Diff of the two buildlogs: -- --- b1/build.log 2025-10-30 18:19:33.129571148 +0000 +++ b2/build.log 2025-10-30 18:23:33.081864342 +0000 @@ -1,8 +1,2550 @@ I: pbuilder: network access will be disabled during build -I: Current time: Thu Oct 30 06:16:05 -12 2025 -I: pbuilder-time-stamp: 1761848165 +I: Current time: Fri Oct 31 08:19:35 +14 2025 +I: pbuilder-time-stamp: 1761848375 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/forky-reproducible-base.tgz] +tar: ./build: time stamp 2026-11-29 07:50:53 is 34039876.327404829 s in the future +tar: ./dev/ptmx: time stamp 2026-11-29 07:46:28 is 34039611.327214159 s in the future +tar: ./dev/zero: time stamp 2026-11-29 07:46:28 is 34039611.327090203 s in the future +tar: ./dev/null: time stamp 2026-11-29 07:46:28 is 34039611.327035812 s in the future +tar: ./dev/console: time stamp 2026-11-29 07:46:28 is 34039611.326954423 s in the future +tar: ./dev/random: time stamp 2026-11-29 07:46:28 is 34039611.32690614 s in the future +tar: ./dev/full: time stamp 2026-11-29 07:46:28 is 34039611.32686272 s in the future +tar: ./dev/pts: time stamp 2026-11-29 07:46:28 is 34039611.326792572 s in the future +tar: ./dev/tty: time stamp 2026-11-29 07:46:28 is 34039611.326713664 s in the future +tar: ./dev/urandom: time stamp 2026-11-29 07:46:28 is 34039611.326663484 s in the future +tar: ./etc/terminfo: time stamp 2026-11-29 07:46:38 is 34039621.326396589 s in the future +tar: ./etc/opt: time stamp 2026-11-29 07:46:28 is 34039611.32489918 s in the future +tar: ./etc/security/opasswd: time stamp 2026-11-29 07:46:40 is 34039623.324481988 s in the future +tar: ./etc/security: time stamp 2026-11-29 07:46:40 is 34039623.32377243 s in the future +tar: ./etc/shadow: time stamp 2026-11-29 07:51:36 is 34039919.3235755 s in the future +tar: ./etc/cron.daily: time stamp 2026-11-30 23:31:29 is 34182712.323365617 s in the future +tar: ./etc/apt/apt.conf.d/80proxy: time stamp 2026-11-29 07:52:40 is 34039983.323131394 s in the future +tar: ./etc/apt/apt.conf.d/15pbuilder: time stamp 2026-11-29 07:50:53 is 34039876.323059998 s in the future +tar: ./etc/apt/apt.conf.d: time stamp 2026-11-30 23:31:29 is 34182712.322960972 s in the future +tar: ./etc/apt/sources.list: time stamp 2026-11-29 07:52:40 is 34039983.322870149 s in the future +tar: ./etc/apt/trusted.gpg.d: time stamp 2026-11-29 07:50:07 is 34039830.322326145 s in the future +tar: ./etc/apt: time stamp 2026-11-29 07:52:40 is 34039983.322234692 s in the future +tar: ./etc/fstab: time stamp 2026-11-29 07:46:28 is 34039611.322163876 s in the future +tar: ./etc/update-motd.d: time stamp 2026-11-29 07:46:39 is 34039622.321478953 s in the future +tar: ./etc/ld.so.conf.d: time stamp 2026-11-29 07:52:00 is 34039943.321165864 s in the future +tar: ./etc/gshadow: time stamp 2026-11-29 07:51:36 is 34039919.320976424 s in the future +tar: ./etc/shells: time stamp 2026-11-29 07:46:39 is 34039622.320883559 s in the future +tar: ./etc/default: time stamp 2026-11-29 07:51:35 is 34039918.320500069 s in the future +tar: ./etc/group: time stamp 2026-11-29 07:51:36 is 34039919.320382199 s in the future +tar: ./etc/ld.so.cache: time stamp 2026-12-02 23:31:27 is 34355510.320188707 s in the future +tar: ./etc/.pwd.lock: time stamp 2026-11-29 07:51:36 is 34039919.320025808 s in the future +tar: ./etc/skel: time stamp 2026-11-29 07:46:39 is 34039622.319798874 s in the future +tar: ./etc/group-: time stamp 2026-11-29 07:46:28 is 34039611.319682853 s in the future +tar: ./etc/subuid: time stamp 2026-11-29 07:46:28 is 34039611.319625756 s in the future +tar: ./etc/logrotate.d: time stamp 2026-11-30 23:31:29 is 34182712.318963243 s in the future +tar: ./etc/pam.d/common-session: time stamp 2026-11-29 07:46:40 is 34039623.318821957 s in the future +tar: ./etc/pam.d/common-password: time stamp 2026-11-29 07:46:40 is 34039623.318659219 s in the future +tar: ./etc/pam.d/common-account: time stamp 2026-11-29 07:46:40 is 34039623.318559062 s in the future +tar: ./etc/pam.d/common-session-noninteractive: time stamp 2026-11-29 07:46:40 is 34039623.318462908 s in the future +tar: ./etc/pam.d/common-auth: time stamp 2026-11-29 07:46:40 is 34039623.318378798 s in the future +tar: ./etc/pam.d: time stamp 2026-12-02 20:29:29 is 34344592.318157955 s in the future +tar: ./etc/subgid: time stamp 2026-11-29 07:46:28 is 34039611.318061032 s in the future +tar: ./etc/passwd-: time stamp 2026-11-29 07:46:28 is 34039611.317999637 s in the future +tar: ./etc/environment: time stamp 2026-11-29 07:46:40 is 34039623.317907804 s in the future +tar: ./etc/perl/Net: time stamp 2026-11-29 07:50:24 is 34039847.317767052 s in the future +tar: ./etc/perl: time stamp 2026-11-29 07:50:02 is 34039825.317708535 s in the future +tar: ./etc/selinux: time stamp 2026-11-29 07:51:27 is 34039910.317536125 s in the future +tar: ./etc/kernel: time stamp 2026-11-29 07:46:41 is 34039624.31745924 s in the future +tar: ./etc/passwd: time stamp 2026-11-29 07:51:36 is 34039919.317417176 s in the future +tar: ./etc/dpkg/origins/default: time stamp 2026-11-29 07:46:28 is 34039611.317305673 s in the future +tar: ./etc/dpkg/origins: time stamp 2026-11-29 07:46:39 is 34039622.317219574 s in the future +tar: ./etc/dpkg/dpkg.cfg.d/02speedup: time stamp 2026-11-29 07:52:40 is 34039983.316982211 s in the future +tar: ./etc/dpkg/dpkg.cfg.d: time stamp 2026-11-29 07:52:40 is 34039983.316957922 s in the future +tar: ./etc/dpkg: time stamp 2026-11-29 07:50:27 is 34039850.316947102 s in the future +tar: ./media: time stamp 2026-11-29 07:46:28 is 34039611.316721205 s in the future +tar: ./mnt: time stamp 2026-11-29 07:46:28 is 34039611.316656423 s in the future +tar: ./opt: time stamp 2026-11-29 07:46:28 is 34039611.316599232 s in the future +tar: ./root: time stamp 2026-11-29 07:46:28 is 34039611.316424069 s in the future +tar: ./run/lock: time stamp 2026-11-29 07:46:28 is 34039611.316335111 s in the future +tar: ./srv: time stamp 2026-11-29 07:46:28 is 34039611.316271564 s in the future +tar: ./usr/share/base-passwd: time stamp 2026-11-29 07:46:31 is 34039614.315954499 s in the future +tar: ./usr/share/terminfo/a: time stamp 2026-11-29 07:46:37 is 34039620.315772905 s in the future +tar: ./usr/share/terminfo/p: time stamp 2026-11-29 07:46:37 is 34039620.315658035 s in the future +tar: ./usr/share/terminfo/E: time stamp 2026-11-29 07:46:37 is 34039620.315498448 s in the future +tar: ./usr/share/terminfo/s: time stamp 2026-11-29 07:46:37 is 34039620.315057209 s in the future +tar: ./usr/share/terminfo/h: time stamp 2026-11-29 07:46:37 is 34039620.314907926 s in the future +tar: ./usr/share/terminfo/d: time stamp 2026-11-29 07:46:37 is 34039620.314813739 s in the future +tar: ./usr/share/terminfo/l: time stamp 2026-11-29 07:46:37 is 34039620.314650769 s in the future +tar: ./usr/share/terminfo/m: time stamp 2026-11-29 07:46:37 is 34039620.314374329 s in the future +tar: ./usr/share/terminfo/c: time stamp 2026-11-29 07:46:37 is 34039620.314261874 s in the future +tar: ./usr/share/terminfo/v: time stamp 2026-11-29 07:46:37 is 34039620.314035655 s in the future +tar: ./usr/share/terminfo/w: time stamp 2026-11-29 07:46:37 is 34039620.313813275 s in the future +tar: ./usr/share/terminfo/x: time stamp 2026-11-29 07:46:37 is 34039620.313317371 s in the future +tar: ./usr/share/terminfo/t: time stamp 2026-11-29 07:46:37 is 34039620.313174115 s in the future +tar: ./usr/share/terminfo/r: time stamp 2026-11-29 07:46:37 is 34039620.312837182 s in the future +tar: ./usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu: time stamp 2026-12-02 23:31:17 is 34355500.312573402 s in the future +tar: ./usr/share/gdb/auto-load/usr/lib: time stamp 2026-11-29 07:47:59 is 34039702.312547457 s in the future +tar: ./usr/share/gdb/auto-load/usr: time stamp 2026-11-29 07:47:59 is 34039702.312531433 s in the future +tar: ./usr/share/gdb/auto-load/lib/x86_64-linux-gnu: time stamp 2026-11-29 07:47:28 is 34039671.312368194 s in the future +tar: ./usr/share/gdb/auto-load/lib: time stamp 2026-11-29 07:47:28 is 34039671.312344915 s in the future +tar: ./usr/share/gdb/auto-load: time stamp 2026-11-29 07:47:59 is 34039702.312330761 s in the future +tar: ./usr/share/gdb: time stamp 2026-11-29 07:47:28 is 34039671.312316506 s in the future +tar: ./usr/share/info: time stamp 2026-11-29 07:46:38 is 34039621.308147655 s in the future +tar: ./usr/share/build-essential: time stamp 2026-11-29 07:46:42 is 34039625.307988079 s in the future +tar: ./usr/share/libc-bin: time stamp 2026-11-29 07:46:34 is 34039617.307883347 s in the future +tar: ./usr/share/zsh/vendor-completions: time stamp 2026-11-29 07:50:05 is 34039828.307681304 s in the future +tar: ./usr/share/zsh: time stamp 2026-11-29 07:46:47 is 34039630.307656347 s in the future +tar: ./usr/share/apt: time stamp 2026-11-30 23:31:29 is 34182712.307564868 s in the future +tar: ./usr/share/man/fr/man7: time stamp 2026-11-30 23:31:29 is 34182712.307355384 s in the future +tar: ./usr/share/man/fr/man5: time stamp 2026-11-30 23:31:29 is 34182712.301451634 s in the future +tar: ./usr/share/man/fi/man1: time stamp 2026-11-29 07:51:34 is 34039917.301149877 s in the future +tar: ./usr/share/man/id/man8: time stamp 2026-11-29 07:51:34 is 34039917.300567214 s in the future +tar: ./usr/share/man/id/man1: time stamp 2026-11-29 07:51:34 is 34039917.300461108 s in the future +tar: ./usr/share/man/id: time stamp 2026-11-29 07:51:34 is 34039917.300437181 s in the future +tar: ./usr/share/man/uk/man8: time stamp 2026-11-29 07:51:34 is 34039917.29963512 s in the future +tar: ./usr/share/man/uk/man1: time stamp 2026-11-29 07:51:34 is 34039917.298965517 s in the future +tar: ./usr/share/man/uk/man5: time stamp 2026-11-29 07:51:34 is 34039917.298662262 s in the future +tar: ./usr/share/man/uk: time stamp 2026-11-29 07:51:34 is 34039917.298636666 s in the future +tar: ./usr/share/man/pt/man7: time stamp 2026-11-30 23:31:29 is 34182712.298420278 s in the future +tar: ./usr/share/man/pt/man8: time stamp 2026-11-30 23:31:29 is 34182712.297426632 s in the future +tar: ./usr/share/man/pt/man1: time stamp 2026-11-30 23:31:29 is 34182712.295421202 s in the future +tar: ./usr/share/man/pt/man5: time stamp 2026-11-30 23:31:29 is 34182712.293755473 s in the future +tar: ./usr/share/man/pt: time stamp 2026-11-29 07:46:41 is 34039624.293728042 s in the future +tar: ./usr/share/man/sr/man1: time stamp 2026-11-29 07:50:06 is 34039829.293290722 s in the future +tar: ./usr/share/man/sr: time stamp 2026-11-29 07:50:06 is 34039829.293265172 s in the future +tar: ./usr/share/man/nl/man7: time stamp 2026-11-30 23:31:29 is 34182712.293045444 s in the future +tar: ./usr/share/man/nl/man8: time stamp 2026-11-30 23:31:29 is 34182712.292247474 s in the future +tar: ./usr/share/man/nl/man1: time stamp 2026-11-30 23:31:29 is 34182712.290030392 s in the future +tar: ./usr/share/man/nl/man5: time stamp 2026-11-30 23:31:29 is 34182712.288376773 s in the future +tar: ./usr/share/man/nl: time stamp 2026-11-29 07:46:41 is 34039624.288346297 s in the future +tar: ./usr/share/man/zh_CN/man8: time stamp 2026-11-29 07:51:34 is 34039917.287541114 s in the future +tar: ./usr/share/man/zh_CN/man1: time stamp 2026-11-29 07:51:34 is 34039917.287191028 s in the future +tar: ./usr/share/man/zh_CN/man5: time stamp 2026-11-29 07:51:34 is 34039917.286947597 s in the future +tar: ./usr/share/man/zh_CN: time stamp 2026-11-29 07:51:34 is 34039917.286923337 s in the future +tar: ./usr/share/man/tr/man8: time stamp 2026-11-29 07:51:34 is 34039917.286562843 s in the future +tar: ./usr/share/man/tr/man1: time stamp 2026-11-29 07:51:34 is 34039917.286401495 s in the future +tar: ./usr/share/man/tr/man5: time stamp 2026-11-29 07:51:34 is 34039917.286264418 s in the future +tar: ./usr/share/man/tr: time stamp 2026-11-29 07:51:34 is 34039917.286239635 s in the future +tar: ./usr/share/man/hu/man1: time stamp 2026-11-29 07:51:34 is 34039917.286028829 s in the future +tar: ./usr/share/man/hu/man5: time stamp 2026-11-29 07:51:34 is 34039917.285921691 s in the future +tar: ./usr/share/man/hu: time stamp 2026-11-29 07:51:34 is 34039917.28590367 s in the future +tar: ./usr/share/man/sv/man7: time stamp 2026-11-29 07:46:47 is 34039630.285748763 s in the future +tar: ./usr/share/man/sv/man8: time stamp 2026-11-29 07:51:34 is 34039917.285315055 s in the future +tar: ./usr/share/man/sv/man5: time stamp 2026-11-29 07:51:37 is 34039920.281547718 s in the future +tar: ./usr/share/man/es/man5: time stamp 2026-11-30 23:31:29 is 34182712.280324469 s in the future +tar: ./usr/share/man/ro/man8: time stamp 2026-11-29 07:51:37 is 34039920.279930364 s in the future +tar: ./usr/share/man/ro/man1: time stamp 2026-11-29 07:51:45 is 34039928.27928846 s in the future +tar: ./usr/share/man/ro/man5: time stamp 2026-11-29 07:51:37 is 34039920.279147209 s in the future +tar: ./usr/share/man/ro: time stamp 2026-11-29 07:51:37 is 34039920.279123261 s in the future +tar: ./usr/share/man/ko/man8: time stamp 2026-11-29 07:51:34 is 34039917.278947328 s in the future +tar: ./usr/share/man/ko/man1: time stamp 2026-11-29 07:51:34 is 34039917.278388391 s in the future +tar: ./usr/share/man/ko/man5: time stamp 2026-11-29 07:51:34 is 34039917.278274761 s in the future +tar: ./usr/share/man/ko: time stamp 2026-11-29 07:51:34 is 34039917.278250321 s in the future +tar: ./usr/share/man/de/man7: time stamp 2026-11-30 23:31:29 is 34182712.27803021 s in the future +tar: ./usr/share/man/de/man5: time stamp 2026-11-30 23:31:29 is 34182712.27133711 s in the future +tar: ./usr/share/man/zh_TW/man8: time stamp 2026-11-29 07:51:34 is 34039917.26434756 s in the future +tar: ./usr/share/man/zh_TW/man1: time stamp 2026-11-29 07:51:34 is 34039917.26419446 s in the future +tar: ./usr/share/man/zh_TW/man5: time stamp 2026-11-29 07:51:34 is 34039917.264110601 s in the future +tar: ./usr/share/man/zh_TW: time stamp 2026-11-29 07:51:34 is 34039917.26409214 s in the future +tar: ./usr/share/man/pl/man5: time stamp 2026-11-30 23:31:29 is 34182712.242564775 s in the future +tar: ./usr/share/man/ja/man5: time stamp 2026-11-30 23:31:29 is 34182712.240287098 s in the future +tar: ./usr/share/man/it/man5: time stamp 2026-11-30 23:31:29 is 34182712.237275898 s in the future +tar: ./usr/share/man/man3: time stamp 2026-11-29 07:47:29 is 34039672.232765352 s in the future +tar: ./usr/share/man/da/man8: time stamp 2026-11-29 07:51:34 is 34039917.232563943 s in the future +tar: ./usr/share/man/da/man1: time stamp 2026-11-29 07:51:34 is 34039917.232451008 s in the future +tar: ./usr/share/man/da/man5: time stamp 2026-11-29 07:51:37 is 34039920.232344486 s in the future +tar: ./usr/share/man/da: time stamp 2026-11-29 07:51:34 is 34039917.232327544 s in the future +tar: ./usr/share/man/cs/man1: time stamp 2026-11-29 07:51:34 is 34039917.231844214 s in the future +tar: ./usr/share/man/cs/man5: time stamp 2026-11-29 07:51:34 is 34039917.231672643 s in the future +tar: ./usr/share/man/man5: time stamp 2026-12-02 20:29:29 is 34344592.228326514 s in the future +tar: ./usr/share/man/sl/man8: time stamp 2026-11-29 07:46:32 is 34039615.228010542 s in the future +tar: ./usr/share/man/pt_BR/man8: time stamp 2026-11-29 07:51:34 is 34039917.227729929 s in the future +tar: ./usr/share/man/pt_BR/man1: time stamp 2026-11-29 07:51:34 is 34039917.227334008 s in the future +tar: ./usr/share/man/pt_BR/man5: time stamp 2026-11-29 07:51:34 is 34039917.227194357 s in the future +tar: ./usr/share/man/pt_BR: time stamp 2026-11-29 07:51:34 is 34039917.227168384 s in the future +tar: ./usr/share/man/ru/man8: time stamp 2026-11-29 07:51:39 is 34039922.226259767 s in the future +tar: ./usr/share/man/ru/man1: time stamp 2026-11-29 07:51:39 is 34039922.225825821 s in the future +tar: ./usr/share/man/ru/man5: time stamp 2026-11-29 07:51:37 is 34039920.2255217 s in the future +tar: ./usr/share/man/ru: time stamp 2026-11-29 07:51:34 is 34039917.225501498 s in the future +tar: ./usr/share/util-linux/logcheck/ignore.d.server: time stamp 2026-12-02 20:29:28 is 34344591.225329101 s in the future +tar: ./usr/share/keyrings: time stamp 2026-11-29 07:46:47 is 34039630.223182856 s in the future +tar: ./usr/share/tabset: time stamp 2026-11-29 07:46:37 is 34039620.22290095 s in the future +tar: ./usr/share/polkit-1/actions: time stamp 2026-11-29 07:46:32 is 34039615.222704164 s in the future +tar: ./usr/share/pam: time stamp 2026-11-29 07:46:36 is 34039619.222233832 s in the future +tar: ./usr/share/debianutils/shells.d: time stamp 2026-11-29 07:46:32 is 34039615.222016222 s in the future +tar: ./usr/share/debianutils: time stamp 2026-11-29 07:46:32 is 34039615.221992803 s in the future +tar: ./usr/share/fish/completions: time stamp 2026-11-29 07:50:05 is 34039828.221898986 s in the future +tar: ./usr/share/fish: time stamp 2026-11-29 07:50:05 is 34039828.221882165 s in the future +tar: ./usr/share/common-licenses: time stamp 2026-11-29 07:46:31 is 34039614.220118362 s in the future +tar: ./usr/share/menu: time stamp 2026-11-29 07:51:41 is 34039924.219873569 s in the future +tar: ./usr/share/lintian/overrides: time stamp 2026-12-02 23:31:26 is 34355509.21737878 s in the future +tar: ./usr/share/lintian/profiles/dpkg: time stamp 2026-11-29 07:46:32 is 34039615.217187913 s in the future +tar: ./usr/share/lintian: time stamp 2026-11-29 07:46:27 is 34039610.217153413 s in the future +tar: ./usr/share/locale/fr/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182712.189892087 s in the future +tar: ./usr/share/locale/th/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182712.186828209 s in the future +tar: ./usr/share/locale/fi/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182712.172651925 s in the future +tar: ./usr/share/locale/fa/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039619.172489948 s in the future +tar: ./usr/share/locale/id/LC_MESSAGES: time stamp 2026-11-30 05:29:45 is 34117808.159602947 s in the future +tar: ./usr/share/locale/uk/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182712.130650101 s in the future +tar: ./usr/share/locale/pt/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182712.111287756 s in the future +tar: ./usr/share/locale/sr/LC_MESSAGES: time stamp 2026-11-30 05:29:45 is 34117808.091207295 s in the future +tar: ./usr/share/locale/bn_IN/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039619.091030567 s in the future +tar: ./usr/share/locale/as/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039619.090863979 s in the future +tar: ./usr/share/locale/lv/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039619.090623197 s in the future +tar: ./usr/share/locale/sr@latin/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039619.090463627 s in the future +tar: ./usr/share/locale/nl/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182712.075467859 s in the future +tar: ./usr/share/locale/ml/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039619.075281936 s in the future +tar: ./usr/share/locale/my/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039619.075131512 s in the future +tar: ./usr/share/locale/zh_CN/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182712.060590481 s in the future +tar: ./usr/share/locale/he/LC_MESSAGES: time stamp 2026-11-29 07:51:34 is 34039917.060172559 s in the future +tar: ./usr/share/locale/te/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039619.060009216 s in the future +tar: ./usr/share/locale/et/LC_MESSAGES: time stamp 2026-11-29 07:46:38 is 34039621.054998607 s in the future +tar: ./usr/share/locale/nb/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182712.046225854 s in the future +tar: ./usr/share/locale/tr/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182712.031275868 s in the future +tar: ./usr/share/locale/de_CH/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039619.031117237 s in the future +tar: ./usr/share/locale/hu/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182712.023117399 s in the future +tar: ./usr/share/locale/sv/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.998782568 s in the future +tar: ./usr/share/locale/am/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.998616338 s in the future +tar: ./usr/share/locale/be/LC_MESSAGES: time stamp 2026-11-29 07:51:38 is 34039920.996595302 s in the future +tar: ./usr/share/locale/es/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.975643284 s in the future +tar: ./usr/share/locale/lt/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.973130838 s in the future +tar: ./usr/share/locale/sq/LC_MESSAGES: time stamp 2026-11-29 07:51:34 is 34039916.97289779 s in the future +tar: ./usr/share/locale/mr/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.97100187 s in the future +tar: ./usr/share/locale/ro/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.944087093 s in the future +tar: ./usr/share/locale/cy/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.943704682 s in the future +tar: ./usr/share/locale/ne/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.941815871 s in the future +tar: ./usr/share/locale/en@quot/LC_MESSAGES: time stamp 2026-11-29 07:46:31 is 34039613.940434897 s in the future +tar: ./usr/share/locale/ko/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.931478852 s in the future +tar: ./usr/share/locale/de/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.918116498 s in the future +tar: ./usr/share/locale/ia/LC_MESSAGES: time stamp 2026-11-29 07:51:38 is 34039920.917599042 s in the future +tar: ./usr/share/locale/hi/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.917403085 s in the future +tar: ./usr/share/locale/pa/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.917205738 s in the future +tar: ./usr/share/locale/mn/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.91710058 s in the future +tar: ./usr/share/locale/kn/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.91697395 s in the future +tar: ./usr/share/locale/bn/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.916857561 s in the future +tar: ./usr/share/locale/ast/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.91508589 s in the future +tar: ./usr/share/locale/kk/LC_MESSAGES: time stamp 2026-11-29 07:51:34 is 34039916.914365755 s in the future +tar: ./usr/share/locale/zh_TW/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.904674082 s in the future +tar: ./usr/share/locale/nn/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.903570736 s in the future +tar: ./usr/share/locale/zu/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.903448273 s in the future +tar: ./usr/share/locale/ta/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.902961929 s in the future +tar: ./usr/share/locale/pl/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.891450879 s in the future +tar: ./usr/share/locale/zh_HK/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.891328679 s in the future +tar: ./usr/share/locale/ca/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.878444121 s in the future +tar: ./usr/share/locale/oc/LC_MESSAGES: time stamp 2026-11-29 07:46:32 is 34039614.877732927 s in the future +tar: ./usr/share/locale/ar/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.876967095 s in the future +tar: ./usr/share/locale/ka/LC_MESSAGES: time stamp 2026-11-30 05:29:45 is 34117807.872194822 s in the future +tar: ./usr/share/locale/kw_GB/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.872084346 s in the future +tar: ./usr/share/locale/gl/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.868277903 s in the future +tar: ./usr/share/locale/sk/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.862030791 s in the future +tar: ./usr/share/locale/or/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.861903452 s in the future +tar: ./usr/share/locale/ja/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.849306513 s in the future +tar: ./usr/share/locale/it/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.838969064 s in the future +tar: ./usr/share/locale/ur/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.838856124 s in the future +tar: ./usr/share/locale/az/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.838748999 s in the future +tar: ./usr/share/locale/lg/LC_MESSAGES: time stamp 2026-11-29 07:46:32 is 34039614.837723359 s in the future +tar: ./usr/share/locale/is/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.837617621 s in the future +tar: ./usr/share/locale/bg/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.825853902 s in the future +tar: ./usr/share/locale/ku/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.82552621 s in the future +tar: ./usr/share/locale/ky/LC_MESSAGES: time stamp 2026-11-29 07:46:38 is 34039620.825050945 s in the future +tar: ./usr/share/locale/dz/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.823194952 s in the future +tar: ./usr/share/locale/tl/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.822182395 s in the future +tar: ./usr/share/locale/tg/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.822067229 s in the future +tar: ./usr/share/locale/ms/LC_MESSAGES: time stamp 2026-11-30 05:29:45 is 34117807.82148206 s in the future +tar: ./usr/share/locale/da/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.810136293 s in the future +tar: ./usr/share/locale/km/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.808255807 s in the future +tar: ./usr/share/locale/cs/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.796432592 s in the future +tar: ./usr/share/locale/bs/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.796096975 s in the future +tar: ./usr/share/locale/en_GB/LC_MESSAGES: time stamp 2026-11-29 07:51:38 is 34039920.795971534 s in the future +tar: ./usr/share/locale/en_GB: time stamp 2026-11-29 07:51:38 is 34039920.795953778 s in the future +tar: ./usr/share/locale/el/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.791523924 s in the future +tar: ./usr/share/locale/sl/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.786627024 s in the future +tar: ./usr/share/locale/eu/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.783735169 s in the future +tar: ./usr/share/locale/gas.es/LC_MESSAGES: time stamp 2026-11-30 05:29:45 is 34117807.780225612 s in the future +tar: ./usr/share/locale/gas.es: time stamp 2026-11-29 07:46:41 is 34039623.780205335 s in the future +tar: ./usr/share/locale/mk/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.780097363 s in the future +tar: ./usr/share/locale/eo/LC_MESSAGES: time stamp 2026-11-30 05:29:45 is 34117807.775071847 s in the future +tar: ./usr/share/locale/vi/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.762793441 s in the future +tar: ./usr/share/locale/rw/LC_MESSAGES: time stamp 2026-11-30 05:29:45 is 34117807.762568542 s in the future +tar: ./usr/share/locale/rw: time stamp 2026-11-29 07:46:41 is 34039623.762550456 s in the future +tar: ./usr/share/locale/hr/LC_MESSAGES: time stamp 2026-11-30 05:29:45 is 34117807.75367956 s in the future +tar: ./usr/share/locale/gu/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.753551388 s in the future +tar: ./usr/share/locale/pt_BR/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.741960329 s in the future +tar: ./usr/share/locale/af/LC_MESSAGES: time stamp 2026-11-29 07:46:38 is 34039620.741638697 s in the future +tar: ./usr/share/locale/ru/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.718321669 s in the future +tar: ./usr/share/locale/yo/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.718203475 s in the future +tar: ./usr/share/locale/si/LC_MESSAGES: time stamp 2026-11-29 07:46:36 is 34039618.7180811 s in the future +tar: ./usr/share/locale/en@boldquot/LC_MESSAGES: time stamp 2026-11-29 07:46:31 is 34039613.716887969 s in the future +tar: ./usr/share/locale/ga/LC_MESSAGES: time stamp 2026-11-30 23:31:29 is 34182711.712459825 s in the future +tar: ./usr/share/debconf: time stamp 2026-11-29 07:46:32 is 34039614.712250975 s in the future +tar: ./usr/share/pixmaps: time stamp 2026-11-29 07:46:32 is 34039614.712169436 s in the future +tar: ./usr/share/initramfs-tools/hooks: time stamp 2026-11-29 07:51:42 is 34039924.712054712 s in the future +tar: ./usr/share/initramfs-tools: time stamp 2026-11-29 07:51:42 is 34039924.712037272 s in the future +tar: ./usr/share/doc/base-passwd: time stamp 2026-11-29 07:46:31 is 34039613.711779689 s in the future +tar: ./usr/share/doc/gzip: time stamp 2026-11-29 07:46:33 is 34039615.711518269 s in the future +tar: ./usr/share/doc/libdb5.3t64: time stamp 2026-11-29 07:46:35 is 34039617.711369548 s in the future +tar: ./usr/share/doc/libpcre2-8-0: time stamp 2026-11-29 07:46:36 is 34039618.710973285 s in the future +tar: ./usr/share/doc/mawk/examples: time stamp 2026-11-29 07:46:37 is 34039619.710394409 s in the future +tar: ./usr/share/doc/mawk: time stamp 2026-11-29 07:46:37 is 34039619.710238192 s in the future +tar: ./usr/share/doc/libuuid1: time stamp 2026-12-02 20:29:28 is 34344590.709788345 s in the future +tar: ./usr/share/doc/libnettle8t64: time stamp 2026-11-29 07:49:18 is 34039780.708708655 s in the future +tar: ./usr/share/doc/make: time stamp 2026-11-29 07:50:02 is 34039824.707416678 s in the future +tar: ./usr/share/doc/libc-bin: time stamp 2026-11-29 07:46:34 is 34039616.706929185 s in the future +tar: ./usr/share/doc/libfuse3-4: time stamp 2026-11-29 07:51:42 is 34039924.706447675 s in the future +tar: ./usr/share/doc/libgdbm6t64: time stamp 2026-11-29 07:47:30 is 34039672.706234838 s in the future +tar: ./usr/share/doc/libudev1: time stamp 2026-12-02 05:32:28 is 34290770.705792616 s in the future +tar: ./usr/share/doc/libc6: time stamp 2026-11-29 07:46:35 is 34039617.704256625 s in the future +tar: ./usr/share/doc/binutils/bfd: time stamp 2026-11-30 05:29:45 is 34117807.703379052 s in the future +tar: ./usr/share/doc/binutils/ld: time stamp 2026-11-30 05:29:45 is 34117807.70307424 s in the future +tar: ./usr/share/doc/binutils/gas: time stamp 2026-11-30 05:29:45 is 34117807.702720782 s in the future +tar: ./usr/share/doc/binutils/gprof: time stamp 2026-11-30 05:29:45 is 34117807.702514499 s in the future +tar: ./usr/share/doc/binutils: time stamp 2026-11-30 05:29:45 is 34117807.702258341 s in the future +tar: ./usr/share/doc/openssl-provider-legacy: time stamp 2026-11-29 07:46:37 is 34039619.700416041 s in the future +tar: ./usr/share/doc/libboost-iostreams1.83.0: time stamp 2026-11-29 07:51:39 is 34039921.68938269 s in the future +tar: ./usr/share/doc/libcap-ng0: time stamp 2026-11-29 07:46:35 is 34039617.689178808 s in the future +tar: ./usr/share/doc/apt/examples: time stamp 2026-11-30 23:31:29 is 34182711.688827331 s in the future +tar: ./usr/share/doc/apt: time stamp 2026-11-30 23:31:29 is 34182711.688769646 s in the future +tar: ./usr/share/doc/sqv: time stamp 2026-11-29 07:50:05 is 34039827.688591986 s in the future +tar: ./usr/share/doc/libpam-modules-bin: time stamp 2026-11-29 07:46:36 is 34039618.6884295 s in the future +tar: ./usr/share/doc/libc-dev-bin: time stamp 2026-11-29 07:47:28 is 34039670.687927036 s in the future +tar: ./usr/share/doc/libbsd0: time stamp 2026-11-29 07:51:10 is 34039892.687083279 s in the future +tar: ./usr/share/doc/libsqlite3-0: time stamp 2026-11-29 07:46:36 is 34039618.685594127 s in the future +tar: ./usr/share/doc/perl-modules-5.40: time stamp 2026-11-29 07:50:04 is 34039826.684762265 s in the future +tar: ./usr/share/doc/util-linux/releases: time stamp 2026-12-02 20:29:28 is 34344590.681596011 s in the future +tar: ./usr/share/doc/util-linux/examples: time stamp 2026-12-02 20:29:28 is 34344590.681119703 s in the future +tar: ./usr/share/doc/util-linux: time stamp 2026-12-02 20:29:28 is 34344590.680703963 s in the future +tar: ./usr/share/doc/libpam0g: time stamp 2026-11-29 07:46:36 is 34039618.680434667 s in the future +tar: ./usr/share/doc/fuse3: time stamp 2026-11-29 07:51:42 is 34039924.680219732 s in the future +tar: ./usr/share/doc/libacl1: time stamp 2026-11-29 07:46:34 is 34039616.680037085 s in the future +tar: ./usr/share/doc/login.defs: time stamp 2026-11-29 07:51:00 is 34039882.678996759 s in the future +tar: ./usr/share/doc/liblz4-1: time stamp 2026-11-29 07:48:47 is 34039749.678884758 s in the future +tar: ./usr/share/doc/rpcsvc-proto: time stamp 2026-11-29 07:50:05 is 34039827.678732947 s in the future +tar: ./usr/share/doc/libc6-dev: time stamp 2026-11-29 07:47:28 is 34039670.67824196 s in the future +tar: ./usr/share/doc/libhogweed6t64: time stamp 2026-11-29 07:47:43 is 34039685.677217946 s in the future +tar: ./usr/share/doc/libgmp10: time stamp 2026-11-29 07:46:35 is 34039617.67504007 s in the future +tar: ./usr/share/doc/libdpkg-perl: time stamp 2026-11-29 07:47:29 is 34039671.673506803 s in the future +tar: ./usr/share/doc/xz-utils: time stamp 2026-11-29 07:50:06 is 34039828.671947493 s in the future +tar: ./usr/share/doc/libmpc3: time stamp 2026-11-29 07:48:57 is 34039759.671828672 s in the future +tar: ./usr/share/doc/disorderfs: time stamp 2026-11-29 07:51:45 is 34039927.671677593 s in the future +tar: ./usr/share/doc/libcrypt1: time stamp 2026-11-29 07:46:35 is 34039617.671524771 s in the future +tar: ./usr/share/doc/debianutils: time stamp 2026-11-29 07:46:32 is 34039614.671367579 s in the future +tar: ./usr/share/doc/liblastlog2-2: time stamp 2026-12-02 20:29:29 is 34344591.671057115 s in the future +tar: ./usr/share/doc/tar: time stamp 2026-11-29 07:46:38 is 34039620.669151459 s in the future +tar: ./usr/share/doc/libcwidget4: time stamp 2026-11-29 07:51:40 is 34039922.668907301 s in the future +tar: ./usr/share/doc/libfakeroot: time stamp 2026-11-29 07:51:45 is 34039927.668675016 s in the future +tar: ./usr/share/doc/libblkid1: time stamp 2026-12-02 20:29:28 is 34344590.668437911 s in the future +tar: ./usr/share/doc/libbz2-1.0: time stamp 2026-11-29 07:46:34 is 34039616.668262226 s in the future +tar: ./usr/share/doc/bzip2: time stamp 2026-11-29 07:46:42 is 34039624.66807998 s in the future +tar: ./usr/share/doc/libsframe2: time stamp 2026-11-30 05:29:45 is 34117807.667746043 s in the future +tar: ./usr/share/doc/dash: time stamp 2026-11-29 07:46:32 is 34039614.66753691 s in the future +tar: ./usr/share/doc/libsemanage2: time stamp 2026-11-29 07:51:30 is 34039912.667428968 s in the future +tar: ./usr/share/doc/libsepol2: time stamp 2026-11-29 07:51:29 is 34039911.66731736 s in the future +tar: ./usr/share/doc/libaudit-common: time stamp 2026-11-29 07:46:34 is 34039616.667172567 s in the future +tar: ./usr/share/doc/libmd0: time stamp 2026-11-29 07:46:35 is 34039617.66696043 s in the future +tar: ./usr/share/doc/aptitude-common: time stamp 2026-11-29 07:51:39 is 34039921.666838079 s in the future +tar: ./usr/share/doc/libisl23: time stamp 2026-11-29 07:48:00 is 34039702.666668742 s in the future +tar: ./usr/share/doc/liblzma5: time stamp 2026-11-29 07:46:35 is 34039617.665276424 s in the future +tar: ./usr/share/doc/libseccomp2: time stamp 2026-11-29 07:49:58 is 34039820.665100531 s in the future +tar: ./usr/share/doc/libperl5.40: time stamp 2026-11-29 07:49:46 is 34039808.664435539 s in the future +tar: ./usr/share/doc/libtinfo6: time stamp 2026-11-29 07:46:36 is 34039618.662675497 s in the future +tar: ./usr/share/doc/libaudit1: time stamp 2026-11-29 07:46:34 is 34039616.662476879 s in the future +tar: ./usr/share/doc/libmpfr6: time stamp 2026-11-29 07:49:08 is 34039770.65971272 s in the future +tar: ./usr/share/doc/mount/examples: time stamp 2026-12-02 20:29:29 is 34344591.659431824 s in the future +tar: ./usr/share/doc/mount: time stamp 2026-12-02 20:29:29 is 34344591.659266978 s in the future +tar: ./usr/share/doc/diffutils: time stamp 2026-11-29 07:46:32 is 34039614.658799339 s in the future +tar: ./usr/share/doc/libsemanage-common: time stamp 2026-11-29 07:51:23 is 34039905.658570419 s in the future +tar: ./usr/share/doc/libjansson4/examples: time stamp 2026-11-29 07:48:24 is 34039726.658350189 s in the future +tar: ./usr/share/doc/libjansson4: time stamp 2026-11-29 07:48:24 is 34039726.658269754 s in the future +tar: ./usr/share/doc/libgdbm-compat4t64: time stamp 2026-11-29 07:47:30 is 34039672.658037587 s in the future +tar: ./usr/share/doc/libcap2: time stamp 2026-11-29 07:46:35 is 34039617.65786039 s in the future +tar: ./usr/share/doc/findutils: time stamp 2026-11-29 07:46:32 is 34039614.65645239 s in the future +tar: ./usr/share/doc/hostname: time stamp 2026-11-29 07:46:33 is 34039615.65633277 s in the future +tar: ./usr/share/doc/debian-archive-keyring: time stamp 2026-11-29 07:46:47 is 34039629.65613343 s in the future +tar: ./usr/share/doc/sed/examples: time stamp 2026-11-29 07:46:38 is 34039620.65517973 s in the future +tar: ./usr/share/doc/sed: time stamp 2026-11-29 07:46:38 is 34039620.655031368 s in the future +tar: ./usr/share/doc/grep: time stamp 2026-11-29 07:46:33 is 34039615.653643746 s in the future +tar: ./usr/share/doc/debconf: time stamp 2026-11-29 07:46:32 is 34039614.653463214 s in the future +tar: ./usr/share/doc/ncurses-base: time stamp 2026-11-29 07:46:37 is 34039619.651888964 s in the future +tar: ./usr/share/doc/libattr1: time stamp 2026-11-29 07:46:34 is 34039616.651712028 s in the future +tar: ./usr/share/doc/base-files: time stamp 2026-11-29 07:46:31 is 34039613.651460814 s in the future +tar: ./usr/share/doc/libsystemd0: time stamp 2026-12-02 05:32:05 is 34290747.65110739 s in the future +tar: ./usr/share/doc/libxapian30: time stamp 2026-11-29 07:51:41 is 34039923.64840195 s in the future +tar: ./usr/share/doc/libc-l10n: time stamp 2026-11-29 07:51:38 is 34039920.647855016 s in the future +tar: ./usr/share/doc/gcc-15-base/itm: time stamp 2026-12-02 23:31:17 is 34355499.647677456 s in the future +tar: ./usr/share/doc/gcc-15-base/C++: time stamp 2026-12-02 23:31:19 is 34355501.644184609 s in the future +tar: ./usr/share/doc/gcc-15-base/gcc: time stamp 2026-12-02 23:31:17 is 34355499.64315489 s in the future +tar: ./usr/share/doc/gcc-15-base/quadmath: time stamp 2026-12-02 23:31:17 is 34355499.642659473 s in the future +tar: ./usr/share/doc/gcc-15-base/gomp: time stamp 2026-12-02 23:31:17 is 34355499.641076641 s in the future +tar: ./usr/share/doc/gcc-15-base/sanitizer: time stamp 2026-12-02 23:31:17 is 34355499.640862798 s in the future +tar: ./usr/share/doc/gcc-15-base: time stamp 2026-12-02 23:31:17 is 34355499.640844909 s in the future +tar: ./usr/share/doc/libzstd1: time stamp 2026-11-29 07:46:37 is 34039619.640569624 s in the future +tar: ./usr/share/doc/patch: time stamp 2026-11-29 07:50:02 is 34039824.640171878 s in the future +tar: ./usr/share/doc/bash: time stamp 2026-11-29 07:46:31 is 34039613.637726542 s in the future +tar: ./usr/share/doc/ncurses-bin: time stamp 2026-11-29 07:46:37 is 34039619.636174713 s in the future +tar: ./usr/share/doc/linux-libc-dev: time stamp 2026-11-29 07:50:02 is 34039824.630435013 s in the future +tar: ./usr/share/doc/sysvinit-utils: time stamp 2026-11-29 07:46:38 is 34039620.63030244 s in the future +tar: ./usr/share/doc/libpam-runtime: time stamp 2026-11-29 07:46:36 is 34039618.630108494 s in the future +tar: ./usr/share/doc/zlib1g: time stamp 2026-11-29 07:46:38 is 34039620.629822849 s in the future +tar: ./usr/share/doc/libdebconfclient0: time stamp 2026-11-29 07:46:35 is 34039617.62966527 s in the future +tar: ./usr/share/doc/perl: time stamp 2026-11-29 07:50:02 is 34039824.628783589 s in the future +tar: ./usr/share/doc/libsigc++-2.0-0v5: time stamp 2026-11-29 07:51:40 is 34039922.628507768 s in the future +tar: ./usr/share/doc/binutils-common: time stamp 2026-11-30 05:29:45 is 34117807.628135223 s in the future +tar: ./usr/share/doc/libapt-pkg7.0: time stamp 2026-11-30 23:31:29 is 34182711.627749874 s in the future +tar: ./usr/share/doc/libselinux1: time stamp 2026-11-29 07:46:36 is 34039618.627590075 s in the future +tar: ./usr/share/doc/fuse: time stamp 2026-11-29 07:51:43 is 34039925.627274176 s in the future +tar: ./usr/share/doc/libssl3t64: time stamp 2026-11-29 07:46:36 is 34039618.625621201 s in the future +tar: ./usr/share/doc/dpkg-dev: time stamp 2026-11-29 07:46:47 is 34039629.624122433 s in the future +tar: ./usr/share/doc/libcrypt-dev: time stamp 2026-11-29 07:47:29 is 34039671.623877503 s in the future +tar: ./usr/share/doc/libxxhash0: time stamp 2026-11-29 07:50:00 is 34039822.623715012 s in the future +tar: ./usr/share/doc/perl-base: time stamp 2026-11-29 07:46:37 is 34039619.623095467 s in the future +tar: ./usr/share/doc/init-system-helpers: time stamp 2026-11-29 07:46:33 is 34039615.622848757 s in the future +tar: ./usr/share/doc/coreutils: time stamp 2026-11-29 07:46:31 is 34039613.620919474 s in the future +tar: ./usr/share/doc/passwd/examples: time stamp 2026-11-29 07:51:34 is 34039916.619954948 s in the future +tar: ./usr/share/doc/passwd: time stamp 2026-11-29 07:51:34 is 34039916.619779236 s in the future +tar: ./usr/share/doc/adduser/examples/adduser.local.conf.examples/skel: time stamp 2026-11-29 07:51:37 is 34039919.619214758 s in the future +tar: ./usr/share/doc/adduser/examples/adduser.local.conf.examples/skel.other: time stamp 2026-11-29 07:51:37 is 34039919.619134848 s in the future +tar: ./usr/share/doc/adduser/examples/adduser.local.conf.examples: time stamp 2026-11-29 07:51:37 is 34039919.619086761 s in the future +tar: ./usr/share/doc/adduser/examples: time stamp 2026-11-29 07:51:37 is 34039919.618964787 s in the future +tar: ./usr/share/doc/adduser: time stamp 2026-11-29 07:51:37 is 34039919.618868893 s in the future +tar: ./usr/share/doc/dpkg/spec: time stamp 2026-11-29 07:46:47 is 34039629.618489316 s in the future +tar: ./usr/share/doc/dpkg: time stamp 2026-11-29 07:46:47 is 34039629.616721335 s in the future +tar: ./usr/share/doc/libmount1: time stamp 2026-12-02 20:29:28 is 34344590.616437804 s in the future +tar: ./usr/share/doc/libfuse2t64: time stamp 2026-11-29 07:51:43 is 34039925.615945575 s in the future +tar: ./usr/share/doc/fakeroot: time stamp 2026-11-29 07:51:45 is 34039927.615644195 s in the future +tar: ./usr/share/doc/locales-all: time stamp 2026-11-29 07:51:59 is 34039941.615070886 s in the future +tar: ./usr/share/doc/libpam-modules/examples: time stamp 2026-11-29 07:46:36 is 34039618.614886552 s in the future +tar: ./usr/share/doc/libpam-modules: time stamp 2026-11-29 07:46:36 is 34039618.614792064 s in the future +tar: ./usr/share/doc/libsmartcols1: time stamp 2026-12-02 20:29:28 is 34344590.614328879 s in the future +tar: ./usr/share/doc/libctf-nobfd0: time stamp 2026-11-30 05:29:44 is 34117806.613857602 s in the future +tar: ./usr/share/base-files: time stamp 2026-11-29 07:46:31 is 34039613.61356085 s in the future +tar: ./usr/share/perl5/Debian/DebConf/Client: time stamp 2026-11-29 07:46:32 is 34039614.613293321 s in the future +tar: ./usr/share/perl5/Debian: time stamp 2026-11-29 07:51:37 is 34039919.613207713 s in the future +tar: ./usr/share/perl5/Debconf/Format: time stamp 2026-11-29 07:46:32 is 34039614.613073857 s in the future +tar: ./usr/share/perl5/Debconf/DbDriver: time stamp 2026-11-29 07:46:32 is 34039614.612625338 s in the future +tar: ./usr/share/perl5/Debconf/Element/Web: time stamp 2026-11-29 07:46:32 is 34039614.611930133 s in the future +tar: ./usr/share/perl5/Debconf/Element/Teletype: time stamp 2026-11-29 07:46:32 is 34039614.611586942 s in the future +tar: ./usr/share/perl5/Debconf/Element/Gnome: time stamp 2026-11-29 07:46:32 is 34039614.611242817 s in the future +tar: ./usr/share/perl5/Debconf/Element/Noninteractive: time stamp 2026-11-29 07:46:32 is 34039614.610882594 s in the future +tar: ./usr/share/perl5/Debconf/Element/Editor: time stamp 2026-11-29 07:46:32 is 34039614.610547718 s in the future +tar: ./usr/share/perl5/Debconf/Element/Dialog: time stamp 2026-11-29 07:46:32 is 34039614.610206914 s in the future +tar: ./usr/share/perl5/Debconf/Element: time stamp 2026-11-29 07:46:32 is 34039614.610155653 s in the future +tar: ./usr/share/perl5/Debconf/FrontEnd: time stamp 2026-11-29 07:46:32 is 34039614.609516992 s in the future +tar: ./usr/share/perl5/Debconf/Template: time stamp 2026-11-29 07:46:32 is 34039614.609398077 s in the future +tar: ./usr/share/perl5/Debconf/Client: time stamp 2026-11-29 07:46:32 is 34039614.609103397 s in the future +tar: ./usr/share/perl5/Debconf: time stamp 2026-11-29 07:46:32 is 34039614.609003614 s in the future +tar: ./usr/share/perl5/Dpkg/Vendor: time stamp 2026-11-29 07:47:29 is 34039671.60867341 s in the future +tar: ./usr/share/perl5/Dpkg/Deps: time stamp 2026-11-29 07:47:29 is 34039671.608420405 s in the future +tar: ./usr/share/perl5/Dpkg/Control/HashCore: time stamp 2026-11-29 07:47:29 is 34039671.608191585 s in the future +tar: ./usr/share/perl5/Dpkg/Control/Tests: time stamp 2026-11-29 07:47:29 is 34039671.608081172 s in the future +tar: ./usr/share/perl5/Dpkg/Control: time stamp 2026-11-29 07:47:29 is 34039671.607788476 s in the future +tar: ./usr/share/perl5/Dpkg/Interface: time stamp 2026-11-29 07:47:29 is 34039671.607597757 s in the future +tar: ./usr/share/perl5/Dpkg/Shlibs/Objdump: time stamp 2026-11-29 07:47:29 is 34039671.607324401 s in the future +tar: ./usr/share/perl5/Dpkg/Shlibs: time stamp 2026-11-29 07:47:29 is 34039671.607264871 s in the future +tar: ./usr/share/perl5/Dpkg/BuildDriver: time stamp 2026-11-29 07:47:29 is 34039671.606803748 s in the future +tar: ./usr/share/perl5/Dpkg/Changelog/Entry: time stamp 2026-11-29 07:47:29 is 34039671.606616436 s in the future +tar: ./usr/share/perl5/Dpkg/Changelog: time stamp 2026-11-29 07:47:29 is 34039671.606488509 s in the future +tar: ./usr/share/perl5/Dpkg/Archive: time stamp 2026-11-29 07:47:29 is 34039671.606300034 s in the future +tar: ./usr/share/perl5/Dpkg/Dist: time stamp 2026-11-29 07:47:29 is 34039671.606019366 s in the future +tar: ./usr/share/perl5/Dpkg/OpenPGP/Backend: time stamp 2026-11-29 07:47:29 is 34039671.605737543 s in the future +tar: ./usr/share/perl5/Dpkg/OpenPGP: time stamp 2026-11-29 07:47:29 is 34039671.605645134 s in the future +tar: ./usr/share/perl5/Dpkg/Source/Package/V3: time stamp 2026-11-29 07:47:29 is 34039671.605052763 s in the future +tar: ./usr/share/perl5/Dpkg/Source/Package: time stamp 2026-11-29 07:47:29 is 34039671.604927391 s in the future +tar: ./usr/share/perl5/Dpkg/Source: time stamp 2026-11-29 07:47:29 is 34039671.60470644 s in the future +tar: ./usr/share/perl5/Dpkg/Compression: time stamp 2026-11-29 07:47:29 is 34039671.604094097 s in the future +tar: ./usr/share/perl5/Dpkg/Build: time stamp 2026-11-29 07:47:29 is 34039671.603981242 s in the future +tar: ./usr/share/perl5/Dpkg: time stamp 2026-11-29 07:47:29 is 34039671.603936339 s in the future +tar: ./usr/share/perl5: time stamp 2026-11-29 07:47:29 is 34039671.603920129 s in the future +tar: ./usr/share/gcc/python/libstdcxx/v6: time stamp 2026-12-02 23:31:17 is 34355499.602914698 s in the future +tar: ./usr/share/gcc/python/libstdcxx: time stamp 2026-12-02 23:31:17 is 34355499.602859856 s in the future +tar: ./usr/share/gcc/python: time stamp 2026-11-29 07:49:59 is 34039821.602842895 s in the future +tar: ./usr/share/gcc: time stamp 2026-11-29 07:49:59 is 34039821.602832913 s in the future +tar: ./usr/share/perl/5.40.1/User: time stamp 2026-11-29 07:50:04 is 34039826.602559826 s in the future +tar: ./usr/share/perl/5.40.1/Getopt/Long: time stamp 2026-11-29 07:50:04 is 34039826.602285967 s in the future +tar: ./usr/share/perl/5.40.1/Getopt: time stamp 2026-11-29 07:50:04 is 34039826.601698155 s in the future +tar: ./usr/share/perl/5.40.1/Filter: time stamp 2026-11-29 07:50:04 is 34039826.600969603 s in the future +tar: ./usr/share/perl/5.40.1/Carp: time stamp 2026-11-29 07:50:04 is 34039826.60071579 s in the future +tar: ./usr/share/perl/5.40.1/pod: time stamp 2026-11-29 07:50:04 is 34039826.598657199 s in the future +tar: ./usr/share/perl/5.40.1/Thread: time stamp 2026-11-29 07:50:04 is 34039826.598389606 s in the future +tar: ./usr/share/perl/5.40.1/Class: time stamp 2026-11-29 07:50:04 is 34039826.597968665 s in the future +tar: ./usr/share/perl/5.40.1/Perl: time stamp 2026-11-29 07:50:04 is 34039826.597405546 s in the future +tar: ./usr/share/perl/5.40.1/CPAN/LWP: time stamp 2026-11-29 07:50:04 is 34039826.597036897 s in the future +tar: ./usr/share/perl/5.40.1/CPAN/HTTP: time stamp 2026-11-29 07:50:04 is 34039826.596919323 s in the future +tar: ./usr/share/perl/5.40.1/CPAN/Kwalify: time stamp 2026-11-29 07:50:04 is 34039826.596783677 s in the future +tar: ./usr/share/perl/5.40.1/CPAN/FTP: time stamp 2026-11-29 07:50:04 is 34039826.595000275 s in the future +tar: ./usr/share/perl/5.40.1/CPAN/Plugin: time stamp 2026-11-29 07:50:04 is 34039826.594907329 s in the future +tar: ./usr/share/perl/5.40.1/CPAN/Exception: time stamp 2026-11-29 07:50:04 is 34039826.594628278 s in the future +tar: ./usr/share/perl/5.40.1/CPAN/API: time stamp 2026-11-29 07:50:04 is 34039826.594339288 s in the future +tar: ./usr/share/perl/5.40.1/CPAN/Meta/Requirements: time stamp 2026-11-29 07:50:04 is 34039826.593993866 s in the future +tar: ./usr/share/perl/5.40.1/CPAN/Meta/History: time stamp 2026-11-29 07:50:04 is 34039826.593571677 s in the future +tar: ./usr/share/perl/5.40.1/CPAN/Meta: time stamp 2026-11-29 07:50:04 is 34039826.592446951 s in the future +tar: ./usr/share/perl/5.40.1/CPAN: time stamp 2026-11-29 07:50:04 is 34039826.591119854 s in the future +tar: ./usr/share/perl/5.40.1/Net/FTP: time stamp 2026-11-29 07:50:04 is 34039826.590318523 s in the future +tar: ./usr/share/perl/5.40.1/Net: time stamp 2026-11-29 07:50:04 is 34039826.588647259 s in the future +tar: ./usr/share/perl/5.40.1/Digest: time stamp 2026-11-29 07:50:04 is 34039826.588490036 s in the future +tar: ./usr/share/perl/5.40.1/Devel: time stamp 2026-11-29 07:50:04 is 34039826.58837511 s in the future +tar: ./usr/share/perl/5.40.1/overload: time stamp 2026-11-29 07:50:04 is 34039826.588288683 s in the future +tar: ./usr/share/perl/5.40.1/Params: time stamp 2026-11-29 07:50:04 is 34039826.588188422 s in the future +tar: ./usr/share/perl/5.40.1/HTTP: time stamp 2026-11-29 07:50:04 is 34039826.587897109 s in the future +tar: ./usr/share/perl/5.40.1/Term/Table: time stamp 2026-11-29 07:50:04 is 34039826.586655036 s in the future +tar: ./usr/share/perl/5.40.1/Term: time stamp 2026-11-29 07:50:04 is 34039826.586291048 s in the future +tar: ./usr/share/perl/5.40.1/unicore/To: time stamp 2026-11-29 07:50:04 is 34039826.579719111 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/SB: time stamp 2026-11-29 07:50:05 is 34039827.563785877 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Age: time stamp 2026-11-29 07:50:04 is 34039826.563009653 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/EPres: time stamp 2026-11-29 07:50:04 is 34039826.562927771 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/GCB: time stamp 2026-11-29 07:50:04 is 34039826.562653036 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/IDC: time stamp 2026-11-29 07:50:04 is 34039826.562558615 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Nt: time stamp 2026-11-29 07:50:05 is 34039827.562422517 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/ExtPict: time stamp 2026-11-29 07:50:04 is 34039826.562344857 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Lower: time stamp 2026-11-29 07:50:05 is 34039827.562221387 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/QMark: time stamp 2026-11-29 07:50:05 is 34039827.562144104 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/NFKDQC: time stamp 2026-11-29 07:50:05 is 34039827.562024341 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Cased: time stamp 2026-11-29 07:50:04 is 34039826.561942271 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Bc: time stamp 2026-11-29 07:50:04 is 34039826.561452437 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Perl: time stamp 2026-11-29 07:50:05 is 34039827.560553534 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Upper: time stamp 2026-11-29 07:50:05 is 34039827.560459464 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Nv: time stamp 2026-11-29 07:50:05 is 34039827.558005827 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/GrExt: time stamp 2026-11-29 07:50:04 is 34039826.557912358 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/EComp: time stamp 2026-11-29 07:50:04 is 34039826.557834867 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/SD: time stamp 2026-11-29 07:50:05 is 34039827.557680468 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/CI: time stamp 2026-11-29 07:50:04 is 34039826.557592332 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Hyphen: time stamp 2026-11-29 07:50:04 is 34039826.557513089 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/CE: time stamp 2026-11-29 07:50:04 is 34039826.557429791 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Dt: time stamp 2026-11-29 07:50:04 is 34039826.556972059 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Term: time stamp 2026-11-29 07:50:05 is 34039827.556885872 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Lb: time stamp 2026-11-29 07:50:05 is 34039827.556224354 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Ccc: time stamp 2026-11-29 07:50:04 is 34039826.555840625 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/CWT: time stamp 2026-11-29 07:50:04 is 34039826.555736995 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/UIdeo: time stamp 2026-11-29 07:50:05 is 34039827.555658191 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/VS: time stamp 2026-11-29 07:50:05 is 34039827.555572512 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/CWL: time stamp 2026-11-29 07:50:04 is 34039826.555490248 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Jg: time stamp 2026-11-29 07:50:04 is 34039826.554632894 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Dia: time stamp 2026-11-29 07:50:04 is 34039826.554541817 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/IdStatus: time stamp 2026-11-29 07:50:04 is 34039826.554421699 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/BidiM: time stamp 2026-11-29 07:50:04 is 34039826.554334389 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Bpt: time stamp 2026-11-29 07:50:04 is 34039826.554134959 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/NFDQC: time stamp 2026-11-29 07:50:05 is 34039827.554018549 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Hex: time stamp 2026-11-29 07:50:04 is 34039826.553931699 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/EBase: time stamp 2026-11-29 07:50:04 is 34039826.553856806 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/CWKCF: time stamp 2026-11-29 07:50:04 is 34039826.553760932 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Blk: time stamp 2026-11-29 07:50:04 is 34039826.553684879 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/BidiC: time stamp 2026-11-29 07:50:04 is 34039826.553607996 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Vo: time stamp 2026-11-29 07:50:05 is 34039827.553399618 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/IDS: time stamp 2026-11-29 07:50:04 is 34039826.553304535 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/CompEx: time stamp 2026-11-29 07:50:04 is 34039826.553228416 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Gc: time stamp 2026-11-29 07:50:04 is 34039826.551973969 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/IdType: time stamp 2026-11-29 07:50:04 is 34039826.551426103 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/NFKCQC: time stamp 2026-11-29 07:50:05 is 34039827.551219467 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Ea: time stamp 2026-11-29 07:50:04 is 34039826.550878632 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Dash: time stamp 2026-11-29 07:50:04 is 34039826.550796686 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/XIDS: time stamp 2026-11-29 07:50:05 is 34039827.550707433 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/STerm: time stamp 2026-11-29 07:50:05 is 34039827.55056684 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/PatSyn: time stamp 2026-11-29 07:50:05 is 34039827.550457468 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/CWCM: time stamp 2026-11-29 07:50:04 is 34039826.550368073 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Dep: time stamp 2026-11-29 07:50:04 is 34039826.550290557 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Emoji: time stamp 2026-11-29 07:50:04 is 34039826.550213306 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/CWCF: time stamp 2026-11-29 07:50:04 is 34039826.550123906 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Math: time stamp 2026-11-29 07:50:05 is 34039827.550046956 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Scx: time stamp 2026-11-29 07:50:05 is 34039827.547440469 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Alpha: time stamp 2026-11-29 07:50:04 is 34039826.5473089 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/GrBase: time stamp 2026-11-29 07:50:04 is 34039826.547211252 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/PCM: time stamp 2026-11-29 07:50:05 is 34039827.547130457 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/InSC: time stamp 2026-11-29 07:50:04 is 34039826.546334491 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/In: time stamp 2026-11-29 07:50:04 is 34039826.545426263 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Sc: time stamp 2026-11-29 07:50:05 is 34039827.544236447 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/NFCQC: time stamp 2026-11-29 07:50:05 is 34039827.544124994 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/DI: time stamp 2026-11-29 07:50:04 is 34039826.544036091 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/WB: time stamp 2026-11-29 07:50:05 is 34039827.543589654 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Ext: time stamp 2026-11-29 07:50:04 is 34039826.543474953 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/InPC: time stamp 2026-11-29 07:50:04 is 34039826.543034064 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Hst: time stamp 2026-11-29 07:50:04 is 34039826.542949059 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Jt: time stamp 2026-11-29 07:50:04 is 34039826.542695642 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/Ideo: time stamp 2026-11-29 07:50:04 is 34039826.542580939 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/XIDC: time stamp 2026-11-29 07:50:05 is 34039827.542486856 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib/CWU: time stamp 2026-11-29 07:50:04 is 34039826.542396162 s in the future +tar: ./usr/share/perl/5.40.1/unicore/lib: time stamp 2026-11-29 07:50:04 is 34039826.542380047 s in the future +tar: ./usr/share/perl/5.40.1/unicore: time stamp 2026-11-29 07:50:05 is 34039827.542240776 s in the future +tar: ./usr/share/perl/5.40.1/App/Prove/State/Result: time stamp 2026-11-29 07:50:04 is 34039826.542088559 s in the future +tar: ./usr/share/perl/5.40.1/App/Prove/State: time stamp 2026-11-29 07:50:04 is 34039826.542033704 s in the future +tar: ./usr/share/perl/5.40.1/App/Prove: time stamp 2026-11-29 07:50:04 is 34039826.541976541 s in the future +tar: ./usr/share/perl/5.40.1/App: time stamp 2026-11-29 07:50:04 is 34039826.541585138 s in the future +tar: ./usr/share/perl/5.40.1/IPC: time stamp 2026-11-29 07:50:04 is 34039826.540854294 s in the future +tar: ./usr/share/perl/5.40.1/Encode: time stamp 2026-11-29 07:50:04 is 34039826.5374673 s in the future +tar: ./usr/share/perl/5.40.1/File: time stamp 2026-11-29 07:50:04 is 34039826.535137907 s in the future +tar: ./usr/share/perl/5.40.1/Compress: time stamp 2026-11-29 07:50:04 is 34039826.534702659 s in the future +tar: ./usr/share/perl/5.40.1/DBM_Filter: time stamp 2026-11-29 07:50:04 is 34039826.534318456 s in the future +tar: ./usr/share/perl/5.40.1/PerlIO/via: time stamp 2026-11-29 07:50:04 is 34039826.533163659 s in the future +tar: ./usr/share/perl/5.40.1/PerlIO: time stamp 2026-11-29 07:50:04 is 34039826.533144342 s in the future +tar: ./usr/share/perl/5.40.1/warnings: time stamp 2026-11-29 07:50:05 is 34039827.532739021 s in the future +tar: ./usr/share/perl/5.40.1/Unicode/Collate/CJK: time stamp 2026-11-29 07:50:04 is 34039826.526156372 s in the future +tar: ./usr/share/perl/5.40.1/Unicode/Collate/Locale: time stamp 2026-11-29 07:50:04 is 34039826.520677426 s in the future +tar: ./usr/share/perl/5.40.1/Unicode/Collate: time stamp 2026-11-29 07:50:04 is 34039826.510115759 s in the future +tar: ./usr/share/perl/5.40.1/Unicode: time stamp 2026-11-29 07:50:04 is 34039826.510095136 s in the future +tar: ./usr/share/perl/5.40.1/TAP/Formatter/Console: time stamp 2026-11-29 07:50:04 is 34039826.5095447 s in the future +tar: ./usr/share/perl/5.40.1/TAP/Formatter/File: time stamp 2026-11-29 07:50:04 is 34039826.509424725 s in the future +tar: ./usr/share/perl/5.40.1/TAP/Formatter: time stamp 2026-11-29 07:50:04 is 34039826.509285348 s in the future +tar: ./usr/share/perl/5.40.1/TAP/Harness: time stamp 2026-11-29 07:50:04 is 34039826.509062535 s in the future +tar: ./usr/share/perl/5.40.1/TAP/Parser/Scheduler: time stamp 2026-11-29 07:50:04 is 34039826.508752751 s in the future +tar: ./usr/share/perl/5.40.1/TAP/Parser/Iterator: time stamp 2026-11-29 07:50:04 is 34039826.50859721 s in the future +tar: ./usr/share/perl/5.40.1/TAP/Parser/Result: time stamp 2026-11-29 07:50:04 is 34039826.508270605 s in the future +tar: ./usr/share/perl/5.40.1/TAP/Parser/SourceHandler: time stamp 2026-11-29 07:50:04 is 34039826.507988305 s in the future +tar: ./usr/share/perl/5.40.1/TAP/Parser/YAMLish: time stamp 2026-11-29 07:50:04 is 34039826.507858004 s in the future +tar: ./usr/share/perl/5.40.1/TAP/Parser: time stamp 2026-11-29 07:50:04 is 34039826.507720554 s in the future +tar: ./usr/share/perl/5.40.1/TAP: time stamp 2026-11-29 07:50:04 is 34039826.507633025 s in the future +tar: ./usr/share/perl/5.40.1/Time: time stamp 2026-11-29 07:50:04 is 34039826.507415102 s in the future +tar: ./usr/share/perl/5.40.1/Exporter: time stamp 2026-11-29 07:50:04 is 34039826.507171719 s in the future +tar: ./usr/share/perl/5.40.1/Archive/Tar: time stamp 2026-11-29 07:50:04 is 34039826.507011765 s in the future +tar: ./usr/share/perl/5.40.1/Archive: time stamp 2026-11-29 07:50:04 is 34039826.506572787 s in the future +tar: ./usr/share/perl/5.40.1/Text: time stamp 2026-11-29 07:50:04 is 34039826.505681549 s in the future +tar: ./usr/share/perl/5.40.1/Pod/Perldoc: time stamp 2026-11-29 07:50:04 is 34039826.503331833 s in the future +tar: ./usr/share/perl/5.40.1/Pod/Text: time stamp 2026-11-29 07:50:04 is 34039826.503170484 s in the future +tar: ./usr/share/perl/5.40.1/Pod/Html: time stamp 2026-11-29 07:50:04 is 34039826.503076978 s in the future +tar: ./usr/share/perl/5.40.1/Pod/Simple: time stamp 2026-11-29 07:50:04 is 34039826.499544049 s in the future +tar: ./usr/share/perl/5.40.1/Pod: time stamp 2026-11-29 07:50:04 is 34039826.499289347 s in the future +tar: ./usr/share/perl/5.40.1/JSON/PP: time stamp 2026-11-29 07:50:04 is 34039826.498977634 s in the future +tar: ./usr/share/perl/5.40.1/JSON: time stamp 2026-11-29 07:50:04 is 34039826.498396392 s in the future +tar: ./usr/share/perl/5.40.1/Locale/Maketext: time stamp 2026-11-29 07:50:04 is 34039826.497282709 s in the future +tar: ./usr/share/perl/5.40.1/Locale: time stamp 2026-11-29 07:50:04 is 34039826.49726256 s in the future +tar: ./usr/share/perl/5.40.1/B: time stamp 2026-11-29 07:50:04 is 34039826.495557863 s in the future +tar: ./usr/share/perl/5.40.1/Module/CoreList: time stamp 2026-11-29 07:50:04 is 34039826.489691662 s in the future +tar: ./usr/share/perl/5.40.1/Module/Load: time stamp 2026-11-29 07:50:04 is 34039826.489584798 s in the future +tar: ./usr/share/perl/5.40.1/Module: time stamp 2026-11-29 07:50:04 is 34039826.489222845 s in the future +tar: ./usr/share/perl/5.40.1/version: time stamp 2026-11-29 07:50:05 is 34039827.488978184 s in the future +tar: ./usr/share/perl/5.40.1/Config/Perl: time stamp 2026-11-29 07:50:04 is 34039826.488856857 s in the future +tar: ./usr/share/perl/5.40.1/Config: time stamp 2026-11-29 07:50:04 is 34039826.488807517 s in the future +tar: ./usr/share/perl/5.40.1/ExtUtils/MakeMaker: time stamp 2026-11-29 07:50:04 is 34039826.48847222 s in the future +tar: ./usr/share/perl/5.40.1/ExtUtils/CBuilder/Platform/Windows: time stamp 2026-11-29 07:50:04 is 34039826.485672614 s in the future +tar: ./usr/share/perl/5.40.1/ExtUtils/CBuilder/Platform: time stamp 2026-11-29 07:50:04 is 34039826.485446225 s in the future +tar: ./usr/share/perl/5.40.1/ExtUtils/CBuilder: time stamp 2026-11-29 07:50:04 is 34039826.485428468 s in the future +tar: ./usr/share/perl/5.40.1/ExtUtils/Typemaps: time stamp 2026-11-29 07:50:04 is 34039826.484931993 s in the future +tar: ./usr/share/perl/5.40.1/ExtUtils/Constant: time stamp 2026-11-29 07:50:04 is 34039826.48450529 s in the future +tar: ./usr/share/perl/5.40.1/ExtUtils/Command: time stamp 2026-11-29 07:50:04 is 34039826.484269293 s in the future +tar: ./usr/share/perl/5.40.1/ExtUtils/Liblist: time stamp 2026-11-29 07:50:04 is 34039826.483882168 s in the future +tar: ./usr/share/perl/5.40.1/ExtUtils/ParseXS: time stamp 2026-11-29 07:50:04 is 34039826.481868939 s in the future +tar: ./usr/share/perl/5.40.1/ExtUtils: time stamp 2026-11-29 07:50:04 is 34039826.481737923 s in the future +tar: ./usr/share/perl/5.40.1/autodie/Scope: time stamp 2026-11-29 07:50:04 is 34039826.481432168 s in the future +tar: ./usr/share/perl/5.40.1/autodie/exception: time stamp 2026-11-29 07:50:04 is 34039826.481347782 s in the future +tar: ./usr/share/perl/5.40.1/autodie: time stamp 2026-11-29 07:50:04 is 34039826.481251577 s in the future +tar: ./usr/share/perl/5.40.1/Attribute: time stamp 2026-11-29 07:50:04 is 34039826.481053655 s in the future +tar: ./usr/share/perl/5.40.1/I18N/LangTags: time stamp 2026-11-29 07:50:04 is 34039826.480505975 s in the future +tar: ./usr/share/perl/5.40.1/I18N: time stamp 2026-11-29 07:50:04 is 34039826.480259684 s in the future +tar: ./usr/share/perl/5.40.1/encoding: time stamp 2026-11-29 07:50:04 is 34039826.479722678 s in the future +tar: ./usr/share/perl/5.40.1/Math/BigFloat: time stamp 2026-11-29 07:50:04 is 34039826.479491867 s in the future +tar: ./usr/share/perl/5.40.1/Math/BigInt: time stamp 2026-11-29 07:50:04 is 34039826.478531586 s in the future +tar: ./usr/share/perl/5.40.1/Math/BigRat: time stamp 2026-11-29 07:50:04 is 34039826.478366673 s in the future +tar: ./usr/share/perl/5.40.1/Math: time stamp 2026-11-29 07:50:04 is 34039826.474258493 s in the future +tar: ./usr/share/perl/5.40.1/Test/Builder/Tester: time stamp 2026-11-29 07:50:04 is 34039826.473603102 s in the future +tar: ./usr/share/perl/5.40.1/Test/Builder/IO: time stamp 2026-11-29 07:50:04 is 34039826.473383434 s in the future +tar: ./usr/share/perl/5.40.1/Test/Builder: time stamp 2026-11-29 07:50:04 is 34039826.47336519 s in the future +tar: ./usr/share/perl/5.40.1/Test/Tester: time stamp 2026-11-29 07:50:04 is 34039826.472791346 s in the future +tar: ./usr/share/perl/5.40.1/Test/use: time stamp 2026-11-29 07:50:04 is 34039826.472268433 s in the future +tar: ./usr/share/perl/5.40.1/Test: time stamp 2026-11-29 07:50:04 is 34039826.472068716 s in the future +tar: ./usr/share/perl/5.40.1/Parse/CPAN: time stamp 2026-11-29 07:50:04 is 34039826.471631744 s in the future +tar: ./usr/share/perl/5.40.1/Parse: time stamp 2026-11-29 07:50:03 is 34039825.471612748 s in the future +tar: ./usr/share/perl/5.40.1/Search: time stamp 2026-11-29 07:50:04 is 34039826.471419506 s in the future +tar: ./usr/share/perl/5.40.1/Memoize: time stamp 2026-11-29 07:50:04 is 34039826.471110387 s in the future +tar: ./usr/share/perl/5.40.1/Tie/Hash: time stamp 2026-11-29 07:50:04 is 34039826.470879043 s in the future +tar: ./usr/share/perl/5.40.1/Tie: time stamp 2026-11-29 07:50:04 is 34039826.470362499 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Event/TAP: time stamp 2026-11-29 07:50:04 is 34039826.469409973 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Event: time stamp 2026-11-29 07:50:04 is 34039826.469020433 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Formatter: time stamp 2026-11-29 07:50:04 is 34039826.468853859 s in the future +tar: ./usr/share/perl/5.40.1/Test2/IPC/Driver: time stamp 2026-11-29 07:50:04 is 34039826.468708749 s in the future +tar: ./usr/share/perl/5.40.1/Test2/IPC: time stamp 2026-11-29 07:50:04 is 34039826.468620801 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Workflow/Task: time stamp 2026-11-29 07:50:04 is 34039826.468311789 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Workflow: time stamp 2026-11-29 07:50:04 is 34039826.468293475 s in the future +tar: ./usr/share/perl/5.40.1/Test2/EventFacet/Info: time stamp 2026-11-29 07:50:04 is 34039826.467831258 s in the future +tar: ./usr/share/perl/5.40.1/Test2/EventFacet: time stamp 2026-11-29 07:50:04 is 34039826.467708362 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Hub/Interceptor: time stamp 2026-11-29 07:50:04 is 34039826.467571795 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Hub: time stamp 2026-11-29 07:50:04 is 34039826.467487302 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Plugin: time stamp 2026-11-29 07:50:04 is 34039826.467212115 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Bundle: time stamp 2026-11-29 07:50:04 is 34039826.466988794 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Require: time stamp 2026-11-29 07:50:04 is 34039826.466516747 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Compare: time stamp 2026-11-29 07:50:04 is 34039826.465357896 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Tools: time stamp 2026-11-29 07:50:04 is 34039826.464229781 s in the future +tar: ./usr/share/perl/5.40.1/Test2/AsyncSubtest/Event: time stamp 2026-11-29 07:50:04 is 34039826.464027548 s in the future +tar: ./usr/share/perl/5.40.1/Test2/AsyncSubtest: time stamp 2026-11-29 07:50:04 is 34039826.463977996 s in the future +tar: ./usr/share/perl/5.40.1/Test2/API/InterceptResult: time stamp 2026-11-29 07:50:04 is 34039826.463655754 s in the future +tar: ./usr/share/perl/5.40.1/Test2/API: time stamp 2026-11-29 07:50:04 is 34039826.463322998 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Manual/Tooling/Plugin: time stamp 2026-11-29 07:50:04 is 34039826.462381675 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Manual/Tooling: time stamp 2026-11-29 07:50:04 is 34039826.462265988 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Manual/Testing: time stamp 2026-11-29 07:50:04 is 34039826.462074478 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Manual/Anatomy: time stamp 2026-11-29 07:50:04 is 34039826.461792654 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Manual: time stamp 2026-11-29 07:50:04 is 34039826.461740664 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Util/Table: time stamp 2026-11-29 07:50:04 is 34039826.461059362 s in the future +tar: ./usr/share/perl/5.40.1/Test2/Util: time stamp 2026-11-29 07:50:04 is 34039826.460969403 s in the future +tar: ./usr/share/perl/5.40.1/Test2: time stamp 2026-11-29 07:50:04 is 34039826.460924656 s in the future +tar: ./usr/share/perl/5.40.1/IO/Compress/Adapter: time stamp 2026-11-29 07:50:04 is 34039826.460674037 s in the future +tar: ./usr/share/perl/5.40.1/IO/Compress/Zip: time stamp 2026-11-29 07:50:04 is 34039826.460486174 s in the future +tar: ./usr/share/perl/5.40.1/IO/Compress/Base: time stamp 2026-11-29 07:50:04 is 34039826.460150892 s in the future +tar: ./usr/share/perl/5.40.1/IO/Compress/Zlib: time stamp 2026-11-29 07:50:04 is 34039826.459223245 s in the future +tar: ./usr/share/perl/5.40.1/IO/Compress/Gzip: time stamp 2026-11-29 07:50:04 is 34039826.459110825 s in the future +tar: ./usr/share/perl/5.40.1/IO/Compress: time stamp 2026-11-29 07:50:04 is 34039826.459092631 s in the future +tar: ./usr/share/perl/5.40.1/IO/Socket: time stamp 2026-11-29 07:50:04 is 34039826.458943742 s in the future +tar: ./usr/share/perl/5.40.1/IO/Uncompress/Adapter: time stamp 2026-11-29 07:50:04 is 34039826.458613349 s in the future +tar: ./usr/share/perl/5.40.1/IO/Uncompress: time stamp 2026-11-29 07:50:04 is 34039826.457267674 s in the future +tar: ./usr/share/perl/5.40.1/IO: time stamp 2026-11-29 07:50:04 is 34039826.457249177 s in the future +tar: ./usr/share/perl/5.40.1: time stamp 2026-11-29 07:50:05 is 34039827.45723846 s in the future +tar: ./usr/share/perl: time stamp 2026-11-29 07:50:05 is 34039827.457228274 s in the future +tar: ./usr/share/bash-completion/completions: time stamp 2026-12-02 20:29:29 is 34344591.455219017 s in the future +tar: ./usr/share/dpkg/sh: time stamp 2026-11-29 07:46:32 is 34039614.454955542 s in the future +tar: ./usr/share/dpkg: time stamp 2026-11-29 07:47:29 is 34039671.454468809 s in the future +tar: ./usr/share/doc-base: time stamp 2026-11-29 07:46:32 is 34039614.454359912 s in the future +tar: ./usr/share/bug/binutils: time stamp 2026-11-30 05:29:45 is 34117807.454250078 s in the future +tar: ./usr/share/bug/apt: time stamp 2026-11-30 23:31:29 is 34182711.454163188 s in the future +tar: ./usr/share/bug/init-system-helpers: time stamp 2026-11-29 07:46:33 is 34039615.454088785 s in the future +tar: ./usr/share/bug: time stamp 2026-11-29 07:51:41 is 34039923.454042067 s in the future +tar: ./usr/share/pam-configs: time stamp 2026-11-29 07:46:36 is 34039618.453938761 s in the future +tar: ./usr/local/src: time stamp 2026-11-29 07:46:28 is 34039610.420227388 s in the future +tar: ./usr/local/share/man: time stamp 2026-11-29 07:46:28 is 34039610.42014334 s in the future +tar: ./usr/local/share: time stamp 2026-11-29 07:46:28 is 34039610.420126123 s in the future +tar: ./usr/local/games: time stamp 2026-11-29 07:46:28 is 34039610.420087208 s in the future +tar: ./usr/local/man: time stamp 2026-11-29 07:46:28 is 34039610.420055183 s in the future +tar: ./usr/local/sbin: time stamp 2026-11-29 07:46:28 is 34039610.420011834 s in the future +tar: ./usr/local/include: time stamp 2026-11-29 07:46:28 is 34039610.419969598 s in the future +tar: ./usr/local/libexec: time stamp 2026-11-29 07:46:28 is 34039610.419925929 s in the future +tar: ./usr/local/etc: time stamp 2026-11-29 07:46:28 is 34039610.419881074 s in the future +tar: ./usr/local/lib: time stamp 2026-11-29 07:46:28 is 34039610.419835278 s in the future +tar: ./usr/local/bin: time stamp 2026-11-29 07:46:28 is 34039610.419772827 s in the future +tar: ./usr/local: time stamp 2026-11-29 07:46:28 is 34039610.419756774 s in the future +tar: ./usr/sbin/policy-rc.d: time stamp 2026-11-29 07:50:53 is 34039875.412964526 s in the future +tar: ./usr/include/x86_64-linux-gnu/bits/types: time stamp 2026-11-29 07:47:28 is 34039670.372950637 s in the future +tar: ./usr/include/x86_64-linux-gnu/bits/platform: time stamp 2026-11-29 07:47:28 is 34039670.370966687 s in the future +tar: ./usr/include/x86_64-linux-gnu/bits: time stamp 2026-11-29 07:47:28 is 34039670.368903123 s in the future +tar: ./usr/include/x86_64-linux-gnu/gnu: time stamp 2026-11-29 07:47:28 is 34039670.368691036 s in the future +tar: ./usr/include/x86_64-linux-gnu/sys/platform: time stamp 2026-11-29 07:47:28 is 34039670.366330771 s in the future +tar: ./usr/include/x86_64-linux-gnu/sys: time stamp 2026-11-29 07:47:28 is 34039670.365436521 s in the future +tar: ./usr/include/x86_64-linux-gnu/c++/15/bits: time stamp 2026-12-02 23:31:19 is 34355501.362869722 s in the future +tar: ./usr/include/x86_64-linux-gnu/c++/15/ext: time stamp 2026-12-02 23:31:19 is 34355501.362774767 s in the future +tar: ./usr/include/x86_64-linux-gnu/c++/15: time stamp 2026-11-29 07:49:59 is 34039821.362756578 s in the future +tar: ./usr/include/x86_64-linux-gnu/c++: time stamp 2026-11-29 07:49:59 is 34039821.362746026 s in the future +tar: ./usr/include/protocols: time stamp 2026-11-29 07:47:28 is 34039670.362394784 s in the future +tar: ./usr/include/cxl: time stamp 2026-11-29 07:50:01 is 34039823.359866368 s in the future +tar: ./usr/include/netpacket: time stamp 2026-11-29 07:47:28 is 34039670.358628943 s in the future +tar: ./usr/include/netrose: time stamp 2026-11-29 07:47:28 is 34039670.358502904 s in the future +tar: ./usr/include/sound/sof: time stamp 2026-11-29 07:50:01 is 34039823.357201846 s in the future +tar: ./usr/include/sound/intel/avs: time stamp 2026-11-29 07:50:01 is 34039823.356468861 s in the future +tar: ./usr/include/sound/intel: time stamp 2026-11-29 07:50:01 is 34039823.356449059 s in the future +tar: ./usr/include/sound: time stamp 2026-11-29 07:50:01 is 34039823.356151419 s in the future +tar: ./usr/include/asm-generic: time stamp 2026-11-29 07:50:01 is 34039823.351589555 s in the future +tar: ./usr/include/rpc: time stamp 2026-11-29 07:47:28 is 34039670.348567249 s in the future +tar: ./usr/include/fwctl: time stamp 2026-11-29 07:50:01 is 34039823.34835557 s in the future +tar: ./usr/include/regulator: time stamp 2026-11-29 07:50:01 is 34039823.348199266 s in the future +tar: ./usr/include/rdma/hfi: time stamp 2026-11-29 07:50:01 is 34039823.343924562 s in the future +tar: ./usr/include/rdma: time stamp 2026-11-29 07:50:01 is 34039823.343777761 s in the future +tar: ./usr/include/net: time stamp 2026-11-29 07:47:28 is 34039670.34118051 s in the future +tar: ./usr/include/c++/15/bits: time stamp 2026-12-02 23:31:19 is 34355501.310876536 s in the future +tar: ./usr/include/c++/15/tr2: time stamp 2026-12-02 23:31:19 is 34355501.310349034 s in the future +tar: ./usr/include/c++/15/backward: time stamp 2026-12-02 23:31:19 is 34355501.308937075 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/ov_tree_map_: time stamp 2026-12-02 23:31:19 is 34355501.305653501 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/hash_fn: time stamp 2026-12-02 23:31:19 is 34355501.305173179 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/bin_search_tree_: time stamp 2026-12-02 23:31:19 is 34355501.304596521 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/binomial_heap_: time stamp 2026-12-02 23:31:19 is 34355501.304419777 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/unordered_iterator: time stamp 2026-12-02 23:31:19 is 34355501.304242325 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/branch_policy: time stamp 2026-12-02 23:31:19 is 34355501.304096263 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/trie_policy: time stamp 2026-12-02 23:31:19 is 34355501.303796507 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/cc_hash_table_map_: time stamp 2026-12-02 23:31:19 is 34355501.302858575 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/pairing_heap_: time stamp 2026-12-02 23:31:19 is 34355501.302572141 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/resize_policy: time stamp 2026-12-02 23:31:19 is 34355501.302187854 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/binary_heap_: time stamp 2026-12-02 23:31:19 is 34355501.301469491 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/left_child_next_sibling_heap_: time stamp 2026-12-02 23:31:19 is 34355501.3009493 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/rc_binomial_heap_: time stamp 2026-12-02 23:31:19 is 34355501.300627935 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/thin_heap_: time stamp 2026-12-02 23:31:19 is 34355501.300298753 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/gp_hash_table_map_: time stamp 2026-12-02 23:31:19 is 34355501.299373379 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/tree_policy: time stamp 2026-12-02 23:31:19 is 34355501.299173699 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/list_update_map_: time stamp 2026-12-02 23:31:19 is 34355501.298716621 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/pat_trie_: time stamp 2026-12-02 23:31:19 is 34355501.29794064 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/binomial_heap_base_: time stamp 2026-12-02 23:31:19 is 34355501.297621378 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/rb_tree_map_: time stamp 2026-12-02 23:31:19 is 34355501.297099387 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/splay_tree_: time stamp 2026-12-02 23:31:19 is 34355501.296679349 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/list_update_policy: time stamp 2026-12-02 23:31:19 is 34355501.296526069 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail/eq_fn: time stamp 2026-12-02 23:31:19 is 34355501.296383989 s in the future +tar: ./usr/include/c++/15/ext/pb_ds/detail: time stamp 2026-12-02 23:31:19 is 34355501.296366176 s in the future +tar: ./usr/include/c++/15/ext/pb_ds: time stamp 2026-12-02 23:31:19 is 34355501.296255547 s in the future +tar: ./usr/include/c++/15/ext: time stamp 2026-12-02 23:31:19 is 34355501.291939851 s in the future +tar: ./usr/include/c++/15/decimal: time stamp 2026-12-02 23:31:19 is 34355501.290373023 s in the future +tar: ./usr/include/c++/15/tr1: time stamp 2026-12-02 23:31:19 is 34355501.286136746 s in the future +tar: ./usr/include/c++/15/experimental/bits: time stamp 2026-12-02 23:31:19 is 34355501.27949326 s in the future +tar: ./usr/include/c++/15/experimental: time stamp 2026-12-02 23:31:19 is 34355501.277031421 s in the future +tar: ./usr/include/c++/15/parallel: time stamp 2026-12-02 23:31:19 is 34355501.268566898 s in the future +tar: ./usr/include/c++/15/pstl: time stamp 2026-12-02 23:31:19 is 34355501.26574828 s in the future +tar: ./usr/include/c++/15/debug: time stamp 2026-12-02 23:31:19 is 34355501.262562738 s in the future +tar: ./usr/include/c++/15: time stamp 2026-12-02 23:31:19 is 34355501.262486891 s in the future +tar: ./usr/include/c++: time stamp 2026-11-29 07:49:58 is 34039820.262468751 s in the future +tar: ./usr/include/video: time stamp 2026-11-29 07:50:01 is 34039823.262278364 s in the future +tar: ./usr/include/linux/surface_aggregator: time stamp 2026-11-29 07:50:01 is 34039823.260055803 s in the future +tar: ./usr/include/linux/genwqe: time stamp 2026-11-29 07:50:01 is 34039823.256285027 s in the future +tar: ./usr/include/linux/caif: time stamp 2026-11-29 07:50:01 is 34039823.254959053 s in the future +tar: ./usr/include/linux/byteorder: time stamp 2026-11-29 07:50:01 is 34039823.254334319 s in the future +tar: ./usr/include/linux/cifs: time stamp 2026-11-29 07:50:01 is 34039823.253517985 s in the future +tar: ./usr/include/linux/sched: time stamp 2026-11-29 07:50:01 is 34039823.252745205 s in the future +tar: ./usr/include/linux/hdlc: time stamp 2026-11-29 07:50:01 is 34039823.251556057 s in the future +tar: ./usr/include/linux/tc_act: time stamp 2026-11-29 07:50:01 is 34039823.250946239 s in the future +tar: ./usr/include/linux/netfilter_bridge: time stamp 2026-11-29 07:50:01 is 34039823.245398612 s in the future +tar: ./usr/include/linux/isdn: time stamp 2026-11-29 07:50:01 is 34039823.244621797 s in the future +tar: ./usr/include/linux/tc_ematch: time stamp 2026-11-29 07:50:01 is 34039823.24405688 s in the future +tar: ./usr/include/linux/netfilter_ipv6: time stamp 2026-11-29 07:50:01 is 34039823.243195399 s in the future +tar: ./usr/include/linux/media/amlogic: time stamp 2026-11-29 07:50:01 is 34039823.239208798 s in the future +tar: ./usr/include/linux/media/raspberrypi: time stamp 2026-11-29 07:50:01 is 34039823.238946908 s in the future +tar: ./usr/include/linux/media: time stamp 2026-11-29 07:50:01 is 34039823.238928177 s in the future +tar: ./usr/include/linux/mmc: time stamp 2026-11-29 07:50:01 is 34039823.238766384 s in the future +tar: ./usr/include/linux/netfilter_arp: time stamp 2026-11-29 07:50:01 is 34039823.23729465 s in the future +tar: ./usr/include/linux/netfilter_ipv4: time stamp 2026-11-29 07:50:01 is 34039823.235649941 s in the future +tar: ./usr/include/linux/dvb: time stamp 2026-11-29 07:50:01 is 34039823.23525609 s in the future +tar: ./usr/include/linux/hsi: time stamp 2026-11-29 07:50:01 is 34039823.234418927 s in the future +tar: ./usr/include/linux/usb: time stamp 2026-11-29 07:50:01 is 34039823.233478622 s in the future +tar: ./usr/include/linux/can: time stamp 2026-11-29 07:50:01 is 34039823.231085773 s in the future +tar: ./usr/include/linux/sunrpc: time stamp 2026-11-29 07:50:01 is 34039823.229629817 s in the future +tar: ./usr/include/linux/spi: time stamp 2026-11-29 07:50:01 is 34039823.229416091 s in the future +tar: ./usr/include/linux/android: time stamp 2026-11-29 07:50:01 is 34039823.228826776 s in the future +tar: ./usr/include/linux/iio: time stamp 2026-11-29 07:50:01 is 34039823.228421821 s in the future +tar: ./usr/include/linux/counter: time stamp 2026-11-29 07:50:01 is 34039823.226310901 s in the future +tar: ./usr/include/linux/nfsd: time stamp 2026-11-29 07:50:01 is 34039823.224889066 s in the future +tar: ./usr/include/linux/misc: time stamp 2026-11-29 07:50:01 is 34039823.224157836 s in the future +tar: ./usr/include/linux/raid: time stamp 2026-11-29 07:50:01 is 34039823.223810735 s in the future +tar: ./usr/include/linux/netfilter/ipset: time stamp 2026-11-29 07:50:01 is 34039823.220069977 s in the future +tar: ./usr/include/linux/netfilter: time stamp 2026-11-29 07:50:01 is 34039823.218164973 s in the future +tar: ./usr/include/linux: time stamp 2026-11-29 07:50:01 is 34039823.218145784 s in the future +tar: ./usr/include/netipx: time stamp 2026-11-29 07:47:28 is 34039670.218051992 s in the future +tar: ./usr/include/finclude/x86_64-linux-gnu: time stamp 2026-11-29 07:47:28 is 34039670.217859134 s in the future +tar: ./usr/include/finclude: time stamp 2026-11-29 07:47:28 is 34039670.217841564 s in the future +tar: ./usr/include/netatalk: time stamp 2026-11-29 07:47:28 is 34039670.217755685 s in the future +tar: ./usr/include/drm: time stamp 2026-11-29 07:50:01 is 34039823.211885357 s in the future +tar: ./usr/include/netash: time stamp 2026-11-29 07:47:28 is 34039670.211776726 s in the future +tar: ./usr/include/netax25: time stamp 2026-11-29 07:47:28 is 34039670.210557469 s in the future +tar: ./usr/include/mtd: time stamp 2026-11-29 07:50:01 is 34039823.210318414 s in the future +tar: ./usr/include/nfs: time stamp 2026-11-29 07:47:28 is 34039670.210230591 s in the future +tar: ./usr/include/netrom: time stamp 2026-11-29 07:47:28 is 34039670.210149048 s in the future +tar: ./usr/include/scsi/fc: time stamp 2026-11-29 07:50:01 is 34039823.209787858 s in the future +tar: ./usr/include/scsi: time stamp 2026-11-29 07:50:01 is 34039823.209523989 s in the future +tar: ./usr/include/arpa: time stamp 2026-11-29 07:47:28 is 34039670.207152687 s in the future +tar: ./usr/include/rpcsvc: time stamp 2026-11-29 07:50:05 is 34039827.206222633 s in the future +tar: ./usr/include/netiucv: time stamp 2026-11-29 07:47:28 is 34039670.206019206 s in the future +tar: ./usr/include/neteconet: time stamp 2026-11-29 07:47:28 is 34039670.205844913 s in the future +tar: ./usr/include/misc/uacce: time stamp 2026-11-29 07:50:01 is 34039823.205592183 s in the future +tar: ./usr/include/misc: time stamp 2026-11-29 07:50:01 is 34039823.205428725 s in the future +tar: ./usr/include/xen: time stamp 2026-11-29 07:50:01 is 34039823.205197264 s in the future +tar: ./usr/include/netinet: time stamp 2026-11-29 07:47:28 is 34039670.204673915 s in the future +tar: ./usr/libexec/libselinux: time stamp 2026-11-29 07:46:36 is 34039618.202519473 s in the future +tar: ./usr/libexec/gcc/x86_64-linux-gnu/15: time stamp 2026-12-02 23:31:26 is 34355507.240049626 s in the future +tar: ./usr/libexec/gcc/x86_64-linux-gnu: time stamp 2026-11-29 07:46:42 is 34039623.240012764 s in the future +tar: ./usr/libexec/gcc: time stamp 2026-11-29 07:46:42 is 34039623.240001277 s in the future +tar: ./usr/libexec/coreutils: time stamp 2026-11-29 07:46:31 is 34039612.23978837 s in the future +tar: ./usr/libexec/dpkg: time stamp 2026-11-29 07:46:32 is 34039613.23966549 s in the future +tar: ./usr/libexec: time stamp 2026-11-29 07:46:42 is 34039623.239647748 s in the future +tar: ./usr/lib/x86_64-linux-gnu/engines-3: time stamp 2026-11-29 07:46:36 is 34039617.227842293 s in the future +tar: ./usr/lib/x86_64-linux-gnu/security: time stamp 2026-11-29 07:46:36 is 34039617.202218107 s in the future +tar: ./usr/lib/x86_64-linux-gnu/gconv/gconv-modules.d: time stamp 2026-11-29 07:46:35 is 34039616.164769515 s in the future +tar: ./usr/lib/x86_64-linux-gnu/gconv: time stamp 2026-11-29 07:46:35 is 34039616.142715327 s in the future +tar: ./usr/lib/x86_64-linux-gnu/pkgconfig: time stamp 2026-11-29 07:47:29 is 34039669.922327354 s in the future +tar: ./usr/lib/x86_64-linux-gnu/ossl-modules: time stamp 2026-11-29 07:46:37 is 34039617.804671614 s in the future +tar: ./usr/lib/x86_64-linux-gnu/libfakeroot: time stamp 2026-11-29 07:51:45 is 34039925.771847117 s in the future +tar: ./usr/lib/x86_64-linux-gnu/gprofng: time stamp 2026-11-30 05:29:44 is 34117804.724971267 s in the future +tar: ./usr/lib/x86_64-linux-gnu/bfd-plugins: time stamp 2026-11-30 05:29:44 is 34117804.56395002 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Data: time stamp 2026-11-29 07:49:45 is 34039805.560648158 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Filter/Util: time stamp 2026-11-29 07:49:45 is 34039805.560065111 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Filter: time stamp 2026-11-29 07:49:32 is 34039792.560045654 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Scalar: time stamp 2026-11-29 07:49:45 is 34039805.559959358 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/bits/types: time stamp 2026-11-29 07:49:46 is 34039806.558536287 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/bits: time stamp 2026-11-29 07:49:46 is 34039806.557203275 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Digest: time stamp 2026-11-29 07:49:45 is 34039805.556555007 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Devel: time stamp 2026-11-29 07:49:45 is 34039805.552056298 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/gnu: time stamp 2026-11-29 07:49:46 is 34039806.551499006 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/IPC: time stamp 2026-11-29 07:49:45 is 34039805.551307535 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Encode/KR: time stamp 2026-11-29 07:49:45 is 34039805.550691181 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Encode/CN: time stamp 2026-11-29 07:49:45 is 34039805.550604534 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Encode/JP: time stamp 2026-11-29 07:49:45 is 34039805.550457239 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Encode/Unicode: time stamp 2026-11-29 07:49:45 is 34039805.550279789 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Encode/MIME/Header: time stamp 2026-11-29 07:49:45 is 34039805.550055633 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Encode/MIME: time stamp 2026-11-29 07:49:45 is 34039805.549961757 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Encode: time stamp 2026-11-29 07:49:45 is 34039805.549774864 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/File/Spec: time stamp 2026-11-29 07:49:45 is 34039805.549346805 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/File: time stamp 2026-11-29 07:49:45 is 34039805.549111505 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Compress/Raw: time stamp 2026-11-29 07:49:45 is 34039805.548714296 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Compress: time stamp 2026-11-29 07:49:29 is 34039789.548694804 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/asm-generic: time stamp 2026-11-29 07:49:45 is 34039805.548279101 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/sys: time stamp 2026-11-29 07:49:46 is 34039806.547766724 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Hash/Util: time stamp 2026-11-29 07:49:45 is 34039805.547383184 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Hash: time stamp 2026-11-29 07:49:45 is 34039805.547355199 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/PerlIO: time stamp 2026-11-29 07:49:45 is 34039805.546959637 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Unicode/Collate: time stamp 2026-11-29 07:49:45 is 34039805.546189345 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Unicode: time stamp 2026-11-29 07:49:45 is 34039805.546169695 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Time: time stamp 2026-11-29 07:49:45 is 34039805.545542158 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Data/Dumper: time stamp 2026-11-29 07:49:45 is 34039805.545298735 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Data: time stamp 2026-11-29 07:49:35 is 34039795.545279551 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Fcntl: time stamp 2026-11-29 07:49:45 is 34039805.54507683 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Filter/Util/Call: time stamp 2026-11-29 07:49:46 is 34039806.544909082 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Filter/Util: time stamp 2026-11-29 07:49:37 is 34039797.544888778 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Filter: time stamp 2026-11-29 07:49:37 is 34039797.544878074 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/NDBM_File: time stamp 2026-11-29 07:49:46 is 34039806.544776657 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Digest/SHA: time stamp 2026-11-29 07:49:45 is 34039805.544292001 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Digest/MD5: time stamp 2026-11-29 07:49:45 is 34039805.544179211 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Digest: time stamp 2026-11-29 07:49:35 is 34039795.544153299 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Devel/Peek: time stamp 2026-11-29 07:49:45 is 34039805.543994411 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Devel: time stamp 2026-11-29 07:49:35 is 34039795.543975417 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/DB_File: time stamp 2026-11-29 07:49:45 is 34039805.543579909 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/POSIX: time stamp 2026-11-29 07:49:46 is 34039806.542792528 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/attributes: time stamp 2026-11-29 07:49:46 is 34039806.542587346 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/ODBM_File: time stamp 2026-11-29 07:49:46 is 34039806.542446358 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/IPC/SysV: time stamp 2026-11-29 07:49:46 is 34039806.542285881 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/IPC: time stamp 2026-11-29 07:49:37 is 34039797.542265886 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/re: time stamp 2026-11-29 07:49:46 is 34039806.536375897 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Encode/KR: time stamp 2026-11-29 07:49:45 is 34039805.522539284 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Encode/CN: time stamp 2026-11-29 07:49:45 is 34039805.510045926 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Encode/JP: time stamp 2026-11-29 07:49:45 is 34039805.494175176 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Encode/Unicode: time stamp 2026-11-29 07:49:45 is 34039805.494026744 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Encode/TW: time stamp 2026-11-29 07:49:45 is 34039805.481809364 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Encode/Symbol: time stamp 2026-11-29 07:49:45 is 34039805.481396863 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Encode/Byte: time stamp 2026-11-29 07:49:45 is 34039805.479108699 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Encode/EBCDIC: time stamp 2026-11-29 07:49:45 is 34039805.47846535 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Encode: time stamp 2026-11-29 07:49:45 is 34039805.478445899 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/File/DosGlob: time stamp 2026-11-29 07:49:45 is 34039805.478324444 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/File/Glob: time stamp 2026-11-29 07:49:46 is 34039806.478183616 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/File: time stamp 2026-11-29 07:49:36 is 34039796.47816548 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Compress/Raw/Bzip2: time stamp 2026-11-29 07:49:45 is 34039805.477942633 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Compress/Raw/Zlib: time stamp 2026-11-29 07:49:45 is 34039805.47721026 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Compress/Raw: time stamp 2026-11-29 07:49:35 is 34039795.477191191 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Compress: time stamp 2026-11-29 07:49:35 is 34039795.477180692 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Opcode: time stamp 2026-11-29 07:49:46 is 34039806.477078904 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Storable: time stamp 2026-11-29 07:49:46 is 34039806.476058883 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Hash/Util/FieldHash: time stamp 2026-11-29 07:49:46 is 34039806.475749264 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Hash/Util: time stamp 2026-11-29 07:49:46 is 34039806.475728895 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Hash: time stamp 2026-11-29 07:49:37 is 34039797.47571825 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/PerlIO/via: time stamp 2026-11-29 07:49:46 is 34039806.475557194 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/PerlIO/encoding: time stamp 2026-11-29 07:49:46 is 34039806.475380733 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/PerlIO/mmap: time stamp 2026-11-29 07:49:46 is 34039806.475210219 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/PerlIO: time stamp 2026-11-29 07:49:38 is 34039798.475192079 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Unicode/Collate: time stamp 2026-11-29 07:49:46 is 34039806.466505708 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Unicode/Normalize: time stamp 2026-11-29 07:49:46 is 34039806.463224619 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Unicode: time stamp 2026-11-29 07:49:39 is 34039799.463203524 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/GDBM_File: time stamp 2026-11-29 07:49:46 is 34039806.462714787 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Time/HiRes: time stamp 2026-11-29 07:49:46 is 34039806.462452877 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Time/Piece: time stamp 2026-11-29 07:49:46 is 34039806.46222263 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Time: time stamp 2026-11-29 07:49:39 is 34039799.462204441 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Socket: time stamp 2026-11-29 07:49:46 is 34039806.461965514 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/mro: time stamp 2026-11-29 07:49:46 is 34039806.461686569 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/B: time stamp 2026-11-29 07:49:45 is 34039805.460990899 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/List/Util: time stamp 2026-11-29 07:49:46 is 34039806.460250845 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/List: time stamp 2026-11-29 07:49:38 is 34039798.460231856 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/MIME/Base64: time stamp 2026-11-29 07:49:46 is 34039806.460101564 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/MIME: time stamp 2026-11-29 07:49:38 is 34039798.460083211 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/I18N/Langinfo: time stamp 2026-11-29 07:49:46 is 34039806.459964608 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/I18N: time stamp 2026-11-29 07:49:37 is 34039797.459946754 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Math/BigInt/FastCalc: time stamp 2026-11-29 07:49:46 is 34039806.459790869 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Math/BigInt: time stamp 2026-11-29 07:49:38 is 34039798.459771699 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Math: time stamp 2026-11-29 07:49:38 is 34039798.459761042 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Cwd: time stamp 2026-11-29 07:49:45 is 34039805.459660352 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Sys/Hostname: time stamp 2026-11-29 07:49:46 is 34039806.45953733 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Sys/Syslog: time stamp 2026-11-29 07:49:46 is 34039806.459432566 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/Sys: time stamp 2026-11-29 07:49:39 is 34039799.459415201 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/threads/shared: time stamp 2026-11-29 07:49:46 is 34039806.458996571 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/threads: time stamp 2026-11-29 07:49:46 is 34039806.458735346 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/IO: time stamp 2026-11-29 07:49:46 is 34039806.458513876 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto/SDBM_File: time stamp 2026-11-29 07:49:46 is 34039806.45829627 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/auto: time stamp 2026-11-29 07:49:40 is 34039800.458277851 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/B: time stamp 2026-11-29 07:49:45 is 34039805.457226968 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/List/Util: time stamp 2026-11-29 07:49:45 is 34039805.457037289 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/List: time stamp 2026-11-29 07:49:45 is 34039805.457018 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Sub: time stamp 2026-11-29 07:49:45 is 34039805.456841219 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/CORE: time stamp 2026-11-29 07:49:45 is 34039805.412357822 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/asm: time stamp 2026-11-29 07:49:45 is 34039805.411658195 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/MIME: time stamp 2026-11-29 07:49:45 is 34039805.41146334 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/I18N: time stamp 2026-11-29 07:49:45 is 34039805.411327395 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Math/BigInt: time stamp 2026-11-29 07:49:45 is 34039805.411178902 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Math: time stamp 2026-11-29 07:49:32 is 34039792.411159513 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/linux: time stamp 2026-11-29 07:49:46 is 34039806.41094879 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/Sys: time stamp 2026-11-29 07:49:45 is 34039805.409837343 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/threads: time stamp 2026-11-29 07:49:46 is 34039806.409599481 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/IO/Socket: time stamp 2026-11-29 07:49:45 is 34039805.408693674 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1/IO: time stamp 2026-11-29 07:49:45 is 34039805.408551387 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/5.40.1: time stamp 2026-11-29 07:49:46 is 34039806.408528757 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/debian-config-data-5.40.1: time stamp 2026-11-29 07:49:46 is 34039806.408251377 s in the future +tar: ./usr/lib/x86_64-linux-gnu/ldscripts: time stamp 2026-11-30 05:29:44 is 34117804.378086254 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/Getopt/Long: time stamp 2026-11-29 07:46:37 is 34039617.376977478 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/Getopt: time stamp 2026-11-29 07:46:37 is 34039617.376870551 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/Carp: time stamp 2026-11-29 07:46:37 is 34039617.376562386 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/Scalar: time stamp 2026-11-29 07:46:37 is 34039617.376417881 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/To: time stamp 2026-11-29 07:46:37 is 34039617.369664364 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/SB: time stamp 2026-11-29 07:46:37 is 34039617.369174923 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Age: time stamp 2026-11-29 07:46:37 is 34039617.36842562 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/EPres: time stamp 2026-11-29 07:46:37 is 34039617.36834439 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/GCB: time stamp 2026-11-29 07:46:37 is 34039617.368039854 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/IDC: time stamp 2026-11-29 07:46:37 is 34039617.367949572 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Nt: time stamp 2026-11-29 07:46:37 is 34039617.367780364 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/ExtPict: time stamp 2026-11-29 07:46:37 is 34039617.367693811 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Lower: time stamp 2026-11-29 07:46:37 is 34039617.367602634 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/QMark: time stamp 2026-11-29 07:46:37 is 34039617.367524986 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/NFKDQC: time stamp 2026-11-29 07:46:37 is 34039617.367398128 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Cased: time stamp 2026-11-29 07:46:37 is 34039617.367320669 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Bc: time stamp 2026-11-29 07:46:37 is 34039617.366851909 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Perl: time stamp 2026-11-29 07:46:37 is 34039617.365932404 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Upper: time stamp 2026-11-29 07:46:37 is 34039617.36584038 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Nv: time stamp 2026-11-29 07:46:37 is 34039617.363419238 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/GrExt: time stamp 2026-11-29 07:46:37 is 34039617.363330585 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/EComp: time stamp 2026-11-29 07:46:37 is 34039617.363242936 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/SD: time stamp 2026-11-29 07:46:37 is 34039617.363131919 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/CI: time stamp 2026-11-29 07:46:37 is 34039617.363016657 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Hyphen: time stamp 2026-11-29 07:46:37 is 34039617.362932 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/CE: time stamp 2026-11-29 07:46:37 is 34039617.362840949 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Dt: time stamp 2026-11-29 07:46:37 is 34039617.362357169 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Term: time stamp 2026-11-29 07:46:37 is 34039617.362274218 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Lb: time stamp 2026-11-29 07:46:37 is 34039617.361583256 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Ccc: time stamp 2026-11-29 07:46:37 is 34039617.361207279 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/CWT: time stamp 2026-11-29 07:46:37 is 34039617.361114753 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/UIdeo: time stamp 2026-11-29 07:46:37 is 34039617.361036052 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/VS: time stamp 2026-11-29 07:46:37 is 34039617.360950099 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/CWL: time stamp 2026-11-29 07:46:37 is 34039617.360869264 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Jg: time stamp 2026-11-29 07:46:37 is 34039617.360124506 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Dia: time stamp 2026-11-29 07:46:37 is 34039617.360008666 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/IdStatus: time stamp 2026-11-29 07:46:37 is 34039617.359874644 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/BidiM: time stamp 2026-11-29 07:46:37 is 34039617.359779719 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Bpt: time stamp 2026-11-29 07:46:37 is 34039617.35963285 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/NFDQC: time stamp 2026-11-29 07:46:37 is 34039617.359498569 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Hex: time stamp 2026-11-29 07:46:37 is 34039617.359408737 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/EBase: time stamp 2026-11-29 07:46:37 is 34039617.359316621 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/CWKCF: time stamp 2026-11-29 07:46:37 is 34039617.359196995 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Blk: time stamp 2026-11-29 07:46:37 is 34039617.359117139 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/BidiC: time stamp 2026-11-29 07:46:37 is 34039617.359036472 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Vo: time stamp 2026-11-29 07:46:37 is 34039617.358837926 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/IDS: time stamp 2026-11-29 07:46:37 is 34039617.358738984 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/CompEx: time stamp 2026-11-29 07:46:37 is 34039617.358658666 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Gc: time stamp 2026-11-29 07:46:37 is 34039617.357430767 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/IdType: time stamp 2026-11-29 07:46:37 is 34039617.356941135 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/NFKCQC: time stamp 2026-11-29 07:46:37 is 34039617.356794797 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Ea: time stamp 2026-11-29 07:46:37 is 34039617.356478249 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Dash: time stamp 2026-11-29 07:46:37 is 34039617.356362356 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/XIDS: time stamp 2026-11-29 07:46:37 is 34039617.356229499 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/STerm: time stamp 2026-11-29 07:46:37 is 34039617.356122732 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/PatSyn: time stamp 2026-11-29 07:46:37 is 34039617.356007145 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/CWCM: time stamp 2026-11-29 07:46:37 is 34039617.355882018 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Dep: time stamp 2026-11-29 07:46:37 is 34039617.355779539 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Emoji: time stamp 2026-11-29 07:46:37 is 34039617.355698119 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/CWCF: time stamp 2026-11-29 07:46:37 is 34039617.355602077 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Math: time stamp 2026-11-29 07:46:37 is 34039617.355522148 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Scx: time stamp 2026-11-29 07:46:37 is 34039617.352950949 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Alpha: time stamp 2026-11-29 07:46:37 is 34039617.352794944 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/GrBase: time stamp 2026-11-29 07:46:37 is 34039617.352672441 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/PCM: time stamp 2026-11-29 07:46:37 is 34039617.352590927 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/InSC: time stamp 2026-11-29 07:46:37 is 34039617.351742336 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/In: time stamp 2026-11-29 07:46:37 is 34039617.350779592 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Sc: time stamp 2026-11-29 07:46:37 is 34039617.349661719 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/NFCQC: time stamp 2026-11-29 07:46:37 is 34039617.349545842 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/DI: time stamp 2026-11-29 07:46:37 is 34039617.349455804 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/WB: time stamp 2026-11-29 07:46:37 is 34039617.349032448 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Ext: time stamp 2026-11-29 07:46:37 is 34039617.348944131 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/InPC: time stamp 2026-11-29 07:46:37 is 34039617.348485322 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Hst: time stamp 2026-11-29 07:46:37 is 34039617.348398761 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Jt: time stamp 2026-11-29 07:46:37 is 34039617.348147136 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/Ideo: time stamp 2026-11-29 07:46:37 is 34039617.34806478 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/XIDC: time stamp 2026-11-29 07:46:37 is 34039617.347940793 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/unicore/lib/CWU: time stamp 2026-11-29 07:46:37 is 34039617.347845355 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/IPC: time stamp 2026-11-29 07:46:37 is 34039617.347698742 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/File/Spec: time stamp 2026-11-29 07:46:37 is 34039617.346817702 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/File: time stamp 2026-11-29 07:46:37 is 34039617.346607024 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/Hash: time stamp 2026-11-29 07:46:37 is 34039617.346410691 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/warnings: time stamp 2026-11-29 07:46:37 is 34039617.346268542 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/Exporter: time stamp 2026-11-29 07:46:37 is 34039617.346059898 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/auto/Fcntl: time stamp 2026-11-29 07:46:37 is 34039617.345918557 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/auto/POSIX: time stamp 2026-11-29 07:46:37 is 34039617.345334812 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/auto/attributes: time stamp 2026-11-29 07:46:37 is 34039617.345169464 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/auto/re: time stamp 2026-11-29 07:46:37 is 34039617.339339629 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/auto/File/Glob: time stamp 2026-11-29 07:46:37 is 34039617.338990126 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/auto/Hash/Util: time stamp 2026-11-29 07:46:37 is 34039617.338779939 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/auto/Socket: time stamp 2026-11-29 07:46:37 is 34039617.338256496 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/auto/List/Util: time stamp 2026-11-29 07:46:37 is 34039617.337764344 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/auto/Cwd: time stamp 2026-11-29 07:46:37 is 34039617.337653383 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/auto/IO: time stamp 2026-11-29 07:46:37 is 34039617.337524897 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/Text: time stamp 2026-11-29 07:46:37 is 34039617.337360845 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/List: time stamp 2026-11-29 07:46:37 is 34039617.337170494 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/Tie: time stamp 2026-11-29 07:46:37 is 34039617.336772583 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/IO/Socket: time stamp 2026-11-29 07:46:37 is 34039617.336235903 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base/IO: time stamp 2026-11-29 07:46:37 is 34039617.336208725 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl-base: time stamp 2026-11-29 07:46:37 is 34039617.33619204 s in the future +tar: ./usr/lib/x86_64-linux-gnu/audit: time stamp 2026-11-29 07:47:28 is 34039668.218286753 s in the future +tar: ./usr/lib/apt/solvers: time stamp 2026-11-30 23:31:29 is 34182709.218047602 s in the future +tar: ./usr/lib/apt/methods: time stamp 2026-11-30 23:31:29 is 34182709.209967019 s in the future +tar: ./usr/lib/systemd/system: time stamp 2026-12-02 20:29:28 is 34344588.208932221 s in the future +tar: ./usr/lib/init: time stamp 2026-11-29 07:46:38 is 34039618.208799118 s in the future +tar: ./usr/lib/tmpfiles.d: time stamp 2026-11-29 07:51:34 is 34039914.208689819 s in the future +tar: ./usr/lib/lsb/init-functions.d: time stamp 2026-11-29 07:46:38 is 34039618.208580668 s in the future +tar: ./usr/lib/lsb: time stamp 2026-11-29 07:46:38 is 34039618.208512879 s in the future +tar: ./usr/lib/locale/sw_KE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.20812963 s in the future +tar: ./usr/lib/locale/fr_BE@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.207746582 s in the future +tar: ./usr/lib/locale/he_IL/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.205503173 s in the future +tar: ./usr/lib/locale/he_IL: time stamp 2026-11-29 07:51:59 is 34039939.205430876 s in the future +tar: ./usr/lib/locale/se_NO/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.204983774 s in the future +tar: ./usr/lib/locale/pap_AW/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.188228561 s in the future +tar: ./usr/lib/locale/ga_IE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.187794697 s in the future +tar: ./usr/lib/locale/zgh_MA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.187004246 s in the future +tar: ./usr/lib/locale/id_ID.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.186454155 s in the future +tar: ./usr/lib/locale/yi_US/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.184214848 s in the future +tar: ./usr/lib/locale/yi_US: time stamp 2026-11-29 07:51:59 is 34039939.184112021 s in the future +tar: ./usr/lib/locale/en_AG/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.183666121 s in the future +tar: ./usr/lib/locale/gv_GB/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.182233273 s in the future +tar: ./usr/lib/locale/ber_DZ/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.181875181 s in the future +tar: ./usr/lib/locale/wae_CH/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.166115216 s in the future +tar: ./usr/lib/locale/nan_TW/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.165358054 s in the future +tar: ./usr/lib/locale/su_ID/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.164660048 s in the future +tar: ./usr/lib/locale/bg_BG.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.164249252 s in the future +tar: ./usr/lib/locale/af_ZA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.147997684 s in the future +tar: ./usr/lib/locale/fil_PH/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.146883964 s in the future +tar: ./usr/lib/locale/th_TH/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.129268224 s in the future +tar: ./usr/lib/locale/th_TH: time stamp 2026-11-29 07:51:59 is 34039939.129130405 s in the future +tar: ./usr/lib/locale/tg_TJ.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.127932933 s in the future +tar: ./usr/lib/locale/eu_ES.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.112115228 s in the future +tar: ./usr/lib/locale/ta_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.109935504 s in the future +tar: ./usr/lib/locale/nhn_MX/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.109559181 s in the future +tar: ./usr/lib/locale/vi_VN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.107159885 s in the future +tar: ./usr/lib/locale/tr_CY.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.090685133 s in the future +tar: ./usr/lib/locale/kk_KZ.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.074874019 s in the future +tar: ./usr/lib/locale/bem_ZM/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.058389211 s in the future +tar: ./usr/lib/locale/crh_RU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.057996895 s in the future +tar: ./usr/lib/locale/ar_AE.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.042157236 s in the future +tar: ./usr/lib/locale/zu_ZA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.041099693 s in the future +tar: ./usr/lib/locale/pa_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.037754388 s in the future +tar: ./usr/lib/locale/bs_BA.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.037311149 s in the future +tar: ./usr/lib/locale/bo_CN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039939.021405756 s in the future +tar: ./usr/lib/locale/nan_TW@latin/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.988906971 s in the future +tar: ./usr/lib/locale/kl_GL/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.972869728 s in the future +tar: ./usr/lib/locale/so_DJ.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.972115157 s in the future +tar: ./usr/lib/locale/nb_NO/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.970812781 s in the future +tar: ./usr/lib/locale/tn_ZA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.970062057 s in the future +tar: ./usr/lib/locale/oc_FR/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.969623261 s in the future +tar: ./usr/lib/locale/uk_UA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.968117697 s in the future +tar: ./usr/lib/locale/ve_ZA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.967689054 s in the future +tar: ./usr/lib/locale/kk_KZ/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.96536332 s in the future +tar: ./usr/lib/locale/kk_KZ: time stamp 2026-11-29 07:51:59 is 34039938.96527552 s in the future +tar: ./usr/lib/locale/unm_US/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.964920417 s in the future +tar: ./usr/lib/locale/cy_GB.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.963425403 s in the future +tar: ./usr/lib/locale/hne_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.94757922 s in the future +tar: ./usr/lib/locale/ka_GE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.945270209 s in the future +tar: ./usr/lib/locale/ka_GE: time stamp 2026-11-29 07:51:59 is 34039938.945191414 s in the future +tar: ./usr/lib/locale/is_IS/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.94478553 s in the future +tar: ./usr/lib/locale/pt_PT@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.944352965 s in the future +tar: ./usr/lib/locale/ks_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.943186841 s in the future +tar: ./usr/lib/locale/br_FR@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.942337395 s in the future +tar: ./usr/lib/locale/uz_UZ.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.940074768 s in the future +tar: ./usr/lib/locale/ltg_LV.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.923438008 s in the future +tar: ./usr/lib/locale/tt_RU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.907474428 s in the future +tar: ./usr/lib/locale/nso_ZA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.888601767 s in the future +tar: ./usr/lib/locale/fr_CA.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.888203278 s in the future +tar: ./usr/lib/locale/be_BY@latin/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.870744935 s in the future +tar: ./usr/lib/locale/bho_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.870314681 s in the future +tar: ./usr/lib/locale/wa_BE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.867751498 s in the future +tar: ./usr/lib/locale/so_DJ/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.866588579 s in the future +tar: ./usr/lib/locale/sah_RU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.865830673 s in the future +tar: ./usr/lib/locale/gl_ES@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.849607175 s in the future +tar: ./usr/lib/locale/sl_SI/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.849198502 s in the future +tar: ./usr/lib/locale/byn_ER/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.848754408 s in the future +tar: ./usr/lib/locale/gbm_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.848359779 s in the future +tar: ./usr/lib/locale/ig_NG/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.847961763 s in the future +tar: ./usr/lib/locale/pl_PL/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.832141926 s in the future +tar: ./usr/lib/locale/doi_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.831614803 s in the future +tar: ./usr/lib/locale/hak_TW/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.829236528 s in the future +tar: ./usr/lib/locale/om_ET/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.828798421 s in the future +tar: ./usr/lib/locale/ne_NP/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.812946063 s in the future +tar: ./usr/lib/locale/an_ES.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.811790584 s in the future +tar: ./usr/lib/locale/cv_RU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.795939553 s in the future +tar: ./usr/lib/locale/so_KE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.779961905 s in the future +tar: ./usr/lib/locale/fur_IT/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.779522646 s in the future +tar: ./usr/lib/locale/eu_ES/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.763720043 s in the future +tar: ./usr/lib/locale/en_DK.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.763299492 s in the future +tar: ./usr/lib/locale/tok/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.762869636 s in the future +tar: ./usr/lib/locale/is_IS.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.762454386 s in the future +tar: ./usr/lib/locale/fo_FO.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.745589749 s in the future +tar: ./usr/lib/locale/wo_SN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.744393443 s in the future +tar: ./usr/lib/locale/it_IT@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.743603258 s in the future +tar: ./usr/lib/locale/zh_CN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.741383528 s in the future +tar: ./usr/lib/locale/zh_CN: time stamp 2026-11-29 07:51:59 is 34039938.738890364 s in the future +tar: ./usr/lib/locale/rw_RW/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.738079758 s in the future +tar: ./usr/lib/locale/hif_FJ/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.737303289 s in the future +tar: ./usr/lib/locale/aa_DJ.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.73445338 s in the future +tar: ./usr/lib/locale/aa_DJ.utf8: time stamp 2026-11-29 07:51:59 is 34039938.718813427 s in the future +tar: ./usr/lib/locale/xh_ZA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.71835396 s in the future +tar: ./usr/lib/locale/yue_HK/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.716119411 s in the future +tar: ./usr/lib/locale/bi_VU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.715329198 s in the future +tar: ./usr/lib/locale/zh_CN.gb18030/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.710872778 s in the future +tar: ./usr/lib/locale/zh_CN.gb18030: time stamp 2026-11-29 07:51:59 is 34039938.694073989 s in the future +tar: ./usr/lib/locale/mnw_MM/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.69216783 s in the future +tar: ./usr/lib/locale/ast_ES.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.691747386 s in the future +tar: ./usr/lib/locale/csb_PL/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.691325141 s in the future +tar: ./usr/lib/locale/ku_TR/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.674651265 s in the future +tar: ./usr/lib/locale/ku_TR: time stamp 2026-11-29 07:51:59 is 34039938.674570491 s in the future +tar: ./usr/lib/locale/sm_WS/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.674173651 s in the future +tar: ./usr/lib/locale/hy_AM/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.673746509 s in the future +tar: ./usr/lib/locale/ru_RU.koi8r/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.671720311 s in the future +tar: ./usr/lib/locale/ru_RU.koi8r: time stamp 2026-11-29 07:51:59 is 34039938.671645347 s in the future +tar: ./usr/lib/locale/ak_GH/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.667552666 s in the future +tar: ./usr/lib/locale/sv_FI/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.666754104 s in the future +tar: ./usr/lib/locale/cs_CZ.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.666293604 s in the future +tar: ./usr/lib/locale/el_CY.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.650580802 s in the future +tar: ./usr/lib/locale/hu_HU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.648803756 s in the future +tar: ./usr/lib/locale/mdf_RU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.647477392 s in the future +tar: ./usr/lib/locale/fa_IR/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.630876794 s in the future +tar: ./usr/lib/locale/mfe_MU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.630412517 s in the future +tar: ./usr/lib/locale/sat_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.629981446 s in the future +tar: ./usr/lib/locale/fi_FI.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.629155689 s in the future +tar: ./usr/lib/locale/os_RU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.612990522 s in the future +tar: ./usr/lib/locale/mg_MG/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.597061712 s in the future +tar: ./usr/lib/locale/an_ES/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.595145716 s in the future +tar: ./usr/lib/locale/an_ES: time stamp 2026-11-29 07:51:59 is 34039938.595031011 s in the future +tar: ./usr/lib/locale/anp_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.59272944 s in the future +tar: ./usr/lib/locale/et_EE.iso885915/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.592255526 s in the future +tar: ./usr/lib/locale/te_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.590019903 s in the future +tar: ./usr/lib/locale/de_AT.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.585074028 s in the future +tar: ./usr/lib/locale/lg_UG.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.583862187 s in the future +tar: ./usr/lib/locale/he_IL.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.582820544 s in the future +tar: ./usr/lib/locale/ko_KR.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.577799337 s in the future +tar: ./usr/lib/locale/or_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.57173746 s in the future +tar: ./usr/lib/locale/ik_CA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.555096661 s in the future +tar: ./usr/lib/locale/quz_PE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.538789427 s in the future +tar: ./usr/lib/locale/fr_CH.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.538373396 s in the future +tar: ./usr/lib/locale/to_TO/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.537594949 s in the future +tar: ./usr/lib/locale/uz_UZ/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.535633476 s in the future +tar: ./usr/lib/locale/niu_NU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.535140096 s in the future +tar: ./usr/lib/locale/ia_FR/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.533914266 s in the future +tar: ./usr/lib/locale/th_TH.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.533469441 s in the future +tar: ./usr/lib/locale/hsb_DE.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.517254061 s in the future +tar: ./usr/lib/locale/aa_DJ/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.483840281 s in the future +tar: ./usr/lib/locale/aa_DJ: time stamp 2026-11-29 07:51:59 is 34039938.483718483 s in the future +tar: ./usr/lib/locale/zh_CN.gbk/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.481667745 s in the future +tar: ./usr/lib/locale/zh_CN.gbk: time stamp 2026-11-29 07:51:59 is 34039938.475935117 s in the future +tar: ./usr/lib/locale/ts_ZA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.475515137 s in the future +tar: ./usr/lib/locale/uz_UZ@cyrillic/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.473389752 s in the future +tar: ./usr/lib/locale/pa_PK/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.470387774 s in the future +tar: ./usr/lib/locale/tpi_PG/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.46994501 s in the future +tar: ./usr/lib/locale/en_AU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.4687339 s in the future +tar: ./usr/lib/locale/en_GB.iso885915/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.468286609 s in the future +tar: ./usr/lib/locale/mt_MT.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.467860464 s in the future +tar: ./usr/lib/locale/af_ZA.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.451990458 s in the future +tar: ./usr/lib/locale/ur_PK/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.451529204 s in the future +tar: ./usr/lib/locale/wa_BE@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.449642011 s in the future +tar: ./usr/lib/locale/lv_LV.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.448760074 s in the future +tar: ./usr/lib/locale/gd_GB/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.4479139 s in the future +tar: ./usr/lib/locale/el_CY/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.445629331 s in the future +tar: ./usr/lib/locale/el_CY: time stamp 2026-11-29 07:51:59 is 34039938.445522299 s in the future +tar: ./usr/lib/locale/fr_BE.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.445122588 s in the future +tar: ./usr/lib/locale/mk_MK.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.444682484 s in the future +tar: ./usr/lib/locale/ug_CN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.443810498 s in the future +tar: ./usr/lib/locale/gl_ES.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.427573805 s in the future +tar: ./usr/lib/locale/da_DK.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.424745704 s in the future +tar: ./usr/lib/locale/mhr_RU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.408586223 s in the future +tar: ./usr/lib/locale/szl_PL/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.407347579 s in the future +tar: ./usr/lib/locale/it_IT.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.391549243 s in the future +tar: ./usr/lib/locale/hy_AM.armscii8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.389699076 s in the future +tar: ./usr/lib/locale/hy_AM.armscii8: time stamp 2026-11-29 07:51:59 is 34039938.389624622 s in the future +tar: ./usr/lib/locale/it_IT/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.387998176 s in the future +tar: ./usr/lib/locale/si_LK/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.387516513 s in the future +tar: ./usr/lib/locale/hsb_DE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.368931425 s in the future +tar: ./usr/lib/locale/zh_TW/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.36678024 s in the future +tar: ./usr/lib/locale/zh_TW: time stamp 2026-11-29 07:51:59 is 34039938.363051733 s in the future +tar: ./usr/lib/locale/sq_AL/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.362603793 s in the future +tar: ./usr/lib/locale/oc_FR.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.361719966 s in the future +tar: ./usr/lib/locale/sq_AL.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.361265893 s in the future +tar: ./usr/lib/locale/en_DK.iso885915/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.345299515 s in the future +tar: ./usr/lib/locale/lt_LT/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.343487298 s in the future +tar: ./usr/lib/locale/lt_LT: time stamp 2026-11-29 07:51:59 is 34039938.343359429 s in the future +tar: ./usr/lib/locale/sr_ME/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.34249823 s in the future +tar: ./usr/lib/locale/ff_SN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.340811133 s in the future +tar: ./usr/lib/locale/zh_HK/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.338167577 s in the future +tar: ./usr/lib/locale/zh_HK: time stamp 2026-11-29 07:51:59 is 34039938.333599106 s in the future +tar: ./usr/lib/locale/pl_PL.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.333169758 s in the future +tar: ./usr/lib/locale/kv_RU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.317349572 s in the future +tar: ./usr/lib/locale/lij_IT/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.316500958 s in the future +tar: ./usr/lib/locale/kw_GB/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.31605475 s in the future +tar: ./usr/lib/locale/lo_LA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.315177054 s in the future +tar: ./usr/lib/locale/gl_ES/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.314705261 s in the future +tar: ./usr/lib/locale/ast_ES/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.314224338 s in the future +tar: ./usr/lib/locale/ku_TR.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.313736564 s in the future +tar: ./usr/lib/locale/bn_BD/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.297720049 s in the future +tar: ./usr/lib/locale/zh_TW.euctw/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.295200119 s in the future +tar: ./usr/lib/locale/zh_TW.euctw: time stamp 2026-11-29 07:51:59 is 34039938.289929059 s in the future +tar: ./usr/lib/locale/kk_KZ.rk1048/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.287914997 s in the future +tar: ./usr/lib/locale/kk_KZ.rk1048: time stamp 2026-11-29 07:51:59 is 34039938.287801302 s in the future +tar: ./usr/lib/locale/ru_RU.cp1251/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.287392738 s in the future +tar: ./usr/lib/locale/ks_IN@devanagari/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.286052775 s in the future +tar: ./usr/lib/locale/sl_SI.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.285586938 s in the future +tar: ./usr/lib/locale/sa_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.269778525 s in the future +tar: ./usr/lib/locale/fi_FI/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.269318349 s in the future +tar: ./usr/lib/locale/ru_RU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.268420812 s in the future +tar: ./usr/lib/locale/az_IR/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.265806054 s in the future +tar: ./usr/lib/locale/kn_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.247310041 s in the future +tar: ./usr/lib/locale/tt_RU@iqtelif/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.24651552 s in the future +tar: ./usr/lib/locale/nb_NO.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.246053987 s in the future +tar: ./usr/lib/locale/sv_FI@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.228383412 s in the future +tar: ./usr/lib/locale/am_ET/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.227473367 s in the future +tar: ./usr/lib/locale/fr_BE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.2270273 s in the future +tar: ./usr/lib/locale/kok_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.226192084 s in the future +tar: ./usr/lib/locale/ko_KR.euckr/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.221155007 s in the future +tar: ./usr/lib/locale/ko_KR.euckr: time stamp 2026-11-29 07:51:59 is 34039938.218853262 s in the future +tar: ./usr/lib/locale/ps_AF/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.216380882 s in the future +tar: ./usr/lib/locale/yuw_PG/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.200240099 s in the future +tar: ./usr/lib/locale/gez_ET/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.199379083 s in the future +tar: ./usr/lib/locale/ha_NG/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.198851627 s in the future +tar: ./usr/lib/locale/chr_US/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.182983573 s in the future +tar: ./usr/lib/locale/ja_JP.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.178561017 s in the future +tar: ./usr/lib/locale/et_EE.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.174485553 s in the future +tar: ./usr/lib/locale/rif_MA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.158662827 s in the future +tar: ./usr/lib/locale/ro_RO/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.15630057 s in the future +tar: ./usr/lib/locale/my_MM/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.15399404 s in the future +tar: ./usr/lib/locale/fo_FO/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.153419662 s in the future +tar: ./usr/lib/locale/ss_ZA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.148677993 s in the future +tar: ./usr/lib/locale/ja_JP.eucjp/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.143436424 s in the future +tar: ./usr/lib/locale/ja_JP.eucjp: time stamp 2026-11-29 07:51:59 is 34039938.140516921 s in the future +tar: ./usr/lib/locale/agr_PE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.140016896 s in the future +tar: ./usr/lib/locale/sc_IT/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.138617703 s in the future +tar: ./usr/lib/locale/ca_ES/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.13808822 s in the future +tar: ./usr/lib/locale/ru_UA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.135635037 s in the future +tar: ./usr/lib/locale/ru_UA: time stamp 2026-11-29 07:51:59 is 34039938.135510043 s in the future +tar: ./usr/lib/locale/eu_ES@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.134917534 s in the future +tar: ./usr/lib/locale/iu_CA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.133961308 s in the future +tar: ./usr/lib/locale/yo_NG/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.132009443 s in the future +tar: ./usr/lib/locale/de_AT@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.114454717 s in the future +tar: ./usr/lib/locale/en_CA.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.113858785 s in the future +tar: ./usr/lib/locale/mg_MG.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.096535825 s in the future +tar: ./usr/lib/locale/ht_HT/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.095976777 s in the future +tar: ./usr/lib/locale/ayc_PE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.095018802 s in the future +tar: ./usr/lib/locale/xh_ZA.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.094045664 s in the future +tar: ./usr/lib/locale/pt_BR.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.093011812 s in the future +tar: ./usr/lib/locale/ru_RU.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.092488456 s in the future +tar: ./usr/lib/locale/ln_CD/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.091460484 s in the future +tar: ./usr/lib/locale/gu_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.072468824 s in the future +tar: ./usr/lib/locale/tr_CY/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.071916466 s in the future +tar: ./usr/lib/locale/dsb_DE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.071297698 s in the future +tar: ./usr/lib/locale/lg_UG/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.053226826 s in the future +tar: ./usr/lib/locale/lg_UG: time stamp 2026-11-29 07:51:59 is 34039938.053064754 s in the future +tar: ./usr/lib/locale/en_DK/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.052104753 s in the future +tar: ./usr/lib/locale/ga_IE@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.051552471 s in the future +tar: ./usr/lib/locale/bhb_IN.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.050053434 s in the future +tar: ./usr/lib/locale/fr_CA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.049045086 s in the future +tar: ./usr/lib/locale/et_EE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.048428794 s in the future +tar: ./usr/lib/locale/mk_MK/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.046297077 s in the future +tar: ./usr/lib/locale/mk_MK: time stamp 2026-11-29 07:51:59 is 34039938.046123602 s in the future +tar: ./usr/lib/locale/hu_HU.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.042747533 s in the future +tar: ./usr/lib/locale/wa_BE.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.023936044 s in the future +tar: ./usr/lib/locale/de_AT/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.022044877 s in the future +tar: ./usr/lib/locale/ky_KG/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.021502715 s in the future +tar: ./usr/lib/locale/syr/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.005567402 s in the future +tar: ./usr/lib/locale/shn_MM/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.002237568 s in the future +tar: ./usr/lib/locale/lt_LT.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039938.001232883 s in the future +tar: ./usr/lib/locale/zu_ZA.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.984086955 s in the future +tar: ./usr/lib/locale/tl_PH/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.983527914 s in the future +tar: ./usr/lib/locale/bg_BG/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.982960879 s in the future +tar: ./usr/lib/locale/lzh_TW/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.981491981 s in the future +tar: ./usr/lib/locale/ga_IE.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.974539547 s in the future +tar: ./usr/lib/locale/ce_RU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.973988451 s in the future +tar: ./usr/lib/locale/be_BY.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.973429938 s in the future +tar: ./usr/lib/locale/fr_FR@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.957351287 s in the future +tar: ./usr/lib/locale/cs_CZ/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.956360664 s in the future +tar: ./usr/lib/locale/ro_RO.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.954017824 s in the future +tar: ./usr/lib/locale/km_KH/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.937782809 s in the future +tar: ./usr/lib/locale/lv_LV/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.936774993 s in the future +tar: ./usr/lib/locale/br_FR.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.936212483 s in the future +tar: ./usr/lib/locale/sv_FI.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.920272189 s in the future +tar: ./usr/lib/locale/be_BY/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.902851606 s in the future +tar: ./usr/lib/locale/be_BY: time stamp 2026-11-29 07:51:59 is 34039937.902737678 s in the future +tar: ./usr/lib/locale/gd_GB.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.901765461 s in the future +tar: ./usr/lib/locale/eo/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.901231199 s in the future +tar: ./usr/lib/locale/mi_NZ/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.885193205 s in the future +tar: ./usr/lib/locale/dz_BT/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.883218587 s in the future +tar: ./usr/lib/locale/tk_TM/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.880850824 s in the future +tar: ./usr/lib/locale/scn_IT/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.862780447 s in the future +tar: ./usr/lib/locale/fr_CH/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.862221671 s in the future +tar: ./usr/lib/locale/nl_BE@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.861217434 s in the future +tar: ./usr/lib/locale/mt_MT/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.858746857 s in the future +tar: ./usr/lib/locale/mt_MT: time stamp 2026-11-29 07:51:59 is 34039937.858631702 s in the future +tar: ./usr/lib/locale/ml_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.856341421 s in the future +tar: ./usr/lib/locale/as_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.817164122 s in the future +tar: ./usr/lib/locale/li_NL/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.81658627 s in the future +tar: ./usr/lib/locale/zh_HK.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.815067729 s in the future +tar: ./usr/lib/locale/om_KE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.813996414 s in the future +tar: ./usr/lib/locale/az_AZ/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.812424063 s in the future +tar: ./usr/lib/locale/tg_TJ/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.794687043 s in the future +tar: ./usr/lib/locale/tg_TJ: time stamp 2026-11-29 07:51:59 is 34039937.79457117 s in the future +tar: ./usr/lib/locale/lb_LU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.792348377 s in the future +tar: ./usr/lib/locale/cy_GB/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.789285065 s in the future +tar: ./usr/lib/locale/cy_GB: time stamp 2026-11-29 07:51:59 is 34039937.789161436 s in the future +tar: ./usr/lib/locale/pt_BR/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.788638238 s in the future +tar: ./usr/lib/locale/ber_MA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.788034731 s in the future +tar: ./usr/lib/locale/cmn_TW/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.770262577 s in the future +tar: ./usr/lib/locale/mr_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.736557304 s in the future +tar: ./usr/lib/locale/mi_NZ.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.720392872 s in the future +tar: ./usr/lib/locale/fi_FI@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.704330838 s in the future +tar: ./usr/lib/locale/kl_GL.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.703238849 s in the future +tar: ./usr/lib/locale/it_CH/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.701788924 s in the future +tar: ./usr/lib/locale/kw_GB.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.701224345 s in the future +tar: ./usr/lib/locale/uk_UA.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.698657076 s in the future +tar: ./usr/lib/locale/ur_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.681659737 s in the future +tar: ./usr/lib/locale/ckb_IQ/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.681093785 s in the future +tar: ./usr/lib/locale/da_DK/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.663682298 s in the future +tar: ./usr/lib/locale/crh_UA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.661185407 s in the future +tar: ./usr/lib/locale/id_ID/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.643986569 s in the future +tar: ./usr/lib/locale/bs_BA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.641884846 s in the future +tar: ./usr/lib/locale/bs_BA: time stamp 2026-11-29 07:51:59 is 34039937.641769307 s in the future +tar: ./usr/lib/locale/en_CA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.641276606 s in the future +tar: ./usr/lib/locale/yi_US.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.638363676 s in the future +tar: ./usr/lib/locale/ka_GE.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.622024117 s in the future +tar: ./usr/lib/locale/br_FR/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.6214574 s in the future +tar: ./usr/lib/locale/ar_AE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.619549909 s in the future +tar: ./usr/lib/locale/ar_AE: time stamp 2026-11-29 07:51:59 is 34039937.619441403 s in the future +tar: ./usr/lib/locale/C.utf8/LC_MESSAGES: time stamp 2026-11-29 07:46:34 is 34039612.617078769 s in the future +tar: ./usr/lib/locale/C.utf8: time stamp 2026-11-29 07:46:34 is 34039612.616996669 s in the future +tar: ./usr/lib/locale/sgs_LT/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.613970095 s in the future +tar: ./usr/lib/locale/so_KE.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.612935266 s in the future +tar: ./usr/lib/locale/sk_SK/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.612374488 s in the future +tar: ./usr/lib/locale/mn_MN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.60993008 s in the future +tar: ./usr/lib/locale/sk_SK.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039937.609322193 s in the future +tar: ./usr/lib/mime/packages: time stamp 2026-12-02 20:29:28 is 34344586.609003384 s in the future +tar: ./usr/lib/linux/uapi/s390/asm: time stamp 2026-11-29 07:50:02 is 34039820.606505192 s in the future +tar: ./usr/lib/linux/uapi/s390: time stamp 2026-11-29 07:50:01 is 34039819.60647805 s in the future +tar: ./usr/lib/linux/uapi/alpha/asm: time stamp 2026-11-29 07:50:02 is 34039820.604908295 s in the future +tar: ./usr/lib/linux/uapi/alpha: time stamp 2026-11-29 07:50:01 is 34039819.60488005 s in the future +tar: ./usr/lib/linux/uapi/sh/asm: time stamp 2026-11-29 07:50:02 is 34039820.603481488 s in the future +tar: ./usr/lib/linux/uapi/sh: time stamp 2026-11-29 07:50:01 is 34039819.603452098 s in the future +tar: ./usr/lib/linux/uapi/loongarch/asm: time stamp 2026-11-29 07:50:02 is 34039820.60205087 s in the future +tar: ./usr/lib/linux/uapi/loongarch: time stamp 2026-11-29 07:50:01 is 34039819.602022656 s in the future +tar: ./usr/lib/linux/uapi/powerpc/asm: time stamp 2026-11-29 07:50:02 is 34039820.600041897 s in the future +tar: ./usr/lib/linux/uapi/powerpc: time stamp 2026-11-29 07:50:01 is 34039819.600013865 s in the future +tar: ./usr/lib/linux/uapi/arm/asm: time stamp 2026-11-29 07:50:02 is 34039820.598687648 s in the future +tar: ./usr/lib/linux/uapi/arm: time stamp 2026-11-29 07:50:01 is 34039819.598659639 s in the future +tar: ./usr/lib/linux/uapi/m68k/asm: time stamp 2026-11-29 07:50:02 is 34039820.597126659 s in the future +tar: ./usr/lib/linux/uapi/m68k: time stamp 2026-11-29 07:50:01 is 34039819.597100566 s in the future +tar: ./usr/lib/linux/uapi/mips/asm: time stamp 2026-11-29 07:50:02 is 34039820.59537088 s in the future +tar: ./usr/lib/linux/uapi/mips: time stamp 2026-11-29 07:50:01 is 34039819.595341728 s in the future +tar: ./usr/lib/linux/uapi/sparc/asm: time stamp 2026-11-29 07:50:02 is 34039820.593573833 s in the future +tar: ./usr/lib/linux/uapi/sparc: time stamp 2026-11-29 07:50:01 is 34039819.593547294 s in the future +tar: ./usr/lib/linux/uapi/riscv/asm/vendor: time stamp 2026-11-29 07:50:02 is 34039820.59206433 s in the future +tar: ./usr/lib/linux/uapi/riscv/asm: time stamp 2026-11-29 07:50:02 is 34039820.591954326 s in the future +tar: ./usr/lib/linux/uapi/riscv: time stamp 2026-11-29 07:50:01 is 34039819.59192777 s in the future +tar: ./usr/lib/linux/uapi/parisc/asm: time stamp 2026-11-29 07:50:02 is 34039820.590593437 s in the future +tar: ./usr/lib/linux/uapi/parisc: time stamp 2026-11-29 07:50:01 is 34039819.590565407 s in the future +tar: ./usr/lib/linux/uapi/x86/asm: time stamp 2026-11-29 07:50:02 is 34039820.588189292 s in the future +tar: ./usr/lib/linux/uapi/x86: time stamp 2026-11-29 07:50:01 is 34039819.588163055 s in the future +tar: ./usr/lib/linux/uapi/arm64/asm: time stamp 2026-11-29 07:50:02 is 34039820.586759303 s in the future +tar: ./usr/lib/linux/uapi/arm64: time stamp 2026-11-29 07:50:01 is 34039819.586732767 s in the future +tar: ./usr/lib/linux/uapi: time stamp 2026-11-29 07:50:01 is 34039819.586714616 s in the future +tar: ./usr/lib/linux: time stamp 2026-11-29 07:50:01 is 34039819.586697803 s in the future +tar: ./usr/lib/gcc/x86_64-linux-gnu/15/include/sanitizer: time stamp 2026-12-02 23:31:20 is 34355498.543573636 s in the future +tar: ./usr/lib/gcc/x86_64-linux-gnu/15/include: time stamp 2026-12-02 23:31:20 is 34355498.542539726 s in the future +tar: ./usr/lib/gcc/x86_64-linux-gnu/15/plugin: time stamp 2026-12-02 23:31:24 is 34355502.426689168 s in the future +tar: ./usr/lib/dpkg/methods/apt: time stamp 2026-11-30 23:31:29 is 34182707.413508387 s in the future +tar: ./usr/lib/dpkg/methods: time stamp 2026-11-29 07:46:41 is 34039619.413477513 s in the future +tar: ./usr/lib/dpkg: time stamp 2026-11-29 07:46:41 is 34039619.413459867 s in the future +tar: ./var/opt: time stamp 2026-11-29 07:46:28 is 34039605.785365829 s in the future +tar: ./var/cache/apt/archives/lock: time stamp 2026-11-29 07:50:56 is 34039873.784949697 s in the future +tar: ./var/cache/apt/archives/partial: time stamp 2026-12-02 23:31:01 is 34355478.784824864 s in the future +tar: ./var/cache/apt/archives: time stamp 2026-12-02 23:31:28 is 34355505.784799048 s in the future +tar: ./var/cache/debconf/config.dat-old: time stamp 2026-11-29 07:50:53 is 34039870.784654911 s in the future +tar: ./var/cache/debconf/config.dat: time stamp 2026-11-29 07:52:40 is 34039977.784583328 s in the future +tar: ./var/cache/debconf/templates.dat-old: time stamp 2026-11-29 07:50:53 is 34039870.781330711 s in the future +tar: ./var/cache/debconf/passwords.dat: time stamp 2026-11-29 07:46:28 is 34039605.781264926 s in the future +tar: ./var/cache/debconf/templates.dat: time stamp 2026-11-29 07:52:40 is 34039977.777917317 s in the future +tar: ./var/cache/debconf/tmp.ci: time stamp 2026-11-29 07:50:57 is 34039874.77779837 s in the future +tar: ./var/cache/debconf: time stamp 2026-11-29 07:52:40 is 34039977.777766981 s in the future +tar: ./var/cache/ldconfig/aux-cache: time stamp 2026-12-02 23:31:27 is 34355504.777610926 s in the future +tar: ./var/cache/ldconfig: time stamp 2026-12-02 23:31:27 is 34355504.777578608 s in the future +tar: ./var/cache: time stamp 2026-11-29 07:51:37 is 34039914.777474737 s in the future +tar: ./var/log/bootstrap.log: time stamp 2026-11-29 07:50:27 is 34039844.777261829 s in the future +tar: ./var/log/apt/eipp.log.xz: time stamp 2026-12-02 23:31:01 is 34355478.777103908 s in the future +tar: ./var/log/apt/history.log: time stamp 2026-12-02 23:31:27 is 34355504.777027235 s in the future +tar: ./var/log/apt/term.log: time stamp 2026-12-02 23:31:27 is 34355504.77688125 s in the future +tar: ./var/log/apt: time stamp 2026-12-02 23:31:01 is 34355478.776821873 s in the future +tar: ./var/log/alternatives.log: time stamp 2026-12-02 20:29:29 is 34344586.7767495 s in the future +tar: ./var/log/lastlog: time stamp 2026-11-29 07:46:28 is 34039605.776671541 s in the future +tar: ./var/log/wtmp: time stamp 2026-11-29 07:46:28 is 34039605.776616799 s in the future +tar: ./var/log/dpkg.log: time stamp 2026-12-02 23:31:27 is 34355504.776244263 s in the future +tar: ./var/log/btmp: time stamp 2026-11-29 07:46:28 is 34039605.776157598 s in the future +tar: ./var/log: time stamp 2026-11-29 07:50:28 is 34039845.776106338 s in the future +tar: ./var/mail: time stamp 2026-11-29 07:46:28 is 34039605.775905202 s in the future +tar: ./var/lib/apt/lists/lock: time stamp 2026-11-29 07:50:54 is 34039871.413066429 s in the future +tar: ./var/lib/apt/lists/auxfiles: time stamp 2026-11-29 07:50:54 is 34039871.412811911 s in the future +tar: ./var/lib/apt/extended_states: time stamp 2026-12-02 23:31:01 is 34355478.41265374 s in the future +tar: ./var/lib/apt/mirrors: time stamp 2026-11-29 07:46:41 is 34039618.412473564 s in the future +tar: ./var/lib/apt: time stamp 2026-12-02 23:31:01 is 34355478.412357346 s in the future +tar: ./var/lib/systemd/deb-systemd-helper-enabled/apt-daily.timer.dsh-also: time stamp 2026-11-30 23:31:30 is 34182707.412166314 s in the future +tar: ./var/lib/systemd/deb-systemd-helper-enabled/timers.target.wants/apt-daily-upgrade.timer: time stamp 2026-11-29 07:50:27 is 34039844.412022262 s in the future +tar: ./var/lib/systemd/deb-systemd-helper-enabled/timers.target.wants/fstrim.timer: time stamp 2026-11-29 07:46:40 is 34039617.411965989 s in the future +tar: ./var/lib/systemd/deb-systemd-helper-enabled/timers.target.wants/apt-daily.timer: time stamp 2026-11-29 07:50:27 is 34039844.411910804 s in the future +tar: ./var/lib/systemd/deb-systemd-helper-enabled/timers.target.wants/dpkg-db-backup.timer: time stamp 2026-11-29 07:46:29 is 34039606.411861784 s in the future +tar: ./var/lib/systemd/deb-systemd-helper-enabled/timers.target.wants: time stamp 2026-11-29 07:50:27 is 34039844.411832139 s in the future +tar: ./var/lib/systemd/deb-systemd-helper-enabled/fstrim.timer.dsh-also: time stamp 2026-12-02 20:29:29 is 34344586.411756255 s in the future +tar: ./var/lib/systemd/deb-systemd-helper-enabled/dpkg-db-backup.timer.dsh-also: time stamp 2026-11-29 07:46:40 is 34039617.411694758 s in the future +tar: ./var/lib/systemd/deb-systemd-helper-enabled/apt-daily-upgrade.timer.dsh-also: time stamp 2026-11-30 23:31:30 is 34182707.411634222 s in the future +tar: ./var/lib/systemd/deb-systemd-helper-enabled: time stamp 2026-12-02 20:29:29 is 34344586.411606173 s in the future +tar: ./var/lib/systemd: time stamp 2026-11-29 07:46:29 is 34039606.411587076 s in the future +tar: ./var/lib/pam/account: time stamp 2026-11-29 07:46:40 is 34039617.411456065 s in the future +tar: ./var/lib/pam/session: time stamp 2026-11-29 07:46:40 is 34039617.41138728 s in the future +tar: ./var/lib/pam/seen: time stamp 2026-11-29 07:46:40 is 34039617.411326025 s in the future +tar: ./var/lib/pam/session-noninteractive: time stamp 2026-11-29 07:46:40 is 34039617.411261514 s in the future +tar: ./var/lib/pam/auth: time stamp 2026-11-29 07:46:40 is 34039617.411181857 s in the future +tar: ./var/lib/pam/password: time stamp 2026-11-29 07:46:40 is 34039617.411112607 s in the future +tar: ./var/lib/pam: time stamp 2026-11-29 07:46:40 is 34039617.411080686 s in the future +tar: ./var/lib/shells.state: time stamp 2026-11-29 07:46:39 is 34039616.411021191 s in the future +tar: ./var/lib/dpkg/alternatives/awk: time stamp 2026-11-29 07:46:39 is 34039616.410778829 s in the future +tar: ./var/lib/dpkg/alternatives/c99: time stamp 2026-11-29 07:50:27 is 34039844.410710766 s in the future +tar: ./var/lib/dpkg/alternatives/pager: time stamp 2026-12-02 20:29:29 is 34344586.410650124 s in the future +tar: ./var/lib/dpkg/alternatives/lzma: time stamp 2026-11-29 07:50:08 is 34039825.410583053 s in the future +tar: ./var/lib/dpkg/alternatives/rmt: time stamp 2026-11-29 07:46:40 is 34039617.410518432 s in the future +tar: ./var/lib/dpkg/alternatives/which: time stamp 2026-11-29 07:46:39 is 34039616.410433361 s in the future +tar: ./var/lib/dpkg/alternatives/builtins.7.gz: time stamp 2026-11-29 07:46:39 is 34039616.410365622 s in the future +tar: ./var/lib/dpkg/alternatives/cc: time stamp 2026-11-29 07:50:27 is 34039844.41029868 s in the future +tar: ./var/lib/dpkg/alternatives/c++: time stamp 2026-11-29 07:50:27 is 34039844.410235375 s in the future +tar: ./var/lib/dpkg/alternatives/c89: time stamp 2026-11-29 07:50:27 is 34039844.410171842 s in the future +tar: ./var/lib/dpkg/alternatives/cpp: time stamp 2026-11-29 07:50:27 is 34039844.410104524 s in the future +tar: ./var/lib/dpkg/alternatives/fakeroot: time stamp 2026-11-29 07:52:00 is 34039937.410034289 s in the future +tar: ./var/lib/dpkg/alternatives/aptitude: time stamp 2026-11-29 07:52:01 is 34039938.409963615 s in the future +tar: ./var/lib/dpkg/alternatives: time stamp 2026-12-02 20:29:29 is 34344586.409923337 s in the future +tar: ./var/lib/dpkg/info/libboost-iostreams1.83.0:amd64.list: time stamp 2026-11-29 07:51:39 is 34039916.409796577 s in the future +tar: ./var/lib/dpkg/info/build-essential.list: time stamp 2026-11-29 07:46:42 is 34039619.409392526 s in the future +tar: ./var/lib/dpkg/info/locales-all.list: time stamp 2026-11-29 07:52:00 is 34039937.408257297 s in the future +tar: ./var/lib/dpkg/info/base-files.list: time stamp 2026-11-29 07:46:31 is 34039608.408181096 s in the future +tar: ./var/lib/dpkg/info/debconf.list: time stamp 2026-11-29 07:46:32 is 34039609.406564297 s in the future +tar: ./var/lib/dpkg/info/gcc-x86-64-linux-gnu.list: time stamp 2026-11-29 07:47:17 is 34039654.406299853 s in the future +tar: ./var/lib/dpkg/info/libubsan1:amd64.list: time stamp 2026-12-02 23:31:16 is 34355493.406127697 s in the future +tar: ./var/lib/dpkg/info/libjansson4:amd64.list: time stamp 2026-11-29 07:48:24 is 34039721.405848287 s in the future +tar: ./var/lib/dpkg/info/openssl-provider-legacy.list: time stamp 2026-11-29 07:46:37 is 34039614.405370059 s in the future +tar: ./var/lib/dpkg/info/format: time stamp 2026-11-29 07:46:28 is 34039605.405160084 s in the future +tar: ./var/lib/dpkg/info/libpam0g:amd64.list: time stamp 2026-11-29 07:46:36 is 34039613.404767265 s in the future +tar: ./var/lib/dpkg/info/patch.list: time stamp 2026-11-29 07:50:02 is 34039819.404642849 s in the future +tar: ./var/lib/dpkg/info/g++.list: time stamp 2026-11-29 07:46:48 is 34039625.404119456 s in the future +tar: ./var/lib/dpkg/info/libsemanage-common.list: time stamp 2026-11-29 07:51:23 is 34039900.403993168 s in the future +tar: ./var/lib/dpkg/info/perl-modules-5.40.list: time stamp 2026-11-29 07:50:05 is 34039822.403638537 s in the future +tar: ./var/lib/dpkg/info/libctf0:amd64.list: time stamp 2026-11-30 05:29:44 is 34117801.40350915 s in the future +tar: ./var/lib/dpkg/info/libc-dev-bin.list: time stamp 2026-11-29 07:47:28 is 34039665.403376183 s in the future +tar: ./var/lib/dpkg/info/libaudit-common.list: time stamp 2026-11-29 07:46:34 is 34039611.403149642 s in the future +tar: ./var/lib/dpkg/info/fakeroot.list: time stamp 2026-11-29 07:51:45 is 34039922.403078017 s in the future +tar: ./var/lib/dpkg/info/rpcsvc-proto.list: time stamp 2026-11-29 07:50:05 is 34039822.402897104 s in the future +tar: ./var/lib/dpkg/info/libcap2:amd64.list: time stamp 2026-11-29 07:46:35 is 34039612.402771325 s in the future +tar: ./var/lib/dpkg/info/gcc.list: time stamp 2026-11-29 07:47:05 is 34039642.401784549 s in the future +tar: ./var/lib/dpkg/info/gzip.list: time stamp 2026-11-29 07:46:33 is 34039610.401338658 s in the future +tar: ./var/lib/dpkg/info/libssl3t64:amd64.list: time stamp 2026-11-29 07:46:36 is 34039613.40106944 s in the future +tar: ./var/lib/dpkg/info/libsemanage2:amd64.list: time stamp 2026-11-29 07:51:30 is 34039907.400860179 s in the future +tar: ./var/lib/dpkg/info/base-passwd.list: time stamp 2026-11-29 07:46:31 is 34039608.400666811 s in the future +tar: ./var/lib/dpkg/info/sysvinit-utils.list: time stamp 2026-11-29 07:46:38 is 34039615.40058006 s in the future +tar: ./var/lib/dpkg/info/libgdbm-compat4t64:amd64.list: time stamp 2026-11-29 07:47:30 is 34039667.400434204 s in the future +tar: ./var/lib/dpkg/info/libc-l10n.list: time stamp 2026-11-29 07:51:38 is 34039915.400358013 s in the future +tar: ./var/lib/dpkg/info/libdb5.3t64:amd64.list: time stamp 2026-11-29 07:46:35 is 34039612.400276736 s in the future +tar: ./var/lib/dpkg/info/tar.list: time stamp 2026-11-29 07:46:38 is 34039615.4000954 s in the future +tar: ./var/lib/dpkg/info/debianutils.list: time stamp 2026-11-29 07:46:32 is 34039609.400013978 s in the future +tar: ./var/lib/dpkg/info/dash.list: time stamp 2026-11-29 07:46:32 is 34039609.399873709 s in the future +tar: ./var/lib/dpkg/info/diffutils.list: time stamp 2026-11-29 07:46:32 is 34039609.399357387 s in the future +tar: ./var/lib/dpkg/info/passwd.list: time stamp 2026-11-29 07:51:34 is 34039911.39868024 s in the future +tar: ./var/lib/dpkg/info/libbsd0:amd64.list: time stamp 2026-11-29 07:51:11 is 34039888.398460382 s in the future +tar: ./var/lib/dpkg/info/liblz4-1:amd64.list: time stamp 2026-11-29 07:48:47 is 34039744.398385981 s in the future +tar: ./var/lib/dpkg/info/liblzma5:amd64.list: time stamp 2026-11-29 07:46:35 is 34039612.398212587 s in the future +tar: ./var/lib/dpkg/info/g++-x86-64-linux-gnu.list: time stamp 2026-11-29 07:47:05 is 34039642.397570211 s in the future +tar: ./var/lib/dpkg/info/libdpkg-perl.list: time stamp 2026-11-29 07:47:29 is 34039666.397194242 s in the future +tar: ./var/lib/dpkg/info/libbz2-1.0:amd64.list: time stamp 2026-11-29 07:46:34 is 34039611.397115574 s in the future +tar: ./var/lib/dpkg/info/libuuid1:amd64.list: time stamp 2026-12-02 20:29:28 is 34344585.396982541 s in the future +tar: ./var/lib/dpkg/info/cpp-15-x86-64-linux-gnu.list: time stamp 2026-12-02 23:31:25 is 34355502.396675662 s in the future +tar: ./var/lib/dpkg/info/linux-libc-dev.list: time stamp 2026-11-29 07:50:02 is 34039819.395437863 s in the future +tar: ./var/lib/dpkg/info/libctf-nobfd0:amd64.list: time stamp 2026-11-30 05:29:44 is 34117801.394646923 s in the future +tar: ./var/lib/dpkg/info/findutils.list: time stamp 2026-11-29 07:46:32 is 34039609.394369082 s in the future +tar: ./var/lib/dpkg/info/libsframe2:amd64.list: time stamp 2026-11-30 05:29:45 is 34117802.394162904 s in the future +tar: ./var/lib/dpkg/info/coreutils.list: time stamp 2026-11-29 07:46:31 is 34039608.393921487 s in the future +tar: ./var/lib/dpkg/info/libsmartcols1:amd64.list: time stamp 2026-12-02 20:29:28 is 34344585.39385071 s in the future +tar: ./var/lib/dpkg/info/liblastlog2-2:amd64.list: time stamp 2026-12-02 20:29:29 is 34344586.393761125 s in the future +tar: ./var/lib/dpkg/info/libcap-ng0:amd64.list: time stamp 2026-11-29 07:46:35 is 34039612.393329443 s in the future +tar: ./var/lib/dpkg/info/liblsan0:amd64.list: time stamp 2026-12-02 23:31:15 is 34355492.393159693 s in the future +tar: ./var/lib/dpkg/info/libisl23:amd64.list: time stamp 2026-11-29 07:48:00 is 34039697.392917502 s in the future +tar: ./var/lib/dpkg/info/libperl5.40:amd64.list: time stamp 2026-11-29 07:49:46 is 34039803.392713461 s in the future +tar: ./var/lib/dpkg/info/binutils.list: time stamp 2026-11-30 05:29:45 is 34117802.392638698 s in the future +tar: ./var/lib/dpkg/info/disorderfs.list: time stamp 2026-11-29 07:51:45 is 34039922.392498995 s in the future +tar: ./var/lib/dpkg/info/gcc-15-x86-64-linux-gnu.list: time stamp 2026-12-02 23:31:23 is 34355500.392314062 s in the future +tar: ./var/lib/dpkg/info/libmd0:amd64.list: time stamp 2026-11-29 07:46:35 is 34039612.391908646 s in the future +tar: ./var/lib/dpkg/info/libsepol2:amd64.list: time stamp 2026-11-29 07:51:29 is 34039906.3918357 s in the future +tar: ./var/lib/dpkg/info/login.defs.list: time stamp 2026-11-29 07:51:01 is 34039878.391693324 s in the future +tar: ./var/lib/dpkg/info/mount.list: time stamp 2026-12-02 20:29:29 is 34344586.391047939 s in the future +tar: ./var/lib/dpkg/info/zlib1g:amd64.list: time stamp 2026-11-29 07:46:38 is 34039615.390891762 s in the future +tar: ./var/lib/dpkg/info/libudev1:amd64.list: time stamp 2026-12-02 05:32:25 is 34290762.390161724 s in the future +tar: ./var/lib/dpkg/info/libsystemd0:amd64.list: time stamp 2026-12-02 05:32:04 is 34290741.389421023 s in the future +tar: ./var/lib/dpkg/info/libfuse3-4:amd64.list: time stamp 2026-11-29 07:51:42 is 34039919.389194605 s in the future +tar: ./var/lib/dpkg/info/sed.list: time stamp 2026-11-29 07:46:37 is 34039614.389046766 s in the future +tar: ./var/lib/dpkg/info/cpp-15.list: time stamp 2026-12-02 23:31:26 is 34355503.388857729 s in the future +tar: ./var/lib/dpkg/info/libattr1:amd64.list: time stamp 2026-11-29 07:46:34 is 34039611.38846541 s in the future +tar: ./var/lib/dpkg/info/libtinfo6:amd64.list: time stamp 2026-11-29 07:46:36 is 34039613.386744389 s in the future +tar: ./var/lib/dpkg/info/debian-archive-keyring.list: time stamp 2026-11-29 07:46:47 is 34039624.386460333 s in the future +tar: ./var/lib/dpkg/info/libasan8:amd64.list: time stamp 2026-12-02 23:31:15 is 34355492.386379102 s in the future +tar: ./var/lib/dpkg/info/libtsan2:amd64.list: time stamp 2026-12-02 23:31:16 is 34355493.386307457 s in the future +tar: ./var/lib/dpkg/info/libsqlite3-0:amd64.list: time stamp 2026-11-29 07:46:36 is 34039613.386171911 s in the future +tar: ./var/lib/dpkg/info/libmount1:amd64.list: time stamp 2026-12-02 20:29:28 is 34344585.385860938 s in the future +tar: ./var/lib/dpkg/info/libfuse2t64:amd64.list: time stamp 2026-11-29 07:51:43 is 34039920.385522612 s in the future +tar: ./var/lib/dpkg/info/perl-base.list: time stamp 2026-11-29 07:46:37 is 34039614.385383645 s in the future +tar: ./var/lib/dpkg/info/libpam-modules-bin.list: time stamp 2026-11-29 07:46:36 is 34039613.385119211 s in the future +tar: ./var/lib/dpkg/info/libatomic1:amd64.list: time stamp 2026-12-02 23:31:15 is 34355492.385027941 s in the future +tar: ./var/lib/dpkg/info/fuse3.list: time stamp 2026-11-29 07:51:42 is 34039919.384845748 s in the future +tar: ./var/lib/dpkg/info/libstdc++6:amd64.list: time stamp 2026-12-02 23:31:17 is 34355494.384662181 s in the future +tar: ./var/lib/dpkg/info/libpam-modules:amd64.list: time stamp 2026-11-29 07:46:36 is 34039613.384487194 s in the future +tar: ./var/lib/dpkg/info/libmpc3:amd64.list: time stamp 2026-11-29 07:48:57 is 34039754.384264339 s in the future +tar: ./var/lib/dpkg/info/libgdbm6t64:amd64.list: time stamp 2026-11-29 07:47:30 is 34039667.384087698 s in the future +tar: ./var/lib/dpkg/info/grep.list: time stamp 2026-11-29 07:46:32 is 34039609.383781241 s in the future +tar: ./var/lib/dpkg/info/apt.list: time stamp 2026-11-30 23:31:29 is 34182706.383524064 s in the future +tar: ./var/lib/dpkg/info/bzip2.list: time stamp 2026-11-29 07:46:42 is 34039619.382635538 s in the future +tar: ./var/lib/dpkg/info/libgprofng0:amd64.list: time stamp 2026-11-30 05:29:44 is 34117801.382417735 s in the future +tar: ./var/lib/dpkg/info/libmpfr6:amd64.list: time stamp 2026-11-29 07:49:08 is 34039765.381373495 s in the future +tar: ./var/lib/dpkg/info/libfakeroot:amd64.list: time stamp 2026-11-29 07:51:45 is 34039922.381293101 s in the future +tar: ./var/lib/dpkg/info/libapt-pkg7.0:amd64.list: time stamp 2026-11-30 23:31:29 is 34182706.381050935 s in the future +tar: ./var/lib/dpkg/info/bash.list: time stamp 2026-11-29 07:46:31 is 34039608.380958245 s in the future +tar: ./var/lib/dpkg/info/libquadmath0:amd64.list: time stamp 2026-12-02 23:31:16 is 34355493.38082603 s in the future +tar: ./var/lib/dpkg/info/binutils-common:amd64.list: time stamp 2026-11-30 05:29:45 is 34117802.380687197 s in the future +tar: ./var/lib/dpkg/info/libgomp1:amd64.list: time stamp 2026-12-02 23:31:15 is 34355492.38054447 s in the future +tar: ./var/lib/dpkg/info/libselinux1:amd64.list: time stamp 2026-11-29 07:46:36 is 34039613.380465434 s in the future +tar: ./var/lib/dpkg/info/libxapian30:amd64.list: time stamp 2026-11-29 07:51:41 is 34039918.380099517 s in the future +tar: ./var/lib/dpkg/info/gcc-15.list: time stamp 2026-12-02 23:31:17 is 34355494.379965588 s in the future +tar: ./var/lib/dpkg/info/libgcc-15-dev:amd64.list: time stamp 2026-12-02 23:31:20 is 34355497.379788659 s in the future +tar: ./var/lib/dpkg/info/make.list: time stamp 2026-11-29 07:50:02 is 34039819.379103666 s in the future +tar: ./var/lib/dpkg/info/libhogweed6t64:amd64.list: time stamp 2026-11-29 07:47:43 is 34039680.378975938 s in the future +tar: ./var/lib/dpkg/info/libpcre2-8-0:amd64.list: time stamp 2026-11-29 07:46:36 is 34039613.378763219 s in the future +tar: ./var/lib/dpkg/info/libcwidget4:amd64.list: time stamp 2026-11-29 07:51:40 is 34039917.378610559 s in the future +tar: ./var/lib/dpkg/info/libncursesw6:amd64.list: time stamp 2026-11-29 07:51:40 is 34039917.378535902 s in the future +tar: ./var/lib/dpkg/info/init-system-helpers.list: time stamp 2026-11-29 07:46:33 is 34039610.37800037 s in the future +tar: ./var/lib/dpkg/info/fuse.list: time stamp 2026-11-29 07:51:43 is 34039920.37755173 s in the future +tar: ./var/lib/dpkg/info/sqv.list: time stamp 2026-11-29 07:50:05 is 34039822.377203649 s in the future +tar: ./var/lib/dpkg/info/libpam-runtime.list: time stamp 2026-11-29 07:46:36 is 34039613.376865235 s in the future +tar: ./var/lib/dpkg/info/libgmp10:amd64.list: time stamp 2026-11-29 07:46:35 is 34039612.376488837 s in the future +tar: ./var/lib/dpkg/info/aptitude-common.list: time stamp 2026-11-29 07:51:39 is 34039916.376283517 s in the future +tar: ./var/lib/dpkg/info/dpkg.list: time stamp 2026-11-29 07:46:32 is 34039609.376130987 s in the future +tar: ./var/lib/dpkg/info/libxxhash0:amd64.list: time stamp 2026-11-29 07:50:00 is 34039817.375941349 s in the future +tar: ./var/lib/dpkg/info/libitm1:amd64.list: time stamp 2026-12-02 23:31:15 is 34355492.375584314 s in the future +tar: ./var/lib/dpkg/info/libseccomp2:amd64.list: time stamp 2026-11-29 07:49:58 is 34039815.375456196 s in the future +tar: ./var/lib/dpkg/info/libc6-dev:amd64.list: time stamp 2026-11-29 07:47:28 is 34039665.375192159 s in the future +tar: ./var/lib/dpkg/info/libhwasan0:amd64.list: time stamp 2026-12-02 23:31:16 is 34355493.374803861 s in the future +tar: ./var/lib/dpkg/info/cpp-x86-64-linux-gnu.list: time stamp 2026-11-29 07:46:47 is 34039624.374521484 s in the future +tar: ./var/lib/dpkg/info/g++-15.list: time stamp 2026-12-02 23:31:17 is 34355494.374285892 s in the future +tar: ./var/lib/dpkg/info/libblkid1:amd64.list: time stamp 2026-12-02 20:29:28 is 34344585.374214157 s in the future +tar: ./var/lib/dpkg/info/libcc1-0:amd64.list: time stamp 2026-12-02 23:31:11 is 34355488.37394279 s in the future +tar: ./var/lib/dpkg/info/libaudit1:amd64.list: time stamp 2026-11-29 07:46:34 is 34039611.373660662 s in the future +tar: ./var/lib/dpkg/info/libbinutils:amd64.list: time stamp 2026-11-30 05:29:45 is 34117802.372972119 s in the future +tar: ./var/lib/dpkg/info/cpp.list: time stamp 2026-11-29 07:46:42 is 34039619.372677304 s in the future +tar: ./var/lib/dpkg/info/libnettle8t64:amd64.list: time stamp 2026-11-29 07:49:18 is 34039775.372478574 s in the future +tar: ./var/lib/dpkg/info/dpkg-dev.list: time stamp 2026-11-29 07:46:47 is 34039624.370832982 s in the future +tar: ./var/lib/dpkg/info/libc6:amd64.list: time stamp 2026-11-29 07:46:35 is 34039612.370629469 s in the future +tar: ./var/lib/dpkg/info/libc-bin.list: time stamp 2026-11-29 07:46:34 is 34039611.370552159 s in the future +tar: ./var/lib/dpkg/info/libdebconfclient0:amd64.list: time stamp 2026-11-29 07:46:35 is 34039612.370322086 s in the future +tar: ./var/lib/dpkg/info/ncurses-bin.list: time stamp 2026-11-29 07:46:37 is 34039614.37012441 s in the future +tar: ./var/lib/dpkg/info/libsigc++-2.0-0v5:amd64.list: time stamp 2026-11-29 07:51:40 is 34039917.370048345 s in the future +tar: ./var/lib/dpkg/info/util-linux.list: time stamp 2026-12-02 20:29:28 is 34344585.369839296 s in the future +tar: ./var/lib/dpkg/info/mawk.list: time stamp 2026-11-29 07:46:37 is 34039614.369582545 s in the future +tar: ./var/lib/dpkg/info/hostname.list: time stamp 2026-11-29 07:46:33 is 34039610.367760967 s in the future +tar: ./var/lib/dpkg/info/binutils-x86-64-linux-gnu.list: time stamp 2026-11-30 05:29:44 is 34117801.367663815 s in the future +tar: ./var/lib/dpkg/info/libcrypt1:amd64.list: time stamp 2026-11-29 07:46:35 is 34039612.367588842 s in the future +tar: ./var/lib/dpkg/info/perl.list: time stamp 2026-11-29 07:50:02 is 34039819.367457316 s in the future +tar: ./var/lib/dpkg/info/gcc-15-base:amd64.list: time stamp 2026-12-02 23:31:16 is 34355493.367325549 s in the future +tar: ./var/lib/dpkg/info/libstdc++-15-dev:amd64.list: time stamp 2026-12-02 23:31:19 is 34355496.367131458 s in the future +tar: ./var/lib/dpkg/info/adduser.list: time stamp 2026-11-29 07:51:37 is 34039914.367052313 s in the future +tar: ./var/lib/dpkg/info/libcrypt-dev:amd64.list: time stamp 2026-11-29 07:47:29 is 34039666.36686341 s in the future +tar: ./var/lib/dpkg/info/libgcc-s1:amd64.list: time stamp 2026-12-02 23:31:26 is 34355503.365441453 s in the future +tar: ./var/lib/dpkg/info/ncurses-base.list: time stamp 2026-11-29 07:46:37 is 34039614.36520425 s in the future +tar: ./var/lib/dpkg/info/libzstd1:amd64.list: time stamp 2026-11-29 07:46:36 is 34039613.364881454 s in the future +tar: ./var/lib/dpkg/info/xz-utils.list: time stamp 2026-11-29 07:50:06 is 34039823.364799129 s in the future +tar: ./var/lib/dpkg/info/g++-15-x86-64-linux-gnu.list: time stamp 2026-12-02 23:31:18 is 34355495.364729693 s in the future +tar: ./var/lib/dpkg/info/libacl1:amd64.list: time stamp 2026-11-29 07:46:34 is 34039611.364522013 s in the future +tar: ./var/lib/dpkg/info/aptitude.list: time stamp 2026-11-29 07:51:41 is 34039918.364377387 s in the future +tar: ./var/lib/dpkg/info: time stamp 2026-12-02 23:31:26 is 34355503.364237416 s in the future +tar: ./var/lib/dpkg/lock-frontend: time stamp 2026-11-29 07:50:06 is 34039823.364183376 s in the future +tar: ./var/lib/dpkg/diversions-old: time stamp 2026-11-29 07:52:01 is 34039938.3641212 s in the future +tar: ./var/lib/dpkg/diversions: time stamp 2026-11-29 07:52:01 is 34039938.364062635 s in the future +tar: ./var/lib/dpkg/triggers/File: time stamp 2026-11-29 07:46:36 is 34039613.363921 s in the future +tar: ./var/lib/dpkg/triggers/Lock: time stamp 2026-12-02 23:31:26 is 34355503.363860978 s in the future +tar: ./var/lib/dpkg/triggers/ldconfig: time stamp 2026-11-29 07:46:34 is 34039611.363767482 s in the future +tar: ./var/lib/dpkg/triggers/Unincorp: time stamp 2026-11-29 07:46:39 is 34039616.363716946 s in the future +tar: ./var/lib/dpkg/triggers: time stamp 2026-11-29 07:46:39 is 34039616.363686901 s in the future +tar: ./var/lib/dpkg/lock: time stamp 2026-12-02 23:31:26 is 34355503.363644138 s in the future +tar: ./var/lib/dpkg/updates: time stamp 2026-12-02 23:31:27 is 34355504.363541103 s in the future +tar: ./var/lib/dpkg/status-old: time stamp 2026-12-02 23:31:26 is 34355503.362887177 s in the future +tar: ./var/lib/dpkg/cmethopt: time stamp 2026-11-29 07:46:38 is 34039615.362816332 s in the future +tar: ./var/lib/dpkg/arch-native: time stamp 2026-11-29 07:46:40 is 34039617.362735439 s in the future +tar: ./var/lib/dpkg/available: time stamp 2026-11-29 07:46:41 is 34039618.362184906 s in the future +tar: ./var/lib/dpkg/status: time stamp 2026-12-02 23:31:27 is 34355504.361401965 s in the future +tar: ./var/lib/dpkg: time stamp 2026-12-02 23:31:27 is 34355504.361365083 s in the future +tar: ./var/lib: time stamp 2026-11-29 07:51:41 is 34039918.36125233 s in the future +tar: ./dev/fd: time stamp 2026-11-29 07:46:28 is 34039605.361030242 s in the future +tar: ./dev/stdin: time stamp 2026-11-29 07:46:28 is 34039605.36096256 s in the future +tar: ./dev/stderr: time stamp 2026-11-29 07:46:28 is 34039605.360905937 s in the future +tar: ./dev/stdout: time stamp 2026-11-29 07:46:28 is 34039605.360848719 s in the future +tar: ./etc/mtab: time stamp 2026-11-29 07:50:53 is 34039870.358941387 s in the future +tar: ./etc/systemd/system/timers.target.wants/apt-daily-upgrade.timer: time stamp 2026-11-29 07:50:27 is 34039844.358869677 s in the future +tar: ./etc/systemd/system/timers.target.wants/fstrim.timer: time stamp 2026-11-29 07:46:40 is 34039617.358808693 s in the future +tar: ./etc/systemd/system/timers.target.wants/apt-daily.timer: time stamp 2026-11-29 07:50:27 is 34039844.358749304 s in the future +tar: ./etc/systemd/system/timers.target.wants/dpkg-db-backup.timer: time stamp 2026-11-29 07:46:29 is 34039606.35868937 s in the future +tar: ./etc/localtime: time stamp 2026-11-29 07:46:30 is 34039607.35853109 s in the future +tar: ./var/spool/mail: time stamp 2026-11-29 07:46:28 is 34039605.241164669 s in the future +tar: ./var/lock: time stamp 2026-11-29 07:46:28 is 34039605.24108458 s in the future +tar: ./var/run: time stamp 2026-11-29 07:46:28 is 34039605.241031898 s in the future +tar: ./var/spool: time stamp 2026-11-29 07:46:28 is 34039605.241012776 s in the future +tar: ./var: time stamp 2026-11-29 07:46:28 is 34039605.240995208 s in the future +tar: ./usr/bin: time stamp 2026-12-02 23:31:26 is 34355503.240977709 s in the future +tar: ./usr/lib/bfd-plugins: time stamp 2026-11-29 07:47:05 is 34039642.240959197 s in the future +tar: ./usr/lib/gcc/x86_64-linux-gnu/15: time stamp 2026-12-02 23:31:24 is 34355501.240938301 s in the future +tar: ./usr/lib/gcc/x86_64-linux-gnu: time stamp 2026-11-29 07:47:06 is 34039643.240917619 s in the future +tar: ./usr/lib/gcc: time stamp 2026-11-29 07:47:06 is 34039643.24089765 s in the future +tar: ./usr/lib/compat-ld: time stamp 2026-11-30 05:29:45 is 34117802.240876536 s in the future +tar: ./usr/lib/locale/sk_SK.utf8: time stamp 2026-11-29 07:52:00 is 34039937.240856326 s in the future +tar: ./usr/lib/locale/mn_MN: time stamp 2026-11-29 07:51:59 is 34039936.240836449 s in the future +tar: ./usr/lib/locale/sk_SK: time stamp 2026-11-29 07:52:00 is 34039937.240815866 s in the future +tar: ./usr/lib/locale/so_KE.utf8: time stamp 2026-11-29 07:52:00 is 34039937.24079625 s in the future +tar: ./usr/lib/locale/gez_ER/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.240776726 s in the future +tar: ./usr/lib/locale/gez_ER: time stamp 2026-11-29 07:51:59 is 34039936.240756472 s in the future +tar: ./usr/lib/locale/sgs_LT: time stamp 2026-11-29 07:52:00 is 34039937.240736324 s in the future +tar: ./usr/lib/locale/es_PR/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.240714321 s in the future +tar: ./usr/lib/locale/es_PR: time stamp 2026-11-29 07:51:59 is 34039936.240693781 s in the future +tar: ./usr/lib/locale/fr_LU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.240675016 s in the future +tar: ./usr/lib/locale/fr_LU: time stamp 2026-11-29 07:51:59 is 34039936.240654332 s in the future +tar: ./usr/lib/locale/sd_IN@devanagari/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.240633795 s in the future +tar: ./usr/lib/locale/sd_IN@devanagari: time stamp 2026-11-29 07:52:00 is 34039937.240611865 s in the future +tar: ./usr/lib/locale/es_MX.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.240592205 s in the future +tar: ./usr/lib/locale/es_MX.utf8: time stamp 2026-11-29 07:51:59 is 34039936.240571398 s in the future +tar: ./usr/lib/locale/st_ZA.utf8/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.240551938 s in the future +tar: ./usr/lib/locale/st_ZA.utf8: time stamp 2026-11-29 07:52:00 is 34039937.240531316 s in the future +tar: ./usr/lib/locale/br_FR: time stamp 2026-11-29 07:51:59 is 34039936.240510533 s in the future +tar: ./usr/lib/locale/ka_GE.utf8: time stamp 2026-11-29 07:51:59 is 34039936.240490301 s in the future +tar: ./usr/lib/locale/yi_US.utf8: time stamp 2026-11-29 07:52:00 is 34039937.240468977 s in the future +tar: ./usr/lib/locale/st_ZA/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.240449099 s in the future +tar: ./usr/lib/locale/st_ZA: time stamp 2026-11-29 07:52:00 is 34039937.240427526 s in the future +tar: ./usr/lib/locale/en_CA: time stamp 2026-11-29 07:51:59 is 34039936.240406427 s in the future +tar: ./usr/lib/locale/id_ID: time stamp 2026-11-29 07:51:59 is 34039936.240386705 s in the future +tar: ./usr/lib/locale/sw_TZ/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.240365611 s in the future +tar: ./usr/lib/locale/sw_TZ: time stamp 2026-11-29 07:52:00 is 34039937.240346887 s in the future +tar: ./usr/lib/locale/ca_FR.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.240326726 s in the future +tar: ./usr/lib/locale/ca_FR.utf8: time stamp 2026-11-29 07:51:59 is 34039936.240306476 s in the future +tar: ./usr/lib/locale/crh_UA: time stamp 2026-11-29 07:51:59 is 34039936.240286242 s in the future +tar: ./usr/lib/locale/es_UY/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.240265582 s in the future +tar: ./usr/lib/locale/es_UY: time stamp 2026-11-29 07:51:59 is 34039936.240244678 s in the future +tar: ./usr/lib/locale/en_GB.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.240225667 s in the future +tar: ./usr/lib/locale/en_GB.utf8: time stamp 2026-11-29 07:51:59 is 34039936.240204751 s in the future +tar: ./usr/lib/locale/da_DK: time stamp 2026-11-29 07:51:59 is 34039936.240184612 s in the future +tar: ./usr/lib/locale/ckb_IQ: time stamp 2026-11-29 07:51:59 is 34039936.240165213 s in the future +tar: ./usr/lib/locale/ur_IN: time stamp 2026-11-29 07:52:00 is 34039937.240143928 s in the future +tar: ./usr/lib/locale/so_SO/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.240124157 s in the future +tar: ./usr/lib/locale/so_SO: time stamp 2026-11-29 07:52:00 is 34039937.240104287 s in the future +tar: ./usr/lib/locale/tl_PH.utf8/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.240083163 s in the future +tar: ./usr/lib/locale/tl_PH.utf8: time stamp 2026-11-29 07:52:00 is 34039937.240063794 s in the future +tar: ./usr/lib/locale/uk_UA.utf8: time stamp 2026-11-29 07:52:00 is 34039937.240043536 s in the future +tar: ./usr/lib/locale/brx_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.240024272 s in the future +tar: ./usr/lib/locale/brx_IN: time stamp 2026-11-29 07:51:59 is 34039936.24000367 s in the future +tar: ./usr/lib/locale/kw_GB.utf8: time stamp 2026-11-29 07:51:59 is 34039936.239983069 s in the future +tar: ./usr/lib/locale/it_CH: time stamp 2026-11-29 07:51:59 is 34039936.239963291 s in the future +tar: ./usr/lib/locale/es_CR/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.239942634 s in the future +tar: ./usr/lib/locale/es_CR: time stamp 2026-11-29 07:51:59 is 34039936.23992253 s in the future +tar: ./usr/lib/locale/es_PA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.239902631 s in the future +tar: ./usr/lib/locale/es_PA: time stamp 2026-11-29 07:51:59 is 34039936.239881264 s in the future +tar: ./usr/lib/locale/kl_GL.utf8: time stamp 2026-11-29 07:51:59 is 34039936.239861366 s in the future +tar: ./usr/lib/locale/en_BW.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.239840144 s in the future +tar: ./usr/lib/locale/en_BW.utf8: time stamp 2026-11-29 07:51:59 is 34039936.239801554 s in the future +tar: ./usr/lib/locale/fi_FI@euro: time stamp 2026-11-29 07:51:59 is 34039936.239780685 s in the future +tar: ./usr/lib/locale/mi_NZ.utf8: time stamp 2026-11-29 07:51:59 is 34039936.239760713 s in the future +tar: ./usr/lib/locale/mr_IN: time stamp 2026-11-29 07:51:59 is 34039936.239740349 s in the future +tar: ./usr/lib/locale/zh_TW.utf8/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.239720242 s in the future +tar: ./usr/lib/locale/zh_TW.utf8: time stamp 2026-11-29 07:52:00 is 34039937.239699138 s in the future +tar: ./usr/lib/locale/en_IE.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.239679051 s in the future +tar: ./usr/lib/locale/en_IE.utf8: time stamp 2026-11-29 07:51:59 is 34039936.239658466 s in the future +tar: ./usr/lib/locale/ar_SS/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.239638134 s in the future +tar: ./usr/lib/locale/ar_SS: time stamp 2026-11-29 07:51:59 is 34039936.239617933 s in the future +tar: ./usr/lib/locale/dv_MV/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.239597506 s in the future +tar: ./usr/lib/locale/dv_MV: time stamp 2026-11-29 07:51:59 is 34039936.239577798 s in the future +tar: ./usr/lib/locale/es_GT/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.23955708 s in the future +tar: ./usr/lib/locale/es_GT: time stamp 2026-11-29 07:51:59 is 34039936.239536991 s in the future +tar: ./usr/lib/locale/cmn_TW: time stamp 2026-11-29 07:51:59 is 34039936.239516145 s in the future +tar: ./usr/lib/locale/ber_MA: time stamp 2026-11-29 07:51:59 is 34039936.239495327 s in the future +tar: ./usr/lib/locale/pt_BR: time stamp 2026-11-29 07:52:00 is 34039937.239474472 s in the future +tar: ./usr/lib/locale/es_AR.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.239453025 s in the future +tar: ./usr/lib/locale/es_AR.utf8: time stamp 2026-11-29 07:51:59 is 34039936.239431632 s in the future +tar: ./usr/lib/locale/es_VE.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.239410955 s in the future +tar: ./usr/lib/locale/es_VE.utf8: time stamp 2026-11-29 07:51:59 is 34039936.239389561 s in the future +tar: ./usr/lib/locale/lb_LU: time stamp 2026-11-29 07:51:59 is 34039936.239369133 s in the future +tar: ./usr/lib/locale/az_AZ: time stamp 2026-11-29 07:51:59 is 34039936.239348125 s in the future +tar: ./usr/lib/locale/om_KE: time stamp 2026-11-29 07:52:00 is 34039937.239327552 s in the future +tar: ./usr/lib/locale/en_HK/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.239307261 s in the future +tar: ./usr/lib/locale/en_HK: time stamp 2026-11-29 07:51:59 is 34039936.239285779 s in the future +tar: ./usr/lib/locale/zh_HK.utf8: time stamp 2026-11-29 07:52:00 is 34039937.239265517 s in the future +tar: ./usr/lib/locale/fr_LU.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.23924525 s in the future +tar: ./usr/lib/locale/fr_LU.utf8: time stamp 2026-11-29 07:51:59 is 34039936.239224544 s in the future +tar: ./usr/lib/locale/li_BE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.239204034 s in the future +tar: ./usr/lib/locale/li_BE: time stamp 2026-11-29 07:51:59 is 34039936.239182931 s in the future +tar: ./usr/lib/locale/li_NL: time stamp 2026-11-29 07:51:59 is 34039936.239162899 s in the future +tar: ./usr/lib/locale/as_IN: time stamp 2026-11-29 07:51:59 is 34039936.239142507 s in the future +tar: ./usr/lib/locale/tr_TR.utf8/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.239121249 s in the future +tar: ./usr/lib/locale/tr_TR.utf8: time stamp 2026-11-29 07:52:00 is 34039937.239101358 s in the future +tar: ./usr/lib/locale/zh_CN.utf8/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.239080584 s in the future +tar: ./usr/lib/locale/zh_CN.utf8: time stamp 2026-11-29 07:52:00 is 34039937.239059498 s in the future +tar: ./usr/lib/locale/aa_ER/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.239038234 s in the future +tar: ./usr/lib/locale/aa_ER: time stamp 2026-11-29 07:51:59 is 34039936.23901712 s in the future +tar: ./usr/lib/locale/ml_IN: time stamp 2026-11-29 07:51:59 is 34039936.238995811 s in the future +tar: ./usr/lib/locale/de_IT.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.238974449 s in the future +tar: ./usr/lib/locale/de_IT.utf8: time stamp 2026-11-29 07:51:59 is 34039936.23895295 s in the future +tar: ./usr/lib/locale/nl_BE@euro: time stamp 2026-11-29 07:51:59 is 34039936.238932602 s in the future +tar: ./usr/lib/locale/ar_LY.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.238911681 s in the future +tar: ./usr/lib/locale/ar_LY.utf8: time stamp 2026-11-29 07:51:59 is 34039936.238891658 s in the future +tar: ./usr/lib/locale/fr_CH: time stamp 2026-11-29 07:51:59 is 34039936.238870494 s in the future +tar: ./usr/lib/locale/scn_IT: time stamp 2026-11-29 07:52:00 is 34039937.238848893 s in the future +tar: ./usr/lib/locale/tk_TM: time stamp 2026-11-29 07:52:00 is 34039937.238827056 s in the future +tar: ./usr/lib/locale/dz_BT: time stamp 2026-11-29 07:51:59 is 34039936.238806075 s in the future +tar: ./usr/lib/locale/tig_ER/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.238784859 s in the future +tar: ./usr/lib/locale/tig_ER: time stamp 2026-11-29 07:52:00 is 34039937.238763541 s in the future +tar: ./usr/lib/locale/en_ZA.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.238743402 s in the future +tar: ./usr/lib/locale/en_ZA.utf8: time stamp 2026-11-29 07:51:59 is 34039936.238722396 s in the future +tar: ./usr/lib/locale/bo_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.238701982 s in the future +tar: ./usr/lib/locale/bo_IN: time stamp 2026-11-29 07:51:59 is 34039936.238681785 s in the future +tar: ./usr/lib/locale/mi_NZ: time stamp 2026-11-29 07:51:59 is 34039936.238660404 s in the future +tar: ./usr/lib/locale/eo: time stamp 2026-11-29 07:51:59 is 34039936.238639729 s in the future +tar: ./usr/lib/locale/gd_GB.utf8: time stamp 2026-11-29 07:51:59 is 34039936.238619046 s in the future +tar: ./usr/lib/locale/en_US.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.238598007 s in the future +tar: ./usr/lib/locale/en_US.utf8: time stamp 2026-11-29 07:51:59 is 34039936.238576941 s in the future +tar: ./usr/lib/locale/sv_FI.utf8: time stamp 2026-11-29 07:52:00 is 34039937.238555709 s in the future +tar: ./usr/lib/locale/br_FR.utf8: time stamp 2026-11-29 07:51:59 is 34039936.238535084 s in the future +tar: ./usr/lib/locale/lv_LV: time stamp 2026-11-29 07:51:59 is 34039936.238515083 s in the future +tar: ./usr/lib/locale/kab_DZ/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.238493908 s in the future +tar: ./usr/lib/locale/kab_DZ: time stamp 2026-11-29 07:51:59 is 34039936.238473586 s in the future +tar: ./usr/lib/locale/km_KH: time stamp 2026-11-29 07:51:59 is 34039936.238452604 s in the future +tar: ./usr/lib/locale/ro_RO.utf8: time stamp 2026-11-29 07:52:00 is 34039937.238431352 s in the future +tar: ./usr/lib/locale/cs_CZ: time stamp 2026-11-29 07:51:59 is 34039936.238409428 s in the future +tar: ./usr/lib/locale/es_NI.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.23838842 s in the future +tar: ./usr/lib/locale/es_NI.utf8: time stamp 2026-11-29 07:51:59 is 34039936.238367196 s in the future +tar: ./usr/lib/locale/fr_FR@euro: time stamp 2026-11-29 07:51:59 is 34039936.238346064 s in the future +tar: ./usr/lib/locale/be_BY.utf8: time stamp 2026-11-29 07:51:59 is 34039936.238325044 s in the future +tar: ./usr/lib/locale/ce_RU: time stamp 2026-11-29 07:51:59 is 34039936.23830487 s in the future +tar: ./usr/lib/locale/ga_IE.utf8: time stamp 2026-11-29 07:51:59 is 34039936.238284049 s in the future +tar: ./usr/lib/locale/es_PR.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.238264827 s in the future +tar: ./usr/lib/locale/es_PR.utf8: time stamp 2026-11-29 07:51:59 is 34039936.238243791 s in the future +tar: ./usr/lib/locale/zh_SG.gbk/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.238223223 s in the future +tar: ./usr/lib/locale/zh_SG.gbk: time stamp 2026-11-29 07:52:00 is 34039937.23820197 s in the future +tar: ./usr/lib/locale/nn_NO.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.238062823 s in the future +tar: ./usr/lib/locale/nn_NO.utf8: time stamp 2026-11-29 07:51:59 is 34039936.238030981 s in the future +tar: ./usr/lib/locale/lzh_TW: time stamp 2026-11-29 07:51:59 is 34039936.237969865 s in the future +tar: ./usr/lib/locale/es_VE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.237901127 s in the future +tar: ./usr/lib/locale/es_VE: time stamp 2026-11-29 07:51:59 is 34039936.237877046 s in the future +tar: ./usr/lib/locale/sd_IN/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.237856235 s in the future +tar: ./usr/lib/locale/sd_IN: time stamp 2026-11-29 07:52:00 is 34039937.237834991 s in the future +tar: ./usr/lib/locale/bg_BG: time stamp 2026-11-29 07:51:59 is 34039936.237759537 s in the future +tar: ./usr/lib/locale/tl_PH: time stamp 2026-11-29 07:52:00 is 34039937.237728543 s in the future +tar: ./usr/lib/locale/zu_ZA.utf8: time stamp 2026-11-29 07:52:00 is 34039937.237706732 s in the future +tar: ./usr/lib/locale/en_GB/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.237686651 s in the future +tar: ./usr/lib/locale/en_GB: time stamp 2026-11-29 07:51:59 is 34039936.237666645 s in the future +tar: ./usr/lib/locale/ar_LB/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.237600284 s in the future +tar: ./usr/lib/locale/ar_LB: time stamp 2026-11-29 07:51:59 is 34039936.237502939 s in the future +tar: ./usr/lib/locale/de_BE@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.237480488 s in the future +tar: ./usr/lib/locale/de_BE@euro: time stamp 2026-11-29 07:51:59 is 34039936.237459443 s in the future +tar: ./usr/lib/locale/lt_LT.utf8: time stamp 2026-11-29 07:51:59 is 34039936.237368532 s in the future +tar: ./usr/lib/locale/eu_FR/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.237334437 s in the future +tar: ./usr/lib/locale/eu_FR: time stamp 2026-11-29 07:51:59 is 34039936.237312698 s in the future +tar: ./usr/lib/locale/shn_MM: time stamp 2026-11-29 07:52:00 is 34039937.237293194 s in the future +tar: ./usr/lib/locale/mni_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.237271675 s in the future +tar: ./usr/lib/locale/mni_IN: time stamp 2026-11-29 07:51:59 is 34039936.237252004 s in the future +tar: ./usr/lib/locale/es_NI/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.23723088 s in the future +tar: ./usr/lib/locale/es_NI: time stamp 2026-11-29 07:51:59 is 34039936.237125595 s in the future +tar: ./usr/lib/locale/syr: time stamp 2026-11-29 07:52:00 is 34039937.237056651 s in the future +tar: ./usr/lib/locale/ky_KG: time stamp 2026-11-29 07:51:59 is 34039936.237034287 s in the future +tar: ./usr/lib/locale/de_AT: time stamp 2026-11-29 07:51:59 is 34039936.237014268 s in the future +tar: ./usr/lib/locale/wa_BE.utf8: time stamp 2026-11-29 07:52:00 is 34039937.236992944 s in the future +tar: ./usr/lib/locale/el_GR/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.236973474 s in the future +tar: ./usr/lib/locale/el_GR: time stamp 2026-11-29 07:51:59 is 34039936.236952438 s in the future +tar: ./usr/lib/locale/pap_CW/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.23688887 s in the future +tar: ./usr/lib/locale/pap_CW: time stamp 2026-11-29 07:52:00 is 34039937.236865933 s in the future +tar: ./usr/lib/locale/hu_HU.utf8: time stamp 2026-11-29 07:51:59 is 34039936.236844931 s in the future +tar: ./usr/lib/locale/mjw_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.236823866 s in the future +tar: ./usr/lib/locale/mjw_IN: time stamp 2026-11-29 07:51:59 is 34039936.236803623 s in the future +tar: ./usr/lib/locale/en_US/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.236782737 s in the future +tar: ./usr/lib/locale/en_US: time stamp 2026-11-29 07:51:59 is 34039936.23676316 s in the future +tar: ./usr/lib/locale/et_EE: time stamp 2026-11-29 07:51:59 is 34039936.236742171 s in the future +tar: ./usr/lib/locale/fr_CA: time stamp 2026-11-29 07:51:59 is 34039936.236721767 s in the future +tar: ./usr/lib/locale/tr_TR/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.236701748 s in the future +tar: ./usr/lib/locale/tr_TR: time stamp 2026-11-29 07:52:00 is 34039937.236681712 s in the future +tar: ./usr/lib/locale/bhb_IN.utf8: time stamp 2026-11-29 07:51:59 is 34039936.236660199 s in the future +tar: ./usr/lib/locale/om_KE.utf8/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.236640291 s in the future +tar: ./usr/lib/locale/om_KE.utf8: time stamp 2026-11-29 07:52:00 is 34039937.236619816 s in the future +tar: ./usr/lib/locale/ru_UA.utf8/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.236599049 s in the future +tar: ./usr/lib/locale/ru_UA.utf8: time stamp 2026-11-29 07:52:00 is 34039937.236579004 s in the future +tar: ./usr/lib/locale/ga_IE@euro: time stamp 2026-11-29 07:51:59 is 34039936.236557575 s in the future +tar: ./usr/lib/locale/en_DK: time stamp 2026-11-29 07:51:59 is 34039936.236538428 s in the future +tar: ./usr/lib/locale/es_CO.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.236517775 s in the future +tar: ./usr/lib/locale/es_CO.utf8: time stamp 2026-11-29 07:51:59 is 34039936.236497547 s in the future +tar: ./usr/lib/locale/es_SV/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.236476916 s in the future +tar: ./usr/lib/locale/es_SV: time stamp 2026-11-29 07:51:59 is 34039936.236456166 s in the future +tar: ./usr/lib/locale/ar_SA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.236435558 s in the future +tar: ./usr/lib/locale/ar_SA: time stamp 2026-11-29 07:51:59 is 34039936.23641457 s in the future +tar: ./usr/lib/locale/dsb_DE: time stamp 2026-11-29 07:51:59 is 34039936.236393623 s in the future +tar: ./usr/lib/locale/tr_CY: time stamp 2026-11-29 07:52:00 is 34039937.236372267 s in the future +tar: ./usr/lib/locale/gu_IN: time stamp 2026-11-29 07:51:59 is 34039936.236350143 s in the future +tar: ./usr/lib/locale/ar_DZ.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.236329373 s in the future +tar: ./usr/lib/locale/ar_DZ.utf8: time stamp 2026-11-29 07:51:59 is 34039936.23630894 s in the future +tar: ./usr/lib/locale/bho_NP/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.236287648 s in the future +tar: ./usr/lib/locale/bho_NP: time stamp 2026-11-29 07:51:59 is 34039936.236267575 s in the future +tar: ./usr/lib/locale/en_BW/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.236247431 s in the future +tar: ./usr/lib/locale/en_BW: time stamp 2026-11-29 07:51:59 is 34039936.236226485 s in the future +tar: ./usr/lib/locale/ln_CD: time stamp 2026-11-29 07:51:59 is 34039936.236205968 s in the future +tar: ./usr/lib/locale/de_BE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.236185796 s in the future +tar: ./usr/lib/locale/de_BE: time stamp 2026-11-29 07:51:59 is 34039936.236164587 s in the future +tar: ./usr/lib/locale/ru_RU.utf8: time stamp 2026-11-29 07:52:00 is 34039937.23614326 s in the future +tar: ./usr/lib/locale/pt_BR.utf8: time stamp 2026-11-29 07:52:00 is 34039937.236122061 s in the future +tar: ./usr/lib/locale/es_ES.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.236101442 s in the future +tar: ./usr/lib/locale/es_ES.utf8: time stamp 2026-11-29 07:51:59 is 34039936.236080348 s in the future +tar: ./usr/lib/locale/xh_ZA.utf8: time stamp 2026-11-29 07:52:00 is 34039937.236047557 s in the future +tar: ./usr/lib/locale/hi_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.236023584 s in the future +tar: ./usr/lib/locale/hi_IN: time stamp 2026-11-29 07:51:59 is 34039936.236003395 s in the future +tar: ./usr/lib/locale/ayc_PE: time stamp 2026-11-29 07:51:59 is 34039936.235983033 s in the future +tar: ./usr/lib/locale/en_NZ.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.235961937 s in the future +tar: ./usr/lib/locale/en_NZ.utf8: time stamp 2026-11-29 07:51:59 is 34039936.235941752 s in the future +tar: ./usr/lib/locale/ht_HT: time stamp 2026-11-29 07:51:59 is 34039936.235920585 s in the future +tar: ./usr/lib/locale/mg_MG.utf8: time stamp 2026-11-29 07:51:59 is 34039936.235899525 s in the future +tar: ./usr/lib/locale/ar_EG.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.235878847 s in the future +tar: ./usr/lib/locale/ar_EG.utf8: time stamp 2026-11-29 07:51:59 is 34039936.235857528 s in the future +tar: ./usr/lib/locale/ar_MA.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.235836848 s in the future +tar: ./usr/lib/locale/ar_MA.utf8: time stamp 2026-11-29 07:51:59 is 34039936.235757798 s in the future +tar: ./usr/lib/locale/zh_SG.utf8/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.235734074 s in the future +tar: ./usr/lib/locale/zh_SG.utf8: time stamp 2026-11-29 07:52:00 is 34039937.235712304 s in the future +tar: ./usr/lib/locale/en_CA.utf8: time stamp 2026-11-29 07:51:59 is 34039936.235691031 s in the future +tar: ./usr/lib/locale/de_AT@euro: time stamp 2026-11-29 07:51:59 is 34039936.235670302 s in the future +tar: ./usr/lib/locale/ar_SA.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.235649299 s in the future +tar: ./usr/lib/locale/ar_SA.utf8: time stamp 2026-11-29 07:51:59 is 34039936.235628178 s in the future +tar: ./usr/lib/locale/yo_NG: time stamp 2026-11-29 07:52:00 is 34039937.235607304 s in the future +tar: ./usr/lib/locale/es_ES/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.23558592 s in the future +tar: ./usr/lib/locale/es_ES: time stamp 2026-11-29 07:51:59 is 34039936.235565202 s in the future +tar: ./usr/lib/locale/es_HN.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.235543582 s in the future +tar: ./usr/lib/locale/es_HN.utf8: time stamp 2026-11-29 07:51:59 is 34039936.235522221 s in the future +tar: ./usr/lib/locale/en_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.235501455 s in the future +tar: ./usr/lib/locale/en_IN: time stamp 2026-11-29 07:51:59 is 34039936.235480051 s in the future +tar: ./usr/lib/locale/iu_CA: time stamp 2026-11-29 07:51:59 is 34039936.235458879 s in the future +tar: ./usr/lib/locale/es_CU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.235438728 s in the future +tar: ./usr/lib/locale/es_CU: time stamp 2026-11-29 07:51:59 is 34039936.235417122 s in the future +tar: ./usr/lib/locale/eu_ES@euro: time stamp 2026-11-29 07:51:59 is 34039936.235353817 s in the future +tar: ./usr/lib/locale/ar_LB.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.235329445 s in the future +tar: ./usr/lib/locale/ar_LB.utf8: time stamp 2026-11-29 07:51:59 is 34039936.235308497 s in the future +tar: ./usr/lib/locale/ca_ES: time stamp 2026-11-29 07:51:59 is 34039936.235287357 s in the future +tar: ./usr/lib/locale/sc_IT: time stamp 2026-11-29 07:52:00 is 34039937.23526652 s in the future +tar: ./usr/lib/locale/it_CH.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.235245815 s in the future +tar: ./usr/lib/locale/it_CH.utf8: time stamp 2026-11-29 07:51:59 is 34039936.235224498 s in the future +tar: ./usr/lib/locale/en_PH/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.235203184 s in the future +tar: ./usr/lib/locale/en_PH: time stamp 2026-11-29 07:51:59 is 34039936.235181891 s in the future +tar: ./usr/lib/locale/agr_PE: time stamp 2026-11-29 07:51:59 is 34039936.235160083 s in the future +tar: ./usr/lib/locale/ar_IQ/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.235138361 s in the future +tar: ./usr/lib/locale/ar_IQ: time stamp 2026-11-29 07:51:59 is 34039936.235116265 s in the future +tar: ./usr/lib/locale/gv_GB.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.235094924 s in the future +tar: ./usr/lib/locale/gv_GB.utf8: time stamp 2026-11-29 07:51:59 is 34039936.235074135 s in the future +tar: ./usr/lib/locale/en_ZW.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.235052728 s in the future +tar: ./usr/lib/locale/en_ZW.utf8: time stamp 2026-11-29 07:51:59 is 34039936.235032138 s in the future +tar: ./usr/lib/locale/ss_ZA: time stamp 2026-11-29 07:52:00 is 34039937.23501126 s in the future +tar: ./usr/lib/locale/ti_ER/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.234990069 s in the future +tar: ./usr/lib/locale/ti_ER: time stamp 2026-11-29 07:52:00 is 34039937.234967972 s in the future +tar: ./usr/lib/locale/de_DE.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.234947618 s in the future +tar: ./usr/lib/locale/de_DE.utf8: time stamp 2026-11-29 07:51:59 is 34039936.234926041 s in the future +tar: ./usr/lib/locale/es_CL.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.234905761 s in the future +tar: ./usr/lib/locale/es_CL.utf8: time stamp 2026-11-29 07:51:59 is 34039936.234884211 s in the future +tar: ./usr/lib/locale/hr_HR.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.234863473 s in the future +tar: ./usr/lib/locale/hr_HR.utf8: time stamp 2026-11-29 07:51:59 is 34039936.234842307 s in the future +tar: ./usr/lib/locale/es_BO/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.234822411 s in the future +tar: ./usr/lib/locale/es_BO: time stamp 2026-11-29 07:51:59 is 34039936.234800944 s in the future +tar: ./usr/lib/locale/es_PE.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.234779803 s in the future +tar: ./usr/lib/locale/es_PE.utf8: time stamp 2026-11-29 07:51:59 is 34039936.234759481 s in the future +tar: ./usr/lib/locale/fo_FO: time stamp 2026-11-29 07:51:59 is 34039936.234738304 s in the future +tar: ./usr/lib/locale/my_MM: time stamp 2026-11-29 07:51:59 is 34039936.23471712 s in the future +tar: ./usr/lib/locale/so_ET/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.234695478 s in the future +tar: ./usr/lib/locale/so_ET: time stamp 2026-11-29 07:52:00 is 34039937.234673903 s in the future +tar: ./usr/lib/locale/ca_IT/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.234653132 s in the future +tar: ./usr/lib/locale/ca_IT: time stamp 2026-11-29 07:51:59 is 34039936.234631601 s in the future +tar: ./usr/lib/locale/ar_JO/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.23461117 s in the future +tar: ./usr/lib/locale/ar_JO: time stamp 2026-11-29 07:51:59 is 34039936.234589786 s in the future +tar: ./usr/lib/locale/es_SV.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.234569065 s in the future +tar: ./usr/lib/locale/es_SV.utf8: time stamp 2026-11-29 07:51:59 is 34039936.234547098 s in the future +tar: ./usr/lib/locale/ro_RO: time stamp 2026-11-29 07:52:00 is 34039937.234526097 s in the future +tar: ./usr/lib/locale/fr_FR/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.234505205 s in the future +tar: ./usr/lib/locale/fr_FR: time stamp 2026-11-29 07:51:59 is 34039936.234485075 s in the future +tar: ./usr/lib/locale/rif_MA: time stamp 2026-11-29 07:52:00 is 34039937.234464209 s in the future +tar: ./usr/lib/locale/et_EE.utf8: time stamp 2026-11-29 07:51:59 is 34039936.23444312 s in the future +tar: ./usr/lib/locale/ja_JP.utf8: time stamp 2026-11-29 07:51:59 is 34039936.234422104 s in the future +tar: ./usr/lib/locale/mai_NP/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.234400842 s in the future +tar: ./usr/lib/locale/mai_NP: time stamp 2026-11-29 07:51:59 is 34039936.234379373 s in the future +tar: ./usr/lib/locale/chr_US: time stamp 2026-11-29 07:51:59 is 34039936.234358341 s in the future +tar: ./usr/lib/locale/ha_NG: time stamp 2026-11-29 07:51:59 is 34039936.234130152 s in the future +tar: ./usr/lib/locale/gez_ET: time stamp 2026-11-29 07:51:59 is 34039936.234060222 s in the future +tar: ./usr/lib/locale/mai_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.234025933 s in the future +tar: ./usr/lib/locale/mai_IN: time stamp 2026-11-29 07:51:59 is 34039936.234005335 s in the future +tar: ./usr/lib/locale/yuw_PG: time stamp 2026-11-29 07:52:00 is 34039937.233984581 s in the future +tar: ./usr/lib/locale/ps_AF: time stamp 2026-11-29 07:52:00 is 34039937.233963526 s in the future +tar: ./usr/lib/locale/miq_NI/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.233942649 s in the future +tar: ./usr/lib/locale/miq_NI: time stamp 2026-11-29 07:51:59 is 34039936.233921933 s in the future +tar: ./usr/lib/locale/ssy_ER/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.233768394 s in the future +tar: ./usr/lib/locale/ssy_ER: time stamp 2026-11-29 07:52:00 is 34039937.233745729 s in the future +tar: ./usr/lib/locale/kok_IN: time stamp 2026-11-29 07:51:59 is 34039936.23372469 s in the future +tar: ./usr/lib/locale/ar_BH/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.233704451 s in the future +tar: ./usr/lib/locale/ar_BH: time stamp 2026-11-29 07:51:59 is 34039936.233683602 s in the future +tar: ./usr/lib/locale/fr_BE: time stamp 2026-11-29 07:51:59 is 34039936.233663173 s in the future +tar: ./usr/lib/locale/am_ET: time stamp 2026-11-29 07:51:59 is 34039936.233642385 s in the future +tar: ./usr/lib/locale/es_EC/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.233621266 s in the future +tar: ./usr/lib/locale/es_EC: time stamp 2026-11-29 07:51:59 is 34039936.233602069 s in the future +tar: ./usr/lib/locale/sv_FI@euro: time stamp 2026-11-29 07:52:00 is 34039937.233581779 s in the future +tar: ./usr/lib/locale/nb_NO.utf8: time stamp 2026-11-29 07:51:59 is 34039936.233560085 s in the future +tar: ./usr/lib/locale/tt_RU@iqtelif: time stamp 2026-11-29 07:52:00 is 34039937.233539648 s in the future +tar: ./usr/lib/locale/kn_IN: time stamp 2026-11-29 07:51:59 is 34039936.233518097 s in the future +tar: ./usr/lib/locale/eu_FR.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.233497736 s in the future +tar: ./usr/lib/locale/eu_FR.utf8: time stamp 2026-11-29 07:51:59 is 34039936.233477399 s in the future +tar: ./usr/lib/locale/az_IR: time stamp 2026-11-29 07:51:59 is 34039936.233457257 s in the future +tar: ./usr/lib/locale/es_CO/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.233436562 s in the future +tar: ./usr/lib/locale/es_CO: time stamp 2026-11-29 07:51:59 is 34039936.233415883 s in the future +tar: ./usr/lib/locale/ru_RU: time stamp 2026-11-29 07:52:00 is 34039937.233395204 s in the future +tar: ./usr/lib/locale/en_IE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.233374925 s in the future +tar: ./usr/lib/locale/en_IE: time stamp 2026-11-29 07:51:59 is 34039936.233353811 s in the future +tar: ./usr/lib/locale/fi_FI: time stamp 2026-11-29 07:51:59 is 34039936.233333418 s in the future +tar: ./usr/lib/locale/sa_IN: time stamp 2026-11-29 07:52:00 is 34039937.233312375 s in the future +tar: ./usr/lib/locale/sl_SI.utf8: time stamp 2026-11-29 07:52:00 is 34039937.233290828 s in the future +tar: ./usr/lib/locale/ks_IN@devanagari: time stamp 2026-11-29 07:51:59 is 34039936.233270352 s in the future +tar: ./usr/lib/locale/ca_ES.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.233248598 s in the future +tar: ./usr/lib/locale/ca_ES.utf8: time stamp 2026-11-29 07:51:59 is 34039936.233229272 s in the future +tar: ./usr/lib/locale/ca_ES@valencia/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.233207953 s in the future +tar: ./usr/lib/locale/ca_ES@valencia: time stamp 2026-11-29 07:51:59 is 34039936.233187005 s in the future +tar: ./usr/lib/locale/ru_RU.cp1251: time stamp 2026-11-29 07:52:00 is 34039937.23316675 s in the future +tar: ./usr/lib/locale/bn_BD: time stamp 2026-11-29 07:51:59 is 34039936.233145905 s in the future +tar: ./usr/lib/locale/ku_TR.utf8: time stamp 2026-11-29 07:51:59 is 34039936.23312554 s in the future +tar: ./usr/lib/locale/ast_ES: time stamp 2026-11-29 07:51:59 is 34039936.233104692 s in the future +tar: ./usr/lib/locale/gl_ES: time stamp 2026-11-29 07:51:59 is 34039936.233084078 s in the future +tar: ./usr/lib/locale/lo_LA: time stamp 2026-11-29 07:51:59 is 34039936.233064492 s in the future +tar: ./usr/lib/locale/en_NZ/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.233043777 s in the future +tar: ./usr/lib/locale/en_NZ: time stamp 2026-11-29 07:51:59 is 34039936.233022984 s in the future +tar: ./usr/lib/locale/kw_GB: time stamp 2026-11-29 07:51:59 is 34039936.233001709 s in the future +tar: ./usr/lib/locale/lij_IT: time stamp 2026-11-29 07:51:59 is 34039936.232980516 s in the future +tar: ./usr/lib/locale/de_DE@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.232960179 s in the future +tar: ./usr/lib/locale/de_DE@euro: time stamp 2026-11-29 07:51:59 is 34039936.232938919 s in the future +tar: ./usr/lib/locale/kv_RU: time stamp 2026-11-29 07:51:59 is 34039936.232917763 s in the future +tar: ./usr/lib/locale/pl_PL.utf8: time stamp 2026-11-29 07:52:00 is 34039937.232896072 s in the future +tar: ./usr/lib/locale/ff_SN: time stamp 2026-11-29 07:51:59 is 34039936.232874894 s in the future +tar: ./usr/lib/locale/en_ZM/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.232853502 s in the future +tar: ./usr/lib/locale/en_ZM: time stamp 2026-11-29 07:51:59 is 34039936.232832974 s in the future +tar: ./usr/lib/locale/en_SC.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.232811905 s in the future +tar: ./usr/lib/locale/en_SC.utf8: time stamp 2026-11-29 07:51:59 is 34039936.232790229 s in the future +tar: ./usr/lib/locale/el_GR.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.232768841 s in the future +tar: ./usr/lib/locale/el_GR.utf8: time stamp 2026-11-29 07:51:59 is 34039936.23274788 s in the future +tar: ./usr/lib/locale/sr_ME: time stamp 2026-11-29 07:52:00 is 34039937.232725885 s in the future +tar: ./usr/lib/locale/en_US.iso885915/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.232704578 s in the future +tar: ./usr/lib/locale/en_US.iso885915: time stamp 2026-11-29 07:51:59 is 34039936.232683104 s in the future +tar: ./usr/lib/locale/en_DK.iso885915: time stamp 2026-11-29 07:51:59 is 34039936.232662028 s in the future +tar: ./usr/lib/locale/sq_AL.utf8: time stamp 2026-11-29 07:52:00 is 34039937.232641066 s in the future +tar: ./usr/lib/locale/oc_FR.utf8: time stamp 2026-11-29 07:52:00 is 34039937.232569801 s in the future +tar: ./usr/lib/locale/ar_IQ.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.232547974 s in the future +tar: ./usr/lib/locale/ar_IQ.utf8: time stamp 2026-11-29 07:51:59 is 34039936.232526229 s in the future +tar: ./usr/lib/locale/sq_AL: time stamp 2026-11-29 07:52:00 is 34039937.232504691 s in the future +tar: ./usr/lib/locale/hsb_DE: time stamp 2026-11-29 07:51:59 is 34039936.232483731 s in the future +tar: ./usr/lib/locale/ar_YE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.232462724 s in the future +tar: ./usr/lib/locale/ar_YE: time stamp 2026-11-29 07:51:59 is 34039936.232441648 s in the future +tar: ./usr/lib/locale/fy_NL/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.23242076 s in the future +tar: ./usr/lib/locale/fy_NL: time stamp 2026-11-29 07:51:59 is 34039936.232398591 s in the future +tar: ./usr/lib/locale/ar_TN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.232377587 s in the future +tar: ./usr/lib/locale/ar_TN: time stamp 2026-11-29 07:51:59 is 34039936.232357032 s in the future +tar: ./usr/lib/locale/si_LK: time stamp 2026-11-29 07:52:00 is 34039937.232335753 s in the future +tar: ./usr/lib/locale/it_IT: time stamp 2026-11-29 07:51:59 is 34039936.23231379 s in the future +tar: ./usr/lib/locale/de_IT/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.232293694 s in the future +tar: ./usr/lib/locale/de_IT: time stamp 2026-11-29 07:51:59 is 34039936.23227227 s in the future +tar: ./usr/lib/locale/es_PA.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.232251424 s in the future +tar: ./usr/lib/locale/es_PA.utf8: time stamp 2026-11-29 07:51:59 is 34039936.232229715 s in the future +tar: ./usr/lib/locale/ar_QA.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.232207765 s in the future +tar: ./usr/lib/locale/ar_QA.utf8: time stamp 2026-11-29 07:51:59 is 34039936.232186633 s in the future +tar: ./usr/lib/locale/it_IT.utf8: time stamp 2026-11-29 07:51:59 is 34039936.232165455 s in the future +tar: ./usr/lib/locale/szl_PL: time stamp 2026-11-29 07:52:00 is 34039937.232144067 s in the future +tar: ./usr/lib/locale/ar_SY.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.232123238 s in the future +tar: ./usr/lib/locale/ar_SY.utf8: time stamp 2026-11-29 07:51:59 is 34039936.232103978 s in the future +tar: ./usr/lib/locale/en_ZA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.232083974 s in the future +tar: ./usr/lib/locale/en_ZA: time stamp 2026-11-29 07:51:59 is 34039936.232064195 s in the future +tar: ./usr/lib/locale/mhr_RU: time stamp 2026-11-29 07:51:59 is 34039936.232043436 s in the future +tar: ./usr/lib/locale/da_DK.utf8: time stamp 2026-11-29 07:51:59 is 34039936.232022888 s in the future +tar: ./usr/lib/locale/es_DO/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.232001789 s in the future +tar: ./usr/lib/locale/es_DO: time stamp 2026-11-29 07:51:59 is 34039936.231979485 s in the future +tar: ./usr/lib/locale/gl_ES.utf8: time stamp 2026-11-29 07:51:59 is 34039936.23195805 s in the future +tar: ./usr/lib/locale/mag_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.231936336 s in the future +tar: ./usr/lib/locale/mag_IN: time stamp 2026-11-29 07:51:59 is 34039936.231915745 s in the future +tar: ./usr/lib/locale/ug_CN: time stamp 2026-11-29 07:52:00 is 34039937.231893421 s in the future +tar: ./usr/lib/locale/nl_NL.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.231872603 s in the future +tar: ./usr/lib/locale/nl_NL.utf8: time stamp 2026-11-29 07:51:59 is 34039936.231850906 s in the future +tar: ./usr/lib/locale/mk_MK.utf8: time stamp 2026-11-29 07:51:59 is 34039936.231829178 s in the future +tar: ./usr/lib/locale/fr_BE.utf8: time stamp 2026-11-29 07:51:59 is 34039936.231792852 s in the future +tar: ./usr/lib/locale/sv_SE/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.231773724 s in the future +tar: ./usr/lib/locale/sv_SE: time stamp 2026-11-29 07:52:00 is 34039937.231755116 s in the future +tar: ./usr/lib/locale/gd_GB: time stamp 2026-11-29 07:51:59 is 34039936.231735261 s in the future +tar: ./usr/lib/locale/ar_KW.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.23171519 s in the future +tar: ./usr/lib/locale/ar_KW.utf8: time stamp 2026-11-29 07:51:59 is 34039936.231693472 s in the future +tar: ./usr/lib/locale/lv_LV.utf8: time stamp 2026-11-29 07:51:59 is 34039936.231671846 s in the future +tar: ./usr/lib/locale/en_AU.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.231650839 s in the future +tar: ./usr/lib/locale/en_AU.utf8: time stamp 2026-11-29 07:51:59 is 34039936.23158309 s in the future +tar: ./usr/lib/locale/wa_BE@euro: time stamp 2026-11-29 07:52:00 is 34039937.231554434 s in the future +tar: ./usr/lib/locale/ur_PK: time stamp 2026-11-29 07:52:00 is 34039937.23153374 s in the future +tar: ./usr/lib/locale/af_ZA.utf8: time stamp 2026-11-29 07:51:59 is 34039936.231512607 s in the future +tar: ./usr/lib/locale/mt_MT.utf8: time stamp 2026-11-29 07:51:59 is 34039936.231490884 s in the future +tar: ./usr/lib/locale/en_GB.iso885915: time stamp 2026-11-29 07:51:59 is 34039936.231469396 s in the future +tar: ./usr/lib/locale/en_AU: time stamp 2026-11-29 07:51:59 is 34039936.231448171 s in the future +tar: ./usr/lib/locale/nn_NO/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.231381764 s in the future +tar: ./usr/lib/locale/nn_NO: time stamp 2026-11-29 07:51:59 is 34039936.231356965 s in the future +tar: ./usr/lib/locale/pt_PT.utf8/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.231334735 s in the future +tar: ./usr/lib/locale/pt_PT.utf8: time stamp 2026-11-29 07:52:00 is 34039937.231312338 s in the future +tar: ./usr/lib/locale/tpi_PG: time stamp 2026-11-29 07:52:00 is 34039937.231290431 s in the future +tar: ./usr/lib/locale/pa_PK: time stamp 2026-11-29 07:52:00 is 34039937.231201452 s in the future +tar: ./usr/lib/locale/en_ZW/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.231168401 s in the future +tar: ./usr/lib/locale/en_ZW: time stamp 2026-11-29 07:51:59 is 34039936.231146851 s in the future +tar: ./usr/lib/locale/ca_IT.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.231127133 s in the future +tar: ./usr/lib/locale/ca_IT.utf8: time stamp 2026-11-29 07:51:59 is 34039936.231105764 s in the future +tar: ./usr/lib/locale/uz_UZ@cyrillic: time stamp 2026-11-29 07:52:00 is 34039937.231085462 s in the future +tar: ./usr/lib/locale/ts_ZA: time stamp 2026-11-29 07:52:00 is 34039937.231063398 s in the future +tar: ./usr/lib/locale/gez_ER@abegede/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.231041614 s in the future +tar: ./usr/lib/locale/gez_ER@abegede: time stamp 2026-11-29 07:51:59 is 34039936.23101982 s in the future +tar: ./usr/lib/locale/hsb_DE.utf8: time stamp 2026-11-29 07:51:59 is 34039936.23100023 s in the future +tar: ./usr/lib/locale/es_DO.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.230977741 s in the future +tar: ./usr/lib/locale/es_DO.utf8: time stamp 2026-11-29 07:51:59 is 34039936.230914106 s in the future +tar: ./usr/lib/locale/th_TH.utf8: time stamp 2026-11-29 07:52:00 is 34039937.230889139 s in the future +tar: ./usr/lib/locale/ia_FR: time stamp 2026-11-29 07:51:59 is 34039936.230826462 s in the future +tar: ./usr/lib/locale/aa_ET/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.230800735 s in the future +tar: ./usr/lib/locale/aa_ET: time stamp 2026-11-29 07:51:59 is 34039936.230778462 s in the future +tar: ./usr/lib/locale/ar_OM/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.230757702 s in the future +tar: ./usr/lib/locale/ar_OM: time stamp 2026-11-29 07:51:59 is 34039936.230737017 s in the future +tar: ./usr/lib/locale/niu_NU: time stamp 2026-11-29 07:51:59 is 34039936.230671064 s in the future +tar: ./usr/lib/locale/uz_UZ: time stamp 2026-11-29 07:52:00 is 34039937.230648011 s in the future +tar: ./usr/lib/locale/to_TO: time stamp 2026-11-29 07:52:00 is 34039937.230627176 s in the future +tar: ./usr/lib/locale/ca_FR/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.230565251 s in the future +tar: ./usr/lib/locale/ca_FR: time stamp 2026-11-29 07:51:59 is 34039936.230541292 s in the future +tar: ./usr/lib/locale/fr_CH.utf8: time stamp 2026-11-29 07:51:59 is 34039936.230480217 s in the future +tar: ./usr/lib/locale/quz_PE: time stamp 2026-11-29 07:52:00 is 34039937.230453554 s in the future +tar: ./usr/lib/locale/de_LI.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.230431702 s in the future +tar: ./usr/lib/locale/de_LI.utf8: time stamp 2026-11-29 07:51:59 is 34039936.230410467 s in the future +tar: ./usr/lib/locale/ik_CA: time stamp 2026-11-29 07:51:59 is 34039936.230388259 s in the future +tar: ./usr/lib/locale/nl_NL@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.230367652 s in the future +tar: ./usr/lib/locale/nl_NL@euro: time stamp 2026-11-29 07:51:59 is 34039936.230345678 s in the future +tar: ./usr/lib/locale/or_IN: time stamp 2026-11-29 07:52:00 is 34039937.230324657 s in the future +tar: ./usr/lib/locale/ko_KR.utf8: time stamp 2026-11-29 07:51:59 is 34039936.230303969 s in the future +tar: ./usr/lib/locale/en_SG.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.230282585 s in the future +tar: ./usr/lib/locale/en_SG.utf8: time stamp 2026-11-29 07:51:59 is 34039936.23026219 s in the future +tar: ./usr/lib/locale/he_IL.utf8: time stamp 2026-11-29 07:51:59 is 34039936.230241412 s in the future +tar: ./usr/lib/locale/es_US.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.230219711 s in the future +tar: ./usr/lib/locale/es_US.utf8: time stamp 2026-11-29 07:51:59 is 34039936.230199931 s in the future +tar: ./usr/lib/locale/lg_UG.utf8: time stamp 2026-11-29 07:51:59 is 34039936.230178278 s in the future +tar: ./usr/lib/locale/es_EC.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.230158421 s in the future +tar: ./usr/lib/locale/es_EC.utf8: time stamp 2026-11-29 07:51:59 is 34039936.23013703 s in the future +tar: ./usr/lib/locale/es_AR/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.230115917 s in the future +tar: ./usr/lib/locale/es_AR: time stamp 2026-11-29 07:51:59 is 34039936.230095051 s in the future +tar: ./usr/lib/locale/de_AT.utf8: time stamp 2026-11-29 07:51:59 is 34039936.230073364 s in the future +tar: ./usr/lib/locale/zh_SG/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.230052909 s in the future +tar: ./usr/lib/locale/zh_SG: time stamp 2026-11-29 07:52:00 is 34039937.230031152 s in the future +tar: ./usr/lib/locale/te_IN: time stamp 2026-11-29 07:52:00 is 34039937.230009329 s in the future +tar: ./usr/lib/locale/et_EE.iso885915: time stamp 2026-11-29 07:51:59 is 34039936.229989329 s in the future +tar: ./usr/lib/locale/anp_IN: time stamp 2026-11-29 07:51:59 is 34039936.229967729 s in the future +tar: ./usr/lib/locale/mg_MG: time stamp 2026-11-29 07:51:59 is 34039936.229945606 s in the future +tar: ./usr/lib/locale/os_RU: time stamp 2026-11-29 07:52:00 is 34039937.229924261 s in the future +tar: ./usr/lib/locale/ar_QA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.229902486 s in the future +tar: ./usr/lib/locale/ar_QA: time stamp 2026-11-29 07:51:59 is 34039936.229880972 s in the future +tar: ./usr/lib/locale/fi_FI.utf8: time stamp 2026-11-29 07:51:59 is 34039936.229858833 s in the future +tar: ./usr/lib/locale/es_UY.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.229836779 s in the future +tar: ./usr/lib/locale/es_UY.utf8: time stamp 2026-11-29 07:51:59 is 34039936.229814624 s in the future +tar: ./usr/lib/locale/sat_IN: time stamp 2026-11-29 07:52:00 is 34039937.229793881 s in the future +tar: ./usr/lib/locale/mfe_MU: time stamp 2026-11-29 07:51:59 is 34039936.229771621 s in the future +tar: ./usr/lib/locale/fa_IR: time stamp 2026-11-29 07:51:59 is 34039936.229750122 s in the future +tar: ./usr/lib/locale/ar_DZ/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.229728482 s in the future +tar: ./usr/lib/locale/ar_DZ: time stamp 2026-11-29 07:51:59 is 34039936.229706844 s in the future +tar: ./usr/lib/locale/ti_ET/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.229686022 s in the future +tar: ./usr/lib/locale/ti_ET: time stamp 2026-11-29 07:52:00 is 34039937.229664314 s in the future +tar: ./usr/lib/locale/mdf_RU: time stamp 2026-11-29 07:51:59 is 34039936.229642548 s in the future +tar: ./usr/lib/locale/ar_KW/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.22962089 s in the future +tar: ./usr/lib/locale/ar_KW: time stamp 2026-11-29 07:51:59 is 34039936.229598469 s in the future +tar: ./usr/lib/locale/so_SO.utf8/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.229577237 s in the future +tar: ./usr/lib/locale/so_SO.utf8: time stamp 2026-11-29 07:52:00 is 34039937.229555235 s in the future +tar: ./usr/lib/locale/hu_HU: time stamp 2026-11-29 07:51:59 is 34039936.229534671 s in the future +tar: ./usr/lib/locale/el_CY.utf8: time stamp 2026-11-29 07:51:59 is 34039936.229513405 s in the future +tar: ./usr/lib/locale/cs_CZ.utf8: time stamp 2026-11-29 07:51:59 is 34039936.229491532 s in the future +tar: ./usr/lib/locale/sv_FI: time stamp 2026-11-29 07:52:00 is 34039937.229469157 s in the future +tar: ./usr/lib/locale/de_CH/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.229405094 s in the future +tar: ./usr/lib/locale/de_CH: time stamp 2026-11-29 07:51:59 is 34039936.229377399 s in the future +tar: ./usr/lib/locale/ak_GH: time stamp 2026-11-29 07:51:59 is 34039936.229355759 s in the future +tar: ./usr/lib/locale/sr_RS/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.229334804 s in the future +tar: ./usr/lib/locale/sr_RS: time stamp 2026-11-29 07:52:00 is 34039937.229312541 s in the future +tar: ./usr/lib/locale/es_ES@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.229290642 s in the future +tar: ./usr/lib/locale/es_ES@euro: time stamp 2026-11-29 07:51:59 is 34039936.229267971 s in the future +tar: ./usr/lib/locale/ar_OM.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.229246827 s in the future +tar: ./usr/lib/locale/ar_OM.utf8: time stamp 2026-11-29 07:51:59 is 34039936.229225167 s in the future +tar: ./usr/lib/locale/nl_NL/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.229202561 s in the future +tar: ./usr/lib/locale/nl_NL: time stamp 2026-11-29 07:51:59 is 34039936.229182203 s in the future +tar: ./usr/lib/locale/sr_RS@latin/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.229161715 s in the future +tar: ./usr/lib/locale/sr_RS@latin: time stamp 2026-11-29 07:52:00 is 34039937.229139677 s in the future +tar: ./usr/lib/locale/hy_AM: time stamp 2026-11-29 07:51:59 is 34039936.229116972 s in the future +tar: ./usr/lib/locale/sm_WS: time stamp 2026-11-29 07:52:00 is 34039937.22909611 s in the future +tar: ./usr/lib/locale/csb_PL: time stamp 2026-11-29 07:51:59 is 34039936.229073704 s in the future +tar: ./usr/lib/locale/ast_ES.utf8: time stamp 2026-11-29 07:51:59 is 34039936.229051447 s in the future +tar: ./usr/lib/locale/mnw_MM: time stamp 2026-11-29 07:51:59 is 34039936.229030896 s in the future +tar: ./usr/lib/locale/nl_BE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.229007977 s in the future +tar: ./usr/lib/locale/nl_BE: time stamp 2026-11-29 07:51:59 is 34039936.228985067 s in the future +tar: ./usr/lib/locale/wal_ET/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.228963309 s in the future +tar: ./usr/lib/locale/wal_ET: time stamp 2026-11-29 07:52:00 is 34039937.228941755 s in the future +tar: ./usr/lib/locale/es_PY/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.228920886 s in the future +tar: ./usr/lib/locale/es_PY: time stamp 2026-11-29 07:51:59 is 34039936.228898625 s in the future +tar: ./usr/lib/locale/the_NP/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.228876599 s in the future +tar: ./usr/lib/locale/the_NP: time stamp 2026-11-29 07:52:00 is 34039937.228854859 s in the future +tar: ./usr/lib/locale/de_CH.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.228832381 s in the future +tar: ./usr/lib/locale/de_CH.utf8: time stamp 2026-11-29 07:51:59 is 34039936.228810048 s in the future +tar: ./usr/lib/locale/es_MX/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.228788646 s in the future +tar: ./usr/lib/locale/es_MX: time stamp 2026-11-29 07:51:59 is 34039936.228765978 s in the future +tar: ./usr/lib/locale/bi_VU: time stamp 2026-11-29 07:51:59 is 34039936.228745089 s in the future +tar: ./usr/lib/locale/yue_HK: time stamp 2026-11-29 07:52:00 is 34039937.228722607 s in the future +tar: ./usr/lib/locale/xh_ZA: time stamp 2026-11-29 07:52:00 is 34039937.228700679 s in the future +tar: ./usr/lib/locale/sid_ET/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.22867795 s in the future +tar: ./usr/lib/locale/sid_ET: time stamp 2026-11-29 07:52:00 is 34039937.228655318 s in the future +tar: ./usr/lib/locale/hif_FJ: time stamp 2026-11-29 07:51:59 is 34039936.228632455 s in the future +tar: ./usr/lib/locale/es_CL/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.228610934 s in the future +tar: ./usr/lib/locale/es_CL: time stamp 2026-11-29 07:51:59 is 34039936.228589014 s in the future +tar: ./usr/lib/locale/rw_RW: time stamp 2026-11-29 07:52:00 is 34039937.228567585 s in the future +tar: ./usr/lib/locale/sv_SE.iso885915/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.228545644 s in the future +tar: ./usr/lib/locale/sv_SE.iso885915: time stamp 2026-11-29 07:52:00 is 34039937.228524095 s in the future +tar: ./usr/lib/locale/it_IT@euro: time stamp 2026-11-29 07:51:59 is 34039936.228501858 s in the future +tar: ./usr/lib/locale/sv_SE.utf8/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.228479639 s in the future +tar: ./usr/lib/locale/sv_SE.utf8: time stamp 2026-11-29 07:52:00 is 34039937.228456721 s in the future +tar: ./usr/lib/locale/wo_SN: time stamp 2026-11-29 07:52:00 is 34039937.228434294 s in the future +tar: ./usr/lib/locale/ar_JO.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.22841206 s in the future +tar: ./usr/lib/locale/ar_JO.utf8: time stamp 2026-11-29 07:51:59 is 34039936.228389704 s in the future +tar: ./usr/lib/locale/es_CR.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.228368868 s in the future +tar: ./usr/lib/locale/es_CR.utf8: time stamp 2026-11-29 07:51:59 is 34039936.228346662 s in the future +tar: ./usr/lib/locale/fo_FO.utf8: time stamp 2026-11-29 07:51:59 is 34039936.228325431 s in the future +tar: ./usr/lib/locale/es_US/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.228303199 s in the future +tar: ./usr/lib/locale/es_US: time stamp 2026-11-29 07:51:59 is 34039936.228282503 s in the future +tar: ./usr/lib/locale/nr_ZA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.228260107 s in the future +tar: ./usr/lib/locale/nr_ZA: time stamp 2026-11-29 07:51:59 is 34039936.228238382 s in the future +tar: ./usr/lib/locale/fr_LU@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.228214757 s in the future +tar: ./usr/lib/locale/fr_LU@euro: time stamp 2026-11-29 07:51:59 is 34039936.228192661 s in the future +tar: ./usr/lib/locale/is_IS.utf8: time stamp 2026-11-29 07:51:59 is 34039936.228170102 s in the future +tar: ./usr/lib/locale/tok: time stamp 2026-11-29 07:52:00 is 34039937.228146888 s in the future +tar: ./usr/lib/locale/en_DK.utf8: time stamp 2026-11-29 07:51:59 is 34039936.228067565 s in the future +tar: ./usr/lib/locale/eu_ES: time stamp 2026-11-29 07:51:59 is 34039936.228033561 s in the future +tar: ./usr/lib/locale/fur_IT: time stamp 2026-11-29 07:51:59 is 34039936.228011125 s in the future +tar: ./usr/lib/locale/so_KE: time stamp 2026-11-29 07:52:00 is 34039937.22799023 s in the future +tar: ./usr/lib/locale/cv_RU: time stamp 2026-11-29 07:51:59 is 34039936.22796857 s in the future +tar: ./usr/lib/locale/an_ES.utf8: time stamp 2026-11-29 07:51:59 is 34039936.227947332 s in the future +tar: ./usr/lib/locale/nds_DE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.227925508 s in the future +tar: ./usr/lib/locale/nds_DE: time stamp 2026-11-29 07:51:59 is 34039936.227900964 s in the future +tar: ./usr/lib/locale/raj_IN/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.227877937 s in the future +tar: ./usr/lib/locale/raj_IN: time stamp 2026-11-29 07:52:00 is 34039937.227855028 s in the future +tar: ./usr/lib/locale/ne_NP: time stamp 2026-11-29 07:51:59 is 34039936.227833178 s in the future +tar: ./usr/lib/locale/om_ET: time stamp 2026-11-29 07:52:00 is 34039937.227597481 s in the future +tar: ./usr/lib/locale/hak_TW: time stamp 2026-11-29 07:51:59 is 34039936.227570648 s in the future +tar: ./usr/lib/locale/es_HN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.227548758 s in the future +tar: ./usr/lib/locale/es_HN: time stamp 2026-11-29 07:51:59 is 34039936.227484672 s in the future +tar: ./usr/lib/locale/doi_IN: time stamp 2026-11-29 07:51:59 is 34039936.227420397 s in the future +tar: ./usr/lib/locale/pl_PL: time stamp 2026-11-29 07:52:00 is 34039937.227392607 s in the future +tar: ./usr/lib/locale/ig_NG: time stamp 2026-11-29 07:51:59 is 34039936.227371416 s in the future +tar: ./usr/lib/locale/gbm_IN: time stamp 2026-11-29 07:51:59 is 34039936.227349427 s in the future +tar: ./usr/lib/locale/byn_ER: time stamp 2026-11-29 07:51:59 is 34039936.22732686 s in the future +tar: ./usr/lib/locale/sl_SI: time stamp 2026-11-29 07:52:00 is 34039937.227304722 s in the future +tar: ./usr/lib/locale/gl_ES@euro: time stamp 2026-11-29 07:51:59 is 34039936.2272815 s in the future +tar: ./usr/lib/locale/ar_BH.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.227259878 s in the future +tar: ./usr/lib/locale/ar_BH.utf8: time stamp 2026-11-29 07:51:59 is 34039936.227238046 s in the future +tar: ./usr/lib/locale/sah_RU: time stamp 2026-11-29 07:52:00 is 34039937.227215672 s in the future +tar: ./usr/lib/locale/es_PY.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.22719456 s in the future +tar: ./usr/lib/locale/es_PY.utf8: time stamp 2026-11-29 07:51:59 is 34039936.227172817 s in the future +tar: ./usr/lib/locale/so_DJ: time stamp 2026-11-29 07:52:00 is 34039937.227151495 s in the future +tar: ./usr/lib/locale/sq_MK/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.227129014 s in the future +tar: ./usr/lib/locale/sq_MK: time stamp 2026-11-29 07:52:00 is 34039937.22710798 s in the future +tar: ./usr/lib/locale/en_IE@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.227086377 s in the future +tar: ./usr/lib/locale/en_IE@euro: time stamp 2026-11-29 07:51:59 is 34039936.227064441 s in the future +tar: ./usr/lib/locale/wa_BE: time stamp 2026-11-29 07:52:00 is 34039937.227042161 s in the future +tar: ./usr/lib/locale/el_GR@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.227020387 s in the future +tar: ./usr/lib/locale/el_GR@euro: time stamp 2026-11-29 07:51:59 is 34039936.226999602 s in the future +tar: ./usr/lib/locale/de_LU/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.226978257 s in the future +tar: ./usr/lib/locale/de_LU: time stamp 2026-11-29 07:51:59 is 34039936.22695846 s in the future +tar: ./usr/lib/locale/bho_IN: time stamp 2026-11-29 07:51:59 is 34039936.226937013 s in the future +tar: ./usr/lib/locale/be_BY@latin: time stamp 2026-11-29 07:51:59 is 34039936.226914721 s in the future +tar: ./usr/lib/locale/fr_CA.utf8: time stamp 2026-11-29 07:51:59 is 34039936.226893283 s in the future +tar: ./usr/lib/locale/nso_ZA: time stamp 2026-11-29 07:51:59 is 34039936.226871365 s in the future +tar: ./usr/lib/locale/ar_YE.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.226848904 s in the future +tar: ./usr/lib/locale/ar_YE.utf8: time stamp 2026-11-29 07:51:59 is 34039936.226827886 s in the future +tar: ./usr/lib/locale/hr_HR/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.226806146 s in the future +tar: ./usr/lib/locale/hr_HR: time stamp 2026-11-29 07:51:59 is 34039936.226785212 s in the future +tar: ./usr/lib/locale/ar_LY/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.226763457 s in the future +tar: ./usr/lib/locale/ar_LY: time stamp 2026-11-29 07:51:59 is 34039936.226741373 s in the future +tar: ./usr/lib/locale/gez_ET@abegede/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.226719305 s in the future +tar: ./usr/lib/locale/gez_ET@abegede: time stamp 2026-11-29 07:51:59 is 34039936.226698408 s in the future +tar: ./usr/lib/locale/tt_RU: time stamp 2026-11-29 07:52:00 is 34039937.226675253 s in the future +tar: ./usr/lib/locale/ltg_LV.utf8: time stamp 2026-11-29 07:51:59 is 34039936.226652328 s in the future +tar: ./usr/lib/locale/ca_AD/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.226629737 s in the future +tar: ./usr/lib/locale/ca_AD: time stamp 2026-11-29 07:51:59 is 34039936.22660784 s in the future +tar: ./usr/lib/locale/uz_UZ.utf8: time stamp 2026-11-29 07:52:00 is 34039937.226585175 s in the future +tar: ./usr/lib/locale/br_FR@euro: time stamp 2026-11-29 07:51:59 is 34039936.226562023 s in the future +tar: ./usr/lib/locale/nl_AW/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.226539333 s in the future +tar: ./usr/lib/locale/nl_AW: time stamp 2026-11-29 07:51:59 is 34039936.226517508 s in the future +tar: ./usr/lib/locale/ks_IN: time stamp 2026-11-29 07:51:59 is 34039936.226495472 s in the future +tar: ./usr/lib/locale/en_NG/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.226473296 s in the future +tar: ./usr/lib/locale/en_NG: time stamp 2026-11-29 07:51:59 is 34039936.226452087 s in the future +tar: ./usr/lib/locale/ar_MA/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.226428999 s in the future +tar: ./usr/lib/locale/ar_MA: time stamp 2026-11-29 07:51:59 is 34039936.226406833 s in the future +tar: ./usr/lib/locale/pt_PT@euro: time stamp 2026-11-29 07:52:00 is 34039937.226385135 s in the future +tar: ./usr/lib/locale/is_IS: time stamp 2026-11-29 07:51:59 is 34039936.226362741 s in the future +tar: ./usr/lib/locale/ms_MY.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.226291742 s in the future +tar: ./usr/lib/locale/ms_MY.utf8: time stamp 2026-11-29 07:51:59 is 34039936.226267188 s in the future +tar: ./usr/lib/locale/hne_IN: time stamp 2026-11-29 07:51:59 is 34039936.226244227 s in the future +tar: ./usr/lib/locale/cy_GB.utf8: time stamp 2026-11-29 07:51:59 is 34039936.226221133 s in the future +tar: ./usr/lib/locale/bn_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.226199737 s in the future +tar: ./usr/lib/locale/bn_IN: time stamp 2026-11-29 07:51:59 is 34039936.226178114 s in the future +tar: ./usr/lib/locale/tcy_IN.utf8/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.226155081 s in the future +tar: ./usr/lib/locale/tcy_IN.utf8: time stamp 2026-11-29 07:52:00 is 34039937.226132902 s in the future +tar: ./usr/lib/locale/es_GT.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.226109976 s in the future +tar: ./usr/lib/locale/es_GT.utf8: time stamp 2026-11-29 07:51:59 is 34039936.226087126 s in the future +tar: ./usr/lib/locale/unm_US: time stamp 2026-11-29 07:52:00 is 34039937.226064665 s in the future +tar: ./usr/lib/locale/es_PE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.22602943 s in the future +tar: ./usr/lib/locale/es_PE: time stamp 2026-11-29 07:51:59 is 34039936.226007385 s in the future +tar: ./usr/lib/locale/ve_ZA: time stamp 2026-11-29 07:52:00 is 34039937.225984798 s in the future +tar: ./usr/lib/locale/uk_UA: time stamp 2026-11-29 07:52:00 is 34039937.22596177 s in the future +tar: ./usr/lib/locale/oc_FR: time stamp 2026-11-29 07:52:00 is 34039937.225940372 s in the future +tar: ./usr/lib/locale/tn_ZA: time stamp 2026-11-29 07:52:00 is 34039937.225918196 s in the future +tar: ./usr/lib/locale/pt_PT/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.225896807 s in the future +tar: ./usr/lib/locale/pt_PT: time stamp 2026-11-29 07:52:00 is 34039937.22587481 s in the future +tar: ./usr/lib/locale/nb_NO: time stamp 2026-11-29 07:51:59 is 34039936.22585142 s in the future +tar: ./usr/lib/locale/ar_TN.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.225829451 s in the future +tar: ./usr/lib/locale/ar_TN.utf8: time stamp 2026-11-29 07:51:59 is 34039936.225806789 s in the future +tar: ./usr/lib/locale/en_HK.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.225784961 s in the future +tar: ./usr/lib/locale/en_HK.utf8: time stamp 2026-11-29 07:51:59 is 34039936.225762834 s in the future +tar: ./usr/lib/locale/so_DJ.utf8: time stamp 2026-11-29 07:52:00 is 34039937.225740185 s in the future +tar: ./usr/lib/locale/ms_MY/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.225718772 s in the future +tar: ./usr/lib/locale/ms_MY: time stamp 2026-11-29 07:51:59 is 34039936.225696777 s in the future +tar: ./usr/lib/locale/kl_GL: time stamp 2026-11-29 07:51:59 is 34039936.225676034 s in the future +tar: ./usr/lib/locale/nan_TW@latin: time stamp 2026-11-29 07:51:59 is 34039936.22565242 s in the future +tar: ./usr/lib/locale/shs_CA/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.225629858 s in the future +tar: ./usr/lib/locale/shs_CA: time stamp 2026-11-29 07:52:00 is 34039937.225608067 s in the future +tar: ./usr/lib/locale/bo_CN: time stamp 2026-11-29 07:51:59 is 34039936.225585587 s in the future +tar: ./usr/lib/locale/bs_BA.utf8: time stamp 2026-11-29 07:51:59 is 34039936.225563417 s in the future +tar: ./usr/lib/locale/pa_IN: time stamp 2026-11-29 07:52:00 is 34039937.225540242 s in the future +tar: ./usr/lib/locale/en_PH.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.225518192 s in the future +tar: ./usr/lib/locale/en_PH.utf8: time stamp 2026-11-29 07:51:59 is 34039936.225496276 s in the future +tar: ./usr/lib/locale/ta_LK/LC_MESSAGES: time stamp 2026-11-29 07:52:00 is 34039937.225474553 s in the future +tar: ./usr/lib/locale/ta_LK: time stamp 2026-11-29 07:52:00 is 34039937.225451938 s in the future +tar: ./usr/lib/locale/de_LU.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.22538611 s in the future +tar: ./usr/lib/locale/de_LU.utf8: time stamp 2026-11-29 07:51:59 is 34039936.225356137 s in the future +tar: ./usr/lib/locale/zu_ZA: time stamp 2026-11-29 07:52:00 is 34039937.22533259 s in the future +tar: ./usr/lib/locale/ar_SD.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.225306552 s in the future +tar: ./usr/lib/locale/ar_SD.utf8: time stamp 2026-11-29 07:51:59 is 34039936.225283026 s in the future +tar: ./usr/lib/locale/ar_SY/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.225261597 s in the future +tar: ./usr/lib/locale/ar_SY: time stamp 2026-11-29 07:51:59 is 34039936.225239671 s in the future +tar: ./usr/lib/locale/ar_AE.utf8: time stamp 2026-11-29 07:51:59 is 34039936.225174298 s in the future +tar: ./usr/lib/locale/crh_RU: time stamp 2026-11-29 07:51:59 is 34039936.22514084 s in the future +tar: ./usr/lib/locale/bem_ZM: time stamp 2026-11-29 07:51:59 is 34039936.225117324 s in the future +tar: ./usr/lib/locale/fy_DE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.225094932 s in the future +tar: ./usr/lib/locale/fy_DE: time stamp 2026-11-29 07:51:59 is 34039936.225072528 s in the future +tar: ./usr/lib/locale/kk_KZ.utf8: time stamp 2026-11-29 07:51:59 is 34039936.225051106 s in the future +tar: ./usr/lib/locale/tr_CY.utf8: time stamp 2026-11-29 07:52:00 is 34039937.225029804 s in the future +tar: ./usr/lib/locale/ca_AD.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.224952853 s in the future +tar: ./usr/lib/locale/ca_AD.utf8: time stamp 2026-11-29 07:51:59 is 34039936.224925027 s in the future +tar: ./usr/lib/locale/vi_VN: time stamp 2026-11-29 07:52:00 is 34039937.224901683 s in the future +tar: ./usr/lib/locale/nhn_MX: time stamp 2026-11-29 07:51:59 is 34039936.224840636 s in the future +tar: ./usr/lib/locale/ta_IN: time stamp 2026-11-29 07:52:00 is 34039937.224815779 s in the future +tar: ./usr/lib/locale/eu_ES.utf8: time stamp 2026-11-29 07:51:59 is 34039936.224793635 s in the future +tar: ./usr/lib/locale/tg_TJ.utf8: time stamp 2026-11-29 07:52:00 is 34039937.224771026 s in the future +tar: ./usr/lib/locale/ar_SD/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.224749657 s in the future +tar: ./usr/lib/locale/ar_SD: time stamp 2026-11-29 07:51:59 is 34039936.224728261 s in the future +tar: ./usr/lib/locale/fr_FR.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.224705576 s in the future +tar: ./usr/lib/locale/fr_FR.utf8: time stamp 2026-11-29 07:51:59 is 34039936.224679383 s in the future +tar: ./usr/lib/locale/ar_EG/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.224656038 s in the future +tar: ./usr/lib/locale/ar_EG: time stamp 2026-11-29 07:51:59 is 34039936.22459237 s in the future +tar: ./usr/lib/locale/fil_PH: time stamp 2026-11-29 07:51:59 is 34039936.224568636 s in the future +tar: ./usr/lib/locale/eu_FR@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.224505597 s in the future +tar: ./usr/lib/locale/eu_FR@euro: time stamp 2026-11-29 07:51:59 is 34039936.224480619 s in the future +tar: ./usr/lib/locale/nl_BE.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.22441439 s in the future +tar: ./usr/lib/locale/nl_BE.utf8: time stamp 2026-11-29 07:51:59 is 34039936.224389236 s in the future +tar: ./usr/lib/locale/af_ZA: time stamp 2026-11-29 07:51:59 is 34039936.224326223 s in the future +tar: ./usr/lib/locale/en_SG/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.224261603 s in the future +tar: ./usr/lib/locale/en_SG: time stamp 2026-11-29 07:51:59 is 34039936.224233492 s in the future +tar: ./usr/lib/locale/bg_BG.utf8: time stamp 2026-11-29 07:51:59 is 34039936.224209574 s in the future +tar: ./usr/lib/locale/su_ID: time stamp 2026-11-29 07:52:00 is 34039937.224188816 s in the future +tar: ./usr/lib/locale/en_IL/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.224123776 s in the future +tar: ./usr/lib/locale/en_IL: time stamp 2026-11-29 07:51:59 is 34039936.224100838 s in the future +tar: ./usr/lib/locale/nan_TW: time stamp 2026-11-29 07:51:59 is 34039936.22407795 s in the future +tar: ./usr/lib/locale/ar_IN/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.224055921 s in the future +tar: ./usr/lib/locale/ar_IN: time stamp 2026-11-29 07:51:59 is 34039936.224033664 s in the future +tar: ./usr/lib/locale/wae_CH: time stamp 2026-11-29 07:52:00 is 34039937.224010473 s in the future +tar: ./usr/lib/locale/ber_DZ: time stamp 2026-11-29 07:51:59 is 34039936.223989825 s in the future +tar: ./usr/lib/locale/gv_GB: time stamp 2026-11-29 07:51:59 is 34039936.223967381 s in the future +tar: ./usr/lib/locale/es_BO.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.223944755 s in the future +tar: ./usr/lib/locale/es_BO.utf8: time stamp 2026-11-29 07:51:59 is 34039936.223922171 s in the future +tar: ./usr/lib/locale/niu_NZ/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.223900797 s in the future +tar: ./usr/lib/locale/niu_NZ: time stamp 2026-11-29 07:51:59 is 34039936.223880371 s in the future +tar: ./usr/lib/locale/de_LU@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.22385751 s in the future +tar: ./usr/lib/locale/de_LU@euro: time stamp 2026-11-29 07:51:59 is 34039936.22383793 s in the future +tar: ./usr/lib/locale/en_AG: time stamp 2026-11-29 07:51:59 is 34039936.223798261 s in the future +tar: ./usr/lib/locale/id_ID.utf8: time stamp 2026-11-29 07:51:59 is 34039936.223775816 s in the future +tar: ./usr/lib/locale/zgh_MA: time stamp 2026-11-29 07:52:00 is 34039937.223753482 s in the future +tar: ./usr/lib/locale/nds_NL/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.223731742 s in the future +tar: ./usr/lib/locale/nds_NL: time stamp 2026-11-29 07:51:59 is 34039936.22370995 s in the future +tar: ./usr/lib/locale/ga_IE: time stamp 2026-11-29 07:51:59 is 34039936.223688068 s in the future +tar: ./usr/lib/locale/pap_AW: time stamp 2026-11-29 07:52:00 is 34039937.223666203 s in the future +tar: ./usr/lib/locale/de_BE.utf8/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.223643619 s in the future +tar: ./usr/lib/locale/de_BE.utf8: time stamp 2026-11-29 07:51:59 is 34039936.223621209 s in the future +tar: ./usr/lib/locale/ca_ES@euro/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.223600429 s in the future +tar: ./usr/lib/locale/ca_ES@euro: time stamp 2026-11-29 07:51:59 is 34039936.223578257 s in the future +tar: ./usr/lib/locale/se_NO: time stamp 2026-11-29 07:52:00 is 34039937.223556366 s in the future +tar: ./usr/lib/locale/de_DE/LC_MESSAGES: time stamp 2026-11-29 07:51:59 is 34039936.223533576 s in the future +tar: ./usr/lib/locale/de_DE: time stamp 2026-11-29 07:51:59 is 34039936.223511174 s in the future +tar: ./usr/lib/locale/fr_BE@euro: time stamp 2026-11-29 07:51:59 is 34039936.223486872 s in the future +tar: ./usr/lib/locale/sw_KE: time stamp 2026-11-29 07:52:00 is 34039937.223464441 s in the future +tar: ./usr/lib/locale: time stamp 2026-11-29 07:51:59 is 34039936.223443558 s in the future +tar: ./usr/lib/apt/planners: time stamp 2026-11-30 23:31:29 is 34182706.223420268 s in the future +tar: ./usr/lib/apt: time stamp 2026-11-30 23:31:29 is 34182706.22339764 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl/cross-config-5.40.1: time stamp 2026-11-29 07:49:46 is 34039803.223373808 s in the future +tar: ./usr/lib/x86_64-linux-gnu/perl: time stamp 2026-11-29 07:49:46 is 34039803.223348457 s in the future +tar: ./usr/lib/x86_64-linux-gnu: time stamp 2026-12-02 23:31:26 is 34355503.223325864 s in the future +tar: ./usr/lib: time stamp 2026-11-29 07:50:27 is 34039844.223306281 s in the future +tar: ./usr/lib64: time stamp 2026-11-29 07:46:35 is 34039612.22328483 s in the future +tar: ./usr/include/x86_64-linux-gnux32/asm: time stamp 2026-11-29 07:50:02 is 34039819.223260842 s in the future +tar: ./usr/include/x86_64-linux-gnux32: time stamp 2026-11-29 07:50:01 is 34039818.22323967 s in the future +tar: ./usr/include/riscv64-linux-gnu/asm/vendor: time stamp 2026-11-29 07:50:02 is 34039819.223215571 s in the future +tar: ./usr/include/riscv64-linux-gnu/asm: time stamp 2026-11-29 07:50:02 is 34039819.223192762 s in the future +tar: ./usr/include/riscv64-linux-gnu: time stamp 2026-11-29 07:50:01 is 34039818.223171652 s in the future +tar: ./usr/include/alpha-linux-gnu/asm: time stamp 2026-11-29 07:50:02 is 34039819.223147276 s in the future +tar: ./usr/include/alpha-linux-gnu: time stamp 2026-11-29 07:50:00 is 34039817.223126395 s in the future +tar: ./usr/include/powerpc64-linux-gnu/asm: time stamp 2026-11-29 07:50:02 is 34039819.223103536 s in the future +tar: ./usr/include/powerpc64-linux-gnu: time stamp 2026-11-29 07:50:01 is 34039818.223035105 s in the future +tar: ./usr/include/aarch64-linux-gnu/asm: time stamp 2026-11-29 07:50:02 is 34039819.223008418 s in the future +tar: ./usr/include/aarch64-linux-gnu: time stamp 2026-11-29 07:50:00 is 34039817.222986776 s in the future +tar: ./usr/include/sh4-linux-gnu/asm: time stamp 2026-11-29 07:50:02 is 34039819.222963475 s in the future +tar: ./usr/include/sh4-linux-gnu: time stamp 2026-11-29 07:50:01 is 34039818.222941477 s in the future +tar: ./usr/include/sparc64-linux-gnu/asm: time stamp 2026-11-29 07:50:02 is 34039819.22291825 s in the future +tar: ./usr/include/sparc64-linux-gnu: time stamp 2026-11-29 07:50:01 is 34039818.222897251 s in the future +tar: ./usr/include/powerpc64le-linux-gnu/asm: time stamp 2026-11-29 07:50:02 is 34039819.22287469 s in the future +tar: ./usr/include/powerpc64le-linux-gnu: time stamp 2026-11-29 07:50:01 is 34039818.222853082 s in the future +tar: ./usr/include/mipsel-linux-gnu/asm: time stamp 2026-11-29 07:50:02 is 34039819.222829885 s in the future +tar: ./usr/include/mipsel-linux-gnu: time stamp 2026-11-29 07:50:01 is 34039818.222808787 s in the future +tar: ./usr/include/hppa-linux-gnu/asm: time stamp 2026-11-29 07:50:02 is 34039819.222785959 s in the future +tar: ./usr/include/hppa-linux-gnu: time stamp 2026-11-29 07:50:00 is 34039817.222764006 s in the future +tar: ./usr/include/m68k-linux-gnu/asm: time stamp 2026-11-29 07:50:02 is 34039819.222739792 s in the future +tar: ./usr/include/m68k-linux-gnu: time stamp 2026-11-29 07:50:01 is 34039818.222718338 s in the future +tar: ./usr/include/mips64el-linux-gnuabi64/asm: time stamp 2026-11-29 07:50:02 is 34039819.222695272 s in the future +tar: ./usr/include/mips64el-linux-gnuabi64: time stamp 2026-11-29 07:50:01 is 34039818.222674272 s in the future +tar: ./usr/include/arm-linux-gnueabihf/asm: time stamp 2026-11-29 07:50:02 is 34039819.222650927 s in the future +tar: ./usr/include/arm-linux-gnueabihf: time stamp 2026-11-29 07:50:00 is 34039817.222629685 s in the future +tar: ./usr/include/mipsisa64r6el-linux-gnuabi64/asm: time stamp 2026-11-29 07:50:02 is 34039819.222606043 s in the future +tar: ./usr/include/mipsisa64r6el-linux-gnuabi64: time stamp 2026-11-29 07:50:01 is 34039818.222586375 s in the future +tar: ./usr/include/mips-linux-gnu/asm: time stamp 2026-11-29 07:50:02 is 34039819.222562376 s in the future +tar: ./usr/include/mips-linux-gnu: time stamp 2026-11-29 07:50:01 is 34039818.222540966 s in the future +tar: ./usr/include/mips64-linux-gnuabi64/asm: time stamp 2026-11-29 07:50:02 is 34039819.222516865 s in the future +tar: ./usr/include/mips64-linux-gnuabi64: time stamp 2026-11-29 07:50:01 is 34039818.222494206 s in the future +tar: ./usr/include/arm-linux-gnueabi/asm: time stamp 2026-11-29 07:50:02 is 34039819.222470959 s in the future +tar: ./usr/include/arm-linux-gnueabi: time stamp 2026-11-29 07:50:00 is 34039817.222448918 s in the future +tar: ./usr/include/i386-linux-gnu/asm: time stamp 2026-11-29 07:50:02 is 34039819.222425674 s in the future +tar: ./usr/include/i386-linux-gnu: time stamp 2026-11-29 07:50:00 is 34039817.222403852 s in the future +tar: ./usr/include/x86_64-linux-gnu/asm: time stamp 2026-11-29 07:50:02 is 34039819.222379923 s in the future +tar: ./usr/include/x86_64-linux-gnu: time stamp 2026-11-29 07:50:01 is 34039818.222354851 s in the future +tar: ./usr/include/loongarch64-linux-gnu/asm: time stamp 2026-11-29 07:50:02 is 34039819.222330156 s in the future +tar: ./usr/include/loongarch64-linux-gnu: time stamp 2026-11-29 07:50:01 is 34039818.222307012 s in the future +tar: ./usr/include/s390x-linux-gnu/asm: time stamp 2026-11-29 07:50:02 is 34039819.222244062 s in the future +tar: ./usr/include/s390x-linux-gnu: time stamp 2026-11-29 07:50:01 is 34039818.22221727 s in the future +tar: ./usr/include/powerpc-linux-gnu/asm: time stamp 2026-11-29 07:50:02 is 34039819.222193465 s in the future +tar: ./usr/include/powerpc-linux-gnu: time stamp 2026-11-29 07:50:01 is 34039818.222172103 s in the future +tar: ./usr/include: time stamp 2026-11-29 07:50:05 is 34039822.222151533 s in the future +tar: ./usr/sbin: time stamp 2026-12-02 20:29:29 is 34344586.222128443 s in the future +tar: ./usr/share/aptitude: time stamp 2026-11-29 07:51:39 is 34039916.222104706 s in the future +tar: ./usr/share/doc/aptitude: time stamp 2026-11-29 07:51:41 is 34039918.222080738 s in the future +tar: ./usr/share/doc/cpp-x86-64-linux-gnu: time stamp 2026-11-29 07:47:05 is 34039642.222013707 s in the future +tar: ./usr/share/doc/build-essential: time stamp 2026-11-29 07:46:42 is 34039619.221941576 s in the future +tar: ./usr/share/doc: time stamp 2026-11-29 07:51:59 is 34039936.221915361 s in the future +tar: ./usr/share/locale/ga/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.221889701 s in the future +tar: ./usr/share/locale/ga: time stamp 2026-11-29 07:46:26 is 34039603.221868406 s in the future +tar: ./usr/share/locale/ru/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.221843815 s in the future +tar: ./usr/share/locale/ru: time stamp 2026-11-29 07:46:26 is 34039603.22177154 s in the future +tar: ./usr/share/locale/af/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.221734805 s in the future +tar: ./usr/share/locale/af: time stamp 2026-11-29 07:46:26 is 34039603.221713837 s in the future +tar: ./usr/share/locale/pt_BR/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.221690403 s in the future +tar: ./usr/share/locale/pt_BR: time stamp 2026-11-29 07:46:26 is 34039603.221622865 s in the future +tar: ./usr/share/locale/hr/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.221596466 s in the future +tar: ./usr/share/locale/hr: time stamp 2026-11-29 07:46:26 is 34039603.221575389 s in the future +tar: ./usr/share/locale/vi/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.221512212 s in the future +tar: ./usr/share/locale/vi: time stamp 2026-11-29 07:46:26 is 34039603.221488572 s in the future +tar: ./usr/share/locale/eo/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.221464351 s in the future +tar: ./usr/share/locale/eo: time stamp 2026-11-29 07:46:26 is 34039603.221394088 s in the future +tar: ./usr/share/locale/eu/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.221369524 s in the future +tar: ./usr/share/locale/sl/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.221288197 s in the future +tar: ./usr/share/locale/sl: time stamp 2026-11-29 07:46:26 is 34039603.221217596 s in the future +tar: ./usr/share/locale/el/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.221190956 s in the future +tar: ./usr/share/locale/el: time stamp 2026-11-29 07:46:26 is 34039603.221169731 s in the future +tar: ./usr/share/locale/cs/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.221089668 s in the future +tar: ./usr/share/locale/cs: time stamp 2026-11-29 07:46:26 is 34039603.221060738 s in the future +tar: ./usr/share/locale/da/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.221037142 s in the future +tar: ./usr/share/locale/da: time stamp 2026-11-29 07:46:26 is 34039603.220973171 s in the future +tar: ./usr/share/locale/ms/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220950766 s in the future +tar: ./usr/share/locale/bg/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220911952 s in the future +tar: ./usr/share/locale/bg: time stamp 2026-11-29 07:46:26 is 34039603.220891157 s in the future +tar: ./usr/share/locale/lg/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220868086 s in the future +tar: ./usr/share/locale/it/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220831326 s in the future +tar: ./usr/share/locale/it: time stamp 2026-11-29 07:46:26 is 34039603.220808651 s in the future +tar: ./usr/share/locale/ja/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220785805 s in the future +tar: ./usr/share/locale/ja: time stamp 2026-11-29 07:46:26 is 34039603.220764191 s in the future +tar: ./usr/share/locale/sk/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220742054 s in the future +tar: ./usr/share/locale/sk: time stamp 2026-11-29 07:46:26 is 34039603.220720043 s in the future +tar: ./usr/share/locale/gl/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220698037 s in the future +tar: ./usr/share/locale/gl: time stamp 2026-11-29 07:46:26 is 34039603.220676654 s in the future +tar: ./usr/share/locale/ka/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220653637 s in the future +tar: ./usr/share/locale/ka: time stamp 2026-11-29 07:46:26 is 34039603.220632482 s in the future +tar: ./usr/share/locale/ca/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220610693 s in the future +tar: ./usr/share/locale/ca: time stamp 2026-11-29 07:46:26 is 34039603.220590009 s in the future +tar: ./usr/share/locale/pl/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.2205673 s in the future +tar: ./usr/share/locale/pl: time stamp 2026-11-29 07:46:26 is 34039603.220546863 s in the future +tar: ./usr/share/locale/ta/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220524331 s in the future +tar: ./usr/share/locale/zh_TW/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220487469 s in the future +tar: ./usr/share/locale/zh_TW: time stamp 2026-11-29 07:46:26 is 34039603.220465531 s in the future +tar: ./usr/share/locale/kk/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220442104 s in the future +tar: ./usr/share/locale/ia/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220405387 s in the future +tar: ./usr/share/locale/de/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220367932 s in the future +tar: ./usr/share/locale/de: time stamp 2026-11-29 07:46:26 is 34039603.220342321 s in the future +tar: ./usr/share/locale/ko/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220319499 s in the future +tar: ./usr/share/locale/ko: time stamp 2026-11-29 07:46:26 is 34039603.22029762 s in the future +tar: ./usr/share/locale/ro/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220273649 s in the future +tar: ./usr/share/locale/ro: time stamp 2026-11-29 07:46:26 is 34039603.2202519 s in the future +tar: ./usr/share/locale/lt/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220228621 s in the future +tar: ./usr/share/locale/lt: time stamp 2026-11-29 07:46:26 is 34039603.220207535 s in the future +tar: ./usr/share/locale/es/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220184502 s in the future +tar: ./usr/share/locale/es: time stamp 2026-11-29 07:46:26 is 34039603.220162659 s in the future +tar: ./usr/share/locale/be/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220139944 s in the future +tar: ./usr/share/locale/sv/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220101855 s in the future +tar: ./usr/share/locale/sv: time stamp 2026-11-29 07:46:26 is 34039603.220080528 s in the future +tar: ./usr/share/locale/hu/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220056761 s in the future +tar: ./usr/share/locale/hu: time stamp 2026-11-29 07:46:26 is 34039603.220035485 s in the future +tar: ./usr/share/locale/tr/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.220012328 s in the future +tar: ./usr/share/locale/tr: time stamp 2026-11-29 07:46:26 is 34039603.219991388 s in the future +tar: ./usr/share/locale/nb/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.219966225 s in the future +tar: ./usr/share/locale/nb: time stamp 2026-11-29 07:46:26 is 34039603.219944482 s in the future +tar: ./usr/share/locale/et/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.219920897 s in the future +tar: ./usr/share/locale/et: time stamp 2026-11-29 07:46:26 is 34039603.219847706 s in the future +tar: ./usr/share/locale/zh_CN/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.219809419 s in the future +tar: ./usr/share/locale/zh_CN: time stamp 2026-11-29 07:46:26 is 34039603.219785787 s in the future +tar: ./usr/share/locale/nl/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.219762356 s in the future +tar: ./usr/share/locale/nl: time stamp 2026-11-29 07:46:26 is 34039603.21973806 s in the future +tar: ./usr/share/locale/sr/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.219714462 s in the future +tar: ./usr/share/locale/sr: time stamp 2026-11-29 07:46:26 is 34039603.219693116 s in the future +tar: ./usr/share/locale/pt/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.219667981 s in the future +tar: ./usr/share/locale/pt: time stamp 2026-11-29 07:46:26 is 34039603.219646027 s in the future +tar: ./usr/share/locale/uk/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.219621505 s in the future +tar: ./usr/share/locale/uk: time stamp 2026-11-29 07:46:26 is 34039603.219600244 s in the future +tar: ./usr/share/locale/id/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.219576441 s in the future +tar: ./usr/share/locale/id: time stamp 2026-11-29 07:46:26 is 34039603.219554464 s in the future +tar: ./usr/share/locale/fi/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.219530142 s in the future +tar: ./usr/share/locale/fi: time stamp 2026-11-29 07:46:26 is 34039603.219508567 s in the future +tar: ./usr/share/locale/fr/LC_TIME: time stamp 2026-11-29 07:46:31 is 34039608.219484775 s in the future +tar: ./usr/share/locale/fr: time stamp 2026-11-29 07:46:26 is 34039603.219462693 s in the future +tar: ./usr/share/locale: time stamp 2026-11-29 07:51:38 is 34039915.219441202 s in the future +tar: ./usr/share/man/sl/man1: time stamp 2026-11-29 07:46:39 is 34039616.219416517 s in the future +tar: ./usr/share/man/cs/man8: time stamp 2026-11-29 07:52:01 is 34039938.219375458 s in the future +tar: ./usr/share/man/cs: time stamp 2026-11-29 07:51:34 is 34039911.219353027 s in the future +tar: ./usr/share/man/it/man1: time stamp 2026-11-29 07:51:39 is 34039916.219328977 s in the future +tar: ./usr/share/man/it/man8: time stamp 2026-11-30 23:31:29 is 34182706.219306934 s in the future +tar: ./usr/share/man/it: time stamp 2026-11-29 07:46:27 is 34039604.219284668 s in the future +tar: ./usr/share/man/ja/man1: time stamp 2026-11-29 07:51:34 is 34039911.219261426 s in the future +tar: ./usr/share/man/ja/man8: time stamp 2026-11-30 23:31:29 is 34182706.219238396 s in the future +tar: ./usr/share/man/ja: time stamp 2026-11-29 07:46:27 is 34039604.219217793 s in the future +tar: ./usr/share/man/gl/man8: time stamp 2026-11-29 07:52:01 is 34039938.219193419 s in the future +tar: ./usr/share/man/gl: time stamp 2026-11-29 07:51:39 is 34039916.219171469 s in the future +tar: ./usr/share/man/pl/man1: time stamp 2026-11-29 07:51:34 is 34039911.219146586 s in the future +tar: ./usr/share/man/pl/man8: time stamp 2026-11-30 23:31:29 is 34182706.219122763 s in the future +tar: ./usr/share/man/pl: time stamp 2026-11-29 07:46:27 is 34039604.219101193 s in the future +tar: ./usr/share/man/man1: time stamp 2026-12-02 23:31:24 is 34355501.219022951 s in the future +tar: ./usr/share/man/man8: time stamp 2026-12-02 20:29:29 is 34344586.218988771 s in the future +tar: ./usr/share/man/de/man1: time stamp 2026-11-30 23:31:29 is 34182706.218964969 s in the future +tar: ./usr/share/man/de/man8: time stamp 2026-11-30 23:31:29 is 34182706.218942273 s in the future +tar: ./usr/share/man/de: time stamp 2026-11-29 07:46:41 is 34039618.218920761 s in the future +tar: ./usr/share/man/es/man1: time stamp 2026-11-29 07:52:00 is 34039937.218897533 s in the future +tar: ./usr/share/man/es/man8: time stamp 2026-11-30 23:31:29 is 34182706.218873302 s in the future +tar: ./usr/share/man/es: time stamp 2026-11-29 07:46:27 is 34039604.218852331 s in the future +tar: ./usr/share/man/sv/man1: time stamp 2026-11-29 07:52:00 is 34039937.218827902 s in the future +tar: ./usr/share/man/sv: time stamp 2026-11-29 07:46:47 is 34039624.218804708 s in the future +tar: ./usr/share/man/man7: time stamp 2026-11-30 23:31:29 is 34182706.21872502 s in the future +tar: ./usr/share/man/fi/man8: time stamp 2026-11-29 07:52:01 is 34039938.218694927 s in the future +tar: ./usr/share/man/fi: time stamp 2026-11-29 07:51:39 is 34039916.218632045 s in the future +tar: ./usr/share/man/fr/man1: time stamp 2026-11-30 23:31:29 is 34182706.21860591 s in the future +tar: ./usr/share/man/fr/man8: time stamp 2026-11-30 23:31:29 is 34182706.218581871 s in the future +tar: ./usr/share/man/fr: time stamp 2026-11-29 07:46:41 is 34039618.218560323 s in the future +tar: ./usr/share/man: time stamp 2026-11-29 07:51:39 is 34039916.218539972 s in the future +tar: ./usr/share: time stamp 2026-11-29 07:51:42 is 34039919.218517514 s in the future +tar: ./usr: time stamp 2026-11-29 07:46:28 is 34039605.218497266 s in the future +tar: ./etc/systemd/system/timers.target.wants: time stamp 2026-11-29 07:50:27 is 34039844.218414983 s in the future +tar: ./etc/systemd/system: time stamp 2026-11-29 07:46:29 is 34039606.218386229 s in the future +tar: ./etc/alternatives: time stamp 2026-11-29 07:52:01 is 34039938.218302327 s in the future I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem @@ -24,53 +2566,85 @@ dpkg-source: info: applying 0001-Skip-unreliable-tests.patch I: Not using root during the build. I: Installing the build-deps -I: user script /srv/workspace/pbuilder/228153/tmp/hooks/D02_print_environment starting +I: user script /srv/workspace/pbuilder/1573928/tmp/hooks/D01_modify_environment starting +debug: Running on ionos15-amd64. +I: Changing host+domainname to test build reproducibility +I: Adding a custom variable just for the fun of it... +I: Changing /bin/sh to bash +'/bin/sh' -> '/bin/bash' +lrwxrwxrwx 1 root root 9 Oct 30 18:19 /bin/sh -> /bin/bash +I: Setting pbuilder2's login shell to /bin/bash +I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other +I: user script /srv/workspace/pbuilder/1573928/tmp/hooks/D01_modify_environment finished +I: user script /srv/workspace/pbuilder/1573928/tmp/hooks/D02_print_environment starting I: set - BUILDDIR='/build/reproducible-path' - BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' - BUILDUSERNAME='pbuilder1' - BUILD_ARCH='amd64' - DEBIAN_FRONTEND='noninteractive' - DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=40 ' - DISTRIBUTION='forky' - HOME='/root' - HOST_ARCH='amd64' + BASH=/bin/sh + BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:globskipdots:hostcomplete:interactive_comments:patsub_replacement:progcomp:promptvars:sourcepath + BASH_ALIASES=() + BASH_ARGC=() + BASH_ARGV=() + BASH_CMDS=() + BASH_LINENO=([0]="12" [1]="0") + BASH_LOADABLES_PATH=/usr/local/lib/bash:/usr/lib/bash:/opt/local/lib/bash:/usr/pkg/lib/bash:/opt/pkg/lib/bash:. + BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment") + BASH_VERSINFO=([0]="5" [1]="3" [2]="3" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu") + BASH_VERSION='5.3.3(1)-release' + BUILDDIR=/build/reproducible-path + BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other' + BUILDUSERNAME=pbuilder2 + BUILD_ARCH=amd64 + DEBIAN_FRONTEND=noninteractive + DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=42 ' + DIRSTACK=() + DISTRIBUTION=forky + EUID=0 + FUNCNAME=([0]="Echo" [1]="main") + GROUPS=() + HOME=/root + HOSTNAME=i-capture-the-hostname + HOSTTYPE=x86_64 + HOST_ARCH=amd64 IFS=' ' - INVOCATION_ID='e730915eb9b04aec9d96ad94fe1f8ca9' - LANG='C' - LANGUAGE='en_US:en' - LC_ALL='C' - MAIL='/var/mail/root' - OPTIND='1' - PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' - PBCURRENTCOMMANDLINEOPERATION='build' - PBUILDER_OPERATION='build' - PBUILDER_PKGDATADIR='/usr/share/pbuilder' - PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' - PBUILDER_SYSCONFDIR='/etc' - PPID='228153' - PS1='# ' - PS2='> ' + INVOCATION_ID=b5b9285bd57e462791c64a52c4583aa2 + LANG=C + LANGUAGE=et_EE:et + LC_ALL=C + MACHTYPE=x86_64-pc-linux-gnu + MAIL=/var/mail/root + OPTERR=1 + OPTIND=1 + OSTYPE=linux-gnu + PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path + PBCURRENTCOMMANDLINEOPERATION=build + PBUILDER_OPERATION=build + PBUILDER_PKGDATADIR=/usr/share/pbuilder + PBUILDER_PKGLIBDIR=/usr/lib/pbuilder + PBUILDER_SYSCONFDIR=/etc + PIPESTATUS=([0]="0") + POSIXLY_CORRECT=y + PPID=1573928 PS4='+ ' - PWD='/' - SHELL='/bin/bash' - SHLVL='2' - SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.29Yc7fJR/pbuilderrc_98bK --distribution forky --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/forky-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.29Yc7fJR/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc' - SUDO_GID='110' - SUDO_HOME='/var/lib/jenkins' - SUDO_UID='105' - SUDO_USER='jenkins' - TERM='unknown' - TZ='/usr/share/zoneinfo/Etc/GMT+12' - USER='root' - _='/usr/bin/systemd-run' - http_proxy='http://46.16.76.132:3128' + PWD=/ + SHELL=/bin/bash + SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix + SHLVL=3 + SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.29Yc7fJR/pbuilderrc_X8ss --distribution forky --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/forky-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.29Yc7fJR/b2 --logfile b2/build.log golang-github-hashicorp-raft_1.5.0-1.dsc' + SUDO_GID=111 + SUDO_HOME=/var/lib/jenkins + SUDO_UID=106 + SUDO_USER=jenkins + TERM=unknown + TZ=/usr/share/zoneinfo/Etc/GMT-14 + UID=0 + USER=root + _='I: set' + http_proxy=http://213.165.73.152:3128 I: uname -a - Linux ionos1-amd64 6.12.48+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.48-1 (2025-09-20) x86_64 GNU/Linux + Linux i-capture-the-hostname 6.12.48+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.48-1 (2025-09-20) x86_64 GNU/Linux I: ls -l /bin lrwxrwxrwx 1 root root 7 Aug 10 12:30 /bin -> usr/bin -I: user script /srv/workspace/pbuilder/228153/tmp/hooks/D02_print_environment finished +I: user script /srv/workspace/pbuilder/1573928/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy @@ -226,7 +2800,7 @@ Get: 106 http://deb.debian.org/debian forky/main amd64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-2 [31.2 kB] Get: 107 http://deb.debian.org/debian forky/main amd64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 108 http://deb.debian.org/debian forky/main amd64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] -Fetched 126 MB in 5s (26.9 MB/s) +Fetched 126 MB in 2s (59.4 MB/s) Preconfiguring packages ... Selecting previously unselected package sensible-utils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19869 files and directories currently installed.) @@ -673,7 +3247,11 @@ Building tag database... -> Finished parsing the build-deps I: Building the package -I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.5.0-1_source.changes +I: user script /srv/workspace/pbuilder/1573928/tmp/hooks/A99_set_merged_usr starting +Not re-configuring usrmerge for forky +I: user script /srv/workspace/pbuilder/1573928/tmp/hooks/A99_set_merged_usr finished +hostname: Name or service not known +I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.5.0-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.5.0-1 dpkg-buildpackage: info: source distribution unstable @@ -691,93 +3269,93 @@ dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang - cd obj-x86_64-linux-gnu && go install -trimpath -v -p 40 github.com/hashicorp/raft github.com/hashicorp/raft/bench -encoding + cd obj-x86_64-linux-gnu && go install -trimpath -v -p 42 github.com/hashicorp/raft github.com/hashicorp/raft/bench +internal/unsafeheader +internal/coverage/rtcov internal/goarch -vendor/golang.org/x/crypto/cryptobyte/asn1 -crypto/internal/fips140/alias +internal/goos internal/itoa +internal/godebugs +container/list +vendor/golang.org/x/crypto/internal/alias +internal/msan +vendor/golang.org/x/crypto/cryptobyte/asn1 +log/internal +encoding +internal/asan unicode/utf16 cmp -container/list +internal/profilerecord +unicode/utf8 internal/byteorder -log/internal -internal/coverage/rtcov -vendor/golang.org/x/crypto/internal/alias -sync/atomic internal/nettrace +crypto/internal/fips140/alias math/bits -internal/profilerecord -unicode/utf8 -internal/goexperiment -internal/goos -crypto/internal/boring/sig +internal/cpu internal/runtime/syscall -internal/unsafeheader +internal/goexperiment +sync/atomic unicode -internal/cpu -internal/msan internal/runtime/atomic -internal/godebugs -internal/asan -internal/abi +crypto/internal/boring/sig internal/runtime/math +internal/runtime/sys +internal/abi crypto/internal/fips140/subtle crypto/internal/fips140deps/byteorder internal/chacha8rand -internal/runtime/sys -internal/runtime/exithook crypto/internal/fips140deps/cpu internal/bytealg math +internal/runtime/exithook internal/stringslite internal/race internal/sync internal/runtime/maps runtime -internal/reflectlite iter +weak +internal/reflectlite crypto/subtle sync -weak maps slices errors sort internal/bisect -internal/testlog internal/singleflight +internal/testlog unique internal/oserror -github.com/hashicorp/golang-lru/simplelru io -vendor/golang.org/x/net/dns/dnsmessage path +github.com/hashicorp/golang-lru/simplelru math/rand/v2 strconv +vendor/golang.org/x/net/dns/dnsmessage runtime/cgo syscall -internal/godebug +strings +bytes internal/saferio +internal/godebug hash crypto/internal/randutil -strings -bytes -net/netip -reflect -crypto hash/crc64 hash/crc32 -crypto/internal/fips140deps/godebug -math/rand +crypto +net/netip +reflect vendor/golang.org/x/text/transform -bufio -crypto/internal/fips140 -regexp/syntax crypto/internal/impl net/http/internal/ascii +bufio github.com/hashicorp/go-immutable-radix +regexp/syntax html +crypto/internal/fips140deps/godebug +math/rand +crypto/internal/fips140 crypto/tls/internal/fips140tls crypto/internal/fips140/sha256 crypto/internal/fips140/sha512 @@ -785,15 +3363,15 @@ internal/syscall/execenv internal/syscall/unix time -crypto/internal/fips140/hmac crypto/sha3 +crypto/internal/fips140/hmac crypto/internal/fips140/check crypto/internal/fips140hash crypto/internal/fips140/edwards25519/field -crypto/internal/fips140/bigmod +crypto/internal/fips140/aes crypto/internal/fips140/hkdf crypto/internal/fips140/tls12 -crypto/internal/fips140/aes +crypto/internal/fips140/bigmod crypto/internal/fips140/nistec/fiat crypto/internal/fips140/tls13 regexp @@ -801,8 +3379,8 @@ context io/fs internal/poll -internal/filepathlite crypto/internal/fips140/nistec +internal/filepathlite os internal/fmtsort encoding/binary @@ -811,73 +3389,73 @@ golang.org/x/sys/unix encoding/pem crypto/internal/sysrand -path/filepath +internal/sysinfo os/signal io/ioutil +path/filepath vendor/golang.org/x/sys/cpu -internal/sysinfo fmt -net crypto/internal/entropy crypto/internal/fips140/drbg crypto/internal/fips140only crypto/internal/fips140/ecdh crypto/internal/fips140/ed25519 crypto/internal/fips140/ecdsa +crypto/internal/fips140/mlkem crypto/internal/fips140/aes/gcm crypto/internal/fips140/rsa -crypto/internal/fips140/mlkem crypto/rc4 crypto/md5 crypto/cipher -crypto/internal/boring -crypto/des -vendor/golang.org/x/crypto/chacha20 -log -encoding/hex -net/url -vendor/golang.org/x/net/http2/hpack -net/http/internal -compress/flate runtime/trace +encoding/hex mime/quotedprintable +net/http/internal +log +net/url text/template/parse -go/token flag -mime -encoding/gob +runtime/debug encoding/json +vendor/golang.org/x/net/http2/hpack +go/token +compress/flate math/big +mime vendor/golang.org/x/text/unicode/norm -runtime/debug +encoding/gob +vendor/golang.org/x/crypto/chacha20 +crypto/des +crypto/internal/boring crypto/aes -crypto/sha1 -crypto/sha256 -crypto/ecdh crypto/sha512 +crypto/ecdh crypto/hmac -vendor/golang.org/x/crypto/chacha20poly1305 +crypto/sha256 +crypto/sha1 vendor/golang.org/x/text/unicode/bidi -compress/gzip +vendor/golang.org/x/crypto/chacha20poly1305 testing +compress/gzip vendor/golang.org/x/text/secure/bidirule +net text/template -vendor/golang.org/x/net/idna github.com/mattn/go-isatty +vendor/golang.org/x/net/idna github.com/mattn/go-colorable github.com/fatih/color +github.com/hashicorp/go-hclog crypto/internal/boring/bbig crypto/rand crypto/elliptic -crypto/dsa encoding/asn1 +crypto/dsa crypto/ed25519 crypto/internal/hpke crypto/rsa -github.com/hashicorp/go-hclog html/template -vendor/golang.org/x/crypto/cryptobyte crypto/x509/pkix +vendor/golang.org/x/crypto/cryptobyte crypto/ecdsa vendor/golang.org/x/net/http/httpproxy net/textproto @@ -895,7 +3473,7 @@ debian/rules override_dh_auto_test make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' dh_auto_test - cd obj-x86_64-linux-gnu && go test -vet=off -v -p 40 github.com/hashicorp/raft github.com/hashicorp/raft/bench + cd obj-x86_64-linux-gnu && go test -vet=off -v -p 42 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max @@ -937,33 +3515,33 @@ === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir -2025-10-30T18:17:33.723Z [INFO] creating new snapshot: path=/tmp/raft1301350448/raft1623531546/snapshots/3-10-1761848253723.tmp ---- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s) +2025-10-30T18:21:16.929Z [INFO] creating new snapshot: path=/tmp/raft3606365585/raft1143225429/snapshots/3-10-1761848476929.tmp +--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.42s) === RUN TestFileSS_CreateSnapshot -2025-10-30T18:17:33.727Z [INFO] creating new snapshot: path=/tmp/raft1842263271/snapshots/3-10-1761848253727.tmp -2025-10-30T18:17:33.729Z [WARN] found temporary snapshot: name=3-10-1761848253727.tmp ---- PASS: TestFileSS_CreateSnapshot (0.01s) +2025-10-30T18:21:17.347Z [INFO] creating new snapshot: path=/tmp/raft2332989989/snapshots/3-10-1761848477347.tmp +2025-10-30T18:21:17.617Z [WARN] found temporary snapshot: name=3-10-1761848477347.tmp +--- PASS: TestFileSS_CreateSnapshot (1.06s) === RUN TestFileSS_CancelSnapshot -2025-10-30T18:17:33.736Z [INFO] creating new snapshot: path=/tmp/raft2525840641/snapshots/3-10-1761848253736.tmp ---- PASS: TestFileSS_CancelSnapshot (0.00s) +2025-10-30T18:21:18.405Z [INFO] creating new snapshot: path=/tmp/raft669348248/snapshots/3-10-1761848478405.tmp +--- PASS: TestFileSS_CancelSnapshot (0.01s) === RUN TestFileSS_Retention -2025-10-30T18:17:33.740Z [INFO] creating new snapshot: path=/tmp/raft3199211494/snapshots/3-10-1761848253740.tmp -2025-10-30T18:17:33.747Z [INFO] creating new snapshot: path=/tmp/raft3199211494/snapshots/3-11-1761848253747.tmp -2025-10-30T18:17:33.753Z [INFO] creating new snapshot: path=/tmp/raft3199211494/snapshots/3-12-1761848253753.tmp -2025-10-30T18:17:33.759Z [INFO] reaping snapshot: path=/tmp/raft3199211494/snapshots/3-10-1761848253740 -2025-10-30T18:17:33.759Z [INFO] creating new snapshot: path=/tmp/raft3199211494/snapshots/3-13-1761848253759.tmp -2025-10-30T18:17:33.769Z [INFO] reaping snapshot: path=/tmp/raft3199211494/snapshots/3-11-1761848253747 -2025-10-30T18:17:33.769Z [INFO] creating new snapshot: path=/tmp/raft3199211494/snapshots/3-14-1761848253769.tmp -2025-10-30T18:17:33.774Z [INFO] reaping snapshot: path=/tmp/raft3199211494/snapshots/3-12-1761848253753 ---- PASS: TestFileSS_Retention (0.03s) +2025-10-30T18:21:18.414Z [INFO] creating new snapshot: path=/tmp/raft3271981974/snapshots/3-10-1761848478414.tmp +2025-10-30T18:21:18.427Z [INFO] creating new snapshot: path=/tmp/raft3271981974/snapshots/3-11-1761848478427.tmp +2025-10-30T18:21:18.507Z [INFO] creating new snapshot: path=/tmp/raft3271981974/snapshots/3-12-1761848478507.tmp +2025-10-30T18:21:18.518Z [INFO] reaping snapshot: path=/tmp/raft3271981974/snapshots/3-10-1761848478414 +2025-10-30T18:21:18.518Z [INFO] creating new snapshot: path=/tmp/raft3271981974/snapshots/3-13-1761848478518.tmp +2025-10-30T18:21:18.529Z [INFO] reaping snapshot: path=/tmp/raft3271981974/snapshots/3-11-1761848478427 +2025-10-30T18:21:18.529Z [INFO] creating new snapshot: path=/tmp/raft3271981974/snapshots/3-14-1761848478529.tmp +2025-10-30T18:21:18.539Z [INFO] reaping snapshot: path=/tmp/raft3271981974/snapshots/3-12-1761848478507 +--- PASS: TestFileSS_Retention (0.13s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering -2025-10-30T18:17:33.775Z [INFO] creating new snapshot: path=/tmp/raft1018714603/snapshots/5-130350-1761848253775.tmp -2025-10-30T18:17:33.781Z [INFO] creating new snapshot: path=/tmp/raft1018714603/snapshots/36-204917-1761848253781.tmp ---- PASS: TestFileSS_Ordering (0.01s) +2025-10-30T18:21:18.540Z [INFO] creating new snapshot: path=/tmp/raft32755784/snapshots/5-130350-1761848478540.tmp +2025-10-30T18:21:18.550Z [INFO] creating new snapshot: path=/tmp/raft32755784/snapshots/36-204917-1761848478550.tmp +--- PASS: TestFileSS_Ordering (0.02s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError @@ -1012,12 +3590,12 @@ --- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.02s) === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 -2025-10-30T18:17:33.882Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" +2025-10-30T18:21:18.653Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 +2025-10-30T18:21:18.705Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 -2025-10-30T18:17:33.933Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" -2025-10-30T18:17:33.986Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" +2025-10-30T18:21:18.758Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s) @@ -1034,27 +3612,27 @@ === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff - net_transport_test.go:839: countingWriter: 2025-10-30T18:17:33.992Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2025-10-30T18:21:18.763Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2025-10-30T18:17:33.997Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2025-10-30T18:21:18.769Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2025-10-30T18:17:34.008Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2025-10-30T18:21:18.779Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2025-10-30T18:17:34.029Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2025-10-30T18:21:18.800Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2025-10-30T18:17:34.069Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2025-10-30T18:21:18.840Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2025-10-30T18:17:34.150Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2025-10-30T18:21:18.921Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2025-10-30T18:17:34.311Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2025-10-30T18:21:19.081Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2025-10-30T18:17:34.631Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2025-10-30T18:21:19.402Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2025-10-30T18:17:35.273Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2025-10-30T18:21:20.043Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2025-10-30T18:17:36.275Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2025-10-30T18:21:21.045Z [ERROR] test: failed to accept connection: error="intentional error in test" - net_transport_test.go:839: countingWriter: 2025-10-30T18:17:37.278Z [ERROR] test: failed to accept connection: error="intentional error in test" + net_transport_test.go:839: countingWriter: 2025-10-30T18:21:22.047Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration @@ -1064,2562 +3642,2753 @@ === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop -2025-10-30T18:17:37.995Z [INFO] server-9b6a8f26-9120-f2d4-7a0a-1d887770fcd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b6a8f26-9120-f2d4-7a0a-1d887770fcd1 Address:9b6a8f26-9120-f2d4-7a0a-1d887770fcd1}]" +2025-10-30T18:21:22.766Z [INFO] server-25de5e47-1377-dc6a-8229-d938f76f3a07: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25de5e47-1377-dc6a-8229-d938f76f3a07 Address:25de5e47-1377-dc6a-8229-d938f76f3a07}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown -2025-10-30T18:17:37.996Z [INFO] server-60e5c947-a6a2-144b-240f-a0de50f4e1b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60e5c947-a6a2-144b-240f-a0de50f4e1b9 Address:60e5c947-a6a2-144b-240f-a0de50f4e1b9}]" +2025-10-30T18:21:22.766Z [INFO] server-d2cb449d-e740-0fcd-5590-4189037e0aaf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2cb449d-e740-0fcd-5590-4189037e0aaf Address:d2cb449d-e740-0fcd-5590-4189037e0aaf}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap -2025-10-30T18:17:37.996Z [INFO] server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: initial configuration: index=0 servers=[] -2025-10-30T18:17:37.996Z [INFO] server-b4888378-eb1e-f397-e610-7d6790cf66c5: initial configuration: index=0 servers=[] -2025-10-30T18:17:37.996Z [INFO] server-69fac77c-a8eb-eaae-d882-c1a0fb928fb7: initial configuration: index=0 servers=[] -2025-10-30T18:17:37.996Z [INFO] server-b4888378-eb1e-f397-e610-7d6790cf66c5: entering follower state: follower="Node at b4888378-eb1e-f397-e610-7d6790cf66c5 [Follower]" leader-address= leader-id= -2025-10-30T18:17:37.996Z [INFO] server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: entering follower state: follower="Node at 27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4 [Follower]" leader-address= leader-id= -2025-10-30T18:17:37.996Z [INFO] server-69fac77c-a8eb-eaae-d882-c1a0fb928fb7: entering follower state: follower="Node at 69fac77c-a8eb-eaae-d882-c1a0fb928fb7 [Follower]" leader-address= leader-id= -2025-10-30T18:17:37.996Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:17:38.067Z [WARN] server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:38.067Z [INFO] server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: entering candidate state: node="Node at 27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4 [Candidate]" term=2 -2025-10-30T18:17:38.067Z [INFO] server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: election won: term=2 tally=2 -2025-10-30T18:17:38.067Z [INFO] server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: entering leader state: leader="Node at 27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4 [Leader]" -2025-10-30T18:17:38.067Z [INFO] server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: added peer, starting replication: peer=server-b4888378-eb1e-f397-e610-7d6790cf66c5 -2025-10-30T18:17:38.067Z [INFO] server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: added peer, starting replication: peer=server-69fac77c-a8eb-eaae-d882-c1a0fb928fb7 -2025-10-30T18:17:38.067Z [WARN] server-69fac77c-a8eb-eaae-d882-c1a0fb928fb7: failed to get previous log: previous-index=1 last-index=0 error="log not found" -2025-10-30T18:17:38.067Z [WARN] server-b4888378-eb1e-f397-e610-7d6790cf66c5: failed to get previous log: previous-index=1 last-index=0 error="log not found" -2025-10-30T18:17:38.067Z [WARN] server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: appendEntries rejected, sending older logs: peer="{Voter server-69fac77c-a8eb-eaae-d882-c1a0fb928fb7 69fac77c-a8eb-eaae-d882-c1a0fb928fb7}" next=1 -2025-10-30T18:17:38.067Z [WARN] server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: appendEntries rejected, sending older logs: peer="{Voter server-b4888378-eb1e-f397-e610-7d6790cf66c5 b4888378-eb1e-f397-e610-7d6790cf66c5}" next=1 -2025-10-30T18:17:38.067Z [INFO] server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: pipelining replication: peer="{Voter server-69fac77c-a8eb-eaae-d882-c1a0fb928fb7 69fac77c-a8eb-eaae-d882-c1a0fb928fb7}" -2025-10-30T18:17:38.067Z [INFO] server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: pipelining replication: peer="{Voter server-b4888378-eb1e-f397-e610-7d6790cf66c5 b4888378-eb1e-f397-e610-7d6790cf66c5}" -2025-10-30T18:17:38.172Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:17:38.067494818 +0000 UTC m=+4.346168729 (2 nodes), 70.567714ms from start of poll, 70.914544ms from cluster start. Timeout at 2025-10-30 18:17:38.172497366 +0000 UTC m=+4.451171274, 105.002545ms after stability -2025-10-30T18:17:38.172Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:38.278Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:38.172853242 +0000 UTC m=+4.451527142 (1 nodes), 2.726µs from start of poll, 176.272957ms from cluster start. Timeout at 2025-10-30 18:17:38.277856571 +0000 UTC m=+4.556530469, 105.003327ms after stability -2025-10-30T18:17:38.289Z [INFO] server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: aborting pipeline replication: peer="{Voter server-69fac77c-a8eb-eaae-d882-c1a0fb928fb7 69fac77c-a8eb-eaae-d882-c1a0fb928fb7}" -2025-10-30T18:17:38.289Z [INFO] server-27fc6cb2-ef7b-fa28-1929-7d9f14d66dd4: aborting pipeline replication: peer="{Voter server-b4888378-eb1e-f397-e610-7d6790cf66c5 b4888378-eb1e-f397-e610-7d6790cf66c5}" +2025-10-30T18:21:22.767Z [INFO] server-0e73348d-0805-a8d1-cc3b-96fbf1bea5cb: initial configuration: index=0 servers=[] +2025-10-30T18:21:22.767Z [INFO] server-7bc2384a-0ee6-a925-8173-9efc8b212a7c: initial configuration: index=0 servers=[] +2025-10-30T18:21:22.767Z [INFO] server-0e73348d-0805-a8d1-cc3b-96fbf1bea5cb: entering follower state: follower="Node at 0e73348d-0805-a8d1-cc3b-96fbf1bea5cb [Follower]" leader-address= leader-id= +2025-10-30T18:21:22.767Z [INFO] server-9b7cd2d1-4b4e-e610-4269-1f54e2421d64: initial configuration: index=0 servers=[] +2025-10-30T18:21:22.767Z [INFO] server-7bc2384a-0ee6-a925-8173-9efc8b212a7c: entering follower state: follower="Node at 7bc2384a-0ee6-a925-8173-9efc8b212a7c [Follower]" leader-address= leader-id= +2025-10-30T18:21:22.767Z [INFO] server-9b7cd2d1-4b4e-e610-4269-1f54e2421d64: entering follower state: follower="Node at 9b7cd2d1-4b4e-e610-4269-1f54e2421d64 [Follower]" leader-address= leader-id= +2025-10-30T18:21:22.767Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:21:22.830Z [WARN] server-0e73348d-0805-a8d1-cc3b-96fbf1bea5cb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:22.830Z [INFO] server-0e73348d-0805-a8d1-cc3b-96fbf1bea5cb: entering candidate state: node="Node at 0e73348d-0805-a8d1-cc3b-96fbf1bea5cb [Candidate]" term=2 +2025-10-30T18:21:22.830Z [INFO] server-0e73348d-0805-a8d1-cc3b-96fbf1bea5cb: election won: term=2 tally=2 +2025-10-30T18:21:22.830Z [INFO] server-0e73348d-0805-a8d1-cc3b-96fbf1bea5cb: entering leader state: leader="Node at 0e73348d-0805-a8d1-cc3b-96fbf1bea5cb [Leader]" +2025-10-30T18:21:22.830Z [INFO] server-0e73348d-0805-a8d1-cc3b-96fbf1bea5cb: added peer, starting replication: peer=server-7bc2384a-0ee6-a925-8173-9efc8b212a7c +2025-10-30T18:21:22.830Z [INFO] server-0e73348d-0805-a8d1-cc3b-96fbf1bea5cb: added peer, starting replication: peer=server-9b7cd2d1-4b4e-e610-4269-1f54e2421d64 +2025-10-30T18:21:22.831Z [WARN] server-9b7cd2d1-4b4e-e610-4269-1f54e2421d64: failed to get previous log: previous-index=1 last-index=0 error="log not found" +2025-10-30T18:21:22.831Z [WARN] server-7bc2384a-0ee6-a925-8173-9efc8b212a7c: failed to get previous log: previous-index=1 last-index=0 error="log not found" +2025-10-30T18:21:22.831Z [WARN] server-0e73348d-0805-a8d1-cc3b-96fbf1bea5cb: appendEntries rejected, sending older logs: peer="{Voter server-9b7cd2d1-4b4e-e610-4269-1f54e2421d64 9b7cd2d1-4b4e-e610-4269-1f54e2421d64}" next=1 +2025-10-30T18:21:22.831Z [WARN] server-0e73348d-0805-a8d1-cc3b-96fbf1bea5cb: appendEntries rejected, sending older logs: peer="{Voter server-7bc2384a-0ee6-a925-8173-9efc8b212a7c 7bc2384a-0ee6-a925-8173-9efc8b212a7c}" next=1 +2025-10-30T18:21:22.831Z [INFO] server-0e73348d-0805-a8d1-cc3b-96fbf1bea5cb: pipelining replication: peer="{Voter server-9b7cd2d1-4b4e-e610-4269-1f54e2421d64 9b7cd2d1-4b4e-e610-4269-1f54e2421d64}" +2025-10-30T18:21:22.831Z [INFO] server-0e73348d-0805-a8d1-cc3b-96fbf1bea5cb: pipelining replication: peer="{Voter server-7bc2384a-0ee6-a925-8173-9efc8b212a7c 7bc2384a-0ee6-a925-8173-9efc8b212a7c}" +2025-10-30T18:21:22.936Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:21:22.830947117 +0000 UTC m=+5.904476411 (2 nodes), 63.545101ms from start of poll, 63.870691ms from cluster start. Timeout at 2025-10-30 18:21:22.935948379 +0000 UTC m=+6.009477672, 105.001261ms after stability +2025-10-30T18:21:22.936Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:23.041Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:22.936603002 +0000 UTC m=+6.010132297 (1 nodes), 696ns from start of poll, 169.526577ms from cluster start. Timeout at 2025-10-30 18:21:23.041605012 +0000 UTC m=+6.115134305, 105.002008ms after stability +2025-10-30T18:21:23.058Z [INFO] server-0e73348d-0805-a8d1-cc3b-96fbf1bea5cb: aborting pipeline replication: peer="{Voter server-9b7cd2d1-4b4e-e610-4269-1f54e2421d64 9b7cd2d1-4b4e-e610-4269-1f54e2421d64}" +2025-10-30T18:21:23.058Z [INFO] server-0e73348d-0805-a8d1-cc3b-96fbf1bea5cb: aborting pipeline replication: peer="{Voter server-7bc2384a-0ee6-a925-8173-9efc8b212a7c 7bc2384a-0ee6-a925-8173-9efc8b212a7c}" --- PASS: TestRaft_LiveBootstrap (0.29s) === RUN TestRaft_LiveBootstrap_From_NonVoter -2025-10-30T18:17:38.289Z [INFO] server-3bc2969f-9d21-f095-ac30-7b886e1c9150: initial configuration: index=0 servers=[] -2025-10-30T18:17:38.290Z [INFO] server-3d228883-6ce2-d4ae-29dc-540d24c2e147: initial configuration: index=0 servers=[] -2025-10-30T18:17:38.290Z [INFO] server-3bc2969f-9d21-f095-ac30-7b886e1c9150: entering follower state: follower="Node at 3bc2969f-9d21-f095-ac30-7b886e1c9150 [Follower]" leader-address= leader-id= -2025-10-30T18:17:38.290Z [INFO] server-3d228883-6ce2-d4ae-29dc-540d24c2e147: entering follower state: follower="Node at 3d228883-6ce2-d4ae-29dc-540d24c2e147 [Follower]" leader-address= leader-id= +2025-10-30T18:21:23.058Z [INFO] server-454cd653-8b9d-1845-e846-25f914c25075: initial configuration: index=0 servers=[] +2025-10-30T18:21:23.058Z [INFO] server-ea229c5c-4d02-9f13-c233-f0053f0c487f: initial configuration: index=0 servers=[] +2025-10-30T18:21:23.058Z [INFO] server-454cd653-8b9d-1845-e846-25f914c25075: entering follower state: follower="Node at 454cd653-8b9d-1845-e846-25f914c25075 [Follower]" leader-address= leader-id= +2025-10-30T18:21:23.058Z [INFO] server-ea229c5c-4d02-9f13-c233-f0053f0c487f: entering follower state: follower="Node at ea229c5c-4d02-9f13-c233-f0053f0c487f [Follower]" leader-address= leader-id= --- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s) === RUN TestRaft_RecoverCluster_NoState -2025-10-30T18:17:38.291Z [INFO] server-e32e11de-8c24-8ddd-6f57-c2afa4630df3: initial configuration: index=0 servers=[] -2025-10-30T18:17:38.291Z [INFO] server-e32e11de-8c24-8ddd-6f57-c2afa4630df3: entering follower state: follower="Node at e32e11de-8c24-8ddd-6f57-c2afa4630df3 [Shutdown]" leader-address= leader-id= +2025-10-30T18:21:23.059Z [INFO] server-b455d9f1-1380-cd9e-b84c-db8d228dbfe9: initial configuration: index=0 servers=[] +2025-10-30T18:21:23.059Z [INFO] server-b455d9f1-1380-cd9e-b84c-db8d228dbfe9: entering follower state: follower="Node at b455d9f1-1380-cd9e-b84c-db8d228dbfe9 [Shutdown]" leader-address= leader-id= --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster === RUN TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs -2025-10-30T18:17:38.292Z [INFO] server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f Address:92be0f91-b5b2-75a3-4841-1bc480c6fa3f} {Suffrage:Voter ID:server-d21182ed-9821-e0a5-d00a-fa1fdf90f057 Address:d21182ed-9821-e0a5-d00a-fa1fdf90f057} {Suffrage:Voter ID:server-9401301f-d510-4987-3d54-346790456ef9 Address:9401301f-d510-4987-3d54-346790456ef9}]" -2025-10-30T18:17:38.292Z [INFO] server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f: entering follower state: follower="Node at 92be0f91-b5b2-75a3-4841-1bc480c6fa3f [Follower]" leader-address= leader-id= -2025-10-30T18:17:38.292Z [INFO] server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f Address:92be0f91-b5b2-75a3-4841-1bc480c6fa3f} {Suffrage:Voter ID:server-d21182ed-9821-e0a5-d00a-fa1fdf90f057 Address:d21182ed-9821-e0a5-d00a-fa1fdf90f057} {Suffrage:Voter ID:server-9401301f-d510-4987-3d54-346790456ef9 Address:9401301f-d510-4987-3d54-346790456ef9}]" -2025-10-30T18:17:38.292Z [INFO] server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: entering follower state: follower="Node at d21182ed-9821-e0a5-d00a-fa1fdf90f057 [Follower]" leader-address= leader-id= -2025-10-30T18:17:38.292Z [INFO] server-9401301f-d510-4987-3d54-346790456ef9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f Address:92be0f91-b5b2-75a3-4841-1bc480c6fa3f} {Suffrage:Voter ID:server-d21182ed-9821-e0a5-d00a-fa1fdf90f057 Address:d21182ed-9821-e0a5-d00a-fa1fdf90f057} {Suffrage:Voter ID:server-9401301f-d510-4987-3d54-346790456ef9 Address:9401301f-d510-4987-3d54-346790456ef9}]" -2025-10-30T18:17:38.292Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:38.292Z [INFO] server-9401301f-d510-4987-3d54-346790456ef9: entering follower state: follower="Node at 9401301f-d510-4987-3d54-346790456ef9 [Follower]" leader-address= leader-id= -2025-10-30T18:17:38.355Z [WARN] server-9401301f-d510-4987-3d54-346790456ef9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:38.355Z [INFO] server-9401301f-d510-4987-3d54-346790456ef9: entering candidate state: node="Node at 9401301f-d510-4987-3d54-346790456ef9 [Candidate]" term=2 -2025-10-30T18:17:38.355Z [INFO] server-9401301f-d510-4987-3d54-346790456ef9: election won: term=2 tally=2 -2025-10-30T18:17:38.355Z [INFO] server-9401301f-d510-4987-3d54-346790456ef9: entering leader state: leader="Node at 9401301f-d510-4987-3d54-346790456ef9 [Leader]" -2025-10-30T18:17:38.355Z [INFO] server-9401301f-d510-4987-3d54-346790456ef9: added peer, starting replication: peer=server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f -2025-10-30T18:17:38.355Z [INFO] server-9401301f-d510-4987-3d54-346790456ef9: added peer, starting replication: peer=server-d21182ed-9821-e0a5-d00a-fa1fdf90f057 -2025-10-30T18:17:38.355Z [INFO] server-9401301f-d510-4987-3d54-346790456ef9: pipelining replication: peer="{Voter server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f 92be0f91-b5b2-75a3-4841-1bc480c6fa3f}" -2025-10-30T18:17:38.355Z [INFO] server-9401301f-d510-4987-3d54-346790456ef9: pipelining replication: peer="{Voter server-d21182ed-9821-e0a5-d00a-fa1fdf90f057 d21182ed-9821-e0a5-d00a-fa1fdf90f057}" -2025-10-30T18:17:38.461Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:38.355734303 +0000 UTC m=+4.634408211 (1 nodes), 63.02207ms from start of poll, 63.290007ms from cluster start. Timeout at 2025-10-30 18:17:38.460737941 +0000 UTC m=+4.739411838, 105.003627ms after stability -2025-10-30T18:17:38.461Z [INFO] server-9401301f-d510-4987-3d54-346790456ef9: aborting pipeline replication: peer="{Voter server-d21182ed-9821-e0a5-d00a-fa1fdf90f057 d21182ed-9821-e0a5-d00a-fa1fdf90f057}" -2025-10-30T18:17:38.461Z [INFO] server-9401301f-d510-4987-3d54-346790456ef9: aborting pipeline replication: peer="{Voter server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f 92be0f91-b5b2-75a3-4841-1bc480c6fa3f}" -2025-10-30T18:17:38.461Z [INFO] creating new snapshot: path=/tmp/raft514602561/snapshots/2-2-1761848258461.tmp -2025-10-30T18:17:38.462Z [INFO] server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f: starting restore from snapshot: id=2-2-1761848258461 last-index=2 last-term=2 size-in-bytes=1 -2025-10-30T18:17:38.462Z [INFO] server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f: snapshot restore progress: id=2-2-1761848258461 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% -2025-10-30T18:17:38.462Z [INFO] server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f: restored from snapshot: id=2-2-1761848258461 last-index=2 last-term=2 size-in-bytes=1 -2025-10-30T18:17:38.462Z [INFO] server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f Address:92be0f91-b5b2-75a3-4841-1bc480c6fa3f} {Suffrage:Voter ID:server-d21182ed-9821-e0a5-d00a-fa1fdf90f057 Address:d21182ed-9821-e0a5-d00a-fa1fdf90f057} {Suffrage:Voter ID:server-9401301f-d510-4987-3d54-346790456ef9 Address:9401301f-d510-4987-3d54-346790456ef9}]" -2025-10-30T18:17:38.462Z [INFO] creating new snapshot: path=/tmp/raft2579846394/snapshots/2-2-1761848258462.tmp -2025-10-30T18:17:38.462Z [INFO] server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f: entering follower state: follower="Node at 92be0f91-b5b2-75a3-4841-1bc480c6fa3f [Follower]" leader-address= leader-id= -2025-10-30T18:17:38.462Z [INFO] server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: starting restore from snapshot: id=2-2-1761848258462 last-index=2 last-term=2 size-in-bytes=1 -2025-10-30T18:17:38.463Z [INFO] server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: snapshot restore progress: id=2-2-1761848258462 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% -2025-10-30T18:17:38.463Z [INFO] server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: restored from snapshot: id=2-2-1761848258462 last-index=2 last-term=2 size-in-bytes=1 -2025-10-30T18:17:38.463Z [INFO] server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f Address:92be0f91-b5b2-75a3-4841-1bc480c6fa3f} {Suffrage:Voter ID:server-d21182ed-9821-e0a5-d00a-fa1fdf90f057 Address:d21182ed-9821-e0a5-d00a-fa1fdf90f057} {Suffrage:Voter ID:server-9401301f-d510-4987-3d54-346790456ef9 Address:9401301f-d510-4987-3d54-346790456ef9}]" -2025-10-30T18:17:38.463Z [INFO] creating new snapshot: path=/tmp/raft3251839670/snapshots/2-2-1761848258463.tmp -2025-10-30T18:17:38.463Z [INFO] server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: entering follower state: follower="Node at d21182ed-9821-e0a5-d00a-fa1fdf90f057 [Follower]" leader-address= leader-id= -2025-10-30T18:17:38.463Z [INFO] server-9401301f-d510-4987-3d54-346790456ef9: starting restore from snapshot: id=2-2-1761848258463 last-index=2 last-term=2 size-in-bytes=1 -2025-10-30T18:17:38.464Z [INFO] server-9401301f-d510-4987-3d54-346790456ef9: snapshot restore progress: id=2-2-1761848258463 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% -2025-10-30T18:17:38.464Z [INFO] server-9401301f-d510-4987-3d54-346790456ef9: restored from snapshot: id=2-2-1761848258463 last-index=2 last-term=2 size-in-bytes=1 -2025-10-30T18:17:38.464Z [INFO] server-9401301f-d510-4987-3d54-346790456ef9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f Address:92be0f91-b5b2-75a3-4841-1bc480c6fa3f} {Suffrage:Voter ID:server-d21182ed-9821-e0a5-d00a-fa1fdf90f057 Address:d21182ed-9821-e0a5-d00a-fa1fdf90f057} {Suffrage:Voter ID:server-9401301f-d510-4987-3d54-346790456ef9 Address:9401301f-d510-4987-3d54-346790456ef9}]" -2025-10-30T18:17:38.464Z [INFO] server-9401301f-d510-4987-3d54-346790456ef9: entering follower state: follower="Node at 9401301f-d510-4987-3d54-346790456ef9 [Follower]" leader-address= leader-id= -2025-10-30T18:17:38.530Z [WARN] server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:38.530Z [INFO] server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: entering candidate state: node="Node at d21182ed-9821-e0a5-d00a-fa1fdf90f057 [Candidate]" term=3 -2025-10-30T18:17:38.530Z [INFO] server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: election won: term=3 tally=2 -2025-10-30T18:17:38.530Z [INFO] server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: entering leader state: leader="Node at d21182ed-9821-e0a5-d00a-fa1fdf90f057 [Leader]" -2025-10-30T18:17:38.530Z [INFO] server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: added peer, starting replication: peer=server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f -2025-10-30T18:17:38.530Z [INFO] server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: added peer, starting replication: peer=server-9401301f-d510-4987-3d54-346790456ef9 -2025-10-30T18:17:38.531Z [INFO] server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: pipelining replication: peer="{Voter server-9401301f-d510-4987-3d54-346790456ef9 9401301f-d510-4987-3d54-346790456ef9}" -2025-10-30T18:17:38.531Z [INFO] server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: pipelining replication: peer="{Voter server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f 92be0f91-b5b2-75a3-4841-1bc480c6fa3f}" -2025-10-30T18:17:38.779Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:38.885Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:38.779311619 +0000 UTC m=+5.057985519 (1 nodes), 23.978µs from start of poll, 486.867315ms from cluster start. Timeout at 2025-10-30 18:17:38.884314389 +0000 UTC m=+5.162988287, 105.002768ms after stability -2025-10-30T18:17:38.885Z [INFO] server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: aborting pipeline replication: peer="{Voter server-9401301f-d510-4987-3d54-346790456ef9 9401301f-d510-4987-3d54-346790456ef9}" -2025-10-30T18:17:38.886Z [INFO] server-d21182ed-9821-e0a5-d00a-fa1fdf90f057: aborting pipeline replication: peer="{Voter server-92be0f91-b5b2-75a3-4841-1bc480c6fa3f 92be0f91-b5b2-75a3-4841-1bc480c6fa3f}" +2025-10-30T18:21:23.060Z [INFO] server-407305c6-1fa4-5cd1-58c5-713512628787: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-407305c6-1fa4-5cd1-58c5-713512628787 Address:407305c6-1fa4-5cd1-58c5-713512628787} {Suffrage:Voter ID:server-5776c553-0d51-d34f-bd20-075166790ece Address:5776c553-0d51-d34f-bd20-075166790ece} {Suffrage:Voter ID:server-547af5f2-a1af-3c7b-2dc3-593a057426de Address:547af5f2-a1af-3c7b-2dc3-593a057426de}]" +2025-10-30T18:21:23.060Z [INFO] server-407305c6-1fa4-5cd1-58c5-713512628787: entering follower state: follower="Node at 407305c6-1fa4-5cd1-58c5-713512628787 [Follower]" leader-address= leader-id= +2025-10-30T18:21:23.060Z [INFO] server-5776c553-0d51-d34f-bd20-075166790ece: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-407305c6-1fa4-5cd1-58c5-713512628787 Address:407305c6-1fa4-5cd1-58c5-713512628787} {Suffrage:Voter ID:server-5776c553-0d51-d34f-bd20-075166790ece Address:5776c553-0d51-d34f-bd20-075166790ece} {Suffrage:Voter ID:server-547af5f2-a1af-3c7b-2dc3-593a057426de Address:547af5f2-a1af-3c7b-2dc3-593a057426de}]" +2025-10-30T18:21:23.060Z [INFO] server-5776c553-0d51-d34f-bd20-075166790ece: entering follower state: follower="Node at 5776c553-0d51-d34f-bd20-075166790ece [Follower]" leader-address= leader-id= +2025-10-30T18:21:23.060Z [INFO] server-547af5f2-a1af-3c7b-2dc3-593a057426de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-407305c6-1fa4-5cd1-58c5-713512628787 Address:407305c6-1fa4-5cd1-58c5-713512628787} {Suffrage:Voter ID:server-5776c553-0d51-d34f-bd20-075166790ece Address:5776c553-0d51-d34f-bd20-075166790ece} {Suffrage:Voter ID:server-547af5f2-a1af-3c7b-2dc3-593a057426de Address:547af5f2-a1af-3c7b-2dc3-593a057426de}]" +2025-10-30T18:21:23.060Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:23.060Z [INFO] server-547af5f2-a1af-3c7b-2dc3-593a057426de: entering follower state: follower="Node at 547af5f2-a1af-3c7b-2dc3-593a057426de [Follower]" leader-address= leader-id= +2025-10-30T18:21:23.124Z [WARN] server-547af5f2-a1af-3c7b-2dc3-593a057426de: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:23.124Z [INFO] server-547af5f2-a1af-3c7b-2dc3-593a057426de: entering candidate state: node="Node at 547af5f2-a1af-3c7b-2dc3-593a057426de [Candidate]" term=2 +2025-10-30T18:21:23.124Z [INFO] server-547af5f2-a1af-3c7b-2dc3-593a057426de: election won: term=2 tally=2 +2025-10-30T18:21:23.124Z [INFO] server-547af5f2-a1af-3c7b-2dc3-593a057426de: entering leader state: leader="Node at 547af5f2-a1af-3c7b-2dc3-593a057426de [Leader]" +2025-10-30T18:21:23.124Z [INFO] server-547af5f2-a1af-3c7b-2dc3-593a057426de: added peer, starting replication: peer=server-407305c6-1fa4-5cd1-58c5-713512628787 +2025-10-30T18:21:23.124Z [INFO] server-547af5f2-a1af-3c7b-2dc3-593a057426de: added peer, starting replication: peer=server-5776c553-0d51-d34f-bd20-075166790ece +2025-10-30T18:21:23.124Z [INFO] server-547af5f2-a1af-3c7b-2dc3-593a057426de: pipelining replication: peer="{Voter server-5776c553-0d51-d34f-bd20-075166790ece 5776c553-0d51-d34f-bd20-075166790ece}" +2025-10-30T18:21:23.124Z [INFO] server-547af5f2-a1af-3c7b-2dc3-593a057426de: pipelining replication: peer="{Voter server-407305c6-1fa4-5cd1-58c5-713512628787 407305c6-1fa4-5cd1-58c5-713512628787}" +2025-10-30T18:21:23.229Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:23.124371446 +0000 UTC m=+6.197900741 (1 nodes), 63.494027ms from start of poll, 63.871572ms from cluster start. Timeout at 2025-10-30 18:21:23.229375596 +0000 UTC m=+6.302904888, 105.004147ms after stability +2025-10-30T18:21:23.229Z [INFO] server-547af5f2-a1af-3c7b-2dc3-593a057426de: aborting pipeline replication: peer="{Voter server-5776c553-0d51-d34f-bd20-075166790ece 5776c553-0d51-d34f-bd20-075166790ece}" +2025-10-30T18:21:23.229Z [INFO] server-547af5f2-a1af-3c7b-2dc3-593a057426de: aborting pipeline replication: peer="{Voter server-407305c6-1fa4-5cd1-58c5-713512628787 407305c6-1fa4-5cd1-58c5-713512628787}" +2025-10-30T18:21:23.229Z [INFO] creating new snapshot: path=/tmp/raft356390343/snapshots/2-2-1761848483229.tmp +2025-10-30T18:21:23.234Z [INFO] server-407305c6-1fa4-5cd1-58c5-713512628787: starting restore from snapshot: id=2-2-1761848483229 last-index=2 last-term=2 size-in-bytes=1 +2025-10-30T18:21:23.234Z [INFO] server-407305c6-1fa4-5cd1-58c5-713512628787: snapshot restore progress: id=2-2-1761848483229 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% +2025-10-30T18:21:23.234Z [INFO] server-407305c6-1fa4-5cd1-58c5-713512628787: restored from snapshot: id=2-2-1761848483229 last-index=2 last-term=2 size-in-bytes=1 +2025-10-30T18:21:23.234Z [INFO] server-407305c6-1fa4-5cd1-58c5-713512628787: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-407305c6-1fa4-5cd1-58c5-713512628787 Address:407305c6-1fa4-5cd1-58c5-713512628787} {Suffrage:Voter ID:server-5776c553-0d51-d34f-bd20-075166790ece Address:5776c553-0d51-d34f-bd20-075166790ece} {Suffrage:Voter ID:server-547af5f2-a1af-3c7b-2dc3-593a057426de Address:547af5f2-a1af-3c7b-2dc3-593a057426de}]" +2025-10-30T18:21:23.234Z [INFO] server-407305c6-1fa4-5cd1-58c5-713512628787: entering follower state: follower="Node at 407305c6-1fa4-5cd1-58c5-713512628787 [Follower]" leader-address= leader-id= +2025-10-30T18:21:23.234Z [INFO] creating new snapshot: path=/tmp/raft3805134209/snapshots/2-2-1761848483234.tmp +2025-10-30T18:21:23.234Z [INFO] server-5776c553-0d51-d34f-bd20-075166790ece: starting restore from snapshot: id=2-2-1761848483234 last-index=2 last-term=2 size-in-bytes=1 +2025-10-30T18:21:23.235Z [INFO] server-5776c553-0d51-d34f-bd20-075166790ece: snapshot restore progress: id=2-2-1761848483234 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% +2025-10-30T18:21:23.235Z [INFO] server-5776c553-0d51-d34f-bd20-075166790ece: restored from snapshot: id=2-2-1761848483234 last-index=2 last-term=2 size-in-bytes=1 +2025-10-30T18:21:23.235Z [INFO] server-5776c553-0d51-d34f-bd20-075166790ece: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-407305c6-1fa4-5cd1-58c5-713512628787 Address:407305c6-1fa4-5cd1-58c5-713512628787} {Suffrage:Voter ID:server-5776c553-0d51-d34f-bd20-075166790ece Address:5776c553-0d51-d34f-bd20-075166790ece} {Suffrage:Voter ID:server-547af5f2-a1af-3c7b-2dc3-593a057426de Address:547af5f2-a1af-3c7b-2dc3-593a057426de}]" +2025-10-30T18:21:23.235Z [INFO] creating new snapshot: path=/tmp/raft2508247057/snapshots/2-2-1761848483235.tmp +2025-10-30T18:21:23.235Z [INFO] server-5776c553-0d51-d34f-bd20-075166790ece: entering follower state: follower="Node at 5776c553-0d51-d34f-bd20-075166790ece [Follower]" leader-address= leader-id= +2025-10-30T18:21:23.306Z [WARN] server-407305c6-1fa4-5cd1-58c5-713512628787: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:23.306Z [INFO] server-407305c6-1fa4-5cd1-58c5-713512628787: entering candidate state: node="Node at 407305c6-1fa4-5cd1-58c5-713512628787 [Candidate]" term=3 +2025-10-30T18:21:23.306Z [ERROR] server-407305c6-1fa4-5cd1-58c5-713512628787: failed to make requestVote RPC: target="{Voter server-547af5f2-a1af-3c7b-2dc3-593a057426de 547af5f2-a1af-3c7b-2dc3-593a057426de}" error="failed to connect to peer: 547af5f2-a1af-3c7b-2dc3-593a057426de" term=3 +2025-10-30T18:21:23.306Z [ERROR] server-407305c6-1fa4-5cd1-58c5-713512628787: failed to make requestVote RPC: target="{Voter server-5776c553-0d51-d34f-bd20-075166790ece 5776c553-0d51-d34f-bd20-075166790ece}" error="failed to connect to peer: 5776c553-0d51-d34f-bd20-075166790ece" term=3 +2025-10-30T18:21:23.315Z [WARN] server-5776c553-0d51-d34f-bd20-075166790ece: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:23.315Z [INFO] server-5776c553-0d51-d34f-bd20-075166790ece: entering candidate state: node="Node at 5776c553-0d51-d34f-bd20-075166790ece [Candidate]" term=3 +2025-10-30T18:21:23.316Z [ERROR] server-5776c553-0d51-d34f-bd20-075166790ece: failed to make requestVote RPC: target="{Voter server-547af5f2-a1af-3c7b-2dc3-593a057426de 547af5f2-a1af-3c7b-2dc3-593a057426de}" error="failed to connect to peer: 547af5f2-a1af-3c7b-2dc3-593a057426de" term=3 +2025-10-30T18:21:23.316Z [ERROR] server-5776c553-0d51-d34f-bd20-075166790ece: failed to make requestVote RPC: target="{Voter server-407305c6-1fa4-5cd1-58c5-713512628787 407305c6-1fa4-5cd1-58c5-713512628787}" error="failed to connect to peer: 407305c6-1fa4-5cd1-58c5-713512628787" term=3 +2025-10-30T18:21:23.328Z [INFO] server-547af5f2-a1af-3c7b-2dc3-593a057426de: starting restore from snapshot: id=2-2-1761848483235 last-index=2 last-term=2 size-in-bytes=1 +2025-10-30T18:21:23.329Z [INFO] server-547af5f2-a1af-3c7b-2dc3-593a057426de: snapshot restore progress: id=2-2-1761848483235 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% +2025-10-30T18:21:23.329Z [INFO] server-547af5f2-a1af-3c7b-2dc3-593a057426de: restored from snapshot: id=2-2-1761848483235 last-index=2 last-term=2 size-in-bytes=1 +2025-10-30T18:21:23.329Z [INFO] server-547af5f2-a1af-3c7b-2dc3-593a057426de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-407305c6-1fa4-5cd1-58c5-713512628787 Address:407305c6-1fa4-5cd1-58c5-713512628787} {Suffrage:Voter ID:server-5776c553-0d51-d34f-bd20-075166790ece Address:5776c553-0d51-d34f-bd20-075166790ece} {Suffrage:Voter ID:server-547af5f2-a1af-3c7b-2dc3-593a057426de Address:547af5f2-a1af-3c7b-2dc3-593a057426de}]" +2025-10-30T18:21:23.329Z [INFO] server-547af5f2-a1af-3c7b-2dc3-593a057426de: entering follower state: follower="Node at 547af5f2-a1af-3c7b-2dc3-593a057426de [Follower]" leader-address= leader-id= +2025-10-30T18:21:23.381Z [WARN] server-5776c553-0d51-d34f-bd20-075166790ece: Election timeout reached, restarting election +2025-10-30T18:21:23.381Z [INFO] server-5776c553-0d51-d34f-bd20-075166790ece: entering candidate state: node="Node at 5776c553-0d51-d34f-bd20-075166790ece [Candidate]" term=4 +2025-10-30T18:21:23.381Z [INFO] server-5776c553-0d51-d34f-bd20-075166790ece: election won: term=4 tally=2 +2025-10-30T18:21:23.381Z [INFO] server-5776c553-0d51-d34f-bd20-075166790ece: entering leader state: leader="Node at 5776c553-0d51-d34f-bd20-075166790ece [Leader]" +2025-10-30T18:21:23.381Z [INFO] server-5776c553-0d51-d34f-bd20-075166790ece: added peer, starting replication: peer=server-407305c6-1fa4-5cd1-58c5-713512628787 +2025-10-30T18:21:23.381Z [INFO] server-5776c553-0d51-d34f-bd20-075166790ece: added peer, starting replication: peer=server-547af5f2-a1af-3c7b-2dc3-593a057426de +2025-10-30T18:21:23.381Z [INFO] server-407305c6-1fa4-5cd1-58c5-713512628787: entering follower state: follower="Node at 407305c6-1fa4-5cd1-58c5-713512628787 [Follower]" leader-address= leader-id= +2025-10-30T18:21:23.381Z [INFO] server-5776c553-0d51-d34f-bd20-075166790ece: pipelining replication: peer="{Voter server-407305c6-1fa4-5cd1-58c5-713512628787 407305c6-1fa4-5cd1-58c5-713512628787}" +2025-10-30T18:21:23.381Z [INFO] server-5776c553-0d51-d34f-bd20-075166790ece: pipelining replication: peer="{Voter server-547af5f2-a1af-3c7b-2dc3-593a057426de 547af5f2-a1af-3c7b-2dc3-593a057426de}" +2025-10-30T18:21:23.644Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:23.749Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:23.644283314 +0000 UTC m=+6.717812604 (1 nodes), 41.965µs from start of poll, 583.783435ms from cluster start. Timeout at 2025-10-30 18:21:23.749289647 +0000 UTC m=+6.822818939, 105.006335ms after stability +2025-10-30T18:21:23.749Z [INFO] server-5776c553-0d51-d34f-bd20-075166790ece: aborting pipeline replication: peer="{Voter server-547af5f2-a1af-3c7b-2dc3-593a057426de 547af5f2-a1af-3c7b-2dc3-593a057426de}" +2025-10-30T18:21:23.749Z [INFO] server-5776c553-0d51-d34f-bd20-075166790ece: aborting pipeline replication: peer="{Voter server-407305c6-1fa4-5cd1-58c5-713512628787 407305c6-1fa4-5cd1-58c5-713512628787}" === RUN TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs -2025-10-30T18:17:38.886Z [INFO] server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8b39909-a245-775b-ac90-3f65d0dc0b2a Address:c8b39909-a245-775b-ac90-3f65d0dc0b2a} {Suffrage:Voter ID:server-4cc2270b-0977-d62a-f36a-642c6b5a18b2 Address:4cc2270b-0977-d62a-f36a-642c6b5a18b2} {Suffrage:Voter ID:server-5c971473-666b-247a-7af5-7befb1d9604b Address:5c971473-666b-247a-7af5-7befb1d9604b}]" -2025-10-30T18:17:38.887Z [INFO] server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8b39909-a245-775b-ac90-3f65d0dc0b2a Address:c8b39909-a245-775b-ac90-3f65d0dc0b2a} {Suffrage:Voter ID:server-4cc2270b-0977-d62a-f36a-642c6b5a18b2 Address:4cc2270b-0977-d62a-f36a-642c6b5a18b2} {Suffrage:Voter ID:server-5c971473-666b-247a-7af5-7befb1d9604b Address:5c971473-666b-247a-7af5-7befb1d9604b}]" -2025-10-30T18:17:38.887Z [INFO] server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: entering follower state: follower="Node at c8b39909-a245-775b-ac90-3f65d0dc0b2a [Follower]" leader-address= leader-id= -2025-10-30T18:17:38.887Z [INFO] server-5c971473-666b-247a-7af5-7befb1d9604b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8b39909-a245-775b-ac90-3f65d0dc0b2a Address:c8b39909-a245-775b-ac90-3f65d0dc0b2a} {Suffrage:Voter ID:server-4cc2270b-0977-d62a-f36a-642c6b5a18b2 Address:4cc2270b-0977-d62a-f36a-642c6b5a18b2} {Suffrage:Voter ID:server-5c971473-666b-247a-7af5-7befb1d9604b Address:5c971473-666b-247a-7af5-7befb1d9604b}]" -2025-10-30T18:17:38.887Z [INFO] server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: entering follower state: follower="Node at 4cc2270b-0977-d62a-f36a-642c6b5a18b2 [Follower]" leader-address= leader-id= -2025-10-30T18:17:38.887Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:38.887Z [INFO] server-5c971473-666b-247a-7af5-7befb1d9604b: entering follower state: follower="Node at 5c971473-666b-247a-7af5-7befb1d9604b [Follower]" leader-address= leader-id= -2025-10-30T18:17:38.938Z [WARN] server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:38.938Z [INFO] server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: entering candidate state: node="Node at 4cc2270b-0977-d62a-f36a-642c6b5a18b2 [Candidate]" term=2 -2025-10-30T18:17:38.938Z [INFO] server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: election won: term=2 tally=2 -2025-10-30T18:17:38.938Z [INFO] server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: entering leader state: leader="Node at 4cc2270b-0977-d62a-f36a-642c6b5a18b2 [Leader]" -2025-10-30T18:17:38.938Z [INFO] server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: added peer, starting replication: peer=server-c8b39909-a245-775b-ac90-3f65d0dc0b2a -2025-10-30T18:17:38.938Z [INFO] server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: added peer, starting replication: peer=server-5c971473-666b-247a-7af5-7befb1d9604b -2025-10-30T18:17:38.939Z [INFO] server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: pipelining replication: peer="{Voter server-c8b39909-a245-775b-ac90-3f65d0dc0b2a c8b39909-a245-775b-ac90-3f65d0dc0b2a}" -2025-10-30T18:17:38.939Z [INFO] server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: pipelining replication: peer="{Voter server-5c971473-666b-247a-7af5-7befb1d9604b 5c971473-666b-247a-7af5-7befb1d9604b}" -2025-10-30T18:17:39.044Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:38.938953002 +0000 UTC m=+5.217626900 (1 nodes), 51.809402ms from start of poll, 52.0603ms from cluster start. Timeout at 2025-10-30 18:17:39.043954359 +0000 UTC m=+5.322628260, 105.00136ms after stability -2025-10-30T18:17:39.044Z [INFO] server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: aborting pipeline replication: peer="{Voter server-5c971473-666b-247a-7af5-7befb1d9604b 5c971473-666b-247a-7af5-7befb1d9604b}" -2025-10-30T18:17:39.044Z [INFO] server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: aborting pipeline replication: peer="{Voter server-c8b39909-a245-775b-ac90-3f65d0dc0b2a c8b39909-a245-775b-ac90-3f65d0dc0b2a}" -2025-10-30T18:17:39.044Z [INFO] creating new snapshot: path=/tmp/raft4199091538/snapshots/2-6-1761848259044.tmp -2025-10-30T18:17:39.044Z [INFO] server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: starting restore from snapshot: id=2-6-1761848259044 last-index=6 last-term=2 size-in-bytes=25 -2025-10-30T18:17:39.045Z [INFO] server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: snapshot restore progress: id=2-6-1761848259044 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% -2025-10-30T18:17:39.045Z [INFO] server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: restored from snapshot: id=2-6-1761848259044 last-index=6 last-term=2 size-in-bytes=25 -2025-10-30T18:17:39.045Z [INFO] server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8b39909-a245-775b-ac90-3f65d0dc0b2a Address:c8b39909-a245-775b-ac90-3f65d0dc0b2a} {Suffrage:Voter ID:server-4cc2270b-0977-d62a-f36a-642c6b5a18b2 Address:4cc2270b-0977-d62a-f36a-642c6b5a18b2} {Suffrage:Voter ID:server-5c971473-666b-247a-7af5-7befb1d9604b Address:5c971473-666b-247a-7af5-7befb1d9604b}]" -2025-10-30T18:17:39.045Z [INFO] server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: entering follower state: follower="Node at c8b39909-a245-775b-ac90-3f65d0dc0b2a [Follower]" leader-address= leader-id= -2025-10-30T18:17:39.045Z [INFO] creating new snapshot: path=/tmp/raft850242973/snapshots/2-6-1761848259045.tmp -2025-10-30T18:17:39.045Z [INFO] server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: starting restore from snapshot: id=2-6-1761848259045 last-index=6 last-term=2 size-in-bytes=25 -2025-10-30T18:17:39.045Z [INFO] server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: snapshot restore progress: id=2-6-1761848259045 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% -2025-10-30T18:17:39.045Z [INFO] server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: restored from snapshot: id=2-6-1761848259045 last-index=6 last-term=2 size-in-bytes=25 -2025-10-30T18:17:39.045Z [INFO] server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8b39909-a245-775b-ac90-3f65d0dc0b2a Address:c8b39909-a245-775b-ac90-3f65d0dc0b2a} {Suffrage:Voter ID:server-4cc2270b-0977-d62a-f36a-642c6b5a18b2 Address:4cc2270b-0977-d62a-f36a-642c6b5a18b2} {Suffrage:Voter ID:server-5c971473-666b-247a-7af5-7befb1d9604b Address:5c971473-666b-247a-7af5-7befb1d9604b}]" -2025-10-30T18:17:39.045Z [INFO] creating new snapshot: path=/tmp/raft2530727206/snapshots/2-6-1761848259045.tmp -2025-10-30T18:17:39.045Z [INFO] server-4cc2270b-0977-d62a-f36a-642c6b5a18b2: entering follower state: follower="Node at 4cc2270b-0977-d62a-f36a-642c6b5a18b2 [Follower]" leader-address= leader-id= -2025-10-30T18:17:39.046Z [INFO] server-5c971473-666b-247a-7af5-7befb1d9604b: starting restore from snapshot: id=2-6-1761848259045 last-index=6 last-term=2 size-in-bytes=25 -2025-10-30T18:17:39.046Z [INFO] server-5c971473-666b-247a-7af5-7befb1d9604b: snapshot restore progress: id=2-6-1761848259045 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% -2025-10-30T18:17:39.046Z [INFO] server-5c971473-666b-247a-7af5-7befb1d9604b: restored from snapshot: id=2-6-1761848259045 last-index=6 last-term=2 size-in-bytes=25 -2025-10-30T18:17:39.046Z [INFO] server-5c971473-666b-247a-7af5-7befb1d9604b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8b39909-a245-775b-ac90-3f65d0dc0b2a Address:c8b39909-a245-775b-ac90-3f65d0dc0b2a} {Suffrage:Voter ID:server-4cc2270b-0977-d62a-f36a-642c6b5a18b2 Address:4cc2270b-0977-d62a-f36a-642c6b5a18b2} {Suffrage:Voter ID:server-5c971473-666b-247a-7af5-7befb1d9604b Address:5c971473-666b-247a-7af5-7befb1d9604b}]" -2025-10-30T18:17:39.046Z [INFO] server-5c971473-666b-247a-7af5-7befb1d9604b: entering follower state: follower="Node at 5c971473-666b-247a-7af5-7befb1d9604b [Follower]" leader-address= leader-id= -2025-10-30T18:17:39.112Z [WARN] server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:39.112Z [INFO] server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: entering candidate state: node="Node at c8b39909-a245-775b-ac90-3f65d0dc0b2a [Candidate]" term=3 -2025-10-30T18:17:39.112Z [INFO] server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: election won: term=3 tally=2 -2025-10-30T18:17:39.112Z [INFO] server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: entering leader state: leader="Node at c8b39909-a245-775b-ac90-3f65d0dc0b2a [Leader]" -2025-10-30T18:17:39.112Z [INFO] server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: added peer, starting replication: peer=server-4cc2270b-0977-d62a-f36a-642c6b5a18b2 -2025-10-30T18:17:39.112Z [INFO] server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: added peer, starting replication: peer=server-5c971473-666b-247a-7af5-7befb1d9604b -2025-10-30T18:17:39.112Z [INFO] server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: pipelining replication: peer="{Voter server-5c971473-666b-247a-7af5-7befb1d9604b 5c971473-666b-247a-7af5-7befb1d9604b}" -2025-10-30T18:17:39.112Z [INFO] server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: pipelining replication: peer="{Voter server-4cc2270b-0977-d62a-f36a-642c6b5a18b2 4cc2270b-0977-d62a-f36a-642c6b5a18b2}" -2025-10-30T18:17:39.361Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:39.467Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:39.36185612 +0000 UTC m=+5.640530017 (1 nodes), 19.775µs from start of poll, 474.963417ms from cluster start. Timeout at 2025-10-30 18:17:39.466862959 +0000 UTC m=+5.745536856, 105.006839ms after stability -2025-10-30T18:17:39.467Z [INFO] server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: aborting pipeline replication: peer="{Voter server-5c971473-666b-247a-7af5-7befb1d9604b 5c971473-666b-247a-7af5-7befb1d9604b}" -2025-10-30T18:17:39.467Z [INFO] server-c8b39909-a245-775b-ac90-3f65d0dc0b2a: aborting pipeline replication: peer="{Voter server-4cc2270b-0977-d62a-f36a-642c6b5a18b2 4cc2270b-0977-d62a-f36a-642c6b5a18b2}" +2025-10-30T18:21:23.750Z [INFO] server-374d9012-23f9-cd16-b099-77fe5a90a1bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-374d9012-23f9-cd16-b099-77fe5a90a1bb Address:374d9012-23f9-cd16-b099-77fe5a90a1bb} {Suffrage:Voter ID:server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975 Address:ad0c7b27-9819-d0d7-0f30-8f9ab8aed975} {Suffrage:Voter ID:server-05d4d475-3c69-55da-8935-226cd01d6d5e Address:05d4d475-3c69-55da-8935-226cd01d6d5e}]" +2025-10-30T18:21:23.750Z [INFO] server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-374d9012-23f9-cd16-b099-77fe5a90a1bb Address:374d9012-23f9-cd16-b099-77fe5a90a1bb} {Suffrage:Voter ID:server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975 Address:ad0c7b27-9819-d0d7-0f30-8f9ab8aed975} {Suffrage:Voter ID:server-05d4d475-3c69-55da-8935-226cd01d6d5e Address:05d4d475-3c69-55da-8935-226cd01d6d5e}]" +2025-10-30T18:21:23.750Z [INFO] server-374d9012-23f9-cd16-b099-77fe5a90a1bb: entering follower state: follower="Node at 374d9012-23f9-cd16-b099-77fe5a90a1bb [Follower]" leader-address= leader-id= +2025-10-30T18:21:23.750Z [INFO] server-05d4d475-3c69-55da-8935-226cd01d6d5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-374d9012-23f9-cd16-b099-77fe5a90a1bb Address:374d9012-23f9-cd16-b099-77fe5a90a1bb} {Suffrage:Voter ID:server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975 Address:ad0c7b27-9819-d0d7-0f30-8f9ab8aed975} {Suffrage:Voter ID:server-05d4d475-3c69-55da-8935-226cd01d6d5e Address:05d4d475-3c69-55da-8935-226cd01d6d5e}]" +2025-10-30T18:21:23.750Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:23.750Z [INFO] server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975: entering follower state: follower="Node at ad0c7b27-9819-d0d7-0f30-8f9ab8aed975 [Follower]" leader-address= leader-id= +2025-10-30T18:21:23.750Z [INFO] server-05d4d475-3c69-55da-8935-226cd01d6d5e: entering follower state: follower="Node at 05d4d475-3c69-55da-8935-226cd01d6d5e [Follower]" leader-address= leader-id= +2025-10-30T18:21:23.813Z [WARN] server-05d4d475-3c69-55da-8935-226cd01d6d5e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:23.813Z [INFO] server-05d4d475-3c69-55da-8935-226cd01d6d5e: entering candidate state: node="Node at 05d4d475-3c69-55da-8935-226cd01d6d5e [Candidate]" term=2 +2025-10-30T18:21:23.813Z [INFO] server-05d4d475-3c69-55da-8935-226cd01d6d5e: election won: term=2 tally=2 +2025-10-30T18:21:23.813Z [INFO] server-05d4d475-3c69-55da-8935-226cd01d6d5e: entering leader state: leader="Node at 05d4d475-3c69-55da-8935-226cd01d6d5e [Leader]" +2025-10-30T18:21:23.813Z [INFO] server-05d4d475-3c69-55da-8935-226cd01d6d5e: added peer, starting replication: peer=server-374d9012-23f9-cd16-b099-77fe5a90a1bb +2025-10-30T18:21:23.813Z [INFO] server-05d4d475-3c69-55da-8935-226cd01d6d5e: added peer, starting replication: peer=server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975 +2025-10-30T18:21:23.813Z [INFO] server-05d4d475-3c69-55da-8935-226cd01d6d5e: pipelining replication: peer="{Voter server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975 ad0c7b27-9819-d0d7-0f30-8f9ab8aed975}" +2025-10-30T18:21:23.813Z [INFO] server-05d4d475-3c69-55da-8935-226cd01d6d5e: pipelining replication: peer="{Voter server-374d9012-23f9-cd16-b099-77fe5a90a1bb 374d9012-23f9-cd16-b099-77fe5a90a1bb}" +2025-10-30T18:21:23.919Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:23.813799005 +0000 UTC m=+6.887328298 (1 nodes), 62.873474ms from start of poll, 63.192764ms from cluster start. Timeout at 2025-10-30 18:21:23.918801935 +0000 UTC m=+6.992331227, 105.002929ms after stability +2025-10-30T18:21:23.921Z [INFO] server-05d4d475-3c69-55da-8935-226cd01d6d5e: aborting pipeline replication: peer="{Voter server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975 ad0c7b27-9819-d0d7-0f30-8f9ab8aed975}" +2025-10-30T18:21:23.921Z [INFO] server-05d4d475-3c69-55da-8935-226cd01d6d5e: aborting pipeline replication: peer="{Voter server-374d9012-23f9-cd16-b099-77fe5a90a1bb 374d9012-23f9-cd16-b099-77fe5a90a1bb}" +2025-10-30T18:21:23.921Z [INFO] creating new snapshot: path=/tmp/raft2262415446/snapshots/2-6-1761848483921.tmp +2025-10-30T18:21:23.924Z [INFO] server-374d9012-23f9-cd16-b099-77fe5a90a1bb: starting restore from snapshot: id=2-6-1761848483921 last-index=6 last-term=2 size-in-bytes=25 +2025-10-30T18:21:23.924Z [INFO] server-374d9012-23f9-cd16-b099-77fe5a90a1bb: snapshot restore progress: id=2-6-1761848483921 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% +2025-10-30T18:21:23.924Z [INFO] server-374d9012-23f9-cd16-b099-77fe5a90a1bb: restored from snapshot: id=2-6-1761848483921 last-index=6 last-term=2 size-in-bytes=25 +2025-10-30T18:21:23.924Z [INFO] server-374d9012-23f9-cd16-b099-77fe5a90a1bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-374d9012-23f9-cd16-b099-77fe5a90a1bb Address:374d9012-23f9-cd16-b099-77fe5a90a1bb} {Suffrage:Voter ID:server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975 Address:ad0c7b27-9819-d0d7-0f30-8f9ab8aed975} {Suffrage:Voter ID:server-05d4d475-3c69-55da-8935-226cd01d6d5e Address:05d4d475-3c69-55da-8935-226cd01d6d5e}]" +2025-10-30T18:21:23.924Z [INFO] creating new snapshot: path=/tmp/raft3445458034/snapshots/2-6-1761848483924.tmp +2025-10-30T18:21:23.924Z [INFO] server-374d9012-23f9-cd16-b099-77fe5a90a1bb: entering follower state: follower="Node at 374d9012-23f9-cd16-b099-77fe5a90a1bb [Follower]" leader-address= leader-id= +2025-10-30T18:21:23.925Z [INFO] server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975: starting restore from snapshot: id=2-6-1761848483924 last-index=6 last-term=2 size-in-bytes=25 +2025-10-30T18:21:23.925Z [INFO] server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975: snapshot restore progress: id=2-6-1761848483924 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% +2025-10-30T18:21:23.925Z [INFO] server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975: restored from snapshot: id=2-6-1761848483924 last-index=6 last-term=2 size-in-bytes=25 +2025-10-30T18:21:23.925Z [INFO] server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-374d9012-23f9-cd16-b099-77fe5a90a1bb Address:374d9012-23f9-cd16-b099-77fe5a90a1bb} {Suffrage:Voter ID:server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975 Address:ad0c7b27-9819-d0d7-0f30-8f9ab8aed975} {Suffrage:Voter ID:server-05d4d475-3c69-55da-8935-226cd01d6d5e Address:05d4d475-3c69-55da-8935-226cd01d6d5e}]" +2025-10-30T18:21:23.925Z [INFO] creating new snapshot: path=/tmp/raft1866281110/snapshots/2-6-1761848483925.tmp +2025-10-30T18:21:23.925Z [INFO] server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975: entering follower state: follower="Node at ad0c7b27-9819-d0d7-0f30-8f9ab8aed975 [Follower]" leader-address= leader-id= +2025-10-30T18:21:23.925Z [INFO] server-05d4d475-3c69-55da-8935-226cd01d6d5e: starting restore from snapshot: id=2-6-1761848483925 last-index=6 last-term=2 size-in-bytes=25 +2025-10-30T18:21:23.925Z [INFO] server-05d4d475-3c69-55da-8935-226cd01d6d5e: snapshot restore progress: id=2-6-1761848483925 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% +2025-10-30T18:21:23.926Z [INFO] server-05d4d475-3c69-55da-8935-226cd01d6d5e: restored from snapshot: id=2-6-1761848483925 last-index=6 last-term=2 size-in-bytes=25 +2025-10-30T18:21:23.926Z [INFO] server-05d4d475-3c69-55da-8935-226cd01d6d5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-374d9012-23f9-cd16-b099-77fe5a90a1bb Address:374d9012-23f9-cd16-b099-77fe5a90a1bb} {Suffrage:Voter ID:server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975 Address:ad0c7b27-9819-d0d7-0f30-8f9ab8aed975} {Suffrage:Voter ID:server-05d4d475-3c69-55da-8935-226cd01d6d5e Address:05d4d475-3c69-55da-8935-226cd01d6d5e}]" +2025-10-30T18:21:23.926Z [INFO] server-05d4d475-3c69-55da-8935-226cd01d6d5e: entering follower state: follower="Node at 05d4d475-3c69-55da-8935-226cd01d6d5e [Follower]" leader-address= leader-id= +2025-10-30T18:21:23.985Z [WARN] server-374d9012-23f9-cd16-b099-77fe5a90a1bb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:23.985Z [INFO] server-374d9012-23f9-cd16-b099-77fe5a90a1bb: entering candidate state: node="Node at 374d9012-23f9-cd16-b099-77fe5a90a1bb [Candidate]" term=3 +2025-10-30T18:21:23.985Z [INFO] server-374d9012-23f9-cd16-b099-77fe5a90a1bb: election won: term=3 tally=2 +2025-10-30T18:21:23.985Z [INFO] server-374d9012-23f9-cd16-b099-77fe5a90a1bb: entering leader state: leader="Node at 374d9012-23f9-cd16-b099-77fe5a90a1bb [Leader]" +2025-10-30T18:21:23.985Z [INFO] server-374d9012-23f9-cd16-b099-77fe5a90a1bb: added peer, starting replication: peer=server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975 +2025-10-30T18:21:23.985Z [INFO] server-374d9012-23f9-cd16-b099-77fe5a90a1bb: added peer, starting replication: peer=server-05d4d475-3c69-55da-8935-226cd01d6d5e +2025-10-30T18:21:23.985Z [INFO] server-374d9012-23f9-cd16-b099-77fe5a90a1bb: pipelining replication: peer="{Voter server-05d4d475-3c69-55da-8935-226cd01d6d5e 05d4d475-3c69-55da-8935-226cd01d6d5e}" +2025-10-30T18:21:23.985Z [INFO] server-374d9012-23f9-cd16-b099-77fe5a90a1bb: pipelining replication: peer="{Voter server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975 ad0c7b27-9819-d0d7-0f30-8f9ab8aed975}" +2025-10-30T18:21:24.241Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:24.347Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:24.241867626 +0000 UTC m=+7.315396925 (1 nodes), 30.538µs from start of poll, 491.261391ms from cluster start. Timeout at 2025-10-30 18:21:24.34687222 +0000 UTC m=+7.420401508, 105.004583ms after stability +2025-10-30T18:21:24.347Z [INFO] server-374d9012-23f9-cd16-b099-77fe5a90a1bb: aborting pipeline replication: peer="{Voter server-05d4d475-3c69-55da-8935-226cd01d6d5e 05d4d475-3c69-55da-8935-226cd01d6d5e}" +2025-10-30T18:21:24.347Z [INFO] server-374d9012-23f9-cd16-b099-77fe5a90a1bb: aborting pipeline replication: peer="{Voter server-ad0c7b27-9819-d0d7-0f30-8f9ab8aed975 ad0c7b27-9819-d0d7-0f30-8f9ab8aed975}" === RUN TestRaft_RecoverCluster/snapshot,_with_trailing_logs -2025-10-30T18:17:39.468Z [INFO] server-abbab455-2a1d-ab78-f456-f78ffa09bbea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbab455-2a1d-ab78-f456-f78ffa09bbea Address:abbab455-2a1d-ab78-f456-f78ffa09bbea} {Suffrage:Voter ID:server-6039fdef-4f5f-b129-7db1-72c176ed8608 Address:6039fdef-4f5f-b129-7db1-72c176ed8608} {Suffrage:Voter ID:server-d52a0d34-73b1-3c51-d424-a546abfa0ead Address:d52a0d34-73b1-3c51-d424-a546abfa0ead}]" -2025-10-30T18:17:39.468Z [INFO] server-abbab455-2a1d-ab78-f456-f78ffa09bbea: entering follower state: follower="Node at abbab455-2a1d-ab78-f456-f78ffa09bbea [Follower]" leader-address= leader-id= -2025-10-30T18:17:39.468Z [INFO] server-6039fdef-4f5f-b129-7db1-72c176ed8608: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbab455-2a1d-ab78-f456-f78ffa09bbea Address:abbab455-2a1d-ab78-f456-f78ffa09bbea} {Suffrage:Voter ID:server-6039fdef-4f5f-b129-7db1-72c176ed8608 Address:6039fdef-4f5f-b129-7db1-72c176ed8608} {Suffrage:Voter ID:server-d52a0d34-73b1-3c51-d424-a546abfa0ead Address:d52a0d34-73b1-3c51-d424-a546abfa0ead}]" -2025-10-30T18:17:39.468Z [INFO] server-6039fdef-4f5f-b129-7db1-72c176ed8608: entering follower state: follower="Node at 6039fdef-4f5f-b129-7db1-72c176ed8608 [Follower]" leader-address= leader-id= -2025-10-30T18:17:39.468Z [INFO] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbab455-2a1d-ab78-f456-f78ffa09bbea Address:abbab455-2a1d-ab78-f456-f78ffa09bbea} {Suffrage:Voter ID:server-6039fdef-4f5f-b129-7db1-72c176ed8608 Address:6039fdef-4f5f-b129-7db1-72c176ed8608} {Suffrage:Voter ID:server-d52a0d34-73b1-3c51-d424-a546abfa0ead Address:d52a0d34-73b1-3c51-d424-a546abfa0ead}]" -2025-10-30T18:17:39.468Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:39.468Z [INFO] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: entering follower state: follower="Node at d52a0d34-73b1-3c51-d424-a546abfa0ead [Follower]" leader-address= leader-id= -2025-10-30T18:17:39.527Z [WARN] server-6039fdef-4f5f-b129-7db1-72c176ed8608: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:39.527Z [INFO] server-6039fdef-4f5f-b129-7db1-72c176ed8608: entering candidate state: node="Node at 6039fdef-4f5f-b129-7db1-72c176ed8608 [Candidate]" term=2 -2025-10-30T18:17:39.527Z [INFO] server-6039fdef-4f5f-b129-7db1-72c176ed8608: election won: term=2 tally=2 -2025-10-30T18:17:39.527Z [INFO] server-6039fdef-4f5f-b129-7db1-72c176ed8608: entering leader state: leader="Node at 6039fdef-4f5f-b129-7db1-72c176ed8608 [Leader]" -2025-10-30T18:17:39.527Z [INFO] server-6039fdef-4f5f-b129-7db1-72c176ed8608: added peer, starting replication: peer=server-abbab455-2a1d-ab78-f456-f78ffa09bbea -2025-10-30T18:17:39.527Z [INFO] server-6039fdef-4f5f-b129-7db1-72c176ed8608: added peer, starting replication: peer=server-d52a0d34-73b1-3c51-d424-a546abfa0ead -2025-10-30T18:17:39.528Z [INFO] server-6039fdef-4f5f-b129-7db1-72c176ed8608: pipelining replication: peer="{Voter server-abbab455-2a1d-ab78-f456-f78ffa09bbea abbab455-2a1d-ab78-f456-f78ffa09bbea}" -2025-10-30T18:17:39.528Z [INFO] server-6039fdef-4f5f-b129-7db1-72c176ed8608: pipelining replication: peer="{Voter server-d52a0d34-73b1-3c51-d424-a546abfa0ead d52a0d34-73b1-3c51-d424-a546abfa0ead}" -2025-10-30T18:17:39.633Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:39.527901003 +0000 UTC m=+5.806574899 (1 nodes), 59.037888ms from start of poll, 59.414779ms from cluster start. Timeout at 2025-10-30 18:17:39.632903401 +0000 UTC m=+5.911577299, 105.0024ms after stability -2025-10-30T18:17:39.635Z [INFO] server-6039fdef-4f5f-b129-7db1-72c176ed8608: aborting pipeline replication: peer="{Voter server-d52a0d34-73b1-3c51-d424-a546abfa0ead d52a0d34-73b1-3c51-d424-a546abfa0ead}" -2025-10-30T18:17:39.635Z [INFO] server-6039fdef-4f5f-b129-7db1-72c176ed8608: aborting pipeline replication: peer="{Voter server-abbab455-2a1d-ab78-f456-f78ffa09bbea abbab455-2a1d-ab78-f456-f78ffa09bbea}" -2025-10-30T18:17:39.635Z [INFO] creating new snapshot: path=/tmp/raft2050563930/snapshots/2-27-1761848259635.tmp -2025-10-30T18:17:39.635Z [INFO] server-abbab455-2a1d-ab78-f456-f78ffa09bbea: starting restore from snapshot: id=2-27-1761848259635 last-index=27 last-term=2 size-in-bytes=168 -2025-10-30T18:17:39.635Z [INFO] server-abbab455-2a1d-ab78-f456-f78ffa09bbea: snapshot restore progress: id=2-27-1761848259635 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% -2025-10-30T18:17:39.635Z [INFO] server-abbab455-2a1d-ab78-f456-f78ffa09bbea: restored from snapshot: id=2-27-1761848259635 last-index=27 last-term=2 size-in-bytes=168 -2025-10-30T18:17:39.636Z [INFO] server-abbab455-2a1d-ab78-f456-f78ffa09bbea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbab455-2a1d-ab78-f456-f78ffa09bbea Address:abbab455-2a1d-ab78-f456-f78ffa09bbea} {Suffrage:Voter ID:server-6039fdef-4f5f-b129-7db1-72c176ed8608 Address:6039fdef-4f5f-b129-7db1-72c176ed8608} {Suffrage:Voter ID:server-d52a0d34-73b1-3c51-d424-a546abfa0ead Address:d52a0d34-73b1-3c51-d424-a546abfa0ead}]" -2025-10-30T18:17:39.636Z [INFO] server-abbab455-2a1d-ab78-f456-f78ffa09bbea: entering follower state: follower="Node at abbab455-2a1d-ab78-f456-f78ffa09bbea [Follower]" leader-address= leader-id= -2025-10-30T18:17:39.636Z [INFO] creating new snapshot: path=/tmp/raft542924102/snapshots/2-27-1761848259636.tmp -2025-10-30T18:17:39.636Z [INFO] server-6039fdef-4f5f-b129-7db1-72c176ed8608: starting restore from snapshot: id=2-27-1761848259636 last-index=27 last-term=2 size-in-bytes=168 -2025-10-30T18:17:39.636Z [INFO] server-6039fdef-4f5f-b129-7db1-72c176ed8608: snapshot restore progress: id=2-27-1761848259636 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% -2025-10-30T18:17:39.636Z [INFO] server-6039fdef-4f5f-b129-7db1-72c176ed8608: restored from snapshot: id=2-27-1761848259636 last-index=27 last-term=2 size-in-bytes=168 -2025-10-30T18:17:39.636Z [INFO] server-6039fdef-4f5f-b129-7db1-72c176ed8608: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbab455-2a1d-ab78-f456-f78ffa09bbea Address:abbab455-2a1d-ab78-f456-f78ffa09bbea} {Suffrage:Voter ID:server-6039fdef-4f5f-b129-7db1-72c176ed8608 Address:6039fdef-4f5f-b129-7db1-72c176ed8608} {Suffrage:Voter ID:server-d52a0d34-73b1-3c51-d424-a546abfa0ead Address:d52a0d34-73b1-3c51-d424-a546abfa0ead}]" -2025-10-30T18:17:39.636Z [INFO] creating new snapshot: path=/tmp/raft1881663974/snapshots/2-27-1761848259636.tmp -2025-10-30T18:17:39.636Z [INFO] server-6039fdef-4f5f-b129-7db1-72c176ed8608: entering follower state: follower="Node at 6039fdef-4f5f-b129-7db1-72c176ed8608 [Follower]" leader-address= leader-id= -2025-10-30T18:17:39.636Z [INFO] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: starting restore from snapshot: id=2-27-1761848259636 last-index=27 last-term=2 size-in-bytes=168 -2025-10-30T18:17:39.637Z [INFO] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: snapshot restore progress: id=2-27-1761848259636 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% -2025-10-30T18:17:39.637Z [INFO] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: restored from snapshot: id=2-27-1761848259636 last-index=27 last-term=2 size-in-bytes=168 -2025-10-30T18:17:39.637Z [INFO] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abbab455-2a1d-ab78-f456-f78ffa09bbea Address:abbab455-2a1d-ab78-f456-f78ffa09bbea} {Suffrage:Voter ID:server-6039fdef-4f5f-b129-7db1-72c176ed8608 Address:6039fdef-4f5f-b129-7db1-72c176ed8608} {Suffrage:Voter ID:server-d52a0d34-73b1-3c51-d424-a546abfa0ead Address:d52a0d34-73b1-3c51-d424-a546abfa0ead}]" -2025-10-30T18:17:39.637Z [INFO] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: entering follower state: follower="Node at d52a0d34-73b1-3c51-d424-a546abfa0ead [Follower]" leader-address= leader-id= -2025-10-30T18:17:39.709Z [WARN] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:39.709Z [INFO] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: entering candidate state: node="Node at d52a0d34-73b1-3c51-d424-a546abfa0ead [Candidate]" term=3 -2025-10-30T18:17:39.709Z [INFO] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: election won: term=3 tally=2 -2025-10-30T18:17:39.709Z [INFO] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: entering leader state: leader="Node at d52a0d34-73b1-3c51-d424-a546abfa0ead [Leader]" -2025-10-30T18:17:39.709Z [INFO] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: added peer, starting replication: peer=server-abbab455-2a1d-ab78-f456-f78ffa09bbea -2025-10-30T18:17:39.709Z [WARN] server-abbab455-2a1d-ab78-f456-f78ffa09bbea: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:39.709Z [INFO] server-abbab455-2a1d-ab78-f456-f78ffa09bbea: entering candidate state: node="Node at abbab455-2a1d-ab78-f456-f78ffa09bbea [Candidate]" term=3 -2025-10-30T18:17:39.709Z [INFO] server-abbab455-2a1d-ab78-f456-f78ffa09bbea: duplicate requestVote for same term: term=3 -2025-10-30T18:17:39.709Z [INFO] server-abbab455-2a1d-ab78-f456-f78ffa09bbea: entering follower state: follower="Node at abbab455-2a1d-ab78-f456-f78ffa09bbea [Follower]" leader-address=d52a0d34-73b1-3c51-d424-a546abfa0ead leader-id=server-d52a0d34-73b1-3c51-d424-a546abfa0ead -2025-10-30T18:17:39.709Z [INFO] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: added peer, starting replication: peer=server-6039fdef-4f5f-b129-7db1-72c176ed8608 -2025-10-30T18:17:39.709Z [INFO] server-6039fdef-4f5f-b129-7db1-72c176ed8608: duplicate requestVote for same term: term=3 -2025-10-30T18:17:39.709Z [INFO] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: pipelining replication: peer="{Voter server-abbab455-2a1d-ab78-f456-f78ffa09bbea abbab455-2a1d-ab78-f456-f78ffa09bbea}" -2025-10-30T18:17:39.709Z [WARN] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: rejecting vote request since we have a leader: from=abbab455-2a1d-ab78-f456-f78ffa09bbea leader=d52a0d34-73b1-3c51-d424-a546abfa0ead leader-id=server-d52a0d34-73b1-3c51-d424-a546abfa0ead -2025-10-30T18:17:39.709Z [INFO] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: pipelining replication: peer="{Voter server-6039fdef-4f5f-b129-7db1-72c176ed8608 6039fdef-4f5f-b129-7db1-72c176ed8608}" -2025-10-30T18:17:39.953Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:40.059Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:39.953642724 +0000 UTC m=+6.232316621 (1 nodes), 21.882µs from start of poll, 485.156501ms from cluster start. Timeout at 2025-10-30 18:17:40.058647922 +0000 UTC m=+6.337321819, 105.005198ms after stability -2025-10-30T18:17:40.059Z [INFO] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: aborting pipeline replication: peer="{Voter server-abbab455-2a1d-ab78-f456-f78ffa09bbea abbab455-2a1d-ab78-f456-f78ffa09bbea}" -2025-10-30T18:17:40.059Z [INFO] server-d52a0d34-73b1-3c51-d424-a546abfa0ead: aborting pipeline replication: peer="{Voter server-6039fdef-4f5f-b129-7db1-72c176ed8608 6039fdef-4f5f-b129-7db1-72c176ed8608}" ---- PASS: TestRaft_RecoverCluster (1.77s) - --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.60s) - --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.58s) - --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.59s) +2025-10-30T18:21:24.392Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0aefa1b3-5b19-c7b1-f405-be42793b8dff Address:0aefa1b3-5b19-c7b1-f405-be42793b8dff} {Suffrage:Voter ID:server-c441637f-d6a7-8780-c73d-2bc0c527a041 Address:c441637f-d6a7-8780-c73d-2bc0c527a041} {Suffrage:Voter ID:server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 Address:0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456}]" +2025-10-30T18:21:24.392Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: entering follower state: follower="Node at 0aefa1b3-5b19-c7b1-f405-be42793b8dff [Follower]" leader-address= leader-id= +2025-10-30T18:21:24.392Z [INFO] server-c441637f-d6a7-8780-c73d-2bc0c527a041: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0aefa1b3-5b19-c7b1-f405-be42793b8dff Address:0aefa1b3-5b19-c7b1-f405-be42793b8dff} {Suffrage:Voter ID:server-c441637f-d6a7-8780-c73d-2bc0c527a041 Address:c441637f-d6a7-8780-c73d-2bc0c527a041} {Suffrage:Voter ID:server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 Address:0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456}]" +2025-10-30T18:21:24.392Z [INFO] server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0aefa1b3-5b19-c7b1-f405-be42793b8dff Address:0aefa1b3-5b19-c7b1-f405-be42793b8dff} {Suffrage:Voter ID:server-c441637f-d6a7-8780-c73d-2bc0c527a041 Address:c441637f-d6a7-8780-c73d-2bc0c527a041} {Suffrage:Voter ID:server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 Address:0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456}]" +2025-10-30T18:21:24.392Z [INFO] server-c441637f-d6a7-8780-c73d-2bc0c527a041: entering follower state: follower="Node at c441637f-d6a7-8780-c73d-2bc0c527a041 [Follower]" leader-address= leader-id= +2025-10-30T18:21:24.392Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:24.392Z [INFO] server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456: entering follower state: follower="Node at 0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 [Follower]" leader-address= leader-id= +2025-10-30T18:21:24.455Z [WARN] server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:24.455Z [INFO] server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456: entering candidate state: node="Node at 0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 [Candidate]" term=2 +2025-10-30T18:21:24.455Z [INFO] server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456: election won: term=2 tally=2 +2025-10-30T18:21:24.455Z [INFO] server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456: entering leader state: leader="Node at 0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 [Leader]" +2025-10-30T18:21:24.455Z [INFO] server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456: added peer, starting replication: peer=server-0aefa1b3-5b19-c7b1-f405-be42793b8dff +2025-10-30T18:21:24.455Z [INFO] server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456: added peer, starting replication: peer=server-c441637f-d6a7-8780-c73d-2bc0c527a041 +2025-10-30T18:21:24.455Z [INFO] server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456: pipelining replication: peer="{Voter server-c441637f-d6a7-8780-c73d-2bc0c527a041 c441637f-d6a7-8780-c73d-2bc0c527a041}" +2025-10-30T18:21:24.455Z [INFO] server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456: pipelining replication: peer="{Voter server-0aefa1b3-5b19-c7b1-f405-be42793b8dff 0aefa1b3-5b19-c7b1-f405-be42793b8dff}" +2025-10-30T18:21:24.560Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:24.455492789 +0000 UTC m=+7.529022096 (1 nodes), 63.002948ms from start of poll, 63.364085ms from cluster start. Timeout at 2025-10-30 18:21:24.560495128 +0000 UTC m=+7.634024417, 105.002321ms after stability +2025-10-30T18:21:24.563Z [INFO] server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456: aborting pipeline replication: peer="{Voter server-c441637f-d6a7-8780-c73d-2bc0c527a041 c441637f-d6a7-8780-c73d-2bc0c527a041}" +2025-10-30T18:21:24.563Z [INFO] server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456: aborting pipeline replication: peer="{Voter server-0aefa1b3-5b19-c7b1-f405-be42793b8dff 0aefa1b3-5b19-c7b1-f405-be42793b8dff}" +2025-10-30T18:21:24.563Z [INFO] creating new snapshot: path=/tmp/raft2503074049/snapshots/2-27-1761848484563.tmp +2025-10-30T18:21:24.739Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: starting restore from snapshot: id=2-27-1761848484563 last-index=27 last-term=2 size-in-bytes=168 +2025-10-30T18:21:24.739Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: snapshot restore progress: id=2-27-1761848484563 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% +2025-10-30T18:21:24.739Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: restored from snapshot: id=2-27-1761848484563 last-index=27 last-term=2 size-in-bytes=168 +2025-10-30T18:21:24.739Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0aefa1b3-5b19-c7b1-f405-be42793b8dff Address:0aefa1b3-5b19-c7b1-f405-be42793b8dff} {Suffrage:Voter ID:server-c441637f-d6a7-8780-c73d-2bc0c527a041 Address:c441637f-d6a7-8780-c73d-2bc0c527a041} {Suffrage:Voter ID:server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 Address:0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456}]" +2025-10-30T18:21:24.739Z [INFO] creating new snapshot: path=/tmp/raft2419836470/snapshots/2-27-1761848484739.tmp +2025-10-30T18:21:24.739Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: entering follower state: follower="Node at 0aefa1b3-5b19-c7b1-f405-be42793b8dff [Follower]" leader-address= leader-id= +2025-10-30T18:21:24.805Z [WARN] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:24.805Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: entering candidate state: node="Node at 0aefa1b3-5b19-c7b1-f405-be42793b8dff [Candidate]" term=3 +2025-10-30T18:21:24.805Z [ERROR] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: failed to make requestVote RPC: target="{Voter server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456}" error="failed to connect to peer: 0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456" term=3 +2025-10-30T18:21:24.805Z [ERROR] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: failed to make requestVote RPC: target="{Voter server-c441637f-d6a7-8780-c73d-2bc0c527a041 c441637f-d6a7-8780-c73d-2bc0c527a041}" error="failed to connect to peer: c441637f-d6a7-8780-c73d-2bc0c527a041" term=3 +2025-10-30T18:21:24.873Z [INFO] server-c441637f-d6a7-8780-c73d-2bc0c527a041: starting restore from snapshot: id=2-27-1761848484739 last-index=27 last-term=2 size-in-bytes=168 +2025-10-30T18:21:24.873Z [INFO] server-c441637f-d6a7-8780-c73d-2bc0c527a041: snapshot restore progress: id=2-27-1761848484739 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% +2025-10-30T18:21:24.873Z [INFO] server-c441637f-d6a7-8780-c73d-2bc0c527a041: restored from snapshot: id=2-27-1761848484739 last-index=27 last-term=2 size-in-bytes=168 +2025-10-30T18:21:24.873Z [INFO] server-c441637f-d6a7-8780-c73d-2bc0c527a041: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0aefa1b3-5b19-c7b1-f405-be42793b8dff Address:0aefa1b3-5b19-c7b1-f405-be42793b8dff} {Suffrage:Voter ID:server-c441637f-d6a7-8780-c73d-2bc0c527a041 Address:c441637f-d6a7-8780-c73d-2bc0c527a041} {Suffrage:Voter ID:server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 Address:0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456}]" +2025-10-30T18:21:24.873Z [INFO] server-c441637f-d6a7-8780-c73d-2bc0c527a041: entering follower state: follower="Node at c441637f-d6a7-8780-c73d-2bc0c527a041 [Follower]" leader-address= leader-id= +2025-10-30T18:21:24.873Z [INFO] creating new snapshot: path=/tmp/raft252724502/snapshots/2-27-1761848484873.tmp +2025-10-30T18:21:24.886Z [WARN] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: Election timeout reached, restarting election +2025-10-30T18:21:24.886Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: entering candidate state: node="Node at 0aefa1b3-5b19-c7b1-f405-be42793b8dff [Candidate]" term=4 +2025-10-30T18:21:24.886Z [ERROR] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: failed to make requestVote RPC: target="{Voter server-c441637f-d6a7-8780-c73d-2bc0c527a041 c441637f-d6a7-8780-c73d-2bc0c527a041}" error="failed to connect to peer: c441637f-d6a7-8780-c73d-2bc0c527a041" term=4 +2025-10-30T18:21:24.886Z [ERROR] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: failed to make requestVote RPC: target="{Voter server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456}" error="failed to connect to peer: 0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456" term=4 +2025-10-30T18:21:24.930Z [WARN] server-c441637f-d6a7-8780-c73d-2bc0c527a041: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:24.930Z [INFO] server-c441637f-d6a7-8780-c73d-2bc0c527a041: entering candidate state: node="Node at c441637f-d6a7-8780-c73d-2bc0c527a041 [Candidate]" term=3 +2025-10-30T18:21:24.930Z [ERROR] server-c441637f-d6a7-8780-c73d-2bc0c527a041: failed to make requestVote RPC: target="{Voter server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456}" error="failed to connect to peer: 0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456" term=3 +2025-10-30T18:21:24.930Z [ERROR] server-c441637f-d6a7-8780-c73d-2bc0c527a041: failed to make requestVote RPC: target="{Voter server-0aefa1b3-5b19-c7b1-f405-be42793b8dff 0aefa1b3-5b19-c7b1-f405-be42793b8dff}" error="failed to connect to peer: 0aefa1b3-5b19-c7b1-f405-be42793b8dff" term=3 +2025-10-30T18:21:24.970Z [WARN] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: Election timeout reached, restarting election +2025-10-30T18:21:24.970Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: entering candidate state: node="Node at 0aefa1b3-5b19-c7b1-f405-be42793b8dff [Candidate]" term=5 +2025-10-30T18:21:24.970Z [ERROR] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: failed to make requestVote RPC: target="{Voter server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456}" error="failed to connect to peer: 0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456" term=5 +2025-10-30T18:21:24.970Z [ERROR] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: failed to make requestVote RPC: target="{Voter server-c441637f-d6a7-8780-c73d-2bc0c527a041 c441637f-d6a7-8780-c73d-2bc0c527a041}" error="failed to connect to peer: c441637f-d6a7-8780-c73d-2bc0c527a041" term=5 +2025-10-30T18:21:24.998Z [WARN] server-c441637f-d6a7-8780-c73d-2bc0c527a041: Election timeout reached, restarting election +2025-10-30T18:21:24.998Z [INFO] server-c441637f-d6a7-8780-c73d-2bc0c527a041: entering candidate state: node="Node at c441637f-d6a7-8780-c73d-2bc0c527a041 [Candidate]" term=4 +2025-10-30T18:21:24.998Z [ERROR] server-c441637f-d6a7-8780-c73d-2bc0c527a041: failed to make requestVote RPC: target="{Voter server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456}" error="failed to connect to peer: 0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456" term=4 +2025-10-30T18:21:24.998Z [ERROR] server-c441637f-d6a7-8780-c73d-2bc0c527a041: failed to make requestVote RPC: target="{Voter server-0aefa1b3-5b19-c7b1-f405-be42793b8dff 0aefa1b3-5b19-c7b1-f405-be42793b8dff}" error="failed to connect to peer: 0aefa1b3-5b19-c7b1-f405-be42793b8dff" term=4 +2025-10-30T18:21:24.999Z [INFO] server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456: starting restore from snapshot: id=2-27-1761848484873 last-index=27 last-term=2 size-in-bytes=168 +2025-10-30T18:21:24.999Z [INFO] server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456: snapshot restore progress: id=2-27-1761848484873 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% +2025-10-30T18:21:24.999Z [INFO] server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456: restored from snapshot: id=2-27-1761848484873 last-index=27 last-term=2 size-in-bytes=168 +2025-10-30T18:21:24.999Z [INFO] server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0aefa1b3-5b19-c7b1-f405-be42793b8dff Address:0aefa1b3-5b19-c7b1-f405-be42793b8dff} {Suffrage:Voter ID:server-c441637f-d6a7-8780-c73d-2bc0c527a041 Address:c441637f-d6a7-8780-c73d-2bc0c527a041} {Suffrage:Voter ID:server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 Address:0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456}]" +2025-10-30T18:21:24.999Z [INFO] server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456: entering follower state: follower="Node at 0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 [Follower]" leader-address= leader-id= +2025-10-30T18:21:25.060Z [WARN] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: Election timeout reached, restarting election +2025-10-30T18:21:25.060Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: entering candidate state: node="Node at 0aefa1b3-5b19-c7b1-f405-be42793b8dff [Candidate]" term=6 +2025-10-30T18:21:25.060Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: election won: term=6 tally=2 +2025-10-30T18:21:25.060Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: entering leader state: leader="Node at 0aefa1b3-5b19-c7b1-f405-be42793b8dff [Leader]" +2025-10-30T18:21:25.060Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: added peer, starting replication: peer=server-c441637f-d6a7-8780-c73d-2bc0c527a041 +2025-10-30T18:21:25.060Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: added peer, starting replication: peer=server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 +2025-10-30T18:21:25.060Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: pipelining replication: peer="{Voter server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456}" +2025-10-30T18:21:25.060Z [INFO] server-c441637f-d6a7-8780-c73d-2bc0c527a041: entering follower state: follower="Node at c441637f-d6a7-8780-c73d-2bc0c527a041 [Follower]" leader-address= leader-id= +2025-10-30T18:21:25.060Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: pipelining replication: peer="{Voter server-c441637f-d6a7-8780-c73d-2bc0c527a041 c441637f-d6a7-8780-c73d-2bc0c527a041}" +2025-10-30T18:21:25.314Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:25.420Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:25.314775423 +0000 UTC m=+8.388304727 (1 nodes), 24.015µs from start of poll, 922.646716ms from cluster start. Timeout at 2025-10-30 18:21:25.419781696 +0000 UTC m=+8.493311059, 105.006332ms after stability +2025-10-30T18:21:25.420Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: aborting pipeline replication: peer="{Voter server-0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456 0eeabdb7-ebc7-cc25-6e41-a3bf6f9e3456}" +2025-10-30T18:21:25.420Z [INFO] server-0aefa1b3-5b19-c7b1-f405-be42793b8dff: aborting pipeline replication: peer="{Voter server-c441637f-d6a7-8780-c73d-2bc0c527a041 c441637f-d6a7-8780-c73d-2bc0c527a041}" +--- PASS: TestRaft_RecoverCluster (2.39s) + --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.69s) + --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.64s) + --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (1.06s) === RUN TestRaft_HasExistingState raft_test.go:258: Skipping unreliable test --- SKIP: TestRaft_HasExistingState (0.00s) === RUN TestRaft_SingleNode -2025-10-30T18:17:40.059Z [INFO] server-7189ecfb-ee69-aa78-562d-57a224cd7851: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7189ecfb-ee69-aa78-562d-57a224cd7851 Address:7189ecfb-ee69-aa78-562d-57a224cd7851}]" -2025-10-30T18:17:40.060Z [INFO] server-7189ecfb-ee69-aa78-562d-57a224cd7851: entering follower state: follower="Node at 7189ecfb-ee69-aa78-562d-57a224cd7851 [Follower]" leader-address= leader-id= -2025-10-30T18:17:40.123Z [WARN] server-7189ecfb-ee69-aa78-562d-57a224cd7851: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:40.123Z [INFO] server-7189ecfb-ee69-aa78-562d-57a224cd7851: entering candidate state: node="Node at 7189ecfb-ee69-aa78-562d-57a224cd7851 [Candidate]" term=2 -2025-10-30T18:17:40.123Z [INFO] server-7189ecfb-ee69-aa78-562d-57a224cd7851: election won: term=2 tally=1 -2025-10-30T18:17:40.123Z [INFO] server-7189ecfb-ee69-aa78-562d-57a224cd7851: entering leader state: leader="Node at 7189ecfb-ee69-aa78-562d-57a224cd7851 [Leader]" ---- PASS: TestRaft_SingleNode (0.06s) +2025-10-30T18:21:25.449Z [INFO] server-0250e1fe-9bdc-372f-ca33-582d1e869c12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0250e1fe-9bdc-372f-ca33-582d1e869c12 Address:0250e1fe-9bdc-372f-ca33-582d1e869c12}]" +2025-10-30T18:21:25.449Z [INFO] server-0250e1fe-9bdc-372f-ca33-582d1e869c12: entering follower state: follower="Node at 0250e1fe-9bdc-372f-ca33-582d1e869c12 [Follower]" leader-address= leader-id= +2025-10-30T18:21:25.538Z [WARN] server-0250e1fe-9bdc-372f-ca33-582d1e869c12: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:25.539Z [INFO] server-0250e1fe-9bdc-372f-ca33-582d1e869c12: entering candidate state: node="Node at 0250e1fe-9bdc-372f-ca33-582d1e869c12 [Candidate]" term=2 +2025-10-30T18:21:25.539Z [INFO] server-0250e1fe-9bdc-372f-ca33-582d1e869c12: election won: term=2 tally=1 +2025-10-30T18:21:25.539Z [INFO] server-0250e1fe-9bdc-372f-ca33-582d1e869c12: entering leader state: leader="Node at 0250e1fe-9bdc-372f-ca33-582d1e869c12 [Leader]" +--- PASS: TestRaft_SingleNode (0.09s) === RUN TestRaft_TripleNode -2025-10-30T18:17:40.124Z [INFO] server-32fe0f14-6e18-47c3-e923-cdb478e3a3b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32fe0f14-6e18-47c3-e923-cdb478e3a3b6 Address:32fe0f14-6e18-47c3-e923-cdb478e3a3b6} {Suffrage:Voter ID:server-544d3a6f-07c8-1695-9728-8192af9796d9 Address:544d3a6f-07c8-1695-9728-8192af9796d9} {Suffrage:Voter ID:server-24b6f6f6-d117-a22d-869b-0ba0aca8f188 Address:24b6f6f6-d117-a22d-869b-0ba0aca8f188}]" -2025-10-30T18:17:40.124Z [INFO] server-544d3a6f-07c8-1695-9728-8192af9796d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32fe0f14-6e18-47c3-e923-cdb478e3a3b6 Address:32fe0f14-6e18-47c3-e923-cdb478e3a3b6} {Suffrage:Voter ID:server-544d3a6f-07c8-1695-9728-8192af9796d9 Address:544d3a6f-07c8-1695-9728-8192af9796d9} {Suffrage:Voter ID:server-24b6f6f6-d117-a22d-869b-0ba0aca8f188 Address:24b6f6f6-d117-a22d-869b-0ba0aca8f188}]" -2025-10-30T18:17:40.124Z [INFO] server-32fe0f14-6e18-47c3-e923-cdb478e3a3b6: entering follower state: follower="Node at 32fe0f14-6e18-47c3-e923-cdb478e3a3b6 [Follower]" leader-address= leader-id= -2025-10-30T18:17:40.124Z [INFO] server-544d3a6f-07c8-1695-9728-8192af9796d9: entering follower state: follower="Node at 544d3a6f-07c8-1695-9728-8192af9796d9 [Follower]" leader-address= leader-id= -2025-10-30T18:17:40.124Z [INFO] server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32fe0f14-6e18-47c3-e923-cdb478e3a3b6 Address:32fe0f14-6e18-47c3-e923-cdb478e3a3b6} {Suffrage:Voter ID:server-544d3a6f-07c8-1695-9728-8192af9796d9 Address:544d3a6f-07c8-1695-9728-8192af9796d9} {Suffrage:Voter ID:server-24b6f6f6-d117-a22d-869b-0ba0aca8f188 Address:24b6f6f6-d117-a22d-869b-0ba0aca8f188}]" -2025-10-30T18:17:40.124Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:17:40.124Z [INFO] server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: entering follower state: follower="Node at 24b6f6f6-d117-a22d-869b-0ba0aca8f188 [Follower]" leader-address= leader-id= -2025-10-30T18:17:40.187Z [WARN] server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:40.187Z [INFO] server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: entering candidate state: node="Node at 24b6f6f6-d117-a22d-869b-0ba0aca8f188 [Candidate]" term=2 -2025-10-30T18:17:40.187Z [INFO] server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: election won: term=2 tally=2 -2025-10-30T18:17:40.187Z [INFO] server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: entering leader state: leader="Node at 24b6f6f6-d117-a22d-869b-0ba0aca8f188 [Leader]" -2025-10-30T18:17:40.187Z [INFO] server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: added peer, starting replication: peer=server-32fe0f14-6e18-47c3-e923-cdb478e3a3b6 -2025-10-30T18:17:40.187Z [INFO] server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: added peer, starting replication: peer=server-544d3a6f-07c8-1695-9728-8192af9796d9 -2025-10-30T18:17:40.187Z [INFO] server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: pipelining replication: peer="{Voter server-32fe0f14-6e18-47c3-e923-cdb478e3a3b6 32fe0f14-6e18-47c3-e923-cdb478e3a3b6}" -2025-10-30T18:17:40.187Z [INFO] server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: pipelining replication: peer="{Voter server-544d3a6f-07c8-1695-9728-8192af9796d9 544d3a6f-07c8-1695-9728-8192af9796d9}" -2025-10-30T18:17:40.292Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:17:40.187220837 +0000 UTC m=+6.465894735 (2 nodes), 62.373027ms from start of poll, 62.626567ms from cluster start. Timeout at 2025-10-30 18:17:40.292222693 +0000 UTC m=+6.570896590, 105.001855ms after stability -2025-10-30T18:17:40.292Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:40.397Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:40.292388294 +0000 UTC m=+6.571062195 (1 nodes), 2.083µs from start of poll, 167.794027ms from cluster start. Timeout at 2025-10-30 18:17:40.397392056 +0000 UTC m=+6.676065952, 105.003757ms after stability -2025-10-30T18:17:40.408Z [INFO] server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: aborting pipeline replication: peer="{Voter server-544d3a6f-07c8-1695-9728-8192af9796d9 544d3a6f-07c8-1695-9728-8192af9796d9}" -2025-10-30T18:17:40.408Z [INFO] server-24b6f6f6-d117-a22d-869b-0ba0aca8f188: aborting pipeline replication: peer="{Voter server-32fe0f14-6e18-47c3-e923-cdb478e3a3b6 32fe0f14-6e18-47c3-e923-cdb478e3a3b6}" ---- PASS: TestRaft_TripleNode (0.28s) +2025-10-30T18:21:25.539Z [INFO] server-bb7813bd-a1d9-56d8-7a4f-a38b220475f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb7813bd-a1d9-56d8-7a4f-a38b220475f1 Address:bb7813bd-a1d9-56d8-7a4f-a38b220475f1} {Suffrage:Voter ID:server-766d51d3-e734-e5b4-bc24-a2f04c2cc3ba Address:766d51d3-e734-e5b4-bc24-a2f04c2cc3ba} {Suffrage:Voter ID:server-573a809a-1a6e-7dd2-657e-add2205b8c4f Address:573a809a-1a6e-7dd2-657e-add2205b8c4f}]" +2025-10-30T18:21:25.540Z [INFO] server-bb7813bd-a1d9-56d8-7a4f-a38b220475f1: entering follower state: follower="Node at bb7813bd-a1d9-56d8-7a4f-a38b220475f1 [Follower]" leader-address= leader-id= +2025-10-30T18:21:25.540Z [INFO] server-766d51d3-e734-e5b4-bc24-a2f04c2cc3ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb7813bd-a1d9-56d8-7a4f-a38b220475f1 Address:bb7813bd-a1d9-56d8-7a4f-a38b220475f1} {Suffrage:Voter ID:server-766d51d3-e734-e5b4-bc24-a2f04c2cc3ba Address:766d51d3-e734-e5b4-bc24-a2f04c2cc3ba} {Suffrage:Voter ID:server-573a809a-1a6e-7dd2-657e-add2205b8c4f Address:573a809a-1a6e-7dd2-657e-add2205b8c4f}]" +2025-10-30T18:21:25.540Z [INFO] server-766d51d3-e734-e5b4-bc24-a2f04c2cc3ba: entering follower state: follower="Node at 766d51d3-e734-e5b4-bc24-a2f04c2cc3ba [Follower]" leader-address= leader-id= +2025-10-30T18:21:25.540Z [INFO] server-573a809a-1a6e-7dd2-657e-add2205b8c4f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb7813bd-a1d9-56d8-7a4f-a38b220475f1 Address:bb7813bd-a1d9-56d8-7a4f-a38b220475f1} {Suffrage:Voter ID:server-766d51d3-e734-e5b4-bc24-a2f04c2cc3ba Address:766d51d3-e734-e5b4-bc24-a2f04c2cc3ba} {Suffrage:Voter ID:server-573a809a-1a6e-7dd2-657e-add2205b8c4f Address:573a809a-1a6e-7dd2-657e-add2205b8c4f}]" +2025-10-30T18:21:25.540Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:21:25.540Z [INFO] server-573a809a-1a6e-7dd2-657e-add2205b8c4f: entering follower state: follower="Node at 573a809a-1a6e-7dd2-657e-add2205b8c4f [Follower]" leader-address= leader-id= +2025-10-30T18:21:25.597Z [WARN] server-766d51d3-e734-e5b4-bc24-a2f04c2cc3ba: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:25.597Z [INFO] server-766d51d3-e734-e5b4-bc24-a2f04c2cc3ba: entering candidate state: node="Node at 766d51d3-e734-e5b4-bc24-a2f04c2cc3ba [Candidate]" term=2 +2025-10-30T18:21:25.597Z [INFO] server-766d51d3-e734-e5b4-bc24-a2f04c2cc3ba: election won: term=2 tally=2 +2025-10-30T18:21:25.597Z [INFO] server-766d51d3-e734-e5b4-bc24-a2f04c2cc3ba: entering leader state: leader="Node at 766d51d3-e734-e5b4-bc24-a2f04c2cc3ba [Leader]" +2025-10-30T18:21:25.597Z [INFO] server-766d51d3-e734-e5b4-bc24-a2f04c2cc3ba: added peer, starting replication: peer=server-bb7813bd-a1d9-56d8-7a4f-a38b220475f1 +2025-10-30T18:21:25.597Z [INFO] server-766d51d3-e734-e5b4-bc24-a2f04c2cc3ba: added peer, starting replication: peer=server-573a809a-1a6e-7dd2-657e-add2205b8c4f +2025-10-30T18:21:25.597Z [INFO] server-766d51d3-e734-e5b4-bc24-a2f04c2cc3ba: pipelining replication: peer="{Voter server-bb7813bd-a1d9-56d8-7a4f-a38b220475f1 bb7813bd-a1d9-56d8-7a4f-a38b220475f1}" +2025-10-30T18:21:25.597Z [INFO] server-766d51d3-e734-e5b4-bc24-a2f04c2cc3ba: pipelining replication: peer="{Voter server-573a809a-1a6e-7dd2-657e-add2205b8c4f 573a809a-1a6e-7dd2-657e-add2205b8c4f}" +2025-10-30T18:21:25.703Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:21:25.597634167 +0000 UTC m=+8.671163496 (2 nodes), 57.358717ms from start of poll, 57.732484ms from cluster start. Timeout at 2025-10-30 18:21:25.702634829 +0000 UTC m=+8.776164121, 105.000625ms after stability +2025-10-30T18:21:25.703Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:25.808Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:25.70313993 +0000 UTC m=+8.776669218 (1 nodes), 503ns from start of poll, 163.238206ms from cluster start. Timeout at 2025-10-30 18:21:25.808144447 +0000 UTC m=+8.881673739, 105.004521ms after stability +2025-10-30T18:21:25.819Z [INFO] server-766d51d3-e734-e5b4-bc24-a2f04c2cc3ba: aborting pipeline replication: peer="{Voter server-573a809a-1a6e-7dd2-657e-add2205b8c4f 573a809a-1a6e-7dd2-657e-add2205b8c4f}" +2025-10-30T18:21:25.819Z [INFO] server-766d51d3-e734-e5b4-bc24-a2f04c2cc3ba: aborting pipeline replication: peer="{Voter server-bb7813bd-a1d9-56d8-7a4f-a38b220475f1 bb7813bd-a1d9-56d8-7a4f-a38b220475f1}" +2025-10-30T18:21:26.319Z [ERROR] server-766d51d3-e734-e5b4-bc24-a2f04c2cc3ba: failed to heartbeat to: peer=bb7813bd-a1d9-56d8-7a4f-a38b220475f1 backoff time=10ms error="command timed out" +--- PASS: TestRaft_TripleNode (0.78s) === RUN TestRaft_LeaderFail -2025-10-30T18:17:40.409Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72284db2-fe2e-3801-8490-c7d6b8c15771 Address:72284db2-fe2e-3801-8490-c7d6b8c15771} {Suffrage:Voter ID:server-7d247416-3214-8134-dd1a-65dc0c50b23b Address:7d247416-3214-8134-dd1a-65dc0c50b23b} {Suffrage:Voter ID:server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b Address:cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}]" -2025-10-30T18:17:40.409Z [INFO] server-7d247416-3214-8134-dd1a-65dc0c50b23b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72284db2-fe2e-3801-8490-c7d6b8c15771 Address:72284db2-fe2e-3801-8490-c7d6b8c15771} {Suffrage:Voter ID:server-7d247416-3214-8134-dd1a-65dc0c50b23b Address:7d247416-3214-8134-dd1a-65dc0c50b23b} {Suffrage:Voter ID:server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b Address:cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}]" -2025-10-30T18:17:40.409Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: entering follower state: follower="Node at 72284db2-fe2e-3801-8490-c7d6b8c15771 [Follower]" leader-address= leader-id= -2025-10-30T18:17:40.409Z [INFO] server-7d247416-3214-8134-dd1a-65dc0c50b23b: entering follower state: follower="Node at 7d247416-3214-8134-dd1a-65dc0c50b23b [Follower]" leader-address= leader-id= -2025-10-30T18:17:40.409Z [INFO] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72284db2-fe2e-3801-8490-c7d6b8c15771 Address:72284db2-fe2e-3801-8490-c7d6b8c15771} {Suffrage:Voter ID:server-7d247416-3214-8134-dd1a-65dc0c50b23b Address:7d247416-3214-8134-dd1a-65dc0c50b23b} {Suffrage:Voter ID:server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b Address:cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}]" -2025-10-30T18:17:40.409Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:17:40.409Z [INFO] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering follower state: follower="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Follower]" leader-address= leader-id= -2025-10-30T18:17:40.492Z [WARN] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:40.492Z [INFO] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering candidate state: node="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Candidate]" term=2 -2025-10-30T18:17:40.492Z [INFO] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: election won: term=2 tally=2 -2025-10-30T18:17:40.492Z [INFO] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering leader state: leader="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Leader]" -2025-10-30T18:17:40.492Z [INFO] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: added peer, starting replication: peer=server-72284db2-fe2e-3801-8490-c7d6b8c15771 -2025-10-30T18:17:40.492Z [INFO] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: added peer, starting replication: peer=server-7d247416-3214-8134-dd1a-65dc0c50b23b -2025-10-30T18:17:40.492Z [INFO] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: pipelining replication: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" -2025-10-30T18:17:40.492Z [INFO] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: pipelining replication: peer="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" -2025-10-30T18:17:40.598Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:17:40.492584101 +0000 UTC m=+6.771258010 (2 nodes), 82.806213ms from start of poll, 83.078196ms from cluster start. Timeout at 2025-10-30 18:17:40.597587434 +0000 UTC m=+6.876261332, 105.003322ms after stability -2025-10-30T18:17:40.598Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:40.703Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:40.598126151 +0000 UTC m=+6.876800061 (1 nodes), 1.942µs from start of poll, 188.620247ms from cluster start. Timeout at 2025-10-30 18:17:40.703131699 +0000 UTC m=+6.981805599, 105.005538ms after stability - raft_test.go:378: [INFO] Disconnecting Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Leader] -2025-10-30T18:17:40.723Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to pipeline appendEntries: peer="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" error="append pipeline closed" -2025-10-30T18:17:40.723Z [INFO] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: aborting pipeline replication: peer="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" -2025-10-30T18:17:40.723Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to pipeline appendEntries: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" error="append pipeline closed" -2025-10-30T18:17:40.723Z [INFO] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: aborting pipeline replication: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" -2025-10-30T18:17:40.724Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to heartbeat to: peer=7d247416-3214-8134-dd1a-65dc0c50b23b backoff time=10ms error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b" -2025-10-30T18:17:40.724Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:40.727Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to heartbeat to: peer=72284db2-fe2e-3801-8490-c7d6b8c15771 backoff time=10ms error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771" -2025-10-30T18:17:40.731Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to appendEntries to: peer="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771" -2025-10-30T18:17:40.733Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to appendEntries to: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b" -2025-10-30T18:17:40.743Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to heartbeat to: peer=72284db2-fe2e-3801-8490-c7d6b8c15771 backoff time=10ms error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771" -2025-10-30T18:17:40.744Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to heartbeat to: peer=7d247416-3214-8134-dd1a-65dc0c50b23b backoff time=10ms error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b" -2025-10-30T18:17:40.751Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to appendEntries to: peer="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771" -2025-10-30T18:17:40.752Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to appendEntries to: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b" -2025-10-30T18:17:40.764Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to heartbeat to: peer=7d247416-3214-8134-dd1a-65dc0c50b23b backoff time=10ms error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b" -2025-10-30T18:17:40.764Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to heartbeat to: peer=72284db2-fe2e-3801-8490-c7d6b8c15771 backoff time=10ms error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771" -2025-10-30T18:17:40.765Z [WARN] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to contact: server-id=server-7d247416-3214-8134-dd1a-65dc0c50b23b time=50.65026ms -2025-10-30T18:17:40.768Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to appendEntries to: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b" -2025-10-30T18:17:40.768Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to appendEntries to: peer="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771" -2025-10-30T18:17:40.776Z [WARN] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to contact: server-id=server-72284db2-fe2e-3801-8490-c7d6b8c15771 time=58.507745ms -2025-10-30T18:17:40.776Z [WARN] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to contact: server-id=server-7d247416-3214-8134-dd1a-65dc0c50b23b time=61.601075ms -2025-10-30T18:17:40.776Z [WARN] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to contact quorum of nodes, stepping down -2025-10-30T18:17:40.776Z [INFO] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering follower state: follower="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Follower]" leader-address= leader-id= -2025-10-30T18:17:40.781Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to heartbeat to: peer=7d247416-3214-8134-dd1a-65dc0c50b23b backoff time=20ms error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b" -2025-10-30T18:17:40.783Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to heartbeat to: peer=72284db2-fe2e-3801-8490-c7d6b8c15771 backoff time=20ms error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771" -2025-10-30T18:17:40.795Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to appendEntries to: peer="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771" -2025-10-30T18:17:40.795Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to appendEntries to: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b" -2025-10-30T18:17:40.808Z [WARN] server-72284db2-fe2e-3801-8490-c7d6b8c15771: heartbeat timeout reached, starting election: last-leader-addr=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b last-leader-id=server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b -2025-10-30T18:17:40.808Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: entering candidate state: node="Node at 72284db2-fe2e-3801-8490-c7d6b8c15771 [Candidate]" term=3 -2025-10-30T18:17:40.808Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to make requestVote RPC: target="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b" term=3 -2025-10-30T18:17:40.808Z [WARN] server-7d247416-3214-8134-dd1a-65dc0c50b23b: rejecting vote request since we have a leader: from=72284db2-fe2e-3801-8490-c7d6b8c15771 leader=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b leader-id=server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b -2025-10-30T18:17:40.834Z [WARN] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:40.834Z [INFO] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering candidate state: node="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Candidate]" term=3 -2025-10-30T18:17:40.834Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to make requestVote RPC: target="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b" term=3 -2025-10-30T18:17:40.834Z [WARN] server-7d247416-3214-8134-dd1a-65dc0c50b23b: heartbeat timeout reached, starting election: last-leader-addr=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b last-leader-id=server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b -2025-10-30T18:17:40.834Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to make requestVote RPC: target="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771" term=3 -2025-10-30T18:17:40.834Z [INFO] server-7d247416-3214-8134-dd1a-65dc0c50b23b: entering candidate state: node="Node at 7d247416-3214-8134-dd1a-65dc0c50b23b [Candidate]" term=3 -2025-10-30T18:17:40.834Z [ERROR] server-7d247416-3214-8134-dd1a-65dc0c50b23b: failed to make requestVote RPC: target="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b" term=3 -2025-10-30T18:17:40.834Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: duplicate requestVote for same term: term=3 -2025-10-30T18:17:40.896Z [WARN] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: Election timeout reached, restarting election -2025-10-30T18:17:40.896Z [INFO] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering candidate state: node="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Candidate]" term=4 -2025-10-30T18:17:40.896Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to make requestVote RPC: target="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b" term=4 -2025-10-30T18:17:40.896Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to make requestVote RPC: target="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771" term=4 -2025-10-30T18:17:40.907Z [WARN] server-72284db2-fe2e-3801-8490-c7d6b8c15771: Election timeout reached, restarting election -2025-10-30T18:17:40.907Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: entering candidate state: node="Node at 72284db2-fe2e-3801-8490-c7d6b8c15771 [Candidate]" term=4 -2025-10-30T18:17:40.907Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to make requestVote RPC: target="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b" term=4 -2025-10-30T18:17:40.907Z [INFO] server-7d247416-3214-8134-dd1a-65dc0c50b23b: entering follower state: follower="Node at 7d247416-3214-8134-dd1a-65dc0c50b23b [Follower]" leader-address= leader-id= -2025-10-30T18:17:40.907Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: election won: term=4 tally=2 -2025-10-30T18:17:40.907Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: entering leader state: leader="Node at 72284db2-fe2e-3801-8490-c7d6b8c15771 [Leader]" -2025-10-30T18:17:40.907Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: added peer, starting replication: peer=server-7d247416-3214-8134-dd1a-65dc0c50b23b -2025-10-30T18:17:40.907Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: added peer, starting replication: peer=server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b -2025-10-30T18:17:40.907Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to appendEntries to: peer="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b" -2025-10-30T18:17:40.907Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: pipelining replication: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" -2025-10-30T18:17:40.914Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to heartbeat to: peer=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b backoff time=10ms error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b" -2025-10-30T18:17:40.924Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to appendEntries to: peer="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b" -2025-10-30T18:17:40.933Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to heartbeat to: peer=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b backoff time=10ms error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b" -2025-10-30T18:17:40.943Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to appendEntries to: peer="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b" -2025-10-30T18:17:40.953Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to heartbeat to: peer=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b backoff time=10ms error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b" -2025-10-30T18:17:40.957Z [WARN] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to contact: server-id=server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b time=50.173984ms -2025-10-30T18:17:40.966Z [WARN] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: Election timeout reached, restarting election -2025-10-30T18:17:40.966Z [INFO] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering candidate state: node="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Candidate]" term=5 -2025-10-30T18:17:40.966Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to make requestVote RPC: target="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" error="failed to connect to peer: 7d247416-3214-8134-dd1a-65dc0c50b23b" term=5 -2025-10-30T18:17:40.966Z [ERROR] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to make requestVote RPC: target="{Voter server-72284db2-fe2e-3801-8490-c7d6b8c15771 72284db2-fe2e-3801-8490-c7d6b8c15771}" error="failed to connect to peer: 72284db2-fe2e-3801-8490-c7d6b8c15771" term=5 -2025-10-30T18:17:40.970Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to heartbeat to: peer=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b backoff time=20ms error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b" -2025-10-30T18:17:40.974Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to appendEntries to: peer="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b" -2025-10-30T18:17:40.996Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to heartbeat to: peer=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b backoff time=25ms error="failed to connect to peer: cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b" -2025-10-30T18:17:41.006Z [WARN] server-72284db2-fe2e-3801-8490-c7d6b8c15771: failed to contact: server-id=server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b time=99.043092ms -2025-10-30T18:17:41.013Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:40.907601833 +0000 UTC m=+7.186275730 (1 nodes), 182.800078ms from start of poll, 498.095916ms from cluster start. Timeout at 2025-10-30 18:17:41.01260239 +0000 UTC m=+7.291276288, 105.000558ms after stability - raft_test.go:413: [INFO] Reconnecting Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Candidate] -2025-10-30T18:17:41.022Z [ERROR] server-72284db2-fe2e-3801-8490-c7d6b8c15771: peer has newer term, stopping replication: peer="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" -2025-10-30T18:17:41.022Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: entering follower state: follower="Node at 72284db2-fe2e-3801-8490-c7d6b8c15771 [Follower]" leader-address= leader-id= -2025-10-30T18:17:41.022Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: aborting pipeline replication: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" -2025-10-30T18:17:41.066Z [WARN] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: Election timeout reached, restarting election -2025-10-30T18:17:41.066Z [INFO] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering candidate state: node="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Candidate]" term=6 -2025-10-30T18:17:41.066Z [WARN] server-7d247416-3214-8134-dd1a-65dc0c50b23b: rejecting vote request since we have a leader: from=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b leader=72284db2-fe2e-3801-8490-c7d6b8c15771 leader-id=server-72284db2-fe2e-3801-8490-c7d6b8c15771 -2025-10-30T18:17:41.067Z [WARN] server-72284db2-fe2e-3801-8490-c7d6b8c15771: rejecting vote request since our last term is greater: candidate=cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b last-term=4 last-candidate-term=2 -2025-10-30T18:17:41.091Z [WARN] server-72284db2-fe2e-3801-8490-c7d6b8c15771: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:41.091Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: entering candidate state: node="Node at 72284db2-fe2e-3801-8490-c7d6b8c15771 [Candidate]" term=7 -2025-10-30T18:17:41.091Z [INFO] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: entering follower state: follower="Node at cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b [Follower]" leader-address= leader-id= -2025-10-30T18:17:41.091Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: election won: term=7 tally=2 -2025-10-30T18:17:41.091Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: entering leader state: leader="Node at 72284db2-fe2e-3801-8490-c7d6b8c15771 [Leader]" -2025-10-30T18:17:41.091Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: added peer, starting replication: peer=server-7d247416-3214-8134-dd1a-65dc0c50b23b -2025-10-30T18:17:41.091Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: added peer, starting replication: peer=server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b -2025-10-30T18:17:41.092Z [WARN] server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b: failed to get previous log: previous-index=5 last-index=3 error="log not found" -2025-10-30T18:17:41.092Z [WARN] server-72284db2-fe2e-3801-8490-c7d6b8c15771: appendEntries rejected, sending older logs: peer="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" next=4 -2025-10-30T18:17:41.092Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: pipelining replication: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" -2025-10-30T18:17:41.092Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: pipelining replication: peer="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" -2025-10-30T18:17:41.097Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: aborting pipeline replication: peer="{Voter server-7d247416-3214-8134-dd1a-65dc0c50b23b 7d247416-3214-8134-dd1a-65dc0c50b23b}" -2025-10-30T18:17:41.097Z [INFO] server-72284db2-fe2e-3801-8490-c7d6b8c15771: aborting pipeline replication: peer="{Voter server-cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b cfd51d7b-0ba4-9e0f-1eb7-21ac6c04977b}" ---- PASS: TestRaft_LeaderFail (0.69s) +2025-10-30T18:21:26.320Z [INFO] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c6757aad-7cb8-fc3a-9523-207c22d836e2 Address:c6757aad-7cb8-fc3a-9523-207c22d836e2} {Suffrage:Voter ID:server-e2235e25-c9a5-3367-091e-66764b1ae646 Address:e2235e25-c9a5-3367-091e-66764b1ae646} {Suffrage:Voter ID:server-13801704-4d2a-dc39-e551-40722d1f0e42 Address:13801704-4d2a-dc39-e551-40722d1f0e42}]" +2025-10-30T18:21:26.320Z [INFO] server-e2235e25-c9a5-3367-091e-66764b1ae646: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c6757aad-7cb8-fc3a-9523-207c22d836e2 Address:c6757aad-7cb8-fc3a-9523-207c22d836e2} {Suffrage:Voter ID:server-e2235e25-c9a5-3367-091e-66764b1ae646 Address:e2235e25-c9a5-3367-091e-66764b1ae646} {Suffrage:Voter ID:server-13801704-4d2a-dc39-e551-40722d1f0e42 Address:13801704-4d2a-dc39-e551-40722d1f0e42}]" +2025-10-30T18:21:26.320Z [INFO] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: entering follower state: follower="Node at c6757aad-7cb8-fc3a-9523-207c22d836e2 [Follower]" leader-address= leader-id= +2025-10-30T18:21:26.321Z [INFO] server-13801704-4d2a-dc39-e551-40722d1f0e42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c6757aad-7cb8-fc3a-9523-207c22d836e2 Address:c6757aad-7cb8-fc3a-9523-207c22d836e2} {Suffrage:Voter ID:server-e2235e25-c9a5-3367-091e-66764b1ae646 Address:e2235e25-c9a5-3367-091e-66764b1ae646} {Suffrage:Voter ID:server-13801704-4d2a-dc39-e551-40722d1f0e42 Address:13801704-4d2a-dc39-e551-40722d1f0e42}]" +2025-10-30T18:21:26.321Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:21:26.320Z [INFO] server-e2235e25-c9a5-3367-091e-66764b1ae646: entering follower state: follower="Node at e2235e25-c9a5-3367-091e-66764b1ae646 [Follower]" leader-address= leader-id= +2025-10-30T18:21:26.321Z [INFO] server-13801704-4d2a-dc39-e551-40722d1f0e42: entering follower state: follower="Node at 13801704-4d2a-dc39-e551-40722d1f0e42 [Follower]" leader-address= leader-id= +2025-10-30T18:21:26.402Z [WARN] server-e2235e25-c9a5-3367-091e-66764b1ae646: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:26.402Z [WARN] server-13801704-4d2a-dc39-e551-40722d1f0e42: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:26.402Z [INFO] server-13801704-4d2a-dc39-e551-40722d1f0e42: entering candidate state: node="Node at 13801704-4d2a-dc39-e551-40722d1f0e42 [Candidate]" term=2 +2025-10-30T18:21:26.402Z [INFO] server-e2235e25-c9a5-3367-091e-66764b1ae646: entering candidate state: node="Node at e2235e25-c9a5-3367-091e-66764b1ae646 [Candidate]" term=2 +2025-10-30T18:21:26.402Z [INFO] server-e2235e25-c9a5-3367-091e-66764b1ae646: duplicate requestVote for same term: term=2 +2025-10-30T18:21:26.402Z [INFO] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: duplicate requestVote for same term: term=2 +2025-10-30T18:21:26.402Z [INFO] server-13801704-4d2a-dc39-e551-40722d1f0e42: duplicate requestVote for same term: term=2 +2025-10-30T18:21:26.402Z [INFO] server-13801704-4d2a-dc39-e551-40722d1f0e42: election won: term=2 tally=2 +2025-10-30T18:21:26.402Z [INFO] server-13801704-4d2a-dc39-e551-40722d1f0e42: entering leader state: leader="Node at 13801704-4d2a-dc39-e551-40722d1f0e42 [Leader]" +2025-10-30T18:21:26.402Z [INFO] server-13801704-4d2a-dc39-e551-40722d1f0e42: added peer, starting replication: peer=server-c6757aad-7cb8-fc3a-9523-207c22d836e2 +2025-10-30T18:21:26.402Z [INFO] server-13801704-4d2a-dc39-e551-40722d1f0e42: added peer, starting replication: peer=server-e2235e25-c9a5-3367-091e-66764b1ae646 +2025-10-30T18:21:26.402Z [INFO] server-e2235e25-c9a5-3367-091e-66764b1ae646: entering follower state: follower="Node at e2235e25-c9a5-3367-091e-66764b1ae646 [Follower]" leader-address=13801704-4d2a-dc39-e551-40722d1f0e42 leader-id=server-13801704-4d2a-dc39-e551-40722d1f0e42 +2025-10-30T18:21:26.402Z [INFO] server-13801704-4d2a-dc39-e551-40722d1f0e42: pipelining replication: peer="{Voter server-c6757aad-7cb8-fc3a-9523-207c22d836e2 c6757aad-7cb8-fc3a-9523-207c22d836e2}" +2025-10-30T18:21:26.402Z [INFO] server-13801704-4d2a-dc39-e551-40722d1f0e42: pipelining replication: peer="{Voter server-e2235e25-c9a5-3367-091e-66764b1ae646 e2235e25-c9a5-3367-091e-66764b1ae646}" +2025-10-30T18:21:26.508Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:21:26.402866567 +0000 UTC m=+9.476395871 (2 nodes), 81.806726ms from start of poll, 82.081463ms from cluster start. Timeout at 2025-10-30 18:21:26.507874622 +0000 UTC m=+9.581403968, 105.008097ms after stability +2025-10-30T18:21:26.508Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:26.613Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:26.508404769 +0000 UTC m=+9.581934061 (1 nodes), 1.024µs from start of poll, 187.619653ms from cluster start. Timeout at 2025-10-30 18:21:26.613409591 +0000 UTC m=+9.686938880, 105.004819ms after stability + raft_test.go:378: [INFO] Disconnecting Node at 13801704-4d2a-dc39-e551-40722d1f0e42 [Leader] +2025-10-30T18:21:26.625Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to heartbeat to: peer=c6757aad-7cb8-fc3a-9523-207c22d836e2 backoff time=10ms error="failed to connect to peer: c6757aad-7cb8-fc3a-9523-207c22d836e2" +2025-10-30T18:21:26.625Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:26.627Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to heartbeat to: peer=e2235e25-c9a5-3367-091e-66764b1ae646 backoff time=10ms error="failed to connect to peer: e2235e25-c9a5-3367-091e-66764b1ae646" +2025-10-30T18:21:26.627Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to pipeline appendEntries: peer="{Voter server-c6757aad-7cb8-fc3a-9523-207c22d836e2 c6757aad-7cb8-fc3a-9523-207c22d836e2}" error="append pipeline closed" +2025-10-30T18:21:26.627Z [INFO] server-13801704-4d2a-dc39-e551-40722d1f0e42: aborting pipeline replication: peer="{Voter server-c6757aad-7cb8-fc3a-9523-207c22d836e2 c6757aad-7cb8-fc3a-9523-207c22d836e2}" +2025-10-30T18:21:26.632Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to pipeline appendEntries: peer="{Voter server-e2235e25-c9a5-3367-091e-66764b1ae646 e2235e25-c9a5-3367-091e-66764b1ae646}" error="append pipeline closed" +2025-10-30T18:21:26.632Z [INFO] server-13801704-4d2a-dc39-e551-40722d1f0e42: aborting pipeline replication: peer="{Voter server-e2235e25-c9a5-3367-091e-66764b1ae646 e2235e25-c9a5-3367-091e-66764b1ae646}" +2025-10-30T18:21:26.634Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to appendEntries to: peer="{Voter server-c6757aad-7cb8-fc3a-9523-207c22d836e2 c6757aad-7cb8-fc3a-9523-207c22d836e2}" error="failed to connect to peer: c6757aad-7cb8-fc3a-9523-207c22d836e2" +2025-10-30T18:21:26.638Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to appendEntries to: peer="{Voter server-e2235e25-c9a5-3367-091e-66764b1ae646 e2235e25-c9a5-3367-091e-66764b1ae646}" error="failed to connect to peer: e2235e25-c9a5-3367-091e-66764b1ae646" +2025-10-30T18:21:26.644Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to heartbeat to: peer=e2235e25-c9a5-3367-091e-66764b1ae646 backoff time=10ms error="failed to connect to peer: e2235e25-c9a5-3367-091e-66764b1ae646" +2025-10-30T18:21:26.645Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to heartbeat to: peer=c6757aad-7cb8-fc3a-9523-207c22d836e2 backoff time=10ms error="failed to connect to peer: c6757aad-7cb8-fc3a-9523-207c22d836e2" +2025-10-30T18:21:26.654Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to appendEntries to: peer="{Voter server-c6757aad-7cb8-fc3a-9523-207c22d836e2 c6757aad-7cb8-fc3a-9523-207c22d836e2}" error="failed to connect to peer: c6757aad-7cb8-fc3a-9523-207c22d836e2" +2025-10-30T18:21:26.655Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to appendEntries to: peer="{Voter server-e2235e25-c9a5-3367-091e-66764b1ae646 e2235e25-c9a5-3367-091e-66764b1ae646}" error="failed to connect to peer: e2235e25-c9a5-3367-091e-66764b1ae646" +2025-10-30T18:21:26.661Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to heartbeat to: peer=e2235e25-c9a5-3367-091e-66764b1ae646 backoff time=10ms error="failed to connect to peer: e2235e25-c9a5-3367-091e-66764b1ae646" +2025-10-30T18:21:26.664Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to heartbeat to: peer=c6757aad-7cb8-fc3a-9523-207c22d836e2 backoff time=10ms error="failed to connect to peer: c6757aad-7cb8-fc3a-9523-207c22d836e2" +2025-10-30T18:21:26.671Z [WARN] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to contact: server-id=server-c6757aad-7cb8-fc3a-9523-207c22d836e2 time=50.656059ms +2025-10-30T18:21:26.672Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to appendEntries to: peer="{Voter server-c6757aad-7cb8-fc3a-9523-207c22d836e2 c6757aad-7cb8-fc3a-9523-207c22d836e2}" error="failed to connect to peer: c6757aad-7cb8-fc3a-9523-207c22d836e2" +2025-10-30T18:21:26.674Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to appendEntries to: peer="{Voter server-e2235e25-c9a5-3367-091e-66764b1ae646 e2235e25-c9a5-3367-091e-66764b1ae646}" error="failed to connect to peer: e2235e25-c9a5-3367-091e-66764b1ae646" +2025-10-30T18:21:26.678Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to heartbeat to: peer=e2235e25-c9a5-3367-091e-66764b1ae646 backoff time=20ms error="failed to connect to peer: e2235e25-c9a5-3367-091e-66764b1ae646" +2025-10-30T18:21:26.681Z [WARN] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to contact: server-id=server-c6757aad-7cb8-fc3a-9523-207c22d836e2 time=61.322658ms +2025-10-30T18:21:26.681Z [WARN] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to contact: server-id=server-e2235e25-c9a5-3367-091e-66764b1ae646 time=58.075141ms +2025-10-30T18:21:26.681Z [WARN] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to contact quorum of nodes, stepping down +2025-10-30T18:21:26.681Z [INFO] server-13801704-4d2a-dc39-e551-40722d1f0e42: entering follower state: follower="Node at 13801704-4d2a-dc39-e551-40722d1f0e42 [Follower]" leader-address= leader-id= +2025-10-30T18:21:26.683Z [WARN] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: heartbeat timeout reached, starting election: last-leader-addr=13801704-4d2a-dc39-e551-40722d1f0e42 last-leader-id=server-13801704-4d2a-dc39-e551-40722d1f0e42 +2025-10-30T18:21:26.683Z [INFO] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: entering candidate state: node="Node at c6757aad-7cb8-fc3a-9523-207c22d836e2 [Candidate]" term=3 +2025-10-30T18:21:26.684Z [ERROR] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: failed to make requestVote RPC: target="{Voter server-13801704-4d2a-dc39-e551-40722d1f0e42 13801704-4d2a-dc39-e551-40722d1f0e42}" error="failed to connect to peer: 13801704-4d2a-dc39-e551-40722d1f0e42" term=3 +2025-10-30T18:21:26.684Z [WARN] server-e2235e25-c9a5-3367-091e-66764b1ae646: rejecting vote request since we have a leader: from=c6757aad-7cb8-fc3a-9523-207c22d836e2 leader=13801704-4d2a-dc39-e551-40722d1f0e42 leader-id=server-13801704-4d2a-dc39-e551-40722d1f0e42 +2025-10-30T18:21:26.701Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to appendEntries to: peer="{Voter server-e2235e25-c9a5-3367-091e-66764b1ae646 e2235e25-c9a5-3367-091e-66764b1ae646}" error="failed to connect to peer: e2235e25-c9a5-3367-091e-66764b1ae646" +2025-10-30T18:21:26.711Z [WARN] server-e2235e25-c9a5-3367-091e-66764b1ae646: heartbeat timeout reached, starting election: last-leader-addr=13801704-4d2a-dc39-e551-40722d1f0e42 last-leader-id=server-13801704-4d2a-dc39-e551-40722d1f0e42 +2025-10-30T18:21:26.711Z [INFO] server-e2235e25-c9a5-3367-091e-66764b1ae646: entering candidate state: node="Node at e2235e25-c9a5-3367-091e-66764b1ae646 [Candidate]" term=3 +2025-10-30T18:21:26.711Z [ERROR] server-e2235e25-c9a5-3367-091e-66764b1ae646: failed to make requestVote RPC: target="{Voter server-13801704-4d2a-dc39-e551-40722d1f0e42 13801704-4d2a-dc39-e551-40722d1f0e42}" error="failed to connect to peer: 13801704-4d2a-dc39-e551-40722d1f0e42" term=3 +2025-10-30T18:21:26.711Z [INFO] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: duplicate requestVote for same term: term=3 +2025-10-30T18:21:26.747Z [WARN] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: Election timeout reached, restarting election +2025-10-30T18:21:26.747Z [INFO] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: entering candidate state: node="Node at c6757aad-7cb8-fc3a-9523-207c22d836e2 [Candidate]" term=4 +2025-10-30T18:21:26.747Z [ERROR] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: failed to make requestVote RPC: target="{Voter server-13801704-4d2a-dc39-e551-40722d1f0e42 13801704-4d2a-dc39-e551-40722d1f0e42}" error="failed to connect to peer: 13801704-4d2a-dc39-e551-40722d1f0e42" term=4 +2025-10-30T18:21:26.747Z [INFO] server-e2235e25-c9a5-3367-091e-66764b1ae646: entering follower state: follower="Node at e2235e25-c9a5-3367-091e-66764b1ae646 [Follower]" leader-address= leader-id= +2025-10-30T18:21:26.747Z [INFO] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: election won: term=4 tally=2 +2025-10-30T18:21:26.747Z [INFO] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: entering leader state: leader="Node at c6757aad-7cb8-fc3a-9523-207c22d836e2 [Leader]" +2025-10-30T18:21:26.747Z [INFO] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: added peer, starting replication: peer=server-e2235e25-c9a5-3367-091e-66764b1ae646 +2025-10-30T18:21:26.747Z [INFO] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: added peer, starting replication: peer=server-13801704-4d2a-dc39-e551-40722d1f0e42 +2025-10-30T18:21:26.747Z [ERROR] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: failed to appendEntries to: peer="{Voter server-13801704-4d2a-dc39-e551-40722d1f0e42 13801704-4d2a-dc39-e551-40722d1f0e42}" error="failed to connect to peer: 13801704-4d2a-dc39-e551-40722d1f0e42" +2025-10-30T18:21:26.747Z [INFO] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: pipelining replication: peer="{Voter server-e2235e25-c9a5-3367-091e-66764b1ae646 e2235e25-c9a5-3367-091e-66764b1ae646}" +2025-10-30T18:21:26.755Z [ERROR] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: failed to heartbeat to: peer=13801704-4d2a-dc39-e551-40722d1f0e42 backoff time=10ms error="failed to connect to peer: 13801704-4d2a-dc39-e551-40722d1f0e42" +2025-10-30T18:21:26.766Z [ERROR] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: failed to appendEntries to: peer="{Voter server-13801704-4d2a-dc39-e551-40722d1f0e42 13801704-4d2a-dc39-e551-40722d1f0e42}" error="failed to connect to peer: 13801704-4d2a-dc39-e551-40722d1f0e42" +2025-10-30T18:21:26.773Z [WARN] server-13801704-4d2a-dc39-e551-40722d1f0e42: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:26.773Z [INFO] server-13801704-4d2a-dc39-e551-40722d1f0e42: entering candidate state: node="Node at 13801704-4d2a-dc39-e551-40722d1f0e42 [Candidate]" term=3 +2025-10-30T18:21:26.773Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to make requestVote RPC: target="{Voter server-e2235e25-c9a5-3367-091e-66764b1ae646 e2235e25-c9a5-3367-091e-66764b1ae646}" error="failed to connect to peer: e2235e25-c9a5-3367-091e-66764b1ae646" term=3 +2025-10-30T18:21:26.773Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to make requestVote RPC: target="{Voter server-c6757aad-7cb8-fc3a-9523-207c22d836e2 c6757aad-7cb8-fc3a-9523-207c22d836e2}" error="failed to connect to peer: c6757aad-7cb8-fc3a-9523-207c22d836e2" term=3 +2025-10-30T18:21:26.774Z [ERROR] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: failed to heartbeat to: peer=13801704-4d2a-dc39-e551-40722d1f0e42 backoff time=10ms error="failed to connect to peer: 13801704-4d2a-dc39-e551-40722d1f0e42" +2025-10-30T18:21:26.786Z [ERROR] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: failed to appendEntries to: peer="{Voter server-13801704-4d2a-dc39-e551-40722d1f0e42 13801704-4d2a-dc39-e551-40722d1f0e42}" error="failed to connect to peer: 13801704-4d2a-dc39-e551-40722d1f0e42" +2025-10-30T18:21:26.794Z [ERROR] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: failed to heartbeat to: peer=13801704-4d2a-dc39-e551-40722d1f0e42 backoff time=10ms error="failed to connect to peer: 13801704-4d2a-dc39-e551-40722d1f0e42" +2025-10-30T18:21:26.798Z [WARN] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: failed to contact: server-id=server-13801704-4d2a-dc39-e551-40722d1f0e42 time=50.770359ms +2025-10-30T18:21:26.814Z [ERROR] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: failed to heartbeat to: peer=13801704-4d2a-dc39-e551-40722d1f0e42 backoff time=20ms error="failed to connect to peer: 13801704-4d2a-dc39-e551-40722d1f0e42" +2025-10-30T18:21:26.816Z [ERROR] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: failed to appendEntries to: peer="{Voter server-13801704-4d2a-dc39-e551-40722d1f0e42 13801704-4d2a-dc39-e551-40722d1f0e42}" error="failed to connect to peer: 13801704-4d2a-dc39-e551-40722d1f0e42" +2025-10-30T18:21:26.842Z [ERROR] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: failed to heartbeat to: peer=13801704-4d2a-dc39-e551-40722d1f0e42 backoff time=25ms error="failed to connect to peer: 13801704-4d2a-dc39-e551-40722d1f0e42" +2025-10-30T18:21:26.847Z [WARN] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: failed to contact: server-id=server-13801704-4d2a-dc39-e551-40722d1f0e42 time=100.271475ms +2025-10-30T18:21:26.857Z [WARN] server-13801704-4d2a-dc39-e551-40722d1f0e42: Election timeout reached, restarting election +2025-10-30T18:21:26.857Z [INFO] server-13801704-4d2a-dc39-e551-40722d1f0e42: entering candidate state: node="Node at 13801704-4d2a-dc39-e551-40722d1f0e42 [Candidate]" term=4 +2025-10-30T18:21:26.857Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to make requestVote RPC: target="{Voter server-e2235e25-c9a5-3367-091e-66764b1ae646 e2235e25-c9a5-3367-091e-66764b1ae646}" error="failed to connect to peer: e2235e25-c9a5-3367-091e-66764b1ae646" term=4 +2025-10-30T18:21:26.857Z [ERROR] server-13801704-4d2a-dc39-e551-40722d1f0e42: failed to make requestVote RPC: target="{Voter server-c6757aad-7cb8-fc3a-9523-207c22d836e2 c6757aad-7cb8-fc3a-9523-207c22d836e2}" error="failed to connect to peer: c6757aad-7cb8-fc3a-9523-207c22d836e2" term=4 +2025-10-30T18:21:26.865Z [ERROR] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: failed to appendEntries to: peer="{Voter server-13801704-4d2a-dc39-e551-40722d1f0e42 13801704-4d2a-dc39-e551-40722d1f0e42}" error="failed to connect to peer: 13801704-4d2a-dc39-e551-40722d1f0e42" +2025-10-30T18:21:26.873Z [ERROR] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: failed to heartbeat to: peer=13801704-4d2a-dc39-e551-40722d1f0e42 backoff time=25ms error="failed to connect to peer: 13801704-4d2a-dc39-e551-40722d1f0e42" +2025-10-30T18:21:26.879Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:26.773712127 +0000 UTC m=+9.847241416 (1 nodes), 147.723623ms from start of poll, 452.927008ms from cluster start. Timeout at 2025-10-30 18:21:26.878714318 +0000 UTC m=+9.952243611, 105.002195ms after stability + raft_test.go:413: [INFO] Reconnecting Node at 13801704-4d2a-dc39-e551-40722d1f0e42 [Candidate] +2025-10-30T18:21:26.906Z [INFO] server-13801704-4d2a-dc39-e551-40722d1f0e42: entering follower state: follower="Node at 13801704-4d2a-dc39-e551-40722d1f0e42 [Follower]" leader-address=c6757aad-7cb8-fc3a-9523-207c22d836e2 leader-id=server-c6757aad-7cb8-fc3a-9523-207c22d836e2 +2025-10-30T18:21:26.956Z [INFO] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: pipelining replication: peer="{Voter server-13801704-4d2a-dc39-e551-40722d1f0e42 13801704-4d2a-dc39-e551-40722d1f0e42}" +2025-10-30T18:21:26.960Z [INFO] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: aborting pipeline replication: peer="{Voter server-13801704-4d2a-dc39-e551-40722d1f0e42 13801704-4d2a-dc39-e551-40722d1f0e42}" +2025-10-30T18:21:26.960Z [INFO] server-c6757aad-7cb8-fc3a-9523-207c22d836e2: aborting pipeline replication: peer="{Voter server-e2235e25-c9a5-3367-091e-66764b1ae646 e2235e25-c9a5-3367-091e-66764b1ae646}" +--- PASS: TestRaft_LeaderFail (0.64s) === RUN TestRaft_BehindFollower -2025-10-30T18:17:41.098Z [INFO] server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 Address:821f7b28-91bf-3de7-6f56-2f94f9ddf0e5} {Suffrage:Voter ID:server-9ffd1b46-b571-6cad-22bc-9fb55c496434 Address:9ffd1b46-b571-6cad-22bc-9fb55c496434} {Suffrage:Voter ID:server-1e38d4d5-371c-cbd6-430d-abb6f87207aa Address:1e38d4d5-371c-cbd6-430d-abb6f87207aa}]" -2025-10-30T18:17:41.098Z [INFO] server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5: entering follower state: follower="Node at 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 [Follower]" leader-address= leader-id= -2025-10-30T18:17:41.098Z [INFO] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 Address:821f7b28-91bf-3de7-6f56-2f94f9ddf0e5} {Suffrage:Voter ID:server-9ffd1b46-b571-6cad-22bc-9fb55c496434 Address:9ffd1b46-b571-6cad-22bc-9fb55c496434} {Suffrage:Voter ID:server-1e38d4d5-371c-cbd6-430d-abb6f87207aa Address:1e38d4d5-371c-cbd6-430d-abb6f87207aa}]" -2025-10-30T18:17:41.098Z [INFO] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: entering follower state: follower="Node at 9ffd1b46-b571-6cad-22bc-9fb55c496434 [Follower]" leader-address= leader-id= -2025-10-30T18:17:41.098Z [INFO] server-1e38d4d5-371c-cbd6-430d-abb6f87207aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 Address:821f7b28-91bf-3de7-6f56-2f94f9ddf0e5} {Suffrage:Voter ID:server-9ffd1b46-b571-6cad-22bc-9fb55c496434 Address:9ffd1b46-b571-6cad-22bc-9fb55c496434} {Suffrage:Voter ID:server-1e38d4d5-371c-cbd6-430d-abb6f87207aa Address:1e38d4d5-371c-cbd6-430d-abb6f87207aa}]" -2025-10-30T18:17:41.098Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:41.098Z [INFO] server-1e38d4d5-371c-cbd6-430d-abb6f87207aa: entering follower state: follower="Node at 1e38d4d5-371c-cbd6-430d-abb6f87207aa [Follower]" leader-address= leader-id= -2025-10-30T18:17:41.151Z [WARN] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:41.151Z [INFO] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: entering candidate state: node="Node at 9ffd1b46-b571-6cad-22bc-9fb55c496434 [Candidate]" term=2 -2025-10-30T18:17:41.151Z [INFO] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: election won: term=2 tally=2 -2025-10-30T18:17:41.151Z [INFO] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: entering leader state: leader="Node at 9ffd1b46-b571-6cad-22bc-9fb55c496434 [Leader]" -2025-10-30T18:17:41.151Z [INFO] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: added peer, starting replication: peer=server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 -2025-10-30T18:17:41.151Z [INFO] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: added peer, starting replication: peer=server-1e38d4d5-371c-cbd6-430d-abb6f87207aa -2025-10-30T18:17:41.151Z [INFO] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: pipelining replication: peer="{Voter server-1e38d4d5-371c-cbd6-430d-abb6f87207aa 1e38d4d5-371c-cbd6-430d-abb6f87207aa}" -2025-10-30T18:17:41.151Z [INFO] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: pipelining replication: peer="{Voter server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5}" -2025-10-30T18:17:41.257Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:41.151757967 +0000 UTC m=+7.430431864 (1 nodes), 53.178225ms from start of poll, 53.491248ms from cluster start. Timeout at 2025-10-30 18:17:41.256760661 +0000 UTC m=+7.535434559, 105.002695ms after stability -2025-10-30T18:17:41.257Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:17:41.363Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:17:41.257735638 +0000 UTC m=+7.536409537 (2 nodes), 1.464µs from start of poll, 159.468921ms from cluster start. Timeout at 2025-10-30 18:17:41.362745368 +0000 UTC m=+7.641419336, 105.009799ms after stability -2025-10-30T18:17:41.363Z [ERROR] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: failed to pipeline appendEntries: peer="{Voter server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5}" error="append pipeline closed" -2025-10-30T18:17:41.363Z [INFO] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: aborting pipeline replication: peer="{Voter server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5}" -2025-10-30T18:17:41.363Z [ERROR] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: failed to appendEntries to: peer="{Voter server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5}" error="failed to connect to peer: 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5" -2025-10-30T18:17:41.365Z [ERROR] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: failed to heartbeat to: peer=821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 backoff time=10ms error="failed to connect to peer: 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5" +2025-10-30T18:21:26.961Z [INFO] server-abf86516-d756-54fc-5ce3-3c49accd3d99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abf86516-d756-54fc-5ce3-3c49accd3d99 Address:abf86516-d756-54fc-5ce3-3c49accd3d99} {Suffrage:Voter ID:server-87396456-90f2-b71a-1b24-bd38019fa44f Address:87396456-90f2-b71a-1b24-bd38019fa44f} {Suffrage:Voter ID:server-c33af5af-9400-29a2-2d58-65b84cdbeb35 Address:c33af5af-9400-29a2-2d58-65b84cdbeb35}]" +2025-10-30T18:21:26.961Z [INFO] server-abf86516-d756-54fc-5ce3-3c49accd3d99: entering follower state: follower="Node at abf86516-d756-54fc-5ce3-3c49accd3d99 [Follower]" leader-address= leader-id= +2025-10-30T18:21:26.961Z [INFO] server-87396456-90f2-b71a-1b24-bd38019fa44f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abf86516-d756-54fc-5ce3-3c49accd3d99 Address:abf86516-d756-54fc-5ce3-3c49accd3d99} {Suffrage:Voter ID:server-87396456-90f2-b71a-1b24-bd38019fa44f Address:87396456-90f2-b71a-1b24-bd38019fa44f} {Suffrage:Voter ID:server-c33af5af-9400-29a2-2d58-65b84cdbeb35 Address:c33af5af-9400-29a2-2d58-65b84cdbeb35}]" +2025-10-30T18:21:26.961Z [INFO] server-87396456-90f2-b71a-1b24-bd38019fa44f: entering follower state: follower="Node at 87396456-90f2-b71a-1b24-bd38019fa44f [Follower]" leader-address= leader-id= +2025-10-30T18:21:26.961Z [INFO] server-c33af5af-9400-29a2-2d58-65b84cdbeb35: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abf86516-d756-54fc-5ce3-3c49accd3d99 Address:abf86516-d756-54fc-5ce3-3c49accd3d99} {Suffrage:Voter ID:server-87396456-90f2-b71a-1b24-bd38019fa44f Address:87396456-90f2-b71a-1b24-bd38019fa44f} {Suffrage:Voter ID:server-c33af5af-9400-29a2-2d58-65b84cdbeb35 Address:c33af5af-9400-29a2-2d58-65b84cdbeb35}]" +2025-10-30T18:21:26.961Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:26.961Z [INFO] server-c33af5af-9400-29a2-2d58-65b84cdbeb35: entering follower state: follower="Node at c33af5af-9400-29a2-2d58-65b84cdbeb35 [Follower]" leader-address= leader-id= +2025-10-30T18:21:27.038Z [WARN] server-abf86516-d756-54fc-5ce3-3c49accd3d99: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:27.038Z [INFO] server-abf86516-d756-54fc-5ce3-3c49accd3d99: entering candidate state: node="Node at abf86516-d756-54fc-5ce3-3c49accd3d99 [Candidate]" term=2 +2025-10-30T18:21:27.038Z [INFO] server-abf86516-d756-54fc-5ce3-3c49accd3d99: election won: term=2 tally=2 +2025-10-30T18:21:27.038Z [INFO] server-abf86516-d756-54fc-5ce3-3c49accd3d99: entering leader state: leader="Node at abf86516-d756-54fc-5ce3-3c49accd3d99 [Leader]" +2025-10-30T18:21:27.038Z [INFO] server-abf86516-d756-54fc-5ce3-3c49accd3d99: added peer, starting replication: peer=server-87396456-90f2-b71a-1b24-bd38019fa44f +2025-10-30T18:21:27.038Z [INFO] server-abf86516-d756-54fc-5ce3-3c49accd3d99: added peer, starting replication: peer=server-c33af5af-9400-29a2-2d58-65b84cdbeb35 +2025-10-30T18:21:27.038Z [INFO] server-abf86516-d756-54fc-5ce3-3c49accd3d99: pipelining replication: peer="{Voter server-c33af5af-9400-29a2-2d58-65b84cdbeb35 c33af5af-9400-29a2-2d58-65b84cdbeb35}" +2025-10-30T18:21:27.038Z [INFO] server-abf86516-d756-54fc-5ce3-3c49accd3d99: pipelining replication: peer="{Voter server-87396456-90f2-b71a-1b24-bd38019fa44f 87396456-90f2-b71a-1b24-bd38019fa44f}" +2025-10-30T18:21:27.143Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:27.03835626 +0000 UTC m=+10.111885566 (1 nodes), 76.381648ms from start of poll, 76.704963ms from cluster start. Timeout at 2025-10-30 18:21:27.143358266 +0000 UTC m=+10.216887559, 105.001993ms after stability +2025-10-30T18:21:27.143Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:21:27.249Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:21:27.143463103 +0000 UTC m=+10.216992409 (2 nodes), 1.223µs from start of poll, 181.811806ms from cluster start. Timeout at 2025-10-30 18:21:27.248469742 +0000 UTC m=+10.321999036, 105.006627ms after stability +2025-10-30T18:21:27.249Z [ERROR] server-abf86516-d756-54fc-5ce3-3c49accd3d99: failed to pipeline appendEntries: peer="{Voter server-87396456-90f2-b71a-1b24-bd38019fa44f 87396456-90f2-b71a-1b24-bd38019fa44f}" error="append pipeline closed" +2025-10-30T18:21:27.249Z [INFO] server-abf86516-d756-54fc-5ce3-3c49accd3d99: aborting pipeline replication: peer="{Voter server-87396456-90f2-b71a-1b24-bd38019fa44f 87396456-90f2-b71a-1b24-bd38019fa44f}" +2025-10-30T18:21:27.249Z [ERROR] server-abf86516-d756-54fc-5ce3-3c49accd3d99: failed to appendEntries to: peer="{Voter server-87396456-90f2-b71a-1b24-bd38019fa44f 87396456-90f2-b71a-1b24-bd38019fa44f}" error="failed to connect to peer: 87396456-90f2-b71a-1b24-bd38019fa44f" +2025-10-30T18:21:27.250Z [ERROR] server-abf86516-d756-54fc-5ce3-3c49accd3d99: failed to heartbeat to: peer=87396456-90f2-b71a-1b24-bd38019fa44f backoff time=10ms error="failed to connect to peer: 87396456-90f2-b71a-1b24-bd38019fa44f" raft_test.go:459: [INFO] Finished apply without behind follower -2025-10-30T18:17:41.373Z [INFO] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: pipelining replication: peer="{Voter server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5}" -2025-10-30T18:17:41.384Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:41.490Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:41.384686542 +0000 UTC m=+7.663360439 (1 nodes), 542ns from start of poll, 286.419823ms from cluster start. Timeout at 2025-10-30 18:17:41.489690042 +0000 UTC m=+7.768363939, 105.0035ms after stability -2025-10-30T18:17:41.490Z [INFO] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: aborting pipeline replication: peer="{Voter server-821f7b28-91bf-3de7-6f56-2f94f9ddf0e5 821f7b28-91bf-3de7-6f56-2f94f9ddf0e5}" -2025-10-30T18:17:41.490Z [INFO] server-9ffd1b46-b571-6cad-22bc-9fb55c496434: aborting pipeline replication: peer="{Voter server-1e38d4d5-371c-cbd6-430d-abb6f87207aa 1e38d4d5-371c-cbd6-430d-abb6f87207aa}" ---- PASS: TestRaft_BehindFollower (0.39s) +2025-10-30T18:21:27.260Z [INFO] server-abf86516-d756-54fc-5ce3-3c49accd3d99: pipelining replication: peer="{Voter server-87396456-90f2-b71a-1b24-bd38019fa44f 87396456-90f2-b71a-1b24-bd38019fa44f}" +2025-10-30T18:21:27.271Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:27.376Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:27.271597429 +0000 UTC m=+10.345126724 (1 nodes), 715ns from start of poll, 309.946121ms from cluster start. Timeout at 2025-10-30 18:21:27.37660401 +0000 UTC m=+10.450133303, 105.006579ms after stability +2025-10-30T18:21:27.376Z [INFO] server-abf86516-d756-54fc-5ce3-3c49accd3d99: aborting pipeline replication: peer="{Voter server-c33af5af-9400-29a2-2d58-65b84cdbeb35 c33af5af-9400-29a2-2d58-65b84cdbeb35}" +2025-10-30T18:21:27.376Z [INFO] server-abf86516-d756-54fc-5ce3-3c49accd3d99: aborting pipeline replication: peer="{Voter server-87396456-90f2-b71a-1b24-bd38019fa44f 87396456-90f2-b71a-1b24-bd38019fa44f}" +--- PASS: TestRaft_BehindFollower (0.42s) === RUN TestRaft_ApplyNonLeader -2025-10-30T18:17:41.491Z [INFO] server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-182d8643-b95e-ea6e-5c2c-fc19155a90ad Address:182d8643-b95e-ea6e-5c2c-fc19155a90ad} {Suffrage:Voter ID:server-d16cfb49-4715-64c3-a972-38aabb975406 Address:d16cfb49-4715-64c3-a972-38aabb975406} {Suffrage:Voter ID:server-4be31e34-d481-0c47-a717-9820c6d48a3e Address:4be31e34-d481-0c47-a717-9820c6d48a3e}]" -2025-10-30T18:17:41.491Z [INFO] server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: entering follower state: follower="Node at 182d8643-b95e-ea6e-5c2c-fc19155a90ad [Follower]" leader-address= leader-id= -2025-10-30T18:17:41.491Z [INFO] server-d16cfb49-4715-64c3-a972-38aabb975406: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-182d8643-b95e-ea6e-5c2c-fc19155a90ad Address:182d8643-b95e-ea6e-5c2c-fc19155a90ad} {Suffrage:Voter ID:server-d16cfb49-4715-64c3-a972-38aabb975406 Address:d16cfb49-4715-64c3-a972-38aabb975406} {Suffrage:Voter ID:server-4be31e34-d481-0c47-a717-9820c6d48a3e Address:4be31e34-d481-0c47-a717-9820c6d48a3e}]" -2025-10-30T18:17:41.491Z [INFO] server-d16cfb49-4715-64c3-a972-38aabb975406: entering follower state: follower="Node at d16cfb49-4715-64c3-a972-38aabb975406 [Follower]" leader-address= leader-id= -2025-10-30T18:17:41.491Z [INFO] server-4be31e34-d481-0c47-a717-9820c6d48a3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-182d8643-b95e-ea6e-5c2c-fc19155a90ad Address:182d8643-b95e-ea6e-5c2c-fc19155a90ad} {Suffrage:Voter ID:server-d16cfb49-4715-64c3-a972-38aabb975406 Address:d16cfb49-4715-64c3-a972-38aabb975406} {Suffrage:Voter ID:server-4be31e34-d481-0c47-a717-9820c6d48a3e Address:4be31e34-d481-0c47-a717-9820c6d48a3e}]" -2025-10-30T18:17:41.491Z [INFO] server-4be31e34-d481-0c47-a717-9820c6d48a3e: entering follower state: follower="Node at 4be31e34-d481-0c47-a717-9820c6d48a3e [Follower]" leader-address= leader-id= -2025-10-30T18:17:41.491Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:41.569Z [WARN] server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:41.569Z [INFO] server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: entering candidate state: node="Node at 182d8643-b95e-ea6e-5c2c-fc19155a90ad [Candidate]" term=2 -2025-10-30T18:17:41.569Z [INFO] server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: election won: term=2 tally=2 -2025-10-30T18:17:41.569Z [INFO] server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: entering leader state: leader="Node at 182d8643-b95e-ea6e-5c2c-fc19155a90ad [Leader]" -2025-10-30T18:17:41.569Z [INFO] server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: added peer, starting replication: peer=server-d16cfb49-4715-64c3-a972-38aabb975406 -2025-10-30T18:17:41.569Z [INFO] server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: added peer, starting replication: peer=server-4be31e34-d481-0c47-a717-9820c6d48a3e -2025-10-30T18:17:41.569Z [INFO] server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: pipelining replication: peer="{Voter server-4be31e34-d481-0c47-a717-9820c6d48a3e 4be31e34-d481-0c47-a717-9820c6d48a3e}" -2025-10-30T18:17:41.570Z [INFO] server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: pipelining replication: peer="{Voter server-d16cfb49-4715-64c3-a972-38aabb975406 d16cfb49-4715-64c3-a972-38aabb975406}" -2025-10-30T18:17:41.675Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:41.56996994 +0000 UTC m=+7.848643840 (1 nodes), 78.195768ms from start of poll, 78.99977ms from cluster start. Timeout at 2025-10-30 18:17:41.674972497 +0000 UTC m=+7.953646394, 105.002554ms after stability -2025-10-30T18:17:41.675Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:17:41.780Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:17:41.675234975 +0000 UTC m=+7.953908876 (2 nodes), 852ns from start of poll, 184.264806ms from cluster start. Timeout at 2025-10-30 18:17:41.780239191 +0000 UTC m=+8.058913104, 105.004228ms after stability -2025-10-30T18:17:41.780Z [INFO] server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: aborting pipeline replication: peer="{Voter server-4be31e34-d481-0c47-a717-9820c6d48a3e 4be31e34-d481-0c47-a717-9820c6d48a3e}" -2025-10-30T18:17:41.780Z [INFO] server-182d8643-b95e-ea6e-5c2c-fc19155a90ad: aborting pipeline replication: peer="{Voter server-d16cfb49-4715-64c3-a972-38aabb975406 d16cfb49-4715-64c3-a972-38aabb975406}" ---- PASS: TestRaft_ApplyNonLeader (0.29s) +2025-10-30T18:21:27.378Z [INFO] server-e4aaa6e4-f397-66d1-fa1f-d5efeb194d94: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e4aaa6e4-f397-66d1-fa1f-d5efeb194d94 Address:e4aaa6e4-f397-66d1-fa1f-d5efeb194d94} {Suffrage:Voter ID:server-9cd4e1bf-217b-0453-9d9c-59d59e880876 Address:9cd4e1bf-217b-0453-9d9c-59d59e880876} {Suffrage:Voter ID:server-3e820db2-47e1-2394-14d6-515190f39986 Address:3e820db2-47e1-2394-14d6-515190f39986}]" +2025-10-30T18:21:27.378Z [INFO] server-9cd4e1bf-217b-0453-9d9c-59d59e880876: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e4aaa6e4-f397-66d1-fa1f-d5efeb194d94 Address:e4aaa6e4-f397-66d1-fa1f-d5efeb194d94} {Suffrage:Voter ID:server-9cd4e1bf-217b-0453-9d9c-59d59e880876 Address:9cd4e1bf-217b-0453-9d9c-59d59e880876} {Suffrage:Voter ID:server-3e820db2-47e1-2394-14d6-515190f39986 Address:3e820db2-47e1-2394-14d6-515190f39986}]" +2025-10-30T18:21:27.378Z [INFO] server-e4aaa6e4-f397-66d1-fa1f-d5efeb194d94: entering follower state: follower="Node at e4aaa6e4-f397-66d1-fa1f-d5efeb194d94 [Follower]" leader-address= leader-id= +2025-10-30T18:21:27.378Z [INFO] server-9cd4e1bf-217b-0453-9d9c-59d59e880876: entering follower state: follower="Node at 9cd4e1bf-217b-0453-9d9c-59d59e880876 [Follower]" leader-address= leader-id= +2025-10-30T18:21:27.378Z [INFO] server-3e820db2-47e1-2394-14d6-515190f39986: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e4aaa6e4-f397-66d1-fa1f-d5efeb194d94 Address:e4aaa6e4-f397-66d1-fa1f-d5efeb194d94} {Suffrage:Voter ID:server-9cd4e1bf-217b-0453-9d9c-59d59e880876 Address:9cd4e1bf-217b-0453-9d9c-59d59e880876} {Suffrage:Voter ID:server-3e820db2-47e1-2394-14d6-515190f39986 Address:3e820db2-47e1-2394-14d6-515190f39986}]" +2025-10-30T18:21:27.378Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:27.378Z [INFO] server-3e820db2-47e1-2394-14d6-515190f39986: entering follower state: follower="Node at 3e820db2-47e1-2394-14d6-515190f39986 [Follower]" leader-address= leader-id= +2025-10-30T18:21:27.432Z [WARN] server-3e820db2-47e1-2394-14d6-515190f39986: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:27.432Z [INFO] server-3e820db2-47e1-2394-14d6-515190f39986: entering candidate state: node="Node at 3e820db2-47e1-2394-14d6-515190f39986 [Candidate]" term=2 +2025-10-30T18:21:27.432Z [INFO] server-3e820db2-47e1-2394-14d6-515190f39986: election won: term=2 tally=2 +2025-10-30T18:21:27.432Z [INFO] server-3e820db2-47e1-2394-14d6-515190f39986: entering leader state: leader="Node at 3e820db2-47e1-2394-14d6-515190f39986 [Leader]" +2025-10-30T18:21:27.432Z [INFO] server-3e820db2-47e1-2394-14d6-515190f39986: added peer, starting replication: peer=server-e4aaa6e4-f397-66d1-fa1f-d5efeb194d94 +2025-10-30T18:21:27.432Z [INFO] server-3e820db2-47e1-2394-14d6-515190f39986: added peer, starting replication: peer=server-9cd4e1bf-217b-0453-9d9c-59d59e880876 +2025-10-30T18:21:27.432Z [INFO] server-3e820db2-47e1-2394-14d6-515190f39986: pipelining replication: peer="{Voter server-9cd4e1bf-217b-0453-9d9c-59d59e880876 9cd4e1bf-217b-0453-9d9c-59d59e880876}" +2025-10-30T18:21:27.432Z [INFO] server-3e820db2-47e1-2394-14d6-515190f39986: pipelining replication: peer="{Voter server-e4aaa6e4-f397-66d1-fa1f-d5efeb194d94 e4aaa6e4-f397-66d1-fa1f-d5efeb194d94}" +2025-10-30T18:21:27.537Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:27.432365348 +0000 UTC m=+10.505894644 (1 nodes), 53.421065ms from start of poll, 53.741329ms from cluster start. Timeout at 2025-10-30 18:21:27.537367483 +0000 UTC m=+10.610896775, 105.002131ms after stability +2025-10-30T18:21:27.537Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:21:27.643Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:21:27.537769977 +0000 UTC m=+10.611299267 (2 nodes), 554ns from start of poll, 159.145952ms from cluster start. Timeout at 2025-10-30 18:21:27.642776987 +0000 UTC m=+10.716306280, 105.007013ms after stability +2025-10-30T18:21:27.643Z [INFO] server-3e820db2-47e1-2394-14d6-515190f39986: aborting pipeline replication: peer="{Voter server-9cd4e1bf-217b-0453-9d9c-59d59e880876 9cd4e1bf-217b-0453-9d9c-59d59e880876}" +2025-10-30T18:21:27.643Z [INFO] server-3e820db2-47e1-2394-14d6-515190f39986: aborting pipeline replication: peer="{Voter server-e4aaa6e4-f397-66d1-fa1f-d5efeb194d94 e4aaa6e4-f397-66d1-fa1f-d5efeb194d94}" +--- PASS: TestRaft_ApplyNonLeader (0.27s) === RUN TestRaft_ApplyConcurrent -2025-10-30T18:17:41.781Z [INFO] server-66edc0d3-3d91-9f9d-750e-6a98d61cf093: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66edc0d3-3d91-9f9d-750e-6a98d61cf093 Address:66edc0d3-3d91-9f9d-750e-6a98d61cf093} {Suffrage:Voter ID:server-ead9fc6b-97cc-9600-d687-c58362a702f9 Address:ead9fc6b-97cc-9600-d687-c58362a702f9} {Suffrage:Voter ID:server-a40ee663-de6f-2f30-67c9-70ff84a126d2 Address:a40ee663-de6f-2f30-67c9-70ff84a126d2}]" -2025-10-30T18:17:41.781Z [INFO] server-66edc0d3-3d91-9f9d-750e-6a98d61cf093: entering follower state: follower="Node at 66edc0d3-3d91-9f9d-750e-6a98d61cf093 [Follower]" leader-address= leader-id= -2025-10-30T18:17:41.781Z [INFO] server-ead9fc6b-97cc-9600-d687-c58362a702f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66edc0d3-3d91-9f9d-750e-6a98d61cf093 Address:66edc0d3-3d91-9f9d-750e-6a98d61cf093} {Suffrage:Voter ID:server-ead9fc6b-97cc-9600-d687-c58362a702f9 Address:ead9fc6b-97cc-9600-d687-c58362a702f9} {Suffrage:Voter ID:server-a40ee663-de6f-2f30-67c9-70ff84a126d2 Address:a40ee663-de6f-2f30-67c9-70ff84a126d2}]" -2025-10-30T18:17:41.781Z [INFO] server-ead9fc6b-97cc-9600-d687-c58362a702f9: entering follower state: follower="Node at ead9fc6b-97cc-9600-d687-c58362a702f9 [Follower]" leader-address= leader-id= -2025-10-30T18:17:41.781Z [INFO] server-a40ee663-de6f-2f30-67c9-70ff84a126d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66edc0d3-3d91-9f9d-750e-6a98d61cf093 Address:66edc0d3-3d91-9f9d-750e-6a98d61cf093} {Suffrage:Voter ID:server-ead9fc6b-97cc-9600-d687-c58362a702f9 Address:ead9fc6b-97cc-9600-d687-c58362a702f9} {Suffrage:Voter ID:server-a40ee663-de6f-2f30-67c9-70ff84a126d2 Address:a40ee663-de6f-2f30-67c9-70ff84a126d2}]" -2025-10-30T18:17:41.781Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:41.781Z [INFO] server-a40ee663-de6f-2f30-67c9-70ff84a126d2: entering follower state: follower="Node at a40ee663-de6f-2f30-67c9-70ff84a126d2 [Follower]" leader-address= leader-id= -2025-10-30T18:17:41.908Z [WARN] server-ead9fc6b-97cc-9600-d687-c58362a702f9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:41.908Z [INFO] server-ead9fc6b-97cc-9600-d687-c58362a702f9: entering candidate state: node="Node at ead9fc6b-97cc-9600-d687-c58362a702f9 [Candidate]" term=2 -2025-10-30T18:17:41.908Z [INFO] server-ead9fc6b-97cc-9600-d687-c58362a702f9: election won: term=2 tally=2 -2025-10-30T18:17:41.908Z [INFO] server-ead9fc6b-97cc-9600-d687-c58362a702f9: entering leader state: leader="Node at ead9fc6b-97cc-9600-d687-c58362a702f9 [Leader]" -2025-10-30T18:17:41.908Z [INFO] server-ead9fc6b-97cc-9600-d687-c58362a702f9: added peer, starting replication: peer=server-66edc0d3-3d91-9f9d-750e-6a98d61cf093 -2025-10-30T18:17:41.908Z [INFO] server-ead9fc6b-97cc-9600-d687-c58362a702f9: added peer, starting replication: peer=server-a40ee663-de6f-2f30-67c9-70ff84a126d2 -2025-10-30T18:17:41.908Z [INFO] server-ead9fc6b-97cc-9600-d687-c58362a702f9: pipelining replication: peer="{Voter server-66edc0d3-3d91-9f9d-750e-6a98d61cf093 66edc0d3-3d91-9f9d-750e-6a98d61cf093}" -2025-10-30T18:17:41.908Z [INFO] server-ead9fc6b-97cc-9600-d687-c58362a702f9: pipelining replication: peer="{Voter server-a40ee663-de6f-2f30-67c9-70ff84a126d2 a40ee663-de6f-2f30-67c9-70ff84a126d2}" -2025-10-30T18:17:42.114Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:41.908810143 +0000 UTC m=+8.187484041 (1 nodes), 127.386697ms from start of poll, 127.69849ms from cluster start. Timeout at 2025-10-30 18:17:42.113810582 +0000 UTC m=+8.392484479, 205.000438ms after stability -2025-10-30T18:17:42.128Z [INFO] server-ead9fc6b-97cc-9600-d687-c58362a702f9: aborting pipeline replication: peer="{Voter server-a40ee663-de6f-2f30-67c9-70ff84a126d2 a40ee663-de6f-2f30-67c9-70ff84a126d2}" -2025-10-30T18:17:42.128Z [INFO] server-ead9fc6b-97cc-9600-d687-c58362a702f9: aborting pipeline replication: peer="{Voter server-66edc0d3-3d91-9f9d-750e-6a98d61cf093 66edc0d3-3d91-9f9d-750e-6a98d61cf093}" ---- PASS: TestRaft_ApplyConcurrent (0.35s) +2025-10-30T18:21:27.644Z [INFO] server-916d197c-ed71-4416-e14c-de23bacd79b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-916d197c-ed71-4416-e14c-de23bacd79b3 Address:916d197c-ed71-4416-e14c-de23bacd79b3} {Suffrage:Voter ID:server-15403757-eba5-3863-127b-7b5fafdc3f67 Address:15403757-eba5-3863-127b-7b5fafdc3f67} {Suffrage:Voter ID:server-65068693-8e78-6c51-e3f8-72f52c6d3c5e Address:65068693-8e78-6c51-e3f8-72f52c6d3c5e}]" +2025-10-30T18:21:27.644Z [INFO] server-15403757-eba5-3863-127b-7b5fafdc3f67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-916d197c-ed71-4416-e14c-de23bacd79b3 Address:916d197c-ed71-4416-e14c-de23bacd79b3} {Suffrage:Voter ID:server-15403757-eba5-3863-127b-7b5fafdc3f67 Address:15403757-eba5-3863-127b-7b5fafdc3f67} {Suffrage:Voter ID:server-65068693-8e78-6c51-e3f8-72f52c6d3c5e Address:65068693-8e78-6c51-e3f8-72f52c6d3c5e}]" +2025-10-30T18:21:27.644Z [INFO] server-916d197c-ed71-4416-e14c-de23bacd79b3: entering follower state: follower="Node at 916d197c-ed71-4416-e14c-de23bacd79b3 [Follower]" leader-address= leader-id= +2025-10-30T18:21:27.644Z [INFO] server-65068693-8e78-6c51-e3f8-72f52c6d3c5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-916d197c-ed71-4416-e14c-de23bacd79b3 Address:916d197c-ed71-4416-e14c-de23bacd79b3} {Suffrage:Voter ID:server-15403757-eba5-3863-127b-7b5fafdc3f67 Address:15403757-eba5-3863-127b-7b5fafdc3f67} {Suffrage:Voter ID:server-65068693-8e78-6c51-e3f8-72f52c6d3c5e Address:65068693-8e78-6c51-e3f8-72f52c6d3c5e}]" +2025-10-30T18:21:27.644Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:27.644Z [INFO] server-15403757-eba5-3863-127b-7b5fafdc3f67: entering follower state: follower="Node at 15403757-eba5-3863-127b-7b5fafdc3f67 [Follower]" leader-address= leader-id= +2025-10-30T18:21:27.644Z [INFO] server-65068693-8e78-6c51-e3f8-72f52c6d3c5e: entering follower state: follower="Node at 65068693-8e78-6c51-e3f8-72f52c6d3c5e [Follower]" leader-address= leader-id= +2025-10-30T18:21:27.788Z [WARN] server-15403757-eba5-3863-127b-7b5fafdc3f67: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:27.788Z [INFO] server-15403757-eba5-3863-127b-7b5fafdc3f67: entering candidate state: node="Node at 15403757-eba5-3863-127b-7b5fafdc3f67 [Candidate]" term=2 +2025-10-30T18:21:27.788Z [INFO] server-15403757-eba5-3863-127b-7b5fafdc3f67: election won: term=2 tally=2 +2025-10-30T18:21:27.788Z [INFO] server-15403757-eba5-3863-127b-7b5fafdc3f67: entering leader state: leader="Node at 15403757-eba5-3863-127b-7b5fafdc3f67 [Leader]" +2025-10-30T18:21:27.788Z [INFO] server-15403757-eba5-3863-127b-7b5fafdc3f67: added peer, starting replication: peer=server-916d197c-ed71-4416-e14c-de23bacd79b3 +2025-10-30T18:21:27.788Z [INFO] server-15403757-eba5-3863-127b-7b5fafdc3f67: added peer, starting replication: peer=server-65068693-8e78-6c51-e3f8-72f52c6d3c5e +2025-10-30T18:21:27.788Z [INFO] server-15403757-eba5-3863-127b-7b5fafdc3f67: pipelining replication: peer="{Voter server-916d197c-ed71-4416-e14c-de23bacd79b3 916d197c-ed71-4416-e14c-de23bacd79b3}" +2025-10-30T18:21:27.788Z [INFO] server-15403757-eba5-3863-127b-7b5fafdc3f67: pipelining replication: peer="{Voter server-65068693-8e78-6c51-e3f8-72f52c6d3c5e 65068693-8e78-6c51-e3f8-72f52c6d3c5e}" +2025-10-30T18:21:27.993Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:27.78861166 +0000 UTC m=+10.862140953 (1 nodes), 143.846491ms from start of poll, 144.109286ms from cluster start. Timeout at 2025-10-30 18:21:27.993614056 +0000 UTC m=+11.067143348, 205.002395ms after stability +2025-10-30T18:21:28.006Z [INFO] server-15403757-eba5-3863-127b-7b5fafdc3f67: aborting pipeline replication: peer="{Voter server-65068693-8e78-6c51-e3f8-72f52c6d3c5e 65068693-8e78-6c51-e3f8-72f52c6d3c5e}" +2025-10-30T18:21:28.006Z [INFO] server-15403757-eba5-3863-127b-7b5fafdc3f67: aborting pipeline replication: peer="{Voter server-916d197c-ed71-4416-e14c-de23bacd79b3 916d197c-ed71-4416-e14c-de23bacd79b3}" +2025-10-30T18:21:28.507Z [ERROR] server-15403757-eba5-3863-127b-7b5fafdc3f67: failed to heartbeat to: peer=916d197c-ed71-4416-e14c-de23bacd79b3 backoff time=10ms error="command timed out" +--- PASS: TestRaft_ApplyConcurrent (0.86s) === RUN TestRaft_ApplyConcurrent_Timeout -2025-10-30T18:17:42.129Z [INFO] server-4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1 Address:4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1}]" -2025-10-30T18:17:42.129Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:42.129Z [INFO] server-4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1: entering follower state: follower="Node at 4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1 [Follower]" leader-address= leader-id= -2025-10-30T18:17:42.233Z [WARN] server-4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:42.233Z [INFO] server-4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1: entering candidate state: node="Node at 4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1 [Candidate]" term=2 -2025-10-30T18:17:42.233Z [INFO] server-4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1: election won: term=2 tally=1 -2025-10-30T18:17:42.233Z [INFO] server-4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1: entering leader state: leader="Node at 4f40a0b5-4c8e-41dd-1d8c-57a8a676a0c1 [Leader]" -2025-10-30T18:17:42.435Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:42.2333469 +0000 UTC m=+8.512020810 (1 nodes), 103.997156ms from start of poll, 104.439594ms from cluster start. Timeout at 2025-10-30 18:17:42.434351306 +0000 UTC m=+8.713025204, 201.004394ms after stability ---- PASS: TestRaft_ApplyConcurrent_Timeout (0.31s) +2025-10-30T18:21:28.508Z [INFO] server-e75e956f-e200-7687-399b-7768fa7d9f18: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e75e956f-e200-7687-399b-7768fa7d9f18 Address:e75e956f-e200-7687-399b-7768fa7d9f18}]" +2025-10-30T18:21:28.508Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:28.508Z [INFO] server-e75e956f-e200-7687-399b-7768fa7d9f18: entering follower state: follower="Node at e75e956f-e200-7687-399b-7768fa7d9f18 [Follower]" leader-address= leader-id= +2025-10-30T18:21:28.652Z [WARN] server-e75e956f-e200-7687-399b-7768fa7d9f18: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:28.652Z [INFO] server-e75e956f-e200-7687-399b-7768fa7d9f18: entering candidate state: node="Node at e75e956f-e200-7687-399b-7768fa7d9f18 [Candidate]" term=2 +2025-10-30T18:21:28.653Z [INFO] server-e75e956f-e200-7687-399b-7768fa7d9f18: election won: term=2 tally=1 +2025-10-30T18:21:28.653Z [INFO] server-e75e956f-e200-7687-399b-7768fa7d9f18: entering leader state: leader="Node at e75e956f-e200-7687-399b-7768fa7d9f18 [Leader]" +2025-10-30T18:21:28.855Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:28.653058625 +0000 UTC m=+11.726587914 (1 nodes), 144.490839ms from start of poll, 144.646421ms from cluster start. Timeout at 2025-10-30 18:21:28.854061251 +0000 UTC m=+11.927590540, 201.002626ms after stability +--- PASS: TestRaft_ApplyConcurrent_Timeout (0.35s) === RUN TestRaft_JoinNode -2025-10-30T18:17:42.436Z [INFO] server-6dc082e1-6ada-515e-6f4f-106b5c56e7ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6dc082e1-6ada-515e-6f4f-106b5c56e7ee Address:6dc082e1-6ada-515e-6f4f-106b5c56e7ee} {Suffrage:Voter ID:server-74eaed57-0880-317d-24fa-046bae12554c Address:74eaed57-0880-317d-24fa-046bae12554c}]" -2025-10-30T18:17:42.436Z [INFO] server-74eaed57-0880-317d-24fa-046bae12554c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6dc082e1-6ada-515e-6f4f-106b5c56e7ee Address:6dc082e1-6ada-515e-6f4f-106b5c56e7ee} {Suffrage:Voter ID:server-74eaed57-0880-317d-24fa-046bae12554c Address:74eaed57-0880-317d-24fa-046bae12554c}]" -2025-10-30T18:17:42.436Z [INFO] server-6dc082e1-6ada-515e-6f4f-106b5c56e7ee: entering follower state: follower="Node at 6dc082e1-6ada-515e-6f4f-106b5c56e7ee [Follower]" leader-address= leader-id= -2025-10-30T18:17:42.436Z [INFO] server-74eaed57-0880-317d-24fa-046bae12554c: entering follower state: follower="Node at 74eaed57-0880-317d-24fa-046bae12554c [Follower]" leader-address= leader-id= -2025-10-30T18:17:42.437Z [INFO] server-e40d87c6-80a4-6509-a3fa-850f5b180f88: initial configuration: index=0 servers=[] -2025-10-30T18:17:42.437Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:42.437Z [INFO] server-e40d87c6-80a4-6509-a3fa-850f5b180f88: entering follower state: follower="Node at e40d87c6-80a4-6509-a3fa-850f5b180f88 [Follower]" leader-address= leader-id= -2025-10-30T18:17:42.490Z [WARN] server-74eaed57-0880-317d-24fa-046bae12554c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:42.490Z [INFO] server-74eaed57-0880-317d-24fa-046bae12554c: entering candidate state: node="Node at 74eaed57-0880-317d-24fa-046bae12554c [Candidate]" term=2 -2025-10-30T18:17:42.490Z [INFO] server-74eaed57-0880-317d-24fa-046bae12554c: election won: term=2 tally=2 -2025-10-30T18:17:42.490Z [INFO] server-74eaed57-0880-317d-24fa-046bae12554c: entering leader state: leader="Node at 74eaed57-0880-317d-24fa-046bae12554c [Leader]" -2025-10-30T18:17:42.490Z [INFO] server-74eaed57-0880-317d-24fa-046bae12554c: added peer, starting replication: peer=server-6dc082e1-6ada-515e-6f4f-106b5c56e7ee -2025-10-30T18:17:42.490Z [INFO] server-74eaed57-0880-317d-24fa-046bae12554c: pipelining replication: peer="{Voter server-6dc082e1-6ada-515e-6f4f-106b5c56e7ee 6dc082e1-6ada-515e-6f4f-106b5c56e7ee}" -2025-10-30T18:17:42.518Z [WARN] server-e40d87c6-80a4-6509-a3fa-850f5b180f88: no known peers, aborting election -2025-10-30T18:17:42.596Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:42.49044263 +0000 UTC m=+8.769116538 (1 nodes), 53.274818ms from start of poll, 53.686432ms from cluster start. Timeout at 2025-10-30 18:17:42.595444536 +0000 UTC m=+8.874118434, 105.001896ms after stability -2025-10-30T18:17:42.596Z [INFO] server-74eaed57-0880-317d-24fa-046bae12554c: updating configuration: command=AddVoter server-id=server-e40d87c6-80a4-6509-a3fa-850f5b180f88 server-addr=e40d87c6-80a4-6509-a3fa-850f5b180f88 servers="[{Suffrage:Voter ID:server-6dc082e1-6ada-515e-6f4f-106b5c56e7ee Address:6dc082e1-6ada-515e-6f4f-106b5c56e7ee} {Suffrage:Voter ID:server-74eaed57-0880-317d-24fa-046bae12554c Address:74eaed57-0880-317d-24fa-046bae12554c} {Suffrage:Voter ID:server-e40d87c6-80a4-6509-a3fa-850f5b180f88 Address:e40d87c6-80a4-6509-a3fa-850f5b180f88}]" -2025-10-30T18:17:42.596Z [INFO] server-74eaed57-0880-317d-24fa-046bae12554c: added peer, starting replication: peer=server-e40d87c6-80a4-6509-a3fa-850f5b180f88 -2025-10-30T18:17:42.596Z [WARN] server-e40d87c6-80a4-6509-a3fa-850f5b180f88: failed to get previous log: previous-index=3 last-index=0 error="log not found" -2025-10-30T18:17:42.596Z [WARN] server-74eaed57-0880-317d-24fa-046bae12554c: appendEntries rejected, sending older logs: peer="{Voter server-e40d87c6-80a4-6509-a3fa-850f5b180f88 e40d87c6-80a4-6509-a3fa-850f5b180f88}" next=1 -2025-10-30T18:17:42.596Z [INFO] server-74eaed57-0880-317d-24fa-046bae12554c: pipelining replication: peer="{Voter server-e40d87c6-80a4-6509-a3fa-850f5b180f88 e40d87c6-80a4-6509-a3fa-850f5b180f88}" -2025-10-30T18:17:42.596Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:42.701Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:42.596440668 +0000 UTC m=+8.875114566 (1 nodes), 571ns from start of poll, 159.68446ms from cluster start. Timeout at 2025-10-30 18:17:42.701448847 +0000 UTC m=+8.980122761, 105.008195ms after stability -2025-10-30T18:17:42.701Z [INFO] server-74eaed57-0880-317d-24fa-046bae12554c: aborting pipeline replication: peer="{Voter server-e40d87c6-80a4-6509-a3fa-850f5b180f88 e40d87c6-80a4-6509-a3fa-850f5b180f88}" -2025-10-30T18:17:42.701Z [INFO] server-74eaed57-0880-317d-24fa-046bae12554c: aborting pipeline replication: peer="{Voter server-6dc082e1-6ada-515e-6f4f-106b5c56e7ee 6dc082e1-6ada-515e-6f4f-106b5c56e7ee}" +2025-10-30T18:21:28.856Z [INFO] server-e3361898-e24c-a1da-e950-908eb5a95a6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3361898-e24c-a1da-e950-908eb5a95a6f Address:e3361898-e24c-a1da-e950-908eb5a95a6f} {Suffrage:Voter ID:server-ad03f314-f119-5ab7-6209-ab7e3d611f5a Address:ad03f314-f119-5ab7-6209-ab7e3d611f5a}]" +2025-10-30T18:21:28.856Z [INFO] server-e3361898-e24c-a1da-e950-908eb5a95a6f: entering follower state: follower="Node at e3361898-e24c-a1da-e950-908eb5a95a6f [Follower]" leader-address= leader-id= +2025-10-30T18:21:28.856Z [INFO] server-ad03f314-f119-5ab7-6209-ab7e3d611f5a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3361898-e24c-a1da-e950-908eb5a95a6f Address:e3361898-e24c-a1da-e950-908eb5a95a6f} {Suffrage:Voter ID:server-ad03f314-f119-5ab7-6209-ab7e3d611f5a Address:ad03f314-f119-5ab7-6209-ab7e3d611f5a}]" +2025-10-30T18:21:28.856Z [INFO] server-ad03f314-f119-5ab7-6209-ab7e3d611f5a: entering follower state: follower="Node at ad03f314-f119-5ab7-6209-ab7e3d611f5a [Follower]" leader-address= leader-id= +2025-10-30T18:21:28.856Z [INFO] server-ed207402-2869-f0ce-04d2-81cfa693d6e4: initial configuration: index=0 servers=[] +2025-10-30T18:21:28.856Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:28.856Z [INFO] server-ed207402-2869-f0ce-04d2-81cfa693d6e4: entering follower state: follower="Node at ed207402-2869-f0ce-04d2-81cfa693d6e4 [Follower]" leader-address= leader-id= +2025-10-30T18:21:28.910Z [WARN] server-e3361898-e24c-a1da-e950-908eb5a95a6f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:28.910Z [INFO] server-e3361898-e24c-a1da-e950-908eb5a95a6f: entering candidate state: node="Node at e3361898-e24c-a1da-e950-908eb5a95a6f [Candidate]" term=2 +2025-10-30T18:21:28.911Z [INFO] server-e3361898-e24c-a1da-e950-908eb5a95a6f: election won: term=2 tally=2 +2025-10-30T18:21:28.911Z [INFO] server-e3361898-e24c-a1da-e950-908eb5a95a6f: entering leader state: leader="Node at e3361898-e24c-a1da-e950-908eb5a95a6f [Leader]" +2025-10-30T18:21:28.911Z [INFO] server-e3361898-e24c-a1da-e950-908eb5a95a6f: added peer, starting replication: peer=server-ad03f314-f119-5ab7-6209-ab7e3d611f5a +2025-10-30T18:21:28.911Z [INFO] server-e3361898-e24c-a1da-e950-908eb5a95a6f: pipelining replication: peer="{Voter server-ad03f314-f119-5ab7-6209-ab7e3d611f5a ad03f314-f119-5ab7-6209-ab7e3d611f5a}" +2025-10-30T18:21:28.930Z [WARN] server-ed207402-2869-f0ce-04d2-81cfa693d6e4: no known peers, aborting election +2025-10-30T18:21:29.016Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:28.911093623 +0000 UTC m=+11.984622915 (1 nodes), 54.457793ms from start of poll, 55.121992ms from cluster start. Timeout at 2025-10-30 18:21:29.016094498 +0000 UTC m=+12.089623790, 105.000875ms after stability +2025-10-30T18:21:29.016Z [INFO] server-e3361898-e24c-a1da-e950-908eb5a95a6f: updating configuration: command=AddVoter server-id=server-ed207402-2869-f0ce-04d2-81cfa693d6e4 server-addr=ed207402-2869-f0ce-04d2-81cfa693d6e4 servers="[{Suffrage:Voter ID:server-e3361898-e24c-a1da-e950-908eb5a95a6f Address:e3361898-e24c-a1da-e950-908eb5a95a6f} {Suffrage:Voter ID:server-ad03f314-f119-5ab7-6209-ab7e3d611f5a Address:ad03f314-f119-5ab7-6209-ab7e3d611f5a} {Suffrage:Voter ID:server-ed207402-2869-f0ce-04d2-81cfa693d6e4 Address:ed207402-2869-f0ce-04d2-81cfa693d6e4}]" +2025-10-30T18:21:29.016Z [INFO] server-e3361898-e24c-a1da-e950-908eb5a95a6f: added peer, starting replication: peer=server-ed207402-2869-f0ce-04d2-81cfa693d6e4 +2025-10-30T18:21:29.016Z [WARN] server-ed207402-2869-f0ce-04d2-81cfa693d6e4: failed to get previous log: previous-index=3 last-index=0 error="log not found" +2025-10-30T18:21:29.016Z [WARN] server-e3361898-e24c-a1da-e950-908eb5a95a6f: appendEntries rejected, sending older logs: peer="{Voter server-ed207402-2869-f0ce-04d2-81cfa693d6e4 ed207402-2869-f0ce-04d2-81cfa693d6e4}" next=1 +2025-10-30T18:21:29.016Z [INFO] server-e3361898-e24c-a1da-e950-908eb5a95a6f: pipelining replication: peer="{Voter server-ed207402-2869-f0ce-04d2-81cfa693d6e4 ed207402-2869-f0ce-04d2-81cfa693d6e4}" +2025-10-30T18:21:29.016Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:29.121Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:29.016569362 +0000 UTC m=+12.090098656 (1 nodes), 923ns from start of poll, 160.597733ms from cluster start. Timeout at 2025-10-30 18:21:29.121572098 +0000 UTC m=+12.195101387, 105.002731ms after stability +2025-10-30T18:21:29.121Z [INFO] server-e3361898-e24c-a1da-e950-908eb5a95a6f: aborting pipeline replication: peer="{Voter server-ed207402-2869-f0ce-04d2-81cfa693d6e4 ed207402-2869-f0ce-04d2-81cfa693d6e4}" +2025-10-30T18:21:29.121Z [INFO] server-e3361898-e24c-a1da-e950-908eb5a95a6f: aborting pipeline replication: peer="{Voter server-ad03f314-f119-5ab7-6209-ab7e3d611f5a ad03f314-f119-5ab7-6209-ab7e3d611f5a}" --- PASS: TestRaft_JoinNode (0.27s) === RUN TestRaft_JoinNode_ConfigStore -2025-10-30T18:17:42.702Z [INFO] server-f36477ac-c939-3fe3-cad8-9249aacc4a99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f36477ac-c939-3fe3-cad8-9249aacc4a99 Address:f36477ac-c939-3fe3-cad8-9249aacc4a99}]" -2025-10-30T18:17:42.702Z [INFO] server-f36477ac-c939-3fe3-cad8-9249aacc4a99: entering follower state: follower="Node at f36477ac-c939-3fe3-cad8-9249aacc4a99 [Follower]" leader-address= leader-id= -2025-10-30T18:17:42.702Z [INFO] server-dcc76a28-b151-9ba9-84f7-34704a47b6e4: initial configuration: index=0 servers=[] -2025-10-30T18:17:42.702Z [INFO] server-dcc76a28-b151-9ba9-84f7-34704a47b6e4: entering follower state: follower="Node at dcc76a28-b151-9ba9-84f7-34704a47b6e4 [Follower]" leader-address= leader-id= -2025-10-30T18:17:42.702Z [INFO] server-6eb2330a-bd33-58b7-4292-326c07b03977: initial configuration: index=0 servers=[] -2025-10-30T18:17:42.702Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:42.702Z [INFO] server-6eb2330a-bd33-58b7-4292-326c07b03977: entering follower state: follower="Node at 6eb2330a-bd33-58b7-4292-326c07b03977 [Follower]" leader-address= leader-id= -2025-10-30T18:17:42.784Z [WARN] server-6eb2330a-bd33-58b7-4292-326c07b03977: no known peers, aborting election -2025-10-30T18:17:42.785Z [WARN] server-dcc76a28-b151-9ba9-84f7-34704a47b6e4: no known peers, aborting election -2025-10-30T18:17:42.785Z [WARN] server-f36477ac-c939-3fe3-cad8-9249aacc4a99: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:42.785Z [INFO] server-f36477ac-c939-3fe3-cad8-9249aacc4a99: entering candidate state: node="Node at f36477ac-c939-3fe3-cad8-9249aacc4a99 [Candidate]" term=2 -2025-10-30T18:17:42.785Z [INFO] server-f36477ac-c939-3fe3-cad8-9249aacc4a99: election won: term=2 tally=1 -2025-10-30T18:17:42.785Z [INFO] server-f36477ac-c939-3fe3-cad8-9249aacc4a99: entering leader state: leader="Node at f36477ac-c939-3fe3-cad8-9249aacc4a99 [Leader]" -2025-10-30T18:17:42.891Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:42.785470408 +0000 UTC m=+9.064144320 (1 nodes), 82.482645ms from start of poll, 83.086063ms from cluster start. Timeout at 2025-10-30 18:17:42.890473913 +0000 UTC m=+9.169147813, 105.003493ms after stability -2025-10-30T18:17:42.891Z [INFO] server-f36477ac-c939-3fe3-cad8-9249aacc4a99: updating configuration: command=AddVoter server-id=server-dcc76a28-b151-9ba9-84f7-34704a47b6e4 server-addr=dcc76a28-b151-9ba9-84f7-34704a47b6e4 servers="[{Suffrage:Voter ID:server-f36477ac-c939-3fe3-cad8-9249aacc4a99 Address:f36477ac-c939-3fe3-cad8-9249aacc4a99} {Suffrage:Voter ID:server-dcc76a28-b151-9ba9-84f7-34704a47b6e4 Address:dcc76a28-b151-9ba9-84f7-34704a47b6e4}]" -2025-10-30T18:17:42.891Z [INFO] server-f36477ac-c939-3fe3-cad8-9249aacc4a99: added peer, starting replication: peer=server-dcc76a28-b151-9ba9-84f7-34704a47b6e4 -2025-10-30T18:17:42.891Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:42.891Z [WARN] server-dcc76a28-b151-9ba9-84f7-34704a47b6e4: failed to get previous log: previous-index=3 last-index=0 error="log not found" -2025-10-30T18:17:42.891Z [WARN] server-f36477ac-c939-3fe3-cad8-9249aacc4a99: appendEntries rejected, sending older logs: peer="{Voter server-dcc76a28-b151-9ba9-84f7-34704a47b6e4 dcc76a28-b151-9ba9-84f7-34704a47b6e4}" next=1 -2025-10-30T18:17:42.891Z [INFO] server-f36477ac-c939-3fe3-cad8-9249aacc4a99: pipelining replication: peer="{Voter server-dcc76a28-b151-9ba9-84f7-34704a47b6e4 dcc76a28-b151-9ba9-84f7-34704a47b6e4}" -2025-10-30T18:17:42.997Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:42.891583551 +0000 UTC m=+9.170257449 (1 nodes), 792ns from start of poll, 189.199192ms from cluster start. Timeout at 2025-10-30 18:17:42.99659325 +0000 UTC m=+9.275267147, 105.009698ms after stability -2025-10-30T18:17:42.997Z [INFO] server-f36477ac-c939-3fe3-cad8-9249aacc4a99: updating configuration: command=AddVoter server-id=server-6eb2330a-bd33-58b7-4292-326c07b03977 server-addr=6eb2330a-bd33-58b7-4292-326c07b03977 servers="[{Suffrage:Voter ID:server-f36477ac-c939-3fe3-cad8-9249aacc4a99 Address:f36477ac-c939-3fe3-cad8-9249aacc4a99} {Suffrage:Voter ID:server-dcc76a28-b151-9ba9-84f7-34704a47b6e4 Address:dcc76a28-b151-9ba9-84f7-34704a47b6e4} {Suffrage:Voter ID:server-6eb2330a-bd33-58b7-4292-326c07b03977 Address:6eb2330a-bd33-58b7-4292-326c07b03977}]" -2025-10-30T18:17:42.997Z [INFO] server-f36477ac-c939-3fe3-cad8-9249aacc4a99: added peer, starting replication: peer=server-6eb2330a-bd33-58b7-4292-326c07b03977 -2025-10-30T18:17:42.997Z [WARN] server-6eb2330a-bd33-58b7-4292-326c07b03977: failed to get previous log: previous-index=4 last-index=0 error="log not found" -2025-10-30T18:17:42.997Z [WARN] server-f36477ac-c939-3fe3-cad8-9249aacc4a99: appendEntries rejected, sending older logs: peer="{Voter server-6eb2330a-bd33-58b7-4292-326c07b03977 6eb2330a-bd33-58b7-4292-326c07b03977}" next=1 -2025-10-30T18:17:42.997Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:42.998Z [INFO] server-f36477ac-c939-3fe3-cad8-9249aacc4a99: pipelining replication: peer="{Voter server-6eb2330a-bd33-58b7-4292-326c07b03977 6eb2330a-bd33-58b7-4292-326c07b03977}" -2025-10-30T18:17:43.103Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:42.99801285 +0000 UTC m=+9.276686764 (1 nodes), 6.438µs from start of poll, 295.628507ms from cluster start. Timeout at 2025-10-30 18:17:43.103015289 +0000 UTC m=+9.381689186, 105.002422ms after stability -2025-10-30T18:17:43.103Z [INFO] server-f36477ac-c939-3fe3-cad8-9249aacc4a99: aborting pipeline replication: peer="{Voter server-6eb2330a-bd33-58b7-4292-326c07b03977 6eb2330a-bd33-58b7-4292-326c07b03977}" -2025-10-30T18:17:43.103Z [INFO] server-f36477ac-c939-3fe3-cad8-9249aacc4a99: aborting pipeline replication: peer="{Voter server-dcc76a28-b151-9ba9-84f7-34704a47b6e4 dcc76a28-b151-9ba9-84f7-34704a47b6e4}" ---- PASS: TestRaft_JoinNode_ConfigStore (0.40s) +2025-10-30T18:21:29.122Z [INFO] server-0d171c59-9637-27b0-d9e8-0c0559c13f79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d171c59-9637-27b0-d9e8-0c0559c13f79 Address:0d171c59-9637-27b0-d9e8-0c0559c13f79}]" +2025-10-30T18:21:29.122Z [INFO] server-0d171c59-9637-27b0-d9e8-0c0559c13f79: entering follower state: follower="Node at 0d171c59-9637-27b0-d9e8-0c0559c13f79 [Follower]" leader-address= leader-id= +2025-10-30T18:21:29.122Z [INFO] server-993dfba6-3d82-0ff1-3f01-894187bee953: initial configuration: index=0 servers=[] +2025-10-30T18:21:29.123Z [INFO] server-993dfba6-3d82-0ff1-3f01-894187bee953: entering follower state: follower="Node at 993dfba6-3d82-0ff1-3f01-894187bee953 [Follower]" leader-address= leader-id= +2025-10-30T18:21:29.123Z [INFO] server-409e174e-1dc9-9152-3ae3-707c995f31a6: initial configuration: index=0 servers=[] +2025-10-30T18:21:29.123Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:29.123Z [INFO] server-409e174e-1dc9-9152-3ae3-707c995f31a6: entering follower state: follower="Node at 409e174e-1dc9-9152-3ae3-707c995f31a6 [Follower]" leader-address= leader-id= +2025-10-30T18:21:29.173Z [WARN] server-0d171c59-9637-27b0-d9e8-0c0559c13f79: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:29.173Z [INFO] server-0d171c59-9637-27b0-d9e8-0c0559c13f79: entering candidate state: node="Node at 0d171c59-9637-27b0-d9e8-0c0559c13f79 [Candidate]" term=2 +2025-10-30T18:21:29.173Z [INFO] server-0d171c59-9637-27b0-d9e8-0c0559c13f79: election won: term=2 tally=1 +2025-10-30T18:21:29.174Z [INFO] server-0d171c59-9637-27b0-d9e8-0c0559c13f79: entering leader state: leader="Node at 0d171c59-9637-27b0-d9e8-0c0559c13f79 [Leader]" +2025-10-30T18:21:29.212Z [WARN] server-409e174e-1dc9-9152-3ae3-707c995f31a6: no known peers, aborting election +2025-10-30T18:21:29.214Z [WARN] server-993dfba6-3d82-0ff1-3f01-894187bee953: no known peers, aborting election +2025-10-30T18:21:29.279Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:29.174022899 +0000 UTC m=+12.247552194 (1 nodes), 50.77867ms from start of poll, 51.357552ms from cluster start. Timeout at 2025-10-30 18:21:29.279025493 +0000 UTC m=+12.352554786, 105.002592ms after stability +2025-10-30T18:21:29.279Z [INFO] server-0d171c59-9637-27b0-d9e8-0c0559c13f79: updating configuration: command=AddVoter server-id=server-993dfba6-3d82-0ff1-3f01-894187bee953 server-addr=993dfba6-3d82-0ff1-3f01-894187bee953 servers="[{Suffrage:Voter ID:server-0d171c59-9637-27b0-d9e8-0c0559c13f79 Address:0d171c59-9637-27b0-d9e8-0c0559c13f79} {Suffrage:Voter ID:server-993dfba6-3d82-0ff1-3f01-894187bee953 Address:993dfba6-3d82-0ff1-3f01-894187bee953}]" +2025-10-30T18:21:29.279Z [INFO] server-0d171c59-9637-27b0-d9e8-0c0559c13f79: added peer, starting replication: peer=server-993dfba6-3d82-0ff1-3f01-894187bee953 +2025-10-30T18:21:29.279Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:29.279Z [WARN] server-993dfba6-3d82-0ff1-3f01-894187bee953: failed to get previous log: previous-index=3 last-index=0 error="log not found" +2025-10-30T18:21:29.279Z [WARN] server-0d171c59-9637-27b0-d9e8-0c0559c13f79: appendEntries rejected, sending older logs: peer="{Voter server-993dfba6-3d82-0ff1-3f01-894187bee953 993dfba6-3d82-0ff1-3f01-894187bee953}" next=1 +2025-10-30T18:21:29.279Z [INFO] server-0d171c59-9637-27b0-d9e8-0c0559c13f79: pipelining replication: peer="{Voter server-993dfba6-3d82-0ff1-3f01-894187bee953 993dfba6-3d82-0ff1-3f01-894187bee953}" +2025-10-30T18:21:29.385Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:29.279715481 +0000 UTC m=+12.353244788 (1 nodes), 1.917µs from start of poll, 157.050146ms from cluster start. Timeout at 2025-10-30 18:21:29.384720922 +0000 UTC m=+12.458250214, 105.005426ms after stability +2025-10-30T18:21:29.385Z [INFO] server-0d171c59-9637-27b0-d9e8-0c0559c13f79: updating configuration: command=AddVoter server-id=server-409e174e-1dc9-9152-3ae3-707c995f31a6 server-addr=409e174e-1dc9-9152-3ae3-707c995f31a6 servers="[{Suffrage:Voter ID:server-0d171c59-9637-27b0-d9e8-0c0559c13f79 Address:0d171c59-9637-27b0-d9e8-0c0559c13f79} {Suffrage:Voter ID:server-993dfba6-3d82-0ff1-3f01-894187bee953 Address:993dfba6-3d82-0ff1-3f01-894187bee953} {Suffrage:Voter ID:server-409e174e-1dc9-9152-3ae3-707c995f31a6 Address:409e174e-1dc9-9152-3ae3-707c995f31a6}]" +2025-10-30T18:21:29.385Z [INFO] server-0d171c59-9637-27b0-d9e8-0c0559c13f79: added peer, starting replication: peer=server-409e174e-1dc9-9152-3ae3-707c995f31a6 +2025-10-30T18:21:29.385Z [WARN] server-409e174e-1dc9-9152-3ae3-707c995f31a6: failed to get previous log: previous-index=4 last-index=0 error="log not found" +2025-10-30T18:21:29.386Z [WARN] server-0d171c59-9637-27b0-d9e8-0c0559c13f79: appendEntries rejected, sending older logs: peer="{Voter server-409e174e-1dc9-9152-3ae3-707c995f31a6 409e174e-1dc9-9152-3ae3-707c995f31a6}" next=1 +2025-10-30T18:21:29.386Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:29.386Z [INFO] server-0d171c59-9637-27b0-d9e8-0c0559c13f79: pipelining replication: peer="{Voter server-409e174e-1dc9-9152-3ae3-707c995f31a6 409e174e-1dc9-9152-3ae3-707c995f31a6}" +2025-10-30T18:21:29.491Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:29.386088483 +0000 UTC m=+12.459617775 (1 nodes), 817ns from start of poll, 263.423133ms from cluster start. Timeout at 2025-10-30 18:21:29.491092032 +0000 UTC m=+12.564621325, 105.00355ms after stability +2025-10-30T18:21:29.492Z [INFO] server-0d171c59-9637-27b0-d9e8-0c0559c13f79: aborting pipeline replication: peer="{Voter server-409e174e-1dc9-9152-3ae3-707c995f31a6 409e174e-1dc9-9152-3ae3-707c995f31a6}" +2025-10-30T18:21:29.492Z [INFO] server-0d171c59-9637-27b0-d9e8-0c0559c13f79: aborting pipeline replication: peer="{Voter server-993dfba6-3d82-0ff1-3f01-894187bee953 993dfba6-3d82-0ff1-3f01-894187bee953}" +--- PASS: TestRaft_JoinNode_ConfigStore (0.37s) === RUN TestRaft_RemoveFollower -2025-10-30T18:17:43.103Z [INFO] server-ea27d9af-655f-75e1-d6d5-fd4a0529b188: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea27d9af-655f-75e1-d6d5-fd4a0529b188 Address:ea27d9af-655f-75e1-d6d5-fd4a0529b188} {Suffrage:Voter ID:server-b08b8bcf-9ef1-57d3-c485-03d0dce82392 Address:b08b8bcf-9ef1-57d3-c485-03d0dce82392} {Suffrage:Voter ID:server-dc43b1ef-6b4e-c802-b04d-f590d8336d33 Address:dc43b1ef-6b4e-c802-b04d-f590d8336d33}]" -2025-10-30T18:17:43.104Z [INFO] server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea27d9af-655f-75e1-d6d5-fd4a0529b188 Address:ea27d9af-655f-75e1-d6d5-fd4a0529b188} {Suffrage:Voter ID:server-b08b8bcf-9ef1-57d3-c485-03d0dce82392 Address:b08b8bcf-9ef1-57d3-c485-03d0dce82392} {Suffrage:Voter ID:server-dc43b1ef-6b4e-c802-b04d-f590d8336d33 Address:dc43b1ef-6b4e-c802-b04d-f590d8336d33}]" -2025-10-30T18:17:43.104Z [INFO] server-ea27d9af-655f-75e1-d6d5-fd4a0529b188: entering follower state: follower="Node at ea27d9af-655f-75e1-d6d5-fd4a0529b188 [Follower]" leader-address= leader-id= -2025-10-30T18:17:43.104Z [INFO] server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: entering follower state: follower="Node at b08b8bcf-9ef1-57d3-c485-03d0dce82392 [Follower]" leader-address= leader-id= -2025-10-30T18:17:43.104Z [INFO] server-dc43b1ef-6b4e-c802-b04d-f590d8336d33: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea27d9af-655f-75e1-d6d5-fd4a0529b188 Address:ea27d9af-655f-75e1-d6d5-fd4a0529b188} {Suffrage:Voter ID:server-b08b8bcf-9ef1-57d3-c485-03d0dce82392 Address:b08b8bcf-9ef1-57d3-c485-03d0dce82392} {Suffrage:Voter ID:server-dc43b1ef-6b4e-c802-b04d-f590d8336d33 Address:dc43b1ef-6b4e-c802-b04d-f590d8336d33}]" -2025-10-30T18:17:43.104Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:43.104Z [INFO] server-dc43b1ef-6b4e-c802-b04d-f590d8336d33: entering follower state: follower="Node at dc43b1ef-6b4e-c802-b04d-f590d8336d33 [Follower]" leader-address= leader-id= -2025-10-30T18:17:43.156Z [WARN] server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:43.156Z [INFO] server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: entering candidate state: node="Node at b08b8bcf-9ef1-57d3-c485-03d0dce82392 [Candidate]" term=2 -2025-10-30T18:17:43.157Z [INFO] server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: election won: term=2 tally=2 -2025-10-30T18:17:43.157Z [INFO] server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: entering leader state: leader="Node at b08b8bcf-9ef1-57d3-c485-03d0dce82392 [Leader]" -2025-10-30T18:17:43.157Z [INFO] server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: added peer, starting replication: peer=server-ea27d9af-655f-75e1-d6d5-fd4a0529b188 -2025-10-30T18:17:43.157Z [INFO] server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: added peer, starting replication: peer=server-dc43b1ef-6b4e-c802-b04d-f590d8336d33 -2025-10-30T18:17:43.157Z [INFO] server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: pipelining replication: peer="{Voter server-ea27d9af-655f-75e1-d6d5-fd4a0529b188 ea27d9af-655f-75e1-d6d5-fd4a0529b188}" -2025-10-30T18:17:43.157Z [INFO] server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: pipelining replication: peer="{Voter server-dc43b1ef-6b4e-c802-b04d-f590d8336d33 dc43b1ef-6b4e-c802-b04d-f590d8336d33}" -2025-10-30T18:17:43.262Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:43.157435794 +0000 UTC m=+9.436109696 (1 nodes), 53.316198ms from start of poll, 53.570634ms from cluster start. Timeout at 2025-10-30 18:17:43.262439098 +0000 UTC m=+9.541112997, 105.003301ms after stability -2025-10-30T18:17:43.268Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:17:43.374Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:17:43.268233946 +0000 UTC m=+9.546907844 (2 nodes), 2.408µs from start of poll, 164.368782ms from cluster start. Timeout at 2025-10-30 18:17:43.373238298 +0000 UTC m=+9.651912197, 105.004353ms after stability -2025-10-30T18:17:43.374Z [INFO] server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: updating configuration: command=RemoveServer server-id=server-ea27d9af-655f-75e1-d6d5-fd4a0529b188 server-addr= servers="[{Suffrage:Voter ID:server-b08b8bcf-9ef1-57d3-c485-03d0dce82392 Address:b08b8bcf-9ef1-57d3-c485-03d0dce82392} {Suffrage:Voter ID:server-dc43b1ef-6b4e-c802-b04d-f590d8336d33 Address:dc43b1ef-6b4e-c802-b04d-f590d8336d33}]" -2025-10-30T18:17:43.374Z [INFO] server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: removed peer, stopping replication: peer=server-ea27d9af-655f-75e1-d6d5-fd4a0529b188 last-index=3 -2025-10-30T18:17:43.374Z [INFO] server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: aborting pipeline replication: peer="{Voter server-ea27d9af-655f-75e1-d6d5-fd4a0529b188 ea27d9af-655f-75e1-d6d5-fd4a0529b188}" -2025-10-30T18:17:43.480Z [INFO] server-b08b8bcf-9ef1-57d3-c485-03d0dce82392: aborting pipeline replication: peer="{Voter server-dc43b1ef-6b4e-c802-b04d-f590d8336d33 dc43b1ef-6b4e-c802-b04d-f590d8336d33}" ---- PASS: TestRaft_RemoveFollower (0.38s) +2025-10-30T18:21:29.493Z [INFO] server-c8c1e874-e165-110a-dc48-a97857096cc0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8c1e874-e165-110a-dc48-a97857096cc0 Address:c8c1e874-e165-110a-dc48-a97857096cc0} {Suffrage:Voter ID:server-b090c113-770f-2a35-d3b1-551bfcdc7055 Address:b090c113-770f-2a35-d3b1-551bfcdc7055} {Suffrage:Voter ID:server-908012fa-568e-5824-620b-b93973204080 Address:908012fa-568e-5824-620b-b93973204080}]" +2025-10-30T18:21:29.493Z [INFO] server-c8c1e874-e165-110a-dc48-a97857096cc0: entering follower state: follower="Node at c8c1e874-e165-110a-dc48-a97857096cc0 [Follower]" leader-address= leader-id= +2025-10-30T18:21:29.493Z [INFO] server-b090c113-770f-2a35-d3b1-551bfcdc7055: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8c1e874-e165-110a-dc48-a97857096cc0 Address:c8c1e874-e165-110a-dc48-a97857096cc0} {Suffrage:Voter ID:server-b090c113-770f-2a35-d3b1-551bfcdc7055 Address:b090c113-770f-2a35-d3b1-551bfcdc7055} {Suffrage:Voter ID:server-908012fa-568e-5824-620b-b93973204080 Address:908012fa-568e-5824-620b-b93973204080}]" +2025-10-30T18:21:29.493Z [INFO] server-908012fa-568e-5824-620b-b93973204080: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c8c1e874-e165-110a-dc48-a97857096cc0 Address:c8c1e874-e165-110a-dc48-a97857096cc0} {Suffrage:Voter ID:server-b090c113-770f-2a35-d3b1-551bfcdc7055 Address:b090c113-770f-2a35-d3b1-551bfcdc7055} {Suffrage:Voter ID:server-908012fa-568e-5824-620b-b93973204080 Address:908012fa-568e-5824-620b-b93973204080}]" +2025-10-30T18:21:29.493Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:29.493Z [INFO] server-b090c113-770f-2a35-d3b1-551bfcdc7055: entering follower state: follower="Node at b090c113-770f-2a35-d3b1-551bfcdc7055 [Follower]" leader-address= leader-id= +2025-10-30T18:21:29.493Z [INFO] server-908012fa-568e-5824-620b-b93973204080: entering follower state: follower="Node at 908012fa-568e-5824-620b-b93973204080 [Follower]" leader-address= leader-id= +2025-10-30T18:21:29.567Z [WARN] server-908012fa-568e-5824-620b-b93973204080: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:29.567Z [INFO] server-908012fa-568e-5824-620b-b93973204080: entering candidate state: node="Node at 908012fa-568e-5824-620b-b93973204080 [Candidate]" term=2 +2025-10-30T18:21:29.567Z [INFO] server-908012fa-568e-5824-620b-b93973204080: election won: term=2 tally=2 +2025-10-30T18:21:29.567Z [INFO] server-908012fa-568e-5824-620b-b93973204080: entering leader state: leader="Node at 908012fa-568e-5824-620b-b93973204080 [Leader]" +2025-10-30T18:21:29.567Z [INFO] server-908012fa-568e-5824-620b-b93973204080: added peer, starting replication: peer=server-c8c1e874-e165-110a-dc48-a97857096cc0 +2025-10-30T18:21:29.567Z [INFO] server-908012fa-568e-5824-620b-b93973204080: added peer, starting replication: peer=server-b090c113-770f-2a35-d3b1-551bfcdc7055 +2025-10-30T18:21:29.567Z [INFO] server-908012fa-568e-5824-620b-b93973204080: pipelining replication: peer="{Voter server-c8c1e874-e165-110a-dc48-a97857096cc0 c8c1e874-e165-110a-dc48-a97857096cc0}" +2025-10-30T18:21:29.567Z [INFO] server-908012fa-568e-5824-620b-b93973204080: pipelining replication: peer="{Voter server-b090c113-770f-2a35-d3b1-551bfcdc7055 b090c113-770f-2a35-d3b1-551bfcdc7055}" +2025-10-30T18:21:29.673Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:29.567855067 +0000 UTC m=+12.641384355 (1 nodes), 74.462955ms from start of poll, 74.759024ms from cluster start. Timeout at 2025-10-30 18:21:29.672858132 +0000 UTC m=+12.746387425, 105.00307ms after stability +2025-10-30T18:21:29.678Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:21:29.783Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:21:29.678601312 +0000 UTC m=+12.752130607 (2 nodes), 3.851µs from start of poll, 185.505276ms from cluster start. Timeout at 2025-10-30 18:21:29.783606519 +0000 UTC m=+12.857135815, 105.005208ms after stability +2025-10-30T18:21:29.784Z [INFO] server-908012fa-568e-5824-620b-b93973204080: updating configuration: command=RemoveServer server-id=server-c8c1e874-e165-110a-dc48-a97857096cc0 server-addr= servers="[{Suffrage:Voter ID:server-b090c113-770f-2a35-d3b1-551bfcdc7055 Address:b090c113-770f-2a35-d3b1-551bfcdc7055} {Suffrage:Voter ID:server-908012fa-568e-5824-620b-b93973204080 Address:908012fa-568e-5824-620b-b93973204080}]" +2025-10-30T18:21:29.784Z [INFO] server-908012fa-568e-5824-620b-b93973204080: removed peer, stopping replication: peer=server-c8c1e874-e165-110a-dc48-a97857096cc0 last-index=3 +2025-10-30T18:21:29.784Z [INFO] server-908012fa-568e-5824-620b-b93973204080: aborting pipeline replication: peer="{Voter server-c8c1e874-e165-110a-dc48-a97857096cc0 c8c1e874-e165-110a-dc48-a97857096cc0}" +2025-10-30T18:21:29.890Z [INFO] server-908012fa-568e-5824-620b-b93973204080: aborting pipeline replication: peer="{Voter server-b090c113-770f-2a35-d3b1-551bfcdc7055 b090c113-770f-2a35-d3b1-551bfcdc7055}" +--- PASS: TestRaft_RemoveFollower (0.40s) === RUN TestRaft_RemoveLeader -2025-10-30T18:17:43.481Z [INFO] server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15cef41c-b5cf-52ae-2fb6-8c893bd80901 Address:15cef41c-b5cf-52ae-2fb6-8c893bd80901} {Suffrage:Voter ID:server-64e25359-002d-da89-15dd-c22b9a1aacbf Address:64e25359-002d-da89-15dd-c22b9a1aacbf} {Suffrage:Voter ID:server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa Address:fed267d7-71aa-1f7c-0eea-ea089f2b92fa}]" -2025-10-30T18:17:43.481Z [INFO] server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: entering follower state: follower="Node at 15cef41c-b5cf-52ae-2fb6-8c893bd80901 [Follower]" leader-address= leader-id= -2025-10-30T18:17:43.481Z [INFO] server-64e25359-002d-da89-15dd-c22b9a1aacbf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15cef41c-b5cf-52ae-2fb6-8c893bd80901 Address:15cef41c-b5cf-52ae-2fb6-8c893bd80901} {Suffrage:Voter ID:server-64e25359-002d-da89-15dd-c22b9a1aacbf Address:64e25359-002d-da89-15dd-c22b9a1aacbf} {Suffrage:Voter ID:server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa Address:fed267d7-71aa-1f7c-0eea-ea089f2b92fa}]" -2025-10-30T18:17:43.481Z [INFO] server-64e25359-002d-da89-15dd-c22b9a1aacbf: entering follower state: follower="Node at 64e25359-002d-da89-15dd-c22b9a1aacbf [Follower]" leader-address= leader-id= -2025-10-30T18:17:43.481Z [INFO] server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15cef41c-b5cf-52ae-2fb6-8c893bd80901 Address:15cef41c-b5cf-52ae-2fb6-8c893bd80901} {Suffrage:Voter ID:server-64e25359-002d-da89-15dd-c22b9a1aacbf Address:64e25359-002d-da89-15dd-c22b9a1aacbf} {Suffrage:Voter ID:server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa Address:fed267d7-71aa-1f7c-0eea-ea089f2b92fa}]" -2025-10-30T18:17:43.481Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:43.481Z [INFO] server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa: entering follower state: follower="Node at fed267d7-71aa-1f7c-0eea-ea089f2b92fa [Follower]" leader-address= leader-id= -2025-10-30T18:17:43.548Z [WARN] server-64e25359-002d-da89-15dd-c22b9a1aacbf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:43.548Z [INFO] server-64e25359-002d-da89-15dd-c22b9a1aacbf: entering candidate state: node="Node at 64e25359-002d-da89-15dd-c22b9a1aacbf [Candidate]" term=2 -2025-10-30T18:17:43.548Z [INFO] server-64e25359-002d-da89-15dd-c22b9a1aacbf: election won: term=2 tally=2 -2025-10-30T18:17:43.548Z [INFO] server-64e25359-002d-da89-15dd-c22b9a1aacbf: entering leader state: leader="Node at 64e25359-002d-da89-15dd-c22b9a1aacbf [Leader]" -2025-10-30T18:17:43.548Z [INFO] server-64e25359-002d-da89-15dd-c22b9a1aacbf: added peer, starting replication: peer=server-15cef41c-b5cf-52ae-2fb6-8c893bd80901 -2025-10-30T18:17:43.548Z [INFO] server-64e25359-002d-da89-15dd-c22b9a1aacbf: added peer, starting replication: peer=server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa -2025-10-30T18:17:43.548Z [INFO] server-64e25359-002d-da89-15dd-c22b9a1aacbf: pipelining replication: peer="{Voter server-15cef41c-b5cf-52ae-2fb6-8c893bd80901 15cef41c-b5cf-52ae-2fb6-8c893bd80901}" -2025-10-30T18:17:43.548Z [INFO] server-64e25359-002d-da89-15dd-c22b9a1aacbf: pipelining replication: peer="{Voter server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa fed267d7-71aa-1f7c-0eea-ea089f2b92fa}" -2025-10-30T18:17:43.654Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:43.548196502 +0000 UTC m=+9.826870399 (1 nodes), 66.463525ms from start of poll, 66.750341ms from cluster start. Timeout at 2025-10-30 18:17:43.653198081 +0000 UTC m=+9.931871982, 105.001583ms after stability -2025-10-30T18:17:43.659Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:17:43.765Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:17:43.659528239 +0000 UTC m=+9.938202149 (2 nodes), 2.258µs from start of poll, 178.082091ms from cluster start. Timeout at 2025-10-30 18:17:43.764532061 +0000 UTC m=+10.043205959, 105.00381ms after stability -2025-10-30T18:17:43.765Z [INFO] server-64e25359-002d-da89-15dd-c22b9a1aacbf: updating configuration: command=RemoveServer server-id=server-64e25359-002d-da89-15dd-c22b9a1aacbf server-addr= servers="[{Suffrage:Voter ID:server-15cef41c-b5cf-52ae-2fb6-8c893bd80901 Address:15cef41c-b5cf-52ae-2fb6-8c893bd80901} {Suffrage:Voter ID:server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa Address:fed267d7-71aa-1f7c-0eea-ea089f2b92fa}]" -2025-10-30T18:17:43.765Z [INFO] server-64e25359-002d-da89-15dd-c22b9a1aacbf: removed ourself, shutting down -2025-10-30T18:17:43.765Z [INFO] server-64e25359-002d-da89-15dd-c22b9a1aacbf: aborting pipeline replication: peer="{Voter server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa fed267d7-71aa-1f7c-0eea-ea089f2b92fa}" -2025-10-30T18:17:43.765Z [INFO] server-64e25359-002d-da89-15dd-c22b9a1aacbf: aborting pipeline replication: peer="{Voter server-15cef41c-b5cf-52ae-2fb6-8c893bd80901 15cef41c-b5cf-52ae-2fb6-8c893bd80901}" -2025-10-30T18:17:43.829Z [WARN] server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: heartbeat timeout reached, starting election: last-leader-addr=64e25359-002d-da89-15dd-c22b9a1aacbf last-leader-id=server-64e25359-002d-da89-15dd-c22b9a1aacbf -2025-10-30T18:17:43.829Z [INFO] server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: entering candidate state: node="Node at 15cef41c-b5cf-52ae-2fb6-8c893bd80901 [Candidate]" term=3 -2025-10-30T18:17:43.829Z [WARN] server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa: rejecting vote request since we have a leader: from=15cef41c-b5cf-52ae-2fb6-8c893bd80901 leader=64e25359-002d-da89-15dd-c22b9a1aacbf leader-id=server-64e25359-002d-da89-15dd-c22b9a1aacbf -2025-10-30T18:17:43.885Z [WARN] server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa: heartbeat timeout reached, starting election: last-leader-addr=64e25359-002d-da89-15dd-c22b9a1aacbf last-leader-id=server-64e25359-002d-da89-15dd-c22b9a1aacbf -2025-10-30T18:17:43.885Z [INFO] server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa: entering candidate state: node="Node at fed267d7-71aa-1f7c-0eea-ea089f2b92fa [Candidate]" term=3 -2025-10-30T18:17:43.885Z [INFO] server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: duplicate requestVote for same term: term=3 -2025-10-30T18:17:43.925Z [WARN] server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: Election timeout reached, restarting election -2025-10-30T18:17:43.925Z [INFO] server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: entering candidate state: node="Node at 15cef41c-b5cf-52ae-2fb6-8c893bd80901 [Candidate]" term=4 -2025-10-30T18:17:43.925Z [INFO] server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa: entering follower state: follower="Node at fed267d7-71aa-1f7c-0eea-ea089f2b92fa [Follower]" leader-address= leader-id= -2025-10-30T18:17:43.926Z [INFO] server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: election won: term=4 tally=2 -2025-10-30T18:17:43.926Z [INFO] server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: entering leader state: leader="Node at 15cef41c-b5cf-52ae-2fb6-8c893bd80901 [Leader]" -2025-10-30T18:17:43.926Z [INFO] server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: added peer, starting replication: peer=server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa -2025-10-30T18:17:43.926Z [INFO] server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: pipelining replication: peer="{Voter server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa fed267d7-71aa-1f7c-0eea-ea089f2b92fa}" -2025-10-30T18:17:43.977Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:44.082Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:43.977215896 +0000 UTC m=+10.255889805 (1 nodes), 34.111µs from start of poll, 495.769747ms from cluster start. Timeout at 2025-10-30 18:17:44.082219034 +0000 UTC m=+10.360892933, 105.003128ms after stability -2025-10-30T18:17:44.082Z [INFO] server-15cef41c-b5cf-52ae-2fb6-8c893bd80901: aborting pipeline replication: peer="{Voter server-fed267d7-71aa-1f7c-0eea-ea089f2b92fa fed267d7-71aa-1f7c-0eea-ea089f2b92fa}" ---- PASS: TestRaft_RemoveLeader (0.60s) +2025-10-30T18:21:29.891Z [INFO] server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6 Address:47ecd9b4-9ccf-1272-7a7e-be8779b320f6} {Suffrage:Voter ID:server-d3ffe671-0e56-caca-b526-46e2b3f286f1 Address:d3ffe671-0e56-caca-b526-46e2b3f286f1} {Suffrage:Voter ID:server-3c061017-04fb-6529-9320-8c84af8e7a0a Address:3c061017-04fb-6529-9320-8c84af8e7a0a}]" +2025-10-30T18:21:29.891Z [INFO] server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6: entering follower state: follower="Node at 47ecd9b4-9ccf-1272-7a7e-be8779b320f6 [Follower]" leader-address= leader-id= +2025-10-30T18:21:29.891Z [INFO] server-d3ffe671-0e56-caca-b526-46e2b3f286f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6 Address:47ecd9b4-9ccf-1272-7a7e-be8779b320f6} {Suffrage:Voter ID:server-d3ffe671-0e56-caca-b526-46e2b3f286f1 Address:d3ffe671-0e56-caca-b526-46e2b3f286f1} {Suffrage:Voter ID:server-3c061017-04fb-6529-9320-8c84af8e7a0a Address:3c061017-04fb-6529-9320-8c84af8e7a0a}]" +2025-10-30T18:21:29.891Z [INFO] server-d3ffe671-0e56-caca-b526-46e2b3f286f1: entering follower state: follower="Node at d3ffe671-0e56-caca-b526-46e2b3f286f1 [Follower]" leader-address= leader-id= +2025-10-30T18:21:29.891Z [INFO] server-3c061017-04fb-6529-9320-8c84af8e7a0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6 Address:47ecd9b4-9ccf-1272-7a7e-be8779b320f6} {Suffrage:Voter ID:server-d3ffe671-0e56-caca-b526-46e2b3f286f1 Address:d3ffe671-0e56-caca-b526-46e2b3f286f1} {Suffrage:Voter ID:server-3c061017-04fb-6529-9320-8c84af8e7a0a Address:3c061017-04fb-6529-9320-8c84af8e7a0a}]" +2025-10-30T18:21:29.891Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:29.891Z [INFO] server-3c061017-04fb-6529-9320-8c84af8e7a0a: entering follower state: follower="Node at 3c061017-04fb-6529-9320-8c84af8e7a0a [Follower]" leader-address= leader-id= +2025-10-30T18:21:29.943Z [WARN] server-3c061017-04fb-6529-9320-8c84af8e7a0a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:29.943Z [INFO] server-3c061017-04fb-6529-9320-8c84af8e7a0a: entering candidate state: node="Node at 3c061017-04fb-6529-9320-8c84af8e7a0a [Candidate]" term=2 +2025-10-30T18:21:29.943Z [INFO] server-3c061017-04fb-6529-9320-8c84af8e7a0a: election won: term=2 tally=2 +2025-10-30T18:21:29.943Z [INFO] server-3c061017-04fb-6529-9320-8c84af8e7a0a: entering leader state: leader="Node at 3c061017-04fb-6529-9320-8c84af8e7a0a [Leader]" +2025-10-30T18:21:29.943Z [INFO] server-3c061017-04fb-6529-9320-8c84af8e7a0a: added peer, starting replication: peer=server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6 +2025-10-30T18:21:29.943Z [INFO] server-3c061017-04fb-6529-9320-8c84af8e7a0a: added peer, starting replication: peer=server-d3ffe671-0e56-caca-b526-46e2b3f286f1 +2025-10-30T18:21:29.943Z [INFO] server-3c061017-04fb-6529-9320-8c84af8e7a0a: pipelining replication: peer="{Voter server-d3ffe671-0e56-caca-b526-46e2b3f286f1 d3ffe671-0e56-caca-b526-46e2b3f286f1}" +2025-10-30T18:21:29.943Z [INFO] server-3c061017-04fb-6529-9320-8c84af8e7a0a: pipelining replication: peer="{Voter server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6 47ecd9b4-9ccf-1272-7a7e-be8779b320f6}" +2025-10-30T18:21:30.049Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:29.943868003 +0000 UTC m=+13.017397296 (1 nodes), 52.336542ms from start of poll, 52.755242ms from cluster start. Timeout at 2025-10-30 18:21:30.048870283 +0000 UTC m=+13.122399575, 105.002279ms after stability +2025-10-30T18:21:30.055Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:21:30.160Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:21:30.055205629 +0000 UTC m=+13.128734925 (2 nodes), 968ns from start of poll, 164.092871ms from cluster start. Timeout at 2025-10-30 18:21:30.160213119 +0000 UTC m=+13.233742408, 105.007483ms after stability +2025-10-30T18:21:30.160Z [INFO] server-3c061017-04fb-6529-9320-8c84af8e7a0a: updating configuration: command=RemoveServer server-id=server-3c061017-04fb-6529-9320-8c84af8e7a0a server-addr= servers="[{Suffrage:Voter ID:server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6 Address:47ecd9b4-9ccf-1272-7a7e-be8779b320f6} {Suffrage:Voter ID:server-d3ffe671-0e56-caca-b526-46e2b3f286f1 Address:d3ffe671-0e56-caca-b526-46e2b3f286f1}]" +2025-10-30T18:21:30.160Z [INFO] server-3c061017-04fb-6529-9320-8c84af8e7a0a: removed ourself, shutting down +2025-10-30T18:21:30.160Z [INFO] server-3c061017-04fb-6529-9320-8c84af8e7a0a: aborting pipeline replication: peer="{Voter server-d3ffe671-0e56-caca-b526-46e2b3f286f1 d3ffe671-0e56-caca-b526-46e2b3f286f1}" +2025-10-30T18:21:30.160Z [INFO] server-3c061017-04fb-6529-9320-8c84af8e7a0a: aborting pipeline replication: peer="{Voter server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6 47ecd9b4-9ccf-1272-7a7e-be8779b320f6}" +2025-10-30T18:21:30.250Z [WARN] server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6: heartbeat timeout reached, starting election: last-leader-addr=3c061017-04fb-6529-9320-8c84af8e7a0a last-leader-id=server-3c061017-04fb-6529-9320-8c84af8e7a0a +2025-10-30T18:21:30.250Z [INFO] server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6: entering candidate state: node="Node at 47ecd9b4-9ccf-1272-7a7e-be8779b320f6 [Candidate]" term=3 +2025-10-30T18:21:30.251Z [WARN] server-d3ffe671-0e56-caca-b526-46e2b3f286f1: rejecting vote request since we have a leader: from=47ecd9b4-9ccf-1272-7a7e-be8779b320f6 leader=3c061017-04fb-6529-9320-8c84af8e7a0a leader-id=server-3c061017-04fb-6529-9320-8c84af8e7a0a +2025-10-30T18:21:30.270Z [WARN] server-d3ffe671-0e56-caca-b526-46e2b3f286f1: heartbeat timeout reached, starting election: last-leader-addr=3c061017-04fb-6529-9320-8c84af8e7a0a last-leader-id=server-3c061017-04fb-6529-9320-8c84af8e7a0a +2025-10-30T18:21:30.270Z [INFO] server-d3ffe671-0e56-caca-b526-46e2b3f286f1: entering candidate state: node="Node at d3ffe671-0e56-caca-b526-46e2b3f286f1 [Candidate]" term=3 +2025-10-30T18:21:30.270Z [INFO] server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6: duplicate requestVote for same term: term=3 +2025-10-30T18:21:30.351Z [WARN] server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6: Election timeout reached, restarting election +2025-10-30T18:21:30.351Z [INFO] server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6: entering candidate state: node="Node at 47ecd9b4-9ccf-1272-7a7e-be8779b320f6 [Candidate]" term=4 +2025-10-30T18:21:30.351Z [WARN] server-d3ffe671-0e56-caca-b526-46e2b3f286f1: Election timeout reached, restarting election +2025-10-30T18:21:30.351Z [INFO] server-d3ffe671-0e56-caca-b526-46e2b3f286f1: entering candidate state: node="Node at d3ffe671-0e56-caca-b526-46e2b3f286f1 [Candidate]" term=4 +2025-10-30T18:21:30.351Z [INFO] server-d3ffe671-0e56-caca-b526-46e2b3f286f1: duplicate requestVote for same term: term=4 +2025-10-30T18:21:30.351Z [INFO] server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6: duplicate requestVote for same term: term=4 +2025-10-30T18:21:30.372Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:30.417Z [WARN] server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6: Election timeout reached, restarting election +2025-10-30T18:21:30.417Z [INFO] server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6: entering candidate state: node="Node at 47ecd9b4-9ccf-1272-7a7e-be8779b320f6 [Candidate]" term=5 +2025-10-30T18:21:30.417Z [INFO] server-d3ffe671-0e56-caca-b526-46e2b3f286f1: entering follower state: follower="Node at d3ffe671-0e56-caca-b526-46e2b3f286f1 [Follower]" leader-address= leader-id= +2025-10-30T18:21:30.417Z [INFO] server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6: election won: term=5 tally=2 +2025-10-30T18:21:30.417Z [INFO] server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6: entering leader state: leader="Node at 47ecd9b4-9ccf-1272-7a7e-be8779b320f6 [Leader]" +2025-10-30T18:21:30.417Z [INFO] server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6: added peer, starting replication: peer=server-d3ffe671-0e56-caca-b526-46e2b3f286f1 +2025-10-30T18:21:30.418Z [INFO] server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6: pipelining replication: peer="{Voter server-d3ffe671-0e56-caca-b526-46e2b3f286f1 d3ffe671-0e56-caca-b526-46e2b3f286f1}" +2025-10-30T18:21:30.523Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:30.417999154 +0000 UTC m=+13.491528444 (1 nodes), 45.959751ms from start of poll, 526.88639ms from cluster start. Timeout at 2025-10-30 18:21:30.523001329 +0000 UTC m=+13.596530621, 105.002177ms after stability +2025-10-30T18:21:30.523Z [INFO] server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6: aborting pipeline replication: peer="{Voter server-d3ffe671-0e56-caca-b526-46e2b3f286f1 d3ffe671-0e56-caca-b526-46e2b3f286f1}" +2025-10-30T18:21:31.023Z [ERROR] server-47ecd9b4-9ccf-1272-7a7e-be8779b320f6: failed to heartbeat to: peer=d3ffe671-0e56-caca-b526-46e2b3f286f1 backoff time=10ms error="command timed out" +--- PASS: TestRaft_RemoveLeader (1.13s) === RUN TestRaft_RemoveLeader_NoShutdown -2025-10-30T18:17:44.083Z [INFO] server-f267be62-7adb-2a57-19e0-e546a6611cab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f267be62-7adb-2a57-19e0-e546a6611cab Address:f267be62-7adb-2a57-19e0-e546a6611cab} {Suffrage:Voter ID:server-9b684f15-58f8-aff8-f15b-c87a818f13a8 Address:9b684f15-58f8-aff8-f15b-c87a818f13a8} {Suffrage:Voter ID:server-c1e1d9ec-e29f-660c-69de-e065a1e69537 Address:c1e1d9ec-e29f-660c-69de-e065a1e69537}]" -2025-10-30T18:17:44.083Z [INFO] server-9b684f15-58f8-aff8-f15b-c87a818f13a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f267be62-7adb-2a57-19e0-e546a6611cab Address:f267be62-7adb-2a57-19e0-e546a6611cab} {Suffrage:Voter ID:server-9b684f15-58f8-aff8-f15b-c87a818f13a8 Address:9b684f15-58f8-aff8-f15b-c87a818f13a8} {Suffrage:Voter ID:server-c1e1d9ec-e29f-660c-69de-e065a1e69537 Address:c1e1d9ec-e29f-660c-69de-e065a1e69537}]" -2025-10-30T18:17:44.083Z [INFO] server-f267be62-7adb-2a57-19e0-e546a6611cab: entering follower state: follower="Node at f267be62-7adb-2a57-19e0-e546a6611cab [Follower]" leader-address= leader-id= -2025-10-30T18:17:44.083Z [INFO] server-9b684f15-58f8-aff8-f15b-c87a818f13a8: entering follower state: follower="Node at 9b684f15-58f8-aff8-f15b-c87a818f13a8 [Follower]" leader-address= leader-id= -2025-10-30T18:17:44.083Z [INFO] server-c1e1d9ec-e29f-660c-69de-e065a1e69537: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f267be62-7adb-2a57-19e0-e546a6611cab Address:f267be62-7adb-2a57-19e0-e546a6611cab} {Suffrage:Voter ID:server-9b684f15-58f8-aff8-f15b-c87a818f13a8 Address:9b684f15-58f8-aff8-f15b-c87a818f13a8} {Suffrage:Voter ID:server-c1e1d9ec-e29f-660c-69de-e065a1e69537 Address:c1e1d9ec-e29f-660c-69de-e065a1e69537}]" -2025-10-30T18:17:44.083Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:17:44.083Z [INFO] server-c1e1d9ec-e29f-660c-69de-e065a1e69537: entering follower state: follower="Node at c1e1d9ec-e29f-660c-69de-e065a1e69537 [Follower]" leader-address= leader-id= -2025-10-30T18:17:44.149Z [WARN] server-c1e1d9ec-e29f-660c-69de-e065a1e69537: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:44.149Z [INFO] server-c1e1d9ec-e29f-660c-69de-e065a1e69537: entering candidate state: node="Node at c1e1d9ec-e29f-660c-69de-e065a1e69537 [Candidate]" term=2 -2025-10-30T18:17:44.149Z [INFO] server-c1e1d9ec-e29f-660c-69de-e065a1e69537: election won: term=2 tally=2 -2025-10-30T18:17:44.149Z [INFO] server-c1e1d9ec-e29f-660c-69de-e065a1e69537: entering leader state: leader="Node at c1e1d9ec-e29f-660c-69de-e065a1e69537 [Leader]" -2025-10-30T18:17:44.149Z [INFO] server-c1e1d9ec-e29f-660c-69de-e065a1e69537: added peer, starting replication: peer=server-f267be62-7adb-2a57-19e0-e546a6611cab -2025-10-30T18:17:44.149Z [INFO] server-c1e1d9ec-e29f-660c-69de-e065a1e69537: added peer, starting replication: peer=server-9b684f15-58f8-aff8-f15b-c87a818f13a8 -2025-10-30T18:17:44.149Z [INFO] server-c1e1d9ec-e29f-660c-69de-e065a1e69537: pipelining replication: peer="{Voter server-f267be62-7adb-2a57-19e0-e546a6611cab f267be62-7adb-2a57-19e0-e546a6611cab}" -2025-10-30T18:17:44.149Z [INFO] server-c1e1d9ec-e29f-660c-69de-e065a1e69537: pipelining replication: peer="{Voter server-9b684f15-58f8-aff8-f15b-c87a818f13a8 9b684f15-58f8-aff8-f15b-c87a818f13a8}" -2025-10-30T18:17:44.255Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:17:44.149644525 +0000 UTC m=+10.428318424 (2 nodes), 65.844399ms from start of poll, 66.084214ms from cluster start. Timeout at 2025-10-30 18:17:44.254647951 +0000 UTC m=+10.533321859, 105.003435ms after stability -2025-10-30T18:17:44.255Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:44.360Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:44.255566222 +0000 UTC m=+10.534240120 (1 nodes), 2.137µs from start of poll, 172.00591ms from cluster start. Timeout at 2025-10-30 18:17:44.360571574 +0000 UTC m=+10.639245480, 105.00536ms after stability -2025-10-30T18:17:44.362Z [INFO] server-c1e1d9ec-e29f-660c-69de-e065a1e69537: updating configuration: command=RemoveServer server-id=server-c1e1d9ec-e29f-660c-69de-e065a1e69537 server-addr= servers="[{Suffrage:Voter ID:server-f267be62-7adb-2a57-19e0-e546a6611cab Address:f267be62-7adb-2a57-19e0-e546a6611cab} {Suffrage:Voter ID:server-9b684f15-58f8-aff8-f15b-c87a818f13a8 Address:9b684f15-58f8-aff8-f15b-c87a818f13a8}]" -2025-10-30T18:17:44.364Z [INFO] server-c1e1d9ec-e29f-660c-69de-e065a1e69537: removed ourself, transitioning to follower -2025-10-30T18:17:44.364Z [INFO] server-c1e1d9ec-e29f-660c-69de-e065a1e69537: entering follower state: follower="Node at c1e1d9ec-e29f-660c-69de-e065a1e69537 [Follower]" leader-address= leader-id= -2025-10-30T18:17:44.364Z [INFO] server-c1e1d9ec-e29f-660c-69de-e065a1e69537: aborting pipeline replication: peer="{Voter server-9b684f15-58f8-aff8-f15b-c87a818f13a8 9b684f15-58f8-aff8-f15b-c87a818f13a8}" -2025-10-30T18:17:44.364Z [INFO] server-c1e1d9ec-e29f-660c-69de-e065a1e69537: aborting pipeline replication: peer="{Voter server-f267be62-7adb-2a57-19e0-e546a6611cab f267be62-7adb-2a57-19e0-e546a6611cab}" -2025-10-30T18:17:44.421Z [WARN] server-9b684f15-58f8-aff8-f15b-c87a818f13a8: heartbeat timeout reached, starting election: last-leader-addr=c1e1d9ec-e29f-660c-69de-e065a1e69537 last-leader-id=server-c1e1d9ec-e29f-660c-69de-e065a1e69537 -2025-10-30T18:17:44.421Z [INFO] server-9b684f15-58f8-aff8-f15b-c87a818f13a8: entering candidate state: node="Node at 9b684f15-58f8-aff8-f15b-c87a818f13a8 [Candidate]" term=3 -2025-10-30T18:17:44.421Z [WARN] server-f267be62-7adb-2a57-19e0-e546a6611cab: rejecting vote request since we have a leader: from=9b684f15-58f8-aff8-f15b-c87a818f13a8 leader=c1e1d9ec-e29f-660c-69de-e065a1e69537 leader-id=server-c1e1d9ec-e29f-660c-69de-e065a1e69537 -2025-10-30T18:17:44.429Z [WARN] server-f267be62-7adb-2a57-19e0-e546a6611cab: heartbeat timeout reached, starting election: last-leader-addr=c1e1d9ec-e29f-660c-69de-e065a1e69537 last-leader-id=server-c1e1d9ec-e29f-660c-69de-e065a1e69537 -2025-10-30T18:17:44.429Z [INFO] server-f267be62-7adb-2a57-19e0-e546a6611cab: entering candidate state: node="Node at f267be62-7adb-2a57-19e0-e546a6611cab [Candidate]" term=3 -2025-10-30T18:17:44.429Z [INFO] server-9b684f15-58f8-aff8-f15b-c87a818f13a8: duplicate requestVote for same term: term=3 -2025-10-30T18:17:44.435Z [WARN] server-c1e1d9ec-e29f-660c-69de-e065a1e69537: not part of stable configuration, aborting election -2025-10-30T18:17:44.470Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:44.494Z [WARN] server-f267be62-7adb-2a57-19e0-e546a6611cab: Election timeout reached, restarting election -2025-10-30T18:17:44.494Z [INFO] server-f267be62-7adb-2a57-19e0-e546a6611cab: entering candidate state: node="Node at f267be62-7adb-2a57-19e0-e546a6611cab [Candidate]" term=4 -2025-10-30T18:17:44.494Z [INFO] server-9b684f15-58f8-aff8-f15b-c87a818f13a8: entering follower state: follower="Node at 9b684f15-58f8-aff8-f15b-c87a818f13a8 [Follower]" leader-address= leader-id= -2025-10-30T18:17:44.494Z [INFO] server-f267be62-7adb-2a57-19e0-e546a6611cab: election won: term=4 tally=2 -2025-10-30T18:17:44.494Z [INFO] server-f267be62-7adb-2a57-19e0-e546a6611cab: entering leader state: leader="Node at f267be62-7adb-2a57-19e0-e546a6611cab [Leader]" -2025-10-30T18:17:44.494Z [INFO] server-f267be62-7adb-2a57-19e0-e546a6611cab: added peer, starting replication: peer=server-9b684f15-58f8-aff8-f15b-c87a818f13a8 -2025-10-30T18:17:44.494Z [INFO] server-f267be62-7adb-2a57-19e0-e546a6611cab: pipelining replication: peer="{Voter server-9b684f15-58f8-aff8-f15b-c87a818f13a8 9b684f15-58f8-aff8-f15b-c87a818f13a8}" -2025-10-30T18:17:44.600Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:44.494728577 +0000 UTC m=+10.773402484 (1 nodes), 24.30947ms from start of poll, 411.168274ms from cluster start. Timeout at 2025-10-30 18:17:44.599730528 +0000 UTC m=+10.878404426, 105.001942ms after stability -2025-10-30T18:17:44.705Z [INFO] server-f267be62-7adb-2a57-19e0-e546a6611cab: aborting pipeline replication: peer="{Voter server-9b684f15-58f8-aff8-f15b-c87a818f13a8 9b684f15-58f8-aff8-f15b-c87a818f13a8}" ---- PASS: TestRaft_RemoveLeader_NoShutdown (0.62s) +2025-10-30T18:21:31.025Z [INFO] server-fd759145-96a0-94e6-57f0-d9b9720a9953: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd759145-96a0-94e6-57f0-d9b9720a9953 Address:fd759145-96a0-94e6-57f0-d9b9720a9953} {Suffrage:Voter ID:server-65aebb4e-ecd9-f913-5432-f4be006b6432 Address:65aebb4e-ecd9-f913-5432-f4be006b6432} {Suffrage:Voter ID:server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5 Address:4dbc09b2-d051-e7ae-55a0-07a7a42b06e5}]" +2025-10-30T18:21:31.025Z [INFO] server-fd759145-96a0-94e6-57f0-d9b9720a9953: entering follower state: follower="Node at fd759145-96a0-94e6-57f0-d9b9720a9953 [Follower]" leader-address= leader-id= +2025-10-30T18:21:31.025Z [INFO] server-65aebb4e-ecd9-f913-5432-f4be006b6432: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd759145-96a0-94e6-57f0-d9b9720a9953 Address:fd759145-96a0-94e6-57f0-d9b9720a9953} {Suffrage:Voter ID:server-65aebb4e-ecd9-f913-5432-f4be006b6432 Address:65aebb4e-ecd9-f913-5432-f4be006b6432} {Suffrage:Voter ID:server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5 Address:4dbc09b2-d051-e7ae-55a0-07a7a42b06e5}]" +2025-10-30T18:21:31.025Z [INFO] server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd759145-96a0-94e6-57f0-d9b9720a9953 Address:fd759145-96a0-94e6-57f0-d9b9720a9953} {Suffrage:Voter ID:server-65aebb4e-ecd9-f913-5432-f4be006b6432 Address:65aebb4e-ecd9-f913-5432-f4be006b6432} {Suffrage:Voter ID:server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5 Address:4dbc09b2-d051-e7ae-55a0-07a7a42b06e5}]" +2025-10-30T18:21:31.025Z [INFO] server-65aebb4e-ecd9-f913-5432-f4be006b6432: entering follower state: follower="Node at 65aebb4e-ecd9-f913-5432-f4be006b6432 [Follower]" leader-address= leader-id= +2025-10-30T18:21:31.025Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:21:31.025Z [INFO] server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5: entering follower state: follower="Node at 4dbc09b2-d051-e7ae-55a0-07a7a42b06e5 [Follower]" leader-address= leader-id= +2025-10-30T18:21:31.076Z [WARN] server-fd759145-96a0-94e6-57f0-d9b9720a9953: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:31.076Z [INFO] server-fd759145-96a0-94e6-57f0-d9b9720a9953: entering candidate state: node="Node at fd759145-96a0-94e6-57f0-d9b9720a9953 [Candidate]" term=2 +2025-10-30T18:21:31.076Z [INFO] server-fd759145-96a0-94e6-57f0-d9b9720a9953: election won: term=2 tally=2 +2025-10-30T18:21:31.076Z [INFO] server-fd759145-96a0-94e6-57f0-d9b9720a9953: entering leader state: leader="Node at fd759145-96a0-94e6-57f0-d9b9720a9953 [Leader]" +2025-10-30T18:21:31.076Z [INFO] server-fd759145-96a0-94e6-57f0-d9b9720a9953: added peer, starting replication: peer=server-65aebb4e-ecd9-f913-5432-f4be006b6432 +2025-10-30T18:21:31.076Z [INFO] server-fd759145-96a0-94e6-57f0-d9b9720a9953: added peer, starting replication: peer=server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5 +2025-10-30T18:21:31.076Z [INFO] server-fd759145-96a0-94e6-57f0-d9b9720a9953: pipelining replication: peer="{Voter server-65aebb4e-ecd9-f913-5432-f4be006b6432 65aebb4e-ecd9-f913-5432-f4be006b6432}" +2025-10-30T18:21:31.076Z [INFO] server-fd759145-96a0-94e6-57f0-d9b9720a9953: pipelining replication: peer="{Voter server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5 4dbc09b2-d051-e7ae-55a0-07a7a42b06e5}" +2025-10-30T18:21:31.182Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:21:31.076609398 +0000 UTC m=+14.150138686 (2 nodes), 51.133844ms from start of poll, 51.522168ms from cluster start. Timeout at 2025-10-30 18:21:31.181686239 +0000 UTC m=+14.255215545, 105.076859ms after stability +2025-10-30T18:21:31.182Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:31.288Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:31.182900829 +0000 UTC m=+14.256430134 (1 nodes), 1.036µs from start of poll, 157.813616ms from cluster start. Timeout at 2025-10-30 18:21:31.287907286 +0000 UTC m=+14.361436582, 105.006448ms after stability +2025-10-30T18:21:31.292Z [INFO] server-fd759145-96a0-94e6-57f0-d9b9720a9953: updating configuration: command=RemoveServer server-id=server-fd759145-96a0-94e6-57f0-d9b9720a9953 server-addr= servers="[{Suffrage:Voter ID:server-65aebb4e-ecd9-f913-5432-f4be006b6432 Address:65aebb4e-ecd9-f913-5432-f4be006b6432} {Suffrage:Voter ID:server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5 Address:4dbc09b2-d051-e7ae-55a0-07a7a42b06e5}]" +2025-10-30T18:21:31.292Z [INFO] server-fd759145-96a0-94e6-57f0-d9b9720a9953: removed ourself, transitioning to follower +2025-10-30T18:21:31.292Z [INFO] server-fd759145-96a0-94e6-57f0-d9b9720a9953: entering follower state: follower="Node at fd759145-96a0-94e6-57f0-d9b9720a9953 [Follower]" leader-address= leader-id= +2025-10-30T18:21:31.292Z [INFO] server-fd759145-96a0-94e6-57f0-d9b9720a9953: aborting pipeline replication: peer="{Voter server-65aebb4e-ecd9-f913-5432-f4be006b6432 65aebb4e-ecd9-f913-5432-f4be006b6432}" +2025-10-30T18:21:31.292Z [INFO] server-fd759145-96a0-94e6-57f0-d9b9720a9953: aborting pipeline replication: peer="{Voter server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5 4dbc09b2-d051-e7ae-55a0-07a7a42b06e5}" +2025-10-30T18:21:31.353Z [WARN] server-fd759145-96a0-94e6-57f0-d9b9720a9953: not part of stable configuration, aborting election +2025-10-30T18:21:31.377Z [WARN] server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5: heartbeat timeout reached, starting election: last-leader-addr=fd759145-96a0-94e6-57f0-d9b9720a9953 last-leader-id=server-fd759145-96a0-94e6-57f0-d9b9720a9953 +2025-10-30T18:21:31.377Z [INFO] server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5: entering candidate state: node="Node at 4dbc09b2-d051-e7ae-55a0-07a7a42b06e5 [Candidate]" term=3 +2025-10-30T18:21:31.377Z [WARN] server-65aebb4e-ecd9-f913-5432-f4be006b6432: rejecting vote request since we have a leader: from=4dbc09b2-d051-e7ae-55a0-07a7a42b06e5 leader=fd759145-96a0-94e6-57f0-d9b9720a9953 leader-id=server-fd759145-96a0-94e6-57f0-d9b9720a9953 +2025-10-30T18:21:31.398Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:31.403Z [WARN] server-65aebb4e-ecd9-f913-5432-f4be006b6432: heartbeat timeout reached, starting election: last-leader-addr=fd759145-96a0-94e6-57f0-d9b9720a9953 last-leader-id=server-fd759145-96a0-94e6-57f0-d9b9720a9953 +2025-10-30T18:21:31.403Z [INFO] server-65aebb4e-ecd9-f913-5432-f4be006b6432: entering candidate state: node="Node at 65aebb4e-ecd9-f913-5432-f4be006b6432 [Candidate]" term=3 +2025-10-30T18:21:31.403Z [INFO] server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5: duplicate requestVote for same term: term=3 +2025-10-30T18:21:31.446Z [WARN] server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5: Election timeout reached, restarting election +2025-10-30T18:21:31.446Z [INFO] server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5: entering candidate state: node="Node at 4dbc09b2-d051-e7ae-55a0-07a7a42b06e5 [Candidate]" term=4 +2025-10-30T18:21:31.446Z [INFO] server-65aebb4e-ecd9-f913-5432-f4be006b6432: entering follower state: follower="Node at 65aebb4e-ecd9-f913-5432-f4be006b6432 [Follower]" leader-address= leader-id= +2025-10-30T18:21:31.446Z [INFO] server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5: election won: term=4 tally=2 +2025-10-30T18:21:31.446Z [INFO] server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5: entering leader state: leader="Node at 4dbc09b2-d051-e7ae-55a0-07a7a42b06e5 [Leader]" +2025-10-30T18:21:31.446Z [INFO] server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5: added peer, starting replication: peer=server-65aebb4e-ecd9-f913-5432-f4be006b6432 +2025-10-30T18:21:31.446Z [INFO] server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5: pipelining replication: peer="{Voter server-65aebb4e-ecd9-f913-5432-f4be006b6432 65aebb4e-ecd9-f913-5432-f4be006b6432}" +2025-10-30T18:21:31.551Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:31.446272929 +0000 UTC m=+14.519802218 (1 nodes), 47.968248ms from start of poll, 421.1857ms from cluster start. Timeout at 2025-10-30 18:21:31.55127521 +0000 UTC m=+14.624804504, 105.002286ms after stability +2025-10-30T18:21:31.657Z [INFO] server-4dbc09b2-d051-e7ae-55a0-07a7a42b06e5: aborting pipeline replication: peer="{Voter server-65aebb4e-ecd9-f913-5432-f4be006b6432 65aebb4e-ecd9-f913-5432-f4be006b6432}" +--- PASS: TestRaft_RemoveLeader_NoShutdown (0.72s) === RUN TestRaft_RemoveFollower_SplitCluster -2025-10-30T18:17:44.706Z [INFO] server-1b28831c-996a-2bea-0305-4c5109095eb6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b28831c-996a-2bea-0305-4c5109095eb6 Address:1b28831c-996a-2bea-0305-4c5109095eb6} {Suffrage:Voter ID:server-e0597101-cc35-a45e-cd75-9fd2031d50fa Address:e0597101-cc35-a45e-cd75-9fd2031d50fa} {Suffrage:Voter ID:server-49f68783-af41-9ea5-b459-040d669e337e Address:49f68783-af41-9ea5-b459-040d669e337e} {Suffrage:Voter ID:server-a6720c93-5bd0-bcde-7597-50e4e7104721 Address:a6720c93-5bd0-bcde-7597-50e4e7104721}]" -2025-10-30T18:17:44.706Z [INFO] server-1b28831c-996a-2bea-0305-4c5109095eb6: entering follower state: follower="Node at 1b28831c-996a-2bea-0305-4c5109095eb6 [Follower]" leader-address= leader-id= -2025-10-30T18:17:44.706Z [INFO] server-e0597101-cc35-a45e-cd75-9fd2031d50fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b28831c-996a-2bea-0305-4c5109095eb6 Address:1b28831c-996a-2bea-0305-4c5109095eb6} {Suffrage:Voter ID:server-e0597101-cc35-a45e-cd75-9fd2031d50fa Address:e0597101-cc35-a45e-cd75-9fd2031d50fa} {Suffrage:Voter ID:server-49f68783-af41-9ea5-b459-040d669e337e Address:49f68783-af41-9ea5-b459-040d669e337e} {Suffrage:Voter ID:server-a6720c93-5bd0-bcde-7597-50e4e7104721 Address:a6720c93-5bd0-bcde-7597-50e4e7104721}]" -2025-10-30T18:17:44.706Z [INFO] server-49f68783-af41-9ea5-b459-040d669e337e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b28831c-996a-2bea-0305-4c5109095eb6 Address:1b28831c-996a-2bea-0305-4c5109095eb6} {Suffrage:Voter ID:server-e0597101-cc35-a45e-cd75-9fd2031d50fa Address:e0597101-cc35-a45e-cd75-9fd2031d50fa} {Suffrage:Voter ID:server-49f68783-af41-9ea5-b459-040d669e337e Address:49f68783-af41-9ea5-b459-040d669e337e} {Suffrage:Voter ID:server-a6720c93-5bd0-bcde-7597-50e4e7104721 Address:a6720c93-5bd0-bcde-7597-50e4e7104721}]" -2025-10-30T18:17:44.706Z [INFO] server-e0597101-cc35-a45e-cd75-9fd2031d50fa: entering follower state: follower="Node at e0597101-cc35-a45e-cd75-9fd2031d50fa [Follower]" leader-address= leader-id= -2025-10-30T18:17:44.706Z [INFO] server-49f68783-af41-9ea5-b459-040d669e337e: entering follower state: follower="Node at 49f68783-af41-9ea5-b459-040d669e337e [Follower]" leader-address= leader-id= -2025-10-30T18:17:44.706Z [INFO] server-a6720c93-5bd0-bcde-7597-50e4e7104721: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b28831c-996a-2bea-0305-4c5109095eb6 Address:1b28831c-996a-2bea-0305-4c5109095eb6} {Suffrage:Voter ID:server-e0597101-cc35-a45e-cd75-9fd2031d50fa Address:e0597101-cc35-a45e-cd75-9fd2031d50fa} {Suffrage:Voter ID:server-49f68783-af41-9ea5-b459-040d669e337e Address:49f68783-af41-9ea5-b459-040d669e337e} {Suffrage:Voter ID:server-a6720c93-5bd0-bcde-7597-50e4e7104721 Address:a6720c93-5bd0-bcde-7597-50e4e7104721}]" -2025-10-30T18:17:44.706Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:44.706Z [INFO] server-a6720c93-5bd0-bcde-7597-50e4e7104721: entering follower state: follower="Node at a6720c93-5bd0-bcde-7597-50e4e7104721 [Follower]" leader-address= leader-id= -2025-10-30T18:17:44.773Z [WARN] server-49f68783-af41-9ea5-b459-040d669e337e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:44.773Z [INFO] server-49f68783-af41-9ea5-b459-040d669e337e: entering candidate state: node="Node at 49f68783-af41-9ea5-b459-040d669e337e [Candidate]" term=2 -2025-10-30T18:17:44.774Z [INFO] server-49f68783-af41-9ea5-b459-040d669e337e: election won: term=2 tally=3 -2025-10-30T18:17:44.774Z [INFO] server-49f68783-af41-9ea5-b459-040d669e337e: entering leader state: leader="Node at 49f68783-af41-9ea5-b459-040d669e337e [Leader]" -2025-10-30T18:17:44.774Z [INFO] server-49f68783-af41-9ea5-b459-040d669e337e: added peer, starting replication: peer=server-1b28831c-996a-2bea-0305-4c5109095eb6 -2025-10-30T18:17:44.774Z [INFO] server-49f68783-af41-9ea5-b459-040d669e337e: added peer, starting replication: peer=server-e0597101-cc35-a45e-cd75-9fd2031d50fa -2025-10-30T18:17:44.774Z [INFO] server-49f68783-af41-9ea5-b459-040d669e337e: added peer, starting replication: peer=server-a6720c93-5bd0-bcde-7597-50e4e7104721 -2025-10-30T18:17:44.774Z [INFO] server-49f68783-af41-9ea5-b459-040d669e337e: pipelining replication: peer="{Voter server-a6720c93-5bd0-bcde-7597-50e4e7104721 a6720c93-5bd0-bcde-7597-50e4e7104721}" -2025-10-30T18:17:44.774Z [INFO] server-49f68783-af41-9ea5-b459-040d669e337e: pipelining replication: peer="{Voter server-1b28831c-996a-2bea-0305-4c5109095eb6 1b28831c-996a-2bea-0305-4c5109095eb6}" -2025-10-30T18:17:44.774Z [INFO] server-49f68783-af41-9ea5-b459-040d669e337e: pipelining replication: peer="{Voter server-e0597101-cc35-a45e-cd75-9fd2031d50fa e0597101-cc35-a45e-cd75-9fd2031d50fa}" -2025-10-30T18:17:44.879Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:44.774090643 +0000 UTC m=+11.052764552 (1 nodes), 67.460259ms from start of poll, 67.854274ms from cluster start. Timeout at 2025-10-30 18:17:44.879092561 +0000 UTC m=+11.157766458, 105.001906ms after stability -2025-10-30T18:17:44.985Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:17:45.090Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:17:44.985520123 +0000 UTC m=+11.264194020 (3 nodes), 2.491µs from start of poll, 279.283742ms from cluster start. Timeout at 2025-10-30 18:17:45.090526232 +0000 UTC m=+11.369200130, 105.00611ms after stability -2025-10-30T18:17:45.090Z [INFO] server-49f68783-af41-9ea5-b459-040d669e337e: updating configuration: command=RemoveServer server-id=server-a6720c93-5bd0-bcde-7597-50e4e7104721 server-addr= servers="[{Suffrage:Voter ID:server-1b28831c-996a-2bea-0305-4c5109095eb6 Address:1b28831c-996a-2bea-0305-4c5109095eb6} {Suffrage:Voter ID:server-e0597101-cc35-a45e-cd75-9fd2031d50fa Address:e0597101-cc35-a45e-cd75-9fd2031d50fa} {Suffrage:Voter ID:server-49f68783-af41-9ea5-b459-040d669e337e Address:49f68783-af41-9ea5-b459-040d669e337e}]" -2025-10-30T18:17:45.090Z [INFO] server-49f68783-af41-9ea5-b459-040d669e337e: removed peer, stopping replication: peer=server-a6720c93-5bd0-bcde-7597-50e4e7104721 last-index=3 -2025-10-30T18:17:45.090Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to pipeline appendEntries: peer="{Voter server-1b28831c-996a-2bea-0305-4c5109095eb6 1b28831c-996a-2bea-0305-4c5109095eb6}" error="append pipeline closed" -2025-10-30T18:17:45.090Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to pipeline appendEntries: peer="{Voter server-e0597101-cc35-a45e-cd75-9fd2031d50fa e0597101-cc35-a45e-cd75-9fd2031d50fa}" error="append pipeline closed" -2025-10-30T18:17:45.091Z [INFO] server-49f68783-af41-9ea5-b459-040d669e337e: aborting pipeline replication: peer="{Voter server-e0597101-cc35-a45e-cd75-9fd2031d50fa e0597101-cc35-a45e-cd75-9fd2031d50fa}" -2025-10-30T18:17:45.090Z [INFO] server-49f68783-af41-9ea5-b459-040d669e337e: aborting pipeline replication: peer="{Voter server-1b28831c-996a-2bea-0305-4c5109095eb6 1b28831c-996a-2bea-0305-4c5109095eb6}" -2025-10-30T18:17:45.090Z [INFO] server-49f68783-af41-9ea5-b459-040d669e337e: aborting pipeline replication: peer="{Voter server-a6720c93-5bd0-bcde-7597-50e4e7104721 a6720c93-5bd0-bcde-7597-50e4e7104721}" -2025-10-30T18:17:45.091Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to heartbeat to: peer=1b28831c-996a-2bea-0305-4c5109095eb6 backoff time=10ms error="failed to connect to peer: 1b28831c-996a-2bea-0305-4c5109095eb6" -2025-10-30T18:17:45.097Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to appendEntries to: peer="{Voter server-1b28831c-996a-2bea-0305-4c5109095eb6 1b28831c-996a-2bea-0305-4c5109095eb6}" error="failed to connect to peer: 1b28831c-996a-2bea-0305-4c5109095eb6" -2025-10-30T18:17:45.097Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to appendEntries to: peer="{Voter server-e0597101-cc35-a45e-cd75-9fd2031d50fa e0597101-cc35-a45e-cd75-9fd2031d50fa}" error="failed to connect to peer: e0597101-cc35-a45e-cd75-9fd2031d50fa" -2025-10-30T18:17:45.099Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to heartbeat to: peer=e0597101-cc35-a45e-cd75-9fd2031d50fa backoff time=10ms error="failed to connect to peer: e0597101-cc35-a45e-cd75-9fd2031d50fa" -2025-10-30T18:17:45.109Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to heartbeat to: peer=1b28831c-996a-2bea-0305-4c5109095eb6 backoff time=10ms error="failed to connect to peer: 1b28831c-996a-2bea-0305-4c5109095eb6" -2025-10-30T18:17:45.115Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to appendEntries to: peer="{Voter server-1b28831c-996a-2bea-0305-4c5109095eb6 1b28831c-996a-2bea-0305-4c5109095eb6}" error="failed to connect to peer: 1b28831c-996a-2bea-0305-4c5109095eb6" -2025-10-30T18:17:45.116Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to appendEntries to: peer="{Voter server-e0597101-cc35-a45e-cd75-9fd2031d50fa e0597101-cc35-a45e-cd75-9fd2031d50fa}" error="failed to connect to peer: e0597101-cc35-a45e-cd75-9fd2031d50fa" -2025-10-30T18:17:45.117Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to heartbeat to: peer=e0597101-cc35-a45e-cd75-9fd2031d50fa backoff time=10ms error="failed to connect to peer: e0597101-cc35-a45e-cd75-9fd2031d50fa" -2025-10-30T18:17:45.128Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to heartbeat to: peer=1b28831c-996a-2bea-0305-4c5109095eb6 backoff time=10ms error="failed to connect to peer: 1b28831c-996a-2bea-0305-4c5109095eb6" -2025-10-30T18:17:45.135Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to appendEntries to: peer="{Voter server-e0597101-cc35-a45e-cd75-9fd2031d50fa e0597101-cc35-a45e-cd75-9fd2031d50fa}" error="failed to connect to peer: e0597101-cc35-a45e-cd75-9fd2031d50fa" -2025-10-30T18:17:45.136Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to heartbeat to: peer=e0597101-cc35-a45e-cd75-9fd2031d50fa backoff time=10ms error="failed to connect to peer: e0597101-cc35-a45e-cd75-9fd2031d50fa" -2025-10-30T18:17:45.136Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to appendEntries to: peer="{Voter server-1b28831c-996a-2bea-0305-4c5109095eb6 1b28831c-996a-2bea-0305-4c5109095eb6}" error="failed to connect to peer: 1b28831c-996a-2bea-0305-4c5109095eb6" -2025-10-30T18:17:45.136Z [WARN] server-49f68783-af41-9ea5-b459-040d669e337e: failed to contact: server-id=server-1b28831c-996a-2bea-0305-4c5109095eb6 time=50.09139ms -2025-10-30T18:17:45.140Z [WARN] server-e0597101-cc35-a45e-cd75-9fd2031d50fa: heartbeat timeout reached, starting election: last-leader-addr=49f68783-af41-9ea5-b459-040d669e337e last-leader-id=server-49f68783-af41-9ea5-b459-040d669e337e -2025-10-30T18:17:45.140Z [INFO] server-e0597101-cc35-a45e-cd75-9fd2031d50fa: entering candidate state: node="Node at e0597101-cc35-a45e-cd75-9fd2031d50fa [Candidate]" term=3 -2025-10-30T18:17:45.140Z [ERROR] server-e0597101-cc35-a45e-cd75-9fd2031d50fa: failed to make requestVote RPC: target="{Voter server-a6720c93-5bd0-bcde-7597-50e4e7104721 a6720c93-5bd0-bcde-7597-50e4e7104721}" error="failed to connect to peer: a6720c93-5bd0-bcde-7597-50e4e7104721" term=3 -2025-10-30T18:17:45.140Z [WARN] server-1b28831c-996a-2bea-0305-4c5109095eb6: rejecting vote request since we have a leader: from=e0597101-cc35-a45e-cd75-9fd2031d50fa leader=49f68783-af41-9ea5-b459-040d669e337e leader-id=server-49f68783-af41-9ea5-b459-040d669e337e -2025-10-30T18:17:45.140Z [ERROR] server-e0597101-cc35-a45e-cd75-9fd2031d50fa: failed to make requestVote RPC: target="{Voter server-49f68783-af41-9ea5-b459-040d669e337e 49f68783-af41-9ea5-b459-040d669e337e}" error="failed to connect to peer: 49f68783-af41-9ea5-b459-040d669e337e" term=3 -2025-10-30T18:17:45.147Z [WARN] server-49f68783-af41-9ea5-b459-040d669e337e: failed to contact: server-id=server-1b28831c-996a-2bea-0305-4c5109095eb6 time=60.953573ms -2025-10-30T18:17:45.147Z [WARN] server-49f68783-af41-9ea5-b459-040d669e337e: failed to contact: server-id=server-e0597101-cc35-a45e-cd75-9fd2031d50fa time=57.712392ms -2025-10-30T18:17:45.147Z [WARN] server-49f68783-af41-9ea5-b459-040d669e337e: failed to contact quorum of nodes, stepping down -2025-10-30T18:17:45.147Z [INFO] server-49f68783-af41-9ea5-b459-040d669e337e: entering follower state: follower="Node at 49f68783-af41-9ea5-b459-040d669e337e [Follower]" leader-address= leader-id= -2025-10-30T18:17:45.147Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to appendEntries to: peer="{Voter server-1b28831c-996a-2bea-0305-4c5109095eb6 1b28831c-996a-2bea-0305-4c5109095eb6}" error="failed to connect to peer: 1b28831c-996a-2bea-0305-4c5109095eb6" -2025-10-30T18:17:45.147Z [ERROR] server-49f68783-af41-9ea5-b459-040d669e337e: failed to appendEntries to: peer="{Voter server-e0597101-cc35-a45e-cd75-9fd2031d50fa e0597101-cc35-a45e-cd75-9fd2031d50fa}" error="failed to connect to peer: e0597101-cc35-a45e-cd75-9fd2031d50fa" ---- PASS: TestRaft_RemoveFollower_SplitCluster (0.44s) +2025-10-30T18:21:31.741Z [INFO] server-af42de2c-87c6-f84a-0c8e-69b292821286: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af42de2c-87c6-f84a-0c8e-69b292821286 Address:af42de2c-87c6-f84a-0c8e-69b292821286} {Suffrage:Voter ID:server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0 Address:25c9ea00-ca51-f3fa-0051-e3c0ddc921c0} {Suffrage:Voter ID:server-b7a9c854-71aa-14d2-2ad1-bc75b4704fb9 Address:b7a9c854-71aa-14d2-2ad1-bc75b4704fb9} {Suffrage:Voter ID:server-258e18c4-6f01-3a05-53e4-2d4a6a991c5a Address:258e18c4-6f01-3a05-53e4-2d4a6a991c5a}]" +2025-10-30T18:21:31.741Z [INFO] server-af42de2c-87c6-f84a-0c8e-69b292821286: entering follower state: follower="Node at af42de2c-87c6-f84a-0c8e-69b292821286 [Follower]" leader-address= leader-id= +2025-10-30T18:21:31.741Z [INFO] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af42de2c-87c6-f84a-0c8e-69b292821286 Address:af42de2c-87c6-f84a-0c8e-69b292821286} {Suffrage:Voter ID:server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0 Address:25c9ea00-ca51-f3fa-0051-e3c0ddc921c0} {Suffrage:Voter ID:server-b7a9c854-71aa-14d2-2ad1-bc75b4704fb9 Address:b7a9c854-71aa-14d2-2ad1-bc75b4704fb9} {Suffrage:Voter ID:server-258e18c4-6f01-3a05-53e4-2d4a6a991c5a Address:258e18c4-6f01-3a05-53e4-2d4a6a991c5a}]" +2025-10-30T18:21:31.741Z [INFO] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: entering follower state: follower="Node at 25c9ea00-ca51-f3fa-0051-e3c0ddc921c0 [Follower]" leader-address= leader-id= +2025-10-30T18:21:31.741Z [INFO] server-b7a9c854-71aa-14d2-2ad1-bc75b4704fb9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af42de2c-87c6-f84a-0c8e-69b292821286 Address:af42de2c-87c6-f84a-0c8e-69b292821286} {Suffrage:Voter ID:server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0 Address:25c9ea00-ca51-f3fa-0051-e3c0ddc921c0} {Suffrage:Voter ID:server-b7a9c854-71aa-14d2-2ad1-bc75b4704fb9 Address:b7a9c854-71aa-14d2-2ad1-bc75b4704fb9} {Suffrage:Voter ID:server-258e18c4-6f01-3a05-53e4-2d4a6a991c5a Address:258e18c4-6f01-3a05-53e4-2d4a6a991c5a}]" +2025-10-30T18:21:31.741Z [INFO] server-258e18c4-6f01-3a05-53e4-2d4a6a991c5a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af42de2c-87c6-f84a-0c8e-69b292821286 Address:af42de2c-87c6-f84a-0c8e-69b292821286} {Suffrage:Voter ID:server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0 Address:25c9ea00-ca51-f3fa-0051-e3c0ddc921c0} {Suffrage:Voter ID:server-b7a9c854-71aa-14d2-2ad1-bc75b4704fb9 Address:b7a9c854-71aa-14d2-2ad1-bc75b4704fb9} {Suffrage:Voter ID:server-258e18c4-6f01-3a05-53e4-2d4a6a991c5a Address:258e18c4-6f01-3a05-53e4-2d4a6a991c5a}]" +2025-10-30T18:21:31.741Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:31.741Z [INFO] server-258e18c4-6f01-3a05-53e4-2d4a6a991c5a: entering follower state: follower="Node at 258e18c4-6f01-3a05-53e4-2d4a6a991c5a [Follower]" leader-address= leader-id= +2025-10-30T18:21:31.741Z [INFO] server-b7a9c854-71aa-14d2-2ad1-bc75b4704fb9: entering follower state: follower="Node at b7a9c854-71aa-14d2-2ad1-bc75b4704fb9 [Follower]" leader-address= leader-id= +2025-10-30T18:21:31.797Z [WARN] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:31.797Z [INFO] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: entering candidate state: node="Node at 25c9ea00-ca51-f3fa-0051-e3c0ddc921c0 [Candidate]" term=2 +2025-10-30T18:21:31.797Z [INFO] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: election won: term=2 tally=3 +2025-10-30T18:21:31.797Z [INFO] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: entering leader state: leader="Node at 25c9ea00-ca51-f3fa-0051-e3c0ddc921c0 [Leader]" +2025-10-30T18:21:31.797Z [INFO] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: added peer, starting replication: peer=server-af42de2c-87c6-f84a-0c8e-69b292821286 +2025-10-30T18:21:31.797Z [INFO] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: added peer, starting replication: peer=server-b7a9c854-71aa-14d2-2ad1-bc75b4704fb9 +2025-10-30T18:21:31.797Z [INFO] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: added peer, starting replication: peer=server-258e18c4-6f01-3a05-53e4-2d4a6a991c5a +2025-10-30T18:21:31.797Z [INFO] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: pipelining replication: peer="{Voter server-258e18c4-6f01-3a05-53e4-2d4a6a991c5a 258e18c4-6f01-3a05-53e4-2d4a6a991c5a}" +2025-10-30T18:21:31.797Z [INFO] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: pipelining replication: peer="{Voter server-b7a9c854-71aa-14d2-2ad1-bc75b4704fb9 b7a9c854-71aa-14d2-2ad1-bc75b4704fb9}" +2025-10-30T18:21:31.798Z [INFO] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: pipelining replication: peer="{Voter server-af42de2c-87c6-f84a-0c8e-69b292821286 af42de2c-87c6-f84a-0c8e-69b292821286}" +2025-10-30T18:21:31.903Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:31.797772302 +0000 UTC m=+14.871301595 (1 nodes), 56.376939ms from start of poll, 56.891506ms from cluster start. Timeout at 2025-10-30 18:21:31.90277475 +0000 UTC m=+14.976304044, 105.002449ms after stability +2025-10-30T18:21:32.008Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:21:32.114Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:21:32.008438874 +0000 UTC m=+15.081968167 (3 nodes), 2.097µs from start of poll, 267.558078ms from cluster start. Timeout at 2025-10-30 18:21:32.113442962 +0000 UTC m=+15.186972251, 105.004084ms after stability +2025-10-30T18:21:32.114Z [INFO] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: updating configuration: command=RemoveServer server-id=server-258e18c4-6f01-3a05-53e4-2d4a6a991c5a server-addr= servers="[{Suffrage:Voter ID:server-af42de2c-87c6-f84a-0c8e-69b292821286 Address:af42de2c-87c6-f84a-0c8e-69b292821286} {Suffrage:Voter ID:server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0 Address:25c9ea00-ca51-f3fa-0051-e3c0ddc921c0} {Suffrage:Voter ID:server-b7a9c854-71aa-14d2-2ad1-bc75b4704fb9 Address:b7a9c854-71aa-14d2-2ad1-bc75b4704fb9}]" +2025-10-30T18:21:32.114Z [INFO] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: removed peer, stopping replication: peer=server-258e18c4-6f01-3a05-53e4-2d4a6a991c5a last-index=3 +2025-10-30T18:21:32.114Z [ERROR] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: failed to pipeline appendEntries: peer="{Voter server-af42de2c-87c6-f84a-0c8e-69b292821286 af42de2c-87c6-f84a-0c8e-69b292821286}" error="append pipeline closed" +2025-10-30T18:21:32.114Z [INFO] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: aborting pipeline replication: peer="{Voter server-258e18c4-6f01-3a05-53e4-2d4a6a991c5a 258e18c4-6f01-3a05-53e4-2d4a6a991c5a}" +2025-10-30T18:21:32.114Z [ERROR] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: failed to pipeline appendEntries: peer="{Voter server-b7a9c854-71aa-14d2-2ad1-bc75b4704fb9 b7a9c854-71aa-14d2-2ad1-bc75b4704fb9}" error="append pipeline closed" +2025-10-30T18:21:32.114Z [INFO] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: aborting pipeline replication: peer="{Voter server-b7a9c854-71aa-14d2-2ad1-bc75b4704fb9 b7a9c854-71aa-14d2-2ad1-bc75b4704fb9}" +2025-10-30T18:21:32.114Z [INFO] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: aborting pipeline replication: peer="{Voter server-af42de2c-87c6-f84a-0c8e-69b292821286 af42de2c-87c6-f84a-0c8e-69b292821286}" +2025-10-30T18:21:32.115Z [ERROR] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: failed to heartbeat to: peer=af42de2c-87c6-f84a-0c8e-69b292821286 backoff time=10ms error="failed to connect to peer: af42de2c-87c6-f84a-0c8e-69b292821286" +2025-10-30T18:21:32.119Z [ERROR] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: failed to heartbeat to: peer=b7a9c854-71aa-14d2-2ad1-bc75b4704fb9 backoff time=10ms error="failed to connect to peer: b7a9c854-71aa-14d2-2ad1-bc75b4704fb9" +2025-10-30T18:21:32.137Z [ERROR] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: failed to heartbeat to: peer=b7a9c854-71aa-14d2-2ad1-bc75b4704fb9 backoff time=10ms error="failed to connect to peer: b7a9c854-71aa-14d2-2ad1-bc75b4704fb9" +2025-10-30T18:21:32.142Z [ERROR] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: failed to appendEntries to: peer="{Voter server-af42de2c-87c6-f84a-0c8e-69b292821286 af42de2c-87c6-f84a-0c8e-69b292821286}" error="failed to connect to peer: af42de2c-87c6-f84a-0c8e-69b292821286" +2025-10-30T18:21:32.142Z [ERROR] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: failed to appendEntries to: peer="{Voter server-b7a9c854-71aa-14d2-2ad1-bc75b4704fb9 b7a9c854-71aa-14d2-2ad1-bc75b4704fb9}" error="failed to connect to peer: b7a9c854-71aa-14d2-2ad1-bc75b4704fb9" +2025-10-30T18:21:32.148Z [ERROR] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: failed to heartbeat to: peer=af42de2c-87c6-f84a-0c8e-69b292821286 backoff time=10ms error="failed to connect to peer: af42de2c-87c6-f84a-0c8e-69b292821286" +2025-10-30T18:21:32.157Z [ERROR] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: failed to heartbeat to: peer=b7a9c854-71aa-14d2-2ad1-bc75b4704fb9 backoff time=10ms error="failed to connect to peer: b7a9c854-71aa-14d2-2ad1-bc75b4704fb9" +2025-10-30T18:21:32.159Z [WARN] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: failed to contact: server-id=server-af42de2c-87c6-f84a-0c8e-69b292821286 time=50.255817ms +2025-10-30T18:21:32.161Z [ERROR] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: failed to appendEntries to: peer="{Voter server-af42de2c-87c6-f84a-0c8e-69b292821286 af42de2c-87c6-f84a-0c8e-69b292821286}" error="failed to connect to peer: af42de2c-87c6-f84a-0c8e-69b292821286" +2025-10-30T18:21:32.169Z [WARN] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: failed to contact: server-id=server-af42de2c-87c6-f84a-0c8e-69b292821286 time=60.663556ms +2025-10-30T18:21:32.169Z [WARN] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: failed to contact: server-id=server-b7a9c854-71aa-14d2-2ad1-bc75b4704fb9 time=55.191573ms +2025-10-30T18:21:32.169Z [WARN] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: failed to contact quorum of nodes, stepping down +2025-10-30T18:21:32.169Z [INFO] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: entering follower state: follower="Node at 25c9ea00-ca51-f3fa-0051-e3c0ddc921c0 [Follower]" leader-address= leader-id= +2025-10-30T18:21:32.169Z [ERROR] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: failed to appendEntries to: peer="{Voter server-af42de2c-87c6-f84a-0c8e-69b292821286 af42de2c-87c6-f84a-0c8e-69b292821286}" error="failed to connect to peer: af42de2c-87c6-f84a-0c8e-69b292821286" +2025-10-30T18:21:32.169Z [ERROR] server-25c9ea00-ca51-f3fa-0051-e3c0ddc921c0: failed to appendEntries to: peer="{Voter server-b7a9c854-71aa-14d2-2ad1-bc75b4704fb9 b7a9c854-71aa-14d2-2ad1-bc75b4704fb9}" error="failed to connect to peer: b7a9c854-71aa-14d2-2ad1-bc75b4704fb9" +--- PASS: TestRaft_RemoveFollower_SplitCluster (0.50s) === RUN TestRaft_AddKnownPeer -2025-10-30T18:17:45.148Z [INFO] server-e708b557-baff-aba1-8de0-8885b7cd651b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e708b557-baff-aba1-8de0-8885b7cd651b Address:e708b557-baff-aba1-8de0-8885b7cd651b} {Suffrage:Voter ID:server-204e12a7-76ed-210c-67d1-51083ffdd6ca Address:204e12a7-76ed-210c-67d1-51083ffdd6ca} {Suffrage:Voter ID:server-5719da0f-81bf-36e5-0290-a9477b806595 Address:5719da0f-81bf-36e5-0290-a9477b806595}]" -2025-10-30T18:17:45.148Z [INFO] server-204e12a7-76ed-210c-67d1-51083ffdd6ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e708b557-baff-aba1-8de0-8885b7cd651b Address:e708b557-baff-aba1-8de0-8885b7cd651b} {Suffrage:Voter ID:server-204e12a7-76ed-210c-67d1-51083ffdd6ca Address:204e12a7-76ed-210c-67d1-51083ffdd6ca} {Suffrage:Voter ID:server-5719da0f-81bf-36e5-0290-a9477b806595 Address:5719da0f-81bf-36e5-0290-a9477b806595}]" -2025-10-30T18:17:45.148Z [INFO] server-e708b557-baff-aba1-8de0-8885b7cd651b: entering follower state: follower="Node at e708b557-baff-aba1-8de0-8885b7cd651b [Follower]" leader-address= leader-id= -2025-10-30T18:17:45.148Z [INFO] server-5719da0f-81bf-36e5-0290-a9477b806595: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e708b557-baff-aba1-8de0-8885b7cd651b Address:e708b557-baff-aba1-8de0-8885b7cd651b} {Suffrage:Voter ID:server-204e12a7-76ed-210c-67d1-51083ffdd6ca Address:204e12a7-76ed-210c-67d1-51083ffdd6ca} {Suffrage:Voter ID:server-5719da0f-81bf-36e5-0290-a9477b806595 Address:5719da0f-81bf-36e5-0290-a9477b806595}]" -2025-10-30T18:17:45.148Z [INFO] server-204e12a7-76ed-210c-67d1-51083ffdd6ca: entering follower state: follower="Node at 204e12a7-76ed-210c-67d1-51083ffdd6ca [Follower]" leader-address= leader-id= -2025-10-30T18:17:45.148Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:45.148Z [INFO] server-5719da0f-81bf-36e5-0290-a9477b806595: entering follower state: follower="Node at 5719da0f-81bf-36e5-0290-a9477b806595 [Follower]" leader-address= leader-id= -2025-10-30T18:17:45.207Z [WARN] server-e708b557-baff-aba1-8de0-8885b7cd651b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:45.207Z [INFO] server-e708b557-baff-aba1-8de0-8885b7cd651b: entering candidate state: node="Node at e708b557-baff-aba1-8de0-8885b7cd651b [Candidate]" term=2 -2025-10-30T18:17:45.207Z [INFO] server-e708b557-baff-aba1-8de0-8885b7cd651b: election won: term=2 tally=2 -2025-10-30T18:17:45.207Z [INFO] server-e708b557-baff-aba1-8de0-8885b7cd651b: entering leader state: leader="Node at e708b557-baff-aba1-8de0-8885b7cd651b [Leader]" -2025-10-30T18:17:45.207Z [INFO] server-e708b557-baff-aba1-8de0-8885b7cd651b: added peer, starting replication: peer=server-204e12a7-76ed-210c-67d1-51083ffdd6ca -2025-10-30T18:17:45.207Z [INFO] server-e708b557-baff-aba1-8de0-8885b7cd651b: added peer, starting replication: peer=server-5719da0f-81bf-36e5-0290-a9477b806595 -2025-10-30T18:17:45.207Z [INFO] server-e708b557-baff-aba1-8de0-8885b7cd651b: pipelining replication: peer="{Voter server-5719da0f-81bf-36e5-0290-a9477b806595 5719da0f-81bf-36e5-0290-a9477b806595}" -2025-10-30T18:17:45.207Z [INFO] server-e708b557-baff-aba1-8de0-8885b7cd651b: pipelining replication: peer="{Voter server-204e12a7-76ed-210c-67d1-51083ffdd6ca 204e12a7-76ed-210c-67d1-51083ffdd6ca}" -2025-10-30T18:17:45.312Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:45.207375765 +0000 UTC m=+11.486049662 (1 nodes), 58.834501ms from start of poll, 59.124858ms from cluster start. Timeout at 2025-10-30 18:17:45.312378772 +0000 UTC m=+11.591052671, 105.003009ms after stability -2025-10-30T18:17:45.312Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:17:45.418Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:17:45.312582325 +0000 UTC m=+11.591256222 (2 nodes), 1.374µs from start of poll, 164.331418ms from cluster start. Timeout at 2025-10-30 18:17:45.41758945 +0000 UTC m=+11.696263359, 105.007137ms after stability -2025-10-30T18:17:45.418Z [INFO] server-e708b557-baff-aba1-8de0-8885b7cd651b: updating configuration: command=AddVoter server-id=server-204e12a7-76ed-210c-67d1-51083ffdd6ca server-addr=204e12a7-76ed-210c-67d1-51083ffdd6ca servers="[{Suffrage:Voter ID:server-e708b557-baff-aba1-8de0-8885b7cd651b Address:e708b557-baff-aba1-8de0-8885b7cd651b} {Suffrage:Voter ID:server-204e12a7-76ed-210c-67d1-51083ffdd6ca Address:204e12a7-76ed-210c-67d1-51083ffdd6ca} {Suffrage:Voter ID:server-5719da0f-81bf-36e5-0290-a9477b806595 Address:5719da0f-81bf-36e5-0290-a9477b806595}]" -2025-10-30T18:17:45.418Z [INFO] server-e708b557-baff-aba1-8de0-8885b7cd651b: aborting pipeline replication: peer="{Voter server-5719da0f-81bf-36e5-0290-a9477b806595 5719da0f-81bf-36e5-0290-a9477b806595}" -2025-10-30T18:17:45.418Z [INFO] server-e708b557-baff-aba1-8de0-8885b7cd651b: aborting pipeline replication: peer="{Voter server-204e12a7-76ed-210c-67d1-51083ffdd6ca 204e12a7-76ed-210c-67d1-51083ffdd6ca}" ---- PASS: TestRaft_AddKnownPeer (0.27s) +2025-10-30T18:21:32.239Z [INFO] server-5b8a1017-cb40-1a27-30ce-e41f926ef0d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5b8a1017-cb40-1a27-30ce-e41f926ef0d9 Address:5b8a1017-cb40-1a27-30ce-e41f926ef0d9} {Suffrage:Voter ID:server-ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98 Address:ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98} {Suffrage:Voter ID:server-a92a664f-2dc4-9856-eba5-43b63ca8e48e Address:a92a664f-2dc4-9856-eba5-43b63ca8e48e}]" +2025-10-30T18:21:32.239Z [INFO] server-ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5b8a1017-cb40-1a27-30ce-e41f926ef0d9 Address:5b8a1017-cb40-1a27-30ce-e41f926ef0d9} {Suffrage:Voter ID:server-ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98 Address:ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98} {Suffrage:Voter ID:server-a92a664f-2dc4-9856-eba5-43b63ca8e48e Address:a92a664f-2dc4-9856-eba5-43b63ca8e48e}]" +2025-10-30T18:21:32.239Z [INFO] server-5b8a1017-cb40-1a27-30ce-e41f926ef0d9: entering follower state: follower="Node at 5b8a1017-cb40-1a27-30ce-e41f926ef0d9 [Follower]" leader-address= leader-id= +2025-10-30T18:21:32.239Z [INFO] server-ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98: entering follower state: follower="Node at ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98 [Follower]" leader-address= leader-id= +2025-10-30T18:21:32.239Z [INFO] server-a92a664f-2dc4-9856-eba5-43b63ca8e48e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5b8a1017-cb40-1a27-30ce-e41f926ef0d9 Address:5b8a1017-cb40-1a27-30ce-e41f926ef0d9} {Suffrage:Voter ID:server-ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98 Address:ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98} {Suffrage:Voter ID:server-a92a664f-2dc4-9856-eba5-43b63ca8e48e Address:a92a664f-2dc4-9856-eba5-43b63ca8e48e}]" +2025-10-30T18:21:32.239Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:32.239Z [INFO] server-a92a664f-2dc4-9856-eba5-43b63ca8e48e: entering follower state: follower="Node at a92a664f-2dc4-9856-eba5-43b63ca8e48e [Follower]" leader-address= leader-id= +2025-10-30T18:21:32.300Z [WARN] server-5b8a1017-cb40-1a27-30ce-e41f926ef0d9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:32.300Z [INFO] server-5b8a1017-cb40-1a27-30ce-e41f926ef0d9: entering candidate state: node="Node at 5b8a1017-cb40-1a27-30ce-e41f926ef0d9 [Candidate]" term=2 +2025-10-30T18:21:32.301Z [INFO] server-5b8a1017-cb40-1a27-30ce-e41f926ef0d9: election won: term=2 tally=2 +2025-10-30T18:21:32.301Z [INFO] server-5b8a1017-cb40-1a27-30ce-e41f926ef0d9: entering leader state: leader="Node at 5b8a1017-cb40-1a27-30ce-e41f926ef0d9 [Leader]" +2025-10-30T18:21:32.301Z [INFO] server-5b8a1017-cb40-1a27-30ce-e41f926ef0d9: added peer, starting replication: peer=server-ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98 +2025-10-30T18:21:32.301Z [INFO] server-5b8a1017-cb40-1a27-30ce-e41f926ef0d9: added peer, starting replication: peer=server-a92a664f-2dc4-9856-eba5-43b63ca8e48e +2025-10-30T18:21:32.301Z [INFO] server-5b8a1017-cb40-1a27-30ce-e41f926ef0d9: pipelining replication: peer="{Voter server-ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98 ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98}" +2025-10-30T18:21:32.301Z [INFO] server-5b8a1017-cb40-1a27-30ce-e41f926ef0d9: pipelining replication: peer="{Voter server-a92a664f-2dc4-9856-eba5-43b63ca8e48e a92a664f-2dc4-9856-eba5-43b63ca8e48e}" +2025-10-30T18:21:32.407Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:32.301693812 +0000 UTC m=+15.375223128 (1 nodes), 62.176435ms from start of poll, 62.417306ms from cluster start. Timeout at 2025-10-30 18:21:32.406696337 +0000 UTC m=+15.480225629, 105.002501ms after stability +2025-10-30T18:21:32.407Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:21:32.512Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:21:32.407275785 +0000 UTC m=+15.480805079 (2 nodes), 753ns from start of poll, 167.999257ms from cluster start. Timeout at 2025-10-30 18:21:32.512287014 +0000 UTC m=+15.585816343, 105.011264ms after stability +2025-10-30T18:21:32.513Z [INFO] server-5b8a1017-cb40-1a27-30ce-e41f926ef0d9: updating configuration: command=AddVoter server-id=server-ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98 server-addr=ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98 servers="[{Suffrage:Voter ID:server-5b8a1017-cb40-1a27-30ce-e41f926ef0d9 Address:5b8a1017-cb40-1a27-30ce-e41f926ef0d9} {Suffrage:Voter ID:server-ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98 Address:ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98} {Suffrage:Voter ID:server-a92a664f-2dc4-9856-eba5-43b63ca8e48e Address:a92a664f-2dc4-9856-eba5-43b63ca8e48e}]" +2025-10-30T18:21:32.513Z [INFO] server-5b8a1017-cb40-1a27-30ce-e41f926ef0d9: aborting pipeline replication: peer="{Voter server-a92a664f-2dc4-9856-eba5-43b63ca8e48e a92a664f-2dc4-9856-eba5-43b63ca8e48e}" +2025-10-30T18:21:32.513Z [INFO] server-5b8a1017-cb40-1a27-30ce-e41f926ef0d9: aborting pipeline replication: peer="{Voter server-ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98 ecd7cb1e-165b-b5fe-3aba-bdce42ff9d98}" +--- PASS: TestRaft_AddKnownPeer (0.38s) === RUN TestRaft_RemoveUnknownPeer -2025-10-30T18:17:45.419Z [INFO] server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9fa270-dec9-b506-4b15-4b05860e2ef2 Address:3a9fa270-dec9-b506-4b15-4b05860e2ef2} {Suffrage:Voter ID:server-a46ee619-979e-32b9-c494-a88bb99679cf Address:a46ee619-979e-32b9-c494-a88bb99679cf} {Suffrage:Voter ID:server-af67b78c-1d7c-0107-49fd-38b7e0560803 Address:af67b78c-1d7c-0107-49fd-38b7e0560803}]" -2025-10-30T18:17:45.419Z [INFO] server-a46ee619-979e-32b9-c494-a88bb99679cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9fa270-dec9-b506-4b15-4b05860e2ef2 Address:3a9fa270-dec9-b506-4b15-4b05860e2ef2} {Suffrage:Voter ID:server-a46ee619-979e-32b9-c494-a88bb99679cf Address:a46ee619-979e-32b9-c494-a88bb99679cf} {Suffrage:Voter ID:server-af67b78c-1d7c-0107-49fd-38b7e0560803 Address:af67b78c-1d7c-0107-49fd-38b7e0560803}]" -2025-10-30T18:17:45.419Z [INFO] server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: entering follower state: follower="Node at 3a9fa270-dec9-b506-4b15-4b05860e2ef2 [Follower]" leader-address= leader-id= -2025-10-30T18:17:45.419Z [INFO] server-af67b78c-1d7c-0107-49fd-38b7e0560803: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a9fa270-dec9-b506-4b15-4b05860e2ef2 Address:3a9fa270-dec9-b506-4b15-4b05860e2ef2} {Suffrage:Voter ID:server-a46ee619-979e-32b9-c494-a88bb99679cf Address:a46ee619-979e-32b9-c494-a88bb99679cf} {Suffrage:Voter ID:server-af67b78c-1d7c-0107-49fd-38b7e0560803 Address:af67b78c-1d7c-0107-49fd-38b7e0560803}]" -2025-10-30T18:17:45.419Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:45.419Z [INFO] server-a46ee619-979e-32b9-c494-a88bb99679cf: entering follower state: follower="Node at a46ee619-979e-32b9-c494-a88bb99679cf [Follower]" leader-address= leader-id= -2025-10-30T18:17:45.419Z [INFO] server-af67b78c-1d7c-0107-49fd-38b7e0560803: entering follower state: follower="Node at af67b78c-1d7c-0107-49fd-38b7e0560803 [Follower]" leader-address= leader-id= -2025-10-30T18:17:45.471Z [WARN] server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:45.471Z [INFO] server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: entering candidate state: node="Node at 3a9fa270-dec9-b506-4b15-4b05860e2ef2 [Candidate]" term=2 -2025-10-30T18:17:45.471Z [INFO] server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: election won: term=2 tally=2 -2025-10-30T18:17:45.471Z [INFO] server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: entering leader state: leader="Node at 3a9fa270-dec9-b506-4b15-4b05860e2ef2 [Leader]" -2025-10-30T18:17:45.471Z [INFO] server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: added peer, starting replication: peer=server-a46ee619-979e-32b9-c494-a88bb99679cf -2025-10-30T18:17:45.471Z [INFO] server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: added peer, starting replication: peer=server-af67b78c-1d7c-0107-49fd-38b7e0560803 -2025-10-30T18:17:45.472Z [INFO] server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: pipelining replication: peer="{Voter server-af67b78c-1d7c-0107-49fd-38b7e0560803 af67b78c-1d7c-0107-49fd-38b7e0560803}" -2025-10-30T18:17:45.472Z [INFO] server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: pipelining replication: peer="{Voter server-a46ee619-979e-32b9-c494-a88bb99679cf a46ee619-979e-32b9-c494-a88bb99679cf}" -2025-10-30T18:17:45.577Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:45.472033551 +0000 UTC m=+11.750707449 (1 nodes), 52.339269ms from start of poll, 52.558451ms from cluster start. Timeout at 2025-10-30 18:17:45.57703952 +0000 UTC m=+11.855713418, 105.005969ms after stability -2025-10-30T18:17:45.577Z [INFO] server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: updating configuration: command=RemoveServer server-id=5ba0583a-0d91-ecfd-83ff-5acefca0c5c8 server-addr= servers="[{Suffrage:Voter ID:server-3a9fa270-dec9-b506-4b15-4b05860e2ef2 Address:3a9fa270-dec9-b506-4b15-4b05860e2ef2} {Suffrage:Voter ID:server-a46ee619-979e-32b9-c494-a88bb99679cf Address:a46ee619-979e-32b9-c494-a88bb99679cf} {Suffrage:Voter ID:server-af67b78c-1d7c-0107-49fd-38b7e0560803 Address:af67b78c-1d7c-0107-49fd-38b7e0560803}]" -2025-10-30T18:17:45.577Z [INFO] server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: aborting pipeline replication: peer="{Voter server-af67b78c-1d7c-0107-49fd-38b7e0560803 af67b78c-1d7c-0107-49fd-38b7e0560803}" -2025-10-30T18:17:45.577Z [INFO] server-3a9fa270-dec9-b506-4b15-4b05860e2ef2: aborting pipeline replication: peer="{Voter server-a46ee619-979e-32b9-c494-a88bb99679cf a46ee619-979e-32b9-c494-a88bb99679cf}" +2025-10-30T18:21:32.622Z [INFO] server-c49c1fa7-129d-6c84-585c-22a3dca68012: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c49c1fa7-129d-6c84-585c-22a3dca68012 Address:c49c1fa7-129d-6c84-585c-22a3dca68012} {Suffrage:Voter ID:server-eb300af1-f997-4141-c47c-8420cc4d8490 Address:eb300af1-f997-4141-c47c-8420cc4d8490} {Suffrage:Voter ID:server-d5f99c08-e28b-6f30-2a97-cbd0a0dad833 Address:d5f99c08-e28b-6f30-2a97-cbd0a0dad833}]" +2025-10-30T18:21:32.622Z [INFO] server-eb300af1-f997-4141-c47c-8420cc4d8490: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c49c1fa7-129d-6c84-585c-22a3dca68012 Address:c49c1fa7-129d-6c84-585c-22a3dca68012} {Suffrage:Voter ID:server-eb300af1-f997-4141-c47c-8420cc4d8490 Address:eb300af1-f997-4141-c47c-8420cc4d8490} {Suffrage:Voter ID:server-d5f99c08-e28b-6f30-2a97-cbd0a0dad833 Address:d5f99c08-e28b-6f30-2a97-cbd0a0dad833}]" +2025-10-30T18:21:32.622Z [INFO] server-eb300af1-f997-4141-c47c-8420cc4d8490: entering follower state: follower="Node at eb300af1-f997-4141-c47c-8420cc4d8490 [Follower]" leader-address= leader-id= +2025-10-30T18:21:32.622Z [INFO] server-c49c1fa7-129d-6c84-585c-22a3dca68012: entering follower state: follower="Node at c49c1fa7-129d-6c84-585c-22a3dca68012 [Follower]" leader-address= leader-id= +2025-10-30T18:21:32.622Z [INFO] server-d5f99c08-e28b-6f30-2a97-cbd0a0dad833: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c49c1fa7-129d-6c84-585c-22a3dca68012 Address:c49c1fa7-129d-6c84-585c-22a3dca68012} {Suffrage:Voter ID:server-eb300af1-f997-4141-c47c-8420cc4d8490 Address:eb300af1-f997-4141-c47c-8420cc4d8490} {Suffrage:Voter ID:server-d5f99c08-e28b-6f30-2a97-cbd0a0dad833 Address:d5f99c08-e28b-6f30-2a97-cbd0a0dad833}]" +2025-10-30T18:21:32.622Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:32.622Z [INFO] server-d5f99c08-e28b-6f30-2a97-cbd0a0dad833: entering follower state: follower="Node at d5f99c08-e28b-6f30-2a97-cbd0a0dad833 [Follower]" leader-address= leader-id= +2025-10-30T18:21:32.673Z [WARN] server-c49c1fa7-129d-6c84-585c-22a3dca68012: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:32.673Z [INFO] server-c49c1fa7-129d-6c84-585c-22a3dca68012: entering candidate state: node="Node at c49c1fa7-129d-6c84-585c-22a3dca68012 [Candidate]" term=2 +2025-10-30T18:21:32.673Z [INFO] server-c49c1fa7-129d-6c84-585c-22a3dca68012: election won: term=2 tally=2 +2025-10-30T18:21:32.673Z [INFO] server-c49c1fa7-129d-6c84-585c-22a3dca68012: entering leader state: leader="Node at c49c1fa7-129d-6c84-585c-22a3dca68012 [Leader]" +2025-10-30T18:21:32.673Z [INFO] server-c49c1fa7-129d-6c84-585c-22a3dca68012: added peer, starting replication: peer=server-eb300af1-f997-4141-c47c-8420cc4d8490 +2025-10-30T18:21:32.673Z [INFO] server-c49c1fa7-129d-6c84-585c-22a3dca68012: added peer, starting replication: peer=server-d5f99c08-e28b-6f30-2a97-cbd0a0dad833 +2025-10-30T18:21:32.673Z [INFO] server-c49c1fa7-129d-6c84-585c-22a3dca68012: pipelining replication: peer="{Voter server-d5f99c08-e28b-6f30-2a97-cbd0a0dad833 d5f99c08-e28b-6f30-2a97-cbd0a0dad833}" +2025-10-30T18:21:32.673Z [INFO] server-c49c1fa7-129d-6c84-585c-22a3dca68012: pipelining replication: peer="{Voter server-eb300af1-f997-4141-c47c-8420cc4d8490 eb300af1-f997-4141-c47c-8420cc4d8490}" +2025-10-30T18:21:32.778Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:32.673297735 +0000 UTC m=+15.746827024 (1 nodes), 50.682787ms from start of poll, 50.97245ms from cluster start. Timeout at 2025-10-30 18:21:32.778300234 +0000 UTC m=+15.851829527, 105.002503ms after stability +2025-10-30T18:21:32.778Z [INFO] server-c49c1fa7-129d-6c84-585c-22a3dca68012: updating configuration: command=RemoveServer server-id=c0bc31cd-2219-4058-c4db-1d5c0b979f14 server-addr= servers="[{Suffrage:Voter ID:server-c49c1fa7-129d-6c84-585c-22a3dca68012 Address:c49c1fa7-129d-6c84-585c-22a3dca68012} {Suffrage:Voter ID:server-eb300af1-f997-4141-c47c-8420cc4d8490 Address:eb300af1-f997-4141-c47c-8420cc4d8490} {Suffrage:Voter ID:server-d5f99c08-e28b-6f30-2a97-cbd0a0dad833 Address:d5f99c08-e28b-6f30-2a97-cbd0a0dad833}]" +2025-10-30T18:21:32.778Z [INFO] server-c49c1fa7-129d-6c84-585c-22a3dca68012: aborting pipeline replication: peer="{Voter server-d5f99c08-e28b-6f30-2a97-cbd0a0dad833 d5f99c08-e28b-6f30-2a97-cbd0a0dad833}" +2025-10-30T18:21:32.778Z [INFO] server-c49c1fa7-129d-6c84-585c-22a3dca68012: aborting pipeline replication: peer="{Voter server-eb300af1-f997-4141-c47c-8420cc4d8490 eb300af1-f997-4141-c47c-8420cc4d8490}" --- PASS: TestRaft_RemoveUnknownPeer (0.16s) === RUN TestRaft_SnapshotRestore -2025-10-30T18:17:45.579Z [INFO] server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365 Address:baefbeb4-4e30-2a7b-5a74-7ce25eedb365}]" -2025-10-30T18:17:45.579Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:45.579Z [INFO] server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: entering follower state: follower="Node at baefbeb4-4e30-2a7b-5a74-7ce25eedb365 [Follower]" leader-address= leader-id= -2025-10-30T18:17:45.668Z [WARN] server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:45.668Z [INFO] server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: entering candidate state: node="Node at baefbeb4-4e30-2a7b-5a74-7ce25eedb365 [Candidate]" term=2 -2025-10-30T18:17:45.668Z [INFO] server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: election won: term=2 tally=1 -2025-10-30T18:17:45.668Z [INFO] server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: entering leader state: leader="Node at baefbeb4-4e30-2a7b-5a74-7ce25eedb365 [Leader]" -2025-10-30T18:17:45.774Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:45.668931776 +0000 UTC m=+11.947605803 (1 nodes), 89.590456ms from start of poll, 90.746214ms from cluster start. Timeout at 2025-10-30 18:17:45.773933737 +0000 UTC m=+12.052607634, 105.001831ms after stability -2025-10-30T18:17:45.775Z [INFO] server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: starting snapshot up to: index=102 -2025-10-30T18:17:45.775Z [INFO] creating new snapshot: path=/tmp/raft17063633/snapshots/2-102-1761848265775.tmp -2025-10-30T18:17:45.775Z [INFO] server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: compacting logs: from=1 to=92 -2025-10-30T18:17:45.775Z [INFO] server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: snapshot complete up to: index=102 -2025-10-30T18:17:45.776Z [INFO] server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: starting restore from snapshot: id=2-102-1761848265775 last-index=102 last-term=2 size-in-bytes=693 -2025-10-30T18:17:45.776Z [INFO] server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: snapshot restore progress: id=2-102-1761848265775 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% -2025-10-30T18:17:45.776Z [INFO] server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: restored from snapshot: id=2-102-1761848265775 last-index=102 last-term=2 size-in-bytes=693 -2025-10-30T18:17:45.776Z [INFO] server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-baefbeb4-4e30-2a7b-5a74-7ce25eedb365 Address:baefbeb4-4e30-2a7b-5a74-7ce25eedb365}]" ---- PASS: TestRaft_SnapshotRestore (0.20s) +2025-10-30T18:21:32.779Z [INFO] server-65e8cd94-6aaf-cf3b-9dc5-aa048d99473a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65e8cd94-6aaf-cf3b-9dc5-aa048d99473a Address:65e8cd94-6aaf-cf3b-9dc5-aa048d99473a}]" +2025-10-30T18:21:32.779Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:32.779Z [INFO] server-65e8cd94-6aaf-cf3b-9dc5-aa048d99473a: entering follower state: follower="Node at 65e8cd94-6aaf-cf3b-9dc5-aa048d99473a [Follower]" leader-address= leader-id= +2025-10-30T18:21:32.869Z [WARN] server-65e8cd94-6aaf-cf3b-9dc5-aa048d99473a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:32.869Z [INFO] server-65e8cd94-6aaf-cf3b-9dc5-aa048d99473a: entering candidate state: node="Node at 65e8cd94-6aaf-cf3b-9dc5-aa048d99473a [Candidate]" term=2 +2025-10-30T18:21:32.869Z [INFO] server-65e8cd94-6aaf-cf3b-9dc5-aa048d99473a: election won: term=2 tally=1 +2025-10-30T18:21:32.869Z [INFO] server-65e8cd94-6aaf-cf3b-9dc5-aa048d99473a: entering leader state: leader="Node at 65e8cd94-6aaf-cf3b-9dc5-aa048d99473a [Leader]" +2025-10-30T18:21:32.975Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:32.869661382 +0000 UTC m=+15.943190708 (1 nodes), 89.987481ms from start of poll, 90.089547ms from cluster start. Timeout at 2025-10-30 18:21:32.974666187 +0000 UTC m=+16.048195517, 105.004809ms after stability +2025-10-30T18:21:32.977Z [INFO] server-65e8cd94-6aaf-cf3b-9dc5-aa048d99473a: starting snapshot up to: index=102 +2025-10-30T18:21:32.977Z [INFO] creating new snapshot: path=/tmp/raft963172864/snapshots/2-102-1761848492977.tmp +2025-10-30T18:21:33.107Z [INFO] server-65e8cd94-6aaf-cf3b-9dc5-aa048d99473a: compacting logs: from=1 to=92 +2025-10-30T18:21:33.107Z [INFO] server-65e8cd94-6aaf-cf3b-9dc5-aa048d99473a: snapshot complete up to: index=102 +2025-10-30T18:21:33.107Z [INFO] server-65e8cd94-6aaf-cf3b-9dc5-aa048d99473a: starting restore from snapshot: id=2-102-1761848492977 last-index=102 last-term=2 size-in-bytes=693 +2025-10-30T18:21:33.107Z [INFO] server-65e8cd94-6aaf-cf3b-9dc5-aa048d99473a: snapshot restore progress: id=2-102-1761848492977 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% +2025-10-30T18:21:33.107Z [INFO] server-65e8cd94-6aaf-cf3b-9dc5-aa048d99473a: restored from snapshot: id=2-102-1761848492977 last-index=102 last-term=2 size-in-bytes=693 +2025-10-30T18:21:33.107Z [INFO] server-65e8cd94-6aaf-cf3b-9dc5-aa048d99473a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65e8cd94-6aaf-cf3b-9dc5-aa048d99473a Address:65e8cd94-6aaf-cf3b-9dc5-aa048d99473a}]" +--- PASS: TestRaft_SnapshotRestore (0.65s) === RUN TestRaft_RestoreSnapshotOnStartup_Monotonic -2025-10-30T18:17:45.778Z [INFO] server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036 Address:d6f95dcb-2b71-8d5a-c485-cbf3a697f036}]" -2025-10-30T18:17:45.778Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:45.778Z [INFO] server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: entering follower state: follower="Node at d6f95dcb-2b71-8d5a-c485-cbf3a697f036 [Follower]" leader-address= leader-id= -2025-10-30T18:17:45.833Z [WARN] server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:45.833Z [INFO] server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: entering candidate state: node="Node at d6f95dcb-2b71-8d5a-c485-cbf3a697f036 [Candidate]" term=2 -2025-10-30T18:17:45.833Z [INFO] server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: election won: term=2 tally=1 -2025-10-30T18:17:45.833Z [INFO] server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: entering leader state: leader="Node at d6f95dcb-2b71-8d5a-c485-cbf3a697f036 [Leader]" -2025-10-30T18:17:45.939Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:45.83384783 +0000 UTC m=+12.112521742 (1 nodes), 55.642154ms from start of poll, 55.732104ms from cluster start. Timeout at 2025-10-30 18:17:45.938850762 +0000 UTC m=+12.217524659, 105.002917ms after stability -2025-10-30T18:17:45.940Z [INFO] server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: starting snapshot up to: index=102 -2025-10-30T18:17:45.940Z [INFO] creating new snapshot: path=/tmp/raft993325267/snapshots/2-102-1761848265940.tmp -2025-10-30T18:17:45.941Z [INFO] server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: compacting logs: from=1 to=92 -2025-10-30T18:17:45.941Z [INFO] server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: snapshot complete up to: index=102 -2025-10-30T18:17:45.941Z [INFO] server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: starting restore from snapshot: id=2-102-1761848265940 last-index=102 last-term=2 size-in-bytes=693 -2025-10-30T18:17:45.941Z [INFO] server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: snapshot restore progress: id=2-102-1761848265940 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% -2025-10-30T18:17:45.941Z [INFO] server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: restored from snapshot: id=2-102-1761848265940 last-index=102 last-term=2 size-in-bytes=693 -2025-10-30T18:17:45.941Z [INFO] server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036 Address:d6f95dcb-2b71-8d5a-c485-cbf3a697f036}]" -2025-10-30T18:17:45.941Z [INFO] server-d6f95dcb-2b71-8d5a-c485-cbf3a697f036: entering follower state: follower="Node at d6f95dcb-2b71-8d5a-c485-cbf3a697f036 [Shutdown]" leader-address= leader-id= ---- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.16s) +2025-10-30T18:21:33.433Z [INFO] server-04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9 Address:04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9}]" +2025-10-30T18:21:33.433Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:33.433Z [INFO] server-04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9: entering follower state: follower="Node at 04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9 [Follower]" leader-address= leader-id= +2025-10-30T18:21:33.498Z [WARN] server-04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:33.498Z [INFO] server-04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9: entering candidate state: node="Node at 04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9 [Candidate]" term=2 +2025-10-30T18:21:33.498Z [INFO] server-04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9: election won: term=2 tally=1 +2025-10-30T18:21:33.498Z [INFO] server-04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9: entering leader state: leader="Node at 04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9 [Leader]" +2025-10-30T18:21:33.604Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:33.498890162 +0000 UTC m=+16.572419450 (1 nodes), 65.635181ms from start of poll, 65.806675ms from cluster start. Timeout at 2025-10-30 18:21:33.603892827 +0000 UTC m=+16.677422121, 105.002671ms after stability +2025-10-30T18:21:33.605Z [INFO] server-04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9: starting snapshot up to: index=102 +2025-10-30T18:21:33.605Z [INFO] creating new snapshot: path=/tmp/raft2380943252/snapshots/2-102-1761848493605.tmp +2025-10-30T18:21:33.646Z [INFO] server-04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9: compacting logs: from=1 to=92 +2025-10-30T18:21:33.646Z [INFO] server-04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9: snapshot complete up to: index=102 +2025-10-30T18:21:33.647Z [INFO] server-04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9: starting restore from snapshot: id=2-102-1761848493605 last-index=102 last-term=2 size-in-bytes=693 +2025-10-30T18:21:33.647Z [INFO] server-04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9: snapshot restore progress: id=2-102-1761848493605 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% +2025-10-30T18:21:33.647Z [INFO] server-04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9: restored from snapshot: id=2-102-1761848493605 last-index=102 last-term=2 size-in-bytes=693 +2025-10-30T18:21:33.647Z [INFO] server-04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9 Address:04b7a778-f2c6-bbf9-8c52-b2c91ee99fb9}]" +--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.71s) === RUN TestRaft_SnapshotRestore_Progress -2025-10-30T18:17:45.942Z [INFO] server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb Address:b8e6f099-ad7d-6443-c8c3-5ad28629b8bb}]" -2025-10-30T18:17:45.942Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:45.942Z [INFO] server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: entering follower state: follower="Node at b8e6f099-ad7d-6443-c8c3-5ad28629b8bb [Follower]" leader-address= leader-id= -2025-10-30T18:17:46.021Z [WARN] server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:46.021Z [INFO] server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: entering candidate state: node="Node at b8e6f099-ad7d-6443-c8c3-5ad28629b8bb [Candidate]" term=2 -2025-10-30T18:17:46.022Z [INFO] server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: election won: term=2 tally=1 -2025-10-30T18:17:46.022Z [INFO] server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: entering leader state: leader="Node at b8e6f099-ad7d-6443-c8c3-5ad28629b8bb [Leader]" -2025-10-30T18:17:46.127Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:46.022019144 +0000 UTC m=+12.300693051 (1 nodes), 79.838574ms from start of poll, 79.926222ms from cluster start. Timeout at 2025-10-30 18:17:46.127023076 +0000 UTC m=+12.405696975, 105.003924ms after stability -2025-10-30T18:17:46.128Z [INFO] server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: starting snapshot up to: index=102 -2025-10-30T18:17:46.128Z [INFO] creating new snapshot: path=/tmp/raft2914980949/snapshots/2-102-1761848266128.tmp -2025-10-30T18:17:46.128Z [INFO] server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: compacting logs: from=1 to=92 -2025-10-30T18:17:46.128Z [INFO] server-b8e6f099-ad7d-6443-c8c3-5ad28629b8bb: snapshot complete up to: index=102 ---- PASS: TestRaft_SnapshotRestore_Progress (0.19s) +2025-10-30T18:21:34.139Z [INFO] server-034f19c5-db63-fbe6-552c-e58fd87a5d36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-034f19c5-db63-fbe6-552c-e58fd87a5d36 Address:034f19c5-db63-fbe6-552c-e58fd87a5d36}]" +2025-10-30T18:21:34.139Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:34.139Z [INFO] server-034f19c5-db63-fbe6-552c-e58fd87a5d36: entering follower state: follower="Node at 034f19c5-db63-fbe6-552c-e58fd87a5d36 [Follower]" leader-address= leader-id= +2025-10-30T18:21:34.196Z [WARN] server-034f19c5-db63-fbe6-552c-e58fd87a5d36: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:34.196Z [INFO] server-034f19c5-db63-fbe6-552c-e58fd87a5d36: entering candidate state: node="Node at 034f19c5-db63-fbe6-552c-e58fd87a5d36 [Candidate]" term=2 +2025-10-30T18:21:34.196Z [INFO] server-034f19c5-db63-fbe6-552c-e58fd87a5d36: election won: term=2 tally=1 +2025-10-30T18:21:34.196Z [INFO] server-034f19c5-db63-fbe6-552c-e58fd87a5d36: entering leader state: leader="Node at 034f19c5-db63-fbe6-552c-e58fd87a5d36 [Leader]" +2025-10-30T18:21:34.301Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:34.196449394 +0000 UTC m=+17.269978764 (1 nodes), 56.618751ms from start of poll, 56.779409ms from cluster start. Timeout at 2025-10-30 18:21:34.30145248 +0000 UTC m=+17.374981773, 105.003009ms after stability +2025-10-30T18:21:34.304Z [INFO] server-034f19c5-db63-fbe6-552c-e58fd87a5d36: starting snapshot up to: index=102 +2025-10-30T18:21:34.304Z [INFO] creating new snapshot: path=/tmp/raft866525940/snapshots/2-102-1761848494304.tmp +2025-10-30T18:21:34.306Z [INFO] server-034f19c5-db63-fbe6-552c-e58fd87a5d36: compacting logs: from=1 to=92 +2025-10-30T18:21:34.306Z [INFO] server-034f19c5-db63-fbe6-552c-e58fd87a5d36: snapshot complete up to: index=102 +--- PASS: TestRaft_SnapshotRestore_Progress (0.48s) === RUN TestRaft_NoRestoreOnStart -2025-10-30T18:17:46.131Z [INFO] server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e Address:19aae2d4-c32d-846d-4f26-0a1aa55ad75e}]" -2025-10-30T18:17:46.131Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:46.131Z [INFO] server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: entering follower state: follower="Node at 19aae2d4-c32d-846d-4f26-0a1aa55ad75e [Follower]" leader-address= leader-id= -2025-10-30T18:17:46.214Z [WARN] server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:46.214Z [INFO] server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: entering candidate state: node="Node at 19aae2d4-c32d-846d-4f26-0a1aa55ad75e [Candidate]" term=2 -2025-10-30T18:17:46.214Z [INFO] server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: election won: term=2 tally=1 -2025-10-30T18:17:46.214Z [INFO] server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: entering leader state: leader="Node at 19aae2d4-c32d-846d-4f26-0a1aa55ad75e [Leader]" -2025-10-30T18:17:46.320Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:46.214361235 +0000 UTC m=+12.493035132 (1 nodes), 83.24442ms from start of poll, 83.377564ms from cluster start. Timeout at 2025-10-30 18:17:46.31936383 +0000 UTC m=+12.598037730, 105.002598ms after stability -2025-10-30T18:17:46.321Z [INFO] server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: starting snapshot up to: index=102 -2025-10-30T18:17:46.321Z [INFO] creating new snapshot: path=/tmp/raft474268794/snapshots/2-102-1761848266321.tmp -2025-10-30T18:17:46.322Z [INFO] server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: compacting logs: from=1 to=92 -2025-10-30T18:17:46.322Z [INFO] server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: snapshot complete up to: index=102 -2025-10-30T18:17:46.322Z [INFO] server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e Address:19aae2d4-c32d-846d-4f26-0a1aa55ad75e}]" ---- PASS: TestRaft_NoRestoreOnStart (0.19s) -2025-10-30T18:17:46.322Z [INFO] server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: entering follower state: follower="Node at 19aae2d4-c32d-846d-4f26-0a1aa55ad75e [Follower]" leader-address= leader-id= +2025-10-30T18:21:34.621Z [INFO] server-fd5515f0-42a0-6162-83d8-6bdb6d555888: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd5515f0-42a0-6162-83d8-6bdb6d555888 Address:fd5515f0-42a0-6162-83d8-6bdb6d555888}]" +2025-10-30T18:21:34.621Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:34.621Z [INFO] server-fd5515f0-42a0-6162-83d8-6bdb6d555888: entering follower state: follower="Node at fd5515f0-42a0-6162-83d8-6bdb6d555888 [Follower]" leader-address= leader-id= +2025-10-30T18:21:34.699Z [WARN] server-fd5515f0-42a0-6162-83d8-6bdb6d555888: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:34.699Z [INFO] server-fd5515f0-42a0-6162-83d8-6bdb6d555888: entering candidate state: node="Node at fd5515f0-42a0-6162-83d8-6bdb6d555888 [Candidate]" term=2 +2025-10-30T18:21:34.699Z [INFO] server-fd5515f0-42a0-6162-83d8-6bdb6d555888: election won: term=2 tally=1 +2025-10-30T18:21:34.699Z [INFO] server-fd5515f0-42a0-6162-83d8-6bdb6d555888: entering leader state: leader="Node at fd5515f0-42a0-6162-83d8-6bdb6d555888 [Leader]" +2025-10-30T18:21:34.805Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:34.699657984 +0000 UTC m=+17.773187278 (1 nodes), 78.155142ms from start of poll, 78.352853ms from cluster start. Timeout at 2025-10-30 18:21:34.804660443 +0000 UTC m=+17.878189737, 105.002459ms after stability +2025-10-30T18:21:34.807Z [INFO] server-fd5515f0-42a0-6162-83d8-6bdb6d555888: starting snapshot up to: index=102 +2025-10-30T18:21:34.807Z [INFO] creating new snapshot: path=/tmp/raft4212574721/snapshots/2-102-1761848494807.tmp +2025-10-30T18:21:34.960Z [INFO] server-fd5515f0-42a0-6162-83d8-6bdb6d555888: compacting logs: from=1 to=92 +2025-10-30T18:21:34.960Z [INFO] server-fd5515f0-42a0-6162-83d8-6bdb6d555888: snapshot complete up to: index=102 +2025-10-30T18:21:34.960Z [INFO] server-fd5515f0-42a0-6162-83d8-6bdb6d555888: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd5515f0-42a0-6162-83d8-6bdb6d555888 Address:fd5515f0-42a0-6162-83d8-6bdb6d555888}]" +--- PASS: TestRaft_NoRestoreOnStart (0.34s) === RUN TestRaft_SnapshotRestore_PeerChange +2025-10-30T18:21:34.960Z [INFO] server-fd5515f0-42a0-6162-83d8-6bdb6d555888: entering follower state: follower="Node at fd5515f0-42a0-6162-83d8-6bdb6d555888 [Follower]" leader-address= leader-id= raft_test.go:1266: Skipping unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot -2025-10-30T18:17:46.322Z [INFO] server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56 Address:b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56}]" -2025-10-30T18:17:46.322Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:46.322Z [INFO] server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: entering follower state: follower="Node at b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56 [Follower]" leader-address= leader-id= -2025-10-30T18:17:46.390Z [WARN] server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:46.390Z [INFO] server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: entering candidate state: node="Node at b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56 [Candidate]" term=2 -2025-10-30T18:17:46.390Z [INFO] server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: election won: term=2 tally=1 -2025-10-30T18:17:46.390Z [INFO] server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: entering leader state: leader="Node at b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56 [Leader]" -2025-10-30T18:17:46.422Z [WARN] server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:46.422Z [INFO] server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: entering candidate state: node="Node at 19aae2d4-c32d-846d-4f26-0a1aa55ad75e [Candidate]" term=3 -2025-10-30T18:17:46.422Z [INFO] server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: election won: term=3 tally=1 -2025-10-30T18:17:46.422Z [INFO] server-19aae2d4-c32d-846d-4f26-0a1aa55ad75e: entering leader state: leader="Node at 19aae2d4-c32d-846d-4f26-0a1aa55ad75e [Leader]" -2025-10-30T18:17:46.495Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:46.390510961 +0000 UTC m=+12.669184868 (1 nodes), 67.717156ms from start of poll, 67.842644ms from cluster start. Timeout at 2025-10-30 18:17:46.49551402 +0000 UTC m=+12.774187917, 105.003049ms after stability -2025-10-30T18:17:46.509Z [INFO] server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: starting snapshot up to: index=102 -2025-10-30T18:17:46.509Z [INFO] creating new snapshot: path=/tmp/raft4100405487/snapshots/2-102-1761848266509.tmp -2025-10-30T18:17:46.549Z [INFO] server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: compacting logs: from=1 to=92 -2025-10-30T18:17:46.549Z [INFO] server-b97a0ee4-5d4f-1fd2-a8eb-6a1945bb8d56: snapshot complete up to: index=102 ---- PASS: TestRaft_AutoSnapshot (0.28s) +2025-10-30T18:21:34.960Z [INFO] server-dc7697b5-4593-ecbd-1cab-fd438f10de1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc7697b5-4593-ecbd-1cab-fd438f10de1a Address:dc7697b5-4593-ecbd-1cab-fd438f10de1a}]" +2025-10-30T18:21:34.961Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:34.961Z [INFO] server-dc7697b5-4593-ecbd-1cab-fd438f10de1a: entering follower state: follower="Node at dc7697b5-4593-ecbd-1cab-fd438f10de1a [Follower]" leader-address= leader-id= +2025-10-30T18:21:35.055Z [WARN] server-dc7697b5-4593-ecbd-1cab-fd438f10de1a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:35.055Z [INFO] server-dc7697b5-4593-ecbd-1cab-fd438f10de1a: entering candidate state: node="Node at dc7697b5-4593-ecbd-1cab-fd438f10de1a [Candidate]" term=2 +2025-10-30T18:21:35.055Z [INFO] server-dc7697b5-4593-ecbd-1cab-fd438f10de1a: election won: term=2 tally=1 +2025-10-30T18:21:35.055Z [INFO] server-dc7697b5-4593-ecbd-1cab-fd438f10de1a: entering leader state: leader="Node at dc7697b5-4593-ecbd-1cab-fd438f10de1a [Leader]" +2025-10-30T18:21:35.057Z [WARN] server-fd5515f0-42a0-6162-83d8-6bdb6d555888: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:35.057Z [INFO] server-fd5515f0-42a0-6162-83d8-6bdb6d555888: entering candidate state: node="Node at fd5515f0-42a0-6162-83d8-6bdb6d555888 [Candidate]" term=3 +2025-10-30T18:21:35.057Z [INFO] server-fd5515f0-42a0-6162-83d8-6bdb6d555888: election won: term=3 tally=1 +2025-10-30T18:21:35.057Z [INFO] server-fd5515f0-42a0-6162-83d8-6bdb6d555888: entering leader state: leader="Node at fd5515f0-42a0-6162-83d8-6bdb6d555888 [Leader]" +2025-10-30T18:21:35.160Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:35.05527758 +0000 UTC m=+18.128806872 (1 nodes), 94.252651ms from start of poll, 94.399615ms from cluster start. Timeout at 2025-10-30 18:21:35.160280096 +0000 UTC m=+18.233809388, 105.002516ms after stability +2025-10-30T18:21:35.164Z [INFO] server-dc7697b5-4593-ecbd-1cab-fd438f10de1a: starting snapshot up to: index=102 +2025-10-30T18:21:35.164Z [INFO] creating new snapshot: path=/tmp/raft3944086209/snapshots/2-102-1761848495164.tmp +2025-10-30T18:21:35.259Z [INFO] server-dc7697b5-4593-ecbd-1cab-fd438f10de1a: compacting logs: from=1 to=92 +2025-10-30T18:21:35.259Z [INFO] server-dc7697b5-4593-ecbd-1cab-fd438f10de1a: snapshot complete up to: index=102 +--- PASS: TestRaft_AutoSnapshot (0.31s) === RUN TestRaft_UserSnapshot -2025-10-30T18:17:46.603Z [INFO] server-c84e118b-1c37-d4b9-5271-cd723944ddd2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c84e118b-1c37-d4b9-5271-cd723944ddd2 Address:c84e118b-1c37-d4b9-5271-cd723944ddd2}]" -2025-10-30T18:17:46.603Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:46.603Z [INFO] server-c84e118b-1c37-d4b9-5271-cd723944ddd2: entering follower state: follower="Node at c84e118b-1c37-d4b9-5271-cd723944ddd2 [Follower]" leader-address= leader-id= -2025-10-30T18:17:46.690Z [WARN] server-c84e118b-1c37-d4b9-5271-cd723944ddd2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:46.690Z [INFO] server-c84e118b-1c37-d4b9-5271-cd723944ddd2: entering candidate state: node="Node at c84e118b-1c37-d4b9-5271-cd723944ddd2 [Candidate]" term=2 -2025-10-30T18:17:46.690Z [INFO] server-c84e118b-1c37-d4b9-5271-cd723944ddd2: election won: term=2 tally=1 -2025-10-30T18:17:46.690Z [INFO] server-c84e118b-1c37-d4b9-5271-cd723944ddd2: entering leader state: leader="Node at c84e118b-1c37-d4b9-5271-cd723944ddd2 [Leader]" -2025-10-30T18:17:46.795Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:46.690496161 +0000 UTC m=+12.969170063 (1 nodes), 86.968628ms from start of poll, 87.07939ms from cluster start. Timeout at 2025-10-30 18:17:46.795497545 +0000 UTC m=+13.074171444, 105.001381ms after stability -2025-10-30T18:17:46.795Z [ERROR] server-c84e118b-1c37-d4b9-5271-cd723944ddd2: failed to take snapshot: error="nothing new to snapshot" -2025-10-30T18:17:46.795Z [INFO] server-c84e118b-1c37-d4b9-5271-cd723944ddd2: starting snapshot up to: index=12 -2025-10-30T18:17:46.796Z [INFO] creating new snapshot: path=/tmp/raft2872604898/snapshots/2-12-1761848266796.tmp -2025-10-30T18:17:46.797Z [INFO] server-c84e118b-1c37-d4b9-5271-cd723944ddd2: compacting logs: from=1 to=2 -2025-10-30T18:17:46.797Z [INFO] server-c84e118b-1c37-d4b9-5271-cd723944ddd2: snapshot complete up to: index=12 ---- PASS: TestRaft_UserSnapshot (0.19s) +2025-10-30T18:21:35.271Z [INFO] server-ea7b0e96-610c-8dcd-8ac6-74e0667417a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea7b0e96-610c-8dcd-8ac6-74e0667417a6 Address:ea7b0e96-610c-8dcd-8ac6-74e0667417a6}]" +2025-10-30T18:21:35.271Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:35.271Z [INFO] server-ea7b0e96-610c-8dcd-8ac6-74e0667417a6: entering follower state: follower="Node at ea7b0e96-610c-8dcd-8ac6-74e0667417a6 [Follower]" leader-address= leader-id= +2025-10-30T18:21:35.361Z [WARN] server-ea7b0e96-610c-8dcd-8ac6-74e0667417a6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:35.361Z [INFO] server-ea7b0e96-610c-8dcd-8ac6-74e0667417a6: entering candidate state: node="Node at ea7b0e96-610c-8dcd-8ac6-74e0667417a6 [Candidate]" term=2 +2025-10-30T18:21:35.361Z [INFO] server-ea7b0e96-610c-8dcd-8ac6-74e0667417a6: election won: term=2 tally=1 +2025-10-30T18:21:35.361Z [INFO] server-ea7b0e96-610c-8dcd-8ac6-74e0667417a6: entering leader state: leader="Node at ea7b0e96-610c-8dcd-8ac6-74e0667417a6 [Leader]" +2025-10-30T18:21:35.466Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:35.361451495 +0000 UTC m=+18.434980805 (1 nodes), 89.782963ms from start of poll, 90.084588ms from cluster start. Timeout at 2025-10-30 18:21:35.466452592 +0000 UTC m=+18.539981918, 105.001113ms after stability +2025-10-30T18:21:35.467Z [ERROR] server-ea7b0e96-610c-8dcd-8ac6-74e0667417a6: failed to take snapshot: error="nothing new to snapshot" +2025-10-30T18:21:35.467Z [INFO] server-ea7b0e96-610c-8dcd-8ac6-74e0667417a6: starting snapshot up to: index=12 +2025-10-30T18:21:35.467Z [INFO] creating new snapshot: path=/tmp/raft3187357819/snapshots/2-12-1761848495467.tmp +2025-10-30T18:21:35.470Z [INFO] server-ea7b0e96-610c-8dcd-8ac6-74e0667417a6: compacting logs: from=1 to=2 +2025-10-30T18:21:35.470Z [INFO] server-ea7b0e96-610c-8dcd-8ac6-74e0667417a6: snapshot complete up to: index=12 +--- PASS: TestRaft_UserSnapshot (0.22s) === RUN TestRaft_UserRestore === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false -2025-10-30T18:17:46.797Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45efb12f-99ac-27dc-797c-5f27cce802aa Address:45efb12f-99ac-27dc-797c-5f27cce802aa} {Suffrage:Voter ID:server-7813d007-382a-7aed-fc45-b527ad4cc34c Address:7813d007-382a-7aed-fc45-b527ad4cc34c} {Suffrage:Voter ID:server-2bd4d265-c072-7715-e62a-6f65e0df8015 Address:2bd4d265-c072-7715-e62a-6f65e0df8015}]" -2025-10-30T18:17:46.798Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: entering follower state: follower="Node at 45efb12f-99ac-27dc-797c-5f27cce802aa [Follower]" leader-address= leader-id= -2025-10-30T18:17:46.798Z [INFO] server-7813d007-382a-7aed-fc45-b527ad4cc34c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45efb12f-99ac-27dc-797c-5f27cce802aa Address:45efb12f-99ac-27dc-797c-5f27cce802aa} {Suffrage:Voter ID:server-7813d007-382a-7aed-fc45-b527ad4cc34c Address:7813d007-382a-7aed-fc45-b527ad4cc34c} {Suffrage:Voter ID:server-2bd4d265-c072-7715-e62a-6f65e0df8015 Address:2bd4d265-c072-7715-e62a-6f65e0df8015}]" -2025-10-30T18:17:46.798Z [INFO] server-2bd4d265-c072-7715-e62a-6f65e0df8015: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45efb12f-99ac-27dc-797c-5f27cce802aa Address:45efb12f-99ac-27dc-797c-5f27cce802aa} {Suffrage:Voter ID:server-7813d007-382a-7aed-fc45-b527ad4cc34c Address:7813d007-382a-7aed-fc45-b527ad4cc34c} {Suffrage:Voter ID:server-2bd4d265-c072-7715-e62a-6f65e0df8015 Address:2bd4d265-c072-7715-e62a-6f65e0df8015}]" -2025-10-30T18:17:46.798Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:46.798Z [INFO] server-2bd4d265-c072-7715-e62a-6f65e0df8015: entering follower state: follower="Node at 2bd4d265-c072-7715-e62a-6f65e0df8015 [Follower]" leader-address= leader-id= -2025-10-30T18:17:46.798Z [INFO] server-7813d007-382a-7aed-fc45-b527ad4cc34c: entering follower state: follower="Node at 7813d007-382a-7aed-fc45-b527ad4cc34c [Follower]" leader-address= leader-id= -2025-10-30T18:17:47.384Z [WARN] server-45efb12f-99ac-27dc-797c-5f27cce802aa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:47.384Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: entering candidate state: node="Node at 45efb12f-99ac-27dc-797c-5f27cce802aa [Candidate]" term=2 -2025-10-30T18:17:47.384Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: election won: term=2 tally=2 -2025-10-30T18:17:47.384Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: entering leader state: leader="Node at 45efb12f-99ac-27dc-797c-5f27cce802aa [Leader]" -2025-10-30T18:17:47.384Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: added peer, starting replication: peer=server-7813d007-382a-7aed-fc45-b527ad4cc34c -2025-10-30T18:17:47.384Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: added peer, starting replication: peer=server-2bd4d265-c072-7715-e62a-6f65e0df8015 -2025-10-30T18:17:47.385Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: pipelining replication: peer="{Voter server-7813d007-382a-7aed-fc45-b527ad4cc34c 7813d007-382a-7aed-fc45-b527ad4cc34c}" -2025-10-30T18:17:47.385Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: pipelining replication: peer="{Voter server-2bd4d265-c072-7715-e62a-6f65e0df8015 2bd4d265-c072-7715-e62a-6f65e0df8015}" -2025-10-30T18:17:48.391Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:47.385032042 +0000 UTC m=+13.663705939 (1 nodes), 586.938627ms from start of poll, 587.143815ms from cluster start. Timeout at 2025-10-30 18:17:48.390042036 +0000 UTC m=+14.668715947, 1.005010008s after stability -2025-10-30T18:17:48.392Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: starting snapshot up to: index=12 -2025-10-30T18:17:48.392Z [INFO] creating new snapshot: path=/tmp/raft2694164508/snapshots/2-12-1761848268392.tmp -2025-10-30T18:17:48.393Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: snapshot complete up to: index=12 -2025-10-30T18:17:48.394Z [INFO] creating new snapshot: path=/tmp/raft2694164508/snapshots/2-23-1761848268394.tmp -2025-10-30T18:17:48.403Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: copied to local snapshot: bytes=71 -2025-10-30T18:17:48.404Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: snapshot restore progress: id=2-23-1761848268394 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:17:48.404Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: restored user snapshot: index=23 -2025-10-30T18:17:48.404Z [ERROR] server-45efb12f-99ac-27dc-797c-5f27cce802aa: failed to get log: index=23 error="log not found" -2025-10-30T18:17:48.404Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: aborting pipeline replication: peer="{Voter server-2bd4d265-c072-7715-e62a-6f65e0df8015 2bd4d265-c072-7715-e62a-6f65e0df8015}" -2025-10-30T18:17:48.404Z [ERROR] server-45efb12f-99ac-27dc-797c-5f27cce802aa: failed to get log: index=23 error="log not found" -2025-10-30T18:17:48.404Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: aborting pipeline replication: peer="{Voter server-7813d007-382a-7aed-fc45-b527ad4cc34c 7813d007-382a-7aed-fc45-b527ad4cc34c}" -2025-10-30T18:17:48.414Z [ERROR] server-45efb12f-99ac-27dc-797c-5f27cce802aa: failed to get log: index=23 error="log not found" -2025-10-30T18:17:48.414Z [ERROR] server-45efb12f-99ac-27dc-797c-5f27cce802aa: failed to get log: index=23 error="log not found" -2025-10-30T18:17:48.414Z [INFO] creating new snapshot: path=/tmp/raft2682460278/snapshots/2-23-1761848268414.tmp -2025-10-30T18:17:48.414Z [INFO] server-2bd4d265-c072-7715-e62a-6f65e0df8015: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:17:48.414Z [INFO] creating new snapshot: path=/tmp/raft1292056249/snapshots/2-23-1761848268414.tmp -2025-10-30T18:17:48.415Z [INFO] server-2bd4d265-c072-7715-e62a-6f65e0df8015: copied to local snapshot: bytes=71 -2025-10-30T18:17:48.415Z [INFO] server-7813d007-382a-7aed-fc45-b527ad4cc34c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:17:48.415Z [INFO] server-2bd4d265-c072-7715-e62a-6f65e0df8015: snapshot restore progress: id=2-23-1761848268414 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:17:48.415Z [INFO] server-2bd4d265-c072-7715-e62a-6f65e0df8015: Installed remote snapshot -2025-10-30T18:17:48.415Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: pipelining replication: peer="{Voter server-2bd4d265-c072-7715-e62a-6f65e0df8015 2bd4d265-c072-7715-e62a-6f65e0df8015}" -2025-10-30T18:17:48.415Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:17:48.415Z [INFO] server-7813d007-382a-7aed-fc45-b527ad4cc34c: copied to local snapshot: bytes=71 -2025-10-30T18:17:48.416Z [INFO] server-7813d007-382a-7aed-fc45-b527ad4cc34c: snapshot restore progress: id=2-23-1761848268414 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:17:48.416Z [INFO] server-7813d007-382a-7aed-fc45-b527ad4cc34c: Installed remote snapshot -2025-10-30T18:17:48.416Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: pipelining replication: peer="{Voter server-7813d007-382a-7aed-fc45-b527ad4cc34c 7813d007-382a-7aed-fc45-b527ad4cc34c}" -2025-10-30T18:17:49.421Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:17:48.415452593 +0000 UTC m=+14.694126491 (2 nodes), 561ns from start of poll, 1.617564367s from cluster start. Timeout at 2025-10-30 18:17:49.420455911 +0000 UTC m=+15.699129808, 1.005003317s after stability -2025-10-30T18:17:49.433Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: aborting pipeline replication: peer="{Voter server-2bd4d265-c072-7715-e62a-6f65e0df8015 2bd4d265-c072-7715-e62a-6f65e0df8015}" -2025-10-30T18:17:49.433Z [INFO] server-45efb12f-99ac-27dc-797c-5f27cce802aa: aborting pipeline replication: peer="{Voter server-7813d007-382a-7aed-fc45-b527ad4cc34c 7813d007-382a-7aed-fc45-b527ad4cc34c}" +2025-10-30T18:21:35.495Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3 Address:aa94c5b4-7ca9-e1e2-5719-c354a8273fd3} {Suffrage:Voter ID:server-d1a6fa63-3806-17ff-66d2-3e411c1d58b9 Address:d1a6fa63-3806-17ff-66d2-3e411c1d58b9} {Suffrage:Voter ID:server-e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0 Address:e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0}]" +2025-10-30T18:21:35.495Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: entering follower state: follower="Node at aa94c5b4-7ca9-e1e2-5719-c354a8273fd3 [Follower]" leader-address= leader-id= +2025-10-30T18:21:35.495Z [INFO] server-d1a6fa63-3806-17ff-66d2-3e411c1d58b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3 Address:aa94c5b4-7ca9-e1e2-5719-c354a8273fd3} {Suffrage:Voter ID:server-d1a6fa63-3806-17ff-66d2-3e411c1d58b9 Address:d1a6fa63-3806-17ff-66d2-3e411c1d58b9} {Suffrage:Voter ID:server-e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0 Address:e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0}]" +2025-10-30T18:21:35.495Z [INFO] server-d1a6fa63-3806-17ff-66d2-3e411c1d58b9: entering follower state: follower="Node at d1a6fa63-3806-17ff-66d2-3e411c1d58b9 [Follower]" leader-address= leader-id= +2025-10-30T18:21:35.495Z [INFO] server-e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3 Address:aa94c5b4-7ca9-e1e2-5719-c354a8273fd3} {Suffrage:Voter ID:server-d1a6fa63-3806-17ff-66d2-3e411c1d58b9 Address:d1a6fa63-3806-17ff-66d2-3e411c1d58b9} {Suffrage:Voter ID:server-e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0 Address:e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0}]" +2025-10-30T18:21:35.495Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:35.495Z [INFO] server-e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0: entering follower state: follower="Node at e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0 [Follower]" leader-address= leader-id= +2025-10-30T18:21:36.020Z [WARN] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:36.020Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: entering candidate state: node="Node at aa94c5b4-7ca9-e1e2-5719-c354a8273fd3 [Candidate]" term=2 +2025-10-30T18:21:36.020Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: election won: term=2 tally=2 +2025-10-30T18:21:36.020Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: entering leader state: leader="Node at aa94c5b4-7ca9-e1e2-5719-c354a8273fd3 [Leader]" +2025-10-30T18:21:36.020Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: added peer, starting replication: peer=server-d1a6fa63-3806-17ff-66d2-3e411c1d58b9 +2025-10-30T18:21:36.020Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: added peer, starting replication: peer=server-e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0 +2025-10-30T18:21:36.021Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: pipelining replication: peer="{Voter server-d1a6fa63-3806-17ff-66d2-3e411c1d58b9 d1a6fa63-3806-17ff-66d2-3e411c1d58b9}" +2025-10-30T18:21:36.021Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: pipelining replication: peer="{Voter server-e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0 e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0}" +2025-10-30T18:21:37.026Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:36.020840583 +0000 UTC m=+19.094369875 (1 nodes), 525.189455ms from start of poll, 525.535382ms from cluster start. Timeout at 2025-10-30 18:21:37.025841524 +0000 UTC m=+20.099370812, 1.005000937s after stability +2025-10-30T18:21:37.026Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: starting snapshot up to: index=12 +2025-10-30T18:21:37.026Z [INFO] creating new snapshot: path=/tmp/raft1063981571/snapshots/2-12-1761848497026.tmp +2025-10-30T18:21:37.136Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: snapshot complete up to: index=12 +2025-10-30T18:21:37.137Z [INFO] creating new snapshot: path=/tmp/raft1063981571/snapshots/2-23-1761848497136.tmp +2025-10-30T18:21:37.266Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: copied to local snapshot: bytes=71 +2025-10-30T18:21:37.267Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: snapshot restore progress: id=2-23-1761848497136 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:37.267Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: restored user snapshot: index=23 +2025-10-30T18:21:37.267Z [ERROR] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: failed to get log: index=23 error="log not found" +2025-10-30T18:21:37.267Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: aborting pipeline replication: peer="{Voter server-d1a6fa63-3806-17ff-66d2-3e411c1d58b9 d1a6fa63-3806-17ff-66d2-3e411c1d58b9}" +2025-10-30T18:21:37.267Z [ERROR] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: failed to get log: index=23 error="log not found" +2025-10-30T18:21:37.267Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: aborting pipeline replication: peer="{Voter server-e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0 e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0}" +2025-10-30T18:21:37.274Z [ERROR] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: failed to get log: index=23 error="log not found" +2025-10-30T18:21:37.275Z [INFO] creating new snapshot: path=/tmp/raft3099701761/snapshots/2-23-1761848497275.tmp +2025-10-30T18:21:37.275Z [INFO] server-e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:37.277Z [ERROR] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: failed to get log: index=23 error="log not found" +2025-10-30T18:21:37.278Z [INFO] creating new snapshot: path=/tmp/raft111430821/snapshots/2-23-1761848497278.tmp +2025-10-30T18:21:37.278Z [INFO] server-d1a6fa63-3806-17ff-66d2-3e411c1d58b9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:37.622Z [INFO] server-e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0: copied to local snapshot: bytes=71 +2025-10-30T18:21:37.622Z [INFO] server-d1a6fa63-3806-17ff-66d2-3e411c1d58b9: copied to local snapshot: bytes=71 +2025-10-30T18:21:37.622Z [INFO] server-e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0: snapshot restore progress: id=2-23-1761848497275 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:37.622Z [INFO] server-d1a6fa63-3806-17ff-66d2-3e411c1d58b9: snapshot restore progress: id=2-23-1761848497278 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:37.622Z [INFO] server-e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0: Installed remote snapshot +2025-10-30T18:21:37.622Z [INFO] server-d1a6fa63-3806-17ff-66d2-3e411c1d58b9: Installed remote snapshot +2025-10-30T18:21:37.622Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: pipelining replication: peer="{Voter server-d1a6fa63-3806-17ff-66d2-3e411c1d58b9 d1a6fa63-3806-17ff-66d2-3e411c1d58b9}" +2025-10-30T18:21:37.622Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:21:37.622Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: pipelining replication: peer="{Voter server-e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0 e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0}" +2025-10-30T18:21:38.628Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:21:37.622518843 +0000 UTC m=+20.696048146 (2 nodes), 556ns from start of poll, 2.127213653s from cluster start. Timeout at 2025-10-30 18:21:38.627524972 +0000 UTC m=+21.701054261, 1.005006115s after stability +2025-10-30T18:21:38.640Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: aborting pipeline replication: peer="{Voter server-e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0 e50efa9a-e59b-e8cc-bc7f-1e7cf84ddba0}" +2025-10-30T18:21:38.640Z [INFO] server-aa94c5b4-7ca9-e1e2-5719-c354a8273fd3: aborting pipeline replication: peer="{Voter server-d1a6fa63-3806-17ff-66d2-3e411c1d58b9 d1a6fa63-3806-17ff-66d2-3e411c1d58b9}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false -2025-10-30T18:17:49.434Z [INFO] server-57817e28-20c9-beea-71a8-6c74eeadb438: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57817e28-20c9-beea-71a8-6c74eeadb438 Address:57817e28-20c9-beea-71a8-6c74eeadb438} {Suffrage:Voter ID:server-608786fe-c4a9-d217-e925-0891ab0573af Address:608786fe-c4a9-d217-e925-0891ab0573af} {Suffrage:Voter ID:server-38d43f82-a7a3-03e6-da08-5604df1b6901 Address:38d43f82-a7a3-03e6-da08-5604df1b6901}]" -2025-10-30T18:17:49.434Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57817e28-20c9-beea-71a8-6c74eeadb438 Address:57817e28-20c9-beea-71a8-6c74eeadb438} {Suffrage:Voter ID:server-608786fe-c4a9-d217-e925-0891ab0573af Address:608786fe-c4a9-d217-e925-0891ab0573af} {Suffrage:Voter ID:server-38d43f82-a7a3-03e6-da08-5604df1b6901 Address:38d43f82-a7a3-03e6-da08-5604df1b6901}]" -2025-10-30T18:17:49.434Z [INFO] server-57817e28-20c9-beea-71a8-6c74eeadb438: entering follower state: follower="Node at 57817e28-20c9-beea-71a8-6c74eeadb438 [Follower]" leader-address= leader-id= -2025-10-30T18:17:49.434Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: entering follower state: follower="Node at 608786fe-c4a9-d217-e925-0891ab0573af [Follower]" leader-address= leader-id= -2025-10-30T18:17:49.434Z [INFO] server-38d43f82-a7a3-03e6-da08-5604df1b6901: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57817e28-20c9-beea-71a8-6c74eeadb438 Address:57817e28-20c9-beea-71a8-6c74eeadb438} {Suffrage:Voter ID:server-608786fe-c4a9-d217-e925-0891ab0573af Address:608786fe-c4a9-d217-e925-0891ab0573af} {Suffrage:Voter ID:server-38d43f82-a7a3-03e6-da08-5604df1b6901 Address:38d43f82-a7a3-03e6-da08-5604df1b6901}]" -2025-10-30T18:17:49.434Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:49.434Z [INFO] server-38d43f82-a7a3-03e6-da08-5604df1b6901: entering follower state: follower="Node at 38d43f82-a7a3-03e6-da08-5604df1b6901 [Follower]" leader-address= leader-id= -2025-10-30T18:17:50.098Z [WARN] server-608786fe-c4a9-d217-e925-0891ab0573af: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:50.098Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: entering candidate state: node="Node at 608786fe-c4a9-d217-e925-0891ab0573af [Candidate]" term=2 -2025-10-30T18:17:50.098Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: election won: term=2 tally=2 -2025-10-30T18:17:50.098Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: entering leader state: leader="Node at 608786fe-c4a9-d217-e925-0891ab0573af [Leader]" -2025-10-30T18:17:50.098Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: added peer, starting replication: peer=server-57817e28-20c9-beea-71a8-6c74eeadb438 -2025-10-30T18:17:50.098Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: added peer, starting replication: peer=server-38d43f82-a7a3-03e6-da08-5604df1b6901 -2025-10-30T18:17:50.099Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: pipelining replication: peer="{Voter server-57817e28-20c9-beea-71a8-6c74eeadb438 57817e28-20c9-beea-71a8-6c74eeadb438}" -2025-10-30T18:17:50.099Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: pipelining replication: peer="{Voter server-38d43f82-a7a3-03e6-da08-5604df1b6901 38d43f82-a7a3-03e6-da08-5604df1b6901}" -2025-10-30T18:17:51.104Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:50.09897598 +0000 UTC m=+16.377649878 (1 nodes), 664.683163ms from start of poll, 664.92151ms from cluster start. Timeout at 2025-10-30 18:17:51.103978867 +0000 UTC m=+17.382652765, 1.005002887s after stability -2025-10-30T18:17:51.104Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: starting snapshot up to: index=12 -2025-10-30T18:17:51.104Z [INFO] creating new snapshot: path=/tmp/raft3530938663/snapshots/2-12-1761848271104.tmp -2025-10-30T18:17:51.105Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: snapshot complete up to: index=12 -2025-10-30T18:17:51.105Z [INFO] creating new snapshot: path=/tmp/raft3530938663/snapshots/2-23-1761848271105.tmp -2025-10-30T18:17:51.106Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: copied to local snapshot: bytes=71 -2025-10-30T18:17:51.106Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: snapshot restore progress: id=2-23-1761848271105 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:17:51.106Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: removing all old logs from log store -2025-10-30T18:17:51.106Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: compacting logs: from=1 to=22 -2025-10-30T18:17:51.106Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: restored user snapshot: index=23 -2025-10-30T18:17:51.106Z [ERROR] server-608786fe-c4a9-d217-e925-0891ab0573af: failed to get log: index=22 error="log not found" -2025-10-30T18:17:51.106Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: aborting pipeline replication: peer="{Voter server-38d43f82-a7a3-03e6-da08-5604df1b6901 38d43f82-a7a3-03e6-da08-5604df1b6901}" -2025-10-30T18:17:51.106Z [ERROR] server-608786fe-c4a9-d217-e925-0891ab0573af: failed to get log: index=22 error="log not found" -2025-10-30T18:17:51.106Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: aborting pipeline replication: peer="{Voter server-57817e28-20c9-beea-71a8-6c74eeadb438 57817e28-20c9-beea-71a8-6c74eeadb438}" -2025-10-30T18:17:51.113Z [ERROR] server-608786fe-c4a9-d217-e925-0891ab0573af: failed to get log: index=22 error="log not found" -2025-10-30T18:17:51.113Z [INFO] creating new snapshot: path=/tmp/raft205117741/snapshots/2-23-1761848271113.tmp -2025-10-30T18:17:51.114Z [INFO] server-38d43f82-a7a3-03e6-da08-5604df1b6901: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:17:51.114Z [INFO] server-38d43f82-a7a3-03e6-da08-5604df1b6901: copied to local snapshot: bytes=71 -2025-10-30T18:17:51.114Z [ERROR] server-608786fe-c4a9-d217-e925-0891ab0573af: failed to get log: index=22 error="log not found" -2025-10-30T18:17:51.114Z [INFO] server-38d43f82-a7a3-03e6-da08-5604df1b6901: snapshot restore progress: id=2-23-1761848271113 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:17:51.114Z [INFO] server-38d43f82-a7a3-03e6-da08-5604df1b6901: removing all old logs from log store -2025-10-30T18:17:51.114Z [INFO] server-38d43f82-a7a3-03e6-da08-5604df1b6901: compacting logs: from=1 to=22 -2025-10-30T18:17:51.114Z [INFO] server-38d43f82-a7a3-03e6-da08-5604df1b6901: Installed remote snapshot -2025-10-30T18:17:51.114Z [WARN] server-38d43f82-a7a3-03e6-da08-5604df1b6901: skipping application of old log: index=22 -2025-10-30T18:17:51.114Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: pipelining replication: peer="{Voter server-38d43f82-a7a3-03e6-da08-5604df1b6901 38d43f82-a7a3-03e6-da08-5604df1b6901}" -2025-10-30T18:17:51.114Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:17:51.115Z [INFO] creating new snapshot: path=/tmp/raft1192626479/snapshots/2-23-1761848271115.tmp -2025-10-30T18:17:51.115Z [INFO] server-57817e28-20c9-beea-71a8-6c74eeadb438: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:17:51.115Z [INFO] server-57817e28-20c9-beea-71a8-6c74eeadb438: copied to local snapshot: bytes=71 -2025-10-30T18:17:51.115Z [INFO] server-57817e28-20c9-beea-71a8-6c74eeadb438: snapshot restore progress: id=2-23-1761848271115 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:17:51.115Z [INFO] server-57817e28-20c9-beea-71a8-6c74eeadb438: removing all old logs from log store -2025-10-30T18:17:51.115Z [INFO] server-57817e28-20c9-beea-71a8-6c74eeadb438: compacting logs: from=1 to=22 -2025-10-30T18:17:51.115Z [INFO] server-57817e28-20c9-beea-71a8-6c74eeadb438: Installed remote snapshot -2025-10-30T18:17:51.115Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: pipelining replication: peer="{Voter server-57817e28-20c9-beea-71a8-6c74eeadb438 57817e28-20c9-beea-71a8-6c74eeadb438}" -2025-10-30T18:17:52.120Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:17:51.114991304 +0000 UTC m=+17.393665210 (2 nodes), 615ns from start of poll, 1.680936842s from cluster start. Timeout at 2025-10-30 18:17:52.11999406 +0000 UTC m=+18.398667958, 1.005002748s after stability -2025-10-30T18:17:52.132Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: aborting pipeline replication: peer="{Voter server-38d43f82-a7a3-03e6-da08-5604df1b6901 38d43f82-a7a3-03e6-da08-5604df1b6901}" -2025-10-30T18:17:52.132Z [INFO] server-608786fe-c4a9-d217-e925-0891ab0573af: aborting pipeline replication: peer="{Voter server-57817e28-20c9-beea-71a8-6c74eeadb438 57817e28-20c9-beea-71a8-6c74eeadb438}" +2025-10-30T18:21:38.775Z [INFO] server-0056941c-0598-0b8d-c77e-037b0b0d5539: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0056941c-0598-0b8d-c77e-037b0b0d5539 Address:0056941c-0598-0b8d-c77e-037b0b0d5539} {Suffrage:Voter ID:server-5e5493c5-455c-6fbc-2a42-8408924a19cb Address:5e5493c5-455c-6fbc-2a42-8408924a19cb} {Suffrage:Voter ID:server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08 Address:528a34e1-5c6c-9d7a-a23d-7a82caa5ac08}]" +2025-10-30T18:21:38.775Z [INFO] server-0056941c-0598-0b8d-c77e-037b0b0d5539: entering follower state: follower="Node at 0056941c-0598-0b8d-c77e-037b0b0d5539 [Follower]" leader-address= leader-id= +2025-10-30T18:21:38.775Z [INFO] server-5e5493c5-455c-6fbc-2a42-8408924a19cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0056941c-0598-0b8d-c77e-037b0b0d5539 Address:0056941c-0598-0b8d-c77e-037b0b0d5539} {Suffrage:Voter ID:server-5e5493c5-455c-6fbc-2a42-8408924a19cb Address:5e5493c5-455c-6fbc-2a42-8408924a19cb} {Suffrage:Voter ID:server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08 Address:528a34e1-5c6c-9d7a-a23d-7a82caa5ac08}]" +2025-10-30T18:21:38.775Z [INFO] server-5e5493c5-455c-6fbc-2a42-8408924a19cb: entering follower state: follower="Node at 5e5493c5-455c-6fbc-2a42-8408924a19cb [Follower]" leader-address= leader-id= +2025-10-30T18:21:38.775Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0056941c-0598-0b8d-c77e-037b0b0d5539 Address:0056941c-0598-0b8d-c77e-037b0b0d5539} {Suffrage:Voter ID:server-5e5493c5-455c-6fbc-2a42-8408924a19cb Address:5e5493c5-455c-6fbc-2a42-8408924a19cb} {Suffrage:Voter ID:server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08 Address:528a34e1-5c6c-9d7a-a23d-7a82caa5ac08}]" +2025-10-30T18:21:38.775Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:38.775Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: entering follower state: follower="Node at 528a34e1-5c6c-9d7a-a23d-7a82caa5ac08 [Follower]" leader-address= leader-id= +2025-10-30T18:21:39.469Z [WARN] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:39.470Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: entering candidate state: node="Node at 528a34e1-5c6c-9d7a-a23d-7a82caa5ac08 [Candidate]" term=2 +2025-10-30T18:21:39.470Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: election won: term=2 tally=2 +2025-10-30T18:21:39.470Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: entering leader state: leader="Node at 528a34e1-5c6c-9d7a-a23d-7a82caa5ac08 [Leader]" +2025-10-30T18:21:39.470Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: added peer, starting replication: peer=server-0056941c-0598-0b8d-c77e-037b0b0d5539 +2025-10-30T18:21:39.470Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: added peer, starting replication: peer=server-5e5493c5-455c-6fbc-2a42-8408924a19cb +2025-10-30T18:21:39.470Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: pipelining replication: peer="{Voter server-0056941c-0598-0b8d-c77e-037b0b0d5539 0056941c-0598-0b8d-c77e-037b0b0d5539}" +2025-10-30T18:21:39.470Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: pipelining replication: peer="{Voter server-5e5493c5-455c-6fbc-2a42-8408924a19cb 5e5493c5-455c-6fbc-2a42-8408924a19cb}" +2025-10-30T18:21:40.475Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:39.470189759 +0000 UTC m=+22.543719071 (1 nodes), 694.736175ms from start of poll, 695.181762ms from cluster start. Timeout at 2025-10-30 18:21:40.475193087 +0000 UTC m=+23.548722379, 1.005003308s after stability +2025-10-30T18:21:40.476Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: starting snapshot up to: index=12 +2025-10-30T18:21:40.476Z [INFO] creating new snapshot: path=/tmp/raft1256622950/snapshots/2-12-1761848500476.tmp +2025-10-30T18:21:40.575Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: snapshot complete up to: index=12 +2025-10-30T18:21:40.576Z [INFO] creating new snapshot: path=/tmp/raft1256622950/snapshots/2-23-1761848500576.tmp +2025-10-30T18:21:40.577Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: copied to local snapshot: bytes=71 +2025-10-30T18:21:40.577Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: snapshot restore progress: id=2-23-1761848500576 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:40.577Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: removing all old logs from log store +2025-10-30T18:21:40.577Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: compacting logs: from=1 to=22 +2025-10-30T18:21:40.577Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: restored user snapshot: index=23 +2025-10-30T18:21:40.577Z [ERROR] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: failed to get log: index=22 error="log not found" +2025-10-30T18:21:40.577Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: aborting pipeline replication: peer="{Voter server-5e5493c5-455c-6fbc-2a42-8408924a19cb 5e5493c5-455c-6fbc-2a42-8408924a19cb}" +2025-10-30T18:21:40.577Z [ERROR] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: failed to get log: index=22 error="log not found" +2025-10-30T18:21:40.577Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: aborting pipeline replication: peer="{Voter server-0056941c-0598-0b8d-c77e-037b0b0d5539 0056941c-0598-0b8d-c77e-037b0b0d5539}" +2025-10-30T18:21:40.583Z [ERROR] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: failed to get log: index=22 error="log not found" +2025-10-30T18:21:40.583Z [ERROR] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: failed to get log: index=22 error="log not found" +2025-10-30T18:21:40.583Z [INFO] creating new snapshot: path=/tmp/raft769198165/snapshots/2-23-1761848500583.tmp +2025-10-30T18:21:40.583Z [INFO] creating new snapshot: path=/tmp/raft2694250089/snapshots/2-23-1761848500583.tmp +2025-10-30T18:21:40.583Z [INFO] server-5e5493c5-455c-6fbc-2a42-8408924a19cb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:40.584Z [INFO] server-0056941c-0598-0b8d-c77e-037b0b0d5539: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:40.765Z [INFO] server-0056941c-0598-0b8d-c77e-037b0b0d5539: copied to local snapshot: bytes=71 +2025-10-30T18:21:40.765Z [INFO] server-5e5493c5-455c-6fbc-2a42-8408924a19cb: copied to local snapshot: bytes=71 +2025-10-30T18:21:40.766Z [INFO] server-0056941c-0598-0b8d-c77e-037b0b0d5539: snapshot restore progress: id=2-23-1761848500583 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:40.766Z [INFO] server-0056941c-0598-0b8d-c77e-037b0b0d5539: removing all old logs from log store +2025-10-30T18:21:40.766Z [INFO] server-0056941c-0598-0b8d-c77e-037b0b0d5539: compacting logs: from=1 to=22 +2025-10-30T18:21:40.766Z [INFO] server-0056941c-0598-0b8d-c77e-037b0b0d5539: Installed remote snapshot +2025-10-30T18:21:40.766Z [INFO] server-5e5493c5-455c-6fbc-2a42-8408924a19cb: snapshot restore progress: id=2-23-1761848500583 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:40.766Z [INFO] server-5e5493c5-455c-6fbc-2a42-8408924a19cb: removing all old logs from log store +2025-10-30T18:21:40.766Z [INFO] server-5e5493c5-455c-6fbc-2a42-8408924a19cb: compacting logs: from=1 to=22 +2025-10-30T18:21:40.766Z [INFO] server-5e5493c5-455c-6fbc-2a42-8408924a19cb: Installed remote snapshot +2025-10-30T18:21:40.766Z [WARN] server-0056941c-0598-0b8d-c77e-037b0b0d5539: skipping application of old log: index=22 +2025-10-30T18:21:40.766Z [WARN] server-5e5493c5-455c-6fbc-2a42-8408924a19cb: skipping application of old log: index=23 +2025-10-30T18:21:40.766Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: pipelining replication: peer="{Voter server-0056941c-0598-0b8d-c77e-037b0b0d5539 0056941c-0598-0b8d-c77e-037b0b0d5539}" +2025-10-30T18:21:40.766Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:21:40.766Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: pipelining replication: peer="{Voter server-5e5493c5-455c-6fbc-2a42-8408924a19cb 5e5493c5-455c-6fbc-2a42-8408924a19cb}" +2025-10-30T18:21:41.772Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:21:40.766620261 +0000 UTC m=+23.840149554 (2 nodes), 1.067µs from start of poll, 1.991612245s from cluster start. Timeout at 2025-10-30 18:21:41.771626047 +0000 UTC m=+24.845155338, 1.005005784s after stability +2025-10-30T18:21:41.784Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: aborting pipeline replication: peer="{Voter server-5e5493c5-455c-6fbc-2a42-8408924a19cb 5e5493c5-455c-6fbc-2a42-8408924a19cb}" +2025-10-30T18:21:41.784Z [INFO] server-528a34e1-5c6c-9d7a-a23d-7a82caa5ac08: aborting pipeline replication: peer="{Voter server-0056941c-0598-0b8d-c77e-037b0b0d5539 0056941c-0598-0b8d-c77e-037b0b0d5539}" === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true -2025-10-30T18:17:52.134Z [INFO] server-85255c93-7cbb-829a-97b1-be16cf8dc383: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85255c93-7cbb-829a-97b1-be16cf8dc383 Address:85255c93-7cbb-829a-97b1-be16cf8dc383} {Suffrage:Voter ID:server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67 Address:c95bd67b-03c8-e2f4-643f-4a8dedcf5f67} {Suffrage:Voter ID:server-94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869 Address:94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869}]" -2025-10-30T18:17:52.134Z [INFO] server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85255c93-7cbb-829a-97b1-be16cf8dc383 Address:85255c93-7cbb-829a-97b1-be16cf8dc383} {Suffrage:Voter ID:server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67 Address:c95bd67b-03c8-e2f4-643f-4a8dedcf5f67} {Suffrage:Voter ID:server-94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869 Address:94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869}]" -2025-10-30T18:17:52.134Z [INFO] server-85255c93-7cbb-829a-97b1-be16cf8dc383: entering follower state: follower="Node at 85255c93-7cbb-829a-97b1-be16cf8dc383 [Follower]" leader-address= leader-id= -2025-10-30T18:17:52.134Z [INFO] server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: entering follower state: follower="Node at c95bd67b-03c8-e2f4-643f-4a8dedcf5f67 [Follower]" leader-address= leader-id= -2025-10-30T18:17:52.134Z [INFO] server-94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85255c93-7cbb-829a-97b1-be16cf8dc383 Address:85255c93-7cbb-829a-97b1-be16cf8dc383} {Suffrage:Voter ID:server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67 Address:c95bd67b-03c8-e2f4-643f-4a8dedcf5f67} {Suffrage:Voter ID:server-94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869 Address:94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869}]" -2025-10-30T18:17:52.134Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:52.134Z [INFO] server-94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869: entering follower state: follower="Node at 94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869 [Follower]" leader-address= leader-id= -2025-10-30T18:17:52.679Z [WARN] server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:52.679Z [INFO] server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: entering candidate state: node="Node at c95bd67b-03c8-e2f4-643f-4a8dedcf5f67 [Candidate]" term=2 -2025-10-30T18:17:52.679Z [INFO] server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: election won: term=2 tally=2 -2025-10-30T18:17:52.679Z [INFO] server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: entering leader state: leader="Node at c95bd67b-03c8-e2f4-643f-4a8dedcf5f67 [Leader]" -2025-10-30T18:17:52.679Z [INFO] server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: added peer, starting replication: peer=server-85255c93-7cbb-829a-97b1-be16cf8dc383 -2025-10-30T18:17:52.679Z [INFO] server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: added peer, starting replication: peer=server-94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869 -2025-10-30T18:17:52.679Z [INFO] server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: pipelining replication: peer="{Voter server-94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869 94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869}" -2025-10-30T18:17:52.679Z [INFO] server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: pipelining replication: peer="{Voter server-85255c93-7cbb-829a-97b1-be16cf8dc383 85255c93-7cbb-829a-97b1-be16cf8dc383}" -2025-10-30T18:17:53.685Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:52.679742649 +0000 UTC m=+18.958416566 (1 nodes), 545.421524ms from start of poll, 545.686385ms from cluster start. Timeout at 2025-10-30 18:17:53.684743611 +0000 UTC m=+19.963417508, 1.005000942s after stability -2025-10-30T18:17:53.686Z [INFO] server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: starting snapshot up to: index=12 -2025-10-30T18:17:53.686Z [INFO] creating new snapshot: path=/tmp/raft383714054/snapshots/2-12-1761848273686.tmp -2025-10-30T18:17:53.687Z [INFO] server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: snapshot complete up to: index=12 -2025-10-30T18:17:53.687Z [INFO] server-f99e0af7-b17b-34c6-5053-ee1c133a3e03: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f99e0af7-b17b-34c6-5053-ee1c133a3e03 Address:f99e0af7-b17b-34c6-5053-ee1c133a3e03} {Suffrage:Voter ID:server-cde94500-b21c-20c9-98e1-e9f97439f399 Address:cde94500-b21c-20c9-98e1-e9f97439f399} {Suffrage:Voter ID:server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca Address:2e6de459-da77-8a6d-61ca-f7ccc4b33eca}]" -2025-10-30T18:17:53.687Z [INFO] server-f99e0af7-b17b-34c6-5053-ee1c133a3e03: entering follower state: follower="Node at f99e0af7-b17b-34c6-5053-ee1c133a3e03 [Follower]" leader-address= leader-id= -2025-10-30T18:17:53.687Z [INFO] server-cde94500-b21c-20c9-98e1-e9f97439f399: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f99e0af7-b17b-34c6-5053-ee1c133a3e03 Address:f99e0af7-b17b-34c6-5053-ee1c133a3e03} {Suffrage:Voter ID:server-cde94500-b21c-20c9-98e1-e9f97439f399 Address:cde94500-b21c-20c9-98e1-e9f97439f399} {Suffrage:Voter ID:server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca Address:2e6de459-da77-8a6d-61ca-f7ccc4b33eca}]" -2025-10-30T18:17:53.688Z [INFO] server-cde94500-b21c-20c9-98e1-e9f97439f399: entering follower state: follower="Node at cde94500-b21c-20c9-98e1-e9f97439f399 [Follower]" leader-address= leader-id= -2025-10-30T18:17:53.688Z [INFO] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f99e0af7-b17b-34c6-5053-ee1c133a3e03 Address:f99e0af7-b17b-34c6-5053-ee1c133a3e03} {Suffrage:Voter ID:server-cde94500-b21c-20c9-98e1-e9f97439f399 Address:cde94500-b21c-20c9-98e1-e9f97439f399} {Suffrage:Voter ID:server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca Address:2e6de459-da77-8a6d-61ca-f7ccc4b33eca}]" -2025-10-30T18:17:53.688Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:53.688Z [INFO] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: entering follower state: follower="Node at 2e6de459-da77-8a6d-61ca-f7ccc4b33eca [Follower]" leader-address= leader-id= -2025-10-30T18:17:54.256Z [WARN] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:54.256Z [INFO] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: entering candidate state: node="Node at 2e6de459-da77-8a6d-61ca-f7ccc4b33eca [Candidate]" term=2 -2025-10-30T18:17:54.256Z [INFO] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: election won: term=2 tally=2 -2025-10-30T18:17:54.256Z [INFO] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: entering leader state: leader="Node at 2e6de459-da77-8a6d-61ca-f7ccc4b33eca [Leader]" -2025-10-30T18:17:54.256Z [INFO] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: added peer, starting replication: peer=server-f99e0af7-b17b-34c6-5053-ee1c133a3e03 -2025-10-30T18:17:54.256Z [INFO] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: added peer, starting replication: peer=server-cde94500-b21c-20c9-98e1-e9f97439f399 -2025-10-30T18:17:54.257Z [INFO] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: pipelining replication: peer="{Voter server-cde94500-b21c-20c9-98e1-e9f97439f399 cde94500-b21c-20c9-98e1-e9f97439f399}" -2025-10-30T18:17:54.257Z [INFO] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: pipelining replication: peer="{Voter server-f99e0af7-b17b-34c6-5053-ee1c133a3e03 f99e0af7-b17b-34c6-5053-ee1c133a3e03}" -2025-10-30T18:17:55.263Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:54.257473978 +0000 UTC m=+20.536147876 (1 nodes), 569.277941ms from start of poll, 569.712348ms from cluster start. Timeout at 2025-10-30 18:17:55.262476031 +0000 UTC m=+21.541149928, 1.005002052s after stability -2025-10-30T18:17:55.263Z [INFO] creating new snapshot: path=/tmp/raft2247454591/snapshots/2-13-1761848275263.tmp -2025-10-30T18:17:55.366Z [INFO] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: copied to local snapshot: bytes=71 -2025-10-30T18:17:55.366Z [INFO] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: snapshot restore progress: id=2-13-1761848275263 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:17:55.366Z [INFO] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: restored user snapshot: index=13 -2025-10-30T18:17:55.366Z [ERROR] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: failed to get log: index=3 error="log not found" -2025-10-30T18:17:55.366Z [ERROR] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: failed to get log: index=3 error="log not found" -2025-10-30T18:17:55.366Z [INFO] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: aborting pipeline replication: peer="{Voter server-cde94500-b21c-20c9-98e1-e9f97439f399 cde94500-b21c-20c9-98e1-e9f97439f399}" -2025-10-30T18:17:55.366Z [INFO] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: aborting pipeline replication: peer="{Voter server-f99e0af7-b17b-34c6-5053-ee1c133a3e03 f99e0af7-b17b-34c6-5053-ee1c133a3e03}" -2025-10-30T18:17:55.373Z [ERROR] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: failed to get log: index=3 error="log not found" -2025-10-30T18:17:55.374Z [INFO] creating new snapshot: path=/tmp/raft2761616345/snapshots/2-13-1761848275374.tmp -2025-10-30T18:17:55.374Z [INFO] server-f99e0af7-b17b-34c6-5053-ee1c133a3e03: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:17:55.375Z [ERROR] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: failed to get log: index=3 error="log not found" -2025-10-30T18:17:55.375Z [INFO] creating new snapshot: path=/tmp/raft4078536202/snapshots/2-13-1761848275375.tmp -2025-10-30T18:17:55.375Z [INFO] server-cde94500-b21c-20c9-98e1-e9f97439f399: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:17:55.523Z [INFO] server-f99e0af7-b17b-34c6-5053-ee1c133a3e03: copied to local snapshot: bytes=71 -2025-10-30T18:17:55.523Z [INFO] server-cde94500-b21c-20c9-98e1-e9f97439f399: copied to local snapshot: bytes=71 -2025-10-30T18:17:55.523Z [INFO] server-f99e0af7-b17b-34c6-5053-ee1c133a3e03: snapshot restore progress: id=2-13-1761848275374 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:17:55.523Z [INFO] server-f99e0af7-b17b-34c6-5053-ee1c133a3e03: Installed remote snapshot -2025-10-30T18:17:55.523Z [INFO] server-cde94500-b21c-20c9-98e1-e9f97439f399: snapshot restore progress: id=2-13-1761848275375 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:17:55.523Z [INFO] server-cde94500-b21c-20c9-98e1-e9f97439f399: Installed remote snapshot -2025-10-30T18:17:55.523Z [INFO] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: pipelining replication: peer="{Voter server-f99e0af7-b17b-34c6-5053-ee1c133a3e03 f99e0af7-b17b-34c6-5053-ee1c133a3e03}" -2025-10-30T18:17:55.523Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:17:55.523Z [INFO] server-2e6de459-da77-8a6d-61ca-f7ccc4b33eca: pipelining replication: peer="{Voter server-cde94500-b21c-20c9-98e1-e9f97439f399 cde94500-b21c-20c9-98e1-e9f97439f399}" -2025-10-30T18:17:56.529Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:17:55.523675657 +0000 UTC m=+21.802349567 (2 nodes), 2.126µs from start of poll, 1.835914039s from cluster start. Timeout at 2025-10-30 18:17:56.528679292 +0000 UTC m=+22.807353189, 1.005003622s after stability -2025-10-30T18:17:56.545Z [INFO] server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: aborting pipeline replication: peer="{Voter server-94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869 94a6e5ed-f7f3-1ddc-1cc8-4f4184a64869}" -2025-10-30T18:17:56.546Z [INFO] server-c95bd67b-03c8-e2f4-643f-4a8dedcf5f67: aborting pipeline replication: peer="{Voter server-85255c93-7cbb-829a-97b1-be16cf8dc383 85255c93-7cbb-829a-97b1-be16cf8dc383}" +2025-10-30T18:21:41.785Z [INFO] server-be76ed67-3cbe-cabf-0f1b-224e6e8d4886: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be76ed67-3cbe-cabf-0f1b-224e6e8d4886 Address:be76ed67-3cbe-cabf-0f1b-224e6e8d4886} {Suffrage:Voter ID:server-99ae1d77-3e35-f243-7a28-4166a2b0283e Address:99ae1d77-3e35-f243-7a28-4166a2b0283e} {Suffrage:Voter ID:server-b531d734-6d1a-6180-24db-0b62d90af508 Address:b531d734-6d1a-6180-24db-0b62d90af508}]" +2025-10-30T18:21:41.785Z [INFO] server-be76ed67-3cbe-cabf-0f1b-224e6e8d4886: entering follower state: follower="Node at be76ed67-3cbe-cabf-0f1b-224e6e8d4886 [Follower]" leader-address= leader-id= +2025-10-30T18:21:41.785Z [INFO] server-99ae1d77-3e35-f243-7a28-4166a2b0283e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be76ed67-3cbe-cabf-0f1b-224e6e8d4886 Address:be76ed67-3cbe-cabf-0f1b-224e6e8d4886} {Suffrage:Voter ID:server-99ae1d77-3e35-f243-7a28-4166a2b0283e Address:99ae1d77-3e35-f243-7a28-4166a2b0283e} {Suffrage:Voter ID:server-b531d734-6d1a-6180-24db-0b62d90af508 Address:b531d734-6d1a-6180-24db-0b62d90af508}]" +2025-10-30T18:21:41.785Z [INFO] server-99ae1d77-3e35-f243-7a28-4166a2b0283e: entering follower state: follower="Node at 99ae1d77-3e35-f243-7a28-4166a2b0283e [Follower]" leader-address= leader-id= +2025-10-30T18:21:41.785Z [INFO] server-b531d734-6d1a-6180-24db-0b62d90af508: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be76ed67-3cbe-cabf-0f1b-224e6e8d4886 Address:be76ed67-3cbe-cabf-0f1b-224e6e8d4886} {Suffrage:Voter ID:server-99ae1d77-3e35-f243-7a28-4166a2b0283e Address:99ae1d77-3e35-f243-7a28-4166a2b0283e} {Suffrage:Voter ID:server-b531d734-6d1a-6180-24db-0b62d90af508 Address:b531d734-6d1a-6180-24db-0b62d90af508}]" +2025-10-30T18:21:41.785Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:41.785Z [INFO] server-b531d734-6d1a-6180-24db-0b62d90af508: entering follower state: follower="Node at b531d734-6d1a-6180-24db-0b62d90af508 [Follower]" leader-address= leader-id= +2025-10-30T18:21:42.446Z [WARN] server-99ae1d77-3e35-f243-7a28-4166a2b0283e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:42.446Z [INFO] server-99ae1d77-3e35-f243-7a28-4166a2b0283e: entering candidate state: node="Node at 99ae1d77-3e35-f243-7a28-4166a2b0283e [Candidate]" term=2 +2025-10-30T18:21:42.446Z [INFO] server-99ae1d77-3e35-f243-7a28-4166a2b0283e: election won: term=2 tally=2 +2025-10-30T18:21:42.446Z [INFO] server-99ae1d77-3e35-f243-7a28-4166a2b0283e: entering leader state: leader="Node at 99ae1d77-3e35-f243-7a28-4166a2b0283e [Leader]" +2025-10-30T18:21:42.446Z [INFO] server-99ae1d77-3e35-f243-7a28-4166a2b0283e: added peer, starting replication: peer=server-be76ed67-3cbe-cabf-0f1b-224e6e8d4886 +2025-10-30T18:21:42.446Z [INFO] server-99ae1d77-3e35-f243-7a28-4166a2b0283e: added peer, starting replication: peer=server-b531d734-6d1a-6180-24db-0b62d90af508 +2025-10-30T18:21:42.446Z [INFO] server-99ae1d77-3e35-f243-7a28-4166a2b0283e: pipelining replication: peer="{Voter server-b531d734-6d1a-6180-24db-0b62d90af508 b531d734-6d1a-6180-24db-0b62d90af508}" +2025-10-30T18:21:42.446Z [INFO] server-99ae1d77-3e35-f243-7a28-4166a2b0283e: pipelining replication: peer="{Voter server-be76ed67-3cbe-cabf-0f1b-224e6e8d4886 be76ed67-3cbe-cabf-0f1b-224e6e8d4886}" +2025-10-30T18:21:43.451Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:42.446546663 +0000 UTC m=+25.520075989 (1 nodes), 660.83851ms from start of poll, 661.1212ms from cluster start. Timeout at 2025-10-30 18:21:43.451547567 +0000 UTC m=+26.525076863, 1.005000874s after stability +2025-10-30T18:21:43.452Z [INFO] server-99ae1d77-3e35-f243-7a28-4166a2b0283e: starting snapshot up to: index=12 +2025-10-30T18:21:43.452Z [INFO] creating new snapshot: path=/tmp/raft1932337805/snapshots/2-12-1761848503452.tmp +2025-10-30T18:21:43.453Z [INFO] server-99ae1d77-3e35-f243-7a28-4166a2b0283e: snapshot complete up to: index=12 +2025-10-30T18:21:43.453Z [INFO] server-72bdba35-233c-a205-b189-c6faa07c7d5b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72bdba35-233c-a205-b189-c6faa07c7d5b Address:72bdba35-233c-a205-b189-c6faa07c7d5b} {Suffrage:Voter ID:server-94cb357e-bf8e-3d97-650e-e6c1381e15e9 Address:94cb357e-bf8e-3d97-650e-e6c1381e15e9} {Suffrage:Voter ID:server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb Address:7ee4c18a-a936-15e4-7ff6-4d04a9031adb}]" +2025-10-30T18:21:43.454Z [INFO] server-72bdba35-233c-a205-b189-c6faa07c7d5b: entering follower state: follower="Node at 72bdba35-233c-a205-b189-c6faa07c7d5b [Follower]" leader-address= leader-id= +2025-10-30T18:21:43.454Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72bdba35-233c-a205-b189-c6faa07c7d5b Address:72bdba35-233c-a205-b189-c6faa07c7d5b} {Suffrage:Voter ID:server-94cb357e-bf8e-3d97-650e-e6c1381e15e9 Address:94cb357e-bf8e-3d97-650e-e6c1381e15e9} {Suffrage:Voter ID:server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb Address:7ee4c18a-a936-15e4-7ff6-4d04a9031adb}]" +2025-10-30T18:21:43.454Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: entering follower state: follower="Node at 94cb357e-bf8e-3d97-650e-e6c1381e15e9 [Follower]" leader-address= leader-id= +2025-10-30T18:21:43.454Z [INFO] server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-72bdba35-233c-a205-b189-c6faa07c7d5b Address:72bdba35-233c-a205-b189-c6faa07c7d5b} {Suffrage:Voter ID:server-94cb357e-bf8e-3d97-650e-e6c1381e15e9 Address:94cb357e-bf8e-3d97-650e-e6c1381e15e9} {Suffrage:Voter ID:server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb Address:7ee4c18a-a936-15e4-7ff6-4d04a9031adb}]" +2025-10-30T18:21:43.454Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:43.454Z [INFO] server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb: entering follower state: follower="Node at 7ee4c18a-a936-15e4-7ff6-4d04a9031adb [Follower]" leader-address= leader-id= +2025-10-30T18:21:44.147Z [WARN] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:44.147Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: entering candidate state: node="Node at 94cb357e-bf8e-3d97-650e-e6c1381e15e9 [Candidate]" term=2 +2025-10-30T18:21:44.147Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: election won: term=2 tally=2 +2025-10-30T18:21:44.147Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: entering leader state: leader="Node at 94cb357e-bf8e-3d97-650e-e6c1381e15e9 [Leader]" +2025-10-30T18:21:44.147Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: added peer, starting replication: peer=server-72bdba35-233c-a205-b189-c6faa07c7d5b +2025-10-30T18:21:44.147Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: added peer, starting replication: peer=server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb +2025-10-30T18:21:44.147Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: pipelining replication: peer="{Voter server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb 7ee4c18a-a936-15e4-7ff6-4d04a9031adb}" +2025-10-30T18:21:44.147Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: pipelining replication: peer="{Voter server-72bdba35-233c-a205-b189-c6faa07c7d5b 72bdba35-233c-a205-b189-c6faa07c7d5b}" +2025-10-30T18:21:45.152Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:44.14730744 +0000 UTC m=+27.220836728 (1 nodes), 693.084961ms from start of poll, 693.41375ms from cluster start. Timeout at 2025-10-30 18:21:45.152313113 +0000 UTC m=+28.225842439, 1.005005711s after stability +2025-10-30T18:21:45.152Z [INFO] creating new snapshot: path=/tmp/raft1311299548/snapshots/2-13-1761848505152.tmp +2025-10-30T18:21:45.239Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: copied to local snapshot: bytes=71 +2025-10-30T18:21:45.239Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: snapshot restore progress: id=2-13-1761848505152 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:45.239Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: restored user snapshot: index=13 +2025-10-30T18:21:45.239Z [ERROR] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: failed to get log: index=3 error="log not found" +2025-10-30T18:21:45.239Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: aborting pipeline replication: peer="{Voter server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb 7ee4c18a-a936-15e4-7ff6-4d04a9031adb}" +2025-10-30T18:21:45.239Z [ERROR] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: failed to get log: index=3 error="log not found" +2025-10-30T18:21:45.239Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: aborting pipeline replication: peer="{Voter server-72bdba35-233c-a205-b189-c6faa07c7d5b 72bdba35-233c-a205-b189-c6faa07c7d5b}" +2025-10-30T18:21:45.248Z [ERROR] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: failed to get log: index=3 error="log not found" +2025-10-30T18:21:45.248Z [INFO] creating new snapshot: path=/tmp/raft3666514503/snapshots/2-13-1761848505248.tmp +2025-10-30T18:21:45.248Z [ERROR] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: failed to get log: index=3 error="log not found" +2025-10-30T18:21:45.248Z [INFO] server-72bdba35-233c-a205-b189-c6faa07c7d5b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:45.249Z [INFO] creating new snapshot: path=/tmp/raft799880107/snapshots/2-13-1761848505249.tmp +2025-10-30T18:21:45.249Z [INFO] server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:45.404Z [INFO] server-72bdba35-233c-a205-b189-c6faa07c7d5b: copied to local snapshot: bytes=71 +2025-10-30T18:21:45.404Z [INFO] server-72bdba35-233c-a205-b189-c6faa07c7d5b: snapshot restore progress: id=2-13-1761848505248 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:45.404Z [INFO] server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb: copied to local snapshot: bytes=71 +2025-10-30T18:21:45.404Z [INFO] server-72bdba35-233c-a205-b189-c6faa07c7d5b: Installed remote snapshot +2025-10-30T18:21:45.404Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: pipelining replication: peer="{Voter server-72bdba35-233c-a205-b189-c6faa07c7d5b 72bdba35-233c-a205-b189-c6faa07c7d5b}" +2025-10-30T18:21:45.404Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:21:45.404Z [INFO] server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb: snapshot restore progress: id=2-13-1761848505249 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:45.404Z [INFO] server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb: Installed remote snapshot +2025-10-30T18:21:45.404Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: pipelining replication: peer="{Voter server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb 7ee4c18a-a936-15e4-7ff6-4d04a9031adb}" +2025-10-30T18:21:46.410Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:21:45.404417374 +0000 UTC m=+28.477946663 (2 nodes), 499ns from start of poll, 1.950523685s from cluster start. Timeout at 2025-10-30 18:21:46.409420294 +0000 UTC m=+29.482949587, 1.005002924s after stability +2025-10-30T18:21:46.422Z [INFO] server-99ae1d77-3e35-f243-7a28-4166a2b0283e: aborting pipeline replication: peer="{Voter server-b531d734-6d1a-6180-24db-0b62d90af508 b531d734-6d1a-6180-24db-0b62d90af508}" +2025-10-30T18:21:46.422Z [INFO] server-99ae1d77-3e35-f243-7a28-4166a2b0283e: aborting pipeline replication: peer="{Voter server-be76ed67-3cbe-cabf-0f1b-224e6e8d4886 be76ed67-3cbe-cabf-0f1b-224e6e8d4886}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true -2025-10-30T18:17:56.546Z [INFO] server-5fb0a50c-2338-4c38-9356-035502ac6b85: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fb0a50c-2338-4c38-9356-035502ac6b85 Address:5fb0a50c-2338-4c38-9356-035502ac6b85} {Suffrage:Voter ID:server-5ac2565b-6f42-222c-779b-cb1b4ead74b2 Address:5ac2565b-6f42-222c-779b-cb1b4ead74b2} {Suffrage:Voter ID:server-6d8545df-d9b9-a2ec-9ef7-5e538b8e6850 Address:6d8545df-d9b9-a2ec-9ef7-5e538b8e6850}]" -2025-10-30T18:17:56.546Z [INFO] server-5ac2565b-6f42-222c-779b-cb1b4ead74b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fb0a50c-2338-4c38-9356-035502ac6b85 Address:5fb0a50c-2338-4c38-9356-035502ac6b85} {Suffrage:Voter ID:server-5ac2565b-6f42-222c-779b-cb1b4ead74b2 Address:5ac2565b-6f42-222c-779b-cb1b4ead74b2} {Suffrage:Voter ID:server-6d8545df-d9b9-a2ec-9ef7-5e538b8e6850 Address:6d8545df-d9b9-a2ec-9ef7-5e538b8e6850}]" -2025-10-30T18:17:56.546Z [INFO] server-5fb0a50c-2338-4c38-9356-035502ac6b85: entering follower state: follower="Node at 5fb0a50c-2338-4c38-9356-035502ac6b85 [Follower]" leader-address= leader-id= -2025-10-30T18:17:56.546Z [INFO] server-5ac2565b-6f42-222c-779b-cb1b4ead74b2: entering follower state: follower="Node at 5ac2565b-6f42-222c-779b-cb1b4ead74b2 [Follower]" leader-address= leader-id= -2025-10-30T18:17:56.547Z [INFO] server-6d8545df-d9b9-a2ec-9ef7-5e538b8e6850: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fb0a50c-2338-4c38-9356-035502ac6b85 Address:5fb0a50c-2338-4c38-9356-035502ac6b85} {Suffrage:Voter ID:server-5ac2565b-6f42-222c-779b-cb1b4ead74b2 Address:5ac2565b-6f42-222c-779b-cb1b4ead74b2} {Suffrage:Voter ID:server-6d8545df-d9b9-a2ec-9ef7-5e538b8e6850 Address:6d8545df-d9b9-a2ec-9ef7-5e538b8e6850}]" -2025-10-30T18:17:56.547Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:56.547Z [INFO] server-6d8545df-d9b9-a2ec-9ef7-5e538b8e6850: entering follower state: follower="Node at 6d8545df-d9b9-a2ec-9ef7-5e538b8e6850 [Follower]" leader-address= leader-id= -2025-10-30T18:17:57.072Z [WARN] server-5fb0a50c-2338-4c38-9356-035502ac6b85: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:57.072Z [INFO] server-5fb0a50c-2338-4c38-9356-035502ac6b85: entering candidate state: node="Node at 5fb0a50c-2338-4c38-9356-035502ac6b85 [Candidate]" term=2 -2025-10-30T18:17:57.073Z [INFO] server-5fb0a50c-2338-4c38-9356-035502ac6b85: election won: term=2 tally=2 -2025-10-30T18:17:57.073Z [INFO] server-5fb0a50c-2338-4c38-9356-035502ac6b85: entering leader state: leader="Node at 5fb0a50c-2338-4c38-9356-035502ac6b85 [Leader]" -2025-10-30T18:17:57.073Z [INFO] server-5fb0a50c-2338-4c38-9356-035502ac6b85: added peer, starting replication: peer=server-5ac2565b-6f42-222c-779b-cb1b4ead74b2 -2025-10-30T18:17:57.073Z [INFO] server-5fb0a50c-2338-4c38-9356-035502ac6b85: added peer, starting replication: peer=server-6d8545df-d9b9-a2ec-9ef7-5e538b8e6850 -2025-10-30T18:17:57.073Z [INFO] server-5fb0a50c-2338-4c38-9356-035502ac6b85: pipelining replication: peer="{Voter server-6d8545df-d9b9-a2ec-9ef7-5e538b8e6850 6d8545df-d9b9-a2ec-9ef7-5e538b8e6850}" -2025-10-30T18:17:57.073Z [INFO] server-5fb0a50c-2338-4c38-9356-035502ac6b85: pipelining replication: peer="{Voter server-5ac2565b-6f42-222c-779b-cb1b4ead74b2 5ac2565b-6f42-222c-779b-cb1b4ead74b2}" -2025-10-30T18:17:58.078Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:57.073097908 +0000 UTC m=+23.351771805 (1 nodes), 525.983431ms from start of poll, 526.298966ms from cluster start. Timeout at 2025-10-30 18:17:58.078102332 +0000 UTC m=+24.356776229, 1.005004424s after stability -2025-10-30T18:17:58.079Z [INFO] server-5fb0a50c-2338-4c38-9356-035502ac6b85: starting snapshot up to: index=12 -2025-10-30T18:17:58.079Z [INFO] creating new snapshot: path=/tmp/raft2888405852/snapshots/2-12-1761848278079.tmp -2025-10-30T18:17:58.259Z [INFO] server-5fb0a50c-2338-4c38-9356-035502ac6b85: snapshot complete up to: index=12 -2025-10-30T18:17:58.259Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83251805-6ece-3215-0036-8804211416f2 Address:83251805-6ece-3215-0036-8804211416f2} {Suffrage:Voter ID:server-7c958cae-37cf-4466-86a9-3d991e3b0a49 Address:7c958cae-37cf-4466-86a9-3d991e3b0a49} {Suffrage:Voter ID:server-6277ade7-660e-73bf-b161-d75bf3a57370 Address:6277ade7-660e-73bf-b161-d75bf3a57370}]" -2025-10-30T18:17:58.259Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: entering follower state: follower="Node at 83251805-6ece-3215-0036-8804211416f2 [Follower]" leader-address= leader-id= -2025-10-30T18:17:58.259Z [INFO] server-7c958cae-37cf-4466-86a9-3d991e3b0a49: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83251805-6ece-3215-0036-8804211416f2 Address:83251805-6ece-3215-0036-8804211416f2} {Suffrage:Voter ID:server-7c958cae-37cf-4466-86a9-3d991e3b0a49 Address:7c958cae-37cf-4466-86a9-3d991e3b0a49} {Suffrage:Voter ID:server-6277ade7-660e-73bf-b161-d75bf3a57370 Address:6277ade7-660e-73bf-b161-d75bf3a57370}]" -2025-10-30T18:17:58.260Z [INFO] server-6277ade7-660e-73bf-b161-d75bf3a57370: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83251805-6ece-3215-0036-8804211416f2 Address:83251805-6ece-3215-0036-8804211416f2} {Suffrage:Voter ID:server-7c958cae-37cf-4466-86a9-3d991e3b0a49 Address:7c958cae-37cf-4466-86a9-3d991e3b0a49} {Suffrage:Voter ID:server-6277ade7-660e-73bf-b161-d75bf3a57370 Address:6277ade7-660e-73bf-b161-d75bf3a57370}]" -2025-10-30T18:17:58.260Z [INFO] server-7c958cae-37cf-4466-86a9-3d991e3b0a49: entering follower state: follower="Node at 7c958cae-37cf-4466-86a9-3d991e3b0a49 [Follower]" leader-address= leader-id= -2025-10-30T18:17:58.260Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:17:58.260Z [INFO] server-6277ade7-660e-73bf-b161-d75bf3a57370: entering follower state: follower="Node at 6277ade7-660e-73bf-b161-d75bf3a57370 [Follower]" leader-address= leader-id= -2025-10-30T18:17:58.861Z [WARN] server-83251805-6ece-3215-0036-8804211416f2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:17:58.861Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: entering candidate state: node="Node at 83251805-6ece-3215-0036-8804211416f2 [Candidate]" term=2 -2025-10-30T18:17:58.861Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: election won: term=2 tally=2 -2025-10-30T18:17:58.861Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: entering leader state: leader="Node at 83251805-6ece-3215-0036-8804211416f2 [Leader]" -2025-10-30T18:17:58.861Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: added peer, starting replication: peer=server-7c958cae-37cf-4466-86a9-3d991e3b0a49 -2025-10-30T18:17:58.861Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: added peer, starting replication: peer=server-6277ade7-660e-73bf-b161-d75bf3a57370 -2025-10-30T18:17:58.861Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: pipelining replication: peer="{Voter server-6277ade7-660e-73bf-b161-d75bf3a57370 6277ade7-660e-73bf-b161-d75bf3a57370}" -2025-10-30T18:17:58.861Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: pipelining replication: peer="{Voter server-7c958cae-37cf-4466-86a9-3d991e3b0a49 7c958cae-37cf-4466-86a9-3d991e3b0a49}" -2025-10-30T18:17:59.867Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:17:58.861302354 +0000 UTC m=+25.139976267 (1 nodes), 601.226586ms from start of poll, 601.504743ms from cluster start. Timeout at 2025-10-30 18:17:59.866305725 +0000 UTC m=+26.144979623, 1.005003356s after stability -2025-10-30T18:17:59.998Z [INFO] creating new snapshot: path=/tmp/raft3159913288/snapshots/2-13-1761848279998.tmp -2025-10-30T18:18:00.055Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: copied to local snapshot: bytes=71 -2025-10-30T18:18:00.056Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: snapshot restore progress: id=2-13-1761848279998 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:00.056Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: removing all old logs from log store -2025-10-30T18:18:00.056Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: compacting logs: from=1 to=2 -2025-10-30T18:18:00.056Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: restored user snapshot: index=13 -2025-10-30T18:18:00.056Z [ERROR] server-83251805-6ece-3215-0036-8804211416f2: failed to get log: index=2 error="log not found" -2025-10-30T18:18:00.056Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: aborting pipeline replication: peer="{Voter server-7c958cae-37cf-4466-86a9-3d991e3b0a49 7c958cae-37cf-4466-86a9-3d991e3b0a49}" -2025-10-30T18:18:00.056Z [ERROR] server-83251805-6ece-3215-0036-8804211416f2: failed to get log: index=2 error="log not found" -2025-10-30T18:18:00.056Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: aborting pipeline replication: peer="{Voter server-6277ade7-660e-73bf-b161-d75bf3a57370 6277ade7-660e-73bf-b161-d75bf3a57370}" -2025-10-30T18:18:00.063Z [ERROR] server-83251805-6ece-3215-0036-8804211416f2: failed to get log: index=2 error="log not found" -2025-10-30T18:18:00.063Z [INFO] creating new snapshot: path=/tmp/raft3833466995/snapshots/2-13-1761848280063.tmp -2025-10-30T18:18:00.063Z [INFO] server-7c958cae-37cf-4466-86a9-3d991e3b0a49: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:00.065Z [ERROR] server-83251805-6ece-3215-0036-8804211416f2: failed to get log: index=2 error="log not found" -2025-10-30T18:18:00.065Z [INFO] creating new snapshot: path=/tmp/raft366783963/snapshots/2-13-1761848280065.tmp -2025-10-30T18:18:00.065Z [INFO] server-6277ade7-660e-73bf-b161-d75bf3a57370: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:00.151Z [INFO] server-6277ade7-660e-73bf-b161-d75bf3a57370: copied to local snapshot: bytes=71 -2025-10-30T18:18:00.151Z [INFO] server-7c958cae-37cf-4466-86a9-3d991e3b0a49: copied to local snapshot: bytes=71 -2025-10-30T18:18:00.152Z [INFO] server-7c958cae-37cf-4466-86a9-3d991e3b0a49: snapshot restore progress: id=2-13-1761848280063 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:00.152Z [INFO] server-6277ade7-660e-73bf-b161-d75bf3a57370: snapshot restore progress: id=2-13-1761848280065 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:00.152Z [INFO] server-7c958cae-37cf-4466-86a9-3d991e3b0a49: removing all old logs from log store -2025-10-30T18:18:00.152Z [INFO] server-6277ade7-660e-73bf-b161-d75bf3a57370: removing all old logs from log store -2025-10-30T18:18:00.152Z [INFO] server-7c958cae-37cf-4466-86a9-3d991e3b0a49: compacting logs: from=1 to=2 -2025-10-30T18:18:00.152Z [INFO] server-7c958cae-37cf-4466-86a9-3d991e3b0a49: Installed remote snapshot -2025-10-30T18:18:00.152Z [INFO] server-6277ade7-660e-73bf-b161-d75bf3a57370: compacting logs: from=1 to=2 -2025-10-30T18:18:00.152Z [INFO] server-6277ade7-660e-73bf-b161-d75bf3a57370: Installed remote snapshot -2025-10-30T18:18:00.152Z [WARN] server-7c958cae-37cf-4466-86a9-3d991e3b0a49: skipping application of old log: index=13 -2025-10-30T18:18:00.152Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: pipelining replication: peer="{Voter server-6277ade7-660e-73bf-b161-d75bf3a57370 6277ade7-660e-73bf-b161-d75bf3a57370}" -2025-10-30T18:18:00.152Z [INFO] server-83251805-6ece-3215-0036-8804211416f2: pipelining replication: peer="{Voter server-7c958cae-37cf-4466-86a9-3d991e3b0a49 7c958cae-37cf-4466-86a9-3d991e3b0a49}" -2025-10-30T18:18:00.152Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:18:01.157Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:18:00.152301623 +0000 UTC m=+26.430975521 (2 nodes), 492ns from start of poll, 1.892503997s from cluster start. Timeout at 2025-10-30 18:18:01.157314803 +0000 UTC m=+27.435988710, 1.005013189s after stability -2025-10-30T18:18:01.168Z [INFO] server-5fb0a50c-2338-4c38-9356-035502ac6b85: aborting pipeline replication: peer="{Voter server-6d8545df-d9b9-a2ec-9ef7-5e538b8e6850 6d8545df-d9b9-a2ec-9ef7-5e538b8e6850}" -2025-10-30T18:18:01.168Z [INFO] server-5fb0a50c-2338-4c38-9356-035502ac6b85: aborting pipeline replication: peer="{Voter server-5ac2565b-6f42-222c-779b-cb1b4ead74b2 5ac2565b-6f42-222c-779b-cb1b4ead74b2}" +2025-10-30T18:21:46.423Z [INFO] server-7fd8fe99-a380-9c3a-c7b1-00084ebeef4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fd8fe99-a380-9c3a-c7b1-00084ebeef4b Address:7fd8fe99-a380-9c3a-c7b1-00084ebeef4b} {Suffrage:Voter ID:server-ae1e87fa-09d7-63cc-a083-e33711dd8747 Address:ae1e87fa-09d7-63cc-a083-e33711dd8747} {Suffrage:Voter ID:server-94904dcd-bcad-1f47-a0ee-f8c9b99232f5 Address:94904dcd-bcad-1f47-a0ee-f8c9b99232f5}]" +2025-10-30T18:21:46.423Z [INFO] server-7fd8fe99-a380-9c3a-c7b1-00084ebeef4b: entering follower state: follower="Node at 7fd8fe99-a380-9c3a-c7b1-00084ebeef4b [Follower]" leader-address= leader-id= +2025-10-30T18:21:46.424Z [INFO] server-ae1e87fa-09d7-63cc-a083-e33711dd8747: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fd8fe99-a380-9c3a-c7b1-00084ebeef4b Address:7fd8fe99-a380-9c3a-c7b1-00084ebeef4b} {Suffrage:Voter ID:server-ae1e87fa-09d7-63cc-a083-e33711dd8747 Address:ae1e87fa-09d7-63cc-a083-e33711dd8747} {Suffrage:Voter ID:server-94904dcd-bcad-1f47-a0ee-f8c9b99232f5 Address:94904dcd-bcad-1f47-a0ee-f8c9b99232f5}]" +2025-10-30T18:21:46.424Z [INFO] server-94904dcd-bcad-1f47-a0ee-f8c9b99232f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fd8fe99-a380-9c3a-c7b1-00084ebeef4b Address:7fd8fe99-a380-9c3a-c7b1-00084ebeef4b} {Suffrage:Voter ID:server-ae1e87fa-09d7-63cc-a083-e33711dd8747 Address:ae1e87fa-09d7-63cc-a083-e33711dd8747} {Suffrage:Voter ID:server-94904dcd-bcad-1f47-a0ee-f8c9b99232f5 Address:94904dcd-bcad-1f47-a0ee-f8c9b99232f5}]" +2025-10-30T18:21:46.424Z [INFO] server-ae1e87fa-09d7-63cc-a083-e33711dd8747: entering follower state: follower="Node at ae1e87fa-09d7-63cc-a083-e33711dd8747 [Follower]" leader-address= leader-id= +2025-10-30T18:21:46.424Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:46.424Z [INFO] server-94904dcd-bcad-1f47-a0ee-f8c9b99232f5: entering follower state: follower="Node at 94904dcd-bcad-1f47-a0ee-f8c9b99232f5 [Follower]" leader-address= leader-id= +2025-10-30T18:21:47.025Z [WARN] server-7fd8fe99-a380-9c3a-c7b1-00084ebeef4b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:47.025Z [INFO] server-7fd8fe99-a380-9c3a-c7b1-00084ebeef4b: entering candidate state: node="Node at 7fd8fe99-a380-9c3a-c7b1-00084ebeef4b [Candidate]" term=2 +2025-10-30T18:21:47.025Z [INFO] server-7fd8fe99-a380-9c3a-c7b1-00084ebeef4b: election won: term=2 tally=2 +2025-10-30T18:21:47.025Z [INFO] server-7fd8fe99-a380-9c3a-c7b1-00084ebeef4b: entering leader state: leader="Node at 7fd8fe99-a380-9c3a-c7b1-00084ebeef4b [Leader]" +2025-10-30T18:21:47.025Z [INFO] server-7fd8fe99-a380-9c3a-c7b1-00084ebeef4b: added peer, starting replication: peer=server-ae1e87fa-09d7-63cc-a083-e33711dd8747 +2025-10-30T18:21:47.025Z [INFO] server-7fd8fe99-a380-9c3a-c7b1-00084ebeef4b: added peer, starting replication: peer=server-94904dcd-bcad-1f47-a0ee-f8c9b99232f5 +2025-10-30T18:21:47.025Z [INFO] server-7fd8fe99-a380-9c3a-c7b1-00084ebeef4b: pipelining replication: peer="{Voter server-94904dcd-bcad-1f47-a0ee-f8c9b99232f5 94904dcd-bcad-1f47-a0ee-f8c9b99232f5}" +2025-10-30T18:21:47.025Z [INFO] server-7fd8fe99-a380-9c3a-c7b1-00084ebeef4b: pipelining replication: peer="{Voter server-ae1e87fa-09d7-63cc-a083-e33711dd8747 ae1e87fa-09d7-63cc-a083-e33711dd8747}" +2025-10-30T18:21:48.030Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:47.0255486 +0000 UTC m=+30.099077908 (1 nodes), 601.45809ms from start of poll, 601.73147ms from cluster start. Timeout at 2025-10-30 18:21:48.030551232 +0000 UTC m=+31.104080525, 1.005002617s after stability +2025-10-30T18:21:48.031Z [INFO] server-7fd8fe99-a380-9c3a-c7b1-00084ebeef4b: starting snapshot up to: index=12 +2025-10-30T18:21:48.031Z [INFO] creating new snapshot: path=/tmp/raft272675197/snapshots/2-12-1761848508031.tmp +2025-10-30T18:21:48.031Z [INFO] server-7fd8fe99-a380-9c3a-c7b1-00084ebeef4b: snapshot complete up to: index=12 +2025-10-30T18:21:48.032Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d Address:1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d} {Suffrage:Voter ID:server-efc229cc-054f-2b94-f3b3-9bec31752f46 Address:efc229cc-054f-2b94-f3b3-9bec31752f46} {Suffrage:Voter ID:server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9 Address:26cce6da-6cd7-ea19-6f50-7d526f08c4c9}]" +2025-10-30T18:21:48.032Z [INFO] server-efc229cc-054f-2b94-f3b3-9bec31752f46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d Address:1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d} {Suffrage:Voter ID:server-efc229cc-054f-2b94-f3b3-9bec31752f46 Address:efc229cc-054f-2b94-f3b3-9bec31752f46} {Suffrage:Voter ID:server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9 Address:26cce6da-6cd7-ea19-6f50-7d526f08c4c9}]" +2025-10-30T18:21:48.032Z [INFO] server-efc229cc-054f-2b94-f3b3-9bec31752f46: entering follower state: follower="Node at efc229cc-054f-2b94-f3b3-9bec31752f46 [Follower]" leader-address= leader-id= +2025-10-30T18:21:48.032Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: entering follower state: follower="Node at 1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d [Follower]" leader-address= leader-id= +2025-10-30T18:21:48.033Z [INFO] server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d Address:1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d} {Suffrage:Voter ID:server-efc229cc-054f-2b94-f3b3-9bec31752f46 Address:efc229cc-054f-2b94-f3b3-9bec31752f46} {Suffrage:Voter ID:server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9 Address:26cce6da-6cd7-ea19-6f50-7d526f08c4c9}]" +2025-10-30T18:21:48.033Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:48.033Z [INFO] server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9: entering follower state: follower="Node at 26cce6da-6cd7-ea19-6f50-7d526f08c4c9 [Follower]" leader-address= leader-id= +2025-10-30T18:21:48.540Z [WARN] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:48.540Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: entering candidate state: node="Node at 1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d [Candidate]" term=2 +2025-10-30T18:21:48.540Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: election won: term=2 tally=2 +2025-10-30T18:21:48.540Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: entering leader state: leader="Node at 1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d [Leader]" +2025-10-30T18:21:48.540Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: added peer, starting replication: peer=server-efc229cc-054f-2b94-f3b3-9bec31752f46 +2025-10-30T18:21:48.540Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: added peer, starting replication: peer=server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9 +2025-10-30T18:21:48.540Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: pipelining replication: peer="{Voter server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9 26cce6da-6cd7-ea19-6f50-7d526f08c4c9}" +2025-10-30T18:21:48.540Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: pipelining replication: peer="{Voter server-efc229cc-054f-2b94-f3b3-9bec31752f46 efc229cc-054f-2b94-f3b3-9bec31752f46}" +2025-10-30T18:21:49.545Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:48.540529434 +0000 UTC m=+31.614058731 (1 nodes), 507.461723ms from start of poll, 507.825935ms from cluster start. Timeout at 2025-10-30 18:21:49.545537041 +0000 UTC m=+32.619066367, 1.005007636s after stability +2025-10-30T18:21:49.546Z [INFO] creating new snapshot: path=/tmp/raft2198681720/snapshots/2-13-1761848509546.tmp +2025-10-30T18:21:49.794Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: copied to local snapshot: bytes=71 +2025-10-30T18:21:49.794Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: snapshot restore progress: id=2-13-1761848509546 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:49.794Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: removing all old logs from log store +2025-10-30T18:21:49.794Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: compacting logs: from=1 to=2 +2025-10-30T18:21:49.794Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: restored user snapshot: index=13 +2025-10-30T18:21:49.794Z [ERROR] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: failed to get log: index=2 error="log not found" +2025-10-30T18:21:49.794Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: aborting pipeline replication: peer="{Voter server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9 26cce6da-6cd7-ea19-6f50-7d526f08c4c9}" +2025-10-30T18:21:49.794Z [ERROR] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: failed to get log: index=2 error="log not found" +2025-10-30T18:21:49.794Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: aborting pipeline replication: peer="{Voter server-efc229cc-054f-2b94-f3b3-9bec31752f46 efc229cc-054f-2b94-f3b3-9bec31752f46}" +2025-10-30T18:21:49.801Z [ERROR] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: failed to get log: index=2 error="log not found" +2025-10-30T18:21:49.801Z [ERROR] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: failed to get log: index=2 error="log not found" +2025-10-30T18:21:49.801Z [INFO] creating new snapshot: path=/tmp/raft748123278/snapshots/2-13-1761848509801.tmp +2025-10-30T18:21:50.026Z [INFO] server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:50.027Z [INFO] creating new snapshot: path=/tmp/raft577646322/snapshots/2-13-1761848510027.tmp +2025-10-30T18:21:50.027Z [INFO] server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9: copied to local snapshot: bytes=71 +2025-10-30T18:21:50.027Z [INFO] server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9: snapshot restore progress: id=2-13-1761848509801 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:50.027Z [INFO] server-efc229cc-054f-2b94-f3b3-9bec31752f46: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:50.027Z [INFO] server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9: removing all old logs from log store +2025-10-30T18:21:50.027Z [INFO] server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9: compacting logs: from=1 to=2 +2025-10-30T18:21:50.027Z [INFO] server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9: Installed remote snapshot +2025-10-30T18:21:50.027Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: pipelining replication: peer="{Voter server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9 26cce6da-6cd7-ea19-6f50-7d526f08c4c9}" +2025-10-30T18:21:50.027Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:21:50.027Z [INFO] server-efc229cc-054f-2b94-f3b3-9bec31752f46: copied to local snapshot: bytes=71 +2025-10-30T18:21:50.027Z [INFO] server-efc229cc-054f-2b94-f3b3-9bec31752f46: snapshot restore progress: id=2-13-1761848510027 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:50.027Z [INFO] server-efc229cc-054f-2b94-f3b3-9bec31752f46: removing all old logs from log store +2025-10-30T18:21:50.027Z [INFO] server-efc229cc-054f-2b94-f3b3-9bec31752f46: compacting logs: from=1 to=2 +2025-10-30T18:21:50.027Z [INFO] server-efc229cc-054f-2b94-f3b3-9bec31752f46: Installed remote snapshot +2025-10-30T18:21:50.027Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: pipelining replication: peer="{Voter server-efc229cc-054f-2b94-f3b3-9bec31752f46 efc229cc-054f-2b94-f3b3-9bec31752f46}" +2025-10-30T18:21:51.033Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:21:50.027577418 +0000 UTC m=+33.101106711 (2 nodes), 414ns from start of poll, 1.994873915s from cluster start. Timeout at 2025-10-30 18:21:51.032582607 +0000 UTC m=+34.106111934, 1.005005223s after stability +2025-10-30T18:21:51.044Z [INFO] server-7fd8fe99-a380-9c3a-c7b1-00084ebeef4b: aborting pipeline replication: peer="{Voter server-94904dcd-bcad-1f47-a0ee-f8c9b99232f5 94904dcd-bcad-1f47-a0ee-f8c9b99232f5}" +2025-10-30T18:21:51.044Z [INFO] server-7fd8fe99-a380-9c3a-c7b1-00084ebeef4b: aborting pipeline replication: peer="{Voter server-ae1e87fa-09d7-63cc-a083-e33711dd8747 ae1e87fa-09d7-63cc-a083-e33711dd8747}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false -2025-10-30T18:18:01.169Z [INFO] server-cb0ee475-1152-2541-4716-ac0c8d654162: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb0ee475-1152-2541-4716-ac0c8d654162 Address:cb0ee475-1152-2541-4716-ac0c8d654162} {Suffrage:Voter ID:server-31b4661f-d3ae-2b72-0536-84782ccc54bc Address:31b4661f-d3ae-2b72-0536-84782ccc54bc} {Suffrage:Voter ID:server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837 Address:40f9ef41-3e46-3c01-d6e9-e57f47ad9837}]" -2025-10-30T18:18:01.169Z [INFO] server-31b4661f-d3ae-2b72-0536-84782ccc54bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb0ee475-1152-2541-4716-ac0c8d654162 Address:cb0ee475-1152-2541-4716-ac0c8d654162} {Suffrage:Voter ID:server-31b4661f-d3ae-2b72-0536-84782ccc54bc Address:31b4661f-d3ae-2b72-0536-84782ccc54bc} {Suffrage:Voter ID:server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837 Address:40f9ef41-3e46-3c01-d6e9-e57f47ad9837}]" -2025-10-30T18:18:01.169Z [INFO] server-cb0ee475-1152-2541-4716-ac0c8d654162: entering follower state: follower="Node at cb0ee475-1152-2541-4716-ac0c8d654162 [Follower]" leader-address= leader-id= -2025-10-30T18:18:01.169Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb0ee475-1152-2541-4716-ac0c8d654162 Address:cb0ee475-1152-2541-4716-ac0c8d654162} {Suffrage:Voter ID:server-31b4661f-d3ae-2b72-0536-84782ccc54bc Address:31b4661f-d3ae-2b72-0536-84782ccc54bc} {Suffrage:Voter ID:server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837 Address:40f9ef41-3e46-3c01-d6e9-e57f47ad9837}]" -2025-10-30T18:18:01.169Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:01.169Z [INFO] server-31b4661f-d3ae-2b72-0536-84782ccc54bc: entering follower state: follower="Node at 31b4661f-d3ae-2b72-0536-84782ccc54bc [Follower]" leader-address= leader-id= -2025-10-30T18:18:01.169Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: entering follower state: follower="Node at 40f9ef41-3e46-3c01-d6e9-e57f47ad9837 [Follower]" leader-address= leader-id= -2025-10-30T18:18:01.699Z [WARN] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:01.699Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: entering candidate state: node="Node at 40f9ef41-3e46-3c01-d6e9-e57f47ad9837 [Candidate]" term=2 -2025-10-30T18:18:01.699Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: election won: term=2 tally=2 -2025-10-30T18:18:01.699Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: entering leader state: leader="Node at 40f9ef41-3e46-3c01-d6e9-e57f47ad9837 [Leader]" -2025-10-30T18:18:01.699Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: added peer, starting replication: peer=server-cb0ee475-1152-2541-4716-ac0c8d654162 -2025-10-30T18:18:01.699Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: added peer, starting replication: peer=server-31b4661f-d3ae-2b72-0536-84782ccc54bc -2025-10-30T18:18:01.699Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: pipelining replication: peer="{Voter server-cb0ee475-1152-2541-4716-ac0c8d654162 cb0ee475-1152-2541-4716-ac0c8d654162}" -2025-10-30T18:18:01.699Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: pipelining replication: peer="{Voter server-31b4661f-d3ae-2b72-0536-84782ccc54bc 31b4661f-d3ae-2b72-0536-84782ccc54bc}" -2025-10-30T18:18:02.705Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:01.699742306 +0000 UTC m=+27.978416211 (1 nodes), 530.117757ms from start of poll, 530.349879ms from cluster start. Timeout at 2025-10-30 18:18:02.704744249 +0000 UTC m=+28.983418147, 1.005001936s after stability -2025-10-30T18:18:02.706Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: starting snapshot up to: index=12 -2025-10-30T18:18:02.706Z [INFO] creating new snapshot: path=/tmp/raft3765395359/snapshots/2-12-1761848282706.tmp -2025-10-30T18:18:02.707Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: snapshot complete up to: index=12 -2025-10-30T18:18:02.707Z [INFO] creating new snapshot: path=/tmp/raft3765395359/snapshots/2-23-1761848282707.tmp -2025-10-30T18:18:02.708Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: copied to local snapshot: bytes=71 -2025-10-30T18:18:02.708Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: snapshot restore progress: id=2-23-1761848282707 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:02.708Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: restored user snapshot: index=23 -2025-10-30T18:18:02.708Z [ERROR] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: failed to get log: index=23 error="log not found" -2025-10-30T18:18:02.708Z [ERROR] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: failed to get log: index=23 error="log not found" -2025-10-30T18:18:02.708Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: aborting pipeline replication: peer="{Voter server-31b4661f-d3ae-2b72-0536-84782ccc54bc 31b4661f-d3ae-2b72-0536-84782ccc54bc}" -2025-10-30T18:18:02.708Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: aborting pipeline replication: peer="{Voter server-cb0ee475-1152-2541-4716-ac0c8d654162 cb0ee475-1152-2541-4716-ac0c8d654162}" -2025-10-30T18:18:02.714Z [ERROR] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: failed to get log: index=23 error="log not found" -2025-10-30T18:18:02.714Z [INFO] creating new snapshot: path=/tmp/raft3638930787/snapshots/2-23-1761848282714.tmp -2025-10-30T18:18:02.714Z [INFO] server-cb0ee475-1152-2541-4716-ac0c8d654162: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:02.715Z [INFO] server-cb0ee475-1152-2541-4716-ac0c8d654162: copied to local snapshot: bytes=71 -2025-10-30T18:18:02.715Z [INFO] server-cb0ee475-1152-2541-4716-ac0c8d654162: snapshot restore progress: id=2-23-1761848282714 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:02.715Z [INFO] server-cb0ee475-1152-2541-4716-ac0c8d654162: Installed remote snapshot -2025-10-30T18:18:02.715Z [WARN] server-cb0ee475-1152-2541-4716-ac0c8d654162: skipping application of old log: index=22 -2025-10-30T18:18:02.715Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: pipelining replication: peer="{Voter server-cb0ee475-1152-2541-4716-ac0c8d654162 cb0ee475-1152-2541-4716-ac0c8d654162}" -2025-10-30T18:18:02.715Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:18:02.716Z [ERROR] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: failed to get log: index=23 error="log not found" -2025-10-30T18:18:02.716Z [INFO] creating new snapshot: path=/tmp/raft800884271/snapshots/2-23-1761848282716.tmp -2025-10-30T18:18:02.716Z [INFO] server-31b4661f-d3ae-2b72-0536-84782ccc54bc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:02.717Z [INFO] server-31b4661f-d3ae-2b72-0536-84782ccc54bc: copied to local snapshot: bytes=71 -2025-10-30T18:18:02.717Z [INFO] server-31b4661f-d3ae-2b72-0536-84782ccc54bc: snapshot restore progress: id=2-23-1761848282716 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:02.717Z [INFO] server-31b4661f-d3ae-2b72-0536-84782ccc54bc: Installed remote snapshot -2025-10-30T18:18:02.717Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: pipelining replication: peer="{Voter server-31b4661f-d3ae-2b72-0536-84782ccc54bc 31b4661f-d3ae-2b72-0536-84782ccc54bc}" -2025-10-30T18:18:03.720Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:18:02.715493414 +0000 UTC m=+28.994167314 (2 nodes), 484ns from start of poll, 1.546100982s from cluster start. Timeout at 2025-10-30 18:18:03.720505029 +0000 UTC m=+29.999178944, 1.00501163s after stability -2025-10-30T18:18:03.732Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: aborting pipeline replication: peer="{Voter server-31b4661f-d3ae-2b72-0536-84782ccc54bc 31b4661f-d3ae-2b72-0536-84782ccc54bc}" -2025-10-30T18:18:03.732Z [INFO] server-40f9ef41-3e46-3c01-d6e9-e57f47ad9837: aborting pipeline replication: peer="{Voter server-cb0ee475-1152-2541-4716-ac0c8d654162 cb0ee475-1152-2541-4716-ac0c8d654162}" +2025-10-30T18:21:51.166Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acf36646-6f43-c123-97db-431e00006417 Address:acf36646-6f43-c123-97db-431e00006417} {Suffrage:Voter ID:server-b6b14e42-4dd9-42e5-8d29-ebb80f624bf9 Address:b6b14e42-4dd9-42e5-8d29-ebb80f624bf9} {Suffrage:Voter ID:server-bfae86bb-2461-3b5b-80b5-de11ae1ded7d Address:bfae86bb-2461-3b5b-80b5-de11ae1ded7d}]" +2025-10-30T18:21:51.166Z [INFO] server-b6b14e42-4dd9-42e5-8d29-ebb80f624bf9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acf36646-6f43-c123-97db-431e00006417 Address:acf36646-6f43-c123-97db-431e00006417} {Suffrage:Voter ID:server-b6b14e42-4dd9-42e5-8d29-ebb80f624bf9 Address:b6b14e42-4dd9-42e5-8d29-ebb80f624bf9} {Suffrage:Voter ID:server-bfae86bb-2461-3b5b-80b5-de11ae1ded7d Address:bfae86bb-2461-3b5b-80b5-de11ae1ded7d}]" +2025-10-30T18:21:51.166Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: entering follower state: follower="Node at acf36646-6f43-c123-97db-431e00006417 [Follower]" leader-address= leader-id= +2025-10-30T18:21:51.166Z [INFO] server-b6b14e42-4dd9-42e5-8d29-ebb80f624bf9: entering follower state: follower="Node at b6b14e42-4dd9-42e5-8d29-ebb80f624bf9 [Follower]" leader-address= leader-id= +2025-10-30T18:21:51.166Z [INFO] server-bfae86bb-2461-3b5b-80b5-de11ae1ded7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acf36646-6f43-c123-97db-431e00006417 Address:acf36646-6f43-c123-97db-431e00006417} {Suffrage:Voter ID:server-b6b14e42-4dd9-42e5-8d29-ebb80f624bf9 Address:b6b14e42-4dd9-42e5-8d29-ebb80f624bf9} {Suffrage:Voter ID:server-bfae86bb-2461-3b5b-80b5-de11ae1ded7d Address:bfae86bb-2461-3b5b-80b5-de11ae1ded7d}]" +2025-10-30T18:21:51.166Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:51.166Z [INFO] server-bfae86bb-2461-3b5b-80b5-de11ae1ded7d: entering follower state: follower="Node at bfae86bb-2461-3b5b-80b5-de11ae1ded7d [Follower]" leader-address= leader-id= +2025-10-30T18:21:51.716Z [WARN] server-acf36646-6f43-c123-97db-431e00006417: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:51.716Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: entering candidate state: node="Node at acf36646-6f43-c123-97db-431e00006417 [Candidate]" term=2 +2025-10-30T18:21:51.716Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: election won: term=2 tally=2 +2025-10-30T18:21:51.716Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: entering leader state: leader="Node at acf36646-6f43-c123-97db-431e00006417 [Leader]" +2025-10-30T18:21:51.716Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: added peer, starting replication: peer=server-b6b14e42-4dd9-42e5-8d29-ebb80f624bf9 +2025-10-30T18:21:51.716Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: added peer, starting replication: peer=server-bfae86bb-2461-3b5b-80b5-de11ae1ded7d +2025-10-30T18:21:51.716Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: pipelining replication: peer="{Voter server-bfae86bb-2461-3b5b-80b5-de11ae1ded7d bfae86bb-2461-3b5b-80b5-de11ae1ded7d}" +2025-10-30T18:21:51.716Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: pipelining replication: peer="{Voter server-b6b14e42-4dd9-42e5-8d29-ebb80f624bf9 b6b14e42-4dd9-42e5-8d29-ebb80f624bf9}" +2025-10-30T18:21:52.721Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:51.716209656 +0000 UTC m=+34.789738949 (1 nodes), 549.21648ms from start of poll, 549.623527ms from cluster start. Timeout at 2025-10-30 18:21:52.721214024 +0000 UTC m=+35.794743350, 1.005004401s after stability +2025-10-30T18:21:52.722Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: starting snapshot up to: index=12 +2025-10-30T18:21:52.722Z [INFO] creating new snapshot: path=/tmp/raft3570815290/snapshots/2-12-1761848512722.tmp +2025-10-30T18:21:52.828Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: snapshot complete up to: index=12 +2025-10-30T18:21:52.829Z [INFO] creating new snapshot: path=/tmp/raft3570815290/snapshots/2-23-1761848512829.tmp +2025-10-30T18:21:52.829Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: copied to local snapshot: bytes=71 +2025-10-30T18:21:52.829Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: snapshot restore progress: id=2-23-1761848512829 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:52.829Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: restored user snapshot: index=23 +2025-10-30T18:21:52.830Z [ERROR] server-acf36646-6f43-c123-97db-431e00006417: failed to get log: index=23 error="log not found" +2025-10-30T18:21:52.830Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: aborting pipeline replication: peer="{Voter server-bfae86bb-2461-3b5b-80b5-de11ae1ded7d bfae86bb-2461-3b5b-80b5-de11ae1ded7d}" +2025-10-30T18:21:52.830Z [ERROR] server-acf36646-6f43-c123-97db-431e00006417: failed to get log: index=23 error="log not found" +2025-10-30T18:21:52.830Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: aborting pipeline replication: peer="{Voter server-b6b14e42-4dd9-42e5-8d29-ebb80f624bf9 b6b14e42-4dd9-42e5-8d29-ebb80f624bf9}" +2025-10-30T18:21:52.836Z [ERROR] server-acf36646-6f43-c123-97db-431e00006417: failed to get log: index=23 error="log not found" +2025-10-30T18:21:52.836Z [INFO] creating new snapshot: path=/tmp/raft2826377949/snapshots/2-23-1761848512836.tmp +2025-10-30T18:21:52.836Z [INFO] server-b6b14e42-4dd9-42e5-8d29-ebb80f624bf9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:52.837Z [INFO] server-b6b14e42-4dd9-42e5-8d29-ebb80f624bf9: copied to local snapshot: bytes=71 +2025-10-30T18:21:52.837Z [INFO] server-b6b14e42-4dd9-42e5-8d29-ebb80f624bf9: snapshot restore progress: id=2-23-1761848512836 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:52.837Z [INFO] server-b6b14e42-4dd9-42e5-8d29-ebb80f624bf9: Installed remote snapshot +2025-10-30T18:21:52.837Z [WARN] server-b6b14e42-4dd9-42e5-8d29-ebb80f624bf9: skipping application of old log: index=22 +2025-10-30T18:21:52.837Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: pipelining replication: peer="{Voter server-b6b14e42-4dd9-42e5-8d29-ebb80f624bf9 b6b14e42-4dd9-42e5-8d29-ebb80f624bf9}" +2025-10-30T18:21:52.837Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:21:52.839Z [ERROR] server-acf36646-6f43-c123-97db-431e00006417: failed to get log: index=23 error="log not found" +2025-10-30T18:21:52.840Z [INFO] creating new snapshot: path=/tmp/raft1037103722/snapshots/2-23-1761848512840.tmp +2025-10-30T18:21:52.840Z [INFO] server-bfae86bb-2461-3b5b-80b5-de11ae1ded7d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:52.840Z [INFO] server-bfae86bb-2461-3b5b-80b5-de11ae1ded7d: copied to local snapshot: bytes=71 +2025-10-30T18:21:52.840Z [INFO] server-bfae86bb-2461-3b5b-80b5-de11ae1ded7d: snapshot restore progress: id=2-23-1761848512840 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:52.840Z [INFO] server-bfae86bb-2461-3b5b-80b5-de11ae1ded7d: Installed remote snapshot +2025-10-30T18:21:52.840Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: pipelining replication: peer="{Voter server-bfae86bb-2461-3b5b-80b5-de11ae1ded7d bfae86bb-2461-3b5b-80b5-de11ae1ded7d}" +2025-10-30T18:21:53.843Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:21:52.837569885 +0000 UTC m=+35.911099197 (2 nodes), 692ns from start of poll, 1.670983775s from cluster start. Timeout at 2025-10-30 18:21:53.842576808 +0000 UTC m=+36.916106101, 1.005006904s after stability +2025-10-30T18:21:53.855Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: aborting pipeline replication: peer="{Voter server-bfae86bb-2461-3b5b-80b5-de11ae1ded7d bfae86bb-2461-3b5b-80b5-de11ae1ded7d}" +2025-10-30T18:21:53.855Z [INFO] server-acf36646-6f43-c123-97db-431e00006417: aborting pipeline replication: peer="{Voter server-b6b14e42-4dd9-42e5-8d29-ebb80f624bf9 b6b14e42-4dd9-42e5-8d29-ebb80f624bf9}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false -2025-10-30T18:18:03.733Z [INFO] server-561a8915-c417-41d8-cec1-85c4f701c32b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-561a8915-c417-41d8-cec1-85c4f701c32b Address:561a8915-c417-41d8-cec1-85c4f701c32b} {Suffrage:Voter ID:server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca Address:b7edf66e-f2e7-2b50-89dc-f37cccabb3ca} {Suffrage:Voter ID:server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac Address:8c7628c0-82c7-9a6f-6e0c-c5a6205399ac}]" -2025-10-30T18:18:03.733Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-561a8915-c417-41d8-cec1-85c4f701c32b Address:561a8915-c417-41d8-cec1-85c4f701c32b} {Suffrage:Voter ID:server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca Address:b7edf66e-f2e7-2b50-89dc-f37cccabb3ca} {Suffrage:Voter ID:server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac Address:8c7628c0-82c7-9a6f-6e0c-c5a6205399ac}]" -2025-10-30T18:18:03.733Z [INFO] server-561a8915-c417-41d8-cec1-85c4f701c32b: entering follower state: follower="Node at 561a8915-c417-41d8-cec1-85c4f701c32b [Follower]" leader-address= leader-id= -2025-10-30T18:18:03.733Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: entering follower state: follower="Node at b7edf66e-f2e7-2b50-89dc-f37cccabb3ca [Follower]" leader-address= leader-id= -2025-10-30T18:18:03.733Z [INFO] server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-561a8915-c417-41d8-cec1-85c4f701c32b Address:561a8915-c417-41d8-cec1-85c4f701c32b} {Suffrage:Voter ID:server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca Address:b7edf66e-f2e7-2b50-89dc-f37cccabb3ca} {Suffrage:Voter ID:server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac Address:8c7628c0-82c7-9a6f-6e0c-c5a6205399ac}]" -2025-10-30T18:18:03.733Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:03.733Z [INFO] server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac: entering follower state: follower="Node at 8c7628c0-82c7-9a6f-6e0c-c5a6205399ac [Follower]" leader-address= leader-id= -2025-10-30T18:18:04.326Z [WARN] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:04.326Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: entering candidate state: node="Node at b7edf66e-f2e7-2b50-89dc-f37cccabb3ca [Candidate]" term=2 -2025-10-30T18:18:04.326Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: election won: term=2 tally=2 -2025-10-30T18:18:04.326Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: entering leader state: leader="Node at b7edf66e-f2e7-2b50-89dc-f37cccabb3ca [Leader]" -2025-10-30T18:18:04.326Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: added peer, starting replication: peer=server-561a8915-c417-41d8-cec1-85c4f701c32b -2025-10-30T18:18:04.326Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: added peer, starting replication: peer=server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac -2025-10-30T18:18:04.326Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: pipelining replication: peer="{Voter server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac 8c7628c0-82c7-9a6f-6e0c-c5a6205399ac}" -2025-10-30T18:18:04.326Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: pipelining replication: peer="{Voter server-561a8915-c417-41d8-cec1-85c4f701c32b 561a8915-c417-41d8-cec1-85c4f701c32b}" -2025-10-30T18:18:05.332Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:04.326540601 +0000 UTC m=+30.605214499 (1 nodes), 593.131286ms from start of poll, 593.41544ms from cluster start. Timeout at 2025-10-30 18:18:05.331543008 +0000 UTC m=+31.610216905, 1.005002406s after stability -2025-10-30T18:18:05.332Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: starting snapshot up to: index=12 -2025-10-30T18:18:05.332Z [INFO] creating new snapshot: path=/tmp/raft1758275657/snapshots/2-12-1761848285332.tmp -2025-10-30T18:18:05.333Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: snapshot complete up to: index=12 -2025-10-30T18:18:05.333Z [INFO] creating new snapshot: path=/tmp/raft1758275657/snapshots/2-23-1761848285333.tmp -2025-10-30T18:18:05.333Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: copied to local snapshot: bytes=71 -2025-10-30T18:18:05.334Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: snapshot restore progress: id=2-23-1761848285333 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:05.334Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: removing all old logs from log store -2025-10-30T18:18:05.334Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: compacting logs: from=1 to=22 -2025-10-30T18:18:05.334Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: restored user snapshot: index=23 -2025-10-30T18:18:05.334Z [ERROR] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: failed to get log: index=22 error="log not found" -2025-10-30T18:18:05.334Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: aborting pipeline replication: peer="{Voter server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac 8c7628c0-82c7-9a6f-6e0c-c5a6205399ac}" -2025-10-30T18:18:05.334Z [ERROR] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: failed to get log: index=22 error="log not found" -2025-10-30T18:18:05.334Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: aborting pipeline replication: peer="{Voter server-561a8915-c417-41d8-cec1-85c4f701c32b 561a8915-c417-41d8-cec1-85c4f701c32b}" -2025-10-30T18:18:05.342Z [ERROR] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: failed to get log: index=22 error="log not found" -2025-10-30T18:18:05.342Z [INFO] creating new snapshot: path=/tmp/raft3093969630/snapshots/2-23-1761848285342.tmp -2025-10-30T18:18:05.343Z [INFO] server-561a8915-c417-41d8-cec1-85c4f701c32b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:05.343Z [INFO] server-561a8915-c417-41d8-cec1-85c4f701c32b: copied to local snapshot: bytes=71 -2025-10-30T18:18:05.343Z [INFO] server-561a8915-c417-41d8-cec1-85c4f701c32b: snapshot restore progress: id=2-23-1761848285342 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:05.343Z [INFO] server-561a8915-c417-41d8-cec1-85c4f701c32b: removing all old logs from log store -2025-10-30T18:18:05.343Z [INFO] server-561a8915-c417-41d8-cec1-85c4f701c32b: compacting logs: from=1 to=22 -2025-10-30T18:18:05.343Z [INFO] server-561a8915-c417-41d8-cec1-85c4f701c32b: Installed remote snapshot -2025-10-30T18:18:05.343Z [WARN] server-561a8915-c417-41d8-cec1-85c4f701c32b: skipping application of old log: index=22 -2025-10-30T18:18:05.343Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: pipelining replication: peer="{Voter server-561a8915-c417-41d8-cec1-85c4f701c32b 561a8915-c417-41d8-cec1-85c4f701c32b}" -2025-10-30T18:18:05.343Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:18:05.344Z [ERROR] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: failed to get log: index=22 error="log not found" -2025-10-30T18:18:05.345Z [INFO] creating new snapshot: path=/tmp/raft4163550248/snapshots/2-23-1761848285345.tmp -2025-10-30T18:18:05.345Z [INFO] server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:05.345Z [INFO] server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac: copied to local snapshot: bytes=71 -2025-10-30T18:18:05.346Z [INFO] server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac: snapshot restore progress: id=2-23-1761848285345 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:05.346Z [INFO] server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac: removing all old logs from log store -2025-10-30T18:18:05.346Z [INFO] server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac: compacting logs: from=1 to=22 -2025-10-30T18:18:05.346Z [INFO] server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac: Installed remote snapshot -2025-10-30T18:18:05.347Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: pipelining replication: peer="{Voter server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac 8c7628c0-82c7-9a6f-6e0c-c5a6205399ac}" -2025-10-30T18:18:06.349Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:18:05.343608178 +0000 UTC m=+31.622282087 (2 nodes), 707ns from start of poll, 1.610483028s from cluster start. Timeout at 2025-10-30 18:18:06.348610943 +0000 UTC m=+32.627284840, 1.005002753s after stability -2025-10-30T18:18:06.365Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: aborting pipeline replication: peer="{Voter server-8c7628c0-82c7-9a6f-6e0c-c5a6205399ac 8c7628c0-82c7-9a6f-6e0c-c5a6205399ac}" -2025-10-30T18:18:06.365Z [INFO] server-b7edf66e-f2e7-2b50-89dc-f37cccabb3ca: aborting pipeline replication: peer="{Voter server-561a8915-c417-41d8-cec1-85c4f701c32b 561a8915-c417-41d8-cec1-85c4f701c32b}" +2025-10-30T18:21:53.857Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4 Address:b87327e3-c35d-6ca8-f8a4-ccd7def16fc4} {Suffrage:Voter ID:server-7618fd78-59b4-c2df-ef78-dd96219a27f5 Address:7618fd78-59b4-c2df-ef78-dd96219a27f5} {Suffrage:Voter ID:server-428a9fa2-169a-433b-f98c-e9c90323c1d4 Address:428a9fa2-169a-433b-f98c-e9c90323c1d4}]" +2025-10-30T18:21:53.858Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: entering follower state: follower="Node at b87327e3-c35d-6ca8-f8a4-ccd7def16fc4 [Follower]" leader-address= leader-id= +2025-10-30T18:21:53.858Z [INFO] server-7618fd78-59b4-c2df-ef78-dd96219a27f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4 Address:b87327e3-c35d-6ca8-f8a4-ccd7def16fc4} {Suffrage:Voter ID:server-7618fd78-59b4-c2df-ef78-dd96219a27f5 Address:7618fd78-59b4-c2df-ef78-dd96219a27f5} {Suffrage:Voter ID:server-428a9fa2-169a-433b-f98c-e9c90323c1d4 Address:428a9fa2-169a-433b-f98c-e9c90323c1d4}]" +2025-10-30T18:21:53.858Z [INFO] server-7618fd78-59b4-c2df-ef78-dd96219a27f5: entering follower state: follower="Node at 7618fd78-59b4-c2df-ef78-dd96219a27f5 [Follower]" leader-address= leader-id= +2025-10-30T18:21:53.858Z [INFO] server-428a9fa2-169a-433b-f98c-e9c90323c1d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4 Address:b87327e3-c35d-6ca8-f8a4-ccd7def16fc4} {Suffrage:Voter ID:server-7618fd78-59b4-c2df-ef78-dd96219a27f5 Address:7618fd78-59b4-c2df-ef78-dd96219a27f5} {Suffrage:Voter ID:server-428a9fa2-169a-433b-f98c-e9c90323c1d4 Address:428a9fa2-169a-433b-f98c-e9c90323c1d4}]" +2025-10-30T18:21:53.858Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:53.858Z [INFO] server-428a9fa2-169a-433b-f98c-e9c90323c1d4: entering follower state: follower="Node at 428a9fa2-169a-433b-f98c-e9c90323c1d4 [Follower]" leader-address= leader-id= +2025-10-30T18:21:54.406Z [WARN] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:54.406Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: entering candidate state: node="Node at b87327e3-c35d-6ca8-f8a4-ccd7def16fc4 [Candidate]" term=2 +2025-10-30T18:21:54.406Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: election won: term=2 tally=2 +2025-10-30T18:21:54.406Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: entering leader state: leader="Node at b87327e3-c35d-6ca8-f8a4-ccd7def16fc4 [Leader]" +2025-10-30T18:21:54.407Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: added peer, starting replication: peer=server-7618fd78-59b4-c2df-ef78-dd96219a27f5 +2025-10-30T18:21:54.407Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: added peer, starting replication: peer=server-428a9fa2-169a-433b-f98c-e9c90323c1d4 +2025-10-30T18:21:54.407Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: pipelining replication: peer="{Voter server-428a9fa2-169a-433b-f98c-e9c90323c1d4 428a9fa2-169a-433b-f98c-e9c90323c1d4}" +2025-10-30T18:21:54.407Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: pipelining replication: peer="{Voter server-7618fd78-59b4-c2df-ef78-dd96219a27f5 7618fd78-59b4-c2df-ef78-dd96219a27f5}" +2025-10-30T18:21:55.412Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:54.407009621 +0000 UTC m=+37.480538947 (1 nodes), 548.807908ms from start of poll, 549.181963ms from cluster start. Timeout at 2025-10-30 18:21:55.412077149 +0000 UTC m=+38.485606442, 1.005067495s after stability +2025-10-30T18:21:55.413Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: starting snapshot up to: index=12 +2025-10-30T18:21:55.413Z [INFO] creating new snapshot: path=/tmp/raft943355033/snapshots/2-12-1761848515413.tmp +2025-10-30T18:21:55.413Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: snapshot complete up to: index=12 +2025-10-30T18:21:55.414Z [INFO] creating new snapshot: path=/tmp/raft943355033/snapshots/2-23-1761848515414.tmp +2025-10-30T18:21:55.414Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: copied to local snapshot: bytes=71 +2025-10-30T18:21:55.415Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: snapshot restore progress: id=2-23-1761848515414 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:55.415Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: removing all old logs from log store +2025-10-30T18:21:55.415Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: compacting logs: from=1 to=22 +2025-10-30T18:21:55.415Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: restored user snapshot: index=23 +2025-10-30T18:21:55.415Z [ERROR] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: failed to get log: index=22 error="log not found" +2025-10-30T18:21:55.415Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: aborting pipeline replication: peer="{Voter server-428a9fa2-169a-433b-f98c-e9c90323c1d4 428a9fa2-169a-433b-f98c-e9c90323c1d4}" +2025-10-30T18:21:55.415Z [ERROR] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: failed to get log: index=22 error="log not found" +2025-10-30T18:21:55.415Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: aborting pipeline replication: peer="{Voter server-7618fd78-59b4-c2df-ef78-dd96219a27f5 7618fd78-59b4-c2df-ef78-dd96219a27f5}" +2025-10-30T18:21:55.421Z [ERROR] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: failed to get log: index=22 error="log not found" +2025-10-30T18:21:55.422Z [ERROR] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: failed to get log: index=22 error="log not found" +2025-10-30T18:21:55.422Z [INFO] creating new snapshot: path=/tmp/raft1803224540/snapshots/2-23-1761848515422.tmp +2025-10-30T18:21:55.423Z [INFO] server-7618fd78-59b4-c2df-ef78-dd96219a27f5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:55.423Z [INFO] creating new snapshot: path=/tmp/raft1509511106/snapshots/2-23-1761848515423.tmp +2025-10-30T18:21:55.423Z [INFO] server-7618fd78-59b4-c2df-ef78-dd96219a27f5: copied to local snapshot: bytes=71 +2025-10-30T18:21:55.423Z [INFO] server-428a9fa2-169a-433b-f98c-e9c90323c1d4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:55.424Z [INFO] server-7618fd78-59b4-c2df-ef78-dd96219a27f5: snapshot restore progress: id=2-23-1761848515422 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:55.424Z [INFO] server-7618fd78-59b4-c2df-ef78-dd96219a27f5: removing all old logs from log store +2025-10-30T18:21:55.424Z [INFO] server-7618fd78-59b4-c2df-ef78-dd96219a27f5: compacting logs: from=1 to=22 +2025-10-30T18:21:55.424Z [INFO] server-7618fd78-59b4-c2df-ef78-dd96219a27f5: Installed remote snapshot +2025-10-30T18:21:55.424Z [WARN] server-7618fd78-59b4-c2df-ef78-dd96219a27f5: skipping application of old log: index=22 +2025-10-30T18:21:55.424Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: pipelining replication: peer="{Voter server-7618fd78-59b4-c2df-ef78-dd96219a27f5 7618fd78-59b4-c2df-ef78-dd96219a27f5}" +2025-10-30T18:21:55.424Z [INFO] server-428a9fa2-169a-433b-f98c-e9c90323c1d4: copied to local snapshot: bytes=71 +2025-10-30T18:21:55.424Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:21:55.424Z [INFO] server-428a9fa2-169a-433b-f98c-e9c90323c1d4: snapshot restore progress: id=2-23-1761848515423 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:55.424Z [INFO] server-428a9fa2-169a-433b-f98c-e9c90323c1d4: removing all old logs from log store +2025-10-30T18:21:55.424Z [INFO] server-428a9fa2-169a-433b-f98c-e9c90323c1d4: compacting logs: from=1 to=22 +2025-10-30T18:21:55.424Z [INFO] server-428a9fa2-169a-433b-f98c-e9c90323c1d4: Installed remote snapshot +2025-10-30T18:21:55.424Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: pipelining replication: peer="{Voter server-428a9fa2-169a-433b-f98c-e9c90323c1d4 428a9fa2-169a-433b-f98c-e9c90323c1d4}" +2025-10-30T18:21:56.430Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:21:55.424303254 +0000 UTC m=+38.497832562 (2 nodes), 919ns from start of poll, 1.566475578s from cluster start. Timeout at 2025-10-30 18:21:56.429309692 +0000 UTC m=+39.502838982, 1.00500642s after stability +2025-10-30T18:21:56.441Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: aborting pipeline replication: peer="{Voter server-428a9fa2-169a-433b-f98c-e9c90323c1d4 428a9fa2-169a-433b-f98c-e9c90323c1d4}" +2025-10-30T18:21:56.441Z [INFO] server-b87327e3-c35d-6ca8-f8a4-ccd7def16fc4: aborting pipeline replication: peer="{Voter server-7618fd78-59b4-c2df-ef78-dd96219a27f5 7618fd78-59b4-c2df-ef78-dd96219a27f5}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true -2025-10-30T18:18:06.366Z [INFO] server-e1cb99c7-b8ea-92a4-ee56-8255791394cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1cb99c7-b8ea-92a4-ee56-8255791394cb Address:e1cb99c7-b8ea-92a4-ee56-8255791394cb} {Suffrage:Voter ID:server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef Address:8334db8f-9bd0-cbc8-d840-4dd7803f22ef} {Suffrage:Voter ID:server-4cd8842c-131a-aaec-75a1-e60cd5b8f838 Address:4cd8842c-131a-aaec-75a1-e60cd5b8f838}]" -2025-10-30T18:18:06.366Z [INFO] server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1cb99c7-b8ea-92a4-ee56-8255791394cb Address:e1cb99c7-b8ea-92a4-ee56-8255791394cb} {Suffrage:Voter ID:server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef Address:8334db8f-9bd0-cbc8-d840-4dd7803f22ef} {Suffrage:Voter ID:server-4cd8842c-131a-aaec-75a1-e60cd5b8f838 Address:4cd8842c-131a-aaec-75a1-e60cd5b8f838}]" -2025-10-30T18:18:06.366Z [INFO] server-e1cb99c7-b8ea-92a4-ee56-8255791394cb: entering follower state: follower="Node at e1cb99c7-b8ea-92a4-ee56-8255791394cb [Follower]" leader-address= leader-id= -2025-10-30T18:18:06.366Z [INFO] server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: entering follower state: follower="Node at 8334db8f-9bd0-cbc8-d840-4dd7803f22ef [Follower]" leader-address= leader-id= -2025-10-30T18:18:06.366Z [INFO] server-4cd8842c-131a-aaec-75a1-e60cd5b8f838: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1cb99c7-b8ea-92a4-ee56-8255791394cb Address:e1cb99c7-b8ea-92a4-ee56-8255791394cb} {Suffrage:Voter ID:server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef Address:8334db8f-9bd0-cbc8-d840-4dd7803f22ef} {Suffrage:Voter ID:server-4cd8842c-131a-aaec-75a1-e60cd5b8f838 Address:4cd8842c-131a-aaec-75a1-e60cd5b8f838}]" -2025-10-30T18:18:06.366Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:06.367Z [INFO] server-4cd8842c-131a-aaec-75a1-e60cd5b8f838: entering follower state: follower="Node at 4cd8842c-131a-aaec-75a1-e60cd5b8f838 [Follower]" leader-address= leader-id= -2025-10-30T18:18:07.046Z [WARN] server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:07.046Z [INFO] server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: entering candidate state: node="Node at 8334db8f-9bd0-cbc8-d840-4dd7803f22ef [Candidate]" term=2 -2025-10-30T18:18:07.047Z [INFO] server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: election won: term=2 tally=2 -2025-10-30T18:18:07.047Z [INFO] server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: entering leader state: leader="Node at 8334db8f-9bd0-cbc8-d840-4dd7803f22ef [Leader]" -2025-10-30T18:18:07.047Z [INFO] server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: added peer, starting replication: peer=server-e1cb99c7-b8ea-92a4-ee56-8255791394cb -2025-10-30T18:18:07.047Z [INFO] server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: added peer, starting replication: peer=server-4cd8842c-131a-aaec-75a1-e60cd5b8f838 -2025-10-30T18:18:07.047Z [INFO] server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: pipelining replication: peer="{Voter server-e1cb99c7-b8ea-92a4-ee56-8255791394cb e1cb99c7-b8ea-92a4-ee56-8255791394cb}" -2025-10-30T18:18:07.047Z [INFO] server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: pipelining replication: peer="{Voter server-4cd8842c-131a-aaec-75a1-e60cd5b8f838 4cd8842c-131a-aaec-75a1-e60cd5b8f838}" -2025-10-30T18:18:08.052Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:07.047138003 +0000 UTC m=+33.325811903 (1 nodes), 680.14279ms from start of poll, 680.385909ms from cluster start. Timeout at 2025-10-30 18:18:08.052202772 +0000 UTC m=+34.330876680, 1.005064777s after stability -2025-10-30T18:18:08.052Z [INFO] server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: starting snapshot up to: index=12 -2025-10-30T18:18:08.052Z [INFO] creating new snapshot: path=/tmp/raft3073081136/snapshots/2-12-1761848288052.tmp -2025-10-30T18:18:08.053Z [INFO] server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: snapshot complete up to: index=12 -2025-10-30T18:18:08.055Z [INFO] server-485d23e4-0629-68c4-893b-2d99bc4981e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-485d23e4-0629-68c4-893b-2d99bc4981e7 Address:485d23e4-0629-68c4-893b-2d99bc4981e7} {Suffrage:Voter ID:server-9ebf4678-a528-5952-ef5e-40e366e0542e Address:9ebf4678-a528-5952-ef5e-40e366e0542e} {Suffrage:Voter ID:server-d7338044-b577-8dda-b3bc-fcec970a1452 Address:d7338044-b577-8dda-b3bc-fcec970a1452}]" -2025-10-30T18:18:08.055Z [INFO] server-485d23e4-0629-68c4-893b-2d99bc4981e7: entering follower state: follower="Node at 485d23e4-0629-68c4-893b-2d99bc4981e7 [Follower]" leader-address= leader-id= -2025-10-30T18:18:08.055Z [INFO] server-9ebf4678-a528-5952-ef5e-40e366e0542e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-485d23e4-0629-68c4-893b-2d99bc4981e7 Address:485d23e4-0629-68c4-893b-2d99bc4981e7} {Suffrage:Voter ID:server-9ebf4678-a528-5952-ef5e-40e366e0542e Address:9ebf4678-a528-5952-ef5e-40e366e0542e} {Suffrage:Voter ID:server-d7338044-b577-8dda-b3bc-fcec970a1452 Address:d7338044-b577-8dda-b3bc-fcec970a1452}]" -2025-10-30T18:18:08.055Z [INFO] server-9ebf4678-a528-5952-ef5e-40e366e0542e: entering follower state: follower="Node at 9ebf4678-a528-5952-ef5e-40e366e0542e [Follower]" leader-address= leader-id= -2025-10-30T18:18:08.055Z [INFO] server-d7338044-b577-8dda-b3bc-fcec970a1452: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-485d23e4-0629-68c4-893b-2d99bc4981e7 Address:485d23e4-0629-68c4-893b-2d99bc4981e7} {Suffrage:Voter ID:server-9ebf4678-a528-5952-ef5e-40e366e0542e Address:9ebf4678-a528-5952-ef5e-40e366e0542e} {Suffrage:Voter ID:server-d7338044-b577-8dda-b3bc-fcec970a1452 Address:d7338044-b577-8dda-b3bc-fcec970a1452}]" -2025-10-30T18:18:08.055Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:08.055Z [INFO] server-d7338044-b577-8dda-b3bc-fcec970a1452: entering follower state: follower="Node at d7338044-b577-8dda-b3bc-fcec970a1452 [Follower]" leader-address= leader-id= -2025-10-30T18:18:08.765Z [WARN] server-485d23e4-0629-68c4-893b-2d99bc4981e7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:08.765Z [INFO] server-485d23e4-0629-68c4-893b-2d99bc4981e7: entering candidate state: node="Node at 485d23e4-0629-68c4-893b-2d99bc4981e7 [Candidate]" term=2 -2025-10-30T18:18:08.765Z [INFO] server-485d23e4-0629-68c4-893b-2d99bc4981e7: election won: term=2 tally=2 -2025-10-30T18:18:08.765Z [INFO] server-485d23e4-0629-68c4-893b-2d99bc4981e7: entering leader state: leader="Node at 485d23e4-0629-68c4-893b-2d99bc4981e7 [Leader]" -2025-10-30T18:18:08.765Z [INFO] server-485d23e4-0629-68c4-893b-2d99bc4981e7: added peer, starting replication: peer=server-9ebf4678-a528-5952-ef5e-40e366e0542e -2025-10-30T18:18:08.765Z [INFO] server-485d23e4-0629-68c4-893b-2d99bc4981e7: added peer, starting replication: peer=server-d7338044-b577-8dda-b3bc-fcec970a1452 -2025-10-30T18:18:08.765Z [INFO] server-485d23e4-0629-68c4-893b-2d99bc4981e7: pipelining replication: peer="{Voter server-9ebf4678-a528-5952-ef5e-40e366e0542e 9ebf4678-a528-5952-ef5e-40e366e0542e}" -2025-10-30T18:18:08.765Z [INFO] server-485d23e4-0629-68c4-893b-2d99bc4981e7: pipelining replication: peer="{Voter server-d7338044-b577-8dda-b3bc-fcec970a1452 d7338044-b577-8dda-b3bc-fcec970a1452}" -2025-10-30T18:18:09.771Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:08.765418557 +0000 UTC m=+35.044092470 (1 nodes), 709.832405ms from start of poll, 710.20486ms from cluster start. Timeout at 2025-10-30 18:18:09.770421739 +0000 UTC m=+36.049095636, 1.005003166s after stability -2025-10-30T18:18:09.771Z [INFO] creating new snapshot: path=/tmp/raft1091931536/snapshots/2-14-1761848289771.tmp -2025-10-30T18:18:09.771Z [INFO] server-485d23e4-0629-68c4-893b-2d99bc4981e7: copied to local snapshot: bytes=71 -2025-10-30T18:18:09.771Z [INFO] server-485d23e4-0629-68c4-893b-2d99bc4981e7: snapshot restore progress: id=2-14-1761848289771 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:09.771Z [INFO] server-485d23e4-0629-68c4-893b-2d99bc4981e7: restored user snapshot: index=14 -2025-10-30T18:18:09.771Z [ERROR] server-485d23e4-0629-68c4-893b-2d99bc4981e7: failed to get log: index=3 error="log not found" -2025-10-30T18:18:09.771Z [INFO] server-485d23e4-0629-68c4-893b-2d99bc4981e7: aborting pipeline replication: peer="{Voter server-9ebf4678-a528-5952-ef5e-40e366e0542e 9ebf4678-a528-5952-ef5e-40e366e0542e}" -2025-10-30T18:18:09.771Z [ERROR] server-485d23e4-0629-68c4-893b-2d99bc4981e7: failed to get log: index=3 error="log not found" -2025-10-30T18:18:09.772Z [INFO] server-485d23e4-0629-68c4-893b-2d99bc4981e7: aborting pipeline replication: peer="{Voter server-d7338044-b577-8dda-b3bc-fcec970a1452 d7338044-b577-8dda-b3bc-fcec970a1452}" -2025-10-30T18:18:09.777Z [ERROR] server-485d23e4-0629-68c4-893b-2d99bc4981e7: failed to get log: index=3 error="log not found" -2025-10-30T18:18:09.778Z [INFO] creating new snapshot: path=/tmp/raft3004682100/snapshots/2-14-1761848289778.tmp -2025-10-30T18:18:09.778Z [ERROR] server-485d23e4-0629-68c4-893b-2d99bc4981e7: failed to get log: index=3 error="log not found" -2025-10-30T18:18:09.778Z [INFO] creating new snapshot: path=/tmp/raft1054771965/snapshots/2-14-1761848289778.tmp -2025-10-30T18:18:09.778Z [INFO] server-d7338044-b577-8dda-b3bc-fcec970a1452: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:09.778Z [INFO] server-9ebf4678-a528-5952-ef5e-40e366e0542e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:09.778Z [INFO] server-d7338044-b577-8dda-b3bc-fcec970a1452: copied to local snapshot: bytes=71 -2025-10-30T18:18:09.778Z [INFO] server-9ebf4678-a528-5952-ef5e-40e366e0542e: copied to local snapshot: bytes=71 -2025-10-30T18:18:09.778Z [INFO] server-d7338044-b577-8dda-b3bc-fcec970a1452: snapshot restore progress: id=2-14-1761848289778 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:09.778Z [INFO] server-d7338044-b577-8dda-b3bc-fcec970a1452: Installed remote snapshot -2025-10-30T18:18:09.779Z [INFO] server-9ebf4678-a528-5952-ef5e-40e366e0542e: snapshot restore progress: id=2-14-1761848289778 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:09.780Z [INFO] server-9ebf4678-a528-5952-ef5e-40e366e0542e: Installed remote snapshot -2025-10-30T18:18:09.780Z [INFO] server-485d23e4-0629-68c4-893b-2d99bc4981e7: pipelining replication: peer="{Voter server-d7338044-b577-8dda-b3bc-fcec970a1452 d7338044-b577-8dda-b3bc-fcec970a1452}" -2025-10-30T18:18:09.780Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:18:09.780Z [INFO] server-485d23e4-0629-68c4-893b-2d99bc4981e7: pipelining replication: peer="{Voter server-9ebf4678-a528-5952-ef5e-40e366e0542e 9ebf4678-a528-5952-ef5e-40e366e0542e}" -2025-10-30T18:18:10.785Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:18:09.780150254 +0000 UTC m=+36.058824151 (2 nodes), 1.065µs from start of poll, 1.724936541s from cluster start. Timeout at 2025-10-30 18:18:10.785153921 +0000 UTC m=+37.063827818, 1.005003667s after stability -2025-10-30T18:18:10.796Z [INFO] server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: aborting pipeline replication: peer="{Voter server-4cd8842c-131a-aaec-75a1-e60cd5b8f838 4cd8842c-131a-aaec-75a1-e60cd5b8f838}" -2025-10-30T18:18:10.796Z [INFO] server-8334db8f-9bd0-cbc8-d840-4dd7803f22ef: aborting pipeline replication: peer="{Voter server-e1cb99c7-b8ea-92a4-ee56-8255791394cb e1cb99c7-b8ea-92a4-ee56-8255791394cb}" +2025-10-30T18:21:56.442Z [INFO] server-a001d697-3649-624d-5988-def8a602bd45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a001d697-3649-624d-5988-def8a602bd45 Address:a001d697-3649-624d-5988-def8a602bd45} {Suffrage:Voter ID:server-5710a0f7-96db-ff5b-d991-b4e3de62c5a9 Address:5710a0f7-96db-ff5b-d991-b4e3de62c5a9} {Suffrage:Voter ID:server-77311907-ca18-ca90-bd65-3dfe8fd37e59 Address:77311907-ca18-ca90-bd65-3dfe8fd37e59}]" +2025-10-30T18:21:56.442Z [INFO] server-a001d697-3649-624d-5988-def8a602bd45: entering follower state: follower="Node at a001d697-3649-624d-5988-def8a602bd45 [Follower]" leader-address= leader-id= +2025-10-30T18:21:56.442Z [INFO] server-5710a0f7-96db-ff5b-d991-b4e3de62c5a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a001d697-3649-624d-5988-def8a602bd45 Address:a001d697-3649-624d-5988-def8a602bd45} {Suffrage:Voter ID:server-5710a0f7-96db-ff5b-d991-b4e3de62c5a9 Address:5710a0f7-96db-ff5b-d991-b4e3de62c5a9} {Suffrage:Voter ID:server-77311907-ca18-ca90-bd65-3dfe8fd37e59 Address:77311907-ca18-ca90-bd65-3dfe8fd37e59}]" +2025-10-30T18:21:56.442Z [INFO] server-5710a0f7-96db-ff5b-d991-b4e3de62c5a9: entering follower state: follower="Node at 5710a0f7-96db-ff5b-d991-b4e3de62c5a9 [Follower]" leader-address= leader-id= +2025-10-30T18:21:56.442Z [INFO] server-77311907-ca18-ca90-bd65-3dfe8fd37e59: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a001d697-3649-624d-5988-def8a602bd45 Address:a001d697-3649-624d-5988-def8a602bd45} {Suffrage:Voter ID:server-5710a0f7-96db-ff5b-d991-b4e3de62c5a9 Address:5710a0f7-96db-ff5b-d991-b4e3de62c5a9} {Suffrage:Voter ID:server-77311907-ca18-ca90-bd65-3dfe8fd37e59 Address:77311907-ca18-ca90-bd65-3dfe8fd37e59}]" +2025-10-30T18:21:56.442Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:56.442Z [INFO] server-77311907-ca18-ca90-bd65-3dfe8fd37e59: entering follower state: follower="Node at 77311907-ca18-ca90-bd65-3dfe8fd37e59 [Follower]" leader-address= leader-id= +2025-10-30T18:21:56.956Z [WARN] server-a001d697-3649-624d-5988-def8a602bd45: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:56.956Z [INFO] server-a001d697-3649-624d-5988-def8a602bd45: entering candidate state: node="Node at a001d697-3649-624d-5988-def8a602bd45 [Candidate]" term=2 +2025-10-30T18:21:56.956Z [INFO] server-a001d697-3649-624d-5988-def8a602bd45: election won: term=2 tally=2 +2025-10-30T18:21:56.956Z [INFO] server-a001d697-3649-624d-5988-def8a602bd45: entering leader state: leader="Node at a001d697-3649-624d-5988-def8a602bd45 [Leader]" +2025-10-30T18:21:56.956Z [INFO] server-a001d697-3649-624d-5988-def8a602bd45: added peer, starting replication: peer=server-5710a0f7-96db-ff5b-d991-b4e3de62c5a9 +2025-10-30T18:21:56.956Z [INFO] server-a001d697-3649-624d-5988-def8a602bd45: added peer, starting replication: peer=server-77311907-ca18-ca90-bd65-3dfe8fd37e59 +2025-10-30T18:21:56.956Z [INFO] server-a001d697-3649-624d-5988-def8a602bd45: pipelining replication: peer="{Voter server-77311907-ca18-ca90-bd65-3dfe8fd37e59 77311907-ca18-ca90-bd65-3dfe8fd37e59}" +2025-10-30T18:21:56.956Z [INFO] server-a001d697-3649-624d-5988-def8a602bd45: pipelining replication: peer="{Voter server-5710a0f7-96db-ff5b-d991-b4e3de62c5a9 5710a0f7-96db-ff5b-d991-b4e3de62c5a9}" +2025-10-30T18:21:57.962Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:56.956720704 +0000 UTC m=+40.030250005 (1 nodes), 513.891671ms from start of poll, 514.197643ms from cluster start. Timeout at 2025-10-30 18:21:57.961809852 +0000 UTC m=+41.035339178, 1.005089173s after stability +2025-10-30T18:21:57.963Z [INFO] server-a001d697-3649-624d-5988-def8a602bd45: starting snapshot up to: index=12 +2025-10-30T18:21:57.963Z [INFO] creating new snapshot: path=/tmp/raft2565949190/snapshots/2-12-1761848517963.tmp +2025-10-30T18:21:58.159Z [INFO] server-a001d697-3649-624d-5988-def8a602bd45: snapshot complete up to: index=12 +2025-10-30T18:21:58.160Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d65b504a-8185-4303-efb5-28ea2cf86fac Address:d65b504a-8185-4303-efb5-28ea2cf86fac} {Suffrage:Voter ID:server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18 Address:4c9a84fb-0b48-edaf-fee7-5811a1c40d18} {Suffrage:Voter ID:server-de776368-6add-9f70-7549-04e98aff6559 Address:de776368-6add-9f70-7549-04e98aff6559}]" +2025-10-30T18:21:58.160Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: entering follower state: follower="Node at d65b504a-8185-4303-efb5-28ea2cf86fac [Follower]" leader-address= leader-id= +2025-10-30T18:21:58.160Z [INFO] server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d65b504a-8185-4303-efb5-28ea2cf86fac Address:d65b504a-8185-4303-efb5-28ea2cf86fac} {Suffrage:Voter ID:server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18 Address:4c9a84fb-0b48-edaf-fee7-5811a1c40d18} {Suffrage:Voter ID:server-de776368-6add-9f70-7549-04e98aff6559 Address:de776368-6add-9f70-7549-04e98aff6559}]" +2025-10-30T18:21:58.160Z [INFO] server-de776368-6add-9f70-7549-04e98aff6559: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d65b504a-8185-4303-efb5-28ea2cf86fac Address:d65b504a-8185-4303-efb5-28ea2cf86fac} {Suffrage:Voter ID:server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18 Address:4c9a84fb-0b48-edaf-fee7-5811a1c40d18} {Suffrage:Voter ID:server-de776368-6add-9f70-7549-04e98aff6559 Address:de776368-6add-9f70-7549-04e98aff6559}]" +2025-10-30T18:21:58.160Z [INFO] server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18: entering follower state: follower="Node at 4c9a84fb-0b48-edaf-fee7-5811a1c40d18 [Follower]" leader-address= leader-id= +2025-10-30T18:21:58.160Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:21:58.160Z [INFO] server-de776368-6add-9f70-7549-04e98aff6559: entering follower state: follower="Node at de776368-6add-9f70-7549-04e98aff6559 [Follower]" leader-address= leader-id= +2025-10-30T18:21:58.724Z [WARN] server-d65b504a-8185-4303-efb5-28ea2cf86fac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:21:58.724Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: entering candidate state: node="Node at d65b504a-8185-4303-efb5-28ea2cf86fac [Candidate]" term=2 +2025-10-30T18:21:58.724Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: election won: term=2 tally=2 +2025-10-30T18:21:58.724Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: entering leader state: leader="Node at d65b504a-8185-4303-efb5-28ea2cf86fac [Leader]" +2025-10-30T18:21:58.724Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: added peer, starting replication: peer=server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18 +2025-10-30T18:21:58.724Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: added peer, starting replication: peer=server-de776368-6add-9f70-7549-04e98aff6559 +2025-10-30T18:21:58.724Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: pipelining replication: peer="{Voter server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18 4c9a84fb-0b48-edaf-fee7-5811a1c40d18}" +2025-10-30T18:21:58.724Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: pipelining replication: peer="{Voter server-de776368-6add-9f70-7549-04e98aff6559 de776368-6add-9f70-7549-04e98aff6559}" +2025-10-30T18:21:59.730Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:21:58.724827838 +0000 UTC m=+41.798357140 (1 nodes), 563.846796ms from start of poll, 564.130644ms from cluster start. Timeout at 2025-10-30 18:21:59.729837786 +0000 UTC m=+42.803367112, 1.005009972s after stability +2025-10-30T18:21:59.808Z [INFO] creating new snapshot: path=/tmp/raft1696030174/snapshots/2-14-1761848519808.tmp +2025-10-30T18:21:59.808Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: copied to local snapshot: bytes=71 +2025-10-30T18:21:59.808Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: snapshot restore progress: id=2-14-1761848519808 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:59.808Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: restored user snapshot: index=14 +2025-10-30T18:21:59.808Z [ERROR] server-d65b504a-8185-4303-efb5-28ea2cf86fac: failed to get log: index=3 error="log not found" +2025-10-30T18:21:59.809Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: aborting pipeline replication: peer="{Voter server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18 4c9a84fb-0b48-edaf-fee7-5811a1c40d18}" +2025-10-30T18:21:59.808Z [ERROR] server-d65b504a-8185-4303-efb5-28ea2cf86fac: failed to get log: index=3 error="log not found" +2025-10-30T18:21:59.809Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: aborting pipeline replication: peer="{Voter server-de776368-6add-9f70-7549-04e98aff6559 de776368-6add-9f70-7549-04e98aff6559}" +2025-10-30T18:21:59.818Z [ERROR] server-d65b504a-8185-4303-efb5-28ea2cf86fac: failed to get log: index=3 error="log not found" +2025-10-30T18:21:59.818Z [ERROR] server-d65b504a-8185-4303-efb5-28ea2cf86fac: failed to get log: index=3 error="log not found" +2025-10-30T18:21:59.819Z [INFO] creating new snapshot: path=/tmp/raft1045315568/snapshots/2-14-1761848519819.tmp +2025-10-30T18:21:59.819Z [INFO] creating new snapshot: path=/tmp/raft1525406454/snapshots/2-14-1761848519819.tmp +2025-10-30T18:21:59.819Z [INFO] server-de776368-6add-9f70-7549-04e98aff6559: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:59.819Z [INFO] server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:59.820Z [INFO] server-de776368-6add-9f70-7549-04e98aff6559: copied to local snapshot: bytes=71 +2025-10-30T18:21:59.820Z [INFO] server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18: copied to local snapshot: bytes=71 +2025-10-30T18:21:59.820Z [INFO] server-de776368-6add-9f70-7549-04e98aff6559: snapshot restore progress: id=2-14-1761848519819 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:59.820Z [INFO] server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18: snapshot restore progress: id=2-14-1761848519819 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:21:59.820Z [INFO] server-de776368-6add-9f70-7549-04e98aff6559: Installed remote snapshot +2025-10-30T18:21:59.820Z [INFO] server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18: Installed remote snapshot +2025-10-30T18:21:59.820Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: pipelining replication: peer="{Voter server-de776368-6add-9f70-7549-04e98aff6559 de776368-6add-9f70-7549-04e98aff6559}" +2025-10-30T18:21:59.820Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:21:59.820Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: pipelining replication: peer="{Voter server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18 4c9a84fb-0b48-edaf-fee7-5811a1c40d18}" +2025-10-30T18:22:00.825Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:21:59.820518132 +0000 UTC m=+42.894047431 (2 nodes), 825ns from start of poll, 1.659820935s from cluster start. Timeout at 2025-10-30 18:22:00.825531249 +0000 UTC m=+43.899060574, 1.005013143s after stability +2025-10-30T18:22:00.839Z [INFO] server-a001d697-3649-624d-5988-def8a602bd45: aborting pipeline replication: peer="{Voter server-77311907-ca18-ca90-bd65-3dfe8fd37e59 77311907-ca18-ca90-bd65-3dfe8fd37e59}" +2025-10-30T18:22:00.839Z [INFO] server-a001d697-3649-624d-5988-def8a602bd45: aborting pipeline replication: peer="{Voter server-5710a0f7-96db-ff5b-d991-b4e3de62c5a9 5710a0f7-96db-ff5b-d991-b4e3de62c5a9}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true -2025-10-30T18:18:10.797Z [INFO] server-c2fa9d15-4939-de29-8cf8-0304635b83b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2fa9d15-4939-de29-8cf8-0304635b83b8 Address:c2fa9d15-4939-de29-8cf8-0304635b83b8} {Suffrage:Voter ID:server-ec908aba-30e1-0112-6c5a-573d8880cc2d Address:ec908aba-30e1-0112-6c5a-573d8880cc2d} {Suffrage:Voter ID:server-8403c93b-15bc-75c0-dea7-22426435b181 Address:8403c93b-15bc-75c0-dea7-22426435b181}]" -2025-10-30T18:18:10.797Z [INFO] server-ec908aba-30e1-0112-6c5a-573d8880cc2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2fa9d15-4939-de29-8cf8-0304635b83b8 Address:c2fa9d15-4939-de29-8cf8-0304635b83b8} {Suffrage:Voter ID:server-ec908aba-30e1-0112-6c5a-573d8880cc2d Address:ec908aba-30e1-0112-6c5a-573d8880cc2d} {Suffrage:Voter ID:server-8403c93b-15bc-75c0-dea7-22426435b181 Address:8403c93b-15bc-75c0-dea7-22426435b181}]" -2025-10-30T18:18:10.797Z [INFO] server-c2fa9d15-4939-de29-8cf8-0304635b83b8: entering follower state: follower="Node at c2fa9d15-4939-de29-8cf8-0304635b83b8 [Follower]" leader-address= leader-id= -2025-10-30T18:18:10.798Z [INFO] server-8403c93b-15bc-75c0-dea7-22426435b181: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2fa9d15-4939-de29-8cf8-0304635b83b8 Address:c2fa9d15-4939-de29-8cf8-0304635b83b8} {Suffrage:Voter ID:server-ec908aba-30e1-0112-6c5a-573d8880cc2d Address:ec908aba-30e1-0112-6c5a-573d8880cc2d} {Suffrage:Voter ID:server-8403c93b-15bc-75c0-dea7-22426435b181 Address:8403c93b-15bc-75c0-dea7-22426435b181}]" -2025-10-30T18:18:10.798Z [INFO] server-ec908aba-30e1-0112-6c5a-573d8880cc2d: entering follower state: follower="Node at ec908aba-30e1-0112-6c5a-573d8880cc2d [Follower]" leader-address= leader-id= -2025-10-30T18:18:10.798Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:10.798Z [INFO] server-8403c93b-15bc-75c0-dea7-22426435b181: entering follower state: follower="Node at 8403c93b-15bc-75c0-dea7-22426435b181 [Follower]" leader-address= leader-id= -2025-10-30T18:18:11.309Z [WARN] server-ec908aba-30e1-0112-6c5a-573d8880cc2d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:11.309Z [INFO] server-ec908aba-30e1-0112-6c5a-573d8880cc2d: entering candidate state: node="Node at ec908aba-30e1-0112-6c5a-573d8880cc2d [Candidate]" term=2 -2025-10-30T18:18:11.309Z [INFO] server-ec908aba-30e1-0112-6c5a-573d8880cc2d: election won: term=2 tally=2 -2025-10-30T18:18:11.309Z [INFO] server-ec908aba-30e1-0112-6c5a-573d8880cc2d: entering leader state: leader="Node at ec908aba-30e1-0112-6c5a-573d8880cc2d [Leader]" -2025-10-30T18:18:11.309Z [INFO] server-ec908aba-30e1-0112-6c5a-573d8880cc2d: added peer, starting replication: peer=server-c2fa9d15-4939-de29-8cf8-0304635b83b8 -2025-10-30T18:18:11.309Z [INFO] server-ec908aba-30e1-0112-6c5a-573d8880cc2d: added peer, starting replication: peer=server-8403c93b-15bc-75c0-dea7-22426435b181 -2025-10-30T18:18:11.309Z [INFO] server-ec908aba-30e1-0112-6c5a-573d8880cc2d: pipelining replication: peer="{Voter server-8403c93b-15bc-75c0-dea7-22426435b181 8403c93b-15bc-75c0-dea7-22426435b181}" -2025-10-30T18:18:11.309Z [INFO] server-ec908aba-30e1-0112-6c5a-573d8880cc2d: pipelining replication: peer="{Voter server-c2fa9d15-4939-de29-8cf8-0304635b83b8 c2fa9d15-4939-de29-8cf8-0304635b83b8}" -2025-10-30T18:18:12.315Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:11.309637834 +0000 UTC m=+37.588311733 (1 nodes), 511.581109ms from start of poll, 511.808744ms from cluster start. Timeout at 2025-10-30 18:18:12.314642303 +0000 UTC m=+38.593316203, 1.00500447s after stability -2025-10-30T18:18:12.315Z [INFO] server-ec908aba-30e1-0112-6c5a-573d8880cc2d: starting snapshot up to: index=12 -2025-10-30T18:18:12.315Z [INFO] creating new snapshot: path=/tmp/raft1662924434/snapshots/2-12-1761848292315.tmp -2025-10-30T18:18:12.316Z [INFO] server-ec908aba-30e1-0112-6c5a-573d8880cc2d: snapshot complete up to: index=12 -2025-10-30T18:18:12.316Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2 Address:e3d4ef1d-3da5-6ccf-031c-f9401887edf2} {Suffrage:Voter ID:server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c Address:c3206cd3-2dd0-6fe0-0091-3e917e94ce2c} {Suffrage:Voter ID:server-410c7af0-44b1-9b2a-d251-cb3c0583806f Address:410c7af0-44b1-9b2a-d251-cb3c0583806f}]" -2025-10-30T18:18:12.316Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: entering follower state: follower="Node at e3d4ef1d-3da5-6ccf-031c-f9401887edf2 [Follower]" leader-address= leader-id= -2025-10-30T18:18:12.316Z [INFO] server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2 Address:e3d4ef1d-3da5-6ccf-031c-f9401887edf2} {Suffrage:Voter ID:server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c Address:c3206cd3-2dd0-6fe0-0091-3e917e94ce2c} {Suffrage:Voter ID:server-410c7af0-44b1-9b2a-d251-cb3c0583806f Address:410c7af0-44b1-9b2a-d251-cb3c0583806f}]" -2025-10-30T18:18:12.317Z [INFO] server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c: entering follower state: follower="Node at c3206cd3-2dd0-6fe0-0091-3e917e94ce2c [Follower]" leader-address= leader-id= -2025-10-30T18:18:12.317Z [INFO] server-410c7af0-44b1-9b2a-d251-cb3c0583806f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2 Address:e3d4ef1d-3da5-6ccf-031c-f9401887edf2} {Suffrage:Voter ID:server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c Address:c3206cd3-2dd0-6fe0-0091-3e917e94ce2c} {Suffrage:Voter ID:server-410c7af0-44b1-9b2a-d251-cb3c0583806f Address:410c7af0-44b1-9b2a-d251-cb3c0583806f}]" -2025-10-30T18:18:12.317Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:12.317Z [INFO] server-410c7af0-44b1-9b2a-d251-cb3c0583806f: entering follower state: follower="Node at 410c7af0-44b1-9b2a-d251-cb3c0583806f [Follower]" leader-address= leader-id= -2025-10-30T18:18:12.843Z [WARN] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:12.843Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: entering candidate state: node="Node at e3d4ef1d-3da5-6ccf-031c-f9401887edf2 [Candidate]" term=2 -2025-10-30T18:18:12.843Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: election won: term=2 tally=2 -2025-10-30T18:18:12.843Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: entering leader state: leader="Node at e3d4ef1d-3da5-6ccf-031c-f9401887edf2 [Leader]" -2025-10-30T18:18:12.843Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: added peer, starting replication: peer=server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c -2025-10-30T18:18:12.843Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: added peer, starting replication: peer=server-410c7af0-44b1-9b2a-d251-cb3c0583806f -2025-10-30T18:18:12.843Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: pipelining replication: peer="{Voter server-410c7af0-44b1-9b2a-d251-cb3c0583806f 410c7af0-44b1-9b2a-d251-cb3c0583806f}" -2025-10-30T18:18:12.843Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: pipelining replication: peer="{Voter server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c c3206cd3-2dd0-6fe0-0091-3e917e94ce2c}" -2025-10-30T18:18:13.849Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:12.843229667 +0000 UTC m=+39.121903564 (1 nodes), 526.046674ms from start of poll, 526.378551ms from cluster start. Timeout at 2025-10-30 18:18:13.848231739 +0000 UTC m=+40.126905638, 1.005002074s after stability -2025-10-30T18:18:13.849Z [INFO] creating new snapshot: path=/tmp/raft1194384435/snapshots/2-14-1761848293849.tmp -2025-10-30T18:18:13.849Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: copied to local snapshot: bytes=71 -2025-10-30T18:18:13.850Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: snapshot restore progress: id=2-14-1761848293849 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:13.850Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: removing all old logs from log store -2025-10-30T18:18:13.850Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: compacting logs: from=1 to=2 -2025-10-30T18:18:13.850Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: restored user snapshot: index=14 -2025-10-30T18:18:13.850Z [ERROR] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: failed to get log: index=2 error="log not found" -2025-10-30T18:18:13.850Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: aborting pipeline replication: peer="{Voter server-410c7af0-44b1-9b2a-d251-cb3c0583806f 410c7af0-44b1-9b2a-d251-cb3c0583806f}" -2025-10-30T18:18:13.850Z [ERROR] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: failed to get log: index=2 error="log not found" -2025-10-30T18:18:13.850Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: aborting pipeline replication: peer="{Voter server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c c3206cd3-2dd0-6fe0-0091-3e917e94ce2c}" -2025-10-30T18:18:13.855Z [ERROR] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: failed to get log: index=2 error="log not found" -2025-10-30T18:18:13.855Z [INFO] creating new snapshot: path=/tmp/raft1098516377/snapshots/2-14-1761848293855.tmp -2025-10-30T18:18:13.856Z [INFO] server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:13.856Z [INFO] server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c: copied to local snapshot: bytes=71 -2025-10-30T18:18:13.856Z [INFO] server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c: snapshot restore progress: id=2-14-1761848293855 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:13.856Z [INFO] server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c: removing all old logs from log store -2025-10-30T18:18:13.856Z [INFO] server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c: compacting logs: from=1 to=2 -2025-10-30T18:18:13.856Z [INFO] server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c: Installed remote snapshot -2025-10-30T18:18:13.856Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: pipelining replication: peer="{Voter server-c3206cd3-2dd0-6fe0-0091-3e917e94ce2c c3206cd3-2dd0-6fe0-0091-3e917e94ce2c}" -2025-10-30T18:18:13.856Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:18:13.859Z [ERROR] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: failed to get log: index=2 error="log not found" -2025-10-30T18:18:13.859Z [INFO] creating new snapshot: path=/tmp/raft1159472978/snapshots/2-14-1761848293859.tmp -2025-10-30T18:18:13.859Z [INFO] server-410c7af0-44b1-9b2a-d251-cb3c0583806f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:13.859Z [INFO] server-410c7af0-44b1-9b2a-d251-cb3c0583806f: copied to local snapshot: bytes=71 -2025-10-30T18:18:13.859Z [INFO] server-410c7af0-44b1-9b2a-d251-cb3c0583806f: snapshot restore progress: id=2-14-1761848293859 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:13.860Z [INFO] server-410c7af0-44b1-9b2a-d251-cb3c0583806f: removing all old logs from log store -2025-10-30T18:18:13.860Z [INFO] server-410c7af0-44b1-9b2a-d251-cb3c0583806f: compacting logs: from=1 to=2 -2025-10-30T18:18:13.860Z [INFO] server-410c7af0-44b1-9b2a-d251-cb3c0583806f: Installed remote snapshot -2025-10-30T18:18:13.860Z [INFO] server-e3d4ef1d-3da5-6ccf-031c-f9401887edf2: pipelining replication: peer="{Voter server-410c7af0-44b1-9b2a-d251-cb3c0583806f 410c7af0-44b1-9b2a-d251-cb3c0583806f}" -2025-10-30T18:18:14.862Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:18:13.856723201 +0000 UTC m=+40.135397110 (2 nodes), 784ns from start of poll, 1.539872097s from cluster start. Timeout at 2025-10-30 18:18:14.861737512 +0000 UTC m=+41.140411418, 1.005014308s after stability -2025-10-30T18:18:14.873Z [INFO] server-ec908aba-30e1-0112-6c5a-573d8880cc2d: aborting pipeline replication: peer="{Voter server-8403c93b-15bc-75c0-dea7-22426435b181 8403c93b-15bc-75c0-dea7-22426435b181}" -2025-10-30T18:18:14.873Z [INFO] server-ec908aba-30e1-0112-6c5a-573d8880cc2d: aborting pipeline replication: peer="{Voter server-c2fa9d15-4939-de29-8cf8-0304635b83b8 c2fa9d15-4939-de29-8cf8-0304635b83b8}" +2025-10-30T18:22:00.843Z [INFO] server-e84b7966-6a30-e3d9-7748-c77082e7c396: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e84b7966-6a30-e3d9-7748-c77082e7c396 Address:e84b7966-6a30-e3d9-7748-c77082e7c396} {Suffrage:Voter ID:server-201afbba-4c18-be52-9ec3-4018f7b8ee85 Address:201afbba-4c18-be52-9ec3-4018f7b8ee85} {Suffrage:Voter ID:server-db48f41f-7f95-6e90-155c-8db0687bb4c9 Address:db48f41f-7f95-6e90-155c-8db0687bb4c9}]" +2025-10-30T18:22:00.843Z [INFO] server-e84b7966-6a30-e3d9-7748-c77082e7c396: entering follower state: follower="Node at e84b7966-6a30-e3d9-7748-c77082e7c396 [Follower]" leader-address= leader-id= +2025-10-30T18:22:00.843Z [INFO] server-201afbba-4c18-be52-9ec3-4018f7b8ee85: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e84b7966-6a30-e3d9-7748-c77082e7c396 Address:e84b7966-6a30-e3d9-7748-c77082e7c396} {Suffrage:Voter ID:server-201afbba-4c18-be52-9ec3-4018f7b8ee85 Address:201afbba-4c18-be52-9ec3-4018f7b8ee85} {Suffrage:Voter ID:server-db48f41f-7f95-6e90-155c-8db0687bb4c9 Address:db48f41f-7f95-6e90-155c-8db0687bb4c9}]" +2025-10-30T18:22:00.843Z [INFO] server-201afbba-4c18-be52-9ec3-4018f7b8ee85: entering follower state: follower="Node at 201afbba-4c18-be52-9ec3-4018f7b8ee85 [Follower]" leader-address= leader-id= +2025-10-30T18:22:00.843Z [INFO] server-db48f41f-7f95-6e90-155c-8db0687bb4c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e84b7966-6a30-e3d9-7748-c77082e7c396 Address:e84b7966-6a30-e3d9-7748-c77082e7c396} {Suffrage:Voter ID:server-201afbba-4c18-be52-9ec3-4018f7b8ee85 Address:201afbba-4c18-be52-9ec3-4018f7b8ee85} {Suffrage:Voter ID:server-db48f41f-7f95-6e90-155c-8db0687bb4c9 Address:db48f41f-7f95-6e90-155c-8db0687bb4c9}]" +2025-10-30T18:22:00.843Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:00.843Z [INFO] server-db48f41f-7f95-6e90-155c-8db0687bb4c9: entering follower state: follower="Node at db48f41f-7f95-6e90-155c-8db0687bb4c9 [Follower]" leader-address= leader-id= +2025-10-30T18:22:01.693Z [WARN] server-db48f41f-7f95-6e90-155c-8db0687bb4c9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:01.693Z [INFO] server-db48f41f-7f95-6e90-155c-8db0687bb4c9: entering candidate state: node="Node at db48f41f-7f95-6e90-155c-8db0687bb4c9 [Candidate]" term=2 +2025-10-30T18:22:01.693Z [INFO] server-db48f41f-7f95-6e90-155c-8db0687bb4c9: election won: term=2 tally=2 +2025-10-30T18:22:01.693Z [INFO] server-db48f41f-7f95-6e90-155c-8db0687bb4c9: entering leader state: leader="Node at db48f41f-7f95-6e90-155c-8db0687bb4c9 [Leader]" +2025-10-30T18:22:01.693Z [INFO] server-db48f41f-7f95-6e90-155c-8db0687bb4c9: added peer, starting replication: peer=server-e84b7966-6a30-e3d9-7748-c77082e7c396 +2025-10-30T18:22:01.693Z [INFO] server-db48f41f-7f95-6e90-155c-8db0687bb4c9: added peer, starting replication: peer=server-201afbba-4c18-be52-9ec3-4018f7b8ee85 +2025-10-30T18:22:01.693Z [INFO] server-db48f41f-7f95-6e90-155c-8db0687bb4c9: pipelining replication: peer="{Voter server-201afbba-4c18-be52-9ec3-4018f7b8ee85 201afbba-4c18-be52-9ec3-4018f7b8ee85}" +2025-10-30T18:22:01.693Z [INFO] server-db48f41f-7f95-6e90-155c-8db0687bb4c9: pipelining replication: peer="{Voter server-e84b7966-6a30-e3d9-7748-c77082e7c396 e84b7966-6a30-e3d9-7748-c77082e7c396}" +2025-10-30T18:22:02.699Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:01.69356383 +0000 UTC m=+44.767093131 (1 nodes), 850.367097ms from start of poll, 850.644783ms from cluster start. Timeout at 2025-10-30 18:22:02.698564764 +0000 UTC m=+45.772094056, 1.005000925s after stability +2025-10-30T18:22:02.699Z [INFO] server-db48f41f-7f95-6e90-155c-8db0687bb4c9: starting snapshot up to: index=12 +2025-10-30T18:22:02.700Z [INFO] creating new snapshot: path=/tmp/raft2942306581/snapshots/2-12-1761848522700.tmp +2025-10-30T18:22:02.789Z [INFO] server-db48f41f-7f95-6e90-155c-8db0687bb4c9: snapshot complete up to: index=12 +2025-10-30T18:22:02.824Z [INFO] server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f Address:c7faca8d-37f3-a29f-98a5-7cd3a729c05f} {Suffrage:Voter ID:server-747d9665-bec4-273d-3713-17b0ce5a7a84 Address:747d9665-bec4-273d-3713-17b0ce5a7a84} {Suffrage:Voter ID:server-f583ef02-c74d-37b9-ed32-79a7e7e24ced Address:f583ef02-c74d-37b9-ed32-79a7e7e24ced}]" +2025-10-30T18:22:02.824Z [INFO] server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f: entering follower state: follower="Node at c7faca8d-37f3-a29f-98a5-7cd3a729c05f [Follower]" leader-address= leader-id= +2025-10-30T18:22:02.824Z [INFO] server-747d9665-bec4-273d-3713-17b0ce5a7a84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f Address:c7faca8d-37f3-a29f-98a5-7cd3a729c05f} {Suffrage:Voter ID:server-747d9665-bec4-273d-3713-17b0ce5a7a84 Address:747d9665-bec4-273d-3713-17b0ce5a7a84} {Suffrage:Voter ID:server-f583ef02-c74d-37b9-ed32-79a7e7e24ced Address:f583ef02-c74d-37b9-ed32-79a7e7e24ced}]" +2025-10-30T18:22:02.824Z [INFO] server-747d9665-bec4-273d-3713-17b0ce5a7a84: entering follower state: follower="Node at 747d9665-bec4-273d-3713-17b0ce5a7a84 [Follower]" leader-address= leader-id= +2025-10-30T18:22:02.824Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f Address:c7faca8d-37f3-a29f-98a5-7cd3a729c05f} {Suffrage:Voter ID:server-747d9665-bec4-273d-3713-17b0ce5a7a84 Address:747d9665-bec4-273d-3713-17b0ce5a7a84} {Suffrage:Voter ID:server-f583ef02-c74d-37b9-ed32-79a7e7e24ced Address:f583ef02-c74d-37b9-ed32-79a7e7e24ced}]" +2025-10-30T18:22:02.824Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:02.824Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: entering follower state: follower="Node at f583ef02-c74d-37b9-ed32-79a7e7e24ced [Follower]" leader-address= leader-id= +2025-10-30T18:22:03.390Z [WARN] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:03.390Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: entering candidate state: node="Node at f583ef02-c74d-37b9-ed32-79a7e7e24ced [Candidate]" term=2 +2025-10-30T18:22:03.390Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: election won: term=2 tally=2 +2025-10-30T18:22:03.390Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: entering leader state: leader="Node at f583ef02-c74d-37b9-ed32-79a7e7e24ced [Leader]" +2025-10-30T18:22:03.390Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: added peer, starting replication: peer=server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f +2025-10-30T18:22:03.390Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: added peer, starting replication: peer=server-747d9665-bec4-273d-3713-17b0ce5a7a84 +2025-10-30T18:22:03.391Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: pipelining replication: peer="{Voter server-747d9665-bec4-273d-3713-17b0ce5a7a84 747d9665-bec4-273d-3713-17b0ce5a7a84}" +2025-10-30T18:22:03.391Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: pipelining replication: peer="{Voter server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f c7faca8d-37f3-a29f-98a5-7cd3a729c05f}" +2025-10-30T18:22:04.396Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:03.390940177 +0000 UTC m=+46.464469470 (1 nodes), 566.388132ms from start of poll, 566.824664ms from cluster start. Timeout at 2025-10-30 18:22:04.395945056 +0000 UTC m=+47.469474349, 1.005004879s after stability +2025-10-30T18:22:04.397Z [INFO] creating new snapshot: path=/tmp/raft3773407307/snapshots/2-14-1761848524397.tmp +2025-10-30T18:22:04.398Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: copied to local snapshot: bytes=71 +2025-10-30T18:22:04.398Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: snapshot restore progress: id=2-14-1761848524397 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:04.398Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: removing all old logs from log store +2025-10-30T18:22:04.398Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: compacting logs: from=1 to=2 +2025-10-30T18:22:04.398Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: restored user snapshot: index=14 +2025-10-30T18:22:04.398Z [ERROR] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: failed to get log: index=2 error="log not found" +2025-10-30T18:22:04.398Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: aborting pipeline replication: peer="{Voter server-747d9665-bec4-273d-3713-17b0ce5a7a84 747d9665-bec4-273d-3713-17b0ce5a7a84}" +2025-10-30T18:22:04.398Z [ERROR] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: failed to get log: index=2 error="log not found" +2025-10-30T18:22:04.398Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: aborting pipeline replication: peer="{Voter server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f c7faca8d-37f3-a29f-98a5-7cd3a729c05f}" +2025-10-30T18:22:04.405Z [ERROR] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: failed to get log: index=2 error="log not found" +2025-10-30T18:22:04.406Z [INFO] creating new snapshot: path=/tmp/raft2308454295/snapshots/2-14-1761848524406.tmp +2025-10-30T18:22:04.406Z [INFO] server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:04.406Z [INFO] server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f: copied to local snapshot: bytes=71 +2025-10-30T18:22:04.406Z [INFO] server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f: snapshot restore progress: id=2-14-1761848524406 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:04.406Z [INFO] server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f: removing all old logs from log store +2025-10-30T18:22:04.406Z [INFO] server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f: compacting logs: from=1 to=2 +2025-10-30T18:22:04.407Z [INFO] server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f: Installed remote snapshot +2025-10-30T18:22:04.407Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: pipelining replication: peer="{Voter server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f c7faca8d-37f3-a29f-98a5-7cd3a729c05f}" +2025-10-30T18:22:04.407Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:22:04.408Z [ERROR] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: failed to get log: index=2 error="log not found" +2025-10-30T18:22:04.408Z [INFO] creating new snapshot: path=/tmp/raft2011871050/snapshots/2-14-1761848524408.tmp +2025-10-30T18:22:04.408Z [INFO] server-747d9665-bec4-273d-3713-17b0ce5a7a84: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:04.409Z [INFO] server-747d9665-bec4-273d-3713-17b0ce5a7a84: copied to local snapshot: bytes=71 +2025-10-30T18:22:04.409Z [INFO] server-747d9665-bec4-273d-3713-17b0ce5a7a84: snapshot restore progress: id=2-14-1761848524408 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:04.409Z [INFO] server-747d9665-bec4-273d-3713-17b0ce5a7a84: removing all old logs from log store +2025-10-30T18:22:04.409Z [INFO] server-747d9665-bec4-273d-3713-17b0ce5a7a84: compacting logs: from=1 to=2 +2025-10-30T18:22:04.409Z [INFO] server-747d9665-bec4-273d-3713-17b0ce5a7a84: Installed remote snapshot +2025-10-30T18:22:04.409Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: pipelining replication: peer="{Voter server-747d9665-bec4-273d-3713-17b0ce5a7a84 747d9665-bec4-273d-3713-17b0ce5a7a84}" +2025-10-30T18:22:05.412Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:22:04.407143516 +0000 UTC m=+47.480672813 (2 nodes), 629ns from start of poll, 1.583028007s from cluster start. Timeout at 2025-10-30 18:22:05.412153503 +0000 UTC m=+48.485682814, 1.005010001s after stability +2025-10-30T18:22:05.425Z [INFO] server-db48f41f-7f95-6e90-155c-8db0687bb4c9: aborting pipeline replication: peer="{Voter server-e84b7966-6a30-e3d9-7748-c77082e7c396 e84b7966-6a30-e3d9-7748-c77082e7c396}" +2025-10-30T18:22:05.425Z [INFO] server-db48f41f-7f95-6e90-155c-8db0687bb4c9: aborting pipeline replication: peer="{Voter server-201afbba-4c18-be52-9ec3-4018f7b8ee85 201afbba-4c18-be52-9ec3-4018f7b8ee85}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false -2025-10-30T18:18:14.874Z [INFO] server-8a11aa13-229c-7a1e-5a97-6d583dc3914a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a11aa13-229c-7a1e-5a97-6d583dc3914a Address:8a11aa13-229c-7a1e-5a97-6d583dc3914a} {Suffrage:Voter ID:server-8bd36cd1-dce9-28b1-169e-ae212f20b997 Address:8bd36cd1-dce9-28b1-169e-ae212f20b997} {Suffrage:Voter ID:server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600 Address:e410ad3b-cb93-be72-6fe1-e17c4ddb6600}]" -2025-10-30T18:18:14.874Z [INFO] server-8a11aa13-229c-7a1e-5a97-6d583dc3914a: entering follower state: follower="Node at 8a11aa13-229c-7a1e-5a97-6d583dc3914a [Follower]" leader-address= leader-id= -2025-10-30T18:18:14.874Z [INFO] server-8bd36cd1-dce9-28b1-169e-ae212f20b997: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a11aa13-229c-7a1e-5a97-6d583dc3914a Address:8a11aa13-229c-7a1e-5a97-6d583dc3914a} {Suffrage:Voter ID:server-8bd36cd1-dce9-28b1-169e-ae212f20b997 Address:8bd36cd1-dce9-28b1-169e-ae212f20b997} {Suffrage:Voter ID:server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600 Address:e410ad3b-cb93-be72-6fe1-e17c4ddb6600}]" -2025-10-30T18:18:14.874Z [INFO] server-8bd36cd1-dce9-28b1-169e-ae212f20b997: entering follower state: follower="Node at 8bd36cd1-dce9-28b1-169e-ae212f20b997 [Follower]" leader-address= leader-id= -2025-10-30T18:18:14.874Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a11aa13-229c-7a1e-5a97-6d583dc3914a Address:8a11aa13-229c-7a1e-5a97-6d583dc3914a} {Suffrage:Voter ID:server-8bd36cd1-dce9-28b1-169e-ae212f20b997 Address:8bd36cd1-dce9-28b1-169e-ae212f20b997} {Suffrage:Voter ID:server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600 Address:e410ad3b-cb93-be72-6fe1-e17c4ddb6600}]" -2025-10-30T18:18:14.874Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:14.874Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: entering follower state: follower="Node at e410ad3b-cb93-be72-6fe1-e17c4ddb6600 [Follower]" leader-address= leader-id= -2025-10-30T18:18:15.550Z [WARN] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:15.550Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: entering candidate state: node="Node at e410ad3b-cb93-be72-6fe1-e17c4ddb6600 [Candidate]" term=2 -2025-10-30T18:18:15.550Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: election won: term=2 tally=2 -2025-10-30T18:18:15.550Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: entering leader state: leader="Node at e410ad3b-cb93-be72-6fe1-e17c4ddb6600 [Leader]" -2025-10-30T18:18:15.550Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: added peer, starting replication: peer=server-8a11aa13-229c-7a1e-5a97-6d583dc3914a -2025-10-30T18:18:15.550Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: added peer, starting replication: peer=server-8bd36cd1-dce9-28b1-169e-ae212f20b997 -2025-10-30T18:18:15.550Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: pipelining replication: peer="{Voter server-8bd36cd1-dce9-28b1-169e-ae212f20b997 8bd36cd1-dce9-28b1-169e-ae212f20b997}" -2025-10-30T18:18:15.550Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: pipelining replication: peer="{Voter server-8a11aa13-229c-7a1e-5a97-6d583dc3914a 8a11aa13-229c-7a1e-5a97-6d583dc3914a}" -2025-10-30T18:18:16.556Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:15.550426788 +0000 UTC m=+41.829100684 (1 nodes), 675.82852ms from start of poll, 676.145288ms from cluster start. Timeout at 2025-10-30 18:18:16.555434062 +0000 UTC m=+42.834107959, 1.005007275s after stability -2025-10-30T18:18:16.556Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: starting snapshot up to: index=12 -2025-10-30T18:18:16.556Z [INFO] creating new snapshot: path=/tmp/raft3053875945/snapshots/2-12-1761848296556.tmp -2025-10-30T18:18:16.557Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: snapshot complete up to: index=12 -2025-10-30T18:18:16.557Z [INFO] creating new snapshot: path=/tmp/raft3053875945/snapshots/2-23-1761848296557.tmp -2025-10-30T18:18:16.558Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: copied to local snapshot: bytes=71 -2025-10-30T18:18:16.558Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: snapshot restore progress: id=2-23-1761848296557 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:16.558Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: restored user snapshot: index=23 -2025-10-30T18:18:16.558Z [ERROR] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: failed to get log: index=23 error="log not found" -2025-10-30T18:18:16.558Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: aborting pipeline replication: peer="{Voter server-8bd36cd1-dce9-28b1-169e-ae212f20b997 8bd36cd1-dce9-28b1-169e-ae212f20b997}" -2025-10-30T18:18:16.558Z [ERROR] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: failed to get log: index=23 error="log not found" -2025-10-30T18:18:16.558Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: aborting pipeline replication: peer="{Voter server-8a11aa13-229c-7a1e-5a97-6d583dc3914a 8a11aa13-229c-7a1e-5a97-6d583dc3914a}" -2025-10-30T18:18:16.568Z [ERROR] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: failed to get log: index=23 error="log not found" -2025-10-30T18:18:16.568Z [ERROR] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: failed to get log: index=23 error="log not found" -2025-10-30T18:18:16.568Z [INFO] creating new snapshot: path=/tmp/raft4159777038/snapshots/2-23-1761848296568.tmp -2025-10-30T18:18:16.568Z [INFO] creating new snapshot: path=/tmp/raft471217910/snapshots/2-23-1761848296568.tmp -2025-10-30T18:18:16.568Z [INFO] server-8bd36cd1-dce9-28b1-169e-ae212f20b997: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:16.568Z [INFO] server-8a11aa13-229c-7a1e-5a97-6d583dc3914a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:16.568Z [INFO] server-8bd36cd1-dce9-28b1-169e-ae212f20b997: copied to local snapshot: bytes=71 -2025-10-30T18:18:16.568Z [INFO] server-8a11aa13-229c-7a1e-5a97-6d583dc3914a: copied to local snapshot: bytes=71 -2025-10-30T18:18:16.568Z [INFO] server-8bd36cd1-dce9-28b1-169e-ae212f20b997: snapshot restore progress: id=2-23-1761848296568 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:16.569Z [INFO] server-8bd36cd1-dce9-28b1-169e-ae212f20b997: Installed remote snapshot -2025-10-30T18:18:16.569Z [INFO] server-8a11aa13-229c-7a1e-5a97-6d583dc3914a: snapshot restore progress: id=2-23-1761848296568 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:16.569Z [WARN] server-8bd36cd1-dce9-28b1-169e-ae212f20b997: skipping application of old log: index=22 -2025-10-30T18:18:16.569Z [INFO] server-8a11aa13-229c-7a1e-5a97-6d583dc3914a: Installed remote snapshot -2025-10-30T18:18:16.569Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: pipelining replication: peer="{Voter server-8bd36cd1-dce9-28b1-169e-ae212f20b997 8bd36cd1-dce9-28b1-169e-ae212f20b997}" -2025-10-30T18:18:16.569Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:18:16.569Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: pipelining replication: peer="{Voter server-8a11aa13-229c-7a1e-5a97-6d583dc3914a 8a11aa13-229c-7a1e-5a97-6d583dc3914a}" -2025-10-30T18:18:17.574Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:18:16.569199374 +0000 UTC m=+42.847873275 (2 nodes), 822ns from start of poll, 1.694917879s from cluster start. Timeout at 2025-10-30 18:18:17.574208981 +0000 UTC m=+43.852882890, 1.005009615s after stability -2025-10-30T18:18:17.586Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: aborting pipeline replication: peer="{Voter server-8bd36cd1-dce9-28b1-169e-ae212f20b997 8bd36cd1-dce9-28b1-169e-ae212f20b997}" -2025-10-30T18:18:17.586Z [INFO] server-e410ad3b-cb93-be72-6fe1-e17c4ddb6600: aborting pipeline replication: peer="{Voter server-8a11aa13-229c-7a1e-5a97-6d583dc3914a 8a11aa13-229c-7a1e-5a97-6d583dc3914a}" +2025-10-30T18:22:05.543Z [INFO] server-f8318cf8-5c1b-36dd-ab3f-40b54fd0be36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8318cf8-5c1b-36dd-ab3f-40b54fd0be36 Address:f8318cf8-5c1b-36dd-ab3f-40b54fd0be36} {Suffrage:Voter ID:server-ede83569-0884-1a2d-9fca-67eb57fdad8a Address:ede83569-0884-1a2d-9fca-67eb57fdad8a} {Suffrage:Voter ID:server-47ace765-6e5d-3306-c2cf-dcc90d36cb82 Address:47ace765-6e5d-3306-c2cf-dcc90d36cb82}]" +2025-10-30T18:22:05.543Z [INFO] server-f8318cf8-5c1b-36dd-ab3f-40b54fd0be36: entering follower state: follower="Node at f8318cf8-5c1b-36dd-ab3f-40b54fd0be36 [Follower]" leader-address= leader-id= +2025-10-30T18:22:05.543Z [INFO] server-ede83569-0884-1a2d-9fca-67eb57fdad8a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8318cf8-5c1b-36dd-ab3f-40b54fd0be36 Address:f8318cf8-5c1b-36dd-ab3f-40b54fd0be36} {Suffrage:Voter ID:server-ede83569-0884-1a2d-9fca-67eb57fdad8a Address:ede83569-0884-1a2d-9fca-67eb57fdad8a} {Suffrage:Voter ID:server-47ace765-6e5d-3306-c2cf-dcc90d36cb82 Address:47ace765-6e5d-3306-c2cf-dcc90d36cb82}]" +2025-10-30T18:22:05.543Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8318cf8-5c1b-36dd-ab3f-40b54fd0be36 Address:f8318cf8-5c1b-36dd-ab3f-40b54fd0be36} {Suffrage:Voter ID:server-ede83569-0884-1a2d-9fca-67eb57fdad8a Address:ede83569-0884-1a2d-9fca-67eb57fdad8a} {Suffrage:Voter ID:server-47ace765-6e5d-3306-c2cf-dcc90d36cb82 Address:47ace765-6e5d-3306-c2cf-dcc90d36cb82}]" +2025-10-30T18:22:05.543Z [INFO] server-ede83569-0884-1a2d-9fca-67eb57fdad8a: entering follower state: follower="Node at ede83569-0884-1a2d-9fca-67eb57fdad8a [Follower]" leader-address= leader-id= +2025-10-30T18:22:05.543Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:05.543Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: entering follower state: follower="Node at 47ace765-6e5d-3306-c2cf-dcc90d36cb82 [Follower]" leader-address= leader-id= +2025-10-30T18:22:06.111Z [WARN] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:06.111Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: entering candidate state: node="Node at 47ace765-6e5d-3306-c2cf-dcc90d36cb82 [Candidate]" term=2 +2025-10-30T18:22:06.111Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: election won: term=2 tally=2 +2025-10-30T18:22:06.111Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: entering leader state: leader="Node at 47ace765-6e5d-3306-c2cf-dcc90d36cb82 [Leader]" +2025-10-30T18:22:06.111Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: added peer, starting replication: peer=server-f8318cf8-5c1b-36dd-ab3f-40b54fd0be36 +2025-10-30T18:22:06.111Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: added peer, starting replication: peer=server-ede83569-0884-1a2d-9fca-67eb57fdad8a +2025-10-30T18:22:06.112Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: pipelining replication: peer="{Voter server-ede83569-0884-1a2d-9fca-67eb57fdad8a ede83569-0884-1a2d-9fca-67eb57fdad8a}" +2025-10-30T18:22:06.112Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: pipelining replication: peer="{Voter server-f8318cf8-5c1b-36dd-ab3f-40b54fd0be36 f8318cf8-5c1b-36dd-ab3f-40b54fd0be36}" +2025-10-30T18:22:07.117Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:06.111951225 +0000 UTC m=+49.185480536 (1 nodes), 568.132048ms from start of poll, 568.432655ms from cluster start. Timeout at 2025-10-30 18:22:07.11695517 +0000 UTC m=+50.190484461, 1.005003925s after stability +2025-10-30T18:22:07.119Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: starting snapshot up to: index=12 +2025-10-30T18:22:07.119Z [INFO] creating new snapshot: path=/tmp/raft3791834344/snapshots/2-12-1761848527119.tmp +2025-10-30T18:22:07.119Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: snapshot complete up to: index=12 +2025-10-30T18:22:07.120Z [INFO] creating new snapshot: path=/tmp/raft3791834344/snapshots/2-23-1761848527120.tmp +2025-10-30T18:22:07.121Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: copied to local snapshot: bytes=71 +2025-10-30T18:22:07.121Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: snapshot restore progress: id=2-23-1761848527120 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:07.121Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: restored user snapshot: index=23 +2025-10-30T18:22:07.121Z [ERROR] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: failed to get log: index=23 error="log not found" +2025-10-30T18:22:07.121Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: aborting pipeline replication: peer="{Voter server-ede83569-0884-1a2d-9fca-67eb57fdad8a ede83569-0884-1a2d-9fca-67eb57fdad8a}" +2025-10-30T18:22:07.121Z [ERROR] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: failed to get log: index=23 error="log not found" +2025-10-30T18:22:07.121Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: aborting pipeline replication: peer="{Voter server-f8318cf8-5c1b-36dd-ab3f-40b54fd0be36 f8318cf8-5c1b-36dd-ab3f-40b54fd0be36}" +2025-10-30T18:22:07.127Z [ERROR] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: failed to get log: index=23 error="log not found" +2025-10-30T18:22:07.127Z [INFO] creating new snapshot: path=/tmp/raft839950559/snapshots/2-23-1761848527127.tmp +2025-10-30T18:22:07.127Z [INFO] server-f8318cf8-5c1b-36dd-ab3f-40b54fd0be36: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:07.128Z [INFO] server-f8318cf8-5c1b-36dd-ab3f-40b54fd0be36: copied to local snapshot: bytes=71 +2025-10-30T18:22:07.128Z [ERROR] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: failed to get log: index=23 error="log not found" +2025-10-30T18:22:07.128Z [INFO] server-f8318cf8-5c1b-36dd-ab3f-40b54fd0be36: snapshot restore progress: id=2-23-1761848527127 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:07.128Z [INFO] server-f8318cf8-5c1b-36dd-ab3f-40b54fd0be36: Installed remote snapshot +2025-10-30T18:22:07.128Z [WARN] server-f8318cf8-5c1b-36dd-ab3f-40b54fd0be36: skipping application of old log: index=22 +2025-10-30T18:22:07.128Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: pipelining replication: peer="{Voter server-f8318cf8-5c1b-36dd-ab3f-40b54fd0be36 f8318cf8-5c1b-36dd-ab3f-40b54fd0be36}" +2025-10-30T18:22:07.128Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:22:07.128Z [INFO] creating new snapshot: path=/tmp/raft2767325865/snapshots/2-23-1761848527128.tmp +2025-10-30T18:22:07.128Z [INFO] server-ede83569-0884-1a2d-9fca-67eb57fdad8a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:07.128Z [INFO] server-ede83569-0884-1a2d-9fca-67eb57fdad8a: copied to local snapshot: bytes=71 +2025-10-30T18:22:07.129Z [INFO] server-ede83569-0884-1a2d-9fca-67eb57fdad8a: snapshot restore progress: id=2-23-1761848527128 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:07.129Z [INFO] server-ede83569-0884-1a2d-9fca-67eb57fdad8a: Installed remote snapshot +2025-10-30T18:22:07.129Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: pipelining replication: peer="{Voter server-ede83569-0884-1a2d-9fca-67eb57fdad8a ede83569-0884-1a2d-9fca-67eb57fdad8a}" +2025-10-30T18:22:08.134Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:22:07.128438682 +0000 UTC m=+50.201967974 (2 nodes), 637ns from start of poll, 1.584920093s from cluster start. Timeout at 2025-10-30 18:22:08.133461257 +0000 UTC m=+51.206990580, 1.005022606s after stability +2025-10-30T18:22:08.145Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: aborting pipeline replication: peer="{Voter server-ede83569-0884-1a2d-9fca-67eb57fdad8a ede83569-0884-1a2d-9fca-67eb57fdad8a}" +2025-10-30T18:22:08.145Z [INFO] server-47ace765-6e5d-3306-c2cf-dcc90d36cb82: aborting pipeline replication: peer="{Voter server-f8318cf8-5c1b-36dd-ab3f-40b54fd0be36 f8318cf8-5c1b-36dd-ab3f-40b54fd0be36}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false -2025-10-30T18:18:17.587Z [INFO] server-68eef77e-3208-b381-278a-45983e61fee5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68eef77e-3208-b381-278a-45983e61fee5 Address:68eef77e-3208-b381-278a-45983e61fee5} {Suffrage:Voter ID:server-c623027d-1b46-13b1-30f2-99bf616d7e3b Address:c623027d-1b46-13b1-30f2-99bf616d7e3b} {Suffrage:Voter ID:server-ee861805-a79a-24cd-a55c-27b5dece4f98 Address:ee861805-a79a-24cd-a55c-27b5dece4f98}]" -2025-10-30T18:18:17.588Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68eef77e-3208-b381-278a-45983e61fee5 Address:68eef77e-3208-b381-278a-45983e61fee5} {Suffrage:Voter ID:server-c623027d-1b46-13b1-30f2-99bf616d7e3b Address:c623027d-1b46-13b1-30f2-99bf616d7e3b} {Suffrage:Voter ID:server-ee861805-a79a-24cd-a55c-27b5dece4f98 Address:ee861805-a79a-24cd-a55c-27b5dece4f98}]" -2025-10-30T18:18:17.588Z [INFO] server-68eef77e-3208-b381-278a-45983e61fee5: entering follower state: follower="Node at 68eef77e-3208-b381-278a-45983e61fee5 [Follower]" leader-address= leader-id= -2025-10-30T18:18:17.588Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: entering follower state: follower="Node at c623027d-1b46-13b1-30f2-99bf616d7e3b [Follower]" leader-address= leader-id= -2025-10-30T18:18:17.588Z [INFO] server-ee861805-a79a-24cd-a55c-27b5dece4f98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68eef77e-3208-b381-278a-45983e61fee5 Address:68eef77e-3208-b381-278a-45983e61fee5} {Suffrage:Voter ID:server-c623027d-1b46-13b1-30f2-99bf616d7e3b Address:c623027d-1b46-13b1-30f2-99bf616d7e3b} {Suffrage:Voter ID:server-ee861805-a79a-24cd-a55c-27b5dece4f98 Address:ee861805-a79a-24cd-a55c-27b5dece4f98}]" -2025-10-30T18:18:17.588Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:17.588Z [INFO] server-ee861805-a79a-24cd-a55c-27b5dece4f98: entering follower state: follower="Node at ee861805-a79a-24cd-a55c-27b5dece4f98 [Follower]" leader-address= leader-id= -2025-10-30T18:18:18.316Z [WARN] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:18.316Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: entering candidate state: node="Node at c623027d-1b46-13b1-30f2-99bf616d7e3b [Candidate]" term=2 -2025-10-30T18:18:18.316Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: election won: term=2 tally=2 -2025-10-30T18:18:18.316Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: entering leader state: leader="Node at c623027d-1b46-13b1-30f2-99bf616d7e3b [Leader]" -2025-10-30T18:18:18.316Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: added peer, starting replication: peer=server-68eef77e-3208-b381-278a-45983e61fee5 -2025-10-30T18:18:18.316Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: added peer, starting replication: peer=server-ee861805-a79a-24cd-a55c-27b5dece4f98 -2025-10-30T18:18:18.316Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: pipelining replication: peer="{Voter server-68eef77e-3208-b381-278a-45983e61fee5 68eef77e-3208-b381-278a-45983e61fee5}" -2025-10-30T18:18:18.316Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: pipelining replication: peer="{Voter server-ee861805-a79a-24cd-a55c-27b5dece4f98 ee861805-a79a-24cd-a55c-27b5dece4f98}" -2025-10-30T18:18:19.321Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:18.316583877 +0000 UTC m=+44.595257774 (1 nodes), 728.32873ms from start of poll, 728.705482ms from cluster start. Timeout at 2025-10-30 18:18:19.321585357 +0000 UTC m=+45.600259254, 1.00500148s after stability -2025-10-30T18:18:19.322Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: starting snapshot up to: index=12 -2025-10-30T18:18:19.322Z [INFO] creating new snapshot: path=/tmp/raft529528452/snapshots/2-12-1761848299322.tmp -2025-10-30T18:18:19.322Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: snapshot complete up to: index=12 -2025-10-30T18:18:19.322Z [INFO] creating new snapshot: path=/tmp/raft529528452/snapshots/2-23-1761848299322.tmp -2025-10-30T18:18:19.323Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: copied to local snapshot: bytes=71 -2025-10-30T18:18:19.323Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: snapshot restore progress: id=2-23-1761848299322 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:19.323Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: removing all old logs from log store -2025-10-30T18:18:19.323Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: compacting logs: from=1 to=22 -2025-10-30T18:18:19.323Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: restored user snapshot: index=23 -2025-10-30T18:18:19.323Z [ERROR] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: failed to get log: index=22 error="log not found" -2025-10-30T18:18:19.323Z [ERROR] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: failed to get log: index=22 error="log not found" -2025-10-30T18:18:19.323Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: aborting pipeline replication: peer="{Voter server-68eef77e-3208-b381-278a-45983e61fee5 68eef77e-3208-b381-278a-45983e61fee5}" -2025-10-30T18:18:19.323Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: aborting pipeline replication: peer="{Voter server-ee861805-a79a-24cd-a55c-27b5dece4f98 ee861805-a79a-24cd-a55c-27b5dece4f98}" -2025-10-30T18:18:19.332Z [ERROR] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: failed to get log: index=22 error="log not found" -2025-10-30T18:18:19.332Z [ERROR] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: failed to get log: index=22 error="log not found" -2025-10-30T18:18:19.333Z [INFO] creating new snapshot: path=/tmp/raft792270234/snapshots/2-23-1761848299333.tmp -2025-10-30T18:18:19.333Z [INFO] server-68eef77e-3208-b381-278a-45983e61fee5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:19.333Z [INFO] creating new snapshot: path=/tmp/raft3375558584/snapshots/2-23-1761848299333.tmp -2025-10-30T18:18:19.334Z [INFO] server-ee861805-a79a-24cd-a55c-27b5dece4f98: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:19.334Z [INFO] server-68eef77e-3208-b381-278a-45983e61fee5: copied to local snapshot: bytes=71 -2025-10-30T18:18:19.334Z [INFO] server-ee861805-a79a-24cd-a55c-27b5dece4f98: copied to local snapshot: bytes=71 -2025-10-30T18:18:19.334Z [INFO] server-68eef77e-3208-b381-278a-45983e61fee5: snapshot restore progress: id=2-23-1761848299333 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:19.334Z [INFO] server-68eef77e-3208-b381-278a-45983e61fee5: removing all old logs from log store -2025-10-30T18:18:19.334Z [INFO] server-68eef77e-3208-b381-278a-45983e61fee5: compacting logs: from=1 to=22 -2025-10-30T18:18:19.334Z [INFO] server-68eef77e-3208-b381-278a-45983e61fee5: Installed remote snapshot -2025-10-30T18:18:19.335Z [WARN] server-68eef77e-3208-b381-278a-45983e61fee5: skipping application of old log: index=22 -2025-10-30T18:18:19.335Z [INFO] server-ee861805-a79a-24cd-a55c-27b5dece4f98: snapshot restore progress: id=2-23-1761848299333 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:19.335Z [INFO] server-ee861805-a79a-24cd-a55c-27b5dece4f98: removing all old logs from log store -2025-10-30T18:18:19.335Z [INFO] server-ee861805-a79a-24cd-a55c-27b5dece4f98: compacting logs: from=1 to=22 -2025-10-30T18:18:19.335Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: pipelining replication: peer="{Voter server-68eef77e-3208-b381-278a-45983e61fee5 68eef77e-3208-b381-278a-45983e61fee5}" -2025-10-30T18:18:19.335Z [INFO] server-ee861805-a79a-24cd-a55c-27b5dece4f98: Installed remote snapshot -2025-10-30T18:18:19.335Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:18:19.335Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: pipelining replication: peer="{Voter server-ee861805-a79a-24cd-a55c-27b5dece4f98 ee861805-a79a-24cd-a55c-27b5dece4f98}" -2025-10-30T18:18:20.340Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:18:19.335104056 +0000 UTC m=+45.613777955 (2 nodes), 998ns from start of poll, 1.747225663s from cluster start. Timeout at 2025-10-30 18:18:20.340114096 +0000 UTC m=+46.618788006, 1.005010051s after stability -2025-10-30T18:18:20.351Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: aborting pipeline replication: peer="{Voter server-ee861805-a79a-24cd-a55c-27b5dece4f98 ee861805-a79a-24cd-a55c-27b5dece4f98}" -2025-10-30T18:18:20.351Z [INFO] server-c623027d-1b46-13b1-30f2-99bf616d7e3b: aborting pipeline replication: peer="{Voter server-68eef77e-3208-b381-278a-45983e61fee5 68eef77e-3208-b381-278a-45983e61fee5}" +2025-10-30T18:22:08.147Z [INFO] server-1983cec0-080b-e802-7898-b7557cd32c7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1983cec0-080b-e802-7898-b7557cd32c7c Address:1983cec0-080b-e802-7898-b7557cd32c7c} {Suffrage:Voter ID:server-8865a411-2076-d5e6-a252-1c6b69a9d47d Address:8865a411-2076-d5e6-a252-1c6b69a9d47d} {Suffrage:Voter ID:server-8535457c-ff0b-e4a3-8e01-ca99606dc009 Address:8535457c-ff0b-e4a3-8e01-ca99606dc009}]" +2025-10-30T18:22:08.147Z [INFO] server-1983cec0-080b-e802-7898-b7557cd32c7c: entering follower state: follower="Node at 1983cec0-080b-e802-7898-b7557cd32c7c [Follower]" leader-address= leader-id= +2025-10-30T18:22:08.147Z [INFO] server-8865a411-2076-d5e6-a252-1c6b69a9d47d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1983cec0-080b-e802-7898-b7557cd32c7c Address:1983cec0-080b-e802-7898-b7557cd32c7c} {Suffrage:Voter ID:server-8865a411-2076-d5e6-a252-1c6b69a9d47d Address:8865a411-2076-d5e6-a252-1c6b69a9d47d} {Suffrage:Voter ID:server-8535457c-ff0b-e4a3-8e01-ca99606dc009 Address:8535457c-ff0b-e4a3-8e01-ca99606dc009}]" +2025-10-30T18:22:08.147Z [INFO] server-8865a411-2076-d5e6-a252-1c6b69a9d47d: entering follower state: follower="Node at 8865a411-2076-d5e6-a252-1c6b69a9d47d [Follower]" leader-address= leader-id= +2025-10-30T18:22:08.147Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1983cec0-080b-e802-7898-b7557cd32c7c Address:1983cec0-080b-e802-7898-b7557cd32c7c} {Suffrage:Voter ID:server-8865a411-2076-d5e6-a252-1c6b69a9d47d Address:8865a411-2076-d5e6-a252-1c6b69a9d47d} {Suffrage:Voter ID:server-8535457c-ff0b-e4a3-8e01-ca99606dc009 Address:8535457c-ff0b-e4a3-8e01-ca99606dc009}]" +2025-10-30T18:22:08.147Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:08.147Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: entering follower state: follower="Node at 8535457c-ff0b-e4a3-8e01-ca99606dc009 [Follower]" leader-address= leader-id= +2025-10-30T18:22:08.653Z [WARN] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:08.654Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: entering candidate state: node="Node at 8535457c-ff0b-e4a3-8e01-ca99606dc009 [Candidate]" term=2 +2025-10-30T18:22:08.654Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: election won: term=2 tally=2 +2025-10-30T18:22:08.654Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: entering leader state: leader="Node at 8535457c-ff0b-e4a3-8e01-ca99606dc009 [Leader]" +2025-10-30T18:22:08.654Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: added peer, starting replication: peer=server-1983cec0-080b-e802-7898-b7557cd32c7c +2025-10-30T18:22:08.654Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: added peer, starting replication: peer=server-8865a411-2076-d5e6-a252-1c6b69a9d47d +2025-10-30T18:22:08.654Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: pipelining replication: peer="{Voter server-8865a411-2076-d5e6-a252-1c6b69a9d47d 8865a411-2076-d5e6-a252-1c6b69a9d47d}" +2025-10-30T18:22:08.654Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: pipelining replication: peer="{Voter server-1983cec0-080b-e802-7898-b7557cd32c7c 1983cec0-080b-e802-7898-b7557cd32c7c}" +2025-10-30T18:22:09.659Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:08.654165981 +0000 UTC m=+51.727695351 (1 nodes), 506.830154ms from start of poll, 507.130798ms from cluster start. Timeout at 2025-10-30 18:22:09.659171924 +0000 UTC m=+52.732701212, 1.005005861s after stability +2025-10-30T18:22:09.659Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: starting snapshot up to: index=12 +2025-10-30T18:22:09.659Z [INFO] creating new snapshot: path=/tmp/raft101037422/snapshots/2-12-1761848529659.tmp +2025-10-30T18:22:09.884Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: snapshot complete up to: index=12 +2025-10-30T18:22:09.884Z [INFO] creating new snapshot: path=/tmp/raft101037422/snapshots/2-23-1761848529884.tmp +2025-10-30T18:22:09.885Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: copied to local snapshot: bytes=71 +2025-10-30T18:22:09.885Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: snapshot restore progress: id=2-23-1761848529884 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:09.885Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: removing all old logs from log store +2025-10-30T18:22:09.885Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: compacting logs: from=1 to=22 +2025-10-30T18:22:09.885Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: restored user snapshot: index=23 +2025-10-30T18:22:09.885Z [ERROR] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: failed to get log: index=22 error="log not found" +2025-10-30T18:22:09.885Z [ERROR] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: failed to get log: index=22 error="log not found" +2025-10-30T18:22:09.885Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: aborting pipeline replication: peer="{Voter server-1983cec0-080b-e802-7898-b7557cd32c7c 1983cec0-080b-e802-7898-b7557cd32c7c}" +2025-10-30T18:22:09.885Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: aborting pipeline replication: peer="{Voter server-8865a411-2076-d5e6-a252-1c6b69a9d47d 8865a411-2076-d5e6-a252-1c6b69a9d47d}" +2025-10-30T18:22:09.894Z [ERROR] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: failed to get log: index=22 error="log not found" +2025-10-30T18:22:09.894Z [INFO] creating new snapshot: path=/tmp/raft761836380/snapshots/2-23-1761848529894.tmp +2025-10-30T18:22:09.894Z [ERROR] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: failed to get log: index=22 error="log not found" +2025-10-30T18:22:09.894Z [INFO] server-1983cec0-080b-e802-7898-b7557cd32c7c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:09.895Z [INFO] creating new snapshot: path=/tmp/raft1571239853/snapshots/2-23-1761848529895.tmp +2025-10-30T18:22:09.895Z [INFO] server-8865a411-2076-d5e6-a252-1c6b69a9d47d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:09.910Z [INFO] server-1983cec0-080b-e802-7898-b7557cd32c7c: copied to local snapshot: bytes=71 +2025-10-30T18:22:09.910Z [INFO] server-8865a411-2076-d5e6-a252-1c6b69a9d47d: copied to local snapshot: bytes=71 +2025-10-30T18:22:09.911Z [INFO] server-1983cec0-080b-e802-7898-b7557cd32c7c: snapshot restore progress: id=2-23-1761848529894 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:09.911Z [INFO] server-1983cec0-080b-e802-7898-b7557cd32c7c: removing all old logs from log store +2025-10-30T18:22:09.911Z [INFO] server-1983cec0-080b-e802-7898-b7557cd32c7c: compacting logs: from=1 to=22 +2025-10-30T18:22:09.911Z [INFO] server-8865a411-2076-d5e6-a252-1c6b69a9d47d: snapshot restore progress: id=2-23-1761848529895 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:09.911Z [INFO] server-1983cec0-080b-e802-7898-b7557cd32c7c: Installed remote snapshot +2025-10-30T18:22:09.911Z [INFO] server-8865a411-2076-d5e6-a252-1c6b69a9d47d: removing all old logs from log store +2025-10-30T18:22:09.911Z [INFO] server-8865a411-2076-d5e6-a252-1c6b69a9d47d: compacting logs: from=1 to=22 +2025-10-30T18:22:09.911Z [INFO] server-8865a411-2076-d5e6-a252-1c6b69a9d47d: Installed remote snapshot +2025-10-30T18:22:09.911Z [WARN] server-1983cec0-080b-e802-7898-b7557cd32c7c: skipping application of old log: index=22 +2025-10-30T18:22:09.911Z [WARN] server-8865a411-2076-d5e6-a252-1c6b69a9d47d: skipping application of old log: index=23 +2025-10-30T18:22:09.911Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: pipelining replication: peer="{Voter server-1983cec0-080b-e802-7898-b7557cd32c7c 1983cec0-080b-e802-7898-b7557cd32c7c}" +2025-10-30T18:22:09.911Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:22:09.911Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: pipelining replication: peer="{Voter server-8865a411-2076-d5e6-a252-1c6b69a9d47d 8865a411-2076-d5e6-a252-1c6b69a9d47d}" +2025-10-30T18:22:10.916Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:22:09.911287319 +0000 UTC m=+52.984816617 (2 nodes), 1.041µs from start of poll, 1.764252064s from cluster start. Timeout at 2025-10-30 18:22:10.916318356 +0000 UTC m=+53.989847685, 1.005031068s after stability +2025-10-30T18:22:10.927Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: aborting pipeline replication: peer="{Voter server-8865a411-2076-d5e6-a252-1c6b69a9d47d 8865a411-2076-d5e6-a252-1c6b69a9d47d}" +2025-10-30T18:22:10.928Z [INFO] server-8535457c-ff0b-e4a3-8e01-ca99606dc009: aborting pipeline replication: peer="{Voter server-1983cec0-080b-e802-7898-b7557cd32c7c 1983cec0-080b-e802-7898-b7557cd32c7c}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true -2025-10-30T18:18:20.352Z [INFO] server-e354a99b-aaa3-c6e2-f235-3dfab84fe83e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e354a99b-aaa3-c6e2-f235-3dfab84fe83e Address:e354a99b-aaa3-c6e2-f235-3dfab84fe83e} {Suffrage:Voter ID:server-dac695b8-a066-2646-30f8-d8b9c0078a83 Address:dac695b8-a066-2646-30f8-d8b9c0078a83} {Suffrage:Voter ID:server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049 Address:d96195c3-9dec-9dd9-a246-a0fc2c3a2049}]" -2025-10-30T18:18:20.352Z [INFO] server-dac695b8-a066-2646-30f8-d8b9c0078a83: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e354a99b-aaa3-c6e2-f235-3dfab84fe83e Address:e354a99b-aaa3-c6e2-f235-3dfab84fe83e} {Suffrage:Voter ID:server-dac695b8-a066-2646-30f8-d8b9c0078a83 Address:dac695b8-a066-2646-30f8-d8b9c0078a83} {Suffrage:Voter ID:server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049 Address:d96195c3-9dec-9dd9-a246-a0fc2c3a2049}]" -2025-10-30T18:18:20.352Z [INFO] server-e354a99b-aaa3-c6e2-f235-3dfab84fe83e: entering follower state: follower="Node at e354a99b-aaa3-c6e2-f235-3dfab84fe83e [Follower]" leader-address= leader-id= -2025-10-30T18:18:20.352Z [INFO] server-dac695b8-a066-2646-30f8-d8b9c0078a83: entering follower state: follower="Node at dac695b8-a066-2646-30f8-d8b9c0078a83 [Follower]" leader-address= leader-id= -2025-10-30T18:18:20.352Z [INFO] server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e354a99b-aaa3-c6e2-f235-3dfab84fe83e Address:e354a99b-aaa3-c6e2-f235-3dfab84fe83e} {Suffrage:Voter ID:server-dac695b8-a066-2646-30f8-d8b9c0078a83 Address:dac695b8-a066-2646-30f8-d8b9c0078a83} {Suffrage:Voter ID:server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049 Address:d96195c3-9dec-9dd9-a246-a0fc2c3a2049}]" -2025-10-30T18:18:20.352Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:20.352Z [INFO] server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: entering follower state: follower="Node at d96195c3-9dec-9dd9-a246-a0fc2c3a2049 [Follower]" leader-address= leader-id= -2025-10-30T18:18:21.001Z [WARN] server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:21.001Z [INFO] server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: entering candidate state: node="Node at d96195c3-9dec-9dd9-a246-a0fc2c3a2049 [Candidate]" term=2 -2025-10-30T18:18:21.001Z [INFO] server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: election won: term=2 tally=2 -2025-10-30T18:18:21.001Z [INFO] server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: entering leader state: leader="Node at d96195c3-9dec-9dd9-a246-a0fc2c3a2049 [Leader]" -2025-10-30T18:18:21.001Z [INFO] server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: added peer, starting replication: peer=server-e354a99b-aaa3-c6e2-f235-3dfab84fe83e -2025-10-30T18:18:21.001Z [INFO] server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: added peer, starting replication: peer=server-dac695b8-a066-2646-30f8-d8b9c0078a83 -2025-10-30T18:18:21.002Z [INFO] server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: pipelining replication: peer="{Voter server-dac695b8-a066-2646-30f8-d8b9c0078a83 dac695b8-a066-2646-30f8-d8b9c0078a83}" -2025-10-30T18:18:21.002Z [INFO] server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: pipelining replication: peer="{Voter server-e354a99b-aaa3-c6e2-f235-3dfab84fe83e e354a99b-aaa3-c6e2-f235-3dfab84fe83e}" -2025-10-30T18:18:22.007Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:21.001953009 +0000 UTC m=+47.280626906 (1 nodes), 649.587675ms from start of poll, 649.817248ms from cluster start. Timeout at 2025-10-30 18:18:22.00695676 +0000 UTC m=+48.285630657, 1.005003751s after stability -2025-10-30T18:18:22.008Z [INFO] server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: starting snapshot up to: index=12 -2025-10-30T18:18:22.008Z [INFO] creating new snapshot: path=/tmp/raft3054392106/snapshots/2-12-1761848302008.tmp -2025-10-30T18:18:22.008Z [INFO] server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: snapshot complete up to: index=12 -2025-10-30T18:18:22.010Z [INFO] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acb4f508-92ed-0d88-7103-52bc09f9abd4 Address:acb4f508-92ed-0d88-7103-52bc09f9abd4} {Suffrage:Voter ID:server-15259e9c-4869-743d-fed4-4e0d522d7c7f Address:15259e9c-4869-743d-fed4-4e0d522d7c7f} {Suffrage:Voter ID:server-4389cd1c-3106-5587-c166-6e11581f620b Address:4389cd1c-3106-5587-c166-6e11581f620b}]" -2025-10-30T18:18:22.010Z [INFO] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: entering follower state: follower="Node at acb4f508-92ed-0d88-7103-52bc09f9abd4 [Follower]" leader-address= leader-id= -2025-10-30T18:18:22.010Z [INFO] server-15259e9c-4869-743d-fed4-4e0d522d7c7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acb4f508-92ed-0d88-7103-52bc09f9abd4 Address:acb4f508-92ed-0d88-7103-52bc09f9abd4} {Suffrage:Voter ID:server-15259e9c-4869-743d-fed4-4e0d522d7c7f Address:15259e9c-4869-743d-fed4-4e0d522d7c7f} {Suffrage:Voter ID:server-4389cd1c-3106-5587-c166-6e11581f620b Address:4389cd1c-3106-5587-c166-6e11581f620b}]" -2025-10-30T18:18:22.010Z [INFO] server-15259e9c-4869-743d-fed4-4e0d522d7c7f: entering follower state: follower="Node at 15259e9c-4869-743d-fed4-4e0d522d7c7f [Follower]" leader-address= leader-id= -2025-10-30T18:18:22.010Z [INFO] server-4389cd1c-3106-5587-c166-6e11581f620b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acb4f508-92ed-0d88-7103-52bc09f9abd4 Address:acb4f508-92ed-0d88-7103-52bc09f9abd4} {Suffrage:Voter ID:server-15259e9c-4869-743d-fed4-4e0d522d7c7f Address:15259e9c-4869-743d-fed4-4e0d522d7c7f} {Suffrage:Voter ID:server-4389cd1c-3106-5587-c166-6e11581f620b Address:4389cd1c-3106-5587-c166-6e11581f620b}]" -2025-10-30T18:18:22.010Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:22.010Z [INFO] server-4389cd1c-3106-5587-c166-6e11581f620b: entering follower state: follower="Node at 4389cd1c-3106-5587-c166-6e11581f620b [Follower]" leader-address= leader-id= -2025-10-30T18:18:22.679Z [WARN] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:22.679Z [INFO] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: entering candidate state: node="Node at acb4f508-92ed-0d88-7103-52bc09f9abd4 [Candidate]" term=2 -2025-10-30T18:18:22.680Z [INFO] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: election won: term=2 tally=2 -2025-10-30T18:18:22.680Z [INFO] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: entering leader state: leader="Node at acb4f508-92ed-0d88-7103-52bc09f9abd4 [Leader]" -2025-10-30T18:18:22.680Z [INFO] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: added peer, starting replication: peer=server-15259e9c-4869-743d-fed4-4e0d522d7c7f -2025-10-30T18:18:22.680Z [INFO] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: added peer, starting replication: peer=server-4389cd1c-3106-5587-c166-6e11581f620b -2025-10-30T18:18:22.680Z [INFO] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: pipelining replication: peer="{Voter server-15259e9c-4869-743d-fed4-4e0d522d7c7f 15259e9c-4869-743d-fed4-4e0d522d7c7f}" -2025-10-30T18:18:22.680Z [INFO] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: pipelining replication: peer="{Voter server-4389cd1c-3106-5587-c166-6e11581f620b 4389cd1c-3106-5587-c166-6e11581f620b}" -2025-10-30T18:18:23.685Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:22.680090206 +0000 UTC m=+48.958764102 (1 nodes), 669.394425ms from start of poll, 669.768725ms from cluster start. Timeout at 2025-10-30 18:18:23.685090459 +0000 UTC m=+49.963764355, 1.005000253s after stability -2025-10-30T18:18:23.685Z [INFO] creating new snapshot: path=/tmp/raft779741449/snapshots/2-15-1761848303685.tmp -2025-10-30T18:18:23.701Z [INFO] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: copied to local snapshot: bytes=71 -2025-10-30T18:18:23.701Z [INFO] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: snapshot restore progress: id=2-15-1761848303685 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:23.701Z [INFO] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: restored user snapshot: index=15 -2025-10-30T18:18:23.701Z [ERROR] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: failed to get log: index=3 error="log not found" -2025-10-30T18:18:23.701Z [ERROR] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: failed to get log: index=3 error="log not found" -2025-10-30T18:18:23.701Z [INFO] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: aborting pipeline replication: peer="{Voter server-4389cd1c-3106-5587-c166-6e11581f620b 4389cd1c-3106-5587-c166-6e11581f620b}" -2025-10-30T18:18:23.701Z [INFO] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: aborting pipeline replication: peer="{Voter server-15259e9c-4869-743d-fed4-4e0d522d7c7f 15259e9c-4869-743d-fed4-4e0d522d7c7f}" -2025-10-30T18:18:23.708Z [ERROR] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: failed to get log: index=3 error="log not found" -2025-10-30T18:18:23.711Z [ERROR] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: failed to get log: index=3 error="log not found" -2025-10-30T18:18:23.711Z [INFO] creating new snapshot: path=/tmp/raft3968030613/snapshots/2-15-1761848303711.tmp -2025-10-30T18:18:23.794Z [INFO] server-15259e9c-4869-743d-fed4-4e0d522d7c7f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:23.794Z [INFO] creating new snapshot: path=/tmp/raft2820671455/snapshots/2-15-1761848303794.tmp -2025-10-30T18:18:23.794Z [INFO] server-15259e9c-4869-743d-fed4-4e0d522d7c7f: copied to local snapshot: bytes=71 -2025-10-30T18:18:23.794Z [INFO] server-4389cd1c-3106-5587-c166-6e11581f620b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:23.795Z [INFO] server-15259e9c-4869-743d-fed4-4e0d522d7c7f: snapshot restore progress: id=2-15-1761848303711 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:23.795Z [INFO] server-15259e9c-4869-743d-fed4-4e0d522d7c7f: Installed remote snapshot -2025-10-30T18:18:23.795Z [INFO] server-4389cd1c-3106-5587-c166-6e11581f620b: copied to local snapshot: bytes=71 -2025-10-30T18:18:23.795Z [INFO] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: pipelining replication: peer="{Voter server-15259e9c-4869-743d-fed4-4e0d522d7c7f 15259e9c-4869-743d-fed4-4e0d522d7c7f}" -2025-10-30T18:18:23.795Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:18:23.795Z [INFO] server-4389cd1c-3106-5587-c166-6e11581f620b: snapshot restore progress: id=2-15-1761848303794 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:23.795Z [INFO] server-4389cd1c-3106-5587-c166-6e11581f620b: Installed remote snapshot -2025-10-30T18:18:23.795Z [INFO] server-acb4f508-92ed-0d88-7103-52bc09f9abd4: pipelining replication: peer="{Voter server-4389cd1c-3106-5587-c166-6e11581f620b 4389cd1c-3106-5587-c166-6e11581f620b}" -2025-10-30T18:18:24.800Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:18:23.795219208 +0000 UTC m=+50.073893107 (2 nodes), 1.023µs from start of poll, 1.78489773s from cluster start. Timeout at 2025-10-30 18:18:24.80022228 +0000 UTC m=+51.078896177, 1.00500307s after stability -2025-10-30T18:18:24.811Z [INFO] server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: aborting pipeline replication: peer="{Voter server-e354a99b-aaa3-c6e2-f235-3dfab84fe83e e354a99b-aaa3-c6e2-f235-3dfab84fe83e}" -2025-10-30T18:18:24.811Z [INFO] server-d96195c3-9dec-9dd9-a246-a0fc2c3a2049: aborting pipeline replication: peer="{Voter server-dac695b8-a066-2646-30f8-d8b9c0078a83 dac695b8-a066-2646-30f8-d8b9c0078a83}" +2025-10-30T18:22:10.929Z [INFO] server-48458d17-a12a-b96f-d5d3-1b6430db0662: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48458d17-a12a-b96f-d5d3-1b6430db0662 Address:48458d17-a12a-b96f-d5d3-1b6430db0662} {Suffrage:Voter ID:server-7f4be064-70b5-633f-5c4f-a15eb3ccf98b Address:7f4be064-70b5-633f-5c4f-a15eb3ccf98b} {Suffrage:Voter ID:server-3e13a0ac-ab57-28ea-1471-89e6dd0d332c Address:3e13a0ac-ab57-28ea-1471-89e6dd0d332c}]" +2025-10-30T18:22:10.929Z [INFO] server-48458d17-a12a-b96f-d5d3-1b6430db0662: entering follower state: follower="Node at 48458d17-a12a-b96f-d5d3-1b6430db0662 [Follower]" leader-address= leader-id= +2025-10-30T18:22:10.929Z [INFO] server-7f4be064-70b5-633f-5c4f-a15eb3ccf98b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48458d17-a12a-b96f-d5d3-1b6430db0662 Address:48458d17-a12a-b96f-d5d3-1b6430db0662} {Suffrage:Voter ID:server-7f4be064-70b5-633f-5c4f-a15eb3ccf98b Address:7f4be064-70b5-633f-5c4f-a15eb3ccf98b} {Suffrage:Voter ID:server-3e13a0ac-ab57-28ea-1471-89e6dd0d332c Address:3e13a0ac-ab57-28ea-1471-89e6dd0d332c}]" +2025-10-30T18:22:10.929Z [INFO] server-7f4be064-70b5-633f-5c4f-a15eb3ccf98b: entering follower state: follower="Node at 7f4be064-70b5-633f-5c4f-a15eb3ccf98b [Follower]" leader-address= leader-id= +2025-10-30T18:22:10.929Z [INFO] server-3e13a0ac-ab57-28ea-1471-89e6dd0d332c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48458d17-a12a-b96f-d5d3-1b6430db0662 Address:48458d17-a12a-b96f-d5d3-1b6430db0662} {Suffrage:Voter ID:server-7f4be064-70b5-633f-5c4f-a15eb3ccf98b Address:7f4be064-70b5-633f-5c4f-a15eb3ccf98b} {Suffrage:Voter ID:server-3e13a0ac-ab57-28ea-1471-89e6dd0d332c Address:3e13a0ac-ab57-28ea-1471-89e6dd0d332c}]" +2025-10-30T18:22:10.929Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:10.929Z [INFO] server-3e13a0ac-ab57-28ea-1471-89e6dd0d332c: entering follower state: follower="Node at 3e13a0ac-ab57-28ea-1471-89e6dd0d332c [Follower]" leader-address= leader-id= +2025-10-30T18:22:11.473Z [WARN] server-48458d17-a12a-b96f-d5d3-1b6430db0662: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:11.473Z [INFO] server-48458d17-a12a-b96f-d5d3-1b6430db0662: entering candidate state: node="Node at 48458d17-a12a-b96f-d5d3-1b6430db0662 [Candidate]" term=2 +2025-10-30T18:22:11.473Z [INFO] server-48458d17-a12a-b96f-d5d3-1b6430db0662: election won: term=2 tally=2 +2025-10-30T18:22:11.473Z [INFO] server-48458d17-a12a-b96f-d5d3-1b6430db0662: entering leader state: leader="Node at 48458d17-a12a-b96f-d5d3-1b6430db0662 [Leader]" +2025-10-30T18:22:11.473Z [INFO] server-48458d17-a12a-b96f-d5d3-1b6430db0662: added peer, starting replication: peer=server-7f4be064-70b5-633f-5c4f-a15eb3ccf98b +2025-10-30T18:22:11.473Z [INFO] server-48458d17-a12a-b96f-d5d3-1b6430db0662: added peer, starting replication: peer=server-3e13a0ac-ab57-28ea-1471-89e6dd0d332c +2025-10-30T18:22:11.473Z [INFO] server-48458d17-a12a-b96f-d5d3-1b6430db0662: pipelining replication: peer="{Voter server-7f4be064-70b5-633f-5c4f-a15eb3ccf98b 7f4be064-70b5-633f-5c4f-a15eb3ccf98b}" +2025-10-30T18:22:11.473Z [INFO] server-48458d17-a12a-b96f-d5d3-1b6430db0662: pipelining replication: peer="{Voter server-3e13a0ac-ab57-28ea-1471-89e6dd0d332c 3e13a0ac-ab57-28ea-1471-89e6dd0d332c}" +2025-10-30T18:22:12.478Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:11.473181505 +0000 UTC m=+54.546710797 (1 nodes), 543.560521ms from start of poll, 543.857352ms from cluster start. Timeout at 2025-10-30 18:22:12.478182858 +0000 UTC m=+55.551712150, 1.005001353s after stability +2025-10-30T18:22:12.479Z [INFO] server-48458d17-a12a-b96f-d5d3-1b6430db0662: starting snapshot up to: index=12 +2025-10-30T18:22:12.479Z [INFO] creating new snapshot: path=/tmp/raft489694312/snapshots/2-12-1761848532479.tmp +2025-10-30T18:22:12.561Z [INFO] server-48458d17-a12a-b96f-d5d3-1b6430db0662: snapshot complete up to: index=12 +2025-10-30T18:22:12.562Z [INFO] server-788cf89c-090e-f232-af4b-94504abbc2a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-788cf89c-090e-f232-af4b-94504abbc2a1 Address:788cf89c-090e-f232-af4b-94504abbc2a1} {Suffrage:Voter ID:server-fba31bce-63b2-1d27-8898-8934ad09d77b Address:fba31bce-63b2-1d27-8898-8934ad09d77b} {Suffrage:Voter ID:server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db Address:1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db}]" +2025-10-30T18:22:12.562Z [INFO] server-fba31bce-63b2-1d27-8898-8934ad09d77b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-788cf89c-090e-f232-af4b-94504abbc2a1 Address:788cf89c-090e-f232-af4b-94504abbc2a1} {Suffrage:Voter ID:server-fba31bce-63b2-1d27-8898-8934ad09d77b Address:fba31bce-63b2-1d27-8898-8934ad09d77b} {Suffrage:Voter ID:server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db Address:1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db}]" +2025-10-30T18:22:12.562Z [INFO] server-fba31bce-63b2-1d27-8898-8934ad09d77b: entering follower state: follower="Node at fba31bce-63b2-1d27-8898-8934ad09d77b [Follower]" leader-address= leader-id= +2025-10-30T18:22:12.562Z [INFO] server-788cf89c-090e-f232-af4b-94504abbc2a1: entering follower state: follower="Node at 788cf89c-090e-f232-af4b-94504abbc2a1 [Follower]" leader-address= leader-id= +2025-10-30T18:22:12.562Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-788cf89c-090e-f232-af4b-94504abbc2a1 Address:788cf89c-090e-f232-af4b-94504abbc2a1} {Suffrage:Voter ID:server-fba31bce-63b2-1d27-8898-8934ad09d77b Address:fba31bce-63b2-1d27-8898-8934ad09d77b} {Suffrage:Voter ID:server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db Address:1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db}]" +2025-10-30T18:22:12.562Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:12.562Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: entering follower state: follower="Node at 1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db [Follower]" leader-address= leader-id= +2025-10-30T18:22:13.235Z [WARN] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:13.235Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: entering candidate state: node="Node at 1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db [Candidate]" term=2 +2025-10-30T18:22:13.235Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: election won: term=2 tally=2 +2025-10-30T18:22:13.235Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: entering leader state: leader="Node at 1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db [Leader]" +2025-10-30T18:22:13.235Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: added peer, starting replication: peer=server-788cf89c-090e-f232-af4b-94504abbc2a1 +2025-10-30T18:22:13.235Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: added peer, starting replication: peer=server-fba31bce-63b2-1d27-8898-8934ad09d77b +2025-10-30T18:22:13.235Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: pipelining replication: peer="{Voter server-788cf89c-090e-f232-af4b-94504abbc2a1 788cf89c-090e-f232-af4b-94504abbc2a1}" +2025-10-30T18:22:13.235Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: pipelining replication: peer="{Voter server-fba31bce-63b2-1d27-8898-8934ad09d77b fba31bce-63b2-1d27-8898-8934ad09d77b}" +2025-10-30T18:22:14.240Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:13.235420738 +0000 UTC m=+56.308950041 (1 nodes), 672.592224ms from start of poll, 672.847774ms from cluster start. Timeout at 2025-10-30 18:22:14.240424223 +0000 UTC m=+57.313953515, 1.005003474s after stability +2025-10-30T18:22:14.241Z [INFO] creating new snapshot: path=/tmp/raft3539048265/snapshots/2-15-1761848534241.tmp +2025-10-30T18:22:14.241Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: copied to local snapshot: bytes=71 +2025-10-30T18:22:14.241Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: snapshot restore progress: id=2-15-1761848534241 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:14.241Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: restored user snapshot: index=15 +2025-10-30T18:22:14.241Z [ERROR] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: failed to get log: index=3 error="log not found" +2025-10-30T18:22:14.241Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: aborting pipeline replication: peer="{Voter server-fba31bce-63b2-1d27-8898-8934ad09d77b fba31bce-63b2-1d27-8898-8934ad09d77b}" +2025-10-30T18:22:14.241Z [ERROR] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: failed to get log: index=3 error="log not found" +2025-10-30T18:22:14.241Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: aborting pipeline replication: peer="{Voter server-788cf89c-090e-f232-af4b-94504abbc2a1 788cf89c-090e-f232-af4b-94504abbc2a1}" +2025-10-30T18:22:14.248Z [ERROR] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: failed to get log: index=3 error="log not found" +2025-10-30T18:22:14.249Z [ERROR] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: failed to get log: index=3 error="log not found" +2025-10-30T18:22:14.249Z [INFO] creating new snapshot: path=/tmp/raft835051109/snapshots/2-15-1761848534249.tmp +2025-10-30T18:22:14.250Z [INFO] creating new snapshot: path=/tmp/raft124151994/snapshots/2-15-1761848534250.tmp +2025-10-30T18:22:14.250Z [INFO] server-fba31bce-63b2-1d27-8898-8934ad09d77b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:14.250Z [INFO] server-788cf89c-090e-f232-af4b-94504abbc2a1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:14.250Z [INFO] server-fba31bce-63b2-1d27-8898-8934ad09d77b: copied to local snapshot: bytes=71 +2025-10-30T18:22:14.250Z [INFO] server-788cf89c-090e-f232-af4b-94504abbc2a1: copied to local snapshot: bytes=71 +2025-10-30T18:22:14.250Z [INFO] server-fba31bce-63b2-1d27-8898-8934ad09d77b: snapshot restore progress: id=2-15-1761848534250 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:14.250Z [INFO] server-fba31bce-63b2-1d27-8898-8934ad09d77b: Installed remote snapshot +2025-10-30T18:22:14.250Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: pipelining replication: peer="{Voter server-fba31bce-63b2-1d27-8898-8934ad09d77b fba31bce-63b2-1d27-8898-8934ad09d77b}" +2025-10-30T18:22:14.250Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:22:14.250Z [INFO] server-788cf89c-090e-f232-af4b-94504abbc2a1: snapshot restore progress: id=2-15-1761848534249 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:14.250Z [INFO] server-788cf89c-090e-f232-af4b-94504abbc2a1: Installed remote snapshot +2025-10-30T18:22:14.251Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: pipelining replication: peer="{Voter server-788cf89c-090e-f232-af4b-94504abbc2a1 788cf89c-090e-f232-af4b-94504abbc2a1}" +2025-10-30T18:22:15.255Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:22:14.25088629 +0000 UTC m=+57.324415594 (2 nodes), 707ns from start of poll, 1.688313327s from cluster start. Timeout at 2025-10-30 18:22:15.255898948 +0000 UTC m=+58.329428248, 1.005012654s after stability +2025-10-30T18:22:15.267Z [INFO] server-48458d17-a12a-b96f-d5d3-1b6430db0662: aborting pipeline replication: peer="{Voter server-3e13a0ac-ab57-28ea-1471-89e6dd0d332c 3e13a0ac-ab57-28ea-1471-89e6dd0d332c}" +2025-10-30T18:22:15.267Z [INFO] server-48458d17-a12a-b96f-d5d3-1b6430db0662: aborting pipeline replication: peer="{Voter server-7f4be064-70b5-633f-5c4f-a15eb3ccf98b 7f4be064-70b5-633f-5c4f-a15eb3ccf98b}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true -2025-10-30T18:18:24.812Z [INFO] server-28444ed3-dcd0-621a-7f09-9e964b12f478: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-28444ed3-dcd0-621a-7f09-9e964b12f478 Address:28444ed3-dcd0-621a-7f09-9e964b12f478} {Suffrage:Voter ID:server-8b667098-7eed-2842-f2d9-40fd2a8c15a6 Address:8b667098-7eed-2842-f2d9-40fd2a8c15a6} {Suffrage:Voter ID:server-f9af27c7-42d3-8b29-1523-3a4c4e758488 Address:f9af27c7-42d3-8b29-1523-3a4c4e758488}]" -2025-10-30T18:18:24.812Z [INFO] server-8b667098-7eed-2842-f2d9-40fd2a8c15a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-28444ed3-dcd0-621a-7f09-9e964b12f478 Address:28444ed3-dcd0-621a-7f09-9e964b12f478} {Suffrage:Voter ID:server-8b667098-7eed-2842-f2d9-40fd2a8c15a6 Address:8b667098-7eed-2842-f2d9-40fd2a8c15a6} {Suffrage:Voter ID:server-f9af27c7-42d3-8b29-1523-3a4c4e758488 Address:f9af27c7-42d3-8b29-1523-3a4c4e758488}]" -2025-10-30T18:18:24.812Z [INFO] server-28444ed3-dcd0-621a-7f09-9e964b12f478: entering follower state: follower="Node at 28444ed3-dcd0-621a-7f09-9e964b12f478 [Follower]" leader-address= leader-id= -2025-10-30T18:18:24.812Z [INFO] server-f9af27c7-42d3-8b29-1523-3a4c4e758488: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-28444ed3-dcd0-621a-7f09-9e964b12f478 Address:28444ed3-dcd0-621a-7f09-9e964b12f478} {Suffrage:Voter ID:server-8b667098-7eed-2842-f2d9-40fd2a8c15a6 Address:8b667098-7eed-2842-f2d9-40fd2a8c15a6} {Suffrage:Voter ID:server-f9af27c7-42d3-8b29-1523-3a4c4e758488 Address:f9af27c7-42d3-8b29-1523-3a4c4e758488}]" -2025-10-30T18:18:24.812Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:24.812Z [INFO] server-8b667098-7eed-2842-f2d9-40fd2a8c15a6: entering follower state: follower="Node at 8b667098-7eed-2842-f2d9-40fd2a8c15a6 [Follower]" leader-address= leader-id= -2025-10-30T18:18:24.812Z [INFO] server-f9af27c7-42d3-8b29-1523-3a4c4e758488: entering follower state: follower="Node at f9af27c7-42d3-8b29-1523-3a4c4e758488 [Follower]" leader-address= leader-id= -2025-10-30T18:18:25.336Z [WARN] server-f9af27c7-42d3-8b29-1523-3a4c4e758488: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:25.336Z [INFO] server-f9af27c7-42d3-8b29-1523-3a4c4e758488: entering candidate state: node="Node at f9af27c7-42d3-8b29-1523-3a4c4e758488 [Candidate]" term=2 -2025-10-30T18:18:25.336Z [INFO] server-f9af27c7-42d3-8b29-1523-3a4c4e758488: election won: term=2 tally=2 -2025-10-30T18:18:25.336Z [INFO] server-f9af27c7-42d3-8b29-1523-3a4c4e758488: entering leader state: leader="Node at f9af27c7-42d3-8b29-1523-3a4c4e758488 [Leader]" -2025-10-30T18:18:25.336Z [INFO] server-f9af27c7-42d3-8b29-1523-3a4c4e758488: added peer, starting replication: peer=server-28444ed3-dcd0-621a-7f09-9e964b12f478 -2025-10-30T18:18:25.336Z [INFO] server-f9af27c7-42d3-8b29-1523-3a4c4e758488: added peer, starting replication: peer=server-8b667098-7eed-2842-f2d9-40fd2a8c15a6 -2025-10-30T18:18:25.336Z [INFO] server-f9af27c7-42d3-8b29-1523-3a4c4e758488: pipelining replication: peer="{Voter server-8b667098-7eed-2842-f2d9-40fd2a8c15a6 8b667098-7eed-2842-f2d9-40fd2a8c15a6}" -2025-10-30T18:18:25.336Z [INFO] server-f9af27c7-42d3-8b29-1523-3a4c4e758488: pipelining replication: peer="{Voter server-28444ed3-dcd0-621a-7f09-9e964b12f478 28444ed3-dcd0-621a-7f09-9e964b12f478}" -2025-10-30T18:18:26.342Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:25.337002453 +0000 UTC m=+51.615676358 (1 nodes), 524.330736ms from start of poll, 524.515068ms from cluster start. Timeout at 2025-10-30 18:18:26.342005844 +0000 UTC m=+52.620679741, 1.005003383s after stability -2025-10-30T18:18:26.343Z [INFO] server-f9af27c7-42d3-8b29-1523-3a4c4e758488: starting snapshot up to: index=12 -2025-10-30T18:18:26.343Z [INFO] creating new snapshot: path=/tmp/raft1615071074/snapshots/2-12-1761848306343.tmp -2025-10-30T18:18:26.398Z [INFO] server-f9af27c7-42d3-8b29-1523-3a4c4e758488: snapshot complete up to: index=12 -2025-10-30T18:18:26.399Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d475804d-0f75-c1fe-7644-20bd6b8e50c4 Address:d475804d-0f75-c1fe-7644-20bd6b8e50c4} {Suffrage:Voter ID:server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad Address:7bb1fef8-5766-efd8-52fd-24a1cd26f2ad} {Suffrage:Voter ID:server-2f12b407-dbe3-3e38-9511-92ed05505016 Address:2f12b407-dbe3-3e38-9511-92ed05505016}]" -2025-10-30T18:18:26.399Z [INFO] server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d475804d-0f75-c1fe-7644-20bd6b8e50c4 Address:d475804d-0f75-c1fe-7644-20bd6b8e50c4} {Suffrage:Voter ID:server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad Address:7bb1fef8-5766-efd8-52fd-24a1cd26f2ad} {Suffrage:Voter ID:server-2f12b407-dbe3-3e38-9511-92ed05505016 Address:2f12b407-dbe3-3e38-9511-92ed05505016}]" -2025-10-30T18:18:26.399Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: entering follower state: follower="Node at d475804d-0f75-c1fe-7644-20bd6b8e50c4 [Follower]" leader-address= leader-id= -2025-10-30T18:18:26.399Z [INFO] server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad: entering follower state: follower="Node at 7bb1fef8-5766-efd8-52fd-24a1cd26f2ad [Follower]" leader-address= leader-id= -2025-10-30T18:18:26.399Z [INFO] server-2f12b407-dbe3-3e38-9511-92ed05505016: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d475804d-0f75-c1fe-7644-20bd6b8e50c4 Address:d475804d-0f75-c1fe-7644-20bd6b8e50c4} {Suffrage:Voter ID:server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad Address:7bb1fef8-5766-efd8-52fd-24a1cd26f2ad} {Suffrage:Voter ID:server-2f12b407-dbe3-3e38-9511-92ed05505016 Address:2f12b407-dbe3-3e38-9511-92ed05505016}]" -2025-10-30T18:18:26.399Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:26.399Z [INFO] server-2f12b407-dbe3-3e38-9511-92ed05505016: entering follower state: follower="Node at 2f12b407-dbe3-3e38-9511-92ed05505016 [Follower]" leader-address= leader-id= -2025-10-30T18:18:26.951Z [WARN] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:26.951Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: entering candidate state: node="Node at d475804d-0f75-c1fe-7644-20bd6b8e50c4 [Candidate]" term=2 -2025-10-30T18:18:26.951Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: election won: term=2 tally=2 -2025-10-30T18:18:26.951Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: entering leader state: leader="Node at d475804d-0f75-c1fe-7644-20bd6b8e50c4 [Leader]" -2025-10-30T18:18:26.951Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: added peer, starting replication: peer=server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad -2025-10-30T18:18:26.951Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: added peer, starting replication: peer=server-2f12b407-dbe3-3e38-9511-92ed05505016 -2025-10-30T18:18:26.951Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: pipelining replication: peer="{Voter server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad 7bb1fef8-5766-efd8-52fd-24a1cd26f2ad}" -2025-10-30T18:18:26.951Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: pipelining replication: peer="{Voter server-2f12b407-dbe3-3e38-9511-92ed05505016 2f12b407-dbe3-3e38-9511-92ed05505016}" -2025-10-30T18:18:27.957Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:26.951260439 +0000 UTC m=+53.229934336 (1 nodes), 551.448618ms from start of poll, 551.713368ms from cluster start. Timeout at 2025-10-30 18:18:27.956264579 +0000 UTC m=+54.234938476, 1.00500414s after stability -2025-10-30T18:18:27.957Z [INFO] creating new snapshot: path=/tmp/raft555533492/snapshots/2-15-1761848307957.tmp -2025-10-30T18:18:28.001Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: copied to local snapshot: bytes=71 -2025-10-30T18:18:28.001Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: snapshot restore progress: id=2-15-1761848307957 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:28.001Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: removing all old logs from log store -2025-10-30T18:18:28.001Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: compacting logs: from=1 to=2 -2025-10-30T18:18:28.001Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: restored user snapshot: index=15 -2025-10-30T18:18:28.001Z [ERROR] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: failed to get log: index=2 error="log not found" -2025-10-30T18:18:28.001Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: aborting pipeline replication: peer="{Voter server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad 7bb1fef8-5766-efd8-52fd-24a1cd26f2ad}" -2025-10-30T18:18:28.001Z [ERROR] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: failed to get log: index=2 error="log not found" -2025-10-30T18:18:28.001Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: aborting pipeline replication: peer="{Voter server-2f12b407-dbe3-3e38-9511-92ed05505016 2f12b407-dbe3-3e38-9511-92ed05505016}" -2025-10-30T18:18:28.008Z [ERROR] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: failed to get log: index=2 error="log not found" -2025-10-30T18:18:28.008Z [ERROR] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: failed to get log: index=2 error="log not found" -2025-10-30T18:18:28.008Z [INFO] creating new snapshot: path=/tmp/raft939547717/snapshots/2-15-1761848308008.tmp -2025-10-30T18:18:28.008Z [INFO] creating new snapshot: path=/tmp/raft563965688/snapshots/2-15-1761848308008.tmp -2025-10-30T18:18:28.008Z [INFO] server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:28.008Z [INFO] server-2f12b407-dbe3-3e38-9511-92ed05505016: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:28.101Z [INFO] server-2f12b407-dbe3-3e38-9511-92ed05505016: copied to local snapshot: bytes=71 -2025-10-30T18:18:28.101Z [INFO] server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad: copied to local snapshot: bytes=71 -2025-10-30T18:18:28.101Z [INFO] server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad: snapshot restore progress: id=2-15-1761848308008 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:28.101Z [INFO] server-2f12b407-dbe3-3e38-9511-92ed05505016: snapshot restore progress: id=2-15-1761848308008 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:28.101Z [INFO] server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad: removing all old logs from log store -2025-10-30T18:18:28.101Z [INFO] server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad: compacting logs: from=1 to=2 -2025-10-30T18:18:28.101Z [INFO] server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad: Installed remote snapshot -2025-10-30T18:18:28.101Z [INFO] server-2f12b407-dbe3-3e38-9511-92ed05505016: removing all old logs from log store -2025-10-30T18:18:28.101Z [INFO] server-2f12b407-dbe3-3e38-9511-92ed05505016: compacting logs: from=1 to=2 -2025-10-30T18:18:28.101Z [INFO] server-2f12b407-dbe3-3e38-9511-92ed05505016: Installed remote snapshot -2025-10-30T18:18:28.101Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: pipelining replication: peer="{Voter server-7bb1fef8-5766-efd8-52fd-24a1cd26f2ad 7bb1fef8-5766-efd8-52fd-24a1cd26f2ad}" -2025-10-30T18:18:28.101Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:18:28.101Z [INFO] server-d475804d-0f75-c1fe-7644-20bd6b8e50c4: pipelining replication: peer="{Voter server-2f12b407-dbe3-3e38-9511-92ed05505016 2f12b407-dbe3-3e38-9511-92ed05505016}" -2025-10-30T18:18:29.107Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:18:28.101544281 +0000 UTC m=+54.380218178 (2 nodes), 771ns from start of poll, 1.70199721s from cluster start. Timeout at 2025-10-30 18:18:29.106558365 +0000 UTC m=+55.385232280, 1.005014102s after stability -2025-10-30T18:18:29.118Z [INFO] server-f9af27c7-42d3-8b29-1523-3a4c4e758488: aborting pipeline replication: peer="{Voter server-8b667098-7eed-2842-f2d9-40fd2a8c15a6 8b667098-7eed-2842-f2d9-40fd2a8c15a6}" -2025-10-30T18:18:29.118Z [INFO] server-f9af27c7-42d3-8b29-1523-3a4c4e758488: aborting pipeline replication: peer="{Voter server-28444ed3-dcd0-621a-7f09-9e964b12f478 28444ed3-dcd0-621a-7f09-9e964b12f478}" +2025-10-30T18:22:15.268Z [INFO] server-0559936c-a763-20e1-47bf-3eba83174363: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0559936c-a763-20e1-47bf-3eba83174363 Address:0559936c-a763-20e1-47bf-3eba83174363} {Suffrage:Voter ID:server-99f54e10-0c6b-7468-f9c1-fd6a436a1a1f Address:99f54e10-0c6b-7468-f9c1-fd6a436a1a1f} {Suffrage:Voter ID:server-939a2bd4-0958-9404-b228-793f699887fd Address:939a2bd4-0958-9404-b228-793f699887fd}]" +2025-10-30T18:22:15.269Z [INFO] server-0559936c-a763-20e1-47bf-3eba83174363: entering follower state: follower="Node at 0559936c-a763-20e1-47bf-3eba83174363 [Follower]" leader-address= leader-id= +2025-10-30T18:22:15.269Z [INFO] server-99f54e10-0c6b-7468-f9c1-fd6a436a1a1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0559936c-a763-20e1-47bf-3eba83174363 Address:0559936c-a763-20e1-47bf-3eba83174363} {Suffrage:Voter ID:server-99f54e10-0c6b-7468-f9c1-fd6a436a1a1f Address:99f54e10-0c6b-7468-f9c1-fd6a436a1a1f} {Suffrage:Voter ID:server-939a2bd4-0958-9404-b228-793f699887fd Address:939a2bd4-0958-9404-b228-793f699887fd}]" +2025-10-30T18:22:15.269Z [INFO] server-99f54e10-0c6b-7468-f9c1-fd6a436a1a1f: entering follower state: follower="Node at 99f54e10-0c6b-7468-f9c1-fd6a436a1a1f [Follower]" leader-address= leader-id= +2025-10-30T18:22:15.269Z [INFO] server-939a2bd4-0958-9404-b228-793f699887fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0559936c-a763-20e1-47bf-3eba83174363 Address:0559936c-a763-20e1-47bf-3eba83174363} {Suffrage:Voter ID:server-99f54e10-0c6b-7468-f9c1-fd6a436a1a1f Address:99f54e10-0c6b-7468-f9c1-fd6a436a1a1f} {Suffrage:Voter ID:server-939a2bd4-0958-9404-b228-793f699887fd Address:939a2bd4-0958-9404-b228-793f699887fd}]" +2025-10-30T18:22:15.269Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:15.269Z [INFO] server-939a2bd4-0958-9404-b228-793f699887fd: entering follower state: follower="Node at 939a2bd4-0958-9404-b228-793f699887fd [Follower]" leader-address= leader-id= +2025-10-30T18:22:15.792Z [WARN] server-0559936c-a763-20e1-47bf-3eba83174363: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:15.792Z [INFO] server-0559936c-a763-20e1-47bf-3eba83174363: entering candidate state: node="Node at 0559936c-a763-20e1-47bf-3eba83174363 [Candidate]" term=2 +2025-10-30T18:22:15.792Z [INFO] server-0559936c-a763-20e1-47bf-3eba83174363: election won: term=2 tally=2 +2025-10-30T18:22:15.792Z [INFO] server-0559936c-a763-20e1-47bf-3eba83174363: entering leader state: leader="Node at 0559936c-a763-20e1-47bf-3eba83174363 [Leader]" +2025-10-30T18:22:15.792Z [INFO] server-0559936c-a763-20e1-47bf-3eba83174363: added peer, starting replication: peer=server-99f54e10-0c6b-7468-f9c1-fd6a436a1a1f +2025-10-30T18:22:15.792Z [INFO] server-0559936c-a763-20e1-47bf-3eba83174363: added peer, starting replication: peer=server-939a2bd4-0958-9404-b228-793f699887fd +2025-10-30T18:22:15.792Z [INFO] server-0559936c-a763-20e1-47bf-3eba83174363: pipelining replication: peer="{Voter server-99f54e10-0c6b-7468-f9c1-fd6a436a1a1f 99f54e10-0c6b-7468-f9c1-fd6a436a1a1f}" +2025-10-30T18:22:15.792Z [INFO] server-0559936c-a763-20e1-47bf-3eba83174363: pipelining replication: peer="{Voter server-939a2bd4-0958-9404-b228-793f699887fd 939a2bd4-0958-9404-b228-793f699887fd}" +2025-10-30T18:22:16.797Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:15.792670981 +0000 UTC m=+58.866200276 (1 nodes), 523.461127ms from start of poll, 523.808436ms from cluster start. Timeout at 2025-10-30 18:22:16.797674641 +0000 UTC m=+59.871203935, 1.005003659s after stability +2025-10-30T18:22:16.798Z [INFO] server-0559936c-a763-20e1-47bf-3eba83174363: starting snapshot up to: index=12 +2025-10-30T18:22:16.798Z [INFO] creating new snapshot: path=/tmp/raft2526416208/snapshots/2-12-1761848536798.tmp +2025-10-30T18:22:16.798Z [INFO] server-0559936c-a763-20e1-47bf-3eba83174363: snapshot complete up to: index=12 +2025-10-30T18:22:16.799Z [INFO] server-733c507b-20a5-105e-d60d-6fa40e90cfd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-733c507b-20a5-105e-d60d-6fa40e90cfd1 Address:733c507b-20a5-105e-d60d-6fa40e90cfd1} {Suffrage:Voter ID:server-e2668a63-27b7-abe8-f831-048f62464050 Address:e2668a63-27b7-abe8-f831-048f62464050} {Suffrage:Voter ID:server-691e8f75-eabd-0958-0932-0bda3853fff9 Address:691e8f75-eabd-0958-0932-0bda3853fff9}]" +2025-10-30T18:22:16.799Z [INFO] server-733c507b-20a5-105e-d60d-6fa40e90cfd1: entering follower state: follower="Node at 733c507b-20a5-105e-d60d-6fa40e90cfd1 [Follower]" leader-address= leader-id= +2025-10-30T18:22:16.799Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-733c507b-20a5-105e-d60d-6fa40e90cfd1 Address:733c507b-20a5-105e-d60d-6fa40e90cfd1} {Suffrage:Voter ID:server-e2668a63-27b7-abe8-f831-048f62464050 Address:e2668a63-27b7-abe8-f831-048f62464050} {Suffrage:Voter ID:server-691e8f75-eabd-0958-0932-0bda3853fff9 Address:691e8f75-eabd-0958-0932-0bda3853fff9}]" +2025-10-30T18:22:16.799Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: entering follower state: follower="Node at e2668a63-27b7-abe8-f831-048f62464050 [Follower]" leader-address= leader-id= +2025-10-30T18:22:16.799Z [INFO] server-691e8f75-eabd-0958-0932-0bda3853fff9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-733c507b-20a5-105e-d60d-6fa40e90cfd1 Address:733c507b-20a5-105e-d60d-6fa40e90cfd1} {Suffrage:Voter ID:server-e2668a63-27b7-abe8-f831-048f62464050 Address:e2668a63-27b7-abe8-f831-048f62464050} {Suffrage:Voter ID:server-691e8f75-eabd-0958-0932-0bda3853fff9 Address:691e8f75-eabd-0958-0932-0bda3853fff9}]" +2025-10-30T18:22:16.799Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:16.799Z [INFO] server-691e8f75-eabd-0958-0932-0bda3853fff9: entering follower state: follower="Node at 691e8f75-eabd-0958-0932-0bda3853fff9 [Follower]" leader-address= leader-id= +2025-10-30T18:22:17.510Z [WARN] server-e2668a63-27b7-abe8-f831-048f62464050: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:17.510Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: entering candidate state: node="Node at e2668a63-27b7-abe8-f831-048f62464050 [Candidate]" term=2 +2025-10-30T18:22:17.510Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: election won: term=2 tally=2 +2025-10-30T18:22:17.510Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: entering leader state: leader="Node at e2668a63-27b7-abe8-f831-048f62464050 [Leader]" +2025-10-30T18:22:17.510Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: added peer, starting replication: peer=server-733c507b-20a5-105e-d60d-6fa40e90cfd1 +2025-10-30T18:22:17.510Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: added peer, starting replication: peer=server-691e8f75-eabd-0958-0932-0bda3853fff9 +2025-10-30T18:22:17.510Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: pipelining replication: peer="{Voter server-733c507b-20a5-105e-d60d-6fa40e90cfd1 733c507b-20a5-105e-d60d-6fa40e90cfd1}" +2025-10-30T18:22:17.510Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: pipelining replication: peer="{Voter server-691e8f75-eabd-0958-0932-0bda3853fff9 691e8f75-eabd-0958-0932-0bda3853fff9}" +2025-10-30T18:22:18.515Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:17.51033697 +0000 UTC m=+60.583866270 (1 nodes), 710.436622ms from start of poll, 710.758837ms from cluster start. Timeout at 2025-10-30 18:22:18.515339946 +0000 UTC m=+61.588869239, 1.005002969s after stability +2025-10-30T18:22:18.515Z [INFO] creating new snapshot: path=/tmp/raft4143589905/snapshots/2-15-1761848538515.tmp +2025-10-30T18:22:18.516Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: copied to local snapshot: bytes=71 +2025-10-30T18:22:18.516Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: snapshot restore progress: id=2-15-1761848538515 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:18.516Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: removing all old logs from log store +2025-10-30T18:22:18.516Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: compacting logs: from=1 to=2 +2025-10-30T18:22:18.516Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: restored user snapshot: index=15 +2025-10-30T18:22:18.516Z [ERROR] server-e2668a63-27b7-abe8-f831-048f62464050: failed to get log: index=2 error="log not found" +2025-10-30T18:22:18.516Z [ERROR] server-e2668a63-27b7-abe8-f831-048f62464050: failed to get log: index=2 error="log not found" +2025-10-30T18:22:18.516Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: aborting pipeline replication: peer="{Voter server-733c507b-20a5-105e-d60d-6fa40e90cfd1 733c507b-20a5-105e-d60d-6fa40e90cfd1}" +2025-10-30T18:22:18.516Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: aborting pipeline replication: peer="{Voter server-691e8f75-eabd-0958-0932-0bda3853fff9 691e8f75-eabd-0958-0932-0bda3853fff9}" +2025-10-30T18:22:18.524Z [ERROR] server-e2668a63-27b7-abe8-f831-048f62464050: failed to get log: index=2 error="log not found" +2025-10-30T18:22:18.524Z [ERROR] server-e2668a63-27b7-abe8-f831-048f62464050: failed to get log: index=2 error="log not found" +2025-10-30T18:22:18.524Z [INFO] creating new snapshot: path=/tmp/raft1906094058/snapshots/2-15-1761848538524.tmp +2025-10-30T18:22:18.524Z [INFO] creating new snapshot: path=/tmp/raft2269619720/snapshots/2-15-1761848538524.tmp +2025-10-30T18:22:18.524Z [INFO] server-733c507b-20a5-105e-d60d-6fa40e90cfd1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:18.524Z [INFO] server-691e8f75-eabd-0958-0932-0bda3853fff9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:18.525Z [INFO] server-733c507b-20a5-105e-d60d-6fa40e90cfd1: copied to local snapshot: bytes=71 +2025-10-30T18:22:18.525Z [INFO] server-691e8f75-eabd-0958-0932-0bda3853fff9: copied to local snapshot: bytes=71 +2025-10-30T18:22:18.525Z [INFO] server-733c507b-20a5-105e-d60d-6fa40e90cfd1: snapshot restore progress: id=2-15-1761848538524 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:18.525Z [INFO] server-733c507b-20a5-105e-d60d-6fa40e90cfd1: removing all old logs from log store +2025-10-30T18:22:18.525Z [INFO] server-733c507b-20a5-105e-d60d-6fa40e90cfd1: compacting logs: from=1 to=2 +2025-10-30T18:22:18.525Z [INFO] server-733c507b-20a5-105e-d60d-6fa40e90cfd1: Installed remote snapshot +2025-10-30T18:22:18.525Z [INFO] server-691e8f75-eabd-0958-0932-0bda3853fff9: snapshot restore progress: id=2-15-1761848538524 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:18.525Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: pipelining replication: peer="{Voter server-733c507b-20a5-105e-d60d-6fa40e90cfd1 733c507b-20a5-105e-d60d-6fa40e90cfd1}" +2025-10-30T18:22:18.525Z [INFO] server-691e8f75-eabd-0958-0932-0bda3853fff9: removing all old logs from log store +2025-10-30T18:22:18.525Z [INFO] server-691e8f75-eabd-0958-0932-0bda3853fff9: compacting logs: from=1 to=2 +2025-10-30T18:22:18.525Z [INFO] server-691e8f75-eabd-0958-0932-0bda3853fff9: Installed remote snapshot +2025-10-30T18:22:18.525Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:22:18.525Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: pipelining replication: peer="{Voter server-691e8f75-eabd-0958-0932-0bda3853fff9 691e8f75-eabd-0958-0932-0bda3853fff9}" +2025-10-30T18:22:19.530Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:22:18.525453629 +0000 UTC m=+61.598982922 (2 nodes), 544ns from start of poll, 1.725875489s from cluster start. Timeout at 2025-10-30 18:22:19.530484024 +0000 UTC m=+62.604013340, 1.005030418s after stability +2025-10-30T18:22:19.542Z [INFO] server-0559936c-a763-20e1-47bf-3eba83174363: aborting pipeline replication: peer="{Voter server-99f54e10-0c6b-7468-f9c1-fd6a436a1a1f 99f54e10-0c6b-7468-f9c1-fd6a436a1a1f}" +2025-10-30T18:22:19.542Z [INFO] server-0559936c-a763-20e1-47bf-3eba83174363: aborting pipeline replication: peer="{Voter server-939a2bd4-0958-9404-b228-793f699887fd 939a2bd4-0958-9404-b228-793f699887fd}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false -2025-10-30T18:18:29.119Z [INFO] server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd Address:720f909d-2ddb-b4e7-ab96-41817bb3c6fd} {Suffrage:Voter ID:server-6cb3b18d-9674-c289-dd01-d381a2c35fa6 Address:6cb3b18d-9674-c289-dd01-d381a2c35fa6} {Suffrage:Voter ID:server-6980861d-ae8c-eac8-e365-f435c847cd8c Address:6980861d-ae8c-eac8-e365-f435c847cd8c}]" -2025-10-30T18:18:29.119Z [INFO] server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd: entering follower state: follower="Node at 720f909d-2ddb-b4e7-ab96-41817bb3c6fd [Follower]" leader-address= leader-id= -2025-10-30T18:18:29.119Z [INFO] server-6cb3b18d-9674-c289-dd01-d381a2c35fa6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd Address:720f909d-2ddb-b4e7-ab96-41817bb3c6fd} {Suffrage:Voter ID:server-6cb3b18d-9674-c289-dd01-d381a2c35fa6 Address:6cb3b18d-9674-c289-dd01-d381a2c35fa6} {Suffrage:Voter ID:server-6980861d-ae8c-eac8-e365-f435c847cd8c Address:6980861d-ae8c-eac8-e365-f435c847cd8c}]" -2025-10-30T18:18:29.119Z [INFO] server-6cb3b18d-9674-c289-dd01-d381a2c35fa6: entering follower state: follower="Node at 6cb3b18d-9674-c289-dd01-d381a2c35fa6 [Follower]" leader-address= leader-id= -2025-10-30T18:18:29.120Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd Address:720f909d-2ddb-b4e7-ab96-41817bb3c6fd} {Suffrage:Voter ID:server-6cb3b18d-9674-c289-dd01-d381a2c35fa6 Address:6cb3b18d-9674-c289-dd01-d381a2c35fa6} {Suffrage:Voter ID:server-6980861d-ae8c-eac8-e365-f435c847cd8c Address:6980861d-ae8c-eac8-e365-f435c847cd8c}]" -2025-10-30T18:18:29.120Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:29.120Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: entering follower state: follower="Node at 6980861d-ae8c-eac8-e365-f435c847cd8c [Follower]" leader-address= leader-id= -2025-10-30T18:18:29.787Z [WARN] server-6980861d-ae8c-eac8-e365-f435c847cd8c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:29.787Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: entering candidate state: node="Node at 6980861d-ae8c-eac8-e365-f435c847cd8c [Candidate]" term=2 -2025-10-30T18:18:29.787Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: election won: term=2 tally=2 -2025-10-30T18:18:29.787Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: entering leader state: leader="Node at 6980861d-ae8c-eac8-e365-f435c847cd8c [Leader]" -2025-10-30T18:18:29.787Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: added peer, starting replication: peer=server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd -2025-10-30T18:18:29.787Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: added peer, starting replication: peer=server-6cb3b18d-9674-c289-dd01-d381a2c35fa6 -2025-10-30T18:18:29.787Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: pipelining replication: peer="{Voter server-6cb3b18d-9674-c289-dd01-d381a2c35fa6 6cb3b18d-9674-c289-dd01-d381a2c35fa6}" -2025-10-30T18:18:29.787Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: pipelining replication: peer="{Voter server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd 720f909d-2ddb-b4e7-ab96-41817bb3c6fd}" -2025-10-30T18:18:30.793Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:29.78786796 +0000 UTC m=+56.066541858 (1 nodes), 667.747592ms from start of poll, 668.130229ms from cluster start. Timeout at 2025-10-30 18:18:30.792873827 +0000 UTC m=+57.071547725, 1.005005867s after stability -2025-10-30T18:18:30.794Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: starting snapshot up to: index=12 -2025-10-30T18:18:30.794Z [INFO] creating new snapshot: path=/tmp/raft4184190451/snapshots/2-12-1761848310794.tmp -2025-10-30T18:18:30.811Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: snapshot complete up to: index=12 -2025-10-30T18:18:30.812Z [INFO] creating new snapshot: path=/tmp/raft4184190451/snapshots/2-113-1761848310812.tmp -2025-10-30T18:18:30.858Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: copied to local snapshot: bytes=71 -2025-10-30T18:18:30.859Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: snapshot restore progress: id=2-113-1761848310812 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:30.859Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: restored user snapshot: index=113 -2025-10-30T18:18:30.859Z [ERROR] server-6980861d-ae8c-eac8-e365-f435c847cd8c: failed to get log: index=23 error="log not found" -2025-10-30T18:18:30.859Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: aborting pipeline replication: peer="{Voter server-6cb3b18d-9674-c289-dd01-d381a2c35fa6 6cb3b18d-9674-c289-dd01-d381a2c35fa6}" -2025-10-30T18:18:30.859Z [ERROR] server-6980861d-ae8c-eac8-e365-f435c847cd8c: failed to get log: index=23 error="log not found" -2025-10-30T18:18:30.859Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: aborting pipeline replication: peer="{Voter server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd 720f909d-2ddb-b4e7-ab96-41817bb3c6fd}" -2025-10-30T18:18:30.868Z [ERROR] server-6980861d-ae8c-eac8-e365-f435c847cd8c: failed to get log: index=23 error="log not found" -2025-10-30T18:18:30.868Z [INFO] creating new snapshot: path=/tmp/raft841580653/snapshots/2-113-1761848310868.tmp -2025-10-30T18:18:30.868Z [INFO] server-6cb3b18d-9674-c289-dd01-d381a2c35fa6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:30.870Z [ERROR] server-6980861d-ae8c-eac8-e365-f435c847cd8c: failed to get log: index=23 error="log not found" -2025-10-30T18:18:30.870Z [INFO] creating new snapshot: path=/tmp/raft2020485141/snapshots/2-113-1761848310870.tmp -2025-10-30T18:18:30.870Z [INFO] server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:30.965Z [INFO] server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd: copied to local snapshot: bytes=71 -2025-10-30T18:18:30.965Z [INFO] server-6cb3b18d-9674-c289-dd01-d381a2c35fa6: copied to local snapshot: bytes=71 -2025-10-30T18:18:30.965Z [INFO] server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd: snapshot restore progress: id=2-113-1761848310870 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:30.965Z [INFO] server-6cb3b18d-9674-c289-dd01-d381a2c35fa6: snapshot restore progress: id=2-113-1761848310868 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:30.965Z [INFO] server-6cb3b18d-9674-c289-dd01-d381a2c35fa6: Installed remote snapshot -2025-10-30T18:18:30.965Z [INFO] server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd: Installed remote snapshot -2025-10-30T18:18:30.965Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: pipelining replication: peer="{Voter server-6cb3b18d-9674-c289-dd01-d381a2c35fa6 6cb3b18d-9674-c289-dd01-d381a2c35fa6}" -2025-10-30T18:18:30.965Z [WARN] server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd: skipping application of old log: index=113 -2025-10-30T18:18:30.965Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:18:30.965Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: pipelining replication: peer="{Voter server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd 720f909d-2ddb-b4e7-ab96-41817bb3c6fd}" -2025-10-30T18:18:31.971Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:18:30.965662919 +0000 UTC m=+57.244336817 (2 nodes), 528ns from start of poll, 1.845925188s from cluster start. Timeout at 2025-10-30 18:18:31.970678629 +0000 UTC m=+58.249352544, 1.005015727s after stability -2025-10-30T18:18:31.982Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: aborting pipeline replication: peer="{Voter server-6cb3b18d-9674-c289-dd01-d381a2c35fa6 6cb3b18d-9674-c289-dd01-d381a2c35fa6}" -2025-10-30T18:18:31.982Z [INFO] server-6980861d-ae8c-eac8-e365-f435c847cd8c: aborting pipeline replication: peer="{Voter server-720f909d-2ddb-b4e7-ab96-41817bb3c6fd 720f909d-2ddb-b4e7-ab96-41817bb3c6fd}" +2025-10-30T18:22:19.543Z [INFO] server-b659f1bf-72c4-0d06-bd80-f1acffd977e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b659f1bf-72c4-0d06-bd80-f1acffd977e3 Address:b659f1bf-72c4-0d06-bd80-f1acffd977e3} {Suffrage:Voter ID:server-5513ec05-df46-e160-a38f-f8b0bc7631c8 Address:5513ec05-df46-e160-a38f-f8b0bc7631c8} {Suffrage:Voter ID:server-b0269f33-38a2-4bba-435f-744f57720432 Address:b0269f33-38a2-4bba-435f-744f57720432}]" +2025-10-30T18:22:19.543Z [INFO] server-b659f1bf-72c4-0d06-bd80-f1acffd977e3: entering follower state: follower="Node at b659f1bf-72c4-0d06-bd80-f1acffd977e3 [Follower]" leader-address= leader-id= +2025-10-30T18:22:19.543Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b659f1bf-72c4-0d06-bd80-f1acffd977e3 Address:b659f1bf-72c4-0d06-bd80-f1acffd977e3} {Suffrage:Voter ID:server-5513ec05-df46-e160-a38f-f8b0bc7631c8 Address:5513ec05-df46-e160-a38f-f8b0bc7631c8} {Suffrage:Voter ID:server-b0269f33-38a2-4bba-435f-744f57720432 Address:b0269f33-38a2-4bba-435f-744f57720432}]" +2025-10-30T18:22:19.543Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: entering follower state: follower="Node at 5513ec05-df46-e160-a38f-f8b0bc7631c8 [Follower]" leader-address= leader-id= +2025-10-30T18:22:19.543Z [INFO] server-b0269f33-38a2-4bba-435f-744f57720432: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b659f1bf-72c4-0d06-bd80-f1acffd977e3 Address:b659f1bf-72c4-0d06-bd80-f1acffd977e3} {Suffrage:Voter ID:server-5513ec05-df46-e160-a38f-f8b0bc7631c8 Address:5513ec05-df46-e160-a38f-f8b0bc7631c8} {Suffrage:Voter ID:server-b0269f33-38a2-4bba-435f-744f57720432 Address:b0269f33-38a2-4bba-435f-744f57720432}]" +2025-10-30T18:22:19.543Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:19.543Z [INFO] server-b0269f33-38a2-4bba-435f-744f57720432: entering follower state: follower="Node at b0269f33-38a2-4bba-435f-744f57720432 [Follower]" leader-address= leader-id= +2025-10-30T18:22:20.193Z [WARN] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:20.193Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: entering candidate state: node="Node at 5513ec05-df46-e160-a38f-f8b0bc7631c8 [Candidate]" term=2 +2025-10-30T18:22:20.193Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: election won: term=2 tally=2 +2025-10-30T18:22:20.193Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: entering leader state: leader="Node at 5513ec05-df46-e160-a38f-f8b0bc7631c8 [Leader]" +2025-10-30T18:22:20.193Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: added peer, starting replication: peer=server-b659f1bf-72c4-0d06-bd80-f1acffd977e3 +2025-10-30T18:22:20.193Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: added peer, starting replication: peer=server-b0269f33-38a2-4bba-435f-744f57720432 +2025-10-30T18:22:20.193Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: pipelining replication: peer="{Voter server-b0269f33-38a2-4bba-435f-744f57720432 b0269f33-38a2-4bba-435f-744f57720432}" +2025-10-30T18:22:20.193Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: pipelining replication: peer="{Voter server-b659f1bf-72c4-0d06-bd80-f1acffd977e3 b659f1bf-72c4-0d06-bd80-f1acffd977e3}" +2025-10-30T18:22:21.199Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:20.193458302 +0000 UTC m=+63.266987590 (1 nodes), 649.46891ms from start of poll, 649.82658ms from cluster start. Timeout at 2025-10-30 18:22:21.198459525 +0000 UTC m=+64.271988818, 1.005001228s after stability +2025-10-30T18:22:21.199Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: starting snapshot up to: index=12 +2025-10-30T18:22:21.199Z [INFO] creating new snapshot: path=/tmp/raft1343360849/snapshots/2-12-1761848541199.tmp +2025-10-30T18:22:21.200Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: snapshot complete up to: index=12 +2025-10-30T18:22:21.200Z [INFO] creating new snapshot: path=/tmp/raft1343360849/snapshots/2-113-1761848541200.tmp +2025-10-30T18:22:21.201Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: copied to local snapshot: bytes=71 +2025-10-30T18:22:21.201Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: snapshot restore progress: id=2-113-1761848541200 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:21.201Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: restored user snapshot: index=113 +2025-10-30T18:22:21.201Z [ERROR] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: failed to get log: index=23 error="log not found" +2025-10-30T18:22:21.201Z [ERROR] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: failed to get log: index=23 error="log not found" +2025-10-30T18:22:21.201Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: aborting pipeline replication: peer="{Voter server-b659f1bf-72c4-0d06-bd80-f1acffd977e3 b659f1bf-72c4-0d06-bd80-f1acffd977e3}" +2025-10-30T18:22:21.201Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: aborting pipeline replication: peer="{Voter server-b0269f33-38a2-4bba-435f-744f57720432 b0269f33-38a2-4bba-435f-744f57720432}" +2025-10-30T18:22:21.210Z [ERROR] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: failed to get log: index=23 error="log not found" +2025-10-30T18:22:21.210Z [ERROR] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: failed to get log: index=23 error="log not found" +2025-10-30T18:22:21.210Z [INFO] creating new snapshot: path=/tmp/raft2554589469/snapshots/2-113-1761848541210.tmp +2025-10-30T18:22:21.210Z [INFO] creating new snapshot: path=/tmp/raft2204020646/snapshots/2-113-1761848541210.tmp +2025-10-30T18:22:21.211Z [INFO] server-b0269f33-38a2-4bba-435f-744f57720432: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:21.211Z [INFO] server-b659f1bf-72c4-0d06-bd80-f1acffd977e3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:21.211Z [INFO] server-b0269f33-38a2-4bba-435f-744f57720432: copied to local snapshot: bytes=71 +2025-10-30T18:22:21.211Z [INFO] server-b659f1bf-72c4-0d06-bd80-f1acffd977e3: copied to local snapshot: bytes=71 +2025-10-30T18:22:21.211Z [INFO] server-b0269f33-38a2-4bba-435f-744f57720432: snapshot restore progress: id=2-113-1761848541210 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:21.211Z [INFO] server-b0269f33-38a2-4bba-435f-744f57720432: Installed remote snapshot +2025-10-30T18:22:21.211Z [WARN] server-b0269f33-38a2-4bba-435f-744f57720432: skipping application of old log: index=22 +2025-10-30T18:22:21.211Z [INFO] server-b659f1bf-72c4-0d06-bd80-f1acffd977e3: snapshot restore progress: id=2-113-1761848541210 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:21.211Z [INFO] server-b659f1bf-72c4-0d06-bd80-f1acffd977e3: Installed remote snapshot +2025-10-30T18:22:21.211Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: pipelining replication: peer="{Voter server-b0269f33-38a2-4bba-435f-744f57720432 b0269f33-38a2-4bba-435f-744f57720432}" +2025-10-30T18:22:21.211Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:22:21.211Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: pipelining replication: peer="{Voter server-b659f1bf-72c4-0d06-bd80-f1acffd977e3 b659f1bf-72c4-0d06-bd80-f1acffd977e3}" +2025-10-30T18:22:22.216Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:22:21.211881323 +0000 UTC m=+64.285410619 (2 nodes), 892ns from start of poll, 1.668249609s from cluster start. Timeout at 2025-10-30 18:22:22.216896714 +0000 UTC m=+65.290426030, 1.005015411s after stability +2025-10-30T18:22:22.228Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: aborting pipeline replication: peer="{Voter server-b0269f33-38a2-4bba-435f-744f57720432 b0269f33-38a2-4bba-435f-744f57720432}" +2025-10-30T18:22:22.228Z [INFO] server-5513ec05-df46-e160-a38f-f8b0bc7631c8: aborting pipeline replication: peer="{Voter server-b659f1bf-72c4-0d06-bd80-f1acffd977e3 b659f1bf-72c4-0d06-bd80-f1acffd977e3}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false -2025-10-30T18:18:31.983Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6 Address:c2e2d911-c06f-431e-2fcc-81fd79c71ba6} {Suffrage:Voter ID:server-1b4ab6b8-de7e-133a-41a5-12fb93926aad Address:1b4ab6b8-de7e-133a-41a5-12fb93926aad} {Suffrage:Voter ID:server-f60ce911-6562-57aa-4bdc-18d4ec329d48 Address:f60ce911-6562-57aa-4bdc-18d4ec329d48}]" -2025-10-30T18:18:31.983Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: entering follower state: follower="Node at c2e2d911-c06f-431e-2fcc-81fd79c71ba6 [Follower]" leader-address= leader-id= -2025-10-30T18:18:31.983Z [INFO] server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6 Address:c2e2d911-c06f-431e-2fcc-81fd79c71ba6} {Suffrage:Voter ID:server-1b4ab6b8-de7e-133a-41a5-12fb93926aad Address:1b4ab6b8-de7e-133a-41a5-12fb93926aad} {Suffrage:Voter ID:server-f60ce911-6562-57aa-4bdc-18d4ec329d48 Address:f60ce911-6562-57aa-4bdc-18d4ec329d48}]" -2025-10-30T18:18:31.983Z [INFO] server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: entering follower state: follower="Node at 1b4ab6b8-de7e-133a-41a5-12fb93926aad [Follower]" leader-address= leader-id= -2025-10-30T18:18:31.983Z [INFO] server-f60ce911-6562-57aa-4bdc-18d4ec329d48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6 Address:c2e2d911-c06f-431e-2fcc-81fd79c71ba6} {Suffrage:Voter ID:server-1b4ab6b8-de7e-133a-41a5-12fb93926aad Address:1b4ab6b8-de7e-133a-41a5-12fb93926aad} {Suffrage:Voter ID:server-f60ce911-6562-57aa-4bdc-18d4ec329d48 Address:f60ce911-6562-57aa-4bdc-18d4ec329d48}]" -2025-10-30T18:18:31.983Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:31.983Z [INFO] server-f60ce911-6562-57aa-4bdc-18d4ec329d48: entering follower state: follower="Node at f60ce911-6562-57aa-4bdc-18d4ec329d48 [Follower]" leader-address= leader-id= -2025-10-30T18:18:32.600Z [WARN] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:32.600Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: entering candidate state: node="Node at c2e2d911-c06f-431e-2fcc-81fd79c71ba6 [Candidate]" term=2 -2025-10-30T18:18:32.600Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: election won: term=2 tally=2 -2025-10-30T18:18:32.600Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: entering leader state: leader="Node at c2e2d911-c06f-431e-2fcc-81fd79c71ba6 [Leader]" -2025-10-30T18:18:32.600Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: added peer, starting replication: peer=server-1b4ab6b8-de7e-133a-41a5-12fb93926aad -2025-10-30T18:18:32.600Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: added peer, starting replication: peer=server-f60ce911-6562-57aa-4bdc-18d4ec329d48 -2025-10-30T18:18:32.600Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: pipelining replication: peer="{Voter server-f60ce911-6562-57aa-4bdc-18d4ec329d48 f60ce911-6562-57aa-4bdc-18d4ec329d48}" -2025-10-30T18:18:32.600Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: pipelining replication: peer="{Voter server-1b4ab6b8-de7e-133a-41a5-12fb93926aad 1b4ab6b8-de7e-133a-41a5-12fb93926aad}" -2025-10-30T18:18:33.605Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:32.600594344 +0000 UTC m=+58.879268244 (1 nodes), 616.700385ms from start of poll, 617.10079ms from cluster start. Timeout at 2025-10-30 18:18:33.605594704 +0000 UTC m=+59.884268601, 1.005000357s after stability -2025-10-30T18:18:33.606Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: starting snapshot up to: index=12 -2025-10-30T18:18:33.606Z [INFO] creating new snapshot: path=/tmp/raft3068251824/snapshots/2-12-1761848313606.tmp -2025-10-30T18:18:33.606Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: snapshot complete up to: index=12 -2025-10-30T18:18:33.607Z [INFO] creating new snapshot: path=/tmp/raft3068251824/snapshots/2-113-1761848313607.tmp -2025-10-30T18:18:33.607Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: copied to local snapshot: bytes=71 -2025-10-30T18:18:33.607Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: snapshot restore progress: id=2-113-1761848313607 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:33.607Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: removing all old logs from log store -2025-10-30T18:18:33.607Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: compacting logs: from=1 to=22 -2025-10-30T18:18:33.607Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: restored user snapshot: index=113 -2025-10-30T18:18:33.607Z [ERROR] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: failed to get log: index=22 error="log not found" -2025-10-30T18:18:33.607Z [ERROR] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: failed to get log: index=22 error="log not found" -2025-10-30T18:18:33.607Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: aborting pipeline replication: peer="{Voter server-f60ce911-6562-57aa-4bdc-18d4ec329d48 f60ce911-6562-57aa-4bdc-18d4ec329d48}" -2025-10-30T18:18:33.607Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: aborting pipeline replication: peer="{Voter server-1b4ab6b8-de7e-133a-41a5-12fb93926aad 1b4ab6b8-de7e-133a-41a5-12fb93926aad}" -2025-10-30T18:18:33.617Z [ERROR] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: failed to get log: index=22 error="log not found" -2025-10-30T18:18:33.617Z [INFO] creating new snapshot: path=/tmp/raft1078824647/snapshots/2-113-1761848313617.tmp -2025-10-30T18:18:33.617Z [ERROR] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: failed to get log: index=22 error="log not found" -2025-10-30T18:18:33.617Z [INFO] creating new snapshot: path=/tmp/raft2376243238/snapshots/2-113-1761848313617.tmp -2025-10-30T18:18:33.617Z [INFO] server-f60ce911-6562-57aa-4bdc-18d4ec329d48: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:33.617Z [INFO] server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:33.617Z [INFO] server-f60ce911-6562-57aa-4bdc-18d4ec329d48: copied to local snapshot: bytes=71 -2025-10-30T18:18:33.618Z [INFO] server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: copied to local snapshot: bytes=71 -2025-10-30T18:18:33.618Z [INFO] server-f60ce911-6562-57aa-4bdc-18d4ec329d48: snapshot restore progress: id=2-113-1761848313617 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:33.618Z [INFO] server-f60ce911-6562-57aa-4bdc-18d4ec329d48: removing all old logs from log store -2025-10-30T18:18:33.618Z [INFO] server-f60ce911-6562-57aa-4bdc-18d4ec329d48: compacting logs: from=1 to=22 -2025-10-30T18:18:33.618Z [INFO] server-f60ce911-6562-57aa-4bdc-18d4ec329d48: Installed remote snapshot -2025-10-30T18:18:33.618Z [WARN] server-f60ce911-6562-57aa-4bdc-18d4ec329d48: skipping application of old log: index=22 -2025-10-30T18:18:33.618Z [INFO] server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: snapshot restore progress: id=2-113-1761848313617 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:33.618Z [INFO] server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: removing all old logs from log store -2025-10-30T18:18:33.618Z [INFO] server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: compacting logs: from=1 to=22 -2025-10-30T18:18:33.618Z [INFO] server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: Installed remote snapshot -2025-10-30T18:18:33.618Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: pipelining replication: peer="{Voter server-f60ce911-6562-57aa-4bdc-18d4ec329d48 f60ce911-6562-57aa-4bdc-18d4ec329d48}" -2025-10-30T18:18:33.618Z [WARN] server-1b4ab6b8-de7e-133a-41a5-12fb93926aad: skipping application of old log: index=113 -2025-10-30T18:18:33.618Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:18:33.618Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: pipelining replication: peer="{Voter server-1b4ab6b8-de7e-133a-41a5-12fb93926aad 1b4ab6b8-de7e-133a-41a5-12fb93926aad}" -2025-10-30T18:18:34.623Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:18:33.618344875 +0000 UTC m=+59.897018780 (2 nodes), 760ns from start of poll, 1.634851326s from cluster start. Timeout at 2025-10-30 18:18:34.623354412 +0000 UTC m=+60.902028327, 1.005009547s after stability -2025-10-30T18:18:34.635Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: aborting pipeline replication: peer="{Voter server-1b4ab6b8-de7e-133a-41a5-12fb93926aad 1b4ab6b8-de7e-133a-41a5-12fb93926aad}" -2025-10-30T18:18:34.635Z [INFO] server-c2e2d911-c06f-431e-2fcc-81fd79c71ba6: aborting pipeline replication: peer="{Voter server-f60ce911-6562-57aa-4bdc-18d4ec329d48 f60ce911-6562-57aa-4bdc-18d4ec329d48}" +2025-10-30T18:22:22.230Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f Address:6f97d574-627c-b1b9-03d2-9e363d5cfb8f} {Suffrage:Voter ID:server-ab026865-e22d-adc2-964e-294da319f13d Address:ab026865-e22d-adc2-964e-294da319f13d} {Suffrage:Voter ID:server-caf07b44-de75-a11b-df20-c3e1d8674fd6 Address:caf07b44-de75-a11b-df20-c3e1d8674fd6}]" +2025-10-30T18:22:22.230Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: entering follower state: follower="Node at 6f97d574-627c-b1b9-03d2-9e363d5cfb8f [Follower]" leader-address= leader-id= +2025-10-30T18:22:22.230Z [INFO] server-ab026865-e22d-adc2-964e-294da319f13d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f Address:6f97d574-627c-b1b9-03d2-9e363d5cfb8f} {Suffrage:Voter ID:server-ab026865-e22d-adc2-964e-294da319f13d Address:ab026865-e22d-adc2-964e-294da319f13d} {Suffrage:Voter ID:server-caf07b44-de75-a11b-df20-c3e1d8674fd6 Address:caf07b44-de75-a11b-df20-c3e1d8674fd6}]" +2025-10-30T18:22:22.230Z [INFO] server-ab026865-e22d-adc2-964e-294da319f13d: entering follower state: follower="Node at ab026865-e22d-adc2-964e-294da319f13d [Follower]" leader-address= leader-id= +2025-10-30T18:22:22.230Z [INFO] server-caf07b44-de75-a11b-df20-c3e1d8674fd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f Address:6f97d574-627c-b1b9-03d2-9e363d5cfb8f} {Suffrage:Voter ID:server-ab026865-e22d-adc2-964e-294da319f13d Address:ab026865-e22d-adc2-964e-294da319f13d} {Suffrage:Voter ID:server-caf07b44-de75-a11b-df20-c3e1d8674fd6 Address:caf07b44-de75-a11b-df20-c3e1d8674fd6}]" +2025-10-30T18:22:22.230Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:22.230Z [INFO] server-caf07b44-de75-a11b-df20-c3e1d8674fd6: entering follower state: follower="Node at caf07b44-de75-a11b-df20-c3e1d8674fd6 [Follower]" leader-address= leader-id= +2025-10-30T18:22:22.862Z [WARN] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:22.862Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: entering candidate state: node="Node at 6f97d574-627c-b1b9-03d2-9e363d5cfb8f [Candidate]" term=2 +2025-10-30T18:22:22.862Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: election won: term=2 tally=2 +2025-10-30T18:22:22.862Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: entering leader state: leader="Node at 6f97d574-627c-b1b9-03d2-9e363d5cfb8f [Leader]" +2025-10-30T18:22:22.862Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: added peer, starting replication: peer=server-ab026865-e22d-adc2-964e-294da319f13d +2025-10-30T18:22:22.862Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: added peer, starting replication: peer=server-caf07b44-de75-a11b-df20-c3e1d8674fd6 +2025-10-30T18:22:22.862Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: pipelining replication: peer="{Voter server-caf07b44-de75-a11b-df20-c3e1d8674fd6 caf07b44-de75-a11b-df20-c3e1d8674fd6}" +2025-10-30T18:22:22.862Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: pipelining replication: peer="{Voter server-ab026865-e22d-adc2-964e-294da319f13d ab026865-e22d-adc2-964e-294da319f13d}" +2025-10-30T18:22:23.867Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:22.862384097 +0000 UTC m=+65.935913391 (1 nodes), 631.784979ms from start of poll, 632.109521ms from cluster start. Timeout at 2025-10-30 18:22:23.867386692 +0000 UTC m=+66.940915985, 1.005002594s after stability +2025-10-30T18:22:23.868Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: starting snapshot up to: index=12 +2025-10-30T18:22:23.868Z [INFO] creating new snapshot: path=/tmp/raft3766471411/snapshots/2-12-1761848543868.tmp +2025-10-30T18:22:23.943Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: snapshot complete up to: index=12 +2025-10-30T18:22:23.944Z [INFO] creating new snapshot: path=/tmp/raft3766471411/snapshots/2-113-1761848543944.tmp +2025-10-30T18:22:23.944Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: copied to local snapshot: bytes=71 +2025-10-30T18:22:23.944Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: snapshot restore progress: id=2-113-1761848543944 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:23.944Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: removing all old logs from log store +2025-10-30T18:22:23.944Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: compacting logs: from=1 to=22 +2025-10-30T18:22:23.944Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: restored user snapshot: index=113 +2025-10-30T18:22:23.944Z [ERROR] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: failed to get log: index=22 error="log not found" +2025-10-30T18:22:23.945Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: aborting pipeline replication: peer="{Voter server-caf07b44-de75-a11b-df20-c3e1d8674fd6 caf07b44-de75-a11b-df20-c3e1d8674fd6}" +2025-10-30T18:22:23.945Z [ERROR] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: failed to get log: index=22 error="log not found" +2025-10-30T18:22:23.945Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: aborting pipeline replication: peer="{Voter server-ab026865-e22d-adc2-964e-294da319f13d ab026865-e22d-adc2-964e-294da319f13d}" +2025-10-30T18:22:23.952Z [ERROR] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: failed to get log: index=22 error="log not found" +2025-10-30T18:22:23.952Z [INFO] creating new snapshot: path=/tmp/raft3156699117/snapshots/2-113-1761848543952.tmp +2025-10-30T18:22:23.953Z [INFO] server-caf07b44-de75-a11b-df20-c3e1d8674fd6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:23.954Z [ERROR] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: failed to get log: index=22 error="log not found" +2025-10-30T18:22:23.955Z [INFO] creating new snapshot: path=/tmp/raft3103201320/snapshots/2-113-1761848543955.tmp +2025-10-30T18:22:23.955Z [INFO] server-ab026865-e22d-adc2-964e-294da319f13d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:24.015Z [INFO] server-caf07b44-de75-a11b-df20-c3e1d8674fd6: copied to local snapshot: bytes=71 +2025-10-30T18:22:24.015Z [INFO] server-ab026865-e22d-adc2-964e-294da319f13d: copied to local snapshot: bytes=71 +2025-10-30T18:22:24.016Z [INFO] server-caf07b44-de75-a11b-df20-c3e1d8674fd6: snapshot restore progress: id=2-113-1761848543952 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:24.016Z [INFO] server-caf07b44-de75-a11b-df20-c3e1d8674fd6: removing all old logs from log store +2025-10-30T18:22:24.016Z [INFO] server-caf07b44-de75-a11b-df20-c3e1d8674fd6: compacting logs: from=1 to=22 +2025-10-30T18:22:24.016Z [INFO] server-caf07b44-de75-a11b-df20-c3e1d8674fd6: Installed remote snapshot +2025-10-30T18:22:24.016Z [WARN] server-caf07b44-de75-a11b-df20-c3e1d8674fd6: skipping application of old log: index=22 +2025-10-30T18:22:24.016Z [INFO] server-ab026865-e22d-adc2-964e-294da319f13d: snapshot restore progress: id=2-113-1761848543955 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:24.016Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: pipelining replication: peer="{Voter server-caf07b44-de75-a11b-df20-c3e1d8674fd6 caf07b44-de75-a11b-df20-c3e1d8674fd6}" +2025-10-30T18:22:24.016Z [INFO] server-ab026865-e22d-adc2-964e-294da319f13d: removing all old logs from log store +2025-10-30T18:22:24.016Z [INFO] server-ab026865-e22d-adc2-964e-294da319f13d: compacting logs: from=1 to=22 +2025-10-30T18:22:24.016Z [INFO] server-ab026865-e22d-adc2-964e-294da319f13d: Installed remote snapshot +2025-10-30T18:22:24.016Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:22:24.016Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: pipelining replication: peer="{Voter server-ab026865-e22d-adc2-964e-294da319f13d ab026865-e22d-adc2-964e-294da319f13d}" +2025-10-30T18:22:25.022Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:22:24.016399489 +0000 UTC m=+67.089929010 (2 nodes), 1.157µs from start of poll, 1.78612514s from cluster start. Timeout at 2025-10-30 18:22:25.021412062 +0000 UTC m=+68.094941377, 1.005012367s after stability +2025-10-30T18:22:25.034Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: aborting pipeline replication: peer="{Voter server-caf07b44-de75-a11b-df20-c3e1d8674fd6 caf07b44-de75-a11b-df20-c3e1d8674fd6}" +2025-10-30T18:22:25.034Z [INFO] server-6f97d574-627c-b1b9-03d2-9e363d5cfb8f: aborting pipeline replication: peer="{Voter server-ab026865-e22d-adc2-964e-294da319f13d ab026865-e22d-adc2-964e-294da319f13d}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true -2025-10-30T18:18:34.636Z [INFO] server-1ca17eb3-d82f-eacc-3286-52c6424582e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ca17eb3-d82f-eacc-3286-52c6424582e9 Address:1ca17eb3-d82f-eacc-3286-52c6424582e9} {Suffrage:Voter ID:server-826570be-a109-033c-b9c2-b73e0c64edcf Address:826570be-a109-033c-b9c2-b73e0c64edcf} {Suffrage:Voter ID:server-30cc0303-00d8-773c-a2e0-2b97575db913 Address:30cc0303-00d8-773c-a2e0-2b97575db913}]" -2025-10-30T18:18:34.636Z [INFO] server-1ca17eb3-d82f-eacc-3286-52c6424582e9: entering follower state: follower="Node at 1ca17eb3-d82f-eacc-3286-52c6424582e9 [Follower]" leader-address= leader-id= -2025-10-30T18:18:34.636Z [INFO] server-826570be-a109-033c-b9c2-b73e0c64edcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ca17eb3-d82f-eacc-3286-52c6424582e9 Address:1ca17eb3-d82f-eacc-3286-52c6424582e9} {Suffrage:Voter ID:server-826570be-a109-033c-b9c2-b73e0c64edcf Address:826570be-a109-033c-b9c2-b73e0c64edcf} {Suffrage:Voter ID:server-30cc0303-00d8-773c-a2e0-2b97575db913 Address:30cc0303-00d8-773c-a2e0-2b97575db913}]" -2025-10-30T18:18:34.636Z [INFO] server-826570be-a109-033c-b9c2-b73e0c64edcf: entering follower state: follower="Node at 826570be-a109-033c-b9c2-b73e0c64edcf [Follower]" leader-address= leader-id= -2025-10-30T18:18:34.636Z [INFO] server-30cc0303-00d8-773c-a2e0-2b97575db913: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ca17eb3-d82f-eacc-3286-52c6424582e9 Address:1ca17eb3-d82f-eacc-3286-52c6424582e9} {Suffrage:Voter ID:server-826570be-a109-033c-b9c2-b73e0c64edcf Address:826570be-a109-033c-b9c2-b73e0c64edcf} {Suffrage:Voter ID:server-30cc0303-00d8-773c-a2e0-2b97575db913 Address:30cc0303-00d8-773c-a2e0-2b97575db913}]" -2025-10-30T18:18:34.636Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:34.636Z [INFO] server-30cc0303-00d8-773c-a2e0-2b97575db913: entering follower state: follower="Node at 30cc0303-00d8-773c-a2e0-2b97575db913 [Follower]" leader-address= leader-id= -2025-10-30T18:18:35.567Z [WARN] server-1ca17eb3-d82f-eacc-3286-52c6424582e9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:35.567Z [INFO] server-1ca17eb3-d82f-eacc-3286-52c6424582e9: entering candidate state: node="Node at 1ca17eb3-d82f-eacc-3286-52c6424582e9 [Candidate]" term=2 -2025-10-30T18:18:35.567Z [INFO] server-1ca17eb3-d82f-eacc-3286-52c6424582e9: election won: term=2 tally=2 -2025-10-30T18:18:35.567Z [INFO] server-1ca17eb3-d82f-eacc-3286-52c6424582e9: entering leader state: leader="Node at 1ca17eb3-d82f-eacc-3286-52c6424582e9 [Leader]" -2025-10-30T18:18:35.567Z [INFO] server-1ca17eb3-d82f-eacc-3286-52c6424582e9: added peer, starting replication: peer=server-826570be-a109-033c-b9c2-b73e0c64edcf -2025-10-30T18:18:35.567Z [INFO] server-1ca17eb3-d82f-eacc-3286-52c6424582e9: added peer, starting replication: peer=server-30cc0303-00d8-773c-a2e0-2b97575db913 -2025-10-30T18:18:35.567Z [INFO] server-1ca17eb3-d82f-eacc-3286-52c6424582e9: pipelining replication: peer="{Voter server-826570be-a109-033c-b9c2-b73e0c64edcf 826570be-a109-033c-b9c2-b73e0c64edcf}" -2025-10-30T18:18:35.567Z [INFO] server-1ca17eb3-d82f-eacc-3286-52c6424582e9: pipelining replication: peer="{Voter server-30cc0303-00d8-773c-a2e0-2b97575db913 30cc0303-00d8-773c-a2e0-2b97575db913}" -2025-10-30T18:18:36.572Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:35.567249867 +0000 UTC m=+61.845923767 (1 nodes), 930.732881ms from start of poll, 931.057274ms from cluster start. Timeout at 2025-10-30 18:18:36.572250355 +0000 UTC m=+62.850924253, 1.005000486s after stability -2025-10-30T18:18:36.574Z [INFO] server-1ca17eb3-d82f-eacc-3286-52c6424582e9: starting snapshot up to: index=12 -2025-10-30T18:18:36.574Z [INFO] creating new snapshot: path=/tmp/raft2548335724/snapshots/2-12-1761848316574.tmp -2025-10-30T18:18:36.574Z [INFO] server-1ca17eb3-d82f-eacc-3286-52c6424582e9: snapshot complete up to: index=12 -2025-10-30T18:18:36.575Z [INFO] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b4c036b-584c-1601-4c5e-b08a2c94effd Address:8b4c036b-584c-1601-4c5e-b08a2c94effd} {Suffrage:Voter ID:server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0 Address:bd6b162b-0bd1-3481-49ca-d3e438a8cab0} {Suffrage:Voter ID:server-1ac23055-c326-0117-dc20-6cdfcbc0804a Address:1ac23055-c326-0117-dc20-6cdfcbc0804a}]" -2025-10-30T18:18:36.575Z [INFO] server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b4c036b-584c-1601-4c5e-b08a2c94effd Address:8b4c036b-584c-1601-4c5e-b08a2c94effd} {Suffrage:Voter ID:server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0 Address:bd6b162b-0bd1-3481-49ca-d3e438a8cab0} {Suffrage:Voter ID:server-1ac23055-c326-0117-dc20-6cdfcbc0804a Address:1ac23055-c326-0117-dc20-6cdfcbc0804a}]" -2025-10-30T18:18:36.575Z [INFO] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: entering follower state: follower="Node at 8b4c036b-584c-1601-4c5e-b08a2c94effd [Follower]" leader-address= leader-id= -2025-10-30T18:18:36.575Z [INFO] server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0: entering follower state: follower="Node at bd6b162b-0bd1-3481-49ca-d3e438a8cab0 [Follower]" leader-address= leader-id= -2025-10-30T18:18:36.575Z [INFO] server-1ac23055-c326-0117-dc20-6cdfcbc0804a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b4c036b-584c-1601-4c5e-b08a2c94effd Address:8b4c036b-584c-1601-4c5e-b08a2c94effd} {Suffrage:Voter ID:server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0 Address:bd6b162b-0bd1-3481-49ca-d3e438a8cab0} {Suffrage:Voter ID:server-1ac23055-c326-0117-dc20-6cdfcbc0804a Address:1ac23055-c326-0117-dc20-6cdfcbc0804a}]" -2025-10-30T18:18:36.575Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:36.575Z [INFO] server-1ac23055-c326-0117-dc20-6cdfcbc0804a: entering follower state: follower="Node at 1ac23055-c326-0117-dc20-6cdfcbc0804a [Follower]" leader-address= leader-id= -2025-10-30T18:18:37.192Z [WARN] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:37.192Z [INFO] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: entering candidate state: node="Node at 8b4c036b-584c-1601-4c5e-b08a2c94effd [Candidate]" term=2 -2025-10-30T18:18:37.192Z [INFO] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: election won: term=2 tally=2 -2025-10-30T18:18:37.192Z [INFO] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: entering leader state: leader="Node at 8b4c036b-584c-1601-4c5e-b08a2c94effd [Leader]" -2025-10-30T18:18:37.192Z [INFO] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: added peer, starting replication: peer=server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0 -2025-10-30T18:18:37.192Z [INFO] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: added peer, starting replication: peer=server-1ac23055-c326-0117-dc20-6cdfcbc0804a -2025-10-30T18:18:37.192Z [INFO] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: pipelining replication: peer="{Voter server-1ac23055-c326-0117-dc20-6cdfcbc0804a 1ac23055-c326-0117-dc20-6cdfcbc0804a}" -2025-10-30T18:18:37.192Z [INFO] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: pipelining replication: peer="{Voter server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0 bd6b162b-0bd1-3481-49ca-d3e438a8cab0}" -2025-10-30T18:18:38.198Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:37.192355474 +0000 UTC m=+63.471029372 (1 nodes), 616.627045ms from start of poll, 616.965171ms from cluster start. Timeout at 2025-10-30 18:18:38.197359416 +0000 UTC m=+64.476033316, 1.005003944s after stability -2025-10-30T18:18:38.198Z [INFO] creating new snapshot: path=/tmp/raft3474413342/snapshots/2-113-1761848318198.tmp -2025-10-30T18:18:38.303Z [INFO] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: copied to local snapshot: bytes=71 -2025-10-30T18:18:38.303Z [INFO] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: snapshot restore progress: id=2-113-1761848318198 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:38.303Z [INFO] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: restored user snapshot: index=113 -2025-10-30T18:18:38.303Z [ERROR] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: failed to get log: index=3 error="log not found" -2025-10-30T18:18:38.303Z [INFO] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: aborting pipeline replication: peer="{Voter server-1ac23055-c326-0117-dc20-6cdfcbc0804a 1ac23055-c326-0117-dc20-6cdfcbc0804a}" -2025-10-30T18:18:38.303Z [ERROR] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: failed to get log: index=3 error="log not found" -2025-10-30T18:18:38.303Z [INFO] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: aborting pipeline replication: peer="{Voter server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0 bd6b162b-0bd1-3481-49ca-d3e438a8cab0}" -2025-10-30T18:18:38.310Z [ERROR] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: failed to get log: index=3 error="log not found" -2025-10-30T18:18:38.310Z [INFO] creating new snapshot: path=/tmp/raft729453438/snapshots/2-113-1761848318310.tmp -2025-10-30T18:18:38.310Z [INFO] server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:38.311Z [INFO] server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0: copied to local snapshot: bytes=71 -2025-10-30T18:18:38.311Z [INFO] server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0: snapshot restore progress: id=2-113-1761848318310 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:38.311Z [ERROR] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: failed to get log: index=3 error="log not found" -2025-10-30T18:18:38.311Z [INFO] server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0: Installed remote snapshot -2025-10-30T18:18:38.311Z [INFO] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: pipelining replication: peer="{Voter server-bd6b162b-0bd1-3481-49ca-d3e438a8cab0 bd6b162b-0bd1-3481-49ca-d3e438a8cab0}" -2025-10-30T18:18:38.311Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:18:38.311Z [INFO] creating new snapshot: path=/tmp/raft2921723766/snapshots/2-113-1761848318311.tmp -2025-10-30T18:18:38.311Z [INFO] server-1ac23055-c326-0117-dc20-6cdfcbc0804a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:38.311Z [INFO] server-1ac23055-c326-0117-dc20-6cdfcbc0804a: copied to local snapshot: bytes=71 -2025-10-30T18:18:38.312Z [INFO] server-1ac23055-c326-0117-dc20-6cdfcbc0804a: snapshot restore progress: id=2-113-1761848318311 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:38.312Z [INFO] server-1ac23055-c326-0117-dc20-6cdfcbc0804a: Installed remote snapshot -2025-10-30T18:18:38.312Z [INFO] server-8b4c036b-584c-1601-4c5e-b08a2c94effd: pipelining replication: peer="{Voter server-1ac23055-c326-0117-dc20-6cdfcbc0804a 1ac23055-c326-0117-dc20-6cdfcbc0804a}" -2025-10-30T18:18:39.316Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:18:38.311315189 +0000 UTC m=+64.589989087 (2 nodes), 708ns from start of poll, 1.735924886s from cluster start. Timeout at 2025-10-30 18:18:39.316332764 +0000 UTC m=+65.595006692, 1.005017605s after stability -2025-10-30T18:18:39.327Z [INFO] server-1ca17eb3-d82f-eacc-3286-52c6424582e9: aborting pipeline replication: peer="{Voter server-826570be-a109-033c-b9c2-b73e0c64edcf 826570be-a109-033c-b9c2-b73e0c64edcf}" -2025-10-30T18:18:39.327Z [INFO] server-1ca17eb3-d82f-eacc-3286-52c6424582e9: aborting pipeline replication: peer="{Voter server-30cc0303-00d8-773c-a2e0-2b97575db913 30cc0303-00d8-773c-a2e0-2b97575db913}" +2025-10-30T18:22:25.065Z [INFO] server-668ffbf3-13bb-fb42-fda4-fe846951e4b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-668ffbf3-13bb-fb42-fda4-fe846951e4b0 Address:668ffbf3-13bb-fb42-fda4-fe846951e4b0} {Suffrage:Voter ID:server-4f895aa2-987d-9cf0-485b-5435b333995a Address:4f895aa2-987d-9cf0-485b-5435b333995a} {Suffrage:Voter ID:server-32c89fc0-a485-db55-7626-845b2086d45a Address:32c89fc0-a485-db55-7626-845b2086d45a}]" +2025-10-30T18:22:25.065Z [INFO] server-668ffbf3-13bb-fb42-fda4-fe846951e4b0: entering follower state: follower="Node at 668ffbf3-13bb-fb42-fda4-fe846951e4b0 [Follower]" leader-address= leader-id= +2025-10-30T18:22:25.065Z [INFO] server-4f895aa2-987d-9cf0-485b-5435b333995a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-668ffbf3-13bb-fb42-fda4-fe846951e4b0 Address:668ffbf3-13bb-fb42-fda4-fe846951e4b0} {Suffrage:Voter ID:server-4f895aa2-987d-9cf0-485b-5435b333995a Address:4f895aa2-987d-9cf0-485b-5435b333995a} {Suffrage:Voter ID:server-32c89fc0-a485-db55-7626-845b2086d45a Address:32c89fc0-a485-db55-7626-845b2086d45a}]" +2025-10-30T18:22:25.065Z [INFO] server-32c89fc0-a485-db55-7626-845b2086d45a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-668ffbf3-13bb-fb42-fda4-fe846951e4b0 Address:668ffbf3-13bb-fb42-fda4-fe846951e4b0} {Suffrage:Voter ID:server-4f895aa2-987d-9cf0-485b-5435b333995a Address:4f895aa2-987d-9cf0-485b-5435b333995a} {Suffrage:Voter ID:server-32c89fc0-a485-db55-7626-845b2086d45a Address:32c89fc0-a485-db55-7626-845b2086d45a}]" +2025-10-30T18:22:25.065Z [INFO] server-4f895aa2-987d-9cf0-485b-5435b333995a: entering follower state: follower="Node at 4f895aa2-987d-9cf0-485b-5435b333995a [Follower]" leader-address= leader-id= +2025-10-30T18:22:25.065Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:25.065Z [INFO] server-32c89fc0-a485-db55-7626-845b2086d45a: entering follower state: follower="Node at 32c89fc0-a485-db55-7626-845b2086d45a [Follower]" leader-address= leader-id= +2025-10-30T18:22:25.732Z [WARN] server-4f895aa2-987d-9cf0-485b-5435b333995a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:25.732Z [INFO] server-4f895aa2-987d-9cf0-485b-5435b333995a: entering candidate state: node="Node at 4f895aa2-987d-9cf0-485b-5435b333995a [Candidate]" term=2 +2025-10-30T18:22:25.732Z [INFO] server-4f895aa2-987d-9cf0-485b-5435b333995a: election won: term=2 tally=2 +2025-10-30T18:22:25.732Z [INFO] server-4f895aa2-987d-9cf0-485b-5435b333995a: entering leader state: leader="Node at 4f895aa2-987d-9cf0-485b-5435b333995a [Leader]" +2025-10-30T18:22:25.732Z [INFO] server-4f895aa2-987d-9cf0-485b-5435b333995a: added peer, starting replication: peer=server-668ffbf3-13bb-fb42-fda4-fe846951e4b0 +2025-10-30T18:22:25.732Z [INFO] server-4f895aa2-987d-9cf0-485b-5435b333995a: added peer, starting replication: peer=server-32c89fc0-a485-db55-7626-845b2086d45a +2025-10-30T18:22:25.732Z [INFO] server-4f895aa2-987d-9cf0-485b-5435b333995a: pipelining replication: peer="{Voter server-32c89fc0-a485-db55-7626-845b2086d45a 32c89fc0-a485-db55-7626-845b2086d45a}" +2025-10-30T18:22:25.732Z [INFO] server-4f895aa2-987d-9cf0-485b-5435b333995a: pipelining replication: peer="{Voter server-668ffbf3-13bb-fb42-fda4-fe846951e4b0 668ffbf3-13bb-fb42-fda4-fe846951e4b0}" +2025-10-30T18:22:26.737Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:25.732156449 +0000 UTC m=+68.805685745 (1 nodes), 666.284968ms from start of poll, 666.58433ms from cluster start. Timeout at 2025-10-30 18:22:26.737163348 +0000 UTC m=+69.810692640, 1.005006895s after stability +2025-10-30T18:22:26.738Z [INFO] server-4f895aa2-987d-9cf0-485b-5435b333995a: starting snapshot up to: index=12 +2025-10-30T18:22:26.738Z [INFO] creating new snapshot: path=/tmp/raft1291125899/snapshots/2-12-1761848546738.tmp +2025-10-30T18:22:26.816Z [INFO] server-4f895aa2-987d-9cf0-485b-5435b333995a: snapshot complete up to: index=12 +2025-10-30T18:22:26.817Z [INFO] server-a26ff83d-e22a-136c-9b62-5bb846157d8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a26ff83d-e22a-136c-9b62-5bb846157d8e Address:a26ff83d-e22a-136c-9b62-5bb846157d8e} {Suffrage:Voter ID:server-21909053-afae-878e-2be8-0085c1c63f8a Address:21909053-afae-878e-2be8-0085c1c63f8a} {Suffrage:Voter ID:server-4f165838-c7a2-0648-7438-0652345f61e1 Address:4f165838-c7a2-0648-7438-0652345f61e1}]" +2025-10-30T18:22:26.817Z [INFO] server-a26ff83d-e22a-136c-9b62-5bb846157d8e: entering follower state: follower="Node at a26ff83d-e22a-136c-9b62-5bb846157d8e [Follower]" leader-address= leader-id= +2025-10-30T18:22:26.817Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a26ff83d-e22a-136c-9b62-5bb846157d8e Address:a26ff83d-e22a-136c-9b62-5bb846157d8e} {Suffrage:Voter ID:server-21909053-afae-878e-2be8-0085c1c63f8a Address:21909053-afae-878e-2be8-0085c1c63f8a} {Suffrage:Voter ID:server-4f165838-c7a2-0648-7438-0652345f61e1 Address:4f165838-c7a2-0648-7438-0652345f61e1}]" +2025-10-30T18:22:26.817Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: entering follower state: follower="Node at 21909053-afae-878e-2be8-0085c1c63f8a [Follower]" leader-address= leader-id= +2025-10-30T18:22:26.817Z [INFO] server-4f165838-c7a2-0648-7438-0652345f61e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a26ff83d-e22a-136c-9b62-5bb846157d8e Address:a26ff83d-e22a-136c-9b62-5bb846157d8e} {Suffrage:Voter ID:server-21909053-afae-878e-2be8-0085c1c63f8a Address:21909053-afae-878e-2be8-0085c1c63f8a} {Suffrage:Voter ID:server-4f165838-c7a2-0648-7438-0652345f61e1 Address:4f165838-c7a2-0648-7438-0652345f61e1}]" +2025-10-30T18:22:26.817Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:26.817Z [INFO] server-4f165838-c7a2-0648-7438-0652345f61e1: entering follower state: follower="Node at 4f165838-c7a2-0648-7438-0652345f61e1 [Follower]" leader-address= leader-id= +2025-10-30T18:22:27.499Z [WARN] server-21909053-afae-878e-2be8-0085c1c63f8a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:27.499Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: entering candidate state: node="Node at 21909053-afae-878e-2be8-0085c1c63f8a [Candidate]" term=2 +2025-10-30T18:22:27.499Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: election won: term=2 tally=2 +2025-10-30T18:22:27.499Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: entering leader state: leader="Node at 21909053-afae-878e-2be8-0085c1c63f8a [Leader]" +2025-10-30T18:22:27.500Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: added peer, starting replication: peer=server-a26ff83d-e22a-136c-9b62-5bb846157d8e +2025-10-30T18:22:27.500Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: added peer, starting replication: peer=server-4f165838-c7a2-0648-7438-0652345f61e1 +2025-10-30T18:22:27.500Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: pipelining replication: peer="{Voter server-a26ff83d-e22a-136c-9b62-5bb846157d8e a26ff83d-e22a-136c-9b62-5bb846157d8e}" +2025-10-30T18:22:27.500Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: pipelining replication: peer="{Voter server-4f165838-c7a2-0648-7438-0652345f61e1 4f165838-c7a2-0648-7438-0652345f61e1}" +2025-10-30T18:22:28.505Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:27.500022186 +0000 UTC m=+70.573551479 (1 nodes), 682.087385ms from start of poll, 682.435728ms from cluster start. Timeout at 2025-10-30 18:22:28.505027108 +0000 UTC m=+71.578556397, 1.005004918s after stability +2025-10-30T18:22:28.505Z [INFO] creating new snapshot: path=/tmp/raft2252673934/snapshots/2-113-1761848548505.tmp +2025-10-30T18:22:28.508Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: copied to local snapshot: bytes=71 +2025-10-30T18:22:28.508Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: snapshot restore progress: id=2-113-1761848548505 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:28.508Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: restored user snapshot: index=113 +2025-10-30T18:22:28.509Z [ERROR] server-21909053-afae-878e-2be8-0085c1c63f8a: failed to get log: index=3 error="log not found" +2025-10-30T18:22:28.509Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: aborting pipeline replication: peer="{Voter server-a26ff83d-e22a-136c-9b62-5bb846157d8e a26ff83d-e22a-136c-9b62-5bb846157d8e}" +2025-10-30T18:22:28.509Z [ERROR] server-21909053-afae-878e-2be8-0085c1c63f8a: failed to get log: index=3 error="log not found" +2025-10-30T18:22:28.509Z [ERROR] server-21909053-afae-878e-2be8-0085c1c63f8a: failed to get log: index=3 error="log not found" +2025-10-30T18:22:28.509Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: aborting pipeline replication: peer="{Voter server-4f165838-c7a2-0648-7438-0652345f61e1 4f165838-c7a2-0648-7438-0652345f61e1}" +2025-10-30T18:22:28.509Z [INFO] creating new snapshot: path=/tmp/raft4187921486/snapshots/2-113-1761848548509.tmp +2025-10-30T18:22:28.509Z [INFO] server-a26ff83d-e22a-136c-9b62-5bb846157d8e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:28.509Z [INFO] server-a26ff83d-e22a-136c-9b62-5bb846157d8e: copied to local snapshot: bytes=71 +2025-10-30T18:22:28.510Z [INFO] server-a26ff83d-e22a-136c-9b62-5bb846157d8e: snapshot restore progress: id=2-113-1761848548509 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:28.510Z [INFO] server-a26ff83d-e22a-136c-9b62-5bb846157d8e: Installed remote snapshot +2025-10-30T18:22:28.510Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: pipelining replication: peer="{Voter server-a26ff83d-e22a-136c-9b62-5bb846157d8e a26ff83d-e22a-136c-9b62-5bb846157d8e}" +2025-10-30T18:22:28.510Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:22:28.517Z [ERROR] server-21909053-afae-878e-2be8-0085c1c63f8a: failed to get log: index=3 error="log not found" +2025-10-30T18:22:28.636Z [INFO] creating new snapshot: path=/tmp/raft94077827/snapshots/2-113-1761848548636.tmp +2025-10-30T18:22:28.636Z [INFO] server-4f165838-c7a2-0648-7438-0652345f61e1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:28.744Z [INFO] server-4f165838-c7a2-0648-7438-0652345f61e1: copied to local snapshot: bytes=71 +2025-10-30T18:22:28.745Z [INFO] server-4f165838-c7a2-0648-7438-0652345f61e1: snapshot restore progress: id=2-113-1761848548636 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:28.745Z [INFO] server-4f165838-c7a2-0648-7438-0652345f61e1: Installed remote snapshot +2025-10-30T18:22:28.745Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: pipelining replication: peer="{Voter server-4f165838-c7a2-0648-7438-0652345f61e1 4f165838-c7a2-0648-7438-0652345f61e1}" +2025-10-30T18:22:29.515Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:22:28.510211135 +0000 UTC m=+71.583740427 (2 nodes), 580ns from start of poll, 1.692624676s from cluster start. Timeout at 2025-10-30 18:22:29.515223375 +0000 UTC m=+72.588752667, 1.00501224s after stability +2025-10-30T18:22:29.527Z [INFO] server-4f895aa2-987d-9cf0-485b-5435b333995a: aborting pipeline replication: peer="{Voter server-668ffbf3-13bb-fb42-fda4-fe846951e4b0 668ffbf3-13bb-fb42-fda4-fe846951e4b0}" +2025-10-30T18:22:29.527Z [INFO] server-4f895aa2-987d-9cf0-485b-5435b333995a: aborting pipeline replication: peer="{Voter server-32c89fc0-a485-db55-7626-845b2086d45a 32c89fc0-a485-db55-7626-845b2086d45a}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true -2025-10-30T18:18:39.330Z [INFO] server-748ca023-7768-a08f-8cd7-cf6701cd2f1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-748ca023-7768-a08f-8cd7-cf6701cd2f1a Address:748ca023-7768-a08f-8cd7-cf6701cd2f1a} {Suffrage:Voter ID:server-1818bd9b-16af-9652-9038-d6af42c2c520 Address:1818bd9b-16af-9652-9038-d6af42c2c520} {Suffrage:Voter ID:server-76a58140-ece6-e957-356d-750a6badabda Address:76a58140-ece6-e957-356d-750a6badabda}]" -2025-10-30T18:18:39.330Z [INFO] server-748ca023-7768-a08f-8cd7-cf6701cd2f1a: entering follower state: follower="Node at 748ca023-7768-a08f-8cd7-cf6701cd2f1a [Follower]" leader-address= leader-id= -2025-10-30T18:18:39.330Z [INFO] server-1818bd9b-16af-9652-9038-d6af42c2c520: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-748ca023-7768-a08f-8cd7-cf6701cd2f1a Address:748ca023-7768-a08f-8cd7-cf6701cd2f1a} {Suffrage:Voter ID:server-1818bd9b-16af-9652-9038-d6af42c2c520 Address:1818bd9b-16af-9652-9038-d6af42c2c520} {Suffrage:Voter ID:server-76a58140-ece6-e957-356d-750a6badabda Address:76a58140-ece6-e957-356d-750a6badabda}]" -2025-10-30T18:18:39.330Z [INFO] server-1818bd9b-16af-9652-9038-d6af42c2c520: entering follower state: follower="Node at 1818bd9b-16af-9652-9038-d6af42c2c520 [Follower]" leader-address= leader-id= -2025-10-30T18:18:39.330Z [INFO] server-76a58140-ece6-e957-356d-750a6badabda: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-748ca023-7768-a08f-8cd7-cf6701cd2f1a Address:748ca023-7768-a08f-8cd7-cf6701cd2f1a} {Suffrage:Voter ID:server-1818bd9b-16af-9652-9038-d6af42c2c520 Address:1818bd9b-16af-9652-9038-d6af42c2c520} {Suffrage:Voter ID:server-76a58140-ece6-e957-356d-750a6badabda Address:76a58140-ece6-e957-356d-750a6badabda}]" -2025-10-30T18:18:39.330Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:39.330Z [INFO] server-76a58140-ece6-e957-356d-750a6badabda: entering follower state: follower="Node at 76a58140-ece6-e957-356d-750a6badabda [Follower]" leader-address= leader-id= -2025-10-30T18:18:40.055Z [WARN] server-76a58140-ece6-e957-356d-750a6badabda: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:40.055Z [INFO] server-76a58140-ece6-e957-356d-750a6badabda: entering candidate state: node="Node at 76a58140-ece6-e957-356d-750a6badabda [Candidate]" term=2 -2025-10-30T18:18:40.055Z [INFO] server-76a58140-ece6-e957-356d-750a6badabda: election won: term=2 tally=2 -2025-10-30T18:18:40.055Z [INFO] server-76a58140-ece6-e957-356d-750a6badabda: entering leader state: leader="Node at 76a58140-ece6-e957-356d-750a6badabda [Leader]" -2025-10-30T18:18:40.055Z [INFO] server-76a58140-ece6-e957-356d-750a6badabda: added peer, starting replication: peer=server-748ca023-7768-a08f-8cd7-cf6701cd2f1a -2025-10-30T18:18:40.055Z [INFO] server-76a58140-ece6-e957-356d-750a6badabda: added peer, starting replication: peer=server-1818bd9b-16af-9652-9038-d6af42c2c520 -2025-10-30T18:18:40.055Z [INFO] server-76a58140-ece6-e957-356d-750a6badabda: pipelining replication: peer="{Voter server-1818bd9b-16af-9652-9038-d6af42c2c520 1818bd9b-16af-9652-9038-d6af42c2c520}" -2025-10-30T18:18:40.055Z [INFO] server-76a58140-ece6-e957-356d-750a6badabda: pipelining replication: peer="{Voter server-748ca023-7768-a08f-8cd7-cf6701cd2f1a 748ca023-7768-a08f-8cd7-cf6701cd2f1a}" -2025-10-30T18:18:41.060Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:40.055461955 +0000 UTC m=+66.334135868 (1 nodes), 725.164764ms from start of poll, 725.51142ms from cluster start. Timeout at 2025-10-30 18:18:41.060467108 +0000 UTC m=+67.339141012, 1.005005144s after stability -2025-10-30T18:18:41.061Z [INFO] server-76a58140-ece6-e957-356d-750a6badabda: starting snapshot up to: index=12 -2025-10-30T18:18:41.061Z [INFO] creating new snapshot: path=/tmp/raft3208121412/snapshots/2-12-1761848321061.tmp -2025-10-30T18:18:41.061Z [INFO] server-76a58140-ece6-e957-356d-750a6badabda: snapshot complete up to: index=12 -2025-10-30T18:18:41.062Z [INFO] server-d2c34d70-1753-4cbe-33c3-edb13f282d02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2c34d70-1753-4cbe-33c3-edb13f282d02 Address:d2c34d70-1753-4cbe-33c3-edb13f282d02} {Suffrage:Voter ID:server-e54b2fd9-af1a-74be-3888-dc953cc87ebd Address:e54b2fd9-af1a-74be-3888-dc953cc87ebd} {Suffrage:Voter ID:server-59d300b7-30d6-0976-d683-9115ee0b89ea Address:59d300b7-30d6-0976-d683-9115ee0b89ea}]" -2025-10-30T18:18:41.062Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2c34d70-1753-4cbe-33c3-edb13f282d02 Address:d2c34d70-1753-4cbe-33c3-edb13f282d02} {Suffrage:Voter ID:server-e54b2fd9-af1a-74be-3888-dc953cc87ebd Address:e54b2fd9-af1a-74be-3888-dc953cc87ebd} {Suffrage:Voter ID:server-59d300b7-30d6-0976-d683-9115ee0b89ea Address:59d300b7-30d6-0976-d683-9115ee0b89ea}]" -2025-10-30T18:18:41.062Z [INFO] server-d2c34d70-1753-4cbe-33c3-edb13f282d02: entering follower state: follower="Node at d2c34d70-1753-4cbe-33c3-edb13f282d02 [Follower]" leader-address= leader-id= -2025-10-30T18:18:41.062Z [INFO] server-59d300b7-30d6-0976-d683-9115ee0b89ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2c34d70-1753-4cbe-33c3-edb13f282d02 Address:d2c34d70-1753-4cbe-33c3-edb13f282d02} {Suffrage:Voter ID:server-e54b2fd9-af1a-74be-3888-dc953cc87ebd Address:e54b2fd9-af1a-74be-3888-dc953cc87ebd} {Suffrage:Voter ID:server-59d300b7-30d6-0976-d683-9115ee0b89ea Address:59d300b7-30d6-0976-d683-9115ee0b89ea}]" -2025-10-30T18:18:41.062Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:41.062Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: entering follower state: follower="Node at e54b2fd9-af1a-74be-3888-dc953cc87ebd [Follower]" leader-address= leader-id= -2025-10-30T18:18:41.062Z [INFO] server-59d300b7-30d6-0976-d683-9115ee0b89ea: entering follower state: follower="Node at 59d300b7-30d6-0976-d683-9115ee0b89ea [Follower]" leader-address= leader-id= -2025-10-30T18:18:41.665Z [WARN] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:41.665Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: entering candidate state: node="Node at e54b2fd9-af1a-74be-3888-dc953cc87ebd [Candidate]" term=2 -2025-10-30T18:18:41.665Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: election won: term=2 tally=2 -2025-10-30T18:18:41.665Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: entering leader state: leader="Node at e54b2fd9-af1a-74be-3888-dc953cc87ebd [Leader]" -2025-10-30T18:18:41.665Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: added peer, starting replication: peer=server-d2c34d70-1753-4cbe-33c3-edb13f282d02 -2025-10-30T18:18:41.665Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: added peer, starting replication: peer=server-59d300b7-30d6-0976-d683-9115ee0b89ea -2025-10-30T18:18:41.665Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: pipelining replication: peer="{Voter server-59d300b7-30d6-0976-d683-9115ee0b89ea 59d300b7-30d6-0976-d683-9115ee0b89ea}" -2025-10-30T18:18:41.666Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: pipelining replication: peer="{Voter server-d2c34d70-1753-4cbe-33c3-edb13f282d02 d2c34d70-1753-4cbe-33c3-edb13f282d02}" -2025-10-30T18:18:42.671Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:41.665985715 +0000 UTC m=+67.944659627 (1 nodes), 603.69682ms from start of poll, 603.873856ms from cluster start. Timeout at 2025-10-30 18:18:42.67099003 +0000 UTC m=+68.949663930, 1.005004303s after stability -2025-10-30T18:18:42.671Z [INFO] creating new snapshot: path=/tmp/raft2680253167/snapshots/2-113-1761848322671.tmp -2025-10-30T18:18:42.672Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: copied to local snapshot: bytes=71 -2025-10-30T18:18:42.672Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: snapshot restore progress: id=2-113-1761848322671 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:42.672Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: removing all old logs from log store -2025-10-30T18:18:42.672Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: compacting logs: from=1 to=2 -2025-10-30T18:18:42.672Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: restored user snapshot: index=113 -2025-10-30T18:18:42.672Z [ERROR] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: failed to get log: index=2 error="log not found" -2025-10-30T18:18:42.672Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: aborting pipeline replication: peer="{Voter server-59d300b7-30d6-0976-d683-9115ee0b89ea 59d300b7-30d6-0976-d683-9115ee0b89ea}" -2025-10-30T18:18:42.672Z [ERROR] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: failed to get log: index=2 error="log not found" -2025-10-30T18:18:42.672Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: aborting pipeline replication: peer="{Voter server-d2c34d70-1753-4cbe-33c3-edb13f282d02 d2c34d70-1753-4cbe-33c3-edb13f282d02}" -2025-10-30T18:18:42.681Z [ERROR] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: failed to get log: index=2 error="log not found" -2025-10-30T18:18:42.681Z [INFO] creating new snapshot: path=/tmp/raft2653141277/snapshots/2-113-1761848322681.tmp -2025-10-30T18:18:42.681Z [ERROR] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: failed to get log: index=2 error="log not found" -2025-10-30T18:18:42.682Z [INFO] creating new snapshot: path=/tmp/raft45115175/snapshots/2-113-1761848322682.tmp -2025-10-30T18:18:42.682Z [INFO] server-59d300b7-30d6-0976-d683-9115ee0b89ea: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:42.683Z [INFO] server-d2c34d70-1753-4cbe-33c3-edb13f282d02: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:42.738Z [INFO] server-59d300b7-30d6-0976-d683-9115ee0b89ea: copied to local snapshot: bytes=71 -2025-10-30T18:18:42.738Z [INFO] server-d2c34d70-1753-4cbe-33c3-edb13f282d02: copied to local snapshot: bytes=71 -2025-10-30T18:18:42.738Z [INFO] server-d2c34d70-1753-4cbe-33c3-edb13f282d02: snapshot restore progress: id=2-113-1761848322682 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:42.738Z [INFO] server-59d300b7-30d6-0976-d683-9115ee0b89ea: snapshot restore progress: id=2-113-1761848322681 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:42.738Z [INFO] server-d2c34d70-1753-4cbe-33c3-edb13f282d02: removing all old logs from log store -2025-10-30T18:18:42.738Z [INFO] server-d2c34d70-1753-4cbe-33c3-edb13f282d02: compacting logs: from=1 to=2 -2025-10-30T18:18:42.738Z [INFO] server-d2c34d70-1753-4cbe-33c3-edb13f282d02: Installed remote snapshot -2025-10-30T18:18:42.738Z [INFO] server-59d300b7-30d6-0976-d683-9115ee0b89ea: removing all old logs from log store -2025-10-30T18:18:42.738Z [INFO] server-59d300b7-30d6-0976-d683-9115ee0b89ea: compacting logs: from=1 to=2 -2025-10-30T18:18:42.738Z [INFO] server-59d300b7-30d6-0976-d683-9115ee0b89ea: Installed remote snapshot -2025-10-30T18:18:42.738Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: pipelining replication: peer="{Voter server-d2c34d70-1753-4cbe-33c3-edb13f282d02 d2c34d70-1753-4cbe-33c3-edb13f282d02}" -2025-10-30T18:18:42.738Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:18:42.738Z [INFO] server-e54b2fd9-af1a-74be-3888-dc953cc87ebd: pipelining replication: peer="{Voter server-59d300b7-30d6-0976-d683-9115ee0b89ea 59d300b7-30d6-0976-d683-9115ee0b89ea}" -2025-10-30T18:18:43.743Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:18:42.738427127 +0000 UTC m=+69.017101033 (2 nodes), 662ns from start of poll, 1.676315262s from cluster start. Timeout at 2025-10-30 18:18:43.743448518 +0000 UTC m=+70.022122416, 1.005021383s after stability -2025-10-30T18:18:43.755Z [INFO] server-76a58140-ece6-e957-356d-750a6badabda: aborting pipeline replication: peer="{Voter server-1818bd9b-16af-9652-9038-d6af42c2c520 1818bd9b-16af-9652-9038-d6af42c2c520}" -2025-10-30T18:18:43.755Z [INFO] server-76a58140-ece6-e957-356d-750a6badabda: aborting pipeline replication: peer="{Voter server-748ca023-7768-a08f-8cd7-cf6701cd2f1a 748ca023-7768-a08f-8cd7-cf6701cd2f1a}" +2025-10-30T18:22:29.606Z [INFO] server-71c434e0-3c13-efe9-d953-c64e711f8d17: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-71c434e0-3c13-efe9-d953-c64e711f8d17 Address:71c434e0-3c13-efe9-d953-c64e711f8d17} {Suffrage:Voter ID:server-d026cf77-88fe-806c-88e1-5af56245b924 Address:d026cf77-88fe-806c-88e1-5af56245b924} {Suffrage:Voter ID:server-ab9d0f1e-ea88-c0dc-1a5e-0044fce3fa98 Address:ab9d0f1e-ea88-c0dc-1a5e-0044fce3fa98}]" +2025-10-30T18:22:29.607Z [INFO] server-71c434e0-3c13-efe9-d953-c64e711f8d17: entering follower state: follower="Node at 71c434e0-3c13-efe9-d953-c64e711f8d17 [Follower]" leader-address= leader-id= +2025-10-30T18:22:29.607Z [INFO] server-d026cf77-88fe-806c-88e1-5af56245b924: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-71c434e0-3c13-efe9-d953-c64e711f8d17 Address:71c434e0-3c13-efe9-d953-c64e711f8d17} {Suffrage:Voter ID:server-d026cf77-88fe-806c-88e1-5af56245b924 Address:d026cf77-88fe-806c-88e1-5af56245b924} {Suffrage:Voter ID:server-ab9d0f1e-ea88-c0dc-1a5e-0044fce3fa98 Address:ab9d0f1e-ea88-c0dc-1a5e-0044fce3fa98}]" +2025-10-30T18:22:29.607Z [INFO] server-d026cf77-88fe-806c-88e1-5af56245b924: entering follower state: follower="Node at d026cf77-88fe-806c-88e1-5af56245b924 [Follower]" leader-address= leader-id= +2025-10-30T18:22:29.607Z [INFO] server-ab9d0f1e-ea88-c0dc-1a5e-0044fce3fa98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-71c434e0-3c13-efe9-d953-c64e711f8d17 Address:71c434e0-3c13-efe9-d953-c64e711f8d17} {Suffrage:Voter ID:server-d026cf77-88fe-806c-88e1-5af56245b924 Address:d026cf77-88fe-806c-88e1-5af56245b924} {Suffrage:Voter ID:server-ab9d0f1e-ea88-c0dc-1a5e-0044fce3fa98 Address:ab9d0f1e-ea88-c0dc-1a5e-0044fce3fa98}]" +2025-10-30T18:22:29.607Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:29.607Z [INFO] server-ab9d0f1e-ea88-c0dc-1a5e-0044fce3fa98: entering follower state: follower="Node at ab9d0f1e-ea88-c0dc-1a5e-0044fce3fa98 [Follower]" leader-address= leader-id= +2025-10-30T18:22:30.152Z [WARN] server-71c434e0-3c13-efe9-d953-c64e711f8d17: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:30.152Z [INFO] server-71c434e0-3c13-efe9-d953-c64e711f8d17: entering candidate state: node="Node at 71c434e0-3c13-efe9-d953-c64e711f8d17 [Candidate]" term=2 +2025-10-30T18:22:30.152Z [INFO] server-71c434e0-3c13-efe9-d953-c64e711f8d17: election won: term=2 tally=2 +2025-10-30T18:22:30.152Z [INFO] server-71c434e0-3c13-efe9-d953-c64e711f8d17: entering leader state: leader="Node at 71c434e0-3c13-efe9-d953-c64e711f8d17 [Leader]" +2025-10-30T18:22:30.152Z [INFO] server-71c434e0-3c13-efe9-d953-c64e711f8d17: added peer, starting replication: peer=server-d026cf77-88fe-806c-88e1-5af56245b924 +2025-10-30T18:22:30.152Z [INFO] server-71c434e0-3c13-efe9-d953-c64e711f8d17: added peer, starting replication: peer=server-ab9d0f1e-ea88-c0dc-1a5e-0044fce3fa98 +2025-10-30T18:22:30.152Z [INFO] server-71c434e0-3c13-efe9-d953-c64e711f8d17: pipelining replication: peer="{Voter server-ab9d0f1e-ea88-c0dc-1a5e-0044fce3fa98 ab9d0f1e-ea88-c0dc-1a5e-0044fce3fa98}" +2025-10-30T18:22:30.152Z [INFO] server-71c434e0-3c13-efe9-d953-c64e711f8d17: pipelining replication: peer="{Voter server-d026cf77-88fe-806c-88e1-5af56245b924 d026cf77-88fe-806c-88e1-5af56245b924}" +2025-10-30T18:22:31.158Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:30.152784273 +0000 UTC m=+73.226313568 (1 nodes), 545.58595ms from start of poll, 545.931804ms from cluster start. Timeout at 2025-10-30 18:22:31.157788987 +0000 UTC m=+74.231318280, 1.005004712s after stability +2025-10-30T18:22:31.158Z [INFO] server-71c434e0-3c13-efe9-d953-c64e711f8d17: starting snapshot up to: index=12 +2025-10-30T18:22:31.158Z [INFO] creating new snapshot: path=/tmp/raft243129106/snapshots/2-12-1761848551158.tmp +2025-10-30T18:22:31.255Z [INFO] server-71c434e0-3c13-efe9-d953-c64e711f8d17: snapshot complete up to: index=12 +2025-10-30T18:22:31.256Z [INFO] server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a Address:3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a} {Suffrage:Voter ID:server-d90a6fc0-692f-02d2-8800-1bed4355b62e Address:d90a6fc0-692f-02d2-8800-1bed4355b62e} {Suffrage:Voter ID:server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466 Address:d35ebf55-0d21-fd1e-eb65-c2c2fba58466}]" +2025-10-30T18:22:31.256Z [INFO] server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a: entering follower state: follower="Node at 3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a [Follower]" leader-address= leader-id= +2025-10-30T18:22:31.256Z [INFO] server-d90a6fc0-692f-02d2-8800-1bed4355b62e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a Address:3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a} {Suffrage:Voter ID:server-d90a6fc0-692f-02d2-8800-1bed4355b62e Address:d90a6fc0-692f-02d2-8800-1bed4355b62e} {Suffrage:Voter ID:server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466 Address:d35ebf55-0d21-fd1e-eb65-c2c2fba58466}]" +2025-10-30T18:22:31.256Z [INFO] server-d90a6fc0-692f-02d2-8800-1bed4355b62e: entering follower state: follower="Node at d90a6fc0-692f-02d2-8800-1bed4355b62e [Follower]" leader-address= leader-id= +2025-10-30T18:22:31.256Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a Address:3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a} {Suffrage:Voter ID:server-d90a6fc0-692f-02d2-8800-1bed4355b62e Address:d90a6fc0-692f-02d2-8800-1bed4355b62e} {Suffrage:Voter ID:server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466 Address:d35ebf55-0d21-fd1e-eb65-c2c2fba58466}]" +2025-10-30T18:22:31.256Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:31.256Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: entering follower state: follower="Node at d35ebf55-0d21-fd1e-eb65-c2c2fba58466 [Follower]" leader-address= leader-id= +2025-10-30T18:22:32.128Z [WARN] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:32.128Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: entering candidate state: node="Node at d35ebf55-0d21-fd1e-eb65-c2c2fba58466 [Candidate]" term=2 +2025-10-30T18:22:32.128Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: election won: term=2 tally=2 +2025-10-30T18:22:32.128Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: entering leader state: leader="Node at d35ebf55-0d21-fd1e-eb65-c2c2fba58466 [Leader]" +2025-10-30T18:22:32.128Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: added peer, starting replication: peer=server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a +2025-10-30T18:22:32.128Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: added peer, starting replication: peer=server-d90a6fc0-692f-02d2-8800-1bed4355b62e +2025-10-30T18:22:32.128Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: pipelining replication: peer="{Voter server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a 3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a}" +2025-10-30T18:22:32.128Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: pipelining replication: peer="{Voter server-d90a6fc0-692f-02d2-8800-1bed4355b62e d90a6fc0-692f-02d2-8800-1bed4355b62e}" +2025-10-30T18:22:33.134Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:32.128319395 +0000 UTC m=+75.201848688 (1 nodes), 871.37118ms from start of poll, 871.680532ms from cluster start. Timeout at 2025-10-30 18:22:33.133320313 +0000 UTC m=+76.206849607, 1.005000919s after stability +2025-10-30T18:22:33.226Z [INFO] creating new snapshot: path=/tmp/raft3638428727/snapshots/2-113-1761848553226.tmp +2025-10-30T18:22:33.226Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: copied to local snapshot: bytes=71 +2025-10-30T18:22:33.227Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: snapshot restore progress: id=2-113-1761848553226 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:33.227Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: removing all old logs from log store +2025-10-30T18:22:33.227Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: compacting logs: from=1 to=2 +2025-10-30T18:22:33.227Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: restored user snapshot: index=113 +2025-10-30T18:22:33.227Z [ERROR] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: failed to get log: index=2 error="log not found" +2025-10-30T18:22:33.227Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: aborting pipeline replication: peer="{Voter server-d90a6fc0-692f-02d2-8800-1bed4355b62e d90a6fc0-692f-02d2-8800-1bed4355b62e}" +2025-10-30T18:22:33.227Z [ERROR] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: failed to get log: index=2 error="log not found" +2025-10-30T18:22:33.227Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: aborting pipeline replication: peer="{Voter server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a 3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a}" +2025-10-30T18:22:33.236Z [ERROR] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: failed to get log: index=2 error="log not found" +2025-10-30T18:22:33.236Z [ERROR] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: failed to get log: index=2 error="log not found" +2025-10-30T18:22:33.236Z [INFO] creating new snapshot: path=/tmp/raft2301632452/snapshots/2-113-1761848553236.tmp +2025-10-30T18:22:33.237Z [INFO] creating new snapshot: path=/tmp/raft2676165284/snapshots/2-113-1761848553237.tmp +2025-10-30T18:22:33.237Z [INFO] server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:33.237Z [INFO] server-d90a6fc0-692f-02d2-8800-1bed4355b62e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:33.478Z [INFO] server-d90a6fc0-692f-02d2-8800-1bed4355b62e: copied to local snapshot: bytes=71 +2025-10-30T18:22:33.478Z [INFO] server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a: copied to local snapshot: bytes=71 +2025-10-30T18:22:33.478Z [INFO] server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a: snapshot restore progress: id=2-113-1761848553236 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:33.478Z [INFO] server-d90a6fc0-692f-02d2-8800-1bed4355b62e: snapshot restore progress: id=2-113-1761848553237 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:33.478Z [INFO] server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a: removing all old logs from log store +2025-10-30T18:22:33.478Z [INFO] server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a: compacting logs: from=1 to=2 +2025-10-30T18:22:33.478Z [INFO] server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a: Installed remote snapshot +2025-10-30T18:22:33.478Z [INFO] server-d90a6fc0-692f-02d2-8800-1bed4355b62e: removing all old logs from log store +2025-10-30T18:22:33.478Z [INFO] server-d90a6fc0-692f-02d2-8800-1bed4355b62e: compacting logs: from=1 to=2 +2025-10-30T18:22:33.478Z [INFO] server-d90a6fc0-692f-02d2-8800-1bed4355b62e: Installed remote snapshot +2025-10-30T18:22:33.478Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: pipelining replication: peer="{Voter server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a 3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a}" +2025-10-30T18:22:33.478Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:22:33.478Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: pipelining replication: peer="{Voter server-d90a6fc0-692f-02d2-8800-1bed4355b62e d90a6fc0-692f-02d2-8800-1bed4355b62e}" +2025-10-30T18:22:34.484Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:22:33.478795227 +0000 UTC m=+76.552324522 (2 nodes), 685ns from start of poll, 2.222156366s from cluster start. Timeout at 2025-10-30 18:22:34.483824602 +0000 UTC m=+77.557353905, 1.005029383s after stability +2025-10-30T18:22:34.496Z [INFO] server-71c434e0-3c13-efe9-d953-c64e711f8d17: aborting pipeline replication: peer="{Voter server-ab9d0f1e-ea88-c0dc-1a5e-0044fce3fa98 ab9d0f1e-ea88-c0dc-1a5e-0044fce3fa98}" +2025-10-30T18:22:34.496Z [INFO] server-71c434e0-3c13-efe9-d953-c64e711f8d17: aborting pipeline replication: peer="{Voter server-d026cf77-88fe-806c-88e1-5af56245b924 d026cf77-88fe-806c-88e1-5af56245b924}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false -2025-10-30T18:18:43.756Z [INFO] server-81a0274e-5779-d62b-5744-e91b8b85ae8a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81a0274e-5779-d62b-5744-e91b8b85ae8a Address:81a0274e-5779-d62b-5744-e91b8b85ae8a} {Suffrage:Voter ID:server-842f6bb8-5908-4692-e352-be148691141a Address:842f6bb8-5908-4692-e352-be148691141a} {Suffrage:Voter ID:server-422d2ebc-0dfe-d501-9c4a-903214eb8a05 Address:422d2ebc-0dfe-d501-9c4a-903214eb8a05}]" -2025-10-30T18:18:43.756Z [INFO] server-842f6bb8-5908-4692-e352-be148691141a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81a0274e-5779-d62b-5744-e91b8b85ae8a Address:81a0274e-5779-d62b-5744-e91b8b85ae8a} {Suffrage:Voter ID:server-842f6bb8-5908-4692-e352-be148691141a Address:842f6bb8-5908-4692-e352-be148691141a} {Suffrage:Voter ID:server-422d2ebc-0dfe-d501-9c4a-903214eb8a05 Address:422d2ebc-0dfe-d501-9c4a-903214eb8a05}]" -2025-10-30T18:18:43.756Z [INFO] server-81a0274e-5779-d62b-5744-e91b8b85ae8a: entering follower state: follower="Node at 81a0274e-5779-d62b-5744-e91b8b85ae8a [Follower]" leader-address= leader-id= -2025-10-30T18:18:43.756Z [INFO] server-842f6bb8-5908-4692-e352-be148691141a: entering follower state: follower="Node at 842f6bb8-5908-4692-e352-be148691141a [Follower]" leader-address= leader-id= -2025-10-30T18:18:43.756Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81a0274e-5779-d62b-5744-e91b8b85ae8a Address:81a0274e-5779-d62b-5744-e91b8b85ae8a} {Suffrage:Voter ID:server-842f6bb8-5908-4692-e352-be148691141a Address:842f6bb8-5908-4692-e352-be148691141a} {Suffrage:Voter ID:server-422d2ebc-0dfe-d501-9c4a-903214eb8a05 Address:422d2ebc-0dfe-d501-9c4a-903214eb8a05}]" -2025-10-30T18:18:43.756Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:43.756Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: entering follower state: follower="Node at 422d2ebc-0dfe-d501-9c4a-903214eb8a05 [Follower]" leader-address= leader-id= -2025-10-30T18:18:44.388Z [WARN] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:44.389Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: entering candidate state: node="Node at 422d2ebc-0dfe-d501-9c4a-903214eb8a05 [Candidate]" term=2 -2025-10-30T18:18:44.389Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: election won: term=2 tally=2 -2025-10-30T18:18:44.389Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: entering leader state: leader="Node at 422d2ebc-0dfe-d501-9c4a-903214eb8a05 [Leader]" -2025-10-30T18:18:44.389Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: added peer, starting replication: peer=server-81a0274e-5779-d62b-5744-e91b8b85ae8a -2025-10-30T18:18:44.389Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: added peer, starting replication: peer=server-842f6bb8-5908-4692-e352-be148691141a -2025-10-30T18:18:44.389Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: pipelining replication: peer="{Voter server-842f6bb8-5908-4692-e352-be148691141a 842f6bb8-5908-4692-e352-be148691141a}" -2025-10-30T18:18:44.389Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: pipelining replication: peer="{Voter server-81a0274e-5779-d62b-5744-e91b8b85ae8a 81a0274e-5779-d62b-5744-e91b8b85ae8a}" -2025-10-30T18:18:45.394Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:44.389177553 +0000 UTC m=+70.667851449 (1 nodes), 632.931373ms from start of poll, 633.199885ms from cluster start. Timeout at 2025-10-30 18:18:45.394181221 +0000 UTC m=+71.672855118, 1.005003669s after stability -2025-10-30T18:18:45.395Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: starting snapshot up to: index=12 -2025-10-30T18:18:45.395Z [INFO] creating new snapshot: path=/tmp/raft3146786988/snapshots/2-12-1761848325395.tmp -2025-10-30T18:18:45.520Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: snapshot complete up to: index=12 -2025-10-30T18:18:45.521Z [INFO] creating new snapshot: path=/tmp/raft3146786988/snapshots/2-1013-1761848325521.tmp -2025-10-30T18:18:45.521Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: copied to local snapshot: bytes=71 -2025-10-30T18:18:45.521Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: snapshot restore progress: id=2-1013-1761848325521 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:45.521Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: restored user snapshot: index=1013 -2025-10-30T18:18:45.521Z [ERROR] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: failed to get log: index=23 error="log not found" -2025-10-30T18:18:45.521Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: aborting pipeline replication: peer="{Voter server-842f6bb8-5908-4692-e352-be148691141a 842f6bb8-5908-4692-e352-be148691141a}" -2025-10-30T18:18:45.521Z [ERROR] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: failed to get log: index=23 error="log not found" -2025-10-30T18:18:45.521Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: aborting pipeline replication: peer="{Voter server-81a0274e-5779-d62b-5744-e91b8b85ae8a 81a0274e-5779-d62b-5744-e91b8b85ae8a}" -2025-10-30T18:18:45.528Z [ERROR] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: failed to get log: index=23 error="log not found" -2025-10-30T18:18:45.529Z [ERROR] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: failed to get log: index=23 error="log not found" -2025-10-30T18:18:45.654Z [INFO] creating new snapshot: path=/tmp/raft3184662359/snapshots/2-1013-1761848325654.tmp -2025-10-30T18:18:45.654Z [INFO] creating new snapshot: path=/tmp/raft4105804530/snapshots/2-1013-1761848325654.tmp -2025-10-30T18:18:45.679Z [INFO] server-81a0274e-5779-d62b-5744-e91b8b85ae8a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:45.679Z [INFO] server-842f6bb8-5908-4692-e352-be148691141a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:45.679Z [INFO] server-842f6bb8-5908-4692-e352-be148691141a: copied to local snapshot: bytes=71 -2025-10-30T18:18:45.679Z [INFO] server-81a0274e-5779-d62b-5744-e91b8b85ae8a: copied to local snapshot: bytes=71 -2025-10-30T18:18:45.679Z [INFO] server-842f6bb8-5908-4692-e352-be148691141a: snapshot restore progress: id=2-1013-1761848325654 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:45.679Z [INFO] server-842f6bb8-5908-4692-e352-be148691141a: Installed remote snapshot -2025-10-30T18:18:45.679Z [INFO] server-81a0274e-5779-d62b-5744-e91b8b85ae8a: snapshot restore progress: id=2-1013-1761848325654 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:45.679Z [INFO] server-81a0274e-5779-d62b-5744-e91b8b85ae8a: Installed remote snapshot -2025-10-30T18:18:45.679Z [WARN] server-842f6bb8-5908-4692-e352-be148691141a: skipping application of old log: index=22 -2025-10-30T18:18:45.679Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: pipelining replication: peer="{Voter server-842f6bb8-5908-4692-e352-be148691141a 842f6bb8-5908-4692-e352-be148691141a}" -2025-10-30T18:18:45.679Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:18:45.679Z [WARN] server-81a0274e-5779-d62b-5744-e91b8b85ae8a: skipping application of old log: index=1013 -2025-10-30T18:18:45.679Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: pipelining replication: peer="{Voter server-81a0274e-5779-d62b-5744-e91b8b85ae8a 81a0274e-5779-d62b-5744-e91b8b85ae8a}" -2025-10-30T18:18:46.685Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:18:45.679825985 +0000 UTC m=+71.958499882 (2 nodes), 722ns from start of poll, 1.923848318s from cluster start. Timeout at 2025-10-30 18:18:46.684834769 +0000 UTC m=+72.963508684, 1.005008802s after stability -2025-10-30T18:18:46.696Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: aborting pipeline replication: peer="{Voter server-842f6bb8-5908-4692-e352-be148691141a 842f6bb8-5908-4692-e352-be148691141a}" -2025-10-30T18:18:46.696Z [INFO] server-422d2ebc-0dfe-d501-9c4a-903214eb8a05: aborting pipeline replication: peer="{Voter server-81a0274e-5779-d62b-5744-e91b8b85ae8a 81a0274e-5779-d62b-5744-e91b8b85ae8a}" +2025-10-30T18:22:34.538Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1197e970-431a-4c7e-5044-e629a6af7ce8 Address:1197e970-431a-4c7e-5044-e629a6af7ce8} {Suffrage:Voter ID:server-bc8a923c-0eaa-e72d-87ca-2dca77a14574 Address:bc8a923c-0eaa-e72d-87ca-2dca77a14574} {Suffrage:Voter ID:server-52349645-ff65-1dd4-da1e-f7919df11d22 Address:52349645-ff65-1dd4-da1e-f7919df11d22}]" +2025-10-30T18:22:34.538Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: entering follower state: follower="Node at 1197e970-431a-4c7e-5044-e629a6af7ce8 [Follower]" leader-address= leader-id= +2025-10-30T18:22:34.538Z [INFO] server-bc8a923c-0eaa-e72d-87ca-2dca77a14574: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1197e970-431a-4c7e-5044-e629a6af7ce8 Address:1197e970-431a-4c7e-5044-e629a6af7ce8} {Suffrage:Voter ID:server-bc8a923c-0eaa-e72d-87ca-2dca77a14574 Address:bc8a923c-0eaa-e72d-87ca-2dca77a14574} {Suffrage:Voter ID:server-52349645-ff65-1dd4-da1e-f7919df11d22 Address:52349645-ff65-1dd4-da1e-f7919df11d22}]" +2025-10-30T18:22:34.538Z [INFO] server-bc8a923c-0eaa-e72d-87ca-2dca77a14574: entering follower state: follower="Node at bc8a923c-0eaa-e72d-87ca-2dca77a14574 [Follower]" leader-address= leader-id= +2025-10-30T18:22:34.538Z [INFO] server-52349645-ff65-1dd4-da1e-f7919df11d22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1197e970-431a-4c7e-5044-e629a6af7ce8 Address:1197e970-431a-4c7e-5044-e629a6af7ce8} {Suffrage:Voter ID:server-bc8a923c-0eaa-e72d-87ca-2dca77a14574 Address:bc8a923c-0eaa-e72d-87ca-2dca77a14574} {Suffrage:Voter ID:server-52349645-ff65-1dd4-da1e-f7919df11d22 Address:52349645-ff65-1dd4-da1e-f7919df11d22}]" +2025-10-30T18:22:34.538Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:34.538Z [INFO] server-52349645-ff65-1dd4-da1e-f7919df11d22: entering follower state: follower="Node at 52349645-ff65-1dd4-da1e-f7919df11d22 [Follower]" leader-address= leader-id= +2025-10-30T18:22:35.043Z [WARN] server-1197e970-431a-4c7e-5044-e629a6af7ce8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:35.043Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: entering candidate state: node="Node at 1197e970-431a-4c7e-5044-e629a6af7ce8 [Candidate]" term=2 +2025-10-30T18:22:35.043Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: election won: term=2 tally=2 +2025-10-30T18:22:35.043Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: entering leader state: leader="Node at 1197e970-431a-4c7e-5044-e629a6af7ce8 [Leader]" +2025-10-30T18:22:35.043Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: added peer, starting replication: peer=server-bc8a923c-0eaa-e72d-87ca-2dca77a14574 +2025-10-30T18:22:35.043Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: added peer, starting replication: peer=server-52349645-ff65-1dd4-da1e-f7919df11d22 +2025-10-30T18:22:35.043Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: pipelining replication: peer="{Voter server-52349645-ff65-1dd4-da1e-f7919df11d22 52349645-ff65-1dd4-da1e-f7919df11d22}" +2025-10-30T18:22:35.043Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: pipelining replication: peer="{Voter server-bc8a923c-0eaa-e72d-87ca-2dca77a14574 bc8a923c-0eaa-e72d-87ca-2dca77a14574}" +2025-10-30T18:22:36.048Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:35.04355678 +0000 UTC m=+78.117086076 (1 nodes), 504.891118ms from start of poll, 505.190816ms from cluster start. Timeout at 2025-10-30 18:22:36.048567497 +0000 UTC m=+79.122096785, 1.005010709s after stability +2025-10-30T18:22:36.049Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: starting snapshot up to: index=12 +2025-10-30T18:22:36.049Z [INFO] creating new snapshot: path=/tmp/raft3203681927/snapshots/2-12-1761848556049.tmp +2025-10-30T18:22:36.049Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: snapshot complete up to: index=12 +2025-10-30T18:22:36.051Z [INFO] creating new snapshot: path=/tmp/raft3203681927/snapshots/2-1013-1761848556051.tmp +2025-10-30T18:22:36.051Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: copied to local snapshot: bytes=71 +2025-10-30T18:22:36.052Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: snapshot restore progress: id=2-1013-1761848556051 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:36.052Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: restored user snapshot: index=1013 +2025-10-30T18:22:36.052Z [ERROR] server-1197e970-431a-4c7e-5044-e629a6af7ce8: failed to get log: index=23 error="log not found" +2025-10-30T18:22:36.052Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: aborting pipeline replication: peer="{Voter server-52349645-ff65-1dd4-da1e-f7919df11d22 52349645-ff65-1dd4-da1e-f7919df11d22}" +2025-10-30T18:22:36.052Z [ERROR] server-1197e970-431a-4c7e-5044-e629a6af7ce8: failed to get log: index=23 error="log not found" +2025-10-30T18:22:36.052Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: aborting pipeline replication: peer="{Voter server-bc8a923c-0eaa-e72d-87ca-2dca77a14574 bc8a923c-0eaa-e72d-87ca-2dca77a14574}" +2025-10-30T18:22:36.060Z [ERROR] server-1197e970-431a-4c7e-5044-e629a6af7ce8: failed to get log: index=23 error="log not found" +2025-10-30T18:22:36.060Z [INFO] creating new snapshot: path=/tmp/raft3083180536/snapshots/2-1013-1761848556060.tmp +2025-10-30T18:22:36.060Z [INFO] server-52349645-ff65-1dd4-da1e-f7919df11d22: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:36.060Z [INFO] server-52349645-ff65-1dd4-da1e-f7919df11d22: copied to local snapshot: bytes=71 +2025-10-30T18:22:36.061Z [INFO] server-52349645-ff65-1dd4-da1e-f7919df11d22: snapshot restore progress: id=2-1013-1761848556060 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:36.061Z [INFO] server-52349645-ff65-1dd4-da1e-f7919df11d22: Installed remote snapshot +2025-10-30T18:22:36.061Z [WARN] server-52349645-ff65-1dd4-da1e-f7919df11d22: skipping application of old log: index=22 +2025-10-30T18:22:36.061Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: pipelining replication: peer="{Voter server-52349645-ff65-1dd4-da1e-f7919df11d22 52349645-ff65-1dd4-da1e-f7919df11d22}" +2025-10-30T18:22:36.061Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:22:36.062Z [ERROR] server-1197e970-431a-4c7e-5044-e629a6af7ce8: failed to get log: index=23 error="log not found" +2025-10-30T18:22:36.062Z [INFO] creating new snapshot: path=/tmp/raft2815588671/snapshots/2-1013-1761848556062.tmp +2025-10-30T18:22:36.062Z [INFO] server-bc8a923c-0eaa-e72d-87ca-2dca77a14574: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:36.063Z [INFO] server-bc8a923c-0eaa-e72d-87ca-2dca77a14574: copied to local snapshot: bytes=71 +2025-10-30T18:22:36.063Z [INFO] server-bc8a923c-0eaa-e72d-87ca-2dca77a14574: snapshot restore progress: id=2-1013-1761848556062 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:36.063Z [INFO] server-bc8a923c-0eaa-e72d-87ca-2dca77a14574: Installed remote snapshot +2025-10-30T18:22:36.063Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: pipelining replication: peer="{Voter server-bc8a923c-0eaa-e72d-87ca-2dca77a14574 bc8a923c-0eaa-e72d-87ca-2dca77a14574}" +2025-10-30T18:22:37.066Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:22:36.061275865 +0000 UTC m=+79.134805154 (2 nodes), 391ns from start of poll, 1.522909894s from cluster start. Timeout at 2025-10-30 18:22:37.066293408 +0000 UTC m=+80.139822721, 1.005017567s after stability +2025-10-30T18:22:37.078Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: aborting pipeline replication: peer="{Voter server-52349645-ff65-1dd4-da1e-f7919df11d22 52349645-ff65-1dd4-da1e-f7919df11d22}" +2025-10-30T18:22:37.078Z [INFO] server-1197e970-431a-4c7e-5044-e629a6af7ce8: aborting pipeline replication: peer="{Voter server-bc8a923c-0eaa-e72d-87ca-2dca77a14574 bc8a923c-0eaa-e72d-87ca-2dca77a14574}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false -2025-10-30T18:18:46.697Z [INFO] server-cf027ee5-214a-039a-38b7-d28ee558c1d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf027ee5-214a-039a-38b7-d28ee558c1d4 Address:cf027ee5-214a-039a-38b7-d28ee558c1d4} {Suffrage:Voter ID:server-aa55c372-30da-8a30-8a67-1c2b879e8526 Address:aa55c372-30da-8a30-8a67-1c2b879e8526} {Suffrage:Voter ID:server-7b46b1e4-7fca-26ac-662c-cd4623daf019 Address:7b46b1e4-7fca-26ac-662c-cd4623daf019}]" -2025-10-30T18:18:46.697Z [INFO] server-cf027ee5-214a-039a-38b7-d28ee558c1d4: entering follower state: follower="Node at cf027ee5-214a-039a-38b7-d28ee558c1d4 [Follower]" leader-address= leader-id= -2025-10-30T18:18:46.697Z [INFO] server-aa55c372-30da-8a30-8a67-1c2b879e8526: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf027ee5-214a-039a-38b7-d28ee558c1d4 Address:cf027ee5-214a-039a-38b7-d28ee558c1d4} {Suffrage:Voter ID:server-aa55c372-30da-8a30-8a67-1c2b879e8526 Address:aa55c372-30da-8a30-8a67-1c2b879e8526} {Suffrage:Voter ID:server-7b46b1e4-7fca-26ac-662c-cd4623daf019 Address:7b46b1e4-7fca-26ac-662c-cd4623daf019}]" -2025-10-30T18:18:46.697Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf027ee5-214a-039a-38b7-d28ee558c1d4 Address:cf027ee5-214a-039a-38b7-d28ee558c1d4} {Suffrage:Voter ID:server-aa55c372-30da-8a30-8a67-1c2b879e8526 Address:aa55c372-30da-8a30-8a67-1c2b879e8526} {Suffrage:Voter ID:server-7b46b1e4-7fca-26ac-662c-cd4623daf019 Address:7b46b1e4-7fca-26ac-662c-cd4623daf019}]" -2025-10-30T18:18:46.697Z [INFO] server-aa55c372-30da-8a30-8a67-1c2b879e8526: entering follower state: follower="Node at aa55c372-30da-8a30-8a67-1c2b879e8526 [Follower]" leader-address= leader-id= -2025-10-30T18:18:46.698Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:46.698Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: entering follower state: follower="Node at 7b46b1e4-7fca-26ac-662c-cd4623daf019 [Follower]" leader-address= leader-id= -2025-10-30T18:18:47.217Z [WARN] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:47.217Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: entering candidate state: node="Node at 7b46b1e4-7fca-26ac-662c-cd4623daf019 [Candidate]" term=2 -2025-10-30T18:18:47.217Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: election won: term=2 tally=2 -2025-10-30T18:18:47.217Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: entering leader state: leader="Node at 7b46b1e4-7fca-26ac-662c-cd4623daf019 [Leader]" -2025-10-30T18:18:47.217Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: added peer, starting replication: peer=server-cf027ee5-214a-039a-38b7-d28ee558c1d4 -2025-10-30T18:18:47.217Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: added peer, starting replication: peer=server-aa55c372-30da-8a30-8a67-1c2b879e8526 -2025-10-30T18:18:47.217Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: pipelining replication: peer="{Voter server-cf027ee5-214a-039a-38b7-d28ee558c1d4 cf027ee5-214a-039a-38b7-d28ee558c1d4}" -2025-10-30T18:18:47.217Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: pipelining replication: peer="{Voter server-aa55c372-30da-8a30-8a67-1c2b879e8526 aa55c372-30da-8a30-8a67-1c2b879e8526}" -2025-10-30T18:18:48.222Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:47.217179052 +0000 UTC m=+73.495852950 (1 nodes), 519.150914ms from start of poll, 519.368117ms from cluster start. Timeout at 2025-10-30 18:18:48.222181107 +0000 UTC m=+74.500855004, 1.005002054s after stability -2025-10-30T18:18:48.222Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: starting snapshot up to: index=12 -2025-10-30T18:18:48.222Z [INFO] creating new snapshot: path=/tmp/raft349724726/snapshots/2-12-1761848328222.tmp -2025-10-30T18:18:48.372Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: snapshot complete up to: index=12 -2025-10-30T18:18:48.372Z [INFO] creating new snapshot: path=/tmp/raft349724726/snapshots/2-1013-1761848328372.tmp -2025-10-30T18:18:48.482Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: copied to local snapshot: bytes=71 -2025-10-30T18:18:48.482Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: snapshot restore progress: id=2-1013-1761848328372 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:48.482Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: removing all old logs from log store -2025-10-30T18:18:48.482Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: compacting logs: from=1 to=22 -2025-10-30T18:18:48.482Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: restored user snapshot: index=1013 -2025-10-30T18:18:48.482Z [ERROR] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: failed to get log: index=22 error="log not found" -2025-10-30T18:18:48.482Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: aborting pipeline replication: peer="{Voter server-aa55c372-30da-8a30-8a67-1c2b879e8526 aa55c372-30da-8a30-8a67-1c2b879e8526}" -2025-10-30T18:18:48.482Z [ERROR] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: failed to get log: index=22 error="log not found" -2025-10-30T18:18:48.482Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: aborting pipeline replication: peer="{Voter server-cf027ee5-214a-039a-38b7-d28ee558c1d4 cf027ee5-214a-039a-38b7-d28ee558c1d4}" -2025-10-30T18:18:48.488Z [ERROR] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: failed to get log: index=22 error="log not found" -2025-10-30T18:18:48.488Z [ERROR] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: failed to get log: index=22 error="log not found" -2025-10-30T18:18:48.489Z [INFO] creating new snapshot: path=/tmp/raft2728368928/snapshots/2-1013-1761848328489.tmp -2025-10-30T18:18:48.609Z [INFO] server-aa55c372-30da-8a30-8a67-1c2b879e8526: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:48.609Z [INFO] creating new snapshot: path=/tmp/raft1174517755/snapshots/2-1013-1761848328609.tmp -2025-10-30T18:18:48.609Z [INFO] server-cf027ee5-214a-039a-38b7-d28ee558c1d4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:48.609Z [INFO] server-aa55c372-30da-8a30-8a67-1c2b879e8526: copied to local snapshot: bytes=71 -2025-10-30T18:18:48.609Z [INFO] server-aa55c372-30da-8a30-8a67-1c2b879e8526: snapshot restore progress: id=2-1013-1761848328489 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:48.609Z [INFO] server-aa55c372-30da-8a30-8a67-1c2b879e8526: removing all old logs from log store -2025-10-30T18:18:48.609Z [INFO] server-aa55c372-30da-8a30-8a67-1c2b879e8526: compacting logs: from=1 to=22 -2025-10-30T18:18:48.609Z [INFO] server-aa55c372-30da-8a30-8a67-1c2b879e8526: Installed remote snapshot -2025-10-30T18:18:48.609Z [INFO] server-cf027ee5-214a-039a-38b7-d28ee558c1d4: copied to local snapshot: bytes=71 -2025-10-30T18:18:48.609Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: pipelining replication: peer="{Voter server-aa55c372-30da-8a30-8a67-1c2b879e8526 aa55c372-30da-8a30-8a67-1c2b879e8526}" -2025-10-30T18:18:48.609Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:18:48.610Z [INFO] server-cf027ee5-214a-039a-38b7-d28ee558c1d4: snapshot restore progress: id=2-1013-1761848328609 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:48.610Z [INFO] server-cf027ee5-214a-039a-38b7-d28ee558c1d4: removing all old logs from log store -2025-10-30T18:18:48.610Z [INFO] server-cf027ee5-214a-039a-38b7-d28ee558c1d4: compacting logs: from=1 to=22 -2025-10-30T18:18:48.610Z [INFO] server-cf027ee5-214a-039a-38b7-d28ee558c1d4: Installed remote snapshot -2025-10-30T18:18:48.610Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: pipelining replication: peer="{Voter server-cf027ee5-214a-039a-38b7-d28ee558c1d4 cf027ee5-214a-039a-38b7-d28ee558c1d4}" -2025-10-30T18:18:49.615Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:18:48.609947107 +0000 UTC m=+74.888621006 (2 nodes), 700ns from start of poll, 1.912136173s from cluster start. Timeout at 2025-10-30 18:18:49.614965579 +0000 UTC m=+75.893639476, 1.00501847s after stability -2025-10-30T18:18:49.626Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: aborting pipeline replication: peer="{Voter server-aa55c372-30da-8a30-8a67-1c2b879e8526 aa55c372-30da-8a30-8a67-1c2b879e8526}" -2025-10-30T18:18:49.626Z [INFO] server-7b46b1e4-7fca-26ac-662c-cd4623daf019: aborting pipeline replication: peer="{Voter server-cf027ee5-214a-039a-38b7-d28ee558c1d4 cf027ee5-214a-039a-38b7-d28ee558c1d4}" +2025-10-30T18:22:37.080Z [INFO] server-4238ad44-7a14-07d1-f569-6b50865f7a97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4238ad44-7a14-07d1-f569-6b50865f7a97 Address:4238ad44-7a14-07d1-f569-6b50865f7a97} {Suffrage:Voter ID:server-b361e1d6-4f01-baae-9d49-5890970bc587 Address:b361e1d6-4f01-baae-9d49-5890970bc587} {Suffrage:Voter ID:server-ba134f04-ced7-1603-1cad-cd72aba938ce Address:ba134f04-ced7-1603-1cad-cd72aba938ce}]" +2025-10-30T18:22:37.080Z [INFO] server-4238ad44-7a14-07d1-f569-6b50865f7a97: entering follower state: follower="Node at 4238ad44-7a14-07d1-f569-6b50865f7a97 [Follower]" leader-address= leader-id= +2025-10-30T18:22:37.080Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4238ad44-7a14-07d1-f569-6b50865f7a97 Address:4238ad44-7a14-07d1-f569-6b50865f7a97} {Suffrage:Voter ID:server-b361e1d6-4f01-baae-9d49-5890970bc587 Address:b361e1d6-4f01-baae-9d49-5890970bc587} {Suffrage:Voter ID:server-ba134f04-ced7-1603-1cad-cd72aba938ce Address:ba134f04-ced7-1603-1cad-cd72aba938ce}]" +2025-10-30T18:22:37.080Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: entering follower state: follower="Node at b361e1d6-4f01-baae-9d49-5890970bc587 [Follower]" leader-address= leader-id= +2025-10-30T18:22:37.080Z [INFO] server-ba134f04-ced7-1603-1cad-cd72aba938ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4238ad44-7a14-07d1-f569-6b50865f7a97 Address:4238ad44-7a14-07d1-f569-6b50865f7a97} {Suffrage:Voter ID:server-b361e1d6-4f01-baae-9d49-5890970bc587 Address:b361e1d6-4f01-baae-9d49-5890970bc587} {Suffrage:Voter ID:server-ba134f04-ced7-1603-1cad-cd72aba938ce Address:ba134f04-ced7-1603-1cad-cd72aba938ce}]" +2025-10-30T18:22:37.080Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:37.080Z [INFO] server-ba134f04-ced7-1603-1cad-cd72aba938ce: entering follower state: follower="Node at ba134f04-ced7-1603-1cad-cd72aba938ce [Follower]" leader-address= leader-id= +2025-10-30T18:22:37.658Z [WARN] server-b361e1d6-4f01-baae-9d49-5890970bc587: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:37.658Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: entering candidate state: node="Node at b361e1d6-4f01-baae-9d49-5890970bc587 [Candidate]" term=2 +2025-10-30T18:22:37.658Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: election won: term=2 tally=2 +2025-10-30T18:22:37.658Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: entering leader state: leader="Node at b361e1d6-4f01-baae-9d49-5890970bc587 [Leader]" +2025-10-30T18:22:37.658Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: added peer, starting replication: peer=server-4238ad44-7a14-07d1-f569-6b50865f7a97 +2025-10-30T18:22:37.658Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: added peer, starting replication: peer=server-ba134f04-ced7-1603-1cad-cd72aba938ce +2025-10-30T18:22:37.658Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: pipelining replication: peer="{Voter server-4238ad44-7a14-07d1-f569-6b50865f7a97 4238ad44-7a14-07d1-f569-6b50865f7a97}" +2025-10-30T18:22:37.658Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: pipelining replication: peer="{Voter server-ba134f04-ced7-1603-1cad-cd72aba938ce ba134f04-ced7-1603-1cad-cd72aba938ce}" +2025-10-30T18:22:38.664Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:37.658881666 +0000 UTC m=+80.732410954 (1 nodes), 578.328779ms from start of poll, 578.904366ms from cluster start. Timeout at 2025-10-30 18:22:38.663887255 +0000 UTC m=+81.737416547, 1.005005593s after stability +2025-10-30T18:22:38.664Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: starting snapshot up to: index=12 +2025-10-30T18:22:38.664Z [INFO] creating new snapshot: path=/tmp/raft1355903728/snapshots/2-12-1761848558664.tmp +2025-10-30T18:22:38.734Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: snapshot complete up to: index=12 +2025-10-30T18:22:38.735Z [INFO] creating new snapshot: path=/tmp/raft1355903728/snapshots/2-1013-1761848558735.tmp +2025-10-30T18:22:38.735Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: copied to local snapshot: bytes=71 +2025-10-30T18:22:38.735Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: snapshot restore progress: id=2-1013-1761848558735 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:38.735Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: removing all old logs from log store +2025-10-30T18:22:38.735Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: compacting logs: from=1 to=22 +2025-10-30T18:22:38.736Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: restored user snapshot: index=1013 +2025-10-30T18:22:38.736Z [ERROR] server-b361e1d6-4f01-baae-9d49-5890970bc587: failed to get log: index=22 error="log not found" +2025-10-30T18:22:38.736Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: aborting pipeline replication: peer="{Voter server-ba134f04-ced7-1603-1cad-cd72aba938ce ba134f04-ced7-1603-1cad-cd72aba938ce}" +2025-10-30T18:22:38.736Z [ERROR] server-b361e1d6-4f01-baae-9d49-5890970bc587: failed to get log: index=22 error="log not found" +2025-10-30T18:22:38.736Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: aborting pipeline replication: peer="{Voter server-4238ad44-7a14-07d1-f569-6b50865f7a97 4238ad44-7a14-07d1-f569-6b50865f7a97}" +2025-10-30T18:22:38.745Z [ERROR] server-b361e1d6-4f01-baae-9d49-5890970bc587: failed to get log: index=22 error="log not found" +2025-10-30T18:22:38.758Z [ERROR] server-b361e1d6-4f01-baae-9d49-5890970bc587: failed to get log: index=22 error="log not found" +2025-10-30T18:22:38.886Z [INFO] creating new snapshot: path=/tmp/raft2726288454/snapshots/2-1013-1761848558886.tmp +2025-10-30T18:22:38.886Z [INFO] creating new snapshot: path=/tmp/raft3929744612/snapshots/2-1013-1761848558886.tmp +2025-10-30T18:22:38.887Z [INFO] server-4238ad44-7a14-07d1-f569-6b50865f7a97: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:38.887Z [INFO] server-ba134f04-ced7-1603-1cad-cd72aba938ce: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:38.887Z [INFO] server-ba134f04-ced7-1603-1cad-cd72aba938ce: copied to local snapshot: bytes=71 +2025-10-30T18:22:38.887Z [INFO] server-4238ad44-7a14-07d1-f569-6b50865f7a97: copied to local snapshot: bytes=71 +2025-10-30T18:22:38.887Z [INFO] server-ba134f04-ced7-1603-1cad-cd72aba938ce: snapshot restore progress: id=2-1013-1761848558886 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:38.887Z [INFO] server-ba134f04-ced7-1603-1cad-cd72aba938ce: removing all old logs from log store +2025-10-30T18:22:38.887Z [INFO] server-ba134f04-ced7-1603-1cad-cd72aba938ce: compacting logs: from=1 to=22 +2025-10-30T18:22:38.887Z [INFO] server-4238ad44-7a14-07d1-f569-6b50865f7a97: snapshot restore progress: id=2-1013-1761848558886 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:38.887Z [INFO] server-ba134f04-ced7-1603-1cad-cd72aba938ce: Installed remote snapshot +2025-10-30T18:22:38.887Z [INFO] server-4238ad44-7a14-07d1-f569-6b50865f7a97: removing all old logs from log store +2025-10-30T18:22:38.887Z [INFO] server-4238ad44-7a14-07d1-f569-6b50865f7a97: compacting logs: from=1 to=22 +2025-10-30T18:22:38.887Z [INFO] server-4238ad44-7a14-07d1-f569-6b50865f7a97: Installed remote snapshot +2025-10-30T18:22:38.887Z [WARN] server-ba134f04-ced7-1603-1cad-cd72aba938ce: skipping application of old log: index=1013 +2025-10-30T18:22:38.887Z [WARN] server-4238ad44-7a14-07d1-f569-6b50865f7a97: skipping application of old log: index=1013 +2025-10-30T18:22:38.887Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: pipelining replication: peer="{Voter server-ba134f04-ced7-1603-1cad-cd72aba938ce ba134f04-ced7-1603-1cad-cd72aba938ce}" +2025-10-30T18:22:38.887Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:22:38.887Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: pipelining replication: peer="{Voter server-4238ad44-7a14-07d1-f569-6b50865f7a97 4238ad44-7a14-07d1-f569-6b50865f7a97}" +2025-10-30T18:22:39.893Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:22:38.887730849 +0000 UTC m=+81.961260149 (2 nodes), 1.111µs from start of poll, 1.807753561s from cluster start. Timeout at 2025-10-30 18:22:39.892737965 +0000 UTC m=+82.966267275, 1.005007126s after stability +2025-10-30T18:22:39.904Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: aborting pipeline replication: peer="{Voter server-ba134f04-ced7-1603-1cad-cd72aba938ce ba134f04-ced7-1603-1cad-cd72aba938ce}" +2025-10-30T18:22:39.904Z [INFO] server-b361e1d6-4f01-baae-9d49-5890970bc587: aborting pipeline replication: peer="{Voter server-4238ad44-7a14-07d1-f569-6b50865f7a97 4238ad44-7a14-07d1-f569-6b50865f7a97}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true -2025-10-30T18:18:49.627Z [INFO] server-01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8 Address:01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8} {Suffrage:Voter ID:server-4dab8c44-22ef-72e7-6a5c-e907d894a332 Address:4dab8c44-22ef-72e7-6a5c-e907d894a332} {Suffrage:Voter ID:server-a4a5259c-dcb5-29a9-5212-37b471217af7 Address:a4a5259c-dcb5-29a9-5212-37b471217af7}]" -2025-10-30T18:18:49.627Z [INFO] server-01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8: entering follower state: follower="Node at 01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8 [Follower]" leader-address= leader-id= -2025-10-30T18:18:49.627Z [INFO] server-4dab8c44-22ef-72e7-6a5c-e907d894a332: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8 Address:01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8} {Suffrage:Voter ID:server-4dab8c44-22ef-72e7-6a5c-e907d894a332 Address:4dab8c44-22ef-72e7-6a5c-e907d894a332} {Suffrage:Voter ID:server-a4a5259c-dcb5-29a9-5212-37b471217af7 Address:a4a5259c-dcb5-29a9-5212-37b471217af7}]" -2025-10-30T18:18:49.628Z [INFO] server-4dab8c44-22ef-72e7-6a5c-e907d894a332: entering follower state: follower="Node at 4dab8c44-22ef-72e7-6a5c-e907d894a332 [Follower]" leader-address= leader-id= -2025-10-30T18:18:49.628Z [INFO] server-a4a5259c-dcb5-29a9-5212-37b471217af7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8 Address:01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8} {Suffrage:Voter ID:server-4dab8c44-22ef-72e7-6a5c-e907d894a332 Address:4dab8c44-22ef-72e7-6a5c-e907d894a332} {Suffrage:Voter ID:server-a4a5259c-dcb5-29a9-5212-37b471217af7 Address:a4a5259c-dcb5-29a9-5212-37b471217af7}]" -2025-10-30T18:18:49.628Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:49.628Z [INFO] server-a4a5259c-dcb5-29a9-5212-37b471217af7: entering follower state: follower="Node at a4a5259c-dcb5-29a9-5212-37b471217af7 [Follower]" leader-address= leader-id= -2025-10-30T18:18:50.314Z [WARN] server-4dab8c44-22ef-72e7-6a5c-e907d894a332: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:50.314Z [INFO] server-4dab8c44-22ef-72e7-6a5c-e907d894a332: entering candidate state: node="Node at 4dab8c44-22ef-72e7-6a5c-e907d894a332 [Candidate]" term=2 -2025-10-30T18:18:50.314Z [INFO] server-4dab8c44-22ef-72e7-6a5c-e907d894a332: election won: term=2 tally=2 -2025-10-30T18:18:50.314Z [INFO] server-4dab8c44-22ef-72e7-6a5c-e907d894a332: entering leader state: leader="Node at 4dab8c44-22ef-72e7-6a5c-e907d894a332 [Leader]" -2025-10-30T18:18:50.314Z [INFO] server-4dab8c44-22ef-72e7-6a5c-e907d894a332: added peer, starting replication: peer=server-01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8 -2025-10-30T18:18:50.314Z [INFO] server-4dab8c44-22ef-72e7-6a5c-e907d894a332: added peer, starting replication: peer=server-a4a5259c-dcb5-29a9-5212-37b471217af7 -2025-10-30T18:18:50.314Z [INFO] server-4dab8c44-22ef-72e7-6a5c-e907d894a332: pipelining replication: peer="{Voter server-01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8 01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8}" -2025-10-30T18:18:50.314Z [INFO] server-4dab8c44-22ef-72e7-6a5c-e907d894a332: pipelining replication: peer="{Voter server-a4a5259c-dcb5-29a9-5212-37b471217af7 a4a5259c-dcb5-29a9-5212-37b471217af7}" -2025-10-30T18:18:51.319Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:50.314833088 +0000 UTC m=+76.593507002 (1 nodes), 686.67718ms from start of poll, 686.98231ms from cluster start. Timeout at 2025-10-30 18:18:51.319836153 +0000 UTC m=+77.598510053, 1.005003051s after stability -2025-10-30T18:18:51.321Z [INFO] server-4dab8c44-22ef-72e7-6a5c-e907d894a332: starting snapshot up to: index=12 -2025-10-30T18:18:51.321Z [INFO] creating new snapshot: path=/tmp/raft3852338432/snapshots/2-12-1761848331321.tmp -2025-10-30T18:18:51.322Z [INFO] server-4dab8c44-22ef-72e7-6a5c-e907d894a332: snapshot complete up to: index=12 -2025-10-30T18:18:51.322Z [INFO] server-092a4917-d92e-1e4c-5b89-2061806a9306: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-092a4917-d92e-1e4c-5b89-2061806a9306 Address:092a4917-d92e-1e4c-5b89-2061806a9306} {Suffrage:Voter ID:server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2 Address:6a5b0949-b9a8-6d44-7acf-b636fe4e04b2} {Suffrage:Voter ID:server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0 Address:874e1b19-2a1d-de7e-4f40-d3b88facc7e0}]" -2025-10-30T18:18:51.322Z [INFO] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-092a4917-d92e-1e4c-5b89-2061806a9306 Address:092a4917-d92e-1e4c-5b89-2061806a9306} {Suffrage:Voter ID:server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2 Address:6a5b0949-b9a8-6d44-7acf-b636fe4e04b2} {Suffrage:Voter ID:server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0 Address:874e1b19-2a1d-de7e-4f40-d3b88facc7e0}]" -2025-10-30T18:18:51.322Z [INFO] server-092a4917-d92e-1e4c-5b89-2061806a9306: entering follower state: follower="Node at 092a4917-d92e-1e4c-5b89-2061806a9306 [Follower]" leader-address= leader-id= -2025-10-30T18:18:51.322Z [INFO] server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-092a4917-d92e-1e4c-5b89-2061806a9306 Address:092a4917-d92e-1e4c-5b89-2061806a9306} {Suffrage:Voter ID:server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2 Address:6a5b0949-b9a8-6d44-7acf-b636fe4e04b2} {Suffrage:Voter ID:server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0 Address:874e1b19-2a1d-de7e-4f40-d3b88facc7e0}]" -2025-10-30T18:18:51.322Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:51.322Z [INFO] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: entering follower state: follower="Node at 6a5b0949-b9a8-6d44-7acf-b636fe4e04b2 [Follower]" leader-address= leader-id= -2025-10-30T18:18:51.322Z [INFO] server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0: entering follower state: follower="Node at 874e1b19-2a1d-de7e-4f40-d3b88facc7e0 [Follower]" leader-address= leader-id= -2025-10-30T18:18:52.023Z [WARN] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:52.023Z [INFO] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: entering candidate state: node="Node at 6a5b0949-b9a8-6d44-7acf-b636fe4e04b2 [Candidate]" term=2 -2025-10-30T18:18:52.023Z [INFO] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: election won: term=2 tally=2 -2025-10-30T18:18:52.023Z [INFO] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: entering leader state: leader="Node at 6a5b0949-b9a8-6d44-7acf-b636fe4e04b2 [Leader]" -2025-10-30T18:18:52.023Z [INFO] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: added peer, starting replication: peer=server-092a4917-d92e-1e4c-5b89-2061806a9306 -2025-10-30T18:18:52.023Z [INFO] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: added peer, starting replication: peer=server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0 -2025-10-30T18:18:52.023Z [INFO] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: pipelining replication: peer="{Voter server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0 874e1b19-2a1d-de7e-4f40-d3b88facc7e0}" -2025-10-30T18:18:52.023Z [INFO] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: pipelining replication: peer="{Voter server-092a4917-d92e-1e4c-5b89-2061806a9306 092a4917-d92e-1e4c-5b89-2061806a9306}" -2025-10-30T18:18:53.029Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:52.023839024 +0000 UTC m=+78.302512921 (1 nodes), 700.86965ms from start of poll, 701.071395ms from cluster start. Timeout at 2025-10-30 18:18:53.028848052 +0000 UTC m=+79.307521949, 1.005009028s after stability -2025-10-30T18:18:53.029Z [INFO] creating new snapshot: path=/tmp/raft1381586826/snapshots/2-1013-1761848333029.tmp -2025-10-30T18:18:53.029Z [INFO] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: copied to local snapshot: bytes=71 -2025-10-30T18:18:53.029Z [INFO] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: snapshot restore progress: id=2-1013-1761848333029 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:53.029Z [INFO] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: restored user snapshot: index=1013 -2025-10-30T18:18:53.029Z [ERROR] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: failed to get log: index=3 error="log not found" -2025-10-30T18:18:53.029Z [INFO] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: aborting pipeline replication: peer="{Voter server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0 874e1b19-2a1d-de7e-4f40-d3b88facc7e0}" -2025-10-30T18:18:53.029Z [ERROR] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: failed to get log: index=3 error="log not found" -2025-10-30T18:18:53.029Z [INFO] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: aborting pipeline replication: peer="{Voter server-092a4917-d92e-1e4c-5b89-2061806a9306 092a4917-d92e-1e4c-5b89-2061806a9306}" -2025-10-30T18:18:53.036Z [ERROR] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: failed to get log: index=3 error="log not found" -2025-10-30T18:18:53.036Z [ERROR] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: failed to get log: index=3 error="log not found" -2025-10-30T18:18:53.036Z [INFO] creating new snapshot: path=/tmp/raft3517113085/snapshots/2-1013-1761848333036.tmp -2025-10-30T18:18:53.036Z [INFO] creating new snapshot: path=/tmp/raft1919493842/snapshots/2-1013-1761848333036.tmp -2025-10-30T18:18:53.036Z [INFO] server-092a4917-d92e-1e4c-5b89-2061806a9306: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:53.036Z [INFO] server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:53.036Z [INFO] server-092a4917-d92e-1e4c-5b89-2061806a9306: copied to local snapshot: bytes=71 -2025-10-30T18:18:53.036Z [INFO] server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0: copied to local snapshot: bytes=71 -2025-10-30T18:18:53.037Z [INFO] server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0: snapshot restore progress: id=2-1013-1761848333036 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:53.037Z [INFO] server-092a4917-d92e-1e4c-5b89-2061806a9306: snapshot restore progress: id=2-1013-1761848333036 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:53.037Z [INFO] server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0: Installed remote snapshot -2025-10-30T18:18:53.037Z [INFO] server-092a4917-d92e-1e4c-5b89-2061806a9306: Installed remote snapshot -2025-10-30T18:18:53.037Z [INFO] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: pipelining replication: peer="{Voter server-874e1b19-2a1d-de7e-4f40-d3b88facc7e0 874e1b19-2a1d-de7e-4f40-d3b88facc7e0}" -2025-10-30T18:18:53.037Z [INFO] server-6a5b0949-b9a8-6d44-7acf-b636fe4e04b2: pipelining replication: peer="{Voter server-092a4917-d92e-1e4c-5b89-2061806a9306 092a4917-d92e-1e4c-5b89-2061806a9306}" -2025-10-30T18:18:53.037Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:18:54.042Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:18:53.037298963 +0000 UTC m=+79.315972872 (2 nodes), 1.126µs from start of poll, 1.714531346s from cluster start. Timeout at 2025-10-30 18:18:54.042317552 +0000 UTC m=+80.320991449, 1.005018577s after stability -2025-10-30T18:18:54.054Z [INFO] server-4dab8c44-22ef-72e7-6a5c-e907d894a332: aborting pipeline replication: peer="{Voter server-a4a5259c-dcb5-29a9-5212-37b471217af7 a4a5259c-dcb5-29a9-5212-37b471217af7}" -2025-10-30T18:18:54.054Z [INFO] server-4dab8c44-22ef-72e7-6a5c-e907d894a332: aborting pipeline replication: peer="{Voter server-01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8 01cbc5a5-27c2-73a8-13ab-6bbc69ab4cf8}" +2025-10-30T18:22:39.906Z [INFO] server-3ff698c9-3c7f-5d94-c987-93cc1f751172: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ff698c9-3c7f-5d94-c987-93cc1f751172 Address:3ff698c9-3c7f-5d94-c987-93cc1f751172} {Suffrage:Voter ID:server-de03f9d3-e6d6-c207-6454-a39dce5ece42 Address:de03f9d3-e6d6-c207-6454-a39dce5ece42} {Suffrage:Voter ID:server-8d5d39a6-b032-38e7-7466-6c5e7034dc91 Address:8d5d39a6-b032-38e7-7466-6c5e7034dc91}]" +2025-10-30T18:22:39.906Z [INFO] server-3ff698c9-3c7f-5d94-c987-93cc1f751172: entering follower state: follower="Node at 3ff698c9-3c7f-5d94-c987-93cc1f751172 [Follower]" leader-address= leader-id= +2025-10-30T18:22:39.906Z [INFO] server-de03f9d3-e6d6-c207-6454-a39dce5ece42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ff698c9-3c7f-5d94-c987-93cc1f751172 Address:3ff698c9-3c7f-5d94-c987-93cc1f751172} {Suffrage:Voter ID:server-de03f9d3-e6d6-c207-6454-a39dce5ece42 Address:de03f9d3-e6d6-c207-6454-a39dce5ece42} {Suffrage:Voter ID:server-8d5d39a6-b032-38e7-7466-6c5e7034dc91 Address:8d5d39a6-b032-38e7-7466-6c5e7034dc91}]" +2025-10-30T18:22:39.906Z [INFO] server-de03f9d3-e6d6-c207-6454-a39dce5ece42: entering follower state: follower="Node at de03f9d3-e6d6-c207-6454-a39dce5ece42 [Follower]" leader-address= leader-id= +2025-10-30T18:22:39.906Z [INFO] server-8d5d39a6-b032-38e7-7466-6c5e7034dc91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ff698c9-3c7f-5d94-c987-93cc1f751172 Address:3ff698c9-3c7f-5d94-c987-93cc1f751172} {Suffrage:Voter ID:server-de03f9d3-e6d6-c207-6454-a39dce5ece42 Address:de03f9d3-e6d6-c207-6454-a39dce5ece42} {Suffrage:Voter ID:server-8d5d39a6-b032-38e7-7466-6c5e7034dc91 Address:8d5d39a6-b032-38e7-7466-6c5e7034dc91}]" +2025-10-30T18:22:39.906Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:39.906Z [INFO] server-8d5d39a6-b032-38e7-7466-6c5e7034dc91: entering follower state: follower="Node at 8d5d39a6-b032-38e7-7466-6c5e7034dc91 [Follower]" leader-address= leader-id= +2025-10-30T18:22:40.473Z [WARN] server-de03f9d3-e6d6-c207-6454-a39dce5ece42: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:40.473Z [INFO] server-de03f9d3-e6d6-c207-6454-a39dce5ece42: entering candidate state: node="Node at de03f9d3-e6d6-c207-6454-a39dce5ece42 [Candidate]" term=2 +2025-10-30T18:22:40.473Z [INFO] server-de03f9d3-e6d6-c207-6454-a39dce5ece42: election won: term=2 tally=2 +2025-10-30T18:22:40.473Z [INFO] server-de03f9d3-e6d6-c207-6454-a39dce5ece42: entering leader state: leader="Node at de03f9d3-e6d6-c207-6454-a39dce5ece42 [Leader]" +2025-10-30T18:22:40.473Z [INFO] server-de03f9d3-e6d6-c207-6454-a39dce5ece42: added peer, starting replication: peer=server-3ff698c9-3c7f-5d94-c987-93cc1f751172 +2025-10-30T18:22:40.473Z [INFO] server-de03f9d3-e6d6-c207-6454-a39dce5ece42: added peer, starting replication: peer=server-8d5d39a6-b032-38e7-7466-6c5e7034dc91 +2025-10-30T18:22:40.473Z [INFO] server-de03f9d3-e6d6-c207-6454-a39dce5ece42: pipelining replication: peer="{Voter server-8d5d39a6-b032-38e7-7466-6c5e7034dc91 8d5d39a6-b032-38e7-7466-6c5e7034dc91}" +2025-10-30T18:22:40.473Z [INFO] server-de03f9d3-e6d6-c207-6454-a39dce5ece42: pipelining replication: peer="{Voter server-3ff698c9-3c7f-5d94-c987-93cc1f751172 3ff698c9-3c7f-5d94-c987-93cc1f751172}" +2025-10-30T18:22:41.479Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:40.473226168 +0000 UTC m=+83.546755469 (1 nodes), 566.786207ms from start of poll, 567.115041ms from cluster start. Timeout at 2025-10-30 18:22:41.478229516 +0000 UTC m=+84.551758809, 1.00500334s after stability +2025-10-30T18:22:41.479Z [INFO] server-de03f9d3-e6d6-c207-6454-a39dce5ece42: starting snapshot up to: index=12 +2025-10-30T18:22:41.479Z [INFO] creating new snapshot: path=/tmp/raft2870682463/snapshots/2-12-1761848561479.tmp +2025-10-30T18:22:41.550Z [INFO] server-de03f9d3-e6d6-c207-6454-a39dce5ece42: snapshot complete up to: index=12 +2025-10-30T18:22:41.551Z [INFO] server-b6ac980b-0a86-7a3e-d752-e95f45c45f05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6ac980b-0a86-7a3e-d752-e95f45c45f05 Address:b6ac980b-0a86-7a3e-d752-e95f45c45f05} {Suffrage:Voter ID:server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01 Address:0ac54fe3-b7d6-d944-9b1d-0585a2283f01} {Suffrage:Voter ID:server-84a28fee-22e2-6d5e-43d6-02566865291f Address:84a28fee-22e2-6d5e-43d6-02566865291f}]" +2025-10-30T18:22:41.551Z [INFO] server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6ac980b-0a86-7a3e-d752-e95f45c45f05 Address:b6ac980b-0a86-7a3e-d752-e95f45c45f05} {Suffrage:Voter ID:server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01 Address:0ac54fe3-b7d6-d944-9b1d-0585a2283f01} {Suffrage:Voter ID:server-84a28fee-22e2-6d5e-43d6-02566865291f Address:84a28fee-22e2-6d5e-43d6-02566865291f}]" +2025-10-30T18:22:41.551Z [INFO] server-b6ac980b-0a86-7a3e-d752-e95f45c45f05: entering follower state: follower="Node at b6ac980b-0a86-7a3e-d752-e95f45c45f05 [Follower]" leader-address= leader-id= +2025-10-30T18:22:41.551Z [INFO] server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01: entering follower state: follower="Node at 0ac54fe3-b7d6-d944-9b1d-0585a2283f01 [Follower]" leader-address= leader-id= +2025-10-30T18:22:41.551Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6ac980b-0a86-7a3e-d752-e95f45c45f05 Address:b6ac980b-0a86-7a3e-d752-e95f45c45f05} {Suffrage:Voter ID:server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01 Address:0ac54fe3-b7d6-d944-9b1d-0585a2283f01} {Suffrage:Voter ID:server-84a28fee-22e2-6d5e-43d6-02566865291f Address:84a28fee-22e2-6d5e-43d6-02566865291f}]" +2025-10-30T18:22:41.551Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:41.551Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: entering follower state: follower="Node at 84a28fee-22e2-6d5e-43d6-02566865291f [Follower]" leader-address= leader-id= +2025-10-30T18:22:42.415Z [WARN] server-84a28fee-22e2-6d5e-43d6-02566865291f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:42.415Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: entering candidate state: node="Node at 84a28fee-22e2-6d5e-43d6-02566865291f [Candidate]" term=2 +2025-10-30T18:22:42.415Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: election won: term=2 tally=2 +2025-10-30T18:22:42.415Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: entering leader state: leader="Node at 84a28fee-22e2-6d5e-43d6-02566865291f [Leader]" +2025-10-30T18:22:42.415Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: added peer, starting replication: peer=server-b6ac980b-0a86-7a3e-d752-e95f45c45f05 +2025-10-30T18:22:42.416Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: added peer, starting replication: peer=server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01 +2025-10-30T18:22:42.416Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: pipelining replication: peer="{Voter server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01 0ac54fe3-b7d6-d944-9b1d-0585a2283f01}" +2025-10-30T18:22:42.416Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: pipelining replication: peer="{Voter server-b6ac980b-0a86-7a3e-d752-e95f45c45f05 b6ac980b-0a86-7a3e-d752-e95f45c45f05}" +2025-10-30T18:22:43.421Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:42.41605333 +0000 UTC m=+85.489582624 (1 nodes), 864.713088ms from start of poll, 865.060299ms from cluster start. Timeout at 2025-10-30 18:22:43.421057212 +0000 UTC m=+86.494586501, 1.005003877s after stability +2025-10-30T18:22:43.479Z [INFO] creating new snapshot: path=/tmp/raft3999789336/snapshots/2-1013-1761848563479.tmp +2025-10-30T18:22:43.541Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: copied to local snapshot: bytes=71 +2025-10-30T18:22:43.541Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: snapshot restore progress: id=2-1013-1761848563479 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:43.541Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: restored user snapshot: index=1013 +2025-10-30T18:22:43.541Z [ERROR] server-84a28fee-22e2-6d5e-43d6-02566865291f: failed to get log: index=3 error="log not found" +2025-10-30T18:22:43.541Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: aborting pipeline replication: peer="{Voter server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01 0ac54fe3-b7d6-d944-9b1d-0585a2283f01}" +2025-10-30T18:22:43.541Z [ERROR] server-84a28fee-22e2-6d5e-43d6-02566865291f: failed to get log: index=3 error="log not found" +2025-10-30T18:22:43.541Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: aborting pipeline replication: peer="{Voter server-b6ac980b-0a86-7a3e-d752-e95f45c45f05 b6ac980b-0a86-7a3e-d752-e95f45c45f05}" +2025-10-30T18:22:43.549Z [ERROR] server-84a28fee-22e2-6d5e-43d6-02566865291f: failed to get log: index=3 error="log not found" +2025-10-30T18:22:43.549Z [ERROR] server-84a28fee-22e2-6d5e-43d6-02566865291f: failed to get log: index=3 error="log not found" +2025-10-30T18:22:43.549Z [INFO] creating new snapshot: path=/tmp/raft1243310796/snapshots/2-1013-1761848563549.tmp +2025-10-30T18:22:43.549Z [INFO] creating new snapshot: path=/tmp/raft2743192622/snapshots/2-1013-1761848563549.tmp +2025-10-30T18:22:43.550Z [INFO] server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:43.550Z [INFO] server-b6ac980b-0a86-7a3e-d752-e95f45c45f05: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:43.550Z [INFO] server-b6ac980b-0a86-7a3e-d752-e95f45c45f05: copied to local snapshot: bytes=71 +2025-10-30T18:22:43.550Z [INFO] server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01: copied to local snapshot: bytes=71 +2025-10-30T18:22:43.550Z [INFO] server-b6ac980b-0a86-7a3e-d752-e95f45c45f05: snapshot restore progress: id=2-1013-1761848563549 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:43.550Z [INFO] server-b6ac980b-0a86-7a3e-d752-e95f45c45f05: Installed remote snapshot +2025-10-30T18:22:43.550Z [INFO] server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01: snapshot restore progress: id=2-1013-1761848563549 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:43.550Z [INFO] server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01: Installed remote snapshot +2025-10-30T18:22:43.550Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: pipelining replication: peer="{Voter server-b6ac980b-0a86-7a3e-d752-e95f45c45f05 b6ac980b-0a86-7a3e-d752-e95f45c45f05}" +2025-10-30T18:22:43.550Z [WARN] server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01: skipping application of old log: index=1013 +2025-10-30T18:22:43.550Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:22:43.550Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: pipelining replication: peer="{Voter server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01 0ac54fe3-b7d6-d944-9b1d-0585a2283f01}" +2025-10-30T18:22:45.278Z [ERROR] server-e2668a63-27b7-abe8-f831-048f62464050: failed to heartbeat to: peer=691e8f75-eabd-0958-0932-0bda3853fff9 backoff time=10ms error="command timed out" +2025-10-30T18:22:45.278Z [WARN] server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01: heartbeat timeout reached, starting election: last-leader-addr=84a28fee-22e2-6d5e-43d6-02566865291f last-leader-id=server-84a28fee-22e2-6d5e-43d6-02566865291f +2025-10-30T18:22:45.278Z [INFO] server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01: entering candidate state: node="Node at 0ac54fe3-b7d6-d944-9b1d-0585a2283f01 [Candidate]" term=3 +2025-10-30T18:22:45.278Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:22:43.550650109 +0000 UTC m=+86.624179397 (2 nodes), 578ns from start of poll, 1.999657072s from cluster start. Timeout at 2025-10-30 18:22:44.555687274 +0000 UTC m=+87.629216575, 1.005037178s after stability +2025-10-30T18:22:45.279Z [WARN] server-21909053-afae-878e-2be8-0085c1c63f8a: failed to contact: server-id=server-a26ff83d-e22a-136c-9b62-5bb846157d8e time=1.494225513s +2025-10-30T18:22:45.279Z [WARN] server-84a28fee-22e2-6d5e-43d6-02566865291f: failed to contact: server-id=server-b6ac980b-0a86-7a3e-d752-e95f45c45f05 time=1.496609998s +2025-10-30T18:22:45.279Z [WARN] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: failed to contact: server-id=server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a time=1.491813525s +2025-10-30T18:22:45.279Z [WARN] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: failed to contact: server-id=server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f time=1.492234469s +2025-10-30T18:22:45.279Z [WARN] server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb: heartbeat timeout reached, starting election: last-leader-addr=94cb357e-bf8e-3d97-650e-e6c1381e15e9 last-leader-id=server-94cb357e-bf8e-3d97-650e-e6c1381e15e9 +2025-10-30T18:22:45.279Z [WARN] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: failed to contact: server-id=server-efc229cc-054f-2b94-f3b3-9bec31752f46 time=1.494491981s +2025-10-30T18:22:45.279Z [WARN] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: failed to contact: server-id=server-72bdba35-233c-a205-b189-c6faa07c7d5b time=1.494621483s +2025-10-30T18:22:45.279Z [WARN] server-fba31bce-63b2-1d27-8898-8934ad09d77b: heartbeat timeout reached, starting election: last-leader-addr=1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db last-leader-id=server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db +2025-10-30T18:22:45.279Z [INFO] server-fba31bce-63b2-1d27-8898-8934ad09d77b: entering candidate state: node="Node at fba31bce-63b2-1d27-8898-8934ad09d77b [Candidate]" term=3 +2025-10-30T18:22:45.279Z [WARN] server-b6ac980b-0a86-7a3e-d752-e95f45c45f05: heartbeat timeout reached, starting election: last-leader-addr=84a28fee-22e2-6d5e-43d6-02566865291f last-leader-id=server-84a28fee-22e2-6d5e-43d6-02566865291f +2025-10-30T18:22:45.279Z [WARN] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: failed to contact: server-id=server-747d9665-bec4-273d-3713-17b0ce5a7a84 time=1.497181493s +2025-10-30T18:22:45.279Z [INFO] server-b6ac980b-0a86-7a3e-d752-e95f45c45f05: entering candidate state: node="Node at b6ac980b-0a86-7a3e-d752-e95f45c45f05 [Candidate]" term=3 +2025-10-30T18:22:45.279Z [WARN] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: failed to contact quorum of nodes, stepping down +2025-10-30T18:22:45.279Z [WARN] server-de03f9d3-e6d6-c207-6454-a39dce5ece42: failed to contact: server-id=server-3ff698c9-3c7f-5d94-c987-93cc1f751172 time=1.492284563s +2025-10-30T18:22:45.279Z [WARN] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: failed to contact: server-id=server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9 time=1.491954495s +2025-10-30T18:22:45.279Z [ERROR] server-84a28fee-22e2-6d5e-43d6-02566865291f: failed to heartbeat to: peer=0ac54fe3-b7d6-d944-9b1d-0585a2283f01 backoff time=10ms error="command timed out" +2025-10-30T18:22:45.279Z [WARN] server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f: heartbeat timeout reached, starting election: last-leader-addr=f583ef02-c74d-37b9-ed32-79a7e7e24ced last-leader-id=server-f583ef02-c74d-37b9-ed32-79a7e7e24ced +2025-10-30T18:22:45.279Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: aborting pipeline replication: peer="{Voter server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01 0ac54fe3-b7d6-d944-9b1d-0585a2283f01}" +2025-10-30T18:22:45.279Z [WARN] server-733c507b-20a5-105e-d60d-6fa40e90cfd1: heartbeat timeout reached, starting election: last-leader-addr=e2668a63-27b7-abe8-f831-048f62464050 last-leader-id=server-e2668a63-27b7-abe8-f831-048f62464050 +2025-10-30T18:22:45.279Z [INFO] server-733c507b-20a5-105e-d60d-6fa40e90cfd1: entering candidate state: node="Node at 733c507b-20a5-105e-d60d-6fa40e90cfd1 [Candidate]" term=3 +2025-10-30T18:22:45.279Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: aborting pipeline replication: peer="{Voter server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a 3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a}" +2025-10-30T18:22:45.279Z [WARN] server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9: heartbeat timeout reached, starting election: last-leader-addr=1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d last-leader-id=server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d +2025-10-30T18:22:45.279Z [INFO] server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9: entering candidate state: node="Node at 26cce6da-6cd7-ea19-6f50-7d526f08c4c9 [Candidate]" term=3 +2025-10-30T18:22:45.279Z [WARN] server-691e8f75-eabd-0958-0932-0bda3853fff9: heartbeat timeout reached, starting election: last-leader-addr=e2668a63-27b7-abe8-f831-048f62464050 last-leader-id=server-e2668a63-27b7-abe8-f831-048f62464050 +2025-10-30T18:22:45.279Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: entering follower state: follower="Node at f583ef02-c74d-37b9-ed32-79a7e7e24ced [Follower]" leader-address= leader-id= +2025-10-30T18:22:45.279Z [WARN] server-84a28fee-22e2-6d5e-43d6-02566865291f: rejecting vote request since we have a leader: from=0ac54fe3-b7d6-d944-9b1d-0585a2283f01 leader=84a28fee-22e2-6d5e-43d6-02566865291f leader-id=server-84a28fee-22e2-6d5e-43d6-02566865291f +2025-10-30T18:22:45.279Z [WARN] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: failed to contact: server-id=server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb time=1.497129628s +2025-10-30T18:22:45.279Z [INFO] server-b6ac980b-0a86-7a3e-d752-e95f45c45f05: duplicate requestVote for same term: term=3 +2025-10-30T18:22:45.279Z [INFO] server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb: entering candidate state: node="Node at 7ee4c18a-a936-15e4-7ff6-4d04a9031adb [Candidate]" term=3 +2025-10-30T18:22:45.279Z [INFO] server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f: entering candidate state: node="Node at c7faca8d-37f3-a29f-98a5-7cd3a729c05f [Candidate]" term=3 +2025-10-30T18:22:45.279Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: aborting pipeline replication: peer="{Voter server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f c7faca8d-37f3-a29f-98a5-7cd3a729c05f}" +2025-10-30T18:22:45.279Z [WARN] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: failed to contact: server-id=server-788cf89c-090e-f232-af4b-94504abbc2a1 time=1.494848055s +2025-10-30T18:22:45.279Z [INFO] server-f583ef02-c74d-37b9-ed32-79a7e7e24ced: aborting pipeline replication: peer="{Voter server-747d9665-bec4-273d-3713-17b0ce5a7a84 747d9665-bec4-273d-3713-17b0ce5a7a84}" +2025-10-30T18:22:45.279Z [ERROR] server-84a28fee-22e2-6d5e-43d6-02566865291f: peer has newer term, stopping replication: peer="{Voter server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01 0ac54fe3-b7d6-d944-9b1d-0585a2283f01}" +2025-10-30T18:22:45.279Z [WARN] server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18: heartbeat timeout reached, starting election: last-leader-addr=d65b504a-8185-4303-efb5-28ea2cf86fac last-leader-id=server-d65b504a-8185-4303-efb5-28ea2cf86fac +2025-10-30T18:22:45.279Z [INFO] server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01: duplicate requestVote for same term: term=3 +2025-10-30T18:22:45.279Z [WARN] server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a: heartbeat timeout reached, starting election: last-leader-addr=d35ebf55-0d21-fd1e-eb65-c2c2fba58466 last-leader-id=server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466 +2025-10-30T18:22:45.279Z [WARN] server-d65b504a-8185-4303-efb5-28ea2cf86fac: failed to contact: server-id=server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18 time=1.497149972s +2025-10-30T18:22:45.279Z [INFO] server-de03f9d3-e6d6-c207-6454-a39dce5ece42: aborting pipeline replication: peer="{Voter server-3ff698c9-3c7f-5d94-c987-93cc1f751172 3ff698c9-3c7f-5d94-c987-93cc1f751172}" +2025-10-30T18:22:45.279Z [WARN] server-747d9665-bec4-273d-3713-17b0ce5a7a84: rejecting vote request since we have a leader: from=c7faca8d-37f3-a29f-98a5-7cd3a729c05f leader=f583ef02-c74d-37b9-ed32-79a7e7e24ced leader-id=server-f583ef02-c74d-37b9-ed32-79a7e7e24ced +2025-10-30T18:22:45.279Z [INFO] server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a: entering candidate state: node="Node at 3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a [Candidate]" term=3 +2025-10-30T18:22:45.279Z [WARN] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: failed to contact quorum of nodes, stepping down +2025-10-30T18:22:45.279Z [INFO] server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f: election won: term=3 tally=2 +2025-10-30T18:22:45.279Z [WARN] server-72bdba35-233c-a205-b189-c6faa07c7d5b: rejecting vote request since we have a leader: from=7ee4c18a-a936-15e4-7ff6-4d04a9031adb leader=94cb357e-bf8e-3d97-650e-e6c1381e15e9 leader-id=server-94cb357e-bf8e-3d97-650e-e6c1381e15e9 +2025-10-30T18:22:45.279Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: entering follower state: follower="Node at 94cb357e-bf8e-3d97-650e-e6c1381e15e9 [Follower]" leader-address= leader-id= +2025-10-30T18:22:45.279Z [WARN] server-788cf89c-090e-f232-af4b-94504abbc2a1: heartbeat timeout reached, starting election: last-leader-addr=1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db last-leader-id=server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db +2025-10-30T18:22:45.280Z [INFO] server-788cf89c-090e-f232-af4b-94504abbc2a1: entering candidate state: node="Node at 788cf89c-090e-f232-af4b-94504abbc2a1 [Candidate]" term=3 +2025-10-30T18:22:45.280Z [WARN] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: rejecting vote request since we have a leader: from=3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a leader=d35ebf55-0d21-fd1e-eb65-c2c2fba58466 leader-id=server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466 +2025-10-30T18:22:45.279Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: aborting pipeline replication: peer="{Voter server-b6ac980b-0a86-7a3e-d752-e95f45c45f05 b6ac980b-0a86-7a3e-d752-e95f45c45f05}" +2025-10-30T18:22:45.279Z [WARN] server-efc229cc-054f-2b94-f3b3-9bec31752f46: heartbeat timeout reached, starting election: last-leader-addr=1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d last-leader-id=server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d +2025-10-30T18:22:45.280Z [INFO] server-efc229cc-054f-2b94-f3b3-9bec31752f46: entering candidate state: node="Node at efc229cc-054f-2b94-f3b3-9bec31752f46 [Candidate]" term=3 +2025-10-30T18:22:45.279Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: entering follower state: follower="Node at 84a28fee-22e2-6d5e-43d6-02566865291f [Follower]" leader-address= leader-id= +2025-10-30T18:22:45.279Z [WARN] server-e2668a63-27b7-abe8-f831-048f62464050: failed to contact: server-id=server-733c507b-20a5-105e-d60d-6fa40e90cfd1 time=1.493488753s +2025-10-30T18:22:45.279Z [WARN] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: rejecting vote request since we have a leader: from=fba31bce-63b2-1d27-8898-8934ad09d77b leader=1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db leader-id=server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db +2025-10-30T18:22:45.280Z [INFO] server-fba31bce-63b2-1d27-8898-8934ad09d77b: duplicate requestVote for same term: term=3 +2025-10-30T18:22:45.279Z [INFO] server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f: entering leader state: leader="Node at c7faca8d-37f3-a29f-98a5-7cd3a729c05f [Leader]" +2025-10-30T18:22:45.280Z [INFO] server-efc229cc-054f-2b94-f3b3-9bec31752f46: duplicate requestVote for same term: term=3 +2025-10-30T18:22:45.279Z [INFO] server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb: election won: term=3 tally=2 +2025-10-30T18:22:45.280Z [WARN] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: rejecting vote request since we have a leader: from=788cf89c-090e-f232-af4b-94504abbc2a1 leader=1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db leader-id=server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db +2025-10-30T18:22:45.280Z [INFO] server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f: added peer, starting replication: peer=server-747d9665-bec4-273d-3713-17b0ce5a7a84 +2025-10-30T18:22:45.280Z [INFO] server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb: entering leader state: leader="Node at 7ee4c18a-a936-15e4-7ff6-4d04a9031adb [Leader]" +2025-10-30T18:22:45.279Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: aborting pipeline replication: peer="{Voter server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb 7ee4c18a-a936-15e4-7ff6-4d04a9031adb}" +2025-10-30T18:22:45.280Z [INFO] server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f: added peer, starting replication: peer=server-f583ef02-c74d-37b9-ed32-79a7e7e24ced +2025-10-30T18:22:45.279Z [WARN] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: failed to contact quorum of nodes, stepping down +2025-10-30T18:22:45.280Z [INFO] server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb: added peer, starting replication: peer=server-72bdba35-233c-a205-b189-c6faa07c7d5b +2025-10-30T18:22:45.279Z [INFO] server-691e8f75-eabd-0958-0932-0bda3853fff9: entering candidate state: node="Node at 691e8f75-eabd-0958-0932-0bda3853fff9 [Candidate]" term=3 + raft_test.go:1595: Error Apply new log entries: node is not the leader +2025-10-30T18:22:45.279Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: aborting pipeline replication: peer="{Voter server-fba31bce-63b2-1d27-8898-8934ad09d77b fba31bce-63b2-1d27-8898-8934ad09d77b}" +2025-10-30T18:22:45.280Z [INFO] server-788cf89c-090e-f232-af4b-94504abbc2a1: duplicate requestVote for same term: term=3 +2025-10-30T18:22:45.280Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: entering follower state: follower="Node at 1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d [Follower]" leader-address= leader-id= +2025-10-30T18:22:45.280Z [WARN] server-84a28fee-22e2-6d5e-43d6-02566865291f: rejecting vote request since our last index is greater: candidate=b6ac980b-0a86-7a3e-d752-e95f45c45f05 last-index=1023 last-candidate-index=1014 +2025-10-30T18:22:45.280Z [INFO] server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9: duplicate requestVote for same term: term=3 +2025-10-30T18:22:45.280Z [ERROR] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: peer has newer term, stopping replication: peer="{Voter server-788cf89c-090e-f232-af4b-94504abbc2a1 788cf89c-090e-f232-af4b-94504abbc2a1}" +2025-10-30T18:22:45.279Z [ERROR] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: peer has newer term, stopping replication: peer="{Voter server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9 26cce6da-6cd7-ea19-6f50-7d526f08c4c9}" +2025-10-30T18:22:45.280Z [INFO] server-691e8f75-eabd-0958-0932-0bda3853fff9: duplicate requestVote for same term: term=3 +2025-10-30T18:22:45.280Z [INFO] server-de03f9d3-e6d6-c207-6454-a39dce5ece42: aborting pipeline replication: peer="{Voter server-8d5d39a6-b032-38e7-7466-6c5e7034dc91 8d5d39a6-b032-38e7-7466-6c5e7034dc91}" +2025-10-30T18:22:45.280Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: aborting pipeline replication: peer="{Voter server-788cf89c-090e-f232-af4b-94504abbc2a1 788cf89c-090e-f232-af4b-94504abbc2a1}" +2025-10-30T18:22:45.280Z [INFO] server-733c507b-20a5-105e-d60d-6fa40e90cfd1: duplicate requestVote for same term: term=3 +2025-10-30T18:22:45.280Z [WARN] server-d90a6fc0-692f-02d2-8800-1bed4355b62e: rejecting vote request since we have a leader: from=3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a leader=d35ebf55-0d21-fd1e-eb65-c2c2fba58466 leader-id=server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466 +2025-10-30T18:22:45.279Z [WARN] server-a26ff83d-e22a-136c-9b62-5bb846157d8e: heartbeat timeout reached, starting election: last-leader-addr=21909053-afae-878e-2be8-0085c1c63f8a last-leader-id=server-21909053-afae-878e-2be8-0085c1c63f8a +2025-10-30T18:22:45.279Z [ERROR] server-e2668a63-27b7-abe8-f831-048f62464050: peer has newer term, stopping replication: peer="{Voter server-733c507b-20a5-105e-d60d-6fa40e90cfd1 733c507b-20a5-105e-d60d-6fa40e90cfd1}" +2025-10-30T18:22:45.280Z [INFO] server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9: election won: term=3 tally=2 +2025-10-30T18:22:45.279Z [INFO] server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18: entering candidate state: node="Node at 4c9a84fb-0b48-edaf-fee7-5811a1c40d18 [Candidate]" term=3 +2025-10-30T18:22:45.280Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: aborting pipeline replication: peer="{Voter server-733c507b-20a5-105e-d60d-6fa40e90cfd1 733c507b-20a5-105e-d60d-6fa40e90cfd1}" +2025-10-30T18:22:45.280Z [INFO] server-94cb357e-bf8e-3d97-650e-e6c1381e15e9: aborting pipeline replication: peer="{Voter server-72bdba35-233c-a205-b189-c6faa07c7d5b 72bdba35-233c-a205-b189-c6faa07c7d5b}" +2025-10-30T18:22:45.280Z [WARN] server-e2668a63-27b7-abe8-f831-048f62464050: rejecting vote request since we have a leader: from=733c507b-20a5-105e-d60d-6fa40e90cfd1 leader=e2668a63-27b7-abe8-f831-048f62464050 leader-id=server-e2668a63-27b7-abe8-f831-048f62464050 +2025-10-30T18:22:45.280Z [INFO] server-a26ff83d-e22a-136c-9b62-5bb846157d8e: entering candidate state: node="Node at a26ff83d-e22a-136c-9b62-5bb846157d8e [Candidate]" term=3 +2025-10-30T18:22:45.280Z [INFO] server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9: entering leader state: leader="Node at 26cce6da-6cd7-ea19-6f50-7d526f08c4c9 [Leader]" +2025-10-30T18:22:45.280Z [WARN] server-de776368-6add-9f70-7549-04e98aff6559: rejecting vote request since we have a leader: from=4c9a84fb-0b48-edaf-fee7-5811a1c40d18 leader=d65b504a-8185-4303-efb5-28ea2cf86fac leader-id=server-d65b504a-8185-4303-efb5-28ea2cf86fac +2025-10-30T18:22:45.280Z [INFO] server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f: pipelining replication: peer="{Voter server-f583ef02-c74d-37b9-ed32-79a7e7e24ced f583ef02-c74d-37b9-ed32-79a7e7e24ced}" +2025-10-30T18:22:45.280Z [INFO] server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9: added peer, starting replication: peer=server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d +2025-10-30T18:22:45.280Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: duplicate requestVote for same term: term=3 +2025-10-30T18:22:45.280Z [INFO] server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db: entering follower state: follower="Node at 1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db [Follower]" leader-address= leader-id= +2025-10-30T18:22:45.280Z [WARN] server-d65b504a-8185-4303-efb5-28ea2cf86fac: rejecting vote request since we have a leader: from=4c9a84fb-0b48-edaf-fee7-5811a1c40d18 leader=d65b504a-8185-4303-efb5-28ea2cf86fac leader-id=server-d65b504a-8185-4303-efb5-28ea2cf86fac +2025-10-30T18:22:45.280Z [ERROR] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: peer has newer term, stopping replication: peer="{Voter server-efc229cc-054f-2b94-f3b3-9bec31752f46 efc229cc-054f-2b94-f3b3-9bec31752f46}" +2025-10-30T18:22:45.280Z [ERROR] server-d65b504a-8185-4303-efb5-28ea2cf86fac: peer has newer term, stopping replication: peer="{Voter server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18 4c9a84fb-0b48-edaf-fee7-5811a1c40d18}" +2025-10-30T18:22:45.280Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: aborting pipeline replication: peer="{Voter server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18 4c9a84fb-0b48-edaf-fee7-5811a1c40d18}" +2025-10-30T18:22:45.280Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: aborting pipeline replication: peer="{Voter server-efc229cc-054f-2b94-f3b3-9bec31752f46 efc229cc-054f-2b94-f3b3-9bec31752f46}" +2025-10-30T18:22:45.280Z [INFO] server-c7faca8d-37f3-a29f-98a5-7cd3a729c05f: pipelining replication: peer="{Voter server-747d9665-bec4-273d-3713-17b0ce5a7a84 747d9665-bec4-273d-3713-17b0ce5a7a84}" +2025-10-30T18:22:45.280Z [INFO] server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d: aborting pipeline replication: peer="{Voter server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9 26cce6da-6cd7-ea19-6f50-7d526f08c4c9}" +2025-10-30T18:22:45.280Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: entering follower state: follower="Node at e2668a63-27b7-abe8-f831-048f62464050 [Follower]" leader-address= leader-id= +2025-10-30T18:22:45.280Z [INFO] server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9: added peer, starting replication: peer=server-efc229cc-054f-2b94-f3b3-9bec31752f46 +2025-10-30T18:22:45.280Z [INFO] server-691e8f75-eabd-0958-0932-0bda3853fff9: election won: term=3 tally=2 +2025-10-30T18:22:45.280Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: entering follower state: follower="Node at d65b504a-8185-4303-efb5-28ea2cf86fac [Follower]" leader-address= leader-id= +2025-10-30T18:22:45.280Z [ERROR] server-e2668a63-27b7-abe8-f831-048f62464050: peer has newer term, stopping replication: peer="{Voter server-691e8f75-eabd-0958-0932-0bda3853fff9 691e8f75-eabd-0958-0932-0bda3853fff9}" +2025-10-30T18:22:45.280Z [INFO] server-e2668a63-27b7-abe8-f831-048f62464050: aborting pipeline replication: peer="{Voter server-691e8f75-eabd-0958-0932-0bda3853fff9 691e8f75-eabd-0958-0932-0bda3853fff9}" +2025-10-30T18:22:45.280Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: aborting pipeline replication: peer="{Voter server-de776368-6add-9f70-7549-04e98aff6559 de776368-6add-9f70-7549-04e98aff6559}" +2025-10-30T18:22:45.280Z [INFO] server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb: added peer, starting replication: peer=server-94cb357e-bf8e-3d97-650e-e6c1381e15e9 +2025-10-30T18:22:45.280Z [INFO] server-691e8f75-eabd-0958-0932-0bda3853fff9: entering leader state: leader="Node at 691e8f75-eabd-0958-0932-0bda3853fff9 [Leader]" +2025-10-30T18:22:45.280Z [INFO] server-efc229cc-054f-2b94-f3b3-9bec31752f46: entering follower state: follower="Node at efc229cc-054f-2b94-f3b3-9bec31752f46 [Follower]" leader-address=26cce6da-6cd7-ea19-6f50-7d526f08c4c9 leader-id=server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9 +2025-10-30T18:22:45.280Z [WARN] server-4f165838-c7a2-0648-7438-0652345f61e1: rejecting vote request since we have a leader: from=a26ff83d-e22a-136c-9b62-5bb846157d8e leader=21909053-afae-878e-2be8-0085c1c63f8a leader-id=server-21909053-afae-878e-2be8-0085c1c63f8a +2025-10-30T18:22:45.280Z [INFO] server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9: pipelining replication: peer="{Voter server-1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d 1aacbebd-dcb0-3bc9-d6c8-fe0a5ea0915d}" +2025-10-30T18:22:45.280Z [INFO] server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb: pipelining replication: peer="{Voter server-72bdba35-233c-a205-b189-c6faa07c7d5b 72bdba35-233c-a205-b189-c6faa07c7d5b}" +2025-10-30T18:22:45.280Z [INFO] server-691e8f75-eabd-0958-0932-0bda3853fff9: added peer, starting replication: peer=server-733c507b-20a5-105e-d60d-6fa40e90cfd1 +2025-10-30T18:22:45.280Z [INFO] server-26cce6da-6cd7-ea19-6f50-7d526f08c4c9: pipelining replication: peer="{Voter server-efc229cc-054f-2b94-f3b3-9bec31752f46 efc229cc-054f-2b94-f3b3-9bec31752f46}" +2025-10-30T18:22:45.280Z [INFO] server-691e8f75-eabd-0958-0932-0bda3853fff9: added peer, starting replication: peer=server-e2668a63-27b7-abe8-f831-048f62464050 +2025-10-30T18:22:45.280Z [WARN] server-21909053-afae-878e-2be8-0085c1c63f8a: rejecting vote request since we have a leader: from=a26ff83d-e22a-136c-9b62-5bb846157d8e leader=21909053-afae-878e-2be8-0085c1c63f8a leader-id=server-21909053-afae-878e-2be8-0085c1c63f8a +2025-10-30T18:22:45.280Z [INFO] server-7ee4c18a-a936-15e4-7ff6-4d04a9031adb: pipelining replication: peer="{Voter server-94cb357e-bf8e-3d97-650e-e6c1381e15e9 94cb357e-bf8e-3d97-650e-e6c1381e15e9}" +2025-10-30T18:22:45.280Z [ERROR] server-21909053-afae-878e-2be8-0085c1c63f8a: peer has newer term, stopping replication: peer="{Voter server-a26ff83d-e22a-136c-9b62-5bb846157d8e a26ff83d-e22a-136c-9b62-5bb846157d8e}" +2025-10-30T18:22:45.280Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: aborting pipeline replication: peer="{Voter server-a26ff83d-e22a-136c-9b62-5bb846157d8e a26ff83d-e22a-136c-9b62-5bb846157d8e}" +2025-10-30T18:22:45.280Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: entering follower state: follower="Node at 21909053-afae-878e-2be8-0085c1c63f8a [Follower]" leader-address= leader-id= +2025-10-30T18:22:45.280Z [INFO] server-21909053-afae-878e-2be8-0085c1c63f8a: aborting pipeline replication: peer="{Voter server-4f165838-c7a2-0648-7438-0652345f61e1 4f165838-c7a2-0648-7438-0652345f61e1}" +2025-10-30T18:22:45.280Z [INFO] server-733c507b-20a5-105e-d60d-6fa40e90cfd1: entering follower state: follower="Node at 733c507b-20a5-105e-d60d-6fa40e90cfd1 [Follower]" leader-address=691e8f75-eabd-0958-0932-0bda3853fff9 leader-id=server-691e8f75-eabd-0958-0932-0bda3853fff9 +2025-10-30T18:22:45.280Z [INFO] server-691e8f75-eabd-0958-0932-0bda3853fff9: pipelining replication: peer="{Voter server-e2668a63-27b7-abe8-f831-048f62464050 e2668a63-27b7-abe8-f831-048f62464050}" +2025-10-30T18:22:45.280Z [INFO] server-691e8f75-eabd-0958-0932-0bda3853fff9: pipelining replication: peer="{Voter server-733c507b-20a5-105e-d60d-6fa40e90cfd1 733c507b-20a5-105e-d60d-6fa40e90cfd1}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true -2025-10-30T18:18:54.057Z [INFO] server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fee6882-dfe8-3952-a66f-c947a53ddfe5 Address:6fee6882-dfe8-3952-a66f-c947a53ddfe5} {Suffrage:Voter ID:server-b4aa64b9-6c20-40f0-17ef-35a28b344ffc Address:b4aa64b9-6c20-40f0-17ef-35a28b344ffc} {Suffrage:Voter ID:server-4e16e616-bafa-1c9f-6408-e796c971cc8b Address:4e16e616-bafa-1c9f-6408-e796c971cc8b}]" -2025-10-30T18:18:54.057Z [INFO] server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: entering follower state: follower="Node at 6fee6882-dfe8-3952-a66f-c947a53ddfe5 [Follower]" leader-address= leader-id= -2025-10-30T18:18:54.057Z [INFO] server-b4aa64b9-6c20-40f0-17ef-35a28b344ffc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fee6882-dfe8-3952-a66f-c947a53ddfe5 Address:6fee6882-dfe8-3952-a66f-c947a53ddfe5} {Suffrage:Voter ID:server-b4aa64b9-6c20-40f0-17ef-35a28b344ffc Address:b4aa64b9-6c20-40f0-17ef-35a28b344ffc} {Suffrage:Voter ID:server-4e16e616-bafa-1c9f-6408-e796c971cc8b Address:4e16e616-bafa-1c9f-6408-e796c971cc8b}]" -2025-10-30T18:18:54.057Z [INFO] server-b4aa64b9-6c20-40f0-17ef-35a28b344ffc: entering follower state: follower="Node at b4aa64b9-6c20-40f0-17ef-35a28b344ffc [Follower]" leader-address= leader-id= -2025-10-30T18:18:54.057Z [INFO] server-4e16e616-bafa-1c9f-6408-e796c971cc8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fee6882-dfe8-3952-a66f-c947a53ddfe5 Address:6fee6882-dfe8-3952-a66f-c947a53ddfe5} {Suffrage:Voter ID:server-b4aa64b9-6c20-40f0-17ef-35a28b344ffc Address:b4aa64b9-6c20-40f0-17ef-35a28b344ffc} {Suffrage:Voter ID:server-4e16e616-bafa-1c9f-6408-e796c971cc8b Address:4e16e616-bafa-1c9f-6408-e796c971cc8b}]" -2025-10-30T18:18:54.057Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:54.057Z [INFO] server-4e16e616-bafa-1c9f-6408-e796c971cc8b: entering follower state: follower="Node at 4e16e616-bafa-1c9f-6408-e796c971cc8b [Follower]" leader-address= leader-id= -2025-10-30T18:18:54.680Z [WARN] server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:54.680Z [INFO] server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: entering candidate state: node="Node at 6fee6882-dfe8-3952-a66f-c947a53ddfe5 [Candidate]" term=2 -2025-10-30T18:18:54.681Z [INFO] server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: election won: term=2 tally=2 -2025-10-30T18:18:54.681Z [INFO] server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: entering leader state: leader="Node at 6fee6882-dfe8-3952-a66f-c947a53ddfe5 [Leader]" -2025-10-30T18:18:54.681Z [INFO] server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: added peer, starting replication: peer=server-b4aa64b9-6c20-40f0-17ef-35a28b344ffc -2025-10-30T18:18:54.681Z [INFO] server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: added peer, starting replication: peer=server-4e16e616-bafa-1c9f-6408-e796c971cc8b -2025-10-30T18:18:54.681Z [INFO] server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: pipelining replication: peer="{Voter server-4e16e616-bafa-1c9f-6408-e796c971cc8b 4e16e616-bafa-1c9f-6408-e796c971cc8b}" -2025-10-30T18:18:54.681Z [INFO] server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: pipelining replication: peer="{Voter server-b4aa64b9-6c20-40f0-17ef-35a28b344ffc b4aa64b9-6c20-40f0-17ef-35a28b344ffc}" -2025-10-30T18:18:55.686Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:54.681172406 +0000 UTC m=+80.959846304 (1 nodes), 623.371633ms from start of poll, 623.701198ms from cluster start. Timeout at 2025-10-30 18:18:55.686178178 +0000 UTC m=+81.964852089, 1.005005785s after stability -2025-10-30T18:18:55.687Z [INFO] server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: starting snapshot up to: index=12 -2025-10-30T18:18:55.687Z [INFO] creating new snapshot: path=/tmp/raft3309214673/snapshots/2-12-1761848335687.tmp -2025-10-30T18:18:55.687Z [INFO] server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: snapshot complete up to: index=12 -2025-10-30T18:18:55.688Z [INFO] server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734 Address:2cf29b51-4d7d-232a-cc8f-b7e4b5402734} {Suffrage:Voter ID:server-3759ae26-f796-c659-a13d-88d1d2088f31 Address:3759ae26-f796-c659-a13d-88d1d2088f31} {Suffrage:Voter ID:server-b24e3d03-bd41-bd39-f50e-410926f466a2 Address:b24e3d03-bd41-bd39-f50e-410926f466a2}]" -2025-10-30T18:18:55.689Z [INFO] server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734: entering follower state: follower="Node at 2cf29b51-4d7d-232a-cc8f-b7e4b5402734 [Follower]" leader-address= leader-id= -2025-10-30T18:18:55.689Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734 Address:2cf29b51-4d7d-232a-cc8f-b7e4b5402734} {Suffrage:Voter ID:server-3759ae26-f796-c659-a13d-88d1d2088f31 Address:3759ae26-f796-c659-a13d-88d1d2088f31} {Suffrage:Voter ID:server-b24e3d03-bd41-bd39-f50e-410926f466a2 Address:b24e3d03-bd41-bd39-f50e-410926f466a2}]" -2025-10-30T18:18:55.689Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: entering follower state: follower="Node at 3759ae26-f796-c659-a13d-88d1d2088f31 [Follower]" leader-address= leader-id= -2025-10-30T18:18:55.689Z [INFO] server-b24e3d03-bd41-bd39-f50e-410926f466a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734 Address:2cf29b51-4d7d-232a-cc8f-b7e4b5402734} {Suffrage:Voter ID:server-3759ae26-f796-c659-a13d-88d1d2088f31 Address:3759ae26-f796-c659-a13d-88d1d2088f31} {Suffrage:Voter ID:server-b24e3d03-bd41-bd39-f50e-410926f466a2 Address:b24e3d03-bd41-bd39-f50e-410926f466a2}]" -2025-10-30T18:18:55.689Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:55.689Z [INFO] server-b24e3d03-bd41-bd39-f50e-410926f466a2: entering follower state: follower="Node at b24e3d03-bd41-bd39-f50e-410926f466a2 [Follower]" leader-address= leader-id= -2025-10-30T18:18:56.502Z [WARN] server-3759ae26-f796-c659-a13d-88d1d2088f31: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:56.502Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: entering candidate state: node="Node at 3759ae26-f796-c659-a13d-88d1d2088f31 [Candidate]" term=2 -2025-10-30T18:18:56.502Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: election won: term=2 tally=2 -2025-10-30T18:18:56.502Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: entering leader state: leader="Node at 3759ae26-f796-c659-a13d-88d1d2088f31 [Leader]" -2025-10-30T18:18:56.502Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: added peer, starting replication: peer=server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734 -2025-10-30T18:18:56.502Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: added peer, starting replication: peer=server-b24e3d03-bd41-bd39-f50e-410926f466a2 -2025-10-30T18:18:56.502Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: pipelining replication: peer="{Voter server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734 2cf29b51-4d7d-232a-cc8f-b7e4b5402734}" -2025-10-30T18:18:56.502Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: pipelining replication: peer="{Voter server-b24e3d03-bd41-bd39-f50e-410926f466a2 b24e3d03-bd41-bd39-f50e-410926f466a2}" -2025-10-30T18:18:57.508Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:56.502272751 +0000 UTC m=+82.780946648 (1 nodes), 813.090473ms from start of poll, 813.350554ms from cluster start. Timeout at 2025-10-30 18:18:57.507276137 +0000 UTC m=+83.785950036, 1.005003388s after stability -2025-10-30T18:18:57.508Z [INFO] creating new snapshot: path=/tmp/raft3170197592/snapshots/2-1013-1761848337508.tmp -2025-10-30T18:18:57.573Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: copied to local snapshot: bytes=71 -2025-10-30T18:18:57.573Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: snapshot restore progress: id=2-1013-1761848337508 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:57.573Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: removing all old logs from log store -2025-10-30T18:18:57.573Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: compacting logs: from=1 to=2 -2025-10-30T18:18:57.573Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: restored user snapshot: index=1013 -2025-10-30T18:18:57.573Z [ERROR] server-3759ae26-f796-c659-a13d-88d1d2088f31: failed to get log: index=2 error="log not found" -2025-10-30T18:18:57.573Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: aborting pipeline replication: peer="{Voter server-b24e3d03-bd41-bd39-f50e-410926f466a2 b24e3d03-bd41-bd39-f50e-410926f466a2}" -2025-10-30T18:18:57.573Z [ERROR] server-3759ae26-f796-c659-a13d-88d1d2088f31: failed to get log: index=2 error="log not found" -2025-10-30T18:18:57.573Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: aborting pipeline replication: peer="{Voter server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734 2cf29b51-4d7d-232a-cc8f-b7e4b5402734}" -2025-10-30T18:18:57.581Z [ERROR] server-3759ae26-f796-c659-a13d-88d1d2088f31: failed to get log: index=2 error="log not found" -2025-10-30T18:18:57.581Z [ERROR] server-3759ae26-f796-c659-a13d-88d1d2088f31: failed to get log: index=2 error="log not found" -2025-10-30T18:18:57.581Z [INFO] creating new snapshot: path=/tmp/raft1324536089/snapshots/2-1013-1761848337581.tmp -2025-10-30T18:18:57.581Z [INFO] creating new snapshot: path=/tmp/raft2252923619/snapshots/2-1013-1761848337581.tmp -2025-10-30T18:18:57.681Z [INFO] server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:57.681Z [INFO] server-b24e3d03-bd41-bd39-f50e-410926f466a2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:57.681Z [INFO] server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734: copied to local snapshot: bytes=71 -2025-10-30T18:18:57.681Z [INFO] server-b24e3d03-bd41-bd39-f50e-410926f466a2: copied to local snapshot: bytes=71 -2025-10-30T18:18:57.682Z [INFO] server-b24e3d03-bd41-bd39-f50e-410926f466a2: snapshot restore progress: id=2-1013-1761848337581 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:57.682Z [INFO] server-b24e3d03-bd41-bd39-f50e-410926f466a2: removing all old logs from log store -2025-10-30T18:18:57.682Z [INFO] server-b24e3d03-bd41-bd39-f50e-410926f466a2: compacting logs: from=1 to=2 -2025-10-30T18:18:57.682Z [INFO] server-b24e3d03-bd41-bd39-f50e-410926f466a2: Installed remote snapshot -2025-10-30T18:18:57.682Z [INFO] server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734: snapshot restore progress: id=2-1013-1761848337581 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:18:57.682Z [INFO] server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734: removing all old logs from log store -2025-10-30T18:18:57.682Z [INFO] server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734: compacting logs: from=1 to=2 -2025-10-30T18:18:57.682Z [INFO] server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734: Installed remote snapshot -2025-10-30T18:18:57.682Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: pipelining replication: peer="{Voter server-b24e3d03-bd41-bd39-f50e-410926f466a2 b24e3d03-bd41-bd39-f50e-410926f466a2}" -2025-10-30T18:18:57.682Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:18:57.682Z [INFO] server-3759ae26-f796-c659-a13d-88d1d2088f31: pipelining replication: peer="{Voter server-2cf29b51-4d7d-232a-cc8f-b7e4b5402734 2cf29b51-4d7d-232a-cc8f-b7e4b5402734}" -2025-10-30T18:18:58.687Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:18:57.682206485 +0000 UTC m=+83.960880389 (2 nodes), 899ns from start of poll, 1.993284295s from cluster start. Timeout at 2025-10-30 18:18:58.687209741 +0000 UTC m=+84.965883638, 1.005003249s after stability -2025-10-30T18:18:58.698Z [INFO] server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: aborting pipeline replication: peer="{Voter server-4e16e616-bafa-1c9f-6408-e796c971cc8b 4e16e616-bafa-1c9f-6408-e796c971cc8b}" -2025-10-30T18:18:58.698Z [INFO] server-6fee6882-dfe8-3952-a66f-c947a53ddfe5: aborting pipeline replication: peer="{Voter server-b4aa64b9-6c20-40f0-17ef-35a28b344ffc b4aa64b9-6c20-40f0-17ef-35a28b344ffc}" +2025-10-30T18:22:45.281Z [INFO] server-f62ab646-5f5b-c483-009e-7620ef3a7648: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f62ab646-5f5b-c483-009e-7620ef3a7648 Address:f62ab646-5f5b-c483-009e-7620ef3a7648} {Suffrage:Voter ID:server-268cfe0f-5168-2b3e-08de-28a1b49e536a Address:268cfe0f-5168-2b3e-08de-28a1b49e536a} {Suffrage:Voter ID:server-55a83d9d-6714-3e69-1bc1-8cf6a231a822 Address:55a83d9d-6714-3e69-1bc1-8cf6a231a822}]" +2025-10-30T18:22:45.281Z [INFO] server-f62ab646-5f5b-c483-009e-7620ef3a7648: entering follower state: follower="Node at f62ab646-5f5b-c483-009e-7620ef3a7648 [Follower]" leader-address= leader-id= +2025-10-30T18:22:45.281Z [INFO] server-268cfe0f-5168-2b3e-08de-28a1b49e536a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f62ab646-5f5b-c483-009e-7620ef3a7648 Address:f62ab646-5f5b-c483-009e-7620ef3a7648} {Suffrage:Voter ID:server-268cfe0f-5168-2b3e-08de-28a1b49e536a Address:268cfe0f-5168-2b3e-08de-28a1b49e536a} {Suffrage:Voter ID:server-55a83d9d-6714-3e69-1bc1-8cf6a231a822 Address:55a83d9d-6714-3e69-1bc1-8cf6a231a822}]" +2025-10-30T18:22:45.281Z [INFO] server-268cfe0f-5168-2b3e-08de-28a1b49e536a: entering follower state: follower="Node at 268cfe0f-5168-2b3e-08de-28a1b49e536a [Follower]" leader-address= leader-id= +2025-10-30T18:22:45.281Z [INFO] server-55a83d9d-6714-3e69-1bc1-8cf6a231a822: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f62ab646-5f5b-c483-009e-7620ef3a7648 Address:f62ab646-5f5b-c483-009e-7620ef3a7648} {Suffrage:Voter ID:server-268cfe0f-5168-2b3e-08de-28a1b49e536a Address:268cfe0f-5168-2b3e-08de-28a1b49e536a} {Suffrage:Voter ID:server-55a83d9d-6714-3e69-1bc1-8cf6a231a822 Address:55a83d9d-6714-3e69-1bc1-8cf6a231a822}]" +2025-10-30T18:22:45.281Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:45.281Z [INFO] server-55a83d9d-6714-3e69-1bc1-8cf6a231a822: entering follower state: follower="Node at 55a83d9d-6714-3e69-1bc1-8cf6a231a822 [Follower]" leader-address= leader-id= +2025-10-30T18:22:45.288Z [ERROR] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: peer has newer term, stopping replication: peer="{Voter server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a 3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a}" +2025-10-30T18:22:45.288Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: entering follower state: follower="Node at d35ebf55-0d21-fd1e-eb65-c2c2fba58466 [Follower]" leader-address= leader-id= +2025-10-30T18:22:45.288Z [INFO] server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466: aborting pipeline replication: peer="{Voter server-d90a6fc0-692f-02d2-8800-1bed4355b62e d90a6fc0-692f-02d2-8800-1bed4355b62e}" +2025-10-30T18:22:45.805Z [WARN] server-4f165838-c7a2-0648-7438-0652345f61e1: heartbeat timeout reached, starting election: last-leader-addr=21909053-afae-878e-2be8-0085c1c63f8a last-leader-id=server-21909053-afae-878e-2be8-0085c1c63f8a +2025-10-30T18:22:45.805Z [INFO] server-4f165838-c7a2-0648-7438-0652345f61e1: entering candidate state: node="Node at 4f165838-c7a2-0648-7438-0652345f61e1 [Candidate]" term=3 +2025-10-30T18:22:45.805Z [INFO] server-4f165838-c7a2-0648-7438-0652345f61e1: election won: term=3 tally=2 +2025-10-30T18:22:45.805Z [INFO] server-4f165838-c7a2-0648-7438-0652345f61e1: entering leader state: leader="Node at 4f165838-c7a2-0648-7438-0652345f61e1 [Leader]" +2025-10-30T18:22:45.805Z [INFO] server-a26ff83d-e22a-136c-9b62-5bb846157d8e: duplicate requestVote for same term: term=3 +2025-10-30T18:22:45.805Z [INFO] server-4f165838-c7a2-0648-7438-0652345f61e1: added peer, starting replication: peer=server-a26ff83d-e22a-136c-9b62-5bb846157d8e +2025-10-30T18:22:45.805Z [INFO] server-4f165838-c7a2-0648-7438-0652345f61e1: added peer, starting replication: peer=server-21909053-afae-878e-2be8-0085c1c63f8a +2025-10-30T18:22:45.805Z [INFO] server-a26ff83d-e22a-136c-9b62-5bb846157d8e: entering follower state: follower="Node at a26ff83d-e22a-136c-9b62-5bb846157d8e [Follower]" leader-address=4f165838-c7a2-0648-7438-0652345f61e1 leader-id=server-4f165838-c7a2-0648-7438-0652345f61e1 +2025-10-30T18:22:45.805Z [INFO] server-4f165838-c7a2-0648-7438-0652345f61e1: pipelining replication: peer="{Voter server-21909053-afae-878e-2be8-0085c1c63f8a 21909053-afae-878e-2be8-0085c1c63f8a}" +2025-10-30T18:22:45.805Z [INFO] server-4f165838-c7a2-0648-7438-0652345f61e1: pipelining replication: peer="{Voter server-a26ff83d-e22a-136c-9b62-5bb846157d8e a26ff83d-e22a-136c-9b62-5bb846157d8e}" +2025-10-30T18:22:45.864Z [WARN] server-d90a6fc0-692f-02d2-8800-1bed4355b62e: heartbeat timeout reached, starting election: last-leader-addr=d35ebf55-0d21-fd1e-eb65-c2c2fba58466 last-leader-id=server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466 +2025-10-30T18:22:45.864Z [INFO] server-d90a6fc0-692f-02d2-8800-1bed4355b62e: entering candidate state: node="Node at d90a6fc0-692f-02d2-8800-1bed4355b62e [Candidate]" term=3 +2025-10-30T18:22:45.864Z [INFO] server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a: duplicate requestVote for same term: term=3 +2025-10-30T18:22:45.864Z [INFO] server-d90a6fc0-692f-02d2-8800-1bed4355b62e: election won: term=3 tally=2 +2025-10-30T18:22:45.864Z [INFO] server-d90a6fc0-692f-02d2-8800-1bed4355b62e: entering leader state: leader="Node at d90a6fc0-692f-02d2-8800-1bed4355b62e [Leader]" +2025-10-30T18:22:45.864Z [INFO] server-d90a6fc0-692f-02d2-8800-1bed4355b62e: added peer, starting replication: peer=server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a +2025-10-30T18:22:45.864Z [INFO] server-d90a6fc0-692f-02d2-8800-1bed4355b62e: added peer, starting replication: peer=server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466 +2025-10-30T18:22:45.864Z [INFO] server-d90a6fc0-692f-02d2-8800-1bed4355b62e: pipelining replication: peer="{Voter server-d35ebf55-0d21-fd1e-eb65-c2c2fba58466 d35ebf55-0d21-fd1e-eb65-c2c2fba58466}" +2025-10-30T18:22:45.864Z [INFO] server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a: entering follower state: follower="Node at 3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a [Follower]" leader-address=d90a6fc0-692f-02d2-8800-1bed4355b62e leader-id=server-d90a6fc0-692f-02d2-8800-1bed4355b62e +2025-10-30T18:22:45.864Z [INFO] server-d90a6fc0-692f-02d2-8800-1bed4355b62e: pipelining replication: peer="{Voter server-3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a 3e7fb186-972c-c53d-0bfd-3b9e5ced8c6a}" +2025-10-30T18:22:45.877Z [WARN] server-fba31bce-63b2-1d27-8898-8934ad09d77b: Election timeout reached, restarting election +2025-10-30T18:22:45.877Z [INFO] server-fba31bce-63b2-1d27-8898-8934ad09d77b: entering candidate state: node="Node at fba31bce-63b2-1d27-8898-8934ad09d77b [Candidate]" term=4 +2025-10-30T18:22:45.877Z [INFO] server-fba31bce-63b2-1d27-8898-8934ad09d77b: election won: term=4 tally=2 +2025-10-30T18:22:45.877Z [INFO] server-fba31bce-63b2-1d27-8898-8934ad09d77b: entering leader state: leader="Node at fba31bce-63b2-1d27-8898-8934ad09d77b [Leader]" +2025-10-30T18:22:45.877Z [INFO] server-788cf89c-090e-f232-af4b-94504abbc2a1: entering follower state: follower="Node at 788cf89c-090e-f232-af4b-94504abbc2a1 [Follower]" leader-address= leader-id= +2025-10-30T18:22:45.877Z [INFO] server-fba31bce-63b2-1d27-8898-8934ad09d77b: added peer, starting replication: peer=server-788cf89c-090e-f232-af4b-94504abbc2a1 +2025-10-30T18:22:45.877Z [INFO] server-fba31bce-63b2-1d27-8898-8934ad09d77b: added peer, starting replication: peer=server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db +2025-10-30T18:22:45.877Z [INFO] server-fba31bce-63b2-1d27-8898-8934ad09d77b: pipelining replication: peer="{Voter server-1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db 1dbf71e7-cd7b-ca2e-6a2f-5d942439c9db}" +2025-10-30T18:22:45.877Z [INFO] server-fba31bce-63b2-1d27-8898-8934ad09d77b: pipelining replication: peer="{Voter server-788cf89c-090e-f232-af4b-94504abbc2a1 788cf89c-090e-f232-af4b-94504abbc2a1}" +2025-10-30T18:22:45.940Z [WARN] server-55a83d9d-6714-3e69-1bc1-8cf6a231a822: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:45.940Z [INFO] server-55a83d9d-6714-3e69-1bc1-8cf6a231a822: entering candidate state: node="Node at 55a83d9d-6714-3e69-1bc1-8cf6a231a822 [Candidate]" term=2 +2025-10-30T18:22:45.940Z [INFO] server-55a83d9d-6714-3e69-1bc1-8cf6a231a822: election won: term=2 tally=2 +2025-10-30T18:22:45.940Z [INFO] server-55a83d9d-6714-3e69-1bc1-8cf6a231a822: entering leader state: leader="Node at 55a83d9d-6714-3e69-1bc1-8cf6a231a822 [Leader]" +2025-10-30T18:22:45.940Z [INFO] server-55a83d9d-6714-3e69-1bc1-8cf6a231a822: added peer, starting replication: peer=server-f62ab646-5f5b-c483-009e-7620ef3a7648 +2025-10-30T18:22:45.940Z [INFO] server-55a83d9d-6714-3e69-1bc1-8cf6a231a822: added peer, starting replication: peer=server-268cfe0f-5168-2b3e-08de-28a1b49e536a +2025-10-30T18:22:45.940Z [INFO] server-55a83d9d-6714-3e69-1bc1-8cf6a231a822: pipelining replication: peer="{Voter server-268cfe0f-5168-2b3e-08de-28a1b49e536a 268cfe0f-5168-2b3e-08de-28a1b49e536a}" +2025-10-30T18:22:45.940Z [INFO] server-55a83d9d-6714-3e69-1bc1-8cf6a231a822: pipelining replication: peer="{Voter server-f62ab646-5f5b-c483-009e-7620ef3a7648 f62ab646-5f5b-c483-009e-7620ef3a7648}" +2025-10-30T18:22:45.960Z [WARN] server-d65b504a-8185-4303-efb5-28ea2cf86fac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:45.960Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: entering candidate state: node="Node at d65b504a-8185-4303-efb5-28ea2cf86fac [Candidate]" term=3 +2025-10-30T18:22:45.960Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: election won: term=3 tally=2 +2025-10-30T18:22:45.960Z [INFO] server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18: duplicate requestVote for same term: term=3 +2025-10-30T18:22:45.960Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: entering leader state: leader="Node at d65b504a-8185-4303-efb5-28ea2cf86fac [Leader]" +2025-10-30T18:22:45.960Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: added peer, starting replication: peer=server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18 +2025-10-30T18:22:45.960Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: added peer, starting replication: peer=server-de776368-6add-9f70-7549-04e98aff6559 +2025-10-30T18:22:45.960Z [INFO] server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18: entering follower state: follower="Node at 4c9a84fb-0b48-edaf-fee7-5811a1c40d18 [Follower]" leader-address=d65b504a-8185-4303-efb5-28ea2cf86fac leader-id=server-d65b504a-8185-4303-efb5-28ea2cf86fac +2025-10-30T18:22:45.960Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: pipelining replication: peer="{Voter server-4c9a84fb-0b48-edaf-fee7-5811a1c40d18 4c9a84fb-0b48-edaf-fee7-5811a1c40d18}" +2025-10-30T18:22:45.960Z [INFO] server-d65b504a-8185-4303-efb5-28ea2cf86fac: pipelining replication: peer="{Voter server-de776368-6add-9f70-7549-04e98aff6559 de776368-6add-9f70-7549-04e98aff6559}" +2025-10-30T18:22:46.043Z [WARN] server-84a28fee-22e2-6d5e-43d6-02566865291f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:46.043Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: entering candidate state: node="Node at 84a28fee-22e2-6d5e-43d6-02566865291f [Candidate]" term=4 +2025-10-30T18:22:46.043Z [INFO] server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01: entering follower state: follower="Node at 0ac54fe3-b7d6-d944-9b1d-0585a2283f01 [Follower]" leader-address= leader-id= +2025-10-30T18:22:46.043Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: election won: term=4 tally=2 +2025-10-30T18:22:46.043Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: entering leader state: leader="Node at 84a28fee-22e2-6d5e-43d6-02566865291f [Leader]" +2025-10-30T18:22:46.043Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: added peer, starting replication: peer=server-b6ac980b-0a86-7a3e-d752-e95f45c45f05 +2025-10-30T18:22:46.043Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: added peer, starting replication: peer=server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01 +2025-10-30T18:22:46.043Z [INFO] server-b6ac980b-0a86-7a3e-d752-e95f45c45f05: entering follower state: follower="Node at b6ac980b-0a86-7a3e-d752-e95f45c45f05 [Follower]" leader-address= leader-id= +2025-10-30T18:22:46.043Z [WARN] server-b6ac980b-0a86-7a3e-d752-e95f45c45f05: failed to get previous log: previous-index=1023 last-index=1014 error="log not found" +2025-10-30T18:22:46.043Z [WARN] server-84a28fee-22e2-6d5e-43d6-02566865291f: appendEntries rejected, sending older logs: peer="{Voter server-b6ac980b-0a86-7a3e-d752-e95f45c45f05 b6ac980b-0a86-7a3e-d752-e95f45c45f05}" next=1015 +2025-10-30T18:22:46.043Z [WARN] server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01: failed to get previous log: previous-index=1023 last-index=1014 error="log not found" +2025-10-30T18:22:46.043Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: pipelining replication: peer="{Voter server-b6ac980b-0a86-7a3e-d752-e95f45c45f05 b6ac980b-0a86-7a3e-d752-e95f45c45f05}" +2025-10-30T18:22:46.043Z [WARN] server-84a28fee-22e2-6d5e-43d6-02566865291f: appendEntries rejected, sending older logs: peer="{Voter server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01 0ac54fe3-b7d6-d944-9b1d-0585a2283f01}" next=1015 +2025-10-30T18:22:46.044Z [INFO] server-84a28fee-22e2-6d5e-43d6-02566865291f: pipelining replication: peer="{Voter server-0ac54fe3-b7d6-d944-9b1d-0585a2283f01 0ac54fe3-b7d6-d944-9b1d-0585a2283f01}" +2025-10-30T18:22:46.945Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:45.940339726 +0000 UTC m=+89.013869042 (1 nodes), 658.692106ms from start of poll, 659.000255ms from cluster start. Timeout at 2025-10-30 18:22:46.945342586 +0000 UTC m=+90.018871879, 1.005002837s after stability +2025-10-30T18:22:46.945Z [INFO] server-55a83d9d-6714-3e69-1bc1-8cf6a231a822: starting snapshot up to: index=12 +2025-10-30T18:22:46.945Z [INFO] creating new snapshot: path=/tmp/raft3820123859/snapshots/2-12-1761848566945.tmp +2025-10-30T18:22:47.398Z [INFO] server-55a83d9d-6714-3e69-1bc1-8cf6a231a822: snapshot complete up to: index=12 +2025-10-30T18:22:47.399Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2 Address:cc55bd17-6290-e7a8-4d12-67b6f2072ad2} {Suffrage:Voter ID:server-a7ec6eef-87e8-7401-5961-db7bb6ea8b31 Address:a7ec6eef-87e8-7401-5961-db7bb6ea8b31} {Suffrage:Voter ID:server-1279b367-380a-56ab-165f-8705e215b4d1 Address:1279b367-380a-56ab-165f-8705e215b4d1}]" +2025-10-30T18:22:47.399Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: entering follower state: follower="Node at cc55bd17-6290-e7a8-4d12-67b6f2072ad2 [Follower]" leader-address= leader-id= +2025-10-30T18:22:47.399Z [INFO] server-a7ec6eef-87e8-7401-5961-db7bb6ea8b31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2 Address:cc55bd17-6290-e7a8-4d12-67b6f2072ad2} {Suffrage:Voter ID:server-a7ec6eef-87e8-7401-5961-db7bb6ea8b31 Address:a7ec6eef-87e8-7401-5961-db7bb6ea8b31} {Suffrage:Voter ID:server-1279b367-380a-56ab-165f-8705e215b4d1 Address:1279b367-380a-56ab-165f-8705e215b4d1}]" +2025-10-30T18:22:47.399Z [INFO] server-a7ec6eef-87e8-7401-5961-db7bb6ea8b31: entering follower state: follower="Node at a7ec6eef-87e8-7401-5961-db7bb6ea8b31 [Follower]" leader-address= leader-id= +2025-10-30T18:22:47.399Z [INFO] server-1279b367-380a-56ab-165f-8705e215b4d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2 Address:cc55bd17-6290-e7a8-4d12-67b6f2072ad2} {Suffrage:Voter ID:server-a7ec6eef-87e8-7401-5961-db7bb6ea8b31 Address:a7ec6eef-87e8-7401-5961-db7bb6ea8b31} {Suffrage:Voter ID:server-1279b367-380a-56ab-165f-8705e215b4d1 Address:1279b367-380a-56ab-165f-8705e215b4d1}]" +2025-10-30T18:22:47.399Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:47.399Z [INFO] server-1279b367-380a-56ab-165f-8705e215b4d1: entering follower state: follower="Node at 1279b367-380a-56ab-165f-8705e215b4d1 [Follower]" leader-address= leader-id= +2025-10-30T18:22:47.959Z [WARN] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:47.960Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: entering candidate state: node="Node at cc55bd17-6290-e7a8-4d12-67b6f2072ad2 [Candidate]" term=2 +2025-10-30T18:22:47.960Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: election won: term=2 tally=2 +2025-10-30T18:22:47.960Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: entering leader state: leader="Node at cc55bd17-6290-e7a8-4d12-67b6f2072ad2 [Leader]" +2025-10-30T18:22:47.960Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: added peer, starting replication: peer=server-a7ec6eef-87e8-7401-5961-db7bb6ea8b31 +2025-10-30T18:22:47.960Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: added peer, starting replication: peer=server-1279b367-380a-56ab-165f-8705e215b4d1 +2025-10-30T18:22:47.960Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: pipelining replication: peer="{Voter server-a7ec6eef-87e8-7401-5961-db7bb6ea8b31 a7ec6eef-87e8-7401-5961-db7bb6ea8b31}" +2025-10-30T18:22:47.960Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: pipelining replication: peer="{Voter server-1279b367-380a-56ab-165f-8705e215b4d1 1279b367-380a-56ab-165f-8705e215b4d1}" +2025-10-30T18:22:48.965Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:47.960203239 +0000 UTC m=+91.033732534 (1 nodes), 560.708993ms from start of poll, 561.005163ms from cluster start. Timeout at 2025-10-30 18:22:48.965205893 +0000 UTC m=+92.038735182, 1.005002648s after stability +2025-10-30T18:22:49.184Z [INFO] creating new snapshot: path=/tmp/raft2515668718/snapshots/2-1013-1761848569184.tmp +2025-10-30T18:22:49.282Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: copied to local snapshot: bytes=71 +2025-10-30T18:22:49.283Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: snapshot restore progress: id=2-1013-1761848569184 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:49.283Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: removing all old logs from log store +2025-10-30T18:22:49.283Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: compacting logs: from=1 to=2 +2025-10-30T18:22:49.283Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: restored user snapshot: index=1013 +2025-10-30T18:22:49.283Z [ERROR] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: failed to get log: index=2 error="log not found" +2025-10-30T18:22:49.283Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: aborting pipeline replication: peer="{Voter server-a7ec6eef-87e8-7401-5961-db7bb6ea8b31 a7ec6eef-87e8-7401-5961-db7bb6ea8b31}" +2025-10-30T18:22:49.283Z [ERROR] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: failed to get log: index=2 error="log not found" +2025-10-30T18:22:49.283Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: aborting pipeline replication: peer="{Voter server-1279b367-380a-56ab-165f-8705e215b4d1 1279b367-380a-56ab-165f-8705e215b4d1}" +2025-10-30T18:22:49.289Z [ERROR] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: failed to get log: index=2 error="log not found" +2025-10-30T18:22:49.291Z [ERROR] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: failed to get log: index=2 error="log not found" +2025-10-30T18:22:49.401Z [INFO] creating new snapshot: path=/tmp/raft1160524803/snapshots/2-1013-1761848569401.tmp +2025-10-30T18:22:49.401Z [INFO] creating new snapshot: path=/tmp/raft1556092203/snapshots/2-1013-1761848569401.tmp +2025-10-30T18:22:49.402Z [INFO] server-1279b367-380a-56ab-165f-8705e215b4d1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:49.402Z [INFO] server-a7ec6eef-87e8-7401-5961-db7bb6ea8b31: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:49.411Z [INFO] server-1279b367-380a-56ab-165f-8705e215b4d1: copied to local snapshot: bytes=71 +2025-10-30T18:22:49.411Z [INFO] server-a7ec6eef-87e8-7401-5961-db7bb6ea8b31: copied to local snapshot: bytes=71 +2025-10-30T18:22:49.411Z [INFO] server-a7ec6eef-87e8-7401-5961-db7bb6ea8b31: snapshot restore progress: id=2-1013-1761848569401 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:49.411Z [INFO] server-a7ec6eef-87e8-7401-5961-db7bb6ea8b31: removing all old logs from log store +2025-10-30T18:22:49.411Z [INFO] server-a7ec6eef-87e8-7401-5961-db7bb6ea8b31: compacting logs: from=1 to=2 +2025-10-30T18:22:49.411Z [INFO] server-a7ec6eef-87e8-7401-5961-db7bb6ea8b31: Installed remote snapshot +2025-10-30T18:22:49.411Z [INFO] server-1279b367-380a-56ab-165f-8705e215b4d1: snapshot restore progress: id=2-1013-1761848569401 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:49.411Z [INFO] server-1279b367-380a-56ab-165f-8705e215b4d1: removing all old logs from log store +2025-10-30T18:22:49.411Z [INFO] server-1279b367-380a-56ab-165f-8705e215b4d1: compacting logs: from=1 to=2 +2025-10-30T18:22:49.411Z [INFO] server-1279b367-380a-56ab-165f-8705e215b4d1: Installed remote snapshot +2025-10-30T18:22:49.411Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: pipelining replication: peer="{Voter server-a7ec6eef-87e8-7401-5961-db7bb6ea8b31 a7ec6eef-87e8-7401-5961-db7bb6ea8b31}" +2025-10-30T18:22:49.411Z [INFO] server-cc55bd17-6290-e7a8-4d12-67b6f2072ad2: pipelining replication: peer="{Voter server-1279b367-380a-56ab-165f-8705e215b4d1 1279b367-380a-56ab-165f-8705e215b4d1}" +2025-10-30T18:22:49.412Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:22:50.417Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:22:49.412031471 +0000 UTC m=+92.485560770 (2 nodes), 1.233µs from start of poll, 2.012833399s from cluster start. Timeout at 2025-10-30 18:22:50.417041209 +0000 UTC m=+93.490570520, 1.00500975s after stability +2025-10-30T18:22:50.430Z [INFO] server-55a83d9d-6714-3e69-1bc1-8cf6a231a822: aborting pipeline replication: peer="{Voter server-f62ab646-5f5b-c483-009e-7620ef3a7648 f62ab646-5f5b-c483-009e-7620ef3a7648}" +2025-10-30T18:22:50.430Z [INFO] server-55a83d9d-6714-3e69-1bc1-8cf6a231a822: aborting pipeline replication: peer="{Voter server-268cfe0f-5168-2b3e-08de-28a1b49e536a 268cfe0f-5168-2b3e-08de-28a1b49e536a}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false -2025-10-30T18:18:58.699Z [INFO] server-5467dcbf-11b5-38b4-ed79-f2eb3917af63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5467dcbf-11b5-38b4-ed79-f2eb3917af63 Address:5467dcbf-11b5-38b4-ed79-f2eb3917af63} {Suffrage:Voter ID:server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b Address:56fb0e4e-5a40-4cd5-42ec-41e1050b123b} {Suffrage:Voter ID:server-ac35575a-b6e9-043a-3e9b-d67d217f335a Address:ac35575a-b6e9-043a-3e9b-d67d217f335a}]" -2025-10-30T18:18:58.699Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5467dcbf-11b5-38b4-ed79-f2eb3917af63 Address:5467dcbf-11b5-38b4-ed79-f2eb3917af63} {Suffrage:Voter ID:server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b Address:56fb0e4e-5a40-4cd5-42ec-41e1050b123b} {Suffrage:Voter ID:server-ac35575a-b6e9-043a-3e9b-d67d217f335a Address:ac35575a-b6e9-043a-3e9b-d67d217f335a}]" -2025-10-30T18:18:58.699Z [INFO] server-5467dcbf-11b5-38b4-ed79-f2eb3917af63: entering follower state: follower="Node at 5467dcbf-11b5-38b4-ed79-f2eb3917af63 [Follower]" leader-address= leader-id= -2025-10-30T18:18:58.699Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: entering follower state: follower="Node at 56fb0e4e-5a40-4cd5-42ec-41e1050b123b [Follower]" leader-address= leader-id= -2025-10-30T18:18:58.699Z [INFO] server-ac35575a-b6e9-043a-3e9b-d67d217f335a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5467dcbf-11b5-38b4-ed79-f2eb3917af63 Address:5467dcbf-11b5-38b4-ed79-f2eb3917af63} {Suffrage:Voter ID:server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b Address:56fb0e4e-5a40-4cd5-42ec-41e1050b123b} {Suffrage:Voter ID:server-ac35575a-b6e9-043a-3e9b-d67d217f335a Address:ac35575a-b6e9-043a-3e9b-d67d217f335a}]" -2025-10-30T18:18:58.699Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:18:58.699Z [INFO] server-ac35575a-b6e9-043a-3e9b-d67d217f335a: entering follower state: follower="Node at ac35575a-b6e9-043a-3e9b-d67d217f335a [Follower]" leader-address= leader-id= -2025-10-30T18:18:59.269Z [WARN] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:59.269Z [WARN] server-ac35575a-b6e9-043a-3e9b-d67d217f335a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:18:59.269Z [INFO] server-ac35575a-b6e9-043a-3e9b-d67d217f335a: entering candidate state: node="Node at ac35575a-b6e9-043a-3e9b-d67d217f335a [Candidate]" term=2 -2025-10-30T18:18:59.269Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: entering candidate state: node="Node at 56fb0e4e-5a40-4cd5-42ec-41e1050b123b [Candidate]" term=2 -2025-10-30T18:18:59.269Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: duplicate requestVote for same term: term=2 -2025-10-30T18:18:59.269Z [INFO] server-5467dcbf-11b5-38b4-ed79-f2eb3917af63: duplicate requestVote for same term: term=2 -2025-10-30T18:18:59.269Z [INFO] server-ac35575a-b6e9-043a-3e9b-d67d217f335a: duplicate requestVote for same term: term=2 -2025-10-30T18:18:59.269Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: election won: term=2 tally=2 -2025-10-30T18:18:59.269Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: entering leader state: leader="Node at 56fb0e4e-5a40-4cd5-42ec-41e1050b123b [Leader]" -2025-10-30T18:18:59.269Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: added peer, starting replication: peer=server-5467dcbf-11b5-38b4-ed79-f2eb3917af63 -2025-10-30T18:18:59.269Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: added peer, starting replication: peer=server-ac35575a-b6e9-043a-3e9b-d67d217f335a -2025-10-30T18:18:59.269Z [INFO] server-ac35575a-b6e9-043a-3e9b-d67d217f335a: entering follower state: follower="Node at ac35575a-b6e9-043a-3e9b-d67d217f335a [Follower]" leader-address=56fb0e4e-5a40-4cd5-42ec-41e1050b123b leader-id=server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b -2025-10-30T18:18:59.269Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: pipelining replication: peer="{Voter server-ac35575a-b6e9-043a-3e9b-d67d217f335a ac35575a-b6e9-043a-3e9b-d67d217f335a}" -2025-10-30T18:18:59.269Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: pipelining replication: peer="{Voter server-5467dcbf-11b5-38b4-ed79-f2eb3917af63 5467dcbf-11b5-38b4-ed79-f2eb3917af63}" -2025-10-30T18:19:00.275Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:18:59.269673478 +0000 UTC m=+85.548347388 (1 nodes), 569.679824ms from start of poll, 569.960286ms from cluster start. Timeout at 2025-10-30 18:19:00.274676734 +0000 UTC m=+86.553350631, 1.005003243s after stability -2025-10-30T18:19:00.276Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: starting snapshot up to: index=12 -2025-10-30T18:19:00.276Z [INFO] creating new snapshot: path=/tmp/raft846765429/snapshots/2-12-1761848340276.tmp -2025-10-30T18:19:00.410Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: snapshot complete up to: index=12 -2025-10-30T18:19:00.410Z [INFO] creating new snapshot: path=/tmp/raft846765429/snapshots/2-10013-1761848340410.tmp -2025-10-30T18:19:00.411Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: copied to local snapshot: bytes=71 -2025-10-30T18:19:00.411Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: snapshot restore progress: id=2-10013-1761848340410 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:00.411Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: restored user snapshot: index=10013 -2025-10-30T18:19:00.411Z [ERROR] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: failed to get log: index=23 error="log not found" -2025-10-30T18:19:00.411Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: aborting pipeline replication: peer="{Voter server-ac35575a-b6e9-043a-3e9b-d67d217f335a ac35575a-b6e9-043a-3e9b-d67d217f335a}" -2025-10-30T18:19:00.411Z [ERROR] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: failed to get log: index=23 error="log not found" -2025-10-30T18:19:00.411Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: aborting pipeline replication: peer="{Voter server-5467dcbf-11b5-38b4-ed79-f2eb3917af63 5467dcbf-11b5-38b4-ed79-f2eb3917af63}" -2025-10-30T18:19:00.418Z [ERROR] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: failed to get log: index=23 error="log not found" -2025-10-30T18:19:00.418Z [INFO] creating new snapshot: path=/tmp/raft1870409754/snapshots/2-10013-1761848340418.tmp -2025-10-30T18:19:00.418Z [ERROR] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: failed to get log: index=23 error="log not found" -2025-10-30T18:19:00.418Z [INFO] server-5467dcbf-11b5-38b4-ed79-f2eb3917af63: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:00.418Z [INFO] creating new snapshot: path=/tmp/raft3121484863/snapshots/2-10013-1761848340418.tmp -2025-10-30T18:19:00.419Z [INFO] server-ac35575a-b6e9-043a-3e9b-d67d217f335a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:00.747Z [INFO] server-5467dcbf-11b5-38b4-ed79-f2eb3917af63: copied to local snapshot: bytes=71 -2025-10-30T18:19:00.747Z [INFO] server-ac35575a-b6e9-043a-3e9b-d67d217f335a: copied to local snapshot: bytes=71 -2025-10-30T18:19:00.747Z [INFO] server-5467dcbf-11b5-38b4-ed79-f2eb3917af63: snapshot restore progress: id=2-10013-1761848340418 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:00.747Z [INFO] server-5467dcbf-11b5-38b4-ed79-f2eb3917af63: Installed remote snapshot -2025-10-30T18:19:00.747Z [INFO] server-ac35575a-b6e9-043a-3e9b-d67d217f335a: snapshot restore progress: id=2-10013-1761848340418 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:00.747Z [INFO] server-ac35575a-b6e9-043a-3e9b-d67d217f335a: Installed remote snapshot -2025-10-30T18:19:00.747Z [WARN] server-5467dcbf-11b5-38b4-ed79-f2eb3917af63: skipping application of old log: index=22 -2025-10-30T18:19:00.747Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: pipelining replication: peer="{Voter server-5467dcbf-11b5-38b4-ed79-f2eb3917af63 5467dcbf-11b5-38b4-ed79-f2eb3917af63}" -2025-10-30T18:19:00.747Z [WARN] server-ac35575a-b6e9-043a-3e9b-d67d217f335a: skipping application of old log: index=22 -2025-10-30T18:19:00.747Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:00.747Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: pipelining replication: peer="{Voter server-ac35575a-b6e9-043a-3e9b-d67d217f335a ac35575a-b6e9-043a-3e9b-d67d217f335a}" -2025-10-30T18:19:01.752Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:00.747541416 +0000 UTC m=+87.026215325 (2 nodes), 587ns from start of poll, 2.047828223s from cluster start. Timeout at 2025-10-30 18:19:01.752572413 +0000 UTC m=+88.031246310, 1.005030985s after stability -2025-10-30T18:19:01.764Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: aborting pipeline replication: peer="{Voter server-ac35575a-b6e9-043a-3e9b-d67d217f335a ac35575a-b6e9-043a-3e9b-d67d217f335a}" -2025-10-30T18:19:01.764Z [INFO] server-56fb0e4e-5a40-4cd5-42ec-41e1050b123b: aborting pipeline replication: peer="{Voter server-5467dcbf-11b5-38b4-ed79-f2eb3917af63 5467dcbf-11b5-38b4-ed79-f2eb3917af63}" +2025-10-30T18:22:50.431Z [INFO] server-89247942-b1a8-14d7-0539-65c8f7e60807: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89247942-b1a8-14d7-0539-65c8f7e60807 Address:89247942-b1a8-14d7-0539-65c8f7e60807} {Suffrage:Voter ID:server-f51b0e56-2763-1092-764b-e1f0c36cd3bc Address:f51b0e56-2763-1092-764b-e1f0c36cd3bc} {Suffrage:Voter ID:server-1d0039b0-1aa8-e0fd-ad96-eb98a8b94046 Address:1d0039b0-1aa8-e0fd-ad96-eb98a8b94046}]" +2025-10-30T18:22:50.431Z [INFO] server-89247942-b1a8-14d7-0539-65c8f7e60807: entering follower state: follower="Node at 89247942-b1a8-14d7-0539-65c8f7e60807 [Follower]" leader-address= leader-id= +2025-10-30T18:22:50.431Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89247942-b1a8-14d7-0539-65c8f7e60807 Address:89247942-b1a8-14d7-0539-65c8f7e60807} {Suffrage:Voter ID:server-f51b0e56-2763-1092-764b-e1f0c36cd3bc Address:f51b0e56-2763-1092-764b-e1f0c36cd3bc} {Suffrage:Voter ID:server-1d0039b0-1aa8-e0fd-ad96-eb98a8b94046 Address:1d0039b0-1aa8-e0fd-ad96-eb98a8b94046}]" +2025-10-30T18:22:50.431Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: entering follower state: follower="Node at f51b0e56-2763-1092-764b-e1f0c36cd3bc [Follower]" leader-address= leader-id= +2025-10-30T18:22:50.431Z [INFO] server-1d0039b0-1aa8-e0fd-ad96-eb98a8b94046: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89247942-b1a8-14d7-0539-65c8f7e60807 Address:89247942-b1a8-14d7-0539-65c8f7e60807} {Suffrage:Voter ID:server-f51b0e56-2763-1092-764b-e1f0c36cd3bc Address:f51b0e56-2763-1092-764b-e1f0c36cd3bc} {Suffrage:Voter ID:server-1d0039b0-1aa8-e0fd-ad96-eb98a8b94046 Address:1d0039b0-1aa8-e0fd-ad96-eb98a8b94046}]" +2025-10-30T18:22:50.431Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:50.431Z [INFO] server-1d0039b0-1aa8-e0fd-ad96-eb98a8b94046: entering follower state: follower="Node at 1d0039b0-1aa8-e0fd-ad96-eb98a8b94046 [Follower]" leader-address= leader-id= +2025-10-30T18:22:51.122Z [WARN] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:51.122Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: entering candidate state: node="Node at f51b0e56-2763-1092-764b-e1f0c36cd3bc [Candidate]" term=2 +2025-10-30T18:22:51.122Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: election won: term=2 tally=2 +2025-10-30T18:22:51.122Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: entering leader state: leader="Node at f51b0e56-2763-1092-764b-e1f0c36cd3bc [Leader]" +2025-10-30T18:22:51.122Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: added peer, starting replication: peer=server-89247942-b1a8-14d7-0539-65c8f7e60807 +2025-10-30T18:22:51.122Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: added peer, starting replication: peer=server-1d0039b0-1aa8-e0fd-ad96-eb98a8b94046 +2025-10-30T18:22:51.122Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: pipelining replication: peer="{Voter server-89247942-b1a8-14d7-0539-65c8f7e60807 89247942-b1a8-14d7-0539-65c8f7e60807}" +2025-10-30T18:22:51.122Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: pipelining replication: peer="{Voter server-1d0039b0-1aa8-e0fd-ad96-eb98a8b94046 1d0039b0-1aa8-e0fd-ad96-eb98a8b94046}" +2025-10-30T18:22:52.128Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:51.122816703 +0000 UTC m=+94.196345996 (1 nodes), 691.345516ms from start of poll, 691.684589ms from cluster start. Timeout at 2025-10-30 18:22:52.127820193 +0000 UTC m=+95.201349486, 1.00500349s after stability +2025-10-30T18:22:52.128Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: starting snapshot up to: index=12 +2025-10-30T18:22:52.128Z [INFO] creating new snapshot: path=/tmp/raft3534640099/snapshots/2-12-1761848572128.tmp +2025-10-30T18:22:52.173Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: snapshot complete up to: index=12 +2025-10-30T18:22:52.173Z [INFO] creating new snapshot: path=/tmp/raft3534640099/snapshots/2-10013-1761848572173.tmp +2025-10-30T18:22:52.174Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: copied to local snapshot: bytes=71 +2025-10-30T18:22:52.174Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: snapshot restore progress: id=2-10013-1761848572173 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:52.174Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: restored user snapshot: index=10013 +2025-10-30T18:22:52.174Z [ERROR] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: failed to get log: index=23 error="log not found" +2025-10-30T18:22:52.174Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: aborting pipeline replication: peer="{Voter server-1d0039b0-1aa8-e0fd-ad96-eb98a8b94046 1d0039b0-1aa8-e0fd-ad96-eb98a8b94046}" +2025-10-30T18:22:52.174Z [ERROR] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: failed to get log: index=23 error="log not found" +2025-10-30T18:22:52.174Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: aborting pipeline replication: peer="{Voter server-89247942-b1a8-14d7-0539-65c8f7e60807 89247942-b1a8-14d7-0539-65c8f7e60807}" +2025-10-30T18:22:52.182Z [ERROR] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: failed to get log: index=23 error="log not found" +2025-10-30T18:22:52.183Z [INFO] creating new snapshot: path=/tmp/raft2584875861/snapshots/2-10013-1761848572183.tmp +2025-10-30T18:22:52.183Z [INFO] server-1d0039b0-1aa8-e0fd-ad96-eb98a8b94046: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:52.185Z [ERROR] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: failed to get log: index=23 error="log not found" +2025-10-30T18:22:52.185Z [INFO] creating new snapshot: path=/tmp/raft4102345023/snapshots/2-10013-1761848572185.tmp +2025-10-30T18:22:52.185Z [INFO] server-89247942-b1a8-14d7-0539-65c8f7e60807: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:52.341Z [INFO] server-1d0039b0-1aa8-e0fd-ad96-eb98a8b94046: copied to local snapshot: bytes=71 +2025-10-30T18:22:52.341Z [INFO] server-89247942-b1a8-14d7-0539-65c8f7e60807: copied to local snapshot: bytes=71 +2025-10-30T18:22:52.341Z [INFO] server-1d0039b0-1aa8-e0fd-ad96-eb98a8b94046: snapshot restore progress: id=2-10013-1761848572183 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:52.341Z [INFO] server-1d0039b0-1aa8-e0fd-ad96-eb98a8b94046: Installed remote snapshot +2025-10-30T18:22:52.341Z [INFO] server-89247942-b1a8-14d7-0539-65c8f7e60807: snapshot restore progress: id=2-10013-1761848572185 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:52.341Z [INFO] server-89247942-b1a8-14d7-0539-65c8f7e60807: Installed remote snapshot +2025-10-30T18:22:52.341Z [WARN] server-1d0039b0-1aa8-e0fd-ad96-eb98a8b94046: skipping application of old log: index=22 +2025-10-30T18:22:52.341Z [WARN] server-89247942-b1a8-14d7-0539-65c8f7e60807: skipping application of old log: index=22 +2025-10-30T18:22:52.341Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: pipelining replication: peer="{Voter server-89247942-b1a8-14d7-0539-65c8f7e60807 89247942-b1a8-14d7-0539-65c8f7e60807}" +2025-10-30T18:22:52.341Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:22:52.341Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: pipelining replication: peer="{Voter server-1d0039b0-1aa8-e0fd-ad96-eb98a8b94046 1d0039b0-1aa8-e0fd-ad96-eb98a8b94046}" +2025-10-30T18:22:53.346Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:22:52.341410482 +0000 UTC m=+95.414939776 (2 nodes), 501ns from start of poll, 1.910278369s from cluster start. Timeout at 2025-10-30 18:22:53.346429447 +0000 UTC m=+96.419958741, 1.005018965s after stability +2025-10-30T18:22:53.358Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: aborting pipeline replication: peer="{Voter server-1d0039b0-1aa8-e0fd-ad96-eb98a8b94046 1d0039b0-1aa8-e0fd-ad96-eb98a8b94046}" +2025-10-30T18:22:53.358Z [INFO] server-f51b0e56-2763-1092-764b-e1f0c36cd3bc: aborting pipeline replication: peer="{Voter server-89247942-b1a8-14d7-0539-65c8f7e60807 89247942-b1a8-14d7-0539-65c8f7e60807}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false -2025-10-30T18:19:01.766Z [INFO] server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3 Address:6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3} {Suffrage:Voter ID:server-cdf886e8-c48b-e275-5aa2-6020318b4214 Address:cdf886e8-c48b-e275-5aa2-6020318b4214} {Suffrage:Voter ID:server-74fdb010-82bc-335b-07f0-7fbd3557c721 Address:74fdb010-82bc-335b-07f0-7fbd3557c721}]" -2025-10-30T18:19:01.766Z [INFO] server-cdf886e8-c48b-e275-5aa2-6020318b4214: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3 Address:6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3} {Suffrage:Voter ID:server-cdf886e8-c48b-e275-5aa2-6020318b4214 Address:cdf886e8-c48b-e275-5aa2-6020318b4214} {Suffrage:Voter ID:server-74fdb010-82bc-335b-07f0-7fbd3557c721 Address:74fdb010-82bc-335b-07f0-7fbd3557c721}]" -2025-10-30T18:19:01.766Z [INFO] server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: entering follower state: follower="Node at 6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3 [Follower]" leader-address= leader-id= -2025-10-30T18:19:01.766Z [INFO] server-cdf886e8-c48b-e275-5aa2-6020318b4214: entering follower state: follower="Node at cdf886e8-c48b-e275-5aa2-6020318b4214 [Follower]" leader-address= leader-id= -2025-10-30T18:19:01.766Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3 Address:6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3} {Suffrage:Voter ID:server-cdf886e8-c48b-e275-5aa2-6020318b4214 Address:cdf886e8-c48b-e275-5aa2-6020318b4214} {Suffrage:Voter ID:server-74fdb010-82bc-335b-07f0-7fbd3557c721 Address:74fdb010-82bc-335b-07f0-7fbd3557c721}]" -2025-10-30T18:19:01.766Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:01.766Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: entering follower state: follower="Node at 74fdb010-82bc-335b-07f0-7fbd3557c721 [Follower]" leader-address= leader-id= -2025-10-30T18:19:02.442Z [WARN] server-74fdb010-82bc-335b-07f0-7fbd3557c721: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:02.442Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: entering candidate state: node="Node at 74fdb010-82bc-335b-07f0-7fbd3557c721 [Candidate]" term=2 -2025-10-30T18:19:02.442Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: election won: term=2 tally=2 -2025-10-30T18:19:02.442Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: entering leader state: leader="Node at 74fdb010-82bc-335b-07f0-7fbd3557c721 [Leader]" -2025-10-30T18:19:02.442Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: added peer, starting replication: peer=server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3 -2025-10-30T18:19:02.442Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: added peer, starting replication: peer=server-cdf886e8-c48b-e275-5aa2-6020318b4214 -2025-10-30T18:19:02.442Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: pipelining replication: peer="{Voter server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3 6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3}" -2025-10-30T18:19:02.442Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: pipelining replication: peer="{Voter server-cdf886e8-c48b-e275-5aa2-6020318b4214 cdf886e8-c48b-e275-5aa2-6020318b4214}" -2025-10-30T18:19:03.448Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:02.442878577 +0000 UTC m=+88.721552478 (1 nodes), 676.674323ms from start of poll, 676.971386ms from cluster start. Timeout at 2025-10-30 18:19:03.447881653 +0000 UTC m=+89.726555550, 1.005003072s after stability -2025-10-30T18:19:03.448Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: starting snapshot up to: index=12 -2025-10-30T18:19:03.448Z [INFO] creating new snapshot: path=/tmp/raft2791505475/snapshots/2-12-1761848343448.tmp -2025-10-30T18:19:03.502Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: snapshot complete up to: index=12 -2025-10-30T18:19:03.503Z [INFO] creating new snapshot: path=/tmp/raft2791505475/snapshots/2-10013-1761848343503.tmp -2025-10-30T18:19:03.503Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: copied to local snapshot: bytes=71 -2025-10-30T18:19:03.503Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: snapshot restore progress: id=2-10013-1761848343503 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:03.504Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: removing all old logs from log store -2025-10-30T18:19:03.504Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: compacting logs: from=1 to=22 -2025-10-30T18:19:03.504Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: restored user snapshot: index=10013 -2025-10-30T18:19:03.504Z [ERROR] server-74fdb010-82bc-335b-07f0-7fbd3557c721: failed to get log: index=22 error="log not found" -2025-10-30T18:19:03.504Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: aborting pipeline replication: peer="{Voter server-cdf886e8-c48b-e275-5aa2-6020318b4214 cdf886e8-c48b-e275-5aa2-6020318b4214}" -2025-10-30T18:19:03.504Z [ERROR] server-74fdb010-82bc-335b-07f0-7fbd3557c721: failed to get log: index=22 error="log not found" -2025-10-30T18:19:03.504Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: aborting pipeline replication: peer="{Voter server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3 6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3}" -2025-10-30T18:19:03.509Z [ERROR] server-74fdb010-82bc-335b-07f0-7fbd3557c721: failed to get log: index=22 error="log not found" -2025-10-30T18:19:03.509Z [INFO] creating new snapshot: path=/tmp/raft1624111273/snapshots/2-10013-1761848343509.tmp -2025-10-30T18:19:03.510Z [INFO] server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:03.510Z [INFO] server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: copied to local snapshot: bytes=71 -2025-10-30T18:19:03.510Z [INFO] server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: snapshot restore progress: id=2-10013-1761848343509 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:03.510Z [INFO] server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: removing all old logs from log store -2025-10-30T18:19:03.510Z [INFO] server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: compacting logs: from=1 to=22 -2025-10-30T18:19:03.510Z [INFO] server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: Installed remote snapshot -2025-10-30T18:19:03.510Z [WARN] server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3: skipping application of old log: index=22 -2025-10-30T18:19:03.510Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: pipelining replication: peer="{Voter server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3 6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3}" -2025-10-30T18:19:03.510Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:03.511Z [ERROR] server-74fdb010-82bc-335b-07f0-7fbd3557c721: failed to get log: index=22 error="log not found" -2025-10-30T18:19:03.512Z [INFO] creating new snapshot: path=/tmp/raft2596539982/snapshots/2-10013-1761848343512.tmp -2025-10-30T18:19:03.512Z [INFO] server-cdf886e8-c48b-e275-5aa2-6020318b4214: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:03.513Z [INFO] server-cdf886e8-c48b-e275-5aa2-6020318b4214: copied to local snapshot: bytes=71 -2025-10-30T18:19:03.513Z [INFO] server-cdf886e8-c48b-e275-5aa2-6020318b4214: snapshot restore progress: id=2-10013-1761848343512 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:03.513Z [INFO] server-cdf886e8-c48b-e275-5aa2-6020318b4214: removing all old logs from log store -2025-10-30T18:19:03.513Z [INFO] server-cdf886e8-c48b-e275-5aa2-6020318b4214: compacting logs: from=1 to=22 -2025-10-30T18:19:03.513Z [INFO] server-cdf886e8-c48b-e275-5aa2-6020318b4214: Installed remote snapshot -2025-10-30T18:19:03.513Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: pipelining replication: peer="{Voter server-cdf886e8-c48b-e275-5aa2-6020318b4214 cdf886e8-c48b-e275-5aa2-6020318b4214}" -2025-10-30T18:19:04.516Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:03.510933327 +0000 UTC m=+89.789607224 (2 nodes), 661ns from start of poll, 1.745026132s from cluster start. Timeout at 2025-10-30 18:19:04.515963853 +0000 UTC m=+90.794637766, 1.005030542s after stability -2025-10-30T18:19:04.528Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: aborting pipeline replication: peer="{Voter server-cdf886e8-c48b-e275-5aa2-6020318b4214 cdf886e8-c48b-e275-5aa2-6020318b4214}" -2025-10-30T18:19:04.528Z [INFO] server-74fdb010-82bc-335b-07f0-7fbd3557c721: aborting pipeline replication: peer="{Voter server-6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3 6fab5f6a-5daa-b899-fa2f-bb8e54bf40a3}" +2025-10-30T18:22:53.464Z [INFO] server-b02a043d-dcd0-db56-fa41-3a9796ff1e56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b02a043d-dcd0-db56-fa41-3a9796ff1e56 Address:b02a043d-dcd0-db56-fa41-3a9796ff1e56} {Suffrage:Voter ID:server-980cffd7-0050-4628-9210-2025933b6bde Address:980cffd7-0050-4628-9210-2025933b6bde} {Suffrage:Voter ID:server-b5fac245-3b9a-109e-39c5-d040e0907782 Address:b5fac245-3b9a-109e-39c5-d040e0907782}]" +2025-10-30T18:22:53.464Z [INFO] server-980cffd7-0050-4628-9210-2025933b6bde: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b02a043d-dcd0-db56-fa41-3a9796ff1e56 Address:b02a043d-dcd0-db56-fa41-3a9796ff1e56} {Suffrage:Voter ID:server-980cffd7-0050-4628-9210-2025933b6bde Address:980cffd7-0050-4628-9210-2025933b6bde} {Suffrage:Voter ID:server-b5fac245-3b9a-109e-39c5-d040e0907782 Address:b5fac245-3b9a-109e-39c5-d040e0907782}]" +2025-10-30T18:22:53.464Z [INFO] server-b02a043d-dcd0-db56-fa41-3a9796ff1e56: entering follower state: follower="Node at b02a043d-dcd0-db56-fa41-3a9796ff1e56 [Follower]" leader-address= leader-id= +2025-10-30T18:22:53.464Z [INFO] server-980cffd7-0050-4628-9210-2025933b6bde: entering follower state: follower="Node at 980cffd7-0050-4628-9210-2025933b6bde [Follower]" leader-address= leader-id= +2025-10-30T18:22:53.464Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b02a043d-dcd0-db56-fa41-3a9796ff1e56 Address:b02a043d-dcd0-db56-fa41-3a9796ff1e56} {Suffrage:Voter ID:server-980cffd7-0050-4628-9210-2025933b6bde Address:980cffd7-0050-4628-9210-2025933b6bde} {Suffrage:Voter ID:server-b5fac245-3b9a-109e-39c5-d040e0907782 Address:b5fac245-3b9a-109e-39c5-d040e0907782}]" +2025-10-30T18:22:53.464Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:53.464Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: entering follower state: follower="Node at b5fac245-3b9a-109e-39c5-d040e0907782 [Follower]" leader-address= leader-id= +2025-10-30T18:22:54.139Z [WARN] server-b5fac245-3b9a-109e-39c5-d040e0907782: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:54.139Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: entering candidate state: node="Node at b5fac245-3b9a-109e-39c5-d040e0907782 [Candidate]" term=2 +2025-10-30T18:22:54.139Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: election won: term=2 tally=2 +2025-10-30T18:22:54.139Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: entering leader state: leader="Node at b5fac245-3b9a-109e-39c5-d040e0907782 [Leader]" +2025-10-30T18:22:54.139Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: added peer, starting replication: peer=server-b02a043d-dcd0-db56-fa41-3a9796ff1e56 +2025-10-30T18:22:54.139Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: added peer, starting replication: peer=server-980cffd7-0050-4628-9210-2025933b6bde +2025-10-30T18:22:54.139Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: pipelining replication: peer="{Voter server-b02a043d-dcd0-db56-fa41-3a9796ff1e56 b02a043d-dcd0-db56-fa41-3a9796ff1e56}" +2025-10-30T18:22:54.139Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: pipelining replication: peer="{Voter server-980cffd7-0050-4628-9210-2025933b6bde 980cffd7-0050-4628-9210-2025933b6bde}" +2025-10-30T18:22:55.145Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:54.139669605 +0000 UTC m=+97.213198897 (1 nodes), 675.14523ms from start of poll, 675.523156ms from cluster start. Timeout at 2025-10-30 18:22:55.14467277 +0000 UTC m=+98.218202058, 1.005003161s after stability +2025-10-30T18:22:55.146Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: starting snapshot up to: index=12 +2025-10-30T18:22:55.146Z [INFO] creating new snapshot: path=/tmp/raft827622740/snapshots/2-12-1761848575146.tmp +2025-10-30T18:22:55.255Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: snapshot complete up to: index=12 +2025-10-30T18:22:55.256Z [INFO] creating new snapshot: path=/tmp/raft827622740/snapshots/2-10013-1761848575256.tmp +2025-10-30T18:22:55.415Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: copied to local snapshot: bytes=71 +2025-10-30T18:22:55.415Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: snapshot restore progress: id=2-10013-1761848575256 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:55.415Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: removing all old logs from log store +2025-10-30T18:22:55.415Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: compacting logs: from=1 to=22 +2025-10-30T18:22:55.415Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: restored user snapshot: index=10013 +2025-10-30T18:22:55.415Z [ERROR] server-b5fac245-3b9a-109e-39c5-d040e0907782: failed to get log: index=22 error="log not found" +2025-10-30T18:22:55.416Z [ERROR] server-b5fac245-3b9a-109e-39c5-d040e0907782: failed to get log: index=22 error="log not found" +2025-10-30T18:22:55.416Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: aborting pipeline replication: peer="{Voter server-b02a043d-dcd0-db56-fa41-3a9796ff1e56 b02a043d-dcd0-db56-fa41-3a9796ff1e56}" +2025-10-30T18:22:55.416Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: aborting pipeline replication: peer="{Voter server-980cffd7-0050-4628-9210-2025933b6bde 980cffd7-0050-4628-9210-2025933b6bde}" +2025-10-30T18:22:55.424Z [ERROR] server-b5fac245-3b9a-109e-39c5-d040e0907782: failed to get log: index=22 error="log not found" +2025-10-30T18:22:55.425Z [ERROR] server-b5fac245-3b9a-109e-39c5-d040e0907782: failed to get log: index=22 error="log not found" +2025-10-30T18:22:55.425Z [INFO] creating new snapshot: path=/tmp/raft2871696990/snapshots/2-10013-1761848575425.tmp +2025-10-30T18:22:55.426Z [INFO] creating new snapshot: path=/tmp/raft375508045/snapshots/2-10013-1761848575426.tmp +2025-10-30T18:22:55.462Z [INFO] server-b02a043d-dcd0-db56-fa41-3a9796ff1e56: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:55.462Z [INFO] server-980cffd7-0050-4628-9210-2025933b6bde: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:55.587Z [INFO] server-b02a043d-dcd0-db56-fa41-3a9796ff1e56: copied to local snapshot: bytes=71 +2025-10-30T18:22:55.587Z [INFO] server-980cffd7-0050-4628-9210-2025933b6bde: copied to local snapshot: bytes=71 +2025-10-30T18:22:55.587Z [INFO] server-b02a043d-dcd0-db56-fa41-3a9796ff1e56: snapshot restore progress: id=2-10013-1761848575426 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:55.587Z [INFO] server-b02a043d-dcd0-db56-fa41-3a9796ff1e56: removing all old logs from log store +2025-10-30T18:22:55.587Z [INFO] server-980cffd7-0050-4628-9210-2025933b6bde: snapshot restore progress: id=2-10013-1761848575425 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:22:55.587Z [INFO] server-b02a043d-dcd0-db56-fa41-3a9796ff1e56: compacting logs: from=1 to=22 +2025-10-30T18:22:55.587Z [INFO] server-980cffd7-0050-4628-9210-2025933b6bde: removing all old logs from log store +2025-10-30T18:22:55.587Z [INFO] server-b02a043d-dcd0-db56-fa41-3a9796ff1e56: Installed remote snapshot +2025-10-30T18:22:55.587Z [INFO] server-980cffd7-0050-4628-9210-2025933b6bde: compacting logs: from=1 to=22 +2025-10-30T18:22:55.587Z [INFO] server-980cffd7-0050-4628-9210-2025933b6bde: Installed remote snapshot +2025-10-30T18:22:55.587Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: pipelining replication: peer="{Voter server-b02a043d-dcd0-db56-fa41-3a9796ff1e56 b02a043d-dcd0-db56-fa41-3a9796ff1e56}" +2025-10-30T18:22:55.587Z [WARN] server-980cffd7-0050-4628-9210-2025933b6bde: skipping application of old log: index=10013 +2025-10-30T18:22:55.587Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: pipelining replication: peer="{Voter server-980cffd7-0050-4628-9210-2025933b6bde 980cffd7-0050-4628-9210-2025933b6bde}" +2025-10-30T18:22:55.587Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:22:56.592Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:22:55.587863239 +0000 UTC m=+98.661392545 (2 nodes), 1.287µs from start of poll, 2.123716804s from cluster start. Timeout at 2025-10-30 18:22:56.592877721 +0000 UTC m=+99.666407134, 1.005014589s after stability +2025-10-30T18:22:56.604Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: aborting pipeline replication: peer="{Voter server-980cffd7-0050-4628-9210-2025933b6bde 980cffd7-0050-4628-9210-2025933b6bde}" +2025-10-30T18:22:56.604Z [INFO] server-b5fac245-3b9a-109e-39c5-d040e0907782: aborting pipeline replication: peer="{Voter server-b02a043d-dcd0-db56-fa41-3a9796ff1e56 b02a043d-dcd0-db56-fa41-3a9796ff1e56}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true -2025-10-30T18:19:04.529Z [INFO] server-a12d35d8-4926-c326-572f-aec44863f599: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a12d35d8-4926-c326-572f-aec44863f599 Address:a12d35d8-4926-c326-572f-aec44863f599} {Suffrage:Voter ID:server-38674a90-e3e6-97b6-9933-c36423fb6c83 Address:38674a90-e3e6-97b6-9933-c36423fb6c83} {Suffrage:Voter ID:server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3 Address:0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3}]" -2025-10-30T18:19:04.529Z [INFO] server-a12d35d8-4926-c326-572f-aec44863f599: entering follower state: follower="Node at a12d35d8-4926-c326-572f-aec44863f599 [Follower]" leader-address= leader-id= -2025-10-30T18:19:04.529Z [INFO] server-38674a90-e3e6-97b6-9933-c36423fb6c83: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a12d35d8-4926-c326-572f-aec44863f599 Address:a12d35d8-4926-c326-572f-aec44863f599} {Suffrage:Voter ID:server-38674a90-e3e6-97b6-9933-c36423fb6c83 Address:38674a90-e3e6-97b6-9933-c36423fb6c83} {Suffrage:Voter ID:server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3 Address:0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3}]" -2025-10-30T18:19:04.529Z [INFO] server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a12d35d8-4926-c326-572f-aec44863f599 Address:a12d35d8-4926-c326-572f-aec44863f599} {Suffrage:Voter ID:server-38674a90-e3e6-97b6-9933-c36423fb6c83 Address:38674a90-e3e6-97b6-9933-c36423fb6c83} {Suffrage:Voter ID:server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3 Address:0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3}]" -2025-10-30T18:19:04.529Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:04.529Z [INFO] server-38674a90-e3e6-97b6-9933-c36423fb6c83: entering follower state: follower="Node at 38674a90-e3e6-97b6-9933-c36423fb6c83 [Follower]" leader-address= leader-id= -2025-10-30T18:19:04.529Z [INFO] server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: entering follower state: follower="Node at 0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3 [Follower]" leader-address= leader-id= -2025-10-30T18:19:05.087Z [WARN] server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:05.087Z [INFO] server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: entering candidate state: node="Node at 0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3 [Candidate]" term=2 -2025-10-30T18:19:05.087Z [INFO] server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: election won: term=2 tally=2 -2025-10-30T18:19:05.087Z [INFO] server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: entering leader state: leader="Node at 0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3 [Leader]" -2025-10-30T18:19:05.087Z [INFO] server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: added peer, starting replication: peer=server-a12d35d8-4926-c326-572f-aec44863f599 -2025-10-30T18:19:05.087Z [INFO] server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: added peer, starting replication: peer=server-38674a90-e3e6-97b6-9933-c36423fb6c83 -2025-10-30T18:19:05.088Z [INFO] server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: pipelining replication: peer="{Voter server-a12d35d8-4926-c326-572f-aec44863f599 a12d35d8-4926-c326-572f-aec44863f599}" -2025-10-30T18:19:05.088Z [INFO] server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: pipelining replication: peer="{Voter server-38674a90-e3e6-97b6-9933-c36423fb6c83 38674a90-e3e6-97b6-9933-c36423fb6c83}" -2025-10-30T18:19:06.093Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:05.087966859 +0000 UTC m=+91.366640756 (1 nodes), 558.393062ms from start of poll, 558.616491ms from cluster start. Timeout at 2025-10-30 18:19:06.09296983 +0000 UTC m=+92.371643737, 1.005002981s after stability -2025-10-30T18:19:06.094Z [INFO] server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: starting snapshot up to: index=12 -2025-10-30T18:19:06.094Z [INFO] creating new snapshot: path=/tmp/raft142849237/snapshots/2-12-1761848346094.tmp -2025-10-30T18:19:06.095Z [INFO] server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: snapshot complete up to: index=12 -2025-10-30T18:19:06.096Z [INFO] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5935a42d-564f-f451-1c45-3b1ead50c3f3 Address:5935a42d-564f-f451-1c45-3b1ead50c3f3} {Suffrage:Voter ID:server-54721090-29f0-cb5c-2bed-832f5d577bb8 Address:54721090-29f0-cb5c-2bed-832f5d577bb8} {Suffrage:Voter ID:server-3ea4006e-45c7-de28-d0a5-7e280e66c954 Address:3ea4006e-45c7-de28-d0a5-7e280e66c954}]" -2025-10-30T18:19:06.096Z [INFO] server-54721090-29f0-cb5c-2bed-832f5d577bb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5935a42d-564f-f451-1c45-3b1ead50c3f3 Address:5935a42d-564f-f451-1c45-3b1ead50c3f3} {Suffrage:Voter ID:server-54721090-29f0-cb5c-2bed-832f5d577bb8 Address:54721090-29f0-cb5c-2bed-832f5d577bb8} {Suffrage:Voter ID:server-3ea4006e-45c7-de28-d0a5-7e280e66c954 Address:3ea4006e-45c7-de28-d0a5-7e280e66c954}]" -2025-10-30T18:19:06.096Z [INFO] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: entering follower state: follower="Node at 5935a42d-564f-f451-1c45-3b1ead50c3f3 [Follower]" leader-address= leader-id= -2025-10-30T18:19:06.096Z [INFO] server-54721090-29f0-cb5c-2bed-832f5d577bb8: entering follower state: follower="Node at 54721090-29f0-cb5c-2bed-832f5d577bb8 [Follower]" leader-address= leader-id= -2025-10-30T18:19:06.096Z [INFO] server-3ea4006e-45c7-de28-d0a5-7e280e66c954: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5935a42d-564f-f451-1c45-3b1ead50c3f3 Address:5935a42d-564f-f451-1c45-3b1ead50c3f3} {Suffrage:Voter ID:server-54721090-29f0-cb5c-2bed-832f5d577bb8 Address:54721090-29f0-cb5c-2bed-832f5d577bb8} {Suffrage:Voter ID:server-3ea4006e-45c7-de28-d0a5-7e280e66c954 Address:3ea4006e-45c7-de28-d0a5-7e280e66c954}]" -2025-10-30T18:19:06.096Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:06.096Z [INFO] server-3ea4006e-45c7-de28-d0a5-7e280e66c954: entering follower state: follower="Node at 3ea4006e-45c7-de28-d0a5-7e280e66c954 [Follower]" leader-address= leader-id= -2025-10-30T18:19:06.624Z [WARN] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:06.624Z [INFO] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: entering candidate state: node="Node at 5935a42d-564f-f451-1c45-3b1ead50c3f3 [Candidate]" term=2 -2025-10-30T18:19:06.624Z [INFO] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: election won: term=2 tally=2 -2025-10-30T18:19:06.624Z [INFO] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: entering leader state: leader="Node at 5935a42d-564f-f451-1c45-3b1ead50c3f3 [Leader]" -2025-10-30T18:19:06.624Z [INFO] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: added peer, starting replication: peer=server-54721090-29f0-cb5c-2bed-832f5d577bb8 -2025-10-30T18:19:06.624Z [INFO] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: added peer, starting replication: peer=server-3ea4006e-45c7-de28-d0a5-7e280e66c954 -2025-10-30T18:19:06.624Z [INFO] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: pipelining replication: peer="{Voter server-3ea4006e-45c7-de28-d0a5-7e280e66c954 3ea4006e-45c7-de28-d0a5-7e280e66c954}" -2025-10-30T18:19:06.624Z [INFO] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: pipelining replication: peer="{Voter server-54721090-29f0-cb5c-2bed-832f5d577bb8 54721090-29f0-cb5c-2bed-832f5d577bb8}" -2025-10-30T18:19:07.630Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:06.624781466 +0000 UTC m=+92.903455370 (1 nodes), 528.547614ms from start of poll, 528.772642ms from cluster start. Timeout at 2025-10-30 18:19:07.629782667 +0000 UTC m=+93.908456565, 1.005001195s after stability -2025-10-30T18:19:07.630Z [INFO] creating new snapshot: path=/tmp/raft914815851/snapshots/2-10013-1761848347630.tmp -2025-10-30T18:19:07.631Z [INFO] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: copied to local snapshot: bytes=71 -2025-10-30T18:19:07.631Z [INFO] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: snapshot restore progress: id=2-10013-1761848347630 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:07.631Z [INFO] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: restored user snapshot: index=10013 -2025-10-30T18:19:07.631Z [ERROR] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: failed to get log: index=3 error="log not found" -2025-10-30T18:19:07.631Z [ERROR] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: failed to get log: index=3 error="log not found" -2025-10-30T18:19:07.631Z [INFO] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: aborting pipeline replication: peer="{Voter server-54721090-29f0-cb5c-2bed-832f5d577bb8 54721090-29f0-cb5c-2bed-832f5d577bb8}" -2025-10-30T18:19:07.631Z [INFO] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: aborting pipeline replication: peer="{Voter server-3ea4006e-45c7-de28-d0a5-7e280e66c954 3ea4006e-45c7-de28-d0a5-7e280e66c954}" -2025-10-30T18:19:07.640Z [ERROR] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: failed to get log: index=3 error="log not found" -2025-10-30T18:19:07.640Z [INFO] creating new snapshot: path=/tmp/raft3752347853/snapshots/2-10013-1761848347640.tmp -2025-10-30T18:19:07.641Z [INFO] server-3ea4006e-45c7-de28-d0a5-7e280e66c954: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:07.641Z [ERROR] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: failed to get log: index=3 error="log not found" -2025-10-30T18:19:07.641Z [INFO] creating new snapshot: path=/tmp/raft3719230373/snapshots/2-10013-1761848347641.tmp -2025-10-30T18:19:07.641Z [INFO] server-3ea4006e-45c7-de28-d0a5-7e280e66c954: copied to local snapshot: bytes=71 -2025-10-30T18:19:07.641Z [INFO] server-3ea4006e-45c7-de28-d0a5-7e280e66c954: snapshot restore progress: id=2-10013-1761848347640 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:07.641Z [INFO] server-3ea4006e-45c7-de28-d0a5-7e280e66c954: Installed remote snapshot -2025-10-30T18:19:07.641Z [INFO] server-54721090-29f0-cb5c-2bed-832f5d577bb8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:07.641Z [INFO] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: pipelining replication: peer="{Voter server-3ea4006e-45c7-de28-d0a5-7e280e66c954 3ea4006e-45c7-de28-d0a5-7e280e66c954}" -2025-10-30T18:19:07.641Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:07.641Z [INFO] server-54721090-29f0-cb5c-2bed-832f5d577bb8: copied to local snapshot: bytes=71 -2025-10-30T18:19:07.642Z [INFO] server-54721090-29f0-cb5c-2bed-832f5d577bb8: snapshot restore progress: id=2-10013-1761848347641 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:07.642Z [INFO] server-54721090-29f0-cb5c-2bed-832f5d577bb8: Installed remote snapshot -2025-10-30T18:19:07.642Z [INFO] server-5935a42d-564f-f451-1c45-3b1ead50c3f3: pipelining replication: peer="{Voter server-54721090-29f0-cb5c-2bed-832f5d577bb8 54721090-29f0-cb5c-2bed-832f5d577bb8}" -2025-10-30T18:19:08.647Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:07.641925303 +0000 UTC m=+93.920599202 (2 nodes), 979ns from start of poll, 1.545916474s from cluster start. Timeout at 2025-10-30 18:19:08.646949102 +0000 UTC m=+94.925622999, 1.005023797s after stability -2025-10-30T18:19:08.659Z [INFO] server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: aborting pipeline replication: peer="{Voter server-a12d35d8-4926-c326-572f-aec44863f599 a12d35d8-4926-c326-572f-aec44863f599}" -2025-10-30T18:19:08.659Z [INFO] server-0b3cbf7e-9282-4dfe-e9fc-64c8a9b6d3b3: aborting pipeline replication: peer="{Voter server-38674a90-e3e6-97b6-9933-c36423fb6c83 38674a90-e3e6-97b6-9933-c36423fb6c83}" +2025-10-30T18:22:56.671Z [INFO] server-41cc291e-a8cf-3884-3e45-df08254f74c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41cc291e-a8cf-3884-3e45-df08254f74c3 Address:41cc291e-a8cf-3884-3e45-df08254f74c3} {Suffrage:Voter ID:server-6c54f694-21c7-fdd3-8a51-e82e741b348b Address:6c54f694-21c7-fdd3-8a51-e82e741b348b} {Suffrage:Voter ID:server-ff511afa-7c61-c1c4-0fd5-3b27a5933edc Address:ff511afa-7c61-c1c4-0fd5-3b27a5933edc}]" +2025-10-30T18:22:56.671Z [INFO] server-6c54f694-21c7-fdd3-8a51-e82e741b348b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41cc291e-a8cf-3884-3e45-df08254f74c3 Address:41cc291e-a8cf-3884-3e45-df08254f74c3} {Suffrage:Voter ID:server-6c54f694-21c7-fdd3-8a51-e82e741b348b Address:6c54f694-21c7-fdd3-8a51-e82e741b348b} {Suffrage:Voter ID:server-ff511afa-7c61-c1c4-0fd5-3b27a5933edc Address:ff511afa-7c61-c1c4-0fd5-3b27a5933edc}]" +2025-10-30T18:22:56.671Z [INFO] server-41cc291e-a8cf-3884-3e45-df08254f74c3: entering follower state: follower="Node at 41cc291e-a8cf-3884-3e45-df08254f74c3 [Follower]" leader-address= leader-id= +2025-10-30T18:22:56.671Z [INFO] server-ff511afa-7c61-c1c4-0fd5-3b27a5933edc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41cc291e-a8cf-3884-3e45-df08254f74c3 Address:41cc291e-a8cf-3884-3e45-df08254f74c3} {Suffrage:Voter ID:server-6c54f694-21c7-fdd3-8a51-e82e741b348b Address:6c54f694-21c7-fdd3-8a51-e82e741b348b} {Suffrage:Voter ID:server-ff511afa-7c61-c1c4-0fd5-3b27a5933edc Address:ff511afa-7c61-c1c4-0fd5-3b27a5933edc}]" +2025-10-30T18:22:56.671Z [INFO] server-6c54f694-21c7-fdd3-8a51-e82e741b348b: entering follower state: follower="Node at 6c54f694-21c7-fdd3-8a51-e82e741b348b [Follower]" leader-address= leader-id= +2025-10-30T18:22:56.671Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:56.671Z [INFO] server-ff511afa-7c61-c1c4-0fd5-3b27a5933edc: entering follower state: follower="Node at ff511afa-7c61-c1c4-0fd5-3b27a5933edc [Follower]" leader-address= leader-id= +2025-10-30T18:22:57.514Z [WARN] server-41cc291e-a8cf-3884-3e45-df08254f74c3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:57.514Z [INFO] server-41cc291e-a8cf-3884-3e45-df08254f74c3: entering candidate state: node="Node at 41cc291e-a8cf-3884-3e45-df08254f74c3 [Candidate]" term=2 +2025-10-30T18:22:57.514Z [INFO] server-41cc291e-a8cf-3884-3e45-df08254f74c3: election won: term=2 tally=2 +2025-10-30T18:22:57.514Z [INFO] server-41cc291e-a8cf-3884-3e45-df08254f74c3: entering leader state: leader="Node at 41cc291e-a8cf-3884-3e45-df08254f74c3 [Leader]" +2025-10-30T18:22:57.514Z [INFO] server-41cc291e-a8cf-3884-3e45-df08254f74c3: added peer, starting replication: peer=server-6c54f694-21c7-fdd3-8a51-e82e741b348b +2025-10-30T18:22:57.514Z [INFO] server-41cc291e-a8cf-3884-3e45-df08254f74c3: added peer, starting replication: peer=server-ff511afa-7c61-c1c4-0fd5-3b27a5933edc +2025-10-30T18:22:57.514Z [INFO] server-41cc291e-a8cf-3884-3e45-df08254f74c3: pipelining replication: peer="{Voter server-ff511afa-7c61-c1c4-0fd5-3b27a5933edc ff511afa-7c61-c1c4-0fd5-3b27a5933edc}" +2025-10-30T18:22:57.514Z [INFO] server-41cc291e-a8cf-3884-3e45-df08254f74c3: pipelining replication: peer="{Voter server-6c54f694-21c7-fdd3-8a51-e82e741b348b 6c54f694-21c7-fdd3-8a51-e82e741b348b}" +2025-10-30T18:22:58.520Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:57.514520983 +0000 UTC m=+100.588050273 (1 nodes), 842.981087ms from start of poll, 843.288006ms from cluster start. Timeout at 2025-10-30 18:22:58.519526726 +0000 UTC m=+101.593056052, 1.005005779s after stability +2025-10-30T18:22:58.520Z [INFO] server-41cc291e-a8cf-3884-3e45-df08254f74c3: starting snapshot up to: index=12 +2025-10-30T18:22:58.520Z [INFO] creating new snapshot: path=/tmp/raft112818075/snapshots/2-12-1761848578520.tmp +2025-10-30T18:22:58.562Z [INFO] server-41cc291e-a8cf-3884-3e45-df08254f74c3: snapshot complete up to: index=12 +2025-10-30T18:22:58.563Z [INFO] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d Address:b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d} {Suffrage:Voter ID:server-06b3c7b5-d73c-8a65-0e68-fa74fc32841a Address:06b3c7b5-d73c-8a65-0e68-fa74fc32841a} {Suffrage:Voter ID:server-11152da9-ecd4-60ae-65a5-b0b01cf9480a Address:11152da9-ecd4-60ae-65a5-b0b01cf9480a}]" +2025-10-30T18:22:58.563Z [INFO] server-06b3c7b5-d73c-8a65-0e68-fa74fc32841a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d Address:b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d} {Suffrage:Voter ID:server-06b3c7b5-d73c-8a65-0e68-fa74fc32841a Address:06b3c7b5-d73c-8a65-0e68-fa74fc32841a} {Suffrage:Voter ID:server-11152da9-ecd4-60ae-65a5-b0b01cf9480a Address:11152da9-ecd4-60ae-65a5-b0b01cf9480a}]" +2025-10-30T18:22:58.563Z [INFO] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: entering follower state: follower="Node at b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d [Follower]" leader-address= leader-id= +2025-10-30T18:22:58.563Z [INFO] server-06b3c7b5-d73c-8a65-0e68-fa74fc32841a: entering follower state: follower="Node at 06b3c7b5-d73c-8a65-0e68-fa74fc32841a [Follower]" leader-address= leader-id= +2025-10-30T18:22:58.563Z [INFO] server-11152da9-ecd4-60ae-65a5-b0b01cf9480a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d Address:b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d} {Suffrage:Voter ID:server-06b3c7b5-d73c-8a65-0e68-fa74fc32841a Address:06b3c7b5-d73c-8a65-0e68-fa74fc32841a} {Suffrage:Voter ID:server-11152da9-ecd4-60ae-65a5-b0b01cf9480a Address:11152da9-ecd4-60ae-65a5-b0b01cf9480a}]" +2025-10-30T18:22:58.563Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:22:58.563Z [INFO] server-11152da9-ecd4-60ae-65a5-b0b01cf9480a: entering follower state: follower="Node at 11152da9-ecd4-60ae-65a5-b0b01cf9480a [Follower]" leader-address= leader-id= +2025-10-30T18:22:59.063Z [WARN] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:22:59.063Z [INFO] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: entering candidate state: node="Node at b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d [Candidate]" term=2 +2025-10-30T18:22:59.063Z [INFO] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: election won: term=2 tally=2 +2025-10-30T18:22:59.063Z [INFO] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: entering leader state: leader="Node at b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d [Leader]" +2025-10-30T18:22:59.063Z [INFO] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: added peer, starting replication: peer=server-06b3c7b5-d73c-8a65-0e68-fa74fc32841a +2025-10-30T18:22:59.063Z [INFO] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: added peer, starting replication: peer=server-11152da9-ecd4-60ae-65a5-b0b01cf9480a +2025-10-30T18:22:59.064Z [INFO] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: pipelining replication: peer="{Voter server-06b3c7b5-d73c-8a65-0e68-fa74fc32841a 06b3c7b5-d73c-8a65-0e68-fa74fc32841a}" +2025-10-30T18:22:59.064Z [INFO] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: pipelining replication: peer="{Voter server-11152da9-ecd4-60ae-65a5-b0b01cf9480a 11152da9-ecd4-60ae-65a5-b0b01cf9480a}" +2025-10-30T18:23:00.069Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:22:59.063925269 +0000 UTC m=+102.137454558 (1 nodes), 500.670805ms from start of poll, 500.952569ms from cluster start. Timeout at 2025-10-30 18:23:00.068929304 +0000 UTC m=+103.142458598, 1.00500404s after stability +2025-10-30T18:23:00.095Z [INFO] creating new snapshot: path=/tmp/raft3379969300/snapshots/2-10013-1761848580095.tmp +2025-10-30T18:23:00.208Z [INFO] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: copied to local snapshot: bytes=71 +2025-10-30T18:23:00.209Z [INFO] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: snapshot restore progress: id=2-10013-1761848580095 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:23:00.209Z [INFO] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: restored user snapshot: index=10013 +2025-10-30T18:23:00.209Z [ERROR] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: failed to get log: index=3 error="log not found" +2025-10-30T18:23:00.209Z [INFO] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: aborting pipeline replication: peer="{Voter server-06b3c7b5-d73c-8a65-0e68-fa74fc32841a 06b3c7b5-d73c-8a65-0e68-fa74fc32841a}" +2025-10-30T18:23:00.209Z [ERROR] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: failed to get log: index=3 error="log not found" +2025-10-30T18:23:00.209Z [INFO] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: aborting pipeline replication: peer="{Voter server-11152da9-ecd4-60ae-65a5-b0b01cf9480a 11152da9-ecd4-60ae-65a5-b0b01cf9480a}" +2025-10-30T18:23:00.218Z [ERROR] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: failed to get log: index=3 error="log not found" +2025-10-30T18:23:00.218Z [INFO] creating new snapshot: path=/tmp/raft126151848/snapshots/2-10013-1761848580218.tmp +2025-10-30T18:23:00.218Z [ERROR] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: failed to get log: index=3 error="log not found" +2025-10-30T18:23:00.218Z [INFO] server-06b3c7b5-d73c-8a65-0e68-fa74fc32841a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:23:00.219Z [INFO] creating new snapshot: path=/tmp/raft2314199795/snapshots/2-10013-1761848580219.tmp +2025-10-30T18:23:00.219Z [INFO] server-11152da9-ecd4-60ae-65a5-b0b01cf9480a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:23:00.300Z [INFO] server-06b3c7b5-d73c-8a65-0e68-fa74fc32841a: copied to local snapshot: bytes=71 +2025-10-30T18:23:00.300Z [INFO] server-11152da9-ecd4-60ae-65a5-b0b01cf9480a: copied to local snapshot: bytes=71 +2025-10-30T18:23:00.300Z [INFO] server-06b3c7b5-d73c-8a65-0e68-fa74fc32841a: snapshot restore progress: id=2-10013-1761848580218 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:23:00.300Z [INFO] server-11152da9-ecd4-60ae-65a5-b0b01cf9480a: snapshot restore progress: id=2-10013-1761848580219 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:23:00.300Z [INFO] server-11152da9-ecd4-60ae-65a5-b0b01cf9480a: Installed remote snapshot +2025-10-30T18:23:00.300Z [INFO] server-06b3c7b5-d73c-8a65-0e68-fa74fc32841a: Installed remote snapshot +2025-10-30T18:23:00.300Z [INFO] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: pipelining replication: peer="{Voter server-11152da9-ecd4-60ae-65a5-b0b01cf9480a 11152da9-ecd4-60ae-65a5-b0b01cf9480a}" +2025-10-30T18:23:00.300Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:00.300Z [INFO] server-b84ae0b4-55a5-6b0f-fda9-1daca56b4e5d: pipelining replication: peer="{Voter server-06b3c7b5-d73c-8a65-0e68-fa74fc32841a 06b3c7b5-d73c-8a65-0e68-fa74fc32841a}" +2025-10-30T18:23:01.305Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:00.300704584 +0000 UTC m=+103.374233873 (2 nodes), 1.403µs from start of poll, 1.737731884s from cluster start. Timeout at 2025-10-30 18:23:01.305708801 +0000 UTC m=+104.379238091, 1.005004218s after stability +2025-10-30T18:23:01.317Z [INFO] server-41cc291e-a8cf-3884-3e45-df08254f74c3: aborting pipeline replication: peer="{Voter server-ff511afa-7c61-c1c4-0fd5-3b27a5933edc ff511afa-7c61-c1c4-0fd5-3b27a5933edc}" +2025-10-30T18:23:01.317Z [INFO] server-41cc291e-a8cf-3884-3e45-df08254f74c3: aborting pipeline replication: peer="{Voter server-6c54f694-21c7-fdd3-8a51-e82e741b348b 6c54f694-21c7-fdd3-8a51-e82e741b348b}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true -2025-10-30T18:19:08.660Z [INFO] server-c275bf99-ff0d-a74b-3547-f4d8386c899d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c275bf99-ff0d-a74b-3547-f4d8386c899d Address:c275bf99-ff0d-a74b-3547-f4d8386c899d} {Suffrage:Voter ID:server-fd78e202-034d-df7b-0446-3c02197f42a1 Address:fd78e202-034d-df7b-0446-3c02197f42a1} {Suffrage:Voter ID:server-c495c0bd-a40e-b036-c40c-82fa0659a361 Address:c495c0bd-a40e-b036-c40c-82fa0659a361}]" -2025-10-30T18:19:08.660Z [INFO] server-fd78e202-034d-df7b-0446-3c02197f42a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c275bf99-ff0d-a74b-3547-f4d8386c899d Address:c275bf99-ff0d-a74b-3547-f4d8386c899d} {Suffrage:Voter ID:server-fd78e202-034d-df7b-0446-3c02197f42a1 Address:fd78e202-034d-df7b-0446-3c02197f42a1} {Suffrage:Voter ID:server-c495c0bd-a40e-b036-c40c-82fa0659a361 Address:c495c0bd-a40e-b036-c40c-82fa0659a361}]" -2025-10-30T18:19:08.660Z [INFO] server-c275bf99-ff0d-a74b-3547-f4d8386c899d: entering follower state: follower="Node at c275bf99-ff0d-a74b-3547-f4d8386c899d [Follower]" leader-address= leader-id= -2025-10-30T18:19:08.660Z [INFO] server-fd78e202-034d-df7b-0446-3c02197f42a1: entering follower state: follower="Node at fd78e202-034d-df7b-0446-3c02197f42a1 [Follower]" leader-address= leader-id= -2025-10-30T18:19:08.660Z [INFO] server-c495c0bd-a40e-b036-c40c-82fa0659a361: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c275bf99-ff0d-a74b-3547-f4d8386c899d Address:c275bf99-ff0d-a74b-3547-f4d8386c899d} {Suffrage:Voter ID:server-fd78e202-034d-df7b-0446-3c02197f42a1 Address:fd78e202-034d-df7b-0446-3c02197f42a1} {Suffrage:Voter ID:server-c495c0bd-a40e-b036-c40c-82fa0659a361 Address:c495c0bd-a40e-b036-c40c-82fa0659a361}]" -2025-10-30T18:19:08.660Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:08.660Z [INFO] server-c495c0bd-a40e-b036-c40c-82fa0659a361: entering follower state: follower="Node at c495c0bd-a40e-b036-c40c-82fa0659a361 [Follower]" leader-address= leader-id= -2025-10-30T18:19:09.241Z [WARN] server-fd78e202-034d-df7b-0446-3c02197f42a1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:09.241Z [INFO] server-fd78e202-034d-df7b-0446-3c02197f42a1: entering candidate state: node="Node at fd78e202-034d-df7b-0446-3c02197f42a1 [Candidate]" term=2 -2025-10-30T18:19:09.241Z [INFO] server-fd78e202-034d-df7b-0446-3c02197f42a1: election won: term=2 tally=2 -2025-10-30T18:19:09.241Z [INFO] server-fd78e202-034d-df7b-0446-3c02197f42a1: entering leader state: leader="Node at fd78e202-034d-df7b-0446-3c02197f42a1 [Leader]" -2025-10-30T18:19:09.241Z [INFO] server-fd78e202-034d-df7b-0446-3c02197f42a1: added peer, starting replication: peer=server-c275bf99-ff0d-a74b-3547-f4d8386c899d -2025-10-30T18:19:09.241Z [INFO] server-fd78e202-034d-df7b-0446-3c02197f42a1: added peer, starting replication: peer=server-c495c0bd-a40e-b036-c40c-82fa0659a361 -2025-10-30T18:19:09.241Z [INFO] server-fd78e202-034d-df7b-0446-3c02197f42a1: pipelining replication: peer="{Voter server-c495c0bd-a40e-b036-c40c-82fa0659a361 c495c0bd-a40e-b036-c40c-82fa0659a361}" -2025-10-30T18:19:09.242Z [INFO] server-fd78e202-034d-df7b-0446-3c02197f42a1: pipelining replication: peer="{Voter server-c275bf99-ff0d-a74b-3547-f4d8386c899d c275bf99-ff0d-a74b-3547-f4d8386c899d}" -2025-10-30T18:19:10.247Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:09.241952763 +0000 UTC m=+95.520626660 (1 nodes), 581.076255ms from start of poll, 581.318129ms from cluster start. Timeout at 2025-10-30 18:19:10.246954926 +0000 UTC m=+96.525628823, 1.005002163s after stability -2025-10-30T18:19:10.248Z [INFO] server-fd78e202-034d-df7b-0446-3c02197f42a1: starting snapshot up to: index=12 -2025-10-30T18:19:10.248Z [INFO] creating new snapshot: path=/tmp/raft265069148/snapshots/2-12-1761848350248.tmp -2025-10-30T18:19:10.283Z [INFO] server-fd78e202-034d-df7b-0446-3c02197f42a1: snapshot complete up to: index=12 -2025-10-30T18:19:10.284Z [INFO] server-d4487d22-db28-ad97-459e-bdc76e06a8fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d4487d22-db28-ad97-459e-bdc76e06a8fc Address:d4487d22-db28-ad97-459e-bdc76e06a8fc} {Suffrage:Voter ID:server-7961c63f-0644-b43d-47d3-1c6aed691c58 Address:7961c63f-0644-b43d-47d3-1c6aed691c58} {Suffrage:Voter ID:server-d97fd847-4517-e92e-2b4c-3b09900a54e0 Address:d97fd847-4517-e92e-2b4c-3b09900a54e0}]" -2025-10-30T18:19:10.284Z [INFO] server-d4487d22-db28-ad97-459e-bdc76e06a8fc: entering follower state: follower="Node at d4487d22-db28-ad97-459e-bdc76e06a8fc [Follower]" leader-address= leader-id= -2025-10-30T18:19:10.284Z [INFO] server-7961c63f-0644-b43d-47d3-1c6aed691c58: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d4487d22-db28-ad97-459e-bdc76e06a8fc Address:d4487d22-db28-ad97-459e-bdc76e06a8fc} {Suffrage:Voter ID:server-7961c63f-0644-b43d-47d3-1c6aed691c58 Address:7961c63f-0644-b43d-47d3-1c6aed691c58} {Suffrage:Voter ID:server-d97fd847-4517-e92e-2b4c-3b09900a54e0 Address:d97fd847-4517-e92e-2b4c-3b09900a54e0}]" -2025-10-30T18:19:10.284Z [INFO] server-7961c63f-0644-b43d-47d3-1c6aed691c58: entering follower state: follower="Node at 7961c63f-0644-b43d-47d3-1c6aed691c58 [Follower]" leader-address= leader-id= -2025-10-30T18:19:10.284Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d4487d22-db28-ad97-459e-bdc76e06a8fc Address:d4487d22-db28-ad97-459e-bdc76e06a8fc} {Suffrage:Voter ID:server-7961c63f-0644-b43d-47d3-1c6aed691c58 Address:7961c63f-0644-b43d-47d3-1c6aed691c58} {Suffrage:Voter ID:server-d97fd847-4517-e92e-2b4c-3b09900a54e0 Address:d97fd847-4517-e92e-2b4c-3b09900a54e0}]" -2025-10-30T18:19:10.284Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:10.284Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: entering follower state: follower="Node at d97fd847-4517-e92e-2b4c-3b09900a54e0 [Follower]" leader-address= leader-id= -2025-10-30T18:19:10.846Z [WARN] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:10.846Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: entering candidate state: node="Node at d97fd847-4517-e92e-2b4c-3b09900a54e0 [Candidate]" term=2 -2025-10-30T18:19:10.846Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: election won: term=2 tally=2 -2025-10-30T18:19:10.846Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: entering leader state: leader="Node at d97fd847-4517-e92e-2b4c-3b09900a54e0 [Leader]" -2025-10-30T18:19:10.846Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: added peer, starting replication: peer=server-d4487d22-db28-ad97-459e-bdc76e06a8fc -2025-10-30T18:19:10.846Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: added peer, starting replication: peer=server-7961c63f-0644-b43d-47d3-1c6aed691c58 -2025-10-30T18:19:10.846Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: pipelining replication: peer="{Voter server-7961c63f-0644-b43d-47d3-1c6aed691c58 7961c63f-0644-b43d-47d3-1c6aed691c58}" -2025-10-30T18:19:10.846Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: pipelining replication: peer="{Voter server-d4487d22-db28-ad97-459e-bdc76e06a8fc d4487d22-db28-ad97-459e-bdc76e06a8fc}" -2025-10-30T18:19:11.852Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:10.84659963 +0000 UTC m=+97.125273527 (1 nodes), 561.662825ms from start of poll, 561.989686ms from cluster start. Timeout at 2025-10-30 18:19:11.851601919 +0000 UTC m=+98.130275815, 1.005002288s after stability -2025-10-30T18:19:11.852Z [INFO] creating new snapshot: path=/tmp/raft3705812841/snapshots/2-10013-1761848351852.tmp -2025-10-30T18:19:11.974Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: copied to local snapshot: bytes=71 -2025-10-30T18:19:11.974Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: snapshot restore progress: id=2-10013-1761848351852 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:11.974Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: removing all old logs from log store -2025-10-30T18:19:11.974Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: compacting logs: from=1 to=2 -2025-10-30T18:19:11.974Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: restored user snapshot: index=10013 -2025-10-30T18:19:11.974Z [ERROR] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: failed to get log: index=2 error="log not found" -2025-10-30T18:19:11.974Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: aborting pipeline replication: peer="{Voter server-7961c63f-0644-b43d-47d3-1c6aed691c58 7961c63f-0644-b43d-47d3-1c6aed691c58}" -2025-10-30T18:19:11.974Z [ERROR] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: failed to get log: index=2 error="log not found" -2025-10-30T18:19:11.974Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: aborting pipeline replication: peer="{Voter server-d4487d22-db28-ad97-459e-bdc76e06a8fc d4487d22-db28-ad97-459e-bdc76e06a8fc}" -2025-10-30T18:19:11.983Z [ERROR] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: failed to get log: index=2 error="log not found" -2025-10-30T18:19:11.985Z [ERROR] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: failed to get log: index=2 error="log not found" -2025-10-30T18:19:11.985Z [INFO] creating new snapshot: path=/tmp/raft3916884886/snapshots/2-10013-1761848351985.tmp -2025-10-30T18:19:12.021Z [INFO] creating new snapshot: path=/tmp/raft2705774192/snapshots/2-10013-1761848352021.tmp -2025-10-30T18:19:12.025Z [INFO] server-7961c63f-0644-b43d-47d3-1c6aed691c58: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:12.025Z [INFO] server-d4487d22-db28-ad97-459e-bdc76e06a8fc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:12.025Z [INFO] server-d4487d22-db28-ad97-459e-bdc76e06a8fc: copied to local snapshot: bytes=71 -2025-10-30T18:19:12.025Z [INFO] server-7961c63f-0644-b43d-47d3-1c6aed691c58: copied to local snapshot: bytes=71 -2025-10-30T18:19:12.025Z [INFO] server-7961c63f-0644-b43d-47d3-1c6aed691c58: snapshot restore progress: id=2-10013-1761848352021 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:12.025Z [INFO] server-7961c63f-0644-b43d-47d3-1c6aed691c58: removing all old logs from log store -2025-10-30T18:19:12.025Z [INFO] server-7961c63f-0644-b43d-47d3-1c6aed691c58: compacting logs: from=1 to=2 -2025-10-30T18:19:12.025Z [INFO] server-7961c63f-0644-b43d-47d3-1c6aed691c58: Installed remote snapshot -2025-10-30T18:19:12.025Z [INFO] server-d4487d22-db28-ad97-459e-bdc76e06a8fc: snapshot restore progress: id=2-10013-1761848351985 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% -2025-10-30T18:19:12.025Z [INFO] server-d4487d22-db28-ad97-459e-bdc76e06a8fc: removing all old logs from log store -2025-10-30T18:19:12.025Z [INFO] server-d4487d22-db28-ad97-459e-bdc76e06a8fc: compacting logs: from=1 to=2 -2025-10-30T18:19:12.025Z [INFO] server-d4487d22-db28-ad97-459e-bdc76e06a8fc: Installed remote snapshot -2025-10-30T18:19:12.026Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: pipelining replication: peer="{Voter server-7961c63f-0644-b43d-47d3-1c6aed691c58 7961c63f-0644-b43d-47d3-1c6aed691c58}" -2025-10-30T18:19:12.026Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:12.026Z [WARN] server-d4487d22-db28-ad97-459e-bdc76e06a8fc: skipping application of old log: index=10013 -2025-10-30T18:19:12.026Z [INFO] server-d97fd847-4517-e92e-2b4c-3b09900a54e0: pipelining replication: peer="{Voter server-d4487d22-db28-ad97-459e-bdc76e06a8fc d4487d22-db28-ad97-459e-bdc76e06a8fc}" -2025-10-30T18:19:13.031Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:12.026048671 +0000 UTC m=+98.304722569 (2 nodes), 639ns from start of poll, 1.741438728s from cluster start. Timeout at 2025-10-30 18:19:13.031085221 +0000 UTC m=+99.309759118, 1.005036549s after stability -2025-10-30T18:19:13.043Z [INFO] server-fd78e202-034d-df7b-0446-3c02197f42a1: aborting pipeline replication: peer="{Voter server-c495c0bd-a40e-b036-c40c-82fa0659a361 c495c0bd-a40e-b036-c40c-82fa0659a361}" -2025-10-30T18:19:13.043Z [INFO] server-fd78e202-034d-df7b-0446-3c02197f42a1: aborting pipeline replication: peer="{Voter server-c275bf99-ff0d-a74b-3547-f4d8386c899d c275bf99-ff0d-a74b-3547-f4d8386c899d}" ---- PASS: TestRaft_UserRestore (86.25s) - --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.64s) - --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.70s) - --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.41s) - --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.62s) - --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.56s) - --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.63s) - --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.43s) - --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.08s) - --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.71s) - --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.76s) - --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.46s) - --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.31s) - --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.86s) - --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.65s) - --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.69s) - --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.43s) - --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.94s) - --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.93s) - --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.43s) - --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.64s) - --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (3.07s) - --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.76s) - --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.13s) - --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.38s) +2025-10-30T18:23:01.318Z [INFO] server-18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4 Address:18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4} {Suffrage:Voter ID:server-8954336e-2e58-6aeb-e9aa-63a38794b5f9 Address:8954336e-2e58-6aeb-e9aa-63a38794b5f9} {Suffrage:Voter ID:server-f4a1ae43-8912-d954-a110-4eb8dc2c9ba4 Address:f4a1ae43-8912-d954-a110-4eb8dc2c9ba4}]" +2025-10-30T18:23:01.318Z [INFO] server-18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4: entering follower state: follower="Node at 18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4 [Follower]" leader-address= leader-id= +2025-10-30T18:23:01.318Z [INFO] server-8954336e-2e58-6aeb-e9aa-63a38794b5f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4 Address:18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4} {Suffrage:Voter ID:server-8954336e-2e58-6aeb-e9aa-63a38794b5f9 Address:8954336e-2e58-6aeb-e9aa-63a38794b5f9} {Suffrage:Voter ID:server-f4a1ae43-8912-d954-a110-4eb8dc2c9ba4 Address:f4a1ae43-8912-d954-a110-4eb8dc2c9ba4}]" +2025-10-30T18:23:01.318Z [INFO] server-8954336e-2e58-6aeb-e9aa-63a38794b5f9: entering follower state: follower="Node at 8954336e-2e58-6aeb-e9aa-63a38794b5f9 [Follower]" leader-address= leader-id= +2025-10-30T18:23:01.318Z [INFO] server-f4a1ae43-8912-d954-a110-4eb8dc2c9ba4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4 Address:18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4} {Suffrage:Voter ID:server-8954336e-2e58-6aeb-e9aa-63a38794b5f9 Address:8954336e-2e58-6aeb-e9aa-63a38794b5f9} {Suffrage:Voter ID:server-f4a1ae43-8912-d954-a110-4eb8dc2c9ba4 Address:f4a1ae43-8912-d954-a110-4eb8dc2c9ba4}]" +2025-10-30T18:23:01.318Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:01.318Z [INFO] server-f4a1ae43-8912-d954-a110-4eb8dc2c9ba4: entering follower state: follower="Node at f4a1ae43-8912-d954-a110-4eb8dc2c9ba4 [Follower]" leader-address= leader-id= +2025-10-30T18:23:01.840Z [WARN] server-18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:01.840Z [INFO] server-18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4: entering candidate state: node="Node at 18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4 [Candidate]" term=2 +2025-10-30T18:23:01.840Z [INFO] server-18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4: election won: term=2 tally=2 +2025-10-30T18:23:01.840Z [INFO] server-18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4: entering leader state: leader="Node at 18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4 [Leader]" +2025-10-30T18:23:01.840Z [INFO] server-18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4: added peer, starting replication: peer=server-8954336e-2e58-6aeb-e9aa-63a38794b5f9 +2025-10-30T18:23:01.840Z [INFO] server-18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4: added peer, starting replication: peer=server-f4a1ae43-8912-d954-a110-4eb8dc2c9ba4 +2025-10-30T18:23:01.840Z [INFO] server-18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4: pipelining replication: peer="{Voter server-8954336e-2e58-6aeb-e9aa-63a38794b5f9 8954336e-2e58-6aeb-e9aa-63a38794b5f9}" +2025-10-30T18:23:01.840Z [INFO] server-18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4: pipelining replication: peer="{Voter server-f4a1ae43-8912-d954-a110-4eb8dc2c9ba4 f4a1ae43-8912-d954-a110-4eb8dc2c9ba4}" +2025-10-30T18:23:02.845Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:01.840636619 +0000 UTC m=+104.914165918 (1 nodes), 521.866739ms from start of poll, 522.263126ms from cluster start. Timeout at 2025-10-30 18:23:02.845637645 +0000 UTC m=+105.919166937, 1.005001019s after stability +2025-10-30T18:23:02.846Z [INFO] server-18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4: starting snapshot up to: index=12 +2025-10-30T18:23:02.846Z [INFO] creating new snapshot: path=/tmp/raft1377355932/snapshots/2-12-1761848582846.tmp +2025-10-30T18:23:02.957Z [INFO] server-18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4: snapshot complete up to: index=12 +2025-10-30T18:23:02.959Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f Address:94c4d3f2-e0eb-aac9-48d3-93b5520b465f} {Suffrage:Voter ID:server-84f1add1-f0b9-7b00-56d1-6327c4d551f4 Address:84f1add1-f0b9-7b00-56d1-6327c4d551f4} {Suffrage:Voter ID:server-42d018ee-81f1-7e1d-c227-92ef6d67b969 Address:42d018ee-81f1-7e1d-c227-92ef6d67b969}]" +2025-10-30T18:23:02.959Z [INFO] server-84f1add1-f0b9-7b00-56d1-6327c4d551f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f Address:94c4d3f2-e0eb-aac9-48d3-93b5520b465f} {Suffrage:Voter ID:server-84f1add1-f0b9-7b00-56d1-6327c4d551f4 Address:84f1add1-f0b9-7b00-56d1-6327c4d551f4} {Suffrage:Voter ID:server-42d018ee-81f1-7e1d-c227-92ef6d67b969 Address:42d018ee-81f1-7e1d-c227-92ef6d67b969}]" +2025-10-30T18:23:02.959Z [INFO] server-84f1add1-f0b9-7b00-56d1-6327c4d551f4: entering follower state: follower="Node at 84f1add1-f0b9-7b00-56d1-6327c4d551f4 [Follower]" leader-address= leader-id= +2025-10-30T18:23:02.959Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: entering follower state: follower="Node at 94c4d3f2-e0eb-aac9-48d3-93b5520b465f [Follower]" leader-address= leader-id= +2025-10-30T18:23:02.959Z [INFO] server-42d018ee-81f1-7e1d-c227-92ef6d67b969: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f Address:94c4d3f2-e0eb-aac9-48d3-93b5520b465f} {Suffrage:Voter ID:server-84f1add1-f0b9-7b00-56d1-6327c4d551f4 Address:84f1add1-f0b9-7b00-56d1-6327c4d551f4} {Suffrage:Voter ID:server-42d018ee-81f1-7e1d-c227-92ef6d67b969 Address:42d018ee-81f1-7e1d-c227-92ef6d67b969}]" +2025-10-30T18:23:02.959Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:02.959Z [INFO] server-42d018ee-81f1-7e1d-c227-92ef6d67b969: entering follower state: follower="Node at 42d018ee-81f1-7e1d-c227-92ef6d67b969 [Follower]" leader-address= leader-id= +2025-10-30T18:23:03.597Z [WARN] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:03.597Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: entering candidate state: node="Node at 94c4d3f2-e0eb-aac9-48d3-93b5520b465f [Candidate]" term=2 +2025-10-30T18:23:03.597Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: election won: term=2 tally=2 +2025-10-30T18:23:03.597Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: entering leader state: leader="Node at 94c4d3f2-e0eb-aac9-48d3-93b5520b465f [Leader]" +2025-10-30T18:23:03.597Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: added peer, starting replication: peer=server-84f1add1-f0b9-7b00-56d1-6327c4d551f4 +2025-10-30T18:23:03.597Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: added peer, starting replication: peer=server-42d018ee-81f1-7e1d-c227-92ef6d67b969 +2025-10-30T18:23:03.597Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: pipelining replication: peer="{Voter server-42d018ee-81f1-7e1d-c227-92ef6d67b969 42d018ee-81f1-7e1d-c227-92ef6d67b969}" +2025-10-30T18:23:03.597Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: pipelining replication: peer="{Voter server-84f1add1-f0b9-7b00-56d1-6327c4d551f4 84f1add1-f0b9-7b00-56d1-6327c4d551f4}" +2025-10-30T18:23:04.602Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:03.597844364 +0000 UTC m=+106.671373657 (1 nodes), 637.948555ms from start of poll, 638.75386ms from cluster start. Timeout at 2025-10-30 18:23:04.602846679 +0000 UTC m=+107.676376042, 1.005002385s after stability +2025-10-30T18:23:04.632Z [INFO] creating new snapshot: path=/tmp/raft497902750/snapshots/2-10013-1761848584632.tmp +2025-10-30T18:23:04.732Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: copied to local snapshot: bytes=71 +2025-10-30T18:23:04.732Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: snapshot restore progress: id=2-10013-1761848584632 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:23:04.732Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: removing all old logs from log store +2025-10-30T18:23:04.732Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: compacting logs: from=1 to=2 +2025-10-30T18:23:04.732Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: restored user snapshot: index=10013 +2025-10-30T18:23:04.733Z [ERROR] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: failed to get log: index=2 error="log not found" +2025-10-30T18:23:04.733Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: aborting pipeline replication: peer="{Voter server-42d018ee-81f1-7e1d-c227-92ef6d67b969 42d018ee-81f1-7e1d-c227-92ef6d67b969}" +2025-10-30T18:23:04.733Z [ERROR] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: failed to get log: index=2 error="log not found" +2025-10-30T18:23:04.733Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: aborting pipeline replication: peer="{Voter server-84f1add1-f0b9-7b00-56d1-6327c4d551f4 84f1add1-f0b9-7b00-56d1-6327c4d551f4}" +2025-10-30T18:23:04.739Z [ERROR] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: failed to get log: index=2 error="log not found" +2025-10-30T18:23:04.743Z [ERROR] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: failed to get log: index=2 error="log not found" +2025-10-30T18:23:04.743Z [INFO] creating new snapshot: path=/tmp/raft657778651/snapshots/2-10013-1761848584743.tmp +2025-10-30T18:23:04.796Z [INFO] server-42d018ee-81f1-7e1d-c227-92ef6d67b969: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:23:04.796Z [INFO] creating new snapshot: path=/tmp/raft3694866978/snapshots/2-10013-1761848584796.tmp +2025-10-30T18:23:04.796Z [INFO] server-42d018ee-81f1-7e1d-c227-92ef6d67b969: copied to local snapshot: bytes=71 +2025-10-30T18:23:04.796Z [INFO] server-84f1add1-f0b9-7b00-56d1-6327c4d551f4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% +2025-10-30T18:23:04.797Z [INFO] server-42d018ee-81f1-7e1d-c227-92ef6d67b969: snapshot restore progress: id=2-10013-1761848584743 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:23:04.797Z [INFO] server-42d018ee-81f1-7e1d-c227-92ef6d67b969: removing all old logs from log store +2025-10-30T18:23:04.797Z [INFO] server-42d018ee-81f1-7e1d-c227-92ef6d67b969: compacting logs: from=1 to=2 +2025-10-30T18:23:04.797Z [INFO] server-42d018ee-81f1-7e1d-c227-92ef6d67b969: Installed remote snapshot +2025-10-30T18:23:04.797Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: pipelining replication: peer="{Voter server-42d018ee-81f1-7e1d-c227-92ef6d67b969 42d018ee-81f1-7e1d-c227-92ef6d67b969}" +2025-10-30T18:23:04.797Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:04.930Z [INFO] server-84f1add1-f0b9-7b00-56d1-6327c4d551f4: copied to local snapshot: bytes=71 +2025-10-30T18:23:04.930Z [INFO] server-84f1add1-f0b9-7b00-56d1-6327c4d551f4: snapshot restore progress: id=2-10013-1761848584796 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% +2025-10-30T18:23:04.930Z [INFO] server-84f1add1-f0b9-7b00-56d1-6327c4d551f4: removing all old logs from log store +2025-10-30T18:23:04.931Z [INFO] server-84f1add1-f0b9-7b00-56d1-6327c4d551f4: compacting logs: from=1 to=2 +2025-10-30T18:23:04.931Z [INFO] server-84f1add1-f0b9-7b00-56d1-6327c4d551f4: Installed remote snapshot +2025-10-30T18:23:04.931Z [INFO] server-94c4d3f2-e0eb-aac9-48d3-93b5520b465f: pipelining replication: peer="{Voter server-84f1add1-f0b9-7b00-56d1-6327c4d551f4 84f1add1-f0b9-7b00-56d1-6327c4d551f4}" +2025-10-30T18:23:05.803Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:04.797193517 +0000 UTC m=+107.870722808 (2 nodes), 526ns from start of poll, 1.838103011s from cluster start. Timeout at 2025-10-30 18:23:05.802221255 +0000 UTC m=+108.875750582, 1.005027774s after stability +2025-10-30T18:23:05.813Z [INFO] server-18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4: aborting pipeline replication: peer="{Voter server-8954336e-2e58-6aeb-e9aa-63a38794b5f9 8954336e-2e58-6aeb-e9aa-63a38794b5f9}" +2025-10-30T18:23:05.813Z [INFO] server-18abdf5d-fb1f-ef38-fb5c-c9fa2cda32a4: aborting pipeline replication: peer="{Voter server-f4a1ae43-8912-d954-a110-4eb8dc2c9ba4 f4a1ae43-8912-d954-a110-4eb8dc2c9ba4}" +--- FAIL: TestRaft_UserRestore (90.38s) + --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (3.28s) + --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (3.01s) + --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.64s) + --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.74s) + --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.69s) + --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.58s) + --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.40s) + --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.70s) + --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.60s) + --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.78s) + --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.34s) + --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.27s) + --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.69s) + --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.84s) + --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.54s) + --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.93s) + --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.54s) + --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.83s) + --- FAIL: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (5.38s) + --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (5.15s) + --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (3.03s) + --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (3.21s) + --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.65s) + --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.55s) === RUN TestRaft_SendSnapshotFollower -2025-10-30T18:19:13.044Z [INFO] server-7583d3a5-2c6c-1760-d3f1-4b0d55322785: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7583d3a5-2c6c-1760-d3f1-4b0d55322785 Address:7583d3a5-2c6c-1760-d3f1-4b0d55322785} {Suffrage:Voter ID:server-c7342f45-e086-d997-bac9-cfee8eb5839e Address:c7342f45-e086-d997-bac9-cfee8eb5839e} {Suffrage:Voter ID:server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89 Address:8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89}]" -2025-10-30T18:19:13.044Z [INFO] server-7583d3a5-2c6c-1760-d3f1-4b0d55322785: entering follower state: follower="Node at 7583d3a5-2c6c-1760-d3f1-4b0d55322785 [Follower]" leader-address= leader-id= -2025-10-30T18:19:13.044Z [INFO] server-c7342f45-e086-d997-bac9-cfee8eb5839e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7583d3a5-2c6c-1760-d3f1-4b0d55322785 Address:7583d3a5-2c6c-1760-d3f1-4b0d55322785} {Suffrage:Voter ID:server-c7342f45-e086-d997-bac9-cfee8eb5839e Address:c7342f45-e086-d997-bac9-cfee8eb5839e} {Suffrage:Voter ID:server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89 Address:8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89}]" -2025-10-30T18:19:13.044Z [INFO] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7583d3a5-2c6c-1760-d3f1-4b0d55322785 Address:7583d3a5-2c6c-1760-d3f1-4b0d55322785} {Suffrage:Voter ID:server-c7342f45-e086-d997-bac9-cfee8eb5839e Address:c7342f45-e086-d997-bac9-cfee8eb5839e} {Suffrage:Voter ID:server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89 Address:8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89}]" -2025-10-30T18:19:13.044Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:13.044Z [INFO] server-c7342f45-e086-d997-bac9-cfee8eb5839e: entering follower state: follower="Node at c7342f45-e086-d997-bac9-cfee8eb5839e [Follower]" leader-address= leader-id= -2025-10-30T18:19:13.044Z [INFO] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: entering follower state: follower="Node at 8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89 [Follower]" leader-address= leader-id= -2025-10-30T18:19:13.111Z [WARN] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:13.111Z [INFO] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: entering candidate state: node="Node at 8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89 [Candidate]" term=2 -2025-10-30T18:19:13.111Z [INFO] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: election won: term=2 tally=2 -2025-10-30T18:19:13.111Z [INFO] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: entering leader state: leader="Node at 8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89 [Leader]" -2025-10-30T18:19:13.111Z [INFO] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: added peer, starting replication: peer=server-7583d3a5-2c6c-1760-d3f1-4b0d55322785 -2025-10-30T18:19:13.111Z [INFO] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: added peer, starting replication: peer=server-c7342f45-e086-d997-bac9-cfee8eb5839e -2025-10-30T18:19:13.111Z [INFO] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: pipelining replication: peer="{Voter server-7583d3a5-2c6c-1760-d3f1-4b0d55322785 7583d3a5-2c6c-1760-d3f1-4b0d55322785}" -2025-10-30T18:19:13.111Z [INFO] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: pipelining replication: peer="{Voter server-c7342f45-e086-d997-bac9-cfee8eb5839e c7342f45-e086-d997-bac9-cfee8eb5839e}" -2025-10-30T18:19:13.216Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:13.111256348 +0000 UTC m=+99.389930251 (2 nodes), 66.958009ms from start of poll, 67.187701ms from cluster start. Timeout at 2025-10-30 18:19:13.216259568 +0000 UTC m=+99.494933478, 105.003227ms after stability -2025-10-30T18:19:13.216Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:13.322Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:13.216767018 +0000 UTC m=+99.495440916 (1 nodes), 2.033µs from start of poll, 172.698366ms from cluster start. Timeout at 2025-10-30 18:19:13.321802796 +0000 UTC m=+99.600476695, 105.035779ms after stability -2025-10-30T18:19:13.322Z [ERROR] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: failed to pipeline appendEntries: peer="{Voter server-7583d3a5-2c6c-1760-d3f1-4b0d55322785 7583d3a5-2c6c-1760-d3f1-4b0d55322785}" error="append pipeline closed" -2025-10-30T18:19:13.322Z [INFO] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: aborting pipeline replication: peer="{Voter server-7583d3a5-2c6c-1760-d3f1-4b0d55322785 7583d3a5-2c6c-1760-d3f1-4b0d55322785}" -2025-10-30T18:19:13.322Z [ERROR] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: failed to appendEntries to: peer="{Voter server-7583d3a5-2c6c-1760-d3f1-4b0d55322785 7583d3a5-2c6c-1760-d3f1-4b0d55322785}" error="failed to connect to peer: 7583d3a5-2c6c-1760-d3f1-4b0d55322785" +2025-10-30T18:23:05.872Z [INFO] server-3378b003-12e2-7a1c-1ceb-440241677cb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3378b003-12e2-7a1c-1ceb-440241677cb2 Address:3378b003-12e2-7a1c-1ceb-440241677cb2} {Suffrage:Voter ID:server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad Address:aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad} {Suffrage:Voter ID:server-d30a66e0-1a89-df27-a7b0-d631c383d33d Address:d30a66e0-1a89-df27-a7b0-d631c383d33d}]" +2025-10-30T18:23:05.872Z [INFO] server-3378b003-12e2-7a1c-1ceb-440241677cb2: entering follower state: follower="Node at 3378b003-12e2-7a1c-1ceb-440241677cb2 [Follower]" leader-address= leader-id= +2025-10-30T18:23:05.873Z [INFO] server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3378b003-12e2-7a1c-1ceb-440241677cb2 Address:3378b003-12e2-7a1c-1ceb-440241677cb2} {Suffrage:Voter ID:server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad Address:aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad} {Suffrage:Voter ID:server-d30a66e0-1a89-df27-a7b0-d631c383d33d Address:d30a66e0-1a89-df27-a7b0-d631c383d33d}]" +2025-10-30T18:23:05.873Z [INFO] server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad: entering follower state: follower="Node at aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad [Follower]" leader-address= leader-id= +2025-10-30T18:23:05.873Z [INFO] server-d30a66e0-1a89-df27-a7b0-d631c383d33d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3378b003-12e2-7a1c-1ceb-440241677cb2 Address:3378b003-12e2-7a1c-1ceb-440241677cb2} {Suffrage:Voter ID:server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad Address:aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad} {Suffrage:Voter ID:server-d30a66e0-1a89-df27-a7b0-d631c383d33d Address:d30a66e0-1a89-df27-a7b0-d631c383d33d}]" +2025-10-30T18:23:05.873Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:05.873Z [INFO] server-d30a66e0-1a89-df27-a7b0-d631c383d33d: entering follower state: follower="Node at d30a66e0-1a89-df27-a7b0-d631c383d33d [Follower]" leader-address= leader-id= +2025-10-30T18:23:05.924Z [WARN] server-3378b003-12e2-7a1c-1ceb-440241677cb2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:05.924Z [INFO] server-3378b003-12e2-7a1c-1ceb-440241677cb2: entering candidate state: node="Node at 3378b003-12e2-7a1c-1ceb-440241677cb2 [Candidate]" term=2 +2025-10-30T18:23:05.924Z [INFO] server-3378b003-12e2-7a1c-1ceb-440241677cb2: election won: term=2 tally=2 +2025-10-30T18:23:05.924Z [INFO] server-3378b003-12e2-7a1c-1ceb-440241677cb2: entering leader state: leader="Node at 3378b003-12e2-7a1c-1ceb-440241677cb2 [Leader]" +2025-10-30T18:23:05.924Z [INFO] server-3378b003-12e2-7a1c-1ceb-440241677cb2: added peer, starting replication: peer=server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad +2025-10-30T18:23:05.924Z [INFO] server-3378b003-12e2-7a1c-1ceb-440241677cb2: added peer, starting replication: peer=server-d30a66e0-1a89-df27-a7b0-d631c383d33d +2025-10-30T18:23:05.924Z [INFO] server-3378b003-12e2-7a1c-1ceb-440241677cb2: pipelining replication: peer="{Voter server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad}" +2025-10-30T18:23:05.924Z [INFO] server-3378b003-12e2-7a1c-1ceb-440241677cb2: pipelining replication: peer="{Voter server-d30a66e0-1a89-df27-a7b0-d631c383d33d d30a66e0-1a89-df27-a7b0-d631c383d33d}" +2025-10-30T18:23:06.029Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:05.924491211 +0000 UTC m=+108.998020516 (2 nodes), 51.214206ms from start of poll, 51.655868ms from cluster start. Timeout at 2025-10-30 18:23:06.029494813 +0000 UTC m=+109.103024102, 105.003586ms after stability +2025-10-30T18:23:06.029Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:06.134Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:06.029755827 +0000 UTC m=+109.103285121 (1 nodes), 1.925µs from start of poll, 156.920473ms from cluster start. Timeout at 2025-10-30 18:23:06.13476664 +0000 UTC m=+109.208295964, 105.010843ms after stability +2025-10-30T18:23:06.134Z [ERROR] server-3378b003-12e2-7a1c-1ceb-440241677cb2: failed to pipeline appendEntries: peer="{Voter server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad}" error="append pipeline closed" +2025-10-30T18:23:06.134Z [INFO] server-3378b003-12e2-7a1c-1ceb-440241677cb2: aborting pipeline replication: peer="{Voter server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad}" +2025-10-30T18:23:06.134Z [ERROR] server-3378b003-12e2-7a1c-1ceb-440241677cb2: failed to appendEntries to: peer="{Voter server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad}" error="failed to connect to peer: aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad" raft_test.go:1627: [INFO] Finished apply without behind follower -2025-10-30T18:19:13.326Z [ERROR] server-7583d3a5-2c6c-1760-d3f1-4b0d55322785: failed to take snapshot: error="nothing new to snapshot" -2025-10-30T18:19:13.326Z [INFO] server-c7342f45-e086-d997-bac9-cfee8eb5839e: starting snapshot up to: index=55 -2025-10-30T18:19:13.326Z [INFO] creating new snapshot: path=/tmp/raft1374354312/snapshots/2-55-1761848353326.tmp -2025-10-30T18:19:13.326Z [INFO] server-c7342f45-e086-d997-bac9-cfee8eb5839e: compacting logs: from=1 to=55 -2025-10-30T18:19:13.326Z [INFO] server-c7342f45-e086-d997-bac9-cfee8eb5839e: snapshot complete up to: index=55 -2025-10-30T18:19:13.326Z [INFO] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: starting snapshot up to: index=102 -2025-10-30T18:19:13.326Z [INFO] creating new snapshot: path=/tmp/raft3709220428/snapshots/2-102-1761848353326.tmp -2025-10-30T18:19:13.330Z [ERROR] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: failed to heartbeat to: peer=7583d3a5-2c6c-1760-d3f1-4b0d55322785 backoff time=10ms error="failed to connect to peer: 7583d3a5-2c6c-1760-d3f1-4b0d55322785" -2025-10-30T18:19:13.333Z [ERROR] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: failed to appendEntries to: peer="{Voter server-7583d3a5-2c6c-1760-d3f1-4b0d55322785 7583d3a5-2c6c-1760-d3f1-4b0d55322785}" error="failed to connect to peer: 7583d3a5-2c6c-1760-d3f1-4b0d55322785" -2025-10-30T18:19:13.335Z [INFO] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: compacting logs: from=1 to=92 -2025-10-30T18:19:13.335Z [INFO] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: snapshot complete up to: index=102 -2025-10-30T18:19:13.344Z [ERROR] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: failed to get log: index=2 error="log not found" -2025-10-30T18:19:13.344Z [INFO] creating new snapshot: path=/tmp/raft608591271/snapshots/2-102-1761848353344.tmp -2025-10-30T18:19:13.344Z [INFO] server-7583d3a5-2c6c-1760-d3f1-4b0d55322785: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% -2025-10-30T18:19:13.344Z [INFO] server-7583d3a5-2c6c-1760-d3f1-4b0d55322785: copied to local snapshot: bytes=693 -2025-10-30T18:19:13.345Z [INFO] server-7583d3a5-2c6c-1760-d3f1-4b0d55322785: snapshot restore progress: id=2-102-1761848353344 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% -2025-10-30T18:19:13.345Z [INFO] server-7583d3a5-2c6c-1760-d3f1-4b0d55322785: Installed remote snapshot -2025-10-30T18:19:13.346Z [INFO] server-8064f8b8-cb99-3d1c-ad36-25bd7e8a4d89: aborting pipeline replication: peer="{Voter server-c7342f45-e086-d997-bac9-cfee8eb5839e c7342f45-e086-d997-bac9-cfee8eb5839e}" ---- PASS: TestRaft_SendSnapshotFollower (0.31s) +2025-10-30T18:23:06.136Z [INFO] server-3378b003-12e2-7a1c-1ceb-440241677cb2: starting snapshot up to: index=102 +2025-10-30T18:23:06.136Z [INFO] creating new snapshot: path=/tmp/raft2606622649/snapshots/2-102-1761848586136.tmp +2025-10-30T18:23:06.138Z [ERROR] server-3378b003-12e2-7a1c-1ceb-440241677cb2: failed to heartbeat to: peer=aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad backoff time=10ms error="failed to connect to peer: aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad" +2025-10-30T18:23:06.145Z [ERROR] server-3378b003-12e2-7a1c-1ceb-440241677cb2: failed to appendEntries to: peer="{Voter server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad}" error="failed to connect to peer: aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad" +2025-10-30T18:23:06.155Z [ERROR] server-3378b003-12e2-7a1c-1ceb-440241677cb2: failed to appendEntries to: peer="{Voter server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad}" error="failed to connect to peer: aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad" +2025-10-30T18:23:06.157Z [ERROR] server-3378b003-12e2-7a1c-1ceb-440241677cb2: failed to heartbeat to: peer=aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad backoff time=10ms error="failed to connect to peer: aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad" +2025-10-30T18:23:06.174Z [ERROR] server-3378b003-12e2-7a1c-1ceb-440241677cb2: failed to heartbeat to: peer=aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad backoff time=10ms error="failed to connect to peer: aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad" +2025-10-30T18:23:06.184Z [WARN] server-3378b003-12e2-7a1c-1ceb-440241677cb2: failed to contact: server-id=server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad time=50.823987ms +2025-10-30T18:23:06.185Z [ERROR] server-3378b003-12e2-7a1c-1ceb-440241677cb2: failed to appendEntries to: peer="{Voter server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad}" error="failed to connect to peer: aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad" +2025-10-30T18:23:06.186Z [INFO] server-3378b003-12e2-7a1c-1ceb-440241677cb2: compacting logs: from=1 to=92 +2025-10-30T18:23:06.186Z [INFO] server-3378b003-12e2-7a1c-1ceb-440241677cb2: snapshot complete up to: index=102 +2025-10-30T18:23:06.186Z [ERROR] server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad: failed to take snapshot: error="nothing new to snapshot" +2025-10-30T18:23:06.186Z [INFO] server-d30a66e0-1a89-df27-a7b0-d631c383d33d: starting snapshot up to: index=102 +2025-10-30T18:23:06.186Z [INFO] creating new snapshot: path=/tmp/raft1926093767/snapshots/2-102-1761848586186.tmp +2025-10-30T18:23:06.187Z [INFO] server-d30a66e0-1a89-df27-a7b0-d631c383d33d: compacting logs: from=1 to=92 +2025-10-30T18:23:06.187Z [INFO] server-d30a66e0-1a89-df27-a7b0-d631c383d33d: snapshot complete up to: index=102 +2025-10-30T18:23:06.233Z [ERROR] server-3378b003-12e2-7a1c-1ceb-440241677cb2: failed to get log: index=2 error="log not found" +2025-10-30T18:23:06.319Z [INFO] creating new snapshot: path=/tmp/raft1632941424/snapshots/2-102-1761848586319.tmp +2025-10-30T18:23:06.361Z [WARN] server-3378b003-12e2-7a1c-1ceb-440241677cb2: failed to contact: server-id=server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad time=50.987715ms +2025-10-30T18:23:06.405Z [WARN] server-3378b003-12e2-7a1c-1ceb-440241677cb2: failed to contact: server-id=server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad time=94.237144ms +2025-10-30T18:23:06.440Z [INFO] server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% +2025-10-30T18:23:06.455Z [WARN] server-3378b003-12e2-7a1c-1ceb-440241677cb2: failed to contact: server-id=server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad time=144.155606ms +2025-10-30T18:23:06.561Z [INFO] server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad: copied to local snapshot: bytes=693 +2025-10-30T18:23:06.562Z [INFO] server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad: snapshot restore progress: id=2-102-1761848586319 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% +2025-10-30T18:23:06.562Z [INFO] server-aa6fdd11-f24f-dfe6-cd49-fe5c11aec3ad: Installed remote snapshot +2025-10-30T18:23:06.567Z [INFO] server-3378b003-12e2-7a1c-1ceb-440241677cb2: aborting pipeline replication: peer="{Voter server-d30a66e0-1a89-df27-a7b0-d631c383d33d d30a66e0-1a89-df27-a7b0-d631c383d33d}" +--- PASS: TestRaft_SendSnapshotFollower (0.70s) === RUN TestRaft_SendSnapshotAndLogsFollower -2025-10-30T18:19:13.353Z [INFO] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5dbca4c-3605-f49a-1631-4d44d5c97422 Address:f5dbca4c-3605-f49a-1631-4d44d5c97422} {Suffrage:Voter ID:server-a6a5222a-667c-6f68-0950-b006697a26c5 Address:a6a5222a-667c-6f68-0950-b006697a26c5} {Suffrage:Voter ID:server-f16cefb6-79a8-9b1c-ab72-06db40988785 Address:f16cefb6-79a8-9b1c-ab72-06db40988785}]" -2025-10-30T18:19:13.353Z [INFO] server-a6a5222a-667c-6f68-0950-b006697a26c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5dbca4c-3605-f49a-1631-4d44d5c97422 Address:f5dbca4c-3605-f49a-1631-4d44d5c97422} {Suffrage:Voter ID:server-a6a5222a-667c-6f68-0950-b006697a26c5 Address:a6a5222a-667c-6f68-0950-b006697a26c5} {Suffrage:Voter ID:server-f16cefb6-79a8-9b1c-ab72-06db40988785 Address:f16cefb6-79a8-9b1c-ab72-06db40988785}]" -2025-10-30T18:19:13.353Z [INFO] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: entering follower state: follower="Node at f5dbca4c-3605-f49a-1631-4d44d5c97422 [Follower]" leader-address= leader-id= -2025-10-30T18:19:13.353Z [INFO] server-a6a5222a-667c-6f68-0950-b006697a26c5: entering follower state: follower="Node at a6a5222a-667c-6f68-0950-b006697a26c5 [Follower]" leader-address= leader-id= -2025-10-30T18:19:13.353Z [INFO] server-f16cefb6-79a8-9b1c-ab72-06db40988785: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f5dbca4c-3605-f49a-1631-4d44d5c97422 Address:f5dbca4c-3605-f49a-1631-4d44d5c97422} {Suffrage:Voter ID:server-a6a5222a-667c-6f68-0950-b006697a26c5 Address:a6a5222a-667c-6f68-0950-b006697a26c5} {Suffrage:Voter ID:server-f16cefb6-79a8-9b1c-ab72-06db40988785 Address:f16cefb6-79a8-9b1c-ab72-06db40988785}]" -2025-10-30T18:19:13.353Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:13.353Z [INFO] server-f16cefb6-79a8-9b1c-ab72-06db40988785: entering follower state: follower="Node at f16cefb6-79a8-9b1c-ab72-06db40988785 [Follower]" leader-address= leader-id= -2025-10-30T18:19:13.407Z [WARN] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:13.407Z [INFO] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: entering candidate state: node="Node at f5dbca4c-3605-f49a-1631-4d44d5c97422 [Candidate]" term=2 -2025-10-30T18:19:13.407Z [INFO] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: election won: term=2 tally=2 -2025-10-30T18:19:13.407Z [INFO] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: entering leader state: leader="Node at f5dbca4c-3605-f49a-1631-4d44d5c97422 [Leader]" -2025-10-30T18:19:13.407Z [INFO] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: added peer, starting replication: peer=server-a6a5222a-667c-6f68-0950-b006697a26c5 -2025-10-30T18:19:13.407Z [INFO] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: added peer, starting replication: peer=server-f16cefb6-79a8-9b1c-ab72-06db40988785 -2025-10-30T18:19:13.407Z [INFO] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: pipelining replication: peer="{Voter server-f16cefb6-79a8-9b1c-ab72-06db40988785 f16cefb6-79a8-9b1c-ab72-06db40988785}" -2025-10-30T18:19:13.407Z [INFO] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: pipelining replication: peer="{Voter server-a6a5222a-667c-6f68-0950-b006697a26c5 a6a5222a-667c-6f68-0950-b006697a26c5}" -2025-10-30T18:19:13.513Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:13.407850738 +0000 UTC m=+99.686524650 (2 nodes), 54.285608ms from start of poll, 54.645364ms from cluster start. Timeout at 2025-10-30 18:19:13.512855387 +0000 UTC m=+99.791529299, 105.004649ms after stability -2025-10-30T18:19:13.513Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:13.619Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:13.513271522 +0000 UTC m=+99.791945424 (1 nodes), 1.476µs from start of poll, 160.066138ms from cluster start. Timeout at 2025-10-30 18:19:13.618288243 +0000 UTC m=+99.896962160, 105.016736ms after stability -2025-10-30T18:19:13.619Z [ERROR] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: failed to pipeline appendEntries: peer="{Voter server-a6a5222a-667c-6f68-0950-b006697a26c5 a6a5222a-667c-6f68-0950-b006697a26c5}" error="append pipeline closed" -2025-10-30T18:19:13.619Z [INFO] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: aborting pipeline replication: peer="{Voter server-a6a5222a-667c-6f68-0950-b006697a26c5 a6a5222a-667c-6f68-0950-b006697a26c5}" -2025-10-30T18:19:13.619Z [ERROR] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: failed to appendEntries to: peer="{Voter server-a6a5222a-667c-6f68-0950-b006697a26c5 a6a5222a-667c-6f68-0950-b006697a26c5}" error="failed to connect to peer: a6a5222a-667c-6f68-0950-b006697a26c5" +2025-10-30T18:23:06.571Z [INFO] server-c92c55bf-e06a-af12-3241-60bc928a652c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c92c55bf-e06a-af12-3241-60bc928a652c Address:c92c55bf-e06a-af12-3241-60bc928a652c} {Suffrage:Voter ID:server-e8b88f79-6156-d4ab-575d-cb48571102eb Address:e8b88f79-6156-d4ab-575d-cb48571102eb} {Suffrage:Voter ID:server-e05623fb-6099-a8dc-0f19-2fbb880c94fe Address:e05623fb-6099-a8dc-0f19-2fbb880c94fe}]" +2025-10-30T18:23:06.571Z [INFO] server-c92c55bf-e06a-af12-3241-60bc928a652c: entering follower state: follower="Node at c92c55bf-e06a-af12-3241-60bc928a652c [Follower]" leader-address= leader-id= +2025-10-30T18:23:06.571Z [INFO] server-e8b88f79-6156-d4ab-575d-cb48571102eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c92c55bf-e06a-af12-3241-60bc928a652c Address:c92c55bf-e06a-af12-3241-60bc928a652c} {Suffrage:Voter ID:server-e8b88f79-6156-d4ab-575d-cb48571102eb Address:e8b88f79-6156-d4ab-575d-cb48571102eb} {Suffrage:Voter ID:server-e05623fb-6099-a8dc-0f19-2fbb880c94fe Address:e05623fb-6099-a8dc-0f19-2fbb880c94fe}]" +2025-10-30T18:23:06.571Z [INFO] server-e05623fb-6099-a8dc-0f19-2fbb880c94fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c92c55bf-e06a-af12-3241-60bc928a652c Address:c92c55bf-e06a-af12-3241-60bc928a652c} {Suffrage:Voter ID:server-e8b88f79-6156-d4ab-575d-cb48571102eb Address:e8b88f79-6156-d4ab-575d-cb48571102eb} {Suffrage:Voter ID:server-e05623fb-6099-a8dc-0f19-2fbb880c94fe Address:e05623fb-6099-a8dc-0f19-2fbb880c94fe}]" +2025-10-30T18:23:06.571Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:06.571Z [INFO] server-e8b88f79-6156-d4ab-575d-cb48571102eb: entering follower state: follower="Node at e8b88f79-6156-d4ab-575d-cb48571102eb [Follower]" leader-address= leader-id= +2025-10-30T18:23:06.571Z [INFO] server-e05623fb-6099-a8dc-0f19-2fbb880c94fe: entering follower state: follower="Node at e05623fb-6099-a8dc-0f19-2fbb880c94fe [Follower]" leader-address= leader-id= +2025-10-30T18:23:06.636Z [WARN] server-c92c55bf-e06a-af12-3241-60bc928a652c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:06.636Z [INFO] server-c92c55bf-e06a-af12-3241-60bc928a652c: entering candidate state: node="Node at c92c55bf-e06a-af12-3241-60bc928a652c [Candidate]" term=2 +2025-10-30T18:23:06.637Z [INFO] server-c92c55bf-e06a-af12-3241-60bc928a652c: election won: term=2 tally=2 +2025-10-30T18:23:06.637Z [INFO] server-c92c55bf-e06a-af12-3241-60bc928a652c: entering leader state: leader="Node at c92c55bf-e06a-af12-3241-60bc928a652c [Leader]" +2025-10-30T18:23:06.637Z [INFO] server-c92c55bf-e06a-af12-3241-60bc928a652c: added peer, starting replication: peer=server-e8b88f79-6156-d4ab-575d-cb48571102eb +2025-10-30T18:23:06.637Z [INFO] server-c92c55bf-e06a-af12-3241-60bc928a652c: added peer, starting replication: peer=server-e05623fb-6099-a8dc-0f19-2fbb880c94fe +2025-10-30T18:23:06.637Z [INFO] server-c92c55bf-e06a-af12-3241-60bc928a652c: pipelining replication: peer="{Voter server-e05623fb-6099-a8dc-0f19-2fbb880c94fe e05623fb-6099-a8dc-0f19-2fbb880c94fe}" +2025-10-30T18:23:06.637Z [INFO] server-c92c55bf-e06a-af12-3241-60bc928a652c: pipelining replication: peer="{Voter server-e8b88f79-6156-d4ab-575d-cb48571102eb e8b88f79-6156-d4ab-575d-cb48571102eb}" +2025-10-30T18:23:06.742Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:06.637144397 +0000 UTC m=+109.710673686 (2 nodes), 65.664723ms from start of poll, 65.981001ms from cluster start. Timeout at 2025-10-30 18:23:06.742149849 +0000 UTC m=+109.815679143, 105.005457ms after stability +2025-10-30T18:23:06.742Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:06.848Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:06.742922601 +0000 UTC m=+109.816451893 (1 nodes), 1.076µs from start of poll, 171.759208ms from cluster start. Timeout at 2025-10-30 18:23:06.847938515 +0000 UTC m=+109.921467812, 105.015919ms after stability +2025-10-30T18:23:06.848Z [ERROR] server-c92c55bf-e06a-af12-3241-60bc928a652c: failed to pipeline appendEntries: peer="{Voter server-e8b88f79-6156-d4ab-575d-cb48571102eb e8b88f79-6156-d4ab-575d-cb48571102eb}" error="append pipeline closed" +2025-10-30T18:23:06.848Z [INFO] server-c92c55bf-e06a-af12-3241-60bc928a652c: aborting pipeline replication: peer="{Voter server-e8b88f79-6156-d4ab-575d-cb48571102eb e8b88f79-6156-d4ab-575d-cb48571102eb}" +2025-10-30T18:23:06.848Z [ERROR] server-c92c55bf-e06a-af12-3241-60bc928a652c: failed to appendEntries to: peer="{Voter server-e8b88f79-6156-d4ab-575d-cb48571102eb e8b88f79-6156-d4ab-575d-cb48571102eb}" error="failed to connect to peer: e8b88f79-6156-d4ab-575d-cb48571102eb" +2025-10-30T18:23:06.849Z [ERROR] server-c92c55bf-e06a-af12-3241-60bc928a652c: failed to heartbeat to: peer=e8b88f79-6156-d4ab-575d-cb48571102eb backoff time=10ms error="failed to connect to peer: e8b88f79-6156-d4ab-575d-cb48571102eb" raft_test.go:1669: [INFO] Finished apply without behind follower -2025-10-30T18:19:13.621Z [INFO] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: starting snapshot up to: index=102 -2025-10-30T18:19:13.621Z [INFO] creating new snapshot: path=/tmp/raft2299293677/snapshots/2-102-1761848353621.tmp -2025-10-30T18:19:13.623Z [ERROR] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: failed to heartbeat to: peer=a6a5222a-667c-6f68-0950-b006697a26c5 backoff time=10ms error="failed to connect to peer: a6a5222a-667c-6f68-0950-b006697a26c5" -2025-10-30T18:19:13.624Z [INFO] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: compacting logs: from=1 to=92 -2025-10-30T18:19:13.624Z [INFO] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: snapshot complete up to: index=102 -2025-10-30T18:19:13.624Z [ERROR] server-a6a5222a-667c-6f68-0950-b006697a26c5: failed to take snapshot: error="nothing new to snapshot" -2025-10-30T18:19:13.624Z [INFO] server-f16cefb6-79a8-9b1c-ab72-06db40988785: starting snapshot up to: index=94 -2025-10-30T18:19:13.624Z [INFO] creating new snapshot: path=/tmp/raft1020817440/snapshots/2-94-1761848353624.tmp -2025-10-30T18:19:13.624Z [INFO] server-f16cefb6-79a8-9b1c-ab72-06db40988785: compacting logs: from=1 to=92 -2025-10-30T18:19:13.624Z [INFO] server-f16cefb6-79a8-9b1c-ab72-06db40988785: snapshot complete up to: index=94 +2025-10-30T18:23:06.850Z [INFO] server-c92c55bf-e06a-af12-3241-60bc928a652c: starting snapshot up to: index=102 +2025-10-30T18:23:06.850Z [INFO] creating new snapshot: path=/tmp/raft1582957164/snapshots/2-102-1761848586850.tmp +2025-10-30T18:23:06.859Z [ERROR] server-c92c55bf-e06a-af12-3241-60bc928a652c: failed to appendEntries to: peer="{Voter server-e8b88f79-6156-d4ab-575d-cb48571102eb e8b88f79-6156-d4ab-575d-cb48571102eb}" error="failed to connect to peer: e8b88f79-6156-d4ab-575d-cb48571102eb" +2025-10-30T18:23:06.866Z [ERROR] server-c92c55bf-e06a-af12-3241-60bc928a652c: failed to heartbeat to: peer=e8b88f79-6156-d4ab-575d-cb48571102eb backoff time=10ms error="failed to connect to peer: e8b88f79-6156-d4ab-575d-cb48571102eb" +2025-10-30T18:23:06.869Z [ERROR] server-c92c55bf-e06a-af12-3241-60bc928a652c: failed to appendEntries to: peer="{Voter server-e8b88f79-6156-d4ab-575d-cb48571102eb e8b88f79-6156-d4ab-575d-cb48571102eb}" error="failed to connect to peer: e8b88f79-6156-d4ab-575d-cb48571102eb" +2025-10-30T18:23:06.869Z [INFO] server-c92c55bf-e06a-af12-3241-60bc928a652c: compacting logs: from=1 to=92 +2025-10-30T18:23:06.869Z [INFO] server-c92c55bf-e06a-af12-3241-60bc928a652c: snapshot complete up to: index=102 +2025-10-30T18:23:06.869Z [ERROR] server-e8b88f79-6156-d4ab-575d-cb48571102eb: failed to take snapshot: error="nothing new to snapshot" +2025-10-30T18:23:06.870Z [INFO] server-e05623fb-6099-a8dc-0f19-2fbb880c94fe: starting snapshot up to: index=102 +2025-10-30T18:23:06.870Z [INFO] creating new snapshot: path=/tmp/raft1515291532/snapshots/2-102-1761848586870.tmp +2025-10-30T18:23:06.870Z [INFO] server-e05623fb-6099-a8dc-0f19-2fbb880c94fe: compacting logs: from=1 to=92 +2025-10-30T18:23:06.870Z [INFO] server-e05623fb-6099-a8dc-0f19-2fbb880c94fe: snapshot complete up to: index=102 raft_test.go:1690: [INFO] Finished apply without behind follower -2025-10-30T18:19:13.629Z [ERROR] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: failed to get log: index=2 error="log not found" -2025-10-30T18:19:13.630Z [INFO] creating new snapshot: path=/tmp/raft3269466841/snapshots/2-102-1761848353630.tmp -2025-10-30T18:19:13.630Z [INFO] server-a6a5222a-667c-6f68-0950-b006697a26c5: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% -2025-10-30T18:19:13.637Z [INFO] server-a6a5222a-667c-6f68-0950-b006697a26c5: copied to local snapshot: bytes=693 -2025-10-30T18:19:13.637Z [INFO] server-a6a5222a-667c-6f68-0950-b006697a26c5: snapshot restore progress: id=2-102-1761848353630 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% -2025-10-30T18:19:13.637Z [INFO] server-a6a5222a-667c-6f68-0950-b006697a26c5: Installed remote snapshot -2025-10-30T18:19:13.637Z [INFO] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: pipelining replication: peer="{Voter server-a6a5222a-667c-6f68-0950-b006697a26c5 a6a5222a-667c-6f68-0950-b006697a26c5}" -2025-10-30T18:19:13.638Z [INFO] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: aborting pipeline replication: peer="{Voter server-f16cefb6-79a8-9b1c-ab72-06db40988785 f16cefb6-79a8-9b1c-ab72-06db40988785}" -2025-10-30T18:19:13.638Z [INFO] server-f5dbca4c-3605-f49a-1631-4d44d5c97422: aborting pipeline replication: peer="{Voter server-a6a5222a-667c-6f68-0950-b006697a26c5 a6a5222a-667c-6f68-0950-b006697a26c5}" ---- PASS: TestRaft_SendSnapshotAndLogsFollower (0.29s) +2025-10-30T18:23:06.891Z [ERROR] server-c92c55bf-e06a-af12-3241-60bc928a652c: failed to get log: index=2 error="log not found" +2025-10-30T18:23:06.945Z [INFO] creating new snapshot: path=/tmp/raft662291687/snapshots/2-102-1761848586945.tmp +2025-10-30T18:23:06.945Z [INFO] server-e8b88f79-6156-d4ab-575d-cb48571102eb: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% +2025-10-30T18:23:06.985Z [INFO] server-e8b88f79-6156-d4ab-575d-cb48571102eb: copied to local snapshot: bytes=693 +2025-10-30T18:23:06.985Z [INFO] server-e8b88f79-6156-d4ab-575d-cb48571102eb: snapshot restore progress: id=2-102-1761848586945 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% +2025-10-30T18:23:06.985Z [INFO] server-e8b88f79-6156-d4ab-575d-cb48571102eb: Installed remote snapshot +2025-10-30T18:23:06.985Z [INFO] server-c92c55bf-e06a-af12-3241-60bc928a652c: pipelining replication: peer="{Voter server-e8b88f79-6156-d4ab-575d-cb48571102eb e8b88f79-6156-d4ab-575d-cb48571102eb}" +2025-10-30T18:23:06.995Z [INFO] server-c92c55bf-e06a-af12-3241-60bc928a652c: aborting pipeline replication: peer="{Voter server-e05623fb-6099-a8dc-0f19-2fbb880c94fe e05623fb-6099-a8dc-0f19-2fbb880c94fe}" +2025-10-30T18:23:06.995Z [INFO] server-c92c55bf-e06a-af12-3241-60bc928a652c: aborting pipeline replication: peer="{Voter server-e8b88f79-6156-d4ab-575d-cb48571102eb e8b88f79-6156-d4ab-575d-cb48571102eb}" +--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.52s) === RUN TestRaft_ReJoinFollower -2025-10-30T18:19:13.639Z [INFO] server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 Address:a07c40a4-ba32-bc7d-3ace-5e0a7e76c959} {Suffrage:Voter ID:server-b33acf52-3b79-ca0d-3159-dcb3eef07e01 Address:b33acf52-3b79-ca0d-3159-dcb3eef07e01} {Suffrage:Voter ID:server-ed4a13cc-5082-e111-3c27-598fb280b423 Address:ed4a13cc-5082-e111-3c27-598fb280b423}]" -2025-10-30T18:19:13.639Z [INFO] server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959: entering follower state: follower="Node at a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 [Follower]" leader-address= leader-id= -2025-10-30T18:19:13.639Z [INFO] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 Address:a07c40a4-ba32-bc7d-3ace-5e0a7e76c959} {Suffrage:Voter ID:server-b33acf52-3b79-ca0d-3159-dcb3eef07e01 Address:b33acf52-3b79-ca0d-3159-dcb3eef07e01} {Suffrage:Voter ID:server-ed4a13cc-5082-e111-3c27-598fb280b423 Address:ed4a13cc-5082-e111-3c27-598fb280b423}]" -2025-10-30T18:19:13.639Z [INFO] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: entering follower state: follower="Node at b33acf52-3b79-ca0d-3159-dcb3eef07e01 [Follower]" leader-address= leader-id= -2025-10-30T18:19:13.639Z [INFO] server-ed4a13cc-5082-e111-3c27-598fb280b423: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 Address:a07c40a4-ba32-bc7d-3ace-5e0a7e76c959} {Suffrage:Voter ID:server-b33acf52-3b79-ca0d-3159-dcb3eef07e01 Address:b33acf52-3b79-ca0d-3159-dcb3eef07e01} {Suffrage:Voter ID:server-ed4a13cc-5082-e111-3c27-598fb280b423 Address:ed4a13cc-5082-e111-3c27-598fb280b423}]" -2025-10-30T18:19:13.639Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:13.639Z [INFO] server-ed4a13cc-5082-e111-3c27-598fb280b423: entering follower state: follower="Node at ed4a13cc-5082-e111-3c27-598fb280b423 [Follower]" leader-address= leader-id= -2025-10-30T18:19:13.700Z [WARN] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:13.700Z [INFO] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: entering candidate state: node="Node at b33acf52-3b79-ca0d-3159-dcb3eef07e01 [Candidate]" term=2 -2025-10-30T18:19:13.700Z [INFO] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: election won: term=2 tally=2 -2025-10-30T18:19:13.700Z [INFO] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: entering leader state: leader="Node at b33acf52-3b79-ca0d-3159-dcb3eef07e01 [Leader]" -2025-10-30T18:19:13.700Z [INFO] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: added peer, starting replication: peer=server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 -2025-10-30T18:19:13.700Z [INFO] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: added peer, starting replication: peer=server-ed4a13cc-5082-e111-3c27-598fb280b423 -2025-10-30T18:19:13.700Z [INFO] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: pipelining replication: peer="{Voter server-ed4a13cc-5082-e111-3c27-598fb280b423 ed4a13cc-5082-e111-3c27-598fb280b423}" -2025-10-30T18:19:13.700Z [INFO] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: pipelining replication: peer="{Voter server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 a07c40a4-ba32-bc7d-3ace-5e0a7e76c959}" -2025-10-30T18:19:13.805Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:13.700589231 +0000 UTC m=+99.979263129 (1 nodes), 60.726454ms from start of poll, 60.971752ms from cluster start. Timeout at 2025-10-30 18:19:13.805593901 +0000 UTC m=+100.084267799, 105.00467ms after stability -2025-10-30T18:19:13.810Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:13.916Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:13.810947792 +0000 UTC m=+100.089621690 (2 nodes), 1.103µs from start of poll, 171.330313ms from cluster start. Timeout at 2025-10-30 18:19:13.915971615 +0000 UTC m=+100.194645512, 105.023822ms after stability -2025-10-30T18:19:13.916Z [INFO] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: updating configuration: command=RemoveServer server-id=server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 server-addr= servers="[{Suffrage:Voter ID:server-b33acf52-3b79-ca0d-3159-dcb3eef07e01 Address:b33acf52-3b79-ca0d-3159-dcb3eef07e01} {Suffrage:Voter ID:server-ed4a13cc-5082-e111-3c27-598fb280b423 Address:ed4a13cc-5082-e111-3c27-598fb280b423}]" -2025-10-30T18:19:13.916Z [INFO] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: removed peer, stopping replication: peer=server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 last-index=3 -2025-10-30T18:19:13.916Z [INFO] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: aborting pipeline replication: peer="{Voter server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 a07c40a4-ba32-bc7d-3ace-5e0a7e76c959}" -2025-10-30T18:19:14.021Z [INFO] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: updating configuration: command=AddVoter server-id=server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 server-addr=a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 servers="[{Suffrage:Voter ID:server-b33acf52-3b79-ca0d-3159-dcb3eef07e01 Address:b33acf52-3b79-ca0d-3159-dcb3eef07e01} {Suffrage:Voter ID:server-ed4a13cc-5082-e111-3c27-598fb280b423 Address:ed4a13cc-5082-e111-3c27-598fb280b423} {Suffrage:Voter ID:server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 Address:a07c40a4-ba32-bc7d-3ace-5e0a7e76c959}]" -2025-10-30T18:19:14.021Z [INFO] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: added peer, starting replication: peer=server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 -2025-10-30T18:19:14.021Z [WARN] server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959: failed to get previous log: previous-index=4 last-index=3 error="log not found" -2025-10-30T18:19:14.021Z [WARN] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: appendEntries rejected, sending older logs: peer="{Voter server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 a07c40a4-ba32-bc7d-3ace-5e0a7e76c959}" next=4 -2025-10-30T18:19:14.022Z [INFO] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: pipelining replication: peer="{Voter server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 a07c40a4-ba32-bc7d-3ace-5e0a7e76c959}" -2025-10-30T18:19:14.022Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:14.127Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:14.02207931 +0000 UTC m=+100.300753207 (1 nodes), 434ns from start of poll, 382.46183ms from cluster start. Timeout at 2025-10-30 18:19:14.127106464 +0000 UTC m=+100.405780360, 105.027153ms after stability -2025-10-30T18:19:14.127Z [INFO] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: aborting pipeline replication: peer="{Voter server-a07c40a4-ba32-bc7d-3ace-5e0a7e76c959 a07c40a4-ba32-bc7d-3ace-5e0a7e76c959}" -2025-10-30T18:19:14.127Z [INFO] server-b33acf52-3b79-ca0d-3159-dcb3eef07e01: aborting pipeline replication: peer="{Voter server-ed4a13cc-5082-e111-3c27-598fb280b423 ed4a13cc-5082-e111-3c27-598fb280b423}" ---- PASS: TestRaft_ReJoinFollower (0.49s) +2025-10-30T18:23:07.090Z [INFO] server-05e24d96-1581-5c9e-9398-95cb915c96e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05e24d96-1581-5c9e-9398-95cb915c96e0 Address:05e24d96-1581-5c9e-9398-95cb915c96e0} {Suffrage:Voter ID:server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f Address:ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f} {Suffrage:Voter ID:server-64584aad-59ff-fb24-f8a8-889ff6b68827 Address:64584aad-59ff-fb24-f8a8-889ff6b68827}]" +2025-10-30T18:23:07.090Z [INFO] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05e24d96-1581-5c9e-9398-95cb915c96e0 Address:05e24d96-1581-5c9e-9398-95cb915c96e0} {Suffrage:Voter ID:server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f Address:ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f} {Suffrage:Voter ID:server-64584aad-59ff-fb24-f8a8-889ff6b68827 Address:64584aad-59ff-fb24-f8a8-889ff6b68827}]" +2025-10-30T18:23:07.090Z [INFO] server-05e24d96-1581-5c9e-9398-95cb915c96e0: entering follower state: follower="Node at 05e24d96-1581-5c9e-9398-95cb915c96e0 [Follower]" leader-address= leader-id= +2025-10-30T18:23:07.090Z [INFO] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: entering follower state: follower="Node at ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f [Follower]" leader-address= leader-id= +2025-10-30T18:23:07.090Z [INFO] server-64584aad-59ff-fb24-f8a8-889ff6b68827: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05e24d96-1581-5c9e-9398-95cb915c96e0 Address:05e24d96-1581-5c9e-9398-95cb915c96e0} {Suffrage:Voter ID:server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f Address:ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f} {Suffrage:Voter ID:server-64584aad-59ff-fb24-f8a8-889ff6b68827 Address:64584aad-59ff-fb24-f8a8-889ff6b68827}]" +2025-10-30T18:23:07.090Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:07.090Z [INFO] server-64584aad-59ff-fb24-f8a8-889ff6b68827: entering follower state: follower="Node at 64584aad-59ff-fb24-f8a8-889ff6b68827 [Follower]" leader-address= leader-id= +2025-10-30T18:23:07.141Z [WARN] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:07.141Z [INFO] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: entering candidate state: node="Node at ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f [Candidate]" term=2 +2025-10-30T18:23:07.142Z [INFO] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: election won: term=2 tally=2 +2025-10-30T18:23:07.142Z [INFO] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: entering leader state: leader="Node at ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f [Leader]" +2025-10-30T18:23:07.142Z [INFO] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: added peer, starting replication: peer=server-05e24d96-1581-5c9e-9398-95cb915c96e0 +2025-10-30T18:23:07.142Z [INFO] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: added peer, starting replication: peer=server-64584aad-59ff-fb24-f8a8-889ff6b68827 +2025-10-30T18:23:07.142Z [INFO] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: pipelining replication: peer="{Voter server-64584aad-59ff-fb24-f8a8-889ff6b68827 64584aad-59ff-fb24-f8a8-889ff6b68827}" +2025-10-30T18:23:07.142Z [INFO] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: pipelining replication: peer="{Voter server-05e24d96-1581-5c9e-9398-95cb915c96e0 05e24d96-1581-5c9e-9398-95cb915c96e0}" +2025-10-30T18:23:07.247Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:07.142107097 +0000 UTC m=+110.215636393 (1 nodes), 51.874342ms from start of poll, 52.142076ms from cluster start. Timeout at 2025-10-30 18:23:07.247107899 +0000 UTC m=+110.320637197, 105.000804ms after stability +2025-10-30T18:23:07.252Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:07.357Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:07.2528848 +0000 UTC m=+110.326414092 (2 nodes), 1.597µs from start of poll, 162.919775ms from cluster start. Timeout at 2025-10-30 18:23:07.357919046 +0000 UTC m=+110.431448373, 105.034281ms after stability +2025-10-30T18:23:07.358Z [INFO] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: updating configuration: command=RemoveServer server-id=server-05e24d96-1581-5c9e-9398-95cb915c96e0 server-addr= servers="[{Suffrage:Voter ID:server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f Address:ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f} {Suffrage:Voter ID:server-64584aad-59ff-fb24-f8a8-889ff6b68827 Address:64584aad-59ff-fb24-f8a8-889ff6b68827}]" +2025-10-30T18:23:07.358Z [INFO] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: removed peer, stopping replication: peer=server-05e24d96-1581-5c9e-9398-95cb915c96e0 last-index=3 +2025-10-30T18:23:07.358Z [INFO] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: aborting pipeline replication: peer="{Voter server-05e24d96-1581-5c9e-9398-95cb915c96e0 05e24d96-1581-5c9e-9398-95cb915c96e0}" +2025-10-30T18:23:07.429Z [WARN] server-05e24d96-1581-5c9e-9398-95cb915c96e0: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election +2025-10-30T18:23:07.464Z [INFO] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: updating configuration: command=AddVoter server-id=server-05e24d96-1581-5c9e-9398-95cb915c96e0 server-addr=05e24d96-1581-5c9e-9398-95cb915c96e0 servers="[{Suffrage:Voter ID:server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f Address:ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f} {Suffrage:Voter ID:server-64584aad-59ff-fb24-f8a8-889ff6b68827 Address:64584aad-59ff-fb24-f8a8-889ff6b68827} {Suffrage:Voter ID:server-05e24d96-1581-5c9e-9398-95cb915c96e0 Address:05e24d96-1581-5c9e-9398-95cb915c96e0}]" +2025-10-30T18:23:07.464Z [INFO] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: added peer, starting replication: peer=server-05e24d96-1581-5c9e-9398-95cb915c96e0 +2025-10-30T18:23:07.465Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:07.465Z [WARN] server-05e24d96-1581-5c9e-9398-95cb915c96e0: failed to get previous log: previous-index=4 last-index=3 error="log not found" +2025-10-30T18:23:07.465Z [WARN] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: appendEntries rejected, sending older logs: peer="{Voter server-05e24d96-1581-5c9e-9398-95cb915c96e0 05e24d96-1581-5c9e-9398-95cb915c96e0}" next=4 +2025-10-30T18:23:07.465Z [INFO] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: pipelining replication: peer="{Voter server-05e24d96-1581-5c9e-9398-95cb915c96e0 05e24d96-1581-5c9e-9398-95cb915c96e0}" +2025-10-30T18:23:07.570Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:07.465147354 +0000 UTC m=+110.538676646 (1 nodes), 840ns from start of poll, 375.182329ms from cluster start. Timeout at 2025-10-30 18:23:07.570153527 +0000 UTC m=+110.643682857, 105.006211ms after stability +2025-10-30T18:23:07.570Z [INFO] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: aborting pipeline replication: peer="{Voter server-64584aad-59ff-fb24-f8a8-889ff6b68827 64584aad-59ff-fb24-f8a8-889ff6b68827}" +2025-10-30T18:23:07.571Z [INFO] server-ea71ecf8-735d-3972-9f5d-c6bf5f82fa3f: aborting pipeline replication: peer="{Voter server-05e24d96-1581-5c9e-9398-95cb915c96e0 05e24d96-1581-5c9e-9398-95cb915c96e0}" +--- PASS: TestRaft_ReJoinFollower (0.57s) === RUN TestRaft_LeaderLeaseExpire -2025-10-30T18:19:14.128Z [INFO] server-75d17107-02cc-ceaa-2114-cb6425e8648f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75d17107-02cc-ceaa-2114-cb6425e8648f Address:75d17107-02cc-ceaa-2114-cb6425e8648f} {Suffrage:Voter ID:server-1e3da886-edcc-b52d-ed64-fa60185a48e1 Address:1e3da886-edcc-b52d-ed64-fa60185a48e1}]" -2025-10-30T18:19:14.128Z [INFO] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75d17107-02cc-ceaa-2114-cb6425e8648f Address:75d17107-02cc-ceaa-2114-cb6425e8648f} {Suffrage:Voter ID:server-1e3da886-edcc-b52d-ed64-fa60185a48e1 Address:1e3da886-edcc-b52d-ed64-fa60185a48e1}]" -2025-10-30T18:19:14.128Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:14.128Z [INFO] server-75d17107-02cc-ceaa-2114-cb6425e8648f: entering follower state: follower="Node at 75d17107-02cc-ceaa-2114-cb6425e8648f [Follower]" leader-address= leader-id= -2025-10-30T18:19:14.128Z [INFO] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: entering follower state: follower="Node at 1e3da886-edcc-b52d-ed64-fa60185a48e1 [Follower]" leader-address= leader-id= -2025-10-30T18:19:14.206Z [WARN] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:14.206Z [INFO] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: entering candidate state: node="Node at 1e3da886-edcc-b52d-ed64-fa60185a48e1 [Candidate]" term=2 -2025-10-30T18:19:14.206Z [INFO] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: election won: term=2 tally=2 -2025-10-30T18:19:14.206Z [INFO] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: entering leader state: leader="Node at 1e3da886-edcc-b52d-ed64-fa60185a48e1 [Leader]" -2025-10-30T18:19:14.206Z [INFO] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: added peer, starting replication: peer=server-75d17107-02cc-ceaa-2114-cb6425e8648f -2025-10-30T18:19:14.207Z [INFO] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: pipelining replication: peer="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" -2025-10-30T18:19:14.312Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:14.206855865 +0000 UTC m=+100.485529762 (1 nodes), 78.501942ms from start of poll, 78.673553ms from cluster start. Timeout at 2025-10-30 18:19:14.311856938 +0000 UTC m=+100.590530836, 105.001074ms after stability -2025-10-30T18:19:14.318Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:14.423Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:14.318548305 +0000 UTC m=+100.597222204 (1 nodes), 1.716µs from start of poll, 190.365995ms from cluster start. Timeout at 2025-10-30 18:19:14.423570529 +0000 UTC m=+100.702244426, 105.022222ms after stability - raft_test.go:1799: [INFO] Disconnecting Node at 75d17107-02cc-ceaa-2114-cb6425e8648f [Follower] -2025-10-30T18:19:14.425Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to heartbeat to: peer=75d17107-02cc-ceaa-2114-cb6425e8648f backoff time=10ms error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f" -2025-10-30T18:19:14.426Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to pipeline appendEntries: peer="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" error="append pipeline closed" -2025-10-30T18:19:14.426Z [INFO] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: aborting pipeline replication: peer="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" -2025-10-30T18:19:14.431Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to appendEntries to: peer="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f" -2025-10-30T18:19:14.442Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to heartbeat to: peer=75d17107-02cc-ceaa-2114-cb6425e8648f backoff time=10ms error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f" -2025-10-30T18:19:14.451Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to appendEntries to: peer="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f" -2025-10-30T18:19:14.459Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to heartbeat to: peer=75d17107-02cc-ceaa-2114-cb6425e8648f backoff time=10ms error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f" -2025-10-30T18:19:14.468Z [WARN] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to contact: server-id=server-75d17107-02cc-ceaa-2114-cb6425e8648f time=50.12124ms -2025-10-30T18:19:14.468Z [WARN] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to contact quorum of nodes, stepping down -2025-10-30T18:19:14.468Z [INFO] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: entering follower state: follower="Node at 1e3da886-edcc-b52d-ed64-fa60185a48e1 [Follower]" leader-address= leader-id= -2025-10-30T18:19:14.468Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:14.468Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to appendEntries to: peer="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f" -2025-10-30T18:19:14.473Z [WARN] server-75d17107-02cc-ceaa-2114-cb6425e8648f: heartbeat timeout reached, starting election: last-leader-addr=1e3da886-edcc-b52d-ed64-fa60185a48e1 last-leader-id=server-1e3da886-edcc-b52d-ed64-fa60185a48e1 -2025-10-30T18:19:14.473Z [INFO] server-75d17107-02cc-ceaa-2114-cb6425e8648f: entering candidate state: node="Node at 75d17107-02cc-ceaa-2114-cb6425e8648f [Candidate]" term=3 -2025-10-30T18:19:14.473Z [ERROR] server-75d17107-02cc-ceaa-2114-cb6425e8648f: failed to make requestVote RPC: target="{Voter server-1e3da886-edcc-b52d-ed64-fa60185a48e1 1e3da886-edcc-b52d-ed64-fa60185a48e1}" error="failed to connect to peer: 1e3da886-edcc-b52d-ed64-fa60185a48e1" term=3 -2025-10-30T18:19:14.521Z [WARN] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:14.521Z [INFO] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: entering candidate state: node="Node at 1e3da886-edcc-b52d-ed64-fa60185a48e1 [Candidate]" term=3 -2025-10-30T18:19:14.521Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to make requestVote RPC: target="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f" term=3 -2025-10-30T18:19:14.536Z [WARN] server-75d17107-02cc-ceaa-2114-cb6425e8648f: Election timeout reached, restarting election -2025-10-30T18:19:14.536Z [INFO] server-75d17107-02cc-ceaa-2114-cb6425e8648f: entering candidate state: node="Node at 75d17107-02cc-ceaa-2114-cb6425e8648f [Candidate]" term=4 -2025-10-30T18:19:14.536Z [ERROR] server-75d17107-02cc-ceaa-2114-cb6425e8648f: failed to make requestVote RPC: target="{Voter server-1e3da886-edcc-b52d-ed64-fa60185a48e1 1e3da886-edcc-b52d-ed64-fa60185a48e1}" error="failed to connect to peer: 1e3da886-edcc-b52d-ed64-fa60185a48e1" term=4 -2025-10-30T18:19:14.573Z [WARN] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: Election timeout reached, restarting election -2025-10-30T18:19:14.573Z [INFO] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: entering candidate state: node="Node at 1e3da886-edcc-b52d-ed64-fa60185a48e1 [Candidate]" term=4 -2025-10-30T18:19:14.573Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to make requestVote RPC: target="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f" term=4 -2025-10-30T18:19:14.588Z [WARN] server-75d17107-02cc-ceaa-2114-cb6425e8648f: Election timeout reached, restarting election -2025-10-30T18:19:14.588Z [INFO] server-75d17107-02cc-ceaa-2114-cb6425e8648f: entering candidate state: node="Node at 75d17107-02cc-ceaa-2114-cb6425e8648f [Candidate]" term=5 -2025-10-30T18:19:14.588Z [ERROR] server-75d17107-02cc-ceaa-2114-cb6425e8648f: failed to make requestVote RPC: target="{Voter server-1e3da886-edcc-b52d-ed64-fa60185a48e1 1e3da886-edcc-b52d-ed64-fa60185a48e1}" error="failed to connect to peer: 1e3da886-edcc-b52d-ed64-fa60185a48e1" term=5 -2025-10-30T18:19:14.627Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:14.521909276 +0000 UTC m=+100.800583175 (0 nodes), 53.757186ms from start of poll, 393.726966ms from cluster start. Timeout at 2025-10-30 18:19:14.626911988 +0000 UTC m=+100.905585886, 105.002711ms after stability -2025-10-30T18:19:14.638Z [WARN] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: Election timeout reached, restarting election -2025-10-30T18:19:14.638Z [INFO] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: entering candidate state: node="Node at 1e3da886-edcc-b52d-ed64-fa60185a48e1 [Candidate]" term=5 -2025-10-30T18:19:14.639Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to make requestVote RPC: target="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f" term=5 -2025-10-30T18:19:14.660Z [WARN] server-75d17107-02cc-ceaa-2114-cb6425e8648f: Election timeout reached, restarting election -2025-10-30T18:19:14.660Z [INFO] server-75d17107-02cc-ceaa-2114-cb6425e8648f: entering candidate state: node="Node at 75d17107-02cc-ceaa-2114-cb6425e8648f [Candidate]" term=6 -2025-10-30T18:19:14.660Z [ERROR] server-75d17107-02cc-ceaa-2114-cb6425e8648f: failed to make requestVote RPC: target="{Voter server-1e3da886-edcc-b52d-ed64-fa60185a48e1 1e3da886-edcc-b52d-ed64-fa60185a48e1}" error="failed to connect to peer: 1e3da886-edcc-b52d-ed64-fa60185a48e1" term=6 -2025-10-30T18:19:14.716Z [WARN] server-75d17107-02cc-ceaa-2114-cb6425e8648f: Election timeout reached, restarting election -2025-10-30T18:19:14.716Z [INFO] server-75d17107-02cc-ceaa-2114-cb6425e8648f: entering candidate state: node="Node at 75d17107-02cc-ceaa-2114-cb6425e8648f [Candidate]" term=7 -2025-10-30T18:19:14.716Z [ERROR] server-75d17107-02cc-ceaa-2114-cb6425e8648f: failed to make requestVote RPC: target="{Voter server-1e3da886-edcc-b52d-ed64-fa60185a48e1 1e3da886-edcc-b52d-ed64-fa60185a48e1}" error="failed to connect to peer: 1e3da886-edcc-b52d-ed64-fa60185a48e1" term=7 -2025-10-30T18:19:14.728Z [WARN] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: Election timeout reached, restarting election -2025-10-30T18:19:14.728Z [INFO] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: entering candidate state: node="Node at 1e3da886-edcc-b52d-ed64-fa60185a48e1 [Candidate]" term=6 -2025-10-30T18:19:14.728Z [ERROR] server-1e3da886-edcc-b52d-ed64-fa60185a48e1: failed to make requestVote RPC: target="{Voter server-75d17107-02cc-ceaa-2114-cb6425e8648f 75d17107-02cc-ceaa-2114-cb6425e8648f}" error="failed to connect to peer: 75d17107-02cc-ceaa-2114-cb6425e8648f" term=6 ---- PASS: TestRaft_LeaderLeaseExpire (0.61s) +2025-10-30T18:23:07.658Z [INFO] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35377f8d-37f5-272f-9f4f-4c840e4161bb Address:35377f8d-37f5-272f-9f4f-4c840e4161bb} {Suffrage:Voter ID:server-6eaf8def-756d-0632-01ba-e4f297e64e9a Address:6eaf8def-756d-0632-01ba-e4f297e64e9a}]" +2025-10-30T18:23:07.658Z [INFO] server-6eaf8def-756d-0632-01ba-e4f297e64e9a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35377f8d-37f5-272f-9f4f-4c840e4161bb Address:35377f8d-37f5-272f-9f4f-4c840e4161bb} {Suffrage:Voter ID:server-6eaf8def-756d-0632-01ba-e4f297e64e9a Address:6eaf8def-756d-0632-01ba-e4f297e64e9a}]" +2025-10-30T18:23:07.658Z [INFO] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: entering follower state: follower="Node at 35377f8d-37f5-272f-9f4f-4c840e4161bb [Follower]" leader-address= leader-id= +2025-10-30T18:23:07.658Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:07.658Z [INFO] server-6eaf8def-756d-0632-01ba-e4f297e64e9a: entering follower state: follower="Node at 6eaf8def-756d-0632-01ba-e4f297e64e9a [Follower]" leader-address= leader-id= +2025-10-30T18:23:07.712Z [WARN] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:07.712Z [INFO] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: entering candidate state: node="Node at 35377f8d-37f5-272f-9f4f-4c840e4161bb [Candidate]" term=2 +2025-10-30T18:23:07.712Z [INFO] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: election won: term=2 tally=2 +2025-10-30T18:23:07.712Z [INFO] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: entering leader state: leader="Node at 35377f8d-37f5-272f-9f4f-4c840e4161bb [Leader]" +2025-10-30T18:23:07.712Z [INFO] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: added peer, starting replication: peer=server-6eaf8def-756d-0632-01ba-e4f297e64e9a +2025-10-30T18:23:07.712Z [INFO] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: pipelining replication: peer="{Voter server-6eaf8def-756d-0632-01ba-e4f297e64e9a 6eaf8def-756d-0632-01ba-e4f297e64e9a}" +2025-10-30T18:23:07.818Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:07.712815188 +0000 UTC m=+110.786344481 (1 nodes), 54.168206ms from start of poll, 54.37999ms from cluster start. Timeout at 2025-10-30 18:23:07.817819316 +0000 UTC m=+110.891348609, 105.004128ms after stability +2025-10-30T18:23:07.824Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:07.930Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:07.824655307 +0000 UTC m=+110.898184599 (1 nodes), 728ns from start of poll, 166.220108ms from cluster start. Timeout at 2025-10-30 18:23:07.929676808 +0000 UTC m=+111.003206188, 105.021589ms after stability + raft_test.go:1799: [INFO] Disconnecting Node at 6eaf8def-756d-0632-01ba-e4f297e64e9a [Follower] +2025-10-30T18:23:07.933Z [ERROR] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: failed to heartbeat to: peer=6eaf8def-756d-0632-01ba-e4f297e64e9a backoff time=10ms error="failed to connect to peer: 6eaf8def-756d-0632-01ba-e4f297e64e9a" +2025-10-30T18:23:07.934Z [ERROR] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: failed to pipeline appendEntries: peer="{Voter server-6eaf8def-756d-0632-01ba-e4f297e64e9a 6eaf8def-756d-0632-01ba-e4f297e64e9a}" error="append pipeline closed" +2025-10-30T18:23:07.934Z [INFO] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: aborting pipeline replication: peer="{Voter server-6eaf8def-756d-0632-01ba-e4f297e64e9a 6eaf8def-756d-0632-01ba-e4f297e64e9a}" +2025-10-30T18:23:07.941Z [ERROR] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: failed to appendEntries to: peer="{Voter server-6eaf8def-756d-0632-01ba-e4f297e64e9a 6eaf8def-756d-0632-01ba-e4f297e64e9a}" error="failed to connect to peer: 6eaf8def-756d-0632-01ba-e4f297e64e9a" +2025-10-30T18:23:07.951Z [ERROR] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: failed to heartbeat to: peer=6eaf8def-756d-0632-01ba-e4f297e64e9a backoff time=10ms error="failed to connect to peer: 6eaf8def-756d-0632-01ba-e4f297e64e9a" +2025-10-30T18:23:07.962Z [ERROR] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: failed to appendEntries to: peer="{Voter server-6eaf8def-756d-0632-01ba-e4f297e64e9a 6eaf8def-756d-0632-01ba-e4f297e64e9a}" error="failed to connect to peer: 6eaf8def-756d-0632-01ba-e4f297e64e9a" +2025-10-30T18:23:07.970Z [ERROR] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: failed to heartbeat to: peer=6eaf8def-756d-0632-01ba-e4f297e64e9a backoff time=10ms error="failed to connect to peer: 6eaf8def-756d-0632-01ba-e4f297e64e9a" +2025-10-30T18:23:07.979Z [WARN] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: failed to contact: server-id=server-6eaf8def-756d-0632-01ba-e4f297e64e9a time=51.105909ms +2025-10-30T18:23:07.979Z [WARN] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: failed to contact quorum of nodes, stepping down +2025-10-30T18:23:07.979Z [INFO] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: entering follower state: follower="Node at 35377f8d-37f5-272f-9f4f-4c840e4161bb [Follower]" leader-address= leader-id= +2025-10-30T18:23:07.979Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:07.980Z [ERROR] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: failed to appendEntries to: peer="{Voter server-6eaf8def-756d-0632-01ba-e4f297e64e9a 6eaf8def-756d-0632-01ba-e4f297e64e9a}" error="failed to connect to peer: 6eaf8def-756d-0632-01ba-e4f297e64e9a" +2025-10-30T18:23:07.981Z [WARN] server-6eaf8def-756d-0632-01ba-e4f297e64e9a: heartbeat timeout reached, starting election: last-leader-addr=35377f8d-37f5-272f-9f4f-4c840e4161bb last-leader-id=server-35377f8d-37f5-272f-9f4f-4c840e4161bb +2025-10-30T18:23:07.982Z [INFO] server-6eaf8def-756d-0632-01ba-e4f297e64e9a: entering candidate state: node="Node at 6eaf8def-756d-0632-01ba-e4f297e64e9a [Candidate]" term=3 +2025-10-30T18:23:07.982Z [ERROR] server-6eaf8def-756d-0632-01ba-e4f297e64e9a: failed to make requestVote RPC: target="{Voter server-35377f8d-37f5-272f-9f4f-4c840e4161bb 35377f8d-37f5-272f-9f4f-4c840e4161bb}" error="failed to connect to peer: 35377f8d-37f5-272f-9f4f-4c840e4161bb" term=3 +2025-10-30T18:23:08.033Z [WARN] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:08.033Z [INFO] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: entering candidate state: node="Node at 35377f8d-37f5-272f-9f4f-4c840e4161bb [Candidate]" term=3 +2025-10-30T18:23:08.033Z [ERROR] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: failed to make requestVote RPC: target="{Voter server-6eaf8def-756d-0632-01ba-e4f297e64e9a 6eaf8def-756d-0632-01ba-e4f297e64e9a}" error="failed to connect to peer: 6eaf8def-756d-0632-01ba-e4f297e64e9a" term=3 +2025-10-30T18:23:08.041Z [WARN] server-6eaf8def-756d-0632-01ba-e4f297e64e9a: Election timeout reached, restarting election +2025-10-30T18:23:08.041Z [INFO] server-6eaf8def-756d-0632-01ba-e4f297e64e9a: entering candidate state: node="Node at 6eaf8def-756d-0632-01ba-e4f297e64e9a [Candidate]" term=4 +2025-10-30T18:23:08.041Z [ERROR] server-6eaf8def-756d-0632-01ba-e4f297e64e9a: failed to make requestVote RPC: target="{Voter server-35377f8d-37f5-272f-9f4f-4c840e4161bb 35377f8d-37f5-272f-9f4f-4c840e4161bb}" error="failed to connect to peer: 35377f8d-37f5-272f-9f4f-4c840e4161bb" term=4 +2025-10-30T18:23:08.098Z [WARN] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: Election timeout reached, restarting election +2025-10-30T18:23:08.098Z [INFO] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: entering candidate state: node="Node at 35377f8d-37f5-272f-9f4f-4c840e4161bb [Candidate]" term=4 +2025-10-30T18:23:08.098Z [ERROR] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: failed to make requestVote RPC: target="{Voter server-6eaf8def-756d-0632-01ba-e4f297e64e9a 6eaf8def-756d-0632-01ba-e4f297e64e9a}" error="failed to connect to peer: 6eaf8def-756d-0632-01ba-e4f297e64e9a" term=4 +2025-10-30T18:23:08.109Z [WARN] server-6eaf8def-756d-0632-01ba-e4f297e64e9a: Election timeout reached, restarting election +2025-10-30T18:23:08.109Z [INFO] server-6eaf8def-756d-0632-01ba-e4f297e64e9a: entering candidate state: node="Node at 6eaf8def-756d-0632-01ba-e4f297e64e9a [Candidate]" term=5 +2025-10-30T18:23:08.109Z [ERROR] server-6eaf8def-756d-0632-01ba-e4f297e64e9a: failed to make requestVote RPC: target="{Voter server-35377f8d-37f5-272f-9f4f-4c840e4161bb 35377f8d-37f5-272f-9f4f-4c840e4161bb}" error="failed to connect to peer: 35377f8d-37f5-272f-9f4f-4c840e4161bb" term=5 +2025-10-30T18:23:08.139Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:08.033626465 +0000 UTC m=+111.107155753 (0 nodes), 53.958097ms from start of poll, 375.191262ms from cluster start. Timeout at 2025-10-30 18:23:08.138633571 +0000 UTC m=+111.212162897, 105.007144ms after stability +2025-10-30T18:23:08.151Z [WARN] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: Election timeout reached, restarting election +2025-10-30T18:23:08.151Z [INFO] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: entering candidate state: node="Node at 35377f8d-37f5-272f-9f4f-4c840e4161bb [Candidate]" term=5 +2025-10-30T18:23:08.151Z [ERROR] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: failed to make requestVote RPC: target="{Voter server-6eaf8def-756d-0632-01ba-e4f297e64e9a 6eaf8def-756d-0632-01ba-e4f297e64e9a}" error="failed to connect to peer: 6eaf8def-756d-0632-01ba-e4f297e64e9a" term=5 +2025-10-30T18:23:08.186Z [WARN] server-6eaf8def-756d-0632-01ba-e4f297e64e9a: Election timeout reached, restarting election +2025-10-30T18:23:08.186Z [INFO] server-6eaf8def-756d-0632-01ba-e4f297e64e9a: entering candidate state: node="Node at 6eaf8def-756d-0632-01ba-e4f297e64e9a [Candidate]" term=6 +2025-10-30T18:23:08.186Z [ERROR] server-6eaf8def-756d-0632-01ba-e4f297e64e9a: failed to make requestVote RPC: target="{Voter server-35377f8d-37f5-272f-9f4f-4c840e4161bb 35377f8d-37f5-272f-9f4f-4c840e4161bb}" error="failed to connect to peer: 35377f8d-37f5-272f-9f4f-4c840e4161bb" term=6 +2025-10-30T18:23:08.236Z [WARN] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: Election timeout reached, restarting election +2025-10-30T18:23:08.236Z [INFO] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: entering candidate state: node="Node at 35377f8d-37f5-272f-9f4f-4c840e4161bb [Candidate]" term=6 +2025-10-30T18:23:08.236Z [ERROR] server-35377f8d-37f5-272f-9f4f-4c840e4161bb: failed to make requestVote RPC: target="{Voter server-6eaf8def-756d-0632-01ba-e4f297e64e9a 6eaf8def-756d-0632-01ba-e4f297e64e9a}" error="failed to connect to peer: 6eaf8def-756d-0632-01ba-e4f297e64e9a" term=6 +2025-10-30T18:23:08.238Z [WARN] server-6eaf8def-756d-0632-01ba-e4f297e64e9a: Election timeout reached, restarting election +2025-10-30T18:23:08.238Z [INFO] server-6eaf8def-756d-0632-01ba-e4f297e64e9a: entering candidate state: node="Node at 6eaf8def-756d-0632-01ba-e4f297e64e9a [Candidate]" term=7 +2025-10-30T18:23:08.238Z [ERROR] server-6eaf8def-756d-0632-01ba-e4f297e64e9a: failed to make requestVote RPC: target="{Voter server-35377f8d-37f5-272f-9f4f-4c840e4161bb 35377f8d-37f5-272f-9f4f-4c840e4161bb}" error="failed to connect to peer: 35377f8d-37f5-272f-9f4f-4c840e4161bb" term=7 +--- PASS: TestRaft_LeaderLeaseExpire (0.59s) === RUN TestRaft_Barrier -2025-10-30T18:19:14.734Z [INFO] server-8cf18d3e-1e0d-d84e-d353-cae4f5c353dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8cf18d3e-1e0d-d84e-d353-cae4f5c353dd Address:8cf18d3e-1e0d-d84e-d353-cae4f5c353dd} {Suffrage:Voter ID:server-711dc642-b30b-8ad0-6681-e3d4769045b2 Address:711dc642-b30b-8ad0-6681-e3d4769045b2} {Suffrage:Voter ID:server-88924579-3ef7-7d49-cbc3-d7054a5eba5f Address:88924579-3ef7-7d49-cbc3-d7054a5eba5f}]" -2025-10-30T18:19:14.734Z [INFO] server-711dc642-b30b-8ad0-6681-e3d4769045b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8cf18d3e-1e0d-d84e-d353-cae4f5c353dd Address:8cf18d3e-1e0d-d84e-d353-cae4f5c353dd} {Suffrage:Voter ID:server-711dc642-b30b-8ad0-6681-e3d4769045b2 Address:711dc642-b30b-8ad0-6681-e3d4769045b2} {Suffrage:Voter ID:server-88924579-3ef7-7d49-cbc3-d7054a5eba5f Address:88924579-3ef7-7d49-cbc3-d7054a5eba5f}]" -2025-10-30T18:19:14.734Z [INFO] server-8cf18d3e-1e0d-d84e-d353-cae4f5c353dd: entering follower state: follower="Node at 8cf18d3e-1e0d-d84e-d353-cae4f5c353dd [Follower]" leader-address= leader-id= -2025-10-30T18:19:14.734Z [INFO] server-711dc642-b30b-8ad0-6681-e3d4769045b2: entering follower state: follower="Node at 711dc642-b30b-8ad0-6681-e3d4769045b2 [Follower]" leader-address= leader-id= -2025-10-30T18:19:14.734Z [INFO] server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8cf18d3e-1e0d-d84e-d353-cae4f5c353dd Address:8cf18d3e-1e0d-d84e-d353-cae4f5c353dd} {Suffrage:Voter ID:server-711dc642-b30b-8ad0-6681-e3d4769045b2 Address:711dc642-b30b-8ad0-6681-e3d4769045b2} {Suffrage:Voter ID:server-88924579-3ef7-7d49-cbc3-d7054a5eba5f Address:88924579-3ef7-7d49-cbc3-d7054a5eba5f}]" -2025-10-30T18:19:14.734Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:14.734Z [INFO] server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: entering follower state: follower="Node at 88924579-3ef7-7d49-cbc3-d7054a5eba5f [Follower]" leader-address= leader-id= -2025-10-30T18:19:14.801Z [WARN] server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:14.801Z [INFO] server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: entering candidate state: node="Node at 88924579-3ef7-7d49-cbc3-d7054a5eba5f [Candidate]" term=2 -2025-10-30T18:19:14.801Z [INFO] server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: election won: term=2 tally=2 -2025-10-30T18:19:14.801Z [INFO] server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: entering leader state: leader="Node at 88924579-3ef7-7d49-cbc3-d7054a5eba5f [Leader]" -2025-10-30T18:19:14.801Z [INFO] server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: added peer, starting replication: peer=server-8cf18d3e-1e0d-d84e-d353-cae4f5c353dd -2025-10-30T18:19:14.801Z [INFO] server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: added peer, starting replication: peer=server-711dc642-b30b-8ad0-6681-e3d4769045b2 -2025-10-30T18:19:14.801Z [INFO] server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: pipelining replication: peer="{Voter server-711dc642-b30b-8ad0-6681-e3d4769045b2 711dc642-b30b-8ad0-6681-e3d4769045b2}" -2025-10-30T18:19:14.801Z [INFO] server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: pipelining replication: peer="{Voter server-8cf18d3e-1e0d-d84e-d353-cae4f5c353dd 8cf18d3e-1e0d-d84e-d353-cae4f5c353dd}" -2025-10-30T18:19:14.906Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:14.801299484 +0000 UTC m=+101.079973381 (1 nodes), 67.046642ms from start of poll, 67.328163ms from cluster start. Timeout at 2025-10-30 18:19:14.906302998 +0000 UTC m=+101.184976895, 105.003514ms after stability -2025-10-30T18:19:14.914Z [INFO] server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: aborting pipeline replication: peer="{Voter server-711dc642-b30b-8ad0-6681-e3d4769045b2 711dc642-b30b-8ad0-6681-e3d4769045b2}" -2025-10-30T18:19:14.914Z [INFO] server-88924579-3ef7-7d49-cbc3-d7054a5eba5f: aborting pipeline replication: peer="{Voter server-8cf18d3e-1e0d-d84e-d353-cae4f5c353dd 8cf18d3e-1e0d-d84e-d353-cae4f5c353dd}" +2025-10-30T18:23:08.246Z [INFO] server-ef073c8b-de76-ace5-37d4-d30265feeb42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef073c8b-de76-ace5-37d4-d30265feeb42 Address:ef073c8b-de76-ace5-37d4-d30265feeb42} {Suffrage:Voter ID:server-90d67bc8-db3d-06a0-f235-ff01b65f7251 Address:90d67bc8-db3d-06a0-f235-ff01b65f7251} {Suffrage:Voter ID:server-4b878452-f00a-6e01-8ab7-38bea099ceb3 Address:4b878452-f00a-6e01-8ab7-38bea099ceb3}]" +2025-10-30T18:23:08.246Z [INFO] server-ef073c8b-de76-ace5-37d4-d30265feeb42: entering follower state: follower="Node at ef073c8b-de76-ace5-37d4-d30265feeb42 [Follower]" leader-address= leader-id= +2025-10-30T18:23:08.246Z [INFO] server-90d67bc8-db3d-06a0-f235-ff01b65f7251: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef073c8b-de76-ace5-37d4-d30265feeb42 Address:ef073c8b-de76-ace5-37d4-d30265feeb42} {Suffrage:Voter ID:server-90d67bc8-db3d-06a0-f235-ff01b65f7251 Address:90d67bc8-db3d-06a0-f235-ff01b65f7251} {Suffrage:Voter ID:server-4b878452-f00a-6e01-8ab7-38bea099ceb3 Address:4b878452-f00a-6e01-8ab7-38bea099ceb3}]" +2025-10-30T18:23:08.246Z [INFO] server-90d67bc8-db3d-06a0-f235-ff01b65f7251: entering follower state: follower="Node at 90d67bc8-db3d-06a0-f235-ff01b65f7251 [Follower]" leader-address= leader-id= +2025-10-30T18:23:08.246Z [INFO] server-4b878452-f00a-6e01-8ab7-38bea099ceb3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef073c8b-de76-ace5-37d4-d30265feeb42 Address:ef073c8b-de76-ace5-37d4-d30265feeb42} {Suffrage:Voter ID:server-90d67bc8-db3d-06a0-f235-ff01b65f7251 Address:90d67bc8-db3d-06a0-f235-ff01b65f7251} {Suffrage:Voter ID:server-4b878452-f00a-6e01-8ab7-38bea099ceb3 Address:4b878452-f00a-6e01-8ab7-38bea099ceb3}]" +2025-10-30T18:23:08.246Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:08.246Z [INFO] server-4b878452-f00a-6e01-8ab7-38bea099ceb3: entering follower state: follower="Node at 4b878452-f00a-6e01-8ab7-38bea099ceb3 [Follower]" leader-address= leader-id= +2025-10-30T18:23:08.306Z [WARN] server-4b878452-f00a-6e01-8ab7-38bea099ceb3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:08.306Z [INFO] server-4b878452-f00a-6e01-8ab7-38bea099ceb3: entering candidate state: node="Node at 4b878452-f00a-6e01-8ab7-38bea099ceb3 [Candidate]" term=2 +2025-10-30T18:23:08.306Z [INFO] server-4b878452-f00a-6e01-8ab7-38bea099ceb3: election won: term=2 tally=2 +2025-10-30T18:23:08.306Z [INFO] server-4b878452-f00a-6e01-8ab7-38bea099ceb3: entering leader state: leader="Node at 4b878452-f00a-6e01-8ab7-38bea099ceb3 [Leader]" +2025-10-30T18:23:08.306Z [INFO] server-4b878452-f00a-6e01-8ab7-38bea099ceb3: added peer, starting replication: peer=server-ef073c8b-de76-ace5-37d4-d30265feeb42 +2025-10-30T18:23:08.306Z [INFO] server-4b878452-f00a-6e01-8ab7-38bea099ceb3: added peer, starting replication: peer=server-90d67bc8-db3d-06a0-f235-ff01b65f7251 +2025-10-30T18:23:08.306Z [INFO] server-4b878452-f00a-6e01-8ab7-38bea099ceb3: pipelining replication: peer="{Voter server-90d67bc8-db3d-06a0-f235-ff01b65f7251 90d67bc8-db3d-06a0-f235-ff01b65f7251}" +2025-10-30T18:23:08.306Z [INFO] server-4b878452-f00a-6e01-8ab7-38bea099ceb3: pipelining replication: peer="{Voter server-ef073c8b-de76-ace5-37d4-d30265feeb42 ef073c8b-de76-ace5-37d4-d30265feeb42}" +2025-10-30T18:23:08.412Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:08.306761393 +0000 UTC m=+111.380290722 (1 nodes), 60.389408ms from start of poll, 60.809402ms from cluster start. Timeout at 2025-10-30 18:23:08.411762402 +0000 UTC m=+111.485291690, 105.000968ms after stability +2025-10-30T18:23:08.425Z [INFO] server-4b878452-f00a-6e01-8ab7-38bea099ceb3: aborting pipeline replication: peer="{Voter server-90d67bc8-db3d-06a0-f235-ff01b65f7251 90d67bc8-db3d-06a0-f235-ff01b65f7251}" +2025-10-30T18:23:08.425Z [INFO] server-4b878452-f00a-6e01-8ab7-38bea099ceb3: aborting pipeline replication: peer="{Voter server-ef073c8b-de76-ace5-37d4-d30265feeb42 ef073c8b-de76-ace5-37d4-d30265feeb42}" --- PASS: TestRaft_Barrier (0.18s) === RUN TestRaft_VerifyLeader -2025-10-30T18:19:14.915Z [INFO] server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4 Address:6afa2e74-f001-cc97-84b6-0f31d1ee37c4} {Suffrage:Voter ID:server-a809c745-30a2-6a02-9cc9-76d2a00fac93 Address:a809c745-30a2-6a02-9cc9-76d2a00fac93} {Suffrage:Voter ID:server-108d01e4-d13b-876c-5cf0-0ce4f78224cd Address:108d01e4-d13b-876c-5cf0-0ce4f78224cd}]" -2025-10-30T18:19:14.915Z [INFO] server-a809c745-30a2-6a02-9cc9-76d2a00fac93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4 Address:6afa2e74-f001-cc97-84b6-0f31d1ee37c4} {Suffrage:Voter ID:server-a809c745-30a2-6a02-9cc9-76d2a00fac93 Address:a809c745-30a2-6a02-9cc9-76d2a00fac93} {Suffrage:Voter ID:server-108d01e4-d13b-876c-5cf0-0ce4f78224cd Address:108d01e4-d13b-876c-5cf0-0ce4f78224cd}]" -2025-10-30T18:19:14.915Z [INFO] server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: entering follower state: follower="Node at 6afa2e74-f001-cc97-84b6-0f31d1ee37c4 [Follower]" leader-address= leader-id= -2025-10-30T18:19:14.915Z [INFO] server-a809c745-30a2-6a02-9cc9-76d2a00fac93: entering follower state: follower="Node at a809c745-30a2-6a02-9cc9-76d2a00fac93 [Follower]" leader-address= leader-id= -2025-10-30T18:19:14.915Z [INFO] server-108d01e4-d13b-876c-5cf0-0ce4f78224cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4 Address:6afa2e74-f001-cc97-84b6-0f31d1ee37c4} {Suffrage:Voter ID:server-a809c745-30a2-6a02-9cc9-76d2a00fac93 Address:a809c745-30a2-6a02-9cc9-76d2a00fac93} {Suffrage:Voter ID:server-108d01e4-d13b-876c-5cf0-0ce4f78224cd Address:108d01e4-d13b-876c-5cf0-0ce4f78224cd}]" -2025-10-30T18:19:14.915Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:14.915Z [INFO] server-108d01e4-d13b-876c-5cf0-0ce4f78224cd: entering follower state: follower="Node at 108d01e4-d13b-876c-5cf0-0ce4f78224cd [Follower]" leader-address= leader-id= -2025-10-30T18:19:14.975Z [WARN] server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:14.975Z [INFO] server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: entering candidate state: node="Node at 6afa2e74-f001-cc97-84b6-0f31d1ee37c4 [Candidate]" term=2 -2025-10-30T18:19:14.975Z [INFO] server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: election won: term=2 tally=2 -2025-10-30T18:19:14.975Z [INFO] server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: entering leader state: leader="Node at 6afa2e74-f001-cc97-84b6-0f31d1ee37c4 [Leader]" -2025-10-30T18:19:14.975Z [INFO] server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: added peer, starting replication: peer=server-a809c745-30a2-6a02-9cc9-76d2a00fac93 -2025-10-30T18:19:14.975Z [INFO] server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: added peer, starting replication: peer=server-108d01e4-d13b-876c-5cf0-0ce4f78224cd -2025-10-30T18:19:14.975Z [INFO] server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: pipelining replication: peer="{Voter server-a809c745-30a2-6a02-9cc9-76d2a00fac93 a809c745-30a2-6a02-9cc9-76d2a00fac93}" -2025-10-30T18:19:14.975Z [INFO] server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: pipelining replication: peer="{Voter server-108d01e4-d13b-876c-5cf0-0ce4f78224cd 108d01e4-d13b-876c-5cf0-0ce4f78224cd}" -2025-10-30T18:19:15.080Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:14.975169202 +0000 UTC m=+101.253843116 (1 nodes), 59.640239ms from start of poll, 59.905188ms from cluster start. Timeout at 2025-10-30 18:19:15.08018794 +0000 UTC m=+101.358861837, 105.018721ms after stability -2025-10-30T18:19:15.080Z [INFO] server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: aborting pipeline replication: peer="{Voter server-108d01e4-d13b-876c-5cf0-0ce4f78224cd 108d01e4-d13b-876c-5cf0-0ce4f78224cd}" -2025-10-30T18:19:15.080Z [INFO] server-6afa2e74-f001-cc97-84b6-0f31d1ee37c4: aborting pipeline replication: peer="{Voter server-a809c745-30a2-6a02-9cc9-76d2a00fac93 a809c745-30a2-6a02-9cc9-76d2a00fac93}" ---- PASS: TestRaft_VerifyLeader (0.17s) +2025-10-30T18:23:08.426Z [INFO] server-896e608a-2442-a612-1150-03c72d10a6fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-896e608a-2442-a612-1150-03c72d10a6fc Address:896e608a-2442-a612-1150-03c72d10a6fc} {Suffrage:Voter ID:server-ecb9dbeb-e7bd-4829-39ea-bb45b4e361e9 Address:ecb9dbeb-e7bd-4829-39ea-bb45b4e361e9} {Suffrage:Voter ID:server-20531532-1961-3e6f-a909-0c94ab0a863a Address:20531532-1961-3e6f-a909-0c94ab0a863a}]" +2025-10-30T18:23:08.426Z [INFO] server-896e608a-2442-a612-1150-03c72d10a6fc: entering follower state: follower="Node at 896e608a-2442-a612-1150-03c72d10a6fc [Follower]" leader-address= leader-id= +2025-10-30T18:23:08.426Z [INFO] server-ecb9dbeb-e7bd-4829-39ea-bb45b4e361e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-896e608a-2442-a612-1150-03c72d10a6fc Address:896e608a-2442-a612-1150-03c72d10a6fc} {Suffrage:Voter ID:server-ecb9dbeb-e7bd-4829-39ea-bb45b4e361e9 Address:ecb9dbeb-e7bd-4829-39ea-bb45b4e361e9} {Suffrage:Voter ID:server-20531532-1961-3e6f-a909-0c94ab0a863a Address:20531532-1961-3e6f-a909-0c94ab0a863a}]" +2025-10-30T18:23:08.426Z [INFO] server-ecb9dbeb-e7bd-4829-39ea-bb45b4e361e9: entering follower state: follower="Node at ecb9dbeb-e7bd-4829-39ea-bb45b4e361e9 [Follower]" leader-address= leader-id= +2025-10-30T18:23:08.426Z [INFO] server-20531532-1961-3e6f-a909-0c94ab0a863a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-896e608a-2442-a612-1150-03c72d10a6fc Address:896e608a-2442-a612-1150-03c72d10a6fc} {Suffrage:Voter ID:server-ecb9dbeb-e7bd-4829-39ea-bb45b4e361e9 Address:ecb9dbeb-e7bd-4829-39ea-bb45b4e361e9} {Suffrage:Voter ID:server-20531532-1961-3e6f-a909-0c94ab0a863a Address:20531532-1961-3e6f-a909-0c94ab0a863a}]" +2025-10-30T18:23:08.426Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:08.426Z [INFO] server-20531532-1961-3e6f-a909-0c94ab0a863a: entering follower state: follower="Node at 20531532-1961-3e6f-a909-0c94ab0a863a [Follower]" leader-address= leader-id= +2025-10-30T18:23:08.492Z [WARN] server-20531532-1961-3e6f-a909-0c94ab0a863a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:08.492Z [INFO] server-20531532-1961-3e6f-a909-0c94ab0a863a: entering candidate state: node="Node at 20531532-1961-3e6f-a909-0c94ab0a863a [Candidate]" term=2 +2025-10-30T18:23:08.492Z [INFO] server-20531532-1961-3e6f-a909-0c94ab0a863a: election won: term=2 tally=2 +2025-10-30T18:23:08.492Z [INFO] server-20531532-1961-3e6f-a909-0c94ab0a863a: entering leader state: leader="Node at 20531532-1961-3e6f-a909-0c94ab0a863a [Leader]" +2025-10-30T18:23:08.492Z [INFO] server-20531532-1961-3e6f-a909-0c94ab0a863a: added peer, starting replication: peer=server-896e608a-2442-a612-1150-03c72d10a6fc +2025-10-30T18:23:08.492Z [INFO] server-20531532-1961-3e6f-a909-0c94ab0a863a: added peer, starting replication: peer=server-ecb9dbeb-e7bd-4829-39ea-bb45b4e361e9 +2025-10-30T18:23:08.492Z [INFO] server-20531532-1961-3e6f-a909-0c94ab0a863a: pipelining replication: peer="{Voter server-ecb9dbeb-e7bd-4829-39ea-bb45b4e361e9 ecb9dbeb-e7bd-4829-39ea-bb45b4e361e9}" +2025-10-30T18:23:08.492Z [INFO] server-20531532-1961-3e6f-a909-0c94ab0a863a: pipelining replication: peer="{Voter server-896e608a-2442-a612-1150-03c72d10a6fc 896e608a-2442-a612-1150-03c72d10a6fc}" +2025-10-30T18:23:08.597Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:08.492457013 +0000 UTC m=+111.565986308 (1 nodes), 65.505572ms from start of poll, 65.903432ms from cluster start. Timeout at 2025-10-30 18:23:08.59746508 +0000 UTC m=+111.670994406, 105.008098ms after stability +2025-10-30T18:23:08.597Z [INFO] server-20531532-1961-3e6f-a909-0c94ab0a863a: aborting pipeline replication: peer="{Voter server-ecb9dbeb-e7bd-4829-39ea-bb45b4e361e9 ecb9dbeb-e7bd-4829-39ea-bb45b4e361e9}" +2025-10-30T18:23:08.597Z [INFO] server-20531532-1961-3e6f-a909-0c94ab0a863a: aborting pipeline replication: peer="{Voter server-896e608a-2442-a612-1150-03c72d10a6fc 896e608a-2442-a612-1150-03c72d10a6fc}" +--- PASS: TestRaft_VerifyLeader (0.24s) === RUN TestRaft_VerifyLeader_Single -2025-10-30T18:19:15.081Z [INFO] server-f88da151-a65b-93d4-84bb-81ea9c60df4c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f88da151-a65b-93d4-84bb-81ea9c60df4c Address:f88da151-a65b-93d4-84bb-81ea9c60df4c}]" -2025-10-30T18:19:15.081Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:15.081Z [INFO] server-f88da151-a65b-93d4-84bb-81ea9c60df4c: entering follower state: follower="Node at f88da151-a65b-93d4-84bb-81ea9c60df4c [Follower]" leader-address= leader-id= -2025-10-30T18:19:15.160Z [WARN] server-f88da151-a65b-93d4-84bb-81ea9c60df4c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:15.161Z [INFO] server-f88da151-a65b-93d4-84bb-81ea9c60df4c: entering candidate state: node="Node at f88da151-a65b-93d4-84bb-81ea9c60df4c [Candidate]" term=2 -2025-10-30T18:19:15.161Z [INFO] server-f88da151-a65b-93d4-84bb-81ea9c60df4c: election won: term=2 tally=1 -2025-10-30T18:19:15.161Z [INFO] server-f88da151-a65b-93d4-84bb-81ea9c60df4c: entering leader state: leader="Node at f88da151-a65b-93d4-84bb-81ea9c60df4c [Leader]" -2025-10-30T18:19:15.266Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:15.16113857 +0000 UTC m=+101.439812468 (1 nodes), 79.942375ms from start of poll, 80.077021ms from cluster start. Timeout at 2025-10-30 18:19:15.266139208 +0000 UTC m=+101.544813106, 105.000638ms after stability ---- PASS: TestRaft_VerifyLeader_Single (0.19s) +2025-10-30T18:23:08.670Z [INFO] server-fe8ac7c6-376e-754f-a3bd-48b77043d8b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe8ac7c6-376e-754f-a3bd-48b77043d8b9 Address:fe8ac7c6-376e-754f-a3bd-48b77043d8b9}]" +2025-10-30T18:23:08.670Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:08.670Z [INFO] server-fe8ac7c6-376e-754f-a3bd-48b77043d8b9: entering follower state: follower="Node at fe8ac7c6-376e-754f-a3bd-48b77043d8b9 [Follower]" leader-address= leader-id= +2025-10-30T18:23:08.723Z [WARN] server-fe8ac7c6-376e-754f-a3bd-48b77043d8b9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:08.723Z [INFO] server-fe8ac7c6-376e-754f-a3bd-48b77043d8b9: entering candidate state: node="Node at fe8ac7c6-376e-754f-a3bd-48b77043d8b9 [Candidate]" term=2 +2025-10-30T18:23:08.723Z [INFO] server-fe8ac7c6-376e-754f-a3bd-48b77043d8b9: election won: term=2 tally=1 +2025-10-30T18:23:08.724Z [INFO] server-fe8ac7c6-376e-754f-a3bd-48b77043d8b9: entering leader state: leader="Node at fe8ac7c6-376e-754f-a3bd-48b77043d8b9 [Leader]" +2025-10-30T18:23:08.829Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:08.724055602 +0000 UTC m=+111.797584901 (1 nodes), 53.678381ms from start of poll, 53.819837ms from cluster start. Timeout at 2025-10-30 18:23:08.829066981 +0000 UTC m=+111.902596317, 105.011416ms after stability +--- PASS: TestRaft_VerifyLeader_Single (0.33s) === RUN TestRaft_VerifyLeader_Fail -2025-10-30T18:19:15.267Z [INFO] server-a330f9ea-7751-d1dc-7e24-68647d055d99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a330f9ea-7751-d1dc-7e24-68647d055d99 Address:a330f9ea-7751-d1dc-7e24-68647d055d99} {Suffrage:Voter ID:server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a Address:b96e07e7-ed22-c5bc-b36e-b6524d2eb36a}]" -2025-10-30T18:19:15.267Z [INFO] server-a330f9ea-7751-d1dc-7e24-68647d055d99: entering follower state: follower="Node at a330f9ea-7751-d1dc-7e24-68647d055d99 [Follower]" leader-address= leader-id= -2025-10-30T18:19:15.267Z [INFO] server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a330f9ea-7751-d1dc-7e24-68647d055d99 Address:a330f9ea-7751-d1dc-7e24-68647d055d99} {Suffrage:Voter ID:server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a Address:b96e07e7-ed22-c5bc-b36e-b6524d2eb36a}]" -2025-10-30T18:19:15.267Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:15.267Z [INFO] server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: entering follower state: follower="Node at b96e07e7-ed22-c5bc-b36e-b6524d2eb36a [Follower]" leader-address= leader-id= -2025-10-30T18:19:15.319Z [WARN] server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:15.319Z [WARN] server-a330f9ea-7751-d1dc-7e24-68647d055d99: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:15.319Z [INFO] server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: entering candidate state: node="Node at b96e07e7-ed22-c5bc-b36e-b6524d2eb36a [Candidate]" term=2 -2025-10-30T18:19:15.319Z [INFO] server-a330f9ea-7751-d1dc-7e24-68647d055d99: entering candidate state: node="Node at a330f9ea-7751-d1dc-7e24-68647d055d99 [Candidate]" term=2 -2025-10-30T18:19:15.319Z [INFO] server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: duplicate requestVote for same term: term=2 -2025-10-30T18:19:15.319Z [INFO] server-a330f9ea-7751-d1dc-7e24-68647d055d99: duplicate requestVote for same term: term=2 -2025-10-30T18:19:15.388Z [WARN] server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: Election timeout reached, restarting election -2025-10-30T18:19:15.388Z [INFO] server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: entering candidate state: node="Node at b96e07e7-ed22-c5bc-b36e-b6524d2eb36a [Candidate]" term=3 -2025-10-30T18:19:15.388Z [INFO] server-a330f9ea-7751-d1dc-7e24-68647d055d99: entering follower state: follower="Node at a330f9ea-7751-d1dc-7e24-68647d055d99 [Follower]" leader-address= leader-id= -2025-10-30T18:19:15.388Z [INFO] server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: election won: term=3 tally=2 -2025-10-30T18:19:15.388Z [INFO] server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: entering leader state: leader="Node at b96e07e7-ed22-c5bc-b36e-b6524d2eb36a [Leader]" -2025-10-30T18:19:15.389Z [INFO] server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: added peer, starting replication: peer=server-a330f9ea-7751-d1dc-7e24-68647d055d99 -2025-10-30T18:19:15.389Z [INFO] server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: pipelining replication: peer="{Voter server-a330f9ea-7751-d1dc-7e24-68647d055d99 a330f9ea-7751-d1dc-7e24-68647d055d99}" -2025-10-30T18:19:15.494Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:15.389029851 +0000 UTC m=+101.667703756 (1 nodes), 121.074135ms from start of poll, 121.274325ms from cluster start. Timeout at 2025-10-30 18:19:15.494041355 +0000 UTC m=+101.772715271, 105.011515ms after stability -2025-10-30T18:19:15.494Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:15.600Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:15.494985255 +0000 UTC m=+101.773659152 (1 nodes), 1.733µs from start of poll, 227.229721ms from cluster start. Timeout at 2025-10-30 18:19:15.600004403 +0000 UTC m=+101.878678318, 105.019166ms after stability -2025-10-30T18:19:15.605Z [ERROR] server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: peer has newer term, stopping replication: peer="{Voter server-a330f9ea-7751-d1dc-7e24-68647d055d99 a330f9ea-7751-d1dc-7e24-68647d055d99}" -2025-10-30T18:19:15.605Z [INFO] server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: aborting pipeline replication: peer="{Voter server-a330f9ea-7751-d1dc-7e24-68647d055d99 a330f9ea-7751-d1dc-7e24-68647d055d99}" -2025-10-30T18:19:15.605Z [INFO] server-b96e07e7-ed22-c5bc-b36e-b6524d2eb36a: entering follower state: follower="Node at b96e07e7-ed22-c5bc-b36e-b6524d2eb36a [Follower]" leader-address= leader-id= ---- PASS: TestRaft_VerifyLeader_Fail (0.34s) +2025-10-30T18:23:08.999Z [INFO] server-1e3a3ec4-294a-c4bd-cb53-55d48410452c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e3a3ec4-294a-c4bd-cb53-55d48410452c Address:1e3a3ec4-294a-c4bd-cb53-55d48410452c} {Suffrage:Voter ID:server-5b1f9762-8846-4cd8-e5da-f1d6552f47ae Address:5b1f9762-8846-4cd8-e5da-f1d6552f47ae}]" +2025-10-30T18:23:08.999Z [INFO] server-5b1f9762-8846-4cd8-e5da-f1d6552f47ae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e3a3ec4-294a-c4bd-cb53-55d48410452c Address:1e3a3ec4-294a-c4bd-cb53-55d48410452c} {Suffrage:Voter ID:server-5b1f9762-8846-4cd8-e5da-f1d6552f47ae Address:5b1f9762-8846-4cd8-e5da-f1d6552f47ae}]" +2025-10-30T18:23:08.999Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:08.999Z [INFO] server-1e3a3ec4-294a-c4bd-cb53-55d48410452c: entering follower state: follower="Node at 1e3a3ec4-294a-c4bd-cb53-55d48410452c [Follower]" leader-address= leader-id= +2025-10-30T18:23:08.999Z [INFO] server-5b1f9762-8846-4cd8-e5da-f1d6552f47ae: entering follower state: follower="Node at 5b1f9762-8846-4cd8-e5da-f1d6552f47ae [Follower]" leader-address= leader-id= +2025-10-30T18:23:09.055Z [WARN] server-1e3a3ec4-294a-c4bd-cb53-55d48410452c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:09.055Z [INFO] server-1e3a3ec4-294a-c4bd-cb53-55d48410452c: entering candidate state: node="Node at 1e3a3ec4-294a-c4bd-cb53-55d48410452c [Candidate]" term=2 +2025-10-30T18:23:09.055Z [INFO] server-1e3a3ec4-294a-c4bd-cb53-55d48410452c: election won: term=2 tally=2 +2025-10-30T18:23:09.055Z [INFO] server-1e3a3ec4-294a-c4bd-cb53-55d48410452c: entering leader state: leader="Node at 1e3a3ec4-294a-c4bd-cb53-55d48410452c [Leader]" +2025-10-30T18:23:09.055Z [INFO] server-1e3a3ec4-294a-c4bd-cb53-55d48410452c: added peer, starting replication: peer=server-5b1f9762-8846-4cd8-e5da-f1d6552f47ae +2025-10-30T18:23:09.056Z [INFO] server-1e3a3ec4-294a-c4bd-cb53-55d48410452c: pipelining replication: peer="{Voter server-5b1f9762-8846-4cd8-e5da-f1d6552f47ae 5b1f9762-8846-4cd8-e5da-f1d6552f47ae}" +2025-10-30T18:23:09.160Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:09.055907192 +0000 UTC m=+112.129436486 (1 nodes), 56.570912ms from start of poll, 56.816698ms from cluster start. Timeout at 2025-10-30 18:23:09.160908307 +0000 UTC m=+112.234437600, 105.001114ms after stability +2025-10-30T18:23:09.160Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:09.266Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:09.160979257 +0000 UTC m=+112.234508551 (1 nodes), 859ns from start of poll, 161.888763ms from cluster start. Timeout at 2025-10-30 18:23:09.265990223 +0000 UTC m=+112.339519553, 105.011002ms after stability +2025-10-30T18:23:09.269Z [ERROR] server-1e3a3ec4-294a-c4bd-cb53-55d48410452c: peer has newer term, stopping replication: peer="{Voter server-5b1f9762-8846-4cd8-e5da-f1d6552f47ae 5b1f9762-8846-4cd8-e5da-f1d6552f47ae}" +2025-10-30T18:23:09.269Z [INFO] server-1e3a3ec4-294a-c4bd-cb53-55d48410452c: aborting pipeline replication: peer="{Voter server-5b1f9762-8846-4cd8-e5da-f1d6552f47ae 5b1f9762-8846-4cd8-e5da-f1d6552f47ae}" +2025-10-30T18:23:09.269Z [INFO] server-1e3a3ec4-294a-c4bd-cb53-55d48410452c: entering follower state: follower="Node at 1e3a3ec4-294a-c4bd-cb53-55d48410452c [Follower]" leader-address= leader-id= +--- PASS: TestRaft_VerifyLeader_Fail (0.29s) === RUN TestRaft_VerifyLeader_PartialConnect -2025-10-30T18:19:15.606Z [INFO] server-c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6 Address:c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6} {Suffrage:Voter ID:server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b Address:8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b} {Suffrage:Voter ID:server-a9c50915-fc5f-a683-a5bc-01d4469b8d33 Address:a9c50915-fc5f-a683-a5bc-01d4469b8d33}]" -2025-10-30T18:19:15.606Z [INFO] server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6 Address:c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6} {Suffrage:Voter ID:server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b Address:8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b} {Suffrage:Voter ID:server-a9c50915-fc5f-a683-a5bc-01d4469b8d33 Address:a9c50915-fc5f-a683-a5bc-01d4469b8d33}]" -2025-10-30T18:19:15.606Z [INFO] server-c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6: entering follower state: follower="Node at c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6 [Follower]" leader-address= leader-id= -2025-10-30T18:19:15.606Z [INFO] server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: entering follower state: follower="Node at 8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b [Follower]" leader-address= leader-id= -2025-10-30T18:19:15.606Z [INFO] server-a9c50915-fc5f-a683-a5bc-01d4469b8d33: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6 Address:c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6} {Suffrage:Voter ID:server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b Address:8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b} {Suffrage:Voter ID:server-a9c50915-fc5f-a683-a5bc-01d4469b8d33 Address:a9c50915-fc5f-a683-a5bc-01d4469b8d33}]" -2025-10-30T18:19:15.606Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:15.606Z [INFO] server-a9c50915-fc5f-a683-a5bc-01d4469b8d33: entering follower state: follower="Node at a9c50915-fc5f-a683-a5bc-01d4469b8d33 [Follower]" leader-address= leader-id= -2025-10-30T18:19:15.670Z [WARN] server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:15.670Z [INFO] server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: entering candidate state: node="Node at 8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b [Candidate]" term=2 -2025-10-30T18:19:15.670Z [INFO] server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: election won: term=2 tally=2 -2025-10-30T18:19:15.670Z [INFO] server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: entering leader state: leader="Node at 8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b [Leader]" -2025-10-30T18:19:15.670Z [INFO] server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: added peer, starting replication: peer=server-c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6 -2025-10-30T18:19:15.670Z [INFO] server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: added peer, starting replication: peer=server-a9c50915-fc5f-a683-a5bc-01d4469b8d33 -2025-10-30T18:19:15.670Z [INFO] server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: pipelining replication: peer="{Voter server-a9c50915-fc5f-a683-a5bc-01d4469b8d33 a9c50915-fc5f-a683-a5bc-01d4469b8d33}" -2025-10-30T18:19:15.670Z [INFO] server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: pipelining replication: peer="{Voter server-c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6 c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6}" -2025-10-30T18:19:15.776Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:15.670566719 +0000 UTC m=+101.949240626 (1 nodes), 64.190128ms from start of poll, 64.425618ms from cluster start. Timeout at 2025-10-30 18:19:15.775572524 +0000 UTC m=+102.054246422, 105.005796ms after stability -2025-10-30T18:19:15.781Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:15.887Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:15.781705463 +0000 UTC m=+102.060379360 (2 nodes), 839ns from start of poll, 175.564352ms from cluster start. Timeout at 2025-10-30 18:19:15.88672434 +0000 UTC m=+102.165398255, 105.018895ms after stability - raft_test.go:1968: [INFO] Disconnecting Node at c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6 [Follower] -2025-10-30T18:19:15.887Z [ERROR] server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: failed to heartbeat to: peer=c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6 backoff time=10ms error="failed to connect to peer: c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6" -2025-10-30T18:19:15.887Z [INFO] server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: aborting pipeline replication: peer="{Voter server-c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6 c72ec189-3cdb-2e5e-cc18-2ad5d85a52e6}" -2025-10-30T18:19:15.887Z [INFO] server-8ac3161d-9cfb-5ffe-d0a5-b95cd0646b3b: aborting pipeline replication: peer="{Voter server-a9c50915-fc5f-a683-a5bc-01d4469b8d33 a9c50915-fc5f-a683-a5bc-01d4469b8d33}" ---- PASS: TestRaft_VerifyLeader_PartialConnect (0.28s) +2025-10-30T18:23:09.284Z [INFO] server-05296371-6fab-b0b0-3b5f-26579f967527: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05296371-6fab-b0b0-3b5f-26579f967527 Address:05296371-6fab-b0b0-3b5f-26579f967527} {Suffrage:Voter ID:server-f5b3bae8-7e9a-14cf-f0cd-a86bcda98472 Address:f5b3bae8-7e9a-14cf-f0cd-a86bcda98472} {Suffrage:Voter ID:server-dc63d173-568f-bbd5-1888-f55de4ee9443 Address:dc63d173-568f-bbd5-1888-f55de4ee9443}]" +2025-10-30T18:23:09.284Z [INFO] server-05296371-6fab-b0b0-3b5f-26579f967527: entering follower state: follower="Node at 05296371-6fab-b0b0-3b5f-26579f967527 [Follower]" leader-address= leader-id= +2025-10-30T18:23:09.284Z [INFO] server-f5b3bae8-7e9a-14cf-f0cd-a86bcda98472: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05296371-6fab-b0b0-3b5f-26579f967527 Address:05296371-6fab-b0b0-3b5f-26579f967527} {Suffrage:Voter ID:server-f5b3bae8-7e9a-14cf-f0cd-a86bcda98472 Address:f5b3bae8-7e9a-14cf-f0cd-a86bcda98472} {Suffrage:Voter ID:server-dc63d173-568f-bbd5-1888-f55de4ee9443 Address:dc63d173-568f-bbd5-1888-f55de4ee9443}]" +2025-10-30T18:23:09.284Z [INFO] server-dc63d173-568f-bbd5-1888-f55de4ee9443: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05296371-6fab-b0b0-3b5f-26579f967527 Address:05296371-6fab-b0b0-3b5f-26579f967527} {Suffrage:Voter ID:server-f5b3bae8-7e9a-14cf-f0cd-a86bcda98472 Address:f5b3bae8-7e9a-14cf-f0cd-a86bcda98472} {Suffrage:Voter ID:server-dc63d173-568f-bbd5-1888-f55de4ee9443 Address:dc63d173-568f-bbd5-1888-f55de4ee9443}]" +2025-10-30T18:23:09.284Z [INFO] server-f5b3bae8-7e9a-14cf-f0cd-a86bcda98472: entering follower state: follower="Node at f5b3bae8-7e9a-14cf-f0cd-a86bcda98472 [Follower]" leader-address= leader-id= +2025-10-30T18:23:09.284Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:09.284Z [INFO] server-dc63d173-568f-bbd5-1888-f55de4ee9443: entering follower state: follower="Node at dc63d173-568f-bbd5-1888-f55de4ee9443 [Follower]" leader-address= leader-id= +2025-10-30T18:23:09.365Z [WARN] server-05296371-6fab-b0b0-3b5f-26579f967527: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:09.365Z [INFO] server-05296371-6fab-b0b0-3b5f-26579f967527: entering candidate state: node="Node at 05296371-6fab-b0b0-3b5f-26579f967527 [Candidate]" term=2 +2025-10-30T18:23:09.365Z [INFO] server-05296371-6fab-b0b0-3b5f-26579f967527: election won: term=2 tally=2 +2025-10-30T18:23:09.365Z [INFO] server-05296371-6fab-b0b0-3b5f-26579f967527: entering leader state: leader="Node at 05296371-6fab-b0b0-3b5f-26579f967527 [Leader]" +2025-10-30T18:23:09.365Z [INFO] server-05296371-6fab-b0b0-3b5f-26579f967527: added peer, starting replication: peer=server-f5b3bae8-7e9a-14cf-f0cd-a86bcda98472 +2025-10-30T18:23:09.365Z [INFO] server-05296371-6fab-b0b0-3b5f-26579f967527: added peer, starting replication: peer=server-dc63d173-568f-bbd5-1888-f55de4ee9443 +2025-10-30T18:23:09.365Z [INFO] server-05296371-6fab-b0b0-3b5f-26579f967527: pipelining replication: peer="{Voter server-dc63d173-568f-bbd5-1888-f55de4ee9443 dc63d173-568f-bbd5-1888-f55de4ee9443}" +2025-10-30T18:23:09.365Z [INFO] server-05296371-6fab-b0b0-3b5f-26579f967527: pipelining replication: peer="{Voter server-f5b3bae8-7e9a-14cf-f0cd-a86bcda98472 f5b3bae8-7e9a-14cf-f0cd-a86bcda98472}" +2025-10-30T18:23:09.470Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:09.365641901 +0000 UTC m=+112.439171227 (1 nodes), 81.024608ms from start of poll, 81.360475ms from cluster start. Timeout at 2025-10-30 18:23:09.470645914 +0000 UTC m=+112.544175244, 105.004017ms after stability +2025-10-30T18:23:09.477Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:09.583Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:09.47720553 +0000 UTC m=+112.550734889 (2 nodes), 2.896µs from start of poll, 192.924137ms from cluster start. Timeout at 2025-10-30 18:23:09.582225937 +0000 UTC m=+112.655755313, 105.020424ms after stability + raft_test.go:1968: [INFO] Disconnecting Node at f5b3bae8-7e9a-14cf-f0cd-a86bcda98472 [Follower] +2025-10-30T18:23:09.583Z [ERROR] server-05296371-6fab-b0b0-3b5f-26579f967527: failed to heartbeat to: peer=f5b3bae8-7e9a-14cf-f0cd-a86bcda98472 backoff time=10ms error="failed to connect to peer: f5b3bae8-7e9a-14cf-f0cd-a86bcda98472" +2025-10-30T18:23:09.583Z [INFO] server-05296371-6fab-b0b0-3b5f-26579f967527: aborting pipeline replication: peer="{Voter server-dc63d173-568f-bbd5-1888-f55de4ee9443 dc63d173-568f-bbd5-1888-f55de4ee9443}" +2025-10-30T18:23:09.583Z [INFO] server-05296371-6fab-b0b0-3b5f-26579f967527: aborting pipeline replication: peer="{Voter server-f5b3bae8-7e9a-14cf-f0cd-a86bcda98472 f5b3bae8-7e9a-14cf-f0cd-a86bcda98472}" +--- PASS: TestRaft_VerifyLeader_PartialConnect (0.31s) === RUN TestRaft_NotifyCh -2025-10-30T18:19:15.888Z [INFO] server-37369f83-96dd-21bf-2579-f8fab1b3fbbd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37369f83-96dd-21bf-2579-f8fab1b3fbbd Address:37369f83-96dd-21bf-2579-f8fab1b3fbbd}]" -2025-10-30T18:19:15.888Z [INFO] server-37369f83-96dd-21bf-2579-f8fab1b3fbbd: entering follower state: follower="Node at 37369f83-96dd-21bf-2579-f8fab1b3fbbd [Follower]" leader-address= leader-id= -2025-10-30T18:19:15.958Z [WARN] server-37369f83-96dd-21bf-2579-f8fab1b3fbbd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:15.958Z [INFO] server-37369f83-96dd-21bf-2579-f8fab1b3fbbd: entering candidate state: node="Node at 37369f83-96dd-21bf-2579-f8fab1b3fbbd [Candidate]" term=2 -2025-10-30T18:19:15.958Z [INFO] server-37369f83-96dd-21bf-2579-f8fab1b3fbbd: election won: term=2 tally=1 -2025-10-30T18:19:15.958Z [INFO] server-37369f83-96dd-21bf-2579-f8fab1b3fbbd: entering leader state: leader="Node at 37369f83-96dd-21bf-2579-f8fab1b3fbbd [Leader]" ---- PASS: TestRaft_NotifyCh (0.07s) +2025-10-30T18:23:09.594Z [INFO] server-1e7d3642-2388-4bce-2365-773d4bb7d4f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1e7d3642-2388-4bce-2365-773d4bb7d4f3 Address:1e7d3642-2388-4bce-2365-773d4bb7d4f3}]" +2025-10-30T18:23:09.594Z [INFO] server-1e7d3642-2388-4bce-2365-773d4bb7d4f3: entering follower state: follower="Node at 1e7d3642-2388-4bce-2365-773d4bb7d4f3 [Follower]" leader-address= leader-id= +2025-10-30T18:23:09.667Z [WARN] server-1e7d3642-2388-4bce-2365-773d4bb7d4f3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:09.667Z [INFO] server-1e7d3642-2388-4bce-2365-773d4bb7d4f3: entering candidate state: node="Node at 1e7d3642-2388-4bce-2365-773d4bb7d4f3 [Candidate]" term=2 +2025-10-30T18:23:09.667Z [INFO] server-1e7d3642-2388-4bce-2365-773d4bb7d4f3: election won: term=2 tally=1 +2025-10-30T18:23:09.667Z [INFO] server-1e7d3642-2388-4bce-2365-773d4bb7d4f3: entering leader state: leader="Node at 1e7d3642-2388-4bce-2365-773d4bb7d4f3 [Leader]" +--- PASS: TestRaft_NotifyCh (0.13s) === RUN TestRaft_AppendEntry -2025-10-30T18:19:15.959Z [INFO] server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd Address:b9a79db0-ed00-f7aa-d578-70cbe6cd62cd} {Suffrage:Voter ID:server-1d351432-1cc1-a366-b6ed-6fbb47728198 Address:1d351432-1cc1-a366-b6ed-6fbb47728198} {Suffrage:Voter ID:server-62da28e8-8262-542a-1a41-f01daa8d3931 Address:62da28e8-8262-542a-1a41-f01daa8d3931}]" -2025-10-30T18:19:15.959Z [INFO] server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd: entering follower state: follower="Node at b9a79db0-ed00-f7aa-d578-70cbe6cd62cd [Follower]" leader-address= leader-id= -2025-10-30T18:19:15.959Z [INFO] server-1d351432-1cc1-a366-b6ed-6fbb47728198: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd Address:b9a79db0-ed00-f7aa-d578-70cbe6cd62cd} {Suffrage:Voter ID:server-1d351432-1cc1-a366-b6ed-6fbb47728198 Address:1d351432-1cc1-a366-b6ed-6fbb47728198} {Suffrage:Voter ID:server-62da28e8-8262-542a-1a41-f01daa8d3931 Address:62da28e8-8262-542a-1a41-f01daa8d3931}]" -2025-10-30T18:19:15.959Z [INFO] server-1d351432-1cc1-a366-b6ed-6fbb47728198: entering follower state: follower="Node at 1d351432-1cc1-a366-b6ed-6fbb47728198 [Follower]" leader-address= leader-id= -2025-10-30T18:19:15.959Z [INFO] server-62da28e8-8262-542a-1a41-f01daa8d3931: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd Address:b9a79db0-ed00-f7aa-d578-70cbe6cd62cd} {Suffrage:Voter ID:server-1d351432-1cc1-a366-b6ed-6fbb47728198 Address:1d351432-1cc1-a366-b6ed-6fbb47728198} {Suffrage:Voter ID:server-62da28e8-8262-542a-1a41-f01daa8d3931 Address:62da28e8-8262-542a-1a41-f01daa8d3931}]" -2025-10-30T18:19:15.959Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:15.959Z [INFO] server-62da28e8-8262-542a-1a41-f01daa8d3931: entering follower state: follower="Node at 62da28e8-8262-542a-1a41-f01daa8d3931 [Follower]" leader-address= leader-id= -2025-10-30T18:19:16.033Z [WARN] server-1d351432-1cc1-a366-b6ed-6fbb47728198: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:16.033Z [INFO] server-1d351432-1cc1-a366-b6ed-6fbb47728198: entering candidate state: node="Node at 1d351432-1cc1-a366-b6ed-6fbb47728198 [Candidate]" term=2 -2025-10-30T18:19:16.033Z [INFO] server-1d351432-1cc1-a366-b6ed-6fbb47728198: election won: term=2 tally=2 -2025-10-30T18:19:16.033Z [INFO] server-1d351432-1cc1-a366-b6ed-6fbb47728198: entering leader state: leader="Node at 1d351432-1cc1-a366-b6ed-6fbb47728198 [Leader]" -2025-10-30T18:19:16.033Z [INFO] server-1d351432-1cc1-a366-b6ed-6fbb47728198: added peer, starting replication: peer=server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd -2025-10-30T18:19:16.033Z [INFO] server-1d351432-1cc1-a366-b6ed-6fbb47728198: added peer, starting replication: peer=server-62da28e8-8262-542a-1a41-f01daa8d3931 -2025-10-30T18:19:16.033Z [INFO] server-1d351432-1cc1-a366-b6ed-6fbb47728198: pipelining replication: peer="{Voter server-62da28e8-8262-542a-1a41-f01daa8d3931 62da28e8-8262-542a-1a41-f01daa8d3931}" -2025-10-30T18:19:16.033Z [INFO] server-1d351432-1cc1-a366-b6ed-6fbb47728198: pipelining replication: peer="{Voter server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd b9a79db0-ed00-f7aa-d578-70cbe6cd62cd}" -2025-10-30T18:19:16.139Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:16.033455888 +0000 UTC m=+102.312129786 (2 nodes), 73.727249ms from start of poll, 73.9971ms from cluster start. Timeout at 2025-10-30 18:19:16.138457977 +0000 UTC m=+102.417131874, 105.002088ms after stability -2025-10-30T18:19:16.139Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:16.244Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:16.139410424 +0000 UTC m=+102.418084322 (1 nodes), 1.212µs from start of poll, 179.951636ms from cluster start. Timeout at 2025-10-30 18:19:16.244422031 +0000 UTC m=+102.523095927, 105.011605ms after stability -2025-10-30T18:19:16.244Z [WARN] server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd: clearing log suffix: from=1 to=2 -2025-10-30T18:19:16.244Z [WARN] server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd: skipping application of old log: index=1 -2025-10-30T18:19:16.244Z [WARN] server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd: skipping application of old log: index=1 -2025-10-30T18:19:16.244Z [INFO] server-1d351432-1cc1-a366-b6ed-6fbb47728198: aborting pipeline replication: peer="{Voter server-62da28e8-8262-542a-1a41-f01daa8d3931 62da28e8-8262-542a-1a41-f01daa8d3931}" -2025-10-30T18:19:16.244Z [INFO] server-1d351432-1cc1-a366-b6ed-6fbb47728198: aborting pipeline replication: peer="{Voter server-b9a79db0-ed00-f7aa-d578-70cbe6cd62cd b9a79db0-ed00-f7aa-d578-70cbe6cd62cd}" ---- PASS: TestRaft_AppendEntry (0.29s) +2025-10-30T18:23:09.727Z [INFO] server-0f068a67-09f7-8185-8cbb-d173adba2149: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f068a67-09f7-8185-8cbb-d173adba2149 Address:0f068a67-09f7-8185-8cbb-d173adba2149} {Suffrage:Voter ID:server-6a6ecae5-9bce-2e02-e3a4-e18c5ecbb33c Address:6a6ecae5-9bce-2e02-e3a4-e18c5ecbb33c} {Suffrage:Voter ID:server-fe2b68d0-2db2-927a-a4bc-275b5cde8c40 Address:fe2b68d0-2db2-927a-a4bc-275b5cde8c40}]" +2025-10-30T18:23:09.727Z [INFO] server-0f068a67-09f7-8185-8cbb-d173adba2149: entering follower state: follower="Node at 0f068a67-09f7-8185-8cbb-d173adba2149 [Follower]" leader-address= leader-id= +2025-10-30T18:23:09.727Z [INFO] server-6a6ecae5-9bce-2e02-e3a4-e18c5ecbb33c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f068a67-09f7-8185-8cbb-d173adba2149 Address:0f068a67-09f7-8185-8cbb-d173adba2149} {Suffrage:Voter ID:server-6a6ecae5-9bce-2e02-e3a4-e18c5ecbb33c Address:6a6ecae5-9bce-2e02-e3a4-e18c5ecbb33c} {Suffrage:Voter ID:server-fe2b68d0-2db2-927a-a4bc-275b5cde8c40 Address:fe2b68d0-2db2-927a-a4bc-275b5cde8c40}]" +2025-10-30T18:23:09.727Z [INFO] server-6a6ecae5-9bce-2e02-e3a4-e18c5ecbb33c: entering follower state: follower="Node at 6a6ecae5-9bce-2e02-e3a4-e18c5ecbb33c [Follower]" leader-address= leader-id= +2025-10-30T18:23:09.727Z [INFO] server-fe2b68d0-2db2-927a-a4bc-275b5cde8c40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f068a67-09f7-8185-8cbb-d173adba2149 Address:0f068a67-09f7-8185-8cbb-d173adba2149} {Suffrage:Voter ID:server-6a6ecae5-9bce-2e02-e3a4-e18c5ecbb33c Address:6a6ecae5-9bce-2e02-e3a4-e18c5ecbb33c} {Suffrage:Voter ID:server-fe2b68d0-2db2-927a-a4bc-275b5cde8c40 Address:fe2b68d0-2db2-927a-a4bc-275b5cde8c40}]" +2025-10-30T18:23:09.727Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:09.727Z [INFO] server-fe2b68d0-2db2-927a-a4bc-275b5cde8c40: entering follower state: follower="Node at fe2b68d0-2db2-927a-a4bc-275b5cde8c40 [Follower]" leader-address= leader-id= +2025-10-30T18:23:09.799Z [WARN] server-0f068a67-09f7-8185-8cbb-d173adba2149: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:09.799Z [INFO] server-0f068a67-09f7-8185-8cbb-d173adba2149: entering candidate state: node="Node at 0f068a67-09f7-8185-8cbb-d173adba2149 [Candidate]" term=2 +2025-10-30T18:23:09.800Z [INFO] server-0f068a67-09f7-8185-8cbb-d173adba2149: election won: term=2 tally=2 +2025-10-30T18:23:09.800Z [INFO] server-0f068a67-09f7-8185-8cbb-d173adba2149: entering leader state: leader="Node at 0f068a67-09f7-8185-8cbb-d173adba2149 [Leader]" +2025-10-30T18:23:09.800Z [INFO] server-0f068a67-09f7-8185-8cbb-d173adba2149: added peer, starting replication: peer=server-6a6ecae5-9bce-2e02-e3a4-e18c5ecbb33c +2025-10-30T18:23:09.800Z [INFO] server-0f068a67-09f7-8185-8cbb-d173adba2149: added peer, starting replication: peer=server-fe2b68d0-2db2-927a-a4bc-275b5cde8c40 +2025-10-30T18:23:09.800Z [INFO] server-0f068a67-09f7-8185-8cbb-d173adba2149: pipelining replication: peer="{Voter server-fe2b68d0-2db2-927a-a4bc-275b5cde8c40 fe2b68d0-2db2-927a-a4bc-275b5cde8c40}" +2025-10-30T18:23:09.800Z [INFO] server-0f068a67-09f7-8185-8cbb-d173adba2149: pipelining replication: peer="{Voter server-6a6ecae5-9bce-2e02-e3a4-e18c5ecbb33c 6a6ecae5-9bce-2e02-e3a4-e18c5ecbb33c}" +2025-10-30T18:23:09.905Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:09.800081152 +0000 UTC m=+112.873610450 (2 nodes), 72.614959ms from start of poll, 72.969221ms from cluster start. Timeout at 2025-10-30 18:23:09.905088807 +0000 UTC m=+112.978618097, 105.007647ms after stability +2025-10-30T18:23:09.905Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:10.011Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:09.905878631 +0000 UTC m=+112.979408029 (1 nodes), 1.025µs from start of poll, 178.7668ms from cluster start. Timeout at 2025-10-30 18:23:10.010908782 +0000 UTC m=+113.084438071, 105.030042ms after stability +2025-10-30T18:23:10.011Z [WARN] server-6a6ecae5-9bce-2e02-e3a4-e18c5ecbb33c: clearing log suffix: from=1 to=2 +2025-10-30T18:23:10.011Z [WARN] server-6a6ecae5-9bce-2e02-e3a4-e18c5ecbb33c: skipping application of old log: index=1 +2025-10-30T18:23:10.011Z [WARN] server-6a6ecae5-9bce-2e02-e3a4-e18c5ecbb33c: skipping application of old log: index=1 +2025-10-30T18:23:10.011Z [INFO] server-0f068a67-09f7-8185-8cbb-d173adba2149: aborting pipeline replication: peer="{Voter server-6a6ecae5-9bce-2e02-e3a4-e18c5ecbb33c 6a6ecae5-9bce-2e02-e3a4-e18c5ecbb33c}" +2025-10-30T18:23:10.011Z [INFO] server-0f068a67-09f7-8185-8cbb-d173adba2149: aborting pipeline replication: peer="{Voter server-fe2b68d0-2db2-927a-a4bc-275b5cde8c40 fe2b68d0-2db2-927a-a4bc-275b5cde8c40}" +--- PASS: TestRaft_AppendEntry (0.35s) === RUN TestRaft_VotingGrant_WhenLeaderAvailable -2025-10-30T18:19:16.245Z [INFO] server-4e41b38b-487e-8e96-d1c9-1403a9ab11ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e41b38b-487e-8e96-d1c9-1403a9ab11ba Address:4e41b38b-487e-8e96-d1c9-1403a9ab11ba} {Suffrage:Voter ID:server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4 Address:b7e83b9e-b376-ca23-7e64-6163d1e3dab4} {Suffrage:Voter ID:server-e7b86f93-5674-2aad-7242-c0f0f55c0398 Address:e7b86f93-5674-2aad-7242-c0f0f55c0398}]" -2025-10-30T18:19:16.245Z [INFO] server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e41b38b-487e-8e96-d1c9-1403a9ab11ba Address:4e41b38b-487e-8e96-d1c9-1403a9ab11ba} {Suffrage:Voter ID:server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4 Address:b7e83b9e-b376-ca23-7e64-6163d1e3dab4} {Suffrage:Voter ID:server-e7b86f93-5674-2aad-7242-c0f0f55c0398 Address:e7b86f93-5674-2aad-7242-c0f0f55c0398}]" -2025-10-30T18:19:16.245Z [INFO] server-4e41b38b-487e-8e96-d1c9-1403a9ab11ba: entering follower state: follower="Node at 4e41b38b-487e-8e96-d1c9-1403a9ab11ba [Follower]" leader-address= leader-id= -2025-10-30T18:19:16.245Z [INFO] server-e7b86f93-5674-2aad-7242-c0f0f55c0398: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e41b38b-487e-8e96-d1c9-1403a9ab11ba Address:4e41b38b-487e-8e96-d1c9-1403a9ab11ba} {Suffrage:Voter ID:server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4 Address:b7e83b9e-b376-ca23-7e64-6163d1e3dab4} {Suffrage:Voter ID:server-e7b86f93-5674-2aad-7242-c0f0f55c0398 Address:e7b86f93-5674-2aad-7242-c0f0f55c0398}]" -2025-10-30T18:19:16.245Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:16.245Z [INFO] server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4: entering follower state: follower="Node at b7e83b9e-b376-ca23-7e64-6163d1e3dab4 [Follower]" leader-address= leader-id= -2025-10-30T18:19:16.245Z [INFO] server-e7b86f93-5674-2aad-7242-c0f0f55c0398: entering follower state: follower="Node at e7b86f93-5674-2aad-7242-c0f0f55c0398 [Follower]" leader-address= leader-id= -2025-10-30T18:19:16.298Z [WARN] server-e7b86f93-5674-2aad-7242-c0f0f55c0398: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:16.298Z [INFO] server-e7b86f93-5674-2aad-7242-c0f0f55c0398: entering candidate state: node="Node at e7b86f93-5674-2aad-7242-c0f0f55c0398 [Candidate]" term=2 -2025-10-30T18:19:16.298Z [INFO] server-e7b86f93-5674-2aad-7242-c0f0f55c0398: election won: term=2 tally=2 -2025-10-30T18:19:16.298Z [INFO] server-e7b86f93-5674-2aad-7242-c0f0f55c0398: entering leader state: leader="Node at e7b86f93-5674-2aad-7242-c0f0f55c0398 [Leader]" -2025-10-30T18:19:16.298Z [INFO] server-e7b86f93-5674-2aad-7242-c0f0f55c0398: added peer, starting replication: peer=server-4e41b38b-487e-8e96-d1c9-1403a9ab11ba -2025-10-30T18:19:16.298Z [INFO] server-e7b86f93-5674-2aad-7242-c0f0f55c0398: added peer, starting replication: peer=server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4 -2025-10-30T18:19:16.298Z [INFO] server-e7b86f93-5674-2aad-7242-c0f0f55c0398: pipelining replication: peer="{Voter server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4 b7e83b9e-b376-ca23-7e64-6163d1e3dab4}" -2025-10-30T18:19:16.298Z [INFO] server-e7b86f93-5674-2aad-7242-c0f0f55c0398: pipelining replication: peer="{Voter server-4e41b38b-487e-8e96-d1c9-1403a9ab11ba 4e41b38b-487e-8e96-d1c9-1403a9ab11ba}" -2025-10-30T18:19:16.404Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:16.298494455 +0000 UTC m=+102.577168352 (2 nodes), 52.861663ms from start of poll, 53.027326ms from cluster start. Timeout at 2025-10-30 18:19:16.403497379 +0000 UTC m=+102.682171275, 105.002923ms after stability -2025-10-30T18:19:16.404Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:16.509Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:16.404064949 +0000 UTC m=+102.682738846 (1 nodes), 1.385µs from start of poll, 158.59782ms from cluster start. Timeout at 2025-10-30 18:19:16.509077072 +0000 UTC m=+102.787750986, 105.01214ms after stability -2025-10-30T18:19:16.509Z [WARN] server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4: rejecting vote request since we have a leader: from=4e41b38b-487e-8e96-d1c9-1403a9ab11ba leader=e7b86f93-5674-2aad-7242-c0f0f55c0398 leader-id=server-e7b86f93-5674-2aad-7242-c0f0f55c0398 -2025-10-30T18:19:16.509Z [INFO] server-e7b86f93-5674-2aad-7242-c0f0f55c0398: aborting pipeline replication: peer="{Voter server-b7e83b9e-b376-ca23-7e64-6163d1e3dab4 b7e83b9e-b376-ca23-7e64-6163d1e3dab4}" -2025-10-30T18:19:16.509Z [INFO] server-e7b86f93-5674-2aad-7242-c0f0f55c0398: aborting pipeline replication: peer="{Voter server-4e41b38b-487e-8e96-d1c9-1403a9ab11ba 4e41b38b-487e-8e96-d1c9-1403a9ab11ba}" ---- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.26s) +2025-10-30T18:23:10.081Z [INFO] server-7ff64a3c-96a3-4789-c714-f5b73013ce8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ff64a3c-96a3-4789-c714-f5b73013ce8c Address:7ff64a3c-96a3-4789-c714-f5b73013ce8c} {Suffrage:Voter ID:server-4fda7584-fe3b-a9e8-0689-b146e6470869 Address:4fda7584-fe3b-a9e8-0689-b146e6470869} {Suffrage:Voter ID:server-92aa94b4-dd29-c2b6-e012-e4fa73b706cc Address:92aa94b4-dd29-c2b6-e012-e4fa73b706cc}]" +2025-10-30T18:23:10.081Z [INFO] server-4fda7584-fe3b-a9e8-0689-b146e6470869: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ff64a3c-96a3-4789-c714-f5b73013ce8c Address:7ff64a3c-96a3-4789-c714-f5b73013ce8c} {Suffrage:Voter ID:server-4fda7584-fe3b-a9e8-0689-b146e6470869 Address:4fda7584-fe3b-a9e8-0689-b146e6470869} {Suffrage:Voter ID:server-92aa94b4-dd29-c2b6-e012-e4fa73b706cc Address:92aa94b4-dd29-c2b6-e012-e4fa73b706cc}]" +2025-10-30T18:23:10.081Z [INFO] server-7ff64a3c-96a3-4789-c714-f5b73013ce8c: entering follower state: follower="Node at 7ff64a3c-96a3-4789-c714-f5b73013ce8c [Follower]" leader-address= leader-id= +2025-10-30T18:23:10.081Z [INFO] server-92aa94b4-dd29-c2b6-e012-e4fa73b706cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ff64a3c-96a3-4789-c714-f5b73013ce8c Address:7ff64a3c-96a3-4789-c714-f5b73013ce8c} {Suffrage:Voter ID:server-4fda7584-fe3b-a9e8-0689-b146e6470869 Address:4fda7584-fe3b-a9e8-0689-b146e6470869} {Suffrage:Voter ID:server-92aa94b4-dd29-c2b6-e012-e4fa73b706cc Address:92aa94b4-dd29-c2b6-e012-e4fa73b706cc}]" +2025-10-30T18:23:10.081Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:10.081Z [INFO] server-4fda7584-fe3b-a9e8-0689-b146e6470869: entering follower state: follower="Node at 4fda7584-fe3b-a9e8-0689-b146e6470869 [Follower]" leader-address= leader-id= +2025-10-30T18:23:10.081Z [INFO] server-92aa94b4-dd29-c2b6-e012-e4fa73b706cc: entering follower state: follower="Node at 92aa94b4-dd29-c2b6-e012-e4fa73b706cc [Follower]" leader-address= leader-id= +2025-10-30T18:23:10.153Z [WARN] server-7ff64a3c-96a3-4789-c714-f5b73013ce8c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:10.153Z [INFO] server-7ff64a3c-96a3-4789-c714-f5b73013ce8c: entering candidate state: node="Node at 7ff64a3c-96a3-4789-c714-f5b73013ce8c [Candidate]" term=2 +2025-10-30T18:23:10.153Z [INFO] server-7ff64a3c-96a3-4789-c714-f5b73013ce8c: election won: term=2 tally=2 +2025-10-30T18:23:10.153Z [INFO] server-7ff64a3c-96a3-4789-c714-f5b73013ce8c: entering leader state: leader="Node at 7ff64a3c-96a3-4789-c714-f5b73013ce8c [Leader]" +2025-10-30T18:23:10.153Z [INFO] server-7ff64a3c-96a3-4789-c714-f5b73013ce8c: added peer, starting replication: peer=server-4fda7584-fe3b-a9e8-0689-b146e6470869 +2025-10-30T18:23:10.153Z [INFO] server-7ff64a3c-96a3-4789-c714-f5b73013ce8c: added peer, starting replication: peer=server-92aa94b4-dd29-c2b6-e012-e4fa73b706cc +2025-10-30T18:23:10.153Z [INFO] server-7ff64a3c-96a3-4789-c714-f5b73013ce8c: pipelining replication: peer="{Voter server-4fda7584-fe3b-a9e8-0689-b146e6470869 4fda7584-fe3b-a9e8-0689-b146e6470869}" +2025-10-30T18:23:10.153Z [INFO] server-7ff64a3c-96a3-4789-c714-f5b73013ce8c: pipelining replication: peer="{Voter server-92aa94b4-dd29-c2b6-e012-e4fa73b706cc 92aa94b4-dd29-c2b6-e012-e4fa73b706cc}" +2025-10-30T18:23:10.259Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:10.153258301 +0000 UTC m=+113.226787609 (2 nodes), 71.619288ms from start of poll, 71.906331ms from cluster start. Timeout at 2025-10-30 18:23:10.258261589 +0000 UTC m=+113.331790881, 105.003272ms after stability +2025-10-30T18:23:10.259Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:10.364Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:10.259645742 +0000 UTC m=+113.333175034 (1 nodes), 1.258µs from start of poll, 178.293756ms from cluster start. Timeout at 2025-10-30 18:23:10.364670678 +0000 UTC m=+113.438200071, 105.025037ms after stability +2025-10-30T18:23:10.365Z [WARN] server-92aa94b4-dd29-c2b6-e012-e4fa73b706cc: rejecting vote request since we have a leader: from=4fda7584-fe3b-a9e8-0689-b146e6470869 leader=7ff64a3c-96a3-4789-c714-f5b73013ce8c leader-id=server-7ff64a3c-96a3-4789-c714-f5b73013ce8c +2025-10-30T18:23:10.365Z [INFO] server-7ff64a3c-96a3-4789-c714-f5b73013ce8c: aborting pipeline replication: peer="{Voter server-92aa94b4-dd29-c2b6-e012-e4fa73b706cc 92aa94b4-dd29-c2b6-e012-e4fa73b706cc}" +2025-10-30T18:23:10.365Z [INFO] server-7ff64a3c-96a3-4789-c714-f5b73013ce8c: aborting pipeline replication: peer="{Voter server-4fda7584-fe3b-a9e8-0689-b146e6470869 4fda7584-fe3b-a9e8-0689-b146e6470869}" +--- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.28s) === RUN TestRaft_ProtocolVersion_RejectRPC -2025-10-30T18:19:16.510Z [INFO] server-7030b20f-c40c-43d2-361d-5572c0e40e94: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7030b20f-c40c-43d2-361d-5572c0e40e94 Address:7030b20f-c40c-43d2-361d-5572c0e40e94} {Suffrage:Voter ID:server-c6194370-bdf8-f920-7a3e-e0d629f35383 Address:c6194370-bdf8-f920-7a3e-e0d629f35383} {Suffrage:Voter ID:server-e43aa209-0d35-4525-2170-eb66f4e3cee6 Address:e43aa209-0d35-4525-2170-eb66f4e3cee6}]" -2025-10-30T18:19:16.510Z [INFO] server-c6194370-bdf8-f920-7a3e-e0d629f35383: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7030b20f-c40c-43d2-361d-5572c0e40e94 Address:7030b20f-c40c-43d2-361d-5572c0e40e94} {Suffrage:Voter ID:server-c6194370-bdf8-f920-7a3e-e0d629f35383 Address:c6194370-bdf8-f920-7a3e-e0d629f35383} {Suffrage:Voter ID:server-e43aa209-0d35-4525-2170-eb66f4e3cee6 Address:e43aa209-0d35-4525-2170-eb66f4e3cee6}]" -2025-10-30T18:19:16.510Z [INFO] server-7030b20f-c40c-43d2-361d-5572c0e40e94: entering follower state: follower="Node at 7030b20f-c40c-43d2-361d-5572c0e40e94 [Follower]" leader-address= leader-id= -2025-10-30T18:19:16.510Z [INFO] server-c6194370-bdf8-f920-7a3e-e0d629f35383: entering follower state: follower="Node at c6194370-bdf8-f920-7a3e-e0d629f35383 [Follower]" leader-address= leader-id= -2025-10-30T18:19:16.510Z [INFO] server-e43aa209-0d35-4525-2170-eb66f4e3cee6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7030b20f-c40c-43d2-361d-5572c0e40e94 Address:7030b20f-c40c-43d2-361d-5572c0e40e94} {Suffrage:Voter ID:server-c6194370-bdf8-f920-7a3e-e0d629f35383 Address:c6194370-bdf8-f920-7a3e-e0d629f35383} {Suffrage:Voter ID:server-e43aa209-0d35-4525-2170-eb66f4e3cee6 Address:e43aa209-0d35-4525-2170-eb66f4e3cee6}]" -2025-10-30T18:19:16.510Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:16.510Z [INFO] server-e43aa209-0d35-4525-2170-eb66f4e3cee6: entering follower state: follower="Node at e43aa209-0d35-4525-2170-eb66f4e3cee6 [Follower]" leader-address= leader-id= -2025-10-30T18:19:16.567Z [WARN] server-7030b20f-c40c-43d2-361d-5572c0e40e94: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:16.567Z [INFO] server-7030b20f-c40c-43d2-361d-5572c0e40e94: entering candidate state: node="Node at 7030b20f-c40c-43d2-361d-5572c0e40e94 [Candidate]" term=2 -2025-10-30T18:19:16.567Z [INFO] server-7030b20f-c40c-43d2-361d-5572c0e40e94: election won: term=2 tally=2 -2025-10-30T18:19:16.567Z [INFO] server-7030b20f-c40c-43d2-361d-5572c0e40e94: entering leader state: leader="Node at 7030b20f-c40c-43d2-361d-5572c0e40e94 [Leader]" -2025-10-30T18:19:16.567Z [INFO] server-7030b20f-c40c-43d2-361d-5572c0e40e94: added peer, starting replication: peer=server-c6194370-bdf8-f920-7a3e-e0d629f35383 -2025-10-30T18:19:16.567Z [INFO] server-7030b20f-c40c-43d2-361d-5572c0e40e94: added peer, starting replication: peer=server-e43aa209-0d35-4525-2170-eb66f4e3cee6 -2025-10-30T18:19:16.568Z [INFO] server-7030b20f-c40c-43d2-361d-5572c0e40e94: pipelining replication: peer="{Voter server-e43aa209-0d35-4525-2170-eb66f4e3cee6 e43aa209-0d35-4525-2170-eb66f4e3cee6}" -2025-10-30T18:19:16.568Z [INFO] server-7030b20f-c40c-43d2-361d-5572c0e40e94: pipelining replication: peer="{Voter server-c6194370-bdf8-f920-7a3e-e0d629f35383 c6194370-bdf8-f920-7a3e-e0d629f35383}" -2025-10-30T18:19:16.674Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:16.567998487 +0000 UTC m=+102.846672443 (2 nodes), 57.567362ms from start of poll, 57.754841ms from cluster start. Timeout at 2025-10-30 18:19:16.673007074 +0000 UTC m=+102.951681076, 105.008633ms after stability -2025-10-30T18:19:16.674Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:16.780Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:16.674373653 +0000 UTC m=+102.953047620 (1 nodes), 1.605µs from start of poll, 164.130018ms from cluster start. Timeout at 2025-10-30 18:19:16.779399217 +0000 UTC m=+103.058073115, 105.025495ms after stability -2025-10-30T18:19:16.780Z [INFO] server-7030b20f-c40c-43d2-361d-5572c0e40e94: aborting pipeline replication: peer="{Voter server-e43aa209-0d35-4525-2170-eb66f4e3cee6 e43aa209-0d35-4525-2170-eb66f4e3cee6}" -2025-10-30T18:19:16.780Z [INFO] server-7030b20f-c40c-43d2-361d-5572c0e40e94: aborting pipeline replication: peer="{Voter server-c6194370-bdf8-f920-7a3e-e0d629f35383 c6194370-bdf8-f920-7a3e-e0d629f35383}" +2025-10-30T18:23:10.366Z [INFO] server-df747f96-77e6-aded-bb7f-04358e2416d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df747f96-77e6-aded-bb7f-04358e2416d0 Address:df747f96-77e6-aded-bb7f-04358e2416d0} {Suffrage:Voter ID:server-53dd390a-ee20-3db6-53dc-4f9682338774 Address:53dd390a-ee20-3db6-53dc-4f9682338774} {Suffrage:Voter ID:server-68339132-45f7-0f67-028e-5727f32898a5 Address:68339132-45f7-0f67-028e-5727f32898a5}]" +2025-10-30T18:23:10.366Z [INFO] server-53dd390a-ee20-3db6-53dc-4f9682338774: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df747f96-77e6-aded-bb7f-04358e2416d0 Address:df747f96-77e6-aded-bb7f-04358e2416d0} {Suffrage:Voter ID:server-53dd390a-ee20-3db6-53dc-4f9682338774 Address:53dd390a-ee20-3db6-53dc-4f9682338774} {Suffrage:Voter ID:server-68339132-45f7-0f67-028e-5727f32898a5 Address:68339132-45f7-0f67-028e-5727f32898a5}]" +2025-10-30T18:23:10.366Z [INFO] server-68339132-45f7-0f67-028e-5727f32898a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df747f96-77e6-aded-bb7f-04358e2416d0 Address:df747f96-77e6-aded-bb7f-04358e2416d0} {Suffrage:Voter ID:server-53dd390a-ee20-3db6-53dc-4f9682338774 Address:53dd390a-ee20-3db6-53dc-4f9682338774} {Suffrage:Voter ID:server-68339132-45f7-0f67-028e-5727f32898a5 Address:68339132-45f7-0f67-028e-5727f32898a5}]" +2025-10-30T18:23:10.366Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:10.366Z [INFO] server-53dd390a-ee20-3db6-53dc-4f9682338774: entering follower state: follower="Node at 53dd390a-ee20-3db6-53dc-4f9682338774 [Follower]" leader-address= leader-id= +2025-10-30T18:23:10.366Z [INFO] server-df747f96-77e6-aded-bb7f-04358e2416d0: entering follower state: follower="Node at df747f96-77e6-aded-bb7f-04358e2416d0 [Follower]" leader-address= leader-id= +2025-10-30T18:23:10.366Z [INFO] server-68339132-45f7-0f67-028e-5727f32898a5: entering follower state: follower="Node at 68339132-45f7-0f67-028e-5727f32898a5 [Follower]" leader-address= leader-id= +2025-10-30T18:23:10.424Z [WARN] server-df747f96-77e6-aded-bb7f-04358e2416d0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:10.424Z [INFO] server-df747f96-77e6-aded-bb7f-04358e2416d0: entering candidate state: node="Node at df747f96-77e6-aded-bb7f-04358e2416d0 [Candidate]" term=2 +2025-10-30T18:23:10.424Z [INFO] server-df747f96-77e6-aded-bb7f-04358e2416d0: election won: term=2 tally=2 +2025-10-30T18:23:10.424Z [INFO] server-df747f96-77e6-aded-bb7f-04358e2416d0: entering leader state: leader="Node at df747f96-77e6-aded-bb7f-04358e2416d0 [Leader]" +2025-10-30T18:23:10.424Z [INFO] server-df747f96-77e6-aded-bb7f-04358e2416d0: added peer, starting replication: peer=server-53dd390a-ee20-3db6-53dc-4f9682338774 +2025-10-30T18:23:10.424Z [INFO] server-df747f96-77e6-aded-bb7f-04358e2416d0: added peer, starting replication: peer=server-68339132-45f7-0f67-028e-5727f32898a5 +2025-10-30T18:23:10.424Z [INFO] server-df747f96-77e6-aded-bb7f-04358e2416d0: pipelining replication: peer="{Voter server-68339132-45f7-0f67-028e-5727f32898a5 68339132-45f7-0f67-028e-5727f32898a5}" +2025-10-30T18:23:10.425Z [INFO] server-df747f96-77e6-aded-bb7f-04358e2416d0: pipelining replication: peer="{Voter server-53dd390a-ee20-3db6-53dc-4f9682338774 53dd390a-ee20-3db6-53dc-4f9682338774}" +2025-10-30T18:23:10.530Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:10.424987094 +0000 UTC m=+113.498516383 (2 nodes), 58.475463ms from start of poll, 58.790643ms from cluster start. Timeout at 2025-10-30 18:23:10.52998878 +0000 UTC m=+113.603518072, 105.001689ms after stability +2025-10-30T18:23:10.530Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:10.636Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:10.530792571 +0000 UTC m=+113.604321863 (1 nodes), 1.857µs from start of poll, 164.596123ms from cluster start. Timeout at 2025-10-30 18:23:10.635813852 +0000 UTC m=+113.709343163, 105.0213ms after stability +2025-10-30T18:23:10.636Z [INFO] server-df747f96-77e6-aded-bb7f-04358e2416d0: aborting pipeline replication: peer="{Voter server-53dd390a-ee20-3db6-53dc-4f9682338774 53dd390a-ee20-3db6-53dc-4f9682338774}" +2025-10-30T18:23:10.636Z [INFO] server-df747f96-77e6-aded-bb7f-04358e2416d0: aborting pipeline replication: peer="{Voter server-68339132-45f7-0f67-028e-5727f32898a5 68339132-45f7-0f67-028e-5727f32898a5}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 raft_test.go:2151: Skipping unreliable test @@ -3628,269 +6397,268 @@ raft_test.go:2196: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s) === RUN TestRaft_LeaderID_Propagated -2025-10-30T18:19:16.780Z [INFO] server-9198ff2a-70f9-06e1-00bb-0b84d2192c90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9198ff2a-70f9-06e1-00bb-0b84d2192c90 Address:9198ff2a-70f9-06e1-00bb-0b84d2192c90} {Suffrage:Voter ID:server-ff9db0e7-abab-3741-a819-52041abca8f0 Address:ff9db0e7-abab-3741-a819-52041abca8f0} {Suffrage:Voter ID:server-329d1528-de09-adf4-689b-8190f06b39d8 Address:329d1528-de09-adf4-689b-8190f06b39d8}]" -2025-10-30T18:19:16.781Z [INFO] server-ff9db0e7-abab-3741-a819-52041abca8f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9198ff2a-70f9-06e1-00bb-0b84d2192c90 Address:9198ff2a-70f9-06e1-00bb-0b84d2192c90} {Suffrage:Voter ID:server-ff9db0e7-abab-3741-a819-52041abca8f0 Address:ff9db0e7-abab-3741-a819-52041abca8f0} {Suffrage:Voter ID:server-329d1528-de09-adf4-689b-8190f06b39d8 Address:329d1528-de09-adf4-689b-8190f06b39d8}]" -2025-10-30T18:19:16.781Z [INFO] server-9198ff2a-70f9-06e1-00bb-0b84d2192c90: entering follower state: follower="Node at 9198ff2a-70f9-06e1-00bb-0b84d2192c90 [Follower]" leader-address= leader-id= -2025-10-30T18:19:16.781Z [INFO] server-329d1528-de09-adf4-689b-8190f06b39d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9198ff2a-70f9-06e1-00bb-0b84d2192c90 Address:9198ff2a-70f9-06e1-00bb-0b84d2192c90} {Suffrage:Voter ID:server-ff9db0e7-abab-3741-a819-52041abca8f0 Address:ff9db0e7-abab-3741-a819-52041abca8f0} {Suffrage:Voter ID:server-329d1528-de09-adf4-689b-8190f06b39d8 Address:329d1528-de09-adf4-689b-8190f06b39d8}]" -2025-10-30T18:19:16.781Z [INFO] server-ff9db0e7-abab-3741-a819-52041abca8f0: entering follower state: follower="Node at ff9db0e7-abab-3741-a819-52041abca8f0 [Follower]" leader-address= leader-id= -2025-10-30T18:19:16.781Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:16.781Z [INFO] server-329d1528-de09-adf4-689b-8190f06b39d8: entering follower state: follower="Node at 329d1528-de09-adf4-689b-8190f06b39d8 [Follower]" leader-address= leader-id= -2025-10-30T18:19:16.838Z [WARN] server-ff9db0e7-abab-3741-a819-52041abca8f0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:16.838Z [INFO] server-ff9db0e7-abab-3741-a819-52041abca8f0: entering candidate state: node="Node at ff9db0e7-abab-3741-a819-52041abca8f0 [Candidate]" term=2 -2025-10-30T18:19:16.838Z [INFO] server-ff9db0e7-abab-3741-a819-52041abca8f0: election won: term=2 tally=2 -2025-10-30T18:19:16.838Z [INFO] server-ff9db0e7-abab-3741-a819-52041abca8f0: entering leader state: leader="Node at ff9db0e7-abab-3741-a819-52041abca8f0 [Leader]" -2025-10-30T18:19:16.838Z [INFO] server-ff9db0e7-abab-3741-a819-52041abca8f0: added peer, starting replication: peer=server-9198ff2a-70f9-06e1-00bb-0b84d2192c90 -2025-10-30T18:19:16.838Z [INFO] server-ff9db0e7-abab-3741-a819-52041abca8f0: added peer, starting replication: peer=server-329d1528-de09-adf4-689b-8190f06b39d8 -2025-10-30T18:19:16.838Z [INFO] server-ff9db0e7-abab-3741-a819-52041abca8f0: pipelining replication: peer="{Voter server-329d1528-de09-adf4-689b-8190f06b39d8 329d1528-de09-adf4-689b-8190f06b39d8}" -2025-10-30T18:19:16.838Z [INFO] server-ff9db0e7-abab-3741-a819-52041abca8f0: pipelining replication: peer="{Voter server-9198ff2a-70f9-06e1-00bb-0b84d2192c90 9198ff2a-70f9-06e1-00bb-0b84d2192c90}" -2025-10-30T18:19:16.943Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:16.838186085 +0000 UTC m=+103.116859986 (1 nodes), 57.024257ms from start of poll, 57.282854ms from cluster start. Timeout at 2025-10-30 18:19:16.943190987 +0000 UTC m=+103.221864885, 105.004899ms after stability -2025-10-30T18:19:16.943Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:17.049Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:16.943970108 +0000 UTC m=+103.222644005 (1 nodes), 739ns from start of poll, 163.066873ms from cluster start. Timeout at 2025-10-30 18:19:17.048986705 +0000 UTC m=+103.327660604, 105.016599ms after stability -2025-10-30T18:19:17.049Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:17.154Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:17.049426365 +0000 UTC m=+103.328100264 (1 nodes), 564ns from start of poll, 268.523132ms from cluster start. Timeout at 2025-10-30 18:19:17.154436693 +0000 UTC m=+103.433110619, 105.010355ms after stability -2025-10-30T18:19:17.154Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:17.260Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:17.154889451 +0000 UTC m=+103.433563348 (1 nodes), 483ns from start of poll, 373.986216ms from cluster start. Timeout at 2025-10-30 18:19:17.259907096 +0000 UTC m=+103.538581011, 105.017663ms after stability -2025-10-30T18:19:17.260Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:17.366Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:17.260488994 +0000 UTC m=+103.539162891 (1 nodes), 596ns from start of poll, 479.585759ms from cluster start. Timeout at 2025-10-30 18:19:17.365500804 +0000 UTC m=+103.644174719, 105.011828ms after stability -2025-10-30T18:19:17.366Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:17.472Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:17.366508448 +0000 UTC m=+103.645182345 (1 nodes), 548ns from start of poll, 585.605213ms from cluster start. Timeout at 2025-10-30 18:19:17.471524973 +0000 UTC m=+103.750198888, 105.016543ms after stability -2025-10-30T18:19:17.578Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:17.684Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:17.578218462 +0000 UTC m=+103.856892360 (1 nodes), 1.057µs from start of poll, 797.315228ms from cluster start. Timeout at 2025-10-30 18:19:17.683238543 +0000 UTC m=+103.961912469, 105.020109ms after stability -2025-10-30T18:19:17.684Z [INFO] server-ff9db0e7-abab-3741-a819-52041abca8f0: aborting pipeline replication: peer="{Voter server-329d1528-de09-adf4-689b-8190f06b39d8 329d1528-de09-adf4-689b-8190f06b39d8}" -2025-10-30T18:19:17.684Z [INFO] server-ff9db0e7-abab-3741-a819-52041abca8f0: aborting pipeline replication: peer="{Voter server-9198ff2a-70f9-06e1-00bb-0b84d2192c90 9198ff2a-70f9-06e1-00bb-0b84d2192c90}" +2025-10-30T18:23:10.637Z [INFO] server-efc89374-40d5-6b8e-e142-1f0e447a027c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-efc89374-40d5-6b8e-e142-1f0e447a027c Address:efc89374-40d5-6b8e-e142-1f0e447a027c} {Suffrage:Voter ID:server-849c64fd-d778-f3fc-425c-eed60e9d57ea Address:849c64fd-d778-f3fc-425c-eed60e9d57ea} {Suffrage:Voter ID:server-78166ba5-e16b-a456-3c0f-4a44184f8b11 Address:78166ba5-e16b-a456-3c0f-4a44184f8b11}]" +2025-10-30T18:23:10.637Z [INFO] server-efc89374-40d5-6b8e-e142-1f0e447a027c: entering follower state: follower="Node at efc89374-40d5-6b8e-e142-1f0e447a027c [Follower]" leader-address= leader-id= +2025-10-30T18:23:10.637Z [INFO] server-849c64fd-d778-f3fc-425c-eed60e9d57ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-efc89374-40d5-6b8e-e142-1f0e447a027c Address:efc89374-40d5-6b8e-e142-1f0e447a027c} {Suffrage:Voter ID:server-849c64fd-d778-f3fc-425c-eed60e9d57ea Address:849c64fd-d778-f3fc-425c-eed60e9d57ea} {Suffrage:Voter ID:server-78166ba5-e16b-a456-3c0f-4a44184f8b11 Address:78166ba5-e16b-a456-3c0f-4a44184f8b11}]" +2025-10-30T18:23:10.637Z [INFO] server-78166ba5-e16b-a456-3c0f-4a44184f8b11: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-efc89374-40d5-6b8e-e142-1f0e447a027c Address:efc89374-40d5-6b8e-e142-1f0e447a027c} {Suffrage:Voter ID:server-849c64fd-d778-f3fc-425c-eed60e9d57ea Address:849c64fd-d778-f3fc-425c-eed60e9d57ea} {Suffrage:Voter ID:server-78166ba5-e16b-a456-3c0f-4a44184f8b11 Address:78166ba5-e16b-a456-3c0f-4a44184f8b11}]" +2025-10-30T18:23:10.637Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:10.637Z [INFO] server-849c64fd-d778-f3fc-425c-eed60e9d57ea: entering follower state: follower="Node at 849c64fd-d778-f3fc-425c-eed60e9d57ea [Follower]" leader-address= leader-id= +2025-10-30T18:23:10.637Z [INFO] server-78166ba5-e16b-a456-3c0f-4a44184f8b11: entering follower state: follower="Node at 78166ba5-e16b-a456-3c0f-4a44184f8b11 [Follower]" leader-address= leader-id= +2025-10-30T18:23:10.691Z [WARN] server-78166ba5-e16b-a456-3c0f-4a44184f8b11: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:10.691Z [INFO] server-78166ba5-e16b-a456-3c0f-4a44184f8b11: entering candidate state: node="Node at 78166ba5-e16b-a456-3c0f-4a44184f8b11 [Candidate]" term=2 +2025-10-30T18:23:10.691Z [INFO] server-78166ba5-e16b-a456-3c0f-4a44184f8b11: election won: term=2 tally=2 +2025-10-30T18:23:10.691Z [INFO] server-78166ba5-e16b-a456-3c0f-4a44184f8b11: entering leader state: leader="Node at 78166ba5-e16b-a456-3c0f-4a44184f8b11 [Leader]" +2025-10-30T18:23:10.691Z [INFO] server-78166ba5-e16b-a456-3c0f-4a44184f8b11: added peer, starting replication: peer=server-efc89374-40d5-6b8e-e142-1f0e447a027c +2025-10-30T18:23:10.691Z [INFO] server-78166ba5-e16b-a456-3c0f-4a44184f8b11: added peer, starting replication: peer=server-849c64fd-d778-f3fc-425c-eed60e9d57ea +2025-10-30T18:23:10.691Z [INFO] server-78166ba5-e16b-a456-3c0f-4a44184f8b11: pipelining replication: peer="{Voter server-849c64fd-d778-f3fc-425c-eed60e9d57ea 849c64fd-d778-f3fc-425c-eed60e9d57ea}" +2025-10-30T18:23:10.691Z [INFO] server-78166ba5-e16b-a456-3c0f-4a44184f8b11: pipelining replication: peer="{Voter server-efc89374-40d5-6b8e-e142-1f0e447a027c efc89374-40d5-6b8e-e142-1f0e447a027c}" +2025-10-30T18:23:10.797Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:10.691872602 +0000 UTC m=+113.765401897 (1 nodes), 53.91977ms from start of poll, 54.268207ms from cluster start. Timeout at 2025-10-30 18:23:10.796875575 +0000 UTC m=+113.870404868, 105.002971ms after stability +2025-10-30T18:23:10.797Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:10.903Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:10.797875362 +0000 UTC m=+113.871404667 (1 nodes), 558ns from start of poll, 160.270977ms from cluster start. Timeout at 2025-10-30 18:23:10.90290387 +0000 UTC m=+113.976433200, 105.028533ms after stability +2025-10-30T18:23:10.904Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:11.009Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:10.904216745 +0000 UTC m=+113.977746039 (1 nodes), 753ns from start of poll, 266.612349ms from cluster start. Timeout at 2025-10-30 18:23:11.009250088 +0000 UTC m=+114.082779382, 105.033343ms after stability +2025-10-30T18:23:11.009Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:11.115Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:11.00992915 +0000 UTC m=+114.083458442 (1 nodes), 761ns from start of poll, 372.324752ms from cluster start. Timeout at 2025-10-30 18:23:11.114955906 +0000 UTC m=+114.188485225, 105.026783ms after stability +2025-10-30T18:23:11.115Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:11.221Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:11.115953024 +0000 UTC m=+114.189482318 (1 nodes), 632ns from start of poll, 478.348628ms from cluster start. Timeout at 2025-10-30 18:23:11.220980925 +0000 UTC m=+114.294510218, 105.0279ms after stability +2025-10-30T18:23:11.221Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:11.326Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:11.221231874 +0000 UTC m=+114.294761170 (1 nodes), 689ns from start of poll, 583.62748ms from cluster start. Timeout at 2025-10-30 18:23:11.326258562 +0000 UTC m=+114.399787853, 105.026683ms after stability +2025-10-30T18:23:11.432Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:11.538Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:11.432899153 +0000 UTC m=+114.506428443 (1 nodes), 1.607µs from start of poll, 795.294753ms from cluster start. Timeout at 2025-10-30 18:23:11.537920138 +0000 UTC m=+114.611449427, 105.020984ms after stability +2025-10-30T18:23:11.538Z [INFO] server-78166ba5-e16b-a456-3c0f-4a44184f8b11: aborting pipeline replication: peer="{Voter server-849c64fd-d778-f3fc-425c-eed60e9d57ea 849c64fd-d778-f3fc-425c-eed60e9d57ea}" +2025-10-30T18:23:11.538Z [INFO] server-78166ba5-e16b-a456-3c0f-4a44184f8b11: aborting pipeline replication: peer="{Voter server-efc89374-40d5-6b8e-e142-1f0e447a027c efc89374-40d5-6b8e-e142-1f0e447a027c}" --- PASS: TestRaft_LeaderID_Propagated (0.90s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer -2025-10-30T18:19:17.685Z [INFO] server-47071578-4369-2c98-ab20-01a9096717e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47071578-4369-2c98-ab20-01a9096717e9 Address:47071578-4369-2c98-ab20-01a9096717e9} {Suffrage:Voter ID:server-4866ae6b-dc4a-a1d1-4f2d-15976e391147 Address:4866ae6b-dc4a-a1d1-4f2d-15976e391147} {Suffrage:Voter ID:server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff Address:9d9636fd-26a6-15e4-9ec2-c631de0a94ff}]" -2025-10-30T18:19:17.685Z [INFO] server-47071578-4369-2c98-ab20-01a9096717e9: entering follower state: follower="Node at 47071578-4369-2c98-ab20-01a9096717e9 [Follower]" leader-address= leader-id= -2025-10-30T18:19:17.685Z [INFO] server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47071578-4369-2c98-ab20-01a9096717e9 Address:47071578-4369-2c98-ab20-01a9096717e9} {Suffrage:Voter ID:server-4866ae6b-dc4a-a1d1-4f2d-15976e391147 Address:4866ae6b-dc4a-a1d1-4f2d-15976e391147} {Suffrage:Voter ID:server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff Address:9d9636fd-26a6-15e4-9ec2-c631de0a94ff}]" -2025-10-30T18:19:17.685Z [INFO] server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: entering follower state: follower="Node at 4866ae6b-dc4a-a1d1-4f2d-15976e391147 [Follower]" leader-address= leader-id= -2025-10-30T18:19:17.685Z [INFO] server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47071578-4369-2c98-ab20-01a9096717e9 Address:47071578-4369-2c98-ab20-01a9096717e9} {Suffrage:Voter ID:server-4866ae6b-dc4a-a1d1-4f2d-15976e391147 Address:4866ae6b-dc4a-a1d1-4f2d-15976e391147} {Suffrage:Voter ID:server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff Address:9d9636fd-26a6-15e4-9ec2-c631de0a94ff}]" -2025-10-30T18:19:17.685Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:17.685Z [INFO] server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff: entering follower state: follower="Node at 9d9636fd-26a6-15e4-9ec2-c631de0a94ff [Follower]" leader-address= leader-id= -2025-10-30T18:19:17.747Z [WARN] server-47071578-4369-2c98-ab20-01a9096717e9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:17.747Z [INFO] server-47071578-4369-2c98-ab20-01a9096717e9: entering candidate state: node="Node at 47071578-4369-2c98-ab20-01a9096717e9 [Candidate]" term=2 -2025-10-30T18:19:17.748Z [INFO] server-47071578-4369-2c98-ab20-01a9096717e9: election won: term=2 tally=2 -2025-10-30T18:19:17.748Z [INFO] server-47071578-4369-2c98-ab20-01a9096717e9: entering leader state: leader="Node at 47071578-4369-2c98-ab20-01a9096717e9 [Leader]" -2025-10-30T18:19:17.748Z [INFO] server-47071578-4369-2c98-ab20-01a9096717e9: added peer, starting replication: peer=server-4866ae6b-dc4a-a1d1-4f2d-15976e391147 -2025-10-30T18:19:17.748Z [INFO] server-47071578-4369-2c98-ab20-01a9096717e9: added peer, starting replication: peer=server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff -2025-10-30T18:19:17.748Z [INFO] server-47071578-4369-2c98-ab20-01a9096717e9: pipelining replication: peer="{Voter server-4866ae6b-dc4a-a1d1-4f2d-15976e391147 4866ae6b-dc4a-a1d1-4f2d-15976e391147}" -2025-10-30T18:19:17.748Z [INFO] server-47071578-4369-2c98-ab20-01a9096717e9: pipelining replication: peer="{Voter server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff 9d9636fd-26a6-15e4-9ec2-c631de0a94ff}" -2025-10-30T18:19:17.853Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:17.748056234 +0000 UTC m=+104.026730131 (1 nodes), 62.466257ms from start of poll, 62.712596ms from cluster start. Timeout at 2025-10-30 18:19:17.853057959 +0000 UTC m=+104.131731856, 105.001725ms after stability -2025-10-30T18:19:17.853Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:17.959Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:17.853250683 +0000 UTC m=+104.131924584 (1 nodes), 1.47µs from start of poll, 167.907049ms from cluster start. Timeout at 2025-10-30 18:19:17.958268955 +0000 UTC m=+104.236942875, 105.018291ms after stability -2025-10-30T18:19:17.959Z [INFO] server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: entering candidate state: node="Node at 4866ae6b-dc4a-a1d1-4f2d-15976e391147 [Candidate]" term=3 -2025-10-30T18:19:17.959Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:17.959Z [INFO] server-47071578-4369-2c98-ab20-01a9096717e9: entering follower state: follower="Node at 47071578-4369-2c98-ab20-01a9096717e9 [Follower]" leader-address= leader-id= -2025-10-30T18:19:17.959Z [INFO] server-47071578-4369-2c98-ab20-01a9096717e9: aborting pipeline replication: peer="{Voter server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff 9d9636fd-26a6-15e4-9ec2-c631de0a94ff}" -2025-10-30T18:19:17.959Z [INFO] server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: election won: term=3 tally=2 -2025-10-30T18:19:17.959Z [INFO] server-47071578-4369-2c98-ab20-01a9096717e9: aborting pipeline replication: peer="{Voter server-4866ae6b-dc4a-a1d1-4f2d-15976e391147 4866ae6b-dc4a-a1d1-4f2d-15976e391147}" -2025-10-30T18:19:17.959Z [INFO] server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: entering leader state: leader="Node at 4866ae6b-dc4a-a1d1-4f2d-15976e391147 [Leader]" -2025-10-30T18:19:17.959Z [INFO] server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: added peer, starting replication: peer=server-47071578-4369-2c98-ab20-01a9096717e9 -2025-10-30T18:19:17.959Z [INFO] server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: added peer, starting replication: peer=server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff -2025-10-30T18:19:17.959Z [INFO] server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: pipelining replication: peer="{Voter server-47071578-4369-2c98-ab20-01a9096717e9 47071578-4369-2c98-ab20-01a9096717e9}" -2025-10-30T18:19:17.959Z [INFO] server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: pipelining replication: peer="{Voter server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff 9d9636fd-26a6-15e4-9ec2-c631de0a94ff}" -2025-10-30T18:19:18.064Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:17.959483337 +0000 UTC m=+104.238157237 (1 nodes), 67.887µs from start of poll, 274.139702ms from cluster start. Timeout at 2025-10-30 18:19:18.064560628 +0000 UTC m=+104.343234524, 105.077287ms after stability -2025-10-30T18:19:18.064Z [INFO] server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: aborting pipeline replication: peer="{Voter server-9d9636fd-26a6-15e4-9ec2-c631de0a94ff 9d9636fd-26a6-15e4-9ec2-c631de0a94ff}" -2025-10-30T18:19:18.065Z [INFO] server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: aborting pipeline replication: peer="{Voter server-47071578-4369-2c98-ab20-01a9096717e9 47071578-4369-2c98-ab20-01a9096717e9}" -2025-10-30T18:19:18.565Z [ERROR] server-4866ae6b-dc4a-a1d1-4f2d-15976e391147: failed to heartbeat to: peer=9d9636fd-26a6-15e4-9ec2-c631de0a94ff backoff time=10ms error="command timed out" ---- PASS: TestRaft_LeadershipTransfer (0.88s) +2025-10-30T18:23:11.539Z [INFO] server-e910e356-f493-59f1-f99c-2a1460e356ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e910e356-f493-59f1-f99c-2a1460e356ff Address:e910e356-f493-59f1-f99c-2a1460e356ff} {Suffrage:Voter ID:server-8513d17f-00d6-cb16-910e-34e03dabdf59 Address:8513d17f-00d6-cb16-910e-34e03dabdf59} {Suffrage:Voter ID:server-01521f45-8ecf-17ce-d239-9f435eb908cb Address:01521f45-8ecf-17ce-d239-9f435eb908cb}]" +2025-10-30T18:23:11.539Z [INFO] server-e910e356-f493-59f1-f99c-2a1460e356ff: entering follower state: follower="Node at e910e356-f493-59f1-f99c-2a1460e356ff [Follower]" leader-address= leader-id= +2025-10-30T18:23:11.539Z [INFO] server-8513d17f-00d6-cb16-910e-34e03dabdf59: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e910e356-f493-59f1-f99c-2a1460e356ff Address:e910e356-f493-59f1-f99c-2a1460e356ff} {Suffrage:Voter ID:server-8513d17f-00d6-cb16-910e-34e03dabdf59 Address:8513d17f-00d6-cb16-910e-34e03dabdf59} {Suffrage:Voter ID:server-01521f45-8ecf-17ce-d239-9f435eb908cb Address:01521f45-8ecf-17ce-d239-9f435eb908cb}]" +2025-10-30T18:23:11.539Z [INFO] server-01521f45-8ecf-17ce-d239-9f435eb908cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e910e356-f493-59f1-f99c-2a1460e356ff Address:e910e356-f493-59f1-f99c-2a1460e356ff} {Suffrage:Voter ID:server-8513d17f-00d6-cb16-910e-34e03dabdf59 Address:8513d17f-00d6-cb16-910e-34e03dabdf59} {Suffrage:Voter ID:server-01521f45-8ecf-17ce-d239-9f435eb908cb Address:01521f45-8ecf-17ce-d239-9f435eb908cb}]" +2025-10-30T18:23:11.539Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:11.539Z [INFO] server-8513d17f-00d6-cb16-910e-34e03dabdf59: entering follower state: follower="Node at 8513d17f-00d6-cb16-910e-34e03dabdf59 [Follower]" leader-address= leader-id= +2025-10-30T18:23:11.539Z [INFO] server-01521f45-8ecf-17ce-d239-9f435eb908cb: entering follower state: follower="Node at 01521f45-8ecf-17ce-d239-9f435eb908cb [Follower]" leader-address= leader-id= +2025-10-30T18:23:11.610Z [WARN] server-e910e356-f493-59f1-f99c-2a1460e356ff: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:11.610Z [INFO] server-e910e356-f493-59f1-f99c-2a1460e356ff: entering candidate state: node="Node at e910e356-f493-59f1-f99c-2a1460e356ff [Candidate]" term=2 +2025-10-30T18:23:11.610Z [INFO] server-e910e356-f493-59f1-f99c-2a1460e356ff: election won: term=2 tally=2 +2025-10-30T18:23:11.610Z [INFO] server-e910e356-f493-59f1-f99c-2a1460e356ff: entering leader state: leader="Node at e910e356-f493-59f1-f99c-2a1460e356ff [Leader]" +2025-10-30T18:23:11.610Z [INFO] server-e910e356-f493-59f1-f99c-2a1460e356ff: added peer, starting replication: peer=server-8513d17f-00d6-cb16-910e-34e03dabdf59 +2025-10-30T18:23:11.610Z [INFO] server-e910e356-f493-59f1-f99c-2a1460e356ff: added peer, starting replication: peer=server-01521f45-8ecf-17ce-d239-9f435eb908cb +2025-10-30T18:23:11.610Z [INFO] server-e910e356-f493-59f1-f99c-2a1460e356ff: pipelining replication: peer="{Voter server-01521f45-8ecf-17ce-d239-9f435eb908cb 01521f45-8ecf-17ce-d239-9f435eb908cb}" +2025-10-30T18:23:11.610Z [INFO] server-e910e356-f493-59f1-f99c-2a1460e356ff: pipelining replication: peer="{Voter server-8513d17f-00d6-cb16-910e-34e03dabdf59 8513d17f-00d6-cb16-910e-34e03dabdf59}" +2025-10-30T18:23:11.715Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:11.610704957 +0000 UTC m=+114.684234247 (1 nodes), 70.862086ms from start of poll, 71.191408ms from cluster start. Timeout at 2025-10-30 18:23:11.715710629 +0000 UTC m=+114.789239919, 105.005672ms after stability +2025-10-30T18:23:11.715Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:11.820Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:11.715819893 +0000 UTC m=+114.789349185 (1 nodes), 1.913µs from start of poll, 176.306346ms from cluster start. Timeout at 2025-10-30 18:23:11.82085392 +0000 UTC m=+114.894383247, 105.034062ms after stability +2025-10-30T18:23:11.821Z [INFO] server-8513d17f-00d6-cb16-910e-34e03dabdf59: entering candidate state: node="Node at 8513d17f-00d6-cb16-910e-34e03dabdf59 [Candidate]" term=3 +2025-10-30T18:23:11.821Z [INFO] server-8513d17f-00d6-cb16-910e-34e03dabdf59: election won: term=3 tally=2 +2025-10-30T18:23:11.821Z [INFO] server-8513d17f-00d6-cb16-910e-34e03dabdf59: entering leader state: leader="Node at 8513d17f-00d6-cb16-910e-34e03dabdf59 [Leader]" +2025-10-30T18:23:11.821Z [INFO] server-8513d17f-00d6-cb16-910e-34e03dabdf59: added peer, starting replication: peer=server-e910e356-f493-59f1-f99c-2a1460e356ff +2025-10-30T18:23:11.821Z [INFO] server-8513d17f-00d6-cb16-910e-34e03dabdf59: added peer, starting replication: peer=server-01521f45-8ecf-17ce-d239-9f435eb908cb +2025-10-30T18:23:11.821Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:11.821Z [INFO] server-e910e356-f493-59f1-f99c-2a1460e356ff: entering follower state: follower="Node at e910e356-f493-59f1-f99c-2a1460e356ff [Follower]" leader-address= leader-id= +2025-10-30T18:23:11.821Z [INFO] server-e910e356-f493-59f1-f99c-2a1460e356ff: aborting pipeline replication: peer="{Voter server-8513d17f-00d6-cb16-910e-34e03dabdf59 8513d17f-00d6-cb16-910e-34e03dabdf59}" +2025-10-30T18:23:11.821Z [INFO] server-8513d17f-00d6-cb16-910e-34e03dabdf59: pipelining replication: peer="{Voter server-01521f45-8ecf-17ce-d239-9f435eb908cb 01521f45-8ecf-17ce-d239-9f435eb908cb}" +2025-10-30T18:23:11.821Z [INFO] server-e910e356-f493-59f1-f99c-2a1460e356ff: aborting pipeline replication: peer="{Voter server-01521f45-8ecf-17ce-d239-9f435eb908cb 01521f45-8ecf-17ce-d239-9f435eb908cb}" +2025-10-30T18:23:11.821Z [INFO] server-8513d17f-00d6-cb16-910e-34e03dabdf59: pipelining replication: peer="{Voter server-e910e356-f493-59f1-f99c-2a1460e356ff e910e356-f493-59f1-f99c-2a1460e356ff}" +2025-10-30T18:23:11.926Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:11.821218178 +0000 UTC m=+114.894747475 (1 nodes), 30.732µs from start of poll, 281.704636ms from cluster start. Timeout at 2025-10-30 18:23:11.926229945 +0000 UTC m=+114.999759238, 105.011763ms after stability +2025-10-30T18:23:11.926Z [INFO] server-8513d17f-00d6-cb16-910e-34e03dabdf59: aborting pipeline replication: peer="{Voter server-e910e356-f493-59f1-f99c-2a1460e356ff e910e356-f493-59f1-f99c-2a1460e356ff}" +2025-10-30T18:23:11.926Z [INFO] server-8513d17f-00d6-cb16-910e-34e03dabdf59: aborting pipeline replication: peer="{Voter server-01521f45-8ecf-17ce-d239-9f435eb908cb 01521f45-8ecf-17ce-d239-9f435eb908cb}" +--- PASS: TestRaft_LeadershipTransfer (0.39s) === RUN TestRaft_LeadershipTransferWithOneNode -2025-10-30T18:19:18.566Z [INFO] server-4ece6e09-724c-c5be-a0e3-1710b10f65c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ece6e09-724c-c5be-a0e3-1710b10f65c3 Address:4ece6e09-724c-c5be-a0e3-1710b10f65c3}]" -2025-10-30T18:19:18.566Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:18.566Z [INFO] server-4ece6e09-724c-c5be-a0e3-1710b10f65c3: entering follower state: follower="Node at 4ece6e09-724c-c5be-a0e3-1710b10f65c3 [Follower]" leader-address= leader-id= -2025-10-30T18:19:18.635Z [WARN] server-4ece6e09-724c-c5be-a0e3-1710b10f65c3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:18.635Z [INFO] server-4ece6e09-724c-c5be-a0e3-1710b10f65c3: entering candidate state: node="Node at 4ece6e09-724c-c5be-a0e3-1710b10f65c3 [Candidate]" term=2 -2025-10-30T18:19:18.635Z [INFO] server-4ece6e09-724c-c5be-a0e3-1710b10f65c3: election won: term=2 tally=1 -2025-10-30T18:19:18.635Z [INFO] server-4ece6e09-724c-c5be-a0e3-1710b10f65c3: entering leader state: leader="Node at 4ece6e09-724c-c5be-a0e3-1710b10f65c3 [Leader]" -2025-10-30T18:19:18.740Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:18.635131062 +0000 UTC m=+104.913804973 (1 nodes), 68.737698ms from start of poll, 68.859778ms from cluster start. Timeout at 2025-10-30 18:19:18.740133039 +0000 UTC m=+105.018806936, 105.001963ms after stability ---- PASS: TestRaft_LeadershipTransferWithOneNode (0.17s) +2025-10-30T18:23:11.927Z [INFO] server-629e3650-cb02-8d7d-bbe2-0b3949c61432: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-629e3650-cb02-8d7d-bbe2-0b3949c61432 Address:629e3650-cb02-8d7d-bbe2-0b3949c61432}]" +2025-10-30T18:23:11.927Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:11.927Z [INFO] server-629e3650-cb02-8d7d-bbe2-0b3949c61432: entering follower state: follower="Node at 629e3650-cb02-8d7d-bbe2-0b3949c61432 [Follower]" leader-address= leader-id= +2025-10-30T18:23:12.003Z [WARN] server-629e3650-cb02-8d7d-bbe2-0b3949c61432: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:12.003Z [INFO] server-629e3650-cb02-8d7d-bbe2-0b3949c61432: entering candidate state: node="Node at 629e3650-cb02-8d7d-bbe2-0b3949c61432 [Candidate]" term=2 +2025-10-30T18:23:12.003Z [INFO] server-629e3650-cb02-8d7d-bbe2-0b3949c61432: election won: term=2 tally=1 +2025-10-30T18:23:12.003Z [INFO] server-629e3650-cb02-8d7d-bbe2-0b3949c61432: entering leader state: leader="Node at 629e3650-cb02-8d7d-bbe2-0b3949c61432 [Leader]" +2025-10-30T18:23:12.109Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:12.003735451 +0000 UTC m=+115.077264745 (1 nodes), 76.028333ms from start of poll, 76.158796ms from cluster start. Timeout at 2025-10-30 18:23:12.1087386 +0000 UTC m=+115.182267888, 105.003143ms after stability +--- PASS: TestRaft_LeadershipTransferWithOneNode (0.18s) === RUN TestRaft_LeadershipTransferWithSevenNodes -2025-10-30T18:19:18.741Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6350e70-f150-321b-0246-33eabaad8d7c Address:f6350e70-f150-321b-0246-33eabaad8d7c} {Suffrage:Voter ID:server-ad273008-82c9-92fd-612b-af9ff515bcf2 Address:ad273008-82c9-92fd-612b-af9ff515bcf2} {Suffrage:Voter ID:server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 Address:bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5} {Suffrage:Voter ID:server-8ba7c1e3-74e7-3184-0426-968ea78aa51d Address:8ba7c1e3-74e7-3184-0426-968ea78aa51d} {Suffrage:Voter ID:server-af119d47-ed92-4917-5b41-5baf3da5f549 Address:af119d47-ed92-4917-5b41-5baf3da5f549} {Suffrage:Voter ID:server-627401af-c75a-b265-6083-abf09ffa2855 Address:627401af-c75a-b265-6083-abf09ffa2855} {Suffrage:Voter ID:server-646fc452-30a0-b610-72a7-583056961d5d Address:646fc452-30a0-b610-72a7-583056961d5d}]" -2025-10-30T18:19:18.741Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: entering follower state: follower="Node at f6350e70-f150-321b-0246-33eabaad8d7c [Follower]" leader-address= leader-id= -2025-10-30T18:19:18.741Z [INFO] server-ad273008-82c9-92fd-612b-af9ff515bcf2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6350e70-f150-321b-0246-33eabaad8d7c Address:f6350e70-f150-321b-0246-33eabaad8d7c} {Suffrage:Voter ID:server-ad273008-82c9-92fd-612b-af9ff515bcf2 Address:ad273008-82c9-92fd-612b-af9ff515bcf2} {Suffrage:Voter ID:server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 Address:bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5} {Suffrage:Voter ID:server-8ba7c1e3-74e7-3184-0426-968ea78aa51d Address:8ba7c1e3-74e7-3184-0426-968ea78aa51d} {Suffrage:Voter ID:server-af119d47-ed92-4917-5b41-5baf3da5f549 Address:af119d47-ed92-4917-5b41-5baf3da5f549} {Suffrage:Voter ID:server-627401af-c75a-b265-6083-abf09ffa2855 Address:627401af-c75a-b265-6083-abf09ffa2855} {Suffrage:Voter ID:server-646fc452-30a0-b610-72a7-583056961d5d Address:646fc452-30a0-b610-72a7-583056961d5d}]" -2025-10-30T18:19:18.742Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6350e70-f150-321b-0246-33eabaad8d7c Address:f6350e70-f150-321b-0246-33eabaad8d7c} {Suffrage:Voter ID:server-ad273008-82c9-92fd-612b-af9ff515bcf2 Address:ad273008-82c9-92fd-612b-af9ff515bcf2} {Suffrage:Voter ID:server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 Address:bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5} {Suffrage:Voter ID:server-8ba7c1e3-74e7-3184-0426-968ea78aa51d Address:8ba7c1e3-74e7-3184-0426-968ea78aa51d} {Suffrage:Voter ID:server-af119d47-ed92-4917-5b41-5baf3da5f549 Address:af119d47-ed92-4917-5b41-5baf3da5f549} {Suffrage:Voter ID:server-627401af-c75a-b265-6083-abf09ffa2855 Address:627401af-c75a-b265-6083-abf09ffa2855} {Suffrage:Voter ID:server-646fc452-30a0-b610-72a7-583056961d5d Address:646fc452-30a0-b610-72a7-583056961d5d}]" -2025-10-30T18:19:18.742Z [INFO] server-ad273008-82c9-92fd-612b-af9ff515bcf2: entering follower state: follower="Node at ad273008-82c9-92fd-612b-af9ff515bcf2 [Follower]" leader-address= leader-id= -2025-10-30T18:19:18.742Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: entering follower state: follower="Node at bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 [Follower]" leader-address= leader-id= -2025-10-30T18:19:18.742Z [INFO] server-8ba7c1e3-74e7-3184-0426-968ea78aa51d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6350e70-f150-321b-0246-33eabaad8d7c Address:f6350e70-f150-321b-0246-33eabaad8d7c} {Suffrage:Voter ID:server-ad273008-82c9-92fd-612b-af9ff515bcf2 Address:ad273008-82c9-92fd-612b-af9ff515bcf2} {Suffrage:Voter ID:server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 Address:bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5} {Suffrage:Voter ID:server-8ba7c1e3-74e7-3184-0426-968ea78aa51d Address:8ba7c1e3-74e7-3184-0426-968ea78aa51d} {Suffrage:Voter ID:server-af119d47-ed92-4917-5b41-5baf3da5f549 Address:af119d47-ed92-4917-5b41-5baf3da5f549} {Suffrage:Voter ID:server-627401af-c75a-b265-6083-abf09ffa2855 Address:627401af-c75a-b265-6083-abf09ffa2855} {Suffrage:Voter ID:server-646fc452-30a0-b610-72a7-583056961d5d Address:646fc452-30a0-b610-72a7-583056961d5d}]" -2025-10-30T18:19:18.742Z [INFO] server-8ba7c1e3-74e7-3184-0426-968ea78aa51d: entering follower state: follower="Node at 8ba7c1e3-74e7-3184-0426-968ea78aa51d [Follower]" leader-address= leader-id= -2025-10-30T18:19:18.742Z [INFO] server-af119d47-ed92-4917-5b41-5baf3da5f549: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6350e70-f150-321b-0246-33eabaad8d7c Address:f6350e70-f150-321b-0246-33eabaad8d7c} {Suffrage:Voter ID:server-ad273008-82c9-92fd-612b-af9ff515bcf2 Address:ad273008-82c9-92fd-612b-af9ff515bcf2} {Suffrage:Voter ID:server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 Address:bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5} {Suffrage:Voter ID:server-8ba7c1e3-74e7-3184-0426-968ea78aa51d Address:8ba7c1e3-74e7-3184-0426-968ea78aa51d} {Suffrage:Voter ID:server-af119d47-ed92-4917-5b41-5baf3da5f549 Address:af119d47-ed92-4917-5b41-5baf3da5f549} {Suffrage:Voter ID:server-627401af-c75a-b265-6083-abf09ffa2855 Address:627401af-c75a-b265-6083-abf09ffa2855} {Suffrage:Voter ID:server-646fc452-30a0-b610-72a7-583056961d5d Address:646fc452-30a0-b610-72a7-583056961d5d}]" -2025-10-30T18:19:18.742Z [INFO] server-af119d47-ed92-4917-5b41-5baf3da5f549: entering follower state: follower="Node at af119d47-ed92-4917-5b41-5baf3da5f549 [Follower]" leader-address= leader-id= -2025-10-30T18:19:18.742Z [INFO] server-627401af-c75a-b265-6083-abf09ffa2855: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6350e70-f150-321b-0246-33eabaad8d7c Address:f6350e70-f150-321b-0246-33eabaad8d7c} {Suffrage:Voter ID:server-ad273008-82c9-92fd-612b-af9ff515bcf2 Address:ad273008-82c9-92fd-612b-af9ff515bcf2} {Suffrage:Voter ID:server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 Address:bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5} {Suffrage:Voter ID:server-8ba7c1e3-74e7-3184-0426-968ea78aa51d Address:8ba7c1e3-74e7-3184-0426-968ea78aa51d} {Suffrage:Voter ID:server-af119d47-ed92-4917-5b41-5baf3da5f549 Address:af119d47-ed92-4917-5b41-5baf3da5f549} {Suffrage:Voter ID:server-627401af-c75a-b265-6083-abf09ffa2855 Address:627401af-c75a-b265-6083-abf09ffa2855} {Suffrage:Voter ID:server-646fc452-30a0-b610-72a7-583056961d5d Address:646fc452-30a0-b610-72a7-583056961d5d}]" -2025-10-30T18:19:18.743Z [INFO] server-646fc452-30a0-b610-72a7-583056961d5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6350e70-f150-321b-0246-33eabaad8d7c Address:f6350e70-f150-321b-0246-33eabaad8d7c} {Suffrage:Voter ID:server-ad273008-82c9-92fd-612b-af9ff515bcf2 Address:ad273008-82c9-92fd-612b-af9ff515bcf2} {Suffrage:Voter ID:server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 Address:bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5} {Suffrage:Voter ID:server-8ba7c1e3-74e7-3184-0426-968ea78aa51d Address:8ba7c1e3-74e7-3184-0426-968ea78aa51d} {Suffrage:Voter ID:server-af119d47-ed92-4917-5b41-5baf3da5f549 Address:af119d47-ed92-4917-5b41-5baf3da5f549} {Suffrage:Voter ID:server-627401af-c75a-b265-6083-abf09ffa2855 Address:627401af-c75a-b265-6083-abf09ffa2855} {Suffrage:Voter ID:server-646fc452-30a0-b610-72a7-583056961d5d Address:646fc452-30a0-b610-72a7-583056961d5d}]" -2025-10-30T18:19:18.743Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:18.743Z [INFO] server-627401af-c75a-b265-6083-abf09ffa2855: entering follower state: follower="Node at 627401af-c75a-b265-6083-abf09ffa2855 [Follower]" leader-address= leader-id= -2025-10-30T18:19:18.743Z [INFO] server-646fc452-30a0-b610-72a7-583056961d5d: entering follower state: follower="Node at 646fc452-30a0-b610-72a7-583056961d5d [Follower]" leader-address= leader-id= -2025-10-30T18:19:18.803Z [WARN] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:18.803Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: entering candidate state: node="Node at bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 [Candidate]" term=2 -2025-10-30T18:19:18.803Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: election won: term=2 tally=4 -2025-10-30T18:19:18.803Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: entering leader state: leader="Node at bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 [Leader]" -2025-10-30T18:19:18.803Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: added peer, starting replication: peer=server-f6350e70-f150-321b-0246-33eabaad8d7c -2025-10-30T18:19:18.803Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: added peer, starting replication: peer=server-ad273008-82c9-92fd-612b-af9ff515bcf2 -2025-10-30T18:19:18.803Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: added peer, starting replication: peer=server-8ba7c1e3-74e7-3184-0426-968ea78aa51d -2025-10-30T18:19:18.803Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: added peer, starting replication: peer=server-af119d47-ed92-4917-5b41-5baf3da5f549 -2025-10-30T18:19:18.803Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: added peer, starting replication: peer=server-627401af-c75a-b265-6083-abf09ffa2855 -2025-10-30T18:19:18.803Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: added peer, starting replication: peer=server-646fc452-30a0-b610-72a7-583056961d5d -2025-10-30T18:19:18.803Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: pipelining replication: peer="{Voter server-646fc452-30a0-b610-72a7-583056961d5d 646fc452-30a0-b610-72a7-583056961d5d}" -2025-10-30T18:19:18.803Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: pipelining replication: peer="{Voter server-af119d47-ed92-4917-5b41-5baf3da5f549 af119d47-ed92-4917-5b41-5baf3da5f549}" -2025-10-30T18:19:18.803Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: pipelining replication: peer="{Voter server-627401af-c75a-b265-6083-abf09ffa2855 627401af-c75a-b265-6083-abf09ffa2855}" -2025-10-30T18:19:18.803Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: pipelining replication: peer="{Voter server-f6350e70-f150-321b-0246-33eabaad8d7c f6350e70-f150-321b-0246-33eabaad8d7c}" -2025-10-30T18:19:18.803Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: pipelining replication: peer="{Voter server-ad273008-82c9-92fd-612b-af9ff515bcf2 ad273008-82c9-92fd-612b-af9ff515bcf2}" -2025-10-30T18:19:18.803Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: pipelining replication: peer="{Voter server-8ba7c1e3-74e7-3184-0426-968ea78aa51d 8ba7c1e3-74e7-3184-0426-968ea78aa51d}" -2025-10-30T18:19:18.909Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:18.803553715 +0000 UTC m=+105.082227612 (1 nodes), 60.328763ms from start of poll, 61.734374ms from cluster start. Timeout at 2025-10-30 18:19:18.908555941 +0000 UTC m=+105.187229839, 105.002227ms after stability -2025-10-30T18:19:18.909Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:19.015Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:18.909620838 +0000 UTC m=+105.188294735 (6 nodes), 1.801µs from start of poll, 167.801497ms from cluster start. Timeout at 2025-10-30 18:19:19.014638026 +0000 UTC m=+105.293311922, 105.017187ms after stability -2025-10-30T18:19:19.015Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:19.121Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:19.015318083 +0000 UTC m=+105.293991992 (1 nodes), 2.028µs from start of poll, 273.498754ms from cluster start. Timeout at 2025-10-30 18:19:19.12033941 +0000 UTC m=+105.399013317, 105.021325ms after stability -2025-10-30T18:19:19.121Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: entering candidate state: node="Node at f6350e70-f150-321b-0246-33eabaad8d7c [Candidate]" term=3 -2025-10-30T18:19:19.121Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:19.121Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: election won: term=3 tally=4 -2025-10-30T18:19:19.121Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: entering leader state: leader="Node at f6350e70-f150-321b-0246-33eabaad8d7c [Leader]" -2025-10-30T18:19:19.121Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: added peer, starting replication: peer=server-ad273008-82c9-92fd-612b-af9ff515bcf2 -2025-10-30T18:19:19.121Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: added peer, starting replication: peer=server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 -2025-10-30T18:19:19.121Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: added peer, starting replication: peer=server-8ba7c1e3-74e7-3184-0426-968ea78aa51d -2025-10-30T18:19:19.121Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: added peer, starting replication: peer=server-af119d47-ed92-4917-5b41-5baf3da5f549 -2025-10-30T18:19:19.121Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: added peer, starting replication: peer=server-627401af-c75a-b265-6083-abf09ffa2855 -2025-10-30T18:19:19.121Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: added peer, starting replication: peer=server-646fc452-30a0-b610-72a7-583056961d5d -2025-10-30T18:19:19.121Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: entering follower state: follower="Node at bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 [Follower]" leader-address= leader-id= -2025-10-30T18:19:19.121Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: aborting pipeline replication: peer="{Voter server-af119d47-ed92-4917-5b41-5baf3da5f549 af119d47-ed92-4917-5b41-5baf3da5f549}" -2025-10-30T18:19:19.121Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: aborting pipeline replication: peer="{Voter server-ad273008-82c9-92fd-612b-af9ff515bcf2 ad273008-82c9-92fd-612b-af9ff515bcf2}" -2025-10-30T18:19:19.121Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: pipelining replication: peer="{Voter server-646fc452-30a0-b610-72a7-583056961d5d 646fc452-30a0-b610-72a7-583056961d5d}" -2025-10-30T18:19:19.121Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: pipelining replication: peer="{Voter server-8ba7c1e3-74e7-3184-0426-968ea78aa51d 8ba7c1e3-74e7-3184-0426-968ea78aa51d}" -2025-10-30T18:19:19.121Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: aborting pipeline replication: peer="{Voter server-8ba7c1e3-74e7-3184-0426-968ea78aa51d 8ba7c1e3-74e7-3184-0426-968ea78aa51d}" -2025-10-30T18:19:19.121Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: pipelining replication: peer="{Voter server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5}" -2025-10-30T18:19:19.121Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: aborting pipeline replication: peer="{Voter server-646fc452-30a0-b610-72a7-583056961d5d 646fc452-30a0-b610-72a7-583056961d5d}" -2025-10-30T18:19:19.121Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: pipelining replication: peer="{Voter server-627401af-c75a-b265-6083-abf09ffa2855 627401af-c75a-b265-6083-abf09ffa2855}" -2025-10-30T18:19:19.121Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: aborting pipeline replication: peer="{Voter server-627401af-c75a-b265-6083-abf09ffa2855 627401af-c75a-b265-6083-abf09ffa2855}" -2025-10-30T18:19:19.121Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: pipelining replication: peer="{Voter server-af119d47-ed92-4917-5b41-5baf3da5f549 af119d47-ed92-4917-5b41-5baf3da5f549}" -2025-10-30T18:19:19.121Z [INFO] server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5: aborting pipeline replication: peer="{Voter server-f6350e70-f150-321b-0246-33eabaad8d7c f6350e70-f150-321b-0246-33eabaad8d7c}" -2025-10-30T18:19:19.121Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: pipelining replication: peer="{Voter server-ad273008-82c9-92fd-612b-af9ff515bcf2 ad273008-82c9-92fd-612b-af9ff515bcf2}" -2025-10-30T18:19:19.226Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:19.121376329 +0000 UTC m=+105.400050226 (1 nodes), 73.956µs from start of poll, 379.556988ms from cluster start. Timeout at 2025-10-30 18:19:19.226398439 +0000 UTC m=+105.505072354, 105.022128ms after stability -2025-10-30T18:19:19.227Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: aborting pipeline replication: peer="{Voter server-ad273008-82c9-92fd-612b-af9ff515bcf2 ad273008-82c9-92fd-612b-af9ff515bcf2}" -2025-10-30T18:19:19.227Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: aborting pipeline replication: peer="{Voter server-af119d47-ed92-4917-5b41-5baf3da5f549 af119d47-ed92-4917-5b41-5baf3da5f549}" -2025-10-30T18:19:19.227Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: aborting pipeline replication: peer="{Voter server-627401af-c75a-b265-6083-abf09ffa2855 627401af-c75a-b265-6083-abf09ffa2855}" -2025-10-30T18:19:19.227Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: aborting pipeline replication: peer="{Voter server-bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5 bb289ea5-de2e-3a8b-1c97-3e19dc1ee7a5}" -2025-10-30T18:19:19.227Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: aborting pipeline replication: peer="{Voter server-646fc452-30a0-b610-72a7-583056961d5d 646fc452-30a0-b610-72a7-583056961d5d}" -2025-10-30T18:19:19.227Z [INFO] server-f6350e70-f150-321b-0246-33eabaad8d7c: aborting pipeline replication: peer="{Voter server-8ba7c1e3-74e7-3184-0426-968ea78aa51d 8ba7c1e3-74e7-3184-0426-968ea78aa51d}" -2025-10-30T18:19:19.727Z [ERROR] server-f6350e70-f150-321b-0246-33eabaad8d7c: failed to heartbeat to: peer=ad273008-82c9-92fd-612b-af9ff515bcf2 backoff time=10ms error="command timed out" ---- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.99s) +2025-10-30T18:23:12.110Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd Address:a9b2140a-f479-0dd7-ee02-0d27a6b584bd} {Suffrage:Voter ID:server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca Address:b3871abe-a3ec-f14b-2c06-d0fd52c164ca} {Suffrage:Voter ID:server-616e10b4-7a12-bccc-7e93-342905c5c3ab Address:616e10b4-7a12-bccc-7e93-342905c5c3ab} {Suffrage:Voter ID:server-9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a Address:9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a} {Suffrage:Voter ID:server-cb5afc99-831e-2477-4a75-b931f28f333d Address:cb5afc99-831e-2477-4a75-b931f28f333d} {Suffrage:Voter ID:server-06918917-d944-6882-9ed7-a6d83ca08530 Address:06918917-d944-6882-9ed7-a6d83ca08530} {Suffrage:Voter ID:server-58514aa7-6db4-b415-6105-f3dcf60065dd Address:58514aa7-6db4-b415-6105-f3dcf60065dd}]" +2025-10-30T18:23:12.110Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: entering follower state: follower="Node at a9b2140a-f479-0dd7-ee02-0d27a6b584bd [Follower]" leader-address= leader-id= +2025-10-30T18:23:12.110Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd Address:a9b2140a-f479-0dd7-ee02-0d27a6b584bd} {Suffrage:Voter ID:server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca Address:b3871abe-a3ec-f14b-2c06-d0fd52c164ca} {Suffrage:Voter ID:server-616e10b4-7a12-bccc-7e93-342905c5c3ab Address:616e10b4-7a12-bccc-7e93-342905c5c3ab} {Suffrage:Voter ID:server-9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a Address:9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a} {Suffrage:Voter ID:server-cb5afc99-831e-2477-4a75-b931f28f333d Address:cb5afc99-831e-2477-4a75-b931f28f333d} {Suffrage:Voter ID:server-06918917-d944-6882-9ed7-a6d83ca08530 Address:06918917-d944-6882-9ed7-a6d83ca08530} {Suffrage:Voter ID:server-58514aa7-6db4-b415-6105-f3dcf60065dd Address:58514aa7-6db4-b415-6105-f3dcf60065dd}]" +2025-10-30T18:23:12.110Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: entering follower state: follower="Node at b3871abe-a3ec-f14b-2c06-d0fd52c164ca [Follower]" leader-address= leader-id= +2025-10-30T18:23:12.110Z [INFO] server-616e10b4-7a12-bccc-7e93-342905c5c3ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd Address:a9b2140a-f479-0dd7-ee02-0d27a6b584bd} {Suffrage:Voter ID:server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca Address:b3871abe-a3ec-f14b-2c06-d0fd52c164ca} {Suffrage:Voter ID:server-616e10b4-7a12-bccc-7e93-342905c5c3ab Address:616e10b4-7a12-bccc-7e93-342905c5c3ab} {Suffrage:Voter ID:server-9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a Address:9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a} {Suffrage:Voter ID:server-cb5afc99-831e-2477-4a75-b931f28f333d Address:cb5afc99-831e-2477-4a75-b931f28f333d} {Suffrage:Voter ID:server-06918917-d944-6882-9ed7-a6d83ca08530 Address:06918917-d944-6882-9ed7-a6d83ca08530} {Suffrage:Voter ID:server-58514aa7-6db4-b415-6105-f3dcf60065dd Address:58514aa7-6db4-b415-6105-f3dcf60065dd}]" +2025-10-30T18:23:12.110Z [INFO] server-616e10b4-7a12-bccc-7e93-342905c5c3ab: entering follower state: follower="Node at 616e10b4-7a12-bccc-7e93-342905c5c3ab [Follower]" leader-address= leader-id= +2025-10-30T18:23:12.110Z [INFO] server-9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd Address:a9b2140a-f479-0dd7-ee02-0d27a6b584bd} {Suffrage:Voter ID:server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca Address:b3871abe-a3ec-f14b-2c06-d0fd52c164ca} {Suffrage:Voter ID:server-616e10b4-7a12-bccc-7e93-342905c5c3ab Address:616e10b4-7a12-bccc-7e93-342905c5c3ab} {Suffrage:Voter ID:server-9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a Address:9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a} {Suffrage:Voter ID:server-cb5afc99-831e-2477-4a75-b931f28f333d Address:cb5afc99-831e-2477-4a75-b931f28f333d} {Suffrage:Voter ID:server-06918917-d944-6882-9ed7-a6d83ca08530 Address:06918917-d944-6882-9ed7-a6d83ca08530} {Suffrage:Voter ID:server-58514aa7-6db4-b415-6105-f3dcf60065dd Address:58514aa7-6db4-b415-6105-f3dcf60065dd}]" +2025-10-30T18:23:12.110Z [INFO] server-9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a: entering follower state: follower="Node at 9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a [Follower]" leader-address= leader-id= +2025-10-30T18:23:12.110Z [INFO] server-cb5afc99-831e-2477-4a75-b931f28f333d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd Address:a9b2140a-f479-0dd7-ee02-0d27a6b584bd} {Suffrage:Voter ID:server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca Address:b3871abe-a3ec-f14b-2c06-d0fd52c164ca} {Suffrage:Voter ID:server-616e10b4-7a12-bccc-7e93-342905c5c3ab Address:616e10b4-7a12-bccc-7e93-342905c5c3ab} {Suffrage:Voter ID:server-9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a Address:9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a} {Suffrage:Voter ID:server-cb5afc99-831e-2477-4a75-b931f28f333d Address:cb5afc99-831e-2477-4a75-b931f28f333d} {Suffrage:Voter ID:server-06918917-d944-6882-9ed7-a6d83ca08530 Address:06918917-d944-6882-9ed7-a6d83ca08530} {Suffrage:Voter ID:server-58514aa7-6db4-b415-6105-f3dcf60065dd Address:58514aa7-6db4-b415-6105-f3dcf60065dd}]" +2025-10-30T18:23:12.111Z [INFO] server-cb5afc99-831e-2477-4a75-b931f28f333d: entering follower state: follower="Node at cb5afc99-831e-2477-4a75-b931f28f333d [Follower]" leader-address= leader-id= +2025-10-30T18:23:12.111Z [INFO] server-06918917-d944-6882-9ed7-a6d83ca08530: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd Address:a9b2140a-f479-0dd7-ee02-0d27a6b584bd} {Suffrage:Voter ID:server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca Address:b3871abe-a3ec-f14b-2c06-d0fd52c164ca} {Suffrage:Voter ID:server-616e10b4-7a12-bccc-7e93-342905c5c3ab Address:616e10b4-7a12-bccc-7e93-342905c5c3ab} {Suffrage:Voter ID:server-9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a Address:9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a} {Suffrage:Voter ID:server-cb5afc99-831e-2477-4a75-b931f28f333d Address:cb5afc99-831e-2477-4a75-b931f28f333d} {Suffrage:Voter ID:server-06918917-d944-6882-9ed7-a6d83ca08530 Address:06918917-d944-6882-9ed7-a6d83ca08530} {Suffrage:Voter ID:server-58514aa7-6db4-b415-6105-f3dcf60065dd Address:58514aa7-6db4-b415-6105-f3dcf60065dd}]" +2025-10-30T18:23:12.111Z [INFO] server-06918917-d944-6882-9ed7-a6d83ca08530: entering follower state: follower="Node at 06918917-d944-6882-9ed7-a6d83ca08530 [Follower]" leader-address= leader-id= +2025-10-30T18:23:12.111Z [INFO] server-58514aa7-6db4-b415-6105-f3dcf60065dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd Address:a9b2140a-f479-0dd7-ee02-0d27a6b584bd} {Suffrage:Voter ID:server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca Address:b3871abe-a3ec-f14b-2c06-d0fd52c164ca} {Suffrage:Voter ID:server-616e10b4-7a12-bccc-7e93-342905c5c3ab Address:616e10b4-7a12-bccc-7e93-342905c5c3ab} {Suffrage:Voter ID:server-9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a Address:9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a} {Suffrage:Voter ID:server-cb5afc99-831e-2477-4a75-b931f28f333d Address:cb5afc99-831e-2477-4a75-b931f28f333d} {Suffrage:Voter ID:server-06918917-d944-6882-9ed7-a6d83ca08530 Address:06918917-d944-6882-9ed7-a6d83ca08530} {Suffrage:Voter ID:server-58514aa7-6db4-b415-6105-f3dcf60065dd Address:58514aa7-6db4-b415-6105-f3dcf60065dd}]" +2025-10-30T18:23:12.111Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:12.111Z [INFO] server-58514aa7-6db4-b415-6105-f3dcf60065dd: entering follower state: follower="Node at 58514aa7-6db4-b415-6105-f3dcf60065dd [Follower]" leader-address= leader-id= +2025-10-30T18:23:12.167Z [WARN] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:12.167Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: entering candidate state: node="Node at a9b2140a-f479-0dd7-ee02-0d27a6b584bd [Candidate]" term=2 +2025-10-30T18:23:12.167Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: election won: term=2 tally=4 +2025-10-30T18:23:12.167Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: entering leader state: leader="Node at a9b2140a-f479-0dd7-ee02-0d27a6b584bd [Leader]" +2025-10-30T18:23:12.167Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: added peer, starting replication: peer=server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca +2025-10-30T18:23:12.167Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: added peer, starting replication: peer=server-616e10b4-7a12-bccc-7e93-342905c5c3ab +2025-10-30T18:23:12.167Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: added peer, starting replication: peer=server-9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a +2025-10-30T18:23:12.167Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: added peer, starting replication: peer=server-cb5afc99-831e-2477-4a75-b931f28f333d +2025-10-30T18:23:12.167Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: added peer, starting replication: peer=server-06918917-d944-6882-9ed7-a6d83ca08530 +2025-10-30T18:23:12.167Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: added peer, starting replication: peer=server-58514aa7-6db4-b415-6105-f3dcf60065dd +2025-10-30T18:23:12.167Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: pipelining replication: peer="{Voter server-616e10b4-7a12-bccc-7e93-342905c5c3ab 616e10b4-7a12-bccc-7e93-342905c5c3ab}" +2025-10-30T18:23:12.167Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: pipelining replication: peer="{Voter server-9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a 9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a}" +2025-10-30T18:23:12.167Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: pipelining replication: peer="{Voter server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca b3871abe-a3ec-f14b-2c06-d0fd52c164ca}" +2025-10-30T18:23:12.167Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: pipelining replication: peer="{Voter server-06918917-d944-6882-9ed7-a6d83ca08530 06918917-d944-6882-9ed7-a6d83ca08530}" +2025-10-30T18:23:12.168Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: pipelining replication: peer="{Voter server-58514aa7-6db4-b415-6105-f3dcf60065dd 58514aa7-6db4-b415-6105-f3dcf60065dd}" +2025-10-30T18:23:12.168Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: pipelining replication: peer="{Voter server-cb5afc99-831e-2477-4a75-b931f28f333d cb5afc99-831e-2477-4a75-b931f28f333d}" +2025-10-30T18:23:12.273Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:12.16773979 +0000 UTC m=+115.241269093 (1 nodes), 56.446927ms from start of poll, 57.314288ms from cluster start. Timeout at 2025-10-30 18:23:12.27274782 +0000 UTC m=+115.346277128, 105.008035ms after stability +2025-10-30T18:23:12.273Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:12.378Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:12.273397022 +0000 UTC m=+115.346926318 (6 nodes), 2.383µs from start of poll, 162.971513ms from cluster start. Timeout at 2025-10-30 18:23:12.378435281 +0000 UTC m=+115.451964607, 105.038289ms after stability +2025-10-30T18:23:12.378Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:12.484Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:12.378734052 +0000 UTC m=+115.452263392 (1 nodes), 2.813µs from start of poll, 268.308587ms from cluster start. Timeout at 2025-10-30 18:23:12.483761169 +0000 UTC m=+115.557290495, 105.027103ms after stability +2025-10-30T18:23:12.484Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: entering candidate state: node="Node at b3871abe-a3ec-f14b-2c06-d0fd52c164ca [Candidate]" term=3 +2025-10-30T18:23:12.484Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:12.484Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: election won: term=3 tally=4 +2025-10-30T18:23:12.484Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: entering leader state: leader="Node at b3871abe-a3ec-f14b-2c06-d0fd52c164ca [Leader]" +2025-10-30T18:23:12.484Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: added peer, starting replication: peer=server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd +2025-10-30T18:23:12.484Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: entering follower state: follower="Node at a9b2140a-f479-0dd7-ee02-0d27a6b584bd [Follower]" leader-address= leader-id= +2025-10-30T18:23:12.484Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: aborting pipeline replication: peer="{Voter server-616e10b4-7a12-bccc-7e93-342905c5c3ab 616e10b4-7a12-bccc-7e93-342905c5c3ab}" +2025-10-30T18:23:12.484Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: added peer, starting replication: peer=server-616e10b4-7a12-bccc-7e93-342905c5c3ab +2025-10-30T18:23:12.484Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: aborting pipeline replication: peer="{Voter server-cb5afc99-831e-2477-4a75-b931f28f333d cb5afc99-831e-2477-4a75-b931f28f333d}" +2025-10-30T18:23:12.485Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: added peer, starting replication: peer=server-9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a +2025-10-30T18:23:12.485Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: pipelining replication: peer="{Voter server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd a9b2140a-f479-0dd7-ee02-0d27a6b584bd}" +2025-10-30T18:23:12.484Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: aborting pipeline replication: peer="{Voter server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca b3871abe-a3ec-f14b-2c06-d0fd52c164ca}" +2025-10-30T18:23:12.485Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: pipelining replication: peer="{Voter server-616e10b4-7a12-bccc-7e93-342905c5c3ab 616e10b4-7a12-bccc-7e93-342905c5c3ab}" +2025-10-30T18:23:12.484Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: aborting pipeline replication: peer="{Voter server-9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a 9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a}" +2025-10-30T18:23:12.485Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: pipelining replication: peer="{Voter server-9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a 9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a}" +2025-10-30T18:23:12.484Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: aborting pipeline replication: peer="{Voter server-06918917-d944-6882-9ed7-a6d83ca08530 06918917-d944-6882-9ed7-a6d83ca08530}" +2025-10-30T18:23:12.485Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: added peer, starting replication: peer=server-cb5afc99-831e-2477-4a75-b931f28f333d +2025-10-30T18:23:12.484Z [INFO] server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd: aborting pipeline replication: peer="{Voter server-58514aa7-6db4-b415-6105-f3dcf60065dd 58514aa7-6db4-b415-6105-f3dcf60065dd}" +2025-10-30T18:23:12.485Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: added peer, starting replication: peer=server-06918917-d944-6882-9ed7-a6d83ca08530 +2025-10-30T18:23:12.485Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: added peer, starting replication: peer=server-58514aa7-6db4-b415-6105-f3dcf60065dd +2025-10-30T18:23:12.485Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: pipelining replication: peer="{Voter server-cb5afc99-831e-2477-4a75-b931f28f333d cb5afc99-831e-2477-4a75-b931f28f333d}" +2025-10-30T18:23:12.485Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: pipelining replication: peer="{Voter server-06918917-d944-6882-9ed7-a6d83ca08530 06918917-d944-6882-9ed7-a6d83ca08530}" +2025-10-30T18:23:12.485Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: pipelining replication: peer="{Voter server-58514aa7-6db4-b415-6105-f3dcf60065dd 58514aa7-6db4-b415-6105-f3dcf60065dd}" +2025-10-30T18:23:12.590Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:12.484952348 +0000 UTC m=+115.558481644 (1 nodes), 60.406µs from start of poll, 374.526839ms from cluster start. Timeout at 2025-10-30 18:23:12.589979199 +0000 UTC m=+115.663508523, 105.026879ms after stability +2025-10-30T18:23:12.590Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: aborting pipeline replication: peer="{Voter server-58514aa7-6db4-b415-6105-f3dcf60065dd 58514aa7-6db4-b415-6105-f3dcf60065dd}" +2025-10-30T18:23:12.590Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: aborting pipeline replication: peer="{Voter server-a9b2140a-f479-0dd7-ee02-0d27a6b584bd a9b2140a-f479-0dd7-ee02-0d27a6b584bd}" +2025-10-30T18:23:12.590Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: aborting pipeline replication: peer="{Voter server-cb5afc99-831e-2477-4a75-b931f28f333d cb5afc99-831e-2477-4a75-b931f28f333d}" +2025-10-30T18:23:12.590Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: aborting pipeline replication: peer="{Voter server-06918917-d944-6882-9ed7-a6d83ca08530 06918917-d944-6882-9ed7-a6d83ca08530}" +2025-10-30T18:23:12.590Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: aborting pipeline replication: peer="{Voter server-616e10b4-7a12-bccc-7e93-342905c5c3ab 616e10b4-7a12-bccc-7e93-342905c5c3ab}" +2025-10-30T18:23:12.590Z [INFO] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: aborting pipeline replication: peer="{Voter server-9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a 9b7c866b-8f7d-0daa-79c0-e4b2edcbf39a}" +2025-10-30T18:23:13.090Z [ERROR] server-b3871abe-a3ec-f14b-2c06-d0fd52c164ca: failed to heartbeat to: peer=a9b2140a-f479-0dd7-ee02-0d27a6b584bd backoff time=10ms error="command timed out" +--- PASS: TestRaft_LeadershipTransferWithSevenNodes (1.00s) === RUN TestRaft_LeadershipTransferToInvalidID -2025-10-30T18:19:19.728Z [INFO] server-06d01780-d7b5-3c1b-6649-648a00bb9517: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06d01780-d7b5-3c1b-6649-648a00bb9517 Address:06d01780-d7b5-3c1b-6649-648a00bb9517} {Suffrage:Voter ID:server-c2316925-47e6-dcc4-9570-a76f6f244b07 Address:c2316925-47e6-dcc4-9570-a76f6f244b07} {Suffrage:Voter ID:server-97c82631-dd2e-1258-25b9-5cd8830525ce Address:97c82631-dd2e-1258-25b9-5cd8830525ce}]" -2025-10-30T18:19:19.728Z [INFO] server-06d01780-d7b5-3c1b-6649-648a00bb9517: entering follower state: follower="Node at 06d01780-d7b5-3c1b-6649-648a00bb9517 [Follower]" leader-address= leader-id= -2025-10-30T18:19:19.728Z [INFO] server-c2316925-47e6-dcc4-9570-a76f6f244b07: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06d01780-d7b5-3c1b-6649-648a00bb9517 Address:06d01780-d7b5-3c1b-6649-648a00bb9517} {Suffrage:Voter ID:server-c2316925-47e6-dcc4-9570-a76f6f244b07 Address:c2316925-47e6-dcc4-9570-a76f6f244b07} {Suffrage:Voter ID:server-97c82631-dd2e-1258-25b9-5cd8830525ce Address:97c82631-dd2e-1258-25b9-5cd8830525ce}]" -2025-10-30T18:19:19.728Z [INFO] server-c2316925-47e6-dcc4-9570-a76f6f244b07: entering follower state: follower="Node at c2316925-47e6-dcc4-9570-a76f6f244b07 [Follower]" leader-address= leader-id= -2025-10-30T18:19:19.729Z [INFO] server-97c82631-dd2e-1258-25b9-5cd8830525ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06d01780-d7b5-3c1b-6649-648a00bb9517 Address:06d01780-d7b5-3c1b-6649-648a00bb9517} {Suffrage:Voter ID:server-c2316925-47e6-dcc4-9570-a76f6f244b07 Address:c2316925-47e6-dcc4-9570-a76f6f244b07} {Suffrage:Voter ID:server-97c82631-dd2e-1258-25b9-5cd8830525ce Address:97c82631-dd2e-1258-25b9-5cd8830525ce}]" -2025-10-30T18:19:19.729Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:19.729Z [INFO] server-97c82631-dd2e-1258-25b9-5cd8830525ce: entering follower state: follower="Node at 97c82631-dd2e-1258-25b9-5cd8830525ce [Follower]" leader-address= leader-id= -2025-10-30T18:19:19.823Z [WARN] server-97c82631-dd2e-1258-25b9-5cd8830525ce: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:19.823Z [INFO] server-97c82631-dd2e-1258-25b9-5cd8830525ce: entering candidate state: node="Node at 97c82631-dd2e-1258-25b9-5cd8830525ce [Candidate]" term=2 -2025-10-30T18:19:19.823Z [INFO] server-97c82631-dd2e-1258-25b9-5cd8830525ce: election won: term=2 tally=2 -2025-10-30T18:19:19.823Z [INFO] server-97c82631-dd2e-1258-25b9-5cd8830525ce: entering leader state: leader="Node at 97c82631-dd2e-1258-25b9-5cd8830525ce [Leader]" -2025-10-30T18:19:19.823Z [INFO] server-97c82631-dd2e-1258-25b9-5cd8830525ce: added peer, starting replication: peer=server-06d01780-d7b5-3c1b-6649-648a00bb9517 -2025-10-30T18:19:19.823Z [INFO] server-97c82631-dd2e-1258-25b9-5cd8830525ce: added peer, starting replication: peer=server-c2316925-47e6-dcc4-9570-a76f6f244b07 -2025-10-30T18:19:19.823Z [INFO] server-97c82631-dd2e-1258-25b9-5cd8830525ce: pipelining replication: peer="{Voter server-c2316925-47e6-dcc4-9570-a76f6f244b07 c2316925-47e6-dcc4-9570-a76f6f244b07}" -2025-10-30T18:19:19.823Z [INFO] server-97c82631-dd2e-1258-25b9-5cd8830525ce: pipelining replication: peer="{Voter server-06d01780-d7b5-3c1b-6649-648a00bb9517 06d01780-d7b5-3c1b-6649-648a00bb9517}" -2025-10-30T18:19:19.929Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:19.82365916 +0000 UTC m=+106.102333058 (1 nodes), 94.577523ms from start of poll, 94.913226ms from cluster start. Timeout at 2025-10-30 18:19:19.928665145 +0000 UTC m=+106.207339043, 105.005985ms after stability -2025-10-30T18:19:19.929Z [INFO] server-97c82631-dd2e-1258-25b9-5cd8830525ce: aborting pipeline replication: peer="{Voter server-c2316925-47e6-dcc4-9570-a76f6f244b07 c2316925-47e6-dcc4-9570-a76f6f244b07}" -2025-10-30T18:19:19.929Z [INFO] server-97c82631-dd2e-1258-25b9-5cd8830525ce: aborting pipeline replication: peer="{Voter server-06d01780-d7b5-3c1b-6649-648a00bb9517 06d01780-d7b5-3c1b-6649-648a00bb9517}" ---- PASS: TestRaft_LeadershipTransferToInvalidID (0.20s) +2025-10-30T18:23:13.112Z [INFO] server-b21b3b38-40c9-252f-ca4f-6df8088f5792: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b21b3b38-40c9-252f-ca4f-6df8088f5792 Address:b21b3b38-40c9-252f-ca4f-6df8088f5792} {Suffrage:Voter ID:server-4a55c443-e7e7-f90b-8d24-ce7fd81a3a75 Address:4a55c443-e7e7-f90b-8d24-ce7fd81a3a75} {Suffrage:Voter ID:server-72c1f5a1-2de0-0ce2-d909-28fe555ee216 Address:72c1f5a1-2de0-0ce2-d909-28fe555ee216}]" +2025-10-30T18:23:13.112Z [INFO] server-b21b3b38-40c9-252f-ca4f-6df8088f5792: entering follower state: follower="Node at b21b3b38-40c9-252f-ca4f-6df8088f5792 [Follower]" leader-address= leader-id= +2025-10-30T18:23:13.112Z [INFO] server-4a55c443-e7e7-f90b-8d24-ce7fd81a3a75: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b21b3b38-40c9-252f-ca4f-6df8088f5792 Address:b21b3b38-40c9-252f-ca4f-6df8088f5792} {Suffrage:Voter ID:server-4a55c443-e7e7-f90b-8d24-ce7fd81a3a75 Address:4a55c443-e7e7-f90b-8d24-ce7fd81a3a75} {Suffrage:Voter ID:server-72c1f5a1-2de0-0ce2-d909-28fe555ee216 Address:72c1f5a1-2de0-0ce2-d909-28fe555ee216}]" +2025-10-30T18:23:13.112Z [INFO] server-4a55c443-e7e7-f90b-8d24-ce7fd81a3a75: entering follower state: follower="Node at 4a55c443-e7e7-f90b-8d24-ce7fd81a3a75 [Follower]" leader-address= leader-id= +2025-10-30T18:23:13.112Z [INFO] server-72c1f5a1-2de0-0ce2-d909-28fe555ee216: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b21b3b38-40c9-252f-ca4f-6df8088f5792 Address:b21b3b38-40c9-252f-ca4f-6df8088f5792} {Suffrage:Voter ID:server-4a55c443-e7e7-f90b-8d24-ce7fd81a3a75 Address:4a55c443-e7e7-f90b-8d24-ce7fd81a3a75} {Suffrage:Voter ID:server-72c1f5a1-2de0-0ce2-d909-28fe555ee216 Address:72c1f5a1-2de0-0ce2-d909-28fe555ee216}]" +2025-10-30T18:23:13.112Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:13.112Z [INFO] server-72c1f5a1-2de0-0ce2-d909-28fe555ee216: entering follower state: follower="Node at 72c1f5a1-2de0-0ce2-d909-28fe555ee216 [Follower]" leader-address= leader-id= +2025-10-30T18:23:13.171Z [WARN] server-b21b3b38-40c9-252f-ca4f-6df8088f5792: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:13.171Z [INFO] server-b21b3b38-40c9-252f-ca4f-6df8088f5792: entering candidate state: node="Node at b21b3b38-40c9-252f-ca4f-6df8088f5792 [Candidate]" term=2 +2025-10-30T18:23:13.171Z [INFO] server-b21b3b38-40c9-252f-ca4f-6df8088f5792: election won: term=2 tally=2 +2025-10-30T18:23:13.171Z [INFO] server-b21b3b38-40c9-252f-ca4f-6df8088f5792: entering leader state: leader="Node at b21b3b38-40c9-252f-ca4f-6df8088f5792 [Leader]" +2025-10-30T18:23:13.171Z [INFO] server-b21b3b38-40c9-252f-ca4f-6df8088f5792: added peer, starting replication: peer=server-4a55c443-e7e7-f90b-8d24-ce7fd81a3a75 +2025-10-30T18:23:13.171Z [INFO] server-b21b3b38-40c9-252f-ca4f-6df8088f5792: added peer, starting replication: peer=server-72c1f5a1-2de0-0ce2-d909-28fe555ee216 +2025-10-30T18:23:13.171Z [INFO] server-b21b3b38-40c9-252f-ca4f-6df8088f5792: pipelining replication: peer="{Voter server-72c1f5a1-2de0-0ce2-d909-28fe555ee216 72c1f5a1-2de0-0ce2-d909-28fe555ee216}" +2025-10-30T18:23:13.171Z [INFO] server-b21b3b38-40c9-252f-ca4f-6df8088f5792: pipelining replication: peer="{Voter server-4a55c443-e7e7-f90b-8d24-ce7fd81a3a75 4a55c443-e7e7-f90b-8d24-ce7fd81a3a75}" +2025-10-30T18:23:13.276Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:13.1714085 +0000 UTC m=+116.244937793 (1 nodes), 58.503948ms from start of poll, 58.824357ms from cluster start. Timeout at 2025-10-30 18:23:13.276410304 +0000 UTC m=+116.349939597, 105.001804ms after stability +2025-10-30T18:23:13.277Z [INFO] server-b21b3b38-40c9-252f-ca4f-6df8088f5792: aborting pipeline replication: peer="{Voter server-72c1f5a1-2de0-0ce2-d909-28fe555ee216 72c1f5a1-2de0-0ce2-d909-28fe555ee216}" +2025-10-30T18:23:13.277Z [INFO] server-b21b3b38-40c9-252f-ca4f-6df8088f5792: aborting pipeline replication: peer="{Voter server-4a55c443-e7e7-f90b-8d24-ce7fd81a3a75 4a55c443-e7e7-f90b-8d24-ce7fd81a3a75}" +--- PASS: TestRaft_LeadershipTransferToInvalidID (0.17s) === RUN TestRaft_LeadershipTransferToInvalidAddress -2025-10-30T18:19:19.930Z [INFO] server-2da7005f-7e4e-e9cc-a9bc-26c7e70156d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2da7005f-7e4e-e9cc-a9bc-26c7e70156d9 Address:2da7005f-7e4e-e9cc-a9bc-26c7e70156d9} {Suffrage:Voter ID:server-a0d1989a-6eb7-1bed-ff62-8987cad84b42 Address:a0d1989a-6eb7-1bed-ff62-8987cad84b42} {Suffrage:Voter ID:server-6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784 Address:6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784}]" -2025-10-30T18:19:19.930Z [INFO] server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2da7005f-7e4e-e9cc-a9bc-26c7e70156d9 Address:2da7005f-7e4e-e9cc-a9bc-26c7e70156d9} {Suffrage:Voter ID:server-a0d1989a-6eb7-1bed-ff62-8987cad84b42 Address:a0d1989a-6eb7-1bed-ff62-8987cad84b42} {Suffrage:Voter ID:server-6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784 Address:6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784}]" -2025-10-30T18:19:19.930Z [INFO] server-2da7005f-7e4e-e9cc-a9bc-26c7e70156d9: entering follower state: follower="Node at 2da7005f-7e4e-e9cc-a9bc-26c7e70156d9 [Follower]" leader-address= leader-id= -2025-10-30T18:19:19.930Z [INFO] server-6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2da7005f-7e4e-e9cc-a9bc-26c7e70156d9 Address:2da7005f-7e4e-e9cc-a9bc-26c7e70156d9} {Suffrage:Voter ID:server-a0d1989a-6eb7-1bed-ff62-8987cad84b42 Address:a0d1989a-6eb7-1bed-ff62-8987cad84b42} {Suffrage:Voter ID:server-6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784 Address:6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784}]" -2025-10-30T18:19:19.930Z [INFO] server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: entering follower state: follower="Node at a0d1989a-6eb7-1bed-ff62-8987cad84b42 [Follower]" leader-address= leader-id= -2025-10-30T18:19:19.930Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:19.930Z [INFO] server-6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784: entering follower state: follower="Node at 6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784 [Follower]" leader-address= leader-id= -2025-10-30T18:19:19.985Z [WARN] server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:19.985Z [INFO] server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: entering candidate state: node="Node at a0d1989a-6eb7-1bed-ff62-8987cad84b42 [Candidate]" term=2 -2025-10-30T18:19:19.985Z [INFO] server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: election won: term=2 tally=2 -2025-10-30T18:19:19.985Z [INFO] server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: entering leader state: leader="Node at a0d1989a-6eb7-1bed-ff62-8987cad84b42 [Leader]" -2025-10-30T18:19:19.985Z [INFO] server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: added peer, starting replication: peer=server-2da7005f-7e4e-e9cc-a9bc-26c7e70156d9 -2025-10-30T18:19:19.985Z [INFO] server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: added peer, starting replication: peer=server-6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784 -2025-10-30T18:19:19.985Z [INFO] server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: pipelining replication: peer="{Voter server-6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784 6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784}" -2025-10-30T18:19:19.985Z [INFO] server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: pipelining replication: peer="{Voter server-2da7005f-7e4e-e9cc-a9bc-26c7e70156d9 2da7005f-7e4e-e9cc-a9bc-26c7e70156d9}" -2025-10-30T18:19:20.091Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:19.985495834 +0000 UTC m=+106.264169730 (2 nodes), 55.26613ms from start of poll, 55.513154ms from cluster start. Timeout at 2025-10-30 18:19:20.09049638 +0000 UTC m=+106.369170277, 105.000547ms after stability -2025-10-30T18:19:20.091Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:20.196Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:20.091108403 +0000 UTC m=+106.369782312 (1 nodes), 1.829µs from start of poll, 161.125736ms from cluster start. Timeout at 2025-10-30 18:19:20.196131257 +0000 UTC m=+106.474805172, 105.02286ms after stability -2025-10-30T18:19:20.196Z [INFO] server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: aborting pipeline replication: peer="{Voter server-6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784 6e77df98-b8ba-fc3b-3f5c-3a76bc0d3784}" -2025-10-30T18:19:20.196Z [INFO] server-a0d1989a-6eb7-1bed-ff62-8987cad84b42: aborting pipeline replication: peer="{Voter server-2da7005f-7e4e-e9cc-a9bc-26c7e70156d9 2da7005f-7e4e-e9cc-a9bc-26c7e70156d9}" +2025-10-30T18:23:13.278Z [INFO] server-69fc4b10-eed4-a51d-c6e3-31d301e0d92e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69fc4b10-eed4-a51d-c6e3-31d301e0d92e Address:69fc4b10-eed4-a51d-c6e3-31d301e0d92e} {Suffrage:Voter ID:server-c1899a22-274f-c482-2124-b1cb5a02bda0 Address:c1899a22-274f-c482-2124-b1cb5a02bda0} {Suffrage:Voter ID:server-4e85223b-491b-9340-01fb-a7f41e0e5cd5 Address:4e85223b-491b-9340-01fb-a7f41e0e5cd5}]" +2025-10-30T18:23:13.278Z [INFO] server-69fc4b10-eed4-a51d-c6e3-31d301e0d92e: entering follower state: follower="Node at 69fc4b10-eed4-a51d-c6e3-31d301e0d92e [Follower]" leader-address= leader-id= +2025-10-30T18:23:13.278Z [INFO] server-c1899a22-274f-c482-2124-b1cb5a02bda0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69fc4b10-eed4-a51d-c6e3-31d301e0d92e Address:69fc4b10-eed4-a51d-c6e3-31d301e0d92e} {Suffrage:Voter ID:server-c1899a22-274f-c482-2124-b1cb5a02bda0 Address:c1899a22-274f-c482-2124-b1cb5a02bda0} {Suffrage:Voter ID:server-4e85223b-491b-9340-01fb-a7f41e0e5cd5 Address:4e85223b-491b-9340-01fb-a7f41e0e5cd5}]" +2025-10-30T18:23:13.278Z [INFO] server-c1899a22-274f-c482-2124-b1cb5a02bda0: entering follower state: follower="Node at c1899a22-274f-c482-2124-b1cb5a02bda0 [Follower]" leader-address= leader-id= +2025-10-30T18:23:13.279Z [INFO] server-4e85223b-491b-9340-01fb-a7f41e0e5cd5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69fc4b10-eed4-a51d-c6e3-31d301e0d92e Address:69fc4b10-eed4-a51d-c6e3-31d301e0d92e} {Suffrage:Voter ID:server-c1899a22-274f-c482-2124-b1cb5a02bda0 Address:c1899a22-274f-c482-2124-b1cb5a02bda0} {Suffrage:Voter ID:server-4e85223b-491b-9340-01fb-a7f41e0e5cd5 Address:4e85223b-491b-9340-01fb-a7f41e0e5cd5}]" +2025-10-30T18:23:13.279Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:13.279Z [INFO] server-4e85223b-491b-9340-01fb-a7f41e0e5cd5: entering follower state: follower="Node at 4e85223b-491b-9340-01fb-a7f41e0e5cd5 [Follower]" leader-address= leader-id= +2025-10-30T18:23:13.333Z [WARN] server-c1899a22-274f-c482-2124-b1cb5a02bda0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:13.333Z [INFO] server-c1899a22-274f-c482-2124-b1cb5a02bda0: entering candidate state: node="Node at c1899a22-274f-c482-2124-b1cb5a02bda0 [Candidate]" term=2 +2025-10-30T18:23:13.333Z [INFO] server-c1899a22-274f-c482-2124-b1cb5a02bda0: election won: term=2 tally=2 +2025-10-30T18:23:13.333Z [INFO] server-c1899a22-274f-c482-2124-b1cb5a02bda0: entering leader state: leader="Node at c1899a22-274f-c482-2124-b1cb5a02bda0 [Leader]" +2025-10-30T18:23:13.333Z [INFO] server-c1899a22-274f-c482-2124-b1cb5a02bda0: added peer, starting replication: peer=server-69fc4b10-eed4-a51d-c6e3-31d301e0d92e +2025-10-30T18:23:13.333Z [INFO] server-c1899a22-274f-c482-2124-b1cb5a02bda0: added peer, starting replication: peer=server-4e85223b-491b-9340-01fb-a7f41e0e5cd5 +2025-10-30T18:23:13.334Z [INFO] server-c1899a22-274f-c482-2124-b1cb5a02bda0: pipelining replication: peer="{Voter server-69fc4b10-eed4-a51d-c6e3-31d301e0d92e 69fc4b10-eed4-a51d-c6e3-31d301e0d92e}" +2025-10-30T18:23:13.334Z [INFO] server-c1899a22-274f-c482-2124-b1cb5a02bda0: pipelining replication: peer="{Voter server-4e85223b-491b-9340-01fb-a7f41e0e5cd5 4e85223b-491b-9340-01fb-a7f41e0e5cd5}" +2025-10-30T18:23:13.439Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:13.333996015 +0000 UTC m=+116.407525322 (2 nodes), 54.835243ms from start of poll, 55.260704ms from cluster start. Timeout at 2025-10-30 18:23:13.439004638 +0000 UTC m=+116.512533930, 105.008608ms after stability +2025-10-30T18:23:13.439Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:13.544Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:13.439270878 +0000 UTC m=+116.512800170 (1 nodes), 2.05µs from start of poll, 160.535552ms from cluster start. Timeout at 2025-10-30 18:23:13.544298649 +0000 UTC m=+116.617827965, 105.027795ms after stability +2025-10-30T18:23:13.544Z [INFO] server-c1899a22-274f-c482-2124-b1cb5a02bda0: aborting pipeline replication: peer="{Voter server-4e85223b-491b-9340-01fb-a7f41e0e5cd5 4e85223b-491b-9340-01fb-a7f41e0e5cd5}" +2025-10-30T18:23:13.544Z [INFO] server-c1899a22-274f-c482-2124-b1cb5a02bda0: aborting pipeline replication: peer="{Voter server-69fc4b10-eed4-a51d-c6e3-31d301e0d92e 69fc4b10-eed4-a51d-c6e3-31d301e0d92e}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.27s) === RUN TestRaft_LeadershipTransferToBehindServer -2025-10-30T18:19:20.197Z [INFO] server-8e4aaee5-22de-b546-297d-acc9c20b31e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4aaee5-22de-b546-297d-acc9c20b31e2 Address:8e4aaee5-22de-b546-297d-acc9c20b31e2} {Suffrage:Voter ID:server-513b9a68-100d-87a9-3572-bedd306b52a7 Address:513b9a68-100d-87a9-3572-bedd306b52a7} {Suffrage:Voter ID:server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c Address:a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c}]" -2025-10-30T18:19:20.197Z [INFO] server-513b9a68-100d-87a9-3572-bedd306b52a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4aaee5-22de-b546-297d-acc9c20b31e2 Address:8e4aaee5-22de-b546-297d-acc9c20b31e2} {Suffrage:Voter ID:server-513b9a68-100d-87a9-3572-bedd306b52a7 Address:513b9a68-100d-87a9-3572-bedd306b52a7} {Suffrage:Voter ID:server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c Address:a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c}]" -2025-10-30T18:19:20.197Z [INFO] server-8e4aaee5-22de-b546-297d-acc9c20b31e2: entering follower state: follower="Node at 8e4aaee5-22de-b546-297d-acc9c20b31e2 [Follower]" leader-address= leader-id= -2025-10-30T18:19:20.197Z [INFO] server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4aaee5-22de-b546-297d-acc9c20b31e2 Address:8e4aaee5-22de-b546-297d-acc9c20b31e2} {Suffrage:Voter ID:server-513b9a68-100d-87a9-3572-bedd306b52a7 Address:513b9a68-100d-87a9-3572-bedd306b52a7} {Suffrage:Voter ID:server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c Address:a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c}]" -2025-10-30T18:19:20.197Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:20.197Z [INFO] server-513b9a68-100d-87a9-3572-bedd306b52a7: entering follower state: follower="Node at 513b9a68-100d-87a9-3572-bedd306b52a7 [Follower]" leader-address= leader-id= -2025-10-30T18:19:20.197Z [INFO] server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: entering follower state: follower="Node at a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c [Follower]" leader-address= leader-id= -2025-10-30T18:19:20.252Z [WARN] server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:20.252Z [INFO] server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: entering candidate state: node="Node at a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c [Candidate]" term=2 -2025-10-30T18:19:20.252Z [INFO] server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: election won: term=2 tally=2 -2025-10-30T18:19:20.252Z [INFO] server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: entering leader state: leader="Node at a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c [Leader]" -2025-10-30T18:19:20.253Z [INFO] server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: added peer, starting replication: peer=server-8e4aaee5-22de-b546-297d-acc9c20b31e2 -2025-10-30T18:19:20.253Z [INFO] server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: added peer, starting replication: peer=server-513b9a68-100d-87a9-3572-bedd306b52a7 -2025-10-30T18:19:20.253Z [INFO] server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: pipelining replication: peer="{Voter server-8e4aaee5-22de-b546-297d-acc9c20b31e2 8e4aaee5-22de-b546-297d-acc9c20b31e2}" -2025-10-30T18:19:20.253Z [INFO] server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: pipelining replication: peer="{Voter server-513b9a68-100d-87a9-3572-bedd306b52a7 513b9a68-100d-87a9-3572-bedd306b52a7}" -2025-10-30T18:19:20.358Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:20.253046761 +0000 UTC m=+106.531720657 (1 nodes), 55.57619ms from start of poll, 55.752564ms from cluster start. Timeout at 2025-10-30 18:19:20.358048865 +0000 UTC m=+106.636722762, 105.002105ms after stability -2025-10-30T18:19:20.358Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:20.464Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:20.358195087 +0000 UTC m=+106.636868984 (2 nodes), 1.14µs from start of poll, 160.900891ms from cluster start. Timeout at 2025-10-30 18:19:20.463211463 +0000 UTC m=+106.741885377, 105.016393ms after stability -2025-10-30T18:19:20.487Z [INFO] server-8e4aaee5-22de-b546-297d-acc9c20b31e2: entering candidate state: node="Node at 8e4aaee5-22de-b546-297d-acc9c20b31e2 [Candidate]" term=3 -2025-10-30T18:19:20.487Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:20.487Z [INFO] server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: entering follower state: follower="Node at a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c [Follower]" leader-address= leader-id= -2025-10-30T18:19:20.487Z [INFO] server-8e4aaee5-22de-b546-297d-acc9c20b31e2: election won: term=3 tally=2 -2025-10-30T18:19:20.487Z [INFO] server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: aborting pipeline replication: peer="{Voter server-513b9a68-100d-87a9-3572-bedd306b52a7 513b9a68-100d-87a9-3572-bedd306b52a7}" -2025-10-30T18:19:20.487Z [INFO] server-8e4aaee5-22de-b546-297d-acc9c20b31e2: entering leader state: leader="Node at 8e4aaee5-22de-b546-297d-acc9c20b31e2 [Leader]" -2025-10-30T18:19:20.487Z [INFO] server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c: aborting pipeline replication: peer="{Voter server-8e4aaee5-22de-b546-297d-acc9c20b31e2 8e4aaee5-22de-b546-297d-acc9c20b31e2}" -2025-10-30T18:19:20.487Z [INFO] server-8e4aaee5-22de-b546-297d-acc9c20b31e2: added peer, starting replication: peer=server-513b9a68-100d-87a9-3572-bedd306b52a7 -2025-10-30T18:19:20.487Z [INFO] server-8e4aaee5-22de-b546-297d-acc9c20b31e2: added peer, starting replication: peer=server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c -2025-10-30T18:19:20.487Z [INFO] server-8e4aaee5-22de-b546-297d-acc9c20b31e2: pipelining replication: peer="{Voter server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c}" -2025-10-30T18:19:20.487Z [INFO] server-8e4aaee5-22de-b546-297d-acc9c20b31e2: pipelining replication: peer="{Voter server-513b9a68-100d-87a9-3572-bedd306b52a7 513b9a68-100d-87a9-3572-bedd306b52a7}" -2025-10-30T18:19:20.593Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:20.487483266 +0000 UTC m=+106.766157166 (1 nodes), 43.874µs from start of poll, 290.189073ms from cluster start. Timeout at 2025-10-30 18:19:20.592516616 +0000 UTC m=+106.871190543, 105.033377ms after stability -2025-10-30T18:19:20.593Z [INFO] server-8e4aaee5-22de-b546-297d-acc9c20b31e2: aborting pipeline replication: peer="{Voter server-513b9a68-100d-87a9-3572-bedd306b52a7 513b9a68-100d-87a9-3572-bedd306b52a7}" -2025-10-30T18:19:20.593Z [INFO] server-8e4aaee5-22de-b546-297d-acc9c20b31e2: aborting pipeline replication: peer="{Voter server-a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c a3db48c3-6bc6-3eb7-01ba-4b69fb1ac02c}" ---- PASS: TestRaft_LeadershipTransferToBehindServer (0.40s) +2025-10-30T18:23:13.546Z [INFO] server-67f0e80f-a5b0-30ab-e741-ecf05c92a500: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67f0e80f-a5b0-30ab-e741-ecf05c92a500 Address:67f0e80f-a5b0-30ab-e741-ecf05c92a500} {Suffrage:Voter ID:server-b889c713-2893-de43-4f70-df5f8c192587 Address:b889c713-2893-de43-4f70-df5f8c192587} {Suffrage:Voter ID:server-eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2 Address:eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2}]" +2025-10-30T18:23:13.546Z [INFO] server-67f0e80f-a5b0-30ab-e741-ecf05c92a500: entering follower state: follower="Node at 67f0e80f-a5b0-30ab-e741-ecf05c92a500 [Follower]" leader-address= leader-id= +2025-10-30T18:23:13.546Z [INFO] server-b889c713-2893-de43-4f70-df5f8c192587: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67f0e80f-a5b0-30ab-e741-ecf05c92a500 Address:67f0e80f-a5b0-30ab-e741-ecf05c92a500} {Suffrage:Voter ID:server-b889c713-2893-de43-4f70-df5f8c192587 Address:b889c713-2893-de43-4f70-df5f8c192587} {Suffrage:Voter ID:server-eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2 Address:eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2}]" +2025-10-30T18:23:13.547Z [INFO] server-b889c713-2893-de43-4f70-df5f8c192587: entering follower state: follower="Node at b889c713-2893-de43-4f70-df5f8c192587 [Follower]" leader-address= leader-id= +2025-10-30T18:23:13.547Z [INFO] server-eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67f0e80f-a5b0-30ab-e741-ecf05c92a500 Address:67f0e80f-a5b0-30ab-e741-ecf05c92a500} {Suffrage:Voter ID:server-b889c713-2893-de43-4f70-df5f8c192587 Address:b889c713-2893-de43-4f70-df5f8c192587} {Suffrage:Voter ID:server-eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2 Address:eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2}]" +2025-10-30T18:23:13.547Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:13.547Z [INFO] server-eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2: entering follower state: follower="Node at eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2 [Follower]" leader-address= leader-id= +2025-10-30T18:23:13.631Z [WARN] server-b889c713-2893-de43-4f70-df5f8c192587: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:13.631Z [INFO] server-b889c713-2893-de43-4f70-df5f8c192587: entering candidate state: node="Node at b889c713-2893-de43-4f70-df5f8c192587 [Candidate]" term=2 +2025-10-30T18:23:13.631Z [INFO] server-b889c713-2893-de43-4f70-df5f8c192587: election won: term=2 tally=2 +2025-10-30T18:23:13.631Z [INFO] server-b889c713-2893-de43-4f70-df5f8c192587: entering leader state: leader="Node at b889c713-2893-de43-4f70-df5f8c192587 [Leader]" +2025-10-30T18:23:13.632Z [INFO] server-b889c713-2893-de43-4f70-df5f8c192587: added peer, starting replication: peer=server-67f0e80f-a5b0-30ab-e741-ecf05c92a500 +2025-10-30T18:23:13.632Z [INFO] server-b889c713-2893-de43-4f70-df5f8c192587: added peer, starting replication: peer=server-eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2 +2025-10-30T18:23:13.632Z [INFO] server-b889c713-2893-de43-4f70-df5f8c192587: pipelining replication: peer="{Voter server-67f0e80f-a5b0-30ab-e741-ecf05c92a500 67f0e80f-a5b0-30ab-e741-ecf05c92a500}" +2025-10-30T18:23:13.632Z [INFO] server-b889c713-2893-de43-4f70-df5f8c192587: pipelining replication: peer="{Voter server-eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2 eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2}" +2025-10-30T18:23:13.737Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:13.632056975 +0000 UTC m=+116.705586267 (1 nodes), 84.930321ms from start of poll, 85.295522ms from cluster start. Timeout at 2025-10-30 18:23:13.737065789 +0000 UTC m=+116.810595095, 105.008828ms after stability +2025-10-30T18:23:13.737Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:13.843Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:13.737953878 +0000 UTC m=+116.811483172 (2 nodes), 1.161µs from start of poll, 191.192427ms from cluster start. Timeout at 2025-10-30 18:23:13.842965087 +0000 UTC m=+116.916494422, 105.01125ms after stability +2025-10-30T18:23:13.869Z [INFO] server-67f0e80f-a5b0-30ab-e741-ecf05c92a500: entering candidate state: node="Node at 67f0e80f-a5b0-30ab-e741-ecf05c92a500 [Candidate]" term=3 +2025-10-30T18:23:13.869Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:13.869Z [INFO] server-67f0e80f-a5b0-30ab-e741-ecf05c92a500: election won: term=3 tally=2 +2025-10-30T18:23:13.869Z [INFO] server-67f0e80f-a5b0-30ab-e741-ecf05c92a500: entering leader state: leader="Node at 67f0e80f-a5b0-30ab-e741-ecf05c92a500 [Leader]" +2025-10-30T18:23:13.869Z [INFO] server-67f0e80f-a5b0-30ab-e741-ecf05c92a500: added peer, starting replication: peer=server-b889c713-2893-de43-4f70-df5f8c192587 +2025-10-30T18:23:13.869Z [INFO] server-b889c713-2893-de43-4f70-df5f8c192587: entering follower state: follower="Node at b889c713-2893-de43-4f70-df5f8c192587 [Follower]" leader-address= leader-id= +2025-10-30T18:23:13.869Z [INFO] server-67f0e80f-a5b0-30ab-e741-ecf05c92a500: added peer, starting replication: peer=server-eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2 +2025-10-30T18:23:13.869Z [INFO] server-67f0e80f-a5b0-30ab-e741-ecf05c92a500: pipelining replication: peer="{Voter server-b889c713-2893-de43-4f70-df5f8c192587 b889c713-2893-de43-4f70-df5f8c192587}" +2025-10-30T18:23:13.869Z [INFO] server-b889c713-2893-de43-4f70-df5f8c192587: aborting pipeline replication: peer="{Voter server-eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2 eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2}" +2025-10-30T18:23:13.869Z [INFO] server-b889c713-2893-de43-4f70-df5f8c192587: aborting pipeline replication: peer="{Voter server-67f0e80f-a5b0-30ab-e741-ecf05c92a500 67f0e80f-a5b0-30ab-e741-ecf05c92a500}" +2025-10-30T18:23:13.869Z [INFO] server-67f0e80f-a5b0-30ab-e741-ecf05c92a500: pipelining replication: peer="{Voter server-eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2 eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2}" +2025-10-30T18:23:13.975Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:13.869383139 +0000 UTC m=+116.942912430 (1 nodes), 88.597µs from start of poll, 322.621685ms from cluster start. Timeout at 2025-10-30 18:23:13.974398542 +0000 UTC m=+117.047927832, 105.015402ms after stability +2025-10-30T18:23:13.975Z [INFO] server-67f0e80f-a5b0-30ab-e741-ecf05c92a500: aborting pipeline replication: peer="{Voter server-eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2 eec437a4-2bbe-4b4b-57cd-cdbf3bc2b4d2}" +2025-10-30T18:23:13.975Z [INFO] server-67f0e80f-a5b0-30ab-e741-ecf05c92a500: aborting pipeline replication: peer="{Voter server-b889c713-2893-de43-4f70-df5f8c192587 b889c713-2893-de43-4f70-df5f8c192587}" +--- PASS: TestRaft_LeadershipTransferToBehindServer (0.43s) === RUN TestRaft_LeadershipTransferToItself -2025-10-30T18:19:20.593Z [INFO] server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49 Address:ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49} {Suffrage:Voter ID:server-09fec08a-0dbb-78e4-7618-a30d320038fb Address:09fec08a-0dbb-78e4-7618-a30d320038fb} {Suffrage:Voter ID:server-0efdd2fe-733b-f9a2-f4f8-60bee2c94a70 Address:0efdd2fe-733b-f9a2-f4f8-60bee2c94a70}]" -2025-10-30T18:19:20.593Z [INFO] server-09fec08a-0dbb-78e4-7618-a30d320038fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49 Address:ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49} {Suffrage:Voter ID:server-09fec08a-0dbb-78e4-7618-a30d320038fb Address:09fec08a-0dbb-78e4-7618-a30d320038fb} {Suffrage:Voter ID:server-0efdd2fe-733b-f9a2-f4f8-60bee2c94a70 Address:0efdd2fe-733b-f9a2-f4f8-60bee2c94a70}]" -2025-10-30T18:19:20.593Z [INFO] server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: entering follower state: follower="Node at ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49 [Follower]" leader-address= leader-id= -2025-10-30T18:19:20.593Z [INFO] server-09fec08a-0dbb-78e4-7618-a30d320038fb: entering follower state: follower="Node at 09fec08a-0dbb-78e4-7618-a30d320038fb [Follower]" leader-address= leader-id= -2025-10-30T18:19:20.593Z [INFO] server-0efdd2fe-733b-f9a2-f4f8-60bee2c94a70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49 Address:ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49} {Suffrage:Voter ID:server-09fec08a-0dbb-78e4-7618-a30d320038fb Address:09fec08a-0dbb-78e4-7618-a30d320038fb} {Suffrage:Voter ID:server-0efdd2fe-733b-f9a2-f4f8-60bee2c94a70 Address:0efdd2fe-733b-f9a2-f4f8-60bee2c94a70}]" -2025-10-30T18:19:20.593Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:20.593Z [INFO] server-0efdd2fe-733b-f9a2-f4f8-60bee2c94a70: entering follower state: follower="Node at 0efdd2fe-733b-f9a2-f4f8-60bee2c94a70 [Follower]" leader-address= leader-id= -2025-10-30T18:19:20.665Z [WARN] server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:20.665Z [INFO] server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: entering candidate state: node="Node at ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49 [Candidate]" term=2 -2025-10-30T18:19:20.665Z [INFO] server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: election won: term=2 tally=2 -2025-10-30T18:19:20.665Z [INFO] server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: entering leader state: leader="Node at ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49 [Leader]" -2025-10-30T18:19:20.665Z [INFO] server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: added peer, starting replication: peer=server-09fec08a-0dbb-78e4-7618-a30d320038fb -2025-10-30T18:19:20.665Z [INFO] server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: added peer, starting replication: peer=server-0efdd2fe-733b-f9a2-f4f8-60bee2c94a70 -2025-10-30T18:19:20.665Z [INFO] server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: pipelining replication: peer="{Voter server-09fec08a-0dbb-78e4-7618-a30d320038fb 09fec08a-0dbb-78e4-7618-a30d320038fb}" -2025-10-30T18:19:20.665Z [INFO] server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: pipelining replication: peer="{Voter server-0efdd2fe-733b-f9a2-f4f8-60bee2c94a70 0efdd2fe-733b-f9a2-f4f8-60bee2c94a70}" -2025-10-30T18:19:20.771Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:20.665193252 +0000 UTC m=+106.943867149 (1 nodes), 71.222273ms from start of poll, 71.413698ms from cluster start. Timeout at 2025-10-30 18:19:20.77019608 +0000 UTC m=+107.048869977, 105.002828ms after stability -2025-10-30T18:19:20.771Z [INFO] server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: cannot transfer leadership to itself -2025-10-30T18:19:20.771Z [INFO] server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: aborting pipeline replication: peer="{Voter server-0efdd2fe-733b-f9a2-f4f8-60bee2c94a70 0efdd2fe-733b-f9a2-f4f8-60bee2c94a70}" -2025-10-30T18:19:20.771Z [INFO] server-ed3c112c-5d86-3d09-f0cf-83ecfa4e6c49: aborting pipeline replication: peer="{Voter server-09fec08a-0dbb-78e4-7618-a30d320038fb 09fec08a-0dbb-78e4-7618-a30d320038fb}" ---- PASS: TestRaft_LeadershipTransferToItself (0.18s) +2025-10-30T18:23:13.976Z [INFO] server-4fd067eb-e556-00fe-bc71-54fe480342b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fd067eb-e556-00fe-bc71-54fe480342b9 Address:4fd067eb-e556-00fe-bc71-54fe480342b9} {Suffrage:Voter ID:server-34845bb2-4f4b-be79-2066-cab44d40cb0c Address:34845bb2-4f4b-be79-2066-cab44d40cb0c} {Suffrage:Voter ID:server-5e4f83bf-7766-d7e1-a9a5-2b9e9f15d2c1 Address:5e4f83bf-7766-d7e1-a9a5-2b9e9f15d2c1}]" +2025-10-30T18:23:13.976Z [INFO] server-34845bb2-4f4b-be79-2066-cab44d40cb0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fd067eb-e556-00fe-bc71-54fe480342b9 Address:4fd067eb-e556-00fe-bc71-54fe480342b9} {Suffrage:Voter ID:server-34845bb2-4f4b-be79-2066-cab44d40cb0c Address:34845bb2-4f4b-be79-2066-cab44d40cb0c} {Suffrage:Voter ID:server-5e4f83bf-7766-d7e1-a9a5-2b9e9f15d2c1 Address:5e4f83bf-7766-d7e1-a9a5-2b9e9f15d2c1}]" +2025-10-30T18:23:13.976Z [INFO] server-4fd067eb-e556-00fe-bc71-54fe480342b9: entering follower state: follower="Node at 4fd067eb-e556-00fe-bc71-54fe480342b9 [Follower]" leader-address= leader-id= +2025-10-30T18:23:13.976Z [INFO] server-34845bb2-4f4b-be79-2066-cab44d40cb0c: entering follower state: follower="Node at 34845bb2-4f4b-be79-2066-cab44d40cb0c [Follower]" leader-address= leader-id= +2025-10-30T18:23:13.976Z [INFO] server-5e4f83bf-7766-d7e1-a9a5-2b9e9f15d2c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fd067eb-e556-00fe-bc71-54fe480342b9 Address:4fd067eb-e556-00fe-bc71-54fe480342b9} {Suffrage:Voter ID:server-34845bb2-4f4b-be79-2066-cab44d40cb0c Address:34845bb2-4f4b-be79-2066-cab44d40cb0c} {Suffrage:Voter ID:server-5e4f83bf-7766-d7e1-a9a5-2b9e9f15d2c1 Address:5e4f83bf-7766-d7e1-a9a5-2b9e9f15d2c1}]" +2025-10-30T18:23:13.976Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:13.976Z [INFO] server-5e4f83bf-7766-d7e1-a9a5-2b9e9f15d2c1: entering follower state: follower="Node at 5e4f83bf-7766-d7e1-a9a5-2b9e9f15d2c1 [Follower]" leader-address= leader-id= +2025-10-30T18:23:14.037Z [WARN] server-34845bb2-4f4b-be79-2066-cab44d40cb0c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:14.037Z [INFO] server-34845bb2-4f4b-be79-2066-cab44d40cb0c: entering candidate state: node="Node at 34845bb2-4f4b-be79-2066-cab44d40cb0c [Candidate]" term=2 +2025-10-30T18:23:14.037Z [INFO] server-34845bb2-4f4b-be79-2066-cab44d40cb0c: election won: term=2 tally=2 +2025-10-30T18:23:14.037Z [INFO] server-34845bb2-4f4b-be79-2066-cab44d40cb0c: entering leader state: leader="Node at 34845bb2-4f4b-be79-2066-cab44d40cb0c [Leader]" +2025-10-30T18:23:14.037Z [INFO] server-34845bb2-4f4b-be79-2066-cab44d40cb0c: added peer, starting replication: peer=server-4fd067eb-e556-00fe-bc71-54fe480342b9 +2025-10-30T18:23:14.037Z [INFO] server-34845bb2-4f4b-be79-2066-cab44d40cb0c: added peer, starting replication: peer=server-5e4f83bf-7766-d7e1-a9a5-2b9e9f15d2c1 +2025-10-30T18:23:14.037Z [INFO] server-34845bb2-4f4b-be79-2066-cab44d40cb0c: pipelining replication: peer="{Voter server-4fd067eb-e556-00fe-bc71-54fe480342b9 4fd067eb-e556-00fe-bc71-54fe480342b9}" +2025-10-30T18:23:14.037Z [INFO] server-34845bb2-4f4b-be79-2066-cab44d40cb0c: pipelining replication: peer="{Voter server-5e4f83bf-7766-d7e1-a9a5-2b9e9f15d2c1 5e4f83bf-7766-d7e1-a9a5-2b9e9f15d2c1}" +2025-10-30T18:23:14.142Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:14.037558868 +0000 UTC m=+117.111088161 (1 nodes), 60.977476ms from start of poll, 61.307649ms from cluster start. Timeout at 2025-10-30 18:23:14.142561511 +0000 UTC m=+117.216090803, 105.002642ms after stability +2025-10-30T18:23:14.143Z [INFO] server-34845bb2-4f4b-be79-2066-cab44d40cb0c: cannot transfer leadership to itself +2025-10-30T18:23:14.143Z [INFO] server-34845bb2-4f4b-be79-2066-cab44d40cb0c: aborting pipeline replication: peer="{Voter server-5e4f83bf-7766-d7e1-a9a5-2b9e9f15d2c1 5e4f83bf-7766-d7e1-a9a5-2b9e9f15d2c1}" +2025-10-30T18:23:14.143Z [INFO] server-34845bb2-4f4b-be79-2066-cab44d40cb0c: aborting pipeline replication: peer="{Voter server-4fd067eb-e556-00fe-bc71-54fe480342b9 4fd067eb-e556-00fe-bc71-54fe480342b9}" +--- PASS: TestRaft_LeadershipTransferToItself (0.47s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests -2025-10-30T18:19:20.771Z [INFO] server-60942810-3fdc-b2a7-f08f-e8d847f4ead5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60942810-3fdc-b2a7-f08f-e8d847f4ead5 Address:60942810-3fdc-b2a7-f08f-e8d847f4ead5} {Suffrage:Voter ID:server-7e0cd94e-84ab-f3f0-066d-f89e7fffead9 Address:7e0cd94e-84ab-f3f0-066d-f89e7fffead9} {Suffrage:Voter ID:server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5 Address:abfd1a5d-9716-03d2-5ca1-834aa14da6a5}]" -2025-10-30T18:19:20.772Z [INFO] server-7e0cd94e-84ab-f3f0-066d-f89e7fffead9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60942810-3fdc-b2a7-f08f-e8d847f4ead5 Address:60942810-3fdc-b2a7-f08f-e8d847f4ead5} {Suffrage:Voter ID:server-7e0cd94e-84ab-f3f0-066d-f89e7fffead9 Address:7e0cd94e-84ab-f3f0-066d-f89e7fffead9} {Suffrage:Voter ID:server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5 Address:abfd1a5d-9716-03d2-5ca1-834aa14da6a5}]" -2025-10-30T18:19:20.771Z [INFO] server-60942810-3fdc-b2a7-f08f-e8d847f4ead5: entering follower state: follower="Node at 60942810-3fdc-b2a7-f08f-e8d847f4ead5 [Follower]" leader-address= leader-id= -2025-10-30T18:19:20.772Z [INFO] server-7e0cd94e-84ab-f3f0-066d-f89e7fffead9: entering follower state: follower="Node at 7e0cd94e-84ab-f3f0-066d-f89e7fffead9 [Follower]" leader-address= leader-id= -2025-10-30T18:19:20.772Z [INFO] server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60942810-3fdc-b2a7-f08f-e8d847f4ead5 Address:60942810-3fdc-b2a7-f08f-e8d847f4ead5} {Suffrage:Voter ID:server-7e0cd94e-84ab-f3f0-066d-f89e7fffead9 Address:7e0cd94e-84ab-f3f0-066d-f89e7fffead9} {Suffrage:Voter ID:server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5 Address:abfd1a5d-9716-03d2-5ca1-834aa14da6a5}]" -2025-10-30T18:19:20.772Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:20.772Z [INFO] server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: entering follower state: follower="Node at abfd1a5d-9716-03d2-5ca1-834aa14da6a5 [Follower]" leader-address= leader-id= -2025-10-30T18:19:20.824Z [WARN] server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:20.824Z [INFO] server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: entering candidate state: node="Node at abfd1a5d-9716-03d2-5ca1-834aa14da6a5 [Candidate]" term=2 -2025-10-30T18:19:20.824Z [INFO] server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: election won: term=2 tally=2 -2025-10-30T18:19:20.824Z [INFO] server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: entering leader state: leader="Node at abfd1a5d-9716-03d2-5ca1-834aa14da6a5 [Leader]" -2025-10-30T18:19:20.824Z [INFO] server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: added peer, starting replication: peer=server-60942810-3fdc-b2a7-f08f-e8d847f4ead5 -2025-10-30T18:19:20.824Z [INFO] server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: added peer, starting replication: peer=server-7e0cd94e-84ab-f3f0-066d-f89e7fffead9 -2025-10-30T18:19:20.824Z [INFO] server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: pipelining replication: peer="{Voter server-60942810-3fdc-b2a7-f08f-e8d847f4ead5 60942810-3fdc-b2a7-f08f-e8d847f4ead5}" -2025-10-30T18:19:20.824Z [INFO] server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: pipelining replication: peer="{Voter server-7e0cd94e-84ab-f3f0-066d-f89e7fffead9 7e0cd94e-84ab-f3f0-066d-f89e7fffead9}" -2025-10-30T18:19:20.930Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:20.824164711 +0000 UTC m=+107.102838608 (1 nodes), 52.071538ms from start of poll, 52.289366ms from cluster start. Timeout at 2025-10-30 18:19:20.929165205 +0000 UTC m=+107.207839102, 105.000494ms after stability +2025-10-30T18:23:14.450Z [INFO] server-be6023c8-12be-2fad-eff8-369d8e18921c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be6023c8-12be-2fad-eff8-369d8e18921c Address:be6023c8-12be-2fad-eff8-369d8e18921c} {Suffrage:Voter ID:server-54e2927f-420c-96e8-e8bf-898a0dfd0044 Address:54e2927f-420c-96e8-e8bf-898a0dfd0044} {Suffrage:Voter ID:server-e3867b02-95c7-6cb6-9db0-c16bc8fcf5c2 Address:e3867b02-95c7-6cb6-9db0-c16bc8fcf5c2}]" +2025-10-30T18:23:14.450Z [INFO] server-54e2927f-420c-96e8-e8bf-898a0dfd0044: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be6023c8-12be-2fad-eff8-369d8e18921c Address:be6023c8-12be-2fad-eff8-369d8e18921c} {Suffrage:Voter ID:server-54e2927f-420c-96e8-e8bf-898a0dfd0044 Address:54e2927f-420c-96e8-e8bf-898a0dfd0044} {Suffrage:Voter ID:server-e3867b02-95c7-6cb6-9db0-c16bc8fcf5c2 Address:e3867b02-95c7-6cb6-9db0-c16bc8fcf5c2}]" +2025-10-30T18:23:14.450Z [INFO] server-be6023c8-12be-2fad-eff8-369d8e18921c: entering follower state: follower="Node at be6023c8-12be-2fad-eff8-369d8e18921c [Follower]" leader-address= leader-id= +2025-10-30T18:23:14.450Z [INFO] server-e3867b02-95c7-6cb6-9db0-c16bc8fcf5c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be6023c8-12be-2fad-eff8-369d8e18921c Address:be6023c8-12be-2fad-eff8-369d8e18921c} {Suffrage:Voter ID:server-54e2927f-420c-96e8-e8bf-898a0dfd0044 Address:54e2927f-420c-96e8-e8bf-898a0dfd0044} {Suffrage:Voter ID:server-e3867b02-95c7-6cb6-9db0-c16bc8fcf5c2 Address:e3867b02-95c7-6cb6-9db0-c16bc8fcf5c2}]" +2025-10-30T18:23:14.450Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:14.450Z [INFO] server-54e2927f-420c-96e8-e8bf-898a0dfd0044: entering follower state: follower="Node at 54e2927f-420c-96e8-e8bf-898a0dfd0044 [Follower]" leader-address= leader-id= +2025-10-30T18:23:14.450Z [INFO] server-e3867b02-95c7-6cb6-9db0-c16bc8fcf5c2: entering follower state: follower="Node at e3867b02-95c7-6cb6-9db0-c16bc8fcf5c2 [Follower]" leader-address= leader-id= +2025-10-30T18:23:14.524Z [WARN] server-be6023c8-12be-2fad-eff8-369d8e18921c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:14.524Z [INFO] server-be6023c8-12be-2fad-eff8-369d8e18921c: entering candidate state: node="Node at be6023c8-12be-2fad-eff8-369d8e18921c [Candidate]" term=2 +2025-10-30T18:23:14.524Z [INFO] server-be6023c8-12be-2fad-eff8-369d8e18921c: election won: term=2 tally=2 +2025-10-30T18:23:14.524Z [INFO] server-be6023c8-12be-2fad-eff8-369d8e18921c: entering leader state: leader="Node at be6023c8-12be-2fad-eff8-369d8e18921c [Leader]" +2025-10-30T18:23:14.524Z [INFO] server-be6023c8-12be-2fad-eff8-369d8e18921c: added peer, starting replication: peer=server-54e2927f-420c-96e8-e8bf-898a0dfd0044 +2025-10-30T18:23:14.524Z [INFO] server-be6023c8-12be-2fad-eff8-369d8e18921c: added peer, starting replication: peer=server-e3867b02-95c7-6cb6-9db0-c16bc8fcf5c2 +2025-10-30T18:23:14.524Z [INFO] server-be6023c8-12be-2fad-eff8-369d8e18921c: pipelining replication: peer="{Voter server-54e2927f-420c-96e8-e8bf-898a0dfd0044 54e2927f-420c-96e8-e8bf-898a0dfd0044}" +2025-10-30T18:23:14.524Z [INFO] server-be6023c8-12be-2fad-eff8-369d8e18921c: pipelining replication: peer="{Voter server-e3867b02-95c7-6cb6-9db0-c16bc8fcf5c2 e3867b02-95c7-6cb6-9db0-c16bc8fcf5c2}" +2025-10-30T18:23:14.629Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:14.524355863 +0000 UTC m=+117.597885155 (1 nodes), 73.868635ms from start of poll, 74.111183ms from cluster start. Timeout at 2025-10-30 18:23:14.629358399 +0000 UTC m=+117.702887688, 105.002533ms after stability raft_test.go:2457: waiting on future 0 raft_test.go:2457: waiting on future 1 raft_test.go:2457: waiting on future 2 @@ -3898,344 +6666,348 @@ raft_test.go:2457: waiting on future 4 raft_test.go:2457: waiting on future 5 raft_test.go:2457: waiting on future 6 -2025-10-30T18:19:20.930Z [INFO] server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: aborting pipeline replication: peer="{Voter server-7e0cd94e-84ab-f3f0-066d-f89e7fffead9 7e0cd94e-84ab-f3f0-066d-f89e7fffead9}" -2025-10-30T18:19:20.930Z [INFO] server-abfd1a5d-9716-03d2-5ca1-834aa14da6a5: aborting pipeline replication: peer="{Voter server-60942810-3fdc-b2a7-f08f-e8d847f4ead5 60942810-3fdc-b2a7-f08f-e8d847f4ead5}" ---- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.16s) +2025-10-30T18:23:14.629Z [INFO] server-be6023c8-12be-2fad-eff8-369d8e18921c: aborting pipeline replication: peer="{Voter server-e3867b02-95c7-6cb6-9db0-c16bc8fcf5c2 e3867b02-95c7-6cb6-9db0-c16bc8fcf5c2}" +2025-10-30T18:23:14.629Z [INFO] server-be6023c8-12be-2fad-eff8-369d8e18921c: aborting pipeline replication: peer="{Voter server-54e2927f-420c-96e8-e8bf-898a0dfd0044 54e2927f-420c-96e8-e8bf-898a0dfd0044}" +--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.30s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout -2025-10-30T18:19:20.931Z [INFO] server-fa35787c-946b-1b2b-d093-7511376ec0f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa35787c-946b-1b2b-d093-7511376ec0f9 Address:fa35787c-946b-1b2b-d093-7511376ec0f9} {Suffrage:Voter ID:server-2efc3f40-7050-ad1e-0635-0e2d7807522b Address:2efc3f40-7050-ad1e-0635-0e2d7807522b} {Suffrage:Voter ID:server-4094de85-65f8-2fca-23fd-1f775544793f Address:4094de85-65f8-2fca-23fd-1f775544793f}]" -2025-10-30T18:19:20.931Z [INFO] server-2efc3f40-7050-ad1e-0635-0e2d7807522b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa35787c-946b-1b2b-d093-7511376ec0f9 Address:fa35787c-946b-1b2b-d093-7511376ec0f9} {Suffrage:Voter ID:server-2efc3f40-7050-ad1e-0635-0e2d7807522b Address:2efc3f40-7050-ad1e-0635-0e2d7807522b} {Suffrage:Voter ID:server-4094de85-65f8-2fca-23fd-1f775544793f Address:4094de85-65f8-2fca-23fd-1f775544793f}]" -2025-10-30T18:19:20.931Z [INFO] server-fa35787c-946b-1b2b-d093-7511376ec0f9: entering follower state: follower="Node at fa35787c-946b-1b2b-d093-7511376ec0f9 [Follower]" leader-address= leader-id= -2025-10-30T18:19:20.931Z [INFO] server-2efc3f40-7050-ad1e-0635-0e2d7807522b: entering follower state: follower="Node at 2efc3f40-7050-ad1e-0635-0e2d7807522b [Follower]" leader-address= leader-id= -2025-10-30T18:19:20.931Z [INFO] server-4094de85-65f8-2fca-23fd-1f775544793f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa35787c-946b-1b2b-d093-7511376ec0f9 Address:fa35787c-946b-1b2b-d093-7511376ec0f9} {Suffrage:Voter ID:server-2efc3f40-7050-ad1e-0635-0e2d7807522b Address:2efc3f40-7050-ad1e-0635-0e2d7807522b} {Suffrage:Voter ID:server-4094de85-65f8-2fca-23fd-1f775544793f Address:4094de85-65f8-2fca-23fd-1f775544793f}]" -2025-10-30T18:19:20.931Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:20.931Z [INFO] server-4094de85-65f8-2fca-23fd-1f775544793f: entering follower state: follower="Node at 4094de85-65f8-2fca-23fd-1f775544793f [Follower]" leader-address= leader-id= -2025-10-30T18:19:20.983Z [WARN] server-2efc3f40-7050-ad1e-0635-0e2d7807522b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:20.983Z [INFO] server-2efc3f40-7050-ad1e-0635-0e2d7807522b: entering candidate state: node="Node at 2efc3f40-7050-ad1e-0635-0e2d7807522b [Candidate]" term=2 -2025-10-30T18:19:20.983Z [INFO] server-2efc3f40-7050-ad1e-0635-0e2d7807522b: election won: term=2 tally=2 -2025-10-30T18:19:20.983Z [INFO] server-2efc3f40-7050-ad1e-0635-0e2d7807522b: entering leader state: leader="Node at 2efc3f40-7050-ad1e-0635-0e2d7807522b [Leader]" -2025-10-30T18:19:20.983Z [INFO] server-2efc3f40-7050-ad1e-0635-0e2d7807522b: added peer, starting replication: peer=server-fa35787c-946b-1b2b-d093-7511376ec0f9 -2025-10-30T18:19:20.983Z [INFO] server-2efc3f40-7050-ad1e-0635-0e2d7807522b: added peer, starting replication: peer=server-4094de85-65f8-2fca-23fd-1f775544793f -2025-10-30T18:19:20.983Z [INFO] server-2efc3f40-7050-ad1e-0635-0e2d7807522b: pipelining replication: peer="{Voter server-4094de85-65f8-2fca-23fd-1f775544793f 4094de85-65f8-2fca-23fd-1f775544793f}" -2025-10-30T18:19:20.983Z [INFO] server-2efc3f40-7050-ad1e-0635-0e2d7807522b: pipelining replication: peer="{Voter server-fa35787c-946b-1b2b-d093-7511376ec0f9 fa35787c-946b-1b2b-d093-7511376ec0f9}" -2025-10-30T18:19:21.088Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:20.983649135 +0000 UTC m=+107.262323033 (1 nodes), 52.39573ms from start of poll, 52.615356ms from cluster start. Timeout at 2025-10-30 18:19:21.088652125 +0000 UTC m=+107.367326023, 105.00299ms after stability -2025-10-30T18:19:21.088Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:21.194Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:21.088805807 +0000 UTC m=+107.367479705 (2 nodes), 1.684µs from start of poll, 157.772028ms from cluster start. Timeout at 2025-10-30 18:19:21.193828965 +0000 UTC m=+107.472502862, 105.023157ms after stability -2025-10-30T18:19:21.393Z [INFO] server-2efc3f40-7050-ad1e-0635-0e2d7807522b: aborting pipeline replication: peer="{Voter server-4094de85-65f8-2fca-23fd-1f775544793f 4094de85-65f8-2fca-23fd-1f775544793f}" -2025-10-30T18:19:21.393Z [INFO] server-2efc3f40-7050-ad1e-0635-0e2d7807522b: aborting pipeline replication: peer="{Voter server-fa35787c-946b-1b2b-d093-7511376ec0f9 fa35787c-946b-1b2b-d093-7511376ec0f9}" ---- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.46s) +2025-10-30T18:23:14.751Z [INFO] server-4a293cbe-1c35-7fa6-1023-fb8669dffeba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a293cbe-1c35-7fa6-1023-fb8669dffeba Address:4a293cbe-1c35-7fa6-1023-fb8669dffeba} {Suffrage:Voter ID:server-e50501f4-fe5a-05bb-6294-bffb06a17713 Address:e50501f4-fe5a-05bb-6294-bffb06a17713} {Suffrage:Voter ID:server-759a6aac-f645-d815-f9cb-8f2a81e044f7 Address:759a6aac-f645-d815-f9cb-8f2a81e044f7}]" +2025-10-30T18:23:14.751Z [INFO] server-e50501f4-fe5a-05bb-6294-bffb06a17713: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a293cbe-1c35-7fa6-1023-fb8669dffeba Address:4a293cbe-1c35-7fa6-1023-fb8669dffeba} {Suffrage:Voter ID:server-e50501f4-fe5a-05bb-6294-bffb06a17713 Address:e50501f4-fe5a-05bb-6294-bffb06a17713} {Suffrage:Voter ID:server-759a6aac-f645-d815-f9cb-8f2a81e044f7 Address:759a6aac-f645-d815-f9cb-8f2a81e044f7}]" +2025-10-30T18:23:14.751Z [INFO] server-4a293cbe-1c35-7fa6-1023-fb8669dffeba: entering follower state: follower="Node at 4a293cbe-1c35-7fa6-1023-fb8669dffeba [Follower]" leader-address= leader-id= +2025-10-30T18:23:14.751Z [INFO] server-759a6aac-f645-d815-f9cb-8f2a81e044f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a293cbe-1c35-7fa6-1023-fb8669dffeba Address:4a293cbe-1c35-7fa6-1023-fb8669dffeba} {Suffrage:Voter ID:server-e50501f4-fe5a-05bb-6294-bffb06a17713 Address:e50501f4-fe5a-05bb-6294-bffb06a17713} {Suffrage:Voter ID:server-759a6aac-f645-d815-f9cb-8f2a81e044f7 Address:759a6aac-f645-d815-f9cb-8f2a81e044f7}]" +2025-10-30T18:23:14.751Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:14.751Z [INFO] server-e50501f4-fe5a-05bb-6294-bffb06a17713: entering follower state: follower="Node at e50501f4-fe5a-05bb-6294-bffb06a17713 [Follower]" leader-address= leader-id= +2025-10-30T18:23:14.751Z [INFO] server-759a6aac-f645-d815-f9cb-8f2a81e044f7: entering follower state: follower="Node at 759a6aac-f645-d815-f9cb-8f2a81e044f7 [Follower]" leader-address= leader-id= +2025-10-30T18:23:14.809Z [WARN] server-759a6aac-f645-d815-f9cb-8f2a81e044f7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:14.809Z [INFO] server-759a6aac-f645-d815-f9cb-8f2a81e044f7: entering candidate state: node="Node at 759a6aac-f645-d815-f9cb-8f2a81e044f7 [Candidate]" term=2 +2025-10-30T18:23:14.809Z [INFO] server-759a6aac-f645-d815-f9cb-8f2a81e044f7: election won: term=2 tally=2 +2025-10-30T18:23:14.809Z [INFO] server-759a6aac-f645-d815-f9cb-8f2a81e044f7: entering leader state: leader="Node at 759a6aac-f645-d815-f9cb-8f2a81e044f7 [Leader]" +2025-10-30T18:23:14.809Z [INFO] server-759a6aac-f645-d815-f9cb-8f2a81e044f7: added peer, starting replication: peer=server-4a293cbe-1c35-7fa6-1023-fb8669dffeba +2025-10-30T18:23:14.809Z [INFO] server-759a6aac-f645-d815-f9cb-8f2a81e044f7: added peer, starting replication: peer=server-e50501f4-fe5a-05bb-6294-bffb06a17713 +2025-10-30T18:23:14.809Z [INFO] server-759a6aac-f645-d815-f9cb-8f2a81e044f7: pipelining replication: peer="{Voter server-e50501f4-fe5a-05bb-6294-bffb06a17713 e50501f4-fe5a-05bb-6294-bffb06a17713}" +2025-10-30T18:23:14.809Z [INFO] server-759a6aac-f645-d815-f9cb-8f2a81e044f7: pipelining replication: peer="{Voter server-4a293cbe-1c35-7fa6-1023-fb8669dffeba 4a293cbe-1c35-7fa6-1023-fb8669dffeba}" +2025-10-30T18:23:14.914Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:14.809641738 +0000 UTC m=+117.883171042 (1 nodes), 58.384438ms from start of poll, 58.683899ms from cluster start. Timeout at 2025-10-30 18:23:14.914712971 +0000 UTC m=+117.988242260, 105.071218ms after stability +2025-10-30T18:23:14.914Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:15.020Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:14.914878516 +0000 UTC m=+117.988407808 (2 nodes), 1.153µs from start of poll, 163.920665ms from cluster start. Timeout at 2025-10-30 18:23:15.019901878 +0000 UTC m=+118.093431172, 105.023364ms after stability +2025-10-30T18:23:15.253Z [INFO] server-759a6aac-f645-d815-f9cb-8f2a81e044f7: aborting pipeline replication: peer="{Voter server-e50501f4-fe5a-05bb-6294-bffb06a17713 e50501f4-fe5a-05bb-6294-bffb06a17713}" +2025-10-30T18:23:15.253Z [INFO] server-759a6aac-f645-d815-f9cb-8f2a81e044f7: aborting pipeline replication: peer="{Voter server-4a293cbe-1c35-7fa6-1023-fb8669dffeba 4a293cbe-1c35-7fa6-1023-fb8669dffeba}" +--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.50s) === RUN TestRaft_LeadershipTransferIgnoresNonvoters -2025-10-30T18:19:21.393Z [INFO] server-b374f233-db7b-bf1a-4a12-ed6c4b52db93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b374f233-db7b-bf1a-4a12-ed6c4b52db93 Address:b374f233-db7b-bf1a-4a12-ed6c4b52db93} {Suffrage:Voter ID:server-636418b4-0b2f-925b-a178-214aab4f6ac9 Address:636418b4-0b2f-925b-a178-214aab4f6ac9}]" -2025-10-30T18:19:21.393Z [INFO] server-636418b4-0b2f-925b-a178-214aab4f6ac9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b374f233-db7b-bf1a-4a12-ed6c4b52db93 Address:b374f233-db7b-bf1a-4a12-ed6c4b52db93} {Suffrage:Voter ID:server-636418b4-0b2f-925b-a178-214aab4f6ac9 Address:636418b4-0b2f-925b-a178-214aab4f6ac9}]" -2025-10-30T18:19:21.393Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:21.393Z [INFO] server-b374f233-db7b-bf1a-4a12-ed6c4b52db93: entering follower state: follower="Node at b374f233-db7b-bf1a-4a12-ed6c4b52db93 [Follower]" leader-address= leader-id= -2025-10-30T18:19:21.393Z [INFO] server-636418b4-0b2f-925b-a178-214aab4f6ac9: entering follower state: follower="Node at 636418b4-0b2f-925b-a178-214aab4f6ac9 [Follower]" leader-address= leader-id= -2025-10-30T18:19:21.457Z [WARN] server-636418b4-0b2f-925b-a178-214aab4f6ac9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:21.457Z [INFO] server-636418b4-0b2f-925b-a178-214aab4f6ac9: entering candidate state: node="Node at 636418b4-0b2f-925b-a178-214aab4f6ac9 [Candidate]" term=2 -2025-10-30T18:19:21.457Z [INFO] server-636418b4-0b2f-925b-a178-214aab4f6ac9: election won: term=2 tally=2 -2025-10-30T18:19:21.457Z [INFO] server-636418b4-0b2f-925b-a178-214aab4f6ac9: entering leader state: leader="Node at 636418b4-0b2f-925b-a178-214aab4f6ac9 [Leader]" -2025-10-30T18:19:21.457Z [INFO] server-636418b4-0b2f-925b-a178-214aab4f6ac9: added peer, starting replication: peer=server-b374f233-db7b-bf1a-4a12-ed6c4b52db93 -2025-10-30T18:19:21.457Z [INFO] server-636418b4-0b2f-925b-a178-214aab4f6ac9: pipelining replication: peer="{Voter server-b374f233-db7b-bf1a-4a12-ed6c4b52db93 b374f233-db7b-bf1a-4a12-ed6c4b52db93}" -2025-10-30T18:19:21.562Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:21.457291338 +0000 UTC m=+107.735965235 (1 nodes), 63.391326ms from start of poll, 63.523175ms from cluster start. Timeout at 2025-10-30 18:19:21.562291941 +0000 UTC m=+107.840965838, 105.000603ms after stability -2025-10-30T18:19:21.562Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:21.668Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:21.56259575 +0000 UTC m=+107.841269647 (1 nodes), 635ns from start of poll, 168.827587ms from cluster start. Timeout at 2025-10-30 18:19:21.667682502 +0000 UTC m=+107.946356442, 105.086795ms after stability -2025-10-30T18:19:21.668Z [INFO] server-636418b4-0b2f-925b-a178-214aab4f6ac9: updating configuration: command=DemoteVoter server-id=server-b374f233-db7b-bf1a-4a12-ed6c4b52db93 server-addr= servers="[{Suffrage:Nonvoter ID:server-b374f233-db7b-bf1a-4a12-ed6c4b52db93 Address:b374f233-db7b-bf1a-4a12-ed6c4b52db93} {Suffrage:Voter ID:server-636418b4-0b2f-925b-a178-214aab4f6ac9 Address:636418b4-0b2f-925b-a178-214aab4f6ac9}]" -2025-10-30T18:19:21.668Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:21.774Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:21.668494793 +0000 UTC m=+107.947168690 (1 nodes), 404ns from start of poll, 274.72663ms from cluster start. Timeout at 2025-10-30 18:19:21.773533911 +0000 UTC m=+108.052207807, 105.039117ms after stability -2025-10-30T18:19:21.774Z [INFO] server-636418b4-0b2f-925b-a178-214aab4f6ac9: aborting pipeline replication: peer="{Voter server-b374f233-db7b-bf1a-4a12-ed6c4b52db93 b374f233-db7b-bf1a-4a12-ed6c4b52db93}" ---- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.38s) +2025-10-30T18:23:15.254Z [INFO] server-ad69a45d-2ab0-7665-7bc8-6ad95cbc5436: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad69a45d-2ab0-7665-7bc8-6ad95cbc5436 Address:ad69a45d-2ab0-7665-7bc8-6ad95cbc5436} {Suffrage:Voter ID:server-b7493676-df42-63f3-4e8c-519046e2f8d9 Address:b7493676-df42-63f3-4e8c-519046e2f8d9}]" +2025-10-30T18:23:15.254Z [INFO] server-b7493676-df42-63f3-4e8c-519046e2f8d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad69a45d-2ab0-7665-7bc8-6ad95cbc5436 Address:ad69a45d-2ab0-7665-7bc8-6ad95cbc5436} {Suffrage:Voter ID:server-b7493676-df42-63f3-4e8c-519046e2f8d9 Address:b7493676-df42-63f3-4e8c-519046e2f8d9}]" +2025-10-30T18:23:15.254Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:15.254Z [INFO] server-ad69a45d-2ab0-7665-7bc8-6ad95cbc5436: entering follower state: follower="Node at ad69a45d-2ab0-7665-7bc8-6ad95cbc5436 [Follower]" leader-address= leader-id= +2025-10-30T18:23:15.255Z [INFO] server-b7493676-df42-63f3-4e8c-519046e2f8d9: entering follower state: follower="Node at b7493676-df42-63f3-4e8c-519046e2f8d9 [Follower]" leader-address= leader-id= +2025-10-30T18:23:15.313Z [WARN] server-ad69a45d-2ab0-7665-7bc8-6ad95cbc5436: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:15.313Z [INFO] server-ad69a45d-2ab0-7665-7bc8-6ad95cbc5436: entering candidate state: node="Node at ad69a45d-2ab0-7665-7bc8-6ad95cbc5436 [Candidate]" term=2 +2025-10-30T18:23:15.313Z [INFO] server-ad69a45d-2ab0-7665-7bc8-6ad95cbc5436: election won: term=2 tally=2 +2025-10-30T18:23:15.313Z [INFO] server-ad69a45d-2ab0-7665-7bc8-6ad95cbc5436: entering leader state: leader="Node at ad69a45d-2ab0-7665-7bc8-6ad95cbc5436 [Leader]" +2025-10-30T18:23:15.313Z [INFO] server-ad69a45d-2ab0-7665-7bc8-6ad95cbc5436: added peer, starting replication: peer=server-b7493676-df42-63f3-4e8c-519046e2f8d9 +2025-10-30T18:23:15.313Z [INFO] server-ad69a45d-2ab0-7665-7bc8-6ad95cbc5436: pipelining replication: peer="{Voter server-b7493676-df42-63f3-4e8c-519046e2f8d9 b7493676-df42-63f3-4e8c-519046e2f8d9}" +2025-10-30T18:23:15.419Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:15.313745603 +0000 UTC m=+118.387274907 (1 nodes), 58.733ms from start of poll, 58.987586ms from cluster start. Timeout at 2025-10-30 18:23:15.418818026 +0000 UTC m=+118.492347319, 105.072412ms after stability +2025-10-30T18:23:15.419Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:15.525Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:15.419312383 +0000 UTC m=+118.492841677 (1 nodes), 1.493µs from start of poll, 164.554356ms from cluster start. Timeout at 2025-10-30 18:23:15.524424942 +0000 UTC m=+118.597954279, 105.112602ms after stability +2025-10-30T18:23:15.525Z [INFO] server-ad69a45d-2ab0-7665-7bc8-6ad95cbc5436: updating configuration: command=DemoteVoter server-id=server-b7493676-df42-63f3-4e8c-519046e2f8d9 server-addr= servers="[{Suffrage:Voter ID:server-ad69a45d-2ab0-7665-7bc8-6ad95cbc5436 Address:ad69a45d-2ab0-7665-7bc8-6ad95cbc5436} {Suffrage:Nonvoter ID:server-b7493676-df42-63f3-4e8c-519046e2f8d9 Address:b7493676-df42-63f3-4e8c-519046e2f8d9}]" +2025-10-30T18:23:15.525Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:15.631Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:15.525620735 +0000 UTC m=+118.599150023 (1 nodes), 841ns from start of poll, 270.862702ms from cluster start. Timeout at 2025-10-30 18:23:15.630697252 +0000 UTC m=+118.704226541, 105.076518ms after stability +2025-10-30T18:23:15.631Z [INFO] server-ad69a45d-2ab0-7665-7bc8-6ad95cbc5436: aborting pipeline replication: peer="{Voter server-b7493676-df42-63f3-4e8c-519046e2f8d9 b7493676-df42-63f3-4e8c-519046e2f8d9}" +--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.40s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap -2025-10-30T18:19:21.774Z [INFO] server-9f6634f9-477d-72bc-ce02-f90dd8e69273: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f6634f9-477d-72bc-ce02-f90dd8e69273 Address:9f6634f9-477d-72bc-ce02-f90dd8e69273} {Suffrage:Voter ID:server-28f891b1-7ab4-4221-09da-1925aed3535b Address:28f891b1-7ab4-4221-09da-1925aed3535b}]" -2025-10-30T18:19:21.774Z [INFO] server-9f6634f9-477d-72bc-ce02-f90dd8e69273: entering follower state: follower="Node at 9f6634f9-477d-72bc-ce02-f90dd8e69273 [Follower]" leader-address= leader-id= -2025-10-30T18:19:21.774Z [INFO] server-28f891b1-7ab4-4221-09da-1925aed3535b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f6634f9-477d-72bc-ce02-f90dd8e69273 Address:9f6634f9-477d-72bc-ce02-f90dd8e69273} {Suffrage:Voter ID:server-28f891b1-7ab4-4221-09da-1925aed3535b Address:28f891b1-7ab4-4221-09da-1925aed3535b}]" -2025-10-30T18:19:21.774Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:21.774Z [INFO] server-28f891b1-7ab4-4221-09da-1925aed3535b: entering follower state: follower="Node at 28f891b1-7ab4-4221-09da-1925aed3535b [Follower]" leader-address= leader-id= -2025-10-30T18:19:21.847Z [WARN] server-9f6634f9-477d-72bc-ce02-f90dd8e69273: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:21.847Z [INFO] server-9f6634f9-477d-72bc-ce02-f90dd8e69273: entering candidate state: node="Node at 9f6634f9-477d-72bc-ce02-f90dd8e69273 [Candidate]" term=2 -2025-10-30T18:19:21.848Z [INFO] server-9f6634f9-477d-72bc-ce02-f90dd8e69273: election won: term=2 tally=2 -2025-10-30T18:19:21.848Z [INFO] server-9f6634f9-477d-72bc-ce02-f90dd8e69273: entering leader state: leader="Node at 9f6634f9-477d-72bc-ce02-f90dd8e69273 [Leader]" -2025-10-30T18:19:21.848Z [INFO] server-9f6634f9-477d-72bc-ce02-f90dd8e69273: added peer, starting replication: peer=server-28f891b1-7ab4-4221-09da-1925aed3535b -2025-10-30T18:19:21.848Z [INFO] server-9f6634f9-477d-72bc-ce02-f90dd8e69273: pipelining replication: peer="{Voter server-28f891b1-7ab4-4221-09da-1925aed3535b 28f891b1-7ab4-4221-09da-1925aed3535b}" -2025-10-30T18:19:21.954Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:21.848111175 +0000 UTC m=+108.126785072 (1 nodes), 73.12359ms from start of poll, 73.277005ms from cluster start. Timeout at 2025-10-30 18:19:21.953116354 +0000 UTC m=+108.231790250, 105.005178ms after stability -2025-10-30T18:19:21.954Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:22.059Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:21.954226448 +0000 UTC m=+108.232900345 (1 nodes), 1.333µs from start of poll, 179.392278ms from cluster start. Timeout at 2025-10-30 18:19:22.059242874 +0000 UTC m=+108.337916785, 105.01644ms after stability -2025-10-30T18:19:22.070Z [INFO] server-28f891b1-7ab4-4221-09da-1925aed3535b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f6634f9-477d-72bc-ce02-f90dd8e69273 Address:9f6634f9-477d-72bc-ce02-f90dd8e69273} {Suffrage:Voter ID:server-28f891b1-7ab4-4221-09da-1925aed3535b Address:28f891b1-7ab4-4221-09da-1925aed3535b}]" -2025-10-30T18:19:22.070Z [INFO] server-9f6634f9-477d-72bc-ce02-f90dd8e69273: aborting pipeline replication: peer="{Voter server-28f891b1-7ab4-4221-09da-1925aed3535b 28f891b1-7ab4-4221-09da-1925aed3535b}" ---- PASS: TestRaft_GetConfigurationNoBootstrap (0.30s) +2025-10-30T18:23:15.653Z [INFO] server-cdb8fd3d-2aa2-0b0b-81ab-5a1beff5a46f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdb8fd3d-2aa2-0b0b-81ab-5a1beff5a46f Address:cdb8fd3d-2aa2-0b0b-81ab-5a1beff5a46f} {Suffrage:Voter ID:server-6fe1f3f6-6bb3-b4df-71a8-bd901e821975 Address:6fe1f3f6-6bb3-b4df-71a8-bd901e821975}]" +2025-10-30T18:23:15.653Z [INFO] server-6fe1f3f6-6bb3-b4df-71a8-bd901e821975: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdb8fd3d-2aa2-0b0b-81ab-5a1beff5a46f Address:cdb8fd3d-2aa2-0b0b-81ab-5a1beff5a46f} {Suffrage:Voter ID:server-6fe1f3f6-6bb3-b4df-71a8-bd901e821975 Address:6fe1f3f6-6bb3-b4df-71a8-bd901e821975}]" +2025-10-30T18:23:15.653Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:15.653Z [INFO] server-cdb8fd3d-2aa2-0b0b-81ab-5a1beff5a46f: entering follower state: follower="Node at cdb8fd3d-2aa2-0b0b-81ab-5a1beff5a46f [Follower]" leader-address= leader-id= +2025-10-30T18:23:15.653Z [INFO] server-6fe1f3f6-6bb3-b4df-71a8-bd901e821975: entering follower state: follower="Node at 6fe1f3f6-6bb3-b4df-71a8-bd901e821975 [Follower]" leader-address= leader-id= +2025-10-30T18:23:15.705Z [WARN] server-6fe1f3f6-6bb3-b4df-71a8-bd901e821975: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:15.705Z [INFO] server-6fe1f3f6-6bb3-b4df-71a8-bd901e821975: entering candidate state: node="Node at 6fe1f3f6-6bb3-b4df-71a8-bd901e821975 [Candidate]" term=2 +2025-10-30T18:23:15.705Z [INFO] server-6fe1f3f6-6bb3-b4df-71a8-bd901e821975: election won: term=2 tally=2 +2025-10-30T18:23:15.705Z [INFO] server-6fe1f3f6-6bb3-b4df-71a8-bd901e821975: entering leader state: leader="Node at 6fe1f3f6-6bb3-b4df-71a8-bd901e821975 [Leader]" +2025-10-30T18:23:15.705Z [INFO] server-6fe1f3f6-6bb3-b4df-71a8-bd901e821975: added peer, starting replication: peer=server-cdb8fd3d-2aa2-0b0b-81ab-5a1beff5a46f +2025-10-30T18:23:15.705Z [INFO] server-6fe1f3f6-6bb3-b4df-71a8-bd901e821975: pipelining replication: peer="{Voter server-cdb8fd3d-2aa2-0b0b-81ab-5a1beff5a46f cdb8fd3d-2aa2-0b0b-81ab-5a1beff5a46f}" +2025-10-30T18:23:15.811Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:15.70542566 +0000 UTC m=+118.778954954 (1 nodes), 51.837247ms from start of poll, 52.024512ms from cluster start. Timeout at 2025-10-30 18:23:15.810428281 +0000 UTC m=+118.883957573, 105.002619ms after stability +2025-10-30T18:23:15.811Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:15.917Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:15.811184509 +0000 UTC m=+118.884713798 (1 nodes), 545ns from start of poll, 157.783356ms from cluster start. Timeout at 2025-10-30 18:23:15.916282237 +0000 UTC m=+118.989811527, 105.097729ms after stability +2025-10-30T18:23:16.028Z [INFO] server-6fe1f3f6-6bb3-b4df-71a8-bd901e821975: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdb8fd3d-2aa2-0b0b-81ab-5a1beff5a46f Address:cdb8fd3d-2aa2-0b0b-81ab-5a1beff5a46f} {Suffrage:Voter ID:server-6fe1f3f6-6bb3-b4df-71a8-bd901e821975 Address:6fe1f3f6-6bb3-b4df-71a8-bd901e821975}]" +2025-10-30T18:23:16.028Z [INFO] server-6fe1f3f6-6bb3-b4df-71a8-bd901e821975: aborting pipeline replication: peer="{Voter server-cdb8fd3d-2aa2-0b0b-81ab-5a1beff5a46f cdb8fd3d-2aa2-0b0b-81ab-5a1beff5a46f}" +2025-10-30T18:23:16.528Z [ERROR] server-6fe1f3f6-6bb3-b4df-71a8-bd901e821975: failed to heartbeat to: peer=cdb8fd3d-2aa2-0b0b-81ab-5a1beff5a46f backoff time=10ms error="command timed out" +--- PASS: TestRaft_GetConfigurationNoBootstrap (0.88s) === RUN TestRaft_LogStoreIsMonotonic -2025-10-30T18:19:22.071Z [INFO] server-5b298b0f-338b-a374-1fe3-5be47e63d82d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5b298b0f-338b-a374-1fe3-5be47e63d82d Address:5b298b0f-338b-a374-1fe3-5be47e63d82d}]" -2025-10-30T18:19:22.071Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:22.071Z [INFO] server-5b298b0f-338b-a374-1fe3-5be47e63d82d: entering follower state: follower="Node at 5b298b0f-338b-a374-1fe3-5be47e63d82d [Follower]" leader-address= leader-id= -2025-10-30T18:19:22.133Z [WARN] server-5b298b0f-338b-a374-1fe3-5be47e63d82d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:22.133Z [INFO] server-5b298b0f-338b-a374-1fe3-5be47e63d82d: entering candidate state: node="Node at 5b298b0f-338b-a374-1fe3-5be47e63d82d [Candidate]" term=2 -2025-10-30T18:19:22.133Z [INFO] server-5b298b0f-338b-a374-1fe3-5be47e63d82d: election won: term=2 tally=1 -2025-10-30T18:19:22.133Z [INFO] server-5b298b0f-338b-a374-1fe3-5be47e63d82d: entering leader state: leader="Node at 5b298b0f-338b-a374-1fe3-5be47e63d82d [Leader]" -2025-10-30T18:19:22.239Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:22.133375059 +0000 UTC m=+108.412048957 (1 nodes), 61.863089ms from start of poll, 61.948485ms from cluster start. Timeout at 2025-10-30 18:19:22.238375611 +0000 UTC m=+108.517049511, 105.000554ms after stability ---- PASS: TestRaft_LogStoreIsMonotonic (0.17s) +2025-10-30T18:23:16.529Z [INFO] server-a25f15d5-441f-e8c3-32dd-f902d9014091: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a25f15d5-441f-e8c3-32dd-f902d9014091 Address:a25f15d5-441f-e8c3-32dd-f902d9014091}]" +2025-10-30T18:23:16.529Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:16.529Z [INFO] server-a25f15d5-441f-e8c3-32dd-f902d9014091: entering follower state: follower="Node at a25f15d5-441f-e8c3-32dd-f902d9014091 [Follower]" leader-address= leader-id= +2025-10-30T18:23:16.597Z [WARN] server-a25f15d5-441f-e8c3-32dd-f902d9014091: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:16.597Z [INFO] server-a25f15d5-441f-e8c3-32dd-f902d9014091: entering candidate state: node="Node at a25f15d5-441f-e8c3-32dd-f902d9014091 [Candidate]" term=2 +2025-10-30T18:23:16.597Z [INFO] server-a25f15d5-441f-e8c3-32dd-f902d9014091: election won: term=2 tally=1 +2025-10-30T18:23:16.597Z [INFO] server-a25f15d5-441f-e8c3-32dd-f902d9014091: entering leader state: leader="Node at a25f15d5-441f-e8c3-32dd-f902d9014091 [Leader]" +2025-10-30T18:23:16.702Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:16.597552569 +0000 UTC m=+119.671081861 (1 nodes), 67.892594ms from start of poll, 68.033945ms from cluster start. Timeout at 2025-10-30 18:23:16.70255853 +0000 UTC m=+119.776087827, 105.005966ms after stability +--- PASS: TestRaft_LogStoreIsMonotonic (0.21s) === RUN TestRaft_CacheLogWithStoreError -2025-10-30T18:19:22.240Z [INFO] server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1431383-6e28-4ce3-d3f4-c0afd09d061d Address:e1431383-6e28-4ce3-d3f4-c0afd09d061d} {Suffrage:Voter ID:server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988 Address:5fad65b1-0e46-68b6-1f9c-ad508f5f4988}]" -2025-10-30T18:19:22.240Z [INFO] server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1431383-6e28-4ce3-d3f4-c0afd09d061d Address:e1431383-6e28-4ce3-d3f4-c0afd09d061d} {Suffrage:Voter ID:server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988 Address:5fad65b1-0e46-68b6-1f9c-ad508f5f4988}]" -2025-10-30T18:19:22.240Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:22.240Z [INFO] server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: entering follower state: follower="Node at e1431383-6e28-4ce3-d3f4-c0afd09d061d [Follower]" leader-address= leader-id= -2025-10-30T18:19:22.240Z [INFO] server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988: entering follower state: follower="Node at 5fad65b1-0e46-68b6-1f9c-ad508f5f4988 [Follower]" leader-address= leader-id= -2025-10-30T18:19:22.317Z [WARN] server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:22.317Z [INFO] server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: entering candidate state: node="Node at e1431383-6e28-4ce3-d3f4-c0afd09d061d [Candidate]" term=2 -2025-10-30T18:19:22.317Z [INFO] server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: election won: term=2 tally=2 -2025-10-30T18:19:22.317Z [INFO] server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: entering leader state: leader="Node at e1431383-6e28-4ce3-d3f4-c0afd09d061d [Leader]" -2025-10-30T18:19:22.317Z [INFO] server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: added peer, starting replication: peer=server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988 -2025-10-30T18:19:22.317Z [INFO] server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: pipelining replication: peer="{Voter server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988 5fad65b1-0e46-68b6-1f9c-ad508f5f4988}" -2025-10-30T18:19:22.422Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:22.317631434 +0000 UTC m=+108.596305331 (1 nodes), 77.378111ms from start of poll, 77.546331ms from cluster start. Timeout at 2025-10-30 18:19:22.422634678 +0000 UTC m=+108.701308579, 105.003248ms after stability -2025-10-30T18:19:22.422Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:22.528Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:22.422721257 +0000 UTC m=+108.701395159 (1 nodes), 745ns from start of poll, 182.636159ms from cluster start. Timeout at 2025-10-30 18:19:22.527741092 +0000 UTC m=+108.806414999, 105.01984ms after stability -2025-10-30T18:19:22.530Z [ERROR] server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988: failed to append to logs: error="unable to store logs within log store, err: "some error"" -2025-10-30T18:19:22.530Z [WARN] server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988: failed to get previous log: previous-index=9 last-index=7 error="log not found" -2025-10-30T18:19:22.530Z [INFO] server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: aborting pipeline replication: peer="{Voter server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988 5fad65b1-0e46-68b6-1f9c-ad508f5f4988}" -2025-10-30T18:19:22.539Z [INFO] server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: pipelining replication: peer="{Voter server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988 5fad65b1-0e46-68b6-1f9c-ad508f5f4988}" -2025-10-30T18:19:22.539Z [INFO] server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1431383-6e28-4ce3-d3f4-c0afd09d061d Address:e1431383-6e28-4ce3-d3f4-c0afd09d061d} {Suffrage:Voter ID:server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988 Address:5fad65b1-0e46-68b6-1f9c-ad508f5f4988}]" -2025-10-30T18:19:22.540Z [INFO] server-e1431383-6e28-4ce3-d3f4-c0afd09d061d: aborting pipeline replication: peer="{Voter server-5fad65b1-0e46-68b6-1f9c-ad508f5f4988 5fad65b1-0e46-68b6-1f9c-ad508f5f4988}" ---- PASS: TestRaft_CacheLogWithStoreError (0.30s) +2025-10-30T18:23:16.744Z [INFO] server-0b90ca0d-8cff-045c-286c-340b04cc3971: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b90ca0d-8cff-045c-286c-340b04cc3971 Address:0b90ca0d-8cff-045c-286c-340b04cc3971} {Suffrage:Voter ID:server-63c6ced1-b009-38a5-88b5-505bc09d881c Address:63c6ced1-b009-38a5-88b5-505bc09d881c}]" +2025-10-30T18:23:16.744Z [INFO] server-0b90ca0d-8cff-045c-286c-340b04cc3971: entering follower state: follower="Node at 0b90ca0d-8cff-045c-286c-340b04cc3971 [Follower]" leader-address= leader-id= +2025-10-30T18:23:16.744Z [INFO] server-63c6ced1-b009-38a5-88b5-505bc09d881c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b90ca0d-8cff-045c-286c-340b04cc3971 Address:0b90ca0d-8cff-045c-286c-340b04cc3971} {Suffrage:Voter ID:server-63c6ced1-b009-38a5-88b5-505bc09d881c Address:63c6ced1-b009-38a5-88b5-505bc09d881c}]" +2025-10-30T18:23:16.744Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:16.744Z [INFO] server-63c6ced1-b009-38a5-88b5-505bc09d881c: entering follower state: follower="Node at 63c6ced1-b009-38a5-88b5-505bc09d881c [Follower]" leader-address= leader-id= +2025-10-30T18:23:16.809Z [WARN] server-0b90ca0d-8cff-045c-286c-340b04cc3971: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:16.809Z [INFO] server-0b90ca0d-8cff-045c-286c-340b04cc3971: entering candidate state: node="Node at 0b90ca0d-8cff-045c-286c-340b04cc3971 [Candidate]" term=2 +2025-10-30T18:23:16.810Z [INFO] server-0b90ca0d-8cff-045c-286c-340b04cc3971: election won: term=2 tally=2 +2025-10-30T18:23:16.810Z [INFO] server-0b90ca0d-8cff-045c-286c-340b04cc3971: entering leader state: leader="Node at 0b90ca0d-8cff-045c-286c-340b04cc3971 [Leader]" +2025-10-30T18:23:16.810Z [INFO] server-0b90ca0d-8cff-045c-286c-340b04cc3971: added peer, starting replication: peer=server-63c6ced1-b009-38a5-88b5-505bc09d881c +2025-10-30T18:23:16.810Z [INFO] server-0b90ca0d-8cff-045c-286c-340b04cc3971: pipelining replication: peer="{Voter server-63c6ced1-b009-38a5-88b5-505bc09d881c 63c6ced1-b009-38a5-88b5-505bc09d881c}" +2025-10-30T18:23:16.916Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:16.810070334 +0000 UTC m=+119.883599663 (1 nodes), 65.12212ms from start of poll, 65.364297ms from cluster start. Timeout at 2025-10-30 18:23:16.915074149 +0000 UTC m=+119.988603449, 105.003786ms after stability +2025-10-30T18:23:16.916Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:17.022Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:16.916294541 +0000 UTC m=+119.989823832 (1 nodes), 865ns from start of poll, 171.588466ms from cluster start. Timeout at 2025-10-30 18:23:17.021321316 +0000 UTC m=+120.094850627, 105.026795ms after stability +2025-10-30T18:23:17.022Z [ERROR] server-63c6ced1-b009-38a5-88b5-505bc09d881c: failed to append to logs: error="unable to store logs within log store, err: "some error"" +2025-10-30T18:23:17.022Z [WARN] server-63c6ced1-b009-38a5-88b5-505bc09d881c: failed to get previous log: previous-index=9 last-index=7 error="log not found" +2025-10-30T18:23:17.022Z [INFO] server-0b90ca0d-8cff-045c-286c-340b04cc3971: aborting pipeline replication: peer="{Voter server-63c6ced1-b009-38a5-88b5-505bc09d881c 63c6ced1-b009-38a5-88b5-505bc09d881c}" +2025-10-30T18:23:17.029Z [INFO] server-0b90ca0d-8cff-045c-286c-340b04cc3971: pipelining replication: peer="{Voter server-63c6ced1-b009-38a5-88b5-505bc09d881c 63c6ced1-b009-38a5-88b5-505bc09d881c}" +2025-10-30T18:23:17.075Z [INFO] server-63c6ced1-b009-38a5-88b5-505bc09d881c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b90ca0d-8cff-045c-286c-340b04cc3971 Address:0b90ca0d-8cff-045c-286c-340b04cc3971} {Suffrage:Voter ID:server-63c6ced1-b009-38a5-88b5-505bc09d881c Address:63c6ced1-b009-38a5-88b5-505bc09d881c}]" +2025-10-30T18:23:17.075Z [INFO] server-0b90ca0d-8cff-045c-286c-340b04cc3971: aborting pipeline replication: peer="{Voter server-63c6ced1-b009-38a5-88b5-505bc09d881c 63c6ced1-b009-38a5-88b5-505bc09d881c}" +2025-10-30T18:23:17.535Z [ERROR] server-0b90ca0d-8cff-045c-286c-340b04cc3971: failed to heartbeat to: peer=63c6ced1-b009-38a5-88b5-505bc09d881c backoff time=10ms error="command timed out" +--- PASS: TestRaft_CacheLogWithStoreError (0.80s) === RUN TestRaft_ReloadConfig -2025-10-30T18:19:22.540Z [INFO] server-3d212418-56d8-cddd-a2e6-5b0e7ef82d9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d212418-56d8-cddd-a2e6-5b0e7ef82d9b Address:3d212418-56d8-cddd-a2e6-5b0e7ef82d9b}]" -2025-10-30T18:19:22.540Z [INFO] server-3d212418-56d8-cddd-a2e6-5b0e7ef82d9b: entering follower state: follower="Node at 3d212418-56d8-cddd-a2e6-5b0e7ef82d9b [Shutdown]" leader-address= leader-id= +2025-10-30T18:23:17.548Z [INFO] server-46d85059-5fc6-b712-2a0b-f9599f6df8e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-46d85059-5fc6-b712-2a0b-f9599f6df8e3 Address:46d85059-5fc6-b712-2a0b-f9599f6df8e3}]" --- PASS: TestRaft_ReloadConfig (0.00s) === RUN TestRaft_ReloadConfigValidates -2025-10-30T18:19:22.540Z [INFO] server-c060c281-996a-b3bf-90b3-e0998937da23: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c060c281-996a-b3bf-90b3-e0998937da23 Address:c060c281-996a-b3bf-90b3-e0998937da23}]" +2025-10-30T18:23:17.549Z [INFO] server-3dd34655-7063-cf0f-a950-90232623b45b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3dd34655-7063-cf0f-a950-90232623b45b Address:3dd34655-7063-cf0f-a950-90232623b45b}]" --- PASS: TestRaft_ReloadConfigValidates (0.00s) === RUN TestRaft_InstallSnapshot_InvalidPeers -2025-10-30T18:19:22.541Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)" +2025-10-30T18:23:17.549Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)" --- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s) === RUN TestRaft_VoteNotGranted_WhenNodeNotInCluster -2025-10-30T18:19:22.541Z [INFO] server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 Address:405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3} {Suffrage:Voter ID:server-677caaff-e2d1-f687-42eb-19d578168f63 Address:677caaff-e2d1-f687-42eb-19d578168f63} {Suffrage:Voter ID:server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 Address:0b7d726a-82c8-0b38-b52b-3a0fecf7bd83}]" -2025-10-30T18:19:22.541Z [INFO] server-677caaff-e2d1-f687-42eb-19d578168f63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 Address:405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3} {Suffrage:Voter ID:server-677caaff-e2d1-f687-42eb-19d578168f63 Address:677caaff-e2d1-f687-42eb-19d578168f63} {Suffrage:Voter ID:server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 Address:0b7d726a-82c8-0b38-b52b-3a0fecf7bd83}]" -2025-10-30T18:19:22.541Z [INFO] server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3: entering follower state: follower="Node at 405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 [Follower]" leader-address= leader-id= -2025-10-30T18:19:22.541Z [INFO] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 Address:405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3} {Suffrage:Voter ID:server-677caaff-e2d1-f687-42eb-19d578168f63 Address:677caaff-e2d1-f687-42eb-19d578168f63} {Suffrage:Voter ID:server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 Address:0b7d726a-82c8-0b38-b52b-3a0fecf7bd83}]" -2025-10-30T18:19:22.541Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:22.541Z [INFO] server-677caaff-e2d1-f687-42eb-19d578168f63: entering follower state: follower="Node at 677caaff-e2d1-f687-42eb-19d578168f63 [Follower]" leader-address= leader-id= -2025-10-30T18:19:22.541Z [INFO] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: entering follower state: follower="Node at 0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 [Follower]" leader-address= leader-id= -2025-10-30T18:19:22.599Z [WARN] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:22.599Z [INFO] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: entering candidate state: node="Node at 0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 [Candidate]" term=2 -2025-10-30T18:19:22.599Z [INFO] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: election won: term=2 tally=2 -2025-10-30T18:19:22.599Z [INFO] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: entering leader state: leader="Node at 0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 [Leader]" -2025-10-30T18:19:22.599Z [INFO] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: added peer, starting replication: peer=server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 -2025-10-30T18:19:22.599Z [INFO] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: added peer, starting replication: peer=server-677caaff-e2d1-f687-42eb-19d578168f63 -2025-10-30T18:19:22.599Z [INFO] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: pipelining replication: peer="{Voter server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3}" -2025-10-30T18:19:22.599Z [INFO] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: pipelining replication: peer="{Voter server-677caaff-e2d1-f687-42eb-19d578168f63 677caaff-e2d1-f687-42eb-19d578168f63}" -2025-10-30T18:19:22.705Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:22.599502723 +0000 UTC m=+108.878176621 (1 nodes), 57.830932ms from start of poll, 57.982788ms from cluster start. Timeout at 2025-10-30 18:19:22.704505813 +0000 UTC m=+108.983179711, 105.00309ms after stability -2025-10-30T18:19:22.710Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:22.816Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:22.710758083 +0000 UTC m=+108.989431986 (2 nodes), 938ns from start of poll, 169.238153ms from cluster start. Timeout at 2025-10-30 18:19:22.815772816 +0000 UTC m=+109.094446732, 105.014746ms after stability -2025-10-30T18:19:22.816Z [INFO] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: updating configuration: command=RemoveServer server-id=server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 server-addr= servers="[{Suffrage:Voter ID:server-677caaff-e2d1-f687-42eb-19d578168f63 Address:677caaff-e2d1-f687-42eb-19d578168f63} {Suffrage:Voter ID:server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 Address:0b7d726a-82c8-0b38-b52b-3a0fecf7bd83}]" -2025-10-30T18:19:22.816Z [INFO] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: removed peer, stopping replication: peer=server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 last-index=3 -2025-10-30T18:19:22.816Z [INFO] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: aborting pipeline replication: peer="{Voter server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3}" -2025-10-30T18:19:22.868Z [WARN] server-405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election -2025-10-30T18:19:22.926Z [ERROR] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to heartbeat to: peer=677caaff-e2d1-f687-42eb-19d578168f63 backoff time=10ms error="failed to connect to peer: 677caaff-e2d1-f687-42eb-19d578168f63" -2025-10-30T18:19:22.928Z [ERROR] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to pipeline appendEntries: peer="{Voter server-677caaff-e2d1-f687-42eb-19d578168f63 677caaff-e2d1-f687-42eb-19d578168f63}" error="append pipeline closed" -2025-10-30T18:19:22.928Z [INFO] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: aborting pipeline replication: peer="{Voter server-677caaff-e2d1-f687-42eb-19d578168f63 677caaff-e2d1-f687-42eb-19d578168f63}" -2025-10-30T18:19:22.936Z [ERROR] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to appendEntries to: peer="{Voter server-677caaff-e2d1-f687-42eb-19d578168f63 677caaff-e2d1-f687-42eb-19d578168f63}" error="failed to connect to peer: 677caaff-e2d1-f687-42eb-19d578168f63" -2025-10-30T18:19:22.946Z [ERROR] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to heartbeat to: peer=677caaff-e2d1-f687-42eb-19d578168f63 backoff time=10ms error="failed to connect to peer: 677caaff-e2d1-f687-42eb-19d578168f63" -2025-10-30T18:19:22.957Z [ERROR] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to appendEntries to: peer="{Voter server-677caaff-e2d1-f687-42eb-19d578168f63 677caaff-e2d1-f687-42eb-19d578168f63}" error="failed to connect to peer: 677caaff-e2d1-f687-42eb-19d578168f63" -2025-10-30T18:19:22.966Z [ERROR] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to heartbeat to: peer=677caaff-e2d1-f687-42eb-19d578168f63 backoff time=10ms error="failed to connect to peer: 677caaff-e2d1-f687-42eb-19d578168f63" -2025-10-30T18:19:22.970Z [WARN] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to contact: server-id=server-677caaff-e2d1-f687-42eb-19d578168f63 time=50.027089ms -2025-10-30T18:19:22.970Z [WARN] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to contact quorum of nodes, stepping down -2025-10-30T18:19:22.970Z [INFO] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: entering follower state: follower="Node at 0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 [Follower]" leader-address= leader-id= -2025-10-30T18:19:22.974Z [ERROR] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to appendEntries to: peer="{Voter server-677caaff-e2d1-f687-42eb-19d578168f63 677caaff-e2d1-f687-42eb-19d578168f63}" error="failed to connect to peer: 677caaff-e2d1-f687-42eb-19d578168f63" -2025-10-30T18:19:23.045Z [WARN] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:23.046Z [INFO] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: entering candidate state: node="Node at 0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 [Candidate]" term=3 -2025-10-30T18:19:23.046Z [ERROR] server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83: failed to make requestVote RPC: target="{Voter server-677caaff-e2d1-f687-42eb-19d578168f63 677caaff-e2d1-f687-42eb-19d578168f63}" error="failed to connect to peer: 677caaff-e2d1-f687-42eb-19d578168f63" term=3 -2025-10-30T18:19:23.050Z [WARN] server-677caaff-e2d1-f687-42eb-19d578168f63: heartbeat timeout reached, starting election: last-leader-addr=0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 last-leader-id=server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 -2025-10-30T18:19:23.050Z [INFO] server-677caaff-e2d1-f687-42eb-19d578168f63: entering candidate state: node="Node at 677caaff-e2d1-f687-42eb-19d578168f63 [Candidate]" term=3 -2025-10-30T18:19:23.050Z [WARN] server-677caaff-e2d1-f687-42eb-19d578168f63: rejecting vote request since node is not in configuration: from=405b8914-7f6b-7b1b-cc41-6eaaf4b9a0a3 -2025-10-30T18:19:23.050Z [ERROR] server-677caaff-e2d1-f687-42eb-19d578168f63: failed to make requestVote RPC: target="{Voter server-0b7d726a-82c8-0b38-b52b-3a0fecf7bd83 0b7d726a-82c8-0b38-b52b-3a0fecf7bd83}" error="failed to connect to peer: 0b7d726a-82c8-0b38-b52b-3a0fecf7bd83" term=3 ---- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.51s) +2025-10-30T18:23:17.550Z [INFO] server-c2515547-12b2-720b-b1e9-cbe9788e6ec6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2515547-12b2-720b-b1e9-cbe9788e6ec6 Address:c2515547-12b2-720b-b1e9-cbe9788e6ec6} {Suffrage:Voter ID:server-54067828-cecb-736d-6c82-b8f29fed8403 Address:54067828-cecb-736d-6c82-b8f29fed8403} {Suffrage:Voter ID:server-317dcb4d-7d38-1578-d355-6cfa484b90f5 Address:317dcb4d-7d38-1578-d355-6cfa484b90f5}]" +2025-10-30T18:23:17.550Z [INFO] server-c2515547-12b2-720b-b1e9-cbe9788e6ec6: entering follower state: follower="Node at c2515547-12b2-720b-b1e9-cbe9788e6ec6 [Follower]" leader-address= leader-id= +2025-10-30T18:23:17.550Z [INFO] server-54067828-cecb-736d-6c82-b8f29fed8403: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2515547-12b2-720b-b1e9-cbe9788e6ec6 Address:c2515547-12b2-720b-b1e9-cbe9788e6ec6} {Suffrage:Voter ID:server-54067828-cecb-736d-6c82-b8f29fed8403 Address:54067828-cecb-736d-6c82-b8f29fed8403} {Suffrage:Voter ID:server-317dcb4d-7d38-1578-d355-6cfa484b90f5 Address:317dcb4d-7d38-1578-d355-6cfa484b90f5}]" +2025-10-30T18:23:17.550Z [INFO] server-54067828-cecb-736d-6c82-b8f29fed8403: entering follower state: follower="Node at 54067828-cecb-736d-6c82-b8f29fed8403 [Follower]" leader-address= leader-id= +2025-10-30T18:23:17.550Z [INFO] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2515547-12b2-720b-b1e9-cbe9788e6ec6 Address:c2515547-12b2-720b-b1e9-cbe9788e6ec6} {Suffrage:Voter ID:server-54067828-cecb-736d-6c82-b8f29fed8403 Address:54067828-cecb-736d-6c82-b8f29fed8403} {Suffrage:Voter ID:server-317dcb4d-7d38-1578-d355-6cfa484b90f5 Address:317dcb4d-7d38-1578-d355-6cfa484b90f5}]" +2025-10-30T18:23:17.550Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:17.550Z [INFO] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: entering follower state: follower="Node at 317dcb4d-7d38-1578-d355-6cfa484b90f5 [Follower]" leader-address= leader-id= +2025-10-30T18:23:17.606Z [WARN] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:17.607Z [INFO] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: entering candidate state: node="Node at 317dcb4d-7d38-1578-d355-6cfa484b90f5 [Candidate]" term=2 +2025-10-30T18:23:17.607Z [INFO] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: election won: term=2 tally=2 +2025-10-30T18:23:17.607Z [INFO] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: entering leader state: leader="Node at 317dcb4d-7d38-1578-d355-6cfa484b90f5 [Leader]" +2025-10-30T18:23:17.607Z [INFO] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: added peer, starting replication: peer=server-c2515547-12b2-720b-b1e9-cbe9788e6ec6 +2025-10-30T18:23:17.607Z [INFO] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: added peer, starting replication: peer=server-54067828-cecb-736d-6c82-b8f29fed8403 +2025-10-30T18:23:17.607Z [INFO] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: pipelining replication: peer="{Voter server-54067828-cecb-736d-6c82-b8f29fed8403 54067828-cecb-736d-6c82-b8f29fed8403}" +2025-10-30T18:23:17.607Z [INFO] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: pipelining replication: peer="{Voter server-c2515547-12b2-720b-b1e9-cbe9788e6ec6 c2515547-12b2-720b-b1e9-cbe9788e6ec6}" +2025-10-30T18:23:17.712Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:17.607172046 +0000 UTC m=+120.680701382 (1 nodes), 56.378039ms from start of poll, 56.736001ms from cluster start. Timeout at 2025-10-30 18:23:17.712184322 +0000 UTC m=+120.785713612, 105.01223ms after stability +2025-10-30T18:23:17.718Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:17.823Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:17.718578027 +0000 UTC m=+120.792107326 (2 nodes), 2.523µs from start of poll, 168.141945ms from cluster start. Timeout at 2025-10-30 18:23:17.823656106 +0000 UTC m=+120.897185399, 105.078073ms after stability +2025-10-30T18:23:17.823Z [INFO] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: updating configuration: command=RemoveServer server-id=server-c2515547-12b2-720b-b1e9-cbe9788e6ec6 server-addr= servers="[{Suffrage:Voter ID:server-54067828-cecb-736d-6c82-b8f29fed8403 Address:54067828-cecb-736d-6c82-b8f29fed8403} {Suffrage:Voter ID:server-317dcb4d-7d38-1578-d355-6cfa484b90f5 Address:317dcb4d-7d38-1578-d355-6cfa484b90f5}]" +2025-10-30T18:23:17.823Z [INFO] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: removed peer, stopping replication: peer=server-c2515547-12b2-720b-b1e9-cbe9788e6ec6 last-index=3 +2025-10-30T18:23:17.823Z [INFO] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: aborting pipeline replication: peer="{Voter server-c2515547-12b2-720b-b1e9-cbe9788e6ec6 c2515547-12b2-720b-b1e9-cbe9788e6ec6}" +2025-10-30T18:23:17.882Z [WARN] server-c2515547-12b2-720b-b1e9-cbe9788e6ec6: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election +2025-10-30T18:23:17.931Z [ERROR] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: failed to pipeline appendEntries: peer="{Voter server-54067828-cecb-736d-6c82-b8f29fed8403 54067828-cecb-736d-6c82-b8f29fed8403}" error="append pipeline closed" +2025-10-30T18:23:17.931Z [INFO] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: aborting pipeline replication: peer="{Voter server-54067828-cecb-736d-6c82-b8f29fed8403 54067828-cecb-736d-6c82-b8f29fed8403}" +2025-10-30T18:23:17.934Z [ERROR] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: failed to heartbeat to: peer=54067828-cecb-736d-6c82-b8f29fed8403 backoff time=10ms error="failed to connect to peer: 54067828-cecb-736d-6c82-b8f29fed8403" +2025-10-30T18:23:17.938Z [ERROR] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: failed to appendEntries to: peer="{Voter server-54067828-cecb-736d-6c82-b8f29fed8403 54067828-cecb-736d-6c82-b8f29fed8403}" error="failed to connect to peer: 54067828-cecb-736d-6c82-b8f29fed8403" +2025-10-30T18:23:17.951Z [ERROR] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: failed to heartbeat to: peer=54067828-cecb-736d-6c82-b8f29fed8403 backoff time=10ms error="failed to connect to peer: 54067828-cecb-736d-6c82-b8f29fed8403" +2025-10-30T18:23:17.954Z [ERROR] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: failed to appendEntries to: peer="{Voter server-54067828-cecb-736d-6c82-b8f29fed8403 54067828-cecb-736d-6c82-b8f29fed8403}" error="failed to connect to peer: 54067828-cecb-736d-6c82-b8f29fed8403" +2025-10-30T18:23:17.969Z [ERROR] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: failed to heartbeat to: peer=54067828-cecb-736d-6c82-b8f29fed8403 backoff time=10ms error="failed to connect to peer: 54067828-cecb-736d-6c82-b8f29fed8403" +2025-10-30T18:23:17.974Z [ERROR] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: failed to appendEntries to: peer="{Voter server-54067828-cecb-736d-6c82-b8f29fed8403 54067828-cecb-736d-6c82-b8f29fed8403}" error="failed to connect to peer: 54067828-cecb-736d-6c82-b8f29fed8403" +2025-10-30T18:23:17.976Z [WARN] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: failed to contact: server-id=server-54067828-cecb-736d-6c82-b8f29fed8403 time=50.188845ms +2025-10-30T18:23:17.976Z [WARN] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: failed to contact quorum of nodes, stepping down +2025-10-30T18:23:17.976Z [INFO] server-317dcb4d-7d38-1578-d355-6cfa484b90f5: entering follower state: follower="Node at 317dcb4d-7d38-1578-d355-6cfa484b90f5 [Follower]" leader-address= leader-id= +2025-10-30T18:23:17.998Z [WARN] server-54067828-cecb-736d-6c82-b8f29fed8403: heartbeat timeout reached, starting election: last-leader-addr=317dcb4d-7d38-1578-d355-6cfa484b90f5 last-leader-id=server-317dcb4d-7d38-1578-d355-6cfa484b90f5 +2025-10-30T18:23:17.998Z [INFO] server-54067828-cecb-736d-6c82-b8f29fed8403: entering candidate state: node="Node at 54067828-cecb-736d-6c82-b8f29fed8403 [Candidate]" term=3 +2025-10-30T18:23:17.998Z [ERROR] server-54067828-cecb-736d-6c82-b8f29fed8403: failed to make requestVote RPC: target="{Voter server-317dcb4d-7d38-1578-d355-6cfa484b90f5 317dcb4d-7d38-1578-d355-6cfa484b90f5}" error="failed to connect to peer: 317dcb4d-7d38-1578-d355-6cfa484b90f5" term=3 +2025-10-30T18:23:18.035Z [WARN] server-54067828-cecb-736d-6c82-b8f29fed8403: rejecting vote request since node is not in configuration: from=c2515547-12b2-720b-b1e9-cbe9788e6ec6 +--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.52s) === RUN TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin -2025-10-30T18:19:23.051Z [INFO] server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e Address:671fbac5-1c1a-0e00-bcbc-a14f0658cd6e} {Suffrage:Voter ID:server-c88d47c5-be75-34fa-2df0-109d3b6ae46a Address:c88d47c5-be75-34fa-2df0-109d3b6ae46a} {Suffrage:Voter ID:server-efa5b2fa-54b3-cb32-18da-8843719903cd Address:efa5b2fa-54b3-cb32-18da-8843719903cd}]" -2025-10-30T18:19:23.051Z [INFO] server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: entering follower state: follower="Node at 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e [Follower]" leader-address= leader-id= -2025-10-30T18:19:23.051Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e Address:671fbac5-1c1a-0e00-bcbc-a14f0658cd6e} {Suffrage:Voter ID:server-c88d47c5-be75-34fa-2df0-109d3b6ae46a Address:c88d47c5-be75-34fa-2df0-109d3b6ae46a} {Suffrage:Voter ID:server-efa5b2fa-54b3-cb32-18da-8843719903cd Address:efa5b2fa-54b3-cb32-18da-8843719903cd}]" -2025-10-30T18:19:23.051Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: entering follower state: follower="Node at c88d47c5-be75-34fa-2df0-109d3b6ae46a [Follower]" leader-address= leader-id= -2025-10-30T18:19:23.051Z [INFO] server-efa5b2fa-54b3-cb32-18da-8843719903cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e Address:671fbac5-1c1a-0e00-bcbc-a14f0658cd6e} {Suffrage:Voter ID:server-c88d47c5-be75-34fa-2df0-109d3b6ae46a Address:c88d47c5-be75-34fa-2df0-109d3b6ae46a} {Suffrage:Voter ID:server-efa5b2fa-54b3-cb32-18da-8843719903cd Address:efa5b2fa-54b3-cb32-18da-8843719903cd}]" -2025-10-30T18:19:23.051Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:23.051Z [INFO] server-efa5b2fa-54b3-cb32-18da-8843719903cd: entering follower state: follower="Node at efa5b2fa-54b3-cb32-18da-8843719903cd [Follower]" leader-address= leader-id= -2025-10-30T18:19:23.114Z [WARN] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:23.114Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: entering candidate state: node="Node at c88d47c5-be75-34fa-2df0-109d3b6ae46a [Candidate]" term=2 -2025-10-30T18:19:23.114Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: election won: term=2 tally=2 -2025-10-30T18:19:23.114Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: entering leader state: leader="Node at c88d47c5-be75-34fa-2df0-109d3b6ae46a [Leader]" -2025-10-30T18:19:23.114Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: added peer, starting replication: peer=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e -2025-10-30T18:19:23.114Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: added peer, starting replication: peer=server-efa5b2fa-54b3-cb32-18da-8843719903cd -2025-10-30T18:19:23.114Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: pipelining replication: peer="{Voter server-efa5b2fa-54b3-cb32-18da-8843719903cd efa5b2fa-54b3-cb32-18da-8843719903cd}" -2025-10-30T18:19:23.114Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: pipelining replication: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" -2025-10-30T18:19:23.219Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:23.114230494 +0000 UTC m=+109.392904391 (1 nodes), 62.722226ms from start of poll, 62.936244ms from cluster start. Timeout at 2025-10-30 18:19:23.219232755 +0000 UTC m=+109.497906652, 105.002261ms after stability -2025-10-30T18:19:23.225Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:23.331Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:23.225239878 +0000 UTC m=+109.503913775 (2 nodes), 1.549µs from start of poll, 173.945628ms from cluster start. Timeout at 2025-10-30 18:19:23.33025828 +0000 UTC m=+109.608932188, 105.018413ms after stability -2025-10-30T18:19:23.333Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to heartbeat to: peer=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e backoff time=10ms error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e" -2025-10-30T18:19:23.338Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to pipeline appendEntries: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" error="append pipeline closed" -2025-10-30T18:19:23.338Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: aborting pipeline replication: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" -2025-10-30T18:19:23.346Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to appendEntries to: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e" -2025-10-30T18:19:23.352Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to heartbeat to: peer=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e backoff time=10ms error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e" -2025-10-30T18:19:23.363Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to appendEntries to: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e" -2025-10-30T18:19:23.371Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to heartbeat to: peer=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e backoff time=10ms error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e" -2025-10-30T18:19:23.379Z [WARN] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to contact: server-id=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e time=50.621426ms -2025-10-30T18:19:23.380Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to appendEntries to: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e" -2025-10-30T18:19:23.389Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to heartbeat to: peer=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e backoff time=20ms error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e" -2025-10-30T18:19:23.402Z [WARN] server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: heartbeat timeout reached, starting election: last-leader-addr=c88d47c5-be75-34fa-2df0-109d3b6ae46a last-leader-id=server-c88d47c5-be75-34fa-2df0-109d3b6ae46a -2025-10-30T18:19:23.402Z [INFO] server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: entering candidate state: node="Node at 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e [Candidate]" term=3 -2025-10-30T18:19:23.402Z [ERROR] server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: failed to make requestVote RPC: target="{Voter server-efa5b2fa-54b3-cb32-18da-8843719903cd efa5b2fa-54b3-cb32-18da-8843719903cd}" error="failed to connect to peer: efa5b2fa-54b3-cb32-18da-8843719903cd" term=3 -2025-10-30T18:19:23.402Z [ERROR] server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: failed to make requestVote RPC: target="{Voter server-c88d47c5-be75-34fa-2df0-109d3b6ae46a c88d47c5-be75-34fa-2df0-109d3b6ae46a}" error="failed to connect to peer: c88d47c5-be75-34fa-2df0-109d3b6ae46a" term=3 -2025-10-30T18:19:23.409Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to appendEntries to: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e" -2025-10-30T18:19:23.421Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to heartbeat to: peer=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e backoff time=25ms error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e" -2025-10-30T18:19:23.428Z [WARN] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to contact: server-id=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e time=99.200295ms -2025-10-30T18:19:23.436Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: updating configuration: command=RemoveServer server-id=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e server-addr= servers="[{Suffrage:Voter ID:server-c88d47c5-be75-34fa-2df0-109d3b6ae46a Address:c88d47c5-be75-34fa-2df0-109d3b6ae46a} {Suffrage:Voter ID:server-efa5b2fa-54b3-cb32-18da-8843719903cd Address:efa5b2fa-54b3-cb32-18da-8843719903cd}]" -2025-10-30T18:19:23.436Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: removed peer, stopping replication: peer=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e last-index=3 -2025-10-30T18:19:23.436Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: updating configuration: command=AddNonvoter server-id=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e server-addr=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e servers="[{Suffrage:Voter ID:server-c88d47c5-be75-34fa-2df0-109d3b6ae46a Address:c88d47c5-be75-34fa-2df0-109d3b6ae46a} {Suffrage:Voter ID:server-efa5b2fa-54b3-cb32-18da-8843719903cd Address:efa5b2fa-54b3-cb32-18da-8843719903cd} {Suffrage:Nonvoter ID:server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e Address:671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}]" -2025-10-30T18:19:23.436Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: added peer, starting replication: peer=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e -2025-10-30T18:19:23.436Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: failed to appendEntries to: peer="{Nonvoter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" error="failed to connect to peer: 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e" -2025-10-30T18:19:23.453Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: peer has newer term, stopping replication: peer="{Nonvoter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" -2025-10-30T18:19:23.453Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: entering follower state: follower="Node at c88d47c5-be75-34fa-2df0-109d3b6ae46a [Follower]" leader-address= leader-id= -2025-10-30T18:19:23.453Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: aborting pipeline replication: peer="{Voter server-efa5b2fa-54b3-cb32-18da-8843719903cd efa5b2fa-54b3-cb32-18da-8843719903cd}" -2025-10-30T18:19:23.455Z [ERROR] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: peer has newer term, stopping replication: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" -2025-10-30T18:19:23.461Z [WARN] server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: Election timeout reached, restarting election -2025-10-30T18:19:23.461Z [INFO] server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: entering candidate state: node="Node at 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e [Candidate]" term=23 -2025-10-30T18:19:23.461Z [WARN] server-efa5b2fa-54b3-cb32-18da-8843719903cd: rejecting vote request since we have a leader: from=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e leader=c88d47c5-be75-34fa-2df0-109d3b6ae46a leader-id=server-c88d47c5-be75-34fa-2df0-109d3b6ae46a -2025-10-30T18:19:23.461Z [WARN] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: rejecting vote request since node is not a voter: from=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e -2025-10-30T18:19:23.523Z [WARN] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:23.523Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: entering candidate state: node="Node at c88d47c5-be75-34fa-2df0-109d3b6ae46a [Candidate]" term=24 -2025-10-30T18:19:23.523Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: election won: term=24 tally=2 -2025-10-30T18:19:23.523Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: entering leader state: leader="Node at c88d47c5-be75-34fa-2df0-109d3b6ae46a [Leader]" -2025-10-30T18:19:23.523Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: added peer, starting replication: peer=server-efa5b2fa-54b3-cb32-18da-8843719903cd -2025-10-30T18:19:23.523Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: added peer, starting replication: peer=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e -2025-10-30T18:19:23.523Z [WARN] server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: failed to get previous log: previous-index=4 last-index=2 error="log not found" -2025-10-30T18:19:23.523Z [INFO] server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: entering follower state: follower="Node at 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e [Follower]" leader-address=c88d47c5-be75-34fa-2df0-109d3b6ae46a leader-id=server-c88d47c5-be75-34fa-2df0-109d3b6ae46a -2025-10-30T18:19:23.523Z [WARN] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: appendEntries rejected, sending older logs: peer="{Nonvoter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" next=3 -2025-10-30T18:19:23.524Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: pipelining replication: peer="{Nonvoter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" -2025-10-30T18:19:23.524Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: pipelining replication: peer="{Voter server-efa5b2fa-54b3-cb32-18da-8843719903cd efa5b2fa-54b3-cb32-18da-8843719903cd}" -2025-10-30T18:19:23.542Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:23.647Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:23.542460799 +0000 UTC m=+109.821134696 (1 nodes), 22.114µs from start of poll, 491.166549ms from cluster start. Timeout at 2025-10-30 18:19:23.647480981 +0000 UTC m=+109.926154878, 105.020182ms after stability -2025-10-30T18:19:23.666Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: updating configuration: command=RemoveServer server-id=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e server-addr= servers="[{Suffrage:Voter ID:server-c88d47c5-be75-34fa-2df0-109d3b6ae46a Address:c88d47c5-be75-34fa-2df0-109d3b6ae46a} {Suffrage:Voter ID:server-efa5b2fa-54b3-cb32-18da-8843719903cd Address:efa5b2fa-54b3-cb32-18da-8843719903cd}]" -2025-10-30T18:19:23.666Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: removed peer, stopping replication: peer=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e last-index=106 -2025-10-30T18:19:23.666Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: aborting pipeline replication: peer="{Nonvoter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" -2025-10-30T18:19:23.666Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: updating configuration: command=AddVoter server-id=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e server-addr=671fbac5-1c1a-0e00-bcbc-a14f0658cd6e servers="[{Suffrage:Voter ID:server-c88d47c5-be75-34fa-2df0-109d3b6ae46a Address:c88d47c5-be75-34fa-2df0-109d3b6ae46a} {Suffrage:Voter ID:server-efa5b2fa-54b3-cb32-18da-8843719903cd Address:efa5b2fa-54b3-cb32-18da-8843719903cd} {Suffrage:Voter ID:server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e Address:671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}]" -2025-10-30T18:19:23.666Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: added peer, starting replication: peer=server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e -2025-10-30T18:19:23.666Z [WARN] server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e: failed to get previous log: previous-index=107 last-index=106 error="log not found" -2025-10-30T18:19:23.667Z [WARN] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: appendEntries rejected, sending older logs: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" next=107 -2025-10-30T18:19:23.667Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: pipelining replication: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" -2025-10-30T18:19:24.733Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:24.839Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:24.733793999 +0000 UTC m=+111.012467899 (1 nodes), 526ns from start of poll, 1.682499752s from cluster start. Timeout at 2025-10-30 18:19:24.838815442 +0000 UTC m=+111.117489339, 105.02144ms after stability -2025-10-30T18:19:24.839Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: aborting pipeline replication: peer="{Voter server-671fbac5-1c1a-0e00-bcbc-a14f0658cd6e 671fbac5-1c1a-0e00-bcbc-a14f0658cd6e}" -2025-10-30T18:19:24.839Z [INFO] server-c88d47c5-be75-34fa-2df0-109d3b6ae46a: aborting pipeline replication: peer="{Voter server-efa5b2fa-54b3-cb32-18da-8843719903cd efa5b2fa-54b3-cb32-18da-8843719903cd}" +2025-10-30T18:23:18.069Z [INFO] server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 Address:14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4} {Suffrage:Voter ID:server-fd3033b8-2363-c142-9542-b6d3f1edeef0 Address:fd3033b8-2363-c142-9542-b6d3f1edeef0} {Suffrage:Voter ID:server-86bd4e22-3743-9cca-ec5b-5cc68ceafefd Address:86bd4e22-3743-9cca-ec5b-5cc68ceafefd}]" +2025-10-30T18:23:18.070Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 Address:14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4} {Suffrage:Voter ID:server-fd3033b8-2363-c142-9542-b6d3f1edeef0 Address:fd3033b8-2363-c142-9542-b6d3f1edeef0} {Suffrage:Voter ID:server-86bd4e22-3743-9cca-ec5b-5cc68ceafefd Address:86bd4e22-3743-9cca-ec5b-5cc68ceafefd}]" +2025-10-30T18:23:18.070Z [INFO] server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4: entering follower state: follower="Node at 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 [Follower]" leader-address= leader-id= +2025-10-30T18:23:18.070Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: entering follower state: follower="Node at fd3033b8-2363-c142-9542-b6d3f1edeef0 [Follower]" leader-address= leader-id= +2025-10-30T18:23:18.070Z [INFO] server-86bd4e22-3743-9cca-ec5b-5cc68ceafefd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 Address:14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4} {Suffrage:Voter ID:server-fd3033b8-2363-c142-9542-b6d3f1edeef0 Address:fd3033b8-2363-c142-9542-b6d3f1edeef0} {Suffrage:Voter ID:server-86bd4e22-3743-9cca-ec5b-5cc68ceafefd Address:86bd4e22-3743-9cca-ec5b-5cc68ceafefd}]" +2025-10-30T18:23:18.070Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:18.070Z [INFO] server-86bd4e22-3743-9cca-ec5b-5cc68ceafefd: entering follower state: follower="Node at 86bd4e22-3743-9cca-ec5b-5cc68ceafefd [Follower]" leader-address= leader-id= +2025-10-30T18:23:18.131Z [WARN] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:18.131Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: entering candidate state: node="Node at fd3033b8-2363-c142-9542-b6d3f1edeef0 [Candidate]" term=2 +2025-10-30T18:23:18.131Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: election won: term=2 tally=2 +2025-10-30T18:23:18.131Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: entering leader state: leader="Node at fd3033b8-2363-c142-9542-b6d3f1edeef0 [Leader]" +2025-10-30T18:23:18.131Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: added peer, starting replication: peer=server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 +2025-10-30T18:23:18.131Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: added peer, starting replication: peer=server-86bd4e22-3743-9cca-ec5b-5cc68ceafefd +2025-10-30T18:23:18.132Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: pipelining replication: peer="{Voter server-86bd4e22-3743-9cca-ec5b-5cc68ceafefd 86bd4e22-3743-9cca-ec5b-5cc68ceafefd}" +2025-10-30T18:23:18.132Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: pipelining replication: peer="{Voter server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}" +2025-10-30T18:23:18.238Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:18.132043468 +0000 UTC m=+121.205572773 (1 nodes), 61.919534ms from start of poll, 62.17629ms from cluster start. Timeout at 2025-10-30 18:23:18.237048714 +0000 UTC m=+121.310578040, 105.005267ms after stability +2025-10-30T18:23:18.243Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:18.349Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:18.243927787 +0000 UTC m=+121.317457077 (2 nodes), 1.608µs from start of poll, 174.060594ms from cluster start. Timeout at 2025-10-30 18:23:18.348949088 +0000 UTC m=+121.422478433, 105.021356ms after stability +2025-10-30T18:23:18.355Z [ERROR] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: failed to pipeline appendEntries: peer="{Voter server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}" error="append pipeline closed" +2025-10-30T18:23:18.355Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: aborting pipeline replication: peer="{Voter server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}" +2025-10-30T18:23:18.359Z [ERROR] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: failed to heartbeat to: peer=14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 backoff time=10ms error="failed to connect to peer: 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4" +2025-10-30T18:23:18.361Z [ERROR] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: failed to appendEntries to: peer="{Voter server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}" error="failed to connect to peer: 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4" +2025-10-30T18:23:18.378Z [ERROR] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: failed to heartbeat to: peer=14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 backoff time=10ms error="failed to connect to peer: 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4" +2025-10-30T18:23:18.381Z [ERROR] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: failed to appendEntries to: peer="{Voter server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}" error="failed to connect to peer: 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4" +2025-10-30T18:23:18.394Z [ERROR] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: failed to heartbeat to: peer=14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 backoff time=10ms error="failed to connect to peer: 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4" +2025-10-30T18:23:18.399Z [ERROR] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: failed to appendEntries to: peer="{Voter server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}" error="failed to connect to peer: 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4" +2025-10-30T18:23:18.400Z [WARN] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: failed to contact: server-id=server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 time=50.802074ms +2025-10-30T18:23:18.413Z [ERROR] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: failed to heartbeat to: peer=14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 backoff time=20ms error="failed to connect to peer: 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4" +2025-10-30T18:23:18.427Z [ERROR] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: failed to appendEntries to: peer="{Voter server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}" error="failed to connect to peer: 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4" +2025-10-30T18:23:18.439Z [WARN] server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4: heartbeat timeout reached, starting election: last-leader-addr=fd3033b8-2363-c142-9542-b6d3f1edeef0 last-leader-id=server-fd3033b8-2363-c142-9542-b6d3f1edeef0 +2025-10-30T18:23:18.439Z [INFO] server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4: entering candidate state: node="Node at 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 [Candidate]" term=3 +2025-10-30T18:23:18.439Z [ERROR] server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4: failed to make requestVote RPC: target="{Voter server-86bd4e22-3743-9cca-ec5b-5cc68ceafefd 86bd4e22-3743-9cca-ec5b-5cc68ceafefd}" error="failed to connect to peer: 86bd4e22-3743-9cca-ec5b-5cc68ceafefd" term=3 +2025-10-30T18:23:18.439Z [ERROR] server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4: failed to make requestVote RPC: target="{Voter server-fd3033b8-2363-c142-9542-b6d3f1edeef0 fd3033b8-2363-c142-9542-b6d3f1edeef0}" error="failed to connect to peer: fd3033b8-2363-c142-9542-b6d3f1edeef0" term=3 +2025-10-30T18:23:18.442Z [ERROR] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: failed to heartbeat to: peer=14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 backoff time=25ms error="failed to connect to peer: 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4" +2025-10-30T18:23:18.449Z [WARN] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: failed to contact: server-id=server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 time=99.382031ms +2025-10-30T18:23:18.455Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: updating configuration: command=RemoveServer server-id=server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 server-addr= servers="[{Suffrage:Voter ID:server-fd3033b8-2363-c142-9542-b6d3f1edeef0 Address:fd3033b8-2363-c142-9542-b6d3f1edeef0} {Suffrage:Voter ID:server-86bd4e22-3743-9cca-ec5b-5cc68ceafefd Address:86bd4e22-3743-9cca-ec5b-5cc68ceafefd}]" +2025-10-30T18:23:18.455Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: removed peer, stopping replication: peer=server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 last-index=3 +2025-10-30T18:23:18.455Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: updating configuration: command=AddNonvoter server-id=server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 server-addr=14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 servers="[{Suffrage:Voter ID:server-fd3033b8-2363-c142-9542-b6d3f1edeef0 Address:fd3033b8-2363-c142-9542-b6d3f1edeef0} {Suffrage:Voter ID:server-86bd4e22-3743-9cca-ec5b-5cc68ceafefd Address:86bd4e22-3743-9cca-ec5b-5cc68ceafefd} {Suffrage:Nonvoter ID:server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 Address:14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}]" +2025-10-30T18:23:18.455Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: added peer, starting replication: peer=server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 +2025-10-30T18:23:18.455Z [ERROR] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: failed to appendEntries to: peer="{Nonvoter server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}" error="failed to connect to peer: 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4" +2025-10-30T18:23:18.475Z [ERROR] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: peer has newer term, stopping replication: peer="{Nonvoter server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}" +2025-10-30T18:23:18.475Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: entering follower state: follower="Node at fd3033b8-2363-c142-9542-b6d3f1edeef0 [Follower]" leader-address= leader-id= +2025-10-30T18:23:18.475Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: aborting pipeline replication: peer="{Voter server-86bd4e22-3743-9cca-ec5b-5cc68ceafefd 86bd4e22-3743-9cca-ec5b-5cc68ceafefd}" +2025-10-30T18:23:18.477Z [ERROR] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: peer has newer term, stopping replication: peer="{Voter server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}" +2025-10-30T18:23:18.518Z [WARN] server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4: Election timeout reached, restarting election +2025-10-30T18:23:18.518Z [INFO] server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4: entering candidate state: node="Node at 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 [Candidate]" term=23 +2025-10-30T18:23:18.518Z [WARN] server-86bd4e22-3743-9cca-ec5b-5cc68ceafefd: rejecting vote request since we have a leader: from=14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 leader=fd3033b8-2363-c142-9542-b6d3f1edeef0 leader-id=server-fd3033b8-2363-c142-9542-b6d3f1edeef0 +2025-10-30T18:23:18.518Z [WARN] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: rejecting vote request since node is not a voter: from=14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 +2025-10-30T18:23:18.543Z [WARN] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:18.543Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: entering candidate state: node="Node at fd3033b8-2363-c142-9542-b6d3f1edeef0 [Candidate]" term=24 +2025-10-30T18:23:18.543Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: election won: term=24 tally=2 +2025-10-30T18:23:18.543Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: entering leader state: leader="Node at fd3033b8-2363-c142-9542-b6d3f1edeef0 [Leader]" +2025-10-30T18:23:18.543Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: added peer, starting replication: peer=server-86bd4e22-3743-9cca-ec5b-5cc68ceafefd +2025-10-30T18:23:18.543Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: added peer, starting replication: peer=server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 +2025-10-30T18:23:18.543Z [WARN] server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4: failed to get previous log: previous-index=4 last-index=2 error="log not found" +2025-10-30T18:23:18.543Z [INFO] server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4: entering follower state: follower="Node at 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 [Follower]" leader-address=fd3033b8-2363-c142-9542-b6d3f1edeef0 leader-id=server-fd3033b8-2363-c142-9542-b6d3f1edeef0 +2025-10-30T18:23:18.543Z [WARN] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: appendEntries rejected, sending older logs: peer="{Nonvoter server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}" next=3 +2025-10-30T18:23:18.543Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: pipelining replication: peer="{Voter server-86bd4e22-3743-9cca-ec5b-5cc68ceafefd 86bd4e22-3743-9cca-ec5b-5cc68ceafefd}" +2025-10-30T18:23:18.543Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: pipelining replication: peer="{Nonvoter server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}" +2025-10-30T18:23:18.561Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:18.667Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:18.561738391 +0000 UTC m=+121.635267741 (1 nodes), 91.585µs from start of poll, 491.871258ms from cluster start. Timeout at 2025-10-30 18:23:18.666757302 +0000 UTC m=+121.740286594, 105.018853ms after stability +2025-10-30T18:23:18.685Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: updating configuration: command=RemoveServer server-id=server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 server-addr= servers="[{Suffrage:Voter ID:server-fd3033b8-2363-c142-9542-b6d3f1edeef0 Address:fd3033b8-2363-c142-9542-b6d3f1edeef0} {Suffrage:Voter ID:server-86bd4e22-3743-9cca-ec5b-5cc68ceafefd Address:86bd4e22-3743-9cca-ec5b-5cc68ceafefd}]" +2025-10-30T18:23:18.685Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: removed peer, stopping replication: peer=server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 last-index=106 +2025-10-30T18:23:18.685Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: aborting pipeline replication: peer="{Nonvoter server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}" +2025-10-30T18:23:18.685Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: updating configuration: command=AddVoter server-id=server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 server-addr=14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 servers="[{Suffrage:Voter ID:server-fd3033b8-2363-c142-9542-b6d3f1edeef0 Address:fd3033b8-2363-c142-9542-b6d3f1edeef0} {Suffrage:Voter ID:server-86bd4e22-3743-9cca-ec5b-5cc68ceafefd Address:86bd4e22-3743-9cca-ec5b-5cc68ceafefd} {Suffrage:Voter ID:server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 Address:14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}]" +2025-10-30T18:23:18.685Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: added peer, starting replication: peer=server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 +2025-10-30T18:23:18.685Z [WARN] server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4: failed to get previous log: previous-index=107 last-index=106 error="log not found" +2025-10-30T18:23:18.685Z [WARN] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: appendEntries rejected, sending older logs: peer="{Voter server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}" next=107 +2025-10-30T18:23:18.685Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: pipelining replication: peer="{Voter server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}" +2025-10-30T18:23:19.755Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:19.861Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:19.755870779 +0000 UTC m=+122.829400069 (1 nodes), 978ns from start of poll, 1.686003586s from cluster start. Timeout at 2025-10-30 18:23:19.860892108 +0000 UTC m=+122.934421401, 105.021332ms after stability +2025-10-30T18:23:19.861Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: aborting pipeline replication: peer="{Voter server-14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4 14fd7c3d-f3c2-adc2-51ea-2b2a45612fe4}" +2025-10-30T18:23:19.861Z [INFO] server-fd3033b8-2363-c142-9542-b6d3f1edeef0: aborting pipeline replication: peer="{Voter server-86bd4e22-3743-9cca-ec5b-5cc68ceafefd 86bd4e22-3743-9cca-ec5b-5cc68ceafefd}" --- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.79s) === RUN TestRaft_FollowerRemovalNoElection -2025-10-30T18:19:24.840Z [INFO] server-ff614bb3-8186-fbc0-2495-2700e40934be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff614bb3-8186-fbc0-2495-2700e40934be Address:ff614bb3-8186-fbc0-2495-2700e40934be} {Suffrage:Voter ID:server-5d90c322-23c1-7b8b-1254-88cc33c90377 Address:5d90c322-23c1-7b8b-1254-88cc33c90377} {Suffrage:Voter ID:server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77 Address:f1da6b9a-1ee2-d8f4-1294-e9193efb3a77}]" -2025-10-30T18:19:24.840Z [INFO] server-ff614bb3-8186-fbc0-2495-2700e40934be: entering follower state: follower="Node at ff614bb3-8186-fbc0-2495-2700e40934be [Follower]" leader-address= leader-id= -2025-10-30T18:19:24.840Z [INFO] server-5d90c322-23c1-7b8b-1254-88cc33c90377: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff614bb3-8186-fbc0-2495-2700e40934be Address:ff614bb3-8186-fbc0-2495-2700e40934be} {Suffrage:Voter ID:server-5d90c322-23c1-7b8b-1254-88cc33c90377 Address:5d90c322-23c1-7b8b-1254-88cc33c90377} {Suffrage:Voter ID:server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77 Address:f1da6b9a-1ee2-d8f4-1294-e9193efb3a77}]" -2025-10-30T18:19:24.840Z [INFO] server-5d90c322-23c1-7b8b-1254-88cc33c90377: entering follower state: follower="Node at 5d90c322-23c1-7b8b-1254-88cc33c90377 [Follower]" leader-address= leader-id= -2025-10-30T18:19:24.840Z [INFO] server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff614bb3-8186-fbc0-2495-2700e40934be Address:ff614bb3-8186-fbc0-2495-2700e40934be} {Suffrage:Voter ID:server-5d90c322-23c1-7b8b-1254-88cc33c90377 Address:5d90c322-23c1-7b8b-1254-88cc33c90377} {Suffrage:Voter ID:server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77 Address:f1da6b9a-1ee2-d8f4-1294-e9193efb3a77}]" -2025-10-30T18:19:24.840Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:24.840Z [INFO] server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: entering follower state: follower="Node at f1da6b9a-1ee2-d8f4-1294-e9193efb3a77 [Follower]" leader-address= leader-id= -2025-10-30T18:19:24.960Z [WARN] server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:24.960Z [INFO] server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: entering candidate state: node="Node at f1da6b9a-1ee2-d8f4-1294-e9193efb3a77 [Candidate]" term=2 -2025-10-30T18:19:24.960Z [INFO] server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: election won: term=2 tally=2 -2025-10-30T18:19:24.960Z [INFO] server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: entering leader state: leader="Node at f1da6b9a-1ee2-d8f4-1294-e9193efb3a77 [Leader]" -2025-10-30T18:19:24.960Z [INFO] server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: added peer, starting replication: peer=server-ff614bb3-8186-fbc0-2495-2700e40934be -2025-10-30T18:19:24.960Z [INFO] server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: added peer, starting replication: peer=server-5d90c322-23c1-7b8b-1254-88cc33c90377 -2025-10-30T18:19:24.960Z [INFO] server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: pipelining replication: peer="{Voter server-5d90c322-23c1-7b8b-1254-88cc33c90377 5d90c322-23c1-7b8b-1254-88cc33c90377}" -2025-10-30T18:19:24.960Z [INFO] server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: pipelining replication: peer="{Voter server-ff614bb3-8186-fbc0-2495-2700e40934be ff614bb3-8186-fbc0-2495-2700e40934be}" -2025-10-30T18:19:25.166Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:24.960544116 +0000 UTC m=+111.239218023 (1 nodes), 120.095601ms from start of poll, 120.399813ms from cluster start. Timeout at 2025-10-30 18:19:25.165548143 +0000 UTC m=+111.444222039, 205.004016ms after stability -2025-10-30T18:19:25.166Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:25.371Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:25.166381543 +0000 UTC m=+111.445055454 (1 nodes), 1.429µs from start of poll, 326.237244ms from cluster start. Timeout at 2025-10-30 18:19:25.371398265 +0000 UTC m=+111.650072195, 205.016741ms after stability -2025-10-30T18:19:25.377Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:25.583Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:25.37754477 +0000 UTC m=+111.656218672 (2 nodes), 1.425µs from start of poll, 537.400462ms from cluster start. Timeout at 2025-10-30 18:19:25.582566071 +0000 UTC m=+111.861240009, 205.021337ms after stability -2025-10-30T18:19:25.583Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:25.789Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:25.58351426 +0000 UTC m=+111.862188158 (1 nodes), 1.327µs from start of poll, 743.369948ms from cluster start. Timeout at 2025-10-30 18:19:25.788536164 +0000 UTC m=+112.067210085, 205.021927ms after stability - raft_test.go:2961: [INFO] restarting Node at ff614bb3-8186-fbc0-2495-2700e40934be [Follower] -2025-10-30T18:19:25.789Z [INFO] server-ff614bb3-8186-fbc0-2495-2700e40934be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff614bb3-8186-fbc0-2495-2700e40934be Address:ff614bb3-8186-fbc0-2495-2700e40934be} {Suffrage:Voter ID:server-5d90c322-23c1-7b8b-1254-88cc33c90377 Address:5d90c322-23c1-7b8b-1254-88cc33c90377} {Suffrage:Voter ID:server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77 Address:f1da6b9a-1ee2-d8f4-1294-e9193efb3a77}]" -2025-10-30T18:19:25.789Z [INFO] server-ff614bb3-8186-fbc0-2495-2700e40934be: entering follower state: follower="Node at ff614bb3-8186-fbc0-2495-2700e40934be [Follower]" leader-address= leader-id= -2025-10-30T18:19:26.040Z [INFO] server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: aborting pipeline replication: peer="{Voter server-5d90c322-23c1-7b8b-1254-88cc33c90377 5d90c322-23c1-7b8b-1254-88cc33c90377}" -2025-10-30T18:19:26.424Z [ERROR] server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: failed to pipeline appendEntries: peer="{Voter server-ff614bb3-8186-fbc0-2495-2700e40934be ff614bb3-8186-fbc0-2495-2700e40934be}" error="command enqueue timeout" -2025-10-30T18:19:26.424Z [INFO] server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: aborting pipeline replication: peer="{Voter server-ff614bb3-8186-fbc0-2495-2700e40934be ff614bb3-8186-fbc0-2495-2700e40934be}" -2025-10-30T18:19:26.553Z [ERROR] server-f1da6b9a-1ee2-d8f4-1294-e9193efb3a77: failed to heartbeat to: peer=ff614bb3-8186-fbc0-2495-2700e40934be backoff time=10ms error="command timed out" ---- PASS: TestRaft_FollowerRemovalNoElection (1.71s) +2025-10-30T18:23:19.862Z [INFO] server-de39c77e-61f2-d711-ceab-7aa22e5b4f20: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de39c77e-61f2-d711-ceab-7aa22e5b4f20 Address:de39c77e-61f2-d711-ceab-7aa22e5b4f20} {Suffrage:Voter ID:server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e Address:eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e} {Suffrage:Voter ID:server-151c2e7d-1d35-eb6d-e3b0-10d4c008bb03 Address:151c2e7d-1d35-eb6d-e3b0-10d4c008bb03}]" +2025-10-30T18:23:19.862Z [INFO] server-de39c77e-61f2-d711-ceab-7aa22e5b4f20: entering follower state: follower="Node at de39c77e-61f2-d711-ceab-7aa22e5b4f20 [Follower]" leader-address= leader-id= +2025-10-30T18:23:19.862Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de39c77e-61f2-d711-ceab-7aa22e5b4f20 Address:de39c77e-61f2-d711-ceab-7aa22e5b4f20} {Suffrage:Voter ID:server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e Address:eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e} {Suffrage:Voter ID:server-151c2e7d-1d35-eb6d-e3b0-10d4c008bb03 Address:151c2e7d-1d35-eb6d-e3b0-10d4c008bb03}]" +2025-10-30T18:23:19.862Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: entering follower state: follower="Node at eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e [Follower]" leader-address= leader-id= +2025-10-30T18:23:19.863Z [INFO] server-151c2e7d-1d35-eb6d-e3b0-10d4c008bb03: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de39c77e-61f2-d711-ceab-7aa22e5b4f20 Address:de39c77e-61f2-d711-ceab-7aa22e5b4f20} {Suffrage:Voter ID:server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e Address:eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e} {Suffrage:Voter ID:server-151c2e7d-1d35-eb6d-e3b0-10d4c008bb03 Address:151c2e7d-1d35-eb6d-e3b0-10d4c008bb03}]" +2025-10-30T18:23:19.863Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:19.863Z [INFO] server-151c2e7d-1d35-eb6d-e3b0-10d4c008bb03: entering follower state: follower="Node at 151c2e7d-1d35-eb6d-e3b0-10d4c008bb03 [Follower]" leader-address= leader-id= +2025-10-30T18:23:20.014Z [WARN] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:20.014Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: entering candidate state: node="Node at eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e [Candidate]" term=2 +2025-10-30T18:23:20.014Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: election won: term=2 tally=2 +2025-10-30T18:23:20.014Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: entering leader state: leader="Node at eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e [Leader]" +2025-10-30T18:23:20.014Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: added peer, starting replication: peer=server-de39c77e-61f2-d711-ceab-7aa22e5b4f20 +2025-10-30T18:23:20.014Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: added peer, starting replication: peer=server-151c2e7d-1d35-eb6d-e3b0-10d4c008bb03 +2025-10-30T18:23:20.014Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: pipelining replication: peer="{Voter server-de39c77e-61f2-d711-ceab-7aa22e5b4f20 de39c77e-61f2-d711-ceab-7aa22e5b4f20}" +2025-10-30T18:23:20.014Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: pipelining replication: peer="{Voter server-151c2e7d-1d35-eb6d-e3b0-10d4c008bb03 151c2e7d-1d35-eb6d-e3b0-10d4c008bb03}" +2025-10-30T18:23:20.182Z [WARN] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: failed to contact: server-id=server-de39c77e-61f2-d711-ceab-7aa22e5b4f20 time=61.855405ms +2025-10-30T18:23:20.182Z [WARN] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: failed to contact: server-id=server-151c2e7d-1d35-eb6d-e3b0-10d4c008bb03 time=61.891064ms +2025-10-30T18:23:20.182Z [WARN] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: failed to contact quorum of nodes, stepping down +2025-10-30T18:23:20.182Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: entering follower state: follower="Node at eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e [Follower]" leader-address= leader-id= +2025-10-30T18:23:20.182Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: aborting pipeline replication: peer="{Voter server-de39c77e-61f2-d711-ceab-7aa22e5b4f20 de39c77e-61f2-d711-ceab-7aa22e5b4f20}" +2025-10-30T18:23:20.182Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: aborting pipeline replication: peer="{Voter server-151c2e7d-1d35-eb6d-e3b0-10d4c008bb03 151c2e7d-1d35-eb6d-e3b0-10d4c008bb03}" +2025-10-30T18:23:20.293Z [WARN] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:20.293Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: entering candidate state: node="Node at eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e [Candidate]" term=3 +2025-10-30T18:23:20.293Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: election won: term=3 tally=2 +2025-10-30T18:23:20.293Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: entering leader state: leader="Node at eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e [Leader]" +2025-10-30T18:23:20.293Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: added peer, starting replication: peer=server-de39c77e-61f2-d711-ceab-7aa22e5b4f20 +2025-10-30T18:23:20.293Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: added peer, starting replication: peer=server-151c2e7d-1d35-eb6d-e3b0-10d4c008bb03 +2025-10-30T18:23:20.293Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: pipelining replication: peer="{Voter server-151c2e7d-1d35-eb6d-e3b0-10d4c008bb03 151c2e7d-1d35-eb6d-e3b0-10d4c008bb03}" +2025-10-30T18:23:20.293Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: pipelining replication: peer="{Voter server-de39c77e-61f2-d711-ceab-7aa22e5b4f20 de39c77e-61f2-d711-ceab-7aa22e5b4f20}" +2025-10-30T18:23:20.499Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:20.293675315 +0000 UTC m=+123.367204604 (1 nodes), 430.617707ms from start of poll, 430.962678ms from cluster start. Timeout at 2025-10-30 18:23:20.4986813 +0000 UTC m=+123.572210626, 205.006022ms after stability +2025-10-30T18:23:20.499Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:20.704Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:20.499181646 +0000 UTC m=+123.572710992 (1 nodes), 3.535µs from start of poll, 636.469066ms from cluster start. Timeout at 2025-10-30 18:23:20.704210446 +0000 UTC m=+123.777739738, 205.028746ms after stability +2025-10-30T18:23:20.710Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:20.915Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:20.710370369 +0000 UTC m=+123.783899673 (2 nodes), 3.47µs from start of poll, 847.657747ms from cluster start. Timeout at 2025-10-30 18:23:20.915380779 +0000 UTC m=+123.988910101, 205.010428ms after stability +2025-10-30T18:23:20.915Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:21.121Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:20.915910407 +0000 UTC m=+123.989439714 (1 nodes), 1.557µs from start of poll, 1.053197788s from cluster start. Timeout at 2025-10-30 18:23:21.120939439 +0000 UTC m=+124.194468731, 205.029017ms after stability + raft_test.go:2961: [INFO] restarting Node at de39c77e-61f2-d711-ceab-7aa22e5b4f20 [Follower] +2025-10-30T18:23:21.121Z [INFO] server-de39c77e-61f2-d711-ceab-7aa22e5b4f20: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de39c77e-61f2-d711-ceab-7aa22e5b4f20 Address:de39c77e-61f2-d711-ceab-7aa22e5b4f20} {Suffrage:Voter ID:server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e Address:eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e} {Suffrage:Voter ID:server-151c2e7d-1d35-eb6d-e3b0-10d4c008bb03 Address:151c2e7d-1d35-eb6d-e3b0-10d4c008bb03}]" +2025-10-30T18:23:21.121Z [INFO] server-de39c77e-61f2-d711-ceab-7aa22e5b4f20: entering follower state: follower="Node at de39c77e-61f2-d711-ceab-7aa22e5b4f20 [Follower]" leader-address= leader-id= +2025-10-30T18:23:21.371Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: aborting pipeline replication: peer="{Voter server-151c2e7d-1d35-eb6d-e3b0-10d4c008bb03 151c2e7d-1d35-eb6d-e3b0-10d4c008bb03}" +2025-10-30T18:23:21.762Z [ERROR] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: failed to pipeline appendEntries: peer="{Voter server-de39c77e-61f2-d711-ceab-7aa22e5b4f20 de39c77e-61f2-d711-ceab-7aa22e5b4f20}" error="command enqueue timeout" +2025-10-30T18:23:21.762Z [INFO] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: aborting pipeline replication: peer="{Voter server-de39c77e-61f2-d711-ceab-7aa22e5b4f20 de39c77e-61f2-d711-ceab-7aa22e5b4f20}" +2025-10-30T18:23:21.882Z [ERROR] server-eb3233d1-6e4b-f8c5-c8c1-57dcfdc9dd5e: failed to heartbeat to: peer=de39c77e-61f2-d711-ceab-7aa22e5b4f20 backoff time=10ms error="command timed out" +--- PASS: TestRaft_FollowerRemovalNoElection (2.02s) === RUN TestRaft_VoteWithNoIDNoAddr -2025-10-30T18:19:26.554Z [INFO] server-8c96d448-e412-8301-3f0b-8caa6c1a4952: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c96d448-e412-8301-3f0b-8caa6c1a4952 Address:8c96d448-e412-8301-3f0b-8caa6c1a4952} {Suffrage:Voter ID:server-19c612f6-002f-1c88-e89a-66f8cdcca8ac Address:19c612f6-002f-1c88-e89a-66f8cdcca8ac} {Suffrage:Voter ID:server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c Address:103ff97f-365d-3ae8-7ea4-9a35ca72b24c}]" -2025-10-30T18:19:26.554Z [INFO] server-8c96d448-e412-8301-3f0b-8caa6c1a4952: entering follower state: follower="Node at 8c96d448-e412-8301-3f0b-8caa6c1a4952 [Follower]" leader-address= leader-id= -2025-10-30T18:19:26.554Z [INFO] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c96d448-e412-8301-3f0b-8caa6c1a4952 Address:8c96d448-e412-8301-3f0b-8caa6c1a4952} {Suffrage:Voter ID:server-19c612f6-002f-1c88-e89a-66f8cdcca8ac Address:19c612f6-002f-1c88-e89a-66f8cdcca8ac} {Suffrage:Voter ID:server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c Address:103ff97f-365d-3ae8-7ea4-9a35ca72b24c}]" -2025-10-30T18:19:26.554Z [INFO] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: entering follower state: follower="Node at 19c612f6-002f-1c88-e89a-66f8cdcca8ac [Follower]" leader-address= leader-id= -2025-10-30T18:19:26.554Z [INFO] server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c96d448-e412-8301-3f0b-8caa6c1a4952 Address:8c96d448-e412-8301-3f0b-8caa6c1a4952} {Suffrage:Voter ID:server-19c612f6-002f-1c88-e89a-66f8cdcca8ac Address:19c612f6-002f-1c88-e89a-66f8cdcca8ac} {Suffrage:Voter ID:server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c Address:103ff97f-365d-3ae8-7ea4-9a35ca72b24c}]" -2025-10-30T18:19:26.554Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:26.554Z [INFO] server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c: entering follower state: follower="Node at 103ff97f-365d-3ae8-7ea4-9a35ca72b24c [Follower]" leader-address= leader-id= -2025-10-30T18:19:26.624Z [WARN] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:26.624Z [INFO] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: entering candidate state: node="Node at 19c612f6-002f-1c88-e89a-66f8cdcca8ac [Candidate]" term=2 -2025-10-30T18:19:26.624Z [INFO] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: election won: term=2 tally=2 -2025-10-30T18:19:26.624Z [INFO] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: entering leader state: leader="Node at 19c612f6-002f-1c88-e89a-66f8cdcca8ac [Leader]" -2025-10-30T18:19:26.624Z [INFO] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: added peer, starting replication: peer=server-8c96d448-e412-8301-3f0b-8caa6c1a4952 -2025-10-30T18:19:26.624Z [INFO] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: added peer, starting replication: peer=server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c -2025-10-30T18:19:26.624Z [INFO] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: pipelining replication: peer="{Voter server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c 103ff97f-365d-3ae8-7ea4-9a35ca72b24c}" -2025-10-30T18:19:26.624Z [INFO] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: pipelining replication: peer="{Voter server-8c96d448-e412-8301-3f0b-8caa6c1a4952 8c96d448-e412-8301-3f0b-8caa6c1a4952}" -2025-10-30T18:19:26.730Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:26.624394584 +0000 UTC m=+112.903068481 (1 nodes), 69.458553ms from start of poll, 69.729378ms from cluster start. Timeout at 2025-10-30 18:19:26.729395151 +0000 UTC m=+113.008069048, 105.000567ms after stability -2025-10-30T18:19:26.730Z [INFO] cluster: starting stability test: raft-state=Leader -2025-10-30T18:19:26.835Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:19:26.730305679 +0000 UTC m=+113.008979576 (1 nodes), 774ns from start of poll, 175.640473ms from cluster start. Timeout at 2025-10-30 18:19:26.835319913 +0000 UTC m=+113.113993832, 105.014256ms after stability -2025-10-30T18:19:26.841Z [INFO] cluster: starting stability test: raft-state=Follower -2025-10-30T18:19:26.946Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:19:26.841559278 +0000 UTC m=+113.120233176 (2 nodes), 2.104µs from start of poll, 286.894073ms from cluster start. Timeout at 2025-10-30 18:19:26.94656978 +0000 UTC m=+113.225243677, 105.010501ms after stability -2025-10-30T18:19:26.952Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to heartbeat to: peer=8c96d448-e412-8301-3f0b-8caa6c1a4952 backoff time=10ms error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952" -2025-10-30T18:19:26.952Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to pipeline appendEntries: peer="{Voter server-8c96d448-e412-8301-3f0b-8caa6c1a4952 8c96d448-e412-8301-3f0b-8caa6c1a4952}" error="append pipeline closed" -2025-10-30T18:19:26.952Z [INFO] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: aborting pipeline replication: peer="{Voter server-8c96d448-e412-8301-3f0b-8caa6c1a4952 8c96d448-e412-8301-3f0b-8caa6c1a4952}" -2025-10-30T18:19:26.953Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to heartbeat to: peer=103ff97f-365d-3ae8-7ea4-9a35ca72b24c backoff time=10ms error="failed to connect to peer: 103ff97f-365d-3ae8-7ea4-9a35ca72b24c" -2025-10-30T18:19:26.954Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to pipeline appendEntries: peer="{Voter server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c 103ff97f-365d-3ae8-7ea4-9a35ca72b24c}" error="append pipeline closed" -2025-10-30T18:19:26.954Z [INFO] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: aborting pipeline replication: peer="{Voter server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c 103ff97f-365d-3ae8-7ea4-9a35ca72b24c}" -2025-10-30T18:19:26.961Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to appendEntries to: peer="{Voter server-8c96d448-e412-8301-3f0b-8caa6c1a4952 8c96d448-e412-8301-3f0b-8caa6c1a4952}" error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952" -2025-10-30T18:19:26.962Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to appendEntries to: peer="{Voter server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c 103ff97f-365d-3ae8-7ea4-9a35ca72b24c}" error="failed to connect to peer: 103ff97f-365d-3ae8-7ea4-9a35ca72b24c" -2025-10-30T18:19:26.972Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to heartbeat to: peer=8c96d448-e412-8301-3f0b-8caa6c1a4952 backoff time=10ms error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952" -2025-10-30T18:19:26.973Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to heartbeat to: peer=103ff97f-365d-3ae8-7ea4-9a35ca72b24c backoff time=10ms error="failed to connect to peer: 103ff97f-365d-3ae8-7ea4-9a35ca72b24c" -2025-10-30T18:19:26.977Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to appendEntries to: peer="{Voter server-8c96d448-e412-8301-3f0b-8caa6c1a4952 8c96d448-e412-8301-3f0b-8caa6c1a4952}" error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952" -2025-10-30T18:19:26.982Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to appendEntries to: peer="{Voter server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c 103ff97f-365d-3ae8-7ea4-9a35ca72b24c}" error="failed to connect to peer: 103ff97f-365d-3ae8-7ea4-9a35ca72b24c" -2025-10-30T18:19:26.988Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to heartbeat to: peer=8c96d448-e412-8301-3f0b-8caa6c1a4952 backoff time=10ms error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952" -2025-10-30T18:19:26.990Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to heartbeat to: peer=103ff97f-365d-3ae8-7ea4-9a35ca72b24c backoff time=10ms error="failed to connect to peer: 103ff97f-365d-3ae8-7ea4-9a35ca72b24c" -2025-10-30T18:19:26.995Z [WARN] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to contact: server-id=server-8c96d448-e412-8301-3f0b-8caa6c1a4952 time=50.859628ms -2025-10-30T18:19:26.995Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to appendEntries to: peer="{Voter server-8c96d448-e412-8301-3f0b-8caa6c1a4952 8c96d448-e412-8301-3f0b-8caa6c1a4952}" error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952" -2025-10-30T18:19:27.002Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to appendEntries to: peer="{Voter server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c 103ff97f-365d-3ae8-7ea4-9a35ca72b24c}" error="failed to connect to peer: 103ff97f-365d-3ae8-7ea4-9a35ca72b24c" -2025-10-30T18:19:27.005Z [WARN] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to contact: server-id=server-8c96d448-e412-8301-3f0b-8caa6c1a4952 time=61.170251ms -2025-10-30T18:19:27.005Z [WARN] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to contact: server-id=server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c time=58.737576ms -2025-10-30T18:19:27.005Z [WARN] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to contact quorum of nodes, stepping down -2025-10-30T18:19:27.005Z [INFO] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: entering follower state: follower="Node at 19c612f6-002f-1c88-e89a-66f8cdcca8ac [Follower]" leader-address= leader-id= -2025-10-30T18:19:27.005Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to heartbeat to: peer=8c96d448-e412-8301-3f0b-8caa6c1a4952 backoff time=20ms error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952" -2025-10-30T18:19:27.024Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to appendEntries to: peer="{Voter server-8c96d448-e412-8301-3f0b-8caa6c1a4952 8c96d448-e412-8301-3f0b-8caa6c1a4952}" error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952" -2025-10-30T18:19:27.046Z [WARN] server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c: heartbeat timeout reached, starting election: last-leader-addr=19c612f6-002f-1c88-e89a-66f8cdcca8ac last-leader-id=server-19c612f6-002f-1c88-e89a-66f8cdcca8ac -2025-10-30T18:19:27.046Z [INFO] server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c: entering candidate state: node="Node at 103ff97f-365d-3ae8-7ea4-9a35ca72b24c [Candidate]" term=3 -2025-10-30T18:19:27.046Z [ERROR] server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c: failed to make requestVote RPC: target="{Voter server-19c612f6-002f-1c88-e89a-66f8cdcca8ac 19c612f6-002f-1c88-e89a-66f8cdcca8ac}" error="failed to connect to peer: 19c612f6-002f-1c88-e89a-66f8cdcca8ac" term=3 -2025-10-30T18:19:27.046Z [WARN] server-8c96d448-e412-8301-3f0b-8caa6c1a4952: rejecting vote request since we have a leader: from=103ff97f-365d-3ae8-7ea4-9a35ca72b24c leader=19c612f6-002f-1c88-e89a-66f8cdcca8ac leader-id=server-19c612f6-002f-1c88-e89a-66f8cdcca8ac -2025-10-30T18:19:27.071Z [WARN] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= -2025-10-30T18:19:27.071Z [INFO] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: entering candidate state: node="Node at 19c612f6-002f-1c88-e89a-66f8cdcca8ac [Candidate]" term=3 -2025-10-30T18:19:27.071Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to make requestVote RPC: target="{Voter server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c 103ff97f-365d-3ae8-7ea4-9a35ca72b24c}" error="failed to connect to peer: 103ff97f-365d-3ae8-7ea4-9a35ca72b24c" term=3 -2025-10-30T18:19:27.071Z [ERROR] server-19c612f6-002f-1c88-e89a-66f8cdcca8ac: failed to make requestVote RPC: target="{Voter server-8c96d448-e412-8301-3f0b-8caa6c1a4952 8c96d448-e412-8301-3f0b-8caa6c1a4952}" error="failed to connect to peer: 8c96d448-e412-8301-3f0b-8caa6c1a4952" term=3 -2025-10-30T18:19:27.088Z [WARN] server-8c96d448-e412-8301-3f0b-8caa6c1a4952: heartbeat timeout reached, starting election: last-leader-addr=19c612f6-002f-1c88-e89a-66f8cdcca8ac last-leader-id=server-19c612f6-002f-1c88-e89a-66f8cdcca8ac -2025-10-30T18:19:27.088Z [INFO] server-8c96d448-e412-8301-3f0b-8caa6c1a4952: entering candidate state: node="Node at 8c96d448-e412-8301-3f0b-8caa6c1a4952 [Candidate]" term=3 -2025-10-30T18:19:27.088Z [INFO] server-103ff97f-365d-3ae8-7ea4-9a35ca72b24c: duplicate requestVote for same term: term=3 -2025-10-30T18:19:27.088Z [ERROR] server-8c96d448-e412-8301-3f0b-8caa6c1a4952: failed to make requestVote RPC: target="{Voter server-19c612f6-002f-1c88-e89a-66f8cdcca8ac 19c612f6-002f-1c88-e89a-66f8cdcca8ac}" error="failed to connect to peer: 19c612f6-002f-1c88-e89a-66f8cdcca8ac" term=3 -2025-10-30T18:19:27.090Z [INFO] server-8c96d448-e412-8301-3f0b-8caa6c1a4952: entering follower state: follower="Node at 8c96d448-e412-8301-3f0b-8caa6c1a4952 [Follower]" leader-address= leader-id= ---- PASS: TestRaft_VoteWithNoIDNoAddr (0.54s) +2025-10-30T18:23:21.883Z [INFO] server-beb6a00c-9fec-cf86-a270-419a5036d440: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-beb6a00c-9fec-cf86-a270-419a5036d440 Address:beb6a00c-9fec-cf86-a270-419a5036d440} {Suffrage:Voter ID:server-25ada95b-6526-0f3c-c932-4ecf2f5210b5 Address:25ada95b-6526-0f3c-c932-4ecf2f5210b5} {Suffrage:Voter ID:server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0 Address:105e520b-a37b-1efe-d8bd-4efe32cd9cc0}]" +2025-10-30T18:23:21.883Z [INFO] server-beb6a00c-9fec-cf86-a270-419a5036d440: entering follower state: follower="Node at beb6a00c-9fec-cf86-a270-419a5036d440 [Follower]" leader-address= leader-id= +2025-10-30T18:23:21.883Z [INFO] server-25ada95b-6526-0f3c-c932-4ecf2f5210b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-beb6a00c-9fec-cf86-a270-419a5036d440 Address:beb6a00c-9fec-cf86-a270-419a5036d440} {Suffrage:Voter ID:server-25ada95b-6526-0f3c-c932-4ecf2f5210b5 Address:25ada95b-6526-0f3c-c932-4ecf2f5210b5} {Suffrage:Voter ID:server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0 Address:105e520b-a37b-1efe-d8bd-4efe32cd9cc0}]" +2025-10-30T18:23:21.883Z [INFO] server-25ada95b-6526-0f3c-c932-4ecf2f5210b5: entering follower state: follower="Node at 25ada95b-6526-0f3c-c932-4ecf2f5210b5 [Follower]" leader-address= leader-id= +2025-10-30T18:23:21.883Z [INFO] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-beb6a00c-9fec-cf86-a270-419a5036d440 Address:beb6a00c-9fec-cf86-a270-419a5036d440} {Suffrage:Voter ID:server-25ada95b-6526-0f3c-c932-4ecf2f5210b5 Address:25ada95b-6526-0f3c-c932-4ecf2f5210b5} {Suffrage:Voter ID:server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0 Address:105e520b-a37b-1efe-d8bd-4efe32cd9cc0}]" +2025-10-30T18:23:21.883Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:21.883Z [INFO] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: entering follower state: follower="Node at 105e520b-a37b-1efe-d8bd-4efe32cd9cc0 [Follower]" leader-address= leader-id= +2025-10-30T18:23:21.951Z [WARN] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:21.951Z [INFO] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: entering candidate state: node="Node at 105e520b-a37b-1efe-d8bd-4efe32cd9cc0 [Candidate]" term=2 +2025-10-30T18:23:21.951Z [INFO] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: election won: term=2 tally=2 +2025-10-30T18:23:21.952Z [INFO] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: entering leader state: leader="Node at 105e520b-a37b-1efe-d8bd-4efe32cd9cc0 [Leader]" +2025-10-30T18:23:21.952Z [INFO] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: added peer, starting replication: peer=server-beb6a00c-9fec-cf86-a270-419a5036d440 +2025-10-30T18:23:21.952Z [INFO] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: added peer, starting replication: peer=server-25ada95b-6526-0f3c-c932-4ecf2f5210b5 +2025-10-30T18:23:21.952Z [INFO] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: pipelining replication: peer="{Voter server-beb6a00c-9fec-cf86-a270-419a5036d440 beb6a00c-9fec-cf86-a270-419a5036d440}" +2025-10-30T18:23:21.952Z [INFO] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: pipelining replication: peer="{Voter server-25ada95b-6526-0f3c-c932-4ecf2f5210b5 25ada95b-6526-0f3c-c932-4ecf2f5210b5}" +2025-10-30T18:23:22.057Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:21.95204513 +0000 UTC m=+125.025574446 (1 nodes), 68.300608ms from start of poll, 68.665108ms from cluster start. Timeout at 2025-10-30 18:23:22.057054894 +0000 UTC m=+125.130584182, 105.009736ms after stability +2025-10-30T18:23:22.057Z [INFO] cluster: starting stability test: raft-state=Leader +2025-10-30T18:23:22.162Z [INFO] cluster: stable state for Leader reached at 2025-10-30 18:23:22.057507597 +0000 UTC m=+125.131036899 (1 nodes), 675ns from start of poll, 174.127561ms from cluster start. Timeout at 2025-10-30 18:23:22.162522123 +0000 UTC m=+125.236051454, 105.014555ms after stability +2025-10-30T18:23:22.168Z [INFO] cluster: starting stability test: raft-state=Follower +2025-10-30T18:23:22.274Z [INFO] cluster: stable state for Follower reached at 2025-10-30 18:23:22.168343562 +0000 UTC m=+125.241872883 (2 nodes), 3.052µs from start of poll, 284.963545ms from cluster start. Timeout at 2025-10-30 18:23:22.273376255 +0000 UTC m=+125.346905555, 105.032672ms after stability +2025-10-30T18:23:22.275Z [ERROR] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to heartbeat to: peer=25ada95b-6526-0f3c-c932-4ecf2f5210b5 backoff time=10ms error="failed to connect to peer: 25ada95b-6526-0f3c-c932-4ecf2f5210b5" +2025-10-30T18:23:22.277Z [ERROR] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to heartbeat to: peer=beb6a00c-9fec-cf86-a270-419a5036d440 backoff time=10ms error="failed to connect to peer: beb6a00c-9fec-cf86-a270-419a5036d440" +2025-10-30T18:23:22.280Z [ERROR] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to pipeline appendEntries: peer="{Voter server-beb6a00c-9fec-cf86-a270-419a5036d440 beb6a00c-9fec-cf86-a270-419a5036d440}" error="append pipeline closed" +2025-10-30T18:23:22.280Z [INFO] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: aborting pipeline replication: peer="{Voter server-beb6a00c-9fec-cf86-a270-419a5036d440 beb6a00c-9fec-cf86-a270-419a5036d440}" +2025-10-30T18:23:22.283Z [ERROR] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to pipeline appendEntries: peer="{Voter server-25ada95b-6526-0f3c-c932-4ecf2f5210b5 25ada95b-6526-0f3c-c932-4ecf2f5210b5}" error="append pipeline closed" +2025-10-30T18:23:22.283Z [INFO] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: aborting pipeline replication: peer="{Voter server-25ada95b-6526-0f3c-c932-4ecf2f5210b5 25ada95b-6526-0f3c-c932-4ecf2f5210b5}" +2025-10-30T18:23:22.286Z [ERROR] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to appendEntries to: peer="{Voter server-beb6a00c-9fec-cf86-a270-419a5036d440 beb6a00c-9fec-cf86-a270-419a5036d440}" error="failed to connect to peer: beb6a00c-9fec-cf86-a270-419a5036d440" +2025-10-30T18:23:22.294Z [ERROR] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to appendEntries to: peer="{Voter server-25ada95b-6526-0f3c-c932-4ecf2f5210b5 25ada95b-6526-0f3c-c932-4ecf2f5210b5}" error="failed to connect to peer: 25ada95b-6526-0f3c-c932-4ecf2f5210b5" +2025-10-30T18:23:22.295Z [ERROR] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to heartbeat to: peer=25ada95b-6526-0f3c-c932-4ecf2f5210b5 backoff time=10ms error="failed to connect to peer: 25ada95b-6526-0f3c-c932-4ecf2f5210b5" +2025-10-30T18:23:22.297Z [ERROR] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to heartbeat to: peer=beb6a00c-9fec-cf86-a270-419a5036d440 backoff time=10ms error="failed to connect to peer: beb6a00c-9fec-cf86-a270-419a5036d440" +2025-10-30T18:23:22.304Z [ERROR] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to appendEntries to: peer="{Voter server-beb6a00c-9fec-cf86-a270-419a5036d440 beb6a00c-9fec-cf86-a270-419a5036d440}" error="failed to connect to peer: beb6a00c-9fec-cf86-a270-419a5036d440" +2025-10-30T18:23:22.311Z [ERROR] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to appendEntries to: peer="{Voter server-25ada95b-6526-0f3c-c932-4ecf2f5210b5 25ada95b-6526-0f3c-c932-4ecf2f5210b5}" error="failed to connect to peer: 25ada95b-6526-0f3c-c932-4ecf2f5210b5" +2025-10-30T18:23:22.315Z [ERROR] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to heartbeat to: peer=25ada95b-6526-0f3c-c932-4ecf2f5210b5 backoff time=10ms error="failed to connect to peer: 25ada95b-6526-0f3c-c932-4ecf2f5210b5" +2025-10-30T18:23:22.317Z [ERROR] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to heartbeat to: peer=beb6a00c-9fec-cf86-a270-419a5036d440 backoff time=10ms error="failed to connect to peer: beb6a00c-9fec-cf86-a270-419a5036d440" +2025-10-30T18:23:22.321Z [WARN] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to contact: server-id=server-beb6a00c-9fec-cf86-a270-419a5036d440 time=50.212325ms +2025-10-30T18:23:22.322Z [ERROR] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to appendEntries to: peer="{Voter server-beb6a00c-9fec-cf86-a270-419a5036d440 beb6a00c-9fec-cf86-a270-419a5036d440}" error="failed to connect to peer: beb6a00c-9fec-cf86-a270-419a5036d440" +2025-10-30T18:23:22.332Z [WARN] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to contact: server-id=server-beb6a00c-9fec-cf86-a270-419a5036d440 time=60.797226ms +2025-10-30T18:23:22.332Z [ERROR] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to appendEntries to: peer="{Voter server-25ada95b-6526-0f3c-c932-4ecf2f5210b5 25ada95b-6526-0f3c-c932-4ecf2f5210b5}" error="failed to connect to peer: 25ada95b-6526-0f3c-c932-4ecf2f5210b5" +2025-10-30T18:23:22.332Z [WARN] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to contact: server-id=server-25ada95b-6526-0f3c-c932-4ecf2f5210b5 time=58.518399ms +2025-10-30T18:23:22.332Z [WARN] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to contact quorum of nodes, stepping down +2025-10-30T18:23:22.332Z [INFO] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: entering follower state: follower="Node at 105e520b-a37b-1efe-d8bd-4efe32cd9cc0 [Follower]" leader-address= leader-id= +2025-10-30T18:23:22.333Z [ERROR] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to heartbeat to: peer=beb6a00c-9fec-cf86-a270-419a5036d440 backoff time=20ms error="failed to connect to peer: beb6a00c-9fec-cf86-a270-419a5036d440" +2025-10-30T18:23:22.350Z [ERROR] server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0: failed to appendEntries to: peer="{Voter server-beb6a00c-9fec-cf86-a270-419a5036d440 beb6a00c-9fec-cf86-a270-419a5036d440}" error="failed to connect to peer: beb6a00c-9fec-cf86-a270-419a5036d440" +2025-10-30T18:23:22.367Z [WARN] server-beb6a00c-9fec-cf86-a270-419a5036d440: heartbeat timeout reached, starting election: last-leader-addr=105e520b-a37b-1efe-d8bd-4efe32cd9cc0 last-leader-id=server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0 +2025-10-30T18:23:22.367Z [INFO] server-beb6a00c-9fec-cf86-a270-419a5036d440: entering candidate state: node="Node at beb6a00c-9fec-cf86-a270-419a5036d440 [Candidate]" term=3 +2025-10-30T18:23:22.367Z [ERROR] server-beb6a00c-9fec-cf86-a270-419a5036d440: failed to make requestVote RPC: target="{Voter server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0 105e520b-a37b-1efe-d8bd-4efe32cd9cc0}" error="failed to connect to peer: 105e520b-a37b-1efe-d8bd-4efe32cd9cc0" term=3 +2025-10-30T18:23:22.367Z [WARN] server-25ada95b-6526-0f3c-c932-4ecf2f5210b5: rejecting vote request since we have a leader: from=beb6a00c-9fec-cf86-a270-419a5036d440 leader=105e520b-a37b-1efe-d8bd-4efe32cd9cc0 leader-id=server-105e520b-a37b-1efe-d8bd-4efe32cd9cc0 +2025-10-30T18:23:22.380Z [INFO] server-beb6a00c-9fec-cf86-a270-419a5036d440: entering follower state: follower="Node at beb6a00c-9fec-cf86-a270-419a5036d440 [Follower]" leader-address= leader-id= +--- PASS: TestRaft_VoteWithNoIDNoAddr (0.56s) === RUN TestRaft_runFollower_State_Transition === RUN TestRaft_runFollower_State_Transition/NonVoter -2025-10-30T18:19:27.090Z [INFO] 127.0.0.1:38727 :: starting node: addr=127.0.0.1:38727 -2025-10-30T18:19:27.090Z [INFO] 127.0.0.1:38727 :: initial configuration: index=0 servers=[] -2025-10-30T18:19:27.090Z [INFO] 127.0.0.1:38727 :: entering follower state: follower="Node at 127.0.0.1:38727 [Follower]" leader-address= leader-id= -2025-10-30T18:19:27.142Z [WARN] 127.0.0.1:38727 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election +2025-10-30T18:23:22.445Z [INFO] 127.0.0.1:45709 :: starting node: addr=127.0.0.1:45709 +2025-10-30T18:23:22.445Z [INFO] 127.0.0.1:45709 :: initial configuration: index=0 servers=[] +2025-10-30T18:23:22.445Z [INFO] 127.0.0.1:45709 :: entering follower state: follower="Node at 127.0.0.1:45709 [Follower]" leader-address= leader-id= +2025-10-30T18:23:22.543Z [WARN] 127.0.0.1:45709 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election === RUN TestRaft_runFollower_State_Transition/Voter -2025-10-30T18:19:27.241Z [INFO] 127.0.0.1:37503 :: starting node: addr=127.0.0.1:37503 -2025-10-30T18:19:27.241Z [INFO] 127.0.0.1:37503 :: initial configuration: index=0 servers=[] -2025-10-30T18:19:27.241Z [INFO] 127.0.0.1:37503 :: entering follower state: follower="Node at 127.0.0.1:37503 [Follower]" leader-address= leader-id= -2025-10-30T18:19:27.303Z [WARN] 127.0.0.1:37503 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:22.700Z [INFO] 127.0.0.1:45685 :: starting node: addr=127.0.0.1:45685 +2025-10-30T18:23:22.700Z [INFO] 127.0.0.1:45685 :: initial configuration: index=0 servers=[] +2025-10-30T18:23:22.700Z [INFO] 127.0.0.1:45685 :: entering follower state: follower="Node at 127.0.0.1:45685 [Follower]" leader-address= leader-id= +2025-10-30T18:23:22.780Z [WARN] 127.0.0.1:45685 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= === RUN TestRaft_runFollower_State_Transition/Not_in_Config -2025-10-30T18:19:27.392Z [INFO] 127.0.0.1:40225 :: starting node: addr=127.0.0.1:40225 -2025-10-30T18:19:27.392Z [INFO] 127.0.0.1:40225 :: initial configuration: index=0 servers=[] -2025-10-30T18:19:27.392Z [INFO] 127.0.0.1:40225 :: entering follower state: follower="Node at 127.0.0.1:40225 [Follower]" leader-address= leader-id= -2025-10-30T18:19:27.486Z [WARN] 127.0.0.1:40225 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election ---- PASS: TestRaft_runFollower_State_Transition (0.45s) +2025-10-30T18:23:22.904Z [INFO] 127.0.0.1:46173 :: starting node: addr=127.0.0.1:46173 +2025-10-30T18:23:22.904Z [INFO] 127.0.0.1:46173 :: initial configuration: index=0 servers=[] +2025-10-30T18:23:22.904Z [INFO] 127.0.0.1:46173 :: entering follower state: follower="Node at 127.0.0.1:46173 [Follower]" leader-address= leader-id= +2025-10-30T18:23:22.991Z [WARN] 127.0.0.1:46173 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election +--- PASS: TestRaft_runFollower_State_Transition (0.61s) --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s) - --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s) - --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s) + --- PASS: TestRaft_runFollower_State_Transition/Voter (0.26s) + --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.20s) === RUN TestRaft_runFollower_ReloadTimeoutConfigs -2025-10-30T18:19:27.543Z [INFO] 127.0.0.1:38873 :: starting node: addr=127.0.0.1:38873 -2025-10-30T18:19:27.543Z [INFO] 127.0.0.1:38873 :: initial configuration: index=0 servers=[] -2025-10-30T18:19:27.543Z [INFO] 127.0.0.1:38873 :: entering follower state: follower="Node at 127.0.0.1:38873 [Follower]" leader-address= leader-id= -2025-10-30T18:19:27.543Z [WARN] 127.0.0.1:38873 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= +2025-10-30T18:23:23.054Z [INFO] 127.0.0.1:42517 :: starting node: addr=127.0.0.1:42517 +2025-10-30T18:23:23.054Z [INFO] 127.0.0.1:42517 :: initial configuration: index=0 servers=[] +2025-10-30T18:23:23.055Z [INFO] 127.0.0.1:42517 :: entering follower state: follower="Node at 127.0.0.1:42517 [Follower]" leader-address= leader-id= +2025-10-30T18:23:23.055Z [WARN] 127.0.0.1:42517 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= --- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s) === RUN TestSaturationMetric === RUN TestSaturationMetric/without_smoothing @@ -4283,42 +7055,27 @@ --- PASS: TestBackoff (0.00s) === RUN TestOverrideNotifyBool --- PASS: TestOverrideNotifyBool (0.00s) -PASS -ok github.com/hashicorp/raft 113.982s +FAIL +FAIL github.com/hashicorp/raft 126.293s ? github.com/hashicorp/raft/bench [no test files] +FAIL +dh_auto_test: error: cd obj-x86_64-linux-gnu && go test -vet=off -v -p 42 github.com/hashicorp/raft github.com/hashicorp/raft/bench returned exit code 1 +make[1]: *** [debian/rules:17: override_dh_auto_test] Error 25 make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' - create-stamp debian/debhelper-build-stamp - dh_testroot -O--buildsystem=golang - dh_prep -O--buildsystem=golang - dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang - dh_installdocs -O--buildsystem=golang - dh_installchangelogs -O--buildsystem=golang - dh_installsystemduser -O--buildsystem=golang - dh_perl -O--buildsystem=golang - dh_link -O--buildsystem=golang - dh_strip_nondeterminism -O--buildsystem=golang - dh_compress -O--buildsystem=golang - dh_fixperms -O--buildsystem=golang - dh_missing -O--buildsystem=golang - dh_installdeb -O--buildsystem=golang - dh_golang -O--buildsystem=golang - dh_gencontrol -O--buildsystem=golang - dh_md5sums -O--buildsystem=golang - dh_builddeb -O--buildsystem=golang -dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'. - dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_amd64.buildinfo - dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_amd64.changes -dpkg-genchanges: info: binary-only upload (no source code included) - dpkg-source --after-build . -dpkg-buildpackage: info: binary-only upload (no source included) -dpkg-genchanges: info: including full source code in upload +make: *** [debian/rules:14: binary] Error 2 +dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2 I: copying local configuration +E: Failed autobuilding of package +I: user script /srv/workspace/pbuilder/1573928/tmp/hooks/C01_cleanup starting +debug output: disk usage on i-capture-the-hostname at Thu Oct 30 18:23:24 UTC 2025 +Filesystem Size Used Avail Use% Mounted on +tmpfs 40G 0 40G 0% /dev/shm + +I: user script /srv/workspace/pbuilder/1573928/tmp/hooks/C01_cleanup finished I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env -I: removing directory /srv/workspace/pbuilder/228153 and its subdirectories -I: Current time: Thu Oct 30 06:19:32 -12 2025 -I: pbuilder-time-stamp: 1761848372 +I: removing directory /srv/workspace/pbuilder/1573928 and its subdirectories