I: pbuilder: network access will be disabled during build I: Current time: Tue Apr 21 22:08:13 -12 2026 I: pbuilder-time-stamp: 1776852493 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/unstable-reproducible-base.tgz] I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: using eatmydata during job I: Copying source file I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc] I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz] I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz] I: Extracting source dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: unsupported subcommand dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying 0001-Skip-unreliable-tests.patch I: Not using root during the build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/53424/tmp/hooks/D02_print_environment starting I: set BUILDDIR='/build/reproducible-path' BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' BUILDUSERNAME='pbuilder1' BUILD_ARCH='i386' DEBIAN_FRONTEND='noninteractive' DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=22 ' DISTRIBUTION='unstable' HOME='/root' HOST_ARCH='i386' IFS=' ' INVOCATION_ID='1954aad46e9043bfa5d277658b5ab597' LANG='C' LANGUAGE='en_US:en' LC_ALL='C' LD_LIBRARY_PATH='/usr/lib/libeatmydata' LD_PRELOAD='libeatmydata.so' MAIL='/var/mail/root' OPTIND='1' PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' PBCURRENTCOMMANDLINEOPERATION='build' PBUILDER_OPERATION='build' PBUILDER_PKGDATADIR='/usr/share/pbuilder' PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' PBUILDER_SYSCONFDIR='/etc' PPID='53424' PS1='# ' PS2='> ' PS4='+ ' PWD='/' SHELL='/bin/bash' SHLVL='2' SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.H0QjU75I/pbuilderrc_8hoc --distribution unstable --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/unstable-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.H0QjU75I/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc' SUDO_GID='112' SUDO_UID='107' SUDO_USER='jenkins' TERM='unknown' TZ='/usr/share/zoneinfo/Etc/GMT+12' USER='root' _='/usr/bin/systemd-run' http_proxy='http://213.165.73.152:3128' I: uname -a Linux ionos16-i386 6.1.0-32-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.129-1 (2025-03-06) x86_64 GNU/Linux I: ls -l /bin lrwxrwxrwx 1 root root 7 Mar 4 2025 /bin -> usr/bin I: user script /srv/workspace/pbuilder/53424/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: i386 Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 19770 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however: Package debhelper-compat is not installed. pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however: Package dh-sequence-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however: Package golang-github-stretchr-testify-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.24-go{a} golang-1.24-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-prometheus-sigv4-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu76{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libunistring5{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 0 packages upgraded, 107 newly installed, 0 to remove and 0 not upgraded. Need to get 130 MB of archives. After unpacking 708 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian unstable/main i386 sensible-utils all 0.0.24 [24.8 kB] Get: 2 http://deb.debian.org/debian unstable/main i386 libmagic-mgc i386 1:5.46-2 [337 kB] Get: 3 http://deb.debian.org/debian unstable/main i386 libmagic1t64 i386 1:5.46-2 [117 kB] Get: 4 http://deb.debian.org/debian unstable/main i386 file i386 1:5.46-2 [43.4 kB] Get: 5 http://deb.debian.org/debian unstable/main i386 gettext-base i386 0.23.1-1 [245 kB] Get: 6 http://deb.debian.org/debian unstable/main i386 libuchardet0 i386 0.0.8-1+b2 [69.2 kB] Get: 7 http://deb.debian.org/debian unstable/main i386 groff-base i386 1.23.0-7 [1199 kB] Get: 8 http://deb.debian.org/debian unstable/main i386 bsdextrautils i386 2.40.4-5 [96.5 kB] Get: 9 http://deb.debian.org/debian unstable/main i386 libpipeline1 i386 1.5.8-1 [41.2 kB] Get: 10 http://deb.debian.org/debian unstable/main i386 man-db i386 2.13.0-1 [1428 kB] Get: 11 http://deb.debian.org/debian unstable/main i386 m4 i386 1.4.19-7 [301 kB] Get: 12 http://deb.debian.org/debian unstable/main i386 autoconf all 2.72-3 [493 kB] Get: 13 http://deb.debian.org/debian unstable/main i386 autotools-dev all 20220109.1 [51.6 kB] Get: 14 http://deb.debian.org/debian unstable/main i386 automake all 1:1.17-3 [862 kB] Get: 15 http://deb.debian.org/debian unstable/main i386 autopoint all 0.23.1-1 [770 kB] Get: 16 http://deb.debian.org/debian unstable/main i386 libdebhelper-perl all 13.24.1 [90.9 kB] Get: 17 http://deb.debian.org/debian unstable/main i386 libtool all 2.5.4-4 [539 kB] Get: 18 http://deb.debian.org/debian unstable/main i386 dh-autoreconf all 20 [17.1 kB] Get: 19 http://deb.debian.org/debian unstable/main i386 libarchive-zip-perl all 1.68-1 [104 kB] Get: 20 http://deb.debian.org/debian unstable/main i386 libfile-stripnondeterminism-perl all 1.14.1-2 [19.7 kB] Get: 21 http://deb.debian.org/debian unstable/main i386 dh-strip-nondeterminism all 1.14.1-2 [8620 B] Get: 22 http://deb.debian.org/debian unstable/main i386 libelf1t64 i386 0.192-4 [195 kB] Get: 23 http://deb.debian.org/debian unstable/main i386 dwz i386 0.15-1+b1 [116 kB] Get: 24 http://deb.debian.org/debian unstable/main i386 libunistring5 i386 1.3-1 [458 kB] Get: 25 http://deb.debian.org/debian unstable/main i386 libicu76 i386 76.1-3 [9893 kB] Get: 26 http://deb.debian.org/debian unstable/main i386 libxml2 i386 2.12.7+dfsg+really2.9.14-0.3+b1 [734 kB] Get: 27 http://deb.debian.org/debian unstable/main i386 gettext i386 0.23.1-1 [1714 kB] Get: 28 http://deb.debian.org/debian unstable/main i386 intltool-debian all 0.35.0+20060710.6 [22.9 kB] Get: 29 http://deb.debian.org/debian unstable/main i386 po-debconf all 1.0.21+nmu1 [248 kB] Get: 30 http://deb.debian.org/debian unstable/main i386 debhelper all 13.24.1 [920 kB] Get: 31 http://deb.debian.org/debian unstable/main i386 dh-golang all 1.63 [24.4 kB] Get: 32 http://deb.debian.org/debian unstable/main i386 golang-1.24-src all 1.24.1-1 [21.2 MB] Get: 33 http://deb.debian.org/debian unstable/main i386 golang-1.24-go i386 1.24.1-1 [26.8 MB] Get: 34 http://deb.debian.org/debian unstable/main i386 golang-src all 2:1.24~2 [5136 B] Get: 35 http://deb.debian.org/debian unstable/main i386 golang-go i386 2:1.24~2 [44.3 kB] Get: 36 http://deb.debian.org/debian unstable/main i386 golang-any i386 2:1.24~2 [5216 B] Get: 37 http://deb.debian.org/debian unstable/main i386 golang-github-alecthomas-units-dev all 0.0~git20240927.0f3dac3-1 [8880 B] Get: 38 http://deb.debian.org/debian unstable/main i386 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 39 http://deb.debian.org/debian unstable/main i386 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 40 http://deb.debian.org/debian unstable/main i386 golang-golang-x-sys-dev all 0.22.0-1 [431 kB] Get: 41 http://deb.debian.org/debian unstable/main i386 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B] Get: 42 http://deb.debian.org/debian unstable/main i386 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB] Get: 43 http://deb.debian.org/debian unstable/main i386 golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB] Get: 44 http://deb.debian.org/debian unstable/main i386 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB] Get: 45 http://deb.debian.org/debian unstable/main i386 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 46 http://deb.debian.org/debian unstable/main i386 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB] Get: 47 http://deb.debian.org/debian unstable/main i386 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB] Get: 48 http://deb.debian.org/debian unstable/main i386 golang-github-stretchr-testify-dev all 1.10.0-1 [78.4 kB] Get: 49 http://deb.debian.org/debian unstable/main i386 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 50 http://deb.debian.org/debian unstable/main i386 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB] Get: 51 http://deb.debian.org/debian unstable/main i386 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB] Get: 52 http://deb.debian.org/debian unstable/main i386 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B] Get: 53 http://deb.debian.org/debian unstable/main i386 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB] Get: 54 http://deb.debian.org/debian unstable/main i386 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 55 http://deb.debian.org/debian unstable/main i386 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB] Get: 56 http://deb.debian.org/debian unstable/main i386 golang-google-protobuf-dev all 1.36.5-1 [796 kB] Get: 57 http://deb.debian.org/debian unstable/main i386 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB] Get: 58 http://deb.debian.org/debian unstable/main i386 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B] Get: 59 http://deb.debian.org/debian unstable/main i386 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 60 http://deb.debian.org/debian unstable/main i386 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB] Get: 61 http://deb.debian.org/debian unstable/main i386 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB] Get: 62 http://deb.debian.org/debian unstable/main i386 golang-github-beorn7-perks-dev all 1.0.1-3 [12.2 kB] Get: 63 http://deb.debian.org/debian unstable/main i386 golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB] Get: 64 http://deb.debian.org/debian unstable/main i386 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 65 http://deb.debian.org/debian unstable/main i386 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB] Get: 66 http://deb.debian.org/debian unstable/main i386 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB] Get: 67 http://deb.debian.org/debian unstable/main i386 golang-github-klauspost-compress-dev all 1.18.0+ds1-1 [33.3 MB] Get: 68 http://deb.debian.org/debian unstable/main i386 golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB] Get: 69 http://deb.debian.org/debian unstable/main i386 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB] Get: 70 http://deb.debian.org/debian unstable/main i386 golang-gopkg-yaml.v2-dev all 2.4.0-5 [60.5 kB] Get: 71 http://deb.debian.org/debian unstable/main i386 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB] Get: 72 http://deb.debian.org/debian unstable/main i386 golang-golang-x-term-dev all 0.22.0-1 [14.8 kB] Get: 73 http://deb.debian.org/debian unstable/main i386 golang-golang-x-text-dev all 0.22.0-1 [3995 kB] Get: 74 http://deb.debian.org/debian unstable/main i386 golang-golang-x-crypto-dev all 1:0.25.0-1 [1682 kB] Get: 75 http://deb.debian.org/debian unstable/main i386 golang-golang-x-net-dev all 1:0.27.0-1 [898 kB] Get: 76 http://deb.debian.org/debian unstable/main i386 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB] Get: 77 http://deb.debian.org/debian unstable/main i386 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 78 http://deb.debian.org/debian unstable/main i386 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB] Get: 79 http://deb.debian.org/debian unstable/main i386 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB] Get: 80 http://deb.debian.org/debian unstable/main i386 golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B] Get: 81 http://deb.debian.org/debian unstable/main i386 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 82 http://deb.debian.org/debian unstable/main i386 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-4 [14.6 kB] Get: 83 http://deb.debian.org/debian unstable/main i386 golang-github-prometheus-sigv4-dev all 0.1.2-2 [8484 B] Get: 84 http://deb.debian.org/debian unstable/main i386 golang-golang-x-oauth2-dev all 0.27.0-1 [37.2 kB] Get: 85 http://deb.debian.org/debian unstable/main i386 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB] Get: 86 http://deb.debian.org/debian unstable/main i386 golang-github-prometheus-common-dev all 0.62.0-3 [118 kB] Get: 87 http://deb.debian.org/debian unstable/main i386 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB] Get: 88 http://deb.debian.org/debian unstable/main i386 golang-github-prometheus-procfs-dev all 0.15.1-1 [184 kB] Get: 89 http://deb.debian.org/debian unstable/main i386 golang-github-prometheus-client-golang-dev all 1.21.1-1 [194 kB] Get: 90 http://deb.debian.org/debian unstable/main i386 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB] Get: 91 http://deb.debian.org/debian unstable/main i386 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB] Get: 92 http://deb.debian.org/debian unstable/main i386 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B] Get: 93 http://deb.debian.org/debian unstable/main i386 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 94 http://deb.debian.org/debian unstable/main i386 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 95 http://deb.debian.org/debian unstable/main i386 libsasl2-modules-db i386 2.1.28+dfsg1-9 [20.8 kB] Get: 96 http://deb.debian.org/debian unstable/main i386 libsasl2-2 i386 2.1.28+dfsg1-9 [61.2 kB] Get: 97 http://deb.debian.org/debian unstable/main i386 libssl-dev i386 3.4.1-1 [2837 kB] Get: 98 http://deb.debian.org/debian unstable/main i386 libsasl2-dev i386 2.1.28+dfsg1-9 [247 kB] Get: 99 http://deb.debian.org/debian unstable/main i386 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB] Get: 100 http://deb.debian.org/debian unstable/main i386 golang-golang-x-mod-dev all 0.19.0-1 [104 kB] Get: 101 http://deb.debian.org/debian unstable/main i386 golang-github-yuin-goldmark-dev all 1.7.4-1 [113 kB] Get: 102 http://deb.debian.org/debian unstable/main i386 golang-golang-x-tools-dev all 1:0.25.0+ds-1 [1608 kB] Get: 103 http://deb.debian.org/debian unstable/main i386 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB] Get: 104 http://deb.debian.org/debian unstable/main i386 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB] Get: 105 http://deb.debian.org/debian unstable/main i386 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-2 [31.2 kB] Get: 106 http://deb.debian.org/debian unstable/main i386 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 107 http://deb.debian.org/debian unstable/main i386 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 130 MB in 1s (101 MB/s) Preconfiguring packages ... Selecting previously unselected package sensible-utils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19770 files and directories currently installed.) Preparing to unpack .../000-sensible-utils_0.0.24_all.deb ... Unpacking sensible-utils (0.0.24) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../001-libmagic-mgc_1%3a5.46-2_i386.deb ... Unpacking libmagic-mgc (1:5.46-2) ... Selecting previously unselected package libmagic1t64:i386. Preparing to unpack .../002-libmagic1t64_1%3a5.46-2_i386.deb ... Unpacking libmagic1t64:i386 (1:5.46-2) ... Selecting previously unselected package file. Preparing to unpack .../003-file_1%3a5.46-2_i386.deb ... Unpacking file (1:5.46-2) ... Selecting previously unselected package gettext-base. Preparing to unpack .../004-gettext-base_0.23.1-1_i386.deb ... Unpacking gettext-base (0.23.1-1) ... Selecting previously unselected package libuchardet0:i386. Preparing to unpack .../005-libuchardet0_0.0.8-1+b2_i386.deb ... Unpacking libuchardet0:i386 (0.0.8-1+b2) ... Selecting previously unselected package groff-base. Preparing to unpack .../006-groff-base_1.23.0-7_i386.deb ... Unpacking groff-base (1.23.0-7) ... Selecting previously unselected package bsdextrautils. Preparing to unpack .../007-bsdextrautils_2.40.4-5_i386.deb ... Unpacking bsdextrautils (2.40.4-5) ... Selecting previously unselected package libpipeline1:i386. Preparing to unpack .../008-libpipeline1_1.5.8-1_i386.deb ... Unpacking libpipeline1:i386 (1.5.8-1) ... Selecting previously unselected package man-db. Preparing to unpack .../009-man-db_2.13.0-1_i386.deb ... Unpacking man-db (2.13.0-1) ... Selecting previously unselected package m4. Preparing to unpack .../010-m4_1.4.19-7_i386.deb ... Unpacking m4 (1.4.19-7) ... Selecting previously unselected package autoconf. Preparing to unpack .../011-autoconf_2.72-3_all.deb ... Unpacking autoconf (2.72-3) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ... Unpacking autotools-dev (20220109.1) ... Selecting previously unselected package automake. Preparing to unpack .../013-automake_1%3a1.17-3_all.deb ... Unpacking automake (1:1.17-3) ... Selecting previously unselected package autopoint. Preparing to unpack .../014-autopoint_0.23.1-1_all.deb ... Unpacking autopoint (0.23.1-1) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../015-libdebhelper-perl_13.24.1_all.deb ... Unpacking libdebhelper-perl (13.24.1) ... Selecting previously unselected package libtool. Preparing to unpack .../016-libtool_2.5.4-4_all.deb ... Unpacking libtool (2.5.4-4) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../017-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.1-2_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.14.1-2) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../020-dh-strip-nondeterminism_1.14.1-2_all.deb ... Unpacking dh-strip-nondeterminism (1.14.1-2) ... Selecting previously unselected package libelf1t64:i386. Preparing to unpack .../021-libelf1t64_0.192-4_i386.deb ... Unpacking libelf1t64:i386 (0.192-4) ... Selecting previously unselected package dwz. Preparing to unpack .../022-dwz_0.15-1+b1_i386.deb ... Unpacking dwz (0.15-1+b1) ... Selecting previously unselected package libunistring5:i386. Preparing to unpack .../023-libunistring5_1.3-1_i386.deb ... Unpacking libunistring5:i386 (1.3-1) ... Selecting previously unselected package libicu76:i386. Preparing to unpack .../024-libicu76_76.1-3_i386.deb ... Unpacking libicu76:i386 (76.1-3) ... Selecting previously unselected package libxml2:i386. Preparing to unpack .../025-libxml2_2.12.7+dfsg+really2.9.14-0.3+b1_i386.deb ... Unpacking libxml2:i386 (2.12.7+dfsg+really2.9.14-0.3+b1) ... Selecting previously unselected package gettext. Preparing to unpack .../026-gettext_0.23.1-1_i386.deb ... Unpacking gettext (0.23.1-1) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../027-intltool-debian_0.35.0+20060710.6_all.deb ... Unpacking intltool-debian (0.35.0+20060710.6) ... Selecting previously unselected package po-debconf. Preparing to unpack .../028-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../029-debhelper_13.24.1_all.deb ... Unpacking debhelper (13.24.1) ... Selecting previously unselected package dh-golang. Preparing to unpack .../030-dh-golang_1.63_all.deb ... Unpacking dh-golang (1.63) ... Selecting previously unselected package golang-1.24-src. Preparing to unpack .../031-golang-1.24-src_1.24.1-1_all.deb ... Unpacking golang-1.24-src (1.24.1-1) ... Selecting previously unselected package golang-1.24-go. Preparing to unpack .../032-golang-1.24-go_1.24.1-1_i386.deb ... Unpacking golang-1.24-go (1.24.1-1) ... Selecting previously unselected package golang-src. Preparing to unpack .../033-golang-src_2%3a1.24~2_all.deb ... Unpacking golang-src (2:1.24~2) ... Selecting previously unselected package golang-go:i386. Preparing to unpack .../034-golang-go_2%3a1.24~2_i386.deb ... Unpacking golang-go:i386 (2:1.24~2) ... Selecting previously unselected package golang-any:i386. Preparing to unpack .../035-golang-any_2%3a1.24~2_i386.deb ... Unpacking golang-any:i386 (2:1.24~2) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../036-golang-github-alecthomas-units-dev_0.0~git20240927.0f3dac3-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20240927.0f3dac3-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../037-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../038-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../039-golang-golang-x-sys-dev_0.22.0-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.22.0-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../040-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../041-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../042-golang-github-fatih-color-dev_1.17.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.17.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../043-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../044-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../045-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../046-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../047-golang-github-stretchr-testify-dev_1.10.0-1_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.10.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../048-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../049-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../050-golang-github-pkg-errors-dev_0.9.1-3_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-3) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../051-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../052-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../053-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-google-go-cmp-dev. Preparing to unpack .../054-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ... Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ... Selecting previously unselected package golang-google-protobuf-dev. Preparing to unpack .../055-golang-google-protobuf-dev_1.36.5-1_all.deb ... Unpacking golang-google-protobuf-dev (1.36.5-1) ... Selecting previously unselected package golang-github-golang-protobuf-1-5-dev. Preparing to unpack .../056-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ... Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../057-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../058-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../059-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../060-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../061-golang-github-beorn7-perks-dev_1.0.1-3_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-3) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../062-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../063-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../064-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../065-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ... Selecting previously unselected package golang-github-klauspost-compress-dev. Preparing to unpack .../066-golang-github-klauspost-compress-dev_1.18.0+ds1-1_all.deb ... Unpacking golang-github-klauspost-compress-dev (1.18.0+ds1-1) ... Selecting previously unselected package golang-github-kylelemons-godebug-dev. Preparing to unpack .../067-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ... Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../068-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../069-golang-gopkg-yaml.v2-dev_2.4.0-5_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.0-5) ... Selecting previously unselected package golang-github-jmespath-go-jmespath-dev. Preparing to unpack .../070-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ... Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Selecting previously unselected package golang-golang-x-term-dev. Preparing to unpack .../071-golang-golang-x-term-dev_0.22.0-1_all.deb ... Unpacking golang-golang-x-term-dev (0.22.0-1) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../072-golang-golang-x-text-dev_0.22.0-1_all.deb ... Unpacking golang-golang-x-text-dev (0.22.0-1) ... Selecting previously unselected package golang-golang-x-crypto-dev. Preparing to unpack .../073-golang-golang-x-crypto-dev_1%3a0.25.0-1_all.deb ... Unpacking golang-golang-x-crypto-dev (1:0.25.0-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../074-golang-golang-x-net-dev_1%3a0.27.0-1_all.deb ... Unpacking golang-golang-x-net-dev (1:0.27.0-1) ... Selecting previously unselected package golang-github-aws-aws-sdk-go-dev. Preparing to unpack .../075-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ... Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../076-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-kit-log-dev. Preparing to unpack .../077-golang-github-go-kit-log-dev_0.2.1-1_all.deb ... Unpacking golang-github-go-kit-log-dev (0.2.1-1) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../078-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Selecting previously unselected package golang-github-munnerz-goautoneg-dev. Preparing to unpack .../079-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ... Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../080-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../081-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-4_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ... Selecting previously unselected package golang-github-prometheus-sigv4-dev. Preparing to unpack .../082-golang-github-prometheus-sigv4-dev_0.1.2-2_all.deb ... Unpacking golang-github-prometheus-sigv4-dev (0.1.2-2) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../083-golang-golang-x-oauth2-dev_0.27.0-1_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.27.0-1) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../084-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../085-golang-github-prometheus-common-dev_0.62.0-3_all.deb ... Unpacking golang-github-prometheus-common-dev (0.62.0-3) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../086-golang-golang-x-sync-dev_0.7.0-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.7.0-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../087-golang-github-prometheus-procfs-dev_0.15.1-1_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.15.1-1) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../088-golang-github-prometheus-client-golang-dev_1.21.1-1_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.21.1-1) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../089-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../090-golang-github-creack-pty-dev_1.1.21-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.21-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../091-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../092-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../093-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:i386. Preparing to unpack .../094-libsasl2-modules-db_2.1.28+dfsg1-9_i386.deb ... Unpacking libsasl2-modules-db:i386 (2.1.28+dfsg1-9) ... Selecting previously unselected package libsasl2-2:i386. Preparing to unpack .../095-libsasl2-2_2.1.28+dfsg1-9_i386.deb ... Unpacking libsasl2-2:i386 (2.1.28+dfsg1-9) ... Selecting previously unselected package libssl-dev:i386. Preparing to unpack .../096-libssl-dev_3.4.1-1_i386.deb ... Unpacking libssl-dev:i386 (3.4.1-1) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../097-libsasl2-dev_2.1.28+dfsg1-9_i386.deb ... Unpacking libsasl2-dev (2.1.28+dfsg1-9) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../098-golang-github-kr-text-dev_0.2.0-2_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-2) ... Selecting previously unselected package golang-golang-x-mod-dev. Preparing to unpack .../099-golang-golang-x-mod-dev_0.19.0-1_all.deb ... Unpacking golang-golang-x-mod-dev (0.19.0-1) ... Selecting previously unselected package golang-github-yuin-goldmark-dev. Preparing to unpack .../100-golang-github-yuin-goldmark-dev_1.7.4-1_all.deb ... Unpacking golang-github-yuin-goldmark-dev (1.7.4-1) ... Selecting previously unselected package golang-golang-x-tools-dev. Preparing to unpack .../101-golang-golang-x-tools-dev_1%3a0.25.0+ds-1_all.deb ... Unpacking golang-golang-x-tools-dev (1:0.25.0+ds-1) ... Selecting previously unselected package golang-github-rogpeppe-go-internal-dev. Preparing to unpack .../102-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ... Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../103-golang-github-kr-pretty-dev_0.3.1-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.3.1-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../104-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-2_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../105-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../106-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up golang-1.24-src (1.24.1-1) ... Setting up libpipeline1:i386 (1.5.8-1) ... Setting up golang-github-google-go-cmp-dev (0.6.0-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.0-5) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Setting up bsdextrautils (2.40.4-5) ... Setting up golang-github-creack-pty-dev (1.1.21-1) ... Setting up golang-github-stretchr-objx-dev (0.5.2-1) ... Setting up libmagic-mgc (1:5.46-2) ... Setting up golang-github-pkg-errors-dev (0.9.1-3) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ... Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ... Setting up libdebhelper-perl (13.24.1) ... Setting up golang-golang-x-sys-dev (0.22.0-1) ... Setting up libmagic1t64:i386 (1:5.46-2) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.23.1-1) ... Setting up m4 (1.4.19-7) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up golang-golang-x-term-dev (0.22.0-1) ... Setting up file (1:5.46-2) ... Setting up golang-github-beorn7-perks-dev (1.0.1-3) ... Setting up libelf1t64:i386 (0.192-4) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:i386 (2.1.28+dfsg1-9) ... Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ... Setting up autotools-dev (20220109.1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-google-protobuf-dev (1.36.5-1) ... Setting up golang-github-kr-text-dev (0.2.0-2) ... Setting up golang-github-stretchr-testify-dev (1.10.0-1) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Setting up golang-github-yuin-goldmark-dev (1.7.4-1) ... Setting up libunistring5:i386 (1.3-1) ... Setting up libssl-dev:i386 (3.4.1-1) ... Setting up golang-golang-x-sync-dev (0.7.0-1) ... Setting up golang-golang-x-mod-dev (0.19.0-1) ... Setting up autopoint (0.23.1-1) ... Setting up golang-1.24-go (1.24.1-1) ... Setting up golang-github-klauspost-compress-dev (1.18.0+ds1-1) ... Setting up libsasl2-2:i386 (2.1.28+dfsg1-9) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Setting up autoconf (2.72-3) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Setting up dwz (0.15-1+b1) ... Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ... Setting up sensible-utils (0.0.24) ... Setting up golang-golang-x-text-dev (0.22.0-1) ... Setting up libuchardet0:i386 (0.0.8-1+b2) ... Setting up golang-github-prometheus-procfs-dev (0.15.1-1) ... Setting up golang-github-alecthomas-units-dev (0.0~git20240927.0f3dac3-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ... Setting up golang-golang-x-crypto-dev (1:0.25.0-1) ... Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ... Setting up libicu76:i386 (76.1-3) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-golang-x-oauth2-dev (0.27.0-1) ... Setting up golang-src (2:1.24~2) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Setting up libxml2:i386 (2.12.7+dfsg+really2.9.14-0.3+b1) ... Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Setting up automake (1:1.17-3) ... update-alternatives: using /usr/bin/automake-1.17 to provide /usr/bin/automake (automake) in auto mode Setting up libfile-stripnondeterminism-perl (1.14.1-2) ... Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ... Setting up golang-github-go-kit-log-dev (0.2.1-1) ... Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ... Setting up gettext (0.23.1-1) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Setting up libtool (2.5.4-4) ... Setting up golang-go:i386 (2:1.24~2) ... Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ... Setting up golang-github-json-iterator-go-dev (1.1.12-2) ... Setting up golang-any:i386 (2:1.24~2) ... Setting up intltool-debian (0.35.0+20060710.6) ... Setting up dh-autoreconf (20) ... Setting up libsasl2-dev (2.1.28+dfsg1-9) ... Setting up golang-golang-x-net-dev (1:0.27.0-1) ... Setting up dh-strip-nondeterminism (1.14.1-2) ... Setting up groff-base (1.23.0-7) ... Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ... Setting up golang-github-fatih-color-dev (1.17.0-1) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up golang-golang-x-tools-dev (1:0.25.0+ds-1) ... Setting up man-db (2.13.0-1) ... Not building database; man-db/auto-update is not 'true'. Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up golang-github-kr-pretty-dev (0.3.1-1) ... Setting up debhelper (13.24.1) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Setting up dh-golang (1.63) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ... Setting up golang-github-prometheus-sigv4-dev (0.1.2-2) ... Setting up golang-github-prometheus-common-dev (0.62.0-3) ... Setting up golang-github-prometheus-client-golang-dev (1.21.1-1) ... Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ... Processing triggers for libc-bin (2.41-6) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps I: Building the package I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.5.0-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.5.0-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Daniel Swarbrick dpkg-source --before-build . dpkg-buildpackage: info: host architecture i386 debian/rules clean dh clean --buildsystem=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules binary dh binary --buildsystem=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-i686-linux-gnu && go install -trimpath -v -p 22 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/asan internal/goarch container/list internal/coverage/rtcov internal/msan internal/profilerecord internal/goos unicode/utf8 internal/byteorder crypto/internal/fips140/alias math/bits internal/cpu internal/itoa internal/godebugs sync/atomic internal/runtime/atomic internal/goexperiment internal/unsafeheader crypto/internal/boring/sig internal/runtime/syscall cmp unicode encoding unicode/utf16 log/internal internal/nettrace vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/alias internal/runtime/math internal/runtime/sys internal/abi internal/chacha8rand crypto/internal/fips140deps/byteorder math crypto/internal/fips140deps/cpu internal/bytealg internal/runtime/exithook internal/stringslite internal/race internal/sync internal/runtime/maps runtime crypto/internal/fips140/subtle iter internal/reflectlite weak sync maps slices crypto/subtle errors sort internal/testlog internal/singleflight internal/bisect unique internal/oserror path io github.com/hashicorp/golang-lru/simplelru math/rand/v2 vendor/golang.org/x/net/dns/dnsmessage strconv runtime/cgo syscall internal/godebug hash internal/saferio bytes strings crypto/internal/fips140deps/godebug hash/crc64 hash/crc32 math/rand crypto/internal/randutil crypto/internal/fips140 net/netip bufio crypto net/http/internal/ascii github.com/hashicorp/go-immutable-radix crypto/internal/impl vendor/golang.org/x/text/transform regexp/syntax html reflect crypto/internal/fips140/sha3 crypto/internal/fips140/sha512 crypto/internal/fips140/sha256 crypto/tls/internal/fips140tls crypto/sha3 crypto/internal/fips140/hmac crypto/internal/fips140/check crypto/internal/fips140hash crypto/internal/fips140/bigmod crypto/internal/fips140/edwards25519/field crypto/internal/fips140/hkdf crypto/internal/fips140/tls12 crypto/internal/fips140/aes crypto/internal/fips140/nistec/fiat crypto/internal/fips140/tls13 crypto/internal/fips140/edwards25519 internal/syscall/execenv internal/syscall/unix time regexp context io/fs internal/poll internal/fmtsort encoding/binary internal/filepathlite crypto/internal/fips140/nistec os encoding/base64 vendor/golang.org/x/crypto/internal/poly1305 golang.org/x/sys/unix encoding/pem crypto/internal/sysrand fmt path/filepath io/ioutil os/signal internal/sysinfo net crypto/internal/entropy crypto/internal/fips140/drbg crypto/internal/fips140/aes/gcm crypto/internal/fips140only crypto/internal/fips140/ecdh crypto/internal/fips140/ed25519 crypto/internal/fips140/ecdsa crypto/internal/fips140/mlkem crypto/internal/fips140/rsa crypto/md5 crypto/rc4 crypto/cipher encoding/json encoding/hex log net/url compress/flate vendor/golang.org/x/net/http2/hpack math/big vendor/golang.org/x/text/unicode/norm go/token mime mime/quotedprintable net/http/internal encoding/gob text/template/parse flag runtime/trace runtime/debug crypto/des crypto/internal/boring vendor/golang.org/x/crypto/chacha20 vendor/golang.org/x/text/unicode/bidi crypto/aes crypto/ecdh crypto/hmac crypto/sha512 vendor/golang.org/x/crypto/chacha20poly1305 crypto/sha1 crypto/sha256 compress/gzip testing vendor/golang.org/x/text/secure/bidirule github.com/mattn/go-isatty text/template github.com/mattn/go-colorable vendor/golang.org/x/net/idna github.com/fatih/color github.com/hashicorp/go-hclog crypto/rand crypto/elliptic crypto/internal/boring/bbig crypto/dsa encoding/asn1 crypto/ed25519 crypto/internal/hpke crypto/rsa html/template vendor/golang.org/x/crypto/cryptobyte crypto/x509/pkix crypto/ecdsa vendor/golang.org/x/net/http/httpproxy net/textproto crypto/x509 vendor/golang.org/x/net/http/httpguts mime/multipart crypto/tls net/http/httptrace net/http github.com/armon/go-metrics net/rpc github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench debian/rules override_dh_auto_test make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' dh_auto_test cd obj-i686-linux-gnu && go test -vet=off -v -p 22 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir 2026-04-22T10:09:13.978Z [INFO] creating new snapshot: path=/tmp/raft1320065332/raft496559154/snapshots/3-10-1776852553978.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.01s) === RUN TestFileSS_CreateSnapshot 2026-04-22T10:09:13.984Z [INFO] creating new snapshot: path=/tmp/raft960437121/snapshots/3-10-1776852553984.tmp 2026-04-22T10:09:13.990Z [WARN] found temporary snapshot: name=3-10-1776852553984.tmp --- PASS: TestFileSS_CreateSnapshot (0.01s) === RUN TestFileSS_CancelSnapshot 2026-04-22T10:09:13.997Z [INFO] creating new snapshot: path=/tmp/raft1765692347/snapshots/3-10-1776852553997.tmp --- PASS: TestFileSS_CancelSnapshot (0.00s) === RUN TestFileSS_Retention 2026-04-22T10:09:14.002Z [INFO] creating new snapshot: path=/tmp/raft298372192/snapshots/3-10-1776852554002.tmp 2026-04-22T10:09:14.010Z [INFO] creating new snapshot: path=/tmp/raft298372192/snapshots/3-11-1776852554010.tmp 2026-04-22T10:09:14.019Z [INFO] creating new snapshot: path=/tmp/raft298372192/snapshots/3-12-1776852554019.tmp 2026-04-22T10:09:14.029Z [INFO] reaping snapshot: path=/tmp/raft298372192/snapshots/3-10-1776852554002 2026-04-22T10:09:14.029Z [INFO] creating new snapshot: path=/tmp/raft298372192/snapshots/3-13-1776852554029.tmp 2026-04-22T10:09:14.038Z [INFO] reaping snapshot: path=/tmp/raft298372192/snapshots/3-11-1776852554010 2026-04-22T10:09:14.038Z [INFO] creating new snapshot: path=/tmp/raft298372192/snapshots/3-14-1776852554038.tmp 2026-04-22T10:09:14.047Z [INFO] reaping snapshot: path=/tmp/raft298372192/snapshots/3-12-1776852554019 --- PASS: TestFileSS_Retention (0.05s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering 2026-04-22T10:09:14.049Z [INFO] creating new snapshot: path=/tmp/raft174869081/snapshots/5-130350-1776852554049.tmp 2026-04-22T10:09:14.057Z [INFO] creating new snapshot: path=/tmp/raft174869081/snapshots/36-204917-1776852554057.tmp --- PASS: TestFileSS_Ordering (0.02s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestRaft_RestartFollower_LongInitialHeartbeat --- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestLogCacheWithBackendStoreError --- PASS: TestLogCacheWithBackendStoreError (0.00s) === RUN TestOldestLog === RUN TestOldestLog/empty_logs === RUN TestOldestLog/simple_case --- PASS: TestOldestLog (0.00s) --- PASS: TestOldestLog/empty_logs (0.00s) --- PASS: TestOldestLog/simple_case (0.00s) === RUN TestEmitsLogStoreMetrics --- PASS: TestEmitsLogStoreMetrics (0.01s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.00s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams --- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.02s) === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 2026-04-22T10:09:14.164Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 2026-04-22T10:09:14.216Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" 2026-04-22T10:09:14.268Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s) === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:839: countingWriter: 2026-04-22T10:09:14.274Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-22T10:09:14.279Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-22T10:09:14.290Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-22T10:09:14.310Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-22T10:09:14.351Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-22T10:09:14.431Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-22T10:09:14.592Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-22T10:09:14.912Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-22T10:09:15.553Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-22T10:09:16.555Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-04-22T10:09:17.557Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop 2026-04-22T10:09:18.276Z [INFO] server-b0dd2471-59a7-8e19-9e66-0767b24b0baa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b0dd2471-59a7-8e19-9e66-0767b24b0baa Address:b0dd2471-59a7-8e19-9e66-0767b24b0baa}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown 2026-04-22T10:09:18.277Z [INFO] server-9a33264c-a7df-5903-3d05-7a8e92cc4023: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9a33264c-a7df-5903-3d05-7a8e92cc4023 Address:9a33264c-a7df-5903-3d05-7a8e92cc4023}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap 2026-04-22T10:09:18.278Z [INFO] server-52d0754e-b7a3-3283-75d8-98a76ed0eb27: initial configuration: index=0 servers=[] 2026-04-22T10:09:18.278Z [INFO] server-a9238553-d59b-9369-b4a2-be728263ba87: initial configuration: index=0 servers=[] 2026-04-22T10:09:18.278Z [INFO] server-52d0754e-b7a3-3283-75d8-98a76ed0eb27: entering follower state: follower="Node at 52d0754e-b7a3-3283-75d8-98a76ed0eb27 [Follower]" leader-address= leader-id= 2026-04-22T10:09:18.278Z [INFO] server-a9238553-d59b-9369-b4a2-be728263ba87: entering follower state: follower="Node at a9238553-d59b-9369-b4a2-be728263ba87 [Follower]" leader-address= leader-id= 2026-04-22T10:09:18.278Z [INFO] server-05ec8efb-eeb7-eca7-3c4d-c94ed8900f8e: initial configuration: index=0 servers=[] 2026-04-22T10:09:18.278Z [INFO] server-05ec8efb-eeb7-eca7-3c4d-c94ed8900f8e: entering follower state: follower="Node at 05ec8efb-eeb7-eca7-3c4d-c94ed8900f8e [Follower]" leader-address= leader-id= 2026-04-22T10:09:18.278Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:18.332Z [WARN] server-a9238553-d59b-9369-b4a2-be728263ba87: no known peers, aborting election 2026-04-22T10:09:18.341Z [WARN] server-05ec8efb-eeb7-eca7-3c4d-c94ed8900f8e: no known peers, aborting election 2026-04-22T10:09:18.356Z [WARN] server-52d0754e-b7a3-3283-75d8-98a76ed0eb27: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:18.356Z [INFO] server-52d0754e-b7a3-3283-75d8-98a76ed0eb27: entering candidate state: node="Node at 52d0754e-b7a3-3283-75d8-98a76ed0eb27 [Candidate]" term=2 2026-04-22T10:09:18.356Z [INFO] server-52d0754e-b7a3-3283-75d8-98a76ed0eb27: election won: term=2 tally=2 2026-04-22T10:09:18.356Z [INFO] server-52d0754e-b7a3-3283-75d8-98a76ed0eb27: entering leader state: leader="Node at 52d0754e-b7a3-3283-75d8-98a76ed0eb27 [Leader]" 2026-04-22T10:09:18.356Z [INFO] server-52d0754e-b7a3-3283-75d8-98a76ed0eb27: added peer, starting replication: peer=server-a9238553-d59b-9369-b4a2-be728263ba87 2026-04-22T10:09:18.356Z [INFO] server-52d0754e-b7a3-3283-75d8-98a76ed0eb27: added peer, starting replication: peer=server-05ec8efb-eeb7-eca7-3c4d-c94ed8900f8e 2026-04-22T10:09:18.356Z [WARN] server-a9238553-d59b-9369-b4a2-be728263ba87: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2026-04-22T10:09:18.357Z [WARN] server-52d0754e-b7a3-3283-75d8-98a76ed0eb27: appendEntries rejected, sending older logs: peer="{Voter server-a9238553-d59b-9369-b4a2-be728263ba87 a9238553-d59b-9369-b4a2-be728263ba87}" next=1 2026-04-22T10:09:18.357Z [WARN] server-05ec8efb-eeb7-eca7-3c4d-c94ed8900f8e: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2026-04-22T10:09:18.357Z [WARN] server-52d0754e-b7a3-3283-75d8-98a76ed0eb27: appendEntries rejected, sending older logs: peer="{Voter server-05ec8efb-eeb7-eca7-3c4d-c94ed8900f8e 05ec8efb-eeb7-eca7-3c4d-c94ed8900f8e}" next=1 2026-04-22T10:09:18.357Z [INFO] server-52d0754e-b7a3-3283-75d8-98a76ed0eb27: pipelining replication: peer="{Voter server-a9238553-d59b-9369-b4a2-be728263ba87 a9238553-d59b-9369-b4a2-be728263ba87}" 2026-04-22T10:09:18.357Z [INFO] server-52d0754e-b7a3-3283-75d8-98a76ed0eb27: pipelining replication: peer="{Voter server-05ec8efb-eeb7-eca7-3c4d-c94ed8900f8e 05ec8efb-eeb7-eca7-3c4d-c94ed8900f8e}" 2026-04-22T10:09:18.463Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:18.357595267 +0000 UTC m=+4.381391413 (2 nodes), 79.212234ms from start of poll, 79.48729ms from cluster start. Timeout at 2026-04-22 10:09:18.462661232 +0000 UTC m=+4.486457324, 105.065911ms after stability 2026-04-22T10:09:18.463Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:18.569Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:18.463510217 +0000 UTC m=+4.487306305 (1 nodes), 699ns from start of poll, 185.402182ms from cluster start. Timeout at 2026-04-22 10:09:18.568513782 +0000 UTC m=+4.592309868, 105.003563ms after stability 2026-04-22T10:09:18.580Z [INFO] server-52d0754e-b7a3-3283-75d8-98a76ed0eb27: aborting pipeline replication: peer="{Voter server-05ec8efb-eeb7-eca7-3c4d-c94ed8900f8e 05ec8efb-eeb7-eca7-3c4d-c94ed8900f8e}" 2026-04-22T10:09:18.580Z [INFO] server-52d0754e-b7a3-3283-75d8-98a76ed0eb27: aborting pipeline replication: peer="{Voter server-a9238553-d59b-9369-b4a2-be728263ba87 a9238553-d59b-9369-b4a2-be728263ba87}" --- PASS: TestRaft_LiveBootstrap (0.30s) === RUN TestRaft_LiveBootstrap_From_NonVoter 2026-04-22T10:09:18.581Z [INFO] server-227a4b70-2d39-e02e-9ac7-70f56c7eeb64: initial configuration: index=0 servers=[] 2026-04-22T10:09:18.581Z [INFO] server-227a4b70-2d39-e02e-9ac7-70f56c7eeb64: entering follower state: follower="Node at 227a4b70-2d39-e02e-9ac7-70f56c7eeb64 [Follower]" leader-address= leader-id= 2026-04-22T10:09:18.581Z [INFO] server-5e72e0d4-8d44-6b0b-f9f1-e9783a6812cb: initial configuration: index=0 servers=[] 2026-04-22T10:09:18.581Z [INFO] server-5e72e0d4-8d44-6b0b-f9f1-e9783a6812cb: entering follower state: follower="Node at 5e72e0d4-8d44-6b0b-f9f1-e9783a6812cb [Follower]" leader-address= leader-id= --- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s) === RUN TestRaft_RecoverCluster_NoState 2026-04-22T10:09:18.581Z [INFO] server-2d07da6e-b1c4-7aa6-2b0d-7ae5965707fc: initial configuration: index=0 servers=[] 2026-04-22T10:09:18.581Z [INFO] server-2d07da6e-b1c4-7aa6-2b0d-7ae5965707fc: entering follower state: follower="Node at 2d07da6e-b1c4-7aa6-2b0d-7ae5965707fc [Follower]" leader-address= leader-id= --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster === RUN TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs 2026-04-22T10:09:18.582Z [INFO] server-3244534b-f4e9-d8a2-ad82-af518b84b638: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3244534b-f4e9-d8a2-ad82-af518b84b638 Address:3244534b-f4e9-d8a2-ad82-af518b84b638} {Suffrage:Voter ID:server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8 Address:5b4c6018-176a-09c3-0a51-b3aa89ef2ae8} {Suffrage:Voter ID:server-6b596672-1647-4512-b963-f14a6fa59b74 Address:6b596672-1647-4512-b963-f14a6fa59b74}]" 2026-04-22T10:09:18.582Z [INFO] server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3244534b-f4e9-d8a2-ad82-af518b84b638 Address:3244534b-f4e9-d8a2-ad82-af518b84b638} {Suffrage:Voter ID:server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8 Address:5b4c6018-176a-09c3-0a51-b3aa89ef2ae8} {Suffrage:Voter ID:server-6b596672-1647-4512-b963-f14a6fa59b74 Address:6b596672-1647-4512-b963-f14a6fa59b74}]" 2026-04-22T10:09:18.582Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3244534b-f4e9-d8a2-ad82-af518b84b638 Address:3244534b-f4e9-d8a2-ad82-af518b84b638} {Suffrage:Voter ID:server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8 Address:5b4c6018-176a-09c3-0a51-b3aa89ef2ae8} {Suffrage:Voter ID:server-6b596672-1647-4512-b963-f14a6fa59b74 Address:6b596672-1647-4512-b963-f14a6fa59b74}]" 2026-04-22T10:09:18.582Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:18.582Z [INFO] server-3244534b-f4e9-d8a2-ad82-af518b84b638: entering follower state: follower="Node at 3244534b-f4e9-d8a2-ad82-af518b84b638 [Follower]" leader-address= leader-id= 2026-04-22T10:09:18.583Z [INFO] server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8: entering follower state: follower="Node at 5b4c6018-176a-09c3-0a51-b3aa89ef2ae8 [Follower]" leader-address= leader-id= 2026-04-22T10:09:18.583Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: entering follower state: follower="Node at 6b596672-1647-4512-b963-f14a6fa59b74 [Follower]" leader-address= leader-id= 2026-04-22T10:09:18.635Z [WARN] server-6b596672-1647-4512-b963-f14a6fa59b74: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:18.635Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: entering candidate state: node="Node at 6b596672-1647-4512-b963-f14a6fa59b74 [Candidate]" term=2 2026-04-22T10:09:18.636Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: election won: term=2 tally=2 2026-04-22T10:09:18.636Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: entering leader state: leader="Node at 6b596672-1647-4512-b963-f14a6fa59b74 [Leader]" 2026-04-22T10:09:18.636Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: added peer, starting replication: peer=server-3244534b-f4e9-d8a2-ad82-af518b84b638 2026-04-22T10:09:18.636Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: added peer, starting replication: peer=server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8 2026-04-22T10:09:18.636Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: pipelining replication: peer="{Voter server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8 5b4c6018-176a-09c3-0a51-b3aa89ef2ae8}" 2026-04-22T10:09:18.636Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: pipelining replication: peer="{Voter server-3244534b-f4e9-d8a2-ad82-af518b84b638 3244534b-f4e9-d8a2-ad82-af518b84b638}" 2026-04-22T10:09:18.741Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:18.636050015 +0000 UTC m=+4.659846114 (1 nodes), 53.06919ms from start of poll, 53.390068ms from cluster start. Timeout at 2026-04-22 10:09:18.741053327 +0000 UTC m=+4.764849410, 105.003296ms after stability 2026-04-22T10:09:18.741Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: aborting pipeline replication: peer="{Voter server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8 5b4c6018-176a-09c3-0a51-b3aa89ef2ae8}" 2026-04-22T10:09:18.741Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: aborting pipeline replication: peer="{Voter server-3244534b-f4e9-d8a2-ad82-af518b84b638 3244534b-f4e9-d8a2-ad82-af518b84b638}" 2026-04-22T10:09:18.741Z [INFO] creating new snapshot: path=/tmp/raft3187329625/snapshots/2-2-1776852558741.tmp 2026-04-22T10:09:18.742Z [INFO] server-3244534b-f4e9-d8a2-ad82-af518b84b638: starting restore from snapshot: id=2-2-1776852558741 last-index=2 last-term=2 size-in-bytes=1 2026-04-22T10:09:18.742Z [INFO] server-3244534b-f4e9-d8a2-ad82-af518b84b638: snapshot restore progress: id=2-2-1776852558741 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2026-04-22T10:09:18.742Z [INFO] server-3244534b-f4e9-d8a2-ad82-af518b84b638: restored from snapshot: id=2-2-1776852558741 last-index=2 last-term=2 size-in-bytes=1 2026-04-22T10:09:18.742Z [INFO] server-3244534b-f4e9-d8a2-ad82-af518b84b638: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3244534b-f4e9-d8a2-ad82-af518b84b638 Address:3244534b-f4e9-d8a2-ad82-af518b84b638} {Suffrage:Voter ID:server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8 Address:5b4c6018-176a-09c3-0a51-b3aa89ef2ae8} {Suffrage:Voter ID:server-6b596672-1647-4512-b963-f14a6fa59b74 Address:6b596672-1647-4512-b963-f14a6fa59b74}]" 2026-04-22T10:09:18.742Z [INFO] server-3244534b-f4e9-d8a2-ad82-af518b84b638: entering follower state: follower="Node at 3244534b-f4e9-d8a2-ad82-af518b84b638 [Follower]" leader-address= leader-id= 2026-04-22T10:09:18.742Z [INFO] creating new snapshot: path=/tmp/raft201431586/snapshots/2-2-1776852558742.tmp 2026-04-22T10:09:18.742Z [INFO] server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8: starting restore from snapshot: id=2-2-1776852558742 last-index=2 last-term=2 size-in-bytes=1 2026-04-22T10:09:18.742Z [INFO] server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8: snapshot restore progress: id=2-2-1776852558742 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2026-04-22T10:09:18.742Z [INFO] server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8: restored from snapshot: id=2-2-1776852558742 last-index=2 last-term=2 size-in-bytes=1 2026-04-22T10:09:18.742Z [INFO] server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3244534b-f4e9-d8a2-ad82-af518b84b638 Address:3244534b-f4e9-d8a2-ad82-af518b84b638} {Suffrage:Voter ID:server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8 Address:5b4c6018-176a-09c3-0a51-b3aa89ef2ae8} {Suffrage:Voter ID:server-6b596672-1647-4512-b963-f14a6fa59b74 Address:6b596672-1647-4512-b963-f14a6fa59b74}]" 2026-04-22T10:09:18.743Z [INFO] server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8: entering follower state: follower="Node at 5b4c6018-176a-09c3-0a51-b3aa89ef2ae8 [Follower]" leader-address= leader-id= 2026-04-22T10:09:18.743Z [INFO] creating new snapshot: path=/tmp/raft3849371788/snapshots/2-2-1776852558743.tmp 2026-04-22T10:09:18.744Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: starting restore from snapshot: id=2-2-1776852558743 last-index=2 last-term=2 size-in-bytes=1 2026-04-22T10:09:18.744Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: snapshot restore progress: id=2-2-1776852558743 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2026-04-22T10:09:18.744Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: restored from snapshot: id=2-2-1776852558743 last-index=2 last-term=2 size-in-bytes=1 2026-04-22T10:09:18.744Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3244534b-f4e9-d8a2-ad82-af518b84b638 Address:3244534b-f4e9-d8a2-ad82-af518b84b638} {Suffrage:Voter ID:server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8 Address:5b4c6018-176a-09c3-0a51-b3aa89ef2ae8} {Suffrage:Voter ID:server-6b596672-1647-4512-b963-f14a6fa59b74 Address:6b596672-1647-4512-b963-f14a6fa59b74}]" 2026-04-22T10:09:18.744Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: entering follower state: follower="Node at 6b596672-1647-4512-b963-f14a6fa59b74 [Follower]" leader-address= leader-id= 2026-04-22T10:09:18.809Z [WARN] server-6b596672-1647-4512-b963-f14a6fa59b74: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:18.809Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: entering candidate state: node="Node at 6b596672-1647-4512-b963-f14a6fa59b74 [Candidate]" term=3 2026-04-22T10:09:18.809Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: election won: term=3 tally=2 2026-04-22T10:09:18.809Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: entering leader state: leader="Node at 6b596672-1647-4512-b963-f14a6fa59b74 [Leader]" 2026-04-22T10:09:18.809Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: added peer, starting replication: peer=server-3244534b-f4e9-d8a2-ad82-af518b84b638 2026-04-22T10:09:18.809Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: added peer, starting replication: peer=server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8 2026-04-22T10:09:18.810Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: pipelining replication: peer="{Voter server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8 5b4c6018-176a-09c3-0a51-b3aa89ef2ae8}" 2026-04-22T10:09:18.810Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: pipelining replication: peer="{Voter server-3244534b-f4e9-d8a2-ad82-af518b84b638 3244534b-f4e9-d8a2-ad82-af518b84b638}" 2026-04-22T10:09:19.060Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:19.166Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:19.060161906 +0000 UTC m=+5.083957992 (1 nodes), 23.028µs from start of poll, 477.501946ms from cluster start. Timeout at 2026-04-22 10:09:19.165167022 +0000 UTC m=+5.188963130, 105.005138ms after stability 2026-04-22T10:09:19.166Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: aborting pipeline replication: peer="{Voter server-5b4c6018-176a-09c3-0a51-b3aa89ef2ae8 5b4c6018-176a-09c3-0a51-b3aa89ef2ae8}" 2026-04-22T10:09:19.166Z [INFO] server-6b596672-1647-4512-b963-f14a6fa59b74: aborting pipeline replication: peer="{Voter server-3244534b-f4e9-d8a2-ad82-af518b84b638 3244534b-f4e9-d8a2-ad82-af518b84b638}" === RUN TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs 2026-04-22T10:09:19.342Z [INFO] server-77176985-871c-76cf-c0e7-d004dc150816: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77176985-871c-76cf-c0e7-d004dc150816 Address:77176985-871c-76cf-c0e7-d004dc150816} {Suffrage:Voter ID:server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f Address:8ffb20ef-6433-8bfc-fade-d5a74bffbe3f} {Suffrage:Voter ID:server-f5f56c4a-e57b-3330-1976-085698632c76 Address:f5f56c4a-e57b-3330-1976-085698632c76}]" 2026-04-22T10:09:19.342Z [INFO] server-77176985-871c-76cf-c0e7-d004dc150816: entering follower state: follower="Node at 77176985-871c-76cf-c0e7-d004dc150816 [Follower]" leader-address= leader-id= 2026-04-22T10:09:19.342Z [INFO] server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77176985-871c-76cf-c0e7-d004dc150816 Address:77176985-871c-76cf-c0e7-d004dc150816} {Suffrage:Voter ID:server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f Address:8ffb20ef-6433-8bfc-fade-d5a74bffbe3f} {Suffrage:Voter ID:server-f5f56c4a-e57b-3330-1976-085698632c76 Address:f5f56c4a-e57b-3330-1976-085698632c76}]" 2026-04-22T10:09:19.342Z [INFO] server-f5f56c4a-e57b-3330-1976-085698632c76: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77176985-871c-76cf-c0e7-d004dc150816 Address:77176985-871c-76cf-c0e7-d004dc150816} {Suffrage:Voter ID:server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f Address:8ffb20ef-6433-8bfc-fade-d5a74bffbe3f} {Suffrage:Voter ID:server-f5f56c4a-e57b-3330-1976-085698632c76 Address:f5f56c4a-e57b-3330-1976-085698632c76}]" 2026-04-22T10:09:19.342Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:19.342Z [INFO] server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f: entering follower state: follower="Node at 8ffb20ef-6433-8bfc-fade-d5a74bffbe3f [Follower]" leader-address= leader-id= 2026-04-22T10:09:19.342Z [INFO] server-f5f56c4a-e57b-3330-1976-085698632c76: entering follower state: follower="Node at f5f56c4a-e57b-3330-1976-085698632c76 [Follower]" leader-address= leader-id= 2026-04-22T10:09:19.398Z [WARN] server-f5f56c4a-e57b-3330-1976-085698632c76: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:19.398Z [INFO] server-f5f56c4a-e57b-3330-1976-085698632c76: entering candidate state: node="Node at f5f56c4a-e57b-3330-1976-085698632c76 [Candidate]" term=2 2026-04-22T10:09:19.398Z [INFO] server-f5f56c4a-e57b-3330-1976-085698632c76: election won: term=2 tally=2 2026-04-22T10:09:19.398Z [INFO] server-f5f56c4a-e57b-3330-1976-085698632c76: entering leader state: leader="Node at f5f56c4a-e57b-3330-1976-085698632c76 [Leader]" 2026-04-22T10:09:19.398Z [INFO] server-f5f56c4a-e57b-3330-1976-085698632c76: added peer, starting replication: peer=server-77176985-871c-76cf-c0e7-d004dc150816 2026-04-22T10:09:19.398Z [INFO] server-f5f56c4a-e57b-3330-1976-085698632c76: added peer, starting replication: peer=server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f 2026-04-22T10:09:19.398Z [INFO] server-f5f56c4a-e57b-3330-1976-085698632c76: pipelining replication: peer="{Voter server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f 8ffb20ef-6433-8bfc-fade-d5a74bffbe3f}" 2026-04-22T10:09:19.399Z [INFO] server-f5f56c4a-e57b-3330-1976-085698632c76: pipelining replication: peer="{Voter server-77176985-871c-76cf-c0e7-d004dc150816 77176985-871c-76cf-c0e7-d004dc150816}" 2026-04-22T10:09:19.504Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:19.399083805 +0000 UTC m=+5.422879920 (1 nodes), 56.483022ms from start of poll, 57.005586ms from cluster start. Timeout at 2026-04-22 10:09:19.504084536 +0000 UTC m=+5.527880618, 105.000698ms after stability 2026-04-22T10:09:19.505Z [INFO] server-f5f56c4a-e57b-3330-1976-085698632c76: aborting pipeline replication: peer="{Voter server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f 8ffb20ef-6433-8bfc-fade-d5a74bffbe3f}" 2026-04-22T10:09:19.505Z [INFO] server-f5f56c4a-e57b-3330-1976-085698632c76: aborting pipeline replication: peer="{Voter server-77176985-871c-76cf-c0e7-d004dc150816 77176985-871c-76cf-c0e7-d004dc150816}" 2026-04-22T10:09:19.505Z [INFO] creating new snapshot: path=/tmp/raft3727105579/snapshots/2-6-1776852559505.tmp 2026-04-22T10:09:19.549Z [INFO] server-77176985-871c-76cf-c0e7-d004dc150816: starting restore from snapshot: id=2-6-1776852559505 last-index=6 last-term=2 size-in-bytes=25 2026-04-22T10:09:19.549Z [INFO] server-77176985-871c-76cf-c0e7-d004dc150816: snapshot restore progress: id=2-6-1776852559505 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2026-04-22T10:09:19.549Z [INFO] server-77176985-871c-76cf-c0e7-d004dc150816: restored from snapshot: id=2-6-1776852559505 last-index=6 last-term=2 size-in-bytes=25 2026-04-22T10:09:19.550Z [INFO] server-77176985-871c-76cf-c0e7-d004dc150816: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77176985-871c-76cf-c0e7-d004dc150816 Address:77176985-871c-76cf-c0e7-d004dc150816} {Suffrage:Voter ID:server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f Address:8ffb20ef-6433-8bfc-fade-d5a74bffbe3f} {Suffrage:Voter ID:server-f5f56c4a-e57b-3330-1976-085698632c76 Address:f5f56c4a-e57b-3330-1976-085698632c76}]" 2026-04-22T10:09:19.550Z [INFO] creating new snapshot: path=/tmp/raft3326683447/snapshots/2-6-1776852559550.tmp 2026-04-22T10:09:19.550Z [INFO] server-77176985-871c-76cf-c0e7-d004dc150816: entering follower state: follower="Node at 77176985-871c-76cf-c0e7-d004dc150816 [Follower]" leader-address= leader-id= 2026-04-22T10:09:19.550Z [INFO] server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f: starting restore from snapshot: id=2-6-1776852559550 last-index=6 last-term=2 size-in-bytes=25 2026-04-22T10:09:19.550Z [INFO] server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f: snapshot restore progress: id=2-6-1776852559550 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2026-04-22T10:09:19.550Z [INFO] server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f: restored from snapshot: id=2-6-1776852559550 last-index=6 last-term=2 size-in-bytes=25 2026-04-22T10:09:19.550Z [INFO] server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77176985-871c-76cf-c0e7-d004dc150816 Address:77176985-871c-76cf-c0e7-d004dc150816} {Suffrage:Voter ID:server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f Address:8ffb20ef-6433-8bfc-fade-d5a74bffbe3f} {Suffrage:Voter ID:server-f5f56c4a-e57b-3330-1976-085698632c76 Address:f5f56c4a-e57b-3330-1976-085698632c76}]" 2026-04-22T10:09:19.550Z [INFO] creating new snapshot: path=/tmp/raft1487374809/snapshots/2-6-1776852559550.tmp 2026-04-22T10:09:19.550Z [INFO] server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f: entering follower state: follower="Node at 8ffb20ef-6433-8bfc-fade-d5a74bffbe3f [Follower]" leader-address= leader-id= 2026-04-22T10:09:19.606Z [WARN] server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:19.606Z [INFO] server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f: entering candidate state: node="Node at 8ffb20ef-6433-8bfc-fade-d5a74bffbe3f [Candidate]" term=3 2026-04-22T10:09:19.606Z [ERROR] server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f: failed to make requestVote RPC: target="{Voter server-f5f56c4a-e57b-3330-1976-085698632c76 f5f56c4a-e57b-3330-1976-085698632c76}" error="failed to connect to peer: f5f56c4a-e57b-3330-1976-085698632c76" term=3 2026-04-22T10:09:19.606Z [ERROR] server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f: failed to make requestVote RPC: target="{Voter server-77176985-871c-76cf-c0e7-d004dc150816 77176985-871c-76cf-c0e7-d004dc150816}" error="failed to connect to peer: 77176985-871c-76cf-c0e7-d004dc150816" term=3 2026-04-22T10:09:19.636Z [WARN] server-77176985-871c-76cf-c0e7-d004dc150816: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:19.636Z [INFO] server-77176985-871c-76cf-c0e7-d004dc150816: entering candidate state: node="Node at 77176985-871c-76cf-c0e7-d004dc150816 [Candidate]" term=3 2026-04-22T10:09:19.636Z [ERROR] server-77176985-871c-76cf-c0e7-d004dc150816: failed to make requestVote RPC: target="{Voter server-f5f56c4a-e57b-3330-1976-085698632c76 f5f56c4a-e57b-3330-1976-085698632c76}" error="failed to connect to peer: f5f56c4a-e57b-3330-1976-085698632c76" term=3 2026-04-22T10:09:19.636Z [ERROR] server-77176985-871c-76cf-c0e7-d004dc150816: failed to make requestVote RPC: target="{Voter server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f 8ffb20ef-6433-8bfc-fade-d5a74bffbe3f}" error="failed to connect to peer: 8ffb20ef-6433-8bfc-fade-d5a74bffbe3f" term=3 2026-04-22T10:09:19.672Z [WARN] server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f: Election timeout reached, restarting election 2026-04-22T10:09:19.672Z [INFO] server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f: entering candidate state: node="Node at 8ffb20ef-6433-8bfc-fade-d5a74bffbe3f [Candidate]" term=4 2026-04-22T10:09:19.672Z [ERROR] server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f: failed to make requestVote RPC: target="{Voter server-f5f56c4a-e57b-3330-1976-085698632c76 f5f56c4a-e57b-3330-1976-085698632c76}" error="failed to connect to peer: f5f56c4a-e57b-3330-1976-085698632c76" term=4 2026-04-22T10:09:19.672Z [ERROR] server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f: failed to make requestVote RPC: target="{Voter server-77176985-871c-76cf-c0e7-d004dc150816 77176985-871c-76cf-c0e7-d004dc150816}" error="failed to connect to peer: 77176985-871c-76cf-c0e7-d004dc150816" term=4 2026-04-22T10:09:19.689Z [INFO] server-f5f56c4a-e57b-3330-1976-085698632c76: starting restore from snapshot: id=2-6-1776852559550 last-index=6 last-term=2 size-in-bytes=25 2026-04-22T10:09:19.689Z [INFO] server-f5f56c4a-e57b-3330-1976-085698632c76: snapshot restore progress: id=2-6-1776852559550 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2026-04-22T10:09:19.689Z [INFO] server-f5f56c4a-e57b-3330-1976-085698632c76: restored from snapshot: id=2-6-1776852559550 last-index=6 last-term=2 size-in-bytes=25 2026-04-22T10:09:19.689Z [INFO] server-f5f56c4a-e57b-3330-1976-085698632c76: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77176985-871c-76cf-c0e7-d004dc150816 Address:77176985-871c-76cf-c0e7-d004dc150816} {Suffrage:Voter ID:server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f Address:8ffb20ef-6433-8bfc-fade-d5a74bffbe3f} {Suffrage:Voter ID:server-f5f56c4a-e57b-3330-1976-085698632c76 Address:f5f56c4a-e57b-3330-1976-085698632c76}]" 2026-04-22T10:09:19.689Z [INFO] server-f5f56c4a-e57b-3330-1976-085698632c76: entering follower state: follower="Node at f5f56c4a-e57b-3330-1976-085698632c76 [Follower]" leader-address= leader-id= 2026-04-22T10:09:19.716Z [WARN] server-77176985-871c-76cf-c0e7-d004dc150816: Election timeout reached, restarting election 2026-04-22T10:09:19.716Z [INFO] server-77176985-871c-76cf-c0e7-d004dc150816: entering candidate state: node="Node at 77176985-871c-76cf-c0e7-d004dc150816 [Candidate]" term=4 2026-04-22T10:09:19.716Z [INFO] server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f: duplicate requestVote for same term: term=4 2026-04-22T10:09:19.716Z [INFO] server-77176985-871c-76cf-c0e7-d004dc150816: election won: term=4 tally=2 2026-04-22T10:09:19.716Z [INFO] server-77176985-871c-76cf-c0e7-d004dc150816: entering leader state: leader="Node at 77176985-871c-76cf-c0e7-d004dc150816 [Leader]" 2026-04-22T10:09:19.716Z [INFO] server-77176985-871c-76cf-c0e7-d004dc150816: added peer, starting replication: peer=server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f 2026-04-22T10:09:19.716Z [INFO] server-77176985-871c-76cf-c0e7-d004dc150816: added peer, starting replication: peer=server-f5f56c4a-e57b-3330-1976-085698632c76 2026-04-22T10:09:19.716Z [INFO] server-77176985-871c-76cf-c0e7-d004dc150816: pipelining replication: peer="{Voter server-f5f56c4a-e57b-3330-1976-085698632c76 f5f56c4a-e57b-3330-1976-085698632c76}" 2026-04-22T10:09:19.716Z [INFO] server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f: entering follower state: follower="Node at 8ffb20ef-6433-8bfc-fade-d5a74bffbe3f [Follower]" leader-address=77176985-871c-76cf-c0e7-d004dc150816 leader-id=server-77176985-871c-76cf-c0e7-d004dc150816 2026-04-22T10:09:19.716Z [INFO] server-77176985-871c-76cf-c0e7-d004dc150816: pipelining replication: peer="{Voter server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f 8ffb20ef-6433-8bfc-fade-d5a74bffbe3f}" 2026-04-22T10:09:20.005Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:20.110Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:20.005129374 +0000 UTC m=+6.028925477 (1 nodes), 26.098µs from start of poll, 663.051143ms from cluster start. Timeout at 2026-04-22 10:09:20.11013378 +0000 UTC m=+6.133929878, 105.004401ms after stability 2026-04-22T10:09:20.110Z [INFO] server-77176985-871c-76cf-c0e7-d004dc150816: aborting pipeline replication: peer="{Voter server-f5f56c4a-e57b-3330-1976-085698632c76 f5f56c4a-e57b-3330-1976-085698632c76}" 2026-04-22T10:09:20.110Z [INFO] server-77176985-871c-76cf-c0e7-d004dc150816: aborting pipeline replication: peer="{Voter server-8ffb20ef-6433-8bfc-fade-d5a74bffbe3f 8ffb20ef-6433-8bfc-fade-d5a74bffbe3f}" === RUN TestRaft_RecoverCluster/snapshot,_with_trailing_logs 2026-04-22T10:09:20.111Z [INFO] server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c Address:e59ba41c-10e0-f573-f2d8-5b04f955dd0c} {Suffrage:Voter ID:server-5949481e-b89e-a129-4e2d-f55f10cdcf93 Address:5949481e-b89e-a129-4e2d-f55f10cdcf93} {Suffrage:Voter ID:server-c1426241-9fa7-6427-cf86-9b86bc9a8872 Address:c1426241-9fa7-6427-cf86-9b86bc9a8872}]" 2026-04-22T10:09:20.111Z [INFO] server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c: entering follower state: follower="Node at e59ba41c-10e0-f573-f2d8-5b04f955dd0c [Follower]" leader-address= leader-id= 2026-04-22T10:09:20.111Z [INFO] server-5949481e-b89e-a129-4e2d-f55f10cdcf93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c Address:e59ba41c-10e0-f573-f2d8-5b04f955dd0c} {Suffrage:Voter ID:server-5949481e-b89e-a129-4e2d-f55f10cdcf93 Address:5949481e-b89e-a129-4e2d-f55f10cdcf93} {Suffrage:Voter ID:server-c1426241-9fa7-6427-cf86-9b86bc9a8872 Address:c1426241-9fa7-6427-cf86-9b86bc9a8872}]" 2026-04-22T10:09:20.111Z [INFO] server-5949481e-b89e-a129-4e2d-f55f10cdcf93: entering follower state: follower="Node at 5949481e-b89e-a129-4e2d-f55f10cdcf93 [Follower]" leader-address= leader-id= 2026-04-22T10:09:20.112Z [INFO] server-c1426241-9fa7-6427-cf86-9b86bc9a8872: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c Address:e59ba41c-10e0-f573-f2d8-5b04f955dd0c} {Suffrage:Voter ID:server-5949481e-b89e-a129-4e2d-f55f10cdcf93 Address:5949481e-b89e-a129-4e2d-f55f10cdcf93} {Suffrage:Voter ID:server-c1426241-9fa7-6427-cf86-9b86bc9a8872 Address:c1426241-9fa7-6427-cf86-9b86bc9a8872}]" 2026-04-22T10:09:20.112Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:20.112Z [INFO] server-c1426241-9fa7-6427-cf86-9b86bc9a8872: entering follower state: follower="Node at c1426241-9fa7-6427-cf86-9b86bc9a8872 [Follower]" leader-address= leader-id= 2026-04-22T10:09:20.168Z [WARN] server-5949481e-b89e-a129-4e2d-f55f10cdcf93: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:20.168Z [INFO] server-5949481e-b89e-a129-4e2d-f55f10cdcf93: entering candidate state: node="Node at 5949481e-b89e-a129-4e2d-f55f10cdcf93 [Candidate]" term=2 2026-04-22T10:09:20.168Z [INFO] server-5949481e-b89e-a129-4e2d-f55f10cdcf93: election won: term=2 tally=2 2026-04-22T10:09:20.168Z [INFO] server-5949481e-b89e-a129-4e2d-f55f10cdcf93: entering leader state: leader="Node at 5949481e-b89e-a129-4e2d-f55f10cdcf93 [Leader]" 2026-04-22T10:09:20.168Z [INFO] server-5949481e-b89e-a129-4e2d-f55f10cdcf93: added peer, starting replication: peer=server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c 2026-04-22T10:09:20.168Z [INFO] server-5949481e-b89e-a129-4e2d-f55f10cdcf93: added peer, starting replication: peer=server-c1426241-9fa7-6427-cf86-9b86bc9a8872 2026-04-22T10:09:20.168Z [INFO] server-5949481e-b89e-a129-4e2d-f55f10cdcf93: pipelining replication: peer="{Voter server-c1426241-9fa7-6427-cf86-9b86bc9a8872 c1426241-9fa7-6427-cf86-9b86bc9a8872}" 2026-04-22T10:09:20.168Z [INFO] server-5949481e-b89e-a129-4e2d-f55f10cdcf93: pipelining replication: peer="{Voter server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c e59ba41c-10e0-f573-f2d8-5b04f955dd0c}" 2026-04-22T10:09:20.273Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:20.168310757 +0000 UTC m=+6.192106844 (1 nodes), 56.230964ms from start of poll, 56.638213ms from cluster start. Timeout at 2026-04-22 10:09:20.273312006 +0000 UTC m=+6.297108091, 105.001247ms after stability 2026-04-22T10:09:20.276Z [INFO] server-5949481e-b89e-a129-4e2d-f55f10cdcf93: aborting pipeline replication: peer="{Voter server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c e59ba41c-10e0-f573-f2d8-5b04f955dd0c}" 2026-04-22T10:09:20.276Z [INFO] server-5949481e-b89e-a129-4e2d-f55f10cdcf93: aborting pipeline replication: peer="{Voter server-c1426241-9fa7-6427-cf86-9b86bc9a8872 c1426241-9fa7-6427-cf86-9b86bc9a8872}" 2026-04-22T10:09:20.276Z [INFO] creating new snapshot: path=/tmp/raft3522790038/snapshots/2-27-1776852560276.tmp 2026-04-22T10:09:20.279Z [INFO] server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c: starting restore from snapshot: id=2-27-1776852560276 last-index=27 last-term=2 size-in-bytes=168 2026-04-22T10:09:20.279Z [INFO] server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c: snapshot restore progress: id=2-27-1776852560276 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2026-04-22T10:09:20.279Z [INFO] server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c: restored from snapshot: id=2-27-1776852560276 last-index=27 last-term=2 size-in-bytes=168 2026-04-22T10:09:20.279Z [INFO] server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c Address:e59ba41c-10e0-f573-f2d8-5b04f955dd0c} {Suffrage:Voter ID:server-5949481e-b89e-a129-4e2d-f55f10cdcf93 Address:5949481e-b89e-a129-4e2d-f55f10cdcf93} {Suffrage:Voter ID:server-c1426241-9fa7-6427-cf86-9b86bc9a8872 Address:c1426241-9fa7-6427-cf86-9b86bc9a8872}]" 2026-04-22T10:09:20.279Z [INFO] server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c: entering follower state: follower="Node at e59ba41c-10e0-f573-f2d8-5b04f955dd0c [Follower]" leader-address= leader-id= 2026-04-22T10:09:20.279Z [INFO] creating new snapshot: path=/tmp/raft1465422886/snapshots/2-27-1776852560279.tmp 2026-04-22T10:09:20.279Z [INFO] server-5949481e-b89e-a129-4e2d-f55f10cdcf93: starting restore from snapshot: id=2-27-1776852560279 last-index=27 last-term=2 size-in-bytes=168 2026-04-22T10:09:20.279Z [INFO] server-5949481e-b89e-a129-4e2d-f55f10cdcf93: snapshot restore progress: id=2-27-1776852560279 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2026-04-22T10:09:20.280Z [INFO] server-5949481e-b89e-a129-4e2d-f55f10cdcf93: restored from snapshot: id=2-27-1776852560279 last-index=27 last-term=2 size-in-bytes=168 2026-04-22T10:09:20.280Z [INFO] server-5949481e-b89e-a129-4e2d-f55f10cdcf93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c Address:e59ba41c-10e0-f573-f2d8-5b04f955dd0c} {Suffrage:Voter ID:server-5949481e-b89e-a129-4e2d-f55f10cdcf93 Address:5949481e-b89e-a129-4e2d-f55f10cdcf93} {Suffrage:Voter ID:server-c1426241-9fa7-6427-cf86-9b86bc9a8872 Address:c1426241-9fa7-6427-cf86-9b86bc9a8872}]" 2026-04-22T10:09:20.280Z [INFO] server-5949481e-b89e-a129-4e2d-f55f10cdcf93: entering follower state: follower="Node at 5949481e-b89e-a129-4e2d-f55f10cdcf93 [Follower]" leader-address= leader-id= 2026-04-22T10:09:20.280Z [INFO] creating new snapshot: path=/tmp/raft2712672720/snapshots/2-27-1776852560280.tmp 2026-04-22T10:09:20.280Z [INFO] server-c1426241-9fa7-6427-cf86-9b86bc9a8872: starting restore from snapshot: id=2-27-1776852560280 last-index=27 last-term=2 size-in-bytes=168 2026-04-22T10:09:20.280Z [INFO] server-c1426241-9fa7-6427-cf86-9b86bc9a8872: snapshot restore progress: id=2-27-1776852560280 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2026-04-22T10:09:20.280Z [INFO] server-c1426241-9fa7-6427-cf86-9b86bc9a8872: restored from snapshot: id=2-27-1776852560280 last-index=27 last-term=2 size-in-bytes=168 2026-04-22T10:09:20.280Z [INFO] server-c1426241-9fa7-6427-cf86-9b86bc9a8872: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c Address:e59ba41c-10e0-f573-f2d8-5b04f955dd0c} {Suffrage:Voter ID:server-5949481e-b89e-a129-4e2d-f55f10cdcf93 Address:5949481e-b89e-a129-4e2d-f55f10cdcf93} {Suffrage:Voter ID:server-c1426241-9fa7-6427-cf86-9b86bc9a8872 Address:c1426241-9fa7-6427-cf86-9b86bc9a8872}]" 2026-04-22T10:09:20.280Z [INFO] server-c1426241-9fa7-6427-cf86-9b86bc9a8872: entering follower state: follower="Node at c1426241-9fa7-6427-cf86-9b86bc9a8872 [Follower]" leader-address= leader-id= 2026-04-22T10:09:20.346Z [WARN] server-c1426241-9fa7-6427-cf86-9b86bc9a8872: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:20.346Z [INFO] server-c1426241-9fa7-6427-cf86-9b86bc9a8872: entering candidate state: node="Node at c1426241-9fa7-6427-cf86-9b86bc9a8872 [Candidate]" term=3 2026-04-22T10:09:20.346Z [INFO] server-c1426241-9fa7-6427-cf86-9b86bc9a8872: election won: term=3 tally=2 2026-04-22T10:09:20.346Z [INFO] server-c1426241-9fa7-6427-cf86-9b86bc9a8872: entering leader state: leader="Node at c1426241-9fa7-6427-cf86-9b86bc9a8872 [Leader]" 2026-04-22T10:09:20.346Z [INFO] server-c1426241-9fa7-6427-cf86-9b86bc9a8872: added peer, starting replication: peer=server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c 2026-04-22T10:09:20.346Z [INFO] server-c1426241-9fa7-6427-cf86-9b86bc9a8872: added peer, starting replication: peer=server-5949481e-b89e-a129-4e2d-f55f10cdcf93 2026-04-22T10:09:20.346Z [INFO] server-c1426241-9fa7-6427-cf86-9b86bc9a8872: pipelining replication: peer="{Voter server-5949481e-b89e-a129-4e2d-f55f10cdcf93 5949481e-b89e-a129-4e2d-f55f10cdcf93}" 2026-04-22T10:09:20.347Z [INFO] server-c1426241-9fa7-6427-cf86-9b86bc9a8872: pipelining replication: peer="{Voter server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c e59ba41c-10e0-f573-f2d8-5b04f955dd0c}" 2026-04-22T10:09:20.596Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:20.702Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:20.596346642 +0000 UTC m=+6.620142740 (1 nodes), 32.946µs from start of poll, 484.674109ms from cluster start. Timeout at 2026-04-22 10:09:20.70135993 +0000 UTC m=+6.725156056, 105.013316ms after stability 2026-04-22T10:09:20.702Z [INFO] server-c1426241-9fa7-6427-cf86-9b86bc9a8872: aborting pipeline replication: peer="{Voter server-5949481e-b89e-a129-4e2d-f55f10cdcf93 5949481e-b89e-a129-4e2d-f55f10cdcf93}" 2026-04-22T10:09:20.702Z [INFO] server-c1426241-9fa7-6427-cf86-9b86bc9a8872: aborting pipeline replication: peer="{Voter server-e59ba41c-10e0-f573-f2d8-5b04f955dd0c e59ba41c-10e0-f573-f2d8-5b04f955dd0c}" --- PASS: TestRaft_RecoverCluster (2.12s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.76s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.77s) --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.59s) === RUN TestRaft_HasExistingState raft_test.go:258: Skipping unreliable test --- SKIP: TestRaft_HasExistingState (0.00s) === RUN TestRaft_SingleNode 2026-04-22T10:09:20.703Z [INFO] server-281618b9-9da6-85b8-d452-e93bbad90f32: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-281618b9-9da6-85b8-d452-e93bbad90f32 Address:281618b9-9da6-85b8-d452-e93bbad90f32}]" 2026-04-22T10:09:20.703Z [INFO] server-281618b9-9da6-85b8-d452-e93bbad90f32: entering follower state: follower="Node at 281618b9-9da6-85b8-d452-e93bbad90f32 [Follower]" leader-address= leader-id= 2026-04-22T10:09:20.800Z [WARN] server-281618b9-9da6-85b8-d452-e93bbad90f32: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:20.800Z [INFO] server-281618b9-9da6-85b8-d452-e93bbad90f32: entering candidate state: node="Node at 281618b9-9da6-85b8-d452-e93bbad90f32 [Candidate]" term=2 2026-04-22T10:09:20.800Z [INFO] server-281618b9-9da6-85b8-d452-e93bbad90f32: election won: term=2 tally=1 2026-04-22T10:09:20.800Z [INFO] server-281618b9-9da6-85b8-d452-e93bbad90f32: entering leader state: leader="Node at 281618b9-9da6-85b8-d452-e93bbad90f32 [Leader]" --- PASS: TestRaft_SingleNode (0.10s) === RUN TestRaft_TripleNode 2026-04-22T10:09:20.801Z [INFO] server-f87fb15b-f73c-1cd3-a8df-6696bb292d87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f87fb15b-f73c-1cd3-a8df-6696bb292d87 Address:f87fb15b-f73c-1cd3-a8df-6696bb292d87} {Suffrage:Voter ID:server-560b2ce4-268a-bb85-555f-a2efcf570087 Address:560b2ce4-268a-bb85-555f-a2efcf570087} {Suffrage:Voter ID:server-4da96110-e121-eeb4-83eb-7c0e9fa7d6be Address:4da96110-e121-eeb4-83eb-7c0e9fa7d6be}]" 2026-04-22T10:09:20.801Z [INFO] server-f87fb15b-f73c-1cd3-a8df-6696bb292d87: entering follower state: follower="Node at f87fb15b-f73c-1cd3-a8df-6696bb292d87 [Follower]" leader-address= leader-id= 2026-04-22T10:09:20.801Z [INFO] server-560b2ce4-268a-bb85-555f-a2efcf570087: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f87fb15b-f73c-1cd3-a8df-6696bb292d87 Address:f87fb15b-f73c-1cd3-a8df-6696bb292d87} {Suffrage:Voter ID:server-560b2ce4-268a-bb85-555f-a2efcf570087 Address:560b2ce4-268a-bb85-555f-a2efcf570087} {Suffrage:Voter ID:server-4da96110-e121-eeb4-83eb-7c0e9fa7d6be Address:4da96110-e121-eeb4-83eb-7c0e9fa7d6be}]" 2026-04-22T10:09:20.801Z [INFO] server-560b2ce4-268a-bb85-555f-a2efcf570087: entering follower state: follower="Node at 560b2ce4-268a-bb85-555f-a2efcf570087 [Follower]" leader-address= leader-id= 2026-04-22T10:09:20.801Z [INFO] server-4da96110-e121-eeb4-83eb-7c0e9fa7d6be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f87fb15b-f73c-1cd3-a8df-6696bb292d87 Address:f87fb15b-f73c-1cd3-a8df-6696bb292d87} {Suffrage:Voter ID:server-560b2ce4-268a-bb85-555f-a2efcf570087 Address:560b2ce4-268a-bb85-555f-a2efcf570087} {Suffrage:Voter ID:server-4da96110-e121-eeb4-83eb-7c0e9fa7d6be Address:4da96110-e121-eeb4-83eb-7c0e9fa7d6be}]" 2026-04-22T10:09:20.801Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:20.801Z [INFO] server-4da96110-e121-eeb4-83eb-7c0e9fa7d6be: entering follower state: follower="Node at 4da96110-e121-eeb4-83eb-7c0e9fa7d6be [Follower]" leader-address= leader-id= 2026-04-22T10:09:20.859Z [WARN] server-4da96110-e121-eeb4-83eb-7c0e9fa7d6be: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:20.859Z [INFO] server-4da96110-e121-eeb4-83eb-7c0e9fa7d6be: entering candidate state: node="Node at 4da96110-e121-eeb4-83eb-7c0e9fa7d6be [Candidate]" term=2 2026-04-22T10:09:20.859Z [INFO] server-4da96110-e121-eeb4-83eb-7c0e9fa7d6be: election won: term=2 tally=2 2026-04-22T10:09:20.859Z [INFO] server-4da96110-e121-eeb4-83eb-7c0e9fa7d6be: entering leader state: leader="Node at 4da96110-e121-eeb4-83eb-7c0e9fa7d6be [Leader]" 2026-04-22T10:09:20.859Z [INFO] server-4da96110-e121-eeb4-83eb-7c0e9fa7d6be: added peer, starting replication: peer=server-f87fb15b-f73c-1cd3-a8df-6696bb292d87 2026-04-22T10:09:20.859Z [INFO] server-4da96110-e121-eeb4-83eb-7c0e9fa7d6be: added peer, starting replication: peer=server-560b2ce4-268a-bb85-555f-a2efcf570087 2026-04-22T10:09:20.859Z [INFO] server-4da96110-e121-eeb4-83eb-7c0e9fa7d6be: pipelining replication: peer="{Voter server-560b2ce4-268a-bb85-555f-a2efcf570087 560b2ce4-268a-bb85-555f-a2efcf570087}" 2026-04-22T10:09:20.859Z [INFO] server-4da96110-e121-eeb4-83eb-7c0e9fa7d6be: pipelining replication: peer="{Voter server-f87fb15b-f73c-1cd3-a8df-6696bb292d87 f87fb15b-f73c-1cd3-a8df-6696bb292d87}" 2026-04-22T10:09:20.965Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:20.859496678 +0000 UTC m=+6.883292790 (2 nodes), 58.160182ms from start of poll, 58.520757ms from cluster start. Timeout at 2026-04-22 10:09:20.964499852 +0000 UTC m=+6.988295935, 105.003145ms after stability 2026-04-22T10:09:20.965Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:21.070Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:20.965231826 +0000 UTC m=+6.989027920 (1 nodes), 786ns from start of poll, 164.255887ms from cluster start. Timeout at 2026-04-22 10:09:21.070236887 +0000 UTC m=+7.094032969, 105.005049ms after stability 2026-04-22T10:09:21.081Z [INFO] server-4da96110-e121-eeb4-83eb-7c0e9fa7d6be: aborting pipeline replication: peer="{Voter server-560b2ce4-268a-bb85-555f-a2efcf570087 560b2ce4-268a-bb85-555f-a2efcf570087}" 2026-04-22T10:09:21.081Z [INFO] server-4da96110-e121-eeb4-83eb-7c0e9fa7d6be: aborting pipeline replication: peer="{Voter server-f87fb15b-f73c-1cd3-a8df-6696bb292d87 f87fb15b-f73c-1cd3-a8df-6696bb292d87}" --- PASS: TestRaft_TripleNode (0.28s) === RUN TestRaft_LeaderFail 2026-04-22T10:09:21.082Z [INFO] server-35baf66f-8e1d-2cae-d0c0-68d47521f922: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35baf66f-8e1d-2cae-d0c0-68d47521f922 Address:35baf66f-8e1d-2cae-d0c0-68d47521f922} {Suffrage:Voter ID:server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a Address:f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a} {Suffrage:Voter ID:server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b Address:fb6044d1-2b30-69f3-0ac3-bf2f201eb89b}]" 2026-04-22T10:09:21.082Z [INFO] server-35baf66f-8e1d-2cae-d0c0-68d47521f922: entering follower state: follower="Node at 35baf66f-8e1d-2cae-d0c0-68d47521f922 [Follower]" leader-address= leader-id= 2026-04-22T10:09:21.082Z [INFO] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35baf66f-8e1d-2cae-d0c0-68d47521f922 Address:35baf66f-8e1d-2cae-d0c0-68d47521f922} {Suffrage:Voter ID:server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a Address:f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a} {Suffrage:Voter ID:server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b Address:fb6044d1-2b30-69f3-0ac3-bf2f201eb89b}]" 2026-04-22T10:09:21.082Z [INFO] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: entering follower state: follower="Node at f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a [Follower]" leader-address= leader-id= 2026-04-22T10:09:21.082Z [INFO] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35baf66f-8e1d-2cae-d0c0-68d47521f922 Address:35baf66f-8e1d-2cae-d0c0-68d47521f922} {Suffrage:Voter ID:server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a Address:f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a} {Suffrage:Voter ID:server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b Address:fb6044d1-2b30-69f3-0ac3-bf2f201eb89b}]" 2026-04-22T10:09:21.082Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:21.082Z [INFO] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: entering follower state: follower="Node at fb6044d1-2b30-69f3-0ac3-bf2f201eb89b [Follower]" leader-address= leader-id= 2026-04-22T10:09:21.134Z [WARN] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:21.135Z [INFO] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: entering candidate state: node="Node at fb6044d1-2b30-69f3-0ac3-bf2f201eb89b [Candidate]" term=2 2026-04-22T10:09:21.135Z [INFO] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: election won: term=2 tally=2 2026-04-22T10:09:21.135Z [INFO] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: entering leader state: leader="Node at fb6044d1-2b30-69f3-0ac3-bf2f201eb89b [Leader]" 2026-04-22T10:09:21.135Z [INFO] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: added peer, starting replication: peer=server-35baf66f-8e1d-2cae-d0c0-68d47521f922 2026-04-22T10:09:21.135Z [INFO] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: added peer, starting replication: peer=server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a 2026-04-22T10:09:21.135Z [INFO] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: pipelining replication: peer="{Voter server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a}" 2026-04-22T10:09:21.135Z [INFO] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: pipelining replication: peer="{Voter server-35baf66f-8e1d-2cae-d0c0-68d47521f922 35baf66f-8e1d-2cae-d0c0-68d47521f922}" 2026-04-22T10:09:21.241Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:21.135506093 +0000 UTC m=+7.159302218 (2 nodes), 52.837321ms from start of poll, 53.182234ms from cluster start. Timeout at 2026-04-22 10:09:21.240511907 +0000 UTC m=+7.264308022, 105.005804ms after stability 2026-04-22T10:09:21.241Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:21.347Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:21.241320092 +0000 UTC m=+7.265116200 (1 nodes), 939ns from start of poll, 158.996216ms from cluster start. Timeout at 2026-04-22 10:09:21.346329597 +0000 UTC m=+7.370125703, 105.009503ms after stability raft_test.go:378: [INFO] Disconnecting Node at fb6044d1-2b30-69f3-0ac3-bf2f201eb89b [Leader] 2026-04-22T10:09:21.364Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to pipeline appendEntries: peer="{Voter server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a}" error="append pipeline closed" 2026-04-22T10:09:21.364Z [INFO] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: aborting pipeline replication: peer="{Voter server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a}" 2026-04-22T10:09:21.365Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to heartbeat to: peer=f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a backoff time=10ms error="failed to connect to peer: f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a" 2026-04-22T10:09:21.365Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:21.366Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to heartbeat to: peer=35baf66f-8e1d-2cae-d0c0-68d47521f922 backoff time=10ms error="failed to connect to peer: 35baf66f-8e1d-2cae-d0c0-68d47521f922" 2026-04-22T10:09:21.370Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to pipeline appendEntries: peer="{Voter server-35baf66f-8e1d-2cae-d0c0-68d47521f922 35baf66f-8e1d-2cae-d0c0-68d47521f922}" error="append pipeline closed" 2026-04-22T10:09:21.370Z [INFO] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: aborting pipeline replication: peer="{Voter server-35baf66f-8e1d-2cae-d0c0-68d47521f922 35baf66f-8e1d-2cae-d0c0-68d47521f922}" 2026-04-22T10:09:21.373Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to appendEntries to: peer="{Voter server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a}" error="failed to connect to peer: f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a" 2026-04-22T10:09:21.377Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to appendEntries to: peer="{Voter server-35baf66f-8e1d-2cae-d0c0-68d47521f922 35baf66f-8e1d-2cae-d0c0-68d47521f922}" error="failed to connect to peer: 35baf66f-8e1d-2cae-d0c0-68d47521f922" 2026-04-22T10:09:21.384Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to heartbeat to: peer=f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a backoff time=10ms error="failed to connect to peer: f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a" 2026-04-22T10:09:21.386Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to heartbeat to: peer=35baf66f-8e1d-2cae-d0c0-68d47521f922 backoff time=10ms error="failed to connect to peer: 35baf66f-8e1d-2cae-d0c0-68d47521f922" 2026-04-22T10:09:21.394Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to appendEntries to: peer="{Voter server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a}" error="failed to connect to peer: f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a" 2026-04-22T10:09:21.397Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to appendEntries to: peer="{Voter server-35baf66f-8e1d-2cae-d0c0-68d47521f922 35baf66f-8e1d-2cae-d0c0-68d47521f922}" error="failed to connect to peer: 35baf66f-8e1d-2cae-d0c0-68d47521f922" 2026-04-22T10:09:21.402Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to heartbeat to: peer=35baf66f-8e1d-2cae-d0c0-68d47521f922 backoff time=10ms error="failed to connect to peer: 35baf66f-8e1d-2cae-d0c0-68d47521f922" 2026-04-22T10:09:21.405Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to heartbeat to: peer=f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a backoff time=10ms error="failed to connect to peer: f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a" 2026-04-22T10:09:21.408Z [WARN] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to contact: server-id=server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a time=50.356083ms 2026-04-22T10:09:21.415Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to appendEntries to: peer="{Voter server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a}" error="failed to connect to peer: f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a" 2026-04-22T10:09:21.418Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to appendEntries to: peer="{Voter server-35baf66f-8e1d-2cae-d0c0-68d47521f922 35baf66f-8e1d-2cae-d0c0-68d47521f922}" error="failed to connect to peer: 35baf66f-8e1d-2cae-d0c0-68d47521f922" 2026-04-22T10:09:21.419Z [WARN] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to contact: server-id=server-35baf66f-8e1d-2cae-d0c0-68d47521f922 time=58.175808ms 2026-04-22T10:09:21.419Z [WARN] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to contact: server-id=server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a time=61.406952ms 2026-04-22T10:09:21.419Z [WARN] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to contact quorum of nodes, stepping down 2026-04-22T10:09:21.419Z [INFO] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: entering follower state: follower="Node at fb6044d1-2b30-69f3-0ac3-bf2f201eb89b [Follower]" leader-address= leader-id= 2026-04-22T10:09:21.449Z [WARN] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: heartbeat timeout reached, starting election: last-leader-addr=fb6044d1-2b30-69f3-0ac3-bf2f201eb89b last-leader-id=server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b 2026-04-22T10:09:21.449Z [INFO] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: entering candidate state: node="Node at f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a [Candidate]" term=3 2026-04-22T10:09:21.449Z [ERROR] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: failed to make requestVote RPC: target="{Voter server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b fb6044d1-2b30-69f3-0ac3-bf2f201eb89b}" error="failed to connect to peer: fb6044d1-2b30-69f3-0ac3-bf2f201eb89b" term=3 2026-04-22T10:09:21.449Z [WARN] server-35baf66f-8e1d-2cae-d0c0-68d47521f922: rejecting vote request since we have a leader: from=f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a leader=fb6044d1-2b30-69f3-0ac3-bf2f201eb89b leader-id=server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b 2026-04-22T10:09:21.469Z [WARN] server-35baf66f-8e1d-2cae-d0c0-68d47521f922: heartbeat timeout reached, starting election: last-leader-addr=fb6044d1-2b30-69f3-0ac3-bf2f201eb89b last-leader-id=server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b 2026-04-22T10:09:21.469Z [INFO] server-35baf66f-8e1d-2cae-d0c0-68d47521f922: entering candidate state: node="Node at 35baf66f-8e1d-2cae-d0c0-68d47521f922 [Candidate]" term=3 2026-04-22T10:09:21.469Z [ERROR] server-35baf66f-8e1d-2cae-d0c0-68d47521f922: failed to make requestVote RPC: target="{Voter server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b fb6044d1-2b30-69f3-0ac3-bf2f201eb89b}" error="failed to connect to peer: fb6044d1-2b30-69f3-0ac3-bf2f201eb89b" term=3 2026-04-22T10:09:21.469Z [INFO] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: duplicate requestVote for same term: term=3 2026-04-22T10:09:21.511Z [WARN] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:21.511Z [INFO] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: entering candidate state: node="Node at fb6044d1-2b30-69f3-0ac3-bf2f201eb89b [Candidate]" term=3 2026-04-22T10:09:21.511Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to make requestVote RPC: target="{Voter server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a}" error="failed to connect to peer: f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a" term=3 2026-04-22T10:09:21.511Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to make requestVote RPC: target="{Voter server-35baf66f-8e1d-2cae-d0c0-68d47521f922 35baf66f-8e1d-2cae-d0c0-68d47521f922}" error="failed to connect to peer: 35baf66f-8e1d-2cae-d0c0-68d47521f922" term=3 2026-04-22T10:09:21.516Z [WARN] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: Election timeout reached, restarting election 2026-04-22T10:09:21.516Z [INFO] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: entering candidate state: node="Node at f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a [Candidate]" term=4 2026-04-22T10:09:21.516Z [ERROR] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: failed to make requestVote RPC: target="{Voter server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b fb6044d1-2b30-69f3-0ac3-bf2f201eb89b}" error="failed to connect to peer: fb6044d1-2b30-69f3-0ac3-bf2f201eb89b" term=4 2026-04-22T10:09:21.516Z [INFO] server-35baf66f-8e1d-2cae-d0c0-68d47521f922: entering follower state: follower="Node at 35baf66f-8e1d-2cae-d0c0-68d47521f922 [Follower]" leader-address= leader-id= 2026-04-22T10:09:21.516Z [INFO] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: election won: term=4 tally=2 2026-04-22T10:09:21.516Z [INFO] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: entering leader state: leader="Node at f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a [Leader]" 2026-04-22T10:09:21.516Z [INFO] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: added peer, starting replication: peer=server-35baf66f-8e1d-2cae-d0c0-68d47521f922 2026-04-22T10:09:21.516Z [INFO] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: added peer, starting replication: peer=server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b 2026-04-22T10:09:21.516Z [ERROR] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: failed to appendEntries to: peer="{Voter server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b fb6044d1-2b30-69f3-0ac3-bf2f201eb89b}" error="failed to connect to peer: fb6044d1-2b30-69f3-0ac3-bf2f201eb89b" 2026-04-22T10:09:21.516Z [INFO] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: pipelining replication: peer="{Voter server-35baf66f-8e1d-2cae-d0c0-68d47521f922 35baf66f-8e1d-2cae-d0c0-68d47521f922}" 2026-04-22T10:09:21.526Z [ERROR] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: failed to heartbeat to: peer=fb6044d1-2b30-69f3-0ac3-bf2f201eb89b backoff time=10ms error="failed to connect to peer: fb6044d1-2b30-69f3-0ac3-bf2f201eb89b" 2026-04-22T10:09:21.534Z [ERROR] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: failed to appendEntries to: peer="{Voter server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b fb6044d1-2b30-69f3-0ac3-bf2f201eb89b}" error="failed to connect to peer: fb6044d1-2b30-69f3-0ac3-bf2f201eb89b" 2026-04-22T10:09:21.547Z [ERROR] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: failed to heartbeat to: peer=fb6044d1-2b30-69f3-0ac3-bf2f201eb89b backoff time=10ms error="failed to connect to peer: fb6044d1-2b30-69f3-0ac3-bf2f201eb89b" 2026-04-22T10:09:21.553Z [ERROR] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: failed to appendEntries to: peer="{Voter server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b fb6044d1-2b30-69f3-0ac3-bf2f201eb89b}" error="failed to connect to peer: fb6044d1-2b30-69f3-0ac3-bf2f201eb89b" 2026-04-22T10:09:21.566Z [WARN] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: failed to contact: server-id=server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b time=50.117302ms 2026-04-22T10:09:21.569Z [ERROR] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: failed to heartbeat to: peer=fb6044d1-2b30-69f3-0ac3-bf2f201eb89b backoff time=10ms error="failed to connect to peer: fb6044d1-2b30-69f3-0ac3-bf2f201eb89b" 2026-04-22T10:09:21.579Z [ERROR] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: failed to appendEntries to: peer="{Voter server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b fb6044d1-2b30-69f3-0ac3-bf2f201eb89b}" error="failed to connect to peer: fb6044d1-2b30-69f3-0ac3-bf2f201eb89b" 2026-04-22T10:09:21.583Z [WARN] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: Election timeout reached, restarting election 2026-04-22T10:09:21.583Z [INFO] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: entering candidate state: node="Node at fb6044d1-2b30-69f3-0ac3-bf2f201eb89b [Candidate]" term=4 2026-04-22T10:09:21.583Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to make requestVote RPC: target="{Voter server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a}" error="failed to connect to peer: f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a" term=4 2026-04-22T10:09:21.583Z [ERROR] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: failed to make requestVote RPC: target="{Voter server-35baf66f-8e1d-2cae-d0c0-68d47521f922 35baf66f-8e1d-2cae-d0c0-68d47521f922}" error="failed to connect to peer: 35baf66f-8e1d-2cae-d0c0-68d47521f922" term=4 2026-04-22T10:09:21.589Z [ERROR] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: failed to heartbeat to: peer=fb6044d1-2b30-69f3-0ac3-bf2f201eb89b backoff time=20ms error="failed to connect to peer: fb6044d1-2b30-69f3-0ac3-bf2f201eb89b" 2026-04-22T10:09:21.616Z [WARN] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: failed to contact: server-id=server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b time=100.115458ms 2026-04-22T10:09:21.618Z [ERROR] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: failed to heartbeat to: peer=fb6044d1-2b30-69f3-0ac3-bf2f201eb89b backoff time=25ms error="failed to connect to peer: fb6044d1-2b30-69f3-0ac3-bf2f201eb89b" 2026-04-22T10:09:21.622Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:21.516692355 +0000 UTC m=+7.540488465 (1 nodes), 150.954735ms from start of poll, 434.368481ms from cluster start. Timeout at 2026-04-22 10:09:21.621696043 +0000 UTC m=+7.645492125, 105.00366ms after stability raft_test.go:413: [INFO] Reconnecting Node at fb6044d1-2b30-69f3-0ac3-bf2f201eb89b [Candidate] 2026-04-22T10:09:21.627Z [INFO] server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b: entering follower state: follower="Node at fb6044d1-2b30-69f3-0ac3-bf2f201eb89b [Follower]" leader-address=f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a leader-id=server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a 2026-04-22T10:09:21.627Z [INFO] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: pipelining replication: peer="{Voter server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b fb6044d1-2b30-69f3-0ac3-bf2f201eb89b}" 2026-04-22T10:09:21.633Z [INFO] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: aborting pipeline replication: peer="{Voter server-fb6044d1-2b30-69f3-0ac3-bf2f201eb89b fb6044d1-2b30-69f3-0ac3-bf2f201eb89b}" 2026-04-22T10:09:21.633Z [INFO] server-f8102dde-10fd-e9ec-ecee-8a2f8b5eb12a: aborting pipeline replication: peer="{Voter server-35baf66f-8e1d-2cae-d0c0-68d47521f922 35baf66f-8e1d-2cae-d0c0-68d47521f922}" --- PASS: TestRaft_LeaderFail (0.55s) === RUN TestRaft_BehindFollower 2026-04-22T10:09:21.634Z [INFO] server-428683ea-9b24-eef8-bc76-95e6f02e00a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-428683ea-9b24-eef8-bc76-95e6f02e00a0 Address:428683ea-9b24-eef8-bc76-95e6f02e00a0} {Suffrage:Voter ID:server-f75509ed-8427-f556-871d-d8f064505597 Address:f75509ed-8427-f556-871d-d8f064505597} {Suffrage:Voter ID:server-8c7247e0-bda8-4c4e-fa6d-cfc9dad03a43 Address:8c7247e0-bda8-4c4e-fa6d-cfc9dad03a43}]" 2026-04-22T10:09:21.634Z [INFO] server-f75509ed-8427-f556-871d-d8f064505597: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-428683ea-9b24-eef8-bc76-95e6f02e00a0 Address:428683ea-9b24-eef8-bc76-95e6f02e00a0} {Suffrage:Voter ID:server-f75509ed-8427-f556-871d-d8f064505597 Address:f75509ed-8427-f556-871d-d8f064505597} {Suffrage:Voter ID:server-8c7247e0-bda8-4c4e-fa6d-cfc9dad03a43 Address:8c7247e0-bda8-4c4e-fa6d-cfc9dad03a43}]" 2026-04-22T10:09:21.634Z [INFO] server-8c7247e0-bda8-4c4e-fa6d-cfc9dad03a43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-428683ea-9b24-eef8-bc76-95e6f02e00a0 Address:428683ea-9b24-eef8-bc76-95e6f02e00a0} {Suffrage:Voter ID:server-f75509ed-8427-f556-871d-d8f064505597 Address:f75509ed-8427-f556-871d-d8f064505597} {Suffrage:Voter ID:server-8c7247e0-bda8-4c4e-fa6d-cfc9dad03a43 Address:8c7247e0-bda8-4c4e-fa6d-cfc9dad03a43}]" 2026-04-22T10:09:21.634Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:21.634Z [INFO] server-428683ea-9b24-eef8-bc76-95e6f02e00a0: entering follower state: follower="Node at 428683ea-9b24-eef8-bc76-95e6f02e00a0 [Follower]" leader-address= leader-id= 2026-04-22T10:09:21.634Z [INFO] server-f75509ed-8427-f556-871d-d8f064505597: entering follower state: follower="Node at f75509ed-8427-f556-871d-d8f064505597 [Follower]" leader-address= leader-id= 2026-04-22T10:09:21.634Z [INFO] server-8c7247e0-bda8-4c4e-fa6d-cfc9dad03a43: entering follower state: follower="Node at 8c7247e0-bda8-4c4e-fa6d-cfc9dad03a43 [Follower]" leader-address= leader-id= 2026-04-22T10:09:21.697Z [WARN] server-428683ea-9b24-eef8-bc76-95e6f02e00a0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:21.697Z [INFO] server-428683ea-9b24-eef8-bc76-95e6f02e00a0: entering candidate state: node="Node at 428683ea-9b24-eef8-bc76-95e6f02e00a0 [Candidate]" term=2 2026-04-22T10:09:21.698Z [INFO] server-428683ea-9b24-eef8-bc76-95e6f02e00a0: election won: term=2 tally=2 2026-04-22T10:09:21.698Z [INFO] server-428683ea-9b24-eef8-bc76-95e6f02e00a0: entering leader state: leader="Node at 428683ea-9b24-eef8-bc76-95e6f02e00a0 [Leader]" 2026-04-22T10:09:21.698Z [INFO] server-428683ea-9b24-eef8-bc76-95e6f02e00a0: added peer, starting replication: peer=server-f75509ed-8427-f556-871d-d8f064505597 2026-04-22T10:09:21.698Z [INFO] server-428683ea-9b24-eef8-bc76-95e6f02e00a0: added peer, starting replication: peer=server-8c7247e0-bda8-4c4e-fa6d-cfc9dad03a43 2026-04-22T10:09:21.698Z [INFO] server-428683ea-9b24-eef8-bc76-95e6f02e00a0: pipelining replication: peer="{Voter server-8c7247e0-bda8-4c4e-fa6d-cfc9dad03a43 8c7247e0-bda8-4c4e-fa6d-cfc9dad03a43}" 2026-04-22T10:09:21.698Z [INFO] server-428683ea-9b24-eef8-bc76-95e6f02e00a0: pipelining replication: peer="{Voter server-f75509ed-8427-f556-871d-d8f064505597 f75509ed-8427-f556-871d-d8f064505597}" 2026-04-22T10:09:21.804Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:21.698464198 +0000 UTC m=+7.722260309 (1 nodes), 63.825396ms from start of poll, 64.18175ms from cluster start. Timeout at 2026-04-22 10:09:21.803467847 +0000 UTC m=+7.827263930, 105.003621ms after stability 2026-04-22T10:09:21.804Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:21.909Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:21.804421533 +0000 UTC m=+7.828217650 (2 nodes), 2.384µs from start of poll, 170.139091ms from cluster start. Timeout at 2026-04-22 10:09:21.909428341 +0000 UTC m=+7.933224424, 105.006774ms after stability 2026-04-22T10:09:21.910Z [ERROR] server-428683ea-9b24-eef8-bc76-95e6f02e00a0: failed to pipeline appendEntries: peer="{Voter server-f75509ed-8427-f556-871d-d8f064505597 f75509ed-8427-f556-871d-d8f064505597}" error="append pipeline closed" 2026-04-22T10:09:21.910Z [INFO] server-428683ea-9b24-eef8-bc76-95e6f02e00a0: aborting pipeline replication: peer="{Voter server-f75509ed-8427-f556-871d-d8f064505597 f75509ed-8427-f556-871d-d8f064505597}" 2026-04-22T10:09:21.910Z [ERROR] server-428683ea-9b24-eef8-bc76-95e6f02e00a0: failed to appendEntries to: peer="{Voter server-f75509ed-8427-f556-871d-d8f064505597 f75509ed-8427-f556-871d-d8f064505597}" error="failed to connect to peer: f75509ed-8427-f556-871d-d8f064505597" raft_test.go:459: [INFO] Finished apply without behind follower 2026-04-22T10:09:21.921Z [INFO] server-428683ea-9b24-eef8-bc76-95e6f02e00a0: pipelining replication: peer="{Voter server-f75509ed-8427-f556-871d-d8f064505597 f75509ed-8427-f556-871d-d8f064505597}" 2026-04-22T10:09:21.937Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:22.043Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:21.937660194 +0000 UTC m=+7.961456302 (1 nodes), 1.166µs from start of poll, 303.377743ms from cluster start. Timeout at 2026-04-22 10:09:22.042668559 +0000 UTC m=+8.066464652, 105.00835ms after stability 2026-04-22T10:09:22.043Z [INFO] server-428683ea-9b24-eef8-bc76-95e6f02e00a0: aborting pipeline replication: peer="{Voter server-8c7247e0-bda8-4c4e-fa6d-cfc9dad03a43 8c7247e0-bda8-4c4e-fa6d-cfc9dad03a43}" 2026-04-22T10:09:22.043Z [INFO] server-428683ea-9b24-eef8-bc76-95e6f02e00a0: aborting pipeline replication: peer="{Voter server-f75509ed-8427-f556-871d-d8f064505597 f75509ed-8427-f556-871d-d8f064505597}" --- PASS: TestRaft_BehindFollower (0.41s) === RUN TestRaft_ApplyNonLeader 2026-04-22T10:09:22.044Z [INFO] server-051a4dd4-a43e-7302-f030-384fc4d2b135: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-051a4dd4-a43e-7302-f030-384fc4d2b135 Address:051a4dd4-a43e-7302-f030-384fc4d2b135} {Suffrage:Voter ID:server-15ae070a-55f5-558f-2ffb-6c61b7e231f8 Address:15ae070a-55f5-558f-2ffb-6c61b7e231f8} {Suffrage:Voter ID:server-b5df207f-0c84-606c-28c1-6ffe8ae7ae60 Address:b5df207f-0c84-606c-28c1-6ffe8ae7ae60}]" 2026-04-22T10:09:22.044Z [INFO] server-051a4dd4-a43e-7302-f030-384fc4d2b135: entering follower state: follower="Node at 051a4dd4-a43e-7302-f030-384fc4d2b135 [Follower]" leader-address= leader-id= 2026-04-22T10:09:22.044Z [INFO] server-15ae070a-55f5-558f-2ffb-6c61b7e231f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-051a4dd4-a43e-7302-f030-384fc4d2b135 Address:051a4dd4-a43e-7302-f030-384fc4d2b135} {Suffrage:Voter ID:server-15ae070a-55f5-558f-2ffb-6c61b7e231f8 Address:15ae070a-55f5-558f-2ffb-6c61b7e231f8} {Suffrage:Voter ID:server-b5df207f-0c84-606c-28c1-6ffe8ae7ae60 Address:b5df207f-0c84-606c-28c1-6ffe8ae7ae60}]" 2026-04-22T10:09:22.044Z [INFO] server-15ae070a-55f5-558f-2ffb-6c61b7e231f8: entering follower state: follower="Node at 15ae070a-55f5-558f-2ffb-6c61b7e231f8 [Follower]" leader-address= leader-id= 2026-04-22T10:09:22.044Z [INFO] server-b5df207f-0c84-606c-28c1-6ffe8ae7ae60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-051a4dd4-a43e-7302-f030-384fc4d2b135 Address:051a4dd4-a43e-7302-f030-384fc4d2b135} {Suffrage:Voter ID:server-15ae070a-55f5-558f-2ffb-6c61b7e231f8 Address:15ae070a-55f5-558f-2ffb-6c61b7e231f8} {Suffrage:Voter ID:server-b5df207f-0c84-606c-28c1-6ffe8ae7ae60 Address:b5df207f-0c84-606c-28c1-6ffe8ae7ae60}]" 2026-04-22T10:09:22.044Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:22.044Z [INFO] server-b5df207f-0c84-606c-28c1-6ffe8ae7ae60: entering follower state: follower="Node at b5df207f-0c84-606c-28c1-6ffe8ae7ae60 [Follower]" leader-address= leader-id= 2026-04-22T10:09:22.106Z [WARN] server-b5df207f-0c84-606c-28c1-6ffe8ae7ae60: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:22.106Z [INFO] server-b5df207f-0c84-606c-28c1-6ffe8ae7ae60: entering candidate state: node="Node at b5df207f-0c84-606c-28c1-6ffe8ae7ae60 [Candidate]" term=2 2026-04-22T10:09:22.107Z [INFO] server-b5df207f-0c84-606c-28c1-6ffe8ae7ae60: election won: term=2 tally=2 2026-04-22T10:09:22.107Z [INFO] server-b5df207f-0c84-606c-28c1-6ffe8ae7ae60: entering leader state: leader="Node at b5df207f-0c84-606c-28c1-6ffe8ae7ae60 [Leader]" 2026-04-22T10:09:22.107Z [INFO] server-b5df207f-0c84-606c-28c1-6ffe8ae7ae60: added peer, starting replication: peer=server-051a4dd4-a43e-7302-f030-384fc4d2b135 2026-04-22T10:09:22.107Z [INFO] server-b5df207f-0c84-606c-28c1-6ffe8ae7ae60: added peer, starting replication: peer=server-15ae070a-55f5-558f-2ffb-6c61b7e231f8 2026-04-22T10:09:22.107Z [INFO] server-b5df207f-0c84-606c-28c1-6ffe8ae7ae60: pipelining replication: peer="{Voter server-051a4dd4-a43e-7302-f030-384fc4d2b135 051a4dd4-a43e-7302-f030-384fc4d2b135}" 2026-04-22T10:09:22.107Z [INFO] server-b5df207f-0c84-606c-28c1-6ffe8ae7ae60: pipelining replication: peer="{Voter server-15ae070a-55f5-558f-2ffb-6c61b7e231f8 15ae070a-55f5-558f-2ffb-6c61b7e231f8}" 2026-04-22T10:09:22.213Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:22.107215266 +0000 UTC m=+8.131011395 (1 nodes), 62.750275ms from start of poll, 63.083755ms from cluster start. Timeout at 2026-04-22 10:09:22.212216379 +0000 UTC m=+8.236012462, 105.001067ms after stability 2026-04-22T10:09:22.213Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:22.318Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:22.213126202 +0000 UTC m=+8.236922310 (2 nodes), 1.22µs from start of poll, 168.99467ms from cluster start. Timeout at 2026-04-22 10:09:22.318132571 +0000 UTC m=+8.341928651, 105.006341ms after stability 2026-04-22T10:09:22.318Z [INFO] server-b5df207f-0c84-606c-28c1-6ffe8ae7ae60: aborting pipeline replication: peer="{Voter server-15ae070a-55f5-558f-2ffb-6c61b7e231f8 15ae070a-55f5-558f-2ffb-6c61b7e231f8}" 2026-04-22T10:09:22.318Z [INFO] server-b5df207f-0c84-606c-28c1-6ffe8ae7ae60: aborting pipeline replication: peer="{Voter server-051a4dd4-a43e-7302-f030-384fc4d2b135 051a4dd4-a43e-7302-f030-384fc4d2b135}" --- PASS: TestRaft_ApplyNonLeader (0.28s) === RUN TestRaft_ApplyConcurrent 2026-04-22T10:09:22.319Z [INFO] server-7326745e-254a-bfb4-3be1-36e97c978490: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7326745e-254a-bfb4-3be1-36e97c978490 Address:7326745e-254a-bfb4-3be1-36e97c978490} {Suffrage:Voter ID:server-cd403e3c-7dfe-5200-41ee-a75f96447917 Address:cd403e3c-7dfe-5200-41ee-a75f96447917} {Suffrage:Voter ID:server-9a71e383-1c53-d2d4-c5bb-6a24696e0a8f Address:9a71e383-1c53-d2d4-c5bb-6a24696e0a8f}]" 2026-04-22T10:09:22.319Z [INFO] server-7326745e-254a-bfb4-3be1-36e97c978490: entering follower state: follower="Node at 7326745e-254a-bfb4-3be1-36e97c978490 [Follower]" leader-address= leader-id= 2026-04-22T10:09:22.319Z [INFO] server-cd403e3c-7dfe-5200-41ee-a75f96447917: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7326745e-254a-bfb4-3be1-36e97c978490 Address:7326745e-254a-bfb4-3be1-36e97c978490} {Suffrage:Voter ID:server-cd403e3c-7dfe-5200-41ee-a75f96447917 Address:cd403e3c-7dfe-5200-41ee-a75f96447917} {Suffrage:Voter ID:server-9a71e383-1c53-d2d4-c5bb-6a24696e0a8f Address:9a71e383-1c53-d2d4-c5bb-6a24696e0a8f}]" 2026-04-22T10:09:22.319Z [INFO] server-cd403e3c-7dfe-5200-41ee-a75f96447917: entering follower state: follower="Node at cd403e3c-7dfe-5200-41ee-a75f96447917 [Follower]" leader-address= leader-id= 2026-04-22T10:09:22.319Z [INFO] server-9a71e383-1c53-d2d4-c5bb-6a24696e0a8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7326745e-254a-bfb4-3be1-36e97c978490 Address:7326745e-254a-bfb4-3be1-36e97c978490} {Suffrage:Voter ID:server-cd403e3c-7dfe-5200-41ee-a75f96447917 Address:cd403e3c-7dfe-5200-41ee-a75f96447917} {Suffrage:Voter ID:server-9a71e383-1c53-d2d4-c5bb-6a24696e0a8f Address:9a71e383-1c53-d2d4-c5bb-6a24696e0a8f}]" 2026-04-22T10:09:22.319Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:22.319Z [INFO] server-9a71e383-1c53-d2d4-c5bb-6a24696e0a8f: entering follower state: follower="Node at 9a71e383-1c53-d2d4-c5bb-6a24696e0a8f [Follower]" leader-address= leader-id= 2026-04-22T10:09:22.445Z [WARN] server-9a71e383-1c53-d2d4-c5bb-6a24696e0a8f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:22.445Z [INFO] server-9a71e383-1c53-d2d4-c5bb-6a24696e0a8f: entering candidate state: node="Node at 9a71e383-1c53-d2d4-c5bb-6a24696e0a8f [Candidate]" term=2 2026-04-22T10:09:22.446Z [INFO] server-9a71e383-1c53-d2d4-c5bb-6a24696e0a8f: election won: term=2 tally=2 2026-04-22T10:09:22.446Z [INFO] server-9a71e383-1c53-d2d4-c5bb-6a24696e0a8f: entering leader state: leader="Node at 9a71e383-1c53-d2d4-c5bb-6a24696e0a8f [Leader]" 2026-04-22T10:09:22.446Z [INFO] server-9a71e383-1c53-d2d4-c5bb-6a24696e0a8f: added peer, starting replication: peer=server-7326745e-254a-bfb4-3be1-36e97c978490 2026-04-22T10:09:22.446Z [INFO] server-9a71e383-1c53-d2d4-c5bb-6a24696e0a8f: added peer, starting replication: peer=server-cd403e3c-7dfe-5200-41ee-a75f96447917 2026-04-22T10:09:22.446Z [INFO] server-9a71e383-1c53-d2d4-c5bb-6a24696e0a8f: pipelining replication: peer="{Voter server-7326745e-254a-bfb4-3be1-36e97c978490 7326745e-254a-bfb4-3be1-36e97c978490}" 2026-04-22T10:09:22.446Z [INFO] server-9a71e383-1c53-d2d4-c5bb-6a24696e0a8f: pipelining replication: peer="{Voter server-cd403e3c-7dfe-5200-41ee-a75f96447917 cd403e3c-7dfe-5200-41ee-a75f96447917}" 2026-04-22T10:09:22.653Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:22.446108213 +0000 UTC m=+8.469904340 (1 nodes), 126.263698ms from start of poll, 126.596936ms from cluster start. Timeout at 2026-04-22 10:09:22.651114606 +0000 UTC m=+8.674910731, 205.006391ms after stability 2026-04-22T10:09:22.671Z [INFO] server-9a71e383-1c53-d2d4-c5bb-6a24696e0a8f: aborting pipeline replication: peer="{Voter server-cd403e3c-7dfe-5200-41ee-a75f96447917 cd403e3c-7dfe-5200-41ee-a75f96447917}" 2026-04-22T10:09:22.671Z [INFO] server-9a71e383-1c53-d2d4-c5bb-6a24696e0a8f: aborting pipeline replication: peer="{Voter server-7326745e-254a-bfb4-3be1-36e97c978490 7326745e-254a-bfb4-3be1-36e97c978490}" --- PASS: TestRaft_ApplyConcurrent (0.35s) === RUN TestRaft_ApplyConcurrent_Timeout 2026-04-22T10:09:22.672Z [INFO] server-c5ee96ec-5dfc-b691-ee1d-f4f9ee355e8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c5ee96ec-5dfc-b691-ee1d-f4f9ee355e8b Address:c5ee96ec-5dfc-b691-ee1d-f4f9ee355e8b}]" 2026-04-22T10:09:22.672Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:22.672Z [INFO] server-c5ee96ec-5dfc-b691-ee1d-f4f9ee355e8b: entering follower state: follower="Node at c5ee96ec-5dfc-b691-ee1d-f4f9ee355e8b [Follower]" leader-address= leader-id= 2026-04-22T10:09:22.841Z [WARN] server-c5ee96ec-5dfc-b691-ee1d-f4f9ee355e8b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:22.841Z [INFO] server-c5ee96ec-5dfc-b691-ee1d-f4f9ee355e8b: entering candidate state: node="Node at c5ee96ec-5dfc-b691-ee1d-f4f9ee355e8b [Candidate]" term=2 2026-04-22T10:09:22.841Z [INFO] server-c5ee96ec-5dfc-b691-ee1d-f4f9ee355e8b: election won: term=2 tally=1 2026-04-22T10:09:22.841Z [INFO] server-c5ee96ec-5dfc-b691-ee1d-f4f9ee355e8b: entering leader state: leader="Node at c5ee96ec-5dfc-b691-ee1d-f4f9ee355e8b [Leader]" 2026-04-22T10:09:23.045Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:22.841836554 +0000 UTC m=+8.865632668 (1 nodes), 169.126822ms from start of poll, 169.273773ms from cluster start. Timeout at 2026-04-22 10:09:23.042847528 +0000 UTC m=+9.066643640, 201.010972ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.37s) === RUN TestRaft_JoinNode 2026-04-22T10:09:23.046Z [INFO] server-96e251b9-7953-1866-857b-85a162ac79e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96e251b9-7953-1866-857b-85a162ac79e0 Address:96e251b9-7953-1866-857b-85a162ac79e0} {Suffrage:Voter ID:server-567ddd18-ef01-cc99-f387-34ac0aa95ed2 Address:567ddd18-ef01-cc99-f387-34ac0aa95ed2}]" 2026-04-22T10:09:23.046Z [INFO] server-567ddd18-ef01-cc99-f387-34ac0aa95ed2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96e251b9-7953-1866-857b-85a162ac79e0 Address:96e251b9-7953-1866-857b-85a162ac79e0} {Suffrage:Voter ID:server-567ddd18-ef01-cc99-f387-34ac0aa95ed2 Address:567ddd18-ef01-cc99-f387-34ac0aa95ed2}]" 2026-04-22T10:09:23.047Z [INFO] server-96e251b9-7953-1866-857b-85a162ac79e0: entering follower state: follower="Node at 96e251b9-7953-1866-857b-85a162ac79e0 [Follower]" leader-address= leader-id= 2026-04-22T10:09:23.047Z [INFO] server-567ddd18-ef01-cc99-f387-34ac0aa95ed2: entering follower state: follower="Node at 567ddd18-ef01-cc99-f387-34ac0aa95ed2 [Follower]" leader-address= leader-id= 2026-04-22T10:09:23.047Z [INFO] server-ccbf10e9-7aef-6a84-6972-1f6c437570f9: initial configuration: index=0 servers=[] 2026-04-22T10:09:23.047Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:23.047Z [INFO] server-ccbf10e9-7aef-6a84-6972-1f6c437570f9: entering follower state: follower="Node at ccbf10e9-7aef-6a84-6972-1f6c437570f9 [Follower]" leader-address= leader-id= 2026-04-22T10:09:23.104Z [WARN] server-ccbf10e9-7aef-6a84-6972-1f6c437570f9: no known peers, aborting election 2026-04-22T10:09:23.143Z [WARN] server-96e251b9-7953-1866-857b-85a162ac79e0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:23.143Z [INFO] server-96e251b9-7953-1866-857b-85a162ac79e0: entering candidate state: node="Node at 96e251b9-7953-1866-857b-85a162ac79e0 [Candidate]" term=2 2026-04-22T10:09:23.143Z [INFO] server-96e251b9-7953-1866-857b-85a162ac79e0: election won: term=2 tally=2 2026-04-22T10:09:23.143Z [INFO] server-96e251b9-7953-1866-857b-85a162ac79e0: entering leader state: leader="Node at 96e251b9-7953-1866-857b-85a162ac79e0 [Leader]" 2026-04-22T10:09:23.143Z [INFO] server-96e251b9-7953-1866-857b-85a162ac79e0: added peer, starting replication: peer=server-567ddd18-ef01-cc99-f387-34ac0aa95ed2 2026-04-22T10:09:23.143Z [INFO] server-96e251b9-7953-1866-857b-85a162ac79e0: pipelining replication: peer="{Voter server-567ddd18-ef01-cc99-f387-34ac0aa95ed2 567ddd18-ef01-cc99-f387-34ac0aa95ed2}" 2026-04-22T10:09:23.248Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:23.143645313 +0000 UTC m=+9.167441426 (1 nodes), 96.416015ms from start of poll, 96.927241ms from cluster start. Timeout at 2026-04-22 10:09:23.248651561 +0000 UTC m=+9.272447737, 105.006311ms after stability 2026-04-22T10:09:23.249Z [INFO] server-96e251b9-7953-1866-857b-85a162ac79e0: updating configuration: command=AddVoter server-id=server-ccbf10e9-7aef-6a84-6972-1f6c437570f9 server-addr=ccbf10e9-7aef-6a84-6972-1f6c437570f9 servers="[{Suffrage:Voter ID:server-96e251b9-7953-1866-857b-85a162ac79e0 Address:96e251b9-7953-1866-857b-85a162ac79e0} {Suffrage:Voter ID:server-567ddd18-ef01-cc99-f387-34ac0aa95ed2 Address:567ddd18-ef01-cc99-f387-34ac0aa95ed2} {Suffrage:Voter ID:server-ccbf10e9-7aef-6a84-6972-1f6c437570f9 Address:ccbf10e9-7aef-6a84-6972-1f6c437570f9}]" 2026-04-22T10:09:23.249Z [INFO] server-96e251b9-7953-1866-857b-85a162ac79e0: added peer, starting replication: peer=server-ccbf10e9-7aef-6a84-6972-1f6c437570f9 2026-04-22T10:09:23.249Z [WARN] server-ccbf10e9-7aef-6a84-6972-1f6c437570f9: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2026-04-22T10:09:23.249Z [WARN] server-96e251b9-7953-1866-857b-85a162ac79e0: appendEntries rejected, sending older logs: peer="{Voter server-ccbf10e9-7aef-6a84-6972-1f6c437570f9 ccbf10e9-7aef-6a84-6972-1f6c437570f9}" next=1 2026-04-22T10:09:23.249Z [INFO] server-96e251b9-7953-1866-857b-85a162ac79e0: pipelining replication: peer="{Voter server-ccbf10e9-7aef-6a84-6972-1f6c437570f9 ccbf10e9-7aef-6a84-6972-1f6c437570f9}" 2026-04-22T10:09:23.249Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:23.355Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:23.24947961 +0000 UTC m=+9.273275721 (1 nodes), 937ns from start of poll, 202.761536ms from cluster start. Timeout at 2026-04-22 10:09:23.354484172 +0000 UTC m=+9.378280291, 105.00457ms after stability 2026-04-22T10:09:23.355Z [INFO] server-96e251b9-7953-1866-857b-85a162ac79e0: aborting pipeline replication: peer="{Voter server-567ddd18-ef01-cc99-f387-34ac0aa95ed2 567ddd18-ef01-cc99-f387-34ac0aa95ed2}" 2026-04-22T10:09:23.355Z [INFO] server-96e251b9-7953-1866-857b-85a162ac79e0: aborting pipeline replication: peer="{Voter server-ccbf10e9-7aef-6a84-6972-1f6c437570f9 ccbf10e9-7aef-6a84-6972-1f6c437570f9}" --- PASS: TestRaft_JoinNode (0.31s) === RUN TestRaft_JoinNode_ConfigStore 2026-04-22T10:09:23.356Z [INFO] server-a36bc265-32f0-5640-d74f-621e816ac017: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a36bc265-32f0-5640-d74f-621e816ac017 Address:a36bc265-32f0-5640-d74f-621e816ac017}]" 2026-04-22T10:09:23.356Z [INFO] server-a36bc265-32f0-5640-d74f-621e816ac017: entering follower state: follower="Node at a36bc265-32f0-5640-d74f-621e816ac017 [Follower]" leader-address= leader-id= 2026-04-22T10:09:23.356Z [INFO] server-9ba5cedf-8650-8379-8f7b-29b22e9fed8e: initial configuration: index=0 servers=[] 2026-04-22T10:09:23.357Z [INFO] server-f68e78de-7705-7c56-6e40-d5fbcf734a77: initial configuration: index=0 servers=[] 2026-04-22T10:09:23.357Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:23.357Z [INFO] server-9ba5cedf-8650-8379-8f7b-29b22e9fed8e: entering follower state: follower="Node at 9ba5cedf-8650-8379-8f7b-29b22e9fed8e [Follower]" leader-address= leader-id= 2026-04-22T10:09:23.357Z [INFO] server-f68e78de-7705-7c56-6e40-d5fbcf734a77: entering follower state: follower="Node at f68e78de-7705-7c56-6e40-d5fbcf734a77 [Follower]" leader-address= leader-id= 2026-04-22T10:09:23.417Z [WARN] server-a36bc265-32f0-5640-d74f-621e816ac017: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:23.417Z [INFO] server-a36bc265-32f0-5640-d74f-621e816ac017: entering candidate state: node="Node at a36bc265-32f0-5640-d74f-621e816ac017 [Candidate]" term=2 2026-04-22T10:09:23.417Z [INFO] server-a36bc265-32f0-5640-d74f-621e816ac017: election won: term=2 tally=1 2026-04-22T10:09:23.417Z [INFO] server-a36bc265-32f0-5640-d74f-621e816ac017: entering leader state: leader="Node at a36bc265-32f0-5640-d74f-621e816ac017 [Leader]" 2026-04-22T10:09:23.428Z [WARN] server-f68e78de-7705-7c56-6e40-d5fbcf734a77: no known peers, aborting election 2026-04-22T10:09:23.450Z [WARN] server-9ba5cedf-8650-8379-8f7b-29b22e9fed8e: no known peers, aborting election 2026-04-22T10:09:23.525Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:23.417901622 +0000 UTC m=+9.441697713 (1 nodes), 60.750933ms from start of poll, 61.480608ms from cluster start. Timeout at 2026-04-22 10:09:23.522904335 +0000 UTC m=+9.546700500, 105.002787ms after stability 2026-04-22T10:09:23.525Z [INFO] server-a36bc265-32f0-5640-d74f-621e816ac017: updating configuration: command=AddVoter server-id=server-9ba5cedf-8650-8379-8f7b-29b22e9fed8e server-addr=9ba5cedf-8650-8379-8f7b-29b22e9fed8e servers="[{Suffrage:Voter ID:server-a36bc265-32f0-5640-d74f-621e816ac017 Address:a36bc265-32f0-5640-d74f-621e816ac017} {Suffrage:Voter ID:server-9ba5cedf-8650-8379-8f7b-29b22e9fed8e Address:9ba5cedf-8650-8379-8f7b-29b22e9fed8e}]" 2026-04-22T10:09:23.525Z [INFO] server-a36bc265-32f0-5640-d74f-621e816ac017: added peer, starting replication: peer=server-9ba5cedf-8650-8379-8f7b-29b22e9fed8e 2026-04-22T10:09:23.526Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:23.526Z [WARN] server-9ba5cedf-8650-8379-8f7b-29b22e9fed8e: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2026-04-22T10:09:23.526Z [WARN] server-a36bc265-32f0-5640-d74f-621e816ac017: appendEntries rejected, sending older logs: peer="{Voter server-9ba5cedf-8650-8379-8f7b-29b22e9fed8e 9ba5cedf-8650-8379-8f7b-29b22e9fed8e}" next=1 2026-04-22T10:09:23.526Z [INFO] server-a36bc265-32f0-5640-d74f-621e816ac017: pipelining replication: peer="{Voter server-9ba5cedf-8650-8379-8f7b-29b22e9fed8e 9ba5cedf-8650-8379-8f7b-29b22e9fed8e}" 2026-04-22T10:09:23.631Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:23.526050361 +0000 UTC m=+9.549846472 (1 nodes), 917ns from start of poll, 169.629367ms from cluster start. Timeout at 2026-04-22 10:09:23.631054926 +0000 UTC m=+9.654851017, 105.004545ms after stability 2026-04-22T10:09:23.631Z [INFO] server-a36bc265-32f0-5640-d74f-621e816ac017: updating configuration: command=AddVoter server-id=server-f68e78de-7705-7c56-6e40-d5fbcf734a77 server-addr=f68e78de-7705-7c56-6e40-d5fbcf734a77 servers="[{Suffrage:Voter ID:server-a36bc265-32f0-5640-d74f-621e816ac017 Address:a36bc265-32f0-5640-d74f-621e816ac017} {Suffrage:Voter ID:server-9ba5cedf-8650-8379-8f7b-29b22e9fed8e Address:9ba5cedf-8650-8379-8f7b-29b22e9fed8e} {Suffrage:Voter ID:server-f68e78de-7705-7c56-6e40-d5fbcf734a77 Address:f68e78de-7705-7c56-6e40-d5fbcf734a77}]" 2026-04-22T10:09:23.631Z [INFO] server-a36bc265-32f0-5640-d74f-621e816ac017: added peer, starting replication: peer=server-f68e78de-7705-7c56-6e40-d5fbcf734a77 2026-04-22T10:09:23.631Z [WARN] server-f68e78de-7705-7c56-6e40-d5fbcf734a77: failed to get previous log: previous-index=4 last-index=0 error="log not found" 2026-04-22T10:09:23.631Z [WARN] server-a36bc265-32f0-5640-d74f-621e816ac017: appendEntries rejected, sending older logs: peer="{Voter server-f68e78de-7705-7c56-6e40-d5fbcf734a77 f68e78de-7705-7c56-6e40-d5fbcf734a77}" next=1 2026-04-22T10:09:23.631Z [INFO] server-a36bc265-32f0-5640-d74f-621e816ac017: pipelining replication: peer="{Voter server-f68e78de-7705-7c56-6e40-d5fbcf734a77 f68e78de-7705-7c56-6e40-d5fbcf734a77}" 2026-04-22T10:09:23.631Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:23.737Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:23.631843657 +0000 UTC m=+9.655639751 (1 nodes), 900ns from start of poll, 275.422646ms from cluster start. Timeout at 2026-04-22 10:09:23.736849225 +0000 UTC m=+9.760645323, 105.005572ms after stability 2026-04-22T10:09:23.737Z [INFO] server-a36bc265-32f0-5640-d74f-621e816ac017: aborting pipeline replication: peer="{Voter server-9ba5cedf-8650-8379-8f7b-29b22e9fed8e 9ba5cedf-8650-8379-8f7b-29b22e9fed8e}" 2026-04-22T10:09:23.737Z [INFO] server-a36bc265-32f0-5640-d74f-621e816ac017: aborting pipeline replication: peer="{Voter server-f68e78de-7705-7c56-6e40-d5fbcf734a77 f68e78de-7705-7c56-6e40-d5fbcf734a77}" --- PASS: TestRaft_JoinNode_ConfigStore (0.38s) === RUN TestRaft_RemoveFollower 2026-04-22T10:09:23.738Z [INFO] server-9de23053-a6c2-12dd-2043-fdc4836b95a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9de23053-a6c2-12dd-2043-fdc4836b95a7 Address:9de23053-a6c2-12dd-2043-fdc4836b95a7} {Suffrage:Voter ID:server-7834b15f-f109-44af-0c6d-d5d6decc54f2 Address:7834b15f-f109-44af-0c6d-d5d6decc54f2} {Suffrage:Voter ID:server-5fc94849-62b8-4edf-f429-a44e6c269562 Address:5fc94849-62b8-4edf-f429-a44e6c269562}]" 2026-04-22T10:09:23.738Z [INFO] server-9de23053-a6c2-12dd-2043-fdc4836b95a7: entering follower state: follower="Node at 9de23053-a6c2-12dd-2043-fdc4836b95a7 [Follower]" leader-address= leader-id= 2026-04-22T10:09:23.738Z [INFO] server-7834b15f-f109-44af-0c6d-d5d6decc54f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9de23053-a6c2-12dd-2043-fdc4836b95a7 Address:9de23053-a6c2-12dd-2043-fdc4836b95a7} {Suffrage:Voter ID:server-7834b15f-f109-44af-0c6d-d5d6decc54f2 Address:7834b15f-f109-44af-0c6d-d5d6decc54f2} {Suffrage:Voter ID:server-5fc94849-62b8-4edf-f429-a44e6c269562 Address:5fc94849-62b8-4edf-f429-a44e6c269562}]" 2026-04-22T10:09:23.738Z [INFO] server-7834b15f-f109-44af-0c6d-d5d6decc54f2: entering follower state: follower="Node at 7834b15f-f109-44af-0c6d-d5d6decc54f2 [Follower]" leader-address= leader-id= 2026-04-22T10:09:23.738Z [INFO] server-5fc94849-62b8-4edf-f429-a44e6c269562: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9de23053-a6c2-12dd-2043-fdc4836b95a7 Address:9de23053-a6c2-12dd-2043-fdc4836b95a7} {Suffrage:Voter ID:server-7834b15f-f109-44af-0c6d-d5d6decc54f2 Address:7834b15f-f109-44af-0c6d-d5d6decc54f2} {Suffrage:Voter ID:server-5fc94849-62b8-4edf-f429-a44e6c269562 Address:5fc94849-62b8-4edf-f429-a44e6c269562}]" 2026-04-22T10:09:23.738Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:23.738Z [INFO] server-5fc94849-62b8-4edf-f429-a44e6c269562: entering follower state: follower="Node at 5fc94849-62b8-4edf-f429-a44e6c269562 [Follower]" leader-address= leader-id= 2026-04-22T10:09:23.790Z [WARN] server-5fc94849-62b8-4edf-f429-a44e6c269562: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:23.790Z [INFO] server-5fc94849-62b8-4edf-f429-a44e6c269562: entering candidate state: node="Node at 5fc94849-62b8-4edf-f429-a44e6c269562 [Candidate]" term=2 2026-04-22T10:09:23.790Z [INFO] server-5fc94849-62b8-4edf-f429-a44e6c269562: election won: term=2 tally=2 2026-04-22T10:09:23.790Z [INFO] server-5fc94849-62b8-4edf-f429-a44e6c269562: entering leader state: leader="Node at 5fc94849-62b8-4edf-f429-a44e6c269562 [Leader]" 2026-04-22T10:09:23.790Z [INFO] server-5fc94849-62b8-4edf-f429-a44e6c269562: added peer, starting replication: peer=server-9de23053-a6c2-12dd-2043-fdc4836b95a7 2026-04-22T10:09:23.790Z [INFO] server-5fc94849-62b8-4edf-f429-a44e6c269562: added peer, starting replication: peer=server-7834b15f-f109-44af-0c6d-d5d6decc54f2 2026-04-22T10:09:23.791Z [INFO] server-5fc94849-62b8-4edf-f429-a44e6c269562: pipelining replication: peer="{Voter server-7834b15f-f109-44af-0c6d-d5d6decc54f2 7834b15f-f109-44af-0c6d-d5d6decc54f2}" 2026-04-22T10:09:23.791Z [INFO] server-5fc94849-62b8-4edf-f429-a44e6c269562: pipelining replication: peer="{Voter server-9de23053-a6c2-12dd-2043-fdc4836b95a7 9de23053-a6c2-12dd-2043-fdc4836b95a7}" 2026-04-22T10:09:23.896Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:23.791202092 +0000 UTC m=+9.814998202 (1 nodes), 52.288578ms from start of poll, 52.555654ms from cluster start. Timeout at 2026-04-22 10:09:23.896203253 +0000 UTC m=+9.919999340, 105.001138ms after stability 2026-04-22T10:09:23.901Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:24.008Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:23.901923745 +0000 UTC m=+9.925719838 (2 nodes), 1.353µs from start of poll, 163.27729ms from cluster start. Timeout at 2026-04-22 10:09:24.006932065 +0000 UTC m=+10.030728171, 105.008333ms after stability 2026-04-22T10:09:24.008Z [INFO] server-5fc94849-62b8-4edf-f429-a44e6c269562: updating configuration: command=RemoveServer server-id=server-9de23053-a6c2-12dd-2043-fdc4836b95a7 server-addr= servers="[{Suffrage:Voter ID:server-7834b15f-f109-44af-0c6d-d5d6decc54f2 Address:7834b15f-f109-44af-0c6d-d5d6decc54f2} {Suffrage:Voter ID:server-5fc94849-62b8-4edf-f429-a44e6c269562 Address:5fc94849-62b8-4edf-f429-a44e6c269562}]" 2026-04-22T10:09:24.008Z [INFO] server-5fc94849-62b8-4edf-f429-a44e6c269562: removed peer, stopping replication: peer=server-9de23053-a6c2-12dd-2043-fdc4836b95a7 last-index=3 2026-04-22T10:09:24.008Z [INFO] server-5fc94849-62b8-4edf-f429-a44e6c269562: aborting pipeline replication: peer="{Voter server-9de23053-a6c2-12dd-2043-fdc4836b95a7 9de23053-a6c2-12dd-2043-fdc4836b95a7}" 2026-04-22T10:09:24.084Z [WARN] server-9de23053-a6c2-12dd-2043-fdc4836b95a7: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2026-04-22T10:09:24.114Z [INFO] server-5fc94849-62b8-4edf-f429-a44e6c269562: aborting pipeline replication: peer="{Voter server-7834b15f-f109-44af-0c6d-d5d6decc54f2 7834b15f-f109-44af-0c6d-d5d6decc54f2}" --- PASS: TestRaft_RemoveFollower (0.38s) === RUN TestRaft_RemoveLeader 2026-04-22T10:09:24.115Z [INFO] server-6e2a8005-70f4-abd1-fde9-5711dc1e7806: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e2a8005-70f4-abd1-fde9-5711dc1e7806 Address:6e2a8005-70f4-abd1-fde9-5711dc1e7806} {Suffrage:Voter ID:server-e95d6517-f700-823c-e155-07c8d319f1bf Address:e95d6517-f700-823c-e155-07c8d319f1bf} {Suffrage:Voter ID:server-dd64a51e-eb28-25e2-7c64-f19ade66bb87 Address:dd64a51e-eb28-25e2-7c64-f19ade66bb87}]" 2026-04-22T10:09:24.115Z [INFO] server-6e2a8005-70f4-abd1-fde9-5711dc1e7806: entering follower state: follower="Node at 6e2a8005-70f4-abd1-fde9-5711dc1e7806 [Follower]" leader-address= leader-id= 2026-04-22T10:09:24.115Z [INFO] server-e95d6517-f700-823c-e155-07c8d319f1bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e2a8005-70f4-abd1-fde9-5711dc1e7806 Address:6e2a8005-70f4-abd1-fde9-5711dc1e7806} {Suffrage:Voter ID:server-e95d6517-f700-823c-e155-07c8d319f1bf Address:e95d6517-f700-823c-e155-07c8d319f1bf} {Suffrage:Voter ID:server-dd64a51e-eb28-25e2-7c64-f19ade66bb87 Address:dd64a51e-eb28-25e2-7c64-f19ade66bb87}]" 2026-04-22T10:09:24.115Z [INFO] server-e95d6517-f700-823c-e155-07c8d319f1bf: entering follower state: follower="Node at e95d6517-f700-823c-e155-07c8d319f1bf [Follower]" leader-address= leader-id= 2026-04-22T10:09:24.115Z [INFO] server-dd64a51e-eb28-25e2-7c64-f19ade66bb87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e2a8005-70f4-abd1-fde9-5711dc1e7806 Address:6e2a8005-70f4-abd1-fde9-5711dc1e7806} {Suffrage:Voter ID:server-e95d6517-f700-823c-e155-07c8d319f1bf Address:e95d6517-f700-823c-e155-07c8d319f1bf} {Suffrage:Voter ID:server-dd64a51e-eb28-25e2-7c64-f19ade66bb87 Address:dd64a51e-eb28-25e2-7c64-f19ade66bb87}]" 2026-04-22T10:09:24.115Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:24.116Z [INFO] server-dd64a51e-eb28-25e2-7c64-f19ade66bb87: entering follower state: follower="Node at dd64a51e-eb28-25e2-7c64-f19ade66bb87 [Follower]" leader-address= leader-id= 2026-04-22T10:09:24.171Z [WARN] server-e95d6517-f700-823c-e155-07c8d319f1bf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:24.171Z [INFO] server-e95d6517-f700-823c-e155-07c8d319f1bf: entering candidate state: node="Node at e95d6517-f700-823c-e155-07c8d319f1bf [Candidate]" term=2 2026-04-22T10:09:24.171Z [INFO] server-e95d6517-f700-823c-e155-07c8d319f1bf: election won: term=2 tally=2 2026-04-22T10:09:24.171Z [INFO] server-e95d6517-f700-823c-e155-07c8d319f1bf: entering leader state: leader="Node at e95d6517-f700-823c-e155-07c8d319f1bf [Leader]" 2026-04-22T10:09:24.171Z [INFO] server-e95d6517-f700-823c-e155-07c8d319f1bf: added peer, starting replication: peer=server-6e2a8005-70f4-abd1-fde9-5711dc1e7806 2026-04-22T10:09:24.171Z [INFO] server-e95d6517-f700-823c-e155-07c8d319f1bf: added peer, starting replication: peer=server-dd64a51e-eb28-25e2-7c64-f19ade66bb87 2026-04-22T10:09:24.171Z [INFO] server-e95d6517-f700-823c-e155-07c8d319f1bf: pipelining replication: peer="{Voter server-dd64a51e-eb28-25e2-7c64-f19ade66bb87 dd64a51e-eb28-25e2-7c64-f19ade66bb87}" 2026-04-22T10:09:24.171Z [INFO] server-e95d6517-f700-823c-e155-07c8d319f1bf: pipelining replication: peer="{Voter server-6e2a8005-70f4-abd1-fde9-5711dc1e7806 6e2a8005-70f4-abd1-fde9-5711dc1e7806}" 2026-04-22T10:09:24.277Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:24.171511382 +0000 UTC m=+10.195307528 (1 nodes), 55.503459ms from start of poll, 55.899183ms from cluster start. Timeout at 2026-04-22 10:09:24.276515385 +0000 UTC m=+10.300311468, 105.00394ms after stability 2026-04-22T10:09:24.282Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:24.388Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:24.282487251 +0000 UTC m=+10.306283368 (2 nodes), 1.234µs from start of poll, 166.875023ms from cluster start. Timeout at 2026-04-22 10:09:24.387505891 +0000 UTC m=+10.411302022, 105.018654ms after stability 2026-04-22T10:09:24.388Z [INFO] server-e95d6517-f700-823c-e155-07c8d319f1bf: updating configuration: command=RemoveServer server-id=server-e95d6517-f700-823c-e155-07c8d319f1bf server-addr= servers="[{Suffrage:Voter ID:server-6e2a8005-70f4-abd1-fde9-5711dc1e7806 Address:6e2a8005-70f4-abd1-fde9-5711dc1e7806} {Suffrage:Voter ID:server-dd64a51e-eb28-25e2-7c64-f19ade66bb87 Address:dd64a51e-eb28-25e2-7c64-f19ade66bb87}]" 2026-04-22T10:09:24.388Z [INFO] server-e95d6517-f700-823c-e155-07c8d319f1bf: removed ourself, shutting down 2026-04-22T10:09:24.388Z [INFO] server-e95d6517-f700-823c-e155-07c8d319f1bf: aborting pipeline replication: peer="{Voter server-dd64a51e-eb28-25e2-7c64-f19ade66bb87 dd64a51e-eb28-25e2-7c64-f19ade66bb87}" 2026-04-22T10:09:24.389Z [INFO] server-e95d6517-f700-823c-e155-07c8d319f1bf: aborting pipeline replication: peer="{Voter server-6e2a8005-70f4-abd1-fde9-5711dc1e7806 6e2a8005-70f4-abd1-fde9-5711dc1e7806}" 2026-04-22T10:09:24.477Z [WARN] server-6e2a8005-70f4-abd1-fde9-5711dc1e7806: heartbeat timeout reached, starting election: last-leader-addr=e95d6517-f700-823c-e155-07c8d319f1bf last-leader-id=server-e95d6517-f700-823c-e155-07c8d319f1bf 2026-04-22T10:09:24.477Z [INFO] server-6e2a8005-70f4-abd1-fde9-5711dc1e7806: entering candidate state: node="Node at 6e2a8005-70f4-abd1-fde9-5711dc1e7806 [Candidate]" term=3 2026-04-22T10:09:24.477Z [WARN] server-dd64a51e-eb28-25e2-7c64-f19ade66bb87: rejecting vote request since we have a leader: from=6e2a8005-70f4-abd1-fde9-5711dc1e7806 leader=e95d6517-f700-823c-e155-07c8d319f1bf leader-id=server-e95d6517-f700-823c-e155-07c8d319f1bf 2026-04-22T10:09:24.491Z [WARN] server-dd64a51e-eb28-25e2-7c64-f19ade66bb87: heartbeat timeout reached, starting election: last-leader-addr=e95d6517-f700-823c-e155-07c8d319f1bf last-leader-id=server-e95d6517-f700-823c-e155-07c8d319f1bf 2026-04-22T10:09:24.491Z [INFO] server-dd64a51e-eb28-25e2-7c64-f19ade66bb87: entering candidate state: node="Node at dd64a51e-eb28-25e2-7c64-f19ade66bb87 [Candidate]" term=3 2026-04-22T10:09:24.491Z [INFO] server-6e2a8005-70f4-abd1-fde9-5711dc1e7806: duplicate requestVote for same term: term=3 2026-04-22T10:09:24.544Z [WARN] server-6e2a8005-70f4-abd1-fde9-5711dc1e7806: Election timeout reached, restarting election 2026-04-22T10:09:24.544Z [INFO] server-6e2a8005-70f4-abd1-fde9-5711dc1e7806: entering candidate state: node="Node at 6e2a8005-70f4-abd1-fde9-5711dc1e7806 [Candidate]" term=4 2026-04-22T10:09:24.544Z [INFO] server-dd64a51e-eb28-25e2-7c64-f19ade66bb87: entering follower state: follower="Node at dd64a51e-eb28-25e2-7c64-f19ade66bb87 [Follower]" leader-address= leader-id= 2026-04-22T10:09:24.544Z [INFO] server-6e2a8005-70f4-abd1-fde9-5711dc1e7806: election won: term=4 tally=2 2026-04-22T10:09:24.544Z [INFO] server-6e2a8005-70f4-abd1-fde9-5711dc1e7806: entering leader state: leader="Node at 6e2a8005-70f4-abd1-fde9-5711dc1e7806 [Leader]" 2026-04-22T10:09:24.544Z [INFO] server-6e2a8005-70f4-abd1-fde9-5711dc1e7806: added peer, starting replication: peer=server-dd64a51e-eb28-25e2-7c64-f19ade66bb87 2026-04-22T10:09:24.545Z [INFO] server-6e2a8005-70f4-abd1-fde9-5711dc1e7806: pipelining replication: peer="{Voter server-dd64a51e-eb28-25e2-7c64-f19ade66bb87 dd64a51e-eb28-25e2-7c64-f19ade66bb87}" 2026-04-22T10:09:24.602Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:24.708Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:24.602897401 +0000 UTC m=+10.626693504 (1 nodes), 51.211µs from start of poll, 487.285159ms from cluster start. Timeout at 2026-04-22 10:09:24.707926939 +0000 UTC m=+10.731723062, 105.029558ms after stability 2026-04-22T10:09:24.708Z [INFO] server-6e2a8005-70f4-abd1-fde9-5711dc1e7806: aborting pipeline replication: peer="{Voter server-dd64a51e-eb28-25e2-7c64-f19ade66bb87 dd64a51e-eb28-25e2-7c64-f19ade66bb87}" --- PASS: TestRaft_RemoveLeader (0.60s) === RUN TestRaft_RemoveLeader_NoShutdown 2026-04-22T10:09:24.711Z [INFO] server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33 Address:e59d6a67-e1dd-ff37-7dbe-3f8777f65d33} {Suffrage:Voter ID:server-e43f3ac2-a129-5bf7-3b56-66e800a4743a Address:e43f3ac2-a129-5bf7-3b56-66e800a4743a} {Suffrage:Voter ID:server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd Address:92e2ca1b-e1a6-0667-0632-68d88a66d4fd}]" 2026-04-22T10:09:24.711Z [INFO] server-e43f3ac2-a129-5bf7-3b56-66e800a4743a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33 Address:e59d6a67-e1dd-ff37-7dbe-3f8777f65d33} {Suffrage:Voter ID:server-e43f3ac2-a129-5bf7-3b56-66e800a4743a Address:e43f3ac2-a129-5bf7-3b56-66e800a4743a} {Suffrage:Voter ID:server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd Address:92e2ca1b-e1a6-0667-0632-68d88a66d4fd}]" 2026-04-22T10:09:24.711Z [INFO] server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33 Address:e59d6a67-e1dd-ff37-7dbe-3f8777f65d33} {Suffrage:Voter ID:server-e43f3ac2-a129-5bf7-3b56-66e800a4743a Address:e43f3ac2-a129-5bf7-3b56-66e800a4743a} {Suffrage:Voter ID:server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd Address:92e2ca1b-e1a6-0667-0632-68d88a66d4fd}]" 2026-04-22T10:09:24.711Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:24.712Z [INFO] server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33: entering follower state: follower="Node at e59d6a67-e1dd-ff37-7dbe-3f8777f65d33 [Follower]" leader-address= leader-id= 2026-04-22T10:09:24.712Z [INFO] server-e43f3ac2-a129-5bf7-3b56-66e800a4743a: entering follower state: follower="Node at e43f3ac2-a129-5bf7-3b56-66e800a4743a [Follower]" leader-address= leader-id= 2026-04-22T10:09:24.712Z [INFO] server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd: entering follower state: follower="Node at 92e2ca1b-e1a6-0667-0632-68d88a66d4fd [Follower]" leader-address= leader-id= 2026-04-22T10:09:24.768Z [WARN] server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:24.768Z [INFO] server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd: entering candidate state: node="Node at 92e2ca1b-e1a6-0667-0632-68d88a66d4fd [Candidate]" term=2 2026-04-22T10:09:24.768Z [INFO] server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd: election won: term=2 tally=2 2026-04-22T10:09:24.768Z [INFO] server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd: entering leader state: leader="Node at 92e2ca1b-e1a6-0667-0632-68d88a66d4fd [Leader]" 2026-04-22T10:09:24.768Z [INFO] server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd: added peer, starting replication: peer=server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33 2026-04-22T10:09:24.768Z [INFO] server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd: added peer, starting replication: peer=server-e43f3ac2-a129-5bf7-3b56-66e800a4743a 2026-04-22T10:09:24.768Z [INFO] server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd: pipelining replication: peer="{Voter server-e43f3ac2-a129-5bf7-3b56-66e800a4743a e43f3ac2-a129-5bf7-3b56-66e800a4743a}" 2026-04-22T10:09:24.768Z [INFO] server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd: pipelining replication: peer="{Voter server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33 e59d6a67-e1dd-ff37-7dbe-3f8777f65d33}" 2026-04-22T10:09:24.874Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:24.768979347 +0000 UTC m=+10.792775460 (2 nodes), 56.973137ms from start of poll, 57.336611ms from cluster start. Timeout at 2026-04-22 10:09:24.873986189 +0000 UTC m=+10.897782312, 105.006852ms after stability 2026-04-22T10:09:24.874Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:24.981Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:24.874504689 +0000 UTC m=+10.898300796 (1 nodes), 1.589µs from start of poll, 162.861947ms from cluster start. Timeout at 2026-04-22 10:09:24.97951502 +0000 UTC m=+11.003311148, 105.010352ms after stability 2026-04-22T10:09:24.991Z [INFO] server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd: updating configuration: command=RemoveServer server-id=server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd server-addr= servers="[{Suffrage:Voter ID:server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33 Address:e59d6a67-e1dd-ff37-7dbe-3f8777f65d33} {Suffrage:Voter ID:server-e43f3ac2-a129-5bf7-3b56-66e800a4743a Address:e43f3ac2-a129-5bf7-3b56-66e800a4743a}]" 2026-04-22T10:09:24.992Z [INFO] server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd: removed ourself, transitioning to follower 2026-04-22T10:09:24.992Z [INFO] server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd: entering follower state: follower="Node at 92e2ca1b-e1a6-0667-0632-68d88a66d4fd [Follower]" leader-address= leader-id= 2026-04-22T10:09:24.992Z [INFO] server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd: aborting pipeline replication: peer="{Voter server-e43f3ac2-a129-5bf7-3b56-66e800a4743a e43f3ac2-a129-5bf7-3b56-66e800a4743a}" 2026-04-22T10:09:24.992Z [INFO] server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd: aborting pipeline replication: peer="{Voter server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33 e59d6a67-e1dd-ff37-7dbe-3f8777f65d33}" 2026-04-22T10:09:25.047Z [WARN] server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33: heartbeat timeout reached, starting election: last-leader-addr=92e2ca1b-e1a6-0667-0632-68d88a66d4fd last-leader-id=server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd 2026-04-22T10:09:25.047Z [INFO] server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33: entering candidate state: node="Node at e59d6a67-e1dd-ff37-7dbe-3f8777f65d33 [Candidate]" term=3 2026-04-22T10:09:25.047Z [WARN] server-e43f3ac2-a129-5bf7-3b56-66e800a4743a: rejecting vote request since we have a leader: from=e59d6a67-e1dd-ff37-7dbe-3f8777f65d33 leader=92e2ca1b-e1a6-0667-0632-68d88a66d4fd leader-id=server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd 2026-04-22T10:09:25.073Z [WARN] server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd: not part of stable configuration, aborting election 2026-04-22T10:09:25.097Z [WARN] server-e43f3ac2-a129-5bf7-3b56-66e800a4743a: heartbeat timeout reached, starting election: last-leader-addr=92e2ca1b-e1a6-0667-0632-68d88a66d4fd last-leader-id=server-92e2ca1b-e1a6-0667-0632-68d88a66d4fd 2026-04-22T10:09:25.097Z [INFO] server-e43f3ac2-a129-5bf7-3b56-66e800a4743a: entering candidate state: node="Node at e43f3ac2-a129-5bf7-3b56-66e800a4743a [Candidate]" term=3 2026-04-22T10:09:25.097Z [INFO] server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33: duplicate requestVote for same term: term=3 2026-04-22T10:09:25.097Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:25.105Z [WARN] server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33: Election timeout reached, restarting election 2026-04-22T10:09:25.105Z [INFO] server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33: entering candidate state: node="Node at e59d6a67-e1dd-ff37-7dbe-3f8777f65d33 [Candidate]" term=4 2026-04-22T10:09:25.105Z [INFO] server-e43f3ac2-a129-5bf7-3b56-66e800a4743a: entering follower state: follower="Node at e43f3ac2-a129-5bf7-3b56-66e800a4743a [Follower]" leader-address= leader-id= 2026-04-22T10:09:25.105Z [INFO] server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33: election won: term=4 tally=2 2026-04-22T10:09:25.105Z [INFO] server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33: entering leader state: leader="Node at e59d6a67-e1dd-ff37-7dbe-3f8777f65d33 [Leader]" 2026-04-22T10:09:25.105Z [INFO] server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33: added peer, starting replication: peer=server-e43f3ac2-a129-5bf7-3b56-66e800a4743a 2026-04-22T10:09:25.105Z [INFO] server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33: pipelining replication: peer="{Voter server-e43f3ac2-a129-5bf7-3b56-66e800a4743a e43f3ac2-a129-5bf7-3b56-66e800a4743a}" 2026-04-22T10:09:25.649Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:25.10599145 +0000 UTC m=+11.129787579 (1 nodes), 8.259393ms from start of poll, 394.34873ms from cluster start. Timeout at 2026-04-22 10:09:25.211005561 +0000 UTC m=+11.234801674, 105.014095ms after stability 2026-04-22T10:09:25.757Z [INFO] server-e59d6a67-e1dd-ff37-7dbe-3f8777f65d33: aborting pipeline replication: peer="{Voter server-e43f3ac2-a129-5bf7-3b56-66e800a4743a e43f3ac2-a129-5bf7-3b56-66e800a4743a}" --- PASS: TestRaft_RemoveLeader_NoShutdown (1.05s) === RUN TestRaft_RemoveFollower_SplitCluster 2026-04-22T10:09:25.759Z [INFO] server-022c65c4-5b03-4100-9d02-329c05d6d9e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-022c65c4-5b03-4100-9d02-329c05d6d9e8 Address:022c65c4-5b03-4100-9d02-329c05d6d9e8} {Suffrage:Voter ID:server-adffe494-0224-4fde-2822-ff5110673309 Address:adffe494-0224-4fde-2822-ff5110673309} {Suffrage:Voter ID:server-6e2809f4-b3fe-466e-5eb3-6f6394030c30 Address:6e2809f4-b3fe-466e-5eb3-6f6394030c30} {Suffrage:Voter ID:server-a6e2f418-d993-1a67-ab11-d1d8599c19bc Address:a6e2f418-d993-1a67-ab11-d1d8599c19bc}]" 2026-04-22T10:09:25.759Z [INFO] server-adffe494-0224-4fde-2822-ff5110673309: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-022c65c4-5b03-4100-9d02-329c05d6d9e8 Address:022c65c4-5b03-4100-9d02-329c05d6d9e8} {Suffrage:Voter ID:server-adffe494-0224-4fde-2822-ff5110673309 Address:adffe494-0224-4fde-2822-ff5110673309} {Suffrage:Voter ID:server-6e2809f4-b3fe-466e-5eb3-6f6394030c30 Address:6e2809f4-b3fe-466e-5eb3-6f6394030c30} {Suffrage:Voter ID:server-a6e2f418-d993-1a67-ab11-d1d8599c19bc Address:a6e2f418-d993-1a67-ab11-d1d8599c19bc}]" 2026-04-22T10:09:25.759Z [INFO] server-6e2809f4-b3fe-466e-5eb3-6f6394030c30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-022c65c4-5b03-4100-9d02-329c05d6d9e8 Address:022c65c4-5b03-4100-9d02-329c05d6d9e8} {Suffrage:Voter ID:server-adffe494-0224-4fde-2822-ff5110673309 Address:adffe494-0224-4fde-2822-ff5110673309} {Suffrage:Voter ID:server-6e2809f4-b3fe-466e-5eb3-6f6394030c30 Address:6e2809f4-b3fe-466e-5eb3-6f6394030c30} {Suffrage:Voter ID:server-a6e2f418-d993-1a67-ab11-d1d8599c19bc Address:a6e2f418-d993-1a67-ab11-d1d8599c19bc}]" 2026-04-22T10:09:25.759Z [INFO] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-022c65c4-5b03-4100-9d02-329c05d6d9e8 Address:022c65c4-5b03-4100-9d02-329c05d6d9e8} {Suffrage:Voter ID:server-adffe494-0224-4fde-2822-ff5110673309 Address:adffe494-0224-4fde-2822-ff5110673309} {Suffrage:Voter ID:server-6e2809f4-b3fe-466e-5eb3-6f6394030c30 Address:6e2809f4-b3fe-466e-5eb3-6f6394030c30} {Suffrage:Voter ID:server-a6e2f418-d993-1a67-ab11-d1d8599c19bc Address:a6e2f418-d993-1a67-ab11-d1d8599c19bc}]" 2026-04-22T10:09:25.759Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:25.759Z [INFO] server-022c65c4-5b03-4100-9d02-329c05d6d9e8: entering follower state: follower="Node at 022c65c4-5b03-4100-9d02-329c05d6d9e8 [Follower]" leader-address= leader-id= 2026-04-22T10:09:25.759Z [INFO] server-adffe494-0224-4fde-2822-ff5110673309: entering follower state: follower="Node at adffe494-0224-4fde-2822-ff5110673309 [Follower]" leader-address= leader-id= 2026-04-22T10:09:25.759Z [INFO] server-6e2809f4-b3fe-466e-5eb3-6f6394030c30: entering follower state: follower="Node at 6e2809f4-b3fe-466e-5eb3-6f6394030c30 [Follower]" leader-address= leader-id= 2026-04-22T10:09:25.759Z [INFO] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: entering follower state: follower="Node at a6e2f418-d993-1a67-ab11-d1d8599c19bc [Follower]" leader-address= leader-id= 2026-04-22T10:09:25.829Z [WARN] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:25.829Z [INFO] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: entering candidate state: node="Node at a6e2f418-d993-1a67-ab11-d1d8599c19bc [Candidate]" term=2 2026-04-22T10:09:25.829Z [WARN] server-6e2809f4-b3fe-466e-5eb3-6f6394030c30: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:25.829Z [INFO] server-6e2809f4-b3fe-466e-5eb3-6f6394030c30: entering candidate state: node="Node at 6e2809f4-b3fe-466e-5eb3-6f6394030c30 [Candidate]" term=2 2026-04-22T10:09:25.829Z [INFO] server-6e2809f4-b3fe-466e-5eb3-6f6394030c30: duplicate requestVote for same term: term=2 2026-04-22T10:09:25.829Z [INFO] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: election won: term=2 tally=3 2026-04-22T10:09:25.829Z [INFO] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: entering leader state: leader="Node at a6e2f418-d993-1a67-ab11-d1d8599c19bc [Leader]" 2026-04-22T10:09:25.830Z [INFO] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: added peer, starting replication: peer=server-022c65c4-5b03-4100-9d02-329c05d6d9e8 2026-04-22T10:09:25.830Z [INFO] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: added peer, starting replication: peer=server-adffe494-0224-4fde-2822-ff5110673309 2026-04-22T10:09:25.830Z [INFO] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: added peer, starting replication: peer=server-6e2809f4-b3fe-466e-5eb3-6f6394030c30 2026-04-22T10:09:25.830Z [INFO] server-6e2809f4-b3fe-466e-5eb3-6f6394030c30: entering follower state: follower="Node at 6e2809f4-b3fe-466e-5eb3-6f6394030c30 [Follower]" leader-address=a6e2f418-d993-1a67-ab11-d1d8599c19bc leader-id=server-a6e2f418-d993-1a67-ab11-d1d8599c19bc 2026-04-22T10:09:25.830Z [INFO] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: pipelining replication: peer="{Voter server-6e2809f4-b3fe-466e-5eb3-6f6394030c30 6e2809f4-b3fe-466e-5eb3-6f6394030c30}" 2026-04-22T10:09:25.830Z [INFO] server-022c65c4-5b03-4100-9d02-329c05d6d9e8: duplicate requestVote for same term: term=2 2026-04-22T10:09:25.830Z [INFO] server-adffe494-0224-4fde-2822-ff5110673309: duplicate requestVote for same term: term=2 2026-04-22T10:09:25.830Z [WARN] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: rejecting vote request since we have a leader: from=6e2809f4-b3fe-466e-5eb3-6f6394030c30 leader=a6e2f418-d993-1a67-ab11-d1d8599c19bc leader-id=server-a6e2f418-d993-1a67-ab11-d1d8599c19bc 2026-04-22T10:09:25.830Z [INFO] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: pipelining replication: peer="{Voter server-022c65c4-5b03-4100-9d02-329c05d6d9e8 022c65c4-5b03-4100-9d02-329c05d6d9e8}" 2026-04-22T10:09:25.830Z [INFO] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: pipelining replication: peer="{Voter server-adffe494-0224-4fde-2822-ff5110673309 adffe494-0224-4fde-2822-ff5110673309}" 2026-04-22T10:09:25.935Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:25.830343347 +0000 UTC m=+11.854139451 (1 nodes), 70.860199ms from start of poll, 71.332163ms from cluster start. Timeout at 2026-04-22 10:09:25.935344576 +0000 UTC m=+11.959140701, 105.00125ms after stability 2026-04-22T10:09:26.043Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:26.151Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:26.043717356 +0000 UTC m=+12.067513472 (3 nodes), 1.597µs from start of poll, 284.706184ms from cluster start. Timeout at 2026-04-22 10:09:26.148728849 +0000 UTC m=+12.172524959, 105.011487ms after stability 2026-04-22T10:09:26.151Z [INFO] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: updating configuration: command=RemoveServer server-id=server-6e2809f4-b3fe-466e-5eb3-6f6394030c30 server-addr= servers="[{Suffrage:Voter ID:server-022c65c4-5b03-4100-9d02-329c05d6d9e8 Address:022c65c4-5b03-4100-9d02-329c05d6d9e8} {Suffrage:Voter ID:server-adffe494-0224-4fde-2822-ff5110673309 Address:adffe494-0224-4fde-2822-ff5110673309} {Suffrage:Voter ID:server-a6e2f418-d993-1a67-ab11-d1d8599c19bc Address:a6e2f418-d993-1a67-ab11-d1d8599c19bc}]" 2026-04-22T10:09:26.152Z [INFO] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: removed peer, stopping replication: peer=server-6e2809f4-b3fe-466e-5eb3-6f6394030c30 last-index=3 2026-04-22T10:09:26.152Z [INFO] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: aborting pipeline replication: peer="{Voter server-6e2809f4-b3fe-466e-5eb3-6f6394030c30 6e2809f4-b3fe-466e-5eb3-6f6394030c30}" 2026-04-22T10:09:26.152Z [ERROR] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to pipeline appendEntries: peer="{Voter server-022c65c4-5b03-4100-9d02-329c05d6d9e8 022c65c4-5b03-4100-9d02-329c05d6d9e8}" error="append pipeline closed" 2026-04-22T10:09:26.152Z [INFO] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: aborting pipeline replication: peer="{Voter server-022c65c4-5b03-4100-9d02-329c05d6d9e8 022c65c4-5b03-4100-9d02-329c05d6d9e8}" 2026-04-22T10:09:26.152Z [ERROR] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to pipeline appendEntries: peer="{Voter server-adffe494-0224-4fde-2822-ff5110673309 adffe494-0224-4fde-2822-ff5110673309}" error="append pipeline closed" 2026-04-22T10:09:26.152Z [INFO] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: aborting pipeline replication: peer="{Voter server-adffe494-0224-4fde-2822-ff5110673309 adffe494-0224-4fde-2822-ff5110673309}" 2026-04-22T10:09:26.157Z [ERROR] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to heartbeat to: peer=adffe494-0224-4fde-2822-ff5110673309 backoff time=10ms error="failed to connect to peer: adffe494-0224-4fde-2822-ff5110673309" 2026-04-22T10:09:26.157Z [ERROR] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to heartbeat to: peer=022c65c4-5b03-4100-9d02-329c05d6d9e8 backoff time=10ms error="failed to connect to peer: 022c65c4-5b03-4100-9d02-329c05d6d9e8" 2026-04-22T10:09:26.161Z [ERROR] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to appendEntries to: peer="{Voter server-022c65c4-5b03-4100-9d02-329c05d6d9e8 022c65c4-5b03-4100-9d02-329c05d6d9e8}" error="failed to connect to peer: 022c65c4-5b03-4100-9d02-329c05d6d9e8" 2026-04-22T10:09:26.161Z [ERROR] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to appendEntries to: peer="{Voter server-adffe494-0224-4fde-2822-ff5110673309 adffe494-0224-4fde-2822-ff5110673309}" error="failed to connect to peer: adffe494-0224-4fde-2822-ff5110673309" 2026-04-22T10:09:26.176Z [ERROR] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to heartbeat to: peer=022c65c4-5b03-4100-9d02-329c05d6d9e8 backoff time=10ms error="failed to connect to peer: 022c65c4-5b03-4100-9d02-329c05d6d9e8" 2026-04-22T10:09:26.177Z [ERROR] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to heartbeat to: peer=adffe494-0224-4fde-2822-ff5110673309 backoff time=10ms error="failed to connect to peer: adffe494-0224-4fde-2822-ff5110673309" 2026-04-22T10:09:26.184Z [ERROR] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to appendEntries to: peer="{Voter server-022c65c4-5b03-4100-9d02-329c05d6d9e8 022c65c4-5b03-4100-9d02-329c05d6d9e8}" error="failed to connect to peer: 022c65c4-5b03-4100-9d02-329c05d6d9e8" 2026-04-22T10:09:26.184Z [ERROR] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to appendEntries to: peer="{Voter server-adffe494-0224-4fde-2822-ff5110673309 adffe494-0224-4fde-2822-ff5110673309}" error="failed to connect to peer: adffe494-0224-4fde-2822-ff5110673309" 2026-04-22T10:09:26.194Z [ERROR] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to heartbeat to: peer=022c65c4-5b03-4100-9d02-329c05d6d9e8 backoff time=10ms error="failed to connect to peer: 022c65c4-5b03-4100-9d02-329c05d6d9e8" 2026-04-22T10:09:26.195Z [WARN] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to contact: server-id=server-022c65c4-5b03-4100-9d02-329c05d6d9e8 time=50.448013ms 2026-04-22T10:09:26.199Z [ERROR] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to heartbeat to: peer=adffe494-0224-4fde-2822-ff5110673309 backoff time=10ms error="failed to connect to peer: adffe494-0224-4fde-2822-ff5110673309" 2026-04-22T10:09:26.201Z [ERROR] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to appendEntries to: peer="{Voter server-022c65c4-5b03-4100-9d02-329c05d6d9e8 022c65c4-5b03-4100-9d02-329c05d6d9e8}" error="failed to connect to peer: 022c65c4-5b03-4100-9d02-329c05d6d9e8" 2026-04-22T10:09:26.205Z [ERROR] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to appendEntries to: peer="{Voter server-adffe494-0224-4fde-2822-ff5110673309 adffe494-0224-4fde-2822-ff5110673309}" error="failed to connect to peer: adffe494-0224-4fde-2822-ff5110673309" 2026-04-22T10:09:26.206Z [WARN] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to contact: server-id=server-022c65c4-5b03-4100-9d02-329c05d6d9e8 time=61.440102ms 2026-04-22T10:09:26.206Z [WARN] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to contact: server-id=server-adffe494-0224-4fde-2822-ff5110673309 time=55.284054ms 2026-04-22T10:09:26.206Z [WARN] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: failed to contact quorum of nodes, stepping down 2026-04-22T10:09:26.207Z [INFO] server-a6e2f418-d993-1a67-ab11-d1d8599c19bc: entering follower state: follower="Node at a6e2f418-d993-1a67-ab11-d1d8599c19bc [Follower]" leader-address= leader-id= --- PASS: TestRaft_RemoveFollower_SplitCluster (0.45s) === RUN TestRaft_AddKnownPeer 2026-04-22T10:09:26.208Z [INFO] server-21405755-d305-b391-a77a-024ad6d06275: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21405755-d305-b391-a77a-024ad6d06275 Address:21405755-d305-b391-a77a-024ad6d06275} {Suffrage:Voter ID:server-ae32f513-0d09-fbbe-fe92-87e2d4c2710e Address:ae32f513-0d09-fbbe-fe92-87e2d4c2710e} {Suffrage:Voter ID:server-a2243f9e-153c-770f-4ee7-62b06de9eba6 Address:a2243f9e-153c-770f-4ee7-62b06de9eba6}]" 2026-04-22T10:09:26.208Z [INFO] server-ae32f513-0d09-fbbe-fe92-87e2d4c2710e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21405755-d305-b391-a77a-024ad6d06275 Address:21405755-d305-b391-a77a-024ad6d06275} {Suffrage:Voter ID:server-ae32f513-0d09-fbbe-fe92-87e2d4c2710e Address:ae32f513-0d09-fbbe-fe92-87e2d4c2710e} {Suffrage:Voter ID:server-a2243f9e-153c-770f-4ee7-62b06de9eba6 Address:a2243f9e-153c-770f-4ee7-62b06de9eba6}]" 2026-04-22T10:09:26.208Z [INFO] server-a2243f9e-153c-770f-4ee7-62b06de9eba6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21405755-d305-b391-a77a-024ad6d06275 Address:21405755-d305-b391-a77a-024ad6d06275} {Suffrage:Voter ID:server-ae32f513-0d09-fbbe-fe92-87e2d4c2710e Address:ae32f513-0d09-fbbe-fe92-87e2d4c2710e} {Suffrage:Voter ID:server-a2243f9e-153c-770f-4ee7-62b06de9eba6 Address:a2243f9e-153c-770f-4ee7-62b06de9eba6}]" 2026-04-22T10:09:26.208Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:26.208Z [INFO] server-21405755-d305-b391-a77a-024ad6d06275: entering follower state: follower="Node at 21405755-d305-b391-a77a-024ad6d06275 [Follower]" leader-address= leader-id= 2026-04-22T10:09:26.208Z [INFO] server-ae32f513-0d09-fbbe-fe92-87e2d4c2710e: entering follower state: follower="Node at ae32f513-0d09-fbbe-fe92-87e2d4c2710e [Follower]" leader-address= leader-id= 2026-04-22T10:09:26.208Z [INFO] server-a2243f9e-153c-770f-4ee7-62b06de9eba6: entering follower state: follower="Node at a2243f9e-153c-770f-4ee7-62b06de9eba6 [Follower]" leader-address= leader-id= 2026-04-22T10:09:26.269Z [WARN] server-21405755-d305-b391-a77a-024ad6d06275: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:26.269Z [INFO] server-21405755-d305-b391-a77a-024ad6d06275: entering candidate state: node="Node at 21405755-d305-b391-a77a-024ad6d06275 [Candidate]" term=2 2026-04-22T10:09:26.269Z [INFO] server-21405755-d305-b391-a77a-024ad6d06275: election won: term=2 tally=2 2026-04-22T10:09:26.269Z [INFO] server-21405755-d305-b391-a77a-024ad6d06275: entering leader state: leader="Node at 21405755-d305-b391-a77a-024ad6d06275 [Leader]" 2026-04-22T10:09:26.269Z [INFO] server-21405755-d305-b391-a77a-024ad6d06275: added peer, starting replication: peer=server-ae32f513-0d09-fbbe-fe92-87e2d4c2710e 2026-04-22T10:09:26.269Z [INFO] server-21405755-d305-b391-a77a-024ad6d06275: added peer, starting replication: peer=server-a2243f9e-153c-770f-4ee7-62b06de9eba6 2026-04-22T10:09:26.270Z [INFO] server-21405755-d305-b391-a77a-024ad6d06275: pipelining replication: peer="{Voter server-a2243f9e-153c-770f-4ee7-62b06de9eba6 a2243f9e-153c-770f-4ee7-62b06de9eba6}" 2026-04-22T10:09:26.270Z [INFO] server-21405755-d305-b391-a77a-024ad6d06275: pipelining replication: peer="{Voter server-ae32f513-0d09-fbbe-fe92-87e2d4c2710e ae32f513-0d09-fbbe-fe92-87e2d4c2710e}" 2026-04-22T10:09:26.377Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:26.270625208 +0000 UTC m=+12.294421326 (1 nodes), 61.936392ms from start of poll, 62.28398ms from cluster start. Timeout at 2026-04-22 10:09:26.375630407 +0000 UTC m=+12.399426497, 105.005171ms after stability 2026-04-22T10:09:26.377Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:26.483Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:26.377704189 +0000 UTC m=+12.401500304 (2 nodes), 1.244µs from start of poll, 169.362958ms from cluster start. Timeout at 2026-04-22 10:09:26.482718966 +0000 UTC m=+12.506515107, 105.014803ms after stability 2026-04-22T10:09:26.483Z [INFO] server-21405755-d305-b391-a77a-024ad6d06275: updating configuration: command=AddVoter server-id=server-ae32f513-0d09-fbbe-fe92-87e2d4c2710e server-addr=ae32f513-0d09-fbbe-fe92-87e2d4c2710e servers="[{Suffrage:Voter ID:server-21405755-d305-b391-a77a-024ad6d06275 Address:21405755-d305-b391-a77a-024ad6d06275} {Suffrage:Voter ID:server-ae32f513-0d09-fbbe-fe92-87e2d4c2710e Address:ae32f513-0d09-fbbe-fe92-87e2d4c2710e} {Suffrage:Voter ID:server-a2243f9e-153c-770f-4ee7-62b06de9eba6 Address:a2243f9e-153c-770f-4ee7-62b06de9eba6}]" 2026-04-22T10:09:26.483Z [INFO] server-21405755-d305-b391-a77a-024ad6d06275: aborting pipeline replication: peer="{Voter server-a2243f9e-153c-770f-4ee7-62b06de9eba6 a2243f9e-153c-770f-4ee7-62b06de9eba6}" 2026-04-22T10:09:26.483Z [INFO] server-21405755-d305-b391-a77a-024ad6d06275: aborting pipeline replication: peer="{Voter server-ae32f513-0d09-fbbe-fe92-87e2d4c2710e ae32f513-0d09-fbbe-fe92-87e2d4c2710e}" --- PASS: TestRaft_AddKnownPeer (0.28s) === RUN TestRaft_RemoveUnknownPeer 2026-04-22T10:09:26.484Z [INFO] server-42c99b24-97a6-1ae1-98e9-c8bb331808ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c99b24-97a6-1ae1-98e9-c8bb331808ac Address:42c99b24-97a6-1ae1-98e9-c8bb331808ac} {Suffrage:Voter ID:server-164c7a2a-4375-4a74-e697-5275ab269583 Address:164c7a2a-4375-4a74-e697-5275ab269583} {Suffrage:Voter ID:server-443073a1-fe74-5b49-7655-fd141b84bef6 Address:443073a1-fe74-5b49-7655-fd141b84bef6}]" 2026-04-22T10:09:26.484Z [INFO] server-164c7a2a-4375-4a74-e697-5275ab269583: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c99b24-97a6-1ae1-98e9-c8bb331808ac Address:42c99b24-97a6-1ae1-98e9-c8bb331808ac} {Suffrage:Voter ID:server-164c7a2a-4375-4a74-e697-5275ab269583 Address:164c7a2a-4375-4a74-e697-5275ab269583} {Suffrage:Voter ID:server-443073a1-fe74-5b49-7655-fd141b84bef6 Address:443073a1-fe74-5b49-7655-fd141b84bef6}]" 2026-04-22T10:09:26.484Z [INFO] server-42c99b24-97a6-1ae1-98e9-c8bb331808ac: entering follower state: follower="Node at 42c99b24-97a6-1ae1-98e9-c8bb331808ac [Follower]" leader-address= leader-id= 2026-04-22T10:09:26.484Z [INFO] server-443073a1-fe74-5b49-7655-fd141b84bef6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42c99b24-97a6-1ae1-98e9-c8bb331808ac Address:42c99b24-97a6-1ae1-98e9-c8bb331808ac} {Suffrage:Voter ID:server-164c7a2a-4375-4a74-e697-5275ab269583 Address:164c7a2a-4375-4a74-e697-5275ab269583} {Suffrage:Voter ID:server-443073a1-fe74-5b49-7655-fd141b84bef6 Address:443073a1-fe74-5b49-7655-fd141b84bef6}]" 2026-04-22T10:09:26.484Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:26.484Z [INFO] server-164c7a2a-4375-4a74-e697-5275ab269583: entering follower state: follower="Node at 164c7a2a-4375-4a74-e697-5275ab269583 [Follower]" leader-address= leader-id= 2026-04-22T10:09:26.484Z [INFO] server-443073a1-fe74-5b49-7655-fd141b84bef6: entering follower state: follower="Node at 443073a1-fe74-5b49-7655-fd141b84bef6 [Follower]" leader-address= leader-id= 2026-04-22T10:09:26.537Z [WARN] server-164c7a2a-4375-4a74-e697-5275ab269583: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:26.537Z [INFO] server-164c7a2a-4375-4a74-e697-5275ab269583: entering candidate state: node="Node at 164c7a2a-4375-4a74-e697-5275ab269583 [Candidate]" term=2 2026-04-22T10:09:26.537Z [INFO] server-164c7a2a-4375-4a74-e697-5275ab269583: election won: term=2 tally=2 2026-04-22T10:09:26.537Z [INFO] server-164c7a2a-4375-4a74-e697-5275ab269583: entering leader state: leader="Node at 164c7a2a-4375-4a74-e697-5275ab269583 [Leader]" 2026-04-22T10:09:26.537Z [INFO] server-164c7a2a-4375-4a74-e697-5275ab269583: added peer, starting replication: peer=server-42c99b24-97a6-1ae1-98e9-c8bb331808ac 2026-04-22T10:09:26.537Z [INFO] server-164c7a2a-4375-4a74-e697-5275ab269583: added peer, starting replication: peer=server-443073a1-fe74-5b49-7655-fd141b84bef6 2026-04-22T10:09:26.537Z [INFO] server-164c7a2a-4375-4a74-e697-5275ab269583: pipelining replication: peer="{Voter server-443073a1-fe74-5b49-7655-fd141b84bef6 443073a1-fe74-5b49-7655-fd141b84bef6}" 2026-04-22T10:09:26.537Z [WARN] server-42c99b24-97a6-1ae1-98e9-c8bb331808ac: rejecting vote request since our last term is greater: candidate=164c7a2a-4375-4a74-e697-5275ab269583 last-term=2 last-candidate-term=1 2026-04-22T10:09:26.537Z [INFO] server-164c7a2a-4375-4a74-e697-5275ab269583: pipelining replication: peer="{Voter server-42c99b24-97a6-1ae1-98e9-c8bb331808ac 42c99b24-97a6-1ae1-98e9-c8bb331808ac}" 2026-04-22T10:09:26.643Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:26.537931801 +0000 UTC m=+12.561727899 (1 nodes), 53.00164ms from start of poll, 53.240564ms from cluster start. Timeout at 2026-04-22 10:09:26.642936395 +0000 UTC m=+12.666732509, 105.00461ms after stability 2026-04-22T10:09:26.643Z [INFO] server-164c7a2a-4375-4a74-e697-5275ab269583: updating configuration: command=RemoveServer server-id=d52cedc8-f911-b7a6-1db4-db7f31b29de1 server-addr= servers="[{Suffrage:Voter ID:server-42c99b24-97a6-1ae1-98e9-c8bb331808ac Address:42c99b24-97a6-1ae1-98e9-c8bb331808ac} {Suffrage:Voter ID:server-164c7a2a-4375-4a74-e697-5275ab269583 Address:164c7a2a-4375-4a74-e697-5275ab269583} {Suffrage:Voter ID:server-443073a1-fe74-5b49-7655-fd141b84bef6 Address:443073a1-fe74-5b49-7655-fd141b84bef6}]" 2026-04-22T10:09:26.643Z [INFO] server-164c7a2a-4375-4a74-e697-5275ab269583: aborting pipeline replication: peer="{Voter server-443073a1-fe74-5b49-7655-fd141b84bef6 443073a1-fe74-5b49-7655-fd141b84bef6}" 2026-04-22T10:09:26.643Z [INFO] server-164c7a2a-4375-4a74-e697-5275ab269583: aborting pipeline replication: peer="{Voter server-42c99b24-97a6-1ae1-98e9-c8bb331808ac 42c99b24-97a6-1ae1-98e9-c8bb331808ac}" --- PASS: TestRaft_RemoveUnknownPeer (0.16s) === RUN TestRaft_SnapshotRestore 2026-04-22T10:09:26.644Z [INFO] server-e7a732fd-cf69-e0ea-e6c8-3ebca551546c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7a732fd-cf69-e0ea-e6c8-3ebca551546c Address:e7a732fd-cf69-e0ea-e6c8-3ebca551546c}]" 2026-04-22T10:09:26.644Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:26.644Z [INFO] server-e7a732fd-cf69-e0ea-e6c8-3ebca551546c: entering follower state: follower="Node at e7a732fd-cf69-e0ea-e6c8-3ebca551546c [Follower]" leader-address= leader-id= 2026-04-22T10:09:26.716Z [WARN] server-e7a732fd-cf69-e0ea-e6c8-3ebca551546c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:26.716Z [INFO] server-e7a732fd-cf69-e0ea-e6c8-3ebca551546c: entering candidate state: node="Node at e7a732fd-cf69-e0ea-e6c8-3ebca551546c [Candidate]" term=2 2026-04-22T10:09:26.716Z [INFO] server-e7a732fd-cf69-e0ea-e6c8-3ebca551546c: election won: term=2 tally=1 2026-04-22T10:09:26.716Z [INFO] server-e7a732fd-cf69-e0ea-e6c8-3ebca551546c: entering leader state: leader="Node at e7a732fd-cf69-e0ea-e6c8-3ebca551546c [Leader]" 2026-04-22T10:09:26.821Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:26.716399561 +0000 UTC m=+12.740195673 (1 nodes), 72.111323ms from start of poll, 72.216484ms from cluster start. Timeout at 2026-04-22 10:09:26.821401802 +0000 UTC m=+12.845197904, 105.002231ms after stability 2026-04-22T10:09:26.825Z [INFO] server-e7a732fd-cf69-e0ea-e6c8-3ebca551546c: starting snapshot up to: index=102 2026-04-22T10:09:26.825Z [INFO] creating new snapshot: path=/tmp/raft2944875025/snapshots/2-102-1776852566825.tmp 2026-04-22T10:09:26.831Z [INFO] server-e7a732fd-cf69-e0ea-e6c8-3ebca551546c: compacting logs: from=1 to=92 2026-04-22T10:09:26.831Z [INFO] server-e7a732fd-cf69-e0ea-e6c8-3ebca551546c: snapshot complete up to: index=102 2026-04-22T10:09:26.831Z [INFO] server-e7a732fd-cf69-e0ea-e6c8-3ebca551546c: starting restore from snapshot: id=2-102-1776852566825 last-index=102 last-term=2 size-in-bytes=693 2026-04-22T10:09:26.832Z [INFO] server-e7a732fd-cf69-e0ea-e6c8-3ebca551546c: snapshot restore progress: id=2-102-1776852566825 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2026-04-22T10:09:26.832Z [INFO] server-e7a732fd-cf69-e0ea-e6c8-3ebca551546c: restored from snapshot: id=2-102-1776852566825 last-index=102 last-term=2 size-in-bytes=693 2026-04-22T10:09:26.832Z [INFO] server-e7a732fd-cf69-e0ea-e6c8-3ebca551546c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7a732fd-cf69-e0ea-e6c8-3ebca551546c Address:e7a732fd-cf69-e0ea-e6c8-3ebca551546c}]" --- PASS: TestRaft_SnapshotRestore (0.20s) === RUN TestRaft_RestoreSnapshotOnStartup_Monotonic 2026-04-22T10:09:26.840Z [INFO] server-ed20eced-f87a-e739-d489-1ea36fdd710d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed20eced-f87a-e739-d489-1ea36fdd710d Address:ed20eced-f87a-e739-d489-1ea36fdd710d}]" 2026-04-22T10:09:26.840Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:26.840Z [INFO] server-ed20eced-f87a-e739-d489-1ea36fdd710d: entering follower state: follower="Node at ed20eced-f87a-e739-d489-1ea36fdd710d [Follower]" leader-address= leader-id= 2026-04-22T10:09:26.893Z [WARN] server-ed20eced-f87a-e739-d489-1ea36fdd710d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:26.893Z [INFO] server-ed20eced-f87a-e739-d489-1ea36fdd710d: entering candidate state: node="Node at ed20eced-f87a-e739-d489-1ea36fdd710d [Candidate]" term=2 2026-04-22T10:09:26.893Z [INFO] server-ed20eced-f87a-e739-d489-1ea36fdd710d: election won: term=2 tally=1 2026-04-22T10:09:26.893Z [INFO] server-ed20eced-f87a-e739-d489-1ea36fdd710d: entering leader state: leader="Node at ed20eced-f87a-e739-d489-1ea36fdd710d [Leader]" 2026-04-22T10:09:26.999Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:26.893345208 +0000 UTC m=+12.917141296 (1 nodes), 52.965194ms from start of poll, 53.155994ms from cluster start. Timeout at 2026-04-22 10:09:26.998350838 +0000 UTC m=+13.022146964, 105.005668ms after stability 2026-04-22T10:09:27.002Z [INFO] server-ed20eced-f87a-e739-d489-1ea36fdd710d: starting snapshot up to: index=102 2026-04-22T10:09:27.002Z [INFO] creating new snapshot: path=/tmp/raft3802017344/snapshots/2-102-1776852567002.tmp 2026-04-22T10:09:27.031Z [INFO] server-ed20eced-f87a-e739-d489-1ea36fdd710d: compacting logs: from=1 to=92 2026-04-22T10:09:27.031Z [INFO] server-ed20eced-f87a-e739-d489-1ea36fdd710d: snapshot complete up to: index=102 2026-04-22T10:09:27.032Z [INFO] server-ed20eced-f87a-e739-d489-1ea36fdd710d: starting restore from snapshot: id=2-102-1776852567002 last-index=102 last-term=2 size-in-bytes=693 2026-04-22T10:09:27.032Z [INFO] server-ed20eced-f87a-e739-d489-1ea36fdd710d: snapshot restore progress: id=2-102-1776852567002 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2026-04-22T10:09:27.032Z [INFO] server-ed20eced-f87a-e739-d489-1ea36fdd710d: restored from snapshot: id=2-102-1776852567002 last-index=102 last-term=2 size-in-bytes=693 2026-04-22T10:09:27.032Z [INFO] server-ed20eced-f87a-e739-d489-1ea36fdd710d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed20eced-f87a-e739-d489-1ea36fdd710d Address:ed20eced-f87a-e739-d489-1ea36fdd710d}]" 2026-04-22T10:09:27.032Z [INFO] server-ed20eced-f87a-e739-d489-1ea36fdd710d: entering follower state: follower="Node at ed20eced-f87a-e739-d489-1ea36fdd710d [Shutdown]" leader-address= leader-id= --- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.19s) === RUN TestRaft_SnapshotRestore_Progress 2026-04-22T10:09:27.033Z [INFO] server-6501ca49-e2ca-4ff1-a198-23ba0ae26ef0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6501ca49-e2ca-4ff1-a198-23ba0ae26ef0 Address:6501ca49-e2ca-4ff1-a198-23ba0ae26ef0}]" 2026-04-22T10:09:27.033Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:27.033Z [INFO] server-6501ca49-e2ca-4ff1-a198-23ba0ae26ef0: entering follower state: follower="Node at 6501ca49-e2ca-4ff1-a198-23ba0ae26ef0 [Follower]" leader-address= leader-id= 2026-04-22T10:09:27.108Z [WARN] server-6501ca49-e2ca-4ff1-a198-23ba0ae26ef0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:27.108Z [INFO] server-6501ca49-e2ca-4ff1-a198-23ba0ae26ef0: entering candidate state: node="Node at 6501ca49-e2ca-4ff1-a198-23ba0ae26ef0 [Candidate]" term=2 2026-04-22T10:09:27.108Z [INFO] server-6501ca49-e2ca-4ff1-a198-23ba0ae26ef0: election won: term=2 tally=1 2026-04-22T10:09:27.108Z [INFO] server-6501ca49-e2ca-4ff1-a198-23ba0ae26ef0: entering leader state: leader="Node at 6501ca49-e2ca-4ff1-a198-23ba0ae26ef0 [Leader]" 2026-04-22T10:09:27.214Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:27.108895621 +0000 UTC m=+13.132691707 (1 nodes), 75.453211ms from start of poll, 75.604436ms from cluster start. Timeout at 2026-04-22 10:09:27.213898726 +0000 UTC m=+13.237694825, 105.003118ms after stability 2026-04-22T10:09:27.217Z [INFO] server-6501ca49-e2ca-4ff1-a198-23ba0ae26ef0: starting snapshot up to: index=102 2026-04-22T10:09:27.217Z [INFO] creating new snapshot: path=/tmp/raft962602678/snapshots/2-102-1776852567217.tmp 2026-04-22T10:09:27.255Z [INFO] server-6501ca49-e2ca-4ff1-a198-23ba0ae26ef0: compacting logs: from=1 to=92 2026-04-22T10:09:27.255Z [INFO] server-6501ca49-e2ca-4ff1-a198-23ba0ae26ef0: snapshot complete up to: index=102 --- PASS: TestRaft_SnapshotRestore_Progress (0.22s) === RUN TestRaft_NoRestoreOnStart 2026-04-22T10:09:27.257Z [INFO] server-4123a9f7-c209-7b35-e81d-46eb7e57aa0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4123a9f7-c209-7b35-e81d-46eb7e57aa0a Address:4123a9f7-c209-7b35-e81d-46eb7e57aa0a}]" 2026-04-22T10:09:27.257Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:27.257Z [INFO] server-4123a9f7-c209-7b35-e81d-46eb7e57aa0a: entering follower state: follower="Node at 4123a9f7-c209-7b35-e81d-46eb7e57aa0a [Follower]" leader-address= leader-id= 2026-04-22T10:09:27.347Z [WARN] server-4123a9f7-c209-7b35-e81d-46eb7e57aa0a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:27.347Z [INFO] server-4123a9f7-c209-7b35-e81d-46eb7e57aa0a: entering candidate state: node="Node at 4123a9f7-c209-7b35-e81d-46eb7e57aa0a [Candidate]" term=2 2026-04-22T10:09:27.347Z [INFO] server-4123a9f7-c209-7b35-e81d-46eb7e57aa0a: election won: term=2 tally=1 2026-04-22T10:09:27.347Z [INFO] server-4123a9f7-c209-7b35-e81d-46eb7e57aa0a: entering leader state: leader="Node at 4123a9f7-c209-7b35-e81d-46eb7e57aa0a [Leader]" 2026-04-22T10:09:27.452Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:27.347754545 +0000 UTC m=+13.371550662 (1 nodes), 89.79247ms from start of poll, 89.947435ms from cluster start. Timeout at 2026-04-22 10:09:27.452755545 +0000 UTC m=+13.476551639, 105.000977ms after stability 2026-04-22T10:09:27.455Z [INFO] server-4123a9f7-c209-7b35-e81d-46eb7e57aa0a: starting snapshot up to: index=102 2026-04-22T10:09:27.455Z [INFO] creating new snapshot: path=/tmp/raft3042018787/snapshots/2-102-1776852567455.tmp 2026-04-22T10:09:27.480Z [INFO] server-4123a9f7-c209-7b35-e81d-46eb7e57aa0a: compacting logs: from=1 to=92 2026-04-22T10:09:27.480Z [INFO] server-4123a9f7-c209-7b35-e81d-46eb7e57aa0a: snapshot complete up to: index=102 2026-04-22T10:09:27.481Z [INFO] server-4123a9f7-c209-7b35-e81d-46eb7e57aa0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4123a9f7-c209-7b35-e81d-46eb7e57aa0a Address:4123a9f7-c209-7b35-e81d-46eb7e57aa0a}]" --- PASS: TestRaft_NoRestoreOnStart (0.22s) === RUN TestRaft_SnapshotRestore_PeerChange 2026-04-22T10:09:27.481Z [INFO] server-4123a9f7-c209-7b35-e81d-46eb7e57aa0a: entering follower state: follower="Node at 4123a9f7-c209-7b35-e81d-46eb7e57aa0a [Follower]" leader-address= leader-id= raft_test.go:1266: Skipping unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot 2026-04-22T10:09:27.481Z [INFO] server-17ec5c83-a0dc-f67a-81ec-926d5d82326c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17ec5c83-a0dc-f67a-81ec-926d5d82326c Address:17ec5c83-a0dc-f67a-81ec-926d5d82326c}]" 2026-04-22T10:09:27.481Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:27.481Z [INFO] server-17ec5c83-a0dc-f67a-81ec-926d5d82326c: entering follower state: follower="Node at 17ec5c83-a0dc-f67a-81ec-926d5d82326c [Follower]" leader-address= leader-id= 2026-04-22T10:09:27.541Z [WARN] server-4123a9f7-c209-7b35-e81d-46eb7e57aa0a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:27.541Z [INFO] server-4123a9f7-c209-7b35-e81d-46eb7e57aa0a: entering candidate state: node="Node at 4123a9f7-c209-7b35-e81d-46eb7e57aa0a [Candidate]" term=3 2026-04-22T10:09:27.541Z [INFO] server-4123a9f7-c209-7b35-e81d-46eb7e57aa0a: election won: term=3 tally=1 2026-04-22T10:09:27.541Z [INFO] server-4123a9f7-c209-7b35-e81d-46eb7e57aa0a: entering leader state: leader="Node at 4123a9f7-c209-7b35-e81d-46eb7e57aa0a [Leader]" 2026-04-22T10:09:27.557Z [WARN] server-17ec5c83-a0dc-f67a-81ec-926d5d82326c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:27.557Z [INFO] server-17ec5c83-a0dc-f67a-81ec-926d5d82326c: entering candidate state: node="Node at 17ec5c83-a0dc-f67a-81ec-926d5d82326c [Candidate]" term=2 2026-04-22T10:09:27.557Z [INFO] server-17ec5c83-a0dc-f67a-81ec-926d5d82326c: election won: term=2 tally=1 2026-04-22T10:09:27.557Z [INFO] server-17ec5c83-a0dc-f67a-81ec-926d5d82326c: entering leader state: leader="Node at 17ec5c83-a0dc-f67a-81ec-926d5d82326c [Leader]" 2026-04-22T10:09:27.663Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:27.557275027 +0000 UTC m=+13.581071119 (1 nodes), 75.727762ms from start of poll, 75.847369ms from cluster start. Timeout at 2026-04-22 10:09:27.6622777 +0000 UTC m=+13.686073781, 105.002662ms after stability 2026-04-22T10:09:27.667Z [INFO] server-17ec5c83-a0dc-f67a-81ec-926d5d82326c: starting snapshot up to: index=102 2026-04-22T10:09:27.667Z [INFO] creating new snapshot: path=/tmp/raft3916032151/snapshots/2-102-1776852567667.tmp 2026-04-22T10:09:27.694Z [INFO] server-17ec5c83-a0dc-f67a-81ec-926d5d82326c: compacting logs: from=1 to=92 2026-04-22T10:09:27.694Z [INFO] server-17ec5c83-a0dc-f67a-81ec-926d5d82326c: snapshot complete up to: index=102 --- PASS: TestRaft_AutoSnapshot (0.29s) === RUN TestRaft_UserSnapshot 2026-04-22T10:09:27.771Z [INFO] server-124bdfe4-ae45-ab06-5a49-4f9ba36094df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-124bdfe4-ae45-ab06-5a49-4f9ba36094df Address:124bdfe4-ae45-ab06-5a49-4f9ba36094df}]" 2026-04-22T10:09:27.771Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:27.771Z [INFO] server-124bdfe4-ae45-ab06-5a49-4f9ba36094df: entering follower state: follower="Node at 124bdfe4-ae45-ab06-5a49-4f9ba36094df [Follower]" leader-address= leader-id= 2026-04-22T10:09:27.844Z [WARN] server-124bdfe4-ae45-ab06-5a49-4f9ba36094df: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:27.844Z [INFO] server-124bdfe4-ae45-ab06-5a49-4f9ba36094df: entering candidate state: node="Node at 124bdfe4-ae45-ab06-5a49-4f9ba36094df [Candidate]" term=2 2026-04-22T10:09:27.844Z [INFO] server-124bdfe4-ae45-ab06-5a49-4f9ba36094df: election won: term=2 tally=1 2026-04-22T10:09:27.844Z [INFO] server-124bdfe4-ae45-ab06-5a49-4f9ba36094df: entering leader state: leader="Node at 124bdfe4-ae45-ab06-5a49-4f9ba36094df [Leader]" 2026-04-22T10:09:27.949Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:27.844316986 +0000 UTC m=+13.868113083 (1 nodes), 72.334202ms from start of poll, 72.457086ms from cluster start. Timeout at 2026-04-22 10:09:27.949320395 +0000 UTC m=+13.973116488, 105.003405ms after stability 2026-04-22T10:09:27.949Z [ERROR] server-124bdfe4-ae45-ab06-5a49-4f9ba36094df: failed to take snapshot: error="nothing new to snapshot" 2026-04-22T10:09:27.950Z [INFO] server-124bdfe4-ae45-ab06-5a49-4f9ba36094df: starting snapshot up to: index=12 2026-04-22T10:09:27.950Z [INFO] creating new snapshot: path=/tmp/raft3781532292/snapshots/2-12-1776852567950.tmp 2026-04-22T10:09:27.950Z [INFO] server-124bdfe4-ae45-ab06-5a49-4f9ba36094df: compacting logs: from=1 to=2 2026-04-22T10:09:27.950Z [INFO] server-124bdfe4-ae45-ab06-5a49-4f9ba36094df: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.18s) === RUN TestRaft_UserRestore === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false 2026-04-22T10:09:27.951Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31bdcbab-7903-43cf-434e-63d1c8b786c7 Address:31bdcbab-7903-43cf-434e-63d1c8b786c7} {Suffrage:Voter ID:server-004abba6-2704-f02c-4a1d-2f74074e297c Address:004abba6-2704-f02c-4a1d-2f74074e297c} {Suffrage:Voter ID:server-9280583d-a2ed-9ffa-4c97-536f654f8445 Address:9280583d-a2ed-9ffa-4c97-536f654f8445}]" 2026-04-22T10:09:27.951Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: entering follower state: follower="Node at 31bdcbab-7903-43cf-434e-63d1c8b786c7 [Follower]" leader-address= leader-id= 2026-04-22T10:09:27.951Z [INFO] server-004abba6-2704-f02c-4a1d-2f74074e297c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31bdcbab-7903-43cf-434e-63d1c8b786c7 Address:31bdcbab-7903-43cf-434e-63d1c8b786c7} {Suffrage:Voter ID:server-004abba6-2704-f02c-4a1d-2f74074e297c Address:004abba6-2704-f02c-4a1d-2f74074e297c} {Suffrage:Voter ID:server-9280583d-a2ed-9ffa-4c97-536f654f8445 Address:9280583d-a2ed-9ffa-4c97-536f654f8445}]" 2026-04-22T10:09:27.951Z [INFO] server-004abba6-2704-f02c-4a1d-2f74074e297c: entering follower state: follower="Node at 004abba6-2704-f02c-4a1d-2f74074e297c [Follower]" leader-address= leader-id= 2026-04-22T10:09:27.952Z [INFO] server-9280583d-a2ed-9ffa-4c97-536f654f8445: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31bdcbab-7903-43cf-434e-63d1c8b786c7 Address:31bdcbab-7903-43cf-434e-63d1c8b786c7} {Suffrage:Voter ID:server-004abba6-2704-f02c-4a1d-2f74074e297c Address:004abba6-2704-f02c-4a1d-2f74074e297c} {Suffrage:Voter ID:server-9280583d-a2ed-9ffa-4c97-536f654f8445 Address:9280583d-a2ed-9ffa-4c97-536f654f8445}]" 2026-04-22T10:09:27.952Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:27.952Z [INFO] server-9280583d-a2ed-9ffa-4c97-536f654f8445: entering follower state: follower="Node at 9280583d-a2ed-9ffa-4c97-536f654f8445 [Follower]" leader-address= leader-id= 2026-04-22T10:09:28.599Z [WARN] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:28.599Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: entering candidate state: node="Node at 31bdcbab-7903-43cf-434e-63d1c8b786c7 [Candidate]" term=2 2026-04-22T10:09:28.599Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: election won: term=2 tally=2 2026-04-22T10:09:28.599Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: entering leader state: leader="Node at 31bdcbab-7903-43cf-434e-63d1c8b786c7 [Leader]" 2026-04-22T10:09:28.599Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: added peer, starting replication: peer=server-004abba6-2704-f02c-4a1d-2f74074e297c 2026-04-22T10:09:28.599Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: added peer, starting replication: peer=server-9280583d-a2ed-9ffa-4c97-536f654f8445 2026-04-22T10:09:28.599Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: pipelining replication: peer="{Voter server-9280583d-a2ed-9ffa-4c97-536f654f8445 9280583d-a2ed-9ffa-4c97-536f654f8445}" 2026-04-22T10:09:28.599Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: pipelining replication: peer="{Voter server-004abba6-2704-f02c-4a1d-2f74074e297c 004abba6-2704-f02c-4a1d-2f74074e297c}" 2026-04-22T10:09:29.604Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:28.599516741 +0000 UTC m=+14.623312839 (1 nodes), 647.432638ms from start of poll, 647.797295ms from cluster start. Timeout at 2026-04-22 10:09:29.604522824 +0000 UTC m=+15.628318948, 1.005006109s after stability 2026-04-22T10:09:29.605Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: starting snapshot up to: index=12 2026-04-22T10:09:29.605Z [INFO] creating new snapshot: path=/tmp/raft1998147525/snapshots/2-12-1776852569605.tmp 2026-04-22T10:09:29.606Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: snapshot complete up to: index=12 2026-04-22T10:09:29.606Z [INFO] creating new snapshot: path=/tmp/raft1998147525/snapshots/2-23-1776852569606.tmp 2026-04-22T10:09:29.607Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: copied to local snapshot: bytes=71 2026-04-22T10:09:29.607Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: snapshot restore progress: id=2-23-1776852569606 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:29.607Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: restored user snapshot: index=23 2026-04-22T10:09:29.608Z [ERROR] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: failed to get log: index=23 error="log not found" 2026-04-22T10:09:29.608Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: aborting pipeline replication: peer="{Voter server-9280583d-a2ed-9ffa-4c97-536f654f8445 9280583d-a2ed-9ffa-4c97-536f654f8445}" 2026-04-22T10:09:29.608Z [ERROR] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: failed to get log: index=23 error="log not found" 2026-04-22T10:09:29.608Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: aborting pipeline replication: peer="{Voter server-004abba6-2704-f02c-4a1d-2f74074e297c 004abba6-2704-f02c-4a1d-2f74074e297c}" 2026-04-22T10:09:29.617Z [ERROR] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: failed to get log: index=23 error="log not found" 2026-04-22T10:09:29.617Z [ERROR] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: failed to get log: index=23 error="log not found" 2026-04-22T10:09:29.618Z [INFO] creating new snapshot: path=/tmp/raft3391203387/snapshots/2-23-1776852569617.tmp 2026-04-22T10:09:29.618Z [INFO] creating new snapshot: path=/tmp/raft1203374000/snapshots/2-23-1776852569618.tmp 2026-04-22T10:09:29.618Z [INFO] server-9280583d-a2ed-9ffa-4c97-536f654f8445: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:29.618Z [INFO] server-9280583d-a2ed-9ffa-4c97-536f654f8445: copied to local snapshot: bytes=71 2026-04-22T10:09:29.618Z [INFO] server-004abba6-2704-f02c-4a1d-2f74074e297c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:29.618Z [INFO] server-9280583d-a2ed-9ffa-4c97-536f654f8445: snapshot restore progress: id=2-23-1776852569617 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:29.618Z [INFO] server-9280583d-a2ed-9ffa-4c97-536f654f8445: Installed remote snapshot 2026-04-22T10:09:29.619Z [WARN] server-9280583d-a2ed-9ffa-4c97-536f654f8445: skipping application of old log: index=22 2026-04-22T10:09:29.619Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: pipelining replication: peer="{Voter server-9280583d-a2ed-9ffa-4c97-536f654f8445 9280583d-a2ed-9ffa-4c97-536f654f8445}" 2026-04-22T10:09:29.619Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:29.619Z [INFO] server-004abba6-2704-f02c-4a1d-2f74074e297c: copied to local snapshot: bytes=71 2026-04-22T10:09:29.619Z [INFO] server-004abba6-2704-f02c-4a1d-2f74074e297c: snapshot restore progress: id=2-23-1776852569618 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:29.619Z [INFO] server-004abba6-2704-f02c-4a1d-2f74074e297c: Installed remote snapshot 2026-04-22T10:09:29.619Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: pipelining replication: peer="{Voter server-004abba6-2704-f02c-4a1d-2f74074e297c 004abba6-2704-f02c-4a1d-2f74074e297c}" 2026-04-22T10:09:30.624Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:29.619188034 +0000 UTC m=+15.642984122 (2 nodes), 673ns from start of poll, 1.667468578s from cluster start. Timeout at 2026-04-22 10:09:30.624193244 +0000 UTC m=+16.647989326, 1.005005204s after stability 2026-04-22T10:09:30.636Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: aborting pipeline replication: peer="{Voter server-9280583d-a2ed-9ffa-4c97-536f654f8445 9280583d-a2ed-9ffa-4c97-536f654f8445}" 2026-04-22T10:09:30.636Z [INFO] server-31bdcbab-7903-43cf-434e-63d1c8b786c7: aborting pipeline replication: peer="{Voter server-004abba6-2704-f02c-4a1d-2f74074e297c 004abba6-2704-f02c-4a1d-2f74074e297c}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false 2026-04-22T10:09:30.637Z [INFO] server-4e947b87-ab30-61d6-1a41-2880ba821696: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e947b87-ab30-61d6-1a41-2880ba821696 Address:4e947b87-ab30-61d6-1a41-2880ba821696} {Suffrage:Voter ID:server-c45dcf87-f7ea-0bad-e946-91b7beca8a51 Address:c45dcf87-f7ea-0bad-e946-91b7beca8a51} {Suffrage:Voter ID:server-b60e906f-adda-1903-b2a4-386a694bb5db Address:b60e906f-adda-1903-b2a4-386a694bb5db}]" 2026-04-22T10:09:30.637Z [INFO] server-4e947b87-ab30-61d6-1a41-2880ba821696: entering follower state: follower="Node at 4e947b87-ab30-61d6-1a41-2880ba821696 [Follower]" leader-address= leader-id= 2026-04-22T10:09:30.637Z [INFO] server-c45dcf87-f7ea-0bad-e946-91b7beca8a51: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e947b87-ab30-61d6-1a41-2880ba821696 Address:4e947b87-ab30-61d6-1a41-2880ba821696} {Suffrage:Voter ID:server-c45dcf87-f7ea-0bad-e946-91b7beca8a51 Address:c45dcf87-f7ea-0bad-e946-91b7beca8a51} {Suffrage:Voter ID:server-b60e906f-adda-1903-b2a4-386a694bb5db Address:b60e906f-adda-1903-b2a4-386a694bb5db}]" 2026-04-22T10:09:30.638Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e947b87-ab30-61d6-1a41-2880ba821696 Address:4e947b87-ab30-61d6-1a41-2880ba821696} {Suffrage:Voter ID:server-c45dcf87-f7ea-0bad-e946-91b7beca8a51 Address:c45dcf87-f7ea-0bad-e946-91b7beca8a51} {Suffrage:Voter ID:server-b60e906f-adda-1903-b2a4-386a694bb5db Address:b60e906f-adda-1903-b2a4-386a694bb5db}]" 2026-04-22T10:09:30.638Z [INFO] server-c45dcf87-f7ea-0bad-e946-91b7beca8a51: entering follower state: follower="Node at c45dcf87-f7ea-0bad-e946-91b7beca8a51 [Follower]" leader-address= leader-id= 2026-04-22T10:09:30.638Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:30.638Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: entering follower state: follower="Node at b60e906f-adda-1903-b2a4-386a694bb5db [Follower]" leader-address= leader-id= 2026-04-22T10:09:31.147Z [WARN] server-b60e906f-adda-1903-b2a4-386a694bb5db: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:31.147Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: entering candidate state: node="Node at b60e906f-adda-1903-b2a4-386a694bb5db [Candidate]" term=2 2026-04-22T10:09:31.147Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: election won: term=2 tally=2 2026-04-22T10:09:31.147Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: entering leader state: leader="Node at b60e906f-adda-1903-b2a4-386a694bb5db [Leader]" 2026-04-22T10:09:31.147Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: added peer, starting replication: peer=server-4e947b87-ab30-61d6-1a41-2880ba821696 2026-04-22T10:09:31.147Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: added peer, starting replication: peer=server-c45dcf87-f7ea-0bad-e946-91b7beca8a51 2026-04-22T10:09:31.147Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: pipelining replication: peer="{Voter server-c45dcf87-f7ea-0bad-e946-91b7beca8a51 c45dcf87-f7ea-0bad-e946-91b7beca8a51}" 2026-04-22T10:09:31.147Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: pipelining replication: peer="{Voter server-4e947b87-ab30-61d6-1a41-2880ba821696 4e947b87-ab30-61d6-1a41-2880ba821696}" 2026-04-22T10:09:32.153Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:31.147546263 +0000 UTC m=+17.171342351 (1 nodes), 509.490051ms from start of poll, 509.781962ms from cluster start. Timeout at 2026-04-22 10:09:32.152558631 +0000 UTC m=+18.176354751, 1.0050124s after stability 2026-04-22T10:09:32.155Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: starting snapshot up to: index=12 2026-04-22T10:09:32.155Z [INFO] creating new snapshot: path=/tmp/raft422226262/snapshots/2-12-1776852572155.tmp 2026-04-22T10:09:32.157Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: snapshot complete up to: index=12 2026-04-22T10:09:32.709Z [WARN] server-b60e906f-adda-1903-b2a4-386a694bb5db: failed to contact: server-id=server-c45dcf87-f7ea-0bad-e946-91b7beca8a51 time=554.26795ms 2026-04-22T10:09:32.710Z [INFO] creating new snapshot: path=/tmp/raft422226262/snapshots/2-23-1776852572710.tmp 2026-04-22T10:09:32.710Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: copied to local snapshot: bytes=71 2026-04-22T10:09:32.714Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: snapshot restore progress: id=2-23-1776852572710 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:32.714Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: removing all old logs from log store 2026-04-22T10:09:32.714Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: compacting logs: from=1 to=22 2026-04-22T10:09:32.714Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: restored user snapshot: index=23 2026-04-22T10:09:32.714Z [ERROR] server-b60e906f-adda-1903-b2a4-386a694bb5db: failed to get log: index=22 error="log not found" 2026-04-22T10:09:32.714Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: aborting pipeline replication: peer="{Voter server-c45dcf87-f7ea-0bad-e946-91b7beca8a51 c45dcf87-f7ea-0bad-e946-91b7beca8a51}" 2026-04-22T10:09:32.714Z [ERROR] server-b60e906f-adda-1903-b2a4-386a694bb5db: failed to get log: index=22 error="log not found" 2026-04-22T10:09:32.714Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: aborting pipeline replication: peer="{Voter server-4e947b87-ab30-61d6-1a41-2880ba821696 4e947b87-ab30-61d6-1a41-2880ba821696}" 2026-04-22T10:09:32.721Z [ERROR] server-b60e906f-adda-1903-b2a4-386a694bb5db: failed to get log: index=22 error="log not found" 2026-04-22T10:09:32.721Z [INFO] creating new snapshot: path=/tmp/raft3832349352/snapshots/2-23-1776852572721.tmp 2026-04-22T10:09:32.722Z [INFO] server-4e947b87-ab30-61d6-1a41-2880ba821696: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:32.724Z [ERROR] server-b60e906f-adda-1903-b2a4-386a694bb5db: failed to get log: index=22 error="log not found" 2026-04-22T10:09:32.725Z [INFO] creating new snapshot: path=/tmp/raft3371125227/snapshots/2-23-1776852572725.tmp 2026-04-22T10:09:32.726Z [INFO] server-4e947b87-ab30-61d6-1a41-2880ba821696: copied to local snapshot: bytes=71 2026-04-22T10:09:32.726Z [INFO] server-c45dcf87-f7ea-0bad-e946-91b7beca8a51: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:32.726Z [INFO] server-4e947b87-ab30-61d6-1a41-2880ba821696: snapshot restore progress: id=2-23-1776852572721 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:32.726Z [INFO] server-4e947b87-ab30-61d6-1a41-2880ba821696: removing all old logs from log store 2026-04-22T10:09:32.726Z [INFO] server-4e947b87-ab30-61d6-1a41-2880ba821696: compacting logs: from=1 to=22 2026-04-22T10:09:32.726Z [INFO] server-4e947b87-ab30-61d6-1a41-2880ba821696: Installed remote snapshot 2026-04-22T10:09:32.727Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: pipelining replication: peer="{Voter server-4e947b87-ab30-61d6-1a41-2880ba821696 4e947b87-ab30-61d6-1a41-2880ba821696}" 2026-04-22T10:09:32.727Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:32.732Z [INFO] server-c45dcf87-f7ea-0bad-e946-91b7beca8a51: copied to local snapshot: bytes=71 2026-04-22T10:09:32.733Z [INFO] server-c45dcf87-f7ea-0bad-e946-91b7beca8a51: snapshot restore progress: id=2-23-1776852572725 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:32.733Z [INFO] server-c45dcf87-f7ea-0bad-e946-91b7beca8a51: removing all old logs from log store 2026-04-22T10:09:32.733Z [INFO] server-c45dcf87-f7ea-0bad-e946-91b7beca8a51: compacting logs: from=1 to=22 2026-04-22T10:09:32.733Z [INFO] server-c45dcf87-f7ea-0bad-e946-91b7beca8a51: Installed remote snapshot 2026-04-22T10:09:32.733Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: pipelining replication: peer="{Voter server-c45dcf87-f7ea-0bad-e946-91b7beca8a51 c45dcf87-f7ea-0bad-e946-91b7beca8a51}" 2026-04-22T10:09:33.732Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:32.727119719 +0000 UTC m=+18.750915822 (2 nodes), 1.032µs from start of poll, 2.089355433s from cluster start. Timeout at 2026-04-22 10:09:33.732128442 +0000 UTC m=+19.755924524, 1.005008702s after stability 2026-04-22T10:09:33.744Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: aborting pipeline replication: peer="{Voter server-c45dcf87-f7ea-0bad-e946-91b7beca8a51 c45dcf87-f7ea-0bad-e946-91b7beca8a51}" 2026-04-22T10:09:33.744Z [INFO] server-b60e906f-adda-1903-b2a4-386a694bb5db: aborting pipeline replication: peer="{Voter server-4e947b87-ab30-61d6-1a41-2880ba821696 4e947b87-ab30-61d6-1a41-2880ba821696}" === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true 2026-04-22T10:09:33.745Z [INFO] server-e48c12c2-128f-ded3-3904-d56098d34c0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e48c12c2-128f-ded3-3904-d56098d34c0d Address:e48c12c2-128f-ded3-3904-d56098d34c0d} {Suffrage:Voter ID:server-6d674c41-be45-bba7-807d-9dcb068f668a Address:6d674c41-be45-bba7-807d-9dcb068f668a} {Suffrage:Voter ID:server-bc0db1a9-72c2-b3c4-0b86-2dc0c7363639 Address:bc0db1a9-72c2-b3c4-0b86-2dc0c7363639}]" 2026-04-22T10:09:33.745Z [INFO] server-6d674c41-be45-bba7-807d-9dcb068f668a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e48c12c2-128f-ded3-3904-d56098d34c0d Address:e48c12c2-128f-ded3-3904-d56098d34c0d} {Suffrage:Voter ID:server-6d674c41-be45-bba7-807d-9dcb068f668a Address:6d674c41-be45-bba7-807d-9dcb068f668a} {Suffrage:Voter ID:server-bc0db1a9-72c2-b3c4-0b86-2dc0c7363639 Address:bc0db1a9-72c2-b3c4-0b86-2dc0c7363639}]" 2026-04-22T10:09:33.745Z [INFO] server-bc0db1a9-72c2-b3c4-0b86-2dc0c7363639: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e48c12c2-128f-ded3-3904-d56098d34c0d Address:e48c12c2-128f-ded3-3904-d56098d34c0d} {Suffrage:Voter ID:server-6d674c41-be45-bba7-807d-9dcb068f668a Address:6d674c41-be45-bba7-807d-9dcb068f668a} {Suffrage:Voter ID:server-bc0db1a9-72c2-b3c4-0b86-2dc0c7363639 Address:bc0db1a9-72c2-b3c4-0b86-2dc0c7363639}]" 2026-04-22T10:09:33.745Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:33.745Z [INFO] server-6d674c41-be45-bba7-807d-9dcb068f668a: entering follower state: follower="Node at 6d674c41-be45-bba7-807d-9dcb068f668a [Follower]" leader-address= leader-id= 2026-04-22T10:09:33.745Z [INFO] server-e48c12c2-128f-ded3-3904-d56098d34c0d: entering follower state: follower="Node at e48c12c2-128f-ded3-3904-d56098d34c0d [Follower]" leader-address= leader-id= 2026-04-22T10:09:33.745Z [INFO] server-bc0db1a9-72c2-b3c4-0b86-2dc0c7363639: entering follower state: follower="Node at bc0db1a9-72c2-b3c4-0b86-2dc0c7363639 [Follower]" leader-address= leader-id= 2026-04-22T10:09:34.628Z [WARN] server-6d674c41-be45-bba7-807d-9dcb068f668a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:34.628Z [INFO] server-6d674c41-be45-bba7-807d-9dcb068f668a: entering candidate state: node="Node at 6d674c41-be45-bba7-807d-9dcb068f668a [Candidate]" term=2 2026-04-22T10:09:34.629Z [INFO] server-6d674c41-be45-bba7-807d-9dcb068f668a: election won: term=2 tally=2 2026-04-22T10:09:34.629Z [INFO] server-6d674c41-be45-bba7-807d-9dcb068f668a: entering leader state: leader="Node at 6d674c41-be45-bba7-807d-9dcb068f668a [Leader]" 2026-04-22T10:09:34.629Z [INFO] server-6d674c41-be45-bba7-807d-9dcb068f668a: added peer, starting replication: peer=server-e48c12c2-128f-ded3-3904-d56098d34c0d 2026-04-22T10:09:34.629Z [INFO] server-6d674c41-be45-bba7-807d-9dcb068f668a: added peer, starting replication: peer=server-bc0db1a9-72c2-b3c4-0b86-2dc0c7363639 2026-04-22T10:09:34.629Z [INFO] server-6d674c41-be45-bba7-807d-9dcb068f668a: pipelining replication: peer="{Voter server-bc0db1a9-72c2-b3c4-0b86-2dc0c7363639 bc0db1a9-72c2-b3c4-0b86-2dc0c7363639}" 2026-04-22T10:09:34.629Z [INFO] server-6d674c41-be45-bba7-807d-9dcb068f668a: pipelining replication: peer="{Voter server-e48c12c2-128f-ded3-3904-d56098d34c0d e48c12c2-128f-ded3-3904-d56098d34c0d}" 2026-04-22T10:09:35.634Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:34.629146929 +0000 UTC m=+20.652943044 (1 nodes), 883.230492ms from start of poll, 883.478455ms from cluster start. Timeout at 2026-04-22 10:09:35.634148122 +0000 UTC m=+21.657944205, 1.005001161s after stability 2026-04-22T10:09:35.635Z [INFO] server-6d674c41-be45-bba7-807d-9dcb068f668a: starting snapshot up to: index=12 2026-04-22T10:09:35.635Z [INFO] creating new snapshot: path=/tmp/raft3462718255/snapshots/2-12-1776852575635.tmp 2026-04-22T10:09:35.635Z [INFO] server-6d674c41-be45-bba7-807d-9dcb068f668a: snapshot complete up to: index=12 2026-04-22T10:09:35.636Z [INFO] server-abaa14f9-a3eb-f034-af94-cb71988acdbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abaa14f9-a3eb-f034-af94-cb71988acdbc Address:abaa14f9-a3eb-f034-af94-cb71988acdbc} {Suffrage:Voter ID:server-bbf70462-4738-03d9-2c25-2ccdf674f0fa Address:bbf70462-4738-03d9-2c25-2ccdf674f0fa} {Suffrage:Voter ID:server-f728594b-486b-bd20-8915-1e1bc2871708 Address:f728594b-486b-bd20-8915-1e1bc2871708}]" 2026-04-22T10:09:35.636Z [INFO] server-abaa14f9-a3eb-f034-af94-cb71988acdbc: entering follower state: follower="Node at abaa14f9-a3eb-f034-af94-cb71988acdbc [Follower]" leader-address= leader-id= 2026-04-22T10:09:35.636Z [INFO] server-bbf70462-4738-03d9-2c25-2ccdf674f0fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abaa14f9-a3eb-f034-af94-cb71988acdbc Address:abaa14f9-a3eb-f034-af94-cb71988acdbc} {Suffrage:Voter ID:server-bbf70462-4738-03d9-2c25-2ccdf674f0fa Address:bbf70462-4738-03d9-2c25-2ccdf674f0fa} {Suffrage:Voter ID:server-f728594b-486b-bd20-8915-1e1bc2871708 Address:f728594b-486b-bd20-8915-1e1bc2871708}]" 2026-04-22T10:09:35.636Z [INFO] server-bbf70462-4738-03d9-2c25-2ccdf674f0fa: entering follower state: follower="Node at bbf70462-4738-03d9-2c25-2ccdf674f0fa [Follower]" leader-address= leader-id= 2026-04-22T10:09:35.636Z [INFO] server-f728594b-486b-bd20-8915-1e1bc2871708: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abaa14f9-a3eb-f034-af94-cb71988acdbc Address:abaa14f9-a3eb-f034-af94-cb71988acdbc} {Suffrage:Voter ID:server-bbf70462-4738-03d9-2c25-2ccdf674f0fa Address:bbf70462-4738-03d9-2c25-2ccdf674f0fa} {Suffrage:Voter ID:server-f728594b-486b-bd20-8915-1e1bc2871708 Address:f728594b-486b-bd20-8915-1e1bc2871708}]" 2026-04-22T10:09:35.636Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:35.637Z [INFO] server-f728594b-486b-bd20-8915-1e1bc2871708: entering follower state: follower="Node at f728594b-486b-bd20-8915-1e1bc2871708 [Follower]" leader-address= leader-id= 2026-04-22T10:09:36.307Z [WARN] server-f728594b-486b-bd20-8915-1e1bc2871708: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:36.307Z [INFO] server-f728594b-486b-bd20-8915-1e1bc2871708: entering candidate state: node="Node at f728594b-486b-bd20-8915-1e1bc2871708 [Candidate]" term=2 2026-04-22T10:09:36.308Z [INFO] server-f728594b-486b-bd20-8915-1e1bc2871708: election won: term=2 tally=2 2026-04-22T10:09:36.308Z [INFO] server-f728594b-486b-bd20-8915-1e1bc2871708: entering leader state: leader="Node at f728594b-486b-bd20-8915-1e1bc2871708 [Leader]" 2026-04-22T10:09:36.308Z [INFO] server-f728594b-486b-bd20-8915-1e1bc2871708: added peer, starting replication: peer=server-abaa14f9-a3eb-f034-af94-cb71988acdbc 2026-04-22T10:09:36.308Z [INFO] server-f728594b-486b-bd20-8915-1e1bc2871708: added peer, starting replication: peer=server-bbf70462-4738-03d9-2c25-2ccdf674f0fa 2026-04-22T10:09:36.308Z [INFO] server-f728594b-486b-bd20-8915-1e1bc2871708: pipelining replication: peer="{Voter server-bbf70462-4738-03d9-2c25-2ccdf674f0fa bbf70462-4738-03d9-2c25-2ccdf674f0fa}" 2026-04-22T10:09:36.308Z [INFO] server-f728594b-486b-bd20-8915-1e1bc2871708: pipelining replication: peer="{Voter server-abaa14f9-a3eb-f034-af94-cb71988acdbc abaa14f9-a3eb-f034-af94-cb71988acdbc}" 2026-04-22T10:09:37.313Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:36.308092354 +0000 UTC m=+22.331888443 (1 nodes), 671.078979ms from start of poll, 671.420944ms from cluster start. Timeout at 2026-04-22 10:09:37.31309858 +0000 UTC m=+23.336894662, 1.005006219s after stability 2026-04-22T10:09:37.313Z [INFO] creating new snapshot: path=/tmp/raft3318857523/snapshots/2-13-1776852577313.tmp 2026-04-22T10:09:37.314Z [INFO] server-f728594b-486b-bd20-8915-1e1bc2871708: copied to local snapshot: bytes=71 2026-04-22T10:09:37.314Z [INFO] server-f728594b-486b-bd20-8915-1e1bc2871708: snapshot restore progress: id=2-13-1776852577313 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:37.314Z [INFO] server-f728594b-486b-bd20-8915-1e1bc2871708: restored user snapshot: index=13 2026-04-22T10:09:37.315Z [ERROR] server-f728594b-486b-bd20-8915-1e1bc2871708: failed to get log: index=3 error="log not found" 2026-04-22T10:09:37.315Z [ERROR] server-f728594b-486b-bd20-8915-1e1bc2871708: failed to get log: index=3 error="log not found" 2026-04-22T10:09:37.315Z [INFO] server-f728594b-486b-bd20-8915-1e1bc2871708: aborting pipeline replication: peer="{Voter server-bbf70462-4738-03d9-2c25-2ccdf674f0fa bbf70462-4738-03d9-2c25-2ccdf674f0fa}" 2026-04-22T10:09:37.315Z [INFO] server-f728594b-486b-bd20-8915-1e1bc2871708: aborting pipeline replication: peer="{Voter server-abaa14f9-a3eb-f034-af94-cb71988acdbc abaa14f9-a3eb-f034-af94-cb71988acdbc}" 2026-04-22T10:09:37.321Z [ERROR] server-f728594b-486b-bd20-8915-1e1bc2871708: failed to get log: index=3 error="log not found" 2026-04-22T10:09:37.321Z [INFO] creating new snapshot: path=/tmp/raft2747520041/snapshots/2-13-1776852577321.tmp 2026-04-22T10:09:37.322Z [INFO] server-bbf70462-4738-03d9-2c25-2ccdf674f0fa: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:37.322Z [INFO] server-bbf70462-4738-03d9-2c25-2ccdf674f0fa: copied to local snapshot: bytes=71 2026-04-22T10:09:37.322Z [INFO] server-bbf70462-4738-03d9-2c25-2ccdf674f0fa: snapshot restore progress: id=2-13-1776852577321 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:37.323Z [INFO] server-bbf70462-4738-03d9-2c25-2ccdf674f0fa: Installed remote snapshot 2026-04-22T10:09:37.323Z [INFO] server-f728594b-486b-bd20-8915-1e1bc2871708: pipelining replication: peer="{Voter server-bbf70462-4738-03d9-2c25-2ccdf674f0fa bbf70462-4738-03d9-2c25-2ccdf674f0fa}" 2026-04-22T10:09:37.323Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:37.324Z [ERROR] server-f728594b-486b-bd20-8915-1e1bc2871708: failed to get log: index=3 error="log not found" 2026-04-22T10:09:37.325Z [INFO] creating new snapshot: path=/tmp/raft2751207442/snapshots/2-13-1776852577325.tmp 2026-04-22T10:09:37.325Z [INFO] server-abaa14f9-a3eb-f034-af94-cb71988acdbc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:37.325Z [INFO] server-abaa14f9-a3eb-f034-af94-cb71988acdbc: copied to local snapshot: bytes=71 2026-04-22T10:09:37.325Z [INFO] server-abaa14f9-a3eb-f034-af94-cb71988acdbc: snapshot restore progress: id=2-13-1776852577325 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:37.325Z [INFO] server-abaa14f9-a3eb-f034-af94-cb71988acdbc: Installed remote snapshot 2026-04-22T10:09:37.325Z [INFO] server-f728594b-486b-bd20-8915-1e1bc2871708: pipelining replication: peer="{Voter server-abaa14f9-a3eb-f034-af94-cb71988acdbc abaa14f9-a3eb-f034-af94-cb71988acdbc}" 2026-04-22T10:09:38.329Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:37.323262188 +0000 UTC m=+23.347058275 (2 nodes), 859ns from start of poll, 1.686590776s from cluster start. Timeout at 2026-04-22 10:09:38.328269557 +0000 UTC m=+24.352065639, 1.005007364s after stability 2026-04-22T10:09:38.340Z [INFO] server-6d674c41-be45-bba7-807d-9dcb068f668a: aborting pipeline replication: peer="{Voter server-bc0db1a9-72c2-b3c4-0b86-2dc0c7363639 bc0db1a9-72c2-b3c4-0b86-2dc0c7363639}" 2026-04-22T10:09:38.340Z [INFO] server-6d674c41-be45-bba7-807d-9dcb068f668a: aborting pipeline replication: peer="{Voter server-e48c12c2-128f-ded3-3904-d56098d34c0d e48c12c2-128f-ded3-3904-d56098d34c0d}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true 2026-04-22T10:09:38.341Z [INFO] server-c72fd751-ff91-0442-ef68-3a538c0463a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c72fd751-ff91-0442-ef68-3a538c0463a2 Address:c72fd751-ff91-0442-ef68-3a538c0463a2} {Suffrage:Voter ID:server-cc9dabed-11fa-95fe-a765-1e07d6d7e1d1 Address:cc9dabed-11fa-95fe-a765-1e07d6d7e1d1} {Suffrage:Voter ID:server-c3d675a8-dd55-e655-f5fa-542cfc7d640c Address:c3d675a8-dd55-e655-f5fa-542cfc7d640c}]" 2026-04-22T10:09:38.341Z [INFO] server-c72fd751-ff91-0442-ef68-3a538c0463a2: entering follower state: follower="Node at c72fd751-ff91-0442-ef68-3a538c0463a2 [Follower]" leader-address= leader-id= 2026-04-22T10:09:38.341Z [INFO] server-cc9dabed-11fa-95fe-a765-1e07d6d7e1d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c72fd751-ff91-0442-ef68-3a538c0463a2 Address:c72fd751-ff91-0442-ef68-3a538c0463a2} {Suffrage:Voter ID:server-cc9dabed-11fa-95fe-a765-1e07d6d7e1d1 Address:cc9dabed-11fa-95fe-a765-1e07d6d7e1d1} {Suffrage:Voter ID:server-c3d675a8-dd55-e655-f5fa-542cfc7d640c Address:c3d675a8-dd55-e655-f5fa-542cfc7d640c}]" 2026-04-22T10:09:38.341Z [INFO] server-cc9dabed-11fa-95fe-a765-1e07d6d7e1d1: entering follower state: follower="Node at cc9dabed-11fa-95fe-a765-1e07d6d7e1d1 [Follower]" leader-address= leader-id= 2026-04-22T10:09:38.341Z [INFO] server-c3d675a8-dd55-e655-f5fa-542cfc7d640c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c72fd751-ff91-0442-ef68-3a538c0463a2 Address:c72fd751-ff91-0442-ef68-3a538c0463a2} {Suffrage:Voter ID:server-cc9dabed-11fa-95fe-a765-1e07d6d7e1d1 Address:cc9dabed-11fa-95fe-a765-1e07d6d7e1d1} {Suffrage:Voter ID:server-c3d675a8-dd55-e655-f5fa-542cfc7d640c Address:c3d675a8-dd55-e655-f5fa-542cfc7d640c}]" 2026-04-22T10:09:38.341Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:38.341Z [INFO] server-c3d675a8-dd55-e655-f5fa-542cfc7d640c: entering follower state: follower="Node at c3d675a8-dd55-e655-f5fa-542cfc7d640c [Follower]" leader-address= leader-id= 2026-04-22T10:09:38.990Z [WARN] server-cc9dabed-11fa-95fe-a765-1e07d6d7e1d1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:38.990Z [INFO] server-cc9dabed-11fa-95fe-a765-1e07d6d7e1d1: entering candidate state: node="Node at cc9dabed-11fa-95fe-a765-1e07d6d7e1d1 [Candidate]" term=2 2026-04-22T10:09:38.990Z [INFO] server-cc9dabed-11fa-95fe-a765-1e07d6d7e1d1: election won: term=2 tally=2 2026-04-22T10:09:38.990Z [INFO] server-cc9dabed-11fa-95fe-a765-1e07d6d7e1d1: entering leader state: leader="Node at cc9dabed-11fa-95fe-a765-1e07d6d7e1d1 [Leader]" 2026-04-22T10:09:38.990Z [INFO] server-cc9dabed-11fa-95fe-a765-1e07d6d7e1d1: added peer, starting replication: peer=server-c72fd751-ff91-0442-ef68-3a538c0463a2 2026-04-22T10:09:38.990Z [INFO] server-cc9dabed-11fa-95fe-a765-1e07d6d7e1d1: added peer, starting replication: peer=server-c3d675a8-dd55-e655-f5fa-542cfc7d640c 2026-04-22T10:09:38.990Z [INFO] server-cc9dabed-11fa-95fe-a765-1e07d6d7e1d1: pipelining replication: peer="{Voter server-c3d675a8-dd55-e655-f5fa-542cfc7d640c c3d675a8-dd55-e655-f5fa-542cfc7d640c}" 2026-04-22T10:09:38.991Z [INFO] server-cc9dabed-11fa-95fe-a765-1e07d6d7e1d1: pipelining replication: peer="{Voter server-c72fd751-ff91-0442-ef68-3a538c0463a2 c72fd751-ff91-0442-ef68-3a538c0463a2}" 2026-04-22T10:09:39.996Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:38.990875424 +0000 UTC m=+25.014671521 (1 nodes), 649.040722ms from start of poll, 649.351793ms from cluster start. Timeout at 2026-04-22 10:09:39.995878621 +0000 UTC m=+26.019674702, 1.005003181s after stability 2026-04-22T10:09:39.996Z [INFO] server-cc9dabed-11fa-95fe-a765-1e07d6d7e1d1: starting snapshot up to: index=12 2026-04-22T10:09:39.996Z [INFO] creating new snapshot: path=/tmp/raft382920578/snapshots/2-12-1776852579996.tmp 2026-04-22T10:09:39.997Z [INFO] server-cc9dabed-11fa-95fe-a765-1e07d6d7e1d1: snapshot complete up to: index=12 2026-04-22T10:09:39.998Z [INFO] server-1080493c-6516-d56e-c505-f3d641baa6d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1080493c-6516-d56e-c505-f3d641baa6d3 Address:1080493c-6516-d56e-c505-f3d641baa6d3} {Suffrage:Voter ID:server-faadff2b-87e9-2e77-4659-543972aaaa6f Address:faadff2b-87e9-2e77-4659-543972aaaa6f} {Suffrage:Voter ID:server-d057c27f-a5c0-a855-746a-43d9dbf08668 Address:d057c27f-a5c0-a855-746a-43d9dbf08668}]" 2026-04-22T10:09:39.998Z [INFO] server-1080493c-6516-d56e-c505-f3d641baa6d3: entering follower state: follower="Node at 1080493c-6516-d56e-c505-f3d641baa6d3 [Follower]" leader-address= leader-id= 2026-04-22T10:09:39.998Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1080493c-6516-d56e-c505-f3d641baa6d3 Address:1080493c-6516-d56e-c505-f3d641baa6d3} {Suffrage:Voter ID:server-faadff2b-87e9-2e77-4659-543972aaaa6f Address:faadff2b-87e9-2e77-4659-543972aaaa6f} {Suffrage:Voter ID:server-d057c27f-a5c0-a855-746a-43d9dbf08668 Address:d057c27f-a5c0-a855-746a-43d9dbf08668}]" 2026-04-22T10:09:39.998Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: entering follower state: follower="Node at faadff2b-87e9-2e77-4659-543972aaaa6f [Follower]" leader-address= leader-id= 2026-04-22T10:09:39.998Z [INFO] server-d057c27f-a5c0-a855-746a-43d9dbf08668: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1080493c-6516-d56e-c505-f3d641baa6d3 Address:1080493c-6516-d56e-c505-f3d641baa6d3} {Suffrage:Voter ID:server-faadff2b-87e9-2e77-4659-543972aaaa6f Address:faadff2b-87e9-2e77-4659-543972aaaa6f} {Suffrage:Voter ID:server-d057c27f-a5c0-a855-746a-43d9dbf08668 Address:d057c27f-a5c0-a855-746a-43d9dbf08668}]" 2026-04-22T10:09:39.999Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:39.999Z [INFO] server-d057c27f-a5c0-a855-746a-43d9dbf08668: entering follower state: follower="Node at d057c27f-a5c0-a855-746a-43d9dbf08668 [Follower]" leader-address= leader-id= 2026-04-22T10:09:40.623Z [WARN] server-faadff2b-87e9-2e77-4659-543972aaaa6f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:40.623Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: entering candidate state: node="Node at faadff2b-87e9-2e77-4659-543972aaaa6f [Candidate]" term=2 2026-04-22T10:09:40.623Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: election won: term=2 tally=2 2026-04-22T10:09:40.623Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: entering leader state: leader="Node at faadff2b-87e9-2e77-4659-543972aaaa6f [Leader]" 2026-04-22T10:09:40.623Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: added peer, starting replication: peer=server-1080493c-6516-d56e-c505-f3d641baa6d3 2026-04-22T10:09:40.623Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: added peer, starting replication: peer=server-d057c27f-a5c0-a855-746a-43d9dbf08668 2026-04-22T10:09:40.623Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: pipelining replication: peer="{Voter server-1080493c-6516-d56e-c505-f3d641baa6d3 1080493c-6516-d56e-c505-f3d641baa6d3}" 2026-04-22T10:09:40.623Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: pipelining replication: peer="{Voter server-d057c27f-a5c0-a855-746a-43d9dbf08668 d057c27f-a5c0-a855-746a-43d9dbf08668}" 2026-04-22T10:09:41.629Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:40.623625231 +0000 UTC m=+26.647421319 (1 nodes), 624.61151ms from start of poll, 625.078635ms from cluster start. Timeout at 2026-04-22 10:09:41.62862723 +0000 UTC m=+27.652423341, 1.005002022s after stability 2026-04-22T10:09:41.629Z [INFO] creating new snapshot: path=/tmp/raft1247540436/snapshots/2-13-1776852581629.tmp 2026-04-22T10:09:41.629Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: copied to local snapshot: bytes=71 2026-04-22T10:09:41.630Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: snapshot restore progress: id=2-13-1776852581629 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:41.630Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: removing all old logs from log store 2026-04-22T10:09:41.630Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: compacting logs: from=1 to=2 2026-04-22T10:09:41.630Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: restored user snapshot: index=13 2026-04-22T10:09:41.630Z [ERROR] server-faadff2b-87e9-2e77-4659-543972aaaa6f: failed to get log: index=2 error="log not found" 2026-04-22T10:09:41.630Z [ERROR] server-faadff2b-87e9-2e77-4659-543972aaaa6f: failed to get log: index=2 error="log not found" 2026-04-22T10:09:41.630Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: aborting pipeline replication: peer="{Voter server-d057c27f-a5c0-a855-746a-43d9dbf08668 d057c27f-a5c0-a855-746a-43d9dbf08668}" 2026-04-22T10:09:41.630Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: aborting pipeline replication: peer="{Voter server-1080493c-6516-d56e-c505-f3d641baa6d3 1080493c-6516-d56e-c505-f3d641baa6d3}" 2026-04-22T10:09:41.637Z [ERROR] server-faadff2b-87e9-2e77-4659-543972aaaa6f: failed to get log: index=2 error="log not found" 2026-04-22T10:09:41.638Z [ERROR] server-faadff2b-87e9-2e77-4659-543972aaaa6f: failed to get log: index=2 error="log not found" 2026-04-22T10:09:41.638Z [INFO] creating new snapshot: path=/tmp/raft1884505931/snapshots/2-13-1776852581638.tmp 2026-04-22T10:09:41.638Z [INFO] creating new snapshot: path=/tmp/raft3226239647/snapshots/2-13-1776852581638.tmp 2026-04-22T10:09:41.638Z [INFO] server-1080493c-6516-d56e-c505-f3d641baa6d3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:41.638Z [INFO] server-d057c27f-a5c0-a855-746a-43d9dbf08668: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:41.639Z [INFO] server-1080493c-6516-d56e-c505-f3d641baa6d3: copied to local snapshot: bytes=71 2026-04-22T10:09:41.639Z [INFO] server-d057c27f-a5c0-a855-746a-43d9dbf08668: copied to local snapshot: bytes=71 2026-04-22T10:09:41.639Z [INFO] server-1080493c-6516-d56e-c505-f3d641baa6d3: snapshot restore progress: id=2-13-1776852581638 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:41.639Z [INFO] server-d057c27f-a5c0-a855-746a-43d9dbf08668: snapshot restore progress: id=2-13-1776852581638 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:41.639Z [INFO] server-1080493c-6516-d56e-c505-f3d641baa6d3: removing all old logs from log store 2026-04-22T10:09:41.639Z [INFO] server-1080493c-6516-d56e-c505-f3d641baa6d3: compacting logs: from=1 to=2 2026-04-22T10:09:41.639Z [INFO] server-d057c27f-a5c0-a855-746a-43d9dbf08668: removing all old logs from log store 2026-04-22T10:09:41.639Z [INFO] server-1080493c-6516-d56e-c505-f3d641baa6d3: Installed remote snapshot 2026-04-22T10:09:41.639Z [INFO] server-d057c27f-a5c0-a855-746a-43d9dbf08668: compacting logs: from=1 to=2 2026-04-22T10:09:41.639Z [INFO] server-d057c27f-a5c0-a855-746a-43d9dbf08668: Installed remote snapshot 2026-04-22T10:09:41.639Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: pipelining replication: peer="{Voter server-d057c27f-a5c0-a855-746a-43d9dbf08668 d057c27f-a5c0-a855-746a-43d9dbf08668}" 2026-04-22T10:09:41.639Z [INFO] server-faadff2b-87e9-2e77-4659-543972aaaa6f: pipelining replication: peer="{Voter server-1080493c-6516-d56e-c505-f3d641baa6d3 1080493c-6516-d56e-c505-f3d641baa6d3}" 2026-04-22T10:09:41.639Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:42.645Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:41.639699917 +0000 UTC m=+27.663496014 (2 nodes), 868ns from start of poll, 1.64115333s from cluster start. Timeout at 2026-04-22 10:09:42.644703605 +0000 UTC m=+28.668499697, 1.005003683s after stability 2026-04-22T10:09:42.658Z [INFO] server-cc9dabed-11fa-95fe-a765-1e07d6d7e1d1: aborting pipeline replication: peer="{Voter server-c3d675a8-dd55-e655-f5fa-542cfc7d640c c3d675a8-dd55-e655-f5fa-542cfc7d640c}" 2026-04-22T10:09:42.658Z [INFO] server-cc9dabed-11fa-95fe-a765-1e07d6d7e1d1: aborting pipeline replication: peer="{Voter server-c72fd751-ff91-0442-ef68-3a538c0463a2 c72fd751-ff91-0442-ef68-3a538c0463a2}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false 2026-04-22T10:09:42.659Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a32a60e-1bea-6fec-9455-46430bb3a2e7 Address:4a32a60e-1bea-6fec-9455-46430bb3a2e7} {Suffrage:Voter ID:server-f075213b-e48a-3fa2-e465-848c9d1c01f3 Address:f075213b-e48a-3fa2-e465-848c9d1c01f3} {Suffrage:Voter ID:server-ae027834-6ce4-1cc8-fb04-85373f0d9bf2 Address:ae027834-6ce4-1cc8-fb04-85373f0d9bf2}]" 2026-04-22T10:09:42.659Z [INFO] server-f075213b-e48a-3fa2-e465-848c9d1c01f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a32a60e-1bea-6fec-9455-46430bb3a2e7 Address:4a32a60e-1bea-6fec-9455-46430bb3a2e7} {Suffrage:Voter ID:server-f075213b-e48a-3fa2-e465-848c9d1c01f3 Address:f075213b-e48a-3fa2-e465-848c9d1c01f3} {Suffrage:Voter ID:server-ae027834-6ce4-1cc8-fb04-85373f0d9bf2 Address:ae027834-6ce4-1cc8-fb04-85373f0d9bf2}]" 2026-04-22T10:09:42.659Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: entering follower state: follower="Node at 4a32a60e-1bea-6fec-9455-46430bb3a2e7 [Follower]" leader-address= leader-id= 2026-04-22T10:09:42.659Z [INFO] server-f075213b-e48a-3fa2-e465-848c9d1c01f3: entering follower state: follower="Node at f075213b-e48a-3fa2-e465-848c9d1c01f3 [Follower]" leader-address= leader-id= 2026-04-22T10:09:42.659Z [INFO] server-ae027834-6ce4-1cc8-fb04-85373f0d9bf2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4a32a60e-1bea-6fec-9455-46430bb3a2e7 Address:4a32a60e-1bea-6fec-9455-46430bb3a2e7} {Suffrage:Voter ID:server-f075213b-e48a-3fa2-e465-848c9d1c01f3 Address:f075213b-e48a-3fa2-e465-848c9d1c01f3} {Suffrage:Voter ID:server-ae027834-6ce4-1cc8-fb04-85373f0d9bf2 Address:ae027834-6ce4-1cc8-fb04-85373f0d9bf2}]" 2026-04-22T10:09:42.659Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:42.659Z [INFO] server-ae027834-6ce4-1cc8-fb04-85373f0d9bf2: entering follower state: follower="Node at ae027834-6ce4-1cc8-fb04-85373f0d9bf2 [Follower]" leader-address= leader-id= 2026-04-22T10:09:43.167Z [WARN] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:43.167Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: entering candidate state: node="Node at 4a32a60e-1bea-6fec-9455-46430bb3a2e7 [Candidate]" term=2 2026-04-22T10:09:43.167Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: election won: term=2 tally=2 2026-04-22T10:09:43.167Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: entering leader state: leader="Node at 4a32a60e-1bea-6fec-9455-46430bb3a2e7 [Leader]" 2026-04-22T10:09:43.167Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: added peer, starting replication: peer=server-f075213b-e48a-3fa2-e465-848c9d1c01f3 2026-04-22T10:09:43.167Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: added peer, starting replication: peer=server-ae027834-6ce4-1cc8-fb04-85373f0d9bf2 2026-04-22T10:09:43.167Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: pipelining replication: peer="{Voter server-ae027834-6ce4-1cc8-fb04-85373f0d9bf2 ae027834-6ce4-1cc8-fb04-85373f0d9bf2}" 2026-04-22T10:09:43.167Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: pipelining replication: peer="{Voter server-f075213b-e48a-3fa2-e465-848c9d1c01f3 f075213b-e48a-3fa2-e465-848c9d1c01f3}" 2026-04-22T10:09:44.173Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:43.167804558 +0000 UTC m=+29.191600645 (1 nodes), 508.427496ms from start of poll, 508.786947ms from cluster start. Timeout at 2026-04-22 10:09:44.17280549 +0000 UTC m=+30.196601573, 1.005000928s after stability 2026-04-22T10:09:44.173Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: starting snapshot up to: index=12 2026-04-22T10:09:44.173Z [INFO] creating new snapshot: path=/tmp/raft2942690260/snapshots/2-12-1776852584173.tmp 2026-04-22T10:09:44.174Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: snapshot complete up to: index=12 2026-04-22T10:09:44.175Z [INFO] creating new snapshot: path=/tmp/raft2942690260/snapshots/2-23-1776852584175.tmp 2026-04-22T10:09:44.176Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: copied to local snapshot: bytes=71 2026-04-22T10:09:44.176Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: snapshot restore progress: id=2-23-1776852584175 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:44.176Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: restored user snapshot: index=23 2026-04-22T10:09:44.176Z [ERROR] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: failed to get log: index=23 error="log not found" 2026-04-22T10:09:44.176Z [ERROR] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: failed to get log: index=23 error="log not found" 2026-04-22T10:09:44.176Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: aborting pipeline replication: peer="{Voter server-f075213b-e48a-3fa2-e465-848c9d1c01f3 f075213b-e48a-3fa2-e465-848c9d1c01f3}" 2026-04-22T10:09:44.176Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: aborting pipeline replication: peer="{Voter server-ae027834-6ce4-1cc8-fb04-85373f0d9bf2 ae027834-6ce4-1cc8-fb04-85373f0d9bf2}" 2026-04-22T10:09:44.184Z [ERROR] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: failed to get log: index=23 error="log not found" 2026-04-22T10:09:44.184Z [INFO] creating new snapshot: path=/tmp/raft1080222454/snapshots/2-23-1776852584184.tmp 2026-04-22T10:09:44.184Z [INFO] server-f075213b-e48a-3fa2-e465-848c9d1c01f3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:44.185Z [INFO] server-f075213b-e48a-3fa2-e465-848c9d1c01f3: copied to local snapshot: bytes=71 2026-04-22T10:09:44.185Z [INFO] server-f075213b-e48a-3fa2-e465-848c9d1c01f3: snapshot restore progress: id=2-23-1776852584184 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:44.185Z [ERROR] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: failed to get log: index=23 error="log not found" 2026-04-22T10:09:44.185Z [INFO] server-f075213b-e48a-3fa2-e465-848c9d1c01f3: Installed remote snapshot 2026-04-22T10:09:44.185Z [WARN] server-f075213b-e48a-3fa2-e465-848c9d1c01f3: skipping application of old log: index=22 2026-04-22T10:09:44.185Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: pipelining replication: peer="{Voter server-f075213b-e48a-3fa2-e465-848c9d1c01f3 f075213b-e48a-3fa2-e465-848c9d1c01f3}" 2026-04-22T10:09:44.185Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:44.185Z [INFO] creating new snapshot: path=/tmp/raft3183089841/snapshots/2-23-1776852584185.tmp 2026-04-22T10:09:44.186Z [INFO] server-ae027834-6ce4-1cc8-fb04-85373f0d9bf2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:44.186Z [INFO] server-ae027834-6ce4-1cc8-fb04-85373f0d9bf2: copied to local snapshot: bytes=71 2026-04-22T10:09:44.186Z [INFO] server-ae027834-6ce4-1cc8-fb04-85373f0d9bf2: snapshot restore progress: id=2-23-1776852584185 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:44.186Z [INFO] server-ae027834-6ce4-1cc8-fb04-85373f0d9bf2: Installed remote snapshot 2026-04-22T10:09:44.186Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: pipelining replication: peer="{Voter server-ae027834-6ce4-1cc8-fb04-85373f0d9bf2 ae027834-6ce4-1cc8-fb04-85373f0d9bf2}" 2026-04-22T10:09:45.190Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:44.185697819 +0000 UTC m=+30.209493925 (2 nodes), 1.113µs from start of poll, 1.526680227s from cluster start. Timeout at 2026-04-22 10:09:45.190705914 +0000 UTC m=+31.214502005, 1.00500808s after stability 2026-04-22T10:09:45.202Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: aborting pipeline replication: peer="{Voter server-ae027834-6ce4-1cc8-fb04-85373f0d9bf2 ae027834-6ce4-1cc8-fb04-85373f0d9bf2}" 2026-04-22T10:09:45.203Z [INFO] server-4a32a60e-1bea-6fec-9455-46430bb3a2e7: aborting pipeline replication: peer="{Voter server-f075213b-e48a-3fa2-e465-848c9d1c01f3 f075213b-e48a-3fa2-e465-848c9d1c01f3}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false 2026-04-22T10:09:45.204Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0784757-67f4-2897-a35a-71a1dfff95d4 Address:d0784757-67f4-2897-a35a-71a1dfff95d4} {Suffrage:Voter ID:server-46b862c5-258e-7e9f-f70e-c6369394eeb2 Address:46b862c5-258e-7e9f-f70e-c6369394eeb2} {Suffrage:Voter ID:server-7bae69ab-fd8f-bea1-167a-7301dcf276cc Address:7bae69ab-fd8f-bea1-167a-7301dcf276cc}]" 2026-04-22T10:09:45.204Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: entering follower state: follower="Node at d0784757-67f4-2897-a35a-71a1dfff95d4 [Follower]" leader-address= leader-id= 2026-04-22T10:09:45.204Z [INFO] server-46b862c5-258e-7e9f-f70e-c6369394eeb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0784757-67f4-2897-a35a-71a1dfff95d4 Address:d0784757-67f4-2897-a35a-71a1dfff95d4} {Suffrage:Voter ID:server-46b862c5-258e-7e9f-f70e-c6369394eeb2 Address:46b862c5-258e-7e9f-f70e-c6369394eeb2} {Suffrage:Voter ID:server-7bae69ab-fd8f-bea1-167a-7301dcf276cc Address:7bae69ab-fd8f-bea1-167a-7301dcf276cc}]" 2026-04-22T10:09:45.204Z [INFO] server-46b862c5-258e-7e9f-f70e-c6369394eeb2: entering follower state: follower="Node at 46b862c5-258e-7e9f-f70e-c6369394eeb2 [Follower]" leader-address= leader-id= 2026-04-22T10:09:45.204Z [INFO] server-7bae69ab-fd8f-bea1-167a-7301dcf276cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0784757-67f4-2897-a35a-71a1dfff95d4 Address:d0784757-67f4-2897-a35a-71a1dfff95d4} {Suffrage:Voter ID:server-46b862c5-258e-7e9f-f70e-c6369394eeb2 Address:46b862c5-258e-7e9f-f70e-c6369394eeb2} {Suffrage:Voter ID:server-7bae69ab-fd8f-bea1-167a-7301dcf276cc Address:7bae69ab-fd8f-bea1-167a-7301dcf276cc}]" 2026-04-22T10:09:45.204Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:45.204Z [INFO] server-7bae69ab-fd8f-bea1-167a-7301dcf276cc: entering follower state: follower="Node at 7bae69ab-fd8f-bea1-167a-7301dcf276cc [Follower]" leader-address= leader-id= 2026-04-22T10:09:45.733Z [WARN] server-d0784757-67f4-2897-a35a-71a1dfff95d4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:45.733Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: entering candidate state: node="Node at d0784757-67f4-2897-a35a-71a1dfff95d4 [Candidate]" term=2 2026-04-22T10:09:45.733Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: election won: term=2 tally=2 2026-04-22T10:09:45.733Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: entering leader state: leader="Node at d0784757-67f4-2897-a35a-71a1dfff95d4 [Leader]" 2026-04-22T10:09:45.733Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: added peer, starting replication: peer=server-46b862c5-258e-7e9f-f70e-c6369394eeb2 2026-04-22T10:09:45.733Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: added peer, starting replication: peer=server-7bae69ab-fd8f-bea1-167a-7301dcf276cc 2026-04-22T10:09:45.733Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: pipelining replication: peer="{Voter server-7bae69ab-fd8f-bea1-167a-7301dcf276cc 7bae69ab-fd8f-bea1-167a-7301dcf276cc}" 2026-04-22T10:09:45.733Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: pipelining replication: peer="{Voter server-46b862c5-258e-7e9f-f70e-c6369394eeb2 46b862c5-258e-7e9f-f70e-c6369394eeb2}" 2026-04-22T10:09:46.738Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:45.733327294 +0000 UTC m=+31.757123401 (1 nodes), 528.69613ms from start of poll, 529.063332ms from cluster start. Timeout at 2026-04-22 10:09:46.738333988 +0000 UTC m=+32.762130069, 1.005006668s after stability 2026-04-22T10:09:46.739Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: starting snapshot up to: index=12 2026-04-22T10:09:46.739Z [INFO] creating new snapshot: path=/tmp/raft497367915/snapshots/2-12-1776852586739.tmp 2026-04-22T10:09:46.739Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: snapshot complete up to: index=12 2026-04-22T10:09:46.740Z [INFO] creating new snapshot: path=/tmp/raft497367915/snapshots/2-23-1776852586740.tmp 2026-04-22T10:09:46.740Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: copied to local snapshot: bytes=71 2026-04-22T10:09:46.741Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: snapshot restore progress: id=2-23-1776852586740 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:46.741Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: removing all old logs from log store 2026-04-22T10:09:46.741Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: compacting logs: from=1 to=22 2026-04-22T10:09:46.741Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: restored user snapshot: index=23 2026-04-22T10:09:46.741Z [ERROR] server-d0784757-67f4-2897-a35a-71a1dfff95d4: failed to get log: index=22 error="log not found" 2026-04-22T10:09:46.741Z [ERROR] server-d0784757-67f4-2897-a35a-71a1dfff95d4: failed to get log: index=22 error="log not found" 2026-04-22T10:09:46.741Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: aborting pipeline replication: peer="{Voter server-46b862c5-258e-7e9f-f70e-c6369394eeb2 46b862c5-258e-7e9f-f70e-c6369394eeb2}" 2026-04-22T10:09:46.741Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: aborting pipeline replication: peer="{Voter server-7bae69ab-fd8f-bea1-167a-7301dcf276cc 7bae69ab-fd8f-bea1-167a-7301dcf276cc}" 2026-04-22T10:09:46.748Z [ERROR] server-d0784757-67f4-2897-a35a-71a1dfff95d4: failed to get log: index=22 error="log not found" 2026-04-22T10:09:46.748Z [INFO] creating new snapshot: path=/tmp/raft3782626090/snapshots/2-23-1776852586748.tmp 2026-04-22T10:09:46.749Z [INFO] server-7bae69ab-fd8f-bea1-167a-7301dcf276cc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:46.749Z [INFO] server-7bae69ab-fd8f-bea1-167a-7301dcf276cc: copied to local snapshot: bytes=71 2026-04-22T10:09:46.749Z [INFO] server-7bae69ab-fd8f-bea1-167a-7301dcf276cc: snapshot restore progress: id=2-23-1776852586748 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:46.749Z [INFO] server-7bae69ab-fd8f-bea1-167a-7301dcf276cc: removing all old logs from log store 2026-04-22T10:09:46.749Z [INFO] server-7bae69ab-fd8f-bea1-167a-7301dcf276cc: compacting logs: from=1 to=22 2026-04-22T10:09:46.749Z [INFO] server-7bae69ab-fd8f-bea1-167a-7301dcf276cc: Installed remote snapshot 2026-04-22T10:09:46.749Z [WARN] server-7bae69ab-fd8f-bea1-167a-7301dcf276cc: skipping application of old log: index=22 2026-04-22T10:09:46.749Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: pipelining replication: peer="{Voter server-7bae69ab-fd8f-bea1-167a-7301dcf276cc 7bae69ab-fd8f-bea1-167a-7301dcf276cc}" 2026-04-22T10:09:46.749Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:46.751Z [ERROR] server-d0784757-67f4-2897-a35a-71a1dfff95d4: failed to get log: index=22 error="log not found" 2026-04-22T10:09:46.752Z [INFO] creating new snapshot: path=/tmp/raft4186836534/snapshots/2-23-1776852586752.tmp 2026-04-22T10:09:46.752Z [INFO] server-46b862c5-258e-7e9f-f70e-c6369394eeb2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:46.752Z [INFO] server-46b862c5-258e-7e9f-f70e-c6369394eeb2: copied to local snapshot: bytes=71 2026-04-22T10:09:46.752Z [INFO] server-46b862c5-258e-7e9f-f70e-c6369394eeb2: snapshot restore progress: id=2-23-1776852586752 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:46.752Z [INFO] server-46b862c5-258e-7e9f-f70e-c6369394eeb2: removing all old logs from log store 2026-04-22T10:09:46.752Z [INFO] server-46b862c5-258e-7e9f-f70e-c6369394eeb2: compacting logs: from=1 to=22 2026-04-22T10:09:46.752Z [INFO] server-46b862c5-258e-7e9f-f70e-c6369394eeb2: Installed remote snapshot 2026-04-22T10:09:46.752Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: pipelining replication: peer="{Voter server-46b862c5-258e-7e9f-f70e-c6369394eeb2 46b862c5-258e-7e9f-f70e-c6369394eeb2}" 2026-04-22T10:09:47.755Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:46.74981997 +0000 UTC m=+32.773616071 (2 nodes), 1.181µs from start of poll, 1.545556002s from cluster start. Timeout at 2026-04-22 10:09:47.754826817 +0000 UTC m=+33.778622899, 1.005006828s after stability 2026-04-22T10:09:47.767Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: aborting pipeline replication: peer="{Voter server-7bae69ab-fd8f-bea1-167a-7301dcf276cc 7bae69ab-fd8f-bea1-167a-7301dcf276cc}" 2026-04-22T10:09:47.767Z [INFO] server-d0784757-67f4-2897-a35a-71a1dfff95d4: aborting pipeline replication: peer="{Voter server-46b862c5-258e-7e9f-f70e-c6369394eeb2 46b862c5-258e-7e9f-f70e-c6369394eeb2}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true 2026-04-22T10:09:47.769Z [INFO] server-53b930d6-902a-8d19-e980-e348be0ad61e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53b930d6-902a-8d19-e980-e348be0ad61e Address:53b930d6-902a-8d19-e980-e348be0ad61e} {Suffrage:Voter ID:server-cf51392d-7455-69a0-0f12-a233b2f12d74 Address:cf51392d-7455-69a0-0f12-a233b2f12d74} {Suffrage:Voter ID:server-ea2aca6b-3b7c-35b6-d187-cc0568eb2d1b Address:ea2aca6b-3b7c-35b6-d187-cc0568eb2d1b}]" 2026-04-22T10:09:47.769Z [INFO] server-53b930d6-902a-8d19-e980-e348be0ad61e: entering follower state: follower="Node at 53b930d6-902a-8d19-e980-e348be0ad61e [Follower]" leader-address= leader-id= 2026-04-22T10:09:47.769Z [INFO] server-cf51392d-7455-69a0-0f12-a233b2f12d74: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53b930d6-902a-8d19-e980-e348be0ad61e Address:53b930d6-902a-8d19-e980-e348be0ad61e} {Suffrage:Voter ID:server-cf51392d-7455-69a0-0f12-a233b2f12d74 Address:cf51392d-7455-69a0-0f12-a233b2f12d74} {Suffrage:Voter ID:server-ea2aca6b-3b7c-35b6-d187-cc0568eb2d1b Address:ea2aca6b-3b7c-35b6-d187-cc0568eb2d1b}]" 2026-04-22T10:09:47.769Z [INFO] server-cf51392d-7455-69a0-0f12-a233b2f12d74: entering follower state: follower="Node at cf51392d-7455-69a0-0f12-a233b2f12d74 [Follower]" leader-address= leader-id= 2026-04-22T10:09:47.769Z [INFO] server-ea2aca6b-3b7c-35b6-d187-cc0568eb2d1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-53b930d6-902a-8d19-e980-e348be0ad61e Address:53b930d6-902a-8d19-e980-e348be0ad61e} {Suffrage:Voter ID:server-cf51392d-7455-69a0-0f12-a233b2f12d74 Address:cf51392d-7455-69a0-0f12-a233b2f12d74} {Suffrage:Voter ID:server-ea2aca6b-3b7c-35b6-d187-cc0568eb2d1b Address:ea2aca6b-3b7c-35b6-d187-cc0568eb2d1b}]" 2026-04-22T10:09:47.770Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:47.770Z [INFO] server-ea2aca6b-3b7c-35b6-d187-cc0568eb2d1b: entering follower state: follower="Node at ea2aca6b-3b7c-35b6-d187-cc0568eb2d1b [Follower]" leader-address= leader-id= 2026-04-22T10:09:48.503Z [WARN] server-cf51392d-7455-69a0-0f12-a233b2f12d74: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:48.503Z [INFO] server-cf51392d-7455-69a0-0f12-a233b2f12d74: entering candidate state: node="Node at cf51392d-7455-69a0-0f12-a233b2f12d74 [Candidate]" term=2 2026-04-22T10:09:48.503Z [INFO] server-cf51392d-7455-69a0-0f12-a233b2f12d74: election won: term=2 tally=2 2026-04-22T10:09:48.503Z [INFO] server-cf51392d-7455-69a0-0f12-a233b2f12d74: entering leader state: leader="Node at cf51392d-7455-69a0-0f12-a233b2f12d74 [Leader]" 2026-04-22T10:09:48.503Z [INFO] server-cf51392d-7455-69a0-0f12-a233b2f12d74: added peer, starting replication: peer=server-53b930d6-902a-8d19-e980-e348be0ad61e 2026-04-22T10:09:48.503Z [INFO] server-cf51392d-7455-69a0-0f12-a233b2f12d74: added peer, starting replication: peer=server-ea2aca6b-3b7c-35b6-d187-cc0568eb2d1b 2026-04-22T10:09:48.503Z [INFO] server-cf51392d-7455-69a0-0f12-a233b2f12d74: pipelining replication: peer="{Voter server-53b930d6-902a-8d19-e980-e348be0ad61e 53b930d6-902a-8d19-e980-e348be0ad61e}" 2026-04-22T10:09:48.503Z [INFO] server-cf51392d-7455-69a0-0f12-a233b2f12d74: pipelining replication: peer="{Voter server-ea2aca6b-3b7c-35b6-d187-cc0568eb2d1b ea2aca6b-3b7c-35b6-d187-cc0568eb2d1b}" 2026-04-22T10:09:49.509Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:48.503454203 +0000 UTC m=+34.527250306 (1 nodes), 733.41144ms from start of poll, 734.072777ms from cluster start. Timeout at 2026-04-22 10:09:49.508460181 +0000 UTC m=+35.532256264, 1.005005958s after stability 2026-04-22T10:09:49.510Z [INFO] server-cf51392d-7455-69a0-0f12-a233b2f12d74: starting snapshot up to: index=12 2026-04-22T10:09:49.510Z [INFO] creating new snapshot: path=/tmp/raft1055243269/snapshots/2-12-1776852589510.tmp 2026-04-22T10:09:49.511Z [INFO] server-cf51392d-7455-69a0-0f12-a233b2f12d74: snapshot complete up to: index=12 2026-04-22T10:09:49.512Z [INFO] server-7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021 Address:7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021} {Suffrage:Voter ID:server-e3d31228-e3dc-2e66-b62f-fcb19190190f Address:e3d31228-e3dc-2e66-b62f-fcb19190190f} {Suffrage:Voter ID:server-a4c36d7f-f625-4c52-c844-32d87b5ced9e Address:a4c36d7f-f625-4c52-c844-32d87b5ced9e}]" 2026-04-22T10:09:49.512Z [INFO] server-7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021: entering follower state: follower="Node at 7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021 [Follower]" leader-address= leader-id= 2026-04-22T10:09:49.512Z [INFO] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021 Address:7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021} {Suffrage:Voter ID:server-e3d31228-e3dc-2e66-b62f-fcb19190190f Address:e3d31228-e3dc-2e66-b62f-fcb19190190f} {Suffrage:Voter ID:server-a4c36d7f-f625-4c52-c844-32d87b5ced9e Address:a4c36d7f-f625-4c52-c844-32d87b5ced9e}]" 2026-04-22T10:09:49.512Z [INFO] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: entering follower state: follower="Node at e3d31228-e3dc-2e66-b62f-fcb19190190f [Follower]" leader-address= leader-id= 2026-04-22T10:09:49.512Z [INFO] server-a4c36d7f-f625-4c52-c844-32d87b5ced9e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021 Address:7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021} {Suffrage:Voter ID:server-e3d31228-e3dc-2e66-b62f-fcb19190190f Address:e3d31228-e3dc-2e66-b62f-fcb19190190f} {Suffrage:Voter ID:server-a4c36d7f-f625-4c52-c844-32d87b5ced9e Address:a4c36d7f-f625-4c52-c844-32d87b5ced9e}]" 2026-04-22T10:09:49.512Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:49.512Z [INFO] server-a4c36d7f-f625-4c52-c844-32d87b5ced9e: entering follower state: follower="Node at a4c36d7f-f625-4c52-c844-32d87b5ced9e [Follower]" leader-address= leader-id= 2026-04-22T10:09:50.068Z [WARN] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:50.068Z [INFO] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: entering candidate state: node="Node at e3d31228-e3dc-2e66-b62f-fcb19190190f [Candidate]" term=2 2026-04-22T10:09:50.068Z [INFO] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: election won: term=2 tally=2 2026-04-22T10:09:50.068Z [INFO] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: entering leader state: leader="Node at e3d31228-e3dc-2e66-b62f-fcb19190190f [Leader]" 2026-04-22T10:09:50.068Z [INFO] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: added peer, starting replication: peer=server-7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021 2026-04-22T10:09:50.068Z [INFO] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: added peer, starting replication: peer=server-a4c36d7f-f625-4c52-c844-32d87b5ced9e 2026-04-22T10:09:50.068Z [INFO] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: pipelining replication: peer="{Voter server-a4c36d7f-f625-4c52-c844-32d87b5ced9e a4c36d7f-f625-4c52-c844-32d87b5ced9e}" 2026-04-22T10:09:50.068Z [INFO] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: pipelining replication: peer="{Voter server-7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021 7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021}" 2026-04-22T10:09:51.073Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:50.068372724 +0000 UTC m=+36.092168815 (1 nodes), 555.78727ms from start of poll, 556.129253ms from cluster start. Timeout at 2026-04-22 10:09:51.073377139 +0000 UTC m=+37.097173221, 1.005004406s after stability 2026-04-22T10:09:51.074Z [INFO] creating new snapshot: path=/tmp/raft3643439632/snapshots/2-14-1776852591074.tmp 2026-04-22T10:09:51.074Z [INFO] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: copied to local snapshot: bytes=71 2026-04-22T10:09:51.075Z [INFO] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: snapshot restore progress: id=2-14-1776852591074 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:51.075Z [INFO] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: restored user snapshot: index=14 2026-04-22T10:09:51.075Z [ERROR] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: failed to get log: index=3 error="log not found" 2026-04-22T10:09:51.075Z [INFO] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: aborting pipeline replication: peer="{Voter server-a4c36d7f-f625-4c52-c844-32d87b5ced9e a4c36d7f-f625-4c52-c844-32d87b5ced9e}" 2026-04-22T10:09:51.075Z [ERROR] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: failed to get log: index=3 error="log not found" 2026-04-22T10:09:51.075Z [INFO] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: aborting pipeline replication: peer="{Voter server-7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021 7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021}" 2026-04-22T10:09:51.081Z [ERROR] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: failed to get log: index=3 error="log not found" 2026-04-22T10:09:51.081Z [INFO] creating new snapshot: path=/tmp/raft2369900500/snapshots/2-14-1776852591081.tmp 2026-04-22T10:09:51.081Z [INFO] server-a4c36d7f-f625-4c52-c844-32d87b5ced9e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:51.082Z [INFO] server-a4c36d7f-f625-4c52-c844-32d87b5ced9e: copied to local snapshot: bytes=71 2026-04-22T10:09:51.082Z [ERROR] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: failed to get log: index=3 error="log not found" 2026-04-22T10:09:51.082Z [INFO] server-a4c36d7f-f625-4c52-c844-32d87b5ced9e: snapshot restore progress: id=2-14-1776852591081 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:51.082Z [INFO] server-a4c36d7f-f625-4c52-c844-32d87b5ced9e: Installed remote snapshot 2026-04-22T10:09:51.082Z [INFO] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: pipelining replication: peer="{Voter server-a4c36d7f-f625-4c52-c844-32d87b5ced9e a4c36d7f-f625-4c52-c844-32d87b5ced9e}" 2026-04-22T10:09:51.082Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:51.082Z [INFO] creating new snapshot: path=/tmp/raft1668314335/snapshots/2-14-1776852591082.tmp 2026-04-22T10:09:51.082Z [INFO] server-7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:51.083Z [INFO] server-7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021: copied to local snapshot: bytes=71 2026-04-22T10:09:51.083Z [INFO] server-7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021: snapshot restore progress: id=2-14-1776852591082 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:51.083Z [INFO] server-7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021: Installed remote snapshot 2026-04-22T10:09:51.083Z [INFO] server-e3d31228-e3dc-2e66-b62f-fcb19190190f: pipelining replication: peer="{Voter server-7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021 7cbd52b0-7c6b-a761-0e5f-e1ae8ad22021}" 2026-04-22T10:09:52.088Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:51.082630747 +0000 UTC m=+37.106426834 (2 nodes), 637ns from start of poll, 1.570387272s from cluster start. Timeout at 2026-04-22 10:09:52.087635182 +0000 UTC m=+38.111431273, 1.005004439s after stability 2026-04-22T10:09:52.100Z [INFO] server-cf51392d-7455-69a0-0f12-a233b2f12d74: aborting pipeline replication: peer="{Voter server-53b930d6-902a-8d19-e980-e348be0ad61e 53b930d6-902a-8d19-e980-e348be0ad61e}" 2026-04-22T10:09:52.100Z [INFO] server-cf51392d-7455-69a0-0f12-a233b2f12d74: aborting pipeline replication: peer="{Voter server-ea2aca6b-3b7c-35b6-d187-cc0568eb2d1b ea2aca6b-3b7c-35b6-d187-cc0568eb2d1b}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true 2026-04-22T10:09:52.101Z [INFO] server-8b27b89f-d5e4-c93b-e9b1-f353da681c24: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b27b89f-d5e4-c93b-e9b1-f353da681c24 Address:8b27b89f-d5e4-c93b-e9b1-f353da681c24} {Suffrage:Voter ID:server-8aff63e6-9a39-93bf-214d-da04770c33b2 Address:8aff63e6-9a39-93bf-214d-da04770c33b2} {Suffrage:Voter ID:server-a52c44bb-6097-a987-ee9a-5ac877098a02 Address:a52c44bb-6097-a987-ee9a-5ac877098a02}]" 2026-04-22T10:09:52.101Z [INFO] server-8b27b89f-d5e4-c93b-e9b1-f353da681c24: entering follower state: follower="Node at 8b27b89f-d5e4-c93b-e9b1-f353da681c24 [Follower]" leader-address= leader-id= 2026-04-22T10:09:52.101Z [INFO] server-8aff63e6-9a39-93bf-214d-da04770c33b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b27b89f-d5e4-c93b-e9b1-f353da681c24 Address:8b27b89f-d5e4-c93b-e9b1-f353da681c24} {Suffrage:Voter ID:server-8aff63e6-9a39-93bf-214d-da04770c33b2 Address:8aff63e6-9a39-93bf-214d-da04770c33b2} {Suffrage:Voter ID:server-a52c44bb-6097-a987-ee9a-5ac877098a02 Address:a52c44bb-6097-a987-ee9a-5ac877098a02}]" 2026-04-22T10:09:52.101Z [INFO] server-8aff63e6-9a39-93bf-214d-da04770c33b2: entering follower state: follower="Node at 8aff63e6-9a39-93bf-214d-da04770c33b2 [Follower]" leader-address= leader-id= 2026-04-22T10:09:52.101Z [INFO] server-a52c44bb-6097-a987-ee9a-5ac877098a02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b27b89f-d5e4-c93b-e9b1-f353da681c24 Address:8b27b89f-d5e4-c93b-e9b1-f353da681c24} {Suffrage:Voter ID:server-8aff63e6-9a39-93bf-214d-da04770c33b2 Address:8aff63e6-9a39-93bf-214d-da04770c33b2} {Suffrage:Voter ID:server-a52c44bb-6097-a987-ee9a-5ac877098a02 Address:a52c44bb-6097-a987-ee9a-5ac877098a02}]" 2026-04-22T10:09:52.101Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:52.101Z [INFO] server-a52c44bb-6097-a987-ee9a-5ac877098a02: entering follower state: follower="Node at a52c44bb-6097-a987-ee9a-5ac877098a02 [Follower]" leader-address= leader-id= 2026-04-22T10:09:52.698Z [WARN] server-8b27b89f-d5e4-c93b-e9b1-f353da681c24: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:52.698Z [INFO] server-8b27b89f-d5e4-c93b-e9b1-f353da681c24: entering candidate state: node="Node at 8b27b89f-d5e4-c93b-e9b1-f353da681c24 [Candidate]" term=2 2026-04-22T10:09:52.699Z [INFO] server-8b27b89f-d5e4-c93b-e9b1-f353da681c24: election won: term=2 tally=2 2026-04-22T10:09:52.699Z [INFO] server-8b27b89f-d5e4-c93b-e9b1-f353da681c24: entering leader state: leader="Node at 8b27b89f-d5e4-c93b-e9b1-f353da681c24 [Leader]" 2026-04-22T10:09:52.699Z [INFO] server-8b27b89f-d5e4-c93b-e9b1-f353da681c24: added peer, starting replication: peer=server-8aff63e6-9a39-93bf-214d-da04770c33b2 2026-04-22T10:09:52.699Z [INFO] server-8b27b89f-d5e4-c93b-e9b1-f353da681c24: added peer, starting replication: peer=server-a52c44bb-6097-a987-ee9a-5ac877098a02 2026-04-22T10:09:52.699Z [INFO] server-8b27b89f-d5e4-c93b-e9b1-f353da681c24: pipelining replication: peer="{Voter server-8aff63e6-9a39-93bf-214d-da04770c33b2 8aff63e6-9a39-93bf-214d-da04770c33b2}" 2026-04-22T10:09:52.699Z [INFO] server-8b27b89f-d5e4-c93b-e9b1-f353da681c24: pipelining replication: peer="{Voter server-a52c44bb-6097-a987-ee9a-5ac877098a02 a52c44bb-6097-a987-ee9a-5ac877098a02}" 2026-04-22T10:09:53.704Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:52.699046123 +0000 UTC m=+38.722842213 (1 nodes), 597.605527ms from start of poll, 597.945458ms from cluster start. Timeout at 2026-04-22 10:09:53.704054544 +0000 UTC m=+39.727850684, 1.005008471s after stability 2026-04-22T10:09:53.705Z [INFO] server-8b27b89f-d5e4-c93b-e9b1-f353da681c24: starting snapshot up to: index=12 2026-04-22T10:09:53.705Z [INFO] creating new snapshot: path=/tmp/raft3463197649/snapshots/2-12-1776852593705.tmp 2026-04-22T10:09:53.706Z [INFO] server-8b27b89f-d5e4-c93b-e9b1-f353da681c24: snapshot complete up to: index=12 2026-04-22T10:09:53.707Z [INFO] server-9f7439a0-2690-561a-4295-20cccd20213c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f7439a0-2690-561a-4295-20cccd20213c Address:9f7439a0-2690-561a-4295-20cccd20213c} {Suffrage:Voter ID:server-1c6aef7e-7775-6a76-f793-c3d236133f54 Address:1c6aef7e-7775-6a76-f793-c3d236133f54} {Suffrage:Voter ID:server-8a4e42f2-0df2-c039-d5fd-6f32ecbc8053 Address:8a4e42f2-0df2-c039-d5fd-6f32ecbc8053}]" 2026-04-22T10:09:53.707Z [INFO] server-9f7439a0-2690-561a-4295-20cccd20213c: entering follower state: follower="Node at 9f7439a0-2690-561a-4295-20cccd20213c [Follower]" leader-address= leader-id= 2026-04-22T10:09:53.707Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f7439a0-2690-561a-4295-20cccd20213c Address:9f7439a0-2690-561a-4295-20cccd20213c} {Suffrage:Voter ID:server-1c6aef7e-7775-6a76-f793-c3d236133f54 Address:1c6aef7e-7775-6a76-f793-c3d236133f54} {Suffrage:Voter ID:server-8a4e42f2-0df2-c039-d5fd-6f32ecbc8053 Address:8a4e42f2-0df2-c039-d5fd-6f32ecbc8053}]" 2026-04-22T10:09:53.707Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: entering follower state: follower="Node at 1c6aef7e-7775-6a76-f793-c3d236133f54 [Follower]" leader-address= leader-id= 2026-04-22T10:09:53.707Z [INFO] server-8a4e42f2-0df2-c039-d5fd-6f32ecbc8053: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f7439a0-2690-561a-4295-20cccd20213c Address:9f7439a0-2690-561a-4295-20cccd20213c} {Suffrage:Voter ID:server-1c6aef7e-7775-6a76-f793-c3d236133f54 Address:1c6aef7e-7775-6a76-f793-c3d236133f54} {Suffrage:Voter ID:server-8a4e42f2-0df2-c039-d5fd-6f32ecbc8053 Address:8a4e42f2-0df2-c039-d5fd-6f32ecbc8053}]" 2026-04-22T10:09:53.707Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:53.707Z [INFO] server-8a4e42f2-0df2-c039-d5fd-6f32ecbc8053: entering follower state: follower="Node at 8a4e42f2-0df2-c039-d5fd-6f32ecbc8053 [Follower]" leader-address= leader-id= 2026-04-22T10:09:54.284Z [WARN] server-1c6aef7e-7775-6a76-f793-c3d236133f54: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:54.284Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: entering candidate state: node="Node at 1c6aef7e-7775-6a76-f793-c3d236133f54 [Candidate]" term=2 2026-04-22T10:09:54.284Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: election won: term=2 tally=2 2026-04-22T10:09:54.284Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: entering leader state: leader="Node at 1c6aef7e-7775-6a76-f793-c3d236133f54 [Leader]" 2026-04-22T10:09:54.284Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: added peer, starting replication: peer=server-9f7439a0-2690-561a-4295-20cccd20213c 2026-04-22T10:09:54.284Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: added peer, starting replication: peer=server-8a4e42f2-0df2-c039-d5fd-6f32ecbc8053 2026-04-22T10:09:54.284Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: pipelining replication: peer="{Voter server-9f7439a0-2690-561a-4295-20cccd20213c 9f7439a0-2690-561a-4295-20cccd20213c}" 2026-04-22T10:09:54.284Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: pipelining replication: peer="{Voter server-8a4e42f2-0df2-c039-d5fd-6f32ecbc8053 8a4e42f2-0df2-c039-d5fd-6f32ecbc8053}" 2026-04-22T10:09:55.290Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:54.28471058 +0000 UTC m=+40.308506734 (1 nodes), 577.292744ms from start of poll, 577.603244ms from cluster start. Timeout at 2026-04-22 10:09:55.289712879 +0000 UTC m=+41.313509019, 1.005002285s after stability 2026-04-22T10:09:55.290Z [INFO] creating new snapshot: path=/tmp/raft3638447232/snapshots/2-14-1776852595290.tmp 2026-04-22T10:09:55.291Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: copied to local snapshot: bytes=71 2026-04-22T10:09:55.291Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: snapshot restore progress: id=2-14-1776852595290 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:55.291Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: removing all old logs from log store 2026-04-22T10:09:55.291Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: compacting logs: from=1 to=2 2026-04-22T10:09:55.291Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: restored user snapshot: index=14 2026-04-22T10:09:55.291Z [ERROR] server-1c6aef7e-7775-6a76-f793-c3d236133f54: failed to get log: index=2 error="log not found" 2026-04-22T10:09:55.291Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: aborting pipeline replication: peer="{Voter server-9f7439a0-2690-561a-4295-20cccd20213c 9f7439a0-2690-561a-4295-20cccd20213c}" 2026-04-22T10:09:55.291Z [ERROR] server-1c6aef7e-7775-6a76-f793-c3d236133f54: failed to get log: index=2 error="log not found" 2026-04-22T10:09:55.291Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: aborting pipeline replication: peer="{Voter server-8a4e42f2-0df2-c039-d5fd-6f32ecbc8053 8a4e42f2-0df2-c039-d5fd-6f32ecbc8053}" 2026-04-22T10:09:55.299Z [ERROR] server-1c6aef7e-7775-6a76-f793-c3d236133f54: failed to get log: index=2 error="log not found" 2026-04-22T10:09:55.299Z [ERROR] server-1c6aef7e-7775-6a76-f793-c3d236133f54: failed to get log: index=2 error="log not found" 2026-04-22T10:09:55.299Z [INFO] creating new snapshot: path=/tmp/raft2409031489/snapshots/2-14-1776852595299.tmp 2026-04-22T10:09:55.299Z [INFO] creating new snapshot: path=/tmp/raft2495343654/snapshots/2-14-1776852595299.tmp 2026-04-22T10:09:55.299Z [INFO] server-9f7439a0-2690-561a-4295-20cccd20213c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:55.299Z [INFO] server-8a4e42f2-0df2-c039-d5fd-6f32ecbc8053: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:55.299Z [INFO] server-9f7439a0-2690-561a-4295-20cccd20213c: copied to local snapshot: bytes=71 2026-04-22T10:09:55.299Z [INFO] server-8a4e42f2-0df2-c039-d5fd-6f32ecbc8053: copied to local snapshot: bytes=71 2026-04-22T10:09:55.300Z [INFO] server-9f7439a0-2690-561a-4295-20cccd20213c: snapshot restore progress: id=2-14-1776852595299 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:55.300Z [INFO] server-8a4e42f2-0df2-c039-d5fd-6f32ecbc8053: snapshot restore progress: id=2-14-1776852595299 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:55.300Z [INFO] server-9f7439a0-2690-561a-4295-20cccd20213c: removing all old logs from log store 2026-04-22T10:09:55.300Z [INFO] server-9f7439a0-2690-561a-4295-20cccd20213c: compacting logs: from=1 to=2 2026-04-22T10:09:55.300Z [INFO] server-9f7439a0-2690-561a-4295-20cccd20213c: Installed remote snapshot 2026-04-22T10:09:55.300Z [INFO] server-8a4e42f2-0df2-c039-d5fd-6f32ecbc8053: removing all old logs from log store 2026-04-22T10:09:55.300Z [INFO] server-8a4e42f2-0df2-c039-d5fd-6f32ecbc8053: compacting logs: from=1 to=2 2026-04-22T10:09:55.300Z [INFO] server-8a4e42f2-0df2-c039-d5fd-6f32ecbc8053: Installed remote snapshot 2026-04-22T10:09:55.300Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: pipelining replication: peer="{Voter server-9f7439a0-2690-561a-4295-20cccd20213c 9f7439a0-2690-561a-4295-20cccd20213c}" 2026-04-22T10:09:55.300Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:55.300Z [INFO] server-1c6aef7e-7775-6a76-f793-c3d236133f54: pipelining replication: peer="{Voter server-8a4e42f2-0df2-c039-d5fd-6f32ecbc8053 8a4e42f2-0df2-c039-d5fd-6f32ecbc8053}" 2026-04-22T10:09:56.306Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:55.300350688 +0000 UTC m=+41.324146775 (2 nodes), 581ns from start of poll, 1.593243285s from cluster start. Timeout at 2026-04-22 10:09:56.305381477 +0000 UTC m=+42.329177595, 1.00503082s after stability 2026-04-22T10:09:56.317Z [INFO] server-8b27b89f-d5e4-c93b-e9b1-f353da681c24: aborting pipeline replication: peer="{Voter server-a52c44bb-6097-a987-ee9a-5ac877098a02 a52c44bb-6097-a987-ee9a-5ac877098a02}" 2026-04-22T10:09:56.317Z [INFO] server-8b27b89f-d5e4-c93b-e9b1-f353da681c24: aborting pipeline replication: peer="{Voter server-8aff63e6-9a39-93bf-214d-da04770c33b2 8aff63e6-9a39-93bf-214d-da04770c33b2}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false 2026-04-22T10:09:56.318Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b44941bc-d98d-7ed8-846a-0399c47c7e63 Address:b44941bc-d98d-7ed8-846a-0399c47c7e63} {Suffrage:Voter ID:server-bbc31690-20b1-f984-7020-21c4f3d4154c Address:bbc31690-20b1-f984-7020-21c4f3d4154c} {Suffrage:Voter ID:server-741138a3-a55c-751a-0831-eb7b7c09d073 Address:741138a3-a55c-751a-0831-eb7b7c09d073}]" 2026-04-22T10:09:56.318Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: entering follower state: follower="Node at b44941bc-d98d-7ed8-846a-0399c47c7e63 [Follower]" leader-address= leader-id= 2026-04-22T10:09:56.318Z [INFO] server-bbc31690-20b1-f984-7020-21c4f3d4154c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b44941bc-d98d-7ed8-846a-0399c47c7e63 Address:b44941bc-d98d-7ed8-846a-0399c47c7e63} {Suffrage:Voter ID:server-bbc31690-20b1-f984-7020-21c4f3d4154c Address:bbc31690-20b1-f984-7020-21c4f3d4154c} {Suffrage:Voter ID:server-741138a3-a55c-751a-0831-eb7b7c09d073 Address:741138a3-a55c-751a-0831-eb7b7c09d073}]" 2026-04-22T10:09:56.318Z [INFO] server-bbc31690-20b1-f984-7020-21c4f3d4154c: entering follower state: follower="Node at bbc31690-20b1-f984-7020-21c4f3d4154c [Follower]" leader-address= leader-id= 2026-04-22T10:09:56.318Z [INFO] server-741138a3-a55c-751a-0831-eb7b7c09d073: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b44941bc-d98d-7ed8-846a-0399c47c7e63 Address:b44941bc-d98d-7ed8-846a-0399c47c7e63} {Suffrage:Voter ID:server-bbc31690-20b1-f984-7020-21c4f3d4154c Address:bbc31690-20b1-f984-7020-21c4f3d4154c} {Suffrage:Voter ID:server-741138a3-a55c-751a-0831-eb7b7c09d073 Address:741138a3-a55c-751a-0831-eb7b7c09d073}]" 2026-04-22T10:09:56.318Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:56.318Z [INFO] server-741138a3-a55c-751a-0831-eb7b7c09d073: entering follower state: follower="Node at 741138a3-a55c-751a-0831-eb7b7c09d073 [Follower]" leader-address= leader-id= 2026-04-22T10:09:56.840Z [WARN] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:56.840Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: entering candidate state: node="Node at b44941bc-d98d-7ed8-846a-0399c47c7e63 [Candidate]" term=2 2026-04-22T10:09:56.840Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: election won: term=2 tally=2 2026-04-22T10:09:56.840Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: entering leader state: leader="Node at b44941bc-d98d-7ed8-846a-0399c47c7e63 [Leader]" 2026-04-22T10:09:56.840Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: added peer, starting replication: peer=server-bbc31690-20b1-f984-7020-21c4f3d4154c 2026-04-22T10:09:56.840Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: added peer, starting replication: peer=server-741138a3-a55c-751a-0831-eb7b7c09d073 2026-04-22T10:09:56.841Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: pipelining replication: peer="{Voter server-741138a3-a55c-751a-0831-eb7b7c09d073 741138a3-a55c-751a-0831-eb7b7c09d073}" 2026-04-22T10:09:56.841Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: pipelining replication: peer="{Voter server-bbc31690-20b1-f984-7020-21c4f3d4154c bbc31690-20b1-f984-7020-21c4f3d4154c}" 2026-04-22T10:09:57.845Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:56.840932603 +0000 UTC m=+42.864728709 (1 nodes), 522.291191ms from start of poll, 522.608604ms from cluster start. Timeout at 2026-04-22 10:09:57.84593666 +0000 UTC m=+43.869732743, 1.005004034s after stability 2026-04-22T10:09:57.846Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: starting snapshot up to: index=12 2026-04-22T10:09:57.846Z [INFO] creating new snapshot: path=/tmp/raft2974169161/snapshots/2-12-1776852597846.tmp 2026-04-22T10:09:57.846Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: snapshot complete up to: index=12 2026-04-22T10:09:57.847Z [INFO] creating new snapshot: path=/tmp/raft2974169161/snapshots/2-23-1776852597847.tmp 2026-04-22T10:09:57.848Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: copied to local snapshot: bytes=71 2026-04-22T10:09:57.848Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: snapshot restore progress: id=2-23-1776852597847 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:57.848Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: restored user snapshot: index=23 2026-04-22T10:09:57.848Z [ERROR] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: failed to get log: index=23 error="log not found" 2026-04-22T10:09:57.848Z [ERROR] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: failed to get log: index=23 error="log not found" 2026-04-22T10:09:57.848Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: aborting pipeline replication: peer="{Voter server-741138a3-a55c-751a-0831-eb7b7c09d073 741138a3-a55c-751a-0831-eb7b7c09d073}" 2026-04-22T10:09:57.848Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: aborting pipeline replication: peer="{Voter server-bbc31690-20b1-f984-7020-21c4f3d4154c bbc31690-20b1-f984-7020-21c4f3d4154c}" 2026-04-22T10:09:57.857Z [ERROR] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: failed to get log: index=23 error="log not found" 2026-04-22T10:09:57.858Z [INFO] creating new snapshot: path=/tmp/raft1364874649/snapshots/2-23-1776852597858.tmp 2026-04-22T10:09:57.858Z [INFO] server-bbc31690-20b1-f984-7020-21c4f3d4154c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:57.858Z [ERROR] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: failed to get log: index=23 error="log not found" 2026-04-22T10:09:57.858Z [INFO] server-bbc31690-20b1-f984-7020-21c4f3d4154c: copied to local snapshot: bytes=71 2026-04-22T10:09:57.858Z [INFO] creating new snapshot: path=/tmp/raft2172584266/snapshots/2-23-1776852597858.tmp 2026-04-22T10:09:57.858Z [INFO] server-bbc31690-20b1-f984-7020-21c4f3d4154c: snapshot restore progress: id=2-23-1776852597858 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:57.858Z [INFO] server-bbc31690-20b1-f984-7020-21c4f3d4154c: Installed remote snapshot 2026-04-22T10:09:57.859Z [WARN] server-bbc31690-20b1-f984-7020-21c4f3d4154c: skipping application of old log: index=22 2026-04-22T10:09:57.859Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: pipelining replication: peer="{Voter server-bbc31690-20b1-f984-7020-21c4f3d4154c bbc31690-20b1-f984-7020-21c4f3d4154c}" 2026-04-22T10:09:57.859Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:09:57.859Z [INFO] server-741138a3-a55c-751a-0831-eb7b7c09d073: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:57.859Z [INFO] server-741138a3-a55c-751a-0831-eb7b7c09d073: copied to local snapshot: bytes=71 2026-04-22T10:09:57.859Z [INFO] server-741138a3-a55c-751a-0831-eb7b7c09d073: snapshot restore progress: id=2-23-1776852597858 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:09:57.859Z [INFO] server-741138a3-a55c-751a-0831-eb7b7c09d073: Installed remote snapshot 2026-04-22T10:09:57.859Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: pipelining replication: peer="{Voter server-741138a3-a55c-751a-0831-eb7b7c09d073 741138a3-a55c-751a-0831-eb7b7c09d073}" 2026-04-22T10:09:58.864Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:09:57.85910028 +0000 UTC m=+43.882896368 (2 nodes), 612ns from start of poll, 1.540776263s from cluster start. Timeout at 2026-04-22 10:09:58.864113575 +0000 UTC m=+44.887909745, 1.005013377s after stability 2026-04-22T10:09:58.876Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: aborting pipeline replication: peer="{Voter server-741138a3-a55c-751a-0831-eb7b7c09d073 741138a3-a55c-751a-0831-eb7b7c09d073}" 2026-04-22T10:09:58.876Z [INFO] server-b44941bc-d98d-7ed8-846a-0399c47c7e63: aborting pipeline replication: peer="{Voter server-bbc31690-20b1-f984-7020-21c4f3d4154c bbc31690-20b1-f984-7020-21c4f3d4154c}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false 2026-04-22T10:09:58.878Z [INFO] server-25913ccd-91ec-3249-33ec-48237c4fccc6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25913ccd-91ec-3249-33ec-48237c4fccc6 Address:25913ccd-91ec-3249-33ec-48237c4fccc6} {Suffrage:Voter ID:server-e2c87d58-2999-e469-ea52-a1bda0318cd3 Address:e2c87d58-2999-e469-ea52-a1bda0318cd3} {Suffrage:Voter ID:server-cea24096-1ea1-0e49-3a31-ee1f16086f5c Address:cea24096-1ea1-0e49-3a31-ee1f16086f5c}]" 2026-04-22T10:09:58.878Z [INFO] server-25913ccd-91ec-3249-33ec-48237c4fccc6: entering follower state: follower="Node at 25913ccd-91ec-3249-33ec-48237c4fccc6 [Follower]" leader-address= leader-id= 2026-04-22T10:09:58.878Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25913ccd-91ec-3249-33ec-48237c4fccc6 Address:25913ccd-91ec-3249-33ec-48237c4fccc6} {Suffrage:Voter ID:server-e2c87d58-2999-e469-ea52-a1bda0318cd3 Address:e2c87d58-2999-e469-ea52-a1bda0318cd3} {Suffrage:Voter ID:server-cea24096-1ea1-0e49-3a31-ee1f16086f5c Address:cea24096-1ea1-0e49-3a31-ee1f16086f5c}]" 2026-04-22T10:09:58.878Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: entering follower state: follower="Node at e2c87d58-2999-e469-ea52-a1bda0318cd3 [Follower]" leader-address= leader-id= 2026-04-22T10:09:58.878Z [INFO] server-cea24096-1ea1-0e49-3a31-ee1f16086f5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25913ccd-91ec-3249-33ec-48237c4fccc6 Address:25913ccd-91ec-3249-33ec-48237c4fccc6} {Suffrage:Voter ID:server-e2c87d58-2999-e469-ea52-a1bda0318cd3 Address:e2c87d58-2999-e469-ea52-a1bda0318cd3} {Suffrage:Voter ID:server-cea24096-1ea1-0e49-3a31-ee1f16086f5c Address:cea24096-1ea1-0e49-3a31-ee1f16086f5c}]" 2026-04-22T10:09:58.878Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:09:58.878Z [INFO] server-cea24096-1ea1-0e49-3a31-ee1f16086f5c: entering follower state: follower="Node at cea24096-1ea1-0e49-3a31-ee1f16086f5c [Follower]" leader-address= leader-id= 2026-04-22T10:09:59.446Z [WARN] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:09:59.446Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: entering candidate state: node="Node at e2c87d58-2999-e469-ea52-a1bda0318cd3 [Candidate]" term=2 2026-04-22T10:09:59.446Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: election won: term=2 tally=2 2026-04-22T10:09:59.446Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: entering leader state: leader="Node at e2c87d58-2999-e469-ea52-a1bda0318cd3 [Leader]" 2026-04-22T10:09:59.446Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: added peer, starting replication: peer=server-25913ccd-91ec-3249-33ec-48237c4fccc6 2026-04-22T10:09:59.447Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: added peer, starting replication: peer=server-cea24096-1ea1-0e49-3a31-ee1f16086f5c 2026-04-22T10:09:59.447Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: pipelining replication: peer="{Voter server-25913ccd-91ec-3249-33ec-48237c4fccc6 25913ccd-91ec-3249-33ec-48237c4fccc6}" 2026-04-22T10:09:59.447Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: pipelining replication: peer="{Voter server-cea24096-1ea1-0e49-3a31-ee1f16086f5c cea24096-1ea1-0e49-3a31-ee1f16086f5c}" 2026-04-22T10:10:00.453Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:09:59.447037973 +0000 UTC m=+45.470834063 (1 nodes), 568.147711ms from start of poll, 569.081318ms from cluster start. Timeout at 2026-04-22 10:10:00.452046979 +0000 UTC m=+46.475843146, 1.005009083s after stability 2026-04-22T10:10:00.454Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: starting snapshot up to: index=12 2026-04-22T10:10:00.454Z [INFO] creating new snapshot: path=/tmp/raft1439203812/snapshots/2-12-1776852600454.tmp 2026-04-22T10:10:00.455Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: snapshot complete up to: index=12 2026-04-22T10:10:00.456Z [INFO] creating new snapshot: path=/tmp/raft1439203812/snapshots/2-23-1776852600456.tmp 2026-04-22T10:10:00.456Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: copied to local snapshot: bytes=71 2026-04-22T10:10:00.456Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: snapshot restore progress: id=2-23-1776852600456 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:00.456Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: removing all old logs from log store 2026-04-22T10:10:00.456Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: compacting logs: from=1 to=22 2026-04-22T10:10:00.456Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: restored user snapshot: index=23 2026-04-22T10:10:00.456Z [ERROR] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: failed to get log: index=22 error="log not found" 2026-04-22T10:10:00.456Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: aborting pipeline replication: peer="{Voter server-cea24096-1ea1-0e49-3a31-ee1f16086f5c cea24096-1ea1-0e49-3a31-ee1f16086f5c}" 2026-04-22T10:10:00.456Z [ERROR] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: failed to get log: index=22 error="log not found" 2026-04-22T10:10:00.456Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: aborting pipeline replication: peer="{Voter server-25913ccd-91ec-3249-33ec-48237c4fccc6 25913ccd-91ec-3249-33ec-48237c4fccc6}" 2026-04-22T10:10:00.463Z [ERROR] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: failed to get log: index=22 error="log not found" 2026-04-22T10:10:00.464Z [INFO] creating new snapshot: path=/tmp/raft807479831/snapshots/2-23-1776852600464.tmp 2026-04-22T10:10:00.464Z [INFO] server-25913ccd-91ec-3249-33ec-48237c4fccc6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:00.464Z [INFO] server-25913ccd-91ec-3249-33ec-48237c4fccc6: copied to local snapshot: bytes=71 2026-04-22T10:10:00.465Z [INFO] server-25913ccd-91ec-3249-33ec-48237c4fccc6: snapshot restore progress: id=2-23-1776852600464 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:00.465Z [INFO] server-25913ccd-91ec-3249-33ec-48237c4fccc6: removing all old logs from log store 2026-04-22T10:10:00.465Z [INFO] server-25913ccd-91ec-3249-33ec-48237c4fccc6: compacting logs: from=1 to=22 2026-04-22T10:10:00.465Z [INFO] server-25913ccd-91ec-3249-33ec-48237c4fccc6: Installed remote snapshot 2026-04-22T10:10:00.465Z [WARN] server-25913ccd-91ec-3249-33ec-48237c4fccc6: skipping application of old log: index=22 2026-04-22T10:10:00.465Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: pipelining replication: peer="{Voter server-25913ccd-91ec-3249-33ec-48237c4fccc6 25913ccd-91ec-3249-33ec-48237c4fccc6}" 2026-04-22T10:10:00.465Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:00.467Z [ERROR] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: failed to get log: index=22 error="log not found" 2026-04-22T10:10:00.467Z [INFO] creating new snapshot: path=/tmp/raft2693525025/snapshots/2-23-1776852600467.tmp 2026-04-22T10:10:00.468Z [INFO] server-cea24096-1ea1-0e49-3a31-ee1f16086f5c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:00.468Z [INFO] server-cea24096-1ea1-0e49-3a31-ee1f16086f5c: copied to local snapshot: bytes=71 2026-04-22T10:10:00.468Z [INFO] server-cea24096-1ea1-0e49-3a31-ee1f16086f5c: snapshot restore progress: id=2-23-1776852600467 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:00.468Z [INFO] server-cea24096-1ea1-0e49-3a31-ee1f16086f5c: removing all old logs from log store 2026-04-22T10:10:00.468Z [INFO] server-cea24096-1ea1-0e49-3a31-ee1f16086f5c: compacting logs: from=1 to=22 2026-04-22T10:10:00.468Z [INFO] server-cea24096-1ea1-0e49-3a31-ee1f16086f5c: Installed remote snapshot 2026-04-22T10:10:00.468Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: pipelining replication: peer="{Voter server-cea24096-1ea1-0e49-3a31-ee1f16086f5c cea24096-1ea1-0e49-3a31-ee1f16086f5c}" 2026-04-22T10:10:01.470Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:00.465330465 +0000 UTC m=+46.489126575 (2 nodes), 835ns from start of poll, 1.58737383s from cluster start. Timeout at 2026-04-22 10:10:01.47033791 +0000 UTC m=+47.494133993, 1.005007418s after stability 2026-04-22T10:10:01.481Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: aborting pipeline replication: peer="{Voter server-cea24096-1ea1-0e49-3a31-ee1f16086f5c cea24096-1ea1-0e49-3a31-ee1f16086f5c}" 2026-04-22T10:10:01.481Z [INFO] server-e2c87d58-2999-e469-ea52-a1bda0318cd3: aborting pipeline replication: peer="{Voter server-25913ccd-91ec-3249-33ec-48237c4fccc6 25913ccd-91ec-3249-33ec-48237c4fccc6}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true 2026-04-22T10:10:01.487Z [INFO] server-c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700 Address:c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700} {Suffrage:Voter ID:server-1b7504ce-4f7e-2638-6547-ddd88374b9f1 Address:1b7504ce-4f7e-2638-6547-ddd88374b9f1} {Suffrage:Voter ID:server-1e36660f-0e8c-db77-7038-b66ec1360c48 Address:1e36660f-0e8c-db77-7038-b66ec1360c48}]" 2026-04-22T10:10:01.487Z [INFO] server-c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700: entering follower state: follower="Node at c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700 [Follower]" leader-address= leader-id= 2026-04-22T10:10:01.487Z [INFO] server-1b7504ce-4f7e-2638-6547-ddd88374b9f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700 Address:c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700} {Suffrage:Voter ID:server-1b7504ce-4f7e-2638-6547-ddd88374b9f1 Address:1b7504ce-4f7e-2638-6547-ddd88374b9f1} {Suffrage:Voter ID:server-1e36660f-0e8c-db77-7038-b66ec1360c48 Address:1e36660f-0e8c-db77-7038-b66ec1360c48}]" 2026-04-22T10:10:01.487Z [INFO] server-1b7504ce-4f7e-2638-6547-ddd88374b9f1: entering follower state: follower="Node at 1b7504ce-4f7e-2638-6547-ddd88374b9f1 [Follower]" leader-address= leader-id= 2026-04-22T10:10:01.487Z [INFO] server-1e36660f-0e8c-db77-7038-b66ec1360c48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700 Address:c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700} {Suffrage:Voter ID:server-1b7504ce-4f7e-2638-6547-ddd88374b9f1 Address:1b7504ce-4f7e-2638-6547-ddd88374b9f1} {Suffrage:Voter ID:server-1e36660f-0e8c-db77-7038-b66ec1360c48 Address:1e36660f-0e8c-db77-7038-b66ec1360c48}]" 2026-04-22T10:10:01.487Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:01.487Z [INFO] server-1e36660f-0e8c-db77-7038-b66ec1360c48: entering follower state: follower="Node at 1e36660f-0e8c-db77-7038-b66ec1360c48 [Follower]" leader-address= leader-id= 2026-04-22T10:10:02.256Z [WARN] server-c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:02.256Z [INFO] server-c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700: entering candidate state: node="Node at c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700 [Candidate]" term=2 2026-04-22T10:10:02.256Z [INFO] server-c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700: election won: term=2 tally=2 2026-04-22T10:10:02.256Z [INFO] server-c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700: entering leader state: leader="Node at c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700 [Leader]" 2026-04-22T10:10:02.256Z [INFO] server-c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700: added peer, starting replication: peer=server-1b7504ce-4f7e-2638-6547-ddd88374b9f1 2026-04-22T10:10:02.256Z [INFO] server-c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700: added peer, starting replication: peer=server-1e36660f-0e8c-db77-7038-b66ec1360c48 2026-04-22T10:10:02.256Z [INFO] server-c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700: pipelining replication: peer="{Voter server-1b7504ce-4f7e-2638-6547-ddd88374b9f1 1b7504ce-4f7e-2638-6547-ddd88374b9f1}" 2026-04-22T10:10:02.257Z [INFO] server-c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700: pipelining replication: peer="{Voter server-1e36660f-0e8c-db77-7038-b66ec1360c48 1e36660f-0e8c-db77-7038-b66ec1360c48}" 2026-04-22T10:10:03.262Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:02.256898726 +0000 UTC m=+48.280694902 (1 nodes), 769.095536ms from start of poll, 769.396362ms from cluster start. Timeout at 2026-04-22 10:10:03.261900574 +0000 UTC m=+49.285696713, 1.005001811s after stability 2026-04-22T10:10:03.262Z [INFO] server-c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700: starting snapshot up to: index=12 2026-04-22T10:10:03.262Z [INFO] creating new snapshot: path=/tmp/raft1662794114/snapshots/2-12-1776852603262.tmp 2026-04-22T10:10:03.263Z [INFO] server-c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700: snapshot complete up to: index=12 2026-04-22T10:10:03.264Z [INFO] server-ba90a3a8-f980-75cd-6ce0-79e747342937: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba90a3a8-f980-75cd-6ce0-79e747342937 Address:ba90a3a8-f980-75cd-6ce0-79e747342937} {Suffrage:Voter ID:server-d305b1ab-13b7-318b-7a98-ec07f25053a5 Address:d305b1ab-13b7-318b-7a98-ec07f25053a5} {Suffrage:Voter ID:server-c21f2676-5a5a-d8e8-b600-a6a367b12daa Address:c21f2676-5a5a-d8e8-b600-a6a367b12daa}]" 2026-04-22T10:10:03.264Z [INFO] server-ba90a3a8-f980-75cd-6ce0-79e747342937: entering follower state: follower="Node at ba90a3a8-f980-75cd-6ce0-79e747342937 [Follower]" leader-address= leader-id= 2026-04-22T10:10:03.264Z [INFO] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba90a3a8-f980-75cd-6ce0-79e747342937 Address:ba90a3a8-f980-75cd-6ce0-79e747342937} {Suffrage:Voter ID:server-d305b1ab-13b7-318b-7a98-ec07f25053a5 Address:d305b1ab-13b7-318b-7a98-ec07f25053a5} {Suffrage:Voter ID:server-c21f2676-5a5a-d8e8-b600-a6a367b12daa Address:c21f2676-5a5a-d8e8-b600-a6a367b12daa}]" 2026-04-22T10:10:03.264Z [INFO] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: entering follower state: follower="Node at d305b1ab-13b7-318b-7a98-ec07f25053a5 [Follower]" leader-address= leader-id= 2026-04-22T10:10:03.264Z [INFO] server-c21f2676-5a5a-d8e8-b600-a6a367b12daa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba90a3a8-f980-75cd-6ce0-79e747342937 Address:ba90a3a8-f980-75cd-6ce0-79e747342937} {Suffrage:Voter ID:server-d305b1ab-13b7-318b-7a98-ec07f25053a5 Address:d305b1ab-13b7-318b-7a98-ec07f25053a5} {Suffrage:Voter ID:server-c21f2676-5a5a-d8e8-b600-a6a367b12daa Address:c21f2676-5a5a-d8e8-b600-a6a367b12daa}]" 2026-04-22T10:10:03.264Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:03.264Z [INFO] server-c21f2676-5a5a-d8e8-b600-a6a367b12daa: entering follower state: follower="Node at c21f2676-5a5a-d8e8-b600-a6a367b12daa [Follower]" leader-address= leader-id= 2026-04-22T10:10:03.943Z [WARN] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:03.943Z [INFO] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: entering candidate state: node="Node at d305b1ab-13b7-318b-7a98-ec07f25053a5 [Candidate]" term=2 2026-04-22T10:10:03.943Z [INFO] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: election won: term=2 tally=2 2026-04-22T10:10:03.943Z [INFO] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: entering leader state: leader="Node at d305b1ab-13b7-318b-7a98-ec07f25053a5 [Leader]" 2026-04-22T10:10:03.943Z [INFO] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: added peer, starting replication: peer=server-ba90a3a8-f980-75cd-6ce0-79e747342937 2026-04-22T10:10:03.943Z [INFO] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: added peer, starting replication: peer=server-c21f2676-5a5a-d8e8-b600-a6a367b12daa 2026-04-22T10:10:03.944Z [INFO] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: pipelining replication: peer="{Voter server-ba90a3a8-f980-75cd-6ce0-79e747342937 ba90a3a8-f980-75cd-6ce0-79e747342937}" 2026-04-22T10:10:03.944Z [INFO] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: pipelining replication: peer="{Voter server-c21f2676-5a5a-d8e8-b600-a6a367b12daa c21f2676-5a5a-d8e8-b600-a6a367b12daa}" 2026-04-22T10:10:04.949Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:03.943934524 +0000 UTC m=+49.967730613 (1 nodes), 679.160736ms from start of poll, 679.451875ms from cluster start. Timeout at 2026-04-22 10:10:04.948937667 +0000 UTC m=+50.972733749, 1.005003136s after stability 2026-04-22T10:10:04.949Z [INFO] creating new snapshot: path=/tmp/raft3173064347/snapshots/2-15-1776852604949.tmp 2026-04-22T10:10:04.949Z [INFO] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: copied to local snapshot: bytes=71 2026-04-22T10:10:04.949Z [INFO] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: snapshot restore progress: id=2-15-1776852604949 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:04.950Z [INFO] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: restored user snapshot: index=15 2026-04-22T10:10:04.950Z [ERROR] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: failed to get log: index=3 error="log not found" 2026-04-22T10:10:04.950Z [INFO] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: aborting pipeline replication: peer="{Voter server-ba90a3a8-f980-75cd-6ce0-79e747342937 ba90a3a8-f980-75cd-6ce0-79e747342937}" 2026-04-22T10:10:04.950Z [ERROR] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: failed to get log: index=3 error="log not found" 2026-04-22T10:10:04.950Z [INFO] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: aborting pipeline replication: peer="{Voter server-c21f2676-5a5a-d8e8-b600-a6a367b12daa c21f2676-5a5a-d8e8-b600-a6a367b12daa}" 2026-04-22T10:10:04.956Z [ERROR] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: failed to get log: index=3 error="log not found" 2026-04-22T10:10:04.956Z [INFO] creating new snapshot: path=/tmp/raft1966396183/snapshots/2-15-1776852604956.tmp 2026-04-22T10:10:04.956Z [INFO] server-ba90a3a8-f980-75cd-6ce0-79e747342937: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:04.956Z [INFO] server-ba90a3a8-f980-75cd-6ce0-79e747342937: copied to local snapshot: bytes=71 2026-04-22T10:10:04.957Z [INFO] server-ba90a3a8-f980-75cd-6ce0-79e747342937: snapshot restore progress: id=2-15-1776852604956 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:04.957Z [INFO] server-ba90a3a8-f980-75cd-6ce0-79e747342937: Installed remote snapshot 2026-04-22T10:10:04.957Z [INFO] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: pipelining replication: peer="{Voter server-ba90a3a8-f980-75cd-6ce0-79e747342937 ba90a3a8-f980-75cd-6ce0-79e747342937}" 2026-04-22T10:10:04.957Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:04.959Z [ERROR] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: failed to get log: index=3 error="log not found" 2026-04-22T10:10:04.960Z [INFO] creating new snapshot: path=/tmp/raft3599686553/snapshots/2-15-1776852604960.tmp 2026-04-22T10:10:04.960Z [INFO] server-c21f2676-5a5a-d8e8-b600-a6a367b12daa: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:04.960Z [INFO] server-c21f2676-5a5a-d8e8-b600-a6a367b12daa: copied to local snapshot: bytes=71 2026-04-22T10:10:04.960Z [INFO] server-c21f2676-5a5a-d8e8-b600-a6a367b12daa: snapshot restore progress: id=2-15-1776852604960 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:04.960Z [INFO] server-c21f2676-5a5a-d8e8-b600-a6a367b12daa: Installed remote snapshot 2026-04-22T10:10:04.960Z [INFO] server-d305b1ab-13b7-318b-7a98-ec07f25053a5: pipelining replication: peer="{Voter server-c21f2676-5a5a-d8e8-b600-a6a367b12daa c21f2676-5a5a-d8e8-b600-a6a367b12daa}" 2026-04-22T10:10:05.962Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:04.957250922 +0000 UTC m=+50.981047015 (2 nodes), 714ns from start of poll, 1.692768277s from cluster start. Timeout at 2026-04-22 10:10:05.962266763 +0000 UTC m=+51.986062896, 1.005015881s after stability 2026-04-22T10:10:05.974Z [INFO] server-c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700: aborting pipeline replication: peer="{Voter server-1e36660f-0e8c-db77-7038-b66ec1360c48 1e36660f-0e8c-db77-7038-b66ec1360c48}" 2026-04-22T10:10:05.974Z [INFO] server-c0f2d1a6-c5a1-dafe-63dc-6c4ae05d2700: aborting pipeline replication: peer="{Voter server-1b7504ce-4f7e-2638-6547-ddd88374b9f1 1b7504ce-4f7e-2638-6547-ddd88374b9f1}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true 2026-04-22T10:10:05.975Z [INFO] server-a74df70e-ac27-7834-7d95-a4925533c173: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a74df70e-ac27-7834-7d95-a4925533c173 Address:a74df70e-ac27-7834-7d95-a4925533c173} {Suffrage:Voter ID:server-a65a933e-c447-f688-a674-8b3eb17ad0fe Address:a65a933e-c447-f688-a674-8b3eb17ad0fe} {Suffrage:Voter ID:server-99044550-26ba-b3c3-83f7-0d4bb4fb805d Address:99044550-26ba-b3c3-83f7-0d4bb4fb805d}]" 2026-04-22T10:10:05.975Z [INFO] server-a74df70e-ac27-7834-7d95-a4925533c173: entering follower state: follower="Node at a74df70e-ac27-7834-7d95-a4925533c173 [Follower]" leader-address= leader-id= 2026-04-22T10:10:05.975Z [INFO] server-a65a933e-c447-f688-a674-8b3eb17ad0fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a74df70e-ac27-7834-7d95-a4925533c173 Address:a74df70e-ac27-7834-7d95-a4925533c173} {Suffrage:Voter ID:server-a65a933e-c447-f688-a674-8b3eb17ad0fe Address:a65a933e-c447-f688-a674-8b3eb17ad0fe} {Suffrage:Voter ID:server-99044550-26ba-b3c3-83f7-0d4bb4fb805d Address:99044550-26ba-b3c3-83f7-0d4bb4fb805d}]" 2026-04-22T10:10:05.975Z [INFO] server-a65a933e-c447-f688-a674-8b3eb17ad0fe: entering follower state: follower="Node at a65a933e-c447-f688-a674-8b3eb17ad0fe [Follower]" leader-address= leader-id= 2026-04-22T10:10:05.975Z [INFO] server-99044550-26ba-b3c3-83f7-0d4bb4fb805d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a74df70e-ac27-7834-7d95-a4925533c173 Address:a74df70e-ac27-7834-7d95-a4925533c173} {Suffrage:Voter ID:server-a65a933e-c447-f688-a674-8b3eb17ad0fe Address:a65a933e-c447-f688-a674-8b3eb17ad0fe} {Suffrage:Voter ID:server-99044550-26ba-b3c3-83f7-0d4bb4fb805d Address:99044550-26ba-b3c3-83f7-0d4bb4fb805d}]" 2026-04-22T10:10:05.975Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:05.976Z [INFO] server-99044550-26ba-b3c3-83f7-0d4bb4fb805d: entering follower state: follower="Node at 99044550-26ba-b3c3-83f7-0d4bb4fb805d [Follower]" leader-address= leader-id= 2026-04-22T10:10:06.641Z [WARN] server-a74df70e-ac27-7834-7d95-a4925533c173: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:06.641Z [INFO] server-a74df70e-ac27-7834-7d95-a4925533c173: entering candidate state: node="Node at a74df70e-ac27-7834-7d95-a4925533c173 [Candidate]" term=2 2026-04-22T10:10:06.641Z [INFO] server-a74df70e-ac27-7834-7d95-a4925533c173: election won: term=2 tally=2 2026-04-22T10:10:06.641Z [INFO] server-a74df70e-ac27-7834-7d95-a4925533c173: entering leader state: leader="Node at a74df70e-ac27-7834-7d95-a4925533c173 [Leader]" 2026-04-22T10:10:06.641Z [INFO] server-a74df70e-ac27-7834-7d95-a4925533c173: added peer, starting replication: peer=server-a65a933e-c447-f688-a674-8b3eb17ad0fe 2026-04-22T10:10:06.641Z [INFO] server-a74df70e-ac27-7834-7d95-a4925533c173: added peer, starting replication: peer=server-99044550-26ba-b3c3-83f7-0d4bb4fb805d 2026-04-22T10:10:06.641Z [INFO] server-a74df70e-ac27-7834-7d95-a4925533c173: pipelining replication: peer="{Voter server-99044550-26ba-b3c3-83f7-0d4bb4fb805d 99044550-26ba-b3c3-83f7-0d4bb4fb805d}" 2026-04-22T10:10:06.641Z [INFO] server-a74df70e-ac27-7834-7d95-a4925533c173: pipelining replication: peer="{Voter server-a65a933e-c447-f688-a674-8b3eb17ad0fe a65a933e-c447-f688-a674-8b3eb17ad0fe}" 2026-04-22T10:10:07.646Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:06.64180989 +0000 UTC m=+52.665605978 (1 nodes), 665.799391ms from start of poll, 666.107018ms from cluster start. Timeout at 2026-04-22 10:10:07.646815333 +0000 UTC m=+53.670611461, 1.005005483s after stability 2026-04-22T10:10:07.647Z [INFO] server-a74df70e-ac27-7834-7d95-a4925533c173: starting snapshot up to: index=12 2026-04-22T10:10:07.647Z [INFO] creating new snapshot: path=/tmp/raft3131355993/snapshots/2-12-1776852607647.tmp 2026-04-22T10:10:07.648Z [INFO] server-a74df70e-ac27-7834-7d95-a4925533c173: snapshot complete up to: index=12 2026-04-22T10:10:07.649Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a7d1dfc8-d842-46b0-38ae-67983b35e470 Address:a7d1dfc8-d842-46b0-38ae-67983b35e470} {Suffrage:Voter ID:server-1c09e58b-6cb6-b4fc-4c18-d04ed88a9716 Address:1c09e58b-6cb6-b4fc-4c18-d04ed88a9716} {Suffrage:Voter ID:server-d7661377-f98d-d220-70e1-1063ecec9952 Address:d7661377-f98d-d220-70e1-1063ecec9952}]" 2026-04-22T10:10:07.649Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: entering follower state: follower="Node at a7d1dfc8-d842-46b0-38ae-67983b35e470 [Follower]" leader-address= leader-id= 2026-04-22T10:10:07.650Z [INFO] server-1c09e58b-6cb6-b4fc-4c18-d04ed88a9716: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a7d1dfc8-d842-46b0-38ae-67983b35e470 Address:a7d1dfc8-d842-46b0-38ae-67983b35e470} {Suffrage:Voter ID:server-1c09e58b-6cb6-b4fc-4c18-d04ed88a9716 Address:1c09e58b-6cb6-b4fc-4c18-d04ed88a9716} {Suffrage:Voter ID:server-d7661377-f98d-d220-70e1-1063ecec9952 Address:d7661377-f98d-d220-70e1-1063ecec9952}]" 2026-04-22T10:10:07.650Z [INFO] server-1c09e58b-6cb6-b4fc-4c18-d04ed88a9716: entering follower state: follower="Node at 1c09e58b-6cb6-b4fc-4c18-d04ed88a9716 [Follower]" leader-address= leader-id= 2026-04-22T10:10:07.650Z [INFO] server-d7661377-f98d-d220-70e1-1063ecec9952: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a7d1dfc8-d842-46b0-38ae-67983b35e470 Address:a7d1dfc8-d842-46b0-38ae-67983b35e470} {Suffrage:Voter ID:server-1c09e58b-6cb6-b4fc-4c18-d04ed88a9716 Address:1c09e58b-6cb6-b4fc-4c18-d04ed88a9716} {Suffrage:Voter ID:server-d7661377-f98d-d220-70e1-1063ecec9952 Address:d7661377-f98d-d220-70e1-1063ecec9952}]" 2026-04-22T10:10:07.650Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:07.650Z [INFO] server-d7661377-f98d-d220-70e1-1063ecec9952: entering follower state: follower="Node at d7661377-f98d-d220-70e1-1063ecec9952 [Follower]" leader-address= leader-id= 2026-04-22T10:10:08.251Z [WARN] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:08.251Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: entering candidate state: node="Node at a7d1dfc8-d842-46b0-38ae-67983b35e470 [Candidate]" term=2 2026-04-22T10:10:08.251Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: election won: term=2 tally=2 2026-04-22T10:10:08.251Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: entering leader state: leader="Node at a7d1dfc8-d842-46b0-38ae-67983b35e470 [Leader]" 2026-04-22T10:10:08.251Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: added peer, starting replication: peer=server-1c09e58b-6cb6-b4fc-4c18-d04ed88a9716 2026-04-22T10:10:08.251Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: added peer, starting replication: peer=server-d7661377-f98d-d220-70e1-1063ecec9952 2026-04-22T10:10:08.251Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: pipelining replication: peer="{Voter server-d7661377-f98d-d220-70e1-1063ecec9952 d7661377-f98d-d220-70e1-1063ecec9952}" 2026-04-22T10:10:08.251Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: pipelining replication: peer="{Voter server-1c09e58b-6cb6-b4fc-4c18-d04ed88a9716 1c09e58b-6cb6-b4fc-4c18-d04ed88a9716}" 2026-04-22T10:10:09.256Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:08.251251381 +0000 UTC m=+54.275047530 (1 nodes), 600.784251ms from start of poll, 601.643602ms from cluster start. Timeout at 2026-04-22 10:10:09.25626573 +0000 UTC m=+55.280061863, 1.005014333s after stability 2026-04-22T10:10:09.256Z [INFO] creating new snapshot: path=/tmp/raft740047648/snapshots/2-15-1776852609256.tmp 2026-04-22T10:10:09.257Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: copied to local snapshot: bytes=71 2026-04-22T10:10:09.257Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: snapshot restore progress: id=2-15-1776852609256 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:09.257Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: removing all old logs from log store 2026-04-22T10:10:09.257Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: compacting logs: from=1 to=2 2026-04-22T10:10:09.257Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: restored user snapshot: index=15 2026-04-22T10:10:09.257Z [ERROR] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: failed to get log: index=2 error="log not found" 2026-04-22T10:10:09.257Z [ERROR] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: failed to get log: index=2 error="log not found" 2026-04-22T10:10:09.257Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: aborting pipeline replication: peer="{Voter server-1c09e58b-6cb6-b4fc-4c18-d04ed88a9716 1c09e58b-6cb6-b4fc-4c18-d04ed88a9716}" 2026-04-22T10:10:09.257Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: aborting pipeline replication: peer="{Voter server-d7661377-f98d-d220-70e1-1063ecec9952 d7661377-f98d-d220-70e1-1063ecec9952}" 2026-04-22T10:10:09.263Z [ERROR] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: failed to get log: index=2 error="log not found" 2026-04-22T10:10:09.264Z [INFO] creating new snapshot: path=/tmp/raft1970142833/snapshots/2-15-1776852609264.tmp 2026-04-22T10:10:09.264Z [INFO] server-1c09e58b-6cb6-b4fc-4c18-d04ed88a9716: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:09.264Z [INFO] server-1c09e58b-6cb6-b4fc-4c18-d04ed88a9716: copied to local snapshot: bytes=71 2026-04-22T10:10:09.264Z [INFO] server-1c09e58b-6cb6-b4fc-4c18-d04ed88a9716: snapshot restore progress: id=2-15-1776852609264 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:09.264Z [INFO] server-1c09e58b-6cb6-b4fc-4c18-d04ed88a9716: removing all old logs from log store 2026-04-22T10:10:09.264Z [INFO] server-1c09e58b-6cb6-b4fc-4c18-d04ed88a9716: compacting logs: from=1 to=2 2026-04-22T10:10:09.264Z [INFO] server-1c09e58b-6cb6-b4fc-4c18-d04ed88a9716: Installed remote snapshot 2026-04-22T10:10:09.264Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: pipelining replication: peer="{Voter server-1c09e58b-6cb6-b4fc-4c18-d04ed88a9716 1c09e58b-6cb6-b4fc-4c18-d04ed88a9716}" 2026-04-22T10:10:09.265Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:09.267Z [ERROR] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: failed to get log: index=2 error="log not found" 2026-04-22T10:10:09.267Z [INFO] creating new snapshot: path=/tmp/raft1819537510/snapshots/2-15-1776852609267.tmp 2026-04-22T10:10:09.267Z [INFO] server-d7661377-f98d-d220-70e1-1063ecec9952: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:09.268Z [INFO] server-d7661377-f98d-d220-70e1-1063ecec9952: copied to local snapshot: bytes=71 2026-04-22T10:10:09.268Z [INFO] server-d7661377-f98d-d220-70e1-1063ecec9952: snapshot restore progress: id=2-15-1776852609267 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:09.268Z [INFO] server-d7661377-f98d-d220-70e1-1063ecec9952: removing all old logs from log store 2026-04-22T10:10:09.268Z [INFO] server-d7661377-f98d-d220-70e1-1063ecec9952: compacting logs: from=1 to=2 2026-04-22T10:10:09.268Z [INFO] server-d7661377-f98d-d220-70e1-1063ecec9952: Installed remote snapshot 2026-04-22T10:10:09.268Z [INFO] server-a7d1dfc8-d842-46b0-38ae-67983b35e470: pipelining replication: peer="{Voter server-d7661377-f98d-d220-70e1-1063ecec9952 d7661377-f98d-d220-70e1-1063ecec9952}" 2026-04-22T10:10:10.271Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:09.265036493 +0000 UTC m=+55.288832604 (2 nodes), 772ns from start of poll, 1.615428676s from cluster start. Timeout at 2026-04-22 10:10:10.270055596 +0000 UTC m=+56.293851759, 1.005019155s after stability 2026-04-22T10:10:10.283Z [INFO] server-a74df70e-ac27-7834-7d95-a4925533c173: aborting pipeline replication: peer="{Voter server-99044550-26ba-b3c3-83f7-0d4bb4fb805d 99044550-26ba-b3c3-83f7-0d4bb4fb805d}" 2026-04-22T10:10:10.283Z [INFO] server-a74df70e-ac27-7834-7d95-a4925533c173: aborting pipeline replication: peer="{Voter server-a65a933e-c447-f688-a674-8b3eb17ad0fe a65a933e-c447-f688-a674-8b3eb17ad0fe}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false 2026-04-22T10:10:10.284Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6deb9e7b-cfc7-3256-b519-f4245a571e10 Address:6deb9e7b-cfc7-3256-b519-f4245a571e10} {Suffrage:Voter ID:server-9a2c530f-a845-e0a9-6b7c-390938e35697 Address:9a2c530f-a845-e0a9-6b7c-390938e35697} {Suffrage:Voter ID:server-7d14a275-45e0-08d8-9a64-3280fb0444e9 Address:7d14a275-45e0-08d8-9a64-3280fb0444e9}]" 2026-04-22T10:10:10.284Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: entering follower state: follower="Node at 6deb9e7b-cfc7-3256-b519-f4245a571e10 [Follower]" leader-address= leader-id= 2026-04-22T10:10:10.284Z [INFO] server-9a2c530f-a845-e0a9-6b7c-390938e35697: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6deb9e7b-cfc7-3256-b519-f4245a571e10 Address:6deb9e7b-cfc7-3256-b519-f4245a571e10} {Suffrage:Voter ID:server-9a2c530f-a845-e0a9-6b7c-390938e35697 Address:9a2c530f-a845-e0a9-6b7c-390938e35697} {Suffrage:Voter ID:server-7d14a275-45e0-08d8-9a64-3280fb0444e9 Address:7d14a275-45e0-08d8-9a64-3280fb0444e9}]" 2026-04-22T10:10:10.284Z [INFO] server-7d14a275-45e0-08d8-9a64-3280fb0444e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6deb9e7b-cfc7-3256-b519-f4245a571e10 Address:6deb9e7b-cfc7-3256-b519-f4245a571e10} {Suffrage:Voter ID:server-9a2c530f-a845-e0a9-6b7c-390938e35697 Address:9a2c530f-a845-e0a9-6b7c-390938e35697} {Suffrage:Voter ID:server-7d14a275-45e0-08d8-9a64-3280fb0444e9 Address:7d14a275-45e0-08d8-9a64-3280fb0444e9}]" 2026-04-22T10:10:10.284Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:10.284Z [INFO] server-9a2c530f-a845-e0a9-6b7c-390938e35697: entering follower state: follower="Node at 9a2c530f-a845-e0a9-6b7c-390938e35697 [Follower]" leader-address= leader-id= 2026-04-22T10:10:10.284Z [INFO] server-7d14a275-45e0-08d8-9a64-3280fb0444e9: entering follower state: follower="Node at 7d14a275-45e0-08d8-9a64-3280fb0444e9 [Follower]" leader-address= leader-id= 2026-04-22T10:10:10.876Z [WARN] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:10.876Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: entering candidate state: node="Node at 6deb9e7b-cfc7-3256-b519-f4245a571e10 [Candidate]" term=2 2026-04-22T10:10:10.876Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: election won: term=2 tally=2 2026-04-22T10:10:10.876Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: entering leader state: leader="Node at 6deb9e7b-cfc7-3256-b519-f4245a571e10 [Leader]" 2026-04-22T10:10:10.876Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: added peer, starting replication: peer=server-9a2c530f-a845-e0a9-6b7c-390938e35697 2026-04-22T10:10:10.876Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: added peer, starting replication: peer=server-7d14a275-45e0-08d8-9a64-3280fb0444e9 2026-04-22T10:10:10.876Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: pipelining replication: peer="{Voter server-9a2c530f-a845-e0a9-6b7c-390938e35697 9a2c530f-a845-e0a9-6b7c-390938e35697}" 2026-04-22T10:10:10.876Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: pipelining replication: peer="{Voter server-7d14a275-45e0-08d8-9a64-3280fb0444e9 7d14a275-45e0-08d8-9a64-3280fb0444e9}" 2026-04-22T10:10:11.882Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:10.876530042 +0000 UTC m=+56.900326133 (1 nodes), 592.14139ms from start of poll, 592.406649ms from cluster start. Timeout at 2026-04-22 10:10:11.881534196 +0000 UTC m=+57.905330278, 1.005004145s after stability 2026-04-22T10:10:11.882Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: starting snapshot up to: index=12 2026-04-22T10:10:11.882Z [INFO] creating new snapshot: path=/tmp/raft80188224/snapshots/2-12-1776852611882.tmp 2026-04-22T10:10:11.882Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: snapshot complete up to: index=12 2026-04-22T10:10:11.883Z [INFO] creating new snapshot: path=/tmp/raft80188224/snapshots/2-113-1776852611883.tmp 2026-04-22T10:10:11.883Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: copied to local snapshot: bytes=71 2026-04-22T10:10:11.884Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: snapshot restore progress: id=2-113-1776852611883 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:11.884Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: restored user snapshot: index=113 2026-04-22T10:10:11.884Z [ERROR] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: failed to get log: index=23 error="log not found" 2026-04-22T10:10:11.884Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: aborting pipeline replication: peer="{Voter server-7d14a275-45e0-08d8-9a64-3280fb0444e9 7d14a275-45e0-08d8-9a64-3280fb0444e9}" 2026-04-22T10:10:11.884Z [ERROR] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: failed to get log: index=23 error="log not found" 2026-04-22T10:10:11.884Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: aborting pipeline replication: peer="{Voter server-9a2c530f-a845-e0a9-6b7c-390938e35697 9a2c530f-a845-e0a9-6b7c-390938e35697}" 2026-04-22T10:10:11.891Z [ERROR] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: failed to get log: index=23 error="log not found" 2026-04-22T10:10:11.892Z [ERROR] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: failed to get log: index=23 error="log not found" 2026-04-22T10:10:11.892Z [INFO] creating new snapshot: path=/tmp/raft1946938039/snapshots/2-113-1776852611892.tmp 2026-04-22T10:10:11.892Z [INFO] server-9a2c530f-a845-e0a9-6b7c-390938e35697: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:11.892Z [INFO] creating new snapshot: path=/tmp/raft4196342661/snapshots/2-113-1776852611892.tmp 2026-04-22T10:10:11.892Z [INFO] server-7d14a275-45e0-08d8-9a64-3280fb0444e9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:11.892Z [INFO] server-9a2c530f-a845-e0a9-6b7c-390938e35697: copied to local snapshot: bytes=71 2026-04-22T10:10:11.893Z [INFO] server-7d14a275-45e0-08d8-9a64-3280fb0444e9: copied to local snapshot: bytes=71 2026-04-22T10:10:11.893Z [INFO] server-9a2c530f-a845-e0a9-6b7c-390938e35697: snapshot restore progress: id=2-113-1776852611892 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:11.893Z [INFO] server-7d14a275-45e0-08d8-9a64-3280fb0444e9: snapshot restore progress: id=2-113-1776852611892 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:11.893Z [INFO] server-9a2c530f-a845-e0a9-6b7c-390938e35697: Installed remote snapshot 2026-04-22T10:10:11.893Z [INFO] server-7d14a275-45e0-08d8-9a64-3280fb0444e9: Installed remote snapshot 2026-04-22T10:10:11.893Z [WARN] server-7d14a275-45e0-08d8-9a64-3280fb0444e9: skipping application of old log: index=22 2026-04-22T10:10:11.893Z [WARN] server-9a2c530f-a845-e0a9-6b7c-390938e35697: skipping application of old log: index=113 2026-04-22T10:10:11.893Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: pipelining replication: peer="{Voter server-7d14a275-45e0-08d8-9a64-3280fb0444e9 7d14a275-45e0-08d8-9a64-3280fb0444e9}" 2026-04-22T10:10:11.893Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: pipelining replication: peer="{Voter server-9a2c530f-a845-e0a9-6b7c-390938e35697 9a2c530f-a845-e0a9-6b7c-390938e35697}" 2026-04-22T10:10:11.893Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:12.899Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:11.893367689 +0000 UTC m=+57.917163789 (2 nodes), 661ns from start of poll, 1.609244305s from cluster start. Timeout at 2026-04-22 10:10:12.898377136 +0000 UTC m=+58.922173326, 1.005009537s after stability 2026-04-22T10:10:12.910Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: aborting pipeline replication: peer="{Voter server-7d14a275-45e0-08d8-9a64-3280fb0444e9 7d14a275-45e0-08d8-9a64-3280fb0444e9}" 2026-04-22T10:10:12.910Z [INFO] server-6deb9e7b-cfc7-3256-b519-f4245a571e10: aborting pipeline replication: peer="{Voter server-9a2c530f-a845-e0a9-6b7c-390938e35697 9a2c530f-a845-e0a9-6b7c-390938e35697}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false 2026-04-22T10:10:12.911Z [INFO] server-e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe Address:e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe} {Suffrage:Voter ID:server-a95198fd-9b56-6a10-b4f3-9921c1a451f7 Address:a95198fd-9b56-6a10-b4f3-9921c1a451f7} {Suffrage:Voter ID:server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba Address:6fdfb9b0-1513-6fff-c565-d2c7ffc713ba}]" 2026-04-22T10:10:12.911Z [INFO] server-e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe: entering follower state: follower="Node at e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe [Follower]" leader-address= leader-id= 2026-04-22T10:10:12.912Z [INFO] server-a95198fd-9b56-6a10-b4f3-9921c1a451f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe Address:e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe} {Suffrage:Voter ID:server-a95198fd-9b56-6a10-b4f3-9921c1a451f7 Address:a95198fd-9b56-6a10-b4f3-9921c1a451f7} {Suffrage:Voter ID:server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba Address:6fdfb9b0-1513-6fff-c565-d2c7ffc713ba}]" 2026-04-22T10:10:12.912Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe Address:e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe} {Suffrage:Voter ID:server-a95198fd-9b56-6a10-b4f3-9921c1a451f7 Address:a95198fd-9b56-6a10-b4f3-9921c1a451f7} {Suffrage:Voter ID:server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba Address:6fdfb9b0-1513-6fff-c565-d2c7ffc713ba}]" 2026-04-22T10:10:12.912Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:12.912Z [INFO] server-a95198fd-9b56-6a10-b4f3-9921c1a451f7: entering follower state: follower="Node at a95198fd-9b56-6a10-b4f3-9921c1a451f7 [Follower]" leader-address= leader-id= 2026-04-22T10:10:12.912Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: entering follower state: follower="Node at 6fdfb9b0-1513-6fff-c565-d2c7ffc713ba [Follower]" leader-address= leader-id= 2026-04-22T10:10:13.452Z [WARN] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:13.452Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: entering candidate state: node="Node at 6fdfb9b0-1513-6fff-c565-d2c7ffc713ba [Candidate]" term=2 2026-04-22T10:10:13.452Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: election won: term=2 tally=2 2026-04-22T10:10:13.452Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: entering leader state: leader="Node at 6fdfb9b0-1513-6fff-c565-d2c7ffc713ba [Leader]" 2026-04-22T10:10:13.452Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: added peer, starting replication: peer=server-e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe 2026-04-22T10:10:13.452Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: added peer, starting replication: peer=server-a95198fd-9b56-6a10-b4f3-9921c1a451f7 2026-04-22T10:10:13.452Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: pipelining replication: peer="{Voter server-e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe}" 2026-04-22T10:10:13.452Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: pipelining replication: peer="{Voter server-a95198fd-9b56-6a10-b4f3-9921c1a451f7 a95198fd-9b56-6a10-b4f3-9921c1a451f7}" 2026-04-22T10:10:14.459Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:13.452845554 +0000 UTC m=+59.476641645 (1 nodes), 540.724399ms from start of poll, 540.992188ms from cluster start. Timeout at 2026-04-22 10:10:14.457853617 +0000 UTC m=+60.481649754, 1.005008109s after stability 2026-04-22T10:10:14.460Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: starting snapshot up to: index=12 2026-04-22T10:10:14.460Z [INFO] creating new snapshot: path=/tmp/raft343419281/snapshots/2-12-1776852614460.tmp 2026-04-22T10:10:14.460Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: snapshot complete up to: index=12 2026-04-22T10:10:14.461Z [INFO] creating new snapshot: path=/tmp/raft343419281/snapshots/2-113-1776852614461.tmp 2026-04-22T10:10:14.461Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: copied to local snapshot: bytes=71 2026-04-22T10:10:14.462Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: snapshot restore progress: id=2-113-1776852614461 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:14.462Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: removing all old logs from log store 2026-04-22T10:10:14.462Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: compacting logs: from=1 to=22 2026-04-22T10:10:14.462Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: restored user snapshot: index=113 2026-04-22T10:10:14.462Z [ERROR] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: failed to get log: index=22 error="log not found" 2026-04-22T10:10:14.462Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: aborting pipeline replication: peer="{Voter server-e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe}" 2026-04-22T10:10:14.462Z [ERROR] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: failed to get log: index=22 error="log not found" 2026-04-22T10:10:14.462Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: aborting pipeline replication: peer="{Voter server-a95198fd-9b56-6a10-b4f3-9921c1a451f7 a95198fd-9b56-6a10-b4f3-9921c1a451f7}" 2026-04-22T10:10:14.468Z [ERROR] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: failed to get log: index=22 error="log not found" 2026-04-22T10:10:14.469Z [INFO] creating new snapshot: path=/tmp/raft2687155162/snapshots/2-113-1776852614469.tmp 2026-04-22T10:10:14.469Z [INFO] server-e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:14.469Z [INFO] server-e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe: copied to local snapshot: bytes=71 2026-04-22T10:10:14.469Z [INFO] server-e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe: snapshot restore progress: id=2-113-1776852614469 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:14.469Z [INFO] server-e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe: removing all old logs from log store 2026-04-22T10:10:14.469Z [INFO] server-e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe: compacting logs: from=1 to=22 2026-04-22T10:10:14.470Z [INFO] server-e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe: Installed remote snapshot 2026-04-22T10:10:14.470Z [WARN] server-e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe: skipping application of old log: index=22 2026-04-22T10:10:14.470Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: pipelining replication: peer="{Voter server-e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe}" 2026-04-22T10:10:14.470Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:14.471Z [ERROR] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: failed to get log: index=22 error="log not found" 2026-04-22T10:10:14.471Z [INFO] creating new snapshot: path=/tmp/raft758770256/snapshots/2-113-1776852614471.tmp 2026-04-22T10:10:14.471Z [INFO] server-a95198fd-9b56-6a10-b4f3-9921c1a451f7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:14.472Z [INFO] server-a95198fd-9b56-6a10-b4f3-9921c1a451f7: copied to local snapshot: bytes=71 2026-04-22T10:10:14.472Z [INFO] server-a95198fd-9b56-6a10-b4f3-9921c1a451f7: snapshot restore progress: id=2-113-1776852614471 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:14.472Z [INFO] server-a95198fd-9b56-6a10-b4f3-9921c1a451f7: removing all old logs from log store 2026-04-22T10:10:14.472Z [INFO] server-a95198fd-9b56-6a10-b4f3-9921c1a451f7: compacting logs: from=1 to=22 2026-04-22T10:10:14.472Z [INFO] server-a95198fd-9b56-6a10-b4f3-9921c1a451f7: Installed remote snapshot 2026-04-22T10:10:14.472Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: pipelining replication: peer="{Voter server-a95198fd-9b56-6a10-b4f3-9921c1a451f7 a95198fd-9b56-6a10-b4f3-9921c1a451f7}" 2026-04-22T10:10:15.476Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:14.470201545 +0000 UTC m=+60.493997646 (2 nodes), 834ns from start of poll, 1.558348189s from cluster start. Timeout at 2026-04-22 10:10:15.475221414 +0000 UTC m=+61.499017551, 1.005019905s after stability 2026-04-22T10:10:15.488Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: aborting pipeline replication: peer="{Voter server-a95198fd-9b56-6a10-b4f3-9921c1a451f7 a95198fd-9b56-6a10-b4f3-9921c1a451f7}" 2026-04-22T10:10:15.488Z [INFO] server-6fdfb9b0-1513-6fff-c565-d2c7ffc713ba: aborting pipeline replication: peer="{Voter server-e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe e13c70ee-7e16-bab4-8eef-e4b69e0fb9fe}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true 2026-04-22T10:10:15.489Z [INFO] server-a7da2b12-aa24-9080-774f-30aae561cb18: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a7da2b12-aa24-9080-774f-30aae561cb18 Address:a7da2b12-aa24-9080-774f-30aae561cb18} {Suffrage:Voter ID:server-3ce9b743-7fde-59be-af7f-b9908fef017e Address:3ce9b743-7fde-59be-af7f-b9908fef017e} {Suffrage:Voter ID:server-194a846a-8e2c-b051-91e7-5b82addb1490 Address:194a846a-8e2c-b051-91e7-5b82addb1490}]" 2026-04-22T10:10:15.489Z [INFO] server-a7da2b12-aa24-9080-774f-30aae561cb18: entering follower state: follower="Node at a7da2b12-aa24-9080-774f-30aae561cb18 [Follower]" leader-address= leader-id= 2026-04-22T10:10:15.489Z [INFO] server-3ce9b743-7fde-59be-af7f-b9908fef017e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a7da2b12-aa24-9080-774f-30aae561cb18 Address:a7da2b12-aa24-9080-774f-30aae561cb18} {Suffrage:Voter ID:server-3ce9b743-7fde-59be-af7f-b9908fef017e Address:3ce9b743-7fde-59be-af7f-b9908fef017e} {Suffrage:Voter ID:server-194a846a-8e2c-b051-91e7-5b82addb1490 Address:194a846a-8e2c-b051-91e7-5b82addb1490}]" 2026-04-22T10:10:15.489Z [INFO] server-3ce9b743-7fde-59be-af7f-b9908fef017e: entering follower state: follower="Node at 3ce9b743-7fde-59be-af7f-b9908fef017e [Follower]" leader-address= leader-id= 2026-04-22T10:10:15.489Z [INFO] server-194a846a-8e2c-b051-91e7-5b82addb1490: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a7da2b12-aa24-9080-774f-30aae561cb18 Address:a7da2b12-aa24-9080-774f-30aae561cb18} {Suffrage:Voter ID:server-3ce9b743-7fde-59be-af7f-b9908fef017e Address:3ce9b743-7fde-59be-af7f-b9908fef017e} {Suffrage:Voter ID:server-194a846a-8e2c-b051-91e7-5b82addb1490 Address:194a846a-8e2c-b051-91e7-5b82addb1490}]" 2026-04-22T10:10:15.489Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:15.489Z [INFO] server-194a846a-8e2c-b051-91e7-5b82addb1490: entering follower state: follower="Node at 194a846a-8e2c-b051-91e7-5b82addb1490 [Follower]" leader-address= leader-id= 2026-04-22T10:10:16.128Z [WARN] server-3ce9b743-7fde-59be-af7f-b9908fef017e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:16.128Z [INFO] server-3ce9b743-7fde-59be-af7f-b9908fef017e: entering candidate state: node="Node at 3ce9b743-7fde-59be-af7f-b9908fef017e [Candidate]" term=2 2026-04-22T10:10:16.128Z [INFO] server-3ce9b743-7fde-59be-af7f-b9908fef017e: election won: term=2 tally=2 2026-04-22T10:10:16.128Z [INFO] server-3ce9b743-7fde-59be-af7f-b9908fef017e: entering leader state: leader="Node at 3ce9b743-7fde-59be-af7f-b9908fef017e [Leader]" 2026-04-22T10:10:16.128Z [INFO] server-3ce9b743-7fde-59be-af7f-b9908fef017e: added peer, starting replication: peer=server-a7da2b12-aa24-9080-774f-30aae561cb18 2026-04-22T10:10:16.128Z [INFO] server-3ce9b743-7fde-59be-af7f-b9908fef017e: added peer, starting replication: peer=server-194a846a-8e2c-b051-91e7-5b82addb1490 2026-04-22T10:10:16.128Z [INFO] server-3ce9b743-7fde-59be-af7f-b9908fef017e: pipelining replication: peer="{Voter server-194a846a-8e2c-b051-91e7-5b82addb1490 194a846a-8e2c-b051-91e7-5b82addb1490}" 2026-04-22T10:10:16.128Z [INFO] server-3ce9b743-7fde-59be-af7f-b9908fef017e: pipelining replication: peer="{Voter server-a7da2b12-aa24-9080-774f-30aae561cb18 a7da2b12-aa24-9080-774f-30aae561cb18}" 2026-04-22T10:10:17.134Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:16.128742814 +0000 UTC m=+62.152538925 (1 nodes), 638.889887ms from start of poll, 639.278986ms from cluster start. Timeout at 2026-04-22 10:10:17.133750335 +0000 UTC m=+63.157546475, 1.00500755s after stability 2026-04-22T10:10:17.135Z [INFO] server-3ce9b743-7fde-59be-af7f-b9908fef017e: starting snapshot up to: index=12 2026-04-22T10:10:17.136Z [INFO] creating new snapshot: path=/tmp/raft1231272908/snapshots/2-12-1776852617135.tmp 2026-04-22T10:10:17.136Z [INFO] server-3ce9b743-7fde-59be-af7f-b9908fef017e: snapshot complete up to: index=12 2026-04-22T10:10:17.137Z [INFO] server-1416f794-abdc-e710-3c71-735b000aae94: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1416f794-abdc-e710-3c71-735b000aae94 Address:1416f794-abdc-e710-3c71-735b000aae94} {Suffrage:Voter ID:server-112470c4-e61e-b9ab-221f-b74693553f4d Address:112470c4-e61e-b9ab-221f-b74693553f4d} {Suffrage:Voter ID:server-da057de3-7d9d-9080-7b5a-c692af39c8c9 Address:da057de3-7d9d-9080-7b5a-c692af39c8c9}]" 2026-04-22T10:10:17.137Z [INFO] server-1416f794-abdc-e710-3c71-735b000aae94: entering follower state: follower="Node at 1416f794-abdc-e710-3c71-735b000aae94 [Follower]" leader-address= leader-id= 2026-04-22T10:10:17.137Z [INFO] server-112470c4-e61e-b9ab-221f-b74693553f4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1416f794-abdc-e710-3c71-735b000aae94 Address:1416f794-abdc-e710-3c71-735b000aae94} {Suffrage:Voter ID:server-112470c4-e61e-b9ab-221f-b74693553f4d Address:112470c4-e61e-b9ab-221f-b74693553f4d} {Suffrage:Voter ID:server-da057de3-7d9d-9080-7b5a-c692af39c8c9 Address:da057de3-7d9d-9080-7b5a-c692af39c8c9}]" 2026-04-22T10:10:17.137Z [INFO] server-112470c4-e61e-b9ab-221f-b74693553f4d: entering follower state: follower="Node at 112470c4-e61e-b9ab-221f-b74693553f4d [Follower]" leader-address= leader-id= 2026-04-22T10:10:17.137Z [INFO] server-da057de3-7d9d-9080-7b5a-c692af39c8c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1416f794-abdc-e710-3c71-735b000aae94 Address:1416f794-abdc-e710-3c71-735b000aae94} {Suffrage:Voter ID:server-112470c4-e61e-b9ab-221f-b74693553f4d Address:112470c4-e61e-b9ab-221f-b74693553f4d} {Suffrage:Voter ID:server-da057de3-7d9d-9080-7b5a-c692af39c8c9 Address:da057de3-7d9d-9080-7b5a-c692af39c8c9}]" 2026-04-22T10:10:17.137Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:17.137Z [INFO] server-da057de3-7d9d-9080-7b5a-c692af39c8c9: entering follower state: follower="Node at da057de3-7d9d-9080-7b5a-c692af39c8c9 [Follower]" leader-address= leader-id= 2026-04-22T10:10:17.676Z [WARN] server-112470c4-e61e-b9ab-221f-b74693553f4d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:17.676Z [INFO] server-112470c4-e61e-b9ab-221f-b74693553f4d: entering candidate state: node="Node at 112470c4-e61e-b9ab-221f-b74693553f4d [Candidate]" term=2 2026-04-22T10:10:17.676Z [INFO] server-112470c4-e61e-b9ab-221f-b74693553f4d: election won: term=2 tally=2 2026-04-22T10:10:17.677Z [INFO] server-112470c4-e61e-b9ab-221f-b74693553f4d: entering leader state: leader="Node at 112470c4-e61e-b9ab-221f-b74693553f4d [Leader]" 2026-04-22T10:10:17.677Z [INFO] server-112470c4-e61e-b9ab-221f-b74693553f4d: added peer, starting replication: peer=server-1416f794-abdc-e710-3c71-735b000aae94 2026-04-22T10:10:17.677Z [INFO] server-112470c4-e61e-b9ab-221f-b74693553f4d: added peer, starting replication: peer=server-da057de3-7d9d-9080-7b5a-c692af39c8c9 2026-04-22T10:10:17.677Z [INFO] server-112470c4-e61e-b9ab-221f-b74693553f4d: pipelining replication: peer="{Voter server-da057de3-7d9d-9080-7b5a-c692af39c8c9 da057de3-7d9d-9080-7b5a-c692af39c8c9}" 2026-04-22T10:10:17.677Z [INFO] server-112470c4-e61e-b9ab-221f-b74693553f4d: pipelining replication: peer="{Voter server-1416f794-abdc-e710-3c71-735b000aae94 1416f794-abdc-e710-3c71-735b000aae94}" 2026-04-22T10:10:18.682Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:17.677133496 +0000 UTC m=+63.700929583 (1 nodes), 539.459903ms from start of poll, 539.743435ms from cluster start. Timeout at 2026-04-22 10:10:18.682137928 +0000 UTC m=+64.705934011, 1.005004428s after stability 2026-04-22T10:10:18.684Z [INFO] creating new snapshot: path=/tmp/raft4109923467/snapshots/2-113-1776852618684.tmp 2026-04-22T10:10:18.685Z [INFO] server-112470c4-e61e-b9ab-221f-b74693553f4d: copied to local snapshot: bytes=71 2026-04-22T10:10:18.685Z [INFO] server-112470c4-e61e-b9ab-221f-b74693553f4d: snapshot restore progress: id=2-113-1776852618684 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:18.685Z [INFO] server-112470c4-e61e-b9ab-221f-b74693553f4d: restored user snapshot: index=113 2026-04-22T10:10:18.685Z [ERROR] server-112470c4-e61e-b9ab-221f-b74693553f4d: failed to get log: index=3 error="log not found" 2026-04-22T10:10:18.685Z [ERROR] server-112470c4-e61e-b9ab-221f-b74693553f4d: failed to get log: index=3 error="log not found" 2026-04-22T10:10:18.685Z [INFO] server-112470c4-e61e-b9ab-221f-b74693553f4d: aborting pipeline replication: peer="{Voter server-1416f794-abdc-e710-3c71-735b000aae94 1416f794-abdc-e710-3c71-735b000aae94}" 2026-04-22T10:10:18.685Z [INFO] server-112470c4-e61e-b9ab-221f-b74693553f4d: aborting pipeline replication: peer="{Voter server-da057de3-7d9d-9080-7b5a-c692af39c8c9 da057de3-7d9d-9080-7b5a-c692af39c8c9}" 2026-04-22T10:10:18.691Z [ERROR] server-112470c4-e61e-b9ab-221f-b74693553f4d: failed to get log: index=3 error="log not found" 2026-04-22T10:10:18.692Z [INFO] creating new snapshot: path=/tmp/raft2695396732/snapshots/2-113-1776852618692.tmp 2026-04-22T10:10:18.693Z [INFO] server-1416f794-abdc-e710-3c71-735b000aae94: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:18.693Z [INFO] server-1416f794-abdc-e710-3c71-735b000aae94: copied to local snapshot: bytes=71 2026-04-22T10:10:18.694Z [INFO] server-1416f794-abdc-e710-3c71-735b000aae94: snapshot restore progress: id=2-113-1776852618692 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:18.694Z [INFO] server-1416f794-abdc-e710-3c71-735b000aae94: Installed remote snapshot 2026-04-22T10:10:18.694Z [INFO] server-112470c4-e61e-b9ab-221f-b74693553f4d: pipelining replication: peer="{Voter server-1416f794-abdc-e710-3c71-735b000aae94 1416f794-abdc-e710-3c71-735b000aae94}" 2026-04-22T10:10:18.694Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:18.694Z [ERROR] server-112470c4-e61e-b9ab-221f-b74693553f4d: failed to get log: index=3 error="log not found" 2026-04-22T10:10:18.694Z [INFO] creating new snapshot: path=/tmp/raft3514705329/snapshots/2-113-1776852618694.tmp 2026-04-22T10:10:18.694Z [INFO] server-da057de3-7d9d-9080-7b5a-c692af39c8c9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:18.695Z [INFO] server-da057de3-7d9d-9080-7b5a-c692af39c8c9: copied to local snapshot: bytes=71 2026-04-22T10:10:18.695Z [INFO] server-da057de3-7d9d-9080-7b5a-c692af39c8c9: snapshot restore progress: id=2-113-1776852618694 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:18.695Z [INFO] server-da057de3-7d9d-9080-7b5a-c692af39c8c9: Installed remote snapshot 2026-04-22T10:10:18.695Z [INFO] server-112470c4-e61e-b9ab-221f-b74693553f4d: pipelining replication: peer="{Voter server-da057de3-7d9d-9080-7b5a-c692af39c8c9 da057de3-7d9d-9080-7b5a-c692af39c8c9}" 2026-04-22T10:10:19.699Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:18.694238633 +0000 UTC m=+64.718034726 (2 nodes), 616ns from start of poll, 1.556848578s from cluster start. Timeout at 2026-04-22 10:10:19.69924831 +0000 UTC m=+65.723044392, 1.005009666s after stability 2026-04-22T10:10:19.710Z [INFO] server-3ce9b743-7fde-59be-af7f-b9908fef017e: aborting pipeline replication: peer="{Voter server-194a846a-8e2c-b051-91e7-5b82addb1490 194a846a-8e2c-b051-91e7-5b82addb1490}" 2026-04-22T10:10:19.710Z [INFO] server-3ce9b743-7fde-59be-af7f-b9908fef017e: aborting pipeline replication: peer="{Voter server-a7da2b12-aa24-9080-774f-30aae561cb18 a7da2b12-aa24-9080-774f-30aae561cb18}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true 2026-04-22T10:10:19.711Z [INFO] server-07973ad8-9170-e79e-dcf0-990c1e550bc3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07973ad8-9170-e79e-dcf0-990c1e550bc3 Address:07973ad8-9170-e79e-dcf0-990c1e550bc3} {Suffrage:Voter ID:server-76a82980-c888-4e8c-0cb6-2966d429191a Address:76a82980-c888-4e8c-0cb6-2966d429191a} {Suffrage:Voter ID:server-2309cd97-aa43-957c-864f-7bff16a24e84 Address:2309cd97-aa43-957c-864f-7bff16a24e84}]" 2026-04-22T10:10:19.711Z [INFO] server-07973ad8-9170-e79e-dcf0-990c1e550bc3: entering follower state: follower="Node at 07973ad8-9170-e79e-dcf0-990c1e550bc3 [Follower]" leader-address= leader-id= 2026-04-22T10:10:19.711Z [INFO] server-76a82980-c888-4e8c-0cb6-2966d429191a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07973ad8-9170-e79e-dcf0-990c1e550bc3 Address:07973ad8-9170-e79e-dcf0-990c1e550bc3} {Suffrage:Voter ID:server-76a82980-c888-4e8c-0cb6-2966d429191a Address:76a82980-c888-4e8c-0cb6-2966d429191a} {Suffrage:Voter ID:server-2309cd97-aa43-957c-864f-7bff16a24e84 Address:2309cd97-aa43-957c-864f-7bff16a24e84}]" 2026-04-22T10:10:19.711Z [INFO] server-76a82980-c888-4e8c-0cb6-2966d429191a: entering follower state: follower="Node at 76a82980-c888-4e8c-0cb6-2966d429191a [Follower]" leader-address= leader-id= 2026-04-22T10:10:19.711Z [INFO] server-2309cd97-aa43-957c-864f-7bff16a24e84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07973ad8-9170-e79e-dcf0-990c1e550bc3 Address:07973ad8-9170-e79e-dcf0-990c1e550bc3} {Suffrage:Voter ID:server-76a82980-c888-4e8c-0cb6-2966d429191a Address:76a82980-c888-4e8c-0cb6-2966d429191a} {Suffrage:Voter ID:server-2309cd97-aa43-957c-864f-7bff16a24e84 Address:2309cd97-aa43-957c-864f-7bff16a24e84}]" 2026-04-22T10:10:19.711Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:19.711Z [INFO] server-2309cd97-aa43-957c-864f-7bff16a24e84: entering follower state: follower="Node at 2309cd97-aa43-957c-864f-7bff16a24e84 [Follower]" leader-address= leader-id= 2026-04-22T10:10:20.230Z [WARN] server-2309cd97-aa43-957c-864f-7bff16a24e84: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:20.230Z [INFO] server-2309cd97-aa43-957c-864f-7bff16a24e84: entering candidate state: node="Node at 2309cd97-aa43-957c-864f-7bff16a24e84 [Candidate]" term=2 2026-04-22T10:10:20.230Z [INFO] server-2309cd97-aa43-957c-864f-7bff16a24e84: election won: term=2 tally=2 2026-04-22T10:10:20.230Z [INFO] server-2309cd97-aa43-957c-864f-7bff16a24e84: entering leader state: leader="Node at 2309cd97-aa43-957c-864f-7bff16a24e84 [Leader]" 2026-04-22T10:10:20.230Z [INFO] server-2309cd97-aa43-957c-864f-7bff16a24e84: added peer, starting replication: peer=server-07973ad8-9170-e79e-dcf0-990c1e550bc3 2026-04-22T10:10:20.230Z [INFO] server-2309cd97-aa43-957c-864f-7bff16a24e84: added peer, starting replication: peer=server-76a82980-c888-4e8c-0cb6-2966d429191a 2026-04-22T10:10:20.230Z [INFO] server-2309cd97-aa43-957c-864f-7bff16a24e84: pipelining replication: peer="{Voter server-76a82980-c888-4e8c-0cb6-2966d429191a 76a82980-c888-4e8c-0cb6-2966d429191a}" 2026-04-22T10:10:20.230Z [INFO] server-2309cd97-aa43-957c-864f-7bff16a24e84: pipelining replication: peer="{Voter server-07973ad8-9170-e79e-dcf0-990c1e550bc3 07973ad8-9170-e79e-dcf0-990c1e550bc3}" 2026-04-22T10:10:21.235Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:20.230355809 +0000 UTC m=+66.254151906 (1 nodes), 518.406684ms from start of poll, 518.759413ms from cluster start. Timeout at 2026-04-22 10:10:21.235446651 +0000 UTC m=+67.259242835, 1.005090929s after stability 2026-04-22T10:10:21.237Z [INFO] server-2309cd97-aa43-957c-864f-7bff16a24e84: starting snapshot up to: index=12 2026-04-22T10:10:21.237Z [INFO] creating new snapshot: path=/tmp/raft3482561811/snapshots/2-12-1776852621237.tmp 2026-04-22T10:10:21.237Z [INFO] server-2309cd97-aa43-957c-864f-7bff16a24e84: snapshot complete up to: index=12 2026-04-22T10:10:21.238Z [INFO] server-e61722ad-1c51-ff91-373d-9654c034430e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e61722ad-1c51-ff91-373d-9654c034430e Address:e61722ad-1c51-ff91-373d-9654c034430e} {Suffrage:Voter ID:server-a468eb11-0a8c-79e7-346d-5159613d0dbe Address:a468eb11-0a8c-79e7-346d-5159613d0dbe} {Suffrage:Voter ID:server-77538438-2d54-053a-ca33-1e628c71a620 Address:77538438-2d54-053a-ca33-1e628c71a620}]" 2026-04-22T10:10:21.238Z [INFO] server-e61722ad-1c51-ff91-373d-9654c034430e: entering follower state: follower="Node at e61722ad-1c51-ff91-373d-9654c034430e [Follower]" leader-address= leader-id= 2026-04-22T10:10:21.238Z [INFO] server-a468eb11-0a8c-79e7-346d-5159613d0dbe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e61722ad-1c51-ff91-373d-9654c034430e Address:e61722ad-1c51-ff91-373d-9654c034430e} {Suffrage:Voter ID:server-a468eb11-0a8c-79e7-346d-5159613d0dbe Address:a468eb11-0a8c-79e7-346d-5159613d0dbe} {Suffrage:Voter ID:server-77538438-2d54-053a-ca33-1e628c71a620 Address:77538438-2d54-053a-ca33-1e628c71a620}]" 2026-04-22T10:10:21.238Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e61722ad-1c51-ff91-373d-9654c034430e Address:e61722ad-1c51-ff91-373d-9654c034430e} {Suffrage:Voter ID:server-a468eb11-0a8c-79e7-346d-5159613d0dbe Address:a468eb11-0a8c-79e7-346d-5159613d0dbe} {Suffrage:Voter ID:server-77538438-2d54-053a-ca33-1e628c71a620 Address:77538438-2d54-053a-ca33-1e628c71a620}]" 2026-04-22T10:10:21.238Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:21.238Z [INFO] server-a468eb11-0a8c-79e7-346d-5159613d0dbe: entering follower state: follower="Node at a468eb11-0a8c-79e7-346d-5159613d0dbe [Follower]" leader-address= leader-id= 2026-04-22T10:10:21.238Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: entering follower state: follower="Node at 77538438-2d54-053a-ca33-1e628c71a620 [Follower]" leader-address= leader-id= 2026-04-22T10:10:21.808Z [WARN] server-77538438-2d54-053a-ca33-1e628c71a620: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:21.808Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: entering candidate state: node="Node at 77538438-2d54-053a-ca33-1e628c71a620 [Candidate]" term=2 2026-04-22T10:10:21.808Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: election won: term=2 tally=2 2026-04-22T10:10:21.808Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: entering leader state: leader="Node at 77538438-2d54-053a-ca33-1e628c71a620 [Leader]" 2026-04-22T10:10:21.808Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: added peer, starting replication: peer=server-e61722ad-1c51-ff91-373d-9654c034430e 2026-04-22T10:10:21.808Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: added peer, starting replication: peer=server-a468eb11-0a8c-79e7-346d-5159613d0dbe 2026-04-22T10:10:21.808Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: pipelining replication: peer="{Voter server-a468eb11-0a8c-79e7-346d-5159613d0dbe a468eb11-0a8c-79e7-346d-5159613d0dbe}" 2026-04-22T10:10:21.808Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: pipelining replication: peer="{Voter server-e61722ad-1c51-ff91-373d-9654c034430e e61722ad-1c51-ff91-373d-9654c034430e}" 2026-04-22T10:10:22.813Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:21.808510491 +0000 UTC m=+67.832306579 (1 nodes), 569.533184ms from start of poll, 569.757124ms from cluster start. Timeout at 2026-04-22 10:10:22.813532944 +0000 UTC m=+68.837329062, 1.005022483s after stability 2026-04-22T10:10:22.813Z [INFO] creating new snapshot: path=/tmp/raft2452288288/snapshots/2-113-1776852622813.tmp 2026-04-22T10:10:22.814Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: copied to local snapshot: bytes=71 2026-04-22T10:10:22.814Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: snapshot restore progress: id=2-113-1776852622813 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:22.814Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: removing all old logs from log store 2026-04-22T10:10:22.814Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: compacting logs: from=1 to=2 2026-04-22T10:10:22.814Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: restored user snapshot: index=113 2026-04-22T10:10:22.814Z [ERROR] server-77538438-2d54-053a-ca33-1e628c71a620: failed to get log: index=2 error="log not found" 2026-04-22T10:10:22.814Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: aborting pipeline replication: peer="{Voter server-a468eb11-0a8c-79e7-346d-5159613d0dbe a468eb11-0a8c-79e7-346d-5159613d0dbe}" 2026-04-22T10:10:22.814Z [ERROR] server-77538438-2d54-053a-ca33-1e628c71a620: failed to get log: index=2 error="log not found" 2026-04-22T10:10:22.814Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: aborting pipeline replication: peer="{Voter server-e61722ad-1c51-ff91-373d-9654c034430e e61722ad-1c51-ff91-373d-9654c034430e}" 2026-04-22T10:10:22.821Z [ERROR] server-77538438-2d54-053a-ca33-1e628c71a620: failed to get log: index=2 error="log not found" 2026-04-22T10:10:22.821Z [INFO] creating new snapshot: path=/tmp/raft2572710521/snapshots/2-113-1776852622821.tmp 2026-04-22T10:10:22.821Z [INFO] server-e61722ad-1c51-ff91-373d-9654c034430e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:22.821Z [ERROR] server-77538438-2d54-053a-ca33-1e628c71a620: failed to get log: index=2 error="log not found" 2026-04-22T10:10:22.822Z [INFO] creating new snapshot: path=/tmp/raft1937850857/snapshots/2-113-1776852622822.tmp 2026-04-22T10:10:22.822Z [INFO] server-e61722ad-1c51-ff91-373d-9654c034430e: copied to local snapshot: bytes=71 2026-04-22T10:10:22.822Z [INFO] server-a468eb11-0a8c-79e7-346d-5159613d0dbe: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:22.822Z [INFO] server-e61722ad-1c51-ff91-373d-9654c034430e: snapshot restore progress: id=2-113-1776852622821 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:22.822Z [INFO] server-e61722ad-1c51-ff91-373d-9654c034430e: removing all old logs from log store 2026-04-22T10:10:22.822Z [INFO] server-e61722ad-1c51-ff91-373d-9654c034430e: compacting logs: from=1 to=2 2026-04-22T10:10:22.822Z [INFO] server-e61722ad-1c51-ff91-373d-9654c034430e: Installed remote snapshot 2026-04-22T10:10:22.822Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: pipelining replication: peer="{Voter server-e61722ad-1c51-ff91-373d-9654c034430e e61722ad-1c51-ff91-373d-9654c034430e}" 2026-04-22T10:10:22.822Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:22.822Z [INFO] server-a468eb11-0a8c-79e7-346d-5159613d0dbe: copied to local snapshot: bytes=71 2026-04-22T10:10:22.822Z [INFO] server-a468eb11-0a8c-79e7-346d-5159613d0dbe: snapshot restore progress: id=2-113-1776852622822 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:22.822Z [INFO] server-a468eb11-0a8c-79e7-346d-5159613d0dbe: removing all old logs from log store 2026-04-22T10:10:22.822Z [INFO] server-a468eb11-0a8c-79e7-346d-5159613d0dbe: compacting logs: from=1 to=2 2026-04-22T10:10:22.822Z [INFO] server-a468eb11-0a8c-79e7-346d-5159613d0dbe: Installed remote snapshot 2026-04-22T10:10:22.822Z [INFO] server-77538438-2d54-053a-ca33-1e628c71a620: pipelining replication: peer="{Voter server-a468eb11-0a8c-79e7-346d-5159613d0dbe a468eb11-0a8c-79e7-346d-5159613d0dbe}" 2026-04-22T10:10:23.828Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:22.822594655 +0000 UTC m=+68.846390764 (2 nodes), 806ns from start of poll, 1.583841309s from cluster start. Timeout at 2026-04-22 10:10:23.827601802 +0000 UTC m=+69.851397918, 1.005007154s after stability 2026-04-22T10:10:23.845Z [INFO] server-2309cd97-aa43-957c-864f-7bff16a24e84: aborting pipeline replication: peer="{Voter server-07973ad8-9170-e79e-dcf0-990c1e550bc3 07973ad8-9170-e79e-dcf0-990c1e550bc3}" 2026-04-22T10:10:23.845Z [INFO] server-2309cd97-aa43-957c-864f-7bff16a24e84: aborting pipeline replication: peer="{Voter server-76a82980-c888-4e8c-0cb6-2966d429191a 76a82980-c888-4e8c-0cb6-2966d429191a}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false 2026-04-22T10:10:23.849Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f138bdf-0682-e997-6507-1e5b6e93e1a7 Address:8f138bdf-0682-e997-6507-1e5b6e93e1a7} {Suffrage:Voter ID:server-07a4df22-d471-72be-eb48-a813d8a0e9a6 Address:07a4df22-d471-72be-eb48-a813d8a0e9a6} {Suffrage:Voter ID:server-ece219b1-1681-20d8-c65f-784bafa7039c Address:ece219b1-1681-20d8-c65f-784bafa7039c}]" 2026-04-22T10:10:23.849Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: entering follower state: follower="Node at 8f138bdf-0682-e997-6507-1e5b6e93e1a7 [Follower]" leader-address= leader-id= 2026-04-22T10:10:23.849Z [INFO] server-07a4df22-d471-72be-eb48-a813d8a0e9a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f138bdf-0682-e997-6507-1e5b6e93e1a7 Address:8f138bdf-0682-e997-6507-1e5b6e93e1a7} {Suffrage:Voter ID:server-07a4df22-d471-72be-eb48-a813d8a0e9a6 Address:07a4df22-d471-72be-eb48-a813d8a0e9a6} {Suffrage:Voter ID:server-ece219b1-1681-20d8-c65f-784bafa7039c Address:ece219b1-1681-20d8-c65f-784bafa7039c}]" 2026-04-22T10:10:23.849Z [INFO] server-07a4df22-d471-72be-eb48-a813d8a0e9a6: entering follower state: follower="Node at 07a4df22-d471-72be-eb48-a813d8a0e9a6 [Follower]" leader-address= leader-id= 2026-04-22T10:10:23.849Z [INFO] server-ece219b1-1681-20d8-c65f-784bafa7039c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f138bdf-0682-e997-6507-1e5b6e93e1a7 Address:8f138bdf-0682-e997-6507-1e5b6e93e1a7} {Suffrage:Voter ID:server-07a4df22-d471-72be-eb48-a813d8a0e9a6 Address:07a4df22-d471-72be-eb48-a813d8a0e9a6} {Suffrage:Voter ID:server-ece219b1-1681-20d8-c65f-784bafa7039c Address:ece219b1-1681-20d8-c65f-784bafa7039c}]" 2026-04-22T10:10:23.849Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:23.849Z [INFO] server-ece219b1-1681-20d8-c65f-784bafa7039c: entering follower state: follower="Node at ece219b1-1681-20d8-c65f-784bafa7039c [Follower]" leader-address= leader-id= 2026-04-22T10:10:24.357Z [WARN] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:24.357Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: entering candidate state: node="Node at 8f138bdf-0682-e997-6507-1e5b6e93e1a7 [Candidate]" term=2 2026-04-22T10:10:24.357Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: election won: term=2 tally=2 2026-04-22T10:10:24.357Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: entering leader state: leader="Node at 8f138bdf-0682-e997-6507-1e5b6e93e1a7 [Leader]" 2026-04-22T10:10:24.357Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: added peer, starting replication: peer=server-07a4df22-d471-72be-eb48-a813d8a0e9a6 2026-04-22T10:10:24.357Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: added peer, starting replication: peer=server-ece219b1-1681-20d8-c65f-784bafa7039c 2026-04-22T10:10:24.357Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: pipelining replication: peer="{Voter server-ece219b1-1681-20d8-c65f-784bafa7039c ece219b1-1681-20d8-c65f-784bafa7039c}" 2026-04-22T10:10:24.357Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: pipelining replication: peer="{Voter server-07a4df22-d471-72be-eb48-a813d8a0e9a6 07a4df22-d471-72be-eb48-a813d8a0e9a6}" 2026-04-22T10:10:25.362Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:24.357659732 +0000 UTC m=+70.381455845 (1 nodes), 507.73771ms from start of poll, 508.04482ms from cluster start. Timeout at 2026-04-22 10:10:25.362668258 +0000 UTC m=+71.386464404, 1.005008559s after stability 2026-04-22T10:10:25.363Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: starting snapshot up to: index=12 2026-04-22T10:10:25.363Z [INFO] creating new snapshot: path=/tmp/raft364147191/snapshots/2-12-1776852625363.tmp 2026-04-22T10:10:25.364Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: snapshot complete up to: index=12 2026-04-22T10:10:25.364Z [INFO] creating new snapshot: path=/tmp/raft364147191/snapshots/2-1013-1776852625364.tmp 2026-04-22T10:10:25.365Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: copied to local snapshot: bytes=71 2026-04-22T10:10:25.365Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: snapshot restore progress: id=2-1013-1776852625364 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:25.365Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: restored user snapshot: index=1013 2026-04-22T10:10:25.365Z [ERROR] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: failed to get log: index=23 error="log not found" 2026-04-22T10:10:25.365Z [ERROR] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: failed to get log: index=23 error="log not found" 2026-04-22T10:10:25.365Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: aborting pipeline replication: peer="{Voter server-07a4df22-d471-72be-eb48-a813d8a0e9a6 07a4df22-d471-72be-eb48-a813d8a0e9a6}" 2026-04-22T10:10:25.365Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: aborting pipeline replication: peer="{Voter server-ece219b1-1681-20d8-c65f-784bafa7039c ece219b1-1681-20d8-c65f-784bafa7039c}" 2026-04-22T10:10:25.372Z [ERROR] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: failed to get log: index=23 error="log not found" 2026-04-22T10:10:25.372Z [INFO] creating new snapshot: path=/tmp/raft3137963848/snapshots/2-1013-1776852625372.tmp 2026-04-22T10:10:25.372Z [INFO] server-ece219b1-1681-20d8-c65f-784bafa7039c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:25.373Z [INFO] server-ece219b1-1681-20d8-c65f-784bafa7039c: copied to local snapshot: bytes=71 2026-04-22T10:10:25.373Z [INFO] server-ece219b1-1681-20d8-c65f-784bafa7039c: snapshot restore progress: id=2-1013-1776852625372 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:25.373Z [INFO] server-ece219b1-1681-20d8-c65f-784bafa7039c: Installed remote snapshot 2026-04-22T10:10:25.373Z [WARN] server-ece219b1-1681-20d8-c65f-784bafa7039c: skipping application of old log: index=1013 2026-04-22T10:10:25.373Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: pipelining replication: peer="{Voter server-ece219b1-1681-20d8-c65f-784bafa7039c ece219b1-1681-20d8-c65f-784bafa7039c}" 2026-04-22T10:10:25.373Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:25.374Z [ERROR] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: failed to get log: index=23 error="log not found" 2026-04-22T10:10:25.375Z [INFO] creating new snapshot: path=/tmp/raft3639915264/snapshots/2-1013-1776852625375.tmp 2026-04-22T10:10:25.375Z [INFO] server-07a4df22-d471-72be-eb48-a813d8a0e9a6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:25.375Z [INFO] server-07a4df22-d471-72be-eb48-a813d8a0e9a6: copied to local snapshot: bytes=71 2026-04-22T10:10:25.375Z [INFO] server-07a4df22-d471-72be-eb48-a813d8a0e9a6: snapshot restore progress: id=2-1013-1776852625375 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:25.375Z [INFO] server-07a4df22-d471-72be-eb48-a813d8a0e9a6: Installed remote snapshot 2026-04-22T10:10:25.375Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: pipelining replication: peer="{Voter server-07a4df22-d471-72be-eb48-a813d8a0e9a6 07a4df22-d471-72be-eb48-a813d8a0e9a6}" 2026-04-22T10:10:26.379Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:25.37369818 +0000 UTC m=+71.397494282 (2 nodes), 877ns from start of poll, 1.524083257s from cluster start. Timeout at 2026-04-22 10:10:26.378711614 +0000 UTC m=+72.402507838, 1.005013556s after stability 2026-04-22T10:10:26.392Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: aborting pipeline replication: peer="{Voter server-ece219b1-1681-20d8-c65f-784bafa7039c ece219b1-1681-20d8-c65f-784bafa7039c}" 2026-04-22T10:10:26.392Z [INFO] server-8f138bdf-0682-e997-6507-1e5b6e93e1a7: aborting pipeline replication: peer="{Voter server-07a4df22-d471-72be-eb48-a813d8a0e9a6 07a4df22-d471-72be-eb48-a813d8a0e9a6}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false 2026-04-22T10:10:26.393Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-693f3dbf-8ac8-16ec-75dc-1be133385b0e Address:693f3dbf-8ac8-16ec-75dc-1be133385b0e} {Suffrage:Voter ID:server-18b0c58b-eb00-040a-86d9-bb62bea4b584 Address:18b0c58b-eb00-040a-86d9-bb62bea4b584} {Suffrage:Voter ID:server-f1c0d81b-c7ec-45d7-0326-772e73b3c628 Address:f1c0d81b-c7ec-45d7-0326-772e73b3c628}]" 2026-04-22T10:10:26.394Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: entering follower state: follower="Node at 693f3dbf-8ac8-16ec-75dc-1be133385b0e [Follower]" leader-address= leader-id= 2026-04-22T10:10:26.394Z [INFO] server-18b0c58b-eb00-040a-86d9-bb62bea4b584: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-693f3dbf-8ac8-16ec-75dc-1be133385b0e Address:693f3dbf-8ac8-16ec-75dc-1be133385b0e} {Suffrage:Voter ID:server-18b0c58b-eb00-040a-86d9-bb62bea4b584 Address:18b0c58b-eb00-040a-86d9-bb62bea4b584} {Suffrage:Voter ID:server-f1c0d81b-c7ec-45d7-0326-772e73b3c628 Address:f1c0d81b-c7ec-45d7-0326-772e73b3c628}]" 2026-04-22T10:10:26.394Z [INFO] server-18b0c58b-eb00-040a-86d9-bb62bea4b584: entering follower state: follower="Node at 18b0c58b-eb00-040a-86d9-bb62bea4b584 [Follower]" leader-address= leader-id= 2026-04-22T10:10:26.394Z [INFO] server-f1c0d81b-c7ec-45d7-0326-772e73b3c628: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-693f3dbf-8ac8-16ec-75dc-1be133385b0e Address:693f3dbf-8ac8-16ec-75dc-1be133385b0e} {Suffrage:Voter ID:server-18b0c58b-eb00-040a-86d9-bb62bea4b584 Address:18b0c58b-eb00-040a-86d9-bb62bea4b584} {Suffrage:Voter ID:server-f1c0d81b-c7ec-45d7-0326-772e73b3c628 Address:f1c0d81b-c7ec-45d7-0326-772e73b3c628}]" 2026-04-22T10:10:26.394Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:26.394Z [INFO] server-f1c0d81b-c7ec-45d7-0326-772e73b3c628: entering follower state: follower="Node at f1c0d81b-c7ec-45d7-0326-772e73b3c628 [Follower]" leader-address= leader-id= 2026-04-22T10:10:26.995Z [WARN] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:26.995Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: entering candidate state: node="Node at 693f3dbf-8ac8-16ec-75dc-1be133385b0e [Candidate]" term=2 2026-04-22T10:10:26.995Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: election won: term=2 tally=2 2026-04-22T10:10:26.995Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: entering leader state: leader="Node at 693f3dbf-8ac8-16ec-75dc-1be133385b0e [Leader]" 2026-04-22T10:10:26.995Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: added peer, starting replication: peer=server-18b0c58b-eb00-040a-86d9-bb62bea4b584 2026-04-22T10:10:26.995Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: added peer, starting replication: peer=server-f1c0d81b-c7ec-45d7-0326-772e73b3c628 2026-04-22T10:10:26.995Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: pipelining replication: peer="{Voter server-f1c0d81b-c7ec-45d7-0326-772e73b3c628 f1c0d81b-c7ec-45d7-0326-772e73b3c628}" 2026-04-22T10:10:26.995Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: pipelining replication: peer="{Voter server-18b0c58b-eb00-040a-86d9-bb62bea4b584 18b0c58b-eb00-040a-86d9-bb62bea4b584}" 2026-04-22T10:10:28.000Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:26.995467489 +0000 UTC m=+73.019263606 (1 nodes), 600.85111ms from start of poll, 601.7643ms from cluster start. Timeout at 2026-04-22 10:10:28.000473985 +0000 UTC m=+74.024270101, 1.005006495s after stability 2026-04-22T10:10:28.001Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: starting snapshot up to: index=12 2026-04-22T10:10:28.001Z [INFO] creating new snapshot: path=/tmp/raft3083185278/snapshots/2-12-1776852628001.tmp 2026-04-22T10:10:28.001Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: snapshot complete up to: index=12 2026-04-22T10:10:28.002Z [INFO] creating new snapshot: path=/tmp/raft3083185278/snapshots/2-1013-1776852628002.tmp 2026-04-22T10:10:28.003Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: copied to local snapshot: bytes=71 2026-04-22T10:10:28.003Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: snapshot restore progress: id=2-1013-1776852628002 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:28.003Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: removing all old logs from log store 2026-04-22T10:10:28.003Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: compacting logs: from=1 to=22 2026-04-22T10:10:28.003Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: restored user snapshot: index=1013 2026-04-22T10:10:28.003Z [ERROR] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: failed to get log: index=22 error="log not found" 2026-04-22T10:10:28.003Z [ERROR] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: failed to get log: index=22 error="log not found" 2026-04-22T10:10:28.003Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: aborting pipeline replication: peer="{Voter server-18b0c58b-eb00-040a-86d9-bb62bea4b584 18b0c58b-eb00-040a-86d9-bb62bea4b584}" 2026-04-22T10:10:28.003Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: aborting pipeline replication: peer="{Voter server-f1c0d81b-c7ec-45d7-0326-772e73b3c628 f1c0d81b-c7ec-45d7-0326-772e73b3c628}" 2026-04-22T10:10:28.009Z [ERROR] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: failed to get log: index=22 error="log not found" 2026-04-22T10:10:28.010Z [INFO] creating new snapshot: path=/tmp/raft1807093734/snapshots/2-1013-1776852628010.tmp 2026-04-22T10:10:28.010Z [INFO] server-18b0c58b-eb00-040a-86d9-bb62bea4b584: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:28.010Z [INFO] server-18b0c58b-eb00-040a-86d9-bb62bea4b584: copied to local snapshot: bytes=71 2026-04-22T10:10:28.010Z [INFO] server-18b0c58b-eb00-040a-86d9-bb62bea4b584: snapshot restore progress: id=2-1013-1776852628010 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:28.010Z [INFO] server-18b0c58b-eb00-040a-86d9-bb62bea4b584: removing all old logs from log store 2026-04-22T10:10:28.010Z [INFO] server-18b0c58b-eb00-040a-86d9-bb62bea4b584: compacting logs: from=1 to=22 2026-04-22T10:10:28.010Z [INFO] server-18b0c58b-eb00-040a-86d9-bb62bea4b584: Installed remote snapshot 2026-04-22T10:10:28.010Z [WARN] server-18b0c58b-eb00-040a-86d9-bb62bea4b584: skipping application of old log: index=22 2026-04-22T10:10:28.010Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: pipelining replication: peer="{Voter server-18b0c58b-eb00-040a-86d9-bb62bea4b584 18b0c58b-eb00-040a-86d9-bb62bea4b584}" 2026-04-22T10:10:28.010Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:28.013Z [ERROR] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: failed to get log: index=22 error="log not found" 2026-04-22T10:10:28.013Z [INFO] creating new snapshot: path=/tmp/raft718060485/snapshots/2-1013-1776852628013.tmp 2026-04-22T10:10:28.013Z [INFO] server-f1c0d81b-c7ec-45d7-0326-772e73b3c628: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:28.014Z [INFO] server-f1c0d81b-c7ec-45d7-0326-772e73b3c628: copied to local snapshot: bytes=71 2026-04-22T10:10:28.014Z [INFO] server-f1c0d81b-c7ec-45d7-0326-772e73b3c628: snapshot restore progress: id=2-1013-1776852628013 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:28.014Z [INFO] server-f1c0d81b-c7ec-45d7-0326-772e73b3c628: removing all old logs from log store 2026-04-22T10:10:28.014Z [INFO] server-f1c0d81b-c7ec-45d7-0326-772e73b3c628: compacting logs: from=1 to=22 2026-04-22T10:10:28.014Z [INFO] server-f1c0d81b-c7ec-45d7-0326-772e73b3c628: Installed remote snapshot 2026-04-22T10:10:28.015Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: pipelining replication: peer="{Voter server-f1c0d81b-c7ec-45d7-0326-772e73b3c628 f1c0d81b-c7ec-45d7-0326-772e73b3c628}" 2026-04-22T10:10:29.016Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:28.010863136 +0000 UTC m=+74.034659222 (2 nodes), 629ns from start of poll, 1.617159916s from cluster start. Timeout at 2026-04-22 10:10:29.015866278 +0000 UTC m=+75.039662361, 1.005003139s after stability 2026-04-22T10:10:29.028Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: aborting pipeline replication: peer="{Voter server-f1c0d81b-c7ec-45d7-0326-772e73b3c628 f1c0d81b-c7ec-45d7-0326-772e73b3c628}" 2026-04-22T10:10:29.028Z [INFO] server-693f3dbf-8ac8-16ec-75dc-1be133385b0e: aborting pipeline replication: peer="{Voter server-18b0c58b-eb00-040a-86d9-bb62bea4b584 18b0c58b-eb00-040a-86d9-bb62bea4b584}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true 2026-04-22T10:10:29.033Z [INFO] server-9e30c983-0aa8-13c7-5d6e-153fde24f5f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e30c983-0aa8-13c7-5d6e-153fde24f5f3 Address:9e30c983-0aa8-13c7-5d6e-153fde24f5f3} {Suffrage:Voter ID:server-54b3284e-4caa-7ca7-0c3c-a7c933c43379 Address:54b3284e-4caa-7ca7-0c3c-a7c933c43379} {Suffrage:Voter ID:server-fb05d6cf-79f8-cc0f-a861-81fc30896f50 Address:fb05d6cf-79f8-cc0f-a861-81fc30896f50}]" 2026-04-22T10:10:29.033Z [INFO] server-9e30c983-0aa8-13c7-5d6e-153fde24f5f3: entering follower state: follower="Node at 9e30c983-0aa8-13c7-5d6e-153fde24f5f3 [Follower]" leader-address= leader-id= 2026-04-22T10:10:29.033Z [INFO] server-54b3284e-4caa-7ca7-0c3c-a7c933c43379: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e30c983-0aa8-13c7-5d6e-153fde24f5f3 Address:9e30c983-0aa8-13c7-5d6e-153fde24f5f3} {Suffrage:Voter ID:server-54b3284e-4caa-7ca7-0c3c-a7c933c43379 Address:54b3284e-4caa-7ca7-0c3c-a7c933c43379} {Suffrage:Voter ID:server-fb05d6cf-79f8-cc0f-a861-81fc30896f50 Address:fb05d6cf-79f8-cc0f-a861-81fc30896f50}]" 2026-04-22T10:10:29.033Z [INFO] server-54b3284e-4caa-7ca7-0c3c-a7c933c43379: entering follower state: follower="Node at 54b3284e-4caa-7ca7-0c3c-a7c933c43379 [Follower]" leader-address= leader-id= 2026-04-22T10:10:29.033Z [INFO] server-fb05d6cf-79f8-cc0f-a861-81fc30896f50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e30c983-0aa8-13c7-5d6e-153fde24f5f3 Address:9e30c983-0aa8-13c7-5d6e-153fde24f5f3} {Suffrage:Voter ID:server-54b3284e-4caa-7ca7-0c3c-a7c933c43379 Address:54b3284e-4caa-7ca7-0c3c-a7c933c43379} {Suffrage:Voter ID:server-fb05d6cf-79f8-cc0f-a861-81fc30896f50 Address:fb05d6cf-79f8-cc0f-a861-81fc30896f50}]" 2026-04-22T10:10:29.033Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:29.033Z [INFO] server-fb05d6cf-79f8-cc0f-a861-81fc30896f50: entering follower state: follower="Node at fb05d6cf-79f8-cc0f-a861-81fc30896f50 [Follower]" leader-address= leader-id= 2026-04-22T10:10:29.570Z [WARN] server-fb05d6cf-79f8-cc0f-a861-81fc30896f50: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:29.570Z [INFO] server-fb05d6cf-79f8-cc0f-a861-81fc30896f50: entering candidate state: node="Node at fb05d6cf-79f8-cc0f-a861-81fc30896f50 [Candidate]" term=2 2026-04-22T10:10:29.570Z [INFO] server-fb05d6cf-79f8-cc0f-a861-81fc30896f50: election won: term=2 tally=2 2026-04-22T10:10:29.570Z [INFO] server-fb05d6cf-79f8-cc0f-a861-81fc30896f50: entering leader state: leader="Node at fb05d6cf-79f8-cc0f-a861-81fc30896f50 [Leader]" 2026-04-22T10:10:29.570Z [INFO] server-fb05d6cf-79f8-cc0f-a861-81fc30896f50: added peer, starting replication: peer=server-9e30c983-0aa8-13c7-5d6e-153fde24f5f3 2026-04-22T10:10:29.570Z [INFO] server-fb05d6cf-79f8-cc0f-a861-81fc30896f50: added peer, starting replication: peer=server-54b3284e-4caa-7ca7-0c3c-a7c933c43379 2026-04-22T10:10:29.571Z [INFO] server-fb05d6cf-79f8-cc0f-a861-81fc30896f50: pipelining replication: peer="{Voter server-54b3284e-4caa-7ca7-0c3c-a7c933c43379 54b3284e-4caa-7ca7-0c3c-a7c933c43379}" 2026-04-22T10:10:29.571Z [INFO] server-fb05d6cf-79f8-cc0f-a861-81fc30896f50: pipelining replication: peer="{Voter server-9e30c983-0aa8-13c7-5d6e-153fde24f5f3 9e30c983-0aa8-13c7-5d6e-153fde24f5f3}" 2026-04-22T10:10:30.576Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:29.570988637 +0000 UTC m=+75.594784722 (1 nodes), 537.252637ms from start of poll, 538.129476ms from cluster start. Timeout at 2026-04-22 10:10:30.575990752 +0000 UTC m=+76.599786832, 1.00500211s after stability 2026-04-22T10:10:30.576Z [INFO] server-fb05d6cf-79f8-cc0f-a861-81fc30896f50: starting snapshot up to: index=12 2026-04-22T10:10:30.576Z [INFO] creating new snapshot: path=/tmp/raft4151621761/snapshots/2-12-1776852630576.tmp 2026-04-22T10:10:30.577Z [INFO] server-fb05d6cf-79f8-cc0f-a861-81fc30896f50: snapshot complete up to: index=12 2026-04-22T10:10:30.578Z [INFO] server-21c703c0-a4d0-9d9f-ba03-528c4cba67a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21c703c0-a4d0-9d9f-ba03-528c4cba67a2 Address:21c703c0-a4d0-9d9f-ba03-528c4cba67a2} {Suffrage:Voter ID:server-fa73ec27-99bd-601c-41e2-869b7ac99f7d Address:fa73ec27-99bd-601c-41e2-869b7ac99f7d} {Suffrage:Voter ID:server-1bb58cda-7edd-04e7-e576-7ea1fa87ffea Address:1bb58cda-7edd-04e7-e576-7ea1fa87ffea}]" 2026-04-22T10:10:30.578Z [INFO] server-21c703c0-a4d0-9d9f-ba03-528c4cba67a2: entering follower state: follower="Node at 21c703c0-a4d0-9d9f-ba03-528c4cba67a2 [Follower]" leader-address= leader-id= 2026-04-22T10:10:30.578Z [INFO] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21c703c0-a4d0-9d9f-ba03-528c4cba67a2 Address:21c703c0-a4d0-9d9f-ba03-528c4cba67a2} {Suffrage:Voter ID:server-fa73ec27-99bd-601c-41e2-869b7ac99f7d Address:fa73ec27-99bd-601c-41e2-869b7ac99f7d} {Suffrage:Voter ID:server-1bb58cda-7edd-04e7-e576-7ea1fa87ffea Address:1bb58cda-7edd-04e7-e576-7ea1fa87ffea}]" 2026-04-22T10:10:30.578Z [INFO] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: entering follower state: follower="Node at fa73ec27-99bd-601c-41e2-869b7ac99f7d [Follower]" leader-address= leader-id= 2026-04-22T10:10:30.578Z [INFO] server-1bb58cda-7edd-04e7-e576-7ea1fa87ffea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21c703c0-a4d0-9d9f-ba03-528c4cba67a2 Address:21c703c0-a4d0-9d9f-ba03-528c4cba67a2} {Suffrage:Voter ID:server-fa73ec27-99bd-601c-41e2-869b7ac99f7d Address:fa73ec27-99bd-601c-41e2-869b7ac99f7d} {Suffrage:Voter ID:server-1bb58cda-7edd-04e7-e576-7ea1fa87ffea Address:1bb58cda-7edd-04e7-e576-7ea1fa87ffea}]" 2026-04-22T10:10:30.578Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:30.578Z [INFO] server-1bb58cda-7edd-04e7-e576-7ea1fa87ffea: entering follower state: follower="Node at 1bb58cda-7edd-04e7-e576-7ea1fa87ffea [Follower]" leader-address= leader-id= 2026-04-22T10:10:31.212Z [WARN] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:31.212Z [INFO] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: entering candidate state: node="Node at fa73ec27-99bd-601c-41e2-869b7ac99f7d [Candidate]" term=2 2026-04-22T10:10:31.212Z [INFO] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: election won: term=2 tally=2 2026-04-22T10:10:31.212Z [INFO] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: entering leader state: leader="Node at fa73ec27-99bd-601c-41e2-869b7ac99f7d [Leader]" 2026-04-22T10:10:31.212Z [INFO] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: added peer, starting replication: peer=server-21c703c0-a4d0-9d9f-ba03-528c4cba67a2 2026-04-22T10:10:31.212Z [INFO] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: added peer, starting replication: peer=server-1bb58cda-7edd-04e7-e576-7ea1fa87ffea 2026-04-22T10:10:31.212Z [INFO] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: pipelining replication: peer="{Voter server-21c703c0-a4d0-9d9f-ba03-528c4cba67a2 21c703c0-a4d0-9d9f-ba03-528c4cba67a2}" 2026-04-22T10:10:31.212Z [INFO] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: pipelining replication: peer="{Voter server-1bb58cda-7edd-04e7-e576-7ea1fa87ffea 1bb58cda-7edd-04e7-e576-7ea1fa87ffea}" 2026-04-22T10:10:32.218Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:31.212371216 +0000 UTC m=+77.236167307 (1 nodes), 633.818247ms from start of poll, 634.135217ms from cluster start. Timeout at 2026-04-22 10:10:32.217377657 +0000 UTC m=+78.241173740, 1.005006433s after stability 2026-04-22T10:10:32.218Z [INFO] creating new snapshot: path=/tmp/raft972038166/snapshots/2-1013-1776852632218.tmp 2026-04-22T10:10:32.219Z [INFO] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: copied to local snapshot: bytes=71 2026-04-22T10:10:32.219Z [INFO] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: snapshot restore progress: id=2-1013-1776852632218 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:32.219Z [INFO] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: restored user snapshot: index=1013 2026-04-22T10:10:32.219Z [ERROR] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: failed to get log: index=3 error="log not found" 2026-04-22T10:10:32.219Z [INFO] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: aborting pipeline replication: peer="{Voter server-1bb58cda-7edd-04e7-e576-7ea1fa87ffea 1bb58cda-7edd-04e7-e576-7ea1fa87ffea}" 2026-04-22T10:10:32.219Z [ERROR] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: failed to get log: index=3 error="log not found" 2026-04-22T10:10:32.219Z [INFO] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: aborting pipeline replication: peer="{Voter server-21c703c0-a4d0-9d9f-ba03-528c4cba67a2 21c703c0-a4d0-9d9f-ba03-528c4cba67a2}" 2026-04-22T10:10:32.225Z [ERROR] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: failed to get log: index=3 error="log not found" 2026-04-22T10:10:32.225Z [INFO] creating new snapshot: path=/tmp/raft1926815847/snapshots/2-1013-1776852632225.tmp 2026-04-22T10:10:32.225Z [INFO] server-1bb58cda-7edd-04e7-e576-7ea1fa87ffea: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:32.226Z [INFO] server-1bb58cda-7edd-04e7-e576-7ea1fa87ffea: copied to local snapshot: bytes=71 2026-04-22T10:10:32.226Z [INFO] server-1bb58cda-7edd-04e7-e576-7ea1fa87ffea: snapshot restore progress: id=2-1013-1776852632225 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:32.226Z [INFO] server-1bb58cda-7edd-04e7-e576-7ea1fa87ffea: Installed remote snapshot 2026-04-22T10:10:32.226Z [INFO] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: pipelining replication: peer="{Voter server-1bb58cda-7edd-04e7-e576-7ea1fa87ffea 1bb58cda-7edd-04e7-e576-7ea1fa87ffea}" 2026-04-22T10:10:32.226Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:32.230Z [ERROR] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: failed to get log: index=3 error="log not found" 2026-04-22T10:10:32.230Z [INFO] creating new snapshot: path=/tmp/raft64231691/snapshots/2-1013-1776852632230.tmp 2026-04-22T10:10:32.230Z [INFO] server-21c703c0-a4d0-9d9f-ba03-528c4cba67a2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:32.230Z [INFO] server-21c703c0-a4d0-9d9f-ba03-528c4cba67a2: copied to local snapshot: bytes=71 2026-04-22T10:10:32.231Z [INFO] server-21c703c0-a4d0-9d9f-ba03-528c4cba67a2: snapshot restore progress: id=2-1013-1776852632230 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:32.231Z [INFO] server-21c703c0-a4d0-9d9f-ba03-528c4cba67a2: Installed remote snapshot 2026-04-22T10:10:32.231Z [INFO] server-fa73ec27-99bd-601c-41e2-869b7ac99f7d: pipelining replication: peer="{Voter server-21c703c0-a4d0-9d9f-ba03-528c4cba67a2 21c703c0-a4d0-9d9f-ba03-528c4cba67a2}" 2026-04-22T10:10:33.231Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:32.226524437 +0000 UTC m=+78.250320523 (2 nodes), 566ns from start of poll, 1.648288433s from cluster start. Timeout at 2026-04-22 10:10:33.231535163 +0000 UTC m=+79.255331244, 1.005010721s after stability 2026-04-22T10:10:33.242Z [INFO] server-fb05d6cf-79f8-cc0f-a861-81fc30896f50: aborting pipeline replication: peer="{Voter server-9e30c983-0aa8-13c7-5d6e-153fde24f5f3 9e30c983-0aa8-13c7-5d6e-153fde24f5f3}" 2026-04-22T10:10:33.242Z [INFO] server-fb05d6cf-79f8-cc0f-a861-81fc30896f50: aborting pipeline replication: peer="{Voter server-54b3284e-4caa-7ca7-0c3c-a7c933c43379 54b3284e-4caa-7ca7-0c3c-a7c933c43379}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true 2026-04-22T10:10:33.244Z [INFO] server-63c00e54-84bd-a944-ed8e-85e1f6e1a860: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63c00e54-84bd-a944-ed8e-85e1f6e1a860 Address:63c00e54-84bd-a944-ed8e-85e1f6e1a860} {Suffrage:Voter ID:server-ad91fa6a-c384-fb2d-ecd5-d4de76a404b0 Address:ad91fa6a-c384-fb2d-ecd5-d4de76a404b0} {Suffrage:Voter ID:server-f08abc32-a7e8-0985-6a9f-52928273156f Address:f08abc32-a7e8-0985-6a9f-52928273156f}]" 2026-04-22T10:10:33.244Z [INFO] server-63c00e54-84bd-a944-ed8e-85e1f6e1a860: entering follower state: follower="Node at 63c00e54-84bd-a944-ed8e-85e1f6e1a860 [Follower]" leader-address= leader-id= 2026-04-22T10:10:33.245Z [INFO] server-ad91fa6a-c384-fb2d-ecd5-d4de76a404b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63c00e54-84bd-a944-ed8e-85e1f6e1a860 Address:63c00e54-84bd-a944-ed8e-85e1f6e1a860} {Suffrage:Voter ID:server-ad91fa6a-c384-fb2d-ecd5-d4de76a404b0 Address:ad91fa6a-c384-fb2d-ecd5-d4de76a404b0} {Suffrage:Voter ID:server-f08abc32-a7e8-0985-6a9f-52928273156f Address:f08abc32-a7e8-0985-6a9f-52928273156f}]" 2026-04-22T10:10:33.245Z [INFO] server-ad91fa6a-c384-fb2d-ecd5-d4de76a404b0: entering follower state: follower="Node at ad91fa6a-c384-fb2d-ecd5-d4de76a404b0 [Follower]" leader-address= leader-id= 2026-04-22T10:10:33.245Z [INFO] server-f08abc32-a7e8-0985-6a9f-52928273156f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63c00e54-84bd-a944-ed8e-85e1f6e1a860 Address:63c00e54-84bd-a944-ed8e-85e1f6e1a860} {Suffrage:Voter ID:server-ad91fa6a-c384-fb2d-ecd5-d4de76a404b0 Address:ad91fa6a-c384-fb2d-ecd5-d4de76a404b0} {Suffrage:Voter ID:server-f08abc32-a7e8-0985-6a9f-52928273156f Address:f08abc32-a7e8-0985-6a9f-52928273156f}]" 2026-04-22T10:10:33.245Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:33.245Z [INFO] server-f08abc32-a7e8-0985-6a9f-52928273156f: entering follower state: follower="Node at f08abc32-a7e8-0985-6a9f-52928273156f [Follower]" leader-address= leader-id= 2026-04-22T10:10:33.752Z [WARN] server-f08abc32-a7e8-0985-6a9f-52928273156f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:33.753Z [INFO] server-f08abc32-a7e8-0985-6a9f-52928273156f: entering candidate state: node="Node at f08abc32-a7e8-0985-6a9f-52928273156f [Candidate]" term=2 2026-04-22T10:10:33.753Z [INFO] server-f08abc32-a7e8-0985-6a9f-52928273156f: election won: term=2 tally=2 2026-04-22T10:10:33.753Z [INFO] server-f08abc32-a7e8-0985-6a9f-52928273156f: entering leader state: leader="Node at f08abc32-a7e8-0985-6a9f-52928273156f [Leader]" 2026-04-22T10:10:33.753Z [INFO] server-f08abc32-a7e8-0985-6a9f-52928273156f: added peer, starting replication: peer=server-63c00e54-84bd-a944-ed8e-85e1f6e1a860 2026-04-22T10:10:33.753Z [INFO] server-f08abc32-a7e8-0985-6a9f-52928273156f: added peer, starting replication: peer=server-ad91fa6a-c384-fb2d-ecd5-d4de76a404b0 2026-04-22T10:10:33.753Z [INFO] server-f08abc32-a7e8-0985-6a9f-52928273156f: pipelining replication: peer="{Voter server-63c00e54-84bd-a944-ed8e-85e1f6e1a860 63c00e54-84bd-a944-ed8e-85e1f6e1a860}" 2026-04-22T10:10:33.753Z [INFO] server-f08abc32-a7e8-0985-6a9f-52928273156f: pipelining replication: peer="{Voter server-ad91fa6a-c384-fb2d-ecd5-d4de76a404b0 ad91fa6a-c384-fb2d-ecd5-d4de76a404b0}" 2026-04-22T10:10:34.758Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:33.753188901 +0000 UTC m=+79.776985007 (1 nodes), 507.936385ms from start of poll, 508.767238ms from cluster start. Timeout at 2026-04-22 10:10:34.758191958 +0000 UTC m=+80.781988041, 1.005003034s after stability 2026-04-22T10:10:34.759Z [INFO] server-f08abc32-a7e8-0985-6a9f-52928273156f: starting snapshot up to: index=12 2026-04-22T10:10:34.759Z [INFO] creating new snapshot: path=/tmp/raft3859420460/snapshots/2-12-1776852634759.tmp 2026-04-22T10:10:34.764Z [INFO] server-f08abc32-a7e8-0985-6a9f-52928273156f: snapshot complete up to: index=12 2026-04-22T10:10:34.765Z [INFO] server-81157623-98e1-92d4-c3b3-80583eb268f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81157623-98e1-92d4-c3b3-80583eb268f7 Address:81157623-98e1-92d4-c3b3-80583eb268f7} {Suffrage:Voter ID:server-8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988 Address:8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988} {Suffrage:Voter ID:server-63f5c036-ea37-45b4-ccbb-8c472e7db78f Address:63f5c036-ea37-45b4-ccbb-8c472e7db78f}]" 2026-04-22T10:10:34.765Z [INFO] server-81157623-98e1-92d4-c3b3-80583eb268f7: entering follower state: follower="Node at 81157623-98e1-92d4-c3b3-80583eb268f7 [Follower]" leader-address= leader-id= 2026-04-22T10:10:34.766Z [INFO] server-8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81157623-98e1-92d4-c3b3-80583eb268f7 Address:81157623-98e1-92d4-c3b3-80583eb268f7} {Suffrage:Voter ID:server-8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988 Address:8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988} {Suffrage:Voter ID:server-63f5c036-ea37-45b4-ccbb-8c472e7db78f Address:63f5c036-ea37-45b4-ccbb-8c472e7db78f}]" 2026-04-22T10:10:34.766Z [INFO] server-8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988: entering follower state: follower="Node at 8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988 [Follower]" leader-address= leader-id= 2026-04-22T10:10:34.766Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81157623-98e1-92d4-c3b3-80583eb268f7 Address:81157623-98e1-92d4-c3b3-80583eb268f7} {Suffrage:Voter ID:server-8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988 Address:8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988} {Suffrage:Voter ID:server-63f5c036-ea37-45b4-ccbb-8c472e7db78f Address:63f5c036-ea37-45b4-ccbb-8c472e7db78f}]" 2026-04-22T10:10:34.766Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:34.766Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: entering follower state: follower="Node at 63f5c036-ea37-45b4-ccbb-8c472e7db78f [Follower]" leader-address= leader-id= 2026-04-22T10:10:35.458Z [WARN] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:35.458Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: entering candidate state: node="Node at 63f5c036-ea37-45b4-ccbb-8c472e7db78f [Candidate]" term=2 2026-04-22T10:10:35.458Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: election won: term=2 tally=2 2026-04-22T10:10:35.458Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: entering leader state: leader="Node at 63f5c036-ea37-45b4-ccbb-8c472e7db78f [Leader]" 2026-04-22T10:10:35.458Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: added peer, starting replication: peer=server-81157623-98e1-92d4-c3b3-80583eb268f7 2026-04-22T10:10:35.458Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: added peer, starting replication: peer=server-8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988 2026-04-22T10:10:35.458Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: pipelining replication: peer="{Voter server-8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988 8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988}" 2026-04-22T10:10:35.458Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: pipelining replication: peer="{Voter server-81157623-98e1-92d4-c3b3-80583eb268f7 81157623-98e1-92d4-c3b3-80583eb268f7}" 2026-04-22T10:10:36.463Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:35.458773697 +0000 UTC m=+81.482569788 (1 nodes), 692.560636ms from start of poll, 692.972228ms from cluster start. Timeout at 2026-04-22 10:10:36.463775067 +0000 UTC m=+82.487571204, 1.005001416s after stability 2026-04-22T10:10:36.464Z [INFO] creating new snapshot: path=/tmp/raft2679765030/snapshots/2-1013-1776852636464.tmp 2026-04-22T10:10:36.464Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: copied to local snapshot: bytes=71 2026-04-22T10:10:36.464Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: snapshot restore progress: id=2-1013-1776852636464 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:36.464Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: removing all old logs from log store 2026-04-22T10:10:36.464Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: compacting logs: from=1 to=2 2026-04-22T10:10:36.464Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: restored user snapshot: index=1013 2026-04-22T10:10:36.464Z [ERROR] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: failed to get log: index=2 error="log not found" 2026-04-22T10:10:36.464Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: aborting pipeline replication: peer="{Voter server-8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988 8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988}" 2026-04-22T10:10:36.464Z [ERROR] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: failed to get log: index=2 error="log not found" 2026-04-22T10:10:36.464Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: aborting pipeline replication: peer="{Voter server-81157623-98e1-92d4-c3b3-80583eb268f7 81157623-98e1-92d4-c3b3-80583eb268f7}" 2026-04-22T10:10:36.472Z [ERROR] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: failed to get log: index=2 error="log not found" 2026-04-22T10:10:36.472Z [INFO] creating new snapshot: path=/tmp/raft1717177076/snapshots/2-1013-1776852636472.tmp 2026-04-22T10:10:36.473Z [INFO] server-8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:36.473Z [INFO] server-8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988: copied to local snapshot: bytes=71 2026-04-22T10:10:36.473Z [INFO] server-8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988: snapshot restore progress: id=2-1013-1776852636472 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:36.473Z [INFO] server-8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988: removing all old logs from log store 2026-04-22T10:10:36.473Z [INFO] server-8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988: compacting logs: from=1 to=2 2026-04-22T10:10:36.473Z [INFO] server-8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988: Installed remote snapshot 2026-04-22T10:10:36.473Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: pipelining replication: peer="{Voter server-8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988 8b9d8cd2-e8dd-b436-e2f3-d207e2fc8988}" 2026-04-22T10:10:36.473Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:36.474Z [ERROR] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: failed to get log: index=2 error="log not found" 2026-04-22T10:10:36.475Z [INFO] creating new snapshot: path=/tmp/raft1709363962/snapshots/2-1013-1776852636475.tmp 2026-04-22T10:10:36.475Z [INFO] server-81157623-98e1-92d4-c3b3-80583eb268f7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:36.476Z [INFO] server-81157623-98e1-92d4-c3b3-80583eb268f7: copied to local snapshot: bytes=71 2026-04-22T10:10:36.476Z [INFO] server-81157623-98e1-92d4-c3b3-80583eb268f7: snapshot restore progress: id=2-1013-1776852636475 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:36.476Z [INFO] server-81157623-98e1-92d4-c3b3-80583eb268f7: removing all old logs from log store 2026-04-22T10:10:36.476Z [INFO] server-81157623-98e1-92d4-c3b3-80583eb268f7: compacting logs: from=1 to=2 2026-04-22T10:10:36.476Z [INFO] server-81157623-98e1-92d4-c3b3-80583eb268f7: Installed remote snapshot 2026-04-22T10:10:36.476Z [INFO] server-63f5c036-ea37-45b4-ccbb-8c472e7db78f: pipelining replication: peer="{Voter server-81157623-98e1-92d4-c3b3-80583eb268f7 81157623-98e1-92d4-c3b3-80583eb268f7}" 2026-04-22T10:10:37.478Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:36.473755961 +0000 UTC m=+82.497552047 (2 nodes), 612ns from start of poll, 1.707954487s from cluster start. Timeout at 2026-04-22 10:10:37.478775223 +0000 UTC m=+83.502571328, 1.005019281s after stability 2026-04-22T10:10:37.490Z [INFO] server-f08abc32-a7e8-0985-6a9f-52928273156f: aborting pipeline replication: peer="{Voter server-ad91fa6a-c384-fb2d-ecd5-d4de76a404b0 ad91fa6a-c384-fb2d-ecd5-d4de76a404b0}" 2026-04-22T10:10:37.490Z [INFO] server-f08abc32-a7e8-0985-6a9f-52928273156f: aborting pipeline replication: peer="{Voter server-63c00e54-84bd-a944-ed8e-85e1f6e1a860 63c00e54-84bd-a944-ed8e-85e1f6e1a860}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false 2026-04-22T10:10:37.492Z [INFO] server-31d596f7-e985-37c8-1baf-8a4e413a2803: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31d596f7-e985-37c8-1baf-8a4e413a2803 Address:31d596f7-e985-37c8-1baf-8a4e413a2803} {Suffrage:Voter ID:server-abf15593-c66d-f22e-48cb-c85ff73f494a Address:abf15593-c66d-f22e-48cb-c85ff73f494a} {Suffrage:Voter ID:server-4c1aac57-541c-bd52-df89-5fc87a5a399c Address:4c1aac57-541c-bd52-df89-5fc87a5a399c}]" 2026-04-22T10:10:37.492Z [INFO] server-31d596f7-e985-37c8-1baf-8a4e413a2803: entering follower state: follower="Node at 31d596f7-e985-37c8-1baf-8a4e413a2803 [Follower]" leader-address= leader-id= 2026-04-22T10:10:37.492Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31d596f7-e985-37c8-1baf-8a4e413a2803 Address:31d596f7-e985-37c8-1baf-8a4e413a2803} {Suffrage:Voter ID:server-abf15593-c66d-f22e-48cb-c85ff73f494a Address:abf15593-c66d-f22e-48cb-c85ff73f494a} {Suffrage:Voter ID:server-4c1aac57-541c-bd52-df89-5fc87a5a399c Address:4c1aac57-541c-bd52-df89-5fc87a5a399c}]" 2026-04-22T10:10:37.492Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: entering follower state: follower="Node at abf15593-c66d-f22e-48cb-c85ff73f494a [Follower]" leader-address= leader-id= 2026-04-22T10:10:37.493Z [INFO] server-4c1aac57-541c-bd52-df89-5fc87a5a399c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31d596f7-e985-37c8-1baf-8a4e413a2803 Address:31d596f7-e985-37c8-1baf-8a4e413a2803} {Suffrage:Voter ID:server-abf15593-c66d-f22e-48cb-c85ff73f494a Address:abf15593-c66d-f22e-48cb-c85ff73f494a} {Suffrage:Voter ID:server-4c1aac57-541c-bd52-df89-5fc87a5a399c Address:4c1aac57-541c-bd52-df89-5fc87a5a399c}]" 2026-04-22T10:10:37.493Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:37.493Z [INFO] server-4c1aac57-541c-bd52-df89-5fc87a5a399c: entering follower state: follower="Node at 4c1aac57-541c-bd52-df89-5fc87a5a399c [Follower]" leader-address= leader-id= 2026-04-22T10:10:38.140Z [WARN] server-abf15593-c66d-f22e-48cb-c85ff73f494a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:38.140Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: entering candidate state: node="Node at abf15593-c66d-f22e-48cb-c85ff73f494a [Candidate]" term=2 2026-04-22T10:10:38.140Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: election won: term=2 tally=2 2026-04-22T10:10:38.140Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: entering leader state: leader="Node at abf15593-c66d-f22e-48cb-c85ff73f494a [Leader]" 2026-04-22T10:10:38.140Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: added peer, starting replication: peer=server-31d596f7-e985-37c8-1baf-8a4e413a2803 2026-04-22T10:10:38.140Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: added peer, starting replication: peer=server-4c1aac57-541c-bd52-df89-5fc87a5a399c 2026-04-22T10:10:38.140Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: pipelining replication: peer="{Voter server-31d596f7-e985-37c8-1baf-8a4e413a2803 31d596f7-e985-37c8-1baf-8a4e413a2803}" 2026-04-22T10:10:38.140Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: pipelining replication: peer="{Voter server-4c1aac57-541c-bd52-df89-5fc87a5a399c 4c1aac57-541c-bd52-df89-5fc87a5a399c}" 2026-04-22T10:10:39.145Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:38.140307031 +0000 UTC m=+84.164103124 (1 nodes), 647.235162ms from start of poll, 648.113243ms from cluster start. Timeout at 2026-04-22 10:10:39.145312763 +0000 UTC m=+85.169108844, 1.00500572s after stability 2026-04-22T10:10:39.145Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: starting snapshot up to: index=12 2026-04-22T10:10:39.145Z [INFO] creating new snapshot: path=/tmp/raft571035003/snapshots/2-12-1776852639145.tmp 2026-04-22T10:10:39.146Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: snapshot complete up to: index=12 2026-04-22T10:10:39.146Z [INFO] creating new snapshot: path=/tmp/raft571035003/snapshots/2-10013-1776852639146.tmp 2026-04-22T10:10:39.147Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: copied to local snapshot: bytes=71 2026-04-22T10:10:39.147Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: snapshot restore progress: id=2-10013-1776852639146 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:39.147Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: restored user snapshot: index=10013 2026-04-22T10:10:39.147Z [ERROR] server-abf15593-c66d-f22e-48cb-c85ff73f494a: failed to get log: index=23 error="log not found" 2026-04-22T10:10:39.147Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: aborting pipeline replication: peer="{Voter server-4c1aac57-541c-bd52-df89-5fc87a5a399c 4c1aac57-541c-bd52-df89-5fc87a5a399c}" 2026-04-22T10:10:39.147Z [ERROR] server-abf15593-c66d-f22e-48cb-c85ff73f494a: failed to get log: index=23 error="log not found" 2026-04-22T10:10:39.147Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: aborting pipeline replication: peer="{Voter server-31d596f7-e985-37c8-1baf-8a4e413a2803 31d596f7-e985-37c8-1baf-8a4e413a2803}" 2026-04-22T10:10:39.154Z [ERROR] server-abf15593-c66d-f22e-48cb-c85ff73f494a: failed to get log: index=23 error="log not found" 2026-04-22T10:10:39.154Z [INFO] creating new snapshot: path=/tmp/raft594710046/snapshots/2-10013-1776852639154.tmp 2026-04-22T10:10:39.154Z [INFO] server-31d596f7-e985-37c8-1baf-8a4e413a2803: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:39.155Z [INFO] server-31d596f7-e985-37c8-1baf-8a4e413a2803: copied to local snapshot: bytes=71 2026-04-22T10:10:39.155Z [INFO] server-31d596f7-e985-37c8-1baf-8a4e413a2803: snapshot restore progress: id=2-10013-1776852639154 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:39.155Z [INFO] server-31d596f7-e985-37c8-1baf-8a4e413a2803: Installed remote snapshot 2026-04-22T10:10:39.155Z [WARN] server-31d596f7-e985-37c8-1baf-8a4e413a2803: skipping application of old log: index=22 2026-04-22T10:10:39.155Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: pipelining replication: peer="{Voter server-31d596f7-e985-37c8-1baf-8a4e413a2803 31d596f7-e985-37c8-1baf-8a4e413a2803}" 2026-04-22T10:10:39.155Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:39.156Z [ERROR] server-abf15593-c66d-f22e-48cb-c85ff73f494a: failed to get log: index=23 error="log not found" 2026-04-22T10:10:39.156Z [INFO] creating new snapshot: path=/tmp/raft1956719663/snapshots/2-10013-1776852639156.tmp 2026-04-22T10:10:39.156Z [INFO] server-4c1aac57-541c-bd52-df89-5fc87a5a399c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:39.157Z [INFO] server-4c1aac57-541c-bd52-df89-5fc87a5a399c: copied to local snapshot: bytes=71 2026-04-22T10:10:39.157Z [INFO] server-4c1aac57-541c-bd52-df89-5fc87a5a399c: snapshot restore progress: id=2-10013-1776852639156 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:39.157Z [INFO] server-4c1aac57-541c-bd52-df89-5fc87a5a399c: Installed remote snapshot 2026-04-22T10:10:39.157Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: pipelining replication: peer="{Voter server-4c1aac57-541c-bd52-df89-5fc87a5a399c 4c1aac57-541c-bd52-df89-5fc87a5a399c}" 2026-04-22T10:10:40.161Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:39.155553594 +0000 UTC m=+85.179349692 (2 nodes), 1.085µs from start of poll, 1.663359811s from cluster start. Timeout at 2026-04-22 10:10:40.160587306 +0000 UTC m=+86.184383421, 1.005033729s after stability 2026-04-22T10:10:40.177Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: aborting pipeline replication: peer="{Voter server-31d596f7-e985-37c8-1baf-8a4e413a2803 31d596f7-e985-37c8-1baf-8a4e413a2803}" 2026-04-22T10:10:40.177Z [INFO] server-abf15593-c66d-f22e-48cb-c85ff73f494a: aborting pipeline replication: peer="{Voter server-4c1aac57-541c-bd52-df89-5fc87a5a399c 4c1aac57-541c-bd52-df89-5fc87a5a399c}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false 2026-04-22T10:10:40.178Z [INFO] server-37346471-9e74-b1cb-0cb6-0d343376117e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37346471-9e74-b1cb-0cb6-0d343376117e Address:37346471-9e74-b1cb-0cb6-0d343376117e} {Suffrage:Voter ID:server-a25f9558-a32c-26a8-7e96-7570f4c7966a Address:a25f9558-a32c-26a8-7e96-7570f4c7966a} {Suffrage:Voter ID:server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6 Address:f1715582-af1c-1f4d-40ae-b6b574ddd4e6}]" 2026-04-22T10:10:40.178Z [INFO] server-37346471-9e74-b1cb-0cb6-0d343376117e: entering follower state: follower="Node at 37346471-9e74-b1cb-0cb6-0d343376117e [Follower]" leader-address= leader-id= 2026-04-22T10:10:40.179Z [INFO] server-a25f9558-a32c-26a8-7e96-7570f4c7966a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37346471-9e74-b1cb-0cb6-0d343376117e Address:37346471-9e74-b1cb-0cb6-0d343376117e} {Suffrage:Voter ID:server-a25f9558-a32c-26a8-7e96-7570f4c7966a Address:a25f9558-a32c-26a8-7e96-7570f4c7966a} {Suffrage:Voter ID:server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6 Address:f1715582-af1c-1f4d-40ae-b6b574ddd4e6}]" 2026-04-22T10:10:40.179Z [INFO] server-a25f9558-a32c-26a8-7e96-7570f4c7966a: entering follower state: follower="Node at a25f9558-a32c-26a8-7e96-7570f4c7966a [Follower]" leader-address= leader-id= 2026-04-22T10:10:40.179Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37346471-9e74-b1cb-0cb6-0d343376117e Address:37346471-9e74-b1cb-0cb6-0d343376117e} {Suffrage:Voter ID:server-a25f9558-a32c-26a8-7e96-7570f4c7966a Address:a25f9558-a32c-26a8-7e96-7570f4c7966a} {Suffrage:Voter ID:server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6 Address:f1715582-af1c-1f4d-40ae-b6b574ddd4e6}]" 2026-04-22T10:10:40.179Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:40.179Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: entering follower state: follower="Node at f1715582-af1c-1f4d-40ae-b6b574ddd4e6 [Follower]" leader-address= leader-id= 2026-04-22T10:10:40.937Z [WARN] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:40.937Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: entering candidate state: node="Node at f1715582-af1c-1f4d-40ae-b6b574ddd4e6 [Candidate]" term=2 2026-04-22T10:10:40.937Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: election won: term=2 tally=2 2026-04-22T10:10:40.937Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: entering leader state: leader="Node at f1715582-af1c-1f4d-40ae-b6b574ddd4e6 [Leader]" 2026-04-22T10:10:40.937Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: added peer, starting replication: peer=server-37346471-9e74-b1cb-0cb6-0d343376117e 2026-04-22T10:10:40.937Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: added peer, starting replication: peer=server-a25f9558-a32c-26a8-7e96-7570f4c7966a 2026-04-22T10:10:40.937Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: pipelining replication: peer="{Voter server-37346471-9e74-b1cb-0cb6-0d343376117e 37346471-9e74-b1cb-0cb6-0d343376117e}" 2026-04-22T10:10:40.938Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: pipelining replication: peer="{Voter server-a25f9558-a32c-26a8-7e96-7570f4c7966a a25f9558-a32c-26a8-7e96-7570f4c7966a}" 2026-04-22T10:10:41.943Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:40.937932904 +0000 UTC m=+86.961728992 (1 nodes), 758.77344ms from start of poll, 759.078169ms from cluster start. Timeout at 2026-04-22 10:10:41.942934769 +0000 UTC m=+87.966730905, 1.005001913s after stability 2026-04-22T10:10:41.944Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: starting snapshot up to: index=12 2026-04-22T10:10:41.944Z [INFO] creating new snapshot: path=/tmp/raft1111434963/snapshots/2-12-1776852641944.tmp 2026-04-22T10:10:41.944Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: snapshot complete up to: index=12 2026-04-22T10:10:41.945Z [INFO] creating new snapshot: path=/tmp/raft1111434963/snapshots/2-10013-1776852641945.tmp 2026-04-22T10:10:41.945Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: copied to local snapshot: bytes=71 2026-04-22T10:10:41.946Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: snapshot restore progress: id=2-10013-1776852641945 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:41.946Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: removing all old logs from log store 2026-04-22T10:10:41.946Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: compacting logs: from=1 to=22 2026-04-22T10:10:41.946Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: restored user snapshot: index=10013 2026-04-22T10:10:41.946Z [ERROR] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: failed to get log: index=22 error="log not found" 2026-04-22T10:10:41.946Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: aborting pipeline replication: peer="{Voter server-37346471-9e74-b1cb-0cb6-0d343376117e 37346471-9e74-b1cb-0cb6-0d343376117e}" 2026-04-22T10:10:41.946Z [ERROR] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: failed to get log: index=22 error="log not found" 2026-04-22T10:10:41.946Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: aborting pipeline replication: peer="{Voter server-a25f9558-a32c-26a8-7e96-7570f4c7966a a25f9558-a32c-26a8-7e96-7570f4c7966a}" 2026-04-22T10:10:41.952Z [ERROR] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: failed to get log: index=22 error="log not found" 2026-04-22T10:10:41.953Z [INFO] creating new snapshot: path=/tmp/raft3715421644/snapshots/2-10013-1776852641953.tmp 2026-04-22T10:10:41.953Z [INFO] server-37346471-9e74-b1cb-0cb6-0d343376117e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:41.953Z [INFO] server-37346471-9e74-b1cb-0cb6-0d343376117e: copied to local snapshot: bytes=71 2026-04-22T10:10:41.954Z [INFO] server-37346471-9e74-b1cb-0cb6-0d343376117e: snapshot restore progress: id=2-10013-1776852641953 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:41.954Z [INFO] server-37346471-9e74-b1cb-0cb6-0d343376117e: removing all old logs from log store 2026-04-22T10:10:41.954Z [INFO] server-37346471-9e74-b1cb-0cb6-0d343376117e: compacting logs: from=1 to=22 2026-04-22T10:10:41.954Z [INFO] server-37346471-9e74-b1cb-0cb6-0d343376117e: Installed remote snapshot 2026-04-22T10:10:41.954Z [WARN] server-37346471-9e74-b1cb-0cb6-0d343376117e: skipping application of old log: index=22 2026-04-22T10:10:41.954Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: pipelining replication: peer="{Voter server-37346471-9e74-b1cb-0cb6-0d343376117e 37346471-9e74-b1cb-0cb6-0d343376117e}" 2026-04-22T10:10:41.954Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:41.957Z [ERROR] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: failed to get log: index=22 error="log not found" 2026-04-22T10:10:41.958Z [INFO] creating new snapshot: path=/tmp/raft210572504/snapshots/2-10013-1776852641958.tmp 2026-04-22T10:10:41.958Z [INFO] server-a25f9558-a32c-26a8-7e96-7570f4c7966a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:41.958Z [INFO] server-a25f9558-a32c-26a8-7e96-7570f4c7966a: copied to local snapshot: bytes=71 2026-04-22T10:10:41.958Z [INFO] server-a25f9558-a32c-26a8-7e96-7570f4c7966a: snapshot restore progress: id=2-10013-1776852641958 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:41.958Z [INFO] server-a25f9558-a32c-26a8-7e96-7570f4c7966a: removing all old logs from log store 2026-04-22T10:10:41.958Z [INFO] server-a25f9558-a32c-26a8-7e96-7570f4c7966a: compacting logs: from=1 to=22 2026-04-22T10:10:41.958Z [INFO] server-a25f9558-a32c-26a8-7e96-7570f4c7966a: Installed remote snapshot 2026-04-22T10:10:41.958Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: pipelining replication: peer="{Voter server-a25f9558-a32c-26a8-7e96-7570f4c7966a a25f9558-a32c-26a8-7e96-7570f4c7966a}" 2026-04-22T10:10:42.960Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:41.954355363 +0000 UTC m=+87.978151452 (2 nodes), 687ns from start of poll, 1.775500629s from cluster start. Timeout at 2026-04-22 10:10:42.959375143 +0000 UTC m=+88.983171333, 1.005019881s after stability 2026-04-22T10:10:42.971Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: aborting pipeline replication: peer="{Voter server-a25f9558-a32c-26a8-7e96-7570f4c7966a a25f9558-a32c-26a8-7e96-7570f4c7966a}" 2026-04-22T10:10:42.971Z [INFO] server-f1715582-af1c-1f4d-40ae-b6b574ddd4e6: aborting pipeline replication: peer="{Voter server-37346471-9e74-b1cb-0cb6-0d343376117e 37346471-9e74-b1cb-0cb6-0d343376117e}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true 2026-04-22T10:10:42.972Z [INFO] server-74f249c8-e935-8345-8467-911569067747: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-74f249c8-e935-8345-8467-911569067747 Address:74f249c8-e935-8345-8467-911569067747} {Suffrage:Voter ID:server-443b1416-14d3-8558-0989-055ac897e976 Address:443b1416-14d3-8558-0989-055ac897e976} {Suffrage:Voter ID:server-c5e5714b-11dd-6633-9f56-4e373ebb14c3 Address:c5e5714b-11dd-6633-9f56-4e373ebb14c3}]" 2026-04-22T10:10:42.972Z [INFO] server-74f249c8-e935-8345-8467-911569067747: entering follower state: follower="Node at 74f249c8-e935-8345-8467-911569067747 [Follower]" leader-address= leader-id= 2026-04-22T10:10:42.972Z [INFO] server-443b1416-14d3-8558-0989-055ac897e976: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-74f249c8-e935-8345-8467-911569067747 Address:74f249c8-e935-8345-8467-911569067747} {Suffrage:Voter ID:server-443b1416-14d3-8558-0989-055ac897e976 Address:443b1416-14d3-8558-0989-055ac897e976} {Suffrage:Voter ID:server-c5e5714b-11dd-6633-9f56-4e373ebb14c3 Address:c5e5714b-11dd-6633-9f56-4e373ebb14c3}]" 2026-04-22T10:10:42.972Z [INFO] server-443b1416-14d3-8558-0989-055ac897e976: entering follower state: follower="Node at 443b1416-14d3-8558-0989-055ac897e976 [Follower]" leader-address= leader-id= 2026-04-22T10:10:42.972Z [INFO] server-c5e5714b-11dd-6633-9f56-4e373ebb14c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-74f249c8-e935-8345-8467-911569067747 Address:74f249c8-e935-8345-8467-911569067747} {Suffrage:Voter ID:server-443b1416-14d3-8558-0989-055ac897e976 Address:443b1416-14d3-8558-0989-055ac897e976} {Suffrage:Voter ID:server-c5e5714b-11dd-6633-9f56-4e373ebb14c3 Address:c5e5714b-11dd-6633-9f56-4e373ebb14c3}]" 2026-04-22T10:10:42.973Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:42.973Z [INFO] server-c5e5714b-11dd-6633-9f56-4e373ebb14c3: entering follower state: follower="Node at c5e5714b-11dd-6633-9f56-4e373ebb14c3 [Follower]" leader-address= leader-id= 2026-04-22T10:10:43.551Z [WARN] server-443b1416-14d3-8558-0989-055ac897e976: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:43.551Z [INFO] server-443b1416-14d3-8558-0989-055ac897e976: entering candidate state: node="Node at 443b1416-14d3-8558-0989-055ac897e976 [Candidate]" term=2 2026-04-22T10:10:43.551Z [INFO] server-443b1416-14d3-8558-0989-055ac897e976: election won: term=2 tally=2 2026-04-22T10:10:43.551Z [INFO] server-443b1416-14d3-8558-0989-055ac897e976: entering leader state: leader="Node at 443b1416-14d3-8558-0989-055ac897e976 [Leader]" 2026-04-22T10:10:43.551Z [INFO] server-443b1416-14d3-8558-0989-055ac897e976: added peer, starting replication: peer=server-74f249c8-e935-8345-8467-911569067747 2026-04-22T10:10:43.551Z [INFO] server-443b1416-14d3-8558-0989-055ac897e976: added peer, starting replication: peer=server-c5e5714b-11dd-6633-9f56-4e373ebb14c3 2026-04-22T10:10:43.551Z [INFO] server-443b1416-14d3-8558-0989-055ac897e976: pipelining replication: peer="{Voter server-74f249c8-e935-8345-8467-911569067747 74f249c8-e935-8345-8467-911569067747}" 2026-04-22T10:10:43.551Z [INFO] server-443b1416-14d3-8558-0989-055ac897e976: pipelining replication: peer="{Voter server-c5e5714b-11dd-6633-9f56-4e373ebb14c3 c5e5714b-11dd-6633-9f56-4e373ebb14c3}" 2026-04-22T10:10:44.557Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:43.551621209 +0000 UTC m=+89.575417298 (1 nodes), 578.605051ms from start of poll, 578.855412ms from cluster start. Timeout at 2026-04-22 10:10:44.556623603 +0000 UTC m=+90.580419684, 1.005002386s after stability 2026-04-22T10:10:44.557Z [INFO] server-443b1416-14d3-8558-0989-055ac897e976: starting snapshot up to: index=12 2026-04-22T10:10:44.557Z [INFO] creating new snapshot: path=/tmp/raft1634090500/snapshots/2-12-1776852644557.tmp 2026-04-22T10:10:44.558Z [INFO] server-443b1416-14d3-8558-0989-055ac897e976: snapshot complete up to: index=12 2026-04-22T10:10:44.559Z [INFO] server-3c500fa8-e4d3-a9b3-7919-005ff7e42c45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c500fa8-e4d3-a9b3-7919-005ff7e42c45 Address:3c500fa8-e4d3-a9b3-7919-005ff7e42c45} {Suffrage:Voter ID:server-1a378a13-bc14-6333-bf01-60a05f46d57c Address:1a378a13-bc14-6333-bf01-60a05f46d57c} {Suffrage:Voter ID:server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2 Address:65ab38b4-c201-2589-5555-dcbb8e7c2ea2}]" 2026-04-22T10:10:44.559Z [INFO] server-3c500fa8-e4d3-a9b3-7919-005ff7e42c45: entering follower state: follower="Node at 3c500fa8-e4d3-a9b3-7919-005ff7e42c45 [Follower]" leader-address= leader-id= 2026-04-22T10:10:44.559Z [INFO] server-1a378a13-bc14-6333-bf01-60a05f46d57c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c500fa8-e4d3-a9b3-7919-005ff7e42c45 Address:3c500fa8-e4d3-a9b3-7919-005ff7e42c45} {Suffrage:Voter ID:server-1a378a13-bc14-6333-bf01-60a05f46d57c Address:1a378a13-bc14-6333-bf01-60a05f46d57c} {Suffrage:Voter ID:server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2 Address:65ab38b4-c201-2589-5555-dcbb8e7c2ea2}]" 2026-04-22T10:10:44.559Z [INFO] server-1a378a13-bc14-6333-bf01-60a05f46d57c: entering follower state: follower="Node at 1a378a13-bc14-6333-bf01-60a05f46d57c [Follower]" leader-address= leader-id= 2026-04-22T10:10:44.559Z [INFO] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c500fa8-e4d3-a9b3-7919-005ff7e42c45 Address:3c500fa8-e4d3-a9b3-7919-005ff7e42c45} {Suffrage:Voter ID:server-1a378a13-bc14-6333-bf01-60a05f46d57c Address:1a378a13-bc14-6333-bf01-60a05f46d57c} {Suffrage:Voter ID:server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2 Address:65ab38b4-c201-2589-5555-dcbb8e7c2ea2}]" 2026-04-22T10:10:44.559Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:44.559Z [INFO] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: entering follower state: follower="Node at 65ab38b4-c201-2589-5555-dcbb8e7c2ea2 [Follower]" leader-address= leader-id= 2026-04-22T10:10:45.083Z [WARN] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:45.083Z [INFO] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: entering candidate state: node="Node at 65ab38b4-c201-2589-5555-dcbb8e7c2ea2 [Candidate]" term=2 2026-04-22T10:10:45.083Z [INFO] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: election won: term=2 tally=2 2026-04-22T10:10:45.083Z [INFO] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: entering leader state: leader="Node at 65ab38b4-c201-2589-5555-dcbb8e7c2ea2 [Leader]" 2026-04-22T10:10:45.083Z [INFO] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: added peer, starting replication: peer=server-3c500fa8-e4d3-a9b3-7919-005ff7e42c45 2026-04-22T10:10:45.083Z [INFO] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: added peer, starting replication: peer=server-1a378a13-bc14-6333-bf01-60a05f46d57c 2026-04-22T10:10:45.083Z [INFO] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: pipelining replication: peer="{Voter server-1a378a13-bc14-6333-bf01-60a05f46d57c 1a378a13-bc14-6333-bf01-60a05f46d57c}" 2026-04-22T10:10:45.083Z [INFO] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: pipelining replication: peer="{Voter server-3c500fa8-e4d3-a9b3-7919-005ff7e42c45 3c500fa8-e4d3-a9b3-7919-005ff7e42c45}" 2026-04-22T10:10:46.088Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:45.083346273 +0000 UTC m=+91.107142362 (1 nodes), 523.987539ms from start of poll, 524.293942ms from cluster start. Timeout at 2026-04-22 10:10:46.088355301 +0000 UTC m=+92.112151438, 1.005009076s after stability 2026-04-22T10:10:46.088Z [INFO] creating new snapshot: path=/tmp/raft2779065373/snapshots/2-10013-1776852646088.tmp 2026-04-22T10:10:46.089Z [INFO] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: copied to local snapshot: bytes=71 2026-04-22T10:10:46.089Z [INFO] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: snapshot restore progress: id=2-10013-1776852646088 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:46.089Z [INFO] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: restored user snapshot: index=10013 2026-04-22T10:10:46.089Z [ERROR] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: failed to get log: index=3 error="log not found" 2026-04-22T10:10:46.089Z [INFO] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: aborting pipeline replication: peer="{Voter server-3c500fa8-e4d3-a9b3-7919-005ff7e42c45 3c500fa8-e4d3-a9b3-7919-005ff7e42c45}" 2026-04-22T10:10:46.089Z [ERROR] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: failed to get log: index=3 error="log not found" 2026-04-22T10:10:46.089Z [INFO] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: aborting pipeline replication: peer="{Voter server-1a378a13-bc14-6333-bf01-60a05f46d57c 1a378a13-bc14-6333-bf01-60a05f46d57c}" 2026-04-22T10:10:46.097Z [ERROR] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: failed to get log: index=3 error="log not found" 2026-04-22T10:10:46.097Z [INFO] creating new snapshot: path=/tmp/raft1277979547/snapshots/2-10013-1776852646097.tmp 2026-04-22T10:10:46.098Z [INFO] server-1a378a13-bc14-6333-bf01-60a05f46d57c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:46.098Z [INFO] server-1a378a13-bc14-6333-bf01-60a05f46d57c: copied to local snapshot: bytes=71 2026-04-22T10:10:46.098Z [INFO] server-1a378a13-bc14-6333-bf01-60a05f46d57c: snapshot restore progress: id=2-10013-1776852646097 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:46.098Z [INFO] server-1a378a13-bc14-6333-bf01-60a05f46d57c: Installed remote snapshot 2026-04-22T10:10:46.098Z [INFO] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: pipelining replication: peer="{Voter server-1a378a13-bc14-6333-bf01-60a05f46d57c 1a378a13-bc14-6333-bf01-60a05f46d57c}" 2026-04-22T10:10:46.098Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:46.099Z [ERROR] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: failed to get log: index=3 error="log not found" 2026-04-22T10:10:46.100Z [INFO] creating new snapshot: path=/tmp/raft3895872293/snapshots/2-10013-1776852646100.tmp 2026-04-22T10:10:46.100Z [INFO] server-3c500fa8-e4d3-a9b3-7919-005ff7e42c45: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:46.100Z [INFO] server-3c500fa8-e4d3-a9b3-7919-005ff7e42c45: copied to local snapshot: bytes=71 2026-04-22T10:10:46.101Z [INFO] server-3c500fa8-e4d3-a9b3-7919-005ff7e42c45: snapshot restore progress: id=2-10013-1776852646100 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:46.101Z [INFO] server-3c500fa8-e4d3-a9b3-7919-005ff7e42c45: Installed remote snapshot 2026-04-22T10:10:46.101Z [INFO] server-65ab38b4-c201-2589-5555-dcbb8e7c2ea2: pipelining replication: peer="{Voter server-3c500fa8-e4d3-a9b3-7919-005ff7e42c45 3c500fa8-e4d3-a9b3-7919-005ff7e42c45}" 2026-04-22T10:10:47.103Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:46.098738324 +0000 UTC m=+92.122534461 (2 nodes), 1.068µs from start of poll, 1.539686041s from cluster start. Timeout at 2026-04-22 10:10:47.103742241 +0000 UTC m=+93.127538347, 1.005003886s after stability 2026-04-22T10:10:47.115Z [INFO] server-443b1416-14d3-8558-0989-055ac897e976: aborting pipeline replication: peer="{Voter server-c5e5714b-11dd-6633-9f56-4e373ebb14c3 c5e5714b-11dd-6633-9f56-4e373ebb14c3}" 2026-04-22T10:10:47.115Z [INFO] server-443b1416-14d3-8558-0989-055ac897e976: aborting pipeline replication: peer="{Voter server-74f249c8-e935-8345-8467-911569067747 74f249c8-e935-8345-8467-911569067747}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true 2026-04-22T10:10:47.116Z [INFO] server-f8154d60-a386-e233-83d9-9476b3ab25da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8154d60-a386-e233-83d9-9476b3ab25da Address:f8154d60-a386-e233-83d9-9476b3ab25da} {Suffrage:Voter ID:server-1b83c46c-5f2c-7d10-7fd5-24f75fc8daef Address:1b83c46c-5f2c-7d10-7fd5-24f75fc8daef} {Suffrage:Voter ID:server-9916af56-683b-66aa-86e8-0651da84b291 Address:9916af56-683b-66aa-86e8-0651da84b291}]" 2026-04-22T10:10:47.116Z [INFO] server-f8154d60-a386-e233-83d9-9476b3ab25da: entering follower state: follower="Node at f8154d60-a386-e233-83d9-9476b3ab25da [Follower]" leader-address= leader-id= 2026-04-22T10:10:47.116Z [INFO] server-1b83c46c-5f2c-7d10-7fd5-24f75fc8daef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8154d60-a386-e233-83d9-9476b3ab25da Address:f8154d60-a386-e233-83d9-9476b3ab25da} {Suffrage:Voter ID:server-1b83c46c-5f2c-7d10-7fd5-24f75fc8daef Address:1b83c46c-5f2c-7d10-7fd5-24f75fc8daef} {Suffrage:Voter ID:server-9916af56-683b-66aa-86e8-0651da84b291 Address:9916af56-683b-66aa-86e8-0651da84b291}]" 2026-04-22T10:10:47.116Z [INFO] server-1b83c46c-5f2c-7d10-7fd5-24f75fc8daef: entering follower state: follower="Node at 1b83c46c-5f2c-7d10-7fd5-24f75fc8daef [Follower]" leader-address= leader-id= 2026-04-22T10:10:47.116Z [INFO] server-9916af56-683b-66aa-86e8-0651da84b291: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8154d60-a386-e233-83d9-9476b3ab25da Address:f8154d60-a386-e233-83d9-9476b3ab25da} {Suffrage:Voter ID:server-1b83c46c-5f2c-7d10-7fd5-24f75fc8daef Address:1b83c46c-5f2c-7d10-7fd5-24f75fc8daef} {Suffrage:Voter ID:server-9916af56-683b-66aa-86e8-0651da84b291 Address:9916af56-683b-66aa-86e8-0651da84b291}]" 2026-04-22T10:10:47.116Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:47.116Z [INFO] server-9916af56-683b-66aa-86e8-0651da84b291: entering follower state: follower="Node at 9916af56-683b-66aa-86e8-0651da84b291 [Follower]" leader-address= leader-id= 2026-04-22T10:10:47.629Z [WARN] server-f8154d60-a386-e233-83d9-9476b3ab25da: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:47.629Z [INFO] server-f8154d60-a386-e233-83d9-9476b3ab25da: entering candidate state: node="Node at f8154d60-a386-e233-83d9-9476b3ab25da [Candidate]" term=2 2026-04-22T10:10:47.629Z [INFO] server-f8154d60-a386-e233-83d9-9476b3ab25da: election won: term=2 tally=2 2026-04-22T10:10:47.629Z [INFO] server-f8154d60-a386-e233-83d9-9476b3ab25da: entering leader state: leader="Node at f8154d60-a386-e233-83d9-9476b3ab25da [Leader]" 2026-04-22T10:10:47.629Z [INFO] server-f8154d60-a386-e233-83d9-9476b3ab25da: added peer, starting replication: peer=server-1b83c46c-5f2c-7d10-7fd5-24f75fc8daef 2026-04-22T10:10:47.629Z [INFO] server-f8154d60-a386-e233-83d9-9476b3ab25da: added peer, starting replication: peer=server-9916af56-683b-66aa-86e8-0651da84b291 2026-04-22T10:10:47.630Z [INFO] server-f8154d60-a386-e233-83d9-9476b3ab25da: pipelining replication: peer="{Voter server-9916af56-683b-66aa-86e8-0651da84b291 9916af56-683b-66aa-86e8-0651da84b291}" 2026-04-22T10:10:47.630Z [INFO] server-f8154d60-a386-e233-83d9-9476b3ab25da: pipelining replication: peer="{Voter server-1b83c46c-5f2c-7d10-7fd5-24f75fc8daef 1b83c46c-5f2c-7d10-7fd5-24f75fc8daef}" 2026-04-22T10:10:48.635Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:47.629979707 +0000 UTC m=+93.653775795 (1 nodes), 513.463844ms from start of poll, 513.790615ms from cluster start. Timeout at 2026-04-22 10:10:48.634982067 +0000 UTC m=+94.658778150, 1.005002355s after stability 2026-04-22T10:10:48.636Z [INFO] server-f8154d60-a386-e233-83d9-9476b3ab25da: starting snapshot up to: index=12 2026-04-22T10:10:48.636Z [INFO] creating new snapshot: path=/tmp/raft1258858585/snapshots/2-12-1776852648636.tmp 2026-04-22T10:10:48.636Z [INFO] server-f8154d60-a386-e233-83d9-9476b3ab25da: snapshot complete up to: index=12 2026-04-22T10:10:48.637Z [INFO] server-b1362830-6f1a-5a35-4371-c4ed5a83e34b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1362830-6f1a-5a35-4371-c4ed5a83e34b Address:b1362830-6f1a-5a35-4371-c4ed5a83e34b} {Suffrage:Voter ID:server-9887f76b-e10b-919a-2172-a42816a2d600 Address:9887f76b-e10b-919a-2172-a42816a2d600} {Suffrage:Voter ID:server-e6284e92-4356-237c-64a9-d2a644732297 Address:e6284e92-4356-237c-64a9-d2a644732297}]" 2026-04-22T10:10:48.637Z [INFO] server-b1362830-6f1a-5a35-4371-c4ed5a83e34b: entering follower state: follower="Node at b1362830-6f1a-5a35-4371-c4ed5a83e34b [Follower]" leader-address= leader-id= 2026-04-22T10:10:48.637Z [INFO] server-9887f76b-e10b-919a-2172-a42816a2d600: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1362830-6f1a-5a35-4371-c4ed5a83e34b Address:b1362830-6f1a-5a35-4371-c4ed5a83e34b} {Suffrage:Voter ID:server-9887f76b-e10b-919a-2172-a42816a2d600 Address:9887f76b-e10b-919a-2172-a42816a2d600} {Suffrage:Voter ID:server-e6284e92-4356-237c-64a9-d2a644732297 Address:e6284e92-4356-237c-64a9-d2a644732297}]" 2026-04-22T10:10:48.637Z [INFO] server-9887f76b-e10b-919a-2172-a42816a2d600: entering follower state: follower="Node at 9887f76b-e10b-919a-2172-a42816a2d600 [Follower]" leader-address= leader-id= 2026-04-22T10:10:48.637Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1362830-6f1a-5a35-4371-c4ed5a83e34b Address:b1362830-6f1a-5a35-4371-c4ed5a83e34b} {Suffrage:Voter ID:server-9887f76b-e10b-919a-2172-a42816a2d600 Address:9887f76b-e10b-919a-2172-a42816a2d600} {Suffrage:Voter ID:server-e6284e92-4356-237c-64a9-d2a644732297 Address:e6284e92-4356-237c-64a9-d2a644732297}]" 2026-04-22T10:10:48.637Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:48.637Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: entering follower state: follower="Node at e6284e92-4356-237c-64a9-d2a644732297 [Follower]" leader-address= leader-id= 2026-04-22T10:10:49.352Z [WARN] server-e6284e92-4356-237c-64a9-d2a644732297: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:49.352Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: entering candidate state: node="Node at e6284e92-4356-237c-64a9-d2a644732297 [Candidate]" term=2 2026-04-22T10:10:49.352Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: election won: term=2 tally=2 2026-04-22T10:10:49.352Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: entering leader state: leader="Node at e6284e92-4356-237c-64a9-d2a644732297 [Leader]" 2026-04-22T10:10:49.352Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: added peer, starting replication: peer=server-b1362830-6f1a-5a35-4371-c4ed5a83e34b 2026-04-22T10:10:49.352Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: added peer, starting replication: peer=server-9887f76b-e10b-919a-2172-a42816a2d600 2026-04-22T10:10:49.352Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: pipelining replication: peer="{Voter server-9887f76b-e10b-919a-2172-a42816a2d600 9887f76b-e10b-919a-2172-a42816a2d600}" 2026-04-22T10:10:49.352Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: pipelining replication: peer="{Voter server-b1362830-6f1a-5a35-4371-c4ed5a83e34b b1362830-6f1a-5a35-4371-c4ed5a83e34b}" 2026-04-22T10:10:50.358Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:49.352636971 +0000 UTC m=+95.376433121 (1 nodes), 714.8054ms from start of poll, 715.163109ms from cluster start. Timeout at 2026-04-22 10:10:50.35764784 +0000 UTC m=+96.381443974, 1.005010853s after stability 2026-04-22T10:10:50.358Z [INFO] creating new snapshot: path=/tmp/raft2240097003/snapshots/2-10013-1776852650358.tmp 2026-04-22T10:10:50.358Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: copied to local snapshot: bytes=71 2026-04-22T10:10:50.359Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: snapshot restore progress: id=2-10013-1776852650358 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:50.359Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: removing all old logs from log store 2026-04-22T10:10:50.359Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: compacting logs: from=1 to=2 2026-04-22T10:10:50.359Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: restored user snapshot: index=10013 2026-04-22T10:10:50.359Z [ERROR] server-e6284e92-4356-237c-64a9-d2a644732297: failed to get log: index=2 error="log not found" 2026-04-22T10:10:50.359Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: aborting pipeline replication: peer="{Voter server-b1362830-6f1a-5a35-4371-c4ed5a83e34b b1362830-6f1a-5a35-4371-c4ed5a83e34b}" 2026-04-22T10:10:50.359Z [ERROR] server-e6284e92-4356-237c-64a9-d2a644732297: failed to get log: index=2 error="log not found" 2026-04-22T10:10:50.359Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: aborting pipeline replication: peer="{Voter server-9887f76b-e10b-919a-2172-a42816a2d600 9887f76b-e10b-919a-2172-a42816a2d600}" 2026-04-22T10:10:50.365Z [ERROR] server-e6284e92-4356-237c-64a9-d2a644732297: failed to get log: index=2 error="log not found" 2026-04-22T10:10:50.366Z [INFO] creating new snapshot: path=/tmp/raft1510242677/snapshots/2-10013-1776852650366.tmp 2026-04-22T10:10:50.366Z [INFO] server-b1362830-6f1a-5a35-4371-c4ed5a83e34b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:50.366Z [INFO] server-b1362830-6f1a-5a35-4371-c4ed5a83e34b: copied to local snapshot: bytes=71 2026-04-22T10:10:50.367Z [INFO] server-b1362830-6f1a-5a35-4371-c4ed5a83e34b: snapshot restore progress: id=2-10013-1776852650366 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:50.367Z [INFO] server-b1362830-6f1a-5a35-4371-c4ed5a83e34b: removing all old logs from log store 2026-04-22T10:10:50.367Z [INFO] server-b1362830-6f1a-5a35-4371-c4ed5a83e34b: compacting logs: from=1 to=2 2026-04-22T10:10:50.367Z [INFO] server-b1362830-6f1a-5a35-4371-c4ed5a83e34b: Installed remote snapshot 2026-04-22T10:10:50.367Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: pipelining replication: peer="{Voter server-b1362830-6f1a-5a35-4371-c4ed5a83e34b b1362830-6f1a-5a35-4371-c4ed5a83e34b}" 2026-04-22T10:10:50.367Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:50.369Z [ERROR] server-e6284e92-4356-237c-64a9-d2a644732297: failed to get log: index=2 error="log not found" 2026-04-22T10:10:50.369Z [INFO] creating new snapshot: path=/tmp/raft4061300982/snapshots/2-10013-1776852650369.tmp 2026-04-22T10:10:50.370Z [INFO] server-9887f76b-e10b-919a-2172-a42816a2d600: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:50.370Z [INFO] server-9887f76b-e10b-919a-2172-a42816a2d600: copied to local snapshot: bytes=71 2026-04-22T10:10:50.370Z [INFO] server-9887f76b-e10b-919a-2172-a42816a2d600: snapshot restore progress: id=2-10013-1776852650369 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-04-22T10:10:50.370Z [INFO] server-9887f76b-e10b-919a-2172-a42816a2d600: removing all old logs from log store 2026-04-22T10:10:50.370Z [INFO] server-9887f76b-e10b-919a-2172-a42816a2d600: compacting logs: from=1 to=2 2026-04-22T10:10:50.370Z [INFO] server-9887f76b-e10b-919a-2172-a42816a2d600: Installed remote snapshot 2026-04-22T10:10:50.370Z [INFO] server-e6284e92-4356-237c-64a9-d2a644732297: pipelining replication: peer="{Voter server-9887f76b-e10b-919a-2172-a42816a2d600 9887f76b-e10b-919a-2172-a42816a2d600}" 2026-04-22T10:10:51.373Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:50.367839391 +0000 UTC m=+96.391635479 (2 nodes), 619ns from start of poll, 1.730365467s from cluster start. Timeout at 2026-04-22 10:10:51.37286229 +0000 UTC m=+97.396658474, 1.005022995s after stability 2026-04-22T10:10:51.385Z [INFO] server-f8154d60-a386-e233-83d9-9476b3ab25da: aborting pipeline replication: peer="{Voter server-9916af56-683b-66aa-86e8-0651da84b291 9916af56-683b-66aa-86e8-0651da84b291}" 2026-04-22T10:10:51.385Z [INFO] server-f8154d60-a386-e233-83d9-9476b3ab25da: aborting pipeline replication: peer="{Voter server-1b83c46c-5f2c-7d10-7fd5-24f75fc8daef 1b83c46c-5f2c-7d10-7fd5-24f75fc8daef}" --- PASS: TestRaft_UserRestore (83.44s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.69s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (3.11s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.60s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.32s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.55s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.56s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.33s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.22s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.56s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.61s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.49s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.31s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.63s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.58s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.22s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.14s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.54s) --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.64s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.21s) --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.25s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.69s) --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.79s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.14s) --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.27s) === RUN TestRaft_SendSnapshotFollower 2026-04-22T10:10:51.388Z [INFO] server-878ba533-b1f1-6e69-d77a-be501715c960: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-878ba533-b1f1-6e69-d77a-be501715c960 Address:878ba533-b1f1-6e69-d77a-be501715c960} {Suffrage:Voter ID:server-c9fba947-d710-cff4-09c9-eb1a8f0178c7 Address:c9fba947-d710-cff4-09c9-eb1a8f0178c7} {Suffrage:Voter ID:server-0dfc14f3-5266-5ec1-4865-bdb2041a37fb Address:0dfc14f3-5266-5ec1-4865-bdb2041a37fb}]" 2026-04-22T10:10:51.388Z [INFO] server-878ba533-b1f1-6e69-d77a-be501715c960: entering follower state: follower="Node at 878ba533-b1f1-6e69-d77a-be501715c960 [Follower]" leader-address= leader-id= 2026-04-22T10:10:51.388Z [INFO] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-878ba533-b1f1-6e69-d77a-be501715c960 Address:878ba533-b1f1-6e69-d77a-be501715c960} {Suffrage:Voter ID:server-c9fba947-d710-cff4-09c9-eb1a8f0178c7 Address:c9fba947-d710-cff4-09c9-eb1a8f0178c7} {Suffrage:Voter ID:server-0dfc14f3-5266-5ec1-4865-bdb2041a37fb Address:0dfc14f3-5266-5ec1-4865-bdb2041a37fb}]" 2026-04-22T10:10:51.388Z [INFO] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: entering follower state: follower="Node at c9fba947-d710-cff4-09c9-eb1a8f0178c7 [Follower]" leader-address= leader-id= 2026-04-22T10:10:51.388Z [INFO] server-0dfc14f3-5266-5ec1-4865-bdb2041a37fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-878ba533-b1f1-6e69-d77a-be501715c960 Address:878ba533-b1f1-6e69-d77a-be501715c960} {Suffrage:Voter ID:server-c9fba947-d710-cff4-09c9-eb1a8f0178c7 Address:c9fba947-d710-cff4-09c9-eb1a8f0178c7} {Suffrage:Voter ID:server-0dfc14f3-5266-5ec1-4865-bdb2041a37fb Address:0dfc14f3-5266-5ec1-4865-bdb2041a37fb}]" 2026-04-22T10:10:51.388Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:51.388Z [INFO] server-0dfc14f3-5266-5ec1-4865-bdb2041a37fb: entering follower state: follower="Node at 0dfc14f3-5266-5ec1-4865-bdb2041a37fb [Follower]" leader-address= leader-id= 2026-04-22T10:10:51.457Z [WARN] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:51.457Z [INFO] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: entering candidate state: node="Node at c9fba947-d710-cff4-09c9-eb1a8f0178c7 [Candidate]" term=2 2026-04-22T10:10:51.457Z [INFO] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: election won: term=2 tally=2 2026-04-22T10:10:51.457Z [INFO] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: entering leader state: leader="Node at c9fba947-d710-cff4-09c9-eb1a8f0178c7 [Leader]" 2026-04-22T10:10:51.457Z [INFO] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: added peer, starting replication: peer=server-878ba533-b1f1-6e69-d77a-be501715c960 2026-04-22T10:10:51.457Z [INFO] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: added peer, starting replication: peer=server-0dfc14f3-5266-5ec1-4865-bdb2041a37fb 2026-04-22T10:10:51.457Z [INFO] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: pipelining replication: peer="{Voter server-0dfc14f3-5266-5ec1-4865-bdb2041a37fb 0dfc14f3-5266-5ec1-4865-bdb2041a37fb}" 2026-04-22T10:10:51.457Z [INFO] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: pipelining replication: peer="{Voter server-878ba533-b1f1-6e69-d77a-be501715c960 878ba533-b1f1-6e69-d77a-be501715c960}" 2026-04-22T10:10:51.563Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:51.457825045 +0000 UTC m=+97.481621138 (2 nodes), 69.41838ms from start of poll, 69.745681ms from cluster start. Timeout at 2026-04-22 10:10:51.562831654 +0000 UTC m=+97.586627791, 105.006653ms after stability 2026-04-22T10:10:51.563Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:51.668Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:51.563286703 +0000 UTC m=+97.587082880 (1 nodes), 1.38µs from start of poll, 175.207423ms from cluster start. Timeout at 2026-04-22 10:10:51.668298574 +0000 UTC m=+97.692094734, 105.011854ms after stability 2026-04-22T10:10:51.668Z [ERROR] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: failed to pipeline appendEntries: peer="{Voter server-878ba533-b1f1-6e69-d77a-be501715c960 878ba533-b1f1-6e69-d77a-be501715c960}" error="append pipeline closed" 2026-04-22T10:10:51.668Z [INFO] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: aborting pipeline replication: peer="{Voter server-878ba533-b1f1-6e69-d77a-be501715c960 878ba533-b1f1-6e69-d77a-be501715c960}" 2026-04-22T10:10:51.668Z [ERROR] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: failed to appendEntries to: peer="{Voter server-878ba533-b1f1-6e69-d77a-be501715c960 878ba533-b1f1-6e69-d77a-be501715c960}" error="failed to connect to peer: 878ba533-b1f1-6e69-d77a-be501715c960" raft_test.go:1627: [INFO] Finished apply without behind follower 2026-04-22T10:10:51.670Z [ERROR] server-878ba533-b1f1-6e69-d77a-be501715c960: failed to take snapshot: error="nothing new to snapshot" 2026-04-22T10:10:51.670Z [INFO] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: starting snapshot up to: index=102 2026-04-22T10:10:51.670Z [INFO] creating new snapshot: path=/tmp/raft3782968894/snapshots/2-102-1776852651670.tmp 2026-04-22T10:10:51.671Z [INFO] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: compacting logs: from=1 to=92 2026-04-22T10:10:51.671Z [INFO] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: snapshot complete up to: index=102 2026-04-22T10:10:51.671Z [INFO] server-0dfc14f3-5266-5ec1-4865-bdb2041a37fb: starting snapshot up to: index=97 2026-04-22T10:10:51.671Z [INFO] creating new snapshot: path=/tmp/raft2904758255/snapshots/2-97-1776852651671.tmp 2026-04-22T10:10:51.671Z [INFO] server-0dfc14f3-5266-5ec1-4865-bdb2041a37fb: compacting logs: from=1 to=92 2026-04-22T10:10:51.671Z [INFO] server-0dfc14f3-5266-5ec1-4865-bdb2041a37fb: snapshot complete up to: index=97 2026-04-22T10:10:51.679Z [ERROR] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: failed to get log: index=2 error="log not found" 2026-04-22T10:10:51.679Z [INFO] creating new snapshot: path=/tmp/raft2335437208/snapshots/2-102-1776852651679.tmp 2026-04-22T10:10:51.679Z [INFO] server-878ba533-b1f1-6e69-d77a-be501715c960: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2026-04-22T10:10:51.679Z [INFO] server-878ba533-b1f1-6e69-d77a-be501715c960: copied to local snapshot: bytes=693 2026-04-22T10:10:51.680Z [INFO] server-878ba533-b1f1-6e69-d77a-be501715c960: snapshot restore progress: id=2-102-1776852651679 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2026-04-22T10:10:51.680Z [INFO] server-878ba533-b1f1-6e69-d77a-be501715c960: Installed remote snapshot 2026-04-22T10:10:51.680Z [WARN] server-878ba533-b1f1-6e69-d77a-be501715c960: skipping application of old log: index=102 2026-04-22T10:10:51.680Z [INFO] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: pipelining replication: peer="{Voter server-878ba533-b1f1-6e69-d77a-be501715c960 878ba533-b1f1-6e69-d77a-be501715c960}" 2026-04-22T10:10:51.682Z [INFO] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: aborting pipeline replication: peer="{Voter server-0dfc14f3-5266-5ec1-4865-bdb2041a37fb 0dfc14f3-5266-5ec1-4865-bdb2041a37fb}" 2026-04-22T10:10:51.682Z [INFO] server-c9fba947-d710-cff4-09c9-eb1a8f0178c7: aborting pipeline replication: peer="{Voter server-878ba533-b1f1-6e69-d77a-be501715c960 878ba533-b1f1-6e69-d77a-be501715c960}" --- PASS: TestRaft_SendSnapshotFollower (0.30s) === RUN TestRaft_SendSnapshotAndLogsFollower 2026-04-22T10:10:51.683Z [INFO] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e Address:a0de610f-f807-67e4-bb1b-e2528b9dfa1e} {Suffrage:Voter ID:server-88e001bf-6957-d893-ba34-36567703f67e Address:88e001bf-6957-d893-ba34-36567703f67e} {Suffrage:Voter ID:server-a69afb66-fdb1-dd1e-2501-87b7a3e933fb Address:a69afb66-fdb1-dd1e-2501-87b7a3e933fb}]" 2026-04-22T10:10:51.683Z [INFO] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: entering follower state: follower="Node at a0de610f-f807-67e4-bb1b-e2528b9dfa1e [Follower]" leader-address= leader-id= 2026-04-22T10:10:51.683Z [INFO] server-88e001bf-6957-d893-ba34-36567703f67e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e Address:a0de610f-f807-67e4-bb1b-e2528b9dfa1e} {Suffrage:Voter ID:server-88e001bf-6957-d893-ba34-36567703f67e Address:88e001bf-6957-d893-ba34-36567703f67e} {Suffrage:Voter ID:server-a69afb66-fdb1-dd1e-2501-87b7a3e933fb Address:a69afb66-fdb1-dd1e-2501-87b7a3e933fb}]" 2026-04-22T10:10:51.683Z [INFO] server-88e001bf-6957-d893-ba34-36567703f67e: entering follower state: follower="Node at 88e001bf-6957-d893-ba34-36567703f67e [Follower]" leader-address= leader-id= 2026-04-22T10:10:51.684Z [INFO] server-a69afb66-fdb1-dd1e-2501-87b7a3e933fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e Address:a0de610f-f807-67e4-bb1b-e2528b9dfa1e} {Suffrage:Voter ID:server-88e001bf-6957-d893-ba34-36567703f67e Address:88e001bf-6957-d893-ba34-36567703f67e} {Suffrage:Voter ID:server-a69afb66-fdb1-dd1e-2501-87b7a3e933fb Address:a69afb66-fdb1-dd1e-2501-87b7a3e933fb}]" 2026-04-22T10:10:51.684Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:51.684Z [INFO] server-a69afb66-fdb1-dd1e-2501-87b7a3e933fb: entering follower state: follower="Node at a69afb66-fdb1-dd1e-2501-87b7a3e933fb [Follower]" leader-address= leader-id= 2026-04-22T10:10:51.746Z [WARN] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:51.746Z [INFO] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: entering candidate state: node="Node at a0de610f-f807-67e4-bb1b-e2528b9dfa1e [Candidate]" term=2 2026-04-22T10:10:51.746Z [INFO] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: election won: term=2 tally=2 2026-04-22T10:10:51.746Z [INFO] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: entering leader state: leader="Node at a0de610f-f807-67e4-bb1b-e2528b9dfa1e [Leader]" 2026-04-22T10:10:51.746Z [INFO] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: added peer, starting replication: peer=server-88e001bf-6957-d893-ba34-36567703f67e 2026-04-22T10:10:51.746Z [INFO] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: added peer, starting replication: peer=server-a69afb66-fdb1-dd1e-2501-87b7a3e933fb 2026-04-22T10:10:51.746Z [INFO] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: pipelining replication: peer="{Voter server-88e001bf-6957-d893-ba34-36567703f67e 88e001bf-6957-d893-ba34-36567703f67e}" 2026-04-22T10:10:51.746Z [INFO] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: pipelining replication: peer="{Voter server-a69afb66-fdb1-dd1e-2501-87b7a3e933fb a69afb66-fdb1-dd1e-2501-87b7a3e933fb}" 2026-04-22T10:10:51.853Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:51.746709489 +0000 UTC m=+97.770505576 (2 nodes), 62.646146ms from start of poll, 62.942778ms from cluster start. Timeout at 2026-04-22 10:10:51.851719474 +0000 UTC m=+97.875515611, 105.010035ms after stability 2026-04-22T10:10:51.853Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:51.959Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:51.853081605 +0000 UTC m=+97.876877711 (1 nodes), 1.06µs from start of poll, 169.314913ms from cluster start. Timeout at 2026-04-22 10:10:51.958115849 +0000 UTC m=+97.981912029, 105.034318ms after stability 2026-04-22T10:10:51.959Z [ERROR] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: failed to pipeline appendEntries: peer="{Voter server-88e001bf-6957-d893-ba34-36567703f67e 88e001bf-6957-d893-ba34-36567703f67e}" error="append pipeline closed" 2026-04-22T10:10:51.959Z [INFO] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: aborting pipeline replication: peer="{Voter server-88e001bf-6957-d893-ba34-36567703f67e 88e001bf-6957-d893-ba34-36567703f67e}" 2026-04-22T10:10:51.959Z [ERROR] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: failed to appendEntries to: peer="{Voter server-88e001bf-6957-d893-ba34-36567703f67e 88e001bf-6957-d893-ba34-36567703f67e}" error="failed to connect to peer: 88e001bf-6957-d893-ba34-36567703f67e" raft_test.go:1669: [INFO] Finished apply without behind follower 2026-04-22T10:10:51.963Z [INFO] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: starting snapshot up to: index=102 2026-04-22T10:10:51.963Z [INFO] creating new snapshot: path=/tmp/raft3180086824/snapshots/2-102-1776852651963.tmp 2026-04-22T10:10:51.963Z [INFO] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: compacting logs: from=1 to=92 2026-04-22T10:10:51.963Z [INFO] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: snapshot complete up to: index=102 2026-04-22T10:10:51.964Z [ERROR] server-88e001bf-6957-d893-ba34-36567703f67e: failed to take snapshot: error="nothing new to snapshot" 2026-04-22T10:10:51.964Z [INFO] server-a69afb66-fdb1-dd1e-2501-87b7a3e933fb: starting snapshot up to: index=76 2026-04-22T10:10:51.964Z [INFO] creating new snapshot: path=/tmp/raft3022706013/snapshots/2-76-1776852651964.tmp 2026-04-22T10:10:51.964Z [INFO] server-a69afb66-fdb1-dd1e-2501-87b7a3e933fb: compacting logs: from=1 to=76 2026-04-22T10:10:51.964Z [INFO] server-a69afb66-fdb1-dd1e-2501-87b7a3e933fb: snapshot complete up to: index=76 2026-04-22T10:10:51.964Z [ERROR] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: failed to heartbeat to: peer=88e001bf-6957-d893-ba34-36567703f67e backoff time=10ms error="failed to connect to peer: 88e001bf-6957-d893-ba34-36567703f67e" raft_test.go:1690: [INFO] Finished apply without behind follower 2026-04-22T10:10:51.969Z [ERROR] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: failed to get log: index=2 error="log not found" 2026-04-22T10:10:51.970Z [INFO] creating new snapshot: path=/tmp/raft3849518958/snapshots/2-102-1776852651970.tmp 2026-04-22T10:10:51.970Z [INFO] server-88e001bf-6957-d893-ba34-36567703f67e: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2026-04-22T10:10:51.970Z [INFO] server-88e001bf-6957-d893-ba34-36567703f67e: copied to local snapshot: bytes=693 2026-04-22T10:10:51.970Z [INFO] server-88e001bf-6957-d893-ba34-36567703f67e: snapshot restore progress: id=2-102-1776852651970 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2026-04-22T10:10:51.970Z [INFO] server-88e001bf-6957-d893-ba34-36567703f67e: Installed remote snapshot 2026-04-22T10:10:51.971Z [INFO] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: pipelining replication: peer="{Voter server-88e001bf-6957-d893-ba34-36567703f67e 88e001bf-6957-d893-ba34-36567703f67e}" 2026-04-22T10:10:51.979Z [INFO] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: aborting pipeline replication: peer="{Voter server-88e001bf-6957-d893-ba34-36567703f67e 88e001bf-6957-d893-ba34-36567703f67e}" 2026-04-22T10:10:51.979Z [INFO] server-a0de610f-f807-67e4-bb1b-e2528b9dfa1e: aborting pipeline replication: peer="{Voter server-a69afb66-fdb1-dd1e-2501-87b7a3e933fb a69afb66-fdb1-dd1e-2501-87b7a3e933fb}" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.30s) === RUN TestRaft_ReJoinFollower 2026-04-22T10:10:51.980Z [INFO] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bba6b8e-771d-10bf-63ee-ecea10f4e685 Address:0bba6b8e-771d-10bf-63ee-ecea10f4e685} {Suffrage:Voter ID:server-4256db95-139b-8e62-3f6c-f66df7e3a560 Address:4256db95-139b-8e62-3f6c-f66df7e3a560} {Suffrage:Voter ID:server-17e61f28-b457-5169-0688-59198711421d Address:17e61f28-b457-5169-0688-59198711421d}]" 2026-04-22T10:10:51.980Z [INFO] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: entering follower state: follower="Node at 0bba6b8e-771d-10bf-63ee-ecea10f4e685 [Follower]" leader-address= leader-id= 2026-04-22T10:10:51.980Z [INFO] server-4256db95-139b-8e62-3f6c-f66df7e3a560: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bba6b8e-771d-10bf-63ee-ecea10f4e685 Address:0bba6b8e-771d-10bf-63ee-ecea10f4e685} {Suffrage:Voter ID:server-4256db95-139b-8e62-3f6c-f66df7e3a560 Address:4256db95-139b-8e62-3f6c-f66df7e3a560} {Suffrage:Voter ID:server-17e61f28-b457-5169-0688-59198711421d Address:17e61f28-b457-5169-0688-59198711421d}]" 2026-04-22T10:10:51.980Z [INFO] server-4256db95-139b-8e62-3f6c-f66df7e3a560: entering follower state: follower="Node at 4256db95-139b-8e62-3f6c-f66df7e3a560 [Follower]" leader-address= leader-id= 2026-04-22T10:10:51.980Z [INFO] server-17e61f28-b457-5169-0688-59198711421d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0bba6b8e-771d-10bf-63ee-ecea10f4e685 Address:0bba6b8e-771d-10bf-63ee-ecea10f4e685} {Suffrage:Voter ID:server-4256db95-139b-8e62-3f6c-f66df7e3a560 Address:4256db95-139b-8e62-3f6c-f66df7e3a560} {Suffrage:Voter ID:server-17e61f28-b457-5169-0688-59198711421d Address:17e61f28-b457-5169-0688-59198711421d}]" 2026-04-22T10:10:51.980Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:51.980Z [INFO] server-17e61f28-b457-5169-0688-59198711421d: entering follower state: follower="Node at 17e61f28-b457-5169-0688-59198711421d [Follower]" leader-address= leader-id= 2026-04-22T10:10:52.035Z [WARN] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:52.035Z [INFO] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: entering candidate state: node="Node at 0bba6b8e-771d-10bf-63ee-ecea10f4e685 [Candidate]" term=2 2026-04-22T10:10:52.035Z [INFO] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: election won: term=2 tally=2 2026-04-22T10:10:52.035Z [INFO] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: entering leader state: leader="Node at 0bba6b8e-771d-10bf-63ee-ecea10f4e685 [Leader]" 2026-04-22T10:10:52.035Z [INFO] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: added peer, starting replication: peer=server-4256db95-139b-8e62-3f6c-f66df7e3a560 2026-04-22T10:10:52.035Z [INFO] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: added peer, starting replication: peer=server-17e61f28-b457-5169-0688-59198711421d 2026-04-22T10:10:52.035Z [INFO] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: pipelining replication: peer="{Voter server-4256db95-139b-8e62-3f6c-f66df7e3a560 4256db95-139b-8e62-3f6c-f66df7e3a560}" 2026-04-22T10:10:52.035Z [INFO] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: pipelining replication: peer="{Voter server-17e61f28-b457-5169-0688-59198711421d 17e61f28-b457-5169-0688-59198711421d}" 2026-04-22T10:10:52.140Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:52.035215717 +0000 UTC m=+98.059011826 (1 nodes), 54.84582ms from start of poll, 55.15796ms from cluster start. Timeout at 2026-04-22 10:10:52.140222151 +0000 UTC m=+98.164018288, 105.006462ms after stability 2026-04-22T10:10:52.146Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:52.251Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:52.146130655 +0000 UTC m=+98.169929445 (2 nodes), 4.734µs from start of poll, 166.075579ms from cluster start. Timeout at 2026-04-22 10:10:52.25114217 +0000 UTC m=+98.274938313, 105.008868ms after stability 2026-04-22T10:10:52.252Z [INFO] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: updating configuration: command=RemoveServer server-id=server-4256db95-139b-8e62-3f6c-f66df7e3a560 server-addr= servers="[{Suffrage:Voter ID:server-0bba6b8e-771d-10bf-63ee-ecea10f4e685 Address:0bba6b8e-771d-10bf-63ee-ecea10f4e685} {Suffrage:Voter ID:server-17e61f28-b457-5169-0688-59198711421d Address:17e61f28-b457-5169-0688-59198711421d}]" 2026-04-22T10:10:52.252Z [INFO] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: removed peer, stopping replication: peer=server-4256db95-139b-8e62-3f6c-f66df7e3a560 last-index=3 2026-04-22T10:10:52.252Z [INFO] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: aborting pipeline replication: peer="{Voter server-4256db95-139b-8e62-3f6c-f66df7e3a560 4256db95-139b-8e62-3f6c-f66df7e3a560}" 2026-04-22T10:10:52.357Z [INFO] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: updating configuration: command=AddVoter server-id=server-4256db95-139b-8e62-3f6c-f66df7e3a560 server-addr=4256db95-139b-8e62-3f6c-f66df7e3a560 servers="[{Suffrage:Voter ID:server-0bba6b8e-771d-10bf-63ee-ecea10f4e685 Address:0bba6b8e-771d-10bf-63ee-ecea10f4e685} {Suffrage:Voter ID:server-17e61f28-b457-5169-0688-59198711421d Address:17e61f28-b457-5169-0688-59198711421d} {Suffrage:Voter ID:server-4256db95-139b-8e62-3f6c-f66df7e3a560 Address:4256db95-139b-8e62-3f6c-f66df7e3a560}]" 2026-04-22T10:10:52.357Z [INFO] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: added peer, starting replication: peer=server-4256db95-139b-8e62-3f6c-f66df7e3a560 2026-04-22T10:10:52.358Z [WARN] server-4256db95-139b-8e62-3f6c-f66df7e3a560: failed to get previous log: previous-index=4 last-index=3 error="log not found" 2026-04-22T10:10:52.358Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:52.358Z [WARN] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: appendEntries rejected, sending older logs: peer="{Voter server-4256db95-139b-8e62-3f6c-f66df7e3a560 4256db95-139b-8e62-3f6c-f66df7e3a560}" next=4 2026-04-22T10:10:52.358Z [INFO] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: pipelining replication: peer="{Voter server-4256db95-139b-8e62-3f6c-f66df7e3a560 4256db95-139b-8e62-3f6c-f66df7e3a560}" 2026-04-22T10:10:52.463Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:52.358184856 +0000 UTC m=+98.381981006 (1 nodes), 1.039µs from start of poll, 378.12714ms from cluster start. Timeout at 2026-04-22 10:10:52.46321697 +0000 UTC m=+98.487029308, 105.048302ms after stability 2026-04-22T10:10:52.463Z [INFO] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: aborting pipeline replication: peer="{Voter server-17e61f28-b457-5169-0688-59198711421d 17e61f28-b457-5169-0688-59198711421d}" 2026-04-22T10:10:52.463Z [INFO] server-0bba6b8e-771d-10bf-63ee-ecea10f4e685: aborting pipeline replication: peer="{Voter server-4256db95-139b-8e62-3f6c-f66df7e3a560 4256db95-139b-8e62-3f6c-f66df7e3a560}" --- PASS: TestRaft_ReJoinFollower (0.48s) === RUN TestRaft_LeaderLeaseExpire 2026-04-22T10:10:52.464Z [INFO] server-3ed41163-ca1e-01be-5350-08301687ee2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ed41163-ca1e-01be-5350-08301687ee2d Address:3ed41163-ca1e-01be-5350-08301687ee2d} {Suffrage:Voter ID:server-0c19192d-eea8-56e5-8fea-c839cdea3de4 Address:0c19192d-eea8-56e5-8fea-c839cdea3de4}]" 2026-04-22T10:10:52.464Z [INFO] server-3ed41163-ca1e-01be-5350-08301687ee2d: entering follower state: follower="Node at 3ed41163-ca1e-01be-5350-08301687ee2d [Follower]" leader-address= leader-id= 2026-04-22T10:10:52.464Z [INFO] server-0c19192d-eea8-56e5-8fea-c839cdea3de4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ed41163-ca1e-01be-5350-08301687ee2d Address:3ed41163-ca1e-01be-5350-08301687ee2d} {Suffrage:Voter ID:server-0c19192d-eea8-56e5-8fea-c839cdea3de4 Address:0c19192d-eea8-56e5-8fea-c839cdea3de4}]" 2026-04-22T10:10:52.464Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:52.464Z [INFO] server-0c19192d-eea8-56e5-8fea-c839cdea3de4: entering follower state: follower="Node at 0c19192d-eea8-56e5-8fea-c839cdea3de4 [Follower]" leader-address= leader-id= 2026-04-22T10:10:52.523Z [WARN] server-3ed41163-ca1e-01be-5350-08301687ee2d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:52.523Z [INFO] server-3ed41163-ca1e-01be-5350-08301687ee2d: entering candidate state: node="Node at 3ed41163-ca1e-01be-5350-08301687ee2d [Candidate]" term=2 2026-04-22T10:10:52.523Z [INFO] server-3ed41163-ca1e-01be-5350-08301687ee2d: election won: term=2 tally=2 2026-04-22T10:10:52.523Z [INFO] server-3ed41163-ca1e-01be-5350-08301687ee2d: entering leader state: leader="Node at 3ed41163-ca1e-01be-5350-08301687ee2d [Leader]" 2026-04-22T10:10:52.523Z [INFO] server-3ed41163-ca1e-01be-5350-08301687ee2d: added peer, starting replication: peer=server-0c19192d-eea8-56e5-8fea-c839cdea3de4 2026-04-22T10:10:52.523Z [INFO] server-3ed41163-ca1e-01be-5350-08301687ee2d: pipelining replication: peer="{Voter server-0c19192d-eea8-56e5-8fea-c839cdea3de4 0c19192d-eea8-56e5-8fea-c839cdea3de4}" 2026-04-22T10:10:52.629Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:52.523814881 +0000 UTC m=+98.547610996 (1 nodes), 59.155431ms from start of poll, 59.358929ms from cluster start. Timeout at 2026-04-22 10:10:52.628818531 +0000 UTC m=+98.652614625, 105.003629ms after stability 2026-04-22T10:10:52.635Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:52.741Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:52.635914176 +0000 UTC m=+98.659710277 (1 nodes), 584ns from start of poll, 171.45821ms from cluster start. Timeout at 2026-04-22 10:10:52.740932897 +0000 UTC m=+98.764729033, 105.018756ms after stability raft_test.go:1799: [INFO] Disconnecting Node at 0c19192d-eea8-56e5-8fea-c839cdea3de4 [Follower] 2026-04-22T10:10:52.742Z [ERROR] server-3ed41163-ca1e-01be-5350-08301687ee2d: failed to pipeline appendEntries: peer="{Voter server-0c19192d-eea8-56e5-8fea-c839cdea3de4 0c19192d-eea8-56e5-8fea-c839cdea3de4}" error="append pipeline closed" 2026-04-22T10:10:52.742Z [INFO] server-3ed41163-ca1e-01be-5350-08301687ee2d: aborting pipeline replication: peer="{Voter server-0c19192d-eea8-56e5-8fea-c839cdea3de4 0c19192d-eea8-56e5-8fea-c839cdea3de4}" 2026-04-22T10:10:52.743Z [ERROR] server-3ed41163-ca1e-01be-5350-08301687ee2d: failed to heartbeat to: peer=0c19192d-eea8-56e5-8fea-c839cdea3de4 backoff time=10ms error="failed to connect to peer: 0c19192d-eea8-56e5-8fea-c839cdea3de4" 2026-04-22T10:10:52.751Z [ERROR] server-3ed41163-ca1e-01be-5350-08301687ee2d: failed to appendEntries to: peer="{Voter server-0c19192d-eea8-56e5-8fea-c839cdea3de4 0c19192d-eea8-56e5-8fea-c839cdea3de4}" error="failed to connect to peer: 0c19192d-eea8-56e5-8fea-c839cdea3de4" 2026-04-22T10:10:52.762Z [ERROR] server-3ed41163-ca1e-01be-5350-08301687ee2d: failed to heartbeat to: peer=0c19192d-eea8-56e5-8fea-c839cdea3de4 backoff time=10ms error="failed to connect to peer: 0c19192d-eea8-56e5-8fea-c839cdea3de4" 2026-04-22T10:10:52.766Z [ERROR] server-3ed41163-ca1e-01be-5350-08301687ee2d: failed to appendEntries to: peer="{Voter server-0c19192d-eea8-56e5-8fea-c839cdea3de4 0c19192d-eea8-56e5-8fea-c839cdea3de4}" error="failed to connect to peer: 0c19192d-eea8-56e5-8fea-c839cdea3de4" 2026-04-22T10:10:52.781Z [ERROR] server-3ed41163-ca1e-01be-5350-08301687ee2d: failed to heartbeat to: peer=0c19192d-eea8-56e5-8fea-c839cdea3de4 backoff time=10ms error="failed to connect to peer: 0c19192d-eea8-56e5-8fea-c839cdea3de4" 2026-04-22T10:10:52.784Z [ERROR] server-3ed41163-ca1e-01be-5350-08301687ee2d: failed to appendEntries to: peer="{Voter server-0c19192d-eea8-56e5-8fea-c839cdea3de4 0c19192d-eea8-56e5-8fea-c839cdea3de4}" error="failed to connect to peer: 0c19192d-eea8-56e5-8fea-c839cdea3de4" 2026-04-22T10:10:52.784Z [WARN] server-3ed41163-ca1e-01be-5350-08301687ee2d: failed to contact: server-id=server-0c19192d-eea8-56e5-8fea-c839cdea3de4 time=50.089949ms 2026-04-22T10:10:52.784Z [WARN] server-3ed41163-ca1e-01be-5350-08301687ee2d: failed to contact quorum of nodes, stepping down 2026-04-22T10:10:52.784Z [INFO] server-3ed41163-ca1e-01be-5350-08301687ee2d: entering follower state: follower="Node at 3ed41163-ca1e-01be-5350-08301687ee2d [Follower]" leader-address= leader-id= 2026-04-22T10:10:52.784Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:52.833Z [WARN] server-0c19192d-eea8-56e5-8fea-c839cdea3de4: heartbeat timeout reached, starting election: last-leader-addr=3ed41163-ca1e-01be-5350-08301687ee2d last-leader-id=server-3ed41163-ca1e-01be-5350-08301687ee2d 2026-04-22T10:10:52.833Z [INFO] server-0c19192d-eea8-56e5-8fea-c839cdea3de4: entering candidate state: node="Node at 0c19192d-eea8-56e5-8fea-c839cdea3de4 [Candidate]" term=3 2026-04-22T10:10:52.833Z [ERROR] server-0c19192d-eea8-56e5-8fea-c839cdea3de4: failed to make requestVote RPC: target="{Voter server-3ed41163-ca1e-01be-5350-08301687ee2d 3ed41163-ca1e-01be-5350-08301687ee2d}" error="failed to connect to peer: 3ed41163-ca1e-01be-5350-08301687ee2d" term=3 2026-04-22T10:10:52.837Z [WARN] server-3ed41163-ca1e-01be-5350-08301687ee2d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:52.837Z [INFO] server-3ed41163-ca1e-01be-5350-08301687ee2d: entering candidate state: node="Node at 3ed41163-ca1e-01be-5350-08301687ee2d [Candidate]" term=3 2026-04-22T10:10:52.837Z [ERROR] server-3ed41163-ca1e-01be-5350-08301687ee2d: failed to make requestVote RPC: target="{Voter server-0c19192d-eea8-56e5-8fea-c839cdea3de4 0c19192d-eea8-56e5-8fea-c839cdea3de4}" error="failed to connect to peer: 0c19192d-eea8-56e5-8fea-c839cdea3de4" term=3 2026-04-22T10:10:52.910Z [WARN] server-0c19192d-eea8-56e5-8fea-c839cdea3de4: Election timeout reached, restarting election 2026-04-22T10:10:52.910Z [INFO] server-0c19192d-eea8-56e5-8fea-c839cdea3de4: entering candidate state: node="Node at 0c19192d-eea8-56e5-8fea-c839cdea3de4 [Candidate]" term=4 2026-04-22T10:10:52.910Z [ERROR] server-0c19192d-eea8-56e5-8fea-c839cdea3de4: failed to make requestVote RPC: target="{Voter server-3ed41163-ca1e-01be-5350-08301687ee2d 3ed41163-ca1e-01be-5350-08301687ee2d}" error="failed to connect to peer: 3ed41163-ca1e-01be-5350-08301687ee2d" term=4 2026-04-22T10:10:52.934Z [WARN] server-3ed41163-ca1e-01be-5350-08301687ee2d: Election timeout reached, restarting election 2026-04-22T10:10:52.934Z [INFO] server-3ed41163-ca1e-01be-5350-08301687ee2d: entering candidate state: node="Node at 3ed41163-ca1e-01be-5350-08301687ee2d [Candidate]" term=4 2026-04-22T10:10:52.934Z [ERROR] server-3ed41163-ca1e-01be-5350-08301687ee2d: failed to make requestVote RPC: target="{Voter server-0c19192d-eea8-56e5-8fea-c839cdea3de4 0c19192d-eea8-56e5-8fea-c839cdea3de4}" error="failed to connect to peer: 0c19192d-eea8-56e5-8fea-c839cdea3de4" term=4 2026-04-22T10:10:52.942Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:52.837160566 +0000 UTC m=+98.860956716 (0 nodes), 52.637139ms from start of poll, 372.704649ms from cluster start. Timeout at 2026-04-22 10:10:52.942179118 +0000 UTC m=+98.965975345, 105.018629ms after stability 2026-04-22T10:10:52.962Z [WARN] server-0c19192d-eea8-56e5-8fea-c839cdea3de4: Election timeout reached, restarting election 2026-04-22T10:10:52.962Z [INFO] server-0c19192d-eea8-56e5-8fea-c839cdea3de4: entering candidate state: node="Node at 0c19192d-eea8-56e5-8fea-c839cdea3de4 [Candidate]" term=5 2026-04-22T10:10:52.962Z [ERROR] server-0c19192d-eea8-56e5-8fea-c839cdea3de4: failed to make requestVote RPC: target="{Voter server-3ed41163-ca1e-01be-5350-08301687ee2d 3ed41163-ca1e-01be-5350-08301687ee2d}" error="failed to connect to peer: 3ed41163-ca1e-01be-5350-08301687ee2d" term=5 2026-04-22T10:10:52.989Z [WARN] server-3ed41163-ca1e-01be-5350-08301687ee2d: Election timeout reached, restarting election 2026-04-22T10:10:52.989Z [INFO] server-3ed41163-ca1e-01be-5350-08301687ee2d: entering candidate state: node="Node at 3ed41163-ca1e-01be-5350-08301687ee2d [Candidate]" term=5 2026-04-22T10:10:52.989Z [ERROR] server-3ed41163-ca1e-01be-5350-08301687ee2d: failed to make requestVote RPC: target="{Voter server-0c19192d-eea8-56e5-8fea-c839cdea3de4 0c19192d-eea8-56e5-8fea-c839cdea3de4}" error="failed to connect to peer: 0c19192d-eea8-56e5-8fea-c839cdea3de4" term=5 2026-04-22T10:10:53.032Z [WARN] server-0c19192d-eea8-56e5-8fea-c839cdea3de4: Election timeout reached, restarting election 2026-04-22T10:10:53.032Z [INFO] server-0c19192d-eea8-56e5-8fea-c839cdea3de4: entering candidate state: node="Node at 0c19192d-eea8-56e5-8fea-c839cdea3de4 [Candidate]" term=6 2026-04-22T10:10:53.032Z [ERROR] server-0c19192d-eea8-56e5-8fea-c839cdea3de4: failed to make requestVote RPC: target="{Voter server-3ed41163-ca1e-01be-5350-08301687ee2d 3ed41163-ca1e-01be-5350-08301687ee2d}" error="failed to connect to peer: 3ed41163-ca1e-01be-5350-08301687ee2d" term=6 2026-04-22T10:10:53.046Z [WARN] server-3ed41163-ca1e-01be-5350-08301687ee2d: Election timeout reached, restarting election 2026-04-22T10:10:53.046Z [INFO] server-3ed41163-ca1e-01be-5350-08301687ee2d: entering candidate state: node="Node at 3ed41163-ca1e-01be-5350-08301687ee2d [Candidate]" term=6 2026-04-22T10:10:53.046Z [ERROR] server-3ed41163-ca1e-01be-5350-08301687ee2d: failed to make requestVote RPC: target="{Voter server-0c19192d-eea8-56e5-8fea-c839cdea3de4 0c19192d-eea8-56e5-8fea-c839cdea3de4}" error="failed to connect to peer: 0c19192d-eea8-56e5-8fea-c839cdea3de4" term=6 --- PASS: TestRaft_LeaderLeaseExpire (0.58s) === RUN TestRaft_Barrier 2026-04-22T10:10:53.048Z [INFO] server-bc8b56ec-3caa-7235-ef45-e0293dd4bc2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc8b56ec-3caa-7235-ef45-e0293dd4bc2c Address:bc8b56ec-3caa-7235-ef45-e0293dd4bc2c} {Suffrage:Voter ID:server-c4334f10-15ef-328b-d767-8b1455a04f84 Address:c4334f10-15ef-328b-d767-8b1455a04f84} {Suffrage:Voter ID:server-f9c50256-5530-39e2-8527-8b41cefa7f55 Address:f9c50256-5530-39e2-8527-8b41cefa7f55}]" 2026-04-22T10:10:53.048Z [INFO] server-bc8b56ec-3caa-7235-ef45-e0293dd4bc2c: entering follower state: follower="Node at bc8b56ec-3caa-7235-ef45-e0293dd4bc2c [Follower]" leader-address= leader-id= 2026-04-22T10:10:53.048Z [INFO] server-c4334f10-15ef-328b-d767-8b1455a04f84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc8b56ec-3caa-7235-ef45-e0293dd4bc2c Address:bc8b56ec-3caa-7235-ef45-e0293dd4bc2c} {Suffrage:Voter ID:server-c4334f10-15ef-328b-d767-8b1455a04f84 Address:c4334f10-15ef-328b-d767-8b1455a04f84} {Suffrage:Voter ID:server-f9c50256-5530-39e2-8527-8b41cefa7f55 Address:f9c50256-5530-39e2-8527-8b41cefa7f55}]" 2026-04-22T10:10:53.049Z [INFO] server-c4334f10-15ef-328b-d767-8b1455a04f84: entering follower state: follower="Node at c4334f10-15ef-328b-d767-8b1455a04f84 [Follower]" leader-address= leader-id= 2026-04-22T10:10:53.049Z [INFO] server-f9c50256-5530-39e2-8527-8b41cefa7f55: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc8b56ec-3caa-7235-ef45-e0293dd4bc2c Address:bc8b56ec-3caa-7235-ef45-e0293dd4bc2c} {Suffrage:Voter ID:server-c4334f10-15ef-328b-d767-8b1455a04f84 Address:c4334f10-15ef-328b-d767-8b1455a04f84} {Suffrage:Voter ID:server-f9c50256-5530-39e2-8527-8b41cefa7f55 Address:f9c50256-5530-39e2-8527-8b41cefa7f55}]" 2026-04-22T10:10:53.049Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:53.049Z [INFO] server-f9c50256-5530-39e2-8527-8b41cefa7f55: entering follower state: follower="Node at f9c50256-5530-39e2-8527-8b41cefa7f55 [Follower]" leader-address= leader-id= 2026-04-22T10:10:53.099Z [WARN] server-f9c50256-5530-39e2-8527-8b41cefa7f55: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:53.099Z [INFO] server-f9c50256-5530-39e2-8527-8b41cefa7f55: entering candidate state: node="Node at f9c50256-5530-39e2-8527-8b41cefa7f55 [Candidate]" term=2 2026-04-22T10:10:53.099Z [INFO] server-f9c50256-5530-39e2-8527-8b41cefa7f55: election won: term=2 tally=2 2026-04-22T10:10:53.099Z [INFO] server-f9c50256-5530-39e2-8527-8b41cefa7f55: entering leader state: leader="Node at f9c50256-5530-39e2-8527-8b41cefa7f55 [Leader]" 2026-04-22T10:10:53.099Z [INFO] server-f9c50256-5530-39e2-8527-8b41cefa7f55: added peer, starting replication: peer=server-bc8b56ec-3caa-7235-ef45-e0293dd4bc2c 2026-04-22T10:10:53.099Z [INFO] server-f9c50256-5530-39e2-8527-8b41cefa7f55: added peer, starting replication: peer=server-c4334f10-15ef-328b-d767-8b1455a04f84 2026-04-22T10:10:53.099Z [INFO] server-f9c50256-5530-39e2-8527-8b41cefa7f55: pipelining replication: peer="{Voter server-bc8b56ec-3caa-7235-ef45-e0293dd4bc2c bc8b56ec-3caa-7235-ef45-e0293dd4bc2c}" 2026-04-22T10:10:53.099Z [INFO] server-f9c50256-5530-39e2-8527-8b41cefa7f55: pipelining replication: peer="{Voter server-c4334f10-15ef-328b-d767-8b1455a04f84 c4334f10-15ef-328b-d767-8b1455a04f84}" 2026-04-22T10:10:53.206Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:53.099856083 +0000 UTC m=+99.123652172 (1 nodes), 50.743695ms from start of poll, 51.008784ms from cluster start. Timeout at 2026-04-22 10:10:53.204857965 +0000 UTC m=+99.228654105, 105.001933ms after stability 2026-04-22T10:10:53.222Z [INFO] server-f9c50256-5530-39e2-8527-8b41cefa7f55: aborting pipeline replication: peer="{Voter server-bc8b56ec-3caa-7235-ef45-e0293dd4bc2c bc8b56ec-3caa-7235-ef45-e0293dd4bc2c}" 2026-04-22T10:10:53.222Z [INFO] server-f9c50256-5530-39e2-8527-8b41cefa7f55: aborting pipeline replication: peer="{Voter server-c4334f10-15ef-328b-d767-8b1455a04f84 c4334f10-15ef-328b-d767-8b1455a04f84}" --- PASS: TestRaft_Barrier (0.17s) === RUN TestRaft_VerifyLeader 2026-04-22T10:10:53.223Z [INFO] server-d3f15a9d-76f2-d9ff-fbbf-c885d1c91572: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3f15a9d-76f2-d9ff-fbbf-c885d1c91572 Address:d3f15a9d-76f2-d9ff-fbbf-c885d1c91572} {Suffrage:Voter ID:server-94a46356-ce81-0174-d4ff-31db9ca2cc9b Address:94a46356-ce81-0174-d4ff-31db9ca2cc9b} {Suffrage:Voter ID:server-771c181c-f7e9-7f10-6fe1-fc6333e5e63f Address:771c181c-f7e9-7f10-6fe1-fc6333e5e63f}]" 2026-04-22T10:10:53.223Z [INFO] server-d3f15a9d-76f2-d9ff-fbbf-c885d1c91572: entering follower state: follower="Node at d3f15a9d-76f2-d9ff-fbbf-c885d1c91572 [Follower]" leader-address= leader-id= 2026-04-22T10:10:53.223Z [INFO] server-94a46356-ce81-0174-d4ff-31db9ca2cc9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3f15a9d-76f2-d9ff-fbbf-c885d1c91572 Address:d3f15a9d-76f2-d9ff-fbbf-c885d1c91572} {Suffrage:Voter ID:server-94a46356-ce81-0174-d4ff-31db9ca2cc9b Address:94a46356-ce81-0174-d4ff-31db9ca2cc9b} {Suffrage:Voter ID:server-771c181c-f7e9-7f10-6fe1-fc6333e5e63f Address:771c181c-f7e9-7f10-6fe1-fc6333e5e63f}]" 2026-04-22T10:10:53.223Z [INFO] server-94a46356-ce81-0174-d4ff-31db9ca2cc9b: entering follower state: follower="Node at 94a46356-ce81-0174-d4ff-31db9ca2cc9b [Follower]" leader-address= leader-id= 2026-04-22T10:10:53.223Z [INFO] server-771c181c-f7e9-7f10-6fe1-fc6333e5e63f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3f15a9d-76f2-d9ff-fbbf-c885d1c91572 Address:d3f15a9d-76f2-d9ff-fbbf-c885d1c91572} {Suffrage:Voter ID:server-94a46356-ce81-0174-d4ff-31db9ca2cc9b Address:94a46356-ce81-0174-d4ff-31db9ca2cc9b} {Suffrage:Voter ID:server-771c181c-f7e9-7f10-6fe1-fc6333e5e63f Address:771c181c-f7e9-7f10-6fe1-fc6333e5e63f}]" 2026-04-22T10:10:53.223Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:53.223Z [INFO] server-771c181c-f7e9-7f10-6fe1-fc6333e5e63f: entering follower state: follower="Node at 771c181c-f7e9-7f10-6fe1-fc6333e5e63f [Follower]" leader-address= leader-id= 2026-04-22T10:10:53.284Z [WARN] server-94a46356-ce81-0174-d4ff-31db9ca2cc9b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:53.284Z [INFO] server-94a46356-ce81-0174-d4ff-31db9ca2cc9b: entering candidate state: node="Node at 94a46356-ce81-0174-d4ff-31db9ca2cc9b [Candidate]" term=2 2026-04-22T10:10:53.284Z [INFO] server-94a46356-ce81-0174-d4ff-31db9ca2cc9b: election won: term=2 tally=2 2026-04-22T10:10:53.284Z [INFO] server-94a46356-ce81-0174-d4ff-31db9ca2cc9b: entering leader state: leader="Node at 94a46356-ce81-0174-d4ff-31db9ca2cc9b [Leader]" 2026-04-22T10:10:53.284Z [INFO] server-94a46356-ce81-0174-d4ff-31db9ca2cc9b: added peer, starting replication: peer=server-d3f15a9d-76f2-d9ff-fbbf-c885d1c91572 2026-04-22T10:10:53.284Z [INFO] server-94a46356-ce81-0174-d4ff-31db9ca2cc9b: added peer, starting replication: peer=server-771c181c-f7e9-7f10-6fe1-fc6333e5e63f 2026-04-22T10:10:53.284Z [INFO] server-94a46356-ce81-0174-d4ff-31db9ca2cc9b: pipelining replication: peer="{Voter server-d3f15a9d-76f2-d9ff-fbbf-c885d1c91572 d3f15a9d-76f2-d9ff-fbbf-c885d1c91572}" 2026-04-22T10:10:53.284Z [INFO] server-94a46356-ce81-0174-d4ff-31db9ca2cc9b: pipelining replication: peer="{Voter server-771c181c-f7e9-7f10-6fe1-fc6333e5e63f 771c181c-f7e9-7f10-6fe1-fc6333e5e63f}" 2026-04-22T10:10:53.390Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:53.284732067 +0000 UTC m=+99.308528154 (1 nodes), 60.957535ms from start of poll, 61.23912ms from cluster start. Timeout at 2026-04-22 10:10:53.389736113 +0000 UTC m=+99.413532246, 105.004092ms after stability 2026-04-22T10:10:53.390Z [INFO] server-94a46356-ce81-0174-d4ff-31db9ca2cc9b: aborting pipeline replication: peer="{Voter server-771c181c-f7e9-7f10-6fe1-fc6333e5e63f 771c181c-f7e9-7f10-6fe1-fc6333e5e63f}" 2026-04-22T10:10:53.390Z [INFO] server-94a46356-ce81-0174-d4ff-31db9ca2cc9b: aborting pipeline replication: peer="{Voter server-d3f15a9d-76f2-d9ff-fbbf-c885d1c91572 d3f15a9d-76f2-d9ff-fbbf-c885d1c91572}" --- PASS: TestRaft_VerifyLeader (0.17s) === RUN TestRaft_VerifyLeader_Single 2026-04-22T10:10:53.391Z [INFO] server-6b0c8df9-eb4e-e6c5-71b1-6dc84bd7da19: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b0c8df9-eb4e-e6c5-71b1-6dc84bd7da19 Address:6b0c8df9-eb4e-e6c5-71b1-6dc84bd7da19}]" 2026-04-22T10:10:53.391Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:53.391Z [INFO] server-6b0c8df9-eb4e-e6c5-71b1-6dc84bd7da19: entering follower state: follower="Node at 6b0c8df9-eb4e-e6c5-71b1-6dc84bd7da19 [Follower]" leader-address= leader-id= 2026-04-22T10:10:53.465Z [WARN] server-6b0c8df9-eb4e-e6c5-71b1-6dc84bd7da19: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:53.465Z [INFO] server-6b0c8df9-eb4e-e6c5-71b1-6dc84bd7da19: entering candidate state: node="Node at 6b0c8df9-eb4e-e6c5-71b1-6dc84bd7da19 [Candidate]" term=2 2026-04-22T10:10:53.466Z [INFO] server-6b0c8df9-eb4e-e6c5-71b1-6dc84bd7da19: election won: term=2 tally=1 2026-04-22T10:10:53.466Z [INFO] server-6b0c8df9-eb4e-e6c5-71b1-6dc84bd7da19: entering leader state: leader="Node at 6b0c8df9-eb4e-e6c5-71b1-6dc84bd7da19 [Leader]" 2026-04-22T10:10:53.571Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:53.466102731 +0000 UTC m=+99.489898881 (1 nodes), 74.961374ms from start of poll, 75.059268ms from cluster start. Timeout at 2026-04-22 10:10:53.571113034 +0000 UTC m=+99.594909168, 105.010287ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.18s) === RUN TestRaft_VerifyLeader_Fail 2026-04-22T10:10:53.572Z [INFO] server-8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18 Address:8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18} {Suffrage:Voter ID:server-5d7e175b-3019-ada3-d2b3-6219bc2cb083 Address:5d7e175b-3019-ada3-d2b3-6219bc2cb083}]" 2026-04-22T10:10:53.572Z [INFO] server-8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18: entering follower state: follower="Node at 8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18 [Follower]" leader-address= leader-id= 2026-04-22T10:10:53.572Z [INFO] server-5d7e175b-3019-ada3-d2b3-6219bc2cb083: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18 Address:8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18} {Suffrage:Voter ID:server-5d7e175b-3019-ada3-d2b3-6219bc2cb083 Address:5d7e175b-3019-ada3-d2b3-6219bc2cb083}]" 2026-04-22T10:10:53.572Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:53.572Z [INFO] server-5d7e175b-3019-ada3-d2b3-6219bc2cb083: entering follower state: follower="Node at 5d7e175b-3019-ada3-d2b3-6219bc2cb083 [Follower]" leader-address= leader-id= 2026-04-22T10:10:53.662Z [WARN] server-8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:53.662Z [INFO] server-8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18: entering candidate state: node="Node at 8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18 [Candidate]" term=2 2026-04-22T10:10:53.662Z [INFO] server-8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18: election won: term=2 tally=2 2026-04-22T10:10:53.662Z [INFO] server-8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18: entering leader state: leader="Node at 8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18 [Leader]" 2026-04-22T10:10:53.662Z [INFO] server-8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18: added peer, starting replication: peer=server-5d7e175b-3019-ada3-d2b3-6219bc2cb083 2026-04-22T10:10:53.663Z [INFO] server-8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18: pipelining replication: peer="{Voter server-5d7e175b-3019-ada3-d2b3-6219bc2cb083 5d7e175b-3019-ada3-d2b3-6219bc2cb083}" 2026-04-22T10:10:53.768Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:53.662910744 +0000 UTC m=+99.686706847 (1 nodes), 90.770779ms from start of poll, 90.954097ms from cluster start. Timeout at 2026-04-22 10:10:53.767912038 +0000 UTC m=+99.791708122, 105.001275ms after stability 2026-04-22T10:10:53.768Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:53.874Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:53.768822708 +0000 UTC m=+99.792618795 (1 nodes), 549ns from start of poll, 196.866045ms from cluster start. Timeout at 2026-04-22 10:10:53.873837886 +0000 UTC m=+99.897634123, 105.015328ms after stability 2026-04-22T10:10:53.882Z [ERROR] server-8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18: peer has newer term, stopping replication: peer="{Voter server-5d7e175b-3019-ada3-d2b3-6219bc2cb083 5d7e175b-3019-ada3-d2b3-6219bc2cb083}" 2026-04-22T10:10:53.882Z [INFO] server-8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18: aborting pipeline replication: peer="{Voter server-5d7e175b-3019-ada3-d2b3-6219bc2cb083 5d7e175b-3019-ada3-d2b3-6219bc2cb083}" 2026-04-22T10:10:53.882Z [INFO] server-8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18: entering follower state: follower="Node at 8e8d2309-46e5-3dd4-9056-3f6a2ce8ee18 [Follower]" leader-address= leader-id= --- PASS: TestRaft_VerifyLeader_Fail (0.31s) === RUN TestRaft_VerifyLeader_PartialConnect 2026-04-22T10:10:53.883Z [INFO] server-7b6b812b-4bb9-4a38-0720-899dd6c2d7d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b6b812b-4bb9-4a38-0720-899dd6c2d7d6 Address:7b6b812b-4bb9-4a38-0720-899dd6c2d7d6} {Suffrage:Voter ID:server-873c0c5a-f7aa-bc11-17e3-fdcb955be0aa Address:873c0c5a-f7aa-bc11-17e3-fdcb955be0aa} {Suffrage:Voter ID:server-8bcd5410-fcb9-4916-e395-9140369fec2a Address:8bcd5410-fcb9-4916-e395-9140369fec2a}]" 2026-04-22T10:10:53.883Z [INFO] server-7b6b812b-4bb9-4a38-0720-899dd6c2d7d6: entering follower state: follower="Node at 7b6b812b-4bb9-4a38-0720-899dd6c2d7d6 [Follower]" leader-address= leader-id= 2026-04-22T10:10:53.883Z [INFO] server-873c0c5a-f7aa-bc11-17e3-fdcb955be0aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b6b812b-4bb9-4a38-0720-899dd6c2d7d6 Address:7b6b812b-4bb9-4a38-0720-899dd6c2d7d6} {Suffrage:Voter ID:server-873c0c5a-f7aa-bc11-17e3-fdcb955be0aa Address:873c0c5a-f7aa-bc11-17e3-fdcb955be0aa} {Suffrage:Voter ID:server-8bcd5410-fcb9-4916-e395-9140369fec2a Address:8bcd5410-fcb9-4916-e395-9140369fec2a}]" 2026-04-22T10:10:53.883Z [INFO] server-873c0c5a-f7aa-bc11-17e3-fdcb955be0aa: entering follower state: follower="Node at 873c0c5a-f7aa-bc11-17e3-fdcb955be0aa [Follower]" leader-address= leader-id= 2026-04-22T10:10:53.883Z [INFO] server-8bcd5410-fcb9-4916-e395-9140369fec2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b6b812b-4bb9-4a38-0720-899dd6c2d7d6 Address:7b6b812b-4bb9-4a38-0720-899dd6c2d7d6} {Suffrage:Voter ID:server-873c0c5a-f7aa-bc11-17e3-fdcb955be0aa Address:873c0c5a-f7aa-bc11-17e3-fdcb955be0aa} {Suffrage:Voter ID:server-8bcd5410-fcb9-4916-e395-9140369fec2a Address:8bcd5410-fcb9-4916-e395-9140369fec2a}]" 2026-04-22T10:10:53.883Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:53.883Z [INFO] server-8bcd5410-fcb9-4916-e395-9140369fec2a: entering follower state: follower="Node at 8bcd5410-fcb9-4916-e395-9140369fec2a [Follower]" leader-address= leader-id= 2026-04-22T10:10:53.941Z [WARN] server-7b6b812b-4bb9-4a38-0720-899dd6c2d7d6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:53.941Z [INFO] server-7b6b812b-4bb9-4a38-0720-899dd6c2d7d6: entering candidate state: node="Node at 7b6b812b-4bb9-4a38-0720-899dd6c2d7d6 [Candidate]" term=2 2026-04-22T10:10:53.941Z [INFO] server-7b6b812b-4bb9-4a38-0720-899dd6c2d7d6: election won: term=2 tally=2 2026-04-22T10:10:53.941Z [INFO] server-7b6b812b-4bb9-4a38-0720-899dd6c2d7d6: entering leader state: leader="Node at 7b6b812b-4bb9-4a38-0720-899dd6c2d7d6 [Leader]" 2026-04-22T10:10:53.941Z [INFO] server-7b6b812b-4bb9-4a38-0720-899dd6c2d7d6: added peer, starting replication: peer=server-873c0c5a-f7aa-bc11-17e3-fdcb955be0aa 2026-04-22T10:10:53.941Z [INFO] server-7b6b812b-4bb9-4a38-0720-899dd6c2d7d6: added peer, starting replication: peer=server-8bcd5410-fcb9-4916-e395-9140369fec2a 2026-04-22T10:10:53.941Z [INFO] server-7b6b812b-4bb9-4a38-0720-899dd6c2d7d6: pipelining replication: peer="{Voter server-8bcd5410-fcb9-4916-e395-9140369fec2a 8bcd5410-fcb9-4916-e395-9140369fec2a}" 2026-04-22T10:10:53.941Z [INFO] server-7b6b812b-4bb9-4a38-0720-899dd6c2d7d6: pipelining replication: peer="{Voter server-873c0c5a-f7aa-bc11-17e3-fdcb955be0aa 873c0c5a-f7aa-bc11-17e3-fdcb955be0aa}" 2026-04-22T10:10:54.047Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:53.941564522 +0000 UTC m=+99.965360675 (1 nodes), 57.871882ms from start of poll, 58.178032ms from cluster start. Timeout at 2026-04-22 10:10:54.046565806 +0000 UTC m=+100.070361889, 105.001214ms after stability 2026-04-22T10:10:54.053Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:54.159Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:54.053573408 +0000 UTC m=+100.077369495 (2 nodes), 714ns from start of poll, 170.186852ms from cluster start. Timeout at 2026-04-22 10:10:54.158598009 +0000 UTC m=+100.182394117, 105.024622ms after stability raft_test.go:1968: [INFO] Disconnecting Node at 873c0c5a-f7aa-bc11-17e3-fdcb955be0aa [Follower] 2026-04-22T10:10:54.159Z [ERROR] server-7b6b812b-4bb9-4a38-0720-899dd6c2d7d6: failed to heartbeat to: peer=873c0c5a-f7aa-bc11-17e3-fdcb955be0aa backoff time=10ms error="failed to connect to peer: 873c0c5a-f7aa-bc11-17e3-fdcb955be0aa" 2026-04-22T10:10:54.159Z [INFO] server-7b6b812b-4bb9-4a38-0720-899dd6c2d7d6: aborting pipeline replication: peer="{Voter server-8bcd5410-fcb9-4916-e395-9140369fec2a 8bcd5410-fcb9-4916-e395-9140369fec2a}" 2026-04-22T10:10:54.159Z [INFO] server-7b6b812b-4bb9-4a38-0720-899dd6c2d7d6: aborting pipeline replication: peer="{Voter server-873c0c5a-f7aa-bc11-17e3-fdcb955be0aa 873c0c5a-f7aa-bc11-17e3-fdcb955be0aa}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.28s) === RUN TestRaft_NotifyCh 2026-04-22T10:10:54.160Z [INFO] server-4b67986a-8b5a-31fa-0df2-a5d62360c478: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4b67986a-8b5a-31fa-0df2-a5d62360c478 Address:4b67986a-8b5a-31fa-0df2-a5d62360c478}]" 2026-04-22T10:10:54.160Z [INFO] server-4b67986a-8b5a-31fa-0df2-a5d62360c478: entering follower state: follower="Node at 4b67986a-8b5a-31fa-0df2-a5d62360c478 [Follower]" leader-address= leader-id= 2026-04-22T10:10:54.260Z [WARN] server-4b67986a-8b5a-31fa-0df2-a5d62360c478: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:54.260Z [INFO] server-4b67986a-8b5a-31fa-0df2-a5d62360c478: entering candidate state: node="Node at 4b67986a-8b5a-31fa-0df2-a5d62360c478 [Candidate]" term=2 2026-04-22T10:10:54.261Z [INFO] server-4b67986a-8b5a-31fa-0df2-a5d62360c478: election won: term=2 tally=1 2026-04-22T10:10:54.261Z [INFO] server-4b67986a-8b5a-31fa-0df2-a5d62360c478: entering leader state: leader="Node at 4b67986a-8b5a-31fa-0df2-a5d62360c478 [Leader]" --- PASS: TestRaft_NotifyCh (0.10s) === RUN TestRaft_AppendEntry 2026-04-22T10:10:54.261Z [INFO] server-ba7e9a42-0427-82b2-4356-d332b1f3f78c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba7e9a42-0427-82b2-4356-d332b1f3f78c Address:ba7e9a42-0427-82b2-4356-d332b1f3f78c} {Suffrage:Voter ID:server-67ba9f51-83b1-d19a-f5fc-8cf7d97a457c Address:67ba9f51-83b1-d19a-f5fc-8cf7d97a457c} {Suffrage:Voter ID:server-7c832aa0-a138-6137-8701-be41e24a5c01 Address:7c832aa0-a138-6137-8701-be41e24a5c01}]" 2026-04-22T10:10:54.261Z [INFO] server-ba7e9a42-0427-82b2-4356-d332b1f3f78c: entering follower state: follower="Node at ba7e9a42-0427-82b2-4356-d332b1f3f78c [Follower]" leader-address= leader-id= 2026-04-22T10:10:54.261Z [INFO] server-67ba9f51-83b1-d19a-f5fc-8cf7d97a457c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba7e9a42-0427-82b2-4356-d332b1f3f78c Address:ba7e9a42-0427-82b2-4356-d332b1f3f78c} {Suffrage:Voter ID:server-67ba9f51-83b1-d19a-f5fc-8cf7d97a457c Address:67ba9f51-83b1-d19a-f5fc-8cf7d97a457c} {Suffrage:Voter ID:server-7c832aa0-a138-6137-8701-be41e24a5c01 Address:7c832aa0-a138-6137-8701-be41e24a5c01}]" 2026-04-22T10:10:54.261Z [INFO] server-67ba9f51-83b1-d19a-f5fc-8cf7d97a457c: entering follower state: follower="Node at 67ba9f51-83b1-d19a-f5fc-8cf7d97a457c [Follower]" leader-address= leader-id= 2026-04-22T10:10:54.261Z [INFO] server-7c832aa0-a138-6137-8701-be41e24a5c01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba7e9a42-0427-82b2-4356-d332b1f3f78c Address:ba7e9a42-0427-82b2-4356-d332b1f3f78c} {Suffrage:Voter ID:server-67ba9f51-83b1-d19a-f5fc-8cf7d97a457c Address:67ba9f51-83b1-d19a-f5fc-8cf7d97a457c} {Suffrage:Voter ID:server-7c832aa0-a138-6137-8701-be41e24a5c01 Address:7c832aa0-a138-6137-8701-be41e24a5c01}]" 2026-04-22T10:10:54.261Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:54.261Z [INFO] server-7c832aa0-a138-6137-8701-be41e24a5c01: entering follower state: follower="Node at 7c832aa0-a138-6137-8701-be41e24a5c01 [Follower]" leader-address= leader-id= 2026-04-22T10:10:54.323Z [WARN] server-7c832aa0-a138-6137-8701-be41e24a5c01: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:54.323Z [INFO] server-7c832aa0-a138-6137-8701-be41e24a5c01: entering candidate state: node="Node at 7c832aa0-a138-6137-8701-be41e24a5c01 [Candidate]" term=2 2026-04-22T10:10:54.323Z [INFO] server-7c832aa0-a138-6137-8701-be41e24a5c01: election won: term=2 tally=2 2026-04-22T10:10:54.323Z [INFO] server-7c832aa0-a138-6137-8701-be41e24a5c01: entering leader state: leader="Node at 7c832aa0-a138-6137-8701-be41e24a5c01 [Leader]" 2026-04-22T10:10:54.323Z [INFO] server-7c832aa0-a138-6137-8701-be41e24a5c01: added peer, starting replication: peer=server-ba7e9a42-0427-82b2-4356-d332b1f3f78c 2026-04-22T10:10:54.323Z [INFO] server-7c832aa0-a138-6137-8701-be41e24a5c01: added peer, starting replication: peer=server-67ba9f51-83b1-d19a-f5fc-8cf7d97a457c 2026-04-22T10:10:54.323Z [INFO] server-7c832aa0-a138-6137-8701-be41e24a5c01: pipelining replication: peer="{Voter server-ba7e9a42-0427-82b2-4356-d332b1f3f78c ba7e9a42-0427-82b2-4356-d332b1f3f78c}" 2026-04-22T10:10:54.323Z [INFO] server-7c832aa0-a138-6137-8701-be41e24a5c01: pipelining replication: peer="{Voter server-67ba9f51-83b1-d19a-f5fc-8cf7d97a457c 67ba9f51-83b1-d19a-f5fc-8cf7d97a457c}" 2026-04-22T10:10:54.428Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:54.323617079 +0000 UTC m=+100.347413229 (2 nodes), 61.634165ms from start of poll, 61.847962ms from cluster start. Timeout at 2026-04-22 10:10:54.428620839 +0000 UTC m=+100.452416922, 105.003693ms after stability 2026-04-22T10:10:54.428Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:54.534Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:54.428755878 +0000 UTC m=+100.452551966 (1 nodes), 486ns from start of poll, 166.986699ms from cluster start. Timeout at 2026-04-22 10:10:54.5337669 +0000 UTC m=+100.557580044, 105.028078ms after stability 2026-04-22T10:10:54.534Z [WARN] server-ba7e9a42-0427-82b2-4356-d332b1f3f78c: clearing log suffix: from=1 to=2 2026-04-22T10:10:54.534Z [WARN] server-ba7e9a42-0427-82b2-4356-d332b1f3f78c: skipping application of old log: index=1 2026-04-22T10:10:54.534Z [WARN] server-ba7e9a42-0427-82b2-4356-d332b1f3f78c: skipping application of old log: index=1 2026-04-22T10:10:54.534Z [INFO] server-7c832aa0-a138-6137-8701-be41e24a5c01: aborting pipeline replication: peer="{Voter server-67ba9f51-83b1-d19a-f5fc-8cf7d97a457c 67ba9f51-83b1-d19a-f5fc-8cf7d97a457c}" 2026-04-22T10:10:54.534Z [INFO] server-7c832aa0-a138-6137-8701-be41e24a5c01: aborting pipeline replication: peer="{Voter server-ba7e9a42-0427-82b2-4356-d332b1f3f78c ba7e9a42-0427-82b2-4356-d332b1f3f78c}" 2026-04-22T10:10:55.034Z [ERROR] server-7c832aa0-a138-6137-8701-be41e24a5c01: failed to heartbeat to: peer=ba7e9a42-0427-82b2-4356-d332b1f3f78c backoff time=10ms error="command timed out" --- PASS: TestRaft_AppendEntry (0.77s) === RUN TestRaft_VotingGrant_WhenLeaderAvailable 2026-04-22T10:10:55.035Z [INFO] server-70c7850e-9f1f-fe79-ffda-59de795c7c55: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70c7850e-9f1f-fe79-ffda-59de795c7c55 Address:70c7850e-9f1f-fe79-ffda-59de795c7c55} {Suffrage:Voter ID:server-d0b39596-ef96-6c73-8b6c-c2409efae439 Address:d0b39596-ef96-6c73-8b6c-c2409efae439} {Suffrage:Voter ID:server-565fc759-3c12-801d-646c-0f57dba12edf Address:565fc759-3c12-801d-646c-0f57dba12edf}]" 2026-04-22T10:10:55.035Z [INFO] server-70c7850e-9f1f-fe79-ffda-59de795c7c55: entering follower state: follower="Node at 70c7850e-9f1f-fe79-ffda-59de795c7c55 [Follower]" leader-address= leader-id= 2026-04-22T10:10:55.035Z [INFO] server-d0b39596-ef96-6c73-8b6c-c2409efae439: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70c7850e-9f1f-fe79-ffda-59de795c7c55 Address:70c7850e-9f1f-fe79-ffda-59de795c7c55} {Suffrage:Voter ID:server-d0b39596-ef96-6c73-8b6c-c2409efae439 Address:d0b39596-ef96-6c73-8b6c-c2409efae439} {Suffrage:Voter ID:server-565fc759-3c12-801d-646c-0f57dba12edf Address:565fc759-3c12-801d-646c-0f57dba12edf}]" 2026-04-22T10:10:55.035Z [INFO] server-d0b39596-ef96-6c73-8b6c-c2409efae439: entering follower state: follower="Node at d0b39596-ef96-6c73-8b6c-c2409efae439 [Follower]" leader-address= leader-id= 2026-04-22T10:10:55.035Z [INFO] server-565fc759-3c12-801d-646c-0f57dba12edf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70c7850e-9f1f-fe79-ffda-59de795c7c55 Address:70c7850e-9f1f-fe79-ffda-59de795c7c55} {Suffrage:Voter ID:server-d0b39596-ef96-6c73-8b6c-c2409efae439 Address:d0b39596-ef96-6c73-8b6c-c2409efae439} {Suffrage:Voter ID:server-565fc759-3c12-801d-646c-0f57dba12edf Address:565fc759-3c12-801d-646c-0f57dba12edf}]" 2026-04-22T10:10:55.035Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:55.035Z [INFO] server-565fc759-3c12-801d-646c-0f57dba12edf: entering follower state: follower="Node at 565fc759-3c12-801d-646c-0f57dba12edf [Follower]" leader-address= leader-id= 2026-04-22T10:10:55.113Z [WARN] server-70c7850e-9f1f-fe79-ffda-59de795c7c55: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:55.113Z [INFO] server-70c7850e-9f1f-fe79-ffda-59de795c7c55: entering candidate state: node="Node at 70c7850e-9f1f-fe79-ffda-59de795c7c55 [Candidate]" term=2 2026-04-22T10:10:55.113Z [INFO] server-70c7850e-9f1f-fe79-ffda-59de795c7c55: election won: term=2 tally=2 2026-04-22T10:10:55.113Z [INFO] server-70c7850e-9f1f-fe79-ffda-59de795c7c55: entering leader state: leader="Node at 70c7850e-9f1f-fe79-ffda-59de795c7c55 [Leader]" 2026-04-22T10:10:55.113Z [INFO] server-70c7850e-9f1f-fe79-ffda-59de795c7c55: added peer, starting replication: peer=server-d0b39596-ef96-6c73-8b6c-c2409efae439 2026-04-22T10:10:55.113Z [INFO] server-70c7850e-9f1f-fe79-ffda-59de795c7c55: added peer, starting replication: peer=server-565fc759-3c12-801d-646c-0f57dba12edf 2026-04-22T10:10:55.113Z [INFO] server-70c7850e-9f1f-fe79-ffda-59de795c7c55: pipelining replication: peer="{Voter server-565fc759-3c12-801d-646c-0f57dba12edf 565fc759-3c12-801d-646c-0f57dba12edf}" 2026-04-22T10:10:55.113Z [INFO] server-70c7850e-9f1f-fe79-ffda-59de795c7c55: pipelining replication: peer="{Voter server-d0b39596-ef96-6c73-8b6c-c2409efae439 d0b39596-ef96-6c73-8b6c-c2409efae439}" 2026-04-22T10:10:55.218Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:55.113562877 +0000 UTC m=+101.137358976 (2 nodes), 78.002093ms from start of poll, 78.32627ms from cluster start. Timeout at 2026-04-22 10:10:55.218570753 +0000 UTC m=+101.242366940, 105.007964ms after stability 2026-04-22T10:10:55.218Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:55.324Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:55.218772097 +0000 UTC m=+101.242568284 (1 nodes), 1.738µs from start of poll, 183.535578ms from cluster start. Timeout at 2026-04-22 10:10:55.323779757 +0000 UTC m=+101.347575860, 105.007576ms after stability 2026-04-22T10:10:55.324Z [WARN] server-565fc759-3c12-801d-646c-0f57dba12edf: rejecting vote request since we have a leader: from=d0b39596-ef96-6c73-8b6c-c2409efae439 leader=70c7850e-9f1f-fe79-ffda-59de795c7c55 leader-id=server-70c7850e-9f1f-fe79-ffda-59de795c7c55 2026-04-22T10:10:55.324Z [INFO] server-70c7850e-9f1f-fe79-ffda-59de795c7c55: aborting pipeline replication: peer="{Voter server-565fc759-3c12-801d-646c-0f57dba12edf 565fc759-3c12-801d-646c-0f57dba12edf}" 2026-04-22T10:10:55.324Z [INFO] server-70c7850e-9f1f-fe79-ffda-59de795c7c55: aborting pipeline replication: peer="{Voter server-d0b39596-ef96-6c73-8b6c-c2409efae439 d0b39596-ef96-6c73-8b6c-c2409efae439}" --- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.29s) === RUN TestRaft_ProtocolVersion_RejectRPC 2026-04-22T10:10:55.325Z [INFO] server-8a66f41d-ce0d-4584-7df4-28294371d022: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a66f41d-ce0d-4584-7df4-28294371d022 Address:8a66f41d-ce0d-4584-7df4-28294371d022} {Suffrage:Voter ID:server-f8c4b8b0-968f-723f-4b6a-2a3c10d3400e Address:f8c4b8b0-968f-723f-4b6a-2a3c10d3400e} {Suffrage:Voter ID:server-6d52832f-1b94-3729-3fff-62ad9b1cfebd Address:6d52832f-1b94-3729-3fff-62ad9b1cfebd}]" 2026-04-22T10:10:55.325Z [INFO] server-8a66f41d-ce0d-4584-7df4-28294371d022: entering follower state: follower="Node at 8a66f41d-ce0d-4584-7df4-28294371d022 [Follower]" leader-address= leader-id= 2026-04-22T10:10:55.325Z [INFO] server-f8c4b8b0-968f-723f-4b6a-2a3c10d3400e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a66f41d-ce0d-4584-7df4-28294371d022 Address:8a66f41d-ce0d-4584-7df4-28294371d022} {Suffrage:Voter ID:server-f8c4b8b0-968f-723f-4b6a-2a3c10d3400e Address:f8c4b8b0-968f-723f-4b6a-2a3c10d3400e} {Suffrage:Voter ID:server-6d52832f-1b94-3729-3fff-62ad9b1cfebd Address:6d52832f-1b94-3729-3fff-62ad9b1cfebd}]" 2026-04-22T10:10:55.325Z [INFO] server-f8c4b8b0-968f-723f-4b6a-2a3c10d3400e: entering follower state: follower="Node at f8c4b8b0-968f-723f-4b6a-2a3c10d3400e [Follower]" leader-address= leader-id= 2026-04-22T10:10:55.325Z [INFO] server-6d52832f-1b94-3729-3fff-62ad9b1cfebd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8a66f41d-ce0d-4584-7df4-28294371d022 Address:8a66f41d-ce0d-4584-7df4-28294371d022} {Suffrage:Voter ID:server-f8c4b8b0-968f-723f-4b6a-2a3c10d3400e Address:f8c4b8b0-968f-723f-4b6a-2a3c10d3400e} {Suffrage:Voter ID:server-6d52832f-1b94-3729-3fff-62ad9b1cfebd Address:6d52832f-1b94-3729-3fff-62ad9b1cfebd}]" 2026-04-22T10:10:55.325Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:55.325Z [INFO] server-6d52832f-1b94-3729-3fff-62ad9b1cfebd: entering follower state: follower="Node at 6d52832f-1b94-3729-3fff-62ad9b1cfebd [Follower]" leader-address= leader-id= 2026-04-22T10:10:55.393Z [WARN] server-6d52832f-1b94-3729-3fff-62ad9b1cfebd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:55.393Z [INFO] server-6d52832f-1b94-3729-3fff-62ad9b1cfebd: entering candidate state: node="Node at 6d52832f-1b94-3729-3fff-62ad9b1cfebd [Candidate]" term=2 2026-04-22T10:10:55.393Z [INFO] server-6d52832f-1b94-3729-3fff-62ad9b1cfebd: election won: term=2 tally=2 2026-04-22T10:10:55.393Z [INFO] server-6d52832f-1b94-3729-3fff-62ad9b1cfebd: entering leader state: leader="Node at 6d52832f-1b94-3729-3fff-62ad9b1cfebd [Leader]" 2026-04-22T10:10:55.393Z [INFO] server-6d52832f-1b94-3729-3fff-62ad9b1cfebd: added peer, starting replication: peer=server-8a66f41d-ce0d-4584-7df4-28294371d022 2026-04-22T10:10:55.393Z [INFO] server-6d52832f-1b94-3729-3fff-62ad9b1cfebd: added peer, starting replication: peer=server-f8c4b8b0-968f-723f-4b6a-2a3c10d3400e 2026-04-22T10:10:55.393Z [INFO] server-6d52832f-1b94-3729-3fff-62ad9b1cfebd: pipelining replication: peer="{Voter server-8a66f41d-ce0d-4584-7df4-28294371d022 8a66f41d-ce0d-4584-7df4-28294371d022}" 2026-04-22T10:10:55.394Z [INFO] server-6d52832f-1b94-3729-3fff-62ad9b1cfebd: pipelining replication: peer="{Voter server-f8c4b8b0-968f-723f-4b6a-2a3c10d3400e f8c4b8b0-968f-723f-4b6a-2a3c10d3400e}" 2026-04-22T10:10:55.499Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:55.393782559 +0000 UTC m=+101.417578654 (2 nodes), 68.408043ms from start of poll, 68.686154ms from cluster start. Timeout at 2026-04-22 10:10:55.498785766 +0000 UTC m=+101.522581871, 105.003217ms after stability 2026-04-22T10:10:55.499Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:55.604Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:55.49911049 +0000 UTC m=+101.522906578 (1 nodes), 474ns from start of poll, 174.014078ms from cluster start. Timeout at 2026-04-22 10:10:55.604147604 +0000 UTC m=+101.627943740, 105.037162ms after stability 2026-04-22T10:10:55.604Z [INFO] server-6d52832f-1b94-3729-3fff-62ad9b1cfebd: aborting pipeline replication: peer="{Voter server-f8c4b8b0-968f-723f-4b6a-2a3c10d3400e f8c4b8b0-968f-723f-4b6a-2a3c10d3400e}" 2026-04-22T10:10:55.604Z [INFO] server-6d52832f-1b94-3729-3fff-62ad9b1cfebd: aborting pipeline replication: peer="{Voter server-8a66f41d-ce0d-4584-7df4-28294371d022 8a66f41d-ce0d-4584-7df4-28294371d022}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.28s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 raft_test.go:2151: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 raft_test.go:2196: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s) === RUN TestRaft_LeaderID_Propagated 2026-04-22T10:10:55.605Z [INFO] server-85089f2d-ee17-0ff7-a070-ffc4b1f5bc5b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85089f2d-ee17-0ff7-a070-ffc4b1f5bc5b Address:85089f2d-ee17-0ff7-a070-ffc4b1f5bc5b} {Suffrage:Voter ID:server-da481f95-dbde-6efb-7b4b-5c8f29e96189 Address:da481f95-dbde-6efb-7b4b-5c8f29e96189} {Suffrage:Voter ID:server-6c4867b0-4c18-c379-81c4-6a93ed6ebb2c Address:6c4867b0-4c18-c379-81c4-6a93ed6ebb2c}]" 2026-04-22T10:10:55.605Z [INFO] server-85089f2d-ee17-0ff7-a070-ffc4b1f5bc5b: entering follower state: follower="Node at 85089f2d-ee17-0ff7-a070-ffc4b1f5bc5b [Follower]" leader-address= leader-id= 2026-04-22T10:10:55.605Z [INFO] server-da481f95-dbde-6efb-7b4b-5c8f29e96189: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85089f2d-ee17-0ff7-a070-ffc4b1f5bc5b Address:85089f2d-ee17-0ff7-a070-ffc4b1f5bc5b} {Suffrage:Voter ID:server-da481f95-dbde-6efb-7b4b-5c8f29e96189 Address:da481f95-dbde-6efb-7b4b-5c8f29e96189} {Suffrage:Voter ID:server-6c4867b0-4c18-c379-81c4-6a93ed6ebb2c Address:6c4867b0-4c18-c379-81c4-6a93ed6ebb2c}]" 2026-04-22T10:10:55.605Z [INFO] server-da481f95-dbde-6efb-7b4b-5c8f29e96189: entering follower state: follower="Node at da481f95-dbde-6efb-7b4b-5c8f29e96189 [Follower]" leader-address= leader-id= 2026-04-22T10:10:55.605Z [INFO] server-6c4867b0-4c18-c379-81c4-6a93ed6ebb2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85089f2d-ee17-0ff7-a070-ffc4b1f5bc5b Address:85089f2d-ee17-0ff7-a070-ffc4b1f5bc5b} {Suffrage:Voter ID:server-da481f95-dbde-6efb-7b4b-5c8f29e96189 Address:da481f95-dbde-6efb-7b4b-5c8f29e96189} {Suffrage:Voter ID:server-6c4867b0-4c18-c379-81c4-6a93ed6ebb2c Address:6c4867b0-4c18-c379-81c4-6a93ed6ebb2c}]" 2026-04-22T10:10:55.605Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:55.605Z [INFO] server-6c4867b0-4c18-c379-81c4-6a93ed6ebb2c: entering follower state: follower="Node at 6c4867b0-4c18-c379-81c4-6a93ed6ebb2c [Follower]" leader-address= leader-id= 2026-04-22T10:10:55.668Z [WARN] server-da481f95-dbde-6efb-7b4b-5c8f29e96189: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:55.668Z [INFO] server-da481f95-dbde-6efb-7b4b-5c8f29e96189: entering candidate state: node="Node at da481f95-dbde-6efb-7b4b-5c8f29e96189 [Candidate]" term=2 2026-04-22T10:10:55.668Z [INFO] server-da481f95-dbde-6efb-7b4b-5c8f29e96189: election won: term=2 tally=2 2026-04-22T10:10:55.668Z [INFO] server-da481f95-dbde-6efb-7b4b-5c8f29e96189: entering leader state: leader="Node at da481f95-dbde-6efb-7b4b-5c8f29e96189 [Leader]" 2026-04-22T10:10:55.668Z [INFO] server-da481f95-dbde-6efb-7b4b-5c8f29e96189: added peer, starting replication: peer=server-85089f2d-ee17-0ff7-a070-ffc4b1f5bc5b 2026-04-22T10:10:55.668Z [INFO] server-da481f95-dbde-6efb-7b4b-5c8f29e96189: added peer, starting replication: peer=server-6c4867b0-4c18-c379-81c4-6a93ed6ebb2c 2026-04-22T10:10:55.669Z [INFO] server-da481f95-dbde-6efb-7b4b-5c8f29e96189: pipelining replication: peer="{Voter server-85089f2d-ee17-0ff7-a070-ffc4b1f5bc5b 85089f2d-ee17-0ff7-a070-ffc4b1f5bc5b}" 2026-04-22T10:10:55.669Z [INFO] server-da481f95-dbde-6efb-7b4b-5c8f29e96189: pipelining replication: peer="{Voter server-6c4867b0-4c18-c379-81c4-6a93ed6ebb2c 6c4867b0-4c18-c379-81c4-6a93ed6ebb2c}" 2026-04-22T10:10:55.774Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:55.669214364 +0000 UTC m=+101.693010457 (1 nodes), 63.419478ms from start of poll, 63.755006ms from cluster start. Timeout at 2026-04-22 10:10:55.774221735 +0000 UTC m=+101.798017875, 105.007418ms after stability 2026-04-22T10:10:55.774Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:55.880Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:55.77466758 +0000 UTC m=+101.798463674 (1 nodes), 548ns from start of poll, 169.208223ms from cluster start. Timeout at 2026-04-22 10:10:55.879700108 +0000 UTC m=+101.903496311, 105.032637ms after stability 2026-04-22T10:10:55.880Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:55.985Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:55.880355888 +0000 UTC m=+101.904152145 (1 nodes), 4.02µs from start of poll, 274.896694ms from cluster start. Timeout at 2026-04-22 10:10:55.985363339 +0000 UTC m=+102.009159438, 105.007293ms after stability 2026-04-22T10:10:55.985Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:56.090Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:55.985620673 +0000 UTC m=+102.009416761 (1 nodes), 447ns from start of poll, 380.16131ms from cluster start. Timeout at 2026-04-22 10:10:56.090666345 +0000 UTC m=+102.114462426, 105.045665ms after stability 2026-04-22T10:10:56.090Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:56.196Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:56.090852879 +0000 UTC m=+102.114648965 (1 nodes), 574ns from start of poll, 485.393514ms from cluster start. Timeout at 2026-04-22 10:10:56.19588425 +0000 UTC m=+102.219680363, 105.031398ms after stability 2026-04-22T10:10:56.196Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:56.302Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:56.196462042 +0000 UTC m=+102.220258129 (1 nodes), 502ns from start of poll, 591.002678ms from cluster start. Timeout at 2026-04-22 10:10:56.301473161 +0000 UTC m=+102.325269277, 105.011148ms after stability 2026-04-22T10:10:56.408Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:56.514Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:56.408308797 +0000 UTC m=+102.432104885 (1 nodes), 629ns from start of poll, 802.849434ms from cluster start. Timeout at 2026-04-22 10:10:56.513352733 +0000 UTC m=+102.537148893, 105.044008ms after stability 2026-04-22T10:10:56.514Z [INFO] server-da481f95-dbde-6efb-7b4b-5c8f29e96189: aborting pipeline replication: peer="{Voter server-6c4867b0-4c18-c379-81c4-6a93ed6ebb2c 6c4867b0-4c18-c379-81c4-6a93ed6ebb2c}" 2026-04-22T10:10:56.514Z [INFO] server-da481f95-dbde-6efb-7b4b-5c8f29e96189: aborting pipeline replication: peer="{Voter server-85089f2d-ee17-0ff7-a070-ffc4b1f5bc5b 85089f2d-ee17-0ff7-a070-ffc4b1f5bc5b}" --- PASS: TestRaft_LeaderID_Propagated (0.91s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer 2026-04-22T10:10:56.516Z [INFO] server-45631c7f-dc65-64c7-2244-a218e8ab65b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45631c7f-dc65-64c7-2244-a218e8ab65b1 Address:45631c7f-dc65-64c7-2244-a218e8ab65b1} {Suffrage:Voter ID:server-f83b9a4d-5666-f038-68bc-8d477737c76e Address:f83b9a4d-5666-f038-68bc-8d477737c76e} {Suffrage:Voter ID:server-d341e058-e483-b2d3-0509-4356f6738cd5 Address:d341e058-e483-b2d3-0509-4356f6738cd5}]" 2026-04-22T10:10:56.516Z [INFO] server-45631c7f-dc65-64c7-2244-a218e8ab65b1: entering follower state: follower="Node at 45631c7f-dc65-64c7-2244-a218e8ab65b1 [Follower]" leader-address= leader-id= 2026-04-22T10:10:56.516Z [INFO] server-f83b9a4d-5666-f038-68bc-8d477737c76e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45631c7f-dc65-64c7-2244-a218e8ab65b1 Address:45631c7f-dc65-64c7-2244-a218e8ab65b1} {Suffrage:Voter ID:server-f83b9a4d-5666-f038-68bc-8d477737c76e Address:f83b9a4d-5666-f038-68bc-8d477737c76e} {Suffrage:Voter ID:server-d341e058-e483-b2d3-0509-4356f6738cd5 Address:d341e058-e483-b2d3-0509-4356f6738cd5}]" 2026-04-22T10:10:56.516Z [INFO] server-f83b9a4d-5666-f038-68bc-8d477737c76e: entering follower state: follower="Node at f83b9a4d-5666-f038-68bc-8d477737c76e [Follower]" leader-address= leader-id= 2026-04-22T10:10:56.516Z [INFO] server-d341e058-e483-b2d3-0509-4356f6738cd5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45631c7f-dc65-64c7-2244-a218e8ab65b1 Address:45631c7f-dc65-64c7-2244-a218e8ab65b1} {Suffrage:Voter ID:server-f83b9a4d-5666-f038-68bc-8d477737c76e Address:f83b9a4d-5666-f038-68bc-8d477737c76e} {Suffrage:Voter ID:server-d341e058-e483-b2d3-0509-4356f6738cd5 Address:d341e058-e483-b2d3-0509-4356f6738cd5}]" 2026-04-22T10:10:56.516Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:56.516Z [INFO] server-d341e058-e483-b2d3-0509-4356f6738cd5: entering follower state: follower="Node at d341e058-e483-b2d3-0509-4356f6738cd5 [Follower]" leader-address= leader-id= 2026-04-22T10:10:56.569Z [WARN] server-f83b9a4d-5666-f038-68bc-8d477737c76e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:56.569Z [INFO] server-f83b9a4d-5666-f038-68bc-8d477737c76e: entering candidate state: node="Node at f83b9a4d-5666-f038-68bc-8d477737c76e [Candidate]" term=2 2026-04-22T10:10:56.569Z [INFO] server-f83b9a4d-5666-f038-68bc-8d477737c76e: election won: term=2 tally=2 2026-04-22T10:10:56.569Z [INFO] server-f83b9a4d-5666-f038-68bc-8d477737c76e: entering leader state: leader="Node at f83b9a4d-5666-f038-68bc-8d477737c76e [Leader]" 2026-04-22T10:10:56.569Z [INFO] server-f83b9a4d-5666-f038-68bc-8d477737c76e: added peer, starting replication: peer=server-45631c7f-dc65-64c7-2244-a218e8ab65b1 2026-04-22T10:10:56.569Z [INFO] server-f83b9a4d-5666-f038-68bc-8d477737c76e: added peer, starting replication: peer=server-d341e058-e483-b2d3-0509-4356f6738cd5 2026-04-22T10:10:56.569Z [INFO] server-f83b9a4d-5666-f038-68bc-8d477737c76e: pipelining replication: peer="{Voter server-45631c7f-dc65-64c7-2244-a218e8ab65b1 45631c7f-dc65-64c7-2244-a218e8ab65b1}" 2026-04-22T10:10:56.569Z [INFO] server-f83b9a4d-5666-f038-68bc-8d477737c76e: pipelining replication: peer="{Voter server-d341e058-e483-b2d3-0509-4356f6738cd5 d341e058-e483-b2d3-0509-4356f6738cd5}" 2026-04-22T10:10:56.675Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:56.569602878 +0000 UTC m=+102.593398999 (1 nodes), 53.212325ms from start of poll, 53.424252ms from cluster start. Timeout at 2026-04-22 10:10:56.674606918 +0000 UTC m=+102.698402999, 105.004ms after stability 2026-04-22T10:10:56.675Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:56.781Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:56.675325758 +0000 UTC m=+102.699121845 (1 nodes), 428ns from start of poll, 159.147098ms from cluster start. Timeout at 2026-04-22 10:10:56.780333064 +0000 UTC m=+102.804129194, 105.007349ms after stability 2026-04-22T10:10:56.781Z [INFO] server-45631c7f-dc65-64c7-2244-a218e8ab65b1: entering candidate state: node="Node at 45631c7f-dc65-64c7-2244-a218e8ab65b1 [Candidate]" term=3 2026-04-22T10:10:56.781Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:56.781Z [INFO] server-45631c7f-dc65-64c7-2244-a218e8ab65b1: election won: term=3 tally=2 2026-04-22T10:10:56.781Z [INFO] server-45631c7f-dc65-64c7-2244-a218e8ab65b1: entering leader state: leader="Node at 45631c7f-dc65-64c7-2244-a218e8ab65b1 [Leader]" 2026-04-22T10:10:56.781Z [INFO] server-45631c7f-dc65-64c7-2244-a218e8ab65b1: added peer, starting replication: peer=server-f83b9a4d-5666-f038-68bc-8d477737c76e 2026-04-22T10:10:56.781Z [INFO] server-45631c7f-dc65-64c7-2244-a218e8ab65b1: added peer, starting replication: peer=server-d341e058-e483-b2d3-0509-4356f6738cd5 2026-04-22T10:10:56.781Z [INFO] server-f83b9a4d-5666-f038-68bc-8d477737c76e: entering follower state: follower="Node at f83b9a4d-5666-f038-68bc-8d477737c76e [Follower]" leader-address= leader-id= 2026-04-22T10:10:56.781Z [INFO] server-f83b9a4d-5666-f038-68bc-8d477737c76e: aborting pipeline replication: peer="{Voter server-45631c7f-dc65-64c7-2244-a218e8ab65b1 45631c7f-dc65-64c7-2244-a218e8ab65b1}" 2026-04-22T10:10:56.781Z [INFO] server-f83b9a4d-5666-f038-68bc-8d477737c76e: aborting pipeline replication: peer="{Voter server-d341e058-e483-b2d3-0509-4356f6738cd5 d341e058-e483-b2d3-0509-4356f6738cd5}" 2026-04-22T10:10:56.781Z [INFO] server-45631c7f-dc65-64c7-2244-a218e8ab65b1: pipelining replication: peer="{Voter server-d341e058-e483-b2d3-0509-4356f6738cd5 d341e058-e483-b2d3-0509-4356f6738cd5}" 2026-04-22T10:10:56.781Z [INFO] server-45631c7f-dc65-64c7-2244-a218e8ab65b1: pipelining replication: peer="{Voter server-f83b9a4d-5666-f038-68bc-8d477737c76e f83b9a4d-5666-f038-68bc-8d477737c76e}" 2026-04-22T10:10:56.887Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:56.781211886 +0000 UTC m=+102.805007979 (1 nodes), 77.045µs from start of poll, 265.033232ms from cluster start. Timeout at 2026-04-22 10:10:56.886226349 +0000 UTC m=+102.910022589, 105.01461ms after stability 2026-04-22T10:10:56.887Z [INFO] server-45631c7f-dc65-64c7-2244-a218e8ab65b1: aborting pipeline replication: peer="{Voter server-d341e058-e483-b2d3-0509-4356f6738cd5 d341e058-e483-b2d3-0509-4356f6738cd5}" 2026-04-22T10:10:56.887Z [INFO] server-45631c7f-dc65-64c7-2244-a218e8ab65b1: aborting pipeline replication: peer="{Voter server-f83b9a4d-5666-f038-68bc-8d477737c76e f83b9a4d-5666-f038-68bc-8d477737c76e}" --- PASS: TestRaft_LeadershipTransfer (0.37s) === RUN TestRaft_LeadershipTransferWithOneNode 2026-04-22T10:10:56.888Z [INFO] server-bcd41fa6-8ef6-1348-3a49-c92dcba959d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcd41fa6-8ef6-1348-3a49-c92dcba959d7 Address:bcd41fa6-8ef6-1348-3a49-c92dcba959d7}]" 2026-04-22T10:10:56.888Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:56.888Z [INFO] server-bcd41fa6-8ef6-1348-3a49-c92dcba959d7: entering follower state: follower="Node at bcd41fa6-8ef6-1348-3a49-c92dcba959d7 [Follower]" leader-address= leader-id= 2026-04-22T10:10:56.942Z [WARN] server-bcd41fa6-8ef6-1348-3a49-c92dcba959d7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:56.942Z [INFO] server-bcd41fa6-8ef6-1348-3a49-c92dcba959d7: entering candidate state: node="Node at bcd41fa6-8ef6-1348-3a49-c92dcba959d7 [Candidate]" term=2 2026-04-22T10:10:56.942Z [INFO] server-bcd41fa6-8ef6-1348-3a49-c92dcba959d7: election won: term=2 tally=1 2026-04-22T10:10:56.942Z [INFO] server-bcd41fa6-8ef6-1348-3a49-c92dcba959d7: entering leader state: leader="Node at bcd41fa6-8ef6-1348-3a49-c92dcba959d7 [Leader]" 2026-04-22T10:10:57.048Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:56.94295473 +0000 UTC m=+102.966750843 (1 nodes), 54.698009ms from start of poll, 54.820359ms from cluster start. Timeout at 2026-04-22 10:10:57.047965547 +0000 UTC m=+103.071761683, 105.01084ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.16s) === RUN TestRaft_LeadershipTransferWithSevenNodes 2026-04-22T10:10:57.050Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a Address:77ca3a41-b7c2-8c2f-f14d-79c56d6d764a} {Suffrage:Voter ID:server-bf574c04-07df-880c-460e-efc428cd5991 Address:bf574c04-07df-880c-460e-efc428cd5991} {Suffrage:Voter ID:server-6b370faa-80e7-008d-a526-dc6a356b2adf Address:6b370faa-80e7-008d-a526-dc6a356b2adf} {Suffrage:Voter ID:server-6fdcffea-780c-54c9-5b52-66b8d441d70e Address:6fdcffea-780c-54c9-5b52-66b8d441d70e} {Suffrage:Voter ID:server-d1294074-ce41-48b4-1359-1206d3c13bbb Address:d1294074-ce41-48b4-1359-1206d3c13bbb} {Suffrage:Voter ID:server-dadab481-6273-c354-6558-8456d105bfa9 Address:dadab481-6273-c354-6558-8456d105bfa9} {Suffrage:Voter ID:server-34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3 Address:34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3}]" 2026-04-22T10:10:57.050Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: entering follower state: follower="Node at 77ca3a41-b7c2-8c2f-f14d-79c56d6d764a [Follower]" leader-address= leader-id= 2026-04-22T10:10:57.050Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a Address:77ca3a41-b7c2-8c2f-f14d-79c56d6d764a} {Suffrage:Voter ID:server-bf574c04-07df-880c-460e-efc428cd5991 Address:bf574c04-07df-880c-460e-efc428cd5991} {Suffrage:Voter ID:server-6b370faa-80e7-008d-a526-dc6a356b2adf Address:6b370faa-80e7-008d-a526-dc6a356b2adf} {Suffrage:Voter ID:server-6fdcffea-780c-54c9-5b52-66b8d441d70e Address:6fdcffea-780c-54c9-5b52-66b8d441d70e} {Suffrage:Voter ID:server-d1294074-ce41-48b4-1359-1206d3c13bbb Address:d1294074-ce41-48b4-1359-1206d3c13bbb} {Suffrage:Voter ID:server-dadab481-6273-c354-6558-8456d105bfa9 Address:dadab481-6273-c354-6558-8456d105bfa9} {Suffrage:Voter ID:server-34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3 Address:34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3}]" 2026-04-22T10:10:57.050Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: entering follower state: follower="Node at bf574c04-07df-880c-460e-efc428cd5991 [Follower]" leader-address= leader-id= 2026-04-22T10:10:57.050Z [INFO] server-6b370faa-80e7-008d-a526-dc6a356b2adf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a Address:77ca3a41-b7c2-8c2f-f14d-79c56d6d764a} {Suffrage:Voter ID:server-bf574c04-07df-880c-460e-efc428cd5991 Address:bf574c04-07df-880c-460e-efc428cd5991} {Suffrage:Voter ID:server-6b370faa-80e7-008d-a526-dc6a356b2adf Address:6b370faa-80e7-008d-a526-dc6a356b2adf} {Suffrage:Voter ID:server-6fdcffea-780c-54c9-5b52-66b8d441d70e Address:6fdcffea-780c-54c9-5b52-66b8d441d70e} {Suffrage:Voter ID:server-d1294074-ce41-48b4-1359-1206d3c13bbb Address:d1294074-ce41-48b4-1359-1206d3c13bbb} {Suffrage:Voter ID:server-dadab481-6273-c354-6558-8456d105bfa9 Address:dadab481-6273-c354-6558-8456d105bfa9} {Suffrage:Voter ID:server-34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3 Address:34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3}]" 2026-04-22T10:10:57.050Z [INFO] server-6b370faa-80e7-008d-a526-dc6a356b2adf: entering follower state: follower="Node at 6b370faa-80e7-008d-a526-dc6a356b2adf [Follower]" leader-address= leader-id= 2026-04-22T10:10:57.050Z [INFO] server-6fdcffea-780c-54c9-5b52-66b8d441d70e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a Address:77ca3a41-b7c2-8c2f-f14d-79c56d6d764a} {Suffrage:Voter ID:server-bf574c04-07df-880c-460e-efc428cd5991 Address:bf574c04-07df-880c-460e-efc428cd5991} {Suffrage:Voter ID:server-6b370faa-80e7-008d-a526-dc6a356b2adf Address:6b370faa-80e7-008d-a526-dc6a356b2adf} {Suffrage:Voter ID:server-6fdcffea-780c-54c9-5b52-66b8d441d70e Address:6fdcffea-780c-54c9-5b52-66b8d441d70e} {Suffrage:Voter ID:server-d1294074-ce41-48b4-1359-1206d3c13bbb Address:d1294074-ce41-48b4-1359-1206d3c13bbb} {Suffrage:Voter ID:server-dadab481-6273-c354-6558-8456d105bfa9 Address:dadab481-6273-c354-6558-8456d105bfa9} {Suffrage:Voter ID:server-34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3 Address:34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3}]" 2026-04-22T10:10:57.050Z [INFO] server-6fdcffea-780c-54c9-5b52-66b8d441d70e: entering follower state: follower="Node at 6fdcffea-780c-54c9-5b52-66b8d441d70e [Follower]" leader-address= leader-id= 2026-04-22T10:10:57.050Z [INFO] server-d1294074-ce41-48b4-1359-1206d3c13bbb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a Address:77ca3a41-b7c2-8c2f-f14d-79c56d6d764a} {Suffrage:Voter ID:server-bf574c04-07df-880c-460e-efc428cd5991 Address:bf574c04-07df-880c-460e-efc428cd5991} {Suffrage:Voter ID:server-6b370faa-80e7-008d-a526-dc6a356b2adf Address:6b370faa-80e7-008d-a526-dc6a356b2adf} {Suffrage:Voter ID:server-6fdcffea-780c-54c9-5b52-66b8d441d70e Address:6fdcffea-780c-54c9-5b52-66b8d441d70e} {Suffrage:Voter ID:server-d1294074-ce41-48b4-1359-1206d3c13bbb Address:d1294074-ce41-48b4-1359-1206d3c13bbb} {Suffrage:Voter ID:server-dadab481-6273-c354-6558-8456d105bfa9 Address:dadab481-6273-c354-6558-8456d105bfa9} {Suffrage:Voter ID:server-34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3 Address:34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3}]" 2026-04-22T10:10:57.051Z [INFO] server-d1294074-ce41-48b4-1359-1206d3c13bbb: entering follower state: follower="Node at d1294074-ce41-48b4-1359-1206d3c13bbb [Follower]" leader-address= leader-id= 2026-04-22T10:10:57.051Z [INFO] server-dadab481-6273-c354-6558-8456d105bfa9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a Address:77ca3a41-b7c2-8c2f-f14d-79c56d6d764a} {Suffrage:Voter ID:server-bf574c04-07df-880c-460e-efc428cd5991 Address:bf574c04-07df-880c-460e-efc428cd5991} {Suffrage:Voter ID:server-6b370faa-80e7-008d-a526-dc6a356b2adf Address:6b370faa-80e7-008d-a526-dc6a356b2adf} {Suffrage:Voter ID:server-6fdcffea-780c-54c9-5b52-66b8d441d70e Address:6fdcffea-780c-54c9-5b52-66b8d441d70e} {Suffrage:Voter ID:server-d1294074-ce41-48b4-1359-1206d3c13bbb Address:d1294074-ce41-48b4-1359-1206d3c13bbb} {Suffrage:Voter ID:server-dadab481-6273-c354-6558-8456d105bfa9 Address:dadab481-6273-c354-6558-8456d105bfa9} {Suffrage:Voter ID:server-34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3 Address:34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3}]" 2026-04-22T10:10:57.051Z [INFO] server-dadab481-6273-c354-6558-8456d105bfa9: entering follower state: follower="Node at dadab481-6273-c354-6558-8456d105bfa9 [Follower]" leader-address= leader-id= 2026-04-22T10:10:57.051Z [INFO] server-34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a Address:77ca3a41-b7c2-8c2f-f14d-79c56d6d764a} {Suffrage:Voter ID:server-bf574c04-07df-880c-460e-efc428cd5991 Address:bf574c04-07df-880c-460e-efc428cd5991} {Suffrage:Voter ID:server-6b370faa-80e7-008d-a526-dc6a356b2adf Address:6b370faa-80e7-008d-a526-dc6a356b2adf} {Suffrage:Voter ID:server-6fdcffea-780c-54c9-5b52-66b8d441d70e Address:6fdcffea-780c-54c9-5b52-66b8d441d70e} {Suffrage:Voter ID:server-d1294074-ce41-48b4-1359-1206d3c13bbb Address:d1294074-ce41-48b4-1359-1206d3c13bbb} {Suffrage:Voter ID:server-dadab481-6273-c354-6558-8456d105bfa9 Address:dadab481-6273-c354-6558-8456d105bfa9} {Suffrage:Voter ID:server-34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3 Address:34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3}]" 2026-04-22T10:10:57.051Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:57.051Z [INFO] server-34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3: entering follower state: follower="Node at 34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3 [Follower]" leader-address= leader-id= 2026-04-22T10:10:57.115Z [WARN] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:57.115Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: entering candidate state: node="Node at 77ca3a41-b7c2-8c2f-f14d-79c56d6d764a [Candidate]" term=2 2026-04-22T10:10:57.115Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: election won: term=2 tally=4 2026-04-22T10:10:57.115Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: entering leader state: leader="Node at 77ca3a41-b7c2-8c2f-f14d-79c56d6d764a [Leader]" 2026-04-22T10:10:57.115Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: added peer, starting replication: peer=server-bf574c04-07df-880c-460e-efc428cd5991 2026-04-22T10:10:57.115Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: added peer, starting replication: peer=server-6b370faa-80e7-008d-a526-dc6a356b2adf 2026-04-22T10:10:57.115Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: added peer, starting replication: peer=server-6fdcffea-780c-54c9-5b52-66b8d441d70e 2026-04-22T10:10:57.115Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: added peer, starting replication: peer=server-d1294074-ce41-48b4-1359-1206d3c13bbb 2026-04-22T10:10:57.115Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: added peer, starting replication: peer=server-dadab481-6273-c354-6558-8456d105bfa9 2026-04-22T10:10:57.115Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: added peer, starting replication: peer=server-34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3 2026-04-22T10:10:57.116Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: pipelining replication: peer="{Voter server-bf574c04-07df-880c-460e-efc428cd5991 bf574c04-07df-880c-460e-efc428cd5991}" 2026-04-22T10:10:57.116Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: pipelining replication: peer="{Voter server-6b370faa-80e7-008d-a526-dc6a356b2adf 6b370faa-80e7-008d-a526-dc6a356b2adf}" 2026-04-22T10:10:57.116Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: pipelining replication: peer="{Voter server-6fdcffea-780c-54c9-5b52-66b8d441d70e 6fdcffea-780c-54c9-5b52-66b8d441d70e}" 2026-04-22T10:10:57.116Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: pipelining replication: peer="{Voter server-d1294074-ce41-48b4-1359-1206d3c13bbb d1294074-ce41-48b4-1359-1206d3c13bbb}" 2026-04-22T10:10:57.116Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: pipelining replication: peer="{Voter server-34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3 34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3}" 2026-04-22T10:10:57.116Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: pipelining replication: peer="{Voter server-dadab481-6273-c354-6558-8456d105bfa9 dadab481-6273-c354-6558-8456d105bfa9}" 2026-04-22T10:10:57.221Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:57.115914252 +0000 UTC m=+103.139710366 (1 nodes), 64.635982ms from start of poll, 65.41921ms from cluster start. Timeout at 2026-04-22 10:10:57.220920828 +0000 UTC m=+103.244716961, 105.006595ms after stability 2026-04-22T10:10:57.221Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:57.327Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:57.221220771 +0000 UTC m=+103.245016978 (6 nodes), 3.181µs from start of poll, 170.725822ms from cluster start. Timeout at 2026-04-22 10:10:57.326245397 +0000 UTC m=+103.350041594, 105.024616ms after stability 2026-04-22T10:10:57.327Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:57.433Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:57.327249184 +0000 UTC m=+103.351045387 (1 nodes), 1.83µs from start of poll, 276.754231ms from cluster start. Timeout at 2026-04-22 10:10:57.432263477 +0000 UTC m=+103.456059575, 105.014188ms after stability 2026-04-22T10:10:57.433Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: entering candidate state: node="Node at bf574c04-07df-880c-460e-efc428cd5991 [Candidate]" term=3 2026-04-22T10:10:57.433Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:57.433Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: election won: term=3 tally=4 2026-04-22T10:10:57.433Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: entering follower state: follower="Node at 77ca3a41-b7c2-8c2f-f14d-79c56d6d764a [Follower]" leader-address= leader-id= 2026-04-22T10:10:57.433Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: entering leader state: leader="Node at bf574c04-07df-880c-460e-efc428cd5991 [Leader]" 2026-04-22T10:10:57.433Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: added peer, starting replication: peer=server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a 2026-04-22T10:10:57.433Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: aborting pipeline replication: peer="{Voter server-6b370faa-80e7-008d-a526-dc6a356b2adf 6b370faa-80e7-008d-a526-dc6a356b2adf}" 2026-04-22T10:10:57.433Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: added peer, starting replication: peer=server-6b370faa-80e7-008d-a526-dc6a356b2adf 2026-04-22T10:10:57.433Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: added peer, starting replication: peer=server-6fdcffea-780c-54c9-5b52-66b8d441d70e 2026-04-22T10:10:57.433Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: added peer, starting replication: peer=server-d1294074-ce41-48b4-1359-1206d3c13bbb 2026-04-22T10:10:57.433Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: added peer, starting replication: peer=server-dadab481-6273-c354-6558-8456d105bfa9 2026-04-22T10:10:57.433Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: added peer, starting replication: peer=server-34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3 2026-04-22T10:10:57.433Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: pipelining replication: peer="{Voter server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a 77ca3a41-b7c2-8c2f-f14d-79c56d6d764a}" 2026-04-22T10:10:57.433Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: aborting pipeline replication: peer="{Voter server-34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3 34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3}" 2026-04-22T10:10:57.433Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: pipelining replication: peer="{Voter server-34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3 34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3}" 2026-04-22T10:10:57.433Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: aborting pipeline replication: peer="{Voter server-dadab481-6273-c354-6558-8456d105bfa9 dadab481-6273-c354-6558-8456d105bfa9}" 2026-04-22T10:10:57.433Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: pipelining replication: peer="{Voter server-6b370faa-80e7-008d-a526-dc6a356b2adf 6b370faa-80e7-008d-a526-dc6a356b2adf}" 2026-04-22T10:10:57.433Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: pipelining replication: peer="{Voter server-d1294074-ce41-48b4-1359-1206d3c13bbb d1294074-ce41-48b4-1359-1206d3c13bbb}" 2026-04-22T10:10:57.433Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: aborting pipeline replication: peer="{Voter server-6fdcffea-780c-54c9-5b52-66b8d441d70e 6fdcffea-780c-54c9-5b52-66b8d441d70e}" 2026-04-22T10:10:57.433Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: aborting pipeline replication: peer="{Voter server-bf574c04-07df-880c-460e-efc428cd5991 bf574c04-07df-880c-460e-efc428cd5991}" 2026-04-22T10:10:57.433Z [INFO] server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a: aborting pipeline replication: peer="{Voter server-d1294074-ce41-48b4-1359-1206d3c13bbb d1294074-ce41-48b4-1359-1206d3c13bbb}" 2026-04-22T10:10:57.433Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: pipelining replication: peer="{Voter server-dadab481-6273-c354-6558-8456d105bfa9 dadab481-6273-c354-6558-8456d105bfa9}" 2026-04-22T10:10:57.433Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: pipelining replication: peer="{Voter server-6fdcffea-780c-54c9-5b52-66b8d441d70e 6fdcffea-780c-54c9-5b52-66b8d441d70e}" 2026-04-22T10:10:57.538Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:57.433214217 +0000 UTC m=+103.457010321 (1 nodes), 48.602µs from start of poll, 382.719165ms from cluster start. Timeout at 2026-04-22 10:10:57.53821492 +0000 UTC m=+103.562011001, 105.00068ms after stability 2026-04-22T10:10:57.538Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: aborting pipeline replication: peer="{Voter server-d1294074-ce41-48b4-1359-1206d3c13bbb d1294074-ce41-48b4-1359-1206d3c13bbb}" 2026-04-22T10:10:57.538Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: aborting pipeline replication: peer="{Voter server-34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3 34b8cf1b-1b4e-ca4d-fc64-1f1fb7f222c3}" 2026-04-22T10:10:57.538Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: aborting pipeline replication: peer="{Voter server-6b370faa-80e7-008d-a526-dc6a356b2adf 6b370faa-80e7-008d-a526-dc6a356b2adf}" 2026-04-22T10:10:57.538Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: aborting pipeline replication: peer="{Voter server-dadab481-6273-c354-6558-8456d105bfa9 dadab481-6273-c354-6558-8456d105bfa9}" 2026-04-22T10:10:57.538Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: aborting pipeline replication: peer="{Voter server-77ca3a41-b7c2-8c2f-f14d-79c56d6d764a 77ca3a41-b7c2-8c2f-f14d-79c56d6d764a}" 2026-04-22T10:10:57.538Z [INFO] server-bf574c04-07df-880c-460e-efc428cd5991: aborting pipeline replication: peer="{Voter server-6fdcffea-780c-54c9-5b52-66b8d441d70e 6fdcffea-780c-54c9-5b52-66b8d441d70e}" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.49s) === RUN TestRaft_LeadershipTransferToInvalidID 2026-04-22T10:10:57.539Z [INFO] server-9e383c8f-79cf-4589-8a9e-5878b72eca32: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e383c8f-79cf-4589-8a9e-5878b72eca32 Address:9e383c8f-79cf-4589-8a9e-5878b72eca32} {Suffrage:Voter ID:server-36370d60-e93b-f730-cd5e-9f2925538194 Address:36370d60-e93b-f730-cd5e-9f2925538194} {Suffrage:Voter ID:server-85314991-febd-66ee-ffad-ae976750197b Address:85314991-febd-66ee-ffad-ae976750197b}]" 2026-04-22T10:10:57.539Z [INFO] server-9e383c8f-79cf-4589-8a9e-5878b72eca32: entering follower state: follower="Node at 9e383c8f-79cf-4589-8a9e-5878b72eca32 [Follower]" leader-address= leader-id= 2026-04-22T10:10:57.539Z [INFO] server-36370d60-e93b-f730-cd5e-9f2925538194: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e383c8f-79cf-4589-8a9e-5878b72eca32 Address:9e383c8f-79cf-4589-8a9e-5878b72eca32} {Suffrage:Voter ID:server-36370d60-e93b-f730-cd5e-9f2925538194 Address:36370d60-e93b-f730-cd5e-9f2925538194} {Suffrage:Voter ID:server-85314991-febd-66ee-ffad-ae976750197b Address:85314991-febd-66ee-ffad-ae976750197b}]" 2026-04-22T10:10:57.539Z [INFO] server-36370d60-e93b-f730-cd5e-9f2925538194: entering follower state: follower="Node at 36370d60-e93b-f730-cd5e-9f2925538194 [Follower]" leader-address= leader-id= 2026-04-22T10:10:57.539Z [INFO] server-85314991-febd-66ee-ffad-ae976750197b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e383c8f-79cf-4589-8a9e-5878b72eca32 Address:9e383c8f-79cf-4589-8a9e-5878b72eca32} {Suffrage:Voter ID:server-36370d60-e93b-f730-cd5e-9f2925538194 Address:36370d60-e93b-f730-cd5e-9f2925538194} {Suffrage:Voter ID:server-85314991-febd-66ee-ffad-ae976750197b Address:85314991-febd-66ee-ffad-ae976750197b}]" 2026-04-22T10:10:57.539Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:57.539Z [INFO] server-85314991-febd-66ee-ffad-ae976750197b: entering follower state: follower="Node at 85314991-febd-66ee-ffad-ae976750197b [Follower]" leader-address= leader-id= 2026-04-22T10:10:57.629Z [WARN] server-36370d60-e93b-f730-cd5e-9f2925538194: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:57.629Z [INFO] server-36370d60-e93b-f730-cd5e-9f2925538194: entering candidate state: node="Node at 36370d60-e93b-f730-cd5e-9f2925538194 [Candidate]" term=2 2026-04-22T10:10:57.629Z [INFO] server-36370d60-e93b-f730-cd5e-9f2925538194: election won: term=2 tally=2 2026-04-22T10:10:57.629Z [INFO] server-36370d60-e93b-f730-cd5e-9f2925538194: entering leader state: leader="Node at 36370d60-e93b-f730-cd5e-9f2925538194 [Leader]" 2026-04-22T10:10:57.629Z [INFO] server-36370d60-e93b-f730-cd5e-9f2925538194: added peer, starting replication: peer=server-9e383c8f-79cf-4589-8a9e-5878b72eca32 2026-04-22T10:10:57.629Z [INFO] server-36370d60-e93b-f730-cd5e-9f2925538194: added peer, starting replication: peer=server-85314991-febd-66ee-ffad-ae976750197b 2026-04-22T10:10:57.629Z [INFO] server-36370d60-e93b-f730-cd5e-9f2925538194: pipelining replication: peer="{Voter server-9e383c8f-79cf-4589-8a9e-5878b72eca32 9e383c8f-79cf-4589-8a9e-5878b72eca32}" 2026-04-22T10:10:57.629Z [INFO] server-36370d60-e93b-f730-cd5e-9f2925538194: pipelining replication: peer="{Voter server-85314991-febd-66ee-ffad-ae976750197b 85314991-febd-66ee-ffad-ae976750197b}" 2026-04-22T10:10:57.734Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:57.629668175 +0000 UTC m=+103.653464267 (1 nodes), 89.919727ms from start of poll, 90.222727ms from cluster start. Timeout at 2026-04-22 10:10:57.734670792 +0000 UTC m=+103.758466874, 105.002607ms after stability 2026-04-22T10:10:57.735Z [INFO] server-36370d60-e93b-f730-cd5e-9f2925538194: aborting pipeline replication: peer="{Voter server-85314991-febd-66ee-ffad-ae976750197b 85314991-febd-66ee-ffad-ae976750197b}" 2026-04-22T10:10:57.735Z [INFO] server-36370d60-e93b-f730-cd5e-9f2925538194: aborting pipeline replication: peer="{Voter server-9e383c8f-79cf-4589-8a9e-5878b72eca32 9e383c8f-79cf-4589-8a9e-5878b72eca32}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.20s) === RUN TestRaft_LeadershipTransferToInvalidAddress 2026-04-22T10:10:57.735Z [INFO] server-f6fa42a3-cb7e-241b-1f6d-c498de3a52a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6fa42a3-cb7e-241b-1f6d-c498de3a52a5 Address:f6fa42a3-cb7e-241b-1f6d-c498de3a52a5} {Suffrage:Voter ID:server-12bdd049-3952-9719-0454-9ba3514ba81e Address:12bdd049-3952-9719-0454-9ba3514ba81e} {Suffrage:Voter ID:server-a5ccda94-ce02-81bd-b863-b0fcbfd2fcef Address:a5ccda94-ce02-81bd-b863-b0fcbfd2fcef}]" 2026-04-22T10:10:57.735Z [INFO] server-f6fa42a3-cb7e-241b-1f6d-c498de3a52a5: entering follower state: follower="Node at f6fa42a3-cb7e-241b-1f6d-c498de3a52a5 [Follower]" leader-address= leader-id= 2026-04-22T10:10:57.736Z [INFO] server-12bdd049-3952-9719-0454-9ba3514ba81e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6fa42a3-cb7e-241b-1f6d-c498de3a52a5 Address:f6fa42a3-cb7e-241b-1f6d-c498de3a52a5} {Suffrage:Voter ID:server-12bdd049-3952-9719-0454-9ba3514ba81e Address:12bdd049-3952-9719-0454-9ba3514ba81e} {Suffrage:Voter ID:server-a5ccda94-ce02-81bd-b863-b0fcbfd2fcef Address:a5ccda94-ce02-81bd-b863-b0fcbfd2fcef}]" 2026-04-22T10:10:57.736Z [INFO] server-12bdd049-3952-9719-0454-9ba3514ba81e: entering follower state: follower="Node at 12bdd049-3952-9719-0454-9ba3514ba81e [Follower]" leader-address= leader-id= 2026-04-22T10:10:57.736Z [INFO] server-a5ccda94-ce02-81bd-b863-b0fcbfd2fcef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6fa42a3-cb7e-241b-1f6d-c498de3a52a5 Address:f6fa42a3-cb7e-241b-1f6d-c498de3a52a5} {Suffrage:Voter ID:server-12bdd049-3952-9719-0454-9ba3514ba81e Address:12bdd049-3952-9719-0454-9ba3514ba81e} {Suffrage:Voter ID:server-a5ccda94-ce02-81bd-b863-b0fcbfd2fcef Address:a5ccda94-ce02-81bd-b863-b0fcbfd2fcef}]" 2026-04-22T10:10:57.736Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:57.736Z [INFO] server-a5ccda94-ce02-81bd-b863-b0fcbfd2fcef: entering follower state: follower="Node at a5ccda94-ce02-81bd-b863-b0fcbfd2fcef [Follower]" leader-address= leader-id= 2026-04-22T10:10:57.808Z [WARN] server-f6fa42a3-cb7e-241b-1f6d-c498de3a52a5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:57.808Z [INFO] server-f6fa42a3-cb7e-241b-1f6d-c498de3a52a5: entering candidate state: node="Node at f6fa42a3-cb7e-241b-1f6d-c498de3a52a5 [Candidate]" term=2 2026-04-22T10:10:57.809Z [INFO] server-f6fa42a3-cb7e-241b-1f6d-c498de3a52a5: election won: term=2 tally=2 2026-04-22T10:10:57.809Z [INFO] server-f6fa42a3-cb7e-241b-1f6d-c498de3a52a5: entering leader state: leader="Node at f6fa42a3-cb7e-241b-1f6d-c498de3a52a5 [Leader]" 2026-04-22T10:10:57.809Z [INFO] server-f6fa42a3-cb7e-241b-1f6d-c498de3a52a5: added peer, starting replication: peer=server-12bdd049-3952-9719-0454-9ba3514ba81e 2026-04-22T10:10:57.809Z [INFO] server-f6fa42a3-cb7e-241b-1f6d-c498de3a52a5: added peer, starting replication: peer=server-a5ccda94-ce02-81bd-b863-b0fcbfd2fcef 2026-04-22T10:10:57.809Z [INFO] server-f6fa42a3-cb7e-241b-1f6d-c498de3a52a5: pipelining replication: peer="{Voter server-a5ccda94-ce02-81bd-b863-b0fcbfd2fcef a5ccda94-ce02-81bd-b863-b0fcbfd2fcef}" 2026-04-22T10:10:57.809Z [INFO] server-f6fa42a3-cb7e-241b-1f6d-c498de3a52a5: pipelining replication: peer="{Voter server-12bdd049-3952-9719-0454-9ba3514ba81e 12bdd049-3952-9719-0454-9ba3514ba81e}" 2026-04-22T10:10:57.914Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:57.809174215 +0000 UTC m=+103.832970315 (2 nodes), 73.052918ms from start of poll, 73.353813ms from cluster start. Timeout at 2026-04-22 10:10:57.914178852 +0000 UTC m=+103.937974946, 105.004631ms after stability 2026-04-22T10:10:57.914Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:58.020Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:57.914395442 +0000 UTC m=+103.938191528 (1 nodes), 464ns from start of poll, 178.575026ms from cluster start. Timeout at 2026-04-22 10:10:58.019408608 +0000 UTC m=+104.043204710, 105.013182ms after stability 2026-04-22T10:10:58.020Z [INFO] server-f6fa42a3-cb7e-241b-1f6d-c498de3a52a5: aborting pipeline replication: peer="{Voter server-12bdd049-3952-9719-0454-9ba3514ba81e 12bdd049-3952-9719-0454-9ba3514ba81e}" 2026-04-22T10:10:58.020Z [INFO] server-f6fa42a3-cb7e-241b-1f6d-c498de3a52a5: aborting pipeline replication: peer="{Voter server-a5ccda94-ce02-81bd-b863-b0fcbfd2fcef a5ccda94-ce02-81bd-b863-b0fcbfd2fcef}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.29s) === RUN TestRaft_LeadershipTransferToBehindServer 2026-04-22T10:10:58.021Z [INFO] server-d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11 Address:d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11} {Suffrage:Voter ID:server-0b23f749-527c-ef04-5be2-304c03e887bf Address:0b23f749-527c-ef04-5be2-304c03e887bf} {Suffrage:Voter ID:server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25 Address:b2bbf496-81dc-e7b0-6f8c-69cb9178db25}]" 2026-04-22T10:10:58.021Z [INFO] server-d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11: entering follower state: follower="Node at d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11 [Follower]" leader-address= leader-id= 2026-04-22T10:10:58.021Z [INFO] server-0b23f749-527c-ef04-5be2-304c03e887bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11 Address:d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11} {Suffrage:Voter ID:server-0b23f749-527c-ef04-5be2-304c03e887bf Address:0b23f749-527c-ef04-5be2-304c03e887bf} {Suffrage:Voter ID:server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25 Address:b2bbf496-81dc-e7b0-6f8c-69cb9178db25}]" 2026-04-22T10:10:58.021Z [INFO] server-0b23f749-527c-ef04-5be2-304c03e887bf: entering follower state: follower="Node at 0b23f749-527c-ef04-5be2-304c03e887bf [Follower]" leader-address= leader-id= 2026-04-22T10:10:58.021Z [INFO] server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11 Address:d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11} {Suffrage:Voter ID:server-0b23f749-527c-ef04-5be2-304c03e887bf Address:0b23f749-527c-ef04-5be2-304c03e887bf} {Suffrage:Voter ID:server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25 Address:b2bbf496-81dc-e7b0-6f8c-69cb9178db25}]" 2026-04-22T10:10:58.021Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:58.021Z [INFO] server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25: entering follower state: follower="Node at b2bbf496-81dc-e7b0-6f8c-69cb9178db25 [Follower]" leader-address= leader-id= 2026-04-22T10:10:58.094Z [WARN] server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:58.094Z [INFO] server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25: entering candidate state: node="Node at b2bbf496-81dc-e7b0-6f8c-69cb9178db25 [Candidate]" term=2 2026-04-22T10:10:58.094Z [INFO] server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25: election won: term=2 tally=2 2026-04-22T10:10:58.094Z [INFO] server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25: entering leader state: leader="Node at b2bbf496-81dc-e7b0-6f8c-69cb9178db25 [Leader]" 2026-04-22T10:10:58.095Z [INFO] server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25: added peer, starting replication: peer=server-d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11 2026-04-22T10:10:58.095Z [INFO] server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25: added peer, starting replication: peer=server-0b23f749-527c-ef04-5be2-304c03e887bf 2026-04-22T10:10:58.095Z [INFO] server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25: pipelining replication: peer="{Voter server-0b23f749-527c-ef04-5be2-304c03e887bf 0b23f749-527c-ef04-5be2-304c03e887bf}" 2026-04-22T10:10:58.095Z [INFO] server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25: pipelining replication: peer="{Voter server-d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11 d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11}" 2026-04-22T10:10:58.200Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:58.095066385 +0000 UTC m=+104.118862535 (1 nodes), 73.558782ms from start of poll, 73.878448ms from cluster start. Timeout at 2026-04-22 10:10:58.200070211 +0000 UTC m=+104.223866289, 105.003754ms after stability 2026-04-22T10:10:58.200Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:58.305Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:58.200191885 +0000 UTC m=+104.223987974 (2 nodes), 795ns from start of poll, 179.003887ms from cluster start. Timeout at 2026-04-22 10:10:58.305224281 +0000 UTC m=+104.329020417, 105.032443ms after stability 2026-04-22T10:10:58.349Z [INFO] server-d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11: entering candidate state: node="Node at d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11 [Candidate]" term=3 2026-04-22T10:10:58.349Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:58.349Z [INFO] server-d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11: election won: term=3 tally=2 2026-04-22T10:10:58.349Z [INFO] server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25: entering follower state: follower="Node at b2bbf496-81dc-e7b0-6f8c-69cb9178db25 [Follower]" leader-address= leader-id= 2026-04-22T10:10:58.349Z [INFO] server-d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11: entering leader state: leader="Node at d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11 [Leader]" 2026-04-22T10:10:58.349Z [INFO] server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25: aborting pipeline replication: peer="{Voter server-0b23f749-527c-ef04-5be2-304c03e887bf 0b23f749-527c-ef04-5be2-304c03e887bf}" 2026-04-22T10:10:58.349Z [INFO] server-d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11: added peer, starting replication: peer=server-0b23f749-527c-ef04-5be2-304c03e887bf 2026-04-22T10:10:58.349Z [INFO] server-d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11: added peer, starting replication: peer=server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25 2026-04-22T10:10:58.349Z [INFO] server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25: aborting pipeline replication: peer="{Voter server-d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11 d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11}" 2026-04-22T10:10:58.349Z [INFO] server-d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11: pipelining replication: peer="{Voter server-0b23f749-527c-ef04-5be2-304c03e887bf 0b23f749-527c-ef04-5be2-304c03e887bf}" 2026-04-22T10:10:58.349Z [INFO] server-d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11: pipelining replication: peer="{Voter server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25 b2bbf496-81dc-e7b0-6f8c-69cb9178db25}" 2026-04-22T10:10:58.455Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:58.349836182 +0000 UTC m=+104.373632275 (1 nodes), 63.235µs from start of poll, 328.648188ms from cluster start. Timeout at 2026-04-22 10:10:58.454857803 +0000 UTC m=+104.478654040, 105.021765ms after stability 2026-04-22T10:10:58.455Z [INFO] server-d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11: aborting pipeline replication: peer="{Voter server-b2bbf496-81dc-e7b0-6f8c-69cb9178db25 b2bbf496-81dc-e7b0-6f8c-69cb9178db25}" 2026-04-22T10:10:58.455Z [INFO] server-d488d71c-f0a7-d28c-9d8d-3a7d9aae5d11: aborting pipeline replication: peer="{Voter server-0b23f749-527c-ef04-5be2-304c03e887bf 0b23f749-527c-ef04-5be2-304c03e887bf}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.43s) === RUN TestRaft_LeadershipTransferToItself 2026-04-22T10:10:58.456Z [INFO] server-d0efee09-c05d-7ba2-c9e8-f4c704a2cd35: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0efee09-c05d-7ba2-c9e8-f4c704a2cd35 Address:d0efee09-c05d-7ba2-c9e8-f4c704a2cd35} {Suffrage:Voter ID:server-76d5ef58-6f45-5a26-af3d-b6911b83250e Address:76d5ef58-6f45-5a26-af3d-b6911b83250e} {Suffrage:Voter ID:server-b5e7e297-b3c1-2561-7ab5-1a8c858cd19b Address:b5e7e297-b3c1-2561-7ab5-1a8c858cd19b}]" 2026-04-22T10:10:58.456Z [INFO] server-d0efee09-c05d-7ba2-c9e8-f4c704a2cd35: entering follower state: follower="Node at d0efee09-c05d-7ba2-c9e8-f4c704a2cd35 [Follower]" leader-address= leader-id= 2026-04-22T10:10:58.456Z [INFO] server-76d5ef58-6f45-5a26-af3d-b6911b83250e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0efee09-c05d-7ba2-c9e8-f4c704a2cd35 Address:d0efee09-c05d-7ba2-c9e8-f4c704a2cd35} {Suffrage:Voter ID:server-76d5ef58-6f45-5a26-af3d-b6911b83250e Address:76d5ef58-6f45-5a26-af3d-b6911b83250e} {Suffrage:Voter ID:server-b5e7e297-b3c1-2561-7ab5-1a8c858cd19b Address:b5e7e297-b3c1-2561-7ab5-1a8c858cd19b}]" 2026-04-22T10:10:58.456Z [INFO] server-76d5ef58-6f45-5a26-af3d-b6911b83250e: entering follower state: follower="Node at 76d5ef58-6f45-5a26-af3d-b6911b83250e [Follower]" leader-address= leader-id= 2026-04-22T10:10:58.456Z [INFO] server-b5e7e297-b3c1-2561-7ab5-1a8c858cd19b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0efee09-c05d-7ba2-c9e8-f4c704a2cd35 Address:d0efee09-c05d-7ba2-c9e8-f4c704a2cd35} {Suffrage:Voter ID:server-76d5ef58-6f45-5a26-af3d-b6911b83250e Address:76d5ef58-6f45-5a26-af3d-b6911b83250e} {Suffrage:Voter ID:server-b5e7e297-b3c1-2561-7ab5-1a8c858cd19b Address:b5e7e297-b3c1-2561-7ab5-1a8c858cd19b}]" 2026-04-22T10:10:58.456Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:58.456Z [INFO] server-b5e7e297-b3c1-2561-7ab5-1a8c858cd19b: entering follower state: follower="Node at b5e7e297-b3c1-2561-7ab5-1a8c858cd19b [Follower]" leader-address= leader-id= 2026-04-22T10:10:58.538Z [WARN] server-b5e7e297-b3c1-2561-7ab5-1a8c858cd19b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:58.538Z [INFO] server-b5e7e297-b3c1-2561-7ab5-1a8c858cd19b: entering candidate state: node="Node at b5e7e297-b3c1-2561-7ab5-1a8c858cd19b [Candidate]" term=2 2026-04-22T10:10:58.538Z [INFO] server-b5e7e297-b3c1-2561-7ab5-1a8c858cd19b: election won: term=2 tally=2 2026-04-22T10:10:58.538Z [INFO] server-b5e7e297-b3c1-2561-7ab5-1a8c858cd19b: entering leader state: leader="Node at b5e7e297-b3c1-2561-7ab5-1a8c858cd19b [Leader]" 2026-04-22T10:10:58.538Z [INFO] server-b5e7e297-b3c1-2561-7ab5-1a8c858cd19b: added peer, starting replication: peer=server-d0efee09-c05d-7ba2-c9e8-f4c704a2cd35 2026-04-22T10:10:58.538Z [INFO] server-b5e7e297-b3c1-2561-7ab5-1a8c858cd19b: added peer, starting replication: peer=server-76d5ef58-6f45-5a26-af3d-b6911b83250e 2026-04-22T10:10:58.538Z [INFO] server-b5e7e297-b3c1-2561-7ab5-1a8c858cd19b: pipelining replication: peer="{Voter server-76d5ef58-6f45-5a26-af3d-b6911b83250e 76d5ef58-6f45-5a26-af3d-b6911b83250e}" 2026-04-22T10:10:58.538Z [INFO] server-b5e7e297-b3c1-2561-7ab5-1a8c858cd19b: pipelining replication: peer="{Voter server-d0efee09-c05d-7ba2-c9e8-f4c704a2cd35 d0efee09-c05d-7ba2-c9e8-f4c704a2cd35}" 2026-04-22T10:10:58.643Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:58.5383452 +0000 UTC m=+104.562141316 (1 nodes), 82.001685ms from start of poll, 82.310348ms from cluster start. Timeout at 2026-04-22 10:10:58.643351785 +0000 UTC m=+104.667147866, 105.00655ms after stability 2026-04-22T10:10:58.643Z [INFO] server-b5e7e297-b3c1-2561-7ab5-1a8c858cd19b: cannot transfer leadership to itself 2026-04-22T10:10:58.643Z [INFO] server-b5e7e297-b3c1-2561-7ab5-1a8c858cd19b: aborting pipeline replication: peer="{Voter server-76d5ef58-6f45-5a26-af3d-b6911b83250e 76d5ef58-6f45-5a26-af3d-b6911b83250e}" 2026-04-22T10:10:58.643Z [INFO] server-b5e7e297-b3c1-2561-7ab5-1a8c858cd19b: aborting pipeline replication: peer="{Voter server-d0efee09-c05d-7ba2-c9e8-f4c704a2cd35 d0efee09-c05d-7ba2-c9e8-f4c704a2cd35}" --- PASS: TestRaft_LeadershipTransferToItself (0.19s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests 2026-04-22T10:10:58.644Z [INFO] server-3e2578ff-885b-6834-076c-3e753a07ab7b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e2578ff-885b-6834-076c-3e753a07ab7b Address:3e2578ff-885b-6834-076c-3e753a07ab7b} {Suffrage:Voter ID:server-323eda8a-bdc5-b7d2-3f2e-9fbb0fe1a48a Address:323eda8a-bdc5-b7d2-3f2e-9fbb0fe1a48a} {Suffrage:Voter ID:server-32521446-7b52-e275-8b26-f03cc3e57cc4 Address:32521446-7b52-e275-8b26-f03cc3e57cc4}]" 2026-04-22T10:10:58.644Z [INFO] server-3e2578ff-885b-6834-076c-3e753a07ab7b: entering follower state: follower="Node at 3e2578ff-885b-6834-076c-3e753a07ab7b [Follower]" leader-address= leader-id= 2026-04-22T10:10:58.644Z [INFO] server-323eda8a-bdc5-b7d2-3f2e-9fbb0fe1a48a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e2578ff-885b-6834-076c-3e753a07ab7b Address:3e2578ff-885b-6834-076c-3e753a07ab7b} {Suffrage:Voter ID:server-323eda8a-bdc5-b7d2-3f2e-9fbb0fe1a48a Address:323eda8a-bdc5-b7d2-3f2e-9fbb0fe1a48a} {Suffrage:Voter ID:server-32521446-7b52-e275-8b26-f03cc3e57cc4 Address:32521446-7b52-e275-8b26-f03cc3e57cc4}]" 2026-04-22T10:10:58.644Z [INFO] server-32521446-7b52-e275-8b26-f03cc3e57cc4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3e2578ff-885b-6834-076c-3e753a07ab7b Address:3e2578ff-885b-6834-076c-3e753a07ab7b} {Suffrage:Voter ID:server-323eda8a-bdc5-b7d2-3f2e-9fbb0fe1a48a Address:323eda8a-bdc5-b7d2-3f2e-9fbb0fe1a48a} {Suffrage:Voter ID:server-32521446-7b52-e275-8b26-f03cc3e57cc4 Address:32521446-7b52-e275-8b26-f03cc3e57cc4}]" 2026-04-22T10:10:58.644Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:58.644Z [INFO] server-323eda8a-bdc5-b7d2-3f2e-9fbb0fe1a48a: entering follower state: follower="Node at 323eda8a-bdc5-b7d2-3f2e-9fbb0fe1a48a [Follower]" leader-address= leader-id= 2026-04-22T10:10:58.644Z [INFO] server-32521446-7b52-e275-8b26-f03cc3e57cc4: entering follower state: follower="Node at 32521446-7b52-e275-8b26-f03cc3e57cc4 [Follower]" leader-address= leader-id= 2026-04-22T10:10:58.696Z [WARN] server-3e2578ff-885b-6834-076c-3e753a07ab7b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:58.696Z [INFO] server-3e2578ff-885b-6834-076c-3e753a07ab7b: entering candidate state: node="Node at 3e2578ff-885b-6834-076c-3e753a07ab7b [Candidate]" term=2 2026-04-22T10:10:58.696Z [INFO] server-3e2578ff-885b-6834-076c-3e753a07ab7b: election won: term=2 tally=2 2026-04-22T10:10:58.696Z [INFO] server-3e2578ff-885b-6834-076c-3e753a07ab7b: entering leader state: leader="Node at 3e2578ff-885b-6834-076c-3e753a07ab7b [Leader]" 2026-04-22T10:10:58.696Z [INFO] server-3e2578ff-885b-6834-076c-3e753a07ab7b: added peer, starting replication: peer=server-323eda8a-bdc5-b7d2-3f2e-9fbb0fe1a48a 2026-04-22T10:10:58.696Z [INFO] server-3e2578ff-885b-6834-076c-3e753a07ab7b: added peer, starting replication: peer=server-32521446-7b52-e275-8b26-f03cc3e57cc4 2026-04-22T10:10:58.696Z [INFO] server-3e2578ff-885b-6834-076c-3e753a07ab7b: pipelining replication: peer="{Voter server-32521446-7b52-e275-8b26-f03cc3e57cc4 32521446-7b52-e275-8b26-f03cc3e57cc4}" 2026-04-22T10:10:58.696Z [INFO] server-3e2578ff-885b-6834-076c-3e753a07ab7b: pipelining replication: peer="{Voter server-323eda8a-bdc5-b7d2-3f2e-9fbb0fe1a48a 323eda8a-bdc5-b7d2-3f2e-9fbb0fe1a48a}" 2026-04-22T10:10:58.801Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:58.696334668 +0000 UTC m=+104.720130922 (1 nodes), 51.539078ms from start of poll, 51.758775ms from cluster start. Timeout at 2026-04-22 10:10:58.801422625 +0000 UTC m=+104.825218785, 105.087863ms after stability raft_test.go:2457: waiting on future 0 raft_test.go:2457: waiting on future 1 raft_test.go:2457: waiting on future 2 raft_test.go:2457: waiting on future 3 raft_test.go:2457: waiting on future 4 raft_test.go:2457: waiting on future 5 raft_test.go:2457: waiting on future 6 2026-04-22T10:10:58.801Z [INFO] server-3e2578ff-885b-6834-076c-3e753a07ab7b: aborting pipeline replication: peer="{Voter server-32521446-7b52-e275-8b26-f03cc3e57cc4 32521446-7b52-e275-8b26-f03cc3e57cc4}" 2026-04-22T10:10:58.801Z [INFO] server-3e2578ff-885b-6834-076c-3e753a07ab7b: aborting pipeline replication: peer="{Voter server-323eda8a-bdc5-b7d2-3f2e-9fbb0fe1a48a 323eda8a-bdc5-b7d2-3f2e-9fbb0fe1a48a}" --- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.16s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout 2026-04-22T10:10:58.802Z [INFO] server-5c8c2ae5-b485-efb2-23ec-b8bda1a0e0d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c8c2ae5-b485-efb2-23ec-b8bda1a0e0d2 Address:5c8c2ae5-b485-efb2-23ec-b8bda1a0e0d2} {Suffrage:Voter ID:server-b4c6452c-bb0b-9735-aa35-12b4fdbe79a1 Address:b4c6452c-bb0b-9735-aa35-12b4fdbe79a1} {Suffrage:Voter ID:server-f16a6078-2c71-6f1f-b942-eea5ead64c67 Address:f16a6078-2c71-6f1f-b942-eea5ead64c67}]" 2026-04-22T10:10:58.802Z [INFO] server-5c8c2ae5-b485-efb2-23ec-b8bda1a0e0d2: entering follower state: follower="Node at 5c8c2ae5-b485-efb2-23ec-b8bda1a0e0d2 [Follower]" leader-address= leader-id= 2026-04-22T10:10:58.802Z [INFO] server-b4c6452c-bb0b-9735-aa35-12b4fdbe79a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c8c2ae5-b485-efb2-23ec-b8bda1a0e0d2 Address:5c8c2ae5-b485-efb2-23ec-b8bda1a0e0d2} {Suffrage:Voter ID:server-b4c6452c-bb0b-9735-aa35-12b4fdbe79a1 Address:b4c6452c-bb0b-9735-aa35-12b4fdbe79a1} {Suffrage:Voter ID:server-f16a6078-2c71-6f1f-b942-eea5ead64c67 Address:f16a6078-2c71-6f1f-b942-eea5ead64c67}]" 2026-04-22T10:10:58.802Z [INFO] server-b4c6452c-bb0b-9735-aa35-12b4fdbe79a1: entering follower state: follower="Node at b4c6452c-bb0b-9735-aa35-12b4fdbe79a1 [Follower]" leader-address= leader-id= 2026-04-22T10:10:58.802Z [INFO] server-f16a6078-2c71-6f1f-b942-eea5ead64c67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c8c2ae5-b485-efb2-23ec-b8bda1a0e0d2 Address:5c8c2ae5-b485-efb2-23ec-b8bda1a0e0d2} {Suffrage:Voter ID:server-b4c6452c-bb0b-9735-aa35-12b4fdbe79a1 Address:b4c6452c-bb0b-9735-aa35-12b4fdbe79a1} {Suffrage:Voter ID:server-f16a6078-2c71-6f1f-b942-eea5ead64c67 Address:f16a6078-2c71-6f1f-b942-eea5ead64c67}]" 2026-04-22T10:10:58.802Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:58.802Z [INFO] server-f16a6078-2c71-6f1f-b942-eea5ead64c67: entering follower state: follower="Node at f16a6078-2c71-6f1f-b942-eea5ead64c67 [Follower]" leader-address= leader-id= 2026-04-22T10:10:58.854Z [WARN] server-b4c6452c-bb0b-9735-aa35-12b4fdbe79a1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:58.854Z [INFO] server-b4c6452c-bb0b-9735-aa35-12b4fdbe79a1: entering candidate state: node="Node at b4c6452c-bb0b-9735-aa35-12b4fdbe79a1 [Candidate]" term=2 2026-04-22T10:10:58.854Z [INFO] server-b4c6452c-bb0b-9735-aa35-12b4fdbe79a1: election won: term=2 tally=2 2026-04-22T10:10:58.854Z [INFO] server-b4c6452c-bb0b-9735-aa35-12b4fdbe79a1: entering leader state: leader="Node at b4c6452c-bb0b-9735-aa35-12b4fdbe79a1 [Leader]" 2026-04-22T10:10:58.854Z [INFO] server-b4c6452c-bb0b-9735-aa35-12b4fdbe79a1: added peer, starting replication: peer=server-5c8c2ae5-b485-efb2-23ec-b8bda1a0e0d2 2026-04-22T10:10:58.854Z [INFO] server-b4c6452c-bb0b-9735-aa35-12b4fdbe79a1: added peer, starting replication: peer=server-f16a6078-2c71-6f1f-b942-eea5ead64c67 2026-04-22T10:10:58.854Z [INFO] server-b4c6452c-bb0b-9735-aa35-12b4fdbe79a1: pipelining replication: peer="{Voter server-5c8c2ae5-b485-efb2-23ec-b8bda1a0e0d2 5c8c2ae5-b485-efb2-23ec-b8bda1a0e0d2}" 2026-04-22T10:10:58.854Z [INFO] server-b4c6452c-bb0b-9735-aa35-12b4fdbe79a1: pipelining replication: peer="{Voter server-f16a6078-2c71-6f1f-b942-eea5ead64c67 f16a6078-2c71-6f1f-b942-eea5ead64c67}" 2026-04-22T10:10:58.959Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:58.854376027 +0000 UTC m=+104.878172137 (1 nodes), 51.728931ms from start of poll, 51.944616ms from cluster start. Timeout at 2026-04-22 10:10:58.959388421 +0000 UTC m=+104.983184557, 105.01242ms after stability 2026-04-22T10:10:58.959Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:59.065Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:58.960009233 +0000 UTC m=+104.983805390 (2 nodes), 1.594µs from start of poll, 157.577869ms from cluster start. Timeout at 2026-04-22 10:10:59.065026257 +0000 UTC m=+105.088822358, 105.016968ms after stability 2026-04-22T10:10:59.494Z [INFO] server-b4c6452c-bb0b-9735-aa35-12b4fdbe79a1: aborting pipeline replication: peer="{Voter server-5c8c2ae5-b485-efb2-23ec-b8bda1a0e0d2 5c8c2ae5-b485-efb2-23ec-b8bda1a0e0d2}" 2026-04-22T10:10:59.494Z [INFO] server-b4c6452c-bb0b-9735-aa35-12b4fdbe79a1: aborting pipeline replication: peer="{Voter server-f16a6078-2c71-6f1f-b942-eea5ead64c67 f16a6078-2c71-6f1f-b942-eea5ead64c67}" --- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.69s) === RUN TestRaft_LeadershipTransferIgnoresNonvoters 2026-04-22T10:10:59.495Z [INFO] server-ad2669b4-ad95-3474-513d-662790621387: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad2669b4-ad95-3474-513d-662790621387 Address:ad2669b4-ad95-3474-513d-662790621387} {Suffrage:Voter ID:server-fb18f9cf-93b9-85f5-a090-83d23d1bc918 Address:fb18f9cf-93b9-85f5-a090-83d23d1bc918}]" 2026-04-22T10:10:59.495Z [INFO] server-ad2669b4-ad95-3474-513d-662790621387: entering follower state: follower="Node at ad2669b4-ad95-3474-513d-662790621387 [Follower]" leader-address= leader-id= 2026-04-22T10:10:59.495Z [INFO] server-fb18f9cf-93b9-85f5-a090-83d23d1bc918: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad2669b4-ad95-3474-513d-662790621387 Address:ad2669b4-ad95-3474-513d-662790621387} {Suffrage:Voter ID:server-fb18f9cf-93b9-85f5-a090-83d23d1bc918 Address:fb18f9cf-93b9-85f5-a090-83d23d1bc918}]" 2026-04-22T10:10:59.495Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:59.495Z [INFO] server-fb18f9cf-93b9-85f5-a090-83d23d1bc918: entering follower state: follower="Node at fb18f9cf-93b9-85f5-a090-83d23d1bc918 [Follower]" leader-address= leader-id= 2026-04-22T10:10:59.557Z [WARN] server-fb18f9cf-93b9-85f5-a090-83d23d1bc918: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:59.557Z [INFO] server-fb18f9cf-93b9-85f5-a090-83d23d1bc918: entering candidate state: node="Node at fb18f9cf-93b9-85f5-a090-83d23d1bc918 [Candidate]" term=2 2026-04-22T10:10:59.557Z [INFO] server-fb18f9cf-93b9-85f5-a090-83d23d1bc918: election won: term=2 tally=2 2026-04-22T10:10:59.557Z [INFO] server-fb18f9cf-93b9-85f5-a090-83d23d1bc918: entering leader state: leader="Node at fb18f9cf-93b9-85f5-a090-83d23d1bc918 [Leader]" 2026-04-22T10:10:59.557Z [INFO] server-fb18f9cf-93b9-85f5-a090-83d23d1bc918: added peer, starting replication: peer=server-ad2669b4-ad95-3474-513d-662790621387 2026-04-22T10:10:59.557Z [INFO] server-fb18f9cf-93b9-85f5-a090-83d23d1bc918: pipelining replication: peer="{Voter server-ad2669b4-ad95-3474-513d-662790621387 ad2669b4-ad95-3474-513d-662790621387}" 2026-04-22T10:10:59.663Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:59.557527007 +0000 UTC m=+105.581323095 (1 nodes), 62.254982ms from start of poll, 62.417796ms from cluster start. Timeout at 2026-04-22 10:10:59.662557858 +0000 UTC m=+105.686353935, 105.03084ms after stability 2026-04-22T10:10:59.663Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:59.769Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:59.663386955 +0000 UTC m=+105.687183044 (1 nodes), 541ns from start of poll, 168.277745ms from cluster start. Timeout at 2026-04-22 10:10:59.768427958 +0000 UTC m=+105.792224079, 105.041035ms after stability 2026-04-22T10:10:59.769Z [INFO] server-fb18f9cf-93b9-85f5-a090-83d23d1bc918: updating configuration: command=DemoteVoter server-id=server-ad2669b4-ad95-3474-513d-662790621387 server-addr= servers="[{Suffrage:Nonvoter ID:server-ad2669b4-ad95-3474-513d-662790621387 Address:ad2669b4-ad95-3474-513d-662790621387} {Suffrage:Voter ID:server-fb18f9cf-93b9-85f5-a090-83d23d1bc918 Address:fb18f9cf-93b9-85f5-a090-83d23d1bc918}]" 2026-04-22T10:10:59.769Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:10:59.874Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:10:59.769652172 +0000 UTC m=+105.793448260 (1 nodes), 527ns from start of poll, 274.542961ms from cluster start. Timeout at 2026-04-22 10:10:59.874662447 +0000 UTC m=+105.898458556, 105.010296ms after stability 2026-04-22T10:10:59.874Z [INFO] server-fb18f9cf-93b9-85f5-a090-83d23d1bc918: aborting pipeline replication: peer="{Voter server-ad2669b4-ad95-3474-513d-662790621387 ad2669b4-ad95-3474-513d-662790621387}" --- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.38s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap 2026-04-22T10:10:59.875Z [INFO] server-cff6fabb-86d4-b28c-5cc9-fa04da23c815: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cff6fabb-86d4-b28c-5cc9-fa04da23c815 Address:cff6fabb-86d4-b28c-5cc9-fa04da23c815} {Suffrage:Voter ID:server-91e05e7d-5a89-c54b-97ed-6edc9e7b3667 Address:91e05e7d-5a89-c54b-97ed-6edc9e7b3667}]" 2026-04-22T10:10:59.875Z [INFO] server-cff6fabb-86d4-b28c-5cc9-fa04da23c815: entering follower state: follower="Node at cff6fabb-86d4-b28c-5cc9-fa04da23c815 [Follower]" leader-address= leader-id= 2026-04-22T10:10:59.875Z [INFO] server-91e05e7d-5a89-c54b-97ed-6edc9e7b3667: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cff6fabb-86d4-b28c-5cc9-fa04da23c815 Address:cff6fabb-86d4-b28c-5cc9-fa04da23c815} {Suffrage:Voter ID:server-91e05e7d-5a89-c54b-97ed-6edc9e7b3667 Address:91e05e7d-5a89-c54b-97ed-6edc9e7b3667}]" 2026-04-22T10:10:59.875Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:10:59.875Z [INFO] server-91e05e7d-5a89-c54b-97ed-6edc9e7b3667: entering follower state: follower="Node at 91e05e7d-5a89-c54b-97ed-6edc9e7b3667 [Follower]" leader-address= leader-id= 2026-04-22T10:10:59.932Z [WARN] server-cff6fabb-86d4-b28c-5cc9-fa04da23c815: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:10:59.932Z [INFO] server-cff6fabb-86d4-b28c-5cc9-fa04da23c815: entering candidate state: node="Node at cff6fabb-86d4-b28c-5cc9-fa04da23c815 [Candidate]" term=2 2026-04-22T10:10:59.932Z [INFO] server-cff6fabb-86d4-b28c-5cc9-fa04da23c815: election won: term=2 tally=2 2026-04-22T10:10:59.932Z [INFO] server-cff6fabb-86d4-b28c-5cc9-fa04da23c815: entering leader state: leader="Node at cff6fabb-86d4-b28c-5cc9-fa04da23c815 [Leader]" 2026-04-22T10:10:59.932Z [INFO] server-cff6fabb-86d4-b28c-5cc9-fa04da23c815: added peer, starting replication: peer=server-91e05e7d-5a89-c54b-97ed-6edc9e7b3667 2026-04-22T10:10:59.932Z [INFO] server-cff6fabb-86d4-b28c-5cc9-fa04da23c815: pipelining replication: peer="{Voter server-91e05e7d-5a89-c54b-97ed-6edc9e7b3667 91e05e7d-5a89-c54b-97ed-6edc9e7b3667}" 2026-04-22T10:11:00.037Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:10:59.932233109 +0000 UTC m=+105.956029259 (1 nodes), 56.709992ms from start of poll, 56.880935ms from cluster start. Timeout at 2026-04-22 10:11:00.0372395 +0000 UTC m=+106.061035640, 105.006381ms after stability 2026-04-22T10:11:00.037Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:11:00.142Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:11:00.037322588 +0000 UTC m=+106.061118676 (1 nodes), 487ns from start of poll, 161.970352ms from cluster start. Timeout at 2026-04-22 10:11:00.14232888 +0000 UTC m=+106.166124991, 105.006315ms after stability 2026-04-22T10:11:00.154Z [INFO] server-91e05e7d-5a89-c54b-97ed-6edc9e7b3667: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cff6fabb-86d4-b28c-5cc9-fa04da23c815 Address:cff6fabb-86d4-b28c-5cc9-fa04da23c815} {Suffrage:Voter ID:server-91e05e7d-5a89-c54b-97ed-6edc9e7b3667 Address:91e05e7d-5a89-c54b-97ed-6edc9e7b3667}]" 2026-04-22T10:11:00.154Z [INFO] server-cff6fabb-86d4-b28c-5cc9-fa04da23c815: aborting pipeline replication: peer="{Voter server-91e05e7d-5a89-c54b-97ed-6edc9e7b3667 91e05e7d-5a89-c54b-97ed-6edc9e7b3667}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.28s) === RUN TestRaft_LogStoreIsMonotonic 2026-04-22T10:11:00.155Z [INFO] server-b7202c2b-609f-3985-3739-b2be37503658: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b7202c2b-609f-3985-3739-b2be37503658 Address:b7202c2b-609f-3985-3739-b2be37503658}]" 2026-04-22T10:11:00.155Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:11:00.155Z [INFO] server-b7202c2b-609f-3985-3739-b2be37503658: entering follower state: follower="Node at b7202c2b-609f-3985-3739-b2be37503658 [Follower]" leader-address= leader-id= 2026-04-22T10:11:00.210Z [WARN] server-b7202c2b-609f-3985-3739-b2be37503658: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:11:00.210Z [INFO] server-b7202c2b-609f-3985-3739-b2be37503658: entering candidate state: node="Node at b7202c2b-609f-3985-3739-b2be37503658 [Candidate]" term=2 2026-04-22T10:11:00.210Z [INFO] server-b7202c2b-609f-3985-3739-b2be37503658: election won: term=2 tally=1 2026-04-22T10:11:00.210Z [INFO] server-b7202c2b-609f-3985-3739-b2be37503658: entering leader state: leader="Node at b7202c2b-609f-3985-3739-b2be37503658 [Leader]" 2026-04-22T10:11:00.316Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:11:00.210786492 +0000 UTC m=+106.234582582 (1 nodes), 55.368569ms from start of poll, 55.451864ms from cluster start. Timeout at 2026-04-22 10:11:00.31580487 +0000 UTC m=+106.339601006, 105.018424ms after stability --- PASS: TestRaft_LogStoreIsMonotonic (0.16s) === RUN TestRaft_CacheLogWithStoreError 2026-04-22T10:11:00.317Z [INFO] server-fdddf404-8788-b29f-a092-9b27117c0647: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fdddf404-8788-b29f-a092-9b27117c0647 Address:fdddf404-8788-b29f-a092-9b27117c0647} {Suffrage:Voter ID:server-70be57f5-d9b6-84d0-0fa0-abe54230de90 Address:70be57f5-d9b6-84d0-0fa0-abe54230de90}]" 2026-04-22T10:11:00.317Z [INFO] server-fdddf404-8788-b29f-a092-9b27117c0647: entering follower state: follower="Node at fdddf404-8788-b29f-a092-9b27117c0647 [Follower]" leader-address= leader-id= 2026-04-22T10:11:00.317Z [INFO] server-70be57f5-d9b6-84d0-0fa0-abe54230de90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fdddf404-8788-b29f-a092-9b27117c0647 Address:fdddf404-8788-b29f-a092-9b27117c0647} {Suffrage:Voter ID:server-70be57f5-d9b6-84d0-0fa0-abe54230de90 Address:70be57f5-d9b6-84d0-0fa0-abe54230de90}]" 2026-04-22T10:11:00.317Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:11:00.317Z [INFO] server-70be57f5-d9b6-84d0-0fa0-abe54230de90: entering follower state: follower="Node at 70be57f5-d9b6-84d0-0fa0-abe54230de90 [Follower]" leader-address= leader-id= 2026-04-22T10:11:00.384Z [WARN] server-fdddf404-8788-b29f-a092-9b27117c0647: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:11:00.384Z [INFO] server-fdddf404-8788-b29f-a092-9b27117c0647: entering candidate state: node="Node at fdddf404-8788-b29f-a092-9b27117c0647 [Candidate]" term=2 2026-04-22T10:11:00.384Z [INFO] server-fdddf404-8788-b29f-a092-9b27117c0647: election won: term=2 tally=2 2026-04-22T10:11:00.384Z [INFO] server-fdddf404-8788-b29f-a092-9b27117c0647: entering leader state: leader="Node at fdddf404-8788-b29f-a092-9b27117c0647 [Leader]" 2026-04-22T10:11:00.384Z [INFO] server-fdddf404-8788-b29f-a092-9b27117c0647: added peer, starting replication: peer=server-70be57f5-d9b6-84d0-0fa0-abe54230de90 2026-04-22T10:11:00.384Z [INFO] server-fdddf404-8788-b29f-a092-9b27117c0647: pipelining replication: peer="{Voter server-70be57f5-d9b6-84d0-0fa0-abe54230de90 70be57f5-d9b6-84d0-0fa0-abe54230de90}" 2026-04-22T10:11:00.490Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:11:00.384784574 +0000 UTC m=+106.408580666 (1 nodes), 67.015597ms from start of poll, 67.795255ms from cluster start. Timeout at 2026-04-22 10:11:00.489787443 +0000 UTC m=+106.513583526, 105.00286ms after stability 2026-04-22T10:11:00.490Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:11:00.595Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:11:00.490194014 +0000 UTC m=+106.513990101 (1 nodes), 479ns from start of poll, 173.20469ms from cluster start. Timeout at 2026-04-22 10:11:00.595231652 +0000 UTC m=+106.619027763, 105.037662ms after stability 2026-04-22T10:11:00.595Z [ERROR] server-70be57f5-d9b6-84d0-0fa0-abe54230de90: failed to append to logs: error="unable to store logs within log store, err: "some error"" 2026-04-22T10:11:00.595Z [WARN] server-70be57f5-d9b6-84d0-0fa0-abe54230de90: failed to get previous log: previous-index=9 last-index=7 error="log not found" 2026-04-22T10:11:00.595Z [INFO] server-fdddf404-8788-b29f-a092-9b27117c0647: aborting pipeline replication: peer="{Voter server-70be57f5-d9b6-84d0-0fa0-abe54230de90 70be57f5-d9b6-84d0-0fa0-abe54230de90}" 2026-04-22T10:11:00.604Z [INFO] server-fdddf404-8788-b29f-a092-9b27117c0647: pipelining replication: peer="{Voter server-70be57f5-d9b6-84d0-0fa0-abe54230de90 70be57f5-d9b6-84d0-0fa0-abe54230de90}" 2026-04-22T10:11:00.605Z [INFO] server-70be57f5-d9b6-84d0-0fa0-abe54230de90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fdddf404-8788-b29f-a092-9b27117c0647 Address:fdddf404-8788-b29f-a092-9b27117c0647} {Suffrage:Voter ID:server-70be57f5-d9b6-84d0-0fa0-abe54230de90 Address:70be57f5-d9b6-84d0-0fa0-abe54230de90}]" 2026-04-22T10:11:00.605Z [INFO] server-fdddf404-8788-b29f-a092-9b27117c0647: aborting pipeline replication: peer="{Voter server-70be57f5-d9b6-84d0-0fa0-abe54230de90 70be57f5-d9b6-84d0-0fa0-abe54230de90}" 2026-04-22T10:11:01.106Z [ERROR] server-fdddf404-8788-b29f-a092-9b27117c0647: failed to heartbeat to: peer=70be57f5-d9b6-84d0-0fa0-abe54230de90 backoff time=10ms error="command timed out" --- PASS: TestRaft_CacheLogWithStoreError (0.79s) === RUN TestRaft_ReloadConfig 2026-04-22T10:11:01.107Z [INFO] server-14bddd41-5a99-d744-87fe-84d6d56512f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14bddd41-5a99-d744-87fe-84d6d56512f8 Address:14bddd41-5a99-d744-87fe-84d6d56512f8}]" 2026-04-22T10:11:01.107Z [INFO] server-14bddd41-5a99-d744-87fe-84d6d56512f8: entering follower state: follower="Node at 14bddd41-5a99-d744-87fe-84d6d56512f8 [Shutdown]" leader-address= leader-id= --- PASS: TestRaft_ReloadConfig (0.00s) === RUN TestRaft_ReloadConfigValidates 2026-04-22T10:11:01.108Z [INFO] server-c83e52c7-1175-46bd-aadd-78edbb599e90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c83e52c7-1175-46bd-aadd-78edbb599e90 Address:c83e52c7-1175-46bd-aadd-78edbb599e90}]" --- PASS: TestRaft_ReloadConfigValidates (0.00s) === RUN TestRaft_InstallSnapshot_InvalidPeers 2026-04-22T10:11:01.108Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)" --- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s) === RUN TestRaft_VoteNotGranted_WhenNodeNotInCluster 2026-04-22T10:11:01.109Z [INFO] server-eef27240-9675-43d5-15d8-4da355a5007e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eef27240-9675-43d5-15d8-4da355a5007e Address:eef27240-9675-43d5-15d8-4da355a5007e} {Suffrage:Voter ID:server-c050fba8-5421-1404-bc73-6670e7fae1a4 Address:c050fba8-5421-1404-bc73-6670e7fae1a4} {Suffrage:Voter ID:server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed Address:9d694bfd-cdfa-ddf1-d72c-8c6e108652ed}]" 2026-04-22T10:11:01.109Z [INFO] server-eef27240-9675-43d5-15d8-4da355a5007e: entering follower state: follower="Node at eef27240-9675-43d5-15d8-4da355a5007e [Follower]" leader-address= leader-id= 2026-04-22T10:11:01.109Z [INFO] server-c050fba8-5421-1404-bc73-6670e7fae1a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eef27240-9675-43d5-15d8-4da355a5007e Address:eef27240-9675-43d5-15d8-4da355a5007e} {Suffrage:Voter ID:server-c050fba8-5421-1404-bc73-6670e7fae1a4 Address:c050fba8-5421-1404-bc73-6670e7fae1a4} {Suffrage:Voter ID:server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed Address:9d694bfd-cdfa-ddf1-d72c-8c6e108652ed}]" 2026-04-22T10:11:01.109Z [INFO] server-c050fba8-5421-1404-bc73-6670e7fae1a4: entering follower state: follower="Node at c050fba8-5421-1404-bc73-6670e7fae1a4 [Follower]" leader-address= leader-id= 2026-04-22T10:11:01.109Z [INFO] server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eef27240-9675-43d5-15d8-4da355a5007e Address:eef27240-9675-43d5-15d8-4da355a5007e} {Suffrage:Voter ID:server-c050fba8-5421-1404-bc73-6670e7fae1a4 Address:c050fba8-5421-1404-bc73-6670e7fae1a4} {Suffrage:Voter ID:server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed Address:9d694bfd-cdfa-ddf1-d72c-8c6e108652ed}]" 2026-04-22T10:11:01.109Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:11:01.109Z [INFO] server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed: entering follower state: follower="Node at 9d694bfd-cdfa-ddf1-d72c-8c6e108652ed [Follower]" leader-address= leader-id= 2026-04-22T10:11:01.196Z [WARN] server-c050fba8-5421-1404-bc73-6670e7fae1a4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:11:01.196Z [INFO] server-c050fba8-5421-1404-bc73-6670e7fae1a4: entering candidate state: node="Node at c050fba8-5421-1404-bc73-6670e7fae1a4 [Candidate]" term=2 2026-04-22T10:11:01.196Z [INFO] server-c050fba8-5421-1404-bc73-6670e7fae1a4: election won: term=2 tally=2 2026-04-22T10:11:01.196Z [INFO] server-c050fba8-5421-1404-bc73-6670e7fae1a4: entering leader state: leader="Node at c050fba8-5421-1404-bc73-6670e7fae1a4 [Leader]" 2026-04-22T10:11:01.196Z [INFO] server-c050fba8-5421-1404-bc73-6670e7fae1a4: added peer, starting replication: peer=server-eef27240-9675-43d5-15d8-4da355a5007e 2026-04-22T10:11:01.196Z [INFO] server-c050fba8-5421-1404-bc73-6670e7fae1a4: added peer, starting replication: peer=server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed 2026-04-22T10:11:01.196Z [INFO] server-c050fba8-5421-1404-bc73-6670e7fae1a4: pipelining replication: peer="{Voter server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed 9d694bfd-cdfa-ddf1-d72c-8c6e108652ed}" 2026-04-22T10:11:01.196Z [INFO] server-c050fba8-5421-1404-bc73-6670e7fae1a4: pipelining replication: peer="{Voter server-eef27240-9675-43d5-15d8-4da355a5007e eef27240-9675-43d5-15d8-4da355a5007e}" 2026-04-22T10:11:01.301Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:11:01.196597136 +0000 UTC m=+107.220393286 (1 nodes), 87.330477ms from start of poll, 87.56987ms from cluster start. Timeout at 2026-04-22 10:11:01.301601032 +0000 UTC m=+107.325397113, 105.003827ms after stability 2026-04-22T10:11:01.307Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:11:01.412Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:11:01.307508817 +0000 UTC m=+107.331304905 (2 nodes), 717ns from start of poll, 198.481489ms from cluster start. Timeout at 2026-04-22 10:11:01.412527564 +0000 UTC m=+107.436323767, 105.018862ms after stability 2026-04-22T10:11:01.412Z [INFO] server-c050fba8-5421-1404-bc73-6670e7fae1a4: updating configuration: command=RemoveServer server-id=server-eef27240-9675-43d5-15d8-4da355a5007e server-addr= servers="[{Suffrage:Voter ID:server-c050fba8-5421-1404-bc73-6670e7fae1a4 Address:c050fba8-5421-1404-bc73-6670e7fae1a4} {Suffrage:Voter ID:server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed Address:9d694bfd-cdfa-ddf1-d72c-8c6e108652ed}]" 2026-04-22T10:11:01.412Z [INFO] server-c050fba8-5421-1404-bc73-6670e7fae1a4: removed peer, stopping replication: peer=server-eef27240-9675-43d5-15d8-4da355a5007e last-index=3 2026-04-22T10:11:01.412Z [INFO] server-c050fba8-5421-1404-bc73-6670e7fae1a4: aborting pipeline replication: peer="{Voter server-eef27240-9675-43d5-15d8-4da355a5007e eef27240-9675-43d5-15d8-4da355a5007e}" 2026-04-22T10:11:01.519Z [ERROR] server-c050fba8-5421-1404-bc73-6670e7fae1a4: failed to heartbeat to: peer=9d694bfd-cdfa-ddf1-d72c-8c6e108652ed backoff time=10ms error="failed to connect to peer: 9d694bfd-cdfa-ddf1-d72c-8c6e108652ed" 2026-04-22T10:11:01.520Z [ERROR] server-c050fba8-5421-1404-bc73-6670e7fae1a4: failed to pipeline appendEntries: peer="{Voter server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed 9d694bfd-cdfa-ddf1-d72c-8c6e108652ed}" error="append pipeline closed" 2026-04-22T10:11:01.520Z [INFO] server-c050fba8-5421-1404-bc73-6670e7fae1a4: aborting pipeline replication: peer="{Voter server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed 9d694bfd-cdfa-ddf1-d72c-8c6e108652ed}" 2026-04-22T10:11:01.527Z [ERROR] server-c050fba8-5421-1404-bc73-6670e7fae1a4: failed to appendEntries to: peer="{Voter server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed 9d694bfd-cdfa-ddf1-d72c-8c6e108652ed}" error="failed to connect to peer: 9d694bfd-cdfa-ddf1-d72c-8c6e108652ed" 2026-04-22T10:11:01.528Z [WARN] server-eef27240-9675-43d5-15d8-4da355a5007e: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2026-04-22T10:11:01.539Z [ERROR] server-c050fba8-5421-1404-bc73-6670e7fae1a4: failed to heartbeat to: peer=9d694bfd-cdfa-ddf1-d72c-8c6e108652ed backoff time=10ms error="failed to connect to peer: 9d694bfd-cdfa-ddf1-d72c-8c6e108652ed" 2026-04-22T10:11:01.543Z [ERROR] server-c050fba8-5421-1404-bc73-6670e7fae1a4: failed to appendEntries to: peer="{Voter server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed 9d694bfd-cdfa-ddf1-d72c-8c6e108652ed}" error="failed to connect to peer: 9d694bfd-cdfa-ddf1-d72c-8c6e108652ed" 2026-04-22T10:11:01.560Z [ERROR] server-c050fba8-5421-1404-bc73-6670e7fae1a4: failed to heartbeat to: peer=9d694bfd-cdfa-ddf1-d72c-8c6e108652ed backoff time=10ms error="failed to connect to peer: 9d694bfd-cdfa-ddf1-d72c-8c6e108652ed" 2026-04-22T10:11:01.564Z [ERROR] server-c050fba8-5421-1404-bc73-6670e7fae1a4: failed to appendEntries to: peer="{Voter server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed 9d694bfd-cdfa-ddf1-d72c-8c6e108652ed}" error="failed to connect to peer: 9d694bfd-cdfa-ddf1-d72c-8c6e108652ed" 2026-04-22T10:11:01.564Z [WARN] server-c050fba8-5421-1404-bc73-6670e7fae1a4: failed to contact: server-id=server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed time=50.96708ms 2026-04-22T10:11:01.564Z [WARN] server-c050fba8-5421-1404-bc73-6670e7fae1a4: failed to contact quorum of nodes, stepping down 2026-04-22T10:11:01.564Z [INFO] server-c050fba8-5421-1404-bc73-6670e7fae1a4: entering follower state: follower="Node at c050fba8-5421-1404-bc73-6670e7fae1a4 [Follower]" leader-address= leader-id= 2026-04-22T10:11:01.593Z [WARN] server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed: heartbeat timeout reached, starting election: last-leader-addr=c050fba8-5421-1404-bc73-6670e7fae1a4 last-leader-id=server-c050fba8-5421-1404-bc73-6670e7fae1a4 2026-04-22T10:11:01.593Z [INFO] server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed: entering candidate state: node="Node at 9d694bfd-cdfa-ddf1-d72c-8c6e108652ed [Candidate]" term=3 2026-04-22T10:11:01.593Z [ERROR] server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed: failed to make requestVote RPC: target="{Voter server-c050fba8-5421-1404-bc73-6670e7fae1a4 c050fba8-5421-1404-bc73-6670e7fae1a4}" error="failed to connect to peer: c050fba8-5421-1404-bc73-6670e7fae1a4" term=3 2026-04-22T10:11:01.624Z [WARN] server-9d694bfd-cdfa-ddf1-d72c-8c6e108652ed: rejecting vote request since node is not in configuration: from=eef27240-9675-43d5-15d8-4da355a5007e --- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.52s) === RUN TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin 2026-04-22T10:11:01.625Z [INFO] server-c2714c75-176b-29b5-f842-4eb2faa7e453: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2714c75-176b-29b5-f842-4eb2faa7e453 Address:c2714c75-176b-29b5-f842-4eb2faa7e453} {Suffrage:Voter ID:server-be5436b9-396e-6576-ac1d-f3fac73e6ffa Address:be5436b9-396e-6576-ac1d-f3fac73e6ffa} {Suffrage:Voter ID:server-69bde12a-d80b-e84c-c422-5a8d3f0063eb Address:69bde12a-d80b-e84c-c422-5a8d3f0063eb}]" 2026-04-22T10:11:01.625Z [INFO] server-c2714c75-176b-29b5-f842-4eb2faa7e453: entering follower state: follower="Node at c2714c75-176b-29b5-f842-4eb2faa7e453 [Follower]" leader-address= leader-id= 2026-04-22T10:11:01.626Z [INFO] server-be5436b9-396e-6576-ac1d-f3fac73e6ffa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2714c75-176b-29b5-f842-4eb2faa7e453 Address:c2714c75-176b-29b5-f842-4eb2faa7e453} {Suffrage:Voter ID:server-be5436b9-396e-6576-ac1d-f3fac73e6ffa Address:be5436b9-396e-6576-ac1d-f3fac73e6ffa} {Suffrage:Voter ID:server-69bde12a-d80b-e84c-c422-5a8d3f0063eb Address:69bde12a-d80b-e84c-c422-5a8d3f0063eb}]" 2026-04-22T10:11:01.626Z [INFO] server-be5436b9-396e-6576-ac1d-f3fac73e6ffa: entering follower state: follower="Node at be5436b9-396e-6576-ac1d-f3fac73e6ffa [Follower]" leader-address= leader-id= 2026-04-22T10:11:01.626Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2714c75-176b-29b5-f842-4eb2faa7e453 Address:c2714c75-176b-29b5-f842-4eb2faa7e453} {Suffrage:Voter ID:server-be5436b9-396e-6576-ac1d-f3fac73e6ffa Address:be5436b9-396e-6576-ac1d-f3fac73e6ffa} {Suffrage:Voter ID:server-69bde12a-d80b-e84c-c422-5a8d3f0063eb Address:69bde12a-d80b-e84c-c422-5a8d3f0063eb}]" 2026-04-22T10:11:01.626Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:11:01.626Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: entering follower state: follower="Node at 69bde12a-d80b-e84c-c422-5a8d3f0063eb [Follower]" leader-address= leader-id= 2026-04-22T10:11:01.701Z [WARN] server-c2714c75-176b-29b5-f842-4eb2faa7e453: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:11:01.701Z [INFO] server-c2714c75-176b-29b5-f842-4eb2faa7e453: entering candidate state: node="Node at c2714c75-176b-29b5-f842-4eb2faa7e453 [Candidate]" term=2 2026-04-22T10:11:01.701Z [INFO] server-c2714c75-176b-29b5-f842-4eb2faa7e453: election won: term=2 tally=2 2026-04-22T10:11:01.701Z [INFO] server-c2714c75-176b-29b5-f842-4eb2faa7e453: entering leader state: leader="Node at c2714c75-176b-29b5-f842-4eb2faa7e453 [Leader]" 2026-04-22T10:11:01.701Z [INFO] server-c2714c75-176b-29b5-f842-4eb2faa7e453: added peer, starting replication: peer=server-be5436b9-396e-6576-ac1d-f3fac73e6ffa 2026-04-22T10:11:01.701Z [INFO] server-c2714c75-176b-29b5-f842-4eb2faa7e453: added peer, starting replication: peer=server-69bde12a-d80b-e84c-c422-5a8d3f0063eb 2026-04-22T10:11:01.701Z [INFO] server-c2714c75-176b-29b5-f842-4eb2faa7e453: pipelining replication: peer="{Voter server-69bde12a-d80b-e84c-c422-5a8d3f0063eb 69bde12a-d80b-e84c-c422-5a8d3f0063eb}" 2026-04-22T10:11:01.701Z [INFO] server-c2714c75-176b-29b5-f842-4eb2faa7e453: pipelining replication: peer="{Voter server-be5436b9-396e-6576-ac1d-f3fac73e6ffa be5436b9-396e-6576-ac1d-f3fac73e6ffa}" 2026-04-22T10:11:01.806Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:11:01.70125104 +0000 UTC m=+107.725047158 (1 nodes), 75.100766ms from start of poll, 75.417871ms from cluster start. Timeout at 2026-04-22 10:11:01.806256568 +0000 UTC m=+107.830052647, 105.005489ms after stability 2026-04-22T10:11:01.811Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:11:01.917Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:11:01.811992937 +0000 UTC m=+107.835789087 (2 nodes), 1.624µs from start of poll, 186.1598ms from cluster start. Timeout at 2026-04-22 10:11:01.917034535 +0000 UTC m=+107.940830616, 105.041529ms after stability 2026-04-22T10:11:01.918Z [ERROR] server-c2714c75-176b-29b5-f842-4eb2faa7e453: failed to pipeline appendEntries: peer="{Voter server-be5436b9-396e-6576-ac1d-f3fac73e6ffa be5436b9-396e-6576-ac1d-f3fac73e6ffa}" error="append pipeline closed" 2026-04-22T10:11:01.918Z [INFO] server-c2714c75-176b-29b5-f842-4eb2faa7e453: aborting pipeline replication: peer="{Voter server-be5436b9-396e-6576-ac1d-f3fac73e6ffa be5436b9-396e-6576-ac1d-f3fac73e6ffa}" 2026-04-22T10:11:01.919Z [ERROR] server-c2714c75-176b-29b5-f842-4eb2faa7e453: failed to heartbeat to: peer=be5436b9-396e-6576-ac1d-f3fac73e6ffa backoff time=10ms error="failed to connect to peer: be5436b9-396e-6576-ac1d-f3fac73e6ffa" 2026-04-22T10:11:01.924Z [ERROR] server-c2714c75-176b-29b5-f842-4eb2faa7e453: failed to appendEntries to: peer="{Voter server-be5436b9-396e-6576-ac1d-f3fac73e6ffa be5436b9-396e-6576-ac1d-f3fac73e6ffa}" error="failed to connect to peer: be5436b9-396e-6576-ac1d-f3fac73e6ffa" 2026-04-22T10:11:01.935Z [ERROR] server-c2714c75-176b-29b5-f842-4eb2faa7e453: failed to heartbeat to: peer=be5436b9-396e-6576-ac1d-f3fac73e6ffa backoff time=10ms error="failed to connect to peer: be5436b9-396e-6576-ac1d-f3fac73e6ffa" 2026-04-22T10:11:01.941Z [ERROR] server-c2714c75-176b-29b5-f842-4eb2faa7e453: failed to appendEntries to: peer="{Voter server-be5436b9-396e-6576-ac1d-f3fac73e6ffa be5436b9-396e-6576-ac1d-f3fac73e6ffa}" error="failed to connect to peer: be5436b9-396e-6576-ac1d-f3fac73e6ffa" 2026-04-22T10:11:01.954Z [ERROR] server-c2714c75-176b-29b5-f842-4eb2faa7e453: failed to heartbeat to: peer=be5436b9-396e-6576-ac1d-f3fac73e6ffa backoff time=10ms error="failed to connect to peer: be5436b9-396e-6576-ac1d-f3fac73e6ffa" 2026-04-22T10:11:01.959Z [ERROR] server-c2714c75-176b-29b5-f842-4eb2faa7e453: failed to appendEntries to: peer="{Voter server-be5436b9-396e-6576-ac1d-f3fac73e6ffa be5436b9-396e-6576-ac1d-f3fac73e6ffa}" error="failed to connect to peer: be5436b9-396e-6576-ac1d-f3fac73e6ffa" 2026-04-22T10:11:01.963Z [WARN] server-c2714c75-176b-29b5-f842-4eb2faa7e453: failed to contact: server-id=server-be5436b9-396e-6576-ac1d-f3fac73e6ffa time=50.272197ms 2026-04-22T10:11:01.973Z [ERROR] server-c2714c75-176b-29b5-f842-4eb2faa7e453: failed to heartbeat to: peer=be5436b9-396e-6576-ac1d-f3fac73e6ffa backoff time=20ms error="failed to connect to peer: be5436b9-396e-6576-ac1d-f3fac73e6ffa" 2026-04-22T10:11:01.987Z [ERROR] server-c2714c75-176b-29b5-f842-4eb2faa7e453: failed to appendEntries to: peer="{Voter server-be5436b9-396e-6576-ac1d-f3fac73e6ffa be5436b9-396e-6576-ac1d-f3fac73e6ffa}" error="failed to connect to peer: be5436b9-396e-6576-ac1d-f3fac73e6ffa" 2026-04-22T10:11:01.997Z [WARN] server-be5436b9-396e-6576-ac1d-f3fac73e6ffa: heartbeat timeout reached, starting election: last-leader-addr=c2714c75-176b-29b5-f842-4eb2faa7e453 last-leader-id=server-c2714c75-176b-29b5-f842-4eb2faa7e453 2026-04-22T10:11:01.997Z [INFO] server-be5436b9-396e-6576-ac1d-f3fac73e6ffa: entering candidate state: node="Node at be5436b9-396e-6576-ac1d-f3fac73e6ffa [Candidate]" term=3 2026-04-22T10:11:01.997Z [ERROR] server-be5436b9-396e-6576-ac1d-f3fac73e6ffa: failed to make requestVote RPC: target="{Voter server-69bde12a-d80b-e84c-c422-5a8d3f0063eb 69bde12a-d80b-e84c-c422-5a8d3f0063eb}" error="failed to connect to peer: 69bde12a-d80b-e84c-c422-5a8d3f0063eb" term=3 2026-04-22T10:11:01.997Z [ERROR] server-be5436b9-396e-6576-ac1d-f3fac73e6ffa: failed to make requestVote RPC: target="{Voter server-c2714c75-176b-29b5-f842-4eb2faa7e453 c2714c75-176b-29b5-f842-4eb2faa7e453}" error="failed to connect to peer: c2714c75-176b-29b5-f842-4eb2faa7e453" term=3 2026-04-22T10:11:01.999Z [ERROR] server-c2714c75-176b-29b5-f842-4eb2faa7e453: failed to heartbeat to: peer=be5436b9-396e-6576-ac1d-f3fac73e6ffa backoff time=25ms error="failed to connect to peer: be5436b9-396e-6576-ac1d-f3fac73e6ffa" 2026-04-22T10:11:02.012Z [WARN] server-c2714c75-176b-29b5-f842-4eb2faa7e453: failed to contact: server-id=server-be5436b9-396e-6576-ac1d-f3fac73e6ffa time=98.904262ms 2026-04-22T10:11:02.022Z [INFO] server-c2714c75-176b-29b5-f842-4eb2faa7e453: updating configuration: command=RemoveServer server-id=server-be5436b9-396e-6576-ac1d-f3fac73e6ffa server-addr= servers="[{Suffrage:Voter ID:server-c2714c75-176b-29b5-f842-4eb2faa7e453 Address:c2714c75-176b-29b5-f842-4eb2faa7e453} {Suffrage:Voter ID:server-69bde12a-d80b-e84c-c422-5a8d3f0063eb Address:69bde12a-d80b-e84c-c422-5a8d3f0063eb}]" 2026-04-22T10:11:02.022Z [INFO] server-c2714c75-176b-29b5-f842-4eb2faa7e453: removed peer, stopping replication: peer=server-be5436b9-396e-6576-ac1d-f3fac73e6ffa last-index=3 2026-04-22T10:11:02.022Z [INFO] server-c2714c75-176b-29b5-f842-4eb2faa7e453: updating configuration: command=AddNonvoter server-id=server-be5436b9-396e-6576-ac1d-f3fac73e6ffa server-addr=be5436b9-396e-6576-ac1d-f3fac73e6ffa servers="[{Suffrage:Voter ID:server-c2714c75-176b-29b5-f842-4eb2faa7e453 Address:c2714c75-176b-29b5-f842-4eb2faa7e453} {Suffrage:Voter ID:server-69bde12a-d80b-e84c-c422-5a8d3f0063eb Address:69bde12a-d80b-e84c-c422-5a8d3f0063eb} {Suffrage:Nonvoter ID:server-be5436b9-396e-6576-ac1d-f3fac73e6ffa Address:be5436b9-396e-6576-ac1d-f3fac73e6ffa}]" 2026-04-22T10:11:02.022Z [INFO] server-c2714c75-176b-29b5-f842-4eb2faa7e453: added peer, starting replication: peer=server-be5436b9-396e-6576-ac1d-f3fac73e6ffa 2026-04-22T10:11:02.022Z [ERROR] server-c2714c75-176b-29b5-f842-4eb2faa7e453: failed to appendEntries to: peer="{Nonvoter server-be5436b9-396e-6576-ac1d-f3fac73e6ffa be5436b9-396e-6576-ac1d-f3fac73e6ffa}" error="failed to connect to peer: be5436b9-396e-6576-ac1d-f3fac73e6ffa" 2026-04-22T10:11:02.035Z [ERROR] server-c2714c75-176b-29b5-f842-4eb2faa7e453: peer has newer term, stopping replication: peer="{Voter server-be5436b9-396e-6576-ac1d-f3fac73e6ffa be5436b9-396e-6576-ac1d-f3fac73e6ffa}" 2026-04-22T10:11:02.035Z [INFO] server-c2714c75-176b-29b5-f842-4eb2faa7e453: entering follower state: follower="Node at c2714c75-176b-29b5-f842-4eb2faa7e453 [Follower]" leader-address= leader-id= 2026-04-22T10:11:02.035Z [INFO] server-c2714c75-176b-29b5-f842-4eb2faa7e453: aborting pipeline replication: peer="{Voter server-69bde12a-d80b-e84c-c422-5a8d3f0063eb 69bde12a-d80b-e84c-c422-5a8d3f0063eb}" 2026-04-22T10:11:02.040Z [ERROR] server-c2714c75-176b-29b5-f842-4eb2faa7e453: peer has newer term, stopping replication: peer="{Nonvoter server-be5436b9-396e-6576-ac1d-f3fac73e6ffa be5436b9-396e-6576-ac1d-f3fac73e6ffa}" 2026-04-22T10:11:02.086Z [WARN] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: heartbeat timeout reached, starting election: last-leader-addr=c2714c75-176b-29b5-f842-4eb2faa7e453 last-leader-id=server-c2714c75-176b-29b5-f842-4eb2faa7e453 2026-04-22T10:11:02.086Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: entering candidate state: node="Node at 69bde12a-d80b-e84c-c422-5a8d3f0063eb [Candidate]" term=3 2026-04-22T10:11:02.086Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: election won: term=3 tally=2 2026-04-22T10:11:02.086Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: entering leader state: leader="Node at 69bde12a-d80b-e84c-c422-5a8d3f0063eb [Leader]" 2026-04-22T10:11:02.086Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: added peer, starting replication: peer=server-c2714c75-176b-29b5-f842-4eb2faa7e453 2026-04-22T10:11:02.086Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: added peer, starting replication: peer=server-be5436b9-396e-6576-ac1d-f3fac73e6ffa 2026-04-22T10:11:02.086Z [ERROR] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: peer has newer term, stopping replication: peer="{Nonvoter server-be5436b9-396e-6576-ac1d-f3fac73e6ffa be5436b9-396e-6576-ac1d-f3fac73e6ffa}" 2026-04-22T10:11:02.086Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: entering follower state: follower="Node at 69bde12a-d80b-e84c-c422-5a8d3f0063eb [Follower]" leader-address= leader-id= 2026-04-22T10:11:02.091Z [WARN] server-be5436b9-396e-6576-ac1d-f3fac73e6ffa: Election timeout reached, restarting election 2026-04-22T10:11:02.091Z [INFO] server-be5436b9-396e-6576-ac1d-f3fac73e6ffa: entering candidate state: node="Node at be5436b9-396e-6576-ac1d-f3fac73e6ffa [Candidate]" term=23 2026-04-22T10:11:02.091Z [WARN] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: rejecting vote request since node is not a voter: from=be5436b9-396e-6576-ac1d-f3fac73e6ffa 2026-04-22T10:11:02.092Z [WARN] server-c2714c75-176b-29b5-f842-4eb2faa7e453: rejecting vote request since we have a leader: from=be5436b9-396e-6576-ac1d-f3fac73e6ffa leader=69bde12a-d80b-e84c-c422-5a8d3f0063eb leader-id=server-69bde12a-d80b-e84c-c422-5a8d3f0063eb 2026-04-22T10:11:02.128Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:11:02.176Z [WARN] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:11:02.176Z [WARN] server-be5436b9-396e-6576-ac1d-f3fac73e6ffa: Election timeout reached, restarting election 2026-04-22T10:11:02.176Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: entering candidate state: node="Node at 69bde12a-d80b-e84c-c422-5a8d3f0063eb [Candidate]" term=24 2026-04-22T10:11:02.176Z [INFO] server-be5436b9-396e-6576-ac1d-f3fac73e6ffa: entering candidate state: node="Node at be5436b9-396e-6576-ac1d-f3fac73e6ffa [Candidate]" term=24 2026-04-22T10:11:02.176Z [WARN] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: rejecting vote request since node is not a voter: from=be5436b9-396e-6576-ac1d-f3fac73e6ffa 2026-04-22T10:11:02.176Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: election won: term=24 tally=2 2026-04-22T10:11:02.176Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: entering leader state: leader="Node at 69bde12a-d80b-e84c-c422-5a8d3f0063eb [Leader]" 2026-04-22T10:11:02.176Z [WARN] server-c2714c75-176b-29b5-f842-4eb2faa7e453: rejecting vote request since node is not a voter: from=be5436b9-396e-6576-ac1d-f3fac73e6ffa 2026-04-22T10:11:02.176Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: added peer, starting replication: peer=server-c2714c75-176b-29b5-f842-4eb2faa7e453 2026-04-22T10:11:02.176Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: added peer, starting replication: peer=server-be5436b9-396e-6576-ac1d-f3fac73e6ffa 2026-04-22T10:11:02.176Z [WARN] server-be5436b9-396e-6576-ac1d-f3fac73e6ffa: failed to get previous log: previous-index=5 last-index=2 error="log not found" 2026-04-22T10:11:02.176Z [INFO] server-be5436b9-396e-6576-ac1d-f3fac73e6ffa: entering follower state: follower="Node at be5436b9-396e-6576-ac1d-f3fac73e6ffa [Follower]" leader-address=69bde12a-d80b-e84c-c422-5a8d3f0063eb leader-id=server-69bde12a-d80b-e84c-c422-5a8d3f0063eb 2026-04-22T10:11:02.176Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: pipelining replication: peer="{Voter server-c2714c75-176b-29b5-f842-4eb2faa7e453 c2714c75-176b-29b5-f842-4eb2faa7e453}" 2026-04-22T10:11:02.176Z [WARN] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: appendEntries rejected, sending older logs: peer="{Nonvoter server-be5436b9-396e-6576-ac1d-f3fac73e6ffa be5436b9-396e-6576-ac1d-f3fac73e6ffa}" next=3 2026-04-22T10:11:02.176Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: pipelining replication: peer="{Nonvoter server-be5436b9-396e-6576-ac1d-f3fac73e6ffa be5436b9-396e-6576-ac1d-f3fac73e6ffa}" 2026-04-22T10:11:02.282Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:11:02.176492275 +0000 UTC m=+108.200288364 (1 nodes), 48.264256ms from start of poll, 550.659077ms from cluster start. Timeout at 2026-04-22 10:11:02.281493493 +0000 UTC m=+108.305289576, 105.001212ms after stability 2026-04-22T10:11:02.309Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: updating configuration: command=RemoveServer server-id=server-be5436b9-396e-6576-ac1d-f3fac73e6ffa server-addr= servers="[{Suffrage:Voter ID:server-c2714c75-176b-29b5-f842-4eb2faa7e453 Address:c2714c75-176b-29b5-f842-4eb2faa7e453} {Suffrage:Voter ID:server-69bde12a-d80b-e84c-c422-5a8d3f0063eb Address:69bde12a-d80b-e84c-c422-5a8d3f0063eb}]" 2026-04-22T10:11:02.310Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: removed peer, stopping replication: peer=server-be5436b9-396e-6576-ac1d-f3fac73e6ffa last-index=107 2026-04-22T10:11:02.310Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: aborting pipeline replication: peer="{Nonvoter server-be5436b9-396e-6576-ac1d-f3fac73e6ffa be5436b9-396e-6576-ac1d-f3fac73e6ffa}" 2026-04-22T10:11:02.310Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: updating configuration: command=AddVoter server-id=server-be5436b9-396e-6576-ac1d-f3fac73e6ffa server-addr=be5436b9-396e-6576-ac1d-f3fac73e6ffa servers="[{Suffrage:Voter ID:server-c2714c75-176b-29b5-f842-4eb2faa7e453 Address:c2714c75-176b-29b5-f842-4eb2faa7e453} {Suffrage:Voter ID:server-69bde12a-d80b-e84c-c422-5a8d3f0063eb Address:69bde12a-d80b-e84c-c422-5a8d3f0063eb} {Suffrage:Voter ID:server-be5436b9-396e-6576-ac1d-f3fac73e6ffa Address:be5436b9-396e-6576-ac1d-f3fac73e6ffa}]" 2026-04-22T10:11:02.310Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: added peer, starting replication: peer=server-be5436b9-396e-6576-ac1d-f3fac73e6ffa 2026-04-22T10:11:02.310Z [WARN] server-be5436b9-396e-6576-ac1d-f3fac73e6ffa: failed to get previous log: previous-index=108 last-index=107 error="log not found" 2026-04-22T10:11:02.310Z [WARN] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: appendEntries rejected, sending older logs: peer="{Voter server-be5436b9-396e-6576-ac1d-f3fac73e6ffa be5436b9-396e-6576-ac1d-f3fac73e6ffa}" next=108 2026-04-22T10:11:02.310Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: pipelining replication: peer="{Voter server-be5436b9-396e-6576-ac1d-f3fac73e6ffa be5436b9-396e-6576-ac1d-f3fac73e6ffa}" 2026-04-22T10:11:03.381Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:11:03.486Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:11:03.381400079 +0000 UTC m=+109.405196168 (1 nodes), 497ns from start of poll, 1.755566881s from cluster start. Timeout at 2026-04-22 10:11:03.486416745 +0000 UTC m=+109.510212965, 105.016797ms after stability 2026-04-22T10:11:03.486Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: aborting pipeline replication: peer="{Voter server-c2714c75-176b-29b5-f842-4eb2faa7e453 c2714c75-176b-29b5-f842-4eb2faa7e453}" 2026-04-22T10:11:03.486Z [INFO] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: aborting pipeline replication: peer="{Voter server-be5436b9-396e-6576-ac1d-f3fac73e6ffa be5436b9-396e-6576-ac1d-f3fac73e6ffa}" 2026-04-22T10:11:03.986Z [ERROR] server-69bde12a-d80b-e84c-c422-5a8d3f0063eb: failed to heartbeat to: peer=c2714c75-176b-29b5-f842-4eb2faa7e453 backoff time=10ms error="command timed out" --- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (2.36s) === RUN TestRaft_FollowerRemovalNoElection 2026-04-22T10:11:03.987Z [INFO] server-6b068f9e-aa34-7ff0-eb9b-069a24427909: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b068f9e-aa34-7ff0-eb9b-069a24427909 Address:6b068f9e-aa34-7ff0-eb9b-069a24427909} {Suffrage:Voter ID:server-f6470077-8ef4-28b5-feda-91d5c5ba115a Address:f6470077-8ef4-28b5-feda-91d5c5ba115a} {Suffrage:Voter ID:server-73d75d2c-f4ba-de09-f33b-c87003cf7fa2 Address:73d75d2c-f4ba-de09-f33b-c87003cf7fa2}]" 2026-04-22T10:11:03.987Z [INFO] server-6b068f9e-aa34-7ff0-eb9b-069a24427909: entering follower state: follower="Node at 6b068f9e-aa34-7ff0-eb9b-069a24427909 [Follower]" leader-address= leader-id= 2026-04-22T10:11:03.987Z [INFO] server-f6470077-8ef4-28b5-feda-91d5c5ba115a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b068f9e-aa34-7ff0-eb9b-069a24427909 Address:6b068f9e-aa34-7ff0-eb9b-069a24427909} {Suffrage:Voter ID:server-f6470077-8ef4-28b5-feda-91d5c5ba115a Address:f6470077-8ef4-28b5-feda-91d5c5ba115a} {Suffrage:Voter ID:server-73d75d2c-f4ba-de09-f33b-c87003cf7fa2 Address:73d75d2c-f4ba-de09-f33b-c87003cf7fa2}]" 2026-04-22T10:11:03.987Z [INFO] server-f6470077-8ef4-28b5-feda-91d5c5ba115a: entering follower state: follower="Node at f6470077-8ef4-28b5-feda-91d5c5ba115a [Follower]" leader-address= leader-id= 2026-04-22T10:11:03.987Z [INFO] server-73d75d2c-f4ba-de09-f33b-c87003cf7fa2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b068f9e-aa34-7ff0-eb9b-069a24427909 Address:6b068f9e-aa34-7ff0-eb9b-069a24427909} {Suffrage:Voter ID:server-f6470077-8ef4-28b5-feda-91d5c5ba115a Address:f6470077-8ef4-28b5-feda-91d5c5ba115a} {Suffrage:Voter ID:server-73d75d2c-f4ba-de09-f33b-c87003cf7fa2 Address:73d75d2c-f4ba-de09-f33b-c87003cf7fa2}]" 2026-04-22T10:11:03.987Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:11:03.987Z [INFO] server-73d75d2c-f4ba-de09-f33b-c87003cf7fa2: entering follower state: follower="Node at 73d75d2c-f4ba-de09-f33b-c87003cf7fa2 [Follower]" leader-address= leader-id= 2026-04-22T10:11:04.097Z [WARN] server-6b068f9e-aa34-7ff0-eb9b-069a24427909: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:11:04.097Z [INFO] server-6b068f9e-aa34-7ff0-eb9b-069a24427909: entering candidate state: node="Node at 6b068f9e-aa34-7ff0-eb9b-069a24427909 [Candidate]" term=2 2026-04-22T10:11:04.097Z [INFO] server-6b068f9e-aa34-7ff0-eb9b-069a24427909: election won: term=2 tally=2 2026-04-22T10:11:04.097Z [INFO] server-6b068f9e-aa34-7ff0-eb9b-069a24427909: entering leader state: leader="Node at 6b068f9e-aa34-7ff0-eb9b-069a24427909 [Leader]" 2026-04-22T10:11:04.097Z [INFO] server-6b068f9e-aa34-7ff0-eb9b-069a24427909: added peer, starting replication: peer=server-f6470077-8ef4-28b5-feda-91d5c5ba115a 2026-04-22T10:11:04.097Z [INFO] server-6b068f9e-aa34-7ff0-eb9b-069a24427909: added peer, starting replication: peer=server-73d75d2c-f4ba-de09-f33b-c87003cf7fa2 2026-04-22T10:11:04.097Z [INFO] server-6b068f9e-aa34-7ff0-eb9b-069a24427909: pipelining replication: peer="{Voter server-f6470077-8ef4-28b5-feda-91d5c5ba115a f6470077-8ef4-28b5-feda-91d5c5ba115a}" 2026-04-22T10:11:04.097Z [INFO] server-6b068f9e-aa34-7ff0-eb9b-069a24427909: pipelining replication: peer="{Voter server-73d75d2c-f4ba-de09-f33b-c87003cf7fa2 73d75d2c-f4ba-de09-f33b-c87003cf7fa2}" 2026-04-22T10:11:04.302Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:11:04.097218122 +0000 UTC m=+110.121014216 (1 nodes), 109.290462ms from start of poll, 109.602501ms from cluster start. Timeout at 2026-04-22 10:11:04.302227979 +0000 UTC m=+110.326024060, 205.009844ms after stability 2026-04-22T10:11:04.302Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:11:04.508Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:11:04.302320274 +0000 UTC m=+110.326116378 (1 nodes), 733ns from start of poll, 314.704663ms from cluster start. Timeout at 2026-04-22 10:11:04.507345697 +0000 UTC m=+110.531141957, 205.025579ms after stability 2026-04-22T10:11:04.513Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:11:04.719Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:11:04.513687831 +0000 UTC m=+110.537483931 (2 nodes), 1.065µs from start of poll, 526.072216ms from cluster start. Timeout at 2026-04-22 10:11:04.718717808 +0000 UTC m=+110.742514005, 205.030074ms after stability 2026-04-22T10:11:04.719Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:11:04.925Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:11:04.719466682 +0000 UTC m=+110.743262859 (1 nodes), 1.507µs from start of poll, 731.851144ms from cluster start. Timeout at 2026-04-22 10:11:04.924487493 +0000 UTC m=+110.948283611, 205.020752ms after stability raft_test.go:2961: [INFO] restarting Node at f6470077-8ef4-28b5-feda-91d5c5ba115a [Follower] 2026-04-22T10:11:04.925Z [INFO] server-f6470077-8ef4-28b5-feda-91d5c5ba115a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b068f9e-aa34-7ff0-eb9b-069a24427909 Address:6b068f9e-aa34-7ff0-eb9b-069a24427909} {Suffrage:Voter ID:server-f6470077-8ef4-28b5-feda-91d5c5ba115a Address:f6470077-8ef4-28b5-feda-91d5c5ba115a} {Suffrage:Voter ID:server-73d75d2c-f4ba-de09-f33b-c87003cf7fa2 Address:73d75d2c-f4ba-de09-f33b-c87003cf7fa2}]" 2026-04-22T10:11:04.925Z [INFO] server-f6470077-8ef4-28b5-feda-91d5c5ba115a: entering follower state: follower="Node at f6470077-8ef4-28b5-feda-91d5c5ba115a [Follower]" leader-address= leader-id= 2026-04-22T10:11:05.175Z [INFO] server-6b068f9e-aa34-7ff0-eb9b-069a24427909: aborting pipeline replication: peer="{Voter server-73d75d2c-f4ba-de09-f33b-c87003cf7fa2 73d75d2c-f4ba-de09-f33b-c87003cf7fa2}" 2026-04-22T10:11:05.558Z [ERROR] server-6b068f9e-aa34-7ff0-eb9b-069a24427909: failed to pipeline appendEntries: peer="{Voter server-f6470077-8ef4-28b5-feda-91d5c5ba115a f6470077-8ef4-28b5-feda-91d5c5ba115a}" error="command enqueue timeout" 2026-04-22T10:11:05.558Z [INFO] server-6b068f9e-aa34-7ff0-eb9b-069a24427909: aborting pipeline replication: peer="{Voter server-f6470077-8ef4-28b5-feda-91d5c5ba115a f6470077-8ef4-28b5-feda-91d5c5ba115a}" 2026-04-22T10:11:05.683Z [ERROR] server-6b068f9e-aa34-7ff0-eb9b-069a24427909: failed to heartbeat to: peer=f6470077-8ef4-28b5-feda-91d5c5ba115a backoff time=10ms error="command timed out" --- PASS: TestRaft_FollowerRemovalNoElection (1.70s) === RUN TestRaft_VoteWithNoIDNoAddr 2026-04-22T10:11:05.684Z [INFO] server-25c3a33e-ad03-e556-5dae-b650ee62bac2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c3a33e-ad03-e556-5dae-b650ee62bac2 Address:25c3a33e-ad03-e556-5dae-b650ee62bac2} {Suffrage:Voter ID:server-39658725-95ed-aef1-aca9-1f05548374f2 Address:39658725-95ed-aef1-aca9-1f05548374f2} {Suffrage:Voter ID:server-92afba55-9920-6e14-3906-e747a5f2105c Address:92afba55-9920-6e14-3906-e747a5f2105c}]" 2026-04-22T10:11:05.684Z [INFO] server-25c3a33e-ad03-e556-5dae-b650ee62bac2: entering follower state: follower="Node at 25c3a33e-ad03-e556-5dae-b650ee62bac2 [Follower]" leader-address= leader-id= 2026-04-22T10:11:05.684Z [INFO] server-39658725-95ed-aef1-aca9-1f05548374f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c3a33e-ad03-e556-5dae-b650ee62bac2 Address:25c3a33e-ad03-e556-5dae-b650ee62bac2} {Suffrage:Voter ID:server-39658725-95ed-aef1-aca9-1f05548374f2 Address:39658725-95ed-aef1-aca9-1f05548374f2} {Suffrage:Voter ID:server-92afba55-9920-6e14-3906-e747a5f2105c Address:92afba55-9920-6e14-3906-e747a5f2105c}]" 2026-04-22T10:11:05.684Z [INFO] server-39658725-95ed-aef1-aca9-1f05548374f2: entering follower state: follower="Node at 39658725-95ed-aef1-aca9-1f05548374f2 [Follower]" leader-address= leader-id= 2026-04-22T10:11:05.684Z [INFO] server-92afba55-9920-6e14-3906-e747a5f2105c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c3a33e-ad03-e556-5dae-b650ee62bac2 Address:25c3a33e-ad03-e556-5dae-b650ee62bac2} {Suffrage:Voter ID:server-39658725-95ed-aef1-aca9-1f05548374f2 Address:39658725-95ed-aef1-aca9-1f05548374f2} {Suffrage:Voter ID:server-92afba55-9920-6e14-3906-e747a5f2105c Address:92afba55-9920-6e14-3906-e747a5f2105c}]" 2026-04-22T10:11:05.684Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:11:05.684Z [INFO] server-92afba55-9920-6e14-3906-e747a5f2105c: entering follower state: follower="Node at 92afba55-9920-6e14-3906-e747a5f2105c [Follower]" leader-address= leader-id= 2026-04-22T10:11:05.742Z [WARN] server-39658725-95ed-aef1-aca9-1f05548374f2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-04-22T10:11:05.742Z [INFO] server-39658725-95ed-aef1-aca9-1f05548374f2: entering candidate state: node="Node at 39658725-95ed-aef1-aca9-1f05548374f2 [Candidate]" term=2 2026-04-22T10:11:05.742Z [INFO] server-39658725-95ed-aef1-aca9-1f05548374f2: election won: term=2 tally=2 2026-04-22T10:11:05.742Z [INFO] server-39658725-95ed-aef1-aca9-1f05548374f2: entering leader state: leader="Node at 39658725-95ed-aef1-aca9-1f05548374f2 [Leader]" 2026-04-22T10:11:05.742Z [INFO] server-39658725-95ed-aef1-aca9-1f05548374f2: added peer, starting replication: peer=server-25c3a33e-ad03-e556-5dae-b650ee62bac2 2026-04-22T10:11:05.742Z [INFO] server-39658725-95ed-aef1-aca9-1f05548374f2: added peer, starting replication: peer=server-92afba55-9920-6e14-3906-e747a5f2105c 2026-04-22T10:11:05.742Z [INFO] server-39658725-95ed-aef1-aca9-1f05548374f2: pipelining replication: peer="{Voter server-25c3a33e-ad03-e556-5dae-b650ee62bac2 25c3a33e-ad03-e556-5dae-b650ee62bac2}" 2026-04-22T10:11:05.742Z [INFO] server-39658725-95ed-aef1-aca9-1f05548374f2: pipelining replication: peer="{Voter server-92afba55-9920-6e14-3906-e747a5f2105c 92afba55-9920-6e14-3906-e747a5f2105c}" 2026-04-22T10:11:05.847Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:11:05.742251262 +0000 UTC m=+111.766047355 (1 nodes), 57.419163ms from start of poll, 57.724377ms from cluster start. Timeout at 2026-04-22 10:11:05.8472607 +0000 UTC m=+111.871056837, 105.009482ms after stability 2026-04-22T10:11:05.847Z [INFO] cluster: starting stability test: raft-state=Leader 2026-04-22T10:11:05.953Z [INFO] cluster: stable state for Leader reached at 2026-04-22 10:11:05.847889806 +0000 UTC m=+111.871686003 (1 nodes), 1.704µs from start of poll, 163.363025ms from cluster start. Timeout at 2026-04-22 10:11:05.952917761 +0000 UTC m=+111.976713971, 105.027968ms after stability 2026-04-22T10:11:05.958Z [INFO] cluster: starting stability test: raft-state=Follower 2026-04-22T10:11:06.064Z [INFO] cluster: stable state for Follower reached at 2026-04-22 10:11:05.958857117 +0000 UTC m=+111.982653337 (2 nodes), 1.991µs from start of poll, 274.330359ms from cluster start. Timeout at 2026-04-22 10:11:06.063888056 +0000 UTC m=+112.087684236, 105.030899ms after stability 2026-04-22T10:11:06.067Z [ERROR] server-39658725-95ed-aef1-aca9-1f05548374f2: failed to heartbeat to: peer=92afba55-9920-6e14-3906-e747a5f2105c backoff time=10ms error="failed to connect to peer: 92afba55-9920-6e14-3906-e747a5f2105c" 2026-04-22T10:11:06.068Z [ERROR] server-39658725-95ed-aef1-aca9-1f05548374f2: failed to heartbeat to: peer=25c3a33e-ad03-e556-5dae-b650ee62bac2 backoff time=10ms error="failed to connect to peer: 25c3a33e-ad03-e556-5dae-b650ee62bac2" 2026-04-22T10:11:06.068Z [ERROR] server-39658725-95ed-aef1-aca9-1f05548374f2: failed to pipeline appendEntries: peer="{Voter server-92afba55-9920-6e14-3906-e747a5f2105c 92afba55-9920-6e14-3906-e747a5f2105c}" error="append pipeline closed" 2026-04-22T10:11:06.069Z [INFO] server-39658725-95ed-aef1-aca9-1f05548374f2: aborting pipeline replication: peer="{Voter server-92afba55-9920-6e14-3906-e747a5f2105c 92afba55-9920-6e14-3906-e747a5f2105c}" 2026-04-22T10:11:06.069Z [ERROR] server-39658725-95ed-aef1-aca9-1f05548374f2: failed to pipeline appendEntries: peer="{Voter server-25c3a33e-ad03-e556-5dae-b650ee62bac2 25c3a33e-ad03-e556-5dae-b650ee62bac2}" error="append pipeline closed" 2026-04-22T10:11:06.069Z [INFO] server-39658725-95ed-aef1-aca9-1f05548374f2: aborting pipeline replication: peer="{Voter server-25c3a33e-ad03-e556-5dae-b650ee62bac2 25c3a33e-ad03-e556-5dae-b650ee62bac2}" 2026-04-22T10:11:06.075Z [ERROR] server-39658725-95ed-aef1-aca9-1f05548374f2: failed to appendEntries to: peer="{Voter server-25c3a33e-ad03-e556-5dae-b650ee62bac2 25c3a33e-ad03-e556-5dae-b650ee62bac2}" error="failed to connect to peer: 25c3a33e-ad03-e556-5dae-b650ee62bac2" 2026-04-22T10:11:06.077Z [ERROR] server-39658725-95ed-aef1-aca9-1f05548374f2: failed to appendEntries to: peer="{Voter server-92afba55-9920-6e14-3906-e747a5f2105c 92afba55-9920-6e14-3906-e747a5f2105c}" error="failed to connect to peer: 92afba55-9920-6e14-3906-e747a5f2105c" 2026-04-22T10:11:06.085Z [ERROR] server-39658725-95ed-aef1-aca9-1f05548374f2: failed to heartbeat to: peer=25c3a33e-ad03-e556-5dae-b650ee62bac2 backoff time=10ms error="failed to connect to peer: 25c3a33e-ad03-e556-5dae-b650ee62bac2" 2026-04-22T10:11:06.086Z [ERROR] server-39658725-95ed-aef1-aca9-1f05548374f2: failed to heartbeat to: peer=92afba55-9920-6e14-3906-e747a5f2105c backoff time=10ms error="failed to connect to peer: 92afba55-9920-6e14-3906-e747a5f2105c" 2026-04-22T10:11:06.091Z [ERROR] server-39658725-95ed-aef1-aca9-1f05548374f2: failed to appendEntries to: peer="{Voter server-25c3a33e-ad03-e556-5dae-b650ee62bac2 25c3a33e-ad03-e556-5dae-b650ee62bac2}" error="failed to connect to peer: 25c3a33e-ad03-e556-5dae-b650ee62bac2" 2026-04-22T10:11:06.094Z [ERROR] server-39658725-95ed-aef1-aca9-1f05548374f2: failed to appendEntries to: peer="{Voter server-92afba55-9920-6e14-3906-e747a5f2105c 92afba55-9920-6e14-3906-e747a5f2105c}" error="failed to connect to peer: 92afba55-9920-6e14-3906-e747a5f2105c" 2026-04-22T10:11:06.101Z [ERROR] server-39658725-95ed-aef1-aca9-1f05548374f2: failed to heartbeat to: peer=25c3a33e-ad03-e556-5dae-b650ee62bac2 backoff time=10ms error="failed to connect to peer: 25c3a33e-ad03-e556-5dae-b650ee62bac2" 2026-04-22T10:11:06.104Z [ERROR] server-39658725-95ed-aef1-aca9-1f05548374f2: failed to heartbeat to: peer=92afba55-9920-6e14-3906-e747a5f2105c backoff time=10ms error="failed to connect to peer: 92afba55-9920-6e14-3906-e747a5f2105c" 2026-04-22T10:11:06.111Z [ERROR] server-39658725-95ed-aef1-aca9-1f05548374f2: failed to appendEntries to: peer="{Voter server-92afba55-9920-6e14-3906-e747a5f2105c 92afba55-9920-6e14-3906-e747a5f2105c}" error="failed to connect to peer: 92afba55-9920-6e14-3906-e747a5f2105c" 2026-04-22T10:11:06.112Z [ERROR] server-39658725-95ed-aef1-aca9-1f05548374f2: failed to appendEntries to: peer="{Voter server-25c3a33e-ad03-e556-5dae-b650ee62bac2 25c3a33e-ad03-e556-5dae-b650ee62bac2}" error="failed to connect to peer: 25c3a33e-ad03-e556-5dae-b650ee62bac2" 2026-04-22T10:11:06.112Z [WARN] server-39658725-95ed-aef1-aca9-1f05548374f2: failed to contact: server-id=server-25c3a33e-ad03-e556-5dae-b650ee62bac2 time=50.274672ms 2026-04-22T10:11:06.112Z [WARN] server-39658725-95ed-aef1-aca9-1f05548374f2: failed to contact: server-id=server-92afba55-9920-6e14-3906-e747a5f2105c time=50.137334ms 2026-04-22T10:11:06.112Z [WARN] server-39658725-95ed-aef1-aca9-1f05548374f2: failed to contact quorum of nodes, stepping down 2026-04-22T10:11:06.112Z [INFO] server-39658725-95ed-aef1-aca9-1f05548374f2: entering follower state: follower="Node at 39658725-95ed-aef1-aca9-1f05548374f2 [Follower]" leader-address= leader-id= 2026-04-22T10:11:06.166Z [WARN] server-92afba55-9920-6e14-3906-e747a5f2105c: heartbeat timeout reached, starting election: last-leader-addr=39658725-95ed-aef1-aca9-1f05548374f2 last-leader-id=server-39658725-95ed-aef1-aca9-1f05548374f2 2026-04-22T10:11:06.166Z [INFO] server-92afba55-9920-6e14-3906-e747a5f2105c: entering candidate state: node="Node at 92afba55-9920-6e14-3906-e747a5f2105c [Candidate]" term=3 2026-04-22T10:11:06.166Z [ERROR] server-92afba55-9920-6e14-3906-e747a5f2105c: failed to make requestVote RPC: target="{Voter server-39658725-95ed-aef1-aca9-1f05548374f2 39658725-95ed-aef1-aca9-1f05548374f2}" error="failed to connect to peer: 39658725-95ed-aef1-aca9-1f05548374f2" term=3 2026-04-22T10:11:06.166Z [WARN] server-25c3a33e-ad03-e556-5dae-b650ee62bac2: rejecting vote request since we have a leader: from=92afba55-9920-6e14-3906-e747a5f2105c leader=39658725-95ed-aef1-aca9-1f05548374f2 leader-id=server-39658725-95ed-aef1-aca9-1f05548374f2 2026-04-22T10:11:06.175Z [WARN] server-25c3a33e-ad03-e556-5dae-b650ee62bac2: heartbeat timeout reached, starting election: last-leader-addr=39658725-95ed-aef1-aca9-1f05548374f2 last-leader-id=server-39658725-95ed-aef1-aca9-1f05548374f2 2026-04-22T10:11:06.175Z [INFO] server-25c3a33e-ad03-e556-5dae-b650ee62bac2: entering candidate state: node="Node at 25c3a33e-ad03-e556-5dae-b650ee62bac2 [Candidate]" term=3 2026-04-22T10:11:06.175Z [INFO] server-92afba55-9920-6e14-3906-e747a5f2105c: duplicate requestVote for same term: term=3 2026-04-22T10:11:06.175Z [ERROR] server-25c3a33e-ad03-e556-5dae-b650ee62bac2: failed to make requestVote RPC: target="{Voter server-39658725-95ed-aef1-aca9-1f05548374f2 39658725-95ed-aef1-aca9-1f05548374f2}" error="failed to connect to peer: 39658725-95ed-aef1-aca9-1f05548374f2" term=3 2026-04-22T10:11:06.176Z [INFO] server-25c3a33e-ad03-e556-5dae-b650ee62bac2: entering follower state: follower="Node at 25c3a33e-ad03-e556-5dae-b650ee62bac2 [Follower]" leader-address= leader-id= --- PASS: TestRaft_VoteWithNoIDNoAddr (0.49s) === RUN TestRaft_runFollower_State_Transition === RUN TestRaft_runFollower_State_Transition/NonVoter 2026-04-22T10:11:06.177Z [INFO] 127.0.0.1:42657 :: starting node: addr=127.0.0.1:42657 2026-04-22T10:11:06.177Z [INFO] 127.0.0.1:42657 :: initial configuration: index=0 servers=[] 2026-04-22T10:11:06.177Z [INFO] 127.0.0.1:42657 :: entering follower state: follower="Node at 127.0.0.1:42657 [Follower]" leader-address= leader-id= 2026-04-22T10:11:06.238Z [WARN] 127.0.0.1:42657 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election === RUN TestRaft_runFollower_State_Transition/Voter 2026-04-22T10:11:06.329Z [INFO] 127.0.0.1:34381 :: starting node: addr=127.0.0.1:34381 2026-04-22T10:11:06.329Z [INFO] 127.0.0.1:34381 :: initial configuration: index=0 servers=[] 2026-04-22T10:11:06.329Z [INFO] 127.0.0.1:34381 :: entering follower state: follower="Node at 127.0.0.1:34381 [Follower]" leader-address= leader-id= 2026-04-22T10:11:06.400Z [WARN] 127.0.0.1:34381 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= === RUN TestRaft_runFollower_State_Transition/Not_in_Config 2026-04-22T10:11:06.480Z [INFO] 127.0.0.1:38633 :: starting node: addr=127.0.0.1:38633 2026-04-22T10:11:06.480Z [INFO] 127.0.0.1:38633 :: initial configuration: index=0 servers=[] 2026-04-22T10:11:06.480Z [INFO] 127.0.0.1:38633 :: entering follower state: follower="Node at 127.0.0.1:38633 [Follower]" leader-address= leader-id= 2026-04-22T10:11:06.565Z [WARN] 127.0.0.1:38633 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election --- PASS: TestRaft_runFollower_State_Transition (0.45s) --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s) === RUN TestRaft_runFollower_ReloadTimeoutConfigs 2026-04-22T10:11:06.633Z [INFO] 127.0.0.1:35061 :: starting node: addr=127.0.0.1:35061 2026-04-22T10:11:06.634Z [INFO] 127.0.0.1:35061 :: initial configuration: index=0 servers=[] 2026-04-22T10:11:06.634Z [INFO] 127.0.0.1:35061 :: entering follower state: follower="Node at 127.0.0.1:35061 [Follower]" leader-address= leader-id= 2026-04-22T10:11:06.634Z [WARN] 127.0.0.1:35061 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= --- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s) === RUN TestSaturationMetric === RUN TestSaturationMetric/without_smoothing --- PASS: TestSaturationMetric (0.00s) --- PASS: TestSaturationMetric/without_smoothing (0.00s) === RUN TestSaturationMetric_IncorrectUsage === RUN TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_first --- PASS: TestSaturationMetric_IncorrectUsage (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_EmptyAddr --- PASS: TestTCPTransport_EmptyAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) === RUN TestOverrideNotifyBool --- PASS: TestOverrideNotifyBool (0.00s) PASS ok github.com/hashicorp/raft 112.820s ? github.com/hashicorp/raft/bench [no test files] make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' create-stamp debian/debhelper-build-stamp dh_testroot -O--buildsystem=golang dh_prep -O--buildsystem=golang dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang dh_installdocs -O--buildsystem=golang dh_installchangelogs -O--buildsystem=golang dh_installsystemduser -O--buildsystem=golang dh_perl -O--buildsystem=golang dh_link -O--buildsystem=golang dh_strip_nondeterminism -O--buildsystem=golang dh_compress -O--buildsystem=golang dh_fixperms -O--buildsystem=golang dh_missing -O--buildsystem=golang dh_installdeb -O--buildsystem=golang dh_golang -O--buildsystem=golang dh_gencontrol -O--buildsystem=golang dh_md5sums -O--buildsystem=golang dh_builddeb -O--buildsystem=golang dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'. dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_i386.buildinfo dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_i386.changes dpkg-genchanges: info: binary-only upload (no source code included) dpkg-source --after-build . dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/53424 and its subdirectories I: Current time: Tue Apr 21 22:11:10 -12 2026 I: pbuilder-time-stamp: 1776852670