I: pbuilder: network access will be disabled during build I: Current time: Sun May 24 15:40:16 -12 2026 I: pbuilder-time-stamp: 1779680416 I: Building the build Environment I: extracting base tarball [/var/cache/pbuilder/trixie-reproducible-base.tgz] I: copying local configuration W: --override-config is not set; not updating apt.conf Read the manpage for details. I: mounting /proc filesystem I: mounting /sys filesystem I: creating /{dev,run}/shm I: mounting /dev/pts filesystem I: redirecting /dev/ptmx to /dev/pts/ptmx I: policy-rc.d already exists I: Copying source file I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc] I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz] I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz] I: Extracting source dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: unsupported subcommand dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying 0001-Skip-unreliable-tests.patch I: Not using root during the build. I: Installing the build-deps I: user script /srv/workspace/pbuilder/3406684/tmp/hooks/D02_print_environment starting I: set BUILDDIR='/build/reproducible-path' BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other' BUILDUSERNAME='pbuilder1' BUILD_ARCH='arm64' DEBIAN_FRONTEND='noninteractive' DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=12 ' DISTRIBUTION='trixie' HOME='/root' HOST_ARCH='arm64' IFS=' ' INVOCATION_ID='11204da5cdca46b7bfe1f0ba9a1e3f3d' LANG='C' LANGUAGE='en_US:en' LC_ALL='C' MAIL='/var/mail/root' OPTIND='1' PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' PBCURRENTCOMMANDLINEOPERATION='build' PBUILDER_OPERATION='build' PBUILDER_PKGDATADIR='/usr/share/pbuilder' PBUILDER_PKGLIBDIR='/usr/lib/pbuilder' PBUILDER_SYSCONFDIR='/etc' PPID='3406684' PS1='# ' PS2='> ' PS4='+ ' PWD='/' SHELL='/bin/bash' SHLVL='2' SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.qt4E2m4z/pbuilderrc_fopF --distribution trixie --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/trixie-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.qt4E2m4z/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc' SUDO_GID='109' SUDO_UID='104' SUDO_USER='jenkins' TERM='unknown' TZ='/usr/share/zoneinfo/Etc/GMT+12' USER='root' _='/usr/bin/systemd-run' http_proxy='http://192.168.101.4:3128' I: uname -a Linux codethink03-arm64 6.1.0-33-cloud-arm64 #1 SMP Debian 6.1.133-1 (2025-04-10) aarch64 GNU/Linux I: ls -l /bin lrwxrwxrwx 1 root root 7 Mar 4 2025 /bin -> usr/bin I: user script /srv/workspace/pbuilder/3406684/tmp/hooks/D02_print_environment finished -> Attempting to satisfy build-dependencies -> Creating pbuilder-satisfydepends-dummy package Package: pbuilder-satisfydepends-dummy Version: 0.invalid.0 Architecture: arm64 Maintainer: Debian Pbuilder Team Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder This package was created automatically by pbuilder to satisfy the build-dependencies of the package being currently built. Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'. Selecting previously unselected package pbuilder-satisfydepends-dummy. (Reading database ... 19954 files and directories currently installed.) Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ... Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ... dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested: pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however: Package debhelper-compat is not installed. pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however: Package dh-sequence-golang is not installed. pbuilder-satisfydepends-dummy depends on golang-any; however: Package golang-any is not installed. pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however: Package golang-github-armon-go-metrics-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however: Package golang-github-hashicorp-go-hclog-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however: Package golang-github-hashicorp-go-msgpack-dev is not installed. pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however: Package golang-github-stretchr-testify-dev is not installed. Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ... Reading package lists... Building dependency tree... Reading state information... Initializing package states... Writing extended state information... Building tag database... pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0) The following NEW packages will be installed: autoconf{a} automake{a} autopoint{a} autotools-dev{a} binutils-gold{a} binutils-gold-aarch64-linux-gnu{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.24-go{a} golang-1.24-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-prometheus-sigv4-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libunistring5{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} The following packages are RECOMMENDED but will NOT be installed: curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 0 packages upgraded, 108 newly installed, 0 to remove and 0 not upgraded. Need to get 120 MB of archives. After unpacking 687 MB will be used. Writing extended state information... Get: 1 http://deb.debian.org/debian trixie/main arm64 sensible-utils all 0.0.24 [24.8 kB] Get: 2 http://deb.debian.org/debian trixie/main arm64 libmagic-mgc arm64 1:5.46-5 [338 kB] Get: 3 http://deb.debian.org/debian trixie/main arm64 libmagic1t64 arm64 1:5.46-5 [103 kB] Get: 4 http://deb.debian.org/debian trixie/main arm64 file arm64 1:5.46-5 [43.7 kB] Get: 5 http://deb.debian.org/debian trixie/main arm64 gettext-base arm64 0.23.1-1 [241 kB] Get: 6 http://deb.debian.org/debian trixie/main arm64 libuchardet0 arm64 0.0.8-1+b2 [69.2 kB] Get: 7 http://deb.debian.org/debian trixie/main arm64 groff-base arm64 1.23.0-7 [1129 kB] Get: 8 http://deb.debian.org/debian trixie/main arm64 bsdextrautils arm64 2.41-4 [93.9 kB] Get: 9 http://deb.debian.org/debian trixie/main arm64 libpipeline1 arm64 1.5.8-1 [40.2 kB] Get: 10 http://deb.debian.org/debian trixie/main arm64 man-db arm64 2.13.0-1 [1404 kB] Get: 11 http://deb.debian.org/debian trixie/main arm64 m4 arm64 1.4.19-7 [285 kB] Get: 12 http://deb.debian.org/debian trixie/main arm64 autoconf all 2.72-3 [493 kB] Get: 13 http://deb.debian.org/debian trixie/main arm64 autotools-dev all 20240727.1 [60.2 kB] Get: 14 http://deb.debian.org/debian trixie/main arm64 automake all 1:1.17-4 [862 kB] Get: 15 http://deb.debian.org/debian trixie/main arm64 autopoint all 0.23.1-1 [770 kB] Get: 16 http://deb.debian.org/debian trixie/main arm64 binutils-gold-aarch64-linux-gnu arm64 2.44-2 [1700 kB] Get: 17 http://deb.debian.org/debian trixie/main arm64 binutils-gold arm64 2.44-2 [202 kB] Get: 18 http://deb.debian.org/debian trixie/main arm64 libdebhelper-perl all 13.24.2 [90.9 kB] Get: 19 http://deb.debian.org/debian trixie/main arm64 libtool all 2.5.4-4 [539 kB] Get: 20 http://deb.debian.org/debian trixie/main arm64 dh-autoreconf all 20 [17.1 kB] Get: 21 http://deb.debian.org/debian trixie/main arm64 libarchive-zip-perl all 1.68-1 [104 kB] Get: 22 http://deb.debian.org/debian trixie/main arm64 libfile-stripnondeterminism-perl all 1.14.1-2 [19.7 kB] Get: 23 http://deb.debian.org/debian trixie/main arm64 dh-strip-nondeterminism all 1.14.1-2 [8620 B] Get: 24 http://deb.debian.org/debian trixie/main arm64 libelf1t64 arm64 0.192-4 [189 kB] Get: 25 http://deb.debian.org/debian trixie/main arm64 dwz arm64 0.15-1+b1 [102 kB] Get: 26 http://deb.debian.org/debian trixie/main arm64 libunistring5 arm64 1.3-2 [453 kB] Get: 27 http://deb.debian.org/debian trixie/main arm64 libxml2 arm64 2.12.7+dfsg+really2.9.14-0.4 [629 kB] Get: 28 http://deb.debian.org/debian trixie/main arm64 gettext arm64 0.23.1-1 [1610 kB] Get: 29 http://deb.debian.org/debian trixie/main arm64 intltool-debian all 0.35.0+20060710.6 [22.9 kB] Get: 30 http://deb.debian.org/debian trixie/main arm64 po-debconf all 1.0.21+nmu1 [248 kB] Get: 31 http://deb.debian.org/debian trixie/main arm64 debhelper all 13.24.2 [919 kB] Get: 32 http://deb.debian.org/debian trixie/main arm64 dh-golang all 1.63 [24.4 kB] Get: 33 http://deb.debian.org/debian trixie/main arm64 golang-1.24-src all 1.24.2-2 [21.2 MB] Get: 34 http://deb.debian.org/debian trixie/main arm64 golang-1.24-go arm64 1.24.2-2 [24.6 MB] Get: 35 http://deb.debian.org/debian trixie/main arm64 golang-src all 2:1.24~2 [5136 B] Get: 36 http://deb.debian.org/debian trixie/main arm64 golang-go arm64 2:1.24~2 [44.3 kB] Get: 37 http://deb.debian.org/debian trixie/main arm64 golang-any arm64 2:1.24~2 [5216 B] Get: 38 http://deb.debian.org/debian trixie/main arm64 golang-github-alecthomas-units-dev all 0.0~git20240927.0f3dac3-1 [8880 B] Get: 39 http://deb.debian.org/debian trixie/main arm64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB] Get: 40 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB] Get: 41 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-sys-dev all 0.22.0-1 [431 kB] Get: 42 http://deb.debian.org/debian trixie/main arm64 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B] Get: 43 http://deb.debian.org/debian trixie/main arm64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB] Get: 44 http://deb.debian.org/debian trixie/main arm64 golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB] Get: 45 http://deb.debian.org/debian trixie/main arm64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB] Get: 46 http://deb.debian.org/debian trixie/main arm64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB] Get: 47 http://deb.debian.org/debian trixie/main arm64 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB] Get: 48 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB] Get: 49 http://deb.debian.org/debian trixie/main arm64 golang-github-stretchr-testify-dev all 1.10.0-1 [78.4 kB] Get: 50 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB] Get: 51 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB] Get: 52 http://deb.debian.org/debian trixie/main arm64 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB] Get: 53 http://deb.debian.org/debian trixie/main arm64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B] Get: 54 http://deb.debian.org/debian trixie/main arm64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB] Get: 55 http://deb.debian.org/debian trixie/main arm64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB] Get: 56 http://deb.debian.org/debian trixie/main arm64 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB] Get: 57 http://deb.debian.org/debian trixie/main arm64 golang-google-protobuf-dev all 1.36.5-1 [796 kB] Get: 58 http://deb.debian.org/debian trixie/main arm64 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB] Get: 59 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B] Get: 60 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB] Get: 61 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB] Get: 62 http://deb.debian.org/debian trixie/main arm64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB] Get: 63 http://deb.debian.org/debian trixie/main arm64 golang-github-beorn7-perks-dev all 1.0.1-3 [12.2 kB] Get: 64 http://deb.debian.org/debian trixie/main arm64 golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB] Get: 65 http://deb.debian.org/debian trixie/main arm64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B] Get: 66 http://deb.debian.org/debian trixie/main arm64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB] Get: 67 http://deb.debian.org/debian trixie/main arm64 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB] Get: 68 http://deb.debian.org/debian trixie/main arm64 golang-github-klauspost-compress-dev all 1.18.0+ds1-1 [33.3 MB] Get: 69 http://deb.debian.org/debian trixie/main arm64 golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB] Get: 70 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB] Get: 71 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-yaml.v2-dev all 2.4.0-5 [60.5 kB] Get: 72 http://deb.debian.org/debian trixie/main arm64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB] Get: 73 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-term-dev all 0.22.0-1 [14.8 kB] Get: 74 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-text-dev all 0.22.0-1 [3995 kB] Get: 75 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-crypto-dev all 1:0.25.0-1 [1682 kB] Get: 76 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-net-dev all 1:0.27.0-1 [898 kB] Get: 77 http://deb.debian.org/debian trixie/main arm64 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB] Get: 78 http://deb.debian.org/debian trixie/main arm64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB] Get: 79 http://deb.debian.org/debian trixie/main arm64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB] Get: 80 http://deb.debian.org/debian trixie/main arm64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB] Get: 81 http://deb.debian.org/debian trixie/main arm64 golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B] Get: 82 http://deb.debian.org/debian trixie/main arm64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B] Get: 83 http://deb.debian.org/debian trixie/main arm64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-4 [14.6 kB] Get: 84 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-sigv4-dev all 0.1.2-2 [8484 B] Get: 85 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-oauth2-dev all 0.27.0-1 [37.2 kB] Get: 86 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB] Get: 87 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-common-dev all 0.62.0-3 [118 kB] Get: 88 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB] Get: 89 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-procfs-dev all 0.15.1-1 [184 kB] Get: 90 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-client-golang-dev all 1.21.1-1 [194 kB] Get: 91 http://deb.debian.org/debian trixie/main arm64 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB] Get: 92 http://deb.debian.org/debian trixie/main arm64 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB] Get: 93 http://deb.debian.org/debian trixie/main arm64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B] Get: 94 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB] Get: 95 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B] Get: 96 http://deb.debian.org/debian trixie/main arm64 libsasl2-modules-db arm64 2.1.28+dfsg1-9 [20.1 kB] Get: 97 http://deb.debian.org/debian trixie/main arm64 libsasl2-2 arm64 2.1.28+dfsg1-9 [55.6 kB] Get: 98 http://deb.debian.org/debian trixie/main arm64 libssl-dev arm64 3.5.0-1 [3383 kB] Get: 99 http://deb.debian.org/debian trixie/main arm64 libsasl2-dev arm64 2.1.28+dfsg1-9 [229 kB] Get: 100 http://deb.debian.org/debian trixie/main arm64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB] Get: 101 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-mod-dev all 0.19.0-1 [104 kB] Get: 102 http://deb.debian.org/debian trixie/main arm64 golang-github-yuin-goldmark-dev all 1.7.4-1 [113 kB] Get: 103 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-tools-dev all 1:0.25.0+ds-1 [1608 kB] Get: 104 http://deb.debian.org/debian trixie/main arm64 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB] Get: 105 http://deb.debian.org/debian trixie/main arm64 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB] Get: 106 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-2 [31.2 kB] Get: 107 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB] Get: 108 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB] Fetched 120 MB in 1s (235 MB/s) Preconfiguring packages ... Selecting previously unselected package sensible-utils. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 19954 files and directories currently installed.) Preparing to unpack .../000-sensible-utils_0.0.24_all.deb ... Unpacking sensible-utils (0.0.24) ... Selecting previously unselected package libmagic-mgc. Preparing to unpack .../001-libmagic-mgc_1%3a5.46-5_arm64.deb ... Unpacking libmagic-mgc (1:5.46-5) ... Selecting previously unselected package libmagic1t64:arm64. Preparing to unpack .../002-libmagic1t64_1%3a5.46-5_arm64.deb ... Unpacking libmagic1t64:arm64 (1:5.46-5) ... Selecting previously unselected package file. Preparing to unpack .../003-file_1%3a5.46-5_arm64.deb ... Unpacking file (1:5.46-5) ... Selecting previously unselected package gettext-base. Preparing to unpack .../004-gettext-base_0.23.1-1_arm64.deb ... Unpacking gettext-base (0.23.1-1) ... Selecting previously unselected package libuchardet0:arm64. Preparing to unpack .../005-libuchardet0_0.0.8-1+b2_arm64.deb ... Unpacking libuchardet0:arm64 (0.0.8-1+b2) ... Selecting previously unselected package groff-base. Preparing to unpack .../006-groff-base_1.23.0-7_arm64.deb ... Unpacking groff-base (1.23.0-7) ... Selecting previously unselected package bsdextrautils. Preparing to unpack .../007-bsdextrautils_2.41-4_arm64.deb ... Unpacking bsdextrautils (2.41-4) ... Selecting previously unselected package libpipeline1:arm64. Preparing to unpack .../008-libpipeline1_1.5.8-1_arm64.deb ... Unpacking libpipeline1:arm64 (1.5.8-1) ... Selecting previously unselected package man-db. Preparing to unpack .../009-man-db_2.13.0-1_arm64.deb ... Unpacking man-db (2.13.0-1) ... Selecting previously unselected package m4. Preparing to unpack .../010-m4_1.4.19-7_arm64.deb ... Unpacking m4 (1.4.19-7) ... Selecting previously unselected package autoconf. Preparing to unpack .../011-autoconf_2.72-3_all.deb ... Unpacking autoconf (2.72-3) ... Selecting previously unselected package autotools-dev. Preparing to unpack .../012-autotools-dev_20240727.1_all.deb ... Unpacking autotools-dev (20240727.1) ... Selecting previously unselected package automake. Preparing to unpack .../013-automake_1%3a1.17-4_all.deb ... Unpacking automake (1:1.17-4) ... Selecting previously unselected package autopoint. Preparing to unpack .../014-autopoint_0.23.1-1_all.deb ... Unpacking autopoint (0.23.1-1) ... Selecting previously unselected package binutils-gold-aarch64-linux-gnu. Preparing to unpack .../015-binutils-gold-aarch64-linux-gnu_2.44-2_arm64.deb ... Unpacking binutils-gold-aarch64-linux-gnu (2.44-2) ... Selecting previously unselected package binutils-gold. Preparing to unpack .../016-binutils-gold_2.44-2_arm64.deb ... Unpacking binutils-gold (2.44-2) ... Selecting previously unselected package libdebhelper-perl. Preparing to unpack .../017-libdebhelper-perl_13.24.2_all.deb ... Unpacking libdebhelper-perl (13.24.2) ... Selecting previously unselected package libtool. Preparing to unpack .../018-libtool_2.5.4-4_all.deb ... Unpacking libtool (2.5.4-4) ... Selecting previously unselected package dh-autoreconf. Preparing to unpack .../019-dh-autoreconf_20_all.deb ... Unpacking dh-autoreconf (20) ... Selecting previously unselected package libarchive-zip-perl. Preparing to unpack .../020-libarchive-zip-perl_1.68-1_all.deb ... Unpacking libarchive-zip-perl (1.68-1) ... Selecting previously unselected package libfile-stripnondeterminism-perl. Preparing to unpack .../021-libfile-stripnondeterminism-perl_1.14.1-2_all.deb ... Unpacking libfile-stripnondeterminism-perl (1.14.1-2) ... Selecting previously unselected package dh-strip-nondeterminism. Preparing to unpack .../022-dh-strip-nondeterminism_1.14.1-2_all.deb ... Unpacking dh-strip-nondeterminism (1.14.1-2) ... Selecting previously unselected package libelf1t64:arm64. Preparing to unpack .../023-libelf1t64_0.192-4_arm64.deb ... Unpacking libelf1t64:arm64 (0.192-4) ... Selecting previously unselected package dwz. Preparing to unpack .../024-dwz_0.15-1+b1_arm64.deb ... Unpacking dwz (0.15-1+b1) ... Selecting previously unselected package libunistring5:arm64. Preparing to unpack .../025-libunistring5_1.3-2_arm64.deb ... Unpacking libunistring5:arm64 (1.3-2) ... Selecting previously unselected package libxml2:arm64. Preparing to unpack .../026-libxml2_2.12.7+dfsg+really2.9.14-0.4_arm64.deb ... Unpacking libxml2:arm64 (2.12.7+dfsg+really2.9.14-0.4) ... Selecting previously unselected package gettext. Preparing to unpack .../027-gettext_0.23.1-1_arm64.deb ... Unpacking gettext (0.23.1-1) ... Selecting previously unselected package intltool-debian. Preparing to unpack .../028-intltool-debian_0.35.0+20060710.6_all.deb ... Unpacking intltool-debian (0.35.0+20060710.6) ... Selecting previously unselected package po-debconf. Preparing to unpack .../029-po-debconf_1.0.21+nmu1_all.deb ... Unpacking po-debconf (1.0.21+nmu1) ... Selecting previously unselected package debhelper. Preparing to unpack .../030-debhelper_13.24.2_all.deb ... Unpacking debhelper (13.24.2) ... Selecting previously unselected package dh-golang. Preparing to unpack .../031-dh-golang_1.63_all.deb ... Unpacking dh-golang (1.63) ... Selecting previously unselected package golang-1.24-src. Preparing to unpack .../032-golang-1.24-src_1.24.2-2_all.deb ... Unpacking golang-1.24-src (1.24.2-2) ... Selecting previously unselected package golang-1.24-go. Preparing to unpack .../033-golang-1.24-go_1.24.2-2_arm64.deb ... Unpacking golang-1.24-go (1.24.2-2) ... Selecting previously unselected package golang-src. Preparing to unpack .../034-golang-src_2%3a1.24~2_all.deb ... Unpacking golang-src (2:1.24~2) ... Selecting previously unselected package golang-go:arm64. Preparing to unpack .../035-golang-go_2%3a1.24~2_arm64.deb ... Unpacking golang-go:arm64 (2:1.24~2) ... Selecting previously unselected package golang-any:arm64. Preparing to unpack .../036-golang-any_2%3a1.24~2_arm64.deb ... Unpacking golang-any:arm64 (2:1.24~2) ... Selecting previously unselected package golang-github-alecthomas-units-dev. Preparing to unpack .../037-golang-github-alecthomas-units-dev_0.0~git20240927.0f3dac3-1_all.deb ... Unpacking golang-github-alecthomas-units-dev (0.0~git20240927.0f3dac3-1) ... Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev. Preparing to unpack .../038-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ... Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev. Preparing to unpack .../039-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ... Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Selecting previously unselected package golang-golang-x-sys-dev. Preparing to unpack .../040-golang-golang-x-sys-dev_0.22.0-1_all.deb ... Unpacking golang-golang-x-sys-dev (0.22.0-1) ... Selecting previously unselected package golang-github-mattn-go-isatty-dev. Preparing to unpack .../041-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ... Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ... Selecting previously unselected package golang-github-mattn-go-colorable-dev. Preparing to unpack .../042-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ... Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ... Selecting previously unselected package golang-github-fatih-color-dev. Preparing to unpack .../043-golang-github-fatih-color-dev_1.17.0-1_all.deb ... Unpacking golang-github-fatih-color-dev (1.17.0-1) ... Selecting previously unselected package golang-github-davecgh-go-spew-dev. Preparing to unpack .../044-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ... Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ... Selecting previously unselected package golang-github-pmezard-go-difflib-dev. Preparing to unpack .../045-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ... Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Selecting previously unselected package golang-github-stretchr-objx-dev. Preparing to unpack .../046-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ... Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ... Selecting previously unselected package golang-gopkg-yaml.v3-dev. Preparing to unpack .../047-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ... Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ... Selecting previously unselected package golang-github-stretchr-testify-dev. Preparing to unpack .../048-golang-github-stretchr-testify-dev_1.10.0-1_all.deb ... Unpacking golang-github-stretchr-testify-dev (1.10.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-hclog-dev. Preparing to unpack .../049-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ... Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev. Preparing to unpack .../050-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ... Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Selecting previously unselected package golang-github-pkg-errors-dev. Preparing to unpack .../051-golang-github-pkg-errors-dev_0.9.1-3_all.deb ... Unpacking golang-github-pkg-errors-dev (0.9.1-3) ... Selecting previously unselected package golang-github-tv42-httpunix-dev. Preparing to unpack .../052-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ... Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev. Preparing to unpack .../053-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ... Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Selecting previously unselected package golang-github-datadog-datadog-go-dev. Preparing to unpack .../054-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ... Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ... Selecting previously unselected package golang-github-google-go-cmp-dev. Preparing to unpack .../055-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ... Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ... Selecting previously unselected package golang-google-protobuf-dev. Preparing to unpack .../056-golang-google-protobuf-dev_1.36.5-1_all.deb ... Unpacking golang-google-protobuf-dev (1.36.5-1) ... Selecting previously unselected package golang-github-golang-protobuf-1-5-dev. Preparing to unpack .../057-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ... Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ... Selecting previously unselected package golang-github-hashicorp-go-uuid-dev. Preparing to unpack .../058-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ... Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Selecting previously unselected package golang-github-hashicorp-golang-lru-dev. Preparing to unpack .../059-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ... Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev. Preparing to unpack .../060-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ... Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Selecting previously unselected package golang-github-pascaldekloe-goe-dev. Preparing to unpack .../061-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ... Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Selecting previously unselected package golang-github-beorn7-perks-dev. Preparing to unpack .../062-golang-github-beorn7-perks-dev_1.0.1-3_all.deb ... Unpacking golang-github-beorn7-perks-dev (1.0.1-3) ... Selecting previously unselected package golang-github-cespare-xxhash-dev. Preparing to unpack .../063-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ... Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ... Selecting previously unselected package golang-github-modern-go-concurrent-dev. Preparing to unpack .../064-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ... Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Selecting previously unselected package golang-github-modern-go-reflect2-dev. Preparing to unpack .../065-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ... Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ... Selecting previously unselected package golang-github-json-iterator-go-dev. Preparing to unpack .../066-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ... Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ... Selecting previously unselected package golang-github-klauspost-compress-dev. Preparing to unpack .../067-golang-github-klauspost-compress-dev_1.18.0+ds1-1_all.deb ... Unpacking golang-github-klauspost-compress-dev (1.18.0+ds1-1) ... Selecting previously unselected package golang-github-kylelemons-godebug-dev. Preparing to unpack .../068-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ... Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ... Selecting previously unselected package golang-github-prometheus-client-model-dev. Preparing to unpack .../069-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ... Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ... Selecting previously unselected package golang-gopkg-yaml.v2-dev. Preparing to unpack .../070-golang-gopkg-yaml.v2-dev_2.4.0-5_all.deb ... Unpacking golang-gopkg-yaml.v2-dev (2.4.0-5) ... Selecting previously unselected package golang-github-jmespath-go-jmespath-dev. Preparing to unpack .../071-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ... Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Selecting previously unselected package golang-golang-x-term-dev. Preparing to unpack .../072-golang-golang-x-term-dev_0.22.0-1_all.deb ... Unpacking golang-golang-x-term-dev (0.22.0-1) ... Selecting previously unselected package golang-golang-x-text-dev. Preparing to unpack .../073-golang-golang-x-text-dev_0.22.0-1_all.deb ... Unpacking golang-golang-x-text-dev (0.22.0-1) ... Selecting previously unselected package golang-golang-x-crypto-dev. Preparing to unpack .../074-golang-golang-x-crypto-dev_1%3a0.25.0-1_all.deb ... Unpacking golang-golang-x-crypto-dev (1:0.25.0-1) ... Selecting previously unselected package golang-golang-x-net-dev. Preparing to unpack .../075-golang-golang-x-net-dev_1%3a0.27.0-1_all.deb ... Unpacking golang-golang-x-net-dev (1:0.27.0-1) ... Selecting previously unselected package golang-github-aws-aws-sdk-go-dev. Preparing to unpack .../076-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ... Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ... Selecting previously unselected package golang-github-go-logfmt-logfmt-dev. Preparing to unpack .../077-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ... Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Selecting previously unselected package golang-github-go-kit-log-dev. Preparing to unpack .../078-golang-github-go-kit-log-dev_0.2.1-1_all.deb ... Unpacking golang-github-go-kit-log-dev (0.2.1-1) ... Selecting previously unselected package golang-github-julienschmidt-httprouter-dev. Preparing to unpack .../079-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ... Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Selecting previously unselected package golang-github-munnerz-goautoneg-dev. Preparing to unpack .../080-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ... Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ... Selecting previously unselected package golang-github-jpillora-backoff-dev. Preparing to unpack .../081-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ... Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev. Preparing to unpack .../082-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-4_all.deb ... Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ... Selecting previously unselected package golang-github-prometheus-sigv4-dev. Preparing to unpack .../083-golang-github-prometheus-sigv4-dev_0.1.2-2_all.deb ... Unpacking golang-github-prometheus-sigv4-dev (0.1.2-2) ... Selecting previously unselected package golang-golang-x-oauth2-dev. Preparing to unpack .../084-golang-golang-x-oauth2-dev_0.27.0-1_all.deb ... Unpacking golang-golang-x-oauth2-dev (0.27.0-1) ... Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev. Preparing to unpack .../085-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ... Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Selecting previously unselected package golang-github-prometheus-common-dev. Preparing to unpack .../086-golang-github-prometheus-common-dev_0.62.0-3_all.deb ... Unpacking golang-github-prometheus-common-dev (0.62.0-3) ... Selecting previously unselected package golang-golang-x-sync-dev. Preparing to unpack .../087-golang-golang-x-sync-dev_0.7.0-1_all.deb ... Unpacking golang-golang-x-sync-dev (0.7.0-1) ... Selecting previously unselected package golang-github-prometheus-procfs-dev. Preparing to unpack .../088-golang-github-prometheus-procfs-dev_0.15.1-1_all.deb ... Unpacking golang-github-prometheus-procfs-dev (0.15.1-1) ... Selecting previously unselected package golang-github-prometheus-client-golang-dev. Preparing to unpack .../089-golang-github-prometheus-client-golang-dev_1.21.1-1_all.deb ... Unpacking golang-github-prometheus-client-golang-dev (1.21.1-1) ... Selecting previously unselected package golang-github-armon-go-metrics-dev. Preparing to unpack .../090-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ... Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ... Selecting previously unselected package golang-github-creack-pty-dev. Preparing to unpack .../091-golang-github-creack-pty-dev_1.1.21-1_all.deb ... Unpacking golang-github-creack-pty-dev (1.1.21-1) ... Selecting previously unselected package golang-github-vmihailenco-tagparser-dev. Preparing to unpack .../092-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ... Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev. Preparing to unpack .../093-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ... Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Selecting previously unselected package golang-gopkg-tomb.v2-dev. Preparing to unpack .../094-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ... Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Selecting previously unselected package libsasl2-modules-db:arm64. Preparing to unpack .../095-libsasl2-modules-db_2.1.28+dfsg1-9_arm64.deb ... Unpacking libsasl2-modules-db:arm64 (2.1.28+dfsg1-9) ... Selecting previously unselected package libsasl2-2:arm64. Preparing to unpack .../096-libsasl2-2_2.1.28+dfsg1-9_arm64.deb ... Unpacking libsasl2-2:arm64 (2.1.28+dfsg1-9) ... Selecting previously unselected package libssl-dev:arm64. Preparing to unpack .../097-libssl-dev_3.5.0-1_arm64.deb ... Unpacking libssl-dev:arm64 (3.5.0-1) ... Selecting previously unselected package libsasl2-dev. Preparing to unpack .../098-libsasl2-dev_2.1.28+dfsg1-9_arm64.deb ... Unpacking libsasl2-dev (2.1.28+dfsg1-9) ... Selecting previously unselected package golang-github-kr-text-dev. Preparing to unpack .../099-golang-github-kr-text-dev_0.2.0-2_all.deb ... Unpacking golang-github-kr-text-dev (0.2.0-2) ... Selecting previously unselected package golang-golang-x-mod-dev. Preparing to unpack .../100-golang-golang-x-mod-dev_0.19.0-1_all.deb ... Unpacking golang-golang-x-mod-dev (0.19.0-1) ... Selecting previously unselected package golang-github-yuin-goldmark-dev. Preparing to unpack .../101-golang-github-yuin-goldmark-dev_1.7.4-1_all.deb ... Unpacking golang-github-yuin-goldmark-dev (1.7.4-1) ... Selecting previously unselected package golang-golang-x-tools-dev. Preparing to unpack .../102-golang-golang-x-tools-dev_1%3a0.25.0+ds-1_all.deb ... Unpacking golang-golang-x-tools-dev (1:0.25.0+ds-1) ... Selecting previously unselected package golang-github-rogpeppe-go-internal-dev. Preparing to unpack .../103-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ... Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Selecting previously unselected package golang-github-kr-pretty-dev. Preparing to unpack .../104-golang-github-kr-pretty-dev_0.3.1-1_all.deb ... Unpacking golang-github-kr-pretty-dev (0.3.1-1) ... Selecting previously unselected package golang-gopkg-check.v1-dev. Preparing to unpack .../105-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-2_all.deb ... Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ... Selecting previously unselected package golang-gopkg-mgo.v2-dev. Preparing to unpack .../106-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ... Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev. Preparing to unpack .../107-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ... Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up golang-1.24-src (1.24.2-2) ... Setting up libpipeline1:arm64 (1.5.8-1) ... Setting up golang-github-google-go-cmp-dev (0.6.0-1) ... Setting up golang-gopkg-yaml.v2-dev (2.4.0-5) ... Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ... Setting up bsdextrautils (2.41-4) ... Setting up golang-github-creack-pty-dev (1.1.21-1) ... Setting up golang-github-stretchr-objx-dev (0.5.2-1) ... Setting up libmagic-mgc (1:5.46-5) ... Setting up golang-github-pkg-errors-dev (0.9.1-3) ... Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ... Setting up libarchive-zip-perl (1.68-1) ... Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ... Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ... Setting up libdebhelper-perl (13.24.2) ... Setting up golang-golang-x-sys-dev (0.22.0-1) ... Setting up libmagic1t64:arm64 (1:5.46-5) ... Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ... Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ... Setting up gettext-base (0.23.1-1) ... Setting up m4 (1.4.19-7) ... Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ... Setting up binutils-gold-aarch64-linux-gnu (2.44-2) ... Setting up golang-golang-x-term-dev (0.22.0-1) ... Setting up file (1:5.46-5) ... Setting up golang-github-beorn7-perks-dev (1.0.1-3) ... Setting up libelf1t64:arm64 (0.192-4) ... Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ... Setting up libsasl2-modules-db:arm64 (2.1.28+dfsg1-9) ... Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ... Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ... Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ... Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ... Setting up autotools-dev (20240727.1) ... Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ... Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ... Setting up golang-google-protobuf-dev (1.36.5-1) ... Setting up golang-github-kr-text-dev (0.2.0-2) ... Setting up golang-github-stretchr-testify-dev (1.10.0-1) ... Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ... Setting up golang-github-yuin-goldmark-dev (1.7.4-1) ... Setting up libunistring5:arm64 (1.3-2) ... Setting up libssl-dev:arm64 (3.5.0-1) ... Setting up golang-golang-x-sync-dev (0.7.0-1) ... Setting up golang-golang-x-mod-dev (0.19.0-1) ... Setting up autopoint (0.23.1-1) ... Setting up golang-github-klauspost-compress-dev (1.18.0+ds1-1) ... Setting up libsasl2-2:arm64 (2.1.28+dfsg1-9) ... Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ... Setting up autoconf (2.72-3) ... Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ... Setting up dwz (0.15-1+b1) ... Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ... Setting up sensible-utils (0.0.24) ... Setting up golang-golang-x-text-dev (0.22.0-1) ... Setting up libuchardet0:arm64 (0.0.8-1+b2) ... Setting up golang-github-prometheus-procfs-dev (0.15.1-1) ... Setting up golang-github-alecthomas-units-dev (0.0~git20240927.0f3dac3-1) ... Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ... Setting up golang-golang-x-crypto-dev (1:0.25.0-1) ... Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ... Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ... Setting up golang-golang-x-oauth2-dev (0.27.0-1) ... Setting up golang-src (2:1.24~2) ... Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ... Setting up libxml2:arm64 (2.12.7+dfsg+really2.9.14-0.4) ... Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ... Setting up automake (1:1.17-4) ... update-alternatives: using /usr/bin/automake-1.17 to provide /usr/bin/automake (automake) in auto mode Setting up libfile-stripnondeterminism-perl (1.14.1-2) ... Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ... Setting up golang-github-go-kit-log-dev (0.2.1-1) ... Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ... Setting up gettext (0.23.1-1) ... Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ... Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ... Setting up libtool (2.5.4-4) ... Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ... Setting up binutils-gold (2.44-2) ... Setting up golang-github-json-iterator-go-dev (1.1.12-2) ... Setting up intltool-debian (0.35.0+20060710.6) ... Setting up dh-autoreconf (20) ... Setting up libsasl2-dev (2.1.28+dfsg1-9) ... Setting up golang-1.24-go (1.24.2-2) ... Setting up golang-golang-x-net-dev (1:0.27.0-1) ... Setting up dh-strip-nondeterminism (1.14.1-2) ... Setting up groff-base (1.23.0-7) ... Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ... Setting up golang-github-fatih-color-dev (1.17.0-1) ... Setting up golang-go:arm64 (2:1.24~2) ... Setting up po-debconf (1.0.21+nmu1) ... Setting up golang-golang-x-tools-dev (1:0.25.0+ds-1) ... Setting up man-db (2.13.0-1) ... Not building database; man-db/auto-update is not 'true'. Setting up golang-any:arm64 (2:1.24~2) ... Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ... Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ... Setting up golang-github-kr-pretty-dev (0.3.1-1) ... Setting up debhelper (13.24.2) ... Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ... Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ... Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ... Setting up dh-golang (1.63) ... Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ... Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ... Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ... Setting up golang-github-prometheus-sigv4-dev (0.1.2-2) ... Setting up golang-github-prometheus-common-dev (0.62.0-3) ... Setting up golang-github-prometheus-client-golang-dev (1.21.1-1) ... Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ... Processing triggers for libc-bin (2.41-7) ... Reading package lists... Building dependency tree... Reading state information... Reading extended state information... Initializing package states... Writing extended state information... Building tag database... -> Finished parsing the build-deps I: Building the package I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S > ../golang-github-hashicorp-raft_1.5.0-1_source.changes dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.5.0-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Daniel Swarbrick dpkg-source --before-build . dpkg-buildpackage: info: host architecture arm64 debian/rules clean dh clean --buildsystem=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules binary dh binary --buildsystem=golang dh_update_autotools_config -O--buildsystem=golang dh_autoreconf -O--buildsystem=golang dh_auto_configure -O--buildsystem=golang dh_auto_build -O--buildsystem=golang cd obj-aarch64-linux-gnu && go install -trimpath -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/coverage/rtcov internal/goarch internal/byteorder internal/asan internal/godebugs internal/goos internal/goexperiment internal/cpu internal/profilerecord internal/runtime/math internal/unsafeheader internal/runtime/syscall internal/chacha8rand internal/abi internal/msan sync/atomic math/bits unicode/utf8 unicode container/list internal/itoa crypto/internal/fips140/alias crypto/internal/fips140deps/byteorder crypto/internal/fips140/subtle cmp crypto/internal/boring/sig encoding unicode/utf16 log/internal internal/nettrace vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/alias internal/runtime/atomic internal/bytealg internal/runtime/sys crypto/internal/fips140deps/cpu math internal/stringslite internal/runtime/exithook internal/race internal/sync internal/runtime/maps runtime internal/reflectlite weak sync iter crypto/subtle slices maps errors sort internal/bisect internal/testlog internal/singleflight unique runtime/cgo io strconv internal/oserror path vendor/golang.org/x/net/dns/dnsmessage math/rand/v2 github.com/hashicorp/golang-lru/simplelru syscall internal/godebug bytes strings crypto/internal/fips140deps/godebug hash crypto/internal/randutil net/netip reflect internal/saferio math/rand crypto hash/crc32 vendor/golang.org/x/text/transform hash/crc64 bufio crypto/internal/fips140 github.com/hashicorp/go-immutable-radix crypto/internal/impl net/http/internal/ascii regexp/syntax html crypto/internal/fips140/sha3 crypto/internal/fips140/sha256 crypto/internal/fips140/sha512 crypto/tls/internal/fips140tls crypto/sha3 crypto/internal/fips140/hmac crypto/internal/fips140hash time internal/syscall/execenv internal/syscall/unix crypto/internal/fips140/check crypto/internal/fips140/aes crypto/internal/fips140/nistec/fiat crypto/internal/fips140/edwards25519/field crypto/internal/fips140/bigmod crypto/internal/fips140/hkdf crypto/internal/fips140/tls12 regexp crypto/internal/fips140/tls13 crypto/internal/fips140/edwards25519 context io/fs internal/poll crypto/internal/fips140/nistec internal/filepathlite os internal/fmtsort encoding/binary encoding/base64 vendor/golang.org/x/crypto/internal/poly1305 golang.org/x/sys/unix encoding/pem crypto/internal/sysrand fmt io/ioutil os/signal internal/sysinfo path/filepath net crypto/internal/entropy crypto/internal/fips140/drbg crypto/internal/fips140only crypto/internal/fips140/rsa crypto/internal/fips140/mlkem crypto/internal/fips140/ecdh crypto/internal/fips140/ecdsa crypto/internal/fips140/ed25519 crypto/internal/fips140/aes/gcm crypto/md5 crypto/rc4 crypto/cipher encoding/json math/big log net/url compress/flate encoding/hex vendor/golang.org/x/text/unicode/norm vendor/golang.org/x/net/http2/hpack mime crypto/internal/boring crypto/des crypto/aes crypto/ecdh crypto/sha512 crypto/hmac vendor/golang.org/x/crypto/chacha20 crypto/sha1 crypto/sha256 vendor/golang.org/x/text/unicode/bidi mime/quotedprintable compress/gzip net/http/internal encoding/gob go/token vendor/golang.org/x/crypto/chacha20poly1305 text/template/parse flag runtime/debug runtime/trace vendor/golang.org/x/text/secure/bidirule github.com/mattn/go-isatty vendor/golang.org/x/net/idna testing github.com/mattn/go-colorable github.com/fatih/color crypto/rand crypto/elliptic crypto/internal/boring/bbig encoding/asn1 crypto/dsa text/template crypto/ed25519 crypto/internal/hpke crypto/rsa github.com/hashicorp/go-hclog vendor/golang.org/x/crypto/cryptobyte crypto/x509/pkix html/template crypto/ecdsa crypto/x509 net/textproto vendor/golang.org/x/net/http/httpproxy vendor/golang.org/x/net/http/httpguts mime/multipart crypto/tls net/http/httptrace net/http net/rpc github.com/armon/go-metrics github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench debian/rules override_dh_auto_test make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' dh_auto_test cd obj-aarch64-linux-gnu && go test -vet=off -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir 2026-05-25T03:41:13.834Z [INFO] creating new snapshot: path=/tmp/raft3072440277/raft200096240/snapshots/3-10-1779680473834.tmp --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s) === RUN TestFileSS_CreateSnapshot 2026-05-25T03:41:13.835Z [INFO] creating new snapshot: path=/tmp/raft659165990/snapshots/3-10-1779680473835.tmp 2026-05-25T03:41:13.835Z [WARN] found temporary snapshot: name=3-10-1779680473835.tmp --- PASS: TestFileSS_CreateSnapshot (0.00s) === RUN TestFileSS_CancelSnapshot 2026-05-25T03:41:13.836Z [INFO] creating new snapshot: path=/tmp/raft155166837/snapshots/3-10-1779680473836.tmp --- PASS: TestFileSS_CancelSnapshot (0.00s) === RUN TestFileSS_Retention 2026-05-25T03:41:13.836Z [INFO] creating new snapshot: path=/tmp/raft2084448904/snapshots/3-10-1779680473836.tmp 2026-05-25T03:41:13.837Z [INFO] creating new snapshot: path=/tmp/raft2084448904/snapshots/3-11-1779680473837.tmp 2026-05-25T03:41:13.837Z [INFO] creating new snapshot: path=/tmp/raft2084448904/snapshots/3-12-1779680473837.tmp 2026-05-25T03:41:13.837Z [INFO] reaping snapshot: path=/tmp/raft2084448904/snapshots/3-10-1779680473836 2026-05-25T03:41:13.837Z [INFO] creating new snapshot: path=/tmp/raft2084448904/snapshots/3-13-1779680473837.tmp 2026-05-25T03:41:13.838Z [INFO] reaping snapshot: path=/tmp/raft2084448904/snapshots/3-11-1779680473837 2026-05-25T03:41:13.838Z [INFO] creating new snapshot: path=/tmp/raft2084448904/snapshots/3-14-1779680473838.tmp 2026-05-25T03:41:13.838Z [INFO] reaping snapshot: path=/tmp/raft2084448904/snapshots/3-12-1779680473837 --- PASS: TestFileSS_Retention (0.00s) === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering 2026-05-25T03:41:13.839Z [INFO] creating new snapshot: path=/tmp/raft2436104053/snapshots/5-130350-1779680473839.tmp 2026-05-25T03:41:13.839Z [INFO] creating new snapshot: path=/tmp/raft2436104053/snapshots/36-204917-1779680473839.tmp --- PASS: TestFileSS_Ordering (0.00s) === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestRaft_RestartFollower_LongInitialHeartbeat --- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestLogCacheWithBackendStoreError --- PASS: TestLogCacheWithBackendStoreError (0.00s) === RUN TestOldestLog === RUN TestOldestLog/empty_logs === RUN TestOldestLog/simple_case --- PASS: TestOldestLog (0.00s) --- PASS: TestOldestLog/empty_logs (0.00s) --- PASS: TestOldestLog/simple_case (0.00s) === RUN TestEmitsLogStoreMetrics --- PASS: TestEmitsLogStoreMetrics (0.01s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.00s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams --- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.02s) === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 2026-05-25T03:41:13.943Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 === RUN TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 2026-05-25T03:41:13.995Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" 2026-05-25T03:41:14.047Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown" --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s) --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s) === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff net_transport_test.go:839: countingWriter: 2026-05-25T03:41:14.055Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-05-25T03:41:14.061Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-05-25T03:41:14.072Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-05-25T03:41:14.093Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-05-25T03:41:14.134Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-05-25T03:41:14.214Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-05-25T03:41:14.375Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-05-25T03:41:14.696Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-05-25T03:41:15.336Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-05-25T03:41:16.338Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:839: countingWriter: 2026-05-25T03:41:17.340Z [ERROR] test: failed to accept connection: error="intentional error in test" --- PASS: TestNetworkTransport_ListenBackoff (4.00s) === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop 2026-05-25T03:41:18.056Z [INFO] server-dd8c49c9-0db4-025a-917b-b548b69fa4db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd8c49c9-0db4-025a-917b-b548b69fa4db Address:dd8c49c9-0db4-025a-917b-b548b69fa4db}]" --- PASS: TestRaft_StartStop (0.00s) === RUN TestRaft_AfterShutdown 2026-05-25T03:41:18.057Z [INFO] server-2148f0d1-c951-1226-ffc2-74c91459eb0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2148f0d1-c951-1226-ffc2-74c91459eb0e Address:2148f0d1-c951-1226-ffc2-74c91459eb0e}]" --- PASS: TestRaft_AfterShutdown (0.00s) === RUN TestRaft_LiveBootstrap 2026-05-25T03:41:18.057Z [INFO] server-ff57f0c0-a106-b7cd-e1d3-d05e7aa12a7a: initial configuration: index=0 servers=[] 2026-05-25T03:41:18.058Z [INFO] server-d6d6e268-f93f-57ca-a958-e9f5538b90d8: initial configuration: index=0 servers=[] 2026-05-25T03:41:18.058Z [INFO] server-9d07d493-a474-67b5-c9e4-d22897dd6ae1: initial configuration: index=0 servers=[] 2026-05-25T03:41:18.058Z [INFO] server-9d07d493-a474-67b5-c9e4-d22897dd6ae1: entering follower state: follower="Node at 9d07d493-a474-67b5-c9e4-d22897dd6ae1 [Follower]" leader-address= leader-id= 2026-05-25T03:41:18.058Z [INFO] server-ff57f0c0-a106-b7cd-e1d3-d05e7aa12a7a: entering follower state: follower="Node at ff57f0c0-a106-b7cd-e1d3-d05e7aa12a7a [Follower]" leader-address= leader-id= 2026-05-25T03:41:18.058Z [INFO] server-d6d6e268-f93f-57ca-a958-e9f5538b90d8: entering follower state: follower="Node at d6d6e268-f93f-57ca-a958-e9f5538b90d8 [Follower]" leader-address= leader-id= 2026-05-25T03:41:18.058Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:18.111Z [WARN] server-d6d6e268-f93f-57ca-a958-e9f5538b90d8: no known peers, aborting election 2026-05-25T03:41:18.112Z [WARN] server-ff57f0c0-a106-b7cd-e1d3-d05e7aa12a7a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:18.112Z [INFO] server-ff57f0c0-a106-b7cd-e1d3-d05e7aa12a7a: entering candidate state: node="Node at ff57f0c0-a106-b7cd-e1d3-d05e7aa12a7a [Candidate]" term=2 2026-05-25T03:41:18.113Z [INFO] server-ff57f0c0-a106-b7cd-e1d3-d05e7aa12a7a: election won: term=2 tally=2 2026-05-25T03:41:18.113Z [INFO] server-ff57f0c0-a106-b7cd-e1d3-d05e7aa12a7a: entering leader state: leader="Node at ff57f0c0-a106-b7cd-e1d3-d05e7aa12a7a [Leader]" 2026-05-25T03:41:18.113Z [INFO] server-ff57f0c0-a106-b7cd-e1d3-d05e7aa12a7a: added peer, starting replication: peer=server-d6d6e268-f93f-57ca-a958-e9f5538b90d8 2026-05-25T03:41:18.113Z [INFO] server-ff57f0c0-a106-b7cd-e1d3-d05e7aa12a7a: added peer, starting replication: peer=server-9d07d493-a474-67b5-c9e4-d22897dd6ae1 2026-05-25T03:41:18.113Z [WARN] server-9d07d493-a474-67b5-c9e4-d22897dd6ae1: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2026-05-25T03:41:18.113Z [WARN] server-d6d6e268-f93f-57ca-a958-e9f5538b90d8: failed to get previous log: previous-index=1 last-index=0 error="log not found" 2026-05-25T03:41:18.113Z [WARN] server-ff57f0c0-a106-b7cd-e1d3-d05e7aa12a7a: appendEntries rejected, sending older logs: peer="{Voter server-9d07d493-a474-67b5-c9e4-d22897dd6ae1 9d07d493-a474-67b5-c9e4-d22897dd6ae1}" next=1 2026-05-25T03:41:18.113Z [WARN] server-ff57f0c0-a106-b7cd-e1d3-d05e7aa12a7a: appendEntries rejected, sending older logs: peer="{Voter server-d6d6e268-f93f-57ca-a958-e9f5538b90d8 d6d6e268-f93f-57ca-a958-e9f5538b90d8}" next=1 2026-05-25T03:41:18.113Z [INFO] server-ff57f0c0-a106-b7cd-e1d3-d05e7aa12a7a: pipelining replication: peer="{Voter server-9d07d493-a474-67b5-c9e4-d22897dd6ae1 9d07d493-a474-67b5-c9e4-d22897dd6ae1}" 2026-05-25T03:41:18.113Z [INFO] server-ff57f0c0-a106-b7cd-e1d3-d05e7aa12a7a: pipelining replication: peer="{Voter server-d6d6e268-f93f-57ca-a958-e9f5538b90d8 d6d6e268-f93f-57ca-a958-e9f5538b90d8}" 2026-05-25T03:41:18.218Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:18.113136677 +0000 UTC m=+4.286112121 (2 nodes), 54.66892ms from start of poll, 55.2596ms from cluster start. Timeout at 2026-05-25 03:41:18.218139637 +0000 UTC m=+4.391115081, 105.00296ms after stability 2026-05-25T03:41:18.218Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:18.323Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:18.218374917 +0000 UTC m=+4.391350361 (1 nodes), 1µs from start of poll, 160.49784ms from cluster start. Timeout at 2026-05-25 03:41:18.323382677 +0000 UTC m=+4.496358121, 105.00776ms after stability 2026-05-25T03:41:18.335Z [INFO] server-ff57f0c0-a106-b7cd-e1d3-d05e7aa12a7a: aborting pipeline replication: peer="{Voter server-9d07d493-a474-67b5-c9e4-d22897dd6ae1 9d07d493-a474-67b5-c9e4-d22897dd6ae1}" 2026-05-25T03:41:18.335Z [INFO] server-ff57f0c0-a106-b7cd-e1d3-d05e7aa12a7a: aborting pipeline replication: peer="{Voter server-d6d6e268-f93f-57ca-a958-e9f5538b90d8 d6d6e268-f93f-57ca-a958-e9f5538b90d8}" --- PASS: TestRaft_LiveBootstrap (0.28s) === RUN TestRaft_LiveBootstrap_From_NonVoter 2026-05-25T03:41:18.335Z [INFO] server-9b08f29b-daa4-270d-051b-4b90a82ad844: initial configuration: index=0 servers=[] 2026-05-25T03:41:18.335Z [INFO] server-e8adb071-d43f-d8fa-a510-ddf7179f05cd: initial configuration: index=0 servers=[] 2026-05-25T03:41:18.336Z [INFO] server-9b08f29b-daa4-270d-051b-4b90a82ad844: entering follower state: follower="Node at 9b08f29b-daa4-270d-051b-4b90a82ad844 [Follower]" leader-address= leader-id= 2026-05-25T03:41:18.336Z [INFO] server-e8adb071-d43f-d8fa-a510-ddf7179f05cd: entering follower state: follower="Node at e8adb071-d43f-d8fa-a510-ddf7179f05cd [Follower]" leader-address= leader-id= --- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s) === RUN TestRaft_RecoverCluster_NoState 2026-05-25T03:41:18.336Z [INFO] server-d03528a4-4876-5778-6540-6205a1dbc3c0: initial configuration: index=0 servers=[] 2026-05-25T03:41:18.336Z [INFO] server-d03528a4-4876-5778-6540-6205a1dbc3c0: entering follower state: follower="Node at d03528a4-4876-5778-6540-6205a1dbc3c0 [Shutdown]" leader-address= leader-id= --- PASS: TestRaft_RecoverCluster_NoState (0.00s) === RUN TestRaft_RecoverCluster === RUN TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs 2026-05-25T03:41:18.337Z [INFO] server-f17402f3-6f10-78a5-6eb0-d1abc8962896: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f17402f3-6f10-78a5-6eb0-d1abc8962896 Address:f17402f3-6f10-78a5-6eb0-d1abc8962896} {Suffrage:Voter ID:server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816 Address:a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816} {Suffrage:Voter ID:server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a Address:52e7324b-7eb9-5d77-2122-f66ce9f75d0a}]" 2026-05-25T03:41:18.338Z [INFO] server-f17402f3-6f10-78a5-6eb0-d1abc8962896: entering follower state: follower="Node at f17402f3-6f10-78a5-6eb0-d1abc8962896 [Follower]" leader-address= leader-id= 2026-05-25T03:41:18.338Z [INFO] server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f17402f3-6f10-78a5-6eb0-d1abc8962896 Address:f17402f3-6f10-78a5-6eb0-d1abc8962896} {Suffrage:Voter ID:server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816 Address:a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816} {Suffrage:Voter ID:server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a Address:52e7324b-7eb9-5d77-2122-f66ce9f75d0a}]" 2026-05-25T03:41:18.338Z [INFO] server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816: entering follower state: follower="Node at a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816 [Follower]" leader-address= leader-id= 2026-05-25T03:41:18.339Z [INFO] server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f17402f3-6f10-78a5-6eb0-d1abc8962896 Address:f17402f3-6f10-78a5-6eb0-d1abc8962896} {Suffrage:Voter ID:server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816 Address:a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816} {Suffrage:Voter ID:server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a Address:52e7324b-7eb9-5d77-2122-f66ce9f75d0a}]" 2026-05-25T03:41:18.339Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:18.339Z [INFO] server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a: entering follower state: follower="Node at 52e7324b-7eb9-5d77-2122-f66ce9f75d0a [Follower]" leader-address= leader-id= 2026-05-25T03:41:18.390Z [WARN] server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:18.390Z [INFO] server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816: entering candidate state: node="Node at a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816 [Candidate]" term=2 2026-05-25T03:41:18.390Z [INFO] server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816: election won: term=2 tally=2 2026-05-25T03:41:18.390Z [INFO] server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816: entering leader state: leader="Node at a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816 [Leader]" 2026-05-25T03:41:18.390Z [INFO] server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816: added peer, starting replication: peer=server-f17402f3-6f10-78a5-6eb0-d1abc8962896 2026-05-25T03:41:18.390Z [INFO] server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816: added peer, starting replication: peer=server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a 2026-05-25T03:41:18.390Z [INFO] server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816: pipelining replication: peer="{Voter server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a 52e7324b-7eb9-5d77-2122-f66ce9f75d0a}" 2026-05-25T03:41:18.390Z [INFO] server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816: pipelining replication: peer="{Voter server-f17402f3-6f10-78a5-6eb0-d1abc8962896 f17402f3-6f10-78a5-6eb0-d1abc8962896}" 2026-05-25T03:41:18.495Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:18.390749597 +0000 UTC m=+4.563725041 (1 nodes), 51.66908ms from start of poll, 53.34228ms from cluster start. Timeout at 2026-05-25 03:41:18.495758797 +0000 UTC m=+4.668734241, 105.0092ms after stability 2026-05-25T03:41:18.496Z [INFO] server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816: aborting pipeline replication: peer="{Voter server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a 52e7324b-7eb9-5d77-2122-f66ce9f75d0a}" 2026-05-25T03:41:18.496Z [INFO] server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816: aborting pipeline replication: peer="{Voter server-f17402f3-6f10-78a5-6eb0-d1abc8962896 f17402f3-6f10-78a5-6eb0-d1abc8962896}" 2026-05-25T03:41:18.496Z [INFO] creating new snapshot: path=/tmp/raft1303794040/snapshots/2-2-1779680478496.tmp 2026-05-25T03:41:18.497Z [INFO] server-f17402f3-6f10-78a5-6eb0-d1abc8962896: starting restore from snapshot: id=2-2-1779680478496 last-index=2 last-term=2 size-in-bytes=1 2026-05-25T03:41:18.497Z [INFO] server-f17402f3-6f10-78a5-6eb0-d1abc8962896: snapshot restore progress: id=2-2-1779680478496 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2026-05-25T03:41:18.497Z [INFO] server-f17402f3-6f10-78a5-6eb0-d1abc8962896: restored from snapshot: id=2-2-1779680478496 last-index=2 last-term=2 size-in-bytes=1 2026-05-25T03:41:18.497Z [INFO] server-f17402f3-6f10-78a5-6eb0-d1abc8962896: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f17402f3-6f10-78a5-6eb0-d1abc8962896 Address:f17402f3-6f10-78a5-6eb0-d1abc8962896} {Suffrage:Voter ID:server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816 Address:a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816} {Suffrage:Voter ID:server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a Address:52e7324b-7eb9-5d77-2122-f66ce9f75d0a}]" 2026-05-25T03:41:18.497Z [INFO] creating new snapshot: path=/tmp/raft1585533946/snapshots/2-2-1779680478497.tmp 2026-05-25T03:41:18.497Z [INFO] server-f17402f3-6f10-78a5-6eb0-d1abc8962896: entering follower state: follower="Node at f17402f3-6f10-78a5-6eb0-d1abc8962896 [Follower]" leader-address= leader-id= 2026-05-25T03:41:18.497Z [INFO] server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816: starting restore from snapshot: id=2-2-1779680478497 last-index=2 last-term=2 size-in-bytes=1 2026-05-25T03:41:18.498Z [INFO] server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816: snapshot restore progress: id=2-2-1779680478497 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2026-05-25T03:41:18.498Z [INFO] server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816: restored from snapshot: id=2-2-1779680478497 last-index=2 last-term=2 size-in-bytes=1 2026-05-25T03:41:18.498Z [INFO] server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f17402f3-6f10-78a5-6eb0-d1abc8962896 Address:f17402f3-6f10-78a5-6eb0-d1abc8962896} {Suffrage:Voter ID:server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816 Address:a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816} {Suffrage:Voter ID:server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a Address:52e7324b-7eb9-5d77-2122-f66ce9f75d0a}]" 2026-05-25T03:41:18.498Z [INFO] creating new snapshot: path=/tmp/raft2322815597/snapshots/2-2-1779680478498.tmp 2026-05-25T03:41:18.498Z [INFO] server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816: entering follower state: follower="Node at a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816 [Follower]" leader-address= leader-id= 2026-05-25T03:41:18.498Z [INFO] server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a: starting restore from snapshot: id=2-2-1779680478498 last-index=2 last-term=2 size-in-bytes=1 2026-05-25T03:41:18.498Z [INFO] server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a: snapshot restore progress: id=2-2-1779680478498 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00% 2026-05-25T03:41:18.498Z [INFO] server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a: restored from snapshot: id=2-2-1779680478498 last-index=2 last-term=2 size-in-bytes=1 2026-05-25T03:41:18.498Z [INFO] server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f17402f3-6f10-78a5-6eb0-d1abc8962896 Address:f17402f3-6f10-78a5-6eb0-d1abc8962896} {Suffrage:Voter ID:server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816 Address:a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816} {Suffrage:Voter ID:server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a Address:52e7324b-7eb9-5d77-2122-f66ce9f75d0a}]" 2026-05-25T03:41:18.498Z [INFO] server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a: entering follower state: follower="Node at 52e7324b-7eb9-5d77-2122-f66ce9f75d0a [Follower]" leader-address= leader-id= 2026-05-25T03:41:18.555Z [WARN] server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:18.555Z [INFO] server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a: entering candidate state: node="Node at 52e7324b-7eb9-5d77-2122-f66ce9f75d0a [Candidate]" term=3 2026-05-25T03:41:18.555Z [INFO] server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a: election won: term=3 tally=2 2026-05-25T03:41:18.555Z [INFO] server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a: entering leader state: leader="Node at 52e7324b-7eb9-5d77-2122-f66ce9f75d0a [Leader]" 2026-05-25T03:41:18.555Z [INFO] server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a: added peer, starting replication: peer=server-f17402f3-6f10-78a5-6eb0-d1abc8962896 2026-05-25T03:41:18.555Z [INFO] server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a: added peer, starting replication: peer=server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816 2026-05-25T03:41:18.555Z [INFO] server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a: pipelining replication: peer="{Voter server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816 a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816}" 2026-05-25T03:41:18.555Z [INFO] server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a: pipelining replication: peer="{Voter server-f17402f3-6f10-78a5-6eb0-d1abc8962896 f17402f3-6f10-78a5-6eb0-d1abc8962896}" 2026-05-25T03:41:18.814Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:18.920Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:18.814894597 +0000 UTC m=+4.987870001 (1 nodes), 43.16µs from start of poll, 477.48724ms from cluster start. Timeout at 2026-05-25 03:41:18.919909157 +0000 UTC m=+5.092884601, 105.0146ms after stability 2026-05-25T03:41:18.920Z [INFO] server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a: aborting pipeline replication: peer="{Voter server-a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816 a87c3ec5-1ded-5ff0-cb8e-e3d9bdef9816}" 2026-05-25T03:41:18.920Z [INFO] server-52e7324b-7eb9-5d77-2122-f66ce9f75d0a: aborting pipeline replication: peer="{Voter server-f17402f3-6f10-78a5-6eb0-d1abc8962896 f17402f3-6f10-78a5-6eb0-d1abc8962896}" === RUN TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs 2026-05-25T03:41:18.921Z [INFO] server-880132cd-6e1e-02c8-5345-d03b38ad6b0b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-880132cd-6e1e-02c8-5345-d03b38ad6b0b Address:880132cd-6e1e-02c8-5345-d03b38ad6b0b} {Suffrage:Voter ID:server-250199eb-f1a9-3f20-00ed-95926ef02f85 Address:250199eb-f1a9-3f20-00ed-95926ef02f85} {Suffrage:Voter ID:server-3c2539dd-2d18-ddc9-4693-65f78cd4b064 Address:3c2539dd-2d18-ddc9-4693-65f78cd4b064}]" 2026-05-25T03:41:18.921Z [INFO] server-250199eb-f1a9-3f20-00ed-95926ef02f85: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-880132cd-6e1e-02c8-5345-d03b38ad6b0b Address:880132cd-6e1e-02c8-5345-d03b38ad6b0b} {Suffrage:Voter ID:server-250199eb-f1a9-3f20-00ed-95926ef02f85 Address:250199eb-f1a9-3f20-00ed-95926ef02f85} {Suffrage:Voter ID:server-3c2539dd-2d18-ddc9-4693-65f78cd4b064 Address:3c2539dd-2d18-ddc9-4693-65f78cd4b064}]" 2026-05-25T03:41:18.921Z [INFO] server-880132cd-6e1e-02c8-5345-d03b38ad6b0b: entering follower state: follower="Node at 880132cd-6e1e-02c8-5345-d03b38ad6b0b [Follower]" leader-address= leader-id= 2026-05-25T03:41:18.921Z [INFO] server-3c2539dd-2d18-ddc9-4693-65f78cd4b064: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-880132cd-6e1e-02c8-5345-d03b38ad6b0b Address:880132cd-6e1e-02c8-5345-d03b38ad6b0b} {Suffrage:Voter ID:server-250199eb-f1a9-3f20-00ed-95926ef02f85 Address:250199eb-f1a9-3f20-00ed-95926ef02f85} {Suffrage:Voter ID:server-3c2539dd-2d18-ddc9-4693-65f78cd4b064 Address:3c2539dd-2d18-ddc9-4693-65f78cd4b064}]" 2026-05-25T03:41:18.921Z [INFO] server-250199eb-f1a9-3f20-00ed-95926ef02f85: entering follower state: follower="Node at 250199eb-f1a9-3f20-00ed-95926ef02f85 [Follower]" leader-address= leader-id= 2026-05-25T03:41:18.922Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:18.922Z [INFO] server-3c2539dd-2d18-ddc9-4693-65f78cd4b064: entering follower state: follower="Node at 3c2539dd-2d18-ddc9-4693-65f78cd4b064 [Follower]" leader-address= leader-id= 2026-05-25T03:41:18.990Z [WARN] server-250199eb-f1a9-3f20-00ed-95926ef02f85: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:18.990Z [INFO] server-250199eb-f1a9-3f20-00ed-95926ef02f85: entering candidate state: node="Node at 250199eb-f1a9-3f20-00ed-95926ef02f85 [Candidate]" term=2 2026-05-25T03:41:18.990Z [INFO] server-250199eb-f1a9-3f20-00ed-95926ef02f85: election won: term=2 tally=2 2026-05-25T03:41:18.990Z [INFO] server-250199eb-f1a9-3f20-00ed-95926ef02f85: entering leader state: leader="Node at 250199eb-f1a9-3f20-00ed-95926ef02f85 [Leader]" 2026-05-25T03:41:18.990Z [INFO] server-250199eb-f1a9-3f20-00ed-95926ef02f85: added peer, starting replication: peer=server-880132cd-6e1e-02c8-5345-d03b38ad6b0b 2026-05-25T03:41:18.990Z [INFO] server-250199eb-f1a9-3f20-00ed-95926ef02f85: added peer, starting replication: peer=server-3c2539dd-2d18-ddc9-4693-65f78cd4b064 2026-05-25T03:41:18.990Z [INFO] server-250199eb-f1a9-3f20-00ed-95926ef02f85: pipelining replication: peer="{Voter server-3c2539dd-2d18-ddc9-4693-65f78cd4b064 3c2539dd-2d18-ddc9-4693-65f78cd4b064}" 2026-05-25T03:41:18.990Z [INFO] server-250199eb-f1a9-3f20-00ed-95926ef02f85: pipelining replication: peer="{Voter server-880132cd-6e1e-02c8-5345-d03b38ad6b0b 880132cd-6e1e-02c8-5345-d03b38ad6b0b}" 2026-05-25T03:41:19.096Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:18.990620477 +0000 UTC m=+5.163595921 (1 nodes), 68.5928ms from start of poll, 69.04252ms from cluster start. Timeout at 2026-05-25 03:41:19.095623757 +0000 UTC m=+5.268599201, 105.00328ms after stability 2026-05-25T03:41:19.097Z [INFO] server-250199eb-f1a9-3f20-00ed-95926ef02f85: aborting pipeline replication: peer="{Voter server-3c2539dd-2d18-ddc9-4693-65f78cd4b064 3c2539dd-2d18-ddc9-4693-65f78cd4b064}" 2026-05-25T03:41:19.097Z [INFO] server-250199eb-f1a9-3f20-00ed-95926ef02f85: aborting pipeline replication: peer="{Voter server-880132cd-6e1e-02c8-5345-d03b38ad6b0b 880132cd-6e1e-02c8-5345-d03b38ad6b0b}" 2026-05-25T03:41:19.097Z [INFO] creating new snapshot: path=/tmp/raft911102400/snapshots/2-6-1779680479097.tmp 2026-05-25T03:41:19.098Z [INFO] server-880132cd-6e1e-02c8-5345-d03b38ad6b0b: starting restore from snapshot: id=2-6-1779680479097 last-index=6 last-term=2 size-in-bytes=25 2026-05-25T03:41:19.098Z [INFO] server-880132cd-6e1e-02c8-5345-d03b38ad6b0b: snapshot restore progress: id=2-6-1779680479097 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2026-05-25T03:41:19.098Z [INFO] server-880132cd-6e1e-02c8-5345-d03b38ad6b0b: restored from snapshot: id=2-6-1779680479097 last-index=6 last-term=2 size-in-bytes=25 2026-05-25T03:41:19.098Z [INFO] server-880132cd-6e1e-02c8-5345-d03b38ad6b0b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-880132cd-6e1e-02c8-5345-d03b38ad6b0b Address:880132cd-6e1e-02c8-5345-d03b38ad6b0b} {Suffrage:Voter ID:server-250199eb-f1a9-3f20-00ed-95926ef02f85 Address:250199eb-f1a9-3f20-00ed-95926ef02f85} {Suffrage:Voter ID:server-3c2539dd-2d18-ddc9-4693-65f78cd4b064 Address:3c2539dd-2d18-ddc9-4693-65f78cd4b064}]" 2026-05-25T03:41:19.098Z [INFO] creating new snapshot: path=/tmp/raft2431424827/snapshots/2-6-1779680479098.tmp 2026-05-25T03:41:19.098Z [INFO] server-880132cd-6e1e-02c8-5345-d03b38ad6b0b: entering follower state: follower="Node at 880132cd-6e1e-02c8-5345-d03b38ad6b0b [Follower]" leader-address= leader-id= 2026-05-25T03:41:19.099Z [INFO] server-250199eb-f1a9-3f20-00ed-95926ef02f85: starting restore from snapshot: id=2-6-1779680479098 last-index=6 last-term=2 size-in-bytes=25 2026-05-25T03:41:19.099Z [INFO] server-250199eb-f1a9-3f20-00ed-95926ef02f85: snapshot restore progress: id=2-6-1779680479098 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2026-05-25T03:41:19.099Z [INFO] server-250199eb-f1a9-3f20-00ed-95926ef02f85: restored from snapshot: id=2-6-1779680479098 last-index=6 last-term=2 size-in-bytes=25 2026-05-25T03:41:19.099Z [INFO] server-250199eb-f1a9-3f20-00ed-95926ef02f85: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-880132cd-6e1e-02c8-5345-d03b38ad6b0b Address:880132cd-6e1e-02c8-5345-d03b38ad6b0b} {Suffrage:Voter ID:server-250199eb-f1a9-3f20-00ed-95926ef02f85 Address:250199eb-f1a9-3f20-00ed-95926ef02f85} {Suffrage:Voter ID:server-3c2539dd-2d18-ddc9-4693-65f78cd4b064 Address:3c2539dd-2d18-ddc9-4693-65f78cd4b064}]" 2026-05-25T03:41:19.099Z [INFO] creating new snapshot: path=/tmp/raft2839002496/snapshots/2-6-1779680479099.tmp 2026-05-25T03:41:19.099Z [INFO] server-250199eb-f1a9-3f20-00ed-95926ef02f85: entering follower state: follower="Node at 250199eb-f1a9-3f20-00ed-95926ef02f85 [Follower]" leader-address= leader-id= 2026-05-25T03:41:19.100Z [INFO] server-3c2539dd-2d18-ddc9-4693-65f78cd4b064: starting restore from snapshot: id=2-6-1779680479099 last-index=6 last-term=2 size-in-bytes=25 2026-05-25T03:41:19.100Z [INFO] server-3c2539dd-2d18-ddc9-4693-65f78cd4b064: snapshot restore progress: id=2-6-1779680479099 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00% 2026-05-25T03:41:19.100Z [INFO] server-3c2539dd-2d18-ddc9-4693-65f78cd4b064: restored from snapshot: id=2-6-1779680479099 last-index=6 last-term=2 size-in-bytes=25 2026-05-25T03:41:19.100Z [INFO] server-3c2539dd-2d18-ddc9-4693-65f78cd4b064: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-880132cd-6e1e-02c8-5345-d03b38ad6b0b Address:880132cd-6e1e-02c8-5345-d03b38ad6b0b} {Suffrage:Voter ID:server-250199eb-f1a9-3f20-00ed-95926ef02f85 Address:250199eb-f1a9-3f20-00ed-95926ef02f85} {Suffrage:Voter ID:server-3c2539dd-2d18-ddc9-4693-65f78cd4b064 Address:3c2539dd-2d18-ddc9-4693-65f78cd4b064}]" 2026-05-25T03:41:19.100Z [INFO] server-3c2539dd-2d18-ddc9-4693-65f78cd4b064: entering follower state: follower="Node at 3c2539dd-2d18-ddc9-4693-65f78cd4b064 [Follower]" leader-address= leader-id= 2026-05-25T03:41:19.154Z [WARN] server-3c2539dd-2d18-ddc9-4693-65f78cd4b064: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:19.154Z [INFO] server-3c2539dd-2d18-ddc9-4693-65f78cd4b064: entering candidate state: node="Node at 3c2539dd-2d18-ddc9-4693-65f78cd4b064 [Candidate]" term=3 2026-05-25T03:41:19.154Z [INFO] server-3c2539dd-2d18-ddc9-4693-65f78cd4b064: election won: term=3 tally=2 2026-05-25T03:41:19.154Z [INFO] server-3c2539dd-2d18-ddc9-4693-65f78cd4b064: entering leader state: leader="Node at 3c2539dd-2d18-ddc9-4693-65f78cd4b064 [Leader]" 2026-05-25T03:41:19.154Z [INFO] server-3c2539dd-2d18-ddc9-4693-65f78cd4b064: added peer, starting replication: peer=server-880132cd-6e1e-02c8-5345-d03b38ad6b0b 2026-05-25T03:41:19.154Z [INFO] server-3c2539dd-2d18-ddc9-4693-65f78cd4b064: added peer, starting replication: peer=server-250199eb-f1a9-3f20-00ed-95926ef02f85 2026-05-25T03:41:19.154Z [INFO] server-3c2539dd-2d18-ddc9-4693-65f78cd4b064: pipelining replication: peer="{Voter server-880132cd-6e1e-02c8-5345-d03b38ad6b0b 880132cd-6e1e-02c8-5345-d03b38ad6b0b}" 2026-05-25T03:41:19.155Z [INFO] server-3c2539dd-2d18-ddc9-4693-65f78cd4b064: pipelining replication: peer="{Voter server-250199eb-f1a9-3f20-00ed-95926ef02f85 250199eb-f1a9-3f20-00ed-95926ef02f85}" 2026-05-25T03:41:19.416Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:19.522Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:19.416534357 +0000 UTC m=+5.589509801 (1 nodes), 39.48µs from start of poll, 494.9564ms from cluster start. Timeout at 2026-05-25 03:41:19.521540717 +0000 UTC m=+5.694516161, 105.00636ms after stability 2026-05-25T03:41:19.522Z [INFO] server-3c2539dd-2d18-ddc9-4693-65f78cd4b064: aborting pipeline replication: peer="{Voter server-880132cd-6e1e-02c8-5345-d03b38ad6b0b 880132cd-6e1e-02c8-5345-d03b38ad6b0b}" 2026-05-25T03:41:19.522Z [INFO] server-3c2539dd-2d18-ddc9-4693-65f78cd4b064: aborting pipeline replication: peer="{Voter server-250199eb-f1a9-3f20-00ed-95926ef02f85 250199eb-f1a9-3f20-00ed-95926ef02f85}" === RUN TestRaft_RecoverCluster/snapshot,_with_trailing_logs 2026-05-25T03:41:19.523Z [INFO] server-2796fcaf-cf28-8714-c798-6f57db447355: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2796fcaf-cf28-8714-c798-6f57db447355 Address:2796fcaf-cf28-8714-c798-6f57db447355} {Suffrage:Voter ID:server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019 Address:0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019} {Suffrage:Voter ID:server-ca21b63e-e57c-ccfe-75ed-857d1f749534 Address:ca21b63e-e57c-ccfe-75ed-857d1f749534}]" 2026-05-25T03:41:19.523Z [INFO] server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2796fcaf-cf28-8714-c798-6f57db447355 Address:2796fcaf-cf28-8714-c798-6f57db447355} {Suffrage:Voter ID:server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019 Address:0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019} {Suffrage:Voter ID:server-ca21b63e-e57c-ccfe-75ed-857d1f749534 Address:ca21b63e-e57c-ccfe-75ed-857d1f749534}]" 2026-05-25T03:41:19.523Z [INFO] server-2796fcaf-cf28-8714-c798-6f57db447355: entering follower state: follower="Node at 2796fcaf-cf28-8714-c798-6f57db447355 [Follower]" leader-address= leader-id= 2026-05-25T03:41:19.523Z [INFO] server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019: entering follower state: follower="Node at 0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019 [Follower]" leader-address= leader-id= 2026-05-25T03:41:19.523Z [INFO] server-ca21b63e-e57c-ccfe-75ed-857d1f749534: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2796fcaf-cf28-8714-c798-6f57db447355 Address:2796fcaf-cf28-8714-c798-6f57db447355} {Suffrage:Voter ID:server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019 Address:0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019} {Suffrage:Voter ID:server-ca21b63e-e57c-ccfe-75ed-857d1f749534 Address:ca21b63e-e57c-ccfe-75ed-857d1f749534}]" 2026-05-25T03:41:19.523Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:19.523Z [INFO] server-ca21b63e-e57c-ccfe-75ed-857d1f749534: entering follower state: follower="Node at ca21b63e-e57c-ccfe-75ed-857d1f749534 [Follower]" leader-address= leader-id= 2026-05-25T03:41:19.617Z [WARN] server-2796fcaf-cf28-8714-c798-6f57db447355: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:19.617Z [INFO] server-2796fcaf-cf28-8714-c798-6f57db447355: entering candidate state: node="Node at 2796fcaf-cf28-8714-c798-6f57db447355 [Candidate]" term=2 2026-05-25T03:41:19.618Z [INFO] server-2796fcaf-cf28-8714-c798-6f57db447355: election won: term=2 tally=2 2026-05-25T03:41:19.618Z [INFO] server-2796fcaf-cf28-8714-c798-6f57db447355: entering leader state: leader="Node at 2796fcaf-cf28-8714-c798-6f57db447355 [Leader]" 2026-05-25T03:41:19.618Z [INFO] server-2796fcaf-cf28-8714-c798-6f57db447355: added peer, starting replication: peer=server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019 2026-05-25T03:41:19.618Z [INFO] server-2796fcaf-cf28-8714-c798-6f57db447355: added peer, starting replication: peer=server-ca21b63e-e57c-ccfe-75ed-857d1f749534 2026-05-25T03:41:19.618Z [INFO] server-2796fcaf-cf28-8714-c798-6f57db447355: pipelining replication: peer="{Voter server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019 0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019}" 2026-05-25T03:41:19.618Z [INFO] server-2796fcaf-cf28-8714-c798-6f57db447355: pipelining replication: peer="{Voter server-ca21b63e-e57c-ccfe-75ed-857d1f749534 ca21b63e-e57c-ccfe-75ed-857d1f749534}" 2026-05-25T03:41:19.723Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:19.618077477 +0000 UTC m=+5.791052921 (1 nodes), 94.23876ms from start of poll, 94.7026ms from cluster start. Timeout at 2026-05-25 03:41:19.723081037 +0000 UTC m=+5.896056481, 105.00356ms after stability 2026-05-25T03:41:19.727Z [INFO] server-2796fcaf-cf28-8714-c798-6f57db447355: aborting pipeline replication: peer="{Voter server-ca21b63e-e57c-ccfe-75ed-857d1f749534 ca21b63e-e57c-ccfe-75ed-857d1f749534}" 2026-05-25T03:41:19.727Z [INFO] server-2796fcaf-cf28-8714-c798-6f57db447355: aborting pipeline replication: peer="{Voter server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019 0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019}" 2026-05-25T03:41:19.727Z [INFO] creating new snapshot: path=/tmp/raft2045367463/snapshots/2-27-1779680479727.tmp 2026-05-25T03:41:19.728Z [INFO] server-2796fcaf-cf28-8714-c798-6f57db447355: starting restore from snapshot: id=2-27-1779680479727 last-index=27 last-term=2 size-in-bytes=168 2026-05-25T03:41:19.728Z [INFO] server-2796fcaf-cf28-8714-c798-6f57db447355: snapshot restore progress: id=2-27-1779680479727 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2026-05-25T03:41:19.728Z [INFO] server-2796fcaf-cf28-8714-c798-6f57db447355: restored from snapshot: id=2-27-1779680479727 last-index=27 last-term=2 size-in-bytes=168 2026-05-25T03:41:19.728Z [INFO] server-2796fcaf-cf28-8714-c798-6f57db447355: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2796fcaf-cf28-8714-c798-6f57db447355 Address:2796fcaf-cf28-8714-c798-6f57db447355} {Suffrage:Voter ID:server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019 Address:0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019} {Suffrage:Voter ID:server-ca21b63e-e57c-ccfe-75ed-857d1f749534 Address:ca21b63e-e57c-ccfe-75ed-857d1f749534}]" 2026-05-25T03:41:19.728Z [INFO] creating new snapshot: path=/tmp/raft3237526330/snapshots/2-27-1779680479728.tmp 2026-05-25T03:41:19.728Z [INFO] server-2796fcaf-cf28-8714-c798-6f57db447355: entering follower state: follower="Node at 2796fcaf-cf28-8714-c798-6f57db447355 [Follower]" leader-address= leader-id= 2026-05-25T03:41:19.729Z [INFO] server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019: starting restore from snapshot: id=2-27-1779680479728 last-index=27 last-term=2 size-in-bytes=168 2026-05-25T03:41:19.729Z [INFO] server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019: snapshot restore progress: id=2-27-1779680479728 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2026-05-25T03:41:19.729Z [INFO] server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019: restored from snapshot: id=2-27-1779680479728 last-index=27 last-term=2 size-in-bytes=168 2026-05-25T03:41:19.729Z [INFO] server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2796fcaf-cf28-8714-c798-6f57db447355 Address:2796fcaf-cf28-8714-c798-6f57db447355} {Suffrage:Voter ID:server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019 Address:0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019} {Suffrage:Voter ID:server-ca21b63e-e57c-ccfe-75ed-857d1f749534 Address:ca21b63e-e57c-ccfe-75ed-857d1f749534}]" 2026-05-25T03:41:19.729Z [INFO] creating new snapshot: path=/tmp/raft1160110235/snapshots/2-27-1779680479729.tmp 2026-05-25T03:41:19.729Z [INFO] server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019: entering follower state: follower="Node at 0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019 [Follower]" leader-address= leader-id= 2026-05-25T03:41:19.729Z [INFO] server-ca21b63e-e57c-ccfe-75ed-857d1f749534: starting restore from snapshot: id=2-27-1779680479729 last-index=27 last-term=2 size-in-bytes=168 2026-05-25T03:41:19.730Z [INFO] server-ca21b63e-e57c-ccfe-75ed-857d1f749534: snapshot restore progress: id=2-27-1779680479729 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00% 2026-05-25T03:41:19.730Z [INFO] server-ca21b63e-e57c-ccfe-75ed-857d1f749534: restored from snapshot: id=2-27-1779680479729 last-index=27 last-term=2 size-in-bytes=168 2026-05-25T03:41:19.730Z [INFO] server-ca21b63e-e57c-ccfe-75ed-857d1f749534: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2796fcaf-cf28-8714-c798-6f57db447355 Address:2796fcaf-cf28-8714-c798-6f57db447355} {Suffrage:Voter ID:server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019 Address:0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019} {Suffrage:Voter ID:server-ca21b63e-e57c-ccfe-75ed-857d1f749534 Address:ca21b63e-e57c-ccfe-75ed-857d1f749534}]" 2026-05-25T03:41:19.730Z [INFO] server-ca21b63e-e57c-ccfe-75ed-857d1f749534: entering follower state: follower="Node at ca21b63e-e57c-ccfe-75ed-857d1f749534 [Follower]" leader-address= leader-id= 2026-05-25T03:41:19.787Z [WARN] server-ca21b63e-e57c-ccfe-75ed-857d1f749534: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:19.787Z [INFO] server-ca21b63e-e57c-ccfe-75ed-857d1f749534: entering candidate state: node="Node at ca21b63e-e57c-ccfe-75ed-857d1f749534 [Candidate]" term=3 2026-05-25T03:41:19.787Z [INFO] server-ca21b63e-e57c-ccfe-75ed-857d1f749534: election won: term=3 tally=2 2026-05-25T03:41:19.787Z [INFO] server-ca21b63e-e57c-ccfe-75ed-857d1f749534: entering leader state: leader="Node at ca21b63e-e57c-ccfe-75ed-857d1f749534 [Leader]" 2026-05-25T03:41:19.787Z [INFO] server-ca21b63e-e57c-ccfe-75ed-857d1f749534: added peer, starting replication: peer=server-2796fcaf-cf28-8714-c798-6f57db447355 2026-05-25T03:41:19.787Z [INFO] server-ca21b63e-e57c-ccfe-75ed-857d1f749534: added peer, starting replication: peer=server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019 2026-05-25T03:41:19.787Z [INFO] server-ca21b63e-e57c-ccfe-75ed-857d1f749534: pipelining replication: peer="{Voter server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019 0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019}" 2026-05-25T03:41:19.787Z [INFO] server-ca21b63e-e57c-ccfe-75ed-857d1f749534: pipelining replication: peer="{Voter server-2796fcaf-cf28-8714-c798-6f57db447355 2796fcaf-cf28-8714-c798-6f57db447355}" 2026-05-25T03:41:20.045Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:20.151Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:20.046006517 +0000 UTC m=+6.218981961 (1 nodes), 45.52µs from start of poll, 522.63164ms from cluster start. Timeout at 2026-05-25 03:41:20.151010957 +0000 UTC m=+6.323986401, 105.00444ms after stability 2026-05-25T03:41:20.151Z [INFO] server-ca21b63e-e57c-ccfe-75ed-857d1f749534: aborting pipeline replication: peer="{Voter server-0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019 0aeae8d2-c9eb-6af7-f348-6f1bc9a9c019}" 2026-05-25T03:41:20.151Z [INFO] server-ca21b63e-e57c-ccfe-75ed-857d1f749534: aborting pipeline replication: peer="{Voter server-2796fcaf-cf28-8714-c798-6f57db447355 2796fcaf-cf28-8714-c798-6f57db447355}" --- PASS: TestRaft_RecoverCluster (1.82s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.58s) --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.60s) --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.63s) === RUN TestRaft_HasExistingState raft_test.go:258: Skipping unreliable test --- SKIP: TestRaft_HasExistingState (0.00s) === RUN TestRaft_SingleNode 2026-05-25T03:41:20.153Z [INFO] server-184a3fae-bf5c-457c-669b-a52c6e472b86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-184a3fae-bf5c-457c-669b-a52c6e472b86 Address:184a3fae-bf5c-457c-669b-a52c6e472b86}]" 2026-05-25T03:41:20.153Z [INFO] server-184a3fae-bf5c-457c-669b-a52c6e472b86: entering follower state: follower="Node at 184a3fae-bf5c-457c-669b-a52c6e472b86 [Follower]" leader-address= leader-id= 2026-05-25T03:41:20.205Z [WARN] server-184a3fae-bf5c-457c-669b-a52c6e472b86: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:20.205Z [INFO] server-184a3fae-bf5c-457c-669b-a52c6e472b86: entering candidate state: node="Node at 184a3fae-bf5c-457c-669b-a52c6e472b86 [Candidate]" term=2 2026-05-25T03:41:20.205Z [INFO] server-184a3fae-bf5c-457c-669b-a52c6e472b86: election won: term=2 tally=1 2026-05-25T03:41:20.205Z [INFO] server-184a3fae-bf5c-457c-669b-a52c6e472b86: entering leader state: leader="Node at 184a3fae-bf5c-457c-669b-a52c6e472b86 [Leader]" --- PASS: TestRaft_SingleNode (0.05s) === RUN TestRaft_TripleNode 2026-05-25T03:41:20.206Z [INFO] server-50aa2058-cc74-37b5-5bc1-da4c0741f30a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50aa2058-cc74-37b5-5bc1-da4c0741f30a Address:50aa2058-cc74-37b5-5bc1-da4c0741f30a} {Suffrage:Voter ID:server-0b305b71-18b5-d307-e210-d5956a2d8914 Address:0b305b71-18b5-d307-e210-d5956a2d8914} {Suffrage:Voter ID:server-d1f57f73-6dd8-789c-6610-f8e12774a554 Address:d1f57f73-6dd8-789c-6610-f8e12774a554}]" 2026-05-25T03:41:20.207Z [INFO] server-0b305b71-18b5-d307-e210-d5956a2d8914: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50aa2058-cc74-37b5-5bc1-da4c0741f30a Address:50aa2058-cc74-37b5-5bc1-da4c0741f30a} {Suffrage:Voter ID:server-0b305b71-18b5-d307-e210-d5956a2d8914 Address:0b305b71-18b5-d307-e210-d5956a2d8914} {Suffrage:Voter ID:server-d1f57f73-6dd8-789c-6610-f8e12774a554 Address:d1f57f73-6dd8-789c-6610-f8e12774a554}]" 2026-05-25T03:41:20.206Z [INFO] server-50aa2058-cc74-37b5-5bc1-da4c0741f30a: entering follower state: follower="Node at 50aa2058-cc74-37b5-5bc1-da4c0741f30a [Follower]" leader-address= leader-id= 2026-05-25T03:41:20.207Z [INFO] server-d1f57f73-6dd8-789c-6610-f8e12774a554: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50aa2058-cc74-37b5-5bc1-da4c0741f30a Address:50aa2058-cc74-37b5-5bc1-da4c0741f30a} {Suffrage:Voter ID:server-0b305b71-18b5-d307-e210-d5956a2d8914 Address:0b305b71-18b5-d307-e210-d5956a2d8914} {Suffrage:Voter ID:server-d1f57f73-6dd8-789c-6610-f8e12774a554 Address:d1f57f73-6dd8-789c-6610-f8e12774a554}]" 2026-05-25T03:41:20.207Z [INFO] server-0b305b71-18b5-d307-e210-d5956a2d8914: entering follower state: follower="Node at 0b305b71-18b5-d307-e210-d5956a2d8914 [Follower]" leader-address= leader-id= 2026-05-25T03:41:20.207Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:20.207Z [INFO] server-d1f57f73-6dd8-789c-6610-f8e12774a554: entering follower state: follower="Node at d1f57f73-6dd8-789c-6610-f8e12774a554 [Follower]" leader-address= leader-id= 2026-05-25T03:41:20.269Z [WARN] server-50aa2058-cc74-37b5-5bc1-da4c0741f30a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:20.269Z [INFO] server-50aa2058-cc74-37b5-5bc1-da4c0741f30a: entering candidate state: node="Node at 50aa2058-cc74-37b5-5bc1-da4c0741f30a [Candidate]" term=2 2026-05-25T03:41:20.269Z [INFO] server-50aa2058-cc74-37b5-5bc1-da4c0741f30a: election won: term=2 tally=2 2026-05-25T03:41:20.269Z [INFO] server-50aa2058-cc74-37b5-5bc1-da4c0741f30a: entering leader state: leader="Node at 50aa2058-cc74-37b5-5bc1-da4c0741f30a [Leader]" 2026-05-25T03:41:20.269Z [INFO] server-50aa2058-cc74-37b5-5bc1-da4c0741f30a: added peer, starting replication: peer=server-0b305b71-18b5-d307-e210-d5956a2d8914 2026-05-25T03:41:20.269Z [INFO] server-50aa2058-cc74-37b5-5bc1-da4c0741f30a: added peer, starting replication: peer=server-d1f57f73-6dd8-789c-6610-f8e12774a554 2026-05-25T03:41:20.269Z [INFO] server-50aa2058-cc74-37b5-5bc1-da4c0741f30a: pipelining replication: peer="{Voter server-d1f57f73-6dd8-789c-6610-f8e12774a554 d1f57f73-6dd8-789c-6610-f8e12774a554}" 2026-05-25T03:41:20.269Z [INFO] server-50aa2058-cc74-37b5-5bc1-da4c0741f30a: pipelining replication: peer="{Voter server-0b305b71-18b5-d307-e210-d5956a2d8914 0b305b71-18b5-d307-e210-d5956a2d8914}" 2026-05-25T03:41:20.375Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:20.269527637 +0000 UTC m=+6.442503081 (2 nodes), 62.37984ms from start of poll, 62.79232ms from cluster start. Timeout at 2026-05-25 03:41:20.374530037 +0000 UTC m=+6.547505481, 105.0024ms after stability 2026-05-25T03:41:20.375Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:20.481Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:20.375259357 +0000 UTC m=+6.548234801 (1 nodes), 880ns from start of poll, 168.52404ms from cluster start. Timeout at 2026-05-25 03:41:20.480265757 +0000 UTC m=+6.653241201, 105.0064ms after stability 2026-05-25T03:41:20.492Z [INFO] server-50aa2058-cc74-37b5-5bc1-da4c0741f30a: aborting pipeline replication: peer="{Voter server-0b305b71-18b5-d307-e210-d5956a2d8914 0b305b71-18b5-d307-e210-d5956a2d8914}" 2026-05-25T03:41:20.492Z [INFO] server-50aa2058-cc74-37b5-5bc1-da4c0741f30a: aborting pipeline replication: peer="{Voter server-d1f57f73-6dd8-789c-6610-f8e12774a554 d1f57f73-6dd8-789c-6610-f8e12774a554}" --- PASS: TestRaft_TripleNode (0.29s) === RUN TestRaft_LeaderFail 2026-05-25T03:41:20.493Z [INFO] server-31a04182-053a-cd2c-ee98-e53b9894e1e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31a04182-053a-cd2c-ee98-e53b9894e1e8 Address:31a04182-053a-cd2c-ee98-e53b9894e1e8} {Suffrage:Voter ID:server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 Address:2c23feeb-c033-c39e-0f7f-d2c29fd87a73} {Suffrage:Voter ID:server-c2b5e523-4828-a171-748f-73d35327a0d5 Address:c2b5e523-4828-a171-748f-73d35327a0d5}]" 2026-05-25T03:41:20.493Z [INFO] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31a04182-053a-cd2c-ee98-e53b9894e1e8 Address:31a04182-053a-cd2c-ee98-e53b9894e1e8} {Suffrage:Voter ID:server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 Address:2c23feeb-c033-c39e-0f7f-d2c29fd87a73} {Suffrage:Voter ID:server-c2b5e523-4828-a171-748f-73d35327a0d5 Address:c2b5e523-4828-a171-748f-73d35327a0d5}]" 2026-05-25T03:41:20.493Z [INFO] server-31a04182-053a-cd2c-ee98-e53b9894e1e8: entering follower state: follower="Node at 31a04182-053a-cd2c-ee98-e53b9894e1e8 [Follower]" leader-address= leader-id= 2026-05-25T03:41:20.493Z [INFO] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: entering follower state: follower="Node at 2c23feeb-c033-c39e-0f7f-d2c29fd87a73 [Follower]" leader-address= leader-id= 2026-05-25T03:41:20.493Z [INFO] server-c2b5e523-4828-a171-748f-73d35327a0d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31a04182-053a-cd2c-ee98-e53b9894e1e8 Address:31a04182-053a-cd2c-ee98-e53b9894e1e8} {Suffrage:Voter ID:server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 Address:2c23feeb-c033-c39e-0f7f-d2c29fd87a73} {Suffrage:Voter ID:server-c2b5e523-4828-a171-748f-73d35327a0d5 Address:c2b5e523-4828-a171-748f-73d35327a0d5}]" 2026-05-25T03:41:20.493Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:20.493Z [INFO] server-c2b5e523-4828-a171-748f-73d35327a0d5: entering follower state: follower="Node at c2b5e523-4828-a171-748f-73d35327a0d5 [Follower]" leader-address= leader-id= 2026-05-25T03:41:20.569Z [WARN] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:20.569Z [INFO] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: entering candidate state: node="Node at 2c23feeb-c033-c39e-0f7f-d2c29fd87a73 [Candidate]" term=2 2026-05-25T03:41:20.569Z [INFO] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: election won: term=2 tally=2 2026-05-25T03:41:20.569Z [INFO] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: entering leader state: leader="Node at 2c23feeb-c033-c39e-0f7f-d2c29fd87a73 [Leader]" 2026-05-25T03:41:20.569Z [INFO] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: added peer, starting replication: peer=server-31a04182-053a-cd2c-ee98-e53b9894e1e8 2026-05-25T03:41:20.569Z [INFO] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: added peer, starting replication: peer=server-c2b5e523-4828-a171-748f-73d35327a0d5 2026-05-25T03:41:20.569Z [INFO] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: pipelining replication: peer="{Voter server-c2b5e523-4828-a171-748f-73d35327a0d5 c2b5e523-4828-a171-748f-73d35327a0d5}" 2026-05-25T03:41:20.569Z [INFO] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: pipelining replication: peer="{Voter server-31a04182-053a-cd2c-ee98-e53b9894e1e8 31a04182-053a-cd2c-ee98-e53b9894e1e8}" 2026-05-25T03:41:20.674Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:20.569388717 +0000 UTC m=+6.742364161 (2 nodes), 75.92884ms from start of poll, 76.4614ms from cluster start. Timeout at 2026-05-25 03:41:20.674393157 +0000 UTC m=+6.847368601, 105.00444ms after stability 2026-05-25T03:41:20.674Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:20.780Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:20.674982837 +0000 UTC m=+6.847958281 (1 nodes), 760ns from start of poll, 182.05552ms from cluster start. Timeout at 2026-05-25 03:41:20.779990917 +0000 UTC m=+6.952966361, 105.00808ms after stability raft_test.go:378: [INFO] Disconnecting Node at 2c23feeb-c033-c39e-0f7f-d2c29fd87a73 [Leader] 2026-05-25T03:41:20.794Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to heartbeat to: peer=31a04182-053a-cd2c-ee98-e53b9894e1e8 backoff time=10ms error="failed to connect to peer: 31a04182-053a-cd2c-ee98-e53b9894e1e8" 2026-05-25T03:41:20.794Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:20.795Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to pipeline appendEntries: peer="{Voter server-c2b5e523-4828-a171-748f-73d35327a0d5 c2b5e523-4828-a171-748f-73d35327a0d5}" error="append pipeline closed" 2026-05-25T03:41:20.795Z [INFO] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: aborting pipeline replication: peer="{Voter server-c2b5e523-4828-a171-748f-73d35327a0d5 c2b5e523-4828-a171-748f-73d35327a0d5}" 2026-05-25T03:41:20.795Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to pipeline appendEntries: peer="{Voter server-31a04182-053a-cd2c-ee98-e53b9894e1e8 31a04182-053a-cd2c-ee98-e53b9894e1e8}" error="append pipeline closed" 2026-05-25T03:41:20.795Z [INFO] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: aborting pipeline replication: peer="{Voter server-31a04182-053a-cd2c-ee98-e53b9894e1e8 31a04182-053a-cd2c-ee98-e53b9894e1e8}" 2026-05-25T03:41:20.797Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to heartbeat to: peer=c2b5e523-4828-a171-748f-73d35327a0d5 backoff time=10ms error="failed to connect to peer: c2b5e523-4828-a171-748f-73d35327a0d5" 2026-05-25T03:41:20.804Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to appendEntries to: peer="{Voter server-31a04182-053a-cd2c-ee98-e53b9894e1e8 31a04182-053a-cd2c-ee98-e53b9894e1e8}" error="failed to connect to peer: 31a04182-053a-cd2c-ee98-e53b9894e1e8" 2026-05-25T03:41:20.805Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to appendEntries to: peer="{Voter server-c2b5e523-4828-a171-748f-73d35327a0d5 c2b5e523-4828-a171-748f-73d35327a0d5}" error="failed to connect to peer: c2b5e523-4828-a171-748f-73d35327a0d5" 2026-05-25T03:41:20.812Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to heartbeat to: peer=31a04182-053a-cd2c-ee98-e53b9894e1e8 backoff time=10ms error="failed to connect to peer: 31a04182-053a-cd2c-ee98-e53b9894e1e8" 2026-05-25T03:41:20.817Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to heartbeat to: peer=c2b5e523-4828-a171-748f-73d35327a0d5 backoff time=10ms error="failed to connect to peer: c2b5e523-4828-a171-748f-73d35327a0d5" 2026-05-25T03:41:20.819Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to appendEntries to: peer="{Voter server-31a04182-053a-cd2c-ee98-e53b9894e1e8 31a04182-053a-cd2c-ee98-e53b9894e1e8}" error="failed to connect to peer: 31a04182-053a-cd2c-ee98-e53b9894e1e8" 2026-05-25T03:41:20.820Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to appendEntries to: peer="{Voter server-c2b5e523-4828-a171-748f-73d35327a0d5 c2b5e523-4828-a171-748f-73d35327a0d5}" error="failed to connect to peer: c2b5e523-4828-a171-748f-73d35327a0d5" 2026-05-25T03:41:20.829Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to heartbeat to: peer=31a04182-053a-cd2c-ee98-e53b9894e1e8 backoff time=10ms error="failed to connect to peer: 31a04182-053a-cd2c-ee98-e53b9894e1e8" 2026-05-25T03:41:20.837Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to heartbeat to: peer=c2b5e523-4828-a171-748f-73d35327a0d5 backoff time=10ms error="failed to connect to peer: c2b5e523-4828-a171-748f-73d35327a0d5" 2026-05-25T03:41:20.838Z [WARN] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to contact: server-id=server-31a04182-053a-cd2c-ee98-e53b9894e1e8 time=51.05608ms 2026-05-25T03:41:20.838Z [WARN] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to contact: server-id=server-c2b5e523-4828-a171-748f-73d35327a0d5 time=51.02096ms 2026-05-25T03:41:20.838Z [WARN] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to contact quorum of nodes, stepping down 2026-05-25T03:41:20.838Z [INFO] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: entering follower state: follower="Node at 2c23feeb-c033-c39e-0f7f-d2c29fd87a73 [Follower]" leader-address= leader-id= 2026-05-25T03:41:20.839Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to appendEntries to: peer="{Voter server-31a04182-053a-cd2c-ee98-e53b9894e1e8 31a04182-053a-cd2c-ee98-e53b9894e1e8}" error="failed to connect to peer: 31a04182-053a-cd2c-ee98-e53b9894e1e8" 2026-05-25T03:41:20.840Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to appendEntries to: peer="{Voter server-c2b5e523-4828-a171-748f-73d35327a0d5 c2b5e523-4828-a171-748f-73d35327a0d5}" error="failed to connect to peer: c2b5e523-4828-a171-748f-73d35327a0d5" 2026-05-25T03:41:20.847Z [WARN] server-c2b5e523-4828-a171-748f-73d35327a0d5: heartbeat timeout reached, starting election: last-leader-addr=2c23feeb-c033-c39e-0f7f-d2c29fd87a73 last-leader-id=server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 2026-05-25T03:41:20.847Z [INFO] server-c2b5e523-4828-a171-748f-73d35327a0d5: entering candidate state: node="Node at c2b5e523-4828-a171-748f-73d35327a0d5 [Candidate]" term=3 2026-05-25T03:41:20.847Z [ERROR] server-c2b5e523-4828-a171-748f-73d35327a0d5: failed to make requestVote RPC: target="{Voter server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 2c23feeb-c033-c39e-0f7f-d2c29fd87a73}" error="failed to connect to peer: 2c23feeb-c033-c39e-0f7f-d2c29fd87a73" term=3 2026-05-25T03:41:20.847Z [WARN] server-31a04182-053a-cd2c-ee98-e53b9894e1e8: rejecting vote request since we have a leader: from=c2b5e523-4828-a171-748f-73d35327a0d5 leader=2c23feeb-c033-c39e-0f7f-d2c29fd87a73 leader-id=server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 2026-05-25T03:41:20.894Z [WARN] server-31a04182-053a-cd2c-ee98-e53b9894e1e8: heartbeat timeout reached, starting election: last-leader-addr=2c23feeb-c033-c39e-0f7f-d2c29fd87a73 last-leader-id=server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 2026-05-25T03:41:20.894Z [INFO] server-31a04182-053a-cd2c-ee98-e53b9894e1e8: entering candidate state: node="Node at 31a04182-053a-cd2c-ee98-e53b9894e1e8 [Candidate]" term=3 2026-05-25T03:41:20.894Z [INFO] server-c2b5e523-4828-a171-748f-73d35327a0d5: duplicate requestVote for same term: term=3 2026-05-25T03:41:20.895Z [ERROR] server-31a04182-053a-cd2c-ee98-e53b9894e1e8: failed to make requestVote RPC: target="{Voter server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 2c23feeb-c033-c39e-0f7f-d2c29fd87a73}" error="failed to connect to peer: 2c23feeb-c033-c39e-0f7f-d2c29fd87a73" term=3 2026-05-25T03:41:20.915Z [WARN] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:20.915Z [INFO] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: entering candidate state: node="Node at 2c23feeb-c033-c39e-0f7f-d2c29fd87a73 [Candidate]" term=3 2026-05-25T03:41:20.915Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to make requestVote RPC: target="{Voter server-c2b5e523-4828-a171-748f-73d35327a0d5 c2b5e523-4828-a171-748f-73d35327a0d5}" error="failed to connect to peer: c2b5e523-4828-a171-748f-73d35327a0d5" term=3 2026-05-25T03:41:20.915Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to make requestVote RPC: target="{Voter server-31a04182-053a-cd2c-ee98-e53b9894e1e8 31a04182-053a-cd2c-ee98-e53b9894e1e8}" error="failed to connect to peer: 31a04182-053a-cd2c-ee98-e53b9894e1e8" term=3 2026-05-25T03:41:20.943Z [WARN] server-c2b5e523-4828-a171-748f-73d35327a0d5: Election timeout reached, restarting election 2026-05-25T03:41:20.943Z [INFO] server-c2b5e523-4828-a171-748f-73d35327a0d5: entering candidate state: node="Node at c2b5e523-4828-a171-748f-73d35327a0d5 [Candidate]" term=4 2026-05-25T03:41:20.943Z [ERROR] server-c2b5e523-4828-a171-748f-73d35327a0d5: failed to make requestVote RPC: target="{Voter server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 2c23feeb-c033-c39e-0f7f-d2c29fd87a73}" error="failed to connect to peer: 2c23feeb-c033-c39e-0f7f-d2c29fd87a73" term=4 2026-05-25T03:41:20.943Z [INFO] server-31a04182-053a-cd2c-ee98-e53b9894e1e8: entering follower state: follower="Node at 31a04182-053a-cd2c-ee98-e53b9894e1e8 [Follower]" leader-address= leader-id= 2026-05-25T03:41:20.943Z [INFO] server-c2b5e523-4828-a171-748f-73d35327a0d5: election won: term=4 tally=2 2026-05-25T03:41:20.943Z [INFO] server-c2b5e523-4828-a171-748f-73d35327a0d5: entering leader state: leader="Node at c2b5e523-4828-a171-748f-73d35327a0d5 [Leader]" 2026-05-25T03:41:20.943Z [INFO] server-c2b5e523-4828-a171-748f-73d35327a0d5: added peer, starting replication: peer=server-31a04182-053a-cd2c-ee98-e53b9894e1e8 2026-05-25T03:41:20.943Z [INFO] server-c2b5e523-4828-a171-748f-73d35327a0d5: added peer, starting replication: peer=server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 2026-05-25T03:41:20.943Z [ERROR] server-c2b5e523-4828-a171-748f-73d35327a0d5: failed to appendEntries to: peer="{Voter server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 2c23feeb-c033-c39e-0f7f-d2c29fd87a73}" error="failed to connect to peer: 2c23feeb-c033-c39e-0f7f-d2c29fd87a73" 2026-05-25T03:41:20.943Z [INFO] server-c2b5e523-4828-a171-748f-73d35327a0d5: pipelining replication: peer="{Voter server-31a04182-053a-cd2c-ee98-e53b9894e1e8 31a04182-053a-cd2c-ee98-e53b9894e1e8}" 2026-05-25T03:41:20.950Z [ERROR] server-c2b5e523-4828-a171-748f-73d35327a0d5: failed to heartbeat to: peer=2c23feeb-c033-c39e-0f7f-d2c29fd87a73 backoff time=10ms error="failed to connect to peer: 2c23feeb-c033-c39e-0f7f-d2c29fd87a73" 2026-05-25T03:41:20.963Z [ERROR] server-c2b5e523-4828-a171-748f-73d35327a0d5: failed to appendEntries to: peer="{Voter server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 2c23feeb-c033-c39e-0f7f-d2c29fd87a73}" error="failed to connect to peer: 2c23feeb-c033-c39e-0f7f-d2c29fd87a73" 2026-05-25T03:41:20.968Z [ERROR] server-c2b5e523-4828-a171-748f-73d35327a0d5: failed to heartbeat to: peer=2c23feeb-c033-c39e-0f7f-d2c29fd87a73 backoff time=10ms error="failed to connect to peer: 2c23feeb-c033-c39e-0f7f-d2c29fd87a73" 2026-05-25T03:41:20.984Z [ERROR] server-c2b5e523-4828-a171-748f-73d35327a0d5: failed to appendEntries to: peer="{Voter server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 2c23feeb-c033-c39e-0f7f-d2c29fd87a73}" error="failed to connect to peer: 2c23feeb-c033-c39e-0f7f-d2c29fd87a73" 2026-05-25T03:41:20.990Z [ERROR] server-c2b5e523-4828-a171-748f-73d35327a0d5: failed to heartbeat to: peer=2c23feeb-c033-c39e-0f7f-d2c29fd87a73 backoff time=10ms error="failed to connect to peer: 2c23feeb-c033-c39e-0f7f-d2c29fd87a73" 2026-05-25T03:41:20.994Z [WARN] server-c2b5e523-4828-a171-748f-73d35327a0d5: failed to contact: server-id=server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 time=50.86188ms 2026-05-25T03:41:20.999Z [WARN] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: Election timeout reached, restarting election 2026-05-25T03:41:20.999Z [INFO] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: entering candidate state: node="Node at 2c23feeb-c033-c39e-0f7f-d2c29fd87a73 [Candidate]" term=4 2026-05-25T03:41:20.999Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to make requestVote RPC: target="{Voter server-c2b5e523-4828-a171-748f-73d35327a0d5 c2b5e523-4828-a171-748f-73d35327a0d5}" error="failed to connect to peer: c2b5e523-4828-a171-748f-73d35327a0d5" term=4 2026-05-25T03:41:20.999Z [ERROR] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: failed to make requestVote RPC: target="{Voter server-31a04182-053a-cd2c-ee98-e53b9894e1e8 31a04182-053a-cd2c-ee98-e53b9894e1e8}" error="failed to connect to peer: 31a04182-053a-cd2c-ee98-e53b9894e1e8" term=4 2026-05-25T03:41:21.007Z [ERROR] server-c2b5e523-4828-a171-748f-73d35327a0d5: failed to heartbeat to: peer=2c23feeb-c033-c39e-0f7f-d2c29fd87a73 backoff time=20ms error="failed to connect to peer: 2c23feeb-c033-c39e-0f7f-d2c29fd87a73" 2026-05-25T03:41:21.015Z [ERROR] server-c2b5e523-4828-a171-748f-73d35327a0d5: failed to appendEntries to: peer="{Voter server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 2c23feeb-c033-c39e-0f7f-d2c29fd87a73}" error="failed to connect to peer: 2c23feeb-c033-c39e-0f7f-d2c29fd87a73" 2026-05-25T03:41:21.036Z [ERROR] server-c2b5e523-4828-a171-748f-73d35327a0d5: failed to heartbeat to: peer=2c23feeb-c033-c39e-0f7f-d2c29fd87a73 backoff time=25ms error="failed to connect to peer: 2c23feeb-c033-c39e-0f7f-d2c29fd87a73" 2026-05-25T03:41:21.042Z [WARN] server-c2b5e523-4828-a171-748f-73d35327a0d5: failed to contact: server-id=server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 time=99.17464ms 2026-05-25T03:41:21.049Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:20.943664917 +0000 UTC m=+7.116640361 (1 nodes), 149.11016ms from start of poll, 450.7376ms from cluster start. Timeout at 2026-05-25 03:41:21.048667317 +0000 UTC m=+7.221642761, 105.0024ms after stability raft_test.go:413: [INFO] Reconnecting Node at 2c23feeb-c033-c39e-0f7f-d2c29fd87a73 [Candidate] 2026-05-25T03:41:21.065Z [INFO] server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73: entering follower state: follower="Node at 2c23feeb-c033-c39e-0f7f-d2c29fd87a73 [Follower]" leader-address=c2b5e523-4828-a171-748f-73d35327a0d5 leader-id=server-c2b5e523-4828-a171-748f-73d35327a0d5 2026-05-25T03:41:21.065Z [INFO] server-c2b5e523-4828-a171-748f-73d35327a0d5: pipelining replication: peer="{Voter server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 2c23feeb-c033-c39e-0f7f-d2c29fd87a73}" 2026-05-25T03:41:21.069Z [INFO] server-c2b5e523-4828-a171-748f-73d35327a0d5: aborting pipeline replication: peer="{Voter server-2c23feeb-c033-c39e-0f7f-d2c29fd87a73 2c23feeb-c033-c39e-0f7f-d2c29fd87a73}" 2026-05-25T03:41:21.069Z [INFO] server-c2b5e523-4828-a171-748f-73d35327a0d5: aborting pipeline replication: peer="{Voter server-31a04182-053a-cd2c-ee98-e53b9894e1e8 31a04182-053a-cd2c-ee98-e53b9894e1e8}" --- PASS: TestRaft_LeaderFail (0.58s) === RUN TestRaft_BehindFollower 2026-05-25T03:41:21.070Z [INFO] server-d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5 Address:d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5} {Suffrage:Voter ID:server-2dd5411c-27c2-eeb0-3e70-864295f10997 Address:2dd5411c-27c2-eeb0-3e70-864295f10997} {Suffrage:Voter ID:server-3435a418-e8fa-513d-9188-740447c07e1f Address:3435a418-e8fa-513d-9188-740447c07e1f}]" 2026-05-25T03:41:21.070Z [INFO] server-2dd5411c-27c2-eeb0-3e70-864295f10997: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5 Address:d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5} {Suffrage:Voter ID:server-2dd5411c-27c2-eeb0-3e70-864295f10997 Address:2dd5411c-27c2-eeb0-3e70-864295f10997} {Suffrage:Voter ID:server-3435a418-e8fa-513d-9188-740447c07e1f Address:3435a418-e8fa-513d-9188-740447c07e1f}]" 2026-05-25T03:41:21.070Z [INFO] server-d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5: entering follower state: follower="Node at d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5 [Follower]" leader-address= leader-id= 2026-05-25T03:41:21.070Z [INFO] server-2dd5411c-27c2-eeb0-3e70-864295f10997: entering follower state: follower="Node at 2dd5411c-27c2-eeb0-3e70-864295f10997 [Follower]" leader-address= leader-id= 2026-05-25T03:41:21.070Z [INFO] server-3435a418-e8fa-513d-9188-740447c07e1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5 Address:d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5} {Suffrage:Voter ID:server-2dd5411c-27c2-eeb0-3e70-864295f10997 Address:2dd5411c-27c2-eeb0-3e70-864295f10997} {Suffrage:Voter ID:server-3435a418-e8fa-513d-9188-740447c07e1f Address:3435a418-e8fa-513d-9188-740447c07e1f}]" 2026-05-25T03:41:21.070Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:21.070Z [INFO] server-3435a418-e8fa-513d-9188-740447c07e1f: entering follower state: follower="Node at 3435a418-e8fa-513d-9188-740447c07e1f [Follower]" leader-address= leader-id= 2026-05-25T03:41:21.140Z [WARN] server-2dd5411c-27c2-eeb0-3e70-864295f10997: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:21.140Z [INFO] server-2dd5411c-27c2-eeb0-3e70-864295f10997: entering candidate state: node="Node at 2dd5411c-27c2-eeb0-3e70-864295f10997 [Candidate]" term=2 2026-05-25T03:41:21.140Z [INFO] server-2dd5411c-27c2-eeb0-3e70-864295f10997: election won: term=2 tally=2 2026-05-25T03:41:21.140Z [INFO] server-2dd5411c-27c2-eeb0-3e70-864295f10997: entering leader state: leader="Node at 2dd5411c-27c2-eeb0-3e70-864295f10997 [Leader]" 2026-05-25T03:41:21.140Z [INFO] server-2dd5411c-27c2-eeb0-3e70-864295f10997: added peer, starting replication: peer=server-d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5 2026-05-25T03:41:21.140Z [INFO] server-2dd5411c-27c2-eeb0-3e70-864295f10997: added peer, starting replication: peer=server-3435a418-e8fa-513d-9188-740447c07e1f 2026-05-25T03:41:21.140Z [INFO] server-2dd5411c-27c2-eeb0-3e70-864295f10997: pipelining replication: peer="{Voter server-3435a418-e8fa-513d-9188-740447c07e1f 3435a418-e8fa-513d-9188-740447c07e1f}" 2026-05-25T03:41:21.140Z [INFO] server-2dd5411c-27c2-eeb0-3e70-864295f10997: pipelining replication: peer="{Voter server-d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5 d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5}" 2026-05-25T03:41:21.246Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:21.140525477 +0000 UTC m=+7.313500921 (1 nodes), 69.9298ms from start of poll, 70.3416ms from cluster start. Timeout at 2026-05-25 03:41:21.245528397 +0000 UTC m=+7.418503841, 105.00292ms after stability 2026-05-25T03:41:21.246Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:21.351Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:21.246563237 +0000 UTC m=+7.419538681 (2 nodes), 1.24µs from start of poll, 176.37936ms from cluster start. Timeout at 2026-05-25 03:41:21.351569077 +0000 UTC m=+7.524544521, 105.00584ms after stability 2026-05-25T03:41:21.351Z [ERROR] server-2dd5411c-27c2-eeb0-3e70-864295f10997: failed to pipeline appendEntries: peer="{Voter server-d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5 d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5}" error="append pipeline closed" 2026-05-25T03:41:21.351Z [INFO] server-2dd5411c-27c2-eeb0-3e70-864295f10997: aborting pipeline replication: peer="{Voter server-d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5 d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5}" 2026-05-25T03:41:21.352Z [ERROR] server-2dd5411c-27c2-eeb0-3e70-864295f10997: failed to appendEntries to: peer="{Voter server-d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5 d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5}" error="failed to connect to peer: d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5" 2026-05-25T03:41:21.353Z [ERROR] server-2dd5411c-27c2-eeb0-3e70-864295f10997: failed to heartbeat to: peer=d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5 backoff time=10ms error="failed to connect to peer: d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5" raft_test.go:459: [INFO] Finished apply without behind follower 2026-05-25T03:41:21.362Z [INFO] server-2dd5411c-27c2-eeb0-3e70-864295f10997: pipelining replication: peer="{Voter server-d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5 d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5}" 2026-05-25T03:41:21.374Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:21.479Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:21.374137437 +0000 UTC m=+7.547112881 (1 nodes), 2.52µs from start of poll, 303.95356ms from cluster start. Timeout at 2026-05-25 03:41:21.479143677 +0000 UTC m=+7.652119121, 105.00624ms after stability 2026-05-25T03:41:21.479Z [INFO] server-2dd5411c-27c2-eeb0-3e70-864295f10997: aborting pipeline replication: peer="{Voter server-3435a418-e8fa-513d-9188-740447c07e1f 3435a418-e8fa-513d-9188-740447c07e1f}" 2026-05-25T03:41:21.479Z [INFO] server-2dd5411c-27c2-eeb0-3e70-864295f10997: aborting pipeline replication: peer="{Voter server-d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5 d68e1d7d-4f16-f0ca-7f2f-a2566384a6e5}" --- PASS: TestRaft_BehindFollower (0.41s) === RUN TestRaft_ApplyNonLeader 2026-05-25T03:41:21.480Z [INFO] server-95d01c5b-9d8c-8079-3784-9e376bca4d4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95d01c5b-9d8c-8079-3784-9e376bca4d4d Address:95d01c5b-9d8c-8079-3784-9e376bca4d4d} {Suffrage:Voter ID:server-e0a227c0-57fc-e8cf-1078-68003a565620 Address:e0a227c0-57fc-e8cf-1078-68003a565620} {Suffrage:Voter ID:server-aceab36f-da26-bdbf-9499-bc1a67ebe0aa Address:aceab36f-da26-bdbf-9499-bc1a67ebe0aa}]" 2026-05-25T03:41:21.481Z [INFO] server-e0a227c0-57fc-e8cf-1078-68003a565620: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95d01c5b-9d8c-8079-3784-9e376bca4d4d Address:95d01c5b-9d8c-8079-3784-9e376bca4d4d} {Suffrage:Voter ID:server-e0a227c0-57fc-e8cf-1078-68003a565620 Address:e0a227c0-57fc-e8cf-1078-68003a565620} {Suffrage:Voter ID:server-aceab36f-da26-bdbf-9499-bc1a67ebe0aa Address:aceab36f-da26-bdbf-9499-bc1a67ebe0aa}]" 2026-05-25T03:41:21.480Z [INFO] server-95d01c5b-9d8c-8079-3784-9e376bca4d4d: entering follower state: follower="Node at 95d01c5b-9d8c-8079-3784-9e376bca4d4d [Follower]" leader-address= leader-id= 2026-05-25T03:41:21.481Z [INFO] server-aceab36f-da26-bdbf-9499-bc1a67ebe0aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95d01c5b-9d8c-8079-3784-9e376bca4d4d Address:95d01c5b-9d8c-8079-3784-9e376bca4d4d} {Suffrage:Voter ID:server-e0a227c0-57fc-e8cf-1078-68003a565620 Address:e0a227c0-57fc-e8cf-1078-68003a565620} {Suffrage:Voter ID:server-aceab36f-da26-bdbf-9499-bc1a67ebe0aa Address:aceab36f-da26-bdbf-9499-bc1a67ebe0aa}]" 2026-05-25T03:41:21.481Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:21.481Z [INFO] server-e0a227c0-57fc-e8cf-1078-68003a565620: entering follower state: follower="Node at e0a227c0-57fc-e8cf-1078-68003a565620 [Follower]" leader-address= leader-id= 2026-05-25T03:41:21.481Z [INFO] server-aceab36f-da26-bdbf-9499-bc1a67ebe0aa: entering follower state: follower="Node at aceab36f-da26-bdbf-9499-bc1a67ebe0aa [Follower]" leader-address= leader-id= 2026-05-25T03:41:21.546Z [WARN] server-95d01c5b-9d8c-8079-3784-9e376bca4d4d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:21.546Z [INFO] server-95d01c5b-9d8c-8079-3784-9e376bca4d4d: entering candidate state: node="Node at 95d01c5b-9d8c-8079-3784-9e376bca4d4d [Candidate]" term=2 2026-05-25T03:41:21.546Z [INFO] server-95d01c5b-9d8c-8079-3784-9e376bca4d4d: election won: term=2 tally=2 2026-05-25T03:41:21.546Z [INFO] server-95d01c5b-9d8c-8079-3784-9e376bca4d4d: entering leader state: leader="Node at 95d01c5b-9d8c-8079-3784-9e376bca4d4d [Leader]" 2026-05-25T03:41:21.546Z [INFO] server-95d01c5b-9d8c-8079-3784-9e376bca4d4d: added peer, starting replication: peer=server-e0a227c0-57fc-e8cf-1078-68003a565620 2026-05-25T03:41:21.546Z [INFO] server-95d01c5b-9d8c-8079-3784-9e376bca4d4d: added peer, starting replication: peer=server-aceab36f-da26-bdbf-9499-bc1a67ebe0aa 2026-05-25T03:41:21.546Z [INFO] server-95d01c5b-9d8c-8079-3784-9e376bca4d4d: pipelining replication: peer="{Voter server-aceab36f-da26-bdbf-9499-bc1a67ebe0aa aceab36f-da26-bdbf-9499-bc1a67ebe0aa}" 2026-05-25T03:41:21.546Z [INFO] server-95d01c5b-9d8c-8079-3784-9e376bca4d4d: pipelining replication: peer="{Voter server-e0a227c0-57fc-e8cf-1078-68003a565620 e0a227c0-57fc-e8cf-1078-68003a565620}" 2026-05-25T03:41:21.652Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:21.546615277 +0000 UTC m=+7.719590721 (1 nodes), 65.41412ms from start of poll, 65.86132ms from cluster start. Timeout at 2026-05-25 03:41:21.651618037 +0000 UTC m=+7.824593481, 105.00276ms after stability 2026-05-25T03:41:21.653Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:21.758Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:21.653112197 +0000 UTC m=+7.826087641 (2 nodes), 1.36µs from start of poll, 172.35824ms from cluster start. Timeout at 2026-05-25 03:41:21.758134437 +0000 UTC m=+7.931109881, 105.02224ms after stability 2026-05-25T03:41:21.759Z [INFO] server-95d01c5b-9d8c-8079-3784-9e376bca4d4d: aborting pipeline replication: peer="{Voter server-aceab36f-da26-bdbf-9499-bc1a67ebe0aa aceab36f-da26-bdbf-9499-bc1a67ebe0aa}" 2026-05-25T03:41:21.759Z [INFO] server-95d01c5b-9d8c-8079-3784-9e376bca4d4d: aborting pipeline replication: peer="{Voter server-e0a227c0-57fc-e8cf-1078-68003a565620 e0a227c0-57fc-e8cf-1078-68003a565620}" --- PASS: TestRaft_ApplyNonLeader (0.28s) === RUN TestRaft_ApplyConcurrent 2026-05-25T03:41:21.759Z [INFO] server-0c6d02dd-b60a-2330-9308-25d0cee6035b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c6d02dd-b60a-2330-9308-25d0cee6035b Address:0c6d02dd-b60a-2330-9308-25d0cee6035b} {Suffrage:Voter ID:server-17a5c30c-6baa-325d-5784-fa2b1d62ceac Address:17a5c30c-6baa-325d-5784-fa2b1d62ceac} {Suffrage:Voter ID:server-042e7940-434b-99ac-ecb1-e00a9713f21d Address:042e7940-434b-99ac-ecb1-e00a9713f21d}]" 2026-05-25T03:41:21.760Z [INFO] server-17a5c30c-6baa-325d-5784-fa2b1d62ceac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c6d02dd-b60a-2330-9308-25d0cee6035b Address:0c6d02dd-b60a-2330-9308-25d0cee6035b} {Suffrage:Voter ID:server-17a5c30c-6baa-325d-5784-fa2b1d62ceac Address:17a5c30c-6baa-325d-5784-fa2b1d62ceac} {Suffrage:Voter ID:server-042e7940-434b-99ac-ecb1-e00a9713f21d Address:042e7940-434b-99ac-ecb1-e00a9713f21d}]" 2026-05-25T03:41:21.760Z [INFO] server-0c6d02dd-b60a-2330-9308-25d0cee6035b: entering follower state: follower="Node at 0c6d02dd-b60a-2330-9308-25d0cee6035b [Follower]" leader-address= leader-id= 2026-05-25T03:41:21.760Z [INFO] server-042e7940-434b-99ac-ecb1-e00a9713f21d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c6d02dd-b60a-2330-9308-25d0cee6035b Address:0c6d02dd-b60a-2330-9308-25d0cee6035b} {Suffrage:Voter ID:server-17a5c30c-6baa-325d-5784-fa2b1d62ceac Address:17a5c30c-6baa-325d-5784-fa2b1d62ceac} {Suffrage:Voter ID:server-042e7940-434b-99ac-ecb1-e00a9713f21d Address:042e7940-434b-99ac-ecb1-e00a9713f21d}]" 2026-05-25T03:41:21.760Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:21.760Z [INFO] server-17a5c30c-6baa-325d-5784-fa2b1d62ceac: entering follower state: follower="Node at 17a5c30c-6baa-325d-5784-fa2b1d62ceac [Follower]" leader-address= leader-id= 2026-05-25T03:41:21.760Z [INFO] server-042e7940-434b-99ac-ecb1-e00a9713f21d: entering follower state: follower="Node at 042e7940-434b-99ac-ecb1-e00a9713f21d [Follower]" leader-address= leader-id= 2026-05-25T03:41:21.875Z [WARN] server-042e7940-434b-99ac-ecb1-e00a9713f21d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:21.875Z [INFO] server-042e7940-434b-99ac-ecb1-e00a9713f21d: entering candidate state: node="Node at 042e7940-434b-99ac-ecb1-e00a9713f21d [Candidate]" term=2 2026-05-25T03:41:21.875Z [INFO] server-042e7940-434b-99ac-ecb1-e00a9713f21d: election won: term=2 tally=2 2026-05-25T03:41:21.875Z [INFO] server-042e7940-434b-99ac-ecb1-e00a9713f21d: entering leader state: leader="Node at 042e7940-434b-99ac-ecb1-e00a9713f21d [Leader]" 2026-05-25T03:41:21.876Z [INFO] server-042e7940-434b-99ac-ecb1-e00a9713f21d: added peer, starting replication: peer=server-0c6d02dd-b60a-2330-9308-25d0cee6035b 2026-05-25T03:41:21.876Z [INFO] server-042e7940-434b-99ac-ecb1-e00a9713f21d: added peer, starting replication: peer=server-17a5c30c-6baa-325d-5784-fa2b1d62ceac 2026-05-25T03:41:21.876Z [INFO] server-042e7940-434b-99ac-ecb1-e00a9713f21d: pipelining replication: peer="{Voter server-17a5c30c-6baa-325d-5784-fa2b1d62ceac 17a5c30c-6baa-325d-5784-fa2b1d62ceac}" 2026-05-25T03:41:21.876Z [INFO] server-042e7940-434b-99ac-ecb1-e00a9713f21d: pipelining replication: peer="{Voter server-0c6d02dd-b60a-2330-9308-25d0cee6035b 0c6d02dd-b60a-2330-9308-25d0cee6035b}" 2026-05-25T03:41:22.082Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:21.876029797 +0000 UTC m=+8.049005241 (1 nodes), 115.75492ms from start of poll, 116.1922ms from cluster start. Timeout at 2026-05-25 03:41:22.081031477 +0000 UTC m=+8.254006921, 205.00168ms after stability 2026-05-25T03:41:22.096Z [INFO] server-042e7940-434b-99ac-ecb1-e00a9713f21d: aborting pipeline replication: peer="{Voter server-17a5c30c-6baa-325d-5784-fa2b1d62ceac 17a5c30c-6baa-325d-5784-fa2b1d62ceac}" 2026-05-25T03:41:22.096Z [INFO] server-042e7940-434b-99ac-ecb1-e00a9713f21d: aborting pipeline replication: peer="{Voter server-0c6d02dd-b60a-2330-9308-25d0cee6035b 0c6d02dd-b60a-2330-9308-25d0cee6035b}" --- PASS: TestRaft_ApplyConcurrent (0.34s) === RUN TestRaft_ApplyConcurrent_Timeout 2026-05-25T03:41:22.097Z [INFO] server-3a89f977-b373-253e-b8b3-93544e8ac848: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a89f977-b373-253e-b8b3-93544e8ac848 Address:3a89f977-b373-253e-b8b3-93544e8ac848}]" 2026-05-25T03:41:22.097Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:22.097Z [INFO] server-3a89f977-b373-253e-b8b3-93544e8ac848: entering follower state: follower="Node at 3a89f977-b373-253e-b8b3-93544e8ac848 [Follower]" leader-address= leader-id= 2026-05-25T03:41:22.273Z [WARN] server-3a89f977-b373-253e-b8b3-93544e8ac848: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:22.273Z [INFO] server-3a89f977-b373-253e-b8b3-93544e8ac848: entering candidate state: node="Node at 3a89f977-b373-253e-b8b3-93544e8ac848 [Candidate]" term=2 2026-05-25T03:41:22.273Z [INFO] server-3a89f977-b373-253e-b8b3-93544e8ac848: election won: term=2 tally=1 2026-05-25T03:41:22.273Z [INFO] server-3a89f977-b373-253e-b8b3-93544e8ac848: entering leader state: leader="Node at 3a89f977-b373-253e-b8b3-93544e8ac848 [Leader]" 2026-05-25T03:41:22.475Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:22.274079357 +0000 UTC m=+8.447054801 (1 nodes), 176.984ms from start of poll, 177.1798ms from cluster start. Timeout at 2026-05-25 03:41:22.475084037 +0000 UTC m=+8.648059481, 201.00468ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.38s) === RUN TestRaft_JoinNode 2026-05-25T03:41:22.476Z [INFO] server-2cdd46f3-00cc-84d1-8c64-7737a148fab7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cdd46f3-00cc-84d1-8c64-7737a148fab7 Address:2cdd46f3-00cc-84d1-8c64-7737a148fab7} {Suffrage:Voter ID:server-b6464c7e-edbe-6810-32be-d67013979e6c Address:b6464c7e-edbe-6810-32be-d67013979e6c}]" 2026-05-25T03:41:22.476Z [INFO] server-b6464c7e-edbe-6810-32be-d67013979e6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cdd46f3-00cc-84d1-8c64-7737a148fab7 Address:2cdd46f3-00cc-84d1-8c64-7737a148fab7} {Suffrage:Voter ID:server-b6464c7e-edbe-6810-32be-d67013979e6c Address:b6464c7e-edbe-6810-32be-d67013979e6c}]" 2026-05-25T03:41:22.476Z [INFO] server-2cdd46f3-00cc-84d1-8c64-7737a148fab7: entering follower state: follower="Node at 2cdd46f3-00cc-84d1-8c64-7737a148fab7 [Follower]" leader-address= leader-id= 2026-05-25T03:41:22.476Z [INFO] server-b6464c7e-edbe-6810-32be-d67013979e6c: entering follower state: follower="Node at b6464c7e-edbe-6810-32be-d67013979e6c [Follower]" leader-address= leader-id= 2026-05-25T03:41:22.476Z [INFO] server-1f75ce70-8621-f5c7-f6d4-d6ebc726c0db: initial configuration: index=0 servers=[] 2026-05-25T03:41:22.476Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:22.476Z [INFO] server-1f75ce70-8621-f5c7-f6d4-d6ebc726c0db: entering follower state: follower="Node at 1f75ce70-8621-f5c7-f6d4-d6ebc726c0db [Follower]" leader-address= leader-id= 2026-05-25T03:41:22.532Z [WARN] server-b6464c7e-edbe-6810-32be-d67013979e6c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:22.532Z [INFO] server-b6464c7e-edbe-6810-32be-d67013979e6c: entering candidate state: node="Node at b6464c7e-edbe-6810-32be-d67013979e6c [Candidate]" term=2 2026-05-25T03:41:22.532Z [INFO] server-b6464c7e-edbe-6810-32be-d67013979e6c: election won: term=2 tally=2 2026-05-25T03:41:22.532Z [INFO] server-b6464c7e-edbe-6810-32be-d67013979e6c: entering leader state: leader="Node at b6464c7e-edbe-6810-32be-d67013979e6c [Leader]" 2026-05-25T03:41:22.532Z [INFO] server-b6464c7e-edbe-6810-32be-d67013979e6c: added peer, starting replication: peer=server-2cdd46f3-00cc-84d1-8c64-7737a148fab7 2026-05-25T03:41:22.532Z [INFO] server-b6464c7e-edbe-6810-32be-d67013979e6c: pipelining replication: peer="{Voter server-2cdd46f3-00cc-84d1-8c64-7737a148fab7 2cdd46f3-00cc-84d1-8c64-7737a148fab7}" 2026-05-25T03:41:22.545Z [WARN] server-1f75ce70-8621-f5c7-f6d4-d6ebc726c0db: no known peers, aborting election 2026-05-25T03:41:22.638Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:22.532765197 +0000 UTC m=+8.705740641 (1 nodes), 55.9476ms from start of poll, 56.41264ms from cluster start. Timeout at 2026-05-25 03:41:22.637767797 +0000 UTC m=+8.810743241, 105.0026ms after stability 2026-05-25T03:41:22.638Z [INFO] server-b6464c7e-edbe-6810-32be-d67013979e6c: updating configuration: command=AddVoter server-id=server-1f75ce70-8621-f5c7-f6d4-d6ebc726c0db server-addr=1f75ce70-8621-f5c7-f6d4-d6ebc726c0db servers="[{Suffrage:Voter ID:server-2cdd46f3-00cc-84d1-8c64-7737a148fab7 Address:2cdd46f3-00cc-84d1-8c64-7737a148fab7} {Suffrage:Voter ID:server-b6464c7e-edbe-6810-32be-d67013979e6c Address:b6464c7e-edbe-6810-32be-d67013979e6c} {Suffrage:Voter ID:server-1f75ce70-8621-f5c7-f6d4-d6ebc726c0db Address:1f75ce70-8621-f5c7-f6d4-d6ebc726c0db}]" 2026-05-25T03:41:22.639Z [INFO] server-b6464c7e-edbe-6810-32be-d67013979e6c: added peer, starting replication: peer=server-1f75ce70-8621-f5c7-f6d4-d6ebc726c0db 2026-05-25T03:41:22.640Z [WARN] server-1f75ce70-8621-f5c7-f6d4-d6ebc726c0db: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2026-05-25T03:41:22.640Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:22.640Z [WARN] server-b6464c7e-edbe-6810-32be-d67013979e6c: appendEntries rejected, sending older logs: peer="{Voter server-1f75ce70-8621-f5c7-f6d4-d6ebc726c0db 1f75ce70-8621-f5c7-f6d4-d6ebc726c0db}" next=1 2026-05-25T03:41:22.640Z [INFO] server-b6464c7e-edbe-6810-32be-d67013979e6c: pipelining replication: peer="{Voter server-1f75ce70-8621-f5c7-f6d4-d6ebc726c0db 1f75ce70-8621-f5c7-f6d4-d6ebc726c0db}" 2026-05-25T03:41:22.745Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:22.640111037 +0000 UTC m=+8.813086481 (1 nodes), 720ns from start of poll, 163.75848ms from cluster start. Timeout at 2026-05-25 03:41:22.745114277 +0000 UTC m=+8.918089721, 105.00324ms after stability 2026-05-25T03:41:22.745Z [INFO] server-b6464c7e-edbe-6810-32be-d67013979e6c: aborting pipeline replication: peer="{Voter server-1f75ce70-8621-f5c7-f6d4-d6ebc726c0db 1f75ce70-8621-f5c7-f6d4-d6ebc726c0db}" 2026-05-25T03:41:22.745Z [INFO] server-b6464c7e-edbe-6810-32be-d67013979e6c: aborting pipeline replication: peer="{Voter server-2cdd46f3-00cc-84d1-8c64-7737a148fab7 2cdd46f3-00cc-84d1-8c64-7737a148fab7}" --- PASS: TestRaft_JoinNode (0.27s) === RUN TestRaft_JoinNode_ConfigStore 2026-05-25T03:41:22.746Z [INFO] server-d71cf05d-bffd-4c39-90eb-d62dd743b799: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d71cf05d-bffd-4c39-90eb-d62dd743b799 Address:d71cf05d-bffd-4c39-90eb-d62dd743b799}]" 2026-05-25T03:41:22.746Z [INFO] server-d71cf05d-bffd-4c39-90eb-d62dd743b799: entering follower state: follower="Node at d71cf05d-bffd-4c39-90eb-d62dd743b799 [Follower]" leader-address= leader-id= 2026-05-25T03:41:22.746Z [INFO] server-1b740a9c-ae68-dab3-5a9e-1435c910096c: initial configuration: index=0 servers=[] 2026-05-25T03:41:22.746Z [INFO] server-1b740a9c-ae68-dab3-5a9e-1435c910096c: entering follower state: follower="Node at 1b740a9c-ae68-dab3-5a9e-1435c910096c [Follower]" leader-address= leader-id= 2026-05-25T03:41:22.746Z [INFO] server-bbfb0c15-7097-d636-f141-24a6fd9d7176: initial configuration: index=0 servers=[] 2026-05-25T03:41:22.746Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:22.746Z [INFO] server-bbfb0c15-7097-d636-f141-24a6fd9d7176: entering follower state: follower="Node at bbfb0c15-7097-d636-f141-24a6fd9d7176 [Follower]" leader-address= leader-id= 2026-05-25T03:41:22.802Z [WARN] server-bbfb0c15-7097-d636-f141-24a6fd9d7176: no known peers, aborting election 2026-05-25T03:41:22.816Z [WARN] server-1b740a9c-ae68-dab3-5a9e-1435c910096c: no known peers, aborting election 2026-05-25T03:41:22.843Z [WARN] server-d71cf05d-bffd-4c39-90eb-d62dd743b799: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:22.843Z [INFO] server-d71cf05d-bffd-4c39-90eb-d62dd743b799: entering candidate state: node="Node at d71cf05d-bffd-4c39-90eb-d62dd743b799 [Candidate]" term=2 2026-05-25T03:41:22.843Z [INFO] server-d71cf05d-bffd-4c39-90eb-d62dd743b799: election won: term=2 tally=1 2026-05-25T03:41:22.843Z [INFO] server-d71cf05d-bffd-4c39-90eb-d62dd743b799: entering leader state: leader="Node at d71cf05d-bffd-4c39-90eb-d62dd743b799 [Leader]" 2026-05-25T03:41:22.949Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:22.843203197 +0000 UTC m=+9.016178681 (1 nodes), 96.76512ms from start of poll, 97.23956ms from cluster start. Timeout at 2026-05-25 03:41:22.948206117 +0000 UTC m=+9.121181561, 105.00288ms after stability 2026-05-25T03:41:22.949Z [INFO] server-d71cf05d-bffd-4c39-90eb-d62dd743b799: updating configuration: command=AddVoter server-id=server-1b740a9c-ae68-dab3-5a9e-1435c910096c server-addr=1b740a9c-ae68-dab3-5a9e-1435c910096c servers="[{Suffrage:Voter ID:server-d71cf05d-bffd-4c39-90eb-d62dd743b799 Address:d71cf05d-bffd-4c39-90eb-d62dd743b799} {Suffrage:Voter ID:server-1b740a9c-ae68-dab3-5a9e-1435c910096c Address:1b740a9c-ae68-dab3-5a9e-1435c910096c}]" 2026-05-25T03:41:22.949Z [INFO] server-d71cf05d-bffd-4c39-90eb-d62dd743b799: added peer, starting replication: peer=server-1b740a9c-ae68-dab3-5a9e-1435c910096c 2026-05-25T03:41:22.949Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:22.949Z [WARN] server-1b740a9c-ae68-dab3-5a9e-1435c910096c: failed to get previous log: previous-index=3 last-index=0 error="log not found" 2026-05-25T03:41:22.949Z [WARN] server-d71cf05d-bffd-4c39-90eb-d62dd743b799: appendEntries rejected, sending older logs: peer="{Voter server-1b740a9c-ae68-dab3-5a9e-1435c910096c 1b740a9c-ae68-dab3-5a9e-1435c910096c}" next=1 2026-05-25T03:41:22.949Z [INFO] server-d71cf05d-bffd-4c39-90eb-d62dd743b799: pipelining replication: peer="{Voter server-1b740a9c-ae68-dab3-5a9e-1435c910096c 1b740a9c-ae68-dab3-5a9e-1435c910096c}" 2026-05-25T03:41:23.054Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:22.949277637 +0000 UTC m=+9.122253081 (1 nodes), 680ns from start of poll, 203.31396ms from cluster start. Timeout at 2026-05-25 03:41:23.054283717 +0000 UTC m=+9.227259161, 105.00608ms after stability 2026-05-25T03:41:23.054Z [INFO] server-d71cf05d-bffd-4c39-90eb-d62dd743b799: updating configuration: command=AddVoter server-id=server-bbfb0c15-7097-d636-f141-24a6fd9d7176 server-addr=bbfb0c15-7097-d636-f141-24a6fd9d7176 servers="[{Suffrage:Voter ID:server-d71cf05d-bffd-4c39-90eb-d62dd743b799 Address:d71cf05d-bffd-4c39-90eb-d62dd743b799} {Suffrage:Voter ID:server-1b740a9c-ae68-dab3-5a9e-1435c910096c Address:1b740a9c-ae68-dab3-5a9e-1435c910096c} {Suffrage:Voter ID:server-bbfb0c15-7097-d636-f141-24a6fd9d7176 Address:bbfb0c15-7097-d636-f141-24a6fd9d7176}]" 2026-05-25T03:41:23.054Z [INFO] server-d71cf05d-bffd-4c39-90eb-d62dd743b799: added peer, starting replication: peer=server-bbfb0c15-7097-d636-f141-24a6fd9d7176 2026-05-25T03:41:23.054Z [WARN] server-bbfb0c15-7097-d636-f141-24a6fd9d7176: failed to get previous log: previous-index=4 last-index=0 error="log not found" 2026-05-25T03:41:23.054Z [WARN] server-d71cf05d-bffd-4c39-90eb-d62dd743b799: appendEntries rejected, sending older logs: peer="{Voter server-bbfb0c15-7097-d636-f141-24a6fd9d7176 bbfb0c15-7097-d636-f141-24a6fd9d7176}" next=1 2026-05-25T03:41:23.054Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:23.054Z [INFO] server-d71cf05d-bffd-4c39-90eb-d62dd743b799: pipelining replication: peer="{Voter server-bbfb0c15-7097-d636-f141-24a6fd9d7176 bbfb0c15-7097-d636-f141-24a6fd9d7176}" 2026-05-25T03:41:23.160Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:23.054768437 +0000 UTC m=+9.227743881 (1 nodes), 760ns from start of poll, 308.80476ms from cluster start. Timeout at 2026-05-25 03:41:23.159774077 +0000 UTC m=+9.332749521, 105.00564ms after stability 2026-05-25T03:41:23.160Z [INFO] server-d71cf05d-bffd-4c39-90eb-d62dd743b799: aborting pipeline replication: peer="{Voter server-bbfb0c15-7097-d636-f141-24a6fd9d7176 bbfb0c15-7097-d636-f141-24a6fd9d7176}" 2026-05-25T03:41:23.160Z [INFO] server-d71cf05d-bffd-4c39-90eb-d62dd743b799: aborting pipeline replication: peer="{Voter server-1b740a9c-ae68-dab3-5a9e-1435c910096c 1b740a9c-ae68-dab3-5a9e-1435c910096c}" --- PASS: TestRaft_JoinNode_ConfigStore (0.41s) === RUN TestRaft_RemoveFollower 2026-05-25T03:41:23.161Z [INFO] server-25c7a985-30bf-1049-e313-da6d500a8708: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c7a985-30bf-1049-e313-da6d500a8708 Address:25c7a985-30bf-1049-e313-da6d500a8708} {Suffrage:Voter ID:server-fa91574c-2e86-7058-ae53-0245bbff4116 Address:fa91574c-2e86-7058-ae53-0245bbff4116} {Suffrage:Voter ID:server-4ebc5457-38ee-5fcd-c8d9-8f439a48f840 Address:4ebc5457-38ee-5fcd-c8d9-8f439a48f840}]" 2026-05-25T03:41:23.161Z [INFO] server-25c7a985-30bf-1049-e313-da6d500a8708: entering follower state: follower="Node at 25c7a985-30bf-1049-e313-da6d500a8708 [Follower]" leader-address= leader-id= 2026-05-25T03:41:23.161Z [INFO] server-fa91574c-2e86-7058-ae53-0245bbff4116: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c7a985-30bf-1049-e313-da6d500a8708 Address:25c7a985-30bf-1049-e313-da6d500a8708} {Suffrage:Voter ID:server-fa91574c-2e86-7058-ae53-0245bbff4116 Address:fa91574c-2e86-7058-ae53-0245bbff4116} {Suffrage:Voter ID:server-4ebc5457-38ee-5fcd-c8d9-8f439a48f840 Address:4ebc5457-38ee-5fcd-c8d9-8f439a48f840}]" 2026-05-25T03:41:23.161Z [INFO] server-4ebc5457-38ee-5fcd-c8d9-8f439a48f840: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25c7a985-30bf-1049-e313-da6d500a8708 Address:25c7a985-30bf-1049-e313-da6d500a8708} {Suffrage:Voter ID:server-fa91574c-2e86-7058-ae53-0245bbff4116 Address:fa91574c-2e86-7058-ae53-0245bbff4116} {Suffrage:Voter ID:server-4ebc5457-38ee-5fcd-c8d9-8f439a48f840 Address:4ebc5457-38ee-5fcd-c8d9-8f439a48f840}]" 2026-05-25T03:41:23.161Z [INFO] server-fa91574c-2e86-7058-ae53-0245bbff4116: entering follower state: follower="Node at fa91574c-2e86-7058-ae53-0245bbff4116 [Follower]" leader-address= leader-id= 2026-05-25T03:41:23.161Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:23.161Z [INFO] server-4ebc5457-38ee-5fcd-c8d9-8f439a48f840: entering follower state: follower="Node at 4ebc5457-38ee-5fcd-c8d9-8f439a48f840 [Follower]" leader-address= leader-id= 2026-05-25T03:41:23.217Z [WARN] server-fa91574c-2e86-7058-ae53-0245bbff4116: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:23.217Z [INFO] server-fa91574c-2e86-7058-ae53-0245bbff4116: entering candidate state: node="Node at fa91574c-2e86-7058-ae53-0245bbff4116 [Candidate]" term=2 2026-05-25T03:41:23.217Z [INFO] server-fa91574c-2e86-7058-ae53-0245bbff4116: election won: term=2 tally=2 2026-05-25T03:41:23.217Z [INFO] server-fa91574c-2e86-7058-ae53-0245bbff4116: entering leader state: leader="Node at fa91574c-2e86-7058-ae53-0245bbff4116 [Leader]" 2026-05-25T03:41:23.217Z [INFO] server-fa91574c-2e86-7058-ae53-0245bbff4116: added peer, starting replication: peer=server-25c7a985-30bf-1049-e313-da6d500a8708 2026-05-25T03:41:23.217Z [INFO] server-fa91574c-2e86-7058-ae53-0245bbff4116: added peer, starting replication: peer=server-4ebc5457-38ee-5fcd-c8d9-8f439a48f840 2026-05-25T03:41:23.217Z [INFO] server-fa91574c-2e86-7058-ae53-0245bbff4116: pipelining replication: peer="{Voter server-4ebc5457-38ee-5fcd-c8d9-8f439a48f840 4ebc5457-38ee-5fcd-c8d9-8f439a48f840}" 2026-05-25T03:41:23.217Z [INFO] server-fa91574c-2e86-7058-ae53-0245bbff4116: pipelining replication: peer="{Voter server-25c7a985-30bf-1049-e313-da6d500a8708 25c7a985-30bf-1049-e313-da6d500a8708}" 2026-05-25T03:41:23.323Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:23.217269317 +0000 UTC m=+9.390244761 (1 nodes), 55.65392ms from start of poll, 56.2148ms from cluster start. Timeout at 2026-05-25 03:41:23.322272237 +0000 UTC m=+9.495247681, 105.00292ms after stability 2026-05-25T03:41:23.328Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:23.434Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:23.328587557 +0000 UTC m=+9.501563001 (2 nodes), 2.44µs from start of poll, 167.53304ms from cluster start. Timeout at 2026-05-25 03:41:23.433594117 +0000 UTC m=+9.606569561, 105.00656ms after stability 2026-05-25T03:41:23.434Z [INFO] server-fa91574c-2e86-7058-ae53-0245bbff4116: updating configuration: command=RemoveServer server-id=server-25c7a985-30bf-1049-e313-da6d500a8708 server-addr= servers="[{Suffrage:Voter ID:server-fa91574c-2e86-7058-ae53-0245bbff4116 Address:fa91574c-2e86-7058-ae53-0245bbff4116} {Suffrage:Voter ID:server-4ebc5457-38ee-5fcd-c8d9-8f439a48f840 Address:4ebc5457-38ee-5fcd-c8d9-8f439a48f840}]" 2026-05-25T03:41:23.434Z [INFO] server-fa91574c-2e86-7058-ae53-0245bbff4116: removed peer, stopping replication: peer=server-25c7a985-30bf-1049-e313-da6d500a8708 last-index=3 2026-05-25T03:41:23.434Z [INFO] server-fa91574c-2e86-7058-ae53-0245bbff4116: aborting pipeline replication: peer="{Voter server-25c7a985-30bf-1049-e313-da6d500a8708 25c7a985-30bf-1049-e313-da6d500a8708}" 2026-05-25T03:41:23.486Z [WARN] server-25c7a985-30bf-1049-e313-da6d500a8708: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2026-05-25T03:41:23.539Z [INFO] server-fa91574c-2e86-7058-ae53-0245bbff4116: aborting pipeline replication: peer="{Voter server-4ebc5457-38ee-5fcd-c8d9-8f439a48f840 4ebc5457-38ee-5fcd-c8d9-8f439a48f840}" --- PASS: TestRaft_RemoveFollower (0.38s) === RUN TestRaft_RemoveLeader 2026-05-25T03:41:23.540Z [INFO] server-585a94f2-d673-d63a-a8b1-6c03203d281c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-585a94f2-d673-d63a-a8b1-6c03203d281c Address:585a94f2-d673-d63a-a8b1-6c03203d281c} {Suffrage:Voter ID:server-656f43df-ae3c-b501-e919-db56755fe898 Address:656f43df-ae3c-b501-e919-db56755fe898} {Suffrage:Voter ID:server-dc85223a-4c72-c2af-a338-d8cbe72fa756 Address:dc85223a-4c72-c2af-a338-d8cbe72fa756}]" 2026-05-25T03:41:23.540Z [INFO] server-656f43df-ae3c-b501-e919-db56755fe898: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-585a94f2-d673-d63a-a8b1-6c03203d281c Address:585a94f2-d673-d63a-a8b1-6c03203d281c} {Suffrage:Voter ID:server-656f43df-ae3c-b501-e919-db56755fe898 Address:656f43df-ae3c-b501-e919-db56755fe898} {Suffrage:Voter ID:server-dc85223a-4c72-c2af-a338-d8cbe72fa756 Address:dc85223a-4c72-c2af-a338-d8cbe72fa756}]" 2026-05-25T03:41:23.540Z [INFO] server-585a94f2-d673-d63a-a8b1-6c03203d281c: entering follower state: follower="Node at 585a94f2-d673-d63a-a8b1-6c03203d281c [Follower]" leader-address= leader-id= 2026-05-25T03:41:23.541Z [INFO] server-dc85223a-4c72-c2af-a338-d8cbe72fa756: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-585a94f2-d673-d63a-a8b1-6c03203d281c Address:585a94f2-d673-d63a-a8b1-6c03203d281c} {Suffrage:Voter ID:server-656f43df-ae3c-b501-e919-db56755fe898 Address:656f43df-ae3c-b501-e919-db56755fe898} {Suffrage:Voter ID:server-dc85223a-4c72-c2af-a338-d8cbe72fa756 Address:dc85223a-4c72-c2af-a338-d8cbe72fa756}]" 2026-05-25T03:41:23.541Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:23.541Z [INFO] server-656f43df-ae3c-b501-e919-db56755fe898: entering follower state: follower="Node at 656f43df-ae3c-b501-e919-db56755fe898 [Follower]" leader-address= leader-id= 2026-05-25T03:41:23.541Z [INFO] server-dc85223a-4c72-c2af-a338-d8cbe72fa756: entering follower state: follower="Node at dc85223a-4c72-c2af-a338-d8cbe72fa756 [Follower]" leader-address= leader-id= 2026-05-25T03:41:23.598Z [WARN] server-585a94f2-d673-d63a-a8b1-6c03203d281c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:23.598Z [INFO] server-585a94f2-d673-d63a-a8b1-6c03203d281c: entering candidate state: node="Node at 585a94f2-d673-d63a-a8b1-6c03203d281c [Candidate]" term=2 2026-05-25T03:41:23.598Z [INFO] server-585a94f2-d673-d63a-a8b1-6c03203d281c: election won: term=2 tally=2 2026-05-25T03:41:23.598Z [INFO] server-585a94f2-d673-d63a-a8b1-6c03203d281c: entering leader state: leader="Node at 585a94f2-d673-d63a-a8b1-6c03203d281c [Leader]" 2026-05-25T03:41:23.598Z [INFO] server-585a94f2-d673-d63a-a8b1-6c03203d281c: added peer, starting replication: peer=server-656f43df-ae3c-b501-e919-db56755fe898 2026-05-25T03:41:23.598Z [INFO] server-585a94f2-d673-d63a-a8b1-6c03203d281c: added peer, starting replication: peer=server-dc85223a-4c72-c2af-a338-d8cbe72fa756 2026-05-25T03:41:23.598Z [INFO] server-585a94f2-d673-d63a-a8b1-6c03203d281c: pipelining replication: peer="{Voter server-dc85223a-4c72-c2af-a338-d8cbe72fa756 dc85223a-4c72-c2af-a338-d8cbe72fa756}" 2026-05-25T03:41:23.598Z [INFO] server-585a94f2-d673-d63a-a8b1-6c03203d281c: pipelining replication: peer="{Voter server-656f43df-ae3c-b501-e919-db56755fe898 656f43df-ae3c-b501-e919-db56755fe898}" 2026-05-25T03:41:23.704Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:23.598701237 +0000 UTC m=+9.771676681 (1 nodes), 57.59124ms from start of poll, 58.04872ms from cluster start. Timeout at 2026-05-25 03:41:23.703703117 +0000 UTC m=+9.876678561, 105.00188ms after stability 2026-05-25T03:41:23.709Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:23.816Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:23.709983917 +0000 UTC m=+9.882959361 (2 nodes), 1.12µs from start of poll, 169.3314ms from cluster start. Timeout at 2026-05-25 03:41:23.814990157 +0000 UTC m=+9.987965641, 105.00628ms after stability 2026-05-25T03:41:23.816Z [INFO] server-585a94f2-d673-d63a-a8b1-6c03203d281c: updating configuration: command=RemoveServer server-id=server-585a94f2-d673-d63a-a8b1-6c03203d281c server-addr= servers="[{Suffrage:Voter ID:server-656f43df-ae3c-b501-e919-db56755fe898 Address:656f43df-ae3c-b501-e919-db56755fe898} {Suffrage:Voter ID:server-dc85223a-4c72-c2af-a338-d8cbe72fa756 Address:dc85223a-4c72-c2af-a338-d8cbe72fa756}]" 2026-05-25T03:41:23.816Z [INFO] server-585a94f2-d673-d63a-a8b1-6c03203d281c: removed ourself, shutting down 2026-05-25T03:41:23.816Z [INFO] server-585a94f2-d673-d63a-a8b1-6c03203d281c: aborting pipeline replication: peer="{Voter server-dc85223a-4c72-c2af-a338-d8cbe72fa756 dc85223a-4c72-c2af-a338-d8cbe72fa756}" 2026-05-25T03:41:23.816Z [INFO] server-585a94f2-d673-d63a-a8b1-6c03203d281c: aborting pipeline replication: peer="{Voter server-656f43df-ae3c-b501-e919-db56755fe898 656f43df-ae3c-b501-e919-db56755fe898}" 2026-05-25T03:41:23.868Z [WARN] server-dc85223a-4c72-c2af-a338-d8cbe72fa756: heartbeat timeout reached, starting election: last-leader-addr=585a94f2-d673-d63a-a8b1-6c03203d281c last-leader-id=server-585a94f2-d673-d63a-a8b1-6c03203d281c 2026-05-25T03:41:23.868Z [INFO] server-dc85223a-4c72-c2af-a338-d8cbe72fa756: entering candidate state: node="Node at dc85223a-4c72-c2af-a338-d8cbe72fa756 [Candidate]" term=3 2026-05-25T03:41:23.868Z [WARN] server-656f43df-ae3c-b501-e919-db56755fe898: rejecting vote request since we have a leader: from=dc85223a-4c72-c2af-a338-d8cbe72fa756 leader=585a94f2-d673-d63a-a8b1-6c03203d281c leader-id=server-585a94f2-d673-d63a-a8b1-6c03203d281c 2026-05-25T03:41:23.923Z [WARN] server-656f43df-ae3c-b501-e919-db56755fe898: heartbeat timeout reached, starting election: last-leader-addr=585a94f2-d673-d63a-a8b1-6c03203d281c last-leader-id=server-585a94f2-d673-d63a-a8b1-6c03203d281c 2026-05-25T03:41:23.923Z [INFO] server-656f43df-ae3c-b501-e919-db56755fe898: entering candidate state: node="Node at 656f43df-ae3c-b501-e919-db56755fe898 [Candidate]" term=3 2026-05-25T03:41:23.923Z [INFO] server-dc85223a-4c72-c2af-a338-d8cbe72fa756: duplicate requestVote for same term: term=3 2026-05-25T03:41:23.932Z [WARN] server-dc85223a-4c72-c2af-a338-d8cbe72fa756: Election timeout reached, restarting election 2026-05-25T03:41:23.933Z [INFO] server-dc85223a-4c72-c2af-a338-d8cbe72fa756: entering candidate state: node="Node at dc85223a-4c72-c2af-a338-d8cbe72fa756 [Candidate]" term=4 2026-05-25T03:41:23.933Z [INFO] server-656f43df-ae3c-b501-e919-db56755fe898: entering follower state: follower="Node at 656f43df-ae3c-b501-e919-db56755fe898 [Follower]" leader-address= leader-id= 2026-05-25T03:41:23.933Z [INFO] server-dc85223a-4c72-c2af-a338-d8cbe72fa756: election won: term=4 tally=2 2026-05-25T03:41:23.933Z [INFO] server-dc85223a-4c72-c2af-a338-d8cbe72fa756: entering leader state: leader="Node at dc85223a-4c72-c2af-a338-d8cbe72fa756 [Leader]" 2026-05-25T03:41:23.933Z [INFO] server-dc85223a-4c72-c2af-a338-d8cbe72fa756: added peer, starting replication: peer=server-656f43df-ae3c-b501-e919-db56755fe898 2026-05-25T03:41:23.933Z [INFO] server-dc85223a-4c72-c2af-a338-d8cbe72fa756: pipelining replication: peer="{Voter server-656f43df-ae3c-b501-e919-db56755fe898 656f43df-ae3c-b501-e919-db56755fe898}" 2026-05-25T03:41:24.027Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:24.133Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:24.027810477 +0000 UTC m=+10.200785921 (1 nodes), 69.56µs from start of poll, 487.15796ms from cluster start. Timeout at 2026-05-25 03:41:24.132816197 +0000 UTC m=+10.305791641, 105.00572ms after stability 2026-05-25T03:41:24.133Z [INFO] server-dc85223a-4c72-c2af-a338-d8cbe72fa756: aborting pipeline replication: peer="{Voter server-656f43df-ae3c-b501-e919-db56755fe898 656f43df-ae3c-b501-e919-db56755fe898}" --- PASS: TestRaft_RemoveLeader (0.59s) === RUN TestRaft_RemoveLeader_NoShutdown 2026-05-25T03:41:24.135Z [INFO] server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6 Address:f4c8f895-2c89-bf63-b0ad-f994db6d9ad6} {Suffrage:Voter ID:server-dda832dc-9e29-e66a-b96d-b9c3e27c0075 Address:dda832dc-9e29-e66a-b96d-b9c3e27c0075} {Suffrage:Voter ID:server-079a5605-c757-0858-a8e1-e03000b1682e Address:079a5605-c757-0858-a8e1-e03000b1682e}]" 2026-05-25T03:41:24.135Z [INFO] server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6: entering follower state: follower="Node at f4c8f895-2c89-bf63-b0ad-f994db6d9ad6 [Follower]" leader-address= leader-id= 2026-05-25T03:41:24.135Z [INFO] server-dda832dc-9e29-e66a-b96d-b9c3e27c0075: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6 Address:f4c8f895-2c89-bf63-b0ad-f994db6d9ad6} {Suffrage:Voter ID:server-dda832dc-9e29-e66a-b96d-b9c3e27c0075 Address:dda832dc-9e29-e66a-b96d-b9c3e27c0075} {Suffrage:Voter ID:server-079a5605-c757-0858-a8e1-e03000b1682e Address:079a5605-c757-0858-a8e1-e03000b1682e}]" 2026-05-25T03:41:24.135Z [INFO] server-dda832dc-9e29-e66a-b96d-b9c3e27c0075: entering follower state: follower="Node at dda832dc-9e29-e66a-b96d-b9c3e27c0075 [Follower]" leader-address= leader-id= 2026-05-25T03:41:24.135Z [INFO] server-079a5605-c757-0858-a8e1-e03000b1682e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6 Address:f4c8f895-2c89-bf63-b0ad-f994db6d9ad6} {Suffrage:Voter ID:server-dda832dc-9e29-e66a-b96d-b9c3e27c0075 Address:dda832dc-9e29-e66a-b96d-b9c3e27c0075} {Suffrage:Voter ID:server-079a5605-c757-0858-a8e1-e03000b1682e Address:079a5605-c757-0858-a8e1-e03000b1682e}]" 2026-05-25T03:41:24.135Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:24.135Z [INFO] server-079a5605-c757-0858-a8e1-e03000b1682e: entering follower state: follower="Node at 079a5605-c757-0858-a8e1-e03000b1682e [Follower]" leader-address= leader-id= 2026-05-25T03:41:24.187Z [WARN] server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:24.187Z [INFO] server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6: entering candidate state: node="Node at f4c8f895-2c89-bf63-b0ad-f994db6d9ad6 [Candidate]" term=2 2026-05-25T03:41:24.187Z [INFO] server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6: election won: term=2 tally=2 2026-05-25T03:41:24.187Z [INFO] server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6: entering leader state: leader="Node at f4c8f895-2c89-bf63-b0ad-f994db6d9ad6 [Leader]" 2026-05-25T03:41:24.187Z [INFO] server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6: added peer, starting replication: peer=server-dda832dc-9e29-e66a-b96d-b9c3e27c0075 2026-05-25T03:41:24.187Z [INFO] server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6: added peer, starting replication: peer=server-079a5605-c757-0858-a8e1-e03000b1682e 2026-05-25T03:41:24.187Z [INFO] server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6: pipelining replication: peer="{Voter server-079a5605-c757-0858-a8e1-e03000b1682e 079a5605-c757-0858-a8e1-e03000b1682e}" 2026-05-25T03:41:24.187Z [INFO] server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6: pipelining replication: peer="{Voter server-dda832dc-9e29-e66a-b96d-b9c3e27c0075 dda832dc-9e29-e66a-b96d-b9c3e27c0075}" 2026-05-25T03:41:24.293Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:24.187466197 +0000 UTC m=+10.360441641 (2 nodes), 51.69372ms from start of poll, 52.61888ms from cluster start. Timeout at 2026-05-25 03:41:24.292468557 +0000 UTC m=+10.465444001, 105.00236ms after stability 2026-05-25T03:41:24.293Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:24.399Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:24.293608437 +0000 UTC m=+10.466583881 (1 nodes), 720ns from start of poll, 158.76112ms from cluster start. Timeout at 2026-05-25 03:41:24.398615357 +0000 UTC m=+10.571590801, 105.00692ms after stability 2026-05-25T03:41:24.404Z [INFO] server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6: updating configuration: command=RemoveServer server-id=server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6 server-addr= servers="[{Suffrage:Voter ID:server-dda832dc-9e29-e66a-b96d-b9c3e27c0075 Address:dda832dc-9e29-e66a-b96d-b9c3e27c0075} {Suffrage:Voter ID:server-079a5605-c757-0858-a8e1-e03000b1682e Address:079a5605-c757-0858-a8e1-e03000b1682e}]" 2026-05-25T03:41:24.404Z [INFO] server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6: removed ourself, transitioning to follower 2026-05-25T03:41:24.404Z [INFO] server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6: entering follower state: follower="Node at f4c8f895-2c89-bf63-b0ad-f994db6d9ad6 [Follower]" leader-address= leader-id= 2026-05-25T03:41:24.404Z [INFO] server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6: aborting pipeline replication: peer="{Voter server-079a5605-c757-0858-a8e1-e03000b1682e 079a5605-c757-0858-a8e1-e03000b1682e}" 2026-05-25T03:41:24.404Z [INFO] server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6: aborting pipeline replication: peer="{Voter server-dda832dc-9e29-e66a-b96d-b9c3e27c0075 dda832dc-9e29-e66a-b96d-b9c3e27c0075}" 2026-05-25T03:41:24.460Z [WARN] server-dda832dc-9e29-e66a-b96d-b9c3e27c0075: heartbeat timeout reached, starting election: last-leader-addr=f4c8f895-2c89-bf63-b0ad-f994db6d9ad6 last-leader-id=server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6 2026-05-25T03:41:24.460Z [INFO] server-dda832dc-9e29-e66a-b96d-b9c3e27c0075: entering candidate state: node="Node at dda832dc-9e29-e66a-b96d-b9c3e27c0075 [Candidate]" term=3 2026-05-25T03:41:24.460Z [WARN] server-079a5605-c757-0858-a8e1-e03000b1682e: rejecting vote request since we have a leader: from=dda832dc-9e29-e66a-b96d-b9c3e27c0075 leader=f4c8f895-2c89-bf63-b0ad-f994db6d9ad6 leader-id=server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6 2026-05-25T03:41:24.466Z [WARN] server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6: not part of stable configuration, aborting election 2026-05-25T03:41:24.480Z [WARN] server-079a5605-c757-0858-a8e1-e03000b1682e: heartbeat timeout reached, starting election: last-leader-addr=f4c8f895-2c89-bf63-b0ad-f994db6d9ad6 last-leader-id=server-f4c8f895-2c89-bf63-b0ad-f994db6d9ad6 2026-05-25T03:41:24.480Z [INFO] server-079a5605-c757-0858-a8e1-e03000b1682e: entering candidate state: node="Node at 079a5605-c757-0858-a8e1-e03000b1682e [Candidate]" term=3 2026-05-25T03:41:24.480Z [INFO] server-dda832dc-9e29-e66a-b96d-b9c3e27c0075: duplicate requestVote for same term: term=3 2026-05-25T03:41:24.511Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:24.539Z [WARN] server-dda832dc-9e29-e66a-b96d-b9c3e27c0075: Election timeout reached, restarting election 2026-05-25T03:41:24.539Z [INFO] server-dda832dc-9e29-e66a-b96d-b9c3e27c0075: entering candidate state: node="Node at dda832dc-9e29-e66a-b96d-b9c3e27c0075 [Candidate]" term=4 2026-05-25T03:41:24.539Z [INFO] server-079a5605-c757-0858-a8e1-e03000b1682e: entering follower state: follower="Node at 079a5605-c757-0858-a8e1-e03000b1682e [Follower]" leader-address= leader-id= 2026-05-25T03:41:24.539Z [INFO] server-dda832dc-9e29-e66a-b96d-b9c3e27c0075: election won: term=4 tally=2 2026-05-25T03:41:24.539Z [INFO] server-dda832dc-9e29-e66a-b96d-b9c3e27c0075: entering leader state: leader="Node at dda832dc-9e29-e66a-b96d-b9c3e27c0075 [Leader]" 2026-05-25T03:41:24.539Z [INFO] server-dda832dc-9e29-e66a-b96d-b9c3e27c0075: added peer, starting replication: peer=server-079a5605-c757-0858-a8e1-e03000b1682e 2026-05-25T03:41:24.539Z [INFO] server-dda832dc-9e29-e66a-b96d-b9c3e27c0075: pipelining replication: peer="{Voter server-079a5605-c757-0858-a8e1-e03000b1682e 079a5605-c757-0858-a8e1-e03000b1682e}" 2026-05-25T03:41:24.645Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:24.539869637 +0000 UTC m=+10.712845081 (1 nodes), 28.53072ms from start of poll, 405.02232ms from cluster start. Timeout at 2026-05-25 03:41:24.644873717 +0000 UTC m=+10.817849161, 105.00408ms after stability 2026-05-25T03:41:24.751Z [INFO] server-dda832dc-9e29-e66a-b96d-b9c3e27c0075: aborting pipeline replication: peer="{Voter server-079a5605-c757-0858-a8e1-e03000b1682e 079a5605-c757-0858-a8e1-e03000b1682e}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.62s) === RUN TestRaft_RemoveFollower_SplitCluster 2026-05-25T03:41:24.751Z [INFO] server-dc9b0a74-d2b5-de59-d2cd-ca5f24908777: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc9b0a74-d2b5-de59-d2cd-ca5f24908777 Address:dc9b0a74-d2b5-de59-d2cd-ca5f24908777} {Suffrage:Voter ID:server-0313d3b4-6073-6fc0-5d85-2f52a6c32d6c Address:0313d3b4-6073-6fc0-5d85-2f52a6c32d6c} {Suffrage:Voter ID:server-a3bb32f8-ea2a-f897-da14-51ae1a882fc4 Address:a3bb32f8-ea2a-f897-da14-51ae1a882fc4} {Suffrage:Voter ID:server-10f1dd46-0ddf-4ee0-6121-65a0f0613285 Address:10f1dd46-0ddf-4ee0-6121-65a0f0613285}]" 2026-05-25T03:41:24.752Z [INFO] server-0313d3b4-6073-6fc0-5d85-2f52a6c32d6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc9b0a74-d2b5-de59-d2cd-ca5f24908777 Address:dc9b0a74-d2b5-de59-d2cd-ca5f24908777} {Suffrage:Voter ID:server-0313d3b4-6073-6fc0-5d85-2f52a6c32d6c Address:0313d3b4-6073-6fc0-5d85-2f52a6c32d6c} {Suffrage:Voter ID:server-a3bb32f8-ea2a-f897-da14-51ae1a882fc4 Address:a3bb32f8-ea2a-f897-da14-51ae1a882fc4} {Suffrage:Voter ID:server-10f1dd46-0ddf-4ee0-6121-65a0f0613285 Address:10f1dd46-0ddf-4ee0-6121-65a0f0613285}]" 2026-05-25T03:41:24.752Z [INFO] server-dc9b0a74-d2b5-de59-d2cd-ca5f24908777: entering follower state: follower="Node at dc9b0a74-d2b5-de59-d2cd-ca5f24908777 [Follower]" leader-address= leader-id= 2026-05-25T03:41:24.752Z [INFO] server-a3bb32f8-ea2a-f897-da14-51ae1a882fc4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc9b0a74-d2b5-de59-d2cd-ca5f24908777 Address:dc9b0a74-d2b5-de59-d2cd-ca5f24908777} {Suffrage:Voter ID:server-0313d3b4-6073-6fc0-5d85-2f52a6c32d6c Address:0313d3b4-6073-6fc0-5d85-2f52a6c32d6c} {Suffrage:Voter ID:server-a3bb32f8-ea2a-f897-da14-51ae1a882fc4 Address:a3bb32f8-ea2a-f897-da14-51ae1a882fc4} {Suffrage:Voter ID:server-10f1dd46-0ddf-4ee0-6121-65a0f0613285 Address:10f1dd46-0ddf-4ee0-6121-65a0f0613285}]" 2026-05-25T03:41:24.752Z [INFO] server-a3bb32f8-ea2a-f897-da14-51ae1a882fc4: entering follower state: follower="Node at a3bb32f8-ea2a-f897-da14-51ae1a882fc4 [Follower]" leader-address= leader-id= 2026-05-25T03:41:24.752Z [INFO] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc9b0a74-d2b5-de59-d2cd-ca5f24908777 Address:dc9b0a74-d2b5-de59-d2cd-ca5f24908777} {Suffrage:Voter ID:server-0313d3b4-6073-6fc0-5d85-2f52a6c32d6c Address:0313d3b4-6073-6fc0-5d85-2f52a6c32d6c} {Suffrage:Voter ID:server-a3bb32f8-ea2a-f897-da14-51ae1a882fc4 Address:a3bb32f8-ea2a-f897-da14-51ae1a882fc4} {Suffrage:Voter ID:server-10f1dd46-0ddf-4ee0-6121-65a0f0613285 Address:10f1dd46-0ddf-4ee0-6121-65a0f0613285}]" 2026-05-25T03:41:24.752Z [INFO] server-0313d3b4-6073-6fc0-5d85-2f52a6c32d6c: entering follower state: follower="Node at 0313d3b4-6073-6fc0-5d85-2f52a6c32d6c [Follower]" leader-address= leader-id= 2026-05-25T03:41:24.752Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:24.752Z [INFO] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: entering follower state: follower="Node at 10f1dd46-0ddf-4ee0-6121-65a0f0613285 [Follower]" leader-address= leader-id= 2026-05-25T03:41:24.832Z [WARN] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:24.832Z [INFO] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: entering candidate state: node="Node at 10f1dd46-0ddf-4ee0-6121-65a0f0613285 [Candidate]" term=2 2026-05-25T03:41:24.832Z [INFO] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: election won: term=2 tally=3 2026-05-25T03:41:24.832Z [INFO] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: entering leader state: leader="Node at 10f1dd46-0ddf-4ee0-6121-65a0f0613285 [Leader]" 2026-05-25T03:41:24.832Z [INFO] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: added peer, starting replication: peer=server-dc9b0a74-d2b5-de59-d2cd-ca5f24908777 2026-05-25T03:41:24.832Z [INFO] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: added peer, starting replication: peer=server-0313d3b4-6073-6fc0-5d85-2f52a6c32d6c 2026-05-25T03:41:24.832Z [INFO] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: added peer, starting replication: peer=server-a3bb32f8-ea2a-f897-da14-51ae1a882fc4 2026-05-25T03:41:24.832Z [INFO] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: pipelining replication: peer="{Voter server-dc9b0a74-d2b5-de59-d2cd-ca5f24908777 dc9b0a74-d2b5-de59-d2cd-ca5f24908777}" 2026-05-25T03:41:24.832Z [INFO] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: pipelining replication: peer="{Voter server-0313d3b4-6073-6fc0-5d85-2f52a6c32d6c 0313d3b4-6073-6fc0-5d85-2f52a6c32d6c}" 2026-05-25T03:41:24.832Z [INFO] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: pipelining replication: peer="{Voter server-a3bb32f8-ea2a-f897-da14-51ae1a882fc4 a3bb32f8-ea2a-f897-da14-51ae1a882fc4}" 2026-05-25T03:41:24.938Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:24.832650077 +0000 UTC m=+11.005625521 (1 nodes), 80.29144ms from start of poll, 80.81448ms from cluster start. Timeout at 2026-05-25 03:41:24.937653717 +0000 UTC m=+11.110629161, 105.00364ms after stability 2026-05-25T03:41:25.043Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:25.149Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:25.043894677 +0000 UTC m=+11.216870121 (3 nodes), 3.32µs from start of poll, 292.05908ms from cluster start. Timeout at 2026-05-25 03:41:25.148902317 +0000 UTC m=+11.321877761, 105.00764ms after stability 2026-05-25T03:41:25.149Z [INFO] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: updating configuration: command=RemoveServer server-id=server-a3bb32f8-ea2a-f897-da14-51ae1a882fc4 server-addr= servers="[{Suffrage:Voter ID:server-dc9b0a74-d2b5-de59-d2cd-ca5f24908777 Address:dc9b0a74-d2b5-de59-d2cd-ca5f24908777} {Suffrage:Voter ID:server-0313d3b4-6073-6fc0-5d85-2f52a6c32d6c Address:0313d3b4-6073-6fc0-5d85-2f52a6c32d6c} {Suffrage:Voter ID:server-10f1dd46-0ddf-4ee0-6121-65a0f0613285 Address:10f1dd46-0ddf-4ee0-6121-65a0f0613285}]" 2026-05-25T03:41:25.149Z [INFO] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: removed peer, stopping replication: peer=server-a3bb32f8-ea2a-f897-da14-51ae1a882fc4 last-index=3 2026-05-25T03:41:25.149Z [INFO] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: aborting pipeline replication: peer="{Voter server-a3bb32f8-ea2a-f897-da14-51ae1a882fc4 a3bb32f8-ea2a-f897-da14-51ae1a882fc4}" 2026-05-25T03:41:25.149Z [ERROR] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to pipeline appendEntries: peer="{Voter server-dc9b0a74-d2b5-de59-d2cd-ca5f24908777 dc9b0a74-d2b5-de59-d2cd-ca5f24908777}" error="append pipeline closed" 2026-05-25T03:41:25.149Z [INFO] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: aborting pipeline replication: peer="{Voter server-dc9b0a74-d2b5-de59-d2cd-ca5f24908777 dc9b0a74-d2b5-de59-d2cd-ca5f24908777}" 2026-05-25T03:41:25.149Z [ERROR] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to pipeline appendEntries: peer="{Voter server-0313d3b4-6073-6fc0-5d85-2f52a6c32d6c 0313d3b4-6073-6fc0-5d85-2f52a6c32d6c}" error="append pipeline closed" 2026-05-25T03:41:25.149Z [INFO] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: aborting pipeline replication: peer="{Voter server-0313d3b4-6073-6fc0-5d85-2f52a6c32d6c 0313d3b4-6073-6fc0-5d85-2f52a6c32d6c}" 2026-05-25T03:41:25.151Z [ERROR] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to heartbeat to: peer=dc9b0a74-d2b5-de59-d2cd-ca5f24908777 backoff time=10ms error="failed to connect to peer: dc9b0a74-d2b5-de59-d2cd-ca5f24908777" 2026-05-25T03:41:25.155Z [ERROR] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to heartbeat to: peer=0313d3b4-6073-6fc0-5d85-2f52a6c32d6c backoff time=10ms error="failed to connect to peer: 0313d3b4-6073-6fc0-5d85-2f52a6c32d6c" 2026-05-25T03:41:25.157Z [ERROR] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to appendEntries to: peer="{Voter server-0313d3b4-6073-6fc0-5d85-2f52a6c32d6c 0313d3b4-6073-6fc0-5d85-2f52a6c32d6c}" error="failed to connect to peer: 0313d3b4-6073-6fc0-5d85-2f52a6c32d6c" 2026-05-25T03:41:25.158Z [ERROR] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to appendEntries to: peer="{Voter server-dc9b0a74-d2b5-de59-d2cd-ca5f24908777 dc9b0a74-d2b5-de59-d2cd-ca5f24908777}" error="failed to connect to peer: dc9b0a74-d2b5-de59-d2cd-ca5f24908777" 2026-05-25T03:41:25.171Z [ERROR] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to heartbeat to: peer=0313d3b4-6073-6fc0-5d85-2f52a6c32d6c backoff time=10ms error="failed to connect to peer: 0313d3b4-6073-6fc0-5d85-2f52a6c32d6c" 2026-05-25T03:41:25.171Z [ERROR] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to heartbeat to: peer=dc9b0a74-d2b5-de59-d2cd-ca5f24908777 backoff time=10ms error="failed to connect to peer: dc9b0a74-d2b5-de59-d2cd-ca5f24908777" 2026-05-25T03:41:25.175Z [ERROR] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to appendEntries to: peer="{Voter server-dc9b0a74-d2b5-de59-d2cd-ca5f24908777 dc9b0a74-d2b5-de59-d2cd-ca5f24908777}" error="failed to connect to peer: dc9b0a74-d2b5-de59-d2cd-ca5f24908777" 2026-05-25T03:41:25.175Z [ERROR] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to appendEntries to: peer="{Voter server-0313d3b4-6073-6fc0-5d85-2f52a6c32d6c 0313d3b4-6073-6fc0-5d85-2f52a6c32d6c}" error="failed to connect to peer: 0313d3b4-6073-6fc0-5d85-2f52a6c32d6c" 2026-05-25T03:41:25.188Z [ERROR] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to heartbeat to: peer=dc9b0a74-d2b5-de59-d2cd-ca5f24908777 backoff time=10ms error="failed to connect to peer: dc9b0a74-d2b5-de59-d2cd-ca5f24908777" 2026-05-25T03:41:25.189Z [ERROR] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to heartbeat to: peer=0313d3b4-6073-6fc0-5d85-2f52a6c32d6c backoff time=10ms error="failed to connect to peer: 0313d3b4-6073-6fc0-5d85-2f52a6c32d6c" 2026-05-25T03:41:25.194Z [ERROR] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to appendEntries to: peer="{Voter server-dc9b0a74-d2b5-de59-d2cd-ca5f24908777 dc9b0a74-d2b5-de59-d2cd-ca5f24908777}" error="failed to connect to peer: dc9b0a74-d2b5-de59-d2cd-ca5f24908777" 2026-05-25T03:41:25.194Z [ERROR] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to appendEntries to: peer="{Voter server-0313d3b4-6073-6fc0-5d85-2f52a6c32d6c 0313d3b4-6073-6fc0-5d85-2f52a6c32d6c}" error="failed to connect to peer: 0313d3b4-6073-6fc0-5d85-2f52a6c32d6c" 2026-05-25T03:41:25.195Z [WARN] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to contact: server-id=server-dc9b0a74-d2b5-de59-d2cd-ca5f24908777 time=50.404ms 2026-05-25T03:41:25.200Z [WARN] server-a3bb32f8-ea2a-f897-da14-51ae1a882fc4: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2026-05-25T03:41:25.205Z [ERROR] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to heartbeat to: peer=dc9b0a74-d2b5-de59-d2cd-ca5f24908777 backoff time=20ms error="failed to connect to peer: dc9b0a74-d2b5-de59-d2cd-ca5f24908777" 2026-05-25T03:41:25.206Z [WARN] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to contact: server-id=server-dc9b0a74-d2b5-de59-d2cd-ca5f24908777 time=61.39216ms 2026-05-25T03:41:25.206Z [WARN] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to contact: server-id=server-0313d3b4-6073-6fc0-5d85-2f52a6c32d6c time=56.98784ms 2026-05-25T03:41:25.206Z [WARN] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to contact quorum of nodes, stepping down 2026-05-25T03:41:25.206Z [INFO] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: entering follower state: follower="Node at 10f1dd46-0ddf-4ee0-6121-65a0f0613285 [Follower]" leader-address= leader-id= 2026-05-25T03:41:25.206Z [ERROR] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to appendEntries to: peer="{Voter server-0313d3b4-6073-6fc0-5d85-2f52a6c32d6c 0313d3b4-6073-6fc0-5d85-2f52a6c32d6c}" error="failed to connect to peer: 0313d3b4-6073-6fc0-5d85-2f52a6c32d6c" 2026-05-25T03:41:25.206Z [ERROR] server-10f1dd46-0ddf-4ee0-6121-65a0f0613285: failed to appendEntries to: peer="{Voter server-dc9b0a74-d2b5-de59-d2cd-ca5f24908777 dc9b0a74-d2b5-de59-d2cd-ca5f24908777}" error="failed to connect to peer: dc9b0a74-d2b5-de59-d2cd-ca5f24908777" --- PASS: TestRaft_RemoveFollower_SplitCluster (0.46s) === RUN TestRaft_AddKnownPeer 2026-05-25T03:41:25.207Z [INFO] server-5fdbc844-1164-e58c-ef43-71adcc904adb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fdbc844-1164-e58c-ef43-71adcc904adb Address:5fdbc844-1164-e58c-ef43-71adcc904adb} {Suffrage:Voter ID:server-10757a95-99d3-7bde-7134-b70bab703761 Address:10757a95-99d3-7bde-7134-b70bab703761} {Suffrage:Voter ID:server-788bffb3-04be-9146-8096-4e0743cdf855 Address:788bffb3-04be-9146-8096-4e0743cdf855}]" 2026-05-25T03:41:25.207Z [INFO] server-10757a95-99d3-7bde-7134-b70bab703761: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fdbc844-1164-e58c-ef43-71adcc904adb Address:5fdbc844-1164-e58c-ef43-71adcc904adb} {Suffrage:Voter ID:server-10757a95-99d3-7bde-7134-b70bab703761 Address:10757a95-99d3-7bde-7134-b70bab703761} {Suffrage:Voter ID:server-788bffb3-04be-9146-8096-4e0743cdf855 Address:788bffb3-04be-9146-8096-4e0743cdf855}]" 2026-05-25T03:41:25.207Z [INFO] server-5fdbc844-1164-e58c-ef43-71adcc904adb: entering follower state: follower="Node at 5fdbc844-1164-e58c-ef43-71adcc904adb [Follower]" leader-address= leader-id= 2026-05-25T03:41:25.207Z [INFO] server-10757a95-99d3-7bde-7134-b70bab703761: entering follower state: follower="Node at 10757a95-99d3-7bde-7134-b70bab703761 [Follower]" leader-address= leader-id= 2026-05-25T03:41:25.207Z [INFO] server-788bffb3-04be-9146-8096-4e0743cdf855: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fdbc844-1164-e58c-ef43-71adcc904adb Address:5fdbc844-1164-e58c-ef43-71adcc904adb} {Suffrage:Voter ID:server-10757a95-99d3-7bde-7134-b70bab703761 Address:10757a95-99d3-7bde-7134-b70bab703761} {Suffrage:Voter ID:server-788bffb3-04be-9146-8096-4e0743cdf855 Address:788bffb3-04be-9146-8096-4e0743cdf855}]" 2026-05-25T03:41:25.207Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:25.207Z [INFO] server-788bffb3-04be-9146-8096-4e0743cdf855: entering follower state: follower="Node at 788bffb3-04be-9146-8096-4e0743cdf855 [Follower]" leader-address= leader-id= 2026-05-25T03:41:25.284Z [WARN] server-5fdbc844-1164-e58c-ef43-71adcc904adb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:25.284Z [INFO] server-5fdbc844-1164-e58c-ef43-71adcc904adb: entering candidate state: node="Node at 5fdbc844-1164-e58c-ef43-71adcc904adb [Candidate]" term=2 2026-05-25T03:41:25.284Z [INFO] server-5fdbc844-1164-e58c-ef43-71adcc904adb: election won: term=2 tally=2 2026-05-25T03:41:25.284Z [INFO] server-5fdbc844-1164-e58c-ef43-71adcc904adb: entering leader state: leader="Node at 5fdbc844-1164-e58c-ef43-71adcc904adb [Leader]" 2026-05-25T03:41:25.284Z [INFO] server-5fdbc844-1164-e58c-ef43-71adcc904adb: added peer, starting replication: peer=server-10757a95-99d3-7bde-7134-b70bab703761 2026-05-25T03:41:25.284Z [INFO] server-5fdbc844-1164-e58c-ef43-71adcc904adb: added peer, starting replication: peer=server-788bffb3-04be-9146-8096-4e0743cdf855 2026-05-25T03:41:25.284Z [INFO] server-5fdbc844-1164-e58c-ef43-71adcc904adb: pipelining replication: peer="{Voter server-788bffb3-04be-9146-8096-4e0743cdf855 788bffb3-04be-9146-8096-4e0743cdf855}" 2026-05-25T03:41:25.284Z [INFO] server-5fdbc844-1164-e58c-ef43-71adcc904adb: pipelining replication: peer="{Voter server-10757a95-99d3-7bde-7134-b70bab703761 10757a95-99d3-7bde-7134-b70bab703761}" 2026-05-25T03:41:25.390Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:25.284699957 +0000 UTC m=+11.457675401 (1 nodes), 76.75396ms from start of poll, 77.21548ms from cluster start. Timeout at 2026-05-25 03:41:25.389712037 +0000 UTC m=+11.562687481, 105.01208ms after stability 2026-05-25T03:41:25.390Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:25.496Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:25.390441877 +0000 UTC m=+11.563417321 (2 nodes), 3.32µs from start of poll, 182.9574ms from cluster start. Timeout at 2026-05-25 03:41:25.495450357 +0000 UTC m=+11.668425801, 105.00848ms after stability 2026-05-25T03:41:25.496Z [INFO] server-5fdbc844-1164-e58c-ef43-71adcc904adb: updating configuration: command=AddVoter server-id=server-10757a95-99d3-7bde-7134-b70bab703761 server-addr=10757a95-99d3-7bde-7134-b70bab703761 servers="[{Suffrage:Voter ID:server-5fdbc844-1164-e58c-ef43-71adcc904adb Address:5fdbc844-1164-e58c-ef43-71adcc904adb} {Suffrage:Voter ID:server-10757a95-99d3-7bde-7134-b70bab703761 Address:10757a95-99d3-7bde-7134-b70bab703761} {Suffrage:Voter ID:server-788bffb3-04be-9146-8096-4e0743cdf855 Address:788bffb3-04be-9146-8096-4e0743cdf855}]" 2026-05-25T03:41:25.496Z [INFO] server-5fdbc844-1164-e58c-ef43-71adcc904adb: aborting pipeline replication: peer="{Voter server-788bffb3-04be-9146-8096-4e0743cdf855 788bffb3-04be-9146-8096-4e0743cdf855}" 2026-05-25T03:41:25.496Z [INFO] server-5fdbc844-1164-e58c-ef43-71adcc904adb: aborting pipeline replication: peer="{Voter server-10757a95-99d3-7bde-7134-b70bab703761 10757a95-99d3-7bde-7134-b70bab703761}" --- PASS: TestRaft_AddKnownPeer (0.29s) === RUN TestRaft_RemoveUnknownPeer 2026-05-25T03:41:25.497Z [INFO] server-29ec65f1-2f10-1658-7e99-3a60e3ff9954: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29ec65f1-2f10-1658-7e99-3a60e3ff9954 Address:29ec65f1-2f10-1658-7e99-3a60e3ff9954} {Suffrage:Voter ID:server-87400c7b-f801-beb7-4084-779de61fe92e Address:87400c7b-f801-beb7-4084-779de61fe92e} {Suffrage:Voter ID:server-f489e6aa-37b8-dd32-b857-0b6a8f220efe Address:f489e6aa-37b8-dd32-b857-0b6a8f220efe}]" 2026-05-25T03:41:25.497Z [INFO] server-87400c7b-f801-beb7-4084-779de61fe92e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29ec65f1-2f10-1658-7e99-3a60e3ff9954 Address:29ec65f1-2f10-1658-7e99-3a60e3ff9954} {Suffrage:Voter ID:server-87400c7b-f801-beb7-4084-779de61fe92e Address:87400c7b-f801-beb7-4084-779de61fe92e} {Suffrage:Voter ID:server-f489e6aa-37b8-dd32-b857-0b6a8f220efe Address:f489e6aa-37b8-dd32-b857-0b6a8f220efe}]" 2026-05-25T03:41:25.497Z [INFO] server-29ec65f1-2f10-1658-7e99-3a60e3ff9954: entering follower state: follower="Node at 29ec65f1-2f10-1658-7e99-3a60e3ff9954 [Follower]" leader-address= leader-id= 2026-05-25T03:41:25.497Z [INFO] server-f489e6aa-37b8-dd32-b857-0b6a8f220efe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29ec65f1-2f10-1658-7e99-3a60e3ff9954 Address:29ec65f1-2f10-1658-7e99-3a60e3ff9954} {Suffrage:Voter ID:server-87400c7b-f801-beb7-4084-779de61fe92e Address:87400c7b-f801-beb7-4084-779de61fe92e} {Suffrage:Voter ID:server-f489e6aa-37b8-dd32-b857-0b6a8f220efe Address:f489e6aa-37b8-dd32-b857-0b6a8f220efe}]" 2026-05-25T03:41:25.497Z [INFO] server-87400c7b-f801-beb7-4084-779de61fe92e: entering follower state: follower="Node at 87400c7b-f801-beb7-4084-779de61fe92e [Follower]" leader-address= leader-id= 2026-05-25T03:41:25.497Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:25.497Z [INFO] server-f489e6aa-37b8-dd32-b857-0b6a8f220efe: entering follower state: follower="Node at f489e6aa-37b8-dd32-b857-0b6a8f220efe [Follower]" leader-address= leader-id= 2026-05-25T03:41:25.564Z [WARN] server-87400c7b-f801-beb7-4084-779de61fe92e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:25.564Z [INFO] server-87400c7b-f801-beb7-4084-779de61fe92e: entering candidate state: node="Node at 87400c7b-f801-beb7-4084-779de61fe92e [Candidate]" term=2 2026-05-25T03:41:25.564Z [INFO] server-87400c7b-f801-beb7-4084-779de61fe92e: election won: term=2 tally=2 2026-05-25T03:41:25.565Z [INFO] server-87400c7b-f801-beb7-4084-779de61fe92e: entering leader state: leader="Node at 87400c7b-f801-beb7-4084-779de61fe92e [Leader]" 2026-05-25T03:41:25.565Z [INFO] server-87400c7b-f801-beb7-4084-779de61fe92e: added peer, starting replication: peer=server-29ec65f1-2f10-1658-7e99-3a60e3ff9954 2026-05-25T03:41:25.565Z [INFO] server-87400c7b-f801-beb7-4084-779de61fe92e: added peer, starting replication: peer=server-f489e6aa-37b8-dd32-b857-0b6a8f220efe 2026-05-25T03:41:25.565Z [INFO] server-87400c7b-f801-beb7-4084-779de61fe92e: pipelining replication: peer="{Voter server-f489e6aa-37b8-dd32-b857-0b6a8f220efe f489e6aa-37b8-dd32-b857-0b6a8f220efe}" 2026-05-25T03:41:25.565Z [INFO] server-87400c7b-f801-beb7-4084-779de61fe92e: pipelining replication: peer="{Voter server-29ec65f1-2f10-1658-7e99-3a60e3ff9954 29ec65f1-2f10-1658-7e99-3a60e3ff9954}" 2026-05-25T03:41:25.670Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:25.565036677 +0000 UTC m=+11.738012121 (1 nodes), 67.22708ms from start of poll, 67.65936ms from cluster start. Timeout at 2026-05-25 03:41:25.670038517 +0000 UTC m=+11.843013961, 105.00184ms after stability 2026-05-25T03:41:25.670Z [INFO] server-87400c7b-f801-beb7-4084-779de61fe92e: updating configuration: command=RemoveServer server-id=b3d83cfb-0580-d46c-fe84-8b6a06fa0121 server-addr= servers="[{Suffrage:Voter ID:server-29ec65f1-2f10-1658-7e99-3a60e3ff9954 Address:29ec65f1-2f10-1658-7e99-3a60e3ff9954} {Suffrage:Voter ID:server-87400c7b-f801-beb7-4084-779de61fe92e Address:87400c7b-f801-beb7-4084-779de61fe92e} {Suffrage:Voter ID:server-f489e6aa-37b8-dd32-b857-0b6a8f220efe Address:f489e6aa-37b8-dd32-b857-0b6a8f220efe}]" 2026-05-25T03:41:25.670Z [INFO] server-87400c7b-f801-beb7-4084-779de61fe92e: aborting pipeline replication: peer="{Voter server-f489e6aa-37b8-dd32-b857-0b6a8f220efe f489e6aa-37b8-dd32-b857-0b6a8f220efe}" 2026-05-25T03:41:25.670Z [INFO] server-87400c7b-f801-beb7-4084-779de61fe92e: aborting pipeline replication: peer="{Voter server-29ec65f1-2f10-1658-7e99-3a60e3ff9954 29ec65f1-2f10-1658-7e99-3a60e3ff9954}" --- PASS: TestRaft_RemoveUnknownPeer (0.17s) === RUN TestRaft_SnapshotRestore 2026-05-25T03:41:25.671Z [INFO] server-130bf8ba-4166-1f91-ccb4-fc75d431e51f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-130bf8ba-4166-1f91-ccb4-fc75d431e51f Address:130bf8ba-4166-1f91-ccb4-fc75d431e51f}]" 2026-05-25T03:41:25.671Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:25.671Z [INFO] server-130bf8ba-4166-1f91-ccb4-fc75d431e51f: entering follower state: follower="Node at 130bf8ba-4166-1f91-ccb4-fc75d431e51f [Follower]" leader-address= leader-id= 2026-05-25T03:41:25.765Z [WARN] server-130bf8ba-4166-1f91-ccb4-fc75d431e51f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:25.765Z [INFO] server-130bf8ba-4166-1f91-ccb4-fc75d431e51f: entering candidate state: node="Node at 130bf8ba-4166-1f91-ccb4-fc75d431e51f [Candidate]" term=2 2026-05-25T03:41:25.765Z [INFO] server-130bf8ba-4166-1f91-ccb4-fc75d431e51f: election won: term=2 tally=1 2026-05-25T03:41:25.765Z [INFO] server-130bf8ba-4166-1f91-ccb4-fc75d431e51f: entering leader state: leader="Node at 130bf8ba-4166-1f91-ccb4-fc75d431e51f [Leader]" 2026-05-25T03:41:25.871Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:25.765974077 +0000 UTC m=+11.938949561 (1 nodes), 94.47012ms from start of poll, 94.66372ms from cluster start. Timeout at 2026-05-25 03:41:25.870975997 +0000 UTC m=+12.043951441, 105.00188ms after stability 2026-05-25T03:41:25.874Z [INFO] server-130bf8ba-4166-1f91-ccb4-fc75d431e51f: starting snapshot up to: index=102 2026-05-25T03:41:25.874Z [INFO] creating new snapshot: path=/tmp/raft1270033305/snapshots/2-102-1779680485874.tmp 2026-05-25T03:41:25.875Z [INFO] server-130bf8ba-4166-1f91-ccb4-fc75d431e51f: compacting logs: from=1 to=92 2026-05-25T03:41:25.875Z [INFO] server-130bf8ba-4166-1f91-ccb4-fc75d431e51f: snapshot complete up to: index=102 2026-05-25T03:41:25.875Z [INFO] server-130bf8ba-4166-1f91-ccb4-fc75d431e51f: starting restore from snapshot: id=2-102-1779680485874 last-index=102 last-term=2 size-in-bytes=693 2026-05-25T03:41:25.875Z [INFO] server-130bf8ba-4166-1f91-ccb4-fc75d431e51f: snapshot restore progress: id=2-102-1779680485874 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2026-05-25T03:41:25.875Z [INFO] server-130bf8ba-4166-1f91-ccb4-fc75d431e51f: restored from snapshot: id=2-102-1779680485874 last-index=102 last-term=2 size-in-bytes=693 2026-05-25T03:41:25.875Z [INFO] server-130bf8ba-4166-1f91-ccb4-fc75d431e51f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-130bf8ba-4166-1f91-ccb4-fc75d431e51f Address:130bf8ba-4166-1f91-ccb4-fc75d431e51f}]" --- PASS: TestRaft_SnapshotRestore (0.20s) === RUN TestRaft_RestoreSnapshotOnStartup_Monotonic 2026-05-25T03:41:25.876Z [INFO] server-6b7fd4eb-757d-fc6c-804a-bb84a54e17dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b7fd4eb-757d-fc6c-804a-bb84a54e17dc Address:6b7fd4eb-757d-fc6c-804a-bb84a54e17dc}]" 2026-05-25T03:41:25.876Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:25.876Z [INFO] server-6b7fd4eb-757d-fc6c-804a-bb84a54e17dc: entering follower state: follower="Node at 6b7fd4eb-757d-fc6c-804a-bb84a54e17dc [Follower]" leader-address= leader-id= 2026-05-25T03:41:25.977Z [WARN] server-6b7fd4eb-757d-fc6c-804a-bb84a54e17dc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:25.977Z [INFO] server-6b7fd4eb-757d-fc6c-804a-bb84a54e17dc: entering candidate state: node="Node at 6b7fd4eb-757d-fc6c-804a-bb84a54e17dc [Candidate]" term=2 2026-05-25T03:41:25.977Z [INFO] server-6b7fd4eb-757d-fc6c-804a-bb84a54e17dc: election won: term=2 tally=1 2026-05-25T03:41:25.977Z [INFO] server-6b7fd4eb-757d-fc6c-804a-bb84a54e17dc: entering leader state: leader="Node at 6b7fd4eb-757d-fc6c-804a-bb84a54e17dc [Leader]" 2026-05-25T03:41:26.083Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:25.977346477 +0000 UTC m=+12.150321921 (1 nodes), 100.9578ms from start of poll, 101.15056ms from cluster start. Timeout at 2026-05-25 03:41:26.082348677 +0000 UTC m=+12.255324121, 105.0022ms after stability 2026-05-25T03:41:26.086Z [INFO] server-6b7fd4eb-757d-fc6c-804a-bb84a54e17dc: starting snapshot up to: index=102 2026-05-25T03:41:26.086Z [INFO] creating new snapshot: path=/tmp/raft2210881179/snapshots/2-102-1779680486086.tmp 2026-05-25T03:41:26.087Z [INFO] server-6b7fd4eb-757d-fc6c-804a-bb84a54e17dc: compacting logs: from=1 to=92 2026-05-25T03:41:26.087Z [INFO] server-6b7fd4eb-757d-fc6c-804a-bb84a54e17dc: snapshot complete up to: index=102 2026-05-25T03:41:26.087Z [INFO] server-6b7fd4eb-757d-fc6c-804a-bb84a54e17dc: starting restore from snapshot: id=2-102-1779680486086 last-index=102 last-term=2 size-in-bytes=693 2026-05-25T03:41:26.087Z [INFO] server-6b7fd4eb-757d-fc6c-804a-bb84a54e17dc: snapshot restore progress: id=2-102-1779680486086 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2026-05-25T03:41:26.087Z [INFO] server-6b7fd4eb-757d-fc6c-804a-bb84a54e17dc: restored from snapshot: id=2-102-1779680486086 last-index=102 last-term=2 size-in-bytes=693 2026-05-25T03:41:26.087Z [INFO] server-6b7fd4eb-757d-fc6c-804a-bb84a54e17dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b7fd4eb-757d-fc6c-804a-bb84a54e17dc Address:6b7fd4eb-757d-fc6c-804a-bb84a54e17dc}]" --- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.21s) === RUN TestRaft_SnapshotRestore_Progress 2026-05-25T03:41:26.088Z [INFO] server-6cc8ac45-bec0-ba1f-70b9-0384854975b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6cc8ac45-bec0-ba1f-70b9-0384854975b3 Address:6cc8ac45-bec0-ba1f-70b9-0384854975b3}]" 2026-05-25T03:41:26.088Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:26.088Z [INFO] server-6cc8ac45-bec0-ba1f-70b9-0384854975b3: entering follower state: follower="Node at 6cc8ac45-bec0-ba1f-70b9-0384854975b3 [Follower]" leader-address= leader-id= 2026-05-25T03:41:26.155Z [WARN] server-6cc8ac45-bec0-ba1f-70b9-0384854975b3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:26.155Z [INFO] server-6cc8ac45-bec0-ba1f-70b9-0384854975b3: entering candidate state: node="Node at 6cc8ac45-bec0-ba1f-70b9-0384854975b3 [Candidate]" term=2 2026-05-25T03:41:26.155Z [INFO] server-6cc8ac45-bec0-ba1f-70b9-0384854975b3: election won: term=2 tally=1 2026-05-25T03:41:26.155Z [INFO] server-6cc8ac45-bec0-ba1f-70b9-0384854975b3: entering leader state: leader="Node at 6cc8ac45-bec0-ba1f-70b9-0384854975b3 [Leader]" 2026-05-25T03:41:26.261Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:26.156017637 +0000 UTC m=+12.328993081 (1 nodes), 67.646ms from start of poll, 67.8222ms from cluster start. Timeout at 2026-05-25 03:41:26.261019597 +0000 UTC m=+12.433995041, 105.00196ms after stability 2026-05-25T03:41:26.264Z [INFO] server-6cc8ac45-bec0-ba1f-70b9-0384854975b3: starting snapshot up to: index=102 2026-05-25T03:41:26.264Z [INFO] creating new snapshot: path=/tmp/raft2752666818/snapshots/2-102-1779680486264.tmp 2026-05-25T03:41:26.264Z [INFO] server-6cc8ac45-bec0-ba1f-70b9-0384854975b3: compacting logs: from=1 to=92 2026-05-25T03:41:26.264Z [INFO] server-6cc8ac45-bec0-ba1f-70b9-0384854975b3: snapshot complete up to: index=102 --- PASS: TestRaft_SnapshotRestore_Progress (0.18s) === RUN TestRaft_NoRestoreOnStart 2026-05-25T03:41:26.265Z [INFO] server-41863e28-e74a-6164-e9d3-7109c098dbbe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41863e28-e74a-6164-e9d3-7109c098dbbe Address:41863e28-e74a-6164-e9d3-7109c098dbbe}]" 2026-05-25T03:41:26.265Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:26.266Z [INFO] server-41863e28-e74a-6164-e9d3-7109c098dbbe: entering follower state: follower="Node at 41863e28-e74a-6164-e9d3-7109c098dbbe [Follower]" leader-address= leader-id= 2026-05-25T03:41:26.319Z [WARN] server-41863e28-e74a-6164-e9d3-7109c098dbbe: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:26.319Z [INFO] server-41863e28-e74a-6164-e9d3-7109c098dbbe: entering candidate state: node="Node at 41863e28-e74a-6164-e9d3-7109c098dbbe [Candidate]" term=2 2026-05-25T03:41:26.319Z [INFO] server-41863e28-e74a-6164-e9d3-7109c098dbbe: election won: term=2 tally=1 2026-05-25T03:41:26.319Z [INFO] server-41863e28-e74a-6164-e9d3-7109c098dbbe: entering leader state: leader="Node at 41863e28-e74a-6164-e9d3-7109c098dbbe [Leader]" 2026-05-25T03:41:26.425Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:26.320037597 +0000 UTC m=+12.493013041 (1 nodes), 54.05756ms from start of poll, 54.24328ms from cluster start. Timeout at 2026-05-25 03:41:26.425039157 +0000 UTC m=+12.598014601, 105.00156ms after stability 2026-05-25T03:41:26.428Z [INFO] server-41863e28-e74a-6164-e9d3-7109c098dbbe: starting snapshot up to: index=102 2026-05-25T03:41:26.428Z [INFO] creating new snapshot: path=/tmp/raft3414227836/snapshots/2-102-1779680486428.tmp 2026-05-25T03:41:26.428Z [INFO] server-41863e28-e74a-6164-e9d3-7109c098dbbe: compacting logs: from=1 to=92 2026-05-25T03:41:26.428Z [INFO] server-41863e28-e74a-6164-e9d3-7109c098dbbe: snapshot complete up to: index=102 2026-05-25T03:41:26.429Z [INFO] server-41863e28-e74a-6164-e9d3-7109c098dbbe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41863e28-e74a-6164-e9d3-7109c098dbbe Address:41863e28-e74a-6164-e9d3-7109c098dbbe}]" --- PASS: TestRaft_NoRestoreOnStart (0.16s) === RUN TestRaft_SnapshotRestore_PeerChange 2026-05-25T03:41:26.429Z [INFO] server-41863e28-e74a-6164-e9d3-7109c098dbbe: entering follower state: follower="Node at 41863e28-e74a-6164-e9d3-7109c098dbbe [Follower]" leader-address= leader-id= raft_test.go:1266: Skipping unreliable test --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) === RUN TestRaft_AutoSnapshot 2026-05-25T03:41:26.429Z [INFO] server-0606615e-2cce-105d-f9f7-7b5a4976044b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0606615e-2cce-105d-f9f7-7b5a4976044b Address:0606615e-2cce-105d-f9f7-7b5a4976044b}]" 2026-05-25T03:41:26.429Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:26.429Z [INFO] server-0606615e-2cce-105d-f9f7-7b5a4976044b: entering follower state: follower="Node at 0606615e-2cce-105d-f9f7-7b5a4976044b [Follower]" leader-address= leader-id= 2026-05-25T03:41:26.492Z [WARN] server-0606615e-2cce-105d-f9f7-7b5a4976044b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:26.492Z [INFO] server-0606615e-2cce-105d-f9f7-7b5a4976044b: entering candidate state: node="Node at 0606615e-2cce-105d-f9f7-7b5a4976044b [Candidate]" term=2 2026-05-25T03:41:26.492Z [INFO] server-0606615e-2cce-105d-f9f7-7b5a4976044b: election won: term=2 tally=1 2026-05-25T03:41:26.492Z [INFO] server-0606615e-2cce-105d-f9f7-7b5a4976044b: entering leader state: leader="Node at 0606615e-2cce-105d-f9f7-7b5a4976044b [Leader]" 2026-05-25T03:41:26.517Z [WARN] server-41863e28-e74a-6164-e9d3-7109c098dbbe: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:26.517Z [INFO] server-41863e28-e74a-6164-e9d3-7109c098dbbe: entering candidate state: node="Node at 41863e28-e74a-6164-e9d3-7109c098dbbe [Candidate]" term=3 2026-05-25T03:41:26.517Z [INFO] server-41863e28-e74a-6164-e9d3-7109c098dbbe: election won: term=3 tally=1 2026-05-25T03:41:26.517Z [INFO] server-41863e28-e74a-6164-e9d3-7109c098dbbe: entering leader state: leader="Node at 41863e28-e74a-6164-e9d3-7109c098dbbe [Leader]" 2026-05-25T03:41:26.599Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:26.493089357 +0000 UTC m=+12.666064801 (1 nodes), 63.57124ms from start of poll, 63.73484ms from cluster start. Timeout at 2026-05-25 03:41:26.598090677 +0000 UTC m=+12.771066161, 105.00136ms after stability 2026-05-25T03:41:26.602Z [INFO] server-0606615e-2cce-105d-f9f7-7b5a4976044b: starting snapshot up to: index=58 2026-05-25T03:41:26.602Z [INFO] creating new snapshot: path=/tmp/raft1165035291/snapshots/2-58-1779680486602.tmp 2026-05-25T03:41:26.602Z [INFO] server-0606615e-2cce-105d-f9f7-7b5a4976044b: compacting logs: from=1 to=58 2026-05-25T03:41:26.602Z [INFO] server-0606615e-2cce-105d-f9f7-7b5a4976044b: snapshot complete up to: index=58 --- PASS: TestRaft_AutoSnapshot (0.28s) === RUN TestRaft_UserSnapshot 2026-05-25T03:41:26.709Z [INFO] server-65b79847-eab8-68b6-3668-f87a7307d050: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65b79847-eab8-68b6-3668-f87a7307d050 Address:65b79847-eab8-68b6-3668-f87a7307d050}]" 2026-05-25T03:41:26.709Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:26.709Z [INFO] server-65b79847-eab8-68b6-3668-f87a7307d050: entering follower state: follower="Node at 65b79847-eab8-68b6-3668-f87a7307d050 [Follower]" leader-address= leader-id= 2026-05-25T03:41:26.804Z [WARN] server-65b79847-eab8-68b6-3668-f87a7307d050: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:26.804Z [INFO] server-65b79847-eab8-68b6-3668-f87a7307d050: entering candidate state: node="Node at 65b79847-eab8-68b6-3668-f87a7307d050 [Candidate]" term=2 2026-05-25T03:41:26.804Z [INFO] server-65b79847-eab8-68b6-3668-f87a7307d050: election won: term=2 tally=1 2026-05-25T03:41:26.804Z [INFO] server-65b79847-eab8-68b6-3668-f87a7307d050: entering leader state: leader="Node at 65b79847-eab8-68b6-3668-f87a7307d050 [Leader]" 2026-05-25T03:41:26.909Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:26.804441637 +0000 UTC m=+12.977417041 (1 nodes), 94.65244ms from start of poll, 94.8428ms from cluster start. Timeout at 2026-05-25 03:41:26.909443277 +0000 UTC m=+13.082418721, 105.00168ms after stability 2026-05-25T03:41:26.909Z [ERROR] server-65b79847-eab8-68b6-3668-f87a7307d050: failed to take snapshot: error="nothing new to snapshot" 2026-05-25T03:41:26.910Z [INFO] server-65b79847-eab8-68b6-3668-f87a7307d050: starting snapshot up to: index=12 2026-05-25T03:41:26.910Z [INFO] creating new snapshot: path=/tmp/raft1304557301/snapshots/2-12-1779680486910.tmp 2026-05-25T03:41:26.910Z [INFO] server-65b79847-eab8-68b6-3668-f87a7307d050: compacting logs: from=1 to=2 2026-05-25T03:41:26.910Z [INFO] server-65b79847-eab8-68b6-3668-f87a7307d050: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.20s) === RUN TestRaft_UserRestore === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false 2026-05-25T03:41:26.911Z [INFO] server-aef90201-5189-928c-62cd-9b711d06948e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aef90201-5189-928c-62cd-9b711d06948e Address:aef90201-5189-928c-62cd-9b711d06948e} {Suffrage:Voter ID:server-591f5706-3755-6b0e-4392-bcb595f84ccf Address:591f5706-3755-6b0e-4392-bcb595f84ccf} {Suffrage:Voter ID:server-446fec7b-b83b-9db6-f1ac-b14977f09b88 Address:446fec7b-b83b-9db6-f1ac-b14977f09b88}]" 2026-05-25T03:41:26.911Z [INFO] server-591f5706-3755-6b0e-4392-bcb595f84ccf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aef90201-5189-928c-62cd-9b711d06948e Address:aef90201-5189-928c-62cd-9b711d06948e} {Suffrage:Voter ID:server-591f5706-3755-6b0e-4392-bcb595f84ccf Address:591f5706-3755-6b0e-4392-bcb595f84ccf} {Suffrage:Voter ID:server-446fec7b-b83b-9db6-f1ac-b14977f09b88 Address:446fec7b-b83b-9db6-f1ac-b14977f09b88}]" 2026-05-25T03:41:26.911Z [INFO] server-aef90201-5189-928c-62cd-9b711d06948e: entering follower state: follower="Node at aef90201-5189-928c-62cd-9b711d06948e [Follower]" leader-address= leader-id= 2026-05-25T03:41:26.911Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aef90201-5189-928c-62cd-9b711d06948e Address:aef90201-5189-928c-62cd-9b711d06948e} {Suffrage:Voter ID:server-591f5706-3755-6b0e-4392-bcb595f84ccf Address:591f5706-3755-6b0e-4392-bcb595f84ccf} {Suffrage:Voter ID:server-446fec7b-b83b-9db6-f1ac-b14977f09b88 Address:446fec7b-b83b-9db6-f1ac-b14977f09b88}]" 2026-05-25T03:41:26.911Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:26.911Z [INFO] server-591f5706-3755-6b0e-4392-bcb595f84ccf: entering follower state: follower="Node at 591f5706-3755-6b0e-4392-bcb595f84ccf [Follower]" leader-address= leader-id= 2026-05-25T03:41:26.911Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: entering follower state: follower="Node at 446fec7b-b83b-9db6-f1ac-b14977f09b88 [Follower]" leader-address= leader-id= 2026-05-25T03:41:27.453Z [WARN] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:27.453Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: entering candidate state: node="Node at 446fec7b-b83b-9db6-f1ac-b14977f09b88 [Candidate]" term=2 2026-05-25T03:41:27.453Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: election won: term=2 tally=2 2026-05-25T03:41:27.453Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: entering leader state: leader="Node at 446fec7b-b83b-9db6-f1ac-b14977f09b88 [Leader]" 2026-05-25T03:41:27.453Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: added peer, starting replication: peer=server-aef90201-5189-928c-62cd-9b711d06948e 2026-05-25T03:41:27.453Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: added peer, starting replication: peer=server-591f5706-3755-6b0e-4392-bcb595f84ccf 2026-05-25T03:41:27.453Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: pipelining replication: peer="{Voter server-591f5706-3755-6b0e-4392-bcb595f84ccf 591f5706-3755-6b0e-4392-bcb595f84ccf}" 2026-05-25T03:41:27.453Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: pipelining replication: peer="{Voter server-aef90201-5189-928c-62cd-9b711d06948e aef90201-5189-928c-62cd-9b711d06948e}" 2026-05-25T03:41:28.459Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:27.453672957 +0000 UTC m=+13.626648401 (1 nodes), 542.01024ms from start of poll, 542.4028ms from cluster start. Timeout at 2026-05-25 03:41:28.458676037 +0000 UTC m=+14.631651521, 1.00500312s after stability 2026-05-25T03:41:28.460Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: starting snapshot up to: index=12 2026-05-25T03:41:28.460Z [INFO] creating new snapshot: path=/tmp/raft2670834467/snapshots/2-12-1779680488460.tmp 2026-05-25T03:41:28.460Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: snapshot complete up to: index=12 2026-05-25T03:41:28.461Z [INFO] creating new snapshot: path=/tmp/raft2670834467/snapshots/2-23-1779680488461.tmp 2026-05-25T03:41:28.462Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: copied to local snapshot: bytes=71 2026-05-25T03:41:28.462Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: snapshot restore progress: id=2-23-1779680488461 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:28.462Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: restored user snapshot: index=23 2026-05-25T03:41:28.462Z [ERROR] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: failed to get log: index=23 error="log not found" 2026-05-25T03:41:28.462Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: aborting pipeline replication: peer="{Voter server-591f5706-3755-6b0e-4392-bcb595f84ccf 591f5706-3755-6b0e-4392-bcb595f84ccf}" 2026-05-25T03:41:28.462Z [ERROR] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: failed to get log: index=23 error="log not found" 2026-05-25T03:41:28.462Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: aborting pipeline replication: peer="{Voter server-aef90201-5189-928c-62cd-9b711d06948e aef90201-5189-928c-62cd-9b711d06948e}" 2026-05-25T03:41:28.468Z [ERROR] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: failed to get log: index=23 error="log not found" 2026-05-25T03:41:28.468Z [INFO] creating new snapshot: path=/tmp/raft348621269/snapshots/2-23-1779680488468.tmp 2026-05-25T03:41:28.468Z [INFO] server-591f5706-3755-6b0e-4392-bcb595f84ccf: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:28.469Z [INFO] server-591f5706-3755-6b0e-4392-bcb595f84ccf: copied to local snapshot: bytes=71 2026-05-25T03:41:28.469Z [INFO] server-591f5706-3755-6b0e-4392-bcb595f84ccf: snapshot restore progress: id=2-23-1779680488468 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:28.469Z [ERROR] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: failed to get log: index=23 error="log not found" 2026-05-25T03:41:28.469Z [INFO] server-591f5706-3755-6b0e-4392-bcb595f84ccf: Installed remote snapshot 2026-05-25T03:41:28.469Z [WARN] server-591f5706-3755-6b0e-4392-bcb595f84ccf: skipping application of old log: index=22 2026-05-25T03:41:28.469Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: pipelining replication: peer="{Voter server-591f5706-3755-6b0e-4392-bcb595f84ccf 591f5706-3755-6b0e-4392-bcb595f84ccf}" 2026-05-25T03:41:28.469Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:28.469Z [INFO] creating new snapshot: path=/tmp/raft160269320/snapshots/2-23-1779680488469.tmp 2026-05-25T03:41:28.469Z [INFO] server-aef90201-5189-928c-62cd-9b711d06948e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:28.470Z [INFO] server-aef90201-5189-928c-62cd-9b711d06948e: copied to local snapshot: bytes=71 2026-05-25T03:41:28.470Z [INFO] server-aef90201-5189-928c-62cd-9b711d06948e: snapshot restore progress: id=2-23-1779680488469 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:28.470Z [INFO] server-aef90201-5189-928c-62cd-9b711d06948e: Installed remote snapshot 2026-05-25T03:41:28.470Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: pipelining replication: peer="{Voter server-aef90201-5189-928c-62cd-9b711d06948e aef90201-5189-928c-62cd-9b711d06948e}" 2026-05-25T03:41:29.475Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:28.469702277 +0000 UTC m=+14.642677721 (2 nodes), 1.08µs from start of poll, 1.55843212s from cluster start. Timeout at 2026-05-25 03:41:29.474706117 +0000 UTC m=+15.647681561, 1.00500384s after stability 2026-05-25T03:41:29.488Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: aborting pipeline replication: peer="{Voter server-591f5706-3755-6b0e-4392-bcb595f84ccf 591f5706-3755-6b0e-4392-bcb595f84ccf}" 2026-05-25T03:41:29.488Z [INFO] server-446fec7b-b83b-9db6-f1ac-b14977f09b88: aborting pipeline replication: peer="{Voter server-aef90201-5189-928c-62cd-9b711d06948e aef90201-5189-928c-62cd-9b711d06948e}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false 2026-05-25T03:41:29.489Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227 Address:9d621c1f-5b9d-42fe-387e-ca5a4cf75227} {Suffrage:Voter ID:server-e203b5bf-37c0-d5b9-a5ad-5ff413f5684f Address:e203b5bf-37c0-d5b9-a5ad-5ff413f5684f} {Suffrage:Voter ID:server-1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3 Address:1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3}]" 2026-05-25T03:41:29.489Z [INFO] server-e203b5bf-37c0-d5b9-a5ad-5ff413f5684f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227 Address:9d621c1f-5b9d-42fe-387e-ca5a4cf75227} {Suffrage:Voter ID:server-e203b5bf-37c0-d5b9-a5ad-5ff413f5684f Address:e203b5bf-37c0-d5b9-a5ad-5ff413f5684f} {Suffrage:Voter ID:server-1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3 Address:1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3}]" 2026-05-25T03:41:29.489Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: entering follower state: follower="Node at 9d621c1f-5b9d-42fe-387e-ca5a4cf75227 [Follower]" leader-address= leader-id= 2026-05-25T03:41:29.489Z [INFO] server-e203b5bf-37c0-d5b9-a5ad-5ff413f5684f: entering follower state: follower="Node at e203b5bf-37c0-d5b9-a5ad-5ff413f5684f [Follower]" leader-address= leader-id= 2026-05-25T03:41:29.489Z [INFO] server-1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227 Address:9d621c1f-5b9d-42fe-387e-ca5a4cf75227} {Suffrage:Voter ID:server-e203b5bf-37c0-d5b9-a5ad-5ff413f5684f Address:e203b5bf-37c0-d5b9-a5ad-5ff413f5684f} {Suffrage:Voter ID:server-1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3 Address:1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3}]" 2026-05-25T03:41:29.489Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:29.489Z [INFO] server-1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3: entering follower state: follower="Node at 1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3 [Follower]" leader-address= leader-id= 2026-05-25T03:41:30.006Z [WARN] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:30.006Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: entering candidate state: node="Node at 9d621c1f-5b9d-42fe-387e-ca5a4cf75227 [Candidate]" term=2 2026-05-25T03:41:30.006Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: election won: term=2 tally=2 2026-05-25T03:41:30.006Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: entering leader state: leader="Node at 9d621c1f-5b9d-42fe-387e-ca5a4cf75227 [Leader]" 2026-05-25T03:41:30.006Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: added peer, starting replication: peer=server-e203b5bf-37c0-d5b9-a5ad-5ff413f5684f 2026-05-25T03:41:30.006Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: added peer, starting replication: peer=server-1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3 2026-05-25T03:41:30.007Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: pipelining replication: peer="{Voter server-e203b5bf-37c0-d5b9-a5ad-5ff413f5684f e203b5bf-37c0-d5b9-a5ad-5ff413f5684f}" 2026-05-25T03:41:30.007Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: pipelining replication: peer="{Voter server-1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3 1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3}" 2026-05-25T03:41:31.012Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:30.007032637 +0000 UTC m=+16.180008121 (1 nodes), 517.24936ms from start of poll, 517.69848ms from cluster start. Timeout at 2026-05-25 03:41:31.012034317 +0000 UTC m=+17.185009761, 1.00500164s after stability 2026-05-25T03:41:31.013Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: starting snapshot up to: index=12 2026-05-25T03:41:31.013Z [INFO] creating new snapshot: path=/tmp/raft376123107/snapshots/2-12-1779680491013.tmp 2026-05-25T03:41:31.014Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: snapshot complete up to: index=12 2026-05-25T03:41:31.016Z [INFO] creating new snapshot: path=/tmp/raft376123107/snapshots/2-23-1779680491016.tmp 2026-05-25T03:41:31.016Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: copied to local snapshot: bytes=71 2026-05-25T03:41:31.016Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: snapshot restore progress: id=2-23-1779680491016 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:31.017Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: removing all old logs from log store 2026-05-25T03:41:31.017Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: compacting logs: from=1 to=22 2026-05-25T03:41:31.017Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: restored user snapshot: index=23 2026-05-25T03:41:31.017Z [ERROR] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: failed to get log: index=22 error="log not found" 2026-05-25T03:41:31.017Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: aborting pipeline replication: peer="{Voter server-1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3 1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3}" 2026-05-25T03:41:31.017Z [ERROR] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: failed to get log: index=22 error="log not found" 2026-05-25T03:41:31.017Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: aborting pipeline replication: peer="{Voter server-e203b5bf-37c0-d5b9-a5ad-5ff413f5684f e203b5bf-37c0-d5b9-a5ad-5ff413f5684f}" 2026-05-25T03:41:31.025Z [ERROR] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: failed to get log: index=22 error="log not found" 2026-05-25T03:41:31.025Z [ERROR] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: failed to get log: index=22 error="log not found" 2026-05-25T03:41:31.025Z [INFO] creating new snapshot: path=/tmp/raft3959989815/snapshots/2-23-1779680491025.tmp 2026-05-25T03:41:31.026Z [INFO] creating new snapshot: path=/tmp/raft3927474125/snapshots/2-23-1779680491026.tmp 2026-05-25T03:41:31.026Z [INFO] server-1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:31.026Z [INFO] server-e203b5bf-37c0-d5b9-a5ad-5ff413f5684f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:31.026Z [INFO] server-1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3: copied to local snapshot: bytes=71 2026-05-25T03:41:31.026Z [INFO] server-e203b5bf-37c0-d5b9-a5ad-5ff413f5684f: copied to local snapshot: bytes=71 2026-05-25T03:41:31.026Z [INFO] server-1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3: snapshot restore progress: id=2-23-1779680491025 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:31.026Z [INFO] server-1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3: removing all old logs from log store 2026-05-25T03:41:31.027Z [INFO] server-1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3: compacting logs: from=1 to=22 2026-05-25T03:41:31.027Z [INFO] server-1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3: Installed remote snapshot 2026-05-25T03:41:31.027Z [INFO] server-e203b5bf-37c0-d5b9-a5ad-5ff413f5684f: snapshot restore progress: id=2-23-1779680491026 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:31.027Z [INFO] server-e203b5bf-37c0-d5b9-a5ad-5ff413f5684f: removing all old logs from log store 2026-05-25T03:41:31.027Z [INFO] server-e203b5bf-37c0-d5b9-a5ad-5ff413f5684f: compacting logs: from=1 to=22 2026-05-25T03:41:31.027Z [INFO] server-e203b5bf-37c0-d5b9-a5ad-5ff413f5684f: Installed remote snapshot 2026-05-25T03:41:31.027Z [WARN] server-e203b5bf-37c0-d5b9-a5ad-5ff413f5684f: skipping application of old log: index=23 2026-05-25T03:41:31.027Z [WARN] server-1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3: skipping application of old log: index=22 2026-05-25T03:41:31.027Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: pipelining replication: peer="{Voter server-e203b5bf-37c0-d5b9-a5ad-5ff413f5684f e203b5bf-37c0-d5b9-a5ad-5ff413f5684f}" 2026-05-25T03:41:31.027Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: pipelining replication: peer="{Voter server-1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3 1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3}" 2026-05-25T03:41:31.027Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:32.033Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:31.027356597 +0000 UTC m=+17.200332041 (2 nodes), 1µs from start of poll, 1.5380224s from cluster start. Timeout at 2026-05-25 03:41:32.032360357 +0000 UTC m=+18.205335801, 1.00500376s after stability 2026-05-25T03:41:32.045Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: aborting pipeline replication: peer="{Voter server-e203b5bf-37c0-d5b9-a5ad-5ff413f5684f e203b5bf-37c0-d5b9-a5ad-5ff413f5684f}" 2026-05-25T03:41:32.045Z [INFO] server-9d621c1f-5b9d-42fe-387e-ca5a4cf75227: aborting pipeline replication: peer="{Voter server-1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3 1c6995f9-9b7c-64ed-b1bd-8687e0dad0b3}" === RUN TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true 2026-05-25T03:41:32.046Z [INFO] server-ac2e4673-7b21-036e-02b5-91ab4293d98b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac2e4673-7b21-036e-02b5-91ab4293d98b Address:ac2e4673-7b21-036e-02b5-91ab4293d98b} {Suffrage:Voter ID:server-2587a48a-abfa-7325-6a97-238e6e35c4a1 Address:2587a48a-abfa-7325-6a97-238e6e35c4a1} {Suffrage:Voter ID:server-c2f32d00-b065-3337-e998-1920ec694e74 Address:c2f32d00-b065-3337-e998-1920ec694e74}]" 2026-05-25T03:41:32.047Z [INFO] server-2587a48a-abfa-7325-6a97-238e6e35c4a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac2e4673-7b21-036e-02b5-91ab4293d98b Address:ac2e4673-7b21-036e-02b5-91ab4293d98b} {Suffrage:Voter ID:server-2587a48a-abfa-7325-6a97-238e6e35c4a1 Address:2587a48a-abfa-7325-6a97-238e6e35c4a1} {Suffrage:Voter ID:server-c2f32d00-b065-3337-e998-1920ec694e74 Address:c2f32d00-b065-3337-e998-1920ec694e74}]" 2026-05-25T03:41:32.046Z [INFO] server-ac2e4673-7b21-036e-02b5-91ab4293d98b: entering follower state: follower="Node at ac2e4673-7b21-036e-02b5-91ab4293d98b [Follower]" leader-address= leader-id= 2026-05-25T03:41:32.047Z [INFO] server-c2f32d00-b065-3337-e998-1920ec694e74: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac2e4673-7b21-036e-02b5-91ab4293d98b Address:ac2e4673-7b21-036e-02b5-91ab4293d98b} {Suffrage:Voter ID:server-2587a48a-abfa-7325-6a97-238e6e35c4a1 Address:2587a48a-abfa-7325-6a97-238e6e35c4a1} {Suffrage:Voter ID:server-c2f32d00-b065-3337-e998-1920ec694e74 Address:c2f32d00-b065-3337-e998-1920ec694e74}]" 2026-05-25T03:41:32.047Z [INFO] server-2587a48a-abfa-7325-6a97-238e6e35c4a1: entering follower state: follower="Node at 2587a48a-abfa-7325-6a97-238e6e35c4a1 [Follower]" leader-address= leader-id= 2026-05-25T03:41:32.047Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:32.047Z [INFO] server-c2f32d00-b065-3337-e998-1920ec694e74: entering follower state: follower="Node at c2f32d00-b065-3337-e998-1920ec694e74 [Follower]" leader-address= leader-id= 2026-05-25T03:41:32.759Z [WARN] server-c2f32d00-b065-3337-e998-1920ec694e74: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:32.759Z [INFO] server-c2f32d00-b065-3337-e998-1920ec694e74: entering candidate state: node="Node at c2f32d00-b065-3337-e998-1920ec694e74 [Candidate]" term=2 2026-05-25T03:41:32.759Z [INFO] server-c2f32d00-b065-3337-e998-1920ec694e74: election won: term=2 tally=2 2026-05-25T03:41:32.759Z [INFO] server-c2f32d00-b065-3337-e998-1920ec694e74: entering leader state: leader="Node at c2f32d00-b065-3337-e998-1920ec694e74 [Leader]" 2026-05-25T03:41:32.759Z [INFO] server-c2f32d00-b065-3337-e998-1920ec694e74: added peer, starting replication: peer=server-ac2e4673-7b21-036e-02b5-91ab4293d98b 2026-05-25T03:41:32.759Z [INFO] server-c2f32d00-b065-3337-e998-1920ec694e74: added peer, starting replication: peer=server-2587a48a-abfa-7325-6a97-238e6e35c4a1 2026-05-25T03:41:32.759Z [INFO] server-c2f32d00-b065-3337-e998-1920ec694e74: pipelining replication: peer="{Voter server-2587a48a-abfa-7325-6a97-238e6e35c4a1 2587a48a-abfa-7325-6a97-238e6e35c4a1}" 2026-05-25T03:41:32.759Z [INFO] server-c2f32d00-b065-3337-e998-1920ec694e74: pipelining replication: peer="{Voter server-ac2e4673-7b21-036e-02b5-91ab4293d98b ac2e4673-7b21-036e-02b5-91ab4293d98b}" 2026-05-25T03:41:33.765Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:32.759750357 +0000 UTC m=+18.932725801 (1 nodes), 712.58324ms from start of poll, 713.0118ms from cluster start. Timeout at 2026-05-25 03:41:33.764752357 +0000 UTC m=+19.937727801, 1.005002s after stability 2026-05-25T03:41:33.765Z [INFO] server-c2f32d00-b065-3337-e998-1920ec694e74: starting snapshot up to: index=12 2026-05-25T03:41:33.765Z [INFO] creating new snapshot: path=/tmp/raft784843369/snapshots/2-12-1779680493765.tmp 2026-05-25T03:41:33.766Z [INFO] server-c2f32d00-b065-3337-e998-1920ec694e74: snapshot complete up to: index=12 2026-05-25T03:41:33.767Z [INFO] server-07ade529-b79b-bf70-6afa-177e05b5b25b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07ade529-b79b-bf70-6afa-177e05b5b25b Address:07ade529-b79b-bf70-6afa-177e05b5b25b} {Suffrage:Voter ID:server-27d976e6-26d7-2deb-a8c9-9c173ee25928 Address:27d976e6-26d7-2deb-a8c9-9c173ee25928} {Suffrage:Voter ID:server-d317c517-71dd-c456-c19d-a6263de195da Address:d317c517-71dd-c456-c19d-a6263de195da}]" 2026-05-25T03:41:33.767Z [INFO] server-27d976e6-26d7-2deb-a8c9-9c173ee25928: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07ade529-b79b-bf70-6afa-177e05b5b25b Address:07ade529-b79b-bf70-6afa-177e05b5b25b} {Suffrage:Voter ID:server-27d976e6-26d7-2deb-a8c9-9c173ee25928 Address:27d976e6-26d7-2deb-a8c9-9c173ee25928} {Suffrage:Voter ID:server-d317c517-71dd-c456-c19d-a6263de195da Address:d317c517-71dd-c456-c19d-a6263de195da}]" 2026-05-25T03:41:33.767Z [INFO] server-07ade529-b79b-bf70-6afa-177e05b5b25b: entering follower state: follower="Node at 07ade529-b79b-bf70-6afa-177e05b5b25b [Follower]" leader-address= leader-id= 2026-05-25T03:41:33.767Z [INFO] server-d317c517-71dd-c456-c19d-a6263de195da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07ade529-b79b-bf70-6afa-177e05b5b25b Address:07ade529-b79b-bf70-6afa-177e05b5b25b} {Suffrage:Voter ID:server-27d976e6-26d7-2deb-a8c9-9c173ee25928 Address:27d976e6-26d7-2deb-a8c9-9c173ee25928} {Suffrage:Voter ID:server-d317c517-71dd-c456-c19d-a6263de195da Address:d317c517-71dd-c456-c19d-a6263de195da}]" 2026-05-25T03:41:33.767Z [INFO] server-27d976e6-26d7-2deb-a8c9-9c173ee25928: entering follower state: follower="Node at 27d976e6-26d7-2deb-a8c9-9c173ee25928 [Follower]" leader-address= leader-id= 2026-05-25T03:41:33.767Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:33.767Z [INFO] server-d317c517-71dd-c456-c19d-a6263de195da: entering follower state: follower="Node at d317c517-71dd-c456-c19d-a6263de195da [Follower]" leader-address= leader-id= 2026-05-25T03:41:34.305Z [WARN] server-d317c517-71dd-c456-c19d-a6263de195da: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:34.305Z [INFO] server-d317c517-71dd-c456-c19d-a6263de195da: entering candidate state: node="Node at d317c517-71dd-c456-c19d-a6263de195da [Candidate]" term=2 2026-05-25T03:41:34.305Z [INFO] server-d317c517-71dd-c456-c19d-a6263de195da: election won: term=2 tally=2 2026-05-25T03:41:34.305Z [INFO] server-d317c517-71dd-c456-c19d-a6263de195da: entering leader state: leader="Node at d317c517-71dd-c456-c19d-a6263de195da [Leader]" 2026-05-25T03:41:34.305Z [INFO] server-d317c517-71dd-c456-c19d-a6263de195da: added peer, starting replication: peer=server-07ade529-b79b-bf70-6afa-177e05b5b25b 2026-05-25T03:41:34.305Z [INFO] server-d317c517-71dd-c456-c19d-a6263de195da: added peer, starting replication: peer=server-27d976e6-26d7-2deb-a8c9-9c173ee25928 2026-05-25T03:41:34.305Z [INFO] server-d317c517-71dd-c456-c19d-a6263de195da: pipelining replication: peer="{Voter server-27d976e6-26d7-2deb-a8c9-9c173ee25928 27d976e6-26d7-2deb-a8c9-9c173ee25928}" 2026-05-25T03:41:34.306Z [INFO] server-d317c517-71dd-c456-c19d-a6263de195da: pipelining replication: peer="{Voter server-07ade529-b79b-bf70-6afa-177e05b5b25b 07ade529-b79b-bf70-6afa-177e05b5b25b}" 2026-05-25T03:41:35.310Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:34.305581357 +0000 UTC m=+20.478556801 (1 nodes), 537.9376ms from start of poll, 538.39904ms from cluster start. Timeout at 2026-05-25 03:41:35.310584757 +0000 UTC m=+21.483560201, 1.0050034s after stability 2026-05-25T03:41:35.310Z [INFO] creating new snapshot: path=/tmp/raft145704549/snapshots/2-13-1779680495310.tmp 2026-05-25T03:41:35.311Z [INFO] server-d317c517-71dd-c456-c19d-a6263de195da: copied to local snapshot: bytes=71 2026-05-25T03:41:35.311Z [INFO] server-d317c517-71dd-c456-c19d-a6263de195da: snapshot restore progress: id=2-13-1779680495310 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:35.311Z [INFO] server-d317c517-71dd-c456-c19d-a6263de195da: restored user snapshot: index=13 2026-05-25T03:41:35.311Z [ERROR] server-d317c517-71dd-c456-c19d-a6263de195da: failed to get log: index=3 error="log not found" 2026-05-25T03:41:35.311Z [INFO] server-d317c517-71dd-c456-c19d-a6263de195da: aborting pipeline replication: peer="{Voter server-27d976e6-26d7-2deb-a8c9-9c173ee25928 27d976e6-26d7-2deb-a8c9-9c173ee25928}" 2026-05-25T03:41:35.311Z [ERROR] server-d317c517-71dd-c456-c19d-a6263de195da: failed to get log: index=3 error="log not found" 2026-05-25T03:41:35.311Z [INFO] server-d317c517-71dd-c456-c19d-a6263de195da: aborting pipeline replication: peer="{Voter server-07ade529-b79b-bf70-6afa-177e05b5b25b 07ade529-b79b-bf70-6afa-177e05b5b25b}" 2026-05-25T03:41:35.321Z [ERROR] server-d317c517-71dd-c456-c19d-a6263de195da: failed to get log: index=3 error="log not found" 2026-05-25T03:41:35.322Z [INFO] creating new snapshot: path=/tmp/raft536773385/snapshots/2-13-1779680495322.tmp 2026-05-25T03:41:35.322Z [ERROR] server-d317c517-71dd-c456-c19d-a6263de195da: failed to get log: index=3 error="log not found" 2026-05-25T03:41:35.322Z [INFO] server-07ade529-b79b-bf70-6afa-177e05b5b25b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:35.322Z [INFO] creating new snapshot: path=/tmp/raft344323477/snapshots/2-13-1779680495322.tmp 2026-05-25T03:41:35.322Z [INFO] server-07ade529-b79b-bf70-6afa-177e05b5b25b: copied to local snapshot: bytes=71 2026-05-25T03:41:35.322Z [INFO] server-27d976e6-26d7-2deb-a8c9-9c173ee25928: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:35.322Z [INFO] server-07ade529-b79b-bf70-6afa-177e05b5b25b: snapshot restore progress: id=2-13-1779680495322 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:35.323Z [INFO] server-07ade529-b79b-bf70-6afa-177e05b5b25b: Installed remote snapshot 2026-05-25T03:41:35.323Z [INFO] server-d317c517-71dd-c456-c19d-a6263de195da: pipelining replication: peer="{Voter server-07ade529-b79b-bf70-6afa-177e05b5b25b 07ade529-b79b-bf70-6afa-177e05b5b25b}" 2026-05-25T03:41:35.323Z [INFO] server-27d976e6-26d7-2deb-a8c9-9c173ee25928: copied to local snapshot: bytes=71 2026-05-25T03:41:35.323Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:35.323Z [INFO] server-27d976e6-26d7-2deb-a8c9-9c173ee25928: snapshot restore progress: id=2-13-1779680495322 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:35.323Z [INFO] server-27d976e6-26d7-2deb-a8c9-9c173ee25928: Installed remote snapshot 2026-05-25T03:41:35.323Z [INFO] server-d317c517-71dd-c456-c19d-a6263de195da: pipelining replication: peer="{Voter server-27d976e6-26d7-2deb-a8c9-9c173ee25928 27d976e6-26d7-2deb-a8c9-9c173ee25928}" 2026-05-25T03:41:36.328Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:35.323280477 +0000 UTC m=+21.496255921 (2 nodes), 800ns from start of poll, 1.55609816s from cluster start. Timeout at 2026-05-25 03:41:36.328287957 +0000 UTC m=+22.501263401, 1.00500748s after stability 2026-05-25T03:41:36.340Z [INFO] server-c2f32d00-b065-3337-e998-1920ec694e74: aborting pipeline replication: peer="{Voter server-2587a48a-abfa-7325-6a97-238e6e35c4a1 2587a48a-abfa-7325-6a97-238e6e35c4a1}" 2026-05-25T03:41:36.340Z [INFO] server-c2f32d00-b065-3337-e998-1920ec694e74: aborting pipeline replication: peer="{Voter server-ac2e4673-7b21-036e-02b5-91ab4293d98b ac2e4673-7b21-036e-02b5-91ab4293d98b}" === RUN TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true 2026-05-25T03:41:36.341Z [INFO] server-3cd1f4b4-884d-557c-2f75-73cec8aeebb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3cd1f4b4-884d-557c-2f75-73cec8aeebb8 Address:3cd1f4b4-884d-557c-2f75-73cec8aeebb8} {Suffrage:Voter ID:server-6dd98aba-b39c-33bc-de6a-ba9da377194c Address:6dd98aba-b39c-33bc-de6a-ba9da377194c} {Suffrage:Voter ID:server-c4a5683c-7b17-7fc2-778c-ced74c0fc3c6 Address:c4a5683c-7b17-7fc2-778c-ced74c0fc3c6}]" 2026-05-25T03:41:36.341Z [INFO] server-6dd98aba-b39c-33bc-de6a-ba9da377194c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3cd1f4b4-884d-557c-2f75-73cec8aeebb8 Address:3cd1f4b4-884d-557c-2f75-73cec8aeebb8} {Suffrage:Voter ID:server-6dd98aba-b39c-33bc-de6a-ba9da377194c Address:6dd98aba-b39c-33bc-de6a-ba9da377194c} {Suffrage:Voter ID:server-c4a5683c-7b17-7fc2-778c-ced74c0fc3c6 Address:c4a5683c-7b17-7fc2-778c-ced74c0fc3c6}]" 2026-05-25T03:41:36.341Z [INFO] server-3cd1f4b4-884d-557c-2f75-73cec8aeebb8: entering follower state: follower="Node at 3cd1f4b4-884d-557c-2f75-73cec8aeebb8 [Follower]" leader-address= leader-id= 2026-05-25T03:41:36.341Z [INFO] server-6dd98aba-b39c-33bc-de6a-ba9da377194c: entering follower state: follower="Node at 6dd98aba-b39c-33bc-de6a-ba9da377194c [Follower]" leader-address= leader-id= 2026-05-25T03:41:36.342Z [INFO] server-c4a5683c-7b17-7fc2-778c-ced74c0fc3c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3cd1f4b4-884d-557c-2f75-73cec8aeebb8 Address:3cd1f4b4-884d-557c-2f75-73cec8aeebb8} {Suffrage:Voter ID:server-6dd98aba-b39c-33bc-de6a-ba9da377194c Address:6dd98aba-b39c-33bc-de6a-ba9da377194c} {Suffrage:Voter ID:server-c4a5683c-7b17-7fc2-778c-ced74c0fc3c6 Address:c4a5683c-7b17-7fc2-778c-ced74c0fc3c6}]" 2026-05-25T03:41:36.342Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:36.342Z [INFO] server-c4a5683c-7b17-7fc2-778c-ced74c0fc3c6: entering follower state: follower="Node at c4a5683c-7b17-7fc2-778c-ced74c0fc3c6 [Follower]" leader-address= leader-id= 2026-05-25T03:41:37.096Z [WARN] server-6dd98aba-b39c-33bc-de6a-ba9da377194c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:37.096Z [INFO] server-6dd98aba-b39c-33bc-de6a-ba9da377194c: entering candidate state: node="Node at 6dd98aba-b39c-33bc-de6a-ba9da377194c [Candidate]" term=2 2026-05-25T03:41:37.096Z [INFO] server-6dd98aba-b39c-33bc-de6a-ba9da377194c: election won: term=2 tally=2 2026-05-25T03:41:37.096Z [INFO] server-6dd98aba-b39c-33bc-de6a-ba9da377194c: entering leader state: leader="Node at 6dd98aba-b39c-33bc-de6a-ba9da377194c [Leader]" 2026-05-25T03:41:37.096Z [INFO] server-6dd98aba-b39c-33bc-de6a-ba9da377194c: added peer, starting replication: peer=server-3cd1f4b4-884d-557c-2f75-73cec8aeebb8 2026-05-25T03:41:37.096Z [INFO] server-6dd98aba-b39c-33bc-de6a-ba9da377194c: added peer, starting replication: peer=server-c4a5683c-7b17-7fc2-778c-ced74c0fc3c6 2026-05-25T03:41:37.096Z [INFO] server-6dd98aba-b39c-33bc-de6a-ba9da377194c: pipelining replication: peer="{Voter server-3cd1f4b4-884d-557c-2f75-73cec8aeebb8 3cd1f4b4-884d-557c-2f75-73cec8aeebb8}" 2026-05-25T03:41:37.096Z [INFO] server-6dd98aba-b39c-33bc-de6a-ba9da377194c: pipelining replication: peer="{Voter server-c4a5683c-7b17-7fc2-778c-ced74c0fc3c6 c4a5683c-7b17-7fc2-778c-ced74c0fc3c6}" 2026-05-25T03:41:38.101Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:37.096465357 +0000 UTC m=+23.269440801 (1 nodes), 753.96732ms from start of poll, 755.10324ms from cluster start. Timeout at 2026-05-25 03:41:38.101470597 +0000 UTC m=+24.274446081, 1.00500528s after stability 2026-05-25T03:41:38.103Z [INFO] server-6dd98aba-b39c-33bc-de6a-ba9da377194c: starting snapshot up to: index=12 2026-05-25T03:41:38.104Z [INFO] creating new snapshot: path=/tmp/raft1121939270/snapshots/2-12-1779680498104.tmp 2026-05-25T03:41:38.104Z [INFO] server-6dd98aba-b39c-33bc-de6a-ba9da377194c: snapshot complete up to: index=12 2026-05-25T03:41:38.105Z [INFO] server-ea7b237b-c29a-402e-14cd-08bf6c17226c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea7b237b-c29a-402e-14cd-08bf6c17226c Address:ea7b237b-c29a-402e-14cd-08bf6c17226c} {Suffrage:Voter ID:server-b9b12d6f-2d90-9ee3-d12f-0df643e01935 Address:b9b12d6f-2d90-9ee3-d12f-0df643e01935} {Suffrage:Voter ID:server-4bffd149-2b1d-4420-ad2d-08ec4ce78c32 Address:4bffd149-2b1d-4420-ad2d-08ec4ce78c32}]" 2026-05-25T03:41:38.105Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea7b237b-c29a-402e-14cd-08bf6c17226c Address:ea7b237b-c29a-402e-14cd-08bf6c17226c} {Suffrage:Voter ID:server-b9b12d6f-2d90-9ee3-d12f-0df643e01935 Address:b9b12d6f-2d90-9ee3-d12f-0df643e01935} {Suffrage:Voter ID:server-4bffd149-2b1d-4420-ad2d-08ec4ce78c32 Address:4bffd149-2b1d-4420-ad2d-08ec4ce78c32}]" 2026-05-25T03:41:38.105Z [INFO] server-ea7b237b-c29a-402e-14cd-08bf6c17226c: entering follower state: follower="Node at ea7b237b-c29a-402e-14cd-08bf6c17226c [Follower]" leader-address= leader-id= 2026-05-25T03:41:38.105Z [INFO] server-4bffd149-2b1d-4420-ad2d-08ec4ce78c32: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea7b237b-c29a-402e-14cd-08bf6c17226c Address:ea7b237b-c29a-402e-14cd-08bf6c17226c} {Suffrage:Voter ID:server-b9b12d6f-2d90-9ee3-d12f-0df643e01935 Address:b9b12d6f-2d90-9ee3-d12f-0df643e01935} {Suffrage:Voter ID:server-4bffd149-2b1d-4420-ad2d-08ec4ce78c32 Address:4bffd149-2b1d-4420-ad2d-08ec4ce78c32}]" 2026-05-25T03:41:38.105Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: entering follower state: follower="Node at b9b12d6f-2d90-9ee3-d12f-0df643e01935 [Follower]" leader-address= leader-id= 2026-05-25T03:41:38.105Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:38.105Z [INFO] server-4bffd149-2b1d-4420-ad2d-08ec4ce78c32: entering follower state: follower="Node at 4bffd149-2b1d-4420-ad2d-08ec4ce78c32 [Follower]" leader-address= leader-id= 2026-05-25T03:41:38.658Z [WARN] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:38.658Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: entering candidate state: node="Node at b9b12d6f-2d90-9ee3-d12f-0df643e01935 [Candidate]" term=2 2026-05-25T03:41:38.658Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: election won: term=2 tally=2 2026-05-25T03:41:38.658Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: entering leader state: leader="Node at b9b12d6f-2d90-9ee3-d12f-0df643e01935 [Leader]" 2026-05-25T03:41:38.658Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: added peer, starting replication: peer=server-ea7b237b-c29a-402e-14cd-08bf6c17226c 2026-05-25T03:41:38.658Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: added peer, starting replication: peer=server-4bffd149-2b1d-4420-ad2d-08ec4ce78c32 2026-05-25T03:41:38.658Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: pipelining replication: peer="{Voter server-4bffd149-2b1d-4420-ad2d-08ec4ce78c32 4bffd149-2b1d-4420-ad2d-08ec4ce78c32}" 2026-05-25T03:41:38.658Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: pipelining replication: peer="{Voter server-ea7b237b-c29a-402e-14cd-08bf6c17226c ea7b237b-c29a-402e-14cd-08bf6c17226c}" 2026-05-25T03:41:39.663Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:38.658563237 +0000 UTC m=+24.831538681 (1 nodes), 552.79264ms from start of poll, 553.22492ms from cluster start. Timeout at 2026-05-25 03:41:39.663566597 +0000 UTC m=+25.836542041, 1.00500336s after stability 2026-05-25T03:41:39.663Z [INFO] creating new snapshot: path=/tmp/raft857298187/snapshots/2-13-1779680499663.tmp 2026-05-25T03:41:39.664Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: copied to local snapshot: bytes=71 2026-05-25T03:41:39.664Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: snapshot restore progress: id=2-13-1779680499663 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:39.664Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: removing all old logs from log store 2026-05-25T03:41:39.664Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: compacting logs: from=1 to=2 2026-05-25T03:41:39.664Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: restored user snapshot: index=13 2026-05-25T03:41:39.664Z [ERROR] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: failed to get log: index=2 error="log not found" 2026-05-25T03:41:39.664Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: aborting pipeline replication: peer="{Voter server-4bffd149-2b1d-4420-ad2d-08ec4ce78c32 4bffd149-2b1d-4420-ad2d-08ec4ce78c32}" 2026-05-25T03:41:39.664Z [ERROR] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: failed to get log: index=2 error="log not found" 2026-05-25T03:41:39.664Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: aborting pipeline replication: peer="{Voter server-ea7b237b-c29a-402e-14cd-08bf6c17226c ea7b237b-c29a-402e-14cd-08bf6c17226c}" 2026-05-25T03:41:39.672Z [ERROR] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: failed to get log: index=2 error="log not found" 2026-05-25T03:41:39.673Z [ERROR] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: failed to get log: index=2 error="log not found" 2026-05-25T03:41:39.673Z [INFO] creating new snapshot: path=/tmp/raft3237360557/snapshots/2-13-1779680499673.tmp 2026-05-25T03:41:39.673Z [INFO] server-ea7b237b-c29a-402e-14cd-08bf6c17226c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:39.673Z [INFO] creating new snapshot: path=/tmp/raft1347742665/snapshots/2-13-1779680499673.tmp 2026-05-25T03:41:39.673Z [INFO] server-ea7b237b-c29a-402e-14cd-08bf6c17226c: copied to local snapshot: bytes=71 2026-05-25T03:41:39.673Z [INFO] server-4bffd149-2b1d-4420-ad2d-08ec4ce78c32: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:39.674Z [INFO] server-ea7b237b-c29a-402e-14cd-08bf6c17226c: snapshot restore progress: id=2-13-1779680499673 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:39.674Z [INFO] server-ea7b237b-c29a-402e-14cd-08bf6c17226c: removing all old logs from log store 2026-05-25T03:41:39.674Z [INFO] server-ea7b237b-c29a-402e-14cd-08bf6c17226c: compacting logs: from=1 to=2 2026-05-25T03:41:39.674Z [INFO] server-ea7b237b-c29a-402e-14cd-08bf6c17226c: Installed remote snapshot 2026-05-25T03:41:39.674Z [INFO] server-4bffd149-2b1d-4420-ad2d-08ec4ce78c32: copied to local snapshot: bytes=71 2026-05-25T03:41:39.674Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: pipelining replication: peer="{Voter server-ea7b237b-c29a-402e-14cd-08bf6c17226c ea7b237b-c29a-402e-14cd-08bf6c17226c}" 2026-05-25T03:41:39.674Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:39.674Z [INFO] server-4bffd149-2b1d-4420-ad2d-08ec4ce78c32: snapshot restore progress: id=2-13-1779680499673 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:39.674Z [INFO] server-4bffd149-2b1d-4420-ad2d-08ec4ce78c32: removing all old logs from log store 2026-05-25T03:41:39.674Z [INFO] server-4bffd149-2b1d-4420-ad2d-08ec4ce78c32: compacting logs: from=1 to=2 2026-05-25T03:41:39.674Z [INFO] server-4bffd149-2b1d-4420-ad2d-08ec4ce78c32: Installed remote snapshot 2026-05-25T03:41:39.674Z [INFO] server-b9b12d6f-2d90-9ee3-d12f-0df643e01935: pipelining replication: peer="{Voter server-4bffd149-2b1d-4420-ad2d-08ec4ce78c32 4bffd149-2b1d-4420-ad2d-08ec4ce78c32}" 2026-05-25T03:41:40.679Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:39.674330277 +0000 UTC m=+25.847305721 (2 nodes), 1.28µs from start of poll, 1.56899196s from cluster start. Timeout at 2026-05-25 03:41:40.679338357 +0000 UTC m=+26.852313801, 1.00500808s after stability 2026-05-25T03:41:40.692Z [INFO] server-6dd98aba-b39c-33bc-de6a-ba9da377194c: aborting pipeline replication: peer="{Voter server-c4a5683c-7b17-7fc2-778c-ced74c0fc3c6 c4a5683c-7b17-7fc2-778c-ced74c0fc3c6}" 2026-05-25T03:41:40.692Z [INFO] server-6dd98aba-b39c-33bc-de6a-ba9da377194c: aborting pipeline replication: peer="{Voter server-3cd1f4b4-884d-557c-2f75-73cec8aeebb8 3cd1f4b4-884d-557c-2f75-73cec8aeebb8}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false 2026-05-25T03:41:40.694Z [INFO] server-3806be9c-67dc-e310-9b77-16217d8f2dd7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3806be9c-67dc-e310-9b77-16217d8f2dd7 Address:3806be9c-67dc-e310-9b77-16217d8f2dd7} {Suffrage:Voter ID:server-68628901-e7d1-1b6b-1a8e-20d09753de1f Address:68628901-e7d1-1b6b-1a8e-20d09753de1f} {Suffrage:Voter ID:server-9761f9cb-07d1-d17c-8d35-72012c5836ec Address:9761f9cb-07d1-d17c-8d35-72012c5836ec}]" 2026-05-25T03:41:40.694Z [INFO] server-3806be9c-67dc-e310-9b77-16217d8f2dd7: entering follower state: follower="Node at 3806be9c-67dc-e310-9b77-16217d8f2dd7 [Follower]" leader-address= leader-id= 2026-05-25T03:41:40.694Z [INFO] server-68628901-e7d1-1b6b-1a8e-20d09753de1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3806be9c-67dc-e310-9b77-16217d8f2dd7 Address:3806be9c-67dc-e310-9b77-16217d8f2dd7} {Suffrage:Voter ID:server-68628901-e7d1-1b6b-1a8e-20d09753de1f Address:68628901-e7d1-1b6b-1a8e-20d09753de1f} {Suffrage:Voter ID:server-9761f9cb-07d1-d17c-8d35-72012c5836ec Address:9761f9cb-07d1-d17c-8d35-72012c5836ec}]" 2026-05-25T03:41:40.694Z [INFO] server-68628901-e7d1-1b6b-1a8e-20d09753de1f: entering follower state: follower="Node at 68628901-e7d1-1b6b-1a8e-20d09753de1f [Follower]" leader-address= leader-id= 2026-05-25T03:41:40.695Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3806be9c-67dc-e310-9b77-16217d8f2dd7 Address:3806be9c-67dc-e310-9b77-16217d8f2dd7} {Suffrage:Voter ID:server-68628901-e7d1-1b6b-1a8e-20d09753de1f Address:68628901-e7d1-1b6b-1a8e-20d09753de1f} {Suffrage:Voter ID:server-9761f9cb-07d1-d17c-8d35-72012c5836ec Address:9761f9cb-07d1-d17c-8d35-72012c5836ec}]" 2026-05-25T03:41:40.695Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:40.695Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: entering follower state: follower="Node at 9761f9cb-07d1-d17c-8d35-72012c5836ec [Follower]" leader-address= leader-id= 2026-05-25T03:41:41.280Z [WARN] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:41.280Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: entering candidate state: node="Node at 9761f9cb-07d1-d17c-8d35-72012c5836ec [Candidate]" term=2 2026-05-25T03:41:41.280Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: election won: term=2 tally=2 2026-05-25T03:41:41.280Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: entering leader state: leader="Node at 9761f9cb-07d1-d17c-8d35-72012c5836ec [Leader]" 2026-05-25T03:41:41.280Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: added peer, starting replication: peer=server-3806be9c-67dc-e310-9b77-16217d8f2dd7 2026-05-25T03:41:41.280Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: added peer, starting replication: peer=server-68628901-e7d1-1b6b-1a8e-20d09753de1f 2026-05-25T03:41:41.281Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: pipelining replication: peer="{Voter server-68628901-e7d1-1b6b-1a8e-20d09753de1f 68628901-e7d1-1b6b-1a8e-20d09753de1f}" 2026-05-25T03:41:41.281Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: pipelining replication: peer="{Voter server-3806be9c-67dc-e310-9b77-16217d8f2dd7 3806be9c-67dc-e310-9b77-16217d8f2dd7}" 2026-05-25T03:41:42.286Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:41.281024917 +0000 UTC m=+27.454000361 (1 nodes), 585.8774ms from start of poll, 587.35036ms from cluster start. Timeout at 2026-05-25 03:41:42.286032437 +0000 UTC m=+28.459007881, 1.00500752s after stability 2026-05-25T03:41:42.287Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: starting snapshot up to: index=12 2026-05-25T03:41:42.287Z [INFO] creating new snapshot: path=/tmp/raft610644480/snapshots/2-12-1779680502287.tmp 2026-05-25T03:41:42.291Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: snapshot complete up to: index=12 2026-05-25T03:41:42.292Z [INFO] creating new snapshot: path=/tmp/raft610644480/snapshots/2-23-1779680502292.tmp 2026-05-25T03:41:42.293Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: copied to local snapshot: bytes=71 2026-05-25T03:41:42.293Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: snapshot restore progress: id=2-23-1779680502292 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:42.293Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: restored user snapshot: index=23 2026-05-25T03:41:42.293Z [ERROR] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: failed to get log: index=23 error="log not found" 2026-05-25T03:41:42.293Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: aborting pipeline replication: peer="{Voter server-68628901-e7d1-1b6b-1a8e-20d09753de1f 68628901-e7d1-1b6b-1a8e-20d09753de1f}" 2026-05-25T03:41:42.293Z [ERROR] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: failed to get log: index=23 error="log not found" 2026-05-25T03:41:42.293Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: aborting pipeline replication: peer="{Voter server-3806be9c-67dc-e310-9b77-16217d8f2dd7 3806be9c-67dc-e310-9b77-16217d8f2dd7}" 2026-05-25T03:41:42.301Z [ERROR] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: failed to get log: index=23 error="log not found" 2026-05-25T03:41:42.301Z [INFO] creating new snapshot: path=/tmp/raft1820263587/snapshots/2-23-1779680502301.tmp 2026-05-25T03:41:42.302Z [INFO] server-68628901-e7d1-1b6b-1a8e-20d09753de1f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:42.302Z [INFO] server-68628901-e7d1-1b6b-1a8e-20d09753de1f: copied to local snapshot: bytes=71 2026-05-25T03:41:42.302Z [INFO] server-68628901-e7d1-1b6b-1a8e-20d09753de1f: snapshot restore progress: id=2-23-1779680502301 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:42.302Z [INFO] server-68628901-e7d1-1b6b-1a8e-20d09753de1f: Installed remote snapshot 2026-05-25T03:41:42.302Z [WARN] server-68628901-e7d1-1b6b-1a8e-20d09753de1f: skipping application of old log: index=22 2026-05-25T03:41:42.303Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: pipelining replication: peer="{Voter server-68628901-e7d1-1b6b-1a8e-20d09753de1f 68628901-e7d1-1b6b-1a8e-20d09753de1f}" 2026-05-25T03:41:42.303Z [ERROR] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: failed to get log: index=23 error="log not found" 2026-05-25T03:41:42.303Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:42.303Z [INFO] creating new snapshot: path=/tmp/raft1818188527/snapshots/2-23-1779680502303.tmp 2026-05-25T03:41:42.303Z [INFO] server-3806be9c-67dc-e310-9b77-16217d8f2dd7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:42.303Z [INFO] server-3806be9c-67dc-e310-9b77-16217d8f2dd7: copied to local snapshot: bytes=71 2026-05-25T03:41:42.304Z [INFO] server-3806be9c-67dc-e310-9b77-16217d8f2dd7: snapshot restore progress: id=2-23-1779680502303 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:42.304Z [INFO] server-3806be9c-67dc-e310-9b77-16217d8f2dd7: Installed remote snapshot 2026-05-25T03:41:42.304Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: pipelining replication: peer="{Voter server-3806be9c-67dc-e310-9b77-16217d8f2dd7 3806be9c-67dc-e310-9b77-16217d8f2dd7}" 2026-05-25T03:41:43.308Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:42.303106997 +0000 UTC m=+28.476082401 (2 nodes), 1.24µs from start of poll, 1.6094324s from cluster start. Timeout at 2026-05-25 03:41:43.308119677 +0000 UTC m=+29.481095121, 1.00501272s after stability 2026-05-25T03:41:43.320Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: aborting pipeline replication: peer="{Voter server-68628901-e7d1-1b6b-1a8e-20d09753de1f 68628901-e7d1-1b6b-1a8e-20d09753de1f}" 2026-05-25T03:41:43.320Z [INFO] server-9761f9cb-07d1-d17c-8d35-72012c5836ec: aborting pipeline replication: peer="{Voter server-3806be9c-67dc-e310-9b77-16217d8f2dd7 3806be9c-67dc-e310-9b77-16217d8f2dd7}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false 2026-05-25T03:41:43.321Z [INFO] server-c57a6a07-877f-da62-877d-8c6f4459207c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c57a6a07-877f-da62-877d-8c6f4459207c Address:c57a6a07-877f-da62-877d-8c6f4459207c} {Suffrage:Voter ID:server-0df7964f-a92f-07fa-4384-fe2dfdb1b998 Address:0df7964f-a92f-07fa-4384-fe2dfdb1b998} {Suffrage:Voter ID:server-5da4e77d-c945-0905-e78d-31f8c3ed8a56 Address:5da4e77d-c945-0905-e78d-31f8c3ed8a56}]" 2026-05-25T03:41:43.321Z [INFO] server-0df7964f-a92f-07fa-4384-fe2dfdb1b998: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c57a6a07-877f-da62-877d-8c6f4459207c Address:c57a6a07-877f-da62-877d-8c6f4459207c} {Suffrage:Voter ID:server-0df7964f-a92f-07fa-4384-fe2dfdb1b998 Address:0df7964f-a92f-07fa-4384-fe2dfdb1b998} {Suffrage:Voter ID:server-5da4e77d-c945-0905-e78d-31f8c3ed8a56 Address:5da4e77d-c945-0905-e78d-31f8c3ed8a56}]" 2026-05-25T03:41:43.321Z [INFO] server-c57a6a07-877f-da62-877d-8c6f4459207c: entering follower state: follower="Node at c57a6a07-877f-da62-877d-8c6f4459207c [Follower]" leader-address= leader-id= 2026-05-25T03:41:43.322Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c57a6a07-877f-da62-877d-8c6f4459207c Address:c57a6a07-877f-da62-877d-8c6f4459207c} {Suffrage:Voter ID:server-0df7964f-a92f-07fa-4384-fe2dfdb1b998 Address:0df7964f-a92f-07fa-4384-fe2dfdb1b998} {Suffrage:Voter ID:server-5da4e77d-c945-0905-e78d-31f8c3ed8a56 Address:5da4e77d-c945-0905-e78d-31f8c3ed8a56}]" 2026-05-25T03:41:43.322Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:43.322Z [INFO] server-0df7964f-a92f-07fa-4384-fe2dfdb1b998: entering follower state: follower="Node at 0df7964f-a92f-07fa-4384-fe2dfdb1b998 [Follower]" leader-address= leader-id= 2026-05-25T03:41:43.322Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: entering follower state: follower="Node at 5da4e77d-c945-0905-e78d-31f8c3ed8a56 [Follower]" leader-address= leader-id= 2026-05-25T03:41:44.106Z [WARN] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:44.106Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: entering candidate state: node="Node at 5da4e77d-c945-0905-e78d-31f8c3ed8a56 [Candidate]" term=2 2026-05-25T03:41:44.106Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: election won: term=2 tally=2 2026-05-25T03:41:44.106Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: entering leader state: leader="Node at 5da4e77d-c945-0905-e78d-31f8c3ed8a56 [Leader]" 2026-05-25T03:41:44.106Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: added peer, starting replication: peer=server-c57a6a07-877f-da62-877d-8c6f4459207c 2026-05-25T03:41:44.106Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: added peer, starting replication: peer=server-0df7964f-a92f-07fa-4384-fe2dfdb1b998 2026-05-25T03:41:44.106Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: pipelining replication: peer="{Voter server-0df7964f-a92f-07fa-4384-fe2dfdb1b998 0df7964f-a92f-07fa-4384-fe2dfdb1b998}" 2026-05-25T03:41:44.106Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: pipelining replication: peer="{Voter server-c57a6a07-877f-da62-877d-8c6f4459207c c57a6a07-877f-da62-877d-8c6f4459207c}" 2026-05-25T03:41:45.112Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:44.106434757 +0000 UTC m=+30.279410161 (1 nodes), 784.36608ms from start of poll, 784.7634ms from cluster start. Timeout at 2026-05-25 03:41:45.111437877 +0000 UTC m=+31.284413321, 1.00500316s after stability 2026-05-25T03:41:45.112Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: starting snapshot up to: index=12 2026-05-25T03:41:45.112Z [INFO] creating new snapshot: path=/tmp/raft924358576/snapshots/2-12-1779680505112.tmp 2026-05-25T03:41:45.113Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: snapshot complete up to: index=12 2026-05-25T03:41:45.114Z [INFO] creating new snapshot: path=/tmp/raft924358576/snapshots/2-23-1779680505114.tmp 2026-05-25T03:41:45.115Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: copied to local snapshot: bytes=71 2026-05-25T03:41:45.116Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: snapshot restore progress: id=2-23-1779680505114 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:45.116Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: removing all old logs from log store 2026-05-25T03:41:45.116Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: compacting logs: from=1 to=22 2026-05-25T03:41:45.116Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: restored user snapshot: index=23 2026-05-25T03:41:45.116Z [ERROR] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: failed to get log: index=22 error="log not found" 2026-05-25T03:41:45.116Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: aborting pipeline replication: peer="{Voter server-0df7964f-a92f-07fa-4384-fe2dfdb1b998 0df7964f-a92f-07fa-4384-fe2dfdb1b998}" 2026-05-25T03:41:45.116Z [ERROR] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: failed to get log: index=22 error="log not found" 2026-05-25T03:41:45.116Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: aborting pipeline replication: peer="{Voter server-c57a6a07-877f-da62-877d-8c6f4459207c c57a6a07-877f-da62-877d-8c6f4459207c}" 2026-05-25T03:41:45.123Z [ERROR] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: failed to get log: index=22 error="log not found" 2026-05-25T03:41:45.123Z [INFO] creating new snapshot: path=/tmp/raft1270784898/snapshots/2-23-1779680505123.tmp 2026-05-25T03:41:45.124Z [INFO] server-c57a6a07-877f-da62-877d-8c6f4459207c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:45.124Z [INFO] server-c57a6a07-877f-da62-877d-8c6f4459207c: copied to local snapshot: bytes=71 2026-05-25T03:41:45.124Z [INFO] server-c57a6a07-877f-da62-877d-8c6f4459207c: snapshot restore progress: id=2-23-1779680505123 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:45.124Z [INFO] server-c57a6a07-877f-da62-877d-8c6f4459207c: removing all old logs from log store 2026-05-25T03:41:45.124Z [INFO] server-c57a6a07-877f-da62-877d-8c6f4459207c: compacting logs: from=1 to=22 2026-05-25T03:41:45.124Z [INFO] server-c57a6a07-877f-da62-877d-8c6f4459207c: Installed remote snapshot 2026-05-25T03:41:45.124Z [WARN] server-c57a6a07-877f-da62-877d-8c6f4459207c: skipping application of old log: index=22 2026-05-25T03:41:45.124Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: pipelining replication: peer="{Voter server-c57a6a07-877f-da62-877d-8c6f4459207c c57a6a07-877f-da62-877d-8c6f4459207c}" 2026-05-25T03:41:45.124Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:45.126Z [ERROR] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: failed to get log: index=22 error="log not found" 2026-05-25T03:41:45.127Z [INFO] creating new snapshot: path=/tmp/raft1463436465/snapshots/2-23-1779680505127.tmp 2026-05-25T03:41:45.127Z [INFO] server-0df7964f-a92f-07fa-4384-fe2dfdb1b998: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:45.127Z [INFO] server-0df7964f-a92f-07fa-4384-fe2dfdb1b998: copied to local snapshot: bytes=71 2026-05-25T03:41:45.127Z [INFO] server-0df7964f-a92f-07fa-4384-fe2dfdb1b998: snapshot restore progress: id=2-23-1779680505127 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:45.128Z [INFO] server-0df7964f-a92f-07fa-4384-fe2dfdb1b998: removing all old logs from log store 2026-05-25T03:41:45.128Z [INFO] server-0df7964f-a92f-07fa-4384-fe2dfdb1b998: compacting logs: from=1 to=22 2026-05-25T03:41:45.128Z [INFO] server-0df7964f-a92f-07fa-4384-fe2dfdb1b998: Installed remote snapshot 2026-05-25T03:41:45.128Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: pipelining replication: peer="{Voter server-0df7964f-a92f-07fa-4384-fe2dfdb1b998 0df7964f-a92f-07fa-4384-fe2dfdb1b998}" 2026-05-25T03:41:46.130Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:45.124847157 +0000 UTC m=+31.297822601 (2 nodes), 1µs from start of poll, 1.80317584s from cluster start. Timeout at 2026-05-25 03:41:46.129856677 +0000 UTC m=+32.302832121, 1.00500952s after stability 2026-05-25T03:41:46.144Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: aborting pipeline replication: peer="{Voter server-c57a6a07-877f-da62-877d-8c6f4459207c c57a6a07-877f-da62-877d-8c6f4459207c}" 2026-05-25T03:41:46.144Z [INFO] server-5da4e77d-c945-0905-e78d-31f8c3ed8a56: aborting pipeline replication: peer="{Voter server-0df7964f-a92f-07fa-4384-fe2dfdb1b998 0df7964f-a92f-07fa-4384-fe2dfdb1b998}" === RUN TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true 2026-05-25T03:41:46.145Z [INFO] server-7aac5251-9c6a-4c1f-16a6-e3d2e11193cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7aac5251-9c6a-4c1f-16a6-e3d2e11193cd Address:7aac5251-9c6a-4c1f-16a6-e3d2e11193cd} {Suffrage:Voter ID:server-fb1b3c1a-d42f-30a2-9343-5595af8374ce Address:fb1b3c1a-d42f-30a2-9343-5595af8374ce} {Suffrage:Voter ID:server-03fd8e65-417b-2f42-7936-ca06d8ba8e86 Address:03fd8e65-417b-2f42-7936-ca06d8ba8e86}]" 2026-05-25T03:41:46.145Z [INFO] server-fb1b3c1a-d42f-30a2-9343-5595af8374ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7aac5251-9c6a-4c1f-16a6-e3d2e11193cd Address:7aac5251-9c6a-4c1f-16a6-e3d2e11193cd} {Suffrage:Voter ID:server-fb1b3c1a-d42f-30a2-9343-5595af8374ce Address:fb1b3c1a-d42f-30a2-9343-5595af8374ce} {Suffrage:Voter ID:server-03fd8e65-417b-2f42-7936-ca06d8ba8e86 Address:03fd8e65-417b-2f42-7936-ca06d8ba8e86}]" 2026-05-25T03:41:46.145Z [INFO] server-7aac5251-9c6a-4c1f-16a6-e3d2e11193cd: entering follower state: follower="Node at 7aac5251-9c6a-4c1f-16a6-e3d2e11193cd [Follower]" leader-address= leader-id= 2026-05-25T03:41:46.145Z [INFO] server-fb1b3c1a-d42f-30a2-9343-5595af8374ce: entering follower state: follower="Node at fb1b3c1a-d42f-30a2-9343-5595af8374ce [Follower]" leader-address= leader-id= 2026-05-25T03:41:46.145Z [INFO] server-03fd8e65-417b-2f42-7936-ca06d8ba8e86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7aac5251-9c6a-4c1f-16a6-e3d2e11193cd Address:7aac5251-9c6a-4c1f-16a6-e3d2e11193cd} {Suffrage:Voter ID:server-fb1b3c1a-d42f-30a2-9343-5595af8374ce Address:fb1b3c1a-d42f-30a2-9343-5595af8374ce} {Suffrage:Voter ID:server-03fd8e65-417b-2f42-7936-ca06d8ba8e86 Address:03fd8e65-417b-2f42-7936-ca06d8ba8e86}]" 2026-05-25T03:41:46.145Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:46.145Z [INFO] server-03fd8e65-417b-2f42-7936-ca06d8ba8e86: entering follower state: follower="Node at 03fd8e65-417b-2f42-7936-ca06d8ba8e86 [Follower]" leader-address= leader-id= 2026-05-25T03:41:46.667Z [WARN] server-fb1b3c1a-d42f-30a2-9343-5595af8374ce: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:46.668Z [INFO] server-fb1b3c1a-d42f-30a2-9343-5595af8374ce: entering candidate state: node="Node at fb1b3c1a-d42f-30a2-9343-5595af8374ce [Candidate]" term=2 2026-05-25T03:41:46.668Z [INFO] server-fb1b3c1a-d42f-30a2-9343-5595af8374ce: election won: term=2 tally=2 2026-05-25T03:41:46.668Z [INFO] server-fb1b3c1a-d42f-30a2-9343-5595af8374ce: entering leader state: leader="Node at fb1b3c1a-d42f-30a2-9343-5595af8374ce [Leader]" 2026-05-25T03:41:46.668Z [INFO] server-fb1b3c1a-d42f-30a2-9343-5595af8374ce: added peer, starting replication: peer=server-7aac5251-9c6a-4c1f-16a6-e3d2e11193cd 2026-05-25T03:41:46.668Z [INFO] server-fb1b3c1a-d42f-30a2-9343-5595af8374ce: added peer, starting replication: peer=server-03fd8e65-417b-2f42-7936-ca06d8ba8e86 2026-05-25T03:41:46.668Z [INFO] server-fb1b3c1a-d42f-30a2-9343-5595af8374ce: pipelining replication: peer="{Voter server-03fd8e65-417b-2f42-7936-ca06d8ba8e86 03fd8e65-417b-2f42-7936-ca06d8ba8e86}" 2026-05-25T03:41:46.668Z [INFO] server-fb1b3c1a-d42f-30a2-9343-5595af8374ce: pipelining replication: peer="{Voter server-7aac5251-9c6a-4c1f-16a6-e3d2e11193cd 7aac5251-9c6a-4c1f-16a6-e3d2e11193cd}" 2026-05-25T03:41:47.674Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:46.668245837 +0000 UTC m=+32.841221281 (1 nodes), 522.59712ms from start of poll, 523.04056ms from cluster start. Timeout at 2026-05-25 03:41:47.673253477 +0000 UTC m=+33.846228921, 1.00500764s after stability 2026-05-25T03:41:47.675Z [INFO] server-fb1b3c1a-d42f-30a2-9343-5595af8374ce: starting snapshot up to: index=12 2026-05-25T03:41:47.675Z [INFO] creating new snapshot: path=/tmp/raft905804598/snapshots/2-12-1779680507675.tmp 2026-05-25T03:41:47.676Z [INFO] server-fb1b3c1a-d42f-30a2-9343-5595af8374ce: snapshot complete up to: index=12 2026-05-25T03:41:47.677Z [INFO] server-2b506c92-06b4-526f-ce6c-fb957e844288: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b506c92-06b4-526f-ce6c-fb957e844288 Address:2b506c92-06b4-526f-ce6c-fb957e844288} {Suffrage:Voter ID:server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34 Address:be7f4246-ee4f-ae01-e7f4-8ba535a0ee34} {Suffrage:Voter ID:server-7ca976cd-3fc9-c918-164f-62f20d80be50 Address:7ca976cd-3fc9-c918-164f-62f20d80be50}]" 2026-05-25T03:41:47.677Z [INFO] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b506c92-06b4-526f-ce6c-fb957e844288 Address:2b506c92-06b4-526f-ce6c-fb957e844288} {Suffrage:Voter ID:server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34 Address:be7f4246-ee4f-ae01-e7f4-8ba535a0ee34} {Suffrage:Voter ID:server-7ca976cd-3fc9-c918-164f-62f20d80be50 Address:7ca976cd-3fc9-c918-164f-62f20d80be50}]" 2026-05-25T03:41:47.677Z [INFO] server-2b506c92-06b4-526f-ce6c-fb957e844288: entering follower state: follower="Node at 2b506c92-06b4-526f-ce6c-fb957e844288 [Follower]" leader-address= leader-id= 2026-05-25T03:41:47.677Z [INFO] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: entering follower state: follower="Node at be7f4246-ee4f-ae01-e7f4-8ba535a0ee34 [Follower]" leader-address= leader-id= 2026-05-25T03:41:47.677Z [INFO] server-7ca976cd-3fc9-c918-164f-62f20d80be50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b506c92-06b4-526f-ce6c-fb957e844288 Address:2b506c92-06b4-526f-ce6c-fb957e844288} {Suffrage:Voter ID:server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34 Address:be7f4246-ee4f-ae01-e7f4-8ba535a0ee34} {Suffrage:Voter ID:server-7ca976cd-3fc9-c918-164f-62f20d80be50 Address:7ca976cd-3fc9-c918-164f-62f20d80be50}]" 2026-05-25T03:41:47.678Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:47.678Z [INFO] server-7ca976cd-3fc9-c918-164f-62f20d80be50: entering follower state: follower="Node at 7ca976cd-3fc9-c918-164f-62f20d80be50 [Follower]" leader-address= leader-id= 2026-05-25T03:41:48.182Z [WARN] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:48.182Z [INFO] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: entering candidate state: node="Node at be7f4246-ee4f-ae01-e7f4-8ba535a0ee34 [Candidate]" term=2 2026-05-25T03:41:48.182Z [INFO] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: election won: term=2 tally=2 2026-05-25T03:41:48.182Z [INFO] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: entering leader state: leader="Node at be7f4246-ee4f-ae01-e7f4-8ba535a0ee34 [Leader]" 2026-05-25T03:41:48.182Z [INFO] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: added peer, starting replication: peer=server-2b506c92-06b4-526f-ce6c-fb957e844288 2026-05-25T03:41:48.182Z [INFO] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: added peer, starting replication: peer=server-7ca976cd-3fc9-c918-164f-62f20d80be50 2026-05-25T03:41:48.182Z [INFO] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: pipelining replication: peer="{Voter server-7ca976cd-3fc9-c918-164f-62f20d80be50 7ca976cd-3fc9-c918-164f-62f20d80be50}" 2026-05-25T03:41:48.182Z [INFO] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: pipelining replication: peer="{Voter server-2b506c92-06b4-526f-ce6c-fb957e844288 2b506c92-06b4-526f-ce6c-fb957e844288}" 2026-05-25T03:41:49.188Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:48.182760197 +0000 UTC m=+34.355735641 (1 nodes), 504.71752ms from start of poll, 505.14272ms from cluster start. Timeout at 2026-05-25 03:41:49.187761957 +0000 UTC m=+35.360737401, 1.00500176s after stability 2026-05-25T03:41:49.188Z [INFO] creating new snapshot: path=/tmp/raft3379193944/snapshots/2-14-1779680509188.tmp 2026-05-25T03:41:49.189Z [INFO] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: copied to local snapshot: bytes=71 2026-05-25T03:41:49.190Z [INFO] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: snapshot restore progress: id=2-14-1779680509188 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:49.190Z [INFO] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: restored user snapshot: index=14 2026-05-25T03:41:49.190Z [ERROR] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: failed to get log: index=3 error="log not found" 2026-05-25T03:41:49.190Z [ERROR] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: failed to get log: index=3 error="log not found" 2026-05-25T03:41:49.190Z [INFO] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: aborting pipeline replication: peer="{Voter server-2b506c92-06b4-526f-ce6c-fb957e844288 2b506c92-06b4-526f-ce6c-fb957e844288}" 2026-05-25T03:41:49.190Z [INFO] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: aborting pipeline replication: peer="{Voter server-7ca976cd-3fc9-c918-164f-62f20d80be50 7ca976cd-3fc9-c918-164f-62f20d80be50}" 2026-05-25T03:41:49.197Z [ERROR] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: failed to get log: index=3 error="log not found" 2026-05-25T03:41:49.198Z [INFO] creating new snapshot: path=/tmp/raft2026209350/snapshots/2-14-1779680509198.tmp 2026-05-25T03:41:49.198Z [ERROR] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: failed to get log: index=3 error="log not found" 2026-05-25T03:41:49.198Z [INFO] server-2b506c92-06b4-526f-ce6c-fb957e844288: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:49.198Z [INFO] creating new snapshot: path=/tmp/raft1354255936/snapshots/2-14-1779680509198.tmp 2026-05-25T03:41:49.198Z [INFO] server-2b506c92-06b4-526f-ce6c-fb957e844288: copied to local snapshot: bytes=71 2026-05-25T03:41:49.199Z [INFO] server-7ca976cd-3fc9-c918-164f-62f20d80be50: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:49.199Z [INFO] server-2b506c92-06b4-526f-ce6c-fb957e844288: snapshot restore progress: id=2-14-1779680509198 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:49.199Z [INFO] server-7ca976cd-3fc9-c918-164f-62f20d80be50: copied to local snapshot: bytes=71 2026-05-25T03:41:49.199Z [INFO] server-2b506c92-06b4-526f-ce6c-fb957e844288: Installed remote snapshot 2026-05-25T03:41:49.199Z [INFO] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: pipelining replication: peer="{Voter server-2b506c92-06b4-526f-ce6c-fb957e844288 2b506c92-06b4-526f-ce6c-fb957e844288}" 2026-05-25T03:41:49.199Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:49.199Z [INFO] server-7ca976cd-3fc9-c918-164f-62f20d80be50: snapshot restore progress: id=2-14-1779680509198 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:49.199Z [INFO] server-7ca976cd-3fc9-c918-164f-62f20d80be50: Installed remote snapshot 2026-05-25T03:41:49.199Z [INFO] server-be7f4246-ee4f-ae01-e7f4-8ba535a0ee34: pipelining replication: peer="{Voter server-7ca976cd-3fc9-c918-164f-62f20d80be50 7ca976cd-3fc9-c918-164f-62f20d80be50}" 2026-05-25T03:41:50.205Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:49.199564637 +0000 UTC m=+35.372540081 (2 nodes), 920ns from start of poll, 1.52194716s from cluster start. Timeout at 2026-05-25 03:41:50.204582837 +0000 UTC m=+36.377558281, 1.0050182s after stability 2026-05-25T03:41:50.218Z [INFO] server-fb1b3c1a-d42f-30a2-9343-5595af8374ce: aborting pipeline replication: peer="{Voter server-03fd8e65-417b-2f42-7936-ca06d8ba8e86 03fd8e65-417b-2f42-7936-ca06d8ba8e86}" 2026-05-25T03:41:50.218Z [INFO] server-fb1b3c1a-d42f-30a2-9343-5595af8374ce: aborting pipeline replication: peer="{Voter server-7aac5251-9c6a-4c1f-16a6-e3d2e11193cd 7aac5251-9c6a-4c1f-16a6-e3d2e11193cd}" === RUN TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true 2026-05-25T03:41:50.219Z [INFO] server-4c5d895d-2c7d-6275-2ad3-39c6de094504: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c5d895d-2c7d-6275-2ad3-39c6de094504 Address:4c5d895d-2c7d-6275-2ad3-39c6de094504} {Suffrage:Voter ID:server-6e40b8ec-89c9-11b8-7ef0-3955988484be Address:6e40b8ec-89c9-11b8-7ef0-3955988484be} {Suffrage:Voter ID:server-4bdbc2b9-c1e5-c86c-795c-b367e28d4e3a Address:4bdbc2b9-c1e5-c86c-795c-b367e28d4e3a}]" 2026-05-25T03:41:50.219Z [INFO] server-6e40b8ec-89c9-11b8-7ef0-3955988484be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c5d895d-2c7d-6275-2ad3-39c6de094504 Address:4c5d895d-2c7d-6275-2ad3-39c6de094504} {Suffrage:Voter ID:server-6e40b8ec-89c9-11b8-7ef0-3955988484be Address:6e40b8ec-89c9-11b8-7ef0-3955988484be} {Suffrage:Voter ID:server-4bdbc2b9-c1e5-c86c-795c-b367e28d4e3a Address:4bdbc2b9-c1e5-c86c-795c-b367e28d4e3a}]" 2026-05-25T03:41:50.219Z [INFO] server-4c5d895d-2c7d-6275-2ad3-39c6de094504: entering follower state: follower="Node at 4c5d895d-2c7d-6275-2ad3-39c6de094504 [Follower]" leader-address= leader-id= 2026-05-25T03:41:50.219Z [INFO] server-6e40b8ec-89c9-11b8-7ef0-3955988484be: entering follower state: follower="Node at 6e40b8ec-89c9-11b8-7ef0-3955988484be [Follower]" leader-address= leader-id= 2026-05-25T03:41:50.219Z [INFO] server-4bdbc2b9-c1e5-c86c-795c-b367e28d4e3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c5d895d-2c7d-6275-2ad3-39c6de094504 Address:4c5d895d-2c7d-6275-2ad3-39c6de094504} {Suffrage:Voter ID:server-6e40b8ec-89c9-11b8-7ef0-3955988484be Address:6e40b8ec-89c9-11b8-7ef0-3955988484be} {Suffrage:Voter ID:server-4bdbc2b9-c1e5-c86c-795c-b367e28d4e3a Address:4bdbc2b9-c1e5-c86c-795c-b367e28d4e3a}]" 2026-05-25T03:41:50.219Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:50.219Z [INFO] server-4bdbc2b9-c1e5-c86c-795c-b367e28d4e3a: entering follower state: follower="Node at 4bdbc2b9-c1e5-c86c-795c-b367e28d4e3a [Follower]" leader-address= leader-id= 2026-05-25T03:41:50.837Z [WARN] server-6e40b8ec-89c9-11b8-7ef0-3955988484be: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:50.837Z [INFO] server-6e40b8ec-89c9-11b8-7ef0-3955988484be: entering candidate state: node="Node at 6e40b8ec-89c9-11b8-7ef0-3955988484be [Candidate]" term=2 2026-05-25T03:41:50.837Z [INFO] server-6e40b8ec-89c9-11b8-7ef0-3955988484be: election won: term=2 tally=2 2026-05-25T03:41:50.837Z [INFO] server-6e40b8ec-89c9-11b8-7ef0-3955988484be: entering leader state: leader="Node at 6e40b8ec-89c9-11b8-7ef0-3955988484be [Leader]" 2026-05-25T03:41:50.837Z [INFO] server-6e40b8ec-89c9-11b8-7ef0-3955988484be: added peer, starting replication: peer=server-4c5d895d-2c7d-6275-2ad3-39c6de094504 2026-05-25T03:41:50.837Z [INFO] server-6e40b8ec-89c9-11b8-7ef0-3955988484be: added peer, starting replication: peer=server-4bdbc2b9-c1e5-c86c-795c-b367e28d4e3a 2026-05-25T03:41:50.837Z [INFO] server-6e40b8ec-89c9-11b8-7ef0-3955988484be: pipelining replication: peer="{Voter server-4bdbc2b9-c1e5-c86c-795c-b367e28d4e3a 4bdbc2b9-c1e5-c86c-795c-b367e28d4e3a}" 2026-05-25T03:41:50.837Z [INFO] server-6e40b8ec-89c9-11b8-7ef0-3955988484be: pipelining replication: peer="{Voter server-4c5d895d-2c7d-6275-2ad3-39c6de094504 4c5d895d-2c7d-6275-2ad3-39c6de094504}" 2026-05-25T03:41:51.843Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:50.837535557 +0000 UTC m=+37.010511001 (1 nodes), 618.15972ms from start of poll, 618.60832ms from cluster start. Timeout at 2026-05-25 03:41:51.842537917 +0000 UTC m=+38.015513401, 1.0050024s after stability 2026-05-25T03:41:51.843Z [INFO] server-6e40b8ec-89c9-11b8-7ef0-3955988484be: starting snapshot up to: index=12 2026-05-25T03:41:51.843Z [INFO] creating new snapshot: path=/tmp/raft2761061383/snapshots/2-12-1779680511843.tmp 2026-05-25T03:41:51.844Z [INFO] server-6e40b8ec-89c9-11b8-7ef0-3955988484be: snapshot complete up to: index=12 2026-05-25T03:41:51.845Z [INFO] server-474f375e-2b3e-bbc1-3ef1-82aba6c2098e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-474f375e-2b3e-bbc1-3ef1-82aba6c2098e Address:474f375e-2b3e-bbc1-3ef1-82aba6c2098e} {Suffrage:Voter ID:server-d264731d-2730-1866-e9ff-dc98ca5fc22d Address:d264731d-2730-1866-e9ff-dc98ca5fc22d} {Suffrage:Voter ID:server-e3bc5e27-235d-98bd-1cf4-292e84ee086c Address:e3bc5e27-235d-98bd-1cf4-292e84ee086c}]" 2026-05-25T03:41:51.845Z [INFO] server-474f375e-2b3e-bbc1-3ef1-82aba6c2098e: entering follower state: follower="Node at 474f375e-2b3e-bbc1-3ef1-82aba6c2098e [Follower]" leader-address= leader-id= 2026-05-25T03:41:51.845Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-474f375e-2b3e-bbc1-3ef1-82aba6c2098e Address:474f375e-2b3e-bbc1-3ef1-82aba6c2098e} {Suffrage:Voter ID:server-d264731d-2730-1866-e9ff-dc98ca5fc22d Address:d264731d-2730-1866-e9ff-dc98ca5fc22d} {Suffrage:Voter ID:server-e3bc5e27-235d-98bd-1cf4-292e84ee086c Address:e3bc5e27-235d-98bd-1cf4-292e84ee086c}]" 2026-05-25T03:41:51.845Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: entering follower state: follower="Node at d264731d-2730-1866-e9ff-dc98ca5fc22d [Follower]" leader-address= leader-id= 2026-05-25T03:41:51.845Z [INFO] server-e3bc5e27-235d-98bd-1cf4-292e84ee086c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-474f375e-2b3e-bbc1-3ef1-82aba6c2098e Address:474f375e-2b3e-bbc1-3ef1-82aba6c2098e} {Suffrage:Voter ID:server-d264731d-2730-1866-e9ff-dc98ca5fc22d Address:d264731d-2730-1866-e9ff-dc98ca5fc22d} {Suffrage:Voter ID:server-e3bc5e27-235d-98bd-1cf4-292e84ee086c Address:e3bc5e27-235d-98bd-1cf4-292e84ee086c}]" 2026-05-25T03:41:51.845Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:51.845Z [INFO] server-e3bc5e27-235d-98bd-1cf4-292e84ee086c: entering follower state: follower="Node at e3bc5e27-235d-98bd-1cf4-292e84ee086c [Follower]" leader-address= leader-id= 2026-05-25T03:41:52.516Z [WARN] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:52.516Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: entering candidate state: node="Node at d264731d-2730-1866-e9ff-dc98ca5fc22d [Candidate]" term=2 2026-05-25T03:41:52.516Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: election won: term=2 tally=2 2026-05-25T03:41:52.516Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: entering leader state: leader="Node at d264731d-2730-1866-e9ff-dc98ca5fc22d [Leader]" 2026-05-25T03:41:52.516Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: added peer, starting replication: peer=server-474f375e-2b3e-bbc1-3ef1-82aba6c2098e 2026-05-25T03:41:52.516Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: added peer, starting replication: peer=server-e3bc5e27-235d-98bd-1cf4-292e84ee086c 2026-05-25T03:41:52.516Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: pipelining replication: peer="{Voter server-474f375e-2b3e-bbc1-3ef1-82aba6c2098e 474f375e-2b3e-bbc1-3ef1-82aba6c2098e}" 2026-05-25T03:41:52.516Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: pipelining replication: peer="{Voter server-e3bc5e27-235d-98bd-1cf4-292e84ee086c e3bc5e27-235d-98bd-1cf4-292e84ee086c}" 2026-05-25T03:41:53.522Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:52.516842437 +0000 UTC m=+38.689817881 (1 nodes), 671.27844ms from start of poll, 671.82308ms from cluster start. Timeout at 2026-05-25 03:41:53.521845357 +0000 UTC m=+39.694820801, 1.00500292s after stability 2026-05-25T03:41:53.522Z [INFO] creating new snapshot: path=/tmp/raft652263253/snapshots/2-14-1779680513522.tmp 2026-05-25T03:41:53.523Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: copied to local snapshot: bytes=71 2026-05-25T03:41:53.523Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: snapshot restore progress: id=2-14-1779680513522 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:53.523Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: removing all old logs from log store 2026-05-25T03:41:53.523Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: compacting logs: from=1 to=2 2026-05-25T03:41:53.523Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: restored user snapshot: index=14 2026-05-25T03:41:53.523Z [ERROR] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: failed to get log: index=2 error="log not found" 2026-05-25T03:41:53.523Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: aborting pipeline replication: peer="{Voter server-e3bc5e27-235d-98bd-1cf4-292e84ee086c e3bc5e27-235d-98bd-1cf4-292e84ee086c}" 2026-05-25T03:41:53.523Z [ERROR] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: failed to get log: index=2 error="log not found" 2026-05-25T03:41:53.523Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: aborting pipeline replication: peer="{Voter server-474f375e-2b3e-bbc1-3ef1-82aba6c2098e 474f375e-2b3e-bbc1-3ef1-82aba6c2098e}" 2026-05-25T03:41:53.531Z [ERROR] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: failed to get log: index=2 error="log not found" 2026-05-25T03:41:53.531Z [ERROR] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: failed to get log: index=2 error="log not found" 2026-05-25T03:41:53.532Z [INFO] creating new snapshot: path=/tmp/raft2584851033/snapshots/2-14-1779680513532.tmp 2026-05-25T03:41:53.532Z [INFO] creating new snapshot: path=/tmp/raft307515718/snapshots/2-14-1779680513532.tmp 2026-05-25T03:41:53.532Z [INFO] server-474f375e-2b3e-bbc1-3ef1-82aba6c2098e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:53.532Z [INFO] server-e3bc5e27-235d-98bd-1cf4-292e84ee086c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:53.532Z [INFO] server-474f375e-2b3e-bbc1-3ef1-82aba6c2098e: copied to local snapshot: bytes=71 2026-05-25T03:41:53.532Z [INFO] server-e3bc5e27-235d-98bd-1cf4-292e84ee086c: copied to local snapshot: bytes=71 2026-05-25T03:41:53.532Z [INFO] server-474f375e-2b3e-bbc1-3ef1-82aba6c2098e: snapshot restore progress: id=2-14-1779680513532 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:53.532Z [INFO] server-474f375e-2b3e-bbc1-3ef1-82aba6c2098e: removing all old logs from log store 2026-05-25T03:41:53.532Z [INFO] server-474f375e-2b3e-bbc1-3ef1-82aba6c2098e: compacting logs: from=1 to=2 2026-05-25T03:41:53.532Z [INFO] server-474f375e-2b3e-bbc1-3ef1-82aba6c2098e: Installed remote snapshot 2026-05-25T03:41:53.532Z [INFO] server-e3bc5e27-235d-98bd-1cf4-292e84ee086c: snapshot restore progress: id=2-14-1779680513532 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:53.532Z [INFO] server-e3bc5e27-235d-98bd-1cf4-292e84ee086c: removing all old logs from log store 2026-05-25T03:41:53.532Z [INFO] server-e3bc5e27-235d-98bd-1cf4-292e84ee086c: compacting logs: from=1 to=2 2026-05-25T03:41:53.532Z [INFO] server-e3bc5e27-235d-98bd-1cf4-292e84ee086c: Installed remote snapshot 2026-05-25T03:41:53.533Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: pipelining replication: peer="{Voter server-474f375e-2b3e-bbc1-3ef1-82aba6c2098e 474f375e-2b3e-bbc1-3ef1-82aba6c2098e}" 2026-05-25T03:41:53.533Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:53.533Z [INFO] server-d264731d-2730-1866-e9ff-dc98ca5fc22d: pipelining replication: peer="{Voter server-e3bc5e27-235d-98bd-1cf4-292e84ee086c e3bc5e27-235d-98bd-1cf4-292e84ee086c}" 2026-05-25T03:41:54.539Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:53.533151197 +0000 UTC m=+39.706126641 (2 nodes), 1.16µs from start of poll, 1.68813184s from cluster start. Timeout at 2026-05-25 03:41:54.538165357 +0000 UTC m=+40.711140801, 1.00501416s after stability 2026-05-25T03:41:54.550Z [INFO] server-6e40b8ec-89c9-11b8-7ef0-3955988484be: aborting pipeline replication: peer="{Voter server-4bdbc2b9-c1e5-c86c-795c-b367e28d4e3a 4bdbc2b9-c1e5-c86c-795c-b367e28d4e3a}" 2026-05-25T03:41:54.550Z [INFO] server-6e40b8ec-89c9-11b8-7ef0-3955988484be: aborting pipeline replication: peer="{Voter server-4c5d895d-2c7d-6275-2ad3-39c6de094504 4c5d895d-2c7d-6275-2ad3-39c6de094504}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false 2026-05-25T03:41:54.551Z [INFO] server-17900a71-49d8-93f4-3a13-5ef105b46bd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17900a71-49d8-93f4-3a13-5ef105b46bd9 Address:17900a71-49d8-93f4-3a13-5ef105b46bd9} {Suffrage:Voter ID:server-08f0e3ce-ac05-e3a7-fa86-6d079203cf84 Address:08f0e3ce-ac05-e3a7-fa86-6d079203cf84} {Suffrage:Voter ID:server-dec4e6bc-d3d3-7046-3c34-640a021e6a53 Address:dec4e6bc-d3d3-7046-3c34-640a021e6a53}]" 2026-05-25T03:41:54.552Z [INFO] server-17900a71-49d8-93f4-3a13-5ef105b46bd9: entering follower state: follower="Node at 17900a71-49d8-93f4-3a13-5ef105b46bd9 [Follower]" leader-address= leader-id= 2026-05-25T03:41:54.552Z [INFO] server-08f0e3ce-ac05-e3a7-fa86-6d079203cf84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17900a71-49d8-93f4-3a13-5ef105b46bd9 Address:17900a71-49d8-93f4-3a13-5ef105b46bd9} {Suffrage:Voter ID:server-08f0e3ce-ac05-e3a7-fa86-6d079203cf84 Address:08f0e3ce-ac05-e3a7-fa86-6d079203cf84} {Suffrage:Voter ID:server-dec4e6bc-d3d3-7046-3c34-640a021e6a53 Address:dec4e6bc-d3d3-7046-3c34-640a021e6a53}]" 2026-05-25T03:41:54.552Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17900a71-49d8-93f4-3a13-5ef105b46bd9 Address:17900a71-49d8-93f4-3a13-5ef105b46bd9} {Suffrage:Voter ID:server-08f0e3ce-ac05-e3a7-fa86-6d079203cf84 Address:08f0e3ce-ac05-e3a7-fa86-6d079203cf84} {Suffrage:Voter ID:server-dec4e6bc-d3d3-7046-3c34-640a021e6a53 Address:dec4e6bc-d3d3-7046-3c34-640a021e6a53}]" 2026-05-25T03:41:54.552Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:54.552Z [INFO] server-08f0e3ce-ac05-e3a7-fa86-6d079203cf84: entering follower state: follower="Node at 08f0e3ce-ac05-e3a7-fa86-6d079203cf84 [Follower]" leader-address= leader-id= 2026-05-25T03:41:54.552Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: entering follower state: follower="Node at dec4e6bc-d3d3-7046-3c34-640a021e6a53 [Follower]" leader-address= leader-id= 2026-05-25T03:41:55.137Z [WARN] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:55.137Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: entering candidate state: node="Node at dec4e6bc-d3d3-7046-3c34-640a021e6a53 [Candidate]" term=2 2026-05-25T03:41:55.137Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: election won: term=2 tally=2 2026-05-25T03:41:55.138Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: entering leader state: leader="Node at dec4e6bc-d3d3-7046-3c34-640a021e6a53 [Leader]" 2026-05-25T03:41:55.138Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: added peer, starting replication: peer=server-17900a71-49d8-93f4-3a13-5ef105b46bd9 2026-05-25T03:41:55.138Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: added peer, starting replication: peer=server-08f0e3ce-ac05-e3a7-fa86-6d079203cf84 2026-05-25T03:41:55.138Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: pipelining replication: peer="{Voter server-08f0e3ce-ac05-e3a7-fa86-6d079203cf84 08f0e3ce-ac05-e3a7-fa86-6d079203cf84}" 2026-05-25T03:41:55.138Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: pipelining replication: peer="{Voter server-17900a71-49d8-93f4-3a13-5ef105b46bd9 17900a71-49d8-93f4-3a13-5ef105b46bd9}" 2026-05-25T03:41:56.143Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:55.138144677 +0000 UTC m=+41.311120121 (1 nodes), 585.21912ms from start of poll, 586.291ms from cluster start. Timeout at 2026-05-25 03:41:56.143148157 +0000 UTC m=+42.316123601, 1.00500348s after stability 2026-05-25T03:41:56.144Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: starting snapshot up to: index=12 2026-05-25T03:41:56.144Z [INFO] creating new snapshot: path=/tmp/raft712001525/snapshots/2-12-1779680516144.tmp 2026-05-25T03:41:56.145Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: snapshot complete up to: index=12 2026-05-25T03:41:56.145Z [INFO] creating new snapshot: path=/tmp/raft712001525/snapshots/2-23-1779680516145.tmp 2026-05-25T03:41:56.146Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: copied to local snapshot: bytes=71 2026-05-25T03:41:56.146Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: snapshot restore progress: id=2-23-1779680516145 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:56.146Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: restored user snapshot: index=23 2026-05-25T03:41:56.146Z [ERROR] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: failed to get log: index=23 error="log not found" 2026-05-25T03:41:56.146Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: aborting pipeline replication: peer="{Voter server-08f0e3ce-ac05-e3a7-fa86-6d079203cf84 08f0e3ce-ac05-e3a7-fa86-6d079203cf84}" 2026-05-25T03:41:56.146Z [ERROR] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: failed to get log: index=23 error="log not found" 2026-05-25T03:41:56.146Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: aborting pipeline replication: peer="{Voter server-17900a71-49d8-93f4-3a13-5ef105b46bd9 17900a71-49d8-93f4-3a13-5ef105b46bd9}" 2026-05-25T03:41:56.152Z [ERROR] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: failed to get log: index=23 error="log not found" 2026-05-25T03:41:56.153Z [INFO] creating new snapshot: path=/tmp/raft2411287249/snapshots/2-23-1779680516153.tmp 2026-05-25T03:41:56.153Z [INFO] server-08f0e3ce-ac05-e3a7-fa86-6d079203cf84: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:56.153Z [INFO] server-08f0e3ce-ac05-e3a7-fa86-6d079203cf84: copied to local snapshot: bytes=71 2026-05-25T03:41:56.153Z [INFO] server-08f0e3ce-ac05-e3a7-fa86-6d079203cf84: snapshot restore progress: id=2-23-1779680516153 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:56.153Z [INFO] server-08f0e3ce-ac05-e3a7-fa86-6d079203cf84: Installed remote snapshot 2026-05-25T03:41:56.153Z [WARN] server-08f0e3ce-ac05-e3a7-fa86-6d079203cf84: skipping application of old log: index=22 2026-05-25T03:41:56.153Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: pipelining replication: peer="{Voter server-08f0e3ce-ac05-e3a7-fa86-6d079203cf84 08f0e3ce-ac05-e3a7-fa86-6d079203cf84}" 2026-05-25T03:41:56.154Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:56.156Z [ERROR] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: failed to get log: index=23 error="log not found" 2026-05-25T03:41:56.157Z [INFO] creating new snapshot: path=/tmp/raft82498442/snapshots/2-23-1779680516157.tmp 2026-05-25T03:41:56.157Z [INFO] server-17900a71-49d8-93f4-3a13-5ef105b46bd9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:56.157Z [INFO] server-17900a71-49d8-93f4-3a13-5ef105b46bd9: copied to local snapshot: bytes=71 2026-05-25T03:41:56.157Z [INFO] server-17900a71-49d8-93f4-3a13-5ef105b46bd9: snapshot restore progress: id=2-23-1779680516157 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:56.157Z [INFO] server-17900a71-49d8-93f4-3a13-5ef105b46bd9: Installed remote snapshot 2026-05-25T03:41:56.158Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: pipelining replication: peer="{Voter server-17900a71-49d8-93f4-3a13-5ef105b46bd9 17900a71-49d8-93f4-3a13-5ef105b46bd9}" 2026-05-25T03:41:57.159Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:56.154078357 +0000 UTC m=+42.327053801 (2 nodes), 880ns from start of poll, 1.60222468s from cluster start. Timeout at 2026-05-25 03:41:57.159091237 +0000 UTC m=+43.332066681, 1.00501288s after stability 2026-05-25T03:41:57.171Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: aborting pipeline replication: peer="{Voter server-17900a71-49d8-93f4-3a13-5ef105b46bd9 17900a71-49d8-93f4-3a13-5ef105b46bd9}" 2026-05-25T03:41:57.171Z [INFO] server-dec4e6bc-d3d3-7046-3c34-640a021e6a53: aborting pipeline replication: peer="{Voter server-08f0e3ce-ac05-e3a7-fa86-6d079203cf84 08f0e3ce-ac05-e3a7-fa86-6d079203cf84}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false 2026-05-25T03:41:57.172Z [INFO] server-25e4043e-ad06-8839-6dfa-491a2d85ea8d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25e4043e-ad06-8839-6dfa-491a2d85ea8d Address:25e4043e-ad06-8839-6dfa-491a2d85ea8d} {Suffrage:Voter ID:server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5 Address:8990cbe0-d09a-bb1b-0dd8-b13af6c304f5} {Suffrage:Voter ID:server-5430d94a-ab36-0827-a802-6a4238246636 Address:5430d94a-ab36-0827-a802-6a4238246636}]" 2026-05-25T03:41:57.172Z [INFO] server-25e4043e-ad06-8839-6dfa-491a2d85ea8d: entering follower state: follower="Node at 25e4043e-ad06-8839-6dfa-491a2d85ea8d [Follower]" leader-address= leader-id= 2026-05-25T03:41:57.172Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25e4043e-ad06-8839-6dfa-491a2d85ea8d Address:25e4043e-ad06-8839-6dfa-491a2d85ea8d} {Suffrage:Voter ID:server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5 Address:8990cbe0-d09a-bb1b-0dd8-b13af6c304f5} {Suffrage:Voter ID:server-5430d94a-ab36-0827-a802-6a4238246636 Address:5430d94a-ab36-0827-a802-6a4238246636}]" 2026-05-25T03:41:57.172Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: entering follower state: follower="Node at 8990cbe0-d09a-bb1b-0dd8-b13af6c304f5 [Follower]" leader-address= leader-id= 2026-05-25T03:41:57.173Z [INFO] server-5430d94a-ab36-0827-a802-6a4238246636: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-25e4043e-ad06-8839-6dfa-491a2d85ea8d Address:25e4043e-ad06-8839-6dfa-491a2d85ea8d} {Suffrage:Voter ID:server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5 Address:8990cbe0-d09a-bb1b-0dd8-b13af6c304f5} {Suffrage:Voter ID:server-5430d94a-ab36-0827-a802-6a4238246636 Address:5430d94a-ab36-0827-a802-6a4238246636}]" 2026-05-25T03:41:57.173Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:57.173Z [INFO] server-5430d94a-ab36-0827-a802-6a4238246636: entering follower state: follower="Node at 5430d94a-ab36-0827-a802-6a4238246636 [Follower]" leader-address= leader-id= 2026-05-25T03:41:57.688Z [WARN] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:41:57.688Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: entering candidate state: node="Node at 8990cbe0-d09a-bb1b-0dd8-b13af6c304f5 [Candidate]" term=2 2026-05-25T03:41:57.688Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: election won: term=2 tally=2 2026-05-25T03:41:57.688Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: entering leader state: leader="Node at 8990cbe0-d09a-bb1b-0dd8-b13af6c304f5 [Leader]" 2026-05-25T03:41:57.688Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: added peer, starting replication: peer=server-25e4043e-ad06-8839-6dfa-491a2d85ea8d 2026-05-25T03:41:57.688Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: added peer, starting replication: peer=server-5430d94a-ab36-0827-a802-6a4238246636 2026-05-25T03:41:57.688Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: pipelining replication: peer="{Voter server-5430d94a-ab36-0827-a802-6a4238246636 5430d94a-ab36-0827-a802-6a4238246636}" 2026-05-25T03:41:57.688Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: pipelining replication: peer="{Voter server-25e4043e-ad06-8839-6dfa-491a2d85ea8d 25e4043e-ad06-8839-6dfa-491a2d85ea8d}" 2026-05-25T03:41:58.694Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:41:57.688700197 +0000 UTC m=+43.861675641 (1 nodes), 515.65236ms from start of poll, 516.08496ms from cluster start. Timeout at 2026-05-25 03:41:58.693712837 +0000 UTC m=+44.866688321, 1.00501268s after stability 2026-05-25T03:41:58.694Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: starting snapshot up to: index=12 2026-05-25T03:41:58.694Z [INFO] creating new snapshot: path=/tmp/raft1844212567/snapshots/2-12-1779680518694.tmp 2026-05-25T03:41:58.695Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: snapshot complete up to: index=12 2026-05-25T03:41:58.697Z [INFO] creating new snapshot: path=/tmp/raft1844212567/snapshots/2-23-1779680518697.tmp 2026-05-25T03:41:58.698Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: copied to local snapshot: bytes=71 2026-05-25T03:41:58.698Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: snapshot restore progress: id=2-23-1779680518697 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:58.698Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: removing all old logs from log store 2026-05-25T03:41:58.698Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: compacting logs: from=1 to=22 2026-05-25T03:41:58.698Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: restored user snapshot: index=23 2026-05-25T03:41:58.698Z [ERROR] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: failed to get log: index=22 error="log not found" 2026-05-25T03:41:58.698Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: aborting pipeline replication: peer="{Voter server-5430d94a-ab36-0827-a802-6a4238246636 5430d94a-ab36-0827-a802-6a4238246636}" 2026-05-25T03:41:58.698Z [ERROR] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: failed to get log: index=22 error="log not found" 2026-05-25T03:41:58.698Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: aborting pipeline replication: peer="{Voter server-25e4043e-ad06-8839-6dfa-491a2d85ea8d 25e4043e-ad06-8839-6dfa-491a2d85ea8d}" 2026-05-25T03:41:58.706Z [ERROR] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: failed to get log: index=22 error="log not found" 2026-05-25T03:41:58.707Z [INFO] creating new snapshot: path=/tmp/raft3704020522/snapshots/2-23-1779680518707.tmp 2026-05-25T03:41:58.707Z [ERROR] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: failed to get log: index=22 error="log not found" 2026-05-25T03:41:58.707Z [INFO] server-25e4043e-ad06-8839-6dfa-491a2d85ea8d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:58.707Z [INFO] server-25e4043e-ad06-8839-6dfa-491a2d85ea8d: copied to local snapshot: bytes=71 2026-05-25T03:41:58.708Z [INFO] creating new snapshot: path=/tmp/raft1219623050/snapshots/2-23-1779680518708.tmp 2026-05-25T03:41:58.708Z [INFO] server-25e4043e-ad06-8839-6dfa-491a2d85ea8d: snapshot restore progress: id=2-23-1779680518707 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:58.708Z [INFO] server-25e4043e-ad06-8839-6dfa-491a2d85ea8d: removing all old logs from log store 2026-05-25T03:41:58.708Z [INFO] server-25e4043e-ad06-8839-6dfa-491a2d85ea8d: compacting logs: from=1 to=22 2026-05-25T03:41:58.708Z [INFO] server-25e4043e-ad06-8839-6dfa-491a2d85ea8d: Installed remote snapshot 2026-05-25T03:41:58.708Z [INFO] server-5430d94a-ab36-0827-a802-6a4238246636: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:58.708Z [WARN] server-25e4043e-ad06-8839-6dfa-491a2d85ea8d: skipping application of old log: index=22 2026-05-25T03:41:58.708Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: pipelining replication: peer="{Voter server-25e4043e-ad06-8839-6dfa-491a2d85ea8d 25e4043e-ad06-8839-6dfa-491a2d85ea8d}" 2026-05-25T03:41:58.708Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:41:58.708Z [INFO] server-5430d94a-ab36-0827-a802-6a4238246636: copied to local snapshot: bytes=71 2026-05-25T03:41:58.708Z [INFO] server-5430d94a-ab36-0827-a802-6a4238246636: snapshot restore progress: id=2-23-1779680518708 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:41:58.708Z [INFO] server-5430d94a-ab36-0827-a802-6a4238246636: removing all old logs from log store 2026-05-25T03:41:58.708Z [INFO] server-5430d94a-ab36-0827-a802-6a4238246636: compacting logs: from=1 to=22 2026-05-25T03:41:58.708Z [INFO] server-5430d94a-ab36-0827-a802-6a4238246636: Installed remote snapshot 2026-05-25T03:41:58.709Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: pipelining replication: peer="{Voter server-5430d94a-ab36-0827-a802-6a4238246636 5430d94a-ab36-0827-a802-6a4238246636}" 2026-05-25T03:41:59.714Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:41:58.708563917 +0000 UTC m=+44.881539321 (2 nodes), 840ns from start of poll, 1.53594864s from cluster start. Timeout at 2026-05-25 03:41:59.713580277 +0000 UTC m=+45.886555721, 1.0050164s after stability 2026-05-25T03:41:59.726Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: aborting pipeline replication: peer="{Voter server-5430d94a-ab36-0827-a802-6a4238246636 5430d94a-ab36-0827-a802-6a4238246636}" 2026-05-25T03:41:59.726Z [INFO] server-8990cbe0-d09a-bb1b-0dd8-b13af6c304f5: aborting pipeline replication: peer="{Voter server-25e4043e-ad06-8839-6dfa-491a2d85ea8d 25e4043e-ad06-8839-6dfa-491a2d85ea8d}" === RUN TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true 2026-05-25T03:41:59.728Z [INFO] server-9c53ac3f-17b0-dd31-5727-9abd37b7649e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c53ac3f-17b0-dd31-5727-9abd37b7649e Address:9c53ac3f-17b0-dd31-5727-9abd37b7649e} {Suffrage:Voter ID:server-20e7d4be-f68a-f1d8-ee87-8d156a8c1692 Address:20e7d4be-f68a-f1d8-ee87-8d156a8c1692} {Suffrage:Voter ID:server-f06e2665-c718-1a39-7ab9-9fa0409404fd Address:f06e2665-c718-1a39-7ab9-9fa0409404fd}]" 2026-05-25T03:41:59.728Z [INFO] server-9c53ac3f-17b0-dd31-5727-9abd37b7649e: entering follower state: follower="Node at 9c53ac3f-17b0-dd31-5727-9abd37b7649e [Follower]" leader-address= leader-id= 2026-05-25T03:41:59.729Z [INFO] server-20e7d4be-f68a-f1d8-ee87-8d156a8c1692: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c53ac3f-17b0-dd31-5727-9abd37b7649e Address:9c53ac3f-17b0-dd31-5727-9abd37b7649e} {Suffrage:Voter ID:server-20e7d4be-f68a-f1d8-ee87-8d156a8c1692 Address:20e7d4be-f68a-f1d8-ee87-8d156a8c1692} {Suffrage:Voter ID:server-f06e2665-c718-1a39-7ab9-9fa0409404fd Address:f06e2665-c718-1a39-7ab9-9fa0409404fd}]" 2026-05-25T03:41:59.729Z [INFO] server-20e7d4be-f68a-f1d8-ee87-8d156a8c1692: entering follower state: follower="Node at 20e7d4be-f68a-f1d8-ee87-8d156a8c1692 [Follower]" leader-address= leader-id= 2026-05-25T03:41:59.729Z [INFO] server-f06e2665-c718-1a39-7ab9-9fa0409404fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c53ac3f-17b0-dd31-5727-9abd37b7649e Address:9c53ac3f-17b0-dd31-5727-9abd37b7649e} {Suffrage:Voter ID:server-20e7d4be-f68a-f1d8-ee87-8d156a8c1692 Address:20e7d4be-f68a-f1d8-ee87-8d156a8c1692} {Suffrage:Voter ID:server-f06e2665-c718-1a39-7ab9-9fa0409404fd Address:f06e2665-c718-1a39-7ab9-9fa0409404fd}]" 2026-05-25T03:41:59.729Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:41:59.729Z [INFO] server-f06e2665-c718-1a39-7ab9-9fa0409404fd: entering follower state: follower="Node at f06e2665-c718-1a39-7ab9-9fa0409404fd [Follower]" leader-address= leader-id= 2026-05-25T03:42:00.677Z [WARN] server-20e7d4be-f68a-f1d8-ee87-8d156a8c1692: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:00.677Z [INFO] server-20e7d4be-f68a-f1d8-ee87-8d156a8c1692: entering candidate state: node="Node at 20e7d4be-f68a-f1d8-ee87-8d156a8c1692 [Candidate]" term=2 2026-05-25T03:42:00.677Z [INFO] server-20e7d4be-f68a-f1d8-ee87-8d156a8c1692: election won: term=2 tally=2 2026-05-25T03:42:00.677Z [INFO] server-20e7d4be-f68a-f1d8-ee87-8d156a8c1692: entering leader state: leader="Node at 20e7d4be-f68a-f1d8-ee87-8d156a8c1692 [Leader]" 2026-05-25T03:42:00.677Z [INFO] server-20e7d4be-f68a-f1d8-ee87-8d156a8c1692: added peer, starting replication: peer=server-9c53ac3f-17b0-dd31-5727-9abd37b7649e 2026-05-25T03:42:00.677Z [INFO] server-20e7d4be-f68a-f1d8-ee87-8d156a8c1692: added peer, starting replication: peer=server-f06e2665-c718-1a39-7ab9-9fa0409404fd 2026-05-25T03:42:00.678Z [INFO] server-20e7d4be-f68a-f1d8-ee87-8d156a8c1692: pipelining replication: peer="{Voter server-9c53ac3f-17b0-dd31-5727-9abd37b7649e 9c53ac3f-17b0-dd31-5727-9abd37b7649e}" 2026-05-25T03:42:00.678Z [INFO] server-20e7d4be-f68a-f1d8-ee87-8d156a8c1692: pipelining replication: peer="{Voter server-f06e2665-c718-1a39-7ab9-9fa0409404fd f06e2665-c718-1a39-7ab9-9fa0409404fd}" 2026-05-25T03:42:01.683Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:00.678017037 +0000 UTC m=+46.850992481 (1 nodes), 948.77312ms from start of poll, 950.21888ms from cluster start. Timeout at 2026-05-25 03:42:01.683022037 +0000 UTC m=+47.855997481, 1.005005s after stability 2026-05-25T03:42:01.684Z [INFO] server-20e7d4be-f68a-f1d8-ee87-8d156a8c1692: starting snapshot up to: index=12 2026-05-25T03:42:01.684Z [INFO] creating new snapshot: path=/tmp/raft1939374326/snapshots/2-12-1779680521684.tmp 2026-05-25T03:42:01.684Z [INFO] server-20e7d4be-f68a-f1d8-ee87-8d156a8c1692: snapshot complete up to: index=12 2026-05-25T03:42:01.685Z [INFO] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa Address:a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa} {Suffrage:Voter ID:server-cdd852c6-6dd7-830a-ea21-d2dbe48ba05b Address:cdd852c6-6dd7-830a-ea21-d2dbe48ba05b} {Suffrage:Voter ID:server-6d3b9d86-f33f-dfd3-9f89-9117023f7499 Address:6d3b9d86-f33f-dfd3-9f89-9117023f7499}]" 2026-05-25T03:42:01.685Z [INFO] server-cdd852c6-6dd7-830a-ea21-d2dbe48ba05b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa Address:a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa} {Suffrage:Voter ID:server-cdd852c6-6dd7-830a-ea21-d2dbe48ba05b Address:cdd852c6-6dd7-830a-ea21-d2dbe48ba05b} {Suffrage:Voter ID:server-6d3b9d86-f33f-dfd3-9f89-9117023f7499 Address:6d3b9d86-f33f-dfd3-9f89-9117023f7499}]" 2026-05-25T03:42:01.685Z [INFO] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: entering follower state: follower="Node at a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa [Follower]" leader-address= leader-id= 2026-05-25T03:42:01.685Z [INFO] server-6d3b9d86-f33f-dfd3-9f89-9117023f7499: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa Address:a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa} {Suffrage:Voter ID:server-cdd852c6-6dd7-830a-ea21-d2dbe48ba05b Address:cdd852c6-6dd7-830a-ea21-d2dbe48ba05b} {Suffrage:Voter ID:server-6d3b9d86-f33f-dfd3-9f89-9117023f7499 Address:6d3b9d86-f33f-dfd3-9f89-9117023f7499}]" 2026-05-25T03:42:01.685Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:01.685Z [INFO] server-cdd852c6-6dd7-830a-ea21-d2dbe48ba05b: entering follower state: follower="Node at cdd852c6-6dd7-830a-ea21-d2dbe48ba05b [Follower]" leader-address= leader-id= 2026-05-25T03:42:01.685Z [INFO] server-6d3b9d86-f33f-dfd3-9f89-9117023f7499: entering follower state: follower="Node at 6d3b9d86-f33f-dfd3-9f89-9117023f7499 [Follower]" leader-address= leader-id= 2026-05-25T03:42:02.417Z [WARN] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:02.417Z [INFO] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: entering candidate state: node="Node at a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa [Candidate]" term=2 2026-05-25T03:42:02.418Z [INFO] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: election won: term=2 tally=2 2026-05-25T03:42:02.418Z [INFO] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: entering leader state: leader="Node at a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa [Leader]" 2026-05-25T03:42:02.418Z [INFO] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: added peer, starting replication: peer=server-cdd852c6-6dd7-830a-ea21-d2dbe48ba05b 2026-05-25T03:42:02.418Z [INFO] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: added peer, starting replication: peer=server-6d3b9d86-f33f-dfd3-9f89-9117023f7499 2026-05-25T03:42:02.418Z [INFO] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: pipelining replication: peer="{Voter server-6d3b9d86-f33f-dfd3-9f89-9117023f7499 6d3b9d86-f33f-dfd3-9f89-9117023f7499}" 2026-05-25T03:42:02.418Z [INFO] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: pipelining replication: peer="{Voter server-cdd852c6-6dd7-830a-ea21-d2dbe48ba05b cdd852c6-6dd7-830a-ea21-d2dbe48ba05b}" 2026-05-25T03:42:03.423Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:02.418169157 +0000 UTC m=+48.591144601 (1 nodes), 732.42004ms from start of poll, 732.81064ms from cluster start. Timeout at 2026-05-25 03:42:03.423175517 +0000 UTC m=+49.596150921, 1.00500632s after stability 2026-05-25T03:42:03.423Z [INFO] creating new snapshot: path=/tmp/raft1635926859/snapshots/2-15-1779680523423.tmp 2026-05-25T03:42:03.423Z [INFO] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: copied to local snapshot: bytes=71 2026-05-25T03:42:03.424Z [INFO] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: snapshot restore progress: id=2-15-1779680523423 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:03.424Z [INFO] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: restored user snapshot: index=15 2026-05-25T03:42:03.424Z [ERROR] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: failed to get log: index=3 error="log not found" 2026-05-25T03:42:03.424Z [INFO] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: aborting pipeline replication: peer="{Voter server-6d3b9d86-f33f-dfd3-9f89-9117023f7499 6d3b9d86-f33f-dfd3-9f89-9117023f7499}" 2026-05-25T03:42:03.424Z [ERROR] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: failed to get log: index=3 error="log not found" 2026-05-25T03:42:03.424Z [INFO] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: aborting pipeline replication: peer="{Voter server-cdd852c6-6dd7-830a-ea21-d2dbe48ba05b cdd852c6-6dd7-830a-ea21-d2dbe48ba05b}" 2026-05-25T03:42:03.433Z [ERROR] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: failed to get log: index=3 error="log not found" 2026-05-25T03:42:03.433Z [ERROR] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: failed to get log: index=3 error="log not found" 2026-05-25T03:42:03.434Z [INFO] creating new snapshot: path=/tmp/raft2897779979/snapshots/2-15-1779680523434.tmp 2026-05-25T03:42:03.434Z [INFO] creating new snapshot: path=/tmp/raft3297751055/snapshots/2-15-1779680523434.tmp 2026-05-25T03:42:03.434Z [INFO] server-cdd852c6-6dd7-830a-ea21-d2dbe48ba05b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:03.434Z [INFO] server-6d3b9d86-f33f-dfd3-9f89-9117023f7499: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:03.434Z [INFO] server-6d3b9d86-f33f-dfd3-9f89-9117023f7499: copied to local snapshot: bytes=71 2026-05-25T03:42:03.434Z [INFO] server-cdd852c6-6dd7-830a-ea21-d2dbe48ba05b: copied to local snapshot: bytes=71 2026-05-25T03:42:03.435Z [INFO] server-6d3b9d86-f33f-dfd3-9f89-9117023f7499: snapshot restore progress: id=2-15-1779680523434 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:03.435Z [INFO] server-6d3b9d86-f33f-dfd3-9f89-9117023f7499: Installed remote snapshot 2026-05-25T03:42:03.435Z [INFO] server-cdd852c6-6dd7-830a-ea21-d2dbe48ba05b: snapshot restore progress: id=2-15-1779680523434 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:03.435Z [INFO] server-cdd852c6-6dd7-830a-ea21-d2dbe48ba05b: Installed remote snapshot 2026-05-25T03:42:03.435Z [INFO] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: pipelining replication: peer="{Voter server-6d3b9d86-f33f-dfd3-9f89-9117023f7499 6d3b9d86-f33f-dfd3-9f89-9117023f7499}" 2026-05-25T03:42:03.435Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:03.435Z [INFO] server-a774a6c9-e6cc-ad11-e2d3-d4bd917eddaa: pipelining replication: peer="{Voter server-cdd852c6-6dd7-830a-ea21-d2dbe48ba05b cdd852c6-6dd7-830a-ea21-d2dbe48ba05b}" 2026-05-25T03:42:04.440Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:03.435416237 +0000 UTC m=+49.608391681 (2 nodes), 1.08µs from start of poll, 1.75005772s from cluster start. Timeout at 2026-05-25 03:42:04.440426197 +0000 UTC m=+50.613401681, 1.00501s after stability 2026-05-25T03:42:04.452Z [INFO] server-20e7d4be-f68a-f1d8-ee87-8d156a8c1692: aborting pipeline replication: peer="{Voter server-f06e2665-c718-1a39-7ab9-9fa0409404fd f06e2665-c718-1a39-7ab9-9fa0409404fd}" 2026-05-25T03:42:04.452Z [INFO] server-20e7d4be-f68a-f1d8-ee87-8d156a8c1692: aborting pipeline replication: peer="{Voter server-9c53ac3f-17b0-dd31-5727-9abd37b7649e 9c53ac3f-17b0-dd31-5727-9abd37b7649e}" === RUN TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true 2026-05-25T03:42:04.453Z [INFO] server-ed5bd64e-5d7f-82ae-1d24-9ebaeb0b1b23: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed5bd64e-5d7f-82ae-1d24-9ebaeb0b1b23 Address:ed5bd64e-5d7f-82ae-1d24-9ebaeb0b1b23} {Suffrage:Voter ID:server-91894fa7-aa65-6894-d13b-ce2836dec629 Address:91894fa7-aa65-6894-d13b-ce2836dec629} {Suffrage:Voter ID:server-0890b02a-e540-d54b-0fb2-8d92a2ac5608 Address:0890b02a-e540-d54b-0fb2-8d92a2ac5608}]" 2026-05-25T03:42:04.453Z [INFO] server-91894fa7-aa65-6894-d13b-ce2836dec629: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed5bd64e-5d7f-82ae-1d24-9ebaeb0b1b23 Address:ed5bd64e-5d7f-82ae-1d24-9ebaeb0b1b23} {Suffrage:Voter ID:server-91894fa7-aa65-6894-d13b-ce2836dec629 Address:91894fa7-aa65-6894-d13b-ce2836dec629} {Suffrage:Voter ID:server-0890b02a-e540-d54b-0fb2-8d92a2ac5608 Address:0890b02a-e540-d54b-0fb2-8d92a2ac5608}]" 2026-05-25T03:42:04.453Z [INFO] server-ed5bd64e-5d7f-82ae-1d24-9ebaeb0b1b23: entering follower state: follower="Node at ed5bd64e-5d7f-82ae-1d24-9ebaeb0b1b23 [Follower]" leader-address= leader-id= 2026-05-25T03:42:04.453Z [INFO] server-91894fa7-aa65-6894-d13b-ce2836dec629: entering follower state: follower="Node at 91894fa7-aa65-6894-d13b-ce2836dec629 [Follower]" leader-address= leader-id= 2026-05-25T03:42:04.453Z [INFO] server-0890b02a-e540-d54b-0fb2-8d92a2ac5608: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed5bd64e-5d7f-82ae-1d24-9ebaeb0b1b23 Address:ed5bd64e-5d7f-82ae-1d24-9ebaeb0b1b23} {Suffrage:Voter ID:server-91894fa7-aa65-6894-d13b-ce2836dec629 Address:91894fa7-aa65-6894-d13b-ce2836dec629} {Suffrage:Voter ID:server-0890b02a-e540-d54b-0fb2-8d92a2ac5608 Address:0890b02a-e540-d54b-0fb2-8d92a2ac5608}]" 2026-05-25T03:42:04.453Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:04.453Z [INFO] server-0890b02a-e540-d54b-0fb2-8d92a2ac5608: entering follower state: follower="Node at 0890b02a-e540-d54b-0fb2-8d92a2ac5608 [Follower]" leader-address= leader-id= 2026-05-25T03:42:04.996Z [WARN] server-0890b02a-e540-d54b-0fb2-8d92a2ac5608: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:04.996Z [INFO] server-0890b02a-e540-d54b-0fb2-8d92a2ac5608: entering candidate state: node="Node at 0890b02a-e540-d54b-0fb2-8d92a2ac5608 [Candidate]" term=2 2026-05-25T03:42:04.996Z [INFO] server-0890b02a-e540-d54b-0fb2-8d92a2ac5608: election won: term=2 tally=2 2026-05-25T03:42:04.996Z [INFO] server-0890b02a-e540-d54b-0fb2-8d92a2ac5608: entering leader state: leader="Node at 0890b02a-e540-d54b-0fb2-8d92a2ac5608 [Leader]" 2026-05-25T03:42:04.996Z [INFO] server-0890b02a-e540-d54b-0fb2-8d92a2ac5608: added peer, starting replication: peer=server-ed5bd64e-5d7f-82ae-1d24-9ebaeb0b1b23 2026-05-25T03:42:04.996Z [INFO] server-0890b02a-e540-d54b-0fb2-8d92a2ac5608: added peer, starting replication: peer=server-91894fa7-aa65-6894-d13b-ce2836dec629 2026-05-25T03:42:04.996Z [INFO] server-0890b02a-e540-d54b-0fb2-8d92a2ac5608: pipelining replication: peer="{Voter server-91894fa7-aa65-6894-d13b-ce2836dec629 91894fa7-aa65-6894-d13b-ce2836dec629}" 2026-05-25T03:42:04.996Z [INFO] server-0890b02a-e540-d54b-0fb2-8d92a2ac5608: pipelining replication: peer="{Voter server-ed5bd64e-5d7f-82ae-1d24-9ebaeb0b1b23 ed5bd64e-5d7f-82ae-1d24-9ebaeb0b1b23}" 2026-05-25T03:42:06.002Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:04.996792077 +0000 UTC m=+51.169767521 (1 nodes), 543.44844ms from start of poll, 543.91256ms from cluster start. Timeout at 2026-05-25 03:42:06.001795517 +0000 UTC m=+52.174771001, 1.00500348s after stability 2026-05-25T03:42:06.003Z [INFO] server-0890b02a-e540-d54b-0fb2-8d92a2ac5608: starting snapshot up to: index=12 2026-05-25T03:42:06.003Z [INFO] creating new snapshot: path=/tmp/raft65128427/snapshots/2-12-1779680526003.tmp 2026-05-25T03:42:06.003Z [INFO] server-0890b02a-e540-d54b-0fb2-8d92a2ac5608: snapshot complete up to: index=12 2026-05-25T03:42:06.004Z [INFO] server-fbf29af2-0ac8-6a18-38fd-525bb0247897: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbf29af2-0ac8-6a18-38fd-525bb0247897 Address:fbf29af2-0ac8-6a18-38fd-525bb0247897} {Suffrage:Voter ID:server-d34729d0-ebab-041e-a72f-f76ff0f6fab3 Address:d34729d0-ebab-041e-a72f-f76ff0f6fab3} {Suffrage:Voter ID:server-23c34c30-922c-bc7a-bb84-2893b81809b0 Address:23c34c30-922c-bc7a-bb84-2893b81809b0}]" 2026-05-25T03:42:06.005Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbf29af2-0ac8-6a18-38fd-525bb0247897 Address:fbf29af2-0ac8-6a18-38fd-525bb0247897} {Suffrage:Voter ID:server-d34729d0-ebab-041e-a72f-f76ff0f6fab3 Address:d34729d0-ebab-041e-a72f-f76ff0f6fab3} {Suffrage:Voter ID:server-23c34c30-922c-bc7a-bb84-2893b81809b0 Address:23c34c30-922c-bc7a-bb84-2893b81809b0}]" 2026-05-25T03:42:06.005Z [INFO] server-fbf29af2-0ac8-6a18-38fd-525bb0247897: entering follower state: follower="Node at fbf29af2-0ac8-6a18-38fd-525bb0247897 [Follower]" leader-address= leader-id= 2026-05-25T03:42:06.005Z [INFO] server-23c34c30-922c-bc7a-bb84-2893b81809b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fbf29af2-0ac8-6a18-38fd-525bb0247897 Address:fbf29af2-0ac8-6a18-38fd-525bb0247897} {Suffrage:Voter ID:server-d34729d0-ebab-041e-a72f-f76ff0f6fab3 Address:d34729d0-ebab-041e-a72f-f76ff0f6fab3} {Suffrage:Voter ID:server-23c34c30-922c-bc7a-bb84-2893b81809b0 Address:23c34c30-922c-bc7a-bb84-2893b81809b0}]" 2026-05-25T03:42:06.005Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:06.005Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: entering follower state: follower="Node at d34729d0-ebab-041e-a72f-f76ff0f6fab3 [Follower]" leader-address= leader-id= 2026-05-25T03:42:06.005Z [INFO] server-23c34c30-922c-bc7a-bb84-2893b81809b0: entering follower state: follower="Node at 23c34c30-922c-bc7a-bb84-2893b81809b0 [Follower]" leader-address= leader-id= 2026-05-25T03:42:06.597Z [WARN] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:06.597Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: entering candidate state: node="Node at d34729d0-ebab-041e-a72f-f76ff0f6fab3 [Candidate]" term=2 2026-05-25T03:42:06.597Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: election won: term=2 tally=2 2026-05-25T03:42:06.597Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: entering leader state: leader="Node at d34729d0-ebab-041e-a72f-f76ff0f6fab3 [Leader]" 2026-05-25T03:42:06.597Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: added peer, starting replication: peer=server-fbf29af2-0ac8-6a18-38fd-525bb0247897 2026-05-25T03:42:06.597Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: added peer, starting replication: peer=server-23c34c30-922c-bc7a-bb84-2893b81809b0 2026-05-25T03:42:06.597Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: pipelining replication: peer="{Voter server-fbf29af2-0ac8-6a18-38fd-525bb0247897 fbf29af2-0ac8-6a18-38fd-525bb0247897}" 2026-05-25T03:42:06.597Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: pipelining replication: peer="{Voter server-23c34c30-922c-bc7a-bb84-2893b81809b0 23c34c30-922c-bc7a-bb84-2893b81809b0}" 2026-05-25T03:42:07.602Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:06.597567117 +0000 UTC m=+52.770542561 (1 nodes), 592.29152ms from start of poll, 592.78204ms from cluster start. Timeout at 2026-05-25 03:42:07.602568357 +0000 UTC m=+53.775543801, 1.00500124s after stability 2026-05-25T03:42:07.603Z [INFO] creating new snapshot: path=/tmp/raft414591219/snapshots/2-15-1779680527603.tmp 2026-05-25T03:42:07.603Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: copied to local snapshot: bytes=71 2026-05-25T03:42:07.603Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: snapshot restore progress: id=2-15-1779680527603 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:07.603Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: removing all old logs from log store 2026-05-25T03:42:07.603Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: compacting logs: from=1 to=2 2026-05-25T03:42:07.603Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: restored user snapshot: index=15 2026-05-25T03:42:07.603Z [ERROR] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: failed to get log: index=2 error="log not found" 2026-05-25T03:42:07.603Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: aborting pipeline replication: peer="{Voter server-23c34c30-922c-bc7a-bb84-2893b81809b0 23c34c30-922c-bc7a-bb84-2893b81809b0}" 2026-05-25T03:42:07.603Z [ERROR] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: failed to get log: index=2 error="log not found" 2026-05-25T03:42:07.603Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: aborting pipeline replication: peer="{Voter server-fbf29af2-0ac8-6a18-38fd-525bb0247897 fbf29af2-0ac8-6a18-38fd-525bb0247897}" 2026-05-25T03:42:07.610Z [ERROR] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: failed to get log: index=2 error="log not found" 2026-05-25T03:42:07.610Z [INFO] creating new snapshot: path=/tmp/raft1223968946/snapshots/2-15-1779680527610.tmp 2026-05-25T03:42:07.610Z [INFO] server-23c34c30-922c-bc7a-bb84-2893b81809b0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:07.610Z [ERROR] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: failed to get log: index=2 error="log not found" 2026-05-25T03:42:07.611Z [INFO] server-23c34c30-922c-bc7a-bb84-2893b81809b0: copied to local snapshot: bytes=71 2026-05-25T03:42:07.611Z [INFO] server-23c34c30-922c-bc7a-bb84-2893b81809b0: snapshot restore progress: id=2-15-1779680527610 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:07.611Z [INFO] creating new snapshot: path=/tmp/raft4198059639/snapshots/2-15-1779680527611.tmp 2026-05-25T03:42:07.611Z [INFO] server-23c34c30-922c-bc7a-bb84-2893b81809b0: removing all old logs from log store 2026-05-25T03:42:07.611Z [INFO] server-23c34c30-922c-bc7a-bb84-2893b81809b0: compacting logs: from=1 to=2 2026-05-25T03:42:07.611Z [INFO] server-23c34c30-922c-bc7a-bb84-2893b81809b0: Installed remote snapshot 2026-05-25T03:42:07.611Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: pipelining replication: peer="{Voter server-23c34c30-922c-bc7a-bb84-2893b81809b0 23c34c30-922c-bc7a-bb84-2893b81809b0}" 2026-05-25T03:42:07.611Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:07.611Z [INFO] server-fbf29af2-0ac8-6a18-38fd-525bb0247897: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:07.611Z [INFO] server-fbf29af2-0ac8-6a18-38fd-525bb0247897: copied to local snapshot: bytes=71 2026-05-25T03:42:07.612Z [INFO] server-fbf29af2-0ac8-6a18-38fd-525bb0247897: snapshot restore progress: id=2-15-1779680527611 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:07.612Z [INFO] server-fbf29af2-0ac8-6a18-38fd-525bb0247897: removing all old logs from log store 2026-05-25T03:42:07.612Z [INFO] server-fbf29af2-0ac8-6a18-38fd-525bb0247897: compacting logs: from=1 to=2 2026-05-25T03:42:07.612Z [INFO] server-fbf29af2-0ac8-6a18-38fd-525bb0247897: Installed remote snapshot 2026-05-25T03:42:07.612Z [INFO] server-d34729d0-ebab-041e-a72f-f76ff0f6fab3: pipelining replication: peer="{Voter server-fbf29af2-0ac8-6a18-38fd-525bb0247897 fbf29af2-0ac8-6a18-38fd-525bb0247897}" 2026-05-25T03:42:08.617Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:07.611610477 +0000 UTC m=+53.784585921 (2 nodes), 1.12µs from start of poll, 1.6068254s from cluster start. Timeout at 2026-05-25 03:42:08.616637037 +0000 UTC m=+54.789612521, 1.0050266s after stability 2026-05-25T03:42:08.629Z [INFO] server-0890b02a-e540-d54b-0fb2-8d92a2ac5608: aborting pipeline replication: peer="{Voter server-91894fa7-aa65-6894-d13b-ce2836dec629 91894fa7-aa65-6894-d13b-ce2836dec629}" 2026-05-25T03:42:08.629Z [INFO] server-0890b02a-e540-d54b-0fb2-8d92a2ac5608: aborting pipeline replication: peer="{Voter server-ed5bd64e-5d7f-82ae-1d24-9ebaeb0b1b23 ed5bd64e-5d7f-82ae-1d24-9ebaeb0b1b23}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false 2026-05-25T03:42:08.630Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a89fd98a-2087-63f4-4599-a3ffe4551955 Address:a89fd98a-2087-63f4-4599-a3ffe4551955} {Suffrage:Voter ID:server-7e5cbd86-c5c8-138f-db53-ac00743f9c50 Address:7e5cbd86-c5c8-138f-db53-ac00743f9c50} {Suffrage:Voter ID:server-158e49a3-890c-6528-550e-e2ac16df6705 Address:158e49a3-890c-6528-550e-e2ac16df6705}]" 2026-05-25T03:42:08.630Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: entering follower state: follower="Node at a89fd98a-2087-63f4-4599-a3ffe4551955 [Follower]" leader-address= leader-id= 2026-05-25T03:42:08.630Z [INFO] server-7e5cbd86-c5c8-138f-db53-ac00743f9c50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a89fd98a-2087-63f4-4599-a3ffe4551955 Address:a89fd98a-2087-63f4-4599-a3ffe4551955} {Suffrage:Voter ID:server-7e5cbd86-c5c8-138f-db53-ac00743f9c50 Address:7e5cbd86-c5c8-138f-db53-ac00743f9c50} {Suffrage:Voter ID:server-158e49a3-890c-6528-550e-e2ac16df6705 Address:158e49a3-890c-6528-550e-e2ac16df6705}]" 2026-05-25T03:42:08.630Z [INFO] server-7e5cbd86-c5c8-138f-db53-ac00743f9c50: entering follower state: follower="Node at 7e5cbd86-c5c8-138f-db53-ac00743f9c50 [Follower]" leader-address= leader-id= 2026-05-25T03:42:08.631Z [INFO] server-158e49a3-890c-6528-550e-e2ac16df6705: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a89fd98a-2087-63f4-4599-a3ffe4551955 Address:a89fd98a-2087-63f4-4599-a3ffe4551955} {Suffrage:Voter ID:server-7e5cbd86-c5c8-138f-db53-ac00743f9c50 Address:7e5cbd86-c5c8-138f-db53-ac00743f9c50} {Suffrage:Voter ID:server-158e49a3-890c-6528-550e-e2ac16df6705 Address:158e49a3-890c-6528-550e-e2ac16df6705}]" 2026-05-25T03:42:08.631Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:08.631Z [INFO] server-158e49a3-890c-6528-550e-e2ac16df6705: entering follower state: follower="Node at 158e49a3-890c-6528-550e-e2ac16df6705 [Follower]" leader-address= leader-id= 2026-05-25T03:42:09.372Z [WARN] server-a89fd98a-2087-63f4-4599-a3ffe4551955: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:09.372Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: entering candidate state: node="Node at a89fd98a-2087-63f4-4599-a3ffe4551955 [Candidate]" term=2 2026-05-25T03:42:09.372Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: election won: term=2 tally=2 2026-05-25T03:42:09.372Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: entering leader state: leader="Node at a89fd98a-2087-63f4-4599-a3ffe4551955 [Leader]" 2026-05-25T03:42:09.372Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: added peer, starting replication: peer=server-7e5cbd86-c5c8-138f-db53-ac00743f9c50 2026-05-25T03:42:09.372Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: added peer, starting replication: peer=server-158e49a3-890c-6528-550e-e2ac16df6705 2026-05-25T03:42:09.372Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: pipelining replication: peer="{Voter server-158e49a3-890c-6528-550e-e2ac16df6705 158e49a3-890c-6528-550e-e2ac16df6705}" 2026-05-25T03:42:09.373Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: pipelining replication: peer="{Voter server-7e5cbd86-c5c8-138f-db53-ac00743f9c50 7e5cbd86-c5c8-138f-db53-ac00743f9c50}" 2026-05-25T03:42:10.378Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:09.372838997 +0000 UTC m=+55.545814441 (1 nodes), 741.73408ms from start of poll, 742.22356ms from cluster start. Timeout at 2026-05-25 03:42:10.377916957 +0000 UTC m=+56.550892401, 1.00507796s after stability 2026-05-25T03:42:10.378Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: starting snapshot up to: index=12 2026-05-25T03:42:10.379Z [INFO] creating new snapshot: path=/tmp/raft567447717/snapshots/2-12-1779680530378.tmp 2026-05-25T03:42:10.380Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: snapshot complete up to: index=12 2026-05-25T03:42:10.381Z [INFO] creating new snapshot: path=/tmp/raft567447717/snapshots/2-113-1779680530381.tmp 2026-05-25T03:42:10.382Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: copied to local snapshot: bytes=71 2026-05-25T03:42:10.382Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: snapshot restore progress: id=2-113-1779680530381 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:10.382Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: restored user snapshot: index=113 2026-05-25T03:42:10.382Z [ERROR] server-a89fd98a-2087-63f4-4599-a3ffe4551955: failed to get log: index=23 error="log not found" 2026-05-25T03:42:10.382Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: aborting pipeline replication: peer="{Voter server-158e49a3-890c-6528-550e-e2ac16df6705 158e49a3-890c-6528-550e-e2ac16df6705}" 2026-05-25T03:42:10.382Z [ERROR] server-a89fd98a-2087-63f4-4599-a3ffe4551955: failed to get log: index=23 error="log not found" 2026-05-25T03:42:10.382Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: aborting pipeline replication: peer="{Voter server-7e5cbd86-c5c8-138f-db53-ac00743f9c50 7e5cbd86-c5c8-138f-db53-ac00743f9c50}" 2026-05-25T03:42:10.389Z [ERROR] server-a89fd98a-2087-63f4-4599-a3ffe4551955: failed to get log: index=23 error="log not found" 2026-05-25T03:42:10.390Z [INFO] creating new snapshot: path=/tmp/raft171967127/snapshots/2-113-1779680530390.tmp 2026-05-25T03:42:10.390Z [INFO] server-7e5cbd86-c5c8-138f-db53-ac00743f9c50: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:10.390Z [ERROR] server-a89fd98a-2087-63f4-4599-a3ffe4551955: failed to get log: index=23 error="log not found" 2026-05-25T03:42:10.390Z [INFO] server-7e5cbd86-c5c8-138f-db53-ac00743f9c50: copied to local snapshot: bytes=71 2026-05-25T03:42:10.390Z [INFO] server-7e5cbd86-c5c8-138f-db53-ac00743f9c50: snapshot restore progress: id=2-113-1779680530390 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:10.390Z [INFO] server-7e5cbd86-c5c8-138f-db53-ac00743f9c50: Installed remote snapshot 2026-05-25T03:42:10.390Z [INFO] creating new snapshot: path=/tmp/raft1071234996/snapshots/2-113-1779680530390.tmp 2026-05-25T03:42:10.390Z [WARN] server-7e5cbd86-c5c8-138f-db53-ac00743f9c50: skipping application of old log: index=22 2026-05-25T03:42:10.390Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: pipelining replication: peer="{Voter server-7e5cbd86-c5c8-138f-db53-ac00743f9c50 7e5cbd86-c5c8-138f-db53-ac00743f9c50}" 2026-05-25T03:42:10.390Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:10.390Z [INFO] server-158e49a3-890c-6528-550e-e2ac16df6705: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:10.391Z [INFO] server-158e49a3-890c-6528-550e-e2ac16df6705: copied to local snapshot: bytes=71 2026-05-25T03:42:10.391Z [INFO] server-158e49a3-890c-6528-550e-e2ac16df6705: snapshot restore progress: id=2-113-1779680530390 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:10.391Z [INFO] server-158e49a3-890c-6528-550e-e2ac16df6705: Installed remote snapshot 2026-05-25T03:42:10.391Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: pipelining replication: peer="{Voter server-158e49a3-890c-6528-550e-e2ac16df6705 158e49a3-890c-6528-550e-e2ac16df6705}" 2026-05-25T03:42:11.396Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:10.390957917 +0000 UTC m=+56.563933361 (2 nodes), 1.44µs from start of poll, 1.76034248s from cluster start. Timeout at 2026-05-25 03:42:11.395961677 +0000 UTC m=+57.568937121, 1.00500376s after stability 2026-05-25T03:42:11.408Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: aborting pipeline replication: peer="{Voter server-158e49a3-890c-6528-550e-e2ac16df6705 158e49a3-890c-6528-550e-e2ac16df6705}" 2026-05-25T03:42:11.408Z [INFO] server-a89fd98a-2087-63f4-4599-a3ffe4551955: aborting pipeline replication: peer="{Voter server-7e5cbd86-c5c8-138f-db53-ac00743f9c50 7e5cbd86-c5c8-138f-db53-ac00743f9c50}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false 2026-05-25T03:42:11.409Z [INFO] server-2cc46bfb-065c-9379-ac0c-76e9c31bc58d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cc46bfb-065c-9379-ac0c-76e9c31bc58d Address:2cc46bfb-065c-9379-ac0c-76e9c31bc58d} {Suffrage:Voter ID:server-733e5650-9cb1-abca-e3c2-ff0f99d3e9ca Address:733e5650-9cb1-abca-e3c2-ff0f99d3e9ca} {Suffrage:Voter ID:server-753b432f-e0b0-1e7a-ccaf-c437915998f5 Address:753b432f-e0b0-1e7a-ccaf-c437915998f5}]" 2026-05-25T03:42:11.409Z [INFO] server-733e5650-9cb1-abca-e3c2-ff0f99d3e9ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cc46bfb-065c-9379-ac0c-76e9c31bc58d Address:2cc46bfb-065c-9379-ac0c-76e9c31bc58d} {Suffrage:Voter ID:server-733e5650-9cb1-abca-e3c2-ff0f99d3e9ca Address:733e5650-9cb1-abca-e3c2-ff0f99d3e9ca} {Suffrage:Voter ID:server-753b432f-e0b0-1e7a-ccaf-c437915998f5 Address:753b432f-e0b0-1e7a-ccaf-c437915998f5}]" 2026-05-25T03:42:11.409Z [INFO] server-2cc46bfb-065c-9379-ac0c-76e9c31bc58d: entering follower state: follower="Node at 2cc46bfb-065c-9379-ac0c-76e9c31bc58d [Follower]" leader-address= leader-id= 2026-05-25T03:42:11.409Z [INFO] server-733e5650-9cb1-abca-e3c2-ff0f99d3e9ca: entering follower state: follower="Node at 733e5650-9cb1-abca-e3c2-ff0f99d3e9ca [Follower]" leader-address= leader-id= 2026-05-25T03:42:11.409Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cc46bfb-065c-9379-ac0c-76e9c31bc58d Address:2cc46bfb-065c-9379-ac0c-76e9c31bc58d} {Suffrage:Voter ID:server-733e5650-9cb1-abca-e3c2-ff0f99d3e9ca Address:733e5650-9cb1-abca-e3c2-ff0f99d3e9ca} {Suffrage:Voter ID:server-753b432f-e0b0-1e7a-ccaf-c437915998f5 Address:753b432f-e0b0-1e7a-ccaf-c437915998f5}]" 2026-05-25T03:42:11.409Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:11.409Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: entering follower state: follower="Node at 753b432f-e0b0-1e7a-ccaf-c437915998f5 [Follower]" leader-address= leader-id= 2026-05-25T03:42:11.949Z [WARN] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:11.949Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: entering candidate state: node="Node at 753b432f-e0b0-1e7a-ccaf-c437915998f5 [Candidate]" term=2 2026-05-25T03:42:11.950Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: election won: term=2 tally=2 2026-05-25T03:42:11.950Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: entering leader state: leader="Node at 753b432f-e0b0-1e7a-ccaf-c437915998f5 [Leader]" 2026-05-25T03:42:11.950Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: added peer, starting replication: peer=server-2cc46bfb-065c-9379-ac0c-76e9c31bc58d 2026-05-25T03:42:11.950Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: added peer, starting replication: peer=server-733e5650-9cb1-abca-e3c2-ff0f99d3e9ca 2026-05-25T03:42:11.950Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: pipelining replication: peer="{Voter server-733e5650-9cb1-abca-e3c2-ff0f99d3e9ca 733e5650-9cb1-abca-e3c2-ff0f99d3e9ca}" 2026-05-25T03:42:11.950Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: pipelining replication: peer="{Voter server-2cc46bfb-065c-9379-ac0c-76e9c31bc58d 2cc46bfb-065c-9379-ac0c-76e9c31bc58d}" 2026-05-25T03:42:12.956Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:11.950222877 +0000 UTC m=+58.123198321 (1 nodes), 540.40692ms from start of poll, 540.8666ms from cluster start. Timeout at 2026-05-25 03:42:12.955227837 +0000 UTC m=+59.128203281, 1.00500496s after stability 2026-05-25T03:42:12.956Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: starting snapshot up to: index=12 2026-05-25T03:42:12.957Z [INFO] creating new snapshot: path=/tmp/raft1098346269/snapshots/2-12-1779680532956.tmp 2026-05-25T03:42:12.957Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: snapshot complete up to: index=12 2026-05-25T03:42:12.958Z [INFO] creating new snapshot: path=/tmp/raft1098346269/snapshots/2-113-1779680532958.tmp 2026-05-25T03:42:12.958Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: copied to local snapshot: bytes=71 2026-05-25T03:42:12.959Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: snapshot restore progress: id=2-113-1779680532958 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:12.959Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: removing all old logs from log store 2026-05-25T03:42:12.959Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: compacting logs: from=1 to=22 2026-05-25T03:42:12.959Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: restored user snapshot: index=113 2026-05-25T03:42:12.959Z [ERROR] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: failed to get log: index=22 error="log not found" 2026-05-25T03:42:12.959Z [ERROR] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: failed to get log: index=22 error="log not found" 2026-05-25T03:42:12.959Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: aborting pipeline replication: peer="{Voter server-733e5650-9cb1-abca-e3c2-ff0f99d3e9ca 733e5650-9cb1-abca-e3c2-ff0f99d3e9ca}" 2026-05-25T03:42:12.959Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: aborting pipeline replication: peer="{Voter server-2cc46bfb-065c-9379-ac0c-76e9c31bc58d 2cc46bfb-065c-9379-ac0c-76e9c31bc58d}" 2026-05-25T03:42:12.966Z [ERROR] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: failed to get log: index=22 error="log not found" 2026-05-25T03:42:12.967Z [INFO] creating new snapshot: path=/tmp/raft3572646766/snapshots/2-113-1779680532967.tmp 2026-05-25T03:42:12.967Z [INFO] server-2cc46bfb-065c-9379-ac0c-76e9c31bc58d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:12.967Z [INFO] server-2cc46bfb-065c-9379-ac0c-76e9c31bc58d: copied to local snapshot: bytes=71 2026-05-25T03:42:12.968Z [INFO] server-2cc46bfb-065c-9379-ac0c-76e9c31bc58d: snapshot restore progress: id=2-113-1779680532967 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:12.968Z [INFO] server-2cc46bfb-065c-9379-ac0c-76e9c31bc58d: removing all old logs from log store 2026-05-25T03:42:12.968Z [INFO] server-2cc46bfb-065c-9379-ac0c-76e9c31bc58d: compacting logs: from=1 to=22 2026-05-25T03:42:12.968Z [INFO] server-2cc46bfb-065c-9379-ac0c-76e9c31bc58d: Installed remote snapshot 2026-05-25T03:42:12.968Z [WARN] server-2cc46bfb-065c-9379-ac0c-76e9c31bc58d: skipping application of old log: index=22 2026-05-25T03:42:12.968Z [ERROR] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: failed to get log: index=22 error="log not found" 2026-05-25T03:42:12.968Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: pipelining replication: peer="{Voter server-2cc46bfb-065c-9379-ac0c-76e9c31bc58d 2cc46bfb-065c-9379-ac0c-76e9c31bc58d}" 2026-05-25T03:42:12.968Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:12.968Z [INFO] creating new snapshot: path=/tmp/raft2587198234/snapshots/2-113-1779680532968.tmp 2026-05-25T03:42:12.968Z [INFO] server-733e5650-9cb1-abca-e3c2-ff0f99d3e9ca: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:12.969Z [INFO] server-733e5650-9cb1-abca-e3c2-ff0f99d3e9ca: copied to local snapshot: bytes=71 2026-05-25T03:42:12.969Z [INFO] server-733e5650-9cb1-abca-e3c2-ff0f99d3e9ca: snapshot restore progress: id=2-113-1779680532968 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:12.969Z [INFO] server-733e5650-9cb1-abca-e3c2-ff0f99d3e9ca: removing all old logs from log store 2026-05-25T03:42:12.969Z [INFO] server-733e5650-9cb1-abca-e3c2-ff0f99d3e9ca: compacting logs: from=1 to=22 2026-05-25T03:42:12.969Z [INFO] server-733e5650-9cb1-abca-e3c2-ff0f99d3e9ca: Installed remote snapshot 2026-05-25T03:42:12.969Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: pipelining replication: peer="{Voter server-733e5650-9cb1-abca-e3c2-ff0f99d3e9ca 733e5650-9cb1-abca-e3c2-ff0f99d3e9ca}" 2026-05-25T03:42:13.973Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:12.968320317 +0000 UTC m=+59.141295761 (2 nodes), 1µs from start of poll, 1.55896404s from cluster start. Timeout at 2026-05-25 03:42:13.973323957 +0000 UTC m=+60.146299401, 1.00500364s after stability 2026-05-25T03:42:13.984Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: aborting pipeline replication: peer="{Voter server-2cc46bfb-065c-9379-ac0c-76e9c31bc58d 2cc46bfb-065c-9379-ac0c-76e9c31bc58d}" 2026-05-25T03:42:13.984Z [INFO] server-753b432f-e0b0-1e7a-ccaf-c437915998f5: aborting pipeline replication: peer="{Voter server-733e5650-9cb1-abca-e3c2-ff0f99d3e9ca 733e5650-9cb1-abca-e3c2-ff0f99d3e9ca}" === RUN TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true 2026-05-25T03:42:13.985Z [INFO] server-7655f230-1dee-87cd-65f4-3b7bab53bfb9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7655f230-1dee-87cd-65f4-3b7bab53bfb9 Address:7655f230-1dee-87cd-65f4-3b7bab53bfb9} {Suffrage:Voter ID:server-79d975e0-287d-880c-2483-81e92adca15b Address:79d975e0-287d-880c-2483-81e92adca15b} {Suffrage:Voter ID:server-0171d7ae-8c61-4387-de24-d798e29e512b Address:0171d7ae-8c61-4387-de24-d798e29e512b}]" 2026-05-25T03:42:13.985Z [INFO] server-7655f230-1dee-87cd-65f4-3b7bab53bfb9: entering follower state: follower="Node at 7655f230-1dee-87cd-65f4-3b7bab53bfb9 [Follower]" leader-address= leader-id= 2026-05-25T03:42:13.985Z [INFO] server-79d975e0-287d-880c-2483-81e92adca15b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7655f230-1dee-87cd-65f4-3b7bab53bfb9 Address:7655f230-1dee-87cd-65f4-3b7bab53bfb9} {Suffrage:Voter ID:server-79d975e0-287d-880c-2483-81e92adca15b Address:79d975e0-287d-880c-2483-81e92adca15b} {Suffrage:Voter ID:server-0171d7ae-8c61-4387-de24-d798e29e512b Address:0171d7ae-8c61-4387-de24-d798e29e512b}]" 2026-05-25T03:42:13.985Z [INFO] server-79d975e0-287d-880c-2483-81e92adca15b: entering follower state: follower="Node at 79d975e0-287d-880c-2483-81e92adca15b [Follower]" leader-address= leader-id= 2026-05-25T03:42:13.986Z [INFO] server-0171d7ae-8c61-4387-de24-d798e29e512b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7655f230-1dee-87cd-65f4-3b7bab53bfb9 Address:7655f230-1dee-87cd-65f4-3b7bab53bfb9} {Suffrage:Voter ID:server-79d975e0-287d-880c-2483-81e92adca15b Address:79d975e0-287d-880c-2483-81e92adca15b} {Suffrage:Voter ID:server-0171d7ae-8c61-4387-de24-d798e29e512b Address:0171d7ae-8c61-4387-de24-d798e29e512b}]" 2026-05-25T03:42:13.986Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:13.986Z [INFO] server-0171d7ae-8c61-4387-de24-d798e29e512b: entering follower state: follower="Node at 0171d7ae-8c61-4387-de24-d798e29e512b [Follower]" leader-address= leader-id= 2026-05-25T03:42:14.511Z [WARN] server-0171d7ae-8c61-4387-de24-d798e29e512b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:14.511Z [INFO] server-0171d7ae-8c61-4387-de24-d798e29e512b: entering candidate state: node="Node at 0171d7ae-8c61-4387-de24-d798e29e512b [Candidate]" term=2 2026-05-25T03:42:14.511Z [INFO] server-0171d7ae-8c61-4387-de24-d798e29e512b: election won: term=2 tally=2 2026-05-25T03:42:14.511Z [INFO] server-0171d7ae-8c61-4387-de24-d798e29e512b: entering leader state: leader="Node at 0171d7ae-8c61-4387-de24-d798e29e512b [Leader]" 2026-05-25T03:42:14.511Z [INFO] server-0171d7ae-8c61-4387-de24-d798e29e512b: added peer, starting replication: peer=server-7655f230-1dee-87cd-65f4-3b7bab53bfb9 2026-05-25T03:42:14.511Z [INFO] server-0171d7ae-8c61-4387-de24-d798e29e512b: added peer, starting replication: peer=server-79d975e0-287d-880c-2483-81e92adca15b 2026-05-25T03:42:14.511Z [INFO] server-0171d7ae-8c61-4387-de24-d798e29e512b: pipelining replication: peer="{Voter server-79d975e0-287d-880c-2483-81e92adca15b 79d975e0-287d-880c-2483-81e92adca15b}" 2026-05-25T03:42:14.512Z [INFO] server-0171d7ae-8c61-4387-de24-d798e29e512b: pipelining replication: peer="{Voter server-7655f230-1dee-87cd-65f4-3b7bab53bfb9 7655f230-1dee-87cd-65f4-3b7bab53bfb9}" 2026-05-25T03:42:15.517Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:14.511864837 +0000 UTC m=+60.684840281 (1 nodes), 525.71392ms from start of poll, 526.26644ms from cluster start. Timeout at 2026-05-25 03:42:15.516868717 +0000 UTC m=+61.689844161, 1.00500388s after stability 2026-05-25T03:42:15.519Z [INFO] server-0171d7ae-8c61-4387-de24-d798e29e512b: starting snapshot up to: index=12 2026-05-25T03:42:15.519Z [INFO] creating new snapshot: path=/tmp/raft425250099/snapshots/2-12-1779680535519.tmp 2026-05-25T03:42:15.520Z [INFO] server-0171d7ae-8c61-4387-de24-d798e29e512b: snapshot complete up to: index=12 2026-05-25T03:42:15.521Z [INFO] server-42b571f1-9ad6-5a3c-b317-8272b93c4391: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42b571f1-9ad6-5a3c-b317-8272b93c4391 Address:42b571f1-9ad6-5a3c-b317-8272b93c4391} {Suffrage:Voter ID:server-632aa123-123c-a5ea-94fd-5e691f336472 Address:632aa123-123c-a5ea-94fd-5e691f336472} {Suffrage:Voter ID:server-44f7785d-41c4-38ca-c1a8-99be824fe7bc Address:44f7785d-41c4-38ca-c1a8-99be824fe7bc}]" 2026-05-25T03:42:15.521Z [INFO] server-632aa123-123c-a5ea-94fd-5e691f336472: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42b571f1-9ad6-5a3c-b317-8272b93c4391 Address:42b571f1-9ad6-5a3c-b317-8272b93c4391} {Suffrage:Voter ID:server-632aa123-123c-a5ea-94fd-5e691f336472 Address:632aa123-123c-a5ea-94fd-5e691f336472} {Suffrage:Voter ID:server-44f7785d-41c4-38ca-c1a8-99be824fe7bc Address:44f7785d-41c4-38ca-c1a8-99be824fe7bc}]" 2026-05-25T03:42:15.521Z [INFO] server-42b571f1-9ad6-5a3c-b317-8272b93c4391: entering follower state: follower="Node at 42b571f1-9ad6-5a3c-b317-8272b93c4391 [Follower]" leader-address= leader-id= 2026-05-25T03:42:15.521Z [INFO] server-632aa123-123c-a5ea-94fd-5e691f336472: entering follower state: follower="Node at 632aa123-123c-a5ea-94fd-5e691f336472 [Follower]" leader-address= leader-id= 2026-05-25T03:42:15.521Z [INFO] server-44f7785d-41c4-38ca-c1a8-99be824fe7bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42b571f1-9ad6-5a3c-b317-8272b93c4391 Address:42b571f1-9ad6-5a3c-b317-8272b93c4391} {Suffrage:Voter ID:server-632aa123-123c-a5ea-94fd-5e691f336472 Address:632aa123-123c-a5ea-94fd-5e691f336472} {Suffrage:Voter ID:server-44f7785d-41c4-38ca-c1a8-99be824fe7bc Address:44f7785d-41c4-38ca-c1a8-99be824fe7bc}]" 2026-05-25T03:42:15.521Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:15.521Z [INFO] server-44f7785d-41c4-38ca-c1a8-99be824fe7bc: entering follower state: follower="Node at 44f7785d-41c4-38ca-c1a8-99be824fe7bc [Follower]" leader-address= leader-id= 2026-05-25T03:42:16.040Z [WARN] server-632aa123-123c-a5ea-94fd-5e691f336472: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:16.040Z [INFO] server-632aa123-123c-a5ea-94fd-5e691f336472: entering candidate state: node="Node at 632aa123-123c-a5ea-94fd-5e691f336472 [Candidate]" term=2 2026-05-25T03:42:16.040Z [INFO] server-632aa123-123c-a5ea-94fd-5e691f336472: election won: term=2 tally=2 2026-05-25T03:42:16.040Z [INFO] server-632aa123-123c-a5ea-94fd-5e691f336472: entering leader state: leader="Node at 632aa123-123c-a5ea-94fd-5e691f336472 [Leader]" 2026-05-25T03:42:16.040Z [INFO] server-632aa123-123c-a5ea-94fd-5e691f336472: added peer, starting replication: peer=server-42b571f1-9ad6-5a3c-b317-8272b93c4391 2026-05-25T03:42:16.040Z [INFO] server-632aa123-123c-a5ea-94fd-5e691f336472: added peer, starting replication: peer=server-44f7785d-41c4-38ca-c1a8-99be824fe7bc 2026-05-25T03:42:16.040Z [INFO] server-632aa123-123c-a5ea-94fd-5e691f336472: pipelining replication: peer="{Voter server-42b571f1-9ad6-5a3c-b317-8272b93c4391 42b571f1-9ad6-5a3c-b317-8272b93c4391}" 2026-05-25T03:42:16.040Z [INFO] server-632aa123-123c-a5ea-94fd-5e691f336472: pipelining replication: peer="{Voter server-44f7785d-41c4-38ca-c1a8-99be824fe7bc 44f7785d-41c4-38ca-c1a8-99be824fe7bc}" 2026-05-25T03:42:17.046Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:16.040667077 +0000 UTC m=+62.213642521 (1 nodes), 518.9182ms from start of poll, 519.3488ms from cluster start. Timeout at 2026-05-25 03:42:17.045674517 +0000 UTC m=+63.218649961, 1.00500744s after stability 2026-05-25T03:42:17.046Z [INFO] creating new snapshot: path=/tmp/raft1901273712/snapshots/2-113-1779680537046.tmp 2026-05-25T03:42:17.047Z [INFO] server-632aa123-123c-a5ea-94fd-5e691f336472: copied to local snapshot: bytes=71 2026-05-25T03:42:17.047Z [INFO] server-632aa123-123c-a5ea-94fd-5e691f336472: snapshot restore progress: id=2-113-1779680537046 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:17.047Z [INFO] server-632aa123-123c-a5ea-94fd-5e691f336472: restored user snapshot: index=113 2026-05-25T03:42:17.047Z [ERROR] server-632aa123-123c-a5ea-94fd-5e691f336472: failed to get log: index=3 error="log not found" 2026-05-25T03:42:17.047Z [INFO] server-632aa123-123c-a5ea-94fd-5e691f336472: aborting pipeline replication: peer="{Voter server-44f7785d-41c4-38ca-c1a8-99be824fe7bc 44f7785d-41c4-38ca-c1a8-99be824fe7bc}" 2026-05-25T03:42:17.047Z [ERROR] server-632aa123-123c-a5ea-94fd-5e691f336472: failed to get log: index=3 error="log not found" 2026-05-25T03:42:17.047Z [INFO] server-632aa123-123c-a5ea-94fd-5e691f336472: aborting pipeline replication: peer="{Voter server-42b571f1-9ad6-5a3c-b317-8272b93c4391 42b571f1-9ad6-5a3c-b317-8272b93c4391}" 2026-05-25T03:42:17.053Z [ERROR] server-632aa123-123c-a5ea-94fd-5e691f336472: failed to get log: index=3 error="log not found" 2026-05-25T03:42:17.053Z [INFO] creating new snapshot: path=/tmp/raft587899776/snapshots/2-113-1779680537053.tmp 2026-05-25T03:42:17.054Z [INFO] server-44f7785d-41c4-38ca-c1a8-99be824fe7bc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:17.054Z [INFO] server-44f7785d-41c4-38ca-c1a8-99be824fe7bc: copied to local snapshot: bytes=71 2026-05-25T03:42:17.054Z [INFO] server-44f7785d-41c4-38ca-c1a8-99be824fe7bc: snapshot restore progress: id=2-113-1779680537053 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:17.054Z [INFO] server-44f7785d-41c4-38ca-c1a8-99be824fe7bc: Installed remote snapshot 2026-05-25T03:42:17.054Z [INFO] server-632aa123-123c-a5ea-94fd-5e691f336472: pipelining replication: peer="{Voter server-44f7785d-41c4-38ca-c1a8-99be824fe7bc 44f7785d-41c4-38ca-c1a8-99be824fe7bc}" 2026-05-25T03:42:17.054Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:17.057Z [ERROR] server-632aa123-123c-a5ea-94fd-5e691f336472: failed to get log: index=3 error="log not found" 2026-05-25T03:42:17.057Z [INFO] creating new snapshot: path=/tmp/raft1322733538/snapshots/2-113-1779680537057.tmp 2026-05-25T03:42:17.057Z [INFO] server-42b571f1-9ad6-5a3c-b317-8272b93c4391: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:17.057Z [INFO] server-42b571f1-9ad6-5a3c-b317-8272b93c4391: copied to local snapshot: bytes=71 2026-05-25T03:42:17.058Z [INFO] server-42b571f1-9ad6-5a3c-b317-8272b93c4391: snapshot restore progress: id=2-113-1779680537057 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:17.058Z [INFO] server-42b571f1-9ad6-5a3c-b317-8272b93c4391: Installed remote snapshot 2026-05-25T03:42:17.058Z [INFO] server-632aa123-123c-a5ea-94fd-5e691f336472: pipelining replication: peer="{Voter server-42b571f1-9ad6-5a3c-b317-8272b93c4391 42b571f1-9ad6-5a3c-b317-8272b93c4391}" 2026-05-25T03:42:18.060Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:17.054744477 +0000 UTC m=+63.227719921 (2 nodes), 1.16µs from start of poll, 1.5334262s from cluster start. Timeout at 2026-05-25 03:42:18.059747597 +0000 UTC m=+64.232723041, 1.00500312s after stability 2026-05-25T03:42:18.071Z [INFO] server-0171d7ae-8c61-4387-de24-d798e29e512b: aborting pipeline replication: peer="{Voter server-79d975e0-287d-880c-2483-81e92adca15b 79d975e0-287d-880c-2483-81e92adca15b}" 2026-05-25T03:42:18.071Z [INFO] server-0171d7ae-8c61-4387-de24-d798e29e512b: aborting pipeline replication: peer="{Voter server-7655f230-1dee-87cd-65f4-3b7bab53bfb9 7655f230-1dee-87cd-65f4-3b7bab53bfb9}" === RUN TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true 2026-05-25T03:42:18.072Z [INFO] server-1528a53e-4822-d54f-666d-dc81c7ba7e1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1528a53e-4822-d54f-666d-dc81c7ba7e1f Address:1528a53e-4822-d54f-666d-dc81c7ba7e1f} {Suffrage:Voter ID:server-72bbabe0-4ff6-1c27-7cf8-91679adb34db Address:72bbabe0-4ff6-1c27-7cf8-91679adb34db} {Suffrage:Voter ID:server-62d09485-1f39-3af8-f577-cf3a3f0d6468 Address:62d09485-1f39-3af8-f577-cf3a3f0d6468}]" 2026-05-25T03:42:18.072Z [INFO] server-72bbabe0-4ff6-1c27-7cf8-91679adb34db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1528a53e-4822-d54f-666d-dc81c7ba7e1f Address:1528a53e-4822-d54f-666d-dc81c7ba7e1f} {Suffrage:Voter ID:server-72bbabe0-4ff6-1c27-7cf8-91679adb34db Address:72bbabe0-4ff6-1c27-7cf8-91679adb34db} {Suffrage:Voter ID:server-62d09485-1f39-3af8-f577-cf3a3f0d6468 Address:62d09485-1f39-3af8-f577-cf3a3f0d6468}]" 2026-05-25T03:42:18.072Z [INFO] server-1528a53e-4822-d54f-666d-dc81c7ba7e1f: entering follower state: follower="Node at 1528a53e-4822-d54f-666d-dc81c7ba7e1f [Follower]" leader-address= leader-id= 2026-05-25T03:42:18.072Z [INFO] server-62d09485-1f39-3af8-f577-cf3a3f0d6468: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1528a53e-4822-d54f-666d-dc81c7ba7e1f Address:1528a53e-4822-d54f-666d-dc81c7ba7e1f} {Suffrage:Voter ID:server-72bbabe0-4ff6-1c27-7cf8-91679adb34db Address:72bbabe0-4ff6-1c27-7cf8-91679adb34db} {Suffrage:Voter ID:server-62d09485-1f39-3af8-f577-cf3a3f0d6468 Address:62d09485-1f39-3af8-f577-cf3a3f0d6468}]" 2026-05-25T03:42:18.072Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:18.072Z [INFO] server-72bbabe0-4ff6-1c27-7cf8-91679adb34db: entering follower state: follower="Node at 72bbabe0-4ff6-1c27-7cf8-91679adb34db [Follower]" leader-address= leader-id= 2026-05-25T03:42:18.072Z [INFO] server-62d09485-1f39-3af8-f577-cf3a3f0d6468: entering follower state: follower="Node at 62d09485-1f39-3af8-f577-cf3a3f0d6468 [Follower]" leader-address= leader-id= 2026-05-25T03:42:18.620Z [WARN] server-1528a53e-4822-d54f-666d-dc81c7ba7e1f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:18.620Z [INFO] server-1528a53e-4822-d54f-666d-dc81c7ba7e1f: entering candidate state: node="Node at 1528a53e-4822-d54f-666d-dc81c7ba7e1f [Candidate]" term=2 2026-05-25T03:42:18.620Z [INFO] server-1528a53e-4822-d54f-666d-dc81c7ba7e1f: election won: term=2 tally=2 2026-05-25T03:42:18.620Z [INFO] server-1528a53e-4822-d54f-666d-dc81c7ba7e1f: entering leader state: leader="Node at 1528a53e-4822-d54f-666d-dc81c7ba7e1f [Leader]" 2026-05-25T03:42:18.620Z [INFO] server-1528a53e-4822-d54f-666d-dc81c7ba7e1f: added peer, starting replication: peer=server-72bbabe0-4ff6-1c27-7cf8-91679adb34db 2026-05-25T03:42:18.620Z [INFO] server-1528a53e-4822-d54f-666d-dc81c7ba7e1f: added peer, starting replication: peer=server-62d09485-1f39-3af8-f577-cf3a3f0d6468 2026-05-25T03:42:18.620Z [INFO] server-1528a53e-4822-d54f-666d-dc81c7ba7e1f: pipelining replication: peer="{Voter server-72bbabe0-4ff6-1c27-7cf8-91679adb34db 72bbabe0-4ff6-1c27-7cf8-91679adb34db}" 2026-05-25T03:42:18.620Z [INFO] server-1528a53e-4822-d54f-666d-dc81c7ba7e1f: pipelining replication: peer="{Voter server-62d09485-1f39-3af8-f577-cf3a3f0d6468 62d09485-1f39-3af8-f577-cf3a3f0d6468}" 2026-05-25T03:42:19.626Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:18.620587797 +0000 UTC m=+64.793563241 (1 nodes), 547.67952ms from start of poll, 548.13904ms from cluster start. Timeout at 2026-05-25 03:42:19.625589877 +0000 UTC m=+65.798565321, 1.00500208s after stability 2026-05-25T03:42:19.627Z [INFO] server-1528a53e-4822-d54f-666d-dc81c7ba7e1f: starting snapshot up to: index=12 2026-05-25T03:42:19.627Z [INFO] creating new snapshot: path=/tmp/raft793709348/snapshots/2-12-1779680539627.tmp 2026-05-25T03:42:19.627Z [INFO] server-1528a53e-4822-d54f-666d-dc81c7ba7e1f: snapshot complete up to: index=12 2026-05-25T03:42:19.628Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-108b621e-b41a-f704-3ca9-bd3aa0827067 Address:108b621e-b41a-f704-3ca9-bd3aa0827067} {Suffrage:Voter ID:server-fb376935-9ca8-e176-762c-e92787c2d301 Address:fb376935-9ca8-e176-762c-e92787c2d301} {Suffrage:Voter ID:server-3bf3339c-7123-9c7f-ba49-9b39b9f03069 Address:3bf3339c-7123-9c7f-ba49-9b39b9f03069}]" 2026-05-25T03:42:19.629Z [INFO] server-fb376935-9ca8-e176-762c-e92787c2d301: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-108b621e-b41a-f704-3ca9-bd3aa0827067 Address:108b621e-b41a-f704-3ca9-bd3aa0827067} {Suffrage:Voter ID:server-fb376935-9ca8-e176-762c-e92787c2d301 Address:fb376935-9ca8-e176-762c-e92787c2d301} {Suffrage:Voter ID:server-3bf3339c-7123-9c7f-ba49-9b39b9f03069 Address:3bf3339c-7123-9c7f-ba49-9b39b9f03069}]" 2026-05-25T03:42:19.629Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: entering follower state: follower="Node at 108b621e-b41a-f704-3ca9-bd3aa0827067 [Follower]" leader-address= leader-id= 2026-05-25T03:42:19.629Z [INFO] server-fb376935-9ca8-e176-762c-e92787c2d301: entering follower state: follower="Node at fb376935-9ca8-e176-762c-e92787c2d301 [Follower]" leader-address= leader-id= 2026-05-25T03:42:19.629Z [INFO] server-3bf3339c-7123-9c7f-ba49-9b39b9f03069: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-108b621e-b41a-f704-3ca9-bd3aa0827067 Address:108b621e-b41a-f704-3ca9-bd3aa0827067} {Suffrage:Voter ID:server-fb376935-9ca8-e176-762c-e92787c2d301 Address:fb376935-9ca8-e176-762c-e92787c2d301} {Suffrage:Voter ID:server-3bf3339c-7123-9c7f-ba49-9b39b9f03069 Address:3bf3339c-7123-9c7f-ba49-9b39b9f03069}]" 2026-05-25T03:42:19.629Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:19.629Z [INFO] server-3bf3339c-7123-9c7f-ba49-9b39b9f03069: entering follower state: follower="Node at 3bf3339c-7123-9c7f-ba49-9b39b9f03069 [Follower]" leader-address= leader-id= 2026-05-25T03:42:20.176Z [WARN] server-108b621e-b41a-f704-3ca9-bd3aa0827067: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:20.176Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: entering candidate state: node="Node at 108b621e-b41a-f704-3ca9-bd3aa0827067 [Candidate]" term=2 2026-05-25T03:42:20.176Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: election won: term=2 tally=2 2026-05-25T03:42:20.176Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: entering leader state: leader="Node at 108b621e-b41a-f704-3ca9-bd3aa0827067 [Leader]" 2026-05-25T03:42:20.176Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: added peer, starting replication: peer=server-fb376935-9ca8-e176-762c-e92787c2d301 2026-05-25T03:42:20.176Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: added peer, starting replication: peer=server-3bf3339c-7123-9c7f-ba49-9b39b9f03069 2026-05-25T03:42:20.176Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: pipelining replication: peer="{Voter server-fb376935-9ca8-e176-762c-e92787c2d301 fb376935-9ca8-e176-762c-e92787c2d301}" 2026-05-25T03:42:20.176Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: pipelining replication: peer="{Voter server-3bf3339c-7123-9c7f-ba49-9b39b9f03069 3bf3339c-7123-9c7f-ba49-9b39b9f03069}" 2026-05-25T03:42:21.182Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:20.176828597 +0000 UTC m=+66.349804041 (1 nodes), 547.5778ms from start of poll, 548.07424ms from cluster start. Timeout at 2026-05-25 03:42:21.181835917 +0000 UTC m=+67.354811401, 1.00500736s after stability 2026-05-25T03:42:21.182Z [INFO] creating new snapshot: path=/tmp/raft523605189/snapshots/2-113-1779680541182.tmp 2026-05-25T03:42:21.183Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: copied to local snapshot: bytes=71 2026-05-25T03:42:21.183Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: snapshot restore progress: id=2-113-1779680541182 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:21.183Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: removing all old logs from log store 2026-05-25T03:42:21.183Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: compacting logs: from=1 to=2 2026-05-25T03:42:21.183Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: restored user snapshot: index=113 2026-05-25T03:42:21.183Z [ERROR] server-108b621e-b41a-f704-3ca9-bd3aa0827067: failed to get log: index=2 error="log not found" 2026-05-25T03:42:21.183Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: aborting pipeline replication: peer="{Voter server-3bf3339c-7123-9c7f-ba49-9b39b9f03069 3bf3339c-7123-9c7f-ba49-9b39b9f03069}" 2026-05-25T03:42:21.183Z [ERROR] server-108b621e-b41a-f704-3ca9-bd3aa0827067: failed to get log: index=2 error="log not found" 2026-05-25T03:42:21.183Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: aborting pipeline replication: peer="{Voter server-fb376935-9ca8-e176-762c-e92787c2d301 fb376935-9ca8-e176-762c-e92787c2d301}" 2026-05-25T03:42:21.188Z [ERROR] server-108b621e-b41a-f704-3ca9-bd3aa0827067: failed to get log: index=2 error="log not found" 2026-05-25T03:42:21.189Z [INFO] creating new snapshot: path=/tmp/raft1850235646/snapshots/2-113-1779680541189.tmp 2026-05-25T03:42:21.189Z [INFO] server-3bf3339c-7123-9c7f-ba49-9b39b9f03069: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:21.189Z [INFO] server-3bf3339c-7123-9c7f-ba49-9b39b9f03069: copied to local snapshot: bytes=71 2026-05-25T03:42:21.190Z [INFO] server-3bf3339c-7123-9c7f-ba49-9b39b9f03069: snapshot restore progress: id=2-113-1779680541189 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:21.190Z [INFO] server-3bf3339c-7123-9c7f-ba49-9b39b9f03069: removing all old logs from log store 2026-05-25T03:42:21.190Z [INFO] server-3bf3339c-7123-9c7f-ba49-9b39b9f03069: compacting logs: from=1 to=2 2026-05-25T03:42:21.190Z [INFO] server-3bf3339c-7123-9c7f-ba49-9b39b9f03069: Installed remote snapshot 2026-05-25T03:42:21.190Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: pipelining replication: peer="{Voter server-3bf3339c-7123-9c7f-ba49-9b39b9f03069 3bf3339c-7123-9c7f-ba49-9b39b9f03069}" 2026-05-25T03:42:21.190Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:21.192Z [ERROR] server-108b621e-b41a-f704-3ca9-bd3aa0827067: failed to get log: index=2 error="log not found" 2026-05-25T03:42:21.193Z [INFO] creating new snapshot: path=/tmp/raft1560313926/snapshots/2-113-1779680541193.tmp 2026-05-25T03:42:21.193Z [INFO] server-fb376935-9ca8-e176-762c-e92787c2d301: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:21.193Z [INFO] server-fb376935-9ca8-e176-762c-e92787c2d301: copied to local snapshot: bytes=71 2026-05-25T03:42:21.193Z [INFO] server-fb376935-9ca8-e176-762c-e92787c2d301: snapshot restore progress: id=2-113-1779680541193 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:21.193Z [INFO] server-fb376935-9ca8-e176-762c-e92787c2d301: removing all old logs from log store 2026-05-25T03:42:21.193Z [INFO] server-fb376935-9ca8-e176-762c-e92787c2d301: compacting logs: from=1 to=2 2026-05-25T03:42:21.193Z [INFO] server-fb376935-9ca8-e176-762c-e92787c2d301: Installed remote snapshot 2026-05-25T03:42:21.194Z [INFO] server-108b621e-b41a-f704-3ca9-bd3aa0827067: pipelining replication: peer="{Voter server-fb376935-9ca8-e176-762c-e92787c2d301 fb376935-9ca8-e176-762c-e92787c2d301}" 2026-05-25T03:42:22.195Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:21.190320357 +0000 UTC m=+67.363295801 (2 nodes), 1.2µs from start of poll, 1.561566s from cluster start. Timeout at 2026-05-25 03:42:22.195338797 +0000 UTC m=+68.368314241, 1.00501844s after stability 2026-05-25T03:42:22.208Z [INFO] server-1528a53e-4822-d54f-666d-dc81c7ba7e1f: aborting pipeline replication: peer="{Voter server-62d09485-1f39-3af8-f577-cf3a3f0d6468 62d09485-1f39-3af8-f577-cf3a3f0d6468}" 2026-05-25T03:42:22.208Z [INFO] server-1528a53e-4822-d54f-666d-dc81c7ba7e1f: aborting pipeline replication: peer="{Voter server-72bbabe0-4ff6-1c27-7cf8-91679adb34db 72bbabe0-4ff6-1c27-7cf8-91679adb34db}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false 2026-05-25T03:42:22.209Z [INFO] server-5cd42853-1d53-5d80-2dc2-39f4cc4dac60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5cd42853-1d53-5d80-2dc2-39f4cc4dac60 Address:5cd42853-1d53-5d80-2dc2-39f4cc4dac60} {Suffrage:Voter ID:server-ee15e779-3b37-9f79-b78c-dfa24aeac99b Address:ee15e779-3b37-9f79-b78c-dfa24aeac99b} {Suffrage:Voter ID:server-1b275961-137f-d87b-dd69-9cf969ce5d1a Address:1b275961-137f-d87b-dd69-9cf969ce5d1a}]" 2026-05-25T03:42:22.209Z [INFO] server-ee15e779-3b37-9f79-b78c-dfa24aeac99b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5cd42853-1d53-5d80-2dc2-39f4cc4dac60 Address:5cd42853-1d53-5d80-2dc2-39f4cc4dac60} {Suffrage:Voter ID:server-ee15e779-3b37-9f79-b78c-dfa24aeac99b Address:ee15e779-3b37-9f79-b78c-dfa24aeac99b} {Suffrage:Voter ID:server-1b275961-137f-d87b-dd69-9cf969ce5d1a Address:1b275961-137f-d87b-dd69-9cf969ce5d1a}]" 2026-05-25T03:42:22.209Z [INFO] server-5cd42853-1d53-5d80-2dc2-39f4cc4dac60: entering follower state: follower="Node at 5cd42853-1d53-5d80-2dc2-39f4cc4dac60 [Follower]" leader-address= leader-id= 2026-05-25T03:42:22.209Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5cd42853-1d53-5d80-2dc2-39f4cc4dac60 Address:5cd42853-1d53-5d80-2dc2-39f4cc4dac60} {Suffrage:Voter ID:server-ee15e779-3b37-9f79-b78c-dfa24aeac99b Address:ee15e779-3b37-9f79-b78c-dfa24aeac99b} {Suffrage:Voter ID:server-1b275961-137f-d87b-dd69-9cf969ce5d1a Address:1b275961-137f-d87b-dd69-9cf969ce5d1a}]" 2026-05-25T03:42:22.209Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:22.209Z [INFO] server-ee15e779-3b37-9f79-b78c-dfa24aeac99b: entering follower state: follower="Node at ee15e779-3b37-9f79-b78c-dfa24aeac99b [Follower]" leader-address= leader-id= 2026-05-25T03:42:22.209Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: entering follower state: follower="Node at 1b275961-137f-d87b-dd69-9cf969ce5d1a [Follower]" leader-address= leader-id= 2026-05-25T03:42:22.771Z [WARN] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:22.771Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: entering candidate state: node="Node at 1b275961-137f-d87b-dd69-9cf969ce5d1a [Candidate]" term=2 2026-05-25T03:42:22.771Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: election won: term=2 tally=2 2026-05-25T03:42:22.771Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: entering leader state: leader="Node at 1b275961-137f-d87b-dd69-9cf969ce5d1a [Leader]" 2026-05-25T03:42:22.771Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: added peer, starting replication: peer=server-5cd42853-1d53-5d80-2dc2-39f4cc4dac60 2026-05-25T03:42:22.771Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: added peer, starting replication: peer=server-ee15e779-3b37-9f79-b78c-dfa24aeac99b 2026-05-25T03:42:22.771Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: pipelining replication: peer="{Voter server-5cd42853-1d53-5d80-2dc2-39f4cc4dac60 5cd42853-1d53-5d80-2dc2-39f4cc4dac60}" 2026-05-25T03:42:22.772Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: pipelining replication: peer="{Voter server-ee15e779-3b37-9f79-b78c-dfa24aeac99b ee15e779-3b37-9f79-b78c-dfa24aeac99b}" 2026-05-25T03:42:23.777Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:22.771871997 +0000 UTC m=+68.944847441 (1 nodes), 562.13772ms from start of poll, 562.56432ms from cluster start. Timeout at 2026-05-25 03:42:23.776873717 +0000 UTC m=+69.949849161, 1.00500172s after stability 2026-05-25T03:42:23.778Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: starting snapshot up to: index=12 2026-05-25T03:42:23.778Z [INFO] creating new snapshot: path=/tmp/raft3175073544/snapshots/2-12-1779680543778.tmp 2026-05-25T03:42:23.778Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: snapshot complete up to: index=12 2026-05-25T03:42:23.779Z [INFO] creating new snapshot: path=/tmp/raft3175073544/snapshots/2-1013-1779680543779.tmp 2026-05-25T03:42:23.780Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: copied to local snapshot: bytes=71 2026-05-25T03:42:23.780Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: snapshot restore progress: id=2-1013-1779680543779 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:23.780Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: restored user snapshot: index=1013 2026-05-25T03:42:23.780Z [ERROR] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: failed to get log: index=23 error="log not found" 2026-05-25T03:42:23.780Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: aborting pipeline replication: peer="{Voter server-5cd42853-1d53-5d80-2dc2-39f4cc4dac60 5cd42853-1d53-5d80-2dc2-39f4cc4dac60}" 2026-05-25T03:42:23.780Z [ERROR] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: failed to get log: index=23 error="log not found" 2026-05-25T03:42:23.780Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: aborting pipeline replication: peer="{Voter server-ee15e779-3b37-9f79-b78c-dfa24aeac99b ee15e779-3b37-9f79-b78c-dfa24aeac99b}" 2026-05-25T03:42:23.787Z [ERROR] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: failed to get log: index=23 error="log not found" 2026-05-25T03:42:23.788Z [INFO] creating new snapshot: path=/tmp/raft1915791842/snapshots/2-1013-1779680543788.tmp 2026-05-25T03:42:23.788Z [INFO] server-5cd42853-1d53-5d80-2dc2-39f4cc4dac60: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:23.788Z [INFO] server-5cd42853-1d53-5d80-2dc2-39f4cc4dac60: copied to local snapshot: bytes=71 2026-05-25T03:42:23.789Z [INFO] server-5cd42853-1d53-5d80-2dc2-39f4cc4dac60: snapshot restore progress: id=2-1013-1779680543788 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:23.789Z [INFO] server-5cd42853-1d53-5d80-2dc2-39f4cc4dac60: Installed remote snapshot 2026-05-25T03:42:23.789Z [WARN] server-5cd42853-1d53-5d80-2dc2-39f4cc4dac60: skipping application of old log: index=22 2026-05-25T03:42:23.789Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: pipelining replication: peer="{Voter server-5cd42853-1d53-5d80-2dc2-39f4cc4dac60 5cd42853-1d53-5d80-2dc2-39f4cc4dac60}" 2026-05-25T03:42:23.789Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:23.790Z [ERROR] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: failed to get log: index=23 error="log not found" 2026-05-25T03:42:23.790Z [INFO] creating new snapshot: path=/tmp/raft2967568110/snapshots/2-1013-1779680543790.tmp 2026-05-25T03:42:23.791Z [INFO] server-ee15e779-3b37-9f79-b78c-dfa24aeac99b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:23.791Z [INFO] server-ee15e779-3b37-9f79-b78c-dfa24aeac99b: copied to local snapshot: bytes=71 2026-05-25T03:42:23.791Z [INFO] server-ee15e779-3b37-9f79-b78c-dfa24aeac99b: snapshot restore progress: id=2-1013-1779680543790 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:23.791Z [INFO] server-ee15e779-3b37-9f79-b78c-dfa24aeac99b: Installed remote snapshot 2026-05-25T03:42:23.791Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: pipelining replication: peer="{Voter server-ee15e779-3b37-9f79-b78c-dfa24aeac99b ee15e779-3b37-9f79-b78c-dfa24aeac99b}" 2026-05-25T03:42:24.795Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:23.789541077 +0000 UTC m=+69.962516521 (2 nodes), 920ns from start of poll, 1.5802334s from cluster start. Timeout at 2026-05-25 03:42:24.794557117 +0000 UTC m=+70.967532561, 1.00501604s after stability 2026-05-25T03:42:24.810Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: aborting pipeline replication: peer="{Voter server-5cd42853-1d53-5d80-2dc2-39f4cc4dac60 5cd42853-1d53-5d80-2dc2-39f4cc4dac60}" 2026-05-25T03:42:24.810Z [INFO] server-1b275961-137f-d87b-dd69-9cf969ce5d1a: aborting pipeline replication: peer="{Voter server-ee15e779-3b37-9f79-b78c-dfa24aeac99b ee15e779-3b37-9f79-b78c-dfa24aeac99b}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false 2026-05-25T03:42:24.812Z [INFO] server-959c4936-ad6f-89f4-47e5-abebb3ec938f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-959c4936-ad6f-89f4-47e5-abebb3ec938f Address:959c4936-ad6f-89f4-47e5-abebb3ec938f} {Suffrage:Voter ID:server-b1509a41-49b5-9c78-0688-481967a17a90 Address:b1509a41-49b5-9c78-0688-481967a17a90} {Suffrage:Voter ID:server-6b932663-8898-6e30-aa78-767df89a9fbe Address:6b932663-8898-6e30-aa78-767df89a9fbe}]" 2026-05-25T03:42:24.812Z [INFO] server-959c4936-ad6f-89f4-47e5-abebb3ec938f: entering follower state: follower="Node at 959c4936-ad6f-89f4-47e5-abebb3ec938f [Follower]" leader-address= leader-id= 2026-05-25T03:42:24.812Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-959c4936-ad6f-89f4-47e5-abebb3ec938f Address:959c4936-ad6f-89f4-47e5-abebb3ec938f} {Suffrage:Voter ID:server-b1509a41-49b5-9c78-0688-481967a17a90 Address:b1509a41-49b5-9c78-0688-481967a17a90} {Suffrage:Voter ID:server-6b932663-8898-6e30-aa78-767df89a9fbe Address:6b932663-8898-6e30-aa78-767df89a9fbe}]" 2026-05-25T03:42:24.812Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: entering follower state: follower="Node at b1509a41-49b5-9c78-0688-481967a17a90 [Follower]" leader-address= leader-id= 2026-05-25T03:42:24.812Z [INFO] server-6b932663-8898-6e30-aa78-767df89a9fbe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-959c4936-ad6f-89f4-47e5-abebb3ec938f Address:959c4936-ad6f-89f4-47e5-abebb3ec938f} {Suffrage:Voter ID:server-b1509a41-49b5-9c78-0688-481967a17a90 Address:b1509a41-49b5-9c78-0688-481967a17a90} {Suffrage:Voter ID:server-6b932663-8898-6e30-aa78-767df89a9fbe Address:6b932663-8898-6e30-aa78-767df89a9fbe}]" 2026-05-25T03:42:24.812Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:24.812Z [INFO] server-6b932663-8898-6e30-aa78-767df89a9fbe: entering follower state: follower="Node at 6b932663-8898-6e30-aa78-767df89a9fbe [Follower]" leader-address= leader-id= 2026-05-25T03:42:25.422Z [WARN] server-b1509a41-49b5-9c78-0688-481967a17a90: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:25.422Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: entering candidate state: node="Node at b1509a41-49b5-9c78-0688-481967a17a90 [Candidate]" term=2 2026-05-25T03:42:25.422Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: election won: term=2 tally=2 2026-05-25T03:42:25.422Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: entering leader state: leader="Node at b1509a41-49b5-9c78-0688-481967a17a90 [Leader]" 2026-05-25T03:42:25.422Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: added peer, starting replication: peer=server-959c4936-ad6f-89f4-47e5-abebb3ec938f 2026-05-25T03:42:25.422Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: added peer, starting replication: peer=server-6b932663-8898-6e30-aa78-767df89a9fbe 2026-05-25T03:42:25.423Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: pipelining replication: peer="{Voter server-6b932663-8898-6e30-aa78-767df89a9fbe 6b932663-8898-6e30-aa78-767df89a9fbe}" 2026-05-25T03:42:25.423Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: pipelining replication: peer="{Voter server-959c4936-ad6f-89f4-47e5-abebb3ec938f 959c4936-ad6f-89f4-47e5-abebb3ec938f}" 2026-05-25T03:42:26.428Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:25.422929477 +0000 UTC m=+71.595904921 (1 nodes), 610.50628ms from start of poll, 611.06248ms from cluster start. Timeout at 2026-05-25 03:42:26.427936877 +0000 UTC m=+72.600912321, 1.0050074s after stability 2026-05-25T03:42:26.429Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: starting snapshot up to: index=12 2026-05-25T03:42:26.429Z [INFO] creating new snapshot: path=/tmp/raft353486208/snapshots/2-12-1779680546429.tmp 2026-05-25T03:42:26.429Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: snapshot complete up to: index=12 2026-05-25T03:42:26.430Z [INFO] creating new snapshot: path=/tmp/raft353486208/snapshots/2-1013-1779680546430.tmp 2026-05-25T03:42:26.430Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: copied to local snapshot: bytes=71 2026-05-25T03:42:26.431Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: snapshot restore progress: id=2-1013-1779680546430 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:26.431Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: removing all old logs from log store 2026-05-25T03:42:26.431Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: compacting logs: from=1 to=22 2026-05-25T03:42:26.431Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: restored user snapshot: index=1013 2026-05-25T03:42:26.431Z [ERROR] server-b1509a41-49b5-9c78-0688-481967a17a90: failed to get log: index=22 error="log not found" 2026-05-25T03:42:26.431Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: aborting pipeline replication: peer="{Voter server-6b932663-8898-6e30-aa78-767df89a9fbe 6b932663-8898-6e30-aa78-767df89a9fbe}" 2026-05-25T03:42:26.431Z [ERROR] server-b1509a41-49b5-9c78-0688-481967a17a90: failed to get log: index=22 error="log not found" 2026-05-25T03:42:26.431Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: aborting pipeline replication: peer="{Voter server-959c4936-ad6f-89f4-47e5-abebb3ec938f 959c4936-ad6f-89f4-47e5-abebb3ec938f}" 2026-05-25T03:42:26.438Z [ERROR] server-b1509a41-49b5-9c78-0688-481967a17a90: failed to get log: index=22 error="log not found" 2026-05-25T03:42:26.439Z [INFO] creating new snapshot: path=/tmp/raft2176431687/snapshots/2-1013-1779680546439.tmp 2026-05-25T03:42:26.439Z [INFO] server-6b932663-8898-6e30-aa78-767df89a9fbe: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:26.439Z [INFO] server-6b932663-8898-6e30-aa78-767df89a9fbe: copied to local snapshot: bytes=71 2026-05-25T03:42:26.440Z [INFO] server-6b932663-8898-6e30-aa78-767df89a9fbe: snapshot restore progress: id=2-1013-1779680546439 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:26.440Z [INFO] server-6b932663-8898-6e30-aa78-767df89a9fbe: removing all old logs from log store 2026-05-25T03:42:26.440Z [INFO] server-6b932663-8898-6e30-aa78-767df89a9fbe: compacting logs: from=1 to=22 2026-05-25T03:42:26.440Z [INFO] server-6b932663-8898-6e30-aa78-767df89a9fbe: Installed remote snapshot 2026-05-25T03:42:26.440Z [WARN] server-6b932663-8898-6e30-aa78-767df89a9fbe: skipping application of old log: index=22 2026-05-25T03:42:26.440Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: pipelining replication: peer="{Voter server-6b932663-8898-6e30-aa78-767df89a9fbe 6b932663-8898-6e30-aa78-767df89a9fbe}" 2026-05-25T03:42:26.440Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:26.442Z [ERROR] server-b1509a41-49b5-9c78-0688-481967a17a90: failed to get log: index=22 error="log not found" 2026-05-25T03:42:26.442Z [INFO] creating new snapshot: path=/tmp/raft290913231/snapshots/2-1013-1779680546442.tmp 2026-05-25T03:42:26.443Z [INFO] server-959c4936-ad6f-89f4-47e5-abebb3ec938f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:26.443Z [INFO] server-959c4936-ad6f-89f4-47e5-abebb3ec938f: copied to local snapshot: bytes=71 2026-05-25T03:42:26.443Z [INFO] server-959c4936-ad6f-89f4-47e5-abebb3ec938f: snapshot restore progress: id=2-1013-1779680546442 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:26.443Z [INFO] server-959c4936-ad6f-89f4-47e5-abebb3ec938f: removing all old logs from log store 2026-05-25T03:42:26.443Z [INFO] server-959c4936-ad6f-89f4-47e5-abebb3ec938f: compacting logs: from=1 to=22 2026-05-25T03:42:26.443Z [INFO] server-959c4936-ad6f-89f4-47e5-abebb3ec938f: Installed remote snapshot 2026-05-25T03:42:26.444Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: pipelining replication: peer="{Voter server-959c4936-ad6f-89f4-47e5-abebb3ec938f 959c4936-ad6f-89f4-47e5-abebb3ec938f}" 2026-05-25T03:42:27.445Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:26.440366117 +0000 UTC m=+72.613341561 (2 nodes), 1.36µs from start of poll, 1.62849912s from cluster start. Timeout at 2026-05-25 03:42:27.445392957 +0000 UTC m=+73.618368401, 1.00502684s after stability 2026-05-25T03:42:27.456Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: aborting pipeline replication: peer="{Voter server-6b932663-8898-6e30-aa78-767df89a9fbe 6b932663-8898-6e30-aa78-767df89a9fbe}" 2026-05-25T03:42:27.456Z [INFO] server-b1509a41-49b5-9c78-0688-481967a17a90: aborting pipeline replication: peer="{Voter server-959c4936-ad6f-89f4-47e5-abebb3ec938f 959c4936-ad6f-89f4-47e5-abebb3ec938f}" === RUN TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true 2026-05-25T03:42:27.457Z [INFO] server-32bc8e32-147a-f266-46ff-e80faac230b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32bc8e32-147a-f266-46ff-e80faac230b5 Address:32bc8e32-147a-f266-46ff-e80faac230b5} {Suffrage:Voter ID:server-fb7e4293-c0e4-e906-0cd5-0f249fe670d2 Address:fb7e4293-c0e4-e906-0cd5-0f249fe670d2} {Suffrage:Voter ID:server-844c700a-29bd-6522-b610-cd84597e0b8b Address:844c700a-29bd-6522-b610-cd84597e0b8b}]" 2026-05-25T03:42:27.457Z [INFO] server-fb7e4293-c0e4-e906-0cd5-0f249fe670d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32bc8e32-147a-f266-46ff-e80faac230b5 Address:32bc8e32-147a-f266-46ff-e80faac230b5} {Suffrage:Voter ID:server-fb7e4293-c0e4-e906-0cd5-0f249fe670d2 Address:fb7e4293-c0e4-e906-0cd5-0f249fe670d2} {Suffrage:Voter ID:server-844c700a-29bd-6522-b610-cd84597e0b8b Address:844c700a-29bd-6522-b610-cd84597e0b8b}]" 2026-05-25T03:42:27.457Z [INFO] server-32bc8e32-147a-f266-46ff-e80faac230b5: entering follower state: follower="Node at 32bc8e32-147a-f266-46ff-e80faac230b5 [Follower]" leader-address= leader-id= 2026-05-25T03:42:27.457Z [INFO] server-fb7e4293-c0e4-e906-0cd5-0f249fe670d2: entering follower state: follower="Node at fb7e4293-c0e4-e906-0cd5-0f249fe670d2 [Follower]" leader-address= leader-id= 2026-05-25T03:42:27.457Z [INFO] server-844c700a-29bd-6522-b610-cd84597e0b8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32bc8e32-147a-f266-46ff-e80faac230b5 Address:32bc8e32-147a-f266-46ff-e80faac230b5} {Suffrage:Voter ID:server-fb7e4293-c0e4-e906-0cd5-0f249fe670d2 Address:fb7e4293-c0e4-e906-0cd5-0f249fe670d2} {Suffrage:Voter ID:server-844c700a-29bd-6522-b610-cd84597e0b8b Address:844c700a-29bd-6522-b610-cd84597e0b8b}]" 2026-05-25T03:42:27.457Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:27.457Z [INFO] server-844c700a-29bd-6522-b610-cd84597e0b8b: entering follower state: follower="Node at 844c700a-29bd-6522-b610-cd84597e0b8b [Follower]" leader-address= leader-id= 2026-05-25T03:42:28.020Z [WARN] server-32bc8e32-147a-f266-46ff-e80faac230b5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:28.020Z [INFO] server-32bc8e32-147a-f266-46ff-e80faac230b5: entering candidate state: node="Node at 32bc8e32-147a-f266-46ff-e80faac230b5 [Candidate]" term=2 2026-05-25T03:42:28.020Z [INFO] server-32bc8e32-147a-f266-46ff-e80faac230b5: election won: term=2 tally=2 2026-05-25T03:42:28.020Z [INFO] server-32bc8e32-147a-f266-46ff-e80faac230b5: entering leader state: leader="Node at 32bc8e32-147a-f266-46ff-e80faac230b5 [Leader]" 2026-05-25T03:42:28.021Z [INFO] server-32bc8e32-147a-f266-46ff-e80faac230b5: added peer, starting replication: peer=server-fb7e4293-c0e4-e906-0cd5-0f249fe670d2 2026-05-25T03:42:28.021Z [INFO] server-32bc8e32-147a-f266-46ff-e80faac230b5: added peer, starting replication: peer=server-844c700a-29bd-6522-b610-cd84597e0b8b 2026-05-25T03:42:28.021Z [INFO] server-32bc8e32-147a-f266-46ff-e80faac230b5: pipelining replication: peer="{Voter server-844c700a-29bd-6522-b610-cd84597e0b8b 844c700a-29bd-6522-b610-cd84597e0b8b}" 2026-05-25T03:42:28.021Z [INFO] server-32bc8e32-147a-f266-46ff-e80faac230b5: pipelining replication: peer="{Voter server-fb7e4293-c0e4-e906-0cd5-0f249fe670d2 fb7e4293-c0e4-e906-0cd5-0f249fe670d2}" 2026-05-25T03:42:29.026Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:28.021004317 +0000 UTC m=+74.193979761 (1 nodes), 563.37504ms from start of poll, 563.81952ms from cluster start. Timeout at 2026-05-25 03:42:29.026011117 +0000 UTC m=+75.198986561, 1.0050068s after stability 2026-05-25T03:42:29.027Z [INFO] server-32bc8e32-147a-f266-46ff-e80faac230b5: starting snapshot up to: index=12 2026-05-25T03:42:29.027Z [INFO] creating new snapshot: path=/tmp/raft1652290750/snapshots/2-12-1779680549027.tmp 2026-05-25T03:42:29.027Z [INFO] server-32bc8e32-147a-f266-46ff-e80faac230b5: snapshot complete up to: index=12 2026-05-25T03:42:29.028Z [INFO] server-f70eff2d-b5c7-d69e-e346-ea58663e2657: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f70eff2d-b5c7-d69e-e346-ea58663e2657 Address:f70eff2d-b5c7-d69e-e346-ea58663e2657} {Suffrage:Voter ID:server-9bcaa604-2924-721b-6aa7-ff669c2203fe Address:9bcaa604-2924-721b-6aa7-ff669c2203fe} {Suffrage:Voter ID:server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43 Address:2b0df0ee-1f7c-a1bf-efed-03ca234e2a43}]" 2026-05-25T03:42:29.028Z [INFO] server-f70eff2d-b5c7-d69e-e346-ea58663e2657: entering follower state: follower="Node at f70eff2d-b5c7-d69e-e346-ea58663e2657 [Follower]" leader-address= leader-id= 2026-05-25T03:42:29.029Z [INFO] server-9bcaa604-2924-721b-6aa7-ff669c2203fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f70eff2d-b5c7-d69e-e346-ea58663e2657 Address:f70eff2d-b5c7-d69e-e346-ea58663e2657} {Suffrage:Voter ID:server-9bcaa604-2924-721b-6aa7-ff669c2203fe Address:9bcaa604-2924-721b-6aa7-ff669c2203fe} {Suffrage:Voter ID:server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43 Address:2b0df0ee-1f7c-a1bf-efed-03ca234e2a43}]" 2026-05-25T03:42:29.029Z [INFO] server-9bcaa604-2924-721b-6aa7-ff669c2203fe: entering follower state: follower="Node at 9bcaa604-2924-721b-6aa7-ff669c2203fe [Follower]" leader-address= leader-id= 2026-05-25T03:42:29.029Z [INFO] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f70eff2d-b5c7-d69e-e346-ea58663e2657 Address:f70eff2d-b5c7-d69e-e346-ea58663e2657} {Suffrage:Voter ID:server-9bcaa604-2924-721b-6aa7-ff669c2203fe Address:9bcaa604-2924-721b-6aa7-ff669c2203fe} {Suffrage:Voter ID:server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43 Address:2b0df0ee-1f7c-a1bf-efed-03ca234e2a43}]" 2026-05-25T03:42:29.029Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:29.029Z [INFO] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: entering follower state: follower="Node at 2b0df0ee-1f7c-a1bf-efed-03ca234e2a43 [Follower]" leader-address= leader-id= 2026-05-25T03:42:29.717Z [WARN] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:29.717Z [INFO] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: entering candidate state: node="Node at 2b0df0ee-1f7c-a1bf-efed-03ca234e2a43 [Candidate]" term=2 2026-05-25T03:42:29.718Z [INFO] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: election won: term=2 tally=2 2026-05-25T03:42:29.718Z [INFO] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: entering leader state: leader="Node at 2b0df0ee-1f7c-a1bf-efed-03ca234e2a43 [Leader]" 2026-05-25T03:42:29.718Z [INFO] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: added peer, starting replication: peer=server-f70eff2d-b5c7-d69e-e346-ea58663e2657 2026-05-25T03:42:29.718Z [INFO] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: added peer, starting replication: peer=server-9bcaa604-2924-721b-6aa7-ff669c2203fe 2026-05-25T03:42:29.718Z [INFO] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: pipelining replication: peer="{Voter server-f70eff2d-b5c7-d69e-e346-ea58663e2657 f70eff2d-b5c7-d69e-e346-ea58663e2657}" 2026-05-25T03:42:29.718Z [INFO] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: pipelining replication: peer="{Voter server-9bcaa604-2924-721b-6aa7-ff669c2203fe 9bcaa604-2924-721b-6aa7-ff669c2203fe}" 2026-05-25T03:42:30.723Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:29.718156557 +0000 UTC m=+75.891132001 (1 nodes), 688.87088ms from start of poll, 689.43044ms from cluster start. Timeout at 2026-05-25 03:42:30.723159837 +0000 UTC m=+76.896135281, 1.00500328s after stability 2026-05-25T03:42:30.724Z [INFO] creating new snapshot: path=/tmp/raft2429899610/snapshots/2-1013-1779680550724.tmp 2026-05-25T03:42:30.724Z [INFO] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: copied to local snapshot: bytes=71 2026-05-25T03:42:30.724Z [INFO] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: snapshot restore progress: id=2-1013-1779680550724 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:30.724Z [INFO] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: restored user snapshot: index=1013 2026-05-25T03:42:30.724Z [ERROR] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: failed to get log: index=3 error="log not found" 2026-05-25T03:42:30.724Z [INFO] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: aborting pipeline replication: peer="{Voter server-9bcaa604-2924-721b-6aa7-ff669c2203fe 9bcaa604-2924-721b-6aa7-ff669c2203fe}" 2026-05-25T03:42:30.724Z [ERROR] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: failed to get log: index=3 error="log not found" 2026-05-25T03:42:30.724Z [INFO] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: aborting pipeline replication: peer="{Voter server-f70eff2d-b5c7-d69e-e346-ea58663e2657 f70eff2d-b5c7-d69e-e346-ea58663e2657}" 2026-05-25T03:42:30.730Z [ERROR] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: failed to get log: index=3 error="log not found" 2026-05-25T03:42:30.730Z [ERROR] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: failed to get log: index=3 error="log not found" 2026-05-25T03:42:30.730Z [INFO] creating new snapshot: path=/tmp/raft2963166883/snapshots/2-1013-1779680550730.tmp 2026-05-25T03:42:30.731Z [INFO] creating new snapshot: path=/tmp/raft1952339895/snapshots/2-1013-1779680550731.tmp 2026-05-25T03:42:30.731Z [INFO] server-9bcaa604-2924-721b-6aa7-ff669c2203fe: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:30.731Z [INFO] server-9bcaa604-2924-721b-6aa7-ff669c2203fe: copied to local snapshot: bytes=71 2026-05-25T03:42:30.732Z [INFO] server-f70eff2d-b5c7-d69e-e346-ea58663e2657: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:30.732Z [INFO] server-9bcaa604-2924-721b-6aa7-ff669c2203fe: snapshot restore progress: id=2-1013-1779680550731 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:30.732Z [INFO] server-9bcaa604-2924-721b-6aa7-ff669c2203fe: Installed remote snapshot 2026-05-25T03:42:30.732Z [WARN] server-9bcaa604-2924-721b-6aa7-ff669c2203fe: skipping application of old log: index=1013 2026-05-25T03:42:30.733Z [INFO] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: pipelining replication: peer="{Voter server-9bcaa604-2924-721b-6aa7-ff669c2203fe 9bcaa604-2924-721b-6aa7-ff669c2203fe}" 2026-05-25T03:42:30.733Z [INFO] server-f70eff2d-b5c7-d69e-e346-ea58663e2657: copied to local snapshot: bytes=71 2026-05-25T03:42:30.733Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:30.733Z [INFO] server-f70eff2d-b5c7-d69e-e346-ea58663e2657: snapshot restore progress: id=2-1013-1779680550730 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:30.733Z [INFO] server-f70eff2d-b5c7-d69e-e346-ea58663e2657: Installed remote snapshot 2026-05-25T03:42:30.733Z [INFO] server-2b0df0ee-1f7c-a1bf-efed-03ca234e2a43: pipelining replication: peer="{Voter server-f70eff2d-b5c7-d69e-e346-ea58663e2657 f70eff2d-b5c7-d69e-e346-ea58663e2657}" 2026-05-25T03:42:31.739Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:30.733453637 +0000 UTC m=+76.906429081 (2 nodes), 1.72µs from start of poll, 1.70472752s from cluster start. Timeout at 2026-05-25 03:42:31.738494117 +0000 UTC m=+77.911469561, 1.00504048s after stability 2026-05-25T03:42:31.755Z [INFO] server-32bc8e32-147a-f266-46ff-e80faac230b5: aborting pipeline replication: peer="{Voter server-fb7e4293-c0e4-e906-0cd5-0f249fe670d2 fb7e4293-c0e4-e906-0cd5-0f249fe670d2}" 2026-05-25T03:42:31.755Z [INFO] server-32bc8e32-147a-f266-46ff-e80faac230b5: aborting pipeline replication: peer="{Voter server-844c700a-29bd-6522-b610-cd84597e0b8b 844c700a-29bd-6522-b610-cd84597e0b8b}" === RUN TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true 2026-05-25T03:42:31.756Z [INFO] server-cce530f8-413b-e5cd-5fad-e5cf03d082d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cce530f8-413b-e5cd-5fad-e5cf03d082d5 Address:cce530f8-413b-e5cd-5fad-e5cf03d082d5} {Suffrage:Voter ID:server-c13ec5d7-b253-c9ef-2160-b38595199b39 Address:c13ec5d7-b253-c9ef-2160-b38595199b39} {Suffrage:Voter ID:server-fa6ec4ca-c43a-0074-3497-f68e552ec992 Address:fa6ec4ca-c43a-0074-3497-f68e552ec992}]" 2026-05-25T03:42:31.756Z [INFO] server-cce530f8-413b-e5cd-5fad-e5cf03d082d5: entering follower state: follower="Node at cce530f8-413b-e5cd-5fad-e5cf03d082d5 [Follower]" leader-address= leader-id= 2026-05-25T03:42:31.757Z [INFO] server-c13ec5d7-b253-c9ef-2160-b38595199b39: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cce530f8-413b-e5cd-5fad-e5cf03d082d5 Address:cce530f8-413b-e5cd-5fad-e5cf03d082d5} {Suffrage:Voter ID:server-c13ec5d7-b253-c9ef-2160-b38595199b39 Address:c13ec5d7-b253-c9ef-2160-b38595199b39} {Suffrage:Voter ID:server-fa6ec4ca-c43a-0074-3497-f68e552ec992 Address:fa6ec4ca-c43a-0074-3497-f68e552ec992}]" 2026-05-25T03:42:31.757Z [INFO] server-fa6ec4ca-c43a-0074-3497-f68e552ec992: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cce530f8-413b-e5cd-5fad-e5cf03d082d5 Address:cce530f8-413b-e5cd-5fad-e5cf03d082d5} {Suffrage:Voter ID:server-c13ec5d7-b253-c9ef-2160-b38595199b39 Address:c13ec5d7-b253-c9ef-2160-b38595199b39} {Suffrage:Voter ID:server-fa6ec4ca-c43a-0074-3497-f68e552ec992 Address:fa6ec4ca-c43a-0074-3497-f68e552ec992}]" 2026-05-25T03:42:31.757Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:31.757Z [INFO] server-c13ec5d7-b253-c9ef-2160-b38595199b39: entering follower state: follower="Node at c13ec5d7-b253-c9ef-2160-b38595199b39 [Follower]" leader-address= leader-id= 2026-05-25T03:42:31.757Z [INFO] server-fa6ec4ca-c43a-0074-3497-f68e552ec992: entering follower state: follower="Node at fa6ec4ca-c43a-0074-3497-f68e552ec992 [Follower]" leader-address= leader-id= 2026-05-25T03:42:32.410Z [WARN] server-fa6ec4ca-c43a-0074-3497-f68e552ec992: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:32.410Z [INFO] server-fa6ec4ca-c43a-0074-3497-f68e552ec992: entering candidate state: node="Node at fa6ec4ca-c43a-0074-3497-f68e552ec992 [Candidate]" term=2 2026-05-25T03:42:32.410Z [INFO] server-fa6ec4ca-c43a-0074-3497-f68e552ec992: election won: term=2 tally=2 2026-05-25T03:42:32.410Z [INFO] server-fa6ec4ca-c43a-0074-3497-f68e552ec992: entering leader state: leader="Node at fa6ec4ca-c43a-0074-3497-f68e552ec992 [Leader]" 2026-05-25T03:42:32.410Z [INFO] server-fa6ec4ca-c43a-0074-3497-f68e552ec992: added peer, starting replication: peer=server-cce530f8-413b-e5cd-5fad-e5cf03d082d5 2026-05-25T03:42:32.410Z [INFO] server-fa6ec4ca-c43a-0074-3497-f68e552ec992: added peer, starting replication: peer=server-c13ec5d7-b253-c9ef-2160-b38595199b39 2026-05-25T03:42:32.410Z [INFO] server-fa6ec4ca-c43a-0074-3497-f68e552ec992: pipelining replication: peer="{Voter server-cce530f8-413b-e5cd-5fad-e5cf03d082d5 cce530f8-413b-e5cd-5fad-e5cf03d082d5}" 2026-05-25T03:42:32.410Z [INFO] server-fa6ec4ca-c43a-0074-3497-f68e552ec992: pipelining replication: peer="{Voter server-c13ec5d7-b253-c9ef-2160-b38595199b39 c13ec5d7-b253-c9ef-2160-b38595199b39}" 2026-05-25T03:42:33.416Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:32.410810437 +0000 UTC m=+78.583785921 (1 nodes), 653.07116ms from start of poll, 654.23868ms from cluster start. Timeout at 2026-05-25 03:42:33.415812957 +0000 UTC m=+79.588788361, 1.00500244s after stability 2026-05-25T03:42:33.417Z [INFO] server-fa6ec4ca-c43a-0074-3497-f68e552ec992: starting snapshot up to: index=12 2026-05-25T03:42:33.417Z [INFO] creating new snapshot: path=/tmp/raft630188550/snapshots/2-12-1779680553417.tmp 2026-05-25T03:42:33.417Z [INFO] server-fa6ec4ca-c43a-0074-3497-f68e552ec992: snapshot complete up to: index=12 2026-05-25T03:42:33.419Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92371518-cc95-7d1f-60e7-1c4af7153209 Address:92371518-cc95-7d1f-60e7-1c4af7153209} {Suffrage:Voter ID:server-8925ae50-48b4-e064-a841-2dea8217fff5 Address:8925ae50-48b4-e064-a841-2dea8217fff5} {Suffrage:Voter ID:server-3539507b-0484-405e-8e8b-69842ec5a4e8 Address:3539507b-0484-405e-8e8b-69842ec5a4e8}]" 2026-05-25T03:42:33.420Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: entering follower state: follower="Node at 92371518-cc95-7d1f-60e7-1c4af7153209 [Follower]" leader-address= leader-id= 2026-05-25T03:42:33.420Z [INFO] server-8925ae50-48b4-e064-a841-2dea8217fff5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92371518-cc95-7d1f-60e7-1c4af7153209 Address:92371518-cc95-7d1f-60e7-1c4af7153209} {Suffrage:Voter ID:server-8925ae50-48b4-e064-a841-2dea8217fff5 Address:8925ae50-48b4-e064-a841-2dea8217fff5} {Suffrage:Voter ID:server-3539507b-0484-405e-8e8b-69842ec5a4e8 Address:3539507b-0484-405e-8e8b-69842ec5a4e8}]" 2026-05-25T03:42:33.420Z [INFO] server-8925ae50-48b4-e064-a841-2dea8217fff5: entering follower state: follower="Node at 8925ae50-48b4-e064-a841-2dea8217fff5 [Follower]" leader-address= leader-id= 2026-05-25T03:42:33.420Z [INFO] server-3539507b-0484-405e-8e8b-69842ec5a4e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92371518-cc95-7d1f-60e7-1c4af7153209 Address:92371518-cc95-7d1f-60e7-1c4af7153209} {Suffrage:Voter ID:server-8925ae50-48b4-e064-a841-2dea8217fff5 Address:8925ae50-48b4-e064-a841-2dea8217fff5} {Suffrage:Voter ID:server-3539507b-0484-405e-8e8b-69842ec5a4e8 Address:3539507b-0484-405e-8e8b-69842ec5a4e8}]" 2026-05-25T03:42:33.420Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:33.420Z [INFO] server-3539507b-0484-405e-8e8b-69842ec5a4e8: entering follower state: follower="Node at 3539507b-0484-405e-8e8b-69842ec5a4e8 [Follower]" leader-address= leader-id= 2026-05-25T03:42:33.965Z [WARN] server-92371518-cc95-7d1f-60e7-1c4af7153209: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:33.965Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: entering candidate state: node="Node at 92371518-cc95-7d1f-60e7-1c4af7153209 [Candidate]" term=2 2026-05-25T03:42:33.965Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: election won: term=2 tally=2 2026-05-25T03:42:33.965Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: entering leader state: leader="Node at 92371518-cc95-7d1f-60e7-1c4af7153209 [Leader]" 2026-05-25T03:42:33.965Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: added peer, starting replication: peer=server-8925ae50-48b4-e064-a841-2dea8217fff5 2026-05-25T03:42:33.965Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: added peer, starting replication: peer=server-3539507b-0484-405e-8e8b-69842ec5a4e8 2026-05-25T03:42:33.965Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: pipelining replication: peer="{Voter server-8925ae50-48b4-e064-a841-2dea8217fff5 8925ae50-48b4-e064-a841-2dea8217fff5}" 2026-05-25T03:42:33.965Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: pipelining replication: peer="{Voter server-3539507b-0484-405e-8e8b-69842ec5a4e8 3539507b-0484-405e-8e8b-69842ec5a4e8}" 2026-05-25T03:42:34.971Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:33.965575397 +0000 UTC m=+80.138550841 (1 nodes), 544.43956ms from start of poll, 545.85732ms from cluster start. Timeout at 2026-05-25 03:42:34.970581597 +0000 UTC m=+81.143557041, 1.0050062s after stability 2026-05-25T03:42:34.971Z [INFO] creating new snapshot: path=/tmp/raft1669314152/snapshots/2-1013-1779680554971.tmp 2026-05-25T03:42:34.972Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: copied to local snapshot: bytes=71 2026-05-25T03:42:34.972Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: snapshot restore progress: id=2-1013-1779680554971 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:34.972Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: removing all old logs from log store 2026-05-25T03:42:34.972Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: compacting logs: from=1 to=2 2026-05-25T03:42:34.972Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: restored user snapshot: index=1013 2026-05-25T03:42:34.972Z [ERROR] server-92371518-cc95-7d1f-60e7-1c4af7153209: failed to get log: index=2 error="log not found" 2026-05-25T03:42:34.972Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: aborting pipeline replication: peer="{Voter server-3539507b-0484-405e-8e8b-69842ec5a4e8 3539507b-0484-405e-8e8b-69842ec5a4e8}" 2026-05-25T03:42:34.972Z [ERROR] server-92371518-cc95-7d1f-60e7-1c4af7153209: failed to get log: index=2 error="log not found" 2026-05-25T03:42:34.972Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: aborting pipeline replication: peer="{Voter server-8925ae50-48b4-e064-a841-2dea8217fff5 8925ae50-48b4-e064-a841-2dea8217fff5}" 2026-05-25T03:42:34.978Z [ERROR] server-92371518-cc95-7d1f-60e7-1c4af7153209: failed to get log: index=2 error="log not found" 2026-05-25T03:42:34.979Z [INFO] creating new snapshot: path=/tmp/raft2984076337/snapshots/2-1013-1779680554979.tmp 2026-05-25T03:42:34.979Z [INFO] server-8925ae50-48b4-e064-a841-2dea8217fff5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:34.979Z [INFO] server-8925ae50-48b4-e064-a841-2dea8217fff5: copied to local snapshot: bytes=71 2026-05-25T03:42:34.981Z [INFO] server-8925ae50-48b4-e064-a841-2dea8217fff5: snapshot restore progress: id=2-1013-1779680554979 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:34.981Z [INFO] server-8925ae50-48b4-e064-a841-2dea8217fff5: removing all old logs from log store 2026-05-25T03:42:34.981Z [INFO] server-8925ae50-48b4-e064-a841-2dea8217fff5: compacting logs: from=1 to=2 2026-05-25T03:42:34.982Z [INFO] server-8925ae50-48b4-e064-a841-2dea8217fff5: Installed remote snapshot 2026-05-25T03:42:34.982Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: pipelining replication: peer="{Voter server-8925ae50-48b4-e064-a841-2dea8217fff5 8925ae50-48b4-e064-a841-2dea8217fff5}" 2026-05-25T03:42:34.982Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:34.983Z [ERROR] server-92371518-cc95-7d1f-60e7-1c4af7153209: failed to get log: index=2 error="log not found" 2026-05-25T03:42:34.984Z [INFO] creating new snapshot: path=/tmp/raft2708613141/snapshots/2-1013-1779680554984.tmp 2026-05-25T03:42:34.985Z [INFO] server-3539507b-0484-405e-8e8b-69842ec5a4e8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:34.985Z [INFO] server-3539507b-0484-405e-8e8b-69842ec5a4e8: copied to local snapshot: bytes=71 2026-05-25T03:42:34.985Z [INFO] server-3539507b-0484-405e-8e8b-69842ec5a4e8: snapshot restore progress: id=2-1013-1779680554984 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:34.985Z [INFO] server-3539507b-0484-405e-8e8b-69842ec5a4e8: removing all old logs from log store 2026-05-25T03:42:34.985Z [INFO] server-3539507b-0484-405e-8e8b-69842ec5a4e8: compacting logs: from=1 to=2 2026-05-25T03:42:34.985Z [INFO] server-3539507b-0484-405e-8e8b-69842ec5a4e8: Installed remote snapshot 2026-05-25T03:42:34.986Z [INFO] server-92371518-cc95-7d1f-60e7-1c4af7153209: pipelining replication: peer="{Voter server-3539507b-0484-405e-8e8b-69842ec5a4e8 3539507b-0484-405e-8e8b-69842ec5a4e8}" 2026-05-25T03:42:35.988Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:34.982927717 +0000 UTC m=+81.155903161 (2 nodes), 1.44µs from start of poll, 1.56320964s from cluster start. Timeout at 2026-05-25 03:42:35.987932397 +0000 UTC m=+82.160907841, 1.00500468s after stability 2026-05-25T03:42:35.999Z [INFO] server-fa6ec4ca-c43a-0074-3497-f68e552ec992: aborting pipeline replication: peer="{Voter server-c13ec5d7-b253-c9ef-2160-b38595199b39 c13ec5d7-b253-c9ef-2160-b38595199b39}" 2026-05-25T03:42:35.999Z [INFO] server-fa6ec4ca-c43a-0074-3497-f68e552ec992: aborting pipeline replication: peer="{Voter server-cce530f8-413b-e5cd-5fad-e5cf03d082d5 cce530f8-413b-e5cd-5fad-e5cf03d082d5}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false 2026-05-25T03:42:36.000Z [INFO] server-d5110120-3ccc-1492-1992-748e0978ecf8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5110120-3ccc-1492-1992-748e0978ecf8 Address:d5110120-3ccc-1492-1992-748e0978ecf8} {Suffrage:Voter ID:server-9cbdd670-069a-fd22-5abd-c9260004c7d3 Address:9cbdd670-069a-fd22-5abd-c9260004c7d3} {Suffrage:Voter ID:server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a Address:5c8e2f64-dc90-29aa-0fff-bf51969c7b8a}]" 2026-05-25T03:42:36.001Z [INFO] server-9cbdd670-069a-fd22-5abd-c9260004c7d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5110120-3ccc-1492-1992-748e0978ecf8 Address:d5110120-3ccc-1492-1992-748e0978ecf8} {Suffrage:Voter ID:server-9cbdd670-069a-fd22-5abd-c9260004c7d3 Address:9cbdd670-069a-fd22-5abd-c9260004c7d3} {Suffrage:Voter ID:server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a Address:5c8e2f64-dc90-29aa-0fff-bf51969c7b8a}]" 2026-05-25T03:42:36.001Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5110120-3ccc-1492-1992-748e0978ecf8 Address:d5110120-3ccc-1492-1992-748e0978ecf8} {Suffrage:Voter ID:server-9cbdd670-069a-fd22-5abd-c9260004c7d3 Address:9cbdd670-069a-fd22-5abd-c9260004c7d3} {Suffrage:Voter ID:server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a Address:5c8e2f64-dc90-29aa-0fff-bf51969c7b8a}]" 2026-05-25T03:42:36.001Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:36.001Z [INFO] server-d5110120-3ccc-1492-1992-748e0978ecf8: entering follower state: follower="Node at d5110120-3ccc-1492-1992-748e0978ecf8 [Follower]" leader-address= leader-id= 2026-05-25T03:42:36.001Z [INFO] server-9cbdd670-069a-fd22-5abd-c9260004c7d3: entering follower state: follower="Node at 9cbdd670-069a-fd22-5abd-c9260004c7d3 [Follower]" leader-address= leader-id= 2026-05-25T03:42:36.001Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: entering follower state: follower="Node at 5c8e2f64-dc90-29aa-0fff-bf51969c7b8a [Follower]" leader-address= leader-id= 2026-05-25T03:42:36.522Z [WARN] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:36.522Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: entering candidate state: node="Node at 5c8e2f64-dc90-29aa-0fff-bf51969c7b8a [Candidate]" term=2 2026-05-25T03:42:36.522Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: election won: term=2 tally=2 2026-05-25T03:42:36.522Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: entering leader state: leader="Node at 5c8e2f64-dc90-29aa-0fff-bf51969c7b8a [Leader]" 2026-05-25T03:42:36.522Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: added peer, starting replication: peer=server-d5110120-3ccc-1492-1992-748e0978ecf8 2026-05-25T03:42:36.522Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: added peer, starting replication: peer=server-9cbdd670-069a-fd22-5abd-c9260004c7d3 2026-05-25T03:42:36.522Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: pipelining replication: peer="{Voter server-9cbdd670-069a-fd22-5abd-c9260004c7d3 9cbdd670-069a-fd22-5abd-c9260004c7d3}" 2026-05-25T03:42:36.522Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: pipelining replication: peer="{Voter server-d5110120-3ccc-1492-1992-748e0978ecf8 d5110120-3ccc-1492-1992-748e0978ecf8}" 2026-05-25T03:42:37.527Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:36.522817077 +0000 UTC m=+82.695792521 (1 nodes), 521.48896ms from start of poll, 522.05952ms from cluster start. Timeout at 2026-05-25 03:42:37.527818557 +0000 UTC m=+83.700794001, 1.00500148s after stability 2026-05-25T03:42:37.528Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: starting snapshot up to: index=12 2026-05-25T03:42:37.528Z [INFO] creating new snapshot: path=/tmp/raft3605706808/snapshots/2-12-1779680557528.tmp 2026-05-25T03:42:37.529Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: snapshot complete up to: index=12 2026-05-25T03:42:37.530Z [INFO] creating new snapshot: path=/tmp/raft3605706808/snapshots/2-10013-1779680557530.tmp 2026-05-25T03:42:37.530Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: copied to local snapshot: bytes=71 2026-05-25T03:42:37.530Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: snapshot restore progress: id=2-10013-1779680557530 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:37.530Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: restored user snapshot: index=10013 2026-05-25T03:42:37.530Z [ERROR] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: failed to get log: index=23 error="log not found" 2026-05-25T03:42:37.530Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: aborting pipeline replication: peer="{Voter server-9cbdd670-069a-fd22-5abd-c9260004c7d3 9cbdd670-069a-fd22-5abd-c9260004c7d3}" 2026-05-25T03:42:37.530Z [ERROR] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: failed to get log: index=23 error="log not found" 2026-05-25T03:42:37.530Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: aborting pipeline replication: peer="{Voter server-d5110120-3ccc-1492-1992-748e0978ecf8 d5110120-3ccc-1492-1992-748e0978ecf8}" 2026-05-25T03:42:37.538Z [ERROR] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: failed to get log: index=23 error="log not found" 2026-05-25T03:42:37.538Z [INFO] creating new snapshot: path=/tmp/raft4227153352/snapshots/2-10013-1779680557538.tmp 2026-05-25T03:42:37.538Z [ERROR] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: failed to get log: index=23 error="log not found" 2026-05-25T03:42:37.539Z [INFO] server-9cbdd670-069a-fd22-5abd-c9260004c7d3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:37.539Z [INFO] server-9cbdd670-069a-fd22-5abd-c9260004c7d3: copied to local snapshot: bytes=71 2026-05-25T03:42:37.539Z [INFO] creating new snapshot: path=/tmp/raft2415664811/snapshots/2-10013-1779680557539.tmp 2026-05-25T03:42:37.539Z [INFO] server-9cbdd670-069a-fd22-5abd-c9260004c7d3: snapshot restore progress: id=2-10013-1779680557538 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:37.539Z [INFO] server-d5110120-3ccc-1492-1992-748e0978ecf8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:37.539Z [INFO] server-9cbdd670-069a-fd22-5abd-c9260004c7d3: Installed remote snapshot 2026-05-25T03:42:37.539Z [WARN] server-9cbdd670-069a-fd22-5abd-c9260004c7d3: skipping application of old log: index=22 2026-05-25T03:42:37.539Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: pipelining replication: peer="{Voter server-9cbdd670-069a-fd22-5abd-c9260004c7d3 9cbdd670-069a-fd22-5abd-c9260004c7d3}" 2026-05-25T03:42:37.539Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:37.539Z [INFO] server-d5110120-3ccc-1492-1992-748e0978ecf8: copied to local snapshot: bytes=71 2026-05-25T03:42:37.540Z [INFO] server-d5110120-3ccc-1492-1992-748e0978ecf8: snapshot restore progress: id=2-10013-1779680557539 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:37.540Z [INFO] server-d5110120-3ccc-1492-1992-748e0978ecf8: Installed remote snapshot 2026-05-25T03:42:37.540Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: pipelining replication: peer="{Voter server-d5110120-3ccc-1492-1992-748e0978ecf8 d5110120-3ccc-1492-1992-748e0978ecf8}" 2026-05-25T03:42:38.544Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:37.539800837 +0000 UTC m=+83.712776281 (2 nodes), 960ns from start of poll, 1.53904328s from cluster start. Timeout at 2026-05-25 03:42:38.544810677 +0000 UTC m=+84.717786081, 1.0050098s after stability 2026-05-25T03:42:38.553Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: aborting pipeline replication: peer="{Voter server-9cbdd670-069a-fd22-5abd-c9260004c7d3 9cbdd670-069a-fd22-5abd-c9260004c7d3}" 2026-05-25T03:42:38.553Z [INFO] server-5c8e2f64-dc90-29aa-0fff-bf51969c7b8a: aborting pipeline replication: peer="{Voter server-d5110120-3ccc-1492-1992-748e0978ecf8 d5110120-3ccc-1492-1992-748e0978ecf8}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false 2026-05-25T03:42:38.554Z [INFO] server-c2e886cb-61cb-57da-ded9-b168ce4bc990: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2e886cb-61cb-57da-ded9-b168ce4bc990 Address:c2e886cb-61cb-57da-ded9-b168ce4bc990} {Suffrage:Voter ID:server-b71153de-ef2e-7448-3c95-4944a5c0a483 Address:b71153de-ef2e-7448-3c95-4944a5c0a483} {Suffrage:Voter ID:server-76dbb12f-d571-4cc3-72d1-24f734d4a773 Address:76dbb12f-d571-4cc3-72d1-24f734d4a773}]" 2026-05-25T03:42:38.554Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2e886cb-61cb-57da-ded9-b168ce4bc990 Address:c2e886cb-61cb-57da-ded9-b168ce4bc990} {Suffrage:Voter ID:server-b71153de-ef2e-7448-3c95-4944a5c0a483 Address:b71153de-ef2e-7448-3c95-4944a5c0a483} {Suffrage:Voter ID:server-76dbb12f-d571-4cc3-72d1-24f734d4a773 Address:76dbb12f-d571-4cc3-72d1-24f734d4a773}]" 2026-05-25T03:42:38.554Z [INFO] server-c2e886cb-61cb-57da-ded9-b168ce4bc990: entering follower state: follower="Node at c2e886cb-61cb-57da-ded9-b168ce4bc990 [Follower]" leader-address= leader-id= 2026-05-25T03:42:38.554Z [INFO] server-76dbb12f-d571-4cc3-72d1-24f734d4a773: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2e886cb-61cb-57da-ded9-b168ce4bc990 Address:c2e886cb-61cb-57da-ded9-b168ce4bc990} {Suffrage:Voter ID:server-b71153de-ef2e-7448-3c95-4944a5c0a483 Address:b71153de-ef2e-7448-3c95-4944a5c0a483} {Suffrage:Voter ID:server-76dbb12f-d571-4cc3-72d1-24f734d4a773 Address:76dbb12f-d571-4cc3-72d1-24f734d4a773}]" 2026-05-25T03:42:38.554Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: entering follower state: follower="Node at b71153de-ef2e-7448-3c95-4944a5c0a483 [Follower]" leader-address= leader-id= 2026-05-25T03:42:38.554Z [INFO] server-76dbb12f-d571-4cc3-72d1-24f734d4a773: entering follower state: follower="Node at 76dbb12f-d571-4cc3-72d1-24f734d4a773 [Follower]" leader-address= leader-id= 2026-05-25T03:42:38.554Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:39.139Z [WARN] server-b71153de-ef2e-7448-3c95-4944a5c0a483: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:39.139Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: entering candidate state: node="Node at b71153de-ef2e-7448-3c95-4944a5c0a483 [Candidate]" term=2 2026-05-25T03:42:39.139Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: election won: term=2 tally=2 2026-05-25T03:42:39.139Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: entering leader state: leader="Node at b71153de-ef2e-7448-3c95-4944a5c0a483 [Leader]" 2026-05-25T03:42:39.140Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: added peer, starting replication: peer=server-c2e886cb-61cb-57da-ded9-b168ce4bc990 2026-05-25T03:42:39.140Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: added peer, starting replication: peer=server-76dbb12f-d571-4cc3-72d1-24f734d4a773 2026-05-25T03:42:39.140Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: pipelining replication: peer="{Voter server-76dbb12f-d571-4cc3-72d1-24f734d4a773 76dbb12f-d571-4cc3-72d1-24f734d4a773}" 2026-05-25T03:42:39.140Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: pipelining replication: peer="{Voter server-c2e886cb-61cb-57da-ded9-b168ce4bc990 c2e886cb-61cb-57da-ded9-b168ce4bc990}" 2026-05-25T03:42:40.145Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:39.139983437 +0000 UTC m=+85.312958881 (1 nodes), 585.2378ms from start of poll, 585.57392ms from cluster start. Timeout at 2026-05-25 03:42:40.144987317 +0000 UTC m=+86.317962761, 1.00500388s after stability 2026-05-25T03:42:40.146Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: starting snapshot up to: index=12 2026-05-25T03:42:40.146Z [INFO] creating new snapshot: path=/tmp/raft2446383058/snapshots/2-12-1779680560146.tmp 2026-05-25T03:42:40.147Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: snapshot complete up to: index=12 2026-05-25T03:42:40.149Z [INFO] creating new snapshot: path=/tmp/raft2446383058/snapshots/2-10013-1779680560148.tmp 2026-05-25T03:42:40.149Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: copied to local snapshot: bytes=71 2026-05-25T03:42:40.150Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: snapshot restore progress: id=2-10013-1779680560148 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:40.150Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: removing all old logs from log store 2026-05-25T03:42:40.150Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: compacting logs: from=1 to=22 2026-05-25T03:42:40.150Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: restored user snapshot: index=10013 2026-05-25T03:42:40.150Z [ERROR] server-b71153de-ef2e-7448-3c95-4944a5c0a483: failed to get log: index=22 error="log not found" 2026-05-25T03:42:40.150Z [ERROR] server-b71153de-ef2e-7448-3c95-4944a5c0a483: failed to get log: index=22 error="log not found" 2026-05-25T03:42:40.151Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: aborting pipeline replication: peer="{Voter server-c2e886cb-61cb-57da-ded9-b168ce4bc990 c2e886cb-61cb-57da-ded9-b168ce4bc990}" 2026-05-25T03:42:40.150Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: aborting pipeline replication: peer="{Voter server-76dbb12f-d571-4cc3-72d1-24f734d4a773 76dbb12f-d571-4cc3-72d1-24f734d4a773}" 2026-05-25T03:42:40.159Z [ERROR] server-b71153de-ef2e-7448-3c95-4944a5c0a483: failed to get log: index=22 error="log not found" 2026-05-25T03:42:40.160Z [INFO] creating new snapshot: path=/tmp/raft2241768606/snapshots/2-10013-1779680560160.tmp 2026-05-25T03:42:40.160Z [INFO] server-c2e886cb-61cb-57da-ded9-b168ce4bc990: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:40.160Z [INFO] server-c2e886cb-61cb-57da-ded9-b168ce4bc990: copied to local snapshot: bytes=71 2026-05-25T03:42:40.160Z [INFO] server-c2e886cb-61cb-57da-ded9-b168ce4bc990: snapshot restore progress: id=2-10013-1779680560160 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:40.161Z [INFO] server-c2e886cb-61cb-57da-ded9-b168ce4bc990: removing all old logs from log store 2026-05-25T03:42:40.161Z [INFO] server-c2e886cb-61cb-57da-ded9-b168ce4bc990: compacting logs: from=1 to=22 2026-05-25T03:42:40.161Z [INFO] server-c2e886cb-61cb-57da-ded9-b168ce4bc990: Installed remote snapshot 2026-05-25T03:42:40.161Z [ERROR] server-b71153de-ef2e-7448-3c95-4944a5c0a483: failed to get log: index=22 error="log not found" 2026-05-25T03:42:40.161Z [WARN] server-c2e886cb-61cb-57da-ded9-b168ce4bc990: skipping application of old log: index=22 2026-05-25T03:42:40.161Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: pipelining replication: peer="{Voter server-c2e886cb-61cb-57da-ded9-b168ce4bc990 c2e886cb-61cb-57da-ded9-b168ce4bc990}" 2026-05-25T03:42:40.161Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:40.161Z [INFO] creating new snapshot: path=/tmp/raft4294603994/snapshots/2-10013-1779680560161.tmp 2026-05-25T03:42:40.161Z [INFO] server-76dbb12f-d571-4cc3-72d1-24f734d4a773: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:40.162Z [INFO] server-76dbb12f-d571-4cc3-72d1-24f734d4a773: copied to local snapshot: bytes=71 2026-05-25T03:42:40.162Z [INFO] server-76dbb12f-d571-4cc3-72d1-24f734d4a773: snapshot restore progress: id=2-10013-1779680560161 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:40.162Z [INFO] server-76dbb12f-d571-4cc3-72d1-24f734d4a773: removing all old logs from log store 2026-05-25T03:42:40.162Z [INFO] server-76dbb12f-d571-4cc3-72d1-24f734d4a773: compacting logs: from=1 to=22 2026-05-25T03:42:40.162Z [INFO] server-76dbb12f-d571-4cc3-72d1-24f734d4a773: Installed remote snapshot 2026-05-25T03:42:40.162Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: pipelining replication: peer="{Voter server-76dbb12f-d571-4cc3-72d1-24f734d4a773 76dbb12f-d571-4cc3-72d1-24f734d4a773}" 2026-05-25T03:42:41.166Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:40.161320757 +0000 UTC m=+86.334296201 (2 nodes), 960ns from start of poll, 1.60691124s from cluster start. Timeout at 2026-05-25 03:42:41.166349957 +0000 UTC m=+87.339325401, 1.0050292s after stability 2026-05-25T03:42:41.183Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: aborting pipeline replication: peer="{Voter server-76dbb12f-d571-4cc3-72d1-24f734d4a773 76dbb12f-d571-4cc3-72d1-24f734d4a773}" 2026-05-25T03:42:41.183Z [INFO] server-b71153de-ef2e-7448-3c95-4944a5c0a483: aborting pipeline replication: peer="{Voter server-c2e886cb-61cb-57da-ded9-b168ce4bc990 c2e886cb-61cb-57da-ded9-b168ce4bc990}" === RUN TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true 2026-05-25T03:42:41.184Z [INFO] server-4cd27625-8d2b-0a64-fa00-4e2ffc5b9cd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cd27625-8d2b-0a64-fa00-4e2ffc5b9cd6 Address:4cd27625-8d2b-0a64-fa00-4e2ffc5b9cd6} {Suffrage:Voter ID:server-9feb938f-7dd7-56aa-7a7d-6f2556c63def Address:9feb938f-7dd7-56aa-7a7d-6f2556c63def} {Suffrage:Voter ID:server-409b691d-bd45-4732-e1bd-08edb9012c88 Address:409b691d-bd45-4732-e1bd-08edb9012c88}]" 2026-05-25T03:42:41.185Z [INFO] server-9feb938f-7dd7-56aa-7a7d-6f2556c63def: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cd27625-8d2b-0a64-fa00-4e2ffc5b9cd6 Address:4cd27625-8d2b-0a64-fa00-4e2ffc5b9cd6} {Suffrage:Voter ID:server-9feb938f-7dd7-56aa-7a7d-6f2556c63def Address:9feb938f-7dd7-56aa-7a7d-6f2556c63def} {Suffrage:Voter ID:server-409b691d-bd45-4732-e1bd-08edb9012c88 Address:409b691d-bd45-4732-e1bd-08edb9012c88}]" 2026-05-25T03:42:41.185Z [INFO] server-4cd27625-8d2b-0a64-fa00-4e2ffc5b9cd6: entering follower state: follower="Node at 4cd27625-8d2b-0a64-fa00-4e2ffc5b9cd6 [Follower]" leader-address= leader-id= 2026-05-25T03:42:41.185Z [INFO] server-409b691d-bd45-4732-e1bd-08edb9012c88: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cd27625-8d2b-0a64-fa00-4e2ffc5b9cd6 Address:4cd27625-8d2b-0a64-fa00-4e2ffc5b9cd6} {Suffrage:Voter ID:server-9feb938f-7dd7-56aa-7a7d-6f2556c63def Address:9feb938f-7dd7-56aa-7a7d-6f2556c63def} {Suffrage:Voter ID:server-409b691d-bd45-4732-e1bd-08edb9012c88 Address:409b691d-bd45-4732-e1bd-08edb9012c88}]" 2026-05-25T03:42:41.185Z [INFO] server-9feb938f-7dd7-56aa-7a7d-6f2556c63def: entering follower state: follower="Node at 9feb938f-7dd7-56aa-7a7d-6f2556c63def [Follower]" leader-address= leader-id= 2026-05-25T03:42:41.185Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:41.185Z [INFO] server-409b691d-bd45-4732-e1bd-08edb9012c88: entering follower state: follower="Node at 409b691d-bd45-4732-e1bd-08edb9012c88 [Follower]" leader-address= leader-id= 2026-05-25T03:42:41.851Z [WARN] server-409b691d-bd45-4732-e1bd-08edb9012c88: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:41.851Z [INFO] server-409b691d-bd45-4732-e1bd-08edb9012c88: entering candidate state: node="Node at 409b691d-bd45-4732-e1bd-08edb9012c88 [Candidate]" term=2 2026-05-25T03:42:41.852Z [INFO] server-409b691d-bd45-4732-e1bd-08edb9012c88: election won: term=2 tally=2 2026-05-25T03:42:41.852Z [INFO] server-409b691d-bd45-4732-e1bd-08edb9012c88: entering leader state: leader="Node at 409b691d-bd45-4732-e1bd-08edb9012c88 [Leader]" 2026-05-25T03:42:41.852Z [INFO] server-409b691d-bd45-4732-e1bd-08edb9012c88: added peer, starting replication: peer=server-4cd27625-8d2b-0a64-fa00-4e2ffc5b9cd6 2026-05-25T03:42:41.852Z [INFO] server-409b691d-bd45-4732-e1bd-08edb9012c88: added peer, starting replication: peer=server-9feb938f-7dd7-56aa-7a7d-6f2556c63def 2026-05-25T03:42:41.852Z [INFO] server-409b691d-bd45-4732-e1bd-08edb9012c88: pipelining replication: peer="{Voter server-4cd27625-8d2b-0a64-fa00-4e2ffc5b9cd6 4cd27625-8d2b-0a64-fa00-4e2ffc5b9cd6}" 2026-05-25T03:42:41.852Z [INFO] server-409b691d-bd45-4732-e1bd-08edb9012c88: pipelining replication: peer="{Voter server-9feb938f-7dd7-56aa-7a7d-6f2556c63def 9feb938f-7dd7-56aa-7a7d-6f2556c63def}" 2026-05-25T03:42:42.857Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:41.852038917 +0000 UTC m=+88.025014361 (1 nodes), 666.82492ms from start of poll, 667.27804ms from cluster start. Timeout at 2026-05-25 03:42:42.857048077 +0000 UTC m=+89.030023481, 1.00500912s after stability 2026-05-25T03:42:42.858Z [INFO] server-409b691d-bd45-4732-e1bd-08edb9012c88: starting snapshot up to: index=12 2026-05-25T03:42:42.858Z [INFO] creating new snapshot: path=/tmp/raft2691159727/snapshots/2-12-1779680562858.tmp 2026-05-25T03:42:42.859Z [INFO] server-409b691d-bd45-4732-e1bd-08edb9012c88: snapshot complete up to: index=12 2026-05-25T03:42:42.861Z [INFO] server-c9f1050f-52c6-2fba-5d27-f7fb755200fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9f1050f-52c6-2fba-5d27-f7fb755200fb Address:c9f1050f-52c6-2fba-5d27-f7fb755200fb} {Suffrage:Voter ID:server-5b132456-4404-9c41-e575-9e262e259ca0 Address:5b132456-4404-9c41-e575-9e262e259ca0} {Suffrage:Voter ID:server-e9fd2d53-8969-260b-510d-ed618f5ca3c5 Address:e9fd2d53-8969-260b-510d-ed618f5ca3c5}]" 2026-05-25T03:42:42.861Z [INFO] server-5b132456-4404-9c41-e575-9e262e259ca0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9f1050f-52c6-2fba-5d27-f7fb755200fb Address:c9f1050f-52c6-2fba-5d27-f7fb755200fb} {Suffrage:Voter ID:server-5b132456-4404-9c41-e575-9e262e259ca0 Address:5b132456-4404-9c41-e575-9e262e259ca0} {Suffrage:Voter ID:server-e9fd2d53-8969-260b-510d-ed618f5ca3c5 Address:e9fd2d53-8969-260b-510d-ed618f5ca3c5}]" 2026-05-25T03:42:42.861Z [INFO] server-c9f1050f-52c6-2fba-5d27-f7fb755200fb: entering follower state: follower="Node at c9f1050f-52c6-2fba-5d27-f7fb755200fb [Follower]" leader-address= leader-id= 2026-05-25T03:42:42.861Z [INFO] server-5b132456-4404-9c41-e575-9e262e259ca0: entering follower state: follower="Node at 5b132456-4404-9c41-e575-9e262e259ca0 [Follower]" leader-address= leader-id= 2026-05-25T03:42:42.861Z [INFO] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9f1050f-52c6-2fba-5d27-f7fb755200fb Address:c9f1050f-52c6-2fba-5d27-f7fb755200fb} {Suffrage:Voter ID:server-5b132456-4404-9c41-e575-9e262e259ca0 Address:5b132456-4404-9c41-e575-9e262e259ca0} {Suffrage:Voter ID:server-e9fd2d53-8969-260b-510d-ed618f5ca3c5 Address:e9fd2d53-8969-260b-510d-ed618f5ca3c5}]" 2026-05-25T03:42:42.861Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:42.861Z [INFO] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: entering follower state: follower="Node at e9fd2d53-8969-260b-510d-ed618f5ca3c5 [Follower]" leader-address= leader-id= 2026-05-25T03:42:43.460Z [WARN] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:43.461Z [INFO] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: entering candidate state: node="Node at e9fd2d53-8969-260b-510d-ed618f5ca3c5 [Candidate]" term=2 2026-05-25T03:42:43.461Z [INFO] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: election won: term=2 tally=2 2026-05-25T03:42:43.461Z [INFO] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: entering leader state: leader="Node at e9fd2d53-8969-260b-510d-ed618f5ca3c5 [Leader]" 2026-05-25T03:42:43.461Z [INFO] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: added peer, starting replication: peer=server-c9f1050f-52c6-2fba-5d27-f7fb755200fb 2026-05-25T03:42:43.461Z [INFO] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: added peer, starting replication: peer=server-5b132456-4404-9c41-e575-9e262e259ca0 2026-05-25T03:42:43.461Z [INFO] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: pipelining replication: peer="{Voter server-c9f1050f-52c6-2fba-5d27-f7fb755200fb c9f1050f-52c6-2fba-5d27-f7fb755200fb}" 2026-05-25T03:42:43.461Z [INFO] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: pipelining replication: peer="{Voter server-5b132456-4404-9c41-e575-9e262e259ca0 5b132456-4404-9c41-e575-9e262e259ca0}" 2026-05-25T03:42:44.466Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:43.461204797 +0000 UTC m=+89.634180241 (1 nodes), 599.4222ms from start of poll, 599.87944ms from cluster start. Timeout at 2026-05-25 03:42:44.466207677 +0000 UTC m=+90.639183121, 1.00500288s after stability 2026-05-25T03:42:44.467Z [INFO] creating new snapshot: path=/tmp/raft2218886360/snapshots/2-10013-1779680564467.tmp 2026-05-25T03:42:44.467Z [INFO] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: copied to local snapshot: bytes=71 2026-05-25T03:42:44.467Z [INFO] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: snapshot restore progress: id=2-10013-1779680564467 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:44.467Z [INFO] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: restored user snapshot: index=10013 2026-05-25T03:42:44.467Z [ERROR] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: failed to get log: index=3 error="log not found" 2026-05-25T03:42:44.467Z [ERROR] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: failed to get log: index=3 error="log not found" 2026-05-25T03:42:44.467Z [INFO] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: aborting pipeline replication: peer="{Voter server-c9f1050f-52c6-2fba-5d27-f7fb755200fb c9f1050f-52c6-2fba-5d27-f7fb755200fb}" 2026-05-25T03:42:44.467Z [INFO] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: aborting pipeline replication: peer="{Voter server-5b132456-4404-9c41-e575-9e262e259ca0 5b132456-4404-9c41-e575-9e262e259ca0}" 2026-05-25T03:42:44.476Z [ERROR] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: failed to get log: index=3 error="log not found" 2026-05-25T03:42:44.476Z [ERROR] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: failed to get log: index=3 error="log not found" 2026-05-25T03:42:44.476Z [INFO] creating new snapshot: path=/tmp/raft438031504/snapshots/2-10013-1779680564476.tmp 2026-05-25T03:42:44.477Z [INFO] creating new snapshot: path=/tmp/raft1695164310/snapshots/2-10013-1779680564477.tmp 2026-05-25T03:42:44.477Z [INFO] server-c9f1050f-52c6-2fba-5d27-f7fb755200fb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:44.477Z [INFO] server-5b132456-4404-9c41-e575-9e262e259ca0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:44.477Z [INFO] server-c9f1050f-52c6-2fba-5d27-f7fb755200fb: copied to local snapshot: bytes=71 2026-05-25T03:42:44.477Z [INFO] server-5b132456-4404-9c41-e575-9e262e259ca0: copied to local snapshot: bytes=71 2026-05-25T03:42:44.477Z [INFO] server-5b132456-4404-9c41-e575-9e262e259ca0: snapshot restore progress: id=2-10013-1779680564477 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:44.477Z [INFO] server-5b132456-4404-9c41-e575-9e262e259ca0: Installed remote snapshot 2026-05-25T03:42:44.477Z [INFO] server-c9f1050f-52c6-2fba-5d27-f7fb755200fb: snapshot restore progress: id=2-10013-1779680564476 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:44.478Z [INFO] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: pipelining replication: peer="{Voter server-5b132456-4404-9c41-e575-9e262e259ca0 5b132456-4404-9c41-e575-9e262e259ca0}" 2026-05-25T03:42:44.478Z [INFO] server-c9f1050f-52c6-2fba-5d27-f7fb755200fb: Installed remote snapshot 2026-05-25T03:42:44.478Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:44.478Z [INFO] server-e9fd2d53-8969-260b-510d-ed618f5ca3c5: pipelining replication: peer="{Voter server-c9f1050f-52c6-2fba-5d27-f7fb755200fb c9f1050f-52c6-2fba-5d27-f7fb755200fb}" 2026-05-25T03:42:45.483Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:44.478163437 +0000 UTC m=+90.651138881 (2 nodes), 880ns from start of poll, 1.61683808s from cluster start. Timeout at 2026-05-25 03:42:45.483277437 +0000 UTC m=+91.656252881, 1.005114s after stability 2026-05-25T03:42:45.495Z [INFO] server-409b691d-bd45-4732-e1bd-08edb9012c88: aborting pipeline replication: peer="{Voter server-9feb938f-7dd7-56aa-7a7d-6f2556c63def 9feb938f-7dd7-56aa-7a7d-6f2556c63def}" 2026-05-25T03:42:45.495Z [INFO] server-409b691d-bd45-4732-e1bd-08edb9012c88: aborting pipeline replication: peer="{Voter server-4cd27625-8d2b-0a64-fa00-4e2ffc5b9cd6 4cd27625-8d2b-0a64-fa00-4e2ffc5b9cd6}" === RUN TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true 2026-05-25T03:42:45.496Z [INFO] server-2275e49f-a535-7b2c-d0c8-bf86db1dd799: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2275e49f-a535-7b2c-d0c8-bf86db1dd799 Address:2275e49f-a535-7b2c-d0c8-bf86db1dd799} {Suffrage:Voter ID:server-00251c01-e6e7-5927-cc53-c778a1ee044e Address:00251c01-e6e7-5927-cc53-c778a1ee044e} {Suffrage:Voter ID:server-296700f2-8a69-5897-e1ce-d75bc27837f8 Address:296700f2-8a69-5897-e1ce-d75bc27837f8}]" 2026-05-25T03:42:45.496Z [INFO] server-2275e49f-a535-7b2c-d0c8-bf86db1dd799: entering follower state: follower="Node at 2275e49f-a535-7b2c-d0c8-bf86db1dd799 [Follower]" leader-address= leader-id= 2026-05-25T03:42:45.496Z [INFO] server-00251c01-e6e7-5927-cc53-c778a1ee044e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2275e49f-a535-7b2c-d0c8-bf86db1dd799 Address:2275e49f-a535-7b2c-d0c8-bf86db1dd799} {Suffrage:Voter ID:server-00251c01-e6e7-5927-cc53-c778a1ee044e Address:00251c01-e6e7-5927-cc53-c778a1ee044e} {Suffrage:Voter ID:server-296700f2-8a69-5897-e1ce-d75bc27837f8 Address:296700f2-8a69-5897-e1ce-d75bc27837f8}]" 2026-05-25T03:42:45.496Z [INFO] server-00251c01-e6e7-5927-cc53-c778a1ee044e: entering follower state: follower="Node at 00251c01-e6e7-5927-cc53-c778a1ee044e [Follower]" leader-address= leader-id= 2026-05-25T03:42:45.496Z [INFO] server-296700f2-8a69-5897-e1ce-d75bc27837f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2275e49f-a535-7b2c-d0c8-bf86db1dd799 Address:2275e49f-a535-7b2c-d0c8-bf86db1dd799} {Suffrage:Voter ID:server-00251c01-e6e7-5927-cc53-c778a1ee044e Address:00251c01-e6e7-5927-cc53-c778a1ee044e} {Suffrage:Voter ID:server-296700f2-8a69-5897-e1ce-d75bc27837f8 Address:296700f2-8a69-5897-e1ce-d75bc27837f8}]" 2026-05-25T03:42:45.496Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:45.496Z [INFO] server-296700f2-8a69-5897-e1ce-d75bc27837f8: entering follower state: follower="Node at 296700f2-8a69-5897-e1ce-d75bc27837f8 [Follower]" leader-address= leader-id= 2026-05-25T03:42:46.230Z [WARN] server-00251c01-e6e7-5927-cc53-c778a1ee044e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:46.230Z [INFO] server-00251c01-e6e7-5927-cc53-c778a1ee044e: entering candidate state: node="Node at 00251c01-e6e7-5927-cc53-c778a1ee044e [Candidate]" term=2 2026-05-25T03:42:46.230Z [INFO] server-00251c01-e6e7-5927-cc53-c778a1ee044e: election won: term=2 tally=2 2026-05-25T03:42:46.230Z [INFO] server-00251c01-e6e7-5927-cc53-c778a1ee044e: entering leader state: leader="Node at 00251c01-e6e7-5927-cc53-c778a1ee044e [Leader]" 2026-05-25T03:42:46.230Z [INFO] server-00251c01-e6e7-5927-cc53-c778a1ee044e: added peer, starting replication: peer=server-2275e49f-a535-7b2c-d0c8-bf86db1dd799 2026-05-25T03:42:46.230Z [INFO] server-00251c01-e6e7-5927-cc53-c778a1ee044e: added peer, starting replication: peer=server-296700f2-8a69-5897-e1ce-d75bc27837f8 2026-05-25T03:42:46.230Z [INFO] server-00251c01-e6e7-5927-cc53-c778a1ee044e: pipelining replication: peer="{Voter server-296700f2-8a69-5897-e1ce-d75bc27837f8 296700f2-8a69-5897-e1ce-d75bc27837f8}" 2026-05-25T03:42:46.230Z [INFO] server-00251c01-e6e7-5927-cc53-c778a1ee044e: pipelining replication: peer="{Voter server-2275e49f-a535-7b2c-d0c8-bf86db1dd799 2275e49f-a535-7b2c-d0c8-bf86db1dd799}" 2026-05-25T03:42:47.236Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:46.230579397 +0000 UTC m=+92.403554841 (1 nodes), 733.95052ms from start of poll, 734.40096ms from cluster start. Timeout at 2026-05-25 03:42:47.235581437 +0000 UTC m=+93.408556881, 1.00500204s after stability 2026-05-25T03:42:47.237Z [INFO] server-00251c01-e6e7-5927-cc53-c778a1ee044e: starting snapshot up to: index=12 2026-05-25T03:42:47.237Z [INFO] creating new snapshot: path=/tmp/raft398262724/snapshots/2-12-1779680567237.tmp 2026-05-25T03:42:47.237Z [INFO] server-00251c01-e6e7-5927-cc53-c778a1ee044e: snapshot complete up to: index=12 2026-05-25T03:42:47.239Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a11bd8ff-05d1-004c-41cb-dad43224d8ba Address:a11bd8ff-05d1-004c-41cb-dad43224d8ba} {Suffrage:Voter ID:server-55a2c96d-37ac-527e-3bde-33a416a64e86 Address:55a2c96d-37ac-527e-3bde-33a416a64e86} {Suffrage:Voter ID:server-3ac17f5e-8d84-75a1-95fa-713469d04071 Address:3ac17f5e-8d84-75a1-95fa-713469d04071}]" 2026-05-25T03:42:47.239Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: entering follower state: follower="Node at a11bd8ff-05d1-004c-41cb-dad43224d8ba [Follower]" leader-address= leader-id= 2026-05-25T03:42:47.239Z [INFO] server-55a2c96d-37ac-527e-3bde-33a416a64e86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a11bd8ff-05d1-004c-41cb-dad43224d8ba Address:a11bd8ff-05d1-004c-41cb-dad43224d8ba} {Suffrage:Voter ID:server-55a2c96d-37ac-527e-3bde-33a416a64e86 Address:55a2c96d-37ac-527e-3bde-33a416a64e86} {Suffrage:Voter ID:server-3ac17f5e-8d84-75a1-95fa-713469d04071 Address:3ac17f5e-8d84-75a1-95fa-713469d04071}]" 2026-05-25T03:42:47.239Z [INFO] server-55a2c96d-37ac-527e-3bde-33a416a64e86: entering follower state: follower="Node at 55a2c96d-37ac-527e-3bde-33a416a64e86 [Follower]" leader-address= leader-id= 2026-05-25T03:42:47.239Z [INFO] server-3ac17f5e-8d84-75a1-95fa-713469d04071: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a11bd8ff-05d1-004c-41cb-dad43224d8ba Address:a11bd8ff-05d1-004c-41cb-dad43224d8ba} {Suffrage:Voter ID:server-55a2c96d-37ac-527e-3bde-33a416a64e86 Address:55a2c96d-37ac-527e-3bde-33a416a64e86} {Suffrage:Voter ID:server-3ac17f5e-8d84-75a1-95fa-713469d04071 Address:3ac17f5e-8d84-75a1-95fa-713469d04071}]" 2026-05-25T03:42:47.239Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:47.239Z [INFO] server-3ac17f5e-8d84-75a1-95fa-713469d04071: entering follower state: follower="Node at 3ac17f5e-8d84-75a1-95fa-713469d04071 [Follower]" leader-address= leader-id= 2026-05-25T03:42:47.792Z [WARN] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:47.792Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: entering candidate state: node="Node at a11bd8ff-05d1-004c-41cb-dad43224d8ba [Candidate]" term=2 2026-05-25T03:42:47.792Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: election won: term=2 tally=2 2026-05-25T03:42:47.792Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: entering leader state: leader="Node at a11bd8ff-05d1-004c-41cb-dad43224d8ba [Leader]" 2026-05-25T03:42:47.792Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: added peer, starting replication: peer=server-55a2c96d-37ac-527e-3bde-33a416a64e86 2026-05-25T03:42:47.792Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: added peer, starting replication: peer=server-3ac17f5e-8d84-75a1-95fa-713469d04071 2026-05-25T03:42:47.792Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: pipelining replication: peer="{Voter server-55a2c96d-37ac-527e-3bde-33a416a64e86 55a2c96d-37ac-527e-3bde-33a416a64e86}" 2026-05-25T03:42:47.792Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: pipelining replication: peer="{Voter server-3ac17f5e-8d84-75a1-95fa-713469d04071 3ac17f5e-8d84-75a1-95fa-713469d04071}" 2026-05-25T03:42:48.798Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:47.792702317 +0000 UTC m=+93.965677761 (1 nodes), 553.18036ms from start of poll, 553.77844ms from cluster start. Timeout at 2026-05-25 03:42:48.797704677 +0000 UTC m=+94.970680121, 1.00500236s after stability 2026-05-25T03:42:48.798Z [INFO] creating new snapshot: path=/tmp/raft4146590316/snapshots/2-10013-1779680568798.tmp 2026-05-25T03:42:48.799Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: copied to local snapshot: bytes=71 2026-05-25T03:42:48.799Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: snapshot restore progress: id=2-10013-1779680568798 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:48.799Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: removing all old logs from log store 2026-05-25T03:42:48.799Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: compacting logs: from=1 to=2 2026-05-25T03:42:48.799Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: restored user snapshot: index=10013 2026-05-25T03:42:48.799Z [ERROR] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: failed to get log: index=2 error="log not found" 2026-05-25T03:42:48.799Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: aborting pipeline replication: peer="{Voter server-55a2c96d-37ac-527e-3bde-33a416a64e86 55a2c96d-37ac-527e-3bde-33a416a64e86}" 2026-05-25T03:42:48.799Z [ERROR] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: failed to get log: index=2 error="log not found" 2026-05-25T03:42:48.799Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: aborting pipeline replication: peer="{Voter server-3ac17f5e-8d84-75a1-95fa-713469d04071 3ac17f5e-8d84-75a1-95fa-713469d04071}" 2026-05-25T03:42:48.805Z [ERROR] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: failed to get log: index=2 error="log not found" 2026-05-25T03:42:48.805Z [ERROR] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: failed to get log: index=2 error="log not found" 2026-05-25T03:42:48.806Z [INFO] creating new snapshot: path=/tmp/raft2643338262/snapshots/2-10013-1779680568806.tmp 2026-05-25T03:42:48.806Z [INFO] creating new snapshot: path=/tmp/raft1442182703/snapshots/2-10013-1779680568806.tmp 2026-05-25T03:42:48.806Z [INFO] server-55a2c96d-37ac-527e-3bde-33a416a64e86: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:48.806Z [INFO] server-3ac17f5e-8d84-75a1-95fa-713469d04071: snapshot network transfer progress: read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:48.807Z [INFO] server-55a2c96d-37ac-527e-3bde-33a416a64e86: copied to local snapshot: bytes=71 2026-05-25T03:42:48.807Z [INFO] server-3ac17f5e-8d84-75a1-95fa-713469d04071: copied to local snapshot: bytes=71 2026-05-25T03:42:48.807Z [INFO] server-3ac17f5e-8d84-75a1-95fa-713469d04071: snapshot restore progress: id=2-10013-1779680568806 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:48.807Z [INFO] server-55a2c96d-37ac-527e-3bde-33a416a64e86: snapshot restore progress: id=2-10013-1779680568806 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00% 2026-05-25T03:42:48.807Z [INFO] server-3ac17f5e-8d84-75a1-95fa-713469d04071: removing all old logs from log store 2026-05-25T03:42:48.807Z [INFO] server-3ac17f5e-8d84-75a1-95fa-713469d04071: compacting logs: from=1 to=2 2026-05-25T03:42:48.807Z [INFO] server-55a2c96d-37ac-527e-3bde-33a416a64e86: removing all old logs from log store 2026-05-25T03:42:48.807Z [INFO] server-3ac17f5e-8d84-75a1-95fa-713469d04071: Installed remote snapshot 2026-05-25T03:42:48.807Z [INFO] server-55a2c96d-37ac-527e-3bde-33a416a64e86: compacting logs: from=1 to=2 2026-05-25T03:42:48.807Z [INFO] server-55a2c96d-37ac-527e-3bde-33a416a64e86: Installed remote snapshot 2026-05-25T03:42:48.807Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: pipelining replication: peer="{Voter server-3ac17f5e-8d84-75a1-95fa-713469d04071 3ac17f5e-8d84-75a1-95fa-713469d04071}" 2026-05-25T03:42:48.807Z [WARN] server-55a2c96d-37ac-527e-3bde-33a416a64e86: skipping application of old log: index=10013 2026-05-25T03:42:48.807Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:48.807Z [INFO] server-a11bd8ff-05d1-004c-41cb-dad43224d8ba: pipelining replication: peer="{Voter server-55a2c96d-37ac-527e-3bde-33a416a64e86 55a2c96d-37ac-527e-3bde-33a416a64e86}" 2026-05-25T03:42:49.813Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:48.807708517 +0000 UTC m=+94.980683961 (2 nodes), 960ns from start of poll, 1.56878464s from cluster start. Timeout at 2026-05-25 03:42:49.812730157 +0000 UTC m=+95.985705601, 1.00502164s after stability 2026-05-25T03:42:49.825Z [INFO] server-00251c01-e6e7-5927-cc53-c778a1ee044e: aborting pipeline replication: peer="{Voter server-296700f2-8a69-5897-e1ce-d75bc27837f8 296700f2-8a69-5897-e1ce-d75bc27837f8}" 2026-05-25T03:42:49.825Z [INFO] server-00251c01-e6e7-5927-cc53-c778a1ee044e: aborting pipeline replication: peer="{Voter server-2275e49f-a535-7b2c-d0c8-bf86db1dd799 2275e49f-a535-7b2c-d0c8-bf86db1dd799}" --- PASS: TestRaft_UserRestore (82.91s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.58s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.56s) --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.29s) --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.35s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.63s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.82s) --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.07s) --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.33s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.62s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.55s) --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.73s) --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.18s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.78s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.58s) --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.09s) --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.14s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.60s) --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.65s) --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.30s) --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.24s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.55s) --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.63s) --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.31s) --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.33s) === RUN TestRaft_SendSnapshotFollower 2026-05-25T03:42:49.826Z [INFO] server-c5e04314-aea1-4150-1d87-605efe22592b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c5e04314-aea1-4150-1d87-605efe22592b Address:c5e04314-aea1-4150-1d87-605efe22592b} {Suffrage:Voter ID:server-be3d45c0-6a0b-167f-a99d-857c30b7f625 Address:be3d45c0-6a0b-167f-a99d-857c30b7f625} {Suffrage:Voter ID:server-1547771d-a6ad-a7eb-5593-67556052d925 Address:1547771d-a6ad-a7eb-5593-67556052d925}]" 2026-05-25T03:42:49.826Z [INFO] server-be3d45c0-6a0b-167f-a99d-857c30b7f625: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c5e04314-aea1-4150-1d87-605efe22592b Address:c5e04314-aea1-4150-1d87-605efe22592b} {Suffrage:Voter ID:server-be3d45c0-6a0b-167f-a99d-857c30b7f625 Address:be3d45c0-6a0b-167f-a99d-857c30b7f625} {Suffrage:Voter ID:server-1547771d-a6ad-a7eb-5593-67556052d925 Address:1547771d-a6ad-a7eb-5593-67556052d925}]" 2026-05-25T03:42:49.826Z [INFO] server-c5e04314-aea1-4150-1d87-605efe22592b: entering follower state: follower="Node at c5e04314-aea1-4150-1d87-605efe22592b [Follower]" leader-address= leader-id= 2026-05-25T03:42:49.826Z [INFO] server-be3d45c0-6a0b-167f-a99d-857c30b7f625: entering follower state: follower="Node at be3d45c0-6a0b-167f-a99d-857c30b7f625 [Follower]" leader-address= leader-id= 2026-05-25T03:42:49.826Z [INFO] server-1547771d-a6ad-a7eb-5593-67556052d925: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c5e04314-aea1-4150-1d87-605efe22592b Address:c5e04314-aea1-4150-1d87-605efe22592b} {Suffrage:Voter ID:server-be3d45c0-6a0b-167f-a99d-857c30b7f625 Address:be3d45c0-6a0b-167f-a99d-857c30b7f625} {Suffrage:Voter ID:server-1547771d-a6ad-a7eb-5593-67556052d925 Address:1547771d-a6ad-a7eb-5593-67556052d925}]" 2026-05-25T03:42:49.826Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:49.826Z [INFO] server-1547771d-a6ad-a7eb-5593-67556052d925: entering follower state: follower="Node at 1547771d-a6ad-a7eb-5593-67556052d925 [Follower]" leader-address= leader-id= 2026-05-25T03:42:49.884Z [WARN] server-c5e04314-aea1-4150-1d87-605efe22592b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:49.884Z [INFO] server-c5e04314-aea1-4150-1d87-605efe22592b: entering candidate state: node="Node at c5e04314-aea1-4150-1d87-605efe22592b [Candidate]" term=2 2026-05-25T03:42:49.884Z [INFO] server-c5e04314-aea1-4150-1d87-605efe22592b: election won: term=2 tally=2 2026-05-25T03:42:49.884Z [INFO] server-c5e04314-aea1-4150-1d87-605efe22592b: entering leader state: leader="Node at c5e04314-aea1-4150-1d87-605efe22592b [Leader]" 2026-05-25T03:42:49.884Z [INFO] server-c5e04314-aea1-4150-1d87-605efe22592b: added peer, starting replication: peer=server-be3d45c0-6a0b-167f-a99d-857c30b7f625 2026-05-25T03:42:49.884Z [INFO] server-c5e04314-aea1-4150-1d87-605efe22592b: added peer, starting replication: peer=server-1547771d-a6ad-a7eb-5593-67556052d925 2026-05-25T03:42:49.884Z [INFO] server-c5e04314-aea1-4150-1d87-605efe22592b: pipelining replication: peer="{Voter server-be3d45c0-6a0b-167f-a99d-857c30b7f625 be3d45c0-6a0b-167f-a99d-857c30b7f625}" 2026-05-25T03:42:49.884Z [INFO] server-c5e04314-aea1-4150-1d87-605efe22592b: pipelining replication: peer="{Voter server-1547771d-a6ad-a7eb-5593-67556052d925 1547771d-a6ad-a7eb-5593-67556052d925}" 2026-05-25T03:42:49.989Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:49.884213117 +0000 UTC m=+96.057188561 (2 nodes), 57.77672ms from start of poll, 58.20736ms from cluster start. Timeout at 2026-05-25 03:42:49.989216877 +0000 UTC m=+96.162192321, 105.00376ms after stability 2026-05-25T03:42:49.989Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:50.096Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:49.989368117 +0000 UTC m=+96.162343561 (1 nodes), 2.24µs from start of poll, 163.36236ms from cluster start. Timeout at 2026-05-25 03:42:50.094396157 +0000 UTC m=+96.267371561, 105.028ms after stability 2026-05-25T03:42:50.096Z [ERROR] server-c5e04314-aea1-4150-1d87-605efe22592b: failed to pipeline appendEntries: peer="{Voter server-be3d45c0-6a0b-167f-a99d-857c30b7f625 be3d45c0-6a0b-167f-a99d-857c30b7f625}" error="append pipeline closed" 2026-05-25T03:42:50.109Z [ERROR] server-c5e04314-aea1-4150-1d87-605efe22592b: failed to heartbeat to: peer=be3d45c0-6a0b-167f-a99d-857c30b7f625 backoff time=10ms error="failed to connect to peer: be3d45c0-6a0b-167f-a99d-857c30b7f625" 2026-05-25T03:42:50.110Z [INFO] server-c5e04314-aea1-4150-1d87-605efe22592b: aborting pipeline replication: peer="{Voter server-be3d45c0-6a0b-167f-a99d-857c30b7f625 be3d45c0-6a0b-167f-a99d-857c30b7f625}" 2026-05-25T03:42:50.128Z [ERROR] server-c5e04314-aea1-4150-1d87-605efe22592b: failed to appendEntries to: peer="{Voter server-be3d45c0-6a0b-167f-a99d-857c30b7f625 be3d45c0-6a0b-167f-a99d-857c30b7f625}" error="failed to connect to peer: be3d45c0-6a0b-167f-a99d-857c30b7f625" 2026-05-25T03:42:50.136Z [ERROR] server-c5e04314-aea1-4150-1d87-605efe22592b: failed to heartbeat to: peer=be3d45c0-6a0b-167f-a99d-857c30b7f625 backoff time=10ms error="failed to connect to peer: be3d45c0-6a0b-167f-a99d-857c30b7f625" raft_test.go:1627: [INFO] Finished apply without behind follower 2026-05-25T03:42:50.139Z [INFO] server-c5e04314-aea1-4150-1d87-605efe22592b: starting snapshot up to: index=102 2026-05-25T03:42:50.139Z [INFO] creating new snapshot: path=/tmp/raft2008053165/snapshots/2-102-1779680570139.tmp 2026-05-25T03:42:50.139Z [INFO] server-c5e04314-aea1-4150-1d87-605efe22592b: compacting logs: from=1 to=92 2026-05-25T03:42:50.139Z [INFO] server-c5e04314-aea1-4150-1d87-605efe22592b: snapshot complete up to: index=102 2026-05-25T03:42:50.139Z [ERROR] server-c5e04314-aea1-4150-1d87-605efe22592b: failed to get log: index=2 error="log not found" 2026-05-25T03:42:50.139Z [ERROR] server-c5e04314-aea1-4150-1d87-605efe22592b: failed to install snapshot: id=2-102-1779680570139 error="failed to connect to peer: be3d45c0-6a0b-167f-a99d-857c30b7f625" 2026-05-25T03:42:50.139Z [ERROR] server-c5e04314-aea1-4150-1d87-605efe22592b: failed to send snapshot to: peer="{Voter server-be3d45c0-6a0b-167f-a99d-857c30b7f625 be3d45c0-6a0b-167f-a99d-857c30b7f625}" error="failed to connect to peer: be3d45c0-6a0b-167f-a99d-857c30b7f625" 2026-05-25T03:42:50.139Z [ERROR] server-be3d45c0-6a0b-167f-a99d-857c30b7f625: failed to take snapshot: error="nothing new to snapshot" 2026-05-25T03:42:50.139Z [INFO] server-1547771d-a6ad-a7eb-5593-67556052d925: starting snapshot up to: index=98 2026-05-25T03:42:50.139Z [INFO] creating new snapshot: path=/tmp/raft2175666558/snapshots/2-98-1779680570139.tmp 2026-05-25T03:42:50.140Z [INFO] server-1547771d-a6ad-a7eb-5593-67556052d925: compacting logs: from=1 to=92 2026-05-25T03:42:50.140Z [INFO] server-1547771d-a6ad-a7eb-5593-67556052d925: snapshot complete up to: index=98 2026-05-25T03:42:50.140Z [WARN] server-c5e04314-aea1-4150-1d87-605efe22592b: failed to contact: server-id=server-be3d45c0-6a0b-167f-a99d-857c30b7f625 time=51.25532ms 2026-05-25T03:42:50.152Z [ERROR] server-c5e04314-aea1-4150-1d87-605efe22592b: failed to get log: index=2 error="log not found" 2026-05-25T03:42:50.153Z [INFO] creating new snapshot: path=/tmp/raft2774006694/snapshots/2-102-1779680570153.tmp 2026-05-25T03:42:50.153Z [INFO] server-be3d45c0-6a0b-167f-a99d-857c30b7f625: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2026-05-25T03:42:50.153Z [INFO] server-be3d45c0-6a0b-167f-a99d-857c30b7f625: copied to local snapshot: bytes=693 2026-05-25T03:42:50.153Z [INFO] server-be3d45c0-6a0b-167f-a99d-857c30b7f625: snapshot restore progress: id=2-102-1779680570153 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2026-05-25T03:42:50.153Z [INFO] server-be3d45c0-6a0b-167f-a99d-857c30b7f625: Installed remote snapshot 2026-05-25T03:42:50.161Z [INFO] server-c5e04314-aea1-4150-1d87-605efe22592b: aborting pipeline replication: peer="{Voter server-1547771d-a6ad-a7eb-5593-67556052d925 1547771d-a6ad-a7eb-5593-67556052d925}" --- PASS: TestRaft_SendSnapshotFollower (0.34s) === RUN TestRaft_SendSnapshotAndLogsFollower 2026-05-25T03:42:50.161Z [INFO] server-b5fdd94e-00ef-c88e-160a-864b2b930453: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5fdd94e-00ef-c88e-160a-864b2b930453 Address:b5fdd94e-00ef-c88e-160a-864b2b930453} {Suffrage:Voter ID:server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1 Address:91bb2f5e-f793-6ebb-269f-4b10dfb213e1} {Suffrage:Voter ID:server-2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e Address:2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e}]" 2026-05-25T03:42:50.161Z [INFO] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5fdd94e-00ef-c88e-160a-864b2b930453 Address:b5fdd94e-00ef-c88e-160a-864b2b930453} {Suffrage:Voter ID:server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1 Address:91bb2f5e-f793-6ebb-269f-4b10dfb213e1} {Suffrage:Voter ID:server-2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e Address:2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e}]" 2026-05-25T03:42:50.161Z [INFO] server-2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5fdd94e-00ef-c88e-160a-864b2b930453 Address:b5fdd94e-00ef-c88e-160a-864b2b930453} {Suffrage:Voter ID:server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1 Address:91bb2f5e-f793-6ebb-269f-4b10dfb213e1} {Suffrage:Voter ID:server-2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e Address:2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e}]" 2026-05-25T03:42:50.161Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:50.161Z [INFO] server-b5fdd94e-00ef-c88e-160a-864b2b930453: entering follower state: follower="Node at b5fdd94e-00ef-c88e-160a-864b2b930453 [Follower]" leader-address= leader-id= 2026-05-25T03:42:50.161Z [INFO] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: entering follower state: follower="Node at 91bb2f5e-f793-6ebb-269f-4b10dfb213e1 [Follower]" leader-address= leader-id= 2026-05-25T03:42:50.161Z [INFO] server-2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e: entering follower state: follower="Node at 2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e [Follower]" leader-address= leader-id= 2026-05-25T03:42:50.220Z [WARN] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:50.220Z [INFO] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: entering candidate state: node="Node at 91bb2f5e-f793-6ebb-269f-4b10dfb213e1 [Candidate]" term=2 2026-05-25T03:42:50.221Z [WARN] server-2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:50.221Z [INFO] server-2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e: entering candidate state: node="Node at 2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e [Candidate]" term=2 2026-05-25T03:42:50.221Z [INFO] server-2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e: duplicate requestVote for same term: term=2 2026-05-25T03:42:50.221Z [INFO] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: election won: term=2 tally=2 2026-05-25T03:42:50.221Z [INFO] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: entering leader state: leader="Node at 91bb2f5e-f793-6ebb-269f-4b10dfb213e1 [Leader]" 2026-05-25T03:42:50.221Z [INFO] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: added peer, starting replication: peer=server-b5fdd94e-00ef-c88e-160a-864b2b930453 2026-05-25T03:42:50.221Z [INFO] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: added peer, starting replication: peer=server-2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e 2026-05-25T03:42:50.221Z [INFO] server-2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e: entering follower state: follower="Node at 2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e [Follower]" leader-address=91bb2f5e-f793-6ebb-269f-4b10dfb213e1 leader-id=server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1 2026-05-25T03:42:50.221Z [INFO] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: pipelining replication: peer="{Voter server-2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e 2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e}" 2026-05-25T03:42:50.221Z [INFO] server-b5fdd94e-00ef-c88e-160a-864b2b930453: duplicate requestVote for same term: term=2 2026-05-25T03:42:50.221Z [INFO] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: pipelining replication: peer="{Voter server-b5fdd94e-00ef-c88e-160a-864b2b930453 b5fdd94e-00ef-c88e-160a-864b2b930453}" 2026-05-25T03:42:50.221Z [WARN] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: rejecting vote request since we have a leader: from=2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e leader=91bb2f5e-f793-6ebb-269f-4b10dfb213e1 leader-id=server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1 2026-05-25T03:42:50.329Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:50.221796797 +0000 UTC m=+96.394772161 (2 nodes), 59.99692ms from start of poll, 60.18844ms from cluster start. Timeout at 2026-05-25 03:42:50.326799317 +0000 UTC m=+96.499774721, 105.00256ms after stability 2026-05-25T03:42:50.329Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:50.437Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:50.329539117 +0000 UTC m=+96.502514521 (1 nodes), 1.04µs from start of poll, 167.9308ms from cluster start. Timeout at 2026-05-25 03:42:50.434558597 +0000 UTC m=+96.607534001, 105.01948ms after stability 2026-05-25T03:42:50.437Z [ERROR] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: failed to pipeline appendEntries: peer="{Voter server-b5fdd94e-00ef-c88e-160a-864b2b930453 b5fdd94e-00ef-c88e-160a-864b2b930453}" error="append pipeline closed" 2026-05-25T03:42:50.437Z [INFO] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: aborting pipeline replication: peer="{Voter server-b5fdd94e-00ef-c88e-160a-864b2b930453 b5fdd94e-00ef-c88e-160a-864b2b930453}" 2026-05-25T03:42:50.437Z [ERROR] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: failed to appendEntries to: peer="{Voter server-b5fdd94e-00ef-c88e-160a-864b2b930453 b5fdd94e-00ef-c88e-160a-864b2b930453}" error="failed to connect to peer: b5fdd94e-00ef-c88e-160a-864b2b930453" raft_test.go:1669: [INFO] Finished apply without behind follower 2026-05-25T03:42:50.438Z [ERROR] server-b5fdd94e-00ef-c88e-160a-864b2b930453: failed to take snapshot: error="nothing new to snapshot" 2026-05-25T03:42:50.438Z [INFO] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: starting snapshot up to: index=102 2026-05-25T03:42:50.438Z [INFO] creating new snapshot: path=/tmp/raft2081337525/snapshots/2-102-1779680570438.tmp 2026-05-25T03:42:50.439Z [INFO] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: compacting logs: from=1 to=92 2026-05-25T03:42:50.439Z [INFO] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: snapshot complete up to: index=102 2026-05-25T03:42:50.439Z [INFO] server-2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e: starting snapshot up to: index=42 2026-05-25T03:42:50.439Z [INFO] creating new snapshot: path=/tmp/raft2472342750/snapshots/2-42-1779680570439.tmp 2026-05-25T03:42:50.439Z [INFO] server-2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e: compacting logs: from=1 to=42 2026-05-25T03:42:50.439Z [INFO] server-2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e: snapshot complete up to: index=42 2026-05-25T03:42:50.439Z [ERROR] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: failed to heartbeat to: peer=b5fdd94e-00ef-c88e-160a-864b2b930453 backoff time=10ms error="failed to connect to peer: b5fdd94e-00ef-c88e-160a-864b2b930453" 2026-05-25T03:42:50.448Z [ERROR] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: failed to get log: index=2 error="log not found" raft_test.go:1690: [INFO] Finished apply without behind follower 2026-05-25T03:42:50.448Z [INFO] creating new snapshot: path=/tmp/raft1429304599/snapshots/2-102-1779680570448.tmp 2026-05-25T03:42:50.448Z [INFO] server-b5fdd94e-00ef-c88e-160a-864b2b930453: snapshot network transfer progress: read-bytes=693 percent-complete=100.00% 2026-05-25T03:42:50.448Z [INFO] server-b5fdd94e-00ef-c88e-160a-864b2b930453: copied to local snapshot: bytes=693 2026-05-25T03:42:50.448Z [INFO] server-b5fdd94e-00ef-c88e-160a-864b2b930453: snapshot restore progress: id=2-102-1779680570448 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00% 2026-05-25T03:42:50.448Z [INFO] server-b5fdd94e-00ef-c88e-160a-864b2b930453: Installed remote snapshot 2026-05-25T03:42:50.448Z [INFO] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: pipelining replication: peer="{Voter server-b5fdd94e-00ef-c88e-160a-864b2b930453 b5fdd94e-00ef-c88e-160a-864b2b930453}" 2026-05-25T03:42:50.456Z [INFO] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: aborting pipeline replication: peer="{Voter server-2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e 2a1bbaa9-c37f-6f46-9f69-44d4a45ab58e}" 2026-05-25T03:42:50.456Z [INFO] server-91bb2f5e-f793-6ebb-269f-4b10dfb213e1: aborting pipeline replication: peer="{Voter server-b5fdd94e-00ef-c88e-160a-864b2b930453 b5fdd94e-00ef-c88e-160a-864b2b930453}" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.30s) === RUN TestRaft_ReJoinFollower 2026-05-25T03:42:50.457Z [INFO] server-c6c439d2-8bd2-6329-9f62-360456ab1bde: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c6c439d2-8bd2-6329-9f62-360456ab1bde Address:c6c439d2-8bd2-6329-9f62-360456ab1bde} {Suffrage:Voter ID:server-d1e404f1-f656-66d8-ee16-6ec112191469 Address:d1e404f1-f656-66d8-ee16-6ec112191469} {Suffrage:Voter ID:server-802d39ed-c5f5-73f3-cd57-4845b6ecff4b Address:802d39ed-c5f5-73f3-cd57-4845b6ecff4b}]" 2026-05-25T03:42:50.457Z [INFO] server-d1e404f1-f656-66d8-ee16-6ec112191469: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c6c439d2-8bd2-6329-9f62-360456ab1bde Address:c6c439d2-8bd2-6329-9f62-360456ab1bde} {Suffrage:Voter ID:server-d1e404f1-f656-66d8-ee16-6ec112191469 Address:d1e404f1-f656-66d8-ee16-6ec112191469} {Suffrage:Voter ID:server-802d39ed-c5f5-73f3-cd57-4845b6ecff4b Address:802d39ed-c5f5-73f3-cd57-4845b6ecff4b}]" 2026-05-25T03:42:50.457Z [INFO] server-c6c439d2-8bd2-6329-9f62-360456ab1bde: entering follower state: follower="Node at c6c439d2-8bd2-6329-9f62-360456ab1bde [Follower]" leader-address= leader-id= 2026-05-25T03:42:50.457Z [INFO] server-d1e404f1-f656-66d8-ee16-6ec112191469: entering follower state: follower="Node at d1e404f1-f656-66d8-ee16-6ec112191469 [Follower]" leader-address= leader-id= 2026-05-25T03:42:50.457Z [INFO] server-802d39ed-c5f5-73f3-cd57-4845b6ecff4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c6c439d2-8bd2-6329-9f62-360456ab1bde Address:c6c439d2-8bd2-6329-9f62-360456ab1bde} {Suffrage:Voter ID:server-d1e404f1-f656-66d8-ee16-6ec112191469 Address:d1e404f1-f656-66d8-ee16-6ec112191469} {Suffrage:Voter ID:server-802d39ed-c5f5-73f3-cd57-4845b6ecff4b Address:802d39ed-c5f5-73f3-cd57-4845b6ecff4b}]" 2026-05-25T03:42:50.457Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:50.457Z [INFO] server-802d39ed-c5f5-73f3-cd57-4845b6ecff4b: entering follower state: follower="Node at 802d39ed-c5f5-73f3-cd57-4845b6ecff4b [Follower]" leader-address= leader-id= 2026-05-25T03:42:50.516Z [WARN] server-d1e404f1-f656-66d8-ee16-6ec112191469: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:50.516Z [INFO] server-d1e404f1-f656-66d8-ee16-6ec112191469: entering candidate state: node="Node at d1e404f1-f656-66d8-ee16-6ec112191469 [Candidate]" term=2 2026-05-25T03:42:50.516Z [INFO] server-d1e404f1-f656-66d8-ee16-6ec112191469: election won: term=2 tally=2 2026-05-25T03:42:50.516Z [INFO] server-d1e404f1-f656-66d8-ee16-6ec112191469: entering leader state: leader="Node at d1e404f1-f656-66d8-ee16-6ec112191469 [Leader]" 2026-05-25T03:42:50.516Z [INFO] server-d1e404f1-f656-66d8-ee16-6ec112191469: added peer, starting replication: peer=server-c6c439d2-8bd2-6329-9f62-360456ab1bde 2026-05-25T03:42:50.516Z [INFO] server-d1e404f1-f656-66d8-ee16-6ec112191469: added peer, starting replication: peer=server-802d39ed-c5f5-73f3-cd57-4845b6ecff4b 2026-05-25T03:42:50.516Z [INFO] server-d1e404f1-f656-66d8-ee16-6ec112191469: pipelining replication: peer="{Voter server-802d39ed-c5f5-73f3-cd57-4845b6ecff4b 802d39ed-c5f5-73f3-cd57-4845b6ecff4b}" 2026-05-25T03:42:50.516Z [INFO] server-d1e404f1-f656-66d8-ee16-6ec112191469: pipelining replication: peer="{Voter server-c6c439d2-8bd2-6329-9f62-360456ab1bde c6c439d2-8bd2-6329-9f62-360456ab1bde}" 2026-05-25T03:42:50.622Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:50.516833357 +0000 UTC m=+96.689808761 (1 nodes), 59.45964ms from start of poll, 59.66064ms from cluster start. Timeout at 2026-05-25 03:42:50.621838037 +0000 UTC m=+96.794813481, 105.00472ms after stability 2026-05-25T03:42:50.627Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:50.732Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:50.627579837 +0000 UTC m=+96.800555281 (2 nodes), 2.68µs from start of poll, 170.40716ms from cluster start. Timeout at 2026-05-25 03:42:50.732610237 +0000 UTC m=+96.905585681, 105.0304ms after stability 2026-05-25T03:42:50.732Z [INFO] server-d1e404f1-f656-66d8-ee16-6ec112191469: updating configuration: command=RemoveServer server-id=server-c6c439d2-8bd2-6329-9f62-360456ab1bde server-addr= servers="[{Suffrage:Voter ID:server-d1e404f1-f656-66d8-ee16-6ec112191469 Address:d1e404f1-f656-66d8-ee16-6ec112191469} {Suffrage:Voter ID:server-802d39ed-c5f5-73f3-cd57-4845b6ecff4b Address:802d39ed-c5f5-73f3-cd57-4845b6ecff4b}]" 2026-05-25T03:42:50.732Z [INFO] server-d1e404f1-f656-66d8-ee16-6ec112191469: removed peer, stopping replication: peer=server-c6c439d2-8bd2-6329-9f62-360456ab1bde last-index=3 2026-05-25T03:42:50.732Z [INFO] server-d1e404f1-f656-66d8-ee16-6ec112191469: aborting pipeline replication: peer="{Voter server-c6c439d2-8bd2-6329-9f62-360456ab1bde c6c439d2-8bd2-6329-9f62-360456ab1bde}" 2026-05-25T03:42:50.838Z [WARN] server-c6c439d2-8bd2-6329-9f62-360456ab1bde: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2026-05-25T03:42:50.838Z [INFO] server-d1e404f1-f656-66d8-ee16-6ec112191469: updating configuration: command=AddVoter server-id=server-c6c439d2-8bd2-6329-9f62-360456ab1bde server-addr=c6c439d2-8bd2-6329-9f62-360456ab1bde servers="[{Suffrage:Voter ID:server-d1e404f1-f656-66d8-ee16-6ec112191469 Address:d1e404f1-f656-66d8-ee16-6ec112191469} {Suffrage:Voter ID:server-802d39ed-c5f5-73f3-cd57-4845b6ecff4b Address:802d39ed-c5f5-73f3-cd57-4845b6ecff4b} {Suffrage:Voter ID:server-c6c439d2-8bd2-6329-9f62-360456ab1bde Address:c6c439d2-8bd2-6329-9f62-360456ab1bde}]" 2026-05-25T03:42:50.839Z [INFO] server-d1e404f1-f656-66d8-ee16-6ec112191469: added peer, starting replication: peer=server-c6c439d2-8bd2-6329-9f62-360456ab1bde 2026-05-25T03:42:50.839Z [WARN] server-c6c439d2-8bd2-6329-9f62-360456ab1bde: failed to get previous log: previous-index=4 last-index=3 error="log not found" 2026-05-25T03:42:50.839Z [WARN] server-d1e404f1-f656-66d8-ee16-6ec112191469: appendEntries rejected, sending older logs: peer="{Voter server-c6c439d2-8bd2-6329-9f62-360456ab1bde c6c439d2-8bd2-6329-9f62-360456ab1bde}" next=4 2026-05-25T03:42:50.839Z [INFO] server-d1e404f1-f656-66d8-ee16-6ec112191469: pipelining replication: peer="{Voter server-c6c439d2-8bd2-6329-9f62-360456ab1bde c6c439d2-8bd2-6329-9f62-360456ab1bde}" 2026-05-25T03:42:50.839Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:50.944Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:50.839454597 +0000 UTC m=+97.012430041 (1 nodes), 16.2µs from start of poll, 382.28192ms from cluster start. Timeout at 2026-05-25 03:42:50.944469917 +0000 UTC m=+97.117445361, 105.01532ms after stability 2026-05-25T03:42:50.944Z [INFO] server-d1e404f1-f656-66d8-ee16-6ec112191469: aborting pipeline replication: peer="{Voter server-802d39ed-c5f5-73f3-cd57-4845b6ecff4b 802d39ed-c5f5-73f3-cd57-4845b6ecff4b}" 2026-05-25T03:42:50.944Z [INFO] server-d1e404f1-f656-66d8-ee16-6ec112191469: aborting pipeline replication: peer="{Voter server-c6c439d2-8bd2-6329-9f62-360456ab1bde c6c439d2-8bd2-6329-9f62-360456ab1bde}" --- PASS: TestRaft_ReJoinFollower (0.49s) === RUN TestRaft_LeaderLeaseExpire 2026-05-25T03:42:50.945Z [INFO] server-5d1ddd93-819d-21b2-91e8-42f137c3f88e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5d1ddd93-819d-21b2-91e8-42f137c3f88e Address:5d1ddd93-819d-21b2-91e8-42f137c3f88e} {Suffrage:Voter ID:server-16a8803a-ea9b-732b-45a2-4a094944bc03 Address:16a8803a-ea9b-732b-45a2-4a094944bc03}]" 2026-05-25T03:42:50.945Z [INFO] server-16a8803a-ea9b-732b-45a2-4a094944bc03: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5d1ddd93-819d-21b2-91e8-42f137c3f88e Address:5d1ddd93-819d-21b2-91e8-42f137c3f88e} {Suffrage:Voter ID:server-16a8803a-ea9b-732b-45a2-4a094944bc03 Address:16a8803a-ea9b-732b-45a2-4a094944bc03}]" 2026-05-25T03:42:50.945Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:50.945Z [INFO] server-5d1ddd93-819d-21b2-91e8-42f137c3f88e: entering follower state: follower="Node at 5d1ddd93-819d-21b2-91e8-42f137c3f88e [Follower]" leader-address= leader-id= 2026-05-25T03:42:50.945Z [INFO] server-16a8803a-ea9b-732b-45a2-4a094944bc03: entering follower state: follower="Node at 16a8803a-ea9b-732b-45a2-4a094944bc03 [Follower]" leader-address= leader-id= 2026-05-25T03:42:51.026Z [WARN] server-16a8803a-ea9b-732b-45a2-4a094944bc03: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:51.026Z [INFO] server-16a8803a-ea9b-732b-45a2-4a094944bc03: entering candidate state: node="Node at 16a8803a-ea9b-732b-45a2-4a094944bc03 [Candidate]" term=2 2026-05-25T03:42:51.026Z [INFO] server-16a8803a-ea9b-732b-45a2-4a094944bc03: election won: term=2 tally=2 2026-05-25T03:42:51.026Z [INFO] server-16a8803a-ea9b-732b-45a2-4a094944bc03: entering leader state: leader="Node at 16a8803a-ea9b-732b-45a2-4a094944bc03 [Leader]" 2026-05-25T03:42:51.026Z [INFO] server-16a8803a-ea9b-732b-45a2-4a094944bc03: added peer, starting replication: peer=server-5d1ddd93-819d-21b2-91e8-42f137c3f88e 2026-05-25T03:42:51.026Z [INFO] server-16a8803a-ea9b-732b-45a2-4a094944bc03: pipelining replication: peer="{Voter server-5d1ddd93-819d-21b2-91e8-42f137c3f88e 5d1ddd93-819d-21b2-91e8-42f137c3f88e}" 2026-05-25T03:42:51.132Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:51.026398157 +0000 UTC m=+97.199373601 (1 nodes), 80.8186ms from start of poll, 81.11804ms from cluster start. Timeout at 2026-05-25 03:42:51.131401797 +0000 UTC m=+97.304377241, 105.00364ms after stability 2026-05-25T03:42:51.137Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:51.242Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:51.137594677 +0000 UTC m=+97.310570121 (1 nodes), 1.16µs from start of poll, 192.31456ms from cluster start. Timeout at 2026-05-25 03:42:51.242627317 +0000 UTC m=+97.415602761, 105.03264ms after stability raft_test.go:1799: [INFO] Disconnecting Node at 5d1ddd93-819d-21b2-91e8-42f137c3f88e [Follower] 2026-05-25T03:42:51.244Z [ERROR] server-16a8803a-ea9b-732b-45a2-4a094944bc03: failed to pipeline appendEntries: peer="{Voter server-5d1ddd93-819d-21b2-91e8-42f137c3f88e 5d1ddd93-819d-21b2-91e8-42f137c3f88e}" error="append pipeline closed" 2026-05-25T03:42:51.244Z [INFO] server-16a8803a-ea9b-732b-45a2-4a094944bc03: aborting pipeline replication: peer="{Voter server-5d1ddd93-819d-21b2-91e8-42f137c3f88e 5d1ddd93-819d-21b2-91e8-42f137c3f88e}" 2026-05-25T03:42:51.249Z [ERROR] server-16a8803a-ea9b-732b-45a2-4a094944bc03: failed to heartbeat to: peer=5d1ddd93-819d-21b2-91e8-42f137c3f88e backoff time=10ms error="failed to connect to peer: 5d1ddd93-819d-21b2-91e8-42f137c3f88e" 2026-05-25T03:42:51.250Z [ERROR] server-16a8803a-ea9b-732b-45a2-4a094944bc03: failed to appendEntries to: peer="{Voter server-5d1ddd93-819d-21b2-91e8-42f137c3f88e 5d1ddd93-819d-21b2-91e8-42f137c3f88e}" error="failed to connect to peer: 5d1ddd93-819d-21b2-91e8-42f137c3f88e" 2026-05-25T03:42:51.268Z [ERROR] server-16a8803a-ea9b-732b-45a2-4a094944bc03: failed to appendEntries to: peer="{Voter server-5d1ddd93-819d-21b2-91e8-42f137c3f88e 5d1ddd93-819d-21b2-91e8-42f137c3f88e}" error="failed to connect to peer: 5d1ddd93-819d-21b2-91e8-42f137c3f88e" 2026-05-25T03:42:51.268Z [ERROR] server-16a8803a-ea9b-732b-45a2-4a094944bc03: failed to heartbeat to: peer=5d1ddd93-819d-21b2-91e8-42f137c3f88e backoff time=10ms error="failed to connect to peer: 5d1ddd93-819d-21b2-91e8-42f137c3f88e" 2026-05-25T03:42:51.286Z [ERROR] server-16a8803a-ea9b-732b-45a2-4a094944bc03: failed to appendEntries to: peer="{Voter server-5d1ddd93-819d-21b2-91e8-42f137c3f88e 5d1ddd93-819d-21b2-91e8-42f137c3f88e}" error="failed to connect to peer: 5d1ddd93-819d-21b2-91e8-42f137c3f88e" 2026-05-25T03:42:51.288Z [ERROR] server-16a8803a-ea9b-732b-45a2-4a094944bc03: failed to heartbeat to: peer=5d1ddd93-819d-21b2-91e8-42f137c3f88e backoff time=10ms error="failed to connect to peer: 5d1ddd93-819d-21b2-91e8-42f137c3f88e" 2026-05-25T03:42:51.293Z [WARN] server-16a8803a-ea9b-732b-45a2-4a094944bc03: failed to contact: server-id=server-5d1ddd93-819d-21b2-91e8-42f137c3f88e time=50.61144ms 2026-05-25T03:42:51.293Z [WARN] server-16a8803a-ea9b-732b-45a2-4a094944bc03: failed to contact quorum of nodes, stepping down 2026-05-25T03:42:51.293Z [INFO] server-16a8803a-ea9b-732b-45a2-4a094944bc03: entering follower state: follower="Node at 16a8803a-ea9b-732b-45a2-4a094944bc03 [Follower]" leader-address= leader-id= 2026-05-25T03:42:51.293Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:51.355Z [WARN] server-5d1ddd93-819d-21b2-91e8-42f137c3f88e: heartbeat timeout reached, starting election: last-leader-addr=16a8803a-ea9b-732b-45a2-4a094944bc03 last-leader-id=server-16a8803a-ea9b-732b-45a2-4a094944bc03 2026-05-25T03:42:51.355Z [INFO] server-5d1ddd93-819d-21b2-91e8-42f137c3f88e: entering candidate state: node="Node at 5d1ddd93-819d-21b2-91e8-42f137c3f88e [Candidate]" term=3 2026-05-25T03:42:51.355Z [ERROR] server-5d1ddd93-819d-21b2-91e8-42f137c3f88e: failed to make requestVote RPC: target="{Voter server-16a8803a-ea9b-732b-45a2-4a094944bc03 16a8803a-ea9b-732b-45a2-4a094944bc03}" error="failed to connect to peer: 16a8803a-ea9b-732b-45a2-4a094944bc03" term=3 2026-05-25T03:42:51.370Z [WARN] server-16a8803a-ea9b-732b-45a2-4a094944bc03: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:51.370Z [INFO] server-16a8803a-ea9b-732b-45a2-4a094944bc03: entering candidate state: node="Node at 16a8803a-ea9b-732b-45a2-4a094944bc03 [Candidate]" term=3 2026-05-25T03:42:51.370Z [ERROR] server-16a8803a-ea9b-732b-45a2-4a094944bc03: failed to make requestVote RPC: target="{Voter server-5d1ddd93-819d-21b2-91e8-42f137c3f88e 5d1ddd93-819d-21b2-91e8-42f137c3f88e}" error="failed to connect to peer: 5d1ddd93-819d-21b2-91e8-42f137c3f88e" term=3 2026-05-25T03:42:51.448Z [WARN] server-5d1ddd93-819d-21b2-91e8-42f137c3f88e: Election timeout reached, restarting election 2026-05-25T03:42:51.448Z [INFO] server-5d1ddd93-819d-21b2-91e8-42f137c3f88e: entering candidate state: node="Node at 5d1ddd93-819d-21b2-91e8-42f137c3f88e [Candidate]" term=4 2026-05-25T03:42:51.448Z [ERROR] server-5d1ddd93-819d-21b2-91e8-42f137c3f88e: failed to make requestVote RPC: target="{Voter server-16a8803a-ea9b-732b-45a2-4a094944bc03 16a8803a-ea9b-732b-45a2-4a094944bc03}" error="failed to connect to peer: 16a8803a-ea9b-732b-45a2-4a094944bc03" term=4 2026-05-25T03:42:51.460Z [WARN] server-16a8803a-ea9b-732b-45a2-4a094944bc03: Election timeout reached, restarting election 2026-05-25T03:42:51.460Z [INFO] server-16a8803a-ea9b-732b-45a2-4a094944bc03: entering candidate state: node="Node at 16a8803a-ea9b-732b-45a2-4a094944bc03 [Candidate]" term=4 2026-05-25T03:42:51.460Z [ERROR] server-16a8803a-ea9b-732b-45a2-4a094944bc03: failed to make requestVote RPC: target="{Voter server-5d1ddd93-819d-21b2-91e8-42f137c3f88e 5d1ddd93-819d-21b2-91e8-42f137c3f88e}" error="failed to connect to peer: 5d1ddd93-819d-21b2-91e8-42f137c3f88e" term=4 2026-05-25T03:42:51.475Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:51.370303077 +0000 UTC m=+97.543278521 (0 nodes), 76.69016ms from start of poll, 425.02296ms from cluster start. Timeout at 2026-05-25 03:42:51.475311397 +0000 UTC m=+97.648286841, 105.00832ms after stability 2026-05-25T03:42:51.545Z [WARN] server-5d1ddd93-819d-21b2-91e8-42f137c3f88e: Election timeout reached, restarting election 2026-05-25T03:42:51.545Z [INFO] server-5d1ddd93-819d-21b2-91e8-42f137c3f88e: entering candidate state: node="Node at 5d1ddd93-819d-21b2-91e8-42f137c3f88e [Candidate]" term=5 2026-05-25T03:42:51.545Z [ERROR] server-5d1ddd93-819d-21b2-91e8-42f137c3f88e: failed to make requestVote RPC: target="{Voter server-16a8803a-ea9b-732b-45a2-4a094944bc03 16a8803a-ea9b-732b-45a2-4a094944bc03}" error="failed to connect to peer: 16a8803a-ea9b-732b-45a2-4a094944bc03" term=5 2026-05-25T03:42:51.558Z [WARN] server-16a8803a-ea9b-732b-45a2-4a094944bc03: Election timeout reached, restarting election 2026-05-25T03:42:51.558Z [INFO] server-16a8803a-ea9b-732b-45a2-4a094944bc03: entering candidate state: node="Node at 16a8803a-ea9b-732b-45a2-4a094944bc03 [Candidate]" term=5 2026-05-25T03:42:51.558Z [ERROR] server-16a8803a-ea9b-732b-45a2-4a094944bc03: failed to make requestVote RPC: target="{Voter server-5d1ddd93-819d-21b2-91e8-42f137c3f88e 5d1ddd93-819d-21b2-91e8-42f137c3f88e}" error="failed to connect to peer: 5d1ddd93-819d-21b2-91e8-42f137c3f88e" term=5 --- PASS: TestRaft_LeaderLeaseExpire (0.64s) === RUN TestRaft_Barrier 2026-05-25T03:42:51.582Z [INFO] server-dba0d59c-65c2-ea9c-367d-03f321c78d7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dba0d59c-65c2-ea9c-367d-03f321c78d7d Address:dba0d59c-65c2-ea9c-367d-03f321c78d7d} {Suffrage:Voter ID:server-9e6ffaf6-647b-9848-1a99-3b415f6f1532 Address:9e6ffaf6-647b-9848-1a99-3b415f6f1532} {Suffrage:Voter ID:server-4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d Address:4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d}]" 2026-05-25T03:42:51.583Z [INFO] server-dba0d59c-65c2-ea9c-367d-03f321c78d7d: entering follower state: follower="Node at dba0d59c-65c2-ea9c-367d-03f321c78d7d [Follower]" leader-address= leader-id= 2026-05-25T03:42:51.583Z [INFO] server-9e6ffaf6-647b-9848-1a99-3b415f6f1532: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dba0d59c-65c2-ea9c-367d-03f321c78d7d Address:dba0d59c-65c2-ea9c-367d-03f321c78d7d} {Suffrage:Voter ID:server-9e6ffaf6-647b-9848-1a99-3b415f6f1532 Address:9e6ffaf6-647b-9848-1a99-3b415f6f1532} {Suffrage:Voter ID:server-4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d Address:4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d}]" 2026-05-25T03:42:51.583Z [INFO] server-4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dba0d59c-65c2-ea9c-367d-03f321c78d7d Address:dba0d59c-65c2-ea9c-367d-03f321c78d7d} {Suffrage:Voter ID:server-9e6ffaf6-647b-9848-1a99-3b415f6f1532 Address:9e6ffaf6-647b-9848-1a99-3b415f6f1532} {Suffrage:Voter ID:server-4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d Address:4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d}]" 2026-05-25T03:42:51.583Z [INFO] server-9e6ffaf6-647b-9848-1a99-3b415f6f1532: entering follower state: follower="Node at 9e6ffaf6-647b-9848-1a99-3b415f6f1532 [Follower]" leader-address= leader-id= 2026-05-25T03:42:51.583Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:51.583Z [INFO] server-4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d: entering follower state: follower="Node at 4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d [Follower]" leader-address= leader-id= 2026-05-25T03:42:51.650Z [WARN] server-4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:51.650Z [INFO] server-4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d: entering candidate state: node="Node at 4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d [Candidate]" term=2 2026-05-25T03:42:51.650Z [INFO] server-4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d: election won: term=2 tally=2 2026-05-25T03:42:51.650Z [INFO] server-4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d: entering leader state: leader="Node at 4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d [Leader]" 2026-05-25T03:42:51.650Z [INFO] server-4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d: added peer, starting replication: peer=server-dba0d59c-65c2-ea9c-367d-03f321c78d7d 2026-05-25T03:42:51.650Z [INFO] server-4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d: added peer, starting replication: peer=server-9e6ffaf6-647b-9848-1a99-3b415f6f1532 2026-05-25T03:42:51.650Z [INFO] server-4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d: pipelining replication: peer="{Voter server-9e6ffaf6-647b-9848-1a99-3b415f6f1532 9e6ffaf6-647b-9848-1a99-3b415f6f1532}" 2026-05-25T03:42:51.650Z [INFO] server-4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d: pipelining replication: peer="{Voter server-dba0d59c-65c2-ea9c-367d-03f321c78d7d dba0d59c-65c2-ea9c-367d-03f321c78d7d}" 2026-05-25T03:42:51.756Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:51.650693757 +0000 UTC m=+97.823669201 (1 nodes), 67.17604ms from start of poll, 68.21024ms from cluster start. Timeout at 2026-05-25 03:42:51.755786797 +0000 UTC m=+97.928762241, 105.09304ms after stability 2026-05-25T03:42:51.772Z [INFO] server-4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d: aborting pipeline replication: peer="{Voter server-9e6ffaf6-647b-9848-1a99-3b415f6f1532 9e6ffaf6-647b-9848-1a99-3b415f6f1532}" 2026-05-25T03:42:51.772Z [INFO] server-4fd1a3ac-413a-c74e-aafb-b2ab2db9d42d: aborting pipeline replication: peer="{Voter server-dba0d59c-65c2-ea9c-367d-03f321c78d7d dba0d59c-65c2-ea9c-367d-03f321c78d7d}" --- PASS: TestRaft_Barrier (0.19s) === RUN TestRaft_VerifyLeader 2026-05-25T03:42:51.773Z [INFO] server-213ccf8c-8583-93d7-6ea3-d8bc399da4dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-213ccf8c-8583-93d7-6ea3-d8bc399da4dc Address:213ccf8c-8583-93d7-6ea3-d8bc399da4dc} {Suffrage:Voter ID:server-ff01c401-b61c-2271-51dc-2b57eed478dc Address:ff01c401-b61c-2271-51dc-2b57eed478dc} {Suffrage:Voter ID:server-3a8b2132-b261-4775-6b66-937da40bc323 Address:3a8b2132-b261-4775-6b66-937da40bc323}]" 2026-05-25T03:42:51.774Z [INFO] server-213ccf8c-8583-93d7-6ea3-d8bc399da4dc: entering follower state: follower="Node at 213ccf8c-8583-93d7-6ea3-d8bc399da4dc [Follower]" leader-address= leader-id= 2026-05-25T03:42:51.774Z [INFO] server-ff01c401-b61c-2271-51dc-2b57eed478dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-213ccf8c-8583-93d7-6ea3-d8bc399da4dc Address:213ccf8c-8583-93d7-6ea3-d8bc399da4dc} {Suffrage:Voter ID:server-ff01c401-b61c-2271-51dc-2b57eed478dc Address:ff01c401-b61c-2271-51dc-2b57eed478dc} {Suffrage:Voter ID:server-3a8b2132-b261-4775-6b66-937da40bc323 Address:3a8b2132-b261-4775-6b66-937da40bc323}]" 2026-05-25T03:42:51.774Z [INFO] server-ff01c401-b61c-2271-51dc-2b57eed478dc: entering follower state: follower="Node at ff01c401-b61c-2271-51dc-2b57eed478dc [Follower]" leader-address= leader-id= 2026-05-25T03:42:51.774Z [INFO] server-3a8b2132-b261-4775-6b66-937da40bc323: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-213ccf8c-8583-93d7-6ea3-d8bc399da4dc Address:213ccf8c-8583-93d7-6ea3-d8bc399da4dc} {Suffrage:Voter ID:server-ff01c401-b61c-2271-51dc-2b57eed478dc Address:ff01c401-b61c-2271-51dc-2b57eed478dc} {Suffrage:Voter ID:server-3a8b2132-b261-4775-6b66-937da40bc323 Address:3a8b2132-b261-4775-6b66-937da40bc323}]" 2026-05-25T03:42:51.774Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:51.774Z [INFO] server-3a8b2132-b261-4775-6b66-937da40bc323: entering follower state: follower="Node at 3a8b2132-b261-4775-6b66-937da40bc323 [Follower]" leader-address= leader-id= 2026-05-25T03:42:51.841Z [WARN] server-213ccf8c-8583-93d7-6ea3-d8bc399da4dc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:51.841Z [INFO] server-213ccf8c-8583-93d7-6ea3-d8bc399da4dc: entering candidate state: node="Node at 213ccf8c-8583-93d7-6ea3-d8bc399da4dc [Candidate]" term=2 2026-05-25T03:42:51.841Z [INFO] server-213ccf8c-8583-93d7-6ea3-d8bc399da4dc: election won: term=2 tally=2 2026-05-25T03:42:51.841Z [INFO] server-213ccf8c-8583-93d7-6ea3-d8bc399da4dc: entering leader state: leader="Node at 213ccf8c-8583-93d7-6ea3-d8bc399da4dc [Leader]" 2026-05-25T03:42:51.841Z [INFO] server-213ccf8c-8583-93d7-6ea3-d8bc399da4dc: added peer, starting replication: peer=server-ff01c401-b61c-2271-51dc-2b57eed478dc 2026-05-25T03:42:51.842Z [INFO] server-213ccf8c-8583-93d7-6ea3-d8bc399da4dc: added peer, starting replication: peer=server-3a8b2132-b261-4775-6b66-937da40bc323 2026-05-25T03:42:51.842Z [INFO] server-213ccf8c-8583-93d7-6ea3-d8bc399da4dc: pipelining replication: peer="{Voter server-ff01c401-b61c-2271-51dc-2b57eed478dc ff01c401-b61c-2271-51dc-2b57eed478dc}" 2026-05-25T03:42:51.842Z [INFO] server-213ccf8c-8583-93d7-6ea3-d8bc399da4dc: pipelining replication: peer="{Voter server-3a8b2132-b261-4775-6b66-937da40bc323 3a8b2132-b261-4775-6b66-937da40bc323}" 2026-05-25T03:42:51.947Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:51.842027997 +0000 UTC m=+98.015003441 (1 nodes), 67.56464ms from start of poll, 68.24528ms from cluster start. Timeout at 2026-05-25 03:42:51.947041317 +0000 UTC m=+98.120016761, 105.01332ms after stability 2026-05-25T03:42:51.947Z [INFO] server-213ccf8c-8583-93d7-6ea3-d8bc399da4dc: aborting pipeline replication: peer="{Voter server-3a8b2132-b261-4775-6b66-937da40bc323 3a8b2132-b261-4775-6b66-937da40bc323}" 2026-05-25T03:42:51.947Z [INFO] server-213ccf8c-8583-93d7-6ea3-d8bc399da4dc: aborting pipeline replication: peer="{Voter server-ff01c401-b61c-2271-51dc-2b57eed478dc ff01c401-b61c-2271-51dc-2b57eed478dc}" --- PASS: TestRaft_VerifyLeader (0.17s) === RUN TestRaft_VerifyLeader_Single 2026-05-25T03:42:51.948Z [INFO] server-216faa67-9cb1-314b-ad0a-79de05156261: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-216faa67-9cb1-314b-ad0a-79de05156261 Address:216faa67-9cb1-314b-ad0a-79de05156261}]" 2026-05-25T03:42:51.948Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:51.948Z [INFO] server-216faa67-9cb1-314b-ad0a-79de05156261: entering follower state: follower="Node at 216faa67-9cb1-314b-ad0a-79de05156261 [Follower]" leader-address= leader-id= 2026-05-25T03:42:52.039Z [WARN] server-216faa67-9cb1-314b-ad0a-79de05156261: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:52.039Z [INFO] server-216faa67-9cb1-314b-ad0a-79de05156261: entering candidate state: node="Node at 216faa67-9cb1-314b-ad0a-79de05156261 [Candidate]" term=2 2026-05-25T03:42:52.039Z [INFO] server-216faa67-9cb1-314b-ad0a-79de05156261: election won: term=2 tally=1 2026-05-25T03:42:52.039Z [INFO] server-216faa67-9cb1-314b-ad0a-79de05156261: entering leader state: leader="Node at 216faa67-9cb1-314b-ad0a-79de05156261 [Leader]" 2026-05-25T03:42:52.145Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:52.039258917 +0000 UTC m=+98.212234361 (1 nodes), 90.9272ms from start of poll, 91.11544ms from cluster start. Timeout at 2026-05-25 03:42:52.144261037 +0000 UTC m=+98.317236481, 105.00212ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.20s) === RUN TestRaft_VerifyLeader_Fail 2026-05-25T03:42:52.145Z [INFO] server-a0b32855-97cb-26b1-93a7-732ccae006a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0b32855-97cb-26b1-93a7-732ccae006a0 Address:a0b32855-97cb-26b1-93a7-732ccae006a0} {Suffrage:Voter ID:server-841b223b-78bb-ce9d-05a9-82595ee0414d Address:841b223b-78bb-ce9d-05a9-82595ee0414d}]" 2026-05-25T03:42:52.146Z [INFO] server-841b223b-78bb-ce9d-05a9-82595ee0414d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0b32855-97cb-26b1-93a7-732ccae006a0 Address:a0b32855-97cb-26b1-93a7-732ccae006a0} {Suffrage:Voter ID:server-841b223b-78bb-ce9d-05a9-82595ee0414d Address:841b223b-78bb-ce9d-05a9-82595ee0414d}]" 2026-05-25T03:42:52.146Z [INFO] server-a0b32855-97cb-26b1-93a7-732ccae006a0: entering follower state: follower="Node at a0b32855-97cb-26b1-93a7-732ccae006a0 [Follower]" leader-address= leader-id= 2026-05-25T03:42:52.146Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:52.146Z [INFO] server-841b223b-78bb-ce9d-05a9-82595ee0414d: entering follower state: follower="Node at 841b223b-78bb-ce9d-05a9-82595ee0414d [Follower]" leader-address= leader-id= 2026-05-25T03:42:52.213Z [WARN] server-a0b32855-97cb-26b1-93a7-732ccae006a0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:52.213Z [INFO] server-a0b32855-97cb-26b1-93a7-732ccae006a0: entering candidate state: node="Node at a0b32855-97cb-26b1-93a7-732ccae006a0 [Candidate]" term=2 2026-05-25T03:42:52.213Z [INFO] server-a0b32855-97cb-26b1-93a7-732ccae006a0: election won: term=2 tally=2 2026-05-25T03:42:52.213Z [INFO] server-a0b32855-97cb-26b1-93a7-732ccae006a0: entering leader state: leader="Node at a0b32855-97cb-26b1-93a7-732ccae006a0 [Leader]" 2026-05-25T03:42:52.213Z [INFO] server-a0b32855-97cb-26b1-93a7-732ccae006a0: added peer, starting replication: peer=server-841b223b-78bb-ce9d-05a9-82595ee0414d 2026-05-25T03:42:52.213Z [INFO] server-a0b32855-97cb-26b1-93a7-732ccae006a0: pipelining replication: peer="{Voter server-841b223b-78bb-ce9d-05a9-82595ee0414d 841b223b-78bb-ce9d-05a9-82595ee0414d}" 2026-05-25T03:42:52.319Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:52.213957357 +0000 UTC m=+98.386932801 (1 nodes), 67.83336ms from start of poll, 68.17516ms from cluster start. Timeout at 2026-05-25 03:42:52.318966797 +0000 UTC m=+98.491942241, 105.00944ms after stability 2026-05-25T03:42:52.319Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:52.425Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:52.319667717 +0000 UTC m=+98.492643161 (1 nodes), 2.4µs from start of poll, 173.88552ms from cluster start. Timeout at 2026-05-25 03:42:52.424693637 +0000 UTC m=+98.597669081, 105.02592ms after stability 2026-05-25T03:42:52.430Z [ERROR] server-a0b32855-97cb-26b1-93a7-732ccae006a0: peer has newer term, stopping replication: peer="{Voter server-841b223b-78bb-ce9d-05a9-82595ee0414d 841b223b-78bb-ce9d-05a9-82595ee0414d}" 2026-05-25T03:42:52.430Z [INFO] server-a0b32855-97cb-26b1-93a7-732ccae006a0: aborting pipeline replication: peer="{Voter server-841b223b-78bb-ce9d-05a9-82595ee0414d 841b223b-78bb-ce9d-05a9-82595ee0414d}" 2026-05-25T03:42:52.430Z [INFO] server-a0b32855-97cb-26b1-93a7-732ccae006a0: entering follower state: follower="Node at a0b32855-97cb-26b1-93a7-732ccae006a0 [Follower]" leader-address= leader-id= --- PASS: TestRaft_VerifyLeader_Fail (0.29s) === RUN TestRaft_VerifyLeader_PartialConnect 2026-05-25T03:42:52.431Z [INFO] server-ca86cd8a-d7fd-2321-1a15-f315c1c8924a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca86cd8a-d7fd-2321-1a15-f315c1c8924a Address:ca86cd8a-d7fd-2321-1a15-f315c1c8924a} {Suffrage:Voter ID:server-80423722-c2e8-59e4-053f-7ff9507b204c Address:80423722-c2e8-59e4-053f-7ff9507b204c} {Suffrage:Voter ID:server-afbcbf3e-4488-3c43-9fa1-280c71f478f0 Address:afbcbf3e-4488-3c43-9fa1-280c71f478f0}]" 2026-05-25T03:42:52.431Z [INFO] server-ca86cd8a-d7fd-2321-1a15-f315c1c8924a: entering follower state: follower="Node at ca86cd8a-d7fd-2321-1a15-f315c1c8924a [Follower]" leader-address= leader-id= 2026-05-25T03:42:52.431Z [INFO] server-80423722-c2e8-59e4-053f-7ff9507b204c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca86cd8a-d7fd-2321-1a15-f315c1c8924a Address:ca86cd8a-d7fd-2321-1a15-f315c1c8924a} {Suffrage:Voter ID:server-80423722-c2e8-59e4-053f-7ff9507b204c Address:80423722-c2e8-59e4-053f-7ff9507b204c} {Suffrage:Voter ID:server-afbcbf3e-4488-3c43-9fa1-280c71f478f0 Address:afbcbf3e-4488-3c43-9fa1-280c71f478f0}]" 2026-05-25T03:42:52.431Z [INFO] server-80423722-c2e8-59e4-053f-7ff9507b204c: entering follower state: follower="Node at 80423722-c2e8-59e4-053f-7ff9507b204c [Follower]" leader-address= leader-id= 2026-05-25T03:42:52.431Z [INFO] server-afbcbf3e-4488-3c43-9fa1-280c71f478f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca86cd8a-d7fd-2321-1a15-f315c1c8924a Address:ca86cd8a-d7fd-2321-1a15-f315c1c8924a} {Suffrage:Voter ID:server-80423722-c2e8-59e4-053f-7ff9507b204c Address:80423722-c2e8-59e4-053f-7ff9507b204c} {Suffrage:Voter ID:server-afbcbf3e-4488-3c43-9fa1-280c71f478f0 Address:afbcbf3e-4488-3c43-9fa1-280c71f478f0}]" 2026-05-25T03:42:52.431Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:52.431Z [INFO] server-afbcbf3e-4488-3c43-9fa1-280c71f478f0: entering follower state: follower="Node at afbcbf3e-4488-3c43-9fa1-280c71f478f0 [Follower]" leader-address= leader-id= 2026-05-25T03:42:52.494Z [WARN] server-ca86cd8a-d7fd-2321-1a15-f315c1c8924a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:52.494Z [INFO] server-ca86cd8a-d7fd-2321-1a15-f315c1c8924a: entering candidate state: node="Node at ca86cd8a-d7fd-2321-1a15-f315c1c8924a [Candidate]" term=2 2026-05-25T03:42:52.494Z [INFO] server-ca86cd8a-d7fd-2321-1a15-f315c1c8924a: election won: term=2 tally=2 2026-05-25T03:42:52.494Z [INFO] server-ca86cd8a-d7fd-2321-1a15-f315c1c8924a: entering leader state: leader="Node at ca86cd8a-d7fd-2321-1a15-f315c1c8924a [Leader]" 2026-05-25T03:42:52.494Z [INFO] server-ca86cd8a-d7fd-2321-1a15-f315c1c8924a: added peer, starting replication: peer=server-80423722-c2e8-59e4-053f-7ff9507b204c 2026-05-25T03:42:52.494Z [INFO] server-ca86cd8a-d7fd-2321-1a15-f315c1c8924a: added peer, starting replication: peer=server-afbcbf3e-4488-3c43-9fa1-280c71f478f0 2026-05-25T03:42:52.494Z [INFO] server-ca86cd8a-d7fd-2321-1a15-f315c1c8924a: pipelining replication: peer="{Voter server-afbcbf3e-4488-3c43-9fa1-280c71f478f0 afbcbf3e-4488-3c43-9fa1-280c71f478f0}" 2026-05-25T03:42:52.494Z [INFO] server-ca86cd8a-d7fd-2321-1a15-f315c1c8924a: pipelining replication: peer="{Voter server-80423722-c2e8-59e4-053f-7ff9507b204c 80423722-c2e8-59e4-053f-7ff9507b204c}" 2026-05-25T03:42:52.599Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:52.494435397 +0000 UTC m=+98.667410841 (1 nodes), 62.77548ms from start of poll, 63.234ms from cluster start. Timeout at 2026-05-25 03:42:52.599436757 +0000 UTC m=+98.772412201, 105.00136ms after stability 2026-05-25T03:42:52.604Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:52.710Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:52.604788317 +0000 UTC m=+98.777763761 (2 nodes), 3.24µs from start of poll, 173.58692ms from cluster start. Timeout at 2026-05-25 03:42:52.709822237 +0000 UTC m=+98.882797681, 105.03392ms after stability raft_test.go:1968: [INFO] Disconnecting Node at 80423722-c2e8-59e4-053f-7ff9507b204c [Follower] 2026-05-25T03:42:52.710Z [ERROR] server-ca86cd8a-d7fd-2321-1a15-f315c1c8924a: failed to heartbeat to: peer=80423722-c2e8-59e4-053f-7ff9507b204c backoff time=10ms error="failed to connect to peer: 80423722-c2e8-59e4-053f-7ff9507b204c" 2026-05-25T03:42:52.710Z [INFO] server-ca86cd8a-d7fd-2321-1a15-f315c1c8924a: aborting pipeline replication: peer="{Voter server-80423722-c2e8-59e4-053f-7ff9507b204c 80423722-c2e8-59e4-053f-7ff9507b204c}" 2026-05-25T03:42:52.710Z [INFO] server-ca86cd8a-d7fd-2321-1a15-f315c1c8924a: aborting pipeline replication: peer="{Voter server-afbcbf3e-4488-3c43-9fa1-280c71f478f0 afbcbf3e-4488-3c43-9fa1-280c71f478f0}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.28s) === RUN TestRaft_NotifyCh 2026-05-25T03:42:52.711Z [INFO] server-8077babd-cb3c-395b-78cf-b82b008ccda4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8077babd-cb3c-395b-78cf-b82b008ccda4 Address:8077babd-cb3c-395b-78cf-b82b008ccda4}]" 2026-05-25T03:42:52.711Z [INFO] server-8077babd-cb3c-395b-78cf-b82b008ccda4: entering follower state: follower="Node at 8077babd-cb3c-395b-78cf-b82b008ccda4 [Follower]" leader-address= leader-id= 2026-05-25T03:42:52.787Z [WARN] server-8077babd-cb3c-395b-78cf-b82b008ccda4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:52.787Z [INFO] server-8077babd-cb3c-395b-78cf-b82b008ccda4: entering candidate state: node="Node at 8077babd-cb3c-395b-78cf-b82b008ccda4 [Candidate]" term=2 2026-05-25T03:42:52.787Z [INFO] server-8077babd-cb3c-395b-78cf-b82b008ccda4: election won: term=2 tally=1 2026-05-25T03:42:52.787Z [INFO] server-8077babd-cb3c-395b-78cf-b82b008ccda4: entering leader state: leader="Node at 8077babd-cb3c-395b-78cf-b82b008ccda4 [Leader]" --- PASS: TestRaft_NotifyCh (0.08s) === RUN TestRaft_AppendEntry 2026-05-25T03:42:52.788Z [INFO] server-b768ac08-95e3-8dff-0777-4873bc9cc738: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b768ac08-95e3-8dff-0777-4873bc9cc738 Address:b768ac08-95e3-8dff-0777-4873bc9cc738} {Suffrage:Voter ID:server-011a3271-3631-9b39-5707-87ef39ef69a9 Address:011a3271-3631-9b39-5707-87ef39ef69a9} {Suffrage:Voter ID:server-1dd8553c-897d-60f4-53b2-df2869e619fe Address:1dd8553c-897d-60f4-53b2-df2869e619fe}]" 2026-05-25T03:42:52.788Z [INFO] server-b768ac08-95e3-8dff-0777-4873bc9cc738: entering follower state: follower="Node at b768ac08-95e3-8dff-0777-4873bc9cc738 [Follower]" leader-address= leader-id= 2026-05-25T03:42:52.788Z [INFO] server-011a3271-3631-9b39-5707-87ef39ef69a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b768ac08-95e3-8dff-0777-4873bc9cc738 Address:b768ac08-95e3-8dff-0777-4873bc9cc738} {Suffrage:Voter ID:server-011a3271-3631-9b39-5707-87ef39ef69a9 Address:011a3271-3631-9b39-5707-87ef39ef69a9} {Suffrage:Voter ID:server-1dd8553c-897d-60f4-53b2-df2869e619fe Address:1dd8553c-897d-60f4-53b2-df2869e619fe}]" 2026-05-25T03:42:52.788Z [INFO] server-011a3271-3631-9b39-5707-87ef39ef69a9: entering follower state: follower="Node at 011a3271-3631-9b39-5707-87ef39ef69a9 [Follower]" leader-address= leader-id= 2026-05-25T03:42:52.789Z [INFO] server-1dd8553c-897d-60f4-53b2-df2869e619fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b768ac08-95e3-8dff-0777-4873bc9cc738 Address:b768ac08-95e3-8dff-0777-4873bc9cc738} {Suffrage:Voter ID:server-011a3271-3631-9b39-5707-87ef39ef69a9 Address:011a3271-3631-9b39-5707-87ef39ef69a9} {Suffrage:Voter ID:server-1dd8553c-897d-60f4-53b2-df2869e619fe Address:1dd8553c-897d-60f4-53b2-df2869e619fe}]" 2026-05-25T03:42:52.789Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:52.789Z [INFO] server-1dd8553c-897d-60f4-53b2-df2869e619fe: entering follower state: follower="Node at 1dd8553c-897d-60f4-53b2-df2869e619fe [Follower]" leader-address= leader-id= 2026-05-25T03:42:52.852Z [WARN] server-011a3271-3631-9b39-5707-87ef39ef69a9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:52.852Z [INFO] server-011a3271-3631-9b39-5707-87ef39ef69a9: entering candidate state: node="Node at 011a3271-3631-9b39-5707-87ef39ef69a9 [Candidate]" term=2 2026-05-25T03:42:52.852Z [INFO] server-011a3271-3631-9b39-5707-87ef39ef69a9: election won: term=2 tally=2 2026-05-25T03:42:52.852Z [INFO] server-011a3271-3631-9b39-5707-87ef39ef69a9: entering leader state: leader="Node at 011a3271-3631-9b39-5707-87ef39ef69a9 [Leader]" 2026-05-25T03:42:52.852Z [INFO] server-011a3271-3631-9b39-5707-87ef39ef69a9: added peer, starting replication: peer=server-b768ac08-95e3-8dff-0777-4873bc9cc738 2026-05-25T03:42:52.852Z [INFO] server-011a3271-3631-9b39-5707-87ef39ef69a9: added peer, starting replication: peer=server-1dd8553c-897d-60f4-53b2-df2869e619fe 2026-05-25T03:42:52.853Z [INFO] server-011a3271-3631-9b39-5707-87ef39ef69a9: pipelining replication: peer="{Voter server-b768ac08-95e3-8dff-0777-4873bc9cc738 b768ac08-95e3-8dff-0777-4873bc9cc738}" 2026-05-25T03:42:52.853Z [INFO] server-011a3271-3631-9b39-5707-87ef39ef69a9: pipelining replication: peer="{Voter server-1dd8553c-897d-60f4-53b2-df2869e619fe 1dd8553c-897d-60f4-53b2-df2869e619fe}" 2026-05-25T03:42:52.958Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:52.852960197 +0000 UTC m=+99.025935641 (2 nodes), 63.72456ms from start of poll, 64.35908ms from cluster start. Timeout at 2026-05-25 03:42:52.957970157 +0000 UTC m=+99.130945601, 105.00996ms after stability 2026-05-25T03:42:52.958Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:53.063Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:52.958213357 +0000 UTC m=+99.131188801 (1 nodes), 1.92µs from start of poll, 169.61224ms from cluster start. Timeout at 2026-05-25 03:42:53.063217277 +0000 UTC m=+99.236192721, 105.00392ms after stability 2026-05-25T03:42:53.063Z [WARN] server-b768ac08-95e3-8dff-0777-4873bc9cc738: clearing log suffix: from=1 to=2 2026-05-25T03:42:53.063Z [WARN] server-b768ac08-95e3-8dff-0777-4873bc9cc738: skipping application of old log: index=1 2026-05-25T03:42:53.063Z [WARN] server-b768ac08-95e3-8dff-0777-4873bc9cc738: skipping application of old log: index=1 2026-05-25T03:42:53.063Z [INFO] server-011a3271-3631-9b39-5707-87ef39ef69a9: aborting pipeline replication: peer="{Voter server-b768ac08-95e3-8dff-0777-4873bc9cc738 b768ac08-95e3-8dff-0777-4873bc9cc738}" 2026-05-25T03:42:53.063Z [INFO] server-011a3271-3631-9b39-5707-87ef39ef69a9: aborting pipeline replication: peer="{Voter server-1dd8553c-897d-60f4-53b2-df2869e619fe 1dd8553c-897d-60f4-53b2-df2869e619fe}" --- PASS: TestRaft_AppendEntry (0.28s) === RUN TestRaft_VotingGrant_WhenLeaderAvailable 2026-05-25T03:42:53.064Z [INFO] server-31cf0a30-64dc-c211-a9f7-437205d2f757: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31cf0a30-64dc-c211-a9f7-437205d2f757 Address:31cf0a30-64dc-c211-a9f7-437205d2f757} {Suffrage:Voter ID:server-fd649c61-010e-1a95-7e0a-60c99b421163 Address:fd649c61-010e-1a95-7e0a-60c99b421163} {Suffrage:Voter ID:server-775a2209-d974-62a2-e98e-91289e383e05 Address:775a2209-d974-62a2-e98e-91289e383e05}]" 2026-05-25T03:42:53.064Z [INFO] server-31cf0a30-64dc-c211-a9f7-437205d2f757: entering follower state: follower="Node at 31cf0a30-64dc-c211-a9f7-437205d2f757 [Follower]" leader-address= leader-id= 2026-05-25T03:42:53.064Z [INFO] server-fd649c61-010e-1a95-7e0a-60c99b421163: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31cf0a30-64dc-c211-a9f7-437205d2f757 Address:31cf0a30-64dc-c211-a9f7-437205d2f757} {Suffrage:Voter ID:server-fd649c61-010e-1a95-7e0a-60c99b421163 Address:fd649c61-010e-1a95-7e0a-60c99b421163} {Suffrage:Voter ID:server-775a2209-d974-62a2-e98e-91289e383e05 Address:775a2209-d974-62a2-e98e-91289e383e05}]" 2026-05-25T03:42:53.064Z [INFO] server-fd649c61-010e-1a95-7e0a-60c99b421163: entering follower state: follower="Node at fd649c61-010e-1a95-7e0a-60c99b421163 [Follower]" leader-address= leader-id= 2026-05-25T03:42:53.064Z [INFO] server-775a2209-d974-62a2-e98e-91289e383e05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31cf0a30-64dc-c211-a9f7-437205d2f757 Address:31cf0a30-64dc-c211-a9f7-437205d2f757} {Suffrage:Voter ID:server-fd649c61-010e-1a95-7e0a-60c99b421163 Address:fd649c61-010e-1a95-7e0a-60c99b421163} {Suffrage:Voter ID:server-775a2209-d974-62a2-e98e-91289e383e05 Address:775a2209-d974-62a2-e98e-91289e383e05}]" 2026-05-25T03:42:53.064Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:53.065Z [INFO] server-775a2209-d974-62a2-e98e-91289e383e05: entering follower state: follower="Node at 775a2209-d974-62a2-e98e-91289e383e05 [Follower]" leader-address= leader-id= 2026-05-25T03:42:53.118Z [WARN] server-775a2209-d974-62a2-e98e-91289e383e05: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:53.118Z [WARN] server-fd649c61-010e-1a95-7e0a-60c99b421163: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:53.118Z [INFO] server-775a2209-d974-62a2-e98e-91289e383e05: entering candidate state: node="Node at 775a2209-d974-62a2-e98e-91289e383e05 [Candidate]" term=2 2026-05-25T03:42:53.118Z [INFO] server-fd649c61-010e-1a95-7e0a-60c99b421163: entering candidate state: node="Node at fd649c61-010e-1a95-7e0a-60c99b421163 [Candidate]" term=2 2026-05-25T03:42:53.118Z [INFO] server-fd649c61-010e-1a95-7e0a-60c99b421163: duplicate requestVote for same term: term=2 2026-05-25T03:42:53.118Z [INFO] server-31cf0a30-64dc-c211-a9f7-437205d2f757: duplicate requestVote for same term: term=2 2026-05-25T03:42:53.118Z [INFO] server-775a2209-d974-62a2-e98e-91289e383e05: duplicate requestVote for same term: term=2 2026-05-25T03:42:53.118Z [INFO] server-775a2209-d974-62a2-e98e-91289e383e05: election won: term=2 tally=2 2026-05-25T03:42:53.118Z [INFO] server-775a2209-d974-62a2-e98e-91289e383e05: entering leader state: leader="Node at 775a2209-d974-62a2-e98e-91289e383e05 [Leader]" 2026-05-25T03:42:53.118Z [INFO] server-775a2209-d974-62a2-e98e-91289e383e05: added peer, starting replication: peer=server-31cf0a30-64dc-c211-a9f7-437205d2f757 2026-05-25T03:42:53.118Z [INFO] server-775a2209-d974-62a2-e98e-91289e383e05: added peer, starting replication: peer=server-fd649c61-010e-1a95-7e0a-60c99b421163 2026-05-25T03:42:53.118Z [INFO] server-fd649c61-010e-1a95-7e0a-60c99b421163: entering follower state: follower="Node at fd649c61-010e-1a95-7e0a-60c99b421163 [Follower]" leader-address=775a2209-d974-62a2-e98e-91289e383e05 leader-id=server-775a2209-d974-62a2-e98e-91289e383e05 2026-05-25T03:42:53.118Z [INFO] server-775a2209-d974-62a2-e98e-91289e383e05: pipelining replication: peer="{Voter server-31cf0a30-64dc-c211-a9f7-437205d2f757 31cf0a30-64dc-c211-a9f7-437205d2f757}" 2026-05-25T03:42:53.118Z [INFO] server-775a2209-d974-62a2-e98e-91289e383e05: pipelining replication: peer="{Voter server-fd649c61-010e-1a95-7e0a-60c99b421163 fd649c61-010e-1a95-7e0a-60c99b421163}" 2026-05-25T03:42:53.224Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:53.119007557 +0000 UTC m=+99.291983041 (2 nodes), 53.99716ms from start of poll, 54.46796ms from cluster start. Timeout at 2026-05-25 03:42:53.224011637 +0000 UTC m=+99.396987081, 105.00404ms after stability 2026-05-25T03:42:53.224Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:53.330Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:53.224336397 +0000 UTC m=+99.397311841 (1 nodes), 2.48µs from start of poll, 159.79676ms from cluster start. Timeout at 2026-05-25 03:42:53.329357637 +0000 UTC m=+99.502333081, 105.02124ms after stability 2026-05-25T03:42:53.330Z [WARN] server-fd649c61-010e-1a95-7e0a-60c99b421163: rejecting vote request since we have a leader: from=31cf0a30-64dc-c211-a9f7-437205d2f757 leader=775a2209-d974-62a2-e98e-91289e383e05 leader-id=server-775a2209-d974-62a2-e98e-91289e383e05 2026-05-25T03:42:53.330Z [INFO] server-775a2209-d974-62a2-e98e-91289e383e05: aborting pipeline replication: peer="{Voter server-fd649c61-010e-1a95-7e0a-60c99b421163 fd649c61-010e-1a95-7e0a-60c99b421163}" 2026-05-25T03:42:53.330Z [INFO] server-775a2209-d974-62a2-e98e-91289e383e05: aborting pipeline replication: peer="{Voter server-31cf0a30-64dc-c211-a9f7-437205d2f757 31cf0a30-64dc-c211-a9f7-437205d2f757}" --- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.27s) === RUN TestRaft_ProtocolVersion_RejectRPC 2026-05-25T03:42:53.331Z [INFO] server-9ff8b120-9009-27a5-48c9-ad0cf5c1781c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ff8b120-9009-27a5-48c9-ad0cf5c1781c Address:9ff8b120-9009-27a5-48c9-ad0cf5c1781c} {Suffrage:Voter ID:server-8d28039d-76a6-7e88-7dac-1efe9a2a20f7 Address:8d28039d-76a6-7e88-7dac-1efe9a2a20f7} {Suffrage:Voter ID:server-d6c8509e-5225-5d80-1a44-b01919bddfb0 Address:d6c8509e-5225-5d80-1a44-b01919bddfb0}]" 2026-05-25T03:42:53.331Z [INFO] server-8d28039d-76a6-7e88-7dac-1efe9a2a20f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ff8b120-9009-27a5-48c9-ad0cf5c1781c Address:9ff8b120-9009-27a5-48c9-ad0cf5c1781c} {Suffrage:Voter ID:server-8d28039d-76a6-7e88-7dac-1efe9a2a20f7 Address:8d28039d-76a6-7e88-7dac-1efe9a2a20f7} {Suffrage:Voter ID:server-d6c8509e-5225-5d80-1a44-b01919bddfb0 Address:d6c8509e-5225-5d80-1a44-b01919bddfb0}]" 2026-05-25T03:42:53.331Z [INFO] server-9ff8b120-9009-27a5-48c9-ad0cf5c1781c: entering follower state: follower="Node at 9ff8b120-9009-27a5-48c9-ad0cf5c1781c [Follower]" leader-address= leader-id= 2026-05-25T03:42:53.331Z [INFO] server-8d28039d-76a6-7e88-7dac-1efe9a2a20f7: entering follower state: follower="Node at 8d28039d-76a6-7e88-7dac-1efe9a2a20f7 [Follower]" leader-address= leader-id= 2026-05-25T03:42:53.331Z [INFO] server-d6c8509e-5225-5d80-1a44-b01919bddfb0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ff8b120-9009-27a5-48c9-ad0cf5c1781c Address:9ff8b120-9009-27a5-48c9-ad0cf5c1781c} {Suffrage:Voter ID:server-8d28039d-76a6-7e88-7dac-1efe9a2a20f7 Address:8d28039d-76a6-7e88-7dac-1efe9a2a20f7} {Suffrage:Voter ID:server-d6c8509e-5225-5d80-1a44-b01919bddfb0 Address:d6c8509e-5225-5d80-1a44-b01919bddfb0}]" 2026-05-25T03:42:53.331Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:53.331Z [INFO] server-d6c8509e-5225-5d80-1a44-b01919bddfb0: entering follower state: follower="Node at d6c8509e-5225-5d80-1a44-b01919bddfb0 [Follower]" leader-address= leader-id= 2026-05-25T03:42:53.391Z [WARN] server-9ff8b120-9009-27a5-48c9-ad0cf5c1781c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:53.391Z [INFO] server-9ff8b120-9009-27a5-48c9-ad0cf5c1781c: entering candidate state: node="Node at 9ff8b120-9009-27a5-48c9-ad0cf5c1781c [Candidate]" term=2 2026-05-25T03:42:53.391Z [INFO] server-9ff8b120-9009-27a5-48c9-ad0cf5c1781c: election won: term=2 tally=2 2026-05-25T03:42:53.391Z [INFO] server-9ff8b120-9009-27a5-48c9-ad0cf5c1781c: entering leader state: leader="Node at 9ff8b120-9009-27a5-48c9-ad0cf5c1781c [Leader]" 2026-05-25T03:42:53.391Z [INFO] server-9ff8b120-9009-27a5-48c9-ad0cf5c1781c: added peer, starting replication: peer=server-8d28039d-76a6-7e88-7dac-1efe9a2a20f7 2026-05-25T03:42:53.391Z [INFO] server-9ff8b120-9009-27a5-48c9-ad0cf5c1781c: added peer, starting replication: peer=server-d6c8509e-5225-5d80-1a44-b01919bddfb0 2026-05-25T03:42:53.391Z [INFO] server-9ff8b120-9009-27a5-48c9-ad0cf5c1781c: pipelining replication: peer="{Voter server-d6c8509e-5225-5d80-1a44-b01919bddfb0 d6c8509e-5225-5d80-1a44-b01919bddfb0}" 2026-05-25T03:42:53.391Z [INFO] server-9ff8b120-9009-27a5-48c9-ad0cf5c1781c: pipelining replication: peer="{Voter server-8d28039d-76a6-7e88-7dac-1efe9a2a20f7 8d28039d-76a6-7e88-7dac-1efe9a2a20f7}" 2026-05-25T03:42:53.497Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:53.391921237 +0000 UTC m=+99.564896681 (2 nodes), 60.08052ms from start of poll, 60.52008ms from cluster start. Timeout at 2026-05-25 03:42:53.496922757 +0000 UTC m=+99.669898201, 105.00152ms after stability 2026-05-25T03:42:53.497Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:53.603Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:53.497809117 +0000 UTC m=+99.670784561 (1 nodes), 1.28µs from start of poll, 166.40796ms from cluster start. Timeout at 2026-05-25 03:42:53.602830277 +0000 UTC m=+99.775805721, 105.02116ms after stability 2026-05-25T03:42:53.603Z [INFO] server-9ff8b120-9009-27a5-48c9-ad0cf5c1781c: aborting pipeline replication: peer="{Voter server-d6c8509e-5225-5d80-1a44-b01919bddfb0 d6c8509e-5225-5d80-1a44-b01919bddfb0}" 2026-05-25T03:42:53.603Z [INFO] server-9ff8b120-9009-27a5-48c9-ad0cf5c1781c: aborting pipeline replication: peer="{Voter server-8d28039d-76a6-7e88-7dac-1efe9a2a20f7 8d28039d-76a6-7e88-7dac-1efe9a2a20f7}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s) === RUN TestRaft_ProtocolVersion_Upgrade_1_2 raft_test.go:2151: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s) === RUN TestRaft_ProtocolVersion_Upgrade_2_3 raft_test.go:2196: Skipping unreliable test --- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s) === RUN TestRaft_LeaderID_Propagated 2026-05-25T03:42:53.604Z [INFO] server-8e59f600-a1da-5df0-272a-ec342750f4d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e59f600-a1da-5df0-272a-ec342750f4d2 Address:8e59f600-a1da-5df0-272a-ec342750f4d2} {Suffrage:Voter ID:server-ed7a243f-96cf-5c62-2c99-2d3d73a0d803 Address:ed7a243f-96cf-5c62-2c99-2d3d73a0d803} {Suffrage:Voter ID:server-10024e2d-7bc7-bf64-6ba1-2f765f76ef3a Address:10024e2d-7bc7-bf64-6ba1-2f765f76ef3a}]" 2026-05-25T03:42:53.604Z [INFO] server-8e59f600-a1da-5df0-272a-ec342750f4d2: entering follower state: follower="Node at 8e59f600-a1da-5df0-272a-ec342750f4d2 [Follower]" leader-address= leader-id= 2026-05-25T03:42:53.604Z [INFO] server-ed7a243f-96cf-5c62-2c99-2d3d73a0d803: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e59f600-a1da-5df0-272a-ec342750f4d2 Address:8e59f600-a1da-5df0-272a-ec342750f4d2} {Suffrage:Voter ID:server-ed7a243f-96cf-5c62-2c99-2d3d73a0d803 Address:ed7a243f-96cf-5c62-2c99-2d3d73a0d803} {Suffrage:Voter ID:server-10024e2d-7bc7-bf64-6ba1-2f765f76ef3a Address:10024e2d-7bc7-bf64-6ba1-2f765f76ef3a}]" 2026-05-25T03:42:53.604Z [INFO] server-ed7a243f-96cf-5c62-2c99-2d3d73a0d803: entering follower state: follower="Node at ed7a243f-96cf-5c62-2c99-2d3d73a0d803 [Follower]" leader-address= leader-id= 2026-05-25T03:42:53.604Z [INFO] server-10024e2d-7bc7-bf64-6ba1-2f765f76ef3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e59f600-a1da-5df0-272a-ec342750f4d2 Address:8e59f600-a1da-5df0-272a-ec342750f4d2} {Suffrage:Voter ID:server-ed7a243f-96cf-5c62-2c99-2d3d73a0d803 Address:ed7a243f-96cf-5c62-2c99-2d3d73a0d803} {Suffrage:Voter ID:server-10024e2d-7bc7-bf64-6ba1-2f765f76ef3a Address:10024e2d-7bc7-bf64-6ba1-2f765f76ef3a}]" 2026-05-25T03:42:53.605Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:53.605Z [INFO] server-10024e2d-7bc7-bf64-6ba1-2f765f76ef3a: entering follower state: follower="Node at 10024e2d-7bc7-bf64-6ba1-2f765f76ef3a [Follower]" leader-address= leader-id= 2026-05-25T03:42:53.655Z [WARN] server-8e59f600-a1da-5df0-272a-ec342750f4d2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:53.655Z [INFO] server-8e59f600-a1da-5df0-272a-ec342750f4d2: entering candidate state: node="Node at 8e59f600-a1da-5df0-272a-ec342750f4d2 [Candidate]" term=2 2026-05-25T03:42:53.655Z [INFO] server-8e59f600-a1da-5df0-272a-ec342750f4d2: election won: term=2 tally=2 2026-05-25T03:42:53.655Z [INFO] server-8e59f600-a1da-5df0-272a-ec342750f4d2: entering leader state: leader="Node at 8e59f600-a1da-5df0-272a-ec342750f4d2 [Leader]" 2026-05-25T03:42:53.655Z [INFO] server-8e59f600-a1da-5df0-272a-ec342750f4d2: added peer, starting replication: peer=server-ed7a243f-96cf-5c62-2c99-2d3d73a0d803 2026-05-25T03:42:53.655Z [INFO] server-8e59f600-a1da-5df0-272a-ec342750f4d2: added peer, starting replication: peer=server-10024e2d-7bc7-bf64-6ba1-2f765f76ef3a 2026-05-25T03:42:53.655Z [INFO] server-8e59f600-a1da-5df0-272a-ec342750f4d2: pipelining replication: peer="{Voter server-10024e2d-7bc7-bf64-6ba1-2f765f76ef3a 10024e2d-7bc7-bf64-6ba1-2f765f76ef3a}" 2026-05-25T03:42:53.655Z [INFO] server-8e59f600-a1da-5df0-272a-ec342750f4d2: pipelining replication: peer="{Voter server-ed7a243f-96cf-5c62-2c99-2d3d73a0d803 ed7a243f-96cf-5c62-2c99-2d3d73a0d803}" 2026-05-25T03:42:53.761Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:53.655649357 +0000 UTC m=+99.828624801 (1 nodes), 50.59372ms from start of poll, 51.21332ms from cluster start. Timeout at 2026-05-25 03:42:53.760651517 +0000 UTC m=+99.933626961, 105.00216ms after stability 2026-05-25T03:42:53.761Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:53.867Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:53.761117037 +0000 UTC m=+99.934092481 (1 nodes), 2.56µs from start of poll, 156.681ms from cluster start. Timeout at 2026-05-25 03:42:53.866140997 +0000 UTC m=+100.039116441, 105.02396ms after stability 2026-05-25T03:42:53.867Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:53.972Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:53.867496957 +0000 UTC m=+100.040472401 (1 nodes), 920ns from start of poll, 263.06092ms from cluster start. Timeout at 2026-05-25 03:42:53.972542877 +0000 UTC m=+100.145518321, 105.04592ms after stability 2026-05-25T03:42:53.973Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:54.079Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:53.973115517 +0000 UTC m=+100.146090961 (1 nodes), 1.52µs from start of poll, 368.67948ms from cluster start. Timeout at 2026-05-25 03:42:54.078142997 +0000 UTC m=+100.251118441, 105.02748ms after stability 2026-05-25T03:42:54.079Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:54.184Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:54.079514797 +0000 UTC m=+100.252490241 (1 nodes), 1.92µs from start of poll, 475.07876ms from cluster start. Timeout at 2026-05-25 03:42:54.184551117 +0000 UTC m=+100.357526561, 105.03632ms after stability 2026-05-25T03:42:54.185Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:54.290Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:54.185091637 +0000 UTC m=+100.358067081 (1 nodes), 1.04µs from start of poll, 580.6556ms from cluster start. Timeout at 2026-05-25 03:42:54.290133557 +0000 UTC m=+100.463109001, 105.04192ms after stability 2026-05-25T03:42:54.396Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:54.502Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:54.396243557 +0000 UTC m=+100.569219041 (1 nodes), 2.12µs from start of poll, 791.80756ms from cluster start. Timeout at 2026-05-25 03:42:54.501272837 +0000 UTC m=+100.674248281, 105.02924ms after stability 2026-05-25T03:42:54.502Z [INFO] server-8e59f600-a1da-5df0-272a-ec342750f4d2: aborting pipeline replication: peer="{Voter server-10024e2d-7bc7-bf64-6ba1-2f765f76ef3a 10024e2d-7bc7-bf64-6ba1-2f765f76ef3a}" 2026-05-25T03:42:54.502Z [INFO] server-8e59f600-a1da-5df0-272a-ec342750f4d2: aborting pipeline replication: peer="{Voter server-ed7a243f-96cf-5c62-2c99-2d3d73a0d803 ed7a243f-96cf-5c62-2c99-2d3d73a0d803}" --- PASS: TestRaft_LeaderID_Propagated (0.90s) === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer 2026-05-25T03:42:54.503Z [INFO] server-1a564fe4-5d74-fa25-ce5f-c017b76cda86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a564fe4-5d74-fa25-ce5f-c017b76cda86 Address:1a564fe4-5d74-fa25-ce5f-c017b76cda86} {Suffrage:Voter ID:server-ded59c14-2728-326d-47be-718551d3bf02 Address:ded59c14-2728-326d-47be-718551d3bf02} {Suffrage:Voter ID:server-fee999a0-3cda-e7c4-65e3-567c0161d2aa Address:fee999a0-3cda-e7c4-65e3-567c0161d2aa}]" 2026-05-25T03:42:54.503Z [INFO] server-1a564fe4-5d74-fa25-ce5f-c017b76cda86: entering follower state: follower="Node at 1a564fe4-5d74-fa25-ce5f-c017b76cda86 [Follower]" leader-address= leader-id= 2026-05-25T03:42:54.503Z [INFO] server-ded59c14-2728-326d-47be-718551d3bf02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a564fe4-5d74-fa25-ce5f-c017b76cda86 Address:1a564fe4-5d74-fa25-ce5f-c017b76cda86} {Suffrage:Voter ID:server-ded59c14-2728-326d-47be-718551d3bf02 Address:ded59c14-2728-326d-47be-718551d3bf02} {Suffrage:Voter ID:server-fee999a0-3cda-e7c4-65e3-567c0161d2aa Address:fee999a0-3cda-e7c4-65e3-567c0161d2aa}]" 2026-05-25T03:42:54.503Z [INFO] server-ded59c14-2728-326d-47be-718551d3bf02: entering follower state: follower="Node at ded59c14-2728-326d-47be-718551d3bf02 [Follower]" leader-address= leader-id= 2026-05-25T03:42:54.503Z [INFO] server-fee999a0-3cda-e7c4-65e3-567c0161d2aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a564fe4-5d74-fa25-ce5f-c017b76cda86 Address:1a564fe4-5d74-fa25-ce5f-c017b76cda86} {Suffrage:Voter ID:server-ded59c14-2728-326d-47be-718551d3bf02 Address:ded59c14-2728-326d-47be-718551d3bf02} {Suffrage:Voter ID:server-fee999a0-3cda-e7c4-65e3-567c0161d2aa Address:fee999a0-3cda-e7c4-65e3-567c0161d2aa}]" 2026-05-25T03:42:54.503Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:54.503Z [INFO] server-fee999a0-3cda-e7c4-65e3-567c0161d2aa: entering follower state: follower="Node at fee999a0-3cda-e7c4-65e3-567c0161d2aa [Follower]" leader-address= leader-id= 2026-05-25T03:42:54.562Z [WARN] server-ded59c14-2728-326d-47be-718551d3bf02: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:54.562Z [INFO] server-ded59c14-2728-326d-47be-718551d3bf02: entering candidate state: node="Node at ded59c14-2728-326d-47be-718551d3bf02 [Candidate]" term=2 2026-05-25T03:42:54.562Z [INFO] server-ded59c14-2728-326d-47be-718551d3bf02: election won: term=2 tally=2 2026-05-25T03:42:54.562Z [INFO] server-ded59c14-2728-326d-47be-718551d3bf02: entering leader state: leader="Node at ded59c14-2728-326d-47be-718551d3bf02 [Leader]" 2026-05-25T03:42:54.562Z [INFO] server-ded59c14-2728-326d-47be-718551d3bf02: added peer, starting replication: peer=server-1a564fe4-5d74-fa25-ce5f-c017b76cda86 2026-05-25T03:42:54.563Z [INFO] server-ded59c14-2728-326d-47be-718551d3bf02: added peer, starting replication: peer=server-fee999a0-3cda-e7c4-65e3-567c0161d2aa 2026-05-25T03:42:54.563Z [INFO] server-ded59c14-2728-326d-47be-718551d3bf02: pipelining replication: peer="{Voter server-1a564fe4-5d74-fa25-ce5f-c017b76cda86 1a564fe4-5d74-fa25-ce5f-c017b76cda86}" 2026-05-25T03:42:54.563Z [INFO] server-ded59c14-2728-326d-47be-718551d3bf02: pipelining replication: peer="{Voter server-fee999a0-3cda-e7c4-65e3-567c0161d2aa fee999a0-3cda-e7c4-65e3-567c0161d2aa}" 2026-05-25T03:42:54.668Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:54.563010917 +0000 UTC m=+100.735986361 (1 nodes), 59.17732ms from start of poll, 59.72128ms from cluster start. Timeout at 2026-05-25 03:42:54.668018557 +0000 UTC m=+100.840994001, 105.00764ms after stability 2026-05-25T03:42:54.668Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:54.773Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:54.668166477 +0000 UTC m=+100.841141961 (1 nodes), 1µs from start of poll, 164.87688ms from cluster start. Timeout at 2026-05-25 03:42:54.773195757 +0000 UTC m=+100.946171201, 105.02924ms after stability 2026-05-25T03:42:54.774Z [INFO] server-1a564fe4-5d74-fa25-ce5f-c017b76cda86: entering candidate state: node="Node at 1a564fe4-5d74-fa25-ce5f-c017b76cda86 [Candidate]" term=3 2026-05-25T03:42:54.774Z [INFO] server-1a564fe4-5d74-fa25-ce5f-c017b76cda86: election won: term=3 tally=2 2026-05-25T03:42:54.774Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:54.774Z [INFO] server-ded59c14-2728-326d-47be-718551d3bf02: entering follower state: follower="Node at ded59c14-2728-326d-47be-718551d3bf02 [Follower]" leader-address= leader-id= 2026-05-25T03:42:54.774Z [INFO] server-ded59c14-2728-326d-47be-718551d3bf02: aborting pipeline replication: peer="{Voter server-1a564fe4-5d74-fa25-ce5f-c017b76cda86 1a564fe4-5d74-fa25-ce5f-c017b76cda86}" 2026-05-25T03:42:54.774Z [INFO] server-1a564fe4-5d74-fa25-ce5f-c017b76cda86: entering leader state: leader="Node at 1a564fe4-5d74-fa25-ce5f-c017b76cda86 [Leader]" 2026-05-25T03:42:54.774Z [INFO] server-ded59c14-2728-326d-47be-718551d3bf02: aborting pipeline replication: peer="{Voter server-fee999a0-3cda-e7c4-65e3-567c0161d2aa fee999a0-3cda-e7c4-65e3-567c0161d2aa}" 2026-05-25T03:42:54.774Z [INFO] server-1a564fe4-5d74-fa25-ce5f-c017b76cda86: added peer, starting replication: peer=server-ded59c14-2728-326d-47be-718551d3bf02 2026-05-25T03:42:54.774Z [INFO] server-1a564fe4-5d74-fa25-ce5f-c017b76cda86: added peer, starting replication: peer=server-fee999a0-3cda-e7c4-65e3-567c0161d2aa 2026-05-25T03:42:54.774Z [INFO] server-1a564fe4-5d74-fa25-ce5f-c017b76cda86: pipelining replication: peer="{Voter server-fee999a0-3cda-e7c4-65e3-567c0161d2aa fee999a0-3cda-e7c4-65e3-567c0161d2aa}" 2026-05-25T03:42:54.774Z [INFO] server-1a564fe4-5d74-fa25-ce5f-c017b76cda86: pipelining replication: peer="{Voter server-ded59c14-2728-326d-47be-718551d3bf02 ded59c14-2728-326d-47be-718551d3bf02}" 2026-05-25T03:42:54.879Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:54.774246357 +0000 UTC m=+100.947221801 (1 nodes), 38.68µs from start of poll, 270.95672ms from cluster start. Timeout at 2026-05-25 03:42:54.879286957 +0000 UTC m=+101.052262401, 105.0406ms after stability 2026-05-25T03:42:54.879Z [INFO] server-1a564fe4-5d74-fa25-ce5f-c017b76cda86: aborting pipeline replication: peer="{Voter server-fee999a0-3cda-e7c4-65e3-567c0161d2aa fee999a0-3cda-e7c4-65e3-567c0161d2aa}" 2026-05-25T03:42:54.879Z [INFO] server-1a564fe4-5d74-fa25-ce5f-c017b76cda86: aborting pipeline replication: peer="{Voter server-ded59c14-2728-326d-47be-718551d3bf02 ded59c14-2728-326d-47be-718551d3bf02}" 2026-05-25T03:42:55.380Z [ERROR] server-1a564fe4-5d74-fa25-ce5f-c017b76cda86: failed to heartbeat to: peer=ded59c14-2728-326d-47be-718551d3bf02 backoff time=10ms error="command timed out" --- PASS: TestRaft_LeadershipTransfer (0.88s) === RUN TestRaft_LeadershipTransferWithOneNode 2026-05-25T03:42:55.381Z [INFO] server-b42c5b93-54ff-6b25-2efa-bce12c7b0728: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b42c5b93-54ff-6b25-2efa-bce12c7b0728 Address:b42c5b93-54ff-6b25-2efa-bce12c7b0728}]" 2026-05-25T03:42:55.381Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:55.381Z [INFO] server-b42c5b93-54ff-6b25-2efa-bce12c7b0728: entering follower state: follower="Node at b42c5b93-54ff-6b25-2efa-bce12c7b0728 [Follower]" leader-address= leader-id= 2026-05-25T03:42:55.458Z [WARN] server-b42c5b93-54ff-6b25-2efa-bce12c7b0728: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:55.458Z [INFO] server-b42c5b93-54ff-6b25-2efa-bce12c7b0728: entering candidate state: node="Node at b42c5b93-54ff-6b25-2efa-bce12c7b0728 [Candidate]" term=2 2026-05-25T03:42:55.459Z [INFO] server-b42c5b93-54ff-6b25-2efa-bce12c7b0728: election won: term=2 tally=1 2026-05-25T03:42:55.459Z [INFO] server-b42c5b93-54ff-6b25-2efa-bce12c7b0728: entering leader state: leader="Node at b42c5b93-54ff-6b25-2efa-bce12c7b0728 [Leader]" 2026-05-25T03:42:55.564Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:55.459089997 +0000 UTC m=+101.632065441 (1 nodes), 77.3022ms from start of poll, 77.5034ms from cluster start. Timeout at 2026-05-25 03:42:55.564093757 +0000 UTC m=+101.737069201, 105.00376ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.18s) === RUN TestRaft_LeadershipTransferWithSevenNodes 2026-05-25T03:42:55.565Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e8892cc5-54ad-0010-8f46-a14f7d72614f Address:e8892cc5-54ad-0010-8f46-a14f7d72614f} {Suffrage:Voter ID:server-6ee13fef-1721-0acb-d382-856e9606a908 Address:6ee13fef-1721-0acb-d382-856e9606a908} {Suffrage:Voter ID:server-9b890798-bb67-ccfd-751e-a3ce931eb45d Address:9b890798-bb67-ccfd-751e-a3ce931eb45d} {Suffrage:Voter ID:server-58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5 Address:58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5} {Suffrage:Voter ID:server-c91df5f0-5efd-d9b4-72d6-7628bce21e15 Address:c91df5f0-5efd-d9b4-72d6-7628bce21e15} {Suffrage:Voter ID:server-41aa47db-a3d8-1402-5d48-7dfd91af1761 Address:41aa47db-a3d8-1402-5d48-7dfd91af1761} {Suffrage:Voter ID:server-59386b69-cbde-5ab1-8c92-c96b9ecd5caf Address:59386b69-cbde-5ab1-8c92-c96b9ecd5caf}]" 2026-05-25T03:42:55.565Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: entering follower state: follower="Node at e8892cc5-54ad-0010-8f46-a14f7d72614f [Follower]" leader-address= leader-id= 2026-05-25T03:42:55.565Z [INFO] server-6ee13fef-1721-0acb-d382-856e9606a908: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e8892cc5-54ad-0010-8f46-a14f7d72614f Address:e8892cc5-54ad-0010-8f46-a14f7d72614f} {Suffrage:Voter ID:server-6ee13fef-1721-0acb-d382-856e9606a908 Address:6ee13fef-1721-0acb-d382-856e9606a908} {Suffrage:Voter ID:server-9b890798-bb67-ccfd-751e-a3ce931eb45d Address:9b890798-bb67-ccfd-751e-a3ce931eb45d} {Suffrage:Voter ID:server-58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5 Address:58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5} {Suffrage:Voter ID:server-c91df5f0-5efd-d9b4-72d6-7628bce21e15 Address:c91df5f0-5efd-d9b4-72d6-7628bce21e15} {Suffrage:Voter ID:server-41aa47db-a3d8-1402-5d48-7dfd91af1761 Address:41aa47db-a3d8-1402-5d48-7dfd91af1761} {Suffrage:Voter ID:server-59386b69-cbde-5ab1-8c92-c96b9ecd5caf Address:59386b69-cbde-5ab1-8c92-c96b9ecd5caf}]" 2026-05-25T03:42:55.565Z [INFO] server-6ee13fef-1721-0acb-d382-856e9606a908: entering follower state: follower="Node at 6ee13fef-1721-0acb-d382-856e9606a908 [Follower]" leader-address= leader-id= 2026-05-25T03:42:55.565Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e8892cc5-54ad-0010-8f46-a14f7d72614f Address:e8892cc5-54ad-0010-8f46-a14f7d72614f} {Suffrage:Voter ID:server-6ee13fef-1721-0acb-d382-856e9606a908 Address:6ee13fef-1721-0acb-d382-856e9606a908} {Suffrage:Voter ID:server-9b890798-bb67-ccfd-751e-a3ce931eb45d Address:9b890798-bb67-ccfd-751e-a3ce931eb45d} {Suffrage:Voter ID:server-58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5 Address:58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5} {Suffrage:Voter ID:server-c91df5f0-5efd-d9b4-72d6-7628bce21e15 Address:c91df5f0-5efd-d9b4-72d6-7628bce21e15} {Suffrage:Voter ID:server-41aa47db-a3d8-1402-5d48-7dfd91af1761 Address:41aa47db-a3d8-1402-5d48-7dfd91af1761} {Suffrage:Voter ID:server-59386b69-cbde-5ab1-8c92-c96b9ecd5caf Address:59386b69-cbde-5ab1-8c92-c96b9ecd5caf}]" 2026-05-25T03:42:55.565Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: entering follower state: follower="Node at 9b890798-bb67-ccfd-751e-a3ce931eb45d [Follower]" leader-address= leader-id= 2026-05-25T03:42:55.566Z [INFO] server-58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e8892cc5-54ad-0010-8f46-a14f7d72614f Address:e8892cc5-54ad-0010-8f46-a14f7d72614f} {Suffrage:Voter ID:server-6ee13fef-1721-0acb-d382-856e9606a908 Address:6ee13fef-1721-0acb-d382-856e9606a908} {Suffrage:Voter ID:server-9b890798-bb67-ccfd-751e-a3ce931eb45d Address:9b890798-bb67-ccfd-751e-a3ce931eb45d} {Suffrage:Voter ID:server-58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5 Address:58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5} {Suffrage:Voter ID:server-c91df5f0-5efd-d9b4-72d6-7628bce21e15 Address:c91df5f0-5efd-d9b4-72d6-7628bce21e15} {Suffrage:Voter ID:server-41aa47db-a3d8-1402-5d48-7dfd91af1761 Address:41aa47db-a3d8-1402-5d48-7dfd91af1761} {Suffrage:Voter ID:server-59386b69-cbde-5ab1-8c92-c96b9ecd5caf Address:59386b69-cbde-5ab1-8c92-c96b9ecd5caf}]" 2026-05-25T03:42:55.566Z [INFO] server-58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5: entering follower state: follower="Node at 58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5 [Follower]" leader-address= leader-id= 2026-05-25T03:42:55.566Z [INFO] server-c91df5f0-5efd-d9b4-72d6-7628bce21e15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e8892cc5-54ad-0010-8f46-a14f7d72614f Address:e8892cc5-54ad-0010-8f46-a14f7d72614f} {Suffrage:Voter ID:server-6ee13fef-1721-0acb-d382-856e9606a908 Address:6ee13fef-1721-0acb-d382-856e9606a908} {Suffrage:Voter ID:server-9b890798-bb67-ccfd-751e-a3ce931eb45d Address:9b890798-bb67-ccfd-751e-a3ce931eb45d} {Suffrage:Voter ID:server-58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5 Address:58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5} {Suffrage:Voter ID:server-c91df5f0-5efd-d9b4-72d6-7628bce21e15 Address:c91df5f0-5efd-d9b4-72d6-7628bce21e15} {Suffrage:Voter ID:server-41aa47db-a3d8-1402-5d48-7dfd91af1761 Address:41aa47db-a3d8-1402-5d48-7dfd91af1761} {Suffrage:Voter ID:server-59386b69-cbde-5ab1-8c92-c96b9ecd5caf Address:59386b69-cbde-5ab1-8c92-c96b9ecd5caf}]" 2026-05-25T03:42:55.566Z [INFO] server-c91df5f0-5efd-d9b4-72d6-7628bce21e15: entering follower state: follower="Node at c91df5f0-5efd-d9b4-72d6-7628bce21e15 [Follower]" leader-address= leader-id= 2026-05-25T03:42:55.566Z [INFO] server-41aa47db-a3d8-1402-5d48-7dfd91af1761: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e8892cc5-54ad-0010-8f46-a14f7d72614f Address:e8892cc5-54ad-0010-8f46-a14f7d72614f} {Suffrage:Voter ID:server-6ee13fef-1721-0acb-d382-856e9606a908 Address:6ee13fef-1721-0acb-d382-856e9606a908} {Suffrage:Voter ID:server-9b890798-bb67-ccfd-751e-a3ce931eb45d Address:9b890798-bb67-ccfd-751e-a3ce931eb45d} {Suffrage:Voter ID:server-58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5 Address:58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5} {Suffrage:Voter ID:server-c91df5f0-5efd-d9b4-72d6-7628bce21e15 Address:c91df5f0-5efd-d9b4-72d6-7628bce21e15} {Suffrage:Voter ID:server-41aa47db-a3d8-1402-5d48-7dfd91af1761 Address:41aa47db-a3d8-1402-5d48-7dfd91af1761} {Suffrage:Voter ID:server-59386b69-cbde-5ab1-8c92-c96b9ecd5caf Address:59386b69-cbde-5ab1-8c92-c96b9ecd5caf}]" 2026-05-25T03:42:55.566Z [INFO] server-59386b69-cbde-5ab1-8c92-c96b9ecd5caf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e8892cc5-54ad-0010-8f46-a14f7d72614f Address:e8892cc5-54ad-0010-8f46-a14f7d72614f} {Suffrage:Voter ID:server-6ee13fef-1721-0acb-d382-856e9606a908 Address:6ee13fef-1721-0acb-d382-856e9606a908} {Suffrage:Voter ID:server-9b890798-bb67-ccfd-751e-a3ce931eb45d Address:9b890798-bb67-ccfd-751e-a3ce931eb45d} {Suffrage:Voter ID:server-58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5 Address:58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5} {Suffrage:Voter ID:server-c91df5f0-5efd-d9b4-72d6-7628bce21e15 Address:c91df5f0-5efd-d9b4-72d6-7628bce21e15} {Suffrage:Voter ID:server-41aa47db-a3d8-1402-5d48-7dfd91af1761 Address:41aa47db-a3d8-1402-5d48-7dfd91af1761} {Suffrage:Voter ID:server-59386b69-cbde-5ab1-8c92-c96b9ecd5caf Address:59386b69-cbde-5ab1-8c92-c96b9ecd5caf}]" 2026-05-25T03:42:55.566Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:55.567Z [INFO] server-41aa47db-a3d8-1402-5d48-7dfd91af1761: entering follower state: follower="Node at 41aa47db-a3d8-1402-5d48-7dfd91af1761 [Follower]" leader-address= leader-id= 2026-05-25T03:42:55.567Z [INFO] server-59386b69-cbde-5ab1-8c92-c96b9ecd5caf: entering follower state: follower="Node at 59386b69-cbde-5ab1-8c92-c96b9ecd5caf [Follower]" leader-address= leader-id= 2026-05-25T03:42:55.621Z [WARN] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:55.621Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: entering candidate state: node="Node at 9b890798-bb67-ccfd-751e-a3ce931eb45d [Candidate]" term=2 2026-05-25T03:42:55.622Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: election won: term=2 tally=4 2026-05-25T03:42:55.622Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: entering leader state: leader="Node at 9b890798-bb67-ccfd-751e-a3ce931eb45d [Leader]" 2026-05-25T03:42:55.622Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: added peer, starting replication: peer=server-e8892cc5-54ad-0010-8f46-a14f7d72614f 2026-05-25T03:42:55.622Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: added peer, starting replication: peer=server-6ee13fef-1721-0acb-d382-856e9606a908 2026-05-25T03:42:55.622Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: added peer, starting replication: peer=server-58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5 2026-05-25T03:42:55.622Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: added peer, starting replication: peer=server-c91df5f0-5efd-d9b4-72d6-7628bce21e15 2026-05-25T03:42:55.622Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: added peer, starting replication: peer=server-41aa47db-a3d8-1402-5d48-7dfd91af1761 2026-05-25T03:42:55.622Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: added peer, starting replication: peer=server-59386b69-cbde-5ab1-8c92-c96b9ecd5caf 2026-05-25T03:42:55.622Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: pipelining replication: peer="{Voter server-e8892cc5-54ad-0010-8f46-a14f7d72614f e8892cc5-54ad-0010-8f46-a14f7d72614f}" 2026-05-25T03:42:55.622Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: pipelining replication: peer="{Voter server-6ee13fef-1721-0acb-d382-856e9606a908 6ee13fef-1721-0acb-d382-856e9606a908}" 2026-05-25T03:42:55.622Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: pipelining replication: peer="{Voter server-59386b69-cbde-5ab1-8c92-c96b9ecd5caf 59386b69-cbde-5ab1-8c92-c96b9ecd5caf}" 2026-05-25T03:42:55.622Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: pipelining replication: peer="{Voter server-41aa47db-a3d8-1402-5d48-7dfd91af1761 41aa47db-a3d8-1402-5d48-7dfd91af1761}" 2026-05-25T03:42:55.622Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: pipelining replication: peer="{Voter server-58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5 58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5}" 2026-05-25T03:42:55.622Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: pipelining replication: peer="{Voter server-c91df5f0-5efd-d9b4-72d6-7628bce21e15 c91df5f0-5efd-d9b4-72d6-7628bce21e15}" 2026-05-25T03:42:55.727Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:55.622225957 +0000 UTC m=+101.795201401 (1 nodes), 55.2288ms from start of poll, 57.07924ms from cluster start. Timeout at 2026-05-25 03:42:55.727229237 +0000 UTC m=+101.900204681, 105.00328ms after stability 2026-05-25T03:42:55.727Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:55.833Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:55.727332917 +0000 UTC m=+101.900308361 (6 nodes), 3.96µs from start of poll, 162.1862ms from cluster start. Timeout at 2026-05-25 03:42:55.832353877 +0000 UTC m=+102.005329321, 105.02096ms after stability 2026-05-25T03:42:55.833Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:55.939Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:55.833298077 +0000 UTC m=+102.006273521 (1 nodes), 2.84µs from start of poll, 268.15136ms from cluster start. Timeout at 2026-05-25 03:42:55.938342157 +0000 UTC m=+102.111317601, 105.04408ms after stability 2026-05-25T03:42:55.939Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: entering candidate state: node="Node at e8892cc5-54ad-0010-8f46-a14f7d72614f [Candidate]" term=3 2026-05-25T03:42:55.939Z [ERROR] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: peer has newer term, stopping replication: peer="{Voter server-e8892cc5-54ad-0010-8f46-a14f7d72614f e8892cc5-54ad-0010-8f46-a14f7d72614f}" 2026-05-25T03:42:55.939Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: aborting pipeline replication: peer="{Voter server-e8892cc5-54ad-0010-8f46-a14f7d72614f e8892cc5-54ad-0010-8f46-a14f7d72614f}" 2026-05-25T03:42:55.939Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:55.939Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: entering follower state: follower="Node at 9b890798-bb67-ccfd-751e-a3ce931eb45d [Follower]" leader-address= leader-id= 2026-05-25T03:42:55.939Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: election won: term=3 tally=4 2026-05-25T03:42:55.939Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: entering leader state: leader="Node at e8892cc5-54ad-0010-8f46-a14f7d72614f [Leader]" 2026-05-25T03:42:55.939Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: aborting pipeline replication: peer="{Voter server-58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5 58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5}" 2026-05-25T03:42:55.939Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: added peer, starting replication: peer=server-6ee13fef-1721-0acb-d382-856e9606a908 2026-05-25T03:42:55.939Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: added peer, starting replication: peer=server-9b890798-bb67-ccfd-751e-a3ce931eb45d 2026-05-25T03:42:55.939Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: aborting pipeline replication: peer="{Voter server-41aa47db-a3d8-1402-5d48-7dfd91af1761 41aa47db-a3d8-1402-5d48-7dfd91af1761}" 2026-05-25T03:42:55.939Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: added peer, starting replication: peer=server-58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5 2026-05-25T03:42:55.939Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: added peer, starting replication: peer=server-c91df5f0-5efd-d9b4-72d6-7628bce21e15 2026-05-25T03:42:55.939Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: aborting pipeline replication: peer="{Voter server-c91df5f0-5efd-d9b4-72d6-7628bce21e15 c91df5f0-5efd-d9b4-72d6-7628bce21e15}" 2026-05-25T03:42:55.939Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: added peer, starting replication: peer=server-41aa47db-a3d8-1402-5d48-7dfd91af1761 2026-05-25T03:42:55.939Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: added peer, starting replication: peer=server-59386b69-cbde-5ab1-8c92-c96b9ecd5caf 2026-05-25T03:42:55.939Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: aborting pipeline replication: peer="{Voter server-6ee13fef-1721-0acb-d382-856e9606a908 6ee13fef-1721-0acb-d382-856e9606a908}" 2026-05-25T03:42:55.939Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: pipelining replication: peer="{Voter server-9b890798-bb67-ccfd-751e-a3ce931eb45d 9b890798-bb67-ccfd-751e-a3ce931eb45d}" 2026-05-25T03:42:55.939Z [INFO] server-9b890798-bb67-ccfd-751e-a3ce931eb45d: aborting pipeline replication: peer="{Voter server-59386b69-cbde-5ab1-8c92-c96b9ecd5caf 59386b69-cbde-5ab1-8c92-c96b9ecd5caf}" 2026-05-25T03:42:55.940Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: pipelining replication: peer="{Voter server-c91df5f0-5efd-d9b4-72d6-7628bce21e15 c91df5f0-5efd-d9b4-72d6-7628bce21e15}" 2026-05-25T03:42:55.940Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: pipelining replication: peer="{Voter server-59386b69-cbde-5ab1-8c92-c96b9ecd5caf 59386b69-cbde-5ab1-8c92-c96b9ecd5caf}" 2026-05-25T03:42:55.940Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: pipelining replication: peer="{Voter server-41aa47db-a3d8-1402-5d48-7dfd91af1761 41aa47db-a3d8-1402-5d48-7dfd91af1761}" 2026-05-25T03:42:55.940Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: pipelining replication: peer="{Voter server-6ee13fef-1721-0acb-d382-856e9606a908 6ee13fef-1721-0acb-d382-856e9606a908}" 2026-05-25T03:42:55.940Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: pipelining replication: peer="{Voter server-58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5 58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5}" 2026-05-25T03:42:56.045Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:55.939728317 +0000 UTC m=+102.112703761 (1 nodes), 61.96µs from start of poll, 374.5816ms from cluster start. Timeout at 2026-05-25 03:42:56.044762317 +0000 UTC m=+102.217737761, 105.034ms after stability 2026-05-25T03:42:56.045Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: aborting pipeline replication: peer="{Voter server-59386b69-cbde-5ab1-8c92-c96b9ecd5caf 59386b69-cbde-5ab1-8c92-c96b9ecd5caf}" 2026-05-25T03:42:56.045Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: aborting pipeline replication: peer="{Voter server-6ee13fef-1721-0acb-d382-856e9606a908 6ee13fef-1721-0acb-d382-856e9606a908}" 2026-05-25T03:42:56.045Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: aborting pipeline replication: peer="{Voter server-58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5 58b550cf-c5bb-f4d5-5c7c-be7389bdd6e5}" 2026-05-25T03:42:56.045Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: aborting pipeline replication: peer="{Voter server-c91df5f0-5efd-d9b4-72d6-7628bce21e15 c91df5f0-5efd-d9b4-72d6-7628bce21e15}" 2026-05-25T03:42:56.045Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: aborting pipeline replication: peer="{Voter server-9b890798-bb67-ccfd-751e-a3ce931eb45d 9b890798-bb67-ccfd-751e-a3ce931eb45d}" 2026-05-25T03:42:56.045Z [INFO] server-e8892cc5-54ad-0010-8f46-a14f7d72614f: aborting pipeline replication: peer="{Voter server-41aa47db-a3d8-1402-5d48-7dfd91af1761 41aa47db-a3d8-1402-5d48-7dfd91af1761}" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.48s) === RUN TestRaft_LeadershipTransferToInvalidID 2026-05-25T03:42:56.046Z [INFO] server-c47a4801-df80-3757-384a-93c66289e33d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c47a4801-df80-3757-384a-93c66289e33d Address:c47a4801-df80-3757-384a-93c66289e33d} {Suffrage:Voter ID:server-196fdde4-98a8-c46e-9527-5e176d8a5d88 Address:196fdde4-98a8-c46e-9527-5e176d8a5d88} {Suffrage:Voter ID:server-94f6c45d-3e11-b0fc-4cd8-8ba5439c9933 Address:94f6c45d-3e11-b0fc-4cd8-8ba5439c9933}]" 2026-05-25T03:42:56.046Z [INFO] server-196fdde4-98a8-c46e-9527-5e176d8a5d88: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c47a4801-df80-3757-384a-93c66289e33d Address:c47a4801-df80-3757-384a-93c66289e33d} {Suffrage:Voter ID:server-196fdde4-98a8-c46e-9527-5e176d8a5d88 Address:196fdde4-98a8-c46e-9527-5e176d8a5d88} {Suffrage:Voter ID:server-94f6c45d-3e11-b0fc-4cd8-8ba5439c9933 Address:94f6c45d-3e11-b0fc-4cd8-8ba5439c9933}]" 2026-05-25T03:42:56.046Z [INFO] server-c47a4801-df80-3757-384a-93c66289e33d: entering follower state: follower="Node at c47a4801-df80-3757-384a-93c66289e33d [Follower]" leader-address= leader-id= 2026-05-25T03:42:56.046Z [INFO] server-94f6c45d-3e11-b0fc-4cd8-8ba5439c9933: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c47a4801-df80-3757-384a-93c66289e33d Address:c47a4801-df80-3757-384a-93c66289e33d} {Suffrage:Voter ID:server-196fdde4-98a8-c46e-9527-5e176d8a5d88 Address:196fdde4-98a8-c46e-9527-5e176d8a5d88} {Suffrage:Voter ID:server-94f6c45d-3e11-b0fc-4cd8-8ba5439c9933 Address:94f6c45d-3e11-b0fc-4cd8-8ba5439c9933}]" 2026-05-25T03:42:56.046Z [INFO] server-196fdde4-98a8-c46e-9527-5e176d8a5d88: entering follower state: follower="Node at 196fdde4-98a8-c46e-9527-5e176d8a5d88 [Follower]" leader-address= leader-id= 2026-05-25T03:42:56.046Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:56.046Z [INFO] server-94f6c45d-3e11-b0fc-4cd8-8ba5439c9933: entering follower state: follower="Node at 94f6c45d-3e11-b0fc-4cd8-8ba5439c9933 [Follower]" leader-address= leader-id= 2026-05-25T03:42:56.117Z [WARN] server-94f6c45d-3e11-b0fc-4cd8-8ba5439c9933: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:56.117Z [INFO] server-94f6c45d-3e11-b0fc-4cd8-8ba5439c9933: entering candidate state: node="Node at 94f6c45d-3e11-b0fc-4cd8-8ba5439c9933 [Candidate]" term=2 2026-05-25T03:42:56.117Z [INFO] server-94f6c45d-3e11-b0fc-4cd8-8ba5439c9933: election won: term=2 tally=2 2026-05-25T03:42:56.117Z [INFO] server-94f6c45d-3e11-b0fc-4cd8-8ba5439c9933: entering leader state: leader="Node at 94f6c45d-3e11-b0fc-4cd8-8ba5439c9933 [Leader]" 2026-05-25T03:42:56.117Z [INFO] server-94f6c45d-3e11-b0fc-4cd8-8ba5439c9933: added peer, starting replication: peer=server-c47a4801-df80-3757-384a-93c66289e33d 2026-05-25T03:42:56.117Z [INFO] server-94f6c45d-3e11-b0fc-4cd8-8ba5439c9933: added peer, starting replication: peer=server-196fdde4-98a8-c46e-9527-5e176d8a5d88 2026-05-25T03:42:56.117Z [INFO] server-94f6c45d-3e11-b0fc-4cd8-8ba5439c9933: pipelining replication: peer="{Voter server-c47a4801-df80-3757-384a-93c66289e33d c47a4801-df80-3757-384a-93c66289e33d}" 2026-05-25T03:42:56.117Z [INFO] server-94f6c45d-3e11-b0fc-4cd8-8ba5439c9933: pipelining replication: peer="{Voter server-196fdde4-98a8-c46e-9527-5e176d8a5d88 196fdde4-98a8-c46e-9527-5e176d8a5d88}" 2026-05-25T03:42:56.222Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:56.117798597 +0000 UTC m=+102.290774041 (1 nodes), 70.82276ms from start of poll, 71.235ms from cluster start. Timeout at 2026-05-25 03:42:56.222799717 +0000 UTC m=+102.395775161, 105.00112ms after stability 2026-05-25T03:42:56.222Z [INFO] server-94f6c45d-3e11-b0fc-4cd8-8ba5439c9933: aborting pipeline replication: peer="{Voter server-196fdde4-98a8-c46e-9527-5e176d8a5d88 196fdde4-98a8-c46e-9527-5e176d8a5d88}" 2026-05-25T03:42:56.222Z [INFO] server-94f6c45d-3e11-b0fc-4cd8-8ba5439c9933: aborting pipeline replication: peer="{Voter server-c47a4801-df80-3757-384a-93c66289e33d c47a4801-df80-3757-384a-93c66289e33d}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.18s) === RUN TestRaft_LeadershipTransferToInvalidAddress 2026-05-25T03:42:56.223Z [INFO] server-d4299a91-7238-8001-f3f3-30afe293dd29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d4299a91-7238-8001-f3f3-30afe293dd29 Address:d4299a91-7238-8001-f3f3-30afe293dd29} {Suffrage:Voter ID:server-1632ae58-7f3f-78b4-9d00-48d6e68b12e4 Address:1632ae58-7f3f-78b4-9d00-48d6e68b12e4} {Suffrage:Voter ID:server-ff25c907-8e82-8b9b-8c4f-cbb68fc77857 Address:ff25c907-8e82-8b9b-8c4f-cbb68fc77857}]" 2026-05-25T03:42:56.223Z [INFO] server-1632ae58-7f3f-78b4-9d00-48d6e68b12e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d4299a91-7238-8001-f3f3-30afe293dd29 Address:d4299a91-7238-8001-f3f3-30afe293dd29} {Suffrage:Voter ID:server-1632ae58-7f3f-78b4-9d00-48d6e68b12e4 Address:1632ae58-7f3f-78b4-9d00-48d6e68b12e4} {Suffrage:Voter ID:server-ff25c907-8e82-8b9b-8c4f-cbb68fc77857 Address:ff25c907-8e82-8b9b-8c4f-cbb68fc77857}]" 2026-05-25T03:42:56.223Z [INFO] server-d4299a91-7238-8001-f3f3-30afe293dd29: entering follower state: follower="Node at d4299a91-7238-8001-f3f3-30afe293dd29 [Follower]" leader-address= leader-id= 2026-05-25T03:42:56.224Z [INFO] server-1632ae58-7f3f-78b4-9d00-48d6e68b12e4: entering follower state: follower="Node at 1632ae58-7f3f-78b4-9d00-48d6e68b12e4 [Follower]" leader-address= leader-id= 2026-05-25T03:42:56.224Z [INFO] server-ff25c907-8e82-8b9b-8c4f-cbb68fc77857: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d4299a91-7238-8001-f3f3-30afe293dd29 Address:d4299a91-7238-8001-f3f3-30afe293dd29} {Suffrage:Voter ID:server-1632ae58-7f3f-78b4-9d00-48d6e68b12e4 Address:1632ae58-7f3f-78b4-9d00-48d6e68b12e4} {Suffrage:Voter ID:server-ff25c907-8e82-8b9b-8c4f-cbb68fc77857 Address:ff25c907-8e82-8b9b-8c4f-cbb68fc77857}]" 2026-05-25T03:42:56.224Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:56.224Z [INFO] server-ff25c907-8e82-8b9b-8c4f-cbb68fc77857: entering follower state: follower="Node at ff25c907-8e82-8b9b-8c4f-cbb68fc77857 [Follower]" leader-address= leader-id= 2026-05-25T03:42:56.292Z [WARN] server-d4299a91-7238-8001-f3f3-30afe293dd29: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:56.292Z [INFO] server-d4299a91-7238-8001-f3f3-30afe293dd29: entering candidate state: node="Node at d4299a91-7238-8001-f3f3-30afe293dd29 [Candidate]" term=2 2026-05-25T03:42:56.292Z [INFO] server-d4299a91-7238-8001-f3f3-30afe293dd29: election won: term=2 tally=2 2026-05-25T03:42:56.292Z [INFO] server-d4299a91-7238-8001-f3f3-30afe293dd29: entering leader state: leader="Node at d4299a91-7238-8001-f3f3-30afe293dd29 [Leader]" 2026-05-25T03:42:56.292Z [INFO] server-d4299a91-7238-8001-f3f3-30afe293dd29: added peer, starting replication: peer=server-1632ae58-7f3f-78b4-9d00-48d6e68b12e4 2026-05-25T03:42:56.292Z [INFO] server-d4299a91-7238-8001-f3f3-30afe293dd29: added peer, starting replication: peer=server-ff25c907-8e82-8b9b-8c4f-cbb68fc77857 2026-05-25T03:42:56.292Z [INFO] server-d4299a91-7238-8001-f3f3-30afe293dd29: pipelining replication: peer="{Voter server-ff25c907-8e82-8b9b-8c4f-cbb68fc77857 ff25c907-8e82-8b9b-8c4f-cbb68fc77857}" 2026-05-25T03:42:56.292Z [INFO] server-d4299a91-7238-8001-f3f3-30afe293dd29: pipelining replication: peer="{Voter server-1632ae58-7f3f-78b4-9d00-48d6e68b12e4 1632ae58-7f3f-78b4-9d00-48d6e68b12e4}" 2026-05-25T03:42:56.397Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:56.292355357 +0000 UTC m=+102.465330841 (2 nodes), 68.2442ms from start of poll, 68.66756ms from cluster start. Timeout at 2026-05-25 03:42:56.397360077 +0000 UTC m=+102.570335561, 105.00472ms after stability 2026-05-25T03:42:56.397Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:56.503Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:56.397554077 +0000 UTC m=+102.570529521 (1 nodes), 2.88µs from start of poll, 173.86624ms from cluster start. Timeout at 2026-05-25 03:42:56.502565957 +0000 UTC m=+102.675541441, 105.01192ms after stability 2026-05-25T03:42:56.503Z [INFO] server-d4299a91-7238-8001-f3f3-30afe293dd29: aborting pipeline replication: peer="{Voter server-ff25c907-8e82-8b9b-8c4f-cbb68fc77857 ff25c907-8e82-8b9b-8c4f-cbb68fc77857}" 2026-05-25T03:42:56.503Z [INFO] server-d4299a91-7238-8001-f3f3-30afe293dd29: aborting pipeline replication: peer="{Voter server-1632ae58-7f3f-78b4-9d00-48d6e68b12e4 1632ae58-7f3f-78b4-9d00-48d6e68b12e4}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.28s) === RUN TestRaft_LeadershipTransferToBehindServer 2026-05-25T03:42:56.504Z [INFO] server-cf71d5da-19d0-9214-6b0f-3f45faee7935: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf71d5da-19d0-9214-6b0f-3f45faee7935 Address:cf71d5da-19d0-9214-6b0f-3f45faee7935} {Suffrage:Voter ID:server-810f4d41-b1be-8dfd-bf19-25401c6f48b0 Address:810f4d41-b1be-8dfd-bf19-25401c6f48b0} {Suffrage:Voter ID:server-7516ea4e-a567-8d36-8566-40ffc7303dfa Address:7516ea4e-a567-8d36-8566-40ffc7303dfa}]" 2026-05-25T03:42:56.504Z [INFO] server-cf71d5da-19d0-9214-6b0f-3f45faee7935: entering follower state: follower="Node at cf71d5da-19d0-9214-6b0f-3f45faee7935 [Follower]" leader-address= leader-id= 2026-05-25T03:42:56.504Z [INFO] server-810f4d41-b1be-8dfd-bf19-25401c6f48b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf71d5da-19d0-9214-6b0f-3f45faee7935 Address:cf71d5da-19d0-9214-6b0f-3f45faee7935} {Suffrage:Voter ID:server-810f4d41-b1be-8dfd-bf19-25401c6f48b0 Address:810f4d41-b1be-8dfd-bf19-25401c6f48b0} {Suffrage:Voter ID:server-7516ea4e-a567-8d36-8566-40ffc7303dfa Address:7516ea4e-a567-8d36-8566-40ffc7303dfa}]" 2026-05-25T03:42:56.504Z [INFO] server-810f4d41-b1be-8dfd-bf19-25401c6f48b0: entering follower state: follower="Node at 810f4d41-b1be-8dfd-bf19-25401c6f48b0 [Follower]" leader-address= leader-id= 2026-05-25T03:42:56.504Z [INFO] server-7516ea4e-a567-8d36-8566-40ffc7303dfa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf71d5da-19d0-9214-6b0f-3f45faee7935 Address:cf71d5da-19d0-9214-6b0f-3f45faee7935} {Suffrage:Voter ID:server-810f4d41-b1be-8dfd-bf19-25401c6f48b0 Address:810f4d41-b1be-8dfd-bf19-25401c6f48b0} {Suffrage:Voter ID:server-7516ea4e-a567-8d36-8566-40ffc7303dfa Address:7516ea4e-a567-8d36-8566-40ffc7303dfa}]" 2026-05-25T03:42:56.504Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:56.504Z [INFO] server-7516ea4e-a567-8d36-8566-40ffc7303dfa: entering follower state: follower="Node at 7516ea4e-a567-8d36-8566-40ffc7303dfa [Follower]" leader-address= leader-id= 2026-05-25T03:42:56.570Z [WARN] server-810f4d41-b1be-8dfd-bf19-25401c6f48b0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:56.570Z [INFO] server-810f4d41-b1be-8dfd-bf19-25401c6f48b0: entering candidate state: node="Node at 810f4d41-b1be-8dfd-bf19-25401c6f48b0 [Candidate]" term=2 2026-05-25T03:42:56.570Z [INFO] server-810f4d41-b1be-8dfd-bf19-25401c6f48b0: election won: term=2 tally=2 2026-05-25T03:42:56.570Z [INFO] server-810f4d41-b1be-8dfd-bf19-25401c6f48b0: entering leader state: leader="Node at 810f4d41-b1be-8dfd-bf19-25401c6f48b0 [Leader]" 2026-05-25T03:42:56.570Z [INFO] server-810f4d41-b1be-8dfd-bf19-25401c6f48b0: added peer, starting replication: peer=server-cf71d5da-19d0-9214-6b0f-3f45faee7935 2026-05-25T03:42:56.570Z [INFO] server-810f4d41-b1be-8dfd-bf19-25401c6f48b0: added peer, starting replication: peer=server-7516ea4e-a567-8d36-8566-40ffc7303dfa 2026-05-25T03:42:56.570Z [INFO] server-810f4d41-b1be-8dfd-bf19-25401c6f48b0: pipelining replication: peer="{Voter server-7516ea4e-a567-8d36-8566-40ffc7303dfa 7516ea4e-a567-8d36-8566-40ffc7303dfa}" 2026-05-25T03:42:56.570Z [INFO] server-810f4d41-b1be-8dfd-bf19-25401c6f48b0: pipelining replication: peer="{Voter server-cf71d5da-19d0-9214-6b0f-3f45faee7935 cf71d5da-19d0-9214-6b0f-3f45faee7935}" 2026-05-25T03:42:56.676Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:56.570395637 +0000 UTC m=+102.743371081 (1 nodes), 65.47924ms from start of poll, 66.06212ms from cluster start. Timeout at 2026-05-25 03:42:56.675398957 +0000 UTC m=+102.848374401, 105.00332ms after stability 2026-05-25T03:42:56.676Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:56.781Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:56.676298397 +0000 UTC m=+102.849273841 (2 nodes), 3.6µs from start of poll, 171.96488ms from cluster start. Timeout at 2026-05-25 03:42:56.781305437 +0000 UTC m=+102.954280881, 105.00704ms after stability 2026-05-25T03:42:56.835Z [INFO] server-cf71d5da-19d0-9214-6b0f-3f45faee7935: entering candidate state: node="Node at cf71d5da-19d0-9214-6b0f-3f45faee7935 [Candidate]" term=3 2026-05-25T03:42:56.835Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:56.835Z [INFO] server-cf71d5da-19d0-9214-6b0f-3f45faee7935: election won: term=3 tally=2 2026-05-25T03:42:56.835Z [INFO] server-cf71d5da-19d0-9214-6b0f-3f45faee7935: entering leader state: leader="Node at cf71d5da-19d0-9214-6b0f-3f45faee7935 [Leader]" 2026-05-25T03:42:56.835Z [INFO] server-cf71d5da-19d0-9214-6b0f-3f45faee7935: added peer, starting replication: peer=server-810f4d41-b1be-8dfd-bf19-25401c6f48b0 2026-05-25T03:42:56.835Z [INFO] server-810f4d41-b1be-8dfd-bf19-25401c6f48b0: entering follower state: follower="Node at 810f4d41-b1be-8dfd-bf19-25401c6f48b0 [Follower]" leader-address= leader-id= 2026-05-25T03:42:56.835Z [INFO] server-cf71d5da-19d0-9214-6b0f-3f45faee7935: added peer, starting replication: peer=server-7516ea4e-a567-8d36-8566-40ffc7303dfa 2026-05-25T03:42:56.835Z [INFO] server-810f4d41-b1be-8dfd-bf19-25401c6f48b0: aborting pipeline replication: peer="{Voter server-cf71d5da-19d0-9214-6b0f-3f45faee7935 cf71d5da-19d0-9214-6b0f-3f45faee7935}" 2026-05-25T03:42:56.835Z [INFO] server-810f4d41-b1be-8dfd-bf19-25401c6f48b0: aborting pipeline replication: peer="{Voter server-7516ea4e-a567-8d36-8566-40ffc7303dfa 7516ea4e-a567-8d36-8566-40ffc7303dfa}" 2026-05-25T03:42:56.835Z [INFO] server-cf71d5da-19d0-9214-6b0f-3f45faee7935: pipelining replication: peer="{Voter server-810f4d41-b1be-8dfd-bf19-25401c6f48b0 810f4d41-b1be-8dfd-bf19-25401c6f48b0}" 2026-05-25T03:42:56.835Z [INFO] server-cf71d5da-19d0-9214-6b0f-3f45faee7935: pipelining replication: peer="{Voter server-7516ea4e-a567-8d36-8566-40ffc7303dfa 7516ea4e-a567-8d36-8566-40ffc7303dfa}" 2026-05-25T03:42:56.941Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:56.835503877 +0000 UTC m=+103.008479321 (1 nodes), 37.8µs from start of poll, 331.17036ms from cluster start. Timeout at 2026-05-25 03:42:56.940560877 +0000 UTC m=+103.113536321, 105.057ms after stability 2026-05-25T03:42:56.941Z [INFO] server-cf71d5da-19d0-9214-6b0f-3f45faee7935: aborting pipeline replication: peer="{Voter server-7516ea4e-a567-8d36-8566-40ffc7303dfa 7516ea4e-a567-8d36-8566-40ffc7303dfa}" 2026-05-25T03:42:56.941Z [INFO] server-cf71d5da-19d0-9214-6b0f-3f45faee7935: aborting pipeline replication: peer="{Voter server-810f4d41-b1be-8dfd-bf19-25401c6f48b0 810f4d41-b1be-8dfd-bf19-25401c6f48b0}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.44s) === RUN TestRaft_LeadershipTransferToItself 2026-05-25T03:42:56.943Z [INFO] server-c3cc9c0c-d33e-4a3a-3574-2365f5849673: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3cc9c0c-d33e-4a3a-3574-2365f5849673 Address:c3cc9c0c-d33e-4a3a-3574-2365f5849673} {Suffrage:Voter ID:server-e982cd76-9d85-b23e-7aa3-8a31a06ad823 Address:e982cd76-9d85-b23e-7aa3-8a31a06ad823} {Suffrage:Voter ID:server-13690e06-41aa-8dc5-df33-96a81ebf0e13 Address:13690e06-41aa-8dc5-df33-96a81ebf0e13}]" 2026-05-25T03:42:56.943Z [INFO] server-e982cd76-9d85-b23e-7aa3-8a31a06ad823: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3cc9c0c-d33e-4a3a-3574-2365f5849673 Address:c3cc9c0c-d33e-4a3a-3574-2365f5849673} {Suffrage:Voter ID:server-e982cd76-9d85-b23e-7aa3-8a31a06ad823 Address:e982cd76-9d85-b23e-7aa3-8a31a06ad823} {Suffrage:Voter ID:server-13690e06-41aa-8dc5-df33-96a81ebf0e13 Address:13690e06-41aa-8dc5-df33-96a81ebf0e13}]" 2026-05-25T03:42:56.943Z [INFO] server-13690e06-41aa-8dc5-df33-96a81ebf0e13: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c3cc9c0c-d33e-4a3a-3574-2365f5849673 Address:c3cc9c0c-d33e-4a3a-3574-2365f5849673} {Suffrage:Voter ID:server-e982cd76-9d85-b23e-7aa3-8a31a06ad823 Address:e982cd76-9d85-b23e-7aa3-8a31a06ad823} {Suffrage:Voter ID:server-13690e06-41aa-8dc5-df33-96a81ebf0e13 Address:13690e06-41aa-8dc5-df33-96a81ebf0e13}]" 2026-05-25T03:42:56.943Z [INFO] server-e982cd76-9d85-b23e-7aa3-8a31a06ad823: entering follower state: follower="Node at e982cd76-9d85-b23e-7aa3-8a31a06ad823 [Follower]" leader-address= leader-id= 2026-05-25T03:42:56.943Z [INFO] server-c3cc9c0c-d33e-4a3a-3574-2365f5849673: entering follower state: follower="Node at c3cc9c0c-d33e-4a3a-3574-2365f5849673 [Follower]" leader-address= leader-id= 2026-05-25T03:42:56.943Z [INFO] server-13690e06-41aa-8dc5-df33-96a81ebf0e13: entering follower state: follower="Node at 13690e06-41aa-8dc5-df33-96a81ebf0e13 [Follower]" leader-address= leader-id= 2026-05-25T03:42:56.943Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:56.995Z [WARN] server-13690e06-41aa-8dc5-df33-96a81ebf0e13: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:56.995Z [INFO] server-13690e06-41aa-8dc5-df33-96a81ebf0e13: entering candidate state: node="Node at 13690e06-41aa-8dc5-df33-96a81ebf0e13 [Candidate]" term=2 2026-05-25T03:42:56.995Z [INFO] server-13690e06-41aa-8dc5-df33-96a81ebf0e13: election won: term=2 tally=2 2026-05-25T03:42:56.995Z [INFO] server-13690e06-41aa-8dc5-df33-96a81ebf0e13: entering leader state: leader="Node at 13690e06-41aa-8dc5-df33-96a81ebf0e13 [Leader]" 2026-05-25T03:42:56.995Z [INFO] server-13690e06-41aa-8dc5-df33-96a81ebf0e13: added peer, starting replication: peer=server-c3cc9c0c-d33e-4a3a-3574-2365f5849673 2026-05-25T03:42:56.995Z [INFO] server-13690e06-41aa-8dc5-df33-96a81ebf0e13: added peer, starting replication: peer=server-e982cd76-9d85-b23e-7aa3-8a31a06ad823 2026-05-25T03:42:56.995Z [INFO] server-13690e06-41aa-8dc5-df33-96a81ebf0e13: pipelining replication: peer="{Voter server-e982cd76-9d85-b23e-7aa3-8a31a06ad823 e982cd76-9d85-b23e-7aa3-8a31a06ad823}" 2026-05-25T03:42:56.995Z [INFO] server-13690e06-41aa-8dc5-df33-96a81ebf0e13: pipelining replication: peer="{Voter server-c3cc9c0c-d33e-4a3a-3574-2365f5849673 c3cc9c0c-d33e-4a3a-3574-2365f5849673}" 2026-05-25T03:42:57.101Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:56.995785717 +0000 UTC m=+103.168761161 (1 nodes), 52.17384ms from start of poll, 52.85532ms from cluster start. Timeout at 2026-05-25 03:42:57.100789477 +0000 UTC m=+103.273764921, 105.00376ms after stability 2026-05-25T03:42:57.101Z [INFO] server-13690e06-41aa-8dc5-df33-96a81ebf0e13: cannot transfer leadership to itself 2026-05-25T03:42:57.101Z [INFO] server-13690e06-41aa-8dc5-df33-96a81ebf0e13: aborting pipeline replication: peer="{Voter server-e982cd76-9d85-b23e-7aa3-8a31a06ad823 e982cd76-9d85-b23e-7aa3-8a31a06ad823}" 2026-05-25T03:42:57.101Z [INFO] server-13690e06-41aa-8dc5-df33-96a81ebf0e13: aborting pipeline replication: peer="{Voter server-c3cc9c0c-d33e-4a3a-3574-2365f5849673 c3cc9c0c-d33e-4a3a-3574-2365f5849673}" --- PASS: TestRaft_LeadershipTransferToItself (0.16s) === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests 2026-05-25T03:42:57.102Z [INFO] server-35187319-09f3-212b-eab6-34e08df20380: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35187319-09f3-212b-eab6-34e08df20380 Address:35187319-09f3-212b-eab6-34e08df20380} {Suffrage:Voter ID:server-c2127f15-c5dd-1c26-1f56-27366d04eb6b Address:c2127f15-c5dd-1c26-1f56-27366d04eb6b} {Suffrage:Voter ID:server-40f216b6-b518-550c-c87c-1cf1e053b07d Address:40f216b6-b518-550c-c87c-1cf1e053b07d}]" 2026-05-25T03:42:57.102Z [INFO] server-35187319-09f3-212b-eab6-34e08df20380: entering follower state: follower="Node at 35187319-09f3-212b-eab6-34e08df20380 [Follower]" leader-address= leader-id= 2026-05-25T03:42:57.102Z [INFO] server-c2127f15-c5dd-1c26-1f56-27366d04eb6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35187319-09f3-212b-eab6-34e08df20380 Address:35187319-09f3-212b-eab6-34e08df20380} {Suffrage:Voter ID:server-c2127f15-c5dd-1c26-1f56-27366d04eb6b Address:c2127f15-c5dd-1c26-1f56-27366d04eb6b} {Suffrage:Voter ID:server-40f216b6-b518-550c-c87c-1cf1e053b07d Address:40f216b6-b518-550c-c87c-1cf1e053b07d}]" 2026-05-25T03:42:57.102Z [INFO] server-c2127f15-c5dd-1c26-1f56-27366d04eb6b: entering follower state: follower="Node at c2127f15-c5dd-1c26-1f56-27366d04eb6b [Follower]" leader-address= leader-id= 2026-05-25T03:42:57.102Z [INFO] server-40f216b6-b518-550c-c87c-1cf1e053b07d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35187319-09f3-212b-eab6-34e08df20380 Address:35187319-09f3-212b-eab6-34e08df20380} {Suffrage:Voter ID:server-c2127f15-c5dd-1c26-1f56-27366d04eb6b Address:c2127f15-c5dd-1c26-1f56-27366d04eb6b} {Suffrage:Voter ID:server-40f216b6-b518-550c-c87c-1cf1e053b07d Address:40f216b6-b518-550c-c87c-1cf1e053b07d}]" 2026-05-25T03:42:57.102Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:57.102Z [INFO] server-40f216b6-b518-550c-c87c-1cf1e053b07d: entering follower state: follower="Node at 40f216b6-b518-550c-c87c-1cf1e053b07d [Follower]" leader-address= leader-id= 2026-05-25T03:42:57.172Z [WARN] server-40f216b6-b518-550c-c87c-1cf1e053b07d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:57.172Z [INFO] server-40f216b6-b518-550c-c87c-1cf1e053b07d: entering candidate state: node="Node at 40f216b6-b518-550c-c87c-1cf1e053b07d [Candidate]" term=2 2026-05-25T03:42:57.172Z [INFO] server-40f216b6-b518-550c-c87c-1cf1e053b07d: election won: term=2 tally=2 2026-05-25T03:42:57.172Z [INFO] server-40f216b6-b518-550c-c87c-1cf1e053b07d: entering leader state: leader="Node at 40f216b6-b518-550c-c87c-1cf1e053b07d [Leader]" 2026-05-25T03:42:57.172Z [INFO] server-40f216b6-b518-550c-c87c-1cf1e053b07d: added peer, starting replication: peer=server-35187319-09f3-212b-eab6-34e08df20380 2026-05-25T03:42:57.172Z [INFO] server-40f216b6-b518-550c-c87c-1cf1e053b07d: added peer, starting replication: peer=server-c2127f15-c5dd-1c26-1f56-27366d04eb6b 2026-05-25T03:42:57.172Z [INFO] server-40f216b6-b518-550c-c87c-1cf1e053b07d: pipelining replication: peer="{Voter server-c2127f15-c5dd-1c26-1f56-27366d04eb6b c2127f15-c5dd-1c26-1f56-27366d04eb6b}" 2026-05-25T03:42:57.172Z [INFO] server-40f216b6-b518-550c-c87c-1cf1e053b07d: pipelining replication: peer="{Voter server-35187319-09f3-212b-eab6-34e08df20380 35187319-09f3-212b-eab6-34e08df20380}" 2026-05-25T03:42:57.277Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:57.172411397 +0000 UTC m=+103.345386841 (1 nodes), 69.9998ms from start of poll, 70.5402ms from cluster start. Timeout at 2026-05-25 03:42:57.277420037 +0000 UTC m=+103.450395481, 105.00864ms after stability raft_test.go:2457: waiting on future 0 raft_test.go:2457: waiting on future 1 raft_test.go:2457: waiting on future 2 raft_test.go:2457: waiting on future 3 raft_test.go:2457: waiting on future 4 raft_test.go:2457: waiting on future 5 raft_test.go:2457: waiting on future 6 2026-05-25T03:42:57.277Z [INFO] server-40f216b6-b518-550c-c87c-1cf1e053b07d: aborting pipeline replication: peer="{Voter server-c2127f15-c5dd-1c26-1f56-27366d04eb6b c2127f15-c5dd-1c26-1f56-27366d04eb6b}" 2026-05-25T03:42:57.277Z [INFO] server-40f216b6-b518-550c-c87c-1cf1e053b07d: aborting pipeline replication: peer="{Voter server-35187319-09f3-212b-eab6-34e08df20380 35187319-09f3-212b-eab6-34e08df20380}" --- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.18s) === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout 2026-05-25T03:42:57.278Z [INFO] server-85e7ae70-37c0-d329-b1f2-2469addd9e50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85e7ae70-37c0-d329-b1f2-2469addd9e50 Address:85e7ae70-37c0-d329-b1f2-2469addd9e50} {Suffrage:Voter ID:server-50028adc-6165-d44d-f5f1-d9ccc9ebaa78 Address:50028adc-6165-d44d-f5f1-d9ccc9ebaa78} {Suffrage:Voter ID:server-06bffb9a-8228-e491-fa5f-65395950b2df Address:06bffb9a-8228-e491-fa5f-65395950b2df}]" 2026-05-25T03:42:57.279Z [INFO] server-50028adc-6165-d44d-f5f1-d9ccc9ebaa78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85e7ae70-37c0-d329-b1f2-2469addd9e50 Address:85e7ae70-37c0-d329-b1f2-2469addd9e50} {Suffrage:Voter ID:server-50028adc-6165-d44d-f5f1-d9ccc9ebaa78 Address:50028adc-6165-d44d-f5f1-d9ccc9ebaa78} {Suffrage:Voter ID:server-06bffb9a-8228-e491-fa5f-65395950b2df Address:06bffb9a-8228-e491-fa5f-65395950b2df}]" 2026-05-25T03:42:57.279Z [INFO] server-06bffb9a-8228-e491-fa5f-65395950b2df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-85e7ae70-37c0-d329-b1f2-2469addd9e50 Address:85e7ae70-37c0-d329-b1f2-2469addd9e50} {Suffrage:Voter ID:server-50028adc-6165-d44d-f5f1-d9ccc9ebaa78 Address:50028adc-6165-d44d-f5f1-d9ccc9ebaa78} {Suffrage:Voter ID:server-06bffb9a-8228-e491-fa5f-65395950b2df Address:06bffb9a-8228-e491-fa5f-65395950b2df}]" 2026-05-25T03:42:57.279Z [INFO] server-50028adc-6165-d44d-f5f1-d9ccc9ebaa78: entering follower state: follower="Node at 50028adc-6165-d44d-f5f1-d9ccc9ebaa78 [Follower]" leader-address= leader-id= 2026-05-25T03:42:57.278Z [INFO] server-85e7ae70-37c0-d329-b1f2-2469addd9e50: entering follower state: follower="Node at 85e7ae70-37c0-d329-b1f2-2469addd9e50 [Follower]" leader-address= leader-id= 2026-05-25T03:42:57.279Z [INFO] server-06bffb9a-8228-e491-fa5f-65395950b2df: entering follower state: follower="Node at 06bffb9a-8228-e491-fa5f-65395950b2df [Follower]" leader-address= leader-id= 2026-05-25T03:42:57.279Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:57.337Z [WARN] server-85e7ae70-37c0-d329-b1f2-2469addd9e50: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:57.337Z [INFO] server-85e7ae70-37c0-d329-b1f2-2469addd9e50: entering candidate state: node="Node at 85e7ae70-37c0-d329-b1f2-2469addd9e50 [Candidate]" term=2 2026-05-25T03:42:57.337Z [INFO] server-85e7ae70-37c0-d329-b1f2-2469addd9e50: election won: term=2 tally=2 2026-05-25T03:42:57.337Z [INFO] server-85e7ae70-37c0-d329-b1f2-2469addd9e50: entering leader state: leader="Node at 85e7ae70-37c0-d329-b1f2-2469addd9e50 [Leader]" 2026-05-25T03:42:57.337Z [INFO] server-85e7ae70-37c0-d329-b1f2-2469addd9e50: added peer, starting replication: peer=server-50028adc-6165-d44d-f5f1-d9ccc9ebaa78 2026-05-25T03:42:57.337Z [INFO] server-85e7ae70-37c0-d329-b1f2-2469addd9e50: added peer, starting replication: peer=server-06bffb9a-8228-e491-fa5f-65395950b2df 2026-05-25T03:42:57.337Z [INFO] server-85e7ae70-37c0-d329-b1f2-2469addd9e50: pipelining replication: peer="{Voter server-50028adc-6165-d44d-f5f1-d9ccc9ebaa78 50028adc-6165-d44d-f5f1-d9ccc9ebaa78}" 2026-05-25T03:42:57.337Z [INFO] server-85e7ae70-37c0-d329-b1f2-2469addd9e50: pipelining replication: peer="{Voter server-06bffb9a-8228-e491-fa5f-65395950b2df 06bffb9a-8228-e491-fa5f-65395950b2df}" 2026-05-25T03:42:57.443Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:57.337789517 +0000 UTC m=+103.510764961 (1 nodes), 58.53384ms from start of poll, 59.0738ms from cluster start. Timeout at 2026-05-25 03:42:57.442800077 +0000 UTC m=+103.615775521, 105.01056ms after stability 2026-05-25T03:42:57.443Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:57.549Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:57.443716317 +0000 UTC m=+103.616691761 (2 nodes), 1.4µs from start of poll, 165.0006ms from cluster start. Timeout at 2026-05-25 03:42:57.548747277 +0000 UTC m=+103.721722721, 105.03096ms after stability 2026-05-25T03:42:58.069Z [INFO] server-85e7ae70-37c0-d329-b1f2-2469addd9e50: aborting pipeline replication: peer="{Voter server-06bffb9a-8228-e491-fa5f-65395950b2df 06bffb9a-8228-e491-fa5f-65395950b2df}" 2026-05-25T03:42:58.069Z [INFO] server-85e7ae70-37c0-d329-b1f2-2469addd9e50: aborting pipeline replication: peer="{Voter server-50028adc-6165-d44d-f5f1-d9ccc9ebaa78 50028adc-6165-d44d-f5f1-d9ccc9ebaa78}" --- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.79s) === RUN TestRaft_LeadershipTransferIgnoresNonvoters 2026-05-25T03:42:58.070Z [INFO] server-2fa10731-f156-11b7-dc87-18a2ee751d42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fa10731-f156-11b7-dc87-18a2ee751d42 Address:2fa10731-f156-11b7-dc87-18a2ee751d42} {Suffrage:Voter ID:server-ffb8b221-f0f0-3e1d-9998-a5904c7fd568 Address:ffb8b221-f0f0-3e1d-9998-a5904c7fd568}]" 2026-05-25T03:42:58.070Z [INFO] server-ffb8b221-f0f0-3e1d-9998-a5904c7fd568: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2fa10731-f156-11b7-dc87-18a2ee751d42 Address:2fa10731-f156-11b7-dc87-18a2ee751d42} {Suffrage:Voter ID:server-ffb8b221-f0f0-3e1d-9998-a5904c7fd568 Address:ffb8b221-f0f0-3e1d-9998-a5904c7fd568}]" 2026-05-25T03:42:58.070Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:58.070Z [INFO] server-2fa10731-f156-11b7-dc87-18a2ee751d42: entering follower state: follower="Node at 2fa10731-f156-11b7-dc87-18a2ee751d42 [Follower]" leader-address= leader-id= 2026-05-25T03:42:58.070Z [INFO] server-ffb8b221-f0f0-3e1d-9998-a5904c7fd568: entering follower state: follower="Node at ffb8b221-f0f0-3e1d-9998-a5904c7fd568 [Follower]" leader-address= leader-id= 2026-05-25T03:42:58.140Z [WARN] server-2fa10731-f156-11b7-dc87-18a2ee751d42: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:58.140Z [INFO] server-2fa10731-f156-11b7-dc87-18a2ee751d42: entering candidate state: node="Node at 2fa10731-f156-11b7-dc87-18a2ee751d42 [Candidate]" term=2 2026-05-25T03:42:58.140Z [INFO] server-2fa10731-f156-11b7-dc87-18a2ee751d42: election won: term=2 tally=2 2026-05-25T03:42:58.140Z [INFO] server-2fa10731-f156-11b7-dc87-18a2ee751d42: entering leader state: leader="Node at 2fa10731-f156-11b7-dc87-18a2ee751d42 [Leader]" 2026-05-25T03:42:58.140Z [INFO] server-2fa10731-f156-11b7-dc87-18a2ee751d42: added peer, starting replication: peer=server-ffb8b221-f0f0-3e1d-9998-a5904c7fd568 2026-05-25T03:42:58.141Z [INFO] server-2fa10731-f156-11b7-dc87-18a2ee751d42: pipelining replication: peer="{Voter server-ffb8b221-f0f0-3e1d-9998-a5904c7fd568 ffb8b221-f0f0-3e1d-9998-a5904c7fd568}" 2026-05-25T03:42:58.246Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:58.140971437 +0000 UTC m=+104.313946881 (1 nodes), 70.7018ms from start of poll, 70.89104ms from cluster start. Timeout at 2026-05-25 03:42:58.245982677 +0000 UTC m=+104.418958161, 105.01128ms after stability 2026-05-25T03:42:58.246Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:58.352Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:58.246619277 +0000 UTC m=+104.419594721 (1 nodes), 880ns from start of poll, 176.53888ms from cluster start. Timeout at 2026-05-25 03:42:58.351718717 +0000 UTC m=+104.524694161, 105.09944ms after stability 2026-05-25T03:42:58.352Z [INFO] server-2fa10731-f156-11b7-dc87-18a2ee751d42: updating configuration: command=DemoteVoter server-id=server-ffb8b221-f0f0-3e1d-9998-a5904c7fd568 server-addr= servers="[{Suffrage:Voter ID:server-2fa10731-f156-11b7-dc87-18a2ee751d42 Address:2fa10731-f156-11b7-dc87-18a2ee751d42} {Suffrage:Nonvoter ID:server-ffb8b221-f0f0-3e1d-9998-a5904c7fd568 Address:ffb8b221-f0f0-3e1d-9998-a5904c7fd568}]" 2026-05-25T03:42:58.352Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:58.457Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:58.352275757 +0000 UTC m=+104.525251201 (1 nodes), 720ns from start of poll, 282.19536ms from cluster start. Timeout at 2026-05-25 03:42:58.457337117 +0000 UTC m=+104.630312561, 105.06136ms after stability 2026-05-25T03:42:58.457Z [INFO] server-2fa10731-f156-11b7-dc87-18a2ee751d42: aborting pipeline replication: peer="{Voter server-ffb8b221-f0f0-3e1d-9998-a5904c7fd568 ffb8b221-f0f0-3e1d-9998-a5904c7fd568}" --- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.39s) === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap 2026-05-25T03:42:58.458Z [INFO] server-2d80bfb7-17e2-1e89-600a-8dcb6127ac10: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d80bfb7-17e2-1e89-600a-8dcb6127ac10 Address:2d80bfb7-17e2-1e89-600a-8dcb6127ac10} {Suffrage:Voter ID:server-105b6186-18bb-3dd5-73e1-e8c9578b86c4 Address:105b6186-18bb-3dd5-73e1-e8c9578b86c4}]" 2026-05-25T03:42:58.458Z [INFO] server-2d80bfb7-17e2-1e89-600a-8dcb6127ac10: entering follower state: follower="Node at 2d80bfb7-17e2-1e89-600a-8dcb6127ac10 [Follower]" leader-address= leader-id= 2026-05-25T03:42:58.458Z [INFO] server-105b6186-18bb-3dd5-73e1-e8c9578b86c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d80bfb7-17e2-1e89-600a-8dcb6127ac10 Address:2d80bfb7-17e2-1e89-600a-8dcb6127ac10} {Suffrage:Voter ID:server-105b6186-18bb-3dd5-73e1-e8c9578b86c4 Address:105b6186-18bb-3dd5-73e1-e8c9578b86c4}]" 2026-05-25T03:42:58.458Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:58.458Z [INFO] server-105b6186-18bb-3dd5-73e1-e8c9578b86c4: entering follower state: follower="Node at 105b6186-18bb-3dd5-73e1-e8c9578b86c4 [Follower]" leader-address= leader-id= 2026-05-25T03:42:58.528Z [WARN] server-2d80bfb7-17e2-1e89-600a-8dcb6127ac10: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:58.528Z [INFO] server-2d80bfb7-17e2-1e89-600a-8dcb6127ac10: entering candidate state: node="Node at 2d80bfb7-17e2-1e89-600a-8dcb6127ac10 [Candidate]" term=2 2026-05-25T03:42:58.528Z [INFO] server-2d80bfb7-17e2-1e89-600a-8dcb6127ac10: election won: term=2 tally=2 2026-05-25T03:42:58.528Z [INFO] server-2d80bfb7-17e2-1e89-600a-8dcb6127ac10: entering leader state: leader="Node at 2d80bfb7-17e2-1e89-600a-8dcb6127ac10 [Leader]" 2026-05-25T03:42:58.528Z [INFO] server-2d80bfb7-17e2-1e89-600a-8dcb6127ac10: added peer, starting replication: peer=server-105b6186-18bb-3dd5-73e1-e8c9578b86c4 2026-05-25T03:42:58.528Z [INFO] server-2d80bfb7-17e2-1e89-600a-8dcb6127ac10: pipelining replication: peer="{Voter server-105b6186-18bb-3dd5-73e1-e8c9578b86c4 105b6186-18bb-3dd5-73e1-e8c9578b86c4}" 2026-05-25T03:42:58.633Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:58.528418957 +0000 UTC m=+104.701394401 (1 nodes), 69.77552ms from start of poll, 70.22032ms from cluster start. Timeout at 2026-05-25 03:42:58.633423157 +0000 UTC m=+104.806398601, 105.0042ms after stability 2026-05-25T03:42:58.633Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:58.739Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:58.633830557 +0000 UTC m=+104.806806041 (1 nodes), 2.52µs from start of poll, 175.63196ms from cluster start. Timeout at 2026-05-25 03:42:58.738870637 +0000 UTC m=+104.911846081, 105.04004ms after stability 2026-05-25T03:42:58.749Z [INFO] server-105b6186-18bb-3dd5-73e1-e8c9578b86c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d80bfb7-17e2-1e89-600a-8dcb6127ac10 Address:2d80bfb7-17e2-1e89-600a-8dcb6127ac10} {Suffrage:Voter ID:server-105b6186-18bb-3dd5-73e1-e8c9578b86c4 Address:105b6186-18bb-3dd5-73e1-e8c9578b86c4}]" 2026-05-25T03:42:58.749Z [INFO] server-2d80bfb7-17e2-1e89-600a-8dcb6127ac10: aborting pipeline replication: peer="{Voter server-105b6186-18bb-3dd5-73e1-e8c9578b86c4 105b6186-18bb-3dd5-73e1-e8c9578b86c4}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.29s) === RUN TestRaft_LogStoreIsMonotonic 2026-05-25T03:42:58.750Z [INFO] server-9685681a-d098-451d-442f-d73db5ce18be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9685681a-d098-451d-442f-d73db5ce18be Address:9685681a-d098-451d-442f-d73db5ce18be}]" 2026-05-25T03:42:58.750Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:58.750Z [INFO] server-9685681a-d098-451d-442f-d73db5ce18be: entering follower state: follower="Node at 9685681a-d098-451d-442f-d73db5ce18be [Follower]" leader-address= leader-id= 2026-05-25T03:42:58.829Z [WARN] server-9685681a-d098-451d-442f-d73db5ce18be: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:58.829Z [INFO] server-9685681a-d098-451d-442f-d73db5ce18be: entering candidate state: node="Node at 9685681a-d098-451d-442f-d73db5ce18be [Candidate]" term=2 2026-05-25T03:42:58.829Z [INFO] server-9685681a-d098-451d-442f-d73db5ce18be: election won: term=2 tally=1 2026-05-25T03:42:58.829Z [INFO] server-9685681a-d098-451d-442f-d73db5ce18be: entering leader state: leader="Node at 9685681a-d098-451d-442f-d73db5ce18be [Leader]" 2026-05-25T03:42:58.934Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:58.829313437 +0000 UTC m=+105.002288881 (1 nodes), 79.16192ms from start of poll, 79.34288ms from cluster start. Timeout at 2026-05-25 03:42:58.934318397 +0000 UTC m=+105.107293841, 105.00496ms after stability --- PASS: TestRaft_LogStoreIsMonotonic (0.18s) === RUN TestRaft_CacheLogWithStoreError 2026-05-25T03:42:58.935Z [INFO] server-fc6f5873-c5c5-5db6-580e-acd3ac4451ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc6f5873-c5c5-5db6-580e-acd3ac4451ee Address:fc6f5873-c5c5-5db6-580e-acd3ac4451ee} {Suffrage:Voter ID:server-7eff31ac-c75e-67b4-36ee-c6f187c922d3 Address:7eff31ac-c75e-67b4-36ee-c6f187c922d3}]" 2026-05-25T03:42:58.935Z [INFO] server-7eff31ac-c75e-67b4-36ee-c6f187c922d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc6f5873-c5c5-5db6-580e-acd3ac4451ee Address:fc6f5873-c5c5-5db6-580e-acd3ac4451ee} {Suffrage:Voter ID:server-7eff31ac-c75e-67b4-36ee-c6f187c922d3 Address:7eff31ac-c75e-67b4-36ee-c6f187c922d3}]" 2026-05-25T03:42:58.935Z [INFO] server-fc6f5873-c5c5-5db6-580e-acd3ac4451ee: entering follower state: follower="Node at fc6f5873-c5c5-5db6-580e-acd3ac4451ee [Follower]" leader-address= leader-id= 2026-05-25T03:42:58.935Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:58.935Z [INFO] server-7eff31ac-c75e-67b4-36ee-c6f187c922d3: entering follower state: follower="Node at 7eff31ac-c75e-67b4-36ee-c6f187c922d3 [Follower]" leader-address= leader-id= 2026-05-25T03:42:59.007Z [WARN] server-7eff31ac-c75e-67b4-36ee-c6f187c922d3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:59.007Z [INFO] server-7eff31ac-c75e-67b4-36ee-c6f187c922d3: entering candidate state: node="Node at 7eff31ac-c75e-67b4-36ee-c6f187c922d3 [Candidate]" term=2 2026-05-25T03:42:59.007Z [INFO] server-7eff31ac-c75e-67b4-36ee-c6f187c922d3: election won: term=2 tally=2 2026-05-25T03:42:59.007Z [INFO] server-7eff31ac-c75e-67b4-36ee-c6f187c922d3: entering leader state: leader="Node at 7eff31ac-c75e-67b4-36ee-c6f187c922d3 [Leader]" 2026-05-25T03:42:59.007Z [INFO] server-7eff31ac-c75e-67b4-36ee-c6f187c922d3: added peer, starting replication: peer=server-fc6f5873-c5c5-5db6-580e-acd3ac4451ee 2026-05-25T03:42:59.007Z [INFO] server-7eff31ac-c75e-67b4-36ee-c6f187c922d3: pipelining replication: peer="{Voter server-fc6f5873-c5c5-5db6-580e-acd3ac4451ee fc6f5873-c5c5-5db6-580e-acd3ac4451ee}" 2026-05-25T03:42:59.112Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:59.007399757 +0000 UTC m=+105.180375241 (1 nodes), 72.15876ms from start of poll, 72.47908ms from cluster start. Timeout at 2026-05-25 03:42:59.112403677 +0000 UTC m=+105.285379121, 105.00388ms after stability 2026-05-25T03:42:59.112Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:59.218Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:59.112970517 +0000 UTC m=+105.285945961 (1 nodes), 2.12µs from start of poll, 178.0498ms from cluster start. Timeout at 2026-05-25 03:42:59.218001757 +0000 UTC m=+105.390977201, 105.03124ms after stability 2026-05-25T03:42:59.219Z [ERROR] server-fc6f5873-c5c5-5db6-580e-acd3ac4451ee: failed to append to logs: error="unable to store logs within log store, err: "some error"" 2026-05-25T03:42:59.219Z [WARN] server-fc6f5873-c5c5-5db6-580e-acd3ac4451ee: failed to get previous log: previous-index=9 last-index=7 error="log not found" 2026-05-25T03:42:59.219Z [INFO] server-7eff31ac-c75e-67b4-36ee-c6f187c922d3: aborting pipeline replication: peer="{Voter server-fc6f5873-c5c5-5db6-580e-acd3ac4451ee fc6f5873-c5c5-5db6-580e-acd3ac4451ee}" 2026-05-25T03:42:59.226Z [INFO] server-7eff31ac-c75e-67b4-36ee-c6f187c922d3: pipelining replication: peer="{Voter server-fc6f5873-c5c5-5db6-580e-acd3ac4451ee fc6f5873-c5c5-5db6-580e-acd3ac4451ee}" 2026-05-25T03:42:59.226Z [INFO] server-fc6f5873-c5c5-5db6-580e-acd3ac4451ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc6f5873-c5c5-5db6-580e-acd3ac4451ee Address:fc6f5873-c5c5-5db6-580e-acd3ac4451ee} {Suffrage:Voter ID:server-7eff31ac-c75e-67b4-36ee-c6f187c922d3 Address:7eff31ac-c75e-67b4-36ee-c6f187c922d3}]" 2026-05-25T03:42:59.226Z [INFO] server-7eff31ac-c75e-67b4-36ee-c6f187c922d3: aborting pipeline replication: peer="{Voter server-fc6f5873-c5c5-5db6-580e-acd3ac4451ee fc6f5873-c5c5-5db6-580e-acd3ac4451ee}" --- PASS: TestRaft_CacheLogWithStoreError (0.29s) === RUN TestRaft_ReloadConfig 2026-05-25T03:42:59.227Z [INFO] server-15c26539-9958-0e0a-ce62-7016639855c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15c26539-9958-0e0a-ce62-7016639855c5 Address:15c26539-9958-0e0a-ce62-7016639855c5}]" 2026-05-25T03:42:59.227Z [INFO] server-15c26539-9958-0e0a-ce62-7016639855c5: entering follower state: follower="Node at 15c26539-9958-0e0a-ce62-7016639855c5 [Follower]" leader-address= leader-id= --- PASS: TestRaft_ReloadConfig (0.00s) === RUN TestRaft_ReloadConfigValidates 2026-05-25T03:42:59.228Z [INFO] server-0e58b9f8-a0f5-5605-0c2a-ee4dad8fd159: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e58b9f8-a0f5-5605-0c2a-ee4dad8fd159 Address:0e58b9f8-a0f5-5605-0c2a-ee4dad8fd159}]" 2026-05-25T03:42:59.228Z [INFO] server-0e58b9f8-a0f5-5605-0c2a-ee4dad8fd159: entering follower state: follower="Node at 0e58b9f8-a0f5-5605-0c2a-ee4dad8fd159 [Shutdown]" leader-address= leader-id= --- PASS: TestRaft_ReloadConfigValidates (0.00s) === RUN TestRaft_InstallSnapshot_InvalidPeers 2026-05-25T03:42:59.228Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)" --- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s) === RUN TestRaft_VoteNotGranted_WhenNodeNotInCluster 2026-05-25T03:42:59.229Z [INFO] server-49d3bc92-09f3-05f9-6598-ce763596b3cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49d3bc92-09f3-05f9-6598-ce763596b3cc Address:49d3bc92-09f3-05f9-6598-ce763596b3cc} {Suffrage:Voter ID:server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40 Address:cd58e1ff-3c4e-9651-2fb1-39bc6f997b40} {Suffrage:Voter ID:server-90687b77-88d4-2fe2-d31e-18afeba0608b Address:90687b77-88d4-2fe2-d31e-18afeba0608b}]" 2026-05-25T03:42:59.229Z [INFO] server-49d3bc92-09f3-05f9-6598-ce763596b3cc: entering follower state: follower="Node at 49d3bc92-09f3-05f9-6598-ce763596b3cc [Follower]" leader-address= leader-id= 2026-05-25T03:42:59.229Z [INFO] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49d3bc92-09f3-05f9-6598-ce763596b3cc Address:49d3bc92-09f3-05f9-6598-ce763596b3cc} {Suffrage:Voter ID:server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40 Address:cd58e1ff-3c4e-9651-2fb1-39bc6f997b40} {Suffrage:Voter ID:server-90687b77-88d4-2fe2-d31e-18afeba0608b Address:90687b77-88d4-2fe2-d31e-18afeba0608b}]" 2026-05-25T03:42:59.229Z [INFO] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: entering follower state: follower="Node at cd58e1ff-3c4e-9651-2fb1-39bc6f997b40 [Follower]" leader-address= leader-id= 2026-05-25T03:42:59.229Z [INFO] server-90687b77-88d4-2fe2-d31e-18afeba0608b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49d3bc92-09f3-05f9-6598-ce763596b3cc Address:49d3bc92-09f3-05f9-6598-ce763596b3cc} {Suffrage:Voter ID:server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40 Address:cd58e1ff-3c4e-9651-2fb1-39bc6f997b40} {Suffrage:Voter ID:server-90687b77-88d4-2fe2-d31e-18afeba0608b Address:90687b77-88d4-2fe2-d31e-18afeba0608b}]" 2026-05-25T03:42:59.229Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:59.229Z [INFO] server-90687b77-88d4-2fe2-d31e-18afeba0608b: entering follower state: follower="Node at 90687b77-88d4-2fe2-d31e-18afeba0608b [Follower]" leader-address= leader-id= 2026-05-25T03:42:59.288Z [WARN] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:59.288Z [INFO] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: entering candidate state: node="Node at cd58e1ff-3c4e-9651-2fb1-39bc6f997b40 [Candidate]" term=2 2026-05-25T03:42:59.288Z [INFO] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: election won: term=2 tally=2 2026-05-25T03:42:59.288Z [INFO] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: entering leader state: leader="Node at cd58e1ff-3c4e-9651-2fb1-39bc6f997b40 [Leader]" 2026-05-25T03:42:59.288Z [INFO] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: added peer, starting replication: peer=server-49d3bc92-09f3-05f9-6598-ce763596b3cc 2026-05-25T03:42:59.289Z [INFO] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: added peer, starting replication: peer=server-90687b77-88d4-2fe2-d31e-18afeba0608b 2026-05-25T03:42:59.289Z [INFO] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: pipelining replication: peer="{Voter server-49d3bc92-09f3-05f9-6598-ce763596b3cc 49d3bc92-09f3-05f9-6598-ce763596b3cc}" 2026-05-25T03:42:59.289Z [INFO] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: pipelining replication: peer="{Voter server-90687b77-88d4-2fe2-d31e-18afeba0608b 90687b77-88d4-2fe2-d31e-18afeba0608b}" 2026-05-25T03:42:59.394Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:59.289028757 +0000 UTC m=+105.462004201 (1 nodes), 59.4204ms from start of poll, 59.92224ms from cluster start. Timeout at 2026-05-25 03:42:59.394042277 +0000 UTC m=+105.567017761, 105.01356ms after stability 2026-05-25T03:42:59.400Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:42:59.506Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:59.400771557 +0000 UTC m=+105.573747001 (2 nodes), 3.08µs from start of poll, 171.66504ms from cluster start. Timeout at 2026-05-25 03:42:59.505804237 +0000 UTC m=+105.678779681, 105.03268ms after stability 2026-05-25T03:42:59.506Z [INFO] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: updating configuration: command=RemoveServer server-id=server-49d3bc92-09f3-05f9-6598-ce763596b3cc server-addr= servers="[{Suffrage:Voter ID:server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40 Address:cd58e1ff-3c4e-9651-2fb1-39bc6f997b40} {Suffrage:Voter ID:server-90687b77-88d4-2fe2-d31e-18afeba0608b Address:90687b77-88d4-2fe2-d31e-18afeba0608b}]" 2026-05-25T03:42:59.506Z [INFO] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: removed peer, stopping replication: peer=server-49d3bc92-09f3-05f9-6598-ce763596b3cc last-index=3 2026-05-25T03:42:59.507Z [INFO] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: aborting pipeline replication: peer="{Voter server-49d3bc92-09f3-05f9-6598-ce763596b3cc 49d3bc92-09f3-05f9-6598-ce763596b3cc}" 2026-05-25T03:42:59.600Z [WARN] server-49d3bc92-09f3-05f9-6598-ce763596b3cc: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election 2026-05-25T03:42:59.616Z [ERROR] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: failed to pipeline appendEntries: peer="{Voter server-90687b77-88d4-2fe2-d31e-18afeba0608b 90687b77-88d4-2fe2-d31e-18afeba0608b}" error="append pipeline closed" 2026-05-25T03:42:59.616Z [INFO] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: aborting pipeline replication: peer="{Voter server-90687b77-88d4-2fe2-d31e-18afeba0608b 90687b77-88d4-2fe2-d31e-18afeba0608b}" 2026-05-25T03:42:59.620Z [ERROR] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: failed to heartbeat to: peer=90687b77-88d4-2fe2-d31e-18afeba0608b backoff time=10ms error="failed to connect to peer: 90687b77-88d4-2fe2-d31e-18afeba0608b" 2026-05-25T03:42:59.626Z [ERROR] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: failed to appendEntries to: peer="{Voter server-90687b77-88d4-2fe2-d31e-18afeba0608b 90687b77-88d4-2fe2-d31e-18afeba0608b}" error="failed to connect to peer: 90687b77-88d4-2fe2-d31e-18afeba0608b" 2026-05-25T03:42:59.639Z [ERROR] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: failed to heartbeat to: peer=90687b77-88d4-2fe2-d31e-18afeba0608b backoff time=10ms error="failed to connect to peer: 90687b77-88d4-2fe2-d31e-18afeba0608b" 2026-05-25T03:42:59.643Z [ERROR] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: failed to appendEntries to: peer="{Voter server-90687b77-88d4-2fe2-d31e-18afeba0608b 90687b77-88d4-2fe2-d31e-18afeba0608b}" error="failed to connect to peer: 90687b77-88d4-2fe2-d31e-18afeba0608b" 2026-05-25T03:42:59.657Z [ERROR] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: failed to heartbeat to: peer=90687b77-88d4-2fe2-d31e-18afeba0608b backoff time=10ms error="failed to connect to peer: 90687b77-88d4-2fe2-d31e-18afeba0608b" 2026-05-25T03:42:59.659Z [ERROR] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: failed to appendEntries to: peer="{Voter server-90687b77-88d4-2fe2-d31e-18afeba0608b 90687b77-88d4-2fe2-d31e-18afeba0608b}" error="failed to connect to peer: 90687b77-88d4-2fe2-d31e-18afeba0608b" 2026-05-25T03:42:59.662Z [WARN] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: failed to contact: server-id=server-90687b77-88d4-2fe2-d31e-18afeba0608b time=50.97592ms 2026-05-25T03:42:59.662Z [WARN] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: failed to contact quorum of nodes, stepping down 2026-05-25T03:42:59.662Z [INFO] server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40: entering follower state: follower="Node at cd58e1ff-3c4e-9651-2fb1-39bc6f997b40 [Follower]" leader-address= leader-id= 2026-05-25T03:42:59.693Z [WARN] server-90687b77-88d4-2fe2-d31e-18afeba0608b: heartbeat timeout reached, starting election: last-leader-addr=cd58e1ff-3c4e-9651-2fb1-39bc6f997b40 last-leader-id=server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40 2026-05-25T03:42:59.693Z [INFO] server-90687b77-88d4-2fe2-d31e-18afeba0608b: entering candidate state: node="Node at 90687b77-88d4-2fe2-d31e-18afeba0608b [Candidate]" term=3 2026-05-25T03:42:59.693Z [ERROR] server-90687b77-88d4-2fe2-d31e-18afeba0608b: failed to make requestVote RPC: target="{Voter server-cd58e1ff-3c4e-9651-2fb1-39bc6f997b40 cd58e1ff-3c4e-9651-2fb1-39bc6f997b40}" error="failed to connect to peer: cd58e1ff-3c4e-9651-2fb1-39bc6f997b40" term=3 2026-05-25T03:42:59.718Z [WARN] server-90687b77-88d4-2fe2-d31e-18afeba0608b: rejecting vote request since node is not in configuration: from=49d3bc92-09f3-05f9-6598-ce763596b3cc --- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.49s) === RUN TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin 2026-05-25T03:42:59.718Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a48f20c-21e7-2c6d-ba57-19384835115d Address:6a48f20c-21e7-2c6d-ba57-19384835115d} {Suffrage:Voter ID:server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 Address:90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6} {Suffrage:Voter ID:server-3a304c48-e433-4d71-0442-bf3126265694 Address:3a304c48-e433-4d71-0442-bf3126265694}]" 2026-05-25T03:42:59.719Z [INFO] server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a48f20c-21e7-2c6d-ba57-19384835115d Address:6a48f20c-21e7-2c6d-ba57-19384835115d} {Suffrage:Voter ID:server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 Address:90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6} {Suffrage:Voter ID:server-3a304c48-e433-4d71-0442-bf3126265694 Address:3a304c48-e433-4d71-0442-bf3126265694}]" 2026-05-25T03:42:59.719Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: entering follower state: follower="Node at 6a48f20c-21e7-2c6d-ba57-19384835115d [Follower]" leader-address= leader-id= 2026-05-25T03:42:59.719Z [INFO] server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6: entering follower state: follower="Node at 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 [Follower]" leader-address= leader-id= 2026-05-25T03:42:59.719Z [INFO] server-3a304c48-e433-4d71-0442-bf3126265694: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a48f20c-21e7-2c6d-ba57-19384835115d Address:6a48f20c-21e7-2c6d-ba57-19384835115d} {Suffrage:Voter ID:server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 Address:90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6} {Suffrage:Voter ID:server-3a304c48-e433-4d71-0442-bf3126265694 Address:3a304c48-e433-4d71-0442-bf3126265694}]" 2026-05-25T03:42:59.719Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:42:59.719Z [INFO] server-3a304c48-e433-4d71-0442-bf3126265694: entering follower state: follower="Node at 3a304c48-e433-4d71-0442-bf3126265694 [Follower]" leader-address= leader-id= 2026-05-25T03:42:59.793Z [WARN] server-6a48f20c-21e7-2c6d-ba57-19384835115d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:42:59.793Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: entering candidate state: node="Node at 6a48f20c-21e7-2c6d-ba57-19384835115d [Candidate]" term=2 2026-05-25T03:42:59.793Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: election won: term=2 tally=2 2026-05-25T03:42:59.793Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: entering leader state: leader="Node at 6a48f20c-21e7-2c6d-ba57-19384835115d [Leader]" 2026-05-25T03:42:59.793Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: added peer, starting replication: peer=server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 2026-05-25T03:42:59.793Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: added peer, starting replication: peer=server-3a304c48-e433-4d71-0442-bf3126265694 2026-05-25T03:42:59.793Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: pipelining replication: peer="{Voter server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}" 2026-05-25T03:42:59.793Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: pipelining replication: peer="{Voter server-3a304c48-e433-4d71-0442-bf3126265694 3a304c48-e433-4d71-0442-bf3126265694}" 2026-05-25T03:42:59.899Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:42:59.793513117 +0000 UTC m=+105.966488561 (1 nodes), 74.27312ms from start of poll, 74.71608ms from cluster start. Timeout at 2026-05-25 03:42:59.898596077 +0000 UTC m=+106.071571521, 105.08296ms after stability 2026-05-25T03:42:59.904Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:43:00.009Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:42:59.904197277 +0000 UTC m=+106.077172721 (2 nodes), 1.32µs from start of poll, 185.40024ms from cluster start. Timeout at 2026-05-25 03:43:00.009221637 +0000 UTC m=+106.182197121, 105.0244ms after stability 2026-05-25T03:43:00.011Z [ERROR] server-6a48f20c-21e7-2c6d-ba57-19384835115d: failed to heartbeat to: peer=90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 backoff time=10ms error="failed to connect to peer: 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6" 2026-05-25T03:43:00.011Z [ERROR] server-6a48f20c-21e7-2c6d-ba57-19384835115d: failed to pipeline appendEntries: peer="{Voter server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}" error="append pipeline closed" 2026-05-25T03:43:00.011Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: aborting pipeline replication: peer="{Voter server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}" 2026-05-25T03:43:00.019Z [ERROR] server-6a48f20c-21e7-2c6d-ba57-19384835115d: failed to appendEntries to: peer="{Voter server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}" error="failed to connect to peer: 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6" 2026-05-25T03:43:00.032Z [ERROR] server-6a48f20c-21e7-2c6d-ba57-19384835115d: failed to heartbeat to: peer=90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 backoff time=10ms error="failed to connect to peer: 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6" 2026-05-25T03:43:00.037Z [ERROR] server-6a48f20c-21e7-2c6d-ba57-19384835115d: failed to appendEntries to: peer="{Voter server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}" error="failed to connect to peer: 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6" 2026-05-25T03:43:00.050Z [ERROR] server-6a48f20c-21e7-2c6d-ba57-19384835115d: failed to heartbeat to: peer=90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 backoff time=10ms error="failed to connect to peer: 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6" 2026-05-25T03:43:00.053Z [WARN] server-6a48f20c-21e7-2c6d-ba57-19384835115d: failed to contact: server-id=server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 time=50.87388ms 2026-05-25T03:43:00.054Z [ERROR] server-6a48f20c-21e7-2c6d-ba57-19384835115d: failed to appendEntries to: peer="{Voter server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}" error="failed to connect to peer: 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6" 2026-05-25T03:43:00.070Z [ERROR] server-6a48f20c-21e7-2c6d-ba57-19384835115d: failed to heartbeat to: peer=90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 backoff time=20ms error="failed to connect to peer: 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6" 2026-05-25T03:43:00.083Z [ERROR] server-6a48f20c-21e7-2c6d-ba57-19384835115d: failed to appendEntries to: peer="{Voter server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}" error="failed to connect to peer: 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6" 2026-05-25T03:43:00.097Z [ERROR] server-6a48f20c-21e7-2c6d-ba57-19384835115d: failed to heartbeat to: peer=90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 backoff time=25ms error="failed to connect to peer: 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6" 2026-05-25T03:43:00.103Z [WARN] server-6a48f20c-21e7-2c6d-ba57-19384835115d: failed to contact: server-id=server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 time=100.72868ms 2026-05-25T03:43:00.107Z [WARN] server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6: heartbeat timeout reached, starting election: last-leader-addr=6a48f20c-21e7-2c6d-ba57-19384835115d last-leader-id=server-6a48f20c-21e7-2c6d-ba57-19384835115d 2026-05-25T03:43:00.107Z [INFO] server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6: entering candidate state: node="Node at 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 [Candidate]" term=3 2026-05-25T03:43:00.107Z [ERROR] server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6: failed to make requestVote RPC: target="{Voter server-6a48f20c-21e7-2c6d-ba57-19384835115d 6a48f20c-21e7-2c6d-ba57-19384835115d}" error="failed to connect to peer: 6a48f20c-21e7-2c6d-ba57-19384835115d" term=3 2026-05-25T03:43:00.107Z [ERROR] server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6: failed to make requestVote RPC: target="{Voter server-3a304c48-e433-4d71-0442-bf3126265694 3a304c48-e433-4d71-0442-bf3126265694}" error="failed to connect to peer: 3a304c48-e433-4d71-0442-bf3126265694" term=3 2026-05-25T03:43:00.115Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: updating configuration: command=RemoveServer server-id=server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 server-addr= servers="[{Suffrage:Voter ID:server-6a48f20c-21e7-2c6d-ba57-19384835115d Address:6a48f20c-21e7-2c6d-ba57-19384835115d} {Suffrage:Voter ID:server-3a304c48-e433-4d71-0442-bf3126265694 Address:3a304c48-e433-4d71-0442-bf3126265694}]" 2026-05-25T03:43:00.115Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: removed peer, stopping replication: peer=server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 last-index=3 2026-05-25T03:43:00.115Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: updating configuration: command=AddNonvoter server-id=server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 server-addr=90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 servers="[{Suffrage:Voter ID:server-6a48f20c-21e7-2c6d-ba57-19384835115d Address:6a48f20c-21e7-2c6d-ba57-19384835115d} {Suffrage:Voter ID:server-3a304c48-e433-4d71-0442-bf3126265694 Address:3a304c48-e433-4d71-0442-bf3126265694} {Suffrage:Nonvoter ID:server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 Address:90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}]" 2026-05-25T03:43:00.115Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: added peer, starting replication: peer=server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 2026-05-25T03:43:00.115Z [ERROR] server-6a48f20c-21e7-2c6d-ba57-19384835115d: failed to appendEntries to: peer="{Nonvoter server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}" error="failed to connect to peer: 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6" 2026-05-25T03:43:00.133Z [ERROR] server-6a48f20c-21e7-2c6d-ba57-19384835115d: peer has newer term, stopping replication: peer="{Voter server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}" 2026-05-25T03:43:00.133Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: entering follower state: follower="Node at 6a48f20c-21e7-2c6d-ba57-19384835115d [Follower]" leader-address= leader-id= 2026-05-25T03:43:00.133Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: aborting pipeline replication: peer="{Voter server-3a304c48-e433-4d71-0442-bf3126265694 3a304c48-e433-4d71-0442-bf3126265694}" 2026-05-25T03:43:00.133Z [ERROR] server-6a48f20c-21e7-2c6d-ba57-19384835115d: peer has newer term, stopping replication: peer="{Nonvoter server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}" 2026-05-25T03:43:00.200Z [WARN] server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6: Election timeout reached, restarting election 2026-05-25T03:43:00.200Z [INFO] server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6: entering candidate state: node="Node at 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 [Candidate]" term=23 2026-05-25T03:43:00.200Z [WARN] server-3a304c48-e433-4d71-0442-bf3126265694: rejecting vote request since we have a leader: from=90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 leader=6a48f20c-21e7-2c6d-ba57-19384835115d leader-id=server-6a48f20c-21e7-2c6d-ba57-19384835115d 2026-05-25T03:43:00.200Z [WARN] server-6a48f20c-21e7-2c6d-ba57-19384835115d: rejecting vote request since node is not a voter: from=90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 2026-05-25T03:43:00.207Z [WARN] server-3a304c48-e433-4d71-0442-bf3126265694: heartbeat timeout reached, starting election: last-leader-addr=6a48f20c-21e7-2c6d-ba57-19384835115d last-leader-id=server-6a48f20c-21e7-2c6d-ba57-19384835115d 2026-05-25T03:43:00.207Z [INFO] server-3a304c48-e433-4d71-0442-bf3126265694: entering candidate state: node="Node at 3a304c48-e433-4d71-0442-bf3126265694 [Candidate]" term=3 2026-05-25T03:43:00.208Z [INFO] server-3a304c48-e433-4d71-0442-bf3126265694: entering follower state: follower="Node at 3a304c48-e433-4d71-0442-bf3126265694 [Follower]" leader-address= leader-id= 2026-05-25T03:43:00.221Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:43:00.225Z [WARN] server-6a48f20c-21e7-2c6d-ba57-19384835115d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:43:00.225Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: entering candidate state: node="Node at 6a48f20c-21e7-2c6d-ba57-19384835115d [Candidate]" term=24 2026-05-25T03:43:00.225Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: election won: term=24 tally=2 2026-05-25T03:43:00.225Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: entering leader state: leader="Node at 6a48f20c-21e7-2c6d-ba57-19384835115d [Leader]" 2026-05-25T03:43:00.225Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: added peer, starting replication: peer=server-3a304c48-e433-4d71-0442-bf3126265694 2026-05-25T03:43:00.225Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: added peer, starting replication: peer=server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 2026-05-25T03:43:00.225Z [WARN] server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6: failed to get previous log: previous-index=4 last-index=2 error="log not found" 2026-05-25T03:43:00.225Z [INFO] server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6: entering follower state: follower="Node at 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 [Follower]" leader-address=6a48f20c-21e7-2c6d-ba57-19384835115d leader-id=server-6a48f20c-21e7-2c6d-ba57-19384835115d 2026-05-25T03:43:00.225Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: pipelining replication: peer="{Voter server-3a304c48-e433-4d71-0442-bf3126265694 3a304c48-e433-4d71-0442-bf3126265694}" 2026-05-25T03:43:00.225Z [WARN] server-6a48f20c-21e7-2c6d-ba57-19384835115d: appendEntries rejected, sending older logs: peer="{Nonvoter server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}" next=3 2026-05-25T03:43:00.225Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: pipelining replication: peer="{Nonvoter server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}" 2026-05-25T03:43:00.331Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:43:00.225764357 +0000 UTC m=+106.398739801 (1 nodes), 4.19324ms from start of poll, 506.96732ms from cluster start. Timeout at 2026-05-25 03:43:00.330847197 +0000 UTC m=+106.503822641, 105.08284ms after stability 2026-05-25T03:43:00.357Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: updating configuration: command=RemoveServer server-id=server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 server-addr= servers="[{Suffrage:Voter ID:server-6a48f20c-21e7-2c6d-ba57-19384835115d Address:6a48f20c-21e7-2c6d-ba57-19384835115d} {Suffrage:Voter ID:server-3a304c48-e433-4d71-0442-bf3126265694 Address:3a304c48-e433-4d71-0442-bf3126265694}]" 2026-05-25T03:43:00.357Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: removed peer, stopping replication: peer=server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 last-index=106 2026-05-25T03:43:00.357Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: aborting pipeline replication: peer="{Nonvoter server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}" 2026-05-25T03:43:00.357Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: updating configuration: command=AddVoter server-id=server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 server-addr=90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 servers="[{Suffrage:Voter ID:server-6a48f20c-21e7-2c6d-ba57-19384835115d Address:6a48f20c-21e7-2c6d-ba57-19384835115d} {Suffrage:Voter ID:server-3a304c48-e433-4d71-0442-bf3126265694 Address:3a304c48-e433-4d71-0442-bf3126265694} {Suffrage:Voter ID:server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 Address:90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}]" 2026-05-25T03:43:00.357Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: added peer, starting replication: peer=server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 2026-05-25T03:43:00.357Z [WARN] server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6: failed to get previous log: previous-index=107 last-index=106 error="log not found" 2026-05-25T03:43:00.357Z [WARN] server-6a48f20c-21e7-2c6d-ba57-19384835115d: appendEntries rejected, sending older logs: peer="{Voter server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}" next=107 2026-05-25T03:43:00.358Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: pipelining replication: peer="{Voter server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}" 2026-05-25T03:43:01.434Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:43:01.540Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:43:01.434176117 +0000 UTC m=+107.607151561 (1 nodes), 680ns from start of poll, 1.71537908s from cluster start. Timeout at 2026-05-25 03:43:01.539199557 +0000 UTC m=+107.712175001, 105.02344ms after stability 2026-05-25T03:43:01.540Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: aborting pipeline replication: peer="{Voter server-90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6 90d76126-04f2-3bc1-f3f1-69f5fe5ab1d6}" 2026-05-25T03:43:01.540Z [INFO] server-6a48f20c-21e7-2c6d-ba57-19384835115d: aborting pipeline replication: peer="{Voter server-3a304c48-e433-4d71-0442-bf3126265694 3a304c48-e433-4d71-0442-bf3126265694}" --- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.82s) === RUN TestRaft_FollowerRemovalNoElection 2026-05-25T03:43:01.541Z [INFO] server-c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab Address:c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab} {Suffrage:Voter ID:server-6511f649-9e1e-7944-22ad-38a84673099c Address:6511f649-9e1e-7944-22ad-38a84673099c} {Suffrage:Voter ID:server-ddc124a2-4c5e-9b8a-3032-a173a7d32330 Address:ddc124a2-4c5e-9b8a-3032-a173a7d32330}]" 2026-05-25T03:43:01.541Z [INFO] server-6511f649-9e1e-7944-22ad-38a84673099c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab Address:c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab} {Suffrage:Voter ID:server-6511f649-9e1e-7944-22ad-38a84673099c Address:6511f649-9e1e-7944-22ad-38a84673099c} {Suffrage:Voter ID:server-ddc124a2-4c5e-9b8a-3032-a173a7d32330 Address:ddc124a2-4c5e-9b8a-3032-a173a7d32330}]" 2026-05-25T03:43:01.541Z [INFO] server-c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab: entering follower state: follower="Node at c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab [Follower]" leader-address= leader-id= 2026-05-25T03:43:01.541Z [INFO] server-ddc124a2-4c5e-9b8a-3032-a173a7d32330: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab Address:c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab} {Suffrage:Voter ID:server-6511f649-9e1e-7944-22ad-38a84673099c Address:6511f649-9e1e-7944-22ad-38a84673099c} {Suffrage:Voter ID:server-ddc124a2-4c5e-9b8a-3032-a173a7d32330 Address:ddc124a2-4c5e-9b8a-3032-a173a7d32330}]" 2026-05-25T03:43:01.542Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:43:01.542Z [INFO] server-6511f649-9e1e-7944-22ad-38a84673099c: entering follower state: follower="Node at 6511f649-9e1e-7944-22ad-38a84673099c [Follower]" leader-address= leader-id= 2026-05-25T03:43:01.542Z [INFO] server-ddc124a2-4c5e-9b8a-3032-a173a7d32330: entering follower state: follower="Node at ddc124a2-4c5e-9b8a-3032-a173a7d32330 [Follower]" leader-address= leader-id= 2026-05-25T03:43:01.721Z [WARN] server-6511f649-9e1e-7944-22ad-38a84673099c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:43:01.721Z [INFO] server-6511f649-9e1e-7944-22ad-38a84673099c: entering candidate state: node="Node at 6511f649-9e1e-7944-22ad-38a84673099c [Candidate]" term=2 2026-05-25T03:43:01.721Z [INFO] server-6511f649-9e1e-7944-22ad-38a84673099c: election won: term=2 tally=2 2026-05-25T03:43:01.721Z [INFO] server-6511f649-9e1e-7944-22ad-38a84673099c: entering leader state: leader="Node at 6511f649-9e1e-7944-22ad-38a84673099c [Leader]" 2026-05-25T03:43:01.721Z [INFO] server-6511f649-9e1e-7944-22ad-38a84673099c: added peer, starting replication: peer=server-c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab 2026-05-25T03:43:01.721Z [INFO] server-6511f649-9e1e-7944-22ad-38a84673099c: added peer, starting replication: peer=server-ddc124a2-4c5e-9b8a-3032-a173a7d32330 2026-05-25T03:43:01.721Z [INFO] server-6511f649-9e1e-7944-22ad-38a84673099c: pipelining replication: peer="{Voter server-ddc124a2-4c5e-9b8a-3032-a173a7d32330 ddc124a2-4c5e-9b8a-3032-a173a7d32330}" 2026-05-25T03:43:01.721Z [INFO] server-6511f649-9e1e-7944-22ad-38a84673099c: pipelining replication: peer="{Voter server-c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab}" 2026-05-25T03:43:01.927Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:43:01.721781357 +0000 UTC m=+107.894756801 (1 nodes), 179.71824ms from start of poll, 180.69024ms from cluster start. Timeout at 2026-05-25 03:43:01.926797357 +0000 UTC m=+108.099772801, 205.016ms after stability 2026-05-25T03:43:01.927Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:43:02.133Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:43:01.927817717 +0000 UTC m=+108.100793201 (1 nodes), 1.44µs from start of poll, 386.72664ms from cluster start. Timeout at 2026-05-25 03:43:02.132848277 +0000 UTC m=+108.305823721, 205.03052ms after stability 2026-05-25T03:43:02.138Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:43:02.343Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:43:02.138242277 +0000 UTC m=+108.311217721 (2 nodes), 4µs from start of poll, 597.15116ms from cluster start. Timeout at 2026-05-25 03:43:02.343266237 +0000 UTC m=+108.516241681, 205.02396ms after stability 2026-05-25T03:43:02.344Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:43:02.549Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:43:02.344037837 +0000 UTC m=+108.517013281 (1 nodes), 2µs from start of poll, 802.94672ms from cluster start. Timeout at 2026-05-25 03:43:02.549068957 +0000 UTC m=+108.722044401, 205.03112ms after stability raft_test.go:2961: [INFO] restarting Node at c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab [Follower] 2026-05-25T03:43:02.549Z [INFO] server-c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab Address:c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab} {Suffrage:Voter ID:server-6511f649-9e1e-7944-22ad-38a84673099c Address:6511f649-9e1e-7944-22ad-38a84673099c} {Suffrage:Voter ID:server-ddc124a2-4c5e-9b8a-3032-a173a7d32330 Address:ddc124a2-4c5e-9b8a-3032-a173a7d32330}]" 2026-05-25T03:43:02.549Z [INFO] server-c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab: entering follower state: follower="Node at c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab [Follower]" leader-address= leader-id= 2026-05-25T03:43:02.799Z [INFO] server-6511f649-9e1e-7944-22ad-38a84673099c: aborting pipeline replication: peer="{Voter server-ddc124a2-4c5e-9b8a-3032-a173a7d32330 ddc124a2-4c5e-9b8a-3032-a173a7d32330}" 2026-05-25T03:43:03.177Z [ERROR] server-6511f649-9e1e-7944-22ad-38a84673099c: failed to pipeline appendEntries: peer="{Voter server-c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab}" error="command enqueue timeout" 2026-05-25T03:43:03.177Z [INFO] server-6511f649-9e1e-7944-22ad-38a84673099c: aborting pipeline replication: peer="{Voter server-c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab}" 2026-05-25T03:43:03.307Z [ERROR] server-6511f649-9e1e-7944-22ad-38a84673099c: failed to heartbeat to: peer=c5ceee00-7ad0-ffd2-d6dd-6bbb554cdcab backoff time=10ms error="command timed out" --- PASS: TestRaft_FollowerRemovalNoElection (1.77s) === RUN TestRaft_VoteWithNoIDNoAddr 2026-05-25T03:43:03.308Z [INFO] server-d72a2f09-6073-87f4-5db5-22bbd8318100: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d72a2f09-6073-87f4-5db5-22bbd8318100 Address:d72a2f09-6073-87f4-5db5-22bbd8318100} {Suffrage:Voter ID:server-5228102d-3a31-559b-36c7-05c611a98cbc Address:5228102d-3a31-559b-36c7-05c611a98cbc} {Suffrage:Voter ID:server-e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81 Address:e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81}]" 2026-05-25T03:43:03.308Z [INFO] server-5228102d-3a31-559b-36c7-05c611a98cbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d72a2f09-6073-87f4-5db5-22bbd8318100 Address:d72a2f09-6073-87f4-5db5-22bbd8318100} {Suffrage:Voter ID:server-5228102d-3a31-559b-36c7-05c611a98cbc Address:5228102d-3a31-559b-36c7-05c611a98cbc} {Suffrage:Voter ID:server-e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81 Address:e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81}]" 2026-05-25T03:43:03.308Z [INFO] server-d72a2f09-6073-87f4-5db5-22bbd8318100: entering follower state: follower="Node at d72a2f09-6073-87f4-5db5-22bbd8318100 [Follower]" leader-address= leader-id= 2026-05-25T03:43:03.309Z [INFO] server-e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d72a2f09-6073-87f4-5db5-22bbd8318100 Address:d72a2f09-6073-87f4-5db5-22bbd8318100} {Suffrage:Voter ID:server-5228102d-3a31-559b-36c7-05c611a98cbc Address:5228102d-3a31-559b-36c7-05c611a98cbc} {Suffrage:Voter ID:server-e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81 Address:e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81}]" 2026-05-25T03:43:03.309Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:43:03.309Z [INFO] server-5228102d-3a31-559b-36c7-05c611a98cbc: entering follower state: follower="Node at 5228102d-3a31-559b-36c7-05c611a98cbc [Follower]" leader-address= leader-id= 2026-05-25T03:43:03.309Z [INFO] server-e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81: entering follower state: follower="Node at e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81 [Follower]" leader-address= leader-id= 2026-05-25T03:43:03.363Z [WARN] server-d72a2f09-6073-87f4-5db5-22bbd8318100: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= 2026-05-25T03:43:03.363Z [INFO] server-d72a2f09-6073-87f4-5db5-22bbd8318100: entering candidate state: node="Node at d72a2f09-6073-87f4-5db5-22bbd8318100 [Candidate]" term=2 2026-05-25T03:43:03.363Z [INFO] server-d72a2f09-6073-87f4-5db5-22bbd8318100: election won: term=2 tally=2 2026-05-25T03:43:03.363Z [INFO] server-d72a2f09-6073-87f4-5db5-22bbd8318100: entering leader state: leader="Node at d72a2f09-6073-87f4-5db5-22bbd8318100 [Leader]" 2026-05-25T03:43:03.363Z [INFO] server-d72a2f09-6073-87f4-5db5-22bbd8318100: added peer, starting replication: peer=server-5228102d-3a31-559b-36c7-05c611a98cbc 2026-05-25T03:43:03.363Z [INFO] server-d72a2f09-6073-87f4-5db5-22bbd8318100: added peer, starting replication: peer=server-e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81 2026-05-25T03:43:03.363Z [INFO] server-d72a2f09-6073-87f4-5db5-22bbd8318100: pipelining replication: peer="{Voter server-5228102d-3a31-559b-36c7-05c611a98cbc 5228102d-3a31-559b-36c7-05c611a98cbc}" 2026-05-25T03:43:03.363Z [INFO] server-d72a2f09-6073-87f4-5db5-22bbd8318100: pipelining replication: peer="{Voter server-e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81 e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81}" 2026-05-25T03:43:03.468Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:43:03.363290237 +0000 UTC m=+109.536265681 (1 nodes), 54.23048ms from start of poll, 54.6786ms from cluster start. Timeout at 2026-05-25 03:43:03.468294437 +0000 UTC m=+109.641269881, 105.0042ms after stability 2026-05-25T03:43:03.468Z [INFO] cluster: starting stability test: raft-state=Leader 2026-05-25T03:43:03.574Z [INFO] cluster: stable state for Leader reached at 2026-05-25 03:43:03.468539437 +0000 UTC m=+109.641514881 (1 nodes), 2.44µs from start of poll, 159.9278ms from cluster start. Timeout at 2026-05-25 03:43:03.573566197 +0000 UTC m=+109.746541641, 105.02676ms after stability 2026-05-25T03:43:03.579Z [INFO] cluster: starting stability test: raft-state=Follower 2026-05-25T03:43:03.684Z [INFO] cluster: stable state for Follower reached at 2026-05-25 03:43:03.579545797 +0000 UTC m=+109.752521201 (2 nodes), 2.84µs from start of poll, 270.93412ms from cluster start. Timeout at 2026-05-25 03:43:03.684579837 +0000 UTC m=+109.857555281, 105.03408ms after stability 2026-05-25T03:43:03.685Z [ERROR] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to pipeline appendEntries: peer="{Voter server-5228102d-3a31-559b-36c7-05c611a98cbc 5228102d-3a31-559b-36c7-05c611a98cbc}" error="append pipeline closed" 2026-05-25T03:43:03.686Z [INFO] server-d72a2f09-6073-87f4-5db5-22bbd8318100: aborting pipeline replication: peer="{Voter server-5228102d-3a31-559b-36c7-05c611a98cbc 5228102d-3a31-559b-36c7-05c611a98cbc}" 2026-05-25T03:43:03.691Z [ERROR] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to pipeline appendEntries: peer="{Voter server-e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81 e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81}" error="append pipeline closed" 2026-05-25T03:43:03.691Z [INFO] server-d72a2f09-6073-87f4-5db5-22bbd8318100: aborting pipeline replication: peer="{Voter server-e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81 e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81}" 2026-05-25T03:43:03.692Z [ERROR] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to appendEntries to: peer="{Voter server-5228102d-3a31-559b-36c7-05c611a98cbc 5228102d-3a31-559b-36c7-05c611a98cbc}" error="failed to connect to peer: 5228102d-3a31-559b-36c7-05c611a98cbc" 2026-05-25T03:43:03.693Z [ERROR] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to heartbeat to: peer=5228102d-3a31-559b-36c7-05c611a98cbc backoff time=10ms error="failed to connect to peer: 5228102d-3a31-559b-36c7-05c611a98cbc" 2026-05-25T03:43:03.693Z [ERROR] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to heartbeat to: peer=e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81 backoff time=10ms error="failed to connect to peer: e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81" 2026-05-25T03:43:03.701Z [ERROR] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to appendEntries to: peer="{Voter server-e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81 e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81}" error="failed to connect to peer: e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81" 2026-05-25T03:43:03.711Z [ERROR] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to appendEntries to: peer="{Voter server-5228102d-3a31-559b-36c7-05c611a98cbc 5228102d-3a31-559b-36c7-05c611a98cbc}" error="failed to connect to peer: 5228102d-3a31-559b-36c7-05c611a98cbc" 2026-05-25T03:43:03.711Z [ERROR] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to heartbeat to: peer=e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81 backoff time=10ms error="failed to connect to peer: e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81" 2026-05-25T03:43:03.711Z [ERROR] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to heartbeat to: peer=5228102d-3a31-559b-36c7-05c611a98cbc backoff time=10ms error="failed to connect to peer: 5228102d-3a31-559b-36c7-05c611a98cbc" 2026-05-25T03:43:03.719Z [ERROR] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to appendEntries to: peer="{Voter server-e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81 e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81}" error="failed to connect to peer: e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81" 2026-05-25T03:43:03.729Z [ERROR] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to heartbeat to: peer=e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81 backoff time=10ms error="failed to connect to peer: e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81" 2026-05-25T03:43:03.729Z [ERROR] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to appendEntries to: peer="{Voter server-5228102d-3a31-559b-36c7-05c611a98cbc 5228102d-3a31-559b-36c7-05c611a98cbc}" error="failed to connect to peer: 5228102d-3a31-559b-36c7-05c611a98cbc" 2026-05-25T03:43:03.730Z [ERROR] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to heartbeat to: peer=5228102d-3a31-559b-36c7-05c611a98cbc backoff time=10ms error="failed to connect to peer: 5228102d-3a31-559b-36c7-05c611a98cbc" 2026-05-25T03:43:03.733Z [WARN] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to contact: server-id=server-e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81 time=50.07932ms 2026-05-25T03:43:03.739Z [ERROR] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to appendEntries to: peer="{Voter server-e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81 e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81}" error="failed to connect to peer: e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81" 2026-05-25T03:43:03.743Z [WARN] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to contact: server-id=server-5228102d-3a31-559b-36c7-05c611a98cbc time=60.4106ms 2026-05-25T03:43:03.744Z [WARN] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to contact: server-id=server-e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81 time=60.64992ms 2026-05-25T03:43:03.744Z [WARN] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to contact quorum of nodes, stepping down 2026-05-25T03:43:03.744Z [INFO] server-d72a2f09-6073-87f4-5db5-22bbd8318100: entering follower state: follower="Node at d72a2f09-6073-87f4-5db5-22bbd8318100 [Follower]" leader-address= leader-id= 2026-05-25T03:43:03.747Z [WARN] server-5228102d-3a31-559b-36c7-05c611a98cbc: heartbeat timeout reached, starting election: last-leader-addr=d72a2f09-6073-87f4-5db5-22bbd8318100 last-leader-id=server-d72a2f09-6073-87f4-5db5-22bbd8318100 2026-05-25T03:43:03.747Z [INFO] server-5228102d-3a31-559b-36c7-05c611a98cbc: entering candidate state: node="Node at 5228102d-3a31-559b-36c7-05c611a98cbc [Candidate]" term=3 2026-05-25T03:43:03.747Z [WARN] server-e0a1aaab-5b74-0f94-4c6d-f0e11e3e6d81: rejecting vote request since we have a leader: from=5228102d-3a31-559b-36c7-05c611a98cbc leader=d72a2f09-6073-87f4-5db5-22bbd8318100 leader-id=server-d72a2f09-6073-87f4-5db5-22bbd8318100 2026-05-25T03:43:03.747Z [ERROR] server-5228102d-3a31-559b-36c7-05c611a98cbc: failed to make requestVote RPC: target="{Voter server-d72a2f09-6073-87f4-5db5-22bbd8318100 d72a2f09-6073-87f4-5db5-22bbd8318100}" error="failed to connect to peer: d72a2f09-6073-87f4-5db5-22bbd8318100" term=3 2026-05-25T03:43:03.749Z [ERROR] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to heartbeat to: peer=5228102d-3a31-559b-36c7-05c611a98cbc backoff time=20ms error="failed to connect to peer: 5228102d-3a31-559b-36c7-05c611a98cbc" 2026-05-25T03:43:03.756Z [ERROR] server-d72a2f09-6073-87f4-5db5-22bbd8318100: failed to appendEntries to: peer="{Voter server-5228102d-3a31-559b-36c7-05c611a98cbc 5228102d-3a31-559b-36c7-05c611a98cbc}" error="failed to connect to peer: 5228102d-3a31-559b-36c7-05c611a98cbc" 2026-05-25T03:43:03.790Z [INFO] server-5228102d-3a31-559b-36c7-05c611a98cbc: entering follower state: follower="Node at 5228102d-3a31-559b-36c7-05c611a98cbc [Follower]" leader-address= leader-id= --- PASS: TestRaft_VoteWithNoIDNoAddr (0.48s) === RUN TestRaft_runFollower_State_Transition === RUN TestRaft_runFollower_State_Transition/NonVoter 2026-05-25T03:43:03.791Z [INFO] 127.0.0.1:40569 :: starting node: addr=127.0.0.1:40569 2026-05-25T03:43:03.791Z [INFO] 127.0.0.1:40569 :: initial configuration: index=0 servers=[] 2026-05-25T03:43:03.791Z [INFO] 127.0.0.1:40569 :: entering follower state: follower="Node at 127.0.0.1:40569 [Follower]" leader-address= leader-id= 2026-05-25T03:43:03.841Z [WARN] 127.0.0.1:40569 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election === RUN TestRaft_runFollower_State_Transition/Voter 2026-05-25T03:43:03.942Z [INFO] 127.0.0.1:45335 :: starting node: addr=127.0.0.1:45335 2026-05-25T03:43:03.942Z [INFO] 127.0.0.1:45335 :: initial configuration: index=0 servers=[] 2026-05-25T03:43:03.942Z [INFO] 127.0.0.1:45335 :: entering follower state: follower="Node at 127.0.0.1:45335 [Follower]" leader-address= leader-id= 2026-05-25T03:43:04.041Z [WARN] 127.0.0.1:45335 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= === RUN TestRaft_runFollower_State_Transition/Not_in_Config 2026-05-25T03:43:04.093Z [INFO] 127.0.0.1:37221 :: starting node: addr=127.0.0.1:37221 2026-05-25T03:43:04.093Z [INFO] 127.0.0.1:37221 :: initial configuration: index=0 servers=[] 2026-05-25T03:43:04.093Z [INFO] 127.0.0.1:37221 :: entering follower state: follower="Node at 127.0.0.1:37221 [Follower]" leader-address= leader-id= 2026-05-25T03:43:04.161Z [WARN] 127.0.0.1:37221 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election --- PASS: TestRaft_runFollower_State_Transition (0.45s) --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s) --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s) === RUN TestRaft_runFollower_ReloadTimeoutConfigs 2026-05-25T03:43:04.243Z [INFO] 127.0.0.1:46067 :: starting node: addr=127.0.0.1:46067 2026-05-25T03:43:04.244Z [INFO] 127.0.0.1:46067 :: initial configuration: index=0 servers=[] 2026-05-25T03:43:04.244Z [INFO] 127.0.0.1:46067 :: entering follower state: follower="Node at 127.0.0.1:46067 [Follower]" leader-address= leader-id= 2026-05-25T03:43:04.244Z [WARN] 127.0.0.1:46067 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id= --- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s) === RUN TestSaturationMetric === RUN TestSaturationMetric/without_smoothing --- PASS: TestSaturationMetric (0.00s) --- PASS: TestSaturationMetric/without_smoothing (0.00s) === RUN TestSaturationMetric_IncorrectUsage === RUN TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_consecutively === RUN TestSaturationMetric_IncorrectUsage/calling_working()_first --- PASS: TestSaturationMetric_IncorrectUsage (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s) --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s) === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_EmptyAddr --- PASS: TestTCPTransport_EmptyAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) === RUN TestOverrideNotifyBool --- PASS: TestOverrideNotifyBool (0.00s) PASS ok github.com/hashicorp/raft 110.588s ? github.com/hashicorp/raft/bench [no test files] make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0' create-stamp debian/debhelper-build-stamp dh_testroot -O--buildsystem=golang dh_prep -O--buildsystem=golang dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang dh_installdocs -O--buildsystem=golang dh_installchangelogs -O--buildsystem=golang dh_installsystemduser -O--buildsystem=golang dh_perl -O--buildsystem=golang dh_link -O--buildsystem=golang dh_strip_nondeterminism -O--buildsystem=golang dh_compress -O--buildsystem=golang dh_fixperms -O--buildsystem=golang dh_missing -O--buildsystem=golang dh_installdeb -O--buildsystem=golang dh_golang -O--buildsystem=golang dh_gencontrol -O--buildsystem=golang dh_md5sums -O--buildsystem=golang dh_builddeb -O--buildsystem=golang dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'. dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_arm64.buildinfo dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_arm64.changes dpkg-genchanges: info: binary-only upload (no source code included) dpkg-source --after-build . dpkg-buildpackage: info: binary-only upload (no source included) dpkg-genchanges: info: including full source code in upload I: copying local configuration I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: cleaning the build env I: removing directory /srv/workspace/pbuilder/3406684 and its subdirectories I: Current time: Sun May 24 15:43:10 -12 2026 I: pbuilder-time-stamp: 1779680590