I: pbuilder: network access will be disabled during build
I: Current time: Mon May 25 11:20:48 -12 2026
I: pbuilder-time-stamp: 1779751248
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/trixie-reproducible-base.tgz]
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: policy-rc.d already exists
I: Copying source file
I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc]
I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz]
I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz]
I: Extracting source
dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: unsupported subcommand
dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying 0001-Skip-unreliable-tests.patch
I: Not using root during the build.
I: Installing the build-deps
I: user script /srv/workspace/pbuilder/3759238/tmp/hooks/D02_print_environment starting
I: set
  BUILDDIR='/build/reproducible-path'
  BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other'
  BUILDUSERNAME='pbuilder1'
  BUILD_ARCH='amd64'
  DEBIAN_FRONTEND='noninteractive'
  DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=42 '
  DISTRIBUTION='trixie'
  HOME='/root'
  HOST_ARCH='amd64'
  IFS=' 	
  '
  INVOCATION_ID='0f0a5401214b42b48868cc7121423556'
  LANG='C'
  LANGUAGE='en_US:en'
  LC_ALL='C'
  MAIL='/var/mail/root'
  OPTIND='1'
  PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games'
  PBCURRENTCOMMANDLINEOPERATION='build'
  PBUILDER_OPERATION='build'
  PBUILDER_PKGDATADIR='/usr/share/pbuilder'
  PBUILDER_PKGLIBDIR='/usr/lib/pbuilder'
  PBUILDER_SYSCONFDIR='/etc'
  PPID='3759238'
  PS1='# '
  PS2='> '
  PS4='+ '
  PWD='/'
  SHELL='/bin/bash'
  SHLVL='2'
  SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.aPNxpS2z/pbuilderrc_6zSL --distribution trixie --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/trixie-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.aPNxpS2z/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc'
  SUDO_GID='111'
  SUDO_UID='106'
  SUDO_USER='jenkins'
  TERM='unknown'
  TZ='/usr/share/zoneinfo/Etc/GMT+12'
  USER='root'
  _='/usr/bin/systemd-run'
  http_proxy='http://213.165.73.152:3128'
I: uname -a
  Linux ionos15-amd64 6.12.12+bpo-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.12-1~bpo12+1 (2025-02-23) x86_64 GNU/Linux
I: ls -l /bin
  lrwxrwxrwx 1 root root 7 Mar  4  2025 /bin -> usr/bin
I: user script /srv/workspace/pbuilder/3759238/tmp/hooks/D02_print_environment finished
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: amd64
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 19840 files and directories currently installed.)
Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ...
Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however:
  Package debhelper-compat is not installed.
 pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however:
  Package dh-sequence-golang is not installed.
 pbuilder-satisfydepends-dummy depends on golang-any; however:
  Package golang-any is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however:
  Package golang-github-armon-go-metrics-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however:
  Package golang-github-hashicorp-go-hclog-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however:
  Package golang-github-hashicorp-go-msgpack-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however:
  Package golang-github-stretchr-testify-dev is not installed.

Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
Building tag database...
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
The following NEW packages will be installed:
  autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.24-go{a} golang-1.24-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-prometheus-sigv4-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libunistring5{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} 
The following packages are RECOMMENDED but will NOT be installed:
  curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 
0 packages upgraded, 106 newly installed, 0 to remove and 0 not upgraded.
Need to get 122 MB of archives. After unpacking 680 MB will be used.
Writing extended state information...
Get: 1 http://deb.debian.org/debian trixie/main amd64 sensible-utils all 0.0.24 [24.8 kB]
Get: 2 http://deb.debian.org/debian trixie/main amd64 libmagic-mgc amd64 1:5.46-5 [338 kB]
Get: 3 http://deb.debian.org/debian trixie/main amd64 libmagic1t64 amd64 1:5.46-5 [109 kB]
Get: 4 http://deb.debian.org/debian trixie/main amd64 file amd64 1:5.46-5 [43.6 kB]
Get: 5 http://deb.debian.org/debian trixie/main amd64 gettext-base amd64 0.23.1-1 [243 kB]
Get: 6 http://deb.debian.org/debian trixie/main amd64 libuchardet0 amd64 0.0.8-1+b2 [68.9 kB]
Get: 7 http://deb.debian.org/debian trixie/main amd64 groff-base amd64 1.23.0-7 [1185 kB]
Get: 8 http://deb.debian.org/debian trixie/main amd64 bsdextrautils amd64 2.41-4 [94.6 kB]
Get: 9 http://deb.debian.org/debian trixie/main amd64 libpipeline1 amd64 1.5.8-1 [42.0 kB]
Get: 10 http://deb.debian.org/debian trixie/main amd64 man-db amd64 2.13.0-1 [1420 kB]
Get: 11 http://deb.debian.org/debian trixie/main amd64 m4 amd64 1.4.19-7 [294 kB]
Get: 12 http://deb.debian.org/debian trixie/main amd64 autoconf all 2.72-3 [493 kB]
Get: 13 http://deb.debian.org/debian trixie/main amd64 autotools-dev all 20240727.1 [60.2 kB]
Get: 14 http://deb.debian.org/debian trixie/main amd64 automake all 1:1.17-4 [862 kB]
Get: 15 http://deb.debian.org/debian trixie/main amd64 autopoint all 0.23.1-1 [770 kB]
Get: 16 http://deb.debian.org/debian trixie/main amd64 libdebhelper-perl all 13.24.2 [90.9 kB]
Get: 17 http://deb.debian.org/debian trixie/main amd64 libtool all 2.5.4-4 [539 kB]
Get: 18 http://deb.debian.org/debian trixie/main amd64 dh-autoreconf all 20 [17.1 kB]
Get: 19 http://deb.debian.org/debian trixie/main amd64 libarchive-zip-perl all 1.68-1 [104 kB]
Get: 20 http://deb.debian.org/debian trixie/main amd64 libfile-stripnondeterminism-perl all 1.14.1-2 [19.7 kB]
Get: 21 http://deb.debian.org/debian trixie/main amd64 dh-strip-nondeterminism all 1.14.1-2 [8620 B]
Get: 22 http://deb.debian.org/debian trixie/main amd64 libelf1t64 amd64 0.192-4 [189 kB]
Get: 23 http://deb.debian.org/debian trixie/main amd64 dwz amd64 0.15-1+b1 [110 kB]
Get: 24 http://deb.debian.org/debian trixie/main amd64 libunistring5 amd64 1.3-2 [477 kB]
Get: 25 http://deb.debian.org/debian trixie/main amd64 libxml2 amd64 2.12.7+dfsg+really2.9.14-0.4 [698 kB]
Get: 26 http://deb.debian.org/debian trixie/main amd64 gettext amd64 0.23.1-1 [1680 kB]
Get: 27 http://deb.debian.org/debian trixie/main amd64 intltool-debian all 0.35.0+20060710.6 [22.9 kB]
Get: 28 http://deb.debian.org/debian trixie/main amd64 po-debconf all 1.0.21+nmu1 [248 kB]
Get: 29 http://deb.debian.org/debian trixie/main amd64 debhelper all 13.24.2 [919 kB]
Get: 30 http://deb.debian.org/debian trixie/main amd64 dh-golang all 1.63 [24.4 kB]
Get: 31 http://deb.debian.org/debian trixie/main amd64 golang-1.24-src all 1.24.2-2 [21.2 MB]
Get: 32 http://deb.debian.org/debian trixie/main amd64 golang-1.24-go amd64 1.24.2-2 [28.6 MB]
Get: 33 http://deb.debian.org/debian trixie/main amd64 golang-src all 2:1.24~2 [5136 B]
Get: 34 http://deb.debian.org/debian trixie/main amd64 golang-go amd64 2:1.24~2 [44.3 kB]
Get: 35 http://deb.debian.org/debian trixie/main amd64 golang-any amd64 2:1.24~2 [5216 B]
Get: 36 http://deb.debian.org/debian trixie/main amd64 golang-github-alecthomas-units-dev all 0.0~git20240927.0f3dac3-1 [8880 B]
Get: 37 http://deb.debian.org/debian trixie/main amd64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB]
Get: 38 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB]
Get: 39 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-sys-dev all 0.22.0-1 [431 kB]
Get: 40 http://deb.debian.org/debian trixie/main amd64 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B]
Get: 41 http://deb.debian.org/debian trixie/main amd64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB]
Get: 42 http://deb.debian.org/debian trixie/main amd64 golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB]
Get: 43 http://deb.debian.org/debian trixie/main amd64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB]
Get: 44 http://deb.debian.org/debian trixie/main amd64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB]
Get: 45 http://deb.debian.org/debian trixie/main amd64 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB]
Get: 46 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB]
Get: 47 http://deb.debian.org/debian trixie/main amd64 golang-github-stretchr-testify-dev all 1.10.0-1 [78.4 kB]
Get: 48 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB]
Get: 49 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB]
Get: 50 http://deb.debian.org/debian trixie/main amd64 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB]
Get: 51 http://deb.debian.org/debian trixie/main amd64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B]
Get: 52 http://deb.debian.org/debian trixie/main amd64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB]
Get: 53 http://deb.debian.org/debian trixie/main amd64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB]
Get: 54 http://deb.debian.org/debian trixie/main amd64 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB]
Get: 55 http://deb.debian.org/debian trixie/main amd64 golang-google-protobuf-dev all 1.36.5-1 [796 kB]
Get: 56 http://deb.debian.org/debian trixie/main amd64 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB]
Get: 57 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B]
Get: 58 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB]
Get: 59 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB]
Get: 60 http://deb.debian.org/debian trixie/main amd64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB]
Get: 61 http://deb.debian.org/debian trixie/main amd64 golang-github-beorn7-perks-dev all 1.0.1-3 [12.2 kB]
Get: 62 http://deb.debian.org/debian trixie/main amd64 golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB]
Get: 63 http://deb.debian.org/debian trixie/main amd64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B]
Get: 64 http://deb.debian.org/debian trixie/main amd64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB]
Get: 65 http://deb.debian.org/debian trixie/main amd64 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB]
Get: 66 http://deb.debian.org/debian trixie/main amd64 golang-github-klauspost-compress-dev all 1.18.0+ds1-1 [33.3 MB]
Get: 67 http://deb.debian.org/debian trixie/main amd64 golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB]
Get: 68 http://deb.debian.org/debian trixie/main amd64 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB]
Get: 69 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-yaml.v2-dev all 2.4.0-5 [60.5 kB]
Get: 70 http://deb.debian.org/debian trixie/main amd64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB]
Get: 71 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-term-dev all 0.22.0-1 [14.8 kB]
Get: 72 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-text-dev all 0.22.0-1 [3995 kB]
Get: 73 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-crypto-dev all 1:0.25.0-1 [1682 kB]
Get: 74 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-net-dev all 1:0.27.0-1 [898 kB]
Get: 75 http://deb.debian.org/debian trixie/main amd64 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB]
Get: 76 http://deb.debian.org/debian trixie/main amd64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB]
Get: 77 http://deb.debian.org/debian trixie/main amd64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB]
Get: 78 http://deb.debian.org/debian trixie/main amd64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB]
Get: 79 http://deb.debian.org/debian trixie/main amd64 golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B]
Get: 80 http://deb.debian.org/debian trixie/main amd64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B]
Get: 81 http://deb.debian.org/debian trixie/main amd64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-4 [14.6 kB]
Get: 82 http://deb.debian.org/debian trixie/main amd64 golang-github-prometheus-sigv4-dev all 0.1.2-2 [8484 B]
Get: 83 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-oauth2-dev all 0.27.0-1 [37.2 kB]
Get: 84 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB]
Get: 85 http://deb.debian.org/debian trixie/main amd64 golang-github-prometheus-common-dev all 0.62.0-3 [118 kB]
Get: 86 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB]
Get: 87 http://deb.debian.org/debian trixie/main amd64 golang-github-prometheus-procfs-dev all 0.15.1-1 [184 kB]
Get: 88 http://deb.debian.org/debian trixie/main amd64 golang-github-prometheus-client-golang-dev all 1.21.1-1 [194 kB]
Get: 89 http://deb.debian.org/debian trixie/main amd64 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB]
Get: 90 http://deb.debian.org/debian trixie/main amd64 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB]
Get: 91 http://deb.debian.org/debian trixie/main amd64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B]
Get: 92 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB]
Get: 93 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B]
Get: 94 http://deb.debian.org/debian trixie/main amd64 libsasl2-modules-db amd64 2.1.28+dfsg1-9 [19.8 kB]
Get: 95 http://deb.debian.org/debian trixie/main amd64 libsasl2-2 amd64 2.1.28+dfsg1-9 [57.5 kB]
Get: 96 http://deb.debian.org/debian trixie/main amd64 libssl-dev amd64 3.5.0-1 [2949 kB]
Get: 97 http://deb.debian.org/debian trixie/main amd64 libsasl2-dev amd64 2.1.28+dfsg1-9 [234 kB]
Get: 98 http://deb.debian.org/debian trixie/main amd64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB]
Get: 99 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-mod-dev all 0.19.0-1 [104 kB]
Get: 100 http://deb.debian.org/debian trixie/main amd64 golang-github-yuin-goldmark-dev all 1.7.4-1 [113 kB]
Get: 101 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-tools-dev all 1:0.25.0+ds-1 [1608 kB]
Get: 102 http://deb.debian.org/debian trixie/main amd64 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB]
Get: 103 http://deb.debian.org/debian trixie/main amd64 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB]
Get: 104 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-2 [31.2 kB]
Get: 105 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB]
Get: 106 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB]
Fetched 122 MB in 7s (18.4 MB/s)
Preconfiguring packages ...
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 19840 files and directories currently installed.)
Preparing to unpack .../000-sensible-utils_0.0.24_all.deb ...
Unpacking sensible-utils (0.0.24) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../001-libmagic-mgc_1%3a5.46-5_amd64.deb ...
Unpacking libmagic-mgc (1:5.46-5) ...
Selecting previously unselected package libmagic1t64:amd64.
Preparing to unpack .../002-libmagic1t64_1%3a5.46-5_amd64.deb ...
Unpacking libmagic1t64:amd64 (1:5.46-5) ...
Selecting previously unselected package file.
Preparing to unpack .../003-file_1%3a5.46-5_amd64.deb ...
Unpacking file (1:5.46-5) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../004-gettext-base_0.23.1-1_amd64.deb ...
Unpacking gettext-base (0.23.1-1) ...
Selecting previously unselected package libuchardet0:amd64.
Preparing to unpack .../005-libuchardet0_0.0.8-1+b2_amd64.deb ...
Unpacking libuchardet0:amd64 (0.0.8-1+b2) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../006-groff-base_1.23.0-7_amd64.deb ...
Unpacking groff-base (1.23.0-7) ...
Selecting previously unselected package bsdextrautils.
Preparing to unpack .../007-bsdextrautils_2.41-4_amd64.deb ...
Unpacking bsdextrautils (2.41-4) ...
Selecting previously unselected package libpipeline1:amd64.
Preparing to unpack .../008-libpipeline1_1.5.8-1_amd64.deb ...
Unpacking libpipeline1:amd64 (1.5.8-1) ...
Selecting previously unselected package man-db.
Preparing to unpack .../009-man-db_2.13.0-1_amd64.deb ...
Unpacking man-db (2.13.0-1) ...
Selecting previously unselected package m4.
Preparing to unpack .../010-m4_1.4.19-7_amd64.deb ...
Unpacking m4 (1.4.19-7) ...
Selecting previously unselected package autoconf.
Preparing to unpack .../011-autoconf_2.72-3_all.deb ...
Unpacking autoconf (2.72-3) ...
Selecting previously unselected package autotools-dev.
Preparing to unpack .../012-autotools-dev_20240727.1_all.deb ...
Unpacking autotools-dev (20240727.1) ...
Selecting previously unselected package automake.
Preparing to unpack .../013-automake_1%3a1.17-4_all.deb ...
Unpacking automake (1:1.17-4) ...
Selecting previously unselected package autopoint.
Preparing to unpack .../014-autopoint_0.23.1-1_all.deb ...
Unpacking autopoint (0.23.1-1) ...
Selecting previously unselected package libdebhelper-perl.
Preparing to unpack .../015-libdebhelper-perl_13.24.2_all.deb ...
Unpacking libdebhelper-perl (13.24.2) ...
Selecting previously unselected package libtool.
Preparing to unpack .../016-libtool_2.5.4-4_all.deb ...
Unpacking libtool (2.5.4-4) ...
Selecting previously unselected package dh-autoreconf.
Preparing to unpack .../017-dh-autoreconf_20_all.deb ...
Unpacking dh-autoreconf (20) ...
Selecting previously unselected package libarchive-zip-perl.
Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ...
Unpacking libarchive-zip-perl (1.68-1) ...
Selecting previously unselected package libfile-stripnondeterminism-perl.
Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.1-2_all.deb ...
Unpacking libfile-stripnondeterminism-perl (1.14.1-2) ...
Selecting previously unselected package dh-strip-nondeterminism.
Preparing to unpack .../020-dh-strip-nondeterminism_1.14.1-2_all.deb ...
Unpacking dh-strip-nondeterminism (1.14.1-2) ...
Selecting previously unselected package libelf1t64:amd64.
Preparing to unpack .../021-libelf1t64_0.192-4_amd64.deb ...
Unpacking libelf1t64:amd64 (0.192-4) ...
Selecting previously unselected package dwz.
Preparing to unpack .../022-dwz_0.15-1+b1_amd64.deb ...
Unpacking dwz (0.15-1+b1) ...
Selecting previously unselected package libunistring5:amd64.
Preparing to unpack .../023-libunistring5_1.3-2_amd64.deb ...
Unpacking libunistring5:amd64 (1.3-2) ...
Selecting previously unselected package libxml2:amd64.
Preparing to unpack .../024-libxml2_2.12.7+dfsg+really2.9.14-0.4_amd64.deb ...
Unpacking libxml2:amd64 (2.12.7+dfsg+really2.9.14-0.4) ...
Selecting previously unselected package gettext.
Preparing to unpack .../025-gettext_0.23.1-1_amd64.deb ...
Unpacking gettext (0.23.1-1) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../026-intltool-debian_0.35.0+20060710.6_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.6) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../027-po-debconf_1.0.21+nmu1_all.deb ...
Unpacking po-debconf (1.0.21+nmu1) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../028-debhelper_13.24.2_all.deb ...
Unpacking debhelper (13.24.2) ...
Selecting previously unselected package dh-golang.
Preparing to unpack .../029-dh-golang_1.63_all.deb ...
Unpacking dh-golang (1.63) ...
Selecting previously unselected package golang-1.24-src.
Preparing to unpack .../030-golang-1.24-src_1.24.2-2_all.deb ...
Unpacking golang-1.24-src (1.24.2-2) ...
Selecting previously unselected package golang-1.24-go.
Preparing to unpack .../031-golang-1.24-go_1.24.2-2_amd64.deb ...
Unpacking golang-1.24-go (1.24.2-2) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../032-golang-src_2%3a1.24~2_all.deb ...
Unpacking golang-src (2:1.24~2) ...
Selecting previously unselected package golang-go:amd64.
Preparing to unpack .../033-golang-go_2%3a1.24~2_amd64.deb ...
Unpacking golang-go:amd64 (2:1.24~2) ...
Selecting previously unselected package golang-any:amd64.
Preparing to unpack .../034-golang-any_2%3a1.24~2_amd64.deb ...
Unpacking golang-any:amd64 (2:1.24~2) ...
Selecting previously unselected package golang-github-alecthomas-units-dev.
Preparing to unpack .../035-golang-github-alecthomas-units-dev_0.0~git20240927.0f3dac3-1_all.deb ...
Unpacking golang-github-alecthomas-units-dev (0.0~git20240927.0f3dac3-1) ...
Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev.
Preparing to unpack .../036-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ...
Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev.
Preparing to unpack .../037-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ...
Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Selecting previously unselected package golang-golang-x-sys-dev.
Preparing to unpack .../038-golang-golang-x-sys-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-sys-dev (0.22.0-1) ...
Selecting previously unselected package golang-github-mattn-go-isatty-dev.
Preparing to unpack .../039-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ...
Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Selecting previously unselected package golang-github-mattn-go-colorable-dev.
Preparing to unpack .../040-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ...
Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Selecting previously unselected package golang-github-fatih-color-dev.
Preparing to unpack .../041-golang-github-fatih-color-dev_1.17.0-1_all.deb ...
Unpacking golang-github-fatih-color-dev (1.17.0-1) ...
Selecting previously unselected package golang-github-davecgh-go-spew-dev.
Preparing to unpack .../042-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ...
Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Selecting previously unselected package golang-github-pmezard-go-difflib-dev.
Preparing to unpack .../043-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ...
Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Selecting previously unselected package golang-github-stretchr-objx-dev.
Preparing to unpack .../044-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ...
Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ...
Selecting previously unselected package golang-gopkg-yaml.v3-dev.
Preparing to unpack .../045-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ...
Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Selecting previously unselected package golang-github-stretchr-testify-dev.
Preparing to unpack .../046-golang-github-stretchr-testify-dev_1.10.0-1_all.deb ...
Unpacking golang-github-stretchr-testify-dev (1.10.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-hclog-dev.
Preparing to unpack .../047-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ...
Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev.
Preparing to unpack .../048-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ...
Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Selecting previously unselected package golang-github-pkg-errors-dev.
Preparing to unpack .../049-golang-github-pkg-errors-dev_0.9.1-3_all.deb ...
Unpacking golang-github-pkg-errors-dev (0.9.1-3) ...
Selecting previously unselected package golang-github-tv42-httpunix-dev.
Preparing to unpack .../050-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ...
Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev.
Preparing to unpack .../051-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ...
Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Selecting previously unselected package golang-github-datadog-datadog-go-dev.
Preparing to unpack .../052-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ...
Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Selecting previously unselected package golang-github-google-go-cmp-dev.
Preparing to unpack .../053-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ...
Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ...
Selecting previously unselected package golang-google-protobuf-dev.
Preparing to unpack .../054-golang-google-protobuf-dev_1.36.5-1_all.deb ...
Unpacking golang-google-protobuf-dev (1.36.5-1) ...
Selecting previously unselected package golang-github-golang-protobuf-1-5-dev.
Preparing to unpack .../055-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ...
Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Selecting previously unselected package golang-github-hashicorp-go-uuid-dev.
Preparing to unpack .../056-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ...
Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Selecting previously unselected package golang-github-hashicorp-golang-lru-dev.
Preparing to unpack .../057-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ...
Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev.
Preparing to unpack .../058-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ...
Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Selecting previously unselected package golang-github-pascaldekloe-goe-dev.
Preparing to unpack .../059-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ...
Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Selecting previously unselected package golang-github-beorn7-perks-dev.
Preparing to unpack .../060-golang-github-beorn7-perks-dev_1.0.1-3_all.deb ...
Unpacking golang-github-beorn7-perks-dev (1.0.1-3) ...
Selecting previously unselected package golang-github-cespare-xxhash-dev.
Preparing to unpack .../061-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ...
Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ...
Selecting previously unselected package golang-github-modern-go-concurrent-dev.
Preparing to unpack .../062-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ...
Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Selecting previously unselected package golang-github-modern-go-reflect2-dev.
Preparing to unpack .../063-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ...
Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-json-iterator-go-dev.
Preparing to unpack .../064-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ...
Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ...
Selecting previously unselected package golang-github-klauspost-compress-dev.
Preparing to unpack .../065-golang-github-klauspost-compress-dev_1.18.0+ds1-1_all.deb ...
Unpacking golang-github-klauspost-compress-dev (1.18.0+ds1-1) ...
Selecting previously unselected package golang-github-kylelemons-godebug-dev.
Preparing to unpack .../066-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ...
Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Selecting previously unselected package golang-github-prometheus-client-model-dev.
Preparing to unpack .../067-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ...
Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ...
Selecting previously unselected package golang-gopkg-yaml.v2-dev.
Preparing to unpack .../068-golang-gopkg-yaml.v2-dev_2.4.0-5_all.deb ...
Unpacking golang-gopkg-yaml.v2-dev (2.4.0-5) ...
Selecting previously unselected package golang-github-jmespath-go-jmespath-dev.
Preparing to unpack .../069-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ...
Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Selecting previously unselected package golang-golang-x-term-dev.
Preparing to unpack .../070-golang-golang-x-term-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-term-dev (0.22.0-1) ...
Selecting previously unselected package golang-golang-x-text-dev.
Preparing to unpack .../071-golang-golang-x-text-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-text-dev (0.22.0-1) ...
Selecting previously unselected package golang-golang-x-crypto-dev.
Preparing to unpack .../072-golang-golang-x-crypto-dev_1%3a0.25.0-1_all.deb ...
Unpacking golang-golang-x-crypto-dev (1:0.25.0-1) ...
Selecting previously unselected package golang-golang-x-net-dev.
Preparing to unpack .../073-golang-golang-x-net-dev_1%3a0.27.0-1_all.deb ...
Unpacking golang-golang-x-net-dev (1:0.27.0-1) ...
Selecting previously unselected package golang-github-aws-aws-sdk-go-dev.
Preparing to unpack .../074-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ...
Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Selecting previously unselected package golang-github-go-logfmt-logfmt-dev.
Preparing to unpack .../075-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ...
Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Selecting previously unselected package golang-github-go-kit-log-dev.
Preparing to unpack .../076-golang-github-go-kit-log-dev_0.2.1-1_all.deb ...
Unpacking golang-github-go-kit-log-dev (0.2.1-1) ...
Selecting previously unselected package golang-github-julienschmidt-httprouter-dev.
Preparing to unpack .../077-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ...
Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Selecting previously unselected package golang-github-munnerz-goautoneg-dev.
Preparing to unpack .../078-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ...
Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Selecting previously unselected package golang-github-jpillora-backoff-dev.
Preparing to unpack .../079-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ...
Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev.
Preparing to unpack .../080-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-4_all.deb ...
Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ...
Selecting previously unselected package golang-github-prometheus-sigv4-dev.
Preparing to unpack .../081-golang-github-prometheus-sigv4-dev_0.1.2-2_all.deb ...
Unpacking golang-github-prometheus-sigv4-dev (0.1.2-2) ...
Selecting previously unselected package golang-golang-x-oauth2-dev.
Preparing to unpack .../082-golang-golang-x-oauth2-dev_0.27.0-1_all.deb ...
Unpacking golang-golang-x-oauth2-dev (0.27.0-1) ...
Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev.
Preparing to unpack .../083-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ...
Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Selecting previously unselected package golang-github-prometheus-common-dev.
Preparing to unpack .../084-golang-github-prometheus-common-dev_0.62.0-3_all.deb ...
Unpacking golang-github-prometheus-common-dev (0.62.0-3) ...
Selecting previously unselected package golang-golang-x-sync-dev.
Preparing to unpack .../085-golang-golang-x-sync-dev_0.7.0-1_all.deb ...
Unpacking golang-golang-x-sync-dev (0.7.0-1) ...
Selecting previously unselected package golang-github-prometheus-procfs-dev.
Preparing to unpack .../086-golang-github-prometheus-procfs-dev_0.15.1-1_all.deb ...
Unpacking golang-github-prometheus-procfs-dev (0.15.1-1) ...
Selecting previously unselected package golang-github-prometheus-client-golang-dev.
Preparing to unpack .../087-golang-github-prometheus-client-golang-dev_1.21.1-1_all.deb ...
Unpacking golang-github-prometheus-client-golang-dev (1.21.1-1) ...
Selecting previously unselected package golang-github-armon-go-metrics-dev.
Preparing to unpack .../088-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ...
Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ...
Selecting previously unselected package golang-github-creack-pty-dev.
Preparing to unpack .../089-golang-github-creack-pty-dev_1.1.21-1_all.deb ...
Unpacking golang-github-creack-pty-dev (1.1.21-1) ...
Selecting previously unselected package golang-github-vmihailenco-tagparser-dev.
Preparing to unpack .../090-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ...
Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev.
Preparing to unpack .../091-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ...
Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Selecting previously unselected package golang-gopkg-tomb.v2-dev.
Preparing to unpack .../092-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ...
Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../093-libsasl2-modules-db_2.1.28+dfsg1-9_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.28+dfsg1-9) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../094-libsasl2-2_2.1.28+dfsg1-9_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.28+dfsg1-9) ...
Selecting previously unselected package libssl-dev:amd64.
Preparing to unpack .../095-libssl-dev_3.5.0-1_amd64.deb ...
Unpacking libssl-dev:amd64 (3.5.0-1) ...
Selecting previously unselected package libsasl2-dev.
Preparing to unpack .../096-libsasl2-dev_2.1.28+dfsg1-9_amd64.deb ...
Unpacking libsasl2-dev (2.1.28+dfsg1-9) ...
Selecting previously unselected package golang-github-kr-text-dev.
Preparing to unpack .../097-golang-github-kr-text-dev_0.2.0-2_all.deb ...
Unpacking golang-github-kr-text-dev (0.2.0-2) ...
Selecting previously unselected package golang-golang-x-mod-dev.
Preparing to unpack .../098-golang-golang-x-mod-dev_0.19.0-1_all.deb ...
Unpacking golang-golang-x-mod-dev (0.19.0-1) ...
Selecting previously unselected package golang-github-yuin-goldmark-dev.
Preparing to unpack .../099-golang-github-yuin-goldmark-dev_1.7.4-1_all.deb ...
Unpacking golang-github-yuin-goldmark-dev (1.7.4-1) ...
Selecting previously unselected package golang-golang-x-tools-dev.
Preparing to unpack .../100-golang-golang-x-tools-dev_1%3a0.25.0+ds-1_all.deb ...
Unpacking golang-golang-x-tools-dev (1:0.25.0+ds-1) ...
Selecting previously unselected package golang-github-rogpeppe-go-internal-dev.
Preparing to unpack .../101-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ...
Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Selecting previously unselected package golang-github-kr-pretty-dev.
Preparing to unpack .../102-golang-github-kr-pretty-dev_0.3.1-1_all.deb ...
Unpacking golang-github-kr-pretty-dev (0.3.1-1) ...
Selecting previously unselected package golang-gopkg-check.v1-dev.
Preparing to unpack .../103-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-2_all.deb ...
Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ...
Selecting previously unselected package golang-gopkg-mgo.v2-dev.
Preparing to unpack .../104-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ...
Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev.
Preparing to unpack .../105-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ...
Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-1.24-src (1.24.2-2) ...
Setting up libpipeline1:amd64 (1.5.8-1) ...
Setting up golang-github-google-go-cmp-dev (0.6.0-1) ...
Setting up golang-gopkg-yaml.v2-dev (2.4.0-5) ...
Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Setting up bsdextrautils (2.41-4) ...
Setting up golang-github-creack-pty-dev (1.1.21-1) ...
Setting up golang-github-stretchr-objx-dev (0.5.2-1) ...
Setting up libmagic-mgc (1:5.46-5) ...
Setting up golang-github-pkg-errors-dev (0.9.1-3) ...
Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Setting up libarchive-zip-perl (1.68-1) ...
Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Setting up libdebhelper-perl (13.24.2) ...
Setting up golang-golang-x-sys-dev (0.22.0-1) ...
Setting up libmagic1t64:amd64 (1:5.46-5) ...
Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Setting up gettext-base (0.23.1-1) ...
Setting up m4 (1.4.19-7) ...
Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Setting up golang-golang-x-term-dev (0.22.0-1) ...
Setting up file (1:5.46-5) ...
Setting up golang-github-beorn7-perks-dev (1.0.1-3) ...
Setting up libelf1t64:amd64 (0.192-4) ...
Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Setting up libsasl2-modules-db:amd64 (2.1.28+dfsg1-9) ...
Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ...
Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Setting up autotools-dev (20240727.1) ...
Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Setting up golang-google-protobuf-dev (1.36.5-1) ...
Setting up golang-github-kr-text-dev (0.2.0-2) ...
Setting up golang-github-stretchr-testify-dev (1.10.0-1) ...
Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Setting up golang-github-yuin-goldmark-dev (1.7.4-1) ...
Setting up libunistring5:amd64 (1.3-2) ...
Setting up libssl-dev:amd64 (3.5.0-1) ...
Setting up golang-golang-x-sync-dev (0.7.0-1) ...
Setting up golang-golang-x-mod-dev (0.19.0-1) ...
Setting up autopoint (0.23.1-1) ...
Setting up golang-1.24-go (1.24.2-2) ...
Setting up golang-github-klauspost-compress-dev (1.18.0+ds1-1) ...
Setting up libsasl2-2:amd64 (2.1.28+dfsg1-9) ...
Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Setting up autoconf (2.72-3) ...
Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Setting up dwz (0.15-1+b1) ...
Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Setting up sensible-utils (0.0.24) ...
Setting up golang-golang-x-text-dev (0.22.0-1) ...
Setting up libuchardet0:amd64 (0.0.8-1+b2) ...
Setting up golang-github-prometheus-procfs-dev (0.15.1-1) ...
Setting up golang-github-alecthomas-units-dev (0.0~git20240927.0f3dac3-1) ...
Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Setting up golang-golang-x-crypto-dev (1:0.25.0-1) ...
Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ...
Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Setting up golang-golang-x-oauth2-dev (0.27.0-1) ...
Setting up golang-src (2:1.24~2) ...
Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Setting up libxml2:amd64 (2.12.7+dfsg+really2.9.14-0.4) ...
Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Setting up automake (1:1.17-4) ...
update-alternatives: using /usr/bin/automake-1.17 to provide /usr/bin/automake (automake) in auto mode
Setting up libfile-stripnondeterminism-perl (1.14.1-2) ...
Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Setting up golang-github-go-kit-log-dev (0.2.1-1) ...
Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Setting up gettext (0.23.1-1) ...
Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Setting up libtool (2.5.4-4) ...
Setting up golang-go:amd64 (2:1.24~2) ...
Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Setting up golang-github-json-iterator-go-dev (1.1.12-2) ...
Setting up golang-any:amd64 (2:1.24~2) ...
Setting up intltool-debian (0.35.0+20060710.6) ...
Setting up dh-autoreconf (20) ...
Setting up libsasl2-dev (2.1.28+dfsg1-9) ...
Setting up golang-golang-x-net-dev (1:0.27.0-1) ...
Setting up dh-strip-nondeterminism (1.14.1-2) ...
Setting up groff-base (1.23.0-7) ...
Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Setting up golang-github-fatih-color-dev (1.17.0-1) ...
Setting up po-debconf (1.0.21+nmu1) ...
Setting up golang-golang-x-tools-dev (1:0.25.0+ds-1) ...
Setting up man-db (2.13.0-1) ...
Not building database; man-db/auto-update is not 'true'.
Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Setting up golang-github-kr-pretty-dev (0.3.1-1) ...
Setting up debhelper (13.24.2) ...
Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ...
Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Setting up dh-golang (1.63) ...
Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ...
Setting up golang-github-prometheus-sigv4-dev (0.1.2-2) ...
Setting up golang-github-prometheus-common-dev (0.62.0-3) ...
Setting up golang-github-prometheus-client-golang-dev (1.21.1-1) ...
Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ...
Processing triggers for libc-bin (2.41-7) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Building tag database...
 -> Finished parsing the build-deps
I: Building the package
I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S  > ../golang-github-hashicorp-raft_1.5.0-1_source.changes
dpkg-buildpackage: info: source package golang-github-hashicorp-raft
dpkg-buildpackage: info: source version 1.5.0-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Daniel Swarbrick <dswarbrick@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture amd64
 debian/rules clean
dh clean --buildsystem=golang
   dh_auto_clean -O--buildsystem=golang
   dh_autoreconf_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules binary
dh binary --buildsystem=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-x86_64-linux-gnu && go install -trimpath -v -p 42 github.com/hashicorp/raft github.com/hashicorp/raft/bench
vendor/golang.org/x/crypto/cryptobyte/asn1
internal/profilerecord
internal/msan
internal/goarch
internal/coverage/rtcov
internal/goos
unicode/utf16
internal/goexperiment
internal/asan
container/list
crypto/internal/fips140/alias
internal/nettrace
internal/byteorder
math/bits
encoding
internal/unsafeheader
vendor/golang.org/x/crypto/internal/alias
log/internal
unicode/utf8
unicode
cmp
internal/cpu
internal/runtime/atomic
internal/itoa
crypto/internal/boring/sig
internal/runtime/syscall
sync/atomic
internal/godebugs
internal/runtime/math
internal/abi
internal/runtime/sys
crypto/internal/fips140/subtle
crypto/internal/fips140deps/byteorder
internal/chacha8rand
crypto/internal/fips140deps/cpu
internal/bytealg
math
internal/runtime/exithook
internal/stringslite
internal/race
internal/sync
internal/runtime/maps
runtime
internal/reflectlite
iter
crypto/subtle
sync
weak
maps
slices
errors
sort
internal/bisect
internal/testlog
internal/singleflight
unique
runtime/cgo
path
io
internal/oserror
github.com/hashicorp/golang-lru/simplelru
math/rand/v2
vendor/golang.org/x/net/dns/dnsmessage
strconv
internal/godebug
syscall
internal/saferio
hash
crypto/internal/randutil
bytes
strings
hash/crc64
hash/crc32
crypto/internal/fips140deps/godebug
math/rand
crypto
net/netip
reflect
vendor/golang.org/x/text/transform
crypto/internal/impl
crypto/internal/fips140
bufio
net/http/internal/ascii
github.com/hashicorp/go-immutable-radix
html
regexp/syntax
crypto/internal/fips140/sha256
crypto/internal/fips140/sha3
crypto/tls/internal/fips140tls
crypto/internal/fips140/sha512
crypto/internal/fips140/hmac
crypto/sha3
crypto/internal/fips140/check
crypto/internal/fips140hash
crypto/internal/fips140/hkdf
crypto/internal/fips140/tls12
crypto/internal/fips140/edwards25519/field
crypto/internal/fips140/bigmod
crypto/internal/fips140/aes
internal/syscall/execenv
crypto/internal/fips140/nistec/fiat
time
internal/syscall/unix
crypto/internal/fips140/tls13
regexp
crypto/internal/fips140/edwards25519
context
io/fs
internal/poll
crypto/internal/fips140/nistec
internal/filepathlite
os
internal/fmtsort
encoding/binary
encoding/base64
vendor/golang.org/x/crypto/internal/poly1305
golang.org/x/sys/unix
encoding/pem
crypto/internal/sysrand
io/ioutil
os/signal
path/filepath
fmt
vendor/golang.org/x/sys/cpu
internal/sysinfo
net
crypto/internal/entropy
crypto/internal/fips140/drbg
crypto/internal/fips140only
crypto/internal/fips140/ed25519
crypto/internal/fips140/ecdh
crypto/internal/fips140/ecdsa
crypto/internal/fips140/rsa
crypto/internal/fips140/mlkem
crypto/internal/fips140/aes/gcm
crypto/rc4
crypto/md5
crypto/cipher
crypto/internal/boring
vendor/golang.org/x/crypto/chacha20
crypto/des
crypto/sha512
crypto/aes
crypto/hmac
crypto/ecdh
crypto/sha256
crypto/sha1
log
encoding/hex
net/url
compress/flate
encoding/json
mime
net/http/internal
vendor/golang.org/x/net/http2/hpack
runtime/debug
text/template/parse
encoding/gob
vendor/golang.org/x/text/unicode/norm
math/big
go/token
runtime/trace
mime/quotedprintable
flag
vendor/golang.org/x/crypto/chacha20poly1305
vendor/golang.org/x/text/unicode/bidi
testing
compress/gzip
vendor/golang.org/x/text/secure/bidirule
github.com/mattn/go-isatty
text/template
github.com/mattn/go-colorable
github.com/fatih/color
vendor/golang.org/x/net/idna
crypto/internal/boring/bbig
crypto/elliptic
crypto/rand
crypto/dsa
encoding/asn1
github.com/hashicorp/go-hclog
crypto/ed25519
crypto/rsa
crypto/internal/hpke
html/template
crypto/x509/pkix
vendor/golang.org/x/crypto/cryptobyte
crypto/ecdsa
vendor/golang.org/x/net/http/httpproxy
net/textproto
crypto/x509
vendor/golang.org/x/net/http/httpguts
mime/multipart
crypto/tls
net/http/httptrace
net/http
net/rpc
github.com/armon/go-metrics
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/raft
github.com/hashicorp/raft/bench
   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
dh_auto_test
	cd obj-x86_64-linux-gnu && go test -vet=off -v -p 42 github.com/hashicorp/raft github.com/hashicorp/raft/bench
=== RUN   TestCommitment_setVoters
--- PASS: TestCommitment_setVoters (0.00s)
=== RUN   TestCommitment_match_max
--- PASS: TestCommitment_match_max (0.00s)
=== RUN   TestCommitment_match_nonVoting
--- PASS: TestCommitment_match_nonVoting (0.00s)
=== RUN   TestCommitment_recalculate
--- PASS: TestCommitment_recalculate (0.00s)
=== RUN   TestCommitment_recalculate_startIndex
--- PASS: TestCommitment_recalculate_startIndex (0.00s)
=== RUN   TestCommitment_noVoterSanity
--- PASS: TestCommitment_noVoterSanity (0.00s)
=== RUN   TestCommitment_singleVoter
--- PASS: TestCommitment_singleVoter (0.00s)
=== RUN   TestConfiguration_Configuration_Clone
--- PASS: TestConfiguration_Configuration_Clone (0.00s)
=== RUN   TestConfiguration_configurations_Clone
--- PASS: TestConfiguration_configurations_Clone (0.00s)
=== RUN   TestConfiguration_hasVote
--- PASS: TestConfiguration_hasVote (0.00s)
=== RUN   TestConfiguration_checkConfiguration
--- PASS: TestConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_nextConfiguration_table
--- PASS: TestConfiguration_nextConfiguration_table (0.00s)
=== RUN   TestConfiguration_nextConfiguration_prevIndex
--- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
=== RUN   TestConfiguration_nextConfiguration_checkConfiguration
--- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_encodeDecodePeers
--- PASS: TestConfiguration_encodeDecodePeers (0.00s)
=== RUN   TestConfiguration_encodeDecodeConfiguration
--- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
=== RUN   TestDiscardSnapshotStoreImpl
--- PASS: TestDiscardSnapshotStoreImpl (0.00s)
=== RUN   TestDiscardSnapshotSinkImpl
--- PASS: TestDiscardSnapshotSinkImpl (0.00s)
=== RUN   TestFileSnapshotStoreImpl
--- PASS: TestFileSnapshotStoreImpl (0.00s)
=== RUN   TestFileSnapshotSinkImpl
--- PASS: TestFileSnapshotSinkImpl (0.00s)
=== RUN   TestFileSS_CreateSnapshotMissingParentDir
2026-05-25T23:22:24.356Z [INFO]  creating new snapshot: path=/tmp/raft215299860/raft1496913637/snapshots/3-10-1779751344356.tmp
--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s)
=== RUN   TestFileSS_CreateSnapshot
2026-05-25T23:22:24.360Z [INFO]  creating new snapshot: path=/tmp/raft85305387/snapshots/3-10-1779751344360.tmp
2026-05-25T23:22:24.363Z [WARN]  found temporary snapshot: name=3-10-1779751344360.tmp
--- PASS: TestFileSS_CreateSnapshot (0.01s)
=== RUN   TestFileSS_CancelSnapshot
2026-05-25T23:22:24.373Z [INFO]  creating new snapshot: path=/tmp/raft3270299945/snapshots/3-10-1779751344373.tmp
--- PASS: TestFileSS_CancelSnapshot (0.00s)
=== RUN   TestFileSS_Retention
2026-05-25T23:22:24.377Z [INFO]  creating new snapshot: path=/tmp/raft1961542930/snapshots/3-10-1779751344377.tmp
2026-05-25T23:22:24.388Z [INFO]  creating new snapshot: path=/tmp/raft1961542930/snapshots/3-11-1779751344388.tmp
2026-05-25T23:22:24.396Z [INFO]  creating new snapshot: path=/tmp/raft1961542930/snapshots/3-12-1779751344396.tmp
2026-05-25T23:22:24.408Z [INFO]  reaping snapshot: path=/tmp/raft1961542930/snapshots/3-10-1779751344377
2026-05-25T23:22:24.409Z [INFO]  creating new snapshot: path=/tmp/raft1961542930/snapshots/3-13-1779751344409.tmp
2026-05-25T23:22:24.419Z [INFO]  reaping snapshot: path=/tmp/raft1961542930/snapshots/3-11-1779751344388
2026-05-25T23:22:24.420Z [INFO]  creating new snapshot: path=/tmp/raft1961542930/snapshots/3-14-1779751344420.tmp
2026-05-25T23:22:24.428Z [INFO]  reaping snapshot: path=/tmp/raft1961542930/snapshots/3-12-1779751344396
--- PASS: TestFileSS_Retention (0.05s)
=== RUN   TestFileSS_BadPerm
--- PASS: TestFileSS_BadPerm (0.00s)
=== RUN   TestFileSS_MissingParentDir
--- PASS: TestFileSS_MissingParentDir (0.00s)
=== RUN   TestFileSS_Ordering
2026-05-25T23:22:24.429Z [INFO]  creating new snapshot: path=/tmp/raft1932062109/snapshots/5-130350-1779751344429.tmp
2026-05-25T23:22:24.439Z [INFO]  creating new snapshot: path=/tmp/raft1932062109/snapshots/36-204917-1779751344439.tmp
--- PASS: TestFileSS_Ordering (0.02s)
=== RUN   TestDeferFutureSuccess
--- PASS: TestDeferFutureSuccess (0.00s)
=== RUN   TestDeferFutureError
--- PASS: TestDeferFutureError (0.00s)
=== RUN   TestDeferFutureConcurrent
--- PASS: TestDeferFutureConcurrent (0.00s)
=== RUN   TestInmemSnapshotStoreImpl
--- PASS: TestInmemSnapshotStoreImpl (0.00s)
=== RUN   TestInmemSnapshotSinkImpl
--- PASS: TestInmemSnapshotSinkImpl (0.00s)
=== RUN   TestInmemSS_CreateSnapshot
--- PASS: TestInmemSS_CreateSnapshot (0.00s)
=== RUN   TestInmemSS_OpenSnapshotTwice
--- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
=== RUN   TestInmemTransportImpl
--- PASS: TestInmemTransportImpl (0.00s)
=== RUN   TestInmemTransportWriteTimeout
--- PASS: TestInmemTransportWriteTimeout (0.01s)
=== RUN   TestRaft_Integ
--- SKIP: TestRaft_Integ (0.00s)
=== RUN   TestRaft_RestartFollower_LongInitialHeartbeat
--- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s)
=== RUN   TestLogCache
--- PASS: TestLogCache (0.00s)
=== RUN   TestLogCacheWithBackendStoreError
--- PASS: TestLogCacheWithBackendStoreError (0.00s)
=== RUN   TestOldestLog
=== RUN   TestOldestLog/empty_logs
=== RUN   TestOldestLog/simple_case
--- PASS: TestOldestLog (0.00s)
    --- PASS: TestOldestLog/empty_logs (0.00s)
    --- PASS: TestOldestLog/simple_case (0.00s)
=== RUN   TestEmitsLogStoreMetrics
--- PASS: TestEmitsLogStoreMetrics (0.01s)
=== RUN   TestNetworkTransport_CloseStreams
--- PASS: TestNetworkTransport_CloseStreams (0.00s)
=== RUN   TestNetworkTransport_StartStop
--- PASS: TestNetworkTransport_StartStop (0.00s)
=== RUN   TestNetworkTransport_Heartbeat_FastPath
--- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s)
=== RUN   TestNetworkTransport_AppendEntries
--- PASS: TestNetworkTransport_AppendEntries (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline
--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_CloseStreams
--- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.02s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0
2026-05-25T23:22:24.545Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10
2026-05-25T23:22:24.597Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
2026-05-25T23:22:24.649Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.15s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s)
=== RUN   TestNetworkTransport_RequestVote
--- PASS: TestNetworkTransport_RequestVote (0.00s)
=== RUN   TestNetworkTransport_InstallSnapshot
--- PASS: TestNetworkTransport_InstallSnapshot (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode
--- PASS: TestNetworkTransport_EncodeDecode (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode_AddressProvider
--- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
=== RUN   TestNetworkTransport_PooledConn
--- PASS: TestNetworkTransport_PooledConn (0.00s)
=== RUN   TestNetworkTransport_ListenBackoff
    net_transport_test.go:839: countingWriter: 2026-05-25T23:22:24.656Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-05-25T23:22:24.661Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-05-25T23:22:24.672Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-05-25T23:22:24.693Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-05-25T23:22:24.733Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-05-25T23:22:24.814Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-05-25T23:22:24.974Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-05-25T23:22:25.295Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-05-25T23:22:25.935Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-05-25T23:22:26.938Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-05-25T23:22:27.940Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
--- PASS: TestNetworkTransport_ListenBackoff (4.00s)
=== RUN   TestPeersJSON_BadConfiguration
--- PASS: TestPeersJSON_BadConfiguration (0.13s)
=== RUN   TestPeersJSON_ReadPeersJSON
--- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
=== RUN   TestPeersJSON_ReadConfigJSON
--- PASS: TestPeersJSON_ReadConfigJSON (0.07s)
=== RUN   TestRaft_StartStop
2026-05-25T23:22:28.854Z [INFO]  server-32f7f97a-00ec-ef84-04cb-13093892f003: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-32f7f97a-00ec-ef84-04cb-13093892f003 Address:32f7f97a-00ec-ef84-04cb-13093892f003}]"
--- PASS: TestRaft_StartStop (0.00s)
=== RUN   TestRaft_AfterShutdown
2026-05-25T23:22:28.855Z [INFO]  server-d5e55550-421a-fc5d-6f25-9dd53da43348: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5e55550-421a-fc5d-6f25-9dd53da43348 Address:d5e55550-421a-fc5d-6f25-9dd53da43348}]"
--- PASS: TestRaft_AfterShutdown (0.05s)
=== RUN   TestRaft_LiveBootstrap
2026-05-25T23:22:28.909Z [INFO]  server-a18cc808-5da2-f566-0d76-651378e1ef3a: initial configuration: index=0 servers=[]
2026-05-25T23:22:28.909Z [INFO]  server-38b83dba-c91f-18b9-4bd4-21eacfe86b3b: initial configuration: index=0 servers=[]
2026-05-25T23:22:28.910Z [INFO]  server-eca604f8-3552-f157-bfba-9f46f9128dcf: initial configuration: index=0 servers=[]
2026-05-25T23:22:28.909Z [INFO]  server-a18cc808-5da2-f566-0d76-651378e1ef3a: entering follower state: follower="Node at a18cc808-5da2-f566-0d76-651378e1ef3a [Follower]" leader-address= leader-id=
2026-05-25T23:22:28.910Z [INFO]  server-38b83dba-c91f-18b9-4bd4-21eacfe86b3b: entering follower state: follower="Node at 38b83dba-c91f-18b9-4bd4-21eacfe86b3b [Follower]" leader-address= leader-id=
2026-05-25T23:22:28.910Z [INFO]  server-eca604f8-3552-f157-bfba-9f46f9128dcf: entering follower state: follower="Node at eca604f8-3552-f157-bfba-9f46f9128dcf [Follower]" leader-address= leader-id=
2026-05-25T23:22:28.910Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:22:28.970Z [WARN]  server-38b83dba-c91f-18b9-4bd4-21eacfe86b3b: no known peers, aborting election
2026-05-25T23:22:28.976Z [WARN]  server-eca604f8-3552-f157-bfba-9f46f9128dcf: no known peers, aborting election
2026-05-25T23:22:28.997Z [WARN]  server-a18cc808-5da2-f566-0d76-651378e1ef3a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:28.997Z [INFO]  server-a18cc808-5da2-f566-0d76-651378e1ef3a: entering candidate state: node="Node at a18cc808-5da2-f566-0d76-651378e1ef3a [Candidate]" term=2
2026-05-25T23:22:28.998Z [INFO]  server-a18cc808-5da2-f566-0d76-651378e1ef3a: election won: term=2 tally=2
2026-05-25T23:22:28.998Z [INFO]  server-a18cc808-5da2-f566-0d76-651378e1ef3a: entering leader state: leader="Node at a18cc808-5da2-f566-0d76-651378e1ef3a [Leader]"
2026-05-25T23:22:28.998Z [INFO]  server-a18cc808-5da2-f566-0d76-651378e1ef3a: added peer, starting replication: peer=server-38b83dba-c91f-18b9-4bd4-21eacfe86b3b
2026-05-25T23:22:28.998Z [INFO]  server-a18cc808-5da2-f566-0d76-651378e1ef3a: added peer, starting replication: peer=server-eca604f8-3552-f157-bfba-9f46f9128dcf
2026-05-25T23:22:28.998Z [WARN]  server-38b83dba-c91f-18b9-4bd4-21eacfe86b3b: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2026-05-25T23:22:28.998Z [WARN]  server-eca604f8-3552-f157-bfba-9f46f9128dcf: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2026-05-25T23:22:28.998Z [WARN]  server-a18cc808-5da2-f566-0d76-651378e1ef3a: appendEntries rejected, sending older logs: peer="{Voter server-38b83dba-c91f-18b9-4bd4-21eacfe86b3b 38b83dba-c91f-18b9-4bd4-21eacfe86b3b}" next=1
2026-05-25T23:22:28.998Z [WARN]  server-a18cc808-5da2-f566-0d76-651378e1ef3a: appendEntries rejected, sending older logs: peer="{Voter server-eca604f8-3552-f157-bfba-9f46f9128dcf eca604f8-3552-f157-bfba-9f46f9128dcf}" next=1
2026-05-25T23:22:28.998Z [INFO]  server-a18cc808-5da2-f566-0d76-651378e1ef3a: pipelining replication: peer="{Voter server-38b83dba-c91f-18b9-4bd4-21eacfe86b3b 38b83dba-c91f-18b9-4bd4-21eacfe86b3b}"
2026-05-25T23:22:28.998Z [INFO]  server-a18cc808-5da2-f566-0d76-651378e1ef3a: pipelining replication: peer="{Voter server-eca604f8-3552-f157-bfba-9f46f9128dcf eca604f8-3552-f157-bfba-9f46f9128dcf}"
2026-05-25T23:22:29.104Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:22:28.998261054 +0000 UTC m=+4.644708520 (2 nodes), 87.977272ms from start of poll, 88.421581ms from cluster start. Timeout at 2026-05-25 23:22:29.103266457 +0000 UTC m=+4.749713932, 105.005412ms after stability
2026-05-25T23:22:29.104Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:29.209Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:29.104479795 +0000 UTC m=+4.750927248 (1 nodes), 1.2µs from start of poll, 194.640309ms from cluster start. Timeout at 2026-05-25 23:22:29.209483665 +0000 UTC m=+4.855931128, 105.00388ms after stability
2026-05-25T23:22:29.220Z [INFO]  server-a18cc808-5da2-f566-0d76-651378e1ef3a: aborting pipeline replication: peer="{Voter server-eca604f8-3552-f157-bfba-9f46f9128dcf eca604f8-3552-f157-bfba-9f46f9128dcf}"
2026-05-25T23:22:29.220Z [INFO]  server-a18cc808-5da2-f566-0d76-651378e1ef3a: aborting pipeline replication: peer="{Voter server-38b83dba-c91f-18b9-4bd4-21eacfe86b3b 38b83dba-c91f-18b9-4bd4-21eacfe86b3b}"
--- PASS: TestRaft_LiveBootstrap (0.31s)
=== RUN   TestRaft_LiveBootstrap_From_NonVoter
2026-05-25T23:22:29.221Z [INFO]  server-21c97ee8-179b-622f-0509-8880021636d9: initial configuration: index=0 servers=[]
2026-05-25T23:22:29.221Z [INFO]  server-3482c990-6e98-e9dc-20aa-61066cce8469: initial configuration: index=0 servers=[]
2026-05-25T23:22:29.221Z [INFO]  server-3482c990-6e98-e9dc-20aa-61066cce8469: entering follower state: follower="Node at 3482c990-6e98-e9dc-20aa-61066cce8469 [Follower]" leader-address= leader-id=
2026-05-25T23:22:29.221Z [INFO]  server-21c97ee8-179b-622f-0509-8880021636d9: entering follower state: follower="Node at 21c97ee8-179b-622f-0509-8880021636d9 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s)
=== RUN   TestRaft_RecoverCluster_NoState
2026-05-25T23:22:29.222Z [INFO]  server-64bbee5f-858f-0d33-0240-d1f9c7c668d8: initial configuration: index=0 servers=[]
--- PASS: TestRaft_RecoverCluster_NoState (0.00s)
=== RUN   TestRaft_RecoverCluster
=== RUN   TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs
2026-05-25T23:22:29.223Z [INFO]  server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2 Address:2a0d8958-1e44-8ac0-abc3-a6a1947e42d2} {Suffrage:Voter ID:server-a3002cc2-0a65-b208-2120-c438cc6a3183 Address:a3002cc2-0a65-b208-2120-c438cc6a3183} {Suffrage:Voter ID:server-84dc496a-2241-47ce-d3da-3f746de8583f Address:84dc496a-2241-47ce-d3da-3f746de8583f}]"
2026-05-25T23:22:29.223Z [INFO]  server-a3002cc2-0a65-b208-2120-c438cc6a3183: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2 Address:2a0d8958-1e44-8ac0-abc3-a6a1947e42d2} {Suffrage:Voter ID:server-a3002cc2-0a65-b208-2120-c438cc6a3183 Address:a3002cc2-0a65-b208-2120-c438cc6a3183} {Suffrage:Voter ID:server-84dc496a-2241-47ce-d3da-3f746de8583f Address:84dc496a-2241-47ce-d3da-3f746de8583f}]"
2026-05-25T23:22:29.223Z [INFO]  server-a3002cc2-0a65-b208-2120-c438cc6a3183: entering follower state: follower="Node at a3002cc2-0a65-b208-2120-c438cc6a3183 [Follower]" leader-address= leader-id=
2026-05-25T23:22:29.223Z [INFO]  server-84dc496a-2241-47ce-d3da-3f746de8583f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2 Address:2a0d8958-1e44-8ac0-abc3-a6a1947e42d2} {Suffrage:Voter ID:server-a3002cc2-0a65-b208-2120-c438cc6a3183 Address:a3002cc2-0a65-b208-2120-c438cc6a3183} {Suffrage:Voter ID:server-84dc496a-2241-47ce-d3da-3f746de8583f Address:84dc496a-2241-47ce-d3da-3f746de8583f}]"
2026-05-25T23:22:29.223Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:29.223Z [INFO]  server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2: entering follower state: follower="Node at 2a0d8958-1e44-8ac0-abc3-a6a1947e42d2 [Follower]" leader-address= leader-id=
2026-05-25T23:22:29.223Z [INFO]  server-84dc496a-2241-47ce-d3da-3f746de8583f: entering follower state: follower="Node at 84dc496a-2241-47ce-d3da-3f746de8583f [Follower]" leader-address= leader-id=
2026-05-25T23:22:29.279Z [WARN]  server-84dc496a-2241-47ce-d3da-3f746de8583f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:29.279Z [INFO]  server-84dc496a-2241-47ce-d3da-3f746de8583f: entering candidate state: node="Node at 84dc496a-2241-47ce-d3da-3f746de8583f [Candidate]" term=2
2026-05-25T23:22:29.279Z [INFO]  server-84dc496a-2241-47ce-d3da-3f746de8583f: election won: term=2 tally=2
2026-05-25T23:22:29.279Z [INFO]  server-84dc496a-2241-47ce-d3da-3f746de8583f: entering leader state: leader="Node at 84dc496a-2241-47ce-d3da-3f746de8583f [Leader]"
2026-05-25T23:22:29.279Z [INFO]  server-84dc496a-2241-47ce-d3da-3f746de8583f: added peer, starting replication: peer=server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2
2026-05-25T23:22:29.279Z [INFO]  server-84dc496a-2241-47ce-d3da-3f746de8583f: added peer, starting replication: peer=server-a3002cc2-0a65-b208-2120-c438cc6a3183
2026-05-25T23:22:29.279Z [INFO]  server-84dc496a-2241-47ce-d3da-3f746de8583f: pipelining replication: peer="{Voter server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2 2a0d8958-1e44-8ac0-abc3-a6a1947e42d2}"
2026-05-25T23:22:29.279Z [INFO]  server-84dc496a-2241-47ce-d3da-3f746de8583f: pipelining replication: peer="{Voter server-a3002cc2-0a65-b208-2120-c438cc6a3183 a3002cc2-0a65-b208-2120-c438cc6a3183}"
2026-05-25T23:22:29.385Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:29.279799549 +0000 UTC m=+4.926247004 (1 nodes), 56.327127ms from start of poll, 56.627981ms from cluster start. Timeout at 2026-05-25 23:22:29.384802246 +0000 UTC m=+5.031249699, 105.002695ms after stability
2026-05-25T23:22:29.385Z [INFO]  server-84dc496a-2241-47ce-d3da-3f746de8583f: aborting pipeline replication: peer="{Voter server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2 2a0d8958-1e44-8ac0-abc3-a6a1947e42d2}"
2026-05-25T23:22:29.385Z [INFO]  server-84dc496a-2241-47ce-d3da-3f746de8583f: aborting pipeline replication: peer="{Voter server-a3002cc2-0a65-b208-2120-c438cc6a3183 a3002cc2-0a65-b208-2120-c438cc6a3183}"
2026-05-25T23:22:29.385Z [INFO]  creating new snapshot: path=/tmp/raft858542488/snapshots/2-2-1779751349385.tmp
2026-05-25T23:22:29.386Z [INFO]  server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2: starting restore from snapshot: id=2-2-1779751349385 last-index=2 last-term=2 size-in-bytes=1
2026-05-25T23:22:29.386Z [INFO]  server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2: snapshot restore progress: id=2-2-1779751349385 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2026-05-25T23:22:29.386Z [INFO]  server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2: restored from snapshot: id=2-2-1779751349385 last-index=2 last-term=2 size-in-bytes=1
2026-05-25T23:22:29.386Z [INFO]  server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2 Address:2a0d8958-1e44-8ac0-abc3-a6a1947e42d2} {Suffrage:Voter ID:server-a3002cc2-0a65-b208-2120-c438cc6a3183 Address:a3002cc2-0a65-b208-2120-c438cc6a3183} {Suffrage:Voter ID:server-84dc496a-2241-47ce-d3da-3f746de8583f Address:84dc496a-2241-47ce-d3da-3f746de8583f}]"
2026-05-25T23:22:29.386Z [INFO]  creating new snapshot: path=/tmp/raft1405734761/snapshots/2-2-1779751349386.tmp
2026-05-25T23:22:29.386Z [INFO]  server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2: entering follower state: follower="Node at 2a0d8958-1e44-8ac0-abc3-a6a1947e42d2 [Follower]" leader-address= leader-id=
2026-05-25T23:22:29.387Z [INFO]  server-a3002cc2-0a65-b208-2120-c438cc6a3183: starting restore from snapshot: id=2-2-1779751349386 last-index=2 last-term=2 size-in-bytes=1
2026-05-25T23:22:29.387Z [INFO]  server-a3002cc2-0a65-b208-2120-c438cc6a3183: snapshot restore progress: id=2-2-1779751349386 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2026-05-25T23:22:29.388Z [INFO]  server-a3002cc2-0a65-b208-2120-c438cc6a3183: restored from snapshot: id=2-2-1779751349386 last-index=2 last-term=2 size-in-bytes=1
2026-05-25T23:22:29.388Z [INFO]  server-a3002cc2-0a65-b208-2120-c438cc6a3183: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2 Address:2a0d8958-1e44-8ac0-abc3-a6a1947e42d2} {Suffrage:Voter ID:server-a3002cc2-0a65-b208-2120-c438cc6a3183 Address:a3002cc2-0a65-b208-2120-c438cc6a3183} {Suffrage:Voter ID:server-84dc496a-2241-47ce-d3da-3f746de8583f Address:84dc496a-2241-47ce-d3da-3f746de8583f}]"
2026-05-25T23:22:29.388Z [INFO]  server-a3002cc2-0a65-b208-2120-c438cc6a3183: entering follower state: follower="Node at a3002cc2-0a65-b208-2120-c438cc6a3183 [Follower]" leader-address= leader-id=
2026-05-25T23:22:29.388Z [INFO]  creating new snapshot: path=/tmp/raft2022574615/snapshots/2-2-1779751349388.tmp
2026-05-25T23:22:29.390Z [INFO]  server-84dc496a-2241-47ce-d3da-3f746de8583f: starting restore from snapshot: id=2-2-1779751349388 last-index=2 last-term=2 size-in-bytes=1
2026-05-25T23:22:29.390Z [INFO]  server-84dc496a-2241-47ce-d3da-3f746de8583f: snapshot restore progress: id=2-2-1779751349388 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2026-05-25T23:22:29.390Z [INFO]  server-84dc496a-2241-47ce-d3da-3f746de8583f: restored from snapshot: id=2-2-1779751349388 last-index=2 last-term=2 size-in-bytes=1
2026-05-25T23:22:29.390Z [INFO]  server-84dc496a-2241-47ce-d3da-3f746de8583f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2 Address:2a0d8958-1e44-8ac0-abc3-a6a1947e42d2} {Suffrage:Voter ID:server-a3002cc2-0a65-b208-2120-c438cc6a3183 Address:a3002cc2-0a65-b208-2120-c438cc6a3183} {Suffrage:Voter ID:server-84dc496a-2241-47ce-d3da-3f746de8583f Address:84dc496a-2241-47ce-d3da-3f746de8583f}]"
2026-05-25T23:22:29.390Z [INFO]  server-84dc496a-2241-47ce-d3da-3f746de8583f: entering follower state: follower="Node at 84dc496a-2241-47ce-d3da-3f746de8583f [Follower]" leader-address= leader-id=
2026-05-25T23:22:29.463Z [WARN]  server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:29.463Z [INFO]  server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2: entering candidate state: node="Node at 2a0d8958-1e44-8ac0-abc3-a6a1947e42d2 [Candidate]" term=3
2026-05-25T23:22:29.463Z [INFO]  server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2: election won: term=3 tally=2
2026-05-25T23:22:29.463Z [INFO]  server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2: entering leader state: leader="Node at 2a0d8958-1e44-8ac0-abc3-a6a1947e42d2 [Leader]"
2026-05-25T23:22:29.463Z [INFO]  server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2: added peer, starting replication: peer=server-a3002cc2-0a65-b208-2120-c438cc6a3183
2026-05-25T23:22:29.463Z [INFO]  server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2: added peer, starting replication: peer=server-84dc496a-2241-47ce-d3da-3f746de8583f
2026-05-25T23:22:29.463Z [INFO]  server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2: pipelining replication: peer="{Voter server-84dc496a-2241-47ce-d3da-3f746de8583f 84dc496a-2241-47ce-d3da-3f746de8583f}"
2026-05-25T23:22:29.463Z [INFO]  server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2: pipelining replication: peer="{Voter server-a3002cc2-0a65-b208-2120-c438cc6a3183 a3002cc2-0a65-b208-2120-c438cc6a3183}"
2026-05-25T23:22:29.705Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:29.811Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:29.706000449 +0000 UTC m=+5.352447908 (1 nodes), 49.823µs from start of poll, 482.828885ms from cluster start. Timeout at 2026-05-25 23:22:29.811018526 +0000 UTC m=+5.457465999, 105.018091ms after stability
2026-05-25T23:22:29.811Z [INFO]  server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2: aborting pipeline replication: peer="{Voter server-84dc496a-2241-47ce-d3da-3f746de8583f 84dc496a-2241-47ce-d3da-3f746de8583f}"
2026-05-25T23:22:29.811Z [INFO]  server-2a0d8958-1e44-8ac0-abc3-a6a1947e42d2: aborting pipeline replication: peer="{Voter server-a3002cc2-0a65-b208-2120-c438cc6a3183 a3002cc2-0a65-b208-2120-c438cc6a3183}"
=== RUN   TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs
2026-05-25T23:22:29.812Z [INFO]  server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719 Address:0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719} {Suffrage:Voter ID:server-50466357-868c-c39c-c5bb-fc9e1b595324 Address:50466357-868c-c39c-c5bb-fc9e1b595324} {Suffrage:Voter ID:server-b4f814bd-5a38-ac2b-9f93-41782724fadb Address:b4f814bd-5a38-ac2b-9f93-41782724fadb}]"
2026-05-25T23:22:29.812Z [INFO]  server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719: entering follower state: follower="Node at 0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719 [Follower]" leader-address= leader-id=
2026-05-25T23:22:29.812Z [INFO]  server-50466357-868c-c39c-c5bb-fc9e1b595324: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719 Address:0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719} {Suffrage:Voter ID:server-50466357-868c-c39c-c5bb-fc9e1b595324 Address:50466357-868c-c39c-c5bb-fc9e1b595324} {Suffrage:Voter ID:server-b4f814bd-5a38-ac2b-9f93-41782724fadb Address:b4f814bd-5a38-ac2b-9f93-41782724fadb}]"
2026-05-25T23:22:29.813Z [INFO]  server-50466357-868c-c39c-c5bb-fc9e1b595324: entering follower state: follower="Node at 50466357-868c-c39c-c5bb-fc9e1b595324 [Follower]" leader-address= leader-id=
2026-05-25T23:22:29.813Z [INFO]  server-b4f814bd-5a38-ac2b-9f93-41782724fadb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719 Address:0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719} {Suffrage:Voter ID:server-50466357-868c-c39c-c5bb-fc9e1b595324 Address:50466357-868c-c39c-c5bb-fc9e1b595324} {Suffrage:Voter ID:server-b4f814bd-5a38-ac2b-9f93-41782724fadb Address:b4f814bd-5a38-ac2b-9f93-41782724fadb}]"
2026-05-25T23:22:29.813Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:29.813Z [INFO]  server-b4f814bd-5a38-ac2b-9f93-41782724fadb: entering follower state: follower="Node at b4f814bd-5a38-ac2b-9f93-41782724fadb [Follower]" leader-address= leader-id=
2026-05-25T23:22:29.876Z [WARN]  server-50466357-868c-c39c-c5bb-fc9e1b595324: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:29.876Z [INFO]  server-50466357-868c-c39c-c5bb-fc9e1b595324: entering candidate state: node="Node at 50466357-868c-c39c-c5bb-fc9e1b595324 [Candidate]" term=2
2026-05-25T23:22:29.877Z [INFO]  server-50466357-868c-c39c-c5bb-fc9e1b595324: election won: term=2 tally=2
2026-05-25T23:22:29.877Z [INFO]  server-50466357-868c-c39c-c5bb-fc9e1b595324: entering leader state: leader="Node at 50466357-868c-c39c-c5bb-fc9e1b595324 [Leader]"
2026-05-25T23:22:29.877Z [INFO]  server-50466357-868c-c39c-c5bb-fc9e1b595324: added peer, starting replication: peer=server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719
2026-05-25T23:22:29.877Z [INFO]  server-50466357-868c-c39c-c5bb-fc9e1b595324: added peer, starting replication: peer=server-b4f814bd-5a38-ac2b-9f93-41782724fadb
2026-05-25T23:22:29.877Z [INFO]  server-50466357-868c-c39c-c5bb-fc9e1b595324: pipelining replication: peer="{Voter server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719 0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719}"
2026-05-25T23:22:29.877Z [INFO]  server-50466357-868c-c39c-c5bb-fc9e1b595324: pipelining replication: peer="{Voter server-b4f814bd-5a38-ac2b-9f93-41782724fadb b4f814bd-5a38-ac2b-9f93-41782724fadb}"
2026-05-25T23:22:29.983Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:29.877761014 +0000 UTC m=+5.524208481 (1 nodes), 64.538505ms from start of poll, 65.060062ms from cluster start. Timeout at 2026-05-25 23:22:29.982763926 +0000 UTC m=+5.629211381, 105.0029ms after stability
2026-05-25T23:22:29.983Z [INFO]  server-50466357-868c-c39c-c5bb-fc9e1b595324: aborting pipeline replication: peer="{Voter server-b4f814bd-5a38-ac2b-9f93-41782724fadb b4f814bd-5a38-ac2b-9f93-41782724fadb}"
2026-05-25T23:22:29.983Z [INFO]  server-50466357-868c-c39c-c5bb-fc9e1b595324: aborting pipeline replication: peer="{Voter server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719 0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719}"
2026-05-25T23:22:29.983Z [INFO]  creating new snapshot: path=/tmp/raft1480314709/snapshots/2-6-1779751349983.tmp
2026-05-25T23:22:29.984Z [INFO]  server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719: starting restore from snapshot: id=2-6-1779751349983 last-index=6 last-term=2 size-in-bytes=25
2026-05-25T23:22:29.984Z [INFO]  server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719: snapshot restore progress: id=2-6-1779751349983 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2026-05-25T23:22:29.984Z [INFO]  server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719: restored from snapshot: id=2-6-1779751349983 last-index=6 last-term=2 size-in-bytes=25
2026-05-25T23:22:29.984Z [INFO]  server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719 Address:0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719} {Suffrage:Voter ID:server-50466357-868c-c39c-c5bb-fc9e1b595324 Address:50466357-868c-c39c-c5bb-fc9e1b595324} {Suffrage:Voter ID:server-b4f814bd-5a38-ac2b-9f93-41782724fadb Address:b4f814bd-5a38-ac2b-9f93-41782724fadb}]"
2026-05-25T23:22:29.984Z [INFO]  creating new snapshot: path=/tmp/raft1162294647/snapshots/2-6-1779751349984.tmp
2026-05-25T23:22:29.984Z [INFO]  server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719: entering follower state: follower="Node at 0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719 [Follower]" leader-address= leader-id=
2026-05-25T23:22:29.986Z [INFO]  server-50466357-868c-c39c-c5bb-fc9e1b595324: starting restore from snapshot: id=2-6-1779751349984 last-index=6 last-term=2 size-in-bytes=25
2026-05-25T23:22:29.986Z [INFO]  server-50466357-868c-c39c-c5bb-fc9e1b595324: snapshot restore progress: id=2-6-1779751349984 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2026-05-25T23:22:29.986Z [INFO]  server-50466357-868c-c39c-c5bb-fc9e1b595324: restored from snapshot: id=2-6-1779751349984 last-index=6 last-term=2 size-in-bytes=25
2026-05-25T23:22:29.986Z [INFO]  server-50466357-868c-c39c-c5bb-fc9e1b595324: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719 Address:0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719} {Suffrage:Voter ID:server-50466357-868c-c39c-c5bb-fc9e1b595324 Address:50466357-868c-c39c-c5bb-fc9e1b595324} {Suffrage:Voter ID:server-b4f814bd-5a38-ac2b-9f93-41782724fadb Address:b4f814bd-5a38-ac2b-9f93-41782724fadb}]"
2026-05-25T23:22:29.986Z [INFO]  server-50466357-868c-c39c-c5bb-fc9e1b595324: entering follower state: follower="Node at 50466357-868c-c39c-c5bb-fc9e1b595324 [Follower]" leader-address= leader-id=
2026-05-25T23:22:29.987Z [INFO]  creating new snapshot: path=/tmp/raft1377128238/snapshots/2-6-1779751349987.tmp
2026-05-25T23:22:29.987Z [INFO]  server-b4f814bd-5a38-ac2b-9f93-41782724fadb: starting restore from snapshot: id=2-6-1779751349987 last-index=6 last-term=2 size-in-bytes=25
2026-05-25T23:22:29.987Z [INFO]  server-b4f814bd-5a38-ac2b-9f93-41782724fadb: snapshot restore progress: id=2-6-1779751349987 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2026-05-25T23:22:29.987Z [INFO]  server-b4f814bd-5a38-ac2b-9f93-41782724fadb: restored from snapshot: id=2-6-1779751349987 last-index=6 last-term=2 size-in-bytes=25
2026-05-25T23:22:29.987Z [INFO]  server-b4f814bd-5a38-ac2b-9f93-41782724fadb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719 Address:0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719} {Suffrage:Voter ID:server-50466357-868c-c39c-c5bb-fc9e1b595324 Address:50466357-868c-c39c-c5bb-fc9e1b595324} {Suffrage:Voter ID:server-b4f814bd-5a38-ac2b-9f93-41782724fadb Address:b4f814bd-5a38-ac2b-9f93-41782724fadb}]"
2026-05-25T23:22:29.987Z [INFO]  server-b4f814bd-5a38-ac2b-9f93-41782724fadb: entering follower state: follower="Node at b4f814bd-5a38-ac2b-9f93-41782724fadb [Follower]" leader-address= leader-id=
2026-05-25T23:22:30.041Z [WARN]  server-b4f814bd-5a38-ac2b-9f93-41782724fadb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:30.041Z [INFO]  server-b4f814bd-5a38-ac2b-9f93-41782724fadb: entering candidate state: node="Node at b4f814bd-5a38-ac2b-9f93-41782724fadb [Candidate]" term=3
2026-05-25T23:22:30.041Z [INFO]  server-b4f814bd-5a38-ac2b-9f93-41782724fadb: election won: term=3 tally=2
2026-05-25T23:22:30.042Z [INFO]  server-b4f814bd-5a38-ac2b-9f93-41782724fadb: entering leader state: leader="Node at b4f814bd-5a38-ac2b-9f93-41782724fadb [Leader]"
2026-05-25T23:22:30.042Z [INFO]  server-b4f814bd-5a38-ac2b-9f93-41782724fadb: added peer, starting replication: peer=server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719
2026-05-25T23:22:30.042Z [INFO]  server-b4f814bd-5a38-ac2b-9f93-41782724fadb: added peer, starting replication: peer=server-50466357-868c-c39c-c5bb-fc9e1b595324
2026-05-25T23:22:30.042Z [INFO]  server-b4f814bd-5a38-ac2b-9f93-41782724fadb: pipelining replication: peer="{Voter server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719 0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719}"
2026-05-25T23:22:30.042Z [INFO]  server-b4f814bd-5a38-ac2b-9f93-41782724fadb: pipelining replication: peer="{Voter server-50466357-868c-c39c-c5bb-fc9e1b595324 50466357-868c-c39c-c5bb-fc9e1b595324}"
2026-05-25T23:22:30.303Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:30.409Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:30.303682668 +0000 UTC m=+5.950130128 (1 nodes), 63.351µs from start of poll, 490.981709ms from cluster start. Timeout at 2026-05-25 23:22:30.408686757 +0000 UTC m=+6.055134212, 105.004084ms after stability
2026-05-25T23:22:30.409Z [INFO]  server-b4f814bd-5a38-ac2b-9f93-41782724fadb: aborting pipeline replication: peer="{Voter server-50466357-868c-c39c-c5bb-fc9e1b595324 50466357-868c-c39c-c5bb-fc9e1b595324}"
2026-05-25T23:22:30.409Z [INFO]  server-b4f814bd-5a38-ac2b-9f93-41782724fadb: aborting pipeline replication: peer="{Voter server-0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719 0e9d2d1a-14f7-13ad-cc6d-ce00f88ed719}"
=== RUN   TestRaft_RecoverCluster/snapshot,_with_trailing_logs
2026-05-25T23:22:30.410Z [INFO]  server-5785b9cf-93de-260c-c358-f3fc65c4d2ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5785b9cf-93de-260c-c358-f3fc65c4d2ce Address:5785b9cf-93de-260c-c358-f3fc65c4d2ce} {Suffrage:Voter ID:server-517d9170-db11-0278-61d6-20fedd7c725d Address:517d9170-db11-0278-61d6-20fedd7c725d} {Suffrage:Voter ID:server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e Address:7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e}]"
2026-05-25T23:22:30.410Z [INFO]  server-517d9170-db11-0278-61d6-20fedd7c725d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5785b9cf-93de-260c-c358-f3fc65c4d2ce Address:5785b9cf-93de-260c-c358-f3fc65c4d2ce} {Suffrage:Voter ID:server-517d9170-db11-0278-61d6-20fedd7c725d Address:517d9170-db11-0278-61d6-20fedd7c725d} {Suffrage:Voter ID:server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e Address:7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e}]"
2026-05-25T23:22:30.410Z [INFO]  server-5785b9cf-93de-260c-c358-f3fc65c4d2ce: entering follower state: follower="Node at 5785b9cf-93de-260c-c358-f3fc65c4d2ce [Follower]" leader-address= leader-id=
2026-05-25T23:22:30.410Z [INFO]  server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5785b9cf-93de-260c-c358-f3fc65c4d2ce Address:5785b9cf-93de-260c-c358-f3fc65c4d2ce} {Suffrage:Voter ID:server-517d9170-db11-0278-61d6-20fedd7c725d Address:517d9170-db11-0278-61d6-20fedd7c725d} {Suffrage:Voter ID:server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e Address:7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e}]"
2026-05-25T23:22:30.410Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:30.410Z [INFO]  server-517d9170-db11-0278-61d6-20fedd7c725d: entering follower state: follower="Node at 517d9170-db11-0278-61d6-20fedd7c725d [Follower]" leader-address= leader-id=
2026-05-25T23:22:30.410Z [INFO]  server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e: entering follower state: follower="Node at 7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e [Follower]" leader-address= leader-id=
2026-05-25T23:22:30.480Z [WARN]  server-517d9170-db11-0278-61d6-20fedd7c725d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:30.480Z [INFO]  server-517d9170-db11-0278-61d6-20fedd7c725d: entering candidate state: node="Node at 517d9170-db11-0278-61d6-20fedd7c725d [Candidate]" term=2
2026-05-25T23:22:30.480Z [INFO]  server-517d9170-db11-0278-61d6-20fedd7c725d: election won: term=2 tally=2
2026-05-25T23:22:30.480Z [INFO]  server-517d9170-db11-0278-61d6-20fedd7c725d: entering leader state: leader="Node at 517d9170-db11-0278-61d6-20fedd7c725d [Leader]"
2026-05-25T23:22:30.480Z [INFO]  server-517d9170-db11-0278-61d6-20fedd7c725d: added peer, starting replication: peer=server-5785b9cf-93de-260c-c358-f3fc65c4d2ce
2026-05-25T23:22:30.480Z [INFO]  server-517d9170-db11-0278-61d6-20fedd7c725d: added peer, starting replication: peer=server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e
2026-05-25T23:22:30.480Z [INFO]  server-517d9170-db11-0278-61d6-20fedd7c725d: pipelining replication: peer="{Voter server-5785b9cf-93de-260c-c358-f3fc65c4d2ce 5785b9cf-93de-260c-c358-f3fc65c4d2ce}"
2026-05-25T23:22:30.480Z [INFO]  server-517d9170-db11-0278-61d6-20fedd7c725d: pipelining replication: peer="{Voter server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e 7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e}"
2026-05-25T23:22:30.586Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:30.480481059 +0000 UTC m=+6.126928568 (1 nodes), 69.679375ms from start of poll, 70.040316ms from cluster start. Timeout at 2026-05-25 23:22:30.585484382 +0000 UTC m=+6.231931835, 105.003267ms after stability
2026-05-25T23:22:30.587Z [INFO]  server-517d9170-db11-0278-61d6-20fedd7c725d: aborting pipeline replication: peer="{Voter server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e 7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e}"
2026-05-25T23:22:30.587Z [INFO]  server-517d9170-db11-0278-61d6-20fedd7c725d: aborting pipeline replication: peer="{Voter server-5785b9cf-93de-260c-c358-f3fc65c4d2ce 5785b9cf-93de-260c-c358-f3fc65c4d2ce}"
2026-05-25T23:22:30.587Z [INFO]  creating new snapshot: path=/tmp/raft1141293556/snapshots/2-27-1779751350587.tmp
2026-05-25T23:22:30.588Z [INFO]  server-5785b9cf-93de-260c-c358-f3fc65c4d2ce: starting restore from snapshot: id=2-27-1779751350587 last-index=27 last-term=2 size-in-bytes=168
2026-05-25T23:22:30.588Z [INFO]  server-5785b9cf-93de-260c-c358-f3fc65c4d2ce: snapshot restore progress: id=2-27-1779751350587 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2026-05-25T23:22:30.588Z [INFO]  server-5785b9cf-93de-260c-c358-f3fc65c4d2ce: restored from snapshot: id=2-27-1779751350587 last-index=27 last-term=2 size-in-bytes=168
2026-05-25T23:22:30.588Z [INFO]  server-5785b9cf-93de-260c-c358-f3fc65c4d2ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5785b9cf-93de-260c-c358-f3fc65c4d2ce Address:5785b9cf-93de-260c-c358-f3fc65c4d2ce} {Suffrage:Voter ID:server-517d9170-db11-0278-61d6-20fedd7c725d Address:517d9170-db11-0278-61d6-20fedd7c725d} {Suffrage:Voter ID:server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e Address:7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e}]"
2026-05-25T23:22:30.588Z [INFO]  server-5785b9cf-93de-260c-c358-f3fc65c4d2ce: entering follower state: follower="Node at 5785b9cf-93de-260c-c358-f3fc65c4d2ce [Follower]" leader-address= leader-id=
2026-05-25T23:22:30.588Z [INFO]  creating new snapshot: path=/tmp/raft356189008/snapshots/2-27-1779751350588.tmp
2026-05-25T23:22:30.589Z [INFO]  server-517d9170-db11-0278-61d6-20fedd7c725d: starting restore from snapshot: id=2-27-1779751350588 last-index=27 last-term=2 size-in-bytes=168
2026-05-25T23:22:30.589Z [INFO]  server-517d9170-db11-0278-61d6-20fedd7c725d: snapshot restore progress: id=2-27-1779751350588 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2026-05-25T23:22:30.589Z [INFO]  server-517d9170-db11-0278-61d6-20fedd7c725d: restored from snapshot: id=2-27-1779751350588 last-index=27 last-term=2 size-in-bytes=168
2026-05-25T23:22:30.589Z [INFO]  server-517d9170-db11-0278-61d6-20fedd7c725d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5785b9cf-93de-260c-c358-f3fc65c4d2ce Address:5785b9cf-93de-260c-c358-f3fc65c4d2ce} {Suffrage:Voter ID:server-517d9170-db11-0278-61d6-20fedd7c725d Address:517d9170-db11-0278-61d6-20fedd7c725d} {Suffrage:Voter ID:server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e Address:7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e}]"
2026-05-25T23:22:30.589Z [INFO]  server-517d9170-db11-0278-61d6-20fedd7c725d: entering follower state: follower="Node at 517d9170-db11-0278-61d6-20fedd7c725d [Follower]" leader-address= leader-id=
2026-05-25T23:22:30.589Z [INFO]  creating new snapshot: path=/tmp/raft3129702465/snapshots/2-27-1779751350589.tmp
2026-05-25T23:22:30.590Z [INFO]  server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e: starting restore from snapshot: id=2-27-1779751350589 last-index=27 last-term=2 size-in-bytes=168
2026-05-25T23:22:30.590Z [INFO]  server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e: snapshot restore progress: id=2-27-1779751350589 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2026-05-25T23:22:30.590Z [INFO]  server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e: restored from snapshot: id=2-27-1779751350589 last-index=27 last-term=2 size-in-bytes=168
2026-05-25T23:22:30.590Z [INFO]  server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5785b9cf-93de-260c-c358-f3fc65c4d2ce Address:5785b9cf-93de-260c-c358-f3fc65c4d2ce} {Suffrage:Voter ID:server-517d9170-db11-0278-61d6-20fedd7c725d Address:517d9170-db11-0278-61d6-20fedd7c725d} {Suffrage:Voter ID:server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e Address:7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e}]"
2026-05-25T23:22:30.591Z [INFO]  server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e: entering follower state: follower="Node at 7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e [Follower]" leader-address= leader-id=
2026-05-25T23:22:30.649Z [WARN]  server-5785b9cf-93de-260c-c358-f3fc65c4d2ce: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:30.649Z [INFO]  server-5785b9cf-93de-260c-c358-f3fc65c4d2ce: entering candidate state: node="Node at 5785b9cf-93de-260c-c358-f3fc65c4d2ce [Candidate]" term=3
2026-05-25T23:22:30.649Z [INFO]  server-5785b9cf-93de-260c-c358-f3fc65c4d2ce: election won: term=3 tally=2
2026-05-25T23:22:30.649Z [INFO]  server-5785b9cf-93de-260c-c358-f3fc65c4d2ce: entering leader state: leader="Node at 5785b9cf-93de-260c-c358-f3fc65c4d2ce [Leader]"
2026-05-25T23:22:30.649Z [INFO]  server-5785b9cf-93de-260c-c358-f3fc65c4d2ce: added peer, starting replication: peer=server-517d9170-db11-0278-61d6-20fedd7c725d
2026-05-25T23:22:30.649Z [INFO]  server-5785b9cf-93de-260c-c358-f3fc65c4d2ce: added peer, starting replication: peer=server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e
2026-05-25T23:22:30.649Z [INFO]  server-5785b9cf-93de-260c-c358-f3fc65c4d2ce: pipelining replication: peer="{Voter server-517d9170-db11-0278-61d6-20fedd7c725d 517d9170-db11-0278-61d6-20fedd7c725d}"
2026-05-25T23:22:30.649Z [INFO]  server-5785b9cf-93de-260c-c358-f3fc65c4d2ce: pipelining replication: peer="{Voter server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e 7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e}"
2026-05-25T23:22:30.906Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:31.012Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:30.90655618 +0000 UTC m=+6.553003671 (1 nodes), 40.738µs from start of poll, 496.115419ms from cluster start. Timeout at 2026-05-25 23:22:31.011571626 +0000 UTC m=+6.658019116, 105.015445ms after stability
2026-05-25T23:22:31.012Z [INFO]  server-5785b9cf-93de-260c-c358-f3fc65c4d2ce: aborting pipeline replication: peer="{Voter server-517d9170-db11-0278-61d6-20fedd7c725d 517d9170-db11-0278-61d6-20fedd7c725d}"
2026-05-25T23:22:31.012Z [INFO]  server-5785b9cf-93de-260c-c358-f3fc65c4d2ce: aborting pipeline replication: peer="{Voter server-7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e 7f6c7159-20e1-e52d-c6c2-efc2a40a6e2e}"
--- PASS: TestRaft_RecoverCluster (1.79s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.59s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.60s)
    --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.60s)
=== RUN   TestRaft_HasExistingState
    raft_test.go:258: Skipping unreliable test
--- SKIP: TestRaft_HasExistingState (0.00s)
=== RUN   TestRaft_SingleNode
2026-05-25T23:22:31.013Z [INFO]  server-e41eac7d-e009-5ad7-0904-cdfdf96e1831: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e41eac7d-e009-5ad7-0904-cdfdf96e1831 Address:e41eac7d-e009-5ad7-0904-cdfdf96e1831}]"
2026-05-25T23:22:31.013Z [INFO]  server-e41eac7d-e009-5ad7-0904-cdfdf96e1831: entering follower state: follower="Node at e41eac7d-e009-5ad7-0904-cdfdf96e1831 [Follower]" leader-address= leader-id=
2026-05-25T23:22:31.107Z [WARN]  server-e41eac7d-e009-5ad7-0904-cdfdf96e1831: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:31.107Z [INFO]  server-e41eac7d-e009-5ad7-0904-cdfdf96e1831: entering candidate state: node="Node at e41eac7d-e009-5ad7-0904-cdfdf96e1831 [Candidate]" term=2
2026-05-25T23:22:31.107Z [INFO]  server-e41eac7d-e009-5ad7-0904-cdfdf96e1831: election won: term=2 tally=1
2026-05-25T23:22:31.107Z [INFO]  server-e41eac7d-e009-5ad7-0904-cdfdf96e1831: entering leader state: leader="Node at e41eac7d-e009-5ad7-0904-cdfdf96e1831 [Leader]"
--- PASS: TestRaft_SingleNode (0.09s)
=== RUN   TestRaft_TripleNode
2026-05-25T23:22:31.108Z [INFO]  server-4fe48864-8135-739f-d2c4-0a74fe3bf10f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fe48864-8135-739f-d2c4-0a74fe3bf10f Address:4fe48864-8135-739f-d2c4-0a74fe3bf10f} {Suffrage:Voter ID:server-e7a0c691-7d03-70a0-012f-804b5955ac4e Address:e7a0c691-7d03-70a0-012f-804b5955ac4e} {Suffrage:Voter ID:server-2434411b-13fc-ce22-defa-9951af6b48dc Address:2434411b-13fc-ce22-defa-9951af6b48dc}]"
2026-05-25T23:22:31.109Z [INFO]  server-4fe48864-8135-739f-d2c4-0a74fe3bf10f: entering follower state: follower="Node at 4fe48864-8135-739f-d2c4-0a74fe3bf10f [Follower]" leader-address= leader-id=
2026-05-25T23:22:31.109Z [INFO]  server-e7a0c691-7d03-70a0-012f-804b5955ac4e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fe48864-8135-739f-d2c4-0a74fe3bf10f Address:4fe48864-8135-739f-d2c4-0a74fe3bf10f} {Suffrage:Voter ID:server-e7a0c691-7d03-70a0-012f-804b5955ac4e Address:e7a0c691-7d03-70a0-012f-804b5955ac4e} {Suffrage:Voter ID:server-2434411b-13fc-ce22-defa-9951af6b48dc Address:2434411b-13fc-ce22-defa-9951af6b48dc}]"
2026-05-25T23:22:31.109Z [INFO]  server-e7a0c691-7d03-70a0-012f-804b5955ac4e: entering follower state: follower="Node at e7a0c691-7d03-70a0-012f-804b5955ac4e [Follower]" leader-address= leader-id=
2026-05-25T23:22:31.109Z [INFO]  server-2434411b-13fc-ce22-defa-9951af6b48dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fe48864-8135-739f-d2c4-0a74fe3bf10f Address:4fe48864-8135-739f-d2c4-0a74fe3bf10f} {Suffrage:Voter ID:server-e7a0c691-7d03-70a0-012f-804b5955ac4e Address:e7a0c691-7d03-70a0-012f-804b5955ac4e} {Suffrage:Voter ID:server-2434411b-13fc-ce22-defa-9951af6b48dc Address:2434411b-13fc-ce22-defa-9951af6b48dc}]"
2026-05-25T23:22:31.109Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:22:31.109Z [INFO]  server-2434411b-13fc-ce22-defa-9951af6b48dc: entering follower state: follower="Node at 2434411b-13fc-ce22-defa-9951af6b48dc [Follower]" leader-address= leader-id=
2026-05-25T23:22:31.163Z [WARN]  server-2434411b-13fc-ce22-defa-9951af6b48dc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:31.163Z [INFO]  server-2434411b-13fc-ce22-defa-9951af6b48dc: entering candidate state: node="Node at 2434411b-13fc-ce22-defa-9951af6b48dc [Candidate]" term=2
2026-05-25T23:22:31.163Z [INFO]  server-2434411b-13fc-ce22-defa-9951af6b48dc: election won: term=2 tally=2
2026-05-25T23:22:31.163Z [INFO]  server-2434411b-13fc-ce22-defa-9951af6b48dc: entering leader state: leader="Node at 2434411b-13fc-ce22-defa-9951af6b48dc [Leader]"
2026-05-25T23:22:31.163Z [INFO]  server-2434411b-13fc-ce22-defa-9951af6b48dc: added peer, starting replication: peer=server-4fe48864-8135-739f-d2c4-0a74fe3bf10f
2026-05-25T23:22:31.163Z [INFO]  server-2434411b-13fc-ce22-defa-9951af6b48dc: added peer, starting replication: peer=server-e7a0c691-7d03-70a0-012f-804b5955ac4e
2026-05-25T23:22:31.163Z [INFO]  server-2434411b-13fc-ce22-defa-9951af6b48dc: pipelining replication: peer="{Voter server-e7a0c691-7d03-70a0-012f-804b5955ac4e e7a0c691-7d03-70a0-012f-804b5955ac4e}"
2026-05-25T23:22:31.163Z [INFO]  server-2434411b-13fc-ce22-defa-9951af6b48dc: pipelining replication: peer="{Voter server-4fe48864-8135-739f-d2c4-0a74fe3bf10f 4fe48864-8135-739f-d2c4-0a74fe3bf10f}"
2026-05-25T23:22:31.269Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:22:31.163268533 +0000 UTC m=+6.809715984 (2 nodes), 53.750644ms from start of poll, 54.954395ms from cluster start. Timeout at 2026-05-25 23:22:31.268270786 +0000 UTC m=+6.914718240, 105.002256ms after stability
2026-05-25T23:22:31.269Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:31.374Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:31.269338608 +0000 UTC m=+6.915786063 (1 nodes), 731ns from start of poll, 161.024474ms from cluster start. Timeout at 2026-05-25 23:22:31.374342953 +0000 UTC m=+7.020790405, 105.004342ms after stability
2026-05-25T23:22:31.391Z [INFO]  server-2434411b-13fc-ce22-defa-9951af6b48dc: aborting pipeline replication: peer="{Voter server-e7a0c691-7d03-70a0-012f-804b5955ac4e e7a0c691-7d03-70a0-012f-804b5955ac4e}"
2026-05-25T23:22:31.391Z [INFO]  server-2434411b-13fc-ce22-defa-9951af6b48dc: aborting pipeline replication: peer="{Voter server-4fe48864-8135-739f-d2c4-0a74fe3bf10f 4fe48864-8135-739f-d2c4-0a74fe3bf10f}"
--- PASS: TestRaft_TripleNode (0.28s)
=== RUN   TestRaft_LeaderFail
2026-05-25T23:22:31.392Z [INFO]  server-05052cbf-2b06-98e6-100b-1c41c60bd392: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05052cbf-2b06-98e6-100b-1c41c60bd392 Address:05052cbf-2b06-98e6-100b-1c41c60bd392} {Suffrage:Voter ID:server-208acc98-b56b-2490-fbfa-98bae06d5e6c Address:208acc98-b56b-2490-fbfa-98bae06d5e6c} {Suffrage:Voter ID:server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d Address:8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d}]"
2026-05-25T23:22:31.392Z [INFO]  server-05052cbf-2b06-98e6-100b-1c41c60bd392: entering follower state: follower="Node at 05052cbf-2b06-98e6-100b-1c41c60bd392 [Follower]" leader-address= leader-id=
2026-05-25T23:22:31.392Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05052cbf-2b06-98e6-100b-1c41c60bd392 Address:05052cbf-2b06-98e6-100b-1c41c60bd392} {Suffrage:Voter ID:server-208acc98-b56b-2490-fbfa-98bae06d5e6c Address:208acc98-b56b-2490-fbfa-98bae06d5e6c} {Suffrage:Voter ID:server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d Address:8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d}]"
2026-05-25T23:22:31.392Z [INFO]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05052cbf-2b06-98e6-100b-1c41c60bd392 Address:05052cbf-2b06-98e6-100b-1c41c60bd392} {Suffrage:Voter ID:server-208acc98-b56b-2490-fbfa-98bae06d5e6c Address:208acc98-b56b-2490-fbfa-98bae06d5e6c} {Suffrage:Voter ID:server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d Address:8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d}]"
2026-05-25T23:22:31.392Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:22:31.392Z [INFO]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: entering follower state: follower="Node at 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d [Follower]" leader-address= leader-id=
2026-05-25T23:22:31.392Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: entering follower state: follower="Node at 208acc98-b56b-2490-fbfa-98bae06d5e6c [Follower]" leader-address= leader-id=
2026-05-25T23:22:31.443Z [WARN]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:31.443Z [INFO]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: entering candidate state: node="Node at 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d [Candidate]" term=2
2026-05-25T23:22:31.443Z [INFO]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: election won: term=2 tally=2
2026-05-25T23:22:31.443Z [INFO]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: entering leader state: leader="Node at 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d [Leader]"
2026-05-25T23:22:31.443Z [INFO]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: added peer, starting replication: peer=server-05052cbf-2b06-98e6-100b-1c41c60bd392
2026-05-25T23:22:31.443Z [INFO]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: added peer, starting replication: peer=server-208acc98-b56b-2490-fbfa-98bae06d5e6c
2026-05-25T23:22:31.444Z [INFO]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: pipelining replication: peer="{Voter server-208acc98-b56b-2490-fbfa-98bae06d5e6c 208acc98-b56b-2490-fbfa-98bae06d5e6c}"
2026-05-25T23:22:31.444Z [INFO]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: pipelining replication: peer="{Voter server-05052cbf-2b06-98e6-100b-1c41c60bd392 05052cbf-2b06-98e6-100b-1c41c60bd392}"
2026-05-25T23:22:31.549Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:22:31.44397914 +0000 UTC m=+7.090426593 (2 nodes), 51.607761ms from start of poll, 51.873585ms from cluster start. Timeout at 2026-05-25 23:22:31.548981783 +0000 UTC m=+7.195429237, 105.002644ms after stability
2026-05-25T23:22:31.549Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:31.655Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:31.549559446 +0000 UTC m=+7.196006900 (1 nodes), 1.99µs from start of poll, 157.453892ms from cluster start. Timeout at 2026-05-25 23:22:31.654571158 +0000 UTC m=+7.301018682, 105.011782ms after stability
    raft_test.go:378: [INFO] Disconnecting Node at 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d [Leader]
2026-05-25T23:22:31.668Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to heartbeat to: peer=208acc98-b56b-2490-fbfa-98bae06d5e6c backoff time=10ms error="failed to connect to peer: 208acc98-b56b-2490-fbfa-98bae06d5e6c"
2026-05-25T23:22:31.668Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:31.670Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to pipeline appendEntries: peer="{Voter server-05052cbf-2b06-98e6-100b-1c41c60bd392 05052cbf-2b06-98e6-100b-1c41c60bd392}" error="append pipeline closed"
2026-05-25T23:22:31.670Z [INFO]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: aborting pipeline replication: peer="{Voter server-05052cbf-2b06-98e6-100b-1c41c60bd392 05052cbf-2b06-98e6-100b-1c41c60bd392}"
2026-05-25T23:22:31.672Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to pipeline appendEntries: peer="{Voter server-208acc98-b56b-2490-fbfa-98bae06d5e6c 208acc98-b56b-2490-fbfa-98bae06d5e6c}" error="append pipeline closed"
2026-05-25T23:22:31.672Z [INFO]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: aborting pipeline replication: peer="{Voter server-208acc98-b56b-2490-fbfa-98bae06d5e6c 208acc98-b56b-2490-fbfa-98bae06d5e6c}"
2026-05-25T23:22:31.674Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to heartbeat to: peer=05052cbf-2b06-98e6-100b-1c41c60bd392 backoff time=10ms error="failed to connect to peer: 05052cbf-2b06-98e6-100b-1c41c60bd392"
2026-05-25T23:22:31.678Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to appendEntries to: peer="{Voter server-05052cbf-2b06-98e6-100b-1c41c60bd392 05052cbf-2b06-98e6-100b-1c41c60bd392}" error="failed to connect to peer: 05052cbf-2b06-98e6-100b-1c41c60bd392"
2026-05-25T23:22:31.679Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to appendEntries to: peer="{Voter server-208acc98-b56b-2490-fbfa-98bae06d5e6c 208acc98-b56b-2490-fbfa-98bae06d5e6c}" error="failed to connect to peer: 208acc98-b56b-2490-fbfa-98bae06d5e6c"
2026-05-25T23:22:31.685Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to heartbeat to: peer=208acc98-b56b-2490-fbfa-98bae06d5e6c backoff time=10ms error="failed to connect to peer: 208acc98-b56b-2490-fbfa-98bae06d5e6c"
2026-05-25T23:22:31.694Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to heartbeat to: peer=05052cbf-2b06-98e6-100b-1c41c60bd392 backoff time=10ms error="failed to connect to peer: 05052cbf-2b06-98e6-100b-1c41c60bd392"
2026-05-25T23:22:31.698Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to appendEntries to: peer="{Voter server-05052cbf-2b06-98e6-100b-1c41c60bd392 05052cbf-2b06-98e6-100b-1c41c60bd392}" error="failed to connect to peer: 05052cbf-2b06-98e6-100b-1c41c60bd392"
2026-05-25T23:22:31.699Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to appendEntries to: peer="{Voter server-208acc98-b56b-2490-fbfa-98bae06d5e6c 208acc98-b56b-2490-fbfa-98bae06d5e6c}" error="failed to connect to peer: 208acc98-b56b-2490-fbfa-98bae06d5e6c"
2026-05-25T23:22:31.705Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to heartbeat to: peer=208acc98-b56b-2490-fbfa-98bae06d5e6c backoff time=10ms error="failed to connect to peer: 208acc98-b56b-2490-fbfa-98bae06d5e6c"
2026-05-25T23:22:31.711Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to heartbeat to: peer=05052cbf-2b06-98e6-100b-1c41c60bd392 backoff time=10ms error="failed to connect to peer: 05052cbf-2b06-98e6-100b-1c41c60bd392"
2026-05-25T23:22:31.716Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to appendEntries to: peer="{Voter server-05052cbf-2b06-98e6-100b-1c41c60bd392 05052cbf-2b06-98e6-100b-1c41c60bd392}" error="failed to connect to peer: 05052cbf-2b06-98e6-100b-1c41c60bd392"
2026-05-25T23:22:31.716Z [WARN]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to contact: server-id=server-208acc98-b56b-2490-fbfa-98bae06d5e6c time=50.9916ms
2026-05-25T23:22:31.719Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to appendEntries to: peer="{Voter server-208acc98-b56b-2490-fbfa-98bae06d5e6c 208acc98-b56b-2490-fbfa-98bae06d5e6c}" error="failed to connect to peer: 208acc98-b56b-2490-fbfa-98bae06d5e6c"
2026-05-25T23:22:31.722Z [WARN]  server-05052cbf-2b06-98e6-100b-1c41c60bd392: heartbeat timeout reached, starting election: last-leader-addr=8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d last-leader-id=server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d
2026-05-25T23:22:31.722Z [INFO]  server-05052cbf-2b06-98e6-100b-1c41c60bd392: entering candidate state: node="Node at 05052cbf-2b06-98e6-100b-1c41c60bd392 [Candidate]" term=3
2026-05-25T23:22:31.722Z [ERROR] server-05052cbf-2b06-98e6-100b-1c41c60bd392: failed to make requestVote RPC: target="{Voter server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d}" error="failed to connect to peer: 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d" term=3
2026-05-25T23:22:31.722Z [WARN]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: rejecting vote request since we have a leader: from=05052cbf-2b06-98e6-100b-1c41c60bd392 leader=8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d leader-id=server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d
2026-05-25T23:22:31.726Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to heartbeat to: peer=208acc98-b56b-2490-fbfa-98bae06d5e6c backoff time=20ms error="failed to connect to peer: 208acc98-b56b-2490-fbfa-98bae06d5e6c"
2026-05-25T23:22:31.726Z [WARN]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to contact: server-id=server-05052cbf-2b06-98e6-100b-1c41c60bd392 time=60.554776ms
2026-05-25T23:22:31.726Z [WARN]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to contact: server-id=server-208acc98-b56b-2490-fbfa-98bae06d5e6c time=61.673253ms
2026-05-25T23:22:31.726Z [WARN]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to contact quorum of nodes, stepping down
2026-05-25T23:22:31.726Z [INFO]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: entering follower state: follower="Node at 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d [Follower]" leader-address= leader-id=
2026-05-25T23:22:31.728Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to heartbeat to: peer=05052cbf-2b06-98e6-100b-1c41c60bd392 backoff time=20ms error="failed to connect to peer: 05052cbf-2b06-98e6-100b-1c41c60bd392"
2026-05-25T23:22:31.732Z [WARN]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: heartbeat timeout reached, starting election: last-leader-addr=8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d last-leader-id=server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d
2026-05-25T23:22:31.732Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: entering candidate state: node="Node at 208acc98-b56b-2490-fbfa-98bae06d5e6c [Candidate]" term=3
2026-05-25T23:22:31.732Z [ERROR] server-208acc98-b56b-2490-fbfa-98bae06d5e6c: failed to make requestVote RPC: target="{Voter server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d}" error="failed to connect to peer: 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d" term=3
2026-05-25T23:22:31.732Z [INFO]  server-05052cbf-2b06-98e6-100b-1c41c60bd392: duplicate requestVote for same term: term=3
2026-05-25T23:22:31.741Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to appendEntries to: peer="{Voter server-05052cbf-2b06-98e6-100b-1c41c60bd392 05052cbf-2b06-98e6-100b-1c41c60bd392}" error="failed to connect to peer: 05052cbf-2b06-98e6-100b-1c41c60bd392"
2026-05-25T23:22:31.747Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to appendEntries to: peer="{Voter server-208acc98-b56b-2490-fbfa-98bae06d5e6c 208acc98-b56b-2490-fbfa-98bae06d5e6c}" error="failed to connect to peer: 208acc98-b56b-2490-fbfa-98bae06d5e6c"
2026-05-25T23:22:31.801Z [WARN]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: Election timeout reached, restarting election
2026-05-25T23:22:31.801Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: entering candidate state: node="Node at 208acc98-b56b-2490-fbfa-98bae06d5e6c [Candidate]" term=4
2026-05-25T23:22:31.801Z [ERROR] server-208acc98-b56b-2490-fbfa-98bae06d5e6c: failed to make requestVote RPC: target="{Voter server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d}" error="failed to connect to peer: 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d" term=4
2026-05-25T23:22:31.801Z [INFO]  server-05052cbf-2b06-98e6-100b-1c41c60bd392: entering follower state: follower="Node at 05052cbf-2b06-98e6-100b-1c41c60bd392 [Follower]" leader-address= leader-id=
2026-05-25T23:22:31.801Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: election won: term=4 tally=2
2026-05-25T23:22:31.801Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: entering leader state: leader="Node at 208acc98-b56b-2490-fbfa-98bae06d5e6c [Leader]"
2026-05-25T23:22:31.801Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: added peer, starting replication: peer=server-05052cbf-2b06-98e6-100b-1c41c60bd392
2026-05-25T23:22:31.801Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: added peer, starting replication: peer=server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d
2026-05-25T23:22:31.801Z [ERROR] server-208acc98-b56b-2490-fbfa-98bae06d5e6c: failed to appendEntries to: peer="{Voter server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d}" error="failed to connect to peer: 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d"
2026-05-25T23:22:31.801Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: pipelining replication: peer="{Voter server-05052cbf-2b06-98e6-100b-1c41c60bd392 05052cbf-2b06-98e6-100b-1c41c60bd392}"
2026-05-25T23:22:31.809Z [ERROR] server-208acc98-b56b-2490-fbfa-98bae06d5e6c: failed to heartbeat to: peer=8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d backoff time=10ms error="failed to connect to peer: 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d"
2026-05-25T23:22:31.819Z [ERROR] server-208acc98-b56b-2490-fbfa-98bae06d5e6c: failed to appendEntries to: peer="{Voter server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d}" error="failed to connect to peer: 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d"
2026-05-25T23:22:31.824Z [WARN]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:31.824Z [INFO]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: entering candidate state: node="Node at 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d [Candidate]" term=3
2026-05-25T23:22:31.824Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to make requestVote RPC: target="{Voter server-208acc98-b56b-2490-fbfa-98bae06d5e6c 208acc98-b56b-2490-fbfa-98bae06d5e6c}" error="failed to connect to peer: 208acc98-b56b-2490-fbfa-98bae06d5e6c" term=3
2026-05-25T23:22:31.824Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to make requestVote RPC: target="{Voter server-05052cbf-2b06-98e6-100b-1c41c60bd392 05052cbf-2b06-98e6-100b-1c41c60bd392}" error="failed to connect to peer: 05052cbf-2b06-98e6-100b-1c41c60bd392" term=3
2026-05-25T23:22:31.826Z [ERROR] server-208acc98-b56b-2490-fbfa-98bae06d5e6c: failed to heartbeat to: peer=8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d backoff time=10ms error="failed to connect to peer: 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d"
2026-05-25T23:22:31.835Z [ERROR] server-208acc98-b56b-2490-fbfa-98bae06d5e6c: failed to appendEntries to: peer="{Voter server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d}" error="failed to connect to peer: 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d"
2026-05-25T23:22:31.844Z [ERROR] server-208acc98-b56b-2490-fbfa-98bae06d5e6c: failed to heartbeat to: peer=8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d backoff time=10ms error="failed to connect to peer: 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d"
2026-05-25T23:22:31.852Z [WARN]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: failed to contact: server-id=server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d time=51.050951ms
2026-05-25T23:22:31.862Z [ERROR] server-208acc98-b56b-2490-fbfa-98bae06d5e6c: failed to appendEntries to: peer="{Voter server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d}" error="failed to connect to peer: 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d"
2026-05-25T23:22:31.863Z [ERROR] server-208acc98-b56b-2490-fbfa-98bae06d5e6c: failed to heartbeat to: peer=8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d backoff time=20ms error="failed to connect to peer: 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d"
2026-05-25T23:22:31.894Z [ERROR] server-208acc98-b56b-2490-fbfa-98bae06d5e6c: failed to heartbeat to: peer=8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d backoff time=25ms error="failed to connect to peer: 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d"
2026-05-25T23:22:31.895Z [WARN]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: Election timeout reached, restarting election
2026-05-25T23:22:31.895Z [INFO]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: entering candidate state: node="Node at 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d [Candidate]" term=4
2026-05-25T23:22:31.895Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to make requestVote RPC: target="{Voter server-208acc98-b56b-2490-fbfa-98bae06d5e6c 208acc98-b56b-2490-fbfa-98bae06d5e6c}" error="failed to connect to peer: 208acc98-b56b-2490-fbfa-98bae06d5e6c" term=4
2026-05-25T23:22:31.895Z [ERROR] server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to make requestVote RPC: target="{Voter server-05052cbf-2b06-98e6-100b-1c41c60bd392 05052cbf-2b06-98e6-100b-1c41c60bd392}" error="failed to connect to peer: 05052cbf-2b06-98e6-100b-1c41c60bd392" term=4
2026-05-25T23:22:31.900Z [WARN]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: failed to contact: server-id=server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d time=98.732318ms
2026-05-25T23:22:31.912Z [ERROR] server-208acc98-b56b-2490-fbfa-98bae06d5e6c: failed to appendEntries to: peer="{Voter server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d}" error="failed to connect to peer: 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d"
2026-05-25T23:22:31.929Z [ERROR] server-208acc98-b56b-2490-fbfa-98bae06d5e6c: failed to heartbeat to: peer=8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d backoff time=25ms error="failed to connect to peer: 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d"
2026-05-25T23:22:31.930Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:31.824914728 +0000 UTC m=+7.471362177 (1 nodes), 156.315335ms from start of poll, 432.809169ms from cluster start. Timeout at 2026-05-25 23:22:31.929917518 +0000 UTC m=+7.576364972, 105.002795ms after stability
    raft_test.go:413: [INFO] Reconnecting Node at 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d [Candidate]
2026-05-25T23:22:31.948Z [WARN]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: failed to contact: server-id=server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d time=146.942457ms
2026-05-25T23:22:31.957Z [WARN]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: Election timeout reached, restarting election
2026-05-25T23:22:31.957Z [INFO]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: entering candidate state: node="Node at 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d [Candidate]" term=5
2026-05-25T23:22:31.957Z [WARN]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: rejecting vote request since we have a leader: from=8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d leader=208acc98-b56b-2490-fbfa-98bae06d5e6c leader-id=server-208acc98-b56b-2490-fbfa-98bae06d5e6c
2026-05-25T23:22:31.957Z [WARN]  server-05052cbf-2b06-98e6-100b-1c41c60bd392: rejecting vote request since we have a leader: from=8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d leader=208acc98-b56b-2490-fbfa-98bae06d5e6c leader-id=server-208acc98-b56b-2490-fbfa-98bae06d5e6c
2026-05-25T23:22:32.002Z [ERROR] server-208acc98-b56b-2490-fbfa-98bae06d5e6c: peer has newer term, stopping replication: peer="{Voter server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d}"
2026-05-25T23:22:32.002Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: entering follower state: follower="Node at 208acc98-b56b-2490-fbfa-98bae06d5e6c [Follower]" leader-address= leader-id=
2026-05-25T23:22:32.002Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: aborting pipeline replication: peer="{Voter server-05052cbf-2b06-98e6-100b-1c41c60bd392 05052cbf-2b06-98e6-100b-1c41c60bd392}"
2026-05-25T23:22:32.022Z [WARN]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: Election timeout reached, restarting election
2026-05-25T23:22:32.022Z [INFO]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: entering candidate state: node="Node at 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d [Candidate]" term=6
2026-05-25T23:22:32.022Z [WARN]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: rejecting vote request since our last term is greater: candidate=8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d last-term=4 last-candidate-term=2
2026-05-25T23:22:32.022Z [WARN]  server-05052cbf-2b06-98e6-100b-1c41c60bd392: rejecting vote request since we have a leader: from=8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d leader=208acc98-b56b-2490-fbfa-98bae06d5e6c leader-id=server-208acc98-b56b-2490-fbfa-98bae06d5e6c
2026-05-25T23:22:32.096Z [WARN]  server-05052cbf-2b06-98e6-100b-1c41c60bd392: heartbeat timeout reached, starting election: last-leader-addr=208acc98-b56b-2490-fbfa-98bae06d5e6c last-leader-id=server-208acc98-b56b-2490-fbfa-98bae06d5e6c
2026-05-25T23:22:32.096Z [INFO]  server-05052cbf-2b06-98e6-100b-1c41c60bd392: entering candidate state: node="Node at 05052cbf-2b06-98e6-100b-1c41c60bd392 [Candidate]" term=5
2026-05-25T23:22:32.096Z [INFO]  server-05052cbf-2b06-98e6-100b-1c41c60bd392: entering follower state: follower="Node at 05052cbf-2b06-98e6-100b-1c41c60bd392 [Follower]" leader-address= leader-id=
2026-05-25T23:22:32.100Z [WARN]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:32.100Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: entering candidate state: node="Node at 208acc98-b56b-2490-fbfa-98bae06d5e6c [Candidate]" term=7
2026-05-25T23:22:32.100Z [INFO]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: entering follower state: follower="Node at 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d [Follower]" leader-address= leader-id=
2026-05-25T23:22:32.100Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: election won: term=7 tally=2
2026-05-25T23:22:32.100Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: entering leader state: leader="Node at 208acc98-b56b-2490-fbfa-98bae06d5e6c [Leader]"
2026-05-25T23:22:32.101Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: added peer, starting replication: peer=server-05052cbf-2b06-98e6-100b-1c41c60bd392
2026-05-25T23:22:32.101Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: added peer, starting replication: peer=server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d
2026-05-25T23:22:32.101Z [WARN]  server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d: failed to get previous log: previous-index=5 last-index=3 error="log not found"
2026-05-25T23:22:32.101Z [WARN]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: appendEntries rejected, sending older logs: peer="{Voter server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d}" next=4
2026-05-25T23:22:32.101Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: pipelining replication: peer="{Voter server-05052cbf-2b06-98e6-100b-1c41c60bd392 05052cbf-2b06-98e6-100b-1c41c60bd392}"
2026-05-25T23:22:32.101Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: pipelining replication: peer="{Voter server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d}"
2026-05-25T23:22:32.101Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: aborting pipeline replication: peer="{Voter server-8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d 8fce7cd6-2234-ed06-90ee-c1dc7fa2b16d}"
2026-05-25T23:22:32.101Z [INFO]  server-208acc98-b56b-2490-fbfa-98bae06d5e6c: aborting pipeline replication: peer="{Voter server-05052cbf-2b06-98e6-100b-1c41c60bd392 05052cbf-2b06-98e6-100b-1c41c60bd392}"
--- PASS: TestRaft_LeaderFail (0.71s)
=== RUN   TestRaft_BehindFollower
2026-05-25T23:22:32.102Z [INFO]  server-1ac4b5ec-f323-9b73-4303-698905f543c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ac4b5ec-f323-9b73-4303-698905f543c1 Address:1ac4b5ec-f323-9b73-4303-698905f543c1} {Suffrage:Voter ID:server-0c2c83a0-4dc0-ee7f-ca18-e61fa2ec0edb Address:0c2c83a0-4dc0-ee7f-ca18-e61fa2ec0edb} {Suffrage:Voter ID:server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec Address:f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec}]"
2026-05-25T23:22:32.102Z [INFO]  server-1ac4b5ec-f323-9b73-4303-698905f543c1: entering follower state: follower="Node at 1ac4b5ec-f323-9b73-4303-698905f543c1 [Follower]" leader-address= leader-id=
2026-05-25T23:22:32.102Z [INFO]  server-0c2c83a0-4dc0-ee7f-ca18-e61fa2ec0edb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ac4b5ec-f323-9b73-4303-698905f543c1 Address:1ac4b5ec-f323-9b73-4303-698905f543c1} {Suffrage:Voter ID:server-0c2c83a0-4dc0-ee7f-ca18-e61fa2ec0edb Address:0c2c83a0-4dc0-ee7f-ca18-e61fa2ec0edb} {Suffrage:Voter ID:server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec Address:f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec}]"
2026-05-25T23:22:32.102Z [INFO]  server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ac4b5ec-f323-9b73-4303-698905f543c1 Address:1ac4b5ec-f323-9b73-4303-698905f543c1} {Suffrage:Voter ID:server-0c2c83a0-4dc0-ee7f-ca18-e61fa2ec0edb Address:0c2c83a0-4dc0-ee7f-ca18-e61fa2ec0edb} {Suffrage:Voter ID:server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec Address:f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec}]"
2026-05-25T23:22:32.102Z [INFO]  server-0c2c83a0-4dc0-ee7f-ca18-e61fa2ec0edb: entering follower state: follower="Node at 0c2c83a0-4dc0-ee7f-ca18-e61fa2ec0edb [Follower]" leader-address= leader-id=
2026-05-25T23:22:32.102Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:32.102Z [INFO]  server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec: entering follower state: follower="Node at f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec [Follower]" leader-address= leader-id=
2026-05-25T23:22:32.165Z [WARN]  server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:32.165Z [INFO]  server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec: entering candidate state: node="Node at f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec [Candidate]" term=2
2026-05-25T23:22:32.165Z [INFO]  server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec: election won: term=2 tally=2
2026-05-25T23:22:32.165Z [INFO]  server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec: entering leader state: leader="Node at f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec [Leader]"
2026-05-25T23:22:32.165Z [INFO]  server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec: added peer, starting replication: peer=server-1ac4b5ec-f323-9b73-4303-698905f543c1
2026-05-25T23:22:32.165Z [INFO]  server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec: added peer, starting replication: peer=server-0c2c83a0-4dc0-ee7f-ca18-e61fa2ec0edb
2026-05-25T23:22:32.166Z [INFO]  server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec: pipelining replication: peer="{Voter server-0c2c83a0-4dc0-ee7f-ca18-e61fa2ec0edb 0c2c83a0-4dc0-ee7f-ca18-e61fa2ec0edb}"
2026-05-25T23:22:32.166Z [INFO]  server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec: pipelining replication: peer="{Voter server-1ac4b5ec-f323-9b73-4303-698905f543c1 1ac4b5ec-f323-9b73-4303-698905f543c1}"
2026-05-25T23:22:32.271Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:32.165975781 +0000 UTC m=+7.812423254 (1 nodes), 63.407345ms from start of poll, 63.70059ms from cluster start. Timeout at 2026-05-25 23:22:32.270979082 +0000 UTC m=+7.917426533, 105.003279ms after stability
2026-05-25T23:22:32.271Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:22:32.377Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:22:32.271741702 +0000 UTC m=+7.918189155 (2 nodes), 564ns from start of poll, 169.466491ms from cluster start. Timeout at 2026-05-25 23:22:32.376745973 +0000 UTC m=+8.023193460, 105.004305ms after stability
2026-05-25T23:22:32.377Z [ERROR] server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec: failed to pipeline appendEntries: peer="{Voter server-1ac4b5ec-f323-9b73-4303-698905f543c1 1ac4b5ec-f323-9b73-4303-698905f543c1}" error="append pipeline closed"
2026-05-25T23:22:32.377Z [INFO]  server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec: aborting pipeline replication: peer="{Voter server-1ac4b5ec-f323-9b73-4303-698905f543c1 1ac4b5ec-f323-9b73-4303-698905f543c1}"
2026-05-25T23:22:32.377Z [ERROR] server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec: failed to appendEntries to: peer="{Voter server-1ac4b5ec-f323-9b73-4303-698905f543c1 1ac4b5ec-f323-9b73-4303-698905f543c1}" error="failed to connect to peer: 1ac4b5ec-f323-9b73-4303-698905f543c1"
2026-05-25T23:22:32.377Z [ERROR] server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec: failed to heartbeat to: peer=1ac4b5ec-f323-9b73-4303-698905f543c1 backoff time=10ms error="failed to connect to peer: 1ac4b5ec-f323-9b73-4303-698905f543c1"
    raft_test.go:459: [INFO] Finished apply without behind follower
2026-05-25T23:22:32.388Z [INFO]  server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec: pipelining replication: peer="{Voter server-1ac4b5ec-f323-9b73-4303-698905f543c1 1ac4b5ec-f323-9b73-4303-698905f543c1}"
2026-05-25T23:22:32.400Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:32.505Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:32.400284925 +0000 UTC m=+8.046732378 (1 nodes), 747ns from start of poll, 298.009714ms from cluster start. Timeout at 2026-05-25 23:22:32.505289433 +0000 UTC m=+8.151736884, 105.004506ms after stability
2026-05-25T23:22:32.505Z [INFO]  server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec: aborting pipeline replication: peer="{Voter server-1ac4b5ec-f323-9b73-4303-698905f543c1 1ac4b5ec-f323-9b73-4303-698905f543c1}"
2026-05-25T23:22:32.505Z [INFO]  server-f0ca0f2a-bbc6-cc1f-058f-170a331ce3ec: aborting pipeline replication: peer="{Voter server-0c2c83a0-4dc0-ee7f-ca18-e61fa2ec0edb 0c2c83a0-4dc0-ee7f-ca18-e61fa2ec0edb}"
--- PASS: TestRaft_BehindFollower (0.40s)
=== RUN   TestRaft_ApplyNonLeader
2026-05-25T23:22:32.506Z [INFO]  server-57e3f6a4-38d0-0fec-6976-8062457b76ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57e3f6a4-38d0-0fec-6976-8062457b76ac Address:57e3f6a4-38d0-0fec-6976-8062457b76ac} {Suffrage:Voter ID:server-10968d74-b3fe-f465-2379-c3325db217bf Address:10968d74-b3fe-f465-2379-c3325db217bf} {Suffrage:Voter ID:server-7807cdb7-125d-0bdc-ad13-7118af9310dd Address:7807cdb7-125d-0bdc-ad13-7118af9310dd}]"
2026-05-25T23:22:32.506Z [INFO]  server-57e3f6a4-38d0-0fec-6976-8062457b76ac: entering follower state: follower="Node at 57e3f6a4-38d0-0fec-6976-8062457b76ac [Follower]" leader-address= leader-id=
2026-05-25T23:22:32.506Z [INFO]  server-10968d74-b3fe-f465-2379-c3325db217bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57e3f6a4-38d0-0fec-6976-8062457b76ac Address:57e3f6a4-38d0-0fec-6976-8062457b76ac} {Suffrage:Voter ID:server-10968d74-b3fe-f465-2379-c3325db217bf Address:10968d74-b3fe-f465-2379-c3325db217bf} {Suffrage:Voter ID:server-7807cdb7-125d-0bdc-ad13-7118af9310dd Address:7807cdb7-125d-0bdc-ad13-7118af9310dd}]"
2026-05-25T23:22:32.506Z [INFO]  server-10968d74-b3fe-f465-2379-c3325db217bf: entering follower state: follower="Node at 10968d74-b3fe-f465-2379-c3325db217bf [Follower]" leader-address= leader-id=
2026-05-25T23:22:32.506Z [INFO]  server-7807cdb7-125d-0bdc-ad13-7118af9310dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57e3f6a4-38d0-0fec-6976-8062457b76ac Address:57e3f6a4-38d0-0fec-6976-8062457b76ac} {Suffrage:Voter ID:server-10968d74-b3fe-f465-2379-c3325db217bf Address:10968d74-b3fe-f465-2379-c3325db217bf} {Suffrage:Voter ID:server-7807cdb7-125d-0bdc-ad13-7118af9310dd Address:7807cdb7-125d-0bdc-ad13-7118af9310dd}]"
2026-05-25T23:22:32.506Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:32.506Z [INFO]  server-7807cdb7-125d-0bdc-ad13-7118af9310dd: entering follower state: follower="Node at 7807cdb7-125d-0bdc-ad13-7118af9310dd [Follower]" leader-address= leader-id=
2026-05-25T23:22:32.563Z [WARN]  server-7807cdb7-125d-0bdc-ad13-7118af9310dd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:32.563Z [INFO]  server-7807cdb7-125d-0bdc-ad13-7118af9310dd: entering candidate state: node="Node at 7807cdb7-125d-0bdc-ad13-7118af9310dd [Candidate]" term=2
2026-05-25T23:22:32.563Z [INFO]  server-7807cdb7-125d-0bdc-ad13-7118af9310dd: election won: term=2 tally=2
2026-05-25T23:22:32.563Z [INFO]  server-7807cdb7-125d-0bdc-ad13-7118af9310dd: entering leader state: leader="Node at 7807cdb7-125d-0bdc-ad13-7118af9310dd [Leader]"
2026-05-25T23:22:32.563Z [INFO]  server-7807cdb7-125d-0bdc-ad13-7118af9310dd: added peer, starting replication: peer=server-57e3f6a4-38d0-0fec-6976-8062457b76ac
2026-05-25T23:22:32.563Z [INFO]  server-7807cdb7-125d-0bdc-ad13-7118af9310dd: added peer, starting replication: peer=server-10968d74-b3fe-f465-2379-c3325db217bf
2026-05-25T23:22:32.563Z [INFO]  server-7807cdb7-125d-0bdc-ad13-7118af9310dd: pipelining replication: peer="{Voter server-10968d74-b3fe-f465-2379-c3325db217bf 10968d74-b3fe-f465-2379-c3325db217bf}"
2026-05-25T23:22:32.563Z [INFO]  server-7807cdb7-125d-0bdc-ad13-7118af9310dd: pipelining replication: peer="{Voter server-57e3f6a4-38d0-0fec-6976-8062457b76ac 57e3f6a4-38d0-0fec-6976-8062457b76ac}"
2026-05-25T23:22:32.669Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:32.563875959 +0000 UTC m=+8.210323417 (1 nodes), 57.007603ms from start of poll, 57.321866ms from cluster start. Timeout at 2026-05-25 23:22:32.668878354 +0000 UTC m=+8.315325808, 105.002391ms after stability
2026-05-25T23:22:32.669Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:22:32.775Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:22:32.669634911 +0000 UTC m=+8.316082360 (2 nodes), 1.293µs from start of poll, 163.080809ms from cluster start. Timeout at 2026-05-25 23:22:32.774638557 +0000 UTC m=+8.421086007, 105.003647ms after stability
2026-05-25T23:22:32.775Z [INFO]  server-7807cdb7-125d-0bdc-ad13-7118af9310dd: aborting pipeline replication: peer="{Voter server-10968d74-b3fe-f465-2379-c3325db217bf 10968d74-b3fe-f465-2379-c3325db217bf}"
2026-05-25T23:22:32.775Z [INFO]  server-7807cdb7-125d-0bdc-ad13-7118af9310dd: aborting pipeline replication: peer="{Voter server-57e3f6a4-38d0-0fec-6976-8062457b76ac 57e3f6a4-38d0-0fec-6976-8062457b76ac}"
--- PASS: TestRaft_ApplyNonLeader (0.27s)
=== RUN   TestRaft_ApplyConcurrent
2026-05-25T23:22:32.776Z [INFO]  server-e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5 Address:e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5} {Suffrage:Voter ID:server-d946d9fd-ef98-a6c4-919f-dbfdff3250f2 Address:d946d9fd-ef98-a6c4-919f-dbfdff3250f2} {Suffrage:Voter ID:server-89f8d79f-a672-cea6-d7c5-5ad916b7fa6b Address:89f8d79f-a672-cea6-d7c5-5ad916b7fa6b}]"
2026-05-25T23:22:32.776Z [INFO]  server-e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5: entering follower state: follower="Node at e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5 [Follower]" leader-address= leader-id=
2026-05-25T23:22:32.776Z [INFO]  server-d946d9fd-ef98-a6c4-919f-dbfdff3250f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5 Address:e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5} {Suffrage:Voter ID:server-d946d9fd-ef98-a6c4-919f-dbfdff3250f2 Address:d946d9fd-ef98-a6c4-919f-dbfdff3250f2} {Suffrage:Voter ID:server-89f8d79f-a672-cea6-d7c5-5ad916b7fa6b Address:89f8d79f-a672-cea6-d7c5-5ad916b7fa6b}]"
2026-05-25T23:22:32.776Z [INFO]  server-d946d9fd-ef98-a6c4-919f-dbfdff3250f2: entering follower state: follower="Node at d946d9fd-ef98-a6c4-919f-dbfdff3250f2 [Follower]" leader-address= leader-id=
2026-05-25T23:22:32.776Z [INFO]  server-89f8d79f-a672-cea6-d7c5-5ad916b7fa6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5 Address:e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5} {Suffrage:Voter ID:server-d946d9fd-ef98-a6c4-919f-dbfdff3250f2 Address:d946d9fd-ef98-a6c4-919f-dbfdff3250f2} {Suffrage:Voter ID:server-89f8d79f-a672-cea6-d7c5-5ad916b7fa6b Address:89f8d79f-a672-cea6-d7c5-5ad916b7fa6b}]"
2026-05-25T23:22:32.776Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:32.776Z [INFO]  server-89f8d79f-a672-cea6-d7c5-5ad916b7fa6b: entering follower state: follower="Node at 89f8d79f-a672-cea6-d7c5-5ad916b7fa6b [Follower]" leader-address= leader-id=
2026-05-25T23:22:32.883Z [WARN]  server-e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:32.883Z [INFO]  server-e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5: entering candidate state: node="Node at e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5 [Candidate]" term=2
2026-05-25T23:22:32.884Z [INFO]  server-e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5: election won: term=2 tally=2
2026-05-25T23:22:32.884Z [INFO]  server-e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5: entering leader state: leader="Node at e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5 [Leader]"
2026-05-25T23:22:32.884Z [INFO]  server-e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5: added peer, starting replication: peer=server-d946d9fd-ef98-a6c4-919f-dbfdff3250f2
2026-05-25T23:22:32.884Z [INFO]  server-e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5: added peer, starting replication: peer=server-89f8d79f-a672-cea6-d7c5-5ad916b7fa6b
2026-05-25T23:22:32.884Z [INFO]  server-e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5: pipelining replication: peer="{Voter server-89f8d79f-a672-cea6-d7c5-5ad916b7fa6b 89f8d79f-a672-cea6-d7c5-5ad916b7fa6b}"
2026-05-25T23:22:32.884Z [INFO]  server-e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5: pipelining replication: peer="{Voter server-d946d9fd-ef98-a6c4-919f-dbfdff3250f2 d946d9fd-ef98-a6c4-919f-dbfdff3250f2}"
2026-05-25T23:22:33.089Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:32.884243377 +0000 UTC m=+8.530690828 (1 nodes), 107.403913ms from start of poll, 107.701152ms from cluster start. Timeout at 2026-05-25 23:22:33.089308949 +0000 UTC m=+8.735756403, 205.065575ms after stability
2026-05-25T23:22:33.103Z [INFO]  server-e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5: aborting pipeline replication: peer="{Voter server-89f8d79f-a672-cea6-d7c5-5ad916b7fa6b 89f8d79f-a672-cea6-d7c5-5ad916b7fa6b}"
2026-05-25T23:22:33.103Z [INFO]  server-e4cf9a2a-d1d1-bbf1-4fbd-deb3b3ad8da5: aborting pipeline replication: peer="{Voter server-d946d9fd-ef98-a6c4-919f-dbfdff3250f2 d946d9fd-ef98-a6c4-919f-dbfdff3250f2}"
--- PASS: TestRaft_ApplyConcurrent (0.33s)
=== RUN   TestRaft_ApplyConcurrent_Timeout
2026-05-25T23:22:33.104Z [INFO]  server-c981a9e2-3a6b-1d55-161f-2febde7960c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c981a9e2-3a6b-1d55-161f-2febde7960c3 Address:c981a9e2-3a6b-1d55-161f-2febde7960c3}]"
2026-05-25T23:22:33.104Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:33.104Z [INFO]  server-c981a9e2-3a6b-1d55-161f-2febde7960c3: entering follower state: follower="Node at c981a9e2-3a6b-1d55-161f-2febde7960c3 [Follower]" leader-address= leader-id=
2026-05-25T23:22:33.224Z [WARN]  server-c981a9e2-3a6b-1d55-161f-2febde7960c3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:33.224Z [INFO]  server-c981a9e2-3a6b-1d55-161f-2febde7960c3: entering candidate state: node="Node at c981a9e2-3a6b-1d55-161f-2febde7960c3 [Candidate]" term=2
2026-05-25T23:22:33.224Z [INFO]  server-c981a9e2-3a6b-1d55-161f-2febde7960c3: election won: term=2 tally=1
2026-05-25T23:22:33.224Z [INFO]  server-c981a9e2-3a6b-1d55-161f-2febde7960c3: entering leader state: leader="Node at c981a9e2-3a6b-1d55-161f-2febde7960c3 [Leader]"
2026-05-25T23:22:33.426Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:33.224462393 +0000 UTC m=+8.870909913 (1 nodes), 120.164461ms from start of poll, 120.293973ms from cluster start. Timeout at 2026-05-25 23:22:33.425464766 +0000 UTC m=+9.071912221, 201.002308ms after stability
--- PASS: TestRaft_ApplyConcurrent_Timeout (0.32s)
=== RUN   TestRaft_JoinNode
2026-05-25T23:22:33.427Z [INFO]  server-a51529ee-c7c1-43a4-4fd3-0d637efc78f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a51529ee-c7c1-43a4-4fd3-0d637efc78f8 Address:a51529ee-c7c1-43a4-4fd3-0d637efc78f8} {Suffrage:Voter ID:server-b9de1023-3397-b7ab-a4db-2f6b8bd04e86 Address:b9de1023-3397-b7ab-a4db-2f6b8bd04e86}]"
2026-05-25T23:22:33.427Z [INFO]  server-b9de1023-3397-b7ab-a4db-2f6b8bd04e86: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a51529ee-c7c1-43a4-4fd3-0d637efc78f8 Address:a51529ee-c7c1-43a4-4fd3-0d637efc78f8} {Suffrage:Voter ID:server-b9de1023-3397-b7ab-a4db-2f6b8bd04e86 Address:b9de1023-3397-b7ab-a4db-2f6b8bd04e86}]"
2026-05-25T23:22:33.427Z [INFO]  server-a51529ee-c7c1-43a4-4fd3-0d637efc78f8: entering follower state: follower="Node at a51529ee-c7c1-43a4-4fd3-0d637efc78f8 [Follower]" leader-address= leader-id=
2026-05-25T23:22:33.427Z [INFO]  server-b9de1023-3397-b7ab-a4db-2f6b8bd04e86: entering follower state: follower="Node at b9de1023-3397-b7ab-a4db-2f6b8bd04e86 [Follower]" leader-address= leader-id=
2026-05-25T23:22:33.427Z [INFO]  server-4e13ec63-7a53-f29e-879d-1fbd5679d0bf: initial configuration: index=0 servers=[]
2026-05-25T23:22:33.427Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:33.427Z [INFO]  server-4e13ec63-7a53-f29e-879d-1fbd5679d0bf: entering follower state: follower="Node at 4e13ec63-7a53-f29e-879d-1fbd5679d0bf [Follower]" leader-address= leader-id=
2026-05-25T23:22:33.494Z [WARN]  server-b9de1023-3397-b7ab-a4db-2f6b8bd04e86: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:33.494Z [INFO]  server-b9de1023-3397-b7ab-a4db-2f6b8bd04e86: entering candidate state: node="Node at b9de1023-3397-b7ab-a4db-2f6b8bd04e86 [Candidate]" term=2
2026-05-25T23:22:33.494Z [INFO]  server-b9de1023-3397-b7ab-a4db-2f6b8bd04e86: election won: term=2 tally=2
2026-05-25T23:22:33.494Z [INFO]  server-b9de1023-3397-b7ab-a4db-2f6b8bd04e86: entering leader state: leader="Node at b9de1023-3397-b7ab-a4db-2f6b8bd04e86 [Leader]"
2026-05-25T23:22:33.494Z [INFO]  server-b9de1023-3397-b7ab-a4db-2f6b8bd04e86: added peer, starting replication: peer=server-a51529ee-c7c1-43a4-4fd3-0d637efc78f8
2026-05-25T23:22:33.494Z [INFO]  server-b9de1023-3397-b7ab-a4db-2f6b8bd04e86: pipelining replication: peer="{Voter server-a51529ee-c7c1-43a4-4fd3-0d637efc78f8 a51529ee-c7c1-43a4-4fd3-0d637efc78f8}"
2026-05-25T23:22:33.522Z [WARN]  server-4e13ec63-7a53-f29e-879d-1fbd5679d0bf: no known peers, aborting election
2026-05-25T23:22:33.600Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:33.494483422 +0000 UTC m=+9.140930875 (1 nodes), 66.953769ms from start of poll, 67.435214ms from cluster start. Timeout at 2026-05-25 23:22:33.599490456 +0000 UTC m=+9.245937947, 105.007072ms after stability
2026-05-25T23:22:33.600Z [INFO]  server-b9de1023-3397-b7ab-a4db-2f6b8bd04e86: updating configuration: command=AddVoter server-id=server-4e13ec63-7a53-f29e-879d-1fbd5679d0bf server-addr=4e13ec63-7a53-f29e-879d-1fbd5679d0bf servers="[{Suffrage:Voter ID:server-a51529ee-c7c1-43a4-4fd3-0d637efc78f8 Address:a51529ee-c7c1-43a4-4fd3-0d637efc78f8} {Suffrage:Voter ID:server-b9de1023-3397-b7ab-a4db-2f6b8bd04e86 Address:b9de1023-3397-b7ab-a4db-2f6b8bd04e86} {Suffrage:Voter ID:server-4e13ec63-7a53-f29e-879d-1fbd5679d0bf Address:4e13ec63-7a53-f29e-879d-1fbd5679d0bf}]"
2026-05-25T23:22:33.600Z [INFO]  server-b9de1023-3397-b7ab-a4db-2f6b8bd04e86: added peer, starting replication: peer=server-4e13ec63-7a53-f29e-879d-1fbd5679d0bf
2026-05-25T23:22:33.600Z [WARN]  server-4e13ec63-7a53-f29e-879d-1fbd5679d0bf: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2026-05-25T23:22:33.600Z [WARN]  server-b9de1023-3397-b7ab-a4db-2f6b8bd04e86: appendEntries rejected, sending older logs: peer="{Voter server-4e13ec63-7a53-f29e-879d-1fbd5679d0bf 4e13ec63-7a53-f29e-879d-1fbd5679d0bf}" next=1
2026-05-25T23:22:33.600Z [INFO]  server-b9de1023-3397-b7ab-a4db-2f6b8bd04e86: pipelining replication: peer="{Voter server-4e13ec63-7a53-f29e-879d-1fbd5679d0bf 4e13ec63-7a53-f29e-879d-1fbd5679d0bf}"
2026-05-25T23:22:33.600Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:33.706Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:33.600673509 +0000 UTC m=+9.247120970 (1 nodes), 568ns from start of poll, 173.625309ms from cluster start. Timeout at 2026-05-25 23:22:33.705676941 +0000 UTC m=+9.352124401, 105.003431ms after stability
2026-05-25T23:22:33.706Z [INFO]  server-b9de1023-3397-b7ab-a4db-2f6b8bd04e86: aborting pipeline replication: peer="{Voter server-4e13ec63-7a53-f29e-879d-1fbd5679d0bf 4e13ec63-7a53-f29e-879d-1fbd5679d0bf}"
2026-05-25T23:22:33.706Z [INFO]  server-b9de1023-3397-b7ab-a4db-2f6b8bd04e86: aborting pipeline replication: peer="{Voter server-a51529ee-c7c1-43a4-4fd3-0d637efc78f8 a51529ee-c7c1-43a4-4fd3-0d637efc78f8}"
--- PASS: TestRaft_JoinNode (0.28s)
=== RUN   TestRaft_JoinNode_ConfigStore
2026-05-25T23:22:33.707Z [INFO]  server-c13ab050-6443-8319-e736-71aeef42f1a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c13ab050-6443-8319-e736-71aeef42f1a4 Address:c13ab050-6443-8319-e736-71aeef42f1a4}]"
2026-05-25T23:22:33.707Z [INFO]  server-c13ab050-6443-8319-e736-71aeef42f1a4: entering follower state: follower="Node at c13ab050-6443-8319-e736-71aeef42f1a4 [Follower]" leader-address= leader-id=
2026-05-25T23:22:33.707Z [INFO]  server-7e9fca43-bb5a-e874-17c4-8aafd93a0fce: initial configuration: index=0 servers=[]
2026-05-25T23:22:33.707Z [INFO]  server-7e9fca43-bb5a-e874-17c4-8aafd93a0fce: entering follower state: follower="Node at 7e9fca43-bb5a-e874-17c4-8aafd93a0fce [Follower]" leader-address= leader-id=
2026-05-25T23:22:33.708Z [INFO]  server-58fb0112-a322-d17e-1338-fd6b31df25f6: initial configuration: index=0 servers=[]
2026-05-25T23:22:33.708Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:33.708Z [INFO]  server-58fb0112-a322-d17e-1338-fd6b31df25f6: entering follower state: follower="Node at 58fb0112-a322-d17e-1338-fd6b31df25f6 [Follower]" leader-address= leader-id=
2026-05-25T23:22:33.772Z [WARN]  server-58fb0112-a322-d17e-1338-fd6b31df25f6: no known peers, aborting election
2026-05-25T23:22:33.783Z [WARN]  server-c13ab050-6443-8319-e736-71aeef42f1a4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:33.783Z [INFO]  server-c13ab050-6443-8319-e736-71aeef42f1a4: entering candidate state: node="Node at c13ab050-6443-8319-e736-71aeef42f1a4 [Candidate]" term=2
2026-05-25T23:22:33.783Z [INFO]  server-c13ab050-6443-8319-e736-71aeef42f1a4: election won: term=2 tally=1
2026-05-25T23:22:33.783Z [INFO]  server-c13ab050-6443-8319-e736-71aeef42f1a4: entering leader state: leader="Node at c13ab050-6443-8319-e736-71aeef42f1a4 [Leader]"
2026-05-25T23:22:33.796Z [WARN]  server-7e9fca43-bb5a-e874-17c4-8aafd93a0fce: no known peers, aborting election
2026-05-25T23:22:33.888Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:33.783080838 +0000 UTC m=+9.429528293 (1 nodes), 74.943396ms from start of poll, 75.581536ms from cluster start. Timeout at 2026-05-25 23:22:33.88808398 +0000 UTC m=+9.534531434, 105.003141ms after stability
2026-05-25T23:22:33.888Z [INFO]  server-c13ab050-6443-8319-e736-71aeef42f1a4: updating configuration: command=AddVoter server-id=server-7e9fca43-bb5a-e874-17c4-8aafd93a0fce server-addr=7e9fca43-bb5a-e874-17c4-8aafd93a0fce servers="[{Suffrage:Voter ID:server-c13ab050-6443-8319-e736-71aeef42f1a4 Address:c13ab050-6443-8319-e736-71aeef42f1a4} {Suffrage:Voter ID:server-7e9fca43-bb5a-e874-17c4-8aafd93a0fce Address:7e9fca43-bb5a-e874-17c4-8aafd93a0fce}]"
2026-05-25T23:22:33.888Z [INFO]  server-c13ab050-6443-8319-e736-71aeef42f1a4: added peer, starting replication: peer=server-7e9fca43-bb5a-e874-17c4-8aafd93a0fce
2026-05-25T23:22:33.889Z [WARN]  server-7e9fca43-bb5a-e874-17c4-8aafd93a0fce: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2026-05-25T23:22:33.889Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:33.889Z [WARN]  server-c13ab050-6443-8319-e736-71aeef42f1a4: appendEntries rejected, sending older logs: peer="{Voter server-7e9fca43-bb5a-e874-17c4-8aafd93a0fce 7e9fca43-bb5a-e874-17c4-8aafd93a0fce}" next=1
2026-05-25T23:22:33.889Z [INFO]  server-c13ab050-6443-8319-e736-71aeef42f1a4: pipelining replication: peer="{Voter server-7e9fca43-bb5a-e874-17c4-8aafd93a0fce 7e9fca43-bb5a-e874-17c4-8aafd93a0fce}"
2026-05-25T23:22:33.995Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:33.889557287 +0000 UTC m=+9.536004822 (1 nodes), 3.543µs from start of poll, 182.058065ms from cluster start. Timeout at 2026-05-25 23:22:33.99456055 +0000 UTC m=+9.641008005, 105.003183ms after stability
2026-05-25T23:22:33.995Z [INFO]  server-c13ab050-6443-8319-e736-71aeef42f1a4: updating configuration: command=AddVoter server-id=server-58fb0112-a322-d17e-1338-fd6b31df25f6 server-addr=58fb0112-a322-d17e-1338-fd6b31df25f6 servers="[{Suffrage:Voter ID:server-c13ab050-6443-8319-e736-71aeef42f1a4 Address:c13ab050-6443-8319-e736-71aeef42f1a4} {Suffrage:Voter ID:server-7e9fca43-bb5a-e874-17c4-8aafd93a0fce Address:7e9fca43-bb5a-e874-17c4-8aafd93a0fce} {Suffrage:Voter ID:server-58fb0112-a322-d17e-1338-fd6b31df25f6 Address:58fb0112-a322-d17e-1338-fd6b31df25f6}]"
2026-05-25T23:22:33.995Z [INFO]  server-c13ab050-6443-8319-e736-71aeef42f1a4: added peer, starting replication: peer=server-58fb0112-a322-d17e-1338-fd6b31df25f6
2026-05-25T23:22:33.995Z [WARN]  server-58fb0112-a322-d17e-1338-fd6b31df25f6: failed to get previous log: previous-index=4 last-index=0 error="log not found"
2026-05-25T23:22:33.995Z [WARN]  server-c13ab050-6443-8319-e736-71aeef42f1a4: appendEntries rejected, sending older logs: peer="{Voter server-58fb0112-a322-d17e-1338-fd6b31df25f6 58fb0112-a322-d17e-1338-fd6b31df25f6}" next=1
2026-05-25T23:22:33.995Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:33.995Z [INFO]  server-c13ab050-6443-8319-e736-71aeef42f1a4: pipelining replication: peer="{Voter server-58fb0112-a322-d17e-1338-fd6b31df25f6 58fb0112-a322-d17e-1338-fd6b31df25f6}"
2026-05-25T23:22:34.101Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:33.995621026 +0000 UTC m=+9.642068497 (1 nodes), 1.028µs from start of poll, 288.12174ms from cluster start. Timeout at 2026-05-25 23:22:34.100624518 +0000 UTC m=+9.747071971, 105.003474ms after stability
2026-05-25T23:22:34.101Z [INFO]  server-c13ab050-6443-8319-e736-71aeef42f1a4: aborting pipeline replication: peer="{Voter server-58fb0112-a322-d17e-1338-fd6b31df25f6 58fb0112-a322-d17e-1338-fd6b31df25f6}"
2026-05-25T23:22:34.101Z [INFO]  server-c13ab050-6443-8319-e736-71aeef42f1a4: aborting pipeline replication: peer="{Voter server-7e9fca43-bb5a-e874-17c4-8aafd93a0fce 7e9fca43-bb5a-e874-17c4-8aafd93a0fce}"
--- PASS: TestRaft_JoinNode_ConfigStore (0.39s)
=== RUN   TestRaft_RemoveFollower
2026-05-25T23:22:34.102Z [INFO]  server-fa065b57-554c-4c6f-f70e-43c1c3e91597: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa065b57-554c-4c6f-f70e-43c1c3e91597 Address:fa065b57-554c-4c6f-f70e-43c1c3e91597} {Suffrage:Voter ID:server-20d5cab8-c10a-1a4a-adc1-d3922525d28b Address:20d5cab8-c10a-1a4a-adc1-d3922525d28b} {Suffrage:Voter ID:server-98c60309-0b6b-8ee2-7609-4384e3492039 Address:98c60309-0b6b-8ee2-7609-4384e3492039}]"
2026-05-25T23:22:34.102Z [INFO]  server-fa065b57-554c-4c6f-f70e-43c1c3e91597: entering follower state: follower="Node at fa065b57-554c-4c6f-f70e-43c1c3e91597 [Follower]" leader-address= leader-id=
2026-05-25T23:22:34.102Z [INFO]  server-20d5cab8-c10a-1a4a-adc1-d3922525d28b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa065b57-554c-4c6f-f70e-43c1c3e91597 Address:fa065b57-554c-4c6f-f70e-43c1c3e91597} {Suffrage:Voter ID:server-20d5cab8-c10a-1a4a-adc1-d3922525d28b Address:20d5cab8-c10a-1a4a-adc1-d3922525d28b} {Suffrage:Voter ID:server-98c60309-0b6b-8ee2-7609-4384e3492039 Address:98c60309-0b6b-8ee2-7609-4384e3492039}]"
2026-05-25T23:22:34.102Z [INFO]  server-20d5cab8-c10a-1a4a-adc1-d3922525d28b: entering follower state: follower="Node at 20d5cab8-c10a-1a4a-adc1-d3922525d28b [Follower]" leader-address= leader-id=
2026-05-25T23:22:34.102Z [INFO]  server-98c60309-0b6b-8ee2-7609-4384e3492039: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa065b57-554c-4c6f-f70e-43c1c3e91597 Address:fa065b57-554c-4c6f-f70e-43c1c3e91597} {Suffrage:Voter ID:server-20d5cab8-c10a-1a4a-adc1-d3922525d28b Address:20d5cab8-c10a-1a4a-adc1-d3922525d28b} {Suffrage:Voter ID:server-98c60309-0b6b-8ee2-7609-4384e3492039 Address:98c60309-0b6b-8ee2-7609-4384e3492039}]"
2026-05-25T23:22:34.103Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:34.103Z [INFO]  server-98c60309-0b6b-8ee2-7609-4384e3492039: entering follower state: follower="Node at 98c60309-0b6b-8ee2-7609-4384e3492039 [Follower]" leader-address= leader-id=
2026-05-25T23:22:34.177Z [WARN]  server-20d5cab8-c10a-1a4a-adc1-d3922525d28b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:34.177Z [INFO]  server-20d5cab8-c10a-1a4a-adc1-d3922525d28b: entering candidate state: node="Node at 20d5cab8-c10a-1a4a-adc1-d3922525d28b [Candidate]" term=2
2026-05-25T23:22:34.178Z [INFO]  server-20d5cab8-c10a-1a4a-adc1-d3922525d28b: election won: term=2 tally=2
2026-05-25T23:22:34.178Z [INFO]  server-20d5cab8-c10a-1a4a-adc1-d3922525d28b: entering leader state: leader="Node at 20d5cab8-c10a-1a4a-adc1-d3922525d28b [Leader]"
2026-05-25T23:22:34.178Z [INFO]  server-20d5cab8-c10a-1a4a-adc1-d3922525d28b: added peer, starting replication: peer=server-fa065b57-554c-4c6f-f70e-43c1c3e91597
2026-05-25T23:22:34.178Z [INFO]  server-20d5cab8-c10a-1a4a-adc1-d3922525d28b: added peer, starting replication: peer=server-98c60309-0b6b-8ee2-7609-4384e3492039
2026-05-25T23:22:34.178Z [INFO]  server-20d5cab8-c10a-1a4a-adc1-d3922525d28b: pipelining replication: peer="{Voter server-fa065b57-554c-4c6f-f70e-43c1c3e91597 fa065b57-554c-4c6f-f70e-43c1c3e91597}"
2026-05-25T23:22:34.178Z [INFO]  server-20d5cab8-c10a-1a4a-adc1-d3922525d28b: pipelining replication: peer="{Voter server-98c60309-0b6b-8ee2-7609-4384e3492039 98c60309-0b6b-8ee2-7609-4384e3492039}"
2026-05-25T23:22:34.283Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:34.178088543 +0000 UTC m=+9.824535994 (1 nodes), 75.027614ms from start of poll, 75.52659ms from cluster start. Timeout at 2026-05-25 23:22:34.283091363 +0000 UTC m=+9.929538813, 105.002819ms after stability
2026-05-25T23:22:34.289Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:22:34.395Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:22:34.289087848 +0000 UTC m=+9.935535306 (2 nodes), 1.755µs from start of poll, 186.525902ms from cluster start. Timeout at 2026-05-25 23:22:34.394093064 +0000 UTC m=+10.040540514, 105.005208ms after stability
2026-05-25T23:22:34.395Z [INFO]  server-20d5cab8-c10a-1a4a-adc1-d3922525d28b: updating configuration: command=RemoveServer server-id=server-fa065b57-554c-4c6f-f70e-43c1c3e91597 server-addr= servers="[{Suffrage:Voter ID:server-20d5cab8-c10a-1a4a-adc1-d3922525d28b Address:20d5cab8-c10a-1a4a-adc1-d3922525d28b} {Suffrage:Voter ID:server-98c60309-0b6b-8ee2-7609-4384e3492039 Address:98c60309-0b6b-8ee2-7609-4384e3492039}]"
2026-05-25T23:22:34.395Z [INFO]  server-20d5cab8-c10a-1a4a-adc1-d3922525d28b: removed peer, stopping replication: peer=server-fa065b57-554c-4c6f-f70e-43c1c3e91597 last-index=3
2026-05-25T23:22:34.395Z [INFO]  server-20d5cab8-c10a-1a4a-adc1-d3922525d28b: aborting pipeline replication: peer="{Voter server-fa065b57-554c-4c6f-f70e-43c1c3e91597 fa065b57-554c-4c6f-f70e-43c1c3e91597}"
2026-05-25T23:22:34.489Z [WARN]  server-fa065b57-554c-4c6f-f70e-43c1c3e91597: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2026-05-25T23:22:34.501Z [INFO]  server-20d5cab8-c10a-1a4a-adc1-d3922525d28b: aborting pipeline replication: peer="{Voter server-98c60309-0b6b-8ee2-7609-4384e3492039 98c60309-0b6b-8ee2-7609-4384e3492039}"
--- PASS: TestRaft_RemoveFollower (0.40s)
=== RUN   TestRaft_RemoveLeader
2026-05-25T23:22:34.502Z [INFO]  server-714e0810-af49-e8e2-dda1-1b3216176aa5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-714e0810-af49-e8e2-dda1-1b3216176aa5 Address:714e0810-af49-e8e2-dda1-1b3216176aa5} {Suffrage:Voter ID:server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff Address:32fd119d-1fea-b5b2-76d6-5d5d80d79dff} {Suffrage:Voter ID:server-58943e88-ff23-fcd8-b6ce-74bd4b7c1e20 Address:58943e88-ff23-fcd8-b6ce-74bd4b7c1e20}]"
2026-05-25T23:22:34.502Z [INFO]  server-714e0810-af49-e8e2-dda1-1b3216176aa5: entering follower state: follower="Node at 714e0810-af49-e8e2-dda1-1b3216176aa5 [Follower]" leader-address= leader-id=
2026-05-25T23:22:34.502Z [INFO]  server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-714e0810-af49-e8e2-dda1-1b3216176aa5 Address:714e0810-af49-e8e2-dda1-1b3216176aa5} {Suffrage:Voter ID:server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff Address:32fd119d-1fea-b5b2-76d6-5d5d80d79dff} {Suffrage:Voter ID:server-58943e88-ff23-fcd8-b6ce-74bd4b7c1e20 Address:58943e88-ff23-fcd8-b6ce-74bd4b7c1e20}]"
2026-05-25T23:22:34.502Z [INFO]  server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff: entering follower state: follower="Node at 32fd119d-1fea-b5b2-76d6-5d5d80d79dff [Follower]" leader-address= leader-id=
2026-05-25T23:22:34.502Z [INFO]  server-58943e88-ff23-fcd8-b6ce-74bd4b7c1e20: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-714e0810-af49-e8e2-dda1-1b3216176aa5 Address:714e0810-af49-e8e2-dda1-1b3216176aa5} {Suffrage:Voter ID:server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff Address:32fd119d-1fea-b5b2-76d6-5d5d80d79dff} {Suffrage:Voter ID:server-58943e88-ff23-fcd8-b6ce-74bd4b7c1e20 Address:58943e88-ff23-fcd8-b6ce-74bd4b7c1e20}]"
2026-05-25T23:22:34.502Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:34.502Z [INFO]  server-58943e88-ff23-fcd8-b6ce-74bd4b7c1e20: entering follower state: follower="Node at 58943e88-ff23-fcd8-b6ce-74bd4b7c1e20 [Follower]" leader-address= leader-id=
2026-05-25T23:22:34.570Z [WARN]  server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:34.570Z [INFO]  server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff: entering candidate state: node="Node at 32fd119d-1fea-b5b2-76d6-5d5d80d79dff [Candidate]" term=2
2026-05-25T23:22:34.571Z [INFO]  server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff: election won: term=2 tally=2
2026-05-25T23:22:34.571Z [INFO]  server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff: entering leader state: leader="Node at 32fd119d-1fea-b5b2-76d6-5d5d80d79dff [Leader]"
2026-05-25T23:22:34.571Z [INFO]  server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff: added peer, starting replication: peer=server-714e0810-af49-e8e2-dda1-1b3216176aa5
2026-05-25T23:22:34.571Z [INFO]  server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff: added peer, starting replication: peer=server-58943e88-ff23-fcd8-b6ce-74bd4b7c1e20
2026-05-25T23:22:34.571Z [INFO]  server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff: pipelining replication: peer="{Voter server-714e0810-af49-e8e2-dda1-1b3216176aa5 714e0810-af49-e8e2-dda1-1b3216176aa5}"
2026-05-25T23:22:34.571Z [INFO]  server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff: pipelining replication: peer="{Voter server-58943e88-ff23-fcd8-b6ce-74bd4b7c1e20 58943e88-ff23-fcd8-b6ce-74bd4b7c1e20}"
2026-05-25T23:22:34.676Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:34.571095993 +0000 UTC m=+10.217543448 (1 nodes), 68.300716ms from start of poll, 68.666465ms from cluster start. Timeout at 2026-05-25 23:22:34.676100098 +0000 UTC m=+10.322547575, 105.004127ms after stability
2026-05-25T23:22:34.681Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:22:34.787Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:22:34.682004504 +0000 UTC m=+10.328451981 (2 nodes), 851ns from start of poll, 179.574998ms from cluster start. Timeout at 2026-05-25 23:22:34.787008834 +0000 UTC m=+10.433456289, 105.004308ms after stability
2026-05-25T23:22:34.787Z [INFO]  server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff: updating configuration: command=RemoveServer server-id=server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff server-addr= servers="[{Suffrage:Voter ID:server-714e0810-af49-e8e2-dda1-1b3216176aa5 Address:714e0810-af49-e8e2-dda1-1b3216176aa5} {Suffrage:Voter ID:server-58943e88-ff23-fcd8-b6ce-74bd4b7c1e20 Address:58943e88-ff23-fcd8-b6ce-74bd4b7c1e20}]"
2026-05-25T23:22:34.788Z [INFO]  server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff: removed ourself, shutting down
2026-05-25T23:22:34.788Z [INFO]  server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff: aborting pipeline replication: peer="{Voter server-714e0810-af49-e8e2-dda1-1b3216176aa5 714e0810-af49-e8e2-dda1-1b3216176aa5}"
2026-05-25T23:22:34.788Z [INFO]  server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff: aborting pipeline replication: peer="{Voter server-58943e88-ff23-fcd8-b6ce-74bd4b7c1e20 58943e88-ff23-fcd8-b6ce-74bd4b7c1e20}"
2026-05-25T23:22:34.858Z [WARN]  server-58943e88-ff23-fcd8-b6ce-74bd4b7c1e20: heartbeat timeout reached, starting election: last-leader-addr=32fd119d-1fea-b5b2-76d6-5d5d80d79dff last-leader-id=server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff
2026-05-25T23:22:34.858Z [INFO]  server-58943e88-ff23-fcd8-b6ce-74bd4b7c1e20: entering candidate state: node="Node at 58943e88-ff23-fcd8-b6ce-74bd4b7c1e20 [Candidate]" term=3
2026-05-25T23:22:34.858Z [WARN]  server-714e0810-af49-e8e2-dda1-1b3216176aa5: rejecting vote request since we have a leader: from=58943e88-ff23-fcd8-b6ce-74bd4b7c1e20 leader=32fd119d-1fea-b5b2-76d6-5d5d80d79dff leader-id=server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff
2026-05-25T23:22:34.882Z [WARN]  server-714e0810-af49-e8e2-dda1-1b3216176aa5: heartbeat timeout reached, starting election: last-leader-addr=32fd119d-1fea-b5b2-76d6-5d5d80d79dff last-leader-id=server-32fd119d-1fea-b5b2-76d6-5d5d80d79dff
2026-05-25T23:22:34.882Z [INFO]  server-714e0810-af49-e8e2-dda1-1b3216176aa5: entering candidate state: node="Node at 714e0810-af49-e8e2-dda1-1b3216176aa5 [Candidate]" term=3
2026-05-25T23:22:34.882Z [INFO]  server-58943e88-ff23-fcd8-b6ce-74bd4b7c1e20: duplicate requestVote for same term: term=3
2026-05-25T23:22:34.939Z [WARN]  server-714e0810-af49-e8e2-dda1-1b3216176aa5: Election timeout reached, restarting election
2026-05-25T23:22:34.939Z [INFO]  server-714e0810-af49-e8e2-dda1-1b3216176aa5: entering candidate state: node="Node at 714e0810-af49-e8e2-dda1-1b3216176aa5 [Candidate]" term=4
2026-05-25T23:22:34.939Z [INFO]  server-58943e88-ff23-fcd8-b6ce-74bd4b7c1e20: entering follower state: follower="Node at 58943e88-ff23-fcd8-b6ce-74bd4b7c1e20 [Follower]" leader-address= leader-id=
2026-05-25T23:22:34.939Z [INFO]  server-714e0810-af49-e8e2-dda1-1b3216176aa5: election won: term=4 tally=2
2026-05-25T23:22:34.939Z [INFO]  server-714e0810-af49-e8e2-dda1-1b3216176aa5: entering leader state: leader="Node at 714e0810-af49-e8e2-dda1-1b3216176aa5 [Leader]"
2026-05-25T23:22:34.939Z [INFO]  server-714e0810-af49-e8e2-dda1-1b3216176aa5: added peer, starting replication: peer=server-58943e88-ff23-fcd8-b6ce-74bd4b7c1e20
2026-05-25T23:22:34.939Z [INFO]  server-714e0810-af49-e8e2-dda1-1b3216176aa5: pipelining replication: peer="{Voter server-58943e88-ff23-fcd8-b6ce-74bd4b7c1e20 58943e88-ff23-fcd8-b6ce-74bd4b7c1e20}"
2026-05-25T23:22:34.998Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:35.104Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:34.999012887 +0000 UTC m=+10.645460404 (1 nodes), 78.836µs from start of poll, 496.583421ms from cluster start. Timeout at 2026-05-25 23:22:35.104016335 +0000 UTC m=+10.750463789, 105.003385ms after stability
2026-05-25T23:22:35.104Z [INFO]  server-714e0810-af49-e8e2-dda1-1b3216176aa5: aborting pipeline replication: peer="{Voter server-58943e88-ff23-fcd8-b6ce-74bd4b7c1e20 58943e88-ff23-fcd8-b6ce-74bd4b7c1e20}"
--- PASS: TestRaft_RemoveLeader (0.60s)
=== RUN   TestRaft_RemoveLeader_NoShutdown
2026-05-25T23:22:35.105Z [INFO]  server-b8147b6f-f74e-5603-385f-bc81ee945eda: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b8147b6f-f74e-5603-385f-bc81ee945eda Address:b8147b6f-f74e-5603-385f-bc81ee945eda} {Suffrage:Voter ID:server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c Address:9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c} {Suffrage:Voter ID:server-1edb13fd-681d-08ac-2677-607521094b83 Address:1edb13fd-681d-08ac-2677-607521094b83}]"
2026-05-25T23:22:35.105Z [INFO]  server-b8147b6f-f74e-5603-385f-bc81ee945eda: entering follower state: follower="Node at b8147b6f-f74e-5603-385f-bc81ee945eda [Follower]" leader-address= leader-id=
2026-05-25T23:22:35.105Z [INFO]  server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b8147b6f-f74e-5603-385f-bc81ee945eda Address:b8147b6f-f74e-5603-385f-bc81ee945eda} {Suffrage:Voter ID:server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c Address:9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c} {Suffrage:Voter ID:server-1edb13fd-681d-08ac-2677-607521094b83 Address:1edb13fd-681d-08ac-2677-607521094b83}]"
2026-05-25T23:22:35.105Z [INFO]  server-1edb13fd-681d-08ac-2677-607521094b83: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b8147b6f-f74e-5603-385f-bc81ee945eda Address:b8147b6f-f74e-5603-385f-bc81ee945eda} {Suffrage:Voter ID:server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c Address:9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c} {Suffrage:Voter ID:server-1edb13fd-681d-08ac-2677-607521094b83 Address:1edb13fd-681d-08ac-2677-607521094b83}]"
2026-05-25T23:22:35.105Z [INFO]  server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c: entering follower state: follower="Node at 9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c [Follower]" leader-address= leader-id=
2026-05-25T23:22:35.105Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:22:35.105Z [INFO]  server-1edb13fd-681d-08ac-2677-607521094b83: entering follower state: follower="Node at 1edb13fd-681d-08ac-2677-607521094b83 [Follower]" leader-address= leader-id=
2026-05-25T23:22:35.161Z [WARN]  server-1edb13fd-681d-08ac-2677-607521094b83: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:35.161Z [INFO]  server-1edb13fd-681d-08ac-2677-607521094b83: entering candidate state: node="Node at 1edb13fd-681d-08ac-2677-607521094b83 [Candidate]" term=2
2026-05-25T23:22:35.161Z [INFO]  server-1edb13fd-681d-08ac-2677-607521094b83: election won: term=2 tally=2
2026-05-25T23:22:35.161Z [INFO]  server-1edb13fd-681d-08ac-2677-607521094b83: entering leader state: leader="Node at 1edb13fd-681d-08ac-2677-607521094b83 [Leader]"
2026-05-25T23:22:35.161Z [INFO]  server-1edb13fd-681d-08ac-2677-607521094b83: added peer, starting replication: peer=server-b8147b6f-f74e-5603-385f-bc81ee945eda
2026-05-25T23:22:35.161Z [INFO]  server-1edb13fd-681d-08ac-2677-607521094b83: added peer, starting replication: peer=server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c
2026-05-25T23:22:35.161Z [INFO]  server-1edb13fd-681d-08ac-2677-607521094b83: pipelining replication: peer="{Voter server-b8147b6f-f74e-5603-385f-bc81ee945eda b8147b6f-f74e-5603-385f-bc81ee945eda}"
2026-05-25T23:22:35.162Z [INFO]  server-1edb13fd-681d-08ac-2677-607521094b83: pipelining replication: peer="{Voter server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c 9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c}"
2026-05-25T23:22:35.267Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:22:35.161848996 +0000 UTC m=+10.808296446 (2 nodes), 56.013051ms from start of poll, 56.343064ms from cluster start. Timeout at 2026-05-25 23:22:35.26684994 +0000 UTC m=+10.913297427, 105.000981ms after stability
2026-05-25T23:22:35.267Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:35.372Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:35.267332365 +0000 UTC m=+10.913779819 (1 nodes), 925ns from start of poll, 161.826437ms from cluster start. Timeout at 2026-05-25 23:22:35.372339154 +0000 UTC m=+11.018786605, 105.006786ms after stability
2026-05-25T23:22:35.375Z [INFO]  server-1edb13fd-681d-08ac-2677-607521094b83: updating configuration: command=RemoveServer server-id=server-1edb13fd-681d-08ac-2677-607521094b83 server-addr= servers="[{Suffrage:Voter ID:server-b8147b6f-f74e-5603-385f-bc81ee945eda Address:b8147b6f-f74e-5603-385f-bc81ee945eda} {Suffrage:Voter ID:server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c Address:9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c}]"
2026-05-25T23:22:35.376Z [INFO]  server-1edb13fd-681d-08ac-2677-607521094b83: removed ourself, transitioning to follower
2026-05-25T23:22:35.376Z [INFO]  server-1edb13fd-681d-08ac-2677-607521094b83: entering follower state: follower="Node at 1edb13fd-681d-08ac-2677-607521094b83 [Follower]" leader-address= leader-id=
2026-05-25T23:22:35.376Z [INFO]  server-1edb13fd-681d-08ac-2677-607521094b83: aborting pipeline replication: peer="{Voter server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c 9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c}"
2026-05-25T23:22:35.376Z [INFO]  server-1edb13fd-681d-08ac-2677-607521094b83: aborting pipeline replication: peer="{Voter server-b8147b6f-f74e-5603-385f-bc81ee945eda b8147b6f-f74e-5603-385f-bc81ee945eda}"
2026-05-25T23:22:35.453Z [WARN]  server-1edb13fd-681d-08ac-2677-607521094b83: not part of stable configuration, aborting election
2026-05-25T23:22:35.454Z [WARN]  server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c: heartbeat timeout reached, starting election: last-leader-addr=1edb13fd-681d-08ac-2677-607521094b83 last-leader-id=server-1edb13fd-681d-08ac-2677-607521094b83
2026-05-25T23:22:35.454Z [INFO]  server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c: entering candidate state: node="Node at 9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c [Candidate]" term=3
2026-05-25T23:22:35.454Z [WARN]  server-b8147b6f-f74e-5603-385f-bc81ee945eda: rejecting vote request since we have a leader: from=9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c leader=1edb13fd-681d-08ac-2677-607521094b83 leader-id=server-1edb13fd-681d-08ac-2677-607521094b83
2026-05-25T23:22:35.479Z [WARN]  server-b8147b6f-f74e-5603-385f-bc81ee945eda: heartbeat timeout reached, starting election: last-leader-addr=1edb13fd-681d-08ac-2677-607521094b83 last-leader-id=server-1edb13fd-681d-08ac-2677-607521094b83
2026-05-25T23:22:35.479Z [INFO]  server-b8147b6f-f74e-5603-385f-bc81ee945eda: entering candidate state: node="Node at b8147b6f-f74e-5603-385f-bc81ee945eda [Candidate]" term=3
2026-05-25T23:22:35.479Z [INFO]  server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c: duplicate requestVote for same term: term=3
2026-05-25T23:22:35.482Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:35.534Z [WARN]  server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c: Election timeout reached, restarting election
2026-05-25T23:22:35.534Z [INFO]  server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c: entering candidate state: node="Node at 9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c [Candidate]" term=4
2026-05-25T23:22:35.534Z [INFO]  server-b8147b6f-f74e-5603-385f-bc81ee945eda: entering follower state: follower="Node at b8147b6f-f74e-5603-385f-bc81ee945eda [Follower]" leader-address= leader-id=
2026-05-25T23:22:35.534Z [INFO]  server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c: election won: term=4 tally=2
2026-05-25T23:22:35.534Z [INFO]  server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c: entering leader state: leader="Node at 9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c [Leader]"
2026-05-25T23:22:35.534Z [INFO]  server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c: added peer, starting replication: peer=server-b8147b6f-f74e-5603-385f-bc81ee945eda
2026-05-25T23:22:35.534Z [INFO]  server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c: pipelining replication: peer="{Voter server-b8147b6f-f74e-5603-385f-bc81ee945eda b8147b6f-f74e-5603-385f-bc81ee945eda}"
2026-05-25T23:22:35.639Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:35.534765644 +0000 UTC m=+11.181213113 (1 nodes), 52.460548ms from start of poll, 429.259731ms from cluster start. Timeout at 2026-05-25 23:22:35.639768274 +0000 UTC m=+11.286215728, 105.002615ms after stability
2026-05-25T23:22:35.745Z [INFO]  server-9b6494d3-9fc8-1c5d-2dc3-c3603c531b8c: aborting pipeline replication: peer="{Voter server-b8147b6f-f74e-5603-385f-bc81ee945eda b8147b6f-f74e-5603-385f-bc81ee945eda}"
--- PASS: TestRaft_RemoveLeader_NoShutdown (0.64s)
=== RUN   TestRaft_RemoveFollower_SplitCluster
2026-05-25T23:22:35.746Z [INFO]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0591acb1-93c1-c7cb-7529-3c3f21d9c650 Address:0591acb1-93c1-c7cb-7529-3c3f21d9c650} {Suffrage:Voter ID:server-7c0529f4-40bd-43ff-4fb2-a7015606b8a6 Address:7c0529f4-40bd-43ff-4fb2-a7015606b8a6} {Suffrage:Voter ID:server-b87f3e5a-18b6-230b-f7f0-f5645274497e Address:b87f3e5a-18b6-230b-f7f0-f5645274497e} {Suffrage:Voter ID:server-abc53876-1cdb-623b-066c-35d20e505719 Address:abc53876-1cdb-623b-066c-35d20e505719}]"
2026-05-25T23:22:35.747Z [INFO]  server-7c0529f4-40bd-43ff-4fb2-a7015606b8a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0591acb1-93c1-c7cb-7529-3c3f21d9c650 Address:0591acb1-93c1-c7cb-7529-3c3f21d9c650} {Suffrage:Voter ID:server-7c0529f4-40bd-43ff-4fb2-a7015606b8a6 Address:7c0529f4-40bd-43ff-4fb2-a7015606b8a6} {Suffrage:Voter ID:server-b87f3e5a-18b6-230b-f7f0-f5645274497e Address:b87f3e5a-18b6-230b-f7f0-f5645274497e} {Suffrage:Voter ID:server-abc53876-1cdb-623b-066c-35d20e505719 Address:abc53876-1cdb-623b-066c-35d20e505719}]"
2026-05-25T23:22:35.747Z [INFO]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: entering follower state: follower="Node at 0591acb1-93c1-c7cb-7529-3c3f21d9c650 [Follower]" leader-address= leader-id=
2026-05-25T23:22:35.747Z [INFO]  server-7c0529f4-40bd-43ff-4fb2-a7015606b8a6: entering follower state: follower="Node at 7c0529f4-40bd-43ff-4fb2-a7015606b8a6 [Follower]" leader-address= leader-id=
2026-05-25T23:22:35.747Z [INFO]  server-b87f3e5a-18b6-230b-f7f0-f5645274497e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0591acb1-93c1-c7cb-7529-3c3f21d9c650 Address:0591acb1-93c1-c7cb-7529-3c3f21d9c650} {Suffrage:Voter ID:server-7c0529f4-40bd-43ff-4fb2-a7015606b8a6 Address:7c0529f4-40bd-43ff-4fb2-a7015606b8a6} {Suffrage:Voter ID:server-b87f3e5a-18b6-230b-f7f0-f5645274497e Address:b87f3e5a-18b6-230b-f7f0-f5645274497e} {Suffrage:Voter ID:server-abc53876-1cdb-623b-066c-35d20e505719 Address:abc53876-1cdb-623b-066c-35d20e505719}]"
2026-05-25T23:22:35.747Z [INFO]  server-b87f3e5a-18b6-230b-f7f0-f5645274497e: entering follower state: follower="Node at b87f3e5a-18b6-230b-f7f0-f5645274497e [Follower]" leader-address= leader-id=
2026-05-25T23:22:35.747Z [INFO]  server-abc53876-1cdb-623b-066c-35d20e505719: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0591acb1-93c1-c7cb-7529-3c3f21d9c650 Address:0591acb1-93c1-c7cb-7529-3c3f21d9c650} {Suffrage:Voter ID:server-7c0529f4-40bd-43ff-4fb2-a7015606b8a6 Address:7c0529f4-40bd-43ff-4fb2-a7015606b8a6} {Suffrage:Voter ID:server-b87f3e5a-18b6-230b-f7f0-f5645274497e Address:b87f3e5a-18b6-230b-f7f0-f5645274497e} {Suffrage:Voter ID:server-abc53876-1cdb-623b-066c-35d20e505719 Address:abc53876-1cdb-623b-066c-35d20e505719}]"
2026-05-25T23:22:35.747Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:35.747Z [INFO]  server-abc53876-1cdb-623b-066c-35d20e505719: entering follower state: follower="Node at abc53876-1cdb-623b-066c-35d20e505719 [Follower]" leader-address= leader-id=
2026-05-25T23:22:35.821Z [WARN]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:35.821Z [INFO]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: entering candidate state: node="Node at 0591acb1-93c1-c7cb-7529-3c3f21d9c650 [Candidate]" term=2
2026-05-25T23:22:35.821Z [INFO]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: election won: term=2 tally=3
2026-05-25T23:22:35.821Z [INFO]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: entering leader state: leader="Node at 0591acb1-93c1-c7cb-7529-3c3f21d9c650 [Leader]"
2026-05-25T23:22:35.821Z [INFO]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: added peer, starting replication: peer=server-7c0529f4-40bd-43ff-4fb2-a7015606b8a6
2026-05-25T23:22:35.822Z [INFO]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: added peer, starting replication: peer=server-b87f3e5a-18b6-230b-f7f0-f5645274497e
2026-05-25T23:22:35.822Z [INFO]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: added peer, starting replication: peer=server-abc53876-1cdb-623b-066c-35d20e505719
2026-05-25T23:22:35.822Z [INFO]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: pipelining replication: peer="{Voter server-7c0529f4-40bd-43ff-4fb2-a7015606b8a6 7c0529f4-40bd-43ff-4fb2-a7015606b8a6}"
2026-05-25T23:22:35.822Z [INFO]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: pipelining replication: peer="{Voter server-abc53876-1cdb-623b-066c-35d20e505719 abc53876-1cdb-623b-066c-35d20e505719}"
2026-05-25T23:22:35.822Z [INFO]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: pipelining replication: peer="{Voter server-b87f3e5a-18b6-230b-f7f0-f5645274497e b87f3e5a-18b6-230b-f7f0-f5645274497e}"
2026-05-25T23:22:35.927Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:35.8220034 +0000 UTC m=+11.468450870 (1 nodes), 74.612267ms from start of poll, 75.16655ms from cluster start. Timeout at 2026-05-25 23:22:35.927004577 +0000 UTC m=+11.573452031, 105.001161ms after stability
2026-05-25T23:22:36.033Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:22:36.138Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:22:36.03308205 +0000 UTC m=+11.679529514 (3 nodes), 2.337µs from start of poll, 286.245194ms from cluster start. Timeout at 2026-05-25 23:22:36.138090694 +0000 UTC m=+11.784538148, 105.008634ms after stability
2026-05-25T23:22:36.138Z [INFO]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: updating configuration: command=RemoveServer server-id=server-abc53876-1cdb-623b-066c-35d20e505719 server-addr= servers="[{Suffrage:Voter ID:server-0591acb1-93c1-c7cb-7529-3c3f21d9c650 Address:0591acb1-93c1-c7cb-7529-3c3f21d9c650} {Suffrage:Voter ID:server-7c0529f4-40bd-43ff-4fb2-a7015606b8a6 Address:7c0529f4-40bd-43ff-4fb2-a7015606b8a6} {Suffrage:Voter ID:server-b87f3e5a-18b6-230b-f7f0-f5645274497e Address:b87f3e5a-18b6-230b-f7f0-f5645274497e}]"
2026-05-25T23:22:36.138Z [INFO]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: removed peer, stopping replication: peer=server-abc53876-1cdb-623b-066c-35d20e505719 last-index=3
2026-05-25T23:22:36.138Z [ERROR] server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to pipeline appendEntries: peer="{Voter server-7c0529f4-40bd-43ff-4fb2-a7015606b8a6 7c0529f4-40bd-43ff-4fb2-a7015606b8a6}" error="append pipeline closed"
2026-05-25T23:22:36.138Z [INFO]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: aborting pipeline replication: peer="{Voter server-abc53876-1cdb-623b-066c-35d20e505719 abc53876-1cdb-623b-066c-35d20e505719}"
2026-05-25T23:22:36.138Z [ERROR] server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to pipeline appendEntries: peer="{Voter server-b87f3e5a-18b6-230b-f7f0-f5645274497e b87f3e5a-18b6-230b-f7f0-f5645274497e}" error="append pipeline closed"
2026-05-25T23:22:36.138Z [INFO]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: aborting pipeline replication: peer="{Voter server-b87f3e5a-18b6-230b-f7f0-f5645274497e b87f3e5a-18b6-230b-f7f0-f5645274497e}"
2026-05-25T23:22:36.138Z [INFO]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: aborting pipeline replication: peer="{Voter server-7c0529f4-40bd-43ff-4fb2-a7015606b8a6 7c0529f4-40bd-43ff-4fb2-a7015606b8a6}"
2026-05-25T23:22:36.142Z [ERROR] server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to heartbeat to: peer=7c0529f4-40bd-43ff-4fb2-a7015606b8a6 backoff time=10ms error="failed to connect to peer: 7c0529f4-40bd-43ff-4fb2-a7015606b8a6"
2026-05-25T23:22:36.145Z [ERROR] server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to appendEntries to: peer="{Voter server-7c0529f4-40bd-43ff-4fb2-a7015606b8a6 7c0529f4-40bd-43ff-4fb2-a7015606b8a6}" error="failed to connect to peer: 7c0529f4-40bd-43ff-4fb2-a7015606b8a6"
2026-05-25T23:22:36.149Z [ERROR] server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to appendEntries to: peer="{Voter server-b87f3e5a-18b6-230b-f7f0-f5645274497e b87f3e5a-18b6-230b-f7f0-f5645274497e}" error="failed to connect to peer: b87f3e5a-18b6-230b-f7f0-f5645274497e"
2026-05-25T23:22:36.149Z [ERROR] server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to heartbeat to: peer=b87f3e5a-18b6-230b-f7f0-f5645274497e backoff time=10ms error="failed to connect to peer: b87f3e5a-18b6-230b-f7f0-f5645274497e"
2026-05-25T23:22:36.160Z [ERROR] server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to heartbeat to: peer=7c0529f4-40bd-43ff-4fb2-a7015606b8a6 backoff time=10ms error="failed to connect to peer: 7c0529f4-40bd-43ff-4fb2-a7015606b8a6"
2026-05-25T23:22:36.165Z [ERROR] server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to appendEntries to: peer="{Voter server-7c0529f4-40bd-43ff-4fb2-a7015606b8a6 7c0529f4-40bd-43ff-4fb2-a7015606b8a6}" error="failed to connect to peer: 7c0529f4-40bd-43ff-4fb2-a7015606b8a6"
2026-05-25T23:22:36.165Z [ERROR] server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to appendEntries to: peer="{Voter server-b87f3e5a-18b6-230b-f7f0-f5645274497e b87f3e5a-18b6-230b-f7f0-f5645274497e}" error="failed to connect to peer: b87f3e5a-18b6-230b-f7f0-f5645274497e"
2026-05-25T23:22:36.167Z [ERROR] server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to heartbeat to: peer=b87f3e5a-18b6-230b-f7f0-f5645274497e backoff time=10ms error="failed to connect to peer: b87f3e5a-18b6-230b-f7f0-f5645274497e"
2026-05-25T23:22:36.177Z [ERROR] server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to heartbeat to: peer=7c0529f4-40bd-43ff-4fb2-a7015606b8a6 backoff time=10ms error="failed to connect to peer: 7c0529f4-40bd-43ff-4fb2-a7015606b8a6"
2026-05-25T23:22:36.183Z [WARN]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to contact: server-id=server-7c0529f4-40bd-43ff-4fb2-a7015606b8a6 time=50.824789ms
2026-05-25T23:22:36.183Z [ERROR] server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to appendEntries to: peer="{Voter server-b87f3e5a-18b6-230b-f7f0-f5645274497e b87f3e5a-18b6-230b-f7f0-f5645274497e}" error="failed to connect to peer: b87f3e5a-18b6-230b-f7f0-f5645274497e"
2026-05-25T23:22:36.184Z [ERROR] server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to appendEntries to: peer="{Voter server-7c0529f4-40bd-43ff-4fb2-a7015606b8a6 7c0529f4-40bd-43ff-4fb2-a7015606b8a6}" error="failed to connect to peer: 7c0529f4-40bd-43ff-4fb2-a7015606b8a6"
2026-05-25T23:22:36.186Z [ERROR] server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to heartbeat to: peer=b87f3e5a-18b6-230b-f7f0-f5645274497e backoff time=10ms error="failed to connect to peer: b87f3e5a-18b6-230b-f7f0-f5645274497e"
2026-05-25T23:22:36.194Z [WARN]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to contact: server-id=server-7c0529f4-40bd-43ff-4fb2-a7015606b8a6 time=61.663473ms
2026-05-25T23:22:36.194Z [WARN]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to contact: server-id=server-b87f3e5a-18b6-230b-f7f0-f5645274497e time=56.182859ms
2026-05-25T23:22:36.194Z [WARN]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to contact quorum of nodes, stepping down
2026-05-25T23:22:36.194Z [INFO]  server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: entering follower state: follower="Node at 0591acb1-93c1-c7cb-7529-3c3f21d9c650 [Follower]" leader-address= leader-id=
2026-05-25T23:22:36.194Z [ERROR] server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to heartbeat to: peer=7c0529f4-40bd-43ff-4fb2-a7015606b8a6 backoff time=20ms error="failed to connect to peer: 7c0529f4-40bd-43ff-4fb2-a7015606b8a6"
2026-05-25T23:22:36.194Z [ERROR] server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to appendEntries to: peer="{Voter server-b87f3e5a-18b6-230b-f7f0-f5645274497e b87f3e5a-18b6-230b-f7f0-f5645274497e}" error="failed to connect to peer: b87f3e5a-18b6-230b-f7f0-f5645274497e"
2026-05-25T23:22:36.194Z [ERROR] server-0591acb1-93c1-c7cb-7529-3c3f21d9c650: failed to appendEntries to: peer="{Voter server-7c0529f4-40bd-43ff-4fb2-a7015606b8a6 7c0529f4-40bd-43ff-4fb2-a7015606b8a6}" error="failed to connect to peer: 7c0529f4-40bd-43ff-4fb2-a7015606b8a6"
--- PASS: TestRaft_RemoveFollower_SplitCluster (0.45s)
=== RUN   TestRaft_AddKnownPeer
2026-05-25T23:22:36.195Z [INFO]  server-d306d91b-1f27-cd93-f7ea-170324965fbe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d306d91b-1f27-cd93-f7ea-170324965fbe Address:d306d91b-1f27-cd93-f7ea-170324965fbe} {Suffrage:Voter ID:server-ee18230f-d56e-109c-def6-7e85d781e34b Address:ee18230f-d56e-109c-def6-7e85d781e34b} {Suffrage:Voter ID:server-61c98e68-8ea6-65b2-bfbc-c29797cfce91 Address:61c98e68-8ea6-65b2-bfbc-c29797cfce91}]"
2026-05-25T23:22:36.196Z [INFO]  server-ee18230f-d56e-109c-def6-7e85d781e34b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d306d91b-1f27-cd93-f7ea-170324965fbe Address:d306d91b-1f27-cd93-f7ea-170324965fbe} {Suffrage:Voter ID:server-ee18230f-d56e-109c-def6-7e85d781e34b Address:ee18230f-d56e-109c-def6-7e85d781e34b} {Suffrage:Voter ID:server-61c98e68-8ea6-65b2-bfbc-c29797cfce91 Address:61c98e68-8ea6-65b2-bfbc-c29797cfce91}]"
2026-05-25T23:22:36.196Z [INFO]  server-ee18230f-d56e-109c-def6-7e85d781e34b: entering follower state: follower="Node at ee18230f-d56e-109c-def6-7e85d781e34b [Follower]" leader-address= leader-id=
2026-05-25T23:22:36.196Z [INFO]  server-d306d91b-1f27-cd93-f7ea-170324965fbe: entering follower state: follower="Node at d306d91b-1f27-cd93-f7ea-170324965fbe [Follower]" leader-address= leader-id=
2026-05-25T23:22:36.196Z [INFO]  server-61c98e68-8ea6-65b2-bfbc-c29797cfce91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d306d91b-1f27-cd93-f7ea-170324965fbe Address:d306d91b-1f27-cd93-f7ea-170324965fbe} {Suffrage:Voter ID:server-ee18230f-d56e-109c-def6-7e85d781e34b Address:ee18230f-d56e-109c-def6-7e85d781e34b} {Suffrage:Voter ID:server-61c98e68-8ea6-65b2-bfbc-c29797cfce91 Address:61c98e68-8ea6-65b2-bfbc-c29797cfce91}]"
2026-05-25T23:22:36.196Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:36.196Z [INFO]  server-61c98e68-8ea6-65b2-bfbc-c29797cfce91: entering follower state: follower="Node at 61c98e68-8ea6-65b2-bfbc-c29797cfce91 [Follower]" leader-address= leader-id=
2026-05-25T23:22:36.258Z [WARN]  server-61c98e68-8ea6-65b2-bfbc-c29797cfce91: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:36.258Z [INFO]  server-61c98e68-8ea6-65b2-bfbc-c29797cfce91: entering candidate state: node="Node at 61c98e68-8ea6-65b2-bfbc-c29797cfce91 [Candidate]" term=2
2026-05-25T23:22:36.258Z [INFO]  server-61c98e68-8ea6-65b2-bfbc-c29797cfce91: election won: term=2 tally=2
2026-05-25T23:22:36.258Z [INFO]  server-61c98e68-8ea6-65b2-bfbc-c29797cfce91: entering leader state: leader="Node at 61c98e68-8ea6-65b2-bfbc-c29797cfce91 [Leader]"
2026-05-25T23:22:36.258Z [INFO]  server-61c98e68-8ea6-65b2-bfbc-c29797cfce91: added peer, starting replication: peer=server-d306d91b-1f27-cd93-f7ea-170324965fbe
2026-05-25T23:22:36.258Z [INFO]  server-61c98e68-8ea6-65b2-bfbc-c29797cfce91: added peer, starting replication: peer=server-ee18230f-d56e-109c-def6-7e85d781e34b
2026-05-25T23:22:36.258Z [INFO]  server-61c98e68-8ea6-65b2-bfbc-c29797cfce91: pipelining replication: peer="{Voter server-ee18230f-d56e-109c-def6-7e85d781e34b ee18230f-d56e-109c-def6-7e85d781e34b}"
2026-05-25T23:22:36.258Z [INFO]  server-61c98e68-8ea6-65b2-bfbc-c29797cfce91: pipelining replication: peer="{Voter server-d306d91b-1f27-cd93-f7ea-170324965fbe d306d91b-1f27-cd93-f7ea-170324965fbe}"
2026-05-25T23:22:36.364Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:36.258660823 +0000 UTC m=+11.905108285 (1 nodes), 62.50967ms from start of poll, 62.86678ms from cluster start. Timeout at 2026-05-25 23:22:36.363666747 +0000 UTC m=+12.010114237, 105.005952ms after stability
2026-05-25T23:22:36.364Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:22:36.470Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:22:36.364774321 +0000 UTC m=+12.011221812 (2 nodes), 1.273µs from start of poll, 168.980307ms from cluster start. Timeout at 2026-05-25 23:22:36.469784716 +0000 UTC m=+12.116232170, 105.010358ms after stability
2026-05-25T23:22:36.470Z [INFO]  server-61c98e68-8ea6-65b2-bfbc-c29797cfce91: updating configuration: command=AddVoter server-id=server-d306d91b-1f27-cd93-f7ea-170324965fbe server-addr=d306d91b-1f27-cd93-f7ea-170324965fbe servers="[{Suffrage:Voter ID:server-d306d91b-1f27-cd93-f7ea-170324965fbe Address:d306d91b-1f27-cd93-f7ea-170324965fbe} {Suffrage:Voter ID:server-ee18230f-d56e-109c-def6-7e85d781e34b Address:ee18230f-d56e-109c-def6-7e85d781e34b} {Suffrage:Voter ID:server-61c98e68-8ea6-65b2-bfbc-c29797cfce91 Address:61c98e68-8ea6-65b2-bfbc-c29797cfce91}]"
2026-05-25T23:22:36.471Z [INFO]  server-61c98e68-8ea6-65b2-bfbc-c29797cfce91: aborting pipeline replication: peer="{Voter server-ee18230f-d56e-109c-def6-7e85d781e34b ee18230f-d56e-109c-def6-7e85d781e34b}"
2026-05-25T23:22:36.471Z [INFO]  server-61c98e68-8ea6-65b2-bfbc-c29797cfce91: aborting pipeline replication: peer="{Voter server-d306d91b-1f27-cd93-f7ea-170324965fbe d306d91b-1f27-cd93-f7ea-170324965fbe}"
--- PASS: TestRaft_AddKnownPeer (0.28s)
=== RUN   TestRaft_RemoveUnknownPeer
2026-05-25T23:22:36.472Z [INFO]  server-fe77d74f-5aa9-d357-ef42-c04dc859134d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe77d74f-5aa9-d357-ef42-c04dc859134d Address:fe77d74f-5aa9-d357-ef42-c04dc859134d} {Suffrage:Voter ID:server-e070f119-635f-ca93-a434-c427f9e73f6c Address:e070f119-635f-ca93-a434-c427f9e73f6c} {Suffrage:Voter ID:server-d0d140cf-0d93-9fc1-b4b0-c926fe0fa3fe Address:d0d140cf-0d93-9fc1-b4b0-c926fe0fa3fe}]"
2026-05-25T23:22:36.472Z [INFO]  server-fe77d74f-5aa9-d357-ef42-c04dc859134d: entering follower state: follower="Node at fe77d74f-5aa9-d357-ef42-c04dc859134d [Follower]" leader-address= leader-id=
2026-05-25T23:22:36.472Z [INFO]  server-e070f119-635f-ca93-a434-c427f9e73f6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe77d74f-5aa9-d357-ef42-c04dc859134d Address:fe77d74f-5aa9-d357-ef42-c04dc859134d} {Suffrage:Voter ID:server-e070f119-635f-ca93-a434-c427f9e73f6c Address:e070f119-635f-ca93-a434-c427f9e73f6c} {Suffrage:Voter ID:server-d0d140cf-0d93-9fc1-b4b0-c926fe0fa3fe Address:d0d140cf-0d93-9fc1-b4b0-c926fe0fa3fe}]"
2026-05-25T23:22:36.472Z [INFO]  server-e070f119-635f-ca93-a434-c427f9e73f6c: entering follower state: follower="Node at e070f119-635f-ca93-a434-c427f9e73f6c [Follower]" leader-address= leader-id=
2026-05-25T23:22:36.472Z [INFO]  server-d0d140cf-0d93-9fc1-b4b0-c926fe0fa3fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe77d74f-5aa9-d357-ef42-c04dc859134d Address:fe77d74f-5aa9-d357-ef42-c04dc859134d} {Suffrage:Voter ID:server-e070f119-635f-ca93-a434-c427f9e73f6c Address:e070f119-635f-ca93-a434-c427f9e73f6c} {Suffrage:Voter ID:server-d0d140cf-0d93-9fc1-b4b0-c926fe0fa3fe Address:d0d140cf-0d93-9fc1-b4b0-c926fe0fa3fe}]"
2026-05-25T23:22:36.472Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:36.472Z [INFO]  server-d0d140cf-0d93-9fc1-b4b0-c926fe0fa3fe: entering follower state: follower="Node at d0d140cf-0d93-9fc1-b4b0-c926fe0fa3fe [Follower]" leader-address= leader-id=
2026-05-25T23:22:36.529Z [WARN]  server-e070f119-635f-ca93-a434-c427f9e73f6c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:36.529Z [INFO]  server-e070f119-635f-ca93-a434-c427f9e73f6c: entering candidate state: node="Node at e070f119-635f-ca93-a434-c427f9e73f6c [Candidate]" term=2
2026-05-25T23:22:36.529Z [INFO]  server-e070f119-635f-ca93-a434-c427f9e73f6c: election won: term=2 tally=2
2026-05-25T23:22:36.529Z [INFO]  server-e070f119-635f-ca93-a434-c427f9e73f6c: entering leader state: leader="Node at e070f119-635f-ca93-a434-c427f9e73f6c [Leader]"
2026-05-25T23:22:36.529Z [INFO]  server-e070f119-635f-ca93-a434-c427f9e73f6c: added peer, starting replication: peer=server-fe77d74f-5aa9-d357-ef42-c04dc859134d
2026-05-25T23:22:36.529Z [INFO]  server-e070f119-635f-ca93-a434-c427f9e73f6c: added peer, starting replication: peer=server-d0d140cf-0d93-9fc1-b4b0-c926fe0fa3fe
2026-05-25T23:22:36.529Z [INFO]  server-e070f119-635f-ca93-a434-c427f9e73f6c: pipelining replication: peer="{Voter server-d0d140cf-0d93-9fc1-b4b0-c926fe0fa3fe d0d140cf-0d93-9fc1-b4b0-c926fe0fa3fe}"
2026-05-25T23:22:36.529Z [INFO]  server-e070f119-635f-ca93-a434-c427f9e73f6c: pipelining replication: peer="{Voter server-fe77d74f-5aa9-d357-ef42-c04dc859134d fe77d74f-5aa9-d357-ef42-c04dc859134d}"
2026-05-25T23:22:36.635Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:36.529872978 +0000 UTC m=+12.176320465 (1 nodes), 57.402651ms from start of poll, 57.763932ms from cluster start. Timeout at 2026-05-25 23:22:36.634878221 +0000 UTC m=+12.281325711, 105.005246ms after stability
2026-05-25T23:22:36.635Z [INFO]  server-e070f119-635f-ca93-a434-c427f9e73f6c: updating configuration: command=RemoveServer server-id=4731b625-38a6-8db1-05b9-32b4cd1e9aed server-addr= servers="[{Suffrage:Voter ID:server-fe77d74f-5aa9-d357-ef42-c04dc859134d Address:fe77d74f-5aa9-d357-ef42-c04dc859134d} {Suffrage:Voter ID:server-e070f119-635f-ca93-a434-c427f9e73f6c Address:e070f119-635f-ca93-a434-c427f9e73f6c} {Suffrage:Voter ID:server-d0d140cf-0d93-9fc1-b4b0-c926fe0fa3fe Address:d0d140cf-0d93-9fc1-b4b0-c926fe0fa3fe}]"
2026-05-25T23:22:36.635Z [INFO]  server-e070f119-635f-ca93-a434-c427f9e73f6c: aborting pipeline replication: peer="{Voter server-d0d140cf-0d93-9fc1-b4b0-c926fe0fa3fe d0d140cf-0d93-9fc1-b4b0-c926fe0fa3fe}"
2026-05-25T23:22:36.635Z [INFO]  server-e070f119-635f-ca93-a434-c427f9e73f6c: aborting pipeline replication: peer="{Voter server-fe77d74f-5aa9-d357-ef42-c04dc859134d fe77d74f-5aa9-d357-ef42-c04dc859134d}"
--- PASS: TestRaft_RemoveUnknownPeer (0.16s)
=== RUN   TestRaft_SnapshotRestore
2026-05-25T23:22:36.636Z [INFO]  server-d578544c-18f1-c321-2559-e838d7bcf590: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d578544c-18f1-c321-2559-e838d7bcf590 Address:d578544c-18f1-c321-2559-e838d7bcf590}]"
2026-05-25T23:22:36.636Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:36.636Z [INFO]  server-d578544c-18f1-c321-2559-e838d7bcf590: entering follower state: follower="Node at d578544c-18f1-c321-2559-e838d7bcf590 [Follower]" leader-address= leader-id=
2026-05-25T23:22:36.734Z [WARN]  server-d578544c-18f1-c321-2559-e838d7bcf590: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:36.734Z [INFO]  server-d578544c-18f1-c321-2559-e838d7bcf590: entering candidate state: node="Node at d578544c-18f1-c321-2559-e838d7bcf590 [Candidate]" term=2
2026-05-25T23:22:36.734Z [INFO]  server-d578544c-18f1-c321-2559-e838d7bcf590: election won: term=2 tally=1
2026-05-25T23:22:36.734Z [INFO]  server-d578544c-18f1-c321-2559-e838d7bcf590: entering leader state: leader="Node at d578544c-18f1-c321-2559-e838d7bcf590 [Leader]"
2026-05-25T23:22:36.840Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:36.735023885 +0000 UTC m=+12.381471386 (1 nodes), 98.853065ms from start of poll, 98.970091ms from cluster start. Timeout at 2026-05-25 23:22:36.840026251 +0000 UTC m=+12.486473705, 105.002319ms after stability
2026-05-25T23:22:36.841Z [INFO]  server-d578544c-18f1-c321-2559-e838d7bcf590: starting snapshot up to: index=102
2026-05-25T23:22:36.841Z [INFO]  creating new snapshot: path=/tmp/raft4238240543/snapshots/2-102-1779751356841.tmp
2026-05-25T23:22:36.841Z [INFO]  server-d578544c-18f1-c321-2559-e838d7bcf590: compacting logs: from=1 to=92
2026-05-25T23:22:36.841Z [INFO]  server-d578544c-18f1-c321-2559-e838d7bcf590: snapshot complete up to: index=102
2026-05-25T23:22:36.842Z [INFO]  server-d578544c-18f1-c321-2559-e838d7bcf590: starting restore from snapshot: id=2-102-1779751356841 last-index=102 last-term=2 size-in-bytes=693
2026-05-25T23:22:36.842Z [INFO]  server-d578544c-18f1-c321-2559-e838d7bcf590: snapshot restore progress: id=2-102-1779751356841 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-05-25T23:22:36.842Z [INFO]  server-d578544c-18f1-c321-2559-e838d7bcf590: restored from snapshot: id=2-102-1779751356841 last-index=102 last-term=2 size-in-bytes=693
2026-05-25T23:22:36.842Z [INFO]  server-d578544c-18f1-c321-2559-e838d7bcf590: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d578544c-18f1-c321-2559-e838d7bcf590 Address:d578544c-18f1-c321-2559-e838d7bcf590}]"
--- PASS: TestRaft_SnapshotRestore (0.21s)
=== RUN   TestRaft_RestoreSnapshotOnStartup_Monotonic
2026-05-25T23:22:36.843Z [INFO]  server-dbdbb48a-ecfb-3692-2c89-af3fd8028f62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dbdbb48a-ecfb-3692-2c89-af3fd8028f62 Address:dbdbb48a-ecfb-3692-2c89-af3fd8028f62}]"
2026-05-25T23:22:36.843Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:36.843Z [INFO]  server-dbdbb48a-ecfb-3692-2c89-af3fd8028f62: entering follower state: follower="Node at dbdbb48a-ecfb-3692-2c89-af3fd8028f62 [Follower]" leader-address= leader-id=
2026-05-25T23:22:36.937Z [WARN]  server-dbdbb48a-ecfb-3692-2c89-af3fd8028f62: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:36.937Z [INFO]  server-dbdbb48a-ecfb-3692-2c89-af3fd8028f62: entering candidate state: node="Node at dbdbb48a-ecfb-3692-2c89-af3fd8028f62 [Candidate]" term=2
2026-05-25T23:22:36.937Z [INFO]  server-dbdbb48a-ecfb-3692-2c89-af3fd8028f62: election won: term=2 tally=1
2026-05-25T23:22:36.937Z [INFO]  server-dbdbb48a-ecfb-3692-2c89-af3fd8028f62: entering leader state: leader="Node at dbdbb48a-ecfb-3692-2c89-af3fd8028f62 [Leader]"
2026-05-25T23:22:37.042Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:36.937207808 +0000 UTC m=+12.583655274 (1 nodes), 93.721553ms from start of poll, 93.825042ms from cluster start. Timeout at 2026-05-25 23:22:37.042212526 +0000 UTC m=+12.688660016, 105.004742ms after stability
2026-05-25T23:22:37.045Z [INFO]  server-dbdbb48a-ecfb-3692-2c89-af3fd8028f62: starting snapshot up to: index=102
2026-05-25T23:22:37.045Z [INFO]  creating new snapshot: path=/tmp/raft1384947412/snapshots/2-102-1779751357045.tmp
2026-05-25T23:22:37.046Z [INFO]  server-dbdbb48a-ecfb-3692-2c89-af3fd8028f62: compacting logs: from=1 to=92
2026-05-25T23:22:37.046Z [INFO]  server-dbdbb48a-ecfb-3692-2c89-af3fd8028f62: snapshot complete up to: index=102
2026-05-25T23:22:37.046Z [INFO]  server-dbdbb48a-ecfb-3692-2c89-af3fd8028f62: starting restore from snapshot: id=2-102-1779751357045 last-index=102 last-term=2 size-in-bytes=693
2026-05-25T23:22:37.046Z [INFO]  server-dbdbb48a-ecfb-3692-2c89-af3fd8028f62: snapshot restore progress: id=2-102-1779751357045 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-05-25T23:22:37.046Z [INFO]  server-dbdbb48a-ecfb-3692-2c89-af3fd8028f62: restored from snapshot: id=2-102-1779751357045 last-index=102 last-term=2 size-in-bytes=693
2026-05-25T23:22:37.046Z [INFO]  server-dbdbb48a-ecfb-3692-2c89-af3fd8028f62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dbdbb48a-ecfb-3692-2c89-af3fd8028f62 Address:dbdbb48a-ecfb-3692-2c89-af3fd8028f62}]"
--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.20s)
=== RUN   TestRaft_SnapshotRestore_Progress
2026-05-25T23:22:37.047Z [INFO]  server-1a45ec02-ab04-80d6-143a-ef9899a971d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a45ec02-ab04-80d6-143a-ef9899a971d1 Address:1a45ec02-ab04-80d6-143a-ef9899a971d1}]"
2026-05-25T23:22:37.047Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:37.047Z [INFO]  server-1a45ec02-ab04-80d6-143a-ef9899a971d1: entering follower state: follower="Node at 1a45ec02-ab04-80d6-143a-ef9899a971d1 [Follower]" leader-address= leader-id=
2026-05-25T23:22:37.142Z [WARN]  server-1a45ec02-ab04-80d6-143a-ef9899a971d1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:37.142Z [INFO]  server-1a45ec02-ab04-80d6-143a-ef9899a971d1: entering candidate state: node="Node at 1a45ec02-ab04-80d6-143a-ef9899a971d1 [Candidate]" term=2
2026-05-25T23:22:37.142Z [INFO]  server-1a45ec02-ab04-80d6-143a-ef9899a971d1: election won: term=2 tally=1
2026-05-25T23:22:37.142Z [INFO]  server-1a45ec02-ab04-80d6-143a-ef9899a971d1: entering leader state: leader="Node at 1a45ec02-ab04-80d6-143a-ef9899a971d1 [Leader]"
2026-05-25T23:22:37.247Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:37.142603607 +0000 UTC m=+12.789051056 (1 nodes), 95.520363ms from start of poll, 95.630337ms from cluster start. Timeout at 2026-05-25 23:22:37.247604188 +0000 UTC m=+12.894051641, 105.000585ms after stability
2026-05-25T23:22:37.249Z [INFO]  server-1a45ec02-ab04-80d6-143a-ef9899a971d1: starting snapshot up to: index=102
2026-05-25T23:22:37.249Z [INFO]  creating new snapshot: path=/tmp/raft3242302411/snapshots/2-102-1779751357249.tmp
2026-05-25T23:22:37.249Z [INFO]  server-1a45ec02-ab04-80d6-143a-ef9899a971d1: compacting logs: from=1 to=92
2026-05-25T23:22:37.249Z [INFO]  server-1a45ec02-ab04-80d6-143a-ef9899a971d1: snapshot complete up to: index=102
--- PASS: TestRaft_SnapshotRestore_Progress (0.20s)
=== RUN   TestRaft_NoRestoreOnStart
2026-05-25T23:22:37.251Z [INFO]  server-e18c3d1f-57e9-240c-f29a-f8507b32a983: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e18c3d1f-57e9-240c-f29a-f8507b32a983 Address:e18c3d1f-57e9-240c-f29a-f8507b32a983}]"
2026-05-25T23:22:37.251Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:37.251Z [INFO]  server-e18c3d1f-57e9-240c-f29a-f8507b32a983: entering follower state: follower="Node at e18c3d1f-57e9-240c-f29a-f8507b32a983 [Follower]" leader-address= leader-id=
2026-05-25T23:22:37.320Z [WARN]  server-e18c3d1f-57e9-240c-f29a-f8507b32a983: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:37.320Z [INFO]  server-e18c3d1f-57e9-240c-f29a-f8507b32a983: entering candidate state: node="Node at e18c3d1f-57e9-240c-f29a-f8507b32a983 [Candidate]" term=2
2026-05-25T23:22:37.320Z [INFO]  server-e18c3d1f-57e9-240c-f29a-f8507b32a983: election won: term=2 tally=1
2026-05-25T23:22:37.320Z [INFO]  server-e18c3d1f-57e9-240c-f29a-f8507b32a983: entering leader state: leader="Node at e18c3d1f-57e9-240c-f29a-f8507b32a983 [Leader]"
2026-05-25T23:22:37.425Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:37.32042543 +0000 UTC m=+12.966872920 (1 nodes), 68.886362ms from start of poll, 69.030616ms from cluster start. Timeout at 2026-05-25 23:22:37.425426096 +0000 UTC m=+13.071873548, 105.000628ms after stability
2026-05-25T23:22:37.427Z [INFO]  server-e18c3d1f-57e9-240c-f29a-f8507b32a983: starting snapshot up to: index=102
2026-05-25T23:22:37.427Z [INFO]  creating new snapshot: path=/tmp/raft1637184027/snapshots/2-102-1779751357427.tmp
2026-05-25T23:22:37.428Z [INFO]  server-e18c3d1f-57e9-240c-f29a-f8507b32a983: compacting logs: from=1 to=92
2026-05-25T23:22:37.428Z [INFO]  server-e18c3d1f-57e9-240c-f29a-f8507b32a983: snapshot complete up to: index=102
2026-05-25T23:22:37.428Z [INFO]  server-e18c3d1f-57e9-240c-f29a-f8507b32a983: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e18c3d1f-57e9-240c-f29a-f8507b32a983 Address:e18c3d1f-57e9-240c-f29a-f8507b32a983}]"
--- PASS: TestRaft_NoRestoreOnStart (0.18s)
=== RUN   TestRaft_SnapshotRestore_PeerChange
2026-05-25T23:22:37.428Z [INFO]  server-e18c3d1f-57e9-240c-f29a-f8507b32a983: entering follower state: follower="Node at e18c3d1f-57e9-240c-f29a-f8507b32a983 [Follower]" leader-address= leader-id=
    raft_test.go:1266: Skipping unreliable test
--- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
=== RUN   TestRaft_AutoSnapshot
2026-05-25T23:22:37.429Z [INFO]  server-da1c5053-7c7c-6b54-5b29-8ed26063a40f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da1c5053-7c7c-6b54-5b29-8ed26063a40f Address:da1c5053-7c7c-6b54-5b29-8ed26063a40f}]"
2026-05-25T23:22:37.429Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:37.429Z [INFO]  server-da1c5053-7c7c-6b54-5b29-8ed26063a40f: entering follower state: follower="Node at da1c5053-7c7c-6b54-5b29-8ed26063a40f [Follower]" leader-address= leader-id=
2026-05-25T23:22:37.496Z [WARN]  server-da1c5053-7c7c-6b54-5b29-8ed26063a40f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:37.496Z [INFO]  server-da1c5053-7c7c-6b54-5b29-8ed26063a40f: entering candidate state: node="Node at da1c5053-7c7c-6b54-5b29-8ed26063a40f [Candidate]" term=2
2026-05-25T23:22:37.496Z [INFO]  server-da1c5053-7c7c-6b54-5b29-8ed26063a40f: election won: term=2 tally=1
2026-05-25T23:22:37.496Z [INFO]  server-da1c5053-7c7c-6b54-5b29-8ed26063a40f: entering leader state: leader="Node at da1c5053-7c7c-6b54-5b29-8ed26063a40f [Leader]"
2026-05-25T23:22:37.526Z [WARN]  server-e18c3d1f-57e9-240c-f29a-f8507b32a983: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:37.526Z [INFO]  server-e18c3d1f-57e9-240c-f29a-f8507b32a983: entering candidate state: node="Node at e18c3d1f-57e9-240c-f29a-f8507b32a983 [Candidate]" term=3
2026-05-25T23:22:37.526Z [INFO]  server-e18c3d1f-57e9-240c-f29a-f8507b32a983: election won: term=3 tally=1
2026-05-25T23:22:37.526Z [INFO]  server-e18c3d1f-57e9-240c-f29a-f8507b32a983: entering leader state: leader="Node at e18c3d1f-57e9-240c-f29a-f8507b32a983 [Leader]"
2026-05-25T23:22:37.602Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:37.496736402 +0000 UTC m=+13.143183853 (1 nodes), 67.586611ms from start of poll, 67.720848ms from cluster start. Timeout at 2026-05-25 23:22:37.601742318 +0000 UTC m=+13.248189805, 105.005952ms after stability
2026-05-25T23:22:37.620Z [INFO]  server-da1c5053-7c7c-6b54-5b29-8ed26063a40f: starting snapshot up to: index=102
2026-05-25T23:22:37.620Z [INFO]  creating new snapshot: path=/tmp/raft3084670293/snapshots/2-102-1779751357620.tmp
2026-05-25T23:22:37.620Z [INFO]  server-da1c5053-7c7c-6b54-5b29-8ed26063a40f: compacting logs: from=1 to=92
2026-05-25T23:22:37.620Z [INFO]  server-da1c5053-7c7c-6b54-5b29-8ed26063a40f: snapshot complete up to: index=102
--- PASS: TestRaft_AutoSnapshot (0.28s)
=== RUN   TestRaft_UserSnapshot
2026-05-25T23:22:37.710Z [INFO]  server-29ed139d-412c-9bd5-fafb-c1e2e34e90f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29ed139d-412c-9bd5-fafb-c1e2e34e90f1 Address:29ed139d-412c-9bd5-fafb-c1e2e34e90f1}]"
2026-05-25T23:22:37.710Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:37.710Z [INFO]  server-29ed139d-412c-9bd5-fafb-c1e2e34e90f1: entering follower state: follower="Node at 29ed139d-412c-9bd5-fafb-c1e2e34e90f1 [Follower]" leader-address= leader-id=
2026-05-25T23:22:37.808Z [WARN]  server-29ed139d-412c-9bd5-fafb-c1e2e34e90f1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:37.808Z [INFO]  server-29ed139d-412c-9bd5-fafb-c1e2e34e90f1: entering candidate state: node="Node at 29ed139d-412c-9bd5-fafb-c1e2e34e90f1 [Candidate]" term=2
2026-05-25T23:22:37.808Z [INFO]  server-29ed139d-412c-9bd5-fafb-c1e2e34e90f1: election won: term=2 tally=1
2026-05-25T23:22:37.808Z [INFO]  server-29ed139d-412c-9bd5-fafb-c1e2e34e90f1: entering leader state: leader="Node at 29ed139d-412c-9bd5-fafb-c1e2e34e90f1 [Leader]"
2026-05-25T23:22:37.914Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:37.808890182 +0000 UTC m=+13.455337648 (1 nodes), 98.772611ms from start of poll, 98.877903ms from cluster start. Timeout at 2026-05-25 23:22:37.913892825 +0000 UTC m=+13.560340279, 105.002631ms after stability
2026-05-25T23:22:37.914Z [ERROR] server-29ed139d-412c-9bd5-fafb-c1e2e34e90f1: failed to take snapshot: error="nothing new to snapshot"
2026-05-25T23:22:37.914Z [INFO]  server-29ed139d-412c-9bd5-fafb-c1e2e34e90f1: starting snapshot up to: index=12
2026-05-25T23:22:37.914Z [INFO]  creating new snapshot: path=/tmp/raft3091322046/snapshots/2-12-1779751357914.tmp
2026-05-25T23:22:37.915Z [INFO]  server-29ed139d-412c-9bd5-fafb-c1e2e34e90f1: compacting logs: from=1 to=2
2026-05-25T23:22:37.915Z [INFO]  server-29ed139d-412c-9bd5-fafb-c1e2e34e90f1: snapshot complete up to: index=12
--- PASS: TestRaft_UserSnapshot (0.21s)
=== RUN   TestRaft_UserRestore
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false
2026-05-25T23:22:37.916Z [INFO]  server-b1e29b0b-6516-c874-6d32-4826f04971e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1e29b0b-6516-c874-6d32-4826f04971e3 Address:b1e29b0b-6516-c874-6d32-4826f04971e3} {Suffrage:Voter ID:server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44 Address:c07294b6-2cbd-a8d3-12a8-0a922c1b1c44} {Suffrage:Voter ID:server-fe1f5d13-defe-2d5b-4da2-fc5334f5316c Address:fe1f5d13-defe-2d5b-4da2-fc5334f5316c}]"
2026-05-25T23:22:37.916Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1e29b0b-6516-c874-6d32-4826f04971e3 Address:b1e29b0b-6516-c874-6d32-4826f04971e3} {Suffrage:Voter ID:server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44 Address:c07294b6-2cbd-a8d3-12a8-0a922c1b1c44} {Suffrage:Voter ID:server-fe1f5d13-defe-2d5b-4da2-fc5334f5316c Address:fe1f5d13-defe-2d5b-4da2-fc5334f5316c}]"
2026-05-25T23:22:37.916Z [INFO]  server-b1e29b0b-6516-c874-6d32-4826f04971e3: entering follower state: follower="Node at b1e29b0b-6516-c874-6d32-4826f04971e3 [Follower]" leader-address= leader-id=
2026-05-25T23:22:37.916Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: entering follower state: follower="Node at c07294b6-2cbd-a8d3-12a8-0a922c1b1c44 [Follower]" leader-address= leader-id=
2026-05-25T23:22:37.917Z [INFO]  server-fe1f5d13-defe-2d5b-4da2-fc5334f5316c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1e29b0b-6516-c874-6d32-4826f04971e3 Address:b1e29b0b-6516-c874-6d32-4826f04971e3} {Suffrage:Voter ID:server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44 Address:c07294b6-2cbd-a8d3-12a8-0a922c1b1c44} {Suffrage:Voter ID:server-fe1f5d13-defe-2d5b-4da2-fc5334f5316c Address:fe1f5d13-defe-2d5b-4da2-fc5334f5316c}]"
2026-05-25T23:22:37.917Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:37.917Z [INFO]  server-fe1f5d13-defe-2d5b-4da2-fc5334f5316c: entering follower state: follower="Node at fe1f5d13-defe-2d5b-4da2-fc5334f5316c [Follower]" leader-address= leader-id=
2026-05-25T23:22:38.472Z [WARN]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:38.472Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: entering candidate state: node="Node at c07294b6-2cbd-a8d3-12a8-0a922c1b1c44 [Candidate]" term=2
2026-05-25T23:22:38.473Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: election won: term=2 tally=2
2026-05-25T23:22:38.473Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: entering leader state: leader="Node at c07294b6-2cbd-a8d3-12a8-0a922c1b1c44 [Leader]"
2026-05-25T23:22:38.473Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: added peer, starting replication: peer=server-b1e29b0b-6516-c874-6d32-4826f04971e3
2026-05-25T23:22:38.473Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: added peer, starting replication: peer=server-fe1f5d13-defe-2d5b-4da2-fc5334f5316c
2026-05-25T23:22:38.473Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: pipelining replication: peer="{Voter server-b1e29b0b-6516-c874-6d32-4826f04971e3 b1e29b0b-6516-c874-6d32-4826f04971e3}"
2026-05-25T23:22:38.473Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: pipelining replication: peer="{Voter server-fe1f5d13-defe-2d5b-4da2-fc5334f5316c fe1f5d13-defe-2d5b-4da2-fc5334f5316c}"
2026-05-25T23:22:39.478Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:38.47342437 +0000 UTC m=+14.119871842 (1 nodes), 556.333311ms from start of poll, 556.679156ms from cluster start. Timeout at 2026-05-25 23:22:39.478430832 +0000 UTC m=+15.124878323, 1.005006481s after stability
2026-05-25T23:22:39.480Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: starting snapshot up to: index=12
2026-05-25T23:22:39.480Z [INFO]  creating new snapshot: path=/tmp/raft608375298/snapshots/2-12-1779751359480.tmp
2026-05-25T23:22:39.480Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: snapshot complete up to: index=12
2026-05-25T23:22:39.481Z [INFO]  creating new snapshot: path=/tmp/raft608375298/snapshots/2-23-1779751359481.tmp
2026-05-25T23:22:39.481Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: copied to local snapshot: bytes=71
2026-05-25T23:22:39.481Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: snapshot restore progress: id=2-23-1779751359481 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:39.481Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: restored user snapshot: index=23
2026-05-25T23:22:39.481Z [ERROR] server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: failed to get log: index=23 error="log not found"
2026-05-25T23:22:39.481Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: aborting pipeline replication: peer="{Voter server-b1e29b0b-6516-c874-6d32-4826f04971e3 b1e29b0b-6516-c874-6d32-4826f04971e3}"
2026-05-25T23:22:39.481Z [ERROR] server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: failed to get log: index=23 error="log not found"
2026-05-25T23:22:39.481Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: aborting pipeline replication: peer="{Voter server-fe1f5d13-defe-2d5b-4da2-fc5334f5316c fe1f5d13-defe-2d5b-4da2-fc5334f5316c}"
2026-05-25T23:22:39.488Z [ERROR] server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: failed to get log: index=23 error="log not found"
2026-05-25T23:22:39.488Z [INFO]  creating new snapshot: path=/tmp/raft810192906/snapshots/2-23-1779751359488.tmp
2026-05-25T23:22:39.489Z [INFO]  server-fe1f5d13-defe-2d5b-4da2-fc5334f5316c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:39.489Z [INFO]  server-fe1f5d13-defe-2d5b-4da2-fc5334f5316c: copied to local snapshot: bytes=71
2026-05-25T23:22:39.489Z [INFO]  server-fe1f5d13-defe-2d5b-4da2-fc5334f5316c: snapshot restore progress: id=2-23-1779751359488 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:39.489Z [INFO]  server-fe1f5d13-defe-2d5b-4da2-fc5334f5316c: Installed remote snapshot
2026-05-25T23:22:39.489Z [WARN]  server-fe1f5d13-defe-2d5b-4da2-fc5334f5316c: skipping application of old log: index=22
2026-05-25T23:22:39.489Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: pipelining replication: peer="{Voter server-fe1f5d13-defe-2d5b-4da2-fc5334f5316c fe1f5d13-defe-2d5b-4da2-fc5334f5316c}"
2026-05-25T23:22:39.489Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:22:39.490Z [ERROR] server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: failed to get log: index=23 error="log not found"
2026-05-25T23:22:39.491Z [INFO]  creating new snapshot: path=/tmp/raft2129017959/snapshots/2-23-1779751359491.tmp
2026-05-25T23:22:39.491Z [INFO]  server-b1e29b0b-6516-c874-6d32-4826f04971e3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:39.491Z [INFO]  server-b1e29b0b-6516-c874-6d32-4826f04971e3: copied to local snapshot: bytes=71
2026-05-25T23:22:39.492Z [INFO]  server-b1e29b0b-6516-c874-6d32-4826f04971e3: snapshot restore progress: id=2-23-1779751359491 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:39.492Z [INFO]  server-b1e29b0b-6516-c874-6d32-4826f04971e3: Installed remote snapshot
2026-05-25T23:22:39.492Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: pipelining replication: peer="{Voter server-b1e29b0b-6516-c874-6d32-4826f04971e3 b1e29b0b-6516-c874-6d32-4826f04971e3}"
2026-05-25T23:22:40.495Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:22:39.489853624 +0000 UTC m=+15.136301087 (2 nodes), 719ns from start of poll, 1.573108401s from cluster start. Timeout at 2026-05-25 23:22:40.494860269 +0000 UTC m=+16.141307760, 1.005006673s after stability
2026-05-25T23:22:40.507Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: aborting pipeline replication: peer="{Voter server-b1e29b0b-6516-c874-6d32-4826f04971e3 b1e29b0b-6516-c874-6d32-4826f04971e3}"
2026-05-25T23:22:40.507Z [INFO]  server-c07294b6-2cbd-a8d3-12a8-0a922c1b1c44: aborting pipeline replication: peer="{Voter server-fe1f5d13-defe-2d5b-4da2-fc5334f5316c fe1f5d13-defe-2d5b-4da2-fc5334f5316c}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false
2026-05-25T23:22:40.508Z [INFO]  server-dc29617b-75c1-02be-0eaf-831cecfe463d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc29617b-75c1-02be-0eaf-831cecfe463d Address:dc29617b-75c1-02be-0eaf-831cecfe463d} {Suffrage:Voter ID:server-a18b2036-9a84-2704-9d5c-74cfecfaa2c2 Address:a18b2036-9a84-2704-9d5c-74cfecfaa2c2} {Suffrage:Voter ID:server-ba3894ce-3611-539e-79d9-bd4608eaea09 Address:ba3894ce-3611-539e-79d9-bd4608eaea09}]"
2026-05-25T23:22:40.508Z [INFO]  server-dc29617b-75c1-02be-0eaf-831cecfe463d: entering follower state: follower="Node at dc29617b-75c1-02be-0eaf-831cecfe463d [Follower]" leader-address= leader-id=
2026-05-25T23:22:40.509Z [INFO]  server-a18b2036-9a84-2704-9d5c-74cfecfaa2c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc29617b-75c1-02be-0eaf-831cecfe463d Address:dc29617b-75c1-02be-0eaf-831cecfe463d} {Suffrage:Voter ID:server-a18b2036-9a84-2704-9d5c-74cfecfaa2c2 Address:a18b2036-9a84-2704-9d5c-74cfecfaa2c2} {Suffrage:Voter ID:server-ba3894ce-3611-539e-79d9-bd4608eaea09 Address:ba3894ce-3611-539e-79d9-bd4608eaea09}]"
2026-05-25T23:22:40.509Z [INFO]  server-a18b2036-9a84-2704-9d5c-74cfecfaa2c2: entering follower state: follower="Node at a18b2036-9a84-2704-9d5c-74cfecfaa2c2 [Follower]" leader-address= leader-id=
2026-05-25T23:22:40.509Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc29617b-75c1-02be-0eaf-831cecfe463d Address:dc29617b-75c1-02be-0eaf-831cecfe463d} {Suffrage:Voter ID:server-a18b2036-9a84-2704-9d5c-74cfecfaa2c2 Address:a18b2036-9a84-2704-9d5c-74cfecfaa2c2} {Suffrage:Voter ID:server-ba3894ce-3611-539e-79d9-bd4608eaea09 Address:ba3894ce-3611-539e-79d9-bd4608eaea09}]"
2026-05-25T23:22:40.509Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:40.509Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: entering follower state: follower="Node at ba3894ce-3611-539e-79d9-bd4608eaea09 [Follower]" leader-address= leader-id=
2026-05-25T23:22:41.016Z [WARN]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:41.016Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: entering candidate state: node="Node at ba3894ce-3611-539e-79d9-bd4608eaea09 [Candidate]" term=2
2026-05-25T23:22:41.016Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: election won: term=2 tally=2
2026-05-25T23:22:41.016Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: entering leader state: leader="Node at ba3894ce-3611-539e-79d9-bd4608eaea09 [Leader]"
2026-05-25T23:22:41.016Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: added peer, starting replication: peer=server-dc29617b-75c1-02be-0eaf-831cecfe463d
2026-05-25T23:22:41.016Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: added peer, starting replication: peer=server-a18b2036-9a84-2704-9d5c-74cfecfaa2c2
2026-05-25T23:22:41.016Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: pipelining replication: peer="{Voter server-a18b2036-9a84-2704-9d5c-74cfecfaa2c2 a18b2036-9a84-2704-9d5c-74cfecfaa2c2}"
2026-05-25T23:22:41.016Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: pipelining replication: peer="{Voter server-dc29617b-75c1-02be-0eaf-831cecfe463d dc29617b-75c1-02be-0eaf-831cecfe463d}"
2026-05-25T23:22:42.022Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:41.016335168 +0000 UTC m=+16.662782658 (1 nodes), 507.192622ms from start of poll, 507.494309ms from cluster start. Timeout at 2026-05-25 23:22:42.021339922 +0000 UTC m=+17.667787412, 1.005004754s after stability
2026-05-25T23:22:42.023Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: starting snapshot up to: index=12
2026-05-25T23:22:42.023Z [INFO]  creating new snapshot: path=/tmp/raft127235857/snapshots/2-12-1779751362023.tmp
2026-05-25T23:22:42.023Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: snapshot complete up to: index=12
2026-05-25T23:22:42.024Z [INFO]  creating new snapshot: path=/tmp/raft127235857/snapshots/2-23-1779751362024.tmp
2026-05-25T23:22:42.024Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: copied to local snapshot: bytes=71
2026-05-25T23:22:42.025Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: snapshot restore progress: id=2-23-1779751362024 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:42.025Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: removing all old logs from log store
2026-05-25T23:22:42.025Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: compacting logs: from=1 to=22
2026-05-25T23:22:42.025Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: restored user snapshot: index=23
2026-05-25T23:22:42.025Z [ERROR] server-ba3894ce-3611-539e-79d9-bd4608eaea09: failed to get log: index=22 error="log not found"
2026-05-25T23:22:42.025Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: aborting pipeline replication: peer="{Voter server-a18b2036-9a84-2704-9d5c-74cfecfaa2c2 a18b2036-9a84-2704-9d5c-74cfecfaa2c2}"
2026-05-25T23:22:42.025Z [ERROR] server-ba3894ce-3611-539e-79d9-bd4608eaea09: failed to get log: index=22 error="log not found"
2026-05-25T23:22:42.025Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: aborting pipeline replication: peer="{Voter server-dc29617b-75c1-02be-0eaf-831cecfe463d dc29617b-75c1-02be-0eaf-831cecfe463d}"
2026-05-25T23:22:42.034Z [ERROR] server-ba3894ce-3611-539e-79d9-bd4608eaea09: failed to get log: index=22 error="log not found"
2026-05-25T23:22:42.034Z [INFO]  creating new snapshot: path=/tmp/raft1455296492/snapshots/2-23-1779751362034.tmp
2026-05-25T23:22:42.034Z [INFO]  server-a18b2036-9a84-2704-9d5c-74cfecfaa2c2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:42.034Z [INFO]  server-a18b2036-9a84-2704-9d5c-74cfecfaa2c2: copied to local snapshot: bytes=71
2026-05-25T23:22:42.034Z [INFO]  server-a18b2036-9a84-2704-9d5c-74cfecfaa2c2: snapshot restore progress: id=2-23-1779751362034 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:42.035Z [INFO]  server-a18b2036-9a84-2704-9d5c-74cfecfaa2c2: removing all old logs from log store
2026-05-25T23:22:42.035Z [INFO]  server-a18b2036-9a84-2704-9d5c-74cfecfaa2c2: compacting logs: from=1 to=22
2026-05-25T23:22:42.035Z [INFO]  server-a18b2036-9a84-2704-9d5c-74cfecfaa2c2: Installed remote snapshot
2026-05-25T23:22:42.035Z [WARN]  server-a18b2036-9a84-2704-9d5c-74cfecfaa2c2: skipping application of old log: index=22
2026-05-25T23:22:42.035Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: pipelining replication: peer="{Voter server-a18b2036-9a84-2704-9d5c-74cfecfaa2c2 a18b2036-9a84-2704-9d5c-74cfecfaa2c2}"
2026-05-25T23:22:42.035Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:22:42.036Z [ERROR] server-ba3894ce-3611-539e-79d9-bd4608eaea09: failed to get log: index=22 error="log not found"
2026-05-25T23:22:42.036Z [INFO]  creating new snapshot: path=/tmp/raft1601524379/snapshots/2-23-1779751362036.tmp
2026-05-25T23:22:42.036Z [INFO]  server-dc29617b-75c1-02be-0eaf-831cecfe463d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:42.037Z [INFO]  server-dc29617b-75c1-02be-0eaf-831cecfe463d: copied to local snapshot: bytes=71
2026-05-25T23:22:42.037Z [INFO]  server-dc29617b-75c1-02be-0eaf-831cecfe463d: snapshot restore progress: id=2-23-1779751362036 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:42.037Z [INFO]  server-dc29617b-75c1-02be-0eaf-831cecfe463d: removing all old logs from log store
2026-05-25T23:22:42.037Z [INFO]  server-dc29617b-75c1-02be-0eaf-831cecfe463d: compacting logs: from=1 to=22
2026-05-25T23:22:42.037Z [INFO]  server-dc29617b-75c1-02be-0eaf-831cecfe463d: Installed remote snapshot
2026-05-25T23:22:42.037Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: pipelining replication: peer="{Voter server-dc29617b-75c1-02be-0eaf-831cecfe463d dc29617b-75c1-02be-0eaf-831cecfe463d}"
2026-05-25T23:22:43.041Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:22:42.035170412 +0000 UTC m=+17.681617866 (2 nodes), 439ns from start of poll, 1.526329517s from cluster start. Timeout at 2026-05-25 23:22:43.040180433 +0000 UTC m=+18.686627923, 1.005010057s after stability
2026-05-25T23:22:43.053Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: aborting pipeline replication: peer="{Voter server-a18b2036-9a84-2704-9d5c-74cfecfaa2c2 a18b2036-9a84-2704-9d5c-74cfecfaa2c2}"
2026-05-25T23:22:43.053Z [INFO]  server-ba3894ce-3611-539e-79d9-bd4608eaea09: aborting pipeline replication: peer="{Voter server-dc29617b-75c1-02be-0eaf-831cecfe463d dc29617b-75c1-02be-0eaf-831cecfe463d}"
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true
2026-05-25T23:22:43.055Z [INFO]  server-70577eea-c439-8aa5-f01a-b04837250c4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70577eea-c439-8aa5-f01a-b04837250c4b Address:70577eea-c439-8aa5-f01a-b04837250c4b} {Suffrage:Voter ID:server-86b2273e-27f6-bb56-3413-7ac063107ef8 Address:86b2273e-27f6-bb56-3413-7ac063107ef8} {Suffrage:Voter ID:server-a9763b1a-78b1-9544-baa4-2748296c8af2 Address:a9763b1a-78b1-9544-baa4-2748296c8af2}]"
2026-05-25T23:22:43.055Z [INFO]  server-70577eea-c439-8aa5-f01a-b04837250c4b: entering follower state: follower="Node at 70577eea-c439-8aa5-f01a-b04837250c4b [Follower]" leader-address= leader-id=
2026-05-25T23:22:43.055Z [INFO]  server-86b2273e-27f6-bb56-3413-7ac063107ef8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70577eea-c439-8aa5-f01a-b04837250c4b Address:70577eea-c439-8aa5-f01a-b04837250c4b} {Suffrage:Voter ID:server-86b2273e-27f6-bb56-3413-7ac063107ef8 Address:86b2273e-27f6-bb56-3413-7ac063107ef8} {Suffrage:Voter ID:server-a9763b1a-78b1-9544-baa4-2748296c8af2 Address:a9763b1a-78b1-9544-baa4-2748296c8af2}]"
2026-05-25T23:22:43.055Z [INFO]  server-a9763b1a-78b1-9544-baa4-2748296c8af2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70577eea-c439-8aa5-f01a-b04837250c4b Address:70577eea-c439-8aa5-f01a-b04837250c4b} {Suffrage:Voter ID:server-86b2273e-27f6-bb56-3413-7ac063107ef8 Address:86b2273e-27f6-bb56-3413-7ac063107ef8} {Suffrage:Voter ID:server-a9763b1a-78b1-9544-baa4-2748296c8af2 Address:a9763b1a-78b1-9544-baa4-2748296c8af2}]"
2026-05-25T23:22:43.055Z [INFO]  server-86b2273e-27f6-bb56-3413-7ac063107ef8: entering follower state: follower="Node at 86b2273e-27f6-bb56-3413-7ac063107ef8 [Follower]" leader-address= leader-id=
2026-05-25T23:22:43.055Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:43.055Z [INFO]  server-a9763b1a-78b1-9544-baa4-2748296c8af2: entering follower state: follower="Node at a9763b1a-78b1-9544-baa4-2748296c8af2 [Follower]" leader-address= leader-id=
2026-05-25T23:22:43.563Z [WARN]  server-70577eea-c439-8aa5-f01a-b04837250c4b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:43.563Z [INFO]  server-70577eea-c439-8aa5-f01a-b04837250c4b: entering candidate state: node="Node at 70577eea-c439-8aa5-f01a-b04837250c4b [Candidate]" term=2
2026-05-25T23:22:43.563Z [INFO]  server-70577eea-c439-8aa5-f01a-b04837250c4b: election won: term=2 tally=2
2026-05-25T23:22:43.563Z [INFO]  server-70577eea-c439-8aa5-f01a-b04837250c4b: entering leader state: leader="Node at 70577eea-c439-8aa5-f01a-b04837250c4b [Leader]"
2026-05-25T23:22:43.563Z [INFO]  server-70577eea-c439-8aa5-f01a-b04837250c4b: added peer, starting replication: peer=server-86b2273e-27f6-bb56-3413-7ac063107ef8
2026-05-25T23:22:43.563Z [INFO]  server-70577eea-c439-8aa5-f01a-b04837250c4b: added peer, starting replication: peer=server-a9763b1a-78b1-9544-baa4-2748296c8af2
2026-05-25T23:22:43.563Z [INFO]  server-70577eea-c439-8aa5-f01a-b04837250c4b: pipelining replication: peer="{Voter server-86b2273e-27f6-bb56-3413-7ac063107ef8 86b2273e-27f6-bb56-3413-7ac063107ef8}"
2026-05-25T23:22:43.563Z [INFO]  server-70577eea-c439-8aa5-f01a-b04837250c4b: pipelining replication: peer="{Voter server-a9763b1a-78b1-9544-baa4-2748296c8af2 a9763b1a-78b1-9544-baa4-2748296c8af2}"
2026-05-25T23:22:44.568Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:43.563279609 +0000 UTC m=+19.209727076 (1 nodes), 507.991691ms from start of poll, 508.301849ms from cluster start. Timeout at 2026-05-25 23:22:44.568280997 +0000 UTC m=+20.214728451, 1.005001375s after stability
2026-05-25T23:22:44.570Z [INFO]  server-70577eea-c439-8aa5-f01a-b04837250c4b: starting snapshot up to: index=12
2026-05-25T23:22:44.570Z [INFO]  creating new snapshot: path=/tmp/raft2895466833/snapshots/2-12-1779751364570.tmp
2026-05-25T23:22:44.582Z [INFO]  server-70577eea-c439-8aa5-f01a-b04837250c4b: snapshot complete up to: index=12
2026-05-25T23:22:44.583Z [INFO]  server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb Address:b2b88b2b-e5dc-316b-2db7-975d03d33dfb} {Suffrage:Voter ID:server-918e04a4-d2b8-6285-0572-1f11870065de Address:918e04a4-d2b8-6285-0572-1f11870065de} {Suffrage:Voter ID:server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058 Address:0bc7c1d4-cea2-25e5-ba38-79cde9eb4058}]"
2026-05-25T23:22:44.583Z [INFO]  server-918e04a4-d2b8-6285-0572-1f11870065de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb Address:b2b88b2b-e5dc-316b-2db7-975d03d33dfb} {Suffrage:Voter ID:server-918e04a4-d2b8-6285-0572-1f11870065de Address:918e04a4-d2b8-6285-0572-1f11870065de} {Suffrage:Voter ID:server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058 Address:0bc7c1d4-cea2-25e5-ba38-79cde9eb4058}]"
2026-05-25T23:22:44.583Z [INFO]  server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb: entering follower state: follower="Node at b2b88b2b-e5dc-316b-2db7-975d03d33dfb [Follower]" leader-address= leader-id=
2026-05-25T23:22:44.583Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb Address:b2b88b2b-e5dc-316b-2db7-975d03d33dfb} {Suffrage:Voter ID:server-918e04a4-d2b8-6285-0572-1f11870065de Address:918e04a4-d2b8-6285-0572-1f11870065de} {Suffrage:Voter ID:server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058 Address:0bc7c1d4-cea2-25e5-ba38-79cde9eb4058}]"
2026-05-25T23:22:44.583Z [INFO]  server-918e04a4-d2b8-6285-0572-1f11870065de: entering follower state: follower="Node at 918e04a4-d2b8-6285-0572-1f11870065de [Follower]" leader-address= leader-id=
2026-05-25T23:22:44.583Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:44.583Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: entering follower state: follower="Node at 0bc7c1d4-cea2-25e5-ba38-79cde9eb4058 [Follower]" leader-address= leader-id=
2026-05-25T23:22:45.299Z [WARN]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:45.299Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: entering candidate state: node="Node at 0bc7c1d4-cea2-25e5-ba38-79cde9eb4058 [Candidate]" term=2
2026-05-25T23:22:45.299Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: election won: term=2 tally=2
2026-05-25T23:22:45.299Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: entering leader state: leader="Node at 0bc7c1d4-cea2-25e5-ba38-79cde9eb4058 [Leader]"
2026-05-25T23:22:45.299Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: added peer, starting replication: peer=server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb
2026-05-25T23:22:45.299Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: added peer, starting replication: peer=server-918e04a4-d2b8-6285-0572-1f11870065de
2026-05-25T23:22:45.299Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: pipelining replication: peer="{Voter server-918e04a4-d2b8-6285-0572-1f11870065de 918e04a4-d2b8-6285-0572-1f11870065de}"
2026-05-25T23:22:45.299Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: pipelining replication: peer="{Voter server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb b2b88b2b-e5dc-316b-2db7-975d03d33dfb}"
2026-05-25T23:22:46.305Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:45.299792353 +0000 UTC m=+20.946239820 (1 nodes), 716.315253ms from start of poll, 716.675055ms from cluster start. Timeout at 2026-05-25 23:22:46.304797541 +0000 UTC m=+21.951245031, 1.005005211s after stability
2026-05-25T23:22:46.305Z [INFO]  creating new snapshot: path=/tmp/raft849096487/snapshots/2-13-1779751366305.tmp
2026-05-25T23:22:46.306Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: copied to local snapshot: bytes=71
2026-05-25T23:22:46.306Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: snapshot restore progress: id=2-13-1779751366305 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:46.306Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: restored user snapshot: index=13
2026-05-25T23:22:46.306Z [ERROR] server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: failed to get log: index=3 error="log not found"
2026-05-25T23:22:46.306Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: aborting pipeline replication: peer="{Voter server-918e04a4-d2b8-6285-0572-1f11870065de 918e04a4-d2b8-6285-0572-1f11870065de}"
2026-05-25T23:22:46.306Z [ERROR] server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: failed to get log: index=3 error="log not found"
2026-05-25T23:22:46.306Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: aborting pipeline replication: peer="{Voter server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb b2b88b2b-e5dc-316b-2db7-975d03d33dfb}"
2026-05-25T23:22:46.311Z [ERROR] server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: failed to get log: index=3 error="log not found"
2026-05-25T23:22:46.312Z [INFO]  creating new snapshot: path=/tmp/raft2353670227/snapshots/2-13-1779751366312.tmp
2026-05-25T23:22:46.312Z [INFO]  server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:46.312Z [INFO]  server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb: copied to local snapshot: bytes=71
2026-05-25T23:22:46.312Z [INFO]  server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb: snapshot restore progress: id=2-13-1779751366312 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:46.312Z [INFO]  server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb: Installed remote snapshot
2026-05-25T23:22:46.312Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: pipelining replication: peer="{Voter server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb b2b88b2b-e5dc-316b-2db7-975d03d33dfb}"
2026-05-25T23:22:46.312Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:22:46.317Z [ERROR] server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: failed to get log: index=3 error="log not found"
2026-05-25T23:22:46.317Z [INFO]  creating new snapshot: path=/tmp/raft1281778582/snapshots/2-13-1779751366317.tmp
2026-05-25T23:22:46.317Z [INFO]  server-918e04a4-d2b8-6285-0572-1f11870065de: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:46.317Z [INFO]  server-918e04a4-d2b8-6285-0572-1f11870065de: copied to local snapshot: bytes=71
2026-05-25T23:22:46.318Z [INFO]  server-918e04a4-d2b8-6285-0572-1f11870065de: snapshot restore progress: id=2-13-1779751366317 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:46.318Z [INFO]  server-918e04a4-d2b8-6285-0572-1f11870065de: Installed remote snapshot
2026-05-25T23:22:46.318Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: pipelining replication: peer="{Voter server-918e04a4-d2b8-6285-0572-1f11870065de 918e04a4-d2b8-6285-0572-1f11870065de}"
2026-05-25T23:22:47.318Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:22:46.312942682 +0000 UTC m=+21.959390143 (2 nodes), 905ns from start of poll, 1.729825378s from cluster start. Timeout at 2026-05-25 23:22:47.317950152 +0000 UTC m=+22.964397605, 1.005007462s after stability
2026-05-25T23:22:47.330Z [INFO]  server-70577eea-c439-8aa5-f01a-b04837250c4b: aborting pipeline replication: peer="{Voter server-a9763b1a-78b1-9544-baa4-2748296c8af2 a9763b1a-78b1-9544-baa4-2748296c8af2}"
2026-05-25T23:22:47.330Z [INFO]  server-70577eea-c439-8aa5-f01a-b04837250c4b: aborting pipeline replication: peer="{Voter server-86b2273e-27f6-bb56-3413-7ac063107ef8 86b2273e-27f6-bb56-3413-7ac063107ef8}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true
2026-05-25T23:22:47.331Z [INFO]  server-b2afda6f-20c5-100f-9faa-c00cacac1d10: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2afda6f-20c5-100f-9faa-c00cacac1d10 Address:b2afda6f-20c5-100f-9faa-c00cacac1d10} {Suffrage:Voter ID:server-821f2225-94f9-778b-ac3f-eb709c8f2f65 Address:821f2225-94f9-778b-ac3f-eb709c8f2f65} {Suffrage:Voter ID:server-ce107e5a-04a4-61dc-14b5-b34ecaeea19a Address:ce107e5a-04a4-61dc-14b5-b34ecaeea19a}]"
2026-05-25T23:22:47.331Z [INFO]  server-b2afda6f-20c5-100f-9faa-c00cacac1d10: entering follower state: follower="Node at b2afda6f-20c5-100f-9faa-c00cacac1d10 [Follower]" leader-address= leader-id=
2026-05-25T23:22:47.331Z [INFO]  server-821f2225-94f9-778b-ac3f-eb709c8f2f65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2afda6f-20c5-100f-9faa-c00cacac1d10 Address:b2afda6f-20c5-100f-9faa-c00cacac1d10} {Suffrage:Voter ID:server-821f2225-94f9-778b-ac3f-eb709c8f2f65 Address:821f2225-94f9-778b-ac3f-eb709c8f2f65} {Suffrage:Voter ID:server-ce107e5a-04a4-61dc-14b5-b34ecaeea19a Address:ce107e5a-04a4-61dc-14b5-b34ecaeea19a}]"
2026-05-25T23:22:47.331Z [INFO]  server-821f2225-94f9-778b-ac3f-eb709c8f2f65: entering follower state: follower="Node at 821f2225-94f9-778b-ac3f-eb709c8f2f65 [Follower]" leader-address= leader-id=
2026-05-25T23:22:47.331Z [INFO]  server-ce107e5a-04a4-61dc-14b5-b34ecaeea19a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2afda6f-20c5-100f-9faa-c00cacac1d10 Address:b2afda6f-20c5-100f-9faa-c00cacac1d10} {Suffrage:Voter ID:server-821f2225-94f9-778b-ac3f-eb709c8f2f65 Address:821f2225-94f9-778b-ac3f-eb709c8f2f65} {Suffrage:Voter ID:server-ce107e5a-04a4-61dc-14b5-b34ecaeea19a Address:ce107e5a-04a4-61dc-14b5-b34ecaeea19a}]"
2026-05-25T23:22:47.331Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:47.331Z [INFO]  server-ce107e5a-04a4-61dc-14b5-b34ecaeea19a: entering follower state: follower="Node at ce107e5a-04a4-61dc-14b5-b34ecaeea19a [Follower]" leader-address= leader-id=
2026-05-25T23:22:47.992Z [WARN]  server-ce107e5a-04a4-61dc-14b5-b34ecaeea19a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:47.992Z [INFO]  server-ce107e5a-04a4-61dc-14b5-b34ecaeea19a: entering candidate state: node="Node at ce107e5a-04a4-61dc-14b5-b34ecaeea19a [Candidate]" term=2
2026-05-25T23:22:47.992Z [INFO]  server-ce107e5a-04a4-61dc-14b5-b34ecaeea19a: election won: term=2 tally=2
2026-05-25T23:22:47.992Z [INFO]  server-ce107e5a-04a4-61dc-14b5-b34ecaeea19a: entering leader state: leader="Node at ce107e5a-04a4-61dc-14b5-b34ecaeea19a [Leader]"
2026-05-25T23:22:47.992Z [INFO]  server-ce107e5a-04a4-61dc-14b5-b34ecaeea19a: added peer, starting replication: peer=server-b2afda6f-20c5-100f-9faa-c00cacac1d10
2026-05-25T23:22:47.992Z [INFO]  server-ce107e5a-04a4-61dc-14b5-b34ecaeea19a: added peer, starting replication: peer=server-821f2225-94f9-778b-ac3f-eb709c8f2f65
2026-05-25T23:22:47.993Z [INFO]  server-ce107e5a-04a4-61dc-14b5-b34ecaeea19a: pipelining replication: peer="{Voter server-b2afda6f-20c5-100f-9faa-c00cacac1d10 b2afda6f-20c5-100f-9faa-c00cacac1d10}"
2026-05-25T23:22:47.993Z [INFO]  server-ce107e5a-04a4-61dc-14b5-b34ecaeea19a: pipelining replication: peer="{Voter server-821f2225-94f9-778b-ac3f-eb709c8f2f65 821f2225-94f9-778b-ac3f-eb709c8f2f65}"
2026-05-25T23:22:48.998Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:47.993359819 +0000 UTC m=+23.639807269 (1 nodes), 661.533642ms from start of poll, 661.88123ms from cluster start. Timeout at 2026-05-25 23:22:48.998360756 +0000 UTC m=+24.644808211, 1.005000942s after stability
2026-05-25T23:22:48.999Z [INFO]  server-ce107e5a-04a4-61dc-14b5-b34ecaeea19a: starting snapshot up to: index=12
2026-05-25T23:22:48.999Z [INFO]  creating new snapshot: path=/tmp/raft654614049/snapshots/2-12-1779751368999.tmp
2026-05-25T23:22:49.164Z [INFO]  server-ce107e5a-04a4-61dc-14b5-b34ecaeea19a: snapshot complete up to: index=12
2026-05-25T23:22:49.164Z [INFO]  server-1ea7c9cb-c351-9612-c976-e429888b54a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ea7c9cb-c351-9612-c976-e429888b54a7 Address:1ea7c9cb-c351-9612-c976-e429888b54a7} {Suffrage:Voter ID:server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940 Address:8cf9d984-e9cf-b9ad-bd1c-e4135e071940} {Suffrage:Voter ID:server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710 Address:fcb5b37a-b3b5-51e6-b68a-69e91d5df710}]"
2026-05-25T23:22:49.165Z [INFO]  server-1ea7c9cb-c351-9612-c976-e429888b54a7: entering follower state: follower="Node at 1ea7c9cb-c351-9612-c976-e429888b54a7 [Follower]" leader-address= leader-id=
2026-05-25T23:22:49.165Z [INFO]  server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ea7c9cb-c351-9612-c976-e429888b54a7 Address:1ea7c9cb-c351-9612-c976-e429888b54a7} {Suffrage:Voter ID:server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940 Address:8cf9d984-e9cf-b9ad-bd1c-e4135e071940} {Suffrage:Voter ID:server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710 Address:fcb5b37a-b3b5-51e6-b68a-69e91d5df710}]"
2026-05-25T23:22:49.165Z [INFO]  server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940: entering follower state: follower="Node at 8cf9d984-e9cf-b9ad-bd1c-e4135e071940 [Follower]" leader-address= leader-id=
2026-05-25T23:22:49.165Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ea7c9cb-c351-9612-c976-e429888b54a7 Address:1ea7c9cb-c351-9612-c976-e429888b54a7} {Suffrage:Voter ID:server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940 Address:8cf9d984-e9cf-b9ad-bd1c-e4135e071940} {Suffrage:Voter ID:server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710 Address:fcb5b37a-b3b5-51e6-b68a-69e91d5df710}]"
2026-05-25T23:22:49.165Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:49.165Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: entering follower state: follower="Node at fcb5b37a-b3b5-51e6-b68a-69e91d5df710 [Follower]" leader-address= leader-id=
2026-05-25T23:22:49.799Z [WARN]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:49.799Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: entering candidate state: node="Node at fcb5b37a-b3b5-51e6-b68a-69e91d5df710 [Candidate]" term=2
2026-05-25T23:22:49.800Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: election won: term=2 tally=2
2026-05-25T23:22:49.800Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: entering leader state: leader="Node at fcb5b37a-b3b5-51e6-b68a-69e91d5df710 [Leader]"
2026-05-25T23:22:49.800Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: added peer, starting replication: peer=server-1ea7c9cb-c351-9612-c976-e429888b54a7
2026-05-25T23:22:49.800Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: added peer, starting replication: peer=server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940
2026-05-25T23:22:49.800Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: pipelining replication: peer="{Voter server-1ea7c9cb-c351-9612-c976-e429888b54a7 1ea7c9cb-c351-9612-c976-e429888b54a7}"
2026-05-25T23:22:49.800Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: pipelining replication: peer="{Voter server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940 8cf9d984-e9cf-b9ad-bd1c-e4135e071940}"
2026-05-25T23:22:50.806Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:49.800086951 +0000 UTC m=+25.446534653 (1 nodes), 634.694387ms from start of poll, 635.248463ms from cluster start. Timeout at 2026-05-25 23:22:50.805154256 +0000 UTC m=+26.451601710, 1.005067057s after stability
2026-05-25T23:22:50.806Z [INFO]  creating new snapshot: path=/tmp/raft547752820/snapshots/2-13-1779751370806.tmp
2026-05-25T23:22:50.806Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: copied to local snapshot: bytes=71
2026-05-25T23:22:50.807Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: snapshot restore progress: id=2-13-1779751370806 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:50.807Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: removing all old logs from log store
2026-05-25T23:22:50.807Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: compacting logs: from=1 to=2
2026-05-25T23:22:50.807Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: restored user snapshot: index=13
2026-05-25T23:22:50.807Z [ERROR] server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: failed to get log: index=2 error="log not found"
2026-05-25T23:22:50.807Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: aborting pipeline replication: peer="{Voter server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940 8cf9d984-e9cf-b9ad-bd1c-e4135e071940}"
2026-05-25T23:22:50.807Z [ERROR] server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: failed to get log: index=2 error="log not found"
2026-05-25T23:22:50.807Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: aborting pipeline replication: peer="{Voter server-1ea7c9cb-c351-9612-c976-e429888b54a7 1ea7c9cb-c351-9612-c976-e429888b54a7}"
2026-05-25T23:22:50.813Z [ERROR] server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: failed to get log: index=2 error="log not found"
2026-05-25T23:22:50.813Z [ERROR] server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: failed to get log: index=2 error="log not found"
2026-05-25T23:22:50.813Z [INFO]  creating new snapshot: path=/tmp/raft1225518370/snapshots/2-13-1779751370813.tmp
2026-05-25T23:22:50.813Z [INFO]  creating new snapshot: path=/tmp/raft3865471302/snapshots/2-13-1779751370813.tmp
2026-05-25T23:22:50.814Z [INFO]  server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:50.814Z [INFO]  server-1ea7c9cb-c351-9612-c976-e429888b54a7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:50.814Z [INFO]  server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940: copied to local snapshot: bytes=71
2026-05-25T23:22:50.814Z [INFO]  server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940: snapshot restore progress: id=2-13-1779751370813 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:50.814Z [INFO]  server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940: removing all old logs from log store
2026-05-25T23:22:50.814Z [INFO]  server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940: compacting logs: from=1 to=2
2026-05-25T23:22:50.814Z [INFO]  server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940: Installed remote snapshot
2026-05-25T23:22:50.814Z [INFO]  server-1ea7c9cb-c351-9612-c976-e429888b54a7: copied to local snapshot: bytes=71
2026-05-25T23:22:50.814Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: pipelining replication: peer="{Voter server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940 8cf9d984-e9cf-b9ad-bd1c-e4135e071940}"
2026-05-25T23:22:50.814Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:22:50.814Z [INFO]  server-1ea7c9cb-c351-9612-c976-e429888b54a7: snapshot restore progress: id=2-13-1779751370813 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:50.814Z [INFO]  server-1ea7c9cb-c351-9612-c976-e429888b54a7: removing all old logs from log store
2026-05-25T23:22:50.814Z [INFO]  server-1ea7c9cb-c351-9612-c976-e429888b54a7: compacting logs: from=1 to=2
2026-05-25T23:22:50.814Z [INFO]  server-1ea7c9cb-c351-9612-c976-e429888b54a7: Installed remote snapshot
2026-05-25T23:22:50.814Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: pipelining replication: peer="{Voter server-1ea7c9cb-c351-9612-c976-e429888b54a7 1ea7c9cb-c351-9612-c976-e429888b54a7}"
2026-05-25T23:22:51.819Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:22:50.814684577 +0000 UTC m=+26.461132049 (2 nodes), 978ns from start of poll, 1.649845859s from cluster start. Timeout at 2026-05-25 23:22:51.819691577 +0000 UTC m=+27.466139064, 1.005007015s after stability
2026-05-25T23:22:51.832Z [INFO]  server-ce107e5a-04a4-61dc-14b5-b34ecaeea19a: aborting pipeline replication: peer="{Voter server-821f2225-94f9-778b-ac3f-eb709c8f2f65 821f2225-94f9-778b-ac3f-eb709c8f2f65}"
2026-05-25T23:22:51.832Z [INFO]  server-ce107e5a-04a4-61dc-14b5-b34ecaeea19a: aborting pipeline replication: peer="{Voter server-b2afda6f-20c5-100f-9faa-c00cacac1d10 b2afda6f-20c5-100f-9faa-c00cacac1d10}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false
2026-05-25T23:22:51.837Z [INFO]  server-2de34fd7-5523-b8a4-3a08-196b5b2776b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2de34fd7-5523-b8a4-3a08-196b5b2776b4 Address:2de34fd7-5523-b8a4-3a08-196b5b2776b4} {Suffrage:Voter ID:server-b8d2b007-ad0f-8357-e0aa-38f42b4ed177 Address:b8d2b007-ad0f-8357-e0aa-38f42b4ed177} {Suffrage:Voter ID:server-59b35d80-1ced-4f88-353b-5c17e37bf0d6 Address:59b35d80-1ced-4f88-353b-5c17e37bf0d6}]"
2026-05-25T23:22:51.837Z [INFO]  server-2de34fd7-5523-b8a4-3a08-196b5b2776b4: entering follower state: follower="Node at 2de34fd7-5523-b8a4-3a08-196b5b2776b4 [Follower]" leader-address= leader-id=
2026-05-25T23:22:51.837Z [INFO]  server-b8d2b007-ad0f-8357-e0aa-38f42b4ed177: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2de34fd7-5523-b8a4-3a08-196b5b2776b4 Address:2de34fd7-5523-b8a4-3a08-196b5b2776b4} {Suffrage:Voter ID:server-b8d2b007-ad0f-8357-e0aa-38f42b4ed177 Address:b8d2b007-ad0f-8357-e0aa-38f42b4ed177} {Suffrage:Voter ID:server-59b35d80-1ced-4f88-353b-5c17e37bf0d6 Address:59b35d80-1ced-4f88-353b-5c17e37bf0d6}]"
2026-05-25T23:22:51.837Z [INFO]  server-b8d2b007-ad0f-8357-e0aa-38f42b4ed177: entering follower state: follower="Node at b8d2b007-ad0f-8357-e0aa-38f42b4ed177 [Follower]" leader-address= leader-id=
2026-05-25T23:22:51.837Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2de34fd7-5523-b8a4-3a08-196b5b2776b4 Address:2de34fd7-5523-b8a4-3a08-196b5b2776b4} {Suffrage:Voter ID:server-b8d2b007-ad0f-8357-e0aa-38f42b4ed177 Address:b8d2b007-ad0f-8357-e0aa-38f42b4ed177} {Suffrage:Voter ID:server-59b35d80-1ced-4f88-353b-5c17e37bf0d6 Address:59b35d80-1ced-4f88-353b-5c17e37bf0d6}]"
2026-05-25T23:22:51.837Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:51.837Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: entering follower state: follower="Node at 59b35d80-1ced-4f88-353b-5c17e37bf0d6 [Follower]" leader-address= leader-id=
2026-05-25T23:22:52.498Z [WARN]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:52.498Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: entering candidate state: node="Node at 59b35d80-1ced-4f88-353b-5c17e37bf0d6 [Candidate]" term=2
2026-05-25T23:22:52.498Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: election won: term=2 tally=2
2026-05-25T23:22:52.498Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: entering leader state: leader="Node at 59b35d80-1ced-4f88-353b-5c17e37bf0d6 [Leader]"
2026-05-25T23:22:52.498Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: added peer, starting replication: peer=server-2de34fd7-5523-b8a4-3a08-196b5b2776b4
2026-05-25T23:22:52.498Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: added peer, starting replication: peer=server-b8d2b007-ad0f-8357-e0aa-38f42b4ed177
2026-05-25T23:22:52.498Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: pipelining replication: peer="{Voter server-b8d2b007-ad0f-8357-e0aa-38f42b4ed177 b8d2b007-ad0f-8357-e0aa-38f42b4ed177}"
2026-05-25T23:22:52.498Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: pipelining replication: peer="{Voter server-2de34fd7-5523-b8a4-3a08-196b5b2776b4 2de34fd7-5523-b8a4-3a08-196b5b2776b4}"
2026-05-25T23:22:53.504Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:52.498797634 +0000 UTC m=+28.145245106 (1 nodes), 661.488131ms from start of poll, 661.787361ms from cluster start. Timeout at 2026-05-25 23:22:53.50380025 +0000 UTC m=+29.150247705, 1.005002599s after stability
2026-05-25T23:22:53.504Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: starting snapshot up to: index=12
2026-05-25T23:22:53.504Z [INFO]  creating new snapshot: path=/tmp/raft1166578558/snapshots/2-12-1779751373504.tmp
2026-05-25T23:22:53.505Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: snapshot complete up to: index=12
2026-05-25T23:22:53.505Z [INFO]  creating new snapshot: path=/tmp/raft1166578558/snapshots/2-23-1779751373505.tmp
2026-05-25T23:22:53.506Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: copied to local snapshot: bytes=71
2026-05-25T23:22:53.506Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: snapshot restore progress: id=2-23-1779751373505 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:53.506Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: restored user snapshot: index=23
2026-05-25T23:22:53.506Z [ERROR] server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: failed to get log: index=23 error="log not found"
2026-05-25T23:22:53.506Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: aborting pipeline replication: peer="{Voter server-b8d2b007-ad0f-8357-e0aa-38f42b4ed177 b8d2b007-ad0f-8357-e0aa-38f42b4ed177}"
2026-05-25T23:22:53.506Z [ERROR] server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: failed to get log: index=23 error="log not found"
2026-05-25T23:22:53.506Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: aborting pipeline replication: peer="{Voter server-2de34fd7-5523-b8a4-3a08-196b5b2776b4 2de34fd7-5523-b8a4-3a08-196b5b2776b4}"
2026-05-25T23:22:53.515Z [ERROR] server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: failed to get log: index=23 error="log not found"
2026-05-25T23:22:53.516Z [ERROR] server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: failed to get log: index=23 error="log not found"
2026-05-25T23:22:53.516Z [INFO]  creating new snapshot: path=/tmp/raft3836083447/snapshots/2-23-1779751373516.tmp
2026-05-25T23:22:53.516Z [INFO]  server-b8d2b007-ad0f-8357-e0aa-38f42b4ed177: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:53.516Z [INFO]  creating new snapshot: path=/tmp/raft4002872606/snapshots/2-23-1779751373516.tmp
2026-05-25T23:22:53.516Z [INFO]  server-b8d2b007-ad0f-8357-e0aa-38f42b4ed177: copied to local snapshot: bytes=71
2026-05-25T23:22:53.516Z [INFO]  server-2de34fd7-5523-b8a4-3a08-196b5b2776b4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:53.516Z [INFO]  server-b8d2b007-ad0f-8357-e0aa-38f42b4ed177: snapshot restore progress: id=2-23-1779751373516 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:53.516Z [INFO]  server-b8d2b007-ad0f-8357-e0aa-38f42b4ed177: Installed remote snapshot
2026-05-25T23:22:53.516Z [WARN]  server-b8d2b007-ad0f-8357-e0aa-38f42b4ed177: skipping application of old log: index=22
2026-05-25T23:22:53.516Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: pipelining replication: peer="{Voter server-b8d2b007-ad0f-8357-e0aa-38f42b4ed177 b8d2b007-ad0f-8357-e0aa-38f42b4ed177}"
2026-05-25T23:22:53.516Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:22:53.517Z [INFO]  server-2de34fd7-5523-b8a4-3a08-196b5b2776b4: copied to local snapshot: bytes=71
2026-05-25T23:22:53.517Z [INFO]  server-2de34fd7-5523-b8a4-3a08-196b5b2776b4: snapshot restore progress: id=2-23-1779751373516 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:53.517Z [INFO]  server-2de34fd7-5523-b8a4-3a08-196b5b2776b4: Installed remote snapshot
2026-05-25T23:22:53.517Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: pipelining replication: peer="{Voter server-2de34fd7-5523-b8a4-3a08-196b5b2776b4 2de34fd7-5523-b8a4-3a08-196b5b2776b4}"
2026-05-25T23:22:54.522Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:22:53.51691776 +0000 UTC m=+29.163365230 (2 nodes), 784ns from start of poll, 1.679907485s from cluster start. Timeout at 2026-05-25 23:22:54.521926157 +0000 UTC m=+30.168373609, 1.005008379s after stability
2026-05-25T23:22:54.535Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: aborting pipeline replication: peer="{Voter server-b8d2b007-ad0f-8357-e0aa-38f42b4ed177 b8d2b007-ad0f-8357-e0aa-38f42b4ed177}"
2026-05-25T23:22:54.535Z [INFO]  server-59b35d80-1ced-4f88-353b-5c17e37bf0d6: aborting pipeline replication: peer="{Voter server-2de34fd7-5523-b8a4-3a08-196b5b2776b4 2de34fd7-5523-b8a4-3a08-196b5b2776b4}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false
2026-05-25T23:22:54.537Z [INFO]  server-3427fc90-d99f-88be-d147-f80d430d05d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3427fc90-d99f-88be-d147-f80d430d05d8 Address:3427fc90-d99f-88be-d147-f80d430d05d8} {Suffrage:Voter ID:server-cd252465-3cae-db8d-8b9d-d69e2d241ffc Address:cd252465-3cae-db8d-8b9d-d69e2d241ffc} {Suffrage:Voter ID:server-31afc209-49de-67c8-ee30-cfcaa51597fd Address:31afc209-49de-67c8-ee30-cfcaa51597fd}]"
2026-05-25T23:22:54.537Z [INFO]  server-3427fc90-d99f-88be-d147-f80d430d05d8: entering follower state: follower="Node at 3427fc90-d99f-88be-d147-f80d430d05d8 [Follower]" leader-address= leader-id=
2026-05-25T23:22:54.537Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3427fc90-d99f-88be-d147-f80d430d05d8 Address:3427fc90-d99f-88be-d147-f80d430d05d8} {Suffrage:Voter ID:server-cd252465-3cae-db8d-8b9d-d69e2d241ffc Address:cd252465-3cae-db8d-8b9d-d69e2d241ffc} {Suffrage:Voter ID:server-31afc209-49de-67c8-ee30-cfcaa51597fd Address:31afc209-49de-67c8-ee30-cfcaa51597fd}]"
2026-05-25T23:22:54.537Z [INFO]  server-31afc209-49de-67c8-ee30-cfcaa51597fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3427fc90-d99f-88be-d147-f80d430d05d8 Address:3427fc90-d99f-88be-d147-f80d430d05d8} {Suffrage:Voter ID:server-cd252465-3cae-db8d-8b9d-d69e2d241ffc Address:cd252465-3cae-db8d-8b9d-d69e2d241ffc} {Suffrage:Voter ID:server-31afc209-49de-67c8-ee30-cfcaa51597fd Address:31afc209-49de-67c8-ee30-cfcaa51597fd}]"
2026-05-25T23:22:54.537Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:54.537Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: entering follower state: follower="Node at cd252465-3cae-db8d-8b9d-d69e2d241ffc [Follower]" leader-address= leader-id=
2026-05-25T23:22:54.537Z [INFO]  server-31afc209-49de-67c8-ee30-cfcaa51597fd: entering follower state: follower="Node at 31afc209-49de-67c8-ee30-cfcaa51597fd [Follower]" leader-address= leader-id=
2026-05-25T23:22:55.078Z [WARN]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:55.078Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: entering candidate state: node="Node at cd252465-3cae-db8d-8b9d-d69e2d241ffc [Candidate]" term=2
2026-05-25T23:22:55.078Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: election won: term=2 tally=2
2026-05-25T23:22:55.078Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: entering leader state: leader="Node at cd252465-3cae-db8d-8b9d-d69e2d241ffc [Leader]"
2026-05-25T23:22:55.078Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: added peer, starting replication: peer=server-3427fc90-d99f-88be-d147-f80d430d05d8
2026-05-25T23:22:55.078Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: added peer, starting replication: peer=server-31afc209-49de-67c8-ee30-cfcaa51597fd
2026-05-25T23:22:55.078Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: pipelining replication: peer="{Voter server-31afc209-49de-67c8-ee30-cfcaa51597fd 31afc209-49de-67c8-ee30-cfcaa51597fd}"
2026-05-25T23:22:55.078Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: pipelining replication: peer="{Voter server-3427fc90-d99f-88be-d147-f80d430d05d8 3427fc90-d99f-88be-d147-f80d430d05d8}"
2026-05-25T23:22:56.084Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:55.078557338 +0000 UTC m=+30.725004811 (1 nodes), 541.21165ms from start of poll, 541.583815ms from cluster start. Timeout at 2026-05-25 23:22:56.083626033 +0000 UTC m=+31.730073487, 1.005068676s after stability
2026-05-25T23:22:56.085Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: starting snapshot up to: index=12
2026-05-25T23:22:56.085Z [INFO]  creating new snapshot: path=/tmp/raft631745645/snapshots/2-12-1779751376085.tmp
2026-05-25T23:22:56.085Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: snapshot complete up to: index=12
2026-05-25T23:22:56.086Z [INFO]  creating new snapshot: path=/tmp/raft631745645/snapshots/2-23-1779751376086.tmp
2026-05-25T23:22:56.086Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: copied to local snapshot: bytes=71
2026-05-25T23:22:56.087Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: snapshot restore progress: id=2-23-1779751376086 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:56.087Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: removing all old logs from log store
2026-05-25T23:22:56.087Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: compacting logs: from=1 to=22
2026-05-25T23:22:56.087Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: restored user snapshot: index=23
2026-05-25T23:22:56.087Z [ERROR] server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: failed to get log: index=22 error="log not found"
2026-05-25T23:22:56.087Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: aborting pipeline replication: peer="{Voter server-31afc209-49de-67c8-ee30-cfcaa51597fd 31afc209-49de-67c8-ee30-cfcaa51597fd}"
2026-05-25T23:22:56.087Z [ERROR] server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: failed to get log: index=22 error="log not found"
2026-05-25T23:22:56.087Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: aborting pipeline replication: peer="{Voter server-3427fc90-d99f-88be-d147-f80d430d05d8 3427fc90-d99f-88be-d147-f80d430d05d8}"
2026-05-25T23:22:56.094Z [ERROR] server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: failed to get log: index=22 error="log not found"
2026-05-25T23:22:56.095Z [INFO]  creating new snapshot: path=/tmp/raft679168561/snapshots/2-23-1779751376095.tmp
2026-05-25T23:22:56.095Z [INFO]  server-31afc209-49de-67c8-ee30-cfcaa51597fd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:56.095Z [INFO]  server-31afc209-49de-67c8-ee30-cfcaa51597fd: copied to local snapshot: bytes=71
2026-05-25T23:22:56.095Z [INFO]  server-31afc209-49de-67c8-ee30-cfcaa51597fd: snapshot restore progress: id=2-23-1779751376095 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:56.095Z [INFO]  server-31afc209-49de-67c8-ee30-cfcaa51597fd: removing all old logs from log store
2026-05-25T23:22:56.095Z [INFO]  server-31afc209-49de-67c8-ee30-cfcaa51597fd: compacting logs: from=1 to=22
2026-05-25T23:22:56.095Z [INFO]  server-31afc209-49de-67c8-ee30-cfcaa51597fd: Installed remote snapshot
2026-05-25T23:22:56.095Z [WARN]  server-31afc209-49de-67c8-ee30-cfcaa51597fd: skipping application of old log: index=22
2026-05-25T23:22:56.095Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: pipelining replication: peer="{Voter server-31afc209-49de-67c8-ee30-cfcaa51597fd 31afc209-49de-67c8-ee30-cfcaa51597fd}"
2026-05-25T23:22:56.095Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:22:56.096Z [ERROR] server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: failed to get log: index=22 error="log not found"
2026-05-25T23:22:56.097Z [INFO]  creating new snapshot: path=/tmp/raft3191392163/snapshots/2-23-1779751376097.tmp
2026-05-25T23:22:56.097Z [INFO]  server-3427fc90-d99f-88be-d147-f80d430d05d8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:56.097Z [INFO]  server-3427fc90-d99f-88be-d147-f80d430d05d8: copied to local snapshot: bytes=71
2026-05-25T23:22:56.098Z [INFO]  server-3427fc90-d99f-88be-d147-f80d430d05d8: snapshot restore progress: id=2-23-1779751376097 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:22:56.098Z [INFO]  server-3427fc90-d99f-88be-d147-f80d430d05d8: removing all old logs from log store
2026-05-25T23:22:56.098Z [INFO]  server-3427fc90-d99f-88be-d147-f80d430d05d8: compacting logs: from=1 to=22
2026-05-25T23:22:56.098Z [INFO]  server-3427fc90-d99f-88be-d147-f80d430d05d8: Installed remote snapshot
2026-05-25T23:22:56.098Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: pipelining replication: peer="{Voter server-3427fc90-d99f-88be-d147-f80d430d05d8 3427fc90-d99f-88be-d147-f80d430d05d8}"
2026-05-25T23:22:57.101Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:22:56.095939871 +0000 UTC m=+31.742387330 (2 nodes), 746ns from start of poll, 1.558966334s from cluster start. Timeout at 2026-05-25 23:22:57.100953033 +0000 UTC m=+32.747400489, 1.005013159s after stability
2026-05-25T23:22:57.113Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: aborting pipeline replication: peer="{Voter server-31afc209-49de-67c8-ee30-cfcaa51597fd 31afc209-49de-67c8-ee30-cfcaa51597fd}"
2026-05-25T23:22:57.113Z [INFO]  server-cd252465-3cae-db8d-8b9d-d69e2d241ffc: aborting pipeline replication: peer="{Voter server-3427fc90-d99f-88be-d147-f80d430d05d8 3427fc90-d99f-88be-d147-f80d430d05d8}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true
2026-05-25T23:22:57.114Z [INFO]  server-3a7ffa33-d152-7ab4-0883-6986660f6eac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a7ffa33-d152-7ab4-0883-6986660f6eac Address:3a7ffa33-d152-7ab4-0883-6986660f6eac} {Suffrage:Voter ID:server-b127950c-3747-538c-16d5-9d94d9c833cb Address:b127950c-3747-538c-16d5-9d94d9c833cb} {Suffrage:Voter ID:server-3e88b45b-f6c8-a19d-cfdb-07c290cac019 Address:3e88b45b-f6c8-a19d-cfdb-07c290cac019}]"
2026-05-25T23:22:57.114Z [INFO]  server-b127950c-3747-538c-16d5-9d94d9c833cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a7ffa33-d152-7ab4-0883-6986660f6eac Address:3a7ffa33-d152-7ab4-0883-6986660f6eac} {Suffrage:Voter ID:server-b127950c-3747-538c-16d5-9d94d9c833cb Address:b127950c-3747-538c-16d5-9d94d9c833cb} {Suffrage:Voter ID:server-3e88b45b-f6c8-a19d-cfdb-07c290cac019 Address:3e88b45b-f6c8-a19d-cfdb-07c290cac019}]"
2026-05-25T23:22:57.114Z [INFO]  server-3a7ffa33-d152-7ab4-0883-6986660f6eac: entering follower state: follower="Node at 3a7ffa33-d152-7ab4-0883-6986660f6eac [Follower]" leader-address= leader-id=
2026-05-25T23:22:57.114Z [INFO]  server-3e88b45b-f6c8-a19d-cfdb-07c290cac019: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a7ffa33-d152-7ab4-0883-6986660f6eac Address:3a7ffa33-d152-7ab4-0883-6986660f6eac} {Suffrage:Voter ID:server-b127950c-3747-538c-16d5-9d94d9c833cb Address:b127950c-3747-538c-16d5-9d94d9c833cb} {Suffrage:Voter ID:server-3e88b45b-f6c8-a19d-cfdb-07c290cac019 Address:3e88b45b-f6c8-a19d-cfdb-07c290cac019}]"
2026-05-25T23:22:57.114Z [INFO]  server-b127950c-3747-538c-16d5-9d94d9c833cb: entering follower state: follower="Node at b127950c-3747-538c-16d5-9d94d9c833cb [Follower]" leader-address= leader-id=
2026-05-25T23:22:57.114Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:57.114Z [INFO]  server-3e88b45b-f6c8-a19d-cfdb-07c290cac019: entering follower state: follower="Node at 3e88b45b-f6c8-a19d-cfdb-07c290cac019 [Follower]" leader-address= leader-id=
2026-05-25T23:22:57.796Z [WARN]  server-3a7ffa33-d152-7ab4-0883-6986660f6eac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:57.796Z [INFO]  server-3a7ffa33-d152-7ab4-0883-6986660f6eac: entering candidate state: node="Node at 3a7ffa33-d152-7ab4-0883-6986660f6eac [Candidate]" term=2
2026-05-25T23:22:57.796Z [INFO]  server-3a7ffa33-d152-7ab4-0883-6986660f6eac: election won: term=2 tally=2
2026-05-25T23:22:57.796Z [INFO]  server-3a7ffa33-d152-7ab4-0883-6986660f6eac: entering leader state: leader="Node at 3a7ffa33-d152-7ab4-0883-6986660f6eac [Leader]"
2026-05-25T23:22:57.796Z [INFO]  server-3a7ffa33-d152-7ab4-0883-6986660f6eac: added peer, starting replication: peer=server-b127950c-3747-538c-16d5-9d94d9c833cb
2026-05-25T23:22:57.796Z [INFO]  server-3a7ffa33-d152-7ab4-0883-6986660f6eac: added peer, starting replication: peer=server-3e88b45b-f6c8-a19d-cfdb-07c290cac019
2026-05-25T23:22:57.796Z [INFO]  server-3a7ffa33-d152-7ab4-0883-6986660f6eac: pipelining replication: peer="{Voter server-3e88b45b-f6c8-a19d-cfdb-07c290cac019 3e88b45b-f6c8-a19d-cfdb-07c290cac019}"
2026-05-25T23:22:57.796Z [INFO]  server-3a7ffa33-d152-7ab4-0883-6986660f6eac: pipelining replication: peer="{Voter server-b127950c-3747-538c-16d5-9d94d9c833cb b127950c-3747-538c-16d5-9d94d9c833cb}"
2026-05-25T23:22:58.802Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:57.79681116 +0000 UTC m=+33.443258626 (1 nodes), 681.870081ms from start of poll, 682.229563ms from cluster start. Timeout at 2026-05-25 23:22:58.801820035 +0000 UTC m=+34.448267493, 1.005008867s after stability
2026-05-25T23:22:58.804Z [INFO]  server-3a7ffa33-d152-7ab4-0883-6986660f6eac: starting snapshot up to: index=12
2026-05-25T23:22:58.804Z [INFO]  creating new snapshot: path=/tmp/raft2915662174/snapshots/2-12-1779751378804.tmp
2026-05-25T23:22:58.805Z [INFO]  server-3a7ffa33-d152-7ab4-0883-6986660f6eac: snapshot complete up to: index=12
2026-05-25T23:22:58.806Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-164f4764-ebd7-bbaa-3367-321add0910e4 Address:164f4764-ebd7-bbaa-3367-321add0910e4} {Suffrage:Voter ID:server-43deaf51-2664-9283-bde9-1056361f0668 Address:43deaf51-2664-9283-bde9-1056361f0668} {Suffrage:Voter ID:server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f Address:19c71118-7e61-cc98-9b5e-18e0d7ec6f6f}]"
2026-05-25T23:22:58.806Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: entering follower state: follower="Node at 164f4764-ebd7-bbaa-3367-321add0910e4 [Follower]" leader-address= leader-id=
2026-05-25T23:22:58.806Z [INFO]  server-43deaf51-2664-9283-bde9-1056361f0668: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-164f4764-ebd7-bbaa-3367-321add0910e4 Address:164f4764-ebd7-bbaa-3367-321add0910e4} {Suffrage:Voter ID:server-43deaf51-2664-9283-bde9-1056361f0668 Address:43deaf51-2664-9283-bde9-1056361f0668} {Suffrage:Voter ID:server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f Address:19c71118-7e61-cc98-9b5e-18e0d7ec6f6f}]"
2026-05-25T23:22:58.806Z [INFO]  server-43deaf51-2664-9283-bde9-1056361f0668: entering follower state: follower="Node at 43deaf51-2664-9283-bde9-1056361f0668 [Follower]" leader-address= leader-id=
2026-05-25T23:22:58.806Z [INFO]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-164f4764-ebd7-bbaa-3367-321add0910e4 Address:164f4764-ebd7-bbaa-3367-321add0910e4} {Suffrage:Voter ID:server-43deaf51-2664-9283-bde9-1056361f0668 Address:43deaf51-2664-9283-bde9-1056361f0668} {Suffrage:Voter ID:server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f Address:19c71118-7e61-cc98-9b5e-18e0d7ec6f6f}]"
2026-05-25T23:22:58.806Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:22:58.806Z [INFO]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: entering follower state: follower="Node at 19c71118-7e61-cc98-9b5e-18e0d7ec6f6f [Follower]" leader-address= leader-id=
2026-05-25T23:22:59.336Z [WARN]  server-164f4764-ebd7-bbaa-3367-321add0910e4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:22:59.336Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: entering candidate state: node="Node at 164f4764-ebd7-bbaa-3367-321add0910e4 [Candidate]" term=2
2026-05-25T23:22:59.336Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: election won: term=2 tally=2
2026-05-25T23:22:59.336Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: entering leader state: leader="Node at 164f4764-ebd7-bbaa-3367-321add0910e4 [Leader]"
2026-05-25T23:22:59.336Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: added peer, starting replication: peer=server-43deaf51-2664-9283-bde9-1056361f0668
2026-05-25T23:22:59.336Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: added peer, starting replication: peer=server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f
2026-05-25T23:22:59.337Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: pipelining replication: peer="{Voter server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f 19c71118-7e61-cc98-9b5e-18e0d7ec6f6f}"
2026-05-25T23:22:59.337Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: pipelining replication: peer="{Voter server-43deaf51-2664-9283-bde9-1056361f0668 43deaf51-2664-9283-bde9-1056361f0668}"
2026-05-25T23:23:00.342Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:22:59.336857743 +0000 UTC m=+34.983305213 (1 nodes), 530.070034ms from start of poll, 530.433532ms from cluster start. Timeout at 2026-05-25 23:23:00.341862515 +0000 UTC m=+35.988309966, 1.005004753s after stability
2026-05-25T23:23:00.342Z [INFO]  creating new snapshot: path=/tmp/raft2663663008/snapshots/2-14-1779751380342.tmp
2026-05-25T23:23:00.343Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: copied to local snapshot: bytes=71
2026-05-25T23:23:00.343Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: snapshot restore progress: id=2-14-1779751380342 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:00.343Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: restored user snapshot: index=14
2026-05-25T23:23:00.343Z [ERROR] server-164f4764-ebd7-bbaa-3367-321add0910e4: failed to get log: index=3 error="log not found"
2026-05-25T23:23:00.343Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: aborting pipeline replication: peer="{Voter server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f 19c71118-7e61-cc98-9b5e-18e0d7ec6f6f}"
2026-05-25T23:23:00.343Z [ERROR] server-164f4764-ebd7-bbaa-3367-321add0910e4: failed to get log: index=3 error="log not found"
2026-05-25T23:23:00.343Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: aborting pipeline replication: peer="{Voter server-43deaf51-2664-9283-bde9-1056361f0668 43deaf51-2664-9283-bde9-1056361f0668}"
2026-05-25T23:23:00.351Z [ERROR] server-164f4764-ebd7-bbaa-3367-321add0910e4: failed to get log: index=3 error="log not found"
2026-05-25T23:23:00.351Z [INFO]  creating new snapshot: path=/tmp/raft1706078795/snapshots/2-14-1779751380351.tmp
2026-05-25T23:23:00.352Z [ERROR] server-164f4764-ebd7-bbaa-3367-321add0910e4: failed to get log: index=3 error="log not found"
2026-05-25T23:23:00.352Z [INFO]  server-43deaf51-2664-9283-bde9-1056361f0668: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:00.353Z [INFO]  creating new snapshot: path=/tmp/raft3405534013/snapshots/2-14-1779751380353.tmp
2026-05-25T23:23:00.353Z [INFO]  server-43deaf51-2664-9283-bde9-1056361f0668: copied to local snapshot: bytes=71
2026-05-25T23:23:00.353Z [INFO]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:00.353Z [INFO]  server-43deaf51-2664-9283-bde9-1056361f0668: snapshot restore progress: id=2-14-1779751380351 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:00.353Z [INFO]  server-43deaf51-2664-9283-bde9-1056361f0668: Installed remote snapshot
2026-05-25T23:23:00.353Z [INFO]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: copied to local snapshot: bytes=71
2026-05-25T23:23:00.353Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:23:00.353Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: pipelining replication: peer="{Voter server-43deaf51-2664-9283-bde9-1056361f0668 43deaf51-2664-9283-bde9-1056361f0668}"
2026-05-25T23:23:00.354Z [INFO]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: snapshot restore progress: id=2-14-1779751380353 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:00.354Z [INFO]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: Installed remote snapshot
2026-05-25T23:23:00.354Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: pipelining replication: peer="{Voter server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f 19c71118-7e61-cc98-9b5e-18e0d7ec6f6f}"
2026-05-25T23:23:01.359Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:23:00.353933239 +0000 UTC m=+36.000380702 (2 nodes), 933ns from start of poll, 1.547509021s from cluster start. Timeout at 2026-05-25 23:23:01.358938006 +0000 UTC m=+37.005385456, 1.005004754s after stability
2026-05-25T23:23:01.371Z [INFO]  server-3a7ffa33-d152-7ab4-0883-6986660f6eac: aborting pipeline replication: peer="{Voter server-3e88b45b-f6c8-a19d-cfdb-07c290cac019 3e88b45b-f6c8-a19d-cfdb-07c290cac019}"
2026-05-25T23:23:01.371Z [INFO]  server-3a7ffa33-d152-7ab4-0883-6986660f6eac: aborting pipeline replication: peer="{Voter server-b127950c-3747-538c-16d5-9d94d9c833cb b127950c-3747-538c-16d5-9d94d9c833cb}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true
2026-05-25T23:23:01.372Z [INFO]  server-372b8d65-4389-ff97-70bd-694aeac9b5b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-372b8d65-4389-ff97-70bd-694aeac9b5b2 Address:372b8d65-4389-ff97-70bd-694aeac9b5b2} {Suffrage:Voter ID:server-8fb29423-5107-e652-440a-ba04c8d0a62c Address:8fb29423-5107-e652-440a-ba04c8d0a62c} {Suffrage:Voter ID:server-eeed3c0d-53c6-48c7-a0d6-7356611a4586 Address:eeed3c0d-53c6-48c7-a0d6-7356611a4586}]"
2026-05-25T23:23:01.372Z [INFO]  server-372b8d65-4389-ff97-70bd-694aeac9b5b2: entering follower state: follower="Node at 372b8d65-4389-ff97-70bd-694aeac9b5b2 [Follower]" leader-address= leader-id=
2026-05-25T23:23:01.372Z [INFO]  server-8fb29423-5107-e652-440a-ba04c8d0a62c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-372b8d65-4389-ff97-70bd-694aeac9b5b2 Address:372b8d65-4389-ff97-70bd-694aeac9b5b2} {Suffrage:Voter ID:server-8fb29423-5107-e652-440a-ba04c8d0a62c Address:8fb29423-5107-e652-440a-ba04c8d0a62c} {Suffrage:Voter ID:server-eeed3c0d-53c6-48c7-a0d6-7356611a4586 Address:eeed3c0d-53c6-48c7-a0d6-7356611a4586}]"
2026-05-25T23:23:01.372Z [INFO]  server-8fb29423-5107-e652-440a-ba04c8d0a62c: entering follower state: follower="Node at 8fb29423-5107-e652-440a-ba04c8d0a62c [Follower]" leader-address= leader-id=
2026-05-25T23:23:01.372Z [INFO]  server-eeed3c0d-53c6-48c7-a0d6-7356611a4586: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-372b8d65-4389-ff97-70bd-694aeac9b5b2 Address:372b8d65-4389-ff97-70bd-694aeac9b5b2} {Suffrage:Voter ID:server-8fb29423-5107-e652-440a-ba04c8d0a62c Address:8fb29423-5107-e652-440a-ba04c8d0a62c} {Suffrage:Voter ID:server-eeed3c0d-53c6-48c7-a0d6-7356611a4586 Address:eeed3c0d-53c6-48c7-a0d6-7356611a4586}]"
2026-05-25T23:23:01.373Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:01.373Z [INFO]  server-eeed3c0d-53c6-48c7-a0d6-7356611a4586: entering follower state: follower="Node at eeed3c0d-53c6-48c7-a0d6-7356611a4586 [Follower]" leader-address= leader-id=
2026-05-25T23:23:01.881Z [WARN]  server-eeed3c0d-53c6-48c7-a0d6-7356611a4586: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:01.881Z [INFO]  server-eeed3c0d-53c6-48c7-a0d6-7356611a4586: entering candidate state: node="Node at eeed3c0d-53c6-48c7-a0d6-7356611a4586 [Candidate]" term=2
2026-05-25T23:23:01.881Z [INFO]  server-eeed3c0d-53c6-48c7-a0d6-7356611a4586: election won: term=2 tally=2
2026-05-25T23:23:01.881Z [INFO]  server-eeed3c0d-53c6-48c7-a0d6-7356611a4586: entering leader state: leader="Node at eeed3c0d-53c6-48c7-a0d6-7356611a4586 [Leader]"
2026-05-25T23:23:01.881Z [INFO]  server-eeed3c0d-53c6-48c7-a0d6-7356611a4586: added peer, starting replication: peer=server-372b8d65-4389-ff97-70bd-694aeac9b5b2
2026-05-25T23:23:01.881Z [INFO]  server-eeed3c0d-53c6-48c7-a0d6-7356611a4586: added peer, starting replication: peer=server-8fb29423-5107-e652-440a-ba04c8d0a62c
2026-05-25T23:23:01.882Z [INFO]  server-eeed3c0d-53c6-48c7-a0d6-7356611a4586: pipelining replication: peer="{Voter server-8fb29423-5107-e652-440a-ba04c8d0a62c 8fb29423-5107-e652-440a-ba04c8d0a62c}"
2026-05-25T23:23:01.882Z [INFO]  server-eeed3c0d-53c6-48c7-a0d6-7356611a4586: pipelining replication: peer="{Voter server-372b8d65-4389-ff97-70bd-694aeac9b5b2 372b8d65-4389-ff97-70bd-694aeac9b5b2}"
2026-05-25T23:23:02.887Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:01.881986609 +0000 UTC m=+37.528434062 (1 nodes), 508.918568ms from start of poll, 509.738689ms from cluster start. Timeout at 2026-05-25 23:23:02.886990782 +0000 UTC m=+38.533438233, 1.005004171s after stability
2026-05-25T23:23:02.888Z [INFO]  server-eeed3c0d-53c6-48c7-a0d6-7356611a4586: starting snapshot up to: index=12
2026-05-25T23:23:02.888Z [INFO]  creating new snapshot: path=/tmp/raft1349993081/snapshots/2-12-1779751382888.tmp
2026-05-25T23:23:02.888Z [INFO]  server-eeed3c0d-53c6-48c7-a0d6-7356611a4586: snapshot complete up to: index=12
2026-05-25T23:23:02.889Z [INFO]  server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb Address:bc1e4076-200c-c555-eb3f-ab3ef10bbedb} {Suffrage:Voter ID:server-cb02535b-4fa3-e7a6-9c73-0120936129ca Address:cb02535b-4fa3-e7a6-9c73-0120936129ca} {Suffrage:Voter ID:server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc Address:eb7f8dda-b789-58f9-9aee-a08a5ed05fcc}]"
2026-05-25T23:23:02.890Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb Address:bc1e4076-200c-c555-eb3f-ab3ef10bbedb} {Suffrage:Voter ID:server-cb02535b-4fa3-e7a6-9c73-0120936129ca Address:cb02535b-4fa3-e7a6-9c73-0120936129ca} {Suffrage:Voter ID:server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc Address:eb7f8dda-b789-58f9-9aee-a08a5ed05fcc}]"
2026-05-25T23:23:02.890Z [INFO]  server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb: entering follower state: follower="Node at bc1e4076-200c-c555-eb3f-ab3ef10bbedb [Follower]" leader-address= leader-id=
2026-05-25T23:23:02.890Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: entering follower state: follower="Node at cb02535b-4fa3-e7a6-9c73-0120936129ca [Follower]" leader-address= leader-id=
2026-05-25T23:23:02.890Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb Address:bc1e4076-200c-c555-eb3f-ab3ef10bbedb} {Suffrage:Voter ID:server-cb02535b-4fa3-e7a6-9c73-0120936129ca Address:cb02535b-4fa3-e7a6-9c73-0120936129ca} {Suffrage:Voter ID:server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc Address:eb7f8dda-b789-58f9-9aee-a08a5ed05fcc}]"
2026-05-25T23:23:02.891Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:02.891Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: entering follower state: follower="Node at eb7f8dda-b789-58f9-9aee-a08a5ed05fcc [Follower]" leader-address= leader-id=
2026-05-25T23:23:03.437Z [WARN]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:03.437Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: entering candidate state: node="Node at eb7f8dda-b789-58f9-9aee-a08a5ed05fcc [Candidate]" term=2
2026-05-25T23:23:03.437Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: election won: term=2 tally=2
2026-05-25T23:23:03.438Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: entering leader state: leader="Node at eb7f8dda-b789-58f9-9aee-a08a5ed05fcc [Leader]"
2026-05-25T23:23:03.438Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: added peer, starting replication: peer=server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb
2026-05-25T23:23:03.438Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: added peer, starting replication: peer=server-cb02535b-4fa3-e7a6-9c73-0120936129ca
2026-05-25T23:23:03.438Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: pipelining replication: peer="{Voter server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb bc1e4076-200c-c555-eb3f-ab3ef10bbedb}"
2026-05-25T23:23:03.438Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: pipelining replication: peer="{Voter server-cb02535b-4fa3-e7a6-9c73-0120936129ca cb02535b-4fa3-e7a6-9c73-0120936129ca}"
2026-05-25T23:23:04.444Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:03.438094299 +0000 UTC m=+39.084541764 (1 nodes), 546.793352ms from start of poll, 548.520238ms from cluster start. Timeout at 2026-05-25 23:23:04.443097542 +0000 UTC m=+40.089544996, 1.005003232s after stability
2026-05-25T23:23:04.444Z [INFO]  creating new snapshot: path=/tmp/raft1699989054/snapshots/2-14-1779751384444.tmp
2026-05-25T23:23:04.444Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: copied to local snapshot: bytes=71
2026-05-25T23:23:04.445Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: snapshot restore progress: id=2-14-1779751384444 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:04.445Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: removing all old logs from log store
2026-05-25T23:23:04.445Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: compacting logs: from=1 to=2
2026-05-25T23:23:04.445Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: restored user snapshot: index=14
2026-05-25T23:23:04.445Z [ERROR] server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: failed to get log: index=2 error="log not found"
2026-05-25T23:23:04.445Z [ERROR] server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: failed to get log: index=2 error="log not found"
2026-05-25T23:23:04.445Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: aborting pipeline replication: peer="{Voter server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb bc1e4076-200c-c555-eb3f-ab3ef10bbedb}"
2026-05-25T23:23:04.445Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: aborting pipeline replication: peer="{Voter server-cb02535b-4fa3-e7a6-9c73-0120936129ca cb02535b-4fa3-e7a6-9c73-0120936129ca}"
2026-05-25T23:23:04.451Z [ERROR] server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: failed to get log: index=2 error="log not found"
2026-05-25T23:23:04.452Z [INFO]  creating new snapshot: path=/tmp/raft2935504058/snapshots/2-14-1779751384452.tmp
2026-05-25T23:23:04.452Z [INFO]  server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:04.452Z [INFO]  server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb: copied to local snapshot: bytes=71
2026-05-25T23:23:04.452Z [INFO]  server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb: snapshot restore progress: id=2-14-1779751384452 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:04.452Z [INFO]  server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb: removing all old logs from log store
2026-05-25T23:23:04.452Z [INFO]  server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb: compacting logs: from=1 to=2
2026-05-25T23:23:04.452Z [INFO]  server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb: Installed remote snapshot
2026-05-25T23:23:04.453Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: pipelining replication: peer="{Voter server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb bc1e4076-200c-c555-eb3f-ab3ef10bbedb}"
2026-05-25T23:23:04.453Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:23:04.455Z [ERROR] server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: failed to get log: index=2 error="log not found"
2026-05-25T23:23:04.455Z [INFO]  creating new snapshot: path=/tmp/raft300337071/snapshots/2-14-1779751384455.tmp
2026-05-25T23:23:04.455Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:04.455Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: copied to local snapshot: bytes=71
2026-05-25T23:23:04.456Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: snapshot restore progress: id=2-14-1779751384455 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:04.456Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: removing all old logs from log store
2026-05-25T23:23:04.456Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: compacting logs: from=1 to=2
2026-05-25T23:23:04.456Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: Installed remote snapshot
2026-05-25T23:23:04.456Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: pipelining replication: peer="{Voter server-cb02535b-4fa3-e7a6-9c73-0120936129ca cb02535b-4fa3-e7a6-9c73-0120936129ca}"
2026-05-25T23:23:05.458Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:23:04.453140112 +0000 UTC m=+40.099587576 (2 nodes), 1.104µs from start of poll, 1.56356605s from cluster start. Timeout at 2026-05-25 23:23:05.45815279 +0000 UTC m=+41.104600242, 1.005012666s after stability
2026-05-25T23:23:05.470Z [INFO]  server-eeed3c0d-53c6-48c7-a0d6-7356611a4586: aborting pipeline replication: peer="{Voter server-8fb29423-5107-e652-440a-ba04c8d0a62c 8fb29423-5107-e652-440a-ba04c8d0a62c}"
2026-05-25T23:23:05.470Z [INFO]  server-eeed3c0d-53c6-48c7-a0d6-7356611a4586: aborting pipeline replication: peer="{Voter server-372b8d65-4389-ff97-70bd-694aeac9b5b2 372b8d65-4389-ff97-70bd-694aeac9b5b2}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false
2026-05-25T23:23:05.471Z [INFO]  server-976d1385-5cfb-b672-cd40-0a4183017b14: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-976d1385-5cfb-b672-cd40-0a4183017b14 Address:976d1385-5cfb-b672-cd40-0a4183017b14} {Suffrage:Voter ID:server-38f2666a-3365-f818-0d11-2b6208ba8040 Address:38f2666a-3365-f818-0d11-2b6208ba8040} {Suffrage:Voter ID:server-e4005e77-f873-32e8-83d0-25b90c31f4e1 Address:e4005e77-f873-32e8-83d0-25b90c31f4e1}]"
2026-05-25T23:23:05.471Z [INFO]  server-976d1385-5cfb-b672-cd40-0a4183017b14: entering follower state: follower="Node at 976d1385-5cfb-b672-cd40-0a4183017b14 [Follower]" leader-address= leader-id=
2026-05-25T23:23:05.471Z [INFO]  server-38f2666a-3365-f818-0d11-2b6208ba8040: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-976d1385-5cfb-b672-cd40-0a4183017b14 Address:976d1385-5cfb-b672-cd40-0a4183017b14} {Suffrage:Voter ID:server-38f2666a-3365-f818-0d11-2b6208ba8040 Address:38f2666a-3365-f818-0d11-2b6208ba8040} {Suffrage:Voter ID:server-e4005e77-f873-32e8-83d0-25b90c31f4e1 Address:e4005e77-f873-32e8-83d0-25b90c31f4e1}]"
2026-05-25T23:23:05.471Z [INFO]  server-38f2666a-3365-f818-0d11-2b6208ba8040: entering follower state: follower="Node at 38f2666a-3365-f818-0d11-2b6208ba8040 [Follower]" leader-address= leader-id=
2026-05-25T23:23:05.471Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-976d1385-5cfb-b672-cd40-0a4183017b14 Address:976d1385-5cfb-b672-cd40-0a4183017b14} {Suffrage:Voter ID:server-38f2666a-3365-f818-0d11-2b6208ba8040 Address:38f2666a-3365-f818-0d11-2b6208ba8040} {Suffrage:Voter ID:server-e4005e77-f873-32e8-83d0-25b90c31f4e1 Address:e4005e77-f873-32e8-83d0-25b90c31f4e1}]"
2026-05-25T23:23:05.471Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:05.471Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: entering follower state: follower="Node at e4005e77-f873-32e8-83d0-25b90c31f4e1 [Follower]" leader-address= leader-id=
2026-05-25T23:23:06.011Z [WARN]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:06.011Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: entering candidate state: node="Node at e4005e77-f873-32e8-83d0-25b90c31f4e1 [Candidate]" term=2
2026-05-25T23:23:06.011Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: election won: term=2 tally=2
2026-05-25T23:23:06.011Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: entering leader state: leader="Node at e4005e77-f873-32e8-83d0-25b90c31f4e1 [Leader]"
2026-05-25T23:23:06.011Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: added peer, starting replication: peer=server-976d1385-5cfb-b672-cd40-0a4183017b14
2026-05-25T23:23:06.011Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: added peer, starting replication: peer=server-38f2666a-3365-f818-0d11-2b6208ba8040
2026-05-25T23:23:06.011Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: pipelining replication: peer="{Voter server-38f2666a-3365-f818-0d11-2b6208ba8040 38f2666a-3365-f818-0d11-2b6208ba8040}"
2026-05-25T23:23:06.011Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: pipelining replication: peer="{Voter server-976d1385-5cfb-b672-cd40-0a4183017b14 976d1385-5cfb-b672-cd40-0a4183017b14}"
2026-05-25T23:23:07.016Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:06.011685782 +0000 UTC m=+41.658133249 (1 nodes), 540.09346ms from start of poll, 540.385976ms from cluster start. Timeout at 2026-05-25 23:23:07.016689328 +0000 UTC m=+42.663136781, 1.005003532s after stability
2026-05-25T23:23:07.017Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: starting snapshot up to: index=12
2026-05-25T23:23:07.017Z [INFO]  creating new snapshot: path=/tmp/raft121208075/snapshots/2-12-1779751387017.tmp
2026-05-25T23:23:07.017Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: snapshot complete up to: index=12
2026-05-25T23:23:07.020Z [INFO]  creating new snapshot: path=/tmp/raft121208075/snapshots/2-23-1779751387020.tmp
2026-05-25T23:23:07.020Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: copied to local snapshot: bytes=71
2026-05-25T23:23:07.020Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: snapshot restore progress: id=2-23-1779751387020 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:07.020Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: restored user snapshot: index=23
2026-05-25T23:23:07.021Z [ERROR] server-e4005e77-f873-32e8-83d0-25b90c31f4e1: failed to get log: index=23 error="log not found"
2026-05-25T23:23:07.021Z [ERROR] server-e4005e77-f873-32e8-83d0-25b90c31f4e1: failed to get log: index=23 error="log not found"
2026-05-25T23:23:07.021Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: aborting pipeline replication: peer="{Voter server-976d1385-5cfb-b672-cd40-0a4183017b14 976d1385-5cfb-b672-cd40-0a4183017b14}"
2026-05-25T23:23:07.021Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: aborting pipeline replication: peer="{Voter server-38f2666a-3365-f818-0d11-2b6208ba8040 38f2666a-3365-f818-0d11-2b6208ba8040}"
2026-05-25T23:23:07.026Z [ERROR] server-e4005e77-f873-32e8-83d0-25b90c31f4e1: failed to get log: index=23 error="log not found"
2026-05-25T23:23:07.026Z [INFO]  creating new snapshot: path=/tmp/raft3905015219/snapshots/2-23-1779751387026.tmp
2026-05-25T23:23:07.026Z [INFO]  server-38f2666a-3365-f818-0d11-2b6208ba8040: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:07.027Z [INFO]  server-38f2666a-3365-f818-0d11-2b6208ba8040: copied to local snapshot: bytes=71
2026-05-25T23:23:07.027Z [INFO]  server-38f2666a-3365-f818-0d11-2b6208ba8040: snapshot restore progress: id=2-23-1779751387026 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:07.027Z [INFO]  server-38f2666a-3365-f818-0d11-2b6208ba8040: Installed remote snapshot
2026-05-25T23:23:07.027Z [ERROR] server-e4005e77-f873-32e8-83d0-25b90c31f4e1: failed to get log: index=23 error="log not found"
2026-05-25T23:23:07.027Z [WARN]  server-38f2666a-3365-f818-0d11-2b6208ba8040: skipping application of old log: index=22
2026-05-25T23:23:07.027Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: pipelining replication: peer="{Voter server-38f2666a-3365-f818-0d11-2b6208ba8040 38f2666a-3365-f818-0d11-2b6208ba8040}"
2026-05-25T23:23:07.027Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:23:07.027Z [INFO]  creating new snapshot: path=/tmp/raft711207237/snapshots/2-23-1779751387027.tmp
2026-05-25T23:23:07.027Z [INFO]  server-976d1385-5cfb-b672-cd40-0a4183017b14: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:07.028Z [INFO]  server-976d1385-5cfb-b672-cd40-0a4183017b14: copied to local snapshot: bytes=71
2026-05-25T23:23:07.028Z [INFO]  server-976d1385-5cfb-b672-cd40-0a4183017b14: snapshot restore progress: id=2-23-1779751387027 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:07.028Z [INFO]  server-976d1385-5cfb-b672-cd40-0a4183017b14: Installed remote snapshot
2026-05-25T23:23:07.028Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: pipelining replication: peer="{Voter server-976d1385-5cfb-b672-cd40-0a4183017b14 976d1385-5cfb-b672-cd40-0a4183017b14}"
2026-05-25T23:23:08.033Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:23:07.027613032 +0000 UTC m=+42.674060579 (2 nodes), 1.709µs from start of poll, 1.556313306s from cluster start. Timeout at 2026-05-25 23:23:08.032617465 +0000 UTC m=+43.679064914, 1.005004335s after stability
2026-05-25T23:23:08.044Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: aborting pipeline replication: peer="{Voter server-976d1385-5cfb-b672-cd40-0a4183017b14 976d1385-5cfb-b672-cd40-0a4183017b14}"
2026-05-25T23:23:08.044Z [INFO]  server-e4005e77-f873-32e8-83d0-25b90c31f4e1: aborting pipeline replication: peer="{Voter server-38f2666a-3365-f818-0d11-2b6208ba8040 38f2666a-3365-f818-0d11-2b6208ba8040}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false
2026-05-25T23:23:08.046Z [INFO]  server-eae863d9-0745-2c31-b0b4-7ff5632d32c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eae863d9-0745-2c31-b0b4-7ff5632d32c6 Address:eae863d9-0745-2c31-b0b4-7ff5632d32c6} {Suffrage:Voter ID:server-8f1099c3-87c0-f1b3-aec1-5aaf78948493 Address:8f1099c3-87c0-f1b3-aec1-5aaf78948493} {Suffrage:Voter ID:server-848fd31b-7997-dd7e-7849-b1080966402f Address:848fd31b-7997-dd7e-7849-b1080966402f}]"
2026-05-25T23:23:08.046Z [INFO]  server-eae863d9-0745-2c31-b0b4-7ff5632d32c6: entering follower state: follower="Node at eae863d9-0745-2c31-b0b4-7ff5632d32c6 [Follower]" leader-address= leader-id=
2026-05-25T23:23:08.046Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eae863d9-0745-2c31-b0b4-7ff5632d32c6 Address:eae863d9-0745-2c31-b0b4-7ff5632d32c6} {Suffrage:Voter ID:server-8f1099c3-87c0-f1b3-aec1-5aaf78948493 Address:8f1099c3-87c0-f1b3-aec1-5aaf78948493} {Suffrage:Voter ID:server-848fd31b-7997-dd7e-7849-b1080966402f Address:848fd31b-7997-dd7e-7849-b1080966402f}]"
2026-05-25T23:23:08.046Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: entering follower state: follower="Node at 8f1099c3-87c0-f1b3-aec1-5aaf78948493 [Follower]" leader-address= leader-id=
2026-05-25T23:23:08.046Z [INFO]  server-848fd31b-7997-dd7e-7849-b1080966402f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eae863d9-0745-2c31-b0b4-7ff5632d32c6 Address:eae863d9-0745-2c31-b0b4-7ff5632d32c6} {Suffrage:Voter ID:server-8f1099c3-87c0-f1b3-aec1-5aaf78948493 Address:8f1099c3-87c0-f1b3-aec1-5aaf78948493} {Suffrage:Voter ID:server-848fd31b-7997-dd7e-7849-b1080966402f Address:848fd31b-7997-dd7e-7849-b1080966402f}]"
2026-05-25T23:23:08.046Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:08.046Z [INFO]  server-848fd31b-7997-dd7e-7849-b1080966402f: entering follower state: follower="Node at 848fd31b-7997-dd7e-7849-b1080966402f [Follower]" leader-address= leader-id=
2026-05-25T23:23:08.684Z [WARN]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:08.684Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: entering candidate state: node="Node at 8f1099c3-87c0-f1b3-aec1-5aaf78948493 [Candidate]" term=2
2026-05-25T23:23:08.684Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: election won: term=2 tally=2
2026-05-25T23:23:08.684Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: entering leader state: leader="Node at 8f1099c3-87c0-f1b3-aec1-5aaf78948493 [Leader]"
2026-05-25T23:23:08.684Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: added peer, starting replication: peer=server-eae863d9-0745-2c31-b0b4-7ff5632d32c6
2026-05-25T23:23:08.684Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: added peer, starting replication: peer=server-848fd31b-7997-dd7e-7849-b1080966402f
2026-05-25T23:23:08.684Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: pipelining replication: peer="{Voter server-eae863d9-0745-2c31-b0b4-7ff5632d32c6 eae863d9-0745-2c31-b0b4-7ff5632d32c6}"
2026-05-25T23:23:08.684Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: pipelining replication: peer="{Voter server-848fd31b-7997-dd7e-7849-b1080966402f 848fd31b-7997-dd7e-7849-b1080966402f}"
2026-05-25T23:23:09.690Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:08.68425511 +0000 UTC m=+44.330702573 (1 nodes), 637.746207ms from start of poll, 638.180925ms from cluster start. Timeout at 2026-05-25 23:23:09.689255856 +0000 UTC m=+45.335703307, 1.005000734s after stability
2026-05-25T23:23:09.690Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: starting snapshot up to: index=12
2026-05-25T23:23:09.690Z [INFO]  creating new snapshot: path=/tmp/raft3030543500/snapshots/2-12-1779751389690.tmp
2026-05-25T23:23:09.829Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: snapshot complete up to: index=12
2026-05-25T23:23:09.830Z [INFO]  creating new snapshot: path=/tmp/raft3030543500/snapshots/2-23-1779751389830.tmp
2026-05-25T23:23:09.831Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: copied to local snapshot: bytes=71
2026-05-25T23:23:09.831Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: snapshot restore progress: id=2-23-1779751389830 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:09.831Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: removing all old logs from log store
2026-05-25T23:23:09.831Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: compacting logs: from=1 to=22
2026-05-25T23:23:09.831Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: restored user snapshot: index=23
2026-05-25T23:23:09.831Z [ERROR] server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: failed to get log: index=22 error="log not found"
2026-05-25T23:23:09.831Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: aborting pipeline replication: peer="{Voter server-848fd31b-7997-dd7e-7849-b1080966402f 848fd31b-7997-dd7e-7849-b1080966402f}"
2026-05-25T23:23:09.831Z [ERROR] server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: failed to get log: index=22 error="log not found"
2026-05-25T23:23:09.831Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: aborting pipeline replication: peer="{Voter server-eae863d9-0745-2c31-b0b4-7ff5632d32c6 eae863d9-0745-2c31-b0b4-7ff5632d32c6}"
2026-05-25T23:23:09.838Z [ERROR] server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: failed to get log: index=22 error="log not found"
2026-05-25T23:23:09.838Z [INFO]  creating new snapshot: path=/tmp/raft3573124637/snapshots/2-23-1779751389838.tmp
2026-05-25T23:23:09.839Z [INFO]  server-848fd31b-7997-dd7e-7849-b1080966402f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:09.839Z [ERROR] server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: failed to get log: index=22 error="log not found"
2026-05-25T23:23:09.839Z [INFO]  creating new snapshot: path=/tmp/raft186667292/snapshots/2-23-1779751389839.tmp
2026-05-25T23:23:09.840Z [INFO]  server-eae863d9-0745-2c31-b0b4-7ff5632d32c6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:09.991Z [INFO]  server-eae863d9-0745-2c31-b0b4-7ff5632d32c6: copied to local snapshot: bytes=71
2026-05-25T23:23:09.991Z [INFO]  server-848fd31b-7997-dd7e-7849-b1080966402f: copied to local snapshot: bytes=71
2026-05-25T23:23:09.991Z [INFO]  server-eae863d9-0745-2c31-b0b4-7ff5632d32c6: snapshot restore progress: id=2-23-1779751389839 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:09.991Z [INFO]  server-eae863d9-0745-2c31-b0b4-7ff5632d32c6: removing all old logs from log store
2026-05-25T23:23:09.991Z [INFO]  server-eae863d9-0745-2c31-b0b4-7ff5632d32c6: compacting logs: from=1 to=22
2026-05-25T23:23:09.991Z [INFO]  server-eae863d9-0745-2c31-b0b4-7ff5632d32c6: Installed remote snapshot
2026-05-25T23:23:09.991Z [INFO]  server-848fd31b-7997-dd7e-7849-b1080966402f: snapshot restore progress: id=2-23-1779751389838 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:09.992Z [INFO]  server-848fd31b-7997-dd7e-7849-b1080966402f: removing all old logs from log store
2026-05-25T23:23:09.992Z [INFO]  server-848fd31b-7997-dd7e-7849-b1080966402f: compacting logs: from=1 to=22
2026-05-25T23:23:09.992Z [INFO]  server-848fd31b-7997-dd7e-7849-b1080966402f: Installed remote snapshot
2026-05-25T23:23:09.992Z [WARN]  server-eae863d9-0745-2c31-b0b4-7ff5632d32c6: skipping application of old log: index=22
2026-05-25T23:23:09.992Z [WARN]  server-848fd31b-7997-dd7e-7849-b1080966402f: skipping application of old log: index=23
2026-05-25T23:23:09.992Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: pipelining replication: peer="{Voter server-eae863d9-0745-2c31-b0b4-7ff5632d32c6 eae863d9-0745-2c31-b0b4-7ff5632d32c6}"
2026-05-25T23:23:09.992Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:23:09.992Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: pipelining replication: peer="{Voter server-848fd31b-7997-dd7e-7849-b1080966402f 848fd31b-7997-dd7e-7849-b1080966402f}"
2026-05-25T23:23:10.998Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:23:09.992205187 +0000 UTC m=+45.638652636 (2 nodes), 1.829µs from start of poll, 1.946130988s from cluster start. Timeout at 2026-05-25 23:23:10.997216232 +0000 UTC m=+46.643663715, 1.005011079s after stability
2026-05-25T23:23:11.009Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: aborting pipeline replication: peer="{Voter server-eae863d9-0745-2c31-b0b4-7ff5632d32c6 eae863d9-0745-2c31-b0b4-7ff5632d32c6}"
2026-05-25T23:23:11.009Z [INFO]  server-8f1099c3-87c0-f1b3-aec1-5aaf78948493: aborting pipeline replication: peer="{Voter server-848fd31b-7997-dd7e-7849-b1080966402f 848fd31b-7997-dd7e-7849-b1080966402f}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true
2026-05-25T23:23:11.011Z [INFO]  server-d951aaca-54eb-57a3-b2b9-08c207efab8a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d951aaca-54eb-57a3-b2b9-08c207efab8a Address:d951aaca-54eb-57a3-b2b9-08c207efab8a} {Suffrage:Voter ID:server-35693531-07c5-b061-e560-7e9cb8397ed5 Address:35693531-07c5-b061-e560-7e9cb8397ed5} {Suffrage:Voter ID:server-4cb66cc6-60ee-a159-1696-51e77971ccba Address:4cb66cc6-60ee-a159-1696-51e77971ccba}]"
2026-05-25T23:23:11.011Z [INFO]  server-35693531-07c5-b061-e560-7e9cb8397ed5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d951aaca-54eb-57a3-b2b9-08c207efab8a Address:d951aaca-54eb-57a3-b2b9-08c207efab8a} {Suffrage:Voter ID:server-35693531-07c5-b061-e560-7e9cb8397ed5 Address:35693531-07c5-b061-e560-7e9cb8397ed5} {Suffrage:Voter ID:server-4cb66cc6-60ee-a159-1696-51e77971ccba Address:4cb66cc6-60ee-a159-1696-51e77971ccba}]"
2026-05-25T23:23:11.011Z [INFO]  server-d951aaca-54eb-57a3-b2b9-08c207efab8a: entering follower state: follower="Node at d951aaca-54eb-57a3-b2b9-08c207efab8a [Follower]" leader-address= leader-id=
2026-05-25T23:23:11.011Z [INFO]  server-4cb66cc6-60ee-a159-1696-51e77971ccba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d951aaca-54eb-57a3-b2b9-08c207efab8a Address:d951aaca-54eb-57a3-b2b9-08c207efab8a} {Suffrage:Voter ID:server-35693531-07c5-b061-e560-7e9cb8397ed5 Address:35693531-07c5-b061-e560-7e9cb8397ed5} {Suffrage:Voter ID:server-4cb66cc6-60ee-a159-1696-51e77971ccba Address:4cb66cc6-60ee-a159-1696-51e77971ccba}]"
2026-05-25T23:23:11.011Z [INFO]  server-35693531-07c5-b061-e560-7e9cb8397ed5: entering follower state: follower="Node at 35693531-07c5-b061-e560-7e9cb8397ed5 [Follower]" leader-address= leader-id=
2026-05-25T23:23:11.011Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:11.011Z [INFO]  server-4cb66cc6-60ee-a159-1696-51e77971ccba: entering follower state: follower="Node at 4cb66cc6-60ee-a159-1696-51e77971ccba [Follower]" leader-address= leader-id=
2026-05-25T23:23:11.571Z [WARN]  server-35693531-07c5-b061-e560-7e9cb8397ed5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:11.571Z [INFO]  server-35693531-07c5-b061-e560-7e9cb8397ed5: entering candidate state: node="Node at 35693531-07c5-b061-e560-7e9cb8397ed5 [Candidate]" term=2
2026-05-25T23:23:11.571Z [INFO]  server-35693531-07c5-b061-e560-7e9cb8397ed5: election won: term=2 tally=2
2026-05-25T23:23:11.571Z [INFO]  server-35693531-07c5-b061-e560-7e9cb8397ed5: entering leader state: leader="Node at 35693531-07c5-b061-e560-7e9cb8397ed5 [Leader]"
2026-05-25T23:23:11.571Z [INFO]  server-35693531-07c5-b061-e560-7e9cb8397ed5: added peer, starting replication: peer=server-d951aaca-54eb-57a3-b2b9-08c207efab8a
2026-05-25T23:23:11.571Z [INFO]  server-35693531-07c5-b061-e560-7e9cb8397ed5: added peer, starting replication: peer=server-4cb66cc6-60ee-a159-1696-51e77971ccba
2026-05-25T23:23:11.571Z [INFO]  server-35693531-07c5-b061-e560-7e9cb8397ed5: pipelining replication: peer="{Voter server-4cb66cc6-60ee-a159-1696-51e77971ccba 4cb66cc6-60ee-a159-1696-51e77971ccba}"
2026-05-25T23:23:11.571Z [INFO]  server-35693531-07c5-b061-e560-7e9cb8397ed5: pipelining replication: peer="{Voter server-d951aaca-54eb-57a3-b2b9-08c207efab8a d951aaca-54eb-57a3-b2b9-08c207efab8a}"
2026-05-25T23:23:12.577Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:11.57184697 +0000 UTC m=+47.218294419 (1 nodes), 560.398561ms from start of poll, 560.644204ms from cluster start. Timeout at 2026-05-25 23:23:12.57685182 +0000 UTC m=+48.223299274, 1.005004855s after stability
2026-05-25T23:23:12.577Z [INFO]  server-35693531-07c5-b061-e560-7e9cb8397ed5: starting snapshot up to: index=12
2026-05-25T23:23:12.577Z [INFO]  creating new snapshot: path=/tmp/raft2999401087/snapshots/2-12-1779751392577.tmp
2026-05-25T23:23:12.582Z [INFO]  server-35693531-07c5-b061-e560-7e9cb8397ed5: snapshot complete up to: index=12
2026-05-25T23:23:12.583Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5aff39f7-caf5-0cca-b85b-cefd035d60f7 Address:5aff39f7-caf5-0cca-b85b-cefd035d60f7} {Suffrage:Voter ID:server-bb5ff54a-16e0-4125-6196-c4d8b3be7726 Address:bb5ff54a-16e0-4125-6196-c4d8b3be7726} {Suffrage:Voter ID:server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a Address:eedc65c4-92af-c6a4-c7ee-60bd0065ac4a}]"
2026-05-25T23:23:12.583Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: entering follower state: follower="Node at 5aff39f7-caf5-0cca-b85b-cefd035d60f7 [Follower]" leader-address= leader-id=
2026-05-25T23:23:12.583Z [INFO]  server-bb5ff54a-16e0-4125-6196-c4d8b3be7726: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5aff39f7-caf5-0cca-b85b-cefd035d60f7 Address:5aff39f7-caf5-0cca-b85b-cefd035d60f7} {Suffrage:Voter ID:server-bb5ff54a-16e0-4125-6196-c4d8b3be7726 Address:bb5ff54a-16e0-4125-6196-c4d8b3be7726} {Suffrage:Voter ID:server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a Address:eedc65c4-92af-c6a4-c7ee-60bd0065ac4a}]"
2026-05-25T23:23:12.583Z [INFO]  server-bb5ff54a-16e0-4125-6196-c4d8b3be7726: entering follower state: follower="Node at bb5ff54a-16e0-4125-6196-c4d8b3be7726 [Follower]" leader-address= leader-id=
2026-05-25T23:23:12.583Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5aff39f7-caf5-0cca-b85b-cefd035d60f7 Address:5aff39f7-caf5-0cca-b85b-cefd035d60f7} {Suffrage:Voter ID:server-bb5ff54a-16e0-4125-6196-c4d8b3be7726 Address:bb5ff54a-16e0-4125-6196-c4d8b3be7726} {Suffrage:Voter ID:server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a Address:eedc65c4-92af-c6a4-c7ee-60bd0065ac4a}]"
2026-05-25T23:23:12.583Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:12.583Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: entering follower state: follower="Node at eedc65c4-92af-c6a4-c7ee-60bd0065ac4a [Follower]" leader-address= leader-id=
2026-05-25T23:23:13.240Z [WARN]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:13.240Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: entering candidate state: node="Node at 5aff39f7-caf5-0cca-b85b-cefd035d60f7 [Candidate]" term=2
2026-05-25T23:23:13.240Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: election won: term=2 tally=2
2026-05-25T23:23:13.240Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: entering leader state: leader="Node at 5aff39f7-caf5-0cca-b85b-cefd035d60f7 [Leader]"
2026-05-25T23:23:13.240Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: added peer, starting replication: peer=server-bb5ff54a-16e0-4125-6196-c4d8b3be7726
2026-05-25T23:23:13.240Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: added peer, starting replication: peer=server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a
2026-05-25T23:23:13.240Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: pipelining replication: peer="{Voter server-bb5ff54a-16e0-4125-6196-c4d8b3be7726 bb5ff54a-16e0-4125-6196-c4d8b3be7726}"
2026-05-25T23:23:13.240Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: pipelining replication: peer="{Voter server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a eedc65c4-92af-c6a4-c7ee-60bd0065ac4a}"
2026-05-25T23:23:14.246Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:13.240678687 +0000 UTC m=+48.887126146 (1 nodes), 657.243571ms from start of poll, 657.583362ms from cluster start. Timeout at 2026-05-25 23:23:14.245682338 +0000 UTC m=+49.892129792, 1.005003646s after stability
2026-05-25T23:23:14.246Z [INFO]  creating new snapshot: path=/tmp/raft2525907470/snapshots/2-15-1779751394246.tmp
2026-05-25T23:23:14.407Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: copied to local snapshot: bytes=71
2026-05-25T23:23:14.407Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: snapshot restore progress: id=2-15-1779751394246 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:14.407Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: restored user snapshot: index=15
2026-05-25T23:23:14.407Z [ERROR] server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: failed to get log: index=3 error="log not found"
2026-05-25T23:23:14.407Z [ERROR] server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: failed to get log: index=3 error="log not found"
2026-05-25T23:23:14.407Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: aborting pipeline replication: peer="{Voter server-bb5ff54a-16e0-4125-6196-c4d8b3be7726 bb5ff54a-16e0-4125-6196-c4d8b3be7726}"
2026-05-25T23:23:14.407Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: aborting pipeline replication: peer="{Voter server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a eedc65c4-92af-c6a4-c7ee-60bd0065ac4a}"
2026-05-25T23:23:14.416Z [ERROR] server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: failed to get log: index=3 error="log not found"
2026-05-25T23:23:14.416Z [ERROR] server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: failed to get log: index=3 error="log not found"
2026-05-25T23:23:14.416Z [INFO]  creating new snapshot: path=/tmp/raft680358630/snapshots/2-15-1779751394416.tmp
2026-05-25T23:23:14.470Z [INFO]  creating new snapshot: path=/tmp/raft968340988/snapshots/2-15-1779751394470.tmp
2026-05-25T23:23:14.470Z [INFO]  server-bb5ff54a-16e0-4125-6196-c4d8b3be7726: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:14.470Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:14.471Z [INFO]  server-bb5ff54a-16e0-4125-6196-c4d8b3be7726: copied to local snapshot: bytes=71
2026-05-25T23:23:14.471Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: copied to local snapshot: bytes=71
2026-05-25T23:23:14.471Z [INFO]  server-bb5ff54a-16e0-4125-6196-c4d8b3be7726: snapshot restore progress: id=2-15-1779751394416 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:14.471Z [INFO]  server-bb5ff54a-16e0-4125-6196-c4d8b3be7726: Installed remote snapshot
2026-05-25T23:23:14.471Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: pipelining replication: peer="{Voter server-bb5ff54a-16e0-4125-6196-c4d8b3be7726 bb5ff54a-16e0-4125-6196-c4d8b3be7726}"
2026-05-25T23:23:14.471Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:23:14.471Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: snapshot restore progress: id=2-15-1779751394470 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:14.471Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: Installed remote snapshot
2026-05-25T23:23:14.471Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: pipelining replication: peer="{Voter server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a eedc65c4-92af-c6a4-c7ee-60bd0065ac4a}"
2026-05-25T23:23:15.477Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:23:14.471509165 +0000 UTC m=+50.117956615 (2 nodes), 987ns from start of poll, 1.888413831s from cluster start. Timeout at 2026-05-25 23:23:15.476516416 +0000 UTC m=+51.122963906, 1.005007291s after stability
2026-05-25T23:23:15.489Z [INFO]  server-35693531-07c5-b061-e560-7e9cb8397ed5: aborting pipeline replication: peer="{Voter server-d951aaca-54eb-57a3-b2b9-08c207efab8a d951aaca-54eb-57a3-b2b9-08c207efab8a}"
2026-05-25T23:23:15.489Z [INFO]  server-35693531-07c5-b061-e560-7e9cb8397ed5: aborting pipeline replication: peer="{Voter server-4cb66cc6-60ee-a159-1696-51e77971ccba 4cb66cc6-60ee-a159-1696-51e77971ccba}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true
2026-05-25T23:23:15.524Z [INFO]  server-89b2a2a4-c7a8-1e28-5025-bd58052f1fc4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89b2a2a4-c7a8-1e28-5025-bd58052f1fc4 Address:89b2a2a4-c7a8-1e28-5025-bd58052f1fc4} {Suffrage:Voter ID:server-4a37af58-1acd-15f7-ffc0-894eccb946ce Address:4a37af58-1acd-15f7-ffc0-894eccb946ce} {Suffrage:Voter ID:server-42c02025-4853-b952-6a01-5b11166e4c8b Address:42c02025-4853-b952-6a01-5b11166e4c8b}]"
2026-05-25T23:23:15.524Z [INFO]  server-89b2a2a4-c7a8-1e28-5025-bd58052f1fc4: entering follower state: follower="Node at 89b2a2a4-c7a8-1e28-5025-bd58052f1fc4 [Follower]" leader-address= leader-id=
2026-05-25T23:23:15.524Z [INFO]  server-4a37af58-1acd-15f7-ffc0-894eccb946ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89b2a2a4-c7a8-1e28-5025-bd58052f1fc4 Address:89b2a2a4-c7a8-1e28-5025-bd58052f1fc4} {Suffrage:Voter ID:server-4a37af58-1acd-15f7-ffc0-894eccb946ce Address:4a37af58-1acd-15f7-ffc0-894eccb946ce} {Suffrage:Voter ID:server-42c02025-4853-b952-6a01-5b11166e4c8b Address:42c02025-4853-b952-6a01-5b11166e4c8b}]"
2026-05-25T23:23:15.524Z [INFO]  server-4a37af58-1acd-15f7-ffc0-894eccb946ce: entering follower state: follower="Node at 4a37af58-1acd-15f7-ffc0-894eccb946ce [Follower]" leader-address= leader-id=
2026-05-25T23:23:15.524Z [INFO]  server-42c02025-4853-b952-6a01-5b11166e4c8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89b2a2a4-c7a8-1e28-5025-bd58052f1fc4 Address:89b2a2a4-c7a8-1e28-5025-bd58052f1fc4} {Suffrage:Voter ID:server-4a37af58-1acd-15f7-ffc0-894eccb946ce Address:4a37af58-1acd-15f7-ffc0-894eccb946ce} {Suffrage:Voter ID:server-42c02025-4853-b952-6a01-5b11166e4c8b Address:42c02025-4853-b952-6a01-5b11166e4c8b}]"
2026-05-25T23:23:15.524Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:15.524Z [INFO]  server-42c02025-4853-b952-6a01-5b11166e4c8b: entering follower state: follower="Node at 42c02025-4853-b952-6a01-5b11166e4c8b [Follower]" leader-address= leader-id=
2026-05-25T23:23:16.228Z [WARN]  server-42c02025-4853-b952-6a01-5b11166e4c8b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:16.228Z [INFO]  server-42c02025-4853-b952-6a01-5b11166e4c8b: entering candidate state: node="Node at 42c02025-4853-b952-6a01-5b11166e4c8b [Candidate]" term=2
2026-05-25T23:23:16.229Z [INFO]  server-42c02025-4853-b952-6a01-5b11166e4c8b: election won: term=2 tally=2
2026-05-25T23:23:16.229Z [INFO]  server-42c02025-4853-b952-6a01-5b11166e4c8b: entering leader state: leader="Node at 42c02025-4853-b952-6a01-5b11166e4c8b [Leader]"
2026-05-25T23:23:16.229Z [INFO]  server-42c02025-4853-b952-6a01-5b11166e4c8b: added peer, starting replication: peer=server-89b2a2a4-c7a8-1e28-5025-bd58052f1fc4
2026-05-25T23:23:16.229Z [INFO]  server-42c02025-4853-b952-6a01-5b11166e4c8b: added peer, starting replication: peer=server-4a37af58-1acd-15f7-ffc0-894eccb946ce
2026-05-25T23:23:16.229Z [INFO]  server-42c02025-4853-b952-6a01-5b11166e4c8b: pipelining replication: peer="{Voter server-4a37af58-1acd-15f7-ffc0-894eccb946ce 4a37af58-1acd-15f7-ffc0-894eccb946ce}"
2026-05-25T23:23:16.229Z [INFO]  server-42c02025-4853-b952-6a01-5b11166e4c8b: pipelining replication: peer="{Voter server-89b2a2a4-c7a8-1e28-5025-bd58052f1fc4 89b2a2a4-c7a8-1e28-5025-bd58052f1fc4}"
2026-05-25T23:23:17.234Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:16.229171955 +0000 UTC m=+51.875619482 (1 nodes), 704.472852ms from start of poll, 704.756901ms from cluster start. Timeout at 2026-05-25 23:23:17.234177474 +0000 UTC m=+52.880624965, 1.005005483s after stability
2026-05-25T23:23:17.234Z [INFO]  server-42c02025-4853-b952-6a01-5b11166e4c8b: starting snapshot up to: index=12
2026-05-25T23:23:17.234Z [INFO]  creating new snapshot: path=/tmp/raft833559227/snapshots/2-12-1779751397234.tmp
2026-05-25T23:23:17.235Z [INFO]  server-42c02025-4853-b952-6a01-5b11166e4c8b: snapshot complete up to: index=12
2026-05-25T23:23:17.236Z [INFO]  server-ae4d03f3-6164-5c59-99d4-455db47640e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae4d03f3-6164-5c59-99d4-455db47640e4 Address:ae4d03f3-6164-5c59-99d4-455db47640e4} {Suffrage:Voter ID:server-8cdddb1f-717d-965c-d545-807129b2ce3b Address:8cdddb1f-717d-965c-d545-807129b2ce3b} {Suffrage:Voter ID:server-b2d831eb-15f8-614f-ec08-fafe997c4944 Address:b2d831eb-15f8-614f-ec08-fafe997c4944}]"
2026-05-25T23:23:17.236Z [INFO]  server-ae4d03f3-6164-5c59-99d4-455db47640e4: entering follower state: follower="Node at ae4d03f3-6164-5c59-99d4-455db47640e4 [Follower]" leader-address= leader-id=
2026-05-25T23:23:17.236Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae4d03f3-6164-5c59-99d4-455db47640e4 Address:ae4d03f3-6164-5c59-99d4-455db47640e4} {Suffrage:Voter ID:server-8cdddb1f-717d-965c-d545-807129b2ce3b Address:8cdddb1f-717d-965c-d545-807129b2ce3b} {Suffrage:Voter ID:server-b2d831eb-15f8-614f-ec08-fafe997c4944 Address:b2d831eb-15f8-614f-ec08-fafe997c4944}]"
2026-05-25T23:23:17.236Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: entering follower state: follower="Node at 8cdddb1f-717d-965c-d545-807129b2ce3b [Follower]" leader-address= leader-id=
2026-05-25T23:23:17.236Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae4d03f3-6164-5c59-99d4-455db47640e4 Address:ae4d03f3-6164-5c59-99d4-455db47640e4} {Suffrage:Voter ID:server-8cdddb1f-717d-965c-d545-807129b2ce3b Address:8cdddb1f-717d-965c-d545-807129b2ce3b} {Suffrage:Voter ID:server-b2d831eb-15f8-614f-ec08-fafe997c4944 Address:b2d831eb-15f8-614f-ec08-fafe997c4944}]"
2026-05-25T23:23:17.236Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:17.236Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: entering follower state: follower="Node at b2d831eb-15f8-614f-ec08-fafe997c4944 [Follower]" leader-address= leader-id=
2026-05-25T23:23:17.902Z [WARN]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:17.902Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: entering candidate state: node="Node at b2d831eb-15f8-614f-ec08-fafe997c4944 [Candidate]" term=2
2026-05-25T23:23:17.902Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: election won: term=2 tally=2
2026-05-25T23:23:17.903Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: entering leader state: leader="Node at b2d831eb-15f8-614f-ec08-fafe997c4944 [Leader]"
2026-05-25T23:23:17.903Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: added peer, starting replication: peer=server-ae4d03f3-6164-5c59-99d4-455db47640e4
2026-05-25T23:23:17.903Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: added peer, starting replication: peer=server-8cdddb1f-717d-965c-d545-807129b2ce3b
2026-05-25T23:23:17.903Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: pipelining replication: peer="{Voter server-ae4d03f3-6164-5c59-99d4-455db47640e4 ae4d03f3-6164-5c59-99d4-455db47640e4}"
2026-05-25T23:23:17.903Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: pipelining replication: peer="{Voter server-8cdddb1f-717d-965c-d545-807129b2ce3b 8cdddb1f-717d-965c-d545-807129b2ce3b}"
2026-05-25T23:23:18.908Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:17.903024557 +0000 UTC m=+53.549472027 (1 nodes), 666.517405ms from start of poll, 666.86942ms from cluster start. Timeout at 2026-05-25 23:23:18.90803237 +0000 UTC m=+54.554479823, 1.005007796s after stability
2026-05-25T23:23:18.908Z [INFO]  creating new snapshot: path=/tmp/raft1440176519/snapshots/2-15-1779751398908.tmp
2026-05-25T23:23:18.909Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: copied to local snapshot: bytes=71
2026-05-25T23:23:18.909Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: snapshot restore progress: id=2-15-1779751398908 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:18.909Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: removing all old logs from log store
2026-05-25T23:23:18.909Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: compacting logs: from=1 to=2
2026-05-25T23:23:18.909Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: restored user snapshot: index=15
2026-05-25T23:23:18.909Z [ERROR] server-b2d831eb-15f8-614f-ec08-fafe997c4944: failed to get log: index=2 error="log not found"
2026-05-25T23:23:18.909Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: aborting pipeline replication: peer="{Voter server-8cdddb1f-717d-965c-d545-807129b2ce3b 8cdddb1f-717d-965c-d545-807129b2ce3b}"
2026-05-25T23:23:18.909Z [ERROR] server-b2d831eb-15f8-614f-ec08-fafe997c4944: failed to get log: index=2 error="log not found"
2026-05-25T23:23:18.909Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: aborting pipeline replication: peer="{Voter server-ae4d03f3-6164-5c59-99d4-455db47640e4 ae4d03f3-6164-5c59-99d4-455db47640e4}"
2026-05-25T23:23:18.917Z [ERROR] server-b2d831eb-15f8-614f-ec08-fafe997c4944: failed to get log: index=2 error="log not found"
2026-05-25T23:23:18.917Z [INFO]  creating new snapshot: path=/tmp/raft2490682075/snapshots/2-15-1779751398917.tmp
2026-05-25T23:23:18.917Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:18.918Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: copied to local snapshot: bytes=71
2026-05-25T23:23:18.918Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: snapshot restore progress: id=2-15-1779751398917 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:18.918Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: removing all old logs from log store
2026-05-25T23:23:18.918Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: compacting logs: from=1 to=2
2026-05-25T23:23:18.918Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: Installed remote snapshot
2026-05-25T23:23:18.918Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: pipelining replication: peer="{Voter server-8cdddb1f-717d-965c-d545-807129b2ce3b 8cdddb1f-717d-965c-d545-807129b2ce3b}"
2026-05-25T23:23:18.918Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:23:18.919Z [ERROR] server-b2d831eb-15f8-614f-ec08-fafe997c4944: failed to get log: index=2 error="log not found"
2026-05-25T23:23:18.919Z [INFO]  creating new snapshot: path=/tmp/raft2777673433/snapshots/2-15-1779751398919.tmp
2026-05-25T23:23:18.921Z [INFO]  server-ae4d03f3-6164-5c59-99d4-455db47640e4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:18.921Z [INFO]  server-ae4d03f3-6164-5c59-99d4-455db47640e4: copied to local snapshot: bytes=71
2026-05-25T23:23:18.921Z [INFO]  server-ae4d03f3-6164-5c59-99d4-455db47640e4: snapshot restore progress: id=2-15-1779751398919 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:18.921Z [INFO]  server-ae4d03f3-6164-5c59-99d4-455db47640e4: removing all old logs from log store
2026-05-25T23:23:18.921Z [INFO]  server-ae4d03f3-6164-5c59-99d4-455db47640e4: compacting logs: from=1 to=2
2026-05-25T23:23:18.921Z [INFO]  server-ae4d03f3-6164-5c59-99d4-455db47640e4: Installed remote snapshot
2026-05-25T23:23:18.921Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: pipelining replication: peer="{Voter server-ae4d03f3-6164-5c59-99d4-455db47640e4 ae4d03f3-6164-5c59-99d4-455db47640e4}"
2026-05-25T23:23:19.924Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:23:18.918480762 +0000 UTC m=+54.564928212 (2 nodes), 827ns from start of poll, 1.682325605s from cluster start. Timeout at 2026-05-25 23:23:19.923498522 +0000 UTC m=+55.569946013, 1.005017801s after stability
2026-05-25T23:23:19.937Z [INFO]  server-42c02025-4853-b952-6a01-5b11166e4c8b: aborting pipeline replication: peer="{Voter server-4a37af58-1acd-15f7-ffc0-894eccb946ce 4a37af58-1acd-15f7-ffc0-894eccb946ce}"
2026-05-25T23:23:19.937Z [INFO]  server-42c02025-4853-b952-6a01-5b11166e4c8b: aborting pipeline replication: peer="{Voter server-89b2a2a4-c7a8-1e28-5025-bd58052f1fc4 89b2a2a4-c7a8-1e28-5025-bd58052f1fc4}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false
2026-05-25T23:23:19.938Z [INFO]  server-ed2a4713-afae-ca91-702a-ea0394448320: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed2a4713-afae-ca91-702a-ea0394448320 Address:ed2a4713-afae-ca91-702a-ea0394448320} {Suffrage:Voter ID:server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6 Address:de7a7a13-0faa-d6ee-8df1-ef5c2713abd6} {Suffrage:Voter ID:server-681dfd91-d4ac-d2bd-b809-e33fd97d99a8 Address:681dfd91-d4ac-d2bd-b809-e33fd97d99a8}]"
2026-05-25T23:23:19.938Z [INFO]  server-ed2a4713-afae-ca91-702a-ea0394448320: entering follower state: follower="Node at ed2a4713-afae-ca91-702a-ea0394448320 [Follower]" leader-address= leader-id=
2026-05-25T23:23:19.938Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed2a4713-afae-ca91-702a-ea0394448320 Address:ed2a4713-afae-ca91-702a-ea0394448320} {Suffrage:Voter ID:server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6 Address:de7a7a13-0faa-d6ee-8df1-ef5c2713abd6} {Suffrage:Voter ID:server-681dfd91-d4ac-d2bd-b809-e33fd97d99a8 Address:681dfd91-d4ac-d2bd-b809-e33fd97d99a8}]"
2026-05-25T23:23:19.938Z [INFO]  server-681dfd91-d4ac-d2bd-b809-e33fd97d99a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed2a4713-afae-ca91-702a-ea0394448320 Address:ed2a4713-afae-ca91-702a-ea0394448320} {Suffrage:Voter ID:server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6 Address:de7a7a13-0faa-d6ee-8df1-ef5c2713abd6} {Suffrage:Voter ID:server-681dfd91-d4ac-d2bd-b809-e33fd97d99a8 Address:681dfd91-d4ac-d2bd-b809-e33fd97d99a8}]"
2026-05-25T23:23:19.938Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:19.938Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: entering follower state: follower="Node at de7a7a13-0faa-d6ee-8df1-ef5c2713abd6 [Follower]" leader-address= leader-id=
2026-05-25T23:23:19.938Z [INFO]  server-681dfd91-d4ac-d2bd-b809-e33fd97d99a8: entering follower state: follower="Node at 681dfd91-d4ac-d2bd-b809-e33fd97d99a8 [Follower]" leader-address= leader-id=
2026-05-25T23:23:20.730Z [WARN]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:20.730Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: entering candidate state: node="Node at de7a7a13-0faa-d6ee-8df1-ef5c2713abd6 [Candidate]" term=2
2026-05-25T23:23:20.730Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: election won: term=2 tally=2
2026-05-25T23:23:20.731Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: entering leader state: leader="Node at de7a7a13-0faa-d6ee-8df1-ef5c2713abd6 [Leader]"
2026-05-25T23:23:20.731Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: added peer, starting replication: peer=server-ed2a4713-afae-ca91-702a-ea0394448320
2026-05-25T23:23:20.731Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: added peer, starting replication: peer=server-681dfd91-d4ac-d2bd-b809-e33fd97d99a8
2026-05-25T23:23:20.731Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: pipelining replication: peer="{Voter server-ed2a4713-afae-ca91-702a-ea0394448320 ed2a4713-afae-ca91-702a-ea0394448320}"
2026-05-25T23:23:20.731Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: pipelining replication: peer="{Voter server-681dfd91-d4ac-d2bd-b809-e33fd97d99a8 681dfd91-d4ac-d2bd-b809-e33fd97d99a8}"
2026-05-25T23:23:21.736Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:20.73105955 +0000 UTC m=+56.377507037 (1 nodes), 792.428365ms from start of poll, 792.728221ms from cluster start. Timeout at 2026-05-25 23:23:21.736127618 +0000 UTC m=+57.382575069, 1.005068032s after stability
2026-05-25T23:23:21.736Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: starting snapshot up to: index=12
2026-05-25T23:23:21.736Z [INFO]  creating new snapshot: path=/tmp/raft1527442459/snapshots/2-12-1779751401736.tmp
2026-05-25T23:23:21.809Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: snapshot complete up to: index=12
2026-05-25T23:23:21.810Z [INFO]  creating new snapshot: path=/tmp/raft1527442459/snapshots/2-113-1779751401810.tmp
2026-05-25T23:23:21.810Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: copied to local snapshot: bytes=71
2026-05-25T23:23:21.810Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: snapshot restore progress: id=2-113-1779751401810 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:21.810Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: restored user snapshot: index=113
2026-05-25T23:23:21.810Z [ERROR] server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: failed to get log: index=23 error="log not found"
2026-05-25T23:23:21.810Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: aborting pipeline replication: peer="{Voter server-681dfd91-d4ac-d2bd-b809-e33fd97d99a8 681dfd91-d4ac-d2bd-b809-e33fd97d99a8}"
2026-05-25T23:23:21.810Z [ERROR] server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: failed to get log: index=23 error="log not found"
2026-05-25T23:23:21.810Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: aborting pipeline replication: peer="{Voter server-ed2a4713-afae-ca91-702a-ea0394448320 ed2a4713-afae-ca91-702a-ea0394448320}"
2026-05-25T23:23:21.818Z [ERROR] server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: failed to get log: index=23 error="log not found"
2026-05-25T23:23:21.818Z [INFO]  creating new snapshot: path=/tmp/raft2814939928/snapshots/2-113-1779751401818.tmp
2026-05-25T23:23:21.818Z [INFO]  server-681dfd91-d4ac-d2bd-b809-e33fd97d99a8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:21.819Z [ERROR] server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: failed to get log: index=23 error="log not found"
2026-05-25T23:23:21.819Z [INFO]  creating new snapshot: path=/tmp/raft3501646284/snapshots/2-113-1779751401819.tmp
2026-05-25T23:23:21.820Z [INFO]  server-ed2a4713-afae-ca91-702a-ea0394448320: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:21.940Z [INFO]  server-681dfd91-d4ac-d2bd-b809-e33fd97d99a8: copied to local snapshot: bytes=71
2026-05-25T23:23:21.940Z [INFO]  server-ed2a4713-afae-ca91-702a-ea0394448320: copied to local snapshot: bytes=71
2026-05-25T23:23:21.940Z [INFO]  server-681dfd91-d4ac-d2bd-b809-e33fd97d99a8: snapshot restore progress: id=2-113-1779751401818 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:21.940Z [INFO]  server-681dfd91-d4ac-d2bd-b809-e33fd97d99a8: Installed remote snapshot
2026-05-25T23:23:21.940Z [WARN]  server-681dfd91-d4ac-d2bd-b809-e33fd97d99a8: skipping application of old log: index=22
2026-05-25T23:23:21.940Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: pipelining replication: peer="{Voter server-681dfd91-d4ac-d2bd-b809-e33fd97d99a8 681dfd91-d4ac-d2bd-b809-e33fd97d99a8}"
2026-05-25T23:23:21.940Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:23:21.940Z [INFO]  server-ed2a4713-afae-ca91-702a-ea0394448320: snapshot restore progress: id=2-113-1779751401819 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:21.940Z [INFO]  server-ed2a4713-afae-ca91-702a-ea0394448320: Installed remote snapshot
2026-05-25T23:23:21.941Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: pipelining replication: peer="{Voter server-ed2a4713-afae-ca91-702a-ea0394448320 ed2a4713-afae-ca91-702a-ea0394448320}"
2026-05-25T23:23:22.946Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:23:21.940866792 +0000 UTC m=+57.587314260 (2 nodes), 938ns from start of poll, 2.002535444s from cluster start. Timeout at 2026-05-25 23:23:22.945874821 +0000 UTC m=+58.592322308, 1.005008048s after stability
2026-05-25T23:23:22.958Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: aborting pipeline replication: peer="{Voter server-681dfd91-d4ac-d2bd-b809-e33fd97d99a8 681dfd91-d4ac-d2bd-b809-e33fd97d99a8}"
2026-05-25T23:23:22.958Z [INFO]  server-de7a7a13-0faa-d6ee-8df1-ef5c2713abd6: aborting pipeline replication: peer="{Voter server-ed2a4713-afae-ca91-702a-ea0394448320 ed2a4713-afae-ca91-702a-ea0394448320}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false
2026-05-25T23:23:22.959Z [INFO]  server-740e09dd-eaea-e375-c6b5-a0625b1f1481: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-740e09dd-eaea-e375-c6b5-a0625b1f1481 Address:740e09dd-eaea-e375-c6b5-a0625b1f1481} {Suffrage:Voter ID:server-c2a113ce-ada8-5fb4-e260-b69af18ee98c Address:c2a113ce-ada8-5fb4-e260-b69af18ee98c} {Suffrage:Voter ID:server-7581e953-ad63-4973-3240-bf170ede2366 Address:7581e953-ad63-4973-3240-bf170ede2366}]"
2026-05-25T23:23:22.959Z [INFO]  server-740e09dd-eaea-e375-c6b5-a0625b1f1481: entering follower state: follower="Node at 740e09dd-eaea-e375-c6b5-a0625b1f1481 [Follower]" leader-address= leader-id=
2026-05-25T23:23:22.959Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-740e09dd-eaea-e375-c6b5-a0625b1f1481 Address:740e09dd-eaea-e375-c6b5-a0625b1f1481} {Suffrage:Voter ID:server-c2a113ce-ada8-5fb4-e260-b69af18ee98c Address:c2a113ce-ada8-5fb4-e260-b69af18ee98c} {Suffrage:Voter ID:server-7581e953-ad63-4973-3240-bf170ede2366 Address:7581e953-ad63-4973-3240-bf170ede2366}]"
2026-05-25T23:23:22.960Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: entering follower state: follower="Node at c2a113ce-ada8-5fb4-e260-b69af18ee98c [Follower]" leader-address= leader-id=
2026-05-25T23:23:22.960Z [INFO]  server-7581e953-ad63-4973-3240-bf170ede2366: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-740e09dd-eaea-e375-c6b5-a0625b1f1481 Address:740e09dd-eaea-e375-c6b5-a0625b1f1481} {Suffrage:Voter ID:server-c2a113ce-ada8-5fb4-e260-b69af18ee98c Address:c2a113ce-ada8-5fb4-e260-b69af18ee98c} {Suffrage:Voter ID:server-7581e953-ad63-4973-3240-bf170ede2366 Address:7581e953-ad63-4973-3240-bf170ede2366}]"
2026-05-25T23:23:22.960Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:22.960Z [INFO]  server-7581e953-ad63-4973-3240-bf170ede2366: entering follower state: follower="Node at 7581e953-ad63-4973-3240-bf170ede2366 [Follower]" leader-address= leader-id=
2026-05-25T23:23:23.563Z [WARN]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:23.563Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: entering candidate state: node="Node at c2a113ce-ada8-5fb4-e260-b69af18ee98c [Candidate]" term=2
2026-05-25T23:23:23.565Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: election won: term=2 tally=2
2026-05-25T23:23:23.565Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: entering leader state: leader="Node at c2a113ce-ada8-5fb4-e260-b69af18ee98c [Leader]"
2026-05-25T23:23:23.565Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: added peer, starting replication: peer=server-740e09dd-eaea-e375-c6b5-a0625b1f1481
2026-05-25T23:23:23.565Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: added peer, starting replication: peer=server-7581e953-ad63-4973-3240-bf170ede2366
2026-05-25T23:23:23.565Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: pipelining replication: peer="{Voter server-7581e953-ad63-4973-3240-bf170ede2366 7581e953-ad63-4973-3240-bf170ede2366}"
2026-05-25T23:23:23.565Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: pipelining replication: peer="{Voter server-740e09dd-eaea-e375-c6b5-a0625b1f1481 740e09dd-eaea-e375-c6b5-a0625b1f1481}"
2026-05-25T23:23:24.571Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:23.565413786 +0000 UTC m=+59.211861235 (1 nodes), 605.300402ms from start of poll, 605.62867ms from cluster start. Timeout at 2026-05-25 23:23:24.570417634 +0000 UTC m=+60.216865088, 1.005003853s after stability
2026-05-25T23:23:24.571Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: starting snapshot up to: index=12
2026-05-25T23:23:24.571Z [INFO]  creating new snapshot: path=/tmp/raft2650444255/snapshots/2-12-1779751404571.tmp
2026-05-25T23:23:24.600Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: snapshot complete up to: index=12
2026-05-25T23:23:24.600Z [INFO]  creating new snapshot: path=/tmp/raft2650444255/snapshots/2-113-1779751404600.tmp
2026-05-25T23:23:24.601Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: copied to local snapshot: bytes=71
2026-05-25T23:23:24.601Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: snapshot restore progress: id=2-113-1779751404600 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:24.601Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: removing all old logs from log store
2026-05-25T23:23:24.601Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: compacting logs: from=1 to=22
2026-05-25T23:23:24.601Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: restored user snapshot: index=113
2026-05-25T23:23:24.601Z [ERROR] server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: failed to get log: index=22 error="log not found"
2026-05-25T23:23:24.601Z [ERROR] server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: failed to get log: index=22 error="log not found"
2026-05-25T23:23:24.601Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: aborting pipeline replication: peer="{Voter server-740e09dd-eaea-e375-c6b5-a0625b1f1481 740e09dd-eaea-e375-c6b5-a0625b1f1481}"
2026-05-25T23:23:24.601Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: aborting pipeline replication: peer="{Voter server-7581e953-ad63-4973-3240-bf170ede2366 7581e953-ad63-4973-3240-bf170ede2366}"
2026-05-25T23:23:24.609Z [ERROR] server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: failed to get log: index=22 error="log not found"
2026-05-25T23:23:24.609Z [INFO]  creating new snapshot: path=/tmp/raft3097045507/snapshots/2-113-1779751404609.tmp
2026-05-25T23:23:24.609Z [INFO]  server-740e09dd-eaea-e375-c6b5-a0625b1f1481: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:24.612Z [ERROR] server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: failed to get log: index=22 error="log not found"
2026-05-25T23:23:24.612Z [INFO]  creating new snapshot: path=/tmp/raft2570923756/snapshots/2-113-1779751404612.tmp
2026-05-25T23:23:24.613Z [INFO]  server-7581e953-ad63-4973-3240-bf170ede2366: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:24.759Z [INFO]  server-7581e953-ad63-4973-3240-bf170ede2366: copied to local snapshot: bytes=71
2026-05-25T23:23:24.759Z [INFO]  server-740e09dd-eaea-e375-c6b5-a0625b1f1481: copied to local snapshot: bytes=71
2026-05-25T23:23:24.759Z [INFO]  server-740e09dd-eaea-e375-c6b5-a0625b1f1481: snapshot restore progress: id=2-113-1779751404609 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:24.759Z [INFO]  server-740e09dd-eaea-e375-c6b5-a0625b1f1481: removing all old logs from log store
2026-05-25T23:23:24.759Z [INFO]  server-740e09dd-eaea-e375-c6b5-a0625b1f1481: compacting logs: from=1 to=22
2026-05-25T23:23:24.759Z [INFO]  server-7581e953-ad63-4973-3240-bf170ede2366: snapshot restore progress: id=2-113-1779751404612 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:24.759Z [INFO]  server-740e09dd-eaea-e375-c6b5-a0625b1f1481: Installed remote snapshot
2026-05-25T23:23:24.759Z [INFO]  server-7581e953-ad63-4973-3240-bf170ede2366: removing all old logs from log store
2026-05-25T23:23:24.759Z [INFO]  server-7581e953-ad63-4973-3240-bf170ede2366: compacting logs: from=1 to=22
2026-05-25T23:23:24.759Z [INFO]  server-7581e953-ad63-4973-3240-bf170ede2366: Installed remote snapshot
2026-05-25T23:23:24.759Z [WARN]  server-740e09dd-eaea-e375-c6b5-a0625b1f1481: skipping application of old log: index=22
2026-05-25T23:23:24.759Z [WARN]  server-7581e953-ad63-4973-3240-bf170ede2366: skipping application of old log: index=113
2026-05-25T23:23:24.759Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: pipelining replication: peer="{Voter server-740e09dd-eaea-e375-c6b5-a0625b1f1481 740e09dd-eaea-e375-c6b5-a0625b1f1481}"
2026-05-25T23:23:24.759Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: pipelining replication: peer="{Voter server-7581e953-ad63-4973-3240-bf170ede2366 7581e953-ad63-4973-3240-bf170ede2366}"
2026-05-25T23:23:24.759Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:23:25.765Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:23:24.759536241 +0000 UTC m=+60.405983705 (2 nodes), 767ns from start of poll, 1.79975114s from cluster start. Timeout at 2026-05-25 23:23:25.76454225 +0000 UTC m=+61.410989704, 1.005005999s after stability
2026-05-25T23:23:25.777Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: aborting pipeline replication: peer="{Voter server-7581e953-ad63-4973-3240-bf170ede2366 7581e953-ad63-4973-3240-bf170ede2366}"
2026-05-25T23:23:25.777Z [INFO]  server-c2a113ce-ada8-5fb4-e260-b69af18ee98c: aborting pipeline replication: peer="{Voter server-740e09dd-eaea-e375-c6b5-a0625b1f1481 740e09dd-eaea-e375-c6b5-a0625b1f1481}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true
2026-05-25T23:23:25.778Z [INFO]  server-a13cabfe-f7b8-22a1-0d36-6c7cac15dc2b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a13cabfe-f7b8-22a1-0d36-6c7cac15dc2b Address:a13cabfe-f7b8-22a1-0d36-6c7cac15dc2b} {Suffrage:Voter ID:server-80b053d2-4cde-6f6d-16e6-aaa30e2cd65e Address:80b053d2-4cde-6f6d-16e6-aaa30e2cd65e} {Suffrage:Voter ID:server-bb8c13b3-149d-8ac0-16e9-adc727603e91 Address:bb8c13b3-149d-8ac0-16e9-adc727603e91}]"
2026-05-25T23:23:25.778Z [INFO]  server-a13cabfe-f7b8-22a1-0d36-6c7cac15dc2b: entering follower state: follower="Node at a13cabfe-f7b8-22a1-0d36-6c7cac15dc2b [Follower]" leader-address= leader-id=
2026-05-25T23:23:25.778Z [INFO]  server-80b053d2-4cde-6f6d-16e6-aaa30e2cd65e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a13cabfe-f7b8-22a1-0d36-6c7cac15dc2b Address:a13cabfe-f7b8-22a1-0d36-6c7cac15dc2b} {Suffrage:Voter ID:server-80b053d2-4cde-6f6d-16e6-aaa30e2cd65e Address:80b053d2-4cde-6f6d-16e6-aaa30e2cd65e} {Suffrage:Voter ID:server-bb8c13b3-149d-8ac0-16e9-adc727603e91 Address:bb8c13b3-149d-8ac0-16e9-adc727603e91}]"
2026-05-25T23:23:25.778Z [INFO]  server-80b053d2-4cde-6f6d-16e6-aaa30e2cd65e: entering follower state: follower="Node at 80b053d2-4cde-6f6d-16e6-aaa30e2cd65e [Follower]" leader-address= leader-id=
2026-05-25T23:23:25.778Z [INFO]  server-bb8c13b3-149d-8ac0-16e9-adc727603e91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a13cabfe-f7b8-22a1-0d36-6c7cac15dc2b Address:a13cabfe-f7b8-22a1-0d36-6c7cac15dc2b} {Suffrage:Voter ID:server-80b053d2-4cde-6f6d-16e6-aaa30e2cd65e Address:80b053d2-4cde-6f6d-16e6-aaa30e2cd65e} {Suffrage:Voter ID:server-bb8c13b3-149d-8ac0-16e9-adc727603e91 Address:bb8c13b3-149d-8ac0-16e9-adc727603e91}]"
2026-05-25T23:23:25.778Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:25.778Z [INFO]  server-bb8c13b3-149d-8ac0-16e9-adc727603e91: entering follower state: follower="Node at bb8c13b3-149d-8ac0-16e9-adc727603e91 [Follower]" leader-address= leader-id=
2026-05-25T23:23:26.310Z [WARN]  server-80b053d2-4cde-6f6d-16e6-aaa30e2cd65e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:26.310Z [INFO]  server-80b053d2-4cde-6f6d-16e6-aaa30e2cd65e: entering candidate state: node="Node at 80b053d2-4cde-6f6d-16e6-aaa30e2cd65e [Candidate]" term=2
2026-05-25T23:23:26.310Z [INFO]  server-80b053d2-4cde-6f6d-16e6-aaa30e2cd65e: election won: term=2 tally=2
2026-05-25T23:23:26.310Z [INFO]  server-80b053d2-4cde-6f6d-16e6-aaa30e2cd65e: entering leader state: leader="Node at 80b053d2-4cde-6f6d-16e6-aaa30e2cd65e [Leader]"
2026-05-25T23:23:26.310Z [INFO]  server-80b053d2-4cde-6f6d-16e6-aaa30e2cd65e: added peer, starting replication: peer=server-a13cabfe-f7b8-22a1-0d36-6c7cac15dc2b
2026-05-25T23:23:26.310Z [INFO]  server-80b053d2-4cde-6f6d-16e6-aaa30e2cd65e: added peer, starting replication: peer=server-bb8c13b3-149d-8ac0-16e9-adc727603e91
2026-05-25T23:23:26.310Z [INFO]  server-80b053d2-4cde-6f6d-16e6-aaa30e2cd65e: pipelining replication: peer="{Voter server-a13cabfe-f7b8-22a1-0d36-6c7cac15dc2b a13cabfe-f7b8-22a1-0d36-6c7cac15dc2b}"
2026-05-25T23:23:26.310Z [INFO]  server-80b053d2-4cde-6f6d-16e6-aaa30e2cd65e: pipelining replication: peer="{Voter server-bb8c13b3-149d-8ac0-16e9-adc727603e91 bb8c13b3-149d-8ac0-16e9-adc727603e91}"
2026-05-25T23:23:27.316Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:26.310527679 +0000 UTC m=+61.956975157 (1 nodes), 531.629684ms from start of poll, 532.129747ms from cluster start. Timeout at 2026-05-25 23:23:27.315530317 +0000 UTC m=+62.961977770, 1.005002613s after stability
2026-05-25T23:23:27.316Z [INFO]  server-80b053d2-4cde-6f6d-16e6-aaa30e2cd65e: starting snapshot up to: index=12
2026-05-25T23:23:27.316Z [INFO]  creating new snapshot: path=/tmp/raft2976721827/snapshots/2-12-1779751407316.tmp
2026-05-25T23:23:27.317Z [INFO]  server-80b053d2-4cde-6f6d-16e6-aaa30e2cd65e: snapshot complete up to: index=12
2026-05-25T23:23:27.318Z [INFO]  server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d Address:5c8dd68b-1042-03a3-ddab-1955aa5a7d0d} {Suffrage:Voter ID:server-b77f1854-bc55-ea8e-5f00-80df26d0f03e Address:b77f1854-bc55-ea8e-5f00-80df26d0f03e} {Suffrage:Voter ID:server-ce8eaa5e-b4e2-0509-43f1-878737fdf586 Address:ce8eaa5e-b4e2-0509-43f1-878737fdf586}]"
2026-05-25T23:23:27.318Z [INFO]  server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d: entering follower state: follower="Node at 5c8dd68b-1042-03a3-ddab-1955aa5a7d0d [Follower]" leader-address= leader-id=
2026-05-25T23:23:27.318Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d Address:5c8dd68b-1042-03a3-ddab-1955aa5a7d0d} {Suffrage:Voter ID:server-b77f1854-bc55-ea8e-5f00-80df26d0f03e Address:b77f1854-bc55-ea8e-5f00-80df26d0f03e} {Suffrage:Voter ID:server-ce8eaa5e-b4e2-0509-43f1-878737fdf586 Address:ce8eaa5e-b4e2-0509-43f1-878737fdf586}]"
2026-05-25T23:23:27.318Z [INFO]  server-ce8eaa5e-b4e2-0509-43f1-878737fdf586: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d Address:5c8dd68b-1042-03a3-ddab-1955aa5a7d0d} {Suffrage:Voter ID:server-b77f1854-bc55-ea8e-5f00-80df26d0f03e Address:b77f1854-bc55-ea8e-5f00-80df26d0f03e} {Suffrage:Voter ID:server-ce8eaa5e-b4e2-0509-43f1-878737fdf586 Address:ce8eaa5e-b4e2-0509-43f1-878737fdf586}]"
2026-05-25T23:23:27.318Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: entering follower state: follower="Node at b77f1854-bc55-ea8e-5f00-80df26d0f03e [Follower]" leader-address= leader-id=
2026-05-25T23:23:27.318Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:27.318Z [INFO]  server-ce8eaa5e-b4e2-0509-43f1-878737fdf586: entering follower state: follower="Node at ce8eaa5e-b4e2-0509-43f1-878737fdf586 [Follower]" leader-address= leader-id=
2026-05-25T23:23:27.980Z [WARN]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:27.981Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: entering candidate state: node="Node at b77f1854-bc55-ea8e-5f00-80df26d0f03e [Candidate]" term=2
2026-05-25T23:23:27.981Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: election won: term=2 tally=2
2026-05-25T23:23:27.981Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: entering leader state: leader="Node at b77f1854-bc55-ea8e-5f00-80df26d0f03e [Leader]"
2026-05-25T23:23:27.981Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: added peer, starting replication: peer=server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d
2026-05-25T23:23:27.981Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: added peer, starting replication: peer=server-ce8eaa5e-b4e2-0509-43f1-878737fdf586
2026-05-25T23:23:27.981Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: pipelining replication: peer="{Voter server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d 5c8dd68b-1042-03a3-ddab-1955aa5a7d0d}"
2026-05-25T23:23:27.981Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: pipelining replication: peer="{Voter server-ce8eaa5e-b4e2-0509-43f1-878737fdf586 ce8eaa5e-b4e2-0509-43f1-878737fdf586}"
2026-05-25T23:23:28.986Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:27.981212262 +0000 UTC m=+63.627659732 (1 nodes), 662.599236ms from start of poll, 662.905466ms from cluster start. Timeout at 2026-05-25 23:23:28.986217443 +0000 UTC m=+64.632664898, 1.005005166s after stability
2026-05-25T23:23:28.986Z [INFO]  creating new snapshot: path=/tmp/raft775754481/snapshots/2-113-1779751408986.tmp
2026-05-25T23:23:28.987Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: copied to local snapshot: bytes=71
2026-05-25T23:23:28.987Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: snapshot restore progress: id=2-113-1779751408986 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:28.987Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: restored user snapshot: index=113
2026-05-25T23:23:28.987Z [ERROR] server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: failed to get log: index=3 error="log not found"
2026-05-25T23:23:28.987Z [ERROR] server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: failed to get log: index=3 error="log not found"
2026-05-25T23:23:28.987Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: aborting pipeline replication: peer="{Voter server-ce8eaa5e-b4e2-0509-43f1-878737fdf586 ce8eaa5e-b4e2-0509-43f1-878737fdf586}"
2026-05-25T23:23:28.987Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: aborting pipeline replication: peer="{Voter server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d 5c8dd68b-1042-03a3-ddab-1955aa5a7d0d}"
2026-05-25T23:23:28.996Z [ERROR] server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: failed to get log: index=3 error="log not found"
2026-05-25T23:23:28.996Z [ERROR] server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: failed to get log: index=3 error="log not found"
2026-05-25T23:23:28.996Z [INFO]  creating new snapshot: path=/tmp/raft3468775087/snapshots/2-113-1779751408996.tmp
2026-05-25T23:23:28.996Z [INFO]  creating new snapshot: path=/tmp/raft710979166/snapshots/2-113-1779751408996.tmp
2026-05-25T23:23:28.997Z [INFO]  server-ce8eaa5e-b4e2-0509-43f1-878737fdf586: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:28.997Z [INFO]  server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:28.997Z [INFO]  server-ce8eaa5e-b4e2-0509-43f1-878737fdf586: copied to local snapshot: bytes=71
2026-05-25T23:23:28.997Z [INFO]  server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d: copied to local snapshot: bytes=71
2026-05-25T23:23:28.997Z [INFO]  server-ce8eaa5e-b4e2-0509-43f1-878737fdf586: snapshot restore progress: id=2-113-1779751408996 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:28.997Z [INFO]  server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d: snapshot restore progress: id=2-113-1779751408996 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:28.997Z [INFO]  server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d: Installed remote snapshot
2026-05-25T23:23:28.997Z [INFO]  server-ce8eaa5e-b4e2-0509-43f1-878737fdf586: Installed remote snapshot
2026-05-25T23:23:28.997Z [WARN]  server-ce8eaa5e-b4e2-0509-43f1-878737fdf586: skipping application of old log: index=113
2026-05-25T23:23:28.997Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: pipelining replication: peer="{Voter server-ce8eaa5e-b4e2-0509-43f1-878737fdf586 ce8eaa5e-b4e2-0509-43f1-878737fdf586}"
2026-05-25T23:23:28.997Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: pipelining replication: peer="{Voter server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d 5c8dd68b-1042-03a3-ddab-1955aa5a7d0d}"
2026-05-25T23:23:28.997Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:23:30.002Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:23:28.997876938 +0000 UTC m=+64.644324396 (2 nodes), 658ns from start of poll, 1.67957013s from cluster start. Timeout at 2026-05-25 23:23:30.002900101 +0000 UTC m=+65.649347556, 1.00502316s after stability
2026-05-25T23:23:30.014Z [INFO]  server-80b053d2-4cde-6f6d-16e6-aaa30e2cd65e: aborting pipeline replication: peer="{Voter server-bb8c13b3-149d-8ac0-16e9-adc727603e91 bb8c13b3-149d-8ac0-16e9-adc727603e91}"
2026-05-25T23:23:30.014Z [INFO]  server-80b053d2-4cde-6f6d-16e6-aaa30e2cd65e: aborting pipeline replication: peer="{Voter server-a13cabfe-f7b8-22a1-0d36-6c7cac15dc2b a13cabfe-f7b8-22a1-0d36-6c7cac15dc2b}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true
2026-05-25T23:23:30.015Z [INFO]  server-627fb674-0aa0-c7c2-b660-1ded51312add: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-627fb674-0aa0-c7c2-b660-1ded51312add Address:627fb674-0aa0-c7c2-b660-1ded51312add} {Suffrage:Voter ID:server-0f3800ee-f0e9-536d-86c2-b9d246133927 Address:0f3800ee-f0e9-536d-86c2-b9d246133927} {Suffrage:Voter ID:server-58f5fc51-8afc-9619-b356-257a4a5cbfba Address:58f5fc51-8afc-9619-b356-257a4a5cbfba}]"
2026-05-25T23:23:30.015Z [INFO]  server-627fb674-0aa0-c7c2-b660-1ded51312add: entering follower state: follower="Node at 627fb674-0aa0-c7c2-b660-1ded51312add [Follower]" leader-address= leader-id=
2026-05-25T23:23:30.016Z [INFO]  server-0f3800ee-f0e9-536d-86c2-b9d246133927: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-627fb674-0aa0-c7c2-b660-1ded51312add Address:627fb674-0aa0-c7c2-b660-1ded51312add} {Suffrage:Voter ID:server-0f3800ee-f0e9-536d-86c2-b9d246133927 Address:0f3800ee-f0e9-536d-86c2-b9d246133927} {Suffrage:Voter ID:server-58f5fc51-8afc-9619-b356-257a4a5cbfba Address:58f5fc51-8afc-9619-b356-257a4a5cbfba}]"
2026-05-25T23:23:30.016Z [INFO]  server-0f3800ee-f0e9-536d-86c2-b9d246133927: entering follower state: follower="Node at 0f3800ee-f0e9-536d-86c2-b9d246133927 [Follower]" leader-address= leader-id=
2026-05-25T23:23:30.016Z [INFO]  server-58f5fc51-8afc-9619-b356-257a4a5cbfba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-627fb674-0aa0-c7c2-b660-1ded51312add Address:627fb674-0aa0-c7c2-b660-1ded51312add} {Suffrage:Voter ID:server-0f3800ee-f0e9-536d-86c2-b9d246133927 Address:0f3800ee-f0e9-536d-86c2-b9d246133927} {Suffrage:Voter ID:server-58f5fc51-8afc-9619-b356-257a4a5cbfba Address:58f5fc51-8afc-9619-b356-257a4a5cbfba}]"
2026-05-25T23:23:30.016Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:30.016Z [INFO]  server-58f5fc51-8afc-9619-b356-257a4a5cbfba: entering follower state: follower="Node at 58f5fc51-8afc-9619-b356-257a4a5cbfba [Follower]" leader-address= leader-id=
2026-05-25T23:23:30.636Z [WARN]  server-58f5fc51-8afc-9619-b356-257a4a5cbfba: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:30.636Z [INFO]  server-58f5fc51-8afc-9619-b356-257a4a5cbfba: entering candidate state: node="Node at 58f5fc51-8afc-9619-b356-257a4a5cbfba [Candidate]" term=2
2026-05-25T23:23:30.636Z [INFO]  server-58f5fc51-8afc-9619-b356-257a4a5cbfba: election won: term=2 tally=2
2026-05-25T23:23:30.636Z [INFO]  server-58f5fc51-8afc-9619-b356-257a4a5cbfba: entering leader state: leader="Node at 58f5fc51-8afc-9619-b356-257a4a5cbfba [Leader]"
2026-05-25T23:23:30.636Z [INFO]  server-58f5fc51-8afc-9619-b356-257a4a5cbfba: added peer, starting replication: peer=server-627fb674-0aa0-c7c2-b660-1ded51312add
2026-05-25T23:23:30.636Z [INFO]  server-58f5fc51-8afc-9619-b356-257a4a5cbfba: added peer, starting replication: peer=server-0f3800ee-f0e9-536d-86c2-b9d246133927
2026-05-25T23:23:30.636Z [INFO]  server-58f5fc51-8afc-9619-b356-257a4a5cbfba: pipelining replication: peer="{Voter server-627fb674-0aa0-c7c2-b660-1ded51312add 627fb674-0aa0-c7c2-b660-1ded51312add}"
2026-05-25T23:23:30.636Z [INFO]  server-58f5fc51-8afc-9619-b356-257a4a5cbfba: pipelining replication: peer="{Voter server-0f3800ee-f0e9-536d-86c2-b9d246133927 0f3800ee-f0e9-536d-86c2-b9d246133927}"
2026-05-25T23:23:31.642Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:30.636496172 +0000 UTC m=+66.282943699 (1 nodes), 620.392678ms from start of poll, 620.701716ms from cluster start. Timeout at 2026-05-25 23:23:31.641499006 +0000 UTC m=+67.287946461, 1.005002762s after stability
2026-05-25T23:23:31.642Z [INFO]  server-58f5fc51-8afc-9619-b356-257a4a5cbfba: starting snapshot up to: index=12
2026-05-25T23:23:31.642Z [INFO]  creating new snapshot: path=/tmp/raft717140247/snapshots/2-12-1779751411642.tmp
2026-05-25T23:23:31.643Z [INFO]  server-58f5fc51-8afc-9619-b356-257a4a5cbfba: snapshot complete up to: index=12
2026-05-25T23:23:31.645Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-941b9661-74e5-5201-f2fe-39b3f009b868 Address:941b9661-74e5-5201-f2fe-39b3f009b868} {Suffrage:Voter ID:server-116a910e-5eb0-5371-9698-339f44ce34c6 Address:116a910e-5eb0-5371-9698-339f44ce34c6} {Suffrage:Voter ID:server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b Address:f9a39ce0-d9e9-5faf-e108-7822f2b3993b}]"
2026-05-25T23:23:31.645Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-941b9661-74e5-5201-f2fe-39b3f009b868 Address:941b9661-74e5-5201-f2fe-39b3f009b868} {Suffrage:Voter ID:server-116a910e-5eb0-5371-9698-339f44ce34c6 Address:116a910e-5eb0-5371-9698-339f44ce34c6} {Suffrage:Voter ID:server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b Address:f9a39ce0-d9e9-5faf-e108-7822f2b3993b}]"
2026-05-25T23:23:31.645Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: entering follower state: follower="Node at 941b9661-74e5-5201-f2fe-39b3f009b868 [Follower]" leader-address= leader-id=
2026-05-25T23:23:31.645Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: entering follower state: follower="Node at 116a910e-5eb0-5371-9698-339f44ce34c6 [Follower]" leader-address= leader-id=
2026-05-25T23:23:31.645Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-941b9661-74e5-5201-f2fe-39b3f009b868 Address:941b9661-74e5-5201-f2fe-39b3f009b868} {Suffrage:Voter ID:server-116a910e-5eb0-5371-9698-339f44ce34c6 Address:116a910e-5eb0-5371-9698-339f44ce34c6} {Suffrage:Voter ID:server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b Address:f9a39ce0-d9e9-5faf-e108-7822f2b3993b}]"
2026-05-25T23:23:31.645Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:31.645Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: entering follower state: follower="Node at f9a39ce0-d9e9-5faf-e108-7822f2b3993b [Follower]" leader-address= leader-id=
2026-05-25T23:23:32.276Z [WARN]  server-941b9661-74e5-5201-f2fe-39b3f009b868: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:32.276Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: entering candidate state: node="Node at 941b9661-74e5-5201-f2fe-39b3f009b868 [Candidate]" term=2
2026-05-25T23:23:32.277Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: election won: term=2 tally=2
2026-05-25T23:23:32.277Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: entering leader state: leader="Node at 941b9661-74e5-5201-f2fe-39b3f009b868 [Leader]"
2026-05-25T23:23:32.277Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: added peer, starting replication: peer=server-116a910e-5eb0-5371-9698-339f44ce34c6
2026-05-25T23:23:32.277Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: added peer, starting replication: peer=server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b
2026-05-25T23:23:32.277Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: pipelining replication: peer="{Voter server-116a910e-5eb0-5371-9698-339f44ce34c6 116a910e-5eb0-5371-9698-339f44ce34c6}"
2026-05-25T23:23:32.277Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: pipelining replication: peer="{Voter server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b f9a39ce0-d9e9-5faf-e108-7822f2b3993b}"
2026-05-25T23:23:33.282Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:32.277076904 +0000 UTC m=+67.923524358 (1 nodes), 631.765015ms from start of poll, 632.069928ms from cluster start. Timeout at 2026-05-25 23:23:33.282087422 +0000 UTC m=+68.928534891, 1.005010533s after stability
2026-05-25T23:23:33.282Z [INFO]  creating new snapshot: path=/tmp/raft240147463/snapshots/2-113-1779751413282.tmp
2026-05-25T23:23:33.283Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: copied to local snapshot: bytes=71
2026-05-25T23:23:33.283Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: snapshot restore progress: id=2-113-1779751413282 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:33.283Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: removing all old logs from log store
2026-05-25T23:23:33.283Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: compacting logs: from=1 to=2
2026-05-25T23:23:33.283Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: restored user snapshot: index=113
2026-05-25T23:23:33.283Z [ERROR] server-941b9661-74e5-5201-f2fe-39b3f009b868: failed to get log: index=2 error="log not found"
2026-05-25T23:23:33.283Z [ERROR] server-941b9661-74e5-5201-f2fe-39b3f009b868: failed to get log: index=2 error="log not found"
2026-05-25T23:23:33.283Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: aborting pipeline replication: peer="{Voter server-116a910e-5eb0-5371-9698-339f44ce34c6 116a910e-5eb0-5371-9698-339f44ce34c6}"
2026-05-25T23:23:33.283Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: aborting pipeline replication: peer="{Voter server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b f9a39ce0-d9e9-5faf-e108-7822f2b3993b}"
2026-05-25T23:23:33.290Z [ERROR] server-941b9661-74e5-5201-f2fe-39b3f009b868: failed to get log: index=2 error="log not found"
2026-05-25T23:23:33.290Z [INFO]  creating new snapshot: path=/tmp/raft2098650435/snapshots/2-113-1779751413290.tmp
2026-05-25T23:23:33.290Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:33.290Z [ERROR] server-941b9661-74e5-5201-f2fe-39b3f009b868: failed to get log: index=2 error="log not found"
2026-05-25T23:23:33.290Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: copied to local snapshot: bytes=71
2026-05-25T23:23:33.291Z [INFO]  creating new snapshot: path=/tmp/raft4159409475/snapshots/2-113-1779751413291.tmp
2026-05-25T23:23:33.291Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: snapshot restore progress: id=2-113-1779751413290 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:33.291Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: removing all old logs from log store
2026-05-25T23:23:33.291Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: compacting logs: from=1 to=2
2026-05-25T23:23:33.291Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: Installed remote snapshot
2026-05-25T23:23:33.291Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: pipelining replication: peer="{Voter server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b f9a39ce0-d9e9-5faf-e108-7822f2b3993b}"
2026-05-25T23:23:33.291Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:23:33.292Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:33.292Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: copied to local snapshot: bytes=71
2026-05-25T23:23:33.292Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: snapshot restore progress: id=2-113-1779751413291 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:33.292Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: removing all old logs from log store
2026-05-25T23:23:33.292Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: compacting logs: from=1 to=2
2026-05-25T23:23:33.292Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: Installed remote snapshot
2026-05-25T23:23:33.292Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: pipelining replication: peer="{Voter server-116a910e-5eb0-5371-9698-339f44ce34c6 116a910e-5eb0-5371-9698-339f44ce34c6}"
2026-05-25T23:23:34.297Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:23:33.291526064 +0000 UTC m=+68.937973526 (2 nodes), 814ns from start of poll, 1.646519096s from cluster start. Timeout at 2026-05-25 23:23:34.296572959 +0000 UTC m=+69.943020526, 1.005047s after stability
2026-05-25T23:23:34.308Z [INFO]  server-58f5fc51-8afc-9619-b356-257a4a5cbfba: aborting pipeline replication: peer="{Voter server-627fb674-0aa0-c7c2-b660-1ded51312add 627fb674-0aa0-c7c2-b660-1ded51312add}"
2026-05-25T23:23:34.308Z [INFO]  server-58f5fc51-8afc-9619-b356-257a4a5cbfba: aborting pipeline replication: peer="{Voter server-0f3800ee-f0e9-536d-86c2-b9d246133927 0f3800ee-f0e9-536d-86c2-b9d246133927}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false
2026-05-25T23:23:34.342Z [INFO]  server-41037aef-e4cf-ea9c-8180-68ba707d267e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41037aef-e4cf-ea9c-8180-68ba707d267e Address:41037aef-e4cf-ea9c-8180-68ba707d267e} {Suffrage:Voter ID:server-1fe26923-ea55-f0bd-0d7a-994021226697 Address:1fe26923-ea55-f0bd-0d7a-994021226697} {Suffrage:Voter ID:server-17ad52e4-24c4-8844-49af-e3e46041efea Address:17ad52e4-24c4-8844-49af-e3e46041efea}]"
2026-05-25T23:23:34.342Z [INFO]  server-41037aef-e4cf-ea9c-8180-68ba707d267e: entering follower state: follower="Node at 41037aef-e4cf-ea9c-8180-68ba707d267e [Follower]" leader-address= leader-id=
2026-05-25T23:23:34.342Z [INFO]  server-1fe26923-ea55-f0bd-0d7a-994021226697: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41037aef-e4cf-ea9c-8180-68ba707d267e Address:41037aef-e4cf-ea9c-8180-68ba707d267e} {Suffrage:Voter ID:server-1fe26923-ea55-f0bd-0d7a-994021226697 Address:1fe26923-ea55-f0bd-0d7a-994021226697} {Suffrage:Voter ID:server-17ad52e4-24c4-8844-49af-e3e46041efea Address:17ad52e4-24c4-8844-49af-e3e46041efea}]"
2026-05-25T23:23:34.342Z [INFO]  server-1fe26923-ea55-f0bd-0d7a-994021226697: entering follower state: follower="Node at 1fe26923-ea55-f0bd-0d7a-994021226697 [Follower]" leader-address= leader-id=
2026-05-25T23:23:34.342Z [INFO]  server-17ad52e4-24c4-8844-49af-e3e46041efea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41037aef-e4cf-ea9c-8180-68ba707d267e Address:41037aef-e4cf-ea9c-8180-68ba707d267e} {Suffrage:Voter ID:server-1fe26923-ea55-f0bd-0d7a-994021226697 Address:1fe26923-ea55-f0bd-0d7a-994021226697} {Suffrage:Voter ID:server-17ad52e4-24c4-8844-49af-e3e46041efea Address:17ad52e4-24c4-8844-49af-e3e46041efea}]"
2026-05-25T23:23:34.342Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:34.342Z [INFO]  server-17ad52e4-24c4-8844-49af-e3e46041efea: entering follower state: follower="Node at 17ad52e4-24c4-8844-49af-e3e46041efea [Follower]" leader-address= leader-id=
2026-05-25T23:23:34.860Z [WARN]  server-1fe26923-ea55-f0bd-0d7a-994021226697: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:34.860Z [INFO]  server-1fe26923-ea55-f0bd-0d7a-994021226697: entering candidate state: node="Node at 1fe26923-ea55-f0bd-0d7a-994021226697 [Candidate]" term=2
2026-05-25T23:23:34.860Z [INFO]  server-1fe26923-ea55-f0bd-0d7a-994021226697: election won: term=2 tally=2
2026-05-25T23:23:34.860Z [INFO]  server-1fe26923-ea55-f0bd-0d7a-994021226697: entering leader state: leader="Node at 1fe26923-ea55-f0bd-0d7a-994021226697 [Leader]"
2026-05-25T23:23:34.860Z [INFO]  server-1fe26923-ea55-f0bd-0d7a-994021226697: added peer, starting replication: peer=server-41037aef-e4cf-ea9c-8180-68ba707d267e
2026-05-25T23:23:34.860Z [INFO]  server-1fe26923-ea55-f0bd-0d7a-994021226697: added peer, starting replication: peer=server-17ad52e4-24c4-8844-49af-e3e46041efea
2026-05-25T23:23:34.860Z [INFO]  server-1fe26923-ea55-f0bd-0d7a-994021226697: pipelining replication: peer="{Voter server-17ad52e4-24c4-8844-49af-e3e46041efea 17ad52e4-24c4-8844-49af-e3e46041efea}"
2026-05-25T23:23:34.860Z [INFO]  server-1fe26923-ea55-f0bd-0d7a-994021226697: pipelining replication: peer="{Voter server-41037aef-e4cf-ea9c-8180-68ba707d267e 41037aef-e4cf-ea9c-8180-68ba707d267e}"
2026-05-25T23:23:36.814Z [WARN]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: failed to contact: server-id=server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb time=1.085710531s
2026-05-25T23:23:36.814Z [WARN]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: heartbeat timeout reached, starting election: last-leader-addr=5aff39f7-caf5-0cca-b85b-cefd035d60f7 last-leader-id=server-5aff39f7-caf5-0cca-b85b-cefd035d60f7
2026-05-25T23:23:36.814Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: entering candidate state: node="Node at eedc65c4-92af-c6a4-c7ee-60bd0065ac4a [Candidate]" term=3
2026-05-25T23:23:36.814Z [WARN]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: failed to contact: server-id=server-ae4d03f3-6164-5c59-99d4-455db47640e4 time=1.065293767s
2026-05-25T23:23:36.814Z [WARN]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: failed to contact: server-id=server-8cdddb1f-717d-965c-d545-807129b2ce3b time=1.126803197s
2026-05-25T23:23:36.815Z [WARN]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: heartbeat timeout reached, starting election: last-leader-addr=941b9661-74e5-5201-f2fe-39b3f009b868 last-leader-id=server-941b9661-74e5-5201-f2fe-39b3f009b868
2026-05-25T23:23:36.815Z [WARN]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: failed to contact: server-id=server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d time=1.128857851s
2026-05-25T23:23:36.815Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: entering candidate state: node="Node at f9a39ce0-d9e9-5faf-e108-7822f2b3993b [Candidate]" term=3
2026-05-25T23:23:36.815Z [WARN]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: failed to contact: server-id=server-ce8eaa5e-b4e2-0509-43f1-878737fdf586 time=1.126386098s
2026-05-25T23:23:36.814Z [WARN]  server-bb5ff54a-16e0-4125-6196-c4d8b3be7726: heartbeat timeout reached, starting election: last-leader-addr=5aff39f7-caf5-0cca-b85b-cefd035d60f7 last-leader-id=server-5aff39f7-caf5-0cca-b85b-cefd035d60f7
2026-05-25T23:23:36.815Z [WARN]  server-116a910e-5eb0-5371-9698-339f44ce34c6: heartbeat timeout reached, starting election: last-leader-addr=941b9661-74e5-5201-f2fe-39b3f009b868 last-leader-id=server-941b9661-74e5-5201-f2fe-39b3f009b868
2026-05-25T23:23:36.815Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:34.860759863 +0000 UTC m=+70.507207312 (1 nodes), 518.280611ms from start of poll, 518.905067ms from cluster start. Timeout at 2026-05-25 23:23:35.865775942 +0000 UTC m=+71.512223407, 1.005016095s after stability
2026-05-25T23:23:36.814Z [WARN]  server-1fe26923-ea55-f0bd-0d7a-994021226697: failed to contact: server-id=server-41037aef-e4cf-ea9c-8180-68ba707d267e time=1.131475612s
2026-05-25T23:23:36.815Z [WARN]  server-43deaf51-2664-9283-bde9-1056361f0668: heartbeat timeout reached, starting election: last-leader-addr=164f4764-ebd7-bbaa-3367-321add0910e4 last-leader-id=server-164f4764-ebd7-bbaa-3367-321add0910e4
2026-05-25T23:23:36.815Z [WARN]  server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d: heartbeat timeout reached, starting election: last-leader-addr=b77f1854-bc55-ea8e-5f00-80df26d0f03e last-leader-id=server-b77f1854-bc55-ea8e-5f00-80df26d0f03e
2026-05-25T23:23:36.815Z [INFO]  server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d: entering candidate state: node="Node at 5c8dd68b-1042-03a3-ddab-1955aa5a7d0d [Candidate]" term=3
2026-05-25T23:23:36.815Z [WARN]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: failed to contact quorum of nodes, stepping down
2026-05-25T23:23:36.814Z [WARN]  server-17ad52e4-24c4-8844-49af-e3e46041efea: heartbeat timeout reached, starting election: last-leader-addr=1fe26923-ea55-f0bd-0d7a-994021226697 last-leader-id=server-1fe26923-ea55-f0bd-0d7a-994021226697
2026-05-25T23:23:36.815Z [WARN]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: failed to contact: server-id=server-918e04a4-d2b8-6285-0572-1f11870065de time=1.085740509s
2026-05-25T23:23:36.816Z [WARN]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: failed to contact quorum of nodes, stepping down
2026-05-25T23:23:36.816Z [INFO]  server-17ad52e4-24c4-8844-49af-e3e46041efea: entering candidate state: node="Node at 17ad52e4-24c4-8844-49af-e3e46041efea [Candidate]" term=3
2026-05-25T23:23:36.816Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: entering follower state: follower="Node at b77f1854-bc55-ea8e-5f00-80df26d0f03e [Follower]" leader-address= leader-id=
2026-05-25T23:23:36.816Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: entering follower state: follower="Node at 0bc7c1d4-cea2-25e5-ba38-79cde9eb4058 [Follower]" leader-address= leader-id=
2026-05-25T23:23:36.815Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: entering candidate state: node="Node at 116a910e-5eb0-5371-9698-339f44ce34c6 [Candidate]" term=3
2026-05-25T23:23:36.816Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: aborting pipeline replication: peer="{Voter server-ce8eaa5e-b4e2-0509-43f1-878737fdf586 ce8eaa5e-b4e2-0509-43f1-878737fdf586}"
2026-05-25T23:23:36.815Z [WARN]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: heartbeat timeout reached, starting election: last-leader-addr=b2d831eb-15f8-614f-ec08-fafe997c4944 last-leader-id=server-b2d831eb-15f8-614f-ec08-fafe997c4944
2026-05-25T23:23:36.815Z [WARN]  server-1fe26923-ea55-f0bd-0d7a-994021226697: failed to contact: server-id=server-17ad52e4-24c4-8844-49af-e3e46041efea time=1.129243074s
2026-05-25T23:23:36.816Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: entering candidate state: node="Node at 8cdddb1f-717d-965c-d545-807129b2ce3b [Candidate]" term=3
2026-05-25T23:23:36.816Z [WARN]  server-1fe26923-ea55-f0bd-0d7a-994021226697: failed to contact quorum of nodes, stepping down
2026-05-25T23:23:36.816Z [WARN]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: heartbeat timeout reached, starting election: last-leader-addr=164f4764-ebd7-bbaa-3367-321add0910e4 last-leader-id=server-164f4764-ebd7-bbaa-3367-321add0910e4
2026-05-25T23:23:36.816Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: aborting pipeline replication: peer="{Voter server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d 5c8dd68b-1042-03a3-ddab-1955aa5a7d0d}"
2026-05-25T23:23:36.816Z [INFO]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: entering candidate state: node="Node at 19c71118-7e61-cc98-9b5e-18e0d7ec6f6f [Candidate]" term=3
2026-05-25T23:23:36.815Z [WARN]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: failed to contact: server-id=server-bb5ff54a-16e0-4125-6196-c4d8b3be7726 time=1.121953465s
2026-05-25T23:23:36.816Z [INFO]  server-1fe26923-ea55-f0bd-0d7a-994021226697: entering follower state: follower="Node at 1fe26923-ea55-f0bd-0d7a-994021226697 [Follower]" leader-address= leader-id=
2026-05-25T23:23:36.816Z [WARN]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: failed to contact: server-id=server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a time=1.06642383s
2026-05-25T23:23:36.815Z [WARN]  server-41037aef-e4cf-ea9c-8180-68ba707d267e: heartbeat timeout reached, starting election: last-leader-addr=1fe26923-ea55-f0bd-0d7a-994021226697 last-leader-id=server-1fe26923-ea55-f0bd-0d7a-994021226697
2026-05-25T23:23:36.815Z [INFO]  server-bb5ff54a-16e0-4125-6196-c4d8b3be7726: entering candidate state: node="Node at bb5ff54a-16e0-4125-6196-c4d8b3be7726 [Candidate]" term=3
2026-05-25T23:23:36.816Z [ERROR] server-941b9661-74e5-5201-f2fe-39b3f009b868: peer has newer term, stopping replication: peer="{Voter server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b f9a39ce0-d9e9-5faf-e108-7822f2b3993b}"
2026-05-25T23:23:36.816Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: aborting pipeline replication: peer="{Voter server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b f9a39ce0-d9e9-5faf-e108-7822f2b3993b}"
2026-05-25T23:23:36.815Z [WARN]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: failed to contact quorum of nodes, stepping down
2026-05-25T23:23:36.815Z [WARN]  server-164f4764-ebd7-bbaa-3367-321add0910e4: failed to contact: server-id=server-43deaf51-2664-9283-bde9-1056361f0668 time=1.128909416s
2026-05-25T23:23:36.814Z [WARN]  server-ae4d03f3-6164-5c59-99d4-455db47640e4: heartbeat timeout reached, starting election: last-leader-addr=b2d831eb-15f8-614f-ec08-fafe997c4944 last-leader-id=server-b2d831eb-15f8-614f-ec08-fafe997c4944
2026-05-25T23:23:36.816Z [WARN]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: failed to contact quorum of nodes, stepping down
2026-05-25T23:23:36.816Z [INFO]  server-ae4d03f3-6164-5c59-99d4-455db47640e4: entering candidate state: node="Node at ae4d03f3-6164-5c59-99d4-455db47640e4 [Candidate]" term=3
2026-05-25T23:23:36.815Z [INFO]  server-43deaf51-2664-9283-bde9-1056361f0668: entering candidate state: node="Node at 43deaf51-2664-9283-bde9-1056361f0668 [Candidate]" term=3
2026-05-25T23:23:36.816Z [INFO]  server-1fe26923-ea55-f0bd-0d7a-994021226697: aborting pipeline replication: peer="{Voter server-41037aef-e4cf-ea9c-8180-68ba707d267e 41037aef-e4cf-ea9c-8180-68ba707d267e}"
2026-05-25T23:23:36.816Z [ERROR] server-164f4764-ebd7-bbaa-3367-321add0910e4: peer has newer term, stopping replication: peer="{Voter server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f 19c71118-7e61-cc98-9b5e-18e0d7ec6f6f}"
2026-05-25T23:23:36.816Z [INFO]  server-1fe26923-ea55-f0bd-0d7a-994021226697: aborting pipeline replication: peer="{Voter server-17ad52e4-24c4-8844-49af-e3e46041efea 17ad52e4-24c4-8844-49af-e3e46041efea}"
2026-05-25T23:23:36.816Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.816Z [INFO]  server-41037aef-e4cf-ea9c-8180-68ba707d267e: entering candidate state: node="Node at 41037aef-e4cf-ea9c-8180-68ba707d267e [Candidate]" term=3
2026-05-25T23:23:36.816Z [INFO]  server-17ad52e4-24c4-8844-49af-e3e46041efea: election won: term=3 tally=2
2026-05-25T23:23:36.816Z [INFO]  server-17ad52e4-24c4-8844-49af-e3e46041efea: entering leader state: leader="Node at 17ad52e4-24c4-8844-49af-e3e46041efea [Leader]"
2026-05-25T23:23:36.816Z [INFO]  server-17ad52e4-24c4-8844-49af-e3e46041efea: added peer, starting replication: peer=server-41037aef-e4cf-ea9c-8180-68ba707d267e
2026-05-25T23:23:36.816Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: entering follower state: follower="Node at 5aff39f7-caf5-0cca-b85b-cefd035d60f7 [Follower]" leader-address= leader-id=
2026-05-25T23:23:36.816Z [INFO]  server-ae4d03f3-6164-5c59-99d4-455db47640e4: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.812Z [WARN]  server-ce8eaa5e-b4e2-0509-43f1-878737fdf586: heartbeat timeout reached, starting election: last-leader-addr=b77f1854-bc55-ea8e-5f00-80df26d0f03e last-leader-id=server-b77f1854-bc55-ea8e-5f00-80df26d0f03e
2026-05-25T23:23:36.815Z [WARN]  server-918e04a4-d2b8-6285-0572-1f11870065de: heartbeat timeout reached, starting election: last-leader-addr=0bc7c1d4-cea2-25e5-ba38-79cde9eb4058 last-leader-id=server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058
2026-05-25T23:23:36.816Z [INFO]  server-ce8eaa5e-b4e2-0509-43f1-878737fdf586: entering candidate state: node="Node at ce8eaa5e-b4e2-0509-43f1-878737fdf586 [Candidate]" term=3
2026-05-25T23:23:36.816Z [INFO]  server-918e04a4-d2b8-6285-0572-1f11870065de: entering candidate state: node="Node at 918e04a4-d2b8-6285-0572-1f11870065de [Candidate]" term=3
    raft_test.go:1595: Error Apply new log entries: node is not the leader
2026-05-25T23:23:36.816Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.816Z [WARN]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: failed to contact: server-id=server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb time=1.087648617s
2026-05-25T23:23:36.816Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: entering follower state: follower="Node at b2d831eb-15f8-614f-ec08-fafe997c4944 [Follower]" leader-address= leader-id=
2026-05-25T23:23:36.816Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.816Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: election won: term=3 tally=2
2026-05-25T23:23:36.816Z [INFO]  server-43deaf51-2664-9283-bde9-1056361f0668: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.816Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: entering leader state: leader="Node at eedc65c4-92af-c6a4-c7ee-60bd0065ac4a [Leader]"
2026-05-25T23:23:36.816Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: aborting pipeline replication: peer="{Voter server-ae4d03f3-6164-5c59-99d4-455db47640e4 ae4d03f3-6164-5c59-99d4-455db47640e4}"
2026-05-25T23:23:36.816Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: added peer, starting replication: peer=server-5aff39f7-caf5-0cca-b85b-cefd035d60f7
2026-05-25T23:23:36.815Z [WARN]  server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940: heartbeat timeout reached, starting election: last-leader-addr=fcb5b37a-b3b5-51e6-b68a-69e91d5df710 last-leader-id=server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710
2026-05-25T23:23:36.816Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: added peer, starting replication: peer=server-bb5ff54a-16e0-4125-6196-c4d8b3be7726
2026-05-25T23:23:36.816Z [INFO]  server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940: entering candidate state: node="Node at 8cf9d984-e9cf-b9ad-bd1c-e4135e071940 [Candidate]" term=3
2026-05-25T23:23:36.816Z [ERROR] server-941b9661-74e5-5201-f2fe-39b3f009b868: peer has newer term, stopping replication: peer="{Voter server-116a910e-5eb0-5371-9698-339f44ce34c6 116a910e-5eb0-5371-9698-339f44ce34c6}"
2026-05-25T23:23:36.816Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: aborting pipeline replication: peer="{Voter server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f 19c71118-7e61-cc98-9b5e-18e0d7ec6f6f}"
2026-05-25T23:23:36.816Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.816Z [INFO]  server-bb5ff54a-16e0-4125-6196-c4d8b3be7726: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.812Z [WARN]  server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb: heartbeat timeout reached, starting election: last-leader-addr=0bc7c1d4-cea2-25e5-ba38-79cde9eb4058 last-leader-id=server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058
2026-05-25T23:23:36.816Z [INFO]  server-b77f1854-bc55-ea8e-5f00-80df26d0f03e: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.816Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: election won: term=3 tally=2
2026-05-25T23:23:36.816Z [ERROR] server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: peer has newer term, stopping replication: peer="{Voter server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a eedc65c4-92af-c6a4-c7ee-60bd0065ac4a}"
2026-05-25T23:23:36.816Z [INFO]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.816Z [INFO]  server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb: entering candidate state: node="Node at b2b88b2b-e5dc-316b-2db7-975d03d33dfb [Candidate]" term=3
2026-05-25T23:23:36.816Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: aborting pipeline replication: peer="{Voter server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a eedc65c4-92af-c6a4-c7ee-60bd0065ac4a}"
2026-05-25T23:23:36.816Z [WARN]  server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb: heartbeat timeout reached, starting election: last-leader-addr=eb7f8dda-b789-58f9-9aee-a08a5ed05fcc last-leader-id=server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc
2026-05-25T23:23:36.816Z [WARN]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: failed to contact: server-id=server-cb02535b-4fa3-e7a6-9c73-0120936129ca time=1.129991138s
2026-05-25T23:23:36.816Z [INFO]  server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb: entering candidate state: node="Node at bc1e4076-200c-c555-eb3f-ab3ef10bbedb [Candidate]" term=3
2026-05-25T23:23:36.816Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.816Z [INFO]  server-918e04a4-d2b8-6285-0572-1f11870065de: election won: term=3 tally=2
2026-05-25T23:23:36.816Z [INFO]  server-ce8eaa5e-b4e2-0509-43f1-878737fdf586: election won: term=3 tally=2
2026-05-25T23:23:36.816Z [INFO]  server-918e04a4-d2b8-6285-0572-1f11870065de: entering leader state: leader="Node at 918e04a4-d2b8-6285-0572-1f11870065de [Leader]"
2026-05-25T23:23:36.816Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: aborting pipeline replication: peer="{Voter server-918e04a4-d2b8-6285-0572-1f11870065de 918e04a4-d2b8-6285-0572-1f11870065de}"
2026-05-25T23:23:36.816Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: entering leader state: leader="Node at 8cdddb1f-717d-965c-d545-807129b2ce3b [Leader]"
2026-05-25T23:23:36.816Z [WARN]  server-1ea7c9cb-c351-9612-c976-e429888b54a7: rejecting vote request since we have a leader: from=8cf9d984-e9cf-b9ad-bd1c-e4135e071940 leader=fcb5b37a-b3b5-51e6-b68a-69e91d5df710 leader-id=server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710
2026-05-25T23:23:36.816Z [INFO]  server-ce8eaa5e-b4e2-0509-43f1-878737fdf586: entering leader state: leader="Node at ce8eaa5e-b4e2-0509-43f1-878737fdf586 [Leader]"
2026-05-25T23:23:36.815Z [WARN]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: failed to contact: server-id=server-1ea7c9cb-c351-9612-c976-e429888b54a7 time=1.121859262s
2026-05-25T23:23:36.816Z [INFO]  server-17ad52e4-24c4-8844-49af-e3e46041efea: added peer, starting replication: peer=server-1fe26923-ea55-f0bd-0d7a-994021226697
2026-05-25T23:23:36.812Z [WARN]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: heartbeat timeout reached, starting election: last-leader-addr=eb7f8dda-b789-58f9-9aee-a08a5ed05fcc last-leader-id=server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc
2026-05-25T23:23:36.816Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: entering candidate state: node="Node at cb02535b-4fa3-e7a6-9c73-0120936129ca [Candidate]" term=3
2026-05-25T23:23:36.816Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: aborting pipeline replication: peer="{Voter server-bb5ff54a-16e0-4125-6196-c4d8b3be7726 bb5ff54a-16e0-4125-6196-c4d8b3be7726}"
2026-05-25T23:23:36.816Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.816Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: entering follower state: follower="Node at 164f4764-ebd7-bbaa-3367-321add0910e4 [Follower]" leader-address= leader-id=
2026-05-25T23:23:36.816Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.816Z [INFO]  server-bb5ff54a-16e0-4125-6196-c4d8b3be7726: entering follower state: follower="Node at bb5ff54a-16e0-4125-6196-c4d8b3be7726 [Follower]" leader-address=eedc65c4-92af-c6a4-c7ee-60bd0065ac4a leader-id=server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a
2026-05-25T23:23:36.816Z [ERROR] server-b2d831eb-15f8-614f-ec08-fafe997c4944: peer has newer term, stopping replication: peer="{Voter server-8cdddb1f-717d-965c-d545-807129b2ce3b 8cdddb1f-717d-965c-d545-807129b2ce3b}"
2026-05-25T23:23:36.816Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: aborting pipeline replication: peer="{Voter server-8cdddb1f-717d-965c-d545-807129b2ce3b 8cdddb1f-717d-965c-d545-807129b2ce3b}"
2026-05-25T23:23:36.816Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: pipelining replication: peer="{Voter server-5aff39f7-caf5-0cca-b85b-cefd035d60f7 5aff39f7-caf5-0cca-b85b-cefd035d60f7}"
2026-05-25T23:23:36.816Z [INFO]  server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.816Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: aborting pipeline replication: peer="{Voter server-116a910e-5eb0-5371-9698-339f44ce34c6 116a910e-5eb0-5371-9698-339f44ce34c6}"
2026-05-25T23:23:36.816Z [INFO]  server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.816Z [WARN]  server-941b9661-74e5-5201-f2fe-39b3f009b868: rejecting vote request since we have a leader: from=f9a39ce0-d9e9-5faf-e108-7822f2b3993b leader=941b9661-74e5-5201-f2fe-39b3f009b868 leader-id=server-941b9661-74e5-5201-f2fe-39b3f009b868
2026-05-25T23:23:36.816Z [INFO]  server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.816Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.817Z [WARN]  server-941b9661-74e5-5201-f2fe-39b3f009b868: rejecting vote request since we have a leader: from=116a910e-5eb0-5371-9698-339f44ce34c6 leader=941b9661-74e5-5201-f2fe-39b3f009b868 leader-id=server-941b9661-74e5-5201-f2fe-39b3f009b868
2026-05-25T23:23:36.816Z [INFO]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: election won: term=3 tally=2
2026-05-25T23:23:36.817Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: entering follower state: follower="Node at 941b9661-74e5-5201-f2fe-39b3f009b868 [Follower]" leader-address= leader-id=
2026-05-25T23:23:36.817Z [INFO]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: entering leader state: leader="Node at 19c71118-7e61-cc98-9b5e-18e0d7ec6f6f [Leader]"
2026-05-25T23:23:36.816Z [INFO]  server-ce8eaa5e-b4e2-0509-43f1-878737fdf586: added peer, starting replication: peer=server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d
2026-05-25T23:23:36.816Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: aborting pipeline replication: peer="{Voter server-43deaf51-2664-9283-bde9-1056361f0668 43deaf51-2664-9283-bde9-1056361f0668}"
2026-05-25T23:23:36.817Z [INFO]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: added peer, starting replication: peer=server-164f4764-ebd7-bbaa-3367-321add0910e4
2026-05-25T23:23:36.817Z [INFO]  server-ce8eaa5e-b4e2-0509-43f1-878737fdf586: added peer, starting replication: peer=server-b77f1854-bc55-ea8e-5f00-80df26d0f03e
2026-05-25T23:23:36.816Z [INFO]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.816Z [INFO]  server-918e04a4-d2b8-6285-0572-1f11870065de: added peer, starting replication: peer=server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb
2026-05-25T23:23:36.816Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: added peer, starting replication: peer=server-ae4d03f3-6164-5c59-99d4-455db47640e4
2026-05-25T23:23:36.817Z [INFO]  server-918e04a4-d2b8-6285-0572-1f11870065de: added peer, starting replication: peer=server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058
2026-05-25T23:23:36.817Z [WARN]  server-ce8eaa5e-b4e2-0509-43f1-878737fdf586: rejecting vote request since we have a leader: from=5c8dd68b-1042-03a3-ddab-1955aa5a7d0d leader=ce8eaa5e-b4e2-0509-43f1-878737fdf586 leader-id=server-ce8eaa5e-b4e2-0509-43f1-878737fdf586
2026-05-25T23:23:36.816Z [INFO]  server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058: aborting pipeline replication: peer="{Voter server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb b2b88b2b-e5dc-316b-2db7-975d03d33dfb}"
2026-05-25T23:23:36.817Z [WARN]  server-918e04a4-d2b8-6285-0572-1f11870065de: rejecting vote request since we have a leader: from=b2b88b2b-e5dc-316b-2db7-975d03d33dfb leader=918e04a4-d2b8-6285-0572-1f11870065de leader-id=server-918e04a4-d2b8-6285-0572-1f11870065de
2026-05-25T23:23:36.816Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: pipelining replication: peer="{Voter server-bb5ff54a-16e0-4125-6196-c4d8b3be7726 bb5ff54a-16e0-4125-6196-c4d8b3be7726}"
2026-05-25T23:23:36.817Z [INFO]  server-ae4d03f3-6164-5c59-99d4-455db47640e4: entering follower state: follower="Node at ae4d03f3-6164-5c59-99d4-455db47640e4 [Follower]" leader-address=8cdddb1f-717d-965c-d545-807129b2ce3b leader-id=server-8cdddb1f-717d-965c-d545-807129b2ce3b
2026-05-25T23:23:36.817Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: added peer, starting replication: peer=server-b2d831eb-15f8-614f-ec08-fafe997c4944
2026-05-25T23:23:36.816Z [ERROR] server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: peer has newer term, stopping replication: peer="{Voter server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940 8cf9d984-e9cf-b9ad-bd1c-e4135e071940}"
2026-05-25T23:23:36.816Z [WARN]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: failed to contact quorum of nodes, stepping down
2026-05-25T23:23:36.817Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: aborting pipeline replication: peer="{Voter server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940 8cf9d984-e9cf-b9ad-bd1c-e4135e071940}"
2026-05-25T23:23:36.817Z [INFO]  server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb: entering follower state: follower="Node at b2b88b2b-e5dc-316b-2db7-975d03d33dfb [Follower]" leader-address=918e04a4-d2b8-6285-0572-1f11870065de leader-id=server-918e04a4-d2b8-6285-0572-1f11870065de
2026-05-25T23:23:36.817Z [INFO]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: added peer, starting replication: peer=server-43deaf51-2664-9283-bde9-1056361f0668
2026-05-25T23:23:36.816Z [WARN]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: rejecting vote request since we have a leader: from=8cf9d984-e9cf-b9ad-bd1c-e4135e071940 leader=fcb5b37a-b3b5-51e6-b68a-69e91d5df710 leader-id=server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710
2026-05-25T23:23:36.816Z [ERROR] server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: peer has newer term, stopping replication: peer="{Voter server-cb02535b-4fa3-e7a6-9c73-0120936129ca cb02535b-4fa3-e7a6-9c73-0120936129ca}"
2026-05-25T23:23:36.817Z [INFO]  server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d: entering follower state: follower="Node at 5c8dd68b-1042-03a3-ddab-1955aa5a7d0d [Follower]" leader-address=ce8eaa5e-b4e2-0509-43f1-878737fdf586 leader-id=server-ce8eaa5e-b4e2-0509-43f1-878737fdf586
2026-05-25T23:23:36.817Z [INFO]  server-918e04a4-d2b8-6285-0572-1f11870065de: pipelining replication: peer="{Voter server-0bc7c1d4-cea2-25e5-ba38-79cde9eb4058 0bc7c1d4-cea2-25e5-ba38-79cde9eb4058}"
2026-05-25T23:23:36.817Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: aborting pipeline replication: peer="{Voter server-cb02535b-4fa3-e7a6-9c73-0120936129ca cb02535b-4fa3-e7a6-9c73-0120936129ca}"
2026-05-25T23:23:36.817Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: entering follower state: follower="Node at fcb5b37a-b3b5-51e6-b68a-69e91d5df710 [Follower]" leader-address= leader-id=
2026-05-25T23:23:36.817Z [INFO]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: pipelining replication: peer="{Voter server-164f4764-ebd7-bbaa-3367-321add0910e4 164f4764-ebd7-bbaa-3367-321add0910e4}"
2026-05-25T23:23:36.816Z [ERROR] server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: peer has newer term, stopping replication: peer="{Voter server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb bc1e4076-200c-c555-eb3f-ab3ef10bbedb}"
2026-05-25T23:23:36.817Z [INFO]  server-918e04a4-d2b8-6285-0572-1f11870065de: pipelining replication: peer="{Voter server-b2b88b2b-e5dc-316b-2db7-975d03d33dfb b2b88b2b-e5dc-316b-2db7-975d03d33dfb}"
2026-05-25T23:23:36.817Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: aborting pipeline replication: peer="{Voter server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb bc1e4076-200c-c555-eb3f-ab3ef10bbedb}"
2026-05-25T23:23:36.817Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: pipelining replication: peer="{Voter server-ae4d03f3-6164-5c59-99d4-455db47640e4 ae4d03f3-6164-5c59-99d4-455db47640e4}"
2026-05-25T23:23:36.817Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: entering follower state: follower="Node at eb7f8dda-b789-58f9-9aee-a08a5ed05fcc [Follower]" leader-address= leader-id=
2026-05-25T23:23:36.817Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: pipelining replication: peer="{Voter server-b2d831eb-15f8-614f-ec08-fafe997c4944 b2d831eb-15f8-614f-ec08-fafe997c4944}"
2026-05-25T23:23:36.817Z [INFO]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: duplicate requestVote for same term: term=3
2026-05-25T23:23:36.817Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: aborting pipeline replication: peer="{Voter server-1ea7c9cb-c351-9612-c976-e429888b54a7 1ea7c9cb-c351-9612-c976-e429888b54a7}"
2026-05-25T23:23:36.817Z [INFO]  server-43deaf51-2664-9283-bde9-1056361f0668: entering follower state: follower="Node at 43deaf51-2664-9283-bde9-1056361f0668 [Follower]" leader-address=19c71118-7e61-cc98-9b5e-18e0d7ec6f6f leader-id=server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f
2026-05-25T23:23:36.817Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: election won: term=3 tally=2
2026-05-25T23:23:36.817Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: entering leader state: leader="Node at cb02535b-4fa3-e7a6-9c73-0120936129ca [Leader]"
2026-05-25T23:23:36.817Z [INFO]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: pipelining replication: peer="{Voter server-43deaf51-2664-9283-bde9-1056361f0668 43deaf51-2664-9283-bde9-1056361f0668}"
2026-05-25T23:23:36.817Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: added peer, starting replication: peer=server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb
2026-05-25T23:23:36.817Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: added peer, starting replication: peer=server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc
2026-05-25T23:23:36.817Z [INFO]  server-ce8eaa5e-b4e2-0509-43f1-878737fdf586: pipelining replication: peer="{Voter server-b77f1854-bc55-ea8e-5f00-80df26d0f03e b77f1854-bc55-ea8e-5f00-80df26d0f03e}"
2026-05-25T23:23:36.817Z [INFO]  server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb: entering follower state: follower="Node at bc1e4076-200c-c555-eb3f-ab3ef10bbedb [Follower]" leader-address=cb02535b-4fa3-e7a6-9c73-0120936129ca leader-id=server-cb02535b-4fa3-e7a6-9c73-0120936129ca
2026-05-25T23:23:36.817Z [INFO]  server-ce8eaa5e-b4e2-0509-43f1-878737fdf586: pipelining replication: peer="{Voter server-5c8dd68b-1042-03a3-ddab-1955aa5a7d0d 5c8dd68b-1042-03a3-ddab-1955aa5a7d0d}"
2026-05-25T23:23:36.817Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: pipelining replication: peer="{Voter server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc eb7f8dda-b789-58f9-9aee-a08a5ed05fcc}"
2026-05-25T23:23:36.817Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: pipelining replication: peer="{Voter server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb bc1e4076-200c-c555-eb3f-ab3ef10bbedb}"
2026-05-25T23:23:37.315Z [ERROR] server-1fe26923-ea55-f0bd-0d7a-994021226697: failed to heartbeat to: peer=41037aef-e4cf-ea9c-8180-68ba707d267e backoff time=10ms error="command timed out"
2026-05-25T23:23:37.317Z [ERROR] server-17ad52e4-24c4-8844-49af-e3e46041efea: failed to appendEntries to: peer="{Voter server-41037aef-e4cf-ea9c-8180-68ba707d267e 41037aef-e4cf-ea9c-8180-68ba707d267e}" error="command timed out"
2026-05-25T23:23:37.317Z [ERROR] server-17ad52e4-24c4-8844-49af-e3e46041efea: failed to make requestVote RPC: target="{Voter server-41037aef-e4cf-ea9c-8180-68ba707d267e 41037aef-e4cf-ea9c-8180-68ba707d267e}" error="command timed out" term=3
2026-05-25T23:23:37.317Z [ERROR] server-41037aef-e4cf-ea9c-8180-68ba707d267e: failed to make requestVote RPC: target="{Voter server-1fe26923-ea55-f0bd-0d7a-994021226697 1fe26923-ea55-f0bd-0d7a-994021226697}" error="command timed out" term=3
2026-05-25T23:23:37.317Z [ERROR] server-17ad52e4-24c4-8844-49af-e3e46041efea: failed to appendEntries to: peer="{Voter server-1fe26923-ea55-f0bd-0d7a-994021226697 1fe26923-ea55-f0bd-0d7a-994021226697}" error="command timed out"
2026-05-25T23:23:37.317Z [ERROR] server-41037aef-e4cf-ea9c-8180-68ba707d267e: failed to make requestVote RPC: target="{Voter server-17ad52e4-24c4-8844-49af-e3e46041efea 17ad52e4-24c4-8844-49af-e3e46041efea}" error="command timed out" term=3
2026-05-25T23:23:37.385Z [ERROR] server-17ad52e4-24c4-8844-49af-e3e46041efea: failed to heartbeat to: peer=1fe26923-ea55-f0bd-0d7a-994021226697 backoff time=10ms error="command timed out"
2026-05-25T23:23:37.413Z [ERROR] server-17ad52e4-24c4-8844-49af-e3e46041efea: failed to heartbeat to: peer=41037aef-e4cf-ea9c-8180-68ba707d267e backoff time=10ms error="command timed out"
2026-05-25T23:23:37.428Z [WARN]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:37.428Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: entering candidate state: node="Node at fcb5b37a-b3b5-51e6-b68a-69e91d5df710 [Candidate]" term=3
2026-05-25T23:23:37.428Z [INFO]  server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940: duplicate requestVote for same term: term=3
2026-05-25T23:23:37.428Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: election won: term=3 tally=2
2026-05-25T23:23:37.428Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: entering leader state: leader="Node at fcb5b37a-b3b5-51e6-b68a-69e91d5df710 [Leader]"
2026-05-25T23:23:37.428Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: added peer, starting replication: peer=server-1ea7c9cb-c351-9612-c976-e429888b54a7
2026-05-25T23:23:37.428Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: added peer, starting replication: peer=server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940
2026-05-25T23:23:37.428Z [INFO]  server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940: entering follower state: follower="Node at 8cf9d984-e9cf-b9ad-bd1c-e4135e071940 [Follower]" leader-address=fcb5b37a-b3b5-51e6-b68a-69e91d5df710 leader-id=server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710
2026-05-25T23:23:37.428Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: pipelining replication: peer="{Voter server-8cf9d984-e9cf-b9ad-bd1c-e4135e071940 8cf9d984-e9cf-b9ad-bd1c-e4135e071940}"
2026-05-25T23:23:37.428Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: pipelining replication: peer="{Voter server-1ea7c9cb-c351-9612-c976-e429888b54a7 1ea7c9cb-c351-9612-c976-e429888b54a7}"
2026-05-25T23:23:37.490Z [WARN]  server-941b9661-74e5-5201-f2fe-39b3f009b868: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:37.490Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: entering candidate state: node="Node at 941b9661-74e5-5201-f2fe-39b3f009b868 [Candidate]" term=3
2026-05-25T23:23:37.490Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: duplicate requestVote for same term: term=3
2026-05-25T23:23:37.490Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: duplicate requestVote for same term: term=3
2026-05-25T23:23:37.741Z [WARN]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: Election timeout reached, restarting election
2026-05-25T23:23:37.741Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: entering candidate state: node="Node at f9a39ce0-d9e9-5faf-e108-7822f2b3993b [Candidate]" term=4
2026-05-25T23:23:37.741Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: entering follower state: follower="Node at 116a910e-5eb0-5371-9698-339f44ce34c6 [Follower]" leader-address= leader-id=
2026-05-25T23:23:37.741Z [INFO]  server-941b9661-74e5-5201-f2fe-39b3f009b868: entering follower state: follower="Node at 941b9661-74e5-5201-f2fe-39b3f009b868 [Follower]" leader-address= leader-id=
2026-05-25T23:23:37.741Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: election won: term=4 tally=2
2026-05-25T23:23:37.741Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: entering leader state: leader="Node at f9a39ce0-d9e9-5faf-e108-7822f2b3993b [Leader]"
2026-05-25T23:23:37.741Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: added peer, starting replication: peer=server-941b9661-74e5-5201-f2fe-39b3f009b868
2026-05-25T23:23:37.741Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: added peer, starting replication: peer=server-116a910e-5eb0-5371-9698-339f44ce34c6
2026-05-25T23:23:37.741Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: pipelining replication: peer="{Voter server-116a910e-5eb0-5371-9698-339f44ce34c6 116a910e-5eb0-5371-9698-339f44ce34c6}"
2026-05-25T23:23:37.741Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: pipelining replication: peer="{Voter server-941b9661-74e5-5201-f2fe-39b3f009b868 941b9661-74e5-5201-f2fe-39b3f009b868}"
2026-05-25T23:23:37.817Z [ERROR] server-17ad52e4-24c4-8844-49af-e3e46041efea: failed to appendEntries to: peer="{Voter server-1fe26923-ea55-f0bd-0d7a-994021226697 1fe26923-ea55-f0bd-0d7a-994021226697}" error="command timed out"
2026-05-25T23:23:37.959Z [ERROR] server-17ad52e4-24c4-8844-49af-e3e46041efea: failed to heartbeat to: peer=1fe26923-ea55-f0bd-0d7a-994021226697 backoff time=10ms error="command timed out"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false
2026-05-25T23:23:37.960Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138fe7b0-f07c-8f49-2712-1765769aead9 Address:138fe7b0-f07c-8f49-2712-1765769aead9} {Suffrage:Voter ID:server-2927e0a5-0bce-0288-f624-6a9e5e6b3460 Address:2927e0a5-0bce-0288-f624-6a9e5e6b3460} {Suffrage:Voter ID:server-d8d5ea84-513a-e836-8735-cbfab016c745 Address:d8d5ea84-513a-e836-8735-cbfab016c745}]"
2026-05-25T23:23:37.960Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: entering follower state: follower="Node at 138fe7b0-f07c-8f49-2712-1765769aead9 [Follower]" leader-address= leader-id=
2026-05-25T23:23:37.960Z [INFO]  server-2927e0a5-0bce-0288-f624-6a9e5e6b3460: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138fe7b0-f07c-8f49-2712-1765769aead9 Address:138fe7b0-f07c-8f49-2712-1765769aead9} {Suffrage:Voter ID:server-2927e0a5-0bce-0288-f624-6a9e5e6b3460 Address:2927e0a5-0bce-0288-f624-6a9e5e6b3460} {Suffrage:Voter ID:server-d8d5ea84-513a-e836-8735-cbfab016c745 Address:d8d5ea84-513a-e836-8735-cbfab016c745}]"
2026-05-25T23:23:37.960Z [INFO]  server-d8d5ea84-513a-e836-8735-cbfab016c745: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-138fe7b0-f07c-8f49-2712-1765769aead9 Address:138fe7b0-f07c-8f49-2712-1765769aead9} {Suffrage:Voter ID:server-2927e0a5-0bce-0288-f624-6a9e5e6b3460 Address:2927e0a5-0bce-0288-f624-6a9e5e6b3460} {Suffrage:Voter ID:server-d8d5ea84-513a-e836-8735-cbfab016c745 Address:d8d5ea84-513a-e836-8735-cbfab016c745}]"
2026-05-25T23:23:37.960Z [INFO]  server-2927e0a5-0bce-0288-f624-6a9e5e6b3460: entering follower state: follower="Node at 2927e0a5-0bce-0288-f624-6a9e5e6b3460 [Follower]" leader-address= leader-id=
2026-05-25T23:23:37.960Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:37.960Z [INFO]  server-d8d5ea84-513a-e836-8735-cbfab016c745: entering follower state: follower="Node at d8d5ea84-513a-e836-8735-cbfab016c745 [Follower]" leader-address= leader-id=
2026-05-25T23:23:38.634Z [WARN]  server-138fe7b0-f07c-8f49-2712-1765769aead9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:38.634Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: entering candidate state: node="Node at 138fe7b0-f07c-8f49-2712-1765769aead9 [Candidate]" term=2
2026-05-25T23:23:38.634Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: election won: term=2 tally=2
2026-05-25T23:23:38.634Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: entering leader state: leader="Node at 138fe7b0-f07c-8f49-2712-1765769aead9 [Leader]"
2026-05-25T23:23:38.634Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: added peer, starting replication: peer=server-2927e0a5-0bce-0288-f624-6a9e5e6b3460
2026-05-25T23:23:38.634Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: added peer, starting replication: peer=server-d8d5ea84-513a-e836-8735-cbfab016c745
2026-05-25T23:23:38.634Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: pipelining replication: peer="{Voter server-d8d5ea84-513a-e836-8735-cbfab016c745 d8d5ea84-513a-e836-8735-cbfab016c745}"
2026-05-25T23:23:38.634Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: pipelining replication: peer="{Voter server-2927e0a5-0bce-0288-f624-6a9e5e6b3460 2927e0a5-0bce-0288-f624-6a9e5e6b3460}"
2026-05-25T23:23:39.640Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:38.63427252 +0000 UTC m=+74.280720007 (1 nodes), 673.799677ms from start of poll, 674.142349ms from cluster start. Timeout at 2026-05-25 23:23:39.639277882 +0000 UTC m=+75.285725331, 1.005005324s after stability
2026-05-25T23:23:39.640Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: starting snapshot up to: index=12
2026-05-25T23:23:39.640Z [INFO]  creating new snapshot: path=/tmp/raft3257006237/snapshots/2-12-1779751419640.tmp
2026-05-25T23:23:39.641Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: snapshot complete up to: index=12
2026-05-25T23:23:39.642Z [INFO]  creating new snapshot: path=/tmp/raft3257006237/snapshots/2-1013-1779751419642.tmp
2026-05-25T23:23:39.643Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: copied to local snapshot: bytes=71
2026-05-25T23:23:39.644Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: snapshot restore progress: id=2-1013-1779751419642 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:39.645Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: removing all old logs from log store
2026-05-25T23:23:39.645Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: compacting logs: from=1 to=22
2026-05-25T23:23:39.645Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: restored user snapshot: index=1013
2026-05-25T23:23:39.645Z [ERROR] server-138fe7b0-f07c-8f49-2712-1765769aead9: failed to get log: index=22 error="log not found"
2026-05-25T23:23:39.645Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: aborting pipeline replication: peer="{Voter server-2927e0a5-0bce-0288-f624-6a9e5e6b3460 2927e0a5-0bce-0288-f624-6a9e5e6b3460}"
2026-05-25T23:23:39.645Z [ERROR] server-138fe7b0-f07c-8f49-2712-1765769aead9: failed to get log: index=22 error="log not found"
2026-05-25T23:23:39.645Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: aborting pipeline replication: peer="{Voter server-d8d5ea84-513a-e836-8735-cbfab016c745 d8d5ea84-513a-e836-8735-cbfab016c745}"
2026-05-25T23:23:39.651Z [ERROR] server-138fe7b0-f07c-8f49-2712-1765769aead9: failed to get log: index=22 error="log not found"
2026-05-25T23:23:39.651Z [ERROR] server-138fe7b0-f07c-8f49-2712-1765769aead9: failed to get log: index=22 error="log not found"
2026-05-25T23:23:39.651Z [INFO]  creating new snapshot: path=/tmp/raft2021571843/snapshots/2-1013-1779751419651.tmp
2026-05-25T23:23:39.651Z [INFO]  creating new snapshot: path=/tmp/raft39612850/snapshots/2-1013-1779751419651.tmp
2026-05-25T23:23:39.652Z [INFO]  server-2927e0a5-0bce-0288-f624-6a9e5e6b3460: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:39.652Z [INFO]  server-d8d5ea84-513a-e836-8735-cbfab016c745: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:39.652Z [INFO]  server-2927e0a5-0bce-0288-f624-6a9e5e6b3460: copied to local snapshot: bytes=71
2026-05-25T23:23:39.652Z [INFO]  server-d8d5ea84-513a-e836-8735-cbfab016c745: copied to local snapshot: bytes=71
2026-05-25T23:23:39.652Z [INFO]  server-2927e0a5-0bce-0288-f624-6a9e5e6b3460: snapshot restore progress: id=2-1013-1779751419651 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:39.652Z [INFO]  server-2927e0a5-0bce-0288-f624-6a9e5e6b3460: removing all old logs from log store
2026-05-25T23:23:39.652Z [INFO]  server-2927e0a5-0bce-0288-f624-6a9e5e6b3460: compacting logs: from=1 to=22
2026-05-25T23:23:39.652Z [INFO]  server-2927e0a5-0bce-0288-f624-6a9e5e6b3460: Installed remote snapshot
2026-05-25T23:23:39.652Z [INFO]  server-d8d5ea84-513a-e836-8735-cbfab016c745: snapshot restore progress: id=2-1013-1779751419651 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:39.652Z [WARN]  server-2927e0a5-0bce-0288-f624-6a9e5e6b3460: skipping application of old log: index=22
2026-05-25T23:23:39.652Z [INFO]  server-d8d5ea84-513a-e836-8735-cbfab016c745: removing all old logs from log store
2026-05-25T23:23:39.652Z [INFO]  server-d8d5ea84-513a-e836-8735-cbfab016c745: compacting logs: from=1 to=22
2026-05-25T23:23:39.652Z [INFO]  server-d8d5ea84-513a-e836-8735-cbfab016c745: Installed remote snapshot
2026-05-25T23:23:39.652Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: pipelining replication: peer="{Voter server-2927e0a5-0bce-0288-f624-6a9e5e6b3460 2927e0a5-0bce-0288-f624-6a9e5e6b3460}"
2026-05-25T23:23:39.652Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:23:39.652Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: pipelining replication: peer="{Voter server-d8d5ea84-513a-e836-8735-cbfab016c745 d8d5ea84-513a-e836-8735-cbfab016c745}"
2026-05-25T23:23:40.658Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:23:39.652802871 +0000 UTC m=+75.299250332 (2 nodes), 797ns from start of poll, 1.692672674s from cluster start. Timeout at 2026-05-25 23:23:40.657820932 +0000 UTC m=+76.304268405, 1.005018073s after stability
2026-05-25T23:23:40.670Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: aborting pipeline replication: peer="{Voter server-d8d5ea84-513a-e836-8735-cbfab016c745 d8d5ea84-513a-e836-8735-cbfab016c745}"
2026-05-25T23:23:40.670Z [INFO]  server-138fe7b0-f07c-8f49-2712-1765769aead9: aborting pipeline replication: peer="{Voter server-2927e0a5-0bce-0288-f624-6a9e5e6b3460 2927e0a5-0bce-0288-f624-6a9e5e6b3460}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true
2026-05-25T23:23:40.672Z [INFO]  server-ad6864f4-e593-03b3-1d33-74a079f1e1da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad6864f4-e593-03b3-1d33-74a079f1e1da Address:ad6864f4-e593-03b3-1d33-74a079f1e1da} {Suffrage:Voter ID:server-b5200793-f09e-895c-80bb-a57b84f4d144 Address:b5200793-f09e-895c-80bb-a57b84f4d144} {Suffrage:Voter ID:server-ae8a751f-e615-23ed-b191-dd23300f651b Address:ae8a751f-e615-23ed-b191-dd23300f651b}]"
2026-05-25T23:23:40.672Z [INFO]  server-ad6864f4-e593-03b3-1d33-74a079f1e1da: entering follower state: follower="Node at ad6864f4-e593-03b3-1d33-74a079f1e1da [Follower]" leader-address= leader-id=
2026-05-25T23:23:40.672Z [INFO]  server-b5200793-f09e-895c-80bb-a57b84f4d144: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad6864f4-e593-03b3-1d33-74a079f1e1da Address:ad6864f4-e593-03b3-1d33-74a079f1e1da} {Suffrage:Voter ID:server-b5200793-f09e-895c-80bb-a57b84f4d144 Address:b5200793-f09e-895c-80bb-a57b84f4d144} {Suffrage:Voter ID:server-ae8a751f-e615-23ed-b191-dd23300f651b Address:ae8a751f-e615-23ed-b191-dd23300f651b}]"
2026-05-25T23:23:40.672Z [INFO]  server-ae8a751f-e615-23ed-b191-dd23300f651b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad6864f4-e593-03b3-1d33-74a079f1e1da Address:ad6864f4-e593-03b3-1d33-74a079f1e1da} {Suffrage:Voter ID:server-b5200793-f09e-895c-80bb-a57b84f4d144 Address:b5200793-f09e-895c-80bb-a57b84f4d144} {Suffrage:Voter ID:server-ae8a751f-e615-23ed-b191-dd23300f651b Address:ae8a751f-e615-23ed-b191-dd23300f651b}]"
2026-05-25T23:23:40.672Z [INFO]  server-b5200793-f09e-895c-80bb-a57b84f4d144: entering follower state: follower="Node at b5200793-f09e-895c-80bb-a57b84f4d144 [Follower]" leader-address= leader-id=
2026-05-25T23:23:40.672Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:40.672Z [INFO]  server-ae8a751f-e615-23ed-b191-dd23300f651b: entering follower state: follower="Node at ae8a751f-e615-23ed-b191-dd23300f651b [Follower]" leader-address= leader-id=
2026-05-25T23:23:41.546Z [WARN]  server-ae8a751f-e615-23ed-b191-dd23300f651b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:41.546Z [INFO]  server-ae8a751f-e615-23ed-b191-dd23300f651b: entering candidate state: node="Node at ae8a751f-e615-23ed-b191-dd23300f651b [Candidate]" term=2
2026-05-25T23:23:41.546Z [INFO]  server-ae8a751f-e615-23ed-b191-dd23300f651b: election won: term=2 tally=2
2026-05-25T23:23:41.546Z [INFO]  server-ae8a751f-e615-23ed-b191-dd23300f651b: entering leader state: leader="Node at ae8a751f-e615-23ed-b191-dd23300f651b [Leader]"
2026-05-25T23:23:41.546Z [INFO]  server-ae8a751f-e615-23ed-b191-dd23300f651b: added peer, starting replication: peer=server-ad6864f4-e593-03b3-1d33-74a079f1e1da
2026-05-25T23:23:41.546Z [INFO]  server-ae8a751f-e615-23ed-b191-dd23300f651b: added peer, starting replication: peer=server-b5200793-f09e-895c-80bb-a57b84f4d144
2026-05-25T23:23:41.546Z [INFO]  server-ae8a751f-e615-23ed-b191-dd23300f651b: pipelining replication: peer="{Voter server-ad6864f4-e593-03b3-1d33-74a079f1e1da ad6864f4-e593-03b3-1d33-74a079f1e1da}"
2026-05-25T23:23:41.546Z [INFO]  server-ae8a751f-e615-23ed-b191-dd23300f651b: pipelining replication: peer="{Voter server-b5200793-f09e-895c-80bb-a57b84f4d144 b5200793-f09e-895c-80bb-a57b84f4d144}"
2026-05-25T23:23:42.551Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:41.546504282 +0000 UTC m=+77.192951737 (1 nodes), 873.984088ms from start of poll, 874.355414ms from cluster start. Timeout at 2026-05-25 23:23:42.551506072 +0000 UTC m=+78.197953559, 1.005001822s after stability
2026-05-25T23:23:42.552Z [INFO]  server-ae8a751f-e615-23ed-b191-dd23300f651b: starting snapshot up to: index=12
2026-05-25T23:23:42.552Z [INFO]  creating new snapshot: path=/tmp/raft602914042/snapshots/2-12-1779751422552.tmp
2026-05-25T23:23:42.587Z [INFO]  server-ae8a751f-e615-23ed-b191-dd23300f651b: snapshot complete up to: index=12
2026-05-25T23:23:42.589Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96cde625-cbd0-e627-9d39-f11ce9c09632 Address:96cde625-cbd0-e627-9d39-f11ce9c09632} {Suffrage:Voter ID:server-db84d153-4f4c-fe2f-1350-1b880454877a Address:db84d153-4f4c-fe2f-1350-1b880454877a} {Suffrage:Voter ID:server-6315bed1-621a-3f3c-64e5-6197c9662cd5 Address:6315bed1-621a-3f3c-64e5-6197c9662cd5}]"
2026-05-25T23:23:42.589Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: entering follower state: follower="Node at 96cde625-cbd0-e627-9d39-f11ce9c09632 [Follower]" leader-address= leader-id=
2026-05-25T23:23:42.589Z [INFO]  server-db84d153-4f4c-fe2f-1350-1b880454877a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96cde625-cbd0-e627-9d39-f11ce9c09632 Address:96cde625-cbd0-e627-9d39-f11ce9c09632} {Suffrage:Voter ID:server-db84d153-4f4c-fe2f-1350-1b880454877a Address:db84d153-4f4c-fe2f-1350-1b880454877a} {Suffrage:Voter ID:server-6315bed1-621a-3f3c-64e5-6197c9662cd5 Address:6315bed1-621a-3f3c-64e5-6197c9662cd5}]"
2026-05-25T23:23:42.590Z [INFO]  server-db84d153-4f4c-fe2f-1350-1b880454877a: entering follower state: follower="Node at db84d153-4f4c-fe2f-1350-1b880454877a [Follower]" leader-address= leader-id=
2026-05-25T23:23:42.590Z [INFO]  server-6315bed1-621a-3f3c-64e5-6197c9662cd5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96cde625-cbd0-e627-9d39-f11ce9c09632 Address:96cde625-cbd0-e627-9d39-f11ce9c09632} {Suffrage:Voter ID:server-db84d153-4f4c-fe2f-1350-1b880454877a Address:db84d153-4f4c-fe2f-1350-1b880454877a} {Suffrage:Voter ID:server-6315bed1-621a-3f3c-64e5-6197c9662cd5 Address:6315bed1-621a-3f3c-64e5-6197c9662cd5}]"
2026-05-25T23:23:42.590Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:42.590Z [INFO]  server-6315bed1-621a-3f3c-64e5-6197c9662cd5: entering follower state: follower="Node at 6315bed1-621a-3f3c-64e5-6197c9662cd5 [Follower]" leader-address= leader-id=
2026-05-25T23:23:43.122Z [WARN]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:43.122Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: entering candidate state: node="Node at 96cde625-cbd0-e627-9d39-f11ce9c09632 [Candidate]" term=2
2026-05-25T23:23:43.122Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: election won: term=2 tally=2
2026-05-25T23:23:43.122Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: entering leader state: leader="Node at 96cde625-cbd0-e627-9d39-f11ce9c09632 [Leader]"
2026-05-25T23:23:43.122Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: added peer, starting replication: peer=server-db84d153-4f4c-fe2f-1350-1b880454877a
2026-05-25T23:23:43.122Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: added peer, starting replication: peer=server-6315bed1-621a-3f3c-64e5-6197c9662cd5
2026-05-25T23:23:43.122Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: pipelining replication: peer="{Voter server-db84d153-4f4c-fe2f-1350-1b880454877a db84d153-4f4c-fe2f-1350-1b880454877a}"
2026-05-25T23:23:43.122Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: pipelining replication: peer="{Voter server-6315bed1-621a-3f3c-64e5-6197c9662cd5 6315bed1-621a-3f3c-64e5-6197c9662cd5}"
2026-05-25T23:23:44.127Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:43.122655653 +0000 UTC m=+78.769103102 (1 nodes), 532.557555ms from start of poll, 533.36706ms from cluster start. Timeout at 2026-05-25 23:23:44.127658843 +0000 UTC m=+79.774106294, 1.005003192s after stability
2026-05-25T23:23:44.128Z [INFO]  creating new snapshot: path=/tmp/raft3732071074/snapshots/2-1013-1779751424128.tmp
2026-05-25T23:23:44.128Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: copied to local snapshot: bytes=71
2026-05-25T23:23:44.128Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: snapshot restore progress: id=2-1013-1779751424128 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:44.128Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: restored user snapshot: index=1013
2026-05-25T23:23:44.128Z [ERROR] server-96cde625-cbd0-e627-9d39-f11ce9c09632: failed to get log: index=3 error="log not found"
2026-05-25T23:23:44.128Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: aborting pipeline replication: peer="{Voter server-6315bed1-621a-3f3c-64e5-6197c9662cd5 6315bed1-621a-3f3c-64e5-6197c9662cd5}"
2026-05-25T23:23:44.128Z [ERROR] server-96cde625-cbd0-e627-9d39-f11ce9c09632: failed to get log: index=3 error="log not found"
2026-05-25T23:23:44.128Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: aborting pipeline replication: peer="{Voter server-db84d153-4f4c-fe2f-1350-1b880454877a db84d153-4f4c-fe2f-1350-1b880454877a}"
2026-05-25T23:23:44.134Z [ERROR] server-96cde625-cbd0-e627-9d39-f11ce9c09632: failed to get log: index=3 error="log not found"
2026-05-25T23:23:44.134Z [INFO]  creating new snapshot: path=/tmp/raft526240120/snapshots/2-1013-1779751424134.tmp
2026-05-25T23:23:44.135Z [INFO]  server-6315bed1-621a-3f3c-64e5-6197c9662cd5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:44.135Z [INFO]  server-6315bed1-621a-3f3c-64e5-6197c9662cd5: copied to local snapshot: bytes=71
2026-05-25T23:23:44.135Z [INFO]  server-6315bed1-621a-3f3c-64e5-6197c9662cd5: snapshot restore progress: id=2-1013-1779751424134 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:44.135Z [INFO]  server-6315bed1-621a-3f3c-64e5-6197c9662cd5: Installed remote snapshot
2026-05-25T23:23:44.135Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: pipelining replication: peer="{Voter server-6315bed1-621a-3f3c-64e5-6197c9662cd5 6315bed1-621a-3f3c-64e5-6197c9662cd5}"
2026-05-25T23:23:44.135Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:23:44.136Z [ERROR] server-96cde625-cbd0-e627-9d39-f11ce9c09632: failed to get log: index=3 error="log not found"
2026-05-25T23:23:44.137Z [INFO]  creating new snapshot: path=/tmp/raft3489780886/snapshots/2-1013-1779751424137.tmp
2026-05-25T23:23:44.137Z [INFO]  server-db84d153-4f4c-fe2f-1350-1b880454877a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:44.137Z [INFO]  server-db84d153-4f4c-fe2f-1350-1b880454877a: copied to local snapshot: bytes=71
2026-05-25T23:23:44.137Z [INFO]  server-db84d153-4f4c-fe2f-1350-1b880454877a: snapshot restore progress: id=2-1013-1779751424137 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:44.137Z [INFO]  server-db84d153-4f4c-fe2f-1350-1b880454877a: Installed remote snapshot
2026-05-25T23:23:44.138Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: pipelining replication: peer="{Voter server-db84d153-4f4c-fe2f-1350-1b880454877a db84d153-4f4c-fe2f-1350-1b880454877a}"
2026-05-25T23:23:45.141Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:23:44.13573988 +0000 UTC m=+79.782187334 (2 nodes), 729ns from start of poll, 1.546451292s from cluster start. Timeout at 2026-05-25 23:23:45.140825504 +0000 UTC m=+80.787272983, 1.005085649s after stability
2026-05-25T23:23:45.153Z [INFO]  server-ae8a751f-e615-23ed-b191-dd23300f651b: aborting pipeline replication: peer="{Voter server-b5200793-f09e-895c-80bb-a57b84f4d144 b5200793-f09e-895c-80bb-a57b84f4d144}"
2026-05-25T23:23:45.153Z [INFO]  server-ae8a751f-e615-23ed-b191-dd23300f651b: aborting pipeline replication: peer="{Voter server-ad6864f4-e593-03b3-1d33-74a079f1e1da ad6864f4-e593-03b3-1d33-74a079f1e1da}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true
2026-05-25T23:23:45.154Z [INFO]  server-90bc4db0-3d8a-7393-fb02-011e0680e623: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90bc4db0-3d8a-7393-fb02-011e0680e623 Address:90bc4db0-3d8a-7393-fb02-011e0680e623} {Suffrage:Voter ID:server-bef12e0a-96b3-9cb5-cc15-12bf93a7d6c9 Address:bef12e0a-96b3-9cb5-cc15-12bf93a7d6c9} {Suffrage:Voter ID:server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3 Address:8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3}]"
2026-05-25T23:23:45.154Z [INFO]  server-90bc4db0-3d8a-7393-fb02-011e0680e623: entering follower state: follower="Node at 90bc4db0-3d8a-7393-fb02-011e0680e623 [Follower]" leader-address= leader-id=
2026-05-25T23:23:45.154Z [INFO]  server-bef12e0a-96b3-9cb5-cc15-12bf93a7d6c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90bc4db0-3d8a-7393-fb02-011e0680e623 Address:90bc4db0-3d8a-7393-fb02-011e0680e623} {Suffrage:Voter ID:server-bef12e0a-96b3-9cb5-cc15-12bf93a7d6c9 Address:bef12e0a-96b3-9cb5-cc15-12bf93a7d6c9} {Suffrage:Voter ID:server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3 Address:8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3}]"
2026-05-25T23:23:45.155Z [INFO]  server-bef12e0a-96b3-9cb5-cc15-12bf93a7d6c9: entering follower state: follower="Node at bef12e0a-96b3-9cb5-cc15-12bf93a7d6c9 [Follower]" leader-address= leader-id=
2026-05-25T23:23:45.155Z [INFO]  server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90bc4db0-3d8a-7393-fb02-011e0680e623 Address:90bc4db0-3d8a-7393-fb02-011e0680e623} {Suffrage:Voter ID:server-bef12e0a-96b3-9cb5-cc15-12bf93a7d6c9 Address:bef12e0a-96b3-9cb5-cc15-12bf93a7d6c9} {Suffrage:Voter ID:server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3 Address:8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3}]"
2026-05-25T23:23:45.155Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:45.155Z [INFO]  server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3: entering follower state: follower="Node at 8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3 [Follower]" leader-address= leader-id=
2026-05-25T23:23:45.702Z [WARN]  server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:45.702Z [INFO]  server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3: entering candidate state: node="Node at 8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3 [Candidate]" term=2
2026-05-25T23:23:45.703Z [INFO]  server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3: election won: term=2 tally=2
2026-05-25T23:23:45.703Z [INFO]  server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3: entering leader state: leader="Node at 8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3 [Leader]"
2026-05-25T23:23:45.704Z [INFO]  server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3: added peer, starting replication: peer=server-90bc4db0-3d8a-7393-fb02-011e0680e623
2026-05-25T23:23:45.704Z [INFO]  server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3: added peer, starting replication: peer=server-bef12e0a-96b3-9cb5-cc15-12bf93a7d6c9
2026-05-25T23:23:45.704Z [INFO]  server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3: pipelining replication: peer="{Voter server-90bc4db0-3d8a-7393-fb02-011e0680e623 90bc4db0-3d8a-7393-fb02-011e0680e623}"
2026-05-25T23:23:45.704Z [INFO]  server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3: pipelining replication: peer="{Voter server-bef12e0a-96b3-9cb5-cc15-12bf93a7d6c9 bef12e0a-96b3-9cb5-cc15-12bf93a7d6c9}"
2026-05-25T23:23:47.529Z [ERROR] server-cb02535b-4fa3-e7a6-9c73-0120936129ca: failed to heartbeat to: peer=bc1e4076-200c-c555-eb3f-ab3ef10bbedb backoff time=10ms error="command timed out"
2026-05-25T23:23:47.529Z [ERROR] server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: failed to heartbeat to: peer=164f4764-ebd7-bbaa-3367-321add0910e4 backoff time=10ms error="command timed out"
2026-05-25T23:23:47.529Z [ERROR] server-ce8eaa5e-b4e2-0509-43f1-878737fdf586: failed to heartbeat to: peer=b77f1854-bc55-ea8e-5f00-80df26d0f03e backoff time=10ms error="command timed out"
2026-05-25T23:23:47.529Z [ERROR] server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: failed to heartbeat to: peer=1ea7c9cb-c351-9612-c976-e429888b54a7 backoff time=10ms error="command timed out"
2026-05-25T23:23:47.529Z [ERROR] server-96cde625-cbd0-e627-9d39-f11ce9c09632: failed to heartbeat to: peer=db84d153-4f4c-fe2f-1350-1b880454877a backoff time=10ms error="command timed out"
2026-05-25T23:23:47.529Z [ERROR] server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3: failed to heartbeat to: peer=90bc4db0-3d8a-7393-fb02-011e0680e623 backoff time=10ms error="command timed out"
2026-05-25T23:23:47.529Z [ERROR] server-8cdddb1f-717d-965c-d545-807129b2ce3b: failed to heartbeat to: peer=ae4d03f3-6164-5c59-99d4-455db47640e4 backoff time=10ms error="command timed out"
2026-05-25T23:23:47.529Z [ERROR] server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: failed to heartbeat to: peer=bb5ff54a-16e0-4125-6196-c4d8b3be7726 backoff time=10ms error="command timed out"
2026-05-25T23:23:47.529Z [ERROR] server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: failed to heartbeat to: peer=941b9661-74e5-5201-f2fe-39b3f009b868 backoff time=10ms error="command timed out"
2026-05-25T23:23:47.529Z [WARN]  server-bb5ff54a-16e0-4125-6196-c4d8b3be7726: heartbeat timeout reached, starting election: last-leader-addr=eedc65c4-92af-c6a4-c7ee-60bd0065ac4a last-leader-id=server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a
2026-05-25T23:23:47.529Z [WARN]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: heartbeat timeout reached, starting election: last-leader-addr=eedc65c4-92af-c6a4-c7ee-60bd0065ac4a last-leader-id=server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a
2026-05-25T23:23:47.529Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: entering candidate state: node="Node at 5aff39f7-caf5-0cca-b85b-cefd035d60f7 [Candidate]" term=4
2026-05-25T23:23:47.529Z [INFO]  server-bb5ff54a-16e0-4125-6196-c4d8b3be7726: entering candidate state: node="Node at bb5ff54a-16e0-4125-6196-c4d8b3be7726 [Candidate]" term=4
2026-05-25T23:23:47.529Z [WARN]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: failed to contact quorum of nodes, stepping down
2026-05-25T23:23:47.529Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:45.704046587 +0000 UTC m=+81.350494089 (1 nodes), 548.916421ms from start of poll, 549.262891ms from cluster start. Timeout at 2026-05-25 23:23:46.709076927 +0000 UTC m=+82.355524387, 1.005030298s after stability
2026-05-25T23:23:47.529Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: entering follower state: follower="Node at f9a39ce0-d9e9-5faf-e108-7822f2b3993b [Follower]" leader-address= leader-id=
2026-05-25T23:23:47.529Z [ERROR] server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3: failed to heartbeat to: peer=bef12e0a-96b3-9cb5-cc15-12bf93a7d6c9 backoff time=10ms error="command timed out"
2026-05-25T23:23:47.529Z [WARN]  server-ae4d03f3-6164-5c59-99d4-455db47640e4: heartbeat timeout reached, starting election: last-leader-addr=8cdddb1f-717d-965c-d545-807129b2ce3b last-leader-id=server-8cdddb1f-717d-965c-d545-807129b2ce3b
2026-05-25T23:23:47.529Z [INFO]  server-ae4d03f3-6164-5c59-99d4-455db47640e4: entering candidate state: node="Node at ae4d03f3-6164-5c59-99d4-455db47640e4 [Candidate]" term=4
2026-05-25T23:23:47.529Z [ERROR] server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: failed to heartbeat to: peer=8cf9d984-e9cf-b9ad-bd1c-e4135e071940 backoff time=10ms error="command timed out"
2026-05-25T23:23:47.529Z [ERROR] server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: failed to heartbeat to: peer=5aff39f7-caf5-0cca-b85b-cefd035d60f7 backoff time=10ms error="command timed out"
2026-05-25T23:23:47.529Z [WARN]  server-90bc4db0-3d8a-7393-fb02-011e0680e623: heartbeat timeout reached, starting election: last-leader-addr=8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3 last-leader-id=server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3
2026-05-25T23:23:47.529Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: aborting pipeline replication: peer="{Voter server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc eb7f8dda-b789-58f9-9aee-a08a5ed05fcc}"
2026-05-25T23:23:47.529Z [WARN]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: rejecting vote request since we have a leader: from=ae4d03f3-6164-5c59-99d4-455db47640e4 leader=8cdddb1f-717d-965c-d545-807129b2ce3b leader-id=server-8cdddb1f-717d-965c-d545-807129b2ce3b
2026-05-25T23:23:47.529Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: aborting pipeline replication: peer="{Voter server-1ea7c9cb-c351-9612-c976-e429888b54a7 1ea7c9cb-c351-9612-c976-e429888b54a7}"
2026-05-25T23:23:47.529Z [INFO]  server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3: aborting pipeline replication: peer="{Voter server-bef12e0a-96b3-9cb5-cc15-12bf93a7d6c9 bef12e0a-96b3-9cb5-cc15-12bf93a7d6c9}"
2026-05-25T23:23:47.529Z [WARN]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: failed to contact quorum of nodes, stepping down
2026-05-25T23:23:47.529Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: aborting pipeline replication: peer="{Voter server-6315bed1-621a-3f3c-64e5-6197c9662cd5 6315bed1-621a-3f3c-64e5-6197c9662cd5}"
2026-05-25T23:23:47.529Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: aborting pipeline replication: peer="{Voter server-ae4d03f3-6164-5c59-99d4-455db47640e4 ae4d03f3-6164-5c59-99d4-455db47640e4}"
2026-05-25T23:23:47.529Z [ERROR] server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: failed to heartbeat to: peer=116a910e-5eb0-5371-9698-339f44ce34c6 backoff time=10ms error="command timed out"
2026-05-25T23:23:47.529Z [INFO]  server-90bc4db0-3d8a-7393-fb02-011e0680e623: entering candidate state: node="Node at 90bc4db0-3d8a-7393-fb02-011e0680e623 [Candidate]" term=3
2026-05-25T23:23:47.529Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: entering follower state: follower="Node at eedc65c4-92af-c6a4-c7ee-60bd0065ac4a [Follower]" leader-address= leader-id=
2026-05-25T23:23:47.529Z [WARN]  server-164f4764-ebd7-bbaa-3367-321add0910e4: heartbeat timeout reached, starting election: last-leader-addr=19c71118-7e61-cc98-9b5e-18e0d7ec6f6f last-leader-id=server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f
2026-05-25T23:23:47.529Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: entering candidate state: node="Node at 164f4764-ebd7-bbaa-3367-321add0910e4 [Candidate]" term=4
2026-05-25T23:23:47.529Z [WARN]  server-b2d831eb-15f8-614f-ec08-fafe997c4944: rejecting vote request since we have a leader: from=ae4d03f3-6164-5c59-99d4-455db47640e4 leader=8cdddb1f-717d-965c-d545-807129b2ce3b leader-id=server-8cdddb1f-717d-965c-d545-807129b2ce3b
2026-05-25T23:23:47.529Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: duplicate requestVote for same term: term=4
2026-05-25T23:23:47.529Z [WARN]  server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3: rejecting vote request since we have a leader: from=90bc4db0-3d8a-7393-fb02-011e0680e623 leader=8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3 leader-id=server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3
2026-05-25T23:23:47.529Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: aborting pipeline replication: peer="{Voter server-116a910e-5eb0-5371-9698-339f44ce34c6 116a910e-5eb0-5371-9698-339f44ce34c6}"
2026-05-25T23:23:47.529Z [WARN]  server-43deaf51-2664-9283-bde9-1056361f0668: rejecting vote request since we have a leader: from=164f4764-ebd7-bbaa-3367-321add0910e4 leader=19c71118-7e61-cc98-9b5e-18e0d7ec6f6f leader-id=server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f
2026-05-25T23:23:47.529Z [WARN]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: rejecting vote request since we have a leader: from=164f4764-ebd7-bbaa-3367-321add0910e4 leader=19c71118-7e61-cc98-9b5e-18e0d7ec6f6f leader-id=server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f
2026-05-25T23:23:47.529Z [INFO]  server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3: aborting pipeline replication: peer="{Voter server-90bc4db0-3d8a-7393-fb02-011e0680e623 90bc4db0-3d8a-7393-fb02-011e0680e623}"
2026-05-25T23:23:47.529Z [WARN]  server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb: heartbeat timeout reached, starting election: last-leader-addr=cb02535b-4fa3-e7a6-9c73-0120936129ca last-leader-id=server-cb02535b-4fa3-e7a6-9c73-0120936129ca
2026-05-25T23:23:47.529Z [ERROR] server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: peer has newer term, stopping replication: peer="{Voter server-bb5ff54a-16e0-4125-6196-c4d8b3be7726 bb5ff54a-16e0-4125-6196-c4d8b3be7726}"
2026-05-25T23:23:47.529Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: aborting pipeline replication: peer="{Voter server-bb5ff54a-16e0-4125-6196-c4d8b3be7726 bb5ff54a-16e0-4125-6196-c4d8b3be7726}"
2026-05-25T23:23:47.529Z [INFO]  server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb: entering candidate state: node="Node at bc1e4076-200c-c555-eb3f-ab3ef10bbedb [Candidate]" term=4
2026-05-25T23:23:47.530Z [WARN]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: rejecting vote request since we have a leader: from=bc1e4076-200c-c555-eb3f-ab3ef10bbedb leader=cb02535b-4fa3-e7a6-9c73-0120936129ca leader-id=server-cb02535b-4fa3-e7a6-9c73-0120936129ca
2026-05-25T23:23:47.530Z [INFO]  server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a: aborting pipeline replication: peer="{Voter server-5aff39f7-caf5-0cca-b85b-cefd035d60f7 5aff39f7-caf5-0cca-b85b-cefd035d60f7}"
2026-05-25T23:23:47.530Z [WARN]  server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc: rejecting vote request since we have a leader: from=bc1e4076-200c-c555-eb3f-ab3ef10bbedb leader=cb02535b-4fa3-e7a6-9c73-0120936129ca leader-id=server-cb02535b-4fa3-e7a6-9c73-0120936129ca
2026-05-25T23:23:47.530Z [ERROR] server-cb02535b-4fa3-e7a6-9c73-0120936129ca: peer has newer term, stopping replication: peer="{Voter server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb bc1e4076-200c-c555-eb3f-ab3ef10bbedb}"
2026-05-25T23:23:47.530Z [INFO]  server-bb5ff54a-16e0-4125-6196-c4d8b3be7726: duplicate requestVote for same term: term=4
2026-05-25T23:23:47.530Z [ERROR] server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3: peer has newer term, stopping replication: peer="{Voter server-90bc4db0-3d8a-7393-fb02-011e0680e623 90bc4db0-3d8a-7393-fb02-011e0680e623}"
2026-05-25T23:23:47.530Z [WARN]  server-bef12e0a-96b3-9cb5-cc15-12bf93a7d6c9: rejecting vote request since we have a leader: from=90bc4db0-3d8a-7393-fb02-011e0680e623 leader=8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3 leader-id=server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3
2026-05-25T23:23:47.530Z [INFO]  server-8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3: entering follower state: follower="Node at 8bdcea1a-5d0d-49b2-2024-7a7ba5aa84b3 [Follower]" leader-address= leader-id=
2026-05-25T23:23:47.530Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: election won: term=4 tally=2
2026-05-25T23:23:47.530Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: aborting pipeline replication: peer="{Voter server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb bc1e4076-200c-c555-eb3f-ab3ef10bbedb}"
2026-05-25T23:23:47.530Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: entering leader state: leader="Node at 5aff39f7-caf5-0cca-b85b-cefd035d60f7 [Leader]"
2026-05-25T23:23:47.530Z [INFO]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: aborting pipeline replication: peer="{Voter server-164f4764-ebd7-bbaa-3367-321add0910e4 164f4764-ebd7-bbaa-3367-321add0910e4}"
2026-05-25T23:23:47.530Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: entering follower state: follower="Node at cb02535b-4fa3-e7a6-9c73-0120936129ca [Follower]" leader-address= leader-id=
2026-05-25T23:23:47.530Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: added peer, starting replication: peer=server-bb5ff54a-16e0-4125-6196-c4d8b3be7726
2026-05-25T23:23:47.530Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: added peer, starting replication: peer=server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a
    raft_test.go:1598: Error Apply new log entries: node is not the leader
2026-05-25T23:23:47.530Z [INFO]  server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b: aborting pipeline replication: peer="{Voter server-941b9661-74e5-5201-f2fe-39b3f009b868 941b9661-74e5-5201-f2fe-39b3f009b868}"
2026-05-25T23:23:47.530Z [WARN]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: rejecting vote request since we have a leader: from=bb5ff54a-16e0-4125-6196-c4d8b3be7726 leader=5aff39f7-caf5-0cca-b85b-cefd035d60f7 leader-id=server-5aff39f7-caf5-0cca-b85b-cefd035d60f7
2026-05-25T23:23:47.530Z [INFO]  server-bb5ff54a-16e0-4125-6196-c4d8b3be7726: entering follower state: follower="Node at bb5ff54a-16e0-4125-6196-c4d8b3be7726 [Follower]" leader-address=5aff39f7-caf5-0cca-b85b-cefd035d60f7 leader-id=server-5aff39f7-caf5-0cca-b85b-cefd035d60f7
2026-05-25T23:23:47.530Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: pipelining replication: peer="{Voter server-eedc65c4-92af-c6a4-c7ee-60bd0065ac4a eedc65c4-92af-c6a4-c7ee-60bd0065ac4a}"
2026-05-25T23:23:47.530Z [INFO]  server-5aff39f7-caf5-0cca-b85b-cefd035d60f7: pipelining replication: peer="{Voter server-bb5ff54a-16e0-4125-6196-c4d8b3be7726 bb5ff54a-16e0-4125-6196-c4d8b3be7726}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false
2026-05-25T23:23:47.531Z [INFO]  server-63bc1d21-8079-9f02-20e6-9721519814c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63bc1d21-8079-9f02-20e6-9721519814c9 Address:63bc1d21-8079-9f02-20e6-9721519814c9} {Suffrage:Voter ID:server-8e548790-85c7-0f6d-0ba9-e92d7ab6e103 Address:8e548790-85c7-0f6d-0ba9-e92d7ab6e103} {Suffrage:Voter ID:server-4b61b004-1434-9acb-8773-7df6a7a770fd Address:4b61b004-1434-9acb-8773-7df6a7a770fd}]"
2026-05-25T23:23:47.532Z [INFO]  server-8e548790-85c7-0f6d-0ba9-e92d7ab6e103: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63bc1d21-8079-9f02-20e6-9721519814c9 Address:63bc1d21-8079-9f02-20e6-9721519814c9} {Suffrage:Voter ID:server-8e548790-85c7-0f6d-0ba9-e92d7ab6e103 Address:8e548790-85c7-0f6d-0ba9-e92d7ab6e103} {Suffrage:Voter ID:server-4b61b004-1434-9acb-8773-7df6a7a770fd Address:4b61b004-1434-9acb-8773-7df6a7a770fd}]"
2026-05-25T23:23:47.532Z [INFO]  server-63bc1d21-8079-9f02-20e6-9721519814c9: entering follower state: follower="Node at 63bc1d21-8079-9f02-20e6-9721519814c9 [Follower]" leader-address= leader-id=
2026-05-25T23:23:47.532Z [INFO]  server-8e548790-85c7-0f6d-0ba9-e92d7ab6e103: entering follower state: follower="Node at 8e548790-85c7-0f6d-0ba9-e92d7ab6e103 [Follower]" leader-address= leader-id=
2026-05-25T23:23:47.532Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63bc1d21-8079-9f02-20e6-9721519814c9 Address:63bc1d21-8079-9f02-20e6-9721519814c9} {Suffrage:Voter ID:server-8e548790-85c7-0f6d-0ba9-e92d7ab6e103 Address:8e548790-85c7-0f6d-0ba9-e92d7ab6e103} {Suffrage:Voter ID:server-4b61b004-1434-9acb-8773-7df6a7a770fd Address:4b61b004-1434-9acb-8773-7df6a7a770fd}]"
2026-05-25T23:23:47.532Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:47.532Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: entering follower state: follower="Node at 4b61b004-1434-9acb-8773-7df6a7a770fd [Follower]" leader-address= leader-id=
2026-05-25T23:23:47.537Z [ERROR] server-8cdddb1f-717d-965c-d545-807129b2ce3b: peer has newer term, stopping replication: peer="{Voter server-ae4d03f3-6164-5c59-99d4-455db47640e4 ae4d03f3-6164-5c59-99d4-455db47640e4}"
2026-05-25T23:23:47.537Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: entering follower state: follower="Node at 8cdddb1f-717d-965c-d545-807129b2ce3b [Follower]" leader-address= leader-id=
2026-05-25T23:23:47.537Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: aborting pipeline replication: peer="{Voter server-b2d831eb-15f8-614f-ec08-fafe997c4944 b2d831eb-15f8-614f-ec08-fafe997c4944}"
2026-05-25T23:23:47.538Z [INFO]  server-fcb5b37a-b3b5-51e6-b68a-69e91d5df710: pipelining replication: peer="{Voter server-1ea7c9cb-c351-9612-c976-e429888b54a7 1ea7c9cb-c351-9612-c976-e429888b54a7}"
2026-05-25T23:23:47.538Z [ERROR] server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: peer has newer term, stopping replication: peer="{Voter server-164f4764-ebd7-bbaa-3367-321add0910e4 164f4764-ebd7-bbaa-3367-321add0910e4}"
2026-05-25T23:23:47.538Z [INFO]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: entering follower state: follower="Node at 19c71118-7e61-cc98-9b5e-18e0d7ec6f6f [Follower]" leader-address= leader-id=
2026-05-25T23:23:47.538Z [INFO]  server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f: aborting pipeline replication: peer="{Voter server-43deaf51-2664-9283-bde9-1056361f0668 43deaf51-2664-9283-bde9-1056361f0668}"
2026-05-25T23:23:47.539Z [INFO]  server-96cde625-cbd0-e627-9d39-f11ce9c09632: pipelining replication: peer="{Voter server-6315bed1-621a-3f3c-64e5-6197c9662cd5 6315bed1-621a-3f3c-64e5-6197c9662cd5}"
2026-05-25T23:23:48.103Z [WARN]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:48.103Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: entering candidate state: node="Node at cb02535b-4fa3-e7a6-9c73-0120936129ca [Candidate]" term=4
2026-05-25T23:23:48.104Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: election won: term=4 tally=2
2026-05-25T23:23:48.104Z [INFO]  server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb: duplicate requestVote for same term: term=4
2026-05-25T23:23:48.104Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: entering leader state: leader="Node at cb02535b-4fa3-e7a6-9c73-0120936129ca [Leader]"
2026-05-25T23:23:48.104Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: added peer, starting replication: peer=server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb
2026-05-25T23:23:48.104Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: added peer, starting replication: peer=server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc
2026-05-25T23:23:48.104Z [INFO]  server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb: entering follower state: follower="Node at bc1e4076-200c-c555-eb3f-ab3ef10bbedb [Follower]" leader-address=cb02535b-4fa3-e7a6-9c73-0120936129ca leader-id=server-cb02535b-4fa3-e7a6-9c73-0120936129ca
2026-05-25T23:23:48.104Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: pipelining replication: peer="{Voter server-bc1e4076-200c-c555-eb3f-ab3ef10bbedb bc1e4076-200c-c555-eb3f-ab3ef10bbedb}"
2026-05-25T23:23:48.104Z [INFO]  server-cb02535b-4fa3-e7a6-9c73-0120936129ca: pipelining replication: peer="{Voter server-eb7f8dda-b789-58f9-9aee-a08a5ed05fcc eb7f8dda-b789-58f9-9aee-a08a5ed05fcc}"
2026-05-25T23:23:48.137Z [WARN]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:48.137Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: entering candidate state: node="Node at 8cdddb1f-717d-965c-d545-807129b2ce3b [Candidate]" term=4
2026-05-25T23:23:48.137Z [INFO]  server-ae4d03f3-6164-5c59-99d4-455db47640e4: duplicate requestVote for same term: term=4
2026-05-25T23:23:48.137Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: election won: term=4 tally=2
2026-05-25T23:23:48.137Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: entering leader state: leader="Node at 8cdddb1f-717d-965c-d545-807129b2ce3b [Leader]"
2026-05-25T23:23:48.137Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: added peer, starting replication: peer=server-ae4d03f3-6164-5c59-99d4-455db47640e4
2026-05-25T23:23:48.137Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: added peer, starting replication: peer=server-b2d831eb-15f8-614f-ec08-fafe997c4944
2026-05-25T23:23:48.137Z [INFO]  server-ae4d03f3-6164-5c59-99d4-455db47640e4: entering follower state: follower="Node at ae4d03f3-6164-5c59-99d4-455db47640e4 [Follower]" leader-address=8cdddb1f-717d-965c-d545-807129b2ce3b leader-id=server-8cdddb1f-717d-965c-d545-807129b2ce3b
2026-05-25T23:23:48.137Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: pipelining replication: peer="{Voter server-b2d831eb-15f8-614f-ec08-fafe997c4944 b2d831eb-15f8-614f-ec08-fafe997c4944}"
2026-05-25T23:23:48.137Z [INFO]  server-8cdddb1f-717d-965c-d545-807129b2ce3b: pipelining replication: peer="{Voter server-ae4d03f3-6164-5c59-99d4-455db47640e4 ae4d03f3-6164-5c59-99d4-455db47640e4}"
2026-05-25T23:23:48.173Z [WARN]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:48.174Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: entering candidate state: node="Node at 4b61b004-1434-9acb-8773-7df6a7a770fd [Candidate]" term=2
2026-05-25T23:23:48.174Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: election won: term=2 tally=2
2026-05-25T23:23:48.174Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: entering leader state: leader="Node at 4b61b004-1434-9acb-8773-7df6a7a770fd [Leader]"
2026-05-25T23:23:48.174Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: added peer, starting replication: peer=server-63bc1d21-8079-9f02-20e6-9721519814c9
2026-05-25T23:23:48.174Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: added peer, starting replication: peer=server-8e548790-85c7-0f6d-0ba9-e92d7ab6e103
2026-05-25T23:23:48.174Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: pipelining replication: peer="{Voter server-63bc1d21-8079-9f02-20e6-9721519814c9 63bc1d21-8079-9f02-20e6-9721519814c9}"
2026-05-25T23:23:48.175Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: pipelining replication: peer="{Voter server-8e548790-85c7-0f6d-0ba9-e92d7ab6e103 8e548790-85c7-0f6d-0ba9-e92d7ab6e103}"
2026-05-25T23:23:48.282Z [WARN]  server-116a910e-5eb0-5371-9698-339f44ce34c6: heartbeat timeout reached, starting election: last-leader-addr=f9a39ce0-d9e9-5faf-e108-7822f2b3993b last-leader-id=server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b
2026-05-25T23:23:48.282Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: entering candidate state: node="Node at 116a910e-5eb0-5371-9698-339f44ce34c6 [Candidate]" term=5
2026-05-25T23:23:48.282Z [WARN]  server-941b9661-74e5-5201-f2fe-39b3f009b868: rejecting vote request since we have a leader: from=116a910e-5eb0-5371-9698-339f44ce34c6 leader=f9a39ce0-d9e9-5faf-e108-7822f2b3993b leader-id=server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b
2026-05-25T23:23:48.282Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: election won: term=5 tally=2
2026-05-25T23:23:48.282Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: entering leader state: leader="Node at 116a910e-5eb0-5371-9698-339f44ce34c6 [Leader]"
2026-05-25T23:23:48.282Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: added peer, starting replication: peer=server-941b9661-74e5-5201-f2fe-39b3f009b868
2026-05-25T23:23:48.282Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: added peer, starting replication: peer=server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b
2026-05-25T23:23:48.282Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: pipelining replication: peer="{Voter server-941b9661-74e5-5201-f2fe-39b3f009b868 941b9661-74e5-5201-f2fe-39b3f009b868}"
2026-05-25T23:23:48.282Z [INFO]  server-116a910e-5eb0-5371-9698-339f44ce34c6: pipelining replication: peer="{Voter server-f9a39ce0-d9e9-5faf-e108-7822f2b3993b f9a39ce0-d9e9-5faf-e108-7822f2b3993b}"
2026-05-25T23:23:48.320Z [WARN]  server-43deaf51-2664-9283-bde9-1056361f0668: heartbeat timeout reached, starting election: last-leader-addr=19c71118-7e61-cc98-9b5e-18e0d7ec6f6f last-leader-id=server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f
2026-05-25T23:23:48.320Z [INFO]  server-43deaf51-2664-9283-bde9-1056361f0668: entering candidate state: node="Node at 43deaf51-2664-9283-bde9-1056361f0668 [Candidate]" term=4
2026-05-25T23:23:48.320Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: duplicate requestVote for same term: term=4
2026-05-25T23:23:48.320Z [INFO]  server-43deaf51-2664-9283-bde9-1056361f0668: election won: term=4 tally=2
2026-05-25T23:23:48.320Z [INFO]  server-43deaf51-2664-9283-bde9-1056361f0668: entering leader state: leader="Node at 43deaf51-2664-9283-bde9-1056361f0668 [Leader]"
2026-05-25T23:23:48.320Z [INFO]  server-43deaf51-2664-9283-bde9-1056361f0668: added peer, starting replication: peer=server-164f4764-ebd7-bbaa-3367-321add0910e4
2026-05-25T23:23:48.320Z [INFO]  server-43deaf51-2664-9283-bde9-1056361f0668: added peer, starting replication: peer=server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f
2026-05-25T23:23:48.320Z [INFO]  server-164f4764-ebd7-bbaa-3367-321add0910e4: entering follower state: follower="Node at 164f4764-ebd7-bbaa-3367-321add0910e4 [Follower]" leader-address=43deaf51-2664-9283-bde9-1056361f0668 leader-id=server-43deaf51-2664-9283-bde9-1056361f0668
2026-05-25T23:23:48.320Z [INFO]  server-43deaf51-2664-9283-bde9-1056361f0668: pipelining replication: peer="{Voter server-19c71118-7e61-cc98-9b5e-18e0d7ec6f6f 19c71118-7e61-cc98-9b5e-18e0d7ec6f6f}"
2026-05-25T23:23:48.320Z [INFO]  server-43deaf51-2664-9283-bde9-1056361f0668: pipelining replication: peer="{Voter server-164f4764-ebd7-bbaa-3367-321add0910e4 164f4764-ebd7-bbaa-3367-321add0910e4}"
2026-05-25T23:23:49.179Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:48.174171316 +0000 UTC m=+83.820618781 (1 nodes), 641.940277ms from start of poll, 642.347995ms from cluster start. Timeout at 2026-05-25 23:23:49.179174362 +0000 UTC m=+84.825621817, 1.005003036s after stability
2026-05-25T23:23:49.179Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: starting snapshot up to: index=12
2026-05-25T23:23:49.179Z [INFO]  creating new snapshot: path=/tmp/raft3238592162/snapshots/2-12-1779751429179.tmp
2026-05-25T23:23:49.180Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: snapshot complete up to: index=12
2026-05-25T23:23:49.180Z [INFO]  creating new snapshot: path=/tmp/raft3238592162/snapshots/2-10013-1779751429180.tmp
2026-05-25T23:23:49.181Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: copied to local snapshot: bytes=71
2026-05-25T23:23:49.181Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: snapshot restore progress: id=2-10013-1779751429180 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:49.181Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: restored user snapshot: index=10013
2026-05-25T23:23:49.181Z [ERROR] server-4b61b004-1434-9acb-8773-7df6a7a770fd: failed to get log: index=23 error="log not found"
2026-05-25T23:23:49.181Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: aborting pipeline replication: peer="{Voter server-8e548790-85c7-0f6d-0ba9-e92d7ab6e103 8e548790-85c7-0f6d-0ba9-e92d7ab6e103}"
2026-05-25T23:23:49.181Z [ERROR] server-4b61b004-1434-9acb-8773-7df6a7a770fd: failed to get log: index=23 error="log not found"
2026-05-25T23:23:49.181Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: aborting pipeline replication: peer="{Voter server-63bc1d21-8079-9f02-20e6-9721519814c9 63bc1d21-8079-9f02-20e6-9721519814c9}"
2026-05-25T23:23:49.190Z [ERROR] server-4b61b004-1434-9acb-8773-7df6a7a770fd: failed to get log: index=23 error="log not found"
2026-05-25T23:23:49.190Z [INFO]  creating new snapshot: path=/tmp/raft538681014/snapshots/2-10013-1779751429190.tmp
2026-05-25T23:23:49.190Z [INFO]  server-8e548790-85c7-0f6d-0ba9-e92d7ab6e103: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:49.191Z [INFO]  server-8e548790-85c7-0f6d-0ba9-e92d7ab6e103: copied to local snapshot: bytes=71
2026-05-25T23:23:49.191Z [ERROR] server-4b61b004-1434-9acb-8773-7df6a7a770fd: failed to get log: index=23 error="log not found"
2026-05-25T23:23:49.191Z [INFO]  server-8e548790-85c7-0f6d-0ba9-e92d7ab6e103: snapshot restore progress: id=2-10013-1779751429190 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:49.191Z [INFO]  server-8e548790-85c7-0f6d-0ba9-e92d7ab6e103: Installed remote snapshot
2026-05-25T23:23:49.191Z [WARN]  server-8e548790-85c7-0f6d-0ba9-e92d7ab6e103: skipping application of old log: index=22
2026-05-25T23:23:49.191Z [INFO]  creating new snapshot: path=/tmp/raft328948626/snapshots/2-10013-1779751429191.tmp
2026-05-25T23:23:49.191Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: pipelining replication: peer="{Voter server-8e548790-85c7-0f6d-0ba9-e92d7ab6e103 8e548790-85c7-0f6d-0ba9-e92d7ab6e103}"
2026-05-25T23:23:49.191Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:23:49.191Z [INFO]  server-63bc1d21-8079-9f02-20e6-9721519814c9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:49.191Z [INFO]  server-63bc1d21-8079-9f02-20e6-9721519814c9: copied to local snapshot: bytes=71
2026-05-25T23:23:49.191Z [INFO]  server-63bc1d21-8079-9f02-20e6-9721519814c9: snapshot restore progress: id=2-10013-1779751429191 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:49.191Z [INFO]  server-63bc1d21-8079-9f02-20e6-9721519814c9: Installed remote snapshot
2026-05-25T23:23:49.192Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: pipelining replication: peer="{Voter server-63bc1d21-8079-9f02-20e6-9721519814c9 63bc1d21-8079-9f02-20e6-9721519814c9}"
2026-05-25T23:23:50.196Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:23:49.191476024 +0000 UTC m=+84.837923488 (2 nodes), 903ns from start of poll, 1.659652702s from cluster start. Timeout at 2026-05-25 23:23:50.196507374 +0000 UTC m=+85.842954848, 1.00503136s after stability
2026-05-25T23:23:50.209Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: aborting pipeline replication: peer="{Voter server-8e548790-85c7-0f6d-0ba9-e92d7ab6e103 8e548790-85c7-0f6d-0ba9-e92d7ab6e103}"
2026-05-25T23:23:50.209Z [INFO]  server-4b61b004-1434-9acb-8773-7df6a7a770fd: aborting pipeline replication: peer="{Voter server-63bc1d21-8079-9f02-20e6-9721519814c9 63bc1d21-8079-9f02-20e6-9721519814c9}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false
2026-05-25T23:23:50.210Z [INFO]  server-f48823bc-0d6f-de27-4578-ad5cd6143344: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f48823bc-0d6f-de27-4578-ad5cd6143344 Address:f48823bc-0d6f-de27-4578-ad5cd6143344} {Suffrage:Voter ID:server-a3abb60f-9093-64fa-3b48-086d4103705d Address:a3abb60f-9093-64fa-3b48-086d4103705d} {Suffrage:Voter ID:server-cfb3796f-6a21-eb85-d751-532eaef2c8be Address:cfb3796f-6a21-eb85-d751-532eaef2c8be}]"
2026-05-25T23:23:50.210Z [INFO]  server-a3abb60f-9093-64fa-3b48-086d4103705d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f48823bc-0d6f-de27-4578-ad5cd6143344 Address:f48823bc-0d6f-de27-4578-ad5cd6143344} {Suffrage:Voter ID:server-a3abb60f-9093-64fa-3b48-086d4103705d Address:a3abb60f-9093-64fa-3b48-086d4103705d} {Suffrage:Voter ID:server-cfb3796f-6a21-eb85-d751-532eaef2c8be Address:cfb3796f-6a21-eb85-d751-532eaef2c8be}]"
2026-05-25T23:23:50.210Z [INFO]  server-f48823bc-0d6f-de27-4578-ad5cd6143344: entering follower state: follower="Node at f48823bc-0d6f-de27-4578-ad5cd6143344 [Follower]" leader-address= leader-id=
2026-05-25T23:23:50.210Z [INFO]  server-a3abb60f-9093-64fa-3b48-086d4103705d: entering follower state: follower="Node at a3abb60f-9093-64fa-3b48-086d4103705d [Follower]" leader-address= leader-id=
2026-05-25T23:23:50.210Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f48823bc-0d6f-de27-4578-ad5cd6143344 Address:f48823bc-0d6f-de27-4578-ad5cd6143344} {Suffrage:Voter ID:server-a3abb60f-9093-64fa-3b48-086d4103705d Address:a3abb60f-9093-64fa-3b48-086d4103705d} {Suffrage:Voter ID:server-cfb3796f-6a21-eb85-d751-532eaef2c8be Address:cfb3796f-6a21-eb85-d751-532eaef2c8be}]"
2026-05-25T23:23:50.210Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:50.210Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: entering follower state: follower="Node at cfb3796f-6a21-eb85-d751-532eaef2c8be [Follower]" leader-address= leader-id=
2026-05-25T23:23:50.802Z [WARN]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:50.802Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: entering candidate state: node="Node at cfb3796f-6a21-eb85-d751-532eaef2c8be [Candidate]" term=2
2026-05-25T23:23:50.802Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: election won: term=2 tally=2
2026-05-25T23:23:50.802Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: entering leader state: leader="Node at cfb3796f-6a21-eb85-d751-532eaef2c8be [Leader]"
2026-05-25T23:23:50.802Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: added peer, starting replication: peer=server-f48823bc-0d6f-de27-4578-ad5cd6143344
2026-05-25T23:23:50.802Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: added peer, starting replication: peer=server-a3abb60f-9093-64fa-3b48-086d4103705d
2026-05-25T23:23:50.802Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: pipelining replication: peer="{Voter server-a3abb60f-9093-64fa-3b48-086d4103705d a3abb60f-9093-64fa-3b48-086d4103705d}"
2026-05-25T23:23:50.802Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: pipelining replication: peer="{Voter server-f48823bc-0d6f-de27-4578-ad5cd6143344 f48823bc-0d6f-de27-4578-ad5cd6143344}"
2026-05-25T23:23:51.808Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:50.802650308 +0000 UTC m=+86.449097776 (1 nodes), 592.006115ms from start of poll, 592.383777ms from cluster start. Timeout at 2026-05-25 23:23:51.807651862 +0000 UTC m=+87.454099333, 1.005001557s after stability
2026-05-25T23:23:51.809Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: starting snapshot up to: index=12
2026-05-25T23:23:51.809Z [INFO]  creating new snapshot: path=/tmp/raft3725280917/snapshots/2-12-1779751431809.tmp
2026-05-25T23:23:51.815Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: snapshot complete up to: index=12
2026-05-25T23:23:51.816Z [INFO]  creating new snapshot: path=/tmp/raft3725280917/snapshots/2-10013-1779751431816.tmp
2026-05-25T23:23:51.816Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: copied to local snapshot: bytes=71
2026-05-25T23:23:51.816Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: snapshot restore progress: id=2-10013-1779751431816 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:51.816Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: removing all old logs from log store
2026-05-25T23:23:51.816Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: compacting logs: from=1 to=22
2026-05-25T23:23:51.816Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: restored user snapshot: index=10013
2026-05-25T23:23:51.817Z [ERROR] server-cfb3796f-6a21-eb85-d751-532eaef2c8be: failed to get log: index=22 error="log not found"
2026-05-25T23:23:51.817Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: aborting pipeline replication: peer="{Voter server-a3abb60f-9093-64fa-3b48-086d4103705d a3abb60f-9093-64fa-3b48-086d4103705d}"
2026-05-25T23:23:51.817Z [ERROR] server-cfb3796f-6a21-eb85-d751-532eaef2c8be: failed to get log: index=22 error="log not found"
2026-05-25T23:23:51.817Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: aborting pipeline replication: peer="{Voter server-f48823bc-0d6f-de27-4578-ad5cd6143344 f48823bc-0d6f-de27-4578-ad5cd6143344}"
2026-05-25T23:23:51.822Z [ERROR] server-cfb3796f-6a21-eb85-d751-532eaef2c8be: failed to get log: index=22 error="log not found"
2026-05-25T23:23:51.823Z [INFO]  creating new snapshot: path=/tmp/raft1481680732/snapshots/2-10013-1779751431823.tmp
2026-05-25T23:23:51.823Z [INFO]  server-a3abb60f-9093-64fa-3b48-086d4103705d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:51.824Z [INFO]  server-a3abb60f-9093-64fa-3b48-086d4103705d: copied to local snapshot: bytes=71
2026-05-25T23:23:51.824Z [INFO]  server-a3abb60f-9093-64fa-3b48-086d4103705d: snapshot restore progress: id=2-10013-1779751431823 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:51.824Z [INFO]  server-a3abb60f-9093-64fa-3b48-086d4103705d: removing all old logs from log store
2026-05-25T23:23:51.824Z [INFO]  server-a3abb60f-9093-64fa-3b48-086d4103705d: compacting logs: from=1 to=22
2026-05-25T23:23:51.824Z [INFO]  server-a3abb60f-9093-64fa-3b48-086d4103705d: Installed remote snapshot
2026-05-25T23:23:51.824Z [WARN]  server-a3abb60f-9093-64fa-3b48-086d4103705d: skipping application of old log: index=22
2026-05-25T23:23:51.825Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: pipelining replication: peer="{Voter server-a3abb60f-9093-64fa-3b48-086d4103705d a3abb60f-9093-64fa-3b48-086d4103705d}"
2026-05-25T23:23:51.825Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:23:51.826Z [ERROR] server-cfb3796f-6a21-eb85-d751-532eaef2c8be: failed to get log: index=22 error="log not found"
2026-05-25T23:23:51.826Z [INFO]  creating new snapshot: path=/tmp/raft240289254/snapshots/2-10013-1779751431826.tmp
2026-05-25T23:23:51.826Z [INFO]  server-f48823bc-0d6f-de27-4578-ad5cd6143344: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:51.826Z [INFO]  server-f48823bc-0d6f-de27-4578-ad5cd6143344: copied to local snapshot: bytes=71
2026-05-25T23:23:51.827Z [INFO]  server-f48823bc-0d6f-de27-4578-ad5cd6143344: snapshot restore progress: id=2-10013-1779751431826 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:51.827Z [INFO]  server-f48823bc-0d6f-de27-4578-ad5cd6143344: removing all old logs from log store
2026-05-25T23:23:51.827Z [INFO]  server-f48823bc-0d6f-de27-4578-ad5cd6143344: compacting logs: from=1 to=22
2026-05-25T23:23:51.827Z [INFO]  server-f48823bc-0d6f-de27-4578-ad5cd6143344: Installed remote snapshot
2026-05-25T23:23:51.827Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: pipelining replication: peer="{Voter server-f48823bc-0d6f-de27-4578-ad5cd6143344 f48823bc-0d6f-de27-4578-ad5cd6143344}"
2026-05-25T23:23:52.830Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:23:51.825065376 +0000 UTC m=+87.471512842 (2 nodes), 793ns from start of poll, 1.614798843s from cluster start. Timeout at 2026-05-25 23:23:52.830094339 +0000 UTC m=+88.476541840, 1.005028998s after stability
2026-05-25T23:23:52.842Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: aborting pipeline replication: peer="{Voter server-a3abb60f-9093-64fa-3b48-086d4103705d a3abb60f-9093-64fa-3b48-086d4103705d}"
2026-05-25T23:23:52.842Z [INFO]  server-cfb3796f-6a21-eb85-d751-532eaef2c8be: aborting pipeline replication: peer="{Voter server-f48823bc-0d6f-de27-4578-ad5cd6143344 f48823bc-0d6f-de27-4578-ad5cd6143344}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true
2026-05-25T23:23:52.844Z [INFO]  server-83139cd3-2165-5df4-bc01-1dac2c780832: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83139cd3-2165-5df4-bc01-1dac2c780832 Address:83139cd3-2165-5df4-bc01-1dac2c780832} {Suffrage:Voter ID:server-9365a62c-7995-3521-1333-a28b8f4a0383 Address:9365a62c-7995-3521-1333-a28b8f4a0383} {Suffrage:Voter ID:server-c9d0a492-9e25-dcd9-5154-2b7122202eb3 Address:c9d0a492-9e25-dcd9-5154-2b7122202eb3}]"
2026-05-25T23:23:52.844Z [INFO]  server-83139cd3-2165-5df4-bc01-1dac2c780832: entering follower state: follower="Node at 83139cd3-2165-5df4-bc01-1dac2c780832 [Follower]" leader-address= leader-id=
2026-05-25T23:23:52.844Z [INFO]  server-9365a62c-7995-3521-1333-a28b8f4a0383: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83139cd3-2165-5df4-bc01-1dac2c780832 Address:83139cd3-2165-5df4-bc01-1dac2c780832} {Suffrage:Voter ID:server-9365a62c-7995-3521-1333-a28b8f4a0383 Address:9365a62c-7995-3521-1333-a28b8f4a0383} {Suffrage:Voter ID:server-c9d0a492-9e25-dcd9-5154-2b7122202eb3 Address:c9d0a492-9e25-dcd9-5154-2b7122202eb3}]"
2026-05-25T23:23:52.844Z [INFO]  server-9365a62c-7995-3521-1333-a28b8f4a0383: entering follower state: follower="Node at 9365a62c-7995-3521-1333-a28b8f4a0383 [Follower]" leader-address= leader-id=
2026-05-25T23:23:52.844Z [INFO]  server-c9d0a492-9e25-dcd9-5154-2b7122202eb3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83139cd3-2165-5df4-bc01-1dac2c780832 Address:83139cd3-2165-5df4-bc01-1dac2c780832} {Suffrage:Voter ID:server-9365a62c-7995-3521-1333-a28b8f4a0383 Address:9365a62c-7995-3521-1333-a28b8f4a0383} {Suffrage:Voter ID:server-c9d0a492-9e25-dcd9-5154-2b7122202eb3 Address:c9d0a492-9e25-dcd9-5154-2b7122202eb3}]"
2026-05-25T23:23:52.844Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:52.844Z [INFO]  server-c9d0a492-9e25-dcd9-5154-2b7122202eb3: entering follower state: follower="Node at c9d0a492-9e25-dcd9-5154-2b7122202eb3 [Follower]" leader-address= leader-id=
2026-05-25T23:23:53.475Z [WARN]  server-83139cd3-2165-5df4-bc01-1dac2c780832: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:53.475Z [INFO]  server-83139cd3-2165-5df4-bc01-1dac2c780832: entering candidate state: node="Node at 83139cd3-2165-5df4-bc01-1dac2c780832 [Candidate]" term=2
2026-05-25T23:23:53.475Z [INFO]  server-83139cd3-2165-5df4-bc01-1dac2c780832: election won: term=2 tally=2
2026-05-25T23:23:53.475Z [INFO]  server-83139cd3-2165-5df4-bc01-1dac2c780832: entering leader state: leader="Node at 83139cd3-2165-5df4-bc01-1dac2c780832 [Leader]"
2026-05-25T23:23:53.475Z [INFO]  server-83139cd3-2165-5df4-bc01-1dac2c780832: added peer, starting replication: peer=server-9365a62c-7995-3521-1333-a28b8f4a0383
2026-05-25T23:23:53.475Z [INFO]  server-83139cd3-2165-5df4-bc01-1dac2c780832: added peer, starting replication: peer=server-c9d0a492-9e25-dcd9-5154-2b7122202eb3
2026-05-25T23:23:53.475Z [INFO]  server-83139cd3-2165-5df4-bc01-1dac2c780832: pipelining replication: peer="{Voter server-c9d0a492-9e25-dcd9-5154-2b7122202eb3 c9d0a492-9e25-dcd9-5154-2b7122202eb3}"
2026-05-25T23:23:53.475Z [INFO]  server-83139cd3-2165-5df4-bc01-1dac2c780832: pipelining replication: peer="{Voter server-9365a62c-7995-3521-1333-a28b8f4a0383 9365a62c-7995-3521-1333-a28b8f4a0383}"
2026-05-25T23:23:54.481Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:53.475443359 +0000 UTC m=+89.121890812 (1 nodes), 631.186998ms from start of poll, 631.530854ms from cluster start. Timeout at 2026-05-25 23:23:54.480445752 +0000 UTC m=+90.126893207, 1.005002395s after stability
2026-05-25T23:23:54.481Z [INFO]  server-83139cd3-2165-5df4-bc01-1dac2c780832: starting snapshot up to: index=12
2026-05-25T23:23:54.481Z [INFO]  creating new snapshot: path=/tmp/raft860121167/snapshots/2-12-1779751434481.tmp
2026-05-25T23:23:54.627Z [INFO]  server-83139cd3-2165-5df4-bc01-1dac2c780832: snapshot complete up to: index=12
2026-05-25T23:23:54.628Z [INFO]  server-e393e98a-2137-b090-1283-c3f897f3256a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e393e98a-2137-b090-1283-c3f897f3256a Address:e393e98a-2137-b090-1283-c3f897f3256a} {Suffrage:Voter ID:server-45632519-f1e9-2132-49b0-e7672a7d0f40 Address:45632519-f1e9-2132-49b0-e7672a7d0f40} {Suffrage:Voter ID:server-04c82024-d1fe-12c5-010e-35a55b388223 Address:04c82024-d1fe-12c5-010e-35a55b388223}]"
2026-05-25T23:23:54.628Z [INFO]  server-45632519-f1e9-2132-49b0-e7672a7d0f40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e393e98a-2137-b090-1283-c3f897f3256a Address:e393e98a-2137-b090-1283-c3f897f3256a} {Suffrage:Voter ID:server-45632519-f1e9-2132-49b0-e7672a7d0f40 Address:45632519-f1e9-2132-49b0-e7672a7d0f40} {Suffrage:Voter ID:server-04c82024-d1fe-12c5-010e-35a55b388223 Address:04c82024-d1fe-12c5-010e-35a55b388223}]"
2026-05-25T23:23:54.628Z [INFO]  server-e393e98a-2137-b090-1283-c3f897f3256a: entering follower state: follower="Node at e393e98a-2137-b090-1283-c3f897f3256a [Follower]" leader-address= leader-id=
2026-05-25T23:23:54.628Z [INFO]  server-45632519-f1e9-2132-49b0-e7672a7d0f40: entering follower state: follower="Node at 45632519-f1e9-2132-49b0-e7672a7d0f40 [Follower]" leader-address= leader-id=
2026-05-25T23:23:54.628Z [INFO]  server-04c82024-d1fe-12c5-010e-35a55b388223: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e393e98a-2137-b090-1283-c3f897f3256a Address:e393e98a-2137-b090-1283-c3f897f3256a} {Suffrage:Voter ID:server-45632519-f1e9-2132-49b0-e7672a7d0f40 Address:45632519-f1e9-2132-49b0-e7672a7d0f40} {Suffrage:Voter ID:server-04c82024-d1fe-12c5-010e-35a55b388223 Address:04c82024-d1fe-12c5-010e-35a55b388223}]"
2026-05-25T23:23:54.628Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:54.628Z [INFO]  server-04c82024-d1fe-12c5-010e-35a55b388223: entering follower state: follower="Node at 04c82024-d1fe-12c5-010e-35a55b388223 [Follower]" leader-address= leader-id=
2026-05-25T23:23:55.249Z [WARN]  server-04c82024-d1fe-12c5-010e-35a55b388223: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:55.249Z [INFO]  server-04c82024-d1fe-12c5-010e-35a55b388223: entering candidate state: node="Node at 04c82024-d1fe-12c5-010e-35a55b388223 [Candidate]" term=2
2026-05-25T23:23:55.249Z [INFO]  server-04c82024-d1fe-12c5-010e-35a55b388223: election won: term=2 tally=2
2026-05-25T23:23:55.249Z [INFO]  server-04c82024-d1fe-12c5-010e-35a55b388223: entering leader state: leader="Node at 04c82024-d1fe-12c5-010e-35a55b388223 [Leader]"
2026-05-25T23:23:55.249Z [INFO]  server-04c82024-d1fe-12c5-010e-35a55b388223: added peer, starting replication: peer=server-e393e98a-2137-b090-1283-c3f897f3256a
2026-05-25T23:23:55.249Z [INFO]  server-04c82024-d1fe-12c5-010e-35a55b388223: added peer, starting replication: peer=server-45632519-f1e9-2132-49b0-e7672a7d0f40
2026-05-25T23:23:55.249Z [INFO]  server-04c82024-d1fe-12c5-010e-35a55b388223: pipelining replication: peer="{Voter server-45632519-f1e9-2132-49b0-e7672a7d0f40 45632519-f1e9-2132-49b0-e7672a7d0f40}"
2026-05-25T23:23:55.249Z [INFO]  server-04c82024-d1fe-12c5-010e-35a55b388223: pipelining replication: peer="{Voter server-e393e98a-2137-b090-1283-c3f897f3256a e393e98a-2137-b090-1283-c3f897f3256a}"
2026-05-25T23:23:56.255Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:55.24970366 +0000 UTC m=+90.896151114 (1 nodes), 621.363986ms from start of poll, 621.726201ms from cluster start. Timeout at 2026-05-25 23:23:56.254709624 +0000 UTC m=+91.901157074, 1.00500596s after stability
2026-05-25T23:23:56.255Z [INFO]  creating new snapshot: path=/tmp/raft2967810282/snapshots/2-10013-1779751436255.tmp
2026-05-25T23:23:56.255Z [INFO]  server-04c82024-d1fe-12c5-010e-35a55b388223: copied to local snapshot: bytes=71
2026-05-25T23:23:56.256Z [INFO]  server-04c82024-d1fe-12c5-010e-35a55b388223: snapshot restore progress: id=2-10013-1779751436255 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:56.256Z [INFO]  server-04c82024-d1fe-12c5-010e-35a55b388223: restored user snapshot: index=10013
2026-05-25T23:23:56.256Z [ERROR] server-04c82024-d1fe-12c5-010e-35a55b388223: failed to get log: index=3 error="log not found"
2026-05-25T23:23:56.256Z [ERROR] server-04c82024-d1fe-12c5-010e-35a55b388223: failed to get log: index=3 error="log not found"
2026-05-25T23:23:56.256Z [INFO]  server-04c82024-d1fe-12c5-010e-35a55b388223: aborting pipeline replication: peer="{Voter server-45632519-f1e9-2132-49b0-e7672a7d0f40 45632519-f1e9-2132-49b0-e7672a7d0f40}"
2026-05-25T23:23:56.256Z [INFO]  server-04c82024-d1fe-12c5-010e-35a55b388223: aborting pipeline replication: peer="{Voter server-e393e98a-2137-b090-1283-c3f897f3256a e393e98a-2137-b090-1283-c3f897f3256a}"
2026-05-25T23:23:56.264Z [ERROR] server-04c82024-d1fe-12c5-010e-35a55b388223: failed to get log: index=3 error="log not found"
2026-05-25T23:23:56.264Z [INFO]  creating new snapshot: path=/tmp/raft2554357942/snapshots/2-10013-1779751436264.tmp
2026-05-25T23:23:56.265Z [INFO]  server-45632519-f1e9-2132-49b0-e7672a7d0f40: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:56.265Z [INFO]  server-45632519-f1e9-2132-49b0-e7672a7d0f40: copied to local snapshot: bytes=71
2026-05-25T23:23:56.265Z [INFO]  server-45632519-f1e9-2132-49b0-e7672a7d0f40: snapshot restore progress: id=2-10013-1779751436264 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:56.265Z [INFO]  server-45632519-f1e9-2132-49b0-e7672a7d0f40: Installed remote snapshot
2026-05-25T23:23:56.265Z [INFO]  server-04c82024-d1fe-12c5-010e-35a55b388223: pipelining replication: peer="{Voter server-45632519-f1e9-2132-49b0-e7672a7d0f40 45632519-f1e9-2132-49b0-e7672a7d0f40}"
2026-05-25T23:23:56.265Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:23:56.266Z [ERROR] server-04c82024-d1fe-12c5-010e-35a55b388223: failed to get log: index=3 error="log not found"
2026-05-25T23:23:56.266Z [INFO]  creating new snapshot: path=/tmp/raft309682619/snapshots/2-10013-1779751436266.tmp
2026-05-25T23:23:56.267Z [INFO]  server-e393e98a-2137-b090-1283-c3f897f3256a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:56.267Z [INFO]  server-e393e98a-2137-b090-1283-c3f897f3256a: copied to local snapshot: bytes=71
2026-05-25T23:23:56.267Z [INFO]  server-e393e98a-2137-b090-1283-c3f897f3256a: snapshot restore progress: id=2-10013-1779751436266 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:23:56.267Z [INFO]  server-e393e98a-2137-b090-1283-c3f897f3256a: Installed remote snapshot
2026-05-25T23:23:56.267Z [INFO]  server-04c82024-d1fe-12c5-010e-35a55b388223: pipelining replication: peer="{Voter server-e393e98a-2137-b090-1283-c3f897f3256a e393e98a-2137-b090-1283-c3f897f3256a}"
2026-05-25T23:23:57.271Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:23:56.265804409 +0000 UTC m=+91.912251868 (2 nodes), 1.395µs from start of poll, 1.637826955s from cluster start. Timeout at 2026-05-25 23:23:57.270828998 +0000 UTC m=+92.917276486, 1.005024618s after stability
2026-05-25T23:23:57.283Z [INFO]  server-83139cd3-2165-5df4-bc01-1dac2c780832: aborting pipeline replication: peer="{Voter server-c9d0a492-9e25-dcd9-5154-2b7122202eb3 c9d0a492-9e25-dcd9-5154-2b7122202eb3}"
2026-05-25T23:23:57.283Z [INFO]  server-83139cd3-2165-5df4-bc01-1dac2c780832: aborting pipeline replication: peer="{Voter server-9365a62c-7995-3521-1333-a28b8f4a0383 9365a62c-7995-3521-1333-a28b8f4a0383}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true
2026-05-25T23:23:57.284Z [INFO]  server-3eaa7cb2-d8f1-e3ee-6484-ad5f879f5a40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3eaa7cb2-d8f1-e3ee-6484-ad5f879f5a40 Address:3eaa7cb2-d8f1-e3ee-6484-ad5f879f5a40} {Suffrage:Voter ID:server-1f7ea58e-b849-afe6-b965-5f53f0c4fb93 Address:1f7ea58e-b849-afe6-b965-5f53f0c4fb93} {Suffrage:Voter ID:server-e6156019-8ca6-7656-3552-9c7b17412730 Address:e6156019-8ca6-7656-3552-9c7b17412730}]"
2026-05-25T23:23:57.284Z [INFO]  server-1f7ea58e-b849-afe6-b965-5f53f0c4fb93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3eaa7cb2-d8f1-e3ee-6484-ad5f879f5a40 Address:3eaa7cb2-d8f1-e3ee-6484-ad5f879f5a40} {Suffrage:Voter ID:server-1f7ea58e-b849-afe6-b965-5f53f0c4fb93 Address:1f7ea58e-b849-afe6-b965-5f53f0c4fb93} {Suffrage:Voter ID:server-e6156019-8ca6-7656-3552-9c7b17412730 Address:e6156019-8ca6-7656-3552-9c7b17412730}]"
2026-05-25T23:23:57.284Z [INFO]  server-3eaa7cb2-d8f1-e3ee-6484-ad5f879f5a40: entering follower state: follower="Node at 3eaa7cb2-d8f1-e3ee-6484-ad5f879f5a40 [Follower]" leader-address= leader-id=
2026-05-25T23:23:57.284Z [INFO]  server-1f7ea58e-b849-afe6-b965-5f53f0c4fb93: entering follower state: follower="Node at 1f7ea58e-b849-afe6-b965-5f53f0c4fb93 [Follower]" leader-address= leader-id=
2026-05-25T23:23:57.285Z [INFO]  server-e6156019-8ca6-7656-3552-9c7b17412730: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3eaa7cb2-d8f1-e3ee-6484-ad5f879f5a40 Address:3eaa7cb2-d8f1-e3ee-6484-ad5f879f5a40} {Suffrage:Voter ID:server-1f7ea58e-b849-afe6-b965-5f53f0c4fb93 Address:1f7ea58e-b849-afe6-b965-5f53f0c4fb93} {Suffrage:Voter ID:server-e6156019-8ca6-7656-3552-9c7b17412730 Address:e6156019-8ca6-7656-3552-9c7b17412730}]"
2026-05-25T23:23:57.285Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:57.285Z [INFO]  server-e6156019-8ca6-7656-3552-9c7b17412730: entering follower state: follower="Node at e6156019-8ca6-7656-3552-9c7b17412730 [Follower]" leader-address= leader-id=
2026-05-25T23:23:57.842Z [WARN]  server-1f7ea58e-b849-afe6-b965-5f53f0c4fb93: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:57.842Z [INFO]  server-1f7ea58e-b849-afe6-b965-5f53f0c4fb93: entering candidate state: node="Node at 1f7ea58e-b849-afe6-b965-5f53f0c4fb93 [Candidate]" term=2
2026-05-25T23:23:57.842Z [INFO]  server-1f7ea58e-b849-afe6-b965-5f53f0c4fb93: election won: term=2 tally=2
2026-05-25T23:23:57.842Z [INFO]  server-1f7ea58e-b849-afe6-b965-5f53f0c4fb93: entering leader state: leader="Node at 1f7ea58e-b849-afe6-b965-5f53f0c4fb93 [Leader]"
2026-05-25T23:23:57.842Z [INFO]  server-1f7ea58e-b849-afe6-b965-5f53f0c4fb93: added peer, starting replication: peer=server-3eaa7cb2-d8f1-e3ee-6484-ad5f879f5a40
2026-05-25T23:23:57.842Z [INFO]  server-1f7ea58e-b849-afe6-b965-5f53f0c4fb93: added peer, starting replication: peer=server-e6156019-8ca6-7656-3552-9c7b17412730
2026-05-25T23:23:57.842Z [INFO]  server-1f7ea58e-b849-afe6-b965-5f53f0c4fb93: pipelining replication: peer="{Voter server-3eaa7cb2-d8f1-e3ee-6484-ad5f879f5a40 3eaa7cb2-d8f1-e3ee-6484-ad5f879f5a40}"
2026-05-25T23:23:57.843Z [INFO]  server-1f7ea58e-b849-afe6-b965-5f53f0c4fb93: pipelining replication: peer="{Voter server-e6156019-8ca6-7656-3552-9c7b17412730 e6156019-8ca6-7656-3552-9c7b17412730}"
2026-05-25T23:23:58.848Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:57.842908095 +0000 UTC m=+93.489355548 (1 nodes), 557.875561ms from start of poll, 558.199708ms from cluster start. Timeout at 2026-05-25 23:23:58.847911713 +0000 UTC m=+94.494359164, 1.005003616s after stability
2026-05-25T23:23:58.849Z [INFO]  server-1f7ea58e-b849-afe6-b965-5f53f0c4fb93: starting snapshot up to: index=12
2026-05-25T23:23:58.849Z [INFO]  creating new snapshot: path=/tmp/raft310397801/snapshots/2-12-1779751438849.tmp
2026-05-25T23:23:58.849Z [INFO]  server-1f7ea58e-b849-afe6-b965-5f53f0c4fb93: snapshot complete up to: index=12
2026-05-25T23:23:58.850Z [INFO]  server-ff9adc21-2cdb-94e0-7138-39ff979ac038: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff9adc21-2cdb-94e0-7138-39ff979ac038 Address:ff9adc21-2cdb-94e0-7138-39ff979ac038} {Suffrage:Voter ID:server-390addd6-e54e-22e5-b1d7-0fec737b5b43 Address:390addd6-e54e-22e5-b1d7-0fec737b5b43} {Suffrage:Voter ID:server-e3c39d99-dfdd-1da5-3aef-70d9bb164260 Address:e3c39d99-dfdd-1da5-3aef-70d9bb164260}]"
2026-05-25T23:23:58.850Z [INFO]  server-ff9adc21-2cdb-94e0-7138-39ff979ac038: entering follower state: follower="Node at ff9adc21-2cdb-94e0-7138-39ff979ac038 [Follower]" leader-address= leader-id=
2026-05-25T23:23:58.850Z [INFO]  server-390addd6-e54e-22e5-b1d7-0fec737b5b43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff9adc21-2cdb-94e0-7138-39ff979ac038 Address:ff9adc21-2cdb-94e0-7138-39ff979ac038} {Suffrage:Voter ID:server-390addd6-e54e-22e5-b1d7-0fec737b5b43 Address:390addd6-e54e-22e5-b1d7-0fec737b5b43} {Suffrage:Voter ID:server-e3c39d99-dfdd-1da5-3aef-70d9bb164260 Address:e3c39d99-dfdd-1da5-3aef-70d9bb164260}]"
2026-05-25T23:23:58.851Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff9adc21-2cdb-94e0-7138-39ff979ac038 Address:ff9adc21-2cdb-94e0-7138-39ff979ac038} {Suffrage:Voter ID:server-390addd6-e54e-22e5-b1d7-0fec737b5b43 Address:390addd6-e54e-22e5-b1d7-0fec737b5b43} {Suffrage:Voter ID:server-e3c39d99-dfdd-1da5-3aef-70d9bb164260 Address:e3c39d99-dfdd-1da5-3aef-70d9bb164260}]"
2026-05-25T23:23:58.851Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:23:58.850Z [INFO]  server-390addd6-e54e-22e5-b1d7-0fec737b5b43: entering follower state: follower="Node at 390addd6-e54e-22e5-b1d7-0fec737b5b43 [Follower]" leader-address= leader-id=
2026-05-25T23:23:58.851Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: entering follower state: follower="Node at e3c39d99-dfdd-1da5-3aef-70d9bb164260 [Follower]" leader-address= leader-id=
2026-05-25T23:23:59.430Z [WARN]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:23:59.430Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: entering candidate state: node="Node at e3c39d99-dfdd-1da5-3aef-70d9bb164260 [Candidate]" term=2
2026-05-25T23:23:59.430Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: election won: term=2 tally=2
2026-05-25T23:23:59.430Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: entering leader state: leader="Node at e3c39d99-dfdd-1da5-3aef-70d9bb164260 [Leader]"
2026-05-25T23:23:59.430Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: added peer, starting replication: peer=server-ff9adc21-2cdb-94e0-7138-39ff979ac038
2026-05-25T23:23:59.430Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: added peer, starting replication: peer=server-390addd6-e54e-22e5-b1d7-0fec737b5b43
2026-05-25T23:23:59.430Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: pipelining replication: peer="{Voter server-390addd6-e54e-22e5-b1d7-0fec737b5b43 390addd6-e54e-22e5-b1d7-0fec737b5b43}"
2026-05-25T23:23:59.430Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: pipelining replication: peer="{Voter server-ff9adc21-2cdb-94e0-7138-39ff979ac038 ff9adc21-2cdb-94e0-7138-39ff979ac038}"
2026-05-25T23:24:00.436Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:23:59.430861975 +0000 UTC m=+95.077309429 (1 nodes), 579.815833ms from start of poll, 580.182438ms from cluster start. Timeout at 2026-05-25 23:24:00.435868698 +0000 UTC m=+96.082316165, 1.005006736s after stability
2026-05-25T23:24:00.439Z [INFO]  creating new snapshot: path=/tmp/raft2452483247/snapshots/2-10013-1779751440439.tmp
2026-05-25T23:24:00.520Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: copied to local snapshot: bytes=71
2026-05-25T23:24:00.521Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: snapshot restore progress: id=2-10013-1779751440439 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:24:00.521Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: removing all old logs from log store
2026-05-25T23:24:00.521Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: compacting logs: from=1 to=2
2026-05-25T23:24:00.521Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: restored user snapshot: index=10013
2026-05-25T23:24:00.521Z [ERROR] server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: failed to get log: index=2 error="log not found"
2026-05-25T23:24:00.521Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: aborting pipeline replication: peer="{Voter server-390addd6-e54e-22e5-b1d7-0fec737b5b43 390addd6-e54e-22e5-b1d7-0fec737b5b43}"
2026-05-25T23:24:00.521Z [ERROR] server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: failed to get log: index=2 error="log not found"
2026-05-25T23:24:00.521Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: aborting pipeline replication: peer="{Voter server-ff9adc21-2cdb-94e0-7138-39ff979ac038 ff9adc21-2cdb-94e0-7138-39ff979ac038}"
2026-05-25T23:24:00.531Z [ERROR] server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: failed to get log: index=2 error="log not found"
2026-05-25T23:24:00.531Z [ERROR] server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: failed to get log: index=2 error="log not found"
2026-05-25T23:24:00.531Z [INFO]  creating new snapshot: path=/tmp/raft2019511779/snapshots/2-10013-1779751440531.tmp
2026-05-25T23:24:00.531Z [INFO]  creating new snapshot: path=/tmp/raft3171448870/snapshots/2-10013-1779751440531.tmp
2026-05-25T23:24:00.531Z [INFO]  server-ff9adc21-2cdb-94e0-7138-39ff979ac038: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:24:00.532Z [INFO]  server-390addd6-e54e-22e5-b1d7-0fec737b5b43: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-05-25T23:24:00.532Z [INFO]  server-ff9adc21-2cdb-94e0-7138-39ff979ac038: copied to local snapshot: bytes=71
2026-05-25T23:24:00.532Z [INFO]  server-ff9adc21-2cdb-94e0-7138-39ff979ac038: snapshot restore progress: id=2-10013-1779751440531 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:24:00.532Z [INFO]  server-ff9adc21-2cdb-94e0-7138-39ff979ac038: removing all old logs from log store
2026-05-25T23:24:00.532Z [INFO]  server-ff9adc21-2cdb-94e0-7138-39ff979ac038: compacting logs: from=1 to=2
2026-05-25T23:24:00.532Z [INFO]  server-ff9adc21-2cdb-94e0-7138-39ff979ac038: Installed remote snapshot
2026-05-25T23:24:00.532Z [INFO]  server-390addd6-e54e-22e5-b1d7-0fec737b5b43: copied to local snapshot: bytes=71
2026-05-25T23:24:00.532Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: pipelining replication: peer="{Voter server-ff9adc21-2cdb-94e0-7138-39ff979ac038 ff9adc21-2cdb-94e0-7138-39ff979ac038}"
2026-05-25T23:24:00.532Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:00.532Z [INFO]  server-390addd6-e54e-22e5-b1d7-0fec737b5b43: snapshot restore progress: id=2-10013-1779751440531 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-05-25T23:24:00.532Z [INFO]  server-390addd6-e54e-22e5-b1d7-0fec737b5b43: removing all old logs from log store
2026-05-25T23:24:00.532Z [INFO]  server-390addd6-e54e-22e5-b1d7-0fec737b5b43: compacting logs: from=1 to=2
2026-05-25T23:24:00.532Z [INFO]  server-390addd6-e54e-22e5-b1d7-0fec737b5b43: Installed remote snapshot
2026-05-25T23:24:00.532Z [INFO]  server-e3c39d99-dfdd-1da5-3aef-70d9bb164260: pipelining replication: peer="{Voter server-390addd6-e54e-22e5-b1d7-0fec737b5b43 390addd6-e54e-22e5-b1d7-0fec737b5b43}"
2026-05-25T23:24:01.538Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:00.532550491 +0000 UTC m=+96.178997959 (2 nodes), 984ns from start of poll, 1.681870968s from cluster start. Timeout at 2026-05-25 23:24:01.537566843 +0000 UTC m=+97.184014296, 1.005016337s after stability
2026-05-25T23:24:01.549Z [INFO]  server-1f7ea58e-b849-afe6-b965-5f53f0c4fb93: aborting pipeline replication: peer="{Voter server-e6156019-8ca6-7656-3552-9c7b17412730 e6156019-8ca6-7656-3552-9c7b17412730}"
2026-05-25T23:24:01.549Z [INFO]  server-1f7ea58e-b849-afe6-b965-5f53f0c4fb93: aborting pipeline replication: peer="{Voter server-3eaa7cb2-d8f1-e3ee-6484-ad5f879f5a40 3eaa7cb2-d8f1-e3ee-6484-ad5f879f5a40}"
--- FAIL: TestRaft_UserRestore (83.63s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.59s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.55s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.28s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.51s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.70s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.58s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.26s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.10s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.57s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.97s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.51s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.41s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (3.02s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.82s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.24s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.33s)
    --- FAIL: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (3.62s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.71s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.48s)
    --- FAIL: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (2.38s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.68s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.63s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.44s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.27s)
=== RUN   TestRaft_SendSnapshotFollower
2026-05-25T23:24:01.551Z [INFO]  server-9fd490bc-0763-f091-14d1-aaa7110cffa7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9fd490bc-0763-f091-14d1-aaa7110cffa7 Address:9fd490bc-0763-f091-14d1-aaa7110cffa7} {Suffrage:Voter ID:server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f Address:0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f} {Suffrage:Voter ID:server-c362673c-55f1-52a9-5707-4e0c19ed628e Address:c362673c-55f1-52a9-5707-4e0c19ed628e}]"
2026-05-25T23:24:01.551Z [INFO]  server-9fd490bc-0763-f091-14d1-aaa7110cffa7: entering follower state: follower="Node at 9fd490bc-0763-f091-14d1-aaa7110cffa7 [Follower]" leader-address= leader-id=
2026-05-25T23:24:01.551Z [INFO]  server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9fd490bc-0763-f091-14d1-aaa7110cffa7 Address:9fd490bc-0763-f091-14d1-aaa7110cffa7} {Suffrage:Voter ID:server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f Address:0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f} {Suffrage:Voter ID:server-c362673c-55f1-52a9-5707-4e0c19ed628e Address:c362673c-55f1-52a9-5707-4e0c19ed628e}]"
2026-05-25T23:24:01.551Z [INFO]  server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f: entering follower state: follower="Node at 0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f [Follower]" leader-address= leader-id=
2026-05-25T23:24:01.551Z [INFO]  server-c362673c-55f1-52a9-5707-4e0c19ed628e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9fd490bc-0763-f091-14d1-aaa7110cffa7 Address:9fd490bc-0763-f091-14d1-aaa7110cffa7} {Suffrage:Voter ID:server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f Address:0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f} {Suffrage:Voter ID:server-c362673c-55f1-52a9-5707-4e0c19ed628e Address:c362673c-55f1-52a9-5707-4e0c19ed628e}]"
2026-05-25T23:24:01.551Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:01.551Z [INFO]  server-c362673c-55f1-52a9-5707-4e0c19ed628e: entering follower state: follower="Node at c362673c-55f1-52a9-5707-4e0c19ed628e [Follower]" leader-address= leader-id=
2026-05-25T23:24:01.612Z [WARN]  server-9fd490bc-0763-f091-14d1-aaa7110cffa7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:01.612Z [INFO]  server-9fd490bc-0763-f091-14d1-aaa7110cffa7: entering candidate state: node="Node at 9fd490bc-0763-f091-14d1-aaa7110cffa7 [Candidate]" term=2
2026-05-25T23:24:01.612Z [INFO]  server-9fd490bc-0763-f091-14d1-aaa7110cffa7: election won: term=2 tally=2
2026-05-25T23:24:01.612Z [INFO]  server-9fd490bc-0763-f091-14d1-aaa7110cffa7: entering leader state: leader="Node at 9fd490bc-0763-f091-14d1-aaa7110cffa7 [Leader]"
2026-05-25T23:24:01.612Z [INFO]  server-9fd490bc-0763-f091-14d1-aaa7110cffa7: added peer, starting replication: peer=server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f
2026-05-25T23:24:01.612Z [INFO]  server-9fd490bc-0763-f091-14d1-aaa7110cffa7: added peer, starting replication: peer=server-c362673c-55f1-52a9-5707-4e0c19ed628e
2026-05-25T23:24:01.613Z [INFO]  server-9fd490bc-0763-f091-14d1-aaa7110cffa7: pipelining replication: peer="{Voter server-c362673c-55f1-52a9-5707-4e0c19ed628e c362673c-55f1-52a9-5707-4e0c19ed628e}"
2026-05-25T23:24:01.613Z [INFO]  server-9fd490bc-0763-f091-14d1-aaa7110cffa7: pipelining replication: peer="{Voter server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f 0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f}"
2026-05-25T23:24:01.718Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:01.612975147 +0000 UTC m=+97.259422682 (2 nodes), 61.604085ms from start of poll, 61.914338ms from cluster start. Timeout at 2026-05-25 23:24:01.717978175 +0000 UTC m=+97.364425632, 105.00295ms after stability
2026-05-25T23:24:01.718Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:01.824Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:01.718149872 +0000 UTC m=+97.364597347 (1 nodes), 666ns from start of poll, 167.089003ms from cluster start. Timeout at 2026-05-25 23:24:01.823167476 +0000 UTC m=+97.469614927, 105.01758ms after stability
2026-05-25T23:24:01.824Z [ERROR] server-9fd490bc-0763-f091-14d1-aaa7110cffa7: failed to pipeline appendEntries: peer="{Voter server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f 0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f}" error="append pipeline closed"
2026-05-25T23:24:01.824Z [INFO]  server-9fd490bc-0763-f091-14d1-aaa7110cffa7: aborting pipeline replication: peer="{Voter server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f 0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f}"
2026-05-25T23:24:01.824Z [ERROR] server-9fd490bc-0763-f091-14d1-aaa7110cffa7: failed to appendEntries to: peer="{Voter server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f 0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f}" error="failed to connect to peer: 0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f"
    raft_test.go:1627: [INFO] Finished apply without behind follower
2026-05-25T23:24:01.826Z [INFO]  server-9fd490bc-0763-f091-14d1-aaa7110cffa7: starting snapshot up to: index=102
2026-05-25T23:24:01.826Z [INFO]  creating new snapshot: path=/tmp/raft2782677263/snapshots/2-102-1779751441826.tmp
2026-05-25T23:24:01.826Z [ERROR] server-9fd490bc-0763-f091-14d1-aaa7110cffa7: failed to heartbeat to: peer=0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f backoff time=10ms error="failed to connect to peer: 0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f"
2026-05-25T23:24:01.826Z [INFO]  server-9fd490bc-0763-f091-14d1-aaa7110cffa7: compacting logs: from=1 to=92
2026-05-25T23:24:01.826Z [INFO]  server-9fd490bc-0763-f091-14d1-aaa7110cffa7: snapshot complete up to: index=102
2026-05-25T23:24:01.826Z [ERROR] server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f: failed to take snapshot: error="nothing new to snapshot"
2026-05-25T23:24:01.826Z [INFO]  server-c362673c-55f1-52a9-5707-4e0c19ed628e: starting snapshot up to: index=93
2026-05-25T23:24:01.826Z [INFO]  creating new snapshot: path=/tmp/raft3490259636/snapshots/2-93-1779751441826.tmp
2026-05-25T23:24:01.827Z [INFO]  server-c362673c-55f1-52a9-5707-4e0c19ed628e: compacting logs: from=1 to=92
2026-05-25T23:24:01.827Z [INFO]  server-c362673c-55f1-52a9-5707-4e0c19ed628e: snapshot complete up to: index=93
2026-05-25T23:24:01.834Z [ERROR] server-9fd490bc-0763-f091-14d1-aaa7110cffa7: failed to get log: index=2 error="log not found"
2026-05-25T23:24:01.835Z [INFO]  creating new snapshot: path=/tmp/raft4250859965/snapshots/2-102-1779751441835.tmp
2026-05-25T23:24:01.835Z [INFO]  server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2026-05-25T23:24:01.835Z [INFO]  server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f: copied to local snapshot: bytes=693
2026-05-25T23:24:01.836Z [INFO]  server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f: snapshot restore progress: id=2-102-1779751441835 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-05-25T23:24:01.836Z [INFO]  server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f: Installed remote snapshot
2026-05-25T23:24:01.836Z [WARN]  server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f: skipping application of old log: index=102
2026-05-25T23:24:01.836Z [INFO]  server-9fd490bc-0763-f091-14d1-aaa7110cffa7: pipelining replication: peer="{Voter server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f 0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f}"
2026-05-25T23:24:01.838Z [INFO]  server-9fd490bc-0763-f091-14d1-aaa7110cffa7: aborting pipeline replication: peer="{Voter server-c362673c-55f1-52a9-5707-4e0c19ed628e c362673c-55f1-52a9-5707-4e0c19ed628e}"
2026-05-25T23:24:01.838Z [INFO]  server-9fd490bc-0763-f091-14d1-aaa7110cffa7: aborting pipeline replication: peer="{Voter server-0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f 0ea3f3cc-b1fd-c5e6-01f7-1adcc8119a4f}"
--- PASS: TestRaft_SendSnapshotFollower (0.29s)
=== RUN   TestRaft_SendSnapshotAndLogsFollower
2026-05-25T23:24:01.839Z [INFO]  server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e Address:057cbbbf-d14b-0228-63ba-cbc0f9ced40e} {Suffrage:Voter ID:server-85ab180e-e073-15d5-2563-2a559f883db5 Address:85ab180e-e073-15d5-2563-2a559f883db5} {Suffrage:Voter ID:server-8a526b14-c626-b0e9-161c-088cb38fdf9f Address:8a526b14-c626-b0e9-161c-088cb38fdf9f}]"
2026-05-25T23:24:01.839Z [INFO]  server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: entering follower state: follower="Node at 057cbbbf-d14b-0228-63ba-cbc0f9ced40e [Follower]" leader-address= leader-id=
2026-05-25T23:24:01.839Z [INFO]  server-85ab180e-e073-15d5-2563-2a559f883db5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e Address:057cbbbf-d14b-0228-63ba-cbc0f9ced40e} {Suffrage:Voter ID:server-85ab180e-e073-15d5-2563-2a559f883db5 Address:85ab180e-e073-15d5-2563-2a559f883db5} {Suffrage:Voter ID:server-8a526b14-c626-b0e9-161c-088cb38fdf9f Address:8a526b14-c626-b0e9-161c-088cb38fdf9f}]"
2026-05-25T23:24:01.839Z [INFO]  server-8a526b14-c626-b0e9-161c-088cb38fdf9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e Address:057cbbbf-d14b-0228-63ba-cbc0f9ced40e} {Suffrage:Voter ID:server-85ab180e-e073-15d5-2563-2a559f883db5 Address:85ab180e-e073-15d5-2563-2a559f883db5} {Suffrage:Voter ID:server-8a526b14-c626-b0e9-161c-088cb38fdf9f Address:8a526b14-c626-b0e9-161c-088cb38fdf9f}]"
2026-05-25T23:24:01.839Z [INFO]  server-85ab180e-e073-15d5-2563-2a559f883db5: entering follower state: follower="Node at 85ab180e-e073-15d5-2563-2a559f883db5 [Follower]" leader-address= leader-id=
2026-05-25T23:24:01.839Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:01.839Z [INFO]  server-8a526b14-c626-b0e9-161c-088cb38fdf9f: entering follower state: follower="Node at 8a526b14-c626-b0e9-161c-088cb38fdf9f [Follower]" leader-address= leader-id=
2026-05-25T23:24:01.909Z [WARN]  server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:01.909Z [INFO]  server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: entering candidate state: node="Node at 057cbbbf-d14b-0228-63ba-cbc0f9ced40e [Candidate]" term=2
2026-05-25T23:24:01.909Z [INFO]  server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: election won: term=2 tally=2
2026-05-25T23:24:01.909Z [INFO]  server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: entering leader state: leader="Node at 057cbbbf-d14b-0228-63ba-cbc0f9ced40e [Leader]"
2026-05-25T23:24:01.909Z [INFO]  server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: added peer, starting replication: peer=server-85ab180e-e073-15d5-2563-2a559f883db5
2026-05-25T23:24:01.909Z [INFO]  server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: added peer, starting replication: peer=server-8a526b14-c626-b0e9-161c-088cb38fdf9f
2026-05-25T23:24:01.909Z [INFO]  server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: pipelining replication: peer="{Voter server-8a526b14-c626-b0e9-161c-088cb38fdf9f 8a526b14-c626-b0e9-161c-088cb38fdf9f}"
2026-05-25T23:24:01.909Z [INFO]  server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: pipelining replication: peer="{Voter server-85ab180e-e073-15d5-2563-2a559f883db5 85ab180e-e073-15d5-2563-2a559f883db5}"
2026-05-25T23:24:02.014Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:01.909339868 +0000 UTC m=+97.555787318 (2 nodes), 69.453877ms from start of poll, 69.773352ms from cluster start. Timeout at 2026-05-25 23:24:02.014354277 +0000 UTC m=+97.660801756, 105.014438ms after stability
2026-05-25T23:24:02.014Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:02.119Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:02.014560124 +0000 UTC m=+97.661007591 (1 nodes), 2.688µs from start of poll, 174.993625ms from cluster start. Timeout at 2026-05-25 23:24:02.119597017 +0000 UTC m=+97.766044467, 105.036876ms after stability
2026-05-25T23:24:02.119Z [ERROR] server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: failed to pipeline appendEntries: peer="{Voter server-85ab180e-e073-15d5-2563-2a559f883db5 85ab180e-e073-15d5-2563-2a559f883db5}" error="append pipeline closed"
2026-05-25T23:24:02.119Z [INFO]  server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: aborting pipeline replication: peer="{Voter server-85ab180e-e073-15d5-2563-2a559f883db5 85ab180e-e073-15d5-2563-2a559f883db5}"
2026-05-25T23:24:02.120Z [ERROR] server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: failed to appendEntries to: peer="{Voter server-85ab180e-e073-15d5-2563-2a559f883db5 85ab180e-e073-15d5-2563-2a559f883db5}" error="failed to connect to peer: 85ab180e-e073-15d5-2563-2a559f883db5"
    raft_test.go:1669: [INFO] Finished apply without behind follower
2026-05-25T23:24:02.122Z [INFO]  server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: starting snapshot up to: index=102
2026-05-25T23:24:02.122Z [INFO]  creating new snapshot: path=/tmp/raft2961884834/snapshots/2-102-1779751442122.tmp
2026-05-25T23:24:02.122Z [ERROR] server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: failed to heartbeat to: peer=85ab180e-e073-15d5-2563-2a559f883db5 backoff time=10ms error="failed to connect to peer: 85ab180e-e073-15d5-2563-2a559f883db5"
2026-05-25T23:24:02.123Z [INFO]  server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: compacting logs: from=1 to=92
2026-05-25T23:24:02.123Z [INFO]  server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: snapshot complete up to: index=102
2026-05-25T23:24:02.123Z [ERROR] server-85ab180e-e073-15d5-2563-2a559f883db5: failed to take snapshot: error="nothing new to snapshot"
2026-05-25T23:24:02.123Z [INFO]  server-8a526b14-c626-b0e9-161c-088cb38fdf9f: starting snapshot up to: index=97
2026-05-25T23:24:02.123Z [INFO]  creating new snapshot: path=/tmp/raft3696230416/snapshots/2-97-1779751442123.tmp
2026-05-25T23:24:02.124Z [INFO]  server-8a526b14-c626-b0e9-161c-088cb38fdf9f: compacting logs: from=1 to=92
2026-05-25T23:24:02.124Z [INFO]  server-8a526b14-c626-b0e9-161c-088cb38fdf9f: snapshot complete up to: index=97
    raft_test.go:1690: [INFO] Finished apply without behind follower
2026-05-25T23:24:02.130Z [ERROR] server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: failed to get log: index=2 error="log not found"
2026-05-25T23:24:02.130Z [INFO]  creating new snapshot: path=/tmp/raft2744978994/snapshots/2-102-1779751442130.tmp
2026-05-25T23:24:02.130Z [INFO]  server-85ab180e-e073-15d5-2563-2a559f883db5: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2026-05-25T23:24:02.131Z [INFO]  server-85ab180e-e073-15d5-2563-2a559f883db5: copied to local snapshot: bytes=693
2026-05-25T23:24:02.131Z [INFO]  server-85ab180e-e073-15d5-2563-2a559f883db5: snapshot restore progress: id=2-102-1779751442130 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-05-25T23:24:02.131Z [INFO]  server-85ab180e-e073-15d5-2563-2a559f883db5: Installed remote snapshot
2026-05-25T23:24:02.131Z [INFO]  server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: pipelining replication: peer="{Voter server-85ab180e-e073-15d5-2563-2a559f883db5 85ab180e-e073-15d5-2563-2a559f883db5}"
2026-05-25T23:24:02.138Z [INFO]  server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: aborting pipeline replication: peer="{Voter server-8a526b14-c626-b0e9-161c-088cb38fdf9f 8a526b14-c626-b0e9-161c-088cb38fdf9f}"
2026-05-25T23:24:02.138Z [INFO]  server-057cbbbf-d14b-0228-63ba-cbc0f9ced40e: aborting pipeline replication: peer="{Voter server-85ab180e-e073-15d5-2563-2a559f883db5 85ab180e-e073-15d5-2563-2a559f883db5}"
--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.30s)
=== RUN   TestRaft_ReJoinFollower
2026-05-25T23:24:02.139Z [INFO]  server-59b1de5d-58d4-59a4-41a7-b298d7a96f32: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59b1de5d-58d4-59a4-41a7-b298d7a96f32 Address:59b1de5d-58d4-59a4-41a7-b298d7a96f32} {Suffrage:Voter ID:server-a76c04b0-1c33-49d2-5bb9-c8aaa0644973 Address:a76c04b0-1c33-49d2-5bb9-c8aaa0644973} {Suffrage:Voter ID:server-ac95ff9b-b9a9-a049-6b4e-216b154594cf Address:ac95ff9b-b9a9-a049-6b4e-216b154594cf}]"
2026-05-25T23:24:02.139Z [INFO]  server-a76c04b0-1c33-49d2-5bb9-c8aaa0644973: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59b1de5d-58d4-59a4-41a7-b298d7a96f32 Address:59b1de5d-58d4-59a4-41a7-b298d7a96f32} {Suffrage:Voter ID:server-a76c04b0-1c33-49d2-5bb9-c8aaa0644973 Address:a76c04b0-1c33-49d2-5bb9-c8aaa0644973} {Suffrage:Voter ID:server-ac95ff9b-b9a9-a049-6b4e-216b154594cf Address:ac95ff9b-b9a9-a049-6b4e-216b154594cf}]"
2026-05-25T23:24:02.139Z [INFO]  server-59b1de5d-58d4-59a4-41a7-b298d7a96f32: entering follower state: follower="Node at 59b1de5d-58d4-59a4-41a7-b298d7a96f32 [Follower]" leader-address= leader-id=
2026-05-25T23:24:02.139Z [INFO]  server-a76c04b0-1c33-49d2-5bb9-c8aaa0644973: entering follower state: follower="Node at a76c04b0-1c33-49d2-5bb9-c8aaa0644973 [Follower]" leader-address= leader-id=
2026-05-25T23:24:02.139Z [INFO]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-59b1de5d-58d4-59a4-41a7-b298d7a96f32 Address:59b1de5d-58d4-59a4-41a7-b298d7a96f32} {Suffrage:Voter ID:server-a76c04b0-1c33-49d2-5bb9-c8aaa0644973 Address:a76c04b0-1c33-49d2-5bb9-c8aaa0644973} {Suffrage:Voter ID:server-ac95ff9b-b9a9-a049-6b4e-216b154594cf Address:ac95ff9b-b9a9-a049-6b4e-216b154594cf}]"
2026-05-25T23:24:02.139Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:02.139Z [INFO]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: entering follower state: follower="Node at ac95ff9b-b9a9-a049-6b4e-216b154594cf [Follower]" leader-address= leader-id=
2026-05-25T23:24:02.190Z [WARN]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:02.190Z [INFO]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: entering candidate state: node="Node at ac95ff9b-b9a9-a049-6b4e-216b154594cf [Candidate]" term=2
2026-05-25T23:24:02.190Z [INFO]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: election won: term=2 tally=2
2026-05-25T23:24:02.190Z [INFO]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: entering leader state: leader="Node at ac95ff9b-b9a9-a049-6b4e-216b154594cf [Leader]"
2026-05-25T23:24:02.190Z [INFO]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: added peer, starting replication: peer=server-59b1de5d-58d4-59a4-41a7-b298d7a96f32
2026-05-25T23:24:02.190Z [INFO]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: added peer, starting replication: peer=server-a76c04b0-1c33-49d2-5bb9-c8aaa0644973
2026-05-25T23:24:02.190Z [INFO]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: pipelining replication: peer="{Voter server-59b1de5d-58d4-59a4-41a7-b298d7a96f32 59b1de5d-58d4-59a4-41a7-b298d7a96f32}"
2026-05-25T23:24:02.190Z [INFO]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: pipelining replication: peer="{Voter server-a76c04b0-1c33-49d2-5bb9-c8aaa0644973 a76c04b0-1c33-49d2-5bb9-c8aaa0644973}"
2026-05-25T23:24:02.295Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:02.190364744 +0000 UTC m=+97.836812213 (1 nodes), 51.005875ms from start of poll, 51.269649ms from cluster start. Timeout at 2026-05-25 23:24:02.295369189 +0000 UTC m=+97.941816650, 105.004437ms after stability
2026-05-25T23:24:02.300Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:02.406Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:02.300546184 +0000 UTC m=+97.946993650 (2 nodes), 1.904µs from start of poll, 161.451086ms from cluster start. Timeout at 2026-05-25 23:24:02.405580763 +0000 UTC m=+98.052028227, 105.034577ms after stability
2026-05-25T23:24:02.406Z [INFO]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: updating configuration: command=RemoveServer server-id=server-59b1de5d-58d4-59a4-41a7-b298d7a96f32 server-addr= servers="[{Suffrage:Voter ID:server-a76c04b0-1c33-49d2-5bb9-c8aaa0644973 Address:a76c04b0-1c33-49d2-5bb9-c8aaa0644973} {Suffrage:Voter ID:server-ac95ff9b-b9a9-a049-6b4e-216b154594cf Address:ac95ff9b-b9a9-a049-6b4e-216b154594cf}]"
2026-05-25T23:24:02.406Z [INFO]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: removed peer, stopping replication: peer=server-59b1de5d-58d4-59a4-41a7-b298d7a96f32 last-index=3
2026-05-25T23:24:02.406Z [INFO]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: aborting pipeline replication: peer="{Voter server-59b1de5d-58d4-59a4-41a7-b298d7a96f32 59b1de5d-58d4-59a4-41a7-b298d7a96f32}"
2026-05-25T23:24:02.458Z [WARN]  server-59b1de5d-58d4-59a4-41a7-b298d7a96f32: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2026-05-25T23:24:02.511Z [INFO]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: updating configuration: command=AddVoter server-id=server-59b1de5d-58d4-59a4-41a7-b298d7a96f32 server-addr=59b1de5d-58d4-59a4-41a7-b298d7a96f32 servers="[{Suffrage:Voter ID:server-a76c04b0-1c33-49d2-5bb9-c8aaa0644973 Address:a76c04b0-1c33-49d2-5bb9-c8aaa0644973} {Suffrage:Voter ID:server-ac95ff9b-b9a9-a049-6b4e-216b154594cf Address:ac95ff9b-b9a9-a049-6b4e-216b154594cf} {Suffrage:Voter ID:server-59b1de5d-58d4-59a4-41a7-b298d7a96f32 Address:59b1de5d-58d4-59a4-41a7-b298d7a96f32}]"
2026-05-25T23:24:02.511Z [INFO]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: added peer, starting replication: peer=server-59b1de5d-58d4-59a4-41a7-b298d7a96f32
2026-05-25T23:24:02.511Z [WARN]  server-59b1de5d-58d4-59a4-41a7-b298d7a96f32: failed to get previous log: previous-index=4 last-index=3 error="log not found"
2026-05-25T23:24:02.511Z [WARN]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: appendEntries rejected, sending older logs: peer="{Voter server-59b1de5d-58d4-59a4-41a7-b298d7a96f32 59b1de5d-58d4-59a4-41a7-b298d7a96f32}" next=4
2026-05-25T23:24:02.511Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:02.511Z [INFO]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: pipelining replication: peer="{Voter server-59b1de5d-58d4-59a4-41a7-b298d7a96f32 59b1de5d-58d4-59a4-41a7-b298d7a96f32}"
2026-05-25T23:24:02.617Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:02.511819734 +0000 UTC m=+98.158267185 (1 nodes), 443ns from start of poll, 372.724621ms from cluster start. Timeout at 2026-05-25 23:24:02.616845043 +0000 UTC m=+98.263292520, 105.025335ms after stability
2026-05-25T23:24:02.617Z [INFO]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: aborting pipeline replication: peer="{Voter server-a76c04b0-1c33-49d2-5bb9-c8aaa0644973 a76c04b0-1c33-49d2-5bb9-c8aaa0644973}"
2026-05-25T23:24:02.617Z [INFO]  server-ac95ff9b-b9a9-a049-6b4e-216b154594cf: aborting pipeline replication: peer="{Voter server-59b1de5d-58d4-59a4-41a7-b298d7a96f32 59b1de5d-58d4-59a4-41a7-b298d7a96f32}"
--- PASS: TestRaft_ReJoinFollower (0.48s)
=== RUN   TestRaft_LeaderLeaseExpire
2026-05-25T23:24:02.618Z [INFO]  server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b Address:fe61cb5f-e729-e5b2-0623-9f398ca0e90b} {Suffrage:Voter ID:server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa Address:1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa}]"
2026-05-25T23:24:02.618Z [INFO]  server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b: entering follower state: follower="Node at fe61cb5f-e729-e5b2-0623-9f398ca0e90b [Follower]" leader-address= leader-id=
2026-05-25T23:24:02.618Z [INFO]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b Address:fe61cb5f-e729-e5b2-0623-9f398ca0e90b} {Suffrage:Voter ID:server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa Address:1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa}]"
2026-05-25T23:24:02.618Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:02.618Z [INFO]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: entering follower state: follower="Node at 1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa [Follower]" leader-address= leader-id=
2026-05-25T23:24:02.707Z [WARN]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:02.707Z [INFO]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: entering candidate state: node="Node at 1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa [Candidate]" term=2
2026-05-25T23:24:02.707Z [INFO]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: election won: term=2 tally=2
2026-05-25T23:24:02.707Z [INFO]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: entering leader state: leader="Node at 1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa [Leader]"
2026-05-25T23:24:02.707Z [INFO]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: added peer, starting replication: peer=server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b
2026-05-25T23:24:02.708Z [INFO]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: pipelining replication: peer="{Voter server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b fe61cb5f-e729-e5b2-0623-9f398ca0e90b}"
2026-05-25T23:24:02.813Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:02.708007825 +0000 UTC m=+98.354455291 (1 nodes), 89.327331ms from start of poll, 89.584054ms from cluster start. Timeout at 2026-05-25 23:24:02.813008599 +0000 UTC m=+98.459456050, 105.000759ms after stability
2026-05-25T23:24:02.819Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:02.925Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:02.81979635 +0000 UTC m=+98.466243817 (1 nodes), 922ns from start of poll, 201.37258ms from cluster start. Timeout at 2026-05-25 23:24:02.924811623 +0000 UTC m=+98.571259095, 105.015278ms after stability
    raft_test.go:1799: [INFO] Disconnecting Node at fe61cb5f-e729-e5b2-0623-9f398ca0e90b [Follower]
2026-05-25T23:24:02.929Z [ERROR] server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: failed to heartbeat to: peer=fe61cb5f-e729-e5b2-0623-9f398ca0e90b backoff time=10ms error="failed to connect to peer: fe61cb5f-e729-e5b2-0623-9f398ca0e90b"
2026-05-25T23:24:02.935Z [ERROR] server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: failed to pipeline appendEntries: peer="{Voter server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b fe61cb5f-e729-e5b2-0623-9f398ca0e90b}" error="append pipeline closed"
2026-05-25T23:24:02.935Z [INFO]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: aborting pipeline replication: peer="{Voter server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b fe61cb5f-e729-e5b2-0623-9f398ca0e90b}"
2026-05-25T23:24:02.942Z [ERROR] server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: failed to appendEntries to: peer="{Voter server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b fe61cb5f-e729-e5b2-0623-9f398ca0e90b}" error="failed to connect to peer: fe61cb5f-e729-e5b2-0623-9f398ca0e90b"
2026-05-25T23:24:02.946Z [ERROR] server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: failed to heartbeat to: peer=fe61cb5f-e729-e5b2-0623-9f398ca0e90b backoff time=10ms error="failed to connect to peer: fe61cb5f-e729-e5b2-0623-9f398ca0e90b"
2026-05-25T23:24:02.958Z [ERROR] server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: failed to appendEntries to: peer="{Voter server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b fe61cb5f-e729-e5b2-0623-9f398ca0e90b}" error="failed to connect to peer: fe61cb5f-e729-e5b2-0623-9f398ca0e90b"
2026-05-25T23:24:02.963Z [ERROR] server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: failed to heartbeat to: peer=fe61cb5f-e729-e5b2-0623-9f398ca0e90b backoff time=10ms error="failed to connect to peer: fe61cb5f-e729-e5b2-0623-9f398ca0e90b"
2026-05-25T23:24:02.974Z [WARN]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: failed to contact: server-id=server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b time=50.467351ms
2026-05-25T23:24:02.974Z [WARN]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: failed to contact quorum of nodes, stepping down
2026-05-25T23:24:02.974Z [INFO]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: entering follower state: follower="Node at 1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa [Follower]" leader-address= leader-id=
2026-05-25T23:24:02.974Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:02.975Z [ERROR] server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: failed to appendEntries to: peer="{Voter server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b fe61cb5f-e729-e5b2-0623-9f398ca0e90b}" error="failed to connect to peer: fe61cb5f-e729-e5b2-0623-9f398ca0e90b"
2026-05-25T23:24:03.028Z [WARN]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:03.028Z [INFO]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: entering candidate state: node="Node at 1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa [Candidate]" term=3
2026-05-25T23:24:03.028Z [ERROR] server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: failed to make requestVote RPC: target="{Voter server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b fe61cb5f-e729-e5b2-0623-9f398ca0e90b}" error="failed to connect to peer: fe61cb5f-e729-e5b2-0623-9f398ca0e90b" term=3
2026-05-25T23:24:03.033Z [WARN]  server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b: heartbeat timeout reached, starting election: last-leader-addr=1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa last-leader-id=server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa
2026-05-25T23:24:03.033Z [INFO]  server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b: entering candidate state: node="Node at fe61cb5f-e729-e5b2-0623-9f398ca0e90b [Candidate]" term=3
2026-05-25T23:24:03.033Z [ERROR] server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b: failed to make requestVote RPC: target="{Voter server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa 1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa}" error="failed to connect to peer: 1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa" term=3
2026-05-25T23:24:03.089Z [WARN]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: Election timeout reached, restarting election
2026-05-25T23:24:03.089Z [INFO]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: entering candidate state: node="Node at 1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa [Candidate]" term=4
2026-05-25T23:24:03.089Z [ERROR] server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: failed to make requestVote RPC: target="{Voter server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b fe61cb5f-e729-e5b2-0623-9f398ca0e90b}" error="failed to connect to peer: fe61cb5f-e729-e5b2-0623-9f398ca0e90b" term=4
2026-05-25T23:24:03.090Z [WARN]  server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b: Election timeout reached, restarting election
2026-05-25T23:24:03.090Z [INFO]  server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b: entering candidate state: node="Node at fe61cb5f-e729-e5b2-0623-9f398ca0e90b [Candidate]" term=4
2026-05-25T23:24:03.090Z [ERROR] server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b: failed to make requestVote RPC: target="{Voter server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa 1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa}" error="failed to connect to peer: 1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa" term=4
2026-05-25T23:24:03.139Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:03.033257622 +0000 UTC m=+98.679705076 (0 nodes), 58.318765ms from start of poll, 414.833839ms from cluster start. Timeout at 2026-05-25 23:24:03.138263229 +0000 UTC m=+98.784710680, 105.005604ms after stability
2026-05-25T23:24:03.147Z [WARN]  server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b: Election timeout reached, restarting election
2026-05-25T23:24:03.147Z [INFO]  server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b: entering candidate state: node="Node at fe61cb5f-e729-e5b2-0623-9f398ca0e90b [Candidate]" term=5
2026-05-25T23:24:03.147Z [ERROR] server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b: failed to make requestVote RPC: target="{Voter server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa 1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa}" error="failed to connect to peer: 1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa" term=5
2026-05-25T23:24:03.177Z [WARN]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: Election timeout reached, restarting election
2026-05-25T23:24:03.177Z [INFO]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: entering candidate state: node="Node at 1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa [Candidate]" term=5
2026-05-25T23:24:03.177Z [ERROR] server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: failed to make requestVote RPC: target="{Voter server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b fe61cb5f-e729-e5b2-0623-9f398ca0e90b}" error="failed to connect to peer: fe61cb5f-e729-e5b2-0623-9f398ca0e90b" term=5
2026-05-25T23:24:03.225Z [WARN]  server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b: Election timeout reached, restarting election
2026-05-25T23:24:03.225Z [INFO]  server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b: entering candidate state: node="Node at fe61cb5f-e729-e5b2-0623-9f398ca0e90b [Candidate]" term=6
2026-05-25T23:24:03.226Z [ERROR] server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b: failed to make requestVote RPC: target="{Voter server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa 1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa}" error="failed to connect to peer: 1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa" term=6
2026-05-25T23:24:03.238Z [WARN]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: Election timeout reached, restarting election
2026-05-25T23:24:03.238Z [INFO]  server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: entering candidate state: node="Node at 1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa [Candidate]" term=6
2026-05-25T23:24:03.238Z [ERROR] server-1fa6a917-ae6e-cbdd-7761-b31ba9e24ffa: failed to make requestVote RPC: target="{Voter server-fe61cb5f-e729-e5b2-0623-9f398ca0e90b fe61cb5f-e729-e5b2-0623-9f398ca0e90b}" error="failed to connect to peer: fe61cb5f-e729-e5b2-0623-9f398ca0e90b" term=6
--- PASS: TestRaft_LeaderLeaseExpire (0.63s)
=== RUN   TestRaft_Barrier
2026-05-25T23:24:03.245Z [INFO]  server-18970866-a7cb-36c7-fb85-59786f8d61d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18970866-a7cb-36c7-fb85-59786f8d61d6 Address:18970866-a7cb-36c7-fb85-59786f8d61d6} {Suffrage:Voter ID:server-79ab2d70-08a9-fdc6-a2d1-72e7b0d9a221 Address:79ab2d70-08a9-fdc6-a2d1-72e7b0d9a221} {Suffrage:Voter ID:server-a148a064-8358-104d-8010-798e45cac021 Address:a148a064-8358-104d-8010-798e45cac021}]"
2026-05-25T23:24:03.245Z [INFO]  server-18970866-a7cb-36c7-fb85-59786f8d61d6: entering follower state: follower="Node at 18970866-a7cb-36c7-fb85-59786f8d61d6 [Follower]" leader-address= leader-id=
2026-05-25T23:24:03.245Z [INFO]  server-79ab2d70-08a9-fdc6-a2d1-72e7b0d9a221: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18970866-a7cb-36c7-fb85-59786f8d61d6 Address:18970866-a7cb-36c7-fb85-59786f8d61d6} {Suffrage:Voter ID:server-79ab2d70-08a9-fdc6-a2d1-72e7b0d9a221 Address:79ab2d70-08a9-fdc6-a2d1-72e7b0d9a221} {Suffrage:Voter ID:server-a148a064-8358-104d-8010-798e45cac021 Address:a148a064-8358-104d-8010-798e45cac021}]"
2026-05-25T23:24:03.245Z [INFO]  server-79ab2d70-08a9-fdc6-a2d1-72e7b0d9a221: entering follower state: follower="Node at 79ab2d70-08a9-fdc6-a2d1-72e7b0d9a221 [Follower]" leader-address= leader-id=
2026-05-25T23:24:03.245Z [INFO]  server-a148a064-8358-104d-8010-798e45cac021: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18970866-a7cb-36c7-fb85-59786f8d61d6 Address:18970866-a7cb-36c7-fb85-59786f8d61d6} {Suffrage:Voter ID:server-79ab2d70-08a9-fdc6-a2d1-72e7b0d9a221 Address:79ab2d70-08a9-fdc6-a2d1-72e7b0d9a221} {Suffrage:Voter ID:server-a148a064-8358-104d-8010-798e45cac021 Address:a148a064-8358-104d-8010-798e45cac021}]"
2026-05-25T23:24:03.245Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:03.245Z [INFO]  server-a148a064-8358-104d-8010-798e45cac021: entering follower state: follower="Node at a148a064-8358-104d-8010-798e45cac021 [Follower]" leader-address= leader-id=
2026-05-25T23:24:03.309Z [WARN]  server-18970866-a7cb-36c7-fb85-59786f8d61d6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:03.309Z [INFO]  server-18970866-a7cb-36c7-fb85-59786f8d61d6: entering candidate state: node="Node at 18970866-a7cb-36c7-fb85-59786f8d61d6 [Candidate]" term=2
2026-05-25T23:24:03.309Z [INFO]  server-18970866-a7cb-36c7-fb85-59786f8d61d6: election won: term=2 tally=2
2026-05-25T23:24:03.309Z [INFO]  server-18970866-a7cb-36c7-fb85-59786f8d61d6: entering leader state: leader="Node at 18970866-a7cb-36c7-fb85-59786f8d61d6 [Leader]"
2026-05-25T23:24:03.309Z [INFO]  server-18970866-a7cb-36c7-fb85-59786f8d61d6: added peer, starting replication: peer=server-79ab2d70-08a9-fdc6-a2d1-72e7b0d9a221
2026-05-25T23:24:03.309Z [INFO]  server-18970866-a7cb-36c7-fb85-59786f8d61d6: added peer, starting replication: peer=server-a148a064-8358-104d-8010-798e45cac021
2026-05-25T23:24:03.309Z [INFO]  server-18970866-a7cb-36c7-fb85-59786f8d61d6: pipelining replication: peer="{Voter server-79ab2d70-08a9-fdc6-a2d1-72e7b0d9a221 79ab2d70-08a9-fdc6-a2d1-72e7b0d9a221}"
2026-05-25T23:24:03.309Z [INFO]  server-18970866-a7cb-36c7-fb85-59786f8d61d6: pipelining replication: peer="{Voter server-a148a064-8358-104d-8010-798e45cac021 a148a064-8358-104d-8010-798e45cac021}"
2026-05-25T23:24:03.415Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:03.309916802 +0000 UTC m=+98.956364333 (1 nodes), 64.048916ms from start of poll, 64.44284ms from cluster start. Timeout at 2026-05-25 23:24:03.414932175 +0000 UTC m=+99.061379645, 105.015312ms after stability
2026-05-25T23:24:03.428Z [INFO]  server-18970866-a7cb-36c7-fb85-59786f8d61d6: aborting pipeline replication: peer="{Voter server-79ab2d70-08a9-fdc6-a2d1-72e7b0d9a221 79ab2d70-08a9-fdc6-a2d1-72e7b0d9a221}"
2026-05-25T23:24:03.428Z [INFO]  server-18970866-a7cb-36c7-fb85-59786f8d61d6: aborting pipeline replication: peer="{Voter server-a148a064-8358-104d-8010-798e45cac021 a148a064-8358-104d-8010-798e45cac021}"
--- PASS: TestRaft_Barrier (0.18s)
=== RUN   TestRaft_VerifyLeader
2026-05-25T23:24:03.430Z [INFO]  server-5736f165-124f-1217-4c08-e5bbb45cc50e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5736f165-124f-1217-4c08-e5bbb45cc50e Address:5736f165-124f-1217-4c08-e5bbb45cc50e} {Suffrage:Voter ID:server-51b940ed-6ad9-b1ad-04bd-472eef90e541 Address:51b940ed-6ad9-b1ad-04bd-472eef90e541} {Suffrage:Voter ID:server-c595dfbb-85a8-373d-3946-db5daffe4c15 Address:c595dfbb-85a8-373d-3946-db5daffe4c15}]"
2026-05-25T23:24:03.430Z [INFO]  server-51b940ed-6ad9-b1ad-04bd-472eef90e541: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5736f165-124f-1217-4c08-e5bbb45cc50e Address:5736f165-124f-1217-4c08-e5bbb45cc50e} {Suffrage:Voter ID:server-51b940ed-6ad9-b1ad-04bd-472eef90e541 Address:51b940ed-6ad9-b1ad-04bd-472eef90e541} {Suffrage:Voter ID:server-c595dfbb-85a8-373d-3946-db5daffe4c15 Address:c595dfbb-85a8-373d-3946-db5daffe4c15}]"
2026-05-25T23:24:03.430Z [INFO]  server-5736f165-124f-1217-4c08-e5bbb45cc50e: entering follower state: follower="Node at 5736f165-124f-1217-4c08-e5bbb45cc50e [Follower]" leader-address= leader-id=
2026-05-25T23:24:03.430Z [INFO]  server-51b940ed-6ad9-b1ad-04bd-472eef90e541: entering follower state: follower="Node at 51b940ed-6ad9-b1ad-04bd-472eef90e541 [Follower]" leader-address= leader-id=
2026-05-25T23:24:03.430Z [INFO]  server-c595dfbb-85a8-373d-3946-db5daffe4c15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5736f165-124f-1217-4c08-e5bbb45cc50e Address:5736f165-124f-1217-4c08-e5bbb45cc50e} {Suffrage:Voter ID:server-51b940ed-6ad9-b1ad-04bd-472eef90e541 Address:51b940ed-6ad9-b1ad-04bd-472eef90e541} {Suffrage:Voter ID:server-c595dfbb-85a8-373d-3946-db5daffe4c15 Address:c595dfbb-85a8-373d-3946-db5daffe4c15}]"
2026-05-25T23:24:03.430Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:03.430Z [INFO]  server-c595dfbb-85a8-373d-3946-db5daffe4c15: entering follower state: follower="Node at c595dfbb-85a8-373d-3946-db5daffe4c15 [Follower]" leader-address= leader-id=
2026-05-25T23:24:03.509Z [WARN]  server-c595dfbb-85a8-373d-3946-db5daffe4c15: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:03.509Z [INFO]  server-c595dfbb-85a8-373d-3946-db5daffe4c15: entering candidate state: node="Node at c595dfbb-85a8-373d-3946-db5daffe4c15 [Candidate]" term=2
2026-05-25T23:24:03.509Z [INFO]  server-c595dfbb-85a8-373d-3946-db5daffe4c15: election won: term=2 tally=2
2026-05-25T23:24:03.509Z [INFO]  server-c595dfbb-85a8-373d-3946-db5daffe4c15: entering leader state: leader="Node at c595dfbb-85a8-373d-3946-db5daffe4c15 [Leader]"
2026-05-25T23:24:03.509Z [INFO]  server-c595dfbb-85a8-373d-3946-db5daffe4c15: added peer, starting replication: peer=server-5736f165-124f-1217-4c08-e5bbb45cc50e
2026-05-25T23:24:03.509Z [INFO]  server-c595dfbb-85a8-373d-3946-db5daffe4c15: added peer, starting replication: peer=server-51b940ed-6ad9-b1ad-04bd-472eef90e541
2026-05-25T23:24:03.509Z [INFO]  server-c595dfbb-85a8-373d-3946-db5daffe4c15: pipelining replication: peer="{Voter server-5736f165-124f-1217-4c08-e5bbb45cc50e 5736f165-124f-1217-4c08-e5bbb45cc50e}"
2026-05-25T23:24:03.509Z [INFO]  server-c595dfbb-85a8-373d-3946-db5daffe4c15: pipelining replication: peer="{Voter server-51b940ed-6ad9-b1ad-04bd-472eef90e541 51b940ed-6ad9-b1ad-04bd-472eef90e541}"
2026-05-25T23:24:03.614Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:03.509544195 +0000 UTC m=+99.155991697 (1 nodes), 79.334565ms from start of poll, 79.624905ms from cluster start. Timeout at 2026-05-25 23:24:03.614549949 +0000 UTC m=+99.260997398, 105.005701ms after stability
2026-05-25T23:24:03.614Z [INFO]  server-c595dfbb-85a8-373d-3946-db5daffe4c15: aborting pipeline replication: peer="{Voter server-51b940ed-6ad9-b1ad-04bd-472eef90e541 51b940ed-6ad9-b1ad-04bd-472eef90e541}"
2026-05-25T23:24:03.614Z [INFO]  server-c595dfbb-85a8-373d-3946-db5daffe4c15: aborting pipeline replication: peer="{Voter server-5736f165-124f-1217-4c08-e5bbb45cc50e 5736f165-124f-1217-4c08-e5bbb45cc50e}"
--- PASS: TestRaft_VerifyLeader (0.19s)
=== RUN   TestRaft_VerifyLeader_Single
2026-05-25T23:24:03.615Z [INFO]  server-3f9dc938-c960-5534-eb4c-7b93a4157244: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f9dc938-c960-5534-eb4c-7b93a4157244 Address:3f9dc938-c960-5534-eb4c-7b93a4157244}]"
2026-05-25T23:24:03.615Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:03.615Z [INFO]  server-3f9dc938-c960-5534-eb4c-7b93a4157244: entering follower state: follower="Node at 3f9dc938-c960-5534-eb4c-7b93a4157244 [Follower]" leader-address= leader-id=
2026-05-25T23:24:03.696Z [WARN]  server-3f9dc938-c960-5534-eb4c-7b93a4157244: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:03.696Z [INFO]  server-3f9dc938-c960-5534-eb4c-7b93a4157244: entering candidate state: node="Node at 3f9dc938-c960-5534-eb4c-7b93a4157244 [Candidate]" term=2
2026-05-25T23:24:03.696Z [INFO]  server-3f9dc938-c960-5534-eb4c-7b93a4157244: election won: term=2 tally=1
2026-05-25T23:24:03.696Z [INFO]  server-3f9dc938-c960-5534-eb4c-7b93a4157244: entering leader state: leader="Node at 3f9dc938-c960-5534-eb4c-7b93a4157244 [Leader]"
2026-05-25T23:24:03.802Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:03.696913752 +0000 UTC m=+99.343361228 (1 nodes), 81.082676ms from start of poll, 81.230263ms from cluster start. Timeout at 2026-05-25 23:24:03.801920405 +0000 UTC m=+99.448367859, 105.006631ms after stability
--- PASS: TestRaft_VerifyLeader_Single (0.19s)
=== RUN   TestRaft_VerifyLeader_Fail
2026-05-25T23:24:03.802Z [INFO]  server-9ee8a899-8644-3a25-c904-89f65d447cf5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ee8a899-8644-3a25-c904-89f65d447cf5 Address:9ee8a899-8644-3a25-c904-89f65d447cf5} {Suffrage:Voter ID:server-acc091d1-7656-19f6-3a48-e8e2cdabceda Address:acc091d1-7656-19f6-3a48-e8e2cdabceda}]"
2026-05-25T23:24:03.802Z [INFO]  server-9ee8a899-8644-3a25-c904-89f65d447cf5: entering follower state: follower="Node at 9ee8a899-8644-3a25-c904-89f65d447cf5 [Follower]" leader-address= leader-id=
2026-05-25T23:24:03.802Z [INFO]  server-acc091d1-7656-19f6-3a48-e8e2cdabceda: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ee8a899-8644-3a25-c904-89f65d447cf5 Address:9ee8a899-8644-3a25-c904-89f65d447cf5} {Suffrage:Voter ID:server-acc091d1-7656-19f6-3a48-e8e2cdabceda Address:acc091d1-7656-19f6-3a48-e8e2cdabceda}]"
2026-05-25T23:24:03.802Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:03.802Z [INFO]  server-acc091d1-7656-19f6-3a48-e8e2cdabceda: entering follower state: follower="Node at acc091d1-7656-19f6-3a48-e8e2cdabceda [Follower]" leader-address= leader-id=
2026-05-25T23:24:03.892Z [WARN]  server-9ee8a899-8644-3a25-c904-89f65d447cf5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:03.892Z [INFO]  server-9ee8a899-8644-3a25-c904-89f65d447cf5: entering candidate state: node="Node at 9ee8a899-8644-3a25-c904-89f65d447cf5 [Candidate]" term=2
2026-05-25T23:24:03.893Z [INFO]  server-9ee8a899-8644-3a25-c904-89f65d447cf5: election won: term=2 tally=2
2026-05-25T23:24:03.893Z [INFO]  server-9ee8a899-8644-3a25-c904-89f65d447cf5: entering leader state: leader="Node at 9ee8a899-8644-3a25-c904-89f65d447cf5 [Leader]"
2026-05-25T23:24:03.893Z [INFO]  server-9ee8a899-8644-3a25-c904-89f65d447cf5: added peer, starting replication: peer=server-acc091d1-7656-19f6-3a48-e8e2cdabceda
2026-05-25T23:24:03.893Z [INFO]  server-9ee8a899-8644-3a25-c904-89f65d447cf5: pipelining replication: peer="{Voter server-acc091d1-7656-19f6-3a48-e8e2cdabceda acc091d1-7656-19f6-3a48-e8e2cdabceda}"
2026-05-25T23:24:03.998Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:03.893141422 +0000 UTC m=+99.539588889 (1 nodes), 90.196367ms from start of poll, 90.436413ms from cluster start. Timeout at 2026-05-25 23:24:03.998148427 +0000 UTC m=+99.644595917, 105.007028ms after stability
2026-05-25T23:24:03.998Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:04.103Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:03.998728812 +0000 UTC m=+99.645176270 (1 nodes), 1.401µs from start of poll, 196.023794ms from cluster start. Timeout at 2026-05-25 23:24:04.103765719 +0000 UTC m=+99.750213173, 105.036903ms after stability
2026-05-25T23:24:04.111Z [ERROR] server-9ee8a899-8644-3a25-c904-89f65d447cf5: peer has newer term, stopping replication: peer="{Voter server-acc091d1-7656-19f6-3a48-e8e2cdabceda acc091d1-7656-19f6-3a48-e8e2cdabceda}"
2026-05-25T23:24:04.111Z [INFO]  server-9ee8a899-8644-3a25-c904-89f65d447cf5: aborting pipeline replication: peer="{Voter server-acc091d1-7656-19f6-3a48-e8e2cdabceda acc091d1-7656-19f6-3a48-e8e2cdabceda}"
2026-05-25T23:24:04.111Z [INFO]  server-9ee8a899-8644-3a25-c904-89f65d447cf5: entering follower state: follower="Node at 9ee8a899-8644-3a25-c904-89f65d447cf5 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VerifyLeader_Fail (0.31s)
=== RUN   TestRaft_VerifyLeader_PartialConnect
2026-05-25T23:24:04.116Z [INFO]  server-90dd6fb3-f2f9-60d2-9744-f96e342adbf3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90dd6fb3-f2f9-60d2-9744-f96e342adbf3 Address:90dd6fb3-f2f9-60d2-9744-f96e342adbf3} {Suffrage:Voter ID:server-f17ac55d-d4b1-ee09-f647-e45de32689d5 Address:f17ac55d-d4b1-ee09-f647-e45de32689d5} {Suffrage:Voter ID:server-3d154c43-9cf7-c75b-fe41-f6a20a17e63d Address:3d154c43-9cf7-c75b-fe41-f6a20a17e63d}]"
2026-05-25T23:24:04.116Z [INFO]  server-90dd6fb3-f2f9-60d2-9744-f96e342adbf3: entering follower state: follower="Node at 90dd6fb3-f2f9-60d2-9744-f96e342adbf3 [Follower]" leader-address= leader-id=
2026-05-25T23:24:04.116Z [INFO]  server-f17ac55d-d4b1-ee09-f647-e45de32689d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90dd6fb3-f2f9-60d2-9744-f96e342adbf3 Address:90dd6fb3-f2f9-60d2-9744-f96e342adbf3} {Suffrage:Voter ID:server-f17ac55d-d4b1-ee09-f647-e45de32689d5 Address:f17ac55d-d4b1-ee09-f647-e45de32689d5} {Suffrage:Voter ID:server-3d154c43-9cf7-c75b-fe41-f6a20a17e63d Address:3d154c43-9cf7-c75b-fe41-f6a20a17e63d}]"
2026-05-25T23:24:04.116Z [INFO]  server-3d154c43-9cf7-c75b-fe41-f6a20a17e63d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90dd6fb3-f2f9-60d2-9744-f96e342adbf3 Address:90dd6fb3-f2f9-60d2-9744-f96e342adbf3} {Suffrage:Voter ID:server-f17ac55d-d4b1-ee09-f647-e45de32689d5 Address:f17ac55d-d4b1-ee09-f647-e45de32689d5} {Suffrage:Voter ID:server-3d154c43-9cf7-c75b-fe41-f6a20a17e63d Address:3d154c43-9cf7-c75b-fe41-f6a20a17e63d}]"
2026-05-25T23:24:04.116Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:04.116Z [INFO]  server-f17ac55d-d4b1-ee09-f647-e45de32689d5: entering follower state: follower="Node at f17ac55d-d4b1-ee09-f647-e45de32689d5 [Follower]" leader-address= leader-id=
2026-05-25T23:24:04.116Z [INFO]  server-3d154c43-9cf7-c75b-fe41-f6a20a17e63d: entering follower state: follower="Node at 3d154c43-9cf7-c75b-fe41-f6a20a17e63d [Follower]" leader-address= leader-id=
2026-05-25T23:24:04.167Z [WARN]  server-f17ac55d-d4b1-ee09-f647-e45de32689d5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:04.167Z [INFO]  server-f17ac55d-d4b1-ee09-f647-e45de32689d5: entering candidate state: node="Node at f17ac55d-d4b1-ee09-f647-e45de32689d5 [Candidate]" term=2
2026-05-25T23:24:04.167Z [INFO]  server-f17ac55d-d4b1-ee09-f647-e45de32689d5: election won: term=2 tally=2
2026-05-25T23:24:04.167Z [INFO]  server-f17ac55d-d4b1-ee09-f647-e45de32689d5: entering leader state: leader="Node at f17ac55d-d4b1-ee09-f647-e45de32689d5 [Leader]"
2026-05-25T23:24:04.167Z [INFO]  server-f17ac55d-d4b1-ee09-f647-e45de32689d5: added peer, starting replication: peer=server-90dd6fb3-f2f9-60d2-9744-f96e342adbf3
2026-05-25T23:24:04.167Z [INFO]  server-f17ac55d-d4b1-ee09-f647-e45de32689d5: added peer, starting replication: peer=server-3d154c43-9cf7-c75b-fe41-f6a20a17e63d
2026-05-25T23:24:04.167Z [INFO]  server-f17ac55d-d4b1-ee09-f647-e45de32689d5: pipelining replication: peer="{Voter server-3d154c43-9cf7-c75b-fe41-f6a20a17e63d 3d154c43-9cf7-c75b-fe41-f6a20a17e63d}"
2026-05-25T23:24:04.168Z [INFO]  server-f17ac55d-d4b1-ee09-f647-e45de32689d5: pipelining replication: peer="{Voter server-90dd6fb3-f2f9-60d2-9744-f96e342adbf3 90dd6fb3-f2f9-60d2-9744-f96e342adbf3}"
2026-05-25T23:24:04.273Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:04.167926325 +0000 UTC m=+99.814373779 (1 nodes), 51.215595ms from start of poll, 51.535464ms from cluster start. Timeout at 2026-05-25 23:24:04.272926984 +0000 UTC m=+99.919374434, 105.000655ms after stability
2026-05-25T23:24:04.278Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:04.384Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:04.278491374 +0000 UTC m=+99.924938841 (2 nodes), 1.056µs from start of poll, 162.100526ms from cluster start. Timeout at 2026-05-25 23:24:04.383512663 +0000 UTC m=+100.029960118, 105.021277ms after stability
    raft_test.go:1968: [INFO] Disconnecting Node at 90dd6fb3-f2f9-60d2-9744-f96e342adbf3 [Follower]
2026-05-25T23:24:04.384Z [ERROR] server-f17ac55d-d4b1-ee09-f647-e45de32689d5: failed to heartbeat to: peer=90dd6fb3-f2f9-60d2-9744-f96e342adbf3 backoff time=10ms error="failed to connect to peer: 90dd6fb3-f2f9-60d2-9744-f96e342adbf3"
2026-05-25T23:24:04.384Z [INFO]  server-f17ac55d-d4b1-ee09-f647-e45de32689d5: aborting pipeline replication: peer="{Voter server-3d154c43-9cf7-c75b-fe41-f6a20a17e63d 3d154c43-9cf7-c75b-fe41-f6a20a17e63d}"
2026-05-25T23:24:04.384Z [INFO]  server-f17ac55d-d4b1-ee09-f647-e45de32689d5: aborting pipeline replication: peer="{Voter server-90dd6fb3-f2f9-60d2-9744-f96e342adbf3 90dd6fb3-f2f9-60d2-9744-f96e342adbf3}"
--- PASS: TestRaft_VerifyLeader_PartialConnect (0.27s)
=== RUN   TestRaft_NotifyCh
2026-05-25T23:24:04.385Z [INFO]  server-2152f91a-8496-a84c-616d-43626060f123: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2152f91a-8496-a84c-616d-43626060f123 Address:2152f91a-8496-a84c-616d-43626060f123}]"
2026-05-25T23:24:04.385Z [INFO]  server-2152f91a-8496-a84c-616d-43626060f123: entering follower state: follower="Node at 2152f91a-8496-a84c-616d-43626060f123 [Follower]" leader-address= leader-id=
2026-05-25T23:24:04.455Z [WARN]  server-2152f91a-8496-a84c-616d-43626060f123: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:04.455Z [INFO]  server-2152f91a-8496-a84c-616d-43626060f123: entering candidate state: node="Node at 2152f91a-8496-a84c-616d-43626060f123 [Candidate]" term=2
2026-05-25T23:24:04.455Z [INFO]  server-2152f91a-8496-a84c-616d-43626060f123: election won: term=2 tally=1
2026-05-25T23:24:04.455Z [INFO]  server-2152f91a-8496-a84c-616d-43626060f123: entering leader state: leader="Node at 2152f91a-8496-a84c-616d-43626060f123 [Leader]"
--- PASS: TestRaft_NotifyCh (0.07s)
=== RUN   TestRaft_AppendEntry
2026-05-25T23:24:04.456Z [INFO]  server-88775613-f63a-7d24-f5b3-85bb56635b77: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88775613-f63a-7d24-f5b3-85bb56635b77 Address:88775613-f63a-7d24-f5b3-85bb56635b77} {Suffrage:Voter ID:server-668bd6e9-fdd6-1713-43ad-0ff539a75434 Address:668bd6e9-fdd6-1713-43ad-0ff539a75434} {Suffrage:Voter ID:server-c7eed2d9-88f9-aac3-aa1b-71fd5383033d Address:c7eed2d9-88f9-aac3-aa1b-71fd5383033d}]"
2026-05-25T23:24:04.456Z [INFO]  server-668bd6e9-fdd6-1713-43ad-0ff539a75434: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88775613-f63a-7d24-f5b3-85bb56635b77 Address:88775613-f63a-7d24-f5b3-85bb56635b77} {Suffrage:Voter ID:server-668bd6e9-fdd6-1713-43ad-0ff539a75434 Address:668bd6e9-fdd6-1713-43ad-0ff539a75434} {Suffrage:Voter ID:server-c7eed2d9-88f9-aac3-aa1b-71fd5383033d Address:c7eed2d9-88f9-aac3-aa1b-71fd5383033d}]"
2026-05-25T23:24:04.456Z [INFO]  server-88775613-f63a-7d24-f5b3-85bb56635b77: entering follower state: follower="Node at 88775613-f63a-7d24-f5b3-85bb56635b77 [Follower]" leader-address= leader-id=
2026-05-25T23:24:04.456Z [INFO]  server-668bd6e9-fdd6-1713-43ad-0ff539a75434: entering follower state: follower="Node at 668bd6e9-fdd6-1713-43ad-0ff539a75434 [Follower]" leader-address= leader-id=
2026-05-25T23:24:04.456Z [INFO]  server-c7eed2d9-88f9-aac3-aa1b-71fd5383033d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88775613-f63a-7d24-f5b3-85bb56635b77 Address:88775613-f63a-7d24-f5b3-85bb56635b77} {Suffrage:Voter ID:server-668bd6e9-fdd6-1713-43ad-0ff539a75434 Address:668bd6e9-fdd6-1713-43ad-0ff539a75434} {Suffrage:Voter ID:server-c7eed2d9-88f9-aac3-aa1b-71fd5383033d Address:c7eed2d9-88f9-aac3-aa1b-71fd5383033d}]"
2026-05-25T23:24:04.456Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:04.456Z [INFO]  server-c7eed2d9-88f9-aac3-aa1b-71fd5383033d: entering follower state: follower="Node at c7eed2d9-88f9-aac3-aa1b-71fd5383033d [Follower]" leader-address= leader-id=
2026-05-25T23:24:04.508Z [WARN]  server-88775613-f63a-7d24-f5b3-85bb56635b77: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:04.508Z [INFO]  server-88775613-f63a-7d24-f5b3-85bb56635b77: entering candidate state: node="Node at 88775613-f63a-7d24-f5b3-85bb56635b77 [Candidate]" term=2
2026-05-25T23:24:04.508Z [INFO]  server-88775613-f63a-7d24-f5b3-85bb56635b77: election won: term=2 tally=2
2026-05-25T23:24:04.508Z [INFO]  server-88775613-f63a-7d24-f5b3-85bb56635b77: entering leader state: leader="Node at 88775613-f63a-7d24-f5b3-85bb56635b77 [Leader]"
2026-05-25T23:24:04.508Z [INFO]  server-88775613-f63a-7d24-f5b3-85bb56635b77: added peer, starting replication: peer=server-668bd6e9-fdd6-1713-43ad-0ff539a75434
2026-05-25T23:24:04.508Z [INFO]  server-88775613-f63a-7d24-f5b3-85bb56635b77: added peer, starting replication: peer=server-c7eed2d9-88f9-aac3-aa1b-71fd5383033d
2026-05-25T23:24:04.509Z [INFO]  server-88775613-f63a-7d24-f5b3-85bb56635b77: pipelining replication: peer="{Voter server-c7eed2d9-88f9-aac3-aa1b-71fd5383033d c7eed2d9-88f9-aac3-aa1b-71fd5383033d}"
2026-05-25T23:24:04.509Z [INFO]  server-88775613-f63a-7d24-f5b3-85bb56635b77: pipelining replication: peer="{Voter server-668bd6e9-fdd6-1713-43ad-0ff539a75434 668bd6e9-fdd6-1713-43ad-0ff539a75434}"
2026-05-25T23:24:04.613Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:04.508905623 +0000 UTC m=+100.155353074 (2 nodes), 52.247509ms from start of poll, 52.581106ms from cluster start. Timeout at 2026-05-25 23:24:04.613906104 +0000 UTC m=+100.260353558, 105.000484ms after stability
2026-05-25T23:24:04.613Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:04.719Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:04.614009767 +0000 UTC m=+100.260457232 (1 nodes), 2.132µs from start of poll, 157.685264ms from cluster start. Timeout at 2026-05-25 23:24:04.719051416 +0000 UTC m=+100.365498873, 105.041641ms after stability
2026-05-25T23:24:04.719Z [WARN]  server-668bd6e9-fdd6-1713-43ad-0ff539a75434: clearing log suffix: from=1 to=2
2026-05-25T23:24:04.719Z [WARN]  server-668bd6e9-fdd6-1713-43ad-0ff539a75434: skipping application of old log: index=1
2026-05-25T23:24:04.719Z [WARN]  server-668bd6e9-fdd6-1713-43ad-0ff539a75434: skipping application of old log: index=1
2026-05-25T23:24:04.719Z [INFO]  server-88775613-f63a-7d24-f5b3-85bb56635b77: aborting pipeline replication: peer="{Voter server-c7eed2d9-88f9-aac3-aa1b-71fd5383033d c7eed2d9-88f9-aac3-aa1b-71fd5383033d}"
2026-05-25T23:24:04.719Z [INFO]  server-88775613-f63a-7d24-f5b3-85bb56635b77: aborting pipeline replication: peer="{Voter server-668bd6e9-fdd6-1713-43ad-0ff539a75434 668bd6e9-fdd6-1713-43ad-0ff539a75434}"
--- PASS: TestRaft_AppendEntry (0.26s)
=== RUN   TestRaft_VotingGrant_WhenLeaderAvailable
2026-05-25T23:24:04.720Z [INFO]  server-e6bcca96-a875-7c55-1f04-97501216e4d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6bcca96-a875-7c55-1f04-97501216e4d6 Address:e6bcca96-a875-7c55-1f04-97501216e4d6} {Suffrage:Voter ID:server-27ee6aae-1025-bfe1-8477-284d4364b73f Address:27ee6aae-1025-bfe1-8477-284d4364b73f} {Suffrage:Voter ID:server-cfb36e37-a387-c25d-1246-87c7cc2cbafc Address:cfb36e37-a387-c25d-1246-87c7cc2cbafc}]"
2026-05-25T23:24:04.720Z [INFO]  server-e6bcca96-a875-7c55-1f04-97501216e4d6: entering follower state: follower="Node at e6bcca96-a875-7c55-1f04-97501216e4d6 [Follower]" leader-address= leader-id=
2026-05-25T23:24:04.720Z [INFO]  server-27ee6aae-1025-bfe1-8477-284d4364b73f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6bcca96-a875-7c55-1f04-97501216e4d6 Address:e6bcca96-a875-7c55-1f04-97501216e4d6} {Suffrage:Voter ID:server-27ee6aae-1025-bfe1-8477-284d4364b73f Address:27ee6aae-1025-bfe1-8477-284d4364b73f} {Suffrage:Voter ID:server-cfb36e37-a387-c25d-1246-87c7cc2cbafc Address:cfb36e37-a387-c25d-1246-87c7cc2cbafc}]"
2026-05-25T23:24:04.720Z [INFO]  server-27ee6aae-1025-bfe1-8477-284d4364b73f: entering follower state: follower="Node at 27ee6aae-1025-bfe1-8477-284d4364b73f [Follower]" leader-address= leader-id=
2026-05-25T23:24:04.720Z [INFO]  server-cfb36e37-a387-c25d-1246-87c7cc2cbafc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6bcca96-a875-7c55-1f04-97501216e4d6 Address:e6bcca96-a875-7c55-1f04-97501216e4d6} {Suffrage:Voter ID:server-27ee6aae-1025-bfe1-8477-284d4364b73f Address:27ee6aae-1025-bfe1-8477-284d4364b73f} {Suffrage:Voter ID:server-cfb36e37-a387-c25d-1246-87c7cc2cbafc Address:cfb36e37-a387-c25d-1246-87c7cc2cbafc}]"
2026-05-25T23:24:04.720Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:04.720Z [INFO]  server-cfb36e37-a387-c25d-1246-87c7cc2cbafc: entering follower state: follower="Node at cfb36e37-a387-c25d-1246-87c7cc2cbafc [Follower]" leader-address= leader-id=
2026-05-25T23:24:04.783Z [WARN]  server-e6bcca96-a875-7c55-1f04-97501216e4d6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:04.783Z [INFO]  server-e6bcca96-a875-7c55-1f04-97501216e4d6: entering candidate state: node="Node at e6bcca96-a875-7c55-1f04-97501216e4d6 [Candidate]" term=2
2026-05-25T23:24:04.783Z [INFO]  server-e6bcca96-a875-7c55-1f04-97501216e4d6: election won: term=2 tally=2
2026-05-25T23:24:04.783Z [INFO]  server-e6bcca96-a875-7c55-1f04-97501216e4d6: entering leader state: leader="Node at e6bcca96-a875-7c55-1f04-97501216e4d6 [Leader]"
2026-05-25T23:24:04.783Z [INFO]  server-e6bcca96-a875-7c55-1f04-97501216e4d6: added peer, starting replication: peer=server-27ee6aae-1025-bfe1-8477-284d4364b73f
2026-05-25T23:24:04.783Z [INFO]  server-e6bcca96-a875-7c55-1f04-97501216e4d6: added peer, starting replication: peer=server-cfb36e37-a387-c25d-1246-87c7cc2cbafc
2026-05-25T23:24:04.783Z [INFO]  server-e6bcca96-a875-7c55-1f04-97501216e4d6: pipelining replication: peer="{Voter server-cfb36e37-a387-c25d-1246-87c7cc2cbafc cfb36e37-a387-c25d-1246-87c7cc2cbafc}"
2026-05-25T23:24:04.783Z [INFO]  server-e6bcca96-a875-7c55-1f04-97501216e4d6: pipelining replication: peer="{Voter server-27ee6aae-1025-bfe1-8477-284d4364b73f 27ee6aae-1025-bfe1-8477-284d4364b73f}"
2026-05-25T23:24:04.888Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:04.783425511 +0000 UTC m=+100.429872968 (2 nodes), 62.586941ms from start of poll, 63.012154ms from cluster start. Timeout at 2026-05-25 23:24:04.888436406 +0000 UTC m=+100.534883970, 105.011002ms after stability
2026-05-25T23:24:04.888Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:04.994Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:04.888981143 +0000 UTC m=+100.535428676 (1 nodes), 3.678µs from start of poll, 168.567862ms from cluster start. Timeout at 2026-05-25 23:24:04.99399134 +0000 UTC m=+100.640438831, 105.010155ms after stability
2026-05-25T23:24:04.994Z [WARN]  server-cfb36e37-a387-c25d-1246-87c7cc2cbafc: rejecting vote request since we have a leader: from=27ee6aae-1025-bfe1-8477-284d4364b73f leader=e6bcca96-a875-7c55-1f04-97501216e4d6 leader-id=server-e6bcca96-a875-7c55-1f04-97501216e4d6
2026-05-25T23:24:04.994Z [INFO]  server-e6bcca96-a875-7c55-1f04-97501216e4d6: aborting pipeline replication: peer="{Voter server-cfb36e37-a387-c25d-1246-87c7cc2cbafc cfb36e37-a387-c25d-1246-87c7cc2cbafc}"
2026-05-25T23:24:04.994Z [INFO]  server-e6bcca96-a875-7c55-1f04-97501216e4d6: aborting pipeline replication: peer="{Voter server-27ee6aae-1025-bfe1-8477-284d4364b73f 27ee6aae-1025-bfe1-8477-284d4364b73f}"
--- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.27s)
=== RUN   TestRaft_ProtocolVersion_RejectRPC
2026-05-25T23:24:04.995Z [INFO]  server-e85d7ee4-3c53-53c5-5253-16ce2f5be4fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e85d7ee4-3c53-53c5-5253-16ce2f5be4fa Address:e85d7ee4-3c53-53c5-5253-16ce2f5be4fa} {Suffrage:Voter ID:server-fc069fbc-4984-4a96-744f-7ef5e32abcd6 Address:fc069fbc-4984-4a96-744f-7ef5e32abcd6} {Suffrage:Voter ID:server-3b04abf9-bcf8-6199-c7b3-928a2984483f Address:3b04abf9-bcf8-6199-c7b3-928a2984483f}]"
2026-05-25T23:24:04.995Z [INFO]  server-fc069fbc-4984-4a96-744f-7ef5e32abcd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e85d7ee4-3c53-53c5-5253-16ce2f5be4fa Address:e85d7ee4-3c53-53c5-5253-16ce2f5be4fa} {Suffrage:Voter ID:server-fc069fbc-4984-4a96-744f-7ef5e32abcd6 Address:fc069fbc-4984-4a96-744f-7ef5e32abcd6} {Suffrage:Voter ID:server-3b04abf9-bcf8-6199-c7b3-928a2984483f Address:3b04abf9-bcf8-6199-c7b3-928a2984483f}]"
2026-05-25T23:24:04.995Z [INFO]  server-e85d7ee4-3c53-53c5-5253-16ce2f5be4fa: entering follower state: follower="Node at e85d7ee4-3c53-53c5-5253-16ce2f5be4fa [Follower]" leader-address= leader-id=
2026-05-25T23:24:04.995Z [INFO]  server-3b04abf9-bcf8-6199-c7b3-928a2984483f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e85d7ee4-3c53-53c5-5253-16ce2f5be4fa Address:e85d7ee4-3c53-53c5-5253-16ce2f5be4fa} {Suffrage:Voter ID:server-fc069fbc-4984-4a96-744f-7ef5e32abcd6 Address:fc069fbc-4984-4a96-744f-7ef5e32abcd6} {Suffrage:Voter ID:server-3b04abf9-bcf8-6199-c7b3-928a2984483f Address:3b04abf9-bcf8-6199-c7b3-928a2984483f}]"
2026-05-25T23:24:04.995Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:04.995Z [INFO]  server-fc069fbc-4984-4a96-744f-7ef5e32abcd6: entering follower state: follower="Node at fc069fbc-4984-4a96-744f-7ef5e32abcd6 [Follower]" leader-address= leader-id=
2026-05-25T23:24:04.995Z [INFO]  server-3b04abf9-bcf8-6199-c7b3-928a2984483f: entering follower state: follower="Node at 3b04abf9-bcf8-6199-c7b3-928a2984483f [Follower]" leader-address= leader-id=
2026-05-25T23:24:05.067Z [WARN]  server-e85d7ee4-3c53-53c5-5253-16ce2f5be4fa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:05.067Z [INFO]  server-e85d7ee4-3c53-53c5-5253-16ce2f5be4fa: entering candidate state: node="Node at e85d7ee4-3c53-53c5-5253-16ce2f5be4fa [Candidate]" term=2
2026-05-25T23:24:05.067Z [INFO]  server-e85d7ee4-3c53-53c5-5253-16ce2f5be4fa: election won: term=2 tally=2
2026-05-25T23:24:05.067Z [INFO]  server-e85d7ee4-3c53-53c5-5253-16ce2f5be4fa: entering leader state: leader="Node at e85d7ee4-3c53-53c5-5253-16ce2f5be4fa [Leader]"
2026-05-25T23:24:05.067Z [INFO]  server-e85d7ee4-3c53-53c5-5253-16ce2f5be4fa: added peer, starting replication: peer=server-fc069fbc-4984-4a96-744f-7ef5e32abcd6
2026-05-25T23:24:05.067Z [INFO]  server-e85d7ee4-3c53-53c5-5253-16ce2f5be4fa: added peer, starting replication: peer=server-3b04abf9-bcf8-6199-c7b3-928a2984483f
2026-05-25T23:24:05.068Z [INFO]  server-e85d7ee4-3c53-53c5-5253-16ce2f5be4fa: pipelining replication: peer="{Voter server-3b04abf9-bcf8-6199-c7b3-928a2984483f 3b04abf9-bcf8-6199-c7b3-928a2984483f}"
2026-05-25T23:24:05.068Z [INFO]  server-e85d7ee4-3c53-53c5-5253-16ce2f5be4fa: pipelining replication: peer="{Voter server-fc069fbc-4984-4a96-744f-7ef5e32abcd6 fc069fbc-4984-4a96-744f-7ef5e32abcd6}"
2026-05-25T23:24:05.173Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:05.068042056 +0000 UTC m=+100.714489569 (2 nodes), 72.683995ms from start of poll, 72.94988ms from cluster start. Timeout at 2026-05-25 23:24:05.173043392 +0000 UTC m=+100.819490862, 105.001293ms after stability
2026-05-25T23:24:05.173Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:05.279Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:05.173812551 +0000 UTC m=+100.820260005 (1 nodes), 1.072µs from start of poll, 178.720316ms from cluster start. Timeout at 2026-05-25 23:24:05.278833193 +0000 UTC m=+100.925280668, 105.020663ms after stability
2026-05-25T23:24:05.279Z [INFO]  server-e85d7ee4-3c53-53c5-5253-16ce2f5be4fa: aborting pipeline replication: peer="{Voter server-3b04abf9-bcf8-6199-c7b3-928a2984483f 3b04abf9-bcf8-6199-c7b3-928a2984483f}"
2026-05-25T23:24:05.279Z [INFO]  server-e85d7ee4-3c53-53c5-5253-16ce2f5be4fa: aborting pipeline replication: peer="{Voter server-fc069fbc-4984-4a96-744f-7ef5e32abcd6 fc069fbc-4984-4a96-744f-7ef5e32abcd6}"
--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.29s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_1_2
    raft_test.go:2151: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_2_3
    raft_test.go:2196: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s)
=== RUN   TestRaft_LeaderID_Propagated
2026-05-25T23:24:05.282Z [INFO]  server-5eb16126-4c22-7f07-06fc-bfaf6a2d9632: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5eb16126-4c22-7f07-06fc-bfaf6a2d9632 Address:5eb16126-4c22-7f07-06fc-bfaf6a2d9632} {Suffrage:Voter ID:server-799d475b-8a39-913a-28be-02924d1c8ee2 Address:799d475b-8a39-913a-28be-02924d1c8ee2} {Suffrage:Voter ID:server-d15ae51d-76d9-7473-d455-8e0da3523847 Address:d15ae51d-76d9-7473-d455-8e0da3523847}]"
2026-05-25T23:24:05.282Z [INFO]  server-799d475b-8a39-913a-28be-02924d1c8ee2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5eb16126-4c22-7f07-06fc-bfaf6a2d9632 Address:5eb16126-4c22-7f07-06fc-bfaf6a2d9632} {Suffrage:Voter ID:server-799d475b-8a39-913a-28be-02924d1c8ee2 Address:799d475b-8a39-913a-28be-02924d1c8ee2} {Suffrage:Voter ID:server-d15ae51d-76d9-7473-d455-8e0da3523847 Address:d15ae51d-76d9-7473-d455-8e0da3523847}]"
2026-05-25T23:24:05.282Z [INFO]  server-d15ae51d-76d9-7473-d455-8e0da3523847: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5eb16126-4c22-7f07-06fc-bfaf6a2d9632 Address:5eb16126-4c22-7f07-06fc-bfaf6a2d9632} {Suffrage:Voter ID:server-799d475b-8a39-913a-28be-02924d1c8ee2 Address:799d475b-8a39-913a-28be-02924d1c8ee2} {Suffrage:Voter ID:server-d15ae51d-76d9-7473-d455-8e0da3523847 Address:d15ae51d-76d9-7473-d455-8e0da3523847}]"
2026-05-25T23:24:05.282Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:05.282Z [INFO]  server-5eb16126-4c22-7f07-06fc-bfaf6a2d9632: entering follower state: follower="Node at 5eb16126-4c22-7f07-06fc-bfaf6a2d9632 [Follower]" leader-address= leader-id=
2026-05-25T23:24:05.282Z [INFO]  server-d15ae51d-76d9-7473-d455-8e0da3523847: entering follower state: follower="Node at d15ae51d-76d9-7473-d455-8e0da3523847 [Follower]" leader-address= leader-id=
2026-05-25T23:24:05.282Z [INFO]  server-799d475b-8a39-913a-28be-02924d1c8ee2: entering follower state: follower="Node at 799d475b-8a39-913a-28be-02924d1c8ee2 [Follower]" leader-address= leader-id=
2026-05-25T23:24:05.375Z [WARN]  server-799d475b-8a39-913a-28be-02924d1c8ee2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:05.375Z [INFO]  server-799d475b-8a39-913a-28be-02924d1c8ee2: entering candidate state: node="Node at 799d475b-8a39-913a-28be-02924d1c8ee2 [Candidate]" term=2
2026-05-25T23:24:05.375Z [INFO]  server-799d475b-8a39-913a-28be-02924d1c8ee2: election won: term=2 tally=2
2026-05-25T23:24:05.375Z [INFO]  server-799d475b-8a39-913a-28be-02924d1c8ee2: entering leader state: leader="Node at 799d475b-8a39-913a-28be-02924d1c8ee2 [Leader]"
2026-05-25T23:24:05.375Z [INFO]  server-799d475b-8a39-913a-28be-02924d1c8ee2: added peer, starting replication: peer=server-5eb16126-4c22-7f07-06fc-bfaf6a2d9632
2026-05-25T23:24:05.375Z [INFO]  server-799d475b-8a39-913a-28be-02924d1c8ee2: added peer, starting replication: peer=server-d15ae51d-76d9-7473-d455-8e0da3523847
2026-05-25T23:24:05.375Z [INFO]  server-799d475b-8a39-913a-28be-02924d1c8ee2: pipelining replication: peer="{Voter server-d15ae51d-76d9-7473-d455-8e0da3523847 d15ae51d-76d9-7473-d455-8e0da3523847}"
2026-05-25T23:24:05.375Z [INFO]  server-799d475b-8a39-913a-28be-02924d1c8ee2: pipelining replication: peer="{Voter server-5eb16126-4c22-7f07-06fc-bfaf6a2d9632 5eb16126-4c22-7f07-06fc-bfaf6a2d9632}"
2026-05-25T23:24:05.480Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:05.375588181 +0000 UTC m=+101.022035691 (1 nodes), 93.013006ms from start of poll, 93.306865ms from cluster start. Timeout at 2026-05-25 23:24:05.480589412 +0000 UTC m=+101.127036866, 105.001175ms after stability
2026-05-25T23:24:05.481Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:05.586Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:05.481014144 +0000 UTC m=+101.127461597 (1 nodes), 991ns from start of poll, 198.732771ms from cluster start. Timeout at 2026-05-25 23:24:05.586043351 +0000 UTC m=+101.232490805, 105.029208ms after stability
2026-05-25T23:24:05.586Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:05.692Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:05.586834687 +0000 UTC m=+101.233282138 (1 nodes), 925ns from start of poll, 304.553312ms from cluster start. Timeout at 2026-05-25 23:24:05.691853534 +0000 UTC m=+101.338301008, 105.01887ms after stability
2026-05-25T23:24:05.692Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:05.798Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:05.692785652 +0000 UTC m=+101.339233106 (1 nodes), 782ns from start of poll, 410.50428ms from cluster start. Timeout at 2026-05-25 23:24:05.797802914 +0000 UTC m=+101.444250391, 105.017285ms after stability
2026-05-25T23:24:05.798Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:05.904Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:05.798461666 +0000 UTC m=+101.444909120 (1 nodes), 679ns from start of poll, 516.180294ms from cluster start. Timeout at 2026-05-25 23:24:05.903487167 +0000 UTC m=+101.549934621, 105.025501ms after stability
2026-05-25T23:24:05.904Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:06.010Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:05.904479711 +0000 UTC m=+101.550927161 (1 nodes), 815ns from start of poll, 622.198335ms from cluster start. Timeout at 2026-05-25 23:24:06.009491045 +0000 UTC m=+101.655938516, 105.011355ms after stability
2026-05-25T23:24:06.116Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:06.221Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:06.116418307 +0000 UTC m=+101.762865762 (1 nodes), 2.26µs from start of poll, 834.136936ms from cluster start. Timeout at 2026-05-25 23:24:06.221441043 +0000 UTC m=+101.867888496, 105.022734ms after stability
2026-05-25T23:24:06.221Z [INFO]  server-799d475b-8a39-913a-28be-02924d1c8ee2: aborting pipeline replication: peer="{Voter server-d15ae51d-76d9-7473-d455-8e0da3523847 d15ae51d-76d9-7473-d455-8e0da3523847}"
2026-05-25T23:24:06.221Z [INFO]  server-799d475b-8a39-913a-28be-02924d1c8ee2: aborting pipeline replication: peer="{Voter server-5eb16126-4c22-7f07-06fc-bfaf6a2d9632 5eb16126-4c22-7f07-06fc-bfaf6a2d9632}"
--- PASS: TestRaft_LeaderID_Propagated (0.94s)
=== RUN   TestRaft_LeadershipTransferInProgress
--- PASS: TestRaft_LeadershipTransferInProgress (0.00s)
=== RUN   TestRaft_LeadershipTransferPickServer
--- PASS: TestRaft_LeadershipTransferPickServer (0.00s)
=== RUN   TestRaft_LeadershipTransfer
2026-05-25T23:24:06.223Z [INFO]  server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d Address:8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d} {Suffrage:Voter ID:server-5653aade-85db-e77f-3640-7a65072be792 Address:5653aade-85db-e77f-3640-7a65072be792} {Suffrage:Voter ID:server-937e3459-d53e-b16e-f11a-f2519af612ee Address:937e3459-d53e-b16e-f11a-f2519af612ee}]"
2026-05-25T23:24:06.223Z [INFO]  server-5653aade-85db-e77f-3640-7a65072be792: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d Address:8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d} {Suffrage:Voter ID:server-5653aade-85db-e77f-3640-7a65072be792 Address:5653aade-85db-e77f-3640-7a65072be792} {Suffrage:Voter ID:server-937e3459-d53e-b16e-f11a-f2519af612ee Address:937e3459-d53e-b16e-f11a-f2519af612ee}]"
2026-05-25T23:24:06.223Z [INFO]  server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d: entering follower state: follower="Node at 8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d [Follower]" leader-address= leader-id=
2026-05-25T23:24:06.223Z [INFO]  server-5653aade-85db-e77f-3640-7a65072be792: entering follower state: follower="Node at 5653aade-85db-e77f-3640-7a65072be792 [Follower]" leader-address= leader-id=
2026-05-25T23:24:06.223Z [INFO]  server-937e3459-d53e-b16e-f11a-f2519af612ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d Address:8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d} {Suffrage:Voter ID:server-5653aade-85db-e77f-3640-7a65072be792 Address:5653aade-85db-e77f-3640-7a65072be792} {Suffrage:Voter ID:server-937e3459-d53e-b16e-f11a-f2519af612ee Address:937e3459-d53e-b16e-f11a-f2519af612ee}]"
2026-05-25T23:24:06.223Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:06.223Z [INFO]  server-937e3459-d53e-b16e-f11a-f2519af612ee: entering follower state: follower="Node at 937e3459-d53e-b16e-f11a-f2519af612ee [Follower]" leader-address= leader-id=
2026-05-25T23:24:06.282Z [WARN]  server-5653aade-85db-e77f-3640-7a65072be792: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:06.282Z [INFO]  server-5653aade-85db-e77f-3640-7a65072be792: entering candidate state: node="Node at 5653aade-85db-e77f-3640-7a65072be792 [Candidate]" term=2
2026-05-25T23:24:06.282Z [INFO]  server-5653aade-85db-e77f-3640-7a65072be792: election won: term=2 tally=2
2026-05-25T23:24:06.282Z [INFO]  server-5653aade-85db-e77f-3640-7a65072be792: entering leader state: leader="Node at 5653aade-85db-e77f-3640-7a65072be792 [Leader]"
2026-05-25T23:24:06.282Z [INFO]  server-5653aade-85db-e77f-3640-7a65072be792: added peer, starting replication: peer=server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d
2026-05-25T23:24:06.282Z [INFO]  server-5653aade-85db-e77f-3640-7a65072be792: added peer, starting replication: peer=server-937e3459-d53e-b16e-f11a-f2519af612ee
2026-05-25T23:24:06.282Z [INFO]  server-5653aade-85db-e77f-3640-7a65072be792: pipelining replication: peer="{Voter server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d 8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d}"
2026-05-25T23:24:06.282Z [INFO]  server-5653aade-85db-e77f-3640-7a65072be792: pipelining replication: peer="{Voter server-937e3459-d53e-b16e-f11a-f2519af612ee 937e3459-d53e-b16e-f11a-f2519af612ee}"
2026-05-25T23:24:06.388Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:06.282454004 +0000 UTC m=+101.928901458 (1 nodes), 59.203907ms from start of poll, 59.464499ms from cluster start. Timeout at 2026-05-25 23:24:06.387457621 +0000 UTC m=+102.033905075, 105.003617ms after stability
2026-05-25T23:24:06.388Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:06.494Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:06.388143348 +0000 UTC m=+102.034590830 (1 nodes), 1.766µs from start of poll, 165.153871ms from cluster start. Timeout at 2026-05-25 23:24:06.493170981 +0000 UTC m=+102.139618479, 105.027649ms after stability
2026-05-25T23:24:06.494Z [INFO]  server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d: entering candidate state: node="Node at 8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d [Candidate]" term=3
2026-05-25T23:24:06.494Z [ERROR] server-5653aade-85db-e77f-3640-7a65072be792: peer has newer term, stopping replication: peer="{Voter server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d 8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d}"
2026-05-25T23:24:06.494Z [INFO]  server-5653aade-85db-e77f-3640-7a65072be792: aborting pipeline replication: peer="{Voter server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d 8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d}"
2026-05-25T23:24:06.494Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:06.494Z [INFO]  server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d: election won: term=3 tally=2
2026-05-25T23:24:06.494Z [INFO]  server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d: entering leader state: leader="Node at 8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d [Leader]"
2026-05-25T23:24:06.494Z [INFO]  server-5653aade-85db-e77f-3640-7a65072be792: entering follower state: follower="Node at 5653aade-85db-e77f-3640-7a65072be792 [Follower]" leader-address= leader-id=
2026-05-25T23:24:06.494Z [INFO]  server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d: added peer, starting replication: peer=server-5653aade-85db-e77f-3640-7a65072be792
2026-05-25T23:24:06.494Z [INFO]  server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d: added peer, starting replication: peer=server-937e3459-d53e-b16e-f11a-f2519af612ee
2026-05-25T23:24:06.494Z [INFO]  server-5653aade-85db-e77f-3640-7a65072be792: aborting pipeline replication: peer="{Voter server-937e3459-d53e-b16e-f11a-f2519af612ee 937e3459-d53e-b16e-f11a-f2519af612ee}"
2026-05-25T23:24:06.494Z [INFO]  server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d: pipelining replication: peer="{Voter server-937e3459-d53e-b16e-f11a-f2519af612ee 937e3459-d53e-b16e-f11a-f2519af612ee}"
2026-05-25T23:24:06.494Z [INFO]  server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d: pipelining replication: peer="{Voter server-5653aade-85db-e77f-3640-7a65072be792 5653aade-85db-e77f-3640-7a65072be792}"
2026-05-25T23:24:06.600Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:06.49452002 +0000 UTC m=+102.140967482 (1 nodes), 26.286µs from start of poll, 271.530523ms from cluster start. Timeout at 2026-05-25 23:24:06.599545465 +0000 UTC m=+102.245992920, 105.025438ms after stability
2026-05-25T23:24:06.600Z [INFO]  server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d: aborting pipeline replication: peer="{Voter server-937e3459-d53e-b16e-f11a-f2519af612ee 937e3459-d53e-b16e-f11a-f2519af612ee}"
2026-05-25T23:24:06.600Z [INFO]  server-8f68f69f-f4e4-0ffc-7950-f9c8d74f3a0d: aborting pipeline replication: peer="{Voter server-5653aade-85db-e77f-3640-7a65072be792 5653aade-85db-e77f-3640-7a65072be792}"
--- PASS: TestRaft_LeadershipTransfer (0.38s)
=== RUN   TestRaft_LeadershipTransferWithOneNode
2026-05-25T23:24:06.603Z [INFO]  server-f1523eb2-e06a-dffb-87bf-fad0b24a423c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f1523eb2-e06a-dffb-87bf-fad0b24a423c Address:f1523eb2-e06a-dffb-87bf-fad0b24a423c}]"
2026-05-25T23:24:06.603Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:06.603Z [INFO]  server-f1523eb2-e06a-dffb-87bf-fad0b24a423c: entering follower state: follower="Node at f1523eb2-e06a-dffb-87bf-fad0b24a423c [Follower]" leader-address= leader-id=
2026-05-25T23:24:06.676Z [WARN]  server-f1523eb2-e06a-dffb-87bf-fad0b24a423c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:06.676Z [INFO]  server-f1523eb2-e06a-dffb-87bf-fad0b24a423c: entering candidate state: node="Node at f1523eb2-e06a-dffb-87bf-fad0b24a423c [Candidate]" term=2
2026-05-25T23:24:06.676Z [INFO]  server-f1523eb2-e06a-dffb-87bf-fad0b24a423c: election won: term=2 tally=1
2026-05-25T23:24:06.676Z [INFO]  server-f1523eb2-e06a-dffb-87bf-fad0b24a423c: entering leader state: leader="Node at f1523eb2-e06a-dffb-87bf-fad0b24a423c [Leader]"
2026-05-25T23:24:06.782Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:06.676483422 +0000 UTC m=+102.322930875 (1 nodes), 73.323447ms from start of poll, 73.531335ms from cluster start. Timeout at 2026-05-25 23:24:06.781488479 +0000 UTC m=+102.427935934, 105.005059ms after stability
--- PASS: TestRaft_LeadershipTransferWithOneNode (0.18s)
=== RUN   TestRaft_LeadershipTransferWithSevenNodes
2026-05-25T23:24:06.785Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-929b8006-7ea1-1a79-63bb-6a58c4d1e092 Address:929b8006-7ea1-1a79-63bb-6a58c4d1e092} {Suffrage:Voter ID:server-e10e5855-f223-2961-a784-0e843db03bac Address:e10e5855-f223-2961-a784-0e843db03bac} {Suffrage:Voter ID:server-5f0387f6-7f42-da59-2e74-7327f2bec35b Address:5f0387f6-7f42-da59-2e74-7327f2bec35b} {Suffrage:Voter ID:server-0b353397-ccdb-725b-5869-cbd870d35518 Address:0b353397-ccdb-725b-5869-cbd870d35518} {Suffrage:Voter ID:server-7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3 Address:7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3} {Suffrage:Voter ID:server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c Address:2a1d86b6-ef3b-eac4-2464-dbd507a9445c} {Suffrage:Voter ID:server-d8344052-8173-fd29-9633-0d4875958b79 Address:d8344052-8173-fd29-9633-0d4875958b79}]"
2026-05-25T23:24:06.785Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: entering follower state: follower="Node at 929b8006-7ea1-1a79-63bb-6a58c4d1e092 [Follower]" leader-address= leader-id=
2026-05-25T23:24:06.785Z [INFO]  server-e10e5855-f223-2961-a784-0e843db03bac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-929b8006-7ea1-1a79-63bb-6a58c4d1e092 Address:929b8006-7ea1-1a79-63bb-6a58c4d1e092} {Suffrage:Voter ID:server-e10e5855-f223-2961-a784-0e843db03bac Address:e10e5855-f223-2961-a784-0e843db03bac} {Suffrage:Voter ID:server-5f0387f6-7f42-da59-2e74-7327f2bec35b Address:5f0387f6-7f42-da59-2e74-7327f2bec35b} {Suffrage:Voter ID:server-0b353397-ccdb-725b-5869-cbd870d35518 Address:0b353397-ccdb-725b-5869-cbd870d35518} {Suffrage:Voter ID:server-7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3 Address:7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3} {Suffrage:Voter ID:server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c Address:2a1d86b6-ef3b-eac4-2464-dbd507a9445c} {Suffrage:Voter ID:server-d8344052-8173-fd29-9633-0d4875958b79 Address:d8344052-8173-fd29-9633-0d4875958b79}]"
2026-05-25T23:24:06.785Z [INFO]  server-e10e5855-f223-2961-a784-0e843db03bac: entering follower state: follower="Node at e10e5855-f223-2961-a784-0e843db03bac [Follower]" leader-address= leader-id=
2026-05-25T23:24:06.785Z [INFO]  server-5f0387f6-7f42-da59-2e74-7327f2bec35b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-929b8006-7ea1-1a79-63bb-6a58c4d1e092 Address:929b8006-7ea1-1a79-63bb-6a58c4d1e092} {Suffrage:Voter ID:server-e10e5855-f223-2961-a784-0e843db03bac Address:e10e5855-f223-2961-a784-0e843db03bac} {Suffrage:Voter ID:server-5f0387f6-7f42-da59-2e74-7327f2bec35b Address:5f0387f6-7f42-da59-2e74-7327f2bec35b} {Suffrage:Voter ID:server-0b353397-ccdb-725b-5869-cbd870d35518 Address:0b353397-ccdb-725b-5869-cbd870d35518} {Suffrage:Voter ID:server-7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3 Address:7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3} {Suffrage:Voter ID:server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c Address:2a1d86b6-ef3b-eac4-2464-dbd507a9445c} {Suffrage:Voter ID:server-d8344052-8173-fd29-9633-0d4875958b79 Address:d8344052-8173-fd29-9633-0d4875958b79}]"
2026-05-25T23:24:06.785Z [INFO]  server-5f0387f6-7f42-da59-2e74-7327f2bec35b: entering follower state: follower="Node at 5f0387f6-7f42-da59-2e74-7327f2bec35b [Follower]" leader-address= leader-id=
2026-05-25T23:24:06.785Z [INFO]  server-0b353397-ccdb-725b-5869-cbd870d35518: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-929b8006-7ea1-1a79-63bb-6a58c4d1e092 Address:929b8006-7ea1-1a79-63bb-6a58c4d1e092} {Suffrage:Voter ID:server-e10e5855-f223-2961-a784-0e843db03bac Address:e10e5855-f223-2961-a784-0e843db03bac} {Suffrage:Voter ID:server-5f0387f6-7f42-da59-2e74-7327f2bec35b Address:5f0387f6-7f42-da59-2e74-7327f2bec35b} {Suffrage:Voter ID:server-0b353397-ccdb-725b-5869-cbd870d35518 Address:0b353397-ccdb-725b-5869-cbd870d35518} {Suffrage:Voter ID:server-7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3 Address:7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3} {Suffrage:Voter ID:server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c Address:2a1d86b6-ef3b-eac4-2464-dbd507a9445c} {Suffrage:Voter ID:server-d8344052-8173-fd29-9633-0d4875958b79 Address:d8344052-8173-fd29-9633-0d4875958b79}]"
2026-05-25T23:24:06.785Z [INFO]  server-0b353397-ccdb-725b-5869-cbd870d35518: entering follower state: follower="Node at 0b353397-ccdb-725b-5869-cbd870d35518 [Follower]" leader-address= leader-id=
2026-05-25T23:24:06.786Z [INFO]  server-7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-929b8006-7ea1-1a79-63bb-6a58c4d1e092 Address:929b8006-7ea1-1a79-63bb-6a58c4d1e092} {Suffrage:Voter ID:server-e10e5855-f223-2961-a784-0e843db03bac Address:e10e5855-f223-2961-a784-0e843db03bac} {Suffrage:Voter ID:server-5f0387f6-7f42-da59-2e74-7327f2bec35b Address:5f0387f6-7f42-da59-2e74-7327f2bec35b} {Suffrage:Voter ID:server-0b353397-ccdb-725b-5869-cbd870d35518 Address:0b353397-ccdb-725b-5869-cbd870d35518} {Suffrage:Voter ID:server-7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3 Address:7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3} {Suffrage:Voter ID:server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c Address:2a1d86b6-ef3b-eac4-2464-dbd507a9445c} {Suffrage:Voter ID:server-d8344052-8173-fd29-9633-0d4875958b79 Address:d8344052-8173-fd29-9633-0d4875958b79}]"
2026-05-25T23:24:06.786Z [INFO]  server-7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3: entering follower state: follower="Node at 7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3 [Follower]" leader-address= leader-id=
2026-05-25T23:24:06.786Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-929b8006-7ea1-1a79-63bb-6a58c4d1e092 Address:929b8006-7ea1-1a79-63bb-6a58c4d1e092} {Suffrage:Voter ID:server-e10e5855-f223-2961-a784-0e843db03bac Address:e10e5855-f223-2961-a784-0e843db03bac} {Suffrage:Voter ID:server-5f0387f6-7f42-da59-2e74-7327f2bec35b Address:5f0387f6-7f42-da59-2e74-7327f2bec35b} {Suffrage:Voter ID:server-0b353397-ccdb-725b-5869-cbd870d35518 Address:0b353397-ccdb-725b-5869-cbd870d35518} {Suffrage:Voter ID:server-7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3 Address:7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3} {Suffrage:Voter ID:server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c Address:2a1d86b6-ef3b-eac4-2464-dbd507a9445c} {Suffrage:Voter ID:server-d8344052-8173-fd29-9633-0d4875958b79 Address:d8344052-8173-fd29-9633-0d4875958b79}]"
2026-05-25T23:24:06.786Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: entering follower state: follower="Node at 2a1d86b6-ef3b-eac4-2464-dbd507a9445c [Follower]" leader-address= leader-id=
2026-05-25T23:24:06.786Z [INFO]  server-d8344052-8173-fd29-9633-0d4875958b79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-929b8006-7ea1-1a79-63bb-6a58c4d1e092 Address:929b8006-7ea1-1a79-63bb-6a58c4d1e092} {Suffrage:Voter ID:server-e10e5855-f223-2961-a784-0e843db03bac Address:e10e5855-f223-2961-a784-0e843db03bac} {Suffrage:Voter ID:server-5f0387f6-7f42-da59-2e74-7327f2bec35b Address:5f0387f6-7f42-da59-2e74-7327f2bec35b} {Suffrage:Voter ID:server-0b353397-ccdb-725b-5869-cbd870d35518 Address:0b353397-ccdb-725b-5869-cbd870d35518} {Suffrage:Voter ID:server-7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3 Address:7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3} {Suffrage:Voter ID:server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c Address:2a1d86b6-ef3b-eac4-2464-dbd507a9445c} {Suffrage:Voter ID:server-d8344052-8173-fd29-9633-0d4875958b79 Address:d8344052-8173-fd29-9633-0d4875958b79}]"
2026-05-25T23:24:06.786Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:06.786Z [INFO]  server-d8344052-8173-fd29-9633-0d4875958b79: entering follower state: follower="Node at d8344052-8173-fd29-9633-0d4875958b79 [Follower]" leader-address= leader-id=
2026-05-25T23:24:06.837Z [WARN]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:06.837Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: entering candidate state: node="Node at 2a1d86b6-ef3b-eac4-2464-dbd507a9445c [Candidate]" term=2
2026-05-25T23:24:06.838Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: election won: term=2 tally=4
2026-05-25T23:24:06.838Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: entering leader state: leader="Node at 2a1d86b6-ef3b-eac4-2464-dbd507a9445c [Leader]"
2026-05-25T23:24:06.838Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: added peer, starting replication: peer=server-929b8006-7ea1-1a79-63bb-6a58c4d1e092
2026-05-25T23:24:06.838Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: added peer, starting replication: peer=server-e10e5855-f223-2961-a784-0e843db03bac
2026-05-25T23:24:06.838Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: added peer, starting replication: peer=server-5f0387f6-7f42-da59-2e74-7327f2bec35b
2026-05-25T23:24:06.838Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: added peer, starting replication: peer=server-0b353397-ccdb-725b-5869-cbd870d35518
2026-05-25T23:24:06.838Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: added peer, starting replication: peer=server-7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3
2026-05-25T23:24:06.838Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: added peer, starting replication: peer=server-d8344052-8173-fd29-9633-0d4875958b79
2026-05-25T23:24:06.838Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: pipelining replication: peer="{Voter server-929b8006-7ea1-1a79-63bb-6a58c4d1e092 929b8006-7ea1-1a79-63bb-6a58c4d1e092}"
2026-05-25T23:24:06.838Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: pipelining replication: peer="{Voter server-0b353397-ccdb-725b-5869-cbd870d35518 0b353397-ccdb-725b-5869-cbd870d35518}"
2026-05-25T23:24:06.838Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: pipelining replication: peer="{Voter server-5f0387f6-7f42-da59-2e74-7327f2bec35b 5f0387f6-7f42-da59-2e74-7327f2bec35b}"
2026-05-25T23:24:06.838Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: pipelining replication: peer="{Voter server-7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3 7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3}"
2026-05-25T23:24:06.838Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: pipelining replication: peer="{Voter server-e10e5855-f223-2961-a784-0e843db03bac e10e5855-f223-2961-a784-0e843db03bac}"
2026-05-25T23:24:06.838Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: pipelining replication: peer="{Voter server-d8344052-8173-fd29-9633-0d4875958b79 d8344052-8173-fd29-9633-0d4875958b79}"
2026-05-25T23:24:06.943Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:06.838078739 +0000 UTC m=+102.484526205 (1 nodes), 51.692778ms from start of poll, 52.644475ms from cluster start. Timeout at 2026-05-25 23:24:06.943083701 +0000 UTC m=+102.589531152, 105.004947ms after stability
2026-05-25T23:24:06.943Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:07.048Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:06.943194735 +0000 UTC m=+102.589642203 (6 nodes), 3.206µs from start of poll, 157.760473ms from cluster start. Timeout at 2026-05-25 23:24:07.048223713 +0000 UTC m=+102.694671167, 105.028964ms after stability
2026-05-25T23:24:07.048Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:07.154Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:07.048901808 +0000 UTC m=+102.695349259 (1 nodes), 1.724µs from start of poll, 263.467529ms from cluster start. Timeout at 2026-05-25 23:24:07.153914755 +0000 UTC m=+102.800362286, 105.013027ms after stability
2026-05-25T23:24:07.154Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: entering candidate state: node="Node at 929b8006-7ea1-1a79-63bb-6a58c4d1e092 [Candidate]" term=3
2026-05-25T23:24:07.154Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: election won: term=3 tally=4
2026-05-25T23:24:07.154Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: entering leader state: leader="Node at 929b8006-7ea1-1a79-63bb-6a58c4d1e092 [Leader]"
2026-05-25T23:24:07.154Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: entering follower state: follower="Node at 2a1d86b6-ef3b-eac4-2464-dbd507a9445c [Follower]" leader-address= leader-id=
2026-05-25T23:24:07.154Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: added peer, starting replication: peer=server-e10e5855-f223-2961-a784-0e843db03bac
2026-05-25T23:24:07.154Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: aborting pipeline replication: peer="{Voter server-5f0387f6-7f42-da59-2e74-7327f2bec35b 5f0387f6-7f42-da59-2e74-7327f2bec35b}"
2026-05-25T23:24:07.154Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: added peer, starting replication: peer=server-5f0387f6-7f42-da59-2e74-7327f2bec35b
2026-05-25T23:24:07.154Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:07.154Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: added peer, starting replication: peer=server-0b353397-ccdb-725b-5869-cbd870d35518
2026-05-25T23:24:07.154Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: added peer, starting replication: peer=server-7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3
2026-05-25T23:24:07.154Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: added peer, starting replication: peer=server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c
2026-05-25T23:24:07.154Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: added peer, starting replication: peer=server-d8344052-8173-fd29-9633-0d4875958b79
2026-05-25T23:24:07.154Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: aborting pipeline replication: peer="{Voter server-7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3 7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3}"
2026-05-25T23:24:07.154Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: aborting pipeline replication: peer="{Voter server-d8344052-8173-fd29-9633-0d4875958b79 d8344052-8173-fd29-9633-0d4875958b79}"
2026-05-25T23:24:07.154Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: aborting pipeline replication: peer="{Voter server-929b8006-7ea1-1a79-63bb-6a58c4d1e092 929b8006-7ea1-1a79-63bb-6a58c4d1e092}"
2026-05-25T23:24:07.154Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: aborting pipeline replication: peer="{Voter server-0b353397-ccdb-725b-5869-cbd870d35518 0b353397-ccdb-725b-5869-cbd870d35518}"
2026-05-25T23:24:07.154Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: pipelining replication: peer="{Voter server-0b353397-ccdb-725b-5869-cbd870d35518 0b353397-ccdb-725b-5869-cbd870d35518}"
2026-05-25T23:24:07.154Z [INFO]  server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c: aborting pipeline replication: peer="{Voter server-e10e5855-f223-2961-a784-0e843db03bac e10e5855-f223-2961-a784-0e843db03bac}"
2026-05-25T23:24:07.154Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: pipelining replication: peer="{Voter server-5f0387f6-7f42-da59-2e74-7327f2bec35b 5f0387f6-7f42-da59-2e74-7327f2bec35b}"
2026-05-25T23:24:07.154Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: pipelining replication: peer="{Voter server-7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3 7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3}"
2026-05-25T23:24:07.154Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: pipelining replication: peer="{Voter server-e10e5855-f223-2961-a784-0e843db03bac e10e5855-f223-2961-a784-0e843db03bac}"
2026-05-25T23:24:07.154Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: pipelining replication: peer="{Voter server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c 2a1d86b6-ef3b-eac4-2464-dbd507a9445c}"
2026-05-25T23:24:07.154Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: pipelining replication: peer="{Voter server-d8344052-8173-fd29-9633-0d4875958b79 d8344052-8173-fd29-9633-0d4875958b79}"
2026-05-25T23:24:07.260Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:07.154719857 +0000 UTC m=+102.801167328 (1 nodes), 53.042µs from start of poll, 369.285598ms from cluster start. Timeout at 2026-05-25 23:24:07.259725317 +0000 UTC m=+102.906172772, 105.005444ms after stability
2026-05-25T23:24:07.260Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: aborting pipeline replication: peer="{Voter server-0b353397-ccdb-725b-5869-cbd870d35518 0b353397-ccdb-725b-5869-cbd870d35518}"
2026-05-25T23:24:07.260Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: aborting pipeline replication: peer="{Voter server-7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3 7e3dc1f2-ef71-e0aa-9093-f0c734b3b0b3}"
2026-05-25T23:24:07.260Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: aborting pipeline replication: peer="{Voter server-e10e5855-f223-2961-a784-0e843db03bac e10e5855-f223-2961-a784-0e843db03bac}"
2026-05-25T23:24:07.260Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: aborting pipeline replication: peer="{Voter server-2a1d86b6-ef3b-eac4-2464-dbd507a9445c 2a1d86b6-ef3b-eac4-2464-dbd507a9445c}"
2026-05-25T23:24:07.260Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: aborting pipeline replication: peer="{Voter server-d8344052-8173-fd29-9633-0d4875958b79 d8344052-8173-fd29-9633-0d4875958b79}"
2026-05-25T23:24:07.260Z [INFO]  server-929b8006-7ea1-1a79-63bb-6a58c4d1e092: aborting pipeline replication: peer="{Voter server-5f0387f6-7f42-da59-2e74-7327f2bec35b 5f0387f6-7f42-da59-2e74-7327f2bec35b}"
--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.48s)
=== RUN   TestRaft_LeadershipTransferToInvalidID
2026-05-25T23:24:07.264Z [INFO]  server-5faf5fe8-9dce-223a-e254-cc7f4a4e6cee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5faf5fe8-9dce-223a-e254-cc7f4a4e6cee Address:5faf5fe8-9dce-223a-e254-cc7f4a4e6cee} {Suffrage:Voter ID:server-418dca4a-957e-0d14-474f-2994e64979ba Address:418dca4a-957e-0d14-474f-2994e64979ba} {Suffrage:Voter ID:server-395820af-2b20-af86-098c-728a36f46dbb Address:395820af-2b20-af86-098c-728a36f46dbb}]"
2026-05-25T23:24:07.264Z [INFO]  server-418dca4a-957e-0d14-474f-2994e64979ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5faf5fe8-9dce-223a-e254-cc7f4a4e6cee Address:5faf5fe8-9dce-223a-e254-cc7f4a4e6cee} {Suffrage:Voter ID:server-418dca4a-957e-0d14-474f-2994e64979ba Address:418dca4a-957e-0d14-474f-2994e64979ba} {Suffrage:Voter ID:server-395820af-2b20-af86-098c-728a36f46dbb Address:395820af-2b20-af86-098c-728a36f46dbb}]"
2026-05-25T23:24:07.264Z [INFO]  server-5faf5fe8-9dce-223a-e254-cc7f4a4e6cee: entering follower state: follower="Node at 5faf5fe8-9dce-223a-e254-cc7f4a4e6cee [Follower]" leader-address= leader-id=
2026-05-25T23:24:07.264Z [INFO]  server-418dca4a-957e-0d14-474f-2994e64979ba: entering follower state: follower="Node at 418dca4a-957e-0d14-474f-2994e64979ba [Follower]" leader-address= leader-id=
2026-05-25T23:24:07.265Z [INFO]  server-395820af-2b20-af86-098c-728a36f46dbb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5faf5fe8-9dce-223a-e254-cc7f4a4e6cee Address:5faf5fe8-9dce-223a-e254-cc7f4a4e6cee} {Suffrage:Voter ID:server-418dca4a-957e-0d14-474f-2994e64979ba Address:418dca4a-957e-0d14-474f-2994e64979ba} {Suffrage:Voter ID:server-395820af-2b20-af86-098c-728a36f46dbb Address:395820af-2b20-af86-098c-728a36f46dbb}]"
2026-05-25T23:24:07.265Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:07.265Z [INFO]  server-395820af-2b20-af86-098c-728a36f46dbb: entering follower state: follower="Node at 395820af-2b20-af86-098c-728a36f46dbb [Follower]" leader-address= leader-id=
2026-05-25T23:24:07.332Z [WARN]  server-5faf5fe8-9dce-223a-e254-cc7f4a4e6cee: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:07.332Z [INFO]  server-5faf5fe8-9dce-223a-e254-cc7f4a4e6cee: entering candidate state: node="Node at 5faf5fe8-9dce-223a-e254-cc7f4a4e6cee [Candidate]" term=2
2026-05-25T23:24:07.332Z [INFO]  server-5faf5fe8-9dce-223a-e254-cc7f4a4e6cee: election won: term=2 tally=2
2026-05-25T23:24:07.332Z [INFO]  server-5faf5fe8-9dce-223a-e254-cc7f4a4e6cee: entering leader state: leader="Node at 5faf5fe8-9dce-223a-e254-cc7f4a4e6cee [Leader]"
2026-05-25T23:24:07.332Z [INFO]  server-5faf5fe8-9dce-223a-e254-cc7f4a4e6cee: added peer, starting replication: peer=server-418dca4a-957e-0d14-474f-2994e64979ba
2026-05-25T23:24:07.332Z [INFO]  server-5faf5fe8-9dce-223a-e254-cc7f4a4e6cee: added peer, starting replication: peer=server-395820af-2b20-af86-098c-728a36f46dbb
2026-05-25T23:24:07.333Z [INFO]  server-5faf5fe8-9dce-223a-e254-cc7f4a4e6cee: pipelining replication: peer="{Voter server-395820af-2b20-af86-098c-728a36f46dbb 395820af-2b20-af86-098c-728a36f46dbb}"
2026-05-25T23:24:07.333Z [INFO]  server-5faf5fe8-9dce-223a-e254-cc7f4a4e6cee: pipelining replication: peer="{Voter server-418dca4a-957e-0d14-474f-2994e64979ba 418dca4a-957e-0d14-474f-2994e64979ba}"
2026-05-25T23:24:07.438Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:07.333034072 +0000 UTC m=+102.979481538 (1 nodes), 67.997066ms from start of poll, 68.345207ms from cluster start. Timeout at 2026-05-25 23:24:07.438035811 +0000 UTC m=+103.084483260, 105.001722ms after stability
2026-05-25T23:24:07.438Z [INFO]  server-5faf5fe8-9dce-223a-e254-cc7f4a4e6cee: aborting pipeline replication: peer="{Voter server-418dca4a-957e-0d14-474f-2994e64979ba 418dca4a-957e-0d14-474f-2994e64979ba}"
2026-05-25T23:24:07.438Z [INFO]  server-5faf5fe8-9dce-223a-e254-cc7f4a4e6cee: aborting pipeline replication: peer="{Voter server-395820af-2b20-af86-098c-728a36f46dbb 395820af-2b20-af86-098c-728a36f46dbb}"
--- PASS: TestRaft_LeadershipTransferToInvalidID (0.17s)
=== RUN   TestRaft_LeadershipTransferToInvalidAddress
2026-05-25T23:24:07.439Z [INFO]  server-f489948b-b3a5-c90a-86c8-4c5847cd13aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f489948b-b3a5-c90a-86c8-4c5847cd13aa Address:f489948b-b3a5-c90a-86c8-4c5847cd13aa} {Suffrage:Voter ID:server-525230a7-1ea0-ea5f-e6e6-744317267f65 Address:525230a7-1ea0-ea5f-e6e6-744317267f65} {Suffrage:Voter ID:server-6325351b-bbc8-88f0-9047-5dc09f781dfb Address:6325351b-bbc8-88f0-9047-5dc09f781dfb}]"
2026-05-25T23:24:07.439Z [INFO]  server-525230a7-1ea0-ea5f-e6e6-744317267f65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f489948b-b3a5-c90a-86c8-4c5847cd13aa Address:f489948b-b3a5-c90a-86c8-4c5847cd13aa} {Suffrage:Voter ID:server-525230a7-1ea0-ea5f-e6e6-744317267f65 Address:525230a7-1ea0-ea5f-e6e6-744317267f65} {Suffrage:Voter ID:server-6325351b-bbc8-88f0-9047-5dc09f781dfb Address:6325351b-bbc8-88f0-9047-5dc09f781dfb}]"
2026-05-25T23:24:07.439Z [INFO]  server-6325351b-bbc8-88f0-9047-5dc09f781dfb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f489948b-b3a5-c90a-86c8-4c5847cd13aa Address:f489948b-b3a5-c90a-86c8-4c5847cd13aa} {Suffrage:Voter ID:server-525230a7-1ea0-ea5f-e6e6-744317267f65 Address:525230a7-1ea0-ea5f-e6e6-744317267f65} {Suffrage:Voter ID:server-6325351b-bbc8-88f0-9047-5dc09f781dfb Address:6325351b-bbc8-88f0-9047-5dc09f781dfb}]"
2026-05-25T23:24:07.439Z [INFO]  server-525230a7-1ea0-ea5f-e6e6-744317267f65: entering follower state: follower="Node at 525230a7-1ea0-ea5f-e6e6-744317267f65 [Follower]" leader-address= leader-id=
2026-05-25T23:24:07.439Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:07.439Z [INFO]  server-f489948b-b3a5-c90a-86c8-4c5847cd13aa: entering follower state: follower="Node at f489948b-b3a5-c90a-86c8-4c5847cd13aa [Follower]" leader-address= leader-id=
2026-05-25T23:24:07.439Z [INFO]  server-6325351b-bbc8-88f0-9047-5dc09f781dfb: entering follower state: follower="Node at 6325351b-bbc8-88f0-9047-5dc09f781dfb [Follower]" leader-address= leader-id=
2026-05-25T23:24:07.502Z [WARN]  server-f489948b-b3a5-c90a-86c8-4c5847cd13aa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:07.502Z [INFO]  server-f489948b-b3a5-c90a-86c8-4c5847cd13aa: entering candidate state: node="Node at f489948b-b3a5-c90a-86c8-4c5847cd13aa [Candidate]" term=2
2026-05-25T23:24:07.502Z [INFO]  server-f489948b-b3a5-c90a-86c8-4c5847cd13aa: election won: term=2 tally=2
2026-05-25T23:24:07.502Z [INFO]  server-f489948b-b3a5-c90a-86c8-4c5847cd13aa: entering leader state: leader="Node at f489948b-b3a5-c90a-86c8-4c5847cd13aa [Leader]"
2026-05-25T23:24:07.502Z [INFO]  server-f489948b-b3a5-c90a-86c8-4c5847cd13aa: added peer, starting replication: peer=server-525230a7-1ea0-ea5f-e6e6-744317267f65
2026-05-25T23:24:07.502Z [INFO]  server-f489948b-b3a5-c90a-86c8-4c5847cd13aa: added peer, starting replication: peer=server-6325351b-bbc8-88f0-9047-5dc09f781dfb
2026-05-25T23:24:07.502Z [INFO]  server-f489948b-b3a5-c90a-86c8-4c5847cd13aa: pipelining replication: peer="{Voter server-6325351b-bbc8-88f0-9047-5dc09f781dfb 6325351b-bbc8-88f0-9047-5dc09f781dfb}"
2026-05-25T23:24:07.502Z [INFO]  server-f489948b-b3a5-c90a-86c8-4c5847cd13aa: pipelining replication: peer="{Voter server-525230a7-1ea0-ea5f-e6e6-744317267f65 525230a7-1ea0-ea5f-e6e6-744317267f65}"
2026-05-25T23:24:07.608Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:07.50279137 +0000 UTC m=+103.149238826 (2 nodes), 62.807107ms from start of poll, 63.151424ms from cluster start. Timeout at 2026-05-25 23:24:07.607795824 +0000 UTC m=+103.254243279, 105.004453ms after stability
2026-05-25T23:24:07.608Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:07.714Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:07.608410316 +0000 UTC m=+103.254857770 (1 nodes), 1.025µs from start of poll, 168.770368ms from cluster start. Timeout at 2026-05-25 23:24:07.713414615 +0000 UTC m=+103.359862071, 105.004301ms after stability
2026-05-25T23:24:07.714Z [INFO]  server-f489948b-b3a5-c90a-86c8-4c5847cd13aa: aborting pipeline replication: peer="{Voter server-6325351b-bbc8-88f0-9047-5dc09f781dfb 6325351b-bbc8-88f0-9047-5dc09f781dfb}"
2026-05-25T23:24:07.714Z [INFO]  server-f489948b-b3a5-c90a-86c8-4c5847cd13aa: aborting pipeline replication: peer="{Voter server-525230a7-1ea0-ea5f-e6e6-744317267f65 525230a7-1ea0-ea5f-e6e6-744317267f65}"
--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.28s)
=== RUN   TestRaft_LeadershipTransferToBehindServer
2026-05-25T23:24:07.715Z [INFO]  server-cef5fd62-e25c-c220-47a5-54c4d3a91f39: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cef5fd62-e25c-c220-47a5-54c4d3a91f39 Address:cef5fd62-e25c-c220-47a5-54c4d3a91f39} {Suffrage:Voter ID:server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462 Address:61c9f6d9-3787-f5b1-7d7b-1b3c373cb462} {Suffrage:Voter ID:server-c1405c6d-d97a-1704-b64d-c8f8f400153e Address:c1405c6d-d97a-1704-b64d-c8f8f400153e}]"
2026-05-25T23:24:07.715Z [INFO]  server-cef5fd62-e25c-c220-47a5-54c4d3a91f39: entering follower state: follower="Node at cef5fd62-e25c-c220-47a5-54c4d3a91f39 [Follower]" leader-address= leader-id=
2026-05-25T23:24:07.715Z [INFO]  server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cef5fd62-e25c-c220-47a5-54c4d3a91f39 Address:cef5fd62-e25c-c220-47a5-54c4d3a91f39} {Suffrage:Voter ID:server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462 Address:61c9f6d9-3787-f5b1-7d7b-1b3c373cb462} {Suffrage:Voter ID:server-c1405c6d-d97a-1704-b64d-c8f8f400153e Address:c1405c6d-d97a-1704-b64d-c8f8f400153e}]"
2026-05-25T23:24:07.715Z [INFO]  server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462: entering follower state: follower="Node at 61c9f6d9-3787-f5b1-7d7b-1b3c373cb462 [Follower]" leader-address= leader-id=
2026-05-25T23:24:07.715Z [INFO]  server-c1405c6d-d97a-1704-b64d-c8f8f400153e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cef5fd62-e25c-c220-47a5-54c4d3a91f39 Address:cef5fd62-e25c-c220-47a5-54c4d3a91f39} {Suffrage:Voter ID:server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462 Address:61c9f6d9-3787-f5b1-7d7b-1b3c373cb462} {Suffrage:Voter ID:server-c1405c6d-d97a-1704-b64d-c8f8f400153e Address:c1405c6d-d97a-1704-b64d-c8f8f400153e}]"
2026-05-25T23:24:07.715Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:07.715Z [INFO]  server-c1405c6d-d97a-1704-b64d-c8f8f400153e: entering follower state: follower="Node at c1405c6d-d97a-1704-b64d-c8f8f400153e [Follower]" leader-address= leader-id=
2026-05-25T23:24:07.784Z [WARN]  server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:07.784Z [INFO]  server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462: entering candidate state: node="Node at 61c9f6d9-3787-f5b1-7d7b-1b3c373cb462 [Candidate]" term=2
2026-05-25T23:24:07.784Z [INFO]  server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462: election won: term=2 tally=2
2026-05-25T23:24:07.784Z [INFO]  server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462: entering leader state: leader="Node at 61c9f6d9-3787-f5b1-7d7b-1b3c373cb462 [Leader]"
2026-05-25T23:24:07.784Z [INFO]  server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462: added peer, starting replication: peer=server-cef5fd62-e25c-c220-47a5-54c4d3a91f39
2026-05-25T23:24:07.784Z [INFO]  server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462: added peer, starting replication: peer=server-c1405c6d-d97a-1704-b64d-c8f8f400153e
2026-05-25T23:24:07.784Z [INFO]  server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462: pipelining replication: peer="{Voter server-cef5fd62-e25c-c220-47a5-54c4d3a91f39 cef5fd62-e25c-c220-47a5-54c4d3a91f39}"
2026-05-25T23:24:07.784Z [INFO]  server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462: pipelining replication: peer="{Voter server-c1405c6d-d97a-1704-b64d-c8f8f400153e c1405c6d-d97a-1704-b64d-c8f8f400153e}"
2026-05-25T23:24:07.889Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:07.784269622 +0000 UTC m=+103.430717144 (1 nodes), 68.642172ms from start of poll, 69.011192ms from cluster start. Timeout at 2026-05-25 23:24:07.889274806 +0000 UTC m=+103.535722261, 105.005117ms after stability
2026-05-25T23:24:07.889Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:07.994Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:07.889348194 +0000 UTC m=+103.535795649 (2 nodes), 1.584µs from start of poll, 174.089697ms from cluster start. Timeout at 2026-05-25 23:24:07.994384568 +0000 UTC m=+103.640832023, 105.036374ms after stability
2026-05-25T23:24:08.023Z [INFO]  server-cef5fd62-e25c-c220-47a5-54c4d3a91f39: entering candidate state: node="Node at cef5fd62-e25c-c220-47a5-54c4d3a91f39 [Candidate]" term=3
2026-05-25T23:24:08.023Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:08.024Z [INFO]  server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462: entering follower state: follower="Node at 61c9f6d9-3787-f5b1-7d7b-1b3c373cb462 [Follower]" leader-address= leader-id=
2026-05-25T23:24:08.024Z [INFO]  server-cef5fd62-e25c-c220-47a5-54c4d3a91f39: election won: term=3 tally=2
2026-05-25T23:24:08.024Z [INFO]  server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462: aborting pipeline replication: peer="{Voter server-c1405c6d-d97a-1704-b64d-c8f8f400153e c1405c6d-d97a-1704-b64d-c8f8f400153e}"
2026-05-25T23:24:08.024Z [INFO]  server-cef5fd62-e25c-c220-47a5-54c4d3a91f39: entering leader state: leader="Node at cef5fd62-e25c-c220-47a5-54c4d3a91f39 [Leader]"
2026-05-25T23:24:08.024Z [INFO]  server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462: aborting pipeline replication: peer="{Voter server-cef5fd62-e25c-c220-47a5-54c4d3a91f39 cef5fd62-e25c-c220-47a5-54c4d3a91f39}"
2026-05-25T23:24:08.024Z [INFO]  server-cef5fd62-e25c-c220-47a5-54c4d3a91f39: added peer, starting replication: peer=server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462
2026-05-25T23:24:08.024Z [INFO]  server-cef5fd62-e25c-c220-47a5-54c4d3a91f39: added peer, starting replication: peer=server-c1405c6d-d97a-1704-b64d-c8f8f400153e
2026-05-25T23:24:08.024Z [INFO]  server-cef5fd62-e25c-c220-47a5-54c4d3a91f39: pipelining replication: peer="{Voter server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462 61c9f6d9-3787-f5b1-7d7b-1b3c373cb462}"
2026-05-25T23:24:08.024Z [INFO]  server-cef5fd62-e25c-c220-47a5-54c4d3a91f39: pipelining replication: peer="{Voter server-c1405c6d-d97a-1704-b64d-c8f8f400153e c1405c6d-d97a-1704-b64d-c8f8f400153e}"
2026-05-25T23:24:08.129Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:08.024067339 +0000 UTC m=+103.670514806 (1 nodes), 85.756µs from start of poll, 308.808854ms from cluster start. Timeout at 2026-05-25 23:24:08.129089654 +0000 UTC m=+103.775537126, 105.02232ms after stability
2026-05-25T23:24:08.129Z [INFO]  server-cef5fd62-e25c-c220-47a5-54c4d3a91f39: aborting pipeline replication: peer="{Voter server-c1405c6d-d97a-1704-b64d-c8f8f400153e c1405c6d-d97a-1704-b64d-c8f8f400153e}"
2026-05-25T23:24:08.129Z [INFO]  server-cef5fd62-e25c-c220-47a5-54c4d3a91f39: aborting pipeline replication: peer="{Voter server-61c9f6d9-3787-f5b1-7d7b-1b3c373cb462 61c9f6d9-3787-f5b1-7d7b-1b3c373cb462}"
--- PASS: TestRaft_LeadershipTransferToBehindServer (0.42s)
=== RUN   TestRaft_LeadershipTransferToItself
2026-05-25T23:24:08.130Z [INFO]  server-549b22f7-a23a-2d51-1b98-28431b16de56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-549b22f7-a23a-2d51-1b98-28431b16de56 Address:549b22f7-a23a-2d51-1b98-28431b16de56} {Suffrage:Voter ID:server-68292082-1c37-5f83-46c5-03daca8873d9 Address:68292082-1c37-5f83-46c5-03daca8873d9} {Suffrage:Voter ID:server-51a8a380-d21a-5b54-e9df-2dcac57182b4 Address:51a8a380-d21a-5b54-e9df-2dcac57182b4}]"
2026-05-25T23:24:08.130Z [INFO]  server-549b22f7-a23a-2d51-1b98-28431b16de56: entering follower state: follower="Node at 549b22f7-a23a-2d51-1b98-28431b16de56 [Follower]" leader-address= leader-id=
2026-05-25T23:24:08.130Z [INFO]  server-68292082-1c37-5f83-46c5-03daca8873d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-549b22f7-a23a-2d51-1b98-28431b16de56 Address:549b22f7-a23a-2d51-1b98-28431b16de56} {Suffrage:Voter ID:server-68292082-1c37-5f83-46c5-03daca8873d9 Address:68292082-1c37-5f83-46c5-03daca8873d9} {Suffrage:Voter ID:server-51a8a380-d21a-5b54-e9df-2dcac57182b4 Address:51a8a380-d21a-5b54-e9df-2dcac57182b4}]"
2026-05-25T23:24:08.130Z [INFO]  server-68292082-1c37-5f83-46c5-03daca8873d9: entering follower state: follower="Node at 68292082-1c37-5f83-46c5-03daca8873d9 [Follower]" leader-address= leader-id=
2026-05-25T23:24:08.130Z [INFO]  server-51a8a380-d21a-5b54-e9df-2dcac57182b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-549b22f7-a23a-2d51-1b98-28431b16de56 Address:549b22f7-a23a-2d51-1b98-28431b16de56} {Suffrage:Voter ID:server-68292082-1c37-5f83-46c5-03daca8873d9 Address:68292082-1c37-5f83-46c5-03daca8873d9} {Suffrage:Voter ID:server-51a8a380-d21a-5b54-e9df-2dcac57182b4 Address:51a8a380-d21a-5b54-e9df-2dcac57182b4}]"
2026-05-25T23:24:08.131Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:08.131Z [INFO]  server-51a8a380-d21a-5b54-e9df-2dcac57182b4: entering follower state: follower="Node at 51a8a380-d21a-5b54-e9df-2dcac57182b4 [Follower]" leader-address= leader-id=
2026-05-25T23:24:08.199Z [WARN]  server-68292082-1c37-5f83-46c5-03daca8873d9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:08.199Z [INFO]  server-68292082-1c37-5f83-46c5-03daca8873d9: entering candidate state: node="Node at 68292082-1c37-5f83-46c5-03daca8873d9 [Candidate]" term=2
2026-05-25T23:24:08.199Z [INFO]  server-68292082-1c37-5f83-46c5-03daca8873d9: election won: term=2 tally=2
2026-05-25T23:24:08.199Z [INFO]  server-68292082-1c37-5f83-46c5-03daca8873d9: entering leader state: leader="Node at 68292082-1c37-5f83-46c5-03daca8873d9 [Leader]"
2026-05-25T23:24:08.199Z [INFO]  server-68292082-1c37-5f83-46c5-03daca8873d9: added peer, starting replication: peer=server-549b22f7-a23a-2d51-1b98-28431b16de56
2026-05-25T23:24:08.199Z [INFO]  server-68292082-1c37-5f83-46c5-03daca8873d9: added peer, starting replication: peer=server-51a8a380-d21a-5b54-e9df-2dcac57182b4
2026-05-25T23:24:08.199Z [INFO]  server-68292082-1c37-5f83-46c5-03daca8873d9: pipelining replication: peer="{Voter server-51a8a380-d21a-5b54-e9df-2dcac57182b4 51a8a380-d21a-5b54-e9df-2dcac57182b4}"
2026-05-25T23:24:08.199Z [INFO]  server-68292082-1c37-5f83-46c5-03daca8873d9: pipelining replication: peer="{Voter server-549b22f7-a23a-2d51-1b98-28431b16de56 549b22f7-a23a-2d51-1b98-28431b16de56}"
2026-05-25T23:24:08.306Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:08.199909961 +0000 UTC m=+103.846357429 (1 nodes), 68.899835ms from start of poll, 69.335357ms from cluster start. Timeout at 2026-05-25 23:24:08.304911414 +0000 UTC m=+103.951358869, 105.00144ms after stability
2026-05-25T23:24:08.306Z [INFO]  server-68292082-1c37-5f83-46c5-03daca8873d9: cannot transfer leadership to itself
2026-05-25T23:24:08.306Z [INFO]  server-68292082-1c37-5f83-46c5-03daca8873d9: aborting pipeline replication: peer="{Voter server-549b22f7-a23a-2d51-1b98-28431b16de56 549b22f7-a23a-2d51-1b98-28431b16de56}"
2026-05-25T23:24:08.306Z [INFO]  server-68292082-1c37-5f83-46c5-03daca8873d9: aborting pipeline replication: peer="{Voter server-51a8a380-d21a-5b54-e9df-2dcac57182b4 51a8a380-d21a-5b54-e9df-2dcac57182b4}"
2026-05-25T23:24:08.806Z [ERROR] server-68292082-1c37-5f83-46c5-03daca8873d9: failed to heartbeat to: peer=51a8a380-d21a-5b54-e9df-2dcac57182b4 backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeadershipTransferToItself (0.68s)
=== RUN   TestRaft_LeadershipTransferLeaderRejectsClientRequests
2026-05-25T23:24:08.807Z [INFO]  server-d49c9b5a-5ef6-f12e-d0b3-c698c8766a34: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d49c9b5a-5ef6-f12e-d0b3-c698c8766a34 Address:d49c9b5a-5ef6-f12e-d0b3-c698c8766a34} {Suffrage:Voter ID:server-79b8655a-01da-b0e8-6095-065bce48e87d Address:79b8655a-01da-b0e8-6095-065bce48e87d} {Suffrage:Voter ID:server-8f868d49-3d7b-eaca-aa34-f3212f1187a0 Address:8f868d49-3d7b-eaca-aa34-f3212f1187a0}]"
2026-05-25T23:24:08.807Z [INFO]  server-d49c9b5a-5ef6-f12e-d0b3-c698c8766a34: entering follower state: follower="Node at d49c9b5a-5ef6-f12e-d0b3-c698c8766a34 [Follower]" leader-address= leader-id=
2026-05-25T23:24:08.807Z [INFO]  server-79b8655a-01da-b0e8-6095-065bce48e87d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d49c9b5a-5ef6-f12e-d0b3-c698c8766a34 Address:d49c9b5a-5ef6-f12e-d0b3-c698c8766a34} {Suffrage:Voter ID:server-79b8655a-01da-b0e8-6095-065bce48e87d Address:79b8655a-01da-b0e8-6095-065bce48e87d} {Suffrage:Voter ID:server-8f868d49-3d7b-eaca-aa34-f3212f1187a0 Address:8f868d49-3d7b-eaca-aa34-f3212f1187a0}]"
2026-05-25T23:24:08.808Z [INFO]  server-79b8655a-01da-b0e8-6095-065bce48e87d: entering follower state: follower="Node at 79b8655a-01da-b0e8-6095-065bce48e87d [Follower]" leader-address= leader-id=
2026-05-25T23:24:08.808Z [INFO]  server-8f868d49-3d7b-eaca-aa34-f3212f1187a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d49c9b5a-5ef6-f12e-d0b3-c698c8766a34 Address:d49c9b5a-5ef6-f12e-d0b3-c698c8766a34} {Suffrage:Voter ID:server-79b8655a-01da-b0e8-6095-065bce48e87d Address:79b8655a-01da-b0e8-6095-065bce48e87d} {Suffrage:Voter ID:server-8f868d49-3d7b-eaca-aa34-f3212f1187a0 Address:8f868d49-3d7b-eaca-aa34-f3212f1187a0}]"
2026-05-25T23:24:08.808Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:08.808Z [INFO]  server-8f868d49-3d7b-eaca-aa34-f3212f1187a0: entering follower state: follower="Node at 8f868d49-3d7b-eaca-aa34-f3212f1187a0 [Follower]" leader-address= leader-id=
2026-05-25T23:24:08.858Z [WARN]  server-d49c9b5a-5ef6-f12e-d0b3-c698c8766a34: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:08.858Z [INFO]  server-d49c9b5a-5ef6-f12e-d0b3-c698c8766a34: entering candidate state: node="Node at d49c9b5a-5ef6-f12e-d0b3-c698c8766a34 [Candidate]" term=2
2026-05-25T23:24:08.858Z [INFO]  server-d49c9b5a-5ef6-f12e-d0b3-c698c8766a34: election won: term=2 tally=2
2026-05-25T23:24:08.858Z [INFO]  server-d49c9b5a-5ef6-f12e-d0b3-c698c8766a34: entering leader state: leader="Node at d49c9b5a-5ef6-f12e-d0b3-c698c8766a34 [Leader]"
2026-05-25T23:24:08.858Z [INFO]  server-d49c9b5a-5ef6-f12e-d0b3-c698c8766a34: added peer, starting replication: peer=server-79b8655a-01da-b0e8-6095-065bce48e87d
2026-05-25T23:24:08.858Z [INFO]  server-d49c9b5a-5ef6-f12e-d0b3-c698c8766a34: added peer, starting replication: peer=server-8f868d49-3d7b-eaca-aa34-f3212f1187a0
2026-05-25T23:24:08.858Z [INFO]  server-d49c9b5a-5ef6-f12e-d0b3-c698c8766a34: pipelining replication: peer="{Voter server-8f868d49-3d7b-eaca-aa34-f3212f1187a0 8f868d49-3d7b-eaca-aa34-f3212f1187a0}"
2026-05-25T23:24:08.858Z [INFO]  server-d49c9b5a-5ef6-f12e-d0b3-c698c8766a34: pipelining replication: peer="{Voter server-79b8655a-01da-b0e8-6095-065bce48e87d 79b8655a-01da-b0e8-6095-065bce48e87d}"
2026-05-25T23:24:08.964Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:08.858706754 +0000 UTC m=+104.505154215 (1 nodes), 50.540414ms from start of poll, 50.890953ms from cluster start. Timeout at 2026-05-25 23:24:08.963711903 +0000 UTC m=+104.610159357, 105.005142ms after stability
    raft_test.go:2457: waiting on future 0
    raft_test.go:2457: waiting on future 1
    raft_test.go:2457: waiting on future 2
    raft_test.go:2457: waiting on future 3
    raft_test.go:2457: waiting on future 4
    raft_test.go:2457: waiting on future 5
    raft_test.go:2457: waiting on future 6
2026-05-25T23:24:08.966Z [INFO]  server-d49c9b5a-5ef6-f12e-d0b3-c698c8766a34: aborting pipeline replication: peer="{Voter server-8f868d49-3d7b-eaca-aa34-f3212f1187a0 8f868d49-3d7b-eaca-aa34-f3212f1187a0}"
2026-05-25T23:24:08.967Z [INFO]  server-d49c9b5a-5ef6-f12e-d0b3-c698c8766a34: aborting pipeline replication: peer="{Voter server-79b8655a-01da-b0e8-6095-065bce48e87d 79b8655a-01da-b0e8-6095-065bce48e87d}"
--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.16s)
=== RUN   TestRaft_LeadershipTransferLeaderReplicationTimeout
2026-05-25T23:24:08.968Z [INFO]  server-a760903a-a252-ba30-010a-e0f8cafab701: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a760903a-a252-ba30-010a-e0f8cafab701 Address:a760903a-a252-ba30-010a-e0f8cafab701} {Suffrage:Voter ID:server-7ac26b99-2129-56e1-0f92-987e62d39d1b Address:7ac26b99-2129-56e1-0f92-987e62d39d1b} {Suffrage:Voter ID:server-87393c21-b0ba-bb09-e374-718a7fd49670 Address:87393c21-b0ba-bb09-e374-718a7fd49670}]"
2026-05-25T23:24:08.968Z [INFO]  server-7ac26b99-2129-56e1-0f92-987e62d39d1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a760903a-a252-ba30-010a-e0f8cafab701 Address:a760903a-a252-ba30-010a-e0f8cafab701} {Suffrage:Voter ID:server-7ac26b99-2129-56e1-0f92-987e62d39d1b Address:7ac26b99-2129-56e1-0f92-987e62d39d1b} {Suffrage:Voter ID:server-87393c21-b0ba-bb09-e374-718a7fd49670 Address:87393c21-b0ba-bb09-e374-718a7fd49670}]"
2026-05-25T23:24:08.968Z [INFO]  server-a760903a-a252-ba30-010a-e0f8cafab701: entering follower state: follower="Node at a760903a-a252-ba30-010a-e0f8cafab701 [Follower]" leader-address= leader-id=
2026-05-25T23:24:08.968Z [INFO]  server-7ac26b99-2129-56e1-0f92-987e62d39d1b: entering follower state: follower="Node at 7ac26b99-2129-56e1-0f92-987e62d39d1b [Follower]" leader-address= leader-id=
2026-05-25T23:24:08.968Z [INFO]  server-87393c21-b0ba-bb09-e374-718a7fd49670: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a760903a-a252-ba30-010a-e0f8cafab701 Address:a760903a-a252-ba30-010a-e0f8cafab701} {Suffrage:Voter ID:server-7ac26b99-2129-56e1-0f92-987e62d39d1b Address:7ac26b99-2129-56e1-0f92-987e62d39d1b} {Suffrage:Voter ID:server-87393c21-b0ba-bb09-e374-718a7fd49670 Address:87393c21-b0ba-bb09-e374-718a7fd49670}]"
2026-05-25T23:24:08.968Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:08.968Z [INFO]  server-87393c21-b0ba-bb09-e374-718a7fd49670: entering follower state: follower="Node at 87393c21-b0ba-bb09-e374-718a7fd49670 [Follower]" leader-address= leader-id=
2026-05-25T23:24:09.021Z [WARN]  server-a760903a-a252-ba30-010a-e0f8cafab701: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:09.021Z [INFO]  server-a760903a-a252-ba30-010a-e0f8cafab701: entering candidate state: node="Node at a760903a-a252-ba30-010a-e0f8cafab701 [Candidate]" term=2
2026-05-25T23:24:09.021Z [INFO]  server-a760903a-a252-ba30-010a-e0f8cafab701: election won: term=2 tally=2
2026-05-25T23:24:09.021Z [INFO]  server-a760903a-a252-ba30-010a-e0f8cafab701: entering leader state: leader="Node at a760903a-a252-ba30-010a-e0f8cafab701 [Leader]"
2026-05-25T23:24:09.021Z [INFO]  server-a760903a-a252-ba30-010a-e0f8cafab701: added peer, starting replication: peer=server-7ac26b99-2129-56e1-0f92-987e62d39d1b
2026-05-25T23:24:09.021Z [INFO]  server-a760903a-a252-ba30-010a-e0f8cafab701: added peer, starting replication: peer=server-87393c21-b0ba-bb09-e374-718a7fd49670
2026-05-25T23:24:09.021Z [INFO]  server-a760903a-a252-ba30-010a-e0f8cafab701: pipelining replication: peer="{Voter server-7ac26b99-2129-56e1-0f92-987e62d39d1b 7ac26b99-2129-56e1-0f92-987e62d39d1b}"
2026-05-25T23:24:09.021Z [INFO]  server-a760903a-a252-ba30-010a-e0f8cafab701: pipelining replication: peer="{Voter server-87393c21-b0ba-bb09-e374-718a7fd49670 87393c21-b0ba-bb09-e374-718a7fd49670}"
2026-05-25T23:24:09.127Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:09.021759069 +0000 UTC m=+104.668206535 (1 nodes), 53.322511ms from start of poll, 53.755039ms from cluster start. Timeout at 2026-05-25 23:24:09.126762217 +0000 UTC m=+104.773209667, 105.003132ms after stability
2026-05-25T23:24:09.127Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:09.232Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:09.127841545 +0000 UTC m=+104.774289009 (2 nodes), 1.236µs from start of poll, 159.837513ms from cluster start. Timeout at 2026-05-25 23:24:09.232856873 +0000 UTC m=+104.879304358, 105.015349ms after stability
2026-05-25T23:24:09.480Z [INFO]  server-a760903a-a252-ba30-010a-e0f8cafab701: aborting pipeline replication: peer="{Voter server-7ac26b99-2129-56e1-0f92-987e62d39d1b 7ac26b99-2129-56e1-0f92-987e62d39d1b}"
2026-05-25T23:24:09.480Z [INFO]  server-a760903a-a252-ba30-010a-e0f8cafab701: aborting pipeline replication: peer="{Voter server-87393c21-b0ba-bb09-e374-718a7fd49670 87393c21-b0ba-bb09-e374-718a7fd49670}"
--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.51s)
=== RUN   TestRaft_LeadershipTransferIgnoresNonvoters
2026-05-25T23:24:09.482Z [INFO]  server-399b5a6f-99f6-93e8-2cd6-9a5bf66603fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-399b5a6f-99f6-93e8-2cd6-9a5bf66603fd Address:399b5a6f-99f6-93e8-2cd6-9a5bf66603fd} {Suffrage:Voter ID:server-9c950722-000b-053c-8747-7679517131f5 Address:9c950722-000b-053c-8747-7679517131f5}]"
2026-05-25T23:24:09.482Z [INFO]  server-399b5a6f-99f6-93e8-2cd6-9a5bf66603fd: entering follower state: follower="Node at 399b5a6f-99f6-93e8-2cd6-9a5bf66603fd [Follower]" leader-address= leader-id=
2026-05-25T23:24:09.483Z [INFO]  server-9c950722-000b-053c-8747-7679517131f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-399b5a6f-99f6-93e8-2cd6-9a5bf66603fd Address:399b5a6f-99f6-93e8-2cd6-9a5bf66603fd} {Suffrage:Voter ID:server-9c950722-000b-053c-8747-7679517131f5 Address:9c950722-000b-053c-8747-7679517131f5}]"
2026-05-25T23:24:09.483Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:09.483Z [INFO]  server-9c950722-000b-053c-8747-7679517131f5: entering follower state: follower="Node at 9c950722-000b-053c-8747-7679517131f5 [Follower]" leader-address= leader-id=
2026-05-25T23:24:09.539Z [WARN]  server-9c950722-000b-053c-8747-7679517131f5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:09.539Z [INFO]  server-9c950722-000b-053c-8747-7679517131f5: entering candidate state: node="Node at 9c950722-000b-053c-8747-7679517131f5 [Candidate]" term=2
2026-05-25T23:24:09.539Z [INFO]  server-9c950722-000b-053c-8747-7679517131f5: election won: term=2 tally=2
2026-05-25T23:24:09.539Z [INFO]  server-9c950722-000b-053c-8747-7679517131f5: entering leader state: leader="Node at 9c950722-000b-053c-8747-7679517131f5 [Leader]"
2026-05-25T23:24:09.539Z [INFO]  server-9c950722-000b-053c-8747-7679517131f5: added peer, starting replication: peer=server-399b5a6f-99f6-93e8-2cd6-9a5bf66603fd
2026-05-25T23:24:09.539Z [INFO]  server-9c950722-000b-053c-8747-7679517131f5: pipelining replication: peer="{Voter server-399b5a6f-99f6-93e8-2cd6-9a5bf66603fd 399b5a6f-99f6-93e8-2cd6-9a5bf66603fd}"
2026-05-25T23:24:09.645Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:09.539367013 +0000 UTC m=+105.185814464 (1 nodes), 56.298572ms from start of poll, 56.573402ms from cluster start. Timeout at 2026-05-25 23:24:09.644368284 +0000 UTC m=+105.290815733, 105.001269ms after stability
2026-05-25T23:24:09.645Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:09.750Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:09.64537043 +0000 UTC m=+105.291817887 (1 nodes), 2.654µs from start of poll, 162.576825ms from cluster start. Timeout at 2026-05-25 23:24:09.750496847 +0000 UTC m=+105.396944302, 105.126415ms after stability
2026-05-25T23:24:09.750Z [INFO]  server-9c950722-000b-053c-8747-7679517131f5: updating configuration: command=DemoteVoter server-id=server-399b5a6f-99f6-93e8-2cd6-9a5bf66603fd server-addr= servers="[{Suffrage:Nonvoter ID:server-399b5a6f-99f6-93e8-2cd6-9a5bf66603fd Address:399b5a6f-99f6-93e8-2cd6-9a5bf66603fd} {Suffrage:Voter ID:server-9c950722-000b-053c-8747-7679517131f5 Address:9c950722-000b-053c-8747-7679517131f5}]"
2026-05-25T23:24:09.750Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:09.855Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:09.750767082 +0000 UTC m=+105.397214542 (1 nodes), 730ns from start of poll, 267.97348ms from cluster start. Timeout at 2026-05-25 23:24:09.855828236 +0000 UTC m=+105.502275690, 105.061148ms after stability
2026-05-25T23:24:09.855Z [INFO]  server-9c950722-000b-053c-8747-7679517131f5: aborting pipeline replication: peer="{Voter server-399b5a6f-99f6-93e8-2cd6-9a5bf66603fd 399b5a6f-99f6-93e8-2cd6-9a5bf66603fd}"
--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.37s)
=== RUN   TestRaft_LeadershipTransferStopRightAway
--- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s)
=== RUN   TestRaft_GetConfigurationNoBootstrap
2026-05-25T23:24:09.856Z [INFO]  server-d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013 Address:d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013} {Suffrage:Voter ID:server-c7a15913-d079-b18e-40dc-281169e81452 Address:c7a15913-d079-b18e-40dc-281169e81452}]"
2026-05-25T23:24:09.857Z [INFO]  server-d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013: entering follower state: follower="Node at d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013 [Follower]" leader-address= leader-id=
2026-05-25T23:24:09.857Z [INFO]  server-c7a15913-d079-b18e-40dc-281169e81452: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013 Address:d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013} {Suffrage:Voter ID:server-c7a15913-d079-b18e-40dc-281169e81452 Address:c7a15913-d079-b18e-40dc-281169e81452}]"
2026-05-25T23:24:09.857Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:09.857Z [INFO]  server-c7a15913-d079-b18e-40dc-281169e81452: entering follower state: follower="Node at c7a15913-d079-b18e-40dc-281169e81452 [Follower]" leader-address= leader-id=
2026-05-25T23:24:09.910Z [WARN]  server-d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:09.910Z [INFO]  server-d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013: entering candidate state: node="Node at d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013 [Candidate]" term=2
2026-05-25T23:24:09.911Z [INFO]  server-d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013: election won: term=2 tally=2
2026-05-25T23:24:09.911Z [INFO]  server-d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013: entering leader state: leader="Node at d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013 [Leader]"
2026-05-25T23:24:09.911Z [INFO]  server-d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013: added peer, starting replication: peer=server-c7a15913-d079-b18e-40dc-281169e81452
2026-05-25T23:24:09.911Z [INFO]  server-d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013: pipelining replication: peer="{Voter server-c7a15913-d079-b18e-40dc-281169e81452 c7a15913-d079-b18e-40dc-281169e81452}"
2026-05-25T23:24:10.016Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:09.911107465 +0000 UTC m=+105.557554919 (1 nodes), 54.004891ms from start of poll, 54.25451ms from cluster start. Timeout at 2026-05-25 23:24:10.016108121 +0000 UTC m=+105.662555572, 105.000653ms after stability
2026-05-25T23:24:10.016Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:10.121Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:10.016216802 +0000 UTC m=+105.662664261 (1 nodes), 755ns from start of poll, 159.363852ms from cluster start. Timeout at 2026-05-25 23:24:10.121230498 +0000 UTC m=+105.767677963, 105.013702ms after stability
2026-05-25T23:24:10.133Z [INFO]  server-c7a15913-d079-b18e-40dc-281169e81452: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013 Address:d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013} {Suffrage:Voter ID:server-c7a15913-d079-b18e-40dc-281169e81452 Address:c7a15913-d079-b18e-40dc-281169e81452}]"
2026-05-25T23:24:10.133Z [INFO]  server-d3c69a20-5f2f-5c24-b1e2-c0ef45dbe013: aborting pipeline replication: peer="{Voter server-c7a15913-d079-b18e-40dc-281169e81452 c7a15913-d079-b18e-40dc-281169e81452}"
--- PASS: TestRaft_GetConfigurationNoBootstrap (0.28s)
=== RUN   TestRaft_LogStoreIsMonotonic
2026-05-25T23:24:10.133Z [INFO]  server-ae4d3af1-ebe0-a191-b0fa-53dba6dae4bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ae4d3af1-ebe0-a191-b0fa-53dba6dae4bd Address:ae4d3af1-ebe0-a191-b0fa-53dba6dae4bd}]"
2026-05-25T23:24:10.133Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:10.133Z [INFO]  server-ae4d3af1-ebe0-a191-b0fa-53dba6dae4bd: entering follower state: follower="Node at ae4d3af1-ebe0-a191-b0fa-53dba6dae4bd [Follower]" leader-address= leader-id=
2026-05-25T23:24:10.232Z [WARN]  server-ae4d3af1-ebe0-a191-b0fa-53dba6dae4bd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:10.232Z [INFO]  server-ae4d3af1-ebe0-a191-b0fa-53dba6dae4bd: entering candidate state: node="Node at ae4d3af1-ebe0-a191-b0fa-53dba6dae4bd [Candidate]" term=2
2026-05-25T23:24:10.232Z [INFO]  server-ae4d3af1-ebe0-a191-b0fa-53dba6dae4bd: election won: term=2 tally=1
2026-05-25T23:24:10.232Z [INFO]  server-ae4d3af1-ebe0-a191-b0fa-53dba6dae4bd: entering leader state: leader="Node at ae4d3af1-ebe0-a191-b0fa-53dba6dae4bd [Leader]"
2026-05-25T23:24:10.338Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:10.232960264 +0000 UTC m=+105.879407714 (1 nodes), 98.989285ms from start of poll, 99.127027ms from cluster start. Timeout at 2026-05-25 23:24:10.337962841 +0000 UTC m=+105.984410294, 105.00258ms after stability
--- PASS: TestRaft_LogStoreIsMonotonic (0.21s)
=== RUN   TestRaft_CacheLogWithStoreError
2026-05-25T23:24:10.339Z [INFO]  server-ce1b8934-bb39-16b5-816e-1f82cd16a212: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce1b8934-bb39-16b5-816e-1f82cd16a212 Address:ce1b8934-bb39-16b5-816e-1f82cd16a212} {Suffrage:Voter ID:server-808243da-7823-cf6d-a053-030504b36f13 Address:808243da-7823-cf6d-a053-030504b36f13}]"
2026-05-25T23:24:10.339Z [INFO]  server-808243da-7823-cf6d-a053-030504b36f13: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce1b8934-bb39-16b5-816e-1f82cd16a212 Address:ce1b8934-bb39-16b5-816e-1f82cd16a212} {Suffrage:Voter ID:server-808243da-7823-cf6d-a053-030504b36f13 Address:808243da-7823-cf6d-a053-030504b36f13}]"
2026-05-25T23:24:10.339Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:10.339Z [INFO]  server-ce1b8934-bb39-16b5-816e-1f82cd16a212: entering follower state: follower="Node at ce1b8934-bb39-16b5-816e-1f82cd16a212 [Follower]" leader-address= leader-id=
2026-05-25T23:24:10.339Z [INFO]  server-808243da-7823-cf6d-a053-030504b36f13: entering follower state: follower="Node at 808243da-7823-cf6d-a053-030504b36f13 [Follower]" leader-address= leader-id=
2026-05-25T23:24:10.394Z [WARN]  server-808243da-7823-cf6d-a053-030504b36f13: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:10.394Z [INFO]  server-808243da-7823-cf6d-a053-030504b36f13: entering candidate state: node="Node at 808243da-7823-cf6d-a053-030504b36f13 [Candidate]" term=2
2026-05-25T23:24:10.394Z [INFO]  server-808243da-7823-cf6d-a053-030504b36f13: election won: term=2 tally=2
2026-05-25T23:24:10.394Z [INFO]  server-808243da-7823-cf6d-a053-030504b36f13: entering leader state: leader="Node at 808243da-7823-cf6d-a053-030504b36f13 [Leader]"
2026-05-25T23:24:10.394Z [INFO]  server-808243da-7823-cf6d-a053-030504b36f13: added peer, starting replication: peer=server-ce1b8934-bb39-16b5-816e-1f82cd16a212
2026-05-25T23:24:10.394Z [INFO]  server-808243da-7823-cf6d-a053-030504b36f13: pipelining replication: peer="{Voter server-ce1b8934-bb39-16b5-816e-1f82cd16a212 ce1b8934-bb39-16b5-816e-1f82cd16a212}"
2026-05-25T23:24:10.500Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:10.394773318 +0000 UTC m=+106.041220769 (1 nodes), 55.370687ms from start of poll, 55.578822ms from cluster start. Timeout at 2026-05-25 23:24:10.499781837 +0000 UTC m=+106.146229313, 105.008544ms after stability
2026-05-25T23:24:10.500Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:10.606Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:10.500637394 +0000 UTC m=+106.147084865 (1 nodes), 1.089µs from start of poll, 161.442918ms from cluster start. Timeout at 2026-05-25 23:24:10.605658503 +0000 UTC m=+106.252105988, 105.021123ms after stability
2026-05-25T23:24:10.607Z [ERROR] server-ce1b8934-bb39-16b5-816e-1f82cd16a212: failed to append to logs: error="unable to store logs within log store, err: "some error""
2026-05-25T23:24:10.607Z [WARN]  server-ce1b8934-bb39-16b5-816e-1f82cd16a212: failed to get previous log: previous-index=9 last-index=7 error="log not found"
2026-05-25T23:24:10.607Z [INFO]  server-808243da-7823-cf6d-a053-030504b36f13: aborting pipeline replication: peer="{Voter server-ce1b8934-bb39-16b5-816e-1f82cd16a212 ce1b8934-bb39-16b5-816e-1f82cd16a212}"
2026-05-25T23:24:10.615Z [INFO]  server-808243da-7823-cf6d-a053-030504b36f13: pipelining replication: peer="{Voter server-ce1b8934-bb39-16b5-816e-1f82cd16a212 ce1b8934-bb39-16b5-816e-1f82cd16a212}"
2026-05-25T23:24:10.615Z [INFO]  server-ce1b8934-bb39-16b5-816e-1f82cd16a212: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce1b8934-bb39-16b5-816e-1f82cd16a212 Address:ce1b8934-bb39-16b5-816e-1f82cd16a212} {Suffrage:Voter ID:server-808243da-7823-cf6d-a053-030504b36f13 Address:808243da-7823-cf6d-a053-030504b36f13}]"
2026-05-25T23:24:10.615Z [INFO]  server-808243da-7823-cf6d-a053-030504b36f13: aborting pipeline replication: peer="{Voter server-ce1b8934-bb39-16b5-816e-1f82cd16a212 ce1b8934-bb39-16b5-816e-1f82cd16a212}"
--- PASS: TestRaft_CacheLogWithStoreError (0.28s)
=== RUN   TestRaft_ReloadConfig
2026-05-25T23:24:10.615Z [INFO]  server-635d9b77-e71d-5cfd-96db-862c7e722d6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-635d9b77-e71d-5cfd-96db-862c7e722d6c Address:635d9b77-e71d-5cfd-96db-862c7e722d6c}]"
2026-05-25T23:24:10.616Z [INFO]  server-635d9b77-e71d-5cfd-96db-862c7e722d6c: entering follower state: follower="Node at 635d9b77-e71d-5cfd-96db-862c7e722d6c [Shutdown]" leader-address= leader-id=
--- PASS: TestRaft_ReloadConfig (0.00s)
=== RUN   TestRaft_ReloadConfigValidates
2026-05-25T23:24:10.616Z [INFO]  server-28231137-485a-7815-423b-9a7b5d195553: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-28231137-485a-7815-423b-9a7b5d195553 Address:28231137-485a-7815-423b-9a7b5d195553}]"
2026-05-25T23:24:10.616Z [INFO]  server-28231137-485a-7815-423b-9a7b5d195553: entering follower state: follower="Node at 28231137-485a-7815-423b-9a7b5d195553 [Shutdown]" leader-address= leader-id=
--- PASS: TestRaft_ReloadConfigValidates (0.00s)
=== RUN   TestRaft_InstallSnapshot_InvalidPeers
2026-05-25T23:24:10.616Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)"
--- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s)
=== RUN   TestRaft_VoteNotGranted_WhenNodeNotInCluster
2026-05-25T23:24:10.617Z [INFO]  server-0c8c3d50-abcb-792f-1344-34a412d67ed2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c8c3d50-abcb-792f-1344-34a412d67ed2 Address:0c8c3d50-abcb-792f-1344-34a412d67ed2} {Suffrage:Voter ID:server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d Address:1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d} {Suffrage:Voter ID:server-90c5abec-9c9a-8726-a51a-5c70b32526b2 Address:90c5abec-9c9a-8726-a51a-5c70b32526b2}]"
2026-05-25T23:24:10.617Z [INFO]  server-0c8c3d50-abcb-792f-1344-34a412d67ed2: entering follower state: follower="Node at 0c8c3d50-abcb-792f-1344-34a412d67ed2 [Follower]" leader-address= leader-id=
2026-05-25T23:24:10.617Z [INFO]  server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c8c3d50-abcb-792f-1344-34a412d67ed2 Address:0c8c3d50-abcb-792f-1344-34a412d67ed2} {Suffrage:Voter ID:server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d Address:1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d} {Suffrage:Voter ID:server-90c5abec-9c9a-8726-a51a-5c70b32526b2 Address:90c5abec-9c9a-8726-a51a-5c70b32526b2}]"
2026-05-25T23:24:10.617Z [INFO]  server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d: entering follower state: follower="Node at 1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d [Follower]" leader-address= leader-id=
2026-05-25T23:24:10.617Z [INFO]  server-90c5abec-9c9a-8726-a51a-5c70b32526b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c8c3d50-abcb-792f-1344-34a412d67ed2 Address:0c8c3d50-abcb-792f-1344-34a412d67ed2} {Suffrage:Voter ID:server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d Address:1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d} {Suffrage:Voter ID:server-90c5abec-9c9a-8726-a51a-5c70b32526b2 Address:90c5abec-9c9a-8726-a51a-5c70b32526b2}]"
2026-05-25T23:24:10.617Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:10.617Z [INFO]  server-90c5abec-9c9a-8726-a51a-5c70b32526b2: entering follower state: follower="Node at 90c5abec-9c9a-8726-a51a-5c70b32526b2 [Follower]" leader-address= leader-id=
2026-05-25T23:24:10.689Z [WARN]  server-90c5abec-9c9a-8726-a51a-5c70b32526b2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:10.690Z [INFO]  server-90c5abec-9c9a-8726-a51a-5c70b32526b2: entering candidate state: node="Node at 90c5abec-9c9a-8726-a51a-5c70b32526b2 [Candidate]" term=2
2026-05-25T23:24:10.690Z [INFO]  server-90c5abec-9c9a-8726-a51a-5c70b32526b2: election won: term=2 tally=2
2026-05-25T23:24:10.690Z [INFO]  server-90c5abec-9c9a-8726-a51a-5c70b32526b2: entering leader state: leader="Node at 90c5abec-9c9a-8726-a51a-5c70b32526b2 [Leader]"
2026-05-25T23:24:10.690Z [INFO]  server-90c5abec-9c9a-8726-a51a-5c70b32526b2: added peer, starting replication: peer=server-0c8c3d50-abcb-792f-1344-34a412d67ed2
2026-05-25T23:24:10.690Z [INFO]  server-90c5abec-9c9a-8726-a51a-5c70b32526b2: added peer, starting replication: peer=server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d
2026-05-25T23:24:10.690Z [INFO]  server-90c5abec-9c9a-8726-a51a-5c70b32526b2: pipelining replication: peer="{Voter server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d 1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d}"
2026-05-25T23:24:10.690Z [INFO]  server-90c5abec-9c9a-8726-a51a-5c70b32526b2: pipelining replication: peer="{Voter server-0c8c3d50-abcb-792f-1344-34a412d67ed2 0c8c3d50-abcb-792f-1344-34a412d67ed2}"
2026-05-25T23:24:10.795Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:10.690136071 +0000 UTC m=+106.336583525 (1 nodes), 72.552787ms from start of poll, 72.778716ms from cluster start. Timeout at 2026-05-25 23:24:10.795139807 +0000 UTC m=+106.441587258, 105.003733ms after stability
2026-05-25T23:24:10.801Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:10.907Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:10.801269992 +0000 UTC m=+106.447717516 (2 nodes), 1.273µs from start of poll, 183.912707ms from cluster start. Timeout at 2026-05-25 23:24:10.906279117 +0000 UTC m=+106.552726602, 105.009086ms after stability
2026-05-25T23:24:10.907Z [INFO]  server-90c5abec-9c9a-8726-a51a-5c70b32526b2: updating configuration: command=RemoveServer server-id=server-0c8c3d50-abcb-792f-1344-34a412d67ed2 server-addr= servers="[{Suffrage:Voter ID:server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d Address:1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d} {Suffrage:Voter ID:server-90c5abec-9c9a-8726-a51a-5c70b32526b2 Address:90c5abec-9c9a-8726-a51a-5c70b32526b2}]"
2026-05-25T23:24:10.907Z [INFO]  server-90c5abec-9c9a-8726-a51a-5c70b32526b2: removed peer, stopping replication: peer=server-0c8c3d50-abcb-792f-1344-34a412d67ed2 last-index=3
2026-05-25T23:24:10.907Z [INFO]  server-90c5abec-9c9a-8726-a51a-5c70b32526b2: aborting pipeline replication: peer="{Voter server-0c8c3d50-abcb-792f-1344-34a412d67ed2 0c8c3d50-abcb-792f-1344-34a412d67ed2}"
2026-05-25T23:24:10.979Z [WARN]  server-0c8c3d50-abcb-792f-1344-34a412d67ed2: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2026-05-25T23:24:11.013Z [ERROR] server-90c5abec-9c9a-8726-a51a-5c70b32526b2: failed to pipeline appendEntries: peer="{Voter server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d 1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d}" error="append pipeline closed"
2026-05-25T23:24:11.013Z [INFO]  server-90c5abec-9c9a-8726-a51a-5c70b32526b2: aborting pipeline replication: peer="{Voter server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d 1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d}"
2026-05-25T23:24:11.016Z [ERROR] server-90c5abec-9c9a-8726-a51a-5c70b32526b2: failed to heartbeat to: peer=1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d backoff time=10ms error="failed to connect to peer: 1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d"
2026-05-25T23:24:11.022Z [ERROR] server-90c5abec-9c9a-8726-a51a-5c70b32526b2: failed to appendEntries to: peer="{Voter server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d 1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d}" error="failed to connect to peer: 1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d"
2026-05-25T23:24:11.033Z [ERROR] server-90c5abec-9c9a-8726-a51a-5c70b32526b2: failed to heartbeat to: peer=1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d backoff time=10ms error="failed to connect to peer: 1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d"
2026-05-25T23:24:11.041Z [ERROR] server-90c5abec-9c9a-8726-a51a-5c70b32526b2: failed to appendEntries to: peer="{Voter server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d 1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d}" error="failed to connect to peer: 1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d"
2026-05-25T23:24:11.049Z [ERROR] server-90c5abec-9c9a-8726-a51a-5c70b32526b2: failed to heartbeat to: peer=1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d backoff time=10ms error="failed to connect to peer: 1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d"
2026-05-25T23:24:11.058Z [ERROR] server-90c5abec-9c9a-8726-a51a-5c70b32526b2: failed to appendEntries to: peer="{Voter server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d 1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d}" error="failed to connect to peer: 1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d"
2026-05-25T23:24:11.058Z [WARN]  server-90c5abec-9c9a-8726-a51a-5c70b32526b2: failed to contact: server-id=server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d time=50.159887ms
2026-05-25T23:24:11.058Z [WARN]  server-90c5abec-9c9a-8726-a51a-5c70b32526b2: failed to contact quorum of nodes, stepping down
2026-05-25T23:24:11.058Z [INFO]  server-90c5abec-9c9a-8726-a51a-5c70b32526b2: entering follower state: follower="Node at 90c5abec-9c9a-8726-a51a-5c70b32526b2 [Follower]" leader-address= leader-id=
2026-05-25T23:24:11.084Z [WARN]  server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d: heartbeat timeout reached, starting election: last-leader-addr=90c5abec-9c9a-8726-a51a-5c70b32526b2 last-leader-id=server-90c5abec-9c9a-8726-a51a-5c70b32526b2
2026-05-25T23:24:11.084Z [INFO]  server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d: entering candidate state: node="Node at 1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d [Candidate]" term=3
2026-05-25T23:24:11.085Z [ERROR] server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d: failed to make requestVote RPC: target="{Voter server-90c5abec-9c9a-8726-a51a-5c70b32526b2 90c5abec-9c9a-8726-a51a-5c70b32526b2}" error="failed to connect to peer: 90c5abec-9c9a-8726-a51a-5c70b32526b2" term=3
2026-05-25T23:24:11.118Z [WARN]  server-1ed5a5b9-f799-3d50-2f2f-49ad1a58d25d: rejecting vote request since node is not in configuration: from=0c8c3d50-abcb-792f-1344-34a412d67ed2
--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.50s)
=== RUN   TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin
2026-05-25T23:24:11.119Z [INFO]  server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 Address:d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02} {Suffrage:Voter ID:server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95 Address:c294a701-6e3a-15f4-c9ab-1f26ac84bb95} {Suffrage:Voter ID:server-57d88dc4-6000-f325-8239-c867cd4acb7a Address:57d88dc4-6000-f325-8239-c867cd4acb7a}]"
2026-05-25T23:24:11.119Z [INFO]  server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02: entering follower state: follower="Node at d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 [Follower]" leader-address= leader-id=
2026-05-25T23:24:11.119Z [INFO]  server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 Address:d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02} {Suffrage:Voter ID:server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95 Address:c294a701-6e3a-15f4-c9ab-1f26ac84bb95} {Suffrage:Voter ID:server-57d88dc4-6000-f325-8239-c867cd4acb7a Address:57d88dc4-6000-f325-8239-c867cd4acb7a}]"
2026-05-25T23:24:11.119Z [INFO]  server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: entering follower state: follower="Node at c294a701-6e3a-15f4-c9ab-1f26ac84bb95 [Follower]" leader-address= leader-id=
2026-05-25T23:24:11.119Z [INFO]  server-57d88dc4-6000-f325-8239-c867cd4acb7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 Address:d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02} {Suffrage:Voter ID:server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95 Address:c294a701-6e3a-15f4-c9ab-1f26ac84bb95} {Suffrage:Voter ID:server-57d88dc4-6000-f325-8239-c867cd4acb7a Address:57d88dc4-6000-f325-8239-c867cd4acb7a}]"
2026-05-25T23:24:11.119Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:11.119Z [INFO]  server-57d88dc4-6000-f325-8239-c867cd4acb7a: entering follower state: follower="Node at 57d88dc4-6000-f325-8239-c867cd4acb7a [Follower]" leader-address= leader-id=
2026-05-25T23:24:11.174Z [WARN]  server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:11.174Z [INFO]  server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: entering candidate state: node="Node at c294a701-6e3a-15f4-c9ab-1f26ac84bb95 [Candidate]" term=2
2026-05-25T23:24:11.174Z [INFO]  server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: election won: term=2 tally=2
2026-05-25T23:24:11.174Z [INFO]  server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: entering leader state: leader="Node at c294a701-6e3a-15f4-c9ab-1f26ac84bb95 [Leader]"
2026-05-25T23:24:11.174Z [INFO]  server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: added peer, starting replication: peer=server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02
2026-05-25T23:24:11.174Z [INFO]  server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: added peer, starting replication: peer=server-57d88dc4-6000-f325-8239-c867cd4acb7a
2026-05-25T23:24:11.174Z [INFO]  server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: pipelining replication: peer="{Voter server-57d88dc4-6000-f325-8239-c867cd4acb7a 57d88dc4-6000-f325-8239-c867cd4acb7a}"
2026-05-25T23:24:11.174Z [INFO]  server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: pipelining replication: peer="{Voter server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02}"
2026-05-25T23:24:11.279Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:11.174347418 +0000 UTC m=+106.820794968 (1 nodes), 54.448744ms from start of poll, 54.726082ms from cluster start. Timeout at 2026-05-25 23:24:11.279357566 +0000 UTC m=+106.925805056, 105.010088ms after stability
2026-05-25T23:24:11.285Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:11.391Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:11.285844319 +0000 UTC m=+106.932291786 (2 nodes), 2.908µs from start of poll, 166.2229ms from cluster start. Timeout at 2026-05-25 23:24:11.390878475 +0000 UTC m=+107.037325946, 105.03416ms after stability
2026-05-25T23:24:11.392Z [ERROR] server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: failed to heartbeat to: peer=d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 backoff time=10ms error="failed to connect to peer: d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02"
2026-05-25T23:24:11.395Z [ERROR] server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: failed to pipeline appendEntries: peer="{Voter server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02}" error="append pipeline closed"
2026-05-25T23:24:11.395Z [INFO]  server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: aborting pipeline replication: peer="{Voter server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02}"
2026-05-25T23:24:11.405Z [ERROR] server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: failed to appendEntries to: peer="{Voter server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02}" error="failed to connect to peer: d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02"
2026-05-25T23:24:11.409Z [ERROR] server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: failed to heartbeat to: peer=d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 backoff time=10ms error="failed to connect to peer: d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02"
2026-05-25T23:24:11.422Z [ERROR] server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: failed to appendEntries to: peer="{Voter server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02}" error="failed to connect to peer: d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02"
2026-05-25T23:24:11.429Z [ERROR] server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: failed to heartbeat to: peer=d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 backoff time=10ms error="failed to connect to peer: d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02"
2026-05-25T23:24:11.436Z [WARN]  server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: failed to contact: server-id=server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 time=50.142445ms
2026-05-25T23:24:11.444Z [ERROR] server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: failed to appendEntries to: peer="{Voter server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02}" error="failed to connect to peer: d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02"
2026-05-25T23:24:11.818Z [INFO]  server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: updating configuration: command=RemoveServer server-id=server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 server-addr= servers="[{Suffrage:Voter ID:server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95 Address:c294a701-6e3a-15f4-c9ab-1f26ac84bb95} {Suffrage:Voter ID:server-57d88dc4-6000-f325-8239-c867cd4acb7a Address:57d88dc4-6000-f325-8239-c867cd4acb7a}]"
2026-05-25T23:24:11.818Z [WARN]  server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02: heartbeat timeout reached, starting election: last-leader-addr=c294a701-6e3a-15f4-c9ab-1f26ac84bb95 last-leader-id=server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95
2026-05-25T23:24:11.818Z [INFO]  server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02: entering candidate state: node="Node at d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 [Candidate]" term=3
2026-05-25T23:24:11.818Z [INFO]  server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: removed peer, stopping replication: peer=server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 last-index=3
2026-05-25T23:24:11.819Z [WARN]  server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: failed to contact quorum of nodes, stepping down
2026-05-25T23:24:11.819Z [INFO]  server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: entering follower state: follower="Node at c294a701-6e3a-15f4-c9ab-1f26ac84bb95 [Follower]" leader-address= leader-id=
2026-05-25T23:24:11.818Z [ERROR] server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: failed to heartbeat to: peer=d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 backoff time=20ms error="failed to connect to peer: d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02"
    raft_test.go:2865: err: leadership lost while committing log
2026-05-25T23:24:11.819Z [ERROR] server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02: failed to make requestVote RPC: target="{Voter server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95 c294a701-6e3a-15f4-c9ab-1f26ac84bb95}" error="failed to connect to peer: c294a701-6e3a-15f4-c9ab-1f26ac84bb95" term=3
2026-05-25T23:24:11.819Z [INFO]  server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: aborting pipeline replication: peer="{Voter server-57d88dc4-6000-f325-8239-c867cd4acb7a 57d88dc4-6000-f325-8239-c867cd4acb7a}"
2026-05-25T23:24:11.819Z [ERROR] server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02: failed to make requestVote RPC: target="{Voter server-57d88dc4-6000-f325-8239-c867cd4acb7a 57d88dc4-6000-f325-8239-c867cd4acb7a}" error="failed to connect to peer: 57d88dc4-6000-f325-8239-c867cd4acb7a" term=3
2026-05-25T23:24:11.819Z [ERROR] server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: failed to appendEntries to: peer="{Voter server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02}" error="failed to connect to peer: d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02"
2026-05-25T23:24:11.819Z [ERROR] server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: failed to appendEntries to: peer="{Voter server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02}" error="failed to connect to peer: d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02"
2026-05-25T23:24:11.819Z [ERROR] server-c294a701-6e3a-15f4-c9ab-1f26ac84bb95: failed to appendEntries to: peer="{Voter server-d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02 d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02}" error="failed to connect to peer: d3f8e41e-d1ea-80f8-8cdc-1c6838c75d02"
--- FAIL: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (0.70s)
=== RUN   TestRaft_FollowerRemovalNoElection
2026-05-25T23:24:11.820Z [INFO]  server-9d7264df-6f8f-306d-4b93-a8a49164eaf8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d7264df-6f8f-306d-4b93-a8a49164eaf8 Address:9d7264df-6f8f-306d-4b93-a8a49164eaf8} {Suffrage:Voter ID:server-6c93a202-5278-9117-276a-a4a3915afb43 Address:6c93a202-5278-9117-276a-a4a3915afb43} {Suffrage:Voter ID:server-8df1fa0c-6958-dfda-6afd-767e389ee369 Address:8df1fa0c-6958-dfda-6afd-767e389ee369}]"
2026-05-25T23:24:11.820Z [INFO]  server-6c93a202-5278-9117-276a-a4a3915afb43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d7264df-6f8f-306d-4b93-a8a49164eaf8 Address:9d7264df-6f8f-306d-4b93-a8a49164eaf8} {Suffrage:Voter ID:server-6c93a202-5278-9117-276a-a4a3915afb43 Address:6c93a202-5278-9117-276a-a4a3915afb43} {Suffrage:Voter ID:server-8df1fa0c-6958-dfda-6afd-767e389ee369 Address:8df1fa0c-6958-dfda-6afd-767e389ee369}]"
2026-05-25T23:24:11.820Z [INFO]  server-6c93a202-5278-9117-276a-a4a3915afb43: entering follower state: follower="Node at 6c93a202-5278-9117-276a-a4a3915afb43 [Follower]" leader-address= leader-id=
2026-05-25T23:24:11.820Z [INFO]  server-9d7264df-6f8f-306d-4b93-a8a49164eaf8: entering follower state: follower="Node at 9d7264df-6f8f-306d-4b93-a8a49164eaf8 [Follower]" leader-address= leader-id=
2026-05-25T23:24:11.820Z [INFO]  server-8df1fa0c-6958-dfda-6afd-767e389ee369: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d7264df-6f8f-306d-4b93-a8a49164eaf8 Address:9d7264df-6f8f-306d-4b93-a8a49164eaf8} {Suffrage:Voter ID:server-6c93a202-5278-9117-276a-a4a3915afb43 Address:6c93a202-5278-9117-276a-a4a3915afb43} {Suffrage:Voter ID:server-8df1fa0c-6958-dfda-6afd-767e389ee369 Address:8df1fa0c-6958-dfda-6afd-767e389ee369}]"
2026-05-25T23:24:11.820Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:11.820Z [INFO]  server-8df1fa0c-6958-dfda-6afd-767e389ee369: entering follower state: follower="Node at 8df1fa0c-6958-dfda-6afd-767e389ee369 [Follower]" leader-address= leader-id=
2026-05-25T23:24:11.940Z [WARN]  server-8df1fa0c-6958-dfda-6afd-767e389ee369: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:11.940Z [INFO]  server-8df1fa0c-6958-dfda-6afd-767e389ee369: entering candidate state: node="Node at 8df1fa0c-6958-dfda-6afd-767e389ee369 [Candidate]" term=2
2026-05-25T23:24:11.940Z [INFO]  server-8df1fa0c-6958-dfda-6afd-767e389ee369: election won: term=2 tally=2
2026-05-25T23:24:11.940Z [INFO]  server-8df1fa0c-6958-dfda-6afd-767e389ee369: entering leader state: leader="Node at 8df1fa0c-6958-dfda-6afd-767e389ee369 [Leader]"
2026-05-25T23:24:11.940Z [INFO]  server-8df1fa0c-6958-dfda-6afd-767e389ee369: added peer, starting replication: peer=server-9d7264df-6f8f-306d-4b93-a8a49164eaf8
2026-05-25T23:24:11.940Z [INFO]  server-8df1fa0c-6958-dfda-6afd-767e389ee369: added peer, starting replication: peer=server-6c93a202-5278-9117-276a-a4a3915afb43
2026-05-25T23:24:11.940Z [INFO]  server-8df1fa0c-6958-dfda-6afd-767e389ee369: pipelining replication: peer="{Voter server-6c93a202-5278-9117-276a-a4a3915afb43 6c93a202-5278-9117-276a-a4a3915afb43}"
2026-05-25T23:24:11.940Z [INFO]  server-8df1fa0c-6958-dfda-6afd-767e389ee369: pipelining replication: peer="{Voter server-9d7264df-6f8f-306d-4b93-a8a49164eaf8 9d7264df-6f8f-306d-4b93-a8a49164eaf8}"
2026-05-25T23:24:12.146Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:11.940949942 +0000 UTC m=+107.587397414 (1 nodes), 120.655149ms from start of poll, 120.912106ms from cluster start. Timeout at 2026-05-25 23:24:12.145954441 +0000 UTC m=+107.792401895, 205.004481ms after stability
2026-05-25T23:24:12.146Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:12.352Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:12.146751351 +0000 UTC m=+107.793198805 (1 nodes), 2.135µs from start of poll, 326.713497ms from cluster start. Timeout at 2026-05-25 23:24:12.3517744 +0000 UTC m=+107.998221888, 205.023083ms after stability
2026-05-25T23:24:12.358Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:12.564Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:12.358887429 +0000 UTC m=+108.005334880 (2 nodes), 1.913µs from start of poll, 538.849572ms from cluster start. Timeout at 2026-05-25 23:24:12.563908938 +0000 UTC m=+108.210356411, 205.021531ms after stability
2026-05-25T23:24:12.564Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:12.770Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:12.564813123 +0000 UTC m=+108.211260575 (1 nodes), 1.461µs from start of poll, 744.775267ms from cluster start. Timeout at 2026-05-25 23:24:12.769831379 +0000 UTC m=+108.416278860, 205.018285ms after stability
    raft_test.go:2961: [INFO] restarting Node at 9d7264df-6f8f-306d-4b93-a8a49164eaf8 [Follower]
2026-05-25T23:24:12.771Z [INFO]  server-9d7264df-6f8f-306d-4b93-a8a49164eaf8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d7264df-6f8f-306d-4b93-a8a49164eaf8 Address:9d7264df-6f8f-306d-4b93-a8a49164eaf8} {Suffrage:Voter ID:server-6c93a202-5278-9117-276a-a4a3915afb43 Address:6c93a202-5278-9117-276a-a4a3915afb43} {Suffrage:Voter ID:server-8df1fa0c-6958-dfda-6afd-767e389ee369 Address:8df1fa0c-6958-dfda-6afd-767e389ee369}]"
2026-05-25T23:24:12.771Z [INFO]  server-9d7264df-6f8f-306d-4b93-a8a49164eaf8: entering follower state: follower="Node at 9d7264df-6f8f-306d-4b93-a8a49164eaf8 [Follower]" leader-address= leader-id=
2026-05-25T23:24:13.022Z [INFO]  server-8df1fa0c-6958-dfda-6afd-767e389ee369: aborting pipeline replication: peer="{Voter server-6c93a202-5278-9117-276a-a4a3915afb43 6c93a202-5278-9117-276a-a4a3915afb43}"
2026-05-25T23:24:13.411Z [ERROR] server-8df1fa0c-6958-dfda-6afd-767e389ee369: failed to pipeline appendEntries: peer="{Voter server-9d7264df-6f8f-306d-4b93-a8a49164eaf8 9d7264df-6f8f-306d-4b93-a8a49164eaf8}" error="command enqueue timeout"
2026-05-25T23:24:13.411Z [INFO]  server-8df1fa0c-6958-dfda-6afd-767e389ee369: aborting pipeline replication: peer="{Voter server-9d7264df-6f8f-306d-4b93-a8a49164eaf8 9d7264df-6f8f-306d-4b93-a8a49164eaf8}"
2026-05-25T23:24:13.523Z [ERROR] server-8df1fa0c-6958-dfda-6afd-767e389ee369: failed to heartbeat to: peer=9d7264df-6f8f-306d-4b93-a8a49164eaf8 backoff time=10ms error="command timed out"
--- PASS: TestRaft_FollowerRemovalNoElection (1.70s)
=== RUN   TestRaft_VoteWithNoIDNoAddr
2026-05-25T23:24:13.524Z [INFO]  server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1 Address:fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1} {Suffrage:Voter ID:server-35e32fb3-f7fb-72ca-c959-5c685655a828 Address:35e32fb3-f7fb-72ca-c959-5c685655a828} {Suffrage:Voter ID:server-4a5f8030-e902-bb7a-2531-701071cea7f4 Address:4a5f8030-e902-bb7a-2531-701071cea7f4}]"
2026-05-25T23:24:13.524Z [INFO]  server-35e32fb3-f7fb-72ca-c959-5c685655a828: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1 Address:fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1} {Suffrage:Voter ID:server-35e32fb3-f7fb-72ca-c959-5c685655a828 Address:35e32fb3-f7fb-72ca-c959-5c685655a828} {Suffrage:Voter ID:server-4a5f8030-e902-bb7a-2531-701071cea7f4 Address:4a5f8030-e902-bb7a-2531-701071cea7f4}]"
2026-05-25T23:24:13.524Z [INFO]  server-35e32fb3-f7fb-72ca-c959-5c685655a828: entering follower state: follower="Node at 35e32fb3-f7fb-72ca-c959-5c685655a828 [Follower]" leader-address= leader-id=
2026-05-25T23:24:13.524Z [INFO]  server-4a5f8030-e902-bb7a-2531-701071cea7f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1 Address:fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1} {Suffrage:Voter ID:server-35e32fb3-f7fb-72ca-c959-5c685655a828 Address:35e32fb3-f7fb-72ca-c959-5c685655a828} {Suffrage:Voter ID:server-4a5f8030-e902-bb7a-2531-701071cea7f4 Address:4a5f8030-e902-bb7a-2531-701071cea7f4}]"
2026-05-25T23:24:13.524Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:13.524Z [INFO]  server-4a5f8030-e902-bb7a-2531-701071cea7f4: entering follower state: follower="Node at 4a5f8030-e902-bb7a-2531-701071cea7f4 [Follower]" leader-address= leader-id=
2026-05-25T23:24:13.524Z [INFO]  server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1: entering follower state: follower="Node at fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1 [Follower]" leader-address= leader-id=
2026-05-25T23:24:13.592Z [WARN]  server-35e32fb3-f7fb-72ca-c959-5c685655a828: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-05-25T23:24:13.592Z [INFO]  server-35e32fb3-f7fb-72ca-c959-5c685655a828: entering candidate state: node="Node at 35e32fb3-f7fb-72ca-c959-5c685655a828 [Candidate]" term=2
2026-05-25T23:24:13.592Z [INFO]  server-35e32fb3-f7fb-72ca-c959-5c685655a828: election won: term=2 tally=2
2026-05-25T23:24:13.592Z [INFO]  server-35e32fb3-f7fb-72ca-c959-5c685655a828: entering leader state: leader="Node at 35e32fb3-f7fb-72ca-c959-5c685655a828 [Leader]"
2026-05-25T23:24:13.592Z [INFO]  server-35e32fb3-f7fb-72ca-c959-5c685655a828: added peer, starting replication: peer=server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1
2026-05-25T23:24:13.592Z [INFO]  server-35e32fb3-f7fb-72ca-c959-5c685655a828: added peer, starting replication: peer=server-4a5f8030-e902-bb7a-2531-701071cea7f4
2026-05-25T23:24:13.592Z [INFO]  server-35e32fb3-f7fb-72ca-c959-5c685655a828: pipelining replication: peer="{Voter server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1 fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1}"
2026-05-25T23:24:13.592Z [INFO]  server-35e32fb3-f7fb-72ca-c959-5c685655a828: pipelining replication: peer="{Voter server-4a5f8030-e902-bb7a-2531-701071cea7f4 4a5f8030-e902-bb7a-2531-701071cea7f4}"
2026-05-25T23:24:13.697Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:13.592294874 +0000 UTC m=+109.238742329 (1 nodes), 67.417048ms from start of poll, 67.751252ms from cluster start. Timeout at 2026-05-25 23:24:13.69729834 +0000 UTC m=+109.343745794, 105.003465ms after stability
2026-05-25T23:24:13.697Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-05-25T23:24:13.802Z [INFO]  cluster: stable state for Leader reached at 2026-05-25 23:24:13.697655458 +0000 UTC m=+109.344102912 (1 nodes), 748ns from start of poll, 173.111835ms from cluster start. Timeout at 2026-05-25 23:24:13.802666461 +0000 UTC m=+109.449113942, 105.01103ms after stability
2026-05-25T23:24:13.808Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-05-25T23:24:13.914Z [INFO]  cluster: stable state for Follower reached at 2026-05-25 23:24:13.808947584 +0000 UTC m=+109.455395074 (2 nodes), 1.871µs from start of poll, 284.403997ms from cluster start. Timeout at 2026-05-25 23:24:13.913960544 +0000 UTC m=+109.560408094, 105.01302ms after stability
2026-05-25T23:24:13.916Z [ERROR] server-35e32fb3-f7fb-72ca-c959-5c685655a828: failed to heartbeat to: peer=4a5f8030-e902-bb7a-2531-701071cea7f4 backoff time=10ms error="failed to connect to peer: 4a5f8030-e902-bb7a-2531-701071cea7f4"
2026-05-25T23:24:13.921Z [ERROR] server-35e32fb3-f7fb-72ca-c959-5c685655a828: failed to pipeline appendEntries: peer="{Voter server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1 fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1}" error="append pipeline closed"
2026-05-25T23:24:13.921Z [INFO]  server-35e32fb3-f7fb-72ca-c959-5c685655a828: aborting pipeline replication: peer="{Voter server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1 fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1}"
2026-05-25T23:24:13.922Z [ERROR] server-35e32fb3-f7fb-72ca-c959-5c685655a828: failed to pipeline appendEntries: peer="{Voter server-4a5f8030-e902-bb7a-2531-701071cea7f4 4a5f8030-e902-bb7a-2531-701071cea7f4}" error="append pipeline closed"
2026-05-25T23:24:13.922Z [INFO]  server-35e32fb3-f7fb-72ca-c959-5c685655a828: aborting pipeline replication: peer="{Voter server-4a5f8030-e902-bb7a-2531-701071cea7f4 4a5f8030-e902-bb7a-2531-701071cea7f4}"
2026-05-25T23:24:13.922Z [ERROR] server-35e32fb3-f7fb-72ca-c959-5c685655a828: failed to heartbeat to: peer=fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1 backoff time=10ms error="failed to connect to peer: fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1"
2026-05-25T23:24:13.932Z [ERROR] server-35e32fb3-f7fb-72ca-c959-5c685655a828: failed to appendEntries to: peer="{Voter server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1 fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1}" error="failed to connect to peer: fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1"
2026-05-25T23:24:13.932Z [ERROR] server-35e32fb3-f7fb-72ca-c959-5c685655a828: failed to appendEntries to: peer="{Voter server-4a5f8030-e902-bb7a-2531-701071cea7f4 4a5f8030-e902-bb7a-2531-701071cea7f4}" error="failed to connect to peer: 4a5f8030-e902-bb7a-2531-701071cea7f4"
2026-05-25T23:24:13.935Z [ERROR] server-35e32fb3-f7fb-72ca-c959-5c685655a828: failed to heartbeat to: peer=4a5f8030-e902-bb7a-2531-701071cea7f4 backoff time=10ms error="failed to connect to peer: 4a5f8030-e902-bb7a-2531-701071cea7f4"
2026-05-25T23:24:13.940Z [ERROR] server-35e32fb3-f7fb-72ca-c959-5c685655a828: failed to heartbeat to: peer=fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1 backoff time=10ms error="failed to connect to peer: fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1"
2026-05-25T23:24:13.951Z [ERROR] server-35e32fb3-f7fb-72ca-c959-5c685655a828: failed to appendEntries to: peer="{Voter server-4a5f8030-e902-bb7a-2531-701071cea7f4 4a5f8030-e902-bb7a-2531-701071cea7f4}" error="failed to connect to peer: 4a5f8030-e902-bb7a-2531-701071cea7f4"
2026-05-25T23:24:13.951Z [ERROR] server-35e32fb3-f7fb-72ca-c959-5c685655a828: failed to appendEntries to: peer="{Voter server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1 fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1}" error="failed to connect to peer: fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1"
2026-05-25T23:24:13.956Z [ERROR] server-35e32fb3-f7fb-72ca-c959-5c685655a828: failed to heartbeat to: peer=fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1 backoff time=10ms error="failed to connect to peer: fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1"
2026-05-25T23:24:13.956Z [ERROR] server-35e32fb3-f7fb-72ca-c959-5c685655a828: failed to heartbeat to: peer=4a5f8030-e902-bb7a-2531-701071cea7f4 backoff time=10ms error="failed to connect to peer: 4a5f8030-e902-bb7a-2531-701071cea7f4"
2026-05-25T23:24:13.966Z [WARN]  server-35e32fb3-f7fb-72ca-c959-5c685655a828: failed to contact: server-id=server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1 time=53.893937ms
2026-05-25T23:24:13.966Z [WARN]  server-35e32fb3-f7fb-72ca-c959-5c685655a828: failed to contact: server-id=server-4a5f8030-e902-bb7a-2531-701071cea7f4 time=53.905031ms
2026-05-25T23:24:13.966Z [WARN]  server-35e32fb3-f7fb-72ca-c959-5c685655a828: failed to contact quorum of nodes, stepping down
2026-05-25T23:24:13.966Z [INFO]  server-35e32fb3-f7fb-72ca-c959-5c685655a828: entering follower state: follower="Node at 35e32fb3-f7fb-72ca-c959-5c685655a828 [Follower]" leader-address= leader-id=
2026-05-25T23:24:13.971Z [WARN]  server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1: heartbeat timeout reached, starting election: last-leader-addr=35e32fb3-f7fb-72ca-c959-5c685655a828 last-leader-id=server-35e32fb3-f7fb-72ca-c959-5c685655a828
2026-05-25T23:24:13.971Z [ERROR] server-35e32fb3-f7fb-72ca-c959-5c685655a828: failed to appendEntries to: peer="{Voter server-4a5f8030-e902-bb7a-2531-701071cea7f4 4a5f8030-e902-bb7a-2531-701071cea7f4}" error="failed to connect to peer: 4a5f8030-e902-bb7a-2531-701071cea7f4"
2026-05-25T23:24:13.971Z [INFO]  server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1: entering candidate state: node="Node at fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1 [Candidate]" term=3
2026-05-25T23:24:13.971Z [ERROR] server-35e32fb3-f7fb-72ca-c959-5c685655a828: failed to appendEntries to: peer="{Voter server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1 fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1}" error="failed to connect to peer: fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1"
2026-05-25T23:24:13.971Z [ERROR] server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1: failed to make requestVote RPC: target="{Voter server-35e32fb3-f7fb-72ca-c959-5c685655a828 35e32fb3-f7fb-72ca-c959-5c685655a828}" error="failed to connect to peer: 35e32fb3-f7fb-72ca-c959-5c685655a828" term=3
2026-05-25T23:24:13.971Z [WARN]  server-4a5f8030-e902-bb7a-2531-701071cea7f4: rejecting vote request since we have a leader: from=fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1 leader=35e32fb3-f7fb-72ca-c959-5c685655a828 leader-id=server-35e32fb3-f7fb-72ca-c959-5c685655a828
2026-05-25T23:24:13.995Z [WARN]  server-4a5f8030-e902-bb7a-2531-701071cea7f4: heartbeat timeout reached, starting election: last-leader-addr=35e32fb3-f7fb-72ca-c959-5c685655a828 last-leader-id=server-35e32fb3-f7fb-72ca-c959-5c685655a828
2026-05-25T23:24:13.995Z [INFO]  server-4a5f8030-e902-bb7a-2531-701071cea7f4: entering candidate state: node="Node at 4a5f8030-e902-bb7a-2531-701071cea7f4 [Candidate]" term=3
2026-05-25T23:24:13.995Z [INFO]  server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1: duplicate requestVote for same term: term=3
2026-05-25T23:24:13.995Z [ERROR] server-4a5f8030-e902-bb7a-2531-701071cea7f4: failed to make requestVote RPC: target="{Voter server-35e32fb3-f7fb-72ca-c959-5c685655a828 35e32fb3-f7fb-72ca-c959-5c685655a828}" error="failed to connect to peer: 35e32fb3-f7fb-72ca-c959-5c685655a828" term=3
2026-05-25T23:24:14.020Z [INFO]  server-fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1: entering follower state: follower="Node at fd3e516d-aa6e-2dc1-c7e1-c38c241e57b1 [Follower]" leader-address= leader-id=
2026-05-25T23:24:14.020Z [WARN]  server-35e32fb3-f7fb-72ca-c959-5c685655a828: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
--- PASS: TestRaft_VoteWithNoIDNoAddr (0.50s)
=== RUN   TestRaft_runFollower_State_Transition
=== RUN   TestRaft_runFollower_State_Transition/NonVoter
2026-05-25T23:24:14.021Z [INFO]  127.0.0.1:41575 :: starting node: addr=127.0.0.1:41575
2026-05-25T23:24:14.021Z [INFO]  127.0.0.1:41575 :: initial configuration: index=0 servers=[]
2026-05-25T23:24:14.021Z [INFO]  127.0.0.1:41575 :: entering follower state: follower="Node at 127.0.0.1:41575 [Follower]" leader-address= leader-id=
2026-05-25T23:24:14.113Z [WARN]  127.0.0.1:41575 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
=== RUN   TestRaft_runFollower_State_Transition/Voter
2026-05-25T23:24:14.172Z [INFO]  127.0.0.1:39753 :: starting node: addr=127.0.0.1:39753
2026-05-25T23:24:14.172Z [INFO]  127.0.0.1:39753 :: initial configuration: index=0 servers=[]
2026-05-25T23:24:14.172Z [INFO]  127.0.0.1:39753 :: entering follower state: follower="Node at 127.0.0.1:39753 [Follower]" leader-address= leader-id=
2026-05-25T23:24:14.234Z [WARN]  127.0.0.1:39753 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
=== RUN   TestRaft_runFollower_State_Transition/Not_in_Config
2026-05-25T23:24:14.323Z [INFO]  127.0.0.1:45475 :: starting node: addr=127.0.0.1:45475
2026-05-25T23:24:14.323Z [INFO]  127.0.0.1:45475 :: initial configuration: index=0 servers=[]
2026-05-25T23:24:14.323Z [INFO]  127.0.0.1:45475 :: entering follower state: follower="Node at 127.0.0.1:45475 [Follower]" leader-address= leader-id=
2026-05-25T23:24:14.376Z [WARN]  127.0.0.1:45475 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
--- PASS: TestRaft_runFollower_State_Transition (0.45s)
    --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s)
=== RUN   TestRaft_runFollower_ReloadTimeoutConfigs
2026-05-25T23:24:14.474Z [INFO]  127.0.0.1:39401 :: starting node: addr=127.0.0.1:39401
2026-05-25T23:24:14.474Z [INFO]  127.0.0.1:39401 :: initial configuration: index=0 servers=[]
2026-05-25T23:24:14.474Z [INFO]  127.0.0.1:39401 :: entering follower state: follower="Node at 127.0.0.1:39401 [Follower]" leader-address= leader-id=
2026-05-25T23:24:14.475Z [WARN]  127.0.0.1:39401 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
--- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s)
=== RUN   TestSaturationMetric
=== RUN   TestSaturationMetric/without_smoothing
--- PASS: TestSaturationMetric (0.00s)
    --- PASS: TestSaturationMetric/without_smoothing (0.00s)
=== RUN   TestSaturationMetric_IncorrectUsage
=== RUN   TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_first
--- PASS: TestSaturationMetric_IncorrectUsage (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s)
=== RUN   TestTCPTransport_BadAddr
--- PASS: TestTCPTransport_BadAddr (0.00s)
=== RUN   TestTCPTransport_EmptyAddr
--- PASS: TestTCPTransport_EmptyAddr (0.00s)
=== RUN   TestTCPTransport_WithAdvertise
--- PASS: TestTCPTransport_WithAdvertise (0.00s)
=== RUN   TestTransport_StartStop
--- PASS: TestTransport_StartStop (0.00s)
=== RUN   TestTransport_AppendEntries
--- PASS: TestTransport_AppendEntries (0.00s)
=== RUN   TestTransport_AppendEntriesPipeline
--- PASS: TestTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestTransport_RequestVote
--- PASS: TestTransport_RequestVote (0.00s)
=== RUN   TestTransport_InstallSnapshot
--- PASS: TestTransport_InstallSnapshot (0.00s)
=== RUN   TestTransport_EncodeDecode
--- PASS: TestTransport_EncodeDecode (0.00s)
=== RUN   TestRandomTimeout
--- PASS: TestRandomTimeout (0.00s)
=== RUN   TestNewSeed
--- PASS: TestNewSeed (0.00s)
=== RUN   TestRandomTimeout_NoTime
--- PASS: TestRandomTimeout_NoTime (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
=== RUN   TestMax
--- PASS: TestMax (0.00s)
=== RUN   TestGenerateUUID
--- PASS: TestGenerateUUID (0.00s)
=== RUN   TestBackoff
--- PASS: TestBackoff (0.00s)
=== RUN   TestOverrideNotifyBool
--- PASS: TestOverrideNotifyBool (0.00s)
FAIL
FAIL	github.com/hashicorp/raft	110.286s
?   	github.com/hashicorp/raft/bench	[no test files]
FAIL
dh_auto_test: error: cd obj-x86_64-linux-gnu && go test -vet=off -v -p 42 github.com/hashicorp/raft github.com/hashicorp/raft/bench returned exit code 1
make[1]: *** [debian/rules:17: override_dh_auto_test] Error 25
make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
make: *** [debian/rules:14: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
I: copying local configuration
E: Failed autobuilding of package
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /srv/workspace/pbuilder/3759238 and its subdirectories