I: pbuilder: network access will be disabled during build
I: Current time: Fri Feb 27 21:01:07 -12 2026
I: pbuilder-time-stamp: 1772269267
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/unstable-reproducible-base.tgz]
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: policy-rc.d already exists
I: Copying source file
I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc]
I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz]
I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz]
I: Extracting source
dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: unsupported subcommand
dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying 0001-Skip-unreliable-tests.patch
I: Not using root during the build.
I: Installing the build-deps
I: user script /srv/workspace/pbuilder/3472608/tmp/hooks/D02_print_environment starting
I: set
  BUILDDIR='/build/reproducible-path'
  BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other'
  BUILDUSERNAME='pbuilder1'
  BUILD_ARCH='arm64'
  DEBIAN_FRONTEND='noninteractive'
  DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=12 '
  DISTRIBUTION='unstable'
  HOME='/root'
  HOST_ARCH='arm64'
  IFS=' 	
  '
  INVOCATION_ID='2204f66eea3a4965bd77085bfca2b5b0'
  LANG='C'
  LANGUAGE='en_US:en'
  LC_ALL='C'
  MAIL='/var/mail/root'
  OPTIND='1'
  PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games'
  PBCURRENTCOMMANDLINEOPERATION='build'
  PBUILDER_OPERATION='build'
  PBUILDER_PKGDATADIR='/usr/share/pbuilder'
  PBUILDER_PKGLIBDIR='/usr/lib/pbuilder'
  PBUILDER_SYSCONFDIR='/etc'
  PPID='3472608'
  PS1='# '
  PS2='> '
  PS4='+ '
  PWD='/'
  SHELL='/bin/bash'
  SHLVL='2'
  SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.dqTaEGxs/pbuilderrc_394B --distribution unstable --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/unstable-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.dqTaEGxs/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc'
  SUDO_GID='109'
  SUDO_UID='104'
  SUDO_USER='jenkins'
  TERM='unknown'
  TZ='/usr/share/zoneinfo/Etc/GMT+12'
  USER='root'
  _='/usr/bin/systemd-run'
  http_proxy='http://192.168.101.4:3128'
I: uname -a
  Linux codethink03-arm64 6.1.0-30-cloud-arm64 #1 SMP Debian 6.1.124-1 (2025-01-12) aarch64 GNU/Linux
I: ls -l /bin
  lrwxrwxrwx 1 root root 7 Nov 22  2024 /bin -> usr/bin
I: user script /srv/workspace/pbuilder/3472608/tmp/hooks/D02_print_environment finished
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: arm64
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 19946 files and directories currently installed.)
Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ...
Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however:
  Package debhelper-compat is not installed.
 pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however:
  Package dh-sequence-golang is not installed.
 pbuilder-satisfydepends-dummy depends on golang-any; however:
  Package golang-any is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however:
  Package golang-github-armon-go-metrics-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however:
  Package golang-github-hashicorp-go-hclog-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however:
  Package golang-github-hashicorp-go-msgpack-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however:
  Package golang-github-stretchr-testify-dev is not installed.

Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
Building tag database...
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
The following NEW packages will be installed:
  autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.23-go{a} golang-1.23-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libunistring5{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} 
The following packages are RECOMMENDED but will NOT be installed:
  curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 
0 packages upgraded, 106 newly installed, 0 to remove and 0 not upgraded.
Need to get 124 MB of archives. After unpacking 693 MB will be used.
Writing extended state information...
Get: 1 http://deb.debian.org/debian unstable/main arm64 sensible-utils all 0.0.24 [24.8 kB]
Get: 2 http://deb.debian.org/debian unstable/main arm64 libmagic-mgc arm64 1:5.45-3+b1 [314 kB]
Get: 3 http://deb.debian.org/debian unstable/main arm64 libmagic1t64 arm64 1:5.45-3+b1 [102 kB]
Get: 4 http://deb.debian.org/debian unstable/main arm64 file arm64 1:5.45-3+b1 [43.4 kB]
Get: 5 http://deb.debian.org/debian unstable/main arm64 gettext-base arm64 0.23.1-1 [241 kB]
Get: 6 http://deb.debian.org/debian unstable/main arm64 libuchardet0 arm64 0.0.8-1+b2 [69.2 kB]
Get: 7 http://deb.debian.org/debian unstable/main arm64 groff-base arm64 1.23.0-7 [1129 kB]
Get: 8 http://deb.debian.org/debian unstable/main arm64 bsdextrautils arm64 2.40.4-1 [91.6 kB]
Get: 9 http://deb.debian.org/debian unstable/main arm64 libpipeline1 arm64 1.5.8-1 [40.2 kB]
Get: 10 http://deb.debian.org/debian unstable/main arm64 man-db arm64 2.13.0-1 [1404 kB]
Get: 11 http://deb.debian.org/debian unstable/main arm64 m4 arm64 1.4.19-5 [284 kB]
Get: 12 http://deb.debian.org/debian unstable/main arm64 autoconf all 2.72-3 [493 kB]
Get: 13 http://deb.debian.org/debian unstable/main arm64 autotools-dev all 20220109.1 [51.6 kB]
Get: 14 http://deb.debian.org/debian unstable/main arm64 automake all 1:1.16.5-1.3 [823 kB]
Get: 15 http://deb.debian.org/debian unstable/main arm64 autopoint all 0.23.1-1 [770 kB]
Get: 16 http://deb.debian.org/debian unstable/main arm64 libdebhelper-perl all 13.24.1 [90.9 kB]
Get: 17 http://deb.debian.org/debian unstable/main arm64 libtool all 2.5.4-2 [539 kB]
Get: 18 http://deb.debian.org/debian unstable/main arm64 dh-autoreconf all 20 [17.1 kB]
Get: 19 http://deb.debian.org/debian unstable/main arm64 libarchive-zip-perl all 1.68-1 [104 kB]
Get: 20 http://deb.debian.org/debian unstable/main arm64 libfile-stripnondeterminism-perl all 1.14.1-1 [19.6 kB]
Get: 21 http://deb.debian.org/debian unstable/main arm64 dh-strip-nondeterminism all 1.14.1-1 [8584 B]
Get: 22 http://deb.debian.org/debian unstable/main arm64 libelf1t64 arm64 0.192-4 [189 kB]
Get: 23 http://deb.debian.org/debian unstable/main arm64 dwz arm64 0.15-1+b1 [102 kB]
Get: 24 http://deb.debian.org/debian unstable/main arm64 libunistring5 arm64 1.3-1 [449 kB]
Get: 25 http://deb.debian.org/debian unstable/main arm64 libicu72 arm64 72.1-6 [9239 kB]
Get: 26 http://deb.debian.org/debian unstable/main arm64 libxml2 arm64 2.12.7+dfsg+really2.9.14-0.2+b1 [630 kB]
Get: 27 http://deb.debian.org/debian unstable/main arm64 gettext arm64 0.23.1-1 [1610 kB]
Get: 28 http://deb.debian.org/debian unstable/main arm64 intltool-debian all 0.35.0+20060710.6 [22.9 kB]
Get: 29 http://deb.debian.org/debian unstable/main arm64 po-debconf all 1.0.21+nmu1 [248 kB]
Get: 30 http://deb.debian.org/debian unstable/main arm64 debhelper all 13.24.1 [920 kB]
Get: 31 http://deb.debian.org/debian unstable/main arm64 dh-golang all 1.62 [24.7 kB]
Get: 32 http://deb.debian.org/debian unstable/main arm64 golang-1.23-src all 1.23.5-1 [19.1 MB]
Get: 33 http://deb.debian.org/debian unstable/main arm64 golang-1.23-go arm64 1.23.5-1 [23.4 MB]
Get: 34 http://deb.debian.org/debian unstable/main arm64 golang-src all 2:1.23~2 [5088 B]
Get: 35 http://deb.debian.org/debian unstable/main arm64 golang-go arm64 2:1.23~2 [44.3 kB]
Get: 36 http://deb.debian.org/debian unstable/main arm64 golang-any arm64 2:1.23~2 [5172 B]
Get: 37 http://deb.debian.org/debian unstable/main arm64 golang-github-alecthomas-units-dev all 0.0~git20240626.61d1e34-1 [9032 B]
Get: 38 http://deb.debian.org/debian unstable/main arm64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB]
Get: 39 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB]
Get: 40 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-sys-dev all 0.22.0-1 [431 kB]
Get: 41 http://deb.debian.org/debian unstable/main arm64 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B]
Get: 42 http://deb.debian.org/debian unstable/main arm64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB]
Get: 43 http://deb.debian.org/debian unstable/main arm64 golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB]
Get: 44 http://deb.debian.org/debian unstable/main arm64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB]
Get: 45 http://deb.debian.org/debian unstable/main arm64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB]
Get: 46 http://deb.debian.org/debian unstable/main arm64 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB]
Get: 47 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB]
Get: 48 http://deb.debian.org/debian unstable/main arm64 golang-github-stretchr-testify-dev all 1.9.0-1 [74.7 kB]
Get: 49 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB]
Get: 50 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB]
Get: 51 http://deb.debian.org/debian unstable/main arm64 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB]
Get: 52 http://deb.debian.org/debian unstable/main arm64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B]
Get: 53 http://deb.debian.org/debian unstable/main arm64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB]
Get: 54 http://deb.debian.org/debian unstable/main arm64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB]
Get: 55 http://deb.debian.org/debian unstable/main arm64 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB]
Get: 56 http://deb.debian.org/debian unstable/main arm64 golang-google-protobuf-dev all 1.33.0-1 [625 kB]
Get: 57 http://deb.debian.org/debian unstable/main arm64 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB]
Get: 58 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B]
Get: 59 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB]
Get: 60 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB]
Get: 61 http://deb.debian.org/debian unstable/main arm64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB]
Get: 62 http://deb.debian.org/debian unstable/main arm64 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB]
Get: 63 http://deb.debian.org/debian unstable/main arm64 golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB]
Get: 64 http://deb.debian.org/debian unstable/main arm64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B]
Get: 65 http://deb.debian.org/debian unstable/main arm64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB]
Get: 66 http://deb.debian.org/debian unstable/main arm64 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB]
Get: 67 http://deb.debian.org/debian unstable/main arm64 golang-github-klauspost-compress-dev all 1.17.11+ds1-1 [33.3 MB]
Get: 68 http://deb.debian.org/debian unstable/main arm64 golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB]
Get: 69 http://deb.debian.org/debian unstable/main arm64 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB]
Get: 70 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB]
Get: 71 http://deb.debian.org/debian unstable/main arm64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB]
Get: 72 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-term-dev all 0.22.0-1 [14.8 kB]
Get: 73 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-text-dev all 0.16.0-1 [4003 kB]
Get: 74 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-crypto-dev all 1:0.25.0-1 [1682 kB]
Get: 75 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-net-dev all 1:0.27.0-1 [898 kB]
Get: 76 http://deb.debian.org/debian unstable/main arm64 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB]
Get: 77 http://deb.debian.org/debian unstable/main arm64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB]
Get: 78 http://deb.debian.org/debian unstable/main arm64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB]
Get: 79 http://deb.debian.org/debian unstable/main arm64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB]
Get: 80 http://deb.debian.org/debian unstable/main arm64 golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B]
Get: 81 http://deb.debian.org/debian unstable/main arm64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B]
Get: 82 http://deb.debian.org/debian unstable/main arm64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB]
Get: 83 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-oauth2-dev all 0.15.0-1 [38.0 kB]
Get: 84 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB]
Get: 85 http://deb.debian.org/debian unstable/main arm64 golang-github-prometheus-common-dev all 0.61.0-1 [120 kB]
Get: 86 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB]
Get: 87 http://deb.debian.org/debian unstable/main arm64 golang-github-prometheus-procfs-dev all 0.14.0-1 [183 kB]
Get: 88 http://deb.debian.org/debian unstable/main arm64 golang-github-prometheus-client-golang-dev all 1.20.5-1 [188 kB]
Get: 89 http://deb.debian.org/debian unstable/main arm64 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB]
Get: 90 http://deb.debian.org/debian unstable/main arm64 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB]
Get: 91 http://deb.debian.org/debian unstable/main arm64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B]
Get: 92 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB]
Get: 93 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B]
Get: 94 http://deb.debian.org/debian unstable/main arm64 libsasl2-modules-db arm64 2.1.28+dfsg1-8+b1 [20.3 kB]
Get: 95 http://deb.debian.org/debian unstable/main arm64 libsasl2-2 arm64 2.1.28+dfsg1-8+b1 [55.7 kB]
Get: 96 http://deb.debian.org/debian unstable/main arm64 libssl-dev arm64 3.4.0-2 [3234 kB]
Get: 97 http://deb.debian.org/debian unstable/main arm64 libsasl2-dev arm64 2.1.28+dfsg1-8+b1 [230 kB]
Get: 98 http://deb.debian.org/debian unstable/main arm64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB]
Get: 99 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-mod-dev all 0.19.0-1 [104 kB]
Get: 100 http://deb.debian.org/debian unstable/main arm64 golang-github-yuin-goldmark-dev all 1.7.4-1 [113 kB]
Get: 101 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-tools-dev all 1:0.23.0+ds-1 [1593 kB]
Get: 102 http://deb.debian.org/debian unstable/main arm64 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB]
Get: 103 http://deb.debian.org/debian unstable/main arm64 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB]
Get: 104 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB]
Get: 105 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB]
Get: 106 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB]
Fetched 124 MB in 1s (174 MB/s)
Preconfiguring packages ...
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 19946 files and directories currently installed.)
Preparing to unpack .../000-sensible-utils_0.0.24_all.deb ...
Unpacking sensible-utils (0.0.24) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3+b1_arm64.deb ...
Unpacking libmagic-mgc (1:5.45-3+b1) ...
Selecting previously unselected package libmagic1t64:arm64.
Preparing to unpack .../002-libmagic1t64_1%3a5.45-3+b1_arm64.deb ...
Unpacking libmagic1t64:arm64 (1:5.45-3+b1) ...
Selecting previously unselected package file.
Preparing to unpack .../003-file_1%3a5.45-3+b1_arm64.deb ...
Unpacking file (1:5.45-3+b1) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../004-gettext-base_0.23.1-1_arm64.deb ...
Unpacking gettext-base (0.23.1-1) ...
Selecting previously unselected package libuchardet0:arm64.
Preparing to unpack .../005-libuchardet0_0.0.8-1+b2_arm64.deb ...
Unpacking libuchardet0:arm64 (0.0.8-1+b2) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../006-groff-base_1.23.0-7_arm64.deb ...
Unpacking groff-base (1.23.0-7) ...
Selecting previously unselected package bsdextrautils.
Preparing to unpack .../007-bsdextrautils_2.40.4-1_arm64.deb ...
Unpacking bsdextrautils (2.40.4-1) ...
Selecting previously unselected package libpipeline1:arm64.
Preparing to unpack .../008-libpipeline1_1.5.8-1_arm64.deb ...
Unpacking libpipeline1:arm64 (1.5.8-1) ...
Selecting previously unselected package man-db.
Preparing to unpack .../009-man-db_2.13.0-1_arm64.deb ...
Unpacking man-db (2.13.0-1) ...
Selecting previously unselected package m4.
Preparing to unpack .../010-m4_1.4.19-5_arm64.deb ...
Unpacking m4 (1.4.19-5) ...
Selecting previously unselected package autoconf.
Preparing to unpack .../011-autoconf_2.72-3_all.deb ...
Unpacking autoconf (2.72-3) ...
Selecting previously unselected package autotools-dev.
Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ...
Unpacking autotools-dev (20220109.1) ...
Selecting previously unselected package automake.
Preparing to unpack .../013-automake_1%3a1.16.5-1.3_all.deb ...
Unpacking automake (1:1.16.5-1.3) ...
Selecting previously unselected package autopoint.
Preparing to unpack .../014-autopoint_0.23.1-1_all.deb ...
Unpacking autopoint (0.23.1-1) ...
Selecting previously unselected package libdebhelper-perl.
Preparing to unpack .../015-libdebhelper-perl_13.24.1_all.deb ...
Unpacking libdebhelper-perl (13.24.1) ...
Selecting previously unselected package libtool.
Preparing to unpack .../016-libtool_2.5.4-2_all.deb ...
Unpacking libtool (2.5.4-2) ...
Selecting previously unselected package dh-autoreconf.
Preparing to unpack .../017-dh-autoreconf_20_all.deb ...
Unpacking dh-autoreconf (20) ...
Selecting previously unselected package libarchive-zip-perl.
Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ...
Unpacking libarchive-zip-perl (1.68-1) ...
Selecting previously unselected package libfile-stripnondeterminism-perl.
Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.1-1_all.deb ...
Unpacking libfile-stripnondeterminism-perl (1.14.1-1) ...
Selecting previously unselected package dh-strip-nondeterminism.
Preparing to unpack .../020-dh-strip-nondeterminism_1.14.1-1_all.deb ...
Unpacking dh-strip-nondeterminism (1.14.1-1) ...
Selecting previously unselected package libelf1t64:arm64.
Preparing to unpack .../021-libelf1t64_0.192-4_arm64.deb ...
Unpacking libelf1t64:arm64 (0.192-4) ...
Selecting previously unselected package dwz.
Preparing to unpack .../022-dwz_0.15-1+b1_arm64.deb ...
Unpacking dwz (0.15-1+b1) ...
Selecting previously unselected package libunistring5:arm64.
Preparing to unpack .../023-libunistring5_1.3-1_arm64.deb ...
Unpacking libunistring5:arm64 (1.3-1) ...
Selecting previously unselected package libicu72:arm64.
Preparing to unpack .../024-libicu72_72.1-6_arm64.deb ...
Unpacking libicu72:arm64 (72.1-6) ...
Selecting previously unselected package libxml2:arm64.
Preparing to unpack .../025-libxml2_2.12.7+dfsg+really2.9.14-0.2+b1_arm64.deb ...
Unpacking libxml2:arm64 (2.12.7+dfsg+really2.9.14-0.2+b1) ...
Selecting previously unselected package gettext.
Preparing to unpack .../026-gettext_0.23.1-1_arm64.deb ...
Unpacking gettext (0.23.1-1) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../027-intltool-debian_0.35.0+20060710.6_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.6) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../028-po-debconf_1.0.21+nmu1_all.deb ...
Unpacking po-debconf (1.0.21+nmu1) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../029-debhelper_13.24.1_all.deb ...
Unpacking debhelper (13.24.1) ...
Selecting previously unselected package dh-golang.
Preparing to unpack .../030-dh-golang_1.62_all.deb ...
Unpacking dh-golang (1.62) ...
Selecting previously unselected package golang-1.23-src.
Preparing to unpack .../031-golang-1.23-src_1.23.5-1_all.deb ...
Unpacking golang-1.23-src (1.23.5-1) ...
Selecting previously unselected package golang-1.23-go.
Preparing to unpack .../032-golang-1.23-go_1.23.5-1_arm64.deb ...
Unpacking golang-1.23-go (1.23.5-1) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../033-golang-src_2%3a1.23~2_all.deb ...
Unpacking golang-src (2:1.23~2) ...
Selecting previously unselected package golang-go:arm64.
Preparing to unpack .../034-golang-go_2%3a1.23~2_arm64.deb ...
Unpacking golang-go:arm64 (2:1.23~2) ...
Selecting previously unselected package golang-any:arm64.
Preparing to unpack .../035-golang-any_2%3a1.23~2_arm64.deb ...
Unpacking golang-any:arm64 (2:1.23~2) ...
Selecting previously unselected package golang-github-alecthomas-units-dev.
Preparing to unpack .../036-golang-github-alecthomas-units-dev_0.0~git20240626.61d1e34-1_all.deb ...
Unpacking golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ...
Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev.
Preparing to unpack .../037-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ...
Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev.
Preparing to unpack .../038-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ...
Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Selecting previously unselected package golang-golang-x-sys-dev.
Preparing to unpack .../039-golang-golang-x-sys-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-sys-dev (0.22.0-1) ...
Selecting previously unselected package golang-github-mattn-go-isatty-dev.
Preparing to unpack .../040-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ...
Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Selecting previously unselected package golang-github-mattn-go-colorable-dev.
Preparing to unpack .../041-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ...
Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Selecting previously unselected package golang-github-fatih-color-dev.
Preparing to unpack .../042-golang-github-fatih-color-dev_1.17.0-1_all.deb ...
Unpacking golang-github-fatih-color-dev (1.17.0-1) ...
Selecting previously unselected package golang-github-davecgh-go-spew-dev.
Preparing to unpack .../043-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ...
Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Selecting previously unselected package golang-github-pmezard-go-difflib-dev.
Preparing to unpack .../044-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ...
Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Selecting previously unselected package golang-github-stretchr-objx-dev.
Preparing to unpack .../045-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ...
Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ...
Selecting previously unselected package golang-gopkg-yaml.v3-dev.
Preparing to unpack .../046-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ...
Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Selecting previously unselected package golang-github-stretchr-testify-dev.
Preparing to unpack .../047-golang-github-stretchr-testify-dev_1.9.0-1_all.deb ...
Unpacking golang-github-stretchr-testify-dev (1.9.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-hclog-dev.
Preparing to unpack .../048-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ...
Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev.
Preparing to unpack .../049-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ...
Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Selecting previously unselected package golang-github-pkg-errors-dev.
Preparing to unpack .../050-golang-github-pkg-errors-dev_0.9.1-3_all.deb ...
Unpacking golang-github-pkg-errors-dev (0.9.1-3) ...
Selecting previously unselected package golang-github-tv42-httpunix-dev.
Preparing to unpack .../051-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ...
Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev.
Preparing to unpack .../052-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ...
Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Selecting previously unselected package golang-github-datadog-datadog-go-dev.
Preparing to unpack .../053-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ...
Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Selecting previously unselected package golang-github-google-go-cmp-dev.
Preparing to unpack .../054-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ...
Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ...
Selecting previously unselected package golang-google-protobuf-dev.
Preparing to unpack .../055-golang-google-protobuf-dev_1.33.0-1_all.deb ...
Unpacking golang-google-protobuf-dev (1.33.0-1) ...
Selecting previously unselected package golang-github-golang-protobuf-1-5-dev.
Preparing to unpack .../056-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ...
Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Selecting previously unselected package golang-github-hashicorp-go-uuid-dev.
Preparing to unpack .../057-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ...
Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Selecting previously unselected package golang-github-hashicorp-golang-lru-dev.
Preparing to unpack .../058-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ...
Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev.
Preparing to unpack .../059-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ...
Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Selecting previously unselected package golang-github-pascaldekloe-goe-dev.
Preparing to unpack .../060-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ...
Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Selecting previously unselected package golang-github-beorn7-perks-dev.
Preparing to unpack .../061-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ...
Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ...
Selecting previously unselected package golang-github-cespare-xxhash-dev.
Preparing to unpack .../062-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ...
Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ...
Selecting previously unselected package golang-github-modern-go-concurrent-dev.
Preparing to unpack .../063-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ...
Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Selecting previously unselected package golang-github-modern-go-reflect2-dev.
Preparing to unpack .../064-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ...
Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-json-iterator-go-dev.
Preparing to unpack .../065-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ...
Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ...
Selecting previously unselected package golang-github-klauspost-compress-dev.
Preparing to unpack .../066-golang-github-klauspost-compress-dev_1.17.11+ds1-1_all.deb ...
Unpacking golang-github-klauspost-compress-dev (1.17.11+ds1-1) ...
Selecting previously unselected package golang-github-kylelemons-godebug-dev.
Preparing to unpack .../067-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ...
Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Selecting previously unselected package golang-github-prometheus-client-model-dev.
Preparing to unpack .../068-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ...
Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ...
Selecting previously unselected package golang-gopkg-yaml.v2-dev.
Preparing to unpack .../069-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ...
Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Selecting previously unselected package golang-github-jmespath-go-jmespath-dev.
Preparing to unpack .../070-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ...
Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Selecting previously unselected package golang-golang-x-term-dev.
Preparing to unpack .../071-golang-golang-x-term-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-term-dev (0.22.0-1) ...
Selecting previously unselected package golang-golang-x-text-dev.
Preparing to unpack .../072-golang-golang-x-text-dev_0.16.0-1_all.deb ...
Unpacking golang-golang-x-text-dev (0.16.0-1) ...
Selecting previously unselected package golang-golang-x-crypto-dev.
Preparing to unpack .../073-golang-golang-x-crypto-dev_1%3a0.25.0-1_all.deb ...
Unpacking golang-golang-x-crypto-dev (1:0.25.0-1) ...
Selecting previously unselected package golang-golang-x-net-dev.
Preparing to unpack .../074-golang-golang-x-net-dev_1%3a0.27.0-1_all.deb ...
Unpacking golang-golang-x-net-dev (1:0.27.0-1) ...
Selecting previously unselected package golang-github-aws-aws-sdk-go-dev.
Preparing to unpack .../075-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ...
Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Selecting previously unselected package golang-github-go-logfmt-logfmt-dev.
Preparing to unpack .../076-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ...
Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Selecting previously unselected package golang-github-go-kit-log-dev.
Preparing to unpack .../077-golang-github-go-kit-log-dev_0.2.1-1_all.deb ...
Unpacking golang-github-go-kit-log-dev (0.2.1-1) ...
Selecting previously unselected package golang-github-julienschmidt-httprouter-dev.
Preparing to unpack .../078-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ...
Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Selecting previously unselected package golang-github-munnerz-goautoneg-dev.
Preparing to unpack .../079-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ...
Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Selecting previously unselected package golang-github-jpillora-backoff-dev.
Preparing to unpack .../080-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ...
Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev.
Preparing to unpack .../081-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ...
Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Selecting previously unselected package golang-golang-x-oauth2-dev.
Preparing to unpack .../082-golang-golang-x-oauth2-dev_0.15.0-1_all.deb ...
Unpacking golang-golang-x-oauth2-dev (0.15.0-1) ...
Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev.
Preparing to unpack .../083-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ...
Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Selecting previously unselected package golang-github-prometheus-common-dev.
Preparing to unpack .../084-golang-github-prometheus-common-dev_0.61.0-1_all.deb ...
Unpacking golang-github-prometheus-common-dev (0.61.0-1) ...
Selecting previously unselected package golang-golang-x-sync-dev.
Preparing to unpack .../085-golang-golang-x-sync-dev_0.7.0-1_all.deb ...
Unpacking golang-golang-x-sync-dev (0.7.0-1) ...
Selecting previously unselected package golang-github-prometheus-procfs-dev.
Preparing to unpack .../086-golang-github-prometheus-procfs-dev_0.14.0-1_all.deb ...
Unpacking golang-github-prometheus-procfs-dev (0.14.0-1) ...
Selecting previously unselected package golang-github-prometheus-client-golang-dev.
Preparing to unpack .../087-golang-github-prometheus-client-golang-dev_1.20.5-1_all.deb ...
Unpacking golang-github-prometheus-client-golang-dev (1.20.5-1) ...
Selecting previously unselected package golang-github-armon-go-metrics-dev.
Preparing to unpack .../088-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ...
Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ...
Selecting previously unselected package golang-github-creack-pty-dev.
Preparing to unpack .../089-golang-github-creack-pty-dev_1.1.21-1_all.deb ...
Unpacking golang-github-creack-pty-dev (1.1.21-1) ...
Selecting previously unselected package golang-github-vmihailenco-tagparser-dev.
Preparing to unpack .../090-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ...
Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev.
Preparing to unpack .../091-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ...
Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Selecting previously unselected package golang-gopkg-tomb.v2-dev.
Preparing to unpack .../092-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ...
Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Selecting previously unselected package libsasl2-modules-db:arm64.
Preparing to unpack .../093-libsasl2-modules-db_2.1.28+dfsg1-8+b1_arm64.deb ...
Unpacking libsasl2-modules-db:arm64 (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package libsasl2-2:arm64.
Preparing to unpack .../094-libsasl2-2_2.1.28+dfsg1-8+b1_arm64.deb ...
Unpacking libsasl2-2:arm64 (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package libssl-dev:arm64.
Preparing to unpack .../095-libssl-dev_3.4.0-2_arm64.deb ...
Unpacking libssl-dev:arm64 (3.4.0-2) ...
Selecting previously unselected package libsasl2-dev.
Preparing to unpack .../096-libsasl2-dev_2.1.28+dfsg1-8+b1_arm64.deb ...
Unpacking libsasl2-dev (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package golang-github-kr-text-dev.
Preparing to unpack .../097-golang-github-kr-text-dev_0.2.0-2_all.deb ...
Unpacking golang-github-kr-text-dev (0.2.0-2) ...
Selecting previously unselected package golang-golang-x-mod-dev.
Preparing to unpack .../098-golang-golang-x-mod-dev_0.19.0-1_all.deb ...
Unpacking golang-golang-x-mod-dev (0.19.0-1) ...
Selecting previously unselected package golang-github-yuin-goldmark-dev.
Preparing to unpack .../099-golang-github-yuin-goldmark-dev_1.7.4-1_all.deb ...
Unpacking golang-github-yuin-goldmark-dev (1.7.4-1) ...
Selecting previously unselected package golang-golang-x-tools-dev.
Preparing to unpack .../100-golang-golang-x-tools-dev_1%3a0.23.0+ds-1_all.deb ...
Unpacking golang-golang-x-tools-dev (1:0.23.0+ds-1) ...
Selecting previously unselected package golang-github-rogpeppe-go-internal-dev.
Preparing to unpack .../101-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ...
Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Selecting previously unselected package golang-github-kr-pretty-dev.
Preparing to unpack .../102-golang-github-kr-pretty-dev_0.3.1-1_all.deb ...
Unpacking golang-github-kr-pretty-dev (0.3.1-1) ...
Selecting previously unselected package golang-gopkg-check.v1-dev.
Preparing to unpack .../103-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ...
Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Selecting previously unselected package golang-gopkg-mgo.v2-dev.
Preparing to unpack .../104-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ...
Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev.
Preparing to unpack .../105-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ...
Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up libpipeline1:arm64 (1.5.8-1) ...
Setting up golang-github-google-go-cmp-dev (0.6.0-1) ...
Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Setting up libicu72:arm64 (72.1-6) ...
Setting up bsdextrautils (2.40.4-1) ...
Setting up golang-github-creack-pty-dev (1.1.21-1) ...
Setting up golang-github-stretchr-objx-dev (0.5.2-1) ...
Setting up libmagic-mgc (1:5.45-3+b1) ...
Setting up golang-github-pkg-errors-dev (0.9.1-3) ...
Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Setting up libarchive-zip-perl (1.68-1) ...
Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Setting up libdebhelper-perl (13.24.1) ...
Setting up golang-golang-x-sys-dev (0.22.0-1) ...
Setting up libmagic1t64:arm64 (1:5.45-3+b1) ...
Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Setting up gettext-base (0.23.1-1) ...
Setting up m4 (1.4.19-5) ...
Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Setting up golang-golang-x-term-dev (0.22.0-1) ...
Setting up file (1:5.45-3+b1) ...
Setting up golang-github-beorn7-perks-dev (1.0.1-1) ...
Setting up libelf1t64:arm64 (0.192-4) ...
Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Setting up libsasl2-modules-db:arm64 (2.1.28+dfsg1-8+b1) ...
Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ...
Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Setting up autotools-dev (20220109.1) ...
Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Setting up golang-google-protobuf-dev (1.33.0-1) ...
Setting up golang-github-kr-text-dev (0.2.0-2) ...
Setting up golang-github-stretchr-testify-dev (1.9.0-1) ...
Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Setting up golang-github-yuin-goldmark-dev (1.7.4-1) ...
Setting up libunistring5:arm64 (1.3-1) ...
Setting up libssl-dev:arm64 (3.4.0-2) ...
Setting up golang-golang-x-sync-dev (0.7.0-1) ...
Setting up golang-golang-x-mod-dev (0.19.0-1) ...
Setting up golang-1.23-src (1.23.5-1) ...
Setting up autopoint (0.23.1-1) ...
Setting up golang-github-klauspost-compress-dev (1.17.11+ds1-1) ...
Setting up libsasl2-2:arm64 (2.1.28+dfsg1-8+b1) ...
Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Setting up autoconf (2.72-3) ...
Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Setting up dwz (0.15-1+b1) ...
Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Setting up sensible-utils (0.0.24) ...
Setting up golang-golang-x-text-dev (0.16.0-1) ...
Setting up libuchardet0:arm64 (0.0.8-1+b2) ...
Setting up golang-github-prometheus-procfs-dev (0.14.0-1) ...
Setting up golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ...
Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Setting up golang-golang-x-crypto-dev (1:0.25.0-1) ...
Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ...
Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Setting up golang-golang-x-oauth2-dev (0.15.0-1) ...
Setting up golang-src (2:1.23~2) ...
Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Setting up libxml2:arm64 (2.12.7+dfsg+really2.9.14-0.2+b1) ...
Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Setting up automake (1:1.16.5-1.3) ...
update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode
Setting up libfile-stripnondeterminism-perl (1.14.1-1) ...
Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Setting up golang-github-go-kit-log-dev (0.2.1-1) ...
Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Setting up gettext (0.23.1-1) ...
Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Setting up libtool (2.5.4-2) ...
Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Setting up golang-github-json-iterator-go-dev (1.1.12-2) ...
Setting up intltool-debian (0.35.0+20060710.6) ...
Setting up dh-autoreconf (20) ...
Setting up golang-1.23-go (1.23.5-1) ...
Setting up libsasl2-dev (2.1.28+dfsg1-8+b1) ...
Setting up golang-golang-x-net-dev (1:0.27.0-1) ...
Setting up dh-strip-nondeterminism (1.14.1-1) ...
Setting up groff-base (1.23.0-7) ...
Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Setting up golang-github-fatih-color-dev (1.17.0-1) ...
Setting up golang-go:arm64 (2:1.23~2) ...
Setting up po-debconf (1.0.21+nmu1) ...
Setting up golang-golang-x-tools-dev (1:0.23.0+ds-1) ...
Setting up man-db (2.13.0-1) ...
Not building database; man-db/auto-update is not 'true'.
Setting up golang-any:arm64 (2:1.23~2) ...
Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Setting up golang-github-kr-pretty-dev (0.3.1-1) ...
Setting up debhelper (13.24.1) ...
Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Setting up dh-golang (1.62) ...
Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Setting up golang-github-prometheus-common-dev (0.61.0-1) ...
Setting up golang-github-prometheus-client-golang-dev (1.20.5-1) ...
Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ...
Processing triggers for libc-bin (2.40-6) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Building tag database...
 -> Finished parsing the build-deps
I: Building the package
I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S  > ../golang-github-hashicorp-raft_1.5.0-1_source.changes
dpkg-buildpackage: info: source package golang-github-hashicorp-raft
dpkg-buildpackage: info: source version 1.5.0-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Daniel Swarbrick <dswarbrick@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture arm64
 debian/rules clean
dh clean --buildsystem=golang
   dh_auto_clean -O--buildsystem=golang
   dh_autoreconf_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules binary
dh binary --buildsystem=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-aarch64-linux-gnu && go install -trimpath -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench
internal/goos
internal/goarch
internal/cpu
internal/byteorder
internal/coverage/rtcov
internal/profilerecord
internal/godebugs
internal/unsafeheader
internal/goexperiment
internal/runtime/syscall
internal/race
runtime/internal/math
sync/atomic
internal/abi
runtime/internal/sys
unicode/utf8
unicode
container/list
internal/asan
internal/chacha8rand
internal/itoa
internal/msan
crypto/internal/alias
math/bits
crypto/subtle
internal/runtime/atomic
crypto/internal/boring/sig
cmp
encoding
unicode/utf16
internal/bytealg
log/internal
internal/nettrace
vendor/golang.org/x/crypto/cryptobyte/asn1
vendor/golang.org/x/crypto/internal/alias
math
internal/runtime/exithook
internal/stringslite
runtime
internal/reflectlite
iter
internal/weak
sync
slices
maps
internal/bisect
internal/testlog
internal/singleflight
runtime/cgo
errors
sort
internal/godebug
internal/oserror
vendor/golang.org/x/net/dns/dnsmessage
io
path
github.com/hashicorp/golang-lru/simplelru
strconv
syscall
math/rand/v2
crypto/internal/nistec/fiat
crypto/internal/edwards25519/field
bytes
math/rand
strings
hash
crypto/internal/randutil
internal/concurrent
hash/crc32
internal/saferio
crypto
reflect
unique
crypto/md5
crypto/internal/edwards25519
crypto/rc4
crypto/cipher
vendor/golang.org/x/text/transform
hash/crc64
bufio
github.com/hashicorp/go-immutable-radix
net/http/internal/ascii
net/netip
regexp/syntax
crypto/internal/boring
crypto/des
html
crypto/aes
crypto/sha512
crypto/hmac
crypto/sha1
crypto/sha256
vendor/golang.org/x/crypto/hkdf
time
internal/syscall/execenv
internal/syscall/unix
regexp
context
internal/poll
io/fs
internal/fmtsort
encoding/binary
internal/filepathlite
embed
crypto/internal/nistec
os
encoding/base64
vendor/golang.org/x/crypto/chacha20
vendor/golang.org/x/crypto/internal/poly1305
golang.org/x/sys/unix
encoding/pem
vendor/golang.org/x/crypto/chacha20poly1305
crypto/ecdh
fmt
internal/sysinfo
os/signal
path/filepath
vendor/golang.org/x/sys/cpu
io/ioutil
net
vendor/golang.org/x/crypto/sha3
log
encoding/json
net/url
math/big
encoding/hex
vendor/golang.org/x/text/unicode/norm
compress/flate
vendor/golang.org/x/net/http2/hpack
mime/quotedprintable
mime
net/http/internal
encoding/gob
vendor/golang.org/x/text/unicode/bidi
go/token
text/template/parse
flag
compress/gzip
runtime/debug
runtime/trace
vendor/golang.org/x/text/secure/bidirule
vendor/golang.org/x/net/idna
testing
text/template
github.com/mattn/go-isatty
crypto/rand
crypto/elliptic
crypto/internal/bigmod
encoding/asn1
crypto/internal/boring/bbig
crypto/dsa
github.com/mattn/go-colorable
crypto/ed25519
crypto/internal/hpke
crypto/internal/mlkem768
github.com/fatih/color
crypto/rsa
github.com/hashicorp/go-hclog
vendor/golang.org/x/crypto/cryptobyte
crypto/x509/pkix
html/template
crypto/ecdsa
crypto/x509
net/textproto
vendor/golang.org/x/net/http/httpproxy
vendor/golang.org/x/net/http/httpguts
mime/multipart
crypto/tls
net/http/httptrace
net/http
github.com/armon/go-metrics
net/rpc
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/raft
github.com/hashicorp/raft/bench
   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
dh_auto_test
	cd obj-aarch64-linux-gnu && go test -vet=off -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench
?   	github.com/hashicorp/raft/bench	[no test files]
=== RUN   TestCommitment_setVoters
--- PASS: TestCommitment_setVoters (0.00s)
=== RUN   TestCommitment_match_max
--- PASS: TestCommitment_match_max (0.00s)
=== RUN   TestCommitment_match_nonVoting
--- PASS: TestCommitment_match_nonVoting (0.00s)
=== RUN   TestCommitment_recalculate
--- PASS: TestCommitment_recalculate (0.00s)
=== RUN   TestCommitment_recalculate_startIndex
--- PASS: TestCommitment_recalculate_startIndex (0.00s)
=== RUN   TestCommitment_noVoterSanity
--- PASS: TestCommitment_noVoterSanity (0.00s)
=== RUN   TestCommitment_singleVoter
--- PASS: TestCommitment_singleVoter (0.00s)
=== RUN   TestConfiguration_Configuration_Clone
--- PASS: TestConfiguration_Configuration_Clone (0.00s)
=== RUN   TestConfiguration_configurations_Clone
--- PASS: TestConfiguration_configurations_Clone (0.00s)
=== RUN   TestConfiguration_hasVote
--- PASS: TestConfiguration_hasVote (0.00s)
=== RUN   TestConfiguration_checkConfiguration
--- PASS: TestConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_nextConfiguration_table
--- PASS: TestConfiguration_nextConfiguration_table (0.00s)
=== RUN   TestConfiguration_nextConfiguration_prevIndex
--- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
=== RUN   TestConfiguration_nextConfiguration_checkConfiguration
--- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_encodeDecodePeers
--- PASS: TestConfiguration_encodeDecodePeers (0.00s)
=== RUN   TestConfiguration_encodeDecodeConfiguration
--- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
=== RUN   TestDiscardSnapshotStoreImpl
--- PASS: TestDiscardSnapshotStoreImpl (0.00s)
=== RUN   TestDiscardSnapshotSinkImpl
--- PASS: TestDiscardSnapshotSinkImpl (0.00s)
=== RUN   TestFileSnapshotStoreImpl
--- PASS: TestFileSnapshotStoreImpl (0.00s)
=== RUN   TestFileSnapshotSinkImpl
--- PASS: TestFileSnapshotSinkImpl (0.00s)
=== RUN   TestFileSS_CreateSnapshotMissingParentDir
2026-02-28T09:02:02.042Z [INFO]  creating new snapshot: path=/tmp/raft207605535/raft1892673906/snapshots/3-10-1772269322042.tmp
--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s)
=== RUN   TestFileSS_CreateSnapshot
2026-02-28T09:02:02.043Z [INFO]  creating new snapshot: path=/tmp/raft306078197/snapshots/3-10-1772269322043.tmp
2026-02-28T09:02:02.043Z [WARN]  found temporary snapshot: name=3-10-1772269322043.tmp
--- PASS: TestFileSS_CreateSnapshot (0.00s)
=== RUN   TestFileSS_CancelSnapshot
2026-02-28T09:02:02.044Z [INFO]  creating new snapshot: path=/tmp/raft1881047195/snapshots/3-10-1772269322044.tmp
--- PASS: TestFileSS_CancelSnapshot (0.00s)
=== RUN   TestFileSS_Retention
2026-02-28T09:02:02.044Z [INFO]  creating new snapshot: path=/tmp/raft4200889094/snapshots/3-10-1772269322044.tmp
2026-02-28T09:02:02.045Z [INFO]  creating new snapshot: path=/tmp/raft4200889094/snapshots/3-11-1772269322045.tmp
2026-02-28T09:02:02.045Z [INFO]  creating new snapshot: path=/tmp/raft4200889094/snapshots/3-12-1772269322045.tmp
2026-02-28T09:02:02.045Z [INFO]  reaping snapshot: path=/tmp/raft4200889094/snapshots/3-10-1772269322044
2026-02-28T09:02:02.045Z [INFO]  creating new snapshot: path=/tmp/raft4200889094/snapshots/3-13-1772269322045.tmp
2026-02-28T09:02:02.045Z [INFO]  reaping snapshot: path=/tmp/raft4200889094/snapshots/3-11-1772269322045
2026-02-28T09:02:02.045Z [INFO]  creating new snapshot: path=/tmp/raft4200889094/snapshots/3-14-1772269322045.tmp
2026-02-28T09:02:02.046Z [INFO]  reaping snapshot: path=/tmp/raft4200889094/snapshots/3-12-1772269322045
--- PASS: TestFileSS_Retention (0.00s)
=== RUN   TestFileSS_BadPerm
--- PASS: TestFileSS_BadPerm (0.00s)
=== RUN   TestFileSS_MissingParentDir
--- PASS: TestFileSS_MissingParentDir (0.00s)
=== RUN   TestFileSS_Ordering
2026-02-28T09:02:02.047Z [INFO]  creating new snapshot: path=/tmp/raft1076601396/snapshots/5-130350-1772269322047.tmp
2026-02-28T09:02:02.047Z [INFO]  creating new snapshot: path=/tmp/raft1076601396/snapshots/36-204917-1772269322047.tmp
--- PASS: TestFileSS_Ordering (0.00s)
=== RUN   TestDeferFutureSuccess
--- PASS: TestDeferFutureSuccess (0.00s)
=== RUN   TestDeferFutureError
--- PASS: TestDeferFutureError (0.00s)
=== RUN   TestDeferFutureConcurrent
--- PASS: TestDeferFutureConcurrent (0.00s)
=== RUN   TestInmemSnapshotStoreImpl
--- PASS: TestInmemSnapshotStoreImpl (0.00s)
=== RUN   TestInmemSnapshotSinkImpl
--- PASS: TestInmemSnapshotSinkImpl (0.00s)
=== RUN   TestInmemSS_CreateSnapshot
--- PASS: TestInmemSS_CreateSnapshot (0.00s)
=== RUN   TestInmemSS_OpenSnapshotTwice
--- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
=== RUN   TestInmemTransportImpl
--- PASS: TestInmemTransportImpl (0.00s)
=== RUN   TestInmemTransportWriteTimeout
--- PASS: TestInmemTransportWriteTimeout (0.01s)
=== RUN   TestRaft_Integ
--- SKIP: TestRaft_Integ (0.00s)
=== RUN   TestRaft_RestartFollower_LongInitialHeartbeat
--- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s)
=== RUN   TestLogCache
--- PASS: TestLogCache (0.00s)
=== RUN   TestLogCacheWithBackendStoreError
--- PASS: TestLogCacheWithBackendStoreError (0.00s)
=== RUN   TestOldestLog
=== RUN   TestOldestLog/empty_logs
=== RUN   TestOldestLog/simple_case
--- PASS: TestOldestLog (0.00s)
    --- PASS: TestOldestLog/empty_logs (0.00s)
    --- PASS: TestOldestLog/simple_case (0.00s)
=== RUN   TestEmitsLogStoreMetrics
--- PASS: TestEmitsLogStoreMetrics (0.01s)
=== RUN   TestNetworkTransport_CloseStreams
--- PASS: TestNetworkTransport_CloseStreams (0.00s)
=== RUN   TestNetworkTransport_StartStop
--- PASS: TestNetworkTransport_StartStop (0.00s)
=== RUN   TestNetworkTransport_Heartbeat_FastPath
--- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s)
=== RUN   TestNetworkTransport_AppendEntries
--- PASS: TestNetworkTransport_AppendEntries (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline
--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_CloseStreams
--- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.02s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0
2026-02-28T09:02:02.149Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2
2026-02-28T09:02:02.201Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10
2026-02-28T09:02:02.254Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s)
=== RUN   TestNetworkTransport_RequestVote
--- PASS: TestNetworkTransport_RequestVote (0.00s)
=== RUN   TestNetworkTransport_InstallSnapshot
--- PASS: TestNetworkTransport_InstallSnapshot (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode
--- PASS: TestNetworkTransport_EncodeDecode (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode_AddressProvider
--- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
=== RUN   TestNetworkTransport_PooledConn
--- PASS: TestNetworkTransport_PooledConn (0.00s)
=== RUN   TestNetworkTransport_ListenBackoff
    net_transport_test.go:839: countingWriter: 2026-02-28T09:02:02.261Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-02-28T09:02:02.266Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-02-28T09:02:02.277Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-02-28T09:02:02.298Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-02-28T09:02:02.339Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-02-28T09:02:02.420Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-02-28T09:02:02.580Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-02-28T09:02:02.901Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-02-28T09:02:03.542Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-02-28T09:02:04.545Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-02-28T09:02:05.547Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
--- PASS: TestNetworkTransport_ListenBackoff (4.00s)
=== RUN   TestPeersJSON_BadConfiguration
--- PASS: TestPeersJSON_BadConfiguration (0.00s)
=== RUN   TestPeersJSON_ReadPeersJSON
--- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
=== RUN   TestPeersJSON_ReadConfigJSON
--- PASS: TestPeersJSON_ReadConfigJSON (0.00s)
=== RUN   TestRaft_StartStop
2026-02-28T09:02:06.263Z [INFO]  server-9f837117-2ea9-5df0-ac18-1efea45fc556: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f837117-2ea9-5df0-ac18-1efea45fc556 Address:9f837117-2ea9-5df0-ac18-1efea45fc556}]"
--- PASS: TestRaft_StartStop (0.00s)
=== RUN   TestRaft_AfterShutdown
2026-02-28T09:02:06.264Z [INFO]  server-2efc7fc7-77d4-b64b-a5b4-63db52d9c37e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2efc7fc7-77d4-b64b-a5b4-63db52d9c37e Address:2efc7fc7-77d4-b64b-a5b4-63db52d9c37e}]"
--- PASS: TestRaft_AfterShutdown (0.00s)
=== RUN   TestRaft_LiveBootstrap
2026-02-28T09:02:06.265Z [INFO]  server-a62e4839-5efb-2431-6f0c-53e2baaa0567: initial configuration: index=0 servers=[]
2026-02-28T09:02:06.265Z [INFO]  server-917232f1-25d6-1dbe-47a1-7462b25b1838: initial configuration: index=0 servers=[]
2026-02-28T09:02:06.265Z [INFO]  server-e3e02323-156d-9c63-b7e8-ad6abe6ba6ec: initial configuration: index=0 servers=[]
2026-02-28T09:02:06.265Z [INFO]  server-e3e02323-156d-9c63-b7e8-ad6abe6ba6ec: entering follower state: follower="Node at e3e02323-156d-9c63-b7e8-ad6abe6ba6ec [Follower]" leader-address= leader-id=
2026-02-28T09:02:06.265Z [INFO]  server-917232f1-25d6-1dbe-47a1-7462b25b1838: entering follower state: follower="Node at 917232f1-25d6-1dbe-47a1-7462b25b1838 [Follower]" leader-address= leader-id=
2026-02-28T09:02:06.265Z [INFO]  server-a62e4839-5efb-2431-6f0c-53e2baaa0567: entering follower state: follower="Node at a62e4839-5efb-2431-6f0c-53e2baaa0567 [Follower]" leader-address= leader-id=
2026-02-28T09:02:06.265Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:06.351Z [WARN]  server-a62e4839-5efb-2431-6f0c-53e2baaa0567: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:06.351Z [INFO]  server-a62e4839-5efb-2431-6f0c-53e2baaa0567: entering candidate state: node="Node at a62e4839-5efb-2431-6f0c-53e2baaa0567 [Candidate]" term=2
2026-02-28T09:02:06.351Z [WARN]  server-e3e02323-156d-9c63-b7e8-ad6abe6ba6ec: no known peers, aborting election
2026-02-28T09:02:06.351Z [INFO]  server-a62e4839-5efb-2431-6f0c-53e2baaa0567: election won: term=2 tally=2
2026-02-28T09:02:06.351Z [INFO]  server-a62e4839-5efb-2431-6f0c-53e2baaa0567: entering leader state: leader="Node at a62e4839-5efb-2431-6f0c-53e2baaa0567 [Leader]"
2026-02-28T09:02:06.351Z [INFO]  server-a62e4839-5efb-2431-6f0c-53e2baaa0567: added peer, starting replication: peer=server-917232f1-25d6-1dbe-47a1-7462b25b1838
2026-02-28T09:02:06.351Z [INFO]  server-a62e4839-5efb-2431-6f0c-53e2baaa0567: added peer, starting replication: peer=server-e3e02323-156d-9c63-b7e8-ad6abe6ba6ec
2026-02-28T09:02:06.351Z [WARN]  server-e3e02323-156d-9c63-b7e8-ad6abe6ba6ec: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2026-02-28T09:02:06.351Z [WARN]  server-917232f1-25d6-1dbe-47a1-7462b25b1838: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2026-02-28T09:02:06.351Z [WARN]  server-a62e4839-5efb-2431-6f0c-53e2baaa0567: appendEntries rejected, sending older logs: peer="{Voter server-e3e02323-156d-9c63-b7e8-ad6abe6ba6ec e3e02323-156d-9c63-b7e8-ad6abe6ba6ec}" next=1
2026-02-28T09:02:06.352Z [WARN]  server-a62e4839-5efb-2431-6f0c-53e2baaa0567: appendEntries rejected, sending older logs: peer="{Voter server-917232f1-25d6-1dbe-47a1-7462b25b1838 917232f1-25d6-1dbe-47a1-7462b25b1838}" next=1
2026-02-28T09:02:06.352Z [INFO]  server-a62e4839-5efb-2431-6f0c-53e2baaa0567: pipelining replication: peer="{Voter server-e3e02323-156d-9c63-b7e8-ad6abe6ba6ec e3e02323-156d-9c63-b7e8-ad6abe6ba6ec}"
2026-02-28T09:02:06.352Z [INFO]  server-a62e4839-5efb-2431-6f0c-53e2baaa0567: pipelining replication: peer="{Voter server-917232f1-25d6-1dbe-47a1-7462b25b1838 917232f1-25d6-1dbe-47a1-7462b25b1838}"
2026-02-28T09:02:06.457Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:06.351835622 +0000 UTC m=+4.315632161 (2 nodes), 86.36568ms from start of poll, 86.89772ms from cluster start. Timeout at 2026-02-28 09:02:06.456838982 +0000 UTC m=+4.420635561, 105.0034ms after stability
2026-02-28T09:02:06.458Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:06.563Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:06.458034302 +0000 UTC m=+4.421830881 (1 nodes), 880ns from start of poll, 193.09644ms from cluster start. Timeout at 2026-02-28 09:02:06.563038542 +0000 UTC m=+4.526835081, 105.0042ms after stability
2026-02-28T09:02:06.574Z [INFO]  server-a62e4839-5efb-2431-6f0c-53e2baaa0567: aborting pipeline replication: peer="{Voter server-e3e02323-156d-9c63-b7e8-ad6abe6ba6ec e3e02323-156d-9c63-b7e8-ad6abe6ba6ec}"
2026-02-28T09:02:06.574Z [INFO]  server-a62e4839-5efb-2431-6f0c-53e2baaa0567: aborting pipeline replication: peer="{Voter server-917232f1-25d6-1dbe-47a1-7462b25b1838 917232f1-25d6-1dbe-47a1-7462b25b1838}"
2026-02-28T09:02:07.074Z [ERROR] server-a62e4839-5efb-2431-6f0c-53e2baaa0567: failed to heartbeat to: peer=e3e02323-156d-9c63-b7e8-ad6abe6ba6ec backoff time=10ms error="command timed out"
--- PASS: TestRaft_LiveBootstrap (0.81s)
=== RUN   TestRaft_LiveBootstrap_From_NonVoter
2026-02-28T09:02:07.075Z [INFO]  server-7c109bb3-c5ae-cd5b-6011-0738bff70a3f: initial configuration: index=0 servers=[]
2026-02-28T09:02:07.075Z [INFO]  server-e9a420df-771c-f696-6038-01d2139d7e72: initial configuration: index=0 servers=[]
2026-02-28T09:02:07.075Z [INFO]  server-7c109bb3-c5ae-cd5b-6011-0738bff70a3f: entering follower state: follower="Node at 7c109bb3-c5ae-cd5b-6011-0738bff70a3f [Follower]" leader-address= leader-id=
2026-02-28T09:02:07.075Z [INFO]  server-e9a420df-771c-f696-6038-01d2139d7e72: entering follower state: follower="Node at e9a420df-771c-f696-6038-01d2139d7e72 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s)
=== RUN   TestRaft_RecoverCluster_NoState
2026-02-28T09:02:07.076Z [INFO]  server-748dc8c9-ce56-3e8c-64f7-5ab7f728860b: initial configuration: index=0 servers=[]
--- PASS: TestRaft_RecoverCluster_NoState (0.00s)
=== RUN   TestRaft_RecoverCluster
=== RUN   TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs
2026-02-28T09:02:07.077Z [INFO]  server-6b9327b5-61f5-e5f4-fea8-9293adb8a416: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b9327b5-61f5-e5f4-fea8-9293adb8a416 Address:6b9327b5-61f5-e5f4-fea8-9293adb8a416} {Suffrage:Voter ID:server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177 Address:cc0c5407-49fc-a2ac-fe7f-3ef17fc74177} {Suffrage:Voter ID:server-1af307a3-d89f-9f70-0eba-be35aa6e57d3 Address:1af307a3-d89f-9f70-0eba-be35aa6e57d3}]"
2026-02-28T09:02:07.077Z [INFO]  server-6b9327b5-61f5-e5f4-fea8-9293adb8a416: entering follower state: follower="Node at 6b9327b5-61f5-e5f4-fea8-9293adb8a416 [Follower]" leader-address= leader-id=
2026-02-28T09:02:07.077Z [INFO]  server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b9327b5-61f5-e5f4-fea8-9293adb8a416 Address:6b9327b5-61f5-e5f4-fea8-9293adb8a416} {Suffrage:Voter ID:server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177 Address:cc0c5407-49fc-a2ac-fe7f-3ef17fc74177} {Suffrage:Voter ID:server-1af307a3-d89f-9f70-0eba-be35aa6e57d3 Address:1af307a3-d89f-9f70-0eba-be35aa6e57d3}]"
2026-02-28T09:02:07.077Z [INFO]  server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177: entering follower state: follower="Node at cc0c5407-49fc-a2ac-fe7f-3ef17fc74177 [Follower]" leader-address= leader-id=
2026-02-28T09:02:07.077Z [INFO]  server-1af307a3-d89f-9f70-0eba-be35aa6e57d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b9327b5-61f5-e5f4-fea8-9293adb8a416 Address:6b9327b5-61f5-e5f4-fea8-9293adb8a416} {Suffrage:Voter ID:server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177 Address:cc0c5407-49fc-a2ac-fe7f-3ef17fc74177} {Suffrage:Voter ID:server-1af307a3-d89f-9f70-0eba-be35aa6e57d3 Address:1af307a3-d89f-9f70-0eba-be35aa6e57d3}]"
2026-02-28T09:02:07.077Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:07.077Z [INFO]  server-1af307a3-d89f-9f70-0eba-be35aa6e57d3: entering follower state: follower="Node at 1af307a3-d89f-9f70-0eba-be35aa6e57d3 [Follower]" leader-address= leader-id=
2026-02-28T09:02:07.130Z [WARN]  server-6b9327b5-61f5-e5f4-fea8-9293adb8a416: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:07.130Z [INFO]  server-6b9327b5-61f5-e5f4-fea8-9293adb8a416: entering candidate state: node="Node at 6b9327b5-61f5-e5f4-fea8-9293adb8a416 [Candidate]" term=2
2026-02-28T09:02:07.130Z [INFO]  server-6b9327b5-61f5-e5f4-fea8-9293adb8a416: election won: term=2 tally=2
2026-02-28T09:02:07.130Z [INFO]  server-6b9327b5-61f5-e5f4-fea8-9293adb8a416: entering leader state: leader="Node at 6b9327b5-61f5-e5f4-fea8-9293adb8a416 [Leader]"
2026-02-28T09:02:07.130Z [INFO]  server-6b9327b5-61f5-e5f4-fea8-9293adb8a416: added peer, starting replication: peer=server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177
2026-02-28T09:02:07.130Z [INFO]  server-6b9327b5-61f5-e5f4-fea8-9293adb8a416: added peer, starting replication: peer=server-1af307a3-d89f-9f70-0eba-be35aa6e57d3
2026-02-28T09:02:07.130Z [INFO]  server-6b9327b5-61f5-e5f4-fea8-9293adb8a416: pipelining replication: peer="{Voter server-1af307a3-d89f-9f70-0eba-be35aa6e57d3 1af307a3-d89f-9f70-0eba-be35aa6e57d3}"
2026-02-28T09:02:07.131Z [INFO]  server-6b9327b5-61f5-e5f4-fea8-9293adb8a416: pipelining replication: peer="{Voter server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177 cc0c5407-49fc-a2ac-fe7f-3ef17fc74177}"
2026-02-28T09:02:07.236Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:07.130931062 +0000 UTC m=+5.094727601 (1 nodes), 53.2474ms from start of poll, 53.73036ms from cluster start. Timeout at 2026-02-28 09:02:07.235935982 +0000 UTC m=+5.199732521, 105.00492ms after stability
2026-02-28T09:02:07.236Z [INFO]  server-6b9327b5-61f5-e5f4-fea8-9293adb8a416: aborting pipeline replication: peer="{Voter server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177 cc0c5407-49fc-a2ac-fe7f-3ef17fc74177}"
2026-02-28T09:02:07.236Z [INFO]  server-6b9327b5-61f5-e5f4-fea8-9293adb8a416: aborting pipeline replication: peer="{Voter server-1af307a3-d89f-9f70-0eba-be35aa6e57d3 1af307a3-d89f-9f70-0eba-be35aa6e57d3}"
2026-02-28T09:02:07.236Z [INFO]  creating new snapshot: path=/tmp/raft1031754795/snapshots/2-2-1772269327236.tmp
2026-02-28T09:02:07.237Z [INFO]  server-6b9327b5-61f5-e5f4-fea8-9293adb8a416: starting restore from snapshot: id=2-2-1772269327236 last-index=2 last-term=2 size-in-bytes=1
2026-02-28T09:02:07.237Z [INFO]  server-6b9327b5-61f5-e5f4-fea8-9293adb8a416: snapshot restore progress: id=2-2-1772269327236 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2026-02-28T09:02:07.237Z [INFO]  server-6b9327b5-61f5-e5f4-fea8-9293adb8a416: restored from snapshot: id=2-2-1772269327236 last-index=2 last-term=2 size-in-bytes=1
2026-02-28T09:02:07.237Z [INFO]  server-6b9327b5-61f5-e5f4-fea8-9293adb8a416: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b9327b5-61f5-e5f4-fea8-9293adb8a416 Address:6b9327b5-61f5-e5f4-fea8-9293adb8a416} {Suffrage:Voter ID:server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177 Address:cc0c5407-49fc-a2ac-fe7f-3ef17fc74177} {Suffrage:Voter ID:server-1af307a3-d89f-9f70-0eba-be35aa6e57d3 Address:1af307a3-d89f-9f70-0eba-be35aa6e57d3}]"
2026-02-28T09:02:07.237Z [INFO]  creating new snapshot: path=/tmp/raft1250705780/snapshots/2-2-1772269327237.tmp
2026-02-28T09:02:07.237Z [INFO]  server-6b9327b5-61f5-e5f4-fea8-9293adb8a416: entering follower state: follower="Node at 6b9327b5-61f5-e5f4-fea8-9293adb8a416 [Follower]" leader-address= leader-id=
2026-02-28T09:02:07.238Z [INFO]  server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177: starting restore from snapshot: id=2-2-1772269327237 last-index=2 last-term=2 size-in-bytes=1
2026-02-28T09:02:07.238Z [INFO]  server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177: snapshot restore progress: id=2-2-1772269327237 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2026-02-28T09:02:07.238Z [INFO]  server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177: restored from snapshot: id=2-2-1772269327237 last-index=2 last-term=2 size-in-bytes=1
2026-02-28T09:02:07.238Z [INFO]  server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b9327b5-61f5-e5f4-fea8-9293adb8a416 Address:6b9327b5-61f5-e5f4-fea8-9293adb8a416} {Suffrage:Voter ID:server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177 Address:cc0c5407-49fc-a2ac-fe7f-3ef17fc74177} {Suffrage:Voter ID:server-1af307a3-d89f-9f70-0eba-be35aa6e57d3 Address:1af307a3-d89f-9f70-0eba-be35aa6e57d3}]"
2026-02-28T09:02:07.238Z [INFO]  server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177: entering follower state: follower="Node at cc0c5407-49fc-a2ac-fe7f-3ef17fc74177 [Follower]" leader-address= leader-id=
2026-02-28T09:02:07.238Z [INFO]  creating new snapshot: path=/tmp/raft188026581/snapshots/2-2-1772269327238.tmp
2026-02-28T09:02:07.239Z [INFO]  server-1af307a3-d89f-9f70-0eba-be35aa6e57d3: starting restore from snapshot: id=2-2-1772269327238 last-index=2 last-term=2 size-in-bytes=1
2026-02-28T09:02:07.239Z [INFO]  server-1af307a3-d89f-9f70-0eba-be35aa6e57d3: snapshot restore progress: id=2-2-1772269327238 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2026-02-28T09:02:07.239Z [INFO]  server-1af307a3-d89f-9f70-0eba-be35aa6e57d3: restored from snapshot: id=2-2-1772269327238 last-index=2 last-term=2 size-in-bytes=1
2026-02-28T09:02:07.239Z [INFO]  server-1af307a3-d89f-9f70-0eba-be35aa6e57d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b9327b5-61f5-e5f4-fea8-9293adb8a416 Address:6b9327b5-61f5-e5f4-fea8-9293adb8a416} {Suffrage:Voter ID:server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177 Address:cc0c5407-49fc-a2ac-fe7f-3ef17fc74177} {Suffrage:Voter ID:server-1af307a3-d89f-9f70-0eba-be35aa6e57d3 Address:1af307a3-d89f-9f70-0eba-be35aa6e57d3}]"
2026-02-28T09:02:07.239Z [INFO]  server-1af307a3-d89f-9f70-0eba-be35aa6e57d3: entering follower state: follower="Node at 1af307a3-d89f-9f70-0eba-be35aa6e57d3 [Follower]" leader-address= leader-id=
2026-02-28T09:02:07.301Z [WARN]  server-1af307a3-d89f-9f70-0eba-be35aa6e57d3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:07.301Z [INFO]  server-1af307a3-d89f-9f70-0eba-be35aa6e57d3: entering candidate state: node="Node at 1af307a3-d89f-9f70-0eba-be35aa6e57d3 [Candidate]" term=3
2026-02-28T09:02:07.302Z [INFO]  server-1af307a3-d89f-9f70-0eba-be35aa6e57d3: election won: term=3 tally=2
2026-02-28T09:02:07.302Z [INFO]  server-1af307a3-d89f-9f70-0eba-be35aa6e57d3: entering leader state: leader="Node at 1af307a3-d89f-9f70-0eba-be35aa6e57d3 [Leader]"
2026-02-28T09:02:07.302Z [INFO]  server-1af307a3-d89f-9f70-0eba-be35aa6e57d3: added peer, starting replication: peer=server-6b9327b5-61f5-e5f4-fea8-9293adb8a416
2026-02-28T09:02:07.302Z [INFO]  server-1af307a3-d89f-9f70-0eba-be35aa6e57d3: added peer, starting replication: peer=server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177
2026-02-28T09:02:07.302Z [INFO]  server-1af307a3-d89f-9f70-0eba-be35aa6e57d3: pipelining replication: peer="{Voter server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177 cc0c5407-49fc-a2ac-fe7f-3ef17fc74177}"
2026-02-28T09:02:07.302Z [INFO]  server-1af307a3-d89f-9f70-0eba-be35aa6e57d3: pipelining replication: peer="{Voter server-6b9327b5-61f5-e5f4-fea8-9293adb8a416 6b9327b5-61f5-e5f4-fea8-9293adb8a416}"
2026-02-28T09:02:07.554Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:07.660Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:07.555048422 +0000 UTC m=+5.518845001 (1 nodes), 48.08µs from start of poll, 477.84776ms from cluster start. Timeout at 2026-02-28 09:02:07.660052222 +0000 UTC m=+5.623848761, 105.00376ms after stability
2026-02-28T09:02:07.660Z [INFO]  server-1af307a3-d89f-9f70-0eba-be35aa6e57d3: aborting pipeline replication: peer="{Voter server-cc0c5407-49fc-a2ac-fe7f-3ef17fc74177 cc0c5407-49fc-a2ac-fe7f-3ef17fc74177}"
2026-02-28T09:02:07.660Z [INFO]  server-1af307a3-d89f-9f70-0eba-be35aa6e57d3: aborting pipeline replication: peer="{Voter server-6b9327b5-61f5-e5f4-fea8-9293adb8a416 6b9327b5-61f5-e5f4-fea8-9293adb8a416}"
=== RUN   TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs
2026-02-28T09:02:07.661Z [INFO]  server-b7b8adfb-8763-f7fe-f07b-54095d366bcc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b7b8adfb-8763-f7fe-f07b-54095d366bcc Address:b7b8adfb-8763-f7fe-f07b-54095d366bcc} {Suffrage:Voter ID:server-b376b07d-c6b2-7e71-cd65-acd34dbca79a Address:b376b07d-c6b2-7e71-cd65-acd34dbca79a} {Suffrage:Voter ID:server-d90fdcfd-9601-20ad-18f1-3181573f7337 Address:d90fdcfd-9601-20ad-18f1-3181573f7337}]"
2026-02-28T09:02:07.661Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b7b8adfb-8763-f7fe-f07b-54095d366bcc Address:b7b8adfb-8763-f7fe-f07b-54095d366bcc} {Suffrage:Voter ID:server-b376b07d-c6b2-7e71-cd65-acd34dbca79a Address:b376b07d-c6b2-7e71-cd65-acd34dbca79a} {Suffrage:Voter ID:server-d90fdcfd-9601-20ad-18f1-3181573f7337 Address:d90fdcfd-9601-20ad-18f1-3181573f7337}]"
2026-02-28T09:02:07.661Z [INFO]  server-b7b8adfb-8763-f7fe-f07b-54095d366bcc: entering follower state: follower="Node at b7b8adfb-8763-f7fe-f07b-54095d366bcc [Follower]" leader-address= leader-id=
2026-02-28T09:02:07.661Z [INFO]  server-d90fdcfd-9601-20ad-18f1-3181573f7337: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b7b8adfb-8763-f7fe-f07b-54095d366bcc Address:b7b8adfb-8763-f7fe-f07b-54095d366bcc} {Suffrage:Voter ID:server-b376b07d-c6b2-7e71-cd65-acd34dbca79a Address:b376b07d-c6b2-7e71-cd65-acd34dbca79a} {Suffrage:Voter ID:server-d90fdcfd-9601-20ad-18f1-3181573f7337 Address:d90fdcfd-9601-20ad-18f1-3181573f7337}]"
2026-02-28T09:02:07.661Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: entering follower state: follower="Node at b376b07d-c6b2-7e71-cd65-acd34dbca79a [Follower]" leader-address= leader-id=
2026-02-28T09:02:07.661Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:07.661Z [INFO]  server-d90fdcfd-9601-20ad-18f1-3181573f7337: entering follower state: follower="Node at d90fdcfd-9601-20ad-18f1-3181573f7337 [Follower]" leader-address= leader-id=
2026-02-28T09:02:07.714Z [WARN]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:07.714Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: entering candidate state: node="Node at b376b07d-c6b2-7e71-cd65-acd34dbca79a [Candidate]" term=2
2026-02-28T09:02:07.714Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: election won: term=2 tally=2
2026-02-28T09:02:07.714Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: entering leader state: leader="Node at b376b07d-c6b2-7e71-cd65-acd34dbca79a [Leader]"
2026-02-28T09:02:07.714Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: added peer, starting replication: peer=server-b7b8adfb-8763-f7fe-f07b-54095d366bcc
2026-02-28T09:02:07.714Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: added peer, starting replication: peer=server-d90fdcfd-9601-20ad-18f1-3181573f7337
2026-02-28T09:02:07.714Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: pipelining replication: peer="{Voter server-d90fdcfd-9601-20ad-18f1-3181573f7337 d90fdcfd-9601-20ad-18f1-3181573f7337}"
2026-02-28T09:02:07.714Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: pipelining replication: peer="{Voter server-b7b8adfb-8763-f7fe-f07b-54095d366bcc b7b8adfb-8763-f7fe-f07b-54095d366bcc}"
2026-02-28T09:02:07.819Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:07.714349502 +0000 UTC m=+5.678146041 (1 nodes), 52.83568ms from start of poll, 53.2562ms from cluster start. Timeout at 2026-02-28 09:02:07.819355462 +0000 UTC m=+5.783152001, 105.00596ms after stability
2026-02-28T09:02:07.820Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: aborting pipeline replication: peer="{Voter server-d90fdcfd-9601-20ad-18f1-3181573f7337 d90fdcfd-9601-20ad-18f1-3181573f7337}"
2026-02-28T09:02:07.820Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: aborting pipeline replication: peer="{Voter server-b7b8adfb-8763-f7fe-f07b-54095d366bcc b7b8adfb-8763-f7fe-f07b-54095d366bcc}"
2026-02-28T09:02:07.820Z [INFO]  creating new snapshot: path=/tmp/raft4182052260/snapshots/2-6-1772269327820.tmp
2026-02-28T09:02:07.821Z [INFO]  server-b7b8adfb-8763-f7fe-f07b-54095d366bcc: starting restore from snapshot: id=2-6-1772269327820 last-index=6 last-term=2 size-in-bytes=25
2026-02-28T09:02:07.821Z [INFO]  server-b7b8adfb-8763-f7fe-f07b-54095d366bcc: snapshot restore progress: id=2-6-1772269327820 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2026-02-28T09:02:07.821Z [INFO]  server-b7b8adfb-8763-f7fe-f07b-54095d366bcc: restored from snapshot: id=2-6-1772269327820 last-index=6 last-term=2 size-in-bytes=25
2026-02-28T09:02:07.821Z [INFO]  server-b7b8adfb-8763-f7fe-f07b-54095d366bcc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b7b8adfb-8763-f7fe-f07b-54095d366bcc Address:b7b8adfb-8763-f7fe-f07b-54095d366bcc} {Suffrage:Voter ID:server-b376b07d-c6b2-7e71-cd65-acd34dbca79a Address:b376b07d-c6b2-7e71-cd65-acd34dbca79a} {Suffrage:Voter ID:server-d90fdcfd-9601-20ad-18f1-3181573f7337 Address:d90fdcfd-9601-20ad-18f1-3181573f7337}]"
2026-02-28T09:02:07.821Z [INFO]  creating new snapshot: path=/tmp/raft1274148462/snapshots/2-6-1772269327821.tmp
2026-02-28T09:02:07.821Z [INFO]  server-b7b8adfb-8763-f7fe-f07b-54095d366bcc: entering follower state: follower="Node at b7b8adfb-8763-f7fe-f07b-54095d366bcc [Follower]" leader-address= leader-id=
2026-02-28T09:02:07.822Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: starting restore from snapshot: id=2-6-1772269327821 last-index=6 last-term=2 size-in-bytes=25
2026-02-28T09:02:07.822Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: snapshot restore progress: id=2-6-1772269327821 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2026-02-28T09:02:07.822Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: restored from snapshot: id=2-6-1772269327821 last-index=6 last-term=2 size-in-bytes=25
2026-02-28T09:02:07.822Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b7b8adfb-8763-f7fe-f07b-54095d366bcc Address:b7b8adfb-8763-f7fe-f07b-54095d366bcc} {Suffrage:Voter ID:server-b376b07d-c6b2-7e71-cd65-acd34dbca79a Address:b376b07d-c6b2-7e71-cd65-acd34dbca79a} {Suffrage:Voter ID:server-d90fdcfd-9601-20ad-18f1-3181573f7337 Address:d90fdcfd-9601-20ad-18f1-3181573f7337}]"
2026-02-28T09:02:07.822Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: entering follower state: follower="Node at b376b07d-c6b2-7e71-cd65-acd34dbca79a [Follower]" leader-address= leader-id=
2026-02-28T09:02:07.822Z [INFO]  creating new snapshot: path=/tmp/raft3900159820/snapshots/2-6-1772269327822.tmp
2026-02-28T09:02:07.822Z [INFO]  server-d90fdcfd-9601-20ad-18f1-3181573f7337: starting restore from snapshot: id=2-6-1772269327822 last-index=6 last-term=2 size-in-bytes=25
2026-02-28T09:02:07.823Z [INFO]  server-d90fdcfd-9601-20ad-18f1-3181573f7337: snapshot restore progress: id=2-6-1772269327822 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2026-02-28T09:02:07.823Z [INFO]  server-d90fdcfd-9601-20ad-18f1-3181573f7337: restored from snapshot: id=2-6-1772269327822 last-index=6 last-term=2 size-in-bytes=25
2026-02-28T09:02:07.823Z [INFO]  server-d90fdcfd-9601-20ad-18f1-3181573f7337: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b7b8adfb-8763-f7fe-f07b-54095d366bcc Address:b7b8adfb-8763-f7fe-f07b-54095d366bcc} {Suffrage:Voter ID:server-b376b07d-c6b2-7e71-cd65-acd34dbca79a Address:b376b07d-c6b2-7e71-cd65-acd34dbca79a} {Suffrage:Voter ID:server-d90fdcfd-9601-20ad-18f1-3181573f7337 Address:d90fdcfd-9601-20ad-18f1-3181573f7337}]"
2026-02-28T09:02:07.823Z [INFO]  server-d90fdcfd-9601-20ad-18f1-3181573f7337: entering follower state: follower="Node at d90fdcfd-9601-20ad-18f1-3181573f7337 [Follower]" leader-address= leader-id=
2026-02-28T09:02:07.877Z [WARN]  server-b7b8adfb-8763-f7fe-f07b-54095d366bcc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:07.877Z [INFO]  server-b7b8adfb-8763-f7fe-f07b-54095d366bcc: entering candidate state: node="Node at b7b8adfb-8763-f7fe-f07b-54095d366bcc [Candidate]" term=3
2026-02-28T09:02:07.877Z [WARN]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:07.877Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: entering candidate state: node="Node at b376b07d-c6b2-7e71-cd65-acd34dbca79a [Candidate]" term=3
2026-02-28T09:02:07.877Z [INFO]  server-b7b8adfb-8763-f7fe-f07b-54095d366bcc: duplicate requestVote for same term: term=3
2026-02-28T09:02:07.877Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: election won: term=3 tally=2
2026-02-28T09:02:07.877Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: entering leader state: leader="Node at b376b07d-c6b2-7e71-cd65-acd34dbca79a [Leader]"
2026-02-28T09:02:07.877Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: added peer, starting replication: peer=server-b7b8adfb-8763-f7fe-f07b-54095d366bcc
2026-02-28T09:02:07.877Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: added peer, starting replication: peer=server-d90fdcfd-9601-20ad-18f1-3181573f7337
2026-02-28T09:02:07.877Z [INFO]  server-d90fdcfd-9601-20ad-18f1-3181573f7337: duplicate requestVote for same term: term=3
2026-02-28T09:02:07.877Z [WARN]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: rejecting vote request since we have a leader: from=b7b8adfb-8763-f7fe-f07b-54095d366bcc leader=b376b07d-c6b2-7e71-cd65-acd34dbca79a leader-id=server-b376b07d-c6b2-7e71-cd65-acd34dbca79a
2026-02-28T09:02:07.877Z [INFO]  server-b7b8adfb-8763-f7fe-f07b-54095d366bcc: entering follower state: follower="Node at b7b8adfb-8763-f7fe-f07b-54095d366bcc [Follower]" leader-address=b376b07d-c6b2-7e71-cd65-acd34dbca79a leader-id=server-b376b07d-c6b2-7e71-cd65-acd34dbca79a
2026-02-28T09:02:07.877Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: pipelining replication: peer="{Voter server-b7b8adfb-8763-f7fe-f07b-54095d366bcc b7b8adfb-8763-f7fe-f07b-54095d366bcc}"
2026-02-28T09:02:07.877Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: pipelining replication: peer="{Voter server-d90fdcfd-9601-20ad-18f1-3181573f7337 d90fdcfd-9601-20ad-18f1-3181573f7337}"
2026-02-28T09:02:08.138Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:08.244Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:08.138917662 +0000 UTC m=+6.102714241 (1 nodes), 44.8µs from start of poll, 477.8244ms from cluster start. Timeout at 2026-02-28 09:02:08.243926142 +0000 UTC m=+6.207722681, 105.00844ms after stability
2026-02-28T09:02:08.244Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: aborting pipeline replication: peer="{Voter server-d90fdcfd-9601-20ad-18f1-3181573f7337 d90fdcfd-9601-20ad-18f1-3181573f7337}"
2026-02-28T09:02:08.244Z [INFO]  server-b376b07d-c6b2-7e71-cd65-acd34dbca79a: aborting pipeline replication: peer="{Voter server-b7b8adfb-8763-f7fe-f07b-54095d366bcc b7b8adfb-8763-f7fe-f07b-54095d366bcc}"
=== RUN   TestRaft_RecoverCluster/snapshot,_with_trailing_logs
2026-02-28T09:02:08.245Z [INFO]  server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e Address:b9f6860b-57c3-9c4a-8f84-f2a13064db8e} {Suffrage:Voter ID:server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a Address:9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a} {Suffrage:Voter ID:server-7e3feaaa-c03d-0ac6-034e-ea031fa33062 Address:7e3feaaa-c03d-0ac6-034e-ea031fa33062}]"
2026-02-28T09:02:08.245Z [INFO]  server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e: entering follower state: follower="Node at b9f6860b-57c3-9c4a-8f84-f2a13064db8e [Follower]" leader-address= leader-id=
2026-02-28T09:02:08.245Z [INFO]  server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e Address:b9f6860b-57c3-9c4a-8f84-f2a13064db8e} {Suffrage:Voter ID:server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a Address:9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a} {Suffrage:Voter ID:server-7e3feaaa-c03d-0ac6-034e-ea031fa33062 Address:7e3feaaa-c03d-0ac6-034e-ea031fa33062}]"
2026-02-28T09:02:08.245Z [INFO]  server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a: entering follower state: follower="Node at 9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a [Follower]" leader-address= leader-id=
2026-02-28T09:02:08.245Z [INFO]  server-7e3feaaa-c03d-0ac6-034e-ea031fa33062: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e Address:b9f6860b-57c3-9c4a-8f84-f2a13064db8e} {Suffrage:Voter ID:server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a Address:9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a} {Suffrage:Voter ID:server-7e3feaaa-c03d-0ac6-034e-ea031fa33062 Address:7e3feaaa-c03d-0ac6-034e-ea031fa33062}]"
2026-02-28T09:02:08.245Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:08.245Z [INFO]  server-7e3feaaa-c03d-0ac6-034e-ea031fa33062: entering follower state: follower="Node at 7e3feaaa-c03d-0ac6-034e-ea031fa33062 [Follower]" leader-address= leader-id=
2026-02-28T09:02:08.311Z [WARN]  server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:08.311Z [INFO]  server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e: entering candidate state: node="Node at b9f6860b-57c3-9c4a-8f84-f2a13064db8e [Candidate]" term=2
2026-02-28T09:02:08.312Z [INFO]  server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e: election won: term=2 tally=2
2026-02-28T09:02:08.312Z [INFO]  server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e: entering leader state: leader="Node at b9f6860b-57c3-9c4a-8f84-f2a13064db8e [Leader]"
2026-02-28T09:02:08.312Z [INFO]  server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e: added peer, starting replication: peer=server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a
2026-02-28T09:02:08.312Z [INFO]  server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e: added peer, starting replication: peer=server-7e3feaaa-c03d-0ac6-034e-ea031fa33062
2026-02-28T09:02:08.312Z [INFO]  server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e: pipelining replication: peer="{Voter server-7e3feaaa-c03d-0ac6-034e-ea031fa33062 7e3feaaa-c03d-0ac6-034e-ea031fa33062}"
2026-02-28T09:02:08.312Z [INFO]  server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e: pipelining replication: peer="{Voter server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a 9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a}"
2026-02-28T09:02:08.417Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:08.312202542 +0000 UTC m=+6.275999081 (1 nodes), 66.4478ms from start of poll, 67.03448ms from cluster start. Timeout at 2026-02-28 09:02:08.417205902 +0000 UTC m=+6.381002441, 105.00336ms after stability
2026-02-28T09:02:08.421Z [INFO]  server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e: aborting pipeline replication: peer="{Voter server-7e3feaaa-c03d-0ac6-034e-ea031fa33062 7e3feaaa-c03d-0ac6-034e-ea031fa33062}"
2026-02-28T09:02:08.421Z [INFO]  server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e: aborting pipeline replication: peer="{Voter server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a 9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a}"
2026-02-28T09:02:08.421Z [INFO]  creating new snapshot: path=/tmp/raft2746157543/snapshots/2-27-1772269328421.tmp
2026-02-28T09:02:08.421Z [INFO]  server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e: starting restore from snapshot: id=2-27-1772269328421 last-index=27 last-term=2 size-in-bytes=168
2026-02-28T09:02:08.422Z [INFO]  server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e: snapshot restore progress: id=2-27-1772269328421 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2026-02-28T09:02:08.422Z [INFO]  server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e: restored from snapshot: id=2-27-1772269328421 last-index=27 last-term=2 size-in-bytes=168
2026-02-28T09:02:08.422Z [INFO]  server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e Address:b9f6860b-57c3-9c4a-8f84-f2a13064db8e} {Suffrage:Voter ID:server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a Address:9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a} {Suffrage:Voter ID:server-7e3feaaa-c03d-0ac6-034e-ea031fa33062 Address:7e3feaaa-c03d-0ac6-034e-ea031fa33062}]"
2026-02-28T09:02:08.422Z [INFO]  creating new snapshot: path=/tmp/raft2383614402/snapshots/2-27-1772269328422.tmp
2026-02-28T09:02:08.422Z [INFO]  server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e: entering follower state: follower="Node at b9f6860b-57c3-9c4a-8f84-f2a13064db8e [Follower]" leader-address= leader-id=
2026-02-28T09:02:08.422Z [INFO]  server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a: starting restore from snapshot: id=2-27-1772269328422 last-index=27 last-term=2 size-in-bytes=168
2026-02-28T09:02:08.422Z [INFO]  server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a: snapshot restore progress: id=2-27-1772269328422 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2026-02-28T09:02:08.422Z [INFO]  server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a: restored from snapshot: id=2-27-1772269328422 last-index=27 last-term=2 size-in-bytes=168
2026-02-28T09:02:08.422Z [INFO]  server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e Address:b9f6860b-57c3-9c4a-8f84-f2a13064db8e} {Suffrage:Voter ID:server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a Address:9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a} {Suffrage:Voter ID:server-7e3feaaa-c03d-0ac6-034e-ea031fa33062 Address:7e3feaaa-c03d-0ac6-034e-ea031fa33062}]"
2026-02-28T09:02:08.422Z [INFO]  server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a: entering follower state: follower="Node at 9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a [Follower]" leader-address= leader-id=
2026-02-28T09:02:08.422Z [INFO]  creating new snapshot: path=/tmp/raft899492449/snapshots/2-27-1772269328422.tmp
2026-02-28T09:02:08.423Z [INFO]  server-7e3feaaa-c03d-0ac6-034e-ea031fa33062: starting restore from snapshot: id=2-27-1772269328422 last-index=27 last-term=2 size-in-bytes=168
2026-02-28T09:02:08.423Z [INFO]  server-7e3feaaa-c03d-0ac6-034e-ea031fa33062: snapshot restore progress: id=2-27-1772269328422 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2026-02-28T09:02:08.423Z [INFO]  server-7e3feaaa-c03d-0ac6-034e-ea031fa33062: restored from snapshot: id=2-27-1772269328422 last-index=27 last-term=2 size-in-bytes=168
2026-02-28T09:02:08.423Z [INFO]  server-7e3feaaa-c03d-0ac6-034e-ea031fa33062: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e Address:b9f6860b-57c3-9c4a-8f84-f2a13064db8e} {Suffrage:Voter ID:server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a Address:9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a} {Suffrage:Voter ID:server-7e3feaaa-c03d-0ac6-034e-ea031fa33062 Address:7e3feaaa-c03d-0ac6-034e-ea031fa33062}]"
2026-02-28T09:02:08.423Z [INFO]  server-7e3feaaa-c03d-0ac6-034e-ea031fa33062: entering follower state: follower="Node at 7e3feaaa-c03d-0ac6-034e-ea031fa33062 [Follower]" leader-address= leader-id=
2026-02-28T09:02:08.490Z [WARN]  server-7e3feaaa-c03d-0ac6-034e-ea031fa33062: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:08.490Z [INFO]  server-7e3feaaa-c03d-0ac6-034e-ea031fa33062: entering candidate state: node="Node at 7e3feaaa-c03d-0ac6-034e-ea031fa33062 [Candidate]" term=3
2026-02-28T09:02:08.490Z [INFO]  server-7e3feaaa-c03d-0ac6-034e-ea031fa33062: election won: term=3 tally=2
2026-02-28T09:02:08.490Z [INFO]  server-7e3feaaa-c03d-0ac6-034e-ea031fa33062: entering leader state: leader="Node at 7e3feaaa-c03d-0ac6-034e-ea031fa33062 [Leader]"
2026-02-28T09:02:08.491Z [INFO]  server-7e3feaaa-c03d-0ac6-034e-ea031fa33062: added peer, starting replication: peer=server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e
2026-02-28T09:02:08.491Z [INFO]  server-7e3feaaa-c03d-0ac6-034e-ea031fa33062: added peer, starting replication: peer=server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a
2026-02-28T09:02:08.491Z [INFO]  server-7e3feaaa-c03d-0ac6-034e-ea031fa33062: pipelining replication: peer="{Voter server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e b9f6860b-57c3-9c4a-8f84-f2a13064db8e}"
2026-02-28T09:02:08.491Z [INFO]  server-7e3feaaa-c03d-0ac6-034e-ea031fa33062: pipelining replication: peer="{Voter server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a 9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a}"
2026-02-28T09:02:08.739Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:08.844Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:08.739151302 +0000 UTC m=+6.702947841 (1 nodes), 51.32µs from start of poll, 493.98324ms from cluster start. Timeout at 2026-02-28 09:02:08.844165742 +0000 UTC m=+6.807962281, 105.01444ms after stability
2026-02-28T09:02:08.844Z [INFO]  server-7e3feaaa-c03d-0ac6-034e-ea031fa33062: aborting pipeline replication: peer="{Voter server-b9f6860b-57c3-9c4a-8f84-f2a13064db8e b9f6860b-57c3-9c4a-8f84-f2a13064db8e}"
2026-02-28T09:02:08.844Z [INFO]  server-7e3feaaa-c03d-0ac6-034e-ea031fa33062: aborting pipeline replication: peer="{Voter server-9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a 9b4112ce-a7f4-d58a-01bb-c6ad7b16bf5a}"
--- PASS: TestRaft_RecoverCluster (1.77s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.58s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.58s)
    --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.60s)
=== RUN   TestRaft_HasExistingState
    raft_test.go:258: Skipping unreliable test
--- SKIP: TestRaft_HasExistingState (0.00s)
=== RUN   TestRaft_SingleNode
2026-02-28T09:02:08.845Z [INFO]  server-6b5a1326-d39f-2b1c-6f53-4db063a2dd20: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b5a1326-d39f-2b1c-6f53-4db063a2dd20 Address:6b5a1326-d39f-2b1c-6f53-4db063a2dd20}]"
2026-02-28T09:02:08.845Z [INFO]  server-6b5a1326-d39f-2b1c-6f53-4db063a2dd20: entering follower state: follower="Node at 6b5a1326-d39f-2b1c-6f53-4db063a2dd20 [Follower]" leader-address= leader-id=
2026-02-28T09:02:08.933Z [WARN]  server-6b5a1326-d39f-2b1c-6f53-4db063a2dd20: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:08.933Z [INFO]  server-6b5a1326-d39f-2b1c-6f53-4db063a2dd20: entering candidate state: node="Node at 6b5a1326-d39f-2b1c-6f53-4db063a2dd20 [Candidate]" term=2
2026-02-28T09:02:08.933Z [INFO]  server-6b5a1326-d39f-2b1c-6f53-4db063a2dd20: election won: term=2 tally=1
2026-02-28T09:02:08.933Z [INFO]  server-6b5a1326-d39f-2b1c-6f53-4db063a2dd20: entering leader state: leader="Node at 6b5a1326-d39f-2b1c-6f53-4db063a2dd20 [Leader]"
--- PASS: TestRaft_SingleNode (0.09s)
=== RUN   TestRaft_TripleNode
2026-02-28T09:02:08.934Z [INFO]  server-6c94668b-5380-60e1-35bc-b797104c72d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c94668b-5380-60e1-35bc-b797104c72d5 Address:6c94668b-5380-60e1-35bc-b797104c72d5} {Suffrage:Voter ID:server-55454931-f791-63c7-bcf7-96d97b30c8c2 Address:55454931-f791-63c7-bcf7-96d97b30c8c2} {Suffrage:Voter ID:server-5c5b0115-7c28-f751-12c4-2e4c6ebd788d Address:5c5b0115-7c28-f751-12c4-2e4c6ebd788d}]"
2026-02-28T09:02:08.934Z [INFO]  server-6c94668b-5380-60e1-35bc-b797104c72d5: entering follower state: follower="Node at 6c94668b-5380-60e1-35bc-b797104c72d5 [Follower]" leader-address= leader-id=
2026-02-28T09:02:08.934Z [INFO]  server-55454931-f791-63c7-bcf7-96d97b30c8c2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c94668b-5380-60e1-35bc-b797104c72d5 Address:6c94668b-5380-60e1-35bc-b797104c72d5} {Suffrage:Voter ID:server-55454931-f791-63c7-bcf7-96d97b30c8c2 Address:55454931-f791-63c7-bcf7-96d97b30c8c2} {Suffrage:Voter ID:server-5c5b0115-7c28-f751-12c4-2e4c6ebd788d Address:5c5b0115-7c28-f751-12c4-2e4c6ebd788d}]"
2026-02-28T09:02:08.934Z [INFO]  server-55454931-f791-63c7-bcf7-96d97b30c8c2: entering follower state: follower="Node at 55454931-f791-63c7-bcf7-96d97b30c8c2 [Follower]" leader-address= leader-id=
2026-02-28T09:02:08.934Z [INFO]  server-5c5b0115-7c28-f751-12c4-2e4c6ebd788d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c94668b-5380-60e1-35bc-b797104c72d5 Address:6c94668b-5380-60e1-35bc-b797104c72d5} {Suffrage:Voter ID:server-55454931-f791-63c7-bcf7-96d97b30c8c2 Address:55454931-f791-63c7-bcf7-96d97b30c8c2} {Suffrage:Voter ID:server-5c5b0115-7c28-f751-12c4-2e4c6ebd788d Address:5c5b0115-7c28-f751-12c4-2e4c6ebd788d}]"
2026-02-28T09:02:08.934Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:08.934Z [INFO]  server-5c5b0115-7c28-f751-12c4-2e4c6ebd788d: entering follower state: follower="Node at 5c5b0115-7c28-f751-12c4-2e4c6ebd788d [Follower]" leader-address= leader-id=
2026-02-28T09:02:08.988Z [WARN]  server-55454931-f791-63c7-bcf7-96d97b30c8c2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:08.988Z [INFO]  server-55454931-f791-63c7-bcf7-96d97b30c8c2: entering candidate state: node="Node at 55454931-f791-63c7-bcf7-96d97b30c8c2 [Candidate]" term=2
2026-02-28T09:02:08.988Z [INFO]  server-55454931-f791-63c7-bcf7-96d97b30c8c2: election won: term=2 tally=2
2026-02-28T09:02:08.988Z [WARN]  server-6c94668b-5380-60e1-35bc-b797104c72d5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:08.988Z [INFO]  server-55454931-f791-63c7-bcf7-96d97b30c8c2: entering leader state: leader="Node at 55454931-f791-63c7-bcf7-96d97b30c8c2 [Leader]"
2026-02-28T09:02:08.988Z [INFO]  server-6c94668b-5380-60e1-35bc-b797104c72d5: entering candidate state: node="Node at 6c94668b-5380-60e1-35bc-b797104c72d5 [Candidate]" term=2
2026-02-28T09:02:08.988Z [INFO]  server-55454931-f791-63c7-bcf7-96d97b30c8c2: added peer, starting replication: peer=server-6c94668b-5380-60e1-35bc-b797104c72d5
2026-02-28T09:02:08.988Z [INFO]  server-55454931-f791-63c7-bcf7-96d97b30c8c2: added peer, starting replication: peer=server-5c5b0115-7c28-f751-12c4-2e4c6ebd788d
2026-02-28T09:02:08.988Z [INFO]  server-6c94668b-5380-60e1-35bc-b797104c72d5: duplicate requestVote for same term: term=2
2026-02-28T09:02:08.988Z [WARN]  server-55454931-f791-63c7-bcf7-96d97b30c8c2: rejecting vote request since we have a leader: from=6c94668b-5380-60e1-35bc-b797104c72d5 leader=55454931-f791-63c7-bcf7-96d97b30c8c2 leader-id=server-55454931-f791-63c7-bcf7-96d97b30c8c2
2026-02-28T09:02:08.988Z [INFO]  server-55454931-f791-63c7-bcf7-96d97b30c8c2: pipelining replication: peer="{Voter server-5c5b0115-7c28-f751-12c4-2e4c6ebd788d 5c5b0115-7c28-f751-12c4-2e4c6ebd788d}"
2026-02-28T09:02:08.988Z [INFO]  server-6c94668b-5380-60e1-35bc-b797104c72d5: entering follower state: follower="Node at 6c94668b-5380-60e1-35bc-b797104c72d5 [Follower]" leader-address=55454931-f791-63c7-bcf7-96d97b30c8c2 leader-id=server-55454931-f791-63c7-bcf7-96d97b30c8c2
2026-02-28T09:02:08.988Z [WARN]  server-5c5b0115-7c28-f751-12c4-2e4c6ebd788d: rejecting vote request since we have a leader: from=6c94668b-5380-60e1-35bc-b797104c72d5 leader=55454931-f791-63c7-bcf7-96d97b30c8c2 leader-id=server-55454931-f791-63c7-bcf7-96d97b30c8c2
2026-02-28T09:02:08.988Z [INFO]  server-55454931-f791-63c7-bcf7-96d97b30c8c2: pipelining replication: peer="{Voter server-6c94668b-5380-60e1-35bc-b797104c72d5 6c94668b-5380-60e1-35bc-b797104c72d5}"
2026-02-28T09:02:09.094Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:08.988748742 +0000 UTC m=+6.952545281 (2 nodes), 53.90572ms from start of poll, 54.34472ms from cluster start. Timeout at 2026-02-28 09:02:09.093750342 +0000 UTC m=+7.057546881, 105.0016ms after stability
2026-02-28T09:02:09.094Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:09.200Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:09.094748902 +0000 UTC m=+7.058545441 (1 nodes), 2.4µs from start of poll, 160.34488ms from cluster start. Timeout at 2026-02-28 09:02:09.199759262 +0000 UTC m=+7.163555801, 105.01036ms after stability
2026-02-28T09:02:09.211Z [INFO]  server-55454931-f791-63c7-bcf7-96d97b30c8c2: aborting pipeline replication: peer="{Voter server-5c5b0115-7c28-f751-12c4-2e4c6ebd788d 5c5b0115-7c28-f751-12c4-2e4c6ebd788d}"
2026-02-28T09:02:09.211Z [INFO]  server-55454931-f791-63c7-bcf7-96d97b30c8c2: aborting pipeline replication: peer="{Voter server-6c94668b-5380-60e1-35bc-b797104c72d5 6c94668b-5380-60e1-35bc-b797104c72d5}"
--- PASS: TestRaft_TripleNode (0.28s)
=== RUN   TestRaft_LeaderFail
2026-02-28T09:02:09.212Z [INFO]  server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b Address:4e0d1c77-1d55-b30e-24c3-d6c97371c65b} {Suffrage:Voter ID:server-97179f2b-2657-0420-8877-771b7e8e74a8 Address:97179f2b-2657-0420-8877-771b7e8e74a8} {Suffrage:Voter ID:server-a53bbd30-d396-230a-d97b-996e5ee79a99 Address:a53bbd30-d396-230a-d97b-996e5ee79a99}]"
2026-02-28T09:02:09.212Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b Address:4e0d1c77-1d55-b30e-24c3-d6c97371c65b} {Suffrage:Voter ID:server-97179f2b-2657-0420-8877-771b7e8e74a8 Address:97179f2b-2657-0420-8877-771b7e8e74a8} {Suffrage:Voter ID:server-a53bbd30-d396-230a-d97b-996e5ee79a99 Address:a53bbd30-d396-230a-d97b-996e5ee79a99}]"
2026-02-28T09:02:09.212Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: entering follower state: follower="Node at 97179f2b-2657-0420-8877-771b7e8e74a8 [Follower]" leader-address= leader-id=
2026-02-28T09:02:09.212Z [INFO]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b Address:4e0d1c77-1d55-b30e-24c3-d6c97371c65b} {Suffrage:Voter ID:server-97179f2b-2657-0420-8877-771b7e8e74a8 Address:97179f2b-2657-0420-8877-771b7e8e74a8} {Suffrage:Voter ID:server-a53bbd30-d396-230a-d97b-996e5ee79a99 Address:a53bbd30-d396-230a-d97b-996e5ee79a99}]"
2026-02-28T09:02:09.212Z [INFO]  server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b: entering follower state: follower="Node at 4e0d1c77-1d55-b30e-24c3-d6c97371c65b [Follower]" leader-address= leader-id=
2026-02-28T09:02:09.212Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:09.212Z [INFO]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: entering follower state: follower="Node at a53bbd30-d396-230a-d97b-996e5ee79a99 [Follower]" leader-address= leader-id=
2026-02-28T09:02:09.280Z [WARN]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:09.281Z [INFO]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: entering candidate state: node="Node at a53bbd30-d396-230a-d97b-996e5ee79a99 [Candidate]" term=2
2026-02-28T09:02:09.281Z [INFO]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: election won: term=2 tally=2
2026-02-28T09:02:09.281Z [INFO]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: entering leader state: leader="Node at a53bbd30-d396-230a-d97b-996e5ee79a99 [Leader]"
2026-02-28T09:02:09.281Z [INFO]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: added peer, starting replication: peer=server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b
2026-02-28T09:02:09.281Z [INFO]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: added peer, starting replication: peer=server-97179f2b-2657-0420-8877-771b7e8e74a8
2026-02-28T09:02:09.281Z [INFO]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: pipelining replication: peer="{Voter server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b 4e0d1c77-1d55-b30e-24c3-d6c97371c65b}"
2026-02-28T09:02:09.281Z [INFO]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: pipelining replication: peer="{Voter server-97179f2b-2657-0420-8877-771b7e8e74a8 97179f2b-2657-0420-8877-771b7e8e74a8}"
2026-02-28T09:02:09.387Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:09.281219382 +0000 UTC m=+7.245015921 (2 nodes), 68.20764ms from start of poll, 68.66312ms from cluster start. Timeout at 2026-02-28 09:02:09.386224022 +0000 UTC m=+7.350020601, 105.00468ms after stability
2026-02-28T09:02:09.387Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:09.492Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:09.387142862 +0000 UTC m=+7.350939401 (1 nodes), 800ns from start of poll, 174.5866ms from cluster start. Timeout at 2026-02-28 09:02:09.492150022 +0000 UTC m=+7.455946561, 105.00716ms after stability
    raft_test.go:378: [INFO] Disconnecting Node at a53bbd30-d396-230a-d97b-996e5ee79a99 [Leader]
2026-02-28T09:02:09.504Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to pipeline appendEntries: peer="{Voter server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b 4e0d1c77-1d55-b30e-24c3-d6c97371c65b}" error="append pipeline closed"
2026-02-28T09:02:09.504Z [INFO]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: aborting pipeline replication: peer="{Voter server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b 4e0d1c77-1d55-b30e-24c3-d6c97371c65b}"
2026-02-28T09:02:09.504Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to pipeline appendEntries: peer="{Voter server-97179f2b-2657-0420-8877-771b7e8e74a8 97179f2b-2657-0420-8877-771b7e8e74a8}" error="append pipeline closed"
2026-02-28T09:02:09.504Z [INFO]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: aborting pipeline replication: peer="{Voter server-97179f2b-2657-0420-8877-771b7e8e74a8 97179f2b-2657-0420-8877-771b7e8e74a8}"
2026-02-28T09:02:09.506Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to heartbeat to: peer=4e0d1c77-1d55-b30e-24c3-d6c97371c65b backoff time=10ms error="failed to connect to peer: 4e0d1c77-1d55-b30e-24c3-d6c97371c65b"
2026-02-28T09:02:09.506Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:09.507Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to heartbeat to: peer=97179f2b-2657-0420-8877-771b7e8e74a8 backoff time=10ms error="failed to connect to peer: 97179f2b-2657-0420-8877-771b7e8e74a8"
2026-02-28T09:02:09.510Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to appendEntries to: peer="{Voter server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b 4e0d1c77-1d55-b30e-24c3-d6c97371c65b}" error="failed to connect to peer: 4e0d1c77-1d55-b30e-24c3-d6c97371c65b"
2026-02-28T09:02:09.514Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to appendEntries to: peer="{Voter server-97179f2b-2657-0420-8877-771b7e8e74a8 97179f2b-2657-0420-8877-771b7e8e74a8}" error="failed to connect to peer: 97179f2b-2657-0420-8877-771b7e8e74a8"
2026-02-28T09:02:09.524Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to heartbeat to: peer=97179f2b-2657-0420-8877-771b7e8e74a8 backoff time=10ms error="failed to connect to peer: 97179f2b-2657-0420-8877-771b7e8e74a8"
2026-02-28T09:02:09.527Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to heartbeat to: peer=4e0d1c77-1d55-b30e-24c3-d6c97371c65b backoff time=10ms error="failed to connect to peer: 4e0d1c77-1d55-b30e-24c3-d6c97371c65b"
2026-02-28T09:02:09.529Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to appendEntries to: peer="{Voter server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b 4e0d1c77-1d55-b30e-24c3-d6c97371c65b}" error="failed to connect to peer: 4e0d1c77-1d55-b30e-24c3-d6c97371c65b"
2026-02-28T09:02:09.530Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to appendEntries to: peer="{Voter server-97179f2b-2657-0420-8877-771b7e8e74a8 97179f2b-2657-0420-8877-771b7e8e74a8}" error="failed to connect to peer: 97179f2b-2657-0420-8877-771b7e8e74a8"
2026-02-28T09:02:09.544Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to heartbeat to: peer=97179f2b-2657-0420-8877-771b7e8e74a8 backoff time=10ms error="failed to connect to peer: 97179f2b-2657-0420-8877-771b7e8e74a8"
2026-02-28T09:02:09.546Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to appendEntries to: peer="{Voter server-97179f2b-2657-0420-8877-771b7e8e74a8 97179f2b-2657-0420-8877-771b7e8e74a8}" error="failed to connect to peer: 97179f2b-2657-0420-8877-771b7e8e74a8"
2026-02-28T09:02:09.547Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to heartbeat to: peer=4e0d1c77-1d55-b30e-24c3-d6c97371c65b backoff time=10ms error="failed to connect to peer: 4e0d1c77-1d55-b30e-24c3-d6c97371c65b"
2026-02-28T09:02:09.548Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to appendEntries to: peer="{Voter server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b 4e0d1c77-1d55-b30e-24c3-d6c97371c65b}" error="failed to connect to peer: 4e0d1c77-1d55-b30e-24c3-d6c97371c65b"
2026-02-28T09:02:09.550Z [WARN]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to contact: server-id=server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b time=51.04504ms
2026-02-28T09:02:09.550Z [WARN]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to contact: server-id=server-97179f2b-2657-0420-8877-771b7e8e74a8 time=51.095ms
2026-02-28T09:02:09.550Z [WARN]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to contact quorum of nodes, stepping down
2026-02-28T09:02:09.550Z [INFO]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: entering follower state: follower="Node at a53bbd30-d396-230a-d97b-996e5ee79a99 [Follower]" leader-address= leader-id=
2026-02-28T09:02:09.572Z [WARN]  server-97179f2b-2657-0420-8877-771b7e8e74a8: heartbeat timeout reached, starting election: last-leader-addr=a53bbd30-d396-230a-d97b-996e5ee79a99 last-leader-id=server-a53bbd30-d396-230a-d97b-996e5ee79a99
2026-02-28T09:02:09.572Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: entering candidate state: node="Node at 97179f2b-2657-0420-8877-771b7e8e74a8 [Candidate]" term=3
2026-02-28T09:02:09.572Z [ERROR] server-97179f2b-2657-0420-8877-771b7e8e74a8: failed to make requestVote RPC: target="{Voter server-a53bbd30-d396-230a-d97b-996e5ee79a99 a53bbd30-d396-230a-d97b-996e5ee79a99}" error="failed to connect to peer: a53bbd30-d396-230a-d97b-996e5ee79a99" term=3
2026-02-28T09:02:09.573Z [WARN]  server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b: rejecting vote request since we have a leader: from=97179f2b-2657-0420-8877-771b7e8e74a8 leader=a53bbd30-d396-230a-d97b-996e5ee79a99 leader-id=server-a53bbd30-d396-230a-d97b-996e5ee79a99
2026-02-28T09:02:09.601Z [WARN]  server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b: heartbeat timeout reached, starting election: last-leader-addr=a53bbd30-d396-230a-d97b-996e5ee79a99 last-leader-id=server-a53bbd30-d396-230a-d97b-996e5ee79a99
2026-02-28T09:02:09.601Z [INFO]  server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b: entering candidate state: node="Node at 4e0d1c77-1d55-b30e-24c3-d6c97371c65b [Candidate]" term=3
2026-02-28T09:02:09.601Z [ERROR] server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b: failed to make requestVote RPC: target="{Voter server-a53bbd30-d396-230a-d97b-996e5ee79a99 a53bbd30-d396-230a-d97b-996e5ee79a99}" error="failed to connect to peer: a53bbd30-d396-230a-d97b-996e5ee79a99" term=3
2026-02-28T09:02:09.601Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: duplicate requestVote for same term: term=3
2026-02-28T09:02:09.619Z [WARN]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:09.619Z [INFO]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: entering candidate state: node="Node at a53bbd30-d396-230a-d97b-996e5ee79a99 [Candidate]" term=3
2026-02-28T09:02:09.619Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to make requestVote RPC: target="{Voter server-97179f2b-2657-0420-8877-771b7e8e74a8 97179f2b-2657-0420-8877-771b7e8e74a8}" error="failed to connect to peer: 97179f2b-2657-0420-8877-771b7e8e74a8" term=3
2026-02-28T09:02:09.619Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to make requestVote RPC: target="{Voter server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b 4e0d1c77-1d55-b30e-24c3-d6c97371c65b}" error="failed to connect to peer: 4e0d1c77-1d55-b30e-24c3-d6c97371c65b" term=3
2026-02-28T09:02:09.658Z [WARN]  server-97179f2b-2657-0420-8877-771b7e8e74a8: Election timeout reached, restarting election
2026-02-28T09:02:09.658Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: entering candidate state: node="Node at 97179f2b-2657-0420-8877-771b7e8e74a8 [Candidate]" term=4
2026-02-28T09:02:09.658Z [ERROR] server-97179f2b-2657-0420-8877-771b7e8e74a8: failed to make requestVote RPC: target="{Voter server-a53bbd30-d396-230a-d97b-996e5ee79a99 a53bbd30-d396-230a-d97b-996e5ee79a99}" error="failed to connect to peer: a53bbd30-d396-230a-d97b-996e5ee79a99" term=4
2026-02-28T09:02:09.658Z [INFO]  server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b: entering follower state: follower="Node at 4e0d1c77-1d55-b30e-24c3-d6c97371c65b [Follower]" leader-address= leader-id=
2026-02-28T09:02:09.659Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: election won: term=4 tally=2
2026-02-28T09:02:09.659Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: entering leader state: leader="Node at 97179f2b-2657-0420-8877-771b7e8e74a8 [Leader]"
2026-02-28T09:02:09.659Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: added peer, starting replication: peer=server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b
2026-02-28T09:02:09.659Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: added peer, starting replication: peer=server-a53bbd30-d396-230a-d97b-996e5ee79a99
2026-02-28T09:02:09.659Z [ERROR] server-97179f2b-2657-0420-8877-771b7e8e74a8: failed to appendEntries to: peer="{Voter server-a53bbd30-d396-230a-d97b-996e5ee79a99 a53bbd30-d396-230a-d97b-996e5ee79a99}" error="failed to connect to peer: a53bbd30-d396-230a-d97b-996e5ee79a99"
2026-02-28T09:02:09.659Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: pipelining replication: peer="{Voter server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b 4e0d1c77-1d55-b30e-24c3-d6c97371c65b}"
2026-02-28T09:02:09.668Z [ERROR] server-97179f2b-2657-0420-8877-771b7e8e74a8: failed to heartbeat to: peer=a53bbd30-d396-230a-d97b-996e5ee79a99 backoff time=10ms error="failed to connect to peer: a53bbd30-d396-230a-d97b-996e5ee79a99"
2026-02-28T09:02:09.674Z [WARN]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: Election timeout reached, restarting election
2026-02-28T09:02:09.674Z [INFO]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: entering candidate state: node="Node at a53bbd30-d396-230a-d97b-996e5ee79a99 [Candidate]" term=4
2026-02-28T09:02:09.674Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to make requestVote RPC: target="{Voter server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b 4e0d1c77-1d55-b30e-24c3-d6c97371c65b}" error="failed to connect to peer: 4e0d1c77-1d55-b30e-24c3-d6c97371c65b" term=4
2026-02-28T09:02:09.675Z [ERROR] server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to make requestVote RPC: target="{Voter server-97179f2b-2657-0420-8877-771b7e8e74a8 97179f2b-2657-0420-8877-771b7e8e74a8}" error="failed to connect to peer: 97179f2b-2657-0420-8877-771b7e8e74a8" term=4
2026-02-28T09:02:09.679Z [ERROR] server-97179f2b-2657-0420-8877-771b7e8e74a8: failed to appendEntries to: peer="{Voter server-a53bbd30-d396-230a-d97b-996e5ee79a99 a53bbd30-d396-230a-d97b-996e5ee79a99}" error="failed to connect to peer: a53bbd30-d396-230a-d97b-996e5ee79a99"
2026-02-28T09:02:09.685Z [ERROR] server-97179f2b-2657-0420-8877-771b7e8e74a8: failed to heartbeat to: peer=a53bbd30-d396-230a-d97b-996e5ee79a99 backoff time=10ms error="failed to connect to peer: a53bbd30-d396-230a-d97b-996e5ee79a99"
2026-02-28T09:02:09.697Z [ERROR] server-97179f2b-2657-0420-8877-771b7e8e74a8: failed to appendEntries to: peer="{Voter server-a53bbd30-d396-230a-d97b-996e5ee79a99 a53bbd30-d396-230a-d97b-996e5ee79a99}" error="failed to connect to peer: a53bbd30-d396-230a-d97b-996e5ee79a99"
2026-02-28T09:02:09.704Z [ERROR] server-97179f2b-2657-0420-8877-771b7e8e74a8: failed to heartbeat to: peer=a53bbd30-d396-230a-d97b-996e5ee79a99 backoff time=10ms error="failed to connect to peer: a53bbd30-d396-230a-d97b-996e5ee79a99"
2026-02-28T09:02:09.710Z [WARN]  server-97179f2b-2657-0420-8877-771b7e8e74a8: failed to contact: server-id=server-a53bbd30-d396-230a-d97b-996e5ee79a99 time=50.91752ms
2026-02-28T09:02:09.720Z [ERROR] server-97179f2b-2657-0420-8877-771b7e8e74a8: failed to heartbeat to: peer=a53bbd30-d396-230a-d97b-996e5ee79a99 backoff time=20ms error="failed to connect to peer: a53bbd30-d396-230a-d97b-996e5ee79a99"
2026-02-28T09:02:09.726Z [ERROR] server-97179f2b-2657-0420-8877-771b7e8e74a8: failed to appendEntries to: peer="{Voter server-a53bbd30-d396-230a-d97b-996e5ee79a99 a53bbd30-d396-230a-d97b-996e5ee79a99}" error="failed to connect to peer: a53bbd30-d396-230a-d97b-996e5ee79a99"
2026-02-28T09:02:09.750Z [ERROR] server-97179f2b-2657-0420-8877-771b7e8e74a8: failed to heartbeat to: peer=a53bbd30-d396-230a-d97b-996e5ee79a99 backoff time=25ms error="failed to connect to peer: a53bbd30-d396-230a-d97b-996e5ee79a99"
2026-02-28T09:02:09.757Z [WARN]  server-97179f2b-2657-0420-8877-771b7e8e74a8: failed to contact: server-id=server-a53bbd30-d396-230a-d97b-996e5ee79a99 time=97.92296ms
2026-02-28T09:02:09.764Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:09.659173302 +0000 UTC m=+7.622969841 (1 nodes), 153.09196ms from start of poll, 446.61704ms from cluster start. Timeout at 2026-02-28 09:02:09.764177262 +0000 UTC m=+7.727973801, 105.00396ms after stability
    raft_test.go:413: [INFO] Reconnecting Node at a53bbd30-d396-230a-d97b-996e5ee79a99 [Candidate]
2026-02-28T09:02:09.765Z [WARN]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: Election timeout reached, restarting election
2026-02-28T09:02:09.765Z [INFO]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: entering candidate state: node="Node at a53bbd30-d396-230a-d97b-996e5ee79a99 [Candidate]" term=5
2026-02-28T09:02:09.765Z [WARN]  server-97179f2b-2657-0420-8877-771b7e8e74a8: rejecting vote request since we have a leader: from=a53bbd30-d396-230a-d97b-996e5ee79a99 leader=97179f2b-2657-0420-8877-771b7e8e74a8 leader-id=server-97179f2b-2657-0420-8877-771b7e8e74a8
2026-02-28T09:02:09.765Z [WARN]  server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b: rejecting vote request since we have a leader: from=a53bbd30-d396-230a-d97b-996e5ee79a99 leader=97179f2b-2657-0420-8877-771b7e8e74a8 leader-id=server-97179f2b-2657-0420-8877-771b7e8e74a8
2026-02-28T09:02:09.773Z [ERROR] server-97179f2b-2657-0420-8877-771b7e8e74a8: peer has newer term, stopping replication: peer="{Voter server-a53bbd30-d396-230a-d97b-996e5ee79a99 a53bbd30-d396-230a-d97b-996e5ee79a99}"
2026-02-28T09:02:09.773Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: entering follower state: follower="Node at 97179f2b-2657-0420-8877-771b7e8e74a8 [Follower]" leader-address= leader-id=
2026-02-28T09:02:09.773Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: aborting pipeline replication: peer="{Voter server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b 4e0d1c77-1d55-b30e-24c3-d6c97371c65b}"
2026-02-28T09:02:09.827Z [WARN]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: Election timeout reached, restarting election
2026-02-28T09:02:09.827Z [INFO]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: entering candidate state: node="Node at a53bbd30-d396-230a-d97b-996e5ee79a99 [Candidate]" term=6
2026-02-28T09:02:09.827Z [WARN]  server-97179f2b-2657-0420-8877-771b7e8e74a8: rejecting vote request since our last term is greater: candidate=a53bbd30-d396-230a-d97b-996e5ee79a99 last-term=4 last-candidate-term=2
2026-02-28T09:02:09.827Z [WARN]  server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b: rejecting vote request since we have a leader: from=a53bbd30-d396-230a-d97b-996e5ee79a99 leader=97179f2b-2657-0420-8877-771b7e8e74a8 leader-id=server-97179f2b-2657-0420-8877-771b7e8e74a8
2026-02-28T09:02:09.870Z [WARN]  server-97179f2b-2657-0420-8877-771b7e8e74a8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:09.870Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: entering candidate state: node="Node at 97179f2b-2657-0420-8877-771b7e8e74a8 [Candidate]" term=7
2026-02-28T09:02:09.870Z [INFO]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: entering follower state: follower="Node at a53bbd30-d396-230a-d97b-996e5ee79a99 [Follower]" leader-address= leader-id=
2026-02-28T09:02:09.870Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: election won: term=7 tally=2
2026-02-28T09:02:09.870Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: entering leader state: leader="Node at 97179f2b-2657-0420-8877-771b7e8e74a8 [Leader]"
2026-02-28T09:02:09.870Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: added peer, starting replication: peer=server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b
2026-02-28T09:02:09.870Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: added peer, starting replication: peer=server-a53bbd30-d396-230a-d97b-996e5ee79a99
2026-02-28T09:02:09.871Z [WARN]  server-a53bbd30-d396-230a-d97b-996e5ee79a99: failed to get previous log: previous-index=5 last-index=3 error="log not found"
2026-02-28T09:02:09.871Z [WARN]  server-97179f2b-2657-0420-8877-771b7e8e74a8: appendEntries rejected, sending older logs: peer="{Voter server-a53bbd30-d396-230a-d97b-996e5ee79a99 a53bbd30-d396-230a-d97b-996e5ee79a99}" next=4
2026-02-28T09:02:09.871Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: pipelining replication: peer="{Voter server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b 4e0d1c77-1d55-b30e-24c3-d6c97371c65b}"
2026-02-28T09:02:09.871Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: pipelining replication: peer="{Voter server-a53bbd30-d396-230a-d97b-996e5ee79a99 a53bbd30-d396-230a-d97b-996e5ee79a99}"
2026-02-28T09:02:09.876Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: aborting pipeline replication: peer="{Voter server-a53bbd30-d396-230a-d97b-996e5ee79a99 a53bbd30-d396-230a-d97b-996e5ee79a99}"
2026-02-28T09:02:09.876Z [INFO]  server-97179f2b-2657-0420-8877-771b7e8e74a8: aborting pipeline replication: peer="{Voter server-4e0d1c77-1d55-b30e-24c3-d6c97371c65b 4e0d1c77-1d55-b30e-24c3-d6c97371c65b}"
--- PASS: TestRaft_LeaderFail (0.66s)
=== RUN   TestRaft_BehindFollower
2026-02-28T09:02:09.877Z [INFO]  server-4bd858c3-8374-453f-ae97-bbf27e4b8acc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4bd858c3-8374-453f-ae97-bbf27e4b8acc Address:4bd858c3-8374-453f-ae97-bbf27e4b8acc} {Suffrage:Voter ID:server-f214ea24-8cba-f6e8-3623-ed3fb73902a6 Address:f214ea24-8cba-f6e8-3623-ed3fb73902a6} {Suffrage:Voter ID:server-7d9e4a8a-ccd0-dd12-b4a4-295f1bdf56cf Address:7d9e4a8a-ccd0-dd12-b4a4-295f1bdf56cf}]"
2026-02-28T09:02:09.877Z [INFO]  server-4bd858c3-8374-453f-ae97-bbf27e4b8acc: entering follower state: follower="Node at 4bd858c3-8374-453f-ae97-bbf27e4b8acc [Follower]" leader-address= leader-id=
2026-02-28T09:02:09.877Z [INFO]  server-f214ea24-8cba-f6e8-3623-ed3fb73902a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4bd858c3-8374-453f-ae97-bbf27e4b8acc Address:4bd858c3-8374-453f-ae97-bbf27e4b8acc} {Suffrage:Voter ID:server-f214ea24-8cba-f6e8-3623-ed3fb73902a6 Address:f214ea24-8cba-f6e8-3623-ed3fb73902a6} {Suffrage:Voter ID:server-7d9e4a8a-ccd0-dd12-b4a4-295f1bdf56cf Address:7d9e4a8a-ccd0-dd12-b4a4-295f1bdf56cf}]"
2026-02-28T09:02:09.877Z [INFO]  server-f214ea24-8cba-f6e8-3623-ed3fb73902a6: entering follower state: follower="Node at f214ea24-8cba-f6e8-3623-ed3fb73902a6 [Follower]" leader-address= leader-id=
2026-02-28T09:02:09.877Z [INFO]  server-7d9e4a8a-ccd0-dd12-b4a4-295f1bdf56cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4bd858c3-8374-453f-ae97-bbf27e4b8acc Address:4bd858c3-8374-453f-ae97-bbf27e4b8acc} {Suffrage:Voter ID:server-f214ea24-8cba-f6e8-3623-ed3fb73902a6 Address:f214ea24-8cba-f6e8-3623-ed3fb73902a6} {Suffrage:Voter ID:server-7d9e4a8a-ccd0-dd12-b4a4-295f1bdf56cf Address:7d9e4a8a-ccd0-dd12-b4a4-295f1bdf56cf}]"
2026-02-28T09:02:09.877Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:09.877Z [INFO]  server-7d9e4a8a-ccd0-dd12-b4a4-295f1bdf56cf: entering follower state: follower="Node at 7d9e4a8a-ccd0-dd12-b4a4-295f1bdf56cf [Follower]" leader-address= leader-id=
2026-02-28T09:02:09.933Z [WARN]  server-f214ea24-8cba-f6e8-3623-ed3fb73902a6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:09.933Z [INFO]  server-f214ea24-8cba-f6e8-3623-ed3fb73902a6: entering candidate state: node="Node at f214ea24-8cba-f6e8-3623-ed3fb73902a6 [Candidate]" term=2
2026-02-28T09:02:09.933Z [INFO]  server-f214ea24-8cba-f6e8-3623-ed3fb73902a6: election won: term=2 tally=2
2026-02-28T09:02:09.933Z [INFO]  server-f214ea24-8cba-f6e8-3623-ed3fb73902a6: entering leader state: leader="Node at f214ea24-8cba-f6e8-3623-ed3fb73902a6 [Leader]"
2026-02-28T09:02:09.933Z [INFO]  server-f214ea24-8cba-f6e8-3623-ed3fb73902a6: added peer, starting replication: peer=server-4bd858c3-8374-453f-ae97-bbf27e4b8acc
2026-02-28T09:02:09.933Z [INFO]  server-f214ea24-8cba-f6e8-3623-ed3fb73902a6: added peer, starting replication: peer=server-7d9e4a8a-ccd0-dd12-b4a4-295f1bdf56cf
2026-02-28T09:02:09.933Z [INFO]  server-f214ea24-8cba-f6e8-3623-ed3fb73902a6: pipelining replication: peer="{Voter server-7d9e4a8a-ccd0-dd12-b4a4-295f1bdf56cf 7d9e4a8a-ccd0-dd12-b4a4-295f1bdf56cf}"
2026-02-28T09:02:09.933Z [INFO]  server-f214ea24-8cba-f6e8-3623-ed3fb73902a6: pipelining replication: peer="{Voter server-4bd858c3-8374-453f-ae97-bbf27e4b8acc 4bd858c3-8374-453f-ae97-bbf27e4b8acc}"
2026-02-28T09:02:10.038Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:09.933460022 +0000 UTC m=+7.897256561 (1 nodes), 55.84204ms from start of poll, 56.25596ms from cluster start. Timeout at 2026-02-28 09:02:10.038463222 +0000 UTC m=+8.002259761, 105.0032ms after stability
2026-02-28T09:02:10.038Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:10.144Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:10.038704862 +0000 UTC m=+8.002501401 (2 nodes), 1.48µs from start of poll, 161.5008ms from cluster start. Timeout at 2026-02-28 09:02:10.143711222 +0000 UTC m=+8.107507761, 105.00636ms after stability
2026-02-28T09:02:10.144Z [ERROR] server-f214ea24-8cba-f6e8-3623-ed3fb73902a6: failed to pipeline appendEntries: peer="{Voter server-4bd858c3-8374-453f-ae97-bbf27e4b8acc 4bd858c3-8374-453f-ae97-bbf27e4b8acc}" error="append pipeline closed"
2026-02-28T09:02:10.144Z [INFO]  server-f214ea24-8cba-f6e8-3623-ed3fb73902a6: aborting pipeline replication: peer="{Voter server-4bd858c3-8374-453f-ae97-bbf27e4b8acc 4bd858c3-8374-453f-ae97-bbf27e4b8acc}"
2026-02-28T09:02:10.144Z [ERROR] server-f214ea24-8cba-f6e8-3623-ed3fb73902a6: failed to appendEntries to: peer="{Voter server-4bd858c3-8374-453f-ae97-bbf27e4b8acc 4bd858c3-8374-453f-ae97-bbf27e4b8acc}" error="failed to connect to peer: 4bd858c3-8374-453f-ae97-bbf27e4b8acc"
    raft_test.go:459: [INFO] Finished apply without behind follower
2026-02-28T09:02:10.154Z [INFO]  server-f214ea24-8cba-f6e8-3623-ed3fb73902a6: pipelining replication: peer="{Voter server-4bd858c3-8374-453f-ae97-bbf27e4b8acc 4bd858c3-8374-453f-ae97-bbf27e4b8acc}"
2026-02-28T09:02:10.165Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:10.271Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:10.165933742 +0000 UTC m=+8.129730321 (1 nodes), 2.56µs from start of poll, 288.72972ms from cluster start. Timeout at 2026-02-28 09:02:10.270938182 +0000 UTC m=+8.234734721, 105.0044ms after stability
2026-02-28T09:02:10.271Z [INFO]  server-f214ea24-8cba-f6e8-3623-ed3fb73902a6: aborting pipeline replication: peer="{Voter server-7d9e4a8a-ccd0-dd12-b4a4-295f1bdf56cf 7d9e4a8a-ccd0-dd12-b4a4-295f1bdf56cf}"
2026-02-28T09:02:10.271Z [INFO]  server-f214ea24-8cba-f6e8-3623-ed3fb73902a6: aborting pipeline replication: peer="{Voter server-4bd858c3-8374-453f-ae97-bbf27e4b8acc 4bd858c3-8374-453f-ae97-bbf27e4b8acc}"
--- PASS: TestRaft_BehindFollower (0.40s)
=== RUN   TestRaft_ApplyNonLeader
2026-02-28T09:02:10.272Z [INFO]  server-b909efe0-3647-6583-4865-b5beef1e81bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b909efe0-3647-6583-4865-b5beef1e81bc Address:b909efe0-3647-6583-4865-b5beef1e81bc} {Suffrage:Voter ID:server-4868dce3-40ca-f6e4-6ea9-21655896dad7 Address:4868dce3-40ca-f6e4-6ea9-21655896dad7} {Suffrage:Voter ID:server-55dadbbb-70cd-4c7f-faa7-09e3335b04de Address:55dadbbb-70cd-4c7f-faa7-09e3335b04de}]"
2026-02-28T09:02:10.272Z [INFO]  server-4868dce3-40ca-f6e4-6ea9-21655896dad7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b909efe0-3647-6583-4865-b5beef1e81bc Address:b909efe0-3647-6583-4865-b5beef1e81bc} {Suffrage:Voter ID:server-4868dce3-40ca-f6e4-6ea9-21655896dad7 Address:4868dce3-40ca-f6e4-6ea9-21655896dad7} {Suffrage:Voter ID:server-55dadbbb-70cd-4c7f-faa7-09e3335b04de Address:55dadbbb-70cd-4c7f-faa7-09e3335b04de}]"
2026-02-28T09:02:10.272Z [INFO]  server-b909efe0-3647-6583-4865-b5beef1e81bc: entering follower state: follower="Node at b909efe0-3647-6583-4865-b5beef1e81bc [Follower]" leader-address= leader-id=
2026-02-28T09:02:10.273Z [INFO]  server-55dadbbb-70cd-4c7f-faa7-09e3335b04de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b909efe0-3647-6583-4865-b5beef1e81bc Address:b909efe0-3647-6583-4865-b5beef1e81bc} {Suffrage:Voter ID:server-4868dce3-40ca-f6e4-6ea9-21655896dad7 Address:4868dce3-40ca-f6e4-6ea9-21655896dad7} {Suffrage:Voter ID:server-55dadbbb-70cd-4c7f-faa7-09e3335b04de Address:55dadbbb-70cd-4c7f-faa7-09e3335b04de}]"
2026-02-28T09:02:10.273Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:10.273Z [INFO]  server-55dadbbb-70cd-4c7f-faa7-09e3335b04de: entering follower state: follower="Node at 55dadbbb-70cd-4c7f-faa7-09e3335b04de [Follower]" leader-address= leader-id=
2026-02-28T09:02:10.273Z [INFO]  server-4868dce3-40ca-f6e4-6ea9-21655896dad7: entering follower state: follower="Node at 4868dce3-40ca-f6e4-6ea9-21655896dad7 [Follower]" leader-address= leader-id=
2026-02-28T09:02:10.324Z [WARN]  server-55dadbbb-70cd-4c7f-faa7-09e3335b04de: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:10.324Z [INFO]  server-55dadbbb-70cd-4c7f-faa7-09e3335b04de: entering candidate state: node="Node at 55dadbbb-70cd-4c7f-faa7-09e3335b04de [Candidate]" term=2
2026-02-28T09:02:10.324Z [INFO]  server-55dadbbb-70cd-4c7f-faa7-09e3335b04de: election won: term=2 tally=2
2026-02-28T09:02:10.324Z [INFO]  server-55dadbbb-70cd-4c7f-faa7-09e3335b04de: entering leader state: leader="Node at 55dadbbb-70cd-4c7f-faa7-09e3335b04de [Leader]"
2026-02-28T09:02:10.324Z [INFO]  server-55dadbbb-70cd-4c7f-faa7-09e3335b04de: added peer, starting replication: peer=server-b909efe0-3647-6583-4865-b5beef1e81bc
2026-02-28T09:02:10.324Z [INFO]  server-55dadbbb-70cd-4c7f-faa7-09e3335b04de: added peer, starting replication: peer=server-4868dce3-40ca-f6e4-6ea9-21655896dad7
2026-02-28T09:02:10.325Z [INFO]  server-55dadbbb-70cd-4c7f-faa7-09e3335b04de: pipelining replication: peer="{Voter server-4868dce3-40ca-f6e4-6ea9-21655896dad7 4868dce3-40ca-f6e4-6ea9-21655896dad7}"
2026-02-28T09:02:10.325Z [INFO]  server-55dadbbb-70cd-4c7f-faa7-09e3335b04de: pipelining replication: peer="{Voter server-b909efe0-3647-6583-4865-b5beef1e81bc b909efe0-3647-6583-4865-b5beef1e81bc}"
2026-02-28T09:02:10.430Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:10.324851902 +0000 UTC m=+8.288648481 (1 nodes), 51.80684ms from start of poll, 52.21568ms from cluster start. Timeout at 2026-02-28 09:02:10.429856022 +0000 UTC m=+8.393652561, 105.00408ms after stability
2026-02-28T09:02:10.430Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:10.535Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:10.430836622 +0000 UTC m=+8.394633161 (2 nodes), 1.56µs from start of poll, 158.20036ms from cluster start. Timeout at 2026-02-28 09:02:10.535843782 +0000 UTC m=+8.499640321, 105.00716ms after stability
2026-02-28T09:02:10.536Z [INFO]  server-55dadbbb-70cd-4c7f-faa7-09e3335b04de: aborting pipeline replication: peer="{Voter server-4868dce3-40ca-f6e4-6ea9-21655896dad7 4868dce3-40ca-f6e4-6ea9-21655896dad7}"
2026-02-28T09:02:10.536Z [INFO]  server-55dadbbb-70cd-4c7f-faa7-09e3335b04de: aborting pipeline replication: peer="{Voter server-b909efe0-3647-6583-4865-b5beef1e81bc b909efe0-3647-6583-4865-b5beef1e81bc}"
--- PASS: TestRaft_ApplyNonLeader (0.26s)
=== RUN   TestRaft_ApplyConcurrent
2026-02-28T09:02:10.537Z [INFO]  server-4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0 Address:4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0} {Suffrage:Voter ID:server-bc40e10f-68eb-5c58-d35d-81d86b868fb2 Address:bc40e10f-68eb-5c58-d35d-81d86b868fb2} {Suffrage:Voter ID:server-14e09785-9c76-0fe1-6457-cce11261582d Address:14e09785-9c76-0fe1-6457-cce11261582d}]"
2026-02-28T09:02:10.537Z [INFO]  server-bc40e10f-68eb-5c58-d35d-81d86b868fb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0 Address:4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0} {Suffrage:Voter ID:server-bc40e10f-68eb-5c58-d35d-81d86b868fb2 Address:bc40e10f-68eb-5c58-d35d-81d86b868fb2} {Suffrage:Voter ID:server-14e09785-9c76-0fe1-6457-cce11261582d Address:14e09785-9c76-0fe1-6457-cce11261582d}]"
2026-02-28T09:02:10.537Z [INFO]  server-4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0: entering follower state: follower="Node at 4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0 [Follower]" leader-address= leader-id=
2026-02-28T09:02:10.537Z [INFO]  server-14e09785-9c76-0fe1-6457-cce11261582d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0 Address:4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0} {Suffrage:Voter ID:server-bc40e10f-68eb-5c58-d35d-81d86b868fb2 Address:bc40e10f-68eb-5c58-d35d-81d86b868fb2} {Suffrage:Voter ID:server-14e09785-9c76-0fe1-6457-cce11261582d Address:14e09785-9c76-0fe1-6457-cce11261582d}]"
2026-02-28T09:02:10.537Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:10.537Z [INFO]  server-bc40e10f-68eb-5c58-d35d-81d86b868fb2: entering follower state: follower="Node at bc40e10f-68eb-5c58-d35d-81d86b868fb2 [Follower]" leader-address= leader-id=
2026-02-28T09:02:10.537Z [INFO]  server-14e09785-9c76-0fe1-6457-cce11261582d: entering follower state: follower="Node at 14e09785-9c76-0fe1-6457-cce11261582d [Follower]" leader-address= leader-id=
2026-02-28T09:02:10.650Z [WARN]  server-4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:10.650Z [INFO]  server-4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0: entering candidate state: node="Node at 4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0 [Candidate]" term=2
2026-02-28T09:02:10.650Z [INFO]  server-4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0: election won: term=2 tally=2
2026-02-28T09:02:10.650Z [INFO]  server-4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0: entering leader state: leader="Node at 4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0 [Leader]"
2026-02-28T09:02:10.650Z [INFO]  server-4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0: added peer, starting replication: peer=server-bc40e10f-68eb-5c58-d35d-81d86b868fb2
2026-02-28T09:02:10.650Z [INFO]  server-4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0: added peer, starting replication: peer=server-14e09785-9c76-0fe1-6457-cce11261582d
2026-02-28T09:02:10.650Z [INFO]  server-4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0: pipelining replication: peer="{Voter server-14e09785-9c76-0fe1-6457-cce11261582d 14e09785-9c76-0fe1-6457-cce11261582d}"
2026-02-28T09:02:10.650Z [INFO]  server-4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0: pipelining replication: peer="{Voter server-bc40e10f-68eb-5c58-d35d-81d86b868fb2 bc40e10f-68eb-5c58-d35d-81d86b868fb2}"
2026-02-28T09:02:10.855Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:10.650439382 +0000 UTC m=+8.614235921 (1 nodes), 113.07992ms from start of poll, 113.5254ms from cluster start. Timeout at 2026-02-28 09:02:10.855442142 +0000 UTC m=+8.819238721, 205.0028ms after stability
2026-02-28T09:02:10.871Z [INFO]  server-4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0: aborting pipeline replication: peer="{Voter server-bc40e10f-68eb-5c58-d35d-81d86b868fb2 bc40e10f-68eb-5c58-d35d-81d86b868fb2}"
2026-02-28T09:02:10.871Z [INFO]  server-4169f7e6-f6e3-9be7-ffc4-df6ed358e2f0: aborting pipeline replication: peer="{Voter server-14e09785-9c76-0fe1-6457-cce11261582d 14e09785-9c76-0fe1-6457-cce11261582d}"
--- PASS: TestRaft_ApplyConcurrent (0.33s)
=== RUN   TestRaft_ApplyConcurrent_Timeout
2026-02-28T09:02:10.871Z [INFO]  server-3317513e-3ffb-4e32-b48c-222943726d05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3317513e-3ffb-4e32-b48c-222943726d05 Address:3317513e-3ffb-4e32-b48c-222943726d05}]"
2026-02-28T09:02:10.871Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:10.871Z [INFO]  server-3317513e-3ffb-4e32-b48c-222943726d05: entering follower state: follower="Node at 3317513e-3ffb-4e32-b48c-222943726d05 [Follower]" leader-address= leader-id=
2026-02-28T09:02:11.062Z [WARN]  server-3317513e-3ffb-4e32-b48c-222943726d05: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:11.063Z [INFO]  server-3317513e-3ffb-4e32-b48c-222943726d05: entering candidate state: node="Node at 3317513e-3ffb-4e32-b48c-222943726d05 [Candidate]" term=2
2026-02-28T09:02:11.063Z [INFO]  server-3317513e-3ffb-4e32-b48c-222943726d05: election won: term=2 tally=1
2026-02-28T09:02:11.063Z [INFO]  server-3317513e-3ffb-4e32-b48c-222943726d05: entering leader state: leader="Node at 3317513e-3ffb-4e32-b48c-222943726d05 [Leader]"
2026-02-28T09:02:11.264Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:11.063182062 +0000 UTC m=+9.026978601 (1 nodes), 191.23608ms from start of poll, 191.45008ms from cluster start. Timeout at 2026-02-28 09:02:11.264183782 +0000 UTC m=+9.227980321, 201.00172ms after stability
--- PASS: TestRaft_ApplyConcurrent_Timeout (0.39s)
=== RUN   TestRaft_JoinNode
2026-02-28T09:02:11.265Z [INFO]  server-48272baa-c75a-6058-7d93-1e11690586fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48272baa-c75a-6058-7d93-1e11690586fc Address:48272baa-c75a-6058-7d93-1e11690586fc} {Suffrage:Voter ID:server-0278cf51-2c4f-9bf9-c192-10d12a17c591 Address:0278cf51-2c4f-9bf9-c192-10d12a17c591}]"
2026-02-28T09:02:11.265Z [INFO]  server-0278cf51-2c4f-9bf9-c192-10d12a17c591: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48272baa-c75a-6058-7d93-1e11690586fc Address:48272baa-c75a-6058-7d93-1e11690586fc} {Suffrage:Voter ID:server-0278cf51-2c4f-9bf9-c192-10d12a17c591 Address:0278cf51-2c4f-9bf9-c192-10d12a17c591}]"
2026-02-28T09:02:11.265Z [INFO]  server-48272baa-c75a-6058-7d93-1e11690586fc: entering follower state: follower="Node at 48272baa-c75a-6058-7d93-1e11690586fc [Follower]" leader-address= leader-id=
2026-02-28T09:02:11.265Z [INFO]  server-0278cf51-2c4f-9bf9-c192-10d12a17c591: entering follower state: follower="Node at 0278cf51-2c4f-9bf9-c192-10d12a17c591 [Follower]" leader-address= leader-id=
2026-02-28T09:02:11.265Z [INFO]  server-a813b8d8-968e-4864-b91d-38912e4a9832: initial configuration: index=0 servers=[]
2026-02-28T09:02:11.266Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:11.266Z [INFO]  server-a813b8d8-968e-4864-b91d-38912e4a9832: entering follower state: follower="Node at a813b8d8-968e-4864-b91d-38912e4a9832 [Follower]" leader-address= leader-id=
2026-02-28T09:02:11.333Z [WARN]  server-48272baa-c75a-6058-7d93-1e11690586fc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:11.333Z [INFO]  server-48272baa-c75a-6058-7d93-1e11690586fc: entering candidate state: node="Node at 48272baa-c75a-6058-7d93-1e11690586fc [Candidate]" term=2
2026-02-28T09:02:11.333Z [INFO]  server-48272baa-c75a-6058-7d93-1e11690586fc: election won: term=2 tally=2
2026-02-28T09:02:11.333Z [INFO]  server-48272baa-c75a-6058-7d93-1e11690586fc: entering leader state: leader="Node at 48272baa-c75a-6058-7d93-1e11690586fc [Leader]"
2026-02-28T09:02:11.333Z [INFO]  server-48272baa-c75a-6058-7d93-1e11690586fc: added peer, starting replication: peer=server-0278cf51-2c4f-9bf9-c192-10d12a17c591
2026-02-28T09:02:11.333Z [INFO]  server-48272baa-c75a-6058-7d93-1e11690586fc: pipelining replication: peer="{Voter server-0278cf51-2c4f-9bf9-c192-10d12a17c591 0278cf51-2c4f-9bf9-c192-10d12a17c591}"
2026-02-28T09:02:11.355Z [WARN]  server-a813b8d8-968e-4864-b91d-38912e4a9832: no known peers, aborting election
2026-02-28T09:02:11.439Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:11.333801622 +0000 UTC m=+9.297598201 (1 nodes), 67.75572ms from start of poll, 68.23792ms from cluster start. Timeout at 2026-02-28 09:02:11.438803662 +0000 UTC m=+9.402600201, 105.002ms after stability
2026-02-28T09:02:11.439Z [INFO]  server-48272baa-c75a-6058-7d93-1e11690586fc: updating configuration: command=AddVoter server-id=server-a813b8d8-968e-4864-b91d-38912e4a9832 server-addr=a813b8d8-968e-4864-b91d-38912e4a9832 servers="[{Suffrage:Voter ID:server-48272baa-c75a-6058-7d93-1e11690586fc Address:48272baa-c75a-6058-7d93-1e11690586fc} {Suffrage:Voter ID:server-0278cf51-2c4f-9bf9-c192-10d12a17c591 Address:0278cf51-2c4f-9bf9-c192-10d12a17c591} {Suffrage:Voter ID:server-a813b8d8-968e-4864-b91d-38912e4a9832 Address:a813b8d8-968e-4864-b91d-38912e4a9832}]"
2026-02-28T09:02:11.439Z [INFO]  server-48272baa-c75a-6058-7d93-1e11690586fc: added peer, starting replication: peer=server-a813b8d8-968e-4864-b91d-38912e4a9832
2026-02-28T09:02:11.439Z [WARN]  server-a813b8d8-968e-4864-b91d-38912e4a9832: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2026-02-28T09:02:11.439Z [WARN]  server-48272baa-c75a-6058-7d93-1e11690586fc: appendEntries rejected, sending older logs: peer="{Voter server-a813b8d8-968e-4864-b91d-38912e4a9832 a813b8d8-968e-4864-b91d-38912e4a9832}" next=1
2026-02-28T09:02:11.440Z [INFO]  server-48272baa-c75a-6058-7d93-1e11690586fc: pipelining replication: peer="{Voter server-a813b8d8-968e-4864-b91d-38912e4a9832 a813b8d8-968e-4864-b91d-38912e4a9832}"
2026-02-28T09:02:11.440Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:11.545Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:11.440275302 +0000 UTC m=+9.404071841 (1 nodes), 2.96µs from start of poll, 174.71156ms from cluster start. Timeout at 2026-02-28 09:02:11.545279262 +0000 UTC m=+9.509075801, 105.00396ms after stability
2026-02-28T09:02:11.545Z [INFO]  server-48272baa-c75a-6058-7d93-1e11690586fc: aborting pipeline replication: peer="{Voter server-a813b8d8-968e-4864-b91d-38912e4a9832 a813b8d8-968e-4864-b91d-38912e4a9832}"
2026-02-28T09:02:11.545Z [INFO]  server-48272baa-c75a-6058-7d93-1e11690586fc: aborting pipeline replication: peer="{Voter server-0278cf51-2c4f-9bf9-c192-10d12a17c591 0278cf51-2c4f-9bf9-c192-10d12a17c591}"
--- PASS: TestRaft_JoinNode (0.28s)
=== RUN   TestRaft_JoinNode_ConfigStore
2026-02-28T09:02:11.546Z [INFO]  server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07 Address:296e8a5b-7d99-5aa8-8652-f8d5cc727e07}]"
2026-02-28T09:02:11.546Z [INFO]  server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07: entering follower state: follower="Node at 296e8a5b-7d99-5aa8-8652-f8d5cc727e07 [Follower]" leader-address= leader-id=
2026-02-28T09:02:11.546Z [INFO]  server-7c471442-e5c2-0a00-1838-0c940df0f613: initial configuration: index=0 servers=[]
2026-02-28T09:02:11.546Z [INFO]  server-7c471442-e5c2-0a00-1838-0c940df0f613: entering follower state: follower="Node at 7c471442-e5c2-0a00-1838-0c940df0f613 [Follower]" leader-address= leader-id=
2026-02-28T09:02:11.546Z [INFO]  server-f39d1a53-7d41-32c7-fd39-7eb3d2d219ed: initial configuration: index=0 servers=[]
2026-02-28T09:02:11.546Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:11.546Z [INFO]  server-f39d1a53-7d41-32c7-fd39-7eb3d2d219ed: entering follower state: follower="Node at f39d1a53-7d41-32c7-fd39-7eb3d2d219ed [Follower]" leader-address= leader-id=
2026-02-28T09:02:11.599Z [WARN]  server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:11.599Z [INFO]  server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07: entering candidate state: node="Node at 296e8a5b-7d99-5aa8-8652-f8d5cc727e07 [Candidate]" term=2
2026-02-28T09:02:11.599Z [INFO]  server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07: election won: term=2 tally=1
2026-02-28T09:02:11.599Z [INFO]  server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07: entering leader state: leader="Node at 296e8a5b-7d99-5aa8-8652-f8d5cc727e07 [Leader]"
2026-02-28T09:02:11.604Z [WARN]  server-f39d1a53-7d41-32c7-fd39-7eb3d2d219ed: no known peers, aborting election
2026-02-28T09:02:11.637Z [WARN]  server-7c471442-e5c2-0a00-1838-0c940df0f613: no known peers, aborting election
2026-02-28T09:02:11.705Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:11.600022982 +0000 UTC m=+9.563819521 (1 nodes), 53.4024ms from start of poll, 53.87436ms from cluster start. Timeout at 2026-02-28 09:02:11.705027302 +0000 UTC m=+9.668823841, 105.00432ms after stability
2026-02-28T09:02:11.705Z [INFO]  server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07: updating configuration: command=AddVoter server-id=server-7c471442-e5c2-0a00-1838-0c940df0f613 server-addr=7c471442-e5c2-0a00-1838-0c940df0f613 servers="[{Suffrage:Voter ID:server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07 Address:296e8a5b-7d99-5aa8-8652-f8d5cc727e07} {Suffrage:Voter ID:server-7c471442-e5c2-0a00-1838-0c940df0f613 Address:7c471442-e5c2-0a00-1838-0c940df0f613}]"
2026-02-28T09:02:11.705Z [INFO]  server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07: added peer, starting replication: peer=server-7c471442-e5c2-0a00-1838-0c940df0f613
2026-02-28T09:02:11.705Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:11.705Z [WARN]  server-7c471442-e5c2-0a00-1838-0c940df0f613: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2026-02-28T09:02:11.706Z [WARN]  server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07: appendEntries rejected, sending older logs: peer="{Voter server-7c471442-e5c2-0a00-1838-0c940df0f613 7c471442-e5c2-0a00-1838-0c940df0f613}" next=1
2026-02-28T09:02:11.706Z [INFO]  server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07: pipelining replication: peer="{Voter server-7c471442-e5c2-0a00-1838-0c940df0f613 7c471442-e5c2-0a00-1838-0c940df0f613}"
2026-02-28T09:02:11.811Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:11.705952862 +0000 UTC m=+9.669749401 (1 nodes), 840ns from start of poll, 159.80424ms from cluster start. Timeout at 2026-02-28 09:02:11.810959022 +0000 UTC m=+9.774755561, 105.00616ms after stability
2026-02-28T09:02:11.811Z [INFO]  server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07: updating configuration: command=AddVoter server-id=server-f39d1a53-7d41-32c7-fd39-7eb3d2d219ed server-addr=f39d1a53-7d41-32c7-fd39-7eb3d2d219ed servers="[{Suffrage:Voter ID:server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07 Address:296e8a5b-7d99-5aa8-8652-f8d5cc727e07} {Suffrage:Voter ID:server-7c471442-e5c2-0a00-1838-0c940df0f613 Address:7c471442-e5c2-0a00-1838-0c940df0f613} {Suffrage:Voter ID:server-f39d1a53-7d41-32c7-fd39-7eb3d2d219ed Address:f39d1a53-7d41-32c7-fd39-7eb3d2d219ed}]"
2026-02-28T09:02:11.811Z [INFO]  server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07: added peer, starting replication: peer=server-f39d1a53-7d41-32c7-fd39-7eb3d2d219ed
2026-02-28T09:02:11.811Z [WARN]  server-f39d1a53-7d41-32c7-fd39-7eb3d2d219ed: failed to get previous log: previous-index=4 last-index=0 error="log not found"
2026-02-28T09:02:11.811Z [WARN]  server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07: appendEntries rejected, sending older logs: peer="{Voter server-f39d1a53-7d41-32c7-fd39-7eb3d2d219ed f39d1a53-7d41-32c7-fd39-7eb3d2d219ed}" next=1
2026-02-28T09:02:11.811Z [INFO]  server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07: pipelining replication: peer="{Voter server-f39d1a53-7d41-32c7-fd39-7eb3d2d219ed f39d1a53-7d41-32c7-fd39-7eb3d2d219ed}"
2026-02-28T09:02:11.811Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:11.917Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:11.811573262 +0000 UTC m=+9.775369801 (1 nodes), 1µs from start of poll, 265.42464ms from cluster start. Timeout at 2026-02-28 09:02:11.916580342 +0000 UTC m=+9.880376881, 105.00708ms after stability
2026-02-28T09:02:11.917Z [INFO]  server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07: aborting pipeline replication: peer="{Voter server-f39d1a53-7d41-32c7-fd39-7eb3d2d219ed f39d1a53-7d41-32c7-fd39-7eb3d2d219ed}"
2026-02-28T09:02:11.917Z [INFO]  server-296e8a5b-7d99-5aa8-8652-f8d5cc727e07: aborting pipeline replication: peer="{Voter server-7c471442-e5c2-0a00-1838-0c940df0f613 7c471442-e5c2-0a00-1838-0c940df0f613}"
--- PASS: TestRaft_JoinNode_ConfigStore (0.37s)
=== RUN   TestRaft_RemoveFollower
2026-02-28T09:02:11.918Z [INFO]  server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d Address:5e410290-b3bb-4a3a-5776-bbce6c5d0c5d} {Suffrage:Voter ID:server-e551150c-3ba2-ebd6-503a-165aa95c0f8c Address:e551150c-3ba2-ebd6-503a-165aa95c0f8c} {Suffrage:Voter ID:server-1741453a-105d-d2b9-43be-5d23ec820426 Address:1741453a-105d-d2b9-43be-5d23ec820426}]"
2026-02-28T09:02:11.918Z [INFO]  server-e551150c-3ba2-ebd6-503a-165aa95c0f8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d Address:5e410290-b3bb-4a3a-5776-bbce6c5d0c5d} {Suffrage:Voter ID:server-e551150c-3ba2-ebd6-503a-165aa95c0f8c Address:e551150c-3ba2-ebd6-503a-165aa95c0f8c} {Suffrage:Voter ID:server-1741453a-105d-d2b9-43be-5d23ec820426 Address:1741453a-105d-d2b9-43be-5d23ec820426}]"
2026-02-28T09:02:11.918Z [INFO]  server-1741453a-105d-d2b9-43be-5d23ec820426: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d Address:5e410290-b3bb-4a3a-5776-bbce6c5d0c5d} {Suffrage:Voter ID:server-e551150c-3ba2-ebd6-503a-165aa95c0f8c Address:e551150c-3ba2-ebd6-503a-165aa95c0f8c} {Suffrage:Voter ID:server-1741453a-105d-d2b9-43be-5d23ec820426 Address:1741453a-105d-d2b9-43be-5d23ec820426}]"
2026-02-28T09:02:11.918Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:11.918Z [INFO]  server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d: entering follower state: follower="Node at 5e410290-b3bb-4a3a-5776-bbce6c5d0c5d [Follower]" leader-address= leader-id=
2026-02-28T09:02:11.918Z [INFO]  server-e551150c-3ba2-ebd6-503a-165aa95c0f8c: entering follower state: follower="Node at e551150c-3ba2-ebd6-503a-165aa95c0f8c [Follower]" leader-address= leader-id=
2026-02-28T09:02:11.918Z [INFO]  server-1741453a-105d-d2b9-43be-5d23ec820426: entering follower state: follower="Node at 1741453a-105d-d2b9-43be-5d23ec820426 [Follower]" leader-address= leader-id=
2026-02-28T09:02:11.991Z [WARN]  server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:11.992Z [INFO]  server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d: entering candidate state: node="Node at 5e410290-b3bb-4a3a-5776-bbce6c5d0c5d [Candidate]" term=2
2026-02-28T09:02:11.992Z [INFO]  server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d: election won: term=2 tally=2
2026-02-28T09:02:11.992Z [INFO]  server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d: entering leader state: leader="Node at 5e410290-b3bb-4a3a-5776-bbce6c5d0c5d [Leader]"
2026-02-28T09:02:11.992Z [INFO]  server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d: added peer, starting replication: peer=server-e551150c-3ba2-ebd6-503a-165aa95c0f8c
2026-02-28T09:02:11.992Z [INFO]  server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d: added peer, starting replication: peer=server-1741453a-105d-d2b9-43be-5d23ec820426
2026-02-28T09:02:11.992Z [INFO]  server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d: pipelining replication: peer="{Voter server-1741453a-105d-d2b9-43be-5d23ec820426 1741453a-105d-d2b9-43be-5d23ec820426}"
2026-02-28T09:02:11.992Z [INFO]  server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d: pipelining replication: peer="{Voter server-e551150c-3ba2-ebd6-503a-165aa95c0f8c e551150c-3ba2-ebd6-503a-165aa95c0f8c}"
2026-02-28T09:02:12.097Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:11.992271902 +0000 UTC m=+9.956068401 (1 nodes), 73.62776ms from start of poll, 74.04788ms from cluster start. Timeout at 2026-02-28 09:02:12.097274382 +0000 UTC m=+10.061070921, 105.00252ms after stability
2026-02-28T09:02:12.103Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:12.209Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:12.103472982 +0000 UTC m=+10.067269521 (2 nodes), 2.8µs from start of poll, 185.249ms from cluster start. Timeout at 2026-02-28 09:02:12.208477782 +0000 UTC m=+10.172274321, 105.0048ms after stability
2026-02-28T09:02:12.209Z [INFO]  server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d: updating configuration: command=RemoveServer server-id=server-e551150c-3ba2-ebd6-503a-165aa95c0f8c server-addr= servers="[{Suffrage:Voter ID:server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d Address:5e410290-b3bb-4a3a-5776-bbce6c5d0c5d} {Suffrage:Voter ID:server-1741453a-105d-d2b9-43be-5d23ec820426 Address:1741453a-105d-d2b9-43be-5d23ec820426}]"
2026-02-28T09:02:12.209Z [INFO]  server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d: removed peer, stopping replication: peer=server-e551150c-3ba2-ebd6-503a-165aa95c0f8c last-index=3
2026-02-28T09:02:12.209Z [INFO]  server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d: aborting pipeline replication: peer="{Voter server-e551150c-3ba2-ebd6-503a-165aa95c0f8c e551150c-3ba2-ebd6-503a-165aa95c0f8c}"
2026-02-28T09:02:12.284Z [WARN]  server-e551150c-3ba2-ebd6-503a-165aa95c0f8c: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2026-02-28T09:02:12.315Z [INFO]  server-5e410290-b3bb-4a3a-5776-bbce6c5d0c5d: aborting pipeline replication: peer="{Voter server-1741453a-105d-d2b9-43be-5d23ec820426 1741453a-105d-d2b9-43be-5d23ec820426}"
--- PASS: TestRaft_RemoveFollower (0.40s)
=== RUN   TestRaft_RemoveLeader
2026-02-28T09:02:12.316Z [INFO]  server-57dab213-4886-50f0-e42e-679147794d18: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57dab213-4886-50f0-e42e-679147794d18 Address:57dab213-4886-50f0-e42e-679147794d18} {Suffrage:Voter ID:server-f90494ca-a0ba-bf8f-f653-c504fd7388bf Address:f90494ca-a0ba-bf8f-f653-c504fd7388bf} {Suffrage:Voter ID:server-daf4e2d0-d66f-beab-ce03-f984a5facb13 Address:daf4e2d0-d66f-beab-ce03-f984a5facb13}]"
2026-02-28T09:02:12.316Z [INFO]  server-f90494ca-a0ba-bf8f-f653-c504fd7388bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57dab213-4886-50f0-e42e-679147794d18 Address:57dab213-4886-50f0-e42e-679147794d18} {Suffrage:Voter ID:server-f90494ca-a0ba-bf8f-f653-c504fd7388bf Address:f90494ca-a0ba-bf8f-f653-c504fd7388bf} {Suffrage:Voter ID:server-daf4e2d0-d66f-beab-ce03-f984a5facb13 Address:daf4e2d0-d66f-beab-ce03-f984a5facb13}]"
2026-02-28T09:02:12.316Z [INFO]  server-57dab213-4886-50f0-e42e-679147794d18: entering follower state: follower="Node at 57dab213-4886-50f0-e42e-679147794d18 [Follower]" leader-address= leader-id=
2026-02-28T09:02:12.316Z [INFO]  server-f90494ca-a0ba-bf8f-f653-c504fd7388bf: entering follower state: follower="Node at f90494ca-a0ba-bf8f-f653-c504fd7388bf [Follower]" leader-address= leader-id=
2026-02-28T09:02:12.316Z [INFO]  server-daf4e2d0-d66f-beab-ce03-f984a5facb13: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57dab213-4886-50f0-e42e-679147794d18 Address:57dab213-4886-50f0-e42e-679147794d18} {Suffrage:Voter ID:server-f90494ca-a0ba-bf8f-f653-c504fd7388bf Address:f90494ca-a0ba-bf8f-f653-c504fd7388bf} {Suffrage:Voter ID:server-daf4e2d0-d66f-beab-ce03-f984a5facb13 Address:daf4e2d0-d66f-beab-ce03-f984a5facb13}]"
2026-02-28T09:02:12.316Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:12.316Z [INFO]  server-daf4e2d0-d66f-beab-ce03-f984a5facb13: entering follower state: follower="Node at daf4e2d0-d66f-beab-ce03-f984a5facb13 [Follower]" leader-address= leader-id=
2026-02-28T09:02:12.386Z [WARN]  server-f90494ca-a0ba-bf8f-f653-c504fd7388bf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:12.386Z [INFO]  server-f90494ca-a0ba-bf8f-f653-c504fd7388bf: entering candidate state: node="Node at f90494ca-a0ba-bf8f-f653-c504fd7388bf [Candidate]" term=2
2026-02-28T09:02:12.386Z [INFO]  server-f90494ca-a0ba-bf8f-f653-c504fd7388bf: election won: term=2 tally=2
2026-02-28T09:02:12.386Z [INFO]  server-f90494ca-a0ba-bf8f-f653-c504fd7388bf: entering leader state: leader="Node at f90494ca-a0ba-bf8f-f653-c504fd7388bf [Leader]"
2026-02-28T09:02:12.386Z [INFO]  server-f90494ca-a0ba-bf8f-f653-c504fd7388bf: added peer, starting replication: peer=server-57dab213-4886-50f0-e42e-679147794d18
2026-02-28T09:02:12.386Z [INFO]  server-f90494ca-a0ba-bf8f-f653-c504fd7388bf: added peer, starting replication: peer=server-daf4e2d0-d66f-beab-ce03-f984a5facb13
2026-02-28T09:02:12.387Z [INFO]  server-f90494ca-a0ba-bf8f-f653-c504fd7388bf: pipelining replication: peer="{Voter server-daf4e2d0-d66f-beab-ce03-f984a5facb13 daf4e2d0-d66f-beab-ce03-f984a5facb13}"
2026-02-28T09:02:12.387Z [INFO]  server-f90494ca-a0ba-bf8f-f653-c504fd7388bf: pipelining replication: peer="{Voter server-57dab213-4886-50f0-e42e-679147794d18 57dab213-4886-50f0-e42e-679147794d18}"
2026-02-28T09:02:12.492Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:12.386973102 +0000 UTC m=+10.350769641 (1 nodes), 70.09644ms from start of poll, 70.50592ms from cluster start. Timeout at 2026-02-28 09:02:12.491983262 +0000 UTC m=+10.455779801, 105.01016ms after stability
2026-02-28T09:02:12.498Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:12.604Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:12.498329942 +0000 UTC m=+10.462126481 (2 nodes), 3.2µs from start of poll, 181.86276ms from cluster start. Timeout at 2026-02-28 09:02:12.603340262 +0000 UTC m=+10.567136801, 105.01032ms after stability
2026-02-28T09:02:12.604Z [INFO]  server-f90494ca-a0ba-bf8f-f653-c504fd7388bf: updating configuration: command=RemoveServer server-id=server-f90494ca-a0ba-bf8f-f653-c504fd7388bf server-addr= servers="[{Suffrage:Voter ID:server-57dab213-4886-50f0-e42e-679147794d18 Address:57dab213-4886-50f0-e42e-679147794d18} {Suffrage:Voter ID:server-daf4e2d0-d66f-beab-ce03-f984a5facb13 Address:daf4e2d0-d66f-beab-ce03-f984a5facb13}]"
2026-02-28T09:02:12.604Z [INFO]  server-f90494ca-a0ba-bf8f-f653-c504fd7388bf: removed ourself, shutting down
2026-02-28T09:02:12.604Z [INFO]  server-f90494ca-a0ba-bf8f-f653-c504fd7388bf: aborting pipeline replication: peer="{Voter server-daf4e2d0-d66f-beab-ce03-f984a5facb13 daf4e2d0-d66f-beab-ce03-f984a5facb13}"
2026-02-28T09:02:12.604Z [INFO]  server-f90494ca-a0ba-bf8f-f653-c504fd7388bf: aborting pipeline replication: peer="{Voter server-57dab213-4886-50f0-e42e-679147794d18 57dab213-4886-50f0-e42e-679147794d18}"
2026-02-28T09:02:12.700Z [WARN]  server-57dab213-4886-50f0-e42e-679147794d18: heartbeat timeout reached, starting election: last-leader-addr=f90494ca-a0ba-bf8f-f653-c504fd7388bf last-leader-id=server-f90494ca-a0ba-bf8f-f653-c504fd7388bf
2026-02-28T09:02:12.700Z [INFO]  server-57dab213-4886-50f0-e42e-679147794d18: entering candidate state: node="Node at 57dab213-4886-50f0-e42e-679147794d18 [Candidate]" term=3
2026-02-28T09:02:12.700Z [WARN]  server-daf4e2d0-d66f-beab-ce03-f984a5facb13: rejecting vote request since we have a leader: from=57dab213-4886-50f0-e42e-679147794d18 leader=f90494ca-a0ba-bf8f-f653-c504fd7388bf leader-id=server-f90494ca-a0ba-bf8f-f653-c504fd7388bf
2026-02-28T09:02:12.704Z [WARN]  server-daf4e2d0-d66f-beab-ce03-f984a5facb13: heartbeat timeout reached, starting election: last-leader-addr=f90494ca-a0ba-bf8f-f653-c504fd7388bf last-leader-id=server-f90494ca-a0ba-bf8f-f653-c504fd7388bf
2026-02-28T09:02:12.704Z [INFO]  server-daf4e2d0-d66f-beab-ce03-f984a5facb13: entering candidate state: node="Node at daf4e2d0-d66f-beab-ce03-f984a5facb13 [Candidate]" term=3
2026-02-28T09:02:12.704Z [INFO]  server-57dab213-4886-50f0-e42e-679147794d18: duplicate requestVote for same term: term=3
2026-02-28T09:02:12.755Z [WARN]  server-57dab213-4886-50f0-e42e-679147794d18: Election timeout reached, restarting election
2026-02-28T09:02:12.755Z [INFO]  server-57dab213-4886-50f0-e42e-679147794d18: entering candidate state: node="Node at 57dab213-4886-50f0-e42e-679147794d18 [Candidate]" term=4
2026-02-28T09:02:12.756Z [INFO]  server-daf4e2d0-d66f-beab-ce03-f984a5facb13: entering follower state: follower="Node at daf4e2d0-d66f-beab-ce03-f984a5facb13 [Follower]" leader-address= leader-id=
2026-02-28T09:02:12.756Z [INFO]  server-57dab213-4886-50f0-e42e-679147794d18: election won: term=4 tally=2
2026-02-28T09:02:12.756Z [INFO]  server-57dab213-4886-50f0-e42e-679147794d18: entering leader state: leader="Node at 57dab213-4886-50f0-e42e-679147794d18 [Leader]"
2026-02-28T09:02:12.756Z [INFO]  server-57dab213-4886-50f0-e42e-679147794d18: added peer, starting replication: peer=server-daf4e2d0-d66f-beab-ce03-f984a5facb13
2026-02-28T09:02:12.756Z [INFO]  server-57dab213-4886-50f0-e42e-679147794d18: pipelining replication: peer="{Voter server-daf4e2d0-d66f-beab-ce03-f984a5facb13 daf4e2d0-d66f-beab-ce03-f984a5facb13}"
2026-02-28T09:02:12.815Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:12.921Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:12.815793902 +0000 UTC m=+10.779590441 (1 nodes), 66.92µs from start of poll, 499.32672ms from cluster start. Timeout at 2026-02-28 09:02:12.920804942 +0000 UTC m=+10.884601481, 105.01104ms after stability
2026-02-28T09:02:12.921Z [INFO]  server-57dab213-4886-50f0-e42e-679147794d18: aborting pipeline replication: peer="{Voter server-daf4e2d0-d66f-beab-ce03-f984a5facb13 daf4e2d0-d66f-beab-ce03-f984a5facb13}"
--- PASS: TestRaft_RemoveLeader (0.61s)
=== RUN   TestRaft_RemoveLeader_NoShutdown
2026-02-28T09:02:12.922Z [INFO]  server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3 Address:21d770a0-1cfa-f8cd-0880-c0bbe18373d3} {Suffrage:Voter ID:server-4948c08a-2246-856b-cf4f-d0bcb7cbcfc4 Address:4948c08a-2246-856b-cf4f-d0bcb7cbcfc4} {Suffrage:Voter ID:server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7 Address:0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7}]"
2026-02-28T09:02:12.923Z [INFO]  server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3: entering follower state: follower="Node at 21d770a0-1cfa-f8cd-0880-c0bbe18373d3 [Follower]" leader-address= leader-id=
2026-02-28T09:02:12.923Z [INFO]  server-4948c08a-2246-856b-cf4f-d0bcb7cbcfc4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3 Address:21d770a0-1cfa-f8cd-0880-c0bbe18373d3} {Suffrage:Voter ID:server-4948c08a-2246-856b-cf4f-d0bcb7cbcfc4 Address:4948c08a-2246-856b-cf4f-d0bcb7cbcfc4} {Suffrage:Voter ID:server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7 Address:0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7}]"
2026-02-28T09:02:12.923Z [INFO]  server-4948c08a-2246-856b-cf4f-d0bcb7cbcfc4: entering follower state: follower="Node at 4948c08a-2246-856b-cf4f-d0bcb7cbcfc4 [Follower]" leader-address= leader-id=
2026-02-28T09:02:12.923Z [INFO]  server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3 Address:21d770a0-1cfa-f8cd-0880-c0bbe18373d3} {Suffrage:Voter ID:server-4948c08a-2246-856b-cf4f-d0bcb7cbcfc4 Address:4948c08a-2246-856b-cf4f-d0bcb7cbcfc4} {Suffrage:Voter ID:server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7 Address:0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7}]"
2026-02-28T09:02:12.923Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:12.923Z [INFO]  server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7: entering follower state: follower="Node at 0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7 [Follower]" leader-address= leader-id=
2026-02-28T09:02:12.973Z [WARN]  server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:12.973Z [INFO]  server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3: entering candidate state: node="Node at 21d770a0-1cfa-f8cd-0880-c0bbe18373d3 [Candidate]" term=2
2026-02-28T09:02:12.974Z [INFO]  server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3: election won: term=2 tally=2
2026-02-28T09:02:12.974Z [INFO]  server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3: entering leader state: leader="Node at 21d770a0-1cfa-f8cd-0880-c0bbe18373d3 [Leader]"
2026-02-28T09:02:12.974Z [INFO]  server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3: added peer, starting replication: peer=server-4948c08a-2246-856b-cf4f-d0bcb7cbcfc4
2026-02-28T09:02:12.974Z [INFO]  server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3: added peer, starting replication: peer=server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7
2026-02-28T09:02:12.974Z [INFO]  server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3: pipelining replication: peer="{Voter server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7 0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7}"
2026-02-28T09:02:12.974Z [INFO]  server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3: pipelining replication: peer="{Voter server-4948c08a-2246-856b-cf4f-d0bcb7cbcfc4 4948c08a-2246-856b-cf4f-d0bcb7cbcfc4}"
2026-02-28T09:02:13.080Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:12.974221342 +0000 UTC m=+10.938017881 (2 nodes), 50.3122ms from start of poll, 52.09308ms from cluster start. Timeout at 2026-02-28 09:02:13.079224742 +0000 UTC m=+11.043021281, 105.0034ms after stability
2026-02-28T09:02:13.080Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:13.186Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:13.080344462 +0000 UTC m=+11.044141001 (1 nodes), 1.72µs from start of poll, 158.2162ms from cluster start. Timeout at 2026-02-28 09:02:13.185351022 +0000 UTC m=+11.149147561, 105.00656ms after stability
2026-02-28T09:02:13.191Z [INFO]  server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3: updating configuration: command=RemoveServer server-id=server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3 server-addr= servers="[{Suffrage:Voter ID:server-4948c08a-2246-856b-cf4f-d0bcb7cbcfc4 Address:4948c08a-2246-856b-cf4f-d0bcb7cbcfc4} {Suffrage:Voter ID:server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7 Address:0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7}]"
2026-02-28T09:02:13.191Z [INFO]  server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3: removed ourself, transitioning to follower
2026-02-28T09:02:13.191Z [INFO]  server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3: entering follower state: follower="Node at 21d770a0-1cfa-f8cd-0880-c0bbe18373d3 [Follower]" leader-address= leader-id=
2026-02-28T09:02:13.191Z [INFO]  server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3: aborting pipeline replication: peer="{Voter server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7 0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7}"
2026-02-28T09:02:13.191Z [INFO]  server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3: aborting pipeline replication: peer="{Voter server-4948c08a-2246-856b-cf4f-d0bcb7cbcfc4 4948c08a-2246-856b-cf4f-d0bcb7cbcfc4}"
2026-02-28T09:02:13.241Z [WARN]  server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7: heartbeat timeout reached, starting election: last-leader-addr=21d770a0-1cfa-f8cd-0880-c0bbe18373d3 last-leader-id=server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3
2026-02-28T09:02:13.241Z [INFO]  server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7: entering candidate state: node="Node at 0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7 [Candidate]" term=3
2026-02-28T09:02:13.242Z [WARN]  server-4948c08a-2246-856b-cf4f-d0bcb7cbcfc4: rejecting vote request since we have a leader: from=0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7 leader=21d770a0-1cfa-f8cd-0880-c0bbe18373d3 leader-id=server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3
2026-02-28T09:02:13.255Z [WARN]  server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3: not part of stable configuration, aborting election
2026-02-28T09:02:13.263Z [WARN]  server-4948c08a-2246-856b-cf4f-d0bcb7cbcfc4: heartbeat timeout reached, starting election: last-leader-addr=21d770a0-1cfa-f8cd-0880-c0bbe18373d3 last-leader-id=server-21d770a0-1cfa-f8cd-0880-c0bbe18373d3
2026-02-28T09:02:13.263Z [INFO]  server-4948c08a-2246-856b-cf4f-d0bcb7cbcfc4: entering candidate state: node="Node at 4948c08a-2246-856b-cf4f-d0bcb7cbcfc4 [Candidate]" term=3
2026-02-28T09:02:13.263Z [INFO]  server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7: duplicate requestVote for same term: term=3
2026-02-28T09:02:13.296Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:13.319Z [WARN]  server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7: Election timeout reached, restarting election
2026-02-28T09:02:13.319Z [INFO]  server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7: entering candidate state: node="Node at 0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7 [Candidate]" term=4
2026-02-28T09:02:13.320Z [INFO]  server-4948c08a-2246-856b-cf4f-d0bcb7cbcfc4: entering follower state: follower="Node at 4948c08a-2246-856b-cf4f-d0bcb7cbcfc4 [Follower]" leader-address= leader-id=
2026-02-28T09:02:13.320Z [INFO]  server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7: election won: term=4 tally=2
2026-02-28T09:02:13.320Z [INFO]  server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7: entering leader state: leader="Node at 0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7 [Leader]"
2026-02-28T09:02:13.320Z [INFO]  server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7: added peer, starting replication: peer=server-4948c08a-2246-856b-cf4f-d0bcb7cbcfc4
2026-02-28T09:02:13.320Z [INFO]  server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7: pipelining replication: peer="{Voter server-4948c08a-2246-856b-cf4f-d0bcb7cbcfc4 4948c08a-2246-856b-cf4f-d0bcb7cbcfc4}"
2026-02-28T09:02:13.425Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:13.320147742 +0000 UTC m=+11.283944241 (1 nodes), 23.31496ms from start of poll, 398.01944ms from cluster start. Timeout at 2026-02-28 09:02:13.425150582 +0000 UTC m=+11.388947121, 105.00288ms after stability
2026-02-28T09:02:13.530Z [INFO]  server-0d9f5cc0-51c8-cf1f-d1f2-48166ec1e7a7: aborting pipeline replication: peer="{Voter server-4948c08a-2246-856b-cf4f-d0bcb7cbcfc4 4948c08a-2246-856b-cf4f-d0bcb7cbcfc4}"
--- PASS: TestRaft_RemoveLeader_NoShutdown (0.61s)
=== RUN   TestRaft_RemoveFollower_SplitCluster
2026-02-28T09:02:13.531Z [INFO]  server-93f6b1b5-74d0-af0d-636f-4fe46e3659bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93f6b1b5-74d0-af0d-636f-4fe46e3659bf Address:93f6b1b5-74d0-af0d-636f-4fe46e3659bf} {Suffrage:Voter ID:server-5918f179-36b1-c68e-1ee7-0c114b630a2f Address:5918f179-36b1-c68e-1ee7-0c114b630a2f} {Suffrage:Voter ID:server-dced606b-8c19-b9e1-7c1e-cfaae58c492b Address:dced606b-8c19-b9e1-7c1e-cfaae58c492b} {Suffrage:Voter ID:server-b821043a-00ac-58bd-c7cc-98008ba9d313 Address:b821043a-00ac-58bd-c7cc-98008ba9d313}]"
2026-02-28T09:02:13.531Z [INFO]  server-93f6b1b5-74d0-af0d-636f-4fe46e3659bf: entering follower state: follower="Node at 93f6b1b5-74d0-af0d-636f-4fe46e3659bf [Follower]" leader-address= leader-id=
2026-02-28T09:02:13.531Z [INFO]  server-5918f179-36b1-c68e-1ee7-0c114b630a2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93f6b1b5-74d0-af0d-636f-4fe46e3659bf Address:93f6b1b5-74d0-af0d-636f-4fe46e3659bf} {Suffrage:Voter ID:server-5918f179-36b1-c68e-1ee7-0c114b630a2f Address:5918f179-36b1-c68e-1ee7-0c114b630a2f} {Suffrage:Voter ID:server-dced606b-8c19-b9e1-7c1e-cfaae58c492b Address:dced606b-8c19-b9e1-7c1e-cfaae58c492b} {Suffrage:Voter ID:server-b821043a-00ac-58bd-c7cc-98008ba9d313 Address:b821043a-00ac-58bd-c7cc-98008ba9d313}]"
2026-02-28T09:02:13.532Z [INFO]  server-5918f179-36b1-c68e-1ee7-0c114b630a2f: entering follower state: follower="Node at 5918f179-36b1-c68e-1ee7-0c114b630a2f [Follower]" leader-address= leader-id=
2026-02-28T09:02:13.532Z [INFO]  server-dced606b-8c19-b9e1-7c1e-cfaae58c492b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93f6b1b5-74d0-af0d-636f-4fe46e3659bf Address:93f6b1b5-74d0-af0d-636f-4fe46e3659bf} {Suffrage:Voter ID:server-5918f179-36b1-c68e-1ee7-0c114b630a2f Address:5918f179-36b1-c68e-1ee7-0c114b630a2f} {Suffrage:Voter ID:server-dced606b-8c19-b9e1-7c1e-cfaae58c492b Address:dced606b-8c19-b9e1-7c1e-cfaae58c492b} {Suffrage:Voter ID:server-b821043a-00ac-58bd-c7cc-98008ba9d313 Address:b821043a-00ac-58bd-c7cc-98008ba9d313}]"
2026-02-28T09:02:13.532Z [INFO]  server-dced606b-8c19-b9e1-7c1e-cfaae58c492b: entering follower state: follower="Node at dced606b-8c19-b9e1-7c1e-cfaae58c492b [Follower]" leader-address= leader-id=
2026-02-28T09:02:13.532Z [INFO]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93f6b1b5-74d0-af0d-636f-4fe46e3659bf Address:93f6b1b5-74d0-af0d-636f-4fe46e3659bf} {Suffrage:Voter ID:server-5918f179-36b1-c68e-1ee7-0c114b630a2f Address:5918f179-36b1-c68e-1ee7-0c114b630a2f} {Suffrage:Voter ID:server-dced606b-8c19-b9e1-7c1e-cfaae58c492b Address:dced606b-8c19-b9e1-7c1e-cfaae58c492b} {Suffrage:Voter ID:server-b821043a-00ac-58bd-c7cc-98008ba9d313 Address:b821043a-00ac-58bd-c7cc-98008ba9d313}]"
2026-02-28T09:02:13.532Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:13.532Z [INFO]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: entering follower state: follower="Node at b821043a-00ac-58bd-c7cc-98008ba9d313 [Follower]" leader-address= leader-id=
2026-02-28T09:02:13.592Z [WARN]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:13.592Z [INFO]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: entering candidate state: node="Node at b821043a-00ac-58bd-c7cc-98008ba9d313 [Candidate]" term=2
2026-02-28T09:02:13.592Z [INFO]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: election won: term=2 tally=3
2026-02-28T09:02:13.592Z [INFO]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: entering leader state: leader="Node at b821043a-00ac-58bd-c7cc-98008ba9d313 [Leader]"
2026-02-28T09:02:13.592Z [INFO]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: added peer, starting replication: peer=server-93f6b1b5-74d0-af0d-636f-4fe46e3659bf
2026-02-28T09:02:13.592Z [INFO]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: added peer, starting replication: peer=server-5918f179-36b1-c68e-1ee7-0c114b630a2f
2026-02-28T09:02:13.592Z [INFO]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: added peer, starting replication: peer=server-dced606b-8c19-b9e1-7c1e-cfaae58c492b
2026-02-28T09:02:13.593Z [INFO]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: pipelining replication: peer="{Voter server-dced606b-8c19-b9e1-7c1e-cfaae58c492b dced606b-8c19-b9e1-7c1e-cfaae58c492b}"
2026-02-28T09:02:13.593Z [INFO]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: pipelining replication: peer="{Voter server-93f6b1b5-74d0-af0d-636f-4fe46e3659bf 93f6b1b5-74d0-af0d-636f-4fe46e3659bf}"
2026-02-28T09:02:13.593Z [INFO]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: pipelining replication: peer="{Voter server-5918f179-36b1-c68e-1ee7-0c114b630a2f 5918f179-36b1-c68e-1ee7-0c114b630a2f}"
2026-02-28T09:02:13.698Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:13.592859022 +0000 UTC m=+11.556655601 (1 nodes), 60.44608ms from start of poll, 61.2524ms from cluster start. Timeout at 2026-02-28 09:02:13.697863262 +0000 UTC m=+11.661659801, 105.0042ms after stability
2026-02-28T09:02:13.804Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:13.909Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:13.804094382 +0000 UTC m=+11.767890921 (3 nodes), 3.28µs from start of poll, 272.48772ms from cluster start. Timeout at 2026-02-28 09:02:13.909102142 +0000 UTC m=+11.872898681, 105.00776ms after stability
2026-02-28T09:02:13.909Z [INFO]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: updating configuration: command=RemoveServer server-id=server-dced606b-8c19-b9e1-7c1e-cfaae58c492b server-addr= servers="[{Suffrage:Voter ID:server-93f6b1b5-74d0-af0d-636f-4fe46e3659bf Address:93f6b1b5-74d0-af0d-636f-4fe46e3659bf} {Suffrage:Voter ID:server-5918f179-36b1-c68e-1ee7-0c114b630a2f Address:5918f179-36b1-c68e-1ee7-0c114b630a2f} {Suffrage:Voter ID:server-b821043a-00ac-58bd-c7cc-98008ba9d313 Address:b821043a-00ac-58bd-c7cc-98008ba9d313}]"
2026-02-28T09:02:13.909Z [INFO]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: removed peer, stopping replication: peer=server-dced606b-8c19-b9e1-7c1e-cfaae58c492b last-index=3
2026-02-28T09:02:13.909Z [ERROR] server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to pipeline appendEntries: peer="{Voter server-93f6b1b5-74d0-af0d-636f-4fe46e3659bf 93f6b1b5-74d0-af0d-636f-4fe46e3659bf}" error="append pipeline closed"
2026-02-28T09:02:13.909Z [ERROR] server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to pipeline appendEntries: peer="{Voter server-5918f179-36b1-c68e-1ee7-0c114b630a2f 5918f179-36b1-c68e-1ee7-0c114b630a2f}" error="append pipeline closed"
2026-02-28T09:02:13.909Z [INFO]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: aborting pipeline replication: peer="{Voter server-5918f179-36b1-c68e-1ee7-0c114b630a2f 5918f179-36b1-c68e-1ee7-0c114b630a2f}"
2026-02-28T09:02:13.909Z [INFO]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: aborting pipeline replication: peer="{Voter server-93f6b1b5-74d0-af0d-636f-4fe46e3659bf 93f6b1b5-74d0-af0d-636f-4fe46e3659bf}"
2026-02-28T09:02:13.909Z [INFO]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: aborting pipeline replication: peer="{Voter server-dced606b-8c19-b9e1-7c1e-cfaae58c492b dced606b-8c19-b9e1-7c1e-cfaae58c492b}"
2026-02-28T09:02:13.914Z [ERROR] server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to heartbeat to: peer=93f6b1b5-74d0-af0d-636f-4fe46e3659bf backoff time=10ms error="failed to connect to peer: 93f6b1b5-74d0-af0d-636f-4fe46e3659bf"
2026-02-28T09:02:13.914Z [ERROR] server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to heartbeat to: peer=5918f179-36b1-c68e-1ee7-0c114b630a2f backoff time=10ms error="failed to connect to peer: 5918f179-36b1-c68e-1ee7-0c114b630a2f"
2026-02-28T09:02:13.918Z [ERROR] server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to appendEntries to: peer="{Voter server-93f6b1b5-74d0-af0d-636f-4fe46e3659bf 93f6b1b5-74d0-af0d-636f-4fe46e3659bf}" error="failed to connect to peer: 93f6b1b5-74d0-af0d-636f-4fe46e3659bf"
2026-02-28T09:02:13.919Z [ERROR] server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to appendEntries to: peer="{Voter server-5918f179-36b1-c68e-1ee7-0c114b630a2f 5918f179-36b1-c68e-1ee7-0c114b630a2f}" error="failed to connect to peer: 5918f179-36b1-c68e-1ee7-0c114b630a2f"
2026-02-28T09:02:13.932Z [ERROR] server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to heartbeat to: peer=5918f179-36b1-c68e-1ee7-0c114b630a2f backoff time=10ms error="failed to connect to peer: 5918f179-36b1-c68e-1ee7-0c114b630a2f"
2026-02-28T09:02:13.933Z [ERROR] server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to heartbeat to: peer=93f6b1b5-74d0-af0d-636f-4fe46e3659bf backoff time=10ms error="failed to connect to peer: 93f6b1b5-74d0-af0d-636f-4fe46e3659bf"
2026-02-28T09:02:13.937Z [ERROR] server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to appendEntries to: peer="{Voter server-93f6b1b5-74d0-af0d-636f-4fe46e3659bf 93f6b1b5-74d0-af0d-636f-4fe46e3659bf}" error="failed to connect to peer: 93f6b1b5-74d0-af0d-636f-4fe46e3659bf"
2026-02-28T09:02:13.940Z [ERROR] server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to appendEntries to: peer="{Voter server-5918f179-36b1-c68e-1ee7-0c114b630a2f 5918f179-36b1-c68e-1ee7-0c114b630a2f}" error="failed to connect to peer: 5918f179-36b1-c68e-1ee7-0c114b630a2f"
2026-02-28T09:02:13.951Z [ERROR] server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to heartbeat to: peer=5918f179-36b1-c68e-1ee7-0c114b630a2f backoff time=10ms error="failed to connect to peer: 5918f179-36b1-c68e-1ee7-0c114b630a2f"
2026-02-28T09:02:13.952Z [ERROR] server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to heartbeat to: peer=93f6b1b5-74d0-af0d-636f-4fe46e3659bf backoff time=10ms error="failed to connect to peer: 93f6b1b5-74d0-af0d-636f-4fe46e3659bf"
2026-02-28T09:02:13.953Z [WARN]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to contact: server-id=server-5918f179-36b1-c68e-1ee7-0c114b630a2f time=50.58328ms
2026-02-28T09:02:13.957Z [ERROR] server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to appendEntries to: peer="{Voter server-93f6b1b5-74d0-af0d-636f-4fe46e3659bf 93f6b1b5-74d0-af0d-636f-4fe46e3659bf}" error="failed to connect to peer: 93f6b1b5-74d0-af0d-636f-4fe46e3659bf"
2026-02-28T09:02:13.958Z [ERROR] server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to appendEntries to: peer="{Voter server-5918f179-36b1-c68e-1ee7-0c114b630a2f 5918f179-36b1-c68e-1ee7-0c114b630a2f}" error="failed to connect to peer: 5918f179-36b1-c68e-1ee7-0c114b630a2f"
2026-02-28T09:02:13.964Z [WARN]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to contact: server-id=server-93f6b1b5-74d0-af0d-636f-4fe46e3659bf time=60.53324ms
2026-02-28T09:02:13.964Z [WARN]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to contact: server-id=server-5918f179-36b1-c68e-1ee7-0c114b630a2f time=61.5316ms
2026-02-28T09:02:13.964Z [WARN]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to contact quorum of nodes, stepping down
2026-02-28T09:02:13.964Z [INFO]  server-b821043a-00ac-58bd-c7cc-98008ba9d313: entering follower state: follower="Node at b821043a-00ac-58bd-c7cc-98008ba9d313 [Follower]" leader-address= leader-id=
2026-02-28T09:02:13.964Z [ERROR] server-b821043a-00ac-58bd-c7cc-98008ba9d313: failed to appendEntries to: peer="{Voter server-93f6b1b5-74d0-af0d-636f-4fe46e3659bf 93f6b1b5-74d0-af0d-636f-4fe46e3659bf}" error="failed to connect to peer: 93f6b1b5-74d0-af0d-636f-4fe46e3659bf"
--- PASS: TestRaft_RemoveFollower_SplitCluster (0.43s)
=== RUN   TestRaft_AddKnownPeer
2026-02-28T09:02:13.965Z [INFO]  server-62345070-466e-16d8-c810-3fe369d0eec7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62345070-466e-16d8-c810-3fe369d0eec7 Address:62345070-466e-16d8-c810-3fe369d0eec7} {Suffrage:Voter ID:server-667a378e-37b0-bd10-8e0c-4362027e17b1 Address:667a378e-37b0-bd10-8e0c-4362027e17b1} {Suffrage:Voter ID:server-93998e0e-2242-dc02-467d-23ea517a71e3 Address:93998e0e-2242-dc02-467d-23ea517a71e3}]"
2026-02-28T09:02:13.965Z [INFO]  server-667a378e-37b0-bd10-8e0c-4362027e17b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62345070-466e-16d8-c810-3fe369d0eec7 Address:62345070-466e-16d8-c810-3fe369d0eec7} {Suffrage:Voter ID:server-667a378e-37b0-bd10-8e0c-4362027e17b1 Address:667a378e-37b0-bd10-8e0c-4362027e17b1} {Suffrage:Voter ID:server-93998e0e-2242-dc02-467d-23ea517a71e3 Address:93998e0e-2242-dc02-467d-23ea517a71e3}]"
2026-02-28T09:02:13.965Z [INFO]  server-62345070-466e-16d8-c810-3fe369d0eec7: entering follower state: follower="Node at 62345070-466e-16d8-c810-3fe369d0eec7 [Follower]" leader-address= leader-id=
2026-02-28T09:02:13.965Z [INFO]  server-667a378e-37b0-bd10-8e0c-4362027e17b1: entering follower state: follower="Node at 667a378e-37b0-bd10-8e0c-4362027e17b1 [Follower]" leader-address= leader-id=
2026-02-28T09:02:13.965Z [INFO]  server-93998e0e-2242-dc02-467d-23ea517a71e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62345070-466e-16d8-c810-3fe369d0eec7 Address:62345070-466e-16d8-c810-3fe369d0eec7} {Suffrage:Voter ID:server-667a378e-37b0-bd10-8e0c-4362027e17b1 Address:667a378e-37b0-bd10-8e0c-4362027e17b1} {Suffrage:Voter ID:server-93998e0e-2242-dc02-467d-23ea517a71e3 Address:93998e0e-2242-dc02-467d-23ea517a71e3}]"
2026-02-28T09:02:13.965Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:13.965Z [INFO]  server-93998e0e-2242-dc02-467d-23ea517a71e3: entering follower state: follower="Node at 93998e0e-2242-dc02-467d-23ea517a71e3 [Follower]" leader-address= leader-id=
2026-02-28T09:02:14.021Z [WARN]  server-667a378e-37b0-bd10-8e0c-4362027e17b1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:14.021Z [INFO]  server-667a378e-37b0-bd10-8e0c-4362027e17b1: entering candidate state: node="Node at 667a378e-37b0-bd10-8e0c-4362027e17b1 [Candidate]" term=2
2026-02-28T09:02:14.022Z [INFO]  server-667a378e-37b0-bd10-8e0c-4362027e17b1: election won: term=2 tally=2
2026-02-28T09:02:14.022Z [INFO]  server-667a378e-37b0-bd10-8e0c-4362027e17b1: entering leader state: leader="Node at 667a378e-37b0-bd10-8e0c-4362027e17b1 [Leader]"
2026-02-28T09:02:14.022Z [INFO]  server-667a378e-37b0-bd10-8e0c-4362027e17b1: added peer, starting replication: peer=server-62345070-466e-16d8-c810-3fe369d0eec7
2026-02-28T09:02:14.022Z [INFO]  server-667a378e-37b0-bd10-8e0c-4362027e17b1: added peer, starting replication: peer=server-93998e0e-2242-dc02-467d-23ea517a71e3
2026-02-28T09:02:14.022Z [INFO]  server-667a378e-37b0-bd10-8e0c-4362027e17b1: pipelining replication: peer="{Voter server-62345070-466e-16d8-c810-3fe369d0eec7 62345070-466e-16d8-c810-3fe369d0eec7}"
2026-02-28T09:02:14.022Z [INFO]  server-667a378e-37b0-bd10-8e0c-4362027e17b1: pipelining replication: peer="{Voter server-93998e0e-2242-dc02-467d-23ea517a71e3 93998e0e-2242-dc02-467d-23ea517a71e3}"
2026-02-28T09:02:14.128Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:14.022147822 +0000 UTC m=+11.985944361 (1 nodes), 56.28216ms from start of poll, 56.72772ms from cluster start. Timeout at 2026-02-28 09:02:14.127151302 +0000 UTC m=+12.090947841, 105.00348ms after stability
2026-02-28T09:02:14.128Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:14.233Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:14.128255582 +0000 UTC m=+12.092052121 (2 nodes), 2.6µs from start of poll, 162.83548ms from cluster start. Timeout at 2026-02-28 09:02:14.233260582 +0000 UTC m=+12.197057121, 105.005ms after stability
2026-02-28T09:02:14.233Z [INFO]  server-667a378e-37b0-bd10-8e0c-4362027e17b1: updating configuration: command=AddVoter server-id=server-62345070-466e-16d8-c810-3fe369d0eec7 server-addr=62345070-466e-16d8-c810-3fe369d0eec7 servers="[{Suffrage:Voter ID:server-62345070-466e-16d8-c810-3fe369d0eec7 Address:62345070-466e-16d8-c810-3fe369d0eec7} {Suffrage:Voter ID:server-667a378e-37b0-bd10-8e0c-4362027e17b1 Address:667a378e-37b0-bd10-8e0c-4362027e17b1} {Suffrage:Voter ID:server-93998e0e-2242-dc02-467d-23ea517a71e3 Address:93998e0e-2242-dc02-467d-23ea517a71e3}]"
2026-02-28T09:02:14.233Z [INFO]  server-667a378e-37b0-bd10-8e0c-4362027e17b1: aborting pipeline replication: peer="{Voter server-93998e0e-2242-dc02-467d-23ea517a71e3 93998e0e-2242-dc02-467d-23ea517a71e3}"
2026-02-28T09:02:14.233Z [INFO]  server-667a378e-37b0-bd10-8e0c-4362027e17b1: aborting pipeline replication: peer="{Voter server-62345070-466e-16d8-c810-3fe369d0eec7 62345070-466e-16d8-c810-3fe369d0eec7}"
--- PASS: TestRaft_AddKnownPeer (0.27s)
=== RUN   TestRaft_RemoveUnknownPeer
2026-02-28T09:02:14.234Z [INFO]  server-140fb994-8f58-cf99-264a-a1a621fac536: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-140fb994-8f58-cf99-264a-a1a621fac536 Address:140fb994-8f58-cf99-264a-a1a621fac536} {Suffrage:Voter ID:server-a70d3931-d572-2084-8f82-213981d1c4ac Address:a70d3931-d572-2084-8f82-213981d1c4ac} {Suffrage:Voter ID:server-8846f8a4-7312-dfdd-d036-38638c9c978f Address:8846f8a4-7312-dfdd-d036-38638c9c978f}]"
2026-02-28T09:02:14.234Z [INFO]  server-140fb994-8f58-cf99-264a-a1a621fac536: entering follower state: follower="Node at 140fb994-8f58-cf99-264a-a1a621fac536 [Follower]" leader-address= leader-id=
2026-02-28T09:02:14.234Z [INFO]  server-a70d3931-d572-2084-8f82-213981d1c4ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-140fb994-8f58-cf99-264a-a1a621fac536 Address:140fb994-8f58-cf99-264a-a1a621fac536} {Suffrage:Voter ID:server-a70d3931-d572-2084-8f82-213981d1c4ac Address:a70d3931-d572-2084-8f82-213981d1c4ac} {Suffrage:Voter ID:server-8846f8a4-7312-dfdd-d036-38638c9c978f Address:8846f8a4-7312-dfdd-d036-38638c9c978f}]"
2026-02-28T09:02:14.235Z [INFO]  server-a70d3931-d572-2084-8f82-213981d1c4ac: entering follower state: follower="Node at a70d3931-d572-2084-8f82-213981d1c4ac [Follower]" leader-address= leader-id=
2026-02-28T09:02:14.235Z [INFO]  server-8846f8a4-7312-dfdd-d036-38638c9c978f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-140fb994-8f58-cf99-264a-a1a621fac536 Address:140fb994-8f58-cf99-264a-a1a621fac536} {Suffrage:Voter ID:server-a70d3931-d572-2084-8f82-213981d1c4ac Address:a70d3931-d572-2084-8f82-213981d1c4ac} {Suffrage:Voter ID:server-8846f8a4-7312-dfdd-d036-38638c9c978f Address:8846f8a4-7312-dfdd-d036-38638c9c978f}]"
2026-02-28T09:02:14.235Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:14.235Z [INFO]  server-8846f8a4-7312-dfdd-d036-38638c9c978f: entering follower state: follower="Node at 8846f8a4-7312-dfdd-d036-38638c9c978f [Follower]" leader-address= leader-id=
2026-02-28T09:02:14.289Z [WARN]  server-140fb994-8f58-cf99-264a-a1a621fac536: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:14.289Z [INFO]  server-140fb994-8f58-cf99-264a-a1a621fac536: entering candidate state: node="Node at 140fb994-8f58-cf99-264a-a1a621fac536 [Candidate]" term=2
2026-02-28T09:02:14.289Z [INFO]  server-140fb994-8f58-cf99-264a-a1a621fac536: election won: term=2 tally=2
2026-02-28T09:02:14.289Z [INFO]  server-140fb994-8f58-cf99-264a-a1a621fac536: entering leader state: leader="Node at 140fb994-8f58-cf99-264a-a1a621fac536 [Leader]"
2026-02-28T09:02:14.289Z [INFO]  server-140fb994-8f58-cf99-264a-a1a621fac536: added peer, starting replication: peer=server-a70d3931-d572-2084-8f82-213981d1c4ac
2026-02-28T09:02:14.289Z [INFO]  server-140fb994-8f58-cf99-264a-a1a621fac536: added peer, starting replication: peer=server-8846f8a4-7312-dfdd-d036-38638c9c978f
2026-02-28T09:02:14.289Z [INFO]  server-140fb994-8f58-cf99-264a-a1a621fac536: pipelining replication: peer="{Voter server-a70d3931-d572-2084-8f82-213981d1c4ac a70d3931-d572-2084-8f82-213981d1c4ac}"
2026-02-28T09:02:14.289Z [INFO]  server-140fb994-8f58-cf99-264a-a1a621fac536: pipelining replication: peer="{Voter server-8846f8a4-7312-dfdd-d036-38638c9c978f 8846f8a4-7312-dfdd-d036-38638c9c978f}"
2026-02-28T09:02:14.395Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:14.289557902 +0000 UTC m=+12.253354481 (1 nodes), 54.24192ms from start of poll, 54.88248ms from cluster start. Timeout at 2026-02-28 09:02:14.394559662 +0000 UTC m=+12.358356201, 105.00172ms after stability
2026-02-28T09:02:14.395Z [INFO]  server-140fb994-8f58-cf99-264a-a1a621fac536: updating configuration: command=RemoveServer server-id=d05ac1f6-f017-fbfb-51ee-0b341688e7fa server-addr= servers="[{Suffrage:Voter ID:server-140fb994-8f58-cf99-264a-a1a621fac536 Address:140fb994-8f58-cf99-264a-a1a621fac536} {Suffrage:Voter ID:server-a70d3931-d572-2084-8f82-213981d1c4ac Address:a70d3931-d572-2084-8f82-213981d1c4ac} {Suffrage:Voter ID:server-8846f8a4-7312-dfdd-d036-38638c9c978f Address:8846f8a4-7312-dfdd-d036-38638c9c978f}]"
2026-02-28T09:02:14.395Z [INFO]  server-140fb994-8f58-cf99-264a-a1a621fac536: aborting pipeline replication: peer="{Voter server-8846f8a4-7312-dfdd-d036-38638c9c978f 8846f8a4-7312-dfdd-d036-38638c9c978f}"
2026-02-28T09:02:14.395Z [INFO]  server-140fb994-8f58-cf99-264a-a1a621fac536: aborting pipeline replication: peer="{Voter server-a70d3931-d572-2084-8f82-213981d1c4ac a70d3931-d572-2084-8f82-213981d1c4ac}"
--- PASS: TestRaft_RemoveUnknownPeer (0.16s)
=== RUN   TestRaft_SnapshotRestore
2026-02-28T09:02:14.396Z [INFO]  server-d81f8244-b8d2-6be8-04cd-a89fb3df90ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d81f8244-b8d2-6be8-04cd-a89fb3df90ce Address:d81f8244-b8d2-6be8-04cd-a89fb3df90ce}]"
2026-02-28T09:02:14.396Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:14.396Z [INFO]  server-d81f8244-b8d2-6be8-04cd-a89fb3df90ce: entering follower state: follower="Node at d81f8244-b8d2-6be8-04cd-a89fb3df90ce [Follower]" leader-address= leader-id=
2026-02-28T09:02:14.482Z [WARN]  server-d81f8244-b8d2-6be8-04cd-a89fb3df90ce: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:14.482Z [INFO]  server-d81f8244-b8d2-6be8-04cd-a89fb3df90ce: entering candidate state: node="Node at d81f8244-b8d2-6be8-04cd-a89fb3df90ce [Candidate]" term=2
2026-02-28T09:02:14.482Z [INFO]  server-d81f8244-b8d2-6be8-04cd-a89fb3df90ce: election won: term=2 tally=1
2026-02-28T09:02:14.482Z [INFO]  server-d81f8244-b8d2-6be8-04cd-a89fb3df90ce: entering leader state: leader="Node at d81f8244-b8d2-6be8-04cd-a89fb3df90ce [Leader]"
2026-02-28T09:02:14.587Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:14.482297862 +0000 UTC m=+12.446094401 (1 nodes), 85.76124ms from start of poll, 85.94476ms from cluster start. Timeout at 2026-02-28 09:02:14.587301302 +0000 UTC m=+12.551097841, 105.00344ms after stability
2026-02-28T09:02:14.591Z [INFO]  server-d81f8244-b8d2-6be8-04cd-a89fb3df90ce: starting snapshot up to: index=102
2026-02-28T09:02:14.591Z [INFO]  creating new snapshot: path=/tmp/raft3476201573/snapshots/2-102-1772269334591.tmp
2026-02-28T09:02:14.592Z [INFO]  server-d81f8244-b8d2-6be8-04cd-a89fb3df90ce: compacting logs: from=1 to=92
2026-02-28T09:02:14.592Z [INFO]  server-d81f8244-b8d2-6be8-04cd-a89fb3df90ce: snapshot complete up to: index=102
2026-02-28T09:02:14.592Z [INFO]  server-d81f8244-b8d2-6be8-04cd-a89fb3df90ce: starting restore from snapshot: id=2-102-1772269334591 last-index=102 last-term=2 size-in-bytes=693
2026-02-28T09:02:14.592Z [INFO]  server-d81f8244-b8d2-6be8-04cd-a89fb3df90ce: snapshot restore progress: id=2-102-1772269334591 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-02-28T09:02:14.592Z [INFO]  server-d81f8244-b8d2-6be8-04cd-a89fb3df90ce: restored from snapshot: id=2-102-1772269334591 last-index=102 last-term=2 size-in-bytes=693
2026-02-28T09:02:14.592Z [INFO]  server-d81f8244-b8d2-6be8-04cd-a89fb3df90ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d81f8244-b8d2-6be8-04cd-a89fb3df90ce Address:d81f8244-b8d2-6be8-04cd-a89fb3df90ce}]"
--- PASS: TestRaft_SnapshotRestore (0.20s)
=== RUN   TestRaft_RestoreSnapshotOnStartup_Monotonic
2026-02-28T09:02:14.593Z [INFO]  server-bbcb7703-50b3-faa4-dd2c-64607e6e58b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bbcb7703-50b3-faa4-dd2c-64607e6e58b3 Address:bbcb7703-50b3-faa4-dd2c-64607e6e58b3}]"
2026-02-28T09:02:14.593Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:14.593Z [INFO]  server-bbcb7703-50b3-faa4-dd2c-64607e6e58b3: entering follower state: follower="Node at bbcb7703-50b3-faa4-dd2c-64607e6e58b3 [Follower]" leader-address= leader-id=
2026-02-28T09:02:14.656Z [WARN]  server-bbcb7703-50b3-faa4-dd2c-64607e6e58b3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:14.656Z [INFO]  server-bbcb7703-50b3-faa4-dd2c-64607e6e58b3: entering candidate state: node="Node at bbcb7703-50b3-faa4-dd2c-64607e6e58b3 [Candidate]" term=2
2026-02-28T09:02:14.656Z [INFO]  server-bbcb7703-50b3-faa4-dd2c-64607e6e58b3: election won: term=2 tally=1
2026-02-28T09:02:14.656Z [INFO]  server-bbcb7703-50b3-faa4-dd2c-64607e6e58b3: entering leader state: leader="Node at bbcb7703-50b3-faa4-dd2c-64607e6e58b3 [Leader]"
2026-02-28T09:02:14.761Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:14.656421102 +0000 UTC m=+12.620217641 (1 nodes), 62.87152ms from start of poll, 63.04244ms from cluster start. Timeout at 2026-02-28 09:02:14.761424542 +0000 UTC m=+12.725221081, 105.00344ms after stability
2026-02-28T09:02:14.764Z [INFO]  server-bbcb7703-50b3-faa4-dd2c-64607e6e58b3: starting snapshot up to: index=102
2026-02-28T09:02:14.764Z [INFO]  creating new snapshot: path=/tmp/raft4117454905/snapshots/2-102-1772269334764.tmp
2026-02-28T09:02:14.765Z [INFO]  server-bbcb7703-50b3-faa4-dd2c-64607e6e58b3: compacting logs: from=1 to=92
2026-02-28T09:02:14.765Z [INFO]  server-bbcb7703-50b3-faa4-dd2c-64607e6e58b3: snapshot complete up to: index=102
2026-02-28T09:02:14.765Z [INFO]  server-bbcb7703-50b3-faa4-dd2c-64607e6e58b3: starting restore from snapshot: id=2-102-1772269334764 last-index=102 last-term=2 size-in-bytes=693
2026-02-28T09:02:14.765Z [INFO]  server-bbcb7703-50b3-faa4-dd2c-64607e6e58b3: snapshot restore progress: id=2-102-1772269334764 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-02-28T09:02:14.765Z [INFO]  server-bbcb7703-50b3-faa4-dd2c-64607e6e58b3: restored from snapshot: id=2-102-1772269334764 last-index=102 last-term=2 size-in-bytes=693
2026-02-28T09:02:14.765Z [INFO]  server-bbcb7703-50b3-faa4-dd2c-64607e6e58b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bbcb7703-50b3-faa4-dd2c-64607e6e58b3 Address:bbcb7703-50b3-faa4-dd2c-64607e6e58b3}]"
--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.17s)
=== RUN   TestRaft_SnapshotRestore_Progress
2026-02-28T09:02:14.766Z [INFO]  server-c5edd083-991f-8bab-ae12-aba136f993bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c5edd083-991f-8bab-ae12-aba136f993bf Address:c5edd083-991f-8bab-ae12-aba136f993bf}]"
2026-02-28T09:02:14.766Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:14.766Z [INFO]  server-c5edd083-991f-8bab-ae12-aba136f993bf: entering follower state: follower="Node at c5edd083-991f-8bab-ae12-aba136f993bf [Follower]" leader-address= leader-id=
2026-02-28T09:02:14.844Z [WARN]  server-c5edd083-991f-8bab-ae12-aba136f993bf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:14.844Z [INFO]  server-c5edd083-991f-8bab-ae12-aba136f993bf: entering candidate state: node="Node at c5edd083-991f-8bab-ae12-aba136f993bf [Candidate]" term=2
2026-02-28T09:02:14.844Z [INFO]  server-c5edd083-991f-8bab-ae12-aba136f993bf: election won: term=2 tally=1
2026-02-28T09:02:14.844Z [INFO]  server-c5edd083-991f-8bab-ae12-aba136f993bf: entering leader state: leader="Node at c5edd083-991f-8bab-ae12-aba136f993bf [Leader]"
2026-02-28T09:02:14.950Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:14.844294342 +0000 UTC m=+12.808090881 (1 nodes), 77.8388ms from start of poll, 78.02988ms from cluster start. Timeout at 2026-02-28 09:02:14.949298022 +0000 UTC m=+12.913094561, 105.00368ms after stability
2026-02-28T09:02:14.954Z [INFO]  server-c5edd083-991f-8bab-ae12-aba136f993bf: starting snapshot up to: index=102
2026-02-28T09:02:14.954Z [INFO]  creating new snapshot: path=/tmp/raft1219265045/snapshots/2-102-1772269334954.tmp
2026-02-28T09:02:14.954Z [INFO]  server-c5edd083-991f-8bab-ae12-aba136f993bf: compacting logs: from=1 to=92
2026-02-28T09:02:14.954Z [INFO]  server-c5edd083-991f-8bab-ae12-aba136f993bf: snapshot complete up to: index=102
--- PASS: TestRaft_SnapshotRestore_Progress (0.19s)
=== RUN   TestRaft_NoRestoreOnStart
2026-02-28T09:02:14.956Z [INFO]  server-a9292af0-fadd-c45f-8ff5-c5c37e2e99df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9292af0-fadd-c45f-8ff5-c5c37e2e99df Address:a9292af0-fadd-c45f-8ff5-c5c37e2e99df}]"
2026-02-28T09:02:14.956Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:14.956Z [INFO]  server-a9292af0-fadd-c45f-8ff5-c5c37e2e99df: entering follower state: follower="Node at a9292af0-fadd-c45f-8ff5-c5c37e2e99df [Follower]" leader-address= leader-id=
2026-02-28T09:02:15.015Z [WARN]  server-a9292af0-fadd-c45f-8ff5-c5c37e2e99df: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:15.015Z [INFO]  server-a9292af0-fadd-c45f-8ff5-c5c37e2e99df: entering candidate state: node="Node at a9292af0-fadd-c45f-8ff5-c5c37e2e99df [Candidate]" term=2
2026-02-28T09:02:15.015Z [INFO]  server-a9292af0-fadd-c45f-8ff5-c5c37e2e99df: election won: term=2 tally=1
2026-02-28T09:02:15.015Z [INFO]  server-a9292af0-fadd-c45f-8ff5-c5c37e2e99df: entering leader state: leader="Node at a9292af0-fadd-c45f-8ff5-c5c37e2e99df [Leader]"
2026-02-28T09:02:15.121Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:15.015953742 +0000 UTC m=+12.979750281 (1 nodes), 59.76908ms from start of poll, 59.9696ms from cluster start. Timeout at 2026-02-28 09:02:15.120958102 +0000 UTC m=+13.084754641, 105.00436ms after stability
2026-02-28T09:02:15.124Z [INFO]  server-a9292af0-fadd-c45f-8ff5-c5c37e2e99df: starting snapshot up to: index=102
2026-02-28T09:02:15.124Z [INFO]  creating new snapshot: path=/tmp/raft3323618480/snapshots/2-102-1772269335124.tmp
2026-02-28T09:02:15.125Z [INFO]  server-a9292af0-fadd-c45f-8ff5-c5c37e2e99df: compacting logs: from=1 to=92
2026-02-28T09:02:15.125Z [INFO]  server-a9292af0-fadd-c45f-8ff5-c5c37e2e99df: snapshot complete up to: index=102
2026-02-28T09:02:15.125Z [INFO]  server-a9292af0-fadd-c45f-8ff5-c5c37e2e99df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9292af0-fadd-c45f-8ff5-c5c37e2e99df Address:a9292af0-fadd-c45f-8ff5-c5c37e2e99df}]"
--- PASS: TestRaft_NoRestoreOnStart (0.17s)
=== RUN   TestRaft_SnapshotRestore_PeerChange
    raft_test.go:1266: Skipping unreliable test
2026-02-28T09:02:15.125Z [INFO]  server-a9292af0-fadd-c45f-8ff5-c5c37e2e99df: entering follower state: follower="Node at a9292af0-fadd-c45f-8ff5-c5c37e2e99df [Follower]" leader-address= leader-id=
--- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
=== RUN   TestRaft_AutoSnapshot
2026-02-28T09:02:15.125Z [INFO]  server-14173e51-d385-d3c6-1164-acd9646281ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-14173e51-d385-d3c6-1164-acd9646281ea Address:14173e51-d385-d3c6-1164-acd9646281ea}]"
2026-02-28T09:02:15.125Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:15.125Z [INFO]  server-14173e51-d385-d3c6-1164-acd9646281ea: entering follower state: follower="Node at 14173e51-d385-d3c6-1164-acd9646281ea [Follower]" leader-address= leader-id=
2026-02-28T09:02:15.186Z [WARN]  server-a9292af0-fadd-c45f-8ff5-c5c37e2e99df: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:15.186Z [INFO]  server-a9292af0-fadd-c45f-8ff5-c5c37e2e99df: entering candidate state: node="Node at a9292af0-fadd-c45f-8ff5-c5c37e2e99df [Candidate]" term=3
2026-02-28T09:02:15.186Z [INFO]  server-a9292af0-fadd-c45f-8ff5-c5c37e2e99df: election won: term=3 tally=1
2026-02-28T09:02:15.186Z [INFO]  server-a9292af0-fadd-c45f-8ff5-c5c37e2e99df: entering leader state: leader="Node at a9292af0-fadd-c45f-8ff5-c5c37e2e99df [Leader]"
2026-02-28T09:02:15.220Z [WARN]  server-14173e51-d385-d3c6-1164-acd9646281ea: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:15.220Z [INFO]  server-14173e51-d385-d3c6-1164-acd9646281ea: entering candidate state: node="Node at 14173e51-d385-d3c6-1164-acd9646281ea [Candidate]" term=2
2026-02-28T09:02:15.221Z [INFO]  server-14173e51-d385-d3c6-1164-acd9646281ea: election won: term=2 tally=1
2026-02-28T09:02:15.221Z [INFO]  server-14173e51-d385-d3c6-1164-acd9646281ea: entering leader state: leader="Node at 14173e51-d385-d3c6-1164-acd9646281ea [Leader]"
2026-02-28T09:02:15.327Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:15.221113382 +0000 UTC m=+13.184909961 (1 nodes), 95.33832ms from start of poll, 95.50832ms from cluster start. Timeout at 2026-02-28 09:02:15.326114742 +0000 UTC m=+13.289911281, 105.00132ms after stability
2026-02-28T09:02:15.332Z [INFO]  server-14173e51-d385-d3c6-1164-acd9646281ea: starting snapshot up to: index=102
2026-02-28T09:02:15.332Z [INFO]  creating new snapshot: path=/tmp/raft2658119845/snapshots/2-102-1772269335332.tmp
2026-02-28T09:02:15.332Z [INFO]  server-14173e51-d385-d3c6-1164-acd9646281ea: compacting logs: from=1 to=92
2026-02-28T09:02:15.332Z [INFO]  server-14173e51-d385-d3c6-1164-acd9646281ea: snapshot complete up to: index=102
--- PASS: TestRaft_AutoSnapshot (0.31s)
=== RUN   TestRaft_UserSnapshot
2026-02-28T09:02:15.436Z [INFO]  server-e6311e56-b774-389d-2fb3-20ee7a6fcc74: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6311e56-b774-389d-2fb3-20ee7a6fcc74 Address:e6311e56-b774-389d-2fb3-20ee7a6fcc74}]"
2026-02-28T09:02:15.436Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:15.436Z [INFO]  server-e6311e56-b774-389d-2fb3-20ee7a6fcc74: entering follower state: follower="Node at e6311e56-b774-389d-2fb3-20ee7a6fcc74 [Follower]" leader-address= leader-id=
2026-02-28T09:02:15.519Z [WARN]  server-e6311e56-b774-389d-2fb3-20ee7a6fcc74: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:15.519Z [INFO]  server-e6311e56-b774-389d-2fb3-20ee7a6fcc74: entering candidate state: node="Node at e6311e56-b774-389d-2fb3-20ee7a6fcc74 [Candidate]" term=2
2026-02-28T09:02:15.519Z [INFO]  server-e6311e56-b774-389d-2fb3-20ee7a6fcc74: election won: term=2 tally=1
2026-02-28T09:02:15.519Z [INFO]  server-e6311e56-b774-389d-2fb3-20ee7a6fcc74: entering leader state: leader="Node at e6311e56-b774-389d-2fb3-20ee7a6fcc74 [Leader]"
2026-02-28T09:02:15.625Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:15.519426102 +0000 UTC m=+13.483222641 (1 nodes), 82.73056ms from start of poll, 82.9374ms from cluster start. Timeout at 2026-02-28 09:02:15.624430382 +0000 UTC m=+13.588226921, 105.00428ms after stability
2026-02-28T09:02:15.625Z [ERROR] server-e6311e56-b774-389d-2fb3-20ee7a6fcc74: failed to take snapshot: error="nothing new to snapshot"
2026-02-28T09:02:15.625Z [INFO]  server-e6311e56-b774-389d-2fb3-20ee7a6fcc74: starting snapshot up to: index=12
2026-02-28T09:02:15.625Z [INFO]  creating new snapshot: path=/tmp/raft2763565033/snapshots/2-12-1772269335625.tmp
2026-02-28T09:02:15.626Z [INFO]  server-e6311e56-b774-389d-2fb3-20ee7a6fcc74: compacting logs: from=1 to=2
2026-02-28T09:02:15.626Z [INFO]  server-e6311e56-b774-389d-2fb3-20ee7a6fcc74: snapshot complete up to: index=12
--- PASS: TestRaft_UserSnapshot (0.19s)
=== RUN   TestRaft_UserRestore
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false
2026-02-28T09:02:15.627Z [INFO]  server-c60005bc-1a3f-3d65-5ce8-4daf2240c1a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c60005bc-1a3f-3d65-5ce8-4daf2240c1a6 Address:c60005bc-1a3f-3d65-5ce8-4daf2240c1a6} {Suffrage:Voter ID:server-f924816b-c8d3-7a29-d2d7-29409249810e Address:f924816b-c8d3-7a29-d2d7-29409249810e} {Suffrage:Voter ID:server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6 Address:c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6}]"
2026-02-28T09:02:15.627Z [INFO]  server-f924816b-c8d3-7a29-d2d7-29409249810e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c60005bc-1a3f-3d65-5ce8-4daf2240c1a6 Address:c60005bc-1a3f-3d65-5ce8-4daf2240c1a6} {Suffrage:Voter ID:server-f924816b-c8d3-7a29-d2d7-29409249810e Address:f924816b-c8d3-7a29-d2d7-29409249810e} {Suffrage:Voter ID:server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6 Address:c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6}]"
2026-02-28T09:02:15.627Z [INFO]  server-c60005bc-1a3f-3d65-5ce8-4daf2240c1a6: entering follower state: follower="Node at c60005bc-1a3f-3d65-5ce8-4daf2240c1a6 [Follower]" leader-address= leader-id=
2026-02-28T09:02:15.627Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c60005bc-1a3f-3d65-5ce8-4daf2240c1a6 Address:c60005bc-1a3f-3d65-5ce8-4daf2240c1a6} {Suffrage:Voter ID:server-f924816b-c8d3-7a29-d2d7-29409249810e Address:f924816b-c8d3-7a29-d2d7-29409249810e} {Suffrage:Voter ID:server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6 Address:c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6}]"
2026-02-28T09:02:15.627Z [INFO]  server-f924816b-c8d3-7a29-d2d7-29409249810e: entering follower state: follower="Node at f924816b-c8d3-7a29-d2d7-29409249810e [Follower]" leader-address= leader-id=
2026-02-28T09:02:15.627Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:15.627Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: entering follower state: follower="Node at c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6 [Follower]" leader-address= leader-id=
2026-02-28T09:02:16.209Z [WARN]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:16.209Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: entering candidate state: node="Node at c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6 [Candidate]" term=2
2026-02-28T09:02:16.209Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: election won: term=2 tally=2
2026-02-28T09:02:16.209Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: entering leader state: leader="Node at c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6 [Leader]"
2026-02-28T09:02:16.209Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: added peer, starting replication: peer=server-c60005bc-1a3f-3d65-5ce8-4daf2240c1a6
2026-02-28T09:02:16.209Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: added peer, starting replication: peer=server-f924816b-c8d3-7a29-d2d7-29409249810e
2026-02-28T09:02:16.209Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: pipelining replication: peer="{Voter server-f924816b-c8d3-7a29-d2d7-29409249810e f924816b-c8d3-7a29-d2d7-29409249810e}"
2026-02-28T09:02:16.209Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: pipelining replication: peer="{Voter server-c60005bc-1a3f-3d65-5ce8-4daf2240c1a6 c60005bc-1a3f-3d65-5ce8-4daf2240c1a6}"
2026-02-28T09:02:17.215Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:16.209566942 +0000 UTC m=+14.173363521 (1 nodes), 582.18032ms from start of poll, 582.62436ms from cluster start. Timeout at 2026-02-28 09:02:17.214577462 +0000 UTC m=+15.178374001, 1.00501048s after stability
2026-02-28T09:02:17.217Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: starting snapshot up to: index=12
2026-02-28T09:02:17.217Z [INFO]  creating new snapshot: path=/tmp/raft2502959136/snapshots/2-12-1772269337217.tmp
2026-02-28T09:02:17.218Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: snapshot complete up to: index=12
2026-02-28T09:02:17.218Z [INFO]  creating new snapshot: path=/tmp/raft2502959136/snapshots/2-23-1772269337218.tmp
2026-02-28T09:02:17.219Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: copied to local snapshot: bytes=71
2026-02-28T09:02:17.219Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: snapshot restore progress: id=2-23-1772269337218 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:17.219Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: restored user snapshot: index=23
2026-02-28T09:02:17.219Z [ERROR] server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: failed to get log: index=23 error="log not found"
2026-02-28T09:02:17.219Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: aborting pipeline replication: peer="{Voter server-f924816b-c8d3-7a29-d2d7-29409249810e f924816b-c8d3-7a29-d2d7-29409249810e}"
2026-02-28T09:02:17.219Z [ERROR] server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: failed to get log: index=23 error="log not found"
2026-02-28T09:02:17.219Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: aborting pipeline replication: peer="{Voter server-c60005bc-1a3f-3d65-5ce8-4daf2240c1a6 c60005bc-1a3f-3d65-5ce8-4daf2240c1a6}"
2026-02-28T09:02:17.226Z [ERROR] server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: failed to get log: index=23 error="log not found"
2026-02-28T09:02:17.227Z [INFO]  creating new snapshot: path=/tmp/raft872832690/snapshots/2-23-1772269337227.tmp
2026-02-28T09:02:17.227Z [INFO]  server-c60005bc-1a3f-3d65-5ce8-4daf2240c1a6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:17.227Z [INFO]  server-c60005bc-1a3f-3d65-5ce8-4daf2240c1a6: copied to local snapshot: bytes=71
2026-02-28T09:02:17.227Z [INFO]  server-c60005bc-1a3f-3d65-5ce8-4daf2240c1a6: snapshot restore progress: id=2-23-1772269337227 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:17.228Z [INFO]  server-c60005bc-1a3f-3d65-5ce8-4daf2240c1a6: Installed remote snapshot
2026-02-28T09:02:17.228Z [WARN]  server-c60005bc-1a3f-3d65-5ce8-4daf2240c1a6: skipping application of old log: index=22
2026-02-28T09:02:17.228Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: pipelining replication: peer="{Voter server-c60005bc-1a3f-3d65-5ce8-4daf2240c1a6 c60005bc-1a3f-3d65-5ce8-4daf2240c1a6}"
2026-02-28T09:02:17.228Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:17.229Z [ERROR] server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: failed to get log: index=23 error="log not found"
2026-02-28T09:02:17.230Z [INFO]  creating new snapshot: path=/tmp/raft1279919250/snapshots/2-23-1772269337230.tmp
2026-02-28T09:02:17.230Z [INFO]  server-f924816b-c8d3-7a29-d2d7-29409249810e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:17.230Z [INFO]  server-f924816b-c8d3-7a29-d2d7-29409249810e: copied to local snapshot: bytes=71
2026-02-28T09:02:17.231Z [INFO]  server-f924816b-c8d3-7a29-d2d7-29409249810e: snapshot restore progress: id=2-23-1772269337230 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:17.231Z [INFO]  server-f924816b-c8d3-7a29-d2d7-29409249810e: Installed remote snapshot
2026-02-28T09:02:17.231Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: pipelining replication: peer="{Voter server-f924816b-c8d3-7a29-d2d7-29409249810e f924816b-c8d3-7a29-d2d7-29409249810e}"
2026-02-28T09:02:18.233Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:17.228235582 +0000 UTC m=+15.192032121 (2 nodes), 1.08µs from start of poll, 1.60129296s from cluster start. Timeout at 2026-02-28 09:02:18.233239142 +0000 UTC m=+16.197035681, 1.00500356s after stability
2026-02-28T09:02:18.246Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: aborting pipeline replication: peer="{Voter server-c60005bc-1a3f-3d65-5ce8-4daf2240c1a6 c60005bc-1a3f-3d65-5ce8-4daf2240c1a6}"
2026-02-28T09:02:18.246Z [INFO]  server-c67ec6a6-6c00-b6ab-ee65-a9d8d1713ff6: aborting pipeline replication: peer="{Voter server-f924816b-c8d3-7a29-d2d7-29409249810e f924816b-c8d3-7a29-d2d7-29409249810e}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false
2026-02-28T09:02:18.247Z [INFO]  server-84cdbd06-a22b-845e-526a-2960c048e206: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84cdbd06-a22b-845e-526a-2960c048e206 Address:84cdbd06-a22b-845e-526a-2960c048e206} {Suffrage:Voter ID:server-db907de7-ae6e-7207-7ab3-d8b77663a52d Address:db907de7-ae6e-7207-7ab3-d8b77663a52d} {Suffrage:Voter ID:server-21e530cf-314d-5b54-8af5-445ecb8f3403 Address:21e530cf-314d-5b54-8af5-445ecb8f3403}]"
2026-02-28T09:02:18.247Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84cdbd06-a22b-845e-526a-2960c048e206 Address:84cdbd06-a22b-845e-526a-2960c048e206} {Suffrage:Voter ID:server-db907de7-ae6e-7207-7ab3-d8b77663a52d Address:db907de7-ae6e-7207-7ab3-d8b77663a52d} {Suffrage:Voter ID:server-21e530cf-314d-5b54-8af5-445ecb8f3403 Address:21e530cf-314d-5b54-8af5-445ecb8f3403}]"
2026-02-28T09:02:18.247Z [INFO]  server-84cdbd06-a22b-845e-526a-2960c048e206: entering follower state: follower="Node at 84cdbd06-a22b-845e-526a-2960c048e206 [Follower]" leader-address= leader-id=
2026-02-28T09:02:18.247Z [INFO]  server-21e530cf-314d-5b54-8af5-445ecb8f3403: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84cdbd06-a22b-845e-526a-2960c048e206 Address:84cdbd06-a22b-845e-526a-2960c048e206} {Suffrage:Voter ID:server-db907de7-ae6e-7207-7ab3-d8b77663a52d Address:db907de7-ae6e-7207-7ab3-d8b77663a52d} {Suffrage:Voter ID:server-21e530cf-314d-5b54-8af5-445ecb8f3403 Address:21e530cf-314d-5b54-8af5-445ecb8f3403}]"
2026-02-28T09:02:18.247Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:18.247Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: entering follower state: follower="Node at db907de7-ae6e-7207-7ab3-d8b77663a52d [Follower]" leader-address= leader-id=
2026-02-28T09:02:18.247Z [INFO]  server-21e530cf-314d-5b54-8af5-445ecb8f3403: entering follower state: follower="Node at 21e530cf-314d-5b54-8af5-445ecb8f3403 [Follower]" leader-address= leader-id=
2026-02-28T09:02:19.138Z [WARN]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:19.138Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: entering candidate state: node="Node at db907de7-ae6e-7207-7ab3-d8b77663a52d [Candidate]" term=2
2026-02-28T09:02:19.138Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: election won: term=2 tally=2
2026-02-28T09:02:19.138Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: entering leader state: leader="Node at db907de7-ae6e-7207-7ab3-d8b77663a52d [Leader]"
2026-02-28T09:02:19.138Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: added peer, starting replication: peer=server-84cdbd06-a22b-845e-526a-2960c048e206
2026-02-28T09:02:19.138Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: added peer, starting replication: peer=server-21e530cf-314d-5b54-8af5-445ecb8f3403
2026-02-28T09:02:19.138Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: pipelining replication: peer="{Voter server-84cdbd06-a22b-845e-526a-2960c048e206 84cdbd06-a22b-845e-526a-2960c048e206}"
2026-02-28T09:02:19.139Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: pipelining replication: peer="{Voter server-21e530cf-314d-5b54-8af5-445ecb8f3403 21e530cf-314d-5b54-8af5-445ecb8f3403}"
2026-02-28T09:02:20.144Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:19.138839982 +0000 UTC m=+17.102636521 (1 nodes), 891.31212ms from start of poll, 891.74516ms from cluster start. Timeout at 2026-02-28 09:02:20.143841902 +0000 UTC m=+18.107638441, 1.00500192s after stability
2026-02-28T09:02:20.145Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: starting snapshot up to: index=12
2026-02-28T09:02:20.145Z [INFO]  creating new snapshot: path=/tmp/raft830916531/snapshots/2-12-1772269340145.tmp
2026-02-28T09:02:20.145Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: snapshot complete up to: index=12
2026-02-28T09:02:20.147Z [INFO]  creating new snapshot: path=/tmp/raft830916531/snapshots/2-23-1772269340147.tmp
2026-02-28T09:02:20.148Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: copied to local snapshot: bytes=71
2026-02-28T09:02:20.148Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: snapshot restore progress: id=2-23-1772269340147 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:20.148Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: removing all old logs from log store
2026-02-28T09:02:20.148Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: compacting logs: from=1 to=22
2026-02-28T09:02:20.148Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: restored user snapshot: index=23
2026-02-28T09:02:20.148Z [ERROR] server-db907de7-ae6e-7207-7ab3-d8b77663a52d: failed to get log: index=22 error="log not found"
2026-02-28T09:02:20.148Z [ERROR] server-db907de7-ae6e-7207-7ab3-d8b77663a52d: failed to get log: index=22 error="log not found"
2026-02-28T09:02:20.148Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: aborting pipeline replication: peer="{Voter server-84cdbd06-a22b-845e-526a-2960c048e206 84cdbd06-a22b-845e-526a-2960c048e206}"
2026-02-28T09:02:20.148Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: aborting pipeline replication: peer="{Voter server-21e530cf-314d-5b54-8af5-445ecb8f3403 21e530cf-314d-5b54-8af5-445ecb8f3403}"
2026-02-28T09:02:20.156Z [ERROR] server-db907de7-ae6e-7207-7ab3-d8b77663a52d: failed to get log: index=22 error="log not found"
2026-02-28T09:02:20.156Z [INFO]  creating new snapshot: path=/tmp/raft1185378159/snapshots/2-23-1772269340156.tmp
2026-02-28T09:02:20.157Z [INFO]  server-84cdbd06-a22b-845e-526a-2960c048e206: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:20.157Z [INFO]  server-84cdbd06-a22b-845e-526a-2960c048e206: copied to local snapshot: bytes=71
2026-02-28T09:02:20.157Z [ERROR] server-db907de7-ae6e-7207-7ab3-d8b77663a52d: failed to get log: index=22 error="log not found"
2026-02-28T09:02:20.157Z [INFO]  server-84cdbd06-a22b-845e-526a-2960c048e206: snapshot restore progress: id=2-23-1772269340156 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:20.157Z [INFO]  server-84cdbd06-a22b-845e-526a-2960c048e206: removing all old logs from log store
2026-02-28T09:02:20.157Z [INFO]  server-84cdbd06-a22b-845e-526a-2960c048e206: compacting logs: from=1 to=22
2026-02-28T09:02:20.157Z [INFO]  server-84cdbd06-a22b-845e-526a-2960c048e206: Installed remote snapshot
2026-02-28T09:02:20.157Z [WARN]  server-84cdbd06-a22b-845e-526a-2960c048e206: skipping application of old log: index=22
2026-02-28T09:02:20.157Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: pipelining replication: peer="{Voter server-84cdbd06-a22b-845e-526a-2960c048e206 84cdbd06-a22b-845e-526a-2960c048e206}"
2026-02-28T09:02:20.158Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:20.158Z [INFO]  creating new snapshot: path=/tmp/raft3765331459/snapshots/2-23-1772269340158.tmp
2026-02-28T09:02:20.158Z [INFO]  server-21e530cf-314d-5b54-8af5-445ecb8f3403: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:20.158Z [INFO]  server-21e530cf-314d-5b54-8af5-445ecb8f3403: copied to local snapshot: bytes=71
2026-02-28T09:02:20.159Z [INFO]  server-21e530cf-314d-5b54-8af5-445ecb8f3403: snapshot restore progress: id=2-23-1772269340158 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:20.159Z [INFO]  server-21e530cf-314d-5b54-8af5-445ecb8f3403: removing all old logs from log store
2026-02-28T09:02:20.159Z [INFO]  server-21e530cf-314d-5b54-8af5-445ecb8f3403: compacting logs: from=1 to=22
2026-02-28T09:02:20.159Z [INFO]  server-21e530cf-314d-5b54-8af5-445ecb8f3403: Installed remote snapshot
2026-02-28T09:02:20.159Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: pipelining replication: peer="{Voter server-21e530cf-314d-5b54-8af5-445ecb8f3403 21e530cf-314d-5b54-8af5-445ecb8f3403}"
2026-02-28T09:02:21.163Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:20.158077182 +0000 UTC m=+18.121873721 (2 nodes), 1.08µs from start of poll, 1.91098236s from cluster start. Timeout at 2026-02-28 09:02:21.163084862 +0000 UTC m=+19.126881441, 1.00500772s after stability
2026-02-28T09:02:21.176Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: aborting pipeline replication: peer="{Voter server-21e530cf-314d-5b54-8af5-445ecb8f3403 21e530cf-314d-5b54-8af5-445ecb8f3403}"
2026-02-28T09:02:21.176Z [INFO]  server-db907de7-ae6e-7207-7ab3-d8b77663a52d: aborting pipeline replication: peer="{Voter server-84cdbd06-a22b-845e-526a-2960c048e206 84cdbd06-a22b-845e-526a-2960c048e206}"
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true
2026-02-28T09:02:21.177Z [INFO]  server-f02630e6-6b06-c417-186e-c5dd3871a70f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f02630e6-6b06-c417-186e-c5dd3871a70f Address:f02630e6-6b06-c417-186e-c5dd3871a70f} {Suffrage:Voter ID:server-3023cd42-0e7d-0698-81e4-4c03194bbf20 Address:3023cd42-0e7d-0698-81e4-4c03194bbf20} {Suffrage:Voter ID:server-e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe Address:e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe}]"
2026-02-28T09:02:21.177Z [INFO]  server-3023cd42-0e7d-0698-81e4-4c03194bbf20: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f02630e6-6b06-c417-186e-c5dd3871a70f Address:f02630e6-6b06-c417-186e-c5dd3871a70f} {Suffrage:Voter ID:server-3023cd42-0e7d-0698-81e4-4c03194bbf20 Address:3023cd42-0e7d-0698-81e4-4c03194bbf20} {Suffrage:Voter ID:server-e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe Address:e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe}]"
2026-02-28T09:02:21.177Z [INFO]  server-f02630e6-6b06-c417-186e-c5dd3871a70f: entering follower state: follower="Node at f02630e6-6b06-c417-186e-c5dd3871a70f [Follower]" leader-address= leader-id=
2026-02-28T09:02:21.177Z [INFO]  server-3023cd42-0e7d-0698-81e4-4c03194bbf20: entering follower state: follower="Node at 3023cd42-0e7d-0698-81e4-4c03194bbf20 [Follower]" leader-address= leader-id=
2026-02-28T09:02:21.177Z [INFO]  server-e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f02630e6-6b06-c417-186e-c5dd3871a70f Address:f02630e6-6b06-c417-186e-c5dd3871a70f} {Suffrage:Voter ID:server-3023cd42-0e7d-0698-81e4-4c03194bbf20 Address:3023cd42-0e7d-0698-81e4-4c03194bbf20} {Suffrage:Voter ID:server-e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe Address:e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe}]"
2026-02-28T09:02:21.177Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:21.177Z [INFO]  server-e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe: entering follower state: follower="Node at e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe [Follower]" leader-address= leader-id=
2026-02-28T09:02:21.862Z [WARN]  server-e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:21.862Z [INFO]  server-e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe: entering candidate state: node="Node at e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe [Candidate]" term=2
2026-02-28T09:02:21.863Z [INFO]  server-e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe: election won: term=2 tally=2
2026-02-28T09:02:21.863Z [INFO]  server-e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe: entering leader state: leader="Node at e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe [Leader]"
2026-02-28T09:02:21.863Z [INFO]  server-e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe: added peer, starting replication: peer=server-f02630e6-6b06-c417-186e-c5dd3871a70f
2026-02-28T09:02:21.863Z [INFO]  server-e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe: added peer, starting replication: peer=server-3023cd42-0e7d-0698-81e4-4c03194bbf20
2026-02-28T09:02:21.863Z [INFO]  server-e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe: pipelining replication: peer="{Voter server-3023cd42-0e7d-0698-81e4-4c03194bbf20 3023cd42-0e7d-0698-81e4-4c03194bbf20}"
2026-02-28T09:02:21.863Z [INFO]  server-e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe: pipelining replication: peer="{Voter server-f02630e6-6b06-c417-186e-c5dd3871a70f f02630e6-6b06-c417-186e-c5dd3871a70f}"
2026-02-28T09:02:22.869Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:21.863111542 +0000 UTC m=+19.826908081 (1 nodes), 685.41356ms from start of poll, 685.89756ms from cluster start. Timeout at 2026-02-28 09:02:22.868112902 +0000 UTC m=+20.831909441, 1.00500136s after stability
2026-02-28T09:02:22.869Z [INFO]  server-e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe: starting snapshot up to: index=12
2026-02-28T09:02:22.869Z [INFO]  creating new snapshot: path=/tmp/raft3717866538/snapshots/2-12-1772269342869.tmp
2026-02-28T09:02:22.870Z [INFO]  server-e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe: snapshot complete up to: index=12
2026-02-28T09:02:22.871Z [INFO]  server-d860cbac-da48-b5d9-a80e-1dc3964688ae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d860cbac-da48-b5d9-a80e-1dc3964688ae Address:d860cbac-da48-b5d9-a80e-1dc3964688ae} {Suffrage:Voter ID:server-d85055ae-e26a-cb5f-9a29-b8a331c6e570 Address:d85055ae-e26a-cb5f-9a29-b8a331c6e570} {Suffrage:Voter ID:server-11768163-2c42-672b-f3a2-0ae140e519df Address:11768163-2c42-672b-f3a2-0ae140e519df}]"
2026-02-28T09:02:22.871Z [INFO]  server-d860cbac-da48-b5d9-a80e-1dc3964688ae: entering follower state: follower="Node at d860cbac-da48-b5d9-a80e-1dc3964688ae [Follower]" leader-address= leader-id=
2026-02-28T09:02:22.871Z [INFO]  server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d860cbac-da48-b5d9-a80e-1dc3964688ae Address:d860cbac-da48-b5d9-a80e-1dc3964688ae} {Suffrage:Voter ID:server-d85055ae-e26a-cb5f-9a29-b8a331c6e570 Address:d85055ae-e26a-cb5f-9a29-b8a331c6e570} {Suffrage:Voter ID:server-11768163-2c42-672b-f3a2-0ae140e519df Address:11768163-2c42-672b-f3a2-0ae140e519df}]"
2026-02-28T09:02:22.871Z [INFO]  server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: entering follower state: follower="Node at d85055ae-e26a-cb5f-9a29-b8a331c6e570 [Follower]" leader-address= leader-id=
2026-02-28T09:02:22.871Z [INFO]  server-11768163-2c42-672b-f3a2-0ae140e519df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d860cbac-da48-b5d9-a80e-1dc3964688ae Address:d860cbac-da48-b5d9-a80e-1dc3964688ae} {Suffrage:Voter ID:server-d85055ae-e26a-cb5f-9a29-b8a331c6e570 Address:d85055ae-e26a-cb5f-9a29-b8a331c6e570} {Suffrage:Voter ID:server-11768163-2c42-672b-f3a2-0ae140e519df Address:11768163-2c42-672b-f3a2-0ae140e519df}]"
2026-02-28T09:02:22.871Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:22.871Z [INFO]  server-11768163-2c42-672b-f3a2-0ae140e519df: entering follower state: follower="Node at 11768163-2c42-672b-f3a2-0ae140e519df [Follower]" leader-address= leader-id=
2026-02-28T09:02:23.529Z [WARN]  server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:23.529Z [INFO]  server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: entering candidate state: node="Node at d85055ae-e26a-cb5f-9a29-b8a331c6e570 [Candidate]" term=2
2026-02-28T09:02:23.529Z [INFO]  server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: election won: term=2 tally=2
2026-02-28T09:02:23.529Z [INFO]  server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: entering leader state: leader="Node at d85055ae-e26a-cb5f-9a29-b8a331c6e570 [Leader]"
2026-02-28T09:02:23.530Z [INFO]  server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: added peer, starting replication: peer=server-d860cbac-da48-b5d9-a80e-1dc3964688ae
2026-02-28T09:02:23.530Z [INFO]  server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: added peer, starting replication: peer=server-11768163-2c42-672b-f3a2-0ae140e519df
2026-02-28T09:02:23.530Z [INFO]  server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: pipelining replication: peer="{Voter server-11768163-2c42-672b-f3a2-0ae140e519df 11768163-2c42-672b-f3a2-0ae140e519df}"
2026-02-28T09:02:23.530Z [INFO]  server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: pipelining replication: peer="{Voter server-d860cbac-da48-b5d9-a80e-1dc3964688ae d860cbac-da48-b5d9-a80e-1dc3964688ae}"
2026-02-28T09:02:24.536Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:23.530129662 +0000 UTC m=+21.493926241 (1 nodes), 658.42544ms from start of poll, 658.87204ms from cluster start. Timeout at 2026-02-28 09:02:24.535133462 +0000 UTC m=+22.498930001, 1.00500376s after stability
2026-02-28T09:02:24.537Z [INFO]  creating new snapshot: path=/tmp/raft142309826/snapshots/2-13-1772269344537.tmp
2026-02-28T09:02:24.537Z [INFO]  server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: copied to local snapshot: bytes=71
2026-02-28T09:02:24.538Z [INFO]  server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: snapshot restore progress: id=2-13-1772269344537 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:24.538Z [INFO]  server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: restored user snapshot: index=13
2026-02-28T09:02:24.538Z [ERROR] server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: failed to get log: index=3 error="log not found"
2026-02-28T09:02:24.538Z [INFO]  server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: aborting pipeline replication: peer="{Voter server-d860cbac-da48-b5d9-a80e-1dc3964688ae d860cbac-da48-b5d9-a80e-1dc3964688ae}"
2026-02-28T09:02:24.538Z [ERROR] server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: failed to get log: index=3 error="log not found"
2026-02-28T09:02:24.538Z [INFO]  server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: aborting pipeline replication: peer="{Voter server-11768163-2c42-672b-f3a2-0ae140e519df 11768163-2c42-672b-f3a2-0ae140e519df}"
2026-02-28T09:02:24.546Z [ERROR] server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: failed to get log: index=3 error="log not found"
2026-02-28T09:02:24.546Z [ERROR] server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: failed to get log: index=3 error="log not found"
2026-02-28T09:02:24.547Z [INFO]  creating new snapshot: path=/tmp/raft3419005362/snapshots/2-13-1772269344547.tmp
2026-02-28T09:02:24.547Z [INFO]  creating new snapshot: path=/tmp/raft4083607694/snapshots/2-13-1772269344547.tmp
2026-02-28T09:02:24.547Z [INFO]  server-11768163-2c42-672b-f3a2-0ae140e519df: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:24.547Z [INFO]  server-d860cbac-da48-b5d9-a80e-1dc3964688ae: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:24.547Z [INFO]  server-11768163-2c42-672b-f3a2-0ae140e519df: copied to local snapshot: bytes=71
2026-02-28T09:02:24.547Z [INFO]  server-d860cbac-da48-b5d9-a80e-1dc3964688ae: copied to local snapshot: bytes=71
2026-02-28T09:02:24.548Z [INFO]  server-d860cbac-da48-b5d9-a80e-1dc3964688ae: snapshot restore progress: id=2-13-1772269344547 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:24.548Z [INFO]  server-11768163-2c42-672b-f3a2-0ae140e519df: snapshot restore progress: id=2-13-1772269344547 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:24.548Z [INFO]  server-d860cbac-da48-b5d9-a80e-1dc3964688ae: Installed remote snapshot
2026-02-28T09:02:24.548Z [INFO]  server-11768163-2c42-672b-f3a2-0ae140e519df: Installed remote snapshot
2026-02-28T09:02:24.548Z [INFO]  server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: pipelining replication: peer="{Voter server-11768163-2c42-672b-f3a2-0ae140e519df 11768163-2c42-672b-f3a2-0ae140e519df}"
2026-02-28T09:02:24.548Z [INFO]  server-d85055ae-e26a-cb5f-9a29-b8a331c6e570: pipelining replication: peer="{Voter server-d860cbac-da48-b5d9-a80e-1dc3964688ae d860cbac-da48-b5d9-a80e-1dc3964688ae}"
2026-02-28T09:02:24.548Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:25.554Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:24.548565142 +0000 UTC m=+22.512361681 (2 nodes), 1µs from start of poll, 1.67730748s from cluster start. Timeout at 2026-02-28 09:02:25.553575862 +0000 UTC m=+23.517372401, 1.00501072s after stability
2026-02-28T09:02:25.567Z [INFO]  server-e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe: aborting pipeline replication: peer="{Voter server-3023cd42-0e7d-0698-81e4-4c03194bbf20 3023cd42-0e7d-0698-81e4-4c03194bbf20}"
2026-02-28T09:02:25.567Z [INFO]  server-e8bb62e3-63c4-2540-b4dd-c0bbc511a7fe: aborting pipeline replication: peer="{Voter server-f02630e6-6b06-c417-186e-c5dd3871a70f f02630e6-6b06-c417-186e-c5dd3871a70f}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true
2026-02-28T09:02:25.568Z [INFO]  server-75d63eae-573c-0ae3-06a3-e28409f2968f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75d63eae-573c-0ae3-06a3-e28409f2968f Address:75d63eae-573c-0ae3-06a3-e28409f2968f} {Suffrage:Voter ID:server-5ca0b7a2-665e-f6e6-29a7-5d78ca1666ec Address:5ca0b7a2-665e-f6e6-29a7-5d78ca1666ec} {Suffrage:Voter ID:server-42b73ea6-2487-1169-76e2-dc7e462e7817 Address:42b73ea6-2487-1169-76e2-dc7e462e7817}]"
2026-02-28T09:02:25.568Z [INFO]  server-5ca0b7a2-665e-f6e6-29a7-5d78ca1666ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75d63eae-573c-0ae3-06a3-e28409f2968f Address:75d63eae-573c-0ae3-06a3-e28409f2968f} {Suffrage:Voter ID:server-5ca0b7a2-665e-f6e6-29a7-5d78ca1666ec Address:5ca0b7a2-665e-f6e6-29a7-5d78ca1666ec} {Suffrage:Voter ID:server-42b73ea6-2487-1169-76e2-dc7e462e7817 Address:42b73ea6-2487-1169-76e2-dc7e462e7817}]"
2026-02-28T09:02:25.568Z [INFO]  server-75d63eae-573c-0ae3-06a3-e28409f2968f: entering follower state: follower="Node at 75d63eae-573c-0ae3-06a3-e28409f2968f [Follower]" leader-address= leader-id=
2026-02-28T09:02:25.568Z [INFO]  server-5ca0b7a2-665e-f6e6-29a7-5d78ca1666ec: entering follower state: follower="Node at 5ca0b7a2-665e-f6e6-29a7-5d78ca1666ec [Follower]" leader-address= leader-id=
2026-02-28T09:02:25.568Z [INFO]  server-42b73ea6-2487-1169-76e2-dc7e462e7817: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75d63eae-573c-0ae3-06a3-e28409f2968f Address:75d63eae-573c-0ae3-06a3-e28409f2968f} {Suffrage:Voter ID:server-5ca0b7a2-665e-f6e6-29a7-5d78ca1666ec Address:5ca0b7a2-665e-f6e6-29a7-5d78ca1666ec} {Suffrage:Voter ID:server-42b73ea6-2487-1169-76e2-dc7e462e7817 Address:42b73ea6-2487-1169-76e2-dc7e462e7817}]"
2026-02-28T09:02:25.568Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:25.568Z [INFO]  server-42b73ea6-2487-1169-76e2-dc7e462e7817: entering follower state: follower="Node at 42b73ea6-2487-1169-76e2-dc7e462e7817 [Follower]" leader-address= leader-id=
2026-02-28T09:02:26.228Z [WARN]  server-75d63eae-573c-0ae3-06a3-e28409f2968f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:26.228Z [INFO]  server-75d63eae-573c-0ae3-06a3-e28409f2968f: entering candidate state: node="Node at 75d63eae-573c-0ae3-06a3-e28409f2968f [Candidate]" term=2
2026-02-28T09:02:26.228Z [INFO]  server-75d63eae-573c-0ae3-06a3-e28409f2968f: election won: term=2 tally=2
2026-02-28T09:02:26.228Z [INFO]  server-75d63eae-573c-0ae3-06a3-e28409f2968f: entering leader state: leader="Node at 75d63eae-573c-0ae3-06a3-e28409f2968f [Leader]"
2026-02-28T09:02:26.228Z [INFO]  server-75d63eae-573c-0ae3-06a3-e28409f2968f: added peer, starting replication: peer=server-5ca0b7a2-665e-f6e6-29a7-5d78ca1666ec
2026-02-28T09:02:26.228Z [INFO]  server-75d63eae-573c-0ae3-06a3-e28409f2968f: added peer, starting replication: peer=server-42b73ea6-2487-1169-76e2-dc7e462e7817
2026-02-28T09:02:26.228Z [INFO]  server-75d63eae-573c-0ae3-06a3-e28409f2968f: pipelining replication: peer="{Voter server-5ca0b7a2-665e-f6e6-29a7-5d78ca1666ec 5ca0b7a2-665e-f6e6-29a7-5d78ca1666ec}"
2026-02-28T09:02:26.229Z [INFO]  server-75d63eae-573c-0ae3-06a3-e28409f2968f: pipelining replication: peer="{Voter server-42b73ea6-2487-1169-76e2-dc7e462e7817 42b73ea6-2487-1169-76e2-dc7e462e7817}"
2026-02-28T09:02:27.234Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:26.228857982 +0000 UTC m=+24.192654561 (1 nodes), 660.0294ms from start of poll, 660.49716ms from cluster start. Timeout at 2026-02-28 09:02:27.233860422 +0000 UTC m=+25.197656961, 1.0050024s after stability
2026-02-28T09:02:27.234Z [INFO]  server-75d63eae-573c-0ae3-06a3-e28409f2968f: starting snapshot up to: index=12
2026-02-28T09:02:27.234Z [INFO]  creating new snapshot: path=/tmp/raft2341740808/snapshots/2-12-1772269347234.tmp
2026-02-28T09:02:27.235Z [INFO]  server-75d63eae-573c-0ae3-06a3-e28409f2968f: snapshot complete up to: index=12
2026-02-28T09:02:27.236Z [INFO]  server-d0824ac7-8162-fe30-ab21-cb1973edad63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0824ac7-8162-fe30-ab21-cb1973edad63 Address:d0824ac7-8162-fe30-ab21-cb1973edad63} {Suffrage:Voter ID:server-ae0aeae6-4831-4db0-ff77-d3f54baa7018 Address:ae0aeae6-4831-4db0-ff77-d3f54baa7018} {Suffrage:Voter ID:server-c36d2e1a-b3d7-c390-c7b7-5446165f287c Address:c36d2e1a-b3d7-c390-c7b7-5446165f287c}]"
2026-02-28T09:02:27.236Z [INFO]  server-d0824ac7-8162-fe30-ab21-cb1973edad63: entering follower state: follower="Node at d0824ac7-8162-fe30-ab21-cb1973edad63 [Follower]" leader-address= leader-id=
2026-02-28T09:02:27.236Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0824ac7-8162-fe30-ab21-cb1973edad63 Address:d0824ac7-8162-fe30-ab21-cb1973edad63} {Suffrage:Voter ID:server-ae0aeae6-4831-4db0-ff77-d3f54baa7018 Address:ae0aeae6-4831-4db0-ff77-d3f54baa7018} {Suffrage:Voter ID:server-c36d2e1a-b3d7-c390-c7b7-5446165f287c Address:c36d2e1a-b3d7-c390-c7b7-5446165f287c}]"
2026-02-28T09:02:27.237Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: entering follower state: follower="Node at ae0aeae6-4831-4db0-ff77-d3f54baa7018 [Follower]" leader-address= leader-id=
2026-02-28T09:02:27.237Z [INFO]  server-c36d2e1a-b3d7-c390-c7b7-5446165f287c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0824ac7-8162-fe30-ab21-cb1973edad63 Address:d0824ac7-8162-fe30-ab21-cb1973edad63} {Suffrage:Voter ID:server-ae0aeae6-4831-4db0-ff77-d3f54baa7018 Address:ae0aeae6-4831-4db0-ff77-d3f54baa7018} {Suffrage:Voter ID:server-c36d2e1a-b3d7-c390-c7b7-5446165f287c Address:c36d2e1a-b3d7-c390-c7b7-5446165f287c}]"
2026-02-28T09:02:27.237Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:27.237Z [INFO]  server-c36d2e1a-b3d7-c390-c7b7-5446165f287c: entering follower state: follower="Node at c36d2e1a-b3d7-c390-c7b7-5446165f287c [Follower]" leader-address= leader-id=
2026-02-28T09:02:28.044Z [WARN]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:28.044Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: entering candidate state: node="Node at ae0aeae6-4831-4db0-ff77-d3f54baa7018 [Candidate]" term=2
2026-02-28T09:02:28.044Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: election won: term=2 tally=2
2026-02-28T09:02:28.044Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: entering leader state: leader="Node at ae0aeae6-4831-4db0-ff77-d3f54baa7018 [Leader]"
2026-02-28T09:02:28.044Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: added peer, starting replication: peer=server-d0824ac7-8162-fe30-ab21-cb1973edad63
2026-02-28T09:02:28.044Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: added peer, starting replication: peer=server-c36d2e1a-b3d7-c390-c7b7-5446165f287c
2026-02-28T09:02:28.044Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: pipelining replication: peer="{Voter server-c36d2e1a-b3d7-c390-c7b7-5446165f287c c36d2e1a-b3d7-c390-c7b7-5446165f287c}"
2026-02-28T09:02:28.045Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: pipelining replication: peer="{Voter server-d0824ac7-8162-fe30-ab21-cb1973edad63 d0824ac7-8162-fe30-ab21-cb1973edad63}"
2026-02-28T09:02:29.050Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:28.044683262 +0000 UTC m=+26.008479801 (1 nodes), 806.71768ms from start of poll, 808.40768ms from cluster start. Timeout at 2026-02-28 09:02:29.049686422 +0000 UTC m=+27.013482961, 1.00500316s after stability
2026-02-28T09:02:29.050Z [INFO]  creating new snapshot: path=/tmp/raft1607367085/snapshots/2-13-1772269349050.tmp
2026-02-28T09:02:29.051Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: copied to local snapshot: bytes=71
2026-02-28T09:02:29.051Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: snapshot restore progress: id=2-13-1772269349050 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:29.051Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: removing all old logs from log store
2026-02-28T09:02:29.051Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: compacting logs: from=1 to=2
2026-02-28T09:02:29.051Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: restored user snapshot: index=13
2026-02-28T09:02:29.051Z [ERROR] server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: failed to get log: index=2 error="log not found"
2026-02-28T09:02:29.051Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: aborting pipeline replication: peer="{Voter server-c36d2e1a-b3d7-c390-c7b7-5446165f287c c36d2e1a-b3d7-c390-c7b7-5446165f287c}"
2026-02-28T09:02:29.051Z [ERROR] server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: failed to get log: index=2 error="log not found"
2026-02-28T09:02:29.051Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: aborting pipeline replication: peer="{Voter server-d0824ac7-8162-fe30-ab21-cb1973edad63 d0824ac7-8162-fe30-ab21-cb1973edad63}"
2026-02-28T09:02:29.058Z [ERROR] server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: failed to get log: index=2 error="log not found"
2026-02-28T09:02:29.058Z [INFO]  creating new snapshot: path=/tmp/raft127024299/snapshots/2-13-1772269349058.tmp
2026-02-28T09:02:29.059Z [INFO]  server-d0824ac7-8162-fe30-ab21-cb1973edad63: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:29.059Z [INFO]  server-d0824ac7-8162-fe30-ab21-cb1973edad63: copied to local snapshot: bytes=71
2026-02-28T09:02:29.060Z [INFO]  server-d0824ac7-8162-fe30-ab21-cb1973edad63: snapshot restore progress: id=2-13-1772269349058 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:29.060Z [INFO]  server-d0824ac7-8162-fe30-ab21-cb1973edad63: removing all old logs from log store
2026-02-28T09:02:29.060Z [INFO]  server-d0824ac7-8162-fe30-ab21-cb1973edad63: compacting logs: from=1 to=2
2026-02-28T09:02:29.060Z [INFO]  server-d0824ac7-8162-fe30-ab21-cb1973edad63: Installed remote snapshot
2026-02-28T09:02:29.060Z [ERROR] server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: failed to get log: index=2 error="log not found"
2026-02-28T09:02:29.060Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: pipelining replication: peer="{Voter server-d0824ac7-8162-fe30-ab21-cb1973edad63 d0824ac7-8162-fe30-ab21-cb1973edad63}"
2026-02-28T09:02:29.060Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:29.061Z [INFO]  creating new snapshot: path=/tmp/raft3773305949/snapshots/2-13-1772269349061.tmp
2026-02-28T09:02:29.061Z [INFO]  server-c36d2e1a-b3d7-c390-c7b7-5446165f287c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:29.061Z [INFO]  server-c36d2e1a-b3d7-c390-c7b7-5446165f287c: copied to local snapshot: bytes=71
2026-02-28T09:02:29.061Z [INFO]  server-c36d2e1a-b3d7-c390-c7b7-5446165f287c: snapshot restore progress: id=2-13-1772269349061 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:29.061Z [INFO]  server-c36d2e1a-b3d7-c390-c7b7-5446165f287c: removing all old logs from log store
2026-02-28T09:02:29.061Z [INFO]  server-c36d2e1a-b3d7-c390-c7b7-5446165f287c: compacting logs: from=1 to=2
2026-02-28T09:02:29.061Z [INFO]  server-c36d2e1a-b3d7-c390-c7b7-5446165f287c: Installed remote snapshot
2026-02-28T09:02:29.062Z [INFO]  server-ae0aeae6-4831-4db0-ff77-d3f54baa7018: pipelining replication: peer="{Voter server-c36d2e1a-b3d7-c390-c7b7-5446165f287c c36d2e1a-b3d7-c390-c7b7-5446165f287c}"
2026-02-28T09:02:30.066Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:29.060848422 +0000 UTC m=+27.024644961 (2 nodes), 2µs from start of poll, 1.82457284s from cluster start. Timeout at 2026-02-28 09:02:30.065864622 +0000 UTC m=+28.029661161, 1.0050162s after stability
2026-02-28T09:02:30.078Z [INFO]  server-75d63eae-573c-0ae3-06a3-e28409f2968f: aborting pipeline replication: peer="{Voter server-42b73ea6-2487-1169-76e2-dc7e462e7817 42b73ea6-2487-1169-76e2-dc7e462e7817}"
2026-02-28T09:02:30.078Z [INFO]  server-75d63eae-573c-0ae3-06a3-e28409f2968f: aborting pipeline replication: peer="{Voter server-5ca0b7a2-665e-f6e6-29a7-5d78ca1666ec 5ca0b7a2-665e-f6e6-29a7-5d78ca1666ec}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false
2026-02-28T09:02:30.079Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01d12691-5581-6715-1119-9f4d5618a68f Address:01d12691-5581-6715-1119-9f4d5618a68f} {Suffrage:Voter ID:server-927862ef-1d4d-ecf2-353b-534af2245555 Address:927862ef-1d4d-ecf2-353b-534af2245555} {Suffrage:Voter ID:server-95a8e582-cb90-81c6-e118-49313b64b2f8 Address:95a8e582-cb90-81c6-e118-49313b64b2f8}]"
2026-02-28T09:02:30.079Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: entering follower state: follower="Node at 01d12691-5581-6715-1119-9f4d5618a68f [Follower]" leader-address= leader-id=
2026-02-28T09:02:30.079Z [INFO]  server-927862ef-1d4d-ecf2-353b-534af2245555: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01d12691-5581-6715-1119-9f4d5618a68f Address:01d12691-5581-6715-1119-9f4d5618a68f} {Suffrage:Voter ID:server-927862ef-1d4d-ecf2-353b-534af2245555 Address:927862ef-1d4d-ecf2-353b-534af2245555} {Suffrage:Voter ID:server-95a8e582-cb90-81c6-e118-49313b64b2f8 Address:95a8e582-cb90-81c6-e118-49313b64b2f8}]"
2026-02-28T09:02:30.080Z [INFO]  server-927862ef-1d4d-ecf2-353b-534af2245555: entering follower state: follower="Node at 927862ef-1d4d-ecf2-353b-534af2245555 [Follower]" leader-address= leader-id=
2026-02-28T09:02:30.080Z [INFO]  server-95a8e582-cb90-81c6-e118-49313b64b2f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01d12691-5581-6715-1119-9f4d5618a68f Address:01d12691-5581-6715-1119-9f4d5618a68f} {Suffrage:Voter ID:server-927862ef-1d4d-ecf2-353b-534af2245555 Address:927862ef-1d4d-ecf2-353b-534af2245555} {Suffrage:Voter ID:server-95a8e582-cb90-81c6-e118-49313b64b2f8 Address:95a8e582-cb90-81c6-e118-49313b64b2f8}]"
2026-02-28T09:02:30.080Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:30.080Z [INFO]  server-95a8e582-cb90-81c6-e118-49313b64b2f8: entering follower state: follower="Node at 95a8e582-cb90-81c6-e118-49313b64b2f8 [Follower]" leader-address= leader-id=
2026-02-28T09:02:30.599Z [WARN]  server-01d12691-5581-6715-1119-9f4d5618a68f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:30.599Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: entering candidate state: node="Node at 01d12691-5581-6715-1119-9f4d5618a68f [Candidate]" term=2
2026-02-28T09:02:30.599Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: election won: term=2 tally=2
2026-02-28T09:02:30.599Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: entering leader state: leader="Node at 01d12691-5581-6715-1119-9f4d5618a68f [Leader]"
2026-02-28T09:02:30.599Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: added peer, starting replication: peer=server-927862ef-1d4d-ecf2-353b-534af2245555
2026-02-28T09:02:30.599Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: added peer, starting replication: peer=server-95a8e582-cb90-81c6-e118-49313b64b2f8
2026-02-28T09:02:30.599Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: pipelining replication: peer="{Voter server-927862ef-1d4d-ecf2-353b-534af2245555 927862ef-1d4d-ecf2-353b-534af2245555}"
2026-02-28T09:02:30.600Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: pipelining replication: peer="{Voter server-95a8e582-cb90-81c6-e118-49313b64b2f8 95a8e582-cb90-81c6-e118-49313b64b2f8}"
2026-02-28T09:02:31.605Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:30.599904702 +0000 UTC m=+28.563701281 (1 nodes), 519.74428ms from start of poll, 520.24928ms from cluster start. Timeout at 2026-02-28 09:02:31.604908262 +0000 UTC m=+29.568704841, 1.00500356s after stability
2026-02-28T09:02:31.606Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: starting snapshot up to: index=12
2026-02-28T09:02:31.606Z [INFO]  creating new snapshot: path=/tmp/raft2039864754/snapshots/2-12-1772269351606.tmp
2026-02-28T09:02:31.606Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: snapshot complete up to: index=12
2026-02-28T09:02:31.614Z [INFO]  creating new snapshot: path=/tmp/raft2039864754/snapshots/2-23-1772269351614.tmp
2026-02-28T09:02:31.615Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: copied to local snapshot: bytes=71
2026-02-28T09:02:31.615Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: snapshot restore progress: id=2-23-1772269351614 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:31.619Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: restored user snapshot: index=23
2026-02-28T09:02:31.619Z [ERROR] server-01d12691-5581-6715-1119-9f4d5618a68f: failed to get log: index=23 error="log not found"
2026-02-28T09:02:31.619Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: aborting pipeline replication: peer="{Voter server-95a8e582-cb90-81c6-e118-49313b64b2f8 95a8e582-cb90-81c6-e118-49313b64b2f8}"
2026-02-28T09:02:31.619Z [ERROR] server-01d12691-5581-6715-1119-9f4d5618a68f: failed to get log: index=23 error="log not found"
2026-02-28T09:02:31.619Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: aborting pipeline replication: peer="{Voter server-927862ef-1d4d-ecf2-353b-534af2245555 927862ef-1d4d-ecf2-353b-534af2245555}"
2026-02-28T09:02:31.627Z [ERROR] server-01d12691-5581-6715-1119-9f4d5618a68f: failed to get log: index=23 error="log not found"
2026-02-28T09:02:31.627Z [INFO]  creating new snapshot: path=/tmp/raft1446101848/snapshots/2-23-1772269351627.tmp
2026-02-28T09:02:31.628Z [INFO]  server-927862ef-1d4d-ecf2-353b-534af2245555: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:31.628Z [INFO]  server-927862ef-1d4d-ecf2-353b-534af2245555: copied to local snapshot: bytes=71
2026-02-28T09:02:31.628Z [INFO]  server-927862ef-1d4d-ecf2-353b-534af2245555: snapshot restore progress: id=2-23-1772269351627 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:31.628Z [INFO]  server-927862ef-1d4d-ecf2-353b-534af2245555: Installed remote snapshot
2026-02-28T09:02:31.628Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: pipelining replication: peer="{Voter server-927862ef-1d4d-ecf2-353b-534af2245555 927862ef-1d4d-ecf2-353b-534af2245555}"
2026-02-28T09:02:31.628Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:31.629Z [ERROR] server-01d12691-5581-6715-1119-9f4d5618a68f: failed to get log: index=23 error="log not found"
2026-02-28T09:02:31.629Z [INFO]  creating new snapshot: path=/tmp/raft1919354599/snapshots/2-23-1772269351629.tmp
2026-02-28T09:02:31.630Z [INFO]  server-95a8e582-cb90-81c6-e118-49313b64b2f8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:31.630Z [INFO]  server-95a8e582-cb90-81c6-e118-49313b64b2f8: copied to local snapshot: bytes=71
2026-02-28T09:02:31.630Z [INFO]  server-95a8e582-cb90-81c6-e118-49313b64b2f8: snapshot restore progress: id=2-23-1772269351629 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:31.630Z [INFO]  server-95a8e582-cb90-81c6-e118-49313b64b2f8: Installed remote snapshot
2026-02-28T09:02:31.630Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: pipelining replication: peer="{Voter server-95a8e582-cb90-81c6-e118-49313b64b2f8 95a8e582-cb90-81c6-e118-49313b64b2f8}"
2026-02-28T09:02:32.634Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:31.628951862 +0000 UTC m=+29.592748401 (2 nodes), 1.04µs from start of poll, 1.5492964s from cluster start. Timeout at 2026-02-28 09:02:32.633963102 +0000 UTC m=+30.597759641, 1.00501124s after stability
2026-02-28T09:02:32.646Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: aborting pipeline replication: peer="{Voter server-95a8e582-cb90-81c6-e118-49313b64b2f8 95a8e582-cb90-81c6-e118-49313b64b2f8}"
2026-02-28T09:02:32.646Z [INFO]  server-01d12691-5581-6715-1119-9f4d5618a68f: aborting pipeline replication: peer="{Voter server-927862ef-1d4d-ecf2-353b-534af2245555 927862ef-1d4d-ecf2-353b-534af2245555}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false
2026-02-28T09:02:32.647Z [INFO]  server-ac94f060-4bdc-64a5-2197-5237c469e92c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac94f060-4bdc-64a5-2197-5237c469e92c Address:ac94f060-4bdc-64a5-2197-5237c469e92c} {Suffrage:Voter ID:server-5461e43f-b274-a475-ac67-1178dff6d7da Address:5461e43f-b274-a475-ac67-1178dff6d7da} {Suffrage:Voter ID:server-feef2ddb-046c-8ecc-0ba3-43996c079727 Address:feef2ddb-046c-8ecc-0ba3-43996c079727}]"
2026-02-28T09:02:32.648Z [INFO]  server-5461e43f-b274-a475-ac67-1178dff6d7da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac94f060-4bdc-64a5-2197-5237c469e92c Address:ac94f060-4bdc-64a5-2197-5237c469e92c} {Suffrage:Voter ID:server-5461e43f-b274-a475-ac67-1178dff6d7da Address:5461e43f-b274-a475-ac67-1178dff6d7da} {Suffrage:Voter ID:server-feef2ddb-046c-8ecc-0ba3-43996c079727 Address:feef2ddb-046c-8ecc-0ba3-43996c079727}]"
2026-02-28T09:02:32.648Z [INFO]  server-ac94f060-4bdc-64a5-2197-5237c469e92c: entering follower state: follower="Node at ac94f060-4bdc-64a5-2197-5237c469e92c [Follower]" leader-address= leader-id=
2026-02-28T09:02:32.648Z [INFO]  server-5461e43f-b274-a475-ac67-1178dff6d7da: entering follower state: follower="Node at 5461e43f-b274-a475-ac67-1178dff6d7da [Follower]" leader-address= leader-id=
2026-02-28T09:02:32.648Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac94f060-4bdc-64a5-2197-5237c469e92c Address:ac94f060-4bdc-64a5-2197-5237c469e92c} {Suffrage:Voter ID:server-5461e43f-b274-a475-ac67-1178dff6d7da Address:5461e43f-b274-a475-ac67-1178dff6d7da} {Suffrage:Voter ID:server-feef2ddb-046c-8ecc-0ba3-43996c079727 Address:feef2ddb-046c-8ecc-0ba3-43996c079727}]"
2026-02-28T09:02:32.648Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:32.648Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: entering follower state: follower="Node at feef2ddb-046c-8ecc-0ba3-43996c079727 [Follower]" leader-address= leader-id=
2026-02-28T09:02:33.208Z [WARN]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:33.208Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: entering candidate state: node="Node at feef2ddb-046c-8ecc-0ba3-43996c079727 [Candidate]" term=2
2026-02-28T09:02:33.209Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: election won: term=2 tally=2
2026-02-28T09:02:33.209Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: entering leader state: leader="Node at feef2ddb-046c-8ecc-0ba3-43996c079727 [Leader]"
2026-02-28T09:02:33.209Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: added peer, starting replication: peer=server-ac94f060-4bdc-64a5-2197-5237c469e92c
2026-02-28T09:02:33.209Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: added peer, starting replication: peer=server-5461e43f-b274-a475-ac67-1178dff6d7da
2026-02-28T09:02:33.209Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: pipelining replication: peer="{Voter server-5461e43f-b274-a475-ac67-1178dff6d7da 5461e43f-b274-a475-ac67-1178dff6d7da}"
2026-02-28T09:02:33.209Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: pipelining replication: peer="{Voter server-ac94f060-4bdc-64a5-2197-5237c469e92c ac94f060-4bdc-64a5-2197-5237c469e92c}"
2026-02-28T09:02:34.214Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:33.209189342 +0000 UTC m=+31.172985881 (1 nodes), 560.34232ms from start of poll, 561.53364ms from cluster start. Timeout at 2026-02-28 09:02:34.214191982 +0000 UTC m=+32.177988521, 1.00500264s after stability
2026-02-28T09:02:34.214Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: starting snapshot up to: index=12
2026-02-28T09:02:34.215Z [INFO]  creating new snapshot: path=/tmp/raft121670034/snapshots/2-12-1772269354215.tmp
2026-02-28T09:02:34.215Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: snapshot complete up to: index=12
2026-02-28T09:02:34.216Z [INFO]  creating new snapshot: path=/tmp/raft121670034/snapshots/2-23-1772269354216.tmp
2026-02-28T09:02:34.217Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: copied to local snapshot: bytes=71
2026-02-28T09:02:34.217Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: snapshot restore progress: id=2-23-1772269354216 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:34.218Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: removing all old logs from log store
2026-02-28T09:02:34.218Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: compacting logs: from=1 to=22
2026-02-28T09:02:34.218Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: restored user snapshot: index=23
2026-02-28T09:02:34.218Z [ERROR] server-feef2ddb-046c-8ecc-0ba3-43996c079727: failed to get log: index=22 error="log not found"
2026-02-28T09:02:34.218Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: aborting pipeline replication: peer="{Voter server-5461e43f-b274-a475-ac67-1178dff6d7da 5461e43f-b274-a475-ac67-1178dff6d7da}"
2026-02-28T09:02:34.218Z [ERROR] server-feef2ddb-046c-8ecc-0ba3-43996c079727: failed to get log: index=22 error="log not found"
2026-02-28T09:02:34.218Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: aborting pipeline replication: peer="{Voter server-ac94f060-4bdc-64a5-2197-5237c469e92c ac94f060-4bdc-64a5-2197-5237c469e92c}"
2026-02-28T09:02:34.226Z [ERROR] server-feef2ddb-046c-8ecc-0ba3-43996c079727: failed to get log: index=22 error="log not found"
2026-02-28T09:02:34.227Z [INFO]  creating new snapshot: path=/tmp/raft1296515497/snapshots/2-23-1772269354227.tmp
2026-02-28T09:02:34.227Z [INFO]  server-5461e43f-b274-a475-ac67-1178dff6d7da: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:34.227Z [ERROR] server-feef2ddb-046c-8ecc-0ba3-43996c079727: failed to get log: index=22 error="log not found"
2026-02-28T09:02:34.228Z [INFO]  server-5461e43f-b274-a475-ac67-1178dff6d7da: copied to local snapshot: bytes=71
2026-02-28T09:02:34.228Z [INFO]  creating new snapshot: path=/tmp/raft663763837/snapshots/2-23-1772269354228.tmp
2026-02-28T09:02:34.228Z [INFO]  server-5461e43f-b274-a475-ac67-1178dff6d7da: snapshot restore progress: id=2-23-1772269354227 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:34.229Z [INFO]  server-5461e43f-b274-a475-ac67-1178dff6d7da: removing all old logs from log store
2026-02-28T09:02:34.229Z [INFO]  server-5461e43f-b274-a475-ac67-1178dff6d7da: compacting logs: from=1 to=22
2026-02-28T09:02:34.229Z [INFO]  server-5461e43f-b274-a475-ac67-1178dff6d7da: Installed remote snapshot
2026-02-28T09:02:34.229Z [WARN]  server-5461e43f-b274-a475-ac67-1178dff6d7da: skipping application of old log: index=22
2026-02-28T09:02:34.229Z [INFO]  server-ac94f060-4bdc-64a5-2197-5237c469e92c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:34.229Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: pipelining replication: peer="{Voter server-5461e43f-b274-a475-ac67-1178dff6d7da 5461e43f-b274-a475-ac67-1178dff6d7da}"
2026-02-28T09:02:34.229Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:34.229Z [INFO]  server-ac94f060-4bdc-64a5-2197-5237c469e92c: copied to local snapshot: bytes=71
2026-02-28T09:02:34.229Z [INFO]  server-ac94f060-4bdc-64a5-2197-5237c469e92c: snapshot restore progress: id=2-23-1772269354228 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:34.230Z [INFO]  server-ac94f060-4bdc-64a5-2197-5237c469e92c: removing all old logs from log store
2026-02-28T09:02:34.230Z [INFO]  server-ac94f060-4bdc-64a5-2197-5237c469e92c: compacting logs: from=1 to=22
2026-02-28T09:02:34.230Z [INFO]  server-ac94f060-4bdc-64a5-2197-5237c469e92c: Installed remote snapshot
2026-02-28T09:02:34.230Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: pipelining replication: peer="{Voter server-ac94f060-4bdc-64a5-2197-5237c469e92c ac94f060-4bdc-64a5-2197-5237c469e92c}"
2026-02-28T09:02:35.235Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:34.229492262 +0000 UTC m=+32.193288801 (2 nodes), 2.2µs from start of poll, 1.58183656s from cluster start. Timeout at 2026-02-28 09:02:35.234498622 +0000 UTC m=+33.198295161, 1.00500636s after stability
2026-02-28T09:02:35.248Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: aborting pipeline replication: peer="{Voter server-5461e43f-b274-a475-ac67-1178dff6d7da 5461e43f-b274-a475-ac67-1178dff6d7da}"
2026-02-28T09:02:35.248Z [INFO]  server-feef2ddb-046c-8ecc-0ba3-43996c079727: aborting pipeline replication: peer="{Voter server-ac94f060-4bdc-64a5-2197-5237c469e92c ac94f060-4bdc-64a5-2197-5237c469e92c}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true
2026-02-28T09:02:35.249Z [INFO]  server-70091b02-88cf-fd68-bb40-53d7f682f2a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70091b02-88cf-fd68-bb40-53d7f682f2a8 Address:70091b02-88cf-fd68-bb40-53d7f682f2a8} {Suffrage:Voter ID:server-afdb1e71-b266-12d4-153f-1c2df9037cf0 Address:afdb1e71-b266-12d4-153f-1c2df9037cf0} {Suffrage:Voter ID:server-7d4d76ec-6a29-3e72-2754-17ec49570e3f Address:7d4d76ec-6a29-3e72-2754-17ec49570e3f}]"
2026-02-28T09:02:35.249Z [INFO]  server-afdb1e71-b266-12d4-153f-1c2df9037cf0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70091b02-88cf-fd68-bb40-53d7f682f2a8 Address:70091b02-88cf-fd68-bb40-53d7f682f2a8} {Suffrage:Voter ID:server-afdb1e71-b266-12d4-153f-1c2df9037cf0 Address:afdb1e71-b266-12d4-153f-1c2df9037cf0} {Suffrage:Voter ID:server-7d4d76ec-6a29-3e72-2754-17ec49570e3f Address:7d4d76ec-6a29-3e72-2754-17ec49570e3f}]"
2026-02-28T09:02:35.249Z [INFO]  server-70091b02-88cf-fd68-bb40-53d7f682f2a8: entering follower state: follower="Node at 70091b02-88cf-fd68-bb40-53d7f682f2a8 [Follower]" leader-address= leader-id=
2026-02-28T09:02:35.249Z [INFO]  server-afdb1e71-b266-12d4-153f-1c2df9037cf0: entering follower state: follower="Node at afdb1e71-b266-12d4-153f-1c2df9037cf0 [Follower]" leader-address= leader-id=
2026-02-28T09:02:35.249Z [INFO]  server-7d4d76ec-6a29-3e72-2754-17ec49570e3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70091b02-88cf-fd68-bb40-53d7f682f2a8 Address:70091b02-88cf-fd68-bb40-53d7f682f2a8} {Suffrage:Voter ID:server-afdb1e71-b266-12d4-153f-1c2df9037cf0 Address:afdb1e71-b266-12d4-153f-1c2df9037cf0} {Suffrage:Voter ID:server-7d4d76ec-6a29-3e72-2754-17ec49570e3f Address:7d4d76ec-6a29-3e72-2754-17ec49570e3f}]"
2026-02-28T09:02:35.249Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:35.249Z [INFO]  server-7d4d76ec-6a29-3e72-2754-17ec49570e3f: entering follower state: follower="Node at 7d4d76ec-6a29-3e72-2754-17ec49570e3f [Follower]" leader-address= leader-id=
2026-02-28T09:02:35.808Z [WARN]  server-70091b02-88cf-fd68-bb40-53d7f682f2a8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:35.808Z [INFO]  server-70091b02-88cf-fd68-bb40-53d7f682f2a8: entering candidate state: node="Node at 70091b02-88cf-fd68-bb40-53d7f682f2a8 [Candidate]" term=2
2026-02-28T09:02:35.808Z [INFO]  server-70091b02-88cf-fd68-bb40-53d7f682f2a8: election won: term=2 tally=2
2026-02-28T09:02:35.808Z [INFO]  server-70091b02-88cf-fd68-bb40-53d7f682f2a8: entering leader state: leader="Node at 70091b02-88cf-fd68-bb40-53d7f682f2a8 [Leader]"
2026-02-28T09:02:35.808Z [INFO]  server-70091b02-88cf-fd68-bb40-53d7f682f2a8: added peer, starting replication: peer=server-afdb1e71-b266-12d4-153f-1c2df9037cf0
2026-02-28T09:02:35.808Z [INFO]  server-70091b02-88cf-fd68-bb40-53d7f682f2a8: added peer, starting replication: peer=server-7d4d76ec-6a29-3e72-2754-17ec49570e3f
2026-02-28T09:02:35.809Z [INFO]  server-70091b02-88cf-fd68-bb40-53d7f682f2a8: pipelining replication: peer="{Voter server-afdb1e71-b266-12d4-153f-1c2df9037cf0 afdb1e71-b266-12d4-153f-1c2df9037cf0}"
2026-02-28T09:02:35.809Z [INFO]  server-70091b02-88cf-fd68-bb40-53d7f682f2a8: pipelining replication: peer="{Voter server-7d4d76ec-6a29-3e72-2754-17ec49570e3f 7d4d76ec-6a29-3e72-2754-17ec49570e3f}"
2026-02-28T09:02:36.814Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:35.808849782 +0000 UTC m=+33.772646321 (1 nodes), 559.0156ms from start of poll, 559.49732ms from cluster start. Timeout at 2026-02-28 09:02:36.813851622 +0000 UTC m=+34.777648161, 1.00500184s after stability
2026-02-28T09:02:36.815Z [INFO]  server-70091b02-88cf-fd68-bb40-53d7f682f2a8: starting snapshot up to: index=12
2026-02-28T09:02:36.815Z [INFO]  creating new snapshot: path=/tmp/raft26640121/snapshots/2-12-1772269356815.tmp
2026-02-28T09:02:36.815Z [INFO]  server-70091b02-88cf-fd68-bb40-53d7f682f2a8: snapshot complete up to: index=12
2026-02-28T09:02:36.816Z [INFO]  server-7c94535e-e214-8d0c-0d16-838be0297f21: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c94535e-e214-8d0c-0d16-838be0297f21 Address:7c94535e-e214-8d0c-0d16-838be0297f21} {Suffrage:Voter ID:server-cec4b35b-eb12-dd42-1a92-234f116c6628 Address:cec4b35b-eb12-dd42-1a92-234f116c6628} {Suffrage:Voter ID:server-7989b60e-58be-98ad-8b04-7408ca66a815 Address:7989b60e-58be-98ad-8b04-7408ca66a815}]"
2026-02-28T09:02:36.817Z [INFO]  server-cec4b35b-eb12-dd42-1a92-234f116c6628: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c94535e-e214-8d0c-0d16-838be0297f21 Address:7c94535e-e214-8d0c-0d16-838be0297f21} {Suffrage:Voter ID:server-cec4b35b-eb12-dd42-1a92-234f116c6628 Address:cec4b35b-eb12-dd42-1a92-234f116c6628} {Suffrage:Voter ID:server-7989b60e-58be-98ad-8b04-7408ca66a815 Address:7989b60e-58be-98ad-8b04-7408ca66a815}]"
2026-02-28T09:02:36.817Z [INFO]  server-7989b60e-58be-98ad-8b04-7408ca66a815: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c94535e-e214-8d0c-0d16-838be0297f21 Address:7c94535e-e214-8d0c-0d16-838be0297f21} {Suffrage:Voter ID:server-cec4b35b-eb12-dd42-1a92-234f116c6628 Address:cec4b35b-eb12-dd42-1a92-234f116c6628} {Suffrage:Voter ID:server-7989b60e-58be-98ad-8b04-7408ca66a815 Address:7989b60e-58be-98ad-8b04-7408ca66a815}]"
2026-02-28T09:02:36.817Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:36.817Z [INFO]  server-7c94535e-e214-8d0c-0d16-838be0297f21: entering follower state: follower="Node at 7c94535e-e214-8d0c-0d16-838be0297f21 [Follower]" leader-address= leader-id=
2026-02-28T09:02:36.817Z [INFO]  server-cec4b35b-eb12-dd42-1a92-234f116c6628: entering follower state: follower="Node at cec4b35b-eb12-dd42-1a92-234f116c6628 [Follower]" leader-address= leader-id=
2026-02-28T09:02:36.817Z [INFO]  server-7989b60e-58be-98ad-8b04-7408ca66a815: entering follower state: follower="Node at 7989b60e-58be-98ad-8b04-7408ca66a815 [Follower]" leader-address= leader-id=
2026-02-28T09:02:37.422Z [WARN]  server-7989b60e-58be-98ad-8b04-7408ca66a815: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:37.422Z [INFO]  server-7989b60e-58be-98ad-8b04-7408ca66a815: entering candidate state: node="Node at 7989b60e-58be-98ad-8b04-7408ca66a815 [Candidate]" term=2
2026-02-28T09:02:37.422Z [INFO]  server-7989b60e-58be-98ad-8b04-7408ca66a815: election won: term=2 tally=2
2026-02-28T09:02:37.422Z [INFO]  server-7989b60e-58be-98ad-8b04-7408ca66a815: entering leader state: leader="Node at 7989b60e-58be-98ad-8b04-7408ca66a815 [Leader]"
2026-02-28T09:02:37.422Z [INFO]  server-7989b60e-58be-98ad-8b04-7408ca66a815: added peer, starting replication: peer=server-7c94535e-e214-8d0c-0d16-838be0297f21
2026-02-28T09:02:37.422Z [INFO]  server-7989b60e-58be-98ad-8b04-7408ca66a815: added peer, starting replication: peer=server-cec4b35b-eb12-dd42-1a92-234f116c6628
2026-02-28T09:02:37.422Z [INFO]  server-7989b60e-58be-98ad-8b04-7408ca66a815: pipelining replication: peer="{Voter server-cec4b35b-eb12-dd42-1a92-234f116c6628 cec4b35b-eb12-dd42-1a92-234f116c6628}"
2026-02-28T09:02:37.422Z [INFO]  server-7989b60e-58be-98ad-8b04-7408ca66a815: pipelining replication: peer="{Voter server-7c94535e-e214-8d0c-0d16-838be0297f21 7c94535e-e214-8d0c-0d16-838be0297f21}"
2026-02-28T09:02:38.427Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:37.422651542 +0000 UTC m=+35.386448081 (1 nodes), 605.3534ms from start of poll, 605.764ms from cluster start. Timeout at 2026-02-28 09:02:38.427656942 +0000 UTC m=+36.391453481, 1.0050054s after stability
2026-02-28T09:02:38.428Z [INFO]  creating new snapshot: path=/tmp/raft3692453607/snapshots/2-14-1772269358428.tmp
2026-02-28T09:02:38.428Z [INFO]  server-7989b60e-58be-98ad-8b04-7408ca66a815: copied to local snapshot: bytes=71
2026-02-28T09:02:38.428Z [INFO]  server-7989b60e-58be-98ad-8b04-7408ca66a815: snapshot restore progress: id=2-14-1772269358428 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:38.428Z [INFO]  server-7989b60e-58be-98ad-8b04-7408ca66a815: restored user snapshot: index=14
2026-02-28T09:02:38.428Z [ERROR] server-7989b60e-58be-98ad-8b04-7408ca66a815: failed to get log: index=3 error="log not found"
2026-02-28T09:02:38.429Z [INFO]  server-7989b60e-58be-98ad-8b04-7408ca66a815: aborting pipeline replication: peer="{Voter server-cec4b35b-eb12-dd42-1a92-234f116c6628 cec4b35b-eb12-dd42-1a92-234f116c6628}"
2026-02-28T09:02:38.429Z [ERROR] server-7989b60e-58be-98ad-8b04-7408ca66a815: failed to get log: index=3 error="log not found"
2026-02-28T09:02:38.429Z [INFO]  server-7989b60e-58be-98ad-8b04-7408ca66a815: aborting pipeline replication: peer="{Voter server-7c94535e-e214-8d0c-0d16-838be0297f21 7c94535e-e214-8d0c-0d16-838be0297f21}"
2026-02-28T09:02:38.435Z [ERROR] server-7989b60e-58be-98ad-8b04-7408ca66a815: failed to get log: index=3 error="log not found"
2026-02-28T09:02:38.435Z [ERROR] server-7989b60e-58be-98ad-8b04-7408ca66a815: failed to get log: index=3 error="log not found"
2026-02-28T09:02:38.436Z [INFO]  creating new snapshot: path=/tmp/raft3266756632/snapshots/2-14-1772269358436.tmp
2026-02-28T09:02:38.436Z [INFO]  creating new snapshot: path=/tmp/raft543109165/snapshots/2-14-1772269358436.tmp
2026-02-28T09:02:38.436Z [INFO]  server-cec4b35b-eb12-dd42-1a92-234f116c6628: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:38.437Z [INFO]  server-7c94535e-e214-8d0c-0d16-838be0297f21: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:38.437Z [INFO]  server-7c94535e-e214-8d0c-0d16-838be0297f21: copied to local snapshot: bytes=71
2026-02-28T09:02:38.437Z [INFO]  server-cec4b35b-eb12-dd42-1a92-234f116c6628: copied to local snapshot: bytes=71
2026-02-28T09:02:38.437Z [INFO]  server-7c94535e-e214-8d0c-0d16-838be0297f21: snapshot restore progress: id=2-14-1772269358436 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:38.438Z [INFO]  server-7c94535e-e214-8d0c-0d16-838be0297f21: Installed remote snapshot
2026-02-28T09:02:38.438Z [INFO]  server-7989b60e-58be-98ad-8b04-7408ca66a815: pipelining replication: peer="{Voter server-7c94535e-e214-8d0c-0d16-838be0297f21 7c94535e-e214-8d0c-0d16-838be0297f21}"
2026-02-28T09:02:38.438Z [INFO]  server-cec4b35b-eb12-dd42-1a92-234f116c6628: snapshot restore progress: id=2-14-1772269358436 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:38.438Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:38.438Z [INFO]  server-cec4b35b-eb12-dd42-1a92-234f116c6628: Installed remote snapshot
2026-02-28T09:02:38.438Z [INFO]  server-7989b60e-58be-98ad-8b04-7408ca66a815: pipelining replication: peer="{Voter server-cec4b35b-eb12-dd42-1a92-234f116c6628 cec4b35b-eb12-dd42-1a92-234f116c6628}"
2026-02-28T09:02:39.444Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:38.438821942 +0000 UTC m=+36.402618481 (2 nodes), 2.08µs from start of poll, 1.6219344s from cluster start. Timeout at 2026-02-28 09:02:39.443834942 +0000 UTC m=+37.407631481, 1.005013s after stability
2026-02-28T09:02:39.457Z [INFO]  server-70091b02-88cf-fd68-bb40-53d7f682f2a8: aborting pipeline replication: peer="{Voter server-7d4d76ec-6a29-3e72-2754-17ec49570e3f 7d4d76ec-6a29-3e72-2754-17ec49570e3f}"
2026-02-28T09:02:39.457Z [INFO]  server-70091b02-88cf-fd68-bb40-53d7f682f2a8: aborting pipeline replication: peer="{Voter server-afdb1e71-b266-12d4-153f-1c2df9037cf0 afdb1e71-b266-12d4-153f-1c2df9037cf0}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true
2026-02-28T09:02:39.458Z [INFO]  server-5c373227-142f-d261-43e4-cb73159882d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c373227-142f-d261-43e4-cb73159882d0 Address:5c373227-142f-d261-43e4-cb73159882d0} {Suffrage:Voter ID:server-915e770e-ad28-8707-945f-c827317db80e Address:915e770e-ad28-8707-945f-c827317db80e} {Suffrage:Voter ID:server-151ffb7f-ffdd-f8e7-1501-212f13dd634b Address:151ffb7f-ffdd-f8e7-1501-212f13dd634b}]"
2026-02-28T09:02:39.458Z [INFO]  server-5c373227-142f-d261-43e4-cb73159882d0: entering follower state: follower="Node at 5c373227-142f-d261-43e4-cb73159882d0 [Follower]" leader-address= leader-id=
2026-02-28T09:02:39.458Z [INFO]  server-915e770e-ad28-8707-945f-c827317db80e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c373227-142f-d261-43e4-cb73159882d0 Address:5c373227-142f-d261-43e4-cb73159882d0} {Suffrage:Voter ID:server-915e770e-ad28-8707-945f-c827317db80e Address:915e770e-ad28-8707-945f-c827317db80e} {Suffrage:Voter ID:server-151ffb7f-ffdd-f8e7-1501-212f13dd634b Address:151ffb7f-ffdd-f8e7-1501-212f13dd634b}]"
2026-02-28T09:02:39.458Z [INFO]  server-915e770e-ad28-8707-945f-c827317db80e: entering follower state: follower="Node at 915e770e-ad28-8707-945f-c827317db80e [Follower]" leader-address= leader-id=
2026-02-28T09:02:39.458Z [INFO]  server-151ffb7f-ffdd-f8e7-1501-212f13dd634b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c373227-142f-d261-43e4-cb73159882d0 Address:5c373227-142f-d261-43e4-cb73159882d0} {Suffrage:Voter ID:server-915e770e-ad28-8707-945f-c827317db80e Address:915e770e-ad28-8707-945f-c827317db80e} {Suffrage:Voter ID:server-151ffb7f-ffdd-f8e7-1501-212f13dd634b Address:151ffb7f-ffdd-f8e7-1501-212f13dd634b}]"
2026-02-28T09:02:39.459Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:39.459Z [INFO]  server-151ffb7f-ffdd-f8e7-1501-212f13dd634b: entering follower state: follower="Node at 151ffb7f-ffdd-f8e7-1501-212f13dd634b [Follower]" leader-address= leader-id=
2026-02-28T09:02:40.064Z [WARN]  server-5c373227-142f-d261-43e4-cb73159882d0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:40.064Z [INFO]  server-5c373227-142f-d261-43e4-cb73159882d0: entering candidate state: node="Node at 5c373227-142f-d261-43e4-cb73159882d0 [Candidate]" term=2
2026-02-28T09:02:40.065Z [INFO]  server-5c373227-142f-d261-43e4-cb73159882d0: election won: term=2 tally=2
2026-02-28T09:02:40.065Z [INFO]  server-5c373227-142f-d261-43e4-cb73159882d0: entering leader state: leader="Node at 5c373227-142f-d261-43e4-cb73159882d0 [Leader]"
2026-02-28T09:02:40.066Z [INFO]  server-5c373227-142f-d261-43e4-cb73159882d0: added peer, starting replication: peer=server-915e770e-ad28-8707-945f-c827317db80e
2026-02-28T09:02:40.066Z [INFO]  server-5c373227-142f-d261-43e4-cb73159882d0: added peer, starting replication: peer=server-151ffb7f-ffdd-f8e7-1501-212f13dd634b
2026-02-28T09:02:40.066Z [INFO]  server-5c373227-142f-d261-43e4-cb73159882d0: pipelining replication: peer="{Voter server-915e770e-ad28-8707-945f-c827317db80e 915e770e-ad28-8707-945f-c827317db80e}"
2026-02-28T09:02:40.066Z [INFO]  server-5c373227-142f-d261-43e4-cb73159882d0: pipelining replication: peer="{Voter server-151ffb7f-ffdd-f8e7-1501-212f13dd634b 151ffb7f-ffdd-f8e7-1501-212f13dd634b}"
2026-02-28T09:02:41.072Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:40.066663342 +0000 UTC m=+38.030459881 (1 nodes), 607.64424ms from start of poll, 608.22156ms from cluster start. Timeout at 2026-02-28 09:02:41.071666142 +0000 UTC m=+39.035462641, 1.00500276s after stability
2026-02-28T09:02:41.073Z [INFO]  server-5c373227-142f-d261-43e4-cb73159882d0: starting snapshot up to: index=12
2026-02-28T09:02:41.073Z [INFO]  creating new snapshot: path=/tmp/raft3520209681/snapshots/2-12-1772269361073.tmp
2026-02-28T09:02:41.075Z [INFO]  server-5c373227-142f-d261-43e4-cb73159882d0: snapshot complete up to: index=12
2026-02-28T09:02:41.076Z [INFO]  server-6728c77f-6045-4930-a7e5-db11396d4135: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6728c77f-6045-4930-a7e5-db11396d4135 Address:6728c77f-6045-4930-a7e5-db11396d4135} {Suffrage:Voter ID:server-8d836420-7775-0d61-5ff9-02acee984280 Address:8d836420-7775-0d61-5ff9-02acee984280} {Suffrage:Voter ID:server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139 Address:d33a1ddd-7220-2ae1-efe3-0e94d00cf139}]"
2026-02-28T09:02:41.076Z [INFO]  server-8d836420-7775-0d61-5ff9-02acee984280: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6728c77f-6045-4930-a7e5-db11396d4135 Address:6728c77f-6045-4930-a7e5-db11396d4135} {Suffrage:Voter ID:server-8d836420-7775-0d61-5ff9-02acee984280 Address:8d836420-7775-0d61-5ff9-02acee984280} {Suffrage:Voter ID:server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139 Address:d33a1ddd-7220-2ae1-efe3-0e94d00cf139}]"
2026-02-28T09:02:41.076Z [INFO]  server-6728c77f-6045-4930-a7e5-db11396d4135: entering follower state: follower="Node at 6728c77f-6045-4930-a7e5-db11396d4135 [Follower]" leader-address= leader-id=
2026-02-28T09:02:41.076Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6728c77f-6045-4930-a7e5-db11396d4135 Address:6728c77f-6045-4930-a7e5-db11396d4135} {Suffrage:Voter ID:server-8d836420-7775-0d61-5ff9-02acee984280 Address:8d836420-7775-0d61-5ff9-02acee984280} {Suffrage:Voter ID:server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139 Address:d33a1ddd-7220-2ae1-efe3-0e94d00cf139}]"
2026-02-28T09:02:41.076Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:41.076Z [INFO]  server-8d836420-7775-0d61-5ff9-02acee984280: entering follower state: follower="Node at 8d836420-7775-0d61-5ff9-02acee984280 [Follower]" leader-address= leader-id=
2026-02-28T09:02:41.076Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: entering follower state: follower="Node at d33a1ddd-7220-2ae1-efe3-0e94d00cf139 [Follower]" leader-address= leader-id=
2026-02-28T09:02:41.717Z [WARN]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:41.717Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: entering candidate state: node="Node at d33a1ddd-7220-2ae1-efe3-0e94d00cf139 [Candidate]" term=2
2026-02-28T09:02:41.717Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: election won: term=2 tally=2
2026-02-28T09:02:41.717Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: entering leader state: leader="Node at d33a1ddd-7220-2ae1-efe3-0e94d00cf139 [Leader]"
2026-02-28T09:02:41.717Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: added peer, starting replication: peer=server-6728c77f-6045-4930-a7e5-db11396d4135
2026-02-28T09:02:41.717Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: added peer, starting replication: peer=server-8d836420-7775-0d61-5ff9-02acee984280
2026-02-28T09:02:41.717Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: pipelining replication: peer="{Voter server-8d836420-7775-0d61-5ff9-02acee984280 8d836420-7775-0d61-5ff9-02acee984280}"
2026-02-28T09:02:41.718Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: pipelining replication: peer="{Voter server-6728c77f-6045-4930-a7e5-db11396d4135 6728c77f-6045-4930-a7e5-db11396d4135}"
2026-02-28T09:02:42.723Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:41.717864502 +0000 UTC m=+39.681661041 (1 nodes), 641.3396ms from start of poll, 641.78848ms from cluster start. Timeout at 2026-02-28 09:02:42.722879542 +0000 UTC m=+40.686676081, 1.00501504s after stability
2026-02-28T09:02:42.723Z [INFO]  creating new snapshot: path=/tmp/raft1770059816/snapshots/2-14-1772269362723.tmp
2026-02-28T09:02:42.724Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: copied to local snapshot: bytes=71
2026-02-28T09:02:42.724Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: snapshot restore progress: id=2-14-1772269362723 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:42.724Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: removing all old logs from log store
2026-02-28T09:02:42.724Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: compacting logs: from=1 to=2
2026-02-28T09:02:42.724Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: restored user snapshot: index=14
2026-02-28T09:02:42.724Z [ERROR] server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: failed to get log: index=2 error="log not found"
2026-02-28T09:02:42.724Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: aborting pipeline replication: peer="{Voter server-8d836420-7775-0d61-5ff9-02acee984280 8d836420-7775-0d61-5ff9-02acee984280}"
2026-02-28T09:02:42.724Z [ERROR] server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: failed to get log: index=2 error="log not found"
2026-02-28T09:02:42.724Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: aborting pipeline replication: peer="{Voter server-6728c77f-6045-4930-a7e5-db11396d4135 6728c77f-6045-4930-a7e5-db11396d4135}"
2026-02-28T09:02:42.731Z [ERROR] server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: failed to get log: index=2 error="log not found"
2026-02-28T09:02:42.732Z [INFO]  creating new snapshot: path=/tmp/raft876049943/snapshots/2-14-1772269362732.tmp
2026-02-28T09:02:42.732Z [INFO]  server-8d836420-7775-0d61-5ff9-02acee984280: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:42.732Z [INFO]  server-8d836420-7775-0d61-5ff9-02acee984280: copied to local snapshot: bytes=71
2026-02-28T09:02:42.733Z [INFO]  server-8d836420-7775-0d61-5ff9-02acee984280: snapshot restore progress: id=2-14-1772269362732 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:42.733Z [INFO]  server-8d836420-7775-0d61-5ff9-02acee984280: removing all old logs from log store
2026-02-28T09:02:42.733Z [INFO]  server-8d836420-7775-0d61-5ff9-02acee984280: compacting logs: from=1 to=2
2026-02-28T09:02:42.733Z [INFO]  server-8d836420-7775-0d61-5ff9-02acee984280: Installed remote snapshot
2026-02-28T09:02:42.733Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: pipelining replication: peer="{Voter server-8d836420-7775-0d61-5ff9-02acee984280 8d836420-7775-0d61-5ff9-02acee984280}"
2026-02-28T09:02:42.733Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:42.734Z [ERROR] server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: failed to get log: index=2 error="log not found"
2026-02-28T09:02:42.735Z [INFO]  creating new snapshot: path=/tmp/raft1507572676/snapshots/2-14-1772269362735.tmp
2026-02-28T09:02:42.735Z [INFO]  server-6728c77f-6045-4930-a7e5-db11396d4135: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:42.735Z [INFO]  server-6728c77f-6045-4930-a7e5-db11396d4135: copied to local snapshot: bytes=71
2026-02-28T09:02:42.736Z [INFO]  server-6728c77f-6045-4930-a7e5-db11396d4135: snapshot restore progress: id=2-14-1772269362735 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:42.736Z [INFO]  server-6728c77f-6045-4930-a7e5-db11396d4135: removing all old logs from log store
2026-02-28T09:02:42.736Z [INFO]  server-6728c77f-6045-4930-a7e5-db11396d4135: compacting logs: from=1 to=2
2026-02-28T09:02:42.736Z [INFO]  server-6728c77f-6045-4930-a7e5-db11396d4135: Installed remote snapshot
2026-02-28T09:02:42.736Z [INFO]  server-d33a1ddd-7220-2ae1-efe3-0e94d00cf139: pipelining replication: peer="{Voter server-6728c77f-6045-4930-a7e5-db11396d4135 6728c77f-6045-4930-a7e5-db11396d4135}"
2026-02-28T09:02:43.738Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:42.733495022 +0000 UTC m=+40.697291561 (2 nodes), 1.12µs from start of poll, 1.657419s from cluster start. Timeout at 2026-02-28 09:02:43.738514182 +0000 UTC m=+41.702310721, 1.00501916s after stability
2026-02-28T09:02:43.752Z [INFO]  server-5c373227-142f-d261-43e4-cb73159882d0: aborting pipeline replication: peer="{Voter server-151ffb7f-ffdd-f8e7-1501-212f13dd634b 151ffb7f-ffdd-f8e7-1501-212f13dd634b}"
2026-02-28T09:02:43.752Z [INFO]  server-5c373227-142f-d261-43e4-cb73159882d0: aborting pipeline replication: peer="{Voter server-915e770e-ad28-8707-945f-c827317db80e 915e770e-ad28-8707-945f-c827317db80e}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false
2026-02-28T09:02:43.753Z [INFO]  server-b491f50d-93fc-7bc7-7c35-46e1c26ef0a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b491f50d-93fc-7bc7-7c35-46e1c26ef0a7 Address:b491f50d-93fc-7bc7-7c35-46e1c26ef0a7} {Suffrage:Voter ID:server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f Address:eaae1530-cbd7-13fc-5a14-6ac8c06f193f} {Suffrage:Voter ID:server-8dda695a-5bd2-81d1-76e6-2ea803d50123 Address:8dda695a-5bd2-81d1-76e6-2ea803d50123}]"
2026-02-28T09:02:43.753Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b491f50d-93fc-7bc7-7c35-46e1c26ef0a7 Address:b491f50d-93fc-7bc7-7c35-46e1c26ef0a7} {Suffrage:Voter ID:server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f Address:eaae1530-cbd7-13fc-5a14-6ac8c06f193f} {Suffrage:Voter ID:server-8dda695a-5bd2-81d1-76e6-2ea803d50123 Address:8dda695a-5bd2-81d1-76e6-2ea803d50123}]"
2026-02-28T09:02:43.753Z [INFO]  server-b491f50d-93fc-7bc7-7c35-46e1c26ef0a7: entering follower state: follower="Node at b491f50d-93fc-7bc7-7c35-46e1c26ef0a7 [Follower]" leader-address= leader-id=
2026-02-28T09:02:43.753Z [INFO]  server-8dda695a-5bd2-81d1-76e6-2ea803d50123: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b491f50d-93fc-7bc7-7c35-46e1c26ef0a7 Address:b491f50d-93fc-7bc7-7c35-46e1c26ef0a7} {Suffrage:Voter ID:server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f Address:eaae1530-cbd7-13fc-5a14-6ac8c06f193f} {Suffrage:Voter ID:server-8dda695a-5bd2-81d1-76e6-2ea803d50123 Address:8dda695a-5bd2-81d1-76e6-2ea803d50123}]"
2026-02-28T09:02:43.753Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: entering follower state: follower="Node at eaae1530-cbd7-13fc-5a14-6ac8c06f193f [Follower]" leader-address= leader-id=
2026-02-28T09:02:43.754Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:43.754Z [INFO]  server-8dda695a-5bd2-81d1-76e6-2ea803d50123: entering follower state: follower="Node at 8dda695a-5bd2-81d1-76e6-2ea803d50123 [Follower]" leader-address= leader-id=
2026-02-28T09:02:44.319Z [WARN]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:44.319Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: entering candidate state: node="Node at eaae1530-cbd7-13fc-5a14-6ac8c06f193f [Candidate]" term=2
2026-02-28T09:02:44.319Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: election won: term=2 tally=2
2026-02-28T09:02:44.319Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: entering leader state: leader="Node at eaae1530-cbd7-13fc-5a14-6ac8c06f193f [Leader]"
2026-02-28T09:02:44.319Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: added peer, starting replication: peer=server-b491f50d-93fc-7bc7-7c35-46e1c26ef0a7
2026-02-28T09:02:44.319Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: added peer, starting replication: peer=server-8dda695a-5bd2-81d1-76e6-2ea803d50123
2026-02-28T09:02:44.319Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: pipelining replication: peer="{Voter server-8dda695a-5bd2-81d1-76e6-2ea803d50123 8dda695a-5bd2-81d1-76e6-2ea803d50123}"
2026-02-28T09:02:44.320Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: pipelining replication: peer="{Voter server-b491f50d-93fc-7bc7-7c35-46e1c26ef0a7 b491f50d-93fc-7bc7-7c35-46e1c26ef0a7}"
2026-02-28T09:02:45.324Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:44.319790462 +0000 UTC m=+42.283587001 (1 nodes), 565.75712ms from start of poll, 566.201ms from cluster start. Timeout at 2026-02-28 09:02:45.324794502 +0000 UTC m=+43.288591041, 1.00500404s after stability
2026-02-28T09:02:45.325Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: starting snapshot up to: index=12
2026-02-28T09:02:45.325Z [INFO]  creating new snapshot: path=/tmp/raft3881512045/snapshots/2-12-1772269365325.tmp
2026-02-28T09:02:45.326Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: snapshot complete up to: index=12
2026-02-28T09:02:45.327Z [INFO]  creating new snapshot: path=/tmp/raft3881512045/snapshots/2-23-1772269365327.tmp
2026-02-28T09:02:45.327Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: copied to local snapshot: bytes=71
2026-02-28T09:02:45.328Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: snapshot restore progress: id=2-23-1772269365327 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:45.328Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: restored user snapshot: index=23
2026-02-28T09:02:45.328Z [ERROR] server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: failed to get log: index=23 error="log not found"
2026-02-28T09:02:45.328Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: aborting pipeline replication: peer="{Voter server-b491f50d-93fc-7bc7-7c35-46e1c26ef0a7 b491f50d-93fc-7bc7-7c35-46e1c26ef0a7}"
2026-02-28T09:02:45.328Z [ERROR] server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: failed to get log: index=23 error="log not found"
2026-02-28T09:02:45.328Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: aborting pipeline replication: peer="{Voter server-8dda695a-5bd2-81d1-76e6-2ea803d50123 8dda695a-5bd2-81d1-76e6-2ea803d50123}"
2026-02-28T09:02:45.334Z [ERROR] server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: failed to get log: index=23 error="log not found"
2026-02-28T09:02:45.334Z [INFO]  creating new snapshot: path=/tmp/raft180426700/snapshots/2-23-1772269365334.tmp
2026-02-28T09:02:45.335Z [INFO]  server-8dda695a-5bd2-81d1-76e6-2ea803d50123: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:45.335Z [INFO]  server-8dda695a-5bd2-81d1-76e6-2ea803d50123: copied to local snapshot: bytes=71
2026-02-28T09:02:45.335Z [INFO]  server-8dda695a-5bd2-81d1-76e6-2ea803d50123: snapshot restore progress: id=2-23-1772269365334 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:45.335Z [INFO]  server-8dda695a-5bd2-81d1-76e6-2ea803d50123: Installed remote snapshot
2026-02-28T09:02:45.335Z [WARN]  server-8dda695a-5bd2-81d1-76e6-2ea803d50123: skipping application of old log: index=22
2026-02-28T09:02:45.335Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: pipelining replication: peer="{Voter server-8dda695a-5bd2-81d1-76e6-2ea803d50123 8dda695a-5bd2-81d1-76e6-2ea803d50123}"
2026-02-28T09:02:45.335Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:45.351Z [ERROR] server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: failed to get log: index=23 error="log not found"
2026-02-28T09:02:45.351Z [INFO]  creating new snapshot: path=/tmp/raft3499334889/snapshots/2-23-1772269365351.tmp
2026-02-28T09:02:45.352Z [INFO]  server-b491f50d-93fc-7bc7-7c35-46e1c26ef0a7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:45.352Z [INFO]  server-b491f50d-93fc-7bc7-7c35-46e1c26ef0a7: copied to local snapshot: bytes=71
2026-02-28T09:02:45.352Z [INFO]  server-b491f50d-93fc-7bc7-7c35-46e1c26ef0a7: snapshot restore progress: id=2-23-1772269365351 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:45.352Z [INFO]  server-b491f50d-93fc-7bc7-7c35-46e1c26ef0a7: Installed remote snapshot
2026-02-28T09:02:45.353Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: pipelining replication: peer="{Voter server-b491f50d-93fc-7bc7-7c35-46e1c26ef0a7 b491f50d-93fc-7bc7-7c35-46e1c26ef0a7}"
2026-02-28T09:02:46.342Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:45.335910942 +0000 UTC m=+43.299707481 (2 nodes), 920ns from start of poll, 1.58232148s from cluster start. Timeout at 2026-02-28 09:02:46.340935182 +0000 UTC m=+44.304731721, 1.00502424s after stability
2026-02-28T09:02:46.358Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: aborting pipeline replication: peer="{Voter server-8dda695a-5bd2-81d1-76e6-2ea803d50123 8dda695a-5bd2-81d1-76e6-2ea803d50123}"
2026-02-28T09:02:46.358Z [INFO]  server-eaae1530-cbd7-13fc-5a14-6ac8c06f193f: aborting pipeline replication: peer="{Voter server-b491f50d-93fc-7bc7-7c35-46e1c26ef0a7 b491f50d-93fc-7bc7-7c35-46e1c26ef0a7}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false
2026-02-28T09:02:46.360Z [INFO]  server-a3e1eaed-98be-37a4-5663-457d61a9e300: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3e1eaed-98be-37a4-5663-457d61a9e300 Address:a3e1eaed-98be-37a4-5663-457d61a9e300} {Suffrage:Voter ID:server-9d988610-d3fa-16d2-bacd-8af643e97f2f Address:9d988610-d3fa-16d2-bacd-8af643e97f2f} {Suffrage:Voter ID:server-5a6e8fb8-d724-9277-af62-35df0a8590fc Address:5a6e8fb8-d724-9277-af62-35df0a8590fc}]"
2026-02-28T09:02:46.360Z [INFO]  server-9d988610-d3fa-16d2-bacd-8af643e97f2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3e1eaed-98be-37a4-5663-457d61a9e300 Address:a3e1eaed-98be-37a4-5663-457d61a9e300} {Suffrage:Voter ID:server-9d988610-d3fa-16d2-bacd-8af643e97f2f Address:9d988610-d3fa-16d2-bacd-8af643e97f2f} {Suffrage:Voter ID:server-5a6e8fb8-d724-9277-af62-35df0a8590fc Address:5a6e8fb8-d724-9277-af62-35df0a8590fc}]"
2026-02-28T09:02:46.360Z [INFO]  server-a3e1eaed-98be-37a4-5663-457d61a9e300: entering follower state: follower="Node at a3e1eaed-98be-37a4-5663-457d61a9e300 [Follower]" leader-address= leader-id=
2026-02-28T09:02:46.360Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3e1eaed-98be-37a4-5663-457d61a9e300 Address:a3e1eaed-98be-37a4-5663-457d61a9e300} {Suffrage:Voter ID:server-9d988610-d3fa-16d2-bacd-8af643e97f2f Address:9d988610-d3fa-16d2-bacd-8af643e97f2f} {Suffrage:Voter ID:server-5a6e8fb8-d724-9277-af62-35df0a8590fc Address:5a6e8fb8-d724-9277-af62-35df0a8590fc}]"
2026-02-28T09:02:46.360Z [INFO]  server-9d988610-d3fa-16d2-bacd-8af643e97f2f: entering follower state: follower="Node at 9d988610-d3fa-16d2-bacd-8af643e97f2f [Follower]" leader-address= leader-id=
2026-02-28T09:02:46.360Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:46.360Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: entering follower state: follower="Node at 5a6e8fb8-d724-9277-af62-35df0a8590fc [Follower]" leader-address= leader-id=
2026-02-28T09:02:46.877Z [WARN]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:46.877Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: entering candidate state: node="Node at 5a6e8fb8-d724-9277-af62-35df0a8590fc [Candidate]" term=2
2026-02-28T09:02:46.877Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: election won: term=2 tally=2
2026-02-28T09:02:46.877Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: entering leader state: leader="Node at 5a6e8fb8-d724-9277-af62-35df0a8590fc [Leader]"
2026-02-28T09:02:46.877Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: added peer, starting replication: peer=server-a3e1eaed-98be-37a4-5663-457d61a9e300
2026-02-28T09:02:46.877Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: added peer, starting replication: peer=server-9d988610-d3fa-16d2-bacd-8af643e97f2f
2026-02-28T09:02:46.878Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: pipelining replication: peer="{Voter server-9d988610-d3fa-16d2-bacd-8af643e97f2f 9d988610-d3fa-16d2-bacd-8af643e97f2f}"
2026-02-28T09:02:46.878Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: pipelining replication: peer="{Voter server-a3e1eaed-98be-37a4-5663-457d61a9e300 a3e1eaed-98be-37a4-5663-457d61a9e300}"
2026-02-28T09:02:47.883Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:46.878000302 +0000 UTC m=+44.841796841 (1 nodes), 517.6124ms from start of poll, 518.15848ms from cluster start. Timeout at 2026-02-28 09:02:47.883007182 +0000 UTC m=+45.846803721, 1.00500688s after stability
2026-02-28T09:02:47.884Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: starting snapshot up to: index=12
2026-02-28T09:02:47.884Z [INFO]  creating new snapshot: path=/tmp/raft1746327772/snapshots/2-12-1772269367884.tmp
2026-02-28T09:02:47.885Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: snapshot complete up to: index=12
2026-02-28T09:02:47.894Z [INFO]  creating new snapshot: path=/tmp/raft1746327772/snapshots/2-23-1772269367894.tmp
2026-02-28T09:02:47.895Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: copied to local snapshot: bytes=71
2026-02-28T09:02:47.895Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: snapshot restore progress: id=2-23-1772269367894 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:47.895Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: removing all old logs from log store
2026-02-28T09:02:47.895Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: compacting logs: from=1 to=22
2026-02-28T09:02:47.895Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: restored user snapshot: index=23
2026-02-28T09:02:47.895Z [ERROR] server-5a6e8fb8-d724-9277-af62-35df0a8590fc: failed to get log: index=22 error="log not found"
2026-02-28T09:02:47.895Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: aborting pipeline replication: peer="{Voter server-9d988610-d3fa-16d2-bacd-8af643e97f2f 9d988610-d3fa-16d2-bacd-8af643e97f2f}"
2026-02-28T09:02:47.896Z [ERROR] server-5a6e8fb8-d724-9277-af62-35df0a8590fc: failed to get log: index=22 error="log not found"
2026-02-28T09:02:47.896Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: aborting pipeline replication: peer="{Voter server-a3e1eaed-98be-37a4-5663-457d61a9e300 a3e1eaed-98be-37a4-5663-457d61a9e300}"
2026-02-28T09:02:47.902Z [ERROR] server-5a6e8fb8-d724-9277-af62-35df0a8590fc: failed to get log: index=22 error="log not found"
2026-02-28T09:02:47.902Z [ERROR] server-5a6e8fb8-d724-9277-af62-35df0a8590fc: failed to get log: index=22 error="log not found"
2026-02-28T09:02:47.903Z [INFO]  creating new snapshot: path=/tmp/raft1367147757/snapshots/2-23-1772269367903.tmp
2026-02-28T09:02:47.903Z [INFO]  creating new snapshot: path=/tmp/raft4178996569/snapshots/2-23-1772269367903.tmp
2026-02-28T09:02:47.903Z [INFO]  server-9d988610-d3fa-16d2-bacd-8af643e97f2f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:47.903Z [INFO]  server-a3e1eaed-98be-37a4-5663-457d61a9e300: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:47.903Z [INFO]  server-9d988610-d3fa-16d2-bacd-8af643e97f2f: copied to local snapshot: bytes=71
2026-02-28T09:02:47.903Z [INFO]  server-a3e1eaed-98be-37a4-5663-457d61a9e300: copied to local snapshot: bytes=71
2026-02-28T09:02:47.904Z [INFO]  server-9d988610-d3fa-16d2-bacd-8af643e97f2f: snapshot restore progress: id=2-23-1772269367903 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:47.904Z [INFO]  server-a3e1eaed-98be-37a4-5663-457d61a9e300: snapshot restore progress: id=2-23-1772269367903 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:47.904Z [INFO]  server-9d988610-d3fa-16d2-bacd-8af643e97f2f: removing all old logs from log store
2026-02-28T09:02:47.904Z [INFO]  server-9d988610-d3fa-16d2-bacd-8af643e97f2f: compacting logs: from=1 to=22
2026-02-28T09:02:47.904Z [INFO]  server-9d988610-d3fa-16d2-bacd-8af643e97f2f: Installed remote snapshot
2026-02-28T09:02:47.904Z [INFO]  server-a3e1eaed-98be-37a4-5663-457d61a9e300: removing all old logs from log store
2026-02-28T09:02:47.904Z [INFO]  server-a3e1eaed-98be-37a4-5663-457d61a9e300: compacting logs: from=1 to=22
2026-02-28T09:02:47.904Z [INFO]  server-a3e1eaed-98be-37a4-5663-457d61a9e300: Installed remote snapshot
2026-02-28T09:02:47.904Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: pipelining replication: peer="{Voter server-9d988610-d3fa-16d2-bacd-8af643e97f2f 9d988610-d3fa-16d2-bacd-8af643e97f2f}"
2026-02-28T09:02:47.904Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:47.904Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: pipelining replication: peer="{Voter server-a3e1eaed-98be-37a4-5663-457d61a9e300 a3e1eaed-98be-37a4-5663-457d61a9e300}"
2026-02-28T09:02:48.910Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:47.904487662 +0000 UTC m=+45.868284201 (2 nodes), 1.04µs from start of poll, 1.54464584s from cluster start. Timeout at 2026-02-28 09:02:48.909493222 +0000 UTC m=+46.873289801, 1.0050056s after stability
2026-02-28T09:02:48.922Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: aborting pipeline replication: peer="{Voter server-a3e1eaed-98be-37a4-5663-457d61a9e300 a3e1eaed-98be-37a4-5663-457d61a9e300}"
2026-02-28T09:02:48.922Z [INFO]  server-5a6e8fb8-d724-9277-af62-35df0a8590fc: aborting pipeline replication: peer="{Voter server-9d988610-d3fa-16d2-bacd-8af643e97f2f 9d988610-d3fa-16d2-bacd-8af643e97f2f}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true
2026-02-28T09:02:48.923Z [INFO]  server-f889c304-ae9d-f806-53a2-7b163be93de6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f889c304-ae9d-f806-53a2-7b163be93de6 Address:f889c304-ae9d-f806-53a2-7b163be93de6} {Suffrage:Voter ID:server-71d20388-93b7-2cdc-2173-8da77323ccbd Address:71d20388-93b7-2cdc-2173-8da77323ccbd} {Suffrage:Voter ID:server-3627b8fb-41d6-08cb-a7fd-daed3fb25855 Address:3627b8fb-41d6-08cb-a7fd-daed3fb25855}]"
2026-02-28T09:02:48.923Z [INFO]  server-71d20388-93b7-2cdc-2173-8da77323ccbd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f889c304-ae9d-f806-53a2-7b163be93de6 Address:f889c304-ae9d-f806-53a2-7b163be93de6} {Suffrage:Voter ID:server-71d20388-93b7-2cdc-2173-8da77323ccbd Address:71d20388-93b7-2cdc-2173-8da77323ccbd} {Suffrage:Voter ID:server-3627b8fb-41d6-08cb-a7fd-daed3fb25855 Address:3627b8fb-41d6-08cb-a7fd-daed3fb25855}]"
2026-02-28T09:02:48.923Z [INFO]  server-f889c304-ae9d-f806-53a2-7b163be93de6: entering follower state: follower="Node at f889c304-ae9d-f806-53a2-7b163be93de6 [Follower]" leader-address= leader-id=
2026-02-28T09:02:48.923Z [INFO]  server-3627b8fb-41d6-08cb-a7fd-daed3fb25855: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f889c304-ae9d-f806-53a2-7b163be93de6 Address:f889c304-ae9d-f806-53a2-7b163be93de6} {Suffrage:Voter ID:server-71d20388-93b7-2cdc-2173-8da77323ccbd Address:71d20388-93b7-2cdc-2173-8da77323ccbd} {Suffrage:Voter ID:server-3627b8fb-41d6-08cb-a7fd-daed3fb25855 Address:3627b8fb-41d6-08cb-a7fd-daed3fb25855}]"
2026-02-28T09:02:48.923Z [INFO]  server-71d20388-93b7-2cdc-2173-8da77323ccbd: entering follower state: follower="Node at 71d20388-93b7-2cdc-2173-8da77323ccbd [Follower]" leader-address= leader-id=
2026-02-28T09:02:48.923Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:48.923Z [INFO]  server-3627b8fb-41d6-08cb-a7fd-daed3fb25855: entering follower state: follower="Node at 3627b8fb-41d6-08cb-a7fd-daed3fb25855 [Follower]" leader-address= leader-id=
2026-02-28T09:02:49.503Z [WARN]  server-71d20388-93b7-2cdc-2173-8da77323ccbd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:49.503Z [INFO]  server-71d20388-93b7-2cdc-2173-8da77323ccbd: entering candidate state: node="Node at 71d20388-93b7-2cdc-2173-8da77323ccbd [Candidate]" term=2
2026-02-28T09:02:49.503Z [INFO]  server-71d20388-93b7-2cdc-2173-8da77323ccbd: election won: term=2 tally=2
2026-02-28T09:02:49.503Z [INFO]  server-71d20388-93b7-2cdc-2173-8da77323ccbd: entering leader state: leader="Node at 71d20388-93b7-2cdc-2173-8da77323ccbd [Leader]"
2026-02-28T09:02:49.503Z [INFO]  server-71d20388-93b7-2cdc-2173-8da77323ccbd: added peer, starting replication: peer=server-f889c304-ae9d-f806-53a2-7b163be93de6
2026-02-28T09:02:49.503Z [INFO]  server-71d20388-93b7-2cdc-2173-8da77323ccbd: added peer, starting replication: peer=server-3627b8fb-41d6-08cb-a7fd-daed3fb25855
2026-02-28T09:02:49.504Z [INFO]  server-71d20388-93b7-2cdc-2173-8da77323ccbd: pipelining replication: peer="{Voter server-3627b8fb-41d6-08cb-a7fd-daed3fb25855 3627b8fb-41d6-08cb-a7fd-daed3fb25855}"
2026-02-28T09:02:49.504Z [INFO]  server-71d20388-93b7-2cdc-2173-8da77323ccbd: pipelining replication: peer="{Voter server-f889c304-ae9d-f806-53a2-7b163be93de6 f889c304-ae9d-f806-53a2-7b163be93de6}"
2026-02-28T09:02:50.510Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:49.503952982 +0000 UTC m=+47.467749521 (1 nodes), 580.12832ms from start of poll, 580.56616ms from cluster start. Timeout at 2026-02-28 09:02:50.508954822 +0000 UTC m=+48.472751401, 1.00500188s after stability
2026-02-28T09:02:50.510Z [INFO]  server-71d20388-93b7-2cdc-2173-8da77323ccbd: starting snapshot up to: index=12
2026-02-28T09:02:50.510Z [INFO]  creating new snapshot: path=/tmp/raft3834584562/snapshots/2-12-1772269370510.tmp
2026-02-28T09:02:50.511Z [INFO]  server-71d20388-93b7-2cdc-2173-8da77323ccbd: snapshot complete up to: index=12
2026-02-28T09:02:50.512Z [INFO]  server-44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a Address:44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a} {Suffrage:Voter ID:server-077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2 Address:077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2} {Suffrage:Voter ID:server-7b4b2180-7172-1dba-80f9-c3e79f862496 Address:7b4b2180-7172-1dba-80f9-c3e79f862496}]"
2026-02-28T09:02:50.512Z [INFO]  server-077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a Address:44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a} {Suffrage:Voter ID:server-077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2 Address:077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2} {Suffrage:Voter ID:server-7b4b2180-7172-1dba-80f9-c3e79f862496 Address:7b4b2180-7172-1dba-80f9-c3e79f862496}]"
2026-02-28T09:02:50.512Z [INFO]  server-7b4b2180-7172-1dba-80f9-c3e79f862496: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a Address:44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a} {Suffrage:Voter ID:server-077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2 Address:077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2} {Suffrage:Voter ID:server-7b4b2180-7172-1dba-80f9-c3e79f862496 Address:7b4b2180-7172-1dba-80f9-c3e79f862496}]"
2026-02-28T09:02:50.512Z [INFO]  server-077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2: entering follower state: follower="Node at 077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2 [Follower]" leader-address= leader-id=
2026-02-28T09:02:50.512Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:50.512Z [INFO]  server-7b4b2180-7172-1dba-80f9-c3e79f862496: entering follower state: follower="Node at 7b4b2180-7172-1dba-80f9-c3e79f862496 [Follower]" leader-address= leader-id=
2026-02-28T09:02:50.512Z [INFO]  server-44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a: entering follower state: follower="Node at 44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a [Follower]" leader-address= leader-id=
2026-02-28T09:02:51.443Z [WARN]  server-7b4b2180-7172-1dba-80f9-c3e79f862496: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:51.443Z [INFO]  server-7b4b2180-7172-1dba-80f9-c3e79f862496: entering candidate state: node="Node at 7b4b2180-7172-1dba-80f9-c3e79f862496 [Candidate]" term=2
2026-02-28T09:02:51.443Z [INFO]  server-7b4b2180-7172-1dba-80f9-c3e79f862496: election won: term=2 tally=2
2026-02-28T09:02:51.443Z [INFO]  server-7b4b2180-7172-1dba-80f9-c3e79f862496: entering leader state: leader="Node at 7b4b2180-7172-1dba-80f9-c3e79f862496 [Leader]"
2026-02-28T09:02:51.443Z [INFO]  server-7b4b2180-7172-1dba-80f9-c3e79f862496: added peer, starting replication: peer=server-44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a
2026-02-28T09:02:51.443Z [INFO]  server-7b4b2180-7172-1dba-80f9-c3e79f862496: added peer, starting replication: peer=server-077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2
2026-02-28T09:02:51.444Z [INFO]  server-7b4b2180-7172-1dba-80f9-c3e79f862496: pipelining replication: peer="{Voter server-077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2 077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2}"
2026-02-28T09:02:51.444Z [INFO]  server-7b4b2180-7172-1dba-80f9-c3e79f862496: pipelining replication: peer="{Voter server-44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a 44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a}"
2026-02-28T09:02:52.449Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:51.443999262 +0000 UTC m=+49.407795801 (1 nodes), 931.279ms from start of poll, 931.7064ms from cluster start. Timeout at 2026-02-28 09:02:52.449001502 +0000 UTC m=+50.412798041, 1.00500224s after stability
2026-02-28T09:02:52.449Z [INFO]  creating new snapshot: path=/tmp/raft1222448044/snapshots/2-15-1772269372449.tmp
2026-02-28T09:02:52.450Z [INFO]  server-7b4b2180-7172-1dba-80f9-c3e79f862496: copied to local snapshot: bytes=71
2026-02-28T09:02:52.450Z [INFO]  server-7b4b2180-7172-1dba-80f9-c3e79f862496: snapshot restore progress: id=2-15-1772269372449 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:52.450Z [INFO]  server-7b4b2180-7172-1dba-80f9-c3e79f862496: restored user snapshot: index=15
2026-02-28T09:02:52.450Z [ERROR] server-7b4b2180-7172-1dba-80f9-c3e79f862496: failed to get log: index=3 error="log not found"
2026-02-28T09:02:52.450Z [INFO]  server-7b4b2180-7172-1dba-80f9-c3e79f862496: aborting pipeline replication: peer="{Voter server-44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a 44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a}"
2026-02-28T09:02:52.450Z [ERROR] server-7b4b2180-7172-1dba-80f9-c3e79f862496: failed to get log: index=3 error="log not found"
2026-02-28T09:02:52.450Z [INFO]  server-7b4b2180-7172-1dba-80f9-c3e79f862496: aborting pipeline replication: peer="{Voter server-077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2 077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2}"
2026-02-28T09:02:52.456Z [ERROR] server-7b4b2180-7172-1dba-80f9-c3e79f862496: failed to get log: index=3 error="log not found"
2026-02-28T09:02:52.456Z [INFO]  creating new snapshot: path=/tmp/raft45671060/snapshots/2-15-1772269372456.tmp
2026-02-28T09:02:52.456Z [INFO]  server-44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:52.457Z [INFO]  server-44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a: copied to local snapshot: bytes=71
2026-02-28T09:02:52.457Z [INFO]  server-44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a: snapshot restore progress: id=2-15-1772269372456 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:52.458Z [INFO]  server-44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a: Installed remote snapshot
2026-02-28T09:02:52.458Z [INFO]  server-7b4b2180-7172-1dba-80f9-c3e79f862496: pipelining replication: peer="{Voter server-44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a 44bdc52e-1bfb-d80a-f1e4-01db27a4cf6a}"
2026-02-28T09:02:52.458Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:52.460Z [ERROR] server-7b4b2180-7172-1dba-80f9-c3e79f862496: failed to get log: index=3 error="log not found"
2026-02-28T09:02:52.461Z [INFO]  creating new snapshot: path=/tmp/raft3082017998/snapshots/2-15-1772269372461.tmp
2026-02-28T09:02:52.461Z [INFO]  server-077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:52.461Z [INFO]  server-077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2: copied to local snapshot: bytes=71
2026-02-28T09:02:52.462Z [INFO]  server-077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2: snapshot restore progress: id=2-15-1772269372461 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:52.462Z [INFO]  server-077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2: Installed remote snapshot
2026-02-28T09:02:52.462Z [INFO]  server-7b4b2180-7172-1dba-80f9-c3e79f862496: pipelining replication: peer="{Voter server-077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2 077bc539-5d4f-0155-1cd0-b5ba3e3f6cd2}"
2026-02-28T09:02:53.464Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:52.458488422 +0000 UTC m=+50.422284961 (2 nodes), 16.96µs from start of poll, 1.94619556s from cluster start. Timeout at 2026-02-28 09:02:53.463495542 +0000 UTC m=+51.427292081, 1.00500712s after stability
2026-02-28T09:02:53.475Z [INFO]  server-71d20388-93b7-2cdc-2173-8da77323ccbd: aborting pipeline replication: peer="{Voter server-3627b8fb-41d6-08cb-a7fd-daed3fb25855 3627b8fb-41d6-08cb-a7fd-daed3fb25855}"
2026-02-28T09:02:53.475Z [INFO]  server-71d20388-93b7-2cdc-2173-8da77323ccbd: aborting pipeline replication: peer="{Voter server-f889c304-ae9d-f806-53a2-7b163be93de6 f889c304-ae9d-f806-53a2-7b163be93de6}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true
2026-02-28T09:02:53.476Z [INFO]  server-93da9f92-53ac-b6f5-b587-874e11adba2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93da9f92-53ac-b6f5-b587-874e11adba2a Address:93da9f92-53ac-b6f5-b587-874e11adba2a} {Suffrage:Voter ID:server-85ec7f84-0c45-8acc-b149-19f7aebeab24 Address:85ec7f84-0c45-8acc-b149-19f7aebeab24} {Suffrage:Voter ID:server-2c2bf237-a162-8bca-0d76-b6163618ea3b Address:2c2bf237-a162-8bca-0d76-b6163618ea3b}]"
2026-02-28T09:02:53.477Z [INFO]  server-85ec7f84-0c45-8acc-b149-19f7aebeab24: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93da9f92-53ac-b6f5-b587-874e11adba2a Address:93da9f92-53ac-b6f5-b587-874e11adba2a} {Suffrage:Voter ID:server-85ec7f84-0c45-8acc-b149-19f7aebeab24 Address:85ec7f84-0c45-8acc-b149-19f7aebeab24} {Suffrage:Voter ID:server-2c2bf237-a162-8bca-0d76-b6163618ea3b Address:2c2bf237-a162-8bca-0d76-b6163618ea3b}]"
2026-02-28T09:02:53.477Z [INFO]  server-2c2bf237-a162-8bca-0d76-b6163618ea3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93da9f92-53ac-b6f5-b587-874e11adba2a Address:93da9f92-53ac-b6f5-b587-874e11adba2a} {Suffrage:Voter ID:server-85ec7f84-0c45-8acc-b149-19f7aebeab24 Address:85ec7f84-0c45-8acc-b149-19f7aebeab24} {Suffrage:Voter ID:server-2c2bf237-a162-8bca-0d76-b6163618ea3b Address:2c2bf237-a162-8bca-0d76-b6163618ea3b}]"
2026-02-28T09:02:53.477Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:53.477Z [INFO]  server-85ec7f84-0c45-8acc-b149-19f7aebeab24: entering follower state: follower="Node at 85ec7f84-0c45-8acc-b149-19f7aebeab24 [Follower]" leader-address= leader-id=
2026-02-28T09:02:53.477Z [INFO]  server-93da9f92-53ac-b6f5-b587-874e11adba2a: entering follower state: follower="Node at 93da9f92-53ac-b6f5-b587-874e11adba2a [Follower]" leader-address= leader-id=
2026-02-28T09:02:53.477Z [INFO]  server-2c2bf237-a162-8bca-0d76-b6163618ea3b: entering follower state: follower="Node at 2c2bf237-a162-8bca-0d76-b6163618ea3b [Follower]" leader-address= leader-id=
2026-02-28T09:02:54.125Z [WARN]  server-85ec7f84-0c45-8acc-b149-19f7aebeab24: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:54.126Z [INFO]  server-85ec7f84-0c45-8acc-b149-19f7aebeab24: entering candidate state: node="Node at 85ec7f84-0c45-8acc-b149-19f7aebeab24 [Candidate]" term=2
2026-02-28T09:02:54.126Z [INFO]  server-85ec7f84-0c45-8acc-b149-19f7aebeab24: election won: term=2 tally=2
2026-02-28T09:02:54.126Z [INFO]  server-85ec7f84-0c45-8acc-b149-19f7aebeab24: entering leader state: leader="Node at 85ec7f84-0c45-8acc-b149-19f7aebeab24 [Leader]"
2026-02-28T09:02:54.126Z [INFO]  server-85ec7f84-0c45-8acc-b149-19f7aebeab24: added peer, starting replication: peer=server-93da9f92-53ac-b6f5-b587-874e11adba2a
2026-02-28T09:02:54.126Z [INFO]  server-85ec7f84-0c45-8acc-b149-19f7aebeab24: added peer, starting replication: peer=server-2c2bf237-a162-8bca-0d76-b6163618ea3b
2026-02-28T09:02:54.126Z [INFO]  server-85ec7f84-0c45-8acc-b149-19f7aebeab24: pipelining replication: peer="{Voter server-93da9f92-53ac-b6f5-b587-874e11adba2a 93da9f92-53ac-b6f5-b587-874e11adba2a}"
2026-02-28T09:02:54.126Z [INFO]  server-85ec7f84-0c45-8acc-b149-19f7aebeab24: pipelining replication: peer="{Voter server-2c2bf237-a162-8bca-0d76-b6163618ea3b 2c2bf237-a162-8bca-0d76-b6163618ea3b}"
2026-02-28T09:02:55.131Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:54.126280102 +0000 UTC m=+52.090076641 (1 nodes), 649.05164ms from start of poll, 649.5142ms from cluster start. Timeout at 2026-02-28 09:02:55.131287622 +0000 UTC m=+53.095084161, 1.00500752s after stability
2026-02-28T09:02:55.132Z [INFO]  server-85ec7f84-0c45-8acc-b149-19f7aebeab24: starting snapshot up to: index=12
2026-02-28T09:02:55.132Z [INFO]  creating new snapshot: path=/tmp/raft1392614937/snapshots/2-12-1772269375132.tmp
2026-02-28T09:02:55.132Z [INFO]  server-85ec7f84-0c45-8acc-b149-19f7aebeab24: snapshot complete up to: index=12
2026-02-28T09:02:55.134Z [INFO]  server-ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2 Address:ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2} {Suffrage:Voter ID:server-d3b18f04-c2a9-746d-1088-fe45847d80af Address:d3b18f04-c2a9-746d-1088-fe45847d80af} {Suffrage:Voter ID:server-07030bed-83ff-6afa-f20c-60d24711c059 Address:07030bed-83ff-6afa-f20c-60d24711c059}]"
2026-02-28T09:02:55.134Z [INFO]  server-d3b18f04-c2a9-746d-1088-fe45847d80af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2 Address:ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2} {Suffrage:Voter ID:server-d3b18f04-c2a9-746d-1088-fe45847d80af Address:d3b18f04-c2a9-746d-1088-fe45847d80af} {Suffrage:Voter ID:server-07030bed-83ff-6afa-f20c-60d24711c059 Address:07030bed-83ff-6afa-f20c-60d24711c059}]"
2026-02-28T09:02:55.134Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2 Address:ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2} {Suffrage:Voter ID:server-d3b18f04-c2a9-746d-1088-fe45847d80af Address:d3b18f04-c2a9-746d-1088-fe45847d80af} {Suffrage:Voter ID:server-07030bed-83ff-6afa-f20c-60d24711c059 Address:07030bed-83ff-6afa-f20c-60d24711c059}]"
2026-02-28T09:02:55.134Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:55.134Z [INFO]  server-ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2: entering follower state: follower="Node at ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2 [Follower]" leader-address= leader-id=
2026-02-28T09:02:55.134Z [INFO]  server-d3b18f04-c2a9-746d-1088-fe45847d80af: entering follower state: follower="Node at d3b18f04-c2a9-746d-1088-fe45847d80af [Follower]" leader-address= leader-id=
2026-02-28T09:02:55.134Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: entering follower state: follower="Node at 07030bed-83ff-6afa-f20c-60d24711c059 [Follower]" leader-address= leader-id=
2026-02-28T09:02:55.748Z [WARN]  server-07030bed-83ff-6afa-f20c-60d24711c059: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:55.748Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: entering candidate state: node="Node at 07030bed-83ff-6afa-f20c-60d24711c059 [Candidate]" term=2
2026-02-28T09:02:55.748Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: election won: term=2 tally=2
2026-02-28T09:02:55.748Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: entering leader state: leader="Node at 07030bed-83ff-6afa-f20c-60d24711c059 [Leader]"
2026-02-28T09:02:55.748Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: added peer, starting replication: peer=server-ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2
2026-02-28T09:02:55.748Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: added peer, starting replication: peer=server-d3b18f04-c2a9-746d-1088-fe45847d80af
2026-02-28T09:02:55.748Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: pipelining replication: peer="{Voter server-ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2 ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2}"
2026-02-28T09:02:55.748Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: pipelining replication: peer="{Voter server-d3b18f04-c2a9-746d-1088-fe45847d80af d3b18f04-c2a9-746d-1088-fe45847d80af}"
2026-02-28T09:02:56.754Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:55.748908902 +0000 UTC m=+53.712705441 (1 nodes), 614.58828ms from start of poll, 614.9958ms from cluster start. Timeout at 2026-02-28 09:02:56.753919062 +0000 UTC m=+54.717715601, 1.00501016s after stability
2026-02-28T09:02:56.754Z [INFO]  creating new snapshot: path=/tmp/raft2616669016/snapshots/2-15-1772269376754.tmp
2026-02-28T09:02:56.754Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: copied to local snapshot: bytes=71
2026-02-28T09:02:56.755Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: snapshot restore progress: id=2-15-1772269376754 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:56.755Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: removing all old logs from log store
2026-02-28T09:02:56.755Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: compacting logs: from=1 to=2
2026-02-28T09:02:56.755Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: restored user snapshot: index=15
2026-02-28T09:02:56.755Z [ERROR] server-07030bed-83ff-6afa-f20c-60d24711c059: failed to get log: index=2 error="log not found"
2026-02-28T09:02:56.755Z [ERROR] server-07030bed-83ff-6afa-f20c-60d24711c059: failed to get log: index=2 error="log not found"
2026-02-28T09:02:56.755Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: aborting pipeline replication: peer="{Voter server-ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2 ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2}"
2026-02-28T09:02:56.755Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: aborting pipeline replication: peer="{Voter server-d3b18f04-c2a9-746d-1088-fe45847d80af d3b18f04-c2a9-746d-1088-fe45847d80af}"
2026-02-28T09:02:56.764Z [ERROR] server-07030bed-83ff-6afa-f20c-60d24711c059: failed to get log: index=2 error="log not found"
2026-02-28T09:02:56.764Z [ERROR] server-07030bed-83ff-6afa-f20c-60d24711c059: failed to get log: index=2 error="log not found"
2026-02-28T09:02:56.765Z [INFO]  creating new snapshot: path=/tmp/raft1829542785/snapshots/2-15-1772269376765.tmp
2026-02-28T09:02:56.765Z [INFO]  creating new snapshot: path=/tmp/raft1118113465/snapshots/2-15-1772269376765.tmp
2026-02-28T09:02:56.765Z [INFO]  server-d3b18f04-c2a9-746d-1088-fe45847d80af: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:56.765Z [INFO]  server-ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:56.765Z [INFO]  server-d3b18f04-c2a9-746d-1088-fe45847d80af: copied to local snapshot: bytes=71
2026-02-28T09:02:56.766Z [INFO]  server-ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2: copied to local snapshot: bytes=71
2026-02-28T09:02:56.766Z [INFO]  server-d3b18f04-c2a9-746d-1088-fe45847d80af: snapshot restore progress: id=2-15-1772269376765 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:56.766Z [INFO]  server-d3b18f04-c2a9-746d-1088-fe45847d80af: removing all old logs from log store
2026-02-28T09:02:56.766Z [INFO]  server-d3b18f04-c2a9-746d-1088-fe45847d80af: compacting logs: from=1 to=2
2026-02-28T09:02:56.766Z [INFO]  server-d3b18f04-c2a9-746d-1088-fe45847d80af: Installed remote snapshot
2026-02-28T09:02:56.766Z [INFO]  server-ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2: snapshot restore progress: id=2-15-1772269376765 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:56.766Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: pipelining replication: peer="{Voter server-d3b18f04-c2a9-746d-1088-fe45847d80af d3b18f04-c2a9-746d-1088-fe45847d80af}"
2026-02-28T09:02:56.766Z [INFO]  server-ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2: removing all old logs from log store
2026-02-28T09:02:56.766Z [INFO]  server-ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2: compacting logs: from=1 to=2
2026-02-28T09:02:56.766Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:56.766Z [INFO]  server-ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2: Installed remote snapshot
2026-02-28T09:02:56.766Z [INFO]  server-07030bed-83ff-6afa-f20c-60d24711c059: pipelining replication: peer="{Voter server-ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2 ee9d0eaf-ed5a-d4c9-93dc-464c511e57e2}"
2026-02-28T09:02:57.772Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:56.766481502 +0000 UTC m=+54.730278041 (2 nodes), 960ns from start of poll, 1.6325684s from cluster start. Timeout at 2026-02-28 09:02:57.771499862 +0000 UTC m=+55.735296401, 1.00501836s after stability
2026-02-28T09:02:57.784Z [INFO]  server-85ec7f84-0c45-8acc-b149-19f7aebeab24: aborting pipeline replication: peer="{Voter server-2c2bf237-a162-8bca-0d76-b6163618ea3b 2c2bf237-a162-8bca-0d76-b6163618ea3b}"
2026-02-28T09:02:57.784Z [INFO]  server-85ec7f84-0c45-8acc-b149-19f7aebeab24: aborting pipeline replication: peer="{Voter server-93da9f92-53ac-b6f5-b587-874e11adba2a 93da9f92-53ac-b6f5-b587-874e11adba2a}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false
2026-02-28T09:02:57.785Z [INFO]  server-29a491b0-76fa-835d-db63-9ee1b040190e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29a491b0-76fa-835d-db63-9ee1b040190e Address:29a491b0-76fa-835d-db63-9ee1b040190e} {Suffrage:Voter ID:server-37cc16fe-b120-2f28-d434-d4b736fadcac Address:37cc16fe-b120-2f28-d434-d4b736fadcac} {Suffrage:Voter ID:server-a51b44f3-6961-0a2d-49d7-672ff2beb0db Address:a51b44f3-6961-0a2d-49d7-672ff2beb0db}]"
2026-02-28T09:02:57.785Z [INFO]  server-29a491b0-76fa-835d-db63-9ee1b040190e: entering follower state: follower="Node at 29a491b0-76fa-835d-db63-9ee1b040190e [Follower]" leader-address= leader-id=
2026-02-28T09:02:57.785Z [INFO]  server-37cc16fe-b120-2f28-d434-d4b736fadcac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29a491b0-76fa-835d-db63-9ee1b040190e Address:29a491b0-76fa-835d-db63-9ee1b040190e} {Suffrage:Voter ID:server-37cc16fe-b120-2f28-d434-d4b736fadcac Address:37cc16fe-b120-2f28-d434-d4b736fadcac} {Suffrage:Voter ID:server-a51b44f3-6961-0a2d-49d7-672ff2beb0db Address:a51b44f3-6961-0a2d-49d7-672ff2beb0db}]"
2026-02-28T09:02:57.785Z [INFO]  server-37cc16fe-b120-2f28-d434-d4b736fadcac: entering follower state: follower="Node at 37cc16fe-b120-2f28-d434-d4b736fadcac [Follower]" leader-address= leader-id=
2026-02-28T09:02:57.786Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29a491b0-76fa-835d-db63-9ee1b040190e Address:29a491b0-76fa-835d-db63-9ee1b040190e} {Suffrage:Voter ID:server-37cc16fe-b120-2f28-d434-d4b736fadcac Address:37cc16fe-b120-2f28-d434-d4b736fadcac} {Suffrage:Voter ID:server-a51b44f3-6961-0a2d-49d7-672ff2beb0db Address:a51b44f3-6961-0a2d-49d7-672ff2beb0db}]"
2026-02-28T09:02:57.786Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:02:57.786Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: entering follower state: follower="Node at a51b44f3-6961-0a2d-49d7-672ff2beb0db [Follower]" leader-address= leader-id=
2026-02-28T09:02:58.308Z [WARN]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:02:58.308Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: entering candidate state: node="Node at a51b44f3-6961-0a2d-49d7-672ff2beb0db [Candidate]" term=2
2026-02-28T09:02:58.309Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: election won: term=2 tally=2
2026-02-28T09:02:58.309Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: entering leader state: leader="Node at a51b44f3-6961-0a2d-49d7-672ff2beb0db [Leader]"
2026-02-28T09:02:58.309Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: added peer, starting replication: peer=server-29a491b0-76fa-835d-db63-9ee1b040190e
2026-02-28T09:02:58.309Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: added peer, starting replication: peer=server-37cc16fe-b120-2f28-d434-d4b736fadcac
2026-02-28T09:02:58.309Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: pipelining replication: peer="{Voter server-29a491b0-76fa-835d-db63-9ee1b040190e 29a491b0-76fa-835d-db63-9ee1b040190e}"
2026-02-28T09:02:58.309Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: pipelining replication: peer="{Voter server-37cc16fe-b120-2f28-d434-d4b736fadcac 37cc16fe-b120-2f28-d434-d4b736fadcac}"
2026-02-28T09:02:59.314Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:02:58.309060102 +0000 UTC m=+56.272856641 (1 nodes), 522.89196ms from start of poll, 523.49224ms from cluster start. Timeout at 2026-02-28 09:02:59.314067102 +0000 UTC m=+57.277863681, 1.00500704s after stability
2026-02-28T09:02:59.314Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: starting snapshot up to: index=12
2026-02-28T09:02:59.314Z [INFO]  creating new snapshot: path=/tmp/raft3795889472/snapshots/2-12-1772269379314.tmp
2026-02-28T09:02:59.315Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: snapshot complete up to: index=12
2026-02-28T09:02:59.317Z [INFO]  creating new snapshot: path=/tmp/raft3795889472/snapshots/2-113-1772269379317.tmp
2026-02-28T09:02:59.318Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: copied to local snapshot: bytes=71
2026-02-28T09:02:59.318Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: snapshot restore progress: id=2-113-1772269379317 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:59.318Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: restored user snapshot: index=113
2026-02-28T09:02:59.318Z [ERROR] server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: failed to get log: index=23 error="log not found"
2026-02-28T09:02:59.318Z [ERROR] server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: failed to get log: index=23 error="log not found"
2026-02-28T09:02:59.318Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: aborting pipeline replication: peer="{Voter server-29a491b0-76fa-835d-db63-9ee1b040190e 29a491b0-76fa-835d-db63-9ee1b040190e}"
2026-02-28T09:02:59.318Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: aborting pipeline replication: peer="{Voter server-37cc16fe-b120-2f28-d434-d4b736fadcac 37cc16fe-b120-2f28-d434-d4b736fadcac}"
2026-02-28T09:02:59.324Z [ERROR] server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: failed to get log: index=23 error="log not found"
2026-02-28T09:02:59.324Z [INFO]  creating new snapshot: path=/tmp/raft2959286509/snapshots/2-113-1772269379324.tmp
2026-02-28T09:02:59.324Z [INFO]  server-29a491b0-76fa-835d-db63-9ee1b040190e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:59.325Z [INFO]  server-29a491b0-76fa-835d-db63-9ee1b040190e: copied to local snapshot: bytes=71
2026-02-28T09:02:59.325Z [INFO]  server-29a491b0-76fa-835d-db63-9ee1b040190e: snapshot restore progress: id=2-113-1772269379324 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:59.325Z [INFO]  server-29a491b0-76fa-835d-db63-9ee1b040190e: Installed remote snapshot
2026-02-28T09:02:59.325Z [WARN]  server-29a491b0-76fa-835d-db63-9ee1b040190e: skipping application of old log: index=22
2026-02-28T09:02:59.325Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: pipelining replication: peer="{Voter server-29a491b0-76fa-835d-db63-9ee1b040190e 29a491b0-76fa-835d-db63-9ee1b040190e}"
2026-02-28T09:02:59.325Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:02:59.329Z [ERROR] server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: failed to get log: index=23 error="log not found"
2026-02-28T09:02:59.329Z [INFO]  creating new snapshot: path=/tmp/raft1082946597/snapshots/2-113-1772269379329.tmp
2026-02-28T09:02:59.329Z [INFO]  server-37cc16fe-b120-2f28-d434-d4b736fadcac: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:59.330Z [INFO]  server-37cc16fe-b120-2f28-d434-d4b736fadcac: copied to local snapshot: bytes=71
2026-02-28T09:02:59.330Z [INFO]  server-37cc16fe-b120-2f28-d434-d4b736fadcac: snapshot restore progress: id=2-113-1772269379329 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:02:59.330Z [INFO]  server-37cc16fe-b120-2f28-d434-d4b736fadcac: Installed remote snapshot
2026-02-28T09:02:59.330Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: pipelining replication: peer="{Voter server-37cc16fe-b120-2f28-d434-d4b736fadcac 37cc16fe-b120-2f28-d434-d4b736fadcac}"
2026-02-28T09:03:00.330Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:02:59.325696422 +0000 UTC m=+57.289492961 (2 nodes), 1.2µs from start of poll, 1.54012856s from cluster start. Timeout at 2026-02-28 09:03:00.330731142 +0000 UTC m=+58.294527681, 1.00503472s after stability
2026-02-28T09:03:00.343Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: aborting pipeline replication: peer="{Voter server-37cc16fe-b120-2f28-d434-d4b736fadcac 37cc16fe-b120-2f28-d434-d4b736fadcac}"
2026-02-28T09:03:00.343Z [INFO]  server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: aborting pipeline replication: peer="{Voter server-29a491b0-76fa-835d-db63-9ee1b040190e 29a491b0-76fa-835d-db63-9ee1b040190e}"
2026-02-28T09:03:00.844Z [ERROR] server-a51b44f3-6961-0a2d-49d7-672ff2beb0db: failed to heartbeat to: peer=37cc16fe-b120-2f28-d434-d4b736fadcac backoff time=10ms error="command timed out"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false
2026-02-28T09:03:00.845Z [INFO]  server-5d10b3af-b237-f48b-7d68-c55c8574ee16: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5d10b3af-b237-f48b-7d68-c55c8574ee16 Address:5d10b3af-b237-f48b-7d68-c55c8574ee16} {Suffrage:Voter ID:server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567 Address:5dbd1806-1f8e-c835-24e7-d7ce6a0ac567} {Suffrage:Voter ID:server-a6a5fa55-f5df-a0be-2217-a7e41d670f1d Address:a6a5fa55-f5df-a0be-2217-a7e41d670f1d}]"
2026-02-28T09:03:00.845Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5d10b3af-b237-f48b-7d68-c55c8574ee16 Address:5d10b3af-b237-f48b-7d68-c55c8574ee16} {Suffrage:Voter ID:server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567 Address:5dbd1806-1f8e-c835-24e7-d7ce6a0ac567} {Suffrage:Voter ID:server-a6a5fa55-f5df-a0be-2217-a7e41d670f1d Address:a6a5fa55-f5df-a0be-2217-a7e41d670f1d}]"
2026-02-28T09:03:00.845Z [INFO]  server-5d10b3af-b237-f48b-7d68-c55c8574ee16: entering follower state: follower="Node at 5d10b3af-b237-f48b-7d68-c55c8574ee16 [Follower]" leader-address= leader-id=
2026-02-28T09:03:00.845Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: entering follower state: follower="Node at 5dbd1806-1f8e-c835-24e7-d7ce6a0ac567 [Follower]" leader-address= leader-id=
2026-02-28T09:03:00.845Z [INFO]  server-a6a5fa55-f5df-a0be-2217-a7e41d670f1d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5d10b3af-b237-f48b-7d68-c55c8574ee16 Address:5d10b3af-b237-f48b-7d68-c55c8574ee16} {Suffrage:Voter ID:server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567 Address:5dbd1806-1f8e-c835-24e7-d7ce6a0ac567} {Suffrage:Voter ID:server-a6a5fa55-f5df-a0be-2217-a7e41d670f1d Address:a6a5fa55-f5df-a0be-2217-a7e41d670f1d}]"
2026-02-28T09:03:00.845Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:00.845Z [INFO]  server-a6a5fa55-f5df-a0be-2217-a7e41d670f1d: entering follower state: follower="Node at a6a5fa55-f5df-a0be-2217-a7e41d670f1d [Follower]" leader-address= leader-id=
2026-02-28T09:03:01.431Z [WARN]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:01.431Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: entering candidate state: node="Node at 5dbd1806-1f8e-c835-24e7-d7ce6a0ac567 [Candidate]" term=2
2026-02-28T09:03:01.431Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: election won: term=2 tally=2
2026-02-28T09:03:01.431Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: entering leader state: leader="Node at 5dbd1806-1f8e-c835-24e7-d7ce6a0ac567 [Leader]"
2026-02-28T09:03:01.431Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: added peer, starting replication: peer=server-5d10b3af-b237-f48b-7d68-c55c8574ee16
2026-02-28T09:03:01.431Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: added peer, starting replication: peer=server-a6a5fa55-f5df-a0be-2217-a7e41d670f1d
2026-02-28T09:03:01.431Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: pipelining replication: peer="{Voter server-a6a5fa55-f5df-a0be-2217-a7e41d670f1d a6a5fa55-f5df-a0be-2217-a7e41d670f1d}"
2026-02-28T09:03:01.432Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: pipelining replication: peer="{Voter server-5d10b3af-b237-f48b-7d68-c55c8574ee16 5d10b3af-b237-f48b-7d68-c55c8574ee16}"
2026-02-28T09:03:02.436Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:01.431815302 +0000 UTC m=+59.395611841 (1 nodes), 585.91652ms from start of poll, 586.37832ms from cluster start. Timeout at 2026-02-28 09:03:02.436819542 +0000 UTC m=+60.400616081, 1.00500424s after stability
2026-02-28T09:03:02.437Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: starting snapshot up to: index=12
2026-02-28T09:03:02.437Z [INFO]  creating new snapshot: path=/tmp/raft1107502516/snapshots/2-12-1772269382437.tmp
2026-02-28T09:03:02.438Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: snapshot complete up to: index=12
2026-02-28T09:03:02.440Z [INFO]  creating new snapshot: path=/tmp/raft1107502516/snapshots/2-113-1772269382440.tmp
2026-02-28T09:03:02.441Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: copied to local snapshot: bytes=71
2026-02-28T09:03:02.441Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: snapshot restore progress: id=2-113-1772269382440 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:02.441Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: removing all old logs from log store
2026-02-28T09:03:02.441Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: compacting logs: from=1 to=22
2026-02-28T09:03:02.441Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: restored user snapshot: index=113
2026-02-28T09:03:02.441Z [ERROR] server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: failed to get log: index=22 error="log not found"
2026-02-28T09:03:02.441Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: aborting pipeline replication: peer="{Voter server-a6a5fa55-f5df-a0be-2217-a7e41d670f1d a6a5fa55-f5df-a0be-2217-a7e41d670f1d}"
2026-02-28T09:03:02.441Z [ERROR] server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: failed to get log: index=22 error="log not found"
2026-02-28T09:03:02.441Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: aborting pipeline replication: peer="{Voter server-5d10b3af-b237-f48b-7d68-c55c8574ee16 5d10b3af-b237-f48b-7d68-c55c8574ee16}"
2026-02-28T09:03:02.447Z [ERROR] server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: failed to get log: index=22 error="log not found"
2026-02-28T09:03:02.447Z [INFO]  creating new snapshot: path=/tmp/raft1131313940/snapshots/2-113-1772269382447.tmp
2026-02-28T09:03:02.448Z [INFO]  server-a6a5fa55-f5df-a0be-2217-a7e41d670f1d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:02.448Z [INFO]  server-a6a5fa55-f5df-a0be-2217-a7e41d670f1d: copied to local snapshot: bytes=71
2026-02-28T09:03:02.448Z [ERROR] server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: failed to get log: index=22 error="log not found"
2026-02-28T09:03:02.448Z [INFO]  server-a6a5fa55-f5df-a0be-2217-a7e41d670f1d: snapshot restore progress: id=2-113-1772269382447 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:02.448Z [INFO]  server-a6a5fa55-f5df-a0be-2217-a7e41d670f1d: removing all old logs from log store
2026-02-28T09:03:02.448Z [INFO]  server-a6a5fa55-f5df-a0be-2217-a7e41d670f1d: compacting logs: from=1 to=22
2026-02-28T09:03:02.448Z [INFO]  server-a6a5fa55-f5df-a0be-2217-a7e41d670f1d: Installed remote snapshot
2026-02-28T09:03:02.449Z [WARN]  server-a6a5fa55-f5df-a0be-2217-a7e41d670f1d: skipping application of old log: index=22
2026-02-28T09:03:02.449Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: pipelining replication: peer="{Voter server-a6a5fa55-f5df-a0be-2217-a7e41d670f1d a6a5fa55-f5df-a0be-2217-a7e41d670f1d}"
2026-02-28T09:03:02.449Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:02.449Z [INFO]  creating new snapshot: path=/tmp/raft1814488205/snapshots/2-113-1772269382449.tmp
2026-02-28T09:03:02.449Z [INFO]  server-5d10b3af-b237-f48b-7d68-c55c8574ee16: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:02.449Z [INFO]  server-5d10b3af-b237-f48b-7d68-c55c8574ee16: copied to local snapshot: bytes=71
2026-02-28T09:03:02.450Z [INFO]  server-5d10b3af-b237-f48b-7d68-c55c8574ee16: snapshot restore progress: id=2-113-1772269382449 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:02.450Z [INFO]  server-5d10b3af-b237-f48b-7d68-c55c8574ee16: removing all old logs from log store
2026-02-28T09:03:02.450Z [INFO]  server-5d10b3af-b237-f48b-7d68-c55c8574ee16: compacting logs: from=1 to=22
2026-02-28T09:03:02.450Z [INFO]  server-5d10b3af-b237-f48b-7d68-c55c8574ee16: Installed remote snapshot
2026-02-28T09:03:02.450Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: pipelining replication: peer="{Voter server-5d10b3af-b237-f48b-7d68-c55c8574ee16 5d10b3af-b237-f48b-7d68-c55c8574ee16}"
2026-02-28T09:03:03.454Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:02.449156542 +0000 UTC m=+60.412953081 (2 nodes), 1.4µs from start of poll, 1.60371956s from cluster start. Timeout at 2026-02-28 09:03:03.454176222 +0000 UTC m=+61.417972761, 1.00501968s after stability
2026-02-28T09:03:03.467Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: aborting pipeline replication: peer="{Voter server-a6a5fa55-f5df-a0be-2217-a7e41d670f1d a6a5fa55-f5df-a0be-2217-a7e41d670f1d}"
2026-02-28T09:03:03.467Z [INFO]  server-5dbd1806-1f8e-c835-24e7-d7ce6a0ac567: aborting pipeline replication: peer="{Voter server-5d10b3af-b237-f48b-7d68-c55c8574ee16 5d10b3af-b237-f48b-7d68-c55c8574ee16}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true
2026-02-28T09:03:03.468Z [INFO]  server-2266c2e7-2882-5772-3cdc-55a41c8e14e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2266c2e7-2882-5772-3cdc-55a41c8e14e8 Address:2266c2e7-2882-5772-3cdc-55a41c8e14e8} {Suffrage:Voter ID:server-818d071f-c0ab-8f0b-7e17-eb11ad022b29 Address:818d071f-c0ab-8f0b-7e17-eb11ad022b29} {Suffrage:Voter ID:server-5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43 Address:5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43}]"
2026-02-28T09:03:03.468Z [INFO]  server-818d071f-c0ab-8f0b-7e17-eb11ad022b29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2266c2e7-2882-5772-3cdc-55a41c8e14e8 Address:2266c2e7-2882-5772-3cdc-55a41c8e14e8} {Suffrage:Voter ID:server-818d071f-c0ab-8f0b-7e17-eb11ad022b29 Address:818d071f-c0ab-8f0b-7e17-eb11ad022b29} {Suffrage:Voter ID:server-5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43 Address:5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43}]"
2026-02-28T09:03:03.468Z [INFO]  server-2266c2e7-2882-5772-3cdc-55a41c8e14e8: entering follower state: follower="Node at 2266c2e7-2882-5772-3cdc-55a41c8e14e8 [Follower]" leader-address= leader-id=
2026-02-28T09:03:03.468Z [INFO]  server-818d071f-c0ab-8f0b-7e17-eb11ad022b29: entering follower state: follower="Node at 818d071f-c0ab-8f0b-7e17-eb11ad022b29 [Follower]" leader-address= leader-id=
2026-02-28T09:03:03.468Z [INFO]  server-5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2266c2e7-2882-5772-3cdc-55a41c8e14e8 Address:2266c2e7-2882-5772-3cdc-55a41c8e14e8} {Suffrage:Voter ID:server-818d071f-c0ab-8f0b-7e17-eb11ad022b29 Address:818d071f-c0ab-8f0b-7e17-eb11ad022b29} {Suffrage:Voter ID:server-5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43 Address:5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43}]"
2026-02-28T09:03:03.469Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:03.469Z [INFO]  server-5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43: entering follower state: follower="Node at 5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43 [Follower]" leader-address= leader-id=
2026-02-28T09:03:04.047Z [WARN]  server-5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:04.048Z [INFO]  server-5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43: entering candidate state: node="Node at 5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43 [Candidate]" term=2
2026-02-28T09:03:04.048Z [INFO]  server-5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43: election won: term=2 tally=2
2026-02-28T09:03:04.048Z [INFO]  server-5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43: entering leader state: leader="Node at 5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43 [Leader]"
2026-02-28T09:03:04.048Z [INFO]  server-5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43: added peer, starting replication: peer=server-2266c2e7-2882-5772-3cdc-55a41c8e14e8
2026-02-28T09:03:04.048Z [INFO]  server-5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43: added peer, starting replication: peer=server-818d071f-c0ab-8f0b-7e17-eb11ad022b29
2026-02-28T09:03:04.048Z [INFO]  server-5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43: pipelining replication: peer="{Voter server-2266c2e7-2882-5772-3cdc-55a41c8e14e8 2266c2e7-2882-5772-3cdc-55a41c8e14e8}"
2026-02-28T09:03:04.048Z [INFO]  server-5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43: pipelining replication: peer="{Voter server-818d071f-c0ab-8f0b-7e17-eb11ad022b29 818d071f-c0ab-8f0b-7e17-eb11ad022b29}"
2026-02-28T09:03:05.053Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:04.048204662 +0000 UTC m=+62.012001201 (1 nodes), 579.12652ms from start of poll, 579.62852ms from cluster start. Timeout at 2026-02-28 09:03:05.053205702 +0000 UTC m=+63.017002241, 1.00500104s after stability
2026-02-28T09:03:05.054Z [INFO]  server-5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43: starting snapshot up to: index=12
2026-02-28T09:03:05.054Z [INFO]  creating new snapshot: path=/tmp/raft4285102924/snapshots/2-12-1772269385054.tmp
2026-02-28T09:03:05.054Z [INFO]  server-5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43: snapshot complete up to: index=12
2026-02-28T09:03:05.055Z [INFO]  server-63a77502-57e1-c25a-fbf9-6059fc266ef0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63a77502-57e1-c25a-fbf9-6059fc266ef0 Address:63a77502-57e1-c25a-fbf9-6059fc266ef0} {Suffrage:Voter ID:server-1670bd80-c27d-873e-6cae-7fe76fe25b2c Address:1670bd80-c27d-873e-6cae-7fe76fe25b2c} {Suffrage:Voter ID:server-ee049950-2fcd-0820-c8b6-93cf3e3be456 Address:ee049950-2fcd-0820-c8b6-93cf3e3be456}]"
2026-02-28T09:03:05.055Z [INFO]  server-1670bd80-c27d-873e-6cae-7fe76fe25b2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63a77502-57e1-c25a-fbf9-6059fc266ef0 Address:63a77502-57e1-c25a-fbf9-6059fc266ef0} {Suffrage:Voter ID:server-1670bd80-c27d-873e-6cae-7fe76fe25b2c Address:1670bd80-c27d-873e-6cae-7fe76fe25b2c} {Suffrage:Voter ID:server-ee049950-2fcd-0820-c8b6-93cf3e3be456 Address:ee049950-2fcd-0820-c8b6-93cf3e3be456}]"
2026-02-28T09:03:05.055Z [INFO]  server-63a77502-57e1-c25a-fbf9-6059fc266ef0: entering follower state: follower="Node at 63a77502-57e1-c25a-fbf9-6059fc266ef0 [Follower]" leader-address= leader-id=
2026-02-28T09:03:05.055Z [INFO]  server-1670bd80-c27d-873e-6cae-7fe76fe25b2c: entering follower state: follower="Node at 1670bd80-c27d-873e-6cae-7fe76fe25b2c [Follower]" leader-address= leader-id=
2026-02-28T09:03:05.055Z [INFO]  server-ee049950-2fcd-0820-c8b6-93cf3e3be456: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63a77502-57e1-c25a-fbf9-6059fc266ef0 Address:63a77502-57e1-c25a-fbf9-6059fc266ef0} {Suffrage:Voter ID:server-1670bd80-c27d-873e-6cae-7fe76fe25b2c Address:1670bd80-c27d-873e-6cae-7fe76fe25b2c} {Suffrage:Voter ID:server-ee049950-2fcd-0820-c8b6-93cf3e3be456 Address:ee049950-2fcd-0820-c8b6-93cf3e3be456}]"
2026-02-28T09:03:05.055Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:05.056Z [INFO]  server-ee049950-2fcd-0820-c8b6-93cf3e3be456: entering follower state: follower="Node at ee049950-2fcd-0820-c8b6-93cf3e3be456 [Follower]" leader-address= leader-id=
2026-02-28T09:03:05.586Z [WARN]  server-ee049950-2fcd-0820-c8b6-93cf3e3be456: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:05.586Z [INFO]  server-ee049950-2fcd-0820-c8b6-93cf3e3be456: entering candidate state: node="Node at ee049950-2fcd-0820-c8b6-93cf3e3be456 [Candidate]" term=2
2026-02-28T09:03:05.586Z [INFO]  server-ee049950-2fcd-0820-c8b6-93cf3e3be456: election won: term=2 tally=2
2026-02-28T09:03:05.586Z [INFO]  server-ee049950-2fcd-0820-c8b6-93cf3e3be456: entering leader state: leader="Node at ee049950-2fcd-0820-c8b6-93cf3e3be456 [Leader]"
2026-02-28T09:03:05.586Z [INFO]  server-ee049950-2fcd-0820-c8b6-93cf3e3be456: added peer, starting replication: peer=server-63a77502-57e1-c25a-fbf9-6059fc266ef0
2026-02-28T09:03:05.586Z [INFO]  server-ee049950-2fcd-0820-c8b6-93cf3e3be456: added peer, starting replication: peer=server-1670bd80-c27d-873e-6cae-7fe76fe25b2c
2026-02-28T09:03:05.586Z [INFO]  server-ee049950-2fcd-0820-c8b6-93cf3e3be456: pipelining replication: peer="{Voter server-1670bd80-c27d-873e-6cae-7fe76fe25b2c 1670bd80-c27d-873e-6cae-7fe76fe25b2c}"
2026-02-28T09:03:05.586Z [INFO]  server-ee049950-2fcd-0820-c8b6-93cf3e3be456: pipelining replication: peer="{Voter server-63a77502-57e1-c25a-fbf9-6059fc266ef0 63a77502-57e1-c25a-fbf9-6059fc266ef0}"
2026-02-28T09:03:06.591Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:05.586494462 +0000 UTC m=+63.550291001 (1 nodes), 530.49396ms from start of poll, 530.97528ms from cluster start. Timeout at 2026-02-28 09:03:06.591501662 +0000 UTC m=+64.555298201, 1.0050072s after stability
2026-02-28T09:03:06.592Z [INFO]  creating new snapshot: path=/tmp/raft184197214/snapshots/2-113-1772269386592.tmp
2026-02-28T09:03:06.592Z [INFO]  server-ee049950-2fcd-0820-c8b6-93cf3e3be456: copied to local snapshot: bytes=71
2026-02-28T09:03:06.592Z [INFO]  server-ee049950-2fcd-0820-c8b6-93cf3e3be456: snapshot restore progress: id=2-113-1772269386592 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:06.593Z [INFO]  server-ee049950-2fcd-0820-c8b6-93cf3e3be456: restored user snapshot: index=113
2026-02-28T09:03:06.593Z [ERROR] server-ee049950-2fcd-0820-c8b6-93cf3e3be456: failed to get log: index=3 error="log not found"
2026-02-28T09:03:06.593Z [INFO]  server-ee049950-2fcd-0820-c8b6-93cf3e3be456: aborting pipeline replication: peer="{Voter server-1670bd80-c27d-873e-6cae-7fe76fe25b2c 1670bd80-c27d-873e-6cae-7fe76fe25b2c}"
2026-02-28T09:03:06.593Z [ERROR] server-ee049950-2fcd-0820-c8b6-93cf3e3be456: failed to get log: index=3 error="log not found"
2026-02-28T09:03:06.593Z [INFO]  server-ee049950-2fcd-0820-c8b6-93cf3e3be456: aborting pipeline replication: peer="{Voter server-63a77502-57e1-c25a-fbf9-6059fc266ef0 63a77502-57e1-c25a-fbf9-6059fc266ef0}"
2026-02-28T09:03:06.603Z [ERROR] server-ee049950-2fcd-0820-c8b6-93cf3e3be456: failed to get log: index=3 error="log not found"
2026-02-28T09:03:06.603Z [ERROR] server-ee049950-2fcd-0820-c8b6-93cf3e3be456: failed to get log: index=3 error="log not found"
2026-02-28T09:03:06.604Z [INFO]  creating new snapshot: path=/tmp/raft1861608557/snapshots/2-113-1772269386604.tmp
2026-02-28T09:03:06.604Z [INFO]  creating new snapshot: path=/tmp/raft3029719624/snapshots/2-113-1772269386604.tmp
2026-02-28T09:03:06.604Z [INFO]  server-1670bd80-c27d-873e-6cae-7fe76fe25b2c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:06.604Z [INFO]  server-63a77502-57e1-c25a-fbf9-6059fc266ef0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:06.604Z [INFO]  server-1670bd80-c27d-873e-6cae-7fe76fe25b2c: copied to local snapshot: bytes=71
2026-02-28T09:03:06.604Z [INFO]  server-63a77502-57e1-c25a-fbf9-6059fc266ef0: copied to local snapshot: bytes=71
2026-02-28T09:03:06.605Z [INFO]  server-1670bd80-c27d-873e-6cae-7fe76fe25b2c: snapshot restore progress: id=2-113-1772269386604 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:06.605Z [INFO]  server-1670bd80-c27d-873e-6cae-7fe76fe25b2c: Installed remote snapshot
2026-02-28T09:03:06.605Z [INFO]  server-ee049950-2fcd-0820-c8b6-93cf3e3be456: pipelining replication: peer="{Voter server-1670bd80-c27d-873e-6cae-7fe76fe25b2c 1670bd80-c27d-873e-6cae-7fe76fe25b2c}"
2026-02-28T09:03:06.605Z [INFO]  server-63a77502-57e1-c25a-fbf9-6059fc266ef0: snapshot restore progress: id=2-113-1772269386604 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:06.605Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:06.605Z [INFO]  server-63a77502-57e1-c25a-fbf9-6059fc266ef0: Installed remote snapshot
2026-02-28T09:03:06.605Z [INFO]  server-ee049950-2fcd-0820-c8b6-93cf3e3be456: pipelining replication: peer="{Voter server-63a77502-57e1-c25a-fbf9-6059fc266ef0 63a77502-57e1-c25a-fbf9-6059fc266ef0}"
2026-02-28T09:03:07.611Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:06.605372622 +0000 UTC m=+64.569169161 (2 nodes), 23.08µs from start of poll, 1.54985344s from cluster start. Timeout at 2026-02-28 09:03:07.610390742 +0000 UTC m=+65.574187281, 1.00501812s after stability
2026-02-28T09:03:07.623Z [INFO]  server-5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43: aborting pipeline replication: peer="{Voter server-2266c2e7-2882-5772-3cdc-55a41c8e14e8 2266c2e7-2882-5772-3cdc-55a41c8e14e8}"
2026-02-28T09:03:07.623Z [INFO]  server-5eddcc05-a43e-8d4c-67dc-52dd9aa6bd43: aborting pipeline replication: peer="{Voter server-818d071f-c0ab-8f0b-7e17-eb11ad022b29 818d071f-c0ab-8f0b-7e17-eb11ad022b29}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true
2026-02-28T09:03:07.624Z [INFO]  server-e52e85ba-2585-fdfb-3840-d401205f4e81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e52e85ba-2585-fdfb-3840-d401205f4e81 Address:e52e85ba-2585-fdfb-3840-d401205f4e81} {Suffrage:Voter ID:server-242df3b4-60a1-8e36-c754-12a21ff7073b Address:242df3b4-60a1-8e36-c754-12a21ff7073b} {Suffrage:Voter ID:server-12b73594-3776-5d42-adbf-df4bcb7977fa Address:12b73594-3776-5d42-adbf-df4bcb7977fa}]"
2026-02-28T09:03:07.624Z [INFO]  server-242df3b4-60a1-8e36-c754-12a21ff7073b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e52e85ba-2585-fdfb-3840-d401205f4e81 Address:e52e85ba-2585-fdfb-3840-d401205f4e81} {Suffrage:Voter ID:server-242df3b4-60a1-8e36-c754-12a21ff7073b Address:242df3b4-60a1-8e36-c754-12a21ff7073b} {Suffrage:Voter ID:server-12b73594-3776-5d42-adbf-df4bcb7977fa Address:12b73594-3776-5d42-adbf-df4bcb7977fa}]"
2026-02-28T09:03:07.624Z [INFO]  server-12b73594-3776-5d42-adbf-df4bcb7977fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e52e85ba-2585-fdfb-3840-d401205f4e81 Address:e52e85ba-2585-fdfb-3840-d401205f4e81} {Suffrage:Voter ID:server-242df3b4-60a1-8e36-c754-12a21ff7073b Address:242df3b4-60a1-8e36-c754-12a21ff7073b} {Suffrage:Voter ID:server-12b73594-3776-5d42-adbf-df4bcb7977fa Address:12b73594-3776-5d42-adbf-df4bcb7977fa}]"
2026-02-28T09:03:07.624Z [INFO]  server-242df3b4-60a1-8e36-c754-12a21ff7073b: entering follower state: follower="Node at 242df3b4-60a1-8e36-c754-12a21ff7073b [Follower]" leader-address= leader-id=
2026-02-28T09:03:07.624Z [INFO]  server-e52e85ba-2585-fdfb-3840-d401205f4e81: entering follower state: follower="Node at e52e85ba-2585-fdfb-3840-d401205f4e81 [Follower]" leader-address= leader-id=
2026-02-28T09:03:07.624Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:07.624Z [INFO]  server-12b73594-3776-5d42-adbf-df4bcb7977fa: entering follower state: follower="Node at 12b73594-3776-5d42-adbf-df4bcb7977fa [Follower]" leader-address= leader-id=
2026-02-28T09:03:08.429Z [WARN]  server-242df3b4-60a1-8e36-c754-12a21ff7073b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:08.429Z [INFO]  server-242df3b4-60a1-8e36-c754-12a21ff7073b: entering candidate state: node="Node at 242df3b4-60a1-8e36-c754-12a21ff7073b [Candidate]" term=2
2026-02-28T09:03:08.429Z [INFO]  server-242df3b4-60a1-8e36-c754-12a21ff7073b: election won: term=2 tally=2
2026-02-28T09:03:08.429Z [INFO]  server-242df3b4-60a1-8e36-c754-12a21ff7073b: entering leader state: leader="Node at 242df3b4-60a1-8e36-c754-12a21ff7073b [Leader]"
2026-02-28T09:03:08.429Z [INFO]  server-242df3b4-60a1-8e36-c754-12a21ff7073b: added peer, starting replication: peer=server-e52e85ba-2585-fdfb-3840-d401205f4e81
2026-02-28T09:03:08.429Z [INFO]  server-242df3b4-60a1-8e36-c754-12a21ff7073b: added peer, starting replication: peer=server-12b73594-3776-5d42-adbf-df4bcb7977fa
2026-02-28T09:03:08.430Z [INFO]  server-242df3b4-60a1-8e36-c754-12a21ff7073b: pipelining replication: peer="{Voter server-e52e85ba-2585-fdfb-3840-d401205f4e81 e52e85ba-2585-fdfb-3840-d401205f4e81}"
2026-02-28T09:03:08.430Z [INFO]  server-242df3b4-60a1-8e36-c754-12a21ff7073b: pipelining replication: peer="{Voter server-12b73594-3776-5d42-adbf-df4bcb7977fa 12b73594-3776-5d42-adbf-df4bcb7977fa}"
2026-02-28T09:03:09.435Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:08.429880502 +0000 UTC m=+66.393677041 (1 nodes), 804.88004ms from start of poll, 805.35796ms from cluster start. Timeout at 2026-02-28 09:03:09.434885302 +0000 UTC m=+67.398681841, 1.0050048s after stability
2026-02-28T09:03:09.435Z [INFO]  server-242df3b4-60a1-8e36-c754-12a21ff7073b: starting snapshot up to: index=12
2026-02-28T09:03:09.435Z [INFO]  creating new snapshot: path=/tmp/raft361985262/snapshots/2-12-1772269389435.tmp
2026-02-28T09:03:09.436Z [INFO]  server-242df3b4-60a1-8e36-c754-12a21ff7073b: snapshot complete up to: index=12
2026-02-28T09:03:09.437Z [INFO]  server-dfa71713-f2a4-b915-3f0d-84f0fb90cd1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dfa71713-f2a4-b915-3f0d-84f0fb90cd1b Address:dfa71713-f2a4-b915-3f0d-84f0fb90cd1b} {Suffrage:Voter ID:server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e Address:9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e} {Suffrage:Voter ID:server-43a57be0-c7b3-b60f-0bed-5b41f2149763 Address:43a57be0-c7b3-b60f-0bed-5b41f2149763}]"
2026-02-28T09:03:09.437Z [INFO]  server-dfa71713-f2a4-b915-3f0d-84f0fb90cd1b: entering follower state: follower="Node at dfa71713-f2a4-b915-3f0d-84f0fb90cd1b [Follower]" leader-address= leader-id=
2026-02-28T09:03:09.437Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dfa71713-f2a4-b915-3f0d-84f0fb90cd1b Address:dfa71713-f2a4-b915-3f0d-84f0fb90cd1b} {Suffrage:Voter ID:server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e Address:9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e} {Suffrage:Voter ID:server-43a57be0-c7b3-b60f-0bed-5b41f2149763 Address:43a57be0-c7b3-b60f-0bed-5b41f2149763}]"
2026-02-28T09:03:09.437Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: entering follower state: follower="Node at 9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e [Follower]" leader-address= leader-id=
2026-02-28T09:03:09.437Z [INFO]  server-43a57be0-c7b3-b60f-0bed-5b41f2149763: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dfa71713-f2a4-b915-3f0d-84f0fb90cd1b Address:dfa71713-f2a4-b915-3f0d-84f0fb90cd1b} {Suffrage:Voter ID:server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e Address:9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e} {Suffrage:Voter ID:server-43a57be0-c7b3-b60f-0bed-5b41f2149763 Address:43a57be0-c7b3-b60f-0bed-5b41f2149763}]"
2026-02-28T09:03:09.437Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:09.437Z [INFO]  server-43a57be0-c7b3-b60f-0bed-5b41f2149763: entering follower state: follower="Node at 43a57be0-c7b3-b60f-0bed-5b41f2149763 [Follower]" leader-address= leader-id=
2026-02-28T09:03:10.014Z [WARN]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:10.014Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: entering candidate state: node="Node at 9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e [Candidate]" term=2
2026-02-28T09:03:10.014Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: election won: term=2 tally=2
2026-02-28T09:03:10.014Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: entering leader state: leader="Node at 9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e [Leader]"
2026-02-28T09:03:10.014Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: added peer, starting replication: peer=server-dfa71713-f2a4-b915-3f0d-84f0fb90cd1b
2026-02-28T09:03:10.014Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: added peer, starting replication: peer=server-43a57be0-c7b3-b60f-0bed-5b41f2149763
2026-02-28T09:03:10.015Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: pipelining replication: peer="{Voter server-43a57be0-c7b3-b60f-0bed-5b41f2149763 43a57be0-c7b3-b60f-0bed-5b41f2149763}"
2026-02-28T09:03:10.015Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: pipelining replication: peer="{Voter server-dfa71713-f2a4-b915-3f0d-84f0fb90cd1b dfa71713-f2a4-b915-3f0d-84f0fb90cd1b}"
2026-02-28T09:03:11.020Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:10.014987742 +0000 UTC m=+67.978784321 (1 nodes), 577.08972ms from start of poll, 577.63644ms from cluster start. Timeout at 2026-02-28 09:03:11.019993022 +0000 UTC m=+68.983789561, 1.00500524s after stability
2026-02-28T09:03:11.021Z [INFO]  creating new snapshot: path=/tmp/raft1862585947/snapshots/2-113-1772269391021.tmp
2026-02-28T09:03:11.021Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: copied to local snapshot: bytes=71
2026-02-28T09:03:11.021Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: snapshot restore progress: id=2-113-1772269391021 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:11.021Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: removing all old logs from log store
2026-02-28T09:03:11.021Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: compacting logs: from=1 to=2
2026-02-28T09:03:11.021Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: restored user snapshot: index=113
2026-02-28T09:03:11.021Z [ERROR] server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: failed to get log: index=2 error="log not found"
2026-02-28T09:03:11.022Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: aborting pipeline replication: peer="{Voter server-43a57be0-c7b3-b60f-0bed-5b41f2149763 43a57be0-c7b3-b60f-0bed-5b41f2149763}"
2026-02-28T09:03:11.022Z [ERROR] server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: failed to get log: index=2 error="log not found"
2026-02-28T09:03:11.022Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: aborting pipeline replication: peer="{Voter server-dfa71713-f2a4-b915-3f0d-84f0fb90cd1b dfa71713-f2a4-b915-3f0d-84f0fb90cd1b}"
2026-02-28T09:03:11.030Z [ERROR] server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: failed to get log: index=2 error="log not found"
2026-02-28T09:03:11.030Z [ERROR] server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: failed to get log: index=2 error="log not found"
2026-02-28T09:03:11.031Z [INFO]  creating new snapshot: path=/tmp/raft4088030319/snapshots/2-113-1772269391031.tmp
2026-02-28T09:03:11.031Z [INFO]  creating new snapshot: path=/tmp/raft3348671679/snapshots/2-113-1772269391031.tmp
2026-02-28T09:03:11.031Z [INFO]  server-43a57be0-c7b3-b60f-0bed-5b41f2149763: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:11.031Z [INFO]  server-dfa71713-f2a4-b915-3f0d-84f0fb90cd1b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:11.031Z [INFO]  server-dfa71713-f2a4-b915-3f0d-84f0fb90cd1b: copied to local snapshot: bytes=71
2026-02-28T09:03:11.031Z [INFO]  server-43a57be0-c7b3-b60f-0bed-5b41f2149763: copied to local snapshot: bytes=71
2026-02-28T09:03:11.032Z [INFO]  server-dfa71713-f2a4-b915-3f0d-84f0fb90cd1b: snapshot restore progress: id=2-113-1772269391031 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:11.032Z [INFO]  server-dfa71713-f2a4-b915-3f0d-84f0fb90cd1b: removing all old logs from log store
2026-02-28T09:03:11.032Z [INFO]  server-dfa71713-f2a4-b915-3f0d-84f0fb90cd1b: compacting logs: from=1 to=2
2026-02-28T09:03:11.032Z [INFO]  server-dfa71713-f2a4-b915-3f0d-84f0fb90cd1b: Installed remote snapshot
2026-02-28T09:03:11.032Z [INFO]  server-43a57be0-c7b3-b60f-0bed-5b41f2149763: snapshot restore progress: id=2-113-1772269391031 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:11.032Z [INFO]  server-43a57be0-c7b3-b60f-0bed-5b41f2149763: removing all old logs from log store
2026-02-28T09:03:11.032Z [INFO]  server-43a57be0-c7b3-b60f-0bed-5b41f2149763: compacting logs: from=1 to=2
2026-02-28T09:03:11.032Z [INFO]  server-43a57be0-c7b3-b60f-0bed-5b41f2149763: Installed remote snapshot
2026-02-28T09:03:11.032Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: pipelining replication: peer="{Voter server-dfa71713-f2a4-b915-3f0d-84f0fb90cd1b dfa71713-f2a4-b915-3f0d-84f0fb90cd1b}"
2026-02-28T09:03:11.032Z [INFO]  server-9dd9fcb6-4aea-607e-5a47-8d726bdb1b4e: pipelining replication: peer="{Voter server-43a57be0-c7b3-b60f-0bed-5b41f2149763 43a57be0-c7b3-b60f-0bed-5b41f2149763}"
2026-02-28T09:03:11.032Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:12.038Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:11.032466062 +0000 UTC m=+68.996262601 (2 nodes), 1.4µs from start of poll, 1.59511472s from cluster start. Timeout at 2026-02-28 09:03:12.037487942 +0000 UTC m=+70.001284481, 1.00502188s after stability
2026-02-28T09:03:12.050Z [INFO]  server-242df3b4-60a1-8e36-c754-12a21ff7073b: aborting pipeline replication: peer="{Voter server-e52e85ba-2585-fdfb-3840-d401205f4e81 e52e85ba-2585-fdfb-3840-d401205f4e81}"
2026-02-28T09:03:12.050Z [INFO]  server-242df3b4-60a1-8e36-c754-12a21ff7073b: aborting pipeline replication: peer="{Voter server-12b73594-3776-5d42-adbf-df4bcb7977fa 12b73594-3776-5d42-adbf-df4bcb7977fa}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false
2026-02-28T09:03:12.051Z [INFO]  server-fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c Address:fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c} {Suffrage:Voter ID:server-30e4a4c9-8365-12bc-8837-72d112e72814 Address:30e4a4c9-8365-12bc-8837-72d112e72814} {Suffrage:Voter ID:server-bb0db547-72e0-9e56-2ad2-039bd1e0c4df Address:bb0db547-72e0-9e56-2ad2-039bd1e0c4df}]"
2026-02-28T09:03:12.051Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c Address:fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c} {Suffrage:Voter ID:server-30e4a4c9-8365-12bc-8837-72d112e72814 Address:30e4a4c9-8365-12bc-8837-72d112e72814} {Suffrage:Voter ID:server-bb0db547-72e0-9e56-2ad2-039bd1e0c4df Address:bb0db547-72e0-9e56-2ad2-039bd1e0c4df}]"
2026-02-28T09:03:12.051Z [INFO]  server-fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c: entering follower state: follower="Node at fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c [Follower]" leader-address= leader-id=
2026-02-28T09:03:12.051Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: entering follower state: follower="Node at 30e4a4c9-8365-12bc-8837-72d112e72814 [Follower]" leader-address= leader-id=
2026-02-28T09:03:12.051Z [INFO]  server-bb0db547-72e0-9e56-2ad2-039bd1e0c4df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c Address:fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c} {Suffrage:Voter ID:server-30e4a4c9-8365-12bc-8837-72d112e72814 Address:30e4a4c9-8365-12bc-8837-72d112e72814} {Suffrage:Voter ID:server-bb0db547-72e0-9e56-2ad2-039bd1e0c4df Address:bb0db547-72e0-9e56-2ad2-039bd1e0c4df}]"
2026-02-28T09:03:12.051Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:12.051Z [INFO]  server-bb0db547-72e0-9e56-2ad2-039bd1e0c4df: entering follower state: follower="Node at bb0db547-72e0-9e56-2ad2-039bd1e0c4df [Follower]" leader-address= leader-id=
2026-02-28T09:03:12.654Z [WARN]  server-30e4a4c9-8365-12bc-8837-72d112e72814: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:12.654Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: entering candidate state: node="Node at 30e4a4c9-8365-12bc-8837-72d112e72814 [Candidate]" term=2
2026-02-28T09:03:12.655Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: election won: term=2 tally=2
2026-02-28T09:03:12.655Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: entering leader state: leader="Node at 30e4a4c9-8365-12bc-8837-72d112e72814 [Leader]"
2026-02-28T09:03:12.655Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: added peer, starting replication: peer=server-fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c
2026-02-28T09:03:12.655Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: added peer, starting replication: peer=server-bb0db547-72e0-9e56-2ad2-039bd1e0c4df
2026-02-28T09:03:12.655Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: pipelining replication: peer="{Voter server-bb0db547-72e0-9e56-2ad2-039bd1e0c4df bb0db547-72e0-9e56-2ad2-039bd1e0c4df}"
2026-02-28T09:03:12.655Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: pipelining replication: peer="{Voter server-fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c}"
2026-02-28T09:03:13.660Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:12.655229582 +0000 UTC m=+70.619026121 (1 nodes), 603.29108ms from start of poll, 603.80044ms from cluster start. Timeout at 2026-02-28 09:03:13.660234182 +0000 UTC m=+71.624030721, 1.0050046s after stability
2026-02-28T09:03:13.661Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: starting snapshot up to: index=12
2026-02-28T09:03:13.661Z [INFO]  creating new snapshot: path=/tmp/raft3640872490/snapshots/2-12-1772269393661.tmp
2026-02-28T09:03:13.661Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: snapshot complete up to: index=12
2026-02-28T09:03:13.662Z [INFO]  creating new snapshot: path=/tmp/raft3640872490/snapshots/2-1013-1772269393662.tmp
2026-02-28T09:03:13.663Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: copied to local snapshot: bytes=71
2026-02-28T09:03:13.663Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: snapshot restore progress: id=2-1013-1772269393662 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:13.663Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: restored user snapshot: index=1013
2026-02-28T09:03:13.663Z [ERROR] server-30e4a4c9-8365-12bc-8837-72d112e72814: failed to get log: index=23 error="log not found"
2026-02-28T09:03:13.663Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: aborting pipeline replication: peer="{Voter server-bb0db547-72e0-9e56-2ad2-039bd1e0c4df bb0db547-72e0-9e56-2ad2-039bd1e0c4df}"
2026-02-28T09:03:13.663Z [ERROR] server-30e4a4c9-8365-12bc-8837-72d112e72814: failed to get log: index=23 error="log not found"
2026-02-28T09:03:13.663Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: aborting pipeline replication: peer="{Voter server-fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c}"
2026-02-28T09:03:13.670Z [ERROR] server-30e4a4c9-8365-12bc-8837-72d112e72814: failed to get log: index=23 error="log not found"
2026-02-28T09:03:13.671Z [INFO]  creating new snapshot: path=/tmp/raft2976093238/snapshots/2-1013-1772269393671.tmp
2026-02-28T09:03:13.671Z [ERROR] server-30e4a4c9-8365-12bc-8837-72d112e72814: failed to get log: index=23 error="log not found"
2026-02-28T09:03:13.671Z [INFO]  server-fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:13.672Z [INFO]  creating new snapshot: path=/tmp/raft3489719762/snapshots/2-1013-1772269393671.tmp
2026-02-28T09:03:13.672Z [INFO]  server-fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c: copied to local snapshot: bytes=71
2026-02-28T09:03:13.672Z [INFO]  server-bb0db547-72e0-9e56-2ad2-039bd1e0c4df: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:13.672Z [INFO]  server-fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c: snapshot restore progress: id=2-1013-1772269393671 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:13.672Z [INFO]  server-fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c: Installed remote snapshot
2026-02-28T09:03:13.672Z [WARN]  server-fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c: skipping application of old log: index=22
2026-02-28T09:03:13.672Z [INFO]  server-bb0db547-72e0-9e56-2ad2-039bd1e0c4df: copied to local snapshot: bytes=71
2026-02-28T09:03:13.672Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: pipelining replication: peer="{Voter server-fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c}"
2026-02-28T09:03:13.672Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:13.672Z [INFO]  server-bb0db547-72e0-9e56-2ad2-039bd1e0c4df: snapshot restore progress: id=2-1013-1772269393671 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:13.672Z [INFO]  server-bb0db547-72e0-9e56-2ad2-039bd1e0c4df: Installed remote snapshot
2026-02-28T09:03:13.673Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: pipelining replication: peer="{Voter server-bb0db547-72e0-9e56-2ad2-039bd1e0c4df bb0db547-72e0-9e56-2ad2-039bd1e0c4df}"
2026-02-28T09:03:14.678Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:13.672650782 +0000 UTC m=+71.636447321 (2 nodes), 1.08µs from start of poll, 1.62122164s from cluster start. Timeout at 2026-02-28 09:03:14.677674462 +0000 UTC m=+72.641471041, 1.00502372s after stability
2026-02-28T09:03:14.690Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: aborting pipeline replication: peer="{Voter server-bb0db547-72e0-9e56-2ad2-039bd1e0c4df bb0db547-72e0-9e56-2ad2-039bd1e0c4df}"
2026-02-28T09:03:14.690Z [INFO]  server-30e4a4c9-8365-12bc-8837-72d112e72814: aborting pipeline replication: peer="{Voter server-fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c fd06d4ed-92c0-d6ec-a5fa-543b8a3b910c}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false
2026-02-28T09:03:14.691Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9 Address:d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9} {Suffrage:Voter ID:server-dda082cf-9bcc-ea6a-85ae-f2406640d62e Address:dda082cf-9bcc-ea6a-85ae-f2406640d62e} {Suffrage:Voter ID:server-9456247c-bfd9-2bd4-11d2-24c85c2b248d Address:9456247c-bfd9-2bd4-11d2-24c85c2b248d}]"
2026-02-28T09:03:14.691Z [INFO]  server-dda082cf-9bcc-ea6a-85ae-f2406640d62e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9 Address:d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9} {Suffrage:Voter ID:server-dda082cf-9bcc-ea6a-85ae-f2406640d62e Address:dda082cf-9bcc-ea6a-85ae-f2406640d62e} {Suffrage:Voter ID:server-9456247c-bfd9-2bd4-11d2-24c85c2b248d Address:9456247c-bfd9-2bd4-11d2-24c85c2b248d}]"
2026-02-28T09:03:14.691Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: entering follower state: follower="Node at d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9 [Follower]" leader-address= leader-id=
2026-02-28T09:03:14.691Z [INFO]  server-dda082cf-9bcc-ea6a-85ae-f2406640d62e: entering follower state: follower="Node at dda082cf-9bcc-ea6a-85ae-f2406640d62e [Follower]" leader-address= leader-id=
2026-02-28T09:03:14.691Z [INFO]  server-9456247c-bfd9-2bd4-11d2-24c85c2b248d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9 Address:d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9} {Suffrage:Voter ID:server-dda082cf-9bcc-ea6a-85ae-f2406640d62e Address:dda082cf-9bcc-ea6a-85ae-f2406640d62e} {Suffrage:Voter ID:server-9456247c-bfd9-2bd4-11d2-24c85c2b248d Address:9456247c-bfd9-2bd4-11d2-24c85c2b248d}]"
2026-02-28T09:03:14.691Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:14.691Z [INFO]  server-9456247c-bfd9-2bd4-11d2-24c85c2b248d: entering follower state: follower="Node at 9456247c-bfd9-2bd4-11d2-24c85c2b248d [Follower]" leader-address= leader-id=
2026-02-28T09:03:15.261Z [WARN]  server-dda082cf-9bcc-ea6a-85ae-f2406640d62e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:15.261Z [WARN]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:15.261Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: entering candidate state: node="Node at d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9 [Candidate]" term=2
2026-02-28T09:03:15.261Z [INFO]  server-dda082cf-9bcc-ea6a-85ae-f2406640d62e: entering candidate state: node="Node at dda082cf-9bcc-ea6a-85ae-f2406640d62e [Candidate]" term=2
2026-02-28T09:03:15.261Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: election won: term=2 tally=2
2026-02-28T09:03:15.261Z [INFO]  server-dda082cf-9bcc-ea6a-85ae-f2406640d62e: duplicate requestVote for same term: term=2
2026-02-28T09:03:15.262Z [INFO]  server-9456247c-bfd9-2bd4-11d2-24c85c2b248d: duplicate requestVote for same term: term=2
2026-02-28T09:03:15.261Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: entering leader state: leader="Node at d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9 [Leader]"
2026-02-28T09:03:15.262Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: added peer, starting replication: peer=server-dda082cf-9bcc-ea6a-85ae-f2406640d62e
2026-02-28T09:03:15.262Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: added peer, starting replication: peer=server-9456247c-bfd9-2bd4-11d2-24c85c2b248d
2026-02-28T09:03:15.262Z [WARN]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: rejecting vote request since we have a leader: from=dda082cf-9bcc-ea6a-85ae-f2406640d62e leader=d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9 leader-id=server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9
2026-02-28T09:03:15.262Z [INFO]  server-dda082cf-9bcc-ea6a-85ae-f2406640d62e: entering follower state: follower="Node at dda082cf-9bcc-ea6a-85ae-f2406640d62e [Follower]" leader-address=d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9 leader-id=server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9
2026-02-28T09:03:15.262Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: pipelining replication: peer="{Voter server-dda082cf-9bcc-ea6a-85ae-f2406640d62e dda082cf-9bcc-ea6a-85ae-f2406640d62e}"
2026-02-28T09:03:15.262Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: pipelining replication: peer="{Voter server-9456247c-bfd9-2bd4-11d2-24c85c2b248d 9456247c-bfd9-2bd4-11d2-24c85c2b248d}"
2026-02-28T09:03:16.267Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:15.262287142 +0000 UTC m=+73.226083681 (1 nodes), 570.77708ms from start of poll, 571.24656ms from cluster start. Timeout at 2026-02-28 09:03:16.267290982 +0000 UTC m=+74.231087521, 1.00500384s after stability
2026-02-28T09:03:16.268Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: starting snapshot up to: index=12
2026-02-28T09:03:16.268Z [INFO]  creating new snapshot: path=/tmp/raft2740226781/snapshots/2-12-1772269396268.tmp
2026-02-28T09:03:16.269Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: snapshot complete up to: index=12
2026-02-28T09:03:16.270Z [INFO]  creating new snapshot: path=/tmp/raft2740226781/snapshots/2-1013-1772269396270.tmp
2026-02-28T09:03:16.270Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: copied to local snapshot: bytes=71
2026-02-28T09:03:16.270Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: snapshot restore progress: id=2-1013-1772269396270 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:16.270Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: removing all old logs from log store
2026-02-28T09:03:16.270Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: compacting logs: from=1 to=22
2026-02-28T09:03:16.270Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: restored user snapshot: index=1013
2026-02-28T09:03:16.270Z [ERROR] server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: failed to get log: index=22 error="log not found"
2026-02-28T09:03:16.270Z [ERROR] server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: failed to get log: index=22 error="log not found"
2026-02-28T09:03:16.270Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: aborting pipeline replication: peer="{Voter server-dda082cf-9bcc-ea6a-85ae-f2406640d62e dda082cf-9bcc-ea6a-85ae-f2406640d62e}"
2026-02-28T09:03:16.270Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: aborting pipeline replication: peer="{Voter server-9456247c-bfd9-2bd4-11d2-24c85c2b248d 9456247c-bfd9-2bd4-11d2-24c85c2b248d}"
2026-02-28T09:03:16.279Z [ERROR] server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: failed to get log: index=22 error="log not found"
2026-02-28T09:03:16.280Z [INFO]  creating new snapshot: path=/tmp/raft2484992700/snapshots/2-1013-1772269396280.tmp
2026-02-28T09:03:16.280Z [INFO]  server-9456247c-bfd9-2bd4-11d2-24c85c2b248d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:16.280Z [INFO]  server-9456247c-bfd9-2bd4-11d2-24c85c2b248d: copied to local snapshot: bytes=71
2026-02-28T09:03:16.280Z [ERROR] server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: failed to get log: index=22 error="log not found"
2026-02-28T09:03:16.281Z [INFO]  server-9456247c-bfd9-2bd4-11d2-24c85c2b248d: snapshot restore progress: id=2-1013-1772269396280 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:16.281Z [INFO]  server-9456247c-bfd9-2bd4-11d2-24c85c2b248d: removing all old logs from log store
2026-02-28T09:03:16.281Z [INFO]  server-9456247c-bfd9-2bd4-11d2-24c85c2b248d: compacting logs: from=1 to=22
2026-02-28T09:03:16.281Z [INFO]  server-9456247c-bfd9-2bd4-11d2-24c85c2b248d: Installed remote snapshot
2026-02-28T09:03:16.281Z [WARN]  server-9456247c-bfd9-2bd4-11d2-24c85c2b248d: skipping application of old log: index=22
2026-02-28T09:03:16.281Z [INFO]  creating new snapshot: path=/tmp/raft3154519675/snapshots/2-1013-1772269396281.tmp
2026-02-28T09:03:16.281Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: pipelining replication: peer="{Voter server-9456247c-bfd9-2bd4-11d2-24c85c2b248d 9456247c-bfd9-2bd4-11d2-24c85c2b248d}"
2026-02-28T09:03:16.281Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:16.281Z [INFO]  server-dda082cf-9bcc-ea6a-85ae-f2406640d62e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:16.281Z [INFO]  server-dda082cf-9bcc-ea6a-85ae-f2406640d62e: copied to local snapshot: bytes=71
2026-02-28T09:03:16.282Z [INFO]  server-dda082cf-9bcc-ea6a-85ae-f2406640d62e: snapshot restore progress: id=2-1013-1772269396281 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:16.282Z [INFO]  server-dda082cf-9bcc-ea6a-85ae-f2406640d62e: removing all old logs from log store
2026-02-28T09:03:16.282Z [INFO]  server-dda082cf-9bcc-ea6a-85ae-f2406640d62e: compacting logs: from=1 to=22
2026-02-28T09:03:16.282Z [INFO]  server-dda082cf-9bcc-ea6a-85ae-f2406640d62e: Installed remote snapshot
2026-02-28T09:03:16.282Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: pipelining replication: peer="{Voter server-dda082cf-9bcc-ea6a-85ae-f2406640d62e dda082cf-9bcc-ea6a-85ae-f2406640d62e}"
2026-02-28T09:03:17.286Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:16.281430902 +0000 UTC m=+74.245227441 (2 nodes), 1.2µs from start of poll, 1.59039032s from cluster start. Timeout at 2026-02-28 09:03:17.286445422 +0000 UTC m=+75.250241961, 1.00501452s after stability
2026-02-28T09:03:17.297Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: aborting pipeline replication: peer="{Voter server-9456247c-bfd9-2bd4-11d2-24c85c2b248d 9456247c-bfd9-2bd4-11d2-24c85c2b248d}"
2026-02-28T09:03:17.297Z [INFO]  server-d5f6b1a0-3cc0-94e1-5c30-36baaebd35f9: aborting pipeline replication: peer="{Voter server-dda082cf-9bcc-ea6a-85ae-f2406640d62e dda082cf-9bcc-ea6a-85ae-f2406640d62e}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true
2026-02-28T09:03:17.299Z [INFO]  server-10d213ef-2a11-d9a6-bcf5-3a32e79e85aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-10d213ef-2a11-d9a6-bcf5-3a32e79e85aa Address:10d213ef-2a11-d9a6-bcf5-3a32e79e85aa} {Suffrage:Voter ID:server-eebe1dfc-d23f-dd9f-515f-e26bd6b7f43c Address:eebe1dfc-d23f-dd9f-515f-e26bd6b7f43c} {Suffrage:Voter ID:server-282ddc3e-c149-11e7-e886-bbca438611ce Address:282ddc3e-c149-11e7-e886-bbca438611ce}]"
2026-02-28T09:03:17.299Z [INFO]  server-eebe1dfc-d23f-dd9f-515f-e26bd6b7f43c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-10d213ef-2a11-d9a6-bcf5-3a32e79e85aa Address:10d213ef-2a11-d9a6-bcf5-3a32e79e85aa} {Suffrage:Voter ID:server-eebe1dfc-d23f-dd9f-515f-e26bd6b7f43c Address:eebe1dfc-d23f-dd9f-515f-e26bd6b7f43c} {Suffrage:Voter ID:server-282ddc3e-c149-11e7-e886-bbca438611ce Address:282ddc3e-c149-11e7-e886-bbca438611ce}]"
2026-02-28T09:03:17.299Z [INFO]  server-10d213ef-2a11-d9a6-bcf5-3a32e79e85aa: entering follower state: follower="Node at 10d213ef-2a11-d9a6-bcf5-3a32e79e85aa [Follower]" leader-address= leader-id=
2026-02-28T09:03:17.299Z [INFO]  server-eebe1dfc-d23f-dd9f-515f-e26bd6b7f43c: entering follower state: follower="Node at eebe1dfc-d23f-dd9f-515f-e26bd6b7f43c [Follower]" leader-address= leader-id=
2026-02-28T09:03:17.299Z [INFO]  server-282ddc3e-c149-11e7-e886-bbca438611ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-10d213ef-2a11-d9a6-bcf5-3a32e79e85aa Address:10d213ef-2a11-d9a6-bcf5-3a32e79e85aa} {Suffrage:Voter ID:server-eebe1dfc-d23f-dd9f-515f-e26bd6b7f43c Address:eebe1dfc-d23f-dd9f-515f-e26bd6b7f43c} {Suffrage:Voter ID:server-282ddc3e-c149-11e7-e886-bbca438611ce Address:282ddc3e-c149-11e7-e886-bbca438611ce}]"
2026-02-28T09:03:17.299Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:17.299Z [INFO]  server-282ddc3e-c149-11e7-e886-bbca438611ce: entering follower state: follower="Node at 282ddc3e-c149-11e7-e886-bbca438611ce [Follower]" leader-address= leader-id=
2026-02-28T09:03:17.825Z [WARN]  server-10d213ef-2a11-d9a6-bcf5-3a32e79e85aa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:17.825Z [INFO]  server-10d213ef-2a11-d9a6-bcf5-3a32e79e85aa: entering candidate state: node="Node at 10d213ef-2a11-d9a6-bcf5-3a32e79e85aa [Candidate]" term=2
2026-02-28T09:03:17.825Z [INFO]  server-10d213ef-2a11-d9a6-bcf5-3a32e79e85aa: election won: term=2 tally=2
2026-02-28T09:03:17.825Z [INFO]  server-10d213ef-2a11-d9a6-bcf5-3a32e79e85aa: entering leader state: leader="Node at 10d213ef-2a11-d9a6-bcf5-3a32e79e85aa [Leader]"
2026-02-28T09:03:17.826Z [INFO]  server-10d213ef-2a11-d9a6-bcf5-3a32e79e85aa: added peer, starting replication: peer=server-eebe1dfc-d23f-dd9f-515f-e26bd6b7f43c
2026-02-28T09:03:17.826Z [INFO]  server-10d213ef-2a11-d9a6-bcf5-3a32e79e85aa: added peer, starting replication: peer=server-282ddc3e-c149-11e7-e886-bbca438611ce
2026-02-28T09:03:17.826Z [INFO]  server-10d213ef-2a11-d9a6-bcf5-3a32e79e85aa: pipelining replication: peer="{Voter server-eebe1dfc-d23f-dd9f-515f-e26bd6b7f43c eebe1dfc-d23f-dd9f-515f-e26bd6b7f43c}"
2026-02-28T09:03:17.826Z [INFO]  server-10d213ef-2a11-d9a6-bcf5-3a32e79e85aa: pipelining replication: peer="{Voter server-282ddc3e-c149-11e7-e886-bbca438611ce 282ddc3e-c149-11e7-e886-bbca438611ce}"
2026-02-28T09:03:18.832Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:17.826009622 +0000 UTC m=+75.789806161 (1 nodes), 526.18956ms from start of poll, 526.75732ms from cluster start. Timeout at 2026-02-28 09:03:18.831022182 +0000 UTC m=+76.794818721, 1.00501256s after stability
2026-02-28T09:03:18.832Z [INFO]  server-10d213ef-2a11-d9a6-bcf5-3a32e79e85aa: starting snapshot up to: index=12
2026-02-28T09:03:18.832Z [INFO]  creating new snapshot: path=/tmp/raft3869732531/snapshots/2-12-1772269398832.tmp
2026-02-28T09:03:18.833Z [INFO]  server-10d213ef-2a11-d9a6-bcf5-3a32e79e85aa: snapshot complete up to: index=12
2026-02-28T09:03:18.834Z [INFO]  server-ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6 Address:ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6} {Suffrage:Voter ID:server-38d57142-9eb6-281d-b660-561c05b0d7e4 Address:38d57142-9eb6-281d-b660-561c05b0d7e4} {Suffrage:Voter ID:server-2c4d76c4-b56a-313b-8158-d856a98417b8 Address:2c4d76c4-b56a-313b-8158-d856a98417b8}]"
2026-02-28T09:03:18.834Z [INFO]  server-38d57142-9eb6-281d-b660-561c05b0d7e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6 Address:ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6} {Suffrage:Voter ID:server-38d57142-9eb6-281d-b660-561c05b0d7e4 Address:38d57142-9eb6-281d-b660-561c05b0d7e4} {Suffrage:Voter ID:server-2c4d76c4-b56a-313b-8158-d856a98417b8 Address:2c4d76c4-b56a-313b-8158-d856a98417b8}]"
2026-02-28T09:03:18.834Z [INFO]  server-ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6: entering follower state: follower="Node at ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6 [Follower]" leader-address= leader-id=
2026-02-28T09:03:18.834Z [INFO]  server-38d57142-9eb6-281d-b660-561c05b0d7e4: entering follower state: follower="Node at 38d57142-9eb6-281d-b660-561c05b0d7e4 [Follower]" leader-address= leader-id=
2026-02-28T09:03:18.834Z [INFO]  server-2c4d76c4-b56a-313b-8158-d856a98417b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6 Address:ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6} {Suffrage:Voter ID:server-38d57142-9eb6-281d-b660-561c05b0d7e4 Address:38d57142-9eb6-281d-b660-561c05b0d7e4} {Suffrage:Voter ID:server-2c4d76c4-b56a-313b-8158-d856a98417b8 Address:2c4d76c4-b56a-313b-8158-d856a98417b8}]"
2026-02-28T09:03:18.834Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:18.834Z [INFO]  server-2c4d76c4-b56a-313b-8158-d856a98417b8: entering follower state: follower="Node at 2c4d76c4-b56a-313b-8158-d856a98417b8 [Follower]" leader-address= leader-id=
2026-02-28T09:03:19.476Z [WARN]  server-2c4d76c4-b56a-313b-8158-d856a98417b8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:19.476Z [INFO]  server-2c4d76c4-b56a-313b-8158-d856a98417b8: entering candidate state: node="Node at 2c4d76c4-b56a-313b-8158-d856a98417b8 [Candidate]" term=2
2026-02-28T09:03:19.477Z [INFO]  server-2c4d76c4-b56a-313b-8158-d856a98417b8: election won: term=2 tally=2
2026-02-28T09:03:19.477Z [INFO]  server-2c4d76c4-b56a-313b-8158-d856a98417b8: entering leader state: leader="Node at 2c4d76c4-b56a-313b-8158-d856a98417b8 [Leader]"
2026-02-28T09:03:19.477Z [INFO]  server-2c4d76c4-b56a-313b-8158-d856a98417b8: added peer, starting replication: peer=server-ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6
2026-02-28T09:03:19.477Z [INFO]  server-2c4d76c4-b56a-313b-8158-d856a98417b8: added peer, starting replication: peer=server-38d57142-9eb6-281d-b660-561c05b0d7e4
2026-02-28T09:03:19.477Z [INFO]  server-2c4d76c4-b56a-313b-8158-d856a98417b8: pipelining replication: peer="{Voter server-38d57142-9eb6-281d-b660-561c05b0d7e4 38d57142-9eb6-281d-b660-561c05b0d7e4}"
2026-02-28T09:03:19.477Z [INFO]  server-2c4d76c4-b56a-313b-8158-d856a98417b8: pipelining replication: peer="{Voter server-ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6 ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6}"
2026-02-28T09:03:20.482Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:19.477143502 +0000 UTC m=+77.440940041 (1 nodes), 642.67164ms from start of poll, 643.12812ms from cluster start. Timeout at 2026-02-28 09:03:20.482144662 +0000 UTC m=+78.445941201, 1.00500116s after stability
2026-02-28T09:03:20.482Z [INFO]  creating new snapshot: path=/tmp/raft336684456/snapshots/2-1013-1772269400482.tmp
2026-02-28T09:03:20.482Z [INFO]  server-2c4d76c4-b56a-313b-8158-d856a98417b8: copied to local snapshot: bytes=71
2026-02-28T09:03:20.483Z [INFO]  server-2c4d76c4-b56a-313b-8158-d856a98417b8: snapshot restore progress: id=2-1013-1772269400482 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:20.483Z [INFO]  server-2c4d76c4-b56a-313b-8158-d856a98417b8: restored user snapshot: index=1013
2026-02-28T09:03:20.483Z [ERROR] server-2c4d76c4-b56a-313b-8158-d856a98417b8: failed to get log: index=3 error="log not found"
2026-02-28T09:03:20.483Z [INFO]  server-2c4d76c4-b56a-313b-8158-d856a98417b8: aborting pipeline replication: peer="{Voter server-ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6 ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6}"
2026-02-28T09:03:20.483Z [ERROR] server-2c4d76c4-b56a-313b-8158-d856a98417b8: failed to get log: index=3 error="log not found"
2026-02-28T09:03:20.483Z [INFO]  server-2c4d76c4-b56a-313b-8158-d856a98417b8: aborting pipeline replication: peer="{Voter server-38d57142-9eb6-281d-b660-561c05b0d7e4 38d57142-9eb6-281d-b660-561c05b0d7e4}"
2026-02-28T09:03:20.492Z [ERROR] server-2c4d76c4-b56a-313b-8158-d856a98417b8: failed to get log: index=3 error="log not found"
2026-02-28T09:03:20.492Z [ERROR] server-2c4d76c4-b56a-313b-8158-d856a98417b8: failed to get log: index=3 error="log not found"
2026-02-28T09:03:20.492Z [INFO]  creating new snapshot: path=/tmp/raft1594988679/snapshots/2-1013-1772269400492.tmp
2026-02-28T09:03:20.492Z [INFO]  creating new snapshot: path=/tmp/raft3249314332/snapshots/2-1013-1772269400492.tmp
2026-02-28T09:03:20.492Z [INFO]  server-38d57142-9eb6-281d-b660-561c05b0d7e4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:20.492Z [INFO]  server-ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:20.493Z [INFO]  server-38d57142-9eb6-281d-b660-561c05b0d7e4: copied to local snapshot: bytes=71
2026-02-28T09:03:20.493Z [INFO]  server-ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6: copied to local snapshot: bytes=71
2026-02-28T09:03:20.493Z [INFO]  server-38d57142-9eb6-281d-b660-561c05b0d7e4: snapshot restore progress: id=2-1013-1772269400492 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:20.493Z [INFO]  server-38d57142-9eb6-281d-b660-561c05b0d7e4: Installed remote snapshot
2026-02-28T09:03:20.493Z [INFO]  server-2c4d76c4-b56a-313b-8158-d856a98417b8: pipelining replication: peer="{Voter server-38d57142-9eb6-281d-b660-561c05b0d7e4 38d57142-9eb6-281d-b660-561c05b0d7e4}"
2026-02-28T09:03:20.493Z [INFO]  server-ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6: snapshot restore progress: id=2-1013-1772269400492 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:20.493Z [INFO]  server-ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6: Installed remote snapshot
2026-02-28T09:03:20.493Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:20.493Z [INFO]  server-2c4d76c4-b56a-313b-8158-d856a98417b8: pipelining replication: peer="{Voter server-ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6 ad4d7d07-bb71-ea6b-5f91-2cde057a3ae6}"
2026-02-28T09:03:21.499Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:20.493788662 +0000 UTC m=+78.457585201 (2 nodes), 960ns from start of poll, 1.65977328s from cluster start. Timeout at 2026-02-28 09:03:21.498797382 +0000 UTC m=+79.462593921, 1.00500872s after stability
2026-02-28T09:03:21.510Z [INFO]  server-10d213ef-2a11-d9a6-bcf5-3a32e79e85aa: aborting pipeline replication: peer="{Voter server-282ddc3e-c149-11e7-e886-bbca438611ce 282ddc3e-c149-11e7-e886-bbca438611ce}"
2026-02-28T09:03:21.510Z [INFO]  server-10d213ef-2a11-d9a6-bcf5-3a32e79e85aa: aborting pipeline replication: peer="{Voter server-eebe1dfc-d23f-dd9f-515f-e26bd6b7f43c eebe1dfc-d23f-dd9f-515f-e26bd6b7f43c}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true
2026-02-28T09:03:21.511Z [INFO]  server-0fe83323-9eb7-514c-969f-2565ca26a424: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0fe83323-9eb7-514c-969f-2565ca26a424 Address:0fe83323-9eb7-514c-969f-2565ca26a424} {Suffrage:Voter ID:server-d46aa3c5-12c6-972c-0f02-ce50a096ca1a Address:d46aa3c5-12c6-972c-0f02-ce50a096ca1a} {Suffrage:Voter ID:server-22b94053-33a4-2c2f-5b57-8bb22f54d210 Address:22b94053-33a4-2c2f-5b57-8bb22f54d210}]"
2026-02-28T09:03:21.511Z [INFO]  server-d46aa3c5-12c6-972c-0f02-ce50a096ca1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0fe83323-9eb7-514c-969f-2565ca26a424 Address:0fe83323-9eb7-514c-969f-2565ca26a424} {Suffrage:Voter ID:server-d46aa3c5-12c6-972c-0f02-ce50a096ca1a Address:d46aa3c5-12c6-972c-0f02-ce50a096ca1a} {Suffrage:Voter ID:server-22b94053-33a4-2c2f-5b57-8bb22f54d210 Address:22b94053-33a4-2c2f-5b57-8bb22f54d210}]"
2026-02-28T09:03:21.511Z [INFO]  server-0fe83323-9eb7-514c-969f-2565ca26a424: entering follower state: follower="Node at 0fe83323-9eb7-514c-969f-2565ca26a424 [Follower]" leader-address= leader-id=
2026-02-28T09:03:21.511Z [INFO]  server-d46aa3c5-12c6-972c-0f02-ce50a096ca1a: entering follower state: follower="Node at d46aa3c5-12c6-972c-0f02-ce50a096ca1a [Follower]" leader-address= leader-id=
2026-02-28T09:03:21.511Z [INFO]  server-22b94053-33a4-2c2f-5b57-8bb22f54d210: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0fe83323-9eb7-514c-969f-2565ca26a424 Address:0fe83323-9eb7-514c-969f-2565ca26a424} {Suffrage:Voter ID:server-d46aa3c5-12c6-972c-0f02-ce50a096ca1a Address:d46aa3c5-12c6-972c-0f02-ce50a096ca1a} {Suffrage:Voter ID:server-22b94053-33a4-2c2f-5b57-8bb22f54d210 Address:22b94053-33a4-2c2f-5b57-8bb22f54d210}]"
2026-02-28T09:03:21.511Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:21.511Z [INFO]  server-22b94053-33a4-2c2f-5b57-8bb22f54d210: entering follower state: follower="Node at 22b94053-33a4-2c2f-5b57-8bb22f54d210 [Follower]" leader-address= leader-id=
2026-02-28T09:03:22.019Z [WARN]  server-22b94053-33a4-2c2f-5b57-8bb22f54d210: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:22.019Z [INFO]  server-22b94053-33a4-2c2f-5b57-8bb22f54d210: entering candidate state: node="Node at 22b94053-33a4-2c2f-5b57-8bb22f54d210 [Candidate]" term=2
2026-02-28T09:03:22.019Z [WARN]  server-0fe83323-9eb7-514c-969f-2565ca26a424: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:22.019Z [INFO]  server-22b94053-33a4-2c2f-5b57-8bb22f54d210: election won: term=2 tally=2
2026-02-28T09:03:22.019Z [INFO]  server-0fe83323-9eb7-514c-969f-2565ca26a424: entering candidate state: node="Node at 0fe83323-9eb7-514c-969f-2565ca26a424 [Candidate]" term=2
2026-02-28T09:03:22.019Z [INFO]  server-22b94053-33a4-2c2f-5b57-8bb22f54d210: entering leader state: leader="Node at 22b94053-33a4-2c2f-5b57-8bb22f54d210 [Leader]"
2026-02-28T09:03:22.019Z [INFO]  server-22b94053-33a4-2c2f-5b57-8bb22f54d210: added peer, starting replication: peer=server-0fe83323-9eb7-514c-969f-2565ca26a424
2026-02-28T09:03:22.019Z [INFO]  server-0fe83323-9eb7-514c-969f-2565ca26a424: duplicate requestVote for same term: term=2
2026-02-28T09:03:22.019Z [INFO]  server-22b94053-33a4-2c2f-5b57-8bb22f54d210: added peer, starting replication: peer=server-d46aa3c5-12c6-972c-0f02-ce50a096ca1a
2026-02-28T09:03:22.019Z [INFO]  server-d46aa3c5-12c6-972c-0f02-ce50a096ca1a: duplicate requestVote for same term: term=2
2026-02-28T09:03:22.019Z [WARN]  server-22b94053-33a4-2c2f-5b57-8bb22f54d210: rejecting vote request since we have a leader: from=0fe83323-9eb7-514c-969f-2565ca26a424 leader=22b94053-33a4-2c2f-5b57-8bb22f54d210 leader-id=server-22b94053-33a4-2c2f-5b57-8bb22f54d210
2026-02-28T09:03:22.020Z [INFO]  server-0fe83323-9eb7-514c-969f-2565ca26a424: entering follower state: follower="Node at 0fe83323-9eb7-514c-969f-2565ca26a424 [Follower]" leader-address=22b94053-33a4-2c2f-5b57-8bb22f54d210 leader-id=server-22b94053-33a4-2c2f-5b57-8bb22f54d210
2026-02-28T09:03:22.020Z [INFO]  server-22b94053-33a4-2c2f-5b57-8bb22f54d210: pipelining replication: peer="{Voter server-d46aa3c5-12c6-972c-0f02-ce50a096ca1a d46aa3c5-12c6-972c-0f02-ce50a096ca1a}"
2026-02-28T09:03:22.020Z [INFO]  server-22b94053-33a4-2c2f-5b57-8bb22f54d210: pipelining replication: peer="{Voter server-0fe83323-9eb7-514c-969f-2565ca26a424 0fe83323-9eb7-514c-969f-2565ca26a424}"
2026-02-28T09:03:23.025Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:22.020130382 +0000 UTC m=+79.983926961 (1 nodes), 508.1856ms from start of poll, 508.65348ms from cluster start. Timeout at 2026-02-28 09:03:23.025137622 +0000 UTC m=+80.988934161, 1.0050072s after stability
2026-02-28T09:03:23.026Z [INFO]  server-22b94053-33a4-2c2f-5b57-8bb22f54d210: starting snapshot up to: index=12
2026-02-28T09:03:23.026Z [INFO]  creating new snapshot: path=/tmp/raft1874140486/snapshots/2-12-1772269403026.tmp
2026-02-28T09:03:23.027Z [INFO]  server-22b94053-33a4-2c2f-5b57-8bb22f54d210: snapshot complete up to: index=12
2026-02-28T09:03:23.028Z [INFO]  server-70a5cfb7-b897-8539-c52e-f6134c195895: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70a5cfb7-b897-8539-c52e-f6134c195895 Address:70a5cfb7-b897-8539-c52e-f6134c195895} {Suffrage:Voter ID:server-ef2162f2-0304-4362-45b2-f1c20c35675b Address:ef2162f2-0304-4362-45b2-f1c20c35675b} {Suffrage:Voter ID:server-785b9d51-c029-3195-4ac5-0c8d6bc8a568 Address:785b9d51-c029-3195-4ac5-0c8d6bc8a568}]"
2026-02-28T09:03:23.028Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70a5cfb7-b897-8539-c52e-f6134c195895 Address:70a5cfb7-b897-8539-c52e-f6134c195895} {Suffrage:Voter ID:server-ef2162f2-0304-4362-45b2-f1c20c35675b Address:ef2162f2-0304-4362-45b2-f1c20c35675b} {Suffrage:Voter ID:server-785b9d51-c029-3195-4ac5-0c8d6bc8a568 Address:785b9d51-c029-3195-4ac5-0c8d6bc8a568}]"
2026-02-28T09:03:23.028Z [INFO]  server-70a5cfb7-b897-8539-c52e-f6134c195895: entering follower state: follower="Node at 70a5cfb7-b897-8539-c52e-f6134c195895 [Follower]" leader-address= leader-id=
2026-02-28T09:03:23.028Z [INFO]  server-785b9d51-c029-3195-4ac5-0c8d6bc8a568: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70a5cfb7-b897-8539-c52e-f6134c195895 Address:70a5cfb7-b897-8539-c52e-f6134c195895} {Suffrage:Voter ID:server-ef2162f2-0304-4362-45b2-f1c20c35675b Address:ef2162f2-0304-4362-45b2-f1c20c35675b} {Suffrage:Voter ID:server-785b9d51-c029-3195-4ac5-0c8d6bc8a568 Address:785b9d51-c029-3195-4ac5-0c8d6bc8a568}]"
2026-02-28T09:03:23.028Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: entering follower state: follower="Node at ef2162f2-0304-4362-45b2-f1c20c35675b [Follower]" leader-address= leader-id=
2026-02-28T09:03:23.028Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:23.028Z [INFO]  server-785b9d51-c029-3195-4ac5-0c8d6bc8a568: entering follower state: follower="Node at 785b9d51-c029-3195-4ac5-0c8d6bc8a568 [Follower]" leader-address= leader-id=
2026-02-28T09:03:23.634Z [WARN]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:23.634Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: entering candidate state: node="Node at ef2162f2-0304-4362-45b2-f1c20c35675b [Candidate]" term=2
2026-02-28T09:03:23.634Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: election won: term=2 tally=2
2026-02-28T09:03:23.634Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: entering leader state: leader="Node at ef2162f2-0304-4362-45b2-f1c20c35675b [Leader]"
2026-02-28T09:03:23.634Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: added peer, starting replication: peer=server-70a5cfb7-b897-8539-c52e-f6134c195895
2026-02-28T09:03:23.634Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: added peer, starting replication: peer=server-785b9d51-c029-3195-4ac5-0c8d6bc8a568
2026-02-28T09:03:23.635Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: pipelining replication: peer="{Voter server-785b9d51-c029-3195-4ac5-0c8d6bc8a568 785b9d51-c029-3195-4ac5-0c8d6bc8a568}"
2026-02-28T09:03:23.635Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: pipelining replication: peer="{Voter server-70a5cfb7-b897-8539-c52e-f6134c195895 70a5cfb7-b897-8539-c52e-f6134c195895}"
2026-02-28T09:03:24.640Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:23.634898782 +0000 UTC m=+81.598695321 (1 nodes), 606.36044ms from start of poll, 606.85196ms from cluster start. Timeout at 2026-02-28 09:03:24.639902382 +0000 UTC m=+82.603698921, 1.0050036s after stability
2026-02-28T09:03:24.640Z [INFO]  creating new snapshot: path=/tmp/raft3682608301/snapshots/2-1013-1772269404640.tmp
2026-02-28T09:03:24.641Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: copied to local snapshot: bytes=71
2026-02-28T09:03:24.641Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: snapshot restore progress: id=2-1013-1772269404640 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:24.641Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: removing all old logs from log store
2026-02-28T09:03:24.641Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: compacting logs: from=1 to=2
2026-02-28T09:03:24.641Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: restored user snapshot: index=1013
2026-02-28T09:03:24.641Z [ERROR] server-ef2162f2-0304-4362-45b2-f1c20c35675b: failed to get log: index=2 error="log not found"
2026-02-28T09:03:24.641Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: aborting pipeline replication: peer="{Voter server-785b9d51-c029-3195-4ac5-0c8d6bc8a568 785b9d51-c029-3195-4ac5-0c8d6bc8a568}"
2026-02-28T09:03:24.641Z [ERROR] server-ef2162f2-0304-4362-45b2-f1c20c35675b: failed to get log: index=2 error="log not found"
2026-02-28T09:03:24.641Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: aborting pipeline replication: peer="{Voter server-70a5cfb7-b897-8539-c52e-f6134c195895 70a5cfb7-b897-8539-c52e-f6134c195895}"
2026-02-28T09:03:24.649Z [ERROR] server-ef2162f2-0304-4362-45b2-f1c20c35675b: failed to get log: index=2 error="log not found"
2026-02-28T09:03:24.649Z [ERROR] server-ef2162f2-0304-4362-45b2-f1c20c35675b: failed to get log: index=2 error="log not found"
2026-02-28T09:03:24.649Z [INFO]  creating new snapshot: path=/tmp/raft1633669397/snapshots/2-1013-1772269404649.tmp
2026-02-28T09:03:24.649Z [INFO]  creating new snapshot: path=/tmp/raft2607350067/snapshots/2-1013-1772269404649.tmp
2026-02-28T09:03:24.650Z [INFO]  server-785b9d51-c029-3195-4ac5-0c8d6bc8a568: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:24.650Z [INFO]  server-70a5cfb7-b897-8539-c52e-f6134c195895: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:24.650Z [INFO]  server-785b9d51-c029-3195-4ac5-0c8d6bc8a568: copied to local snapshot: bytes=71
2026-02-28T09:03:24.650Z [INFO]  server-70a5cfb7-b897-8539-c52e-f6134c195895: copied to local snapshot: bytes=71
2026-02-28T09:03:24.650Z [INFO]  server-785b9d51-c029-3195-4ac5-0c8d6bc8a568: snapshot restore progress: id=2-1013-1772269404649 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:24.650Z [INFO]  server-785b9d51-c029-3195-4ac5-0c8d6bc8a568: removing all old logs from log store
2026-02-28T09:03:24.650Z [INFO]  server-785b9d51-c029-3195-4ac5-0c8d6bc8a568: compacting logs: from=1 to=2
2026-02-28T09:03:24.650Z [INFO]  server-785b9d51-c029-3195-4ac5-0c8d6bc8a568: Installed remote snapshot
2026-02-28T09:03:24.650Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: pipelining replication: peer="{Voter server-785b9d51-c029-3195-4ac5-0c8d6bc8a568 785b9d51-c029-3195-4ac5-0c8d6bc8a568}"
2026-02-28T09:03:24.650Z [INFO]  server-70a5cfb7-b897-8539-c52e-f6134c195895: snapshot restore progress: id=2-1013-1772269404649 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:24.650Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:24.651Z [INFO]  server-70a5cfb7-b897-8539-c52e-f6134c195895: removing all old logs from log store
2026-02-28T09:03:24.651Z [INFO]  server-70a5cfb7-b897-8539-c52e-f6134c195895: compacting logs: from=1 to=2
2026-02-28T09:03:24.651Z [INFO]  server-70a5cfb7-b897-8539-c52e-f6134c195895: Installed remote snapshot
2026-02-28T09:03:24.651Z [INFO]  server-ef2162f2-0304-4362-45b2-f1c20c35675b: pipelining replication: peer="{Voter server-70a5cfb7-b897-8539-c52e-f6134c195895 70a5cfb7-b897-8539-c52e-f6134c195895}"
2026-02-28T09:03:25.656Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:24.650982142 +0000 UTC m=+82.614778681 (2 nodes), 1.04µs from start of poll, 1.62293532s from cluster start. Timeout at 2026-02-28 09:03:25.656004462 +0000 UTC m=+83.619801001, 1.00502232s after stability
2026-02-28T09:03:25.667Z [INFO]  server-22b94053-33a4-2c2f-5b57-8bb22f54d210: aborting pipeline replication: peer="{Voter server-d46aa3c5-12c6-972c-0f02-ce50a096ca1a d46aa3c5-12c6-972c-0f02-ce50a096ca1a}"
2026-02-28T09:03:25.667Z [INFO]  server-22b94053-33a4-2c2f-5b57-8bb22f54d210: aborting pipeline replication: peer="{Voter server-0fe83323-9eb7-514c-969f-2565ca26a424 0fe83323-9eb7-514c-969f-2565ca26a424}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false
2026-02-28T09:03:25.668Z [INFO]  server-c4516975-bdec-a711-9e3e-a0f961c01942: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c4516975-bdec-a711-9e3e-a0f961c01942 Address:c4516975-bdec-a711-9e3e-a0f961c01942} {Suffrage:Voter ID:server-5d11258e-f81a-bf36-f9db-7712b55506f3 Address:5d11258e-f81a-bf36-f9db-7712b55506f3} {Suffrage:Voter ID:server-89c89b33-579f-f8de-69a8-83cf63816c8b Address:89c89b33-579f-f8de-69a8-83cf63816c8b}]"
2026-02-28T09:03:25.668Z [INFO]  server-5d11258e-f81a-bf36-f9db-7712b55506f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c4516975-bdec-a711-9e3e-a0f961c01942 Address:c4516975-bdec-a711-9e3e-a0f961c01942} {Suffrage:Voter ID:server-5d11258e-f81a-bf36-f9db-7712b55506f3 Address:5d11258e-f81a-bf36-f9db-7712b55506f3} {Suffrage:Voter ID:server-89c89b33-579f-f8de-69a8-83cf63816c8b Address:89c89b33-579f-f8de-69a8-83cf63816c8b}]"
2026-02-28T09:03:25.668Z [INFO]  server-c4516975-bdec-a711-9e3e-a0f961c01942: entering follower state: follower="Node at c4516975-bdec-a711-9e3e-a0f961c01942 [Follower]" leader-address= leader-id=
2026-02-28T09:03:25.668Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c4516975-bdec-a711-9e3e-a0f961c01942 Address:c4516975-bdec-a711-9e3e-a0f961c01942} {Suffrage:Voter ID:server-5d11258e-f81a-bf36-f9db-7712b55506f3 Address:5d11258e-f81a-bf36-f9db-7712b55506f3} {Suffrage:Voter ID:server-89c89b33-579f-f8de-69a8-83cf63816c8b Address:89c89b33-579f-f8de-69a8-83cf63816c8b}]"
2026-02-28T09:03:25.668Z [INFO]  server-5d11258e-f81a-bf36-f9db-7712b55506f3: entering follower state: follower="Node at 5d11258e-f81a-bf36-f9db-7712b55506f3 [Follower]" leader-address= leader-id=
2026-02-28T09:03:25.668Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:25.668Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: entering follower state: follower="Node at 89c89b33-579f-f8de-69a8-83cf63816c8b [Follower]" leader-address= leader-id=
2026-02-28T09:03:26.223Z [WARN]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:26.223Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: entering candidate state: node="Node at 89c89b33-579f-f8de-69a8-83cf63816c8b [Candidate]" term=2
2026-02-28T09:03:26.223Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: election won: term=2 tally=2
2026-02-28T09:03:26.223Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: entering leader state: leader="Node at 89c89b33-579f-f8de-69a8-83cf63816c8b [Leader]"
2026-02-28T09:03:26.223Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: added peer, starting replication: peer=server-c4516975-bdec-a711-9e3e-a0f961c01942
2026-02-28T09:03:26.223Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: added peer, starting replication: peer=server-5d11258e-f81a-bf36-f9db-7712b55506f3
2026-02-28T09:03:26.224Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: pipelining replication: peer="{Voter server-c4516975-bdec-a711-9e3e-a0f961c01942 c4516975-bdec-a711-9e3e-a0f961c01942}"
2026-02-28T09:03:26.224Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: pipelining replication: peer="{Voter server-5d11258e-f81a-bf36-f9db-7712b55506f3 5d11258e-f81a-bf36-f9db-7712b55506f3}"
2026-02-28T09:03:27.228Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:26.223926542 +0000 UTC m=+84.187723081 (1 nodes), 555.23896ms from start of poll, 555.73072ms from cluster start. Timeout at 2026-02-28 09:03:27.228931142 +0000 UTC m=+85.192727681, 1.0050046s after stability
2026-02-28T09:03:27.229Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: starting snapshot up to: index=12
2026-02-28T09:03:27.229Z [INFO]  creating new snapshot: path=/tmp/raft2919474395/snapshots/2-12-1772269407229.tmp
2026-02-28T09:03:27.230Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: snapshot complete up to: index=12
2026-02-28T09:03:27.230Z [INFO]  creating new snapshot: path=/tmp/raft2919474395/snapshots/2-10013-1772269407230.tmp
2026-02-28T09:03:27.231Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: copied to local snapshot: bytes=71
2026-02-28T09:03:27.231Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: snapshot restore progress: id=2-10013-1772269407230 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:27.231Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: restored user snapshot: index=10013
2026-02-28T09:03:27.231Z [ERROR] server-89c89b33-579f-f8de-69a8-83cf63816c8b: failed to get log: index=23 error="log not found"
2026-02-28T09:03:27.231Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: aborting pipeline replication: peer="{Voter server-c4516975-bdec-a711-9e3e-a0f961c01942 c4516975-bdec-a711-9e3e-a0f961c01942}"
2026-02-28T09:03:27.231Z [ERROR] server-89c89b33-579f-f8de-69a8-83cf63816c8b: failed to get log: index=23 error="log not found"
2026-02-28T09:03:27.231Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: aborting pipeline replication: peer="{Voter server-5d11258e-f81a-bf36-f9db-7712b55506f3 5d11258e-f81a-bf36-f9db-7712b55506f3}"
2026-02-28T09:03:27.239Z [ERROR] server-89c89b33-579f-f8de-69a8-83cf63816c8b: failed to get log: index=23 error="log not found"
2026-02-28T09:03:27.239Z [INFO]  creating new snapshot: path=/tmp/raft1901506628/snapshots/2-10013-1772269407239.tmp
2026-02-28T09:03:27.239Z [ERROR] server-89c89b33-579f-f8de-69a8-83cf63816c8b: failed to get log: index=23 error="log not found"
2026-02-28T09:03:27.239Z [INFO]  server-5d11258e-f81a-bf36-f9db-7712b55506f3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:27.240Z [INFO]  server-5d11258e-f81a-bf36-f9db-7712b55506f3: copied to local snapshot: bytes=71
2026-02-28T09:03:27.240Z [INFO]  creating new snapshot: path=/tmp/raft2233326442/snapshots/2-10013-1772269407240.tmp
2026-02-28T09:03:27.240Z [INFO]  server-5d11258e-f81a-bf36-f9db-7712b55506f3: snapshot restore progress: id=2-10013-1772269407239 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:27.240Z [INFO]  server-5d11258e-f81a-bf36-f9db-7712b55506f3: Installed remote snapshot
2026-02-28T09:03:27.240Z [INFO]  server-c4516975-bdec-a711-9e3e-a0f961c01942: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:27.240Z [WARN]  server-5d11258e-f81a-bf36-f9db-7712b55506f3: skipping application of old log: index=22
2026-02-28T09:03:27.240Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: pipelining replication: peer="{Voter server-5d11258e-f81a-bf36-f9db-7712b55506f3 5d11258e-f81a-bf36-f9db-7712b55506f3}"
2026-02-28T09:03:27.240Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:27.240Z [INFO]  server-c4516975-bdec-a711-9e3e-a0f961c01942: copied to local snapshot: bytes=71
2026-02-28T09:03:27.240Z [INFO]  server-c4516975-bdec-a711-9e3e-a0f961c01942: snapshot restore progress: id=2-10013-1772269407240 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:27.241Z [INFO]  server-c4516975-bdec-a711-9e3e-a0f961c01942: Installed remote snapshot
2026-02-28T09:03:27.241Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: pipelining replication: peer="{Voter server-c4516975-bdec-a711-9e3e-a0f961c01942 c4516975-bdec-a711-9e3e-a0f961c01942}"
2026-02-28T09:03:28.246Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:27.240681222 +0000 UTC m=+85.204477761 (2 nodes), 1.44µs from start of poll, 1.5724854s from cluster start. Timeout at 2026-02-28 09:03:28.245696022 +0000 UTC m=+86.209492561, 1.0050148s after stability
2026-02-28T09:03:28.257Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: aborting pipeline replication: peer="{Voter server-c4516975-bdec-a711-9e3e-a0f961c01942 c4516975-bdec-a711-9e3e-a0f961c01942}"
2026-02-28T09:03:28.257Z [INFO]  server-89c89b33-579f-f8de-69a8-83cf63816c8b: aborting pipeline replication: peer="{Voter server-5d11258e-f81a-bf36-f9db-7712b55506f3 5d11258e-f81a-bf36-f9db-7712b55506f3}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false
2026-02-28T09:03:28.258Z [INFO]  server-b215ae0e-c3ea-eefd-0999-fb9d91378e03: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b215ae0e-c3ea-eefd-0999-fb9d91378e03 Address:b215ae0e-c3ea-eefd-0999-fb9d91378e03} {Suffrage:Voter ID:server-6ce4e820-cbae-7fd1-80f6-b9f0485e8a53 Address:6ce4e820-cbae-7fd1-80f6-b9f0485e8a53} {Suffrage:Voter ID:server-e09ace10-e926-4639-7f9e-a098a2b0575c Address:e09ace10-e926-4639-7f9e-a098a2b0575c}]"
2026-02-28T09:03:28.258Z [INFO]  server-6ce4e820-cbae-7fd1-80f6-b9f0485e8a53: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b215ae0e-c3ea-eefd-0999-fb9d91378e03 Address:b215ae0e-c3ea-eefd-0999-fb9d91378e03} {Suffrage:Voter ID:server-6ce4e820-cbae-7fd1-80f6-b9f0485e8a53 Address:6ce4e820-cbae-7fd1-80f6-b9f0485e8a53} {Suffrage:Voter ID:server-e09ace10-e926-4639-7f9e-a098a2b0575c Address:e09ace10-e926-4639-7f9e-a098a2b0575c}]"
2026-02-28T09:03:28.258Z [INFO]  server-b215ae0e-c3ea-eefd-0999-fb9d91378e03: entering follower state: follower="Node at b215ae0e-c3ea-eefd-0999-fb9d91378e03 [Follower]" leader-address= leader-id=
2026-02-28T09:03:28.258Z [INFO]  server-6ce4e820-cbae-7fd1-80f6-b9f0485e8a53: entering follower state: follower="Node at 6ce4e820-cbae-7fd1-80f6-b9f0485e8a53 [Follower]" leader-address= leader-id=
2026-02-28T09:03:28.259Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b215ae0e-c3ea-eefd-0999-fb9d91378e03 Address:b215ae0e-c3ea-eefd-0999-fb9d91378e03} {Suffrage:Voter ID:server-6ce4e820-cbae-7fd1-80f6-b9f0485e8a53 Address:6ce4e820-cbae-7fd1-80f6-b9f0485e8a53} {Suffrage:Voter ID:server-e09ace10-e926-4639-7f9e-a098a2b0575c Address:e09ace10-e926-4639-7f9e-a098a2b0575c}]"
2026-02-28T09:03:28.259Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:28.259Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: entering follower state: follower="Node at e09ace10-e926-4639-7f9e-a098a2b0575c [Follower]" leader-address= leader-id=
2026-02-28T09:03:29.066Z [WARN]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:29.066Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: entering candidate state: node="Node at e09ace10-e926-4639-7f9e-a098a2b0575c [Candidate]" term=2
2026-02-28T09:03:29.066Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: election won: term=2 tally=2
2026-02-28T09:03:29.066Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: entering leader state: leader="Node at e09ace10-e926-4639-7f9e-a098a2b0575c [Leader]"
2026-02-28T09:03:29.066Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: added peer, starting replication: peer=server-b215ae0e-c3ea-eefd-0999-fb9d91378e03
2026-02-28T09:03:29.066Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: added peer, starting replication: peer=server-6ce4e820-cbae-7fd1-80f6-b9f0485e8a53
2026-02-28T09:03:29.066Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: pipelining replication: peer="{Voter server-b215ae0e-c3ea-eefd-0999-fb9d91378e03 b215ae0e-c3ea-eefd-0999-fb9d91378e03}"
2026-02-28T09:03:29.066Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: pipelining replication: peer="{Voter server-6ce4e820-cbae-7fd1-80f6-b9f0485e8a53 6ce4e820-cbae-7fd1-80f6-b9f0485e8a53}"
2026-02-28T09:03:30.072Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:29.066648902 +0000 UTC m=+87.030445481 (1 nodes), 807.59964ms from start of poll, 808.03864ms from cluster start. Timeout at 2026-02-28 09:03:30.071651422 +0000 UTC m=+88.035447961, 1.00500248s after stability
2026-02-28T09:03:30.073Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: starting snapshot up to: index=12
2026-02-28T09:03:30.073Z [INFO]  creating new snapshot: path=/tmp/raft3737928028/snapshots/2-12-1772269410073.tmp
2026-02-28T09:03:30.074Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: snapshot complete up to: index=12
2026-02-28T09:03:30.075Z [INFO]  creating new snapshot: path=/tmp/raft3737928028/snapshots/2-10013-1772269410075.tmp
2026-02-28T09:03:30.075Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: copied to local snapshot: bytes=71
2026-02-28T09:03:30.075Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: snapshot restore progress: id=2-10013-1772269410075 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:30.076Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: removing all old logs from log store
2026-02-28T09:03:30.076Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: compacting logs: from=1 to=22
2026-02-28T09:03:30.076Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: restored user snapshot: index=10013
2026-02-28T09:03:30.076Z [ERROR] server-e09ace10-e926-4639-7f9e-a098a2b0575c: failed to get log: index=22 error="log not found"
2026-02-28T09:03:30.076Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: aborting pipeline replication: peer="{Voter server-6ce4e820-cbae-7fd1-80f6-b9f0485e8a53 6ce4e820-cbae-7fd1-80f6-b9f0485e8a53}"
2026-02-28T09:03:30.076Z [ERROR] server-e09ace10-e926-4639-7f9e-a098a2b0575c: failed to get log: index=22 error="log not found"
2026-02-28T09:03:30.076Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: aborting pipeline replication: peer="{Voter server-b215ae0e-c3ea-eefd-0999-fb9d91378e03 b215ae0e-c3ea-eefd-0999-fb9d91378e03}"
2026-02-28T09:03:30.085Z [ERROR] server-e09ace10-e926-4639-7f9e-a098a2b0575c: failed to get log: index=22 error="log not found"
2026-02-28T09:03:30.086Z [ERROR] server-e09ace10-e926-4639-7f9e-a098a2b0575c: failed to get log: index=22 error="log not found"
2026-02-28T09:03:30.086Z [INFO]  creating new snapshot: path=/tmp/raft3582039937/snapshots/2-10013-1772269410086.tmp
2026-02-28T09:03:30.086Z [INFO]  creating new snapshot: path=/tmp/raft4155844845/snapshots/2-10013-1772269410086.tmp
2026-02-28T09:03:30.086Z [INFO]  server-b215ae0e-c3ea-eefd-0999-fb9d91378e03: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:30.086Z [INFO]  server-6ce4e820-cbae-7fd1-80f6-b9f0485e8a53: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:30.087Z [INFO]  server-b215ae0e-c3ea-eefd-0999-fb9d91378e03: copied to local snapshot: bytes=71
2026-02-28T09:03:30.087Z [INFO]  server-6ce4e820-cbae-7fd1-80f6-b9f0485e8a53: copied to local snapshot: bytes=71
2026-02-28T09:03:30.087Z [INFO]  server-6ce4e820-cbae-7fd1-80f6-b9f0485e8a53: snapshot restore progress: id=2-10013-1772269410086 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:30.087Z [INFO]  server-b215ae0e-c3ea-eefd-0999-fb9d91378e03: snapshot restore progress: id=2-10013-1772269410086 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:30.087Z [INFO]  server-6ce4e820-cbae-7fd1-80f6-b9f0485e8a53: removing all old logs from log store
2026-02-28T09:03:30.087Z [INFO]  server-6ce4e820-cbae-7fd1-80f6-b9f0485e8a53: compacting logs: from=1 to=22
2026-02-28T09:03:30.087Z [INFO]  server-b215ae0e-c3ea-eefd-0999-fb9d91378e03: removing all old logs from log store
2026-02-28T09:03:30.087Z [INFO]  server-b215ae0e-c3ea-eefd-0999-fb9d91378e03: compacting logs: from=1 to=22
2026-02-28T09:03:30.087Z [INFO]  server-b215ae0e-c3ea-eefd-0999-fb9d91378e03: Installed remote snapshot
2026-02-28T09:03:30.087Z [INFO]  server-6ce4e820-cbae-7fd1-80f6-b9f0485e8a53: Installed remote snapshot
2026-02-28T09:03:30.087Z [WARN]  server-b215ae0e-c3ea-eefd-0999-fb9d91378e03: skipping application of old log: index=22
2026-02-28T09:03:30.087Z [WARN]  server-6ce4e820-cbae-7fd1-80f6-b9f0485e8a53: skipping application of old log: index=10013
2026-02-28T09:03:30.087Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: pipelining replication: peer="{Voter server-b215ae0e-c3ea-eefd-0999-fb9d91378e03 b215ae0e-c3ea-eefd-0999-fb9d91378e03}"
2026-02-28T09:03:30.087Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:30.087Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: pipelining replication: peer="{Voter server-6ce4e820-cbae-7fd1-80f6-b9f0485e8a53 6ce4e820-cbae-7fd1-80f6-b9f0485e8a53}"
2026-02-28T09:03:31.093Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:30.087769382 +0000 UTC m=+88.051565921 (2 nodes), 1.04µs from start of poll, 1.82915908s from cluster start. Timeout at 2026-02-28 09:03:31.092793862 +0000 UTC m=+89.056590401, 1.00502448s after stability
2026-02-28T09:03:31.105Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: aborting pipeline replication: peer="{Voter server-6ce4e820-cbae-7fd1-80f6-b9f0485e8a53 6ce4e820-cbae-7fd1-80f6-b9f0485e8a53}"
2026-02-28T09:03:31.105Z [INFO]  server-e09ace10-e926-4639-7f9e-a098a2b0575c: aborting pipeline replication: peer="{Voter server-b215ae0e-c3ea-eefd-0999-fb9d91378e03 b215ae0e-c3ea-eefd-0999-fb9d91378e03}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true
2026-02-28T09:03:31.106Z [INFO]  server-0fde4eb3-ef75-22b9-7f14-06cea4b63cf2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0fde4eb3-ef75-22b9-7f14-06cea4b63cf2 Address:0fde4eb3-ef75-22b9-7f14-06cea4b63cf2} {Suffrage:Voter ID:server-3995b3a5-ecb6-020d-cb45-fb1f8448bc67 Address:3995b3a5-ecb6-020d-cb45-fb1f8448bc67} {Suffrage:Voter ID:server-516b8772-df0e-80ff-9cae-fc0c5a22db92 Address:516b8772-df0e-80ff-9cae-fc0c5a22db92}]"
2026-02-28T09:03:31.106Z [INFO]  server-3995b3a5-ecb6-020d-cb45-fb1f8448bc67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0fde4eb3-ef75-22b9-7f14-06cea4b63cf2 Address:0fde4eb3-ef75-22b9-7f14-06cea4b63cf2} {Suffrage:Voter ID:server-3995b3a5-ecb6-020d-cb45-fb1f8448bc67 Address:3995b3a5-ecb6-020d-cb45-fb1f8448bc67} {Suffrage:Voter ID:server-516b8772-df0e-80ff-9cae-fc0c5a22db92 Address:516b8772-df0e-80ff-9cae-fc0c5a22db92}]"
2026-02-28T09:03:31.106Z [INFO]  server-0fde4eb3-ef75-22b9-7f14-06cea4b63cf2: entering follower state: follower="Node at 0fde4eb3-ef75-22b9-7f14-06cea4b63cf2 [Follower]" leader-address= leader-id=
2026-02-28T09:03:31.106Z [INFO]  server-3995b3a5-ecb6-020d-cb45-fb1f8448bc67: entering follower state: follower="Node at 3995b3a5-ecb6-020d-cb45-fb1f8448bc67 [Follower]" leader-address= leader-id=
2026-02-28T09:03:31.106Z [INFO]  server-516b8772-df0e-80ff-9cae-fc0c5a22db92: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0fde4eb3-ef75-22b9-7f14-06cea4b63cf2 Address:0fde4eb3-ef75-22b9-7f14-06cea4b63cf2} {Suffrage:Voter ID:server-3995b3a5-ecb6-020d-cb45-fb1f8448bc67 Address:3995b3a5-ecb6-020d-cb45-fb1f8448bc67} {Suffrage:Voter ID:server-516b8772-df0e-80ff-9cae-fc0c5a22db92 Address:516b8772-df0e-80ff-9cae-fc0c5a22db92}]"
2026-02-28T09:03:31.106Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:31.106Z [INFO]  server-516b8772-df0e-80ff-9cae-fc0c5a22db92: entering follower state: follower="Node at 516b8772-df0e-80ff-9cae-fc0c5a22db92 [Follower]" leader-address= leader-id=
2026-02-28T09:03:31.627Z [WARN]  server-3995b3a5-ecb6-020d-cb45-fb1f8448bc67: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:31.627Z [INFO]  server-3995b3a5-ecb6-020d-cb45-fb1f8448bc67: entering candidate state: node="Node at 3995b3a5-ecb6-020d-cb45-fb1f8448bc67 [Candidate]" term=2
2026-02-28T09:03:31.627Z [INFO]  server-3995b3a5-ecb6-020d-cb45-fb1f8448bc67: election won: term=2 tally=2
2026-02-28T09:03:31.627Z [INFO]  server-3995b3a5-ecb6-020d-cb45-fb1f8448bc67: entering leader state: leader="Node at 3995b3a5-ecb6-020d-cb45-fb1f8448bc67 [Leader]"
2026-02-28T09:03:31.628Z [INFO]  server-3995b3a5-ecb6-020d-cb45-fb1f8448bc67: added peer, starting replication: peer=server-0fde4eb3-ef75-22b9-7f14-06cea4b63cf2
2026-02-28T09:03:31.628Z [INFO]  server-3995b3a5-ecb6-020d-cb45-fb1f8448bc67: added peer, starting replication: peer=server-516b8772-df0e-80ff-9cae-fc0c5a22db92
2026-02-28T09:03:31.628Z [INFO]  server-3995b3a5-ecb6-020d-cb45-fb1f8448bc67: pipelining replication: peer="{Voter server-516b8772-df0e-80ff-9cae-fc0c5a22db92 516b8772-df0e-80ff-9cae-fc0c5a22db92}"
2026-02-28T09:03:31.628Z [INFO]  server-3995b3a5-ecb6-020d-cb45-fb1f8448bc67: pipelining replication: peer="{Voter server-0fde4eb3-ef75-22b9-7f14-06cea4b63cf2 0fde4eb3-ef75-22b9-7f14-06cea4b63cf2}"
2026-02-28T09:03:32.633Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:31.628104622 +0000 UTC m=+89.591901161 (1 nodes), 521.43876ms from start of poll, 521.92664ms from cluster start. Timeout at 2026-02-28 09:03:32.633108222 +0000 UTC m=+90.596904761, 1.0050036s after stability
2026-02-28T09:03:32.633Z [INFO]  server-3995b3a5-ecb6-020d-cb45-fb1f8448bc67: starting snapshot up to: index=12
2026-02-28T09:03:32.633Z [INFO]  creating new snapshot: path=/tmp/raft2164122722/snapshots/2-12-1772269412633.tmp
2026-02-28T09:03:32.634Z [INFO]  server-3995b3a5-ecb6-020d-cb45-fb1f8448bc67: snapshot complete up to: index=12
2026-02-28T09:03:32.635Z [INFO]  server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95aa27ce-fed1-d4d7-44b0-997a3af550d5 Address:95aa27ce-fed1-d4d7-44b0-997a3af550d5} {Suffrage:Voter ID:server-cd4f9ef6-c61e-12a7-a968-12df171baeee Address:cd4f9ef6-c61e-12a7-a968-12df171baeee} {Suffrage:Voter ID:server-47c42b78-724e-2cd5-0e0a-b95629a2569e Address:47c42b78-724e-2cd5-0e0a-b95629a2569e}]"
2026-02-28T09:03:32.635Z [INFO]  server-cd4f9ef6-c61e-12a7-a968-12df171baeee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95aa27ce-fed1-d4d7-44b0-997a3af550d5 Address:95aa27ce-fed1-d4d7-44b0-997a3af550d5} {Suffrage:Voter ID:server-cd4f9ef6-c61e-12a7-a968-12df171baeee Address:cd4f9ef6-c61e-12a7-a968-12df171baeee} {Suffrage:Voter ID:server-47c42b78-724e-2cd5-0e0a-b95629a2569e Address:47c42b78-724e-2cd5-0e0a-b95629a2569e}]"
2026-02-28T09:03:32.635Z [INFO]  server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: entering follower state: follower="Node at 95aa27ce-fed1-d4d7-44b0-997a3af550d5 [Follower]" leader-address= leader-id=
2026-02-28T09:03:32.635Z [INFO]  server-cd4f9ef6-c61e-12a7-a968-12df171baeee: entering follower state: follower="Node at cd4f9ef6-c61e-12a7-a968-12df171baeee [Follower]" leader-address= leader-id=
2026-02-28T09:03:32.635Z [INFO]  server-47c42b78-724e-2cd5-0e0a-b95629a2569e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95aa27ce-fed1-d4d7-44b0-997a3af550d5 Address:95aa27ce-fed1-d4d7-44b0-997a3af550d5} {Suffrage:Voter ID:server-cd4f9ef6-c61e-12a7-a968-12df171baeee Address:cd4f9ef6-c61e-12a7-a968-12df171baeee} {Suffrage:Voter ID:server-47c42b78-724e-2cd5-0e0a-b95629a2569e Address:47c42b78-724e-2cd5-0e0a-b95629a2569e}]"
2026-02-28T09:03:32.635Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:32.635Z [INFO]  server-47c42b78-724e-2cd5-0e0a-b95629a2569e: entering follower state: follower="Node at 47c42b78-724e-2cd5-0e0a-b95629a2569e [Follower]" leader-address= leader-id=
2026-02-28T09:03:33.330Z [WARN]  server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:33.330Z [INFO]  server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: entering candidate state: node="Node at 95aa27ce-fed1-d4d7-44b0-997a3af550d5 [Candidate]" term=2
2026-02-28T09:03:33.330Z [INFO]  server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: election won: term=2 tally=2
2026-02-28T09:03:33.330Z [INFO]  server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: entering leader state: leader="Node at 95aa27ce-fed1-d4d7-44b0-997a3af550d5 [Leader]"
2026-02-28T09:03:33.330Z [INFO]  server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: added peer, starting replication: peer=server-cd4f9ef6-c61e-12a7-a968-12df171baeee
2026-02-28T09:03:33.330Z [INFO]  server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: added peer, starting replication: peer=server-47c42b78-724e-2cd5-0e0a-b95629a2569e
2026-02-28T09:03:33.330Z [INFO]  server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: pipelining replication: peer="{Voter server-47c42b78-724e-2cd5-0e0a-b95629a2569e 47c42b78-724e-2cd5-0e0a-b95629a2569e}"
2026-02-28T09:03:33.330Z [INFO]  server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: pipelining replication: peer="{Voter server-cd4f9ef6-c61e-12a7-a968-12df171baeee cd4f9ef6-c61e-12a7-a968-12df171baeee}"
2026-02-28T09:03:34.335Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:33.330626502 +0000 UTC m=+91.294423041 (1 nodes), 694.95864ms from start of poll, 695.42076ms from cluster start. Timeout at 2026-02-28 09:03:34.335631022 +0000 UTC m=+92.299427561, 1.00500452s after stability
2026-02-28T09:03:34.336Z [INFO]  creating new snapshot: path=/tmp/raft4262408281/snapshots/2-10013-1772269414336.tmp
2026-02-28T09:03:34.336Z [INFO]  server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: copied to local snapshot: bytes=71
2026-02-28T09:03:34.337Z [INFO]  server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: snapshot restore progress: id=2-10013-1772269414336 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:34.337Z [INFO]  server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: restored user snapshot: index=10013
2026-02-28T09:03:34.337Z [ERROR] server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: failed to get log: index=3 error="log not found"
2026-02-28T09:03:34.337Z [INFO]  server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: aborting pipeline replication: peer="{Voter server-47c42b78-724e-2cd5-0e0a-b95629a2569e 47c42b78-724e-2cd5-0e0a-b95629a2569e}"
2026-02-28T09:03:34.337Z [ERROR] server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: failed to get log: index=3 error="log not found"
2026-02-28T09:03:34.337Z [INFO]  server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: aborting pipeline replication: peer="{Voter server-cd4f9ef6-c61e-12a7-a968-12df171baeee cd4f9ef6-c61e-12a7-a968-12df171baeee}"
2026-02-28T09:03:34.343Z [ERROR] server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: failed to get log: index=3 error="log not found"
2026-02-28T09:03:34.343Z [INFO]  creating new snapshot: path=/tmp/raft1308509756/snapshots/2-10013-1772269414343.tmp
2026-02-28T09:03:34.344Z [INFO]  server-cd4f9ef6-c61e-12a7-a968-12df171baeee: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:34.344Z [INFO]  server-cd4f9ef6-c61e-12a7-a968-12df171baeee: copied to local snapshot: bytes=71
2026-02-28T09:03:34.344Z [INFO]  server-cd4f9ef6-c61e-12a7-a968-12df171baeee: snapshot restore progress: id=2-10013-1772269414343 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:34.344Z [INFO]  server-cd4f9ef6-c61e-12a7-a968-12df171baeee: Installed remote snapshot
2026-02-28T09:03:34.344Z [INFO]  server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: pipelining replication: peer="{Voter server-cd4f9ef6-c61e-12a7-a968-12df171baeee cd4f9ef6-c61e-12a7-a968-12df171baeee}"
2026-02-28T09:03:34.345Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:34.347Z [ERROR] server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: failed to get log: index=3 error="log not found"
2026-02-28T09:03:34.347Z [INFO]  creating new snapshot: path=/tmp/raft2167452175/snapshots/2-10013-1772269414347.tmp
2026-02-28T09:03:34.347Z [INFO]  server-47c42b78-724e-2cd5-0e0a-b95629a2569e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:34.348Z [INFO]  server-47c42b78-724e-2cd5-0e0a-b95629a2569e: copied to local snapshot: bytes=71
2026-02-28T09:03:34.348Z [INFO]  server-47c42b78-724e-2cd5-0e0a-b95629a2569e: snapshot restore progress: id=2-10013-1772269414347 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:34.348Z [INFO]  server-47c42b78-724e-2cd5-0e0a-b95629a2569e: Installed remote snapshot
2026-02-28T09:03:34.348Z [INFO]  server-95aa27ce-fed1-d4d7-44b0-997a3af550d5: pipelining replication: peer="{Voter server-47c42b78-724e-2cd5-0e0a-b95629a2569e 47c42b78-724e-2cd5-0e0a-b95629a2569e}"
2026-02-28T09:03:35.350Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:34.345062022 +0000 UTC m=+92.308858601 (2 nodes), 1.28µs from start of poll, 1.70985632s from cluster start. Timeout at 2026-02-28 09:03:35.350076662 +0000 UTC m=+93.313873201, 1.0050146s after stability
2026-02-28T09:03:35.367Z [INFO]  server-3995b3a5-ecb6-020d-cb45-fb1f8448bc67: aborting pipeline replication: peer="{Voter server-516b8772-df0e-80ff-9cae-fc0c5a22db92 516b8772-df0e-80ff-9cae-fc0c5a22db92}"
2026-02-28T09:03:35.367Z [INFO]  server-3995b3a5-ecb6-020d-cb45-fb1f8448bc67: aborting pipeline replication: peer="{Voter server-0fde4eb3-ef75-22b9-7f14-06cea4b63cf2 0fde4eb3-ef75-22b9-7f14-06cea4b63cf2}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true
2026-02-28T09:03:35.368Z [INFO]  server-f2b4fa81-a65c-8581-cb8f-55806c4607c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f2b4fa81-a65c-8581-cb8f-55806c4607c6 Address:f2b4fa81-a65c-8581-cb8f-55806c4607c6} {Suffrage:Voter ID:server-36cb84a8-634b-94c1-841d-01c603612e0d Address:36cb84a8-634b-94c1-841d-01c603612e0d} {Suffrage:Voter ID:server-6c0537fe-6b4e-cd1b-3992-7a1970adb67f Address:6c0537fe-6b4e-cd1b-3992-7a1970adb67f}]"
2026-02-28T09:03:35.368Z [INFO]  server-36cb84a8-634b-94c1-841d-01c603612e0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f2b4fa81-a65c-8581-cb8f-55806c4607c6 Address:f2b4fa81-a65c-8581-cb8f-55806c4607c6} {Suffrage:Voter ID:server-36cb84a8-634b-94c1-841d-01c603612e0d Address:36cb84a8-634b-94c1-841d-01c603612e0d} {Suffrage:Voter ID:server-6c0537fe-6b4e-cd1b-3992-7a1970adb67f Address:6c0537fe-6b4e-cd1b-3992-7a1970adb67f}]"
2026-02-28T09:03:35.368Z [INFO]  server-6c0537fe-6b4e-cd1b-3992-7a1970adb67f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f2b4fa81-a65c-8581-cb8f-55806c4607c6 Address:f2b4fa81-a65c-8581-cb8f-55806c4607c6} {Suffrage:Voter ID:server-36cb84a8-634b-94c1-841d-01c603612e0d Address:36cb84a8-634b-94c1-841d-01c603612e0d} {Suffrage:Voter ID:server-6c0537fe-6b4e-cd1b-3992-7a1970adb67f Address:6c0537fe-6b4e-cd1b-3992-7a1970adb67f}]"
2026-02-28T09:03:35.368Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:35.368Z [INFO]  server-36cb84a8-634b-94c1-841d-01c603612e0d: entering follower state: follower="Node at 36cb84a8-634b-94c1-841d-01c603612e0d [Follower]" leader-address= leader-id=
2026-02-28T09:03:35.368Z [INFO]  server-6c0537fe-6b4e-cd1b-3992-7a1970adb67f: entering follower state: follower="Node at 6c0537fe-6b4e-cd1b-3992-7a1970adb67f [Follower]" leader-address= leader-id=
2026-02-28T09:03:35.368Z [INFO]  server-f2b4fa81-a65c-8581-cb8f-55806c4607c6: entering follower state: follower="Node at f2b4fa81-a65c-8581-cb8f-55806c4607c6 [Follower]" leader-address= leader-id=
2026-02-28T09:03:35.874Z [WARN]  server-36cb84a8-634b-94c1-841d-01c603612e0d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:35.874Z [INFO]  server-36cb84a8-634b-94c1-841d-01c603612e0d: entering candidate state: node="Node at 36cb84a8-634b-94c1-841d-01c603612e0d [Candidate]" term=2
2026-02-28T09:03:35.874Z [INFO]  server-36cb84a8-634b-94c1-841d-01c603612e0d: election won: term=2 tally=2
2026-02-28T09:03:35.874Z [INFO]  server-36cb84a8-634b-94c1-841d-01c603612e0d: entering leader state: leader="Node at 36cb84a8-634b-94c1-841d-01c603612e0d [Leader]"
2026-02-28T09:03:35.874Z [INFO]  server-36cb84a8-634b-94c1-841d-01c603612e0d: added peer, starting replication: peer=server-f2b4fa81-a65c-8581-cb8f-55806c4607c6
2026-02-28T09:03:35.874Z [INFO]  server-36cb84a8-634b-94c1-841d-01c603612e0d: added peer, starting replication: peer=server-6c0537fe-6b4e-cd1b-3992-7a1970adb67f
2026-02-28T09:03:35.874Z [INFO]  server-36cb84a8-634b-94c1-841d-01c603612e0d: pipelining replication: peer="{Voter server-6c0537fe-6b4e-cd1b-3992-7a1970adb67f 6c0537fe-6b4e-cd1b-3992-7a1970adb67f}"
2026-02-28T09:03:35.874Z [INFO]  server-36cb84a8-634b-94c1-841d-01c603612e0d: pipelining replication: peer="{Voter server-f2b4fa81-a65c-8581-cb8f-55806c4607c6 f2b4fa81-a65c-8581-cb8f-55806c4607c6}"
2026-02-28T09:03:36.880Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:35.874497902 +0000 UTC m=+93.838294481 (1 nodes), 506.12648ms from start of poll, 506.57456ms from cluster start. Timeout at 2026-02-28 09:03:36.879501862 +0000 UTC m=+94.843298401, 1.00500392s after stability
2026-02-28T09:03:36.881Z [INFO]  server-36cb84a8-634b-94c1-841d-01c603612e0d: starting snapshot up to: index=12
2026-02-28T09:03:36.881Z [INFO]  creating new snapshot: path=/tmp/raft2465924093/snapshots/2-12-1772269416881.tmp
2026-02-28T09:03:36.881Z [INFO]  server-36cb84a8-634b-94c1-841d-01c603612e0d: snapshot complete up to: index=12
2026-02-28T09:03:36.882Z [INFO]  server-1b1ef94d-70c0-94a8-eb49-e2bdb292df6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b1ef94d-70c0-94a8-eb49-e2bdb292df6f Address:1b1ef94d-70c0-94a8-eb49-e2bdb292df6f} {Suffrage:Voter ID:server-3ed5c6a6-bf22-eae9-82a8-2da1f262b73a Address:3ed5c6a6-bf22-eae9-82a8-2da1f262b73a} {Suffrage:Voter ID:server-36535af8-d1bd-427f-044d-64b379f166c9 Address:36535af8-d1bd-427f-044d-64b379f166c9}]"
2026-02-28T09:03:36.883Z [INFO]  server-3ed5c6a6-bf22-eae9-82a8-2da1f262b73a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b1ef94d-70c0-94a8-eb49-e2bdb292df6f Address:1b1ef94d-70c0-94a8-eb49-e2bdb292df6f} {Suffrage:Voter ID:server-3ed5c6a6-bf22-eae9-82a8-2da1f262b73a Address:3ed5c6a6-bf22-eae9-82a8-2da1f262b73a} {Suffrage:Voter ID:server-36535af8-d1bd-427f-044d-64b379f166c9 Address:36535af8-d1bd-427f-044d-64b379f166c9}]"
2026-02-28T09:03:36.883Z [INFO]  server-1b1ef94d-70c0-94a8-eb49-e2bdb292df6f: entering follower state: follower="Node at 1b1ef94d-70c0-94a8-eb49-e2bdb292df6f [Follower]" leader-address= leader-id=
2026-02-28T09:03:36.883Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b1ef94d-70c0-94a8-eb49-e2bdb292df6f Address:1b1ef94d-70c0-94a8-eb49-e2bdb292df6f} {Suffrage:Voter ID:server-3ed5c6a6-bf22-eae9-82a8-2da1f262b73a Address:3ed5c6a6-bf22-eae9-82a8-2da1f262b73a} {Suffrage:Voter ID:server-36535af8-d1bd-427f-044d-64b379f166c9 Address:36535af8-d1bd-427f-044d-64b379f166c9}]"
2026-02-28T09:03:36.883Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:36.883Z [INFO]  server-3ed5c6a6-bf22-eae9-82a8-2da1f262b73a: entering follower state: follower="Node at 3ed5c6a6-bf22-eae9-82a8-2da1f262b73a [Follower]" leader-address= leader-id=
2026-02-28T09:03:36.883Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: entering follower state: follower="Node at 36535af8-d1bd-427f-044d-64b379f166c9 [Follower]" leader-address= leader-id=
2026-02-28T09:03:37.418Z [WARN]  server-36535af8-d1bd-427f-044d-64b379f166c9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:37.418Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: entering candidate state: node="Node at 36535af8-d1bd-427f-044d-64b379f166c9 [Candidate]" term=2
2026-02-28T09:03:37.418Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: election won: term=2 tally=2
2026-02-28T09:03:37.418Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: entering leader state: leader="Node at 36535af8-d1bd-427f-044d-64b379f166c9 [Leader]"
2026-02-28T09:03:37.418Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: added peer, starting replication: peer=server-1b1ef94d-70c0-94a8-eb49-e2bdb292df6f
2026-02-28T09:03:37.418Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: added peer, starting replication: peer=server-3ed5c6a6-bf22-eae9-82a8-2da1f262b73a
2026-02-28T09:03:37.418Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: pipelining replication: peer="{Voter server-1b1ef94d-70c0-94a8-eb49-e2bdb292df6f 1b1ef94d-70c0-94a8-eb49-e2bdb292df6f}"
2026-02-28T09:03:37.418Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: pipelining replication: peer="{Voter server-3ed5c6a6-bf22-eae9-82a8-2da1f262b73a 3ed5c6a6-bf22-eae9-82a8-2da1f262b73a}"
2026-02-28T09:03:38.424Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:37.418511582 +0000 UTC m=+95.382308121 (1 nodes), 535.222ms from start of poll, 535.6926ms from cluster start. Timeout at 2026-02-28 09:03:38.423515342 +0000 UTC m=+96.387311881, 1.00500376s after stability
2026-02-28T09:03:38.424Z [INFO]  creating new snapshot: path=/tmp/raft1335611588/snapshots/2-10013-1772269418424.tmp
2026-02-28T09:03:38.425Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: copied to local snapshot: bytes=71
2026-02-28T09:03:38.425Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: snapshot restore progress: id=2-10013-1772269418424 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:38.425Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: removing all old logs from log store
2026-02-28T09:03:38.425Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: compacting logs: from=1 to=2
2026-02-28T09:03:38.425Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: restored user snapshot: index=10013
2026-02-28T09:03:38.425Z [ERROR] server-36535af8-d1bd-427f-044d-64b379f166c9: failed to get log: index=2 error="log not found"
2026-02-28T09:03:38.425Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: aborting pipeline replication: peer="{Voter server-3ed5c6a6-bf22-eae9-82a8-2da1f262b73a 3ed5c6a6-bf22-eae9-82a8-2da1f262b73a}"
2026-02-28T09:03:38.425Z [ERROR] server-36535af8-d1bd-427f-044d-64b379f166c9: failed to get log: index=2 error="log not found"
2026-02-28T09:03:38.425Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: aborting pipeline replication: peer="{Voter server-1b1ef94d-70c0-94a8-eb49-e2bdb292df6f 1b1ef94d-70c0-94a8-eb49-e2bdb292df6f}"
2026-02-28T09:03:38.433Z [ERROR] server-36535af8-d1bd-427f-044d-64b379f166c9: failed to get log: index=2 error="log not found"
2026-02-28T09:03:38.434Z [INFO]  creating new snapshot: path=/tmp/raft3916441081/snapshots/2-10013-1772269418434.tmp
2026-02-28T09:03:38.434Z [INFO]  server-3ed5c6a6-bf22-eae9-82a8-2da1f262b73a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:38.434Z [INFO]  server-3ed5c6a6-bf22-eae9-82a8-2da1f262b73a: copied to local snapshot: bytes=71
2026-02-28T09:03:38.435Z [INFO]  server-3ed5c6a6-bf22-eae9-82a8-2da1f262b73a: snapshot restore progress: id=2-10013-1772269418434 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:38.435Z [INFO]  server-3ed5c6a6-bf22-eae9-82a8-2da1f262b73a: removing all old logs from log store
2026-02-28T09:03:38.435Z [INFO]  server-3ed5c6a6-bf22-eae9-82a8-2da1f262b73a: compacting logs: from=1 to=2
2026-02-28T09:03:38.435Z [INFO]  server-3ed5c6a6-bf22-eae9-82a8-2da1f262b73a: Installed remote snapshot
2026-02-28T09:03:38.435Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: pipelining replication: peer="{Voter server-3ed5c6a6-bf22-eae9-82a8-2da1f262b73a 3ed5c6a6-bf22-eae9-82a8-2da1f262b73a}"
2026-02-28T09:03:38.435Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:38.436Z [ERROR] server-36535af8-d1bd-427f-044d-64b379f166c9: failed to get log: index=2 error="log not found"
2026-02-28T09:03:38.436Z [INFO]  creating new snapshot: path=/tmp/raft2482526776/snapshots/2-10013-1772269418436.tmp
2026-02-28T09:03:38.436Z [INFO]  server-1b1ef94d-70c0-94a8-eb49-e2bdb292df6f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:38.437Z [INFO]  server-1b1ef94d-70c0-94a8-eb49-e2bdb292df6f: copied to local snapshot: bytes=71
2026-02-28T09:03:38.437Z [INFO]  server-1b1ef94d-70c0-94a8-eb49-e2bdb292df6f: snapshot restore progress: id=2-10013-1772269418436 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-02-28T09:03:38.437Z [INFO]  server-1b1ef94d-70c0-94a8-eb49-e2bdb292df6f: removing all old logs from log store
2026-02-28T09:03:38.437Z [INFO]  server-1b1ef94d-70c0-94a8-eb49-e2bdb292df6f: compacting logs: from=1 to=2
2026-02-28T09:03:38.437Z [INFO]  server-1b1ef94d-70c0-94a8-eb49-e2bdb292df6f: Installed remote snapshot
2026-02-28T09:03:38.437Z [INFO]  server-36535af8-d1bd-427f-044d-64b379f166c9: pipelining replication: peer="{Voter server-1b1ef94d-70c0-94a8-eb49-e2bdb292df6f 1b1ef94d-70c0-94a8-eb49-e2bdb292df6f}"
2026-02-28T09:03:39.440Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:38.435444302 +0000 UTC m=+96.399240841 (2 nodes), 1.76µs from start of poll, 1.55262532s from cluster start. Timeout at 2026-02-28 09:03:39.440483462 +0000 UTC m=+97.404280041, 1.0050392s after stability
2026-02-28T09:03:39.453Z [INFO]  server-36cb84a8-634b-94c1-841d-01c603612e0d: aborting pipeline replication: peer="{Voter server-6c0537fe-6b4e-cd1b-3992-7a1970adb67f 6c0537fe-6b4e-cd1b-3992-7a1970adb67f}"
2026-02-28T09:03:39.453Z [INFO]  server-36cb84a8-634b-94c1-841d-01c603612e0d: aborting pipeline replication: peer="{Voter server-f2b4fa81-a65c-8581-cb8f-55806c4607c6 f2b4fa81-a65c-8581-cb8f-55806c4607c6}"
--- PASS: TestRaft_UserRestore (83.83s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.62s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.93s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.39s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.51s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.57s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.60s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.21s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.30s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.56s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.55s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.31s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (3.06s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.62s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.16s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.43s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.64s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.21s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.16s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.59s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.85s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.26s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.09s)
=== RUN   TestRaft_SendSnapshotFollower
2026-02-28T09:03:39.454Z [INFO]  server-1ba99536-df2b-31a1-b304-f80eb31cd00a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ba99536-df2b-31a1-b304-f80eb31cd00a Address:1ba99536-df2b-31a1-b304-f80eb31cd00a} {Suffrage:Voter ID:server-b96fccee-f151-901b-cf7c-124f5f5dde16 Address:b96fccee-f151-901b-cf7c-124f5f5dde16} {Suffrage:Voter ID:server-89eeef75-d3fc-21db-ed88-c3157f970e53 Address:89eeef75-d3fc-21db-ed88-c3157f970e53}]"
2026-02-28T09:03:39.454Z [INFO]  server-1ba99536-df2b-31a1-b304-f80eb31cd00a: entering follower state: follower="Node at 1ba99536-df2b-31a1-b304-f80eb31cd00a [Follower]" leader-address= leader-id=
2026-02-28T09:03:39.454Z [INFO]  server-b96fccee-f151-901b-cf7c-124f5f5dde16: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ba99536-df2b-31a1-b304-f80eb31cd00a Address:1ba99536-df2b-31a1-b304-f80eb31cd00a} {Suffrage:Voter ID:server-b96fccee-f151-901b-cf7c-124f5f5dde16 Address:b96fccee-f151-901b-cf7c-124f5f5dde16} {Suffrage:Voter ID:server-89eeef75-d3fc-21db-ed88-c3157f970e53 Address:89eeef75-d3fc-21db-ed88-c3157f970e53}]"
2026-02-28T09:03:39.454Z [INFO]  server-b96fccee-f151-901b-cf7c-124f5f5dde16: entering follower state: follower="Node at b96fccee-f151-901b-cf7c-124f5f5dde16 [Follower]" leader-address= leader-id=
2026-02-28T09:03:39.454Z [INFO]  server-89eeef75-d3fc-21db-ed88-c3157f970e53: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ba99536-df2b-31a1-b304-f80eb31cd00a Address:1ba99536-df2b-31a1-b304-f80eb31cd00a} {Suffrage:Voter ID:server-b96fccee-f151-901b-cf7c-124f5f5dde16 Address:b96fccee-f151-901b-cf7c-124f5f5dde16} {Suffrage:Voter ID:server-89eeef75-d3fc-21db-ed88-c3157f970e53 Address:89eeef75-d3fc-21db-ed88-c3157f970e53}]"
2026-02-28T09:03:39.454Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:39.454Z [INFO]  server-89eeef75-d3fc-21db-ed88-c3157f970e53: entering follower state: follower="Node at 89eeef75-d3fc-21db-ed88-c3157f970e53 [Follower]" leader-address= leader-id=
2026-02-28T09:03:39.507Z [WARN]  server-1ba99536-df2b-31a1-b304-f80eb31cd00a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:39.507Z [INFO]  server-1ba99536-df2b-31a1-b304-f80eb31cd00a: entering candidate state: node="Node at 1ba99536-df2b-31a1-b304-f80eb31cd00a [Candidate]" term=2
2026-02-28T09:03:39.507Z [INFO]  server-1ba99536-df2b-31a1-b304-f80eb31cd00a: election won: term=2 tally=2
2026-02-28T09:03:39.507Z [INFO]  server-1ba99536-df2b-31a1-b304-f80eb31cd00a: entering leader state: leader="Node at 1ba99536-df2b-31a1-b304-f80eb31cd00a [Leader]"
2026-02-28T09:03:39.507Z [INFO]  server-1ba99536-df2b-31a1-b304-f80eb31cd00a: added peer, starting replication: peer=server-b96fccee-f151-901b-cf7c-124f5f5dde16
2026-02-28T09:03:39.507Z [INFO]  server-1ba99536-df2b-31a1-b304-f80eb31cd00a: added peer, starting replication: peer=server-89eeef75-d3fc-21db-ed88-c3157f970e53
2026-02-28T09:03:39.507Z [INFO]  server-1ba99536-df2b-31a1-b304-f80eb31cd00a: pipelining replication: peer="{Voter server-89eeef75-d3fc-21db-ed88-c3157f970e53 89eeef75-d3fc-21db-ed88-c3157f970e53}"
2026-02-28T09:03:39.508Z [WARN]  server-89eeef75-d3fc-21db-ed88-c3157f970e53: rejecting vote request since our last term is greater: candidate=1ba99536-df2b-31a1-b304-f80eb31cd00a last-term=2 last-candidate-term=1
2026-02-28T09:03:39.508Z [INFO]  server-1ba99536-df2b-31a1-b304-f80eb31cd00a: pipelining replication: peer="{Voter server-b96fccee-f151-901b-cf7c-124f5f5dde16 b96fccee-f151-901b-cf7c-124f5f5dde16}"
2026-02-28T09:03:39.613Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:39.508184982 +0000 UTC m=+97.471981521 (2 nodes), 53.79848ms from start of poll, 54.23648ms from cluster start. Timeout at 2026-02-28 09:03:39.613187182 +0000 UTC m=+97.576983721, 105.0022ms after stability
2026-02-28T09:03:39.613Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:39.719Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:39.613315262 +0000 UTC m=+97.577111801 (1 nodes), 2.84µs from start of poll, 159.36676ms from cluster start. Timeout at 2026-02-28 09:03:39.718344622 +0000 UTC m=+97.682141161, 105.02936ms after stability
2026-02-28T09:03:39.719Z [ERROR] server-1ba99536-df2b-31a1-b304-f80eb31cd00a: failed to pipeline appendEntries: peer="{Voter server-b96fccee-f151-901b-cf7c-124f5f5dde16 b96fccee-f151-901b-cf7c-124f5f5dde16}" error="append pipeline closed"
2026-02-28T09:03:39.719Z [INFO]  server-1ba99536-df2b-31a1-b304-f80eb31cd00a: aborting pipeline replication: peer="{Voter server-b96fccee-f151-901b-cf7c-124f5f5dde16 b96fccee-f151-901b-cf7c-124f5f5dde16}"
2026-02-28T09:03:39.719Z [ERROR] server-1ba99536-df2b-31a1-b304-f80eb31cd00a: failed to appendEntries to: peer="{Voter server-b96fccee-f151-901b-cf7c-124f5f5dde16 b96fccee-f151-901b-cf7c-124f5f5dde16}" error="failed to connect to peer: b96fccee-f151-901b-cf7c-124f5f5dde16"
2026-02-28T09:03:39.723Z [ERROR] server-1ba99536-df2b-31a1-b304-f80eb31cd00a: failed to heartbeat to: peer=b96fccee-f151-901b-cf7c-124f5f5dde16 backoff time=10ms error="failed to connect to peer: b96fccee-f151-901b-cf7c-124f5f5dde16"
    raft_test.go:1627: [INFO] Finished apply without behind follower
2026-02-28T09:03:39.724Z [INFO]  server-1ba99536-df2b-31a1-b304-f80eb31cd00a: starting snapshot up to: index=102
2026-02-28T09:03:39.724Z [INFO]  creating new snapshot: path=/tmp/raft2965224268/snapshots/2-102-1772269419724.tmp
2026-02-28T09:03:39.724Z [INFO]  server-1ba99536-df2b-31a1-b304-f80eb31cd00a: compacting logs: from=1 to=92
2026-02-28T09:03:39.724Z [INFO]  server-1ba99536-df2b-31a1-b304-f80eb31cd00a: snapshot complete up to: index=102
2026-02-28T09:03:39.724Z [ERROR] server-b96fccee-f151-901b-cf7c-124f5f5dde16: failed to take snapshot: error="nothing new to snapshot"
2026-02-28T09:03:39.724Z [INFO]  server-89eeef75-d3fc-21db-ed88-c3157f970e53: starting snapshot up to: index=100
2026-02-28T09:03:39.725Z [INFO]  creating new snapshot: path=/tmp/raft187435312/snapshots/2-100-1772269419724.tmp
2026-02-28T09:03:39.725Z [INFO]  server-89eeef75-d3fc-21db-ed88-c3157f970e53: compacting logs: from=1 to=92
2026-02-28T09:03:39.725Z [INFO]  server-89eeef75-d3fc-21db-ed88-c3157f970e53: snapshot complete up to: index=100
2026-02-28T09:03:39.730Z [ERROR] server-1ba99536-df2b-31a1-b304-f80eb31cd00a: failed to get log: index=2 error="log not found"
2026-02-28T09:03:39.730Z [INFO]  creating new snapshot: path=/tmp/raft3725721905/snapshots/2-102-1772269419730.tmp
2026-02-28T09:03:39.731Z [INFO]  server-b96fccee-f151-901b-cf7c-124f5f5dde16: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2026-02-28T09:03:39.731Z [INFO]  server-b96fccee-f151-901b-cf7c-124f5f5dde16: copied to local snapshot: bytes=693
2026-02-28T09:03:39.731Z [INFO]  server-b96fccee-f151-901b-cf7c-124f5f5dde16: snapshot restore progress: id=2-102-1772269419730 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-02-28T09:03:39.731Z [INFO]  server-b96fccee-f151-901b-cf7c-124f5f5dde16: Installed remote snapshot
2026-02-28T09:03:39.731Z [WARN]  server-b96fccee-f151-901b-cf7c-124f5f5dde16: skipping application of old log: index=102
2026-02-28T09:03:39.731Z [INFO]  server-1ba99536-df2b-31a1-b304-f80eb31cd00a: pipelining replication: peer="{Voter server-b96fccee-f151-901b-cf7c-124f5f5dde16 b96fccee-f151-901b-cf7c-124f5f5dde16}"
2026-02-28T09:03:39.735Z [INFO]  server-1ba99536-df2b-31a1-b304-f80eb31cd00a: aborting pipeline replication: peer="{Voter server-89eeef75-d3fc-21db-ed88-c3157f970e53 89eeef75-d3fc-21db-ed88-c3157f970e53}"
2026-02-28T09:03:39.736Z [INFO]  server-1ba99536-df2b-31a1-b304-f80eb31cd00a: aborting pipeline replication: peer="{Voter server-b96fccee-f151-901b-cf7c-124f5f5dde16 b96fccee-f151-901b-cf7c-124f5f5dde16}"
--- PASS: TestRaft_SendSnapshotFollower (0.28s)
=== RUN   TestRaft_SendSnapshotAndLogsFollower
2026-02-28T09:03:39.737Z [INFO]  server-f8aa0e3e-950f-0482-f656-4984f1e63dec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8aa0e3e-950f-0482-f656-4984f1e63dec Address:f8aa0e3e-950f-0482-f656-4984f1e63dec} {Suffrage:Voter ID:server-8f978db5-d49c-bc51-52a6-834b62a09769 Address:8f978db5-d49c-bc51-52a6-834b62a09769} {Suffrage:Voter ID:server-31997732-423e-bc59-c011-25941e1e0ecb Address:31997732-423e-bc59-c011-25941e1e0ecb}]"
2026-02-28T09:03:39.737Z [INFO]  server-8f978db5-d49c-bc51-52a6-834b62a09769: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8aa0e3e-950f-0482-f656-4984f1e63dec Address:f8aa0e3e-950f-0482-f656-4984f1e63dec} {Suffrage:Voter ID:server-8f978db5-d49c-bc51-52a6-834b62a09769 Address:8f978db5-d49c-bc51-52a6-834b62a09769} {Suffrage:Voter ID:server-31997732-423e-bc59-c011-25941e1e0ecb Address:31997732-423e-bc59-c011-25941e1e0ecb}]"
2026-02-28T09:03:39.737Z [INFO]  server-31997732-423e-bc59-c011-25941e1e0ecb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8aa0e3e-950f-0482-f656-4984f1e63dec Address:f8aa0e3e-950f-0482-f656-4984f1e63dec} {Suffrage:Voter ID:server-8f978db5-d49c-bc51-52a6-834b62a09769 Address:8f978db5-d49c-bc51-52a6-834b62a09769} {Suffrage:Voter ID:server-31997732-423e-bc59-c011-25941e1e0ecb Address:31997732-423e-bc59-c011-25941e1e0ecb}]"
2026-02-28T09:03:39.737Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:39.737Z [INFO]  server-f8aa0e3e-950f-0482-f656-4984f1e63dec: entering follower state: follower="Node at f8aa0e3e-950f-0482-f656-4984f1e63dec [Follower]" leader-address= leader-id=
2026-02-28T09:03:39.737Z [INFO]  server-8f978db5-d49c-bc51-52a6-834b62a09769: entering follower state: follower="Node at 8f978db5-d49c-bc51-52a6-834b62a09769 [Follower]" leader-address= leader-id=
2026-02-28T09:03:39.737Z [INFO]  server-31997732-423e-bc59-c011-25941e1e0ecb: entering follower state: follower="Node at 31997732-423e-bc59-c011-25941e1e0ecb [Follower]" leader-address= leader-id=
2026-02-28T09:03:39.794Z [WARN]  server-31997732-423e-bc59-c011-25941e1e0ecb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:39.794Z [INFO]  server-31997732-423e-bc59-c011-25941e1e0ecb: entering candidate state: node="Node at 31997732-423e-bc59-c011-25941e1e0ecb [Candidate]" term=2
2026-02-28T09:03:39.794Z [INFO]  server-31997732-423e-bc59-c011-25941e1e0ecb: election won: term=2 tally=2
2026-02-28T09:03:39.794Z [INFO]  server-31997732-423e-bc59-c011-25941e1e0ecb: entering leader state: leader="Node at 31997732-423e-bc59-c011-25941e1e0ecb [Leader]"
2026-02-28T09:03:39.794Z [INFO]  server-31997732-423e-bc59-c011-25941e1e0ecb: added peer, starting replication: peer=server-f8aa0e3e-950f-0482-f656-4984f1e63dec
2026-02-28T09:03:39.794Z [INFO]  server-31997732-423e-bc59-c011-25941e1e0ecb: added peer, starting replication: peer=server-8f978db5-d49c-bc51-52a6-834b62a09769
2026-02-28T09:03:39.794Z [WARN]  server-f8aa0e3e-950f-0482-f656-4984f1e63dec: rejecting vote request since our last term is greater: candidate=31997732-423e-bc59-c011-25941e1e0ecb last-term=2 last-candidate-term=1
2026-02-28T09:03:39.794Z [INFO]  server-31997732-423e-bc59-c011-25941e1e0ecb: pipelining replication: peer="{Voter server-8f978db5-d49c-bc51-52a6-834b62a09769 8f978db5-d49c-bc51-52a6-834b62a09769}"
2026-02-28T09:03:39.794Z [INFO]  server-31997732-423e-bc59-c011-25941e1e0ecb: pipelining replication: peer="{Voter server-f8aa0e3e-950f-0482-f656-4984f1e63dec f8aa0e3e-950f-0482-f656-4984f1e63dec}"
2026-02-28T09:03:39.900Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:39.794437182 +0000 UTC m=+97.758233721 (2 nodes), 57.1152ms from start of poll, 57.53268ms from cluster start. Timeout at 2026-02-28 09:03:39.899439702 +0000 UTC m=+97.863236241, 105.00252ms after stability
2026-02-28T09:03:39.900Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:40.006Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:39.900634222 +0000 UTC m=+97.864430761 (1 nodes), 3.2µs from start of poll, 163.72972ms from cluster start. Timeout at 2026-02-28 09:03:40.005651182 +0000 UTC m=+97.969447761, 105.017ms after stability
2026-02-28T09:03:40.006Z [ERROR] server-31997732-423e-bc59-c011-25941e1e0ecb: failed to pipeline appendEntries: peer="{Voter server-f8aa0e3e-950f-0482-f656-4984f1e63dec f8aa0e3e-950f-0482-f656-4984f1e63dec}" error="append pipeline closed"
2026-02-28T09:03:40.006Z [INFO]  server-31997732-423e-bc59-c011-25941e1e0ecb: aborting pipeline replication: peer="{Voter server-f8aa0e3e-950f-0482-f656-4984f1e63dec f8aa0e3e-950f-0482-f656-4984f1e63dec}"
2026-02-28T09:03:40.006Z [ERROR] server-31997732-423e-bc59-c011-25941e1e0ecb: failed to appendEntries to: peer="{Voter server-f8aa0e3e-950f-0482-f656-4984f1e63dec f8aa0e3e-950f-0482-f656-4984f1e63dec}" error="failed to connect to peer: f8aa0e3e-950f-0482-f656-4984f1e63dec"
2026-02-28T09:03:40.007Z [ERROR] server-31997732-423e-bc59-c011-25941e1e0ecb: failed to heartbeat to: peer=f8aa0e3e-950f-0482-f656-4984f1e63dec backoff time=10ms error="failed to connect to peer: f8aa0e3e-950f-0482-f656-4984f1e63dec"
    raft_test.go:1669: [INFO] Finished apply without behind follower
2026-02-28T09:03:40.010Z [ERROR] server-f8aa0e3e-950f-0482-f656-4984f1e63dec: failed to take snapshot: error="nothing new to snapshot"
2026-02-28T09:03:40.010Z [INFO]  server-8f978db5-d49c-bc51-52a6-834b62a09769: starting snapshot up to: index=98
2026-02-28T09:03:40.010Z [INFO]  creating new snapshot: path=/tmp/raft638497112/snapshots/2-98-1772269420010.tmp
2026-02-28T09:03:40.011Z [INFO]  server-8f978db5-d49c-bc51-52a6-834b62a09769: compacting logs: from=1 to=92
2026-02-28T09:03:40.011Z [INFO]  server-8f978db5-d49c-bc51-52a6-834b62a09769: snapshot complete up to: index=98
2026-02-28T09:03:40.011Z [INFO]  server-31997732-423e-bc59-c011-25941e1e0ecb: starting snapshot up to: index=102
2026-02-28T09:03:40.011Z [INFO]  creating new snapshot: path=/tmp/raft2526565710/snapshots/2-102-1772269420011.tmp
2026-02-28T09:03:40.011Z [INFO]  server-31997732-423e-bc59-c011-25941e1e0ecb: compacting logs: from=1 to=92
2026-02-28T09:03:40.012Z [INFO]  server-31997732-423e-bc59-c011-25941e1e0ecb: snapshot complete up to: index=102
    raft_test.go:1690: [INFO] Finished apply without behind follower
2026-02-28T09:03:40.017Z [ERROR] server-31997732-423e-bc59-c011-25941e1e0ecb: failed to get log: index=2 error="log not found"
2026-02-28T09:03:40.017Z [INFO]  creating new snapshot: path=/tmp/raft1175864821/snapshots/2-102-1772269420017.tmp
2026-02-28T09:03:40.017Z [INFO]  server-f8aa0e3e-950f-0482-f656-4984f1e63dec: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2026-02-28T09:03:40.018Z [INFO]  server-f8aa0e3e-950f-0482-f656-4984f1e63dec: copied to local snapshot: bytes=693
2026-02-28T09:03:40.018Z [INFO]  server-f8aa0e3e-950f-0482-f656-4984f1e63dec: snapshot restore progress: id=2-102-1772269420017 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-02-28T09:03:40.018Z [INFO]  server-f8aa0e3e-950f-0482-f656-4984f1e63dec: Installed remote snapshot
2026-02-28T09:03:40.018Z [INFO]  server-31997732-423e-bc59-c011-25941e1e0ecb: pipelining replication: peer="{Voter server-f8aa0e3e-950f-0482-f656-4984f1e63dec f8aa0e3e-950f-0482-f656-4984f1e63dec}"
2026-02-28T09:03:40.026Z [INFO]  server-31997732-423e-bc59-c011-25941e1e0ecb: aborting pipeline replication: peer="{Voter server-8f978db5-d49c-bc51-52a6-834b62a09769 8f978db5-d49c-bc51-52a6-834b62a09769}"
2026-02-28T09:03:40.026Z [INFO]  server-31997732-423e-bc59-c011-25941e1e0ecb: aborting pipeline replication: peer="{Voter server-f8aa0e3e-950f-0482-f656-4984f1e63dec f8aa0e3e-950f-0482-f656-4984f1e63dec}"
--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.29s)
=== RUN   TestRaft_ReJoinFollower
2026-02-28T09:03:40.030Z [INFO]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0f3d8cd-d65f-6f36-367d-feccffaa7684 Address:a0f3d8cd-d65f-6f36-367d-feccffaa7684} {Suffrage:Voter ID:server-e4455e64-1d13-edfe-ee2b-63eb11be423a Address:e4455e64-1d13-edfe-ee2b-63eb11be423a} {Suffrage:Voter ID:server-38e289c3-a54b-6e78-9ad8-d98afdb9fabd Address:38e289c3-a54b-6e78-9ad8-d98afdb9fabd}]"
2026-02-28T09:03:40.031Z [INFO]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: entering follower state: follower="Node at a0f3d8cd-d65f-6f36-367d-feccffaa7684 [Follower]" leader-address= leader-id=
2026-02-28T09:03:40.031Z [INFO]  server-e4455e64-1d13-edfe-ee2b-63eb11be423a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0f3d8cd-d65f-6f36-367d-feccffaa7684 Address:a0f3d8cd-d65f-6f36-367d-feccffaa7684} {Suffrage:Voter ID:server-e4455e64-1d13-edfe-ee2b-63eb11be423a Address:e4455e64-1d13-edfe-ee2b-63eb11be423a} {Suffrage:Voter ID:server-38e289c3-a54b-6e78-9ad8-d98afdb9fabd Address:38e289c3-a54b-6e78-9ad8-d98afdb9fabd}]"
2026-02-28T09:03:40.031Z [INFO]  server-e4455e64-1d13-edfe-ee2b-63eb11be423a: entering follower state: follower="Node at e4455e64-1d13-edfe-ee2b-63eb11be423a [Follower]" leader-address= leader-id=
2026-02-28T09:03:40.031Z [INFO]  server-38e289c3-a54b-6e78-9ad8-d98afdb9fabd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0f3d8cd-d65f-6f36-367d-feccffaa7684 Address:a0f3d8cd-d65f-6f36-367d-feccffaa7684} {Suffrage:Voter ID:server-e4455e64-1d13-edfe-ee2b-63eb11be423a Address:e4455e64-1d13-edfe-ee2b-63eb11be423a} {Suffrage:Voter ID:server-38e289c3-a54b-6e78-9ad8-d98afdb9fabd Address:38e289c3-a54b-6e78-9ad8-d98afdb9fabd}]"
2026-02-28T09:03:40.031Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:40.031Z [INFO]  server-38e289c3-a54b-6e78-9ad8-d98afdb9fabd: entering follower state: follower="Node at 38e289c3-a54b-6e78-9ad8-d98afdb9fabd [Follower]" leader-address= leader-id=
2026-02-28T09:03:40.091Z [WARN]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:40.091Z [INFO]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: entering candidate state: node="Node at a0f3d8cd-d65f-6f36-367d-feccffaa7684 [Candidate]" term=2
2026-02-28T09:03:40.091Z [INFO]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: election won: term=2 tally=2
2026-02-28T09:03:40.091Z [INFO]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: entering leader state: leader="Node at a0f3d8cd-d65f-6f36-367d-feccffaa7684 [Leader]"
2026-02-28T09:03:40.091Z [INFO]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: added peer, starting replication: peer=server-e4455e64-1d13-edfe-ee2b-63eb11be423a
2026-02-28T09:03:40.091Z [INFO]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: added peer, starting replication: peer=server-38e289c3-a54b-6e78-9ad8-d98afdb9fabd
2026-02-28T09:03:40.091Z [INFO]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: pipelining replication: peer="{Voter server-e4455e64-1d13-edfe-ee2b-63eb11be423a e4455e64-1d13-edfe-ee2b-63eb11be423a}"
2026-02-28T09:03:40.091Z [INFO]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: pipelining replication: peer="{Voter server-38e289c3-a54b-6e78-9ad8-d98afdb9fabd 38e289c3-a54b-6e78-9ad8-d98afdb9fabd}"
2026-02-28T09:03:40.196Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:40.091342702 +0000 UTC m=+98.055139241 (1 nodes), 59.20668ms from start of poll, 60.54864ms from cluster start. Timeout at 2026-02-28 09:03:40.196363222 +0000 UTC m=+98.160159761, 105.02052ms after stability
2026-02-28T09:03:40.202Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:40.307Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:40.202597702 +0000 UTC m=+98.166394281 (2 nodes), 4.56µs from start of poll, 171.80368ms from cluster start. Timeout at 2026-02-28 09:03:40.307609742 +0000 UTC m=+98.271406281, 105.012ms after stability
2026-02-28T09:03:40.307Z [INFO]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: updating configuration: command=RemoveServer server-id=server-e4455e64-1d13-edfe-ee2b-63eb11be423a server-addr= servers="[{Suffrage:Voter ID:server-a0f3d8cd-d65f-6f36-367d-feccffaa7684 Address:a0f3d8cd-d65f-6f36-367d-feccffaa7684} {Suffrage:Voter ID:server-38e289c3-a54b-6e78-9ad8-d98afdb9fabd Address:38e289c3-a54b-6e78-9ad8-d98afdb9fabd}]"
2026-02-28T09:03:40.307Z [INFO]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: removed peer, stopping replication: peer=server-e4455e64-1d13-edfe-ee2b-63eb11be423a last-index=3
2026-02-28T09:03:40.308Z [INFO]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: aborting pipeline replication: peer="{Voter server-e4455e64-1d13-edfe-ee2b-63eb11be423a e4455e64-1d13-edfe-ee2b-63eb11be423a}"
2026-02-28T09:03:40.406Z [WARN]  server-e4455e64-1d13-edfe-ee2b-63eb11be423a: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2026-02-28T09:03:40.413Z [INFO]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: updating configuration: command=AddVoter server-id=server-e4455e64-1d13-edfe-ee2b-63eb11be423a server-addr=e4455e64-1d13-edfe-ee2b-63eb11be423a servers="[{Suffrage:Voter ID:server-a0f3d8cd-d65f-6f36-367d-feccffaa7684 Address:a0f3d8cd-d65f-6f36-367d-feccffaa7684} {Suffrage:Voter ID:server-38e289c3-a54b-6e78-9ad8-d98afdb9fabd Address:38e289c3-a54b-6e78-9ad8-d98afdb9fabd} {Suffrage:Voter ID:server-e4455e64-1d13-edfe-ee2b-63eb11be423a Address:e4455e64-1d13-edfe-ee2b-63eb11be423a}]"
2026-02-28T09:03:40.413Z [INFO]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: added peer, starting replication: peer=server-e4455e64-1d13-edfe-ee2b-63eb11be423a
2026-02-28T09:03:40.413Z [WARN]  server-e4455e64-1d13-edfe-ee2b-63eb11be423a: failed to get previous log: previous-index=4 last-index=3 error="log not found"
2026-02-28T09:03:40.413Z [WARN]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: appendEntries rejected, sending older logs: peer="{Voter server-e4455e64-1d13-edfe-ee2b-63eb11be423a e4455e64-1d13-edfe-ee2b-63eb11be423a}" next=4
2026-02-28T09:03:40.413Z [INFO]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: pipelining replication: peer="{Voter server-e4455e64-1d13-edfe-ee2b-63eb11be423a e4455e64-1d13-edfe-ee2b-63eb11be423a}"
2026-02-28T09:03:40.413Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:40.519Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:40.413860262 +0000 UTC m=+98.377656801 (1 nodes), 920ns from start of poll, 383.0662ms from cluster start. Timeout at 2026-02-28 09:03:40.518872702 +0000 UTC m=+98.482669241, 105.01244ms after stability
2026-02-28T09:03:40.519Z [INFO]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: aborting pipeline replication: peer="{Voter server-38e289c3-a54b-6e78-9ad8-d98afdb9fabd 38e289c3-a54b-6e78-9ad8-d98afdb9fabd}"
2026-02-28T09:03:40.519Z [INFO]  server-a0f3d8cd-d65f-6f36-367d-feccffaa7684: aborting pipeline replication: peer="{Voter server-e4455e64-1d13-edfe-ee2b-63eb11be423a e4455e64-1d13-edfe-ee2b-63eb11be423a}"
--- PASS: TestRaft_ReJoinFollower (0.49s)
=== RUN   TestRaft_LeaderLeaseExpire
2026-02-28T09:03:40.520Z [INFO]  server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 Address:92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83} {Suffrage:Voter ID:server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433 Address:2eeb96a3-25cf-b0ff-fef0-7f74ea46a433}]"
2026-02-28T09:03:40.520Z [INFO]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 Address:92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83} {Suffrage:Voter ID:server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433 Address:2eeb96a3-25cf-b0ff-fef0-7f74ea46a433}]"
2026-02-28T09:03:40.520Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:40.520Z [INFO]  server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83: entering follower state: follower="Node at 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 [Follower]" leader-address= leader-id=
2026-02-28T09:03:40.520Z [INFO]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: entering follower state: follower="Node at 2eeb96a3-25cf-b0ff-fef0-7f74ea46a433 [Follower]" leader-address= leader-id=
2026-02-28T09:03:40.572Z [WARN]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:40.572Z [INFO]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: entering candidate state: node="Node at 2eeb96a3-25cf-b0ff-fef0-7f74ea46a433 [Candidate]" term=2
2026-02-28T09:03:40.572Z [INFO]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: election won: term=2 tally=2
2026-02-28T09:03:40.572Z [INFO]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: entering leader state: leader="Node at 2eeb96a3-25cf-b0ff-fef0-7f74ea46a433 [Leader]"
2026-02-28T09:03:40.572Z [INFO]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: added peer, starting replication: peer=server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83
2026-02-28T09:03:40.572Z [INFO]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: pipelining replication: peer="{Voter server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83}"
2026-02-28T09:03:40.678Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:40.572769422 +0000 UTC m=+98.536565961 (1 nodes), 52.02436ms from start of poll, 52.34748ms from cluster start. Timeout at 2026-02-28 09:03:40.677772022 +0000 UTC m=+98.641568561, 105.0026ms after stability
2026-02-28T09:03:40.683Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:40.789Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:40.683769782 +0000 UTC m=+98.647566321 (1 nodes), 2.24µs from start of poll, 163.34784ms from cluster start. Timeout at 2026-02-28 09:03:40.788784222 +0000 UTC m=+98.752580761, 105.01444ms after stability
    raft_test.go:1799: [INFO] Disconnecting Node at 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 [Follower]
2026-02-28T09:03:40.796Z [ERROR] server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: failed to heartbeat to: peer=92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 backoff time=10ms error="failed to connect to peer: 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83"
2026-02-28T09:03:40.797Z [ERROR] server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: failed to pipeline appendEntries: peer="{Voter server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83}" error="append pipeline closed"
2026-02-28T09:03:40.797Z [INFO]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: aborting pipeline replication: peer="{Voter server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83}"
2026-02-28T09:03:40.806Z [ERROR] server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: failed to appendEntries to: peer="{Voter server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83}" error="failed to connect to peer: 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83"
2026-02-28T09:03:40.812Z [ERROR] server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: failed to heartbeat to: peer=92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 backoff time=10ms error="failed to connect to peer: 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83"
2026-02-28T09:03:40.826Z [ERROR] server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: failed to appendEntries to: peer="{Voter server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83}" error="failed to connect to peer: 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83"
2026-02-28T09:03:40.832Z [ERROR] server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: failed to heartbeat to: peer=92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 backoff time=10ms error="failed to connect to peer: 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83"
2026-02-28T09:03:40.838Z [WARN]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: failed to contact: server-id=server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 time=50.0354ms
2026-02-28T09:03:40.838Z [WARN]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: failed to contact quorum of nodes, stepping down
2026-02-28T09:03:40.838Z [INFO]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: entering follower state: follower="Node at 2eeb96a3-25cf-b0ff-fef0-7f74ea46a433 [Follower]" leader-address= leader-id=
2026-02-28T09:03:40.838Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:40.844Z [WARN]  server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83: heartbeat timeout reached, starting election: last-leader-addr=2eeb96a3-25cf-b0ff-fef0-7f74ea46a433 last-leader-id=server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433
2026-02-28T09:03:40.844Z [INFO]  server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83: entering candidate state: node="Node at 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 [Candidate]" term=3
2026-02-28T09:03:40.844Z [ERROR] server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: failed to appendEntries to: peer="{Voter server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83}" error="failed to connect to peer: 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83"
2026-02-28T09:03:40.844Z [ERROR] server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83: failed to make requestVote RPC: target="{Voter server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433 2eeb96a3-25cf-b0ff-fef0-7f74ea46a433}" error="failed to connect to peer: 2eeb96a3-25cf-b0ff-fef0-7f74ea46a433" term=3
2026-02-28T09:03:40.897Z [WARN]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:40.897Z [INFO]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: entering candidate state: node="Node at 2eeb96a3-25cf-b0ff-fef0-7f74ea46a433 [Candidate]" term=3
2026-02-28T09:03:40.897Z [ERROR] server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: failed to make requestVote RPC: target="{Voter server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83}" error="failed to connect to peer: 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83" term=3
2026-02-28T09:03:40.908Z [WARN]  server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83: Election timeout reached, restarting election
2026-02-28T09:03:40.908Z [INFO]  server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83: entering candidate state: node="Node at 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 [Candidate]" term=4
2026-02-28T09:03:40.908Z [ERROR] server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83: failed to make requestVote RPC: target="{Voter server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433 2eeb96a3-25cf-b0ff-fef0-7f74ea46a433}" error="failed to connect to peer: 2eeb96a3-25cf-b0ff-fef0-7f74ea46a433" term=4
2026-02-28T09:03:40.963Z [WARN]  server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83: Election timeout reached, restarting election
2026-02-28T09:03:40.963Z [INFO]  server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83: entering candidate state: node="Node at 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 [Candidate]" term=5
2026-02-28T09:03:40.963Z [ERROR] server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83: failed to make requestVote RPC: target="{Voter server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433 2eeb96a3-25cf-b0ff-fef0-7f74ea46a433}" error="failed to connect to peer: 2eeb96a3-25cf-b0ff-fef0-7f74ea46a433" term=5
2026-02-28T09:03:40.981Z [WARN]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: Election timeout reached, restarting election
2026-02-28T09:03:40.981Z [INFO]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: entering candidate state: node="Node at 2eeb96a3-25cf-b0ff-fef0-7f74ea46a433 [Candidate]" term=4
2026-02-28T09:03:40.981Z [ERROR] server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: failed to make requestVote RPC: target="{Voter server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83}" error="failed to connect to peer: 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83" term=4
2026-02-28T09:03:41.002Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:40.897289702 +0000 UTC m=+98.861086281 (0 nodes), 59.0404ms from start of poll, 376.8678ms from cluster start. Timeout at 2026-02-28 09:03:41.002294102 +0000 UTC m=+98.966090641, 105.00436ms after stability
2026-02-28T09:03:41.040Z [WARN]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: Election timeout reached, restarting election
2026-02-28T09:03:41.040Z [INFO]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: entering candidate state: node="Node at 2eeb96a3-25cf-b0ff-fef0-7f74ea46a433 [Candidate]" term=5
2026-02-28T09:03:41.040Z [ERROR] server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: failed to make requestVote RPC: target="{Voter server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83}" error="failed to connect to peer: 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83" term=5
2026-02-28T09:03:41.055Z [WARN]  server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83: Election timeout reached, restarting election
2026-02-28T09:03:41.055Z [INFO]  server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83: entering candidate state: node="Node at 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 [Candidate]" term=6
2026-02-28T09:03:41.055Z [ERROR] server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83: failed to make requestVote RPC: target="{Voter server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433 2eeb96a3-25cf-b0ff-fef0-7f74ea46a433}" error="failed to connect to peer: 2eeb96a3-25cf-b0ff-fef0-7f74ea46a433" term=6
2026-02-28T09:03:41.102Z [WARN]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: Election timeout reached, restarting election
2026-02-28T09:03:41.102Z [INFO]  server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: entering candidate state: node="Node at 2eeb96a3-25cf-b0ff-fef0-7f74ea46a433 [Candidate]" term=6
2026-02-28T09:03:41.102Z [ERROR] server-2eeb96a3-25cf-b0ff-fef0-7f74ea46a433: failed to make requestVote RPC: target="{Voter server-92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83}" error="failed to connect to peer: 92f1b925-ba8c-9e4b-24e4-adfc0e6e1e83" term=6
--- PASS: TestRaft_LeaderLeaseExpire (0.59s)
=== RUN   TestRaft_Barrier
2026-02-28T09:03:41.108Z [INFO]  server-b2a8c002-8312-082b-df03-a93aca5b695a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2a8c002-8312-082b-df03-a93aca5b695a Address:b2a8c002-8312-082b-df03-a93aca5b695a} {Suffrage:Voter ID:server-2e336d72-1300-67a0-a5b6-61a241c6d683 Address:2e336d72-1300-67a0-a5b6-61a241c6d683} {Suffrage:Voter ID:server-aa286a03-c1db-c5bf-4146-74ce055cf234 Address:aa286a03-c1db-c5bf-4146-74ce055cf234}]"
2026-02-28T09:03:41.108Z [INFO]  server-2e336d72-1300-67a0-a5b6-61a241c6d683: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2a8c002-8312-082b-df03-a93aca5b695a Address:b2a8c002-8312-082b-df03-a93aca5b695a} {Suffrage:Voter ID:server-2e336d72-1300-67a0-a5b6-61a241c6d683 Address:2e336d72-1300-67a0-a5b6-61a241c6d683} {Suffrage:Voter ID:server-aa286a03-c1db-c5bf-4146-74ce055cf234 Address:aa286a03-c1db-c5bf-4146-74ce055cf234}]"
2026-02-28T09:03:41.108Z [INFO]  server-b2a8c002-8312-082b-df03-a93aca5b695a: entering follower state: follower="Node at b2a8c002-8312-082b-df03-a93aca5b695a [Follower]" leader-address= leader-id=
2026-02-28T09:03:41.108Z [INFO]  server-2e336d72-1300-67a0-a5b6-61a241c6d683: entering follower state: follower="Node at 2e336d72-1300-67a0-a5b6-61a241c6d683 [Follower]" leader-address= leader-id=
2026-02-28T09:03:41.108Z [INFO]  server-aa286a03-c1db-c5bf-4146-74ce055cf234: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2a8c002-8312-082b-df03-a93aca5b695a Address:b2a8c002-8312-082b-df03-a93aca5b695a} {Suffrage:Voter ID:server-2e336d72-1300-67a0-a5b6-61a241c6d683 Address:2e336d72-1300-67a0-a5b6-61a241c6d683} {Suffrage:Voter ID:server-aa286a03-c1db-c5bf-4146-74ce055cf234 Address:aa286a03-c1db-c5bf-4146-74ce055cf234}]"
2026-02-28T09:03:41.108Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:41.108Z [INFO]  server-aa286a03-c1db-c5bf-4146-74ce055cf234: entering follower state: follower="Node at aa286a03-c1db-c5bf-4146-74ce055cf234 [Follower]" leader-address= leader-id=
2026-02-28T09:03:41.163Z [WARN]  server-aa286a03-c1db-c5bf-4146-74ce055cf234: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:41.163Z [INFO]  server-aa286a03-c1db-c5bf-4146-74ce055cf234: entering candidate state: node="Node at aa286a03-c1db-c5bf-4146-74ce055cf234 [Candidate]" term=2
2026-02-28T09:03:41.163Z [INFO]  server-aa286a03-c1db-c5bf-4146-74ce055cf234: election won: term=2 tally=2
2026-02-28T09:03:41.163Z [INFO]  server-aa286a03-c1db-c5bf-4146-74ce055cf234: entering leader state: leader="Node at aa286a03-c1db-c5bf-4146-74ce055cf234 [Leader]"
2026-02-28T09:03:41.163Z [INFO]  server-aa286a03-c1db-c5bf-4146-74ce055cf234: added peer, starting replication: peer=server-b2a8c002-8312-082b-df03-a93aca5b695a
2026-02-28T09:03:41.163Z [INFO]  server-aa286a03-c1db-c5bf-4146-74ce055cf234: added peer, starting replication: peer=server-2e336d72-1300-67a0-a5b6-61a241c6d683
2026-02-28T09:03:41.163Z [INFO]  server-aa286a03-c1db-c5bf-4146-74ce055cf234: pipelining replication: peer="{Voter server-2e336d72-1300-67a0-a5b6-61a241c6d683 2e336d72-1300-67a0-a5b6-61a241c6d683}"
2026-02-28T09:03:41.163Z [INFO]  server-aa286a03-c1db-c5bf-4146-74ce055cf234: pipelining replication: peer="{Voter server-b2a8c002-8312-082b-df03-a93aca5b695a b2a8c002-8312-082b-df03-a93aca5b695a}"
2026-02-28T09:03:41.269Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:41.163408742 +0000 UTC m=+99.127205281 (1 nodes), 54.53052ms from start of poll, 55.0752ms from cluster start. Timeout at 2026-02-28 09:03:41.268420062 +0000 UTC m=+99.232216601, 105.01132ms after stability
2026-02-28T09:03:41.285Z [INFO]  server-aa286a03-c1db-c5bf-4146-74ce055cf234: aborting pipeline replication: peer="{Voter server-2e336d72-1300-67a0-a5b6-61a241c6d683 2e336d72-1300-67a0-a5b6-61a241c6d683}"
2026-02-28T09:03:41.285Z [INFO]  server-aa286a03-c1db-c5bf-4146-74ce055cf234: aborting pipeline replication: peer="{Voter server-b2a8c002-8312-082b-df03-a93aca5b695a b2a8c002-8312-082b-df03-a93aca5b695a}"
--- PASS: TestRaft_Barrier (0.18s)
=== RUN   TestRaft_VerifyLeader
2026-02-28T09:03:41.286Z [INFO]  server-3a3d3e40-1753-b036-21b8-67ade1549eeb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a3d3e40-1753-b036-21b8-67ade1549eeb Address:3a3d3e40-1753-b036-21b8-67ade1549eeb} {Suffrage:Voter ID:server-012c9276-3ddc-bb6d-52e2-0a403fbd6b47 Address:012c9276-3ddc-bb6d-52e2-0a403fbd6b47} {Suffrage:Voter ID:server-d2ec6ccf-c51c-1e2f-2c84-b50e27159ce2 Address:d2ec6ccf-c51c-1e2f-2c84-b50e27159ce2}]"
2026-02-28T09:03:41.286Z [INFO]  server-3a3d3e40-1753-b036-21b8-67ade1549eeb: entering follower state: follower="Node at 3a3d3e40-1753-b036-21b8-67ade1549eeb [Follower]" leader-address= leader-id=
2026-02-28T09:03:41.286Z [INFO]  server-012c9276-3ddc-bb6d-52e2-0a403fbd6b47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a3d3e40-1753-b036-21b8-67ade1549eeb Address:3a3d3e40-1753-b036-21b8-67ade1549eeb} {Suffrage:Voter ID:server-012c9276-3ddc-bb6d-52e2-0a403fbd6b47 Address:012c9276-3ddc-bb6d-52e2-0a403fbd6b47} {Suffrage:Voter ID:server-d2ec6ccf-c51c-1e2f-2c84-b50e27159ce2 Address:d2ec6ccf-c51c-1e2f-2c84-b50e27159ce2}]"
2026-02-28T09:03:41.286Z [INFO]  server-012c9276-3ddc-bb6d-52e2-0a403fbd6b47: entering follower state: follower="Node at 012c9276-3ddc-bb6d-52e2-0a403fbd6b47 [Follower]" leader-address= leader-id=
2026-02-28T09:03:41.286Z [INFO]  server-d2ec6ccf-c51c-1e2f-2c84-b50e27159ce2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a3d3e40-1753-b036-21b8-67ade1549eeb Address:3a3d3e40-1753-b036-21b8-67ade1549eeb} {Suffrage:Voter ID:server-012c9276-3ddc-bb6d-52e2-0a403fbd6b47 Address:012c9276-3ddc-bb6d-52e2-0a403fbd6b47} {Suffrage:Voter ID:server-d2ec6ccf-c51c-1e2f-2c84-b50e27159ce2 Address:d2ec6ccf-c51c-1e2f-2c84-b50e27159ce2}]"
2026-02-28T09:03:41.286Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:41.286Z [INFO]  server-d2ec6ccf-c51c-1e2f-2c84-b50e27159ce2: entering follower state: follower="Node at d2ec6ccf-c51c-1e2f-2c84-b50e27159ce2 [Follower]" leader-address= leader-id=
2026-02-28T09:03:41.365Z [WARN]  server-3a3d3e40-1753-b036-21b8-67ade1549eeb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:41.365Z [INFO]  server-3a3d3e40-1753-b036-21b8-67ade1549eeb: entering candidate state: node="Node at 3a3d3e40-1753-b036-21b8-67ade1549eeb [Candidate]" term=2
2026-02-28T09:03:41.365Z [INFO]  server-3a3d3e40-1753-b036-21b8-67ade1549eeb: election won: term=2 tally=2
2026-02-28T09:03:41.365Z [INFO]  server-3a3d3e40-1753-b036-21b8-67ade1549eeb: entering leader state: leader="Node at 3a3d3e40-1753-b036-21b8-67ade1549eeb [Leader]"
2026-02-28T09:03:41.365Z [INFO]  server-3a3d3e40-1753-b036-21b8-67ade1549eeb: added peer, starting replication: peer=server-012c9276-3ddc-bb6d-52e2-0a403fbd6b47
2026-02-28T09:03:41.365Z [INFO]  server-3a3d3e40-1753-b036-21b8-67ade1549eeb: added peer, starting replication: peer=server-d2ec6ccf-c51c-1e2f-2c84-b50e27159ce2
2026-02-28T09:03:41.365Z [INFO]  server-3a3d3e40-1753-b036-21b8-67ade1549eeb: pipelining replication: peer="{Voter server-d2ec6ccf-c51c-1e2f-2c84-b50e27159ce2 d2ec6ccf-c51c-1e2f-2c84-b50e27159ce2}"
2026-02-28T09:03:41.366Z [INFO]  server-3a3d3e40-1753-b036-21b8-67ade1549eeb: pipelining replication: peer="{Voter server-012c9276-3ddc-bb6d-52e2-0a403fbd6b47 012c9276-3ddc-bb6d-52e2-0a403fbd6b47}"
2026-02-28T09:03:41.471Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:41.365926182 +0000 UTC m=+99.329722721 (1 nodes), 79.00792ms from start of poll, 79.68876ms from cluster start. Timeout at 2026-02-28 09:03:41.470933062 +0000 UTC m=+99.434729561, 105.00684ms after stability
2026-02-28T09:03:41.471Z [INFO]  server-3a3d3e40-1753-b036-21b8-67ade1549eeb: aborting pipeline replication: peer="{Voter server-d2ec6ccf-c51c-1e2f-2c84-b50e27159ce2 d2ec6ccf-c51c-1e2f-2c84-b50e27159ce2}"
2026-02-28T09:03:41.471Z [INFO]  server-3a3d3e40-1753-b036-21b8-67ade1549eeb: aborting pipeline replication: peer="{Voter server-012c9276-3ddc-bb6d-52e2-0a403fbd6b47 012c9276-3ddc-bb6d-52e2-0a403fbd6b47}"
--- PASS: TestRaft_VerifyLeader (0.19s)
=== RUN   TestRaft_VerifyLeader_Single
2026-02-28T09:03:41.472Z [INFO]  server-990c18cd-f5dc-76f2-6a91-679117d59d4e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-990c18cd-f5dc-76f2-6a91-679117d59d4e Address:990c18cd-f5dc-76f2-6a91-679117d59d4e}]"
2026-02-28T09:03:41.472Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:41.472Z [INFO]  server-990c18cd-f5dc-76f2-6a91-679117d59d4e: entering follower state: follower="Node at 990c18cd-f5dc-76f2-6a91-679117d59d4e [Follower]" leader-address= leader-id=
2026-02-28T09:03:41.555Z [WARN]  server-990c18cd-f5dc-76f2-6a91-679117d59d4e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:41.555Z [INFO]  server-990c18cd-f5dc-76f2-6a91-679117d59d4e: entering candidate state: node="Node at 990c18cd-f5dc-76f2-6a91-679117d59d4e [Candidate]" term=2
2026-02-28T09:03:41.555Z [INFO]  server-990c18cd-f5dc-76f2-6a91-679117d59d4e: election won: term=2 tally=1
2026-02-28T09:03:41.555Z [INFO]  server-990c18cd-f5dc-76f2-6a91-679117d59d4e: entering leader state: leader="Node at 990c18cd-f5dc-76f2-6a91-679117d59d4e [Leader]"
2026-02-28T09:03:41.661Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:41.555424582 +0000 UTC m=+99.519221161 (1 nodes), 83.0126ms from start of poll, 83.22344ms from cluster start. Timeout at 2026-02-28 09:03:41.660426982 +0000 UTC m=+99.624223521, 105.00236ms after stability
--- PASS: TestRaft_VerifyLeader_Single (0.19s)
=== RUN   TestRaft_VerifyLeader_Fail
2026-02-28T09:03:41.662Z [INFO]  server-0aa48975-941c-4940-50cf-d98b5af2fcdd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0aa48975-941c-4940-50cf-d98b5af2fcdd Address:0aa48975-941c-4940-50cf-d98b5af2fcdd} {Suffrage:Voter ID:server-68f16523-0405-5c3b-d2a7-4c9f308c9892 Address:68f16523-0405-5c3b-d2a7-4c9f308c9892}]"
2026-02-28T09:03:41.662Z [INFO]  server-0aa48975-941c-4940-50cf-d98b5af2fcdd: entering follower state: follower="Node at 0aa48975-941c-4940-50cf-d98b5af2fcdd [Follower]" leader-address= leader-id=
2026-02-28T09:03:41.662Z [INFO]  server-68f16523-0405-5c3b-d2a7-4c9f308c9892: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0aa48975-941c-4940-50cf-d98b5af2fcdd Address:0aa48975-941c-4940-50cf-d98b5af2fcdd} {Suffrage:Voter ID:server-68f16523-0405-5c3b-d2a7-4c9f308c9892 Address:68f16523-0405-5c3b-d2a7-4c9f308c9892}]"
2026-02-28T09:03:41.662Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:41.662Z [INFO]  server-68f16523-0405-5c3b-d2a7-4c9f308c9892: entering follower state: follower="Node at 68f16523-0405-5c3b-d2a7-4c9f308c9892 [Follower]" leader-address= leader-id=
2026-02-28T09:03:41.723Z [WARN]  server-0aa48975-941c-4940-50cf-d98b5af2fcdd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:41.723Z [INFO]  server-0aa48975-941c-4940-50cf-d98b5af2fcdd: entering candidate state: node="Node at 0aa48975-941c-4940-50cf-d98b5af2fcdd [Candidate]" term=2
2026-02-28T09:03:41.723Z [INFO]  server-0aa48975-941c-4940-50cf-d98b5af2fcdd: election won: term=2 tally=2
2026-02-28T09:03:41.723Z [INFO]  server-0aa48975-941c-4940-50cf-d98b5af2fcdd: entering leader state: leader="Node at 0aa48975-941c-4940-50cf-d98b5af2fcdd [Leader]"
2026-02-28T09:03:41.723Z [INFO]  server-0aa48975-941c-4940-50cf-d98b5af2fcdd: added peer, starting replication: peer=server-68f16523-0405-5c3b-d2a7-4c9f308c9892
2026-02-28T09:03:41.724Z [INFO]  server-0aa48975-941c-4940-50cf-d98b5af2fcdd: pipelining replication: peer="{Voter server-68f16523-0405-5c3b-d2a7-4c9f308c9892 68f16523-0405-5c3b-d2a7-4c9f308c9892}"
2026-02-28T09:03:41.829Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:41.723886902 +0000 UTC m=+99.687683441 (1 nodes), 61.38432ms from start of poll, 61.73712ms from cluster start. Timeout at 2026-02-28 09:03:41.828890502 +0000 UTC m=+99.792687041, 105.0036ms after stability
2026-02-28T09:03:41.829Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:41.935Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:41.829191862 +0000 UTC m=+99.792988401 (1 nodes), 1.12µs from start of poll, 167.04208ms from cluster start. Timeout at 2026-02-28 09:03:41.934210622 +0000 UTC m=+99.898007161, 105.01876ms after stability
2026-02-28T09:03:41.944Z [ERROR] server-0aa48975-941c-4940-50cf-d98b5af2fcdd: peer has newer term, stopping replication: peer="{Voter server-68f16523-0405-5c3b-d2a7-4c9f308c9892 68f16523-0405-5c3b-d2a7-4c9f308c9892}"
2026-02-28T09:03:41.944Z [INFO]  server-0aa48975-941c-4940-50cf-d98b5af2fcdd: aborting pipeline replication: peer="{Voter server-68f16523-0405-5c3b-d2a7-4c9f308c9892 68f16523-0405-5c3b-d2a7-4c9f308c9892}"
2026-02-28T09:03:41.944Z [INFO]  server-0aa48975-941c-4940-50cf-d98b5af2fcdd: entering follower state: follower="Node at 0aa48975-941c-4940-50cf-d98b5af2fcdd [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VerifyLeader_Fail (0.28s)
=== RUN   TestRaft_VerifyLeader_PartialConnect
2026-02-28T09:03:41.945Z [INFO]  server-a9327a19-7a3e-1d4e-25b5-a57d335594a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9327a19-7a3e-1d4e-25b5-a57d335594a1 Address:a9327a19-7a3e-1d4e-25b5-a57d335594a1} {Suffrage:Voter ID:server-6e06e7e3-836d-448c-5528-77ba5afe260b Address:6e06e7e3-836d-448c-5528-77ba5afe260b} {Suffrage:Voter ID:server-250f5be9-6454-dd13-5cf2-ff5806068f72 Address:250f5be9-6454-dd13-5cf2-ff5806068f72}]"
2026-02-28T09:03:41.945Z [INFO]  server-6e06e7e3-836d-448c-5528-77ba5afe260b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9327a19-7a3e-1d4e-25b5-a57d335594a1 Address:a9327a19-7a3e-1d4e-25b5-a57d335594a1} {Suffrage:Voter ID:server-6e06e7e3-836d-448c-5528-77ba5afe260b Address:6e06e7e3-836d-448c-5528-77ba5afe260b} {Suffrage:Voter ID:server-250f5be9-6454-dd13-5cf2-ff5806068f72 Address:250f5be9-6454-dd13-5cf2-ff5806068f72}]"
2026-02-28T09:03:41.945Z [INFO]  server-a9327a19-7a3e-1d4e-25b5-a57d335594a1: entering follower state: follower="Node at a9327a19-7a3e-1d4e-25b5-a57d335594a1 [Follower]" leader-address= leader-id=
2026-02-28T09:03:41.945Z [INFO]  server-6e06e7e3-836d-448c-5528-77ba5afe260b: entering follower state: follower="Node at 6e06e7e3-836d-448c-5528-77ba5afe260b [Follower]" leader-address= leader-id=
2026-02-28T09:03:41.945Z [INFO]  server-250f5be9-6454-dd13-5cf2-ff5806068f72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9327a19-7a3e-1d4e-25b5-a57d335594a1 Address:a9327a19-7a3e-1d4e-25b5-a57d335594a1} {Suffrage:Voter ID:server-6e06e7e3-836d-448c-5528-77ba5afe260b Address:6e06e7e3-836d-448c-5528-77ba5afe260b} {Suffrage:Voter ID:server-250f5be9-6454-dd13-5cf2-ff5806068f72 Address:250f5be9-6454-dd13-5cf2-ff5806068f72}]"
2026-02-28T09:03:41.945Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:41.945Z [INFO]  server-250f5be9-6454-dd13-5cf2-ff5806068f72: entering follower state: follower="Node at 250f5be9-6454-dd13-5cf2-ff5806068f72 [Follower]" leader-address= leader-id=
2026-02-28T09:03:42.007Z [WARN]  server-6e06e7e3-836d-448c-5528-77ba5afe260b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:42.007Z [INFO]  server-6e06e7e3-836d-448c-5528-77ba5afe260b: entering candidate state: node="Node at 6e06e7e3-836d-448c-5528-77ba5afe260b [Candidate]" term=2
2026-02-28T09:03:42.007Z [INFO]  server-6e06e7e3-836d-448c-5528-77ba5afe260b: election won: term=2 tally=2
2026-02-28T09:03:42.007Z [INFO]  server-6e06e7e3-836d-448c-5528-77ba5afe260b: entering leader state: leader="Node at 6e06e7e3-836d-448c-5528-77ba5afe260b [Leader]"
2026-02-28T09:03:42.007Z [INFO]  server-6e06e7e3-836d-448c-5528-77ba5afe260b: added peer, starting replication: peer=server-a9327a19-7a3e-1d4e-25b5-a57d335594a1
2026-02-28T09:03:42.007Z [INFO]  server-6e06e7e3-836d-448c-5528-77ba5afe260b: added peer, starting replication: peer=server-250f5be9-6454-dd13-5cf2-ff5806068f72
2026-02-28T09:03:42.007Z [INFO]  server-6e06e7e3-836d-448c-5528-77ba5afe260b: pipelining replication: peer="{Voter server-250f5be9-6454-dd13-5cf2-ff5806068f72 250f5be9-6454-dd13-5cf2-ff5806068f72}"
2026-02-28T09:03:42.007Z [INFO]  server-6e06e7e3-836d-448c-5528-77ba5afe260b: pipelining replication: peer="{Voter server-a9327a19-7a3e-1d4e-25b5-a57d335594a1 a9327a19-7a3e-1d4e-25b5-a57d335594a1}"
2026-02-28T09:03:42.112Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:42.007413702 +0000 UTC m=+99.971210241 (1 nodes), 61.52012ms from start of poll, 61.99992ms from cluster start. Timeout at 2026-02-28 09:03:42.112415022 +0000 UTC m=+100.076211561, 105.00132ms after stability
2026-02-28T09:03:42.117Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:42.222Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:42.117623022 +0000 UTC m=+100.081419561 (2 nodes), 3.88µs from start of poll, 172.20924ms from cluster start. Timeout at 2026-02-28 09:03:42.222643862 +0000 UTC m=+100.186440401, 105.02084ms after stability
    raft_test.go:1968: [INFO] Disconnecting Node at a9327a19-7a3e-1d4e-25b5-a57d335594a1 [Follower]
2026-02-28T09:03:42.222Z [ERROR] server-6e06e7e3-836d-448c-5528-77ba5afe260b: failed to heartbeat to: peer=a9327a19-7a3e-1d4e-25b5-a57d335594a1 backoff time=10ms error="failed to connect to peer: a9327a19-7a3e-1d4e-25b5-a57d335594a1"
2026-02-28T09:03:42.223Z [INFO]  server-6e06e7e3-836d-448c-5528-77ba5afe260b: aborting pipeline replication: peer="{Voter server-250f5be9-6454-dd13-5cf2-ff5806068f72 250f5be9-6454-dd13-5cf2-ff5806068f72}"
2026-02-28T09:03:42.223Z [INFO]  server-6e06e7e3-836d-448c-5528-77ba5afe260b: aborting pipeline replication: peer="{Voter server-a9327a19-7a3e-1d4e-25b5-a57d335594a1 a9327a19-7a3e-1d4e-25b5-a57d335594a1}"
--- PASS: TestRaft_VerifyLeader_PartialConnect (0.28s)
=== RUN   TestRaft_NotifyCh
2026-02-28T09:03:42.223Z [INFO]  server-5c24a12f-709f-e58c-cca5-1c02b9ed6b62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c24a12f-709f-e58c-cca5-1c02b9ed6b62 Address:5c24a12f-709f-e58c-cca5-1c02b9ed6b62}]"
2026-02-28T09:03:42.223Z [INFO]  server-5c24a12f-709f-e58c-cca5-1c02b9ed6b62: entering follower state: follower="Node at 5c24a12f-709f-e58c-cca5-1c02b9ed6b62 [Follower]" leader-address= leader-id=
2026-02-28T09:03:42.277Z [WARN]  server-5c24a12f-709f-e58c-cca5-1c02b9ed6b62: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:42.277Z [INFO]  server-5c24a12f-709f-e58c-cca5-1c02b9ed6b62: entering candidate state: node="Node at 5c24a12f-709f-e58c-cca5-1c02b9ed6b62 [Candidate]" term=2
2026-02-28T09:03:42.277Z [INFO]  server-5c24a12f-709f-e58c-cca5-1c02b9ed6b62: election won: term=2 tally=1
2026-02-28T09:03:42.277Z [INFO]  server-5c24a12f-709f-e58c-cca5-1c02b9ed6b62: entering leader state: leader="Node at 5c24a12f-709f-e58c-cca5-1c02b9ed6b62 [Leader]"
--- PASS: TestRaft_NotifyCh (0.05s)
=== RUN   TestRaft_AppendEntry
2026-02-28T09:03:42.278Z [INFO]  server-ba41d857-7d3c-2683-28f8-5b338e63321e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba41d857-7d3c-2683-28f8-5b338e63321e Address:ba41d857-7d3c-2683-28f8-5b338e63321e} {Suffrage:Voter ID:server-d7837011-d183-a1dc-c285-1dbceb9bb9e0 Address:d7837011-d183-a1dc-c285-1dbceb9bb9e0} {Suffrage:Voter ID:server-7d3a4a2b-1ca4-b17f-57e6-016365dcccd3 Address:7d3a4a2b-1ca4-b17f-57e6-016365dcccd3}]"
2026-02-28T09:03:42.278Z [INFO]  server-d7837011-d183-a1dc-c285-1dbceb9bb9e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba41d857-7d3c-2683-28f8-5b338e63321e Address:ba41d857-7d3c-2683-28f8-5b338e63321e} {Suffrage:Voter ID:server-d7837011-d183-a1dc-c285-1dbceb9bb9e0 Address:d7837011-d183-a1dc-c285-1dbceb9bb9e0} {Suffrage:Voter ID:server-7d3a4a2b-1ca4-b17f-57e6-016365dcccd3 Address:7d3a4a2b-1ca4-b17f-57e6-016365dcccd3}]"
2026-02-28T09:03:42.278Z [INFO]  server-d7837011-d183-a1dc-c285-1dbceb9bb9e0: entering follower state: follower="Node at d7837011-d183-a1dc-c285-1dbceb9bb9e0 [Follower]" leader-address= leader-id=
2026-02-28T09:03:42.278Z [INFO]  server-7d3a4a2b-1ca4-b17f-57e6-016365dcccd3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba41d857-7d3c-2683-28f8-5b338e63321e Address:ba41d857-7d3c-2683-28f8-5b338e63321e} {Suffrage:Voter ID:server-d7837011-d183-a1dc-c285-1dbceb9bb9e0 Address:d7837011-d183-a1dc-c285-1dbceb9bb9e0} {Suffrage:Voter ID:server-7d3a4a2b-1ca4-b17f-57e6-016365dcccd3 Address:7d3a4a2b-1ca4-b17f-57e6-016365dcccd3}]"
2026-02-28T09:03:42.278Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:42.278Z [INFO]  server-ba41d857-7d3c-2683-28f8-5b338e63321e: entering follower state: follower="Node at ba41d857-7d3c-2683-28f8-5b338e63321e [Follower]" leader-address= leader-id=
2026-02-28T09:03:42.278Z [INFO]  server-7d3a4a2b-1ca4-b17f-57e6-016365dcccd3: entering follower state: follower="Node at 7d3a4a2b-1ca4-b17f-57e6-016365dcccd3 [Follower]" leader-address= leader-id=
2026-02-28T09:03:42.340Z [WARN]  server-d7837011-d183-a1dc-c285-1dbceb9bb9e0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:42.340Z [INFO]  server-d7837011-d183-a1dc-c285-1dbceb9bb9e0: entering candidate state: node="Node at d7837011-d183-a1dc-c285-1dbceb9bb9e0 [Candidate]" term=2
2026-02-28T09:03:42.340Z [INFO]  server-d7837011-d183-a1dc-c285-1dbceb9bb9e0: election won: term=2 tally=2
2026-02-28T09:03:42.340Z [INFO]  server-d7837011-d183-a1dc-c285-1dbceb9bb9e0: entering leader state: leader="Node at d7837011-d183-a1dc-c285-1dbceb9bb9e0 [Leader]"
2026-02-28T09:03:42.340Z [INFO]  server-d7837011-d183-a1dc-c285-1dbceb9bb9e0: added peer, starting replication: peer=server-ba41d857-7d3c-2683-28f8-5b338e63321e
2026-02-28T09:03:42.340Z [INFO]  server-d7837011-d183-a1dc-c285-1dbceb9bb9e0: added peer, starting replication: peer=server-7d3a4a2b-1ca4-b17f-57e6-016365dcccd3
2026-02-28T09:03:42.340Z [INFO]  server-d7837011-d183-a1dc-c285-1dbceb9bb9e0: pipelining replication: peer="{Voter server-7d3a4a2b-1ca4-b17f-57e6-016365dcccd3 7d3a4a2b-1ca4-b17f-57e6-016365dcccd3}"
2026-02-28T09:03:42.340Z [INFO]  server-d7837011-d183-a1dc-c285-1dbceb9bb9e0: pipelining replication: peer="{Voter server-ba41d857-7d3c-2683-28f8-5b338e63321e ba41d857-7d3c-2683-28f8-5b338e63321e}"
2026-02-28T09:03:42.446Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:42.340536822 +0000 UTC m=+100.304333361 (2 nodes), 61.7252ms from start of poll, 62.689ms from cluster start. Timeout at 2026-02-28 09:03:42.445541622 +0000 UTC m=+100.409338121, 105.00476ms after stability
2026-02-28T09:03:42.446Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:42.551Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:42.446280942 +0000 UTC m=+100.410077481 (1 nodes), 2.68µs from start of poll, 168.43312ms from cluster start. Timeout at 2026-02-28 09:03:42.551307382 +0000 UTC m=+100.515103921, 105.02644ms after stability
2026-02-28T09:03:42.551Z [WARN]  server-ba41d857-7d3c-2683-28f8-5b338e63321e: clearing log suffix: from=1 to=2
2026-02-28T09:03:42.551Z [WARN]  server-ba41d857-7d3c-2683-28f8-5b338e63321e: skipping application of old log: index=1
2026-02-28T09:03:42.551Z [WARN]  server-ba41d857-7d3c-2683-28f8-5b338e63321e: skipping application of old log: index=1
2026-02-28T09:03:42.551Z [INFO]  server-d7837011-d183-a1dc-c285-1dbceb9bb9e0: aborting pipeline replication: peer="{Voter server-7d3a4a2b-1ca4-b17f-57e6-016365dcccd3 7d3a4a2b-1ca4-b17f-57e6-016365dcccd3}"
2026-02-28T09:03:42.551Z [INFO]  server-d7837011-d183-a1dc-c285-1dbceb9bb9e0: aborting pipeline replication: peer="{Voter server-ba41d857-7d3c-2683-28f8-5b338e63321e ba41d857-7d3c-2683-28f8-5b338e63321e}"
--- PASS: TestRaft_AppendEntry (0.27s)
=== RUN   TestRaft_VotingGrant_WhenLeaderAvailable
2026-02-28T09:03:42.552Z [INFO]  server-4479d9f4-2476-955f-b181-2e7fa313aef2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4479d9f4-2476-955f-b181-2e7fa313aef2 Address:4479d9f4-2476-955f-b181-2e7fa313aef2} {Suffrage:Voter ID:server-0ed81219-22b1-219c-59ee-6d1abb3375a5 Address:0ed81219-22b1-219c-59ee-6d1abb3375a5} {Suffrage:Voter ID:server-1125f8ad-7699-3291-29f2-2790f56b9f13 Address:1125f8ad-7699-3291-29f2-2790f56b9f13}]"
2026-02-28T09:03:42.552Z [INFO]  server-4479d9f4-2476-955f-b181-2e7fa313aef2: entering follower state: follower="Node at 4479d9f4-2476-955f-b181-2e7fa313aef2 [Follower]" leader-address= leader-id=
2026-02-28T09:03:42.552Z [INFO]  server-0ed81219-22b1-219c-59ee-6d1abb3375a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4479d9f4-2476-955f-b181-2e7fa313aef2 Address:4479d9f4-2476-955f-b181-2e7fa313aef2} {Suffrage:Voter ID:server-0ed81219-22b1-219c-59ee-6d1abb3375a5 Address:0ed81219-22b1-219c-59ee-6d1abb3375a5} {Suffrage:Voter ID:server-1125f8ad-7699-3291-29f2-2790f56b9f13 Address:1125f8ad-7699-3291-29f2-2790f56b9f13}]"
2026-02-28T09:03:42.553Z [INFO]  server-0ed81219-22b1-219c-59ee-6d1abb3375a5: entering follower state: follower="Node at 0ed81219-22b1-219c-59ee-6d1abb3375a5 [Follower]" leader-address= leader-id=
2026-02-28T09:03:42.553Z [INFO]  server-1125f8ad-7699-3291-29f2-2790f56b9f13: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4479d9f4-2476-955f-b181-2e7fa313aef2 Address:4479d9f4-2476-955f-b181-2e7fa313aef2} {Suffrage:Voter ID:server-0ed81219-22b1-219c-59ee-6d1abb3375a5 Address:0ed81219-22b1-219c-59ee-6d1abb3375a5} {Suffrage:Voter ID:server-1125f8ad-7699-3291-29f2-2790f56b9f13 Address:1125f8ad-7699-3291-29f2-2790f56b9f13}]"
2026-02-28T09:03:42.553Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:42.553Z [INFO]  server-1125f8ad-7699-3291-29f2-2790f56b9f13: entering follower state: follower="Node at 1125f8ad-7699-3291-29f2-2790f56b9f13 [Follower]" leader-address= leader-id=
2026-02-28T09:03:42.604Z [WARN]  server-0ed81219-22b1-219c-59ee-6d1abb3375a5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:42.604Z [INFO]  server-0ed81219-22b1-219c-59ee-6d1abb3375a5: entering candidate state: node="Node at 0ed81219-22b1-219c-59ee-6d1abb3375a5 [Candidate]" term=2
2026-02-28T09:03:42.605Z [INFO]  server-0ed81219-22b1-219c-59ee-6d1abb3375a5: election won: term=2 tally=2
2026-02-28T09:03:42.605Z [INFO]  server-0ed81219-22b1-219c-59ee-6d1abb3375a5: entering leader state: leader="Node at 0ed81219-22b1-219c-59ee-6d1abb3375a5 [Leader]"
2026-02-28T09:03:42.605Z [INFO]  server-0ed81219-22b1-219c-59ee-6d1abb3375a5: added peer, starting replication: peer=server-4479d9f4-2476-955f-b181-2e7fa313aef2
2026-02-28T09:03:42.605Z [INFO]  server-0ed81219-22b1-219c-59ee-6d1abb3375a5: added peer, starting replication: peer=server-1125f8ad-7699-3291-29f2-2790f56b9f13
2026-02-28T09:03:42.605Z [INFO]  server-0ed81219-22b1-219c-59ee-6d1abb3375a5: pipelining replication: peer="{Voter server-1125f8ad-7699-3291-29f2-2790f56b9f13 1125f8ad-7699-3291-29f2-2790f56b9f13}"
2026-02-28T09:03:42.605Z [INFO]  server-0ed81219-22b1-219c-59ee-6d1abb3375a5: pipelining replication: peer="{Voter server-4479d9f4-2476-955f-b181-2e7fa313aef2 4479d9f4-2476-955f-b181-2e7fa313aef2}"
2026-02-28T09:03:42.711Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:42.605162422 +0000 UTC m=+100.568958961 (2 nodes), 51.92268ms from start of poll, 52.5156ms from cluster start. Timeout at 2026-02-28 09:03:42.710166022 +0000 UTC m=+100.673962561, 105.0036ms after stability
2026-02-28T09:03:42.711Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:42.816Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:42.711404222 +0000 UTC m=+100.675200801 (1 nodes), 3.04µs from start of poll, 158.75744ms from cluster start. Timeout at 2026-02-28 09:03:42.816442422 +0000 UTC m=+100.780238961, 105.03816ms after stability
2026-02-28T09:03:42.816Z [WARN]  server-1125f8ad-7699-3291-29f2-2790f56b9f13: rejecting vote request since we have a leader: from=4479d9f4-2476-955f-b181-2e7fa313aef2 leader=0ed81219-22b1-219c-59ee-6d1abb3375a5 leader-id=server-0ed81219-22b1-219c-59ee-6d1abb3375a5
2026-02-28T09:03:42.816Z [INFO]  server-0ed81219-22b1-219c-59ee-6d1abb3375a5: aborting pipeline replication: peer="{Voter server-1125f8ad-7699-3291-29f2-2790f56b9f13 1125f8ad-7699-3291-29f2-2790f56b9f13}"
2026-02-28T09:03:42.816Z [INFO]  server-0ed81219-22b1-219c-59ee-6d1abb3375a5: aborting pipeline replication: peer="{Voter server-4479d9f4-2476-955f-b181-2e7fa313aef2 4479d9f4-2476-955f-b181-2e7fa313aef2}"
--- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.26s)
=== RUN   TestRaft_ProtocolVersion_RejectRPC
2026-02-28T09:03:42.817Z [INFO]  server-ab3edf31-7a08-fc16-7875-781822fed956: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ab3edf31-7a08-fc16-7875-781822fed956 Address:ab3edf31-7a08-fc16-7875-781822fed956} {Suffrage:Voter ID:server-50b09c2e-f51d-f7d7-accd-c225092fcff4 Address:50b09c2e-f51d-f7d7-accd-c225092fcff4} {Suffrage:Voter ID:server-1b86fb16-a5d3-84db-9bca-0333ec8bb0df Address:1b86fb16-a5d3-84db-9bca-0333ec8bb0df}]"
2026-02-28T09:03:42.817Z [INFO]  server-ab3edf31-7a08-fc16-7875-781822fed956: entering follower state: follower="Node at ab3edf31-7a08-fc16-7875-781822fed956 [Follower]" leader-address= leader-id=
2026-02-28T09:03:42.817Z [INFO]  server-50b09c2e-f51d-f7d7-accd-c225092fcff4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ab3edf31-7a08-fc16-7875-781822fed956 Address:ab3edf31-7a08-fc16-7875-781822fed956} {Suffrage:Voter ID:server-50b09c2e-f51d-f7d7-accd-c225092fcff4 Address:50b09c2e-f51d-f7d7-accd-c225092fcff4} {Suffrage:Voter ID:server-1b86fb16-a5d3-84db-9bca-0333ec8bb0df Address:1b86fb16-a5d3-84db-9bca-0333ec8bb0df}]"
2026-02-28T09:03:42.817Z [INFO]  server-50b09c2e-f51d-f7d7-accd-c225092fcff4: entering follower state: follower="Node at 50b09c2e-f51d-f7d7-accd-c225092fcff4 [Follower]" leader-address= leader-id=
2026-02-28T09:03:42.817Z [INFO]  server-1b86fb16-a5d3-84db-9bca-0333ec8bb0df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ab3edf31-7a08-fc16-7875-781822fed956 Address:ab3edf31-7a08-fc16-7875-781822fed956} {Suffrage:Voter ID:server-50b09c2e-f51d-f7d7-accd-c225092fcff4 Address:50b09c2e-f51d-f7d7-accd-c225092fcff4} {Suffrage:Voter ID:server-1b86fb16-a5d3-84db-9bca-0333ec8bb0df Address:1b86fb16-a5d3-84db-9bca-0333ec8bb0df}]"
2026-02-28T09:03:42.818Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:42.818Z [INFO]  server-1b86fb16-a5d3-84db-9bca-0333ec8bb0df: entering follower state: follower="Node at 1b86fb16-a5d3-84db-9bca-0333ec8bb0df [Follower]" leader-address= leader-id=
2026-02-28T09:03:42.903Z [WARN]  server-ab3edf31-7a08-fc16-7875-781822fed956: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:42.903Z [INFO]  server-ab3edf31-7a08-fc16-7875-781822fed956: entering candidate state: node="Node at ab3edf31-7a08-fc16-7875-781822fed956 [Candidate]" term=2
2026-02-28T09:03:42.904Z [INFO]  server-ab3edf31-7a08-fc16-7875-781822fed956: election won: term=2 tally=2
2026-02-28T09:03:42.904Z [INFO]  server-ab3edf31-7a08-fc16-7875-781822fed956: entering leader state: leader="Node at ab3edf31-7a08-fc16-7875-781822fed956 [Leader]"
2026-02-28T09:03:42.904Z [INFO]  server-ab3edf31-7a08-fc16-7875-781822fed956: added peer, starting replication: peer=server-50b09c2e-f51d-f7d7-accd-c225092fcff4
2026-02-28T09:03:42.904Z [INFO]  server-ab3edf31-7a08-fc16-7875-781822fed956: added peer, starting replication: peer=server-1b86fb16-a5d3-84db-9bca-0333ec8bb0df
2026-02-28T09:03:42.904Z [INFO]  server-ab3edf31-7a08-fc16-7875-781822fed956: pipelining replication: peer="{Voter server-1b86fb16-a5d3-84db-9bca-0333ec8bb0df 1b86fb16-a5d3-84db-9bca-0333ec8bb0df}"
2026-02-28T09:03:42.904Z [INFO]  server-ab3edf31-7a08-fc16-7875-781822fed956: pipelining replication: peer="{Voter server-50b09c2e-f51d-f7d7-accd-c225092fcff4 50b09c2e-f51d-f7d7-accd-c225092fcff4}"
2026-02-28T09:03:43.009Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:42.904097022 +0000 UTC m=+100.867893561 (2 nodes), 86.06116ms from start of poll, 86.65248ms from cluster start. Timeout at 2026-02-28 09:03:43.009105822 +0000 UTC m=+100.972902361, 105.0088ms after stability
2026-02-28T09:03:43.009Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:43.115Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:43.009382862 +0000 UTC m=+100.973179401 (1 nodes), 3.08µs from start of poll, 191.93832ms from cluster start. Timeout at 2026-02-28 09:03:43.114402862 +0000 UTC m=+101.078199401, 105.02ms after stability
2026-02-28T09:03:43.115Z [INFO]  server-ab3edf31-7a08-fc16-7875-781822fed956: aborting pipeline replication: peer="{Voter server-1b86fb16-a5d3-84db-9bca-0333ec8bb0df 1b86fb16-a5d3-84db-9bca-0333ec8bb0df}"
2026-02-28T09:03:43.115Z [INFO]  server-ab3edf31-7a08-fc16-7875-781822fed956: aborting pipeline replication: peer="{Voter server-50b09c2e-f51d-f7d7-accd-c225092fcff4 50b09c2e-f51d-f7d7-accd-c225092fcff4}"
--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.30s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_1_2
    raft_test.go:2151: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_2_3
    raft_test.go:2196: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s)
=== RUN   TestRaft_LeaderID_Propagated
2026-02-28T09:03:43.116Z [INFO]  server-f7e62b26-c7b3-ab0e-f63e-44fbdb1d94f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f7e62b26-c7b3-ab0e-f63e-44fbdb1d94f8 Address:f7e62b26-c7b3-ab0e-f63e-44fbdb1d94f8} {Suffrage:Voter ID:server-cc175e5c-844e-a847-bde6-0f67b5036969 Address:cc175e5c-844e-a847-bde6-0f67b5036969} {Suffrage:Voter ID:server-3609be14-d373-8d4b-4d84-cb20356f29bc Address:3609be14-d373-8d4b-4d84-cb20356f29bc}]"
2026-02-28T09:03:43.116Z [INFO]  server-cc175e5c-844e-a847-bde6-0f67b5036969: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f7e62b26-c7b3-ab0e-f63e-44fbdb1d94f8 Address:f7e62b26-c7b3-ab0e-f63e-44fbdb1d94f8} {Suffrage:Voter ID:server-cc175e5c-844e-a847-bde6-0f67b5036969 Address:cc175e5c-844e-a847-bde6-0f67b5036969} {Suffrage:Voter ID:server-3609be14-d373-8d4b-4d84-cb20356f29bc Address:3609be14-d373-8d4b-4d84-cb20356f29bc}]"
2026-02-28T09:03:43.116Z [INFO]  server-f7e62b26-c7b3-ab0e-f63e-44fbdb1d94f8: entering follower state: follower="Node at f7e62b26-c7b3-ab0e-f63e-44fbdb1d94f8 [Follower]" leader-address= leader-id=
2026-02-28T09:03:43.116Z [INFO]  server-cc175e5c-844e-a847-bde6-0f67b5036969: entering follower state: follower="Node at cc175e5c-844e-a847-bde6-0f67b5036969 [Follower]" leader-address= leader-id=
2026-02-28T09:03:43.116Z [INFO]  server-3609be14-d373-8d4b-4d84-cb20356f29bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f7e62b26-c7b3-ab0e-f63e-44fbdb1d94f8 Address:f7e62b26-c7b3-ab0e-f63e-44fbdb1d94f8} {Suffrage:Voter ID:server-cc175e5c-844e-a847-bde6-0f67b5036969 Address:cc175e5c-844e-a847-bde6-0f67b5036969} {Suffrage:Voter ID:server-3609be14-d373-8d4b-4d84-cb20356f29bc Address:3609be14-d373-8d4b-4d84-cb20356f29bc}]"
2026-02-28T09:03:43.116Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:43.116Z [INFO]  server-3609be14-d373-8d4b-4d84-cb20356f29bc: entering follower state: follower="Node at 3609be14-d373-8d4b-4d84-cb20356f29bc [Follower]" leader-address= leader-id=
2026-02-28T09:03:43.176Z [WARN]  server-cc175e5c-844e-a847-bde6-0f67b5036969: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:43.176Z [INFO]  server-cc175e5c-844e-a847-bde6-0f67b5036969: entering candidate state: node="Node at cc175e5c-844e-a847-bde6-0f67b5036969 [Candidate]" term=2
2026-02-28T09:03:43.177Z [INFO]  server-cc175e5c-844e-a847-bde6-0f67b5036969: election won: term=2 tally=2
2026-02-28T09:03:43.177Z [INFO]  server-cc175e5c-844e-a847-bde6-0f67b5036969: entering leader state: leader="Node at cc175e5c-844e-a847-bde6-0f67b5036969 [Leader]"
2026-02-28T09:03:43.178Z [INFO]  server-cc175e5c-844e-a847-bde6-0f67b5036969: added peer, starting replication: peer=server-f7e62b26-c7b3-ab0e-f63e-44fbdb1d94f8
2026-02-28T09:03:43.178Z [INFO]  server-cc175e5c-844e-a847-bde6-0f67b5036969: added peer, starting replication: peer=server-3609be14-d373-8d4b-4d84-cb20356f29bc
2026-02-28T09:03:43.178Z [INFO]  server-cc175e5c-844e-a847-bde6-0f67b5036969: pipelining replication: peer="{Voter server-3609be14-d373-8d4b-4d84-cb20356f29bc 3609be14-d373-8d4b-4d84-cb20356f29bc}"
2026-02-28T09:03:43.178Z [INFO]  server-cc175e5c-844e-a847-bde6-0f67b5036969: pipelining replication: peer="{Voter server-f7e62b26-c7b3-ab0e-f63e-44fbdb1d94f8 f7e62b26-c7b3-ab0e-f63e-44fbdb1d94f8}"
2026-02-28T09:03:43.283Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:43.178041142 +0000 UTC m=+101.141837681 (1 nodes), 61.32656ms from start of poll, 61.78372ms from cluster start. Timeout at 2026-02-28 09:03:43.283048462 +0000 UTC m=+101.246845001, 105.00732ms after stability
2026-02-28T09:03:43.283Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:43.389Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:43.283712662 +0000 UTC m=+101.247509201 (1 nodes), 1.8µs from start of poll, 167.45524ms from cluster start. Timeout at 2026-02-28 09:03:43.388739702 +0000 UTC m=+101.352536281, 105.02708ms after stability
2026-02-28T09:03:43.389Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:43.494Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:43.389470742 +0000 UTC m=+101.353267281 (1 nodes), 1.32µs from start of poll, 273.21332ms from cluster start. Timeout at 2026-02-28 09:03:43.494511102 +0000 UTC m=+101.458307641, 105.04036ms after stability
2026-02-28T09:03:43.495Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:43.600Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:43.495215422 +0000 UTC m=+101.459011961 (1 nodes), 1µs from start of poll, 378.958ms from cluster start. Timeout at 2026-02-28 09:03:43.600246862 +0000 UTC m=+101.564043441, 105.03148ms after stability
2026-02-28T09:03:43.600Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:43.706Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:43.600713582 +0000 UTC m=+101.564510121 (1 nodes), 1.04µs from start of poll, 484.45616ms from cluster start. Timeout at 2026-02-28 09:03:43.705732142 +0000 UTC m=+101.669528681, 105.01856ms after stability
2026-02-28T09:03:43.706Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:43.812Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:43.706758422 +0000 UTC m=+101.670554961 (1 nodes), 1µs from start of poll, 590.501ms from cluster start. Timeout at 2026-02-28 09:03:43.811785062 +0000 UTC m=+101.775581601, 105.02664ms after stability
2026-02-28T09:03:43.918Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:44.023Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:43.918242662 +0000 UTC m=+101.882039201 (1 nodes), 2.48µs from start of poll, 801.98524ms from cluster start. Timeout at 2026-02-28 09:03:44.023266582 +0000 UTC m=+101.987063121, 105.02392ms after stability
2026-02-28T09:03:44.023Z [INFO]  server-cc175e5c-844e-a847-bde6-0f67b5036969: aborting pipeline replication: peer="{Voter server-3609be14-d373-8d4b-4d84-cb20356f29bc 3609be14-d373-8d4b-4d84-cb20356f29bc}"
2026-02-28T09:03:44.023Z [INFO]  server-cc175e5c-844e-a847-bde6-0f67b5036969: aborting pipeline replication: peer="{Voter server-f7e62b26-c7b3-ab0e-f63e-44fbdb1d94f8 f7e62b26-c7b3-ab0e-f63e-44fbdb1d94f8}"
--- PASS: TestRaft_LeaderID_Propagated (0.91s)
=== RUN   TestRaft_LeadershipTransferInProgress
--- PASS: TestRaft_LeadershipTransferInProgress (0.00s)
=== RUN   TestRaft_LeadershipTransferPickServer
--- PASS: TestRaft_LeadershipTransferPickServer (0.00s)
=== RUN   TestRaft_LeadershipTransfer
2026-02-28T09:03:44.024Z [INFO]  server-93ae09ee-b73f-1c83-fa64-8c70a0b8a757: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93ae09ee-b73f-1c83-fa64-8c70a0b8a757 Address:93ae09ee-b73f-1c83-fa64-8c70a0b8a757} {Suffrage:Voter ID:server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02 Address:8c4e8d7f-08ab-715b-54e6-fa6e8399db02} {Suffrage:Voter ID:server-93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c Address:93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c}]"
2026-02-28T09:03:44.024Z [INFO]  server-93ae09ee-b73f-1c83-fa64-8c70a0b8a757: entering follower state: follower="Node at 93ae09ee-b73f-1c83-fa64-8c70a0b8a757 [Follower]" leader-address= leader-id=
2026-02-28T09:03:44.025Z [INFO]  server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93ae09ee-b73f-1c83-fa64-8c70a0b8a757 Address:93ae09ee-b73f-1c83-fa64-8c70a0b8a757} {Suffrage:Voter ID:server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02 Address:8c4e8d7f-08ab-715b-54e6-fa6e8399db02} {Suffrage:Voter ID:server-93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c Address:93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c}]"
2026-02-28T09:03:44.025Z [INFO]  server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02: entering follower state: follower="Node at 8c4e8d7f-08ab-715b-54e6-fa6e8399db02 [Follower]" leader-address= leader-id=
2026-02-28T09:03:44.025Z [INFO]  server-93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93ae09ee-b73f-1c83-fa64-8c70a0b8a757 Address:93ae09ee-b73f-1c83-fa64-8c70a0b8a757} {Suffrage:Voter ID:server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02 Address:8c4e8d7f-08ab-715b-54e6-fa6e8399db02} {Suffrage:Voter ID:server-93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c Address:93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c}]"
2026-02-28T09:03:44.025Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:44.025Z [INFO]  server-93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c: entering follower state: follower="Node at 93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c [Follower]" leader-address= leader-id=
2026-02-28T09:03:44.085Z [WARN]  server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:44.085Z [INFO]  server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02: entering candidate state: node="Node at 8c4e8d7f-08ab-715b-54e6-fa6e8399db02 [Candidate]" term=2
2026-02-28T09:03:44.085Z [INFO]  server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02: election won: term=2 tally=2
2026-02-28T09:03:44.085Z [INFO]  server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02: entering leader state: leader="Node at 8c4e8d7f-08ab-715b-54e6-fa6e8399db02 [Leader]"
2026-02-28T09:03:44.085Z [INFO]  server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02: added peer, starting replication: peer=server-93ae09ee-b73f-1c83-fa64-8c70a0b8a757
2026-02-28T09:03:44.085Z [INFO]  server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02: added peer, starting replication: peer=server-93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c
2026-02-28T09:03:44.086Z [INFO]  server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02: pipelining replication: peer="{Voter server-93ae09ee-b73f-1c83-fa64-8c70a0b8a757 93ae09ee-b73f-1c83-fa64-8c70a0b8a757}"
2026-02-28T09:03:44.086Z [INFO]  server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02: pipelining replication: peer="{Voter server-93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c 93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c}"
2026-02-28T09:03:44.192Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:44.086365942 +0000 UTC m=+102.050162481 (1 nodes), 61.0138ms from start of poll, 61.64692ms from cluster start. Timeout at 2026-02-28 09:03:44.191372182 +0000 UTC m=+102.155168721, 105.00624ms after stability
2026-02-28T09:03:44.192Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:44.298Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:44.192207382 +0000 UTC m=+102.156003921 (1 nodes), 3.12µs from start of poll, 167.48836ms from cluster start. Timeout at 2026-02-28 09:03:44.297254302 +0000 UTC m=+102.261050841, 105.04692ms after stability
2026-02-28T09:03:44.298Z [INFO]  server-93ae09ee-b73f-1c83-fa64-8c70a0b8a757: entering candidate state: node="Node at 93ae09ee-b73f-1c83-fa64-8c70a0b8a757 [Candidate]" term=3
2026-02-28T09:03:44.298Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:44.298Z [INFO]  server-93ae09ee-b73f-1c83-fa64-8c70a0b8a757: election won: term=3 tally=2
2026-02-28T09:03:44.298Z [INFO]  server-93ae09ee-b73f-1c83-fa64-8c70a0b8a757: entering leader state: leader="Node at 93ae09ee-b73f-1c83-fa64-8c70a0b8a757 [Leader]"
2026-02-28T09:03:44.298Z [INFO]  server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02: entering follower state: follower="Node at 8c4e8d7f-08ab-715b-54e6-fa6e8399db02 [Follower]" leader-address= leader-id=
2026-02-28T09:03:44.298Z [INFO]  server-93ae09ee-b73f-1c83-fa64-8c70a0b8a757: added peer, starting replication: peer=server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02
2026-02-28T09:03:44.298Z [INFO]  server-93ae09ee-b73f-1c83-fa64-8c70a0b8a757: added peer, starting replication: peer=server-93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c
2026-02-28T09:03:44.298Z [INFO]  server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02: aborting pipeline replication: peer="{Voter server-93ae09ee-b73f-1c83-fa64-8c70a0b8a757 93ae09ee-b73f-1c83-fa64-8c70a0b8a757}"
2026-02-28T09:03:44.298Z [INFO]  server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02: aborting pipeline replication: peer="{Voter server-93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c 93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c}"
2026-02-28T09:03:44.298Z [INFO]  server-93ae09ee-b73f-1c83-fa64-8c70a0b8a757: pipelining replication: peer="{Voter server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02 8c4e8d7f-08ab-715b-54e6-fa6e8399db02}"
2026-02-28T09:03:44.298Z [INFO]  server-93ae09ee-b73f-1c83-fa64-8c70a0b8a757: pipelining replication: peer="{Voter server-93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c 93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c}"
2026-02-28T09:03:44.403Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:44.298300982 +0000 UTC m=+102.262097561 (1 nodes), 40.48µs from start of poll, 273.582ms from cluster start. Timeout at 2026-02-28 09:03:44.403327422 +0000 UTC m=+102.367123961, 105.0264ms after stability
2026-02-28T09:03:44.403Z [INFO]  server-93ae09ee-b73f-1c83-fa64-8c70a0b8a757: aborting pipeline replication: peer="{Voter server-93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c 93fc71ab-b7ce-9fe1-fdcf-c11558b9ff7c}"
2026-02-28T09:03:44.403Z [INFO]  server-93ae09ee-b73f-1c83-fa64-8c70a0b8a757: aborting pipeline replication: peer="{Voter server-8c4e8d7f-08ab-715b-54e6-fa6e8399db02 8c4e8d7f-08ab-715b-54e6-fa6e8399db02}"
--- PASS: TestRaft_LeadershipTransfer (0.38s)
=== RUN   TestRaft_LeadershipTransferWithOneNode
2026-02-28T09:03:44.404Z [INFO]  server-f1cdbfc8-0102-99a1-98e2-28162c511402: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f1cdbfc8-0102-99a1-98e2-28162c511402 Address:f1cdbfc8-0102-99a1-98e2-28162c511402}]"
2026-02-28T09:03:44.404Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:44.404Z [INFO]  server-f1cdbfc8-0102-99a1-98e2-28162c511402: entering follower state: follower="Node at f1cdbfc8-0102-99a1-98e2-28162c511402 [Follower]" leader-address= leader-id=
2026-02-28T09:03:44.503Z [WARN]  server-f1cdbfc8-0102-99a1-98e2-28162c511402: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:44.503Z [INFO]  server-f1cdbfc8-0102-99a1-98e2-28162c511402: entering candidate state: node="Node at f1cdbfc8-0102-99a1-98e2-28162c511402 [Candidate]" term=2
2026-02-28T09:03:44.503Z [INFO]  server-f1cdbfc8-0102-99a1-98e2-28162c511402: election won: term=2 tally=1
2026-02-28T09:03:44.503Z [INFO]  server-f1cdbfc8-0102-99a1-98e2-28162c511402: entering leader state: leader="Node at f1cdbfc8-0102-99a1-98e2-28162c511402 [Leader]"
2026-02-28T09:03:44.608Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:44.503454622 +0000 UTC m=+102.467251161 (1 nodes), 98.9316ms from start of poll, 99.14332ms from cluster start. Timeout at 2026-02-28 09:03:44.608457382 +0000 UTC m=+102.572253921, 105.00276ms after stability
--- PASS: TestRaft_LeadershipTransferWithOneNode (0.20s)
=== RUN   TestRaft_LeadershipTransferWithSevenNodes
2026-02-28T09:03:44.609Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-913157f3-e3e1-44a1-6caf-ca400a002155 Address:913157f3-e3e1-44a1-6caf-ca400a002155} {Suffrage:Voter ID:server-17ea8dfa-a55b-3e46-311c-867813534a43 Address:17ea8dfa-a55b-3e46-311c-867813534a43} {Suffrage:Voter ID:server-ccedd4d0-cdb7-2cba-b739-9418307e1284 Address:ccedd4d0-cdb7-2cba-b739-9418307e1284} {Suffrage:Voter ID:server-d05f9034-d3b6-95f5-73b1-429a2f273812 Address:d05f9034-d3b6-95f5-73b1-429a2f273812} {Suffrage:Voter ID:server-aa916c6b-6f79-38e4-4603-c1214fb408d0 Address:aa916c6b-6f79-38e4-4603-c1214fb408d0} {Suffrage:Voter ID:server-9dd98bea-ad84-01b8-9f2c-34d09168a7e1 Address:9dd98bea-ad84-01b8-9f2c-34d09168a7e1} {Suffrage:Voter ID:server-cfd5942e-a425-f10f-13d6-2671e5429b91 Address:cfd5942e-a425-f10f-13d6-2671e5429b91}]"
2026-02-28T09:03:44.609Z [INFO]  server-17ea8dfa-a55b-3e46-311c-867813534a43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-913157f3-e3e1-44a1-6caf-ca400a002155 Address:913157f3-e3e1-44a1-6caf-ca400a002155} {Suffrage:Voter ID:server-17ea8dfa-a55b-3e46-311c-867813534a43 Address:17ea8dfa-a55b-3e46-311c-867813534a43} {Suffrage:Voter ID:server-ccedd4d0-cdb7-2cba-b739-9418307e1284 Address:ccedd4d0-cdb7-2cba-b739-9418307e1284} {Suffrage:Voter ID:server-d05f9034-d3b6-95f5-73b1-429a2f273812 Address:d05f9034-d3b6-95f5-73b1-429a2f273812} {Suffrage:Voter ID:server-aa916c6b-6f79-38e4-4603-c1214fb408d0 Address:aa916c6b-6f79-38e4-4603-c1214fb408d0} {Suffrage:Voter ID:server-9dd98bea-ad84-01b8-9f2c-34d09168a7e1 Address:9dd98bea-ad84-01b8-9f2c-34d09168a7e1} {Suffrage:Voter ID:server-cfd5942e-a425-f10f-13d6-2671e5429b91 Address:cfd5942e-a425-f10f-13d6-2671e5429b91}]"
2026-02-28T09:03:44.609Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: entering follower state: follower="Node at 913157f3-e3e1-44a1-6caf-ca400a002155 [Follower]" leader-address= leader-id=
2026-02-28T09:03:44.610Z [INFO]  server-17ea8dfa-a55b-3e46-311c-867813534a43: entering follower state: follower="Node at 17ea8dfa-a55b-3e46-311c-867813534a43 [Follower]" leader-address= leader-id=
2026-02-28T09:03:44.610Z [INFO]  server-ccedd4d0-cdb7-2cba-b739-9418307e1284: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-913157f3-e3e1-44a1-6caf-ca400a002155 Address:913157f3-e3e1-44a1-6caf-ca400a002155} {Suffrage:Voter ID:server-17ea8dfa-a55b-3e46-311c-867813534a43 Address:17ea8dfa-a55b-3e46-311c-867813534a43} {Suffrage:Voter ID:server-ccedd4d0-cdb7-2cba-b739-9418307e1284 Address:ccedd4d0-cdb7-2cba-b739-9418307e1284} {Suffrage:Voter ID:server-d05f9034-d3b6-95f5-73b1-429a2f273812 Address:d05f9034-d3b6-95f5-73b1-429a2f273812} {Suffrage:Voter ID:server-aa916c6b-6f79-38e4-4603-c1214fb408d0 Address:aa916c6b-6f79-38e4-4603-c1214fb408d0} {Suffrage:Voter ID:server-9dd98bea-ad84-01b8-9f2c-34d09168a7e1 Address:9dd98bea-ad84-01b8-9f2c-34d09168a7e1} {Suffrage:Voter ID:server-cfd5942e-a425-f10f-13d6-2671e5429b91 Address:cfd5942e-a425-f10f-13d6-2671e5429b91}]"
2026-02-28T09:03:44.635Z [INFO]  server-ccedd4d0-cdb7-2cba-b739-9418307e1284: entering follower state: follower="Node at ccedd4d0-cdb7-2cba-b739-9418307e1284 [Follower]" leader-address= leader-id=
2026-02-28T09:03:44.635Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-913157f3-e3e1-44a1-6caf-ca400a002155 Address:913157f3-e3e1-44a1-6caf-ca400a002155} {Suffrage:Voter ID:server-17ea8dfa-a55b-3e46-311c-867813534a43 Address:17ea8dfa-a55b-3e46-311c-867813534a43} {Suffrage:Voter ID:server-ccedd4d0-cdb7-2cba-b739-9418307e1284 Address:ccedd4d0-cdb7-2cba-b739-9418307e1284} {Suffrage:Voter ID:server-d05f9034-d3b6-95f5-73b1-429a2f273812 Address:d05f9034-d3b6-95f5-73b1-429a2f273812} {Suffrage:Voter ID:server-aa916c6b-6f79-38e4-4603-c1214fb408d0 Address:aa916c6b-6f79-38e4-4603-c1214fb408d0} {Suffrage:Voter ID:server-9dd98bea-ad84-01b8-9f2c-34d09168a7e1 Address:9dd98bea-ad84-01b8-9f2c-34d09168a7e1} {Suffrage:Voter ID:server-cfd5942e-a425-f10f-13d6-2671e5429b91 Address:cfd5942e-a425-f10f-13d6-2671e5429b91}]"
2026-02-28T09:03:44.635Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: entering follower state: follower="Node at d05f9034-d3b6-95f5-73b1-429a2f273812 [Follower]" leader-address= leader-id=
2026-02-28T09:03:44.635Z [INFO]  server-aa916c6b-6f79-38e4-4603-c1214fb408d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-913157f3-e3e1-44a1-6caf-ca400a002155 Address:913157f3-e3e1-44a1-6caf-ca400a002155} {Suffrage:Voter ID:server-17ea8dfa-a55b-3e46-311c-867813534a43 Address:17ea8dfa-a55b-3e46-311c-867813534a43} {Suffrage:Voter ID:server-ccedd4d0-cdb7-2cba-b739-9418307e1284 Address:ccedd4d0-cdb7-2cba-b739-9418307e1284} {Suffrage:Voter ID:server-d05f9034-d3b6-95f5-73b1-429a2f273812 Address:d05f9034-d3b6-95f5-73b1-429a2f273812} {Suffrage:Voter ID:server-aa916c6b-6f79-38e4-4603-c1214fb408d0 Address:aa916c6b-6f79-38e4-4603-c1214fb408d0} {Suffrage:Voter ID:server-9dd98bea-ad84-01b8-9f2c-34d09168a7e1 Address:9dd98bea-ad84-01b8-9f2c-34d09168a7e1} {Suffrage:Voter ID:server-cfd5942e-a425-f10f-13d6-2671e5429b91 Address:cfd5942e-a425-f10f-13d6-2671e5429b91}]"
2026-02-28T09:03:44.636Z [INFO]  server-aa916c6b-6f79-38e4-4603-c1214fb408d0: entering follower state: follower="Node at aa916c6b-6f79-38e4-4603-c1214fb408d0 [Follower]" leader-address= leader-id=
2026-02-28T09:03:44.636Z [INFO]  server-9dd98bea-ad84-01b8-9f2c-34d09168a7e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-913157f3-e3e1-44a1-6caf-ca400a002155 Address:913157f3-e3e1-44a1-6caf-ca400a002155} {Suffrage:Voter ID:server-17ea8dfa-a55b-3e46-311c-867813534a43 Address:17ea8dfa-a55b-3e46-311c-867813534a43} {Suffrage:Voter ID:server-ccedd4d0-cdb7-2cba-b739-9418307e1284 Address:ccedd4d0-cdb7-2cba-b739-9418307e1284} {Suffrage:Voter ID:server-d05f9034-d3b6-95f5-73b1-429a2f273812 Address:d05f9034-d3b6-95f5-73b1-429a2f273812} {Suffrage:Voter ID:server-aa916c6b-6f79-38e4-4603-c1214fb408d0 Address:aa916c6b-6f79-38e4-4603-c1214fb408d0} {Suffrage:Voter ID:server-9dd98bea-ad84-01b8-9f2c-34d09168a7e1 Address:9dd98bea-ad84-01b8-9f2c-34d09168a7e1} {Suffrage:Voter ID:server-cfd5942e-a425-f10f-13d6-2671e5429b91 Address:cfd5942e-a425-f10f-13d6-2671e5429b91}]"
2026-02-28T09:03:44.636Z [INFO]  server-9dd98bea-ad84-01b8-9f2c-34d09168a7e1: entering follower state: follower="Node at 9dd98bea-ad84-01b8-9f2c-34d09168a7e1 [Follower]" leader-address= leader-id=
2026-02-28T09:03:44.636Z [INFO]  server-cfd5942e-a425-f10f-13d6-2671e5429b91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-913157f3-e3e1-44a1-6caf-ca400a002155 Address:913157f3-e3e1-44a1-6caf-ca400a002155} {Suffrage:Voter ID:server-17ea8dfa-a55b-3e46-311c-867813534a43 Address:17ea8dfa-a55b-3e46-311c-867813534a43} {Suffrage:Voter ID:server-ccedd4d0-cdb7-2cba-b739-9418307e1284 Address:ccedd4d0-cdb7-2cba-b739-9418307e1284} {Suffrage:Voter ID:server-d05f9034-d3b6-95f5-73b1-429a2f273812 Address:d05f9034-d3b6-95f5-73b1-429a2f273812} {Suffrage:Voter ID:server-aa916c6b-6f79-38e4-4603-c1214fb408d0 Address:aa916c6b-6f79-38e4-4603-c1214fb408d0} {Suffrage:Voter ID:server-9dd98bea-ad84-01b8-9f2c-34d09168a7e1 Address:9dd98bea-ad84-01b8-9f2c-34d09168a7e1} {Suffrage:Voter ID:server-cfd5942e-a425-f10f-13d6-2671e5429b91 Address:cfd5942e-a425-f10f-13d6-2671e5429b91}]"
2026-02-28T09:03:44.636Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:44.636Z [INFO]  server-cfd5942e-a425-f10f-13d6-2671e5429b91: entering follower state: follower="Node at cfd5942e-a425-f10f-13d6-2671e5429b91 [Follower]" leader-address= leader-id=
2026-02-28T09:03:44.687Z [WARN]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:44.687Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: entering candidate state: node="Node at d05f9034-d3b6-95f5-73b1-429a2f273812 [Candidate]" term=2
2026-02-28T09:03:44.687Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: election won: term=2 tally=4
2026-02-28T09:03:44.687Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: entering leader state: leader="Node at d05f9034-d3b6-95f5-73b1-429a2f273812 [Leader]"
2026-02-28T09:03:44.687Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: added peer, starting replication: peer=server-913157f3-e3e1-44a1-6caf-ca400a002155
2026-02-28T09:03:44.687Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: added peer, starting replication: peer=server-17ea8dfa-a55b-3e46-311c-867813534a43
2026-02-28T09:03:44.687Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: added peer, starting replication: peer=server-ccedd4d0-cdb7-2cba-b739-9418307e1284
2026-02-28T09:03:44.687Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: added peer, starting replication: peer=server-aa916c6b-6f79-38e4-4603-c1214fb408d0
2026-02-28T09:03:44.687Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: added peer, starting replication: peer=server-9dd98bea-ad84-01b8-9f2c-34d09168a7e1
2026-02-28T09:03:44.687Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: added peer, starting replication: peer=server-cfd5942e-a425-f10f-13d6-2671e5429b91
2026-02-28T09:03:44.687Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: pipelining replication: peer="{Voter server-913157f3-e3e1-44a1-6caf-ca400a002155 913157f3-e3e1-44a1-6caf-ca400a002155}"
2026-02-28T09:03:44.687Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: pipelining replication: peer="{Voter server-ccedd4d0-cdb7-2cba-b739-9418307e1284 ccedd4d0-cdb7-2cba-b739-9418307e1284}"
2026-02-28T09:03:44.688Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: pipelining replication: peer="{Voter server-cfd5942e-a425-f10f-13d6-2671e5429b91 cfd5942e-a425-f10f-13d6-2671e5429b91}"
2026-02-28T09:03:44.688Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: pipelining replication: peer="{Voter server-9dd98bea-ad84-01b8-9f2c-34d09168a7e1 9dd98bea-ad84-01b8-9f2c-34d09168a7e1}"
2026-02-28T09:03:44.688Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: pipelining replication: peer="{Voter server-17ea8dfa-a55b-3e46-311c-867813534a43 17ea8dfa-a55b-3e46-311c-867813534a43}"
2026-02-28T09:03:44.688Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: pipelining replication: peer="{Voter server-aa916c6b-6f79-38e4-4603-c1214fb408d0 aa916c6b-6f79-38e4-4603-c1214fb408d0}"
2026-02-28T09:03:44.793Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:44.687696262 +0000 UTC m=+102.651492801 (1 nodes), 51.13972ms from start of poll, 78.12064ms from cluster start. Timeout at 2026-02-28 09:03:44.792701542 +0000 UTC m=+102.756498081, 105.00528ms after stability
2026-02-28T09:03:44.793Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:44.899Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:44.793232502 +0000 UTC m=+102.757029041 (6 nodes), 4.52µs from start of poll, 183.65688ms from cluster start. Timeout at 2026-02-28 09:03:44.898284742 +0000 UTC m=+102.862081281, 105.05224ms after stability
2026-02-28T09:03:44.899Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:45.004Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:44.899273742 +0000 UTC m=+102.863070281 (1 nodes), 3.24µs from start of poll, 289.69812ms from cluster start. Timeout at 2026-02-28 09:03:45.004313542 +0000 UTC m=+102.968110121, 105.03984ms after stability
2026-02-28T09:03:45.004Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: entering candidate state: node="Node at 913157f3-e3e1-44a1-6caf-ca400a002155 [Candidate]" term=3
2026-02-28T09:03:45.004Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:45.004Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: election won: term=3 tally=4
2026-02-28T09:03:45.004Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: entering follower state: follower="Node at d05f9034-d3b6-95f5-73b1-429a2f273812 [Follower]" leader-address= leader-id=
2026-02-28T09:03:45.004Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: entering leader state: leader="Node at 913157f3-e3e1-44a1-6caf-ca400a002155 [Leader]"
2026-02-28T09:03:45.004Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: aborting pipeline replication: peer="{Voter server-ccedd4d0-cdb7-2cba-b739-9418307e1284 ccedd4d0-cdb7-2cba-b739-9418307e1284}"
2026-02-28T09:03:45.004Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: added peer, starting replication: peer=server-17ea8dfa-a55b-3e46-311c-867813534a43
2026-02-28T09:03:45.004Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: aborting pipeline replication: peer="{Voter server-17ea8dfa-a55b-3e46-311c-867813534a43 17ea8dfa-a55b-3e46-311c-867813534a43}"
2026-02-28T09:03:45.004Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: added peer, starting replication: peer=server-ccedd4d0-cdb7-2cba-b739-9418307e1284
2026-02-28T09:03:45.004Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: added peer, starting replication: peer=server-d05f9034-d3b6-95f5-73b1-429a2f273812
2026-02-28T09:03:45.004Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: added peer, starting replication: peer=server-aa916c6b-6f79-38e4-4603-c1214fb408d0
2026-02-28T09:03:45.004Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: added peer, starting replication: peer=server-9dd98bea-ad84-01b8-9f2c-34d09168a7e1
2026-02-28T09:03:45.004Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: aborting pipeline replication: peer="{Voter server-913157f3-e3e1-44a1-6caf-ca400a002155 913157f3-e3e1-44a1-6caf-ca400a002155}"
2026-02-28T09:03:45.004Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: pipelining replication: peer="{Voter server-17ea8dfa-a55b-3e46-311c-867813534a43 17ea8dfa-a55b-3e46-311c-867813534a43}"
2026-02-28T09:03:45.004Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: aborting pipeline replication: peer="{Voter server-aa916c6b-6f79-38e4-4603-c1214fb408d0 aa916c6b-6f79-38e4-4603-c1214fb408d0}"
2026-02-28T09:03:45.004Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: added peer, starting replication: peer=server-cfd5942e-a425-f10f-13d6-2671e5429b91
2026-02-28T09:03:45.004Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: aborting pipeline replication: peer="{Voter server-9dd98bea-ad84-01b8-9f2c-34d09168a7e1 9dd98bea-ad84-01b8-9f2c-34d09168a7e1}"
2026-02-28T09:03:45.004Z [INFO]  server-d05f9034-d3b6-95f5-73b1-429a2f273812: aborting pipeline replication: peer="{Voter server-cfd5942e-a425-f10f-13d6-2671e5429b91 cfd5942e-a425-f10f-13d6-2671e5429b91}"
2026-02-28T09:03:45.004Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: pipelining replication: peer="{Voter server-d05f9034-d3b6-95f5-73b1-429a2f273812 d05f9034-d3b6-95f5-73b1-429a2f273812}"
2026-02-28T09:03:45.004Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: pipelining replication: peer="{Voter server-ccedd4d0-cdb7-2cba-b739-9418307e1284 ccedd4d0-cdb7-2cba-b739-9418307e1284}"
2026-02-28T09:03:45.005Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: pipelining replication: peer="{Voter server-cfd5942e-a425-f10f-13d6-2671e5429b91 cfd5942e-a425-f10f-13d6-2671e5429b91}"
2026-02-28T09:03:45.005Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: pipelining replication: peer="{Voter server-aa916c6b-6f79-38e4-4603-c1214fb408d0 aa916c6b-6f79-38e4-4603-c1214fb408d0}"
2026-02-28T09:03:45.005Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: pipelining replication: peer="{Voter server-9dd98bea-ad84-01b8-9f2c-34d09168a7e1 9dd98bea-ad84-01b8-9f2c-34d09168a7e1}"
2026-02-28T09:03:45.109Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:45.004811182 +0000 UTC m=+102.968607721 (1 nodes), 124.32µs from start of poll, 395.23556ms from cluster start. Timeout at 2026-02-28 09:03:45.109833862 +0000 UTC m=+103.073630441, 105.02272ms after stability
2026-02-28T09:03:45.110Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: aborting pipeline replication: peer="{Voter server-d05f9034-d3b6-95f5-73b1-429a2f273812 d05f9034-d3b6-95f5-73b1-429a2f273812}"
2026-02-28T09:03:45.110Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: aborting pipeline replication: peer="{Voter server-aa916c6b-6f79-38e4-4603-c1214fb408d0 aa916c6b-6f79-38e4-4603-c1214fb408d0}"
2026-02-28T09:03:45.110Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: aborting pipeline replication: peer="{Voter server-17ea8dfa-a55b-3e46-311c-867813534a43 17ea8dfa-a55b-3e46-311c-867813534a43}"
2026-02-28T09:03:45.110Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: aborting pipeline replication: peer="{Voter server-ccedd4d0-cdb7-2cba-b739-9418307e1284 ccedd4d0-cdb7-2cba-b739-9418307e1284}"
2026-02-28T09:03:45.110Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: aborting pipeline replication: peer="{Voter server-9dd98bea-ad84-01b8-9f2c-34d09168a7e1 9dd98bea-ad84-01b8-9f2c-34d09168a7e1}"
2026-02-28T09:03:45.110Z [INFO]  server-913157f3-e3e1-44a1-6caf-ca400a002155: aborting pipeline replication: peer="{Voter server-cfd5942e-a425-f10f-13d6-2671e5429b91 cfd5942e-a425-f10f-13d6-2671e5429b91}"
--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.50s)
=== RUN   TestRaft_LeadershipTransferToInvalidID
2026-02-28T09:03:45.111Z [INFO]  server-ea1c08b6-f955-dc7c-f691-5a1cb038f46e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea1c08b6-f955-dc7c-f691-5a1cb038f46e Address:ea1c08b6-f955-dc7c-f691-5a1cb038f46e} {Suffrage:Voter ID:server-0ad7d145-721f-4bfe-f5d7-01d48473a621 Address:0ad7d145-721f-4bfe-f5d7-01d48473a621} {Suffrage:Voter ID:server-2ac257ea-c882-89cf-b1bf-f840f768a590 Address:2ac257ea-c882-89cf-b1bf-f840f768a590}]"
2026-02-28T09:03:45.111Z [INFO]  server-0ad7d145-721f-4bfe-f5d7-01d48473a621: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea1c08b6-f955-dc7c-f691-5a1cb038f46e Address:ea1c08b6-f955-dc7c-f691-5a1cb038f46e} {Suffrage:Voter ID:server-0ad7d145-721f-4bfe-f5d7-01d48473a621 Address:0ad7d145-721f-4bfe-f5d7-01d48473a621} {Suffrage:Voter ID:server-2ac257ea-c882-89cf-b1bf-f840f768a590 Address:2ac257ea-c882-89cf-b1bf-f840f768a590}]"
2026-02-28T09:03:45.111Z [INFO]  server-ea1c08b6-f955-dc7c-f691-5a1cb038f46e: entering follower state: follower="Node at ea1c08b6-f955-dc7c-f691-5a1cb038f46e [Follower]" leader-address= leader-id=
2026-02-28T09:03:45.111Z [INFO]  server-0ad7d145-721f-4bfe-f5d7-01d48473a621: entering follower state: follower="Node at 0ad7d145-721f-4bfe-f5d7-01d48473a621 [Follower]" leader-address= leader-id=
2026-02-28T09:03:45.111Z [INFO]  server-2ac257ea-c882-89cf-b1bf-f840f768a590: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea1c08b6-f955-dc7c-f691-5a1cb038f46e Address:ea1c08b6-f955-dc7c-f691-5a1cb038f46e} {Suffrage:Voter ID:server-0ad7d145-721f-4bfe-f5d7-01d48473a621 Address:0ad7d145-721f-4bfe-f5d7-01d48473a621} {Suffrage:Voter ID:server-2ac257ea-c882-89cf-b1bf-f840f768a590 Address:2ac257ea-c882-89cf-b1bf-f840f768a590}]"
2026-02-28T09:03:45.111Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:45.111Z [INFO]  server-2ac257ea-c882-89cf-b1bf-f840f768a590: entering follower state: follower="Node at 2ac257ea-c882-89cf-b1bf-f840f768a590 [Follower]" leader-address= leader-id=
2026-02-28T09:03:45.190Z [WARN]  server-2ac257ea-c882-89cf-b1bf-f840f768a590: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:45.190Z [INFO]  server-2ac257ea-c882-89cf-b1bf-f840f768a590: entering candidate state: node="Node at 2ac257ea-c882-89cf-b1bf-f840f768a590 [Candidate]" term=2
2026-02-28T09:03:45.190Z [INFO]  server-2ac257ea-c882-89cf-b1bf-f840f768a590: election won: term=2 tally=2
2026-02-28T09:03:45.190Z [INFO]  server-2ac257ea-c882-89cf-b1bf-f840f768a590: entering leader state: leader="Node at 2ac257ea-c882-89cf-b1bf-f840f768a590 [Leader]"
2026-02-28T09:03:45.190Z [INFO]  server-2ac257ea-c882-89cf-b1bf-f840f768a590: added peer, starting replication: peer=server-ea1c08b6-f955-dc7c-f691-5a1cb038f46e
2026-02-28T09:03:45.190Z [INFO]  server-2ac257ea-c882-89cf-b1bf-f840f768a590: added peer, starting replication: peer=server-0ad7d145-721f-4bfe-f5d7-01d48473a621
2026-02-28T09:03:45.191Z [INFO]  server-2ac257ea-c882-89cf-b1bf-f840f768a590: pipelining replication: peer="{Voter server-0ad7d145-721f-4bfe-f5d7-01d48473a621 0ad7d145-721f-4bfe-f5d7-01d48473a621}"
2026-02-28T09:03:45.191Z [INFO]  server-2ac257ea-c882-89cf-b1bf-f840f768a590: pipelining replication: peer="{Voter server-ea1c08b6-f955-dc7c-f691-5a1cb038f46e ea1c08b6-f955-dc7c-f691-5a1cb038f46e}"
2026-02-28T09:03:45.296Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:45.190986862 +0000 UTC m=+103.154783401 (1 nodes), 79.2082ms from start of poll, 79.70936ms from cluster start. Timeout at 2026-02-28 09:03:45.295991222 +0000 UTC m=+103.259787761, 105.00436ms after stability
2026-02-28T09:03:45.304Z [INFO]  server-2ac257ea-c882-89cf-b1bf-f840f768a590: aborting pipeline replication: peer="{Voter server-0ad7d145-721f-4bfe-f5d7-01d48473a621 0ad7d145-721f-4bfe-f5d7-01d48473a621}"
2026-02-28T09:03:45.304Z [INFO]  server-2ac257ea-c882-89cf-b1bf-f840f768a590: aborting pipeline replication: peer="{Voter server-ea1c08b6-f955-dc7c-f691-5a1cb038f46e ea1c08b6-f955-dc7c-f691-5a1cb038f46e}"
2026-02-28T09:03:45.814Z [ERROR] server-2ac257ea-c882-89cf-b1bf-f840f768a590: failed to heartbeat to: peer=ea1c08b6-f955-dc7c-f691-5a1cb038f46e backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeadershipTransferToInvalidID (0.70s)
=== RUN   TestRaft_LeadershipTransferToInvalidAddress
2026-02-28T09:03:45.815Z [INFO]  server-63137944-9bd2-93b4-dbe6-598729e03285: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63137944-9bd2-93b4-dbe6-598729e03285 Address:63137944-9bd2-93b4-dbe6-598729e03285} {Suffrage:Voter ID:server-8b153da5-b832-a524-6194-68431143a479 Address:8b153da5-b832-a524-6194-68431143a479} {Suffrage:Voter ID:server-4c170abe-8255-ea11-3c34-77b59904096d Address:4c170abe-8255-ea11-3c34-77b59904096d}]"
2026-02-28T09:03:45.815Z [INFO]  server-8b153da5-b832-a524-6194-68431143a479: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63137944-9bd2-93b4-dbe6-598729e03285 Address:63137944-9bd2-93b4-dbe6-598729e03285} {Suffrage:Voter ID:server-8b153da5-b832-a524-6194-68431143a479 Address:8b153da5-b832-a524-6194-68431143a479} {Suffrage:Voter ID:server-4c170abe-8255-ea11-3c34-77b59904096d Address:4c170abe-8255-ea11-3c34-77b59904096d}]"
2026-02-28T09:03:45.815Z [INFO]  server-63137944-9bd2-93b4-dbe6-598729e03285: entering follower state: follower="Node at 63137944-9bd2-93b4-dbe6-598729e03285 [Follower]" leader-address= leader-id=
2026-02-28T09:03:45.815Z [INFO]  server-8b153da5-b832-a524-6194-68431143a479: entering follower state: follower="Node at 8b153da5-b832-a524-6194-68431143a479 [Follower]" leader-address= leader-id=
2026-02-28T09:03:45.815Z [INFO]  server-4c170abe-8255-ea11-3c34-77b59904096d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-63137944-9bd2-93b4-dbe6-598729e03285 Address:63137944-9bd2-93b4-dbe6-598729e03285} {Suffrage:Voter ID:server-8b153da5-b832-a524-6194-68431143a479 Address:8b153da5-b832-a524-6194-68431143a479} {Suffrage:Voter ID:server-4c170abe-8255-ea11-3c34-77b59904096d Address:4c170abe-8255-ea11-3c34-77b59904096d}]"
2026-02-28T09:03:45.815Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:45.815Z [INFO]  server-4c170abe-8255-ea11-3c34-77b59904096d: entering follower state: follower="Node at 4c170abe-8255-ea11-3c34-77b59904096d [Follower]" leader-address= leader-id=
2026-02-28T09:03:45.895Z [WARN]  server-4c170abe-8255-ea11-3c34-77b59904096d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:45.895Z [INFO]  server-4c170abe-8255-ea11-3c34-77b59904096d: entering candidate state: node="Node at 4c170abe-8255-ea11-3c34-77b59904096d [Candidate]" term=2
2026-02-28T09:03:45.895Z [INFO]  server-4c170abe-8255-ea11-3c34-77b59904096d: election won: term=2 tally=2
2026-02-28T09:03:45.895Z [INFO]  server-4c170abe-8255-ea11-3c34-77b59904096d: entering leader state: leader="Node at 4c170abe-8255-ea11-3c34-77b59904096d [Leader]"
2026-02-28T09:03:45.895Z [INFO]  server-4c170abe-8255-ea11-3c34-77b59904096d: added peer, starting replication: peer=server-63137944-9bd2-93b4-dbe6-598729e03285
2026-02-28T09:03:45.895Z [INFO]  server-4c170abe-8255-ea11-3c34-77b59904096d: added peer, starting replication: peer=server-8b153da5-b832-a524-6194-68431143a479
2026-02-28T09:03:45.896Z [INFO]  server-4c170abe-8255-ea11-3c34-77b59904096d: pipelining replication: peer="{Voter server-8b153da5-b832-a524-6194-68431143a479 8b153da5-b832-a524-6194-68431143a479}"
2026-02-28T09:03:45.896Z [INFO]  server-4c170abe-8255-ea11-3c34-77b59904096d: pipelining replication: peer="{Voter server-63137944-9bd2-93b4-dbe6-598729e03285 63137944-9bd2-93b4-dbe6-598729e03285}"
2026-02-28T09:03:46.002Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:45.896009742 +0000 UTC m=+103.859806281 (2 nodes), 80.17572ms from start of poll, 80.64272ms from cluster start. Timeout at 2026-02-28 09:03:46.001014342 +0000 UTC m=+103.964810921, 105.00464ms after stability
2026-02-28T09:03:46.002Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:46.107Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:46.002375782 +0000 UTC m=+103.966172321 (1 nodes), 2.6µs from start of poll, 187.00876ms from cluster start. Timeout at 2026-02-28 09:03:46.107406462 +0000 UTC m=+104.071203001, 105.03068ms after stability
2026-02-28T09:03:46.107Z [INFO]  server-4c170abe-8255-ea11-3c34-77b59904096d: aborting pipeline replication: peer="{Voter server-63137944-9bd2-93b4-dbe6-598729e03285 63137944-9bd2-93b4-dbe6-598729e03285}"
2026-02-28T09:03:46.107Z [INFO]  server-4c170abe-8255-ea11-3c34-77b59904096d: aborting pipeline replication: peer="{Voter server-8b153da5-b832-a524-6194-68431143a479 8b153da5-b832-a524-6194-68431143a479}"
--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.29s)
=== RUN   TestRaft_LeadershipTransferToBehindServer
2026-02-28T09:03:46.108Z [INFO]  server-e058bc01-7f31-649b-198c-f01b1eb0a8c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e058bc01-7f31-649b-198c-f01b1eb0a8c9 Address:e058bc01-7f31-649b-198c-f01b1eb0a8c9} {Suffrage:Voter ID:server-4b20ae00-0d25-3e73-7eaf-ff45db3349f1 Address:4b20ae00-0d25-3e73-7eaf-ff45db3349f1} {Suffrage:Voter ID:server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3 Address:bc2214f5-9928-f1ec-a30d-c791cbfc7af3}]"
2026-02-28T09:03:46.108Z [INFO]  server-4b20ae00-0d25-3e73-7eaf-ff45db3349f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e058bc01-7f31-649b-198c-f01b1eb0a8c9 Address:e058bc01-7f31-649b-198c-f01b1eb0a8c9} {Suffrage:Voter ID:server-4b20ae00-0d25-3e73-7eaf-ff45db3349f1 Address:4b20ae00-0d25-3e73-7eaf-ff45db3349f1} {Suffrage:Voter ID:server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3 Address:bc2214f5-9928-f1ec-a30d-c791cbfc7af3}]"
2026-02-28T09:03:46.108Z [INFO]  server-e058bc01-7f31-649b-198c-f01b1eb0a8c9: entering follower state: follower="Node at e058bc01-7f31-649b-198c-f01b1eb0a8c9 [Follower]" leader-address= leader-id=
2026-02-28T09:03:46.108Z [INFO]  server-4b20ae00-0d25-3e73-7eaf-ff45db3349f1: entering follower state: follower="Node at 4b20ae00-0d25-3e73-7eaf-ff45db3349f1 [Follower]" leader-address= leader-id=
2026-02-28T09:03:46.108Z [INFO]  server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e058bc01-7f31-649b-198c-f01b1eb0a8c9 Address:e058bc01-7f31-649b-198c-f01b1eb0a8c9} {Suffrage:Voter ID:server-4b20ae00-0d25-3e73-7eaf-ff45db3349f1 Address:4b20ae00-0d25-3e73-7eaf-ff45db3349f1} {Suffrage:Voter ID:server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3 Address:bc2214f5-9928-f1ec-a30d-c791cbfc7af3}]"
2026-02-28T09:03:46.109Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:46.109Z [INFO]  server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3: entering follower state: follower="Node at bc2214f5-9928-f1ec-a30d-c791cbfc7af3 [Follower]" leader-address= leader-id=
2026-02-28T09:03:46.179Z [WARN]  server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:46.179Z [INFO]  server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3: entering candidate state: node="Node at bc2214f5-9928-f1ec-a30d-c791cbfc7af3 [Candidate]" term=2
2026-02-28T09:03:46.179Z [INFO]  server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3: election won: term=2 tally=2
2026-02-28T09:03:46.179Z [INFO]  server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3: entering leader state: leader="Node at bc2214f5-9928-f1ec-a30d-c791cbfc7af3 [Leader]"
2026-02-28T09:03:46.179Z [INFO]  server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3: added peer, starting replication: peer=server-e058bc01-7f31-649b-198c-f01b1eb0a8c9
2026-02-28T09:03:46.179Z [INFO]  server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3: added peer, starting replication: peer=server-4b20ae00-0d25-3e73-7eaf-ff45db3349f1
2026-02-28T09:03:46.180Z [INFO]  server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3: pipelining replication: peer="{Voter server-4b20ae00-0d25-3e73-7eaf-ff45db3349f1 4b20ae00-0d25-3e73-7eaf-ff45db3349f1}"
2026-02-28T09:03:46.180Z [INFO]  server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3: pipelining replication: peer="{Voter server-e058bc01-7f31-649b-198c-f01b1eb0a8c9 e058bc01-7f31-649b-198c-f01b1eb0a8c9}"
2026-02-28T09:03:46.285Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:46.179840902 +0000 UTC m=+104.143637441 (1 nodes), 70.79732ms from start of poll, 71.22712ms from cluster start. Timeout at 2026-02-28 09:03:46.284846222 +0000 UTC m=+104.248642761, 105.00532ms after stability
2026-02-28T09:03:46.285Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:46.390Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:46.285695902 +0000 UTC m=+104.249492441 (2 nodes), 1.68µs from start of poll, 177.08212ms from cluster start. Timeout at 2026-02-28 09:03:46.390727702 +0000 UTC m=+104.354524241, 105.0318ms after stability
2026-02-28T09:03:46.452Z [INFO]  server-e058bc01-7f31-649b-198c-f01b1eb0a8c9: entering candidate state: node="Node at e058bc01-7f31-649b-198c-f01b1eb0a8c9 [Candidate]" term=3
2026-02-28T09:03:46.452Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:46.452Z [INFO]  server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3: entering follower state: follower="Node at bc2214f5-9928-f1ec-a30d-c791cbfc7af3 [Follower]" leader-address= leader-id=
2026-02-28T09:03:46.452Z [INFO]  server-e058bc01-7f31-649b-198c-f01b1eb0a8c9: election won: term=3 tally=2
2026-02-28T09:03:46.452Z [INFO]  server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3: aborting pipeline replication: peer="{Voter server-4b20ae00-0d25-3e73-7eaf-ff45db3349f1 4b20ae00-0d25-3e73-7eaf-ff45db3349f1}"
2026-02-28T09:03:46.452Z [INFO]  server-e058bc01-7f31-649b-198c-f01b1eb0a8c9: entering leader state: leader="Node at e058bc01-7f31-649b-198c-f01b1eb0a8c9 [Leader]"
2026-02-28T09:03:46.452Z [INFO]  server-e058bc01-7f31-649b-198c-f01b1eb0a8c9: added peer, starting replication: peer=server-4b20ae00-0d25-3e73-7eaf-ff45db3349f1
2026-02-28T09:03:46.452Z [INFO]  server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3: aborting pipeline replication: peer="{Voter server-e058bc01-7f31-649b-198c-f01b1eb0a8c9 e058bc01-7f31-649b-198c-f01b1eb0a8c9}"
2026-02-28T09:03:46.452Z [INFO]  server-e058bc01-7f31-649b-198c-f01b1eb0a8c9: added peer, starting replication: peer=server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3
2026-02-28T09:03:46.452Z [INFO]  server-e058bc01-7f31-649b-198c-f01b1eb0a8c9: pipelining replication: peer="{Voter server-4b20ae00-0d25-3e73-7eaf-ff45db3349f1 4b20ae00-0d25-3e73-7eaf-ff45db3349f1}"
2026-02-28T09:03:46.452Z [INFO]  server-e058bc01-7f31-649b-198c-f01b1eb0a8c9: pipelining replication: peer="{Voter server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3 bc2214f5-9928-f1ec-a30d-c791cbfc7af3}"
2026-02-28T09:03:46.557Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:46.452416102 +0000 UTC m=+104.416212641 (1 nodes), 131.44µs from start of poll, 343.80232ms from cluster start. Timeout at 2026-02-28 09:03:46.557447142 +0000 UTC m=+104.521243681, 105.03104ms after stability
2026-02-28T09:03:46.557Z [INFO]  server-e058bc01-7f31-649b-198c-f01b1eb0a8c9: aborting pipeline replication: peer="{Voter server-4b20ae00-0d25-3e73-7eaf-ff45db3349f1 4b20ae00-0d25-3e73-7eaf-ff45db3349f1}"
2026-02-28T09:03:46.557Z [INFO]  server-e058bc01-7f31-649b-198c-f01b1eb0a8c9: aborting pipeline replication: peer="{Voter server-bc2214f5-9928-f1ec-a30d-c791cbfc7af3 bc2214f5-9928-f1ec-a30d-c791cbfc7af3}"
--- PASS: TestRaft_LeadershipTransferToBehindServer (0.45s)
=== RUN   TestRaft_LeadershipTransferToItself
2026-02-28T09:03:46.558Z [INFO]  server-41c89cb4-1eaf-52ff-5a91-a582f194a4fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41c89cb4-1eaf-52ff-5a91-a582f194a4fa Address:41c89cb4-1eaf-52ff-5a91-a582f194a4fa} {Suffrage:Voter ID:server-ad1c2449-6a53-5b71-1fc8-e67d18bc084f Address:ad1c2449-6a53-5b71-1fc8-e67d18bc084f} {Suffrage:Voter ID:server-1df4dc9d-f653-fd66-dbc8-16348dca4146 Address:1df4dc9d-f653-fd66-dbc8-16348dca4146}]"
2026-02-28T09:03:46.558Z [INFO]  server-41c89cb4-1eaf-52ff-5a91-a582f194a4fa: entering follower state: follower="Node at 41c89cb4-1eaf-52ff-5a91-a582f194a4fa [Follower]" leader-address= leader-id=
2026-02-28T09:03:46.558Z [INFO]  server-ad1c2449-6a53-5b71-1fc8-e67d18bc084f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41c89cb4-1eaf-52ff-5a91-a582f194a4fa Address:41c89cb4-1eaf-52ff-5a91-a582f194a4fa} {Suffrage:Voter ID:server-ad1c2449-6a53-5b71-1fc8-e67d18bc084f Address:ad1c2449-6a53-5b71-1fc8-e67d18bc084f} {Suffrage:Voter ID:server-1df4dc9d-f653-fd66-dbc8-16348dca4146 Address:1df4dc9d-f653-fd66-dbc8-16348dca4146}]"
2026-02-28T09:03:46.558Z [INFO]  server-ad1c2449-6a53-5b71-1fc8-e67d18bc084f: entering follower state: follower="Node at ad1c2449-6a53-5b71-1fc8-e67d18bc084f [Follower]" leader-address= leader-id=
2026-02-28T09:03:46.559Z [INFO]  server-1df4dc9d-f653-fd66-dbc8-16348dca4146: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41c89cb4-1eaf-52ff-5a91-a582f194a4fa Address:41c89cb4-1eaf-52ff-5a91-a582f194a4fa} {Suffrage:Voter ID:server-ad1c2449-6a53-5b71-1fc8-e67d18bc084f Address:ad1c2449-6a53-5b71-1fc8-e67d18bc084f} {Suffrage:Voter ID:server-1df4dc9d-f653-fd66-dbc8-16348dca4146 Address:1df4dc9d-f653-fd66-dbc8-16348dca4146}]"
2026-02-28T09:03:46.559Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:46.559Z [INFO]  server-1df4dc9d-f653-fd66-dbc8-16348dca4146: entering follower state: follower="Node at 1df4dc9d-f653-fd66-dbc8-16348dca4146 [Follower]" leader-address= leader-id=
2026-02-28T09:03:46.613Z [WARN]  server-1df4dc9d-f653-fd66-dbc8-16348dca4146: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:46.613Z [INFO]  server-1df4dc9d-f653-fd66-dbc8-16348dca4146: entering candidate state: node="Node at 1df4dc9d-f653-fd66-dbc8-16348dca4146 [Candidate]" term=2
2026-02-28T09:03:46.613Z [INFO]  server-1df4dc9d-f653-fd66-dbc8-16348dca4146: election won: term=2 tally=2
2026-02-28T09:03:46.613Z [INFO]  server-1df4dc9d-f653-fd66-dbc8-16348dca4146: entering leader state: leader="Node at 1df4dc9d-f653-fd66-dbc8-16348dca4146 [Leader]"
2026-02-28T09:03:46.613Z [INFO]  server-1df4dc9d-f653-fd66-dbc8-16348dca4146: added peer, starting replication: peer=server-41c89cb4-1eaf-52ff-5a91-a582f194a4fa
2026-02-28T09:03:46.613Z [INFO]  server-1df4dc9d-f653-fd66-dbc8-16348dca4146: added peer, starting replication: peer=server-ad1c2449-6a53-5b71-1fc8-e67d18bc084f
2026-02-28T09:03:46.613Z [INFO]  server-1df4dc9d-f653-fd66-dbc8-16348dca4146: pipelining replication: peer="{Voter server-ad1c2449-6a53-5b71-1fc8-e67d18bc084f ad1c2449-6a53-5b71-1fc8-e67d18bc084f}"
2026-02-28T09:03:46.613Z [INFO]  server-1df4dc9d-f653-fd66-dbc8-16348dca4146: pipelining replication: peer="{Voter server-41c89cb4-1eaf-52ff-5a91-a582f194a4fa 41c89cb4-1eaf-52ff-5a91-a582f194a4fa}"
2026-02-28T09:03:46.719Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:46.613592382 +0000 UTC m=+104.577388921 (1 nodes), 54.47832ms from start of poll, 55.07004ms from cluster start. Timeout at 2026-02-28 09:03:46.718605502 +0000 UTC m=+104.682402041, 105.01312ms after stability
2026-02-28T09:03:46.719Z [INFO]  server-1df4dc9d-f653-fd66-dbc8-16348dca4146: cannot transfer leadership to itself
2026-02-28T09:03:46.719Z [INFO]  server-1df4dc9d-f653-fd66-dbc8-16348dca4146: aborting pipeline replication: peer="{Voter server-41c89cb4-1eaf-52ff-5a91-a582f194a4fa 41c89cb4-1eaf-52ff-5a91-a582f194a4fa}"
2026-02-28T09:03:46.719Z [INFO]  server-1df4dc9d-f653-fd66-dbc8-16348dca4146: aborting pipeline replication: peer="{Voter server-ad1c2449-6a53-5b71-1fc8-e67d18bc084f ad1c2449-6a53-5b71-1fc8-e67d18bc084f}"
--- PASS: TestRaft_LeadershipTransferToItself (0.16s)
=== RUN   TestRaft_LeadershipTransferLeaderRejectsClientRequests
2026-02-28T09:03:46.720Z [INFO]  server-b60112bd-7eeb-f5e5-b2f9-b39b2f009506: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b60112bd-7eeb-f5e5-b2f9-b39b2f009506 Address:b60112bd-7eeb-f5e5-b2f9-b39b2f009506} {Suffrage:Voter ID:server-904a1522-59f6-b9b4-755c-d3c5ff8c4651 Address:904a1522-59f6-b9b4-755c-d3c5ff8c4651} {Suffrage:Voter ID:server-5b5f87c5-bc51-47f3-3e18-ab832bcf8a01 Address:5b5f87c5-bc51-47f3-3e18-ab832bcf8a01}]"
2026-02-28T09:03:46.720Z [INFO]  server-904a1522-59f6-b9b4-755c-d3c5ff8c4651: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b60112bd-7eeb-f5e5-b2f9-b39b2f009506 Address:b60112bd-7eeb-f5e5-b2f9-b39b2f009506} {Suffrage:Voter ID:server-904a1522-59f6-b9b4-755c-d3c5ff8c4651 Address:904a1522-59f6-b9b4-755c-d3c5ff8c4651} {Suffrage:Voter ID:server-5b5f87c5-bc51-47f3-3e18-ab832bcf8a01 Address:5b5f87c5-bc51-47f3-3e18-ab832bcf8a01}]"
2026-02-28T09:03:46.720Z [INFO]  server-b60112bd-7eeb-f5e5-b2f9-b39b2f009506: entering follower state: follower="Node at b60112bd-7eeb-f5e5-b2f9-b39b2f009506 [Follower]" leader-address= leader-id=
2026-02-28T09:03:46.720Z [INFO]  server-904a1522-59f6-b9b4-755c-d3c5ff8c4651: entering follower state: follower="Node at 904a1522-59f6-b9b4-755c-d3c5ff8c4651 [Follower]" leader-address= leader-id=
2026-02-28T09:03:46.720Z [INFO]  server-5b5f87c5-bc51-47f3-3e18-ab832bcf8a01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b60112bd-7eeb-f5e5-b2f9-b39b2f009506 Address:b60112bd-7eeb-f5e5-b2f9-b39b2f009506} {Suffrage:Voter ID:server-904a1522-59f6-b9b4-755c-d3c5ff8c4651 Address:904a1522-59f6-b9b4-755c-d3c5ff8c4651} {Suffrage:Voter ID:server-5b5f87c5-bc51-47f3-3e18-ab832bcf8a01 Address:5b5f87c5-bc51-47f3-3e18-ab832bcf8a01}]"
2026-02-28T09:03:46.720Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:46.720Z [INFO]  server-5b5f87c5-bc51-47f3-3e18-ab832bcf8a01: entering follower state: follower="Node at 5b5f87c5-bc51-47f3-3e18-ab832bcf8a01 [Follower]" leader-address= leader-id=
2026-02-28T09:03:46.774Z [WARN]  server-5b5f87c5-bc51-47f3-3e18-ab832bcf8a01: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:46.775Z [INFO]  server-5b5f87c5-bc51-47f3-3e18-ab832bcf8a01: entering candidate state: node="Node at 5b5f87c5-bc51-47f3-3e18-ab832bcf8a01 [Candidate]" term=2
2026-02-28T09:03:46.775Z [INFO]  server-5b5f87c5-bc51-47f3-3e18-ab832bcf8a01: election won: term=2 tally=2
2026-02-28T09:03:46.775Z [INFO]  server-5b5f87c5-bc51-47f3-3e18-ab832bcf8a01: entering leader state: leader="Node at 5b5f87c5-bc51-47f3-3e18-ab832bcf8a01 [Leader]"
2026-02-28T09:03:46.775Z [INFO]  server-5b5f87c5-bc51-47f3-3e18-ab832bcf8a01: added peer, starting replication: peer=server-b60112bd-7eeb-f5e5-b2f9-b39b2f009506
2026-02-28T09:03:46.775Z [INFO]  server-5b5f87c5-bc51-47f3-3e18-ab832bcf8a01: added peer, starting replication: peer=server-904a1522-59f6-b9b4-755c-d3c5ff8c4651
2026-02-28T09:03:46.775Z [INFO]  server-5b5f87c5-bc51-47f3-3e18-ab832bcf8a01: pipelining replication: peer="{Voter server-904a1522-59f6-b9b4-755c-d3c5ff8c4651 904a1522-59f6-b9b4-755c-d3c5ff8c4651}"
2026-02-28T09:03:46.782Z [INFO]  server-5b5f87c5-bc51-47f3-3e18-ab832bcf8a01: pipelining replication: peer="{Voter server-b60112bd-7eeb-f5e5-b2f9-b39b2f009506 b60112bd-7eeb-f5e5-b2f9-b39b2f009506}"
2026-02-28T09:03:46.783Z [WARN]  server-b60112bd-7eeb-f5e5-b2f9-b39b2f009506: rejecting vote request since our last term is greater: candidate=5b5f87c5-bc51-47f3-3e18-ab832bcf8a01 last-term=2 last-candidate-term=1
2026-02-28T09:03:46.888Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:46.783333782 +0000 UTC m=+104.747130321 (1 nodes), 62.56732ms from start of poll, 63.01644ms from cluster start. Timeout at 2026-02-28 09:03:46.888334862 +0000 UTC m=+104.852131441, 105.00112ms after stability
    raft_test.go:2457: waiting on future 0
    raft_test.go:2457: waiting on future 1
    raft_test.go:2457: waiting on future 2
    raft_test.go:2457: waiting on future 3
    raft_test.go:2457: waiting on future 4
    raft_test.go:2457: waiting on future 5
    raft_test.go:2457: waiting on future 6
2026-02-28T09:03:46.888Z [INFO]  server-5b5f87c5-bc51-47f3-3e18-ab832bcf8a01: aborting pipeline replication: peer="{Voter server-b60112bd-7eeb-f5e5-b2f9-b39b2f009506 b60112bd-7eeb-f5e5-b2f9-b39b2f009506}"
2026-02-28T09:03:46.888Z [INFO]  server-5b5f87c5-bc51-47f3-3e18-ab832bcf8a01: aborting pipeline replication: peer="{Voter server-904a1522-59f6-b9b4-755c-d3c5ff8c4651 904a1522-59f6-b9b4-755c-d3c5ff8c4651}"
--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.17s)
=== RUN   TestRaft_LeadershipTransferLeaderReplicationTimeout
2026-02-28T09:03:46.889Z [INFO]  server-2de9c558-4c83-cc43-ca49-14374c1a1779: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2de9c558-4c83-cc43-ca49-14374c1a1779 Address:2de9c558-4c83-cc43-ca49-14374c1a1779} {Suffrage:Voter ID:server-4b982eeb-c3d1-750d-1be0-46ffc23cefb4 Address:4b982eeb-c3d1-750d-1be0-46ffc23cefb4} {Suffrage:Voter ID:server-ec93b378-6eb8-94d6-f99e-614a0bcea112 Address:ec93b378-6eb8-94d6-f99e-614a0bcea112}]"
2026-02-28T09:03:46.889Z [INFO]  server-2de9c558-4c83-cc43-ca49-14374c1a1779: entering follower state: follower="Node at 2de9c558-4c83-cc43-ca49-14374c1a1779 [Follower]" leader-address= leader-id=
2026-02-28T09:03:46.889Z [INFO]  server-4b982eeb-c3d1-750d-1be0-46ffc23cefb4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2de9c558-4c83-cc43-ca49-14374c1a1779 Address:2de9c558-4c83-cc43-ca49-14374c1a1779} {Suffrage:Voter ID:server-4b982eeb-c3d1-750d-1be0-46ffc23cefb4 Address:4b982eeb-c3d1-750d-1be0-46ffc23cefb4} {Suffrage:Voter ID:server-ec93b378-6eb8-94d6-f99e-614a0bcea112 Address:ec93b378-6eb8-94d6-f99e-614a0bcea112}]"
2026-02-28T09:03:46.889Z [INFO]  server-4b982eeb-c3d1-750d-1be0-46ffc23cefb4: entering follower state: follower="Node at 4b982eeb-c3d1-750d-1be0-46ffc23cefb4 [Follower]" leader-address= leader-id=
2026-02-28T09:03:46.889Z [INFO]  server-ec93b378-6eb8-94d6-f99e-614a0bcea112: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2de9c558-4c83-cc43-ca49-14374c1a1779 Address:2de9c558-4c83-cc43-ca49-14374c1a1779} {Suffrage:Voter ID:server-4b982eeb-c3d1-750d-1be0-46ffc23cefb4 Address:4b982eeb-c3d1-750d-1be0-46ffc23cefb4} {Suffrage:Voter ID:server-ec93b378-6eb8-94d6-f99e-614a0bcea112 Address:ec93b378-6eb8-94d6-f99e-614a0bcea112}]"
2026-02-28T09:03:46.889Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:46.889Z [INFO]  server-ec93b378-6eb8-94d6-f99e-614a0bcea112: entering follower state: follower="Node at ec93b378-6eb8-94d6-f99e-614a0bcea112 [Follower]" leader-address= leader-id=
2026-02-28T09:03:46.942Z [WARN]  server-2de9c558-4c83-cc43-ca49-14374c1a1779: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:46.942Z [INFO]  server-2de9c558-4c83-cc43-ca49-14374c1a1779: entering candidate state: node="Node at 2de9c558-4c83-cc43-ca49-14374c1a1779 [Candidate]" term=2
2026-02-28T09:03:46.942Z [INFO]  server-2de9c558-4c83-cc43-ca49-14374c1a1779: election won: term=2 tally=2
2026-02-28T09:03:46.942Z [INFO]  server-2de9c558-4c83-cc43-ca49-14374c1a1779: entering leader state: leader="Node at 2de9c558-4c83-cc43-ca49-14374c1a1779 [Leader]"
2026-02-28T09:03:46.942Z [INFO]  server-2de9c558-4c83-cc43-ca49-14374c1a1779: added peer, starting replication: peer=server-4b982eeb-c3d1-750d-1be0-46ffc23cefb4
2026-02-28T09:03:46.942Z [INFO]  server-2de9c558-4c83-cc43-ca49-14374c1a1779: added peer, starting replication: peer=server-ec93b378-6eb8-94d6-f99e-614a0bcea112
2026-02-28T09:03:46.942Z [INFO]  server-2de9c558-4c83-cc43-ca49-14374c1a1779: pipelining replication: peer="{Voter server-ec93b378-6eb8-94d6-f99e-614a0bcea112 ec93b378-6eb8-94d6-f99e-614a0bcea112}"
2026-02-28T09:03:46.942Z [INFO]  server-2de9c558-4c83-cc43-ca49-14374c1a1779: pipelining replication: peer="{Voter server-4b982eeb-c3d1-750d-1be0-46ffc23cefb4 4b982eeb-c3d1-750d-1be0-46ffc23cefb4}"
2026-02-28T09:03:47.048Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:46.942808942 +0000 UTC m=+104.906605481 (1 nodes), 52.84904ms from start of poll, 53.32248ms from cluster start. Timeout at 2026-02-28 09:03:47.047820942 +0000 UTC m=+105.011617481, 105.012ms after stability
2026-02-28T09:03:47.048Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:47.153Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:47.048268262 +0000 UTC m=+105.012064801 (2 nodes), 3.28µs from start of poll, 158.7818ms from cluster start. Timeout at 2026-02-28 09:03:47.153284422 +0000 UTC m=+105.117080961, 105.01616ms after stability
2026-02-28T09:03:47.686Z [INFO]  server-2de9c558-4c83-cc43-ca49-14374c1a1779: aborting pipeline replication: peer="{Voter server-4b982eeb-c3d1-750d-1be0-46ffc23cefb4 4b982eeb-c3d1-750d-1be0-46ffc23cefb4}"
2026-02-28T09:03:47.686Z [INFO]  server-2de9c558-4c83-cc43-ca49-14374c1a1779: aborting pipeline replication: peer="{Voter server-ec93b378-6eb8-94d6-f99e-614a0bcea112 ec93b378-6eb8-94d6-f99e-614a0bcea112}"
--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.80s)
=== RUN   TestRaft_LeadershipTransferIgnoresNonvoters
2026-02-28T09:03:47.687Z [INFO]  server-765aecc4-8387-c74f-a5a4-f3bcda99c254: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-765aecc4-8387-c74f-a5a4-f3bcda99c254 Address:765aecc4-8387-c74f-a5a4-f3bcda99c254} {Suffrage:Voter ID:server-b17b1227-a546-ad21-523d-3c7cb392c961 Address:b17b1227-a546-ad21-523d-3c7cb392c961}]"
2026-02-28T09:03:47.687Z [INFO]  server-765aecc4-8387-c74f-a5a4-f3bcda99c254: entering follower state: follower="Node at 765aecc4-8387-c74f-a5a4-f3bcda99c254 [Follower]" leader-address= leader-id=
2026-02-28T09:03:47.687Z [INFO]  server-b17b1227-a546-ad21-523d-3c7cb392c961: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-765aecc4-8387-c74f-a5a4-f3bcda99c254 Address:765aecc4-8387-c74f-a5a4-f3bcda99c254} {Suffrage:Voter ID:server-b17b1227-a546-ad21-523d-3c7cb392c961 Address:b17b1227-a546-ad21-523d-3c7cb392c961}]"
2026-02-28T09:03:47.687Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:47.687Z [INFO]  server-b17b1227-a546-ad21-523d-3c7cb392c961: entering follower state: follower="Node at b17b1227-a546-ad21-523d-3c7cb392c961 [Follower]" leader-address= leader-id=
2026-02-28T09:03:47.764Z [WARN]  server-b17b1227-a546-ad21-523d-3c7cb392c961: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:47.764Z [INFO]  server-b17b1227-a546-ad21-523d-3c7cb392c961: entering candidate state: node="Node at b17b1227-a546-ad21-523d-3c7cb392c961 [Candidate]" term=2
2026-02-28T09:03:47.764Z [INFO]  server-b17b1227-a546-ad21-523d-3c7cb392c961: election won: term=2 tally=2
2026-02-28T09:03:47.764Z [INFO]  server-b17b1227-a546-ad21-523d-3c7cb392c961: entering leader state: leader="Node at b17b1227-a546-ad21-523d-3c7cb392c961 [Leader]"
2026-02-28T09:03:47.764Z [INFO]  server-b17b1227-a546-ad21-523d-3c7cb392c961: added peer, starting replication: peer=server-765aecc4-8387-c74f-a5a4-f3bcda99c254
2026-02-28T09:03:47.765Z [INFO]  server-b17b1227-a546-ad21-523d-3c7cb392c961: pipelining replication: peer="{Voter server-765aecc4-8387-c74f-a5a4-f3bcda99c254 765aecc4-8387-c74f-a5a4-f3bcda99c254}"
2026-02-28T09:03:47.870Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:47.764911382 +0000 UTC m=+105.728707921 (1 nodes), 77.01712ms from start of poll, 77.36664ms from cluster start. Timeout at 2026-02-28 09:03:47.869930982 +0000 UTC m=+105.833727521, 105.0196ms after stability
2026-02-28T09:03:47.870Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:47.976Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:47.871017342 +0000 UTC m=+105.834813881 (1 nodes), 2.84µs from start of poll, 183.4726ms from cluster start. Timeout at 2026-02-28 09:03:47.976140622 +0000 UTC m=+105.939937161, 105.12328ms after stability
2026-02-28T09:03:47.977Z [INFO]  server-b17b1227-a546-ad21-523d-3c7cb392c961: updating configuration: command=DemoteVoter server-id=server-765aecc4-8387-c74f-a5a4-f3bcda99c254 server-addr= servers="[{Suffrage:Nonvoter ID:server-765aecc4-8387-c74f-a5a4-f3bcda99c254 Address:765aecc4-8387-c74f-a5a4-f3bcda99c254} {Suffrage:Voter ID:server-b17b1227-a546-ad21-523d-3c7cb392c961 Address:b17b1227-a546-ad21-523d-3c7cb392c961}]"
2026-02-28T09:03:47.977Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:48.082Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:47.977150462 +0000 UTC m=+105.940947001 (1 nodes), 960ns from start of poll, 289.60572ms from cluster start. Timeout at 2026-02-28 09:03:48.082232622 +0000 UTC m=+106.046029201, 105.0822ms after stability
2026-02-28T09:03:48.082Z [INFO]  server-b17b1227-a546-ad21-523d-3c7cb392c961: aborting pipeline replication: peer="{Voter server-765aecc4-8387-c74f-a5a4-f3bcda99c254 765aecc4-8387-c74f-a5a4-f3bcda99c254}"
--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.40s)
=== RUN   TestRaft_LeadershipTransferStopRightAway
--- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s)
=== RUN   TestRaft_GetConfigurationNoBootstrap
2026-02-28T09:03:48.083Z [INFO]  server-0edb84be-adc2-2e68-8028-3772222d4720: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0edb84be-adc2-2e68-8028-3772222d4720 Address:0edb84be-adc2-2e68-8028-3772222d4720} {Suffrage:Voter ID:server-bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e Address:bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e}]"
2026-02-28T09:03:48.083Z [INFO]  server-bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0edb84be-adc2-2e68-8028-3772222d4720 Address:0edb84be-adc2-2e68-8028-3772222d4720} {Suffrage:Voter ID:server-bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e Address:bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e}]"
2026-02-28T09:03:48.083Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:48.083Z [INFO]  server-0edb84be-adc2-2e68-8028-3772222d4720: entering follower state: follower="Node at 0edb84be-adc2-2e68-8028-3772222d4720 [Follower]" leader-address= leader-id=
2026-02-28T09:03:48.083Z [INFO]  server-bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e: entering follower state: follower="Node at bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e [Follower]" leader-address= leader-id=
2026-02-28T09:03:48.176Z [WARN]  server-bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:48.176Z [INFO]  server-bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e: entering candidate state: node="Node at bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e [Candidate]" term=2
2026-02-28T09:03:48.176Z [INFO]  server-bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e: election won: term=2 tally=2
2026-02-28T09:03:48.176Z [INFO]  server-bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e: entering leader state: leader="Node at bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e [Leader]"
2026-02-28T09:03:48.176Z [INFO]  server-bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e: added peer, starting replication: peer=server-0edb84be-adc2-2e68-8028-3772222d4720
2026-02-28T09:03:48.177Z [INFO]  server-bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e: pipelining replication: peer="{Voter server-0edb84be-adc2-2e68-8028-3772222d4720 0edb84be-adc2-2e68-8028-3772222d4720}"
2026-02-28T09:03:48.281Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:48.176939822 +0000 UTC m=+106.140736401 (1 nodes), 93.4632ms from start of poll, 93.79424ms from cluster start. Timeout at 2026-02-28 09:03:48.281943262 +0000 UTC m=+106.245739801, 105.0034ms after stability
2026-02-28T09:03:48.282Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:48.387Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:48.282065662 +0000 UTC m=+106.245862201 (1 nodes), 2.8µs from start of poll, 198.92004ms from cluster start. Timeout at 2026-02-28 09:03:48.387079102 +0000 UTC m=+106.350875681, 105.01348ms after stability
2026-02-28T09:03:48.398Z [INFO]  server-bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0edb84be-adc2-2e68-8028-3772222d4720 Address:0edb84be-adc2-2e68-8028-3772222d4720} {Suffrage:Voter ID:server-bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e Address:bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e}]"
2026-02-28T09:03:48.399Z [INFO]  server-bcc9bfcf-ff5e-ac1e-aa96-f88c1c9b0c5e: aborting pipeline replication: peer="{Voter server-0edb84be-adc2-2e68-8028-3772222d4720 0edb84be-adc2-2e68-8028-3772222d4720}"
--- PASS: TestRaft_GetConfigurationNoBootstrap (0.32s)
=== RUN   TestRaft_LogStoreIsMonotonic
2026-02-28T09:03:48.399Z [INFO]  server-ed4d196c-7863-4f2f-7871-bc2c733959bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed4d196c-7863-4f2f-7871-bc2c733959bf Address:ed4d196c-7863-4f2f-7871-bc2c733959bf}]"
2026-02-28T09:03:48.399Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:48.399Z [INFO]  server-ed4d196c-7863-4f2f-7871-bc2c733959bf: entering follower state: follower="Node at ed4d196c-7863-4f2f-7871-bc2c733959bf [Follower]" leader-address= leader-id=
2026-02-28T09:03:48.486Z [WARN]  server-ed4d196c-7863-4f2f-7871-bc2c733959bf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:48.486Z [INFO]  server-ed4d196c-7863-4f2f-7871-bc2c733959bf: entering candidate state: node="Node at ed4d196c-7863-4f2f-7871-bc2c733959bf [Candidate]" term=2
2026-02-28T09:03:48.486Z [INFO]  server-ed4d196c-7863-4f2f-7871-bc2c733959bf: election won: term=2 tally=1
2026-02-28T09:03:48.486Z [INFO]  server-ed4d196c-7863-4f2f-7871-bc2c733959bf: entering leader state: leader="Node at ed4d196c-7863-4f2f-7871-bc2c733959bf [Leader]"
2026-02-28T09:03:48.592Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:48.486437982 +0000 UTC m=+106.450234561 (1 nodes), 86.60076ms from start of poll, 86.8306ms from cluster start. Timeout at 2026-02-28 09:03:48.591443382 +0000 UTC m=+106.555239921, 105.00536ms after stability
--- PASS: TestRaft_LogStoreIsMonotonic (0.19s)
=== RUN   TestRaft_CacheLogWithStoreError
2026-02-28T09:03:48.592Z [INFO]  server-cbdffaa9-1834-069e-2700-6c57f56ae197: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cbdffaa9-1834-069e-2700-6c57f56ae197 Address:cbdffaa9-1834-069e-2700-6c57f56ae197} {Suffrage:Voter ID:server-2563dd64-3154-f278-87cf-3f8ffefa7076 Address:2563dd64-3154-f278-87cf-3f8ffefa7076}]"
2026-02-28T09:03:48.592Z [INFO]  server-2563dd64-3154-f278-87cf-3f8ffefa7076: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cbdffaa9-1834-069e-2700-6c57f56ae197 Address:cbdffaa9-1834-069e-2700-6c57f56ae197} {Suffrage:Voter ID:server-2563dd64-3154-f278-87cf-3f8ffefa7076 Address:2563dd64-3154-f278-87cf-3f8ffefa7076}]"
2026-02-28T09:03:48.592Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:48.592Z [INFO]  server-cbdffaa9-1834-069e-2700-6c57f56ae197: entering follower state: follower="Node at cbdffaa9-1834-069e-2700-6c57f56ae197 [Follower]" leader-address= leader-id=
2026-02-28T09:03:48.592Z [INFO]  server-2563dd64-3154-f278-87cf-3f8ffefa7076: entering follower state: follower="Node at 2563dd64-3154-f278-87cf-3f8ffefa7076 [Follower]" leader-address= leader-id=
2026-02-28T09:03:48.650Z [WARN]  server-2563dd64-3154-f278-87cf-3f8ffefa7076: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:48.650Z [INFO]  server-2563dd64-3154-f278-87cf-3f8ffefa7076: entering candidate state: node="Node at 2563dd64-3154-f278-87cf-3f8ffefa7076 [Candidate]" term=2
2026-02-28T09:03:48.650Z [INFO]  server-2563dd64-3154-f278-87cf-3f8ffefa7076: election won: term=2 tally=2
2026-02-28T09:03:48.650Z [INFO]  server-2563dd64-3154-f278-87cf-3f8ffefa7076: entering leader state: leader="Node at 2563dd64-3154-f278-87cf-3f8ffefa7076 [Leader]"
2026-02-28T09:03:48.650Z [INFO]  server-2563dd64-3154-f278-87cf-3f8ffefa7076: added peer, starting replication: peer=server-cbdffaa9-1834-069e-2700-6c57f56ae197
2026-02-28T09:03:48.650Z [INFO]  server-2563dd64-3154-f278-87cf-3f8ffefa7076: pipelining replication: peer="{Voter server-cbdffaa9-1834-069e-2700-6c57f56ae197 cbdffaa9-1834-069e-2700-6c57f56ae197}"
2026-02-28T09:03:48.755Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:48.650430422 +0000 UTC m=+106.614226961 (1 nodes), 57.56844ms from start of poll, 57.8812ms from cluster start. Timeout at 2026-02-28 09:03:48.755435622 +0000 UTC m=+106.719232161, 105.0052ms after stability
2026-02-28T09:03:48.756Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:48.861Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:48.756084582 +0000 UTC m=+106.719881121 (1 nodes), 2.32µs from start of poll, 163.53536ms from cluster start. Timeout at 2026-02-28 09:03:48.861106102 +0000 UTC m=+106.824902641, 105.02152ms after stability
2026-02-28T09:03:48.861Z [ERROR] server-cbdffaa9-1834-069e-2700-6c57f56ae197: failed to append to logs: error="unable to store logs within log store, err: "some error""
2026-02-28T09:03:48.861Z [WARN]  server-cbdffaa9-1834-069e-2700-6c57f56ae197: failed to get previous log: previous-index=9 last-index=7 error="log not found"
2026-02-28T09:03:48.862Z [INFO]  server-2563dd64-3154-f278-87cf-3f8ffefa7076: aborting pipeline replication: peer="{Voter server-cbdffaa9-1834-069e-2700-6c57f56ae197 cbdffaa9-1834-069e-2700-6c57f56ae197}"
2026-02-28T09:03:48.868Z [INFO]  server-2563dd64-3154-f278-87cf-3f8ffefa7076: pipelining replication: peer="{Voter server-cbdffaa9-1834-069e-2700-6c57f56ae197 cbdffaa9-1834-069e-2700-6c57f56ae197}"
2026-02-28T09:03:48.868Z [INFO]  server-cbdffaa9-1834-069e-2700-6c57f56ae197: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cbdffaa9-1834-069e-2700-6c57f56ae197 Address:cbdffaa9-1834-069e-2700-6c57f56ae197} {Suffrage:Voter ID:server-2563dd64-3154-f278-87cf-3f8ffefa7076 Address:2563dd64-3154-f278-87cf-3f8ffefa7076}]"
2026-02-28T09:03:48.868Z [INFO]  server-2563dd64-3154-f278-87cf-3f8ffefa7076: aborting pipeline replication: peer="{Voter server-cbdffaa9-1834-069e-2700-6c57f56ae197 cbdffaa9-1834-069e-2700-6c57f56ae197}"
--- PASS: TestRaft_CacheLogWithStoreError (0.28s)
=== RUN   TestRaft_ReloadConfig
2026-02-28T09:03:48.869Z [INFO]  server-18d72b1e-7f2b-5eef-3034-f4094331a9ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18d72b1e-7f2b-5eef-3034-f4094331a9ed Address:18d72b1e-7f2b-5eef-3034-f4094331a9ed}]"
2026-02-28T09:03:48.869Z [INFO]  server-18d72b1e-7f2b-5eef-3034-f4094331a9ed: entering follower state: follower="Node at 18d72b1e-7f2b-5eef-3034-f4094331a9ed [Follower]" leader-address= leader-id=
--- PASS: TestRaft_ReloadConfig (0.00s)
=== RUN   TestRaft_ReloadConfigValidates
2026-02-28T09:03:48.870Z [INFO]  server-84c23d78-2775-5d93-fe6b-9691bf3e3273: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84c23d78-2775-5d93-fe6b-9691bf3e3273 Address:84c23d78-2775-5d93-fe6b-9691bf3e3273}]"
2026-02-28T09:03:48.870Z [INFO]  server-84c23d78-2775-5d93-fe6b-9691bf3e3273: entering follower state: follower="Node at 84c23d78-2775-5d93-fe6b-9691bf3e3273 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_ReloadConfigValidates (0.00s)
=== RUN   TestRaft_InstallSnapshot_InvalidPeers
2026-02-28T09:03:48.870Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)"
--- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s)
=== RUN   TestRaft_VoteNotGranted_WhenNodeNotInCluster
2026-02-28T09:03:48.871Z [INFO]  server-73b275e3-45dd-becf-f56e-6341dfbaf65d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73b275e3-45dd-becf-f56e-6341dfbaf65d Address:73b275e3-45dd-becf-f56e-6341dfbaf65d} {Suffrage:Voter ID:server-377f9234-0d41-3ee9-fd83-b3320887e5cb Address:377f9234-0d41-3ee9-fd83-b3320887e5cb} {Suffrage:Voter ID:server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe Address:80cb0e7f-b199-fa0d-c06d-f51a683e07fe}]"
2026-02-28T09:03:48.871Z [INFO]  server-73b275e3-45dd-becf-f56e-6341dfbaf65d: entering follower state: follower="Node at 73b275e3-45dd-becf-f56e-6341dfbaf65d [Follower]" leader-address= leader-id=
2026-02-28T09:03:48.871Z [INFO]  server-377f9234-0d41-3ee9-fd83-b3320887e5cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73b275e3-45dd-becf-f56e-6341dfbaf65d Address:73b275e3-45dd-becf-f56e-6341dfbaf65d} {Suffrage:Voter ID:server-377f9234-0d41-3ee9-fd83-b3320887e5cb Address:377f9234-0d41-3ee9-fd83-b3320887e5cb} {Suffrage:Voter ID:server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe Address:80cb0e7f-b199-fa0d-c06d-f51a683e07fe}]"
2026-02-28T09:03:48.871Z [INFO]  server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73b275e3-45dd-becf-f56e-6341dfbaf65d Address:73b275e3-45dd-becf-f56e-6341dfbaf65d} {Suffrage:Voter ID:server-377f9234-0d41-3ee9-fd83-b3320887e5cb Address:377f9234-0d41-3ee9-fd83-b3320887e5cb} {Suffrage:Voter ID:server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe Address:80cb0e7f-b199-fa0d-c06d-f51a683e07fe}]"
2026-02-28T09:03:48.871Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:48.871Z [INFO]  server-377f9234-0d41-3ee9-fd83-b3320887e5cb: entering follower state: follower="Node at 377f9234-0d41-3ee9-fd83-b3320887e5cb [Follower]" leader-address= leader-id=
2026-02-28T09:03:48.871Z [INFO]  server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: entering follower state: follower="Node at 80cb0e7f-b199-fa0d-c06d-f51a683e07fe [Follower]" leader-address= leader-id=
2026-02-28T09:03:48.955Z [WARN]  server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:48.955Z [INFO]  server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: entering candidate state: node="Node at 80cb0e7f-b199-fa0d-c06d-f51a683e07fe [Candidate]" term=2
2026-02-28T09:03:48.955Z [INFO]  server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: election won: term=2 tally=2
2026-02-28T09:03:48.955Z [INFO]  server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: entering leader state: leader="Node at 80cb0e7f-b199-fa0d-c06d-f51a683e07fe [Leader]"
2026-02-28T09:03:48.955Z [INFO]  server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: added peer, starting replication: peer=server-73b275e3-45dd-becf-f56e-6341dfbaf65d
2026-02-28T09:03:48.955Z [INFO]  server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: added peer, starting replication: peer=server-377f9234-0d41-3ee9-fd83-b3320887e5cb
2026-02-28T09:03:48.955Z [INFO]  server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: pipelining replication: peer="{Voter server-377f9234-0d41-3ee9-fd83-b3320887e5cb 377f9234-0d41-3ee9-fd83-b3320887e5cb}"
2026-02-28T09:03:48.955Z [INFO]  server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: pipelining replication: peer="{Voter server-73b275e3-45dd-becf-f56e-6341dfbaf65d 73b275e3-45dd-becf-f56e-6341dfbaf65d}"
2026-02-28T09:03:49.061Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:48.955605582 +0000 UTC m=+106.919402121 (1 nodes), 84.14072ms from start of poll, 84.56624ms from cluster start. Timeout at 2026-02-28 09:03:49.060610422 +0000 UTC m=+107.024406961, 105.00484ms after stability
2026-02-28T09:03:49.066Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:49.172Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:49.066868222 +0000 UTC m=+107.030664761 (2 nodes), 1.72µs from start of poll, 195.82888ms from cluster start. Timeout at 2026-02-28 09:03:49.171893462 +0000 UTC m=+107.135690001, 105.02524ms after stability
2026-02-28T09:03:49.172Z [INFO]  server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: updating configuration: command=RemoveServer server-id=server-73b275e3-45dd-becf-f56e-6341dfbaf65d server-addr= servers="[{Suffrage:Voter ID:server-377f9234-0d41-3ee9-fd83-b3320887e5cb Address:377f9234-0d41-3ee9-fd83-b3320887e5cb} {Suffrage:Voter ID:server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe Address:80cb0e7f-b199-fa0d-c06d-f51a683e07fe}]"
2026-02-28T09:03:49.173Z [INFO]  server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: removed peer, stopping replication: peer=server-73b275e3-45dd-becf-f56e-6341dfbaf65d last-index=3
2026-02-28T09:03:49.173Z [INFO]  server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: aborting pipeline replication: peer="{Voter server-73b275e3-45dd-becf-f56e-6341dfbaf65d 73b275e3-45dd-becf-f56e-6341dfbaf65d}"
2026-02-28T09:03:49.265Z [WARN]  server-73b275e3-45dd-becf-f56e-6341dfbaf65d: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2026-02-28T09:03:49.280Z [ERROR] server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: failed to pipeline appendEntries: peer="{Voter server-377f9234-0d41-3ee9-fd83-b3320887e5cb 377f9234-0d41-3ee9-fd83-b3320887e5cb}" error="append pipeline closed"
2026-02-28T09:03:49.280Z [INFO]  server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: aborting pipeline replication: peer="{Voter server-377f9234-0d41-3ee9-fd83-b3320887e5cb 377f9234-0d41-3ee9-fd83-b3320887e5cb}"
2026-02-28T09:03:49.285Z [ERROR] server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: failed to heartbeat to: peer=377f9234-0d41-3ee9-fd83-b3320887e5cb backoff time=10ms error="failed to connect to peer: 377f9234-0d41-3ee9-fd83-b3320887e5cb"
2026-02-28T09:03:49.290Z [ERROR] server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: failed to appendEntries to: peer="{Voter server-377f9234-0d41-3ee9-fd83-b3320887e5cb 377f9234-0d41-3ee9-fd83-b3320887e5cb}" error="failed to connect to peer: 377f9234-0d41-3ee9-fd83-b3320887e5cb"
2026-02-28T09:03:49.303Z [ERROR] server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: failed to heartbeat to: peer=377f9234-0d41-3ee9-fd83-b3320887e5cb backoff time=10ms error="failed to connect to peer: 377f9234-0d41-3ee9-fd83-b3320887e5cb"
2026-02-28T09:03:49.307Z [ERROR] server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: failed to appendEntries to: peer="{Voter server-377f9234-0d41-3ee9-fd83-b3320887e5cb 377f9234-0d41-3ee9-fd83-b3320887e5cb}" error="failed to connect to peer: 377f9234-0d41-3ee9-fd83-b3320887e5cb"
2026-02-28T09:03:49.325Z [ERROR] server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: failed to heartbeat to: peer=377f9234-0d41-3ee9-fd83-b3320887e5cb backoff time=10ms error="failed to connect to peer: 377f9234-0d41-3ee9-fd83-b3320887e5cb"
2026-02-28T09:03:49.328Z [WARN]  server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: failed to contact: server-id=server-377f9234-0d41-3ee9-fd83-b3320887e5cb time=50.1408ms
2026-02-28T09:03:49.328Z [WARN]  server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: failed to contact quorum of nodes, stepping down
2026-02-28T09:03:49.328Z [INFO]  server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: entering follower state: follower="Node at 80cb0e7f-b199-fa0d-c06d-f51a683e07fe [Follower]" leader-address= leader-id=
2026-02-28T09:03:49.328Z [ERROR] server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe: failed to appendEntries to: peer="{Voter server-377f9234-0d41-3ee9-fd83-b3320887e5cb 377f9234-0d41-3ee9-fd83-b3320887e5cb}" error="failed to connect to peer: 377f9234-0d41-3ee9-fd83-b3320887e5cb"
2026-02-28T09:03:49.384Z [WARN]  server-377f9234-0d41-3ee9-fd83-b3320887e5cb: heartbeat timeout reached, starting election: last-leader-addr=80cb0e7f-b199-fa0d-c06d-f51a683e07fe last-leader-id=server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe
2026-02-28T09:03:49.384Z [INFO]  server-377f9234-0d41-3ee9-fd83-b3320887e5cb: entering candidate state: node="Node at 377f9234-0d41-3ee9-fd83-b3320887e5cb [Candidate]" term=3
2026-02-28T09:03:49.384Z [ERROR] server-377f9234-0d41-3ee9-fd83-b3320887e5cb: failed to make requestVote RPC: target="{Voter server-80cb0e7f-b199-fa0d-c06d-f51a683e07fe 80cb0e7f-b199-fa0d-c06d-f51a683e07fe}" error="failed to connect to peer: 80cb0e7f-b199-fa0d-c06d-f51a683e07fe" term=3
2026-02-28T09:03:49.384Z [WARN]  server-377f9234-0d41-3ee9-fd83-b3320887e5cb: rejecting vote request since node is not in configuration: from=73b275e3-45dd-becf-f56e-6341dfbaf65d
--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.51s)
=== RUN   TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin
2026-02-28T09:03:49.385Z [INFO]  server-b6646186-fca6-731d-2559-df9be5188ff7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6646186-fca6-731d-2559-df9be5188ff7 Address:b6646186-fca6-731d-2559-df9be5188ff7} {Suffrage:Voter ID:server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 Address:db34e07e-ff2f-aa90-9a03-dcdfab1a29d6} {Suffrage:Voter ID:server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01 Address:bdef8c89-8b73-26cf-9c07-b5b0a0109d01}]"
2026-02-28T09:03:49.385Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6646186-fca6-731d-2559-df9be5188ff7 Address:b6646186-fca6-731d-2559-df9be5188ff7} {Suffrage:Voter ID:server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 Address:db34e07e-ff2f-aa90-9a03-dcdfab1a29d6} {Suffrage:Voter ID:server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01 Address:bdef8c89-8b73-26cf-9c07-b5b0a0109d01}]"
2026-02-28T09:03:49.385Z [INFO]  server-b6646186-fca6-731d-2559-df9be5188ff7: entering follower state: follower="Node at b6646186-fca6-731d-2559-df9be5188ff7 [Follower]" leader-address= leader-id=
2026-02-28T09:03:49.385Z [INFO]  server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6646186-fca6-731d-2559-df9be5188ff7 Address:b6646186-fca6-731d-2559-df9be5188ff7} {Suffrage:Voter ID:server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 Address:db34e07e-ff2f-aa90-9a03-dcdfab1a29d6} {Suffrage:Voter ID:server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01 Address:bdef8c89-8b73-26cf-9c07-b5b0a0109d01}]"
2026-02-28T09:03:49.385Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:49.385Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: entering follower state: follower="Node at db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 [Follower]" leader-address= leader-id=
2026-02-28T09:03:49.385Z [INFO]  server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01: entering follower state: follower="Node at bdef8c89-8b73-26cf-9c07-b5b0a0109d01 [Follower]" leader-address= leader-id=
2026-02-28T09:03:49.453Z [WARN]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:49.453Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: entering candidate state: node="Node at db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 [Candidate]" term=2
2026-02-28T09:03:49.453Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: election won: term=2 tally=2
2026-02-28T09:03:49.453Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: entering leader state: leader="Node at db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 [Leader]"
2026-02-28T09:03:49.453Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: added peer, starting replication: peer=server-b6646186-fca6-731d-2559-df9be5188ff7
2026-02-28T09:03:49.453Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: added peer, starting replication: peer=server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01
2026-02-28T09:03:49.453Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: pipelining replication: peer="{Voter server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01 bdef8c89-8b73-26cf-9c07-b5b0a0109d01}"
2026-02-28T09:03:49.453Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: pipelining replication: peer="{Voter server-b6646186-fca6-731d-2559-df9be5188ff7 b6646186-fca6-731d-2559-df9be5188ff7}"
2026-02-28T09:03:49.558Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:49.453222382 +0000 UTC m=+107.417018921 (1 nodes), 67.46816ms from start of poll, 67.93332ms from cluster start. Timeout at 2026-02-28 09:03:49.558233502 +0000 UTC m=+107.522030081, 105.01116ms after stability
2026-02-28T09:03:49.564Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:49.669Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:49.564179782 +0000 UTC m=+107.527976321 (2 nodes), 5.16µs from start of poll, 178.89072ms from cluster start. Timeout at 2026-02-28 09:03:49.669199942 +0000 UTC m=+107.632996481, 105.02016ms after stability
2026-02-28T09:03:49.670Z [ERROR] server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: failed to pipeline appendEntries: peer="{Voter server-b6646186-fca6-731d-2559-df9be5188ff7 b6646186-fca6-731d-2559-df9be5188ff7}" error="append pipeline closed"
2026-02-28T09:03:49.670Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: aborting pipeline replication: peer="{Voter server-b6646186-fca6-731d-2559-df9be5188ff7 b6646186-fca6-731d-2559-df9be5188ff7}"
2026-02-28T09:03:49.673Z [ERROR] server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: failed to heartbeat to: peer=b6646186-fca6-731d-2559-df9be5188ff7 backoff time=10ms error="failed to connect to peer: b6646186-fca6-731d-2559-df9be5188ff7"
2026-02-28T09:03:49.677Z [ERROR] server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: failed to appendEntries to: peer="{Voter server-b6646186-fca6-731d-2559-df9be5188ff7 b6646186-fca6-731d-2559-df9be5188ff7}" error="failed to connect to peer: b6646186-fca6-731d-2559-df9be5188ff7"
2026-02-28T09:03:49.692Z [ERROR] server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: failed to heartbeat to: peer=b6646186-fca6-731d-2559-df9be5188ff7 backoff time=10ms error="failed to connect to peer: b6646186-fca6-731d-2559-df9be5188ff7"
2026-02-28T09:03:49.697Z [ERROR] server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: failed to appendEntries to: peer="{Voter server-b6646186-fca6-731d-2559-df9be5188ff7 b6646186-fca6-731d-2559-df9be5188ff7}" error="failed to connect to peer: b6646186-fca6-731d-2559-df9be5188ff7"
2026-02-28T09:03:49.713Z [ERROR] server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: failed to heartbeat to: peer=b6646186-fca6-731d-2559-df9be5188ff7 backoff time=10ms error="failed to connect to peer: b6646186-fca6-731d-2559-df9be5188ff7"
2026-02-28T09:03:49.715Z [ERROR] server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: failed to appendEntries to: peer="{Voter server-b6646186-fca6-731d-2559-df9be5188ff7 b6646186-fca6-731d-2559-df9be5188ff7}" error="failed to connect to peer: b6646186-fca6-731d-2559-df9be5188ff7"
2026-02-28T09:03:49.717Z [WARN]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: failed to contact: server-id=server-b6646186-fca6-731d-2559-df9be5188ff7 time=50.3194ms
2026-02-28T09:03:49.731Z [ERROR] server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: failed to heartbeat to: peer=b6646186-fca6-731d-2559-df9be5188ff7 backoff time=20ms error="failed to connect to peer: b6646186-fca6-731d-2559-df9be5188ff7"
2026-02-28T09:03:49.748Z [ERROR] server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: failed to appendEntries to: peer="{Voter server-b6646186-fca6-731d-2559-df9be5188ff7 b6646186-fca6-731d-2559-df9be5188ff7}" error="failed to connect to peer: b6646186-fca6-731d-2559-df9be5188ff7"
2026-02-28T09:03:49.758Z [ERROR] server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: failed to heartbeat to: peer=b6646186-fca6-731d-2559-df9be5188ff7 backoff time=25ms error="failed to connect to peer: b6646186-fca6-731d-2559-df9be5188ff7"
2026-02-28T09:03:49.763Z [WARN]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: failed to contact: server-id=server-b6646186-fca6-731d-2559-df9be5188ff7 time=97.25584ms
2026-02-28T09:03:49.769Z [WARN]  server-b6646186-fca6-731d-2559-df9be5188ff7: heartbeat timeout reached, starting election: last-leader-addr=db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 last-leader-id=server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6
2026-02-28T09:03:49.769Z [INFO]  server-b6646186-fca6-731d-2559-df9be5188ff7: entering candidate state: node="Node at b6646186-fca6-731d-2559-df9be5188ff7 [Candidate]" term=3
2026-02-28T09:03:49.769Z [ERROR] server-b6646186-fca6-731d-2559-df9be5188ff7: failed to make requestVote RPC: target="{Voter server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01 bdef8c89-8b73-26cf-9c07-b5b0a0109d01}" error="failed to connect to peer: bdef8c89-8b73-26cf-9c07-b5b0a0109d01" term=3
2026-02-28T09:03:49.769Z [ERROR] server-b6646186-fca6-731d-2559-df9be5188ff7: failed to make requestVote RPC: target="{Voter server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 db34e07e-ff2f-aa90-9a03-dcdfab1a29d6}" error="failed to connect to peer: db34e07e-ff2f-aa90-9a03-dcdfab1a29d6" term=3
2026-02-28T09:03:49.774Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: updating configuration: command=RemoveServer server-id=server-b6646186-fca6-731d-2559-df9be5188ff7 server-addr= servers="[{Suffrage:Voter ID:server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 Address:db34e07e-ff2f-aa90-9a03-dcdfab1a29d6} {Suffrage:Voter ID:server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01 Address:bdef8c89-8b73-26cf-9c07-b5b0a0109d01}]"
2026-02-28T09:03:49.774Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: removed peer, stopping replication: peer=server-b6646186-fca6-731d-2559-df9be5188ff7 last-index=3
2026-02-28T09:03:49.774Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: updating configuration: command=AddNonvoter server-id=server-b6646186-fca6-731d-2559-df9be5188ff7 server-addr=b6646186-fca6-731d-2559-df9be5188ff7 servers="[{Suffrage:Voter ID:server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 Address:db34e07e-ff2f-aa90-9a03-dcdfab1a29d6} {Suffrage:Voter ID:server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01 Address:bdef8c89-8b73-26cf-9c07-b5b0a0109d01} {Suffrage:Nonvoter ID:server-b6646186-fca6-731d-2559-df9be5188ff7 Address:b6646186-fca6-731d-2559-df9be5188ff7}]"
2026-02-28T09:03:49.774Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: added peer, starting replication: peer=server-b6646186-fca6-731d-2559-df9be5188ff7
2026-02-28T09:03:49.775Z [ERROR] server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: failed to appendEntries to: peer="{Nonvoter server-b6646186-fca6-731d-2559-df9be5188ff7 b6646186-fca6-731d-2559-df9be5188ff7}" error="failed to connect to peer: b6646186-fca6-731d-2559-df9be5188ff7"
2026-02-28T09:03:49.795Z [ERROR] server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: peer has newer term, stopping replication: peer="{Nonvoter server-b6646186-fca6-731d-2559-df9be5188ff7 b6646186-fca6-731d-2559-df9be5188ff7}"
2026-02-28T09:03:49.795Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: entering follower state: follower="Node at db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 [Follower]" leader-address= leader-id=
2026-02-28T09:03:49.795Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: aborting pipeline replication: peer="{Voter server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01 bdef8c89-8b73-26cf-9c07-b5b0a0109d01}"
2026-02-28T09:03:49.795Z [ERROR] server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: peer has newer term, stopping replication: peer="{Voter server-b6646186-fca6-731d-2559-df9be5188ff7 b6646186-fca6-731d-2559-df9be5188ff7}"
2026-02-28T09:03:49.859Z [WARN]  server-b6646186-fca6-731d-2559-df9be5188ff7: Election timeout reached, restarting election
2026-02-28T09:03:49.859Z [INFO]  server-b6646186-fca6-731d-2559-df9be5188ff7: entering candidate state: node="Node at b6646186-fca6-731d-2559-df9be5188ff7 [Candidate]" term=23
2026-02-28T09:03:49.859Z [WARN]  server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01: rejecting vote request since we have a leader: from=b6646186-fca6-731d-2559-df9be5188ff7 leader=db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 leader-id=server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6
2026-02-28T09:03:49.859Z [WARN]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: rejecting vote request since node is not a voter: from=b6646186-fca6-731d-2559-df9be5188ff7
2026-02-28T09:03:49.862Z [WARN]  server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01: heartbeat timeout reached, starting election: last-leader-addr=db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 last-leader-id=server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6
2026-02-28T09:03:49.862Z [INFO]  server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01: entering candidate state: node="Node at bdef8c89-8b73-26cf-9c07-b5b0a0109d01 [Candidate]" term=3
2026-02-28T09:03:49.862Z [INFO]  server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01: entering follower state: follower="Node at bdef8c89-8b73-26cf-9c07-b5b0a0109d01 [Follower]" leader-address= leader-id=
2026-02-28T09:03:49.880Z [WARN]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:49.880Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: entering candidate state: node="Node at db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 [Candidate]" term=24
2026-02-28T09:03:49.880Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:49.881Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: election won: term=24 tally=2
2026-02-28T09:03:49.881Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: entering leader state: leader="Node at db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 [Leader]"
2026-02-28T09:03:49.881Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: added peer, starting replication: peer=server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01
2026-02-28T09:03:49.881Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: added peer, starting replication: peer=server-b6646186-fca6-731d-2559-df9be5188ff7
2026-02-28T09:03:49.881Z [WARN]  server-b6646186-fca6-731d-2559-df9be5188ff7: failed to get previous log: previous-index=4 last-index=2 error="log not found"
2026-02-28T09:03:49.881Z [INFO]  server-b6646186-fca6-731d-2559-df9be5188ff7: entering follower state: follower="Node at b6646186-fca6-731d-2559-df9be5188ff7 [Follower]" leader-address=db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 leader-id=server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6
2026-02-28T09:03:49.881Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: pipelining replication: peer="{Voter server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01 bdef8c89-8b73-26cf-9c07-b5b0a0109d01}"
2026-02-28T09:03:49.881Z [WARN]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: appendEntries rejected, sending older logs: peer="{Nonvoter server-b6646186-fca6-731d-2559-df9be5188ff7 b6646186-fca6-731d-2559-df9be5188ff7}" next=3
2026-02-28T09:03:49.881Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: pipelining replication: peer="{Nonvoter server-b6646186-fca6-731d-2559-df9be5188ff7 b6646186-fca6-731d-2559-df9be5188ff7}"
2026-02-28T09:03:49.987Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:49.881272302 +0000 UTC m=+107.845068841 (1 nodes), 192.64µs from start of poll, 495.98324ms from cluster start. Timeout at 2026-02-28 09:03:49.986294782 +0000 UTC m=+107.950091321, 105.02248ms after stability
2026-02-28T09:03:50.016Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: updating configuration: command=RemoveServer server-id=server-b6646186-fca6-731d-2559-df9be5188ff7 server-addr= servers="[{Suffrage:Voter ID:server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 Address:db34e07e-ff2f-aa90-9a03-dcdfab1a29d6} {Suffrage:Voter ID:server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01 Address:bdef8c89-8b73-26cf-9c07-b5b0a0109d01}]"
2026-02-28T09:03:50.016Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: removed peer, stopping replication: peer=server-b6646186-fca6-731d-2559-df9be5188ff7 last-index=106
2026-02-28T09:03:50.016Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: aborting pipeline replication: peer="{Nonvoter server-b6646186-fca6-731d-2559-df9be5188ff7 b6646186-fca6-731d-2559-df9be5188ff7}"
2026-02-28T09:03:50.016Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: updating configuration: command=AddVoter server-id=server-b6646186-fca6-731d-2559-df9be5188ff7 server-addr=b6646186-fca6-731d-2559-df9be5188ff7 servers="[{Suffrage:Voter ID:server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6 Address:db34e07e-ff2f-aa90-9a03-dcdfab1a29d6} {Suffrage:Voter ID:server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01 Address:bdef8c89-8b73-26cf-9c07-b5b0a0109d01} {Suffrage:Voter ID:server-b6646186-fca6-731d-2559-df9be5188ff7 Address:b6646186-fca6-731d-2559-df9be5188ff7}]"
2026-02-28T09:03:50.016Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: added peer, starting replication: peer=server-b6646186-fca6-731d-2559-df9be5188ff7
2026-02-28T09:03:50.016Z [WARN]  server-b6646186-fca6-731d-2559-df9be5188ff7: failed to get previous log: previous-index=107 last-index=106 error="log not found"
2026-02-28T09:03:50.016Z [WARN]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: appendEntries rejected, sending older logs: peer="{Voter server-b6646186-fca6-731d-2559-df9be5188ff7 b6646186-fca6-731d-2559-df9be5188ff7}" next=107
2026-02-28T09:03:50.016Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: pipelining replication: peer="{Voter server-b6646186-fca6-731d-2559-df9be5188ff7 b6646186-fca6-731d-2559-df9be5188ff7}"
2026-02-28T09:03:51.097Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:51.203Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:51.097545502 +0000 UTC m=+109.061342041 (1 nodes), 2.44µs from start of poll, 1.71225644s from cluster start. Timeout at 2026-02-28 09:03:51.202581662 +0000 UTC m=+109.166378281, 105.03624ms after stability
2026-02-28T09:03:51.203Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: aborting pipeline replication: peer="{Voter server-b6646186-fca6-731d-2559-df9be5188ff7 b6646186-fca6-731d-2559-df9be5188ff7}"
2026-02-28T09:03:51.203Z [INFO]  server-db34e07e-ff2f-aa90-9a03-dcdfab1a29d6: aborting pipeline replication: peer="{Voter server-bdef8c89-8b73-26cf-9c07-b5b0a0109d01 bdef8c89-8b73-26cf-9c07-b5b0a0109d01}"
--- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.82s)
=== RUN   TestRaft_FollowerRemovalNoElection
2026-02-28T09:03:51.204Z [INFO]  server-b2b43737-cca2-1753-2d91-c51f9f450c96: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2b43737-cca2-1753-2d91-c51f9f450c96 Address:b2b43737-cca2-1753-2d91-c51f9f450c96} {Suffrage:Voter ID:server-b492e48e-75fa-2883-6450-60394b565ee5 Address:b492e48e-75fa-2883-6450-60394b565ee5} {Suffrage:Voter ID:server-79a36a8c-29a3-7841-f2dd-4ea40cf4f65b Address:79a36a8c-29a3-7841-f2dd-4ea40cf4f65b}]"
2026-02-28T09:03:51.204Z [INFO]  server-b492e48e-75fa-2883-6450-60394b565ee5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2b43737-cca2-1753-2d91-c51f9f450c96 Address:b2b43737-cca2-1753-2d91-c51f9f450c96} {Suffrage:Voter ID:server-b492e48e-75fa-2883-6450-60394b565ee5 Address:b492e48e-75fa-2883-6450-60394b565ee5} {Suffrage:Voter ID:server-79a36a8c-29a3-7841-f2dd-4ea40cf4f65b Address:79a36a8c-29a3-7841-f2dd-4ea40cf4f65b}]"
2026-02-28T09:03:51.204Z [INFO]  server-b2b43737-cca2-1753-2d91-c51f9f450c96: entering follower state: follower="Node at b2b43737-cca2-1753-2d91-c51f9f450c96 [Follower]" leader-address= leader-id=
2026-02-28T09:03:51.204Z [INFO]  server-79a36a8c-29a3-7841-f2dd-4ea40cf4f65b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2b43737-cca2-1753-2d91-c51f9f450c96 Address:b2b43737-cca2-1753-2d91-c51f9f450c96} {Suffrage:Voter ID:server-b492e48e-75fa-2883-6450-60394b565ee5 Address:b492e48e-75fa-2883-6450-60394b565ee5} {Suffrage:Voter ID:server-79a36a8c-29a3-7841-f2dd-4ea40cf4f65b Address:79a36a8c-29a3-7841-f2dd-4ea40cf4f65b}]"
2026-02-28T09:03:51.204Z [INFO]  server-b492e48e-75fa-2883-6450-60394b565ee5: entering follower state: follower="Node at b492e48e-75fa-2883-6450-60394b565ee5 [Follower]" leader-address= leader-id=
2026-02-28T09:03:51.204Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:51.204Z [INFO]  server-79a36a8c-29a3-7841-f2dd-4ea40cf4f65b: entering follower state: follower="Node at 79a36a8c-29a3-7841-f2dd-4ea40cf4f65b [Follower]" leader-address= leader-id=
2026-02-28T09:03:51.331Z [WARN]  server-b492e48e-75fa-2883-6450-60394b565ee5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:51.331Z [INFO]  server-b492e48e-75fa-2883-6450-60394b565ee5: entering candidate state: node="Node at b492e48e-75fa-2883-6450-60394b565ee5 [Candidate]" term=2
2026-02-28T09:03:51.331Z [INFO]  server-b492e48e-75fa-2883-6450-60394b565ee5: election won: term=2 tally=2
2026-02-28T09:03:51.331Z [INFO]  server-b492e48e-75fa-2883-6450-60394b565ee5: entering leader state: leader="Node at b492e48e-75fa-2883-6450-60394b565ee5 [Leader]"
2026-02-28T09:03:51.331Z [INFO]  server-b492e48e-75fa-2883-6450-60394b565ee5: added peer, starting replication: peer=server-b2b43737-cca2-1753-2d91-c51f9f450c96
2026-02-28T09:03:51.331Z [INFO]  server-b492e48e-75fa-2883-6450-60394b565ee5: added peer, starting replication: peer=server-79a36a8c-29a3-7841-f2dd-4ea40cf4f65b
2026-02-28T09:03:51.331Z [INFO]  server-b492e48e-75fa-2883-6450-60394b565ee5: pipelining replication: peer="{Voter server-79a36a8c-29a3-7841-f2dd-4ea40cf4f65b 79a36a8c-29a3-7841-f2dd-4ea40cf4f65b}"
2026-02-28T09:03:51.331Z [INFO]  server-b492e48e-75fa-2883-6450-60394b565ee5: pipelining replication: peer="{Voter server-b2b43737-cca2-1753-2d91-c51f9f450c96 b2b43737-cca2-1753-2d91-c51f9f450c96}"
2026-02-28T09:03:51.539Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:51.331486502 +0000 UTC m=+109.295283041 (1 nodes), 126.74248ms from start of poll, 127.15804ms from cluster start. Timeout at 2026-02-28 09:03:51.536500622 +0000 UTC m=+109.500297161, 205.01412ms after stability
2026-02-28T09:03:51.539Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:51.745Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:51.539897942 +0000 UTC m=+109.503694481 (1 nodes), 1.12µs from start of poll, 335.56948ms from cluster start. Timeout at 2026-02-28 09:03:51.744922582 +0000 UTC m=+109.708719121, 205.02464ms after stability
2026-02-28T09:03:51.750Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:51.956Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:51.751013662 +0000 UTC m=+109.714810201 (2 nodes), 3.8µs from start of poll, 546.6852ms from cluster start. Timeout at 2026-02-28 09:03:51.956038422 +0000 UTC m=+109.919834961, 205.02476ms after stability
2026-02-28T09:03:51.957Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:52.162Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:51.957059302 +0000 UTC m=+109.920855841 (1 nodes), 3.36µs from start of poll, 752.73084ms from cluster start. Timeout at 2026-02-28 09:03:52.162082822 +0000 UTC m=+110.125879361, 205.02352ms after stability
    raft_test.go:2961: [INFO] restarting Node at b2b43737-cca2-1753-2d91-c51f9f450c96 [Follower]
2026-02-28T09:03:52.163Z [INFO]  server-b2b43737-cca2-1753-2d91-c51f9f450c96: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2b43737-cca2-1753-2d91-c51f9f450c96 Address:b2b43737-cca2-1753-2d91-c51f9f450c96} {Suffrage:Voter ID:server-b492e48e-75fa-2883-6450-60394b565ee5 Address:b492e48e-75fa-2883-6450-60394b565ee5} {Suffrage:Voter ID:server-79a36a8c-29a3-7841-f2dd-4ea40cf4f65b Address:79a36a8c-29a3-7841-f2dd-4ea40cf4f65b}]"
2026-02-28T09:03:52.163Z [INFO]  server-b2b43737-cca2-1753-2d91-c51f9f450c96: entering follower state: follower="Node at b2b43737-cca2-1753-2d91-c51f9f450c96 [Follower]" leader-address= leader-id=
2026-02-28T09:03:52.413Z [INFO]  server-b492e48e-75fa-2883-6450-60394b565ee5: aborting pipeline replication: peer="{Voter server-79a36a8c-29a3-7841-f2dd-4ea40cf4f65b 79a36a8c-29a3-7841-f2dd-4ea40cf4f65b}"
2026-02-28T09:03:52.801Z [ERROR] server-b492e48e-75fa-2883-6450-60394b565ee5: failed to pipeline appendEntries: peer="{Voter server-b2b43737-cca2-1753-2d91-c51f9f450c96 b2b43737-cca2-1753-2d91-c51f9f450c96}" error="command enqueue timeout"
2026-02-28T09:03:52.801Z [INFO]  server-b492e48e-75fa-2883-6450-60394b565ee5: aborting pipeline replication: peer="{Voter server-b2b43737-cca2-1753-2d91-c51f9f450c96 b2b43737-cca2-1753-2d91-c51f9f450c96}"
2026-02-28T09:03:52.931Z [ERROR] server-b492e48e-75fa-2883-6450-60394b565ee5: failed to heartbeat to: peer=b2b43737-cca2-1753-2d91-c51f9f450c96 backoff time=10ms error="command timed out"
--- PASS: TestRaft_FollowerRemovalNoElection (1.73s)
=== RUN   TestRaft_VoteWithNoIDNoAddr
2026-02-28T09:03:52.932Z [INFO]  server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 Address:822bbf87-c2d1-4f77-e00e-0dd1f2fa5283} {Suffrage:Voter ID:server-3643dfce-7e00-a078-b9a3-37a4cefef690 Address:3643dfce-7e00-a078-b9a3-37a4cefef690} {Suffrage:Voter ID:server-b75fc583-30e6-c9d9-6280-76637187a3a0 Address:b75fc583-30e6-c9d9-6280-76637187a3a0}]"
2026-02-28T09:03:52.932Z [INFO]  server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283: entering follower state: follower="Node at 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 [Follower]" leader-address= leader-id=
2026-02-28T09:03:52.932Z [INFO]  server-3643dfce-7e00-a078-b9a3-37a4cefef690: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 Address:822bbf87-c2d1-4f77-e00e-0dd1f2fa5283} {Suffrage:Voter ID:server-3643dfce-7e00-a078-b9a3-37a4cefef690 Address:3643dfce-7e00-a078-b9a3-37a4cefef690} {Suffrage:Voter ID:server-b75fc583-30e6-c9d9-6280-76637187a3a0 Address:b75fc583-30e6-c9d9-6280-76637187a3a0}]"
2026-02-28T09:03:52.932Z [INFO]  server-3643dfce-7e00-a078-b9a3-37a4cefef690: entering follower state: follower="Node at 3643dfce-7e00-a078-b9a3-37a4cefef690 [Follower]" leader-address= leader-id=
2026-02-28T09:03:52.932Z [INFO]  server-b75fc583-30e6-c9d9-6280-76637187a3a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 Address:822bbf87-c2d1-4f77-e00e-0dd1f2fa5283} {Suffrage:Voter ID:server-3643dfce-7e00-a078-b9a3-37a4cefef690 Address:3643dfce-7e00-a078-b9a3-37a4cefef690} {Suffrage:Voter ID:server-b75fc583-30e6-c9d9-6280-76637187a3a0 Address:b75fc583-30e6-c9d9-6280-76637187a3a0}]"
2026-02-28T09:03:52.932Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:52.932Z [INFO]  server-b75fc583-30e6-c9d9-6280-76637187a3a0: entering follower state: follower="Node at b75fc583-30e6-c9d9-6280-76637187a3a0 [Follower]" leader-address= leader-id=
2026-02-28T09:03:52.983Z [WARN]  server-b75fc583-30e6-c9d9-6280-76637187a3a0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-02-28T09:03:52.983Z [INFO]  server-b75fc583-30e6-c9d9-6280-76637187a3a0: entering candidate state: node="Node at b75fc583-30e6-c9d9-6280-76637187a3a0 [Candidate]" term=2
2026-02-28T09:03:52.983Z [INFO]  server-b75fc583-30e6-c9d9-6280-76637187a3a0: election won: term=2 tally=2
2026-02-28T09:03:52.983Z [INFO]  server-b75fc583-30e6-c9d9-6280-76637187a3a0: entering leader state: leader="Node at b75fc583-30e6-c9d9-6280-76637187a3a0 [Leader]"
2026-02-28T09:03:52.983Z [INFO]  server-b75fc583-30e6-c9d9-6280-76637187a3a0: added peer, starting replication: peer=server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283
2026-02-28T09:03:52.983Z [INFO]  server-b75fc583-30e6-c9d9-6280-76637187a3a0: added peer, starting replication: peer=server-3643dfce-7e00-a078-b9a3-37a4cefef690
2026-02-28T09:03:52.983Z [INFO]  server-b75fc583-30e6-c9d9-6280-76637187a3a0: pipelining replication: peer="{Voter server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283}"
2026-02-28T09:03:52.983Z [INFO]  server-b75fc583-30e6-c9d9-6280-76637187a3a0: pipelining replication: peer="{Voter server-3643dfce-7e00-a078-b9a3-37a4cefef690 3643dfce-7e00-a078-b9a3-37a4cefef690}"
2026-02-28T09:03:53.089Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:52.983763702 +0000 UTC m=+110.947560241 (1 nodes), 50.97424ms from start of poll, 51.54596ms from cluster start. Timeout at 2026-02-28 09:03:53.088770942 +0000 UTC m=+111.052567481, 105.00724ms after stability
2026-02-28T09:03:53.089Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-02-28T09:03:53.194Z [INFO]  cluster: stable state for Leader reached at 2026-02-28 09:03:53.089486582 +0000 UTC m=+111.053283121 (1 nodes), 1.44µs from start of poll, 157.26884ms from cluster start. Timeout at 2026-02-28 09:03:53.194512342 +0000 UTC m=+111.158308841, 105.02572ms after stability
2026-02-28T09:03:53.200Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-02-28T09:03:53.306Z [INFO]  cluster: stable state for Follower reached at 2026-02-28 09:03:53.200364102 +0000 UTC m=+111.164160641 (2 nodes), 9.56µs from start of poll, 268.14636ms from cluster start. Timeout at 2026-02-28 09:03:53.305389942 +0000 UTC m=+111.269186441, 105.0258ms after stability
2026-02-28T09:03:53.307Z [ERROR] server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to heartbeat to: peer=822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 backoff time=10ms error="failed to connect to peer: 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283"
2026-02-28T09:03:53.309Z [ERROR] server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to pipeline appendEntries: peer="{Voter server-3643dfce-7e00-a078-b9a3-37a4cefef690 3643dfce-7e00-a078-b9a3-37a4cefef690}" error="append pipeline closed"
2026-02-28T09:03:53.309Z [INFO]  server-b75fc583-30e6-c9d9-6280-76637187a3a0: aborting pipeline replication: peer="{Voter server-3643dfce-7e00-a078-b9a3-37a4cefef690 3643dfce-7e00-a078-b9a3-37a4cefef690}"
2026-02-28T09:03:53.309Z [ERROR] server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to pipeline appendEntries: peer="{Voter server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283}" error="append pipeline closed"
2026-02-28T09:03:53.309Z [INFO]  server-b75fc583-30e6-c9d9-6280-76637187a3a0: aborting pipeline replication: peer="{Voter server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283}"
2026-02-28T09:03:53.309Z [ERROR] server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to heartbeat to: peer=3643dfce-7e00-a078-b9a3-37a4cefef690 backoff time=10ms error="failed to connect to peer: 3643dfce-7e00-a078-b9a3-37a4cefef690"
2026-02-28T09:03:53.316Z [ERROR] server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to appendEntries to: peer="{Voter server-3643dfce-7e00-a078-b9a3-37a4cefef690 3643dfce-7e00-a078-b9a3-37a4cefef690}" error="failed to connect to peer: 3643dfce-7e00-a078-b9a3-37a4cefef690"
2026-02-28T09:03:53.316Z [ERROR] server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to appendEntries to: peer="{Voter server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283}" error="failed to connect to peer: 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283"
2026-02-28T09:03:53.324Z [ERROR] server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to heartbeat to: peer=822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 backoff time=10ms error="failed to connect to peer: 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283"
2026-02-28T09:03:53.327Z [ERROR] server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to heartbeat to: peer=3643dfce-7e00-a078-b9a3-37a4cefef690 backoff time=10ms error="failed to connect to peer: 3643dfce-7e00-a078-b9a3-37a4cefef690"
2026-02-28T09:03:53.334Z [ERROR] server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to appendEntries to: peer="{Voter server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283}" error="failed to connect to peer: 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283"
2026-02-28T09:03:53.335Z [ERROR] server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to appendEntries to: peer="{Voter server-3643dfce-7e00-a078-b9a3-37a4cefef690 3643dfce-7e00-a078-b9a3-37a4cefef690}" error="failed to connect to peer: 3643dfce-7e00-a078-b9a3-37a4cefef690"
2026-02-28T09:03:53.342Z [ERROR] server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to heartbeat to: peer=822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 backoff time=10ms error="failed to connect to peer: 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283"
2026-02-28T09:03:53.344Z [ERROR] server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to heartbeat to: peer=3643dfce-7e00-a078-b9a3-37a4cefef690 backoff time=10ms error="failed to connect to peer: 3643dfce-7e00-a078-b9a3-37a4cefef690"
2026-02-28T09:03:53.350Z [ERROR] server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to appendEntries to: peer="{Voter server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283}" error="failed to connect to peer: 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283"
2026-02-28T09:03:53.351Z [WARN]  server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to contact: server-id=server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 time=50.04872ms
2026-02-28T09:03:53.354Z [ERROR] server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to appendEntries to: peer="{Voter server-3643dfce-7e00-a078-b9a3-37a4cefef690 3643dfce-7e00-a078-b9a3-37a4cefef690}" error="failed to connect to peer: 3643dfce-7e00-a078-b9a3-37a4cefef690"
2026-02-28T09:03:53.360Z [ERROR] server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to heartbeat to: peer=822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 backoff time=20ms error="failed to connect to peer: 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283"
2026-02-28T09:03:53.360Z [WARN]  server-3643dfce-7e00-a078-b9a3-37a4cefef690: heartbeat timeout reached, starting election: last-leader-addr=b75fc583-30e6-c9d9-6280-76637187a3a0 last-leader-id=server-b75fc583-30e6-c9d9-6280-76637187a3a0
2026-02-28T09:03:53.360Z [INFO]  server-3643dfce-7e00-a078-b9a3-37a4cefef690: entering candidate state: node="Node at 3643dfce-7e00-a078-b9a3-37a4cefef690 [Candidate]" term=3
2026-02-28T09:03:53.360Z [ERROR] server-3643dfce-7e00-a078-b9a3-37a4cefef690: failed to make requestVote RPC: target="{Voter server-b75fc583-30e6-c9d9-6280-76637187a3a0 b75fc583-30e6-c9d9-6280-76637187a3a0}" error="failed to connect to peer: b75fc583-30e6-c9d9-6280-76637187a3a0" term=3
2026-02-28T09:03:53.360Z [WARN]  server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283: rejecting vote request since we have a leader: from=3643dfce-7e00-a078-b9a3-37a4cefef690 leader=b75fc583-30e6-c9d9-6280-76637187a3a0 leader-id=server-b75fc583-30e6-c9d9-6280-76637187a3a0
2026-02-28T09:03:53.361Z [ERROR] server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to heartbeat to: peer=3643dfce-7e00-a078-b9a3-37a4cefef690 backoff time=20ms error="failed to connect to peer: 3643dfce-7e00-a078-b9a3-37a4cefef690"
2026-02-28T09:03:53.361Z [WARN]  server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to contact: server-id=server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 time=60.55088ms
2026-02-28T09:03:53.361Z [WARN]  server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to contact: server-id=server-3643dfce-7e00-a078-b9a3-37a4cefef690 time=57.92544ms
2026-02-28T09:03:53.361Z [WARN]  server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to contact quorum of nodes, stepping down
2026-02-28T09:03:53.361Z [INFO]  server-b75fc583-30e6-c9d9-6280-76637187a3a0: entering follower state: follower="Node at b75fc583-30e6-c9d9-6280-76637187a3a0 [Follower]" leader-address= leader-id=
2026-02-28T09:03:53.379Z [ERROR] server-b75fc583-30e6-c9d9-6280-76637187a3a0: failed to appendEntries to: peer="{Voter server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283}" error="failed to connect to peer: 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283"
2026-02-28T09:03:53.382Z [WARN]  server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283: heartbeat timeout reached, starting election: last-leader-addr=b75fc583-30e6-c9d9-6280-76637187a3a0 last-leader-id=server-b75fc583-30e6-c9d9-6280-76637187a3a0
2026-02-28T09:03:53.382Z [INFO]  server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283: entering candidate state: node="Node at 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 [Candidate]" term=3
2026-02-28T09:03:53.382Z [ERROR] server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283: failed to make requestVote RPC: target="{Voter server-b75fc583-30e6-c9d9-6280-76637187a3a0 b75fc583-30e6-c9d9-6280-76637187a3a0}" error="failed to connect to peer: b75fc583-30e6-c9d9-6280-76637187a3a0" term=3
2026-02-28T09:03:53.382Z [INFO]  server-3643dfce-7e00-a078-b9a3-37a4cefef690: duplicate requestVote for same term: term=3
2026-02-28T09:03:53.411Z [INFO]  server-822bbf87-c2d1-4f77-e00e-0dd1f2fa5283: entering follower state: follower="Node at 822bbf87-c2d1-4f77-e00e-0dd1f2fa5283 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VoteWithNoIDNoAddr (0.48s)
=== RUN   TestRaft_runFollower_State_Transition
=== RUN   TestRaft_runFollower_State_Transition/NonVoter
2026-02-28T09:03:53.412Z [INFO]  127.0.0.1:45129 :: starting node: addr=127.0.0.1:45129
2026-02-28T09:03:53.412Z [INFO]  127.0.0.1:45129 :: initial configuration: index=0 servers=[]
2026-02-28T09:03:53.412Z [INFO]  127.0.0.1:45129 :: entering follower state: follower="Node at 127.0.0.1:45129 [Follower]" leader-address= leader-id=
2026-02-28T09:03:53.471Z [WARN]  127.0.0.1:45129 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
=== RUN   TestRaft_runFollower_State_Transition/Voter
2026-02-28T09:03:53.562Z [INFO]  127.0.0.1:41819 :: starting node: addr=127.0.0.1:41819
2026-02-28T09:03:53.563Z [INFO]  127.0.0.1:41819 :: initial configuration: index=0 servers=[]
2026-02-28T09:03:53.563Z [INFO]  127.0.0.1:41819 :: entering follower state: follower="Node at 127.0.0.1:41819 [Follower]" leader-address= leader-id=
2026-02-28T09:03:53.643Z [WARN]  127.0.0.1:41819 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
=== RUN   TestRaft_runFollower_State_Transition/Not_in_Config
2026-02-28T09:03:53.714Z [INFO]  127.0.0.1:35463 :: starting node: addr=127.0.0.1:35463
2026-02-28T09:03:53.714Z [INFO]  127.0.0.1:35463 :: initial configuration: index=0 servers=[]
2026-02-28T09:03:53.714Z [INFO]  127.0.0.1:35463 :: entering follower state: follower="Node at 127.0.0.1:35463 [Follower]" leader-address= leader-id=
2026-02-28T09:03:53.794Z [WARN]  127.0.0.1:35463 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
--- PASS: TestRaft_runFollower_State_Transition (0.45s)
    --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s)
=== RUN   TestRaft_runFollower_ReloadTimeoutConfigs
2026-02-28T09:03:53.865Z [INFO]  127.0.0.1:43059 :: starting node: addr=127.0.0.1:43059
2026-02-28T09:03:53.865Z [INFO]  127.0.0.1:43059 :: initial configuration: index=0 servers=[]
2026-02-28T09:03:53.865Z [INFO]  127.0.0.1:43059 :: entering follower state: follower="Node at 127.0.0.1:43059 [Follower]" leader-address= leader-id=
2026-02-28T09:03:53.865Z [WARN]  127.0.0.1:43059 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
--- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s)
=== RUN   TestSaturationMetric
=== RUN   TestSaturationMetric/without_smoothing
--- PASS: TestSaturationMetric (0.00s)
    --- PASS: TestSaturationMetric/without_smoothing (0.00s)
=== RUN   TestSaturationMetric_IncorrectUsage
=== RUN   TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_first
--- PASS: TestSaturationMetric_IncorrectUsage (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s)
=== RUN   TestTCPTransport_BadAddr
--- PASS: TestTCPTransport_BadAddr (0.00s)
=== RUN   TestTCPTransport_EmptyAddr
--- PASS: TestTCPTransport_EmptyAddr (0.00s)
=== RUN   TestTCPTransport_WithAdvertise
--- PASS: TestTCPTransport_WithAdvertise (0.00s)
=== RUN   TestTransport_StartStop
--- PASS: TestTransport_StartStop (0.00s)
=== RUN   TestTransport_AppendEntries
--- PASS: TestTransport_AppendEntries (0.00s)
=== RUN   TestTransport_AppendEntriesPipeline
--- PASS: TestTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestTransport_RequestVote
--- PASS: TestTransport_RequestVote (0.00s)
=== RUN   TestTransport_InstallSnapshot
--- PASS: TestTransport_InstallSnapshot (0.00s)
=== RUN   TestTransport_EncodeDecode
--- PASS: TestTransport_EncodeDecode (0.00s)
=== RUN   TestRandomTimeout
--- PASS: TestRandomTimeout (0.00s)
=== RUN   TestNewSeed
--- PASS: TestNewSeed (0.00s)
=== RUN   TestRandomTimeout_NoTime
--- PASS: TestRandomTimeout_NoTime (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
=== RUN   TestMax
--- PASS: TestMax (0.00s)
=== RUN   TestGenerateUUID
--- PASS: TestGenerateUUID (0.00s)
=== RUN   TestBackoff
--- PASS: TestBackoff (0.00s)
=== RUN   TestOverrideNotifyBool
--- PASS: TestOverrideNotifyBool (0.00s)
PASS
ok  	github.com/hashicorp/raft	111.999s
make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
   create-stamp debian/debhelper-build-stamp
   dh_testroot -O--buildsystem=golang
   dh_prep -O--buildsystem=golang
   dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang
   dh_installdocs -O--buildsystem=golang
   dh_installchangelogs -O--buildsystem=golang
   dh_installsystemduser -O--buildsystem=golang
   dh_perl -O--buildsystem=golang
   dh_link -O--buildsystem=golang
   dh_strip_nondeterminism -O--buildsystem=golang
   dh_compress -O--buildsystem=golang
   dh_fixperms -O--buildsystem=golang
   dh_missing -O--buildsystem=golang
   dh_installdeb -O--buildsystem=golang
   dh_golang -O--buildsystem=golang
   dh_gencontrol -O--buildsystem=golang
   dh_md5sums -O--buildsystem=golang
   dh_builddeb -O--buildsystem=golang
dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'.
 dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_arm64.buildinfo
 dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_arm64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-buildpackage: info: binary-only upload (no source included)
dpkg-genchanges: info: including full source code in upload
I: copying local configuration
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /srv/workspace/pbuilder/3472608 and its subdirectories
I: Current time: Fri Feb 27 21:04:00 -12 2026
I: pbuilder-time-stamp: 1772269440