Tue Aug 19 09:00:04 UTC 2025  I: starting to build golang-github-hashicorp-raft/forky/arm64 on jenkins on '2025-08-19 08:59'
Tue Aug 19 09:00:04 UTC 2025  I: The jenkins build log is/was available at https://jenkins.debian.net/userContent/reproducible/debian/build_service/arm64_5/115916/console.log
Tue Aug 19 09:00:04 UTC 2025  I: Downloading source for forky/golang-github-hashicorp-raft=1.5.0-1
--2025-08-19 09:00:04--  http://deb.debian.org/debian/pool/main/g/golang-github-hashicorp-raft/golang-github-hashicorp-raft_1.5.0-1.dsc
Connecting to 46.16.76.132:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2480 (2.4K) [text/prs.lines.tag]
Saving to: ‘golang-github-hashicorp-raft_1.5.0-1.dsc’

     0K ..                                                    100%  292M=0s

2025-08-19 09:00:04 (292 MB/s) - ‘golang-github-hashicorp-raft_1.5.0-1.dsc’ saved [2480/2480]

Tue Aug 19 09:00:04 UTC 2025  I: golang-github-hashicorp-raft_1.5.0-1.dsc
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 3.0 (quilt)
Source: golang-github-hashicorp-raft
Binary: golang-github-hashicorp-raft-dev
Architecture: all
Version: 1.5.0-1
Maintainer: Debian Go Packaging Team <team+pkg-go@tracker.debian.org>
Uploaders: Tim Potter <tpot@hpe.com>, Dmitry Smirnov <onlyjob@debian.org>,
Homepage: https://github.com/hashicorp/raft
Standards-Version: 4.6.2
Vcs-Browser: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft
Vcs-Git: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git
Testsuite: autopkgtest-pkg-go
Build-Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
Package-List:
 golang-github-hashicorp-raft-dev deb golang optional arch=all
Checksums-Sha1:
 cea1c93d584c6f8dfbf53aef3438ab18ec7e7222 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 015fac87182a135b38ce31d1eb99f68ca251ae23 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Checksums-Sha256:
 f4dc9aabc7784a1b9b5da36b85a3797ac37911f2a6dc54a6fd6cb60a6fd643e9 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 0d95ee03abc0b447cfb92e400d914ecee5113fd181a4ea1d51fa5b6c37288658 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Files:
 7ebc703c8f7e2d7fc02568216f2e3b0a 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 1f0ee65ab76bca0e05a75b8f9ad32d8d 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Go-Import-Path: github.com/hashicorp/raft

-----BEGIN PGP SIGNATURE-----

iQJKBAEBCgA0FiEEMD9oek78sa58GjWjtwAXP7uAWikFAmTCiiIWHGRzd2FyYnJp
Y2tAZGViaWFuLm9yZwAKCRC3ABc/u4BaKYS6D/wN81mTx6pt43be/9H7Ovy6z+5p
nv0+kgrOLtH5y4KUjNLZJuPljAp+uoELmotxp1wqkoLSYs1AwCZ11X4NVs8YoLXj
sLd+Ub+QcWb0Gejqjw7Nk9VZqqXjwfFJW54n8NC2Zy8D6OelCl47sDPmiu03v3uz
oRu4dp2DW/uKkC0vb4RczT81GBmJ19Tu2ZdDvtk7kwnvF48cqDhXMhUTQAEjoyVU
Q8kfY13blza+uXZk6xhWyYsc+wpTh6RHZIWsbpgejKmEYnZCoCdaR0UqOoau33Bs
T1OXVNM0P6vrmeXyoiEy66Q2XyHeei+HYRUgED16Ujg0qxPxSEVE5timSwcXHa/u
xb5ZcKQwqEoJISjWTFDG8gww8MBPOKNGW5BKRdQSdtDhNEsnmn58jl1lsRs5L6Xg
+DrufjU6KKqqYyOBS+H+s68KXleg2t/qP+K8GZjJl/eX3+Tw5xOdt3ZWDMWxB2qF
gS+ASmxYV0Uu/pd0MLWXjAFcllnl93MzPVkF14A8eYrghujLjN3+oUOD0RQzxpAb
LHc5b0iPnN4/YQtJX3fZi2y6c62BCZ5Slb5sB0bsgNPbk1T3qQrtMPWJXUfr32ys
VOow6eNA+VDxNxHZ3aSUiVC0lcUm6dJDYqcfk/PgDdBR/E7iigkTi3O/2rgW0Y7Z
14FQMVvbK31veGXJjA==
=okTI
-----END PGP SIGNATURE-----
Tue Aug 19 09:00:04 UTC 2025  I: Checking whether the package is not for us
Tue Aug 19 09:00:04 UTC 2025  I: Starting 1st build on remote node codethink03-arm64.debian.net.
Tue Aug 19 09:00:04 UTC 2025  I: Preparing to do remote build '1' on codethink03-arm64.debian.net.
Tue Aug 19 09:03:05 UTC 2025  I: Deleting $TMPDIR on codethink03-arm64.debian.net.
I: pbuilder: network access will be disabled during build
I: Current time: Mon Sep 21 03:23:06 -12 2026
I: pbuilder-time-stamp: 1790004186
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/forky-reproducible-base.tgz]
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: policy-rc.d already exists
I: Copying source file
I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc]
I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz]
I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz]
I: Extracting source
dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: no acceptable signature found
dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying 0001-Skip-unreliable-tests.patch
I: Not using root during the build.
I: Installing the build-deps
I: user script /srv/workspace/pbuilder/3202066/tmp/hooks/D02_print_environment starting
I: set
  BUILDDIR='/build/reproducible-path'
  BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other'
  BUILDUSERNAME='pbuilder1'
  BUILD_ARCH='arm64'
  DEBIAN_FRONTEND='noninteractive'
  DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=12 '
  DISTRIBUTION='forky'
  HOME='/root'
  HOST_ARCH='arm64'
  IFS=' 	
  '
  INVOCATION_ID='0cca7f5ff9a54da0ae60d869b0488af0'
  LANG='C'
  LANGUAGE='en_US:en'
  LC_ALL='C'
  MAIL='/var/mail/root'
  OPTIND='1'
  PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games'
  PBCURRENTCOMMANDLINEOPERATION='build'
  PBUILDER_OPERATION='build'
  PBUILDER_PKGDATADIR='/usr/share/pbuilder'
  PBUILDER_PKGLIBDIR='/usr/lib/pbuilder'
  PBUILDER_SYSCONFDIR='/etc'
  PPID='3202066'
  PS1='# '
  PS2='> '
  PS4='+ '
  PWD='/'
  SHELL='/bin/bash'
  SHLVL='2'
  SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.OHaJFdYG/pbuilderrc_QrNs --distribution forky --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/forky-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.OHaJFdYG/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc'
  SUDO_GID='109'
  SUDO_HOME='/var/lib/jenkins'
  SUDO_UID='104'
  SUDO_USER='jenkins'
  TERM='unknown'
  TZ='/usr/share/zoneinfo/Etc/GMT+12'
  USER='root'
  _='/usr/bin/systemd-run'
  http_proxy='http://192.168.101.4:3128'
I: uname -a
  Linux codethink03-arm64 6.12.41+deb13-cloud-arm64 #1 SMP Debian 6.12.41-1 (2025-08-12) aarch64 GNU/Linux
I: ls -l /bin
  lrwxrwxrwx 1 root root 7 Aug 10 12:30 /bin -> usr/bin
I: user script /srv/workspace/pbuilder/3202066/tmp/hooks/D02_print_environment finished
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: arm64
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 20009 files and directories currently installed.)
Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ...
Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however:
  Package debhelper-compat is not installed.
 pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however:
  Package dh-sequence-golang is not installed.
 pbuilder-satisfydepends-dummy depends on golang-any; however:
  Package golang-any is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however:
  Package golang-github-armon-go-metrics-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however:
  Package golang-github-hashicorp-go-hclog-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however:
  Package golang-github-hashicorp-go-msgpack-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however:
  Package golang-github-stretchr-testify-dev is not installed.

Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
Building tag database...
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
The following NEW packages will be installed:
  autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.24-go{a} golang-1.24-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-prometheus-sigv4-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libunistring5{a} libxml2-16{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} 
The following packages are RECOMMENDED but will NOT be installed:
  curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 
0 packages upgraded, 106 newly installed, 0 to remove and 0 not upgraded.
Need to get 119 MB of archives. After unpacking 680 MB will be used.
Writing extended state information...
Get: 1 http://deb.debian.org/debian forky/main arm64 sensible-utils all 0.0.25 [25.0 kB]
Get: 2 http://deb.debian.org/debian forky/main arm64 libmagic-mgc arm64 1:5.46-5 [338 kB]
Get: 3 http://deb.debian.org/debian forky/main arm64 libmagic1t64 arm64 1:5.46-5 [103 kB]
Get: 4 http://deb.debian.org/debian forky/main arm64 file arm64 1:5.46-5 [43.7 kB]
Get: 5 http://deb.debian.org/debian forky/main arm64 gettext-base arm64 0.23.1-2+b1 [241 kB]
Get: 6 http://deb.debian.org/debian forky/main arm64 libuchardet0 arm64 0.0.8-1+b2 [69.2 kB]
Get: 7 http://deb.debian.org/debian forky/main arm64 groff-base arm64 1.23.0-9 [1130 kB]
Get: 8 http://deb.debian.org/debian forky/main arm64 bsdextrautils arm64 2.41.1-1 [93.8 kB]
Get: 9 http://deb.debian.org/debian forky/main arm64 libpipeline1 arm64 1.5.8-1 [40.2 kB]
Get: 10 http://deb.debian.org/debian forky/main arm64 man-db arm64 2.13.1-1 [1453 kB]
Get: 11 http://deb.debian.org/debian forky/main arm64 m4 arm64 1.4.19-8 [285 kB]
Get: 12 http://deb.debian.org/debian forky/main arm64 autoconf all 2.72-3.1 [494 kB]
Get: 13 http://deb.debian.org/debian forky/main arm64 autotools-dev all 20240727.1 [60.2 kB]
Get: 14 http://deb.debian.org/debian forky/main arm64 automake all 1:1.17-4 [862 kB]
Get: 15 http://deb.debian.org/debian forky/main arm64 autopoint all 0.23.1-2 [770 kB]
Get: 16 http://deb.debian.org/debian forky/main arm64 libdebhelper-perl all 13.25 [91.6 kB]
Get: 17 http://deb.debian.org/debian forky/main arm64 libtool all 2.5.4-4 [539 kB]
Get: 18 http://deb.debian.org/debian forky/main arm64 dh-autoreconf all 20 [17.1 kB]
Get: 19 http://deb.debian.org/debian forky/main arm64 libarchive-zip-perl all 1.68-1 [104 kB]
Get: 20 http://deb.debian.org/debian forky/main arm64 libfile-stripnondeterminism-perl all 1.14.1-2 [19.7 kB]
Get: 21 http://deb.debian.org/debian forky/main arm64 dh-strip-nondeterminism all 1.14.1-2 [8620 B]
Get: 22 http://deb.debian.org/debian forky/main arm64 libelf1t64 arm64 0.192-4 [189 kB]
Get: 23 http://deb.debian.org/debian forky/main arm64 dwz arm64 0.15-1+b1 [102 kB]
Get: 24 http://deb.debian.org/debian forky/main arm64 libunistring5 arm64 1.3-2 [453 kB]
Get: 25 http://deb.debian.org/debian forky/main arm64 libxml2-16 arm64 2.14.5+dfsg-0.1 [583 kB]
Get: 26 http://deb.debian.org/debian forky/main arm64 gettext arm64 0.23.1-2+b1 [1612 kB]
Get: 27 http://deb.debian.org/debian forky/main arm64 intltool-debian all 0.35.0+20060710.6 [22.9 kB]
Get: 28 http://deb.debian.org/debian forky/main arm64 po-debconf all 1.0.21+nmu1 [248 kB]
Get: 29 http://deb.debian.org/debian forky/main arm64 debhelper all 13.25 [937 kB]
Get: 30 http://deb.debian.org/debian forky/main arm64 dh-golang all 1.63 [24.4 kB]
Get: 31 http://deb.debian.org/debian forky/main arm64 golang-1.24-src all 1.24.4-1 [21.2 MB]
Get: 32 http://deb.debian.org/debian forky/main arm64 golang-1.24-go arm64 1.24.4-1 [24.6 MB]
Get: 33 http://deb.debian.org/debian forky/main arm64 golang-src all 2:1.24~2 [5136 B]
Get: 34 http://deb.debian.org/debian forky/main arm64 golang-go arm64 2:1.24~2 [44.3 kB]
Get: 35 http://deb.debian.org/debian forky/main arm64 golang-any arm64 2:1.24~2 [5216 B]
Get: 36 http://deb.debian.org/debian forky/main arm64 golang-github-alecthomas-units-dev all 0.0~git20240927.0f3dac3-1 [8880 B]
Get: 37 http://deb.debian.org/debian forky/main arm64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB]
Get: 38 http://deb.debian.org/debian forky/main arm64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB]
Get: 39 http://deb.debian.org/debian forky/main arm64 golang-golang-x-sys-dev all 0.22.0-2 [622 kB]
Get: 40 http://deb.debian.org/debian forky/main arm64 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B]
Get: 41 http://deb.debian.org/debian forky/main arm64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB]
Get: 42 http://deb.debian.org/debian forky/main arm64 golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB]
Get: 43 http://deb.debian.org/debian forky/main arm64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB]
Get: 44 http://deb.debian.org/debian forky/main arm64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB]
Get: 45 http://deb.debian.org/debian forky/main arm64 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB]
Get: 46 http://deb.debian.org/debian forky/main arm64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB]
Get: 47 http://deb.debian.org/debian forky/main arm64 golang-github-stretchr-testify-dev all 1.10.0-1 [78.4 kB]
Get: 48 http://deb.debian.org/debian forky/main arm64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB]
Get: 49 http://deb.debian.org/debian forky/main arm64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB]
Get: 50 http://deb.debian.org/debian forky/main arm64 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB]
Get: 51 http://deb.debian.org/debian forky/main arm64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B]
Get: 52 http://deb.debian.org/debian forky/main arm64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB]
Get: 53 http://deb.debian.org/debian forky/main arm64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB]
Get: 54 http://deb.debian.org/debian forky/main arm64 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB]
Get: 55 http://deb.debian.org/debian forky/main arm64 golang-google-protobuf-dev all 1.36.5-1 [796 kB]
Get: 56 http://deb.debian.org/debian forky/main arm64 golang-github-golang-protobuf-1-5-dev all 1.5.4-2 [137 kB]
Get: 57 http://deb.debian.org/debian forky/main arm64 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B]
Get: 58 http://deb.debian.org/debian forky/main arm64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB]
Get: 59 http://deb.debian.org/debian forky/main arm64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB]
Get: 60 http://deb.debian.org/debian forky/main arm64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB]
Get: 61 http://deb.debian.org/debian forky/main arm64 golang-github-beorn7-perks-dev all 1.0.1-3 [12.2 kB]
Get: 62 http://deb.debian.org/debian forky/main arm64 golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB]
Get: 63 http://deb.debian.org/debian forky/main arm64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B]
Get: 64 http://deb.debian.org/debian forky/main arm64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB]
Get: 65 http://deb.debian.org/debian forky/main arm64 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB]
Get: 66 http://deb.debian.org/debian forky/main arm64 golang-github-klauspost-compress-dev all 1.18.0+ds1-1 [33.3 MB]
Get: 67 http://deb.debian.org/debian forky/main arm64 golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB]
Get: 68 http://deb.debian.org/debian forky/main arm64 golang-github-prometheus-client-model-dev all 0.6.2-1 [11.1 kB]
Get: 69 http://deb.debian.org/debian forky/main arm64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB]
Get: 70 http://deb.debian.org/debian forky/main arm64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB]
Get: 71 http://deb.debian.org/debian forky/main arm64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB]
Get: 72 http://deb.debian.org/debian forky/main arm64 golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B]
Get: 73 http://deb.debian.org/debian forky/main arm64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B]
Get: 74 http://deb.debian.org/debian forky/main arm64 golang-golang-x-term-dev all 0.22.0-1 [14.8 kB]
Get: 75 http://deb.debian.org/debian forky/main arm64 golang-golang-x-text-dev all 0.22.0-1 [3995 kB]
Get: 76 http://deb.debian.org/debian forky/main arm64 golang-golang-x-crypto-dev all 1:0.25.0-1 [1682 kB]
Get: 77 http://deb.debian.org/debian forky/main arm64 golang-golang-x-net-dev all 1:0.27.0-2 [904 kB]
Get: 78 http://deb.debian.org/debian forky/main arm64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-4 [14.6 kB]
Get: 79 http://deb.debian.org/debian forky/main arm64 golang-gopkg-yaml.v2-dev all 2.4.0-5 [60.5 kB]
Get: 80 http://deb.debian.org/debian forky/main arm64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB]
Get: 81 http://deb.debian.org/debian forky/main arm64 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB]
Get: 82 http://deb.debian.org/debian forky/main arm64 golang-github-prometheus-sigv4-dev all 0.1.2-2 [8484 B]
Get: 83 http://deb.debian.org/debian forky/main arm64 golang-golang-x-oauth2-dev all 0.27.0-1 [37.2 kB]
Get: 84 http://deb.debian.org/debian forky/main arm64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB]
Get: 85 http://deb.debian.org/debian forky/main arm64 golang-github-prometheus-common-dev all 0.63.0-2 [125 kB]
Get: 86 http://deb.debian.org/debian forky/main arm64 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB]
Get: 87 http://deb.debian.org/debian forky/main arm64 golang-github-prometheus-procfs-dev all 0.15.1-1 [184 kB]
Get: 88 http://deb.debian.org/debian forky/main arm64 golang-github-prometheus-client-golang-dev all 1.21.1-1 [194 kB]
Get: 89 http://deb.debian.org/debian forky/main arm64 golang-github-armon-go-metrics-dev all 0.4.1-2 [35.2 kB]
Get: 90 http://deb.debian.org/debian forky/main arm64 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB]
Get: 91 http://deb.debian.org/debian forky/main arm64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B]
Get: 92 http://deb.debian.org/debian forky/main arm64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB]
Get: 93 http://deb.debian.org/debian forky/main arm64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B]
Get: 94 http://deb.debian.org/debian forky/main arm64 libsasl2-modules-db arm64 2.1.28+dfsg1-9 [20.1 kB]
Get: 95 http://deb.debian.org/debian forky/main arm64 libsasl2-2 arm64 2.1.28+dfsg1-9 [55.6 kB]
Get: 96 http://deb.debian.org/debian forky/main arm64 libssl-dev arm64 3.5.1-1 [3385 kB]
Get: 97 http://deb.debian.org/debian forky/main arm64 libsasl2-dev arm64 2.1.28+dfsg1-9 [229 kB]
Get: 98 http://deb.debian.org/debian forky/main arm64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB]
Get: 99 http://deb.debian.org/debian forky/main arm64 golang-golang-x-mod-dev all 0.19.0-1 [104 kB]
Get: 100 http://deb.debian.org/debian forky/main arm64 golang-github-yuin-goldmark-dev all 1.7.4-1 [113 kB]
Get: 101 http://deb.debian.org/debian forky/main arm64 golang-golang-x-tools-dev all 1:0.25.0+ds-1 [1608 kB]
Get: 102 http://deb.debian.org/debian forky/main arm64 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB]
Get: 103 http://deb.debian.org/debian forky/main arm64 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB]
Get: 104 http://deb.debian.org/debian forky/main arm64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-2 [31.2 kB]
Get: 105 http://deb.debian.org/debian forky/main arm64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB]
Get: 106 http://deb.debian.org/debian forky/main arm64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB]
Fetched 119 MB in 1s (138 MB/s)
Preconfiguring packages ...
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 20009 files and directories currently installed.)
Preparing to unpack .../000-sensible-utils_0.0.25_all.deb ...
Unpacking sensible-utils (0.0.25) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../001-libmagic-mgc_1%3a5.46-5_arm64.deb ...
Unpacking libmagic-mgc (1:5.46-5) ...
Selecting previously unselected package libmagic1t64:arm64.
Preparing to unpack .../002-libmagic1t64_1%3a5.46-5_arm64.deb ...
Unpacking libmagic1t64:arm64 (1:5.46-5) ...
Selecting previously unselected package file.
Preparing to unpack .../003-file_1%3a5.46-5_arm64.deb ...
Unpacking file (1:5.46-5) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../004-gettext-base_0.23.1-2+b1_arm64.deb ...
Unpacking gettext-base (0.23.1-2+b1) ...
Selecting previously unselected package libuchardet0:arm64.
Preparing to unpack .../005-libuchardet0_0.0.8-1+b2_arm64.deb ...
Unpacking libuchardet0:arm64 (0.0.8-1+b2) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../006-groff-base_1.23.0-9_arm64.deb ...
Unpacking groff-base (1.23.0-9) ...
Selecting previously unselected package bsdextrautils.
Preparing to unpack .../007-bsdextrautils_2.41.1-1_arm64.deb ...
Unpacking bsdextrautils (2.41.1-1) ...
Selecting previously unselected package libpipeline1:arm64.
Preparing to unpack .../008-libpipeline1_1.5.8-1_arm64.deb ...
Unpacking libpipeline1:arm64 (1.5.8-1) ...
Selecting previously unselected package man-db.
Preparing to unpack .../009-man-db_2.13.1-1_arm64.deb ...
Unpacking man-db (2.13.1-1) ...
Selecting previously unselected package m4.
Preparing to unpack .../010-m4_1.4.19-8_arm64.deb ...
Unpacking m4 (1.4.19-8) ...
Selecting previously unselected package autoconf.
Preparing to unpack .../011-autoconf_2.72-3.1_all.deb ...
Unpacking autoconf (2.72-3.1) ...
Selecting previously unselected package autotools-dev.
Preparing to unpack .../012-autotools-dev_20240727.1_all.deb ...
Unpacking autotools-dev (20240727.1) ...
Selecting previously unselected package automake.
Preparing to unpack .../013-automake_1%3a1.17-4_all.deb ...
Unpacking automake (1:1.17-4) ...
Selecting previously unselected package autopoint.
Preparing to unpack .../014-autopoint_0.23.1-2_all.deb ...
Unpacking autopoint (0.23.1-2) ...
Selecting previously unselected package libdebhelper-perl.
Preparing to unpack .../015-libdebhelper-perl_13.25_all.deb ...
Unpacking libdebhelper-perl (13.25) ...
Selecting previously unselected package libtool.
Preparing to unpack .../016-libtool_2.5.4-4_all.deb ...
Unpacking libtool (2.5.4-4) ...
Selecting previously unselected package dh-autoreconf.
Preparing to unpack .../017-dh-autoreconf_20_all.deb ...
Unpacking dh-autoreconf (20) ...
Selecting previously unselected package libarchive-zip-perl.
Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ...
Unpacking libarchive-zip-perl (1.68-1) ...
Selecting previously unselected package libfile-stripnondeterminism-perl.
Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.1-2_all.deb ...
Unpacking libfile-stripnondeterminism-perl (1.14.1-2) ...
Selecting previously unselected package dh-strip-nondeterminism.
Preparing to unpack .../020-dh-strip-nondeterminism_1.14.1-2_all.deb ...
Unpacking dh-strip-nondeterminism (1.14.1-2) ...
Selecting previously unselected package libelf1t64:arm64.
Preparing to unpack .../021-libelf1t64_0.192-4_arm64.deb ...
Unpacking libelf1t64:arm64 (0.192-4) ...
Selecting previously unselected package dwz.
Preparing to unpack .../022-dwz_0.15-1+b1_arm64.deb ...
Unpacking dwz (0.15-1+b1) ...
Selecting previously unselected package libunistring5:arm64.
Preparing to unpack .../023-libunistring5_1.3-2_arm64.deb ...
Unpacking libunistring5:arm64 (1.3-2) ...
Selecting previously unselected package libxml2-16:arm64.
Preparing to unpack .../024-libxml2-16_2.14.5+dfsg-0.1_arm64.deb ...
Unpacking libxml2-16:arm64 (2.14.5+dfsg-0.1) ...
Selecting previously unselected package gettext.
Preparing to unpack .../025-gettext_0.23.1-2+b1_arm64.deb ...
Unpacking gettext (0.23.1-2+b1) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../026-intltool-debian_0.35.0+20060710.6_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.6) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../027-po-debconf_1.0.21+nmu1_all.deb ...
Unpacking po-debconf (1.0.21+nmu1) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../028-debhelper_13.25_all.deb ...
Unpacking debhelper (13.25) ...
Selecting previously unselected package dh-golang.
Preparing to unpack .../029-dh-golang_1.63_all.deb ...
Unpacking dh-golang (1.63) ...
Selecting previously unselected package golang-1.24-src.
Preparing to unpack .../030-golang-1.24-src_1.24.4-1_all.deb ...
Unpacking golang-1.24-src (1.24.4-1) ...
Selecting previously unselected package golang-1.24-go.
Preparing to unpack .../031-golang-1.24-go_1.24.4-1_arm64.deb ...
Unpacking golang-1.24-go (1.24.4-1) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../032-golang-src_2%3a1.24~2_all.deb ...
Unpacking golang-src (2:1.24~2) ...
Selecting previously unselected package golang-go:arm64.
Preparing to unpack .../033-golang-go_2%3a1.24~2_arm64.deb ...
Unpacking golang-go:arm64 (2:1.24~2) ...
Selecting previously unselected package golang-any:arm64.
Preparing to unpack .../034-golang-any_2%3a1.24~2_arm64.deb ...
Unpacking golang-any:arm64 (2:1.24~2) ...
Selecting previously unselected package golang-github-alecthomas-units-dev.
Preparing to unpack .../035-golang-github-alecthomas-units-dev_0.0~git20240927.0f3dac3-1_all.deb ...
Unpacking golang-github-alecthomas-units-dev (0.0~git20240927.0f3dac3-1) ...
Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev.
Preparing to unpack .../036-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ...
Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev.
Preparing to unpack .../037-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ...
Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Selecting previously unselected package golang-golang-x-sys-dev.
Preparing to unpack .../038-golang-golang-x-sys-dev_0.22.0-2_all.deb ...
Unpacking golang-golang-x-sys-dev (0.22.0-2) ...
Selecting previously unselected package golang-github-mattn-go-isatty-dev.
Preparing to unpack .../039-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ...
Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Selecting previously unselected package golang-github-mattn-go-colorable-dev.
Preparing to unpack .../040-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ...
Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Selecting previously unselected package golang-github-fatih-color-dev.
Preparing to unpack .../041-golang-github-fatih-color-dev_1.17.0-1_all.deb ...
Unpacking golang-github-fatih-color-dev (1.17.0-1) ...
Selecting previously unselected package golang-github-davecgh-go-spew-dev.
Preparing to unpack .../042-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ...
Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Selecting previously unselected package golang-github-pmezard-go-difflib-dev.
Preparing to unpack .../043-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ...
Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Selecting previously unselected package golang-github-stretchr-objx-dev.
Preparing to unpack .../044-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ...
Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ...
Selecting previously unselected package golang-gopkg-yaml.v3-dev.
Preparing to unpack .../045-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ...
Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Selecting previously unselected package golang-github-stretchr-testify-dev.
Preparing to unpack .../046-golang-github-stretchr-testify-dev_1.10.0-1_all.deb ...
Unpacking golang-github-stretchr-testify-dev (1.10.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-hclog-dev.
Preparing to unpack .../047-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ...
Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev.
Preparing to unpack .../048-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ...
Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Selecting previously unselected package golang-github-pkg-errors-dev.
Preparing to unpack .../049-golang-github-pkg-errors-dev_0.9.1-3_all.deb ...
Unpacking golang-github-pkg-errors-dev (0.9.1-3) ...
Selecting previously unselected package golang-github-tv42-httpunix-dev.
Preparing to unpack .../050-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ...
Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev.
Preparing to unpack .../051-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ...
Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Selecting previously unselected package golang-github-datadog-datadog-go-dev.
Preparing to unpack .../052-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ...
Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Selecting previously unselected package golang-github-google-go-cmp-dev.
Preparing to unpack .../053-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ...
Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ...
Selecting previously unselected package golang-google-protobuf-dev.
Preparing to unpack .../054-golang-google-protobuf-dev_1.36.5-1_all.deb ...
Unpacking golang-google-protobuf-dev (1.36.5-1) ...
Selecting previously unselected package golang-github-golang-protobuf-1-5-dev.
Preparing to unpack .../055-golang-github-golang-protobuf-1-5-dev_1.5.4-2_all.deb ...
Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-uuid-dev.
Preparing to unpack .../056-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ...
Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Selecting previously unselected package golang-github-hashicorp-golang-lru-dev.
Preparing to unpack .../057-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ...
Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev.
Preparing to unpack .../058-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ...
Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Selecting previously unselected package golang-github-pascaldekloe-goe-dev.
Preparing to unpack .../059-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ...
Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Selecting previously unselected package golang-github-beorn7-perks-dev.
Preparing to unpack .../060-golang-github-beorn7-perks-dev_1.0.1-3_all.deb ...
Unpacking golang-github-beorn7-perks-dev (1.0.1-3) ...
Selecting previously unselected package golang-github-cespare-xxhash-dev.
Preparing to unpack .../061-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ...
Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ...
Selecting previously unselected package golang-github-modern-go-concurrent-dev.
Preparing to unpack .../062-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ...
Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Selecting previously unselected package golang-github-modern-go-reflect2-dev.
Preparing to unpack .../063-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ...
Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-json-iterator-go-dev.
Preparing to unpack .../064-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ...
Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ...
Selecting previously unselected package golang-github-klauspost-compress-dev.
Preparing to unpack .../065-golang-github-klauspost-compress-dev_1.18.0+ds1-1_all.deb ...
Unpacking golang-github-klauspost-compress-dev (1.18.0+ds1-1) ...
Selecting previously unselected package golang-github-kylelemons-godebug-dev.
Preparing to unpack .../066-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ...
Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Selecting previously unselected package golang-github-prometheus-client-model-dev.
Preparing to unpack .../067-golang-github-prometheus-client-model-dev_0.6.2-1_all.deb ...
Unpacking golang-github-prometheus-client-model-dev (0.6.2-1) ...
Selecting previously unselected package golang-github-go-logfmt-logfmt-dev.
Preparing to unpack .../068-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ...
Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Selecting previously unselected package golang-github-go-kit-log-dev.
Preparing to unpack .../069-golang-github-go-kit-log-dev_0.2.1-1_all.deb ...
Unpacking golang-github-go-kit-log-dev (0.2.1-1) ...
Selecting previously unselected package golang-github-julienschmidt-httprouter-dev.
Preparing to unpack .../070-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ...
Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Selecting previously unselected package golang-github-munnerz-goautoneg-dev.
Preparing to unpack .../071-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ...
Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Selecting previously unselected package golang-github-jpillora-backoff-dev.
Preparing to unpack .../072-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ...
Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Selecting previously unselected package golang-golang-x-term-dev.
Preparing to unpack .../073-golang-golang-x-term-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-term-dev (0.22.0-1) ...
Selecting previously unselected package golang-golang-x-text-dev.
Preparing to unpack .../074-golang-golang-x-text-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-text-dev (0.22.0-1) ...
Selecting previously unselected package golang-golang-x-crypto-dev.
Preparing to unpack .../075-golang-golang-x-crypto-dev_1%3a0.25.0-1_all.deb ...
Unpacking golang-golang-x-crypto-dev (1:0.25.0-1) ...
Selecting previously unselected package golang-golang-x-net-dev.
Preparing to unpack .../076-golang-golang-x-net-dev_1%3a0.27.0-2_all.deb ...
Unpacking golang-golang-x-net-dev (1:0.27.0-2) ...
Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev.
Preparing to unpack .../077-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-4_all.deb ...
Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ...
Selecting previously unselected package golang-gopkg-yaml.v2-dev.
Preparing to unpack .../078-golang-gopkg-yaml.v2-dev_2.4.0-5_all.deb ...
Unpacking golang-gopkg-yaml.v2-dev (2.4.0-5) ...
Selecting previously unselected package golang-github-jmespath-go-jmespath-dev.
Preparing to unpack .../079-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ...
Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Selecting previously unselected package golang-github-aws-aws-sdk-go-dev.
Preparing to unpack .../080-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ...
Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Selecting previously unselected package golang-github-prometheus-sigv4-dev.
Preparing to unpack .../081-golang-github-prometheus-sigv4-dev_0.1.2-2_all.deb ...
Unpacking golang-github-prometheus-sigv4-dev (0.1.2-2) ...
Selecting previously unselected package golang-golang-x-oauth2-dev.
Preparing to unpack .../082-golang-golang-x-oauth2-dev_0.27.0-1_all.deb ...
Unpacking golang-golang-x-oauth2-dev (0.27.0-1) ...
Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev.
Preparing to unpack .../083-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ...
Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Selecting previously unselected package golang-github-prometheus-common-dev.
Preparing to unpack .../084-golang-github-prometheus-common-dev_0.63.0-2_all.deb ...
Unpacking golang-github-prometheus-common-dev (0.63.0-2) ...
Selecting previously unselected package golang-golang-x-sync-dev.
Preparing to unpack .../085-golang-golang-x-sync-dev_0.7.0-1_all.deb ...
Unpacking golang-golang-x-sync-dev (0.7.0-1) ...
Selecting previously unselected package golang-github-prometheus-procfs-dev.
Preparing to unpack .../086-golang-github-prometheus-procfs-dev_0.15.1-1_all.deb ...
Unpacking golang-github-prometheus-procfs-dev (0.15.1-1) ...
Selecting previously unselected package golang-github-prometheus-client-golang-dev.
Preparing to unpack .../087-golang-github-prometheus-client-golang-dev_1.21.1-1_all.deb ...
Unpacking golang-github-prometheus-client-golang-dev (1.21.1-1) ...
Selecting previously unselected package golang-github-armon-go-metrics-dev.
Preparing to unpack .../088-golang-github-armon-go-metrics-dev_0.4.1-2_all.deb ...
Unpacking golang-github-armon-go-metrics-dev (0.4.1-2) ...
Selecting previously unselected package golang-github-creack-pty-dev.
Preparing to unpack .../089-golang-github-creack-pty-dev_1.1.21-1_all.deb ...
Unpacking golang-github-creack-pty-dev (1.1.21-1) ...
Selecting previously unselected package golang-github-vmihailenco-tagparser-dev.
Preparing to unpack .../090-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ...
Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev.
Preparing to unpack .../091-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ...
Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Selecting previously unselected package golang-gopkg-tomb.v2-dev.
Preparing to unpack .../092-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ...
Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Selecting previously unselected package libsasl2-modules-db:arm64.
Preparing to unpack .../093-libsasl2-modules-db_2.1.28+dfsg1-9_arm64.deb ...
Unpacking libsasl2-modules-db:arm64 (2.1.28+dfsg1-9) ...
Selecting previously unselected package libsasl2-2:arm64.
Preparing to unpack .../094-libsasl2-2_2.1.28+dfsg1-9_arm64.deb ...
Unpacking libsasl2-2:arm64 (2.1.28+dfsg1-9) ...
Selecting previously unselected package libssl-dev:arm64.
Preparing to unpack .../095-libssl-dev_3.5.1-1_arm64.deb ...
Unpacking libssl-dev:arm64 (3.5.1-1) ...
Selecting previously unselected package libsasl2-dev.
Preparing to unpack .../096-libsasl2-dev_2.1.28+dfsg1-9_arm64.deb ...
Unpacking libsasl2-dev (2.1.28+dfsg1-9) ...
Selecting previously unselected package golang-github-kr-text-dev.
Preparing to unpack .../097-golang-github-kr-text-dev_0.2.0-2_all.deb ...
Unpacking golang-github-kr-text-dev (0.2.0-2) ...
Selecting previously unselected package golang-golang-x-mod-dev.
Preparing to unpack .../098-golang-golang-x-mod-dev_0.19.0-1_all.deb ...
Unpacking golang-golang-x-mod-dev (0.19.0-1) ...
Selecting previously unselected package golang-github-yuin-goldmark-dev.
Preparing to unpack .../099-golang-github-yuin-goldmark-dev_1.7.4-1_all.deb ...
Unpacking golang-github-yuin-goldmark-dev (1.7.4-1) ...
Selecting previously unselected package golang-golang-x-tools-dev.
Preparing to unpack .../100-golang-golang-x-tools-dev_1%3a0.25.0+ds-1_all.deb ...
Unpacking golang-golang-x-tools-dev (1:0.25.0+ds-1) ...
Selecting previously unselected package golang-github-rogpeppe-go-internal-dev.
Preparing to unpack .../101-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ...
Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Selecting previously unselected package golang-github-kr-pretty-dev.
Preparing to unpack .../102-golang-github-kr-pretty-dev_0.3.1-1_all.deb ...
Unpacking golang-github-kr-pretty-dev (0.3.1-1) ...
Selecting previously unselected package golang-gopkg-check.v1-dev.
Preparing to unpack .../103-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-2_all.deb ...
Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ...
Selecting previously unselected package golang-gopkg-mgo.v2-dev.
Preparing to unpack .../104-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ...
Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev.
Preparing to unpack .../105-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ...
Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-1.24-src (1.24.4-1) ...
Setting up libpipeline1:arm64 (1.5.8-1) ...
Setting up golang-github-google-go-cmp-dev (0.6.0-1) ...
Setting up golang-gopkg-yaml.v2-dev (2.4.0-5) ...
Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Setting up bsdextrautils (2.41.1-1) ...
Setting up golang-github-creack-pty-dev (1.1.21-1) ...
Setting up golang-github-stretchr-objx-dev (0.5.2-1) ...
Setting up libmagic-mgc (1:5.46-5) ...
Setting up golang-github-pkg-errors-dev (0.9.1-3) ...
Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Setting up libarchive-zip-perl (1.68-1) ...
Setting up libxml2-16:arm64 (2.14.5+dfsg-0.1) ...
Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Setting up libdebhelper-perl (13.25) ...
Setting up golang-golang-x-sys-dev (0.22.0-2) ...
Setting up libmagic1t64:arm64 (1:5.46-5) ...
Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Setting up gettext-base (0.23.1-2+b1) ...
Setting up m4 (1.4.19-8) ...
Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Setting up golang-golang-x-term-dev (0.22.0-1) ...
Setting up file (1:5.46-5) ...
Setting up golang-github-beorn7-perks-dev (1.0.1-3) ...
Setting up libelf1t64:arm64 (0.192-4) ...
Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Setting up libsasl2-modules-db:arm64 (2.1.28+dfsg1-9) ...
Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ...
Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Setting up autotools-dev (20240727.1) ...
Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Setting up golang-google-protobuf-dev (1.36.5-1) ...
Setting up golang-github-kr-text-dev (0.2.0-2) ...
Setting up golang-github-stretchr-testify-dev (1.10.0-1) ...
Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Setting up golang-github-yuin-goldmark-dev (1.7.4-1) ...
Setting up libunistring5:arm64 (1.3-2) ...
Setting up libssl-dev:arm64 (3.5.1-1) ...
Setting up golang-golang-x-sync-dev (0.7.0-1) ...
Setting up golang-golang-x-mod-dev (0.19.0-1) ...
Setting up autopoint (0.23.1-2) ...
Setting up golang-1.24-go (1.24.4-1) ...
Setting up golang-github-klauspost-compress-dev (1.18.0+ds1-1) ...
Setting up libsasl2-2:arm64 (2.1.28+dfsg1-9) ...
Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Setting up autoconf (2.72-3.1) ...
Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Setting up dwz (0.15-1+b1) ...
Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Setting up sensible-utils (0.0.25) ...
Setting up golang-golang-x-text-dev (0.22.0-1) ...
Setting up libuchardet0:arm64 (0.0.8-1+b2) ...
Setting up golang-github-prometheus-procfs-dev (0.15.1-1) ...
Setting up golang-github-alecthomas-units-dev (0.0~git20240927.0f3dac3-1) ...
Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Setting up golang-golang-x-crypto-dev (1:0.25.0-1) ...
Setting up golang-github-prometheus-client-model-dev (0.6.2-1) ...
Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Setting up golang-golang-x-oauth2-dev (0.27.0-1) ...
Setting up golang-src (2:1.24~2) ...
Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Setting up automake (1:1.17-4) ...
update-alternatives: using /usr/bin/automake-1.17 to provide /usr/bin/automake (automake) in auto mode
Setting up libfile-stripnondeterminism-perl (1.14.1-2) ...
Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Setting up golang-github-go-kit-log-dev (0.2.1-1) ...
Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-2) ...
Setting up gettext (0.23.1-2+b1) ...
Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Setting up libtool (2.5.4-4) ...
Setting up golang-go:arm64 (2:1.24~2) ...
Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Setting up golang-github-json-iterator-go-dev (1.1.12-2) ...
Setting up golang-any:arm64 (2:1.24~2) ...
Setting up intltool-debian (0.35.0+20060710.6) ...
Setting up dh-autoreconf (20) ...
Setting up libsasl2-dev (2.1.28+dfsg1-9) ...
Setting up golang-golang-x-net-dev (1:0.27.0-2) ...
Setting up dh-strip-nondeterminism (1.14.1-2) ...
Setting up groff-base (1.23.0-9) ...
Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Setting up golang-github-fatih-color-dev (1.17.0-1) ...
Setting up po-debconf (1.0.21+nmu1) ...
Setting up golang-golang-x-tools-dev (1:0.25.0+ds-1) ...
Setting up man-db (2.13.1-1) ...
Not building database; man-db/auto-update is not 'true'.
Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Setting up golang-github-kr-pretty-dev (0.3.1-1) ...
Setting up debhelper (13.25) ...
Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ...
Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Setting up dh-golang (1.63) ...
Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ...
Setting up golang-github-prometheus-sigv4-dev (0.1.2-2) ...
Setting up golang-github-prometheus-common-dev (0.63.0-2) ...
Setting up golang-github-prometheus-client-golang-dev (1.21.1-1) ...
Setting up golang-github-armon-go-metrics-dev (0.4.1-2) ...
Processing triggers for libc-bin (2.41-12) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Building tag database...
 -> Finished parsing the build-deps
I: Building the package
I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S  > ../golang-github-hashicorp-raft_1.5.0-1_source.changes
dpkg-buildpackage: info: source package golang-github-hashicorp-raft
dpkg-buildpackage: info: source version 1.5.0-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Daniel Swarbrick <dswarbrick@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture arm64
 debian/rules clean
dh clean --buildsystem=golang
   dh_auto_clean -O--buildsystem=golang
   dh_autoreconf_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules binary
dh binary --buildsystem=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-aarch64-linux-gnu && go install -trimpath -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench
internal/godebugs
internal/byteorder
internal/unsafeheader
internal/profilerecord
internal/asan
internal/goexperiment
internal/goos
internal/msan
internal/goarch
internal/runtime/syscall
internal/coverage/rtcov
internal/cpu
sync/atomic
math/bits
unicode/utf8
container/list
internal/itoa
crypto/internal/fips140/alias
crypto/internal/fips140deps/byteorder
internal/chacha8rand
unicode
internal/abi
internal/runtime/math
crypto/internal/fips140/subtle
cmp
crypto/internal/boring/sig
encoding
unicode/utf16
math
log/internal
internal/bytealg
internal/runtime/atomic
internal/runtime/sys
crypto/internal/fips140deps/cpu
internal/nettrace
vendor/golang.org/x/crypto/cryptobyte/asn1
vendor/golang.org/x/crypto/internal/alias
internal/runtime/exithook
internal/stringslite
internal/race
internal/runtime/maps
internal/sync
runtime
sync
crypto/subtle
iter
internal/reflectlite
weak
slices
maps
sort
errors
internal/testlog
internal/singleflight
internal/bisect
unique
runtime/cgo
io
internal/oserror
strconv
path
github.com/hashicorp/golang-lru/simplelru
math/rand/v2
vendor/golang.org/x/net/dns/dnsmessage
internal/godebug
syscall
bytes
hash
internal/saferio
strings
crypto/internal/randutil
crypto/internal/fips140deps/godebug
math/rand
hash/crc32
crypto
hash/crc64
net/netip
reflect
vendor/golang.org/x/text/transform
bufio
crypto/internal/fips140
crypto/internal/impl
github.com/hashicorp/go-immutable-radix
net/http/internal/ascii
regexp/syntax
html
crypto/internal/fips140/sha256
crypto/internal/fips140/sha3
crypto/internal/fips140/sha512
crypto/tls/internal/fips140tls
crypto/sha3
crypto/internal/fips140/hmac
crypto/internal/fips140hash
crypto/internal/fips140/check
internal/syscall/execenv
internal/syscall/unix
time
crypto/internal/fips140/aes
crypto/internal/fips140/edwards25519/field
crypto/internal/fips140/nistec/fiat
crypto/internal/fips140/hkdf
crypto/internal/fips140/bigmod
crypto/internal/fips140/tls12
crypto/internal/fips140/tls13
crypto/internal/fips140/edwards25519
regexp
crypto/internal/fips140/nistec
io/fs
internal/poll
context
internal/filepathlite
os
internal/fmtsort
encoding/binary
vendor/golang.org/x/crypto/internal/poly1305
encoding/base64
golang.org/x/sys/unix
encoding/pem
crypto/internal/sysrand
fmt
internal/sysinfo
os/signal
io/ioutil
path/filepath
net
crypto/internal/entropy
crypto/internal/fips140/drbg
crypto/internal/fips140only
crypto/internal/fips140/ecdh
crypto/internal/fips140/ecdsa
crypto/internal/fips140/ed25519
crypto/internal/fips140/mlkem
crypto/internal/fips140/rsa
crypto/internal/fips140/aes/gcm
crypto/rc4
crypto/md5
crypto/cipher
log
encoding/json
compress/flate
encoding/hex
net/url
mime
vendor/golang.org/x/net/http2/hpack
vendor/golang.org/x/text/unicode/norm
math/big
crypto/internal/boring
crypto/aes
crypto/des
crypto/ecdh
crypto/sha512
crypto/hmac
vendor/golang.org/x/crypto/chacha20
crypto/sha1
crypto/sha256
vendor/golang.org/x/text/unicode/bidi
mime/quotedprintable
net/http/internal
compress/gzip
encoding/gob
vendor/golang.org/x/crypto/chacha20poly1305
go/token
text/template/parse
flag
runtime/debug
runtime/trace
vendor/golang.org/x/text/secure/bidirule
vendor/golang.org/x/net/idna
testing
text/template
crypto/rand
crypto/elliptic
crypto/internal/boring/bbig
encoding/asn1
crypto/dsa
github.com/mattn/go-isatty
crypto/ed25519
crypto/rsa
crypto/internal/hpke
github.com/mattn/go-colorable
github.com/fatih/color
github.com/hashicorp/go-hclog
vendor/golang.org/x/crypto/cryptobyte
crypto/x509/pkix
html/template
crypto/ecdsa
vendor/golang.org/x/net/http/httpproxy
net/textproto
crypto/x509
vendor/golang.org/x/net/http/httpguts
mime/multipart
crypto/tls
net/http/httptrace
net/http
github.com/armon/go-metrics
net/rpc
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/raft
github.com/hashicorp/raft/bench
   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
dh_auto_test
	cd obj-aarch64-linux-gnu && go test -vet=off -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench
=== RUN   TestCommitment_setVoters
--- PASS: TestCommitment_setVoters (0.00s)
=== RUN   TestCommitment_match_max
--- PASS: TestCommitment_match_max (0.00s)
=== RUN   TestCommitment_match_nonVoting
--- PASS: TestCommitment_match_nonVoting (0.00s)
=== RUN   TestCommitment_recalculate
--- PASS: TestCommitment_recalculate (0.00s)
=== RUN   TestCommitment_recalculate_startIndex
--- PASS: TestCommitment_recalculate_startIndex (0.00s)
=== RUN   TestCommitment_noVoterSanity
--- PASS: TestCommitment_noVoterSanity (0.00s)
=== RUN   TestCommitment_singleVoter
--- PASS: TestCommitment_singleVoter (0.00s)
=== RUN   TestConfiguration_Configuration_Clone
--- PASS: TestConfiguration_Configuration_Clone (0.00s)
=== RUN   TestConfiguration_configurations_Clone
--- PASS: TestConfiguration_configurations_Clone (0.00s)
=== RUN   TestConfiguration_hasVote
--- PASS: TestConfiguration_hasVote (0.00s)
=== RUN   TestConfiguration_checkConfiguration
--- PASS: TestConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_nextConfiguration_table
--- PASS: TestConfiguration_nextConfiguration_table (0.00s)
=== RUN   TestConfiguration_nextConfiguration_prevIndex
--- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
=== RUN   TestConfiguration_nextConfiguration_checkConfiguration
--- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_encodeDecodePeers
--- PASS: TestConfiguration_encodeDecodePeers (0.00s)
=== RUN   TestConfiguration_encodeDecodeConfiguration
--- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
=== RUN   TestDiscardSnapshotStoreImpl
--- PASS: TestDiscardSnapshotStoreImpl (0.00s)
=== RUN   TestDiscardSnapshotSinkImpl
--- PASS: TestDiscardSnapshotSinkImpl (0.00s)
=== RUN   TestFileSnapshotStoreImpl
--- PASS: TestFileSnapshotStoreImpl (0.00s)
=== RUN   TestFileSnapshotSinkImpl
--- PASS: TestFileSnapshotSinkImpl (0.00s)
=== RUN   TestFileSS_CreateSnapshotMissingParentDir
2026-09-21T15:24:08.065Z [INFO]  creating new snapshot: path=/tmp/raft2985046737/raft2997965826/snapshots/3-10-1790004248065.tmp
--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s)
=== RUN   TestFileSS_CreateSnapshot
2026-09-21T15:24:08.066Z [INFO]  creating new snapshot: path=/tmp/raft944869434/snapshots/3-10-1790004248066.tmp
2026-09-21T15:24:08.066Z [WARN]  found temporary snapshot: name=3-10-1790004248066.tmp
--- PASS: TestFileSS_CreateSnapshot (0.00s)
=== RUN   TestFileSS_CancelSnapshot
2026-09-21T15:24:08.067Z [INFO]  creating new snapshot: path=/tmp/raft3773239809/snapshots/3-10-1790004248067.tmp
--- PASS: TestFileSS_CancelSnapshot (0.00s)
=== RUN   TestFileSS_Retention
2026-09-21T15:24:08.068Z [INFO]  creating new snapshot: path=/tmp/raft781288397/snapshots/3-10-1790004248068.tmp
2026-09-21T15:24:08.068Z [INFO]  creating new snapshot: path=/tmp/raft781288397/snapshots/3-11-1790004248068.tmp
2026-09-21T15:24:08.068Z [INFO]  creating new snapshot: path=/tmp/raft781288397/snapshots/3-12-1790004248068.tmp
2026-09-21T15:24:08.068Z [INFO]  reaping snapshot: path=/tmp/raft781288397/snapshots/3-10-1790004248068
2026-09-21T15:24:08.069Z [INFO]  creating new snapshot: path=/tmp/raft781288397/snapshots/3-13-1790004248069.tmp
2026-09-21T15:24:08.069Z [INFO]  reaping snapshot: path=/tmp/raft781288397/snapshots/3-11-1790004248068
2026-09-21T15:24:08.069Z [INFO]  creating new snapshot: path=/tmp/raft781288397/snapshots/3-14-1790004248069.tmp
2026-09-21T15:24:08.069Z [INFO]  reaping snapshot: path=/tmp/raft781288397/snapshots/3-12-1790004248068
--- PASS: TestFileSS_Retention (0.00s)
=== RUN   TestFileSS_BadPerm
--- PASS: TestFileSS_BadPerm (0.00s)
=== RUN   TestFileSS_MissingParentDir
--- PASS: TestFileSS_MissingParentDir (0.00s)
=== RUN   TestFileSS_Ordering
2026-09-21T15:24:08.070Z [INFO]  creating new snapshot: path=/tmp/raft951636168/snapshots/5-130350-1790004248070.tmp
2026-09-21T15:24:08.070Z [INFO]  creating new snapshot: path=/tmp/raft951636168/snapshots/36-204917-1790004248070.tmp
--- PASS: TestFileSS_Ordering (0.00s)
=== RUN   TestDeferFutureSuccess
--- PASS: TestDeferFutureSuccess (0.00s)
=== RUN   TestDeferFutureError
--- PASS: TestDeferFutureError (0.00s)
=== RUN   TestDeferFutureConcurrent
--- PASS: TestDeferFutureConcurrent (0.00s)
=== RUN   TestInmemSnapshotStoreImpl
--- PASS: TestInmemSnapshotStoreImpl (0.00s)
=== RUN   TestInmemSnapshotSinkImpl
--- PASS: TestInmemSnapshotSinkImpl (0.00s)
=== RUN   TestInmemSS_CreateSnapshot
--- PASS: TestInmemSS_CreateSnapshot (0.00s)
=== RUN   TestInmemSS_OpenSnapshotTwice
--- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
=== RUN   TestInmemTransportImpl
--- PASS: TestInmemTransportImpl (0.00s)
=== RUN   TestInmemTransportWriteTimeout
--- PASS: TestInmemTransportWriteTimeout (0.01s)
=== RUN   TestRaft_Integ
--- SKIP: TestRaft_Integ (0.00s)
=== RUN   TestRaft_RestartFollower_LongInitialHeartbeat
--- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s)
=== RUN   TestLogCache
--- PASS: TestLogCache (0.00s)
=== RUN   TestLogCacheWithBackendStoreError
--- PASS: TestLogCacheWithBackendStoreError (0.00s)
=== RUN   TestOldestLog
=== RUN   TestOldestLog/empty_logs
=== RUN   TestOldestLog/simple_case
--- PASS: TestOldestLog (0.00s)
    --- PASS: TestOldestLog/empty_logs (0.00s)
    --- PASS: TestOldestLog/simple_case (0.00s)
=== RUN   TestEmitsLogStoreMetrics
--- PASS: TestEmitsLogStoreMetrics (0.01s)
=== RUN   TestNetworkTransport_CloseStreams
--- PASS: TestNetworkTransport_CloseStreams (0.00s)
=== RUN   TestNetworkTransport_StartStop
--- PASS: TestNetworkTransport_StartStop (0.00s)
=== RUN   TestNetworkTransport_Heartbeat_FastPath
--- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s)
=== RUN   TestNetworkTransport_AppendEntries
--- PASS: TestNetworkTransport_AppendEntries (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline
--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.01s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_CloseStreams
--- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.02s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0
2026-09-21T15:24:08.174Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2
2026-09-21T15:24:08.226Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10
2026-09-21T15:24:08.279Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s)
=== RUN   TestNetworkTransport_RequestVote
--- PASS: TestNetworkTransport_RequestVote (0.00s)
=== RUN   TestNetworkTransport_InstallSnapshot
--- PASS: TestNetworkTransport_InstallSnapshot (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode
--- PASS: TestNetworkTransport_EncodeDecode (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode_AddressProvider
--- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
=== RUN   TestNetworkTransport_PooledConn
--- PASS: TestNetworkTransport_PooledConn (0.00s)
=== RUN   TestNetworkTransport_ListenBackoff
    net_transport_test.go:839: countingWriter: 2026-09-21T15:24:08.287Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-09-21T15:24:08.293Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-09-21T15:24:08.304Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-09-21T15:24:08.325Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-09-21T15:24:08.365Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-09-21T15:24:08.446Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-09-21T15:24:08.607Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-09-21T15:24:08.927Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-09-21T15:24:09.568Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-09-21T15:24:10.570Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-09-21T15:24:11.571Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
--- PASS: TestNetworkTransport_ListenBackoff (4.00s)
=== RUN   TestPeersJSON_BadConfiguration
--- PASS: TestPeersJSON_BadConfiguration (0.00s)
=== RUN   TestPeersJSON_ReadPeersJSON
--- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
=== RUN   TestPeersJSON_ReadConfigJSON
--- PASS: TestPeersJSON_ReadConfigJSON (0.00s)
=== RUN   TestRaft_StartStop
2026-09-21T15:24:12.290Z [INFO]  server-7cc9edc5-d8ea-9869-b971-1cb608314e10: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7cc9edc5-d8ea-9869-b971-1cb608314e10 Address:7cc9edc5-d8ea-9869-b971-1cb608314e10}]"
--- PASS: TestRaft_StartStop (0.00s)
=== RUN   TestRaft_AfterShutdown
2026-09-21T15:24:12.290Z [INFO]  server-2e9ee5d0-e4f7-d965-3342-3fadde9df81c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e9ee5d0-e4f7-d965-3342-3fadde9df81c Address:2e9ee5d0-e4f7-d965-3342-3fadde9df81c}]"
--- PASS: TestRaft_AfterShutdown (0.00s)
=== RUN   TestRaft_LiveBootstrap
2026-09-21T15:24:12.290Z [INFO]  server-3c0bf50a-f2c9-a353-8388-b40bf07d013e: initial configuration: index=0 servers=[]
2026-09-21T15:24:12.290Z [INFO]  server-e9957e05-0525-ff82-9dab-741eb93a8d11: initial configuration: index=0 servers=[]
2026-09-21T15:24:12.291Z [INFO]  server-e2469448-178a-af08-c6fb-bc488f9c7781: initial configuration: index=0 servers=[]
2026-09-21T15:24:12.291Z [INFO]  server-e9957e05-0525-ff82-9dab-741eb93a8d11: entering follower state: follower="Node at e9957e05-0525-ff82-9dab-741eb93a8d11 [Follower]" leader-address= leader-id=
2026-09-21T15:24:12.291Z [INFO]  server-3c0bf50a-f2c9-a353-8388-b40bf07d013e: entering follower state: follower="Node at 3c0bf50a-f2c9-a353-8388-b40bf07d013e [Follower]" leader-address= leader-id=
2026-09-21T15:24:12.291Z [INFO]  server-e2469448-178a-af08-c6fb-bc488f9c7781: entering follower state: follower="Node at e2469448-178a-af08-c6fb-bc488f9c7781 [Follower]" leader-address= leader-id=
2026-09-21T15:24:12.291Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:12.344Z [WARN]  server-e2469448-178a-af08-c6fb-bc488f9c7781: no known peers, aborting election
2026-09-21T15:24:12.360Z [WARN]  server-e9957e05-0525-ff82-9dab-741eb93a8d11: no known peers, aborting election
2026-09-21T15:24:12.363Z [WARN]  server-3c0bf50a-f2c9-a353-8388-b40bf07d013e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:12.363Z [INFO]  server-3c0bf50a-f2c9-a353-8388-b40bf07d013e: entering candidate state: node="Node at 3c0bf50a-f2c9-a353-8388-b40bf07d013e [Candidate]" term=2
2026-09-21T15:24:12.363Z [INFO]  server-3c0bf50a-f2c9-a353-8388-b40bf07d013e: election won: term=2 tally=2
2026-09-21T15:24:12.363Z [INFO]  server-3c0bf50a-f2c9-a353-8388-b40bf07d013e: entering leader state: leader="Node at 3c0bf50a-f2c9-a353-8388-b40bf07d013e [Leader]"
2026-09-21T15:24:12.363Z [INFO]  server-3c0bf50a-f2c9-a353-8388-b40bf07d013e: added peer, starting replication: peer=server-e9957e05-0525-ff82-9dab-741eb93a8d11
2026-09-21T15:24:12.363Z [INFO]  server-3c0bf50a-f2c9-a353-8388-b40bf07d013e: added peer, starting replication: peer=server-e2469448-178a-af08-c6fb-bc488f9c7781
2026-09-21T15:24:12.363Z [WARN]  server-e2469448-178a-af08-c6fb-bc488f9c7781: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2026-09-21T15:24:12.363Z [WARN]  server-3c0bf50a-f2c9-a353-8388-b40bf07d013e: appendEntries rejected, sending older logs: peer="{Voter server-e2469448-178a-af08-c6fb-bc488f9c7781 e2469448-178a-af08-c6fb-bc488f9c7781}" next=1
2026-09-21T15:24:12.363Z [WARN]  server-e9957e05-0525-ff82-9dab-741eb93a8d11: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2026-09-21T15:24:12.363Z [WARN]  server-3c0bf50a-f2c9-a353-8388-b40bf07d013e: appendEntries rejected, sending older logs: peer="{Voter server-e9957e05-0525-ff82-9dab-741eb93a8d11 e9957e05-0525-ff82-9dab-741eb93a8d11}" next=1
2026-09-21T15:24:12.364Z [INFO]  server-3c0bf50a-f2c9-a353-8388-b40bf07d013e: pipelining replication: peer="{Voter server-e9957e05-0525-ff82-9dab-741eb93a8d11 e9957e05-0525-ff82-9dab-741eb93a8d11}"
2026-09-21T15:24:12.364Z [INFO]  server-3c0bf50a-f2c9-a353-8388-b40bf07d013e: pipelining replication: peer="{Voter server-e2469448-178a-af08-c6fb-bc488f9c7781 e2469448-178a-af08-c6fb-bc488f9c7781}"
2026-09-21T15:24:12.469Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:12.363585348 +0000 UTC m=+4.304755521 (2 nodes), 72.23528ms from start of poll, 72.6948ms from cluster start. Timeout at 2026-09-21 15:24:12.468590228 +0000 UTC m=+4.409760401, 105.00488ms after stability
2026-09-21T15:24:12.469Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:12.574Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:12.469395188 +0000 UTC m=+4.410565361 (1 nodes), 640ns from start of poll, 178.50464ms from cluster start. Timeout at 2026-09-21 15:24:12.574403428 +0000 UTC m=+4.515573601, 105.00824ms after stability
2026-09-21T15:24:12.585Z [INFO]  server-3c0bf50a-f2c9-a353-8388-b40bf07d013e: aborting pipeline replication: peer="{Voter server-e2469448-178a-af08-c6fb-bc488f9c7781 e2469448-178a-af08-c6fb-bc488f9c7781}"
2026-09-21T15:24:12.585Z [INFO]  server-3c0bf50a-f2c9-a353-8388-b40bf07d013e: aborting pipeline replication: peer="{Voter server-e9957e05-0525-ff82-9dab-741eb93a8d11 e9957e05-0525-ff82-9dab-741eb93a8d11}"
--- PASS: TestRaft_LiveBootstrap (0.30s)
=== RUN   TestRaft_LiveBootstrap_From_NonVoter
2026-09-21T15:24:12.586Z [INFO]  server-81c8d75d-1ed9-c299-0752-ee1678ba2502: initial configuration: index=0 servers=[]
2026-09-21T15:24:12.586Z [INFO]  server-626e37d3-d449-0b27-3df8-73471c89ca67: initial configuration: index=0 servers=[]
2026-09-21T15:24:12.586Z [INFO]  server-626e37d3-d449-0b27-3df8-73471c89ca67: entering follower state: follower="Node at 626e37d3-d449-0b27-3df8-73471c89ca67 [Follower]" leader-address= leader-id=
2026-09-21T15:24:12.586Z [INFO]  server-81c8d75d-1ed9-c299-0752-ee1678ba2502: entering follower state: follower="Node at 81c8d75d-1ed9-c299-0752-ee1678ba2502 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s)
=== RUN   TestRaft_RecoverCluster_NoState
2026-09-21T15:24:12.587Z [INFO]  server-dbc0d02a-8617-490c-d274-244b02903a3d: initial configuration: index=0 servers=[]
--- PASS: TestRaft_RecoverCluster_NoState (0.00s)
=== RUN   TestRaft_RecoverCluster
=== RUN   TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs
2026-09-21T15:24:12.588Z [INFO]  server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12 Address:151edcf3-72ac-c8f5-082a-2dc2c5f0ce12} {Suffrage:Voter ID:server-5e8364ed-6578-f539-7467-d49e93e0407a Address:5e8364ed-6578-f539-7467-d49e93e0407a} {Suffrage:Voter ID:server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922 Address:4ec1fc4f-fd69-4cde-b0cd-518915d6b922}]"
2026-09-21T15:24:12.588Z [INFO]  server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12: entering follower state: follower="Node at 151edcf3-72ac-c8f5-082a-2dc2c5f0ce12 [Follower]" leader-address= leader-id=
2026-09-21T15:24:12.588Z [INFO]  server-5e8364ed-6578-f539-7467-d49e93e0407a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12 Address:151edcf3-72ac-c8f5-082a-2dc2c5f0ce12} {Suffrage:Voter ID:server-5e8364ed-6578-f539-7467-d49e93e0407a Address:5e8364ed-6578-f539-7467-d49e93e0407a} {Suffrage:Voter ID:server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922 Address:4ec1fc4f-fd69-4cde-b0cd-518915d6b922}]"
2026-09-21T15:24:12.588Z [INFO]  server-5e8364ed-6578-f539-7467-d49e93e0407a: entering follower state: follower="Node at 5e8364ed-6578-f539-7467-d49e93e0407a [Follower]" leader-address= leader-id=
2026-09-21T15:24:12.588Z [INFO]  server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12 Address:151edcf3-72ac-c8f5-082a-2dc2c5f0ce12} {Suffrage:Voter ID:server-5e8364ed-6578-f539-7467-d49e93e0407a Address:5e8364ed-6578-f539-7467-d49e93e0407a} {Suffrage:Voter ID:server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922 Address:4ec1fc4f-fd69-4cde-b0cd-518915d6b922}]"
2026-09-21T15:24:12.588Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:12.588Z [INFO]  server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922: entering follower state: follower="Node at 4ec1fc4f-fd69-4cde-b0cd-518915d6b922 [Follower]" leader-address= leader-id=
2026-09-21T15:24:12.645Z [WARN]  server-5e8364ed-6578-f539-7467-d49e93e0407a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:12.645Z [INFO]  server-5e8364ed-6578-f539-7467-d49e93e0407a: entering candidate state: node="Node at 5e8364ed-6578-f539-7467-d49e93e0407a [Candidate]" term=2
2026-09-21T15:24:12.645Z [INFO]  server-5e8364ed-6578-f539-7467-d49e93e0407a: election won: term=2 tally=2
2026-09-21T15:24:12.645Z [INFO]  server-5e8364ed-6578-f539-7467-d49e93e0407a: entering leader state: leader="Node at 5e8364ed-6578-f539-7467-d49e93e0407a [Leader]"
2026-09-21T15:24:12.645Z [INFO]  server-5e8364ed-6578-f539-7467-d49e93e0407a: added peer, starting replication: peer=server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12
2026-09-21T15:24:12.645Z [INFO]  server-5e8364ed-6578-f539-7467-d49e93e0407a: added peer, starting replication: peer=server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922
2026-09-21T15:24:12.645Z [INFO]  server-5e8364ed-6578-f539-7467-d49e93e0407a: pipelining replication: peer="{Voter server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12 151edcf3-72ac-c8f5-082a-2dc2c5f0ce12}"
2026-09-21T15:24:12.645Z [INFO]  server-5e8364ed-6578-f539-7467-d49e93e0407a: pipelining replication: peer="{Voter server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922 4ec1fc4f-fd69-4cde-b0cd-518915d6b922}"
2026-09-21T15:24:12.751Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:12.645672468 +0000 UTC m=+4.586842641 (1 nodes), 57.10484ms from start of poll, 57.67096ms from cluster start. Timeout at 2026-09-21 15:24:12.750678708 +0000 UTC m=+4.691848841, 105.0062ms after stability
2026-09-21T15:24:12.751Z [INFO]  server-5e8364ed-6578-f539-7467-d49e93e0407a: aborting pipeline replication: peer="{Voter server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922 4ec1fc4f-fd69-4cde-b0cd-518915d6b922}"
2026-09-21T15:24:12.751Z [INFO]  server-5e8364ed-6578-f539-7467-d49e93e0407a: aborting pipeline replication: peer="{Voter server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12 151edcf3-72ac-c8f5-082a-2dc2c5f0ce12}"
2026-09-21T15:24:12.751Z [INFO]  creating new snapshot: path=/tmp/raft3342372026/snapshots/2-2-1790004252751.tmp
2026-09-21T15:24:12.752Z [INFO]  server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12: starting restore from snapshot: id=2-2-1790004252751 last-index=2 last-term=2 size-in-bytes=1
2026-09-21T15:24:12.752Z [INFO]  server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12: snapshot restore progress: id=2-2-1790004252751 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2026-09-21T15:24:12.752Z [INFO]  server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12: restored from snapshot: id=2-2-1790004252751 last-index=2 last-term=2 size-in-bytes=1
2026-09-21T15:24:12.752Z [INFO]  server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12 Address:151edcf3-72ac-c8f5-082a-2dc2c5f0ce12} {Suffrage:Voter ID:server-5e8364ed-6578-f539-7467-d49e93e0407a Address:5e8364ed-6578-f539-7467-d49e93e0407a} {Suffrage:Voter ID:server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922 Address:4ec1fc4f-fd69-4cde-b0cd-518915d6b922}]"
2026-09-21T15:24:12.752Z [INFO]  creating new snapshot: path=/tmp/raft3186320848/snapshots/2-2-1790004252752.tmp
2026-09-21T15:24:12.752Z [INFO]  server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12: entering follower state: follower="Node at 151edcf3-72ac-c8f5-082a-2dc2c5f0ce12 [Follower]" leader-address= leader-id=
2026-09-21T15:24:12.752Z [INFO]  server-5e8364ed-6578-f539-7467-d49e93e0407a: starting restore from snapshot: id=2-2-1790004252752 last-index=2 last-term=2 size-in-bytes=1
2026-09-21T15:24:12.753Z [INFO]  server-5e8364ed-6578-f539-7467-d49e93e0407a: snapshot restore progress: id=2-2-1790004252752 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2026-09-21T15:24:12.753Z [INFO]  server-5e8364ed-6578-f539-7467-d49e93e0407a: restored from snapshot: id=2-2-1790004252752 last-index=2 last-term=2 size-in-bytes=1
2026-09-21T15:24:12.753Z [INFO]  server-5e8364ed-6578-f539-7467-d49e93e0407a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12 Address:151edcf3-72ac-c8f5-082a-2dc2c5f0ce12} {Suffrage:Voter ID:server-5e8364ed-6578-f539-7467-d49e93e0407a Address:5e8364ed-6578-f539-7467-d49e93e0407a} {Suffrage:Voter ID:server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922 Address:4ec1fc4f-fd69-4cde-b0cd-518915d6b922}]"
2026-09-21T15:24:12.753Z [INFO]  creating new snapshot: path=/tmp/raft3853694943/snapshots/2-2-1790004252753.tmp
2026-09-21T15:24:12.753Z [INFO]  server-5e8364ed-6578-f539-7467-d49e93e0407a: entering follower state: follower="Node at 5e8364ed-6578-f539-7467-d49e93e0407a [Follower]" leader-address= leader-id=
2026-09-21T15:24:12.753Z [INFO]  server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922: starting restore from snapshot: id=2-2-1790004252753 last-index=2 last-term=2 size-in-bytes=1
2026-09-21T15:24:12.754Z [INFO]  server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922: snapshot restore progress: id=2-2-1790004252753 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2026-09-21T15:24:12.754Z [INFO]  server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922: restored from snapshot: id=2-2-1790004252753 last-index=2 last-term=2 size-in-bytes=1
2026-09-21T15:24:12.754Z [INFO]  server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12 Address:151edcf3-72ac-c8f5-082a-2dc2c5f0ce12} {Suffrage:Voter ID:server-5e8364ed-6578-f539-7467-d49e93e0407a Address:5e8364ed-6578-f539-7467-d49e93e0407a} {Suffrage:Voter ID:server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922 Address:4ec1fc4f-fd69-4cde-b0cd-518915d6b922}]"
2026-09-21T15:24:12.754Z [INFO]  server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922: entering follower state: follower="Node at 4ec1fc4f-fd69-4cde-b0cd-518915d6b922 [Follower]" leader-address= leader-id=
2026-09-21T15:24:12.806Z [WARN]  server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:12.806Z [INFO]  server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922: entering candidate state: node="Node at 4ec1fc4f-fd69-4cde-b0cd-518915d6b922 [Candidate]" term=3
2026-09-21T15:24:12.806Z [INFO]  server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922: election won: term=3 tally=2
2026-09-21T15:24:12.806Z [INFO]  server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922: entering leader state: leader="Node at 4ec1fc4f-fd69-4cde-b0cd-518915d6b922 [Leader]"
2026-09-21T15:24:12.806Z [INFO]  server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922: added peer, starting replication: peer=server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12
2026-09-21T15:24:12.806Z [INFO]  server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922: added peer, starting replication: peer=server-5e8364ed-6578-f539-7467-d49e93e0407a
2026-09-21T15:24:12.806Z [INFO]  server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922: pipelining replication: peer="{Voter server-5e8364ed-6578-f539-7467-d49e93e0407a 5e8364ed-6578-f539-7467-d49e93e0407a}"
2026-09-21T15:24:12.806Z [INFO]  server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922: pipelining replication: peer="{Voter server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12 151edcf3-72ac-c8f5-082a-2dc2c5f0ce12}"
2026-09-21T15:24:13.069Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:13.175Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:13.069416708 +0000 UTC m=+5.010586881 (1 nodes), 39.4µs from start of poll, 481.4152ms from cluster start. Timeout at 2026-09-21 15:24:13.174424988 +0000 UTC m=+5.115595161, 105.00828ms after stability
2026-09-21T15:24:13.175Z [INFO]  server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922: aborting pipeline replication: peer="{Voter server-151edcf3-72ac-c8f5-082a-2dc2c5f0ce12 151edcf3-72ac-c8f5-082a-2dc2c5f0ce12}"
2026-09-21T15:24:13.175Z [INFO]  server-4ec1fc4f-fd69-4cde-b0cd-518915d6b922: aborting pipeline replication: peer="{Voter server-5e8364ed-6578-f539-7467-d49e93e0407a 5e8364ed-6578-f539-7467-d49e93e0407a}"
=== RUN   TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs
2026-09-21T15:24:13.176Z [INFO]  server-09215038-9561-a16e-05d5-8d1fbdd13b02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09215038-9561-a16e-05d5-8d1fbdd13b02 Address:09215038-9561-a16e-05d5-8d1fbdd13b02} {Suffrage:Voter ID:server-c0f5097d-835f-b260-b570-592255ff07f8 Address:c0f5097d-835f-b260-b570-592255ff07f8} {Suffrage:Voter ID:server-69fae30d-db78-15a5-8484-e39e136f4a6b Address:69fae30d-db78-15a5-8484-e39e136f4a6b}]"
2026-09-21T15:24:13.176Z [INFO]  server-c0f5097d-835f-b260-b570-592255ff07f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09215038-9561-a16e-05d5-8d1fbdd13b02 Address:09215038-9561-a16e-05d5-8d1fbdd13b02} {Suffrage:Voter ID:server-c0f5097d-835f-b260-b570-592255ff07f8 Address:c0f5097d-835f-b260-b570-592255ff07f8} {Suffrage:Voter ID:server-69fae30d-db78-15a5-8484-e39e136f4a6b Address:69fae30d-db78-15a5-8484-e39e136f4a6b}]"
2026-09-21T15:24:13.176Z [INFO]  server-09215038-9561-a16e-05d5-8d1fbdd13b02: entering follower state: follower="Node at 09215038-9561-a16e-05d5-8d1fbdd13b02 [Follower]" leader-address= leader-id=
2026-09-21T15:24:13.176Z [INFO]  server-c0f5097d-835f-b260-b570-592255ff07f8: entering follower state: follower="Node at c0f5097d-835f-b260-b570-592255ff07f8 [Follower]" leader-address= leader-id=
2026-09-21T15:24:13.176Z [INFO]  server-69fae30d-db78-15a5-8484-e39e136f4a6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09215038-9561-a16e-05d5-8d1fbdd13b02 Address:09215038-9561-a16e-05d5-8d1fbdd13b02} {Suffrage:Voter ID:server-c0f5097d-835f-b260-b570-592255ff07f8 Address:c0f5097d-835f-b260-b570-592255ff07f8} {Suffrage:Voter ID:server-69fae30d-db78-15a5-8484-e39e136f4a6b Address:69fae30d-db78-15a5-8484-e39e136f4a6b}]"
2026-09-21T15:24:13.176Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:13.176Z [INFO]  server-69fae30d-db78-15a5-8484-e39e136f4a6b: entering follower state: follower="Node at 69fae30d-db78-15a5-8484-e39e136f4a6b [Follower]" leader-address= leader-id=
2026-09-21T15:24:13.228Z [WARN]  server-c0f5097d-835f-b260-b570-592255ff07f8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:13.228Z [INFO]  server-c0f5097d-835f-b260-b570-592255ff07f8: entering candidate state: node="Node at c0f5097d-835f-b260-b570-592255ff07f8 [Candidate]" term=2
2026-09-21T15:24:13.228Z [INFO]  server-c0f5097d-835f-b260-b570-592255ff07f8: election won: term=2 tally=2
2026-09-21T15:24:13.228Z [INFO]  server-c0f5097d-835f-b260-b570-592255ff07f8: entering leader state: leader="Node at c0f5097d-835f-b260-b570-592255ff07f8 [Leader]"
2026-09-21T15:24:13.228Z [INFO]  server-c0f5097d-835f-b260-b570-592255ff07f8: added peer, starting replication: peer=server-09215038-9561-a16e-05d5-8d1fbdd13b02
2026-09-21T15:24:13.228Z [INFO]  server-c0f5097d-835f-b260-b570-592255ff07f8: added peer, starting replication: peer=server-69fae30d-db78-15a5-8484-e39e136f4a6b
2026-09-21T15:24:13.228Z [INFO]  server-c0f5097d-835f-b260-b570-592255ff07f8: pipelining replication: peer="{Voter server-69fae30d-db78-15a5-8484-e39e136f4a6b 69fae30d-db78-15a5-8484-e39e136f4a6b}"
2026-09-21T15:24:13.228Z [INFO]  server-c0f5097d-835f-b260-b570-592255ff07f8: pipelining replication: peer="{Voter server-09215038-9561-a16e-05d5-8d1fbdd13b02 09215038-9561-a16e-05d5-8d1fbdd13b02}"
2026-09-21T15:24:13.334Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:13.228647588 +0000 UTC m=+5.169817761 (1 nodes), 51.92268ms from start of poll, 52.46012ms from cluster start. Timeout at 2026-09-21 15:24:13.333655468 +0000 UTC m=+5.274825681, 105.00792ms after stability
2026-09-21T15:24:13.335Z [INFO]  server-c0f5097d-835f-b260-b570-592255ff07f8: aborting pipeline replication: peer="{Voter server-69fae30d-db78-15a5-8484-e39e136f4a6b 69fae30d-db78-15a5-8484-e39e136f4a6b}"
2026-09-21T15:24:13.335Z [INFO]  server-c0f5097d-835f-b260-b570-592255ff07f8: aborting pipeline replication: peer="{Voter server-09215038-9561-a16e-05d5-8d1fbdd13b02 09215038-9561-a16e-05d5-8d1fbdd13b02}"
2026-09-21T15:24:13.335Z [INFO]  creating new snapshot: path=/tmp/raft3718516615/snapshots/2-6-1790004253335.tmp
2026-09-21T15:24:13.336Z [INFO]  server-09215038-9561-a16e-05d5-8d1fbdd13b02: starting restore from snapshot: id=2-6-1790004253335 last-index=6 last-term=2 size-in-bytes=25
2026-09-21T15:24:13.336Z [INFO]  server-09215038-9561-a16e-05d5-8d1fbdd13b02: snapshot restore progress: id=2-6-1790004253335 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2026-09-21T15:24:13.336Z [INFO]  server-09215038-9561-a16e-05d5-8d1fbdd13b02: restored from snapshot: id=2-6-1790004253335 last-index=6 last-term=2 size-in-bytes=25
2026-09-21T15:24:13.336Z [INFO]  server-09215038-9561-a16e-05d5-8d1fbdd13b02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09215038-9561-a16e-05d5-8d1fbdd13b02 Address:09215038-9561-a16e-05d5-8d1fbdd13b02} {Suffrage:Voter ID:server-c0f5097d-835f-b260-b570-592255ff07f8 Address:c0f5097d-835f-b260-b570-592255ff07f8} {Suffrage:Voter ID:server-69fae30d-db78-15a5-8484-e39e136f4a6b Address:69fae30d-db78-15a5-8484-e39e136f4a6b}]"
2026-09-21T15:24:13.336Z [INFO]  creating new snapshot: path=/tmp/raft2756343482/snapshots/2-6-1790004253336.tmp
2026-09-21T15:24:13.336Z [INFO]  server-09215038-9561-a16e-05d5-8d1fbdd13b02: entering follower state: follower="Node at 09215038-9561-a16e-05d5-8d1fbdd13b02 [Follower]" leader-address= leader-id=
2026-09-21T15:24:13.336Z [INFO]  server-c0f5097d-835f-b260-b570-592255ff07f8: starting restore from snapshot: id=2-6-1790004253336 last-index=6 last-term=2 size-in-bytes=25
2026-09-21T15:24:13.337Z [INFO]  server-c0f5097d-835f-b260-b570-592255ff07f8: snapshot restore progress: id=2-6-1790004253336 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2026-09-21T15:24:13.337Z [INFO]  server-c0f5097d-835f-b260-b570-592255ff07f8: restored from snapshot: id=2-6-1790004253336 last-index=6 last-term=2 size-in-bytes=25
2026-09-21T15:24:13.337Z [INFO]  server-c0f5097d-835f-b260-b570-592255ff07f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09215038-9561-a16e-05d5-8d1fbdd13b02 Address:09215038-9561-a16e-05d5-8d1fbdd13b02} {Suffrage:Voter ID:server-c0f5097d-835f-b260-b570-592255ff07f8 Address:c0f5097d-835f-b260-b570-592255ff07f8} {Suffrage:Voter ID:server-69fae30d-db78-15a5-8484-e39e136f4a6b Address:69fae30d-db78-15a5-8484-e39e136f4a6b}]"
2026-09-21T15:24:13.337Z [INFO]  server-c0f5097d-835f-b260-b570-592255ff07f8: entering follower state: follower="Node at c0f5097d-835f-b260-b570-592255ff07f8 [Follower]" leader-address= leader-id=
2026-09-21T15:24:13.337Z [INFO]  creating new snapshot: path=/tmp/raft1255176302/snapshots/2-6-1790004253337.tmp
2026-09-21T15:24:13.337Z [INFO]  server-69fae30d-db78-15a5-8484-e39e136f4a6b: starting restore from snapshot: id=2-6-1790004253337 last-index=6 last-term=2 size-in-bytes=25
2026-09-21T15:24:13.337Z [INFO]  server-69fae30d-db78-15a5-8484-e39e136f4a6b: snapshot restore progress: id=2-6-1790004253337 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2026-09-21T15:24:13.338Z [INFO]  server-69fae30d-db78-15a5-8484-e39e136f4a6b: restored from snapshot: id=2-6-1790004253337 last-index=6 last-term=2 size-in-bytes=25
2026-09-21T15:24:13.338Z [INFO]  server-69fae30d-db78-15a5-8484-e39e136f4a6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09215038-9561-a16e-05d5-8d1fbdd13b02 Address:09215038-9561-a16e-05d5-8d1fbdd13b02} {Suffrage:Voter ID:server-c0f5097d-835f-b260-b570-592255ff07f8 Address:c0f5097d-835f-b260-b570-592255ff07f8} {Suffrage:Voter ID:server-69fae30d-db78-15a5-8484-e39e136f4a6b Address:69fae30d-db78-15a5-8484-e39e136f4a6b}]"
2026-09-21T15:24:13.338Z [INFO]  server-69fae30d-db78-15a5-8484-e39e136f4a6b: entering follower state: follower="Node at 69fae30d-db78-15a5-8484-e39e136f4a6b [Follower]" leader-address= leader-id=
2026-09-21T15:24:13.401Z [WARN]  server-69fae30d-db78-15a5-8484-e39e136f4a6b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:13.401Z [INFO]  server-69fae30d-db78-15a5-8484-e39e136f4a6b: entering candidate state: node="Node at 69fae30d-db78-15a5-8484-e39e136f4a6b [Candidate]" term=3
2026-09-21T15:24:13.401Z [INFO]  server-69fae30d-db78-15a5-8484-e39e136f4a6b: election won: term=3 tally=2
2026-09-21T15:24:13.401Z [INFO]  server-69fae30d-db78-15a5-8484-e39e136f4a6b: entering leader state: leader="Node at 69fae30d-db78-15a5-8484-e39e136f4a6b [Leader]"
2026-09-21T15:24:13.402Z [INFO]  server-69fae30d-db78-15a5-8484-e39e136f4a6b: added peer, starting replication: peer=server-09215038-9561-a16e-05d5-8d1fbdd13b02
2026-09-21T15:24:13.402Z [INFO]  server-69fae30d-db78-15a5-8484-e39e136f4a6b: added peer, starting replication: peer=server-c0f5097d-835f-b260-b570-592255ff07f8
2026-09-21T15:24:13.402Z [INFO]  server-69fae30d-db78-15a5-8484-e39e136f4a6b: pipelining replication: peer="{Voter server-c0f5097d-835f-b260-b570-592255ff07f8 c0f5097d-835f-b260-b570-592255ff07f8}"
2026-09-21T15:24:13.402Z [INFO]  server-69fae30d-db78-15a5-8484-e39e136f4a6b: pipelining replication: peer="{Voter server-09215038-9561-a16e-05d5-8d1fbdd13b02 09215038-9561-a16e-05d5-8d1fbdd13b02}"
2026-09-21T15:24:13.653Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:13.759Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:13.653957308 +0000 UTC m=+5.595127481 (1 nodes), 42.16µs from start of poll, 477.76984ms from cluster start. Timeout at 2026-09-21 15:24:13.758964788 +0000 UTC m=+5.700134961, 105.00748ms after stability
2026-09-21T15:24:13.759Z [INFO]  server-69fae30d-db78-15a5-8484-e39e136f4a6b: aborting pipeline replication: peer="{Voter server-c0f5097d-835f-b260-b570-592255ff07f8 c0f5097d-835f-b260-b570-592255ff07f8}"
2026-09-21T15:24:13.759Z [INFO]  server-69fae30d-db78-15a5-8484-e39e136f4a6b: aborting pipeline replication: peer="{Voter server-09215038-9561-a16e-05d5-8d1fbdd13b02 09215038-9561-a16e-05d5-8d1fbdd13b02}"
2026-09-21T15:24:14.259Z [ERROR] server-69fae30d-db78-15a5-8484-e39e136f4a6b: failed to heartbeat to: peer=09215038-9561-a16e-05d5-8d1fbdd13b02 backoff time=10ms error="command timed out"
=== RUN   TestRaft_RecoverCluster/snapshot,_with_trailing_logs
2026-09-21T15:24:14.260Z [INFO]  server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b Address:96d806c0-eb1b-fecf-7fe1-74cf4841c45b} {Suffrage:Voter ID:server-ef4faf59-d6bd-fe17-7b86-f735a058d961 Address:ef4faf59-d6bd-fe17-7b86-f735a058d961} {Suffrage:Voter ID:server-a1884d3f-f531-9ce4-fae8-e0369431707d Address:a1884d3f-f531-9ce4-fae8-e0369431707d}]"
2026-09-21T15:24:14.260Z [INFO]  server-ef4faf59-d6bd-fe17-7b86-f735a058d961: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b Address:96d806c0-eb1b-fecf-7fe1-74cf4841c45b} {Suffrage:Voter ID:server-ef4faf59-d6bd-fe17-7b86-f735a058d961 Address:ef4faf59-d6bd-fe17-7b86-f735a058d961} {Suffrage:Voter ID:server-a1884d3f-f531-9ce4-fae8-e0369431707d Address:a1884d3f-f531-9ce4-fae8-e0369431707d}]"
2026-09-21T15:24:14.260Z [INFO]  server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b: entering follower state: follower="Node at 96d806c0-eb1b-fecf-7fe1-74cf4841c45b [Follower]" leader-address= leader-id=
2026-09-21T15:24:14.260Z [INFO]  server-a1884d3f-f531-9ce4-fae8-e0369431707d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b Address:96d806c0-eb1b-fecf-7fe1-74cf4841c45b} {Suffrage:Voter ID:server-ef4faf59-d6bd-fe17-7b86-f735a058d961 Address:ef4faf59-d6bd-fe17-7b86-f735a058d961} {Suffrage:Voter ID:server-a1884d3f-f531-9ce4-fae8-e0369431707d Address:a1884d3f-f531-9ce4-fae8-e0369431707d}]"
2026-09-21T15:24:14.260Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:14.260Z [INFO]  server-ef4faf59-d6bd-fe17-7b86-f735a058d961: entering follower state: follower="Node at ef4faf59-d6bd-fe17-7b86-f735a058d961 [Follower]" leader-address= leader-id=
2026-09-21T15:24:14.260Z [INFO]  server-a1884d3f-f531-9ce4-fae8-e0369431707d: entering follower state: follower="Node at a1884d3f-f531-9ce4-fae8-e0369431707d [Follower]" leader-address= leader-id=
2026-09-21T15:24:14.314Z [WARN]  server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:14.314Z [INFO]  server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b: entering candidate state: node="Node at 96d806c0-eb1b-fecf-7fe1-74cf4841c45b [Candidate]" term=2
2026-09-21T15:24:14.314Z [INFO]  server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b: election won: term=2 tally=2
2026-09-21T15:24:14.314Z [INFO]  server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b: entering leader state: leader="Node at 96d806c0-eb1b-fecf-7fe1-74cf4841c45b [Leader]"
2026-09-21T15:24:14.314Z [INFO]  server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b: added peer, starting replication: peer=server-ef4faf59-d6bd-fe17-7b86-f735a058d961
2026-09-21T15:24:14.314Z [INFO]  server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b: added peer, starting replication: peer=server-a1884d3f-f531-9ce4-fae8-e0369431707d
2026-09-21T15:24:14.314Z [INFO]  server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b: pipelining replication: peer="{Voter server-a1884d3f-f531-9ce4-fae8-e0369431707d a1884d3f-f531-9ce4-fae8-e0369431707d}"
2026-09-21T15:24:14.314Z [INFO]  server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b: pipelining replication: peer="{Voter server-ef4faf59-d6bd-fe17-7b86-f735a058d961 ef4faf59-d6bd-fe17-7b86-f735a058d961}"
2026-09-21T15:24:14.419Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:14.314740668 +0000 UTC m=+6.255910881 (1 nodes), 53.81144ms from start of poll, 54.19788ms from cluster start. Timeout at 2026-09-21 15:24:14.419742228 +0000 UTC m=+6.360912401, 105.00152ms after stability
2026-09-21T15:24:14.423Z [INFO]  server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b: aborting pipeline replication: peer="{Voter server-a1884d3f-f531-9ce4-fae8-e0369431707d a1884d3f-f531-9ce4-fae8-e0369431707d}"
2026-09-21T15:24:14.423Z [INFO]  server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b: aborting pipeline replication: peer="{Voter server-ef4faf59-d6bd-fe17-7b86-f735a058d961 ef4faf59-d6bd-fe17-7b86-f735a058d961}"
2026-09-21T15:24:14.423Z [INFO]  creating new snapshot: path=/tmp/raft887548835/snapshots/2-27-1790004254423.tmp
2026-09-21T15:24:14.424Z [INFO]  server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b: starting restore from snapshot: id=2-27-1790004254423 last-index=27 last-term=2 size-in-bytes=168
2026-09-21T15:24:14.424Z [INFO]  server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b: snapshot restore progress: id=2-27-1790004254423 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2026-09-21T15:24:14.424Z [INFO]  server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b: restored from snapshot: id=2-27-1790004254423 last-index=27 last-term=2 size-in-bytes=168
2026-09-21T15:24:14.424Z [INFO]  server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b Address:96d806c0-eb1b-fecf-7fe1-74cf4841c45b} {Suffrage:Voter ID:server-ef4faf59-d6bd-fe17-7b86-f735a058d961 Address:ef4faf59-d6bd-fe17-7b86-f735a058d961} {Suffrage:Voter ID:server-a1884d3f-f531-9ce4-fae8-e0369431707d Address:a1884d3f-f531-9ce4-fae8-e0369431707d}]"
2026-09-21T15:24:14.424Z [INFO]  creating new snapshot: path=/tmp/raft3421938742/snapshots/2-27-1790004254424.tmp
2026-09-21T15:24:14.424Z [INFO]  server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b: entering follower state: follower="Node at 96d806c0-eb1b-fecf-7fe1-74cf4841c45b [Follower]" leader-address= leader-id=
2026-09-21T15:24:14.425Z [INFO]  server-ef4faf59-d6bd-fe17-7b86-f735a058d961: starting restore from snapshot: id=2-27-1790004254424 last-index=27 last-term=2 size-in-bytes=168
2026-09-21T15:24:14.425Z [INFO]  server-ef4faf59-d6bd-fe17-7b86-f735a058d961: snapshot restore progress: id=2-27-1790004254424 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2026-09-21T15:24:14.425Z [INFO]  server-ef4faf59-d6bd-fe17-7b86-f735a058d961: restored from snapshot: id=2-27-1790004254424 last-index=27 last-term=2 size-in-bytes=168
2026-09-21T15:24:14.425Z [INFO]  server-ef4faf59-d6bd-fe17-7b86-f735a058d961: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b Address:96d806c0-eb1b-fecf-7fe1-74cf4841c45b} {Suffrage:Voter ID:server-ef4faf59-d6bd-fe17-7b86-f735a058d961 Address:ef4faf59-d6bd-fe17-7b86-f735a058d961} {Suffrage:Voter ID:server-a1884d3f-f531-9ce4-fae8-e0369431707d Address:a1884d3f-f531-9ce4-fae8-e0369431707d}]"
2026-09-21T15:24:14.425Z [INFO]  creating new snapshot: path=/tmp/raft2515871978/snapshots/2-27-1790004254425.tmp
2026-09-21T15:24:14.425Z [INFO]  server-ef4faf59-d6bd-fe17-7b86-f735a058d961: entering follower state: follower="Node at ef4faf59-d6bd-fe17-7b86-f735a058d961 [Follower]" leader-address= leader-id=
2026-09-21T15:24:14.426Z [INFO]  server-a1884d3f-f531-9ce4-fae8-e0369431707d: starting restore from snapshot: id=2-27-1790004254425 last-index=27 last-term=2 size-in-bytes=168
2026-09-21T15:24:14.426Z [INFO]  server-a1884d3f-f531-9ce4-fae8-e0369431707d: snapshot restore progress: id=2-27-1790004254425 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2026-09-21T15:24:14.426Z [INFO]  server-a1884d3f-f531-9ce4-fae8-e0369431707d: restored from snapshot: id=2-27-1790004254425 last-index=27 last-term=2 size-in-bytes=168
2026-09-21T15:24:14.426Z [INFO]  server-a1884d3f-f531-9ce4-fae8-e0369431707d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b Address:96d806c0-eb1b-fecf-7fe1-74cf4841c45b} {Suffrage:Voter ID:server-ef4faf59-d6bd-fe17-7b86-f735a058d961 Address:ef4faf59-d6bd-fe17-7b86-f735a058d961} {Suffrage:Voter ID:server-a1884d3f-f531-9ce4-fae8-e0369431707d Address:a1884d3f-f531-9ce4-fae8-e0369431707d}]"
2026-09-21T15:24:14.426Z [INFO]  server-a1884d3f-f531-9ce4-fae8-e0369431707d: entering follower state: follower="Node at a1884d3f-f531-9ce4-fae8-e0369431707d [Follower]" leader-address= leader-id=
2026-09-21T15:24:14.491Z [WARN]  server-a1884d3f-f531-9ce4-fae8-e0369431707d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:14.491Z [INFO]  server-a1884d3f-f531-9ce4-fae8-e0369431707d: entering candidate state: node="Node at a1884d3f-f531-9ce4-fae8-e0369431707d [Candidate]" term=3
2026-09-21T15:24:14.491Z [INFO]  server-a1884d3f-f531-9ce4-fae8-e0369431707d: election won: term=3 tally=2
2026-09-21T15:24:14.491Z [INFO]  server-a1884d3f-f531-9ce4-fae8-e0369431707d: entering leader state: leader="Node at a1884d3f-f531-9ce4-fae8-e0369431707d [Leader]"
2026-09-21T15:24:14.491Z [INFO]  server-a1884d3f-f531-9ce4-fae8-e0369431707d: added peer, starting replication: peer=server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b
2026-09-21T15:24:14.491Z [INFO]  server-a1884d3f-f531-9ce4-fae8-e0369431707d: added peer, starting replication: peer=server-ef4faf59-d6bd-fe17-7b86-f735a058d961
2026-09-21T15:24:14.492Z [INFO]  server-a1884d3f-f531-9ce4-fae8-e0369431707d: pipelining replication: peer="{Voter server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b 96d806c0-eb1b-fecf-7fe1-74cf4841c45b}"
2026-09-21T15:24:14.492Z [INFO]  server-a1884d3f-f531-9ce4-fae8-e0369431707d: pipelining replication: peer="{Voter server-ef4faf59-d6bd-fe17-7b86-f735a058d961 ef4faf59-d6bd-fe17-7b86-f735a058d961}"
2026-09-21T15:24:14.742Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:14.847Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:14.742208908 +0000 UTC m=+6.683379081 (1 nodes), 38.68µs from start of poll, 481.66608ms from cluster start. Timeout at 2026-09-21 15:24:14.847221868 +0000 UTC m=+6.788392041, 105.01296ms after stability
2026-09-21T15:24:14.847Z [INFO]  server-a1884d3f-f531-9ce4-fae8-e0369431707d: aborting pipeline replication: peer="{Voter server-ef4faf59-d6bd-fe17-7b86-f735a058d961 ef4faf59-d6bd-fe17-7b86-f735a058d961}"
2026-09-21T15:24:14.847Z [INFO]  server-a1884d3f-f531-9ce4-fae8-e0369431707d: aborting pipeline replication: peer="{Voter server-96d806c0-eb1b-fecf-7fe1-74cf4841c45b 96d806c0-eb1b-fecf-7fe1-74cf4841c45b}"
--- PASS: TestRaft_RecoverCluster (2.26s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.59s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (1.08s)
    --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.59s)
=== RUN   TestRaft_HasExistingState
    raft_test.go:258: Skipping unreliable test
--- SKIP: TestRaft_HasExistingState (0.00s)
=== RUN   TestRaft_SingleNode
2026-09-21T15:24:14.848Z [INFO]  server-1fe7e00c-ec39-0cf0-993a-9c52f660c632: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1fe7e00c-ec39-0cf0-993a-9c52f660c632 Address:1fe7e00c-ec39-0cf0-993a-9c52f660c632}]"
2026-09-21T15:24:14.848Z [INFO]  server-1fe7e00c-ec39-0cf0-993a-9c52f660c632: entering follower state: follower="Node at 1fe7e00c-ec39-0cf0-993a-9c52f660c632 [Follower]" leader-address= leader-id=
2026-09-21T15:24:14.949Z [WARN]  server-1fe7e00c-ec39-0cf0-993a-9c52f660c632: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:14.949Z [INFO]  server-1fe7e00c-ec39-0cf0-993a-9c52f660c632: entering candidate state: node="Node at 1fe7e00c-ec39-0cf0-993a-9c52f660c632 [Candidate]" term=2
2026-09-21T15:24:14.949Z [INFO]  server-1fe7e00c-ec39-0cf0-993a-9c52f660c632: election won: term=2 tally=1
2026-09-21T15:24:14.949Z [INFO]  server-1fe7e00c-ec39-0cf0-993a-9c52f660c632: entering leader state: leader="Node at 1fe7e00c-ec39-0cf0-993a-9c52f660c632 [Leader]"
--- PASS: TestRaft_SingleNode (0.10s)
=== RUN   TestRaft_TripleNode
2026-09-21T15:24:14.950Z [INFO]  server-d9decc00-e4ed-bb23-d861-2fe96cfd29c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d9decc00-e4ed-bb23-d861-2fe96cfd29c8 Address:d9decc00-e4ed-bb23-d861-2fe96cfd29c8} {Suffrage:Voter ID:server-8d85836f-2b95-2377-3878-891f8f49cb7e Address:8d85836f-2b95-2377-3878-891f8f49cb7e} {Suffrage:Voter ID:server-4c38d740-172d-b14a-9184-f4f7836616e6 Address:4c38d740-172d-b14a-9184-f4f7836616e6}]"
2026-09-21T15:24:14.950Z [INFO]  server-8d85836f-2b95-2377-3878-891f8f49cb7e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d9decc00-e4ed-bb23-d861-2fe96cfd29c8 Address:d9decc00-e4ed-bb23-d861-2fe96cfd29c8} {Suffrage:Voter ID:server-8d85836f-2b95-2377-3878-891f8f49cb7e Address:8d85836f-2b95-2377-3878-891f8f49cb7e} {Suffrage:Voter ID:server-4c38d740-172d-b14a-9184-f4f7836616e6 Address:4c38d740-172d-b14a-9184-f4f7836616e6}]"
2026-09-21T15:24:14.950Z [INFO]  server-d9decc00-e4ed-bb23-d861-2fe96cfd29c8: entering follower state: follower="Node at d9decc00-e4ed-bb23-d861-2fe96cfd29c8 [Follower]" leader-address= leader-id=
2026-09-21T15:24:14.950Z [INFO]  server-4c38d740-172d-b14a-9184-f4f7836616e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d9decc00-e4ed-bb23-d861-2fe96cfd29c8 Address:d9decc00-e4ed-bb23-d861-2fe96cfd29c8} {Suffrage:Voter ID:server-8d85836f-2b95-2377-3878-891f8f49cb7e Address:8d85836f-2b95-2377-3878-891f8f49cb7e} {Suffrage:Voter ID:server-4c38d740-172d-b14a-9184-f4f7836616e6 Address:4c38d740-172d-b14a-9184-f4f7836616e6}]"
2026-09-21T15:24:14.950Z [INFO]  server-8d85836f-2b95-2377-3878-891f8f49cb7e: entering follower state: follower="Node at 8d85836f-2b95-2377-3878-891f8f49cb7e [Follower]" leader-address= leader-id=
2026-09-21T15:24:14.950Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:14.950Z [INFO]  server-4c38d740-172d-b14a-9184-f4f7836616e6: entering follower state: follower="Node at 4c38d740-172d-b14a-9184-f4f7836616e6 [Follower]" leader-address= leader-id=
2026-09-21T15:24:15.024Z [WARN]  server-d9decc00-e4ed-bb23-d861-2fe96cfd29c8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:15.025Z [INFO]  server-d9decc00-e4ed-bb23-d861-2fe96cfd29c8: entering candidate state: node="Node at d9decc00-e4ed-bb23-d861-2fe96cfd29c8 [Candidate]" term=2
2026-09-21T15:24:15.025Z [INFO]  server-d9decc00-e4ed-bb23-d861-2fe96cfd29c8: election won: term=2 tally=2
2026-09-21T15:24:15.025Z [INFO]  server-d9decc00-e4ed-bb23-d861-2fe96cfd29c8: entering leader state: leader="Node at d9decc00-e4ed-bb23-d861-2fe96cfd29c8 [Leader]"
2026-09-21T15:24:15.025Z [INFO]  server-d9decc00-e4ed-bb23-d861-2fe96cfd29c8: added peer, starting replication: peer=server-8d85836f-2b95-2377-3878-891f8f49cb7e
2026-09-21T15:24:15.025Z [INFO]  server-d9decc00-e4ed-bb23-d861-2fe96cfd29c8: added peer, starting replication: peer=server-4c38d740-172d-b14a-9184-f4f7836616e6
2026-09-21T15:24:15.025Z [INFO]  server-d9decc00-e4ed-bb23-d861-2fe96cfd29c8: pipelining replication: peer="{Voter server-4c38d740-172d-b14a-9184-f4f7836616e6 4c38d740-172d-b14a-9184-f4f7836616e6}"
2026-09-21T15:24:15.025Z [INFO]  server-d9decc00-e4ed-bb23-d861-2fe96cfd29c8: pipelining replication: peer="{Voter server-8d85836f-2b95-2377-3878-891f8f49cb7e 8d85836f-2b95-2377-3878-891f8f49cb7e}"
2026-09-21T15:24:15.130Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:15.025261268 +0000 UTC m=+6.966431441 (2 nodes), 74.6856ms from start of poll, 75.07716ms from cluster start. Timeout at 2026-09-21 15:24:15.130265868 +0000 UTC m=+7.071436041, 105.0046ms after stability
2026-09-21T15:24:15.131Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:15.237Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:15.131062508 +0000 UTC m=+7.072232681 (1 nodes), 1.88µs from start of poll, 180.8784ms from cluster start. Timeout at 2026-09-21 15:24:15.236066268 +0000 UTC m=+7.177236441, 105.00376ms after stability
2026-09-21T15:24:15.253Z [INFO]  server-d9decc00-e4ed-bb23-d861-2fe96cfd29c8: aborting pipeline replication: peer="{Voter server-4c38d740-172d-b14a-9184-f4f7836616e6 4c38d740-172d-b14a-9184-f4f7836616e6}"
2026-09-21T15:24:15.253Z [INFO]  server-d9decc00-e4ed-bb23-d861-2fe96cfd29c8: aborting pipeline replication: peer="{Voter server-8d85836f-2b95-2377-3878-891f8f49cb7e 8d85836f-2b95-2377-3878-891f8f49cb7e}"
2026-09-21T15:24:15.753Z [ERROR] server-d9decc00-e4ed-bb23-d861-2fe96cfd29c8: failed to heartbeat to: peer=8d85836f-2b95-2377-3878-891f8f49cb7e backoff time=10ms error="command timed out"
--- PASS: TestRaft_TripleNode (0.80s)
=== RUN   TestRaft_LeaderFail
2026-09-21T15:24:15.754Z [INFO]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef38cc64-9558-79a8-d2bc-d6d014f85d01 Address:ef38cc64-9558-79a8-d2bc-d6d014f85d01} {Suffrage:Voter ID:server-e752b2cb-1ce6-28bb-41bb-27dc30db1390 Address:e752b2cb-1ce6-28bb-41bb-27dc30db1390} {Suffrage:Voter ID:server-b7596def-9d03-d668-b734-c67fa4883619 Address:b7596def-9d03-d668-b734-c67fa4883619}]"
2026-09-21T15:24:15.755Z [INFO]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: entering follower state: follower="Node at ef38cc64-9558-79a8-d2bc-d6d014f85d01 [Follower]" leader-address= leader-id=
2026-09-21T15:24:15.755Z [INFO]  server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef38cc64-9558-79a8-d2bc-d6d014f85d01 Address:ef38cc64-9558-79a8-d2bc-d6d014f85d01} {Suffrage:Voter ID:server-e752b2cb-1ce6-28bb-41bb-27dc30db1390 Address:e752b2cb-1ce6-28bb-41bb-27dc30db1390} {Suffrage:Voter ID:server-b7596def-9d03-d668-b734-c67fa4883619 Address:b7596def-9d03-d668-b734-c67fa4883619}]"
2026-09-21T15:24:15.756Z [INFO]  server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: entering follower state: follower="Node at e752b2cb-1ce6-28bb-41bb-27dc30db1390 [Follower]" leader-address= leader-id=
2026-09-21T15:24:15.756Z [INFO]  server-b7596def-9d03-d668-b734-c67fa4883619: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef38cc64-9558-79a8-d2bc-d6d014f85d01 Address:ef38cc64-9558-79a8-d2bc-d6d014f85d01} {Suffrage:Voter ID:server-e752b2cb-1ce6-28bb-41bb-27dc30db1390 Address:e752b2cb-1ce6-28bb-41bb-27dc30db1390} {Suffrage:Voter ID:server-b7596def-9d03-d668-b734-c67fa4883619 Address:b7596def-9d03-d668-b734-c67fa4883619}]"
2026-09-21T15:24:15.756Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:15.756Z [INFO]  server-b7596def-9d03-d668-b734-c67fa4883619: entering follower state: follower="Node at b7596def-9d03-d668-b734-c67fa4883619 [Follower]" leader-address= leader-id=
2026-09-21T15:24:15.809Z [WARN]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:15.809Z [INFO]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: entering candidate state: node="Node at ef38cc64-9558-79a8-d2bc-d6d014f85d01 [Candidate]" term=2
2026-09-21T15:24:15.809Z [INFO]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: election won: term=2 tally=2
2026-09-21T15:24:15.809Z [INFO]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: entering leader state: leader="Node at ef38cc64-9558-79a8-d2bc-d6d014f85d01 [Leader]"
2026-09-21T15:24:15.809Z [INFO]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: added peer, starting replication: peer=server-e752b2cb-1ce6-28bb-41bb-27dc30db1390
2026-09-21T15:24:15.809Z [INFO]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: added peer, starting replication: peer=server-b7596def-9d03-d668-b734-c67fa4883619
2026-09-21T15:24:15.809Z [INFO]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: pipelining replication: peer="{Voter server-b7596def-9d03-d668-b734-c67fa4883619 b7596def-9d03-d668-b734-c67fa4883619}"
2026-09-21T15:24:15.809Z [INFO]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: pipelining replication: peer="{Voter server-e752b2cb-1ce6-28bb-41bb-27dc30db1390 e752b2cb-1ce6-28bb-41bb-27dc30db1390}"
2026-09-21T15:24:15.915Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:15.809495748 +0000 UTC m=+7.750665921 (2 nodes), 53.34464ms from start of poll, 55.05576ms from cluster start. Timeout at 2026-09-21 15:24:15.914502388 +0000 UTC m=+7.855672561, 105.00664ms after stability
2026-09-21T15:24:15.915Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:16.021Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:15.915756268 +0000 UTC m=+7.856926481 (1 nodes), 1µs from start of poll, 161.31632ms from cluster start. Timeout at 2026-09-21 15:24:16.020760908 +0000 UTC m=+7.961931121, 105.00464ms after stability
    raft_test.go:378: [INFO] Disconnecting Node at ef38cc64-9558-79a8-d2bc-d6d014f85d01 [Leader]
2026-09-21T15:24:16.033Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to heartbeat to: peer=e752b2cb-1ce6-28bb-41bb-27dc30db1390 backoff time=10ms error="failed to connect to peer: e752b2cb-1ce6-28bb-41bb-27dc30db1390"
2026-09-21T15:24:16.033Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:16.037Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to pipeline appendEntries: peer="{Voter server-e752b2cb-1ce6-28bb-41bb-27dc30db1390 e752b2cb-1ce6-28bb-41bb-27dc30db1390}" error="append pipeline closed"
2026-09-21T15:24:16.037Z [INFO]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: aborting pipeline replication: peer="{Voter server-e752b2cb-1ce6-28bb-41bb-27dc30db1390 e752b2cb-1ce6-28bb-41bb-27dc30db1390}"
2026-09-21T15:24:16.038Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to pipeline appendEntries: peer="{Voter server-b7596def-9d03-d668-b734-c67fa4883619 b7596def-9d03-d668-b734-c67fa4883619}" error="append pipeline closed"
2026-09-21T15:24:16.038Z [INFO]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: aborting pipeline replication: peer="{Voter server-b7596def-9d03-d668-b734-c67fa4883619 b7596def-9d03-d668-b734-c67fa4883619}"
2026-09-21T15:24:16.039Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to heartbeat to: peer=b7596def-9d03-d668-b734-c67fa4883619 backoff time=10ms error="failed to connect to peer: b7596def-9d03-d668-b734-c67fa4883619"
2026-09-21T15:24:16.044Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to appendEntries to: peer="{Voter server-e752b2cb-1ce6-28bb-41bb-27dc30db1390 e752b2cb-1ce6-28bb-41bb-27dc30db1390}" error="failed to connect to peer: e752b2cb-1ce6-28bb-41bb-27dc30db1390"
2026-09-21T15:24:16.044Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to appendEntries to: peer="{Voter server-b7596def-9d03-d668-b734-c67fa4883619 b7596def-9d03-d668-b734-c67fa4883619}" error="failed to connect to peer: b7596def-9d03-d668-b734-c67fa4883619"
2026-09-21T15:24:16.055Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to heartbeat to: peer=e752b2cb-1ce6-28bb-41bb-27dc30db1390 backoff time=10ms error="failed to connect to peer: e752b2cb-1ce6-28bb-41bb-27dc30db1390"
2026-09-21T15:24:16.057Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to heartbeat to: peer=b7596def-9d03-d668-b734-c67fa4883619 backoff time=10ms error="failed to connect to peer: b7596def-9d03-d668-b734-c67fa4883619"
2026-09-21T15:24:16.063Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to appendEntries to: peer="{Voter server-e752b2cb-1ce6-28bb-41bb-27dc30db1390 e752b2cb-1ce6-28bb-41bb-27dc30db1390}" error="failed to connect to peer: e752b2cb-1ce6-28bb-41bb-27dc30db1390"
2026-09-21T15:24:16.063Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to appendEntries to: peer="{Voter server-b7596def-9d03-d668-b734-c67fa4883619 b7596def-9d03-d668-b734-c67fa4883619}" error="failed to connect to peer: b7596def-9d03-d668-b734-c67fa4883619"
2026-09-21T15:24:16.074Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to heartbeat to: peer=e752b2cb-1ce6-28bb-41bb-27dc30db1390 backoff time=10ms error="failed to connect to peer: e752b2cb-1ce6-28bb-41bb-27dc30db1390"
2026-09-21T15:24:16.074Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to heartbeat to: peer=b7596def-9d03-d668-b734-c67fa4883619 backoff time=10ms error="failed to connect to peer: b7596def-9d03-d668-b734-c67fa4883619"
2026-09-21T15:24:16.078Z [WARN]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to contact: server-id=server-e752b2cb-1ce6-28bb-41bb-27dc30db1390 time=50.09956ms
2026-09-21T15:24:16.082Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to appendEntries to: peer="{Voter server-b7596def-9d03-d668-b734-c67fa4883619 b7596def-9d03-d668-b734-c67fa4883619}" error="failed to connect to peer: b7596def-9d03-d668-b734-c67fa4883619"
2026-09-21T15:24:16.082Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to appendEntries to: peer="{Voter server-e752b2cb-1ce6-28bb-41bb-27dc30db1390 e752b2cb-1ce6-28bb-41bb-27dc30db1390}" error="failed to connect to peer: e752b2cb-1ce6-28bb-41bb-27dc30db1390"
2026-09-21T15:24:16.089Z [WARN]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to contact: server-id=server-e752b2cb-1ce6-28bb-41bb-27dc30db1390 time=60.85636ms
2026-09-21T15:24:16.089Z [WARN]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to contact: server-id=server-b7596def-9d03-d668-b734-c67fa4883619 time=58.6292ms
2026-09-21T15:24:16.089Z [WARN]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to contact quorum of nodes, stepping down
2026-09-21T15:24:16.089Z [INFO]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: entering follower state: follower="Node at ef38cc64-9558-79a8-d2bc-d6d014f85d01 [Follower]" leader-address= leader-id=
2026-09-21T15:24:16.090Z [WARN]  server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: heartbeat timeout reached, starting election: last-leader-addr=ef38cc64-9558-79a8-d2bc-d6d014f85d01 last-leader-id=server-ef38cc64-9558-79a8-d2bc-d6d014f85d01
2026-09-21T15:24:16.090Z [INFO]  server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: entering candidate state: node="Node at e752b2cb-1ce6-28bb-41bb-27dc30db1390 [Candidate]" term=3
2026-09-21T15:24:16.090Z [WARN]  server-b7596def-9d03-d668-b734-c67fa4883619: rejecting vote request since we have a leader: from=e752b2cb-1ce6-28bb-41bb-27dc30db1390 leader=ef38cc64-9558-79a8-d2bc-d6d014f85d01 leader-id=server-ef38cc64-9558-79a8-d2bc-d6d014f85d01
2026-09-21T15:24:16.090Z [ERROR] server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: failed to make requestVote RPC: target="{Voter server-ef38cc64-9558-79a8-d2bc-d6d014f85d01 ef38cc64-9558-79a8-d2bc-d6d014f85d01}" error="failed to connect to peer: ef38cc64-9558-79a8-d2bc-d6d014f85d01" term=3
2026-09-21T15:24:16.091Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to heartbeat to: peer=b7596def-9d03-d668-b734-c67fa4883619 backoff time=20ms error="failed to connect to peer: b7596def-9d03-d668-b734-c67fa4883619"
2026-09-21T15:24:16.095Z [WARN]  server-b7596def-9d03-d668-b734-c67fa4883619: heartbeat timeout reached, starting election: last-leader-addr=ef38cc64-9558-79a8-d2bc-d6d014f85d01 last-leader-id=server-ef38cc64-9558-79a8-d2bc-d6d014f85d01
2026-09-21T15:24:16.095Z [INFO]  server-b7596def-9d03-d668-b734-c67fa4883619: entering candidate state: node="Node at b7596def-9d03-d668-b734-c67fa4883619 [Candidate]" term=3
2026-09-21T15:24:16.095Z [INFO]  server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: duplicate requestVote for same term: term=3
2026-09-21T15:24:16.095Z [ERROR] server-b7596def-9d03-d668-b734-c67fa4883619: failed to make requestVote RPC: target="{Voter server-ef38cc64-9558-79a8-d2bc-d6d014f85d01 ef38cc64-9558-79a8-d2bc-d6d014f85d01}" error="failed to connect to peer: ef38cc64-9558-79a8-d2bc-d6d014f85d01" term=3
2026-09-21T15:24:16.109Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to appendEntries to: peer="{Voter server-b7596def-9d03-d668-b734-c67fa4883619 b7596def-9d03-d668-b734-c67fa4883619}" error="failed to connect to peer: b7596def-9d03-d668-b734-c67fa4883619"
2026-09-21T15:24:16.156Z [WARN]  server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: Election timeout reached, restarting election
2026-09-21T15:24:16.156Z [INFO]  server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: entering candidate state: node="Node at e752b2cb-1ce6-28bb-41bb-27dc30db1390 [Candidate]" term=4
2026-09-21T15:24:16.156Z [INFO]  server-b7596def-9d03-d668-b734-c67fa4883619: entering follower state: follower="Node at b7596def-9d03-d668-b734-c67fa4883619 [Follower]" leader-address= leader-id=
2026-09-21T15:24:16.156Z [ERROR] server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: failed to make requestVote RPC: target="{Voter server-ef38cc64-9558-79a8-d2bc-d6d014f85d01 ef38cc64-9558-79a8-d2bc-d6d014f85d01}" error="failed to connect to peer: ef38cc64-9558-79a8-d2bc-d6d014f85d01" term=4
2026-09-21T15:24:16.156Z [INFO]  server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: election won: term=4 tally=2
2026-09-21T15:24:16.156Z [INFO]  server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: entering leader state: leader="Node at e752b2cb-1ce6-28bb-41bb-27dc30db1390 [Leader]"
2026-09-21T15:24:16.156Z [INFO]  server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: added peer, starting replication: peer=server-ef38cc64-9558-79a8-d2bc-d6d014f85d01
2026-09-21T15:24:16.156Z [INFO]  server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: added peer, starting replication: peer=server-b7596def-9d03-d668-b734-c67fa4883619
2026-09-21T15:24:16.156Z [ERROR] server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: failed to appendEntries to: peer="{Voter server-ef38cc64-9558-79a8-d2bc-d6d014f85d01 ef38cc64-9558-79a8-d2bc-d6d014f85d01}" error="failed to connect to peer: ef38cc64-9558-79a8-d2bc-d6d014f85d01"
2026-09-21T15:24:16.156Z [INFO]  server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: pipelining replication: peer="{Voter server-b7596def-9d03-d668-b734-c67fa4883619 b7596def-9d03-d668-b734-c67fa4883619}"
2026-09-21T15:24:16.164Z [ERROR] server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: failed to heartbeat to: peer=ef38cc64-9558-79a8-d2bc-d6d014f85d01 backoff time=10ms error="failed to connect to peer: ef38cc64-9558-79a8-d2bc-d6d014f85d01"
2026-09-21T15:24:16.173Z [WARN]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:16.173Z [INFO]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: entering candidate state: node="Node at ef38cc64-9558-79a8-d2bc-d6d014f85d01 [Candidate]" term=3
2026-09-21T15:24:16.173Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to make requestVote RPC: target="{Voter server-b7596def-9d03-d668-b734-c67fa4883619 b7596def-9d03-d668-b734-c67fa4883619}" error="failed to connect to peer: b7596def-9d03-d668-b734-c67fa4883619" term=3
2026-09-21T15:24:16.173Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to make requestVote RPC: target="{Voter server-e752b2cb-1ce6-28bb-41bb-27dc30db1390 e752b2cb-1ce6-28bb-41bb-27dc30db1390}" error="failed to connect to peer: e752b2cb-1ce6-28bb-41bb-27dc30db1390" term=3
2026-09-21T15:24:16.176Z [ERROR] server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: failed to appendEntries to: peer="{Voter server-ef38cc64-9558-79a8-d2bc-d6d014f85d01 ef38cc64-9558-79a8-d2bc-d6d014f85d01}" error="failed to connect to peer: ef38cc64-9558-79a8-d2bc-d6d014f85d01"
2026-09-21T15:24:16.183Z [ERROR] server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: failed to heartbeat to: peer=ef38cc64-9558-79a8-d2bc-d6d014f85d01 backoff time=10ms error="failed to connect to peer: ef38cc64-9558-79a8-d2bc-d6d014f85d01"
2026-09-21T15:24:16.195Z [ERROR] server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: failed to appendEntries to: peer="{Voter server-ef38cc64-9558-79a8-d2bc-d6d014f85d01 ef38cc64-9558-79a8-d2bc-d6d014f85d01}" error="failed to connect to peer: ef38cc64-9558-79a8-d2bc-d6d014f85d01"
2026-09-21T15:24:16.204Z [ERROR] server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: failed to heartbeat to: peer=ef38cc64-9558-79a8-d2bc-d6d014f85d01 backoff time=10ms error="failed to connect to peer: ef38cc64-9558-79a8-d2bc-d6d014f85d01"
2026-09-21T15:24:16.207Z [WARN]  server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: failed to contact: server-id=server-ef38cc64-9558-79a8-d2bc-d6d014f85d01 time=50.59076ms
2026-09-21T15:24:16.222Z [ERROR] server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: failed to heartbeat to: peer=ef38cc64-9558-79a8-d2bc-d6d014f85d01 backoff time=20ms error="failed to connect to peer: ef38cc64-9558-79a8-d2bc-d6d014f85d01"
2026-09-21T15:24:16.223Z [ERROR] server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: failed to appendEntries to: peer="{Voter server-ef38cc64-9558-79a8-d2bc-d6d014f85d01 ef38cc64-9558-79a8-d2bc-d6d014f85d01}" error="failed to connect to peer: ef38cc64-9558-79a8-d2bc-d6d014f85d01"
2026-09-21T15:24:16.252Z [ERROR] server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: failed to heartbeat to: peer=ef38cc64-9558-79a8-d2bc-d6d014f85d01 backoff time=25ms error="failed to connect to peer: ef38cc64-9558-79a8-d2bc-d6d014f85d01"
2026-09-21T15:24:16.254Z [WARN]  server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: failed to contact: server-id=server-ef38cc64-9558-79a8-d2bc-d6d014f85d01 time=98.08648ms
2026-09-21T15:24:16.255Z [WARN]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: Election timeout reached, restarting election
2026-09-21T15:24:16.255Z [INFO]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: entering candidate state: node="Node at ef38cc64-9558-79a8-d2bc-d6d014f85d01 [Candidate]" term=4
2026-09-21T15:24:16.256Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to make requestVote RPC: target="{Voter server-b7596def-9d03-d668-b734-c67fa4883619 b7596def-9d03-d668-b734-c67fa4883619}" error="failed to connect to peer: b7596def-9d03-d668-b734-c67fa4883619" term=4
2026-09-21T15:24:16.256Z [ERROR] server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: failed to make requestVote RPC: target="{Voter server-e752b2cb-1ce6-28bb-41bb-27dc30db1390 e752b2cb-1ce6-28bb-41bb-27dc30db1390}" error="failed to connect to peer: e752b2cb-1ce6-28bb-41bb-27dc30db1390" term=4
2026-09-21T15:24:16.274Z [ERROR] server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: failed to appendEntries to: peer="{Voter server-ef38cc64-9558-79a8-d2bc-d6d014f85d01 ef38cc64-9558-79a8-d2bc-d6d014f85d01}" error="failed to connect to peer: ef38cc64-9558-79a8-d2bc-d6d014f85d01"
2026-09-21T15:24:16.278Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:16.173115428 +0000 UTC m=+8.114285601 (1 nodes), 139.17624ms from start of poll, 418.67544ms from cluster start. Timeout at 2026-09-21 15:24:16.278119228 +0000 UTC m=+8.219289401, 105.0038ms after stability
    raft_test.go:413: [INFO] Reconnecting Node at ef38cc64-9558-79a8-d2bc-d6d014f85d01 [Candidate]
2026-09-21T15:24:16.287Z [INFO]  server-ef38cc64-9558-79a8-d2bc-d6d014f85d01: entering follower state: follower="Node at ef38cc64-9558-79a8-d2bc-d6d014f85d01 [Follower]" leader-address=e752b2cb-1ce6-28bb-41bb-27dc30db1390 leader-id=server-e752b2cb-1ce6-28bb-41bb-27dc30db1390
2026-09-21T15:24:16.359Z [INFO]  server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: pipelining replication: peer="{Voter server-ef38cc64-9558-79a8-d2bc-d6d014f85d01 ef38cc64-9558-79a8-d2bc-d6d014f85d01}"
2026-09-21T15:24:16.361Z [INFO]  server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: aborting pipeline replication: peer="{Voter server-b7596def-9d03-d668-b734-c67fa4883619 b7596def-9d03-d668-b734-c67fa4883619}"
2026-09-21T15:24:16.361Z [INFO]  server-e752b2cb-1ce6-28bb-41bb-27dc30db1390: aborting pipeline replication: peer="{Voter server-ef38cc64-9558-79a8-d2bc-d6d014f85d01 ef38cc64-9558-79a8-d2bc-d6d014f85d01}"
--- PASS: TestRaft_LeaderFail (0.61s)
=== RUN   TestRaft_BehindFollower
2026-09-21T15:24:16.361Z [INFO]  server-a4552266-ff2f-2734-535c-93d78b2a7ba6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4552266-ff2f-2734-535c-93d78b2a7ba6 Address:a4552266-ff2f-2734-535c-93d78b2a7ba6} {Suffrage:Voter ID:server-aaa1370b-31cf-285b-b231-8253ef1a3d14 Address:aaa1370b-31cf-285b-b231-8253ef1a3d14} {Suffrage:Voter ID:server-fff8322d-7769-f10f-2d57-f13eaf24fc49 Address:fff8322d-7769-f10f-2d57-f13eaf24fc49}]"
2026-09-21T15:24:16.362Z [INFO]  server-aaa1370b-31cf-285b-b231-8253ef1a3d14: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4552266-ff2f-2734-535c-93d78b2a7ba6 Address:a4552266-ff2f-2734-535c-93d78b2a7ba6} {Suffrage:Voter ID:server-aaa1370b-31cf-285b-b231-8253ef1a3d14 Address:aaa1370b-31cf-285b-b231-8253ef1a3d14} {Suffrage:Voter ID:server-fff8322d-7769-f10f-2d57-f13eaf24fc49 Address:fff8322d-7769-f10f-2d57-f13eaf24fc49}]"
2026-09-21T15:24:16.362Z [INFO]  server-a4552266-ff2f-2734-535c-93d78b2a7ba6: entering follower state: follower="Node at a4552266-ff2f-2734-535c-93d78b2a7ba6 [Follower]" leader-address= leader-id=
2026-09-21T15:24:16.362Z [INFO]  server-aaa1370b-31cf-285b-b231-8253ef1a3d14: entering follower state: follower="Node at aaa1370b-31cf-285b-b231-8253ef1a3d14 [Follower]" leader-address= leader-id=
2026-09-21T15:24:16.362Z [INFO]  server-fff8322d-7769-f10f-2d57-f13eaf24fc49: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4552266-ff2f-2734-535c-93d78b2a7ba6 Address:a4552266-ff2f-2734-535c-93d78b2a7ba6} {Suffrage:Voter ID:server-aaa1370b-31cf-285b-b231-8253ef1a3d14 Address:aaa1370b-31cf-285b-b231-8253ef1a3d14} {Suffrage:Voter ID:server-fff8322d-7769-f10f-2d57-f13eaf24fc49 Address:fff8322d-7769-f10f-2d57-f13eaf24fc49}]"
2026-09-21T15:24:16.362Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:16.362Z [INFO]  server-fff8322d-7769-f10f-2d57-f13eaf24fc49: entering follower state: follower="Node at fff8322d-7769-f10f-2d57-f13eaf24fc49 [Follower]" leader-address= leader-id=
2026-09-21T15:24:16.422Z [WARN]  server-fff8322d-7769-f10f-2d57-f13eaf24fc49: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:16.422Z [INFO]  server-fff8322d-7769-f10f-2d57-f13eaf24fc49: entering candidate state: node="Node at fff8322d-7769-f10f-2d57-f13eaf24fc49 [Candidate]" term=2
2026-09-21T15:24:16.422Z [INFO]  server-fff8322d-7769-f10f-2d57-f13eaf24fc49: election won: term=2 tally=2
2026-09-21T15:24:16.422Z [INFO]  server-fff8322d-7769-f10f-2d57-f13eaf24fc49: entering leader state: leader="Node at fff8322d-7769-f10f-2d57-f13eaf24fc49 [Leader]"
2026-09-21T15:24:16.422Z [INFO]  server-fff8322d-7769-f10f-2d57-f13eaf24fc49: added peer, starting replication: peer=server-a4552266-ff2f-2734-535c-93d78b2a7ba6
2026-09-21T15:24:16.422Z [INFO]  server-fff8322d-7769-f10f-2d57-f13eaf24fc49: added peer, starting replication: peer=server-aaa1370b-31cf-285b-b231-8253ef1a3d14
2026-09-21T15:24:16.422Z [INFO]  server-fff8322d-7769-f10f-2d57-f13eaf24fc49: pipelining replication: peer="{Voter server-aaa1370b-31cf-285b-b231-8253ef1a3d14 aaa1370b-31cf-285b-b231-8253ef1a3d14}"
2026-09-21T15:24:16.422Z [INFO]  server-fff8322d-7769-f10f-2d57-f13eaf24fc49: pipelining replication: peer="{Voter server-a4552266-ff2f-2734-535c-93d78b2a7ba6 a4552266-ff2f-2734-535c-93d78b2a7ba6}"
2026-09-21T15:24:16.528Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:16.422715388 +0000 UTC m=+8.363885561 (1 nodes), 60.53916ms from start of poll, 60.93104ms from cluster start. Timeout at 2026-09-21 15:24:16.527717748 +0000 UTC m=+8.468887921, 105.00236ms after stability
2026-09-21T15:24:16.528Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:16.634Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:16.528824068 +0000 UTC m=+8.469994241 (2 nodes), 1.24µs from start of poll, 167.03972ms from cluster start. Timeout at 2026-09-21 15:24:16.633830428 +0000 UTC m=+8.575000601, 105.00636ms after stability
2026-09-21T15:24:16.634Z [ERROR] server-fff8322d-7769-f10f-2d57-f13eaf24fc49: failed to pipeline appendEntries: peer="{Voter server-a4552266-ff2f-2734-535c-93d78b2a7ba6 a4552266-ff2f-2734-535c-93d78b2a7ba6}" error="append pipeline closed"
2026-09-21T15:24:16.634Z [INFO]  server-fff8322d-7769-f10f-2d57-f13eaf24fc49: aborting pipeline replication: peer="{Voter server-a4552266-ff2f-2734-535c-93d78b2a7ba6 a4552266-ff2f-2734-535c-93d78b2a7ba6}"
2026-09-21T15:24:16.634Z [ERROR] server-fff8322d-7769-f10f-2d57-f13eaf24fc49: failed to appendEntries to: peer="{Voter server-a4552266-ff2f-2734-535c-93d78b2a7ba6 a4552266-ff2f-2734-535c-93d78b2a7ba6}" error="failed to connect to peer: a4552266-ff2f-2734-535c-93d78b2a7ba6"
2026-09-21T15:24:16.636Z [ERROR] server-fff8322d-7769-f10f-2d57-f13eaf24fc49: failed to heartbeat to: peer=a4552266-ff2f-2734-535c-93d78b2a7ba6 backoff time=10ms error="failed to connect to peer: a4552266-ff2f-2734-535c-93d78b2a7ba6"
    raft_test.go:459: [INFO] Finished apply without behind follower
2026-09-21T15:24:16.645Z [INFO]  server-fff8322d-7769-f10f-2d57-f13eaf24fc49: pipelining replication: peer="{Voter server-a4552266-ff2f-2734-535c-93d78b2a7ba6 a4552266-ff2f-2734-535c-93d78b2a7ba6}"
2026-09-21T15:24:16.656Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:16.762Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:16.656751908 +0000 UTC m=+8.597922081 (1 nodes), 1.28µs from start of poll, 294.96756ms from cluster start. Timeout at 2026-09-21 15:24:16.761760548 +0000 UTC m=+8.702930761, 105.00868ms after stability
2026-09-21T15:24:16.762Z [INFO]  server-fff8322d-7769-f10f-2d57-f13eaf24fc49: aborting pipeline replication: peer="{Voter server-aaa1370b-31cf-285b-b231-8253ef1a3d14 aaa1370b-31cf-285b-b231-8253ef1a3d14}"
2026-09-21T15:24:16.762Z [INFO]  server-fff8322d-7769-f10f-2d57-f13eaf24fc49: aborting pipeline replication: peer="{Voter server-a4552266-ff2f-2734-535c-93d78b2a7ba6 a4552266-ff2f-2734-535c-93d78b2a7ba6}"
--- PASS: TestRaft_BehindFollower (0.40s)
=== RUN   TestRaft_ApplyNonLeader
2026-09-21T15:24:16.763Z [INFO]  server-2640b7e4-4354-2d3c-1bde-a68beaf08ee0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2640b7e4-4354-2d3c-1bde-a68beaf08ee0 Address:2640b7e4-4354-2d3c-1bde-a68beaf08ee0} {Suffrage:Voter ID:server-829c3682-53cd-f943-7a3a-890ba5064175 Address:829c3682-53cd-f943-7a3a-890ba5064175} {Suffrage:Voter ID:server-e3d98bcc-7004-3bc6-4912-72a4065cc4a6 Address:e3d98bcc-7004-3bc6-4912-72a4065cc4a6}]"
2026-09-21T15:24:16.763Z [INFO]  server-2640b7e4-4354-2d3c-1bde-a68beaf08ee0: entering follower state: follower="Node at 2640b7e4-4354-2d3c-1bde-a68beaf08ee0 [Follower]" leader-address= leader-id=
2026-09-21T15:24:16.763Z [INFO]  server-829c3682-53cd-f943-7a3a-890ba5064175: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2640b7e4-4354-2d3c-1bde-a68beaf08ee0 Address:2640b7e4-4354-2d3c-1bde-a68beaf08ee0} {Suffrage:Voter ID:server-829c3682-53cd-f943-7a3a-890ba5064175 Address:829c3682-53cd-f943-7a3a-890ba5064175} {Suffrage:Voter ID:server-e3d98bcc-7004-3bc6-4912-72a4065cc4a6 Address:e3d98bcc-7004-3bc6-4912-72a4065cc4a6}]"
2026-09-21T15:24:16.763Z [INFO]  server-829c3682-53cd-f943-7a3a-890ba5064175: entering follower state: follower="Node at 829c3682-53cd-f943-7a3a-890ba5064175 [Follower]" leader-address= leader-id=
2026-09-21T15:24:16.763Z [INFO]  server-e3d98bcc-7004-3bc6-4912-72a4065cc4a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2640b7e4-4354-2d3c-1bde-a68beaf08ee0 Address:2640b7e4-4354-2d3c-1bde-a68beaf08ee0} {Suffrage:Voter ID:server-829c3682-53cd-f943-7a3a-890ba5064175 Address:829c3682-53cd-f943-7a3a-890ba5064175} {Suffrage:Voter ID:server-e3d98bcc-7004-3bc6-4912-72a4065cc4a6 Address:e3d98bcc-7004-3bc6-4912-72a4065cc4a6}]"
2026-09-21T15:24:16.763Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:16.763Z [INFO]  server-e3d98bcc-7004-3bc6-4912-72a4065cc4a6: entering follower state: follower="Node at e3d98bcc-7004-3bc6-4912-72a4065cc4a6 [Follower]" leader-address= leader-id=
2026-09-21T15:24:16.822Z [WARN]  server-2640b7e4-4354-2d3c-1bde-a68beaf08ee0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:16.822Z [INFO]  server-2640b7e4-4354-2d3c-1bde-a68beaf08ee0: entering candidate state: node="Node at 2640b7e4-4354-2d3c-1bde-a68beaf08ee0 [Candidate]" term=2
2026-09-21T15:24:16.822Z [INFO]  server-2640b7e4-4354-2d3c-1bde-a68beaf08ee0: election won: term=2 tally=2
2026-09-21T15:24:16.822Z [INFO]  server-2640b7e4-4354-2d3c-1bde-a68beaf08ee0: entering leader state: leader="Node at 2640b7e4-4354-2d3c-1bde-a68beaf08ee0 [Leader]"
2026-09-21T15:24:16.823Z [INFO]  server-2640b7e4-4354-2d3c-1bde-a68beaf08ee0: added peer, starting replication: peer=server-829c3682-53cd-f943-7a3a-890ba5064175
2026-09-21T15:24:16.823Z [INFO]  server-2640b7e4-4354-2d3c-1bde-a68beaf08ee0: added peer, starting replication: peer=server-e3d98bcc-7004-3bc6-4912-72a4065cc4a6
2026-09-21T15:24:16.823Z [INFO]  server-2640b7e4-4354-2d3c-1bde-a68beaf08ee0: pipelining replication: peer="{Voter server-e3d98bcc-7004-3bc6-4912-72a4065cc4a6 e3d98bcc-7004-3bc6-4912-72a4065cc4a6}"
2026-09-21T15:24:16.823Z [INFO]  server-2640b7e4-4354-2d3c-1bde-a68beaf08ee0: pipelining replication: peer="{Voter server-829c3682-53cd-f943-7a3a-890ba5064175 829c3682-53cd-f943-7a3a-890ba5064175}"
2026-09-21T15:24:16.928Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:16.823010348 +0000 UTC m=+8.764180521 (1 nodes), 59.34196ms from start of poll, 59.8154ms from cluster start. Timeout at 2026-09-21 15:24:16.928013268 +0000 UTC m=+8.869183441, 105.00292ms after stability
2026-09-21T15:24:16.928Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:17.034Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:16.928967428 +0000 UTC m=+8.870137601 (2 nodes), 1.52µs from start of poll, 165.77248ms from cluster start. Timeout at 2026-09-21 15:24:17.033976588 +0000 UTC m=+8.975146761, 105.00916ms after stability
2026-09-21T15:24:17.034Z [INFO]  server-2640b7e4-4354-2d3c-1bde-a68beaf08ee0: aborting pipeline replication: peer="{Voter server-e3d98bcc-7004-3bc6-4912-72a4065cc4a6 e3d98bcc-7004-3bc6-4912-72a4065cc4a6}"
2026-09-21T15:24:17.034Z [INFO]  server-2640b7e4-4354-2d3c-1bde-a68beaf08ee0: aborting pipeline replication: peer="{Voter server-829c3682-53cd-f943-7a3a-890ba5064175 829c3682-53cd-f943-7a3a-890ba5064175}"
--- PASS: TestRaft_ApplyNonLeader (0.27s)
=== RUN   TestRaft_ApplyConcurrent
2026-09-21T15:24:17.035Z [INFO]  server-a0a9a173-4e30-78a4-48cd-a253c49f0cbd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0a9a173-4e30-78a4-48cd-a253c49f0cbd Address:a0a9a173-4e30-78a4-48cd-a253c49f0cbd} {Suffrage:Voter ID:server-69207442-cc1d-d25f-4680-9453f199c7cb Address:69207442-cc1d-d25f-4680-9453f199c7cb} {Suffrage:Voter ID:server-4b6d15e3-42d9-2321-c0c2-eeb92af3dbc3 Address:4b6d15e3-42d9-2321-c0c2-eeb92af3dbc3}]"
2026-09-21T15:24:17.036Z [INFO]  server-69207442-cc1d-d25f-4680-9453f199c7cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0a9a173-4e30-78a4-48cd-a253c49f0cbd Address:a0a9a173-4e30-78a4-48cd-a253c49f0cbd} {Suffrage:Voter ID:server-69207442-cc1d-d25f-4680-9453f199c7cb Address:69207442-cc1d-d25f-4680-9453f199c7cb} {Suffrage:Voter ID:server-4b6d15e3-42d9-2321-c0c2-eeb92af3dbc3 Address:4b6d15e3-42d9-2321-c0c2-eeb92af3dbc3}]"
2026-09-21T15:24:17.036Z [INFO]  server-a0a9a173-4e30-78a4-48cd-a253c49f0cbd: entering follower state: follower="Node at a0a9a173-4e30-78a4-48cd-a253c49f0cbd [Follower]" leader-address= leader-id=
2026-09-21T15:24:17.036Z [INFO]  server-69207442-cc1d-d25f-4680-9453f199c7cb: entering follower state: follower="Node at 69207442-cc1d-d25f-4680-9453f199c7cb [Follower]" leader-address= leader-id=
2026-09-21T15:24:17.036Z [INFO]  server-4b6d15e3-42d9-2321-c0c2-eeb92af3dbc3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0a9a173-4e30-78a4-48cd-a253c49f0cbd Address:a0a9a173-4e30-78a4-48cd-a253c49f0cbd} {Suffrage:Voter ID:server-69207442-cc1d-d25f-4680-9453f199c7cb Address:69207442-cc1d-d25f-4680-9453f199c7cb} {Suffrage:Voter ID:server-4b6d15e3-42d9-2321-c0c2-eeb92af3dbc3 Address:4b6d15e3-42d9-2321-c0c2-eeb92af3dbc3}]"
2026-09-21T15:24:17.036Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:17.036Z [INFO]  server-4b6d15e3-42d9-2321-c0c2-eeb92af3dbc3: entering follower state: follower="Node at 4b6d15e3-42d9-2321-c0c2-eeb92af3dbc3 [Follower]" leader-address= leader-id=
2026-09-21T15:24:17.154Z [WARN]  server-a0a9a173-4e30-78a4-48cd-a253c49f0cbd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:17.154Z [INFO]  server-a0a9a173-4e30-78a4-48cd-a253c49f0cbd: entering candidate state: node="Node at a0a9a173-4e30-78a4-48cd-a253c49f0cbd [Candidate]" term=2
2026-09-21T15:24:17.154Z [INFO]  server-a0a9a173-4e30-78a4-48cd-a253c49f0cbd: election won: term=2 tally=2
2026-09-21T15:24:17.154Z [INFO]  server-a0a9a173-4e30-78a4-48cd-a253c49f0cbd: entering leader state: leader="Node at a0a9a173-4e30-78a4-48cd-a253c49f0cbd [Leader]"
2026-09-21T15:24:17.154Z [INFO]  server-a0a9a173-4e30-78a4-48cd-a253c49f0cbd: added peer, starting replication: peer=server-69207442-cc1d-d25f-4680-9453f199c7cb
2026-09-21T15:24:17.154Z [INFO]  server-a0a9a173-4e30-78a4-48cd-a253c49f0cbd: added peer, starting replication: peer=server-4b6d15e3-42d9-2321-c0c2-eeb92af3dbc3
2026-09-21T15:24:17.154Z [INFO]  server-a0a9a173-4e30-78a4-48cd-a253c49f0cbd: pipelining replication: peer="{Voter server-4b6d15e3-42d9-2321-c0c2-eeb92af3dbc3 4b6d15e3-42d9-2321-c0c2-eeb92af3dbc3}"
2026-09-21T15:24:17.154Z [INFO]  server-a0a9a173-4e30-78a4-48cd-a253c49f0cbd: pipelining replication: peer="{Voter server-69207442-cc1d-d25f-4680-9453f199c7cb 69207442-cc1d-d25f-4680-9453f199c7cb}"
2026-09-21T15:24:17.360Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:17.154534668 +0000 UTC m=+9.095704841 (1 nodes), 118.33968ms from start of poll, 118.77244ms from cluster start. Timeout at 2026-09-21 15:24:17.359537268 +0000 UTC m=+9.300707441, 205.0026ms after stability
2026-09-21T15:24:17.375Z [INFO]  server-a0a9a173-4e30-78a4-48cd-a253c49f0cbd: aborting pipeline replication: peer="{Voter server-4b6d15e3-42d9-2321-c0c2-eeb92af3dbc3 4b6d15e3-42d9-2321-c0c2-eeb92af3dbc3}"
2026-09-21T15:24:17.375Z [INFO]  server-a0a9a173-4e30-78a4-48cd-a253c49f0cbd: aborting pipeline replication: peer="{Voter server-69207442-cc1d-d25f-4680-9453f199c7cb 69207442-cc1d-d25f-4680-9453f199c7cb}"
--- PASS: TestRaft_ApplyConcurrent (0.34s)
=== RUN   TestRaft_ApplyConcurrent_Timeout
2026-09-21T15:24:17.375Z [INFO]  server-517473cd-d342-e073-7f4d-97d0e188d7f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-517473cd-d342-e073-7f4d-97d0e188d7f4 Address:517473cd-d342-e073-7f4d-97d0e188d7f4}]"
2026-09-21T15:24:17.375Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:17.375Z [INFO]  server-517473cd-d342-e073-7f4d-97d0e188d7f4: entering follower state: follower="Node at 517473cd-d342-e073-7f4d-97d0e188d7f4 [Follower]" leader-address= leader-id=
2026-09-21T15:24:17.529Z [WARN]  server-517473cd-d342-e073-7f4d-97d0e188d7f4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:17.529Z [INFO]  server-517473cd-d342-e073-7f4d-97d0e188d7f4: entering candidate state: node="Node at 517473cd-d342-e073-7f4d-97d0e188d7f4 [Candidate]" term=2
2026-09-21T15:24:17.529Z [INFO]  server-517473cd-d342-e073-7f4d-97d0e188d7f4: election won: term=2 tally=1
2026-09-21T15:24:17.529Z [INFO]  server-517473cd-d342-e073-7f4d-97d0e188d7f4: entering leader state: leader="Node at 517473cd-d342-e073-7f4d-97d0e188d7f4 [Leader]"
2026-09-21T15:24:17.730Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:17.529756148 +0000 UTC m=+9.470926321 (1 nodes), 153.80156ms from start of poll, 153.99916ms from cluster start. Timeout at 2026-09-21 15:24:17.730759228 +0000 UTC m=+9.671929401, 201.00308ms after stability
--- PASS: TestRaft_ApplyConcurrent_Timeout (0.36s)
=== RUN   TestRaft_JoinNode
2026-09-21T15:24:17.732Z [INFO]  server-9e866825-25ab-d538-d7d9-2f3f28354f5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e866825-25ab-d538-d7d9-2f3f28354f5d Address:9e866825-25ab-d538-d7d9-2f3f28354f5d} {Suffrage:Voter ID:server-8ef30ebd-4eb1-304b-a204-7a4c9cd530e0 Address:8ef30ebd-4eb1-304b-a204-7a4c9cd530e0}]"
2026-09-21T15:24:17.732Z [INFO]  server-8ef30ebd-4eb1-304b-a204-7a4c9cd530e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9e866825-25ab-d538-d7d9-2f3f28354f5d Address:9e866825-25ab-d538-d7d9-2f3f28354f5d} {Suffrage:Voter ID:server-8ef30ebd-4eb1-304b-a204-7a4c9cd530e0 Address:8ef30ebd-4eb1-304b-a204-7a4c9cd530e0}]"
2026-09-21T15:24:17.732Z [INFO]  server-9e866825-25ab-d538-d7d9-2f3f28354f5d: entering follower state: follower="Node at 9e866825-25ab-d538-d7d9-2f3f28354f5d [Follower]" leader-address= leader-id=
2026-09-21T15:24:17.732Z [INFO]  server-8ef30ebd-4eb1-304b-a204-7a4c9cd530e0: entering follower state: follower="Node at 8ef30ebd-4eb1-304b-a204-7a4c9cd530e0 [Follower]" leader-address= leader-id=
2026-09-21T15:24:17.732Z [INFO]  server-f41d018b-5213-3ac1-99d6-6a1bc7953714: initial configuration: index=0 servers=[]
2026-09-21T15:24:17.732Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:17.732Z [INFO]  server-f41d018b-5213-3ac1-99d6-6a1bc7953714: entering follower state: follower="Node at f41d018b-5213-3ac1-99d6-6a1bc7953714 [Follower]" leader-address= leader-id=
2026-09-21T15:24:17.795Z [WARN]  server-8ef30ebd-4eb1-304b-a204-7a4c9cd530e0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:17.795Z [INFO]  server-8ef30ebd-4eb1-304b-a204-7a4c9cd530e0: entering candidate state: node="Node at 8ef30ebd-4eb1-304b-a204-7a4c9cd530e0 [Candidate]" term=2
2026-09-21T15:24:17.795Z [INFO]  server-8ef30ebd-4eb1-304b-a204-7a4c9cd530e0: election won: term=2 tally=2
2026-09-21T15:24:17.795Z [INFO]  server-8ef30ebd-4eb1-304b-a204-7a4c9cd530e0: entering leader state: leader="Node at 8ef30ebd-4eb1-304b-a204-7a4c9cd530e0 [Leader]"
2026-09-21T15:24:17.795Z [INFO]  server-8ef30ebd-4eb1-304b-a204-7a4c9cd530e0: added peer, starting replication: peer=server-9e866825-25ab-d538-d7d9-2f3f28354f5d
2026-09-21T15:24:17.796Z [INFO]  server-8ef30ebd-4eb1-304b-a204-7a4c9cd530e0: pipelining replication: peer="{Voter server-9e866825-25ab-d538-d7d9-2f3f28354f5d 9e866825-25ab-d538-d7d9-2f3f28354f5d}"
2026-09-21T15:24:17.817Z [WARN]  server-f41d018b-5213-3ac1-99d6-6a1bc7953714: no known peers, aborting election
2026-09-21T15:24:17.901Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:17.796101108 +0000 UTC m=+9.737273241 (1 nodes), 63.75592ms from start of poll, 64.202ms from cluster start. Timeout at 2026-09-21 15:24:17.901107748 +0000 UTC m=+9.842277921, 105.00468ms after stability
2026-09-21T15:24:17.901Z [INFO]  server-8ef30ebd-4eb1-304b-a204-7a4c9cd530e0: updating configuration: command=AddVoter server-id=server-f41d018b-5213-3ac1-99d6-6a1bc7953714 server-addr=f41d018b-5213-3ac1-99d6-6a1bc7953714 servers="[{Suffrage:Voter ID:server-9e866825-25ab-d538-d7d9-2f3f28354f5d Address:9e866825-25ab-d538-d7d9-2f3f28354f5d} {Suffrage:Voter ID:server-8ef30ebd-4eb1-304b-a204-7a4c9cd530e0 Address:8ef30ebd-4eb1-304b-a204-7a4c9cd530e0} {Suffrage:Voter ID:server-f41d018b-5213-3ac1-99d6-6a1bc7953714 Address:f41d018b-5213-3ac1-99d6-6a1bc7953714}]"
2026-09-21T15:24:17.901Z [INFO]  server-8ef30ebd-4eb1-304b-a204-7a4c9cd530e0: added peer, starting replication: peer=server-f41d018b-5213-3ac1-99d6-6a1bc7953714
2026-09-21T15:24:17.901Z [WARN]  server-f41d018b-5213-3ac1-99d6-6a1bc7953714: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2026-09-21T15:24:17.901Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:17.901Z [WARN]  server-8ef30ebd-4eb1-304b-a204-7a4c9cd530e0: appendEntries rejected, sending older logs: peer="{Voter server-f41d018b-5213-3ac1-99d6-6a1bc7953714 f41d018b-5213-3ac1-99d6-6a1bc7953714}" next=1
2026-09-21T15:24:17.901Z [INFO]  server-8ef30ebd-4eb1-304b-a204-7a4c9cd530e0: pipelining replication: peer="{Voter server-f41d018b-5213-3ac1-99d6-6a1bc7953714 f41d018b-5213-3ac1-99d6-6a1bc7953714}"
2026-09-21T15:24:18.007Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:17.901886788 +0000 UTC m=+9.843056961 (1 nodes), 640ns from start of poll, 169.98572ms from cluster start. Timeout at 2026-09-21 15:24:18.006890748 +0000 UTC m=+9.948060921, 105.00396ms after stability
2026-09-21T15:24:18.007Z [INFO]  server-8ef30ebd-4eb1-304b-a204-7a4c9cd530e0: aborting pipeline replication: peer="{Voter server-f41d018b-5213-3ac1-99d6-6a1bc7953714 f41d018b-5213-3ac1-99d6-6a1bc7953714}"
2026-09-21T15:24:18.007Z [INFO]  server-8ef30ebd-4eb1-304b-a204-7a4c9cd530e0: aborting pipeline replication: peer="{Voter server-9e866825-25ab-d538-d7d9-2f3f28354f5d 9e866825-25ab-d538-d7d9-2f3f28354f5d}"
--- PASS: TestRaft_JoinNode (0.28s)
=== RUN   TestRaft_JoinNode_ConfigStore
2026-09-21T15:24:18.009Z [INFO]  server-408633ee-f10c-8f4c-cb1e-5e6329f63e75: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-408633ee-f10c-8f4c-cb1e-5e6329f63e75 Address:408633ee-f10c-8f4c-cb1e-5e6329f63e75}]"
2026-09-21T15:24:18.009Z [INFO]  server-408633ee-f10c-8f4c-cb1e-5e6329f63e75: entering follower state: follower="Node at 408633ee-f10c-8f4c-cb1e-5e6329f63e75 [Follower]" leader-address= leader-id=
2026-09-21T15:24:18.009Z [INFO]  server-fc766753-9443-67b2-5ba1-b256609e624c: initial configuration: index=0 servers=[]
2026-09-21T15:24:18.009Z [INFO]  server-fc766753-9443-67b2-5ba1-b256609e624c: entering follower state: follower="Node at fc766753-9443-67b2-5ba1-b256609e624c [Follower]" leader-address= leader-id=
2026-09-21T15:24:18.009Z [INFO]  server-c61b1e54-b9bf-5de3-bfe3-6624a31f2a7e: initial configuration: index=0 servers=[]
2026-09-21T15:24:18.010Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:18.010Z [INFO]  server-c61b1e54-b9bf-5de3-bfe3-6624a31f2a7e: entering follower state: follower="Node at c61b1e54-b9bf-5de3-bfe3-6624a31f2a7e [Follower]" leader-address= leader-id=
2026-09-21T15:24:18.076Z [WARN]  server-408633ee-f10c-8f4c-cb1e-5e6329f63e75: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:18.076Z [INFO]  server-408633ee-f10c-8f4c-cb1e-5e6329f63e75: entering candidate state: node="Node at 408633ee-f10c-8f4c-cb1e-5e6329f63e75 [Candidate]" term=2
2026-09-21T15:24:18.076Z [INFO]  server-408633ee-f10c-8f4c-cb1e-5e6329f63e75: election won: term=2 tally=1
2026-09-21T15:24:18.076Z [INFO]  server-408633ee-f10c-8f4c-cb1e-5e6329f63e75: entering leader state: leader="Node at 408633ee-f10c-8f4c-cb1e-5e6329f63e75 [Leader]"
2026-09-21T15:24:18.077Z [WARN]  server-c61b1e54-b9bf-5de3-bfe3-6624a31f2a7e: no known peers, aborting election
2026-09-21T15:24:18.089Z [WARN]  server-fc766753-9443-67b2-5ba1-b256609e624c: no known peers, aborting election
2026-09-21T15:24:18.182Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:18.076622908 +0000 UTC m=+10.017793081 (1 nodes), 66.57924ms from start of poll, 67.11872ms from cluster start. Timeout at 2026-09-21 15:24:18.181627308 +0000 UTC m=+10.122797481, 105.0044ms after stability
2026-09-21T15:24:18.182Z [INFO]  server-408633ee-f10c-8f4c-cb1e-5e6329f63e75: updating configuration: command=AddVoter server-id=server-fc766753-9443-67b2-5ba1-b256609e624c server-addr=fc766753-9443-67b2-5ba1-b256609e624c servers="[{Suffrage:Voter ID:server-408633ee-f10c-8f4c-cb1e-5e6329f63e75 Address:408633ee-f10c-8f4c-cb1e-5e6329f63e75} {Suffrage:Voter ID:server-fc766753-9443-67b2-5ba1-b256609e624c Address:fc766753-9443-67b2-5ba1-b256609e624c}]"
2026-09-21T15:24:18.182Z [INFO]  server-408633ee-f10c-8f4c-cb1e-5e6329f63e75: added peer, starting replication: peer=server-fc766753-9443-67b2-5ba1-b256609e624c
2026-09-21T15:24:18.182Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:18.182Z [WARN]  server-fc766753-9443-67b2-5ba1-b256609e624c: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2026-09-21T15:24:18.182Z [WARN]  server-408633ee-f10c-8f4c-cb1e-5e6329f63e75: appendEntries rejected, sending older logs: peer="{Voter server-fc766753-9443-67b2-5ba1-b256609e624c fc766753-9443-67b2-5ba1-b256609e624c}" next=1
2026-09-21T15:24:18.182Z [INFO]  server-408633ee-f10c-8f4c-cb1e-5e6329f63e75: pipelining replication: peer="{Voter server-fc766753-9443-67b2-5ba1-b256609e624c fc766753-9443-67b2-5ba1-b256609e624c}"
2026-09-21T15:24:18.288Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:18.182593308 +0000 UTC m=+10.123763481 (1 nodes), 640ns from start of poll, 173.08912ms from cluster start. Timeout at 2026-09-21 15:24:18.287600908 +0000 UTC m=+10.228771081, 105.0076ms after stability
2026-09-21T15:24:18.288Z [INFO]  server-408633ee-f10c-8f4c-cb1e-5e6329f63e75: updating configuration: command=AddVoter server-id=server-c61b1e54-b9bf-5de3-bfe3-6624a31f2a7e server-addr=c61b1e54-b9bf-5de3-bfe3-6624a31f2a7e servers="[{Suffrage:Voter ID:server-408633ee-f10c-8f4c-cb1e-5e6329f63e75 Address:408633ee-f10c-8f4c-cb1e-5e6329f63e75} {Suffrage:Voter ID:server-fc766753-9443-67b2-5ba1-b256609e624c Address:fc766753-9443-67b2-5ba1-b256609e624c} {Suffrage:Voter ID:server-c61b1e54-b9bf-5de3-bfe3-6624a31f2a7e Address:c61b1e54-b9bf-5de3-bfe3-6624a31f2a7e}]"
2026-09-21T15:24:18.288Z [INFO]  server-408633ee-f10c-8f4c-cb1e-5e6329f63e75: added peer, starting replication: peer=server-c61b1e54-b9bf-5de3-bfe3-6624a31f2a7e
2026-09-21T15:24:18.288Z [WARN]  server-c61b1e54-b9bf-5de3-bfe3-6624a31f2a7e: failed to get previous log: previous-index=4 last-index=0 error="log not found"
2026-09-21T15:24:18.289Z [WARN]  server-408633ee-f10c-8f4c-cb1e-5e6329f63e75: appendEntries rejected, sending older logs: peer="{Voter server-c61b1e54-b9bf-5de3-bfe3-6624a31f2a7e c61b1e54-b9bf-5de3-bfe3-6624a31f2a7e}" next=1
2026-09-21T15:24:18.289Z [INFO]  server-408633ee-f10c-8f4c-cb1e-5e6329f63e75: pipelining replication: peer="{Voter server-c61b1e54-b9bf-5de3-bfe3-6624a31f2a7e c61b1e54-b9bf-5de3-bfe3-6624a31f2a7e}"
2026-09-21T15:24:18.289Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:18.394Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:18.289199428 +0000 UTC m=+10.230369601 (1 nodes), 560ns from start of poll, 279.69524ms from cluster start. Timeout at 2026-09-21 15:24:18.394205028 +0000 UTC m=+10.335375201, 105.0056ms after stability
2026-09-21T15:24:18.395Z [INFO]  server-408633ee-f10c-8f4c-cb1e-5e6329f63e75: aborting pipeline replication: peer="{Voter server-c61b1e54-b9bf-5de3-bfe3-6624a31f2a7e c61b1e54-b9bf-5de3-bfe3-6624a31f2a7e}"
2026-09-21T15:24:18.395Z [INFO]  server-408633ee-f10c-8f4c-cb1e-5e6329f63e75: aborting pipeline replication: peer="{Voter server-fc766753-9443-67b2-5ba1-b256609e624c fc766753-9443-67b2-5ba1-b256609e624c}"
--- PASS: TestRaft_JoinNode_ConfigStore (0.39s)
=== RUN   TestRaft_RemoveFollower
2026-09-21T15:24:18.395Z [INFO]  server-a4a028f7-6772-0de3-d8f6-47688d5b4368: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4a028f7-6772-0de3-d8f6-47688d5b4368 Address:a4a028f7-6772-0de3-d8f6-47688d5b4368} {Suffrage:Voter ID:server-ce988823-9124-0b0a-0d36-1d7e3c2c24bc Address:ce988823-9124-0b0a-0d36-1d7e3c2c24bc} {Suffrage:Voter ID:server-a38d7637-a577-1da4-5c7e-a1fe74f9d25c Address:a38d7637-a577-1da4-5c7e-a1fe74f9d25c}]"
2026-09-21T15:24:18.396Z [INFO]  server-ce988823-9124-0b0a-0d36-1d7e3c2c24bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4a028f7-6772-0de3-d8f6-47688d5b4368 Address:a4a028f7-6772-0de3-d8f6-47688d5b4368} {Suffrage:Voter ID:server-ce988823-9124-0b0a-0d36-1d7e3c2c24bc Address:ce988823-9124-0b0a-0d36-1d7e3c2c24bc} {Suffrage:Voter ID:server-a38d7637-a577-1da4-5c7e-a1fe74f9d25c Address:a38d7637-a577-1da4-5c7e-a1fe74f9d25c}]"
2026-09-21T15:24:18.396Z [INFO]  server-a4a028f7-6772-0de3-d8f6-47688d5b4368: entering follower state: follower="Node at a4a028f7-6772-0de3-d8f6-47688d5b4368 [Follower]" leader-address= leader-id=
2026-09-21T15:24:18.396Z [INFO]  server-ce988823-9124-0b0a-0d36-1d7e3c2c24bc: entering follower state: follower="Node at ce988823-9124-0b0a-0d36-1d7e3c2c24bc [Follower]" leader-address= leader-id=
2026-09-21T15:24:18.396Z [INFO]  server-a38d7637-a577-1da4-5c7e-a1fe74f9d25c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4a028f7-6772-0de3-d8f6-47688d5b4368 Address:a4a028f7-6772-0de3-d8f6-47688d5b4368} {Suffrage:Voter ID:server-ce988823-9124-0b0a-0d36-1d7e3c2c24bc Address:ce988823-9124-0b0a-0d36-1d7e3c2c24bc} {Suffrage:Voter ID:server-a38d7637-a577-1da4-5c7e-a1fe74f9d25c Address:a38d7637-a577-1da4-5c7e-a1fe74f9d25c}]"
2026-09-21T15:24:18.396Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:18.396Z [INFO]  server-a38d7637-a577-1da4-5c7e-a1fe74f9d25c: entering follower state: follower="Node at a38d7637-a577-1da4-5c7e-a1fe74f9d25c [Follower]" leader-address= leader-id=
2026-09-21T15:24:18.461Z [WARN]  server-a4a028f7-6772-0de3-d8f6-47688d5b4368: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:18.462Z [INFO]  server-a4a028f7-6772-0de3-d8f6-47688d5b4368: entering candidate state: node="Node at a4a028f7-6772-0de3-d8f6-47688d5b4368 [Candidate]" term=2
2026-09-21T15:24:18.462Z [INFO]  server-a4a028f7-6772-0de3-d8f6-47688d5b4368: election won: term=2 tally=2
2026-09-21T15:24:18.462Z [INFO]  server-a4a028f7-6772-0de3-d8f6-47688d5b4368: entering leader state: leader="Node at a4a028f7-6772-0de3-d8f6-47688d5b4368 [Leader]"
2026-09-21T15:24:18.462Z [INFO]  server-a4a028f7-6772-0de3-d8f6-47688d5b4368: added peer, starting replication: peer=server-ce988823-9124-0b0a-0d36-1d7e3c2c24bc
2026-09-21T15:24:18.462Z [INFO]  server-a4a028f7-6772-0de3-d8f6-47688d5b4368: added peer, starting replication: peer=server-a38d7637-a577-1da4-5c7e-a1fe74f9d25c
2026-09-21T15:24:18.462Z [INFO]  server-a4a028f7-6772-0de3-d8f6-47688d5b4368: pipelining replication: peer="{Voter server-a38d7637-a577-1da4-5c7e-a1fe74f9d25c a38d7637-a577-1da4-5c7e-a1fe74f9d25c}"
2026-09-21T15:24:18.462Z [INFO]  server-a4a028f7-6772-0de3-d8f6-47688d5b4368: pipelining replication: peer="{Voter server-ce988823-9124-0b0a-0d36-1d7e3c2c24bc ce988823-9124-0b0a-0d36-1d7e3c2c24bc}"
2026-09-21T15:24:18.567Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:18.462212428 +0000 UTC m=+10.403382601 (1 nodes), 65.77868ms from start of poll, 66.40904ms from cluster start. Timeout at 2026-09-21 15:24:18.567215468 +0000 UTC m=+10.508385681, 105.00308ms after stability
2026-09-21T15:24:18.573Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:18.679Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:18.573260668 +0000 UTC m=+10.514430841 (2 nodes), 1.24µs from start of poll, 177.45728ms from cluster start. Timeout at 2026-09-21 15:24:18.678265628 +0000 UTC m=+10.619435841, 105.005ms after stability
2026-09-21T15:24:18.679Z [INFO]  server-a4a028f7-6772-0de3-d8f6-47688d5b4368: updating configuration: command=RemoveServer server-id=server-ce988823-9124-0b0a-0d36-1d7e3c2c24bc server-addr= servers="[{Suffrage:Voter ID:server-a4a028f7-6772-0de3-d8f6-47688d5b4368 Address:a4a028f7-6772-0de3-d8f6-47688d5b4368} {Suffrage:Voter ID:server-a38d7637-a577-1da4-5c7e-a1fe74f9d25c Address:a38d7637-a577-1da4-5c7e-a1fe74f9d25c}]"
2026-09-21T15:24:18.679Z [INFO]  server-a4a028f7-6772-0de3-d8f6-47688d5b4368: removed peer, stopping replication: peer=server-ce988823-9124-0b0a-0d36-1d7e3c2c24bc last-index=3
2026-09-21T15:24:18.679Z [INFO]  server-a4a028f7-6772-0de3-d8f6-47688d5b4368: aborting pipeline replication: peer="{Voter server-ce988823-9124-0b0a-0d36-1d7e3c2c24bc ce988823-9124-0b0a-0d36-1d7e3c2c24bc}"
2026-09-21T15:24:18.733Z [WARN]  server-ce988823-9124-0b0a-0d36-1d7e3c2c24bc: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2026-09-21T15:24:18.786Z [INFO]  server-a4a028f7-6772-0de3-d8f6-47688d5b4368: aborting pipeline replication: peer="{Voter server-a38d7637-a577-1da4-5c7e-a1fe74f9d25c a38d7637-a577-1da4-5c7e-a1fe74f9d25c}"
--- PASS: TestRaft_RemoveFollower (0.39s)
=== RUN   TestRaft_RemoveLeader
2026-09-21T15:24:18.787Z [INFO]  server-cafb8d2c-5f6c-ca5f-afcd-dc973fddce97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cafb8d2c-5f6c-ca5f-afcd-dc973fddce97 Address:cafb8d2c-5f6c-ca5f-afcd-dc973fddce97} {Suffrage:Voter ID:server-a5817286-4d72-a730-f275-b1d18aa6bc7d Address:a5817286-4d72-a730-f275-b1d18aa6bc7d} {Suffrage:Voter ID:server-df10d181-6e2c-5f49-2e25-84a24171e277 Address:df10d181-6e2c-5f49-2e25-84a24171e277}]"
2026-09-21T15:24:18.787Z [INFO]  server-a5817286-4d72-a730-f275-b1d18aa6bc7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cafb8d2c-5f6c-ca5f-afcd-dc973fddce97 Address:cafb8d2c-5f6c-ca5f-afcd-dc973fddce97} {Suffrage:Voter ID:server-a5817286-4d72-a730-f275-b1d18aa6bc7d Address:a5817286-4d72-a730-f275-b1d18aa6bc7d} {Suffrage:Voter ID:server-df10d181-6e2c-5f49-2e25-84a24171e277 Address:df10d181-6e2c-5f49-2e25-84a24171e277}]"
2026-09-21T15:24:18.787Z [INFO]  server-cafb8d2c-5f6c-ca5f-afcd-dc973fddce97: entering follower state: follower="Node at cafb8d2c-5f6c-ca5f-afcd-dc973fddce97 [Follower]" leader-address= leader-id=
2026-09-21T15:24:18.787Z [INFO]  server-a5817286-4d72-a730-f275-b1d18aa6bc7d: entering follower state: follower="Node at a5817286-4d72-a730-f275-b1d18aa6bc7d [Follower]" leader-address= leader-id=
2026-09-21T15:24:18.787Z [INFO]  server-df10d181-6e2c-5f49-2e25-84a24171e277: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cafb8d2c-5f6c-ca5f-afcd-dc973fddce97 Address:cafb8d2c-5f6c-ca5f-afcd-dc973fddce97} {Suffrage:Voter ID:server-a5817286-4d72-a730-f275-b1d18aa6bc7d Address:a5817286-4d72-a730-f275-b1d18aa6bc7d} {Suffrage:Voter ID:server-df10d181-6e2c-5f49-2e25-84a24171e277 Address:df10d181-6e2c-5f49-2e25-84a24171e277}]"
2026-09-21T15:24:18.787Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:18.787Z [INFO]  server-df10d181-6e2c-5f49-2e25-84a24171e277: entering follower state: follower="Node at df10d181-6e2c-5f49-2e25-84a24171e277 [Follower]" leader-address= leader-id=
2026-09-21T15:24:18.863Z [WARN]  server-a5817286-4d72-a730-f275-b1d18aa6bc7d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:18.863Z [INFO]  server-a5817286-4d72-a730-f275-b1d18aa6bc7d: entering candidate state: node="Node at a5817286-4d72-a730-f275-b1d18aa6bc7d [Candidate]" term=2
2026-09-21T15:24:18.863Z [INFO]  server-a5817286-4d72-a730-f275-b1d18aa6bc7d: election won: term=2 tally=2
2026-09-21T15:24:18.863Z [INFO]  server-a5817286-4d72-a730-f275-b1d18aa6bc7d: entering leader state: leader="Node at a5817286-4d72-a730-f275-b1d18aa6bc7d [Leader]"
2026-09-21T15:24:18.863Z [INFO]  server-a5817286-4d72-a730-f275-b1d18aa6bc7d: added peer, starting replication: peer=server-cafb8d2c-5f6c-ca5f-afcd-dc973fddce97
2026-09-21T15:24:18.863Z [INFO]  server-a5817286-4d72-a730-f275-b1d18aa6bc7d: added peer, starting replication: peer=server-df10d181-6e2c-5f49-2e25-84a24171e277
2026-09-21T15:24:18.863Z [INFO]  server-a5817286-4d72-a730-f275-b1d18aa6bc7d: pipelining replication: peer="{Voter server-df10d181-6e2c-5f49-2e25-84a24171e277 df10d181-6e2c-5f49-2e25-84a24171e277}"
2026-09-21T15:24:18.863Z [INFO]  server-a5817286-4d72-a730-f275-b1d18aa6bc7d: pipelining replication: peer="{Voter server-cafb8d2c-5f6c-ca5f-afcd-dc973fddce97 cafb8d2c-5f6c-ca5f-afcd-dc973fddce97}"
2026-09-21T15:24:18.968Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:18.863254188 +0000 UTC m=+10.804424401 (1 nodes), 75.70268ms from start of poll, 76.1366ms from cluster start. Timeout at 2026-09-21 15:24:18.968256308 +0000 UTC m=+10.909426481, 105.00208ms after stability
2026-09-21T15:24:18.974Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:19.079Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:18.974426028 +0000 UTC m=+10.915596201 (2 nodes), 1.16µs from start of poll, 187.3084ms from cluster start. Timeout at 2026-09-21 15:24:19.079430388 +0000 UTC m=+11.020600561, 105.00436ms after stability
2026-09-21T15:24:19.079Z [INFO]  server-a5817286-4d72-a730-f275-b1d18aa6bc7d: updating configuration: command=RemoveServer server-id=server-a5817286-4d72-a730-f275-b1d18aa6bc7d server-addr= servers="[{Suffrage:Voter ID:server-cafb8d2c-5f6c-ca5f-afcd-dc973fddce97 Address:cafb8d2c-5f6c-ca5f-afcd-dc973fddce97} {Suffrage:Voter ID:server-df10d181-6e2c-5f49-2e25-84a24171e277 Address:df10d181-6e2c-5f49-2e25-84a24171e277}]"
2026-09-21T15:24:19.080Z [INFO]  server-a5817286-4d72-a730-f275-b1d18aa6bc7d: removed ourself, shutting down
2026-09-21T15:24:19.080Z [INFO]  server-a5817286-4d72-a730-f275-b1d18aa6bc7d: aborting pipeline replication: peer="{Voter server-df10d181-6e2c-5f49-2e25-84a24171e277 df10d181-6e2c-5f49-2e25-84a24171e277}"
2026-09-21T15:24:19.080Z [INFO]  server-a5817286-4d72-a730-f275-b1d18aa6bc7d: aborting pipeline replication: peer="{Voter server-cafb8d2c-5f6c-ca5f-afcd-dc973fddce97 cafb8d2c-5f6c-ca5f-afcd-dc973fddce97}"
2026-09-21T15:24:19.141Z [WARN]  server-df10d181-6e2c-5f49-2e25-84a24171e277: heartbeat timeout reached, starting election: last-leader-addr=a5817286-4d72-a730-f275-b1d18aa6bc7d last-leader-id=server-a5817286-4d72-a730-f275-b1d18aa6bc7d
2026-09-21T15:24:19.141Z [INFO]  server-df10d181-6e2c-5f49-2e25-84a24171e277: entering candidate state: node="Node at df10d181-6e2c-5f49-2e25-84a24171e277 [Candidate]" term=3
2026-09-21T15:24:19.141Z [WARN]  server-cafb8d2c-5f6c-ca5f-afcd-dc973fddce97: rejecting vote request since we have a leader: from=df10d181-6e2c-5f49-2e25-84a24171e277 leader=a5817286-4d72-a730-f275-b1d18aa6bc7d leader-id=server-a5817286-4d72-a730-f275-b1d18aa6bc7d
2026-09-21T15:24:19.202Z [WARN]  server-cafb8d2c-5f6c-ca5f-afcd-dc973fddce97: heartbeat timeout reached, starting election: last-leader-addr=a5817286-4d72-a730-f275-b1d18aa6bc7d last-leader-id=server-a5817286-4d72-a730-f275-b1d18aa6bc7d
2026-09-21T15:24:19.202Z [INFO]  server-cafb8d2c-5f6c-ca5f-afcd-dc973fddce97: entering candidate state: node="Node at cafb8d2c-5f6c-ca5f-afcd-dc973fddce97 [Candidate]" term=3
2026-09-21T15:24:19.202Z [INFO]  server-df10d181-6e2c-5f49-2e25-84a24171e277: duplicate requestVote for same term: term=3
2026-09-21T15:24:19.208Z [WARN]  server-df10d181-6e2c-5f49-2e25-84a24171e277: Election timeout reached, restarting election
2026-09-21T15:24:19.208Z [INFO]  server-df10d181-6e2c-5f49-2e25-84a24171e277: entering candidate state: node="Node at df10d181-6e2c-5f49-2e25-84a24171e277 [Candidate]" term=4
2026-09-21T15:24:19.209Z [INFO]  server-cafb8d2c-5f6c-ca5f-afcd-dc973fddce97: entering follower state: follower="Node at cafb8d2c-5f6c-ca5f-afcd-dc973fddce97 [Follower]" leader-address= leader-id=
2026-09-21T15:24:19.209Z [INFO]  server-df10d181-6e2c-5f49-2e25-84a24171e277: election won: term=4 tally=2
2026-09-21T15:24:19.209Z [INFO]  server-df10d181-6e2c-5f49-2e25-84a24171e277: entering leader state: leader="Node at df10d181-6e2c-5f49-2e25-84a24171e277 [Leader]"
2026-09-21T15:24:19.209Z [INFO]  server-df10d181-6e2c-5f49-2e25-84a24171e277: added peer, starting replication: peer=server-cafb8d2c-5f6c-ca5f-afcd-dc973fddce97
2026-09-21T15:24:19.209Z [INFO]  server-df10d181-6e2c-5f49-2e25-84a24171e277: pipelining replication: peer="{Voter server-cafb8d2c-5f6c-ca5f-afcd-dc973fddce97 cafb8d2c-5f6c-ca5f-afcd-dc973fddce97}"
2026-09-21T15:24:19.291Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:19.396Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:19.291310428 +0000 UTC m=+11.232480601 (1 nodes), 70.76µs from start of poll, 504.1928ms from cluster start. Timeout at 2026-09-21 15:24:19.396321588 +0000 UTC m=+11.337491761, 105.01116ms after stability
2026-09-21T15:24:19.396Z [INFO]  server-df10d181-6e2c-5f49-2e25-84a24171e277: aborting pipeline replication: peer="{Voter server-cafb8d2c-5f6c-ca5f-afcd-dc973fddce97 cafb8d2c-5f6c-ca5f-afcd-dc973fddce97}"
--- PASS: TestRaft_RemoveLeader (0.61s)
=== RUN   TestRaft_RemoveLeader_NoShutdown
2026-09-21T15:24:19.397Z [INFO]  server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f Address:7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f} {Suffrage:Voter ID:server-a38d352e-f90d-c91e-e62d-9cf569bb86df Address:a38d352e-f90d-c91e-e62d-9cf569bb86df} {Suffrage:Voter ID:server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e Address:1f0d33e7-f976-39ea-7cf1-d3e490f7d98e}]"
2026-09-21T15:24:19.397Z [INFO]  server-a38d352e-f90d-c91e-e62d-9cf569bb86df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f Address:7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f} {Suffrage:Voter ID:server-a38d352e-f90d-c91e-e62d-9cf569bb86df Address:a38d352e-f90d-c91e-e62d-9cf569bb86df} {Suffrage:Voter ID:server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e Address:1f0d33e7-f976-39ea-7cf1-d3e490f7d98e}]"
2026-09-21T15:24:19.397Z [INFO]  server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f: entering follower state: follower="Node at 7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f [Follower]" leader-address= leader-id=
2026-09-21T15:24:19.397Z [INFO]  server-a38d352e-f90d-c91e-e62d-9cf569bb86df: entering follower state: follower="Node at a38d352e-f90d-c91e-e62d-9cf569bb86df [Follower]" leader-address= leader-id=
2026-09-21T15:24:19.397Z [INFO]  server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f Address:7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f} {Suffrage:Voter ID:server-a38d352e-f90d-c91e-e62d-9cf569bb86df Address:a38d352e-f90d-c91e-e62d-9cf569bb86df} {Suffrage:Voter ID:server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e Address:1f0d33e7-f976-39ea-7cf1-d3e490f7d98e}]"
2026-09-21T15:24:19.397Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:19.397Z [INFO]  server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e: entering follower state: follower="Node at 1f0d33e7-f976-39ea-7cf1-d3e490f7d98e [Follower]" leader-address= leader-id=
2026-09-21T15:24:19.472Z [WARN]  server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:19.472Z [INFO]  server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e: entering candidate state: node="Node at 1f0d33e7-f976-39ea-7cf1-d3e490f7d98e [Candidate]" term=2
2026-09-21T15:24:19.472Z [WARN]  server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:19.472Z [INFO]  server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f: entering candidate state: node="Node at 7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f [Candidate]" term=2
2026-09-21T15:24:19.472Z [INFO]  server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e: election won: term=2 tally=2
2026-09-21T15:24:19.472Z [INFO]  server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f: duplicate requestVote for same term: term=2
2026-09-21T15:24:19.472Z [INFO]  server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e: entering leader state: leader="Node at 1f0d33e7-f976-39ea-7cf1-d3e490f7d98e [Leader]"
2026-09-21T15:24:19.472Z [INFO]  server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e: added peer, starting replication: peer=server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f
2026-09-21T15:24:19.472Z [INFO]  server-a38d352e-f90d-c91e-e62d-9cf569bb86df: duplicate requestVote for same term: term=2
2026-09-21T15:24:19.472Z [INFO]  server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e: added peer, starting replication: peer=server-a38d352e-f90d-c91e-e62d-9cf569bb86df
2026-09-21T15:24:19.472Z [WARN]  server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e: rejecting vote request since we have a leader: from=7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f leader=1f0d33e7-f976-39ea-7cf1-d3e490f7d98e leader-id=server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e
2026-09-21T15:24:19.472Z [INFO]  server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f: entering follower state: follower="Node at 7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f [Follower]" leader-address=1f0d33e7-f976-39ea-7cf1-d3e490f7d98e leader-id=server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e
2026-09-21T15:24:19.472Z [INFO]  server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e: pipelining replication: peer="{Voter server-a38d352e-f90d-c91e-e62d-9cf569bb86df a38d352e-f90d-c91e-e62d-9cf569bb86df}"
2026-09-21T15:24:19.472Z [INFO]  server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e: pipelining replication: peer="{Voter server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f 7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f}"
2026-09-21T15:24:19.578Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:19.472890428 +0000 UTC m=+11.414060601 (2 nodes), 74.96376ms from start of poll, 75.3398ms from cluster start. Timeout at 2026-09-21 15:24:19.577892268 +0000 UTC m=+11.519062441, 105.00184ms after stability
2026-09-21T15:24:19.578Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:19.684Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:19.578697348 +0000 UTC m=+11.519867521 (1 nodes), 1µs from start of poll, 181.14672ms from cluster start. Timeout at 2026-09-21 15:24:19.683778068 +0000 UTC m=+11.624948281, 105.08076ms after stability
2026-09-21T15:24:19.689Z [INFO]  server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e: updating configuration: command=RemoveServer server-id=server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e server-addr= servers="[{Suffrage:Voter ID:server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f Address:7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f} {Suffrage:Voter ID:server-a38d352e-f90d-c91e-e62d-9cf569bb86df Address:a38d352e-f90d-c91e-e62d-9cf569bb86df}]"
2026-09-21T15:24:19.689Z [INFO]  server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e: removed ourself, transitioning to follower
2026-09-21T15:24:19.689Z [INFO]  server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e: entering follower state: follower="Node at 1f0d33e7-f976-39ea-7cf1-d3e490f7d98e [Follower]" leader-address= leader-id=
2026-09-21T15:24:19.689Z [INFO]  server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e: aborting pipeline replication: peer="{Voter server-a38d352e-f90d-c91e-e62d-9cf569bb86df a38d352e-f90d-c91e-e62d-9cf569bb86df}"
2026-09-21T15:24:19.689Z [INFO]  server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e: aborting pipeline replication: peer="{Voter server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f 7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f}"
2026-09-21T15:24:19.755Z [WARN]  server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e: not part of stable configuration, aborting election
2026-09-21T15:24:19.758Z [WARN]  server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f: heartbeat timeout reached, starting election: last-leader-addr=1f0d33e7-f976-39ea-7cf1-d3e490f7d98e last-leader-id=server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e
2026-09-21T15:24:19.758Z [INFO]  server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f: entering candidate state: node="Node at 7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f [Candidate]" term=3
2026-09-21T15:24:19.758Z [WARN]  server-a38d352e-f90d-c91e-e62d-9cf569bb86df: rejecting vote request since we have a leader: from=7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f leader=1f0d33e7-f976-39ea-7cf1-d3e490f7d98e leader-id=server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e
2026-09-21T15:24:19.791Z [WARN]  server-a38d352e-f90d-c91e-e62d-9cf569bb86df: heartbeat timeout reached, starting election: last-leader-addr=1f0d33e7-f976-39ea-7cf1-d3e490f7d98e last-leader-id=server-1f0d33e7-f976-39ea-7cf1-d3e490f7d98e
2026-09-21T15:24:19.791Z [INFO]  server-a38d352e-f90d-c91e-e62d-9cf569bb86df: entering candidate state: node="Node at a38d352e-f90d-c91e-e62d-9cf569bb86df [Candidate]" term=3
2026-09-21T15:24:19.791Z [INFO]  server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f: duplicate requestVote for same term: term=3
2026-09-21T15:24:19.795Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:19.852Z [WARN]  server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f: Election timeout reached, restarting election
2026-09-21T15:24:19.852Z [INFO]  server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f: entering candidate state: node="Node at 7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f [Candidate]" term=4
2026-09-21T15:24:19.852Z [INFO]  server-a38d352e-f90d-c91e-e62d-9cf569bb86df: entering follower state: follower="Node at a38d352e-f90d-c91e-e62d-9cf569bb86df [Follower]" leader-address= leader-id=
2026-09-21T15:24:19.852Z [INFO]  server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f: election won: term=4 tally=2
2026-09-21T15:24:19.852Z [INFO]  server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f: entering leader state: leader="Node at 7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f [Leader]"
2026-09-21T15:24:19.852Z [INFO]  server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f: added peer, starting replication: peer=server-a38d352e-f90d-c91e-e62d-9cf569bb86df
2026-09-21T15:24:19.853Z [INFO]  server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f: pipelining replication: peer="{Voter server-a38d352e-f90d-c91e-e62d-9cf569bb86df a38d352e-f90d-c91e-e62d-9cf569bb86df}"
2026-09-21T15:24:19.958Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:19.852984068 +0000 UTC m=+11.794154281 (1 nodes), 57.2334ms from start of poll, 455.43348ms from cluster start. Timeout at 2026-09-21 15:24:19.957989108 +0000 UTC m=+11.899159281, 105.005ms after stability
2026-09-21T15:24:20.063Z [INFO]  server-7f07ac4e-ea8e-bcc3-6e0a-4aa4ee40e32f: aborting pipeline replication: peer="{Voter server-a38d352e-f90d-c91e-e62d-9cf569bb86df a38d352e-f90d-c91e-e62d-9cf569bb86df}"
--- PASS: TestRaft_RemoveLeader_NoShutdown (0.67s)
=== RUN   TestRaft_RemoveFollower_SplitCluster
2026-09-21T15:24:20.064Z [INFO]  server-7a1e6ec6-4751-f9d6-e0d5-f77e4061d349: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7a1e6ec6-4751-f9d6-e0d5-f77e4061d349 Address:7a1e6ec6-4751-f9d6-e0d5-f77e4061d349} {Suffrage:Voter ID:server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f Address:4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f} {Suffrage:Voter ID:server-d00196cf-d4fd-8903-dfc5-be607e30e732 Address:d00196cf-d4fd-8903-dfc5-be607e30e732} {Suffrage:Voter ID:server-b21f4ebd-1760-a826-b5fd-7b5a52b748c4 Address:b21f4ebd-1760-a826-b5fd-7b5a52b748c4}]"
2026-09-21T15:24:20.064Z [INFO]  server-7a1e6ec6-4751-f9d6-e0d5-f77e4061d349: entering follower state: follower="Node at 7a1e6ec6-4751-f9d6-e0d5-f77e4061d349 [Follower]" leader-address= leader-id=
2026-09-21T15:24:20.064Z [INFO]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7a1e6ec6-4751-f9d6-e0d5-f77e4061d349 Address:7a1e6ec6-4751-f9d6-e0d5-f77e4061d349} {Suffrage:Voter ID:server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f Address:4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f} {Suffrage:Voter ID:server-d00196cf-d4fd-8903-dfc5-be607e30e732 Address:d00196cf-d4fd-8903-dfc5-be607e30e732} {Suffrage:Voter ID:server-b21f4ebd-1760-a826-b5fd-7b5a52b748c4 Address:b21f4ebd-1760-a826-b5fd-7b5a52b748c4}]"
2026-09-21T15:24:20.064Z [INFO]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: entering follower state: follower="Node at 4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f [Follower]" leader-address= leader-id=
2026-09-21T15:24:20.065Z [INFO]  server-d00196cf-d4fd-8903-dfc5-be607e30e732: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7a1e6ec6-4751-f9d6-e0d5-f77e4061d349 Address:7a1e6ec6-4751-f9d6-e0d5-f77e4061d349} {Suffrage:Voter ID:server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f Address:4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f} {Suffrage:Voter ID:server-d00196cf-d4fd-8903-dfc5-be607e30e732 Address:d00196cf-d4fd-8903-dfc5-be607e30e732} {Suffrage:Voter ID:server-b21f4ebd-1760-a826-b5fd-7b5a52b748c4 Address:b21f4ebd-1760-a826-b5fd-7b5a52b748c4}]"
2026-09-21T15:24:20.065Z [INFO]  server-d00196cf-d4fd-8903-dfc5-be607e30e732: entering follower state: follower="Node at d00196cf-d4fd-8903-dfc5-be607e30e732 [Follower]" leader-address= leader-id=
2026-09-21T15:24:20.065Z [INFO]  server-b21f4ebd-1760-a826-b5fd-7b5a52b748c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7a1e6ec6-4751-f9d6-e0d5-f77e4061d349 Address:7a1e6ec6-4751-f9d6-e0d5-f77e4061d349} {Suffrage:Voter ID:server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f Address:4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f} {Suffrage:Voter ID:server-d00196cf-d4fd-8903-dfc5-be607e30e732 Address:d00196cf-d4fd-8903-dfc5-be607e30e732} {Suffrage:Voter ID:server-b21f4ebd-1760-a826-b5fd-7b5a52b748c4 Address:b21f4ebd-1760-a826-b5fd-7b5a52b748c4}]"
2026-09-21T15:24:20.065Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:20.065Z [INFO]  server-b21f4ebd-1760-a826-b5fd-7b5a52b748c4: entering follower state: follower="Node at b21f4ebd-1760-a826-b5fd-7b5a52b748c4 [Follower]" leader-address= leader-id=
2026-09-21T15:24:20.120Z [WARN]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:20.120Z [INFO]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: entering candidate state: node="Node at 4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f [Candidate]" term=2
2026-09-21T15:24:20.120Z [INFO]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: election won: term=2 tally=3
2026-09-21T15:24:20.121Z [INFO]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: entering leader state: leader="Node at 4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f [Leader]"
2026-09-21T15:24:20.121Z [INFO]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: added peer, starting replication: peer=server-7a1e6ec6-4751-f9d6-e0d5-f77e4061d349
2026-09-21T15:24:20.121Z [INFO]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: added peer, starting replication: peer=server-d00196cf-d4fd-8903-dfc5-be607e30e732
2026-09-21T15:24:20.121Z [INFO]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: added peer, starting replication: peer=server-b21f4ebd-1760-a826-b5fd-7b5a52b748c4
2026-09-21T15:24:20.121Z [INFO]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: pipelining replication: peer="{Voter server-b21f4ebd-1760-a826-b5fd-7b5a52b748c4 b21f4ebd-1760-a826-b5fd-7b5a52b748c4}"
2026-09-21T15:24:20.121Z [INFO]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: pipelining replication: peer="{Voter server-7a1e6ec6-4751-f9d6-e0d5-f77e4061d349 7a1e6ec6-4751-f9d6-e0d5-f77e4061d349}"
2026-09-21T15:24:20.121Z [INFO]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: pipelining replication: peer="{Voter server-d00196cf-d4fd-8903-dfc5-be607e30e732 d00196cf-d4fd-8903-dfc5-be607e30e732}"
2026-09-21T15:24:20.227Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:20.121171788 +0000 UTC m=+12.062342001 (1 nodes), 55.8736ms from start of poll, 56.6754ms from cluster start. Timeout at 2026-09-21 15:24:20.226174668 +0000 UTC m=+12.167344881, 105.00288ms after stability
2026-09-21T15:24:20.333Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:20.438Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:20.333368868 +0000 UTC m=+12.274539081 (3 nodes), 2.76µs from start of poll, 268.87248ms from cluster start. Timeout at 2026-09-21 15:24:20.438377068 +0000 UTC m=+12.379547241, 105.00816ms after stability
2026-09-21T15:24:20.438Z [INFO]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: updating configuration: command=RemoveServer server-id=server-b21f4ebd-1760-a826-b5fd-7b5a52b748c4 server-addr= servers="[{Suffrage:Voter ID:server-7a1e6ec6-4751-f9d6-e0d5-f77e4061d349 Address:7a1e6ec6-4751-f9d6-e0d5-f77e4061d349} {Suffrage:Voter ID:server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f Address:4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f} {Suffrage:Voter ID:server-d00196cf-d4fd-8903-dfc5-be607e30e732 Address:d00196cf-d4fd-8903-dfc5-be607e30e732}]"
2026-09-21T15:24:20.438Z [INFO]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: removed peer, stopping replication: peer=server-b21f4ebd-1760-a826-b5fd-7b5a52b748c4 last-index=3
2026-09-21T15:24:20.438Z [ERROR] server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: failed to pipeline appendEntries: peer="{Voter server-7a1e6ec6-4751-f9d6-e0d5-f77e4061d349 7a1e6ec6-4751-f9d6-e0d5-f77e4061d349}" error="append pipeline closed"
2026-09-21T15:24:20.438Z [INFO]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: aborting pipeline replication: peer="{Voter server-7a1e6ec6-4751-f9d6-e0d5-f77e4061d349 7a1e6ec6-4751-f9d6-e0d5-f77e4061d349}"
2026-09-21T15:24:20.438Z [ERROR] server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: failed to pipeline appendEntries: peer="{Voter server-d00196cf-d4fd-8903-dfc5-be607e30e732 d00196cf-d4fd-8903-dfc5-be607e30e732}" error="append pipeline closed"
2026-09-21T15:24:20.438Z [INFO]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: aborting pipeline replication: peer="{Voter server-d00196cf-d4fd-8903-dfc5-be607e30e732 d00196cf-d4fd-8903-dfc5-be607e30e732}"
2026-09-21T15:24:20.438Z [INFO]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: aborting pipeline replication: peer="{Voter server-b21f4ebd-1760-a826-b5fd-7b5a52b748c4 b21f4ebd-1760-a826-b5fd-7b5a52b748c4}"
2026-09-21T15:24:20.446Z [ERROR] server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: failed to heartbeat to: peer=d00196cf-d4fd-8903-dfc5-be607e30e732 backoff time=10ms error="failed to connect to peer: d00196cf-d4fd-8903-dfc5-be607e30e732"
2026-09-21T15:24:20.447Z [ERROR] server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: failed to heartbeat to: peer=7a1e6ec6-4751-f9d6-e0d5-f77e4061d349 backoff time=10ms error="failed to connect to peer: 7a1e6ec6-4751-f9d6-e0d5-f77e4061d349"
2026-09-21T15:24:20.447Z [ERROR] server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: failed to appendEntries to: peer="{Voter server-d00196cf-d4fd-8903-dfc5-be607e30e732 d00196cf-d4fd-8903-dfc5-be607e30e732}" error="failed to connect to peer: d00196cf-d4fd-8903-dfc5-be607e30e732"
2026-09-21T15:24:20.447Z [ERROR] server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: failed to appendEntries to: peer="{Voter server-7a1e6ec6-4751-f9d6-e0d5-f77e4061d349 7a1e6ec6-4751-f9d6-e0d5-f77e4061d349}" error="failed to connect to peer: 7a1e6ec6-4751-f9d6-e0d5-f77e4061d349"
2026-09-21T15:24:20.461Z [ERROR] server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: failed to heartbeat to: peer=d00196cf-d4fd-8903-dfc5-be607e30e732 backoff time=10ms error="failed to connect to peer: d00196cf-d4fd-8903-dfc5-be607e30e732"
2026-09-21T15:24:20.465Z [ERROR] server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: failed to appendEntries to: peer="{Voter server-d00196cf-d4fd-8903-dfc5-be607e30e732 d00196cf-d4fd-8903-dfc5-be607e30e732}" error="failed to connect to peer: d00196cf-d4fd-8903-dfc5-be607e30e732"
2026-09-21T15:24:20.467Z [ERROR] server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: failed to appendEntries to: peer="{Voter server-7a1e6ec6-4751-f9d6-e0d5-f77e4061d349 7a1e6ec6-4751-f9d6-e0d5-f77e4061d349}" error="failed to connect to peer: 7a1e6ec6-4751-f9d6-e0d5-f77e4061d349"
2026-09-21T15:24:20.468Z [ERROR] server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: failed to heartbeat to: peer=7a1e6ec6-4751-f9d6-e0d5-f77e4061d349 backoff time=10ms error="failed to connect to peer: 7a1e6ec6-4751-f9d6-e0d5-f77e4061d349"
2026-09-21T15:24:20.480Z [ERROR] server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: failed to heartbeat to: peer=d00196cf-d4fd-8903-dfc5-be607e30e732 backoff time=10ms error="failed to connect to peer: d00196cf-d4fd-8903-dfc5-be607e30e732"
2026-09-21T15:24:20.484Z [ERROR] server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: failed to heartbeat to: peer=7a1e6ec6-4751-f9d6-e0d5-f77e4061d349 backoff time=10ms error="failed to connect to peer: 7a1e6ec6-4751-f9d6-e0d5-f77e4061d349"
2026-09-21T15:24:20.485Z [ERROR] server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: failed to appendEntries to: peer="{Voter server-7a1e6ec6-4751-f9d6-e0d5-f77e4061d349 7a1e6ec6-4751-f9d6-e0d5-f77e4061d349}" error="failed to connect to peer: 7a1e6ec6-4751-f9d6-e0d5-f77e4061d349"
2026-09-21T15:24:20.486Z [ERROR] server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: failed to appendEntries to: peer="{Voter server-d00196cf-d4fd-8903-dfc5-be607e30e732 d00196cf-d4fd-8903-dfc5-be607e30e732}" error="failed to connect to peer: d00196cf-d4fd-8903-dfc5-be607e30e732"
2026-09-21T15:24:20.489Z [WARN]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: failed to contact: server-id=server-7a1e6ec6-4751-f9d6-e0d5-f77e4061d349 time=50.79228ms
2026-09-21T15:24:20.489Z [WARN]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: failed to contact: server-id=server-d00196cf-d4fd-8903-dfc5-be607e30e732 time=50.74904ms
2026-09-21T15:24:20.489Z [WARN]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: failed to contact quorum of nodes, stepping down
2026-09-21T15:24:20.489Z [INFO]  server-4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f: entering follower state: follower="Node at 4d1228f5-9cd8-d293-0d8b-e9a6c828ce5f [Follower]" leader-address= leader-id=
--- PASS: TestRaft_RemoveFollower_SplitCluster (0.43s)
=== RUN   TestRaft_AddKnownPeer
2026-09-21T15:24:20.490Z [INFO]  server-7ac2bcc6-cbaf-78a5-59d5-91cece5fa344: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ac2bcc6-cbaf-78a5-59d5-91cece5fa344 Address:7ac2bcc6-cbaf-78a5-59d5-91cece5fa344} {Suffrage:Voter ID:server-1f46660f-ee11-a224-1546-8e5b2c46eec4 Address:1f46660f-ee11-a224-1546-8e5b2c46eec4} {Suffrage:Voter ID:server-0f5c1330-1363-a88b-a119-e63ea0d7b1a7 Address:0f5c1330-1363-a88b-a119-e63ea0d7b1a7}]"
2026-09-21T15:24:20.490Z [INFO]  server-1f46660f-ee11-a224-1546-8e5b2c46eec4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ac2bcc6-cbaf-78a5-59d5-91cece5fa344 Address:7ac2bcc6-cbaf-78a5-59d5-91cece5fa344} {Suffrage:Voter ID:server-1f46660f-ee11-a224-1546-8e5b2c46eec4 Address:1f46660f-ee11-a224-1546-8e5b2c46eec4} {Suffrage:Voter ID:server-0f5c1330-1363-a88b-a119-e63ea0d7b1a7 Address:0f5c1330-1363-a88b-a119-e63ea0d7b1a7}]"
2026-09-21T15:24:20.490Z [INFO]  server-7ac2bcc6-cbaf-78a5-59d5-91cece5fa344: entering follower state: follower="Node at 7ac2bcc6-cbaf-78a5-59d5-91cece5fa344 [Follower]" leader-address= leader-id=
2026-09-21T15:24:20.490Z [INFO]  server-1f46660f-ee11-a224-1546-8e5b2c46eec4: entering follower state: follower="Node at 1f46660f-ee11-a224-1546-8e5b2c46eec4 [Follower]" leader-address= leader-id=
2026-09-21T15:24:20.490Z [INFO]  server-0f5c1330-1363-a88b-a119-e63ea0d7b1a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ac2bcc6-cbaf-78a5-59d5-91cece5fa344 Address:7ac2bcc6-cbaf-78a5-59d5-91cece5fa344} {Suffrage:Voter ID:server-1f46660f-ee11-a224-1546-8e5b2c46eec4 Address:1f46660f-ee11-a224-1546-8e5b2c46eec4} {Suffrage:Voter ID:server-0f5c1330-1363-a88b-a119-e63ea0d7b1a7 Address:0f5c1330-1363-a88b-a119-e63ea0d7b1a7}]"
2026-09-21T15:24:20.490Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:20.490Z [INFO]  server-0f5c1330-1363-a88b-a119-e63ea0d7b1a7: entering follower state: follower="Node at 0f5c1330-1363-a88b-a119-e63ea0d7b1a7 [Follower]" leader-address= leader-id=
2026-09-21T15:24:20.572Z [WARN]  server-7ac2bcc6-cbaf-78a5-59d5-91cece5fa344: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:20.572Z [INFO]  server-7ac2bcc6-cbaf-78a5-59d5-91cece5fa344: entering candidate state: node="Node at 7ac2bcc6-cbaf-78a5-59d5-91cece5fa344 [Candidate]" term=2
2026-09-21T15:24:20.572Z [INFO]  server-7ac2bcc6-cbaf-78a5-59d5-91cece5fa344: election won: term=2 tally=2
2026-09-21T15:24:20.572Z [INFO]  server-7ac2bcc6-cbaf-78a5-59d5-91cece5fa344: entering leader state: leader="Node at 7ac2bcc6-cbaf-78a5-59d5-91cece5fa344 [Leader]"
2026-09-21T15:24:20.572Z [INFO]  server-7ac2bcc6-cbaf-78a5-59d5-91cece5fa344: added peer, starting replication: peer=server-1f46660f-ee11-a224-1546-8e5b2c46eec4
2026-09-21T15:24:20.572Z [INFO]  server-7ac2bcc6-cbaf-78a5-59d5-91cece5fa344: added peer, starting replication: peer=server-0f5c1330-1363-a88b-a119-e63ea0d7b1a7
2026-09-21T15:24:20.572Z [INFO]  server-7ac2bcc6-cbaf-78a5-59d5-91cece5fa344: pipelining replication: peer="{Voter server-1f46660f-ee11-a224-1546-8e5b2c46eec4 1f46660f-ee11-a224-1546-8e5b2c46eec4}"
2026-09-21T15:24:20.572Z [INFO]  server-7ac2bcc6-cbaf-78a5-59d5-91cece5fa344: pipelining replication: peer="{Voter server-0f5c1330-1363-a88b-a119-e63ea0d7b1a7 0f5c1330-1363-a88b-a119-e63ea0d7b1a7}"
2026-09-21T15:24:20.678Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:20.572812348 +0000 UTC m=+12.513982521 (1 nodes), 82.02484ms from start of poll, 82.44104ms from cluster start. Timeout at 2026-09-21 15:24:20.677818908 +0000 UTC m=+12.618989081, 105.00656ms after stability
2026-09-21T15:24:20.678Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:20.783Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:20.678061068 +0000 UTC m=+12.619231241 (2 nodes), 2.56µs from start of poll, 187.68976ms from cluster start. Timeout at 2026-09-21 15:24:20.783069308 +0000 UTC m=+12.724239481, 105.00824ms after stability
2026-09-21T15:24:20.783Z [INFO]  server-7ac2bcc6-cbaf-78a5-59d5-91cece5fa344: updating configuration: command=AddVoter server-id=server-1f46660f-ee11-a224-1546-8e5b2c46eec4 server-addr=1f46660f-ee11-a224-1546-8e5b2c46eec4 servers="[{Suffrage:Voter ID:server-7ac2bcc6-cbaf-78a5-59d5-91cece5fa344 Address:7ac2bcc6-cbaf-78a5-59d5-91cece5fa344} {Suffrage:Voter ID:server-1f46660f-ee11-a224-1546-8e5b2c46eec4 Address:1f46660f-ee11-a224-1546-8e5b2c46eec4} {Suffrage:Voter ID:server-0f5c1330-1363-a88b-a119-e63ea0d7b1a7 Address:0f5c1330-1363-a88b-a119-e63ea0d7b1a7}]"
2026-09-21T15:24:20.784Z [INFO]  server-7ac2bcc6-cbaf-78a5-59d5-91cece5fa344: aborting pipeline replication: peer="{Voter server-0f5c1330-1363-a88b-a119-e63ea0d7b1a7 0f5c1330-1363-a88b-a119-e63ea0d7b1a7}"
2026-09-21T15:24:20.784Z [INFO]  server-7ac2bcc6-cbaf-78a5-59d5-91cece5fa344: aborting pipeline replication: peer="{Voter server-1f46660f-ee11-a224-1546-8e5b2c46eec4 1f46660f-ee11-a224-1546-8e5b2c46eec4}"
--- PASS: TestRaft_AddKnownPeer (0.29s)
=== RUN   TestRaft_RemoveUnknownPeer
2026-09-21T15:24:20.785Z [INFO]  server-6f4d71a4-0784-1cfa-74e7-5b3c386e722e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f4d71a4-0784-1cfa-74e7-5b3c386e722e Address:6f4d71a4-0784-1cfa-74e7-5b3c386e722e} {Suffrage:Voter ID:server-350d7fc7-6898-6ee6-7119-790a47f48429 Address:350d7fc7-6898-6ee6-7119-790a47f48429} {Suffrage:Voter ID:server-e31d622f-4989-eb3f-b2a7-ce7adb1c68d2 Address:e31d622f-4989-eb3f-b2a7-ce7adb1c68d2}]"
2026-09-21T15:24:20.785Z [INFO]  server-350d7fc7-6898-6ee6-7119-790a47f48429: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f4d71a4-0784-1cfa-74e7-5b3c386e722e Address:6f4d71a4-0784-1cfa-74e7-5b3c386e722e} {Suffrage:Voter ID:server-350d7fc7-6898-6ee6-7119-790a47f48429 Address:350d7fc7-6898-6ee6-7119-790a47f48429} {Suffrage:Voter ID:server-e31d622f-4989-eb3f-b2a7-ce7adb1c68d2 Address:e31d622f-4989-eb3f-b2a7-ce7adb1c68d2}]"
2026-09-21T15:24:20.785Z [INFO]  server-6f4d71a4-0784-1cfa-74e7-5b3c386e722e: entering follower state: follower="Node at 6f4d71a4-0784-1cfa-74e7-5b3c386e722e [Follower]" leader-address= leader-id=
2026-09-21T15:24:20.785Z [INFO]  server-350d7fc7-6898-6ee6-7119-790a47f48429: entering follower state: follower="Node at 350d7fc7-6898-6ee6-7119-790a47f48429 [Follower]" leader-address= leader-id=
2026-09-21T15:24:20.786Z [INFO]  server-e31d622f-4989-eb3f-b2a7-ce7adb1c68d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f4d71a4-0784-1cfa-74e7-5b3c386e722e Address:6f4d71a4-0784-1cfa-74e7-5b3c386e722e} {Suffrage:Voter ID:server-350d7fc7-6898-6ee6-7119-790a47f48429 Address:350d7fc7-6898-6ee6-7119-790a47f48429} {Suffrage:Voter ID:server-e31d622f-4989-eb3f-b2a7-ce7adb1c68d2 Address:e31d622f-4989-eb3f-b2a7-ce7adb1c68d2}]"
2026-09-21T15:24:20.786Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:20.786Z [INFO]  server-e31d622f-4989-eb3f-b2a7-ce7adb1c68d2: entering follower state: follower="Node at e31d622f-4989-eb3f-b2a7-ce7adb1c68d2 [Follower]" leader-address= leader-id=
2026-09-21T15:24:20.842Z [WARN]  server-6f4d71a4-0784-1cfa-74e7-5b3c386e722e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:20.842Z [INFO]  server-6f4d71a4-0784-1cfa-74e7-5b3c386e722e: entering candidate state: node="Node at 6f4d71a4-0784-1cfa-74e7-5b3c386e722e [Candidate]" term=2
2026-09-21T15:24:20.842Z [INFO]  server-6f4d71a4-0784-1cfa-74e7-5b3c386e722e: election won: term=2 tally=2
2026-09-21T15:24:20.842Z [INFO]  server-6f4d71a4-0784-1cfa-74e7-5b3c386e722e: entering leader state: leader="Node at 6f4d71a4-0784-1cfa-74e7-5b3c386e722e [Leader]"
2026-09-21T15:24:20.842Z [INFO]  server-6f4d71a4-0784-1cfa-74e7-5b3c386e722e: added peer, starting replication: peer=server-350d7fc7-6898-6ee6-7119-790a47f48429
2026-09-21T15:24:20.842Z [INFO]  server-6f4d71a4-0784-1cfa-74e7-5b3c386e722e: added peer, starting replication: peer=server-e31d622f-4989-eb3f-b2a7-ce7adb1c68d2
2026-09-21T15:24:20.842Z [INFO]  server-6f4d71a4-0784-1cfa-74e7-5b3c386e722e: pipelining replication: peer="{Voter server-e31d622f-4989-eb3f-b2a7-ce7adb1c68d2 e31d622f-4989-eb3f-b2a7-ce7adb1c68d2}"
2026-09-21T15:24:20.842Z [INFO]  server-6f4d71a4-0784-1cfa-74e7-5b3c386e722e: pipelining replication: peer="{Voter server-350d7fc7-6898-6ee6-7119-790a47f48429 350d7fc7-6898-6ee6-7119-790a47f48429}"
2026-09-21T15:24:20.948Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:20.842722468 +0000 UTC m=+12.783892641 (1 nodes), 56.61692ms from start of poll, 57.6168ms from cluster start. Timeout at 2026-09-21 15:24:20.947724668 +0000 UTC m=+12.888894841, 105.0022ms after stability
2026-09-21T15:24:20.948Z [INFO]  server-6f4d71a4-0784-1cfa-74e7-5b3c386e722e: updating configuration: command=RemoveServer server-id=a48a13a1-79fa-10c5-d7f4-364af6ad2de6 server-addr= servers="[{Suffrage:Voter ID:server-6f4d71a4-0784-1cfa-74e7-5b3c386e722e Address:6f4d71a4-0784-1cfa-74e7-5b3c386e722e} {Suffrage:Voter ID:server-350d7fc7-6898-6ee6-7119-790a47f48429 Address:350d7fc7-6898-6ee6-7119-790a47f48429} {Suffrage:Voter ID:server-e31d622f-4989-eb3f-b2a7-ce7adb1c68d2 Address:e31d622f-4989-eb3f-b2a7-ce7adb1c68d2}]"
2026-09-21T15:24:20.948Z [INFO]  server-6f4d71a4-0784-1cfa-74e7-5b3c386e722e: aborting pipeline replication: peer="{Voter server-350d7fc7-6898-6ee6-7119-790a47f48429 350d7fc7-6898-6ee6-7119-790a47f48429}"
2026-09-21T15:24:20.948Z [INFO]  server-6f4d71a4-0784-1cfa-74e7-5b3c386e722e: aborting pipeline replication: peer="{Voter server-e31d622f-4989-eb3f-b2a7-ce7adb1c68d2 e31d622f-4989-eb3f-b2a7-ce7adb1c68d2}"
--- PASS: TestRaft_RemoveUnknownPeer (0.16s)
=== RUN   TestRaft_SnapshotRestore
2026-09-21T15:24:20.949Z [INFO]  server-e1e24e52-99b7-17b1-c152-370bf9bf05ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1e24e52-99b7-17b1-c152-370bf9bf05ce Address:e1e24e52-99b7-17b1-c152-370bf9bf05ce}]"
2026-09-21T15:24:20.949Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:20.949Z [INFO]  server-e1e24e52-99b7-17b1-c152-370bf9bf05ce: entering follower state: follower="Node at e1e24e52-99b7-17b1-c152-370bf9bf05ce [Follower]" leader-address= leader-id=
2026-09-21T15:24:21.048Z [WARN]  server-e1e24e52-99b7-17b1-c152-370bf9bf05ce: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:21.048Z [INFO]  server-e1e24e52-99b7-17b1-c152-370bf9bf05ce: entering candidate state: node="Node at e1e24e52-99b7-17b1-c152-370bf9bf05ce [Candidate]" term=2
2026-09-21T15:24:21.048Z [INFO]  server-e1e24e52-99b7-17b1-c152-370bf9bf05ce: election won: term=2 tally=1
2026-09-21T15:24:21.048Z [INFO]  server-e1e24e52-99b7-17b1-c152-370bf9bf05ce: entering leader state: leader="Node at e1e24e52-99b7-17b1-c152-370bf9bf05ce [Leader]"
2026-09-21T15:24:21.154Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:21.048345268 +0000 UTC m=+12.989515441 (1 nodes), 98.91384ms from start of poll, 99.08384ms from cluster start. Timeout at 2026-09-21 15:24:21.153352188 +0000 UTC m=+13.094522361, 105.00692ms after stability
2026-09-21T15:24:21.156Z [INFO]  server-e1e24e52-99b7-17b1-c152-370bf9bf05ce: starting snapshot up to: index=102
2026-09-21T15:24:21.156Z [INFO]  creating new snapshot: path=/tmp/raft2146636685/snapshots/2-102-1790004261156.tmp
2026-09-21T15:24:21.157Z [INFO]  server-e1e24e52-99b7-17b1-c152-370bf9bf05ce: compacting logs: from=1 to=92
2026-09-21T15:24:21.157Z [INFO]  server-e1e24e52-99b7-17b1-c152-370bf9bf05ce: snapshot complete up to: index=102
2026-09-21T15:24:21.157Z [INFO]  server-e1e24e52-99b7-17b1-c152-370bf9bf05ce: starting restore from snapshot: id=2-102-1790004261156 last-index=102 last-term=2 size-in-bytes=693
2026-09-21T15:24:21.158Z [INFO]  server-e1e24e52-99b7-17b1-c152-370bf9bf05ce: snapshot restore progress: id=2-102-1790004261156 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-09-21T15:24:21.158Z [INFO]  server-e1e24e52-99b7-17b1-c152-370bf9bf05ce: restored from snapshot: id=2-102-1790004261156 last-index=102 last-term=2 size-in-bytes=693
2026-09-21T15:24:21.158Z [INFO]  server-e1e24e52-99b7-17b1-c152-370bf9bf05ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1e24e52-99b7-17b1-c152-370bf9bf05ce Address:e1e24e52-99b7-17b1-c152-370bf9bf05ce}]"
--- PASS: TestRaft_SnapshotRestore (0.21s)
=== RUN   TestRaft_RestoreSnapshotOnStartup_Monotonic
2026-09-21T15:24:21.158Z [INFO]  server-1c695c28-d4b7-d091-8f60-ea6157adf82a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c695c28-d4b7-d091-8f60-ea6157adf82a Address:1c695c28-d4b7-d091-8f60-ea6157adf82a}]"
2026-09-21T15:24:21.158Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:21.158Z [INFO]  server-1c695c28-d4b7-d091-8f60-ea6157adf82a: entering follower state: follower="Node at 1c695c28-d4b7-d091-8f60-ea6157adf82a [Follower]" leader-address= leader-id=
2026-09-21T15:24:21.245Z [WARN]  server-1c695c28-d4b7-d091-8f60-ea6157adf82a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:21.245Z [INFO]  server-1c695c28-d4b7-d091-8f60-ea6157adf82a: entering candidate state: node="Node at 1c695c28-d4b7-d091-8f60-ea6157adf82a [Candidate]" term=2
2026-09-21T15:24:21.245Z [INFO]  server-1c695c28-d4b7-d091-8f60-ea6157adf82a: election won: term=2 tally=1
2026-09-21T15:24:21.245Z [INFO]  server-1c695c28-d4b7-d091-8f60-ea6157adf82a: entering leader state: leader="Node at 1c695c28-d4b7-d091-8f60-ea6157adf82a [Leader]"
2026-09-21T15:24:21.350Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:21.245675708 +0000 UTC m=+13.186845881 (1 nodes), 86.8902ms from start of poll, 87.05348ms from cluster start. Timeout at 2026-09-21 15:24:21.350676908 +0000 UTC m=+13.291847081, 105.0012ms after stability
2026-09-21T15:24:21.354Z [INFO]  server-1c695c28-d4b7-d091-8f60-ea6157adf82a: starting snapshot up to: index=102
2026-09-21T15:24:21.354Z [INFO]  creating new snapshot: path=/tmp/raft2764572986/snapshots/2-102-1790004261354.tmp
2026-09-21T15:24:21.354Z [INFO]  server-1c695c28-d4b7-d091-8f60-ea6157adf82a: compacting logs: from=1 to=92
2026-09-21T15:24:21.354Z [INFO]  server-1c695c28-d4b7-d091-8f60-ea6157adf82a: snapshot complete up to: index=102
2026-09-21T15:24:21.354Z [INFO]  server-1c695c28-d4b7-d091-8f60-ea6157adf82a: starting restore from snapshot: id=2-102-1790004261354 last-index=102 last-term=2 size-in-bytes=693
2026-09-21T15:24:21.355Z [INFO]  server-1c695c28-d4b7-d091-8f60-ea6157adf82a: snapshot restore progress: id=2-102-1790004261354 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-09-21T15:24:21.355Z [INFO]  server-1c695c28-d4b7-d091-8f60-ea6157adf82a: restored from snapshot: id=2-102-1790004261354 last-index=102 last-term=2 size-in-bytes=693
2026-09-21T15:24:21.355Z [INFO]  server-1c695c28-d4b7-d091-8f60-ea6157adf82a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c695c28-d4b7-d091-8f60-ea6157adf82a Address:1c695c28-d4b7-d091-8f60-ea6157adf82a}]"
--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.20s)
=== RUN   TestRaft_SnapshotRestore_Progress
2026-09-21T15:24:21.355Z [INFO]  server-e2a73a9c-d6f3-34a6-e394-bc4b397d5f3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e2a73a9c-d6f3-34a6-e394-bc4b397d5f3b Address:e2a73a9c-d6f3-34a6-e394-bc4b397d5f3b}]"
2026-09-21T15:24:21.355Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:21.355Z [INFO]  server-e2a73a9c-d6f3-34a6-e394-bc4b397d5f3b: entering follower state: follower="Node at e2a73a9c-d6f3-34a6-e394-bc4b397d5f3b [Follower]" leader-address= leader-id=
2026-09-21T15:24:21.447Z [WARN]  server-e2a73a9c-d6f3-34a6-e394-bc4b397d5f3b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:21.447Z [INFO]  server-e2a73a9c-d6f3-34a6-e394-bc4b397d5f3b: entering candidate state: node="Node at e2a73a9c-d6f3-34a6-e394-bc4b397d5f3b [Candidate]" term=2
2026-09-21T15:24:21.447Z [INFO]  server-e2a73a9c-d6f3-34a6-e394-bc4b397d5f3b: election won: term=2 tally=1
2026-09-21T15:24:21.447Z [INFO]  server-e2a73a9c-d6f3-34a6-e394-bc4b397d5f3b: entering leader state: leader="Node at e2a73a9c-d6f3-34a6-e394-bc4b397d5f3b [Leader]"
2026-09-21T15:24:21.553Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:21.447811668 +0000 UTC m=+13.388981841 (1 nodes), 91.91652ms from start of poll, 92.08096ms from cluster start. Timeout at 2026-09-21 15:24:21.552813268 +0000 UTC m=+13.493983441, 105.0016ms after stability
2026-09-21T15:24:21.557Z [INFO]  server-e2a73a9c-d6f3-34a6-e394-bc4b397d5f3b: starting snapshot up to: index=102
2026-09-21T15:24:21.557Z [INFO]  creating new snapshot: path=/tmp/raft145848936/snapshots/2-102-1790004261557.tmp
2026-09-21T15:24:21.557Z [INFO]  server-e2a73a9c-d6f3-34a6-e394-bc4b397d5f3b: compacting logs: from=1 to=92
2026-09-21T15:24:21.557Z [INFO]  server-e2a73a9c-d6f3-34a6-e394-bc4b397d5f3b: snapshot complete up to: index=102
--- PASS: TestRaft_SnapshotRestore_Progress (0.20s)
=== RUN   TestRaft_NoRestoreOnStart
2026-09-21T15:24:21.559Z [INFO]  server-fa08baeb-721d-4df4-2694-7986919e3f78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa08baeb-721d-4df4-2694-7986919e3f78 Address:fa08baeb-721d-4df4-2694-7986919e3f78}]"
2026-09-21T15:24:21.559Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:21.559Z [INFO]  server-fa08baeb-721d-4df4-2694-7986919e3f78: entering follower state: follower="Node at fa08baeb-721d-4df4-2694-7986919e3f78 [Follower]" leader-address= leader-id=
2026-09-21T15:24:21.641Z [WARN]  server-fa08baeb-721d-4df4-2694-7986919e3f78: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:21.641Z [INFO]  server-fa08baeb-721d-4df4-2694-7986919e3f78: entering candidate state: node="Node at fa08baeb-721d-4df4-2694-7986919e3f78 [Candidate]" term=2
2026-09-21T15:24:21.641Z [INFO]  server-fa08baeb-721d-4df4-2694-7986919e3f78: election won: term=2 tally=1
2026-09-21T15:24:21.641Z [INFO]  server-fa08baeb-721d-4df4-2694-7986919e3f78: entering leader state: leader="Node at fa08baeb-721d-4df4-2694-7986919e3f78 [Leader]"
2026-09-21T15:24:21.746Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:21.641383788 +0000 UTC m=+13.582553961 (1 nodes), 81.8958ms from start of poll, 82.14416ms from cluster start. Timeout at 2026-09-21 15:24:21.746386868 +0000 UTC m=+13.687557041, 105.00308ms after stability
2026-09-21T15:24:21.749Z [INFO]  server-fa08baeb-721d-4df4-2694-7986919e3f78: starting snapshot up to: index=102
2026-09-21T15:24:21.749Z [INFO]  creating new snapshot: path=/tmp/raft964000465/snapshots/2-102-1790004261749.tmp
2026-09-21T15:24:21.750Z [INFO]  server-fa08baeb-721d-4df4-2694-7986919e3f78: compacting logs: from=1 to=92
2026-09-21T15:24:21.750Z [INFO]  server-fa08baeb-721d-4df4-2694-7986919e3f78: snapshot complete up to: index=102
2026-09-21T15:24:21.750Z [INFO]  server-fa08baeb-721d-4df4-2694-7986919e3f78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa08baeb-721d-4df4-2694-7986919e3f78 Address:fa08baeb-721d-4df4-2694-7986919e3f78}]"
--- PASS: TestRaft_NoRestoreOnStart (0.19s)
=== RUN   TestRaft_SnapshotRestore_PeerChange
2026-09-21T15:24:21.750Z [INFO]  server-fa08baeb-721d-4df4-2694-7986919e3f78: entering follower state: follower="Node at fa08baeb-721d-4df4-2694-7986919e3f78 [Follower]" leader-address= leader-id=
    raft_test.go:1266: Skipping unreliable test
--- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
=== RUN   TestRaft_AutoSnapshot
2026-09-21T15:24:21.750Z [INFO]  server-4d751ba5-cede-37b0-5285-9ddc1301ef89: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d751ba5-cede-37b0-5285-9ddc1301ef89 Address:4d751ba5-cede-37b0-5285-9ddc1301ef89}]"
2026-09-21T15:24:21.751Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:21.751Z [INFO]  server-4d751ba5-cede-37b0-5285-9ddc1301ef89: entering follower state: follower="Node at 4d751ba5-cede-37b0-5285-9ddc1301ef89 [Follower]" leader-address= leader-id=
2026-09-21T15:24:21.837Z [WARN]  server-fa08baeb-721d-4df4-2694-7986919e3f78: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:21.837Z [WARN]  server-4d751ba5-cede-37b0-5285-9ddc1301ef89: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:21.837Z [INFO]  server-fa08baeb-721d-4df4-2694-7986919e3f78: entering candidate state: node="Node at fa08baeb-721d-4df4-2694-7986919e3f78 [Candidate]" term=3
2026-09-21T15:24:21.837Z [INFO]  server-4d751ba5-cede-37b0-5285-9ddc1301ef89: entering candidate state: node="Node at 4d751ba5-cede-37b0-5285-9ddc1301ef89 [Candidate]" term=2
2026-09-21T15:24:21.837Z [INFO]  server-fa08baeb-721d-4df4-2694-7986919e3f78: election won: term=3 tally=1
2026-09-21T15:24:21.837Z [INFO]  server-4d751ba5-cede-37b0-5285-9ddc1301ef89: election won: term=2 tally=1
2026-09-21T15:24:21.837Z [INFO]  server-fa08baeb-721d-4df4-2694-7986919e3f78: entering leader state: leader="Node at fa08baeb-721d-4df4-2694-7986919e3f78 [Leader]"
2026-09-21T15:24:21.837Z [INFO]  server-4d751ba5-cede-37b0-5285-9ddc1301ef89: entering leader state: leader="Node at 4d751ba5-cede-37b0-5285-9ddc1301ef89 [Leader]"
2026-09-21T15:24:21.943Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:21.837902188 +0000 UTC m=+13.779072401 (1 nodes), 86.87804ms from start of poll, 87.04188ms from cluster start. Timeout at 2026-09-21 15:24:21.942906828 +0000 UTC m=+13.884077001, 105.0046ms after stability
2026-09-21T15:24:21.949Z [INFO]  server-4d751ba5-cede-37b0-5285-9ddc1301ef89: starting snapshot up to: index=102
2026-09-21T15:24:21.949Z [INFO]  creating new snapshot: path=/tmp/raft1465946141/snapshots/2-102-1790004261949.tmp
2026-09-21T15:24:21.949Z [INFO]  server-4d751ba5-cede-37b0-5285-9ddc1301ef89: compacting logs: from=1 to=92
2026-09-21T15:24:21.949Z [INFO]  server-4d751ba5-cede-37b0-5285-9ddc1301ef89: snapshot complete up to: index=102
--- PASS: TestRaft_AutoSnapshot (0.30s)
=== RUN   TestRaft_UserSnapshot
2026-09-21T15:24:22.052Z [INFO]  server-68f294c2-2ee3-12e6-bc05-6b373cdfe0f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68f294c2-2ee3-12e6-bc05-6b373cdfe0f3 Address:68f294c2-2ee3-12e6-bc05-6b373cdfe0f3}]"
2026-09-21T15:24:22.052Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:22.052Z [INFO]  server-68f294c2-2ee3-12e6-bc05-6b373cdfe0f3: entering follower state: follower="Node at 68f294c2-2ee3-12e6-bc05-6b373cdfe0f3 [Follower]" leader-address= leader-id=
2026-09-21T15:24:22.136Z [WARN]  server-68f294c2-2ee3-12e6-bc05-6b373cdfe0f3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:22.136Z [INFO]  server-68f294c2-2ee3-12e6-bc05-6b373cdfe0f3: entering candidate state: node="Node at 68f294c2-2ee3-12e6-bc05-6b373cdfe0f3 [Candidate]" term=2
2026-09-21T15:24:22.136Z [INFO]  server-68f294c2-2ee3-12e6-bc05-6b373cdfe0f3: election won: term=2 tally=1
2026-09-21T15:24:22.136Z [INFO]  server-68f294c2-2ee3-12e6-bc05-6b373cdfe0f3: entering leader state: leader="Node at 68f294c2-2ee3-12e6-bc05-6b373cdfe0f3 [Leader]"
2026-09-21T15:24:22.242Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:22.136474268 +0000 UTC m=+14.077644441 (1 nodes), 83.68652ms from start of poll, 83.86448ms from cluster start. Timeout at 2026-09-21 15:24:22.241477508 +0000 UTC m=+14.182647681, 105.00324ms after stability
2026-09-21T15:24:22.242Z [ERROR] server-68f294c2-2ee3-12e6-bc05-6b373cdfe0f3: failed to take snapshot: error="nothing new to snapshot"
2026-09-21T15:24:22.242Z [INFO]  server-68f294c2-2ee3-12e6-bc05-6b373cdfe0f3: starting snapshot up to: index=12
2026-09-21T15:24:22.242Z [INFO]  creating new snapshot: path=/tmp/raft662316138/snapshots/2-12-1790004262242.tmp
2026-09-21T15:24:22.243Z [INFO]  server-68f294c2-2ee3-12e6-bc05-6b373cdfe0f3: compacting logs: from=1 to=2
2026-09-21T15:24:22.243Z [INFO]  server-68f294c2-2ee3-12e6-bc05-6b373cdfe0f3: snapshot complete up to: index=12
--- PASS: TestRaft_UserSnapshot (0.19s)
=== RUN   TestRaft_UserRestore
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false
2026-09-21T15:24:22.244Z [INFO]  server-c891bd03-c5e2-b5d7-0db7-c969d8beb572: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c891bd03-c5e2-b5d7-0db7-c969d8beb572 Address:c891bd03-c5e2-b5d7-0db7-c969d8beb572} {Suffrage:Voter ID:server-4b1f9000-2e1e-5eff-0f2f-557642e55f21 Address:4b1f9000-2e1e-5eff-0f2f-557642e55f21} {Suffrage:Voter ID:server-318e163e-3348-20e3-d6e7-0df297ec46e3 Address:318e163e-3348-20e3-d6e7-0df297ec46e3}]"
2026-09-21T15:24:22.244Z [INFO]  server-c891bd03-c5e2-b5d7-0db7-c969d8beb572: entering follower state: follower="Node at c891bd03-c5e2-b5d7-0db7-c969d8beb572 [Follower]" leader-address= leader-id=
2026-09-21T15:24:22.244Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c891bd03-c5e2-b5d7-0db7-c969d8beb572 Address:c891bd03-c5e2-b5d7-0db7-c969d8beb572} {Suffrage:Voter ID:server-4b1f9000-2e1e-5eff-0f2f-557642e55f21 Address:4b1f9000-2e1e-5eff-0f2f-557642e55f21} {Suffrage:Voter ID:server-318e163e-3348-20e3-d6e7-0df297ec46e3 Address:318e163e-3348-20e3-d6e7-0df297ec46e3}]"
2026-09-21T15:24:22.244Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: entering follower state: follower="Node at 4b1f9000-2e1e-5eff-0f2f-557642e55f21 [Follower]" leader-address= leader-id=
2026-09-21T15:24:22.244Z [INFO]  server-318e163e-3348-20e3-d6e7-0df297ec46e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c891bd03-c5e2-b5d7-0db7-c969d8beb572 Address:c891bd03-c5e2-b5d7-0db7-c969d8beb572} {Suffrage:Voter ID:server-4b1f9000-2e1e-5eff-0f2f-557642e55f21 Address:4b1f9000-2e1e-5eff-0f2f-557642e55f21} {Suffrage:Voter ID:server-318e163e-3348-20e3-d6e7-0df297ec46e3 Address:318e163e-3348-20e3-d6e7-0df297ec46e3}]"
2026-09-21T15:24:22.244Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:22.244Z [INFO]  server-318e163e-3348-20e3-d6e7-0df297ec46e3: entering follower state: follower="Node at 318e163e-3348-20e3-d6e7-0df297ec46e3 [Follower]" leader-address= leader-id=
2026-09-21T15:24:22.778Z [WARN]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:22.779Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: entering candidate state: node="Node at 4b1f9000-2e1e-5eff-0f2f-557642e55f21 [Candidate]" term=2
2026-09-21T15:24:22.779Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: election won: term=2 tally=2
2026-09-21T15:24:22.779Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: entering leader state: leader="Node at 4b1f9000-2e1e-5eff-0f2f-557642e55f21 [Leader]"
2026-09-21T15:24:22.779Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: added peer, starting replication: peer=server-c891bd03-c5e2-b5d7-0db7-c969d8beb572
2026-09-21T15:24:22.779Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: added peer, starting replication: peer=server-318e163e-3348-20e3-d6e7-0df297ec46e3
2026-09-21T15:24:22.779Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: pipelining replication: peer="{Voter server-318e163e-3348-20e3-d6e7-0df297ec46e3 318e163e-3348-20e3-d6e7-0df297ec46e3}"
2026-09-21T15:24:22.779Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: pipelining replication: peer="{Voter server-c891bd03-c5e2-b5d7-0db7-c969d8beb572 c891bd03-c5e2-b5d7-0db7-c969d8beb572}"
2026-09-21T15:24:23.784Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:22.779173988 +0000 UTC m=+14.720344161 (1 nodes), 534.57048ms from start of poll, 535.08496ms from cluster start. Timeout at 2026-09-21 15:24:23.784178108 +0000 UTC m=+15.725348321, 1.00500416s after stability
2026-09-21T15:24:23.786Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: starting snapshot up to: index=12
2026-09-21T15:24:23.786Z [INFO]  creating new snapshot: path=/tmp/raft1656061194/snapshots/2-12-1790004263786.tmp
2026-09-21T15:24:23.787Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: snapshot complete up to: index=12
2026-09-21T15:24:23.788Z [INFO]  creating new snapshot: path=/tmp/raft1656061194/snapshots/2-23-1790004263788.tmp
2026-09-21T15:24:23.788Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: copied to local snapshot: bytes=71
2026-09-21T15:24:23.788Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: snapshot restore progress: id=2-23-1790004263788 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:23.788Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: restored user snapshot: index=23
2026-09-21T15:24:23.788Z [ERROR] server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: failed to get log: index=23 error="log not found"
2026-09-21T15:24:23.788Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: aborting pipeline replication: peer="{Voter server-318e163e-3348-20e3-d6e7-0df297ec46e3 318e163e-3348-20e3-d6e7-0df297ec46e3}"
2026-09-21T15:24:23.788Z [ERROR] server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: failed to get log: index=23 error="log not found"
2026-09-21T15:24:23.788Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: aborting pipeline replication: peer="{Voter server-c891bd03-c5e2-b5d7-0db7-c969d8beb572 c891bd03-c5e2-b5d7-0db7-c969d8beb572}"
2026-09-21T15:24:23.797Z [ERROR] server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: failed to get log: index=23 error="log not found"
2026-09-21T15:24:23.797Z [INFO]  creating new snapshot: path=/tmp/raft4261809996/snapshots/2-23-1790004263797.tmp
2026-09-21T15:24:23.798Z [INFO]  server-318e163e-3348-20e3-d6e7-0df297ec46e3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:23.798Z [INFO]  server-318e163e-3348-20e3-d6e7-0df297ec46e3: copied to local snapshot: bytes=71
2026-09-21T15:24:23.798Z [INFO]  server-318e163e-3348-20e3-d6e7-0df297ec46e3: snapshot restore progress: id=2-23-1790004263797 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:23.798Z [INFO]  server-318e163e-3348-20e3-d6e7-0df297ec46e3: Installed remote snapshot
2026-09-21T15:24:23.798Z [ERROR] server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: failed to get log: index=23 error="log not found"
2026-09-21T15:24:23.798Z [WARN]  server-318e163e-3348-20e3-d6e7-0df297ec46e3: skipping application of old log: index=22
2026-09-21T15:24:23.799Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: pipelining replication: peer="{Voter server-318e163e-3348-20e3-d6e7-0df297ec46e3 318e163e-3348-20e3-d6e7-0df297ec46e3}"
2026-09-21T15:24:23.799Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:23.799Z [INFO]  creating new snapshot: path=/tmp/raft865485655/snapshots/2-23-1790004263799.tmp
2026-09-21T15:24:23.799Z [INFO]  server-c891bd03-c5e2-b5d7-0db7-c969d8beb572: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:23.799Z [INFO]  server-c891bd03-c5e2-b5d7-0db7-c969d8beb572: copied to local snapshot: bytes=71
2026-09-21T15:24:23.800Z [INFO]  server-c891bd03-c5e2-b5d7-0db7-c969d8beb572: snapshot restore progress: id=2-23-1790004263799 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:23.800Z [INFO]  server-c891bd03-c5e2-b5d7-0db7-c969d8beb572: Installed remote snapshot
2026-09-21T15:24:23.800Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: pipelining replication: peer="{Voter server-c891bd03-c5e2-b5d7-0db7-c969d8beb572 c891bd03-c5e2-b5d7-0db7-c969d8beb572}"
2026-09-21T15:24:24.804Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:23.799143308 +0000 UTC m=+15.740313481 (2 nodes), 880ns from start of poll, 1.55505428s from cluster start. Timeout at 2026-09-21 15:24:24.804151508 +0000 UTC m=+16.745321721, 1.00500824s after stability
2026-09-21T15:24:24.817Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: aborting pipeline replication: peer="{Voter server-318e163e-3348-20e3-d6e7-0df297ec46e3 318e163e-3348-20e3-d6e7-0df297ec46e3}"
2026-09-21T15:24:24.817Z [INFO]  server-4b1f9000-2e1e-5eff-0f2f-557642e55f21: aborting pipeline replication: peer="{Voter server-c891bd03-c5e2-b5d7-0db7-c969d8beb572 c891bd03-c5e2-b5d7-0db7-c969d8beb572}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false
2026-09-21T15:24:24.818Z [INFO]  server-75464b18-4fac-df07-19ac-087a898515fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75464b18-4fac-df07-19ac-087a898515fb Address:75464b18-4fac-df07-19ac-087a898515fb} {Suffrage:Voter ID:server-1d341b49-f025-5e68-9f6c-465193d92d15 Address:1d341b49-f025-5e68-9f6c-465193d92d15} {Suffrage:Voter ID:server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc Address:30ecfd79-35af-84a0-f6bf-66bccf0edbdc}]"
2026-09-21T15:24:24.818Z [INFO]  server-1d341b49-f025-5e68-9f6c-465193d92d15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75464b18-4fac-df07-19ac-087a898515fb Address:75464b18-4fac-df07-19ac-087a898515fb} {Suffrage:Voter ID:server-1d341b49-f025-5e68-9f6c-465193d92d15 Address:1d341b49-f025-5e68-9f6c-465193d92d15} {Suffrage:Voter ID:server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc Address:30ecfd79-35af-84a0-f6bf-66bccf0edbdc}]"
2026-09-21T15:24:24.818Z [INFO]  server-75464b18-4fac-df07-19ac-087a898515fb: entering follower state: follower="Node at 75464b18-4fac-df07-19ac-087a898515fb [Follower]" leader-address= leader-id=
2026-09-21T15:24:24.818Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-75464b18-4fac-df07-19ac-087a898515fb Address:75464b18-4fac-df07-19ac-087a898515fb} {Suffrage:Voter ID:server-1d341b49-f025-5e68-9f6c-465193d92d15 Address:1d341b49-f025-5e68-9f6c-465193d92d15} {Suffrage:Voter ID:server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc Address:30ecfd79-35af-84a0-f6bf-66bccf0edbdc}]"
2026-09-21T15:24:24.818Z [INFO]  server-1d341b49-f025-5e68-9f6c-465193d92d15: entering follower state: follower="Node at 1d341b49-f025-5e68-9f6c-465193d92d15 [Follower]" leader-address= leader-id=
2026-09-21T15:24:24.818Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:24.819Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: entering follower state: follower="Node at 30ecfd79-35af-84a0-f6bf-66bccf0edbdc [Follower]" leader-address= leader-id=
2026-09-21T15:24:25.645Z [WARN]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:25.645Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: entering candidate state: node="Node at 30ecfd79-35af-84a0-f6bf-66bccf0edbdc [Candidate]" term=2
2026-09-21T15:24:25.645Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: election won: term=2 tally=2
2026-09-21T15:24:25.645Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: entering leader state: leader="Node at 30ecfd79-35af-84a0-f6bf-66bccf0edbdc [Leader]"
2026-09-21T15:24:25.645Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: added peer, starting replication: peer=server-75464b18-4fac-df07-19ac-087a898515fb
2026-09-21T15:24:25.645Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: added peer, starting replication: peer=server-1d341b49-f025-5e68-9f6c-465193d92d15
2026-09-21T15:24:25.645Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: pipelining replication: peer="{Voter server-1d341b49-f025-5e68-9f6c-465193d92d15 1d341b49-f025-5e68-9f6c-465193d92d15}"
2026-09-21T15:24:25.645Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: pipelining replication: peer="{Voter server-75464b18-4fac-df07-19ac-087a898515fb 75464b18-4fac-df07-19ac-087a898515fb}"
2026-09-21T15:24:26.651Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:25.645793388 +0000 UTC m=+17.586963561 (1 nodes), 826.7792ms from start of poll, 827.21636ms from cluster start. Timeout at 2026-09-21 15:24:26.650795508 +0000 UTC m=+18.591965681, 1.00500212s after stability
2026-09-21T15:24:26.652Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: starting snapshot up to: index=12
2026-09-21T15:24:26.652Z [INFO]  creating new snapshot: path=/tmp/raft2221168589/snapshots/2-12-1790004266652.tmp
2026-09-21T15:24:26.652Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: snapshot complete up to: index=12
2026-09-21T15:24:26.653Z [INFO]  creating new snapshot: path=/tmp/raft2221168589/snapshots/2-23-1790004266653.tmp
2026-09-21T15:24:26.653Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: copied to local snapshot: bytes=71
2026-09-21T15:24:26.654Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: snapshot restore progress: id=2-23-1790004266653 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:26.654Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: removing all old logs from log store
2026-09-21T15:24:26.654Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: compacting logs: from=1 to=22
2026-09-21T15:24:26.654Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: restored user snapshot: index=23
2026-09-21T15:24:26.654Z [ERROR] server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: failed to get log: index=22 error="log not found"
2026-09-21T15:24:26.654Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: aborting pipeline replication: peer="{Voter server-1d341b49-f025-5e68-9f6c-465193d92d15 1d341b49-f025-5e68-9f6c-465193d92d15}"
2026-09-21T15:24:26.654Z [ERROR] server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: failed to get log: index=22 error="log not found"
2026-09-21T15:24:26.654Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: aborting pipeline replication: peer="{Voter server-75464b18-4fac-df07-19ac-087a898515fb 75464b18-4fac-df07-19ac-087a898515fb}"
2026-09-21T15:24:26.660Z [ERROR] server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: failed to get log: index=22 error="log not found"
2026-09-21T15:24:26.661Z [ERROR] server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: failed to get log: index=22 error="log not found"
2026-09-21T15:24:26.662Z [INFO]  creating new snapshot: path=/tmp/raft2390204973/snapshots/2-23-1790004266662.tmp
2026-09-21T15:24:26.662Z [INFO]  creating new snapshot: path=/tmp/raft3877898302/snapshots/2-23-1790004266662.tmp
2026-09-21T15:24:26.662Z [INFO]  server-75464b18-4fac-df07-19ac-087a898515fb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:26.662Z [INFO]  server-1d341b49-f025-5e68-9f6c-465193d92d15: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:26.663Z [INFO]  server-75464b18-4fac-df07-19ac-087a898515fb: copied to local snapshot: bytes=71
2026-09-21T15:24:26.663Z [INFO]  server-1d341b49-f025-5e68-9f6c-465193d92d15: copied to local snapshot: bytes=71
2026-09-21T15:24:26.663Z [INFO]  server-75464b18-4fac-df07-19ac-087a898515fb: snapshot restore progress: id=2-23-1790004266662 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:26.663Z [INFO]  server-1d341b49-f025-5e68-9f6c-465193d92d15: snapshot restore progress: id=2-23-1790004266662 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:26.663Z [INFO]  server-75464b18-4fac-df07-19ac-087a898515fb: removing all old logs from log store
2026-09-21T15:24:26.663Z [INFO]  server-75464b18-4fac-df07-19ac-087a898515fb: compacting logs: from=1 to=22
2026-09-21T15:24:26.663Z [INFO]  server-75464b18-4fac-df07-19ac-087a898515fb: Installed remote snapshot
2026-09-21T15:24:26.663Z [INFO]  server-1d341b49-f025-5e68-9f6c-465193d92d15: removing all old logs from log store
2026-09-21T15:24:26.663Z [WARN]  server-75464b18-4fac-df07-19ac-087a898515fb: skipping application of old log: index=22
2026-09-21T15:24:26.663Z [INFO]  server-1d341b49-f025-5e68-9f6c-465193d92d15: compacting logs: from=1 to=22
2026-09-21T15:24:26.663Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: pipelining replication: peer="{Voter server-75464b18-4fac-df07-19ac-087a898515fb 75464b18-4fac-df07-19ac-087a898515fb}"
2026-09-21T15:24:26.663Z [INFO]  server-1d341b49-f025-5e68-9f6c-465193d92d15: Installed remote snapshot
2026-09-21T15:24:26.663Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:26.663Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: pipelining replication: peer="{Voter server-1d341b49-f025-5e68-9f6c-465193d92d15 1d341b49-f025-5e68-9f6c-465193d92d15}"
2026-09-21T15:24:27.669Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:26.663704188 +0000 UTC m=+18.604874361 (2 nodes), 720ns from start of poll, 1.84512716s from cluster start. Timeout at 2026-09-21 15:24:27.668708708 +0000 UTC m=+19.609878921, 1.00500456s after stability
2026-09-21T15:24:27.681Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: aborting pipeline replication: peer="{Voter server-1d341b49-f025-5e68-9f6c-465193d92d15 1d341b49-f025-5e68-9f6c-465193d92d15}"
2026-09-21T15:24:27.681Z [INFO]  server-30ecfd79-35af-84a0-f6bf-66bccf0edbdc: aborting pipeline replication: peer="{Voter server-75464b18-4fac-df07-19ac-087a898515fb 75464b18-4fac-df07-19ac-087a898515fb}"
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true
2026-09-21T15:24:27.683Z [INFO]  server-2f76599f-e04f-a360-85dd-72458dd4acd7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f76599f-e04f-a360-85dd-72458dd4acd7 Address:2f76599f-e04f-a360-85dd-72458dd4acd7} {Suffrage:Voter ID:server-b2acdeca-dc5d-ce98-48a7-ff58a67eb920 Address:b2acdeca-dc5d-ce98-48a7-ff58a67eb920} {Suffrage:Voter ID:server-fc47edc0-a1e7-3d9f-3b8b-7507d2455483 Address:fc47edc0-a1e7-3d9f-3b8b-7507d2455483}]"
2026-09-21T15:24:27.683Z [INFO]  server-b2acdeca-dc5d-ce98-48a7-ff58a67eb920: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f76599f-e04f-a360-85dd-72458dd4acd7 Address:2f76599f-e04f-a360-85dd-72458dd4acd7} {Suffrage:Voter ID:server-b2acdeca-dc5d-ce98-48a7-ff58a67eb920 Address:b2acdeca-dc5d-ce98-48a7-ff58a67eb920} {Suffrage:Voter ID:server-fc47edc0-a1e7-3d9f-3b8b-7507d2455483 Address:fc47edc0-a1e7-3d9f-3b8b-7507d2455483}]"
2026-09-21T15:24:27.683Z [INFO]  server-2f76599f-e04f-a360-85dd-72458dd4acd7: entering follower state: follower="Node at 2f76599f-e04f-a360-85dd-72458dd4acd7 [Follower]" leader-address= leader-id=
2026-09-21T15:24:27.683Z [INFO]  server-b2acdeca-dc5d-ce98-48a7-ff58a67eb920: entering follower state: follower="Node at b2acdeca-dc5d-ce98-48a7-ff58a67eb920 [Follower]" leader-address= leader-id=
2026-09-21T15:24:27.683Z [INFO]  server-fc47edc0-a1e7-3d9f-3b8b-7507d2455483: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f76599f-e04f-a360-85dd-72458dd4acd7 Address:2f76599f-e04f-a360-85dd-72458dd4acd7} {Suffrage:Voter ID:server-b2acdeca-dc5d-ce98-48a7-ff58a67eb920 Address:b2acdeca-dc5d-ce98-48a7-ff58a67eb920} {Suffrage:Voter ID:server-fc47edc0-a1e7-3d9f-3b8b-7507d2455483 Address:fc47edc0-a1e7-3d9f-3b8b-7507d2455483}]"
2026-09-21T15:24:27.683Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:27.683Z [INFO]  server-fc47edc0-a1e7-3d9f-3b8b-7507d2455483: entering follower state: follower="Node at fc47edc0-a1e7-3d9f-3b8b-7507d2455483 [Follower]" leader-address= leader-id=
2026-09-21T15:24:28.344Z [WARN]  server-fc47edc0-a1e7-3d9f-3b8b-7507d2455483: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:28.344Z [INFO]  server-fc47edc0-a1e7-3d9f-3b8b-7507d2455483: entering candidate state: node="Node at fc47edc0-a1e7-3d9f-3b8b-7507d2455483 [Candidate]" term=2
2026-09-21T15:24:28.344Z [INFO]  server-fc47edc0-a1e7-3d9f-3b8b-7507d2455483: election won: term=2 tally=2
2026-09-21T15:24:28.344Z [INFO]  server-fc47edc0-a1e7-3d9f-3b8b-7507d2455483: entering leader state: leader="Node at fc47edc0-a1e7-3d9f-3b8b-7507d2455483 [Leader]"
2026-09-21T15:24:28.344Z [INFO]  server-fc47edc0-a1e7-3d9f-3b8b-7507d2455483: added peer, starting replication: peer=server-2f76599f-e04f-a360-85dd-72458dd4acd7
2026-09-21T15:24:28.344Z [INFO]  server-fc47edc0-a1e7-3d9f-3b8b-7507d2455483: added peer, starting replication: peer=server-b2acdeca-dc5d-ce98-48a7-ff58a67eb920
2026-09-21T15:24:28.344Z [INFO]  server-fc47edc0-a1e7-3d9f-3b8b-7507d2455483: pipelining replication: peer="{Voter server-b2acdeca-dc5d-ce98-48a7-ff58a67eb920 b2acdeca-dc5d-ce98-48a7-ff58a67eb920}"
2026-09-21T15:24:28.344Z [INFO]  server-fc47edc0-a1e7-3d9f-3b8b-7507d2455483: pipelining replication: peer="{Voter server-2f76599f-e04f-a360-85dd-72458dd4acd7 2f76599f-e04f-a360-85dd-72458dd4acd7}"
2026-09-21T15:24:29.350Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:28.344585068 +0000 UTC m=+20.285755241 (1 nodes), 660.80252ms from start of poll, 661.37292ms from cluster start. Timeout at 2026-09-21 15:24:29.349591988 +0000 UTC m=+21.290762161, 1.00500692s after stability
2026-09-21T15:24:29.351Z [INFO]  server-fc47edc0-a1e7-3d9f-3b8b-7507d2455483: starting snapshot up to: index=12
2026-09-21T15:24:29.352Z [INFO]  creating new snapshot: path=/tmp/raft2974314553/snapshots/2-12-1790004269352.tmp
2026-09-21T15:24:29.352Z [INFO]  server-fc47edc0-a1e7-3d9f-3b8b-7507d2455483: snapshot complete up to: index=12
2026-09-21T15:24:29.353Z [INFO]  server-b6e645f2-832e-0167-7e3c-7f7a9b0bcdce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6e645f2-832e-0167-7e3c-7f7a9b0bcdce Address:b6e645f2-832e-0167-7e3c-7f7a9b0bcdce} {Suffrage:Voter ID:server-43d8cfc4-ec60-53cf-17d6-ee14bec2689f Address:43d8cfc4-ec60-53cf-17d6-ee14bec2689f} {Suffrage:Voter ID:server-fa939c05-33fe-8ed6-60ec-2f286df934c5 Address:fa939c05-33fe-8ed6-60ec-2f286df934c5}]"
2026-09-21T15:24:29.354Z [INFO]  server-b6e645f2-832e-0167-7e3c-7f7a9b0bcdce: entering follower state: follower="Node at b6e645f2-832e-0167-7e3c-7f7a9b0bcdce [Follower]" leader-address= leader-id=
2026-09-21T15:24:29.354Z [INFO]  server-43d8cfc4-ec60-53cf-17d6-ee14bec2689f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6e645f2-832e-0167-7e3c-7f7a9b0bcdce Address:b6e645f2-832e-0167-7e3c-7f7a9b0bcdce} {Suffrage:Voter ID:server-43d8cfc4-ec60-53cf-17d6-ee14bec2689f Address:43d8cfc4-ec60-53cf-17d6-ee14bec2689f} {Suffrage:Voter ID:server-fa939c05-33fe-8ed6-60ec-2f286df934c5 Address:fa939c05-33fe-8ed6-60ec-2f286df934c5}]"
2026-09-21T15:24:29.354Z [INFO]  server-43d8cfc4-ec60-53cf-17d6-ee14bec2689f: entering follower state: follower="Node at 43d8cfc4-ec60-53cf-17d6-ee14bec2689f [Follower]" leader-address= leader-id=
2026-09-21T15:24:29.354Z [INFO]  server-fa939c05-33fe-8ed6-60ec-2f286df934c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6e645f2-832e-0167-7e3c-7f7a9b0bcdce Address:b6e645f2-832e-0167-7e3c-7f7a9b0bcdce} {Suffrage:Voter ID:server-43d8cfc4-ec60-53cf-17d6-ee14bec2689f Address:43d8cfc4-ec60-53cf-17d6-ee14bec2689f} {Suffrage:Voter ID:server-fa939c05-33fe-8ed6-60ec-2f286df934c5 Address:fa939c05-33fe-8ed6-60ec-2f286df934c5}]"
2026-09-21T15:24:29.354Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:29.354Z [INFO]  server-fa939c05-33fe-8ed6-60ec-2f286df934c5: entering follower state: follower="Node at fa939c05-33fe-8ed6-60ec-2f286df934c5 [Follower]" leader-address= leader-id=
2026-09-21T15:24:29.935Z [WARN]  server-fa939c05-33fe-8ed6-60ec-2f286df934c5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:29.935Z [INFO]  server-fa939c05-33fe-8ed6-60ec-2f286df934c5: entering candidate state: node="Node at fa939c05-33fe-8ed6-60ec-2f286df934c5 [Candidate]" term=2
2026-09-21T15:24:29.935Z [INFO]  server-fa939c05-33fe-8ed6-60ec-2f286df934c5: election won: term=2 tally=2
2026-09-21T15:24:29.935Z [INFO]  server-fa939c05-33fe-8ed6-60ec-2f286df934c5: entering leader state: leader="Node at fa939c05-33fe-8ed6-60ec-2f286df934c5 [Leader]"
2026-09-21T15:24:29.935Z [INFO]  server-fa939c05-33fe-8ed6-60ec-2f286df934c5: added peer, starting replication: peer=server-b6e645f2-832e-0167-7e3c-7f7a9b0bcdce
2026-09-21T15:24:29.935Z [INFO]  server-fa939c05-33fe-8ed6-60ec-2f286df934c5: added peer, starting replication: peer=server-43d8cfc4-ec60-53cf-17d6-ee14bec2689f
2026-09-21T15:24:29.935Z [INFO]  server-fa939c05-33fe-8ed6-60ec-2f286df934c5: pipelining replication: peer="{Voter server-b6e645f2-832e-0167-7e3c-7f7a9b0bcdce b6e645f2-832e-0167-7e3c-7f7a9b0bcdce}"
2026-09-21T15:24:29.935Z [INFO]  server-fa939c05-33fe-8ed6-60ec-2f286df934c5: pipelining replication: peer="{Voter server-43d8cfc4-ec60-53cf-17d6-ee14bec2689f 43d8cfc4-ec60-53cf-17d6-ee14bec2689f}"
2026-09-21T15:24:30.941Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:29.935876788 +0000 UTC m=+21.877046961 (1 nodes), 581.4438ms from start of poll, 582.029ms from cluster start. Timeout at 2026-09-21 15:24:30.940882508 +0000 UTC m=+22.882052681, 1.00500572s after stability
2026-09-21T15:24:30.941Z [INFO]  creating new snapshot: path=/tmp/raft559372525/snapshots/2-13-1790004270941.tmp
2026-09-21T15:24:30.942Z [INFO]  server-fa939c05-33fe-8ed6-60ec-2f286df934c5: copied to local snapshot: bytes=71
2026-09-21T15:24:30.942Z [INFO]  server-fa939c05-33fe-8ed6-60ec-2f286df934c5: snapshot restore progress: id=2-13-1790004270941 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:30.942Z [INFO]  server-fa939c05-33fe-8ed6-60ec-2f286df934c5: restored user snapshot: index=13
2026-09-21T15:24:30.942Z [ERROR] server-fa939c05-33fe-8ed6-60ec-2f286df934c5: failed to get log: index=3 error="log not found"
2026-09-21T15:24:30.942Z [INFO]  server-fa939c05-33fe-8ed6-60ec-2f286df934c5: aborting pipeline replication: peer="{Voter server-43d8cfc4-ec60-53cf-17d6-ee14bec2689f 43d8cfc4-ec60-53cf-17d6-ee14bec2689f}"
2026-09-21T15:24:30.942Z [ERROR] server-fa939c05-33fe-8ed6-60ec-2f286df934c5: failed to get log: index=3 error="log not found"
2026-09-21T15:24:30.942Z [INFO]  server-fa939c05-33fe-8ed6-60ec-2f286df934c5: aborting pipeline replication: peer="{Voter server-b6e645f2-832e-0167-7e3c-7f7a9b0bcdce b6e645f2-832e-0167-7e3c-7f7a9b0bcdce}"
2026-09-21T15:24:30.948Z [ERROR] server-fa939c05-33fe-8ed6-60ec-2f286df934c5: failed to get log: index=3 error="log not found"
2026-09-21T15:24:30.949Z [INFO]  creating new snapshot: path=/tmp/raft1702560507/snapshots/2-13-1790004270949.tmp
2026-09-21T15:24:30.949Z [INFO]  server-43d8cfc4-ec60-53cf-17d6-ee14bec2689f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:30.949Z [INFO]  server-43d8cfc4-ec60-53cf-17d6-ee14bec2689f: copied to local snapshot: bytes=71
2026-09-21T15:24:30.950Z [INFO]  server-43d8cfc4-ec60-53cf-17d6-ee14bec2689f: snapshot restore progress: id=2-13-1790004270949 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:30.950Z [INFO]  server-43d8cfc4-ec60-53cf-17d6-ee14bec2689f: Installed remote snapshot
2026-09-21T15:24:30.951Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:30.951Z [INFO]  server-fa939c05-33fe-8ed6-60ec-2f286df934c5: pipelining replication: peer="{Voter server-43d8cfc4-ec60-53cf-17d6-ee14bec2689f 43d8cfc4-ec60-53cf-17d6-ee14bec2689f}"
2026-09-21T15:24:30.952Z [ERROR] server-fa939c05-33fe-8ed6-60ec-2f286df934c5: failed to get log: index=3 error="log not found"
2026-09-21T15:24:30.952Z [INFO]  creating new snapshot: path=/tmp/raft1727936184/snapshots/2-13-1790004270952.tmp
2026-09-21T15:24:30.953Z [INFO]  server-b6e645f2-832e-0167-7e3c-7f7a9b0bcdce: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:30.953Z [INFO]  server-b6e645f2-832e-0167-7e3c-7f7a9b0bcdce: copied to local snapshot: bytes=71
2026-09-21T15:24:30.953Z [INFO]  server-b6e645f2-832e-0167-7e3c-7f7a9b0bcdce: snapshot restore progress: id=2-13-1790004270952 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:30.953Z [INFO]  server-b6e645f2-832e-0167-7e3c-7f7a9b0bcdce: Installed remote snapshot
2026-09-21T15:24:30.953Z [INFO]  server-fa939c05-33fe-8ed6-60ec-2f286df934c5: pipelining replication: peer="{Voter server-b6e645f2-832e-0167-7e3c-7f7a9b0bcdce b6e645f2-832e-0167-7e3c-7f7a9b0bcdce}"
2026-09-21T15:24:31.956Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:30.951260468 +0000 UTC m=+22.892430641 (2 nodes), 920ns from start of poll, 1.59741268s from cluster start. Timeout at 2026-09-21 15:24:31.956268228 +0000 UTC m=+23.897438401, 1.00500776s after stability
2026-09-21T15:24:31.968Z [INFO]  server-fc47edc0-a1e7-3d9f-3b8b-7507d2455483: aborting pipeline replication: peer="{Voter server-b2acdeca-dc5d-ce98-48a7-ff58a67eb920 b2acdeca-dc5d-ce98-48a7-ff58a67eb920}"
2026-09-21T15:24:31.968Z [INFO]  server-fc47edc0-a1e7-3d9f-3b8b-7507d2455483: aborting pipeline replication: peer="{Voter server-2f76599f-e04f-a360-85dd-72458dd4acd7 2f76599f-e04f-a360-85dd-72458dd4acd7}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true
2026-09-21T15:24:31.969Z [INFO]  server-f28faa55-2287-c45c-0f56-3abf39176723: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f28faa55-2287-c45c-0f56-3abf39176723 Address:f28faa55-2287-c45c-0f56-3abf39176723} {Suffrage:Voter ID:server-56afc3bc-fc68-7573-0873-9270668e80c0 Address:56afc3bc-fc68-7573-0873-9270668e80c0} {Suffrage:Voter ID:server-1a3a4df0-6ede-de1c-76d9-37c16055fd4c Address:1a3a4df0-6ede-de1c-76d9-37c16055fd4c}]"
2026-09-21T15:24:31.969Z [INFO]  server-56afc3bc-fc68-7573-0873-9270668e80c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f28faa55-2287-c45c-0f56-3abf39176723 Address:f28faa55-2287-c45c-0f56-3abf39176723} {Suffrage:Voter ID:server-56afc3bc-fc68-7573-0873-9270668e80c0 Address:56afc3bc-fc68-7573-0873-9270668e80c0} {Suffrage:Voter ID:server-1a3a4df0-6ede-de1c-76d9-37c16055fd4c Address:1a3a4df0-6ede-de1c-76d9-37c16055fd4c}]"
2026-09-21T15:24:31.969Z [INFO]  server-f28faa55-2287-c45c-0f56-3abf39176723: entering follower state: follower="Node at f28faa55-2287-c45c-0f56-3abf39176723 [Follower]" leader-address= leader-id=
2026-09-21T15:24:31.969Z [INFO]  server-56afc3bc-fc68-7573-0873-9270668e80c0: entering follower state: follower="Node at 56afc3bc-fc68-7573-0873-9270668e80c0 [Follower]" leader-address= leader-id=
2026-09-21T15:24:31.969Z [INFO]  server-1a3a4df0-6ede-de1c-76d9-37c16055fd4c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f28faa55-2287-c45c-0f56-3abf39176723 Address:f28faa55-2287-c45c-0f56-3abf39176723} {Suffrage:Voter ID:server-56afc3bc-fc68-7573-0873-9270668e80c0 Address:56afc3bc-fc68-7573-0873-9270668e80c0} {Suffrage:Voter ID:server-1a3a4df0-6ede-de1c-76d9-37c16055fd4c Address:1a3a4df0-6ede-de1c-76d9-37c16055fd4c}]"
2026-09-21T15:24:31.969Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:31.969Z [INFO]  server-1a3a4df0-6ede-de1c-76d9-37c16055fd4c: entering follower state: follower="Node at 1a3a4df0-6ede-de1c-76d9-37c16055fd4c [Follower]" leader-address= leader-id=
2026-09-21T15:24:32.577Z [WARN]  server-f28faa55-2287-c45c-0f56-3abf39176723: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:32.577Z [INFO]  server-f28faa55-2287-c45c-0f56-3abf39176723: entering candidate state: node="Node at f28faa55-2287-c45c-0f56-3abf39176723 [Candidate]" term=2
2026-09-21T15:24:32.578Z [INFO]  server-f28faa55-2287-c45c-0f56-3abf39176723: election won: term=2 tally=2
2026-09-21T15:24:32.578Z [INFO]  server-f28faa55-2287-c45c-0f56-3abf39176723: entering leader state: leader="Node at f28faa55-2287-c45c-0f56-3abf39176723 [Leader]"
2026-09-21T15:24:32.578Z [INFO]  server-f28faa55-2287-c45c-0f56-3abf39176723: added peer, starting replication: peer=server-56afc3bc-fc68-7573-0873-9270668e80c0
2026-09-21T15:24:32.578Z [INFO]  server-f28faa55-2287-c45c-0f56-3abf39176723: added peer, starting replication: peer=server-1a3a4df0-6ede-de1c-76d9-37c16055fd4c
2026-09-21T15:24:32.578Z [INFO]  server-f28faa55-2287-c45c-0f56-3abf39176723: pipelining replication: peer="{Voter server-56afc3bc-fc68-7573-0873-9270668e80c0 56afc3bc-fc68-7573-0873-9270668e80c0}"
2026-09-21T15:24:32.578Z [INFO]  server-f28faa55-2287-c45c-0f56-3abf39176723: pipelining replication: peer="{Voter server-1a3a4df0-6ede-de1c-76d9-37c16055fd4c 1a3a4df0-6ede-de1c-76d9-37c16055fd4c}"
2026-09-21T15:24:33.583Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:32.578144068 +0000 UTC m=+24.519314281 (1 nodes), 608.59624ms from start of poll, 609.0408ms from cluster start. Timeout at 2026-09-21 15:24:33.583148188 +0000 UTC m=+25.524318401, 1.00500412s after stability
2026-09-21T15:24:33.584Z [INFO]  server-f28faa55-2287-c45c-0f56-3abf39176723: starting snapshot up to: index=12
2026-09-21T15:24:33.584Z [INFO]  creating new snapshot: path=/tmp/raft385947394/snapshots/2-12-1790004273584.tmp
2026-09-21T15:24:33.584Z [INFO]  server-f28faa55-2287-c45c-0f56-3abf39176723: snapshot complete up to: index=12
2026-09-21T15:24:33.585Z [INFO]  server-920d6caf-74c7-53a1-425e-d4456f75aa88: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-920d6caf-74c7-53a1-425e-d4456f75aa88 Address:920d6caf-74c7-53a1-425e-d4456f75aa88} {Suffrage:Voter ID:server-a8cc7c0d-feb2-35f7-a3a2-1690b993884d Address:a8cc7c0d-feb2-35f7-a3a2-1690b993884d} {Suffrage:Voter ID:server-2a1dc962-4baf-e8ff-7264-944e021d0d08 Address:2a1dc962-4baf-e8ff-7264-944e021d0d08}]"
2026-09-21T15:24:33.586Z [INFO]  server-a8cc7c0d-feb2-35f7-a3a2-1690b993884d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-920d6caf-74c7-53a1-425e-d4456f75aa88 Address:920d6caf-74c7-53a1-425e-d4456f75aa88} {Suffrage:Voter ID:server-a8cc7c0d-feb2-35f7-a3a2-1690b993884d Address:a8cc7c0d-feb2-35f7-a3a2-1690b993884d} {Suffrage:Voter ID:server-2a1dc962-4baf-e8ff-7264-944e021d0d08 Address:2a1dc962-4baf-e8ff-7264-944e021d0d08}]"
2026-09-21T15:24:33.586Z [INFO]  server-920d6caf-74c7-53a1-425e-d4456f75aa88: entering follower state: follower="Node at 920d6caf-74c7-53a1-425e-d4456f75aa88 [Follower]" leader-address= leader-id=
2026-09-21T15:24:33.586Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-920d6caf-74c7-53a1-425e-d4456f75aa88 Address:920d6caf-74c7-53a1-425e-d4456f75aa88} {Suffrage:Voter ID:server-a8cc7c0d-feb2-35f7-a3a2-1690b993884d Address:a8cc7c0d-feb2-35f7-a3a2-1690b993884d} {Suffrage:Voter ID:server-2a1dc962-4baf-e8ff-7264-944e021d0d08 Address:2a1dc962-4baf-e8ff-7264-944e021d0d08}]"
2026-09-21T15:24:33.586Z [INFO]  server-a8cc7c0d-feb2-35f7-a3a2-1690b993884d: entering follower state: follower="Node at a8cc7c0d-feb2-35f7-a3a2-1690b993884d [Follower]" leader-address= leader-id=
2026-09-21T15:24:33.586Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:33.586Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: entering follower state: follower="Node at 2a1dc962-4baf-e8ff-7264-944e021d0d08 [Follower]" leader-address= leader-id=
2026-09-21T15:24:34.088Z [WARN]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:34.088Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: entering candidate state: node="Node at 2a1dc962-4baf-e8ff-7264-944e021d0d08 [Candidate]" term=2
2026-09-21T15:24:34.088Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: election won: term=2 tally=2
2026-09-21T15:24:34.088Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: entering leader state: leader="Node at 2a1dc962-4baf-e8ff-7264-944e021d0d08 [Leader]"
2026-09-21T15:24:34.088Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: added peer, starting replication: peer=server-920d6caf-74c7-53a1-425e-d4456f75aa88
2026-09-21T15:24:34.088Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: added peer, starting replication: peer=server-a8cc7c0d-feb2-35f7-a3a2-1690b993884d
2026-09-21T15:24:34.088Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: pipelining replication: peer="{Voter server-a8cc7c0d-feb2-35f7-a3a2-1690b993884d a8cc7c0d-feb2-35f7-a3a2-1690b993884d}"
2026-09-21T15:24:34.088Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: pipelining replication: peer="{Voter server-920d6caf-74c7-53a1-425e-d4456f75aa88 920d6caf-74c7-53a1-425e-d4456f75aa88}"
2026-09-21T15:24:35.093Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:34.088577148 +0000 UTC m=+26.029747361 (1 nodes), 502.35936ms from start of poll, 502.82724ms from cluster start. Timeout at 2026-09-21 15:24:35.093581068 +0000 UTC m=+27.034751241, 1.00500388s after stability
2026-09-21T15:24:35.094Z [INFO]  creating new snapshot: path=/tmp/raft2275418459/snapshots/2-13-1790004275094.tmp
2026-09-21T15:24:35.094Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: copied to local snapshot: bytes=71
2026-09-21T15:24:35.094Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: snapshot restore progress: id=2-13-1790004275094 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:35.095Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: removing all old logs from log store
2026-09-21T15:24:35.095Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: compacting logs: from=1 to=2
2026-09-21T15:24:35.095Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: restored user snapshot: index=13
2026-09-21T15:24:35.095Z [ERROR] server-2a1dc962-4baf-e8ff-7264-944e021d0d08: failed to get log: index=2 error="log not found"
2026-09-21T15:24:35.095Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: aborting pipeline replication: peer="{Voter server-a8cc7c0d-feb2-35f7-a3a2-1690b993884d a8cc7c0d-feb2-35f7-a3a2-1690b993884d}"
2026-09-21T15:24:35.095Z [ERROR] server-2a1dc962-4baf-e8ff-7264-944e021d0d08: failed to get log: index=2 error="log not found"
2026-09-21T15:24:35.095Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: aborting pipeline replication: peer="{Voter server-920d6caf-74c7-53a1-425e-d4456f75aa88 920d6caf-74c7-53a1-425e-d4456f75aa88}"
2026-09-21T15:24:35.100Z [ERROR] server-2a1dc962-4baf-e8ff-7264-944e021d0d08: failed to get log: index=2 error="log not found"
2026-09-21T15:24:35.101Z [INFO]  creating new snapshot: path=/tmp/raft2081202204/snapshots/2-13-1790004275101.tmp
2026-09-21T15:24:35.101Z [INFO]  server-a8cc7c0d-feb2-35f7-a3a2-1690b993884d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:35.102Z [INFO]  server-a8cc7c0d-feb2-35f7-a3a2-1690b993884d: copied to local snapshot: bytes=71
2026-09-21T15:24:35.102Z [ERROR] server-2a1dc962-4baf-e8ff-7264-944e021d0d08: failed to get log: index=2 error="log not found"
2026-09-21T15:24:35.102Z [INFO]  server-a8cc7c0d-feb2-35f7-a3a2-1690b993884d: snapshot restore progress: id=2-13-1790004275101 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:35.102Z [INFO]  server-a8cc7c0d-feb2-35f7-a3a2-1690b993884d: removing all old logs from log store
2026-09-21T15:24:35.102Z [INFO]  server-a8cc7c0d-feb2-35f7-a3a2-1690b993884d: compacting logs: from=1 to=2
2026-09-21T15:24:35.102Z [INFO]  server-a8cc7c0d-feb2-35f7-a3a2-1690b993884d: Installed remote snapshot
2026-09-21T15:24:35.102Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: pipelining replication: peer="{Voter server-a8cc7c0d-feb2-35f7-a3a2-1690b993884d a8cc7c0d-feb2-35f7-a3a2-1690b993884d}"
2026-09-21T15:24:35.102Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:35.102Z [INFO]  creating new snapshot: path=/tmp/raft1715587870/snapshots/2-13-1790004275102.tmp
2026-09-21T15:24:35.103Z [INFO]  server-920d6caf-74c7-53a1-425e-d4456f75aa88: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:35.103Z [INFO]  server-920d6caf-74c7-53a1-425e-d4456f75aa88: copied to local snapshot: bytes=71
2026-09-21T15:24:35.103Z [INFO]  server-920d6caf-74c7-53a1-425e-d4456f75aa88: snapshot restore progress: id=2-13-1790004275102 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:35.103Z [INFO]  server-920d6caf-74c7-53a1-425e-d4456f75aa88: removing all old logs from log store
2026-09-21T15:24:35.103Z [INFO]  server-920d6caf-74c7-53a1-425e-d4456f75aa88: compacting logs: from=1 to=2
2026-09-21T15:24:35.103Z [INFO]  server-920d6caf-74c7-53a1-425e-d4456f75aa88: Installed remote snapshot
2026-09-21T15:24:35.103Z [INFO]  server-2a1dc962-4baf-e8ff-7264-944e021d0d08: pipelining replication: peer="{Voter server-920d6caf-74c7-53a1-425e-d4456f75aa88 920d6caf-74c7-53a1-425e-d4456f75aa88}"
2026-09-21T15:24:36.108Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:35.102600588 +0000 UTC m=+27.043770761 (2 nodes), 1.04µs from start of poll, 1.51685064s from cluster start. Timeout at 2026-09-21 15:24:36.107609828 +0000 UTC m=+28.048780001, 1.00500924s after stability
2026-09-21T15:24:36.121Z [INFO]  server-f28faa55-2287-c45c-0f56-3abf39176723: aborting pipeline replication: peer="{Voter server-1a3a4df0-6ede-de1c-76d9-37c16055fd4c 1a3a4df0-6ede-de1c-76d9-37c16055fd4c}"
2026-09-21T15:24:36.121Z [INFO]  server-f28faa55-2287-c45c-0f56-3abf39176723: aborting pipeline replication: peer="{Voter server-56afc3bc-fc68-7573-0873-9270668e80c0 56afc3bc-fc68-7573-0873-9270668e80c0}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false
2026-09-21T15:24:36.122Z [INFO]  server-04d03ad6-421c-f672-46c7-07c0cef36210: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04d03ad6-421c-f672-46c7-07c0cef36210 Address:04d03ad6-421c-f672-46c7-07c0cef36210} {Suffrage:Voter ID:server-1447f8a8-8a4f-b295-6877-8705f056ee9a Address:1447f8a8-8a4f-b295-6877-8705f056ee9a} {Suffrage:Voter ID:server-837cfb60-51aa-1058-d766-b59dcdbae3f7 Address:837cfb60-51aa-1058-d766-b59dcdbae3f7}]"
2026-09-21T15:24:36.122Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04d03ad6-421c-f672-46c7-07c0cef36210 Address:04d03ad6-421c-f672-46c7-07c0cef36210} {Suffrage:Voter ID:server-1447f8a8-8a4f-b295-6877-8705f056ee9a Address:1447f8a8-8a4f-b295-6877-8705f056ee9a} {Suffrage:Voter ID:server-837cfb60-51aa-1058-d766-b59dcdbae3f7 Address:837cfb60-51aa-1058-d766-b59dcdbae3f7}]"
2026-09-21T15:24:36.122Z [INFO]  server-04d03ad6-421c-f672-46c7-07c0cef36210: entering follower state: follower="Node at 04d03ad6-421c-f672-46c7-07c0cef36210 [Follower]" leader-address= leader-id=
2026-09-21T15:24:36.122Z [INFO]  server-837cfb60-51aa-1058-d766-b59dcdbae3f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04d03ad6-421c-f672-46c7-07c0cef36210 Address:04d03ad6-421c-f672-46c7-07c0cef36210} {Suffrage:Voter ID:server-1447f8a8-8a4f-b295-6877-8705f056ee9a Address:1447f8a8-8a4f-b295-6877-8705f056ee9a} {Suffrage:Voter ID:server-837cfb60-51aa-1058-d766-b59dcdbae3f7 Address:837cfb60-51aa-1058-d766-b59dcdbae3f7}]"
2026-09-21T15:24:36.122Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: entering follower state: follower="Node at 1447f8a8-8a4f-b295-6877-8705f056ee9a [Follower]" leader-address= leader-id=
2026-09-21T15:24:36.122Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:36.122Z [INFO]  server-837cfb60-51aa-1058-d766-b59dcdbae3f7: entering follower state: follower="Node at 837cfb60-51aa-1058-d766-b59dcdbae3f7 [Follower]" leader-address= leader-id=
2026-09-21T15:24:36.643Z [WARN]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:36.643Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: entering candidate state: node="Node at 1447f8a8-8a4f-b295-6877-8705f056ee9a [Candidate]" term=2
2026-09-21T15:24:36.643Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: election won: term=2 tally=2
2026-09-21T15:24:36.643Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: entering leader state: leader="Node at 1447f8a8-8a4f-b295-6877-8705f056ee9a [Leader]"
2026-09-21T15:24:36.644Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: added peer, starting replication: peer=server-04d03ad6-421c-f672-46c7-07c0cef36210
2026-09-21T15:24:36.644Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: added peer, starting replication: peer=server-837cfb60-51aa-1058-d766-b59dcdbae3f7
2026-09-21T15:24:36.644Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: pipelining replication: peer="{Voter server-837cfb60-51aa-1058-d766-b59dcdbae3f7 837cfb60-51aa-1058-d766-b59dcdbae3f7}"
2026-09-21T15:24:36.644Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: pipelining replication: peer="{Voter server-04d03ad6-421c-f672-46c7-07c0cef36210 04d03ad6-421c-f672-46c7-07c0cef36210}"
2026-09-21T15:24:37.650Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:36.644536228 +0000 UTC m=+28.585706401 (1 nodes), 521.9034ms from start of poll, 522.333ms from cluster start. Timeout at 2026-09-21 15:24:37.649538668 +0000 UTC m=+29.590708841, 1.00500244s after stability
2026-09-21T15:24:37.651Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: starting snapshot up to: index=12
2026-09-21T15:24:37.651Z [INFO]  creating new snapshot: path=/tmp/raft990056322/snapshots/2-12-1790004277651.tmp
2026-09-21T15:24:37.652Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: snapshot complete up to: index=12
2026-09-21T15:24:37.654Z [INFO]  creating new snapshot: path=/tmp/raft990056322/snapshots/2-23-1790004277654.tmp
2026-09-21T15:24:37.655Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: copied to local snapshot: bytes=71
2026-09-21T15:24:37.655Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: snapshot restore progress: id=2-23-1790004277654 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:37.655Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: restored user snapshot: index=23
2026-09-21T15:24:37.655Z [ERROR] server-1447f8a8-8a4f-b295-6877-8705f056ee9a: failed to get log: index=23 error="log not found"
2026-09-21T15:24:37.655Z [ERROR] server-1447f8a8-8a4f-b295-6877-8705f056ee9a: failed to get log: index=23 error="log not found"
2026-09-21T15:24:37.655Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: aborting pipeline replication: peer="{Voter server-04d03ad6-421c-f672-46c7-07c0cef36210 04d03ad6-421c-f672-46c7-07c0cef36210}"
2026-09-21T15:24:37.655Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: aborting pipeline replication: peer="{Voter server-837cfb60-51aa-1058-d766-b59dcdbae3f7 837cfb60-51aa-1058-d766-b59dcdbae3f7}"
2026-09-21T15:24:37.661Z [ERROR] server-1447f8a8-8a4f-b295-6877-8705f056ee9a: failed to get log: index=23 error="log not found"
2026-09-21T15:24:37.662Z [INFO]  creating new snapshot: path=/tmp/raft1088211094/snapshots/2-23-1790004277662.tmp
2026-09-21T15:24:37.662Z [INFO]  server-837cfb60-51aa-1058-d766-b59dcdbae3f7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:37.662Z [INFO]  server-837cfb60-51aa-1058-d766-b59dcdbae3f7: copied to local snapshot: bytes=71
2026-09-21T15:24:37.663Z [INFO]  server-837cfb60-51aa-1058-d766-b59dcdbae3f7: snapshot restore progress: id=2-23-1790004277662 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:37.663Z [INFO]  server-837cfb60-51aa-1058-d766-b59dcdbae3f7: Installed remote snapshot
2026-09-21T15:24:37.663Z [ERROR] server-1447f8a8-8a4f-b295-6877-8705f056ee9a: failed to get log: index=23 error="log not found"
2026-09-21T15:24:37.663Z [WARN]  server-837cfb60-51aa-1058-d766-b59dcdbae3f7: skipping application of old log: index=22
2026-09-21T15:24:37.663Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: pipelining replication: peer="{Voter server-837cfb60-51aa-1058-d766-b59dcdbae3f7 837cfb60-51aa-1058-d766-b59dcdbae3f7}"
2026-09-21T15:24:37.663Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:37.663Z [INFO]  creating new snapshot: path=/tmp/raft768502008/snapshots/2-23-1790004277663.tmp
2026-09-21T15:24:37.663Z [INFO]  server-04d03ad6-421c-f672-46c7-07c0cef36210: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:37.664Z [INFO]  server-04d03ad6-421c-f672-46c7-07c0cef36210: copied to local snapshot: bytes=71
2026-09-21T15:24:37.664Z [INFO]  server-04d03ad6-421c-f672-46c7-07c0cef36210: snapshot restore progress: id=2-23-1790004277663 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:37.664Z [INFO]  server-04d03ad6-421c-f672-46c7-07c0cef36210: Installed remote snapshot
2026-09-21T15:24:37.664Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: pipelining replication: peer="{Voter server-04d03ad6-421c-f672-46c7-07c0cef36210 04d03ad6-421c-f672-46c7-07c0cef36210}"
2026-09-21T15:24:38.669Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:37.663526548 +0000 UTC m=+29.604696721 (2 nodes), 1.2µs from start of poll, 1.54132332s from cluster start. Timeout at 2026-09-21 15:24:38.668539628 +0000 UTC m=+30.609709801, 1.00501308s after stability
2026-09-21T15:24:38.681Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: aborting pipeline replication: peer="{Voter server-837cfb60-51aa-1058-d766-b59dcdbae3f7 837cfb60-51aa-1058-d766-b59dcdbae3f7}"
2026-09-21T15:24:38.681Z [INFO]  server-1447f8a8-8a4f-b295-6877-8705f056ee9a: aborting pipeline replication: peer="{Voter server-04d03ad6-421c-f672-46c7-07c0cef36210 04d03ad6-421c-f672-46c7-07c0cef36210}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false
2026-09-21T15:24:38.682Z [INFO]  server-8ccef60e-4550-8e91-a0a6-36424dd688fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ccef60e-4550-8e91-a0a6-36424dd688fd Address:8ccef60e-4550-8e91-a0a6-36424dd688fd} {Suffrage:Voter ID:server-6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc Address:6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc} {Suffrage:Voter ID:server-c2bd8cfe-b52b-4ab6-d104-125e56850037 Address:c2bd8cfe-b52b-4ab6-d104-125e56850037}]"
2026-09-21T15:24:38.682Z [INFO]  server-8ccef60e-4550-8e91-a0a6-36424dd688fd: entering follower state: follower="Node at 8ccef60e-4550-8e91-a0a6-36424dd688fd [Follower]" leader-address= leader-id=
2026-09-21T15:24:38.682Z [INFO]  server-6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ccef60e-4550-8e91-a0a6-36424dd688fd Address:8ccef60e-4550-8e91-a0a6-36424dd688fd} {Suffrage:Voter ID:server-6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc Address:6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc} {Suffrage:Voter ID:server-c2bd8cfe-b52b-4ab6-d104-125e56850037 Address:c2bd8cfe-b52b-4ab6-d104-125e56850037}]"
2026-09-21T15:24:38.682Z [INFO]  server-6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc: entering follower state: follower="Node at 6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc [Follower]" leader-address= leader-id=
2026-09-21T15:24:38.682Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ccef60e-4550-8e91-a0a6-36424dd688fd Address:8ccef60e-4550-8e91-a0a6-36424dd688fd} {Suffrage:Voter ID:server-6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc Address:6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc} {Suffrage:Voter ID:server-c2bd8cfe-b52b-4ab6-d104-125e56850037 Address:c2bd8cfe-b52b-4ab6-d104-125e56850037}]"
2026-09-21T15:24:38.682Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:38.682Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: entering follower state: follower="Node at c2bd8cfe-b52b-4ab6-d104-125e56850037 [Follower]" leader-address= leader-id=
2026-09-21T15:24:39.505Z [WARN]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:39.505Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: entering candidate state: node="Node at c2bd8cfe-b52b-4ab6-d104-125e56850037 [Candidate]" term=2
2026-09-21T15:24:39.505Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: election won: term=2 tally=2
2026-09-21T15:24:39.505Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: entering leader state: leader="Node at c2bd8cfe-b52b-4ab6-d104-125e56850037 [Leader]"
2026-09-21T15:24:39.505Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: added peer, starting replication: peer=server-8ccef60e-4550-8e91-a0a6-36424dd688fd
2026-09-21T15:24:39.505Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: added peer, starting replication: peer=server-6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc
2026-09-21T15:24:39.505Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: pipelining replication: peer="{Voter server-8ccef60e-4550-8e91-a0a6-36424dd688fd 8ccef60e-4550-8e91-a0a6-36424dd688fd}"
2026-09-21T15:24:39.505Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: pipelining replication: peer="{Voter server-6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc 6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc}"
2026-09-21T15:24:40.511Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:39.505740348 +0000 UTC m=+31.446910521 (1 nodes), 823.07608ms from start of poll, 823.48708ms from cluster start. Timeout at 2026-09-21 15:24:40.510748428 +0000 UTC m=+32.451918601, 1.00500808s after stability
2026-09-21T15:24:40.511Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: starting snapshot up to: index=12
2026-09-21T15:24:40.511Z [INFO]  creating new snapshot: path=/tmp/raft1926343182/snapshots/2-12-1790004280511.tmp
2026-09-21T15:24:40.512Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: snapshot complete up to: index=12
2026-09-21T15:24:40.513Z [INFO]  creating new snapshot: path=/tmp/raft1926343182/snapshots/2-23-1790004280513.tmp
2026-09-21T15:24:40.514Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: copied to local snapshot: bytes=71
2026-09-21T15:24:40.514Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: snapshot restore progress: id=2-23-1790004280513 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:40.514Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: removing all old logs from log store
2026-09-21T15:24:40.514Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: compacting logs: from=1 to=22
2026-09-21T15:24:40.514Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: restored user snapshot: index=23
2026-09-21T15:24:40.514Z [ERROR] server-c2bd8cfe-b52b-4ab6-d104-125e56850037: failed to get log: index=22 error="log not found"
2026-09-21T15:24:40.514Z [ERROR] server-c2bd8cfe-b52b-4ab6-d104-125e56850037: failed to get log: index=22 error="log not found"
2026-09-21T15:24:40.514Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: aborting pipeline replication: peer="{Voter server-8ccef60e-4550-8e91-a0a6-36424dd688fd 8ccef60e-4550-8e91-a0a6-36424dd688fd}"
2026-09-21T15:24:40.514Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: aborting pipeline replication: peer="{Voter server-6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc 6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc}"
2026-09-21T15:24:40.521Z [ERROR] server-c2bd8cfe-b52b-4ab6-d104-125e56850037: failed to get log: index=22 error="log not found"
2026-09-21T15:24:40.522Z [ERROR] server-c2bd8cfe-b52b-4ab6-d104-125e56850037: failed to get log: index=22 error="log not found"
2026-09-21T15:24:40.522Z [INFO]  creating new snapshot: path=/tmp/raft2232183995/snapshots/2-23-1790004280522.tmp
2026-09-21T15:24:40.524Z [INFO]  creating new snapshot: path=/tmp/raft161634215/snapshots/2-23-1790004280524.tmp
2026-09-21T15:24:40.524Z [INFO]  server-6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:40.524Z [INFO]  server-8ccef60e-4550-8e91-a0a6-36424dd688fd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:40.524Z [INFO]  server-6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc: copied to local snapshot: bytes=71
2026-09-21T15:24:40.524Z [INFO]  server-8ccef60e-4550-8e91-a0a6-36424dd688fd: copied to local snapshot: bytes=71
2026-09-21T15:24:40.524Z [INFO]  server-6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc: snapshot restore progress: id=2-23-1790004280522 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:40.524Z [INFO]  server-6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc: removing all old logs from log store
2026-09-21T15:24:40.524Z [INFO]  server-6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc: compacting logs: from=1 to=22
2026-09-21T15:24:40.524Z [INFO]  server-6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc: Installed remote snapshot
2026-09-21T15:24:40.524Z [INFO]  server-8ccef60e-4550-8e91-a0a6-36424dd688fd: snapshot restore progress: id=2-23-1790004280524 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:40.524Z [INFO]  server-8ccef60e-4550-8e91-a0a6-36424dd688fd: removing all old logs from log store
2026-09-21T15:24:40.524Z [INFO]  server-8ccef60e-4550-8e91-a0a6-36424dd688fd: compacting logs: from=1 to=22
2026-09-21T15:24:40.524Z [INFO]  server-8ccef60e-4550-8e91-a0a6-36424dd688fd: Installed remote snapshot
2026-09-21T15:24:40.524Z [WARN]  server-6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc: skipping application of old log: index=22
2026-09-21T15:24:40.524Z [WARN]  server-8ccef60e-4550-8e91-a0a6-36424dd688fd: skipping application of old log: index=23
2026-09-21T15:24:40.525Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:40.524Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: pipelining replication: peer="{Voter server-6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc 6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc}"
2026-09-21T15:24:40.525Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: pipelining replication: peer="{Voter server-8ccef60e-4550-8e91-a0a6-36424dd688fd 8ccef60e-4550-8e91-a0a6-36424dd688fd}"
2026-09-21T15:24:41.530Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:40.525027028 +0000 UTC m=+32.466197201 (2 nodes), 1µs from start of poll, 1.84277376s from cluster start. Timeout at 2026-09-21 15:24:41.530039988 +0000 UTC m=+33.471210161, 1.00501296s after stability
2026-09-21T15:24:41.543Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: aborting pipeline replication: peer="{Voter server-6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc 6ef182c5-00ec-1214-5a6a-24cbdf8c3dbc}"
2026-09-21T15:24:41.543Z [INFO]  server-c2bd8cfe-b52b-4ab6-d104-125e56850037: aborting pipeline replication: peer="{Voter server-8ccef60e-4550-8e91-a0a6-36424dd688fd 8ccef60e-4550-8e91-a0a6-36424dd688fd}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true
2026-09-21T15:24:41.544Z [INFO]  server-03a7a541-59f1-beb5-a17c-c68524bf8a52: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03a7a541-59f1-beb5-a17c-c68524bf8a52 Address:03a7a541-59f1-beb5-a17c-c68524bf8a52} {Suffrage:Voter ID:server-e8d3ea2d-2ba4-6046-6eea-176638d593fe Address:e8d3ea2d-2ba4-6046-6eea-176638d593fe} {Suffrage:Voter ID:server-df58c8ec-4071-7dc6-e317-94d55de0bfc0 Address:df58c8ec-4071-7dc6-e317-94d55de0bfc0}]"
2026-09-21T15:24:41.544Z [INFO]  server-e8d3ea2d-2ba4-6046-6eea-176638d593fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03a7a541-59f1-beb5-a17c-c68524bf8a52 Address:03a7a541-59f1-beb5-a17c-c68524bf8a52} {Suffrage:Voter ID:server-e8d3ea2d-2ba4-6046-6eea-176638d593fe Address:e8d3ea2d-2ba4-6046-6eea-176638d593fe} {Suffrage:Voter ID:server-df58c8ec-4071-7dc6-e317-94d55de0bfc0 Address:df58c8ec-4071-7dc6-e317-94d55de0bfc0}]"
2026-09-21T15:24:41.544Z [INFO]  server-03a7a541-59f1-beb5-a17c-c68524bf8a52: entering follower state: follower="Node at 03a7a541-59f1-beb5-a17c-c68524bf8a52 [Follower]" leader-address= leader-id=
2026-09-21T15:24:41.544Z [INFO]  server-df58c8ec-4071-7dc6-e317-94d55de0bfc0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-03a7a541-59f1-beb5-a17c-c68524bf8a52 Address:03a7a541-59f1-beb5-a17c-c68524bf8a52} {Suffrage:Voter ID:server-e8d3ea2d-2ba4-6046-6eea-176638d593fe Address:e8d3ea2d-2ba4-6046-6eea-176638d593fe} {Suffrage:Voter ID:server-df58c8ec-4071-7dc6-e317-94d55de0bfc0 Address:df58c8ec-4071-7dc6-e317-94d55de0bfc0}]"
2026-09-21T15:24:41.544Z [INFO]  server-e8d3ea2d-2ba4-6046-6eea-176638d593fe: entering follower state: follower="Node at e8d3ea2d-2ba4-6046-6eea-176638d593fe [Follower]" leader-address= leader-id=
2026-09-21T15:24:41.544Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:41.544Z [INFO]  server-df58c8ec-4071-7dc6-e317-94d55de0bfc0: entering follower state: follower="Node at df58c8ec-4071-7dc6-e317-94d55de0bfc0 [Follower]" leader-address= leader-id=
2026-09-21T15:24:42.087Z [WARN]  server-03a7a541-59f1-beb5-a17c-c68524bf8a52: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:42.087Z [INFO]  server-03a7a541-59f1-beb5-a17c-c68524bf8a52: entering candidate state: node="Node at 03a7a541-59f1-beb5-a17c-c68524bf8a52 [Candidate]" term=2
2026-09-21T15:24:42.087Z [INFO]  server-03a7a541-59f1-beb5-a17c-c68524bf8a52: election won: term=2 tally=2
2026-09-21T15:24:42.087Z [INFO]  server-03a7a541-59f1-beb5-a17c-c68524bf8a52: entering leader state: leader="Node at 03a7a541-59f1-beb5-a17c-c68524bf8a52 [Leader]"
2026-09-21T15:24:42.087Z [INFO]  server-03a7a541-59f1-beb5-a17c-c68524bf8a52: added peer, starting replication: peer=server-e8d3ea2d-2ba4-6046-6eea-176638d593fe
2026-09-21T15:24:42.087Z [INFO]  server-03a7a541-59f1-beb5-a17c-c68524bf8a52: added peer, starting replication: peer=server-df58c8ec-4071-7dc6-e317-94d55de0bfc0
2026-09-21T15:24:42.087Z [INFO]  server-03a7a541-59f1-beb5-a17c-c68524bf8a52: pipelining replication: peer="{Voter server-df58c8ec-4071-7dc6-e317-94d55de0bfc0 df58c8ec-4071-7dc6-e317-94d55de0bfc0}"
2026-09-21T15:24:42.087Z [INFO]  server-03a7a541-59f1-beb5-a17c-c68524bf8a52: pipelining replication: peer="{Voter server-e8d3ea2d-2ba4-6046-6eea-176638d593fe e8d3ea2d-2ba4-6046-6eea-176638d593fe}"
2026-09-21T15:24:43.093Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:42.087481668 +0000 UTC m=+34.028651841 (1 nodes), 542.72204ms from start of poll, 543.12328ms from cluster start. Timeout at 2026-09-21 15:24:43.092493108 +0000 UTC m=+35.033663281, 1.00501144s after stability
2026-09-21T15:24:43.094Z [INFO]  server-03a7a541-59f1-beb5-a17c-c68524bf8a52: starting snapshot up to: index=12
2026-09-21T15:24:43.094Z [INFO]  creating new snapshot: path=/tmp/raft4168296706/snapshots/2-12-1790004283094.tmp
2026-09-21T15:24:43.094Z [INFO]  server-03a7a541-59f1-beb5-a17c-c68524bf8a52: snapshot complete up to: index=12
2026-09-21T15:24:43.095Z [INFO]  server-040f45b2-142c-2ab1-3d4f-18f3379404ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-040f45b2-142c-2ab1-3d4f-18f3379404ee Address:040f45b2-142c-2ab1-3d4f-18f3379404ee} {Suffrage:Voter ID:server-65d4c677-262a-3f2e-81a3-144402d822e7 Address:65d4c677-262a-3f2e-81a3-144402d822e7} {Suffrage:Voter ID:server-5cebfcf1-9110-4a47-aa67-f2a712c12f81 Address:5cebfcf1-9110-4a47-aa67-f2a712c12f81}]"
2026-09-21T15:24:43.096Z [INFO]  server-65d4c677-262a-3f2e-81a3-144402d822e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-040f45b2-142c-2ab1-3d4f-18f3379404ee Address:040f45b2-142c-2ab1-3d4f-18f3379404ee} {Suffrage:Voter ID:server-65d4c677-262a-3f2e-81a3-144402d822e7 Address:65d4c677-262a-3f2e-81a3-144402d822e7} {Suffrage:Voter ID:server-5cebfcf1-9110-4a47-aa67-f2a712c12f81 Address:5cebfcf1-9110-4a47-aa67-f2a712c12f81}]"
2026-09-21T15:24:43.095Z [INFO]  server-040f45b2-142c-2ab1-3d4f-18f3379404ee: entering follower state: follower="Node at 040f45b2-142c-2ab1-3d4f-18f3379404ee [Follower]" leader-address= leader-id=
2026-09-21T15:24:43.096Z [INFO]  server-5cebfcf1-9110-4a47-aa67-f2a712c12f81: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-040f45b2-142c-2ab1-3d4f-18f3379404ee Address:040f45b2-142c-2ab1-3d4f-18f3379404ee} {Suffrage:Voter ID:server-65d4c677-262a-3f2e-81a3-144402d822e7 Address:65d4c677-262a-3f2e-81a3-144402d822e7} {Suffrage:Voter ID:server-5cebfcf1-9110-4a47-aa67-f2a712c12f81 Address:5cebfcf1-9110-4a47-aa67-f2a712c12f81}]"
2026-09-21T15:24:43.096Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:43.096Z [INFO]  server-65d4c677-262a-3f2e-81a3-144402d822e7: entering follower state: follower="Node at 65d4c677-262a-3f2e-81a3-144402d822e7 [Follower]" leader-address= leader-id=
2026-09-21T15:24:43.096Z [INFO]  server-5cebfcf1-9110-4a47-aa67-f2a712c12f81: entering follower state: follower="Node at 5cebfcf1-9110-4a47-aa67-f2a712c12f81 [Follower]" leader-address= leader-id=
2026-09-21T15:24:43.790Z [WARN]  server-040f45b2-142c-2ab1-3d4f-18f3379404ee: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:43.790Z [INFO]  server-040f45b2-142c-2ab1-3d4f-18f3379404ee: entering candidate state: node="Node at 040f45b2-142c-2ab1-3d4f-18f3379404ee [Candidate]" term=2
2026-09-21T15:24:43.790Z [INFO]  server-040f45b2-142c-2ab1-3d4f-18f3379404ee: election won: term=2 tally=2
2026-09-21T15:24:43.790Z [INFO]  server-040f45b2-142c-2ab1-3d4f-18f3379404ee: entering leader state: leader="Node at 040f45b2-142c-2ab1-3d4f-18f3379404ee [Leader]"
2026-09-21T15:24:43.790Z [INFO]  server-040f45b2-142c-2ab1-3d4f-18f3379404ee: added peer, starting replication: peer=server-65d4c677-262a-3f2e-81a3-144402d822e7
2026-09-21T15:24:43.790Z [INFO]  server-040f45b2-142c-2ab1-3d4f-18f3379404ee: added peer, starting replication: peer=server-5cebfcf1-9110-4a47-aa67-f2a712c12f81
2026-09-21T15:24:43.790Z [INFO]  server-040f45b2-142c-2ab1-3d4f-18f3379404ee: pipelining replication: peer="{Voter server-5cebfcf1-9110-4a47-aa67-f2a712c12f81 5cebfcf1-9110-4a47-aa67-f2a712c12f81}"
2026-09-21T15:24:43.790Z [INFO]  server-040f45b2-142c-2ab1-3d4f-18f3379404ee: pipelining replication: peer="{Voter server-65d4c677-262a-3f2e-81a3-144402d822e7 65d4c677-262a-3f2e-81a3-144402d822e7}"
2026-09-21T15:24:44.795Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:43.790401308 +0000 UTC m=+35.731571521 (1 nodes), 694.22816ms from start of poll, 694.6328ms from cluster start. Timeout at 2026-09-21 15:24:44.795412108 +0000 UTC m=+36.736582321, 1.0050108s after stability
2026-09-21T15:24:44.796Z [INFO]  creating new snapshot: path=/tmp/raft3302414703/snapshots/2-14-1790004284796.tmp
2026-09-21T15:24:44.796Z [INFO]  server-040f45b2-142c-2ab1-3d4f-18f3379404ee: copied to local snapshot: bytes=71
2026-09-21T15:24:44.796Z [INFO]  server-040f45b2-142c-2ab1-3d4f-18f3379404ee: snapshot restore progress: id=2-14-1790004284796 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:44.797Z [INFO]  server-040f45b2-142c-2ab1-3d4f-18f3379404ee: restored user snapshot: index=14
2026-09-21T15:24:44.797Z [ERROR] server-040f45b2-142c-2ab1-3d4f-18f3379404ee: failed to get log: index=3 error="log not found"
2026-09-21T15:24:44.797Z [ERROR] server-040f45b2-142c-2ab1-3d4f-18f3379404ee: failed to get log: index=3 error="log not found"
2026-09-21T15:24:44.797Z [INFO]  server-040f45b2-142c-2ab1-3d4f-18f3379404ee: aborting pipeline replication: peer="{Voter server-65d4c677-262a-3f2e-81a3-144402d822e7 65d4c677-262a-3f2e-81a3-144402d822e7}"
2026-09-21T15:24:44.797Z [INFO]  server-040f45b2-142c-2ab1-3d4f-18f3379404ee: aborting pipeline replication: peer="{Voter server-5cebfcf1-9110-4a47-aa67-f2a712c12f81 5cebfcf1-9110-4a47-aa67-f2a712c12f81}"
2026-09-21T15:24:44.806Z [ERROR] server-040f45b2-142c-2ab1-3d4f-18f3379404ee: failed to get log: index=3 error="log not found"
2026-09-21T15:24:44.806Z [ERROR] server-040f45b2-142c-2ab1-3d4f-18f3379404ee: failed to get log: index=3 error="log not found"
2026-09-21T15:24:44.806Z [INFO]  creating new snapshot: path=/tmp/raft2972842512/snapshots/2-14-1790004284806.tmp
2026-09-21T15:24:44.806Z [INFO]  creating new snapshot: path=/tmp/raft3935562785/snapshots/2-14-1790004284806.tmp
2026-09-21T15:24:44.806Z [INFO]  server-5cebfcf1-9110-4a47-aa67-f2a712c12f81: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:44.806Z [INFO]  server-65d4c677-262a-3f2e-81a3-144402d822e7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:44.807Z [INFO]  server-5cebfcf1-9110-4a47-aa67-f2a712c12f81: copied to local snapshot: bytes=71
2026-09-21T15:24:44.807Z [INFO]  server-65d4c677-262a-3f2e-81a3-144402d822e7: copied to local snapshot: bytes=71
2026-09-21T15:24:44.807Z [INFO]  server-5cebfcf1-9110-4a47-aa67-f2a712c12f81: snapshot restore progress: id=2-14-1790004284806 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:44.807Z [INFO]  server-5cebfcf1-9110-4a47-aa67-f2a712c12f81: Installed remote snapshot
2026-09-21T15:24:44.807Z [INFO]  server-65d4c677-262a-3f2e-81a3-144402d822e7: snapshot restore progress: id=2-14-1790004284806 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:44.807Z [INFO]  server-65d4c677-262a-3f2e-81a3-144402d822e7: Installed remote snapshot
2026-09-21T15:24:44.807Z [INFO]  server-040f45b2-142c-2ab1-3d4f-18f3379404ee: pipelining replication: peer="{Voter server-5cebfcf1-9110-4a47-aa67-f2a712c12f81 5cebfcf1-9110-4a47-aa67-f2a712c12f81}"
2026-09-21T15:24:44.807Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:44.807Z [INFO]  server-040f45b2-142c-2ab1-3d4f-18f3379404ee: pipelining replication: peer="{Voter server-65d4c677-262a-3f2e-81a3-144402d822e7 65d4c677-262a-3f2e-81a3-144402d822e7}"
2026-09-21T15:24:45.812Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:44.807746268 +0000 UTC m=+36.748916401 (2 nodes), 1.04µs from start of poll, 1.71197768s from cluster start. Timeout at 2026-09-21 15:24:45.812759668 +0000 UTC m=+37.753929881, 1.00501348s after stability
2026-09-21T15:24:45.825Z [INFO]  server-03a7a541-59f1-beb5-a17c-c68524bf8a52: aborting pipeline replication: peer="{Voter server-df58c8ec-4071-7dc6-e317-94d55de0bfc0 df58c8ec-4071-7dc6-e317-94d55de0bfc0}"
2026-09-21T15:24:45.825Z [INFO]  server-03a7a541-59f1-beb5-a17c-c68524bf8a52: aborting pipeline replication: peer="{Voter server-e8d3ea2d-2ba4-6046-6eea-176638d593fe e8d3ea2d-2ba4-6046-6eea-176638d593fe}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true
2026-09-21T15:24:45.826Z [INFO]  server-0f7b82d4-abaa-b7d7-284b-f28df08c5279: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f7b82d4-abaa-b7d7-284b-f28df08c5279 Address:0f7b82d4-abaa-b7d7-284b-f28df08c5279} {Suffrage:Voter ID:server-0d576b24-e395-ba1b-cc46-b6706336ff97 Address:0d576b24-e395-ba1b-cc46-b6706336ff97} {Suffrage:Voter ID:server-d832ed58-e04f-b0e4-1de8-325c032af7d9 Address:d832ed58-e04f-b0e4-1de8-325c032af7d9}]"
2026-09-21T15:24:45.826Z [INFO]  server-0d576b24-e395-ba1b-cc46-b6706336ff97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f7b82d4-abaa-b7d7-284b-f28df08c5279 Address:0f7b82d4-abaa-b7d7-284b-f28df08c5279} {Suffrage:Voter ID:server-0d576b24-e395-ba1b-cc46-b6706336ff97 Address:0d576b24-e395-ba1b-cc46-b6706336ff97} {Suffrage:Voter ID:server-d832ed58-e04f-b0e4-1de8-325c032af7d9 Address:d832ed58-e04f-b0e4-1de8-325c032af7d9}]"
2026-09-21T15:24:45.826Z [INFO]  server-0f7b82d4-abaa-b7d7-284b-f28df08c5279: entering follower state: follower="Node at 0f7b82d4-abaa-b7d7-284b-f28df08c5279 [Follower]" leader-address= leader-id=
2026-09-21T15:24:45.826Z [INFO]  server-0d576b24-e395-ba1b-cc46-b6706336ff97: entering follower state: follower="Node at 0d576b24-e395-ba1b-cc46-b6706336ff97 [Follower]" leader-address= leader-id=
2026-09-21T15:24:45.826Z [INFO]  server-d832ed58-e04f-b0e4-1de8-325c032af7d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f7b82d4-abaa-b7d7-284b-f28df08c5279 Address:0f7b82d4-abaa-b7d7-284b-f28df08c5279} {Suffrage:Voter ID:server-0d576b24-e395-ba1b-cc46-b6706336ff97 Address:0d576b24-e395-ba1b-cc46-b6706336ff97} {Suffrage:Voter ID:server-d832ed58-e04f-b0e4-1de8-325c032af7d9 Address:d832ed58-e04f-b0e4-1de8-325c032af7d9}]"
2026-09-21T15:24:45.826Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:45.826Z [INFO]  server-d832ed58-e04f-b0e4-1de8-325c032af7d9: entering follower state: follower="Node at d832ed58-e04f-b0e4-1de8-325c032af7d9 [Follower]" leader-address= leader-id=
2026-09-21T15:24:46.395Z [WARN]  server-d832ed58-e04f-b0e4-1de8-325c032af7d9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:46.395Z [INFO]  server-d832ed58-e04f-b0e4-1de8-325c032af7d9: entering candidate state: node="Node at d832ed58-e04f-b0e4-1de8-325c032af7d9 [Candidate]" term=2
2026-09-21T15:24:46.395Z [INFO]  server-d832ed58-e04f-b0e4-1de8-325c032af7d9: election won: term=2 tally=2
2026-09-21T15:24:46.395Z [INFO]  server-d832ed58-e04f-b0e4-1de8-325c032af7d9: entering leader state: leader="Node at d832ed58-e04f-b0e4-1de8-325c032af7d9 [Leader]"
2026-09-21T15:24:46.395Z [INFO]  server-d832ed58-e04f-b0e4-1de8-325c032af7d9: added peer, starting replication: peer=server-0f7b82d4-abaa-b7d7-284b-f28df08c5279
2026-09-21T15:24:46.395Z [INFO]  server-d832ed58-e04f-b0e4-1de8-325c032af7d9: added peer, starting replication: peer=server-0d576b24-e395-ba1b-cc46-b6706336ff97
2026-09-21T15:24:46.395Z [INFO]  server-d832ed58-e04f-b0e4-1de8-325c032af7d9: pipelining replication: peer="{Voter server-0d576b24-e395-ba1b-cc46-b6706336ff97 0d576b24-e395-ba1b-cc46-b6706336ff97}"
2026-09-21T15:24:46.395Z [INFO]  server-d832ed58-e04f-b0e4-1de8-325c032af7d9: pipelining replication: peer="{Voter server-0f7b82d4-abaa-b7d7-284b-f28df08c5279 0f7b82d4-abaa-b7d7-284b-f28df08c5279}"
2026-09-21T15:24:47.400Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:46.395246708 +0000 UTC m=+38.336416881 (1 nodes), 568.45448ms from start of poll, 568.87632ms from cluster start. Timeout at 2026-09-21 15:24:47.400249028 +0000 UTC m=+39.341419201, 1.00500232s after stability
2026-09-21T15:24:47.402Z [INFO]  server-d832ed58-e04f-b0e4-1de8-325c032af7d9: starting snapshot up to: index=12
2026-09-21T15:24:47.402Z [INFO]  creating new snapshot: path=/tmp/raft3792788318/snapshots/2-12-1790004287402.tmp
2026-09-21T15:24:47.403Z [INFO]  server-d832ed58-e04f-b0e4-1de8-325c032af7d9: snapshot complete up to: index=12
2026-09-21T15:24:47.404Z [INFO]  server-445070a7-0246-8887-1ccb-f97557f8c6bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-445070a7-0246-8887-1ccb-f97557f8c6bc Address:445070a7-0246-8887-1ccb-f97557f8c6bc} {Suffrage:Voter ID:server-016378df-6123-b6e1-96e8-c50e7b0c6d08 Address:016378df-6123-b6e1-96e8-c50e7b0c6d08} {Suffrage:Voter ID:server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78 Address:8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78}]"
2026-09-21T15:24:47.404Z [INFO]  server-016378df-6123-b6e1-96e8-c50e7b0c6d08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-445070a7-0246-8887-1ccb-f97557f8c6bc Address:445070a7-0246-8887-1ccb-f97557f8c6bc} {Suffrage:Voter ID:server-016378df-6123-b6e1-96e8-c50e7b0c6d08 Address:016378df-6123-b6e1-96e8-c50e7b0c6d08} {Suffrage:Voter ID:server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78 Address:8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78}]"
2026-09-21T15:24:47.404Z [INFO]  server-445070a7-0246-8887-1ccb-f97557f8c6bc: entering follower state: follower="Node at 445070a7-0246-8887-1ccb-f97557f8c6bc [Follower]" leader-address= leader-id=
2026-09-21T15:24:47.404Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-445070a7-0246-8887-1ccb-f97557f8c6bc Address:445070a7-0246-8887-1ccb-f97557f8c6bc} {Suffrage:Voter ID:server-016378df-6123-b6e1-96e8-c50e7b0c6d08 Address:016378df-6123-b6e1-96e8-c50e7b0c6d08} {Suffrage:Voter ID:server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78 Address:8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78}]"
2026-09-21T15:24:47.404Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:47.404Z [INFO]  server-016378df-6123-b6e1-96e8-c50e7b0c6d08: entering follower state: follower="Node at 016378df-6123-b6e1-96e8-c50e7b0c6d08 [Follower]" leader-address= leader-id=
2026-09-21T15:24:47.404Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: entering follower state: follower="Node at 8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78 [Follower]" leader-address= leader-id=
2026-09-21T15:24:48.061Z [WARN]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:48.061Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: entering candidate state: node="Node at 8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78 [Candidate]" term=2
2026-09-21T15:24:48.062Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: election won: term=2 tally=2
2026-09-21T15:24:48.062Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: entering leader state: leader="Node at 8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78 [Leader]"
2026-09-21T15:24:48.062Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: added peer, starting replication: peer=server-445070a7-0246-8887-1ccb-f97557f8c6bc
2026-09-21T15:24:48.062Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: added peer, starting replication: peer=server-016378df-6123-b6e1-96e8-c50e7b0c6d08
2026-09-21T15:24:48.062Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: pipelining replication: peer="{Voter server-445070a7-0246-8887-1ccb-f97557f8c6bc 445070a7-0246-8887-1ccb-f97557f8c6bc}"
2026-09-21T15:24:48.062Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: pipelining replication: peer="{Voter server-016378df-6123-b6e1-96e8-c50e7b0c6d08 016378df-6123-b6e1-96e8-c50e7b0c6d08}"
2026-09-21T15:24:49.068Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:48.062265068 +0000 UTC m=+40.003435241 (1 nodes), 657.51464ms from start of poll, 657.9262ms from cluster start. Timeout at 2026-09-21 15:24:49.067272388 +0000 UTC m=+41.008442521, 1.00500728s after stability
2026-09-21T15:24:49.068Z [INFO]  creating new snapshot: path=/tmp/raft935313073/snapshots/2-14-1790004289068.tmp
2026-09-21T15:24:49.069Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: copied to local snapshot: bytes=71
2026-09-21T15:24:49.069Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: snapshot restore progress: id=2-14-1790004289068 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:49.069Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: removing all old logs from log store
2026-09-21T15:24:49.069Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: compacting logs: from=1 to=2
2026-09-21T15:24:49.069Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: restored user snapshot: index=14
2026-09-21T15:24:49.069Z [ERROR] server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: failed to get log: index=2 error="log not found"
2026-09-21T15:24:49.069Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: aborting pipeline replication: peer="{Voter server-016378df-6123-b6e1-96e8-c50e7b0c6d08 016378df-6123-b6e1-96e8-c50e7b0c6d08}"
2026-09-21T15:24:49.069Z [ERROR] server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: failed to get log: index=2 error="log not found"
2026-09-21T15:24:49.069Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: aborting pipeline replication: peer="{Voter server-445070a7-0246-8887-1ccb-f97557f8c6bc 445070a7-0246-8887-1ccb-f97557f8c6bc}"
2026-09-21T15:24:49.076Z [ERROR] server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: failed to get log: index=2 error="log not found"
2026-09-21T15:24:49.077Z [INFO]  creating new snapshot: path=/tmp/raft1399952623/snapshots/2-14-1790004289077.tmp
2026-09-21T15:24:49.077Z [INFO]  server-445070a7-0246-8887-1ccb-f97557f8c6bc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:49.078Z [ERROR] server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: failed to get log: index=2 error="log not found"
2026-09-21T15:24:49.078Z [INFO]  server-445070a7-0246-8887-1ccb-f97557f8c6bc: copied to local snapshot: bytes=71
2026-09-21T15:24:49.078Z [INFO]  creating new snapshot: path=/tmp/raft3256825374/snapshots/2-14-1790004289078.tmp
2026-09-21T15:24:49.078Z [INFO]  server-016378df-6123-b6e1-96e8-c50e7b0c6d08: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:49.078Z [INFO]  server-445070a7-0246-8887-1ccb-f97557f8c6bc: snapshot restore progress: id=2-14-1790004289077 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:49.078Z [INFO]  server-445070a7-0246-8887-1ccb-f97557f8c6bc: removing all old logs from log store
2026-09-21T15:24:49.078Z [INFO]  server-445070a7-0246-8887-1ccb-f97557f8c6bc: compacting logs: from=1 to=2
2026-09-21T15:24:49.078Z [INFO]  server-445070a7-0246-8887-1ccb-f97557f8c6bc: Installed remote snapshot
2026-09-21T15:24:49.078Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: pipelining replication: peer="{Voter server-445070a7-0246-8887-1ccb-f97557f8c6bc 445070a7-0246-8887-1ccb-f97557f8c6bc}"
2026-09-21T15:24:49.079Z [INFO]  server-016378df-6123-b6e1-96e8-c50e7b0c6d08: copied to local snapshot: bytes=71
2026-09-21T15:24:49.079Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:49.079Z [INFO]  server-016378df-6123-b6e1-96e8-c50e7b0c6d08: snapshot restore progress: id=2-14-1790004289078 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:49.079Z [INFO]  server-016378df-6123-b6e1-96e8-c50e7b0c6d08: removing all old logs from log store
2026-09-21T15:24:49.079Z [INFO]  server-016378df-6123-b6e1-96e8-c50e7b0c6d08: compacting logs: from=1 to=2
2026-09-21T15:24:49.079Z [INFO]  server-016378df-6123-b6e1-96e8-c50e7b0c6d08: Installed remote snapshot
2026-09-21T15:24:49.079Z [INFO]  server-8d9968a1-e035-3a4d-ff1a-0cd9b52d1f78: pipelining replication: peer="{Voter server-016378df-6123-b6e1-96e8-c50e7b0c6d08 016378df-6123-b6e1-96e8-c50e7b0c6d08}"
2026-09-21T15:24:50.084Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:49.079093748 +0000 UTC m=+41.020263921 (2 nodes), 1.52µs from start of poll, 1.67475488s from cluster start. Timeout at 2026-09-21 15:24:50.084117108 +0000 UTC m=+42.025287281, 1.00502336s after stability
2026-09-21T15:24:50.096Z [INFO]  server-d832ed58-e04f-b0e4-1de8-325c032af7d9: aborting pipeline replication: peer="{Voter server-0d576b24-e395-ba1b-cc46-b6706336ff97 0d576b24-e395-ba1b-cc46-b6706336ff97}"
2026-09-21T15:24:50.096Z [INFO]  server-d832ed58-e04f-b0e4-1de8-325c032af7d9: aborting pipeline replication: peer="{Voter server-0f7b82d4-abaa-b7d7-284b-f28df08c5279 0f7b82d4-abaa-b7d7-284b-f28df08c5279}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false
2026-09-21T15:24:50.097Z [INFO]  server-ee32bbe3-0407-e691-d49e-7d01b0b5c50a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee32bbe3-0407-e691-d49e-7d01b0b5c50a Address:ee32bbe3-0407-e691-d49e-7d01b0b5c50a} {Suffrage:Voter ID:server-1478b555-0d8c-094c-ce9e-875826ed9fbf Address:1478b555-0d8c-094c-ce9e-875826ed9fbf} {Suffrage:Voter ID:server-267952b5-2f52-e491-b320-c91e1febc649 Address:267952b5-2f52-e491-b320-c91e1febc649}]"
2026-09-21T15:24:50.097Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee32bbe3-0407-e691-d49e-7d01b0b5c50a Address:ee32bbe3-0407-e691-d49e-7d01b0b5c50a} {Suffrage:Voter ID:server-1478b555-0d8c-094c-ce9e-875826ed9fbf Address:1478b555-0d8c-094c-ce9e-875826ed9fbf} {Suffrage:Voter ID:server-267952b5-2f52-e491-b320-c91e1febc649 Address:267952b5-2f52-e491-b320-c91e1febc649}]"
2026-09-21T15:24:50.097Z [INFO]  server-ee32bbe3-0407-e691-d49e-7d01b0b5c50a: entering follower state: follower="Node at ee32bbe3-0407-e691-d49e-7d01b0b5c50a [Follower]" leader-address= leader-id=
2026-09-21T15:24:50.097Z [INFO]  server-267952b5-2f52-e491-b320-c91e1febc649: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee32bbe3-0407-e691-d49e-7d01b0b5c50a Address:ee32bbe3-0407-e691-d49e-7d01b0b5c50a} {Suffrage:Voter ID:server-1478b555-0d8c-094c-ce9e-875826ed9fbf Address:1478b555-0d8c-094c-ce9e-875826ed9fbf} {Suffrage:Voter ID:server-267952b5-2f52-e491-b320-c91e1febc649 Address:267952b5-2f52-e491-b320-c91e1febc649}]"
2026-09-21T15:24:50.097Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: entering follower state: follower="Node at 1478b555-0d8c-094c-ce9e-875826ed9fbf [Follower]" leader-address= leader-id=
2026-09-21T15:24:50.097Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:50.097Z [INFO]  server-267952b5-2f52-e491-b320-c91e1febc649: entering follower state: follower="Node at 267952b5-2f52-e491-b320-c91e1febc649 [Follower]" leader-address= leader-id=
2026-09-21T15:24:50.628Z [WARN]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:50.628Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: entering candidate state: node="Node at 1478b555-0d8c-094c-ce9e-875826ed9fbf [Candidate]" term=2
2026-09-21T15:24:50.628Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: election won: term=2 tally=2
2026-09-21T15:24:50.628Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: entering leader state: leader="Node at 1478b555-0d8c-094c-ce9e-875826ed9fbf [Leader]"
2026-09-21T15:24:50.628Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: added peer, starting replication: peer=server-ee32bbe3-0407-e691-d49e-7d01b0b5c50a
2026-09-21T15:24:50.628Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: added peer, starting replication: peer=server-267952b5-2f52-e491-b320-c91e1febc649
2026-09-21T15:24:50.628Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: pipelining replication: peer="{Voter server-267952b5-2f52-e491-b320-c91e1febc649 267952b5-2f52-e491-b320-c91e1febc649}"
2026-09-21T15:24:50.629Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: pipelining replication: peer="{Voter server-ee32bbe3-0407-e691-d49e-7d01b0b5c50a ee32bbe3-0407-e691-d49e-7d01b0b5c50a}"
2026-09-21T15:24:51.634Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:50.628411588 +0000 UTC m=+42.569581761 (1 nodes), 530.8532ms from start of poll, 531.26796ms from cluster start. Timeout at 2026-09-21 15:24:51.633415348 +0000 UTC m=+43.574585521, 1.00500376s after stability
2026-09-21T15:24:51.635Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: starting snapshot up to: index=12
2026-09-21T15:24:51.635Z [INFO]  creating new snapshot: path=/tmp/raft1900811789/snapshots/2-12-1790004291635.tmp
2026-09-21T15:24:51.635Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: snapshot complete up to: index=12
2026-09-21T15:24:51.636Z [INFO]  creating new snapshot: path=/tmp/raft1900811789/snapshots/2-23-1790004291636.tmp
2026-09-21T15:24:51.637Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: copied to local snapshot: bytes=71
2026-09-21T15:24:51.637Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: snapshot restore progress: id=2-23-1790004291636 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:51.637Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: restored user snapshot: index=23
2026-09-21T15:24:51.637Z [ERROR] server-1478b555-0d8c-094c-ce9e-875826ed9fbf: failed to get log: index=23 error="log not found"
2026-09-21T15:24:51.637Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: aborting pipeline replication: peer="{Voter server-267952b5-2f52-e491-b320-c91e1febc649 267952b5-2f52-e491-b320-c91e1febc649}"
2026-09-21T15:24:51.637Z [ERROR] server-1478b555-0d8c-094c-ce9e-875826ed9fbf: failed to get log: index=23 error="log not found"
2026-09-21T15:24:51.637Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: aborting pipeline replication: peer="{Voter server-ee32bbe3-0407-e691-d49e-7d01b0b5c50a ee32bbe3-0407-e691-d49e-7d01b0b5c50a}"
2026-09-21T15:24:51.644Z [ERROR] server-1478b555-0d8c-094c-ce9e-875826ed9fbf: failed to get log: index=23 error="log not found"
2026-09-21T15:24:51.644Z [ERROR] server-1478b555-0d8c-094c-ce9e-875826ed9fbf: failed to get log: index=23 error="log not found"
2026-09-21T15:24:51.644Z [INFO]  creating new snapshot: path=/tmp/raft3908258491/snapshots/2-23-1790004291644.tmp
2026-09-21T15:24:51.644Z [INFO]  creating new snapshot: path=/tmp/raft3010123785/snapshots/2-23-1790004291644.tmp
2026-09-21T15:24:51.644Z [INFO]  server-267952b5-2f52-e491-b320-c91e1febc649: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:51.645Z [INFO]  server-ee32bbe3-0407-e691-d49e-7d01b0b5c50a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:51.645Z [INFO]  server-267952b5-2f52-e491-b320-c91e1febc649: copied to local snapshot: bytes=71
2026-09-21T15:24:51.645Z [INFO]  server-267952b5-2f52-e491-b320-c91e1febc649: snapshot restore progress: id=2-23-1790004291644 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:51.645Z [INFO]  server-267952b5-2f52-e491-b320-c91e1febc649: Installed remote snapshot
2026-09-21T15:24:51.645Z [INFO]  server-ee32bbe3-0407-e691-d49e-7d01b0b5c50a: copied to local snapshot: bytes=71
2026-09-21T15:24:51.645Z [WARN]  server-267952b5-2f52-e491-b320-c91e1febc649: skipping application of old log: index=22
2026-09-21T15:24:51.645Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: pipelining replication: peer="{Voter server-267952b5-2f52-e491-b320-c91e1febc649 267952b5-2f52-e491-b320-c91e1febc649}"
2026-09-21T15:24:51.645Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:51.645Z [INFO]  server-ee32bbe3-0407-e691-d49e-7d01b0b5c50a: snapshot restore progress: id=2-23-1790004291644 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:51.645Z [INFO]  server-ee32bbe3-0407-e691-d49e-7d01b0b5c50a: Installed remote snapshot
2026-09-21T15:24:51.646Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: pipelining replication: peer="{Voter server-ee32bbe3-0407-e691-d49e-7d01b0b5c50a ee32bbe3-0407-e691-d49e-7d01b0b5c50a}"
2026-09-21T15:24:52.651Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:51.645758468 +0000 UTC m=+43.586928641 (2 nodes), 920ns from start of poll, 1.54861484s from cluster start. Timeout at 2026-09-21 15:24:52.650768428 +0000 UTC m=+44.591938601, 1.00500996s after stability
2026-09-21T15:24:52.664Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: aborting pipeline replication: peer="{Voter server-267952b5-2f52-e491-b320-c91e1febc649 267952b5-2f52-e491-b320-c91e1febc649}"
2026-09-21T15:24:52.664Z [INFO]  server-1478b555-0d8c-094c-ce9e-875826ed9fbf: aborting pipeline replication: peer="{Voter server-ee32bbe3-0407-e691-d49e-7d01b0b5c50a ee32bbe3-0407-e691-d49e-7d01b0b5c50a}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false
2026-09-21T15:24:52.665Z [INFO]  server-30834b27-dbd7-a638-5da7-d07413852049: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30834b27-dbd7-a638-5da7-d07413852049 Address:30834b27-dbd7-a638-5da7-d07413852049} {Suffrage:Voter ID:server-14a66c91-62ab-6344-0a95-73118883f988 Address:14a66c91-62ab-6344-0a95-73118883f988} {Suffrage:Voter ID:server-9da09b08-bf8a-6f3c-493a-ba99907bf854 Address:9da09b08-bf8a-6f3c-493a-ba99907bf854}]"
2026-09-21T15:24:52.665Z [INFO]  server-14a66c91-62ab-6344-0a95-73118883f988: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30834b27-dbd7-a638-5da7-d07413852049 Address:30834b27-dbd7-a638-5da7-d07413852049} {Suffrage:Voter ID:server-14a66c91-62ab-6344-0a95-73118883f988 Address:14a66c91-62ab-6344-0a95-73118883f988} {Suffrage:Voter ID:server-9da09b08-bf8a-6f3c-493a-ba99907bf854 Address:9da09b08-bf8a-6f3c-493a-ba99907bf854}]"
2026-09-21T15:24:52.665Z [INFO]  server-30834b27-dbd7-a638-5da7-d07413852049: entering follower state: follower="Node at 30834b27-dbd7-a638-5da7-d07413852049 [Follower]" leader-address= leader-id=
2026-09-21T15:24:52.666Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30834b27-dbd7-a638-5da7-d07413852049 Address:30834b27-dbd7-a638-5da7-d07413852049} {Suffrage:Voter ID:server-14a66c91-62ab-6344-0a95-73118883f988 Address:14a66c91-62ab-6344-0a95-73118883f988} {Suffrage:Voter ID:server-9da09b08-bf8a-6f3c-493a-ba99907bf854 Address:9da09b08-bf8a-6f3c-493a-ba99907bf854}]"
2026-09-21T15:24:52.666Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:52.666Z [INFO]  server-14a66c91-62ab-6344-0a95-73118883f988: entering follower state: follower="Node at 14a66c91-62ab-6344-0a95-73118883f988 [Follower]" leader-address= leader-id=
2026-09-21T15:24:52.666Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: entering follower state: follower="Node at 9da09b08-bf8a-6f3c-493a-ba99907bf854 [Follower]" leader-address= leader-id=
2026-09-21T15:24:53.301Z [WARN]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:53.301Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: entering candidate state: node="Node at 9da09b08-bf8a-6f3c-493a-ba99907bf854 [Candidate]" term=2
2026-09-21T15:24:53.302Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: election won: term=2 tally=2
2026-09-21T15:24:53.302Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: entering leader state: leader="Node at 9da09b08-bf8a-6f3c-493a-ba99907bf854 [Leader]"
2026-09-21T15:24:53.302Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: added peer, starting replication: peer=server-30834b27-dbd7-a638-5da7-d07413852049
2026-09-21T15:24:53.302Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: added peer, starting replication: peer=server-14a66c91-62ab-6344-0a95-73118883f988
2026-09-21T15:24:53.302Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: pipelining replication: peer="{Voter server-14a66c91-62ab-6344-0a95-73118883f988 14a66c91-62ab-6344-0a95-73118883f988}"
2026-09-21T15:24:53.302Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: pipelining replication: peer="{Voter server-30834b27-dbd7-a638-5da7-d07413852049 30834b27-dbd7-a638-5da7-d07413852049}"
2026-09-21T15:24:54.308Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:53.302138268 +0000 UTC m=+45.243308441 (1 nodes), 636.06676ms from start of poll, 636.4058ms from cluster start. Timeout at 2026-09-21 15:24:54.307142228 +0000 UTC m=+46.248312401, 1.00500396s after stability
2026-09-21T15:24:54.308Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: starting snapshot up to: index=12
2026-09-21T15:24:54.308Z [INFO]  creating new snapshot: path=/tmp/raft2346651082/snapshots/2-12-1790004294308.tmp
2026-09-21T15:24:54.309Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: snapshot complete up to: index=12
2026-09-21T15:24:54.310Z [INFO]  creating new snapshot: path=/tmp/raft2346651082/snapshots/2-23-1790004294310.tmp
2026-09-21T15:24:54.310Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: copied to local snapshot: bytes=71
2026-09-21T15:24:54.310Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: snapshot restore progress: id=2-23-1790004294310 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:54.311Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: removing all old logs from log store
2026-09-21T15:24:54.311Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: compacting logs: from=1 to=22
2026-09-21T15:24:54.311Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: restored user snapshot: index=23
2026-09-21T15:24:54.311Z [ERROR] server-9da09b08-bf8a-6f3c-493a-ba99907bf854: failed to get log: index=22 error="log not found"
2026-09-21T15:24:54.311Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: aborting pipeline replication: peer="{Voter server-14a66c91-62ab-6344-0a95-73118883f988 14a66c91-62ab-6344-0a95-73118883f988}"
2026-09-21T15:24:54.311Z [ERROR] server-9da09b08-bf8a-6f3c-493a-ba99907bf854: failed to get log: index=22 error="log not found"
2026-09-21T15:24:54.311Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: aborting pipeline replication: peer="{Voter server-30834b27-dbd7-a638-5da7-d07413852049 30834b27-dbd7-a638-5da7-d07413852049}"
2026-09-21T15:24:54.317Z [ERROR] server-9da09b08-bf8a-6f3c-493a-ba99907bf854: failed to get log: index=22 error="log not found"
2026-09-21T15:24:54.317Z [INFO]  creating new snapshot: path=/tmp/raft2130279197/snapshots/2-23-1790004294317.tmp
2026-09-21T15:24:54.318Z [INFO]  server-14a66c91-62ab-6344-0a95-73118883f988: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:54.318Z [INFO]  server-14a66c91-62ab-6344-0a95-73118883f988: copied to local snapshot: bytes=71
2026-09-21T15:24:54.318Z [INFO]  server-14a66c91-62ab-6344-0a95-73118883f988: snapshot restore progress: id=2-23-1790004294317 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:54.318Z [INFO]  server-14a66c91-62ab-6344-0a95-73118883f988: removing all old logs from log store
2026-09-21T15:24:54.318Z [INFO]  server-14a66c91-62ab-6344-0a95-73118883f988: compacting logs: from=1 to=22
2026-09-21T15:24:54.318Z [INFO]  server-14a66c91-62ab-6344-0a95-73118883f988: Installed remote snapshot
2026-09-21T15:24:54.318Z [WARN]  server-14a66c91-62ab-6344-0a95-73118883f988: skipping application of old log: index=22
2026-09-21T15:24:54.319Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: pipelining replication: peer="{Voter server-14a66c91-62ab-6344-0a95-73118883f988 14a66c91-62ab-6344-0a95-73118883f988}"
2026-09-21T15:24:54.319Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:54.320Z [ERROR] server-9da09b08-bf8a-6f3c-493a-ba99907bf854: failed to get log: index=22 error="log not found"
2026-09-21T15:24:54.320Z [INFO]  creating new snapshot: path=/tmp/raft1160555498/snapshots/2-23-1790004294320.tmp
2026-09-21T15:24:54.320Z [INFO]  server-30834b27-dbd7-a638-5da7-d07413852049: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:54.321Z [INFO]  server-30834b27-dbd7-a638-5da7-d07413852049: copied to local snapshot: bytes=71
2026-09-21T15:24:54.321Z [INFO]  server-30834b27-dbd7-a638-5da7-d07413852049: snapshot restore progress: id=2-23-1790004294320 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:54.321Z [INFO]  server-30834b27-dbd7-a638-5da7-d07413852049: removing all old logs from log store
2026-09-21T15:24:54.321Z [INFO]  server-30834b27-dbd7-a638-5da7-d07413852049: compacting logs: from=1 to=22
2026-09-21T15:24:54.321Z [INFO]  server-30834b27-dbd7-a638-5da7-d07413852049: Installed remote snapshot
2026-09-21T15:24:54.321Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: pipelining replication: peer="{Voter server-30834b27-dbd7-a638-5da7-d07413852049 30834b27-dbd7-a638-5da7-d07413852049}"
2026-09-21T15:24:55.324Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:54.319040428 +0000 UTC m=+46.260210601 (2 nodes), 1.08µs from start of poll, 1.65330796s from cluster start. Timeout at 2026-09-21 15:24:55.324054428 +0000 UTC m=+47.265224601, 1.005014s after stability
2026-09-21T15:24:55.337Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: aborting pipeline replication: peer="{Voter server-14a66c91-62ab-6344-0a95-73118883f988 14a66c91-62ab-6344-0a95-73118883f988}"
2026-09-21T15:24:55.337Z [INFO]  server-9da09b08-bf8a-6f3c-493a-ba99907bf854: aborting pipeline replication: peer="{Voter server-30834b27-dbd7-a638-5da7-d07413852049 30834b27-dbd7-a638-5da7-d07413852049}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true
2026-09-21T15:24:55.338Z [INFO]  server-76502eae-95cc-5b15-91a9-6e94ccbe33d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-76502eae-95cc-5b15-91a9-6e94ccbe33d0 Address:76502eae-95cc-5b15-91a9-6e94ccbe33d0} {Suffrage:Voter ID:server-09aada43-2a3b-7ff9-c99a-a0f48cc5f94b Address:09aada43-2a3b-7ff9-c99a-a0f48cc5f94b} {Suffrage:Voter ID:server-df04e680-2d3a-5410-6889-cd70bec3e3ad Address:df04e680-2d3a-5410-6889-cd70bec3e3ad}]"
2026-09-21T15:24:55.338Z [INFO]  server-76502eae-95cc-5b15-91a9-6e94ccbe33d0: entering follower state: follower="Node at 76502eae-95cc-5b15-91a9-6e94ccbe33d0 [Follower]" leader-address= leader-id=
2026-09-21T15:24:55.338Z [INFO]  server-09aada43-2a3b-7ff9-c99a-a0f48cc5f94b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-76502eae-95cc-5b15-91a9-6e94ccbe33d0 Address:76502eae-95cc-5b15-91a9-6e94ccbe33d0} {Suffrage:Voter ID:server-09aada43-2a3b-7ff9-c99a-a0f48cc5f94b Address:09aada43-2a3b-7ff9-c99a-a0f48cc5f94b} {Suffrage:Voter ID:server-df04e680-2d3a-5410-6889-cd70bec3e3ad Address:df04e680-2d3a-5410-6889-cd70bec3e3ad}]"
2026-09-21T15:24:55.338Z [INFO]  server-09aada43-2a3b-7ff9-c99a-a0f48cc5f94b: entering follower state: follower="Node at 09aada43-2a3b-7ff9-c99a-a0f48cc5f94b [Follower]" leader-address= leader-id=
2026-09-21T15:24:55.338Z [INFO]  server-df04e680-2d3a-5410-6889-cd70bec3e3ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-76502eae-95cc-5b15-91a9-6e94ccbe33d0 Address:76502eae-95cc-5b15-91a9-6e94ccbe33d0} {Suffrage:Voter ID:server-09aada43-2a3b-7ff9-c99a-a0f48cc5f94b Address:09aada43-2a3b-7ff9-c99a-a0f48cc5f94b} {Suffrage:Voter ID:server-df04e680-2d3a-5410-6889-cd70bec3e3ad Address:df04e680-2d3a-5410-6889-cd70bec3e3ad}]"
2026-09-21T15:24:55.338Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:55.338Z [INFO]  server-df04e680-2d3a-5410-6889-cd70bec3e3ad: entering follower state: follower="Node at df04e680-2d3a-5410-6889-cd70bec3e3ad [Follower]" leader-address= leader-id=
2026-09-21T15:24:55.979Z [WARN]  server-df04e680-2d3a-5410-6889-cd70bec3e3ad: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:55.979Z [INFO]  server-df04e680-2d3a-5410-6889-cd70bec3e3ad: entering candidate state: node="Node at df04e680-2d3a-5410-6889-cd70bec3e3ad [Candidate]" term=2
2026-09-21T15:24:55.979Z [INFO]  server-df04e680-2d3a-5410-6889-cd70bec3e3ad: election won: term=2 tally=2
2026-09-21T15:24:55.979Z [INFO]  server-df04e680-2d3a-5410-6889-cd70bec3e3ad: entering leader state: leader="Node at df04e680-2d3a-5410-6889-cd70bec3e3ad [Leader]"
2026-09-21T15:24:55.979Z [INFO]  server-df04e680-2d3a-5410-6889-cd70bec3e3ad: added peer, starting replication: peer=server-76502eae-95cc-5b15-91a9-6e94ccbe33d0
2026-09-21T15:24:55.979Z [INFO]  server-df04e680-2d3a-5410-6889-cd70bec3e3ad: added peer, starting replication: peer=server-09aada43-2a3b-7ff9-c99a-a0f48cc5f94b
2026-09-21T15:24:55.979Z [INFO]  server-df04e680-2d3a-5410-6889-cd70bec3e3ad: pipelining replication: peer="{Voter server-09aada43-2a3b-7ff9-c99a-a0f48cc5f94b 09aada43-2a3b-7ff9-c99a-a0f48cc5f94b}"
2026-09-21T15:24:55.980Z [INFO]  server-df04e680-2d3a-5410-6889-cd70bec3e3ad: pipelining replication: peer="{Voter server-76502eae-95cc-5b15-91a9-6e94ccbe33d0 76502eae-95cc-5b15-91a9-6e94ccbe33d0}"
2026-09-21T15:24:56.985Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:55.979843148 +0000 UTC m=+47.921013321 (1 nodes), 641.29008ms from start of poll, 641.81064ms from cluster start. Timeout at 2026-09-21 15:24:56.984845268 +0000 UTC m=+48.926015441, 1.00500212s after stability
2026-09-21T15:24:56.986Z [INFO]  server-df04e680-2d3a-5410-6889-cd70bec3e3ad: starting snapshot up to: index=12
2026-09-21T15:24:56.986Z [INFO]  creating new snapshot: path=/tmp/raft2000355239/snapshots/2-12-1790004296986.tmp
2026-09-21T15:24:56.986Z [INFO]  server-df04e680-2d3a-5410-6889-cd70bec3e3ad: snapshot complete up to: index=12
2026-09-21T15:24:56.987Z [INFO]  server-09494b1f-a844-70b4-3b54-6af216c5b5e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09494b1f-a844-70b4-3b54-6af216c5b5e8 Address:09494b1f-a844-70b4-3b54-6af216c5b5e8} {Suffrage:Voter ID:server-69dd61ec-4dab-541c-f4ef-37365593b56b Address:69dd61ec-4dab-541c-f4ef-37365593b56b} {Suffrage:Voter ID:server-abb4be65-b67b-cd82-0922-61f1a690497f Address:abb4be65-b67b-cd82-0922-61f1a690497f}]"
2026-09-21T15:24:56.987Z [INFO]  server-09494b1f-a844-70b4-3b54-6af216c5b5e8: entering follower state: follower="Node at 09494b1f-a844-70b4-3b54-6af216c5b5e8 [Follower]" leader-address= leader-id=
2026-09-21T15:24:56.987Z [INFO]  server-69dd61ec-4dab-541c-f4ef-37365593b56b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09494b1f-a844-70b4-3b54-6af216c5b5e8 Address:09494b1f-a844-70b4-3b54-6af216c5b5e8} {Suffrage:Voter ID:server-69dd61ec-4dab-541c-f4ef-37365593b56b Address:69dd61ec-4dab-541c-f4ef-37365593b56b} {Suffrage:Voter ID:server-abb4be65-b67b-cd82-0922-61f1a690497f Address:abb4be65-b67b-cd82-0922-61f1a690497f}]"
2026-09-21T15:24:56.987Z [INFO]  server-69dd61ec-4dab-541c-f4ef-37365593b56b: entering follower state: follower="Node at 69dd61ec-4dab-541c-f4ef-37365593b56b [Follower]" leader-address= leader-id=
2026-09-21T15:24:56.988Z [INFO]  server-abb4be65-b67b-cd82-0922-61f1a690497f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09494b1f-a844-70b4-3b54-6af216c5b5e8 Address:09494b1f-a844-70b4-3b54-6af216c5b5e8} {Suffrage:Voter ID:server-69dd61ec-4dab-541c-f4ef-37365593b56b Address:69dd61ec-4dab-541c-f4ef-37365593b56b} {Suffrage:Voter ID:server-abb4be65-b67b-cd82-0922-61f1a690497f Address:abb4be65-b67b-cd82-0922-61f1a690497f}]"
2026-09-21T15:24:56.988Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:56.988Z [INFO]  server-abb4be65-b67b-cd82-0922-61f1a690497f: entering follower state: follower="Node at abb4be65-b67b-cd82-0922-61f1a690497f [Follower]" leader-address= leader-id=
2026-09-21T15:24:57.595Z [WARN]  server-69dd61ec-4dab-541c-f4ef-37365593b56b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:24:57.595Z [INFO]  server-69dd61ec-4dab-541c-f4ef-37365593b56b: entering candidate state: node="Node at 69dd61ec-4dab-541c-f4ef-37365593b56b [Candidate]" term=2
2026-09-21T15:24:57.596Z [INFO]  server-69dd61ec-4dab-541c-f4ef-37365593b56b: election won: term=2 tally=2
2026-09-21T15:24:57.596Z [INFO]  server-69dd61ec-4dab-541c-f4ef-37365593b56b: entering leader state: leader="Node at 69dd61ec-4dab-541c-f4ef-37365593b56b [Leader]"
2026-09-21T15:24:57.596Z [INFO]  server-69dd61ec-4dab-541c-f4ef-37365593b56b: added peer, starting replication: peer=server-09494b1f-a844-70b4-3b54-6af216c5b5e8
2026-09-21T15:24:57.596Z [INFO]  server-69dd61ec-4dab-541c-f4ef-37365593b56b: added peer, starting replication: peer=server-abb4be65-b67b-cd82-0922-61f1a690497f
2026-09-21T15:24:57.596Z [INFO]  server-69dd61ec-4dab-541c-f4ef-37365593b56b: pipelining replication: peer="{Voter server-abb4be65-b67b-cd82-0922-61f1a690497f abb4be65-b67b-cd82-0922-61f1a690497f}"
2026-09-21T15:24:57.596Z [INFO]  server-69dd61ec-4dab-541c-f4ef-37365593b56b: pipelining replication: peer="{Voter server-09494b1f-a844-70b4-3b54-6af216c5b5e8 09494b1f-a844-70b4-3b54-6af216c5b5e8}"
2026-09-21T15:24:58.602Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:24:57.596054188 +0000 UTC m=+49.537224361 (1 nodes), 607.95356ms from start of poll, 608.4476ms from cluster start. Timeout at 2026-09-21 15:24:58.601058188 +0000 UTC m=+50.542228361, 1.005004s after stability
2026-09-21T15:24:58.602Z [INFO]  creating new snapshot: path=/tmp/raft1507338830/snapshots/2-15-1790004298602.tmp
2026-09-21T15:24:58.602Z [INFO]  server-69dd61ec-4dab-541c-f4ef-37365593b56b: copied to local snapshot: bytes=71
2026-09-21T15:24:58.603Z [INFO]  server-69dd61ec-4dab-541c-f4ef-37365593b56b: snapshot restore progress: id=2-15-1790004298602 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:58.603Z [INFO]  server-69dd61ec-4dab-541c-f4ef-37365593b56b: restored user snapshot: index=15
2026-09-21T15:24:58.604Z [ERROR] server-69dd61ec-4dab-541c-f4ef-37365593b56b: failed to get log: index=3 error="log not found"
2026-09-21T15:24:58.604Z [INFO]  server-69dd61ec-4dab-541c-f4ef-37365593b56b: aborting pipeline replication: peer="{Voter server-abb4be65-b67b-cd82-0922-61f1a690497f abb4be65-b67b-cd82-0922-61f1a690497f}"
2026-09-21T15:24:58.604Z [ERROR] server-69dd61ec-4dab-541c-f4ef-37365593b56b: failed to get log: index=3 error="log not found"
2026-09-21T15:24:58.604Z [INFO]  server-69dd61ec-4dab-541c-f4ef-37365593b56b: aborting pipeline replication: peer="{Voter server-09494b1f-a844-70b4-3b54-6af216c5b5e8 09494b1f-a844-70b4-3b54-6af216c5b5e8}"
2026-09-21T15:24:58.609Z [ERROR] server-69dd61ec-4dab-541c-f4ef-37365593b56b: failed to get log: index=3 error="log not found"
2026-09-21T15:24:58.610Z [INFO]  creating new snapshot: path=/tmp/raft3171990386/snapshots/2-15-1790004298610.tmp
2026-09-21T15:24:58.610Z [INFO]  server-abb4be65-b67b-cd82-0922-61f1a690497f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:58.611Z [INFO]  server-abb4be65-b67b-cd82-0922-61f1a690497f: copied to local snapshot: bytes=71
2026-09-21T15:24:58.611Z [ERROR] server-69dd61ec-4dab-541c-f4ef-37365593b56b: failed to get log: index=3 error="log not found"
2026-09-21T15:24:58.611Z [INFO]  server-abb4be65-b67b-cd82-0922-61f1a690497f: snapshot restore progress: id=2-15-1790004298610 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:58.611Z [INFO]  server-abb4be65-b67b-cd82-0922-61f1a690497f: Installed remote snapshot
2026-09-21T15:24:58.611Z [INFO]  server-69dd61ec-4dab-541c-f4ef-37365593b56b: pipelining replication: peer="{Voter server-abb4be65-b67b-cd82-0922-61f1a690497f abb4be65-b67b-cd82-0922-61f1a690497f}"
2026-09-21T15:24:58.611Z [INFO]  creating new snapshot: path=/tmp/raft3293271099/snapshots/2-15-1790004298611.tmp
2026-09-21T15:24:58.611Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:24:58.611Z [INFO]  server-09494b1f-a844-70b4-3b54-6af216c5b5e8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:58.612Z [INFO]  server-09494b1f-a844-70b4-3b54-6af216c5b5e8: copied to local snapshot: bytes=71
2026-09-21T15:24:58.612Z [INFO]  server-09494b1f-a844-70b4-3b54-6af216c5b5e8: snapshot restore progress: id=2-15-1790004298611 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:24:58.612Z [INFO]  server-09494b1f-a844-70b4-3b54-6af216c5b5e8: Installed remote snapshot
2026-09-21T15:24:58.612Z [INFO]  server-69dd61ec-4dab-541c-f4ef-37365593b56b: pipelining replication: peer="{Voter server-09494b1f-a844-70b4-3b54-6af216c5b5e8 09494b1f-a844-70b4-3b54-6af216c5b5e8}"
2026-09-21T15:24:59.616Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:24:58.611587388 +0000 UTC m=+50.552757521 (2 nodes), 1.84µs from start of poll, 1.62398076s from cluster start. Timeout at 2026-09-21 15:24:59.616602748 +0000 UTC m=+51.557772921, 1.0050154s after stability
2026-09-21T15:24:59.629Z [INFO]  server-df04e680-2d3a-5410-6889-cd70bec3e3ad: aborting pipeline replication: peer="{Voter server-76502eae-95cc-5b15-91a9-6e94ccbe33d0 76502eae-95cc-5b15-91a9-6e94ccbe33d0}"
2026-09-21T15:24:59.629Z [INFO]  server-df04e680-2d3a-5410-6889-cd70bec3e3ad: aborting pipeline replication: peer="{Voter server-09aada43-2a3b-7ff9-c99a-a0f48cc5f94b 09aada43-2a3b-7ff9-c99a-a0f48cc5f94b}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true
2026-09-21T15:24:59.630Z [INFO]  server-8deb683f-c090-0a66-6958-b34af820015b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8deb683f-c090-0a66-6958-b34af820015b Address:8deb683f-c090-0a66-6958-b34af820015b} {Suffrage:Voter ID:server-d3be3602-8b3c-fa21-70ec-22fdd5f9aa29 Address:d3be3602-8b3c-fa21-70ec-22fdd5f9aa29} {Suffrage:Voter ID:server-c6880ef6-3e34-b193-3224-98a6d565417a Address:c6880ef6-3e34-b193-3224-98a6d565417a}]"
2026-09-21T15:24:59.630Z [INFO]  server-d3be3602-8b3c-fa21-70ec-22fdd5f9aa29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8deb683f-c090-0a66-6958-b34af820015b Address:8deb683f-c090-0a66-6958-b34af820015b} {Suffrage:Voter ID:server-d3be3602-8b3c-fa21-70ec-22fdd5f9aa29 Address:d3be3602-8b3c-fa21-70ec-22fdd5f9aa29} {Suffrage:Voter ID:server-c6880ef6-3e34-b193-3224-98a6d565417a Address:c6880ef6-3e34-b193-3224-98a6d565417a}]"
2026-09-21T15:24:59.630Z [INFO]  server-8deb683f-c090-0a66-6958-b34af820015b: entering follower state: follower="Node at 8deb683f-c090-0a66-6958-b34af820015b [Follower]" leader-address= leader-id=
2026-09-21T15:24:59.630Z [INFO]  server-d3be3602-8b3c-fa21-70ec-22fdd5f9aa29: entering follower state: follower="Node at d3be3602-8b3c-fa21-70ec-22fdd5f9aa29 [Follower]" leader-address= leader-id=
2026-09-21T15:24:59.630Z [INFO]  server-c6880ef6-3e34-b193-3224-98a6d565417a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8deb683f-c090-0a66-6958-b34af820015b Address:8deb683f-c090-0a66-6958-b34af820015b} {Suffrage:Voter ID:server-d3be3602-8b3c-fa21-70ec-22fdd5f9aa29 Address:d3be3602-8b3c-fa21-70ec-22fdd5f9aa29} {Suffrage:Voter ID:server-c6880ef6-3e34-b193-3224-98a6d565417a Address:c6880ef6-3e34-b193-3224-98a6d565417a}]"
2026-09-21T15:24:59.630Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:24:59.630Z [INFO]  server-c6880ef6-3e34-b193-3224-98a6d565417a: entering follower state: follower="Node at c6880ef6-3e34-b193-3224-98a6d565417a [Follower]" leader-address= leader-id=
2026-09-21T15:25:00.234Z [WARN]  server-d3be3602-8b3c-fa21-70ec-22fdd5f9aa29: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:00.234Z [INFO]  server-d3be3602-8b3c-fa21-70ec-22fdd5f9aa29: entering candidate state: node="Node at d3be3602-8b3c-fa21-70ec-22fdd5f9aa29 [Candidate]" term=2
2026-09-21T15:25:00.234Z [INFO]  server-d3be3602-8b3c-fa21-70ec-22fdd5f9aa29: election won: term=2 tally=2
2026-09-21T15:25:00.234Z [INFO]  server-d3be3602-8b3c-fa21-70ec-22fdd5f9aa29: entering leader state: leader="Node at d3be3602-8b3c-fa21-70ec-22fdd5f9aa29 [Leader]"
2026-09-21T15:25:00.234Z [INFO]  server-d3be3602-8b3c-fa21-70ec-22fdd5f9aa29: added peer, starting replication: peer=server-8deb683f-c090-0a66-6958-b34af820015b
2026-09-21T15:25:00.234Z [INFO]  server-d3be3602-8b3c-fa21-70ec-22fdd5f9aa29: added peer, starting replication: peer=server-c6880ef6-3e34-b193-3224-98a6d565417a
2026-09-21T15:25:00.234Z [INFO]  server-d3be3602-8b3c-fa21-70ec-22fdd5f9aa29: pipelining replication: peer="{Voter server-c6880ef6-3e34-b193-3224-98a6d565417a c6880ef6-3e34-b193-3224-98a6d565417a}"
2026-09-21T15:25:00.234Z [INFO]  server-d3be3602-8b3c-fa21-70ec-22fdd5f9aa29: pipelining replication: peer="{Voter server-8deb683f-c090-0a66-6958-b34af820015b 8deb683f-c090-0a66-6958-b34af820015b}"
2026-09-21T15:25:01.240Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:00.234357828 +0000 UTC m=+52.175528001 (1 nodes), 603.8254ms from start of poll, 604.26548ms from cluster start. Timeout at 2026-09-21 15:25:01.239361228 +0000 UTC m=+53.180531401, 1.0050034s after stability
2026-09-21T15:25:01.240Z [INFO]  server-d3be3602-8b3c-fa21-70ec-22fdd5f9aa29: starting snapshot up to: index=12
2026-09-21T15:25:01.240Z [INFO]  creating new snapshot: path=/tmp/raft4018239123/snapshots/2-12-1790004301240.tmp
2026-09-21T15:25:01.241Z [INFO]  server-d3be3602-8b3c-fa21-70ec-22fdd5f9aa29: snapshot complete up to: index=12
2026-09-21T15:25:01.242Z [INFO]  server-ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331 Address:ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331} {Suffrage:Voter ID:server-6f36b611-28e6-70c2-0826-9c8888897bfe Address:6f36b611-28e6-70c2-0826-9c8888897bfe} {Suffrage:Voter ID:server-868938e7-3c2f-51c4-0bd9-41c02eb2015a Address:868938e7-3c2f-51c4-0bd9-41c02eb2015a}]"
2026-09-21T15:25:01.242Z [INFO]  server-6f36b611-28e6-70c2-0826-9c8888897bfe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331 Address:ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331} {Suffrage:Voter ID:server-6f36b611-28e6-70c2-0826-9c8888897bfe Address:6f36b611-28e6-70c2-0826-9c8888897bfe} {Suffrage:Voter ID:server-868938e7-3c2f-51c4-0bd9-41c02eb2015a Address:868938e7-3c2f-51c4-0bd9-41c02eb2015a}]"
2026-09-21T15:25:01.242Z [INFO]  server-ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331: entering follower state: follower="Node at ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331 [Follower]" leader-address= leader-id=
2026-09-21T15:25:01.243Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331 Address:ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331} {Suffrage:Voter ID:server-6f36b611-28e6-70c2-0826-9c8888897bfe Address:6f36b611-28e6-70c2-0826-9c8888897bfe} {Suffrage:Voter ID:server-868938e7-3c2f-51c4-0bd9-41c02eb2015a Address:868938e7-3c2f-51c4-0bd9-41c02eb2015a}]"
2026-09-21T15:25:01.243Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:01.242Z [INFO]  server-6f36b611-28e6-70c2-0826-9c8888897bfe: entering follower state: follower="Node at 6f36b611-28e6-70c2-0826-9c8888897bfe [Follower]" leader-address= leader-id=
2026-09-21T15:25:01.243Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: entering follower state: follower="Node at 868938e7-3c2f-51c4-0bd9-41c02eb2015a [Follower]" leader-address= leader-id=
2026-09-21T15:25:01.880Z [WARN]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:01.880Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: entering candidate state: node="Node at 868938e7-3c2f-51c4-0bd9-41c02eb2015a [Candidate]" term=2
2026-09-21T15:25:01.880Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: election won: term=2 tally=2
2026-09-21T15:25:01.880Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: entering leader state: leader="Node at 868938e7-3c2f-51c4-0bd9-41c02eb2015a [Leader]"
2026-09-21T15:25:01.880Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: added peer, starting replication: peer=server-ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331
2026-09-21T15:25:01.880Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: added peer, starting replication: peer=server-6f36b611-28e6-70c2-0826-9c8888897bfe
2026-09-21T15:25:01.881Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: pipelining replication: peer="{Voter server-6f36b611-28e6-70c2-0826-9c8888897bfe 6f36b611-28e6-70c2-0826-9c8888897bfe}"
2026-09-21T15:25:01.881Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: pipelining replication: peer="{Voter server-ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331 ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331}"
2026-09-21T15:25:02.887Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:01.881037068 +0000 UTC m=+53.822207241 (1 nodes), 637.9638ms from start of poll, 638.36624ms from cluster start. Timeout at 2026-09-21 15:25:02.886039668 +0000 UTC m=+54.827209881, 1.00500264s after stability
2026-09-21T15:25:02.887Z [INFO]  creating new snapshot: path=/tmp/raft3619150888/snapshots/2-15-1790004302887.tmp
2026-09-21T15:25:02.888Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: copied to local snapshot: bytes=71
2026-09-21T15:25:02.888Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: snapshot restore progress: id=2-15-1790004302887 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:02.888Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: removing all old logs from log store
2026-09-21T15:25:02.888Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: compacting logs: from=1 to=2
2026-09-21T15:25:02.888Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: restored user snapshot: index=15
2026-09-21T15:25:02.889Z [ERROR] server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: failed to get log: index=2 error="log not found"
2026-09-21T15:25:02.889Z [ERROR] server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: failed to get log: index=2 error="log not found"
2026-09-21T15:25:02.889Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: aborting pipeline replication: peer="{Voter server-ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331 ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331}"
2026-09-21T15:25:02.889Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: aborting pipeline replication: peer="{Voter server-6f36b611-28e6-70c2-0826-9c8888897bfe 6f36b611-28e6-70c2-0826-9c8888897bfe}"
2026-09-21T15:25:02.894Z [ERROR] server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: failed to get log: index=2 error="log not found"
2026-09-21T15:25:02.895Z [INFO]  creating new snapshot: path=/tmp/raft4130633598/snapshots/2-15-1790004302895.tmp
2026-09-21T15:25:02.895Z [INFO]  server-6f36b611-28e6-70c2-0826-9c8888897bfe: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:02.895Z [INFO]  server-6f36b611-28e6-70c2-0826-9c8888897bfe: copied to local snapshot: bytes=71
2026-09-21T15:25:02.895Z [INFO]  server-6f36b611-28e6-70c2-0826-9c8888897bfe: snapshot restore progress: id=2-15-1790004302895 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:02.896Z [INFO]  server-6f36b611-28e6-70c2-0826-9c8888897bfe: removing all old logs from log store
2026-09-21T15:25:02.896Z [INFO]  server-6f36b611-28e6-70c2-0826-9c8888897bfe: compacting logs: from=1 to=2
2026-09-21T15:25:02.896Z [INFO]  server-6f36b611-28e6-70c2-0826-9c8888897bfe: Installed remote snapshot
2026-09-21T15:25:02.896Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: pipelining replication: peer="{Voter server-6f36b611-28e6-70c2-0826-9c8888897bfe 6f36b611-28e6-70c2-0826-9c8888897bfe}"
2026-09-21T15:25:02.896Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:02.898Z [ERROR] server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: failed to get log: index=2 error="log not found"
2026-09-21T15:25:02.898Z [INFO]  creating new snapshot: path=/tmp/raft2445903687/snapshots/2-15-1790004302898.tmp
2026-09-21T15:25:02.898Z [INFO]  server-ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:02.899Z [INFO]  server-ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331: copied to local snapshot: bytes=71
2026-09-21T15:25:02.899Z [INFO]  server-ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331: snapshot restore progress: id=2-15-1790004302898 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:02.899Z [INFO]  server-ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331: removing all old logs from log store
2026-09-21T15:25:02.899Z [INFO]  server-ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331: compacting logs: from=1 to=2
2026-09-21T15:25:02.899Z [INFO]  server-ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331: Installed remote snapshot
2026-09-21T15:25:02.899Z [INFO]  server-868938e7-3c2f-51c4-0bd9-41c02eb2015a: pipelining replication: peer="{Voter server-ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331 ad7c9c7a-eb8d-2dd2-5763-37cd0bcfd331}"
2026-09-21T15:25:03.901Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:02.896243028 +0000 UTC m=+54.837413201 (2 nodes), 1µs from start of poll, 1.6535722s from cluster start. Timeout at 2026-09-21 15:25:03.901264628 +0000 UTC m=+55.842434801, 1.0050216s after stability
2026-09-21T15:25:03.913Z [INFO]  server-d3be3602-8b3c-fa21-70ec-22fdd5f9aa29: aborting pipeline replication: peer="{Voter server-c6880ef6-3e34-b193-3224-98a6d565417a c6880ef6-3e34-b193-3224-98a6d565417a}"
2026-09-21T15:25:03.913Z [INFO]  server-d3be3602-8b3c-fa21-70ec-22fdd5f9aa29: aborting pipeline replication: peer="{Voter server-8deb683f-c090-0a66-6958-b34af820015b 8deb683f-c090-0a66-6958-b34af820015b}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false
2026-09-21T15:25:03.915Z [INFO]  server-3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697 Address:3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697} {Suffrage:Voter ID:server-f7483d34-46d9-133b-511c-13b76d64807f Address:f7483d34-46d9-133b-511c-13b76d64807f} {Suffrage:Voter ID:server-cdbc73da-b014-7be3-c6fc-25619e2e57ce Address:cdbc73da-b014-7be3-c6fc-25619e2e57ce}]"
2026-09-21T15:25:03.915Z [INFO]  server-3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697: entering follower state: follower="Node at 3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697 [Follower]" leader-address= leader-id=
2026-09-21T15:25:03.916Z [INFO]  server-f7483d34-46d9-133b-511c-13b76d64807f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697 Address:3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697} {Suffrage:Voter ID:server-f7483d34-46d9-133b-511c-13b76d64807f Address:f7483d34-46d9-133b-511c-13b76d64807f} {Suffrage:Voter ID:server-cdbc73da-b014-7be3-c6fc-25619e2e57ce Address:cdbc73da-b014-7be3-c6fc-25619e2e57ce}]"
2026-09-21T15:25:03.916Z [INFO]  server-f7483d34-46d9-133b-511c-13b76d64807f: entering follower state: follower="Node at f7483d34-46d9-133b-511c-13b76d64807f [Follower]" leader-address= leader-id=
2026-09-21T15:25:03.916Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697 Address:3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697} {Suffrage:Voter ID:server-f7483d34-46d9-133b-511c-13b76d64807f Address:f7483d34-46d9-133b-511c-13b76d64807f} {Suffrage:Voter ID:server-cdbc73da-b014-7be3-c6fc-25619e2e57ce Address:cdbc73da-b014-7be3-c6fc-25619e2e57ce}]"
2026-09-21T15:25:03.916Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:03.916Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: entering follower state: follower="Node at cdbc73da-b014-7be3-c6fc-25619e2e57ce [Follower]" leader-address= leader-id=
2026-09-21T15:25:04.471Z [WARN]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:04.471Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: entering candidate state: node="Node at cdbc73da-b014-7be3-c6fc-25619e2e57ce [Candidate]" term=2
2026-09-21T15:25:04.471Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: election won: term=2 tally=2
2026-09-21T15:25:04.471Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: entering leader state: leader="Node at cdbc73da-b014-7be3-c6fc-25619e2e57ce [Leader]"
2026-09-21T15:25:04.471Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: added peer, starting replication: peer=server-3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697
2026-09-21T15:25:04.471Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: added peer, starting replication: peer=server-f7483d34-46d9-133b-511c-13b76d64807f
2026-09-21T15:25:04.471Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: pipelining replication: peer="{Voter server-3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697 3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697}"
2026-09-21T15:25:04.471Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: pipelining replication: peer="{Voter server-f7483d34-46d9-133b-511c-13b76d64807f f7483d34-46d9-133b-511c-13b76d64807f}"
2026-09-21T15:25:05.476Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:04.471793348 +0000 UTC m=+56.412963521 (1 nodes), 554.8358ms from start of poll, 556.22132ms from cluster start. Timeout at 2026-09-21 15:25:05.476796708 +0000 UTC m=+57.417966881, 1.00500336s after stability
2026-09-21T15:25:05.477Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: starting snapshot up to: index=12
2026-09-21T15:25:05.477Z [INFO]  creating new snapshot: path=/tmp/raft2783002533/snapshots/2-12-1790004305477.tmp
2026-09-21T15:25:05.478Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: snapshot complete up to: index=12
2026-09-21T15:25:05.478Z [INFO]  creating new snapshot: path=/tmp/raft2783002533/snapshots/2-113-1790004305478.tmp
2026-09-21T15:25:05.479Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: copied to local snapshot: bytes=71
2026-09-21T15:25:05.479Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: snapshot restore progress: id=2-113-1790004305478 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:05.479Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: restored user snapshot: index=113
2026-09-21T15:25:05.479Z [ERROR] server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: failed to get log: index=23 error="log not found"
2026-09-21T15:25:05.479Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: aborting pipeline replication: peer="{Voter server-f7483d34-46d9-133b-511c-13b76d64807f f7483d34-46d9-133b-511c-13b76d64807f}"
2026-09-21T15:25:05.479Z [ERROR] server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: failed to get log: index=23 error="log not found"
2026-09-21T15:25:05.479Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: aborting pipeline replication: peer="{Voter server-3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697 3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697}"
2026-09-21T15:25:05.487Z [ERROR] server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: failed to get log: index=23 error="log not found"
2026-09-21T15:25:05.488Z [INFO]  creating new snapshot: path=/tmp/raft1215789592/snapshots/2-113-1790004305488.tmp
2026-09-21T15:25:05.488Z [INFO]  server-f7483d34-46d9-133b-511c-13b76d64807f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:05.488Z [INFO]  server-f7483d34-46d9-133b-511c-13b76d64807f: copied to local snapshot: bytes=71
2026-09-21T15:25:05.489Z [ERROR] server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: failed to get log: index=23 error="log not found"
2026-09-21T15:25:05.489Z [INFO]  server-f7483d34-46d9-133b-511c-13b76d64807f: snapshot restore progress: id=2-113-1790004305488 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:05.489Z [INFO]  server-f7483d34-46d9-133b-511c-13b76d64807f: Installed remote snapshot
2026-09-21T15:25:05.490Z [INFO]  creating new snapshot: path=/tmp/raft3182088537/snapshots/2-113-1790004305490.tmp
2026-09-21T15:25:05.490Z [WARN]  server-f7483d34-46d9-133b-511c-13b76d64807f: skipping application of old log: index=22
2026-09-21T15:25:05.490Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: pipelining replication: peer="{Voter server-f7483d34-46d9-133b-511c-13b76d64807f f7483d34-46d9-133b-511c-13b76d64807f}"
2026-09-21T15:25:05.491Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:05.491Z [INFO]  server-3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:05.491Z [INFO]  server-3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697: copied to local snapshot: bytes=71
2026-09-21T15:25:05.492Z [INFO]  server-3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697: snapshot restore progress: id=2-113-1790004305490 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:05.492Z [INFO]  server-3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697: Installed remote snapshot
2026-09-21T15:25:05.492Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: pipelining replication: peer="{Voter server-3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697 3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697}"
2026-09-21T15:25:06.496Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:05.491275948 +0000 UTC m=+57.432446121 (2 nodes), 2.68µs from start of poll, 1.57570392s from cluster start. Timeout at 2026-09-21 15:25:06.496293668 +0000 UTC m=+58.437463841, 1.00501772s after stability
2026-09-21T15:25:06.509Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: aborting pipeline replication: peer="{Voter server-3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697 3f7c0ff4-f2a6-cb8d-61b1-a9d43714e697}"
2026-09-21T15:25:06.509Z [INFO]  server-cdbc73da-b014-7be3-c6fc-25619e2e57ce: aborting pipeline replication: peer="{Voter server-f7483d34-46d9-133b-511c-13b76d64807f f7483d34-46d9-133b-511c-13b76d64807f}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false
2026-09-21T15:25:06.510Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-390f05f7-34b0-ab37-0380-01e8892e15f6 Address:390f05f7-34b0-ab37-0380-01e8892e15f6} {Suffrage:Voter ID:server-e49c5526-c3c9-9571-bb58-0a98d16f7018 Address:e49c5526-c3c9-9571-bb58-0a98d16f7018} {Suffrage:Voter ID:server-48f3b99f-c77b-14db-f4f4-b4eda0673fa9 Address:48f3b99f-c77b-14db-f4f4-b4eda0673fa9}]"
2026-09-21T15:25:06.510Z [INFO]  server-e49c5526-c3c9-9571-bb58-0a98d16f7018: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-390f05f7-34b0-ab37-0380-01e8892e15f6 Address:390f05f7-34b0-ab37-0380-01e8892e15f6} {Suffrage:Voter ID:server-e49c5526-c3c9-9571-bb58-0a98d16f7018 Address:e49c5526-c3c9-9571-bb58-0a98d16f7018} {Suffrage:Voter ID:server-48f3b99f-c77b-14db-f4f4-b4eda0673fa9 Address:48f3b99f-c77b-14db-f4f4-b4eda0673fa9}]"
2026-09-21T15:25:06.510Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: entering follower state: follower="Node at 390f05f7-34b0-ab37-0380-01e8892e15f6 [Follower]" leader-address= leader-id=
2026-09-21T15:25:06.510Z [INFO]  server-e49c5526-c3c9-9571-bb58-0a98d16f7018: entering follower state: follower="Node at e49c5526-c3c9-9571-bb58-0a98d16f7018 [Follower]" leader-address= leader-id=
2026-09-21T15:25:06.510Z [INFO]  server-48f3b99f-c77b-14db-f4f4-b4eda0673fa9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-390f05f7-34b0-ab37-0380-01e8892e15f6 Address:390f05f7-34b0-ab37-0380-01e8892e15f6} {Suffrage:Voter ID:server-e49c5526-c3c9-9571-bb58-0a98d16f7018 Address:e49c5526-c3c9-9571-bb58-0a98d16f7018} {Suffrage:Voter ID:server-48f3b99f-c77b-14db-f4f4-b4eda0673fa9 Address:48f3b99f-c77b-14db-f4f4-b4eda0673fa9}]"
2026-09-21T15:25:06.510Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:06.510Z [INFO]  server-48f3b99f-c77b-14db-f4f4-b4eda0673fa9: entering follower state: follower="Node at 48f3b99f-c77b-14db-f4f4-b4eda0673fa9 [Follower]" leader-address= leader-id=
2026-09-21T15:25:07.127Z [WARN]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:07.127Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: entering candidate state: node="Node at 390f05f7-34b0-ab37-0380-01e8892e15f6 [Candidate]" term=2
2026-09-21T15:25:07.127Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: election won: term=2 tally=2
2026-09-21T15:25:07.127Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: entering leader state: leader="Node at 390f05f7-34b0-ab37-0380-01e8892e15f6 [Leader]"
2026-09-21T15:25:07.127Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: added peer, starting replication: peer=server-e49c5526-c3c9-9571-bb58-0a98d16f7018
2026-09-21T15:25:07.128Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: added peer, starting replication: peer=server-48f3b99f-c77b-14db-f4f4-b4eda0673fa9
2026-09-21T15:25:07.128Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: pipelining replication: peer="{Voter server-48f3b99f-c77b-14db-f4f4-b4eda0673fa9 48f3b99f-c77b-14db-f4f4-b4eda0673fa9}"
2026-09-21T15:25:07.128Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: pipelining replication: peer="{Voter server-e49c5526-c3c9-9571-bb58-0a98d16f7018 e49c5526-c3c9-9571-bb58-0a98d16f7018}"
2026-09-21T15:25:08.134Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:07.127977908 +0000 UTC m=+59.069148081 (1 nodes), 617.08388ms from start of poll, 617.5476ms from cluster start. Timeout at 2026-09-21 15:25:08.132980388 +0000 UTC m=+60.074150561, 1.00500248s after stability
2026-09-21T15:25:08.134Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: starting snapshot up to: index=12
2026-09-21T15:25:08.134Z [INFO]  creating new snapshot: path=/tmp/raft2690011297/snapshots/2-12-1790004308134.tmp
2026-09-21T15:25:08.135Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: snapshot complete up to: index=12
2026-09-21T15:25:08.137Z [INFO]  creating new snapshot: path=/tmp/raft2690011297/snapshots/2-113-1790004308137.tmp
2026-09-21T15:25:08.138Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: copied to local snapshot: bytes=71
2026-09-21T15:25:08.138Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: snapshot restore progress: id=2-113-1790004308137 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:08.138Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: removing all old logs from log store
2026-09-21T15:25:08.138Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: compacting logs: from=1 to=22
2026-09-21T15:25:08.138Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: restored user snapshot: index=113
2026-09-21T15:25:08.138Z [ERROR] server-390f05f7-34b0-ab37-0380-01e8892e15f6: failed to get log: index=22 error="log not found"
2026-09-21T15:25:08.138Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: aborting pipeline replication: peer="{Voter server-48f3b99f-c77b-14db-f4f4-b4eda0673fa9 48f3b99f-c77b-14db-f4f4-b4eda0673fa9}"
2026-09-21T15:25:08.138Z [ERROR] server-390f05f7-34b0-ab37-0380-01e8892e15f6: failed to get log: index=22 error="log not found"
2026-09-21T15:25:08.138Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: aborting pipeline replication: peer="{Voter server-e49c5526-c3c9-9571-bb58-0a98d16f7018 e49c5526-c3c9-9571-bb58-0a98d16f7018}"
2026-09-21T15:25:08.145Z [ERROR] server-390f05f7-34b0-ab37-0380-01e8892e15f6: failed to get log: index=22 error="log not found"
2026-09-21T15:25:08.146Z [INFO]  creating new snapshot: path=/tmp/raft2049499297/snapshots/2-113-1790004308146.tmp
2026-09-21T15:25:08.146Z [INFO]  server-48f3b99f-c77b-14db-f4f4-b4eda0673fa9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:08.146Z [INFO]  server-48f3b99f-c77b-14db-f4f4-b4eda0673fa9: copied to local snapshot: bytes=71
2026-09-21T15:25:08.146Z [INFO]  server-48f3b99f-c77b-14db-f4f4-b4eda0673fa9: snapshot restore progress: id=2-113-1790004308146 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:08.146Z [INFO]  server-48f3b99f-c77b-14db-f4f4-b4eda0673fa9: removing all old logs from log store
2026-09-21T15:25:08.146Z [INFO]  server-48f3b99f-c77b-14db-f4f4-b4eda0673fa9: compacting logs: from=1 to=22
2026-09-21T15:25:08.146Z [INFO]  server-48f3b99f-c77b-14db-f4f4-b4eda0673fa9: Installed remote snapshot
2026-09-21T15:25:08.147Z [WARN]  server-48f3b99f-c77b-14db-f4f4-b4eda0673fa9: skipping application of old log: index=22
2026-09-21T15:25:08.147Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: pipelining replication: peer="{Voter server-48f3b99f-c77b-14db-f4f4-b4eda0673fa9 48f3b99f-c77b-14db-f4f4-b4eda0673fa9}"
2026-09-21T15:25:08.147Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:08.147Z [ERROR] server-390f05f7-34b0-ab37-0380-01e8892e15f6: failed to get log: index=22 error="log not found"
2026-09-21T15:25:08.148Z [INFO]  creating new snapshot: path=/tmp/raft4067895393/snapshots/2-113-1790004308148.tmp
2026-09-21T15:25:08.148Z [INFO]  server-e49c5526-c3c9-9571-bb58-0a98d16f7018: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:08.148Z [INFO]  server-e49c5526-c3c9-9571-bb58-0a98d16f7018: copied to local snapshot: bytes=71
2026-09-21T15:25:08.149Z [INFO]  server-e49c5526-c3c9-9571-bb58-0a98d16f7018: snapshot restore progress: id=2-113-1790004308148 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:08.149Z [INFO]  server-e49c5526-c3c9-9571-bb58-0a98d16f7018: removing all old logs from log store
2026-09-21T15:25:08.149Z [INFO]  server-e49c5526-c3c9-9571-bb58-0a98d16f7018: compacting logs: from=1 to=22
2026-09-21T15:25:08.149Z [INFO]  server-e49c5526-c3c9-9571-bb58-0a98d16f7018: Installed remote snapshot
2026-09-21T15:25:08.149Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: pipelining replication: peer="{Voter server-e49c5526-c3c9-9571-bb58-0a98d16f7018 e49c5526-c3c9-9571-bb58-0a98d16f7018}"
2026-09-21T15:25:09.152Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:08.147152428 +0000 UTC m=+60.088322601 (2 nodes), 1.24µs from start of poll, 1.63672212s from cluster start. Timeout at 2026-09-21 15:25:09.152164908 +0000 UTC m=+61.093335081, 1.00501248s after stability
2026-09-21T15:25:09.164Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: aborting pipeline replication: peer="{Voter server-e49c5526-c3c9-9571-bb58-0a98d16f7018 e49c5526-c3c9-9571-bb58-0a98d16f7018}"
2026-09-21T15:25:09.164Z [INFO]  server-390f05f7-34b0-ab37-0380-01e8892e15f6: aborting pipeline replication: peer="{Voter server-48f3b99f-c77b-14db-f4f4-b4eda0673fa9 48f3b99f-c77b-14db-f4f4-b4eda0673fa9}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true
2026-09-21T15:25:09.166Z [INFO]  server-6786e347-f362-dd02-15ea-998139c6507e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6786e347-f362-dd02-15ea-998139c6507e Address:6786e347-f362-dd02-15ea-998139c6507e} {Suffrage:Voter ID:server-9e408929-bbf3-4f91-35de-c07b11888721 Address:9e408929-bbf3-4f91-35de-c07b11888721} {Suffrage:Voter ID:server-85918948-25ba-f42c-1faf-0cd11aa4f6fa Address:85918948-25ba-f42c-1faf-0cd11aa4f6fa}]"
2026-09-21T15:25:09.166Z [INFO]  server-9e408929-bbf3-4f91-35de-c07b11888721: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6786e347-f362-dd02-15ea-998139c6507e Address:6786e347-f362-dd02-15ea-998139c6507e} {Suffrage:Voter ID:server-9e408929-bbf3-4f91-35de-c07b11888721 Address:9e408929-bbf3-4f91-35de-c07b11888721} {Suffrage:Voter ID:server-85918948-25ba-f42c-1faf-0cd11aa4f6fa Address:85918948-25ba-f42c-1faf-0cd11aa4f6fa}]"
2026-09-21T15:25:09.166Z [INFO]  server-6786e347-f362-dd02-15ea-998139c6507e: entering follower state: follower="Node at 6786e347-f362-dd02-15ea-998139c6507e [Follower]" leader-address= leader-id=
2026-09-21T15:25:09.166Z [INFO]  server-9e408929-bbf3-4f91-35de-c07b11888721: entering follower state: follower="Node at 9e408929-bbf3-4f91-35de-c07b11888721 [Follower]" leader-address= leader-id=
2026-09-21T15:25:09.166Z [INFO]  server-85918948-25ba-f42c-1faf-0cd11aa4f6fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6786e347-f362-dd02-15ea-998139c6507e Address:6786e347-f362-dd02-15ea-998139c6507e} {Suffrage:Voter ID:server-9e408929-bbf3-4f91-35de-c07b11888721 Address:9e408929-bbf3-4f91-35de-c07b11888721} {Suffrage:Voter ID:server-85918948-25ba-f42c-1faf-0cd11aa4f6fa Address:85918948-25ba-f42c-1faf-0cd11aa4f6fa}]"
2026-09-21T15:25:09.166Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:09.166Z [INFO]  server-85918948-25ba-f42c-1faf-0cd11aa4f6fa: entering follower state: follower="Node at 85918948-25ba-f42c-1faf-0cd11aa4f6fa [Follower]" leader-address= leader-id=
2026-09-21T15:25:09.773Z [WARN]  server-9e408929-bbf3-4f91-35de-c07b11888721: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:09.773Z [INFO]  server-9e408929-bbf3-4f91-35de-c07b11888721: entering candidate state: node="Node at 9e408929-bbf3-4f91-35de-c07b11888721 [Candidate]" term=2
2026-09-21T15:25:09.773Z [INFO]  server-9e408929-bbf3-4f91-35de-c07b11888721: election won: term=2 tally=2
2026-09-21T15:25:09.773Z [INFO]  server-9e408929-bbf3-4f91-35de-c07b11888721: entering leader state: leader="Node at 9e408929-bbf3-4f91-35de-c07b11888721 [Leader]"
2026-09-21T15:25:09.773Z [INFO]  server-9e408929-bbf3-4f91-35de-c07b11888721: added peer, starting replication: peer=server-6786e347-f362-dd02-15ea-998139c6507e
2026-09-21T15:25:09.773Z [INFO]  server-9e408929-bbf3-4f91-35de-c07b11888721: added peer, starting replication: peer=server-85918948-25ba-f42c-1faf-0cd11aa4f6fa
2026-09-21T15:25:09.773Z [INFO]  server-9e408929-bbf3-4f91-35de-c07b11888721: pipelining replication: peer="{Voter server-85918948-25ba-f42c-1faf-0cd11aa4f6fa 85918948-25ba-f42c-1faf-0cd11aa4f6fa}"
2026-09-21T15:25:09.773Z [INFO]  server-9e408929-bbf3-4f91-35de-c07b11888721: pipelining replication: peer="{Voter server-6786e347-f362-dd02-15ea-998139c6507e 6786e347-f362-dd02-15ea-998139c6507e}"
2026-09-21T15:25:10.779Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:09.773531628 +0000 UTC m=+61.714701801 (1 nodes), 607.20708ms from start of poll, 607.62468ms from cluster start. Timeout at 2026-09-21 15:25:10.778534508 +0000 UTC m=+62.719704721, 1.00500292s after stability
2026-09-21T15:25:10.780Z [INFO]  server-9e408929-bbf3-4f91-35de-c07b11888721: starting snapshot up to: index=12
2026-09-21T15:25:10.781Z [INFO]  creating new snapshot: path=/tmp/raft3681524316/snapshots/2-12-1790004310780.tmp
2026-09-21T15:25:10.781Z [INFO]  server-9e408929-bbf3-4f91-35de-c07b11888721: snapshot complete up to: index=12
2026-09-21T15:25:10.782Z [INFO]  server-8f299d1b-8ed4-4bb7-e031-ce0e35c2278e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f299d1b-8ed4-4bb7-e031-ce0e35c2278e Address:8f299d1b-8ed4-4bb7-e031-ce0e35c2278e} {Suffrage:Voter ID:server-c5b739aa-b7ac-d17b-997a-120480ff5207 Address:c5b739aa-b7ac-d17b-997a-120480ff5207} {Suffrage:Voter ID:server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d Address:dc1fe33a-ede1-1ffe-3ac3-e4531746b42d}]"
2026-09-21T15:25:10.783Z [INFO]  server-c5b739aa-b7ac-d17b-997a-120480ff5207: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f299d1b-8ed4-4bb7-e031-ce0e35c2278e Address:8f299d1b-8ed4-4bb7-e031-ce0e35c2278e} {Suffrage:Voter ID:server-c5b739aa-b7ac-d17b-997a-120480ff5207 Address:c5b739aa-b7ac-d17b-997a-120480ff5207} {Suffrage:Voter ID:server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d Address:dc1fe33a-ede1-1ffe-3ac3-e4531746b42d}]"
2026-09-21T15:25:10.783Z [INFO]  server-8f299d1b-8ed4-4bb7-e031-ce0e35c2278e: entering follower state: follower="Node at 8f299d1b-8ed4-4bb7-e031-ce0e35c2278e [Follower]" leader-address= leader-id=
2026-09-21T15:25:10.783Z [INFO]  server-c5b739aa-b7ac-d17b-997a-120480ff5207: entering follower state: follower="Node at c5b739aa-b7ac-d17b-997a-120480ff5207 [Follower]" leader-address= leader-id=
2026-09-21T15:25:10.783Z [INFO]  server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f299d1b-8ed4-4bb7-e031-ce0e35c2278e Address:8f299d1b-8ed4-4bb7-e031-ce0e35c2278e} {Suffrage:Voter ID:server-c5b739aa-b7ac-d17b-997a-120480ff5207 Address:c5b739aa-b7ac-d17b-997a-120480ff5207} {Suffrage:Voter ID:server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d Address:dc1fe33a-ede1-1ffe-3ac3-e4531746b42d}]"
2026-09-21T15:25:10.783Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:10.783Z [INFO]  server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: entering follower state: follower="Node at dc1fe33a-ede1-1ffe-3ac3-e4531746b42d [Follower]" leader-address= leader-id=
2026-09-21T15:25:11.376Z [WARN]  server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:11.376Z [INFO]  server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: entering candidate state: node="Node at dc1fe33a-ede1-1ffe-3ac3-e4531746b42d [Candidate]" term=2
2026-09-21T15:25:11.376Z [INFO]  server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: election won: term=2 tally=2
2026-09-21T15:25:11.376Z [INFO]  server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: entering leader state: leader="Node at dc1fe33a-ede1-1ffe-3ac3-e4531746b42d [Leader]"
2026-09-21T15:25:11.376Z [INFO]  server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: added peer, starting replication: peer=server-8f299d1b-8ed4-4bb7-e031-ce0e35c2278e
2026-09-21T15:25:11.376Z [INFO]  server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: added peer, starting replication: peer=server-c5b739aa-b7ac-d17b-997a-120480ff5207
2026-09-21T15:25:11.376Z [INFO]  server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: pipelining replication: peer="{Voter server-c5b739aa-b7ac-d17b-997a-120480ff5207 c5b739aa-b7ac-d17b-997a-120480ff5207}"
2026-09-21T15:25:11.376Z [INFO]  server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: pipelining replication: peer="{Voter server-8f299d1b-8ed4-4bb7-e031-ce0e35c2278e 8f299d1b-8ed4-4bb7-e031-ce0e35c2278e}"
2026-09-21T15:25:12.382Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:11.376388828 +0000 UTC m=+63.317559001 (1 nodes), 593.1388ms from start of poll, 593.52824ms from cluster start. Timeout at 2026-09-21 15:25:12.381391628 +0000 UTC m=+64.322561801, 1.0050028s after stability
2026-09-21T15:25:12.382Z [INFO]  creating new snapshot: path=/tmp/raft3116009703/snapshots/2-113-1790004312382.tmp
2026-09-21T15:25:12.382Z [INFO]  server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: copied to local snapshot: bytes=71
2026-09-21T15:25:12.383Z [INFO]  server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: snapshot restore progress: id=2-113-1790004312382 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:12.383Z [INFO]  server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: restored user snapshot: index=113
2026-09-21T15:25:12.383Z [ERROR] server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: failed to get log: index=3 error="log not found"
2026-09-21T15:25:12.383Z [INFO]  server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: aborting pipeline replication: peer="{Voter server-8f299d1b-8ed4-4bb7-e031-ce0e35c2278e 8f299d1b-8ed4-4bb7-e031-ce0e35c2278e}"
2026-09-21T15:25:12.383Z [ERROR] server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: failed to get log: index=3 error="log not found"
2026-09-21T15:25:12.383Z [INFO]  server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: aborting pipeline replication: peer="{Voter server-c5b739aa-b7ac-d17b-997a-120480ff5207 c5b739aa-b7ac-d17b-997a-120480ff5207}"
2026-09-21T15:25:12.391Z [ERROR] server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: failed to get log: index=3 error="log not found"
2026-09-21T15:25:12.392Z [INFO]  creating new snapshot: path=/tmp/raft320890455/snapshots/2-113-1790004312392.tmp
2026-09-21T15:25:12.392Z [INFO]  server-8f299d1b-8ed4-4bb7-e031-ce0e35c2278e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:12.392Z [ERROR] server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: failed to get log: index=3 error="log not found"
2026-09-21T15:25:12.392Z [INFO]  server-8f299d1b-8ed4-4bb7-e031-ce0e35c2278e: copied to local snapshot: bytes=71
2026-09-21T15:25:12.392Z [INFO]  creating new snapshot: path=/tmp/raft4071998950/snapshots/2-113-1790004312392.tmp
2026-09-21T15:25:12.392Z [INFO]  server-8f299d1b-8ed4-4bb7-e031-ce0e35c2278e: snapshot restore progress: id=2-113-1790004312392 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:12.393Z [INFO]  server-8f299d1b-8ed4-4bb7-e031-ce0e35c2278e: Installed remote snapshot
2026-09-21T15:25:12.393Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:12.393Z [INFO]  server-c5b739aa-b7ac-d17b-997a-120480ff5207: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:12.393Z [INFO]  server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: pipelining replication: peer="{Voter server-8f299d1b-8ed4-4bb7-e031-ce0e35c2278e 8f299d1b-8ed4-4bb7-e031-ce0e35c2278e}"
2026-09-21T15:25:12.393Z [INFO]  server-c5b739aa-b7ac-d17b-997a-120480ff5207: copied to local snapshot: bytes=71
2026-09-21T15:25:12.393Z [INFO]  server-c5b739aa-b7ac-d17b-997a-120480ff5207: snapshot restore progress: id=2-113-1790004312392 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:12.393Z [INFO]  server-c5b739aa-b7ac-d17b-997a-120480ff5207: Installed remote snapshot
2026-09-21T15:25:12.393Z [INFO]  server-dc1fe33a-ede1-1ffe-3ac3-e4531746b42d: pipelining replication: peer="{Voter server-c5b739aa-b7ac-d17b-997a-120480ff5207 c5b739aa-b7ac-d17b-997a-120480ff5207}"
2026-09-21T15:25:13.399Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:12.393221748 +0000 UTC m=+64.334391921 (2 nodes), 800ns from start of poll, 1.61036116s from cluster start. Timeout at 2026-09-21 15:25:13.398473188 +0000 UTC m=+65.339643321, 1.0052514s after stability
2026-09-21T15:25:13.414Z [INFO]  server-9e408929-bbf3-4f91-35de-c07b11888721: aborting pipeline replication: peer="{Voter server-85918948-25ba-f42c-1faf-0cd11aa4f6fa 85918948-25ba-f42c-1faf-0cd11aa4f6fa}"
2026-09-21T15:25:13.414Z [INFO]  server-9e408929-bbf3-4f91-35de-c07b11888721: aborting pipeline replication: peer="{Voter server-6786e347-f362-dd02-15ea-998139c6507e 6786e347-f362-dd02-15ea-998139c6507e}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true
2026-09-21T15:25:13.415Z [INFO]  server-5e7b2943-12da-02e5-6738-955548d522f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e7b2943-12da-02e5-6738-955548d522f7 Address:5e7b2943-12da-02e5-6738-955548d522f7} {Suffrage:Voter ID:server-2442b06b-544f-70ad-d222-cfc9504afd9a Address:2442b06b-544f-70ad-d222-cfc9504afd9a} {Suffrage:Voter ID:server-e3a9e144-0611-bb9f-c9d0-61f47bd8fb0f Address:e3a9e144-0611-bb9f-c9d0-61f47bd8fb0f}]"
2026-09-21T15:25:13.415Z [INFO]  server-2442b06b-544f-70ad-d222-cfc9504afd9a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e7b2943-12da-02e5-6738-955548d522f7 Address:5e7b2943-12da-02e5-6738-955548d522f7} {Suffrage:Voter ID:server-2442b06b-544f-70ad-d222-cfc9504afd9a Address:2442b06b-544f-70ad-d222-cfc9504afd9a} {Suffrage:Voter ID:server-e3a9e144-0611-bb9f-c9d0-61f47bd8fb0f Address:e3a9e144-0611-bb9f-c9d0-61f47bd8fb0f}]"
2026-09-21T15:25:13.415Z [INFO]  server-5e7b2943-12da-02e5-6738-955548d522f7: entering follower state: follower="Node at 5e7b2943-12da-02e5-6738-955548d522f7 [Follower]" leader-address= leader-id=
2026-09-21T15:25:13.416Z [INFO]  server-e3a9e144-0611-bb9f-c9d0-61f47bd8fb0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e7b2943-12da-02e5-6738-955548d522f7 Address:5e7b2943-12da-02e5-6738-955548d522f7} {Suffrage:Voter ID:server-2442b06b-544f-70ad-d222-cfc9504afd9a Address:2442b06b-544f-70ad-d222-cfc9504afd9a} {Suffrage:Voter ID:server-e3a9e144-0611-bb9f-c9d0-61f47bd8fb0f Address:e3a9e144-0611-bb9f-c9d0-61f47bd8fb0f}]"
2026-09-21T15:25:13.416Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:13.416Z [INFO]  server-2442b06b-544f-70ad-d222-cfc9504afd9a: entering follower state: follower="Node at 2442b06b-544f-70ad-d222-cfc9504afd9a [Follower]" leader-address= leader-id=
2026-09-21T15:25:13.416Z [INFO]  server-e3a9e144-0611-bb9f-c9d0-61f47bd8fb0f: entering follower state: follower="Node at e3a9e144-0611-bb9f-c9d0-61f47bd8fb0f [Follower]" leader-address= leader-id=
2026-09-21T15:25:14.027Z [WARN]  server-2442b06b-544f-70ad-d222-cfc9504afd9a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:14.027Z [INFO]  server-2442b06b-544f-70ad-d222-cfc9504afd9a: entering candidate state: node="Node at 2442b06b-544f-70ad-d222-cfc9504afd9a [Candidate]" term=2
2026-09-21T15:25:14.027Z [INFO]  server-2442b06b-544f-70ad-d222-cfc9504afd9a: election won: term=2 tally=2
2026-09-21T15:25:14.028Z [INFO]  server-2442b06b-544f-70ad-d222-cfc9504afd9a: entering leader state: leader="Node at 2442b06b-544f-70ad-d222-cfc9504afd9a [Leader]"
2026-09-21T15:25:14.028Z [INFO]  server-2442b06b-544f-70ad-d222-cfc9504afd9a: added peer, starting replication: peer=server-5e7b2943-12da-02e5-6738-955548d522f7
2026-09-21T15:25:14.028Z [INFO]  server-2442b06b-544f-70ad-d222-cfc9504afd9a: added peer, starting replication: peer=server-e3a9e144-0611-bb9f-c9d0-61f47bd8fb0f
2026-09-21T15:25:14.028Z [INFO]  server-2442b06b-544f-70ad-d222-cfc9504afd9a: pipelining replication: peer="{Voter server-5e7b2943-12da-02e5-6738-955548d522f7 5e7b2943-12da-02e5-6738-955548d522f7}"
2026-09-21T15:25:14.029Z [INFO]  server-2442b06b-544f-70ad-d222-cfc9504afd9a: pipelining replication: peer="{Voter server-e3a9e144-0611-bb9f-c9d0-61f47bd8fb0f e3a9e144-0611-bb9f-c9d0-61f47bd8fb0f}"
2026-09-21T15:25:15.034Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:14.028528708 +0000 UTC m=+65.969698881 (1 nodes), 612.41124ms from start of poll, 612.82352ms from cluster start. Timeout at 2026-09-21 15:25:15.033537548 +0000 UTC m=+66.974707721, 1.00500884s after stability
2026-09-21T15:25:15.035Z [INFO]  server-2442b06b-544f-70ad-d222-cfc9504afd9a: starting snapshot up to: index=12
2026-09-21T15:25:15.035Z [INFO]  creating new snapshot: path=/tmp/raft3634329852/snapshots/2-12-1790004315035.tmp
2026-09-21T15:25:15.035Z [INFO]  server-2442b06b-544f-70ad-d222-cfc9504afd9a: snapshot complete up to: index=12
2026-09-21T15:25:15.037Z [INFO]  server-b3a44825-7a53-1890-571e-3376dbbe4f08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3a44825-7a53-1890-571e-3376dbbe4f08 Address:b3a44825-7a53-1890-571e-3376dbbe4f08} {Suffrage:Voter ID:server-f87aa301-d2df-9ece-4c3e-09661a1e49a3 Address:f87aa301-d2df-9ece-4c3e-09661a1e49a3} {Suffrage:Voter ID:server-fe91541b-ad19-b557-1adf-9d7997d556b4 Address:fe91541b-ad19-b557-1adf-9d7997d556b4}]"
2026-09-21T15:25:15.037Z [INFO]  server-f87aa301-d2df-9ece-4c3e-09661a1e49a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3a44825-7a53-1890-571e-3376dbbe4f08 Address:b3a44825-7a53-1890-571e-3376dbbe4f08} {Suffrage:Voter ID:server-f87aa301-d2df-9ece-4c3e-09661a1e49a3 Address:f87aa301-d2df-9ece-4c3e-09661a1e49a3} {Suffrage:Voter ID:server-fe91541b-ad19-b557-1adf-9d7997d556b4 Address:fe91541b-ad19-b557-1adf-9d7997d556b4}]"
2026-09-21T15:25:15.037Z [INFO]  server-b3a44825-7a53-1890-571e-3376dbbe4f08: entering follower state: follower="Node at b3a44825-7a53-1890-571e-3376dbbe4f08 [Follower]" leader-address= leader-id=
2026-09-21T15:25:15.037Z [INFO]  server-f87aa301-d2df-9ece-4c3e-09661a1e49a3: entering follower state: follower="Node at f87aa301-d2df-9ece-4c3e-09661a1e49a3 [Follower]" leader-address= leader-id=
2026-09-21T15:25:15.037Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3a44825-7a53-1890-571e-3376dbbe4f08 Address:b3a44825-7a53-1890-571e-3376dbbe4f08} {Suffrage:Voter ID:server-f87aa301-d2df-9ece-4c3e-09661a1e49a3 Address:f87aa301-d2df-9ece-4c3e-09661a1e49a3} {Suffrage:Voter ID:server-fe91541b-ad19-b557-1adf-9d7997d556b4 Address:fe91541b-ad19-b557-1adf-9d7997d556b4}]"
2026-09-21T15:25:15.037Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:15.037Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: entering follower state: follower="Node at fe91541b-ad19-b557-1adf-9d7997d556b4 [Follower]" leader-address= leader-id=
2026-09-21T15:25:15.806Z [WARN]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:15.806Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: entering candidate state: node="Node at fe91541b-ad19-b557-1adf-9d7997d556b4 [Candidate]" term=2
2026-09-21T15:25:15.806Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: election won: term=2 tally=2
2026-09-21T15:25:15.806Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: entering leader state: leader="Node at fe91541b-ad19-b557-1adf-9d7997d556b4 [Leader]"
2026-09-21T15:25:15.806Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: added peer, starting replication: peer=server-b3a44825-7a53-1890-571e-3376dbbe4f08
2026-09-21T15:25:15.806Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: added peer, starting replication: peer=server-f87aa301-d2df-9ece-4c3e-09661a1e49a3
2026-09-21T15:25:15.807Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: pipelining replication: peer="{Voter server-f87aa301-d2df-9ece-4c3e-09661a1e49a3 f87aa301-d2df-9ece-4c3e-09661a1e49a3}"
2026-09-21T15:25:15.807Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: pipelining replication: peer="{Voter server-b3a44825-7a53-1890-571e-3376dbbe4f08 b3a44825-7a53-1890-571e-3376dbbe4f08}"
2026-09-21T15:25:16.812Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:15.806950708 +0000 UTC m=+67.748120881 (1 nodes), 769.57104ms from start of poll, 770.00908ms from cluster start. Timeout at 2026-09-21 15:25:16.811954948 +0000 UTC m=+68.753125121, 1.00500424s after stability
2026-09-21T15:25:16.813Z [INFO]  creating new snapshot: path=/tmp/raft2377666604/snapshots/2-113-1790004316813.tmp
2026-09-21T15:25:16.813Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: copied to local snapshot: bytes=71
2026-09-21T15:25:16.813Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: snapshot restore progress: id=2-113-1790004316813 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:16.814Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: removing all old logs from log store
2026-09-21T15:25:16.814Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: compacting logs: from=1 to=2
2026-09-21T15:25:16.814Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: restored user snapshot: index=113
2026-09-21T15:25:16.814Z [ERROR] server-fe91541b-ad19-b557-1adf-9d7997d556b4: failed to get log: index=2 error="log not found"
2026-09-21T15:25:16.814Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: aborting pipeline replication: peer="{Voter server-f87aa301-d2df-9ece-4c3e-09661a1e49a3 f87aa301-d2df-9ece-4c3e-09661a1e49a3}"
2026-09-21T15:25:16.814Z [ERROR] server-fe91541b-ad19-b557-1adf-9d7997d556b4: failed to get log: index=2 error="log not found"
2026-09-21T15:25:16.814Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: aborting pipeline replication: peer="{Voter server-b3a44825-7a53-1890-571e-3376dbbe4f08 b3a44825-7a53-1890-571e-3376dbbe4f08}"
2026-09-21T15:25:16.822Z [ERROR] server-fe91541b-ad19-b557-1adf-9d7997d556b4: failed to get log: index=2 error="log not found"
2026-09-21T15:25:16.823Z [INFO]  creating new snapshot: path=/tmp/raft3452184736/snapshots/2-113-1790004316823.tmp
2026-09-21T15:25:16.823Z [ERROR] server-fe91541b-ad19-b557-1adf-9d7997d556b4: failed to get log: index=2 error="log not found"
2026-09-21T15:25:16.823Z [INFO]  server-f87aa301-d2df-9ece-4c3e-09661a1e49a3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:16.823Z [INFO]  server-f87aa301-d2df-9ece-4c3e-09661a1e49a3: copied to local snapshot: bytes=71
2026-09-21T15:25:16.824Z [INFO]  creating new snapshot: path=/tmp/raft4042538594/snapshots/2-113-1790004316824.tmp
2026-09-21T15:25:16.824Z [INFO]  server-f87aa301-d2df-9ece-4c3e-09661a1e49a3: snapshot restore progress: id=2-113-1790004316823 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:16.824Z [INFO]  server-f87aa301-d2df-9ece-4c3e-09661a1e49a3: removing all old logs from log store
2026-09-21T15:25:16.824Z [INFO]  server-f87aa301-d2df-9ece-4c3e-09661a1e49a3: compacting logs: from=1 to=2
2026-09-21T15:25:16.824Z [INFO]  server-f87aa301-d2df-9ece-4c3e-09661a1e49a3: Installed remote snapshot
2026-09-21T15:25:16.824Z [INFO]  server-b3a44825-7a53-1890-571e-3376dbbe4f08: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:16.824Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: pipelining replication: peer="{Voter server-f87aa301-d2df-9ece-4c3e-09661a1e49a3 f87aa301-d2df-9ece-4c3e-09661a1e49a3}"
2026-09-21T15:25:16.824Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:16.824Z [INFO]  server-b3a44825-7a53-1890-571e-3376dbbe4f08: copied to local snapshot: bytes=71
2026-09-21T15:25:16.824Z [INFO]  server-b3a44825-7a53-1890-571e-3376dbbe4f08: snapshot restore progress: id=2-113-1790004316824 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:16.824Z [INFO]  server-b3a44825-7a53-1890-571e-3376dbbe4f08: removing all old logs from log store
2026-09-21T15:25:16.824Z [INFO]  server-b3a44825-7a53-1890-571e-3376dbbe4f08: compacting logs: from=1 to=2
2026-09-21T15:25:16.824Z [INFO]  server-b3a44825-7a53-1890-571e-3376dbbe4f08: Installed remote snapshot
2026-09-21T15:25:16.825Z [INFO]  server-fe91541b-ad19-b557-1adf-9d7997d556b4: pipelining replication: peer="{Voter server-b3a44825-7a53-1890-571e-3376dbbe4f08 b3a44825-7a53-1890-571e-3376dbbe4f08}"
2026-09-21T15:25:17.830Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:16.824491028 +0000 UTC m=+68.765661201 (2 nodes), 840ns from start of poll, 1.7875494s from cluster start. Timeout at 2026-09-21 15:25:17.829514708 +0000 UTC m=+69.770684921, 1.00502372s after stability
2026-09-21T15:25:17.843Z [INFO]  server-2442b06b-544f-70ad-d222-cfc9504afd9a: aborting pipeline replication: peer="{Voter server-e3a9e144-0611-bb9f-c9d0-61f47bd8fb0f e3a9e144-0611-bb9f-c9d0-61f47bd8fb0f}"
2026-09-21T15:25:17.843Z [INFO]  server-2442b06b-544f-70ad-d222-cfc9504afd9a: aborting pipeline replication: peer="{Voter server-5e7b2943-12da-02e5-6738-955548d522f7 5e7b2943-12da-02e5-6738-955548d522f7}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false
2026-09-21T15:25:17.844Z [INFO]  server-07b62853-6a30-1aeb-47cf-f3cb4ca68aac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07b62853-6a30-1aeb-47cf-f3cb4ca68aac Address:07b62853-6a30-1aeb-47cf-f3cb4ca68aac} {Suffrage:Voter ID:server-d400dd19-b68e-f675-deaa-66a99f6f7b96 Address:d400dd19-b68e-f675-deaa-66a99f6f7b96} {Suffrage:Voter ID:server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f Address:d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f}]"
2026-09-21T15:25:17.844Z [INFO]  server-d400dd19-b68e-f675-deaa-66a99f6f7b96: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07b62853-6a30-1aeb-47cf-f3cb4ca68aac Address:07b62853-6a30-1aeb-47cf-f3cb4ca68aac} {Suffrage:Voter ID:server-d400dd19-b68e-f675-deaa-66a99f6f7b96 Address:d400dd19-b68e-f675-deaa-66a99f6f7b96} {Suffrage:Voter ID:server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f Address:d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f}]"
2026-09-21T15:25:17.844Z [INFO]  server-07b62853-6a30-1aeb-47cf-f3cb4ca68aac: entering follower state: follower="Node at 07b62853-6a30-1aeb-47cf-f3cb4ca68aac [Follower]" leader-address= leader-id=
2026-09-21T15:25:17.844Z [INFO]  server-d400dd19-b68e-f675-deaa-66a99f6f7b96: entering follower state: follower="Node at d400dd19-b68e-f675-deaa-66a99f6f7b96 [Follower]" leader-address= leader-id=
2026-09-21T15:25:17.844Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07b62853-6a30-1aeb-47cf-f3cb4ca68aac Address:07b62853-6a30-1aeb-47cf-f3cb4ca68aac} {Suffrage:Voter ID:server-d400dd19-b68e-f675-deaa-66a99f6f7b96 Address:d400dd19-b68e-f675-deaa-66a99f6f7b96} {Suffrage:Voter ID:server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f Address:d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f}]"
2026-09-21T15:25:17.844Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:17.844Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: entering follower state: follower="Node at d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f [Follower]" leader-address= leader-id=
2026-09-21T15:25:18.499Z [WARN]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:18.499Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: entering candidate state: node="Node at d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f [Candidate]" term=2
2026-09-21T15:25:18.500Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: election won: term=2 tally=2
2026-09-21T15:25:18.500Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: entering leader state: leader="Node at d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f [Leader]"
2026-09-21T15:25:18.500Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: added peer, starting replication: peer=server-07b62853-6a30-1aeb-47cf-f3cb4ca68aac
2026-09-21T15:25:18.500Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: added peer, starting replication: peer=server-d400dd19-b68e-f675-deaa-66a99f6f7b96
2026-09-21T15:25:18.500Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: pipelining replication: peer="{Voter server-07b62853-6a30-1aeb-47cf-f3cb4ca68aac 07b62853-6a30-1aeb-47cf-f3cb4ca68aac}"
2026-09-21T15:25:18.500Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: pipelining replication: peer="{Voter server-d400dd19-b68e-f675-deaa-66a99f6f7b96 d400dd19-b68e-f675-deaa-66a99f6f7b96}"
2026-09-21T15:25:19.505Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:18.500143108 +0000 UTC m=+70.441313241 (1 nodes), 655.6944ms from start of poll, 656.142ms from cluster start. Timeout at 2026-09-21 15:25:19.505145188 +0000 UTC m=+71.446315361, 1.00500212s after stability
2026-09-21T15:25:19.506Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: starting snapshot up to: index=12
2026-09-21T15:25:19.506Z [INFO]  creating new snapshot: path=/tmp/raft80416305/snapshots/2-12-1790004319506.tmp
2026-09-21T15:25:19.506Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: snapshot complete up to: index=12
2026-09-21T15:25:19.507Z [INFO]  creating new snapshot: path=/tmp/raft80416305/snapshots/2-1013-1790004319507.tmp
2026-09-21T15:25:19.508Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: copied to local snapshot: bytes=71
2026-09-21T15:25:19.508Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: snapshot restore progress: id=2-1013-1790004319507 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:19.508Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: restored user snapshot: index=1013
2026-09-21T15:25:19.508Z [ERROR] server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: failed to get log: index=23 error="log not found"
2026-09-21T15:25:19.508Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: aborting pipeline replication: peer="{Voter server-d400dd19-b68e-f675-deaa-66a99f6f7b96 d400dd19-b68e-f675-deaa-66a99f6f7b96}"
2026-09-21T15:25:19.508Z [ERROR] server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: failed to get log: index=23 error="log not found"
2026-09-21T15:25:19.508Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: aborting pipeline replication: peer="{Voter server-07b62853-6a30-1aeb-47cf-f3cb4ca68aac 07b62853-6a30-1aeb-47cf-f3cb4ca68aac}"
2026-09-21T15:25:19.517Z [ERROR] server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: failed to get log: index=23 error="log not found"
2026-09-21T15:25:19.518Z [INFO]  creating new snapshot: path=/tmp/raft1985710283/snapshots/2-1013-1790004319518.tmp
2026-09-21T15:25:19.518Z [INFO]  server-d400dd19-b68e-f675-deaa-66a99f6f7b96: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:19.518Z [INFO]  server-d400dd19-b68e-f675-deaa-66a99f6f7b96: copied to local snapshot: bytes=71
2026-09-21T15:25:19.518Z [ERROR] server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: failed to get log: index=23 error="log not found"
2026-09-21T15:25:19.518Z [INFO]  server-d400dd19-b68e-f675-deaa-66a99f6f7b96: snapshot restore progress: id=2-1013-1790004319518 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:19.519Z [INFO]  server-d400dd19-b68e-f675-deaa-66a99f6f7b96: Installed remote snapshot
2026-09-21T15:25:19.519Z [WARN]  server-d400dd19-b68e-f675-deaa-66a99f6f7b96: skipping application of old log: index=22
2026-09-21T15:25:19.519Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: pipelining replication: peer="{Voter server-d400dd19-b68e-f675-deaa-66a99f6f7b96 d400dd19-b68e-f675-deaa-66a99f6f7b96}"
2026-09-21T15:25:19.519Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:19.519Z [INFO]  creating new snapshot: path=/tmp/raft3717103959/snapshots/2-1013-1790004319519.tmp
2026-09-21T15:25:19.519Z [INFO]  server-07b62853-6a30-1aeb-47cf-f3cb4ca68aac: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:19.519Z [INFO]  server-07b62853-6a30-1aeb-47cf-f3cb4ca68aac: copied to local snapshot: bytes=71
2026-09-21T15:25:19.520Z [INFO]  server-07b62853-6a30-1aeb-47cf-f3cb4ca68aac: snapshot restore progress: id=2-1013-1790004319519 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:19.520Z [INFO]  server-07b62853-6a30-1aeb-47cf-f3cb4ca68aac: Installed remote snapshot
2026-09-21T15:25:19.520Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: pipelining replication: peer="{Voter server-07b62853-6a30-1aeb-47cf-f3cb4ca68aac 07b62853-6a30-1aeb-47cf-f3cb4ca68aac}"
2026-09-21T15:25:20.524Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:19.519238308 +0000 UTC m=+71.460408521 (2 nodes), 1.04µs from start of poll, 1.67523728s from cluster start. Timeout at 2026-09-21 15:25:20.524261028 +0000 UTC m=+72.465431241, 1.00502272s after stability
2026-09-21T15:25:20.537Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: aborting pipeline replication: peer="{Voter server-d400dd19-b68e-f675-deaa-66a99f6f7b96 d400dd19-b68e-f675-deaa-66a99f6f7b96}"
2026-09-21T15:25:20.537Z [INFO]  server-d92a2cfc-baf6-c1a5-2efd-2c0fc82c722f: aborting pipeline replication: peer="{Voter server-07b62853-6a30-1aeb-47cf-f3cb4ca68aac 07b62853-6a30-1aeb-47cf-f3cb4ca68aac}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false
2026-09-21T15:25:20.538Z [INFO]  server-c7203980-09fb-b44a-6b29-b6f04998d4d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7203980-09fb-b44a-6b29-b6f04998d4d6 Address:c7203980-09fb-b44a-6b29-b6f04998d4d6} {Suffrage:Voter ID:server-1d822775-3d3c-1b7c-4936-8788a6d64e50 Address:1d822775-3d3c-1b7c-4936-8788a6d64e50} {Suffrage:Voter ID:server-1de0bfb3-54c5-5612-7845-d305304e8c48 Address:1de0bfb3-54c5-5612-7845-d305304e8c48}]"
2026-09-21T15:25:20.538Z [INFO]  server-1d822775-3d3c-1b7c-4936-8788a6d64e50: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7203980-09fb-b44a-6b29-b6f04998d4d6 Address:c7203980-09fb-b44a-6b29-b6f04998d4d6} {Suffrage:Voter ID:server-1d822775-3d3c-1b7c-4936-8788a6d64e50 Address:1d822775-3d3c-1b7c-4936-8788a6d64e50} {Suffrage:Voter ID:server-1de0bfb3-54c5-5612-7845-d305304e8c48 Address:1de0bfb3-54c5-5612-7845-d305304e8c48}]"
2026-09-21T15:25:20.538Z [INFO]  server-c7203980-09fb-b44a-6b29-b6f04998d4d6: entering follower state: follower="Node at c7203980-09fb-b44a-6b29-b6f04998d4d6 [Follower]" leader-address= leader-id=
2026-09-21T15:25:20.538Z [INFO]  server-1d822775-3d3c-1b7c-4936-8788a6d64e50: entering follower state: follower="Node at 1d822775-3d3c-1b7c-4936-8788a6d64e50 [Follower]" leader-address= leader-id=
2026-09-21T15:25:20.538Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7203980-09fb-b44a-6b29-b6f04998d4d6 Address:c7203980-09fb-b44a-6b29-b6f04998d4d6} {Suffrage:Voter ID:server-1d822775-3d3c-1b7c-4936-8788a6d64e50 Address:1d822775-3d3c-1b7c-4936-8788a6d64e50} {Suffrage:Voter ID:server-1de0bfb3-54c5-5612-7845-d305304e8c48 Address:1de0bfb3-54c5-5612-7845-d305304e8c48}]"
2026-09-21T15:25:20.538Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:20.538Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: entering follower state: follower="Node at 1de0bfb3-54c5-5612-7845-d305304e8c48 [Follower]" leader-address= leader-id=
2026-09-21T15:25:21.354Z [WARN]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:21.354Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: entering candidate state: node="Node at 1de0bfb3-54c5-5612-7845-d305304e8c48 [Candidate]" term=2
2026-09-21T15:25:21.354Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: election won: term=2 tally=2
2026-09-21T15:25:21.355Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: entering leader state: leader="Node at 1de0bfb3-54c5-5612-7845-d305304e8c48 [Leader]"
2026-09-21T15:25:21.355Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: added peer, starting replication: peer=server-c7203980-09fb-b44a-6b29-b6f04998d4d6
2026-09-21T15:25:21.355Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: added peer, starting replication: peer=server-1d822775-3d3c-1b7c-4936-8788a6d64e50
2026-09-21T15:25:21.355Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: pipelining replication: peer="{Voter server-1d822775-3d3c-1b7c-4936-8788a6d64e50 1d822775-3d3c-1b7c-4936-8788a6d64e50}"
2026-09-21T15:25:21.355Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: pipelining replication: peer="{Voter server-c7203980-09fb-b44a-6b29-b6f04998d4d6 c7203980-09fb-b44a-6b29-b6f04998d4d6}"
2026-09-21T15:25:22.361Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:21.355088548 +0000 UTC m=+73.296258721 (1 nodes), 816.54284ms from start of poll, 816.97816ms from cluster start. Timeout at 2026-09-21 15:25:22.360092308 +0000 UTC m=+74.301262481, 1.00500376s after stability
2026-09-21T15:25:22.361Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: starting snapshot up to: index=12
2026-09-21T15:25:22.361Z [INFO]  creating new snapshot: path=/tmp/raft3162224546/snapshots/2-12-1790004322361.tmp
2026-09-21T15:25:22.362Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: snapshot complete up to: index=12
2026-09-21T15:25:22.362Z [INFO]  creating new snapshot: path=/tmp/raft3162224546/snapshots/2-1013-1790004322362.tmp
2026-09-21T15:25:22.363Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: copied to local snapshot: bytes=71
2026-09-21T15:25:22.363Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: snapshot restore progress: id=2-1013-1790004322362 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:22.363Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: removing all old logs from log store
2026-09-21T15:25:22.363Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: compacting logs: from=1 to=22
2026-09-21T15:25:22.363Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: restored user snapshot: index=1013
2026-09-21T15:25:22.363Z [ERROR] server-1de0bfb3-54c5-5612-7845-d305304e8c48: failed to get log: index=22 error="log not found"
2026-09-21T15:25:22.363Z [ERROR] server-1de0bfb3-54c5-5612-7845-d305304e8c48: failed to get log: index=22 error="log not found"
2026-09-21T15:25:22.364Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: aborting pipeline replication: peer="{Voter server-1d822775-3d3c-1b7c-4936-8788a6d64e50 1d822775-3d3c-1b7c-4936-8788a6d64e50}"
2026-09-21T15:25:22.364Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: aborting pipeline replication: peer="{Voter server-c7203980-09fb-b44a-6b29-b6f04998d4d6 c7203980-09fb-b44a-6b29-b6f04998d4d6}"
2026-09-21T15:25:22.371Z [ERROR] server-1de0bfb3-54c5-5612-7845-d305304e8c48: failed to get log: index=22 error="log not found"
2026-09-21T15:25:22.372Z [ERROR] server-1de0bfb3-54c5-5612-7845-d305304e8c48: failed to get log: index=22 error="log not found"
2026-09-21T15:25:22.372Z [INFO]  creating new snapshot: path=/tmp/raft1877350660/snapshots/2-1013-1790004322372.tmp
2026-09-21T15:25:22.373Z [INFO]  creating new snapshot: path=/tmp/raft2430399781/snapshots/2-1013-1790004322373.tmp
2026-09-21T15:25:22.373Z [INFO]  server-1d822775-3d3c-1b7c-4936-8788a6d64e50: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:22.373Z [INFO]  server-c7203980-09fb-b44a-6b29-b6f04998d4d6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:22.373Z [INFO]  server-1d822775-3d3c-1b7c-4936-8788a6d64e50: copied to local snapshot: bytes=71
2026-09-21T15:25:22.374Z [INFO]  server-1d822775-3d3c-1b7c-4936-8788a6d64e50: snapshot restore progress: id=2-1013-1790004322372 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:22.374Z [INFO]  server-1d822775-3d3c-1b7c-4936-8788a6d64e50: removing all old logs from log store
2026-09-21T15:25:22.374Z [INFO]  server-1d822775-3d3c-1b7c-4936-8788a6d64e50: compacting logs: from=1 to=22
2026-09-21T15:25:22.374Z [INFO]  server-c7203980-09fb-b44a-6b29-b6f04998d4d6: copied to local snapshot: bytes=71
2026-09-21T15:25:22.374Z [INFO]  server-1d822775-3d3c-1b7c-4936-8788a6d64e50: Installed remote snapshot
2026-09-21T15:25:22.374Z [WARN]  server-1d822775-3d3c-1b7c-4936-8788a6d64e50: skipping application of old log: index=22
2026-09-21T15:25:22.374Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: pipelining replication: peer="{Voter server-1d822775-3d3c-1b7c-4936-8788a6d64e50 1d822775-3d3c-1b7c-4936-8788a6d64e50}"
2026-09-21T15:25:22.374Z [INFO]  server-c7203980-09fb-b44a-6b29-b6f04998d4d6: snapshot restore progress: id=2-1013-1790004322373 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:22.374Z [INFO]  server-c7203980-09fb-b44a-6b29-b6f04998d4d6: removing all old logs from log store
2026-09-21T15:25:22.374Z [INFO]  server-c7203980-09fb-b44a-6b29-b6f04998d4d6: compacting logs: from=1 to=22
2026-09-21T15:25:22.374Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:22.374Z [INFO]  server-c7203980-09fb-b44a-6b29-b6f04998d4d6: Installed remote snapshot
2026-09-21T15:25:22.374Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: pipelining replication: peer="{Voter server-c7203980-09fb-b44a-6b29-b6f04998d4d6 c7203980-09fb-b44a-6b29-b6f04998d4d6}"
2026-09-21T15:25:23.380Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:22.374631188 +0000 UTC m=+74.315801361 (2 nodes), 1.24µs from start of poll, 1.8365208s from cluster start. Timeout at 2026-09-21 15:25:23.379634388 +0000 UTC m=+75.320804561, 1.0050032s after stability
2026-09-21T15:25:23.393Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: aborting pipeline replication: peer="{Voter server-1d822775-3d3c-1b7c-4936-8788a6d64e50 1d822775-3d3c-1b7c-4936-8788a6d64e50}"
2026-09-21T15:25:23.393Z [INFO]  server-1de0bfb3-54c5-5612-7845-d305304e8c48: aborting pipeline replication: peer="{Voter server-c7203980-09fb-b44a-6b29-b6f04998d4d6 c7203980-09fb-b44a-6b29-b6f04998d4d6}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true
2026-09-21T15:25:23.394Z [INFO]  server-152702a1-237c-569c-e02a-fccf5081ba83: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-152702a1-237c-569c-e02a-fccf5081ba83 Address:152702a1-237c-569c-e02a-fccf5081ba83} {Suffrage:Voter ID:server-3c5876cd-e444-2397-3018-2755c62270e9 Address:3c5876cd-e444-2397-3018-2755c62270e9} {Suffrage:Voter ID:server-fd08779b-07cf-c730-cecf-42f9588219d9 Address:fd08779b-07cf-c730-cecf-42f9588219d9}]"
2026-09-21T15:25:23.394Z [INFO]  server-3c5876cd-e444-2397-3018-2755c62270e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-152702a1-237c-569c-e02a-fccf5081ba83 Address:152702a1-237c-569c-e02a-fccf5081ba83} {Suffrage:Voter ID:server-3c5876cd-e444-2397-3018-2755c62270e9 Address:3c5876cd-e444-2397-3018-2755c62270e9} {Suffrage:Voter ID:server-fd08779b-07cf-c730-cecf-42f9588219d9 Address:fd08779b-07cf-c730-cecf-42f9588219d9}]"
2026-09-21T15:25:23.394Z [INFO]  server-152702a1-237c-569c-e02a-fccf5081ba83: entering follower state: follower="Node at 152702a1-237c-569c-e02a-fccf5081ba83 [Follower]" leader-address= leader-id=
2026-09-21T15:25:23.394Z [INFO]  server-fd08779b-07cf-c730-cecf-42f9588219d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-152702a1-237c-569c-e02a-fccf5081ba83 Address:152702a1-237c-569c-e02a-fccf5081ba83} {Suffrage:Voter ID:server-3c5876cd-e444-2397-3018-2755c62270e9 Address:3c5876cd-e444-2397-3018-2755c62270e9} {Suffrage:Voter ID:server-fd08779b-07cf-c730-cecf-42f9588219d9 Address:fd08779b-07cf-c730-cecf-42f9588219d9}]"
2026-09-21T15:25:23.394Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:23.394Z [INFO]  server-3c5876cd-e444-2397-3018-2755c62270e9: entering follower state: follower="Node at 3c5876cd-e444-2397-3018-2755c62270e9 [Follower]" leader-address= leader-id=
2026-09-21T15:25:23.394Z [INFO]  server-fd08779b-07cf-c730-cecf-42f9588219d9: entering follower state: follower="Node at fd08779b-07cf-c730-cecf-42f9588219d9 [Follower]" leader-address= leader-id=
2026-09-21T15:25:24.176Z [WARN]  server-3c5876cd-e444-2397-3018-2755c62270e9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:24.176Z [INFO]  server-3c5876cd-e444-2397-3018-2755c62270e9: entering candidate state: node="Node at 3c5876cd-e444-2397-3018-2755c62270e9 [Candidate]" term=2
2026-09-21T15:25:24.176Z [INFO]  server-3c5876cd-e444-2397-3018-2755c62270e9: election won: term=2 tally=2
2026-09-21T15:25:24.176Z [INFO]  server-3c5876cd-e444-2397-3018-2755c62270e9: entering leader state: leader="Node at 3c5876cd-e444-2397-3018-2755c62270e9 [Leader]"
2026-09-21T15:25:24.176Z [INFO]  server-3c5876cd-e444-2397-3018-2755c62270e9: added peer, starting replication: peer=server-152702a1-237c-569c-e02a-fccf5081ba83
2026-09-21T15:25:24.176Z [INFO]  server-3c5876cd-e444-2397-3018-2755c62270e9: added peer, starting replication: peer=server-fd08779b-07cf-c730-cecf-42f9588219d9
2026-09-21T15:25:24.176Z [INFO]  server-3c5876cd-e444-2397-3018-2755c62270e9: pipelining replication: peer="{Voter server-152702a1-237c-569c-e02a-fccf5081ba83 152702a1-237c-569c-e02a-fccf5081ba83}"
2026-09-21T15:25:24.176Z [INFO]  server-3c5876cd-e444-2397-3018-2755c62270e9: pipelining replication: peer="{Voter server-fd08779b-07cf-c730-cecf-42f9588219d9 fd08779b-07cf-c730-cecf-42f9588219d9}"
2026-09-21T15:25:25.182Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:24.176356028 +0000 UTC m=+76.117526201 (1 nodes), 781.7424ms from start of poll, 782.14564ms from cluster start. Timeout at 2026-09-21 15:25:25.181359708 +0000 UTC m=+77.122529881, 1.00500368s after stability
2026-09-21T15:25:25.182Z [INFO]  server-3c5876cd-e444-2397-3018-2755c62270e9: starting snapshot up to: index=12
2026-09-21T15:25:25.182Z [INFO]  creating new snapshot: path=/tmp/raft3899586312/snapshots/2-12-1790004325182.tmp
2026-09-21T15:25:25.183Z [INFO]  server-3c5876cd-e444-2397-3018-2755c62270e9: snapshot complete up to: index=12
2026-09-21T15:25:25.184Z [INFO]  server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc Address:90e3e9a4-23a3-5ffa-d067-6945e0f1adfc} {Suffrage:Voter ID:server-05131cb8-4606-8c23-bed9-04e1df9b2660 Address:05131cb8-4606-8c23-bed9-04e1df9b2660} {Suffrage:Voter ID:server-5f3cd656-5f90-b39f-a53c-138e71229910 Address:5f3cd656-5f90-b39f-a53c-138e71229910}]"
2026-09-21T15:25:25.184Z [INFO]  server-05131cb8-4606-8c23-bed9-04e1df9b2660: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc Address:90e3e9a4-23a3-5ffa-d067-6945e0f1adfc} {Suffrage:Voter ID:server-05131cb8-4606-8c23-bed9-04e1df9b2660 Address:05131cb8-4606-8c23-bed9-04e1df9b2660} {Suffrage:Voter ID:server-5f3cd656-5f90-b39f-a53c-138e71229910 Address:5f3cd656-5f90-b39f-a53c-138e71229910}]"
2026-09-21T15:25:25.184Z [INFO]  server-5f3cd656-5f90-b39f-a53c-138e71229910: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc Address:90e3e9a4-23a3-5ffa-d067-6945e0f1adfc} {Suffrage:Voter ID:server-05131cb8-4606-8c23-bed9-04e1df9b2660 Address:05131cb8-4606-8c23-bed9-04e1df9b2660} {Suffrage:Voter ID:server-5f3cd656-5f90-b39f-a53c-138e71229910 Address:5f3cd656-5f90-b39f-a53c-138e71229910}]"
2026-09-21T15:25:25.184Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:25.185Z [INFO]  server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: entering follower state: follower="Node at 90e3e9a4-23a3-5ffa-d067-6945e0f1adfc [Follower]" leader-address= leader-id=
2026-09-21T15:25:25.185Z [INFO]  server-05131cb8-4606-8c23-bed9-04e1df9b2660: entering follower state: follower="Node at 05131cb8-4606-8c23-bed9-04e1df9b2660 [Follower]" leader-address= leader-id=
2026-09-21T15:25:25.185Z [INFO]  server-5f3cd656-5f90-b39f-a53c-138e71229910: entering follower state: follower="Node at 5f3cd656-5f90-b39f-a53c-138e71229910 [Follower]" leader-address= leader-id=
2026-09-21T15:25:25.695Z [WARN]  server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:25.695Z [INFO]  server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: entering candidate state: node="Node at 90e3e9a4-23a3-5ffa-d067-6945e0f1adfc [Candidate]" term=2
2026-09-21T15:25:25.695Z [INFO]  server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: election won: term=2 tally=2
2026-09-21T15:25:25.695Z [INFO]  server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: entering leader state: leader="Node at 90e3e9a4-23a3-5ffa-d067-6945e0f1adfc [Leader]"
2026-09-21T15:25:25.695Z [INFO]  server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: added peer, starting replication: peer=server-05131cb8-4606-8c23-bed9-04e1df9b2660
2026-09-21T15:25:25.695Z [INFO]  server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: added peer, starting replication: peer=server-5f3cd656-5f90-b39f-a53c-138e71229910
2026-09-21T15:25:25.695Z [INFO]  server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: pipelining replication: peer="{Voter server-5f3cd656-5f90-b39f-a53c-138e71229910 5f3cd656-5f90-b39f-a53c-138e71229910}"
2026-09-21T15:25:25.695Z [WARN]  server-05131cb8-4606-8c23-bed9-04e1df9b2660: rejecting vote request since our last term is greater: candidate=90e3e9a4-23a3-5ffa-d067-6945e0f1adfc last-term=2 last-candidate-term=1
2026-09-21T15:25:25.695Z [INFO]  server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: pipelining replication: peer="{Voter server-05131cb8-4606-8c23-bed9-04e1df9b2660 05131cb8-4606-8c23-bed9-04e1df9b2660}"
2026-09-21T15:25:26.700Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:25.695363188 +0000 UTC m=+77.636533321 (1 nodes), 510.36824ms from start of poll, 510.80628ms from cluster start. Timeout at 2026-09-21 15:25:26.700365308 +0000 UTC m=+78.641535521, 1.0050022s after stability
2026-09-21T15:25:26.700Z [INFO]  creating new snapshot: path=/tmp/raft2181961164/snapshots/2-1013-1790004326700.tmp
2026-09-21T15:25:26.701Z [INFO]  server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: copied to local snapshot: bytes=71
2026-09-21T15:25:26.701Z [INFO]  server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: snapshot restore progress: id=2-1013-1790004326700 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:26.701Z [INFO]  server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: restored user snapshot: index=1013
2026-09-21T15:25:26.701Z [ERROR] server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: failed to get log: index=3 error="log not found"
2026-09-21T15:25:26.701Z [INFO]  server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: aborting pipeline replication: peer="{Voter server-05131cb8-4606-8c23-bed9-04e1df9b2660 05131cb8-4606-8c23-bed9-04e1df9b2660}"
2026-09-21T15:25:26.701Z [ERROR] server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: failed to get log: index=3 error="log not found"
2026-09-21T15:25:26.701Z [INFO]  server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: aborting pipeline replication: peer="{Voter server-5f3cd656-5f90-b39f-a53c-138e71229910 5f3cd656-5f90-b39f-a53c-138e71229910}"
2026-09-21T15:25:26.707Z [ERROR] server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: failed to get log: index=3 error="log not found"
2026-09-21T15:25:26.708Z [INFO]  creating new snapshot: path=/tmp/raft3191821420/snapshots/2-1013-1790004326708.tmp
2026-09-21T15:25:26.708Z [INFO]  server-5f3cd656-5f90-b39f-a53c-138e71229910: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:26.708Z [ERROR] server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: failed to get log: index=3 error="log not found"
2026-09-21T15:25:26.708Z [INFO]  server-5f3cd656-5f90-b39f-a53c-138e71229910: copied to local snapshot: bytes=71
2026-09-21T15:25:26.708Z [INFO]  server-5f3cd656-5f90-b39f-a53c-138e71229910: snapshot restore progress: id=2-1013-1790004326708 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:26.708Z [INFO]  server-5f3cd656-5f90-b39f-a53c-138e71229910: Installed remote snapshot
2026-09-21T15:25:26.708Z [INFO]  server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: pipelining replication: peer="{Voter server-5f3cd656-5f90-b39f-a53c-138e71229910 5f3cd656-5f90-b39f-a53c-138e71229910}"
2026-09-21T15:25:26.708Z [INFO]  creating new snapshot: path=/tmp/raft2924540517/snapshots/2-1013-1790004326708.tmp
2026-09-21T15:25:26.708Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:26.708Z [INFO]  server-05131cb8-4606-8c23-bed9-04e1df9b2660: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:26.709Z [INFO]  server-05131cb8-4606-8c23-bed9-04e1df9b2660: copied to local snapshot: bytes=71
2026-09-21T15:25:26.709Z [INFO]  server-05131cb8-4606-8c23-bed9-04e1df9b2660: snapshot restore progress: id=2-1013-1790004326708 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:26.709Z [INFO]  server-05131cb8-4606-8c23-bed9-04e1df9b2660: Installed remote snapshot
2026-09-21T15:25:26.709Z [INFO]  server-90e3e9a4-23a3-5ffa-d067-6945e0f1adfc: pipelining replication: peer="{Voter server-05131cb8-4606-8c23-bed9-04e1df9b2660 05131cb8-4606-8c23-bed9-04e1df9b2660}"
2026-09-21T15:25:27.713Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:26.708835988 +0000 UTC m=+78.650006121 (2 nodes), 2.16µs from start of poll, 1.52427908s from cluster start. Timeout at 2026-09-21 15:25:27.713856708 +0000 UTC m=+79.655026881, 1.00502076s after stability
2026-09-21T15:25:27.725Z [INFO]  server-3c5876cd-e444-2397-3018-2755c62270e9: aborting pipeline replication: peer="{Voter server-fd08779b-07cf-c730-cecf-42f9588219d9 fd08779b-07cf-c730-cecf-42f9588219d9}"
2026-09-21T15:25:27.725Z [INFO]  server-3c5876cd-e444-2397-3018-2755c62270e9: aborting pipeline replication: peer="{Voter server-152702a1-237c-569c-e02a-fccf5081ba83 152702a1-237c-569c-e02a-fccf5081ba83}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true
2026-09-21T15:25:27.726Z [INFO]  server-680e69b0-2550-39fa-6b26-ab586962a1b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-680e69b0-2550-39fa-6b26-ab586962a1b4 Address:680e69b0-2550-39fa-6b26-ab586962a1b4} {Suffrage:Voter ID:server-3006653f-3fe8-5174-a6af-8c0dff4f6341 Address:3006653f-3fe8-5174-a6af-8c0dff4f6341} {Suffrage:Voter ID:server-49511524-bbe8-5c8d-c601-8e6c6d4fbf14 Address:49511524-bbe8-5c8d-c601-8e6c6d4fbf14}]"
2026-09-21T15:25:27.726Z [INFO]  server-3006653f-3fe8-5174-a6af-8c0dff4f6341: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-680e69b0-2550-39fa-6b26-ab586962a1b4 Address:680e69b0-2550-39fa-6b26-ab586962a1b4} {Suffrage:Voter ID:server-3006653f-3fe8-5174-a6af-8c0dff4f6341 Address:3006653f-3fe8-5174-a6af-8c0dff4f6341} {Suffrage:Voter ID:server-49511524-bbe8-5c8d-c601-8e6c6d4fbf14 Address:49511524-bbe8-5c8d-c601-8e6c6d4fbf14}]"
2026-09-21T15:25:27.726Z [INFO]  server-49511524-bbe8-5c8d-c601-8e6c6d4fbf14: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-680e69b0-2550-39fa-6b26-ab586962a1b4 Address:680e69b0-2550-39fa-6b26-ab586962a1b4} {Suffrage:Voter ID:server-3006653f-3fe8-5174-a6af-8c0dff4f6341 Address:3006653f-3fe8-5174-a6af-8c0dff4f6341} {Suffrage:Voter ID:server-49511524-bbe8-5c8d-c601-8e6c6d4fbf14 Address:49511524-bbe8-5c8d-c601-8e6c6d4fbf14}]"
2026-09-21T15:25:27.726Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:27.726Z [INFO]  server-680e69b0-2550-39fa-6b26-ab586962a1b4: entering follower state: follower="Node at 680e69b0-2550-39fa-6b26-ab586962a1b4 [Follower]" leader-address= leader-id=
2026-09-21T15:25:27.726Z [INFO]  server-3006653f-3fe8-5174-a6af-8c0dff4f6341: entering follower state: follower="Node at 3006653f-3fe8-5174-a6af-8c0dff4f6341 [Follower]" leader-address= leader-id=
2026-09-21T15:25:27.726Z [INFO]  server-49511524-bbe8-5c8d-c601-8e6c6d4fbf14: entering follower state: follower="Node at 49511524-bbe8-5c8d-c601-8e6c6d4fbf14 [Follower]" leader-address= leader-id=
2026-09-21T15:25:28.251Z [WARN]  server-49511524-bbe8-5c8d-c601-8e6c6d4fbf14: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:28.251Z [INFO]  server-49511524-bbe8-5c8d-c601-8e6c6d4fbf14: entering candidate state: node="Node at 49511524-bbe8-5c8d-c601-8e6c6d4fbf14 [Candidate]" term=2
2026-09-21T15:25:28.253Z [INFO]  server-49511524-bbe8-5c8d-c601-8e6c6d4fbf14: election won: term=2 tally=2
2026-09-21T15:25:28.253Z [INFO]  server-49511524-bbe8-5c8d-c601-8e6c6d4fbf14: entering leader state: leader="Node at 49511524-bbe8-5c8d-c601-8e6c6d4fbf14 [Leader]"
2026-09-21T15:25:28.253Z [INFO]  server-49511524-bbe8-5c8d-c601-8e6c6d4fbf14: added peer, starting replication: peer=server-680e69b0-2550-39fa-6b26-ab586962a1b4
2026-09-21T15:25:28.253Z [INFO]  server-49511524-bbe8-5c8d-c601-8e6c6d4fbf14: added peer, starting replication: peer=server-3006653f-3fe8-5174-a6af-8c0dff4f6341
2026-09-21T15:25:28.253Z [INFO]  server-49511524-bbe8-5c8d-c601-8e6c6d4fbf14: pipelining replication: peer="{Voter server-3006653f-3fe8-5174-a6af-8c0dff4f6341 3006653f-3fe8-5174-a6af-8c0dff4f6341}"
2026-09-21T15:25:28.254Z [INFO]  server-49511524-bbe8-5c8d-c601-8e6c6d4fbf14: pipelining replication: peer="{Voter server-680e69b0-2550-39fa-6b26-ab586962a1b4 680e69b0-2550-39fa-6b26-ab586962a1b4}"
2026-09-21T15:25:29.259Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:28.253621828 +0000 UTC m=+80.194792041 (1 nodes), 527.0706ms from start of poll, 527.4584ms from cluster start. Timeout at 2026-09-21 15:25:29.258625828 +0000 UTC m=+81.199795961, 1.00500392s after stability
2026-09-21T15:25:29.259Z [INFO]  server-49511524-bbe8-5c8d-c601-8e6c6d4fbf14: starting snapshot up to: index=12
2026-09-21T15:25:29.259Z [INFO]  creating new snapshot: path=/tmp/raft2666981161/snapshots/2-12-1790004329259.tmp
2026-09-21T15:25:29.260Z [INFO]  server-49511524-bbe8-5c8d-c601-8e6c6d4fbf14: snapshot complete up to: index=12
2026-09-21T15:25:29.260Z [INFO]  server-2dc17095-aac0-d12d-fa0d-375665b55327: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dc17095-aac0-d12d-fa0d-375665b55327 Address:2dc17095-aac0-d12d-fa0d-375665b55327} {Suffrage:Voter ID:server-adba77ff-9766-0cf3-6cce-d099c72be6d8 Address:adba77ff-9766-0cf3-6cce-d099c72be6d8} {Suffrage:Voter ID:server-58e4c98c-1852-61e6-bbff-eb25cae17eb0 Address:58e4c98c-1852-61e6-bbff-eb25cae17eb0}]"
2026-09-21T15:25:29.260Z [INFO]  server-adba77ff-9766-0cf3-6cce-d099c72be6d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dc17095-aac0-d12d-fa0d-375665b55327 Address:2dc17095-aac0-d12d-fa0d-375665b55327} {Suffrage:Voter ID:server-adba77ff-9766-0cf3-6cce-d099c72be6d8 Address:adba77ff-9766-0cf3-6cce-d099c72be6d8} {Suffrage:Voter ID:server-58e4c98c-1852-61e6-bbff-eb25cae17eb0 Address:58e4c98c-1852-61e6-bbff-eb25cae17eb0}]"
2026-09-21T15:25:29.260Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2dc17095-aac0-d12d-fa0d-375665b55327 Address:2dc17095-aac0-d12d-fa0d-375665b55327} {Suffrage:Voter ID:server-adba77ff-9766-0cf3-6cce-d099c72be6d8 Address:adba77ff-9766-0cf3-6cce-d099c72be6d8} {Suffrage:Voter ID:server-58e4c98c-1852-61e6-bbff-eb25cae17eb0 Address:58e4c98c-1852-61e6-bbff-eb25cae17eb0}]"
2026-09-21T15:25:29.260Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:29.260Z [INFO]  server-2dc17095-aac0-d12d-fa0d-375665b55327: entering follower state: follower="Node at 2dc17095-aac0-d12d-fa0d-375665b55327 [Follower]" leader-address= leader-id=
2026-09-21T15:25:29.260Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: entering follower state: follower="Node at 58e4c98c-1852-61e6-bbff-eb25cae17eb0 [Follower]" leader-address= leader-id=
2026-09-21T15:25:29.260Z [INFO]  server-adba77ff-9766-0cf3-6cce-d099c72be6d8: entering follower state: follower="Node at adba77ff-9766-0cf3-6cce-d099c72be6d8 [Follower]" leader-address= leader-id=
2026-09-21T15:25:29.863Z [WARN]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:29.863Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: entering candidate state: node="Node at 58e4c98c-1852-61e6-bbff-eb25cae17eb0 [Candidate]" term=2
2026-09-21T15:25:29.863Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: election won: term=2 tally=2
2026-09-21T15:25:29.863Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: entering leader state: leader="Node at 58e4c98c-1852-61e6-bbff-eb25cae17eb0 [Leader]"
2026-09-21T15:25:29.863Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: added peer, starting replication: peer=server-2dc17095-aac0-d12d-fa0d-375665b55327
2026-09-21T15:25:29.863Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: added peer, starting replication: peer=server-adba77ff-9766-0cf3-6cce-d099c72be6d8
2026-09-21T15:25:29.863Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: pipelining replication: peer="{Voter server-adba77ff-9766-0cf3-6cce-d099c72be6d8 adba77ff-9766-0cf3-6cce-d099c72be6d8}"
2026-09-21T15:25:29.863Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: pipelining replication: peer="{Voter server-2dc17095-aac0-d12d-fa0d-375665b55327 2dc17095-aac0-d12d-fa0d-375665b55327}"
2026-09-21T15:25:30.868Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:29.863686948 +0000 UTC m=+81.804857081 (1 nodes), 602.84716ms from start of poll, 603.06252ms from cluster start. Timeout at 2026-09-21 15:25:30.868688508 +0000 UTC m=+82.809858641, 1.00500156s after stability
2026-09-21T15:25:30.868Z [INFO]  creating new snapshot: path=/tmp/raft1025378715/snapshots/2-1013-1790004330868.tmp
2026-09-21T15:25:30.869Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: copied to local snapshot: bytes=71
2026-09-21T15:25:30.869Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: snapshot restore progress: id=2-1013-1790004330868 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:30.869Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: removing all old logs from log store
2026-09-21T15:25:30.869Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: compacting logs: from=1 to=2
2026-09-21T15:25:30.869Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: restored user snapshot: index=1013
2026-09-21T15:25:30.869Z [ERROR] server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: failed to get log: index=2 error="log not found"
2026-09-21T15:25:30.869Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: aborting pipeline replication: peer="{Voter server-adba77ff-9766-0cf3-6cce-d099c72be6d8 adba77ff-9766-0cf3-6cce-d099c72be6d8}"
2026-09-21T15:25:30.869Z [ERROR] server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: failed to get log: index=2 error="log not found"
2026-09-21T15:25:30.869Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: aborting pipeline replication: peer="{Voter server-2dc17095-aac0-d12d-fa0d-375665b55327 2dc17095-aac0-d12d-fa0d-375665b55327}"
2026-09-21T15:25:30.875Z [ERROR] server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: failed to get log: index=2 error="log not found"
2026-09-21T15:25:30.876Z [INFO]  creating new snapshot: path=/tmp/raft1528362227/snapshots/2-1013-1790004330876.tmp
2026-09-21T15:25:30.876Z [INFO]  server-adba77ff-9766-0cf3-6cce-d099c72be6d8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:30.876Z [INFO]  server-adba77ff-9766-0cf3-6cce-d099c72be6d8: copied to local snapshot: bytes=71
2026-09-21T15:25:30.876Z [INFO]  server-adba77ff-9766-0cf3-6cce-d099c72be6d8: snapshot restore progress: id=2-1013-1790004330876 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:30.876Z [INFO]  server-adba77ff-9766-0cf3-6cce-d099c72be6d8: removing all old logs from log store
2026-09-21T15:25:30.876Z [INFO]  server-adba77ff-9766-0cf3-6cce-d099c72be6d8: compacting logs: from=1 to=2
2026-09-21T15:25:30.876Z [INFO]  server-adba77ff-9766-0cf3-6cce-d099c72be6d8: Installed remote snapshot
2026-09-21T15:25:30.876Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: pipelining replication: peer="{Voter server-adba77ff-9766-0cf3-6cce-d099c72be6d8 adba77ff-9766-0cf3-6cce-d099c72be6d8}"
2026-09-21T15:25:30.876Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:30.879Z [ERROR] server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: failed to get log: index=2 error="log not found"
2026-09-21T15:25:30.879Z [INFO]  creating new snapshot: path=/tmp/raft2462434635/snapshots/2-1013-1790004330879.tmp
2026-09-21T15:25:30.879Z [INFO]  server-2dc17095-aac0-d12d-fa0d-375665b55327: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:30.879Z [INFO]  server-2dc17095-aac0-d12d-fa0d-375665b55327: copied to local snapshot: bytes=71
2026-09-21T15:25:30.879Z [INFO]  server-2dc17095-aac0-d12d-fa0d-375665b55327: snapshot restore progress: id=2-1013-1790004330879 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:30.879Z [INFO]  server-2dc17095-aac0-d12d-fa0d-375665b55327: removing all old logs from log store
2026-09-21T15:25:30.879Z [INFO]  server-2dc17095-aac0-d12d-fa0d-375665b55327: compacting logs: from=1 to=2
2026-09-21T15:25:30.879Z [INFO]  server-2dc17095-aac0-d12d-fa0d-375665b55327: Installed remote snapshot
2026-09-21T15:25:30.880Z [INFO]  server-58e4c98c-1852-61e6-bbff-eb25cae17eb0: pipelining replication: peer="{Voter server-2dc17095-aac0-d12d-fa0d-375665b55327 2dc17095-aac0-d12d-fa0d-375665b55327}"
2026-09-21T15:25:31.881Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:30.876555348 +0000 UTC m=+82.817725481 (2 nodes), 480ns from start of poll, 1.61593092s from cluster start. Timeout at 2026-09-21 15:25:31.881586668 +0000 UTC m=+83.822756761, 1.00503128s after stability
2026-09-21T15:25:31.894Z [INFO]  server-49511524-bbe8-5c8d-c601-8e6c6d4fbf14: aborting pipeline replication: peer="{Voter server-3006653f-3fe8-5174-a6af-8c0dff4f6341 3006653f-3fe8-5174-a6af-8c0dff4f6341}"
2026-09-21T15:25:31.894Z [INFO]  server-49511524-bbe8-5c8d-c601-8e6c6d4fbf14: aborting pipeline replication: peer="{Voter server-680e69b0-2550-39fa-6b26-ab586962a1b4 680e69b0-2550-39fa-6b26-ab586962a1b4}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false
2026-09-21T15:25:31.895Z [INFO]  server-a1f8d771-f1da-c876-96dc-f86b55acc6e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1f8d771-f1da-c876-96dc-f86b55acc6e0 Address:a1f8d771-f1da-c876-96dc-f86b55acc6e0} {Suffrage:Voter ID:server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2 Address:9bae2b6c-bd63-31c5-d34a-72e56ad37db2} {Suffrage:Voter ID:server-9974dbf3-a8e9-deb5-f0f5-b5f88c505156 Address:9974dbf3-a8e9-deb5-f0f5-b5f88c505156}]"
2026-09-21T15:25:31.895Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1f8d771-f1da-c876-96dc-f86b55acc6e0 Address:a1f8d771-f1da-c876-96dc-f86b55acc6e0} {Suffrage:Voter ID:server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2 Address:9bae2b6c-bd63-31c5-d34a-72e56ad37db2} {Suffrage:Voter ID:server-9974dbf3-a8e9-deb5-f0f5-b5f88c505156 Address:9974dbf3-a8e9-deb5-f0f5-b5f88c505156}]"
2026-09-21T15:25:31.895Z [INFO]  server-9974dbf3-a8e9-deb5-f0f5-b5f88c505156: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1f8d771-f1da-c876-96dc-f86b55acc6e0 Address:a1f8d771-f1da-c876-96dc-f86b55acc6e0} {Suffrage:Voter ID:server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2 Address:9bae2b6c-bd63-31c5-d34a-72e56ad37db2} {Suffrage:Voter ID:server-9974dbf3-a8e9-deb5-f0f5-b5f88c505156 Address:9974dbf3-a8e9-deb5-f0f5-b5f88c505156}]"
2026-09-21T15:25:31.895Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:31.895Z [INFO]  server-a1f8d771-f1da-c876-96dc-f86b55acc6e0: entering follower state: follower="Node at a1f8d771-f1da-c876-96dc-f86b55acc6e0 [Follower]" leader-address= leader-id=
2026-09-21T15:25:31.895Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: entering follower state: follower="Node at 9bae2b6c-bd63-31c5-d34a-72e56ad37db2 [Follower]" leader-address= leader-id=
2026-09-21T15:25:31.895Z [INFO]  server-9974dbf3-a8e9-deb5-f0f5-b5f88c505156: entering follower state: follower="Node at 9974dbf3-a8e9-deb5-f0f5-b5f88c505156 [Follower]" leader-address= leader-id=
2026-09-21T15:25:32.493Z [WARN]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:32.493Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: entering candidate state: node="Node at 9bae2b6c-bd63-31c5-d34a-72e56ad37db2 [Candidate]" term=2
2026-09-21T15:25:32.493Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: election won: term=2 tally=2
2026-09-21T15:25:32.493Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: entering leader state: leader="Node at 9bae2b6c-bd63-31c5-d34a-72e56ad37db2 [Leader]"
2026-09-21T15:25:32.493Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: added peer, starting replication: peer=server-a1f8d771-f1da-c876-96dc-f86b55acc6e0
2026-09-21T15:25:32.493Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: added peer, starting replication: peer=server-9974dbf3-a8e9-deb5-f0f5-b5f88c505156
2026-09-21T15:25:32.494Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: pipelining replication: peer="{Voter server-9974dbf3-a8e9-deb5-f0f5-b5f88c505156 9974dbf3-a8e9-deb5-f0f5-b5f88c505156}"
2026-09-21T15:25:32.494Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: pipelining replication: peer="{Voter server-a1f8d771-f1da-c876-96dc-f86b55acc6e0 a1f8d771-f1da-c876-96dc-f86b55acc6e0}"
2026-09-21T15:25:33.503Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:32.494238428 +0000 UTC m=+84.435408561 (1 nodes), 598.80068ms from start of poll, 599.0038ms from cluster start. Timeout at 2026-09-21 15:25:33.499240068 +0000 UTC m=+85.440410161, 1.0050016s after stability
2026-09-21T15:25:33.504Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: starting snapshot up to: index=12
2026-09-21T15:25:33.504Z [INFO]  creating new snapshot: path=/tmp/raft3754957163/snapshots/2-12-1790004333504.tmp
2026-09-21T15:25:33.504Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: snapshot complete up to: index=12
2026-09-21T15:25:33.504Z [INFO]  creating new snapshot: path=/tmp/raft3754957163/snapshots/2-10013-1790004333504.tmp
2026-09-21T15:25:33.505Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: copied to local snapshot: bytes=71
2026-09-21T15:25:33.505Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: snapshot restore progress: id=2-10013-1790004333504 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:33.505Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: restored user snapshot: index=10013
2026-09-21T15:25:33.505Z [ERROR] server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: failed to get log: index=23 error="log not found"
2026-09-21T15:25:33.505Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: aborting pipeline replication: peer="{Voter server-a1f8d771-f1da-c876-96dc-f86b55acc6e0 a1f8d771-f1da-c876-96dc-f86b55acc6e0}"
2026-09-21T15:25:33.505Z [ERROR] server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: failed to get log: index=23 error="log not found"
2026-09-21T15:25:33.505Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: aborting pipeline replication: peer="{Voter server-9974dbf3-a8e9-deb5-f0f5-b5f88c505156 9974dbf3-a8e9-deb5-f0f5-b5f88c505156}"
2026-09-21T15:25:33.511Z [ERROR] server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: failed to get log: index=23 error="log not found"
2026-09-21T15:25:33.511Z [INFO]  creating new snapshot: path=/tmp/raft1910127912/snapshots/2-10013-1790004333511.tmp
2026-09-21T15:25:33.512Z [INFO]  server-a1f8d771-f1da-c876-96dc-f86b55acc6e0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:33.512Z [INFO]  server-a1f8d771-f1da-c876-96dc-f86b55acc6e0: copied to local snapshot: bytes=71
2026-09-21T15:25:33.512Z [INFO]  server-a1f8d771-f1da-c876-96dc-f86b55acc6e0: snapshot restore progress: id=2-10013-1790004333511 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:33.512Z [INFO]  server-a1f8d771-f1da-c876-96dc-f86b55acc6e0: Installed remote snapshot
2026-09-21T15:25:33.512Z [WARN]  server-a1f8d771-f1da-c876-96dc-f86b55acc6e0: skipping application of old log: index=22
2026-09-21T15:25:33.512Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: pipelining replication: peer="{Voter server-a1f8d771-f1da-c876-96dc-f86b55acc6e0 a1f8d771-f1da-c876-96dc-f86b55acc6e0}"
2026-09-21T15:25:33.512Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:33.512Z [ERROR] server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: failed to get log: index=23 error="log not found"
2026-09-21T15:25:33.514Z [INFO]  creating new snapshot: path=/tmp/raft40101620/snapshots/2-10013-1790004333514.tmp
2026-09-21T15:25:33.514Z [INFO]  server-9974dbf3-a8e9-deb5-f0f5-b5f88c505156: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:33.514Z [INFO]  server-9974dbf3-a8e9-deb5-f0f5-b5f88c505156: copied to local snapshot: bytes=71
2026-09-21T15:25:33.514Z [INFO]  server-9974dbf3-a8e9-deb5-f0f5-b5f88c505156: snapshot restore progress: id=2-10013-1790004333514 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:33.514Z [INFO]  server-9974dbf3-a8e9-deb5-f0f5-b5f88c505156: Installed remote snapshot
2026-09-21T15:25:33.514Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: pipelining replication: peer="{Voter server-9974dbf3-a8e9-deb5-f0f5-b5f88c505156 9974dbf3-a8e9-deb5-f0f5-b5f88c505156}"
2026-09-21T15:25:34.519Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:33.514078108 +0000 UTC m=+85.455248201 (2 nodes), 840ns from start of poll, 1.61884344s from cluster start. Timeout at 2026-09-21 15:25:34.519106228 +0000 UTC m=+86.460276361, 1.00502816s after stability
2026-09-21T15:25:34.531Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: aborting pipeline replication: peer="{Voter server-9974dbf3-a8e9-deb5-f0f5-b5f88c505156 9974dbf3-a8e9-deb5-f0f5-b5f88c505156}"
2026-09-21T15:25:34.531Z [INFO]  server-9bae2b6c-bd63-31c5-d34a-72e56ad37db2: aborting pipeline replication: peer="{Voter server-a1f8d771-f1da-c876-96dc-f86b55acc6e0 a1f8d771-f1da-c876-96dc-f86b55acc6e0}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false
2026-09-21T15:25:34.531Z [INFO]  server-0443e2b8-57f1-d798-9646-14ed5889055b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0443e2b8-57f1-d798-9646-14ed5889055b Address:0443e2b8-57f1-d798-9646-14ed5889055b} {Suffrage:Voter ID:server-e80d17b3-1ee3-4a7f-4b7e-94b543950610 Address:e80d17b3-1ee3-4a7f-4b7e-94b543950610} {Suffrage:Voter ID:server-9c9b9099-c631-92ce-7751-f8a9c1bc1749 Address:9c9b9099-c631-92ce-7751-f8a9c1bc1749}]"
2026-09-21T15:25:34.531Z [INFO]  server-e80d17b3-1ee3-4a7f-4b7e-94b543950610: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0443e2b8-57f1-d798-9646-14ed5889055b Address:0443e2b8-57f1-d798-9646-14ed5889055b} {Suffrage:Voter ID:server-e80d17b3-1ee3-4a7f-4b7e-94b543950610 Address:e80d17b3-1ee3-4a7f-4b7e-94b543950610} {Suffrage:Voter ID:server-9c9b9099-c631-92ce-7751-f8a9c1bc1749 Address:9c9b9099-c631-92ce-7751-f8a9c1bc1749}]"
2026-09-21T15:25:34.531Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0443e2b8-57f1-d798-9646-14ed5889055b Address:0443e2b8-57f1-d798-9646-14ed5889055b} {Suffrage:Voter ID:server-e80d17b3-1ee3-4a7f-4b7e-94b543950610 Address:e80d17b3-1ee3-4a7f-4b7e-94b543950610} {Suffrage:Voter ID:server-9c9b9099-c631-92ce-7751-f8a9c1bc1749 Address:9c9b9099-c631-92ce-7751-f8a9c1bc1749}]"
2026-09-21T15:25:34.531Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:34.532Z [INFO]  server-0443e2b8-57f1-d798-9646-14ed5889055b: entering follower state: follower="Node at 0443e2b8-57f1-d798-9646-14ed5889055b [Follower]" leader-address= leader-id=
2026-09-21T15:25:34.532Z [INFO]  server-e80d17b3-1ee3-4a7f-4b7e-94b543950610: entering follower state: follower="Node at e80d17b3-1ee3-4a7f-4b7e-94b543950610 [Follower]" leader-address= leader-id=
2026-09-21T15:25:34.532Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: entering follower state: follower="Node at 9c9b9099-c631-92ce-7751-f8a9c1bc1749 [Follower]" leader-address= leader-id=
2026-09-21T15:25:35.192Z [WARN]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:35.192Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: entering candidate state: node="Node at 9c9b9099-c631-92ce-7751-f8a9c1bc1749 [Candidate]" term=2
2026-09-21T15:25:35.192Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: election won: term=2 tally=2
2026-09-21T15:25:35.192Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: entering leader state: leader="Node at 9c9b9099-c631-92ce-7751-f8a9c1bc1749 [Leader]"
2026-09-21T15:25:35.192Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: added peer, starting replication: peer=server-0443e2b8-57f1-d798-9646-14ed5889055b
2026-09-21T15:25:35.193Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: added peer, starting replication: peer=server-e80d17b3-1ee3-4a7f-4b7e-94b543950610
2026-09-21T15:25:35.193Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: pipelining replication: peer="{Voter server-e80d17b3-1ee3-4a7f-4b7e-94b543950610 e80d17b3-1ee3-4a7f-4b7e-94b543950610}"
2026-09-21T15:25:35.193Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: pipelining replication: peer="{Voter server-0443e2b8-57f1-d798-9646-14ed5889055b 0443e2b8-57f1-d798-9646-14ed5889055b}"
2026-09-21T15:25:36.198Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:35.193207668 +0000 UTC m=+87.134377801 (1 nodes), 661.20584ms from start of poll, 661.4074ms from cluster start. Timeout at 2026-09-21 15:25:36.198209308 +0000 UTC m=+88.139379441, 1.00500164s after stability
2026-09-21T15:25:36.198Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: starting snapshot up to: index=12
2026-09-21T15:25:36.198Z [INFO]  creating new snapshot: path=/tmp/raft284957625/snapshots/2-12-1790004336198.tmp
2026-09-21T15:25:36.199Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: snapshot complete up to: index=12
2026-09-21T15:25:36.199Z [INFO]  creating new snapshot: path=/tmp/raft284957625/snapshots/2-10013-1790004336199.tmp
2026-09-21T15:25:36.199Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: copied to local snapshot: bytes=71
2026-09-21T15:25:36.200Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: snapshot restore progress: id=2-10013-1790004336199 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:36.200Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: removing all old logs from log store
2026-09-21T15:25:36.200Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: compacting logs: from=1 to=22
2026-09-21T15:25:36.200Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: restored user snapshot: index=10013
2026-09-21T15:25:36.200Z [ERROR] server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: failed to get log: index=22 error="log not found"
2026-09-21T15:25:36.200Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: aborting pipeline replication: peer="{Voter server-e80d17b3-1ee3-4a7f-4b7e-94b543950610 e80d17b3-1ee3-4a7f-4b7e-94b543950610}"
2026-09-21T15:25:36.200Z [ERROR] server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: failed to get log: index=22 error="log not found"
2026-09-21T15:25:36.200Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: aborting pipeline replication: peer="{Voter server-0443e2b8-57f1-d798-9646-14ed5889055b 0443e2b8-57f1-d798-9646-14ed5889055b}"
2026-09-21T15:25:36.208Z [ERROR] server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: failed to get log: index=22 error="log not found"
2026-09-21T15:25:36.208Z [INFO]  creating new snapshot: path=/tmp/raft2653816175/snapshots/2-10013-1790004336208.tmp
2026-09-21T15:25:36.208Z [INFO]  server-e80d17b3-1ee3-4a7f-4b7e-94b543950610: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:36.208Z [INFO]  server-e80d17b3-1ee3-4a7f-4b7e-94b543950610: copied to local snapshot: bytes=71
2026-09-21T15:25:36.209Z [INFO]  server-e80d17b3-1ee3-4a7f-4b7e-94b543950610: snapshot restore progress: id=2-10013-1790004336208 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:36.209Z [INFO]  server-e80d17b3-1ee3-4a7f-4b7e-94b543950610: removing all old logs from log store
2026-09-21T15:25:36.209Z [INFO]  server-e80d17b3-1ee3-4a7f-4b7e-94b543950610: compacting logs: from=1 to=22
2026-09-21T15:25:36.209Z [INFO]  server-e80d17b3-1ee3-4a7f-4b7e-94b543950610: Installed remote snapshot
2026-09-21T15:25:36.209Z [WARN]  server-e80d17b3-1ee3-4a7f-4b7e-94b543950610: skipping application of old log: index=22
2026-09-21T15:25:36.209Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: pipelining replication: peer="{Voter server-e80d17b3-1ee3-4a7f-4b7e-94b543950610 e80d17b3-1ee3-4a7f-4b7e-94b543950610}"
2026-09-21T15:25:36.209Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:36.210Z [ERROR] server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: failed to get log: index=22 error="log not found"
2026-09-21T15:25:36.210Z [INFO]  creating new snapshot: path=/tmp/raft1521682782/snapshots/2-10013-1790004336210.tmp
2026-09-21T15:25:36.210Z [INFO]  server-0443e2b8-57f1-d798-9646-14ed5889055b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:36.210Z [INFO]  server-0443e2b8-57f1-d798-9646-14ed5889055b: copied to local snapshot: bytes=71
2026-09-21T15:25:36.211Z [INFO]  server-0443e2b8-57f1-d798-9646-14ed5889055b: snapshot restore progress: id=2-10013-1790004336210 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:36.211Z [INFO]  server-0443e2b8-57f1-d798-9646-14ed5889055b: removing all old logs from log store
2026-09-21T15:25:36.211Z [INFO]  server-0443e2b8-57f1-d798-9646-14ed5889055b: compacting logs: from=1 to=22
2026-09-21T15:25:36.211Z [INFO]  server-0443e2b8-57f1-d798-9646-14ed5889055b: Installed remote snapshot
2026-09-21T15:25:36.211Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: pipelining replication: peer="{Voter server-0443e2b8-57f1-d798-9646-14ed5889055b 0443e2b8-57f1-d798-9646-14ed5889055b}"
2026-09-21T15:25:37.215Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:36.209158308 +0000 UTC m=+88.150328401 (2 nodes), 720ns from start of poll, 1.677358s from cluster start. Timeout at 2026-09-21 15:25:37.214185868 +0000 UTC m=+89.155356001, 1.0050276s after stability
2026-09-21T15:25:37.225Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: aborting pipeline replication: peer="{Voter server-e80d17b3-1ee3-4a7f-4b7e-94b543950610 e80d17b3-1ee3-4a7f-4b7e-94b543950610}"
2026-09-21T15:25:37.226Z [INFO]  server-9c9b9099-c631-92ce-7751-f8a9c1bc1749: aborting pipeline replication: peer="{Voter server-0443e2b8-57f1-d798-9646-14ed5889055b 0443e2b8-57f1-d798-9646-14ed5889055b}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true
2026-09-21T15:25:37.226Z [INFO]  server-12114bc6-9a08-4381-7281-bb42a545d943: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12114bc6-9a08-4381-7281-bb42a545d943 Address:12114bc6-9a08-4381-7281-bb42a545d943} {Suffrage:Voter ID:server-a5de88fb-d0c5-b39d-7f0f-81323bc841f9 Address:a5de88fb-d0c5-b39d-7f0f-81323bc841f9} {Suffrage:Voter ID:server-80ca3719-8c8d-3cca-7538-1aec257dec16 Address:80ca3719-8c8d-3cca-7538-1aec257dec16}]"
2026-09-21T15:25:37.227Z [INFO]  server-a5de88fb-d0c5-b39d-7f0f-81323bc841f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12114bc6-9a08-4381-7281-bb42a545d943 Address:12114bc6-9a08-4381-7281-bb42a545d943} {Suffrage:Voter ID:server-a5de88fb-d0c5-b39d-7f0f-81323bc841f9 Address:a5de88fb-d0c5-b39d-7f0f-81323bc841f9} {Suffrage:Voter ID:server-80ca3719-8c8d-3cca-7538-1aec257dec16 Address:80ca3719-8c8d-3cca-7538-1aec257dec16}]"
2026-09-21T15:25:37.227Z [INFO]  server-80ca3719-8c8d-3cca-7538-1aec257dec16: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12114bc6-9a08-4381-7281-bb42a545d943 Address:12114bc6-9a08-4381-7281-bb42a545d943} {Suffrage:Voter ID:server-a5de88fb-d0c5-b39d-7f0f-81323bc841f9 Address:a5de88fb-d0c5-b39d-7f0f-81323bc841f9} {Suffrage:Voter ID:server-80ca3719-8c8d-3cca-7538-1aec257dec16 Address:80ca3719-8c8d-3cca-7538-1aec257dec16}]"
2026-09-21T15:25:37.227Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:37.230Z [INFO]  server-12114bc6-9a08-4381-7281-bb42a545d943: entering follower state: follower="Node at 12114bc6-9a08-4381-7281-bb42a545d943 [Follower]" leader-address= leader-id=
2026-09-21T15:25:37.235Z [INFO]  server-a5de88fb-d0c5-b39d-7f0f-81323bc841f9: entering follower state: follower="Node at a5de88fb-d0c5-b39d-7f0f-81323bc841f9 [Follower]" leader-address= leader-id=
2026-09-21T15:25:37.235Z [INFO]  server-80ca3719-8c8d-3cca-7538-1aec257dec16: entering follower state: follower="Node at 80ca3719-8c8d-3cca-7538-1aec257dec16 [Follower]" leader-address= leader-id=
2026-09-21T15:25:37.855Z [WARN]  server-a5de88fb-d0c5-b39d-7f0f-81323bc841f9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:37.855Z [INFO]  server-a5de88fb-d0c5-b39d-7f0f-81323bc841f9: entering candidate state: node="Node at a5de88fb-d0c5-b39d-7f0f-81323bc841f9 [Candidate]" term=2
2026-09-21T15:25:37.856Z [INFO]  server-a5de88fb-d0c5-b39d-7f0f-81323bc841f9: election won: term=2 tally=2
2026-09-21T15:25:37.856Z [INFO]  server-a5de88fb-d0c5-b39d-7f0f-81323bc841f9: entering leader state: leader="Node at a5de88fb-d0c5-b39d-7f0f-81323bc841f9 [Leader]"
2026-09-21T15:25:37.856Z [INFO]  server-a5de88fb-d0c5-b39d-7f0f-81323bc841f9: added peer, starting replication: peer=server-12114bc6-9a08-4381-7281-bb42a545d943
2026-09-21T15:25:37.856Z [INFO]  server-a5de88fb-d0c5-b39d-7f0f-81323bc841f9: added peer, starting replication: peer=server-80ca3719-8c8d-3cca-7538-1aec257dec16
2026-09-21T15:25:37.856Z [INFO]  server-a5de88fb-d0c5-b39d-7f0f-81323bc841f9: pipelining replication: peer="{Voter server-80ca3719-8c8d-3cca-7538-1aec257dec16 80ca3719-8c8d-3cca-7538-1aec257dec16}"
2026-09-21T15:25:37.856Z [INFO]  server-a5de88fb-d0c5-b39d-7f0f-81323bc841f9: pipelining replication: peer="{Voter server-12114bc6-9a08-4381-7281-bb42a545d943 12114bc6-9a08-4381-7281-bb42a545d943}"
2026-09-21T15:25:38.861Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:37.856207588 +0000 UTC m=+89.797377721 (1 nodes), 628.63716ms from start of poll, 629.74128ms from cluster start. Timeout at 2026-09-21 15:25:38.861212588 +0000 UTC m=+90.802382721, 1.005005s after stability
2026-09-21T15:25:38.861Z [INFO]  server-a5de88fb-d0c5-b39d-7f0f-81323bc841f9: starting snapshot up to: index=12
2026-09-21T15:25:38.861Z [INFO]  creating new snapshot: path=/tmp/raft3474793236/snapshots/2-12-1790004338861.tmp
2026-09-21T15:25:38.862Z [INFO]  server-a5de88fb-d0c5-b39d-7f0f-81323bc841f9: snapshot complete up to: index=12
2026-09-21T15:25:38.862Z [INFO]  server-29f53828-6792-a8eb-d0d6-1710e5867d8d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29f53828-6792-a8eb-d0d6-1710e5867d8d Address:29f53828-6792-a8eb-d0d6-1710e5867d8d} {Suffrage:Voter ID:server-3e20cbef-1103-f763-5b5d-6d59c64ba7fe Address:3e20cbef-1103-f763-5b5d-6d59c64ba7fe} {Suffrage:Voter ID:server-b7c317ca-f19e-9c44-7d69-3ab3a8978238 Address:b7c317ca-f19e-9c44-7d69-3ab3a8978238}]"
2026-09-21T15:25:38.862Z [INFO]  server-3e20cbef-1103-f763-5b5d-6d59c64ba7fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29f53828-6792-a8eb-d0d6-1710e5867d8d Address:29f53828-6792-a8eb-d0d6-1710e5867d8d} {Suffrage:Voter ID:server-3e20cbef-1103-f763-5b5d-6d59c64ba7fe Address:3e20cbef-1103-f763-5b5d-6d59c64ba7fe} {Suffrage:Voter ID:server-b7c317ca-f19e-9c44-7d69-3ab3a8978238 Address:b7c317ca-f19e-9c44-7d69-3ab3a8978238}]"
2026-09-21T15:25:38.862Z [INFO]  server-b7c317ca-f19e-9c44-7d69-3ab3a8978238: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29f53828-6792-a8eb-d0d6-1710e5867d8d Address:29f53828-6792-a8eb-d0d6-1710e5867d8d} {Suffrage:Voter ID:server-3e20cbef-1103-f763-5b5d-6d59c64ba7fe Address:3e20cbef-1103-f763-5b5d-6d59c64ba7fe} {Suffrage:Voter ID:server-b7c317ca-f19e-9c44-7d69-3ab3a8978238 Address:b7c317ca-f19e-9c44-7d69-3ab3a8978238}]"
2026-09-21T15:25:38.862Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:38.862Z [INFO]  server-29f53828-6792-a8eb-d0d6-1710e5867d8d: entering follower state: follower="Node at 29f53828-6792-a8eb-d0d6-1710e5867d8d [Follower]" leader-address= leader-id=
2026-09-21T15:25:38.862Z [INFO]  server-3e20cbef-1103-f763-5b5d-6d59c64ba7fe: entering follower state: follower="Node at 3e20cbef-1103-f763-5b5d-6d59c64ba7fe [Follower]" leader-address= leader-id=
2026-09-21T15:25:38.862Z [INFO]  server-b7c317ca-f19e-9c44-7d69-3ab3a8978238: entering follower state: follower="Node at b7c317ca-f19e-9c44-7d69-3ab3a8978238 [Follower]" leader-address= leader-id=
2026-09-21T15:25:39.552Z [WARN]  server-29f53828-6792-a8eb-d0d6-1710e5867d8d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:39.552Z [INFO]  server-29f53828-6792-a8eb-d0d6-1710e5867d8d: entering candidate state: node="Node at 29f53828-6792-a8eb-d0d6-1710e5867d8d [Candidate]" term=2
2026-09-21T15:25:39.553Z [INFO]  server-29f53828-6792-a8eb-d0d6-1710e5867d8d: election won: term=2 tally=2
2026-09-21T15:25:39.553Z [INFO]  server-29f53828-6792-a8eb-d0d6-1710e5867d8d: entering leader state: leader="Node at 29f53828-6792-a8eb-d0d6-1710e5867d8d [Leader]"
2026-09-21T15:25:39.553Z [INFO]  server-29f53828-6792-a8eb-d0d6-1710e5867d8d: added peer, starting replication: peer=server-3e20cbef-1103-f763-5b5d-6d59c64ba7fe
2026-09-21T15:25:39.553Z [INFO]  server-29f53828-6792-a8eb-d0d6-1710e5867d8d: added peer, starting replication: peer=server-b7c317ca-f19e-9c44-7d69-3ab3a8978238
2026-09-21T15:25:39.553Z [INFO]  server-29f53828-6792-a8eb-d0d6-1710e5867d8d: pipelining replication: peer="{Voter server-3e20cbef-1103-f763-5b5d-6d59c64ba7fe 3e20cbef-1103-f763-5b5d-6d59c64ba7fe}"
2026-09-21T15:25:39.553Z [INFO]  server-29f53828-6792-a8eb-d0d6-1710e5867d8d: pipelining replication: peer="{Voter server-b7c317ca-f19e-9c44-7d69-3ab3a8978238 b7c317ca-f19e-9c44-7d69-3ab3a8978238}"
2026-09-21T15:25:40.559Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:39.553250988 +0000 UTC m=+91.494421121 (1 nodes), 690.5168ms from start of poll, 690.66968ms from cluster start. Timeout at 2026-09-21 15:25:40.558252228 +0000 UTC m=+92.499422361, 1.00500124s after stability
2026-09-21T15:25:40.559Z [INFO]  creating new snapshot: path=/tmp/raft443261910/snapshots/2-10013-1790004340559.tmp
2026-09-21T15:25:40.559Z [INFO]  server-29f53828-6792-a8eb-d0d6-1710e5867d8d: copied to local snapshot: bytes=71
2026-09-21T15:25:40.560Z [INFO]  server-29f53828-6792-a8eb-d0d6-1710e5867d8d: snapshot restore progress: id=2-10013-1790004340559 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:40.560Z [INFO]  server-29f53828-6792-a8eb-d0d6-1710e5867d8d: restored user snapshot: index=10013
2026-09-21T15:25:40.560Z [ERROR] server-29f53828-6792-a8eb-d0d6-1710e5867d8d: failed to get log: index=3 error="log not found"
2026-09-21T15:25:40.560Z [INFO]  server-29f53828-6792-a8eb-d0d6-1710e5867d8d: aborting pipeline replication: peer="{Voter server-b7c317ca-f19e-9c44-7d69-3ab3a8978238 b7c317ca-f19e-9c44-7d69-3ab3a8978238}"
2026-09-21T15:25:40.560Z [ERROR] server-29f53828-6792-a8eb-d0d6-1710e5867d8d: failed to get log: index=3 error="log not found"
2026-09-21T15:25:40.560Z [INFO]  server-29f53828-6792-a8eb-d0d6-1710e5867d8d: aborting pipeline replication: peer="{Voter server-3e20cbef-1103-f763-5b5d-6d59c64ba7fe 3e20cbef-1103-f763-5b5d-6d59c64ba7fe}"
2026-09-21T15:25:40.567Z [ERROR] server-29f53828-6792-a8eb-d0d6-1710e5867d8d: failed to get log: index=3 error="log not found"
2026-09-21T15:25:40.567Z [INFO]  creating new snapshot: path=/tmp/raft2134544345/snapshots/2-10013-1790004340567.tmp
2026-09-21T15:25:40.567Z [INFO]  server-3e20cbef-1103-f763-5b5d-6d59c64ba7fe: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:40.568Z [INFO]  server-3e20cbef-1103-f763-5b5d-6d59c64ba7fe: copied to local snapshot: bytes=71
2026-09-21T15:25:40.568Z [INFO]  server-3e20cbef-1103-f763-5b5d-6d59c64ba7fe: snapshot restore progress: id=2-10013-1790004340567 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:40.568Z [INFO]  server-3e20cbef-1103-f763-5b5d-6d59c64ba7fe: Installed remote snapshot
2026-09-21T15:25:40.568Z [INFO]  server-29f53828-6792-a8eb-d0d6-1710e5867d8d: pipelining replication: peer="{Voter server-3e20cbef-1103-f763-5b5d-6d59c64ba7fe 3e20cbef-1103-f763-5b5d-6d59c64ba7fe}"
2026-09-21T15:25:40.568Z [ERROR] server-29f53828-6792-a8eb-d0d6-1710e5867d8d: failed to get log: index=3 error="log not found"
2026-09-21T15:25:40.571Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:40.574Z [INFO]  creating new snapshot: path=/tmp/raft11004357/snapshots/2-10013-1790004340574.tmp
2026-09-21T15:25:40.574Z [INFO]  server-b7c317ca-f19e-9c44-7d69-3ab3a8978238: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:40.575Z [INFO]  server-b7c317ca-f19e-9c44-7d69-3ab3a8978238: copied to local snapshot: bytes=71
2026-09-21T15:25:40.575Z [INFO]  server-b7c317ca-f19e-9c44-7d69-3ab3a8978238: snapshot restore progress: id=2-10013-1790004340574 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:40.575Z [INFO]  server-b7c317ca-f19e-9c44-7d69-3ab3a8978238: Installed remote snapshot
2026-09-21T15:25:40.575Z [INFO]  server-29f53828-6792-a8eb-d0d6-1710e5867d8d: pipelining replication: peer="{Voter server-b7c317ca-f19e-9c44-7d69-3ab3a8978238 b7c317ca-f19e-9c44-7d69-3ab3a8978238}"
2026-09-21T15:25:41.577Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:40.571636108 +0000 UTC m=+92.512806241 (2 nodes), 920ns from start of poll, 1.7090548s from cluster start. Timeout at 2026-09-21 15:25:41.576669108 +0000 UTC m=+93.517839241, 1.005033s after stability
2026-09-21T15:25:41.589Z [INFO]  server-a5de88fb-d0c5-b39d-7f0f-81323bc841f9: aborting pipeline replication: peer="{Voter server-12114bc6-9a08-4381-7281-bb42a545d943 12114bc6-9a08-4381-7281-bb42a545d943}"
2026-09-21T15:25:41.589Z [INFO]  server-a5de88fb-d0c5-b39d-7f0f-81323bc841f9: aborting pipeline replication: peer="{Voter server-80ca3719-8c8d-3cca-7538-1aec257dec16 80ca3719-8c8d-3cca-7538-1aec257dec16}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true
2026-09-21T15:25:41.589Z [INFO]  server-0c80d9d2-4aea-4d5a-0897-fe8baad632ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c80d9d2-4aea-4d5a-0897-fe8baad632ec Address:0c80d9d2-4aea-4d5a-0897-fe8baad632ec} {Suffrage:Voter ID:server-b9b682e6-42cb-8fb9-a015-65aafd364ee1 Address:b9b682e6-42cb-8fb9-a015-65aafd364ee1} {Suffrage:Voter ID:server-f27ae759-bef1-c925-c961-8671268ee235 Address:f27ae759-bef1-c925-c961-8671268ee235}]"
2026-09-21T15:25:41.589Z [INFO]  server-b9b682e6-42cb-8fb9-a015-65aafd364ee1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c80d9d2-4aea-4d5a-0897-fe8baad632ec Address:0c80d9d2-4aea-4d5a-0897-fe8baad632ec} {Suffrage:Voter ID:server-b9b682e6-42cb-8fb9-a015-65aafd364ee1 Address:b9b682e6-42cb-8fb9-a015-65aafd364ee1} {Suffrage:Voter ID:server-f27ae759-bef1-c925-c961-8671268ee235 Address:f27ae759-bef1-c925-c961-8671268ee235}]"
2026-09-21T15:25:41.589Z [INFO]  server-f27ae759-bef1-c925-c961-8671268ee235: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c80d9d2-4aea-4d5a-0897-fe8baad632ec Address:0c80d9d2-4aea-4d5a-0897-fe8baad632ec} {Suffrage:Voter ID:server-b9b682e6-42cb-8fb9-a015-65aafd364ee1 Address:b9b682e6-42cb-8fb9-a015-65aafd364ee1} {Suffrage:Voter ID:server-f27ae759-bef1-c925-c961-8671268ee235 Address:f27ae759-bef1-c925-c961-8671268ee235}]"
2026-09-21T15:25:41.589Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:41.589Z [INFO]  server-0c80d9d2-4aea-4d5a-0897-fe8baad632ec: entering follower state: follower="Node at 0c80d9d2-4aea-4d5a-0897-fe8baad632ec [Follower]" leader-address= leader-id=
2026-09-21T15:25:41.589Z [INFO]  server-b9b682e6-42cb-8fb9-a015-65aafd364ee1: entering follower state: follower="Node at b9b682e6-42cb-8fb9-a015-65aafd364ee1 [Follower]" leader-address= leader-id=
2026-09-21T15:25:41.589Z [INFO]  server-f27ae759-bef1-c925-c961-8671268ee235: entering follower state: follower="Node at f27ae759-bef1-c925-c961-8671268ee235 [Follower]" leader-address= leader-id=
2026-09-21T15:25:42.332Z [WARN]  server-b9b682e6-42cb-8fb9-a015-65aafd364ee1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:42.332Z [INFO]  server-b9b682e6-42cb-8fb9-a015-65aafd364ee1: entering candidate state: node="Node at b9b682e6-42cb-8fb9-a015-65aafd364ee1 [Candidate]" term=2
2026-09-21T15:25:42.332Z [INFO]  server-b9b682e6-42cb-8fb9-a015-65aafd364ee1: election won: term=2 tally=2
2026-09-21T15:25:42.332Z [INFO]  server-b9b682e6-42cb-8fb9-a015-65aafd364ee1: entering leader state: leader="Node at b9b682e6-42cb-8fb9-a015-65aafd364ee1 [Leader]"
2026-09-21T15:25:42.332Z [INFO]  server-b9b682e6-42cb-8fb9-a015-65aafd364ee1: added peer, starting replication: peer=server-0c80d9d2-4aea-4d5a-0897-fe8baad632ec
2026-09-21T15:25:42.332Z [INFO]  server-b9b682e6-42cb-8fb9-a015-65aafd364ee1: added peer, starting replication: peer=server-f27ae759-bef1-c925-c961-8671268ee235
2026-09-21T15:25:42.332Z [INFO]  server-b9b682e6-42cb-8fb9-a015-65aafd364ee1: pipelining replication: peer="{Voter server-f27ae759-bef1-c925-c961-8671268ee235 f27ae759-bef1-c925-c961-8671268ee235}"
2026-09-21T15:25:42.332Z [INFO]  server-b9b682e6-42cb-8fb9-a015-65aafd364ee1: pipelining replication: peer="{Voter server-0c80d9d2-4aea-4d5a-0897-fe8baad632ec 0c80d9d2-4aea-4d5a-0897-fe8baad632ec}"
2026-09-21T15:25:43.338Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:42.333022548 +0000 UTC m=+94.274192681 (1 nodes), 743.17664ms from start of poll, 743.38476ms from cluster start. Timeout at 2026-09-21 15:25:43.338026468 +0000 UTC m=+95.279196601, 1.00500392s after stability
2026-09-21T15:25:43.338Z [INFO]  server-b9b682e6-42cb-8fb9-a015-65aafd364ee1: starting snapshot up to: index=12
2026-09-21T15:25:43.338Z [INFO]  creating new snapshot: path=/tmp/raft595517098/snapshots/2-12-1790004343338.tmp
2026-09-21T15:25:43.339Z [INFO]  server-b9b682e6-42cb-8fb9-a015-65aafd364ee1: snapshot complete up to: index=12
2026-09-21T15:25:43.339Z [INFO]  server-6becaf21-97a3-3fbc-7706-b765781ef009: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6becaf21-97a3-3fbc-7706-b765781ef009 Address:6becaf21-97a3-3fbc-7706-b765781ef009} {Suffrage:Voter ID:server-ab4b4de9-de40-d9f5-526e-37f4935c890a Address:ab4b4de9-de40-d9f5-526e-37f4935c890a} {Suffrage:Voter ID:server-cdc72680-9341-cc5b-ea0b-972e6ae9812d Address:cdc72680-9341-cc5b-ea0b-972e6ae9812d}]"
2026-09-21T15:25:43.339Z [INFO]  server-ab4b4de9-de40-d9f5-526e-37f4935c890a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6becaf21-97a3-3fbc-7706-b765781ef009 Address:6becaf21-97a3-3fbc-7706-b765781ef009} {Suffrage:Voter ID:server-ab4b4de9-de40-d9f5-526e-37f4935c890a Address:ab4b4de9-de40-d9f5-526e-37f4935c890a} {Suffrage:Voter ID:server-cdc72680-9341-cc5b-ea0b-972e6ae9812d Address:cdc72680-9341-cc5b-ea0b-972e6ae9812d}]"
2026-09-21T15:25:43.339Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6becaf21-97a3-3fbc-7706-b765781ef009 Address:6becaf21-97a3-3fbc-7706-b765781ef009} {Suffrage:Voter ID:server-ab4b4de9-de40-d9f5-526e-37f4935c890a Address:ab4b4de9-de40-d9f5-526e-37f4935c890a} {Suffrage:Voter ID:server-cdc72680-9341-cc5b-ea0b-972e6ae9812d Address:cdc72680-9341-cc5b-ea0b-972e6ae9812d}]"
2026-09-21T15:25:43.339Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:43.339Z [INFO]  server-6becaf21-97a3-3fbc-7706-b765781ef009: entering follower state: follower="Node at 6becaf21-97a3-3fbc-7706-b765781ef009 [Follower]" leader-address= leader-id=
2026-09-21T15:25:43.340Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: entering follower state: follower="Node at cdc72680-9341-cc5b-ea0b-972e6ae9812d [Follower]" leader-address= leader-id=
2026-09-21T15:25:43.340Z [INFO]  server-ab4b4de9-de40-d9f5-526e-37f4935c890a: entering follower state: follower="Node at ab4b4de9-de40-d9f5-526e-37f4935c890a [Follower]" leader-address= leader-id=
2026-09-21T15:25:44.019Z [WARN]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:44.019Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: entering candidate state: node="Node at cdc72680-9341-cc5b-ea0b-972e6ae9812d [Candidate]" term=2
2026-09-21T15:25:44.019Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: election won: term=2 tally=2
2026-09-21T15:25:44.019Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: entering leader state: leader="Node at cdc72680-9341-cc5b-ea0b-972e6ae9812d [Leader]"
2026-09-21T15:25:44.019Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: added peer, starting replication: peer=server-6becaf21-97a3-3fbc-7706-b765781ef009
2026-09-21T15:25:44.019Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: added peer, starting replication: peer=server-ab4b4de9-de40-d9f5-526e-37f4935c890a
2026-09-21T15:25:44.019Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: pipelining replication: peer="{Voter server-ab4b4de9-de40-d9f5-526e-37f4935c890a ab4b4de9-de40-d9f5-526e-37f4935c890a}"
2026-09-21T15:25:44.019Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: pipelining replication: peer="{Voter server-6becaf21-97a3-3fbc-7706-b765781ef009 6becaf21-97a3-3fbc-7706-b765781ef009}"
2026-09-21T15:25:45.025Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:44.019988468 +0000 UTC m=+95.961158601 (1 nodes), 680.06552ms from start of poll, 680.22308ms from cluster start. Timeout at 2026-09-21 15:25:45.024996868 +0000 UTC m=+96.966167001, 1.0050084s after stability
2026-09-21T15:25:45.026Z [INFO]  creating new snapshot: path=/tmp/raft340405828/snapshots/2-10013-1790004345026.tmp
2026-09-21T15:25:45.026Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: copied to local snapshot: bytes=71
2026-09-21T15:25:45.026Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: snapshot restore progress: id=2-10013-1790004345026 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:45.026Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: removing all old logs from log store
2026-09-21T15:25:45.026Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: compacting logs: from=1 to=2
2026-09-21T15:25:45.027Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: restored user snapshot: index=10013
2026-09-21T15:25:45.027Z [ERROR] server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: failed to get log: index=2 error="log not found"
2026-09-21T15:25:45.027Z [ERROR] server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: failed to get log: index=2 error="log not found"
2026-09-21T15:25:45.027Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: aborting pipeline replication: peer="{Voter server-ab4b4de9-de40-d9f5-526e-37f4935c890a ab4b4de9-de40-d9f5-526e-37f4935c890a}"
2026-09-21T15:25:45.027Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: aborting pipeline replication: peer="{Voter server-6becaf21-97a3-3fbc-7706-b765781ef009 6becaf21-97a3-3fbc-7706-b765781ef009}"
2026-09-21T15:25:45.033Z [ERROR] server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: failed to get log: index=2 error="log not found"
2026-09-21T15:25:45.034Z [INFO]  creating new snapshot: path=/tmp/raft1420426434/snapshots/2-10013-1790004345034.tmp
2026-09-21T15:25:45.034Z [ERROR] server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: failed to get log: index=2 error="log not found"
2026-09-21T15:25:45.034Z [INFO]  creating new snapshot: path=/tmp/raft613450551/snapshots/2-10013-1790004345034.tmp
2026-09-21T15:25:45.034Z [INFO]  server-6becaf21-97a3-3fbc-7706-b765781ef009: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:45.034Z [INFO]  server-6becaf21-97a3-3fbc-7706-b765781ef009: copied to local snapshot: bytes=71
2026-09-21T15:25:45.034Z [INFO]  server-6becaf21-97a3-3fbc-7706-b765781ef009: snapshot restore progress: id=2-10013-1790004345034 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:45.034Z [INFO]  server-6becaf21-97a3-3fbc-7706-b765781ef009: removing all old logs from log store
2026-09-21T15:25:45.034Z [INFO]  server-6becaf21-97a3-3fbc-7706-b765781ef009: compacting logs: from=1 to=2
2026-09-21T15:25:45.034Z [INFO]  server-6becaf21-97a3-3fbc-7706-b765781ef009: Installed remote snapshot
2026-09-21T15:25:45.034Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: pipelining replication: peer="{Voter server-6becaf21-97a3-3fbc-7706-b765781ef009 6becaf21-97a3-3fbc-7706-b765781ef009}"
2026-09-21T15:25:45.035Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:45.035Z [INFO]  server-ab4b4de9-de40-d9f5-526e-37f4935c890a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:45.035Z [INFO]  server-ab4b4de9-de40-d9f5-526e-37f4935c890a: copied to local snapshot: bytes=71
2026-09-21T15:25:45.035Z [INFO]  server-ab4b4de9-de40-d9f5-526e-37f4935c890a: snapshot restore progress: id=2-10013-1790004345034 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-09-21T15:25:45.035Z [INFO]  server-ab4b4de9-de40-d9f5-526e-37f4935c890a: removing all old logs from log store
2026-09-21T15:25:45.035Z [INFO]  server-ab4b4de9-de40-d9f5-526e-37f4935c890a: compacting logs: from=1 to=2
2026-09-21T15:25:45.035Z [INFO]  server-ab4b4de9-de40-d9f5-526e-37f4935c890a: Installed remote snapshot
2026-09-21T15:25:45.036Z [INFO]  server-cdc72680-9341-cc5b-ea0b-972e6ae9812d: pipelining replication: peer="{Voter server-ab4b4de9-de40-d9f5-526e-37f4935c890a ab4b4de9-de40-d9f5-526e-37f4935c890a}"
2026-09-21T15:25:46.040Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:45.035022868 +0000 UTC m=+96.976193001 (2 nodes), 480ns from start of poll, 1.69525748s from cluster start. Timeout at 2026-09-21 15:25:46.040041828 +0000 UTC m=+97.981212001, 1.005019s after stability
2026-09-21T15:25:46.052Z [INFO]  server-b9b682e6-42cb-8fb9-a015-65aafd364ee1: aborting pipeline replication: peer="{Voter server-f27ae759-bef1-c925-c961-8671268ee235 f27ae759-bef1-c925-c961-8671268ee235}"
2026-09-21T15:25:46.052Z [INFO]  server-b9b682e6-42cb-8fb9-a015-65aafd364ee1: aborting pipeline replication: peer="{Voter server-0c80d9d2-4aea-4d5a-0897-fe8baad632ec 0c80d9d2-4aea-4d5a-0897-fe8baad632ec}"
--- PASS: TestRaft_UserRestore (83.81s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.57s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.86s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.29s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.15s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.56s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.86s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.28s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.27s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.57s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.67s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.29s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.28s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.60s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.66s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.25s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.43s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.69s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.86s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.33s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.17s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.64s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.69s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.36s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.46s)
=== RUN   TestRaft_SendSnapshotFollower
2026-09-21T15:25:46.053Z [INFO]  server-9404f5dd-cc2e-638e-c006-70e060100387: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9404f5dd-cc2e-638e-c006-70e060100387 Address:9404f5dd-cc2e-638e-c006-70e060100387} {Suffrage:Voter ID:server-0d6f1627-e828-c564-4096-b32efc5c212d Address:0d6f1627-e828-c564-4096-b32efc5c212d} {Suffrage:Voter ID:server-52401a28-eae3-c0dc-1296-a52dd08197c5 Address:52401a28-eae3-c0dc-1296-a52dd08197c5}]"
2026-09-21T15:25:46.053Z [INFO]  server-0d6f1627-e828-c564-4096-b32efc5c212d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9404f5dd-cc2e-638e-c006-70e060100387 Address:9404f5dd-cc2e-638e-c006-70e060100387} {Suffrage:Voter ID:server-0d6f1627-e828-c564-4096-b32efc5c212d Address:0d6f1627-e828-c564-4096-b32efc5c212d} {Suffrage:Voter ID:server-52401a28-eae3-c0dc-1296-a52dd08197c5 Address:52401a28-eae3-c0dc-1296-a52dd08197c5}]"
2026-09-21T15:25:46.053Z [INFO]  server-52401a28-eae3-c0dc-1296-a52dd08197c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9404f5dd-cc2e-638e-c006-70e060100387 Address:9404f5dd-cc2e-638e-c006-70e060100387} {Suffrage:Voter ID:server-0d6f1627-e828-c564-4096-b32efc5c212d Address:0d6f1627-e828-c564-4096-b32efc5c212d} {Suffrage:Voter ID:server-52401a28-eae3-c0dc-1296-a52dd08197c5 Address:52401a28-eae3-c0dc-1296-a52dd08197c5}]"
2026-09-21T15:25:46.053Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:46.054Z [INFO]  server-9404f5dd-cc2e-638e-c006-70e060100387: entering follower state: follower="Node at 9404f5dd-cc2e-638e-c006-70e060100387 [Follower]" leader-address= leader-id=
2026-09-21T15:25:46.054Z [INFO]  server-0d6f1627-e828-c564-4096-b32efc5c212d: entering follower state: follower="Node at 0d6f1627-e828-c564-4096-b32efc5c212d [Follower]" leader-address= leader-id=
2026-09-21T15:25:46.054Z [INFO]  server-52401a28-eae3-c0dc-1296-a52dd08197c5: entering follower state: follower="Node at 52401a28-eae3-c0dc-1296-a52dd08197c5 [Follower]" leader-address= leader-id=
2026-09-21T15:25:46.132Z [WARN]  server-9404f5dd-cc2e-638e-c006-70e060100387: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:46.133Z [INFO]  server-9404f5dd-cc2e-638e-c006-70e060100387: entering candidate state: node="Node at 9404f5dd-cc2e-638e-c006-70e060100387 [Candidate]" term=2
2026-09-21T15:25:46.133Z [INFO]  server-9404f5dd-cc2e-638e-c006-70e060100387: election won: term=2 tally=2
2026-09-21T15:25:46.133Z [INFO]  server-9404f5dd-cc2e-638e-c006-70e060100387: entering leader state: leader="Node at 9404f5dd-cc2e-638e-c006-70e060100387 [Leader]"
2026-09-21T15:25:46.133Z [INFO]  server-9404f5dd-cc2e-638e-c006-70e060100387: added peer, starting replication: peer=server-0d6f1627-e828-c564-4096-b32efc5c212d
2026-09-21T15:25:46.133Z [INFO]  server-9404f5dd-cc2e-638e-c006-70e060100387: added peer, starting replication: peer=server-52401a28-eae3-c0dc-1296-a52dd08197c5
2026-09-21T15:25:46.134Z [INFO]  server-9404f5dd-cc2e-638e-c006-70e060100387: pipelining replication: peer="{Voter server-52401a28-eae3-c0dc-1296-a52dd08197c5 52401a28-eae3-c0dc-1296-a52dd08197c5}"
2026-09-21T15:25:46.133Z [INFO]  server-9404f5dd-cc2e-638e-c006-70e060100387: pipelining replication: peer="{Voter server-0d6f1627-e828-c564-4096-b32efc5c212d 0d6f1627-e828-c564-4096-b32efc5c212d}"
2026-09-21T15:25:46.239Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:46.133616108 +0000 UTC m=+98.074786281 (2 nodes), 79.61476ms from start of poll, 79.9936ms from cluster start. Timeout at 2026-09-21 15:25:46.238619468 +0000 UTC m=+98.179789601, 105.00332ms after stability
2026-09-21T15:25:46.239Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:46.345Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:46.239766548 +0000 UTC m=+98.180936721 (1 nodes), 2.36µs from start of poll, 186.14404ms from cluster start. Timeout at 2026-09-21 15:25:46.344802908 +0000 UTC m=+98.285973001, 105.03628ms after stability
2026-09-21T15:25:46.345Z [ERROR] server-9404f5dd-cc2e-638e-c006-70e060100387: failed to pipeline appendEntries: peer="{Voter server-0d6f1627-e828-c564-4096-b32efc5c212d 0d6f1627-e828-c564-4096-b32efc5c212d}" error="append pipeline closed"
2026-09-21T15:25:46.346Z [INFO]  server-9404f5dd-cc2e-638e-c006-70e060100387: aborting pipeline replication: peer="{Voter server-0d6f1627-e828-c564-4096-b32efc5c212d 0d6f1627-e828-c564-4096-b32efc5c212d}"
2026-09-21T15:25:46.346Z [ERROR] server-9404f5dd-cc2e-638e-c006-70e060100387: failed to appendEntries to: peer="{Voter server-0d6f1627-e828-c564-4096-b32efc5c212d 0d6f1627-e828-c564-4096-b32efc5c212d}" error="failed to connect to peer: 0d6f1627-e828-c564-4096-b32efc5c212d"
    raft_test.go:1627: [INFO] Finished apply without behind follower
2026-09-21T15:25:46.348Z [INFO]  server-9404f5dd-cc2e-638e-c006-70e060100387: starting snapshot up to: index=102
2026-09-21T15:25:46.348Z [INFO]  creating new snapshot: path=/tmp/raft1093453538/snapshots/2-102-1790004346348.tmp
2026-09-21T15:25:46.348Z [INFO]  server-9404f5dd-cc2e-638e-c006-70e060100387: compacting logs: from=1 to=92
2026-09-21T15:25:46.349Z [INFO]  server-9404f5dd-cc2e-638e-c006-70e060100387: snapshot complete up to: index=102
2026-09-21T15:25:46.349Z [ERROR] server-0d6f1627-e828-c564-4096-b32efc5c212d: failed to take snapshot: error="nothing new to snapshot"
2026-09-21T15:25:46.349Z [INFO]  server-52401a28-eae3-c0dc-1296-a52dd08197c5: starting snapshot up to: index=96
2026-09-21T15:25:46.349Z [INFO]  creating new snapshot: path=/tmp/raft1210737392/snapshots/2-96-1790004346349.tmp
2026-09-21T15:25:46.349Z [INFO]  server-52401a28-eae3-c0dc-1296-a52dd08197c5: compacting logs: from=1 to=92
2026-09-21T15:25:46.349Z [INFO]  server-52401a28-eae3-c0dc-1296-a52dd08197c5: snapshot complete up to: index=96
2026-09-21T15:25:46.356Z [ERROR] server-9404f5dd-cc2e-638e-c006-70e060100387: failed to get log: index=2 error="log not found"
2026-09-21T15:25:46.357Z [INFO]  creating new snapshot: path=/tmp/raft2979306343/snapshots/2-102-1790004346357.tmp
2026-09-21T15:25:46.357Z [INFO]  server-0d6f1627-e828-c564-4096-b32efc5c212d: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2026-09-21T15:25:46.357Z [INFO]  server-0d6f1627-e828-c564-4096-b32efc5c212d: copied to local snapshot: bytes=693
2026-09-21T15:25:46.357Z [INFO]  server-0d6f1627-e828-c564-4096-b32efc5c212d: snapshot restore progress: id=2-102-1790004346357 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-09-21T15:25:46.357Z [INFO]  server-0d6f1627-e828-c564-4096-b32efc5c212d: Installed remote snapshot
2026-09-21T15:25:46.358Z [WARN]  server-0d6f1627-e828-c564-4096-b32efc5c212d: skipping application of old log: index=102
2026-09-21T15:25:46.358Z [INFO]  server-9404f5dd-cc2e-638e-c006-70e060100387: pipelining replication: peer="{Voter server-0d6f1627-e828-c564-4096-b32efc5c212d 0d6f1627-e828-c564-4096-b32efc5c212d}"
2026-09-21T15:25:46.360Z [INFO]  server-9404f5dd-cc2e-638e-c006-70e060100387: aborting pipeline replication: peer="{Voter server-52401a28-eae3-c0dc-1296-a52dd08197c5 52401a28-eae3-c0dc-1296-a52dd08197c5}"
2026-09-21T15:25:46.360Z [INFO]  server-9404f5dd-cc2e-638e-c006-70e060100387: aborting pipeline replication: peer="{Voter server-0d6f1627-e828-c564-4096-b32efc5c212d 0d6f1627-e828-c564-4096-b32efc5c212d}"
--- PASS: TestRaft_SendSnapshotFollower (0.31s)
=== RUN   TestRaft_SendSnapshotAndLogsFollower
2026-09-21T15:25:46.361Z [INFO]  server-992759f1-c476-6e17-a583-0fa79f423bb0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-992759f1-c476-6e17-a583-0fa79f423bb0 Address:992759f1-c476-6e17-a583-0fa79f423bb0} {Suffrage:Voter ID:server-24cb4ff5-df04-7d3a-e4e1-3a8c1a2d830a Address:24cb4ff5-df04-7d3a-e4e1-3a8c1a2d830a} {Suffrage:Voter ID:server-341cd4f9-b83a-72f5-27dd-f34b10bd102b Address:341cd4f9-b83a-72f5-27dd-f34b10bd102b}]"
2026-09-21T15:25:46.361Z [INFO]  server-24cb4ff5-df04-7d3a-e4e1-3a8c1a2d830a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-992759f1-c476-6e17-a583-0fa79f423bb0 Address:992759f1-c476-6e17-a583-0fa79f423bb0} {Suffrage:Voter ID:server-24cb4ff5-df04-7d3a-e4e1-3a8c1a2d830a Address:24cb4ff5-df04-7d3a-e4e1-3a8c1a2d830a} {Suffrage:Voter ID:server-341cd4f9-b83a-72f5-27dd-f34b10bd102b Address:341cd4f9-b83a-72f5-27dd-f34b10bd102b}]"
2026-09-21T15:25:46.361Z [INFO]  server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-992759f1-c476-6e17-a583-0fa79f423bb0 Address:992759f1-c476-6e17-a583-0fa79f423bb0} {Suffrage:Voter ID:server-24cb4ff5-df04-7d3a-e4e1-3a8c1a2d830a Address:24cb4ff5-df04-7d3a-e4e1-3a8c1a2d830a} {Suffrage:Voter ID:server-341cd4f9-b83a-72f5-27dd-f34b10bd102b Address:341cd4f9-b83a-72f5-27dd-f34b10bd102b}]"
2026-09-21T15:25:46.361Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:46.361Z [INFO]  server-992759f1-c476-6e17-a583-0fa79f423bb0: entering follower state: follower="Node at 992759f1-c476-6e17-a583-0fa79f423bb0 [Follower]" leader-address= leader-id=
2026-09-21T15:25:46.361Z [INFO]  server-24cb4ff5-df04-7d3a-e4e1-3a8c1a2d830a: entering follower state: follower="Node at 24cb4ff5-df04-7d3a-e4e1-3a8c1a2d830a [Follower]" leader-address= leader-id=
2026-09-21T15:25:46.361Z [INFO]  server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: entering follower state: follower="Node at 341cd4f9-b83a-72f5-27dd-f34b10bd102b [Follower]" leader-address= leader-id=
2026-09-21T15:25:46.416Z [WARN]  server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:46.416Z [INFO]  server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: entering candidate state: node="Node at 341cd4f9-b83a-72f5-27dd-f34b10bd102b [Candidate]" term=2
2026-09-21T15:25:46.416Z [INFO]  server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: election won: term=2 tally=2
2026-09-21T15:25:46.416Z [INFO]  server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: entering leader state: leader="Node at 341cd4f9-b83a-72f5-27dd-f34b10bd102b [Leader]"
2026-09-21T15:25:46.416Z [INFO]  server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: added peer, starting replication: peer=server-992759f1-c476-6e17-a583-0fa79f423bb0
2026-09-21T15:25:46.416Z [INFO]  server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: added peer, starting replication: peer=server-24cb4ff5-df04-7d3a-e4e1-3a8c1a2d830a
2026-09-21T15:25:46.417Z [INFO]  server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: pipelining replication: peer="{Voter server-24cb4ff5-df04-7d3a-e4e1-3a8c1a2d830a 24cb4ff5-df04-7d3a-e4e1-3a8c1a2d830a}"
2026-09-21T15:25:46.417Z [INFO]  server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: pipelining replication: peer="{Voter server-992759f1-c476-6e17-a583-0fa79f423bb0 992759f1-c476-6e17-a583-0fa79f423bb0}"
2026-09-21T15:25:46.522Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:46.417381788 +0000 UTC m=+98.358551921 (2 nodes), 55.84552ms from start of poll, 56.01976ms from cluster start. Timeout at 2026-09-21 15:25:46.522384668 +0000 UTC m=+98.463554841, 105.00292ms after stability
2026-09-21T15:25:46.522Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:46.627Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:46.522579148 +0000 UTC m=+98.463749361 (1 nodes), 2.16µs from start of poll, 161.2172ms from cluster start. Timeout at 2026-09-21 15:25:46.627613588 +0000 UTC m=+98.568783761, 105.0344ms after stability
2026-09-21T15:25:46.627Z [ERROR] server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: failed to pipeline appendEntries: peer="{Voter server-992759f1-c476-6e17-a583-0fa79f423bb0 992759f1-c476-6e17-a583-0fa79f423bb0}" error="append pipeline closed"
2026-09-21T15:25:46.627Z [INFO]  server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: aborting pipeline replication: peer="{Voter server-992759f1-c476-6e17-a583-0fa79f423bb0 992759f1-c476-6e17-a583-0fa79f423bb0}"
2026-09-21T15:25:46.627Z [ERROR] server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: failed to appendEntries to: peer="{Voter server-992759f1-c476-6e17-a583-0fa79f423bb0 992759f1-c476-6e17-a583-0fa79f423bb0}" error="failed to connect to peer: 992759f1-c476-6e17-a583-0fa79f423bb0"
2026-09-21T15:25:46.628Z [ERROR] server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: failed to heartbeat to: peer=992759f1-c476-6e17-a583-0fa79f423bb0 backoff time=10ms error="failed to connect to peer: 992759f1-c476-6e17-a583-0fa79f423bb0"
    raft_test.go:1669: [INFO] Finished apply without behind follower
2026-09-21T15:25:46.637Z [ERROR] server-992759f1-c476-6e17-a583-0fa79f423bb0: failed to take snapshot: error="nothing new to snapshot"
2026-09-21T15:25:46.637Z [INFO]  server-24cb4ff5-df04-7d3a-e4e1-3a8c1a2d830a: starting snapshot up to: index=16
2026-09-21T15:25:46.637Z [INFO]  creating new snapshot: path=/tmp/raft3526662441/snapshots/2-16-1790004346637.tmp
2026-09-21T15:25:46.637Z [INFO]  server-24cb4ff5-df04-7d3a-e4e1-3a8c1a2d830a: compacting logs: from=1 to=16
2026-09-21T15:25:46.637Z [INFO]  server-24cb4ff5-df04-7d3a-e4e1-3a8c1a2d830a: snapshot complete up to: index=16
2026-09-21T15:25:46.637Z [INFO]  server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: starting snapshot up to: index=102
2026-09-21T15:25:46.637Z [INFO]  creating new snapshot: path=/tmp/raft2896256223/snapshots/2-102-1790004346637.tmp
2026-09-21T15:25:46.638Z [INFO]  server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: compacting logs: from=1 to=92
2026-09-21T15:25:46.638Z [INFO]  server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: snapshot complete up to: index=102
2026-09-21T15:25:46.638Z [ERROR] server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: failed to get log: index=2 error="log not found"
2026-09-21T15:25:46.639Z [ERROR] server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: failed to install snapshot: id=2-102-1790004346637 error="failed to connect to peer: 992759f1-c476-6e17-a583-0fa79f423bb0"
2026-09-21T15:25:46.639Z [ERROR] server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: failed to send snapshot to: peer="{Voter server-992759f1-c476-6e17-a583-0fa79f423bb0 992759f1-c476-6e17-a583-0fa79f423bb0}" error="failed to connect to peer: 992759f1-c476-6e17-a583-0fa79f423bb0"
    raft_test.go:1690: [INFO] Finished apply without behind follower
2026-09-21T15:25:46.650Z [ERROR] server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: failed to get log: index=2 error="log not found"
2026-09-21T15:25:46.650Z [INFO]  creating new snapshot: path=/tmp/raft1196826838/snapshots/2-102-1790004346650.tmp
2026-09-21T15:25:46.651Z [INFO]  server-992759f1-c476-6e17-a583-0fa79f423bb0: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2026-09-21T15:25:46.651Z [INFO]  server-992759f1-c476-6e17-a583-0fa79f423bb0: copied to local snapshot: bytes=693
2026-09-21T15:25:46.651Z [INFO]  server-992759f1-c476-6e17-a583-0fa79f423bb0: snapshot restore progress: id=2-102-1790004346650 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-09-21T15:25:46.651Z [INFO]  server-992759f1-c476-6e17-a583-0fa79f423bb0: Installed remote snapshot
2026-09-21T15:25:46.651Z [INFO]  server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: pipelining replication: peer="{Voter server-992759f1-c476-6e17-a583-0fa79f423bb0 992759f1-c476-6e17-a583-0fa79f423bb0}"
2026-09-21T15:25:46.656Z [INFO]  server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: aborting pipeline replication: peer="{Voter server-24cb4ff5-df04-7d3a-e4e1-3a8c1a2d830a 24cb4ff5-df04-7d3a-e4e1-3a8c1a2d830a}"
2026-09-21T15:25:46.656Z [INFO]  server-341cd4f9-b83a-72f5-27dd-f34b10bd102b: aborting pipeline replication: peer="{Voter server-992759f1-c476-6e17-a583-0fa79f423bb0 992759f1-c476-6e17-a583-0fa79f423bb0}"
--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.30s)
=== RUN   TestRaft_ReJoinFollower
2026-09-21T15:25:46.658Z [INFO]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8 Address:d45f0c1e-55ae-0811-900d-b34c3d0ac4e8} {Suffrage:Voter ID:server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174 Address:5d63277e-de57-f8aa-a3c0-ec95eb5bf174} {Suffrage:Voter ID:server-f7016c22-2402-0ed2-044e-f44d773683ec Address:f7016c22-2402-0ed2-044e-f44d773683ec}]"
2026-09-21T15:25:46.658Z [INFO]  server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8 Address:d45f0c1e-55ae-0811-900d-b34c3d0ac4e8} {Suffrage:Voter ID:server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174 Address:5d63277e-de57-f8aa-a3c0-ec95eb5bf174} {Suffrage:Voter ID:server-f7016c22-2402-0ed2-044e-f44d773683ec Address:f7016c22-2402-0ed2-044e-f44d773683ec}]"
2026-09-21T15:25:46.658Z [INFO]  server-f7016c22-2402-0ed2-044e-f44d773683ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8 Address:d45f0c1e-55ae-0811-900d-b34c3d0ac4e8} {Suffrage:Voter ID:server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174 Address:5d63277e-de57-f8aa-a3c0-ec95eb5bf174} {Suffrage:Voter ID:server-f7016c22-2402-0ed2-044e-f44d773683ec Address:f7016c22-2402-0ed2-044e-f44d773683ec}]"
2026-09-21T15:25:46.658Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:46.658Z [INFO]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: entering follower state: follower="Node at d45f0c1e-55ae-0811-900d-b34c3d0ac4e8 [Follower]" leader-address= leader-id=
2026-09-21T15:25:46.658Z [INFO]  server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174: entering follower state: follower="Node at 5d63277e-de57-f8aa-a3c0-ec95eb5bf174 [Follower]" leader-address= leader-id=
2026-09-21T15:25:46.658Z [INFO]  server-f7016c22-2402-0ed2-044e-f44d773683ec: entering follower state: follower="Node at f7016c22-2402-0ed2-044e-f44d773683ec [Follower]" leader-address= leader-id=
2026-09-21T15:25:46.712Z [WARN]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:46.712Z [INFO]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: entering candidate state: node="Node at d45f0c1e-55ae-0811-900d-b34c3d0ac4e8 [Candidate]" term=2
2026-09-21T15:25:46.712Z [INFO]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: election won: term=2 tally=2
2026-09-21T15:25:46.712Z [INFO]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: entering leader state: leader="Node at d45f0c1e-55ae-0811-900d-b34c3d0ac4e8 [Leader]"
2026-09-21T15:25:46.712Z [INFO]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: added peer, starting replication: peer=server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174
2026-09-21T15:25:46.712Z [INFO]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: added peer, starting replication: peer=server-f7016c22-2402-0ed2-044e-f44d773683ec
2026-09-21T15:25:46.712Z [INFO]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: pipelining replication: peer="{Voter server-f7016c22-2402-0ed2-044e-f44d773683ec f7016c22-2402-0ed2-044e-f44d773683ec}"
2026-09-21T15:25:46.712Z [INFO]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: pipelining replication: peer="{Voter server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174 5d63277e-de57-f8aa-a3c0-ec95eb5bf174}"
2026-09-21T15:25:46.817Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:46.712381908 +0000 UTC m=+98.653552001 (1 nodes), 54.02132ms from start of poll, 54.40556ms from cluster start. Timeout at 2026-09-21 15:25:46.817388468 +0000 UTC m=+98.758558641, 105.00664ms after stability
2026-09-21T15:25:46.823Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:46.928Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:46.823249628 +0000 UTC m=+98.764419801 (2 nodes), 2.44µs from start of poll, 165.27336ms from cluster start. Timeout at 2026-09-21 15:25:46.928280548 +0000 UTC m=+98.869450721, 105.03092ms after stability
2026-09-21T15:25:46.928Z [INFO]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: updating configuration: command=RemoveServer server-id=server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174 server-addr= servers="[{Suffrage:Voter ID:server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8 Address:d45f0c1e-55ae-0811-900d-b34c3d0ac4e8} {Suffrage:Voter ID:server-f7016c22-2402-0ed2-044e-f44d773683ec Address:f7016c22-2402-0ed2-044e-f44d773683ec}]"
2026-09-21T15:25:46.928Z [INFO]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: removed peer, stopping replication: peer=server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174 last-index=3
2026-09-21T15:25:46.928Z [INFO]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: aborting pipeline replication: peer="{Voter server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174 5d63277e-de57-f8aa-a3c0-ec95eb5bf174}"
2026-09-21T15:25:46.997Z [WARN]  server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2026-09-21T15:25:47.033Z [INFO]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: updating configuration: command=AddVoter server-id=server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174 server-addr=5d63277e-de57-f8aa-a3c0-ec95eb5bf174 servers="[{Suffrage:Voter ID:server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8 Address:d45f0c1e-55ae-0811-900d-b34c3d0ac4e8} {Suffrage:Voter ID:server-f7016c22-2402-0ed2-044e-f44d773683ec Address:f7016c22-2402-0ed2-044e-f44d773683ec} {Suffrage:Voter ID:server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174 Address:5d63277e-de57-f8aa-a3c0-ec95eb5bf174}]"
2026-09-21T15:25:47.033Z [INFO]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: added peer, starting replication: peer=server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174
2026-09-21T15:25:47.033Z [WARN]  server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174: failed to get previous log: previous-index=4 last-index=3 error="log not found"
2026-09-21T15:25:47.033Z [WARN]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: appendEntries rejected, sending older logs: peer="{Voter server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174 5d63277e-de57-f8aa-a3c0-ec95eb5bf174}" next=4
2026-09-21T15:25:47.034Z [INFO]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: pipelining replication: peer="{Voter server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174 5d63277e-de57-f8aa-a3c0-ec95eb5bf174}"
2026-09-21T15:25:47.034Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:47.140Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:47.034149148 +0000 UTC m=+98.975319321 (1 nodes), 600ns from start of poll, 376.17288ms from cluster start. Timeout at 2026-09-21 15:25:47.139184068 +0000 UTC m=+99.080354241, 105.03492ms after stability
2026-09-21T15:25:47.140Z [INFO]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: aborting pipeline replication: peer="{Voter server-f7016c22-2402-0ed2-044e-f44d773683ec f7016c22-2402-0ed2-044e-f44d773683ec}"
2026-09-21T15:25:47.140Z [INFO]  server-d45f0c1e-55ae-0811-900d-b34c3d0ac4e8: aborting pipeline replication: peer="{Voter server-5d63277e-de57-f8aa-a3c0-ec95eb5bf174 5d63277e-de57-f8aa-a3c0-ec95eb5bf174}"
--- PASS: TestRaft_ReJoinFollower (0.48s)
=== RUN   TestRaft_LeaderLeaseExpire
2026-09-21T15:25:47.141Z [INFO]  server-e7591341-d516-0a4a-ab8c-232355678a6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7591341-d516-0a4a-ab8c-232355678a6c Address:e7591341-d516-0a4a-ab8c-232355678a6c} {Suffrage:Voter ID:server-cf069b82-b031-1dbb-5e19-09788dc4013b Address:cf069b82-b031-1dbb-5e19-09788dc4013b}]"
2026-09-21T15:25:47.141Z [INFO]  server-e7591341-d516-0a4a-ab8c-232355678a6c: entering follower state: follower="Node at e7591341-d516-0a4a-ab8c-232355678a6c [Follower]" leader-address= leader-id=
2026-09-21T15:25:47.141Z [INFO]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7591341-d516-0a4a-ab8c-232355678a6c Address:e7591341-d516-0a4a-ab8c-232355678a6c} {Suffrage:Voter ID:server-cf069b82-b031-1dbb-5e19-09788dc4013b Address:cf069b82-b031-1dbb-5e19-09788dc4013b}]"
2026-09-21T15:25:47.141Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:47.141Z [INFO]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: entering follower state: follower="Node at cf069b82-b031-1dbb-5e19-09788dc4013b [Follower]" leader-address= leader-id=
2026-09-21T15:25:47.218Z [WARN]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:47.218Z [INFO]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: entering candidate state: node="Node at cf069b82-b031-1dbb-5e19-09788dc4013b [Candidate]" term=2
2026-09-21T15:25:47.218Z [INFO]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: election won: term=2 tally=2
2026-09-21T15:25:47.218Z [INFO]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: entering leader state: leader="Node at cf069b82-b031-1dbb-5e19-09788dc4013b [Leader]"
2026-09-21T15:25:47.218Z [INFO]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: added peer, starting replication: peer=server-e7591341-d516-0a4a-ab8c-232355678a6c
2026-09-21T15:25:47.218Z [INFO]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: pipelining replication: peer="{Voter server-e7591341-d516-0a4a-ab8c-232355678a6c e7591341-d516-0a4a-ab8c-232355678a6c}"
2026-09-21T15:25:47.324Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:47.218637828 +0000 UTC m=+99.159807961 (1 nodes), 76.7708ms from start of poll, 77.0728ms from cluster start. Timeout at 2026-09-21 15:25:47.323640948 +0000 UTC m=+99.264811121, 105.00316ms after stability
2026-09-21T15:25:47.331Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:47.437Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:47.331712788 +0000 UTC m=+99.272882961 (1 nodes), 1.08µs from start of poll, 190.1478ms from cluster start. Timeout at 2026-09-21 15:25:47.436761028 +0000 UTC m=+99.377931201, 105.04824ms after stability
    raft_test.go:1799: [INFO] Disconnecting Node at e7591341-d516-0a4a-ab8c-232355678a6c [Follower]
2026-09-21T15:25:47.439Z [ERROR] server-cf069b82-b031-1dbb-5e19-09788dc4013b: failed to pipeline appendEntries: peer="{Voter server-e7591341-d516-0a4a-ab8c-232355678a6c e7591341-d516-0a4a-ab8c-232355678a6c}" error="append pipeline closed"
2026-09-21T15:25:47.439Z [INFO]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: aborting pipeline replication: peer="{Voter server-e7591341-d516-0a4a-ab8c-232355678a6c e7591341-d516-0a4a-ab8c-232355678a6c}"
2026-09-21T15:25:47.444Z [ERROR] server-cf069b82-b031-1dbb-5e19-09788dc4013b: failed to appendEntries to: peer="{Voter server-e7591341-d516-0a4a-ab8c-232355678a6c e7591341-d516-0a4a-ab8c-232355678a6c}" error="failed to connect to peer: e7591341-d516-0a4a-ab8c-232355678a6c"
2026-09-21T15:25:47.445Z [ERROR] server-cf069b82-b031-1dbb-5e19-09788dc4013b: failed to heartbeat to: peer=e7591341-d516-0a4a-ab8c-232355678a6c backoff time=10ms error="failed to connect to peer: e7591341-d516-0a4a-ab8c-232355678a6c"
2026-09-21T15:25:47.462Z [ERROR] server-cf069b82-b031-1dbb-5e19-09788dc4013b: failed to appendEntries to: peer="{Voter server-e7591341-d516-0a4a-ab8c-232355678a6c e7591341-d516-0a4a-ab8c-232355678a6c}" error="failed to connect to peer: e7591341-d516-0a4a-ab8c-232355678a6c"
2026-09-21T15:25:47.465Z [ERROR] server-cf069b82-b031-1dbb-5e19-09788dc4013b: failed to heartbeat to: peer=e7591341-d516-0a4a-ab8c-232355678a6c backoff time=10ms error="failed to connect to peer: e7591341-d516-0a4a-ab8c-232355678a6c"
2026-09-21T15:25:47.479Z [ERROR] server-cf069b82-b031-1dbb-5e19-09788dc4013b: failed to appendEntries to: peer="{Voter server-e7591341-d516-0a4a-ab8c-232355678a6c e7591341-d516-0a4a-ab8c-232355678a6c}" error="failed to connect to peer: e7591341-d516-0a4a-ab8c-232355678a6c"
2026-09-21T15:25:47.485Z [ERROR] server-cf069b82-b031-1dbb-5e19-09788dc4013b: failed to heartbeat to: peer=e7591341-d516-0a4a-ab8c-232355678a6c backoff time=10ms error="failed to connect to peer: e7591341-d516-0a4a-ab8c-232355678a6c"
2026-09-21T15:25:47.486Z [WARN]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: failed to contact: server-id=server-e7591341-d516-0a4a-ab8c-232355678a6c time=50.48884ms
2026-09-21T15:25:47.486Z [WARN]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: failed to contact quorum of nodes, stepping down
2026-09-21T15:25:47.486Z [INFO]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: entering follower state: follower="Node at cf069b82-b031-1dbb-5e19-09788dc4013b [Follower]" leader-address= leader-id=
2026-09-21T15:25:47.487Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:47.573Z [WARN]  server-e7591341-d516-0a4a-ab8c-232355678a6c: heartbeat timeout reached, starting election: last-leader-addr=cf069b82-b031-1dbb-5e19-09788dc4013b last-leader-id=server-cf069b82-b031-1dbb-5e19-09788dc4013b
2026-09-21T15:25:47.580Z [INFO]  server-e7591341-d516-0a4a-ab8c-232355678a6c: entering candidate state: node="Node at e7591341-d516-0a4a-ab8c-232355678a6c [Candidate]" term=3
2026-09-21T15:25:47.580Z [ERROR] server-e7591341-d516-0a4a-ab8c-232355678a6c: failed to make requestVote RPC: target="{Voter server-cf069b82-b031-1dbb-5e19-09788dc4013b cf069b82-b031-1dbb-5e19-09788dc4013b}" error="failed to connect to peer: cf069b82-b031-1dbb-5e19-09788dc4013b" term=3
2026-09-21T15:25:47.593Z [WARN]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:47.593Z [INFO]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: entering candidate state: node="Node at cf069b82-b031-1dbb-5e19-09788dc4013b [Candidate]" term=3
2026-09-21T15:25:47.593Z [ERROR] server-cf069b82-b031-1dbb-5e19-09788dc4013b: failed to make requestVote RPC: target="{Voter server-e7591341-d516-0a4a-ab8c-232355678a6c e7591341-d516-0a4a-ab8c-232355678a6c}" error="failed to connect to peer: e7591341-d516-0a4a-ab8c-232355678a6c" term=3
2026-09-21T15:25:47.648Z [WARN]  server-e7591341-d516-0a4a-ab8c-232355678a6c: Election timeout reached, restarting election
2026-09-21T15:25:47.648Z [INFO]  server-e7591341-d516-0a4a-ab8c-232355678a6c: entering candidate state: node="Node at e7591341-d516-0a4a-ab8c-232355678a6c [Candidate]" term=4
2026-09-21T15:25:47.648Z [ERROR] server-e7591341-d516-0a4a-ab8c-232355678a6c: failed to make requestVote RPC: target="{Voter server-cf069b82-b031-1dbb-5e19-09788dc4013b cf069b82-b031-1dbb-5e19-09788dc4013b}" error="failed to connect to peer: cf069b82-b031-1dbb-5e19-09788dc4013b" term=4
2026-09-21T15:25:47.653Z [WARN]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: Election timeout reached, restarting election
2026-09-21T15:25:47.653Z [INFO]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: entering candidate state: node="Node at cf069b82-b031-1dbb-5e19-09788dc4013b [Candidate]" term=4
2026-09-21T15:25:47.653Z [ERROR] server-cf069b82-b031-1dbb-5e19-09788dc4013b: failed to make requestVote RPC: target="{Voter server-e7591341-d516-0a4a-ab8c-232355678a6c e7591341-d516-0a4a-ab8c-232355678a6c}" error="failed to connect to peer: e7591341-d516-0a4a-ab8c-232355678a6c" term=4
2026-09-21T15:25:47.699Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:47.593598708 +0000 UTC m=+99.534768841 (0 nodes), 106.55912ms from start of poll, 452.03368ms from cluster start. Timeout at 2026-09-21 15:25:47.698636188 +0000 UTC m=+99.639806401, 105.03756ms after stability
2026-09-21T15:25:47.702Z [WARN]  server-e7591341-d516-0a4a-ab8c-232355678a6c: Election timeout reached, restarting election
2026-09-21T15:25:47.702Z [INFO]  server-e7591341-d516-0a4a-ab8c-232355678a6c: entering candidate state: node="Node at e7591341-d516-0a4a-ab8c-232355678a6c [Candidate]" term=5
2026-09-21T15:25:47.702Z [ERROR] server-e7591341-d516-0a4a-ab8c-232355678a6c: failed to make requestVote RPC: target="{Voter server-cf069b82-b031-1dbb-5e19-09788dc4013b cf069b82-b031-1dbb-5e19-09788dc4013b}" error="failed to connect to peer: cf069b82-b031-1dbb-5e19-09788dc4013b" term=5
2026-09-21T15:25:47.752Z [WARN]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: Election timeout reached, restarting election
2026-09-21T15:25:47.752Z [INFO]  server-cf069b82-b031-1dbb-5e19-09788dc4013b: entering candidate state: node="Node at cf069b82-b031-1dbb-5e19-09788dc4013b [Candidate]" term=5
2026-09-21T15:25:47.752Z [ERROR] server-cf069b82-b031-1dbb-5e19-09788dc4013b: failed to make requestVote RPC: target="{Voter server-e7591341-d516-0a4a-ab8c-232355678a6c e7591341-d516-0a4a-ab8c-232355678a6c}" error="failed to connect to peer: e7591341-d516-0a4a-ab8c-232355678a6c" term=5
2026-09-21T15:25:47.790Z [WARN]  server-e7591341-d516-0a4a-ab8c-232355678a6c: Election timeout reached, restarting election
2026-09-21T15:25:47.790Z [INFO]  server-e7591341-d516-0a4a-ab8c-232355678a6c: entering candidate state: node="Node at e7591341-d516-0a4a-ab8c-232355678a6c [Candidate]" term=6
2026-09-21T15:25:47.790Z [ERROR] server-e7591341-d516-0a4a-ab8c-232355678a6c: failed to make requestVote RPC: target="{Voter server-cf069b82-b031-1dbb-5e19-09788dc4013b cf069b82-b031-1dbb-5e19-09788dc4013b}" error="failed to connect to peer: cf069b82-b031-1dbb-5e19-09788dc4013b" term=6
--- PASS: TestRaft_LeaderLeaseExpire (0.67s)
=== RUN   TestRaft_Barrier
2026-09-21T15:25:47.808Z [INFO]  server-1af732f4-5507-a0bd-57ce-6611baad9bd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1af732f4-5507-a0bd-57ce-6611baad9bd6 Address:1af732f4-5507-a0bd-57ce-6611baad9bd6} {Suffrage:Voter ID:server-f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d Address:f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d} {Suffrage:Voter ID:server-482adece-a595-da0c-d4f1-a8911f3eb9fd Address:482adece-a595-da0c-d4f1-a8911f3eb9fd}]"
2026-09-21T15:25:47.808Z [INFO]  server-f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1af732f4-5507-a0bd-57ce-6611baad9bd6 Address:1af732f4-5507-a0bd-57ce-6611baad9bd6} {Suffrage:Voter ID:server-f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d Address:f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d} {Suffrage:Voter ID:server-482adece-a595-da0c-d4f1-a8911f3eb9fd Address:482adece-a595-da0c-d4f1-a8911f3eb9fd}]"
2026-09-21T15:25:47.808Z [INFO]  server-482adece-a595-da0c-d4f1-a8911f3eb9fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1af732f4-5507-a0bd-57ce-6611baad9bd6 Address:1af732f4-5507-a0bd-57ce-6611baad9bd6} {Suffrage:Voter ID:server-f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d Address:f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d} {Suffrage:Voter ID:server-482adece-a595-da0c-d4f1-a8911f3eb9fd Address:482adece-a595-da0c-d4f1-a8911f3eb9fd}]"
2026-09-21T15:25:47.808Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:47.808Z [INFO]  server-1af732f4-5507-a0bd-57ce-6611baad9bd6: entering follower state: follower="Node at 1af732f4-5507-a0bd-57ce-6611baad9bd6 [Follower]" leader-address= leader-id=
2026-09-21T15:25:47.808Z [INFO]  server-f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d: entering follower state: follower="Node at f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d [Follower]" leader-address= leader-id=
2026-09-21T15:25:47.808Z [INFO]  server-482adece-a595-da0c-d4f1-a8911f3eb9fd: entering follower state: follower="Node at 482adece-a595-da0c-d4f1-a8911f3eb9fd [Follower]" leader-address= leader-id=
2026-09-21T15:25:47.866Z [WARN]  server-f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:47.866Z [INFO]  server-f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d: entering candidate state: node="Node at f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d [Candidate]" term=2
2026-09-21T15:25:47.866Z [INFO]  server-f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d: election won: term=2 tally=2
2026-09-21T15:25:47.866Z [INFO]  server-f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d: entering leader state: leader="Node at f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d [Leader]"
2026-09-21T15:25:47.866Z [INFO]  server-f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d: added peer, starting replication: peer=server-1af732f4-5507-a0bd-57ce-6611baad9bd6
2026-09-21T15:25:47.866Z [INFO]  server-f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d: added peer, starting replication: peer=server-482adece-a595-da0c-d4f1-a8911f3eb9fd
2026-09-21T15:25:47.866Z [INFO]  server-f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d: pipelining replication: peer="{Voter server-482adece-a595-da0c-d4f1-a8911f3eb9fd 482adece-a595-da0c-d4f1-a8911f3eb9fd}"
2026-09-21T15:25:47.866Z [INFO]  server-f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d: pipelining replication: peer="{Voter server-1af732f4-5507-a0bd-57ce-6611baad9bd6 1af732f4-5507-a0bd-57ce-6611baad9bd6}"
2026-09-21T15:25:47.972Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:47.866688788 +0000 UTC m=+99.807858921 (1 nodes), 58.3464ms from start of poll, 58.59068ms from cluster start. Timeout at 2026-09-21 15:25:47.971699308 +0000 UTC m=+99.912869481, 105.01056ms after stability
2026-09-21T15:25:47.986Z [INFO]  server-f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d: aborting pipeline replication: peer="{Voter server-482adece-a595-da0c-d4f1-a8911f3eb9fd 482adece-a595-da0c-d4f1-a8911f3eb9fd}"
2026-09-21T15:25:47.986Z [INFO]  server-f5ffee7b-0aa1-d8fb-8b50-0d9523486b7d: aborting pipeline replication: peer="{Voter server-1af732f4-5507-a0bd-57ce-6611baad9bd6 1af732f4-5507-a0bd-57ce-6611baad9bd6}"
--- PASS: TestRaft_Barrier (0.18s)
=== RUN   TestRaft_VerifyLeader
2026-09-21T15:25:47.996Z [INFO]  server-a7481565-0503-bbd3-fbf0-c5097c9d7949: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a7481565-0503-bbd3-fbf0-c5097c9d7949 Address:a7481565-0503-bbd3-fbf0-c5097c9d7949} {Suffrage:Voter ID:server-8845cdab-e56b-1d57-1176-6b80635571c8 Address:8845cdab-e56b-1d57-1176-6b80635571c8} {Suffrage:Voter ID:server-6f940ef9-2561-76d1-6103-976618b2c11a Address:6f940ef9-2561-76d1-6103-976618b2c11a}]"
2026-09-21T15:25:47.996Z [INFO]  server-a7481565-0503-bbd3-fbf0-c5097c9d7949: entering follower state: follower="Node at a7481565-0503-bbd3-fbf0-c5097c9d7949 [Follower]" leader-address= leader-id=
2026-09-21T15:25:47.996Z [INFO]  server-8845cdab-e56b-1d57-1176-6b80635571c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a7481565-0503-bbd3-fbf0-c5097c9d7949 Address:a7481565-0503-bbd3-fbf0-c5097c9d7949} {Suffrage:Voter ID:server-8845cdab-e56b-1d57-1176-6b80635571c8 Address:8845cdab-e56b-1d57-1176-6b80635571c8} {Suffrage:Voter ID:server-6f940ef9-2561-76d1-6103-976618b2c11a Address:6f940ef9-2561-76d1-6103-976618b2c11a}]"
2026-09-21T15:25:47.997Z [INFO]  server-8845cdab-e56b-1d57-1176-6b80635571c8: entering follower state: follower="Node at 8845cdab-e56b-1d57-1176-6b80635571c8 [Follower]" leader-address= leader-id=
2026-09-21T15:25:47.997Z [INFO]  server-6f940ef9-2561-76d1-6103-976618b2c11a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a7481565-0503-bbd3-fbf0-c5097c9d7949 Address:a7481565-0503-bbd3-fbf0-c5097c9d7949} {Suffrage:Voter ID:server-8845cdab-e56b-1d57-1176-6b80635571c8 Address:8845cdab-e56b-1d57-1176-6b80635571c8} {Suffrage:Voter ID:server-6f940ef9-2561-76d1-6103-976618b2c11a Address:6f940ef9-2561-76d1-6103-976618b2c11a}]"
2026-09-21T15:25:47.997Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:47.997Z [INFO]  server-6f940ef9-2561-76d1-6103-976618b2c11a: entering follower state: follower="Node at 6f940ef9-2561-76d1-6103-976618b2c11a [Follower]" leader-address= leader-id=
2026-09-21T15:25:48.063Z [WARN]  server-a7481565-0503-bbd3-fbf0-c5097c9d7949: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:48.063Z [INFO]  server-a7481565-0503-bbd3-fbf0-c5097c9d7949: entering candidate state: node="Node at a7481565-0503-bbd3-fbf0-c5097c9d7949 [Candidate]" term=2
2026-09-21T15:25:48.063Z [INFO]  server-a7481565-0503-bbd3-fbf0-c5097c9d7949: election won: term=2 tally=2
2026-09-21T15:25:48.063Z [INFO]  server-a7481565-0503-bbd3-fbf0-c5097c9d7949: entering leader state: leader="Node at a7481565-0503-bbd3-fbf0-c5097c9d7949 [Leader]"
2026-09-21T15:25:48.063Z [INFO]  server-a7481565-0503-bbd3-fbf0-c5097c9d7949: added peer, starting replication: peer=server-8845cdab-e56b-1d57-1176-6b80635571c8
2026-09-21T15:25:48.063Z [INFO]  server-a7481565-0503-bbd3-fbf0-c5097c9d7949: added peer, starting replication: peer=server-6f940ef9-2561-76d1-6103-976618b2c11a
2026-09-21T15:25:48.063Z [INFO]  server-a7481565-0503-bbd3-fbf0-c5097c9d7949: pipelining replication: peer="{Voter server-6f940ef9-2561-76d1-6103-976618b2c11a 6f940ef9-2561-76d1-6103-976618b2c11a}"
2026-09-21T15:25:48.063Z [INFO]  server-a7481565-0503-bbd3-fbf0-c5097c9d7949: pipelining replication: peer="{Voter server-8845cdab-e56b-1d57-1176-6b80635571c8 8845cdab-e56b-1d57-1176-6b80635571c8}"
2026-09-21T15:25:48.172Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:48.064028428 +0000 UTC m=+100.005198601 (1 nodes), 66.65912ms from start of poll, 77.31212ms from cluster start. Timeout at 2026-09-21 15:25:48.169029988 +0000 UTC m=+100.110200161, 105.00156ms after stability
2026-09-21T15:25:48.172Z [INFO]  server-a7481565-0503-bbd3-fbf0-c5097c9d7949: aborting pipeline replication: peer="{Voter server-6f940ef9-2561-76d1-6103-976618b2c11a 6f940ef9-2561-76d1-6103-976618b2c11a}"
2026-09-21T15:25:48.172Z [INFO]  server-a7481565-0503-bbd3-fbf0-c5097c9d7949: aborting pipeline replication: peer="{Voter server-8845cdab-e56b-1d57-1176-6b80635571c8 8845cdab-e56b-1d57-1176-6b80635571c8}"
2026-09-21T15:25:48.683Z [ERROR] server-a7481565-0503-bbd3-fbf0-c5097c9d7949: failed to heartbeat to: peer=8845cdab-e56b-1d57-1176-6b80635571c8 backoff time=10ms error="command timed out"
--- PASS: TestRaft_VerifyLeader (0.70s)
=== RUN   TestRaft_VerifyLeader_Single
2026-09-21T15:25:48.684Z [INFO]  server-26704826-b3b4-8092-153c-16024555a23d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26704826-b3b4-8092-153c-16024555a23d Address:26704826-b3b4-8092-153c-16024555a23d}]"
2026-09-21T15:25:48.684Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:48.684Z [INFO]  server-26704826-b3b4-8092-153c-16024555a23d: entering follower state: follower="Node at 26704826-b3b4-8092-153c-16024555a23d [Follower]" leader-address= leader-id=
2026-09-21T15:25:48.749Z [WARN]  server-26704826-b3b4-8092-153c-16024555a23d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:48.749Z [INFO]  server-26704826-b3b4-8092-153c-16024555a23d: entering candidate state: node="Node at 26704826-b3b4-8092-153c-16024555a23d [Candidate]" term=2
2026-09-21T15:25:48.749Z [INFO]  server-26704826-b3b4-8092-153c-16024555a23d: election won: term=2 tally=1
2026-09-21T15:25:48.749Z [INFO]  server-26704826-b3b4-8092-153c-16024555a23d: entering leader state: leader="Node at 26704826-b3b4-8092-153c-16024555a23d [Leader]"
2026-09-21T15:25:48.854Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:48.749789748 +0000 UTC m=+100.690959921 (1 nodes), 65.448ms from start of poll, 65.60936ms from cluster start. Timeout at 2026-09-21 15:25:48.854797348 +0000 UTC m=+100.795967521, 105.0076ms after stability
--- PASS: TestRaft_VerifyLeader_Single (0.17s)
=== RUN   TestRaft_VerifyLeader_Fail
2026-09-21T15:25:48.855Z [INFO]  server-da3167c5-f69e-e9bc-29de-bc1d35ed454c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da3167c5-f69e-e9bc-29de-bc1d35ed454c Address:da3167c5-f69e-e9bc-29de-bc1d35ed454c} {Suffrage:Voter ID:server-8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7 Address:8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7}]"
2026-09-21T15:25:48.855Z [INFO]  server-8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da3167c5-f69e-e9bc-29de-bc1d35ed454c Address:da3167c5-f69e-e9bc-29de-bc1d35ed454c} {Suffrage:Voter ID:server-8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7 Address:8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7}]"
2026-09-21T15:25:48.855Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:48.855Z [INFO]  server-da3167c5-f69e-e9bc-29de-bc1d35ed454c: entering follower state: follower="Node at da3167c5-f69e-e9bc-29de-bc1d35ed454c [Follower]" leader-address= leader-id=
2026-09-21T15:25:48.855Z [INFO]  server-8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7: entering follower state: follower="Node at 8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7 [Follower]" leader-address= leader-id=
2026-09-21T15:25:48.942Z [WARN]  server-8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:48.942Z [INFO]  server-8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7: entering candidate state: node="Node at 8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7 [Candidate]" term=2
2026-09-21T15:25:48.942Z [INFO]  server-8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7: election won: term=2 tally=2
2026-09-21T15:25:48.942Z [INFO]  server-8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7: entering leader state: leader="Node at 8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7 [Leader]"
2026-09-21T15:25:48.942Z [INFO]  server-8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7: added peer, starting replication: peer=server-da3167c5-f69e-e9bc-29de-bc1d35ed454c
2026-09-21T15:25:48.943Z [INFO]  server-8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7: pipelining replication: peer="{Voter server-da3167c5-f69e-e9bc-29de-bc1d35ed454c da3167c5-f69e-e9bc-29de-bc1d35ed454c}"
2026-09-21T15:25:49.048Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:48.943154428 +0000 UTC m=+100.884324601 (1 nodes), 87.35272ms from start of poll, 87.64724ms from cluster start. Timeout at 2026-09-21 15:25:49.048160708 +0000 UTC m=+100.989330881, 105.00628ms after stability
2026-09-21T15:25:49.048Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:49.154Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:49.048806748 +0000 UTC m=+100.989976961 (1 nodes), 2.76µs from start of poll, 193.2996ms from cluster start. Timeout at 2026-09-21 15:25:49.153852188 +0000 UTC m=+101.095022361, 105.0454ms after stability
2026-09-21T15:25:49.154Z [ERROR] server-8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7: peer has newer term, stopping replication: peer="{Voter server-da3167c5-f69e-e9bc-29de-bc1d35ed454c da3167c5-f69e-e9bc-29de-bc1d35ed454c}"
2026-09-21T15:25:49.154Z [INFO]  server-8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7: aborting pipeline replication: peer="{Voter server-da3167c5-f69e-e9bc-29de-bc1d35ed454c da3167c5-f69e-e9bc-29de-bc1d35ed454c}"
2026-09-21T15:25:49.154Z [INFO]  server-8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7: entering follower state: follower="Node at 8b9e9f9b-db4b-6dff-7c21-db3ed5b91dd7 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VerifyLeader_Fail (0.30s)
=== RUN   TestRaft_VerifyLeader_PartialConnect
2026-09-21T15:25:49.155Z [INFO]  server-9b1bbe86-e4c3-399b-2591-b45073019a0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b1bbe86-e4c3-399b-2591-b45073019a0e Address:9b1bbe86-e4c3-399b-2591-b45073019a0e} {Suffrage:Voter ID:server-c7f164b2-3c11-528a-9829-3bedd85eb292 Address:c7f164b2-3c11-528a-9829-3bedd85eb292} {Suffrage:Voter ID:server-bb939321-a474-f066-7d75-88674a4d60eb Address:bb939321-a474-f066-7d75-88674a4d60eb}]"
2026-09-21T15:25:49.155Z [INFO]  server-c7f164b2-3c11-528a-9829-3bedd85eb292: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b1bbe86-e4c3-399b-2591-b45073019a0e Address:9b1bbe86-e4c3-399b-2591-b45073019a0e} {Suffrage:Voter ID:server-c7f164b2-3c11-528a-9829-3bedd85eb292 Address:c7f164b2-3c11-528a-9829-3bedd85eb292} {Suffrage:Voter ID:server-bb939321-a474-f066-7d75-88674a4d60eb Address:bb939321-a474-f066-7d75-88674a4d60eb}]"
2026-09-21T15:25:49.155Z [INFO]  server-bb939321-a474-f066-7d75-88674a4d60eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b1bbe86-e4c3-399b-2591-b45073019a0e Address:9b1bbe86-e4c3-399b-2591-b45073019a0e} {Suffrage:Voter ID:server-c7f164b2-3c11-528a-9829-3bedd85eb292 Address:c7f164b2-3c11-528a-9829-3bedd85eb292} {Suffrage:Voter ID:server-bb939321-a474-f066-7d75-88674a4d60eb Address:bb939321-a474-f066-7d75-88674a4d60eb}]"
2026-09-21T15:25:49.155Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:49.155Z [INFO]  server-9b1bbe86-e4c3-399b-2591-b45073019a0e: entering follower state: follower="Node at 9b1bbe86-e4c3-399b-2591-b45073019a0e [Follower]" leader-address= leader-id=
2026-09-21T15:25:49.155Z [INFO]  server-c7f164b2-3c11-528a-9829-3bedd85eb292: entering follower state: follower="Node at c7f164b2-3c11-528a-9829-3bedd85eb292 [Follower]" leader-address= leader-id=
2026-09-21T15:25:49.155Z [INFO]  server-bb939321-a474-f066-7d75-88674a4d60eb: entering follower state: follower="Node at bb939321-a474-f066-7d75-88674a4d60eb [Follower]" leader-address= leader-id=
2026-09-21T15:25:49.221Z [WARN]  server-bb939321-a474-f066-7d75-88674a4d60eb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:49.221Z [INFO]  server-bb939321-a474-f066-7d75-88674a4d60eb: entering candidate state: node="Node at bb939321-a474-f066-7d75-88674a4d60eb [Candidate]" term=2
2026-09-21T15:25:49.221Z [INFO]  server-bb939321-a474-f066-7d75-88674a4d60eb: election won: term=2 tally=2
2026-09-21T15:25:49.221Z [INFO]  server-bb939321-a474-f066-7d75-88674a4d60eb: entering leader state: leader="Node at bb939321-a474-f066-7d75-88674a4d60eb [Leader]"
2026-09-21T15:25:49.221Z [INFO]  server-bb939321-a474-f066-7d75-88674a4d60eb: added peer, starting replication: peer=server-9b1bbe86-e4c3-399b-2591-b45073019a0e
2026-09-21T15:25:49.221Z [INFO]  server-bb939321-a474-f066-7d75-88674a4d60eb: added peer, starting replication: peer=server-c7f164b2-3c11-528a-9829-3bedd85eb292
2026-09-21T15:25:49.222Z [INFO]  server-bb939321-a474-f066-7d75-88674a4d60eb: pipelining replication: peer="{Voter server-c7f164b2-3c11-528a-9829-3bedd85eb292 c7f164b2-3c11-528a-9829-3bedd85eb292}"
2026-09-21T15:25:49.222Z [INFO]  server-bb939321-a474-f066-7d75-88674a4d60eb: pipelining replication: peer="{Voter server-9b1bbe86-e4c3-399b-2591-b45073019a0e 9b1bbe86-e4c3-399b-2591-b45073019a0e}"
2026-09-21T15:25:49.329Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:49.221925508 +0000 UTC m=+101.163095681 (1 nodes), 66.37132ms from start of poll, 66.76232ms from cluster start. Timeout at 2026-09-21 15:25:49.326927988 +0000 UTC m=+101.268098161, 105.00248ms after stability
2026-09-21T15:25:49.334Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:49.440Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:49.334842708 +0000 UTC m=+101.276012881 (2 nodes), 2.92µs from start of poll, 179.67952ms from cluster start. Timeout at 2026-09-21 15:25:49.439885588 +0000 UTC m=+101.381055761, 105.04288ms after stability
    raft_test.go:1968: [INFO] Disconnecting Node at 9b1bbe86-e4c3-399b-2591-b45073019a0e [Follower]
2026-09-21T15:25:49.440Z [ERROR] server-bb939321-a474-f066-7d75-88674a4d60eb: failed to heartbeat to: peer=9b1bbe86-e4c3-399b-2591-b45073019a0e backoff time=10ms error="failed to connect to peer: 9b1bbe86-e4c3-399b-2591-b45073019a0e"
2026-09-21T15:25:49.440Z [INFO]  server-bb939321-a474-f066-7d75-88674a4d60eb: aborting pipeline replication: peer="{Voter server-c7f164b2-3c11-528a-9829-3bedd85eb292 c7f164b2-3c11-528a-9829-3bedd85eb292}"
2026-09-21T15:25:49.440Z [INFO]  server-bb939321-a474-f066-7d75-88674a4d60eb: aborting pipeline replication: peer="{Voter server-9b1bbe86-e4c3-399b-2591-b45073019a0e 9b1bbe86-e4c3-399b-2591-b45073019a0e}"
--- PASS: TestRaft_VerifyLeader_PartialConnect (0.29s)
=== RUN   TestRaft_NotifyCh
2026-09-21T15:25:49.441Z [INFO]  server-a1d4ad76-ebf1-dbd8-a767-2fba89a1fd1c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1d4ad76-ebf1-dbd8-a767-2fba89a1fd1c Address:a1d4ad76-ebf1-dbd8-a767-2fba89a1fd1c}]"
2026-09-21T15:25:49.441Z [INFO]  server-a1d4ad76-ebf1-dbd8-a767-2fba89a1fd1c: entering follower state: follower="Node at a1d4ad76-ebf1-dbd8-a767-2fba89a1fd1c [Follower]" leader-address= leader-id=
2026-09-21T15:25:49.533Z [WARN]  server-a1d4ad76-ebf1-dbd8-a767-2fba89a1fd1c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:49.533Z [INFO]  server-a1d4ad76-ebf1-dbd8-a767-2fba89a1fd1c: entering candidate state: node="Node at a1d4ad76-ebf1-dbd8-a767-2fba89a1fd1c [Candidate]" term=2
2026-09-21T15:25:49.534Z [INFO]  server-a1d4ad76-ebf1-dbd8-a767-2fba89a1fd1c: election won: term=2 tally=1
2026-09-21T15:25:49.534Z [INFO]  server-a1d4ad76-ebf1-dbd8-a767-2fba89a1fd1c: entering leader state: leader="Node at a1d4ad76-ebf1-dbd8-a767-2fba89a1fd1c [Leader]"
--- PASS: TestRaft_NotifyCh (0.09s)
=== RUN   TestRaft_AppendEntry
2026-09-21T15:25:49.534Z [INFO]  server-b3738294-06cc-2fd2-17be-aa028e6702cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3738294-06cc-2fd2-17be-aa028e6702cd Address:b3738294-06cc-2fd2-17be-aa028e6702cd} {Suffrage:Voter ID:server-8c89a9a8-19c7-c94e-cb36-4c59d81ea230 Address:8c89a9a8-19c7-c94e-cb36-4c59d81ea230} {Suffrage:Voter ID:server-f2414aa4-f376-797f-cb74-7859ae1dd60d Address:f2414aa4-f376-797f-cb74-7859ae1dd60d}]"
2026-09-21T15:25:49.535Z [INFO]  server-8c89a9a8-19c7-c94e-cb36-4c59d81ea230: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3738294-06cc-2fd2-17be-aa028e6702cd Address:b3738294-06cc-2fd2-17be-aa028e6702cd} {Suffrage:Voter ID:server-8c89a9a8-19c7-c94e-cb36-4c59d81ea230 Address:8c89a9a8-19c7-c94e-cb36-4c59d81ea230} {Suffrage:Voter ID:server-f2414aa4-f376-797f-cb74-7859ae1dd60d Address:f2414aa4-f376-797f-cb74-7859ae1dd60d}]"
2026-09-21T15:25:49.535Z [INFO]  server-b3738294-06cc-2fd2-17be-aa028e6702cd: entering follower state: follower="Node at b3738294-06cc-2fd2-17be-aa028e6702cd [Follower]" leader-address= leader-id=
2026-09-21T15:25:49.535Z [INFO]  server-8c89a9a8-19c7-c94e-cb36-4c59d81ea230: entering follower state: follower="Node at 8c89a9a8-19c7-c94e-cb36-4c59d81ea230 [Follower]" leader-address= leader-id=
2026-09-21T15:25:49.535Z [INFO]  server-f2414aa4-f376-797f-cb74-7859ae1dd60d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b3738294-06cc-2fd2-17be-aa028e6702cd Address:b3738294-06cc-2fd2-17be-aa028e6702cd} {Suffrage:Voter ID:server-8c89a9a8-19c7-c94e-cb36-4c59d81ea230 Address:8c89a9a8-19c7-c94e-cb36-4c59d81ea230} {Suffrage:Voter ID:server-f2414aa4-f376-797f-cb74-7859ae1dd60d Address:f2414aa4-f376-797f-cb74-7859ae1dd60d}]"
2026-09-21T15:25:49.535Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:49.535Z [INFO]  server-f2414aa4-f376-797f-cb74-7859ae1dd60d: entering follower state: follower="Node at f2414aa4-f376-797f-cb74-7859ae1dd60d [Follower]" leader-address= leader-id=
2026-09-21T15:25:49.608Z [WARN]  server-b3738294-06cc-2fd2-17be-aa028e6702cd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:49.608Z [INFO]  server-b3738294-06cc-2fd2-17be-aa028e6702cd: entering candidate state: node="Node at b3738294-06cc-2fd2-17be-aa028e6702cd [Candidate]" term=2
2026-09-21T15:25:49.608Z [INFO]  server-b3738294-06cc-2fd2-17be-aa028e6702cd: election won: term=2 tally=2
2026-09-21T15:25:49.608Z [INFO]  server-b3738294-06cc-2fd2-17be-aa028e6702cd: entering leader state: leader="Node at b3738294-06cc-2fd2-17be-aa028e6702cd [Leader]"
2026-09-21T15:25:49.608Z [INFO]  server-b3738294-06cc-2fd2-17be-aa028e6702cd: added peer, starting replication: peer=server-8c89a9a8-19c7-c94e-cb36-4c59d81ea230
2026-09-21T15:25:49.608Z [INFO]  server-b3738294-06cc-2fd2-17be-aa028e6702cd: added peer, starting replication: peer=server-f2414aa4-f376-797f-cb74-7859ae1dd60d
2026-09-21T15:25:49.608Z [INFO]  server-b3738294-06cc-2fd2-17be-aa028e6702cd: pipelining replication: peer="{Voter server-f2414aa4-f376-797f-cb74-7859ae1dd60d f2414aa4-f376-797f-cb74-7859ae1dd60d}"
2026-09-21T15:25:49.608Z [INFO]  server-b3738294-06cc-2fd2-17be-aa028e6702cd: pipelining replication: peer="{Voter server-8c89a9a8-19c7-c94e-cb36-4c59d81ea230 8c89a9a8-19c7-c94e-cb36-4c59d81ea230}"
2026-09-21T15:25:49.715Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:49.609003148 +0000 UTC m=+101.550173321 (2 nodes), 73.12772ms from start of poll, 74.28748ms from cluster start. Timeout at 2026-09-21 15:25:49.714008508 +0000 UTC m=+101.655178641, 105.00532ms after stability
2026-09-21T15:25:49.715Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:49.820Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:49.715747228 +0000 UTC m=+101.656917401 (1 nodes), 920ns from start of poll, 181.03156ms from cluster start. Timeout at 2026-09-21 15:25:49.820781588 +0000 UTC m=+101.761951761, 105.03436ms after stability
2026-09-21T15:25:49.821Z [WARN]  server-8c89a9a8-19c7-c94e-cb36-4c59d81ea230: clearing log suffix: from=1 to=2
2026-09-21T15:25:49.821Z [WARN]  server-8c89a9a8-19c7-c94e-cb36-4c59d81ea230: skipping application of old log: index=1
2026-09-21T15:25:49.822Z [WARN]  server-8c89a9a8-19c7-c94e-cb36-4c59d81ea230: skipping application of old log: index=1
2026-09-21T15:25:49.822Z [INFO]  server-b3738294-06cc-2fd2-17be-aa028e6702cd: aborting pipeline replication: peer="{Voter server-f2414aa4-f376-797f-cb74-7859ae1dd60d f2414aa4-f376-797f-cb74-7859ae1dd60d}"
2026-09-21T15:25:49.822Z [INFO]  server-b3738294-06cc-2fd2-17be-aa028e6702cd: aborting pipeline replication: peer="{Voter server-8c89a9a8-19c7-c94e-cb36-4c59d81ea230 8c89a9a8-19c7-c94e-cb36-4c59d81ea230}"
--- PASS: TestRaft_AppendEntry (0.29s)
=== RUN   TestRaft_VotingGrant_WhenLeaderAvailable
2026-09-21T15:25:49.824Z [INFO]  server-8e65bc83-3ff2-264c-41de-74ed561ee9cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e65bc83-3ff2-264c-41de-74ed561ee9cc Address:8e65bc83-3ff2-264c-41de-74ed561ee9cc} {Suffrage:Voter ID:server-1ca780a4-8dde-7ba5-3d43-37d158afc16a Address:1ca780a4-8dde-7ba5-3d43-37d158afc16a} {Suffrage:Voter ID:server-6b92f9ed-8317-ff9c-b917-678d6f9ea150 Address:6b92f9ed-8317-ff9c-b917-678d6f9ea150}]"
2026-09-21T15:25:49.824Z [INFO]  server-8e65bc83-3ff2-264c-41de-74ed561ee9cc: entering follower state: follower="Node at 8e65bc83-3ff2-264c-41de-74ed561ee9cc [Follower]" leader-address= leader-id=
2026-09-21T15:25:49.824Z [INFO]  server-1ca780a4-8dde-7ba5-3d43-37d158afc16a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e65bc83-3ff2-264c-41de-74ed561ee9cc Address:8e65bc83-3ff2-264c-41de-74ed561ee9cc} {Suffrage:Voter ID:server-1ca780a4-8dde-7ba5-3d43-37d158afc16a Address:1ca780a4-8dde-7ba5-3d43-37d158afc16a} {Suffrage:Voter ID:server-6b92f9ed-8317-ff9c-b917-678d6f9ea150 Address:6b92f9ed-8317-ff9c-b917-678d6f9ea150}]"
2026-09-21T15:25:49.825Z [INFO]  server-1ca780a4-8dde-7ba5-3d43-37d158afc16a: entering follower state: follower="Node at 1ca780a4-8dde-7ba5-3d43-37d158afc16a [Follower]" leader-address= leader-id=
2026-09-21T15:25:49.825Z [INFO]  server-6b92f9ed-8317-ff9c-b917-678d6f9ea150: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e65bc83-3ff2-264c-41de-74ed561ee9cc Address:8e65bc83-3ff2-264c-41de-74ed561ee9cc} {Suffrage:Voter ID:server-1ca780a4-8dde-7ba5-3d43-37d158afc16a Address:1ca780a4-8dde-7ba5-3d43-37d158afc16a} {Suffrage:Voter ID:server-6b92f9ed-8317-ff9c-b917-678d6f9ea150 Address:6b92f9ed-8317-ff9c-b917-678d6f9ea150}]"
2026-09-21T15:25:49.825Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:49.825Z [INFO]  server-6b92f9ed-8317-ff9c-b917-678d6f9ea150: entering follower state: follower="Node at 6b92f9ed-8317-ff9c-b917-678d6f9ea150 [Follower]" leader-address= leader-id=
2026-09-21T15:25:49.878Z [WARN]  server-8e65bc83-3ff2-264c-41de-74ed561ee9cc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:49.878Z [INFO]  server-8e65bc83-3ff2-264c-41de-74ed561ee9cc: entering candidate state: node="Node at 8e65bc83-3ff2-264c-41de-74ed561ee9cc [Candidate]" term=2
2026-09-21T15:25:49.878Z [INFO]  server-8e65bc83-3ff2-264c-41de-74ed561ee9cc: election won: term=2 tally=2
2026-09-21T15:25:49.878Z [INFO]  server-8e65bc83-3ff2-264c-41de-74ed561ee9cc: entering leader state: leader="Node at 8e65bc83-3ff2-264c-41de-74ed561ee9cc [Leader]"
2026-09-21T15:25:49.878Z [INFO]  server-8e65bc83-3ff2-264c-41de-74ed561ee9cc: added peer, starting replication: peer=server-1ca780a4-8dde-7ba5-3d43-37d158afc16a
2026-09-21T15:25:49.878Z [INFO]  server-8e65bc83-3ff2-264c-41de-74ed561ee9cc: added peer, starting replication: peer=server-6b92f9ed-8317-ff9c-b917-678d6f9ea150
2026-09-21T15:25:49.878Z [INFO]  server-8e65bc83-3ff2-264c-41de-74ed561ee9cc: pipelining replication: peer="{Voter server-6b92f9ed-8317-ff9c-b917-678d6f9ea150 6b92f9ed-8317-ff9c-b917-678d6f9ea150}"
2026-09-21T15:25:49.878Z [INFO]  server-8e65bc83-3ff2-264c-41de-74ed561ee9cc: pipelining replication: peer="{Voter server-1ca780a4-8dde-7ba5-3d43-37d158afc16a 1ca780a4-8dde-7ba5-3d43-37d158afc16a}"
2026-09-21T15:25:49.984Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:49.878798308 +0000 UTC m=+101.819968481 (2 nodes), 53.21516ms from start of poll, 54.9388ms from cluster start. Timeout at 2026-09-21 15:25:49.983803148 +0000 UTC m=+101.924973321, 105.00484ms after stability
2026-09-21T15:25:49.984Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:50.089Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:49.984797628 +0000 UTC m=+101.925967801 (1 nodes), 3.6µs from start of poll, 160.93812ms from cluster start. Timeout at 2026-09-21 15:25:50.089828548 +0000 UTC m=+102.030998721, 105.03092ms after stability
2026-09-21T15:25:50.090Z [WARN]  server-6b92f9ed-8317-ff9c-b917-678d6f9ea150: rejecting vote request since we have a leader: from=1ca780a4-8dde-7ba5-3d43-37d158afc16a leader=8e65bc83-3ff2-264c-41de-74ed561ee9cc leader-id=server-8e65bc83-3ff2-264c-41de-74ed561ee9cc
2026-09-21T15:25:50.090Z [INFO]  server-8e65bc83-3ff2-264c-41de-74ed561ee9cc: aborting pipeline replication: peer="{Voter server-6b92f9ed-8317-ff9c-b917-678d6f9ea150 6b92f9ed-8317-ff9c-b917-678d6f9ea150}"
2026-09-21T15:25:50.090Z [INFO]  server-8e65bc83-3ff2-264c-41de-74ed561ee9cc: aborting pipeline replication: peer="{Voter server-1ca780a4-8dde-7ba5-3d43-37d158afc16a 1ca780a4-8dde-7ba5-3d43-37d158afc16a}"
--- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.27s)
=== RUN   TestRaft_ProtocolVersion_RejectRPC
2026-09-21T15:25:50.091Z [INFO]  server-01223b38-84be-aae3-8ca7-27910d2e785b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01223b38-84be-aae3-8ca7-27910d2e785b Address:01223b38-84be-aae3-8ca7-27910d2e785b} {Suffrage:Voter ID:server-77050571-f6e0-7290-da28-cada568ace90 Address:77050571-f6e0-7290-da28-cada568ace90} {Suffrage:Voter ID:server-c0fe35e8-a66d-f893-238c-d3a9c7625403 Address:c0fe35e8-a66d-f893-238c-d3a9c7625403}]"
2026-09-21T15:25:50.091Z [INFO]  server-77050571-f6e0-7290-da28-cada568ace90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01223b38-84be-aae3-8ca7-27910d2e785b Address:01223b38-84be-aae3-8ca7-27910d2e785b} {Suffrage:Voter ID:server-77050571-f6e0-7290-da28-cada568ace90 Address:77050571-f6e0-7290-da28-cada568ace90} {Suffrage:Voter ID:server-c0fe35e8-a66d-f893-238c-d3a9c7625403 Address:c0fe35e8-a66d-f893-238c-d3a9c7625403}]"
2026-09-21T15:25:50.091Z [INFO]  server-c0fe35e8-a66d-f893-238c-d3a9c7625403: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01223b38-84be-aae3-8ca7-27910d2e785b Address:01223b38-84be-aae3-8ca7-27910d2e785b} {Suffrage:Voter ID:server-77050571-f6e0-7290-da28-cada568ace90 Address:77050571-f6e0-7290-da28-cada568ace90} {Suffrage:Voter ID:server-c0fe35e8-a66d-f893-238c-d3a9c7625403 Address:c0fe35e8-a66d-f893-238c-d3a9c7625403}]"
2026-09-21T15:25:50.091Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:50.091Z [INFO]  server-01223b38-84be-aae3-8ca7-27910d2e785b: entering follower state: follower="Node at 01223b38-84be-aae3-8ca7-27910d2e785b [Follower]" leader-address= leader-id=
2026-09-21T15:25:50.091Z [INFO]  server-77050571-f6e0-7290-da28-cada568ace90: entering follower state: follower="Node at 77050571-f6e0-7290-da28-cada568ace90 [Follower]" leader-address= leader-id=
2026-09-21T15:25:50.091Z [INFO]  server-c0fe35e8-a66d-f893-238c-d3a9c7625403: entering follower state: follower="Node at c0fe35e8-a66d-f893-238c-d3a9c7625403 [Follower]" leader-address= leader-id=
2026-09-21T15:25:50.175Z [WARN]  server-77050571-f6e0-7290-da28-cada568ace90: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:50.175Z [INFO]  server-77050571-f6e0-7290-da28-cada568ace90: entering candidate state: node="Node at 77050571-f6e0-7290-da28-cada568ace90 [Candidate]" term=2
2026-09-21T15:25:50.175Z [INFO]  server-77050571-f6e0-7290-da28-cada568ace90: election won: term=2 tally=2
2026-09-21T15:25:50.175Z [INFO]  server-77050571-f6e0-7290-da28-cada568ace90: entering leader state: leader="Node at 77050571-f6e0-7290-da28-cada568ace90 [Leader]"
2026-09-21T15:25:50.175Z [INFO]  server-77050571-f6e0-7290-da28-cada568ace90: added peer, starting replication: peer=server-01223b38-84be-aae3-8ca7-27910d2e785b
2026-09-21T15:25:50.175Z [INFO]  server-77050571-f6e0-7290-da28-cada568ace90: added peer, starting replication: peer=server-c0fe35e8-a66d-f893-238c-d3a9c7625403
2026-09-21T15:25:50.175Z [INFO]  server-77050571-f6e0-7290-da28-cada568ace90: pipelining replication: peer="{Voter server-c0fe35e8-a66d-f893-238c-d3a9c7625403 c0fe35e8-a66d-f893-238c-d3a9c7625403}"
2026-09-21T15:25:50.175Z [INFO]  server-77050571-f6e0-7290-da28-cada568ace90: pipelining replication: peer="{Voter server-01223b38-84be-aae3-8ca7-27910d2e785b 01223b38-84be-aae3-8ca7-27910d2e785b}"
2026-09-21T15:25:50.281Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:50.175329868 +0000 UTC m=+102.116500081 (2 nodes), 83.8604ms from start of poll, 84.29728ms from cluster start. Timeout at 2026-09-21 15:25:50.280333588 +0000 UTC m=+102.221503761, 105.00368ms after stability
2026-09-21T15:25:50.281Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:50.387Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:50.281636668 +0000 UTC m=+102.222806841 (1 nodes), 2.44µs from start of poll, 190.60404ms from cluster start. Timeout at 2026-09-21 15:25:50.386672828 +0000 UTC m=+102.327843001, 105.03616ms after stability
2026-09-21T15:25:50.387Z [INFO]  server-77050571-f6e0-7290-da28-cada568ace90: aborting pipeline replication: peer="{Voter server-c0fe35e8-a66d-f893-238c-d3a9c7625403 c0fe35e8-a66d-f893-238c-d3a9c7625403}"
2026-09-21T15:25:50.387Z [INFO]  server-77050571-f6e0-7290-da28-cada568ace90: aborting pipeline replication: peer="{Voter server-01223b38-84be-aae3-8ca7-27910d2e785b 01223b38-84be-aae3-8ca7-27910d2e785b}"
--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.30s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_1_2
    raft_test.go:2151: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_2_3
    raft_test.go:2196: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s)
=== RUN   TestRaft_LeaderID_Propagated
2026-09-21T15:25:50.388Z [INFO]  server-d3827fb7-5b76-112b-9695-19df420ed23a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3827fb7-5b76-112b-9695-19df420ed23a Address:d3827fb7-5b76-112b-9695-19df420ed23a} {Suffrage:Voter ID:server-b9afba60-8aa3-15fb-33d1-a9849c0707e5 Address:b9afba60-8aa3-15fb-33d1-a9849c0707e5} {Suffrage:Voter ID:server-55e28a3d-ea41-f4ee-1a37-222d4469ea4c Address:55e28a3d-ea41-f4ee-1a37-222d4469ea4c}]"
2026-09-21T15:25:50.388Z [INFO]  server-b9afba60-8aa3-15fb-33d1-a9849c0707e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3827fb7-5b76-112b-9695-19df420ed23a Address:d3827fb7-5b76-112b-9695-19df420ed23a} {Suffrage:Voter ID:server-b9afba60-8aa3-15fb-33d1-a9849c0707e5 Address:b9afba60-8aa3-15fb-33d1-a9849c0707e5} {Suffrage:Voter ID:server-55e28a3d-ea41-f4ee-1a37-222d4469ea4c Address:55e28a3d-ea41-f4ee-1a37-222d4469ea4c}]"
2026-09-21T15:25:50.388Z [INFO]  server-55e28a3d-ea41-f4ee-1a37-222d4469ea4c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3827fb7-5b76-112b-9695-19df420ed23a Address:d3827fb7-5b76-112b-9695-19df420ed23a} {Suffrage:Voter ID:server-b9afba60-8aa3-15fb-33d1-a9849c0707e5 Address:b9afba60-8aa3-15fb-33d1-a9849c0707e5} {Suffrage:Voter ID:server-55e28a3d-ea41-f4ee-1a37-222d4469ea4c Address:55e28a3d-ea41-f4ee-1a37-222d4469ea4c}]"
2026-09-21T15:25:50.388Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:50.388Z [INFO]  server-d3827fb7-5b76-112b-9695-19df420ed23a: entering follower state: follower="Node at d3827fb7-5b76-112b-9695-19df420ed23a [Follower]" leader-address= leader-id=
2026-09-21T15:25:50.388Z [INFO]  server-b9afba60-8aa3-15fb-33d1-a9849c0707e5: entering follower state: follower="Node at b9afba60-8aa3-15fb-33d1-a9849c0707e5 [Follower]" leader-address= leader-id=
2026-09-21T15:25:50.388Z [INFO]  server-55e28a3d-ea41-f4ee-1a37-222d4469ea4c: entering follower state: follower="Node at 55e28a3d-ea41-f4ee-1a37-222d4469ea4c [Follower]" leader-address= leader-id=
2026-09-21T15:25:50.460Z [WARN]  server-55e28a3d-ea41-f4ee-1a37-222d4469ea4c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:50.460Z [INFO]  server-55e28a3d-ea41-f4ee-1a37-222d4469ea4c: entering candidate state: node="Node at 55e28a3d-ea41-f4ee-1a37-222d4469ea4c [Candidate]" term=2
2026-09-21T15:25:50.460Z [INFO]  server-55e28a3d-ea41-f4ee-1a37-222d4469ea4c: election won: term=2 tally=2
2026-09-21T15:25:50.460Z [INFO]  server-55e28a3d-ea41-f4ee-1a37-222d4469ea4c: entering leader state: leader="Node at 55e28a3d-ea41-f4ee-1a37-222d4469ea4c [Leader]"
2026-09-21T15:25:50.460Z [INFO]  server-55e28a3d-ea41-f4ee-1a37-222d4469ea4c: added peer, starting replication: peer=server-d3827fb7-5b76-112b-9695-19df420ed23a
2026-09-21T15:25:50.460Z [INFO]  server-55e28a3d-ea41-f4ee-1a37-222d4469ea4c: added peer, starting replication: peer=server-b9afba60-8aa3-15fb-33d1-a9849c0707e5
2026-09-21T15:25:50.460Z [INFO]  server-55e28a3d-ea41-f4ee-1a37-222d4469ea4c: pipelining replication: peer="{Voter server-b9afba60-8aa3-15fb-33d1-a9849c0707e5 b9afba60-8aa3-15fb-33d1-a9849c0707e5}"
2026-09-21T15:25:50.460Z [INFO]  server-55e28a3d-ea41-f4ee-1a37-222d4469ea4c: pipelining replication: peer="{Voter server-d3827fb7-5b76-112b-9695-19df420ed23a d3827fb7-5b76-112b-9695-19df420ed23a}"
2026-09-21T15:25:50.565Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:50.460580588 +0000 UTC m=+102.401750761 (1 nodes), 71.77312ms from start of poll, 72.14844ms from cluster start. Timeout at 2026-09-21 15:25:50.565586988 +0000 UTC m=+102.506757161, 105.0064ms after stability
2026-09-21T15:25:50.566Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:50.671Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:50.566172748 +0000 UTC m=+102.507342921 (1 nodes), 600ns from start of poll, 177.7406ms from cluster start. Timeout at 2026-09-21 15:25:50.671200148 +0000 UTC m=+102.612370321, 105.0274ms after stability
2026-09-21T15:25:50.672Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:50.777Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:50.672042548 +0000 UTC m=+102.613212721 (1 nodes), 960ns from start of poll, 283.6104ms from cluster start. Timeout at 2026-09-21 15:25:50.777069828 +0000 UTC m=+102.718240001, 105.02728ms after stability
2026-09-21T15:25:50.777Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:50.882Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:50.777400148 +0000 UTC m=+102.718570321 (1 nodes), 1µs from start of poll, 388.968ms from cluster start. Timeout at 2026-09-21 15:25:50.882426188 +0000 UTC m=+102.823596361, 105.02604ms after stability
2026-09-21T15:25:50.882Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:50.987Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:50.882699908 +0000 UTC m=+102.823870081 (1 nodes), 760ns from start of poll, 494.26776ms from cluster start. Timeout at 2026-09-21 15:25:50.987732428 +0000 UTC m=+102.928902601, 105.03252ms after stability
2026-09-21T15:25:50.988Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:51.093Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:50.988098308 +0000 UTC m=+102.929268481 (1 nodes), 880ns from start of poll, 599.66616ms from cluster start. Timeout at 2026-09-21 15:25:51.093138188 +0000 UTC m=+103.034308361, 105.03988ms after stability
2026-09-21T15:25:51.199Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:51.304Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:51.199457508 +0000 UTC m=+103.140627681 (1 nodes), 2.44µs from start of poll, 811.02536ms from cluster start. Timeout at 2026-09-21 15:25:51.304514508 +0000 UTC m=+103.245684681, 105.057ms after stability
2026-09-21T15:25:51.304Z [INFO]  server-55e28a3d-ea41-f4ee-1a37-222d4469ea4c: aborting pipeline replication: peer="{Voter server-d3827fb7-5b76-112b-9695-19df420ed23a d3827fb7-5b76-112b-9695-19df420ed23a}"
2026-09-21T15:25:51.304Z [INFO]  server-55e28a3d-ea41-f4ee-1a37-222d4469ea4c: aborting pipeline replication: peer="{Voter server-b9afba60-8aa3-15fb-33d1-a9849c0707e5 b9afba60-8aa3-15fb-33d1-a9849c0707e5}"
--- PASS: TestRaft_LeaderID_Propagated (0.92s)
=== RUN   TestRaft_LeadershipTransferInProgress
--- PASS: TestRaft_LeadershipTransferInProgress (0.00s)
=== RUN   TestRaft_LeadershipTransferPickServer
--- PASS: TestRaft_LeadershipTransferPickServer (0.00s)
=== RUN   TestRaft_LeadershipTransfer
2026-09-21T15:25:51.305Z [INFO]  server-abef95ad-08ab-70ef-5c0b-edc460f71319: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abef95ad-08ab-70ef-5c0b-edc460f71319 Address:abef95ad-08ab-70ef-5c0b-edc460f71319} {Suffrage:Voter ID:server-9da5cc58-16c7-1b8d-498e-a326505f09c3 Address:9da5cc58-16c7-1b8d-498e-a326505f09c3} {Suffrage:Voter ID:server-a16d63be-38c1-042e-78e5-c53ea878d4e2 Address:a16d63be-38c1-042e-78e5-c53ea878d4e2}]"
2026-09-21T15:25:51.305Z [INFO]  server-9da5cc58-16c7-1b8d-498e-a326505f09c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abef95ad-08ab-70ef-5c0b-edc460f71319 Address:abef95ad-08ab-70ef-5c0b-edc460f71319} {Suffrage:Voter ID:server-9da5cc58-16c7-1b8d-498e-a326505f09c3 Address:9da5cc58-16c7-1b8d-498e-a326505f09c3} {Suffrage:Voter ID:server-a16d63be-38c1-042e-78e5-c53ea878d4e2 Address:a16d63be-38c1-042e-78e5-c53ea878d4e2}]"
2026-09-21T15:25:51.305Z [INFO]  server-abef95ad-08ab-70ef-5c0b-edc460f71319: entering follower state: follower="Node at abef95ad-08ab-70ef-5c0b-edc460f71319 [Follower]" leader-address= leader-id=
2026-09-21T15:25:51.305Z [INFO]  server-9da5cc58-16c7-1b8d-498e-a326505f09c3: entering follower state: follower="Node at 9da5cc58-16c7-1b8d-498e-a326505f09c3 [Follower]" leader-address= leader-id=
2026-09-21T15:25:51.305Z [INFO]  server-a16d63be-38c1-042e-78e5-c53ea878d4e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-abef95ad-08ab-70ef-5c0b-edc460f71319 Address:abef95ad-08ab-70ef-5c0b-edc460f71319} {Suffrage:Voter ID:server-9da5cc58-16c7-1b8d-498e-a326505f09c3 Address:9da5cc58-16c7-1b8d-498e-a326505f09c3} {Suffrage:Voter ID:server-a16d63be-38c1-042e-78e5-c53ea878d4e2 Address:a16d63be-38c1-042e-78e5-c53ea878d4e2}]"
2026-09-21T15:25:51.305Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:51.305Z [INFO]  server-a16d63be-38c1-042e-78e5-c53ea878d4e2: entering follower state: follower="Node at a16d63be-38c1-042e-78e5-c53ea878d4e2 [Follower]" leader-address= leader-id=
2026-09-21T15:25:51.374Z [WARN]  server-9da5cc58-16c7-1b8d-498e-a326505f09c3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:51.374Z [INFO]  server-9da5cc58-16c7-1b8d-498e-a326505f09c3: entering candidate state: node="Node at 9da5cc58-16c7-1b8d-498e-a326505f09c3 [Candidate]" term=2
2026-09-21T15:25:51.374Z [INFO]  server-9da5cc58-16c7-1b8d-498e-a326505f09c3: election won: term=2 tally=2
2026-09-21T15:25:51.374Z [INFO]  server-9da5cc58-16c7-1b8d-498e-a326505f09c3: entering leader state: leader="Node at 9da5cc58-16c7-1b8d-498e-a326505f09c3 [Leader]"
2026-09-21T15:25:51.374Z [INFO]  server-9da5cc58-16c7-1b8d-498e-a326505f09c3: added peer, starting replication: peer=server-abef95ad-08ab-70ef-5c0b-edc460f71319
2026-09-21T15:25:51.374Z [INFO]  server-9da5cc58-16c7-1b8d-498e-a326505f09c3: added peer, starting replication: peer=server-a16d63be-38c1-042e-78e5-c53ea878d4e2
2026-09-21T15:25:51.374Z [INFO]  server-9da5cc58-16c7-1b8d-498e-a326505f09c3: pipelining replication: peer="{Voter server-a16d63be-38c1-042e-78e5-c53ea878d4e2 a16d63be-38c1-042e-78e5-c53ea878d4e2}"
2026-09-21T15:25:51.374Z [WARN]  server-a16d63be-38c1-042e-78e5-c53ea878d4e2: rejecting vote request since our last term is greater: candidate=9da5cc58-16c7-1b8d-498e-a326505f09c3 last-term=2 last-candidate-term=1
2026-09-21T15:25:51.374Z [INFO]  server-9da5cc58-16c7-1b8d-498e-a326505f09c3: pipelining replication: peer="{Voter server-abef95ad-08ab-70ef-5c0b-edc460f71319 abef95ad-08ab-70ef-5c0b-edc460f71319}"
2026-09-21T15:25:51.479Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:51.374639348 +0000 UTC m=+103.315809521 (1 nodes), 68.64904ms from start of poll, 69.04748ms from cluster start. Timeout at 2026-09-21 15:25:51.479642868 +0000 UTC m=+103.420813081, 105.00356ms after stability
2026-09-21T15:25:51.479Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:51.585Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:51.479910588 +0000 UTC m=+103.421080801 (1 nodes), 3.08µs from start of poll, 174.31876ms from cluster start. Timeout at 2026-09-21 15:25:51.584950628 +0000 UTC m=+103.526120801, 105.04ms after stability
2026-09-21T15:25:51.585Z [INFO]  server-abef95ad-08ab-70ef-5c0b-edc460f71319: entering candidate state: node="Node at abef95ad-08ab-70ef-5c0b-edc460f71319 [Candidate]" term=3
2026-09-21T15:25:51.585Z [INFO]  server-abef95ad-08ab-70ef-5c0b-edc460f71319: election won: term=3 tally=2
2026-09-21T15:25:51.585Z [INFO]  server-abef95ad-08ab-70ef-5c0b-edc460f71319: entering leader state: leader="Node at abef95ad-08ab-70ef-5c0b-edc460f71319 [Leader]"
2026-09-21T15:25:51.585Z [INFO]  server-abef95ad-08ab-70ef-5c0b-edc460f71319: added peer, starting replication: peer=server-9da5cc58-16c7-1b8d-498e-a326505f09c3
2026-09-21T15:25:51.585Z [INFO]  server-abef95ad-08ab-70ef-5c0b-edc460f71319: added peer, starting replication: peer=server-a16d63be-38c1-042e-78e5-c53ea878d4e2
2026-09-21T15:25:51.585Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:51.585Z [INFO]  server-9da5cc58-16c7-1b8d-498e-a326505f09c3: entering follower state: follower="Node at 9da5cc58-16c7-1b8d-498e-a326505f09c3 [Follower]" leader-address= leader-id=
2026-09-21T15:25:51.585Z [INFO]  server-abef95ad-08ab-70ef-5c0b-edc460f71319: pipelining replication: peer="{Voter server-a16d63be-38c1-042e-78e5-c53ea878d4e2 a16d63be-38c1-042e-78e5-c53ea878d4e2}"
2026-09-21T15:25:51.585Z [INFO]  server-9da5cc58-16c7-1b8d-498e-a326505f09c3: aborting pipeline replication: peer="{Voter server-a16d63be-38c1-042e-78e5-c53ea878d4e2 a16d63be-38c1-042e-78e5-c53ea878d4e2}"
2026-09-21T15:25:51.585Z [INFO]  server-9da5cc58-16c7-1b8d-498e-a326505f09c3: aborting pipeline replication: peer="{Voter server-abef95ad-08ab-70ef-5c0b-edc460f71319 abef95ad-08ab-70ef-5c0b-edc460f71319}"
2026-09-21T15:25:51.586Z [INFO]  server-abef95ad-08ab-70ef-5c0b-edc460f71319: pipelining replication: peer="{Voter server-9da5cc58-16c7-1b8d-498e-a326505f09c3 9da5cc58-16c7-1b8d-498e-a326505f09c3}"
2026-09-21T15:25:51.691Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:51.585832308 +0000 UTC m=+103.527002481 (1 nodes), 59.72µs from start of poll, 280.24044ms from cluster start. Timeout at 2026-09-21 15:25:51.690883548 +0000 UTC m=+103.632053721, 105.05124ms after stability
2026-09-21T15:25:51.691Z [INFO]  server-abef95ad-08ab-70ef-5c0b-edc460f71319: aborting pipeline replication: peer="{Voter server-a16d63be-38c1-042e-78e5-c53ea878d4e2 a16d63be-38c1-042e-78e5-c53ea878d4e2}"
2026-09-21T15:25:51.691Z [INFO]  server-abef95ad-08ab-70ef-5c0b-edc460f71319: aborting pipeline replication: peer="{Voter server-9da5cc58-16c7-1b8d-498e-a326505f09c3 9da5cc58-16c7-1b8d-498e-a326505f09c3}"
--- PASS: TestRaft_LeadershipTransfer (0.39s)
=== RUN   TestRaft_LeadershipTransferWithOneNode
2026-09-21T15:25:51.691Z [INFO]  server-4f7f0147-1ef5-3d7e-17c0-98a433590093: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f7f0147-1ef5-3d7e-17c0-98a433590093 Address:4f7f0147-1ef5-3d7e-17c0-98a433590093}]"
2026-09-21T15:25:51.692Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:51.692Z [INFO]  server-4f7f0147-1ef5-3d7e-17c0-98a433590093: entering follower state: follower="Node at 4f7f0147-1ef5-3d7e-17c0-98a433590093 [Follower]" leader-address= leader-id=
2026-09-21T15:25:51.760Z [WARN]  server-4f7f0147-1ef5-3d7e-17c0-98a433590093: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:51.760Z [INFO]  server-4f7f0147-1ef5-3d7e-17c0-98a433590093: entering candidate state: node="Node at 4f7f0147-1ef5-3d7e-17c0-98a433590093 [Candidate]" term=2
2026-09-21T15:25:51.761Z [INFO]  server-4f7f0147-1ef5-3d7e-17c0-98a433590093: election won: term=2 tally=1
2026-09-21T15:25:51.761Z [INFO]  server-4f7f0147-1ef5-3d7e-17c0-98a433590093: entering leader state: leader="Node at 4f7f0147-1ef5-3d7e-17c0-98a433590093 [Leader]"
2026-09-21T15:25:51.866Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:51.761108668 +0000 UTC m=+103.702278841 (1 nodes), 69.04864ms from start of poll, 69.24156ms from cluster start. Timeout at 2026-09-21 15:25:51.866125068 +0000 UTC m=+103.807295241, 105.0164ms after stability
--- PASS: TestRaft_LeadershipTransferWithOneNode (0.17s)
=== RUN   TestRaft_LeadershipTransferWithSevenNodes
2026-09-21T15:25:51.867Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22fc3b74-9872-35c5-8077-3b0c1293fabc Address:22fc3b74-9872-35c5-8077-3b0c1293fabc} {Suffrage:Voter ID:server-2572eb21-3200-8e13-b09e-a55c203e1c40 Address:2572eb21-3200-8e13-b09e-a55c203e1c40} {Suffrage:Voter ID:server-dd4665a8-f37b-a3a9-4143-564869b2655b Address:dd4665a8-f37b-a3a9-4143-564869b2655b} {Suffrage:Voter ID:server-10f95e55-9705-3090-f359-8cd2d3c91471 Address:10f95e55-9705-3090-f359-8cd2d3c91471} {Suffrage:Voter ID:server-aca7343f-7208-92b1-6d99-f675731331e9 Address:aca7343f-7208-92b1-6d99-f675731331e9} {Suffrage:Voter ID:server-67bd2b14-8fc9-eee2-256a-e5913327471b Address:67bd2b14-8fc9-eee2-256a-e5913327471b} {Suffrage:Voter ID:server-02749311-e0b7-5400-9782-757dd73c489d Address:02749311-e0b7-5400-9782-757dd73c489d}]"
2026-09-21T15:25:51.867Z [INFO]  server-2572eb21-3200-8e13-b09e-a55c203e1c40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22fc3b74-9872-35c5-8077-3b0c1293fabc Address:22fc3b74-9872-35c5-8077-3b0c1293fabc} {Suffrage:Voter ID:server-2572eb21-3200-8e13-b09e-a55c203e1c40 Address:2572eb21-3200-8e13-b09e-a55c203e1c40} {Suffrage:Voter ID:server-dd4665a8-f37b-a3a9-4143-564869b2655b Address:dd4665a8-f37b-a3a9-4143-564869b2655b} {Suffrage:Voter ID:server-10f95e55-9705-3090-f359-8cd2d3c91471 Address:10f95e55-9705-3090-f359-8cd2d3c91471} {Suffrage:Voter ID:server-aca7343f-7208-92b1-6d99-f675731331e9 Address:aca7343f-7208-92b1-6d99-f675731331e9} {Suffrage:Voter ID:server-67bd2b14-8fc9-eee2-256a-e5913327471b Address:67bd2b14-8fc9-eee2-256a-e5913327471b} {Suffrage:Voter ID:server-02749311-e0b7-5400-9782-757dd73c489d Address:02749311-e0b7-5400-9782-757dd73c489d}]"
2026-09-21T15:25:51.867Z [INFO]  server-dd4665a8-f37b-a3a9-4143-564869b2655b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22fc3b74-9872-35c5-8077-3b0c1293fabc Address:22fc3b74-9872-35c5-8077-3b0c1293fabc} {Suffrage:Voter ID:server-2572eb21-3200-8e13-b09e-a55c203e1c40 Address:2572eb21-3200-8e13-b09e-a55c203e1c40} {Suffrage:Voter ID:server-dd4665a8-f37b-a3a9-4143-564869b2655b Address:dd4665a8-f37b-a3a9-4143-564869b2655b} {Suffrage:Voter ID:server-10f95e55-9705-3090-f359-8cd2d3c91471 Address:10f95e55-9705-3090-f359-8cd2d3c91471} {Suffrage:Voter ID:server-aca7343f-7208-92b1-6d99-f675731331e9 Address:aca7343f-7208-92b1-6d99-f675731331e9} {Suffrage:Voter ID:server-67bd2b14-8fc9-eee2-256a-e5913327471b Address:67bd2b14-8fc9-eee2-256a-e5913327471b} {Suffrage:Voter ID:server-02749311-e0b7-5400-9782-757dd73c489d Address:02749311-e0b7-5400-9782-757dd73c489d}]"
2026-09-21T15:25:51.867Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22fc3b74-9872-35c5-8077-3b0c1293fabc Address:22fc3b74-9872-35c5-8077-3b0c1293fabc} {Suffrage:Voter ID:server-2572eb21-3200-8e13-b09e-a55c203e1c40 Address:2572eb21-3200-8e13-b09e-a55c203e1c40} {Suffrage:Voter ID:server-dd4665a8-f37b-a3a9-4143-564869b2655b Address:dd4665a8-f37b-a3a9-4143-564869b2655b} {Suffrage:Voter ID:server-10f95e55-9705-3090-f359-8cd2d3c91471 Address:10f95e55-9705-3090-f359-8cd2d3c91471} {Suffrage:Voter ID:server-aca7343f-7208-92b1-6d99-f675731331e9 Address:aca7343f-7208-92b1-6d99-f675731331e9} {Suffrage:Voter ID:server-67bd2b14-8fc9-eee2-256a-e5913327471b Address:67bd2b14-8fc9-eee2-256a-e5913327471b} {Suffrage:Voter ID:server-02749311-e0b7-5400-9782-757dd73c489d Address:02749311-e0b7-5400-9782-757dd73c489d}]"
2026-09-21T15:25:51.867Z [INFO]  server-aca7343f-7208-92b1-6d99-f675731331e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22fc3b74-9872-35c5-8077-3b0c1293fabc Address:22fc3b74-9872-35c5-8077-3b0c1293fabc} {Suffrage:Voter ID:server-2572eb21-3200-8e13-b09e-a55c203e1c40 Address:2572eb21-3200-8e13-b09e-a55c203e1c40} {Suffrage:Voter ID:server-dd4665a8-f37b-a3a9-4143-564869b2655b Address:dd4665a8-f37b-a3a9-4143-564869b2655b} {Suffrage:Voter ID:server-10f95e55-9705-3090-f359-8cd2d3c91471 Address:10f95e55-9705-3090-f359-8cd2d3c91471} {Suffrage:Voter ID:server-aca7343f-7208-92b1-6d99-f675731331e9 Address:aca7343f-7208-92b1-6d99-f675731331e9} {Suffrage:Voter ID:server-67bd2b14-8fc9-eee2-256a-e5913327471b Address:67bd2b14-8fc9-eee2-256a-e5913327471b} {Suffrage:Voter ID:server-02749311-e0b7-5400-9782-757dd73c489d Address:02749311-e0b7-5400-9782-757dd73c489d}]"
2026-09-21T15:25:51.867Z [INFO]  server-67bd2b14-8fc9-eee2-256a-e5913327471b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22fc3b74-9872-35c5-8077-3b0c1293fabc Address:22fc3b74-9872-35c5-8077-3b0c1293fabc} {Suffrage:Voter ID:server-2572eb21-3200-8e13-b09e-a55c203e1c40 Address:2572eb21-3200-8e13-b09e-a55c203e1c40} {Suffrage:Voter ID:server-dd4665a8-f37b-a3a9-4143-564869b2655b Address:dd4665a8-f37b-a3a9-4143-564869b2655b} {Suffrage:Voter ID:server-10f95e55-9705-3090-f359-8cd2d3c91471 Address:10f95e55-9705-3090-f359-8cd2d3c91471} {Suffrage:Voter ID:server-aca7343f-7208-92b1-6d99-f675731331e9 Address:aca7343f-7208-92b1-6d99-f675731331e9} {Suffrage:Voter ID:server-67bd2b14-8fc9-eee2-256a-e5913327471b Address:67bd2b14-8fc9-eee2-256a-e5913327471b} {Suffrage:Voter ID:server-02749311-e0b7-5400-9782-757dd73c489d Address:02749311-e0b7-5400-9782-757dd73c489d}]"
2026-09-21T15:25:51.868Z [INFO]  server-02749311-e0b7-5400-9782-757dd73c489d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22fc3b74-9872-35c5-8077-3b0c1293fabc Address:22fc3b74-9872-35c5-8077-3b0c1293fabc} {Suffrage:Voter ID:server-2572eb21-3200-8e13-b09e-a55c203e1c40 Address:2572eb21-3200-8e13-b09e-a55c203e1c40} {Suffrage:Voter ID:server-dd4665a8-f37b-a3a9-4143-564869b2655b Address:dd4665a8-f37b-a3a9-4143-564869b2655b} {Suffrage:Voter ID:server-10f95e55-9705-3090-f359-8cd2d3c91471 Address:10f95e55-9705-3090-f359-8cd2d3c91471} {Suffrage:Voter ID:server-aca7343f-7208-92b1-6d99-f675731331e9 Address:aca7343f-7208-92b1-6d99-f675731331e9} {Suffrage:Voter ID:server-67bd2b14-8fc9-eee2-256a-e5913327471b Address:67bd2b14-8fc9-eee2-256a-e5913327471b} {Suffrage:Voter ID:server-02749311-e0b7-5400-9782-757dd73c489d Address:02749311-e0b7-5400-9782-757dd73c489d}]"
2026-09-21T15:25:51.868Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:51.868Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: entering follower state: follower="Node at 22fc3b74-9872-35c5-8077-3b0c1293fabc [Follower]" leader-address= leader-id=
2026-09-21T15:25:51.868Z [INFO]  server-2572eb21-3200-8e13-b09e-a55c203e1c40: entering follower state: follower="Node at 2572eb21-3200-8e13-b09e-a55c203e1c40 [Follower]" leader-address= leader-id=
2026-09-21T15:25:51.868Z [INFO]  server-dd4665a8-f37b-a3a9-4143-564869b2655b: entering follower state: follower="Node at dd4665a8-f37b-a3a9-4143-564869b2655b [Follower]" leader-address= leader-id=
2026-09-21T15:25:51.868Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: entering follower state: follower="Node at 10f95e55-9705-3090-f359-8cd2d3c91471 [Follower]" leader-address= leader-id=
2026-09-21T15:25:51.868Z [INFO]  server-aca7343f-7208-92b1-6d99-f675731331e9: entering follower state: follower="Node at aca7343f-7208-92b1-6d99-f675731331e9 [Follower]" leader-address= leader-id=
2026-09-21T15:25:51.868Z [INFO]  server-67bd2b14-8fc9-eee2-256a-e5913327471b: entering follower state: follower="Node at 67bd2b14-8fc9-eee2-256a-e5913327471b [Follower]" leader-address= leader-id=
2026-09-21T15:25:51.868Z [INFO]  server-02749311-e0b7-5400-9782-757dd73c489d: entering follower state: follower="Node at 02749311-e0b7-5400-9782-757dd73c489d [Follower]" leader-address= leader-id=
2026-09-21T15:25:51.919Z [WARN]  server-10f95e55-9705-3090-f359-8cd2d3c91471: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:51.919Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: entering candidate state: node="Node at 10f95e55-9705-3090-f359-8cd2d3c91471 [Candidate]" term=2
2026-09-21T15:25:51.919Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: election won: term=2 tally=4
2026-09-21T15:25:51.919Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: entering leader state: leader="Node at 10f95e55-9705-3090-f359-8cd2d3c91471 [Leader]"
2026-09-21T15:25:51.919Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: added peer, starting replication: peer=server-22fc3b74-9872-35c5-8077-3b0c1293fabc
2026-09-21T15:25:51.919Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: added peer, starting replication: peer=server-2572eb21-3200-8e13-b09e-a55c203e1c40
2026-09-21T15:25:51.919Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: added peer, starting replication: peer=server-dd4665a8-f37b-a3a9-4143-564869b2655b
2026-09-21T15:25:51.919Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: added peer, starting replication: peer=server-aca7343f-7208-92b1-6d99-f675731331e9
2026-09-21T15:25:51.919Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: added peer, starting replication: peer=server-67bd2b14-8fc9-eee2-256a-e5913327471b
2026-09-21T15:25:51.919Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: added peer, starting replication: peer=server-02749311-e0b7-5400-9782-757dd73c489d
2026-09-21T15:25:51.919Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: pipelining replication: peer="{Voter server-02749311-e0b7-5400-9782-757dd73c489d 02749311-e0b7-5400-9782-757dd73c489d}"
2026-09-21T15:25:51.920Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: pipelining replication: peer="{Voter server-22fc3b74-9872-35c5-8077-3b0c1293fabc 22fc3b74-9872-35c5-8077-3b0c1293fabc}"
2026-09-21T15:25:51.920Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: pipelining replication: peer="{Voter server-2572eb21-3200-8e13-b09e-a55c203e1c40 2572eb21-3200-8e13-b09e-a55c203e1c40}"
2026-09-21T15:25:51.920Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: pipelining replication: peer="{Voter server-dd4665a8-f37b-a3a9-4143-564869b2655b dd4665a8-f37b-a3a9-4143-564869b2655b}"
2026-09-21T15:25:51.920Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: pipelining replication: peer="{Voter server-aca7343f-7208-92b1-6d99-f675731331e9 aca7343f-7208-92b1-6d99-f675731331e9}"
2026-09-21T15:25:51.920Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: pipelining replication: peer="{Voter server-67bd2b14-8fc9-eee2-256a-e5913327471b 67bd2b14-8fc9-eee2-256a-e5913327471b}"
2026-09-21T15:25:52.025Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:51.920039828 +0000 UTC m=+103.861210001 (1 nodes), 51.91924ms from start of poll, 52.84584ms from cluster start. Timeout at 2026-09-21 15:25:52.025041668 +0000 UTC m=+103.966211841, 105.00184ms after stability
2026-09-21T15:25:52.025Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:52.130Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:52.025527268 +0000 UTC m=+103.966697441 (6 nodes), 3.28µs from start of poll, 158.33328ms from cluster start. Timeout at 2026-09-21 15:25:52.130565348 +0000 UTC m=+104.071735521, 105.03808ms after stability
2026-09-21T15:25:52.131Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:52.236Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:52.131027428 +0000 UTC m=+104.072197601 (1 nodes), 2.72µs from start of poll, 263.83344ms from cluster start. Timeout at 2026-09-21 15:25:52.236068468 +0000 UTC m=+104.177238641, 105.04104ms after stability
2026-09-21T15:25:52.236Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: entering candidate state: node="Node at 22fc3b74-9872-35c5-8077-3b0c1293fabc [Candidate]" term=3
2026-09-21T15:25:52.236Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:52.236Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: entering follower state: follower="Node at 10f95e55-9705-3090-f359-8cd2d3c91471 [Follower]" leader-address= leader-id=
2026-09-21T15:25:52.236Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: aborting pipeline replication: peer="{Voter server-02749311-e0b7-5400-9782-757dd73c489d 02749311-e0b7-5400-9782-757dd73c489d}"
2026-09-21T15:25:52.236Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: election won: term=3 tally=4
2026-09-21T15:25:52.236Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: entering leader state: leader="Node at 22fc3b74-9872-35c5-8077-3b0c1293fabc [Leader]"
2026-09-21T15:25:52.236Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: added peer, starting replication: peer=server-2572eb21-3200-8e13-b09e-a55c203e1c40
2026-09-21T15:25:52.237Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: added peer, starting replication: peer=server-dd4665a8-f37b-a3a9-4143-564869b2655b
2026-09-21T15:25:52.237Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: added peer, starting replication: peer=server-10f95e55-9705-3090-f359-8cd2d3c91471
2026-09-21T15:25:52.237Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: added peer, starting replication: peer=server-aca7343f-7208-92b1-6d99-f675731331e9
2026-09-21T15:25:52.237Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: added peer, starting replication: peer=server-67bd2b14-8fc9-eee2-256a-e5913327471b
2026-09-21T15:25:52.237Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: added peer, starting replication: peer=server-02749311-e0b7-5400-9782-757dd73c489d
2026-09-21T15:25:52.237Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: pipelining replication: peer="{Voter server-02749311-e0b7-5400-9782-757dd73c489d 02749311-e0b7-5400-9782-757dd73c489d}"
2026-09-21T15:25:52.237Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: aborting pipeline replication: peer="{Voter server-22fc3b74-9872-35c5-8077-3b0c1293fabc 22fc3b74-9872-35c5-8077-3b0c1293fabc}"
2026-09-21T15:25:52.237Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: aborting pipeline replication: peer="{Voter server-2572eb21-3200-8e13-b09e-a55c203e1c40 2572eb21-3200-8e13-b09e-a55c203e1c40}"
2026-09-21T15:25:52.237Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: aborting pipeline replication: peer="{Voter server-dd4665a8-f37b-a3a9-4143-564869b2655b dd4665a8-f37b-a3a9-4143-564869b2655b}"
2026-09-21T15:25:52.237Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: aborting pipeline replication: peer="{Voter server-aca7343f-7208-92b1-6d99-f675731331e9 aca7343f-7208-92b1-6d99-f675731331e9}"
2026-09-21T15:25:52.237Z [INFO]  server-10f95e55-9705-3090-f359-8cd2d3c91471: aborting pipeline replication: peer="{Voter server-67bd2b14-8fc9-eee2-256a-e5913327471b 67bd2b14-8fc9-eee2-256a-e5913327471b}"
2026-09-21T15:25:52.237Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: pipelining replication: peer="{Voter server-2572eb21-3200-8e13-b09e-a55c203e1c40 2572eb21-3200-8e13-b09e-a55c203e1c40}"
2026-09-21T15:25:52.237Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: pipelining replication: peer="{Voter server-dd4665a8-f37b-a3a9-4143-564869b2655b dd4665a8-f37b-a3a9-4143-564869b2655b}"
2026-09-21T15:25:52.237Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: pipelining replication: peer="{Voter server-10f95e55-9705-3090-f359-8cd2d3c91471 10f95e55-9705-3090-f359-8cd2d3c91471}"
2026-09-21T15:25:52.237Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: pipelining replication: peer="{Voter server-aca7343f-7208-92b1-6d99-f675731331e9 aca7343f-7208-92b1-6d99-f675731331e9}"
2026-09-21T15:25:52.237Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: pipelining replication: peer="{Voter server-67bd2b14-8fc9-eee2-256a-e5913327471b 67bd2b14-8fc9-eee2-256a-e5913327471b}"
2026-09-21T15:25:52.342Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:52.237301868 +0000 UTC m=+104.178472041 (1 nodes), 663.48µs from start of poll, 370.10788ms from cluster start. Timeout at 2026-09-21 15:25:52.342303788 +0000 UTC m=+104.283473961, 105.00192ms after stability
2026-09-21T15:25:52.342Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: aborting pipeline replication: peer="{Voter server-2572eb21-3200-8e13-b09e-a55c203e1c40 2572eb21-3200-8e13-b09e-a55c203e1c40}"
2026-09-21T15:25:52.342Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: aborting pipeline replication: peer="{Voter server-dd4665a8-f37b-a3a9-4143-564869b2655b dd4665a8-f37b-a3a9-4143-564869b2655b}"
2026-09-21T15:25:52.342Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: aborting pipeline replication: peer="{Voter server-10f95e55-9705-3090-f359-8cd2d3c91471 10f95e55-9705-3090-f359-8cd2d3c91471}"
2026-09-21T15:25:52.342Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: aborting pipeline replication: peer="{Voter server-aca7343f-7208-92b1-6d99-f675731331e9 aca7343f-7208-92b1-6d99-f675731331e9}"
2026-09-21T15:25:52.342Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: aborting pipeline replication: peer="{Voter server-67bd2b14-8fc9-eee2-256a-e5913327471b 67bd2b14-8fc9-eee2-256a-e5913327471b}"
2026-09-21T15:25:52.342Z [INFO]  server-22fc3b74-9872-35c5-8077-3b0c1293fabc: aborting pipeline replication: peer="{Voter server-02749311-e0b7-5400-9782-757dd73c489d 02749311-e0b7-5400-9782-757dd73c489d}"
--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.48s)
=== RUN   TestRaft_LeadershipTransferToInvalidID
2026-09-21T15:25:52.343Z [INFO]  server-7dd8d8ff-46ae-09d9-3a1d-0397644a2ba5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7dd8d8ff-46ae-09d9-3a1d-0397644a2ba5 Address:7dd8d8ff-46ae-09d9-3a1d-0397644a2ba5} {Suffrage:Voter ID:server-b74eccba-1b2f-d996-1249-0e6f4d5146a6 Address:b74eccba-1b2f-d996-1249-0e6f4d5146a6} {Suffrage:Voter ID:server-e8e9e746-19fa-6192-e621-519da114ac79 Address:e8e9e746-19fa-6192-e621-519da114ac79}]"
2026-09-21T15:25:52.343Z [INFO]  server-b74eccba-1b2f-d996-1249-0e6f4d5146a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7dd8d8ff-46ae-09d9-3a1d-0397644a2ba5 Address:7dd8d8ff-46ae-09d9-3a1d-0397644a2ba5} {Suffrage:Voter ID:server-b74eccba-1b2f-d996-1249-0e6f4d5146a6 Address:b74eccba-1b2f-d996-1249-0e6f4d5146a6} {Suffrage:Voter ID:server-e8e9e746-19fa-6192-e621-519da114ac79 Address:e8e9e746-19fa-6192-e621-519da114ac79}]"
2026-09-21T15:25:52.344Z [INFO]  server-e8e9e746-19fa-6192-e621-519da114ac79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7dd8d8ff-46ae-09d9-3a1d-0397644a2ba5 Address:7dd8d8ff-46ae-09d9-3a1d-0397644a2ba5} {Suffrage:Voter ID:server-b74eccba-1b2f-d996-1249-0e6f4d5146a6 Address:b74eccba-1b2f-d996-1249-0e6f4d5146a6} {Suffrage:Voter ID:server-e8e9e746-19fa-6192-e621-519da114ac79 Address:e8e9e746-19fa-6192-e621-519da114ac79}]"
2026-09-21T15:25:52.344Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:52.344Z [INFO]  server-7dd8d8ff-46ae-09d9-3a1d-0397644a2ba5: entering follower state: follower="Node at 7dd8d8ff-46ae-09d9-3a1d-0397644a2ba5 [Follower]" leader-address= leader-id=
2026-09-21T15:25:52.344Z [INFO]  server-b74eccba-1b2f-d996-1249-0e6f4d5146a6: entering follower state: follower="Node at b74eccba-1b2f-d996-1249-0e6f4d5146a6 [Follower]" leader-address= leader-id=
2026-09-21T15:25:52.344Z [INFO]  server-e8e9e746-19fa-6192-e621-519da114ac79: entering follower state: follower="Node at e8e9e746-19fa-6192-e621-519da114ac79 [Follower]" leader-address= leader-id=
2026-09-21T15:25:52.417Z [WARN]  server-e8e9e746-19fa-6192-e621-519da114ac79: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:52.417Z [INFO]  server-e8e9e746-19fa-6192-e621-519da114ac79: entering candidate state: node="Node at e8e9e746-19fa-6192-e621-519da114ac79 [Candidate]" term=2
2026-09-21T15:25:52.417Z [INFO]  server-e8e9e746-19fa-6192-e621-519da114ac79: election won: term=2 tally=2
2026-09-21T15:25:52.417Z [INFO]  server-e8e9e746-19fa-6192-e621-519da114ac79: entering leader state: leader="Node at e8e9e746-19fa-6192-e621-519da114ac79 [Leader]"
2026-09-21T15:25:52.417Z [INFO]  server-e8e9e746-19fa-6192-e621-519da114ac79: added peer, starting replication: peer=server-7dd8d8ff-46ae-09d9-3a1d-0397644a2ba5
2026-09-21T15:25:52.417Z [INFO]  server-e8e9e746-19fa-6192-e621-519da114ac79: added peer, starting replication: peer=server-b74eccba-1b2f-d996-1249-0e6f4d5146a6
2026-09-21T15:25:52.417Z [INFO]  server-e8e9e746-19fa-6192-e621-519da114ac79: pipelining replication: peer="{Voter server-b74eccba-1b2f-d996-1249-0e6f4d5146a6 b74eccba-1b2f-d996-1249-0e6f4d5146a6}"
2026-09-21T15:25:52.417Z [INFO]  server-e8e9e746-19fa-6192-e621-519da114ac79: pipelining replication: peer="{Voter server-7dd8d8ff-46ae-09d9-3a1d-0397644a2ba5 7dd8d8ff-46ae-09d9-3a1d-0397644a2ba5}"
2026-09-21T15:25:52.523Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:52.417499268 +0000 UTC m=+104.358669441 (1 nodes), 73.397ms from start of poll, 73.8132ms from cluster start. Timeout at 2026-09-21 15:25:52.522504028 +0000 UTC m=+104.463674201, 105.00476ms after stability
2026-09-21T15:25:52.523Z [INFO]  server-e8e9e746-19fa-6192-e621-519da114ac79: aborting pipeline replication: peer="{Voter server-7dd8d8ff-46ae-09d9-3a1d-0397644a2ba5 7dd8d8ff-46ae-09d9-3a1d-0397644a2ba5}"
2026-09-21T15:25:52.523Z [INFO]  server-e8e9e746-19fa-6192-e621-519da114ac79: aborting pipeline replication: peer="{Voter server-b74eccba-1b2f-d996-1249-0e6f4d5146a6 b74eccba-1b2f-d996-1249-0e6f4d5146a6}"
--- PASS: TestRaft_LeadershipTransferToInvalidID (0.18s)
=== RUN   TestRaft_LeadershipTransferToInvalidAddress
2026-09-21T15:25:52.525Z [INFO]  server-3c91076d-d9e3-d06a-ebf8-c11da10694a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c91076d-d9e3-d06a-ebf8-c11da10694a1 Address:3c91076d-d9e3-d06a-ebf8-c11da10694a1} {Suffrage:Voter ID:server-6845c226-e219-48cd-9133-f1eb665d5a64 Address:6845c226-e219-48cd-9133-f1eb665d5a64} {Suffrage:Voter ID:server-8e016a49-282b-cf27-53a2-a3753414f735 Address:8e016a49-282b-cf27-53a2-a3753414f735}]"
2026-09-21T15:25:52.525Z [INFO]  server-3c91076d-d9e3-d06a-ebf8-c11da10694a1: entering follower state: follower="Node at 3c91076d-d9e3-d06a-ebf8-c11da10694a1 [Follower]" leader-address= leader-id=
2026-09-21T15:25:52.525Z [INFO]  server-6845c226-e219-48cd-9133-f1eb665d5a64: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c91076d-d9e3-d06a-ebf8-c11da10694a1 Address:3c91076d-d9e3-d06a-ebf8-c11da10694a1} {Suffrage:Voter ID:server-6845c226-e219-48cd-9133-f1eb665d5a64 Address:6845c226-e219-48cd-9133-f1eb665d5a64} {Suffrage:Voter ID:server-8e016a49-282b-cf27-53a2-a3753414f735 Address:8e016a49-282b-cf27-53a2-a3753414f735}]"
2026-09-21T15:25:52.525Z [INFO]  server-8e016a49-282b-cf27-53a2-a3753414f735: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c91076d-d9e3-d06a-ebf8-c11da10694a1 Address:3c91076d-d9e3-d06a-ebf8-c11da10694a1} {Suffrage:Voter ID:server-6845c226-e219-48cd-9133-f1eb665d5a64 Address:6845c226-e219-48cd-9133-f1eb665d5a64} {Suffrage:Voter ID:server-8e016a49-282b-cf27-53a2-a3753414f735 Address:8e016a49-282b-cf27-53a2-a3753414f735}]"
2026-09-21T15:25:52.525Z [INFO]  server-6845c226-e219-48cd-9133-f1eb665d5a64: entering follower state: follower="Node at 6845c226-e219-48cd-9133-f1eb665d5a64 [Follower]" leader-address= leader-id=
2026-09-21T15:25:52.525Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:52.525Z [INFO]  server-8e016a49-282b-cf27-53a2-a3753414f735: entering follower state: follower="Node at 8e016a49-282b-cf27-53a2-a3753414f735 [Follower]" leader-address= leader-id=
2026-09-21T15:25:52.579Z [WARN]  server-3c91076d-d9e3-d06a-ebf8-c11da10694a1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:52.579Z [INFO]  server-3c91076d-d9e3-d06a-ebf8-c11da10694a1: entering candidate state: node="Node at 3c91076d-d9e3-d06a-ebf8-c11da10694a1 [Candidate]" term=2
2026-09-21T15:25:52.579Z [INFO]  server-3c91076d-d9e3-d06a-ebf8-c11da10694a1: election won: term=2 tally=2
2026-09-21T15:25:52.579Z [INFO]  server-3c91076d-d9e3-d06a-ebf8-c11da10694a1: entering leader state: leader="Node at 3c91076d-d9e3-d06a-ebf8-c11da10694a1 [Leader]"
2026-09-21T15:25:52.579Z [INFO]  server-3c91076d-d9e3-d06a-ebf8-c11da10694a1: added peer, starting replication: peer=server-6845c226-e219-48cd-9133-f1eb665d5a64
2026-09-21T15:25:52.579Z [INFO]  server-3c91076d-d9e3-d06a-ebf8-c11da10694a1: added peer, starting replication: peer=server-8e016a49-282b-cf27-53a2-a3753414f735
2026-09-21T15:25:52.580Z [INFO]  server-3c91076d-d9e3-d06a-ebf8-c11da10694a1: pipelining replication: peer="{Voter server-8e016a49-282b-cf27-53a2-a3753414f735 8e016a49-282b-cf27-53a2-a3753414f735}"
2026-09-21T15:25:52.580Z [INFO]  server-3c91076d-d9e3-d06a-ebf8-c11da10694a1: pipelining replication: peer="{Voter server-6845c226-e219-48cd-9133-f1eb665d5a64 6845c226-e219-48cd-9133-f1eb665d5a64}"
2026-09-21T15:25:52.685Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:52.580315148 +0000 UTC m=+104.521485321 (2 nodes), 54.78968ms from start of poll, 55.33988ms from cluster start. Timeout at 2026-09-21 15:25:52.685321828 +0000 UTC m=+104.626492001, 105.00668ms after stability
2026-09-21T15:25:52.685Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:52.791Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:52.685613068 +0000 UTC m=+104.626783241 (1 nodes), 2.16µs from start of poll, 160.6378ms from cluster start. Timeout at 2026-09-21 15:25:52.790639308 +0000 UTC m=+104.731809481, 105.02624ms after stability
2026-09-21T15:25:52.791Z [INFO]  server-3c91076d-d9e3-d06a-ebf8-c11da10694a1: aborting pipeline replication: peer="{Voter server-8e016a49-282b-cf27-53a2-a3753414f735 8e016a49-282b-cf27-53a2-a3753414f735}"
2026-09-21T15:25:52.791Z [INFO]  server-3c91076d-d9e3-d06a-ebf8-c11da10694a1: aborting pipeline replication: peer="{Voter server-6845c226-e219-48cd-9133-f1eb665d5a64 6845c226-e219-48cd-9133-f1eb665d5a64}"
--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.27s)
=== RUN   TestRaft_LeadershipTransferToBehindServer
2026-09-21T15:25:52.792Z [INFO]  server-15207d0d-ad7d-4268-6522-909d99e3ddaa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15207d0d-ad7d-4268-6522-909d99e3ddaa Address:15207d0d-ad7d-4268-6522-909d99e3ddaa} {Suffrage:Voter ID:server-b48d3e06-86b4-1cb5-e511-fafe99bfd108 Address:b48d3e06-86b4-1cb5-e511-fafe99bfd108} {Suffrage:Voter ID:server-531863b4-d09d-0bac-fdda-1869cafdcb52 Address:531863b4-d09d-0bac-fdda-1869cafdcb52}]"
2026-09-21T15:25:52.792Z [INFO]  server-b48d3e06-86b4-1cb5-e511-fafe99bfd108: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15207d0d-ad7d-4268-6522-909d99e3ddaa Address:15207d0d-ad7d-4268-6522-909d99e3ddaa} {Suffrage:Voter ID:server-b48d3e06-86b4-1cb5-e511-fafe99bfd108 Address:b48d3e06-86b4-1cb5-e511-fafe99bfd108} {Suffrage:Voter ID:server-531863b4-d09d-0bac-fdda-1869cafdcb52 Address:531863b4-d09d-0bac-fdda-1869cafdcb52}]"
2026-09-21T15:25:52.792Z [INFO]  server-531863b4-d09d-0bac-fdda-1869cafdcb52: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15207d0d-ad7d-4268-6522-909d99e3ddaa Address:15207d0d-ad7d-4268-6522-909d99e3ddaa} {Suffrage:Voter ID:server-b48d3e06-86b4-1cb5-e511-fafe99bfd108 Address:b48d3e06-86b4-1cb5-e511-fafe99bfd108} {Suffrage:Voter ID:server-531863b4-d09d-0bac-fdda-1869cafdcb52 Address:531863b4-d09d-0bac-fdda-1869cafdcb52}]"
2026-09-21T15:25:52.792Z [INFO]  server-15207d0d-ad7d-4268-6522-909d99e3ddaa: entering follower state: follower="Node at 15207d0d-ad7d-4268-6522-909d99e3ddaa [Follower]" leader-address= leader-id=
2026-09-21T15:25:52.792Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:52.792Z [INFO]  server-b48d3e06-86b4-1cb5-e511-fafe99bfd108: entering follower state: follower="Node at b48d3e06-86b4-1cb5-e511-fafe99bfd108 [Follower]" leader-address= leader-id=
2026-09-21T15:25:52.792Z [INFO]  server-531863b4-d09d-0bac-fdda-1869cafdcb52: entering follower state: follower="Node at 531863b4-d09d-0bac-fdda-1869cafdcb52 [Follower]" leader-address= leader-id=
2026-09-21T15:25:52.873Z [WARN]  server-b48d3e06-86b4-1cb5-e511-fafe99bfd108: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:52.873Z [INFO]  server-b48d3e06-86b4-1cb5-e511-fafe99bfd108: entering candidate state: node="Node at b48d3e06-86b4-1cb5-e511-fafe99bfd108 [Candidate]" term=2
2026-09-21T15:25:52.873Z [INFO]  server-b48d3e06-86b4-1cb5-e511-fafe99bfd108: election won: term=2 tally=2
2026-09-21T15:25:52.873Z [INFO]  server-b48d3e06-86b4-1cb5-e511-fafe99bfd108: entering leader state: leader="Node at b48d3e06-86b4-1cb5-e511-fafe99bfd108 [Leader]"
2026-09-21T15:25:52.873Z [INFO]  server-b48d3e06-86b4-1cb5-e511-fafe99bfd108: added peer, starting replication: peer=server-15207d0d-ad7d-4268-6522-909d99e3ddaa
2026-09-21T15:25:52.873Z [INFO]  server-b48d3e06-86b4-1cb5-e511-fafe99bfd108: added peer, starting replication: peer=server-531863b4-d09d-0bac-fdda-1869cafdcb52
2026-09-21T15:25:52.873Z [INFO]  server-b48d3e06-86b4-1cb5-e511-fafe99bfd108: pipelining replication: peer="{Voter server-531863b4-d09d-0bac-fdda-1869cafdcb52 531863b4-d09d-0bac-fdda-1869cafdcb52}"
2026-09-21T15:25:52.873Z [INFO]  server-b48d3e06-86b4-1cb5-e511-fafe99bfd108: pipelining replication: peer="{Voter server-15207d0d-ad7d-4268-6522-909d99e3ddaa 15207d0d-ad7d-4268-6522-909d99e3ddaa}"
2026-09-21T15:25:52.979Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:52.873670228 +0000 UTC m=+104.814840401 (1 nodes), 81.24448ms from start of poll, 81.67896ms from cluster start. Timeout at 2026-09-21 15:25:52.978680028 +0000 UTC m=+104.919850201, 105.0098ms after stability
2026-09-21T15:25:52.980Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:53.085Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:52.980018708 +0000 UTC m=+104.921188881 (2 nodes), 3.52µs from start of poll, 188.02744ms from cluster start. Timeout at 2026-09-21 15:25:53.085051908 +0000 UTC m=+105.026222081, 105.0332ms after stability
2026-09-21T15:25:53.140Z [INFO]  server-15207d0d-ad7d-4268-6522-909d99e3ddaa: entering candidate state: node="Node at 15207d0d-ad7d-4268-6522-909d99e3ddaa [Candidate]" term=3
2026-09-21T15:25:53.140Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:53.140Z [INFO]  server-15207d0d-ad7d-4268-6522-909d99e3ddaa: election won: term=3 tally=2
2026-09-21T15:25:53.140Z [INFO]  server-15207d0d-ad7d-4268-6522-909d99e3ddaa: entering leader state: leader="Node at 15207d0d-ad7d-4268-6522-909d99e3ddaa [Leader]"
2026-09-21T15:25:53.140Z [INFO]  server-15207d0d-ad7d-4268-6522-909d99e3ddaa: added peer, starting replication: peer=server-b48d3e06-86b4-1cb5-e511-fafe99bfd108
2026-09-21T15:25:53.140Z [INFO]  server-15207d0d-ad7d-4268-6522-909d99e3ddaa: added peer, starting replication: peer=server-531863b4-d09d-0bac-fdda-1869cafdcb52
2026-09-21T15:25:53.140Z [INFO]  server-15207d0d-ad7d-4268-6522-909d99e3ddaa: pipelining replication: peer="{Voter server-531863b4-d09d-0bac-fdda-1869cafdcb52 531863b4-d09d-0bac-fdda-1869cafdcb52}"
2026-09-21T15:25:53.140Z [INFO]  server-b48d3e06-86b4-1cb5-e511-fafe99bfd108: entering follower state: follower="Node at b48d3e06-86b4-1cb5-e511-fafe99bfd108 [Follower]" leader-address=15207d0d-ad7d-4268-6522-909d99e3ddaa leader-id=server-15207d0d-ad7d-4268-6522-909d99e3ddaa
2026-09-21T15:25:53.140Z [INFO]  server-b48d3e06-86b4-1cb5-e511-fafe99bfd108: aborting pipeline replication: peer="{Voter server-531863b4-d09d-0bac-fdda-1869cafdcb52 531863b4-d09d-0bac-fdda-1869cafdcb52}"
2026-09-21T15:25:53.140Z [INFO]  server-15207d0d-ad7d-4268-6522-909d99e3ddaa: pipelining replication: peer="{Voter server-b48d3e06-86b4-1cb5-e511-fafe99bfd108 b48d3e06-86b4-1cb5-e511-fafe99bfd108}"
2026-09-21T15:25:53.140Z [INFO]  server-b48d3e06-86b4-1cb5-e511-fafe99bfd108: aborting pipeline replication: peer="{Voter server-15207d0d-ad7d-4268-6522-909d99e3ddaa 15207d0d-ad7d-4268-6522-909d99e3ddaa}"
2026-09-21T15:25:53.140Z [WARN]  server-b48d3e06-86b4-1cb5-e511-fafe99bfd108: rejecting vote request since our last term is greater: candidate=15207d0d-ad7d-4268-6522-909d99e3ddaa last-term=3 last-candidate-term=2
2026-09-21T15:25:53.247Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:53.140980668 +0000 UTC m=+105.082150841 (1 nodes), 678.32µs from start of poll, 348.9894ms from cluster start. Timeout at 2026-09-21 15:25:53.245989348 +0000 UTC m=+105.187159521, 105.00868ms after stability
2026-09-21T15:25:53.247Z [INFO]  server-15207d0d-ad7d-4268-6522-909d99e3ddaa: aborting pipeline replication: peer="{Voter server-b48d3e06-86b4-1cb5-e511-fafe99bfd108 b48d3e06-86b4-1cb5-e511-fafe99bfd108}"
2026-09-21T15:25:53.247Z [INFO]  server-15207d0d-ad7d-4268-6522-909d99e3ddaa: aborting pipeline replication: peer="{Voter server-531863b4-d09d-0bac-fdda-1869cafdcb52 531863b4-d09d-0bac-fdda-1869cafdcb52}"
--- PASS: TestRaft_LeadershipTransferToBehindServer (0.46s)
=== RUN   TestRaft_LeadershipTransferToItself
2026-09-21T15:25:53.248Z [INFO]  server-255a691c-5c36-f3de-8042-8112e3fa227f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-255a691c-5c36-f3de-8042-8112e3fa227f Address:255a691c-5c36-f3de-8042-8112e3fa227f} {Suffrage:Voter ID:server-02c601bc-ba63-ec1c-37ae-198f6ae0fd42 Address:02c601bc-ba63-ec1c-37ae-198f6ae0fd42} {Suffrage:Voter ID:server-321b32ba-b4d2-0e83-e7cd-d640aad390cb Address:321b32ba-b4d2-0e83-e7cd-d640aad390cb}]"
2026-09-21T15:25:53.248Z [INFO]  server-02c601bc-ba63-ec1c-37ae-198f6ae0fd42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-255a691c-5c36-f3de-8042-8112e3fa227f Address:255a691c-5c36-f3de-8042-8112e3fa227f} {Suffrage:Voter ID:server-02c601bc-ba63-ec1c-37ae-198f6ae0fd42 Address:02c601bc-ba63-ec1c-37ae-198f6ae0fd42} {Suffrage:Voter ID:server-321b32ba-b4d2-0e83-e7cd-d640aad390cb Address:321b32ba-b4d2-0e83-e7cd-d640aad390cb}]"
2026-09-21T15:25:53.249Z [INFO]  server-321b32ba-b4d2-0e83-e7cd-d640aad390cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-255a691c-5c36-f3de-8042-8112e3fa227f Address:255a691c-5c36-f3de-8042-8112e3fa227f} {Suffrage:Voter ID:server-02c601bc-ba63-ec1c-37ae-198f6ae0fd42 Address:02c601bc-ba63-ec1c-37ae-198f6ae0fd42} {Suffrage:Voter ID:server-321b32ba-b4d2-0e83-e7cd-d640aad390cb Address:321b32ba-b4d2-0e83-e7cd-d640aad390cb}]"
2026-09-21T15:25:53.249Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:53.249Z [INFO]  server-255a691c-5c36-f3de-8042-8112e3fa227f: entering follower state: follower="Node at 255a691c-5c36-f3de-8042-8112e3fa227f [Follower]" leader-address= leader-id=
2026-09-21T15:25:53.249Z [INFO]  server-02c601bc-ba63-ec1c-37ae-198f6ae0fd42: entering follower state: follower="Node at 02c601bc-ba63-ec1c-37ae-198f6ae0fd42 [Follower]" leader-address= leader-id=
2026-09-21T15:25:53.249Z [INFO]  server-321b32ba-b4d2-0e83-e7cd-d640aad390cb: entering follower state: follower="Node at 321b32ba-b4d2-0e83-e7cd-d640aad390cb [Follower]" leader-address= leader-id=
2026-09-21T15:25:53.310Z [WARN]  server-02c601bc-ba63-ec1c-37ae-198f6ae0fd42: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:53.310Z [INFO]  server-02c601bc-ba63-ec1c-37ae-198f6ae0fd42: entering candidate state: node="Node at 02c601bc-ba63-ec1c-37ae-198f6ae0fd42 [Candidate]" term=2
2026-09-21T15:25:53.310Z [INFO]  server-02c601bc-ba63-ec1c-37ae-198f6ae0fd42: election won: term=2 tally=2
2026-09-21T15:25:53.310Z [INFO]  server-02c601bc-ba63-ec1c-37ae-198f6ae0fd42: entering leader state: leader="Node at 02c601bc-ba63-ec1c-37ae-198f6ae0fd42 [Leader]"
2026-09-21T15:25:53.310Z [INFO]  server-02c601bc-ba63-ec1c-37ae-198f6ae0fd42: added peer, starting replication: peer=server-255a691c-5c36-f3de-8042-8112e3fa227f
2026-09-21T15:25:53.310Z [INFO]  server-02c601bc-ba63-ec1c-37ae-198f6ae0fd42: added peer, starting replication: peer=server-321b32ba-b4d2-0e83-e7cd-d640aad390cb
2026-09-21T15:25:53.310Z [INFO]  server-02c601bc-ba63-ec1c-37ae-198f6ae0fd42: pipelining replication: peer="{Voter server-255a691c-5c36-f3de-8042-8112e3fa227f 255a691c-5c36-f3de-8042-8112e3fa227f}"
2026-09-21T15:25:53.310Z [INFO]  server-02c601bc-ba63-ec1c-37ae-198f6ae0fd42: pipelining replication: peer="{Voter server-321b32ba-b4d2-0e83-e7cd-d640aad390cb 321b32ba-b4d2-0e83-e7cd-d640aad390cb}"
2026-09-21T15:25:53.415Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:53.310477828 +0000 UTC m=+105.251648001 (1 nodes), 61.3542ms from start of poll, 61.8104ms from cluster start. Timeout at 2026-09-21 15:25:53.415479068 +0000 UTC m=+105.356649241, 105.00124ms after stability
2026-09-21T15:25:53.415Z [INFO]  server-02c601bc-ba63-ec1c-37ae-198f6ae0fd42: cannot transfer leadership to itself
2026-09-21T15:25:53.415Z [INFO]  server-02c601bc-ba63-ec1c-37ae-198f6ae0fd42: aborting pipeline replication: peer="{Voter server-321b32ba-b4d2-0e83-e7cd-d640aad390cb 321b32ba-b4d2-0e83-e7cd-d640aad390cb}"
2026-09-21T15:25:53.415Z [INFO]  server-02c601bc-ba63-ec1c-37ae-198f6ae0fd42: aborting pipeline replication: peer="{Voter server-255a691c-5c36-f3de-8042-8112e3fa227f 255a691c-5c36-f3de-8042-8112e3fa227f}"
--- PASS: TestRaft_LeadershipTransferToItself (0.17s)
=== RUN   TestRaft_LeadershipTransferLeaderRejectsClientRequests
2026-09-21T15:25:53.416Z [INFO]  server-17b9e7df-386e-0161-24c1-c04bc003d3b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17b9e7df-386e-0161-24c1-c04bc003d3b5 Address:17b9e7df-386e-0161-24c1-c04bc003d3b5} {Suffrage:Voter ID:server-cedd6800-e9b3-b4ad-03bc-611e814d70e4 Address:cedd6800-e9b3-b4ad-03bc-611e814d70e4} {Suffrage:Voter ID:server-67663dc3-d1b7-4686-be3d-2ac77b1d0767 Address:67663dc3-d1b7-4686-be3d-2ac77b1d0767}]"
2026-09-21T15:25:53.416Z [INFO]  server-cedd6800-e9b3-b4ad-03bc-611e814d70e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17b9e7df-386e-0161-24c1-c04bc003d3b5 Address:17b9e7df-386e-0161-24c1-c04bc003d3b5} {Suffrage:Voter ID:server-cedd6800-e9b3-b4ad-03bc-611e814d70e4 Address:cedd6800-e9b3-b4ad-03bc-611e814d70e4} {Suffrage:Voter ID:server-67663dc3-d1b7-4686-be3d-2ac77b1d0767 Address:67663dc3-d1b7-4686-be3d-2ac77b1d0767}]"
2026-09-21T15:25:53.416Z [INFO]  server-67663dc3-d1b7-4686-be3d-2ac77b1d0767: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17b9e7df-386e-0161-24c1-c04bc003d3b5 Address:17b9e7df-386e-0161-24c1-c04bc003d3b5} {Suffrage:Voter ID:server-cedd6800-e9b3-b4ad-03bc-611e814d70e4 Address:cedd6800-e9b3-b4ad-03bc-611e814d70e4} {Suffrage:Voter ID:server-67663dc3-d1b7-4686-be3d-2ac77b1d0767 Address:67663dc3-d1b7-4686-be3d-2ac77b1d0767}]"
2026-09-21T15:25:53.416Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:53.416Z [INFO]  server-17b9e7df-386e-0161-24c1-c04bc003d3b5: entering follower state: follower="Node at 17b9e7df-386e-0161-24c1-c04bc003d3b5 [Follower]" leader-address= leader-id=
2026-09-21T15:25:53.416Z [INFO]  server-cedd6800-e9b3-b4ad-03bc-611e814d70e4: entering follower state: follower="Node at cedd6800-e9b3-b4ad-03bc-611e814d70e4 [Follower]" leader-address= leader-id=
2026-09-21T15:25:53.416Z [INFO]  server-67663dc3-d1b7-4686-be3d-2ac77b1d0767: entering follower state: follower="Node at 67663dc3-d1b7-4686-be3d-2ac77b1d0767 [Follower]" leader-address= leader-id=
2026-09-21T15:25:53.481Z [WARN]  server-cedd6800-e9b3-b4ad-03bc-611e814d70e4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:53.481Z [INFO]  server-cedd6800-e9b3-b4ad-03bc-611e814d70e4: entering candidate state: node="Node at cedd6800-e9b3-b4ad-03bc-611e814d70e4 [Candidate]" term=2
2026-09-21T15:25:53.482Z [INFO]  server-cedd6800-e9b3-b4ad-03bc-611e814d70e4: election won: term=2 tally=2
2026-09-21T15:25:53.482Z [INFO]  server-cedd6800-e9b3-b4ad-03bc-611e814d70e4: entering leader state: leader="Node at cedd6800-e9b3-b4ad-03bc-611e814d70e4 [Leader]"
2026-09-21T15:25:53.482Z [INFO]  server-cedd6800-e9b3-b4ad-03bc-611e814d70e4: added peer, starting replication: peer=server-17b9e7df-386e-0161-24c1-c04bc003d3b5
2026-09-21T15:25:53.482Z [INFO]  server-cedd6800-e9b3-b4ad-03bc-611e814d70e4: added peer, starting replication: peer=server-67663dc3-d1b7-4686-be3d-2ac77b1d0767
2026-09-21T15:25:53.482Z [INFO]  server-cedd6800-e9b3-b4ad-03bc-611e814d70e4: pipelining replication: peer="{Voter server-17b9e7df-386e-0161-24c1-c04bc003d3b5 17b9e7df-386e-0161-24c1-c04bc003d3b5}"
2026-09-21T15:25:53.482Z [INFO]  server-cedd6800-e9b3-b4ad-03bc-611e814d70e4: pipelining replication: peer="{Voter server-67663dc3-d1b7-4686-be3d-2ac77b1d0767 67663dc3-d1b7-4686-be3d-2ac77b1d0767}"
2026-09-21T15:25:53.588Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:53.482144028 +0000 UTC m=+105.423314201 (1 nodes), 65.41076ms from start of poll, 65.78928ms from cluster start. Timeout at 2026-09-21 15:25:53.587147148 +0000 UTC m=+105.528317321, 105.00312ms after stability
    raft_test.go:2457: waiting on future 0
    raft_test.go:2457: waiting on future 1
    raft_test.go:2457: waiting on future 2
    raft_test.go:2457: waiting on future 3
    raft_test.go:2457: waiting on future 4
    raft_test.go:2457: waiting on future 5
    raft_test.go:2457: waiting on future 6
2026-09-21T15:25:53.588Z [INFO]  server-cedd6800-e9b3-b4ad-03bc-611e814d70e4: aborting pipeline replication: peer="{Voter server-67663dc3-d1b7-4686-be3d-2ac77b1d0767 67663dc3-d1b7-4686-be3d-2ac77b1d0767}"
2026-09-21T15:25:53.588Z [INFO]  server-cedd6800-e9b3-b4ad-03bc-611e814d70e4: aborting pipeline replication: peer="{Voter server-17b9e7df-386e-0161-24c1-c04bc003d3b5 17b9e7df-386e-0161-24c1-c04bc003d3b5}"
2026-09-21T15:25:54.089Z [ERROR] server-cedd6800-e9b3-b4ad-03bc-611e814d70e4: failed to heartbeat to: peer=17b9e7df-386e-0161-24c1-c04bc003d3b5 backoff time=10ms error="command timed out"
2026-09-21T15:25:54.089Z [ERROR] server-cedd6800-e9b3-b4ad-03bc-611e814d70e4: failed to heartbeat to: peer=67663dc3-d1b7-4686-be3d-2ac77b1d0767 backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.67s)
=== RUN   TestRaft_LeadershipTransferLeaderReplicationTimeout
2026-09-21T15:25:54.090Z [INFO]  server-dab7a11b-1a04-197c-05a2-579b715cda8d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dab7a11b-1a04-197c-05a2-579b715cda8d Address:dab7a11b-1a04-197c-05a2-579b715cda8d} {Suffrage:Voter ID:server-c78320b1-6f78-999a-a193-d037d40b4fdd Address:c78320b1-6f78-999a-a193-d037d40b4fdd} {Suffrage:Voter ID:server-7f430f46-c238-06f7-0489-5688ee577e7c Address:7f430f46-c238-06f7-0489-5688ee577e7c}]"
2026-09-21T15:25:54.090Z [INFO]  server-c78320b1-6f78-999a-a193-d037d40b4fdd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dab7a11b-1a04-197c-05a2-579b715cda8d Address:dab7a11b-1a04-197c-05a2-579b715cda8d} {Suffrage:Voter ID:server-c78320b1-6f78-999a-a193-d037d40b4fdd Address:c78320b1-6f78-999a-a193-d037d40b4fdd} {Suffrage:Voter ID:server-7f430f46-c238-06f7-0489-5688ee577e7c Address:7f430f46-c238-06f7-0489-5688ee577e7c}]"
2026-09-21T15:25:54.090Z [INFO]  server-dab7a11b-1a04-197c-05a2-579b715cda8d: entering follower state: follower="Node at dab7a11b-1a04-197c-05a2-579b715cda8d [Follower]" leader-address= leader-id=
2026-09-21T15:25:54.090Z [INFO]  server-7f430f46-c238-06f7-0489-5688ee577e7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dab7a11b-1a04-197c-05a2-579b715cda8d Address:dab7a11b-1a04-197c-05a2-579b715cda8d} {Suffrage:Voter ID:server-c78320b1-6f78-999a-a193-d037d40b4fdd Address:c78320b1-6f78-999a-a193-d037d40b4fdd} {Suffrage:Voter ID:server-7f430f46-c238-06f7-0489-5688ee577e7c Address:7f430f46-c238-06f7-0489-5688ee577e7c}]"
2026-09-21T15:25:54.090Z [INFO]  server-c78320b1-6f78-999a-a193-d037d40b4fdd: entering follower state: follower="Node at c78320b1-6f78-999a-a193-d037d40b4fdd [Follower]" leader-address= leader-id=
2026-09-21T15:25:54.090Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:54.090Z [INFO]  server-7f430f46-c238-06f7-0489-5688ee577e7c: entering follower state: follower="Node at 7f430f46-c238-06f7-0489-5688ee577e7c [Follower]" leader-address= leader-id=
2026-09-21T15:25:54.150Z [WARN]  server-c78320b1-6f78-999a-a193-d037d40b4fdd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:54.150Z [INFO]  server-c78320b1-6f78-999a-a193-d037d40b4fdd: entering candidate state: node="Node at c78320b1-6f78-999a-a193-d037d40b4fdd [Candidate]" term=2
2026-09-21T15:25:54.150Z [INFO]  server-c78320b1-6f78-999a-a193-d037d40b4fdd: election won: term=2 tally=2
2026-09-21T15:25:54.150Z [INFO]  server-c78320b1-6f78-999a-a193-d037d40b4fdd: entering leader state: leader="Node at c78320b1-6f78-999a-a193-d037d40b4fdd [Leader]"
2026-09-21T15:25:54.150Z [INFO]  server-c78320b1-6f78-999a-a193-d037d40b4fdd: added peer, starting replication: peer=server-dab7a11b-1a04-197c-05a2-579b715cda8d
2026-09-21T15:25:54.150Z [INFO]  server-c78320b1-6f78-999a-a193-d037d40b4fdd: added peer, starting replication: peer=server-7f430f46-c238-06f7-0489-5688ee577e7c
2026-09-21T15:25:54.150Z [INFO]  server-c78320b1-6f78-999a-a193-d037d40b4fdd: pipelining replication: peer="{Voter server-7f430f46-c238-06f7-0489-5688ee577e7c 7f430f46-c238-06f7-0489-5688ee577e7c}"
2026-09-21T15:25:54.151Z [INFO]  server-c78320b1-6f78-999a-a193-d037d40b4fdd: pipelining replication: peer="{Voter server-dab7a11b-1a04-197c-05a2-579b715cda8d dab7a11b-1a04-197c-05a2-579b715cda8d}"
2026-09-21T15:25:54.256Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:54.150846228 +0000 UTC m=+106.092016401 (1 nodes), 60.24436ms from start of poll, 60.64188ms from cluster start. Timeout at 2026-09-21 15:25:54.255848268 +0000 UTC m=+106.197018441, 105.00204ms after stability
2026-09-21T15:25:54.256Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:54.362Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:54.256802068 +0000 UTC m=+106.197972241 (2 nodes), 2.68µs from start of poll, 166.59772ms from cluster start. Timeout at 2026-09-21 15:25:54.361848588 +0000 UTC m=+106.303018761, 105.04652ms after stability
2026-09-21T15:25:55.298Z [ERROR] server-c78320b1-6f78-999a-a193-d037d40b4fdd: failed to pipeline appendEntries: peer="{Voter server-dab7a11b-1a04-197c-05a2-579b715cda8d dab7a11b-1a04-197c-05a2-579b715cda8d}" error="command enqueue timeout"
2026-09-21T15:25:55.298Z [INFO]  server-c78320b1-6f78-999a-a193-d037d40b4fdd: aborting pipeline replication: peer="{Voter server-dab7a11b-1a04-197c-05a2-579b715cda8d dab7a11b-1a04-197c-05a2-579b715cda8d}"
2026-09-21T15:25:55.298Z [ERROR] server-c78320b1-6f78-999a-a193-d037d40b4fdd: failed to pipeline appendEntries: peer="{Voter server-7f430f46-c238-06f7-0489-5688ee577e7c 7f430f46-c238-06f7-0489-5688ee577e7c}" error="command enqueue timeout"
2026-09-21T15:25:55.299Z [INFO]  server-c78320b1-6f78-999a-a193-d037d40b4fdd: aborting pipeline replication: peer="{Voter server-7f430f46-c238-06f7-0489-5688ee577e7c 7f430f46-c238-06f7-0489-5688ee577e7c}"
2026-09-21T15:25:55.302Z [ERROR] server-c78320b1-6f78-999a-a193-d037d40b4fdd: failed to heartbeat to: peer=dab7a11b-1a04-197c-05a2-579b715cda8d backoff time=10ms error="send timed out"
2026-09-21T15:25:55.302Z [ERROR] server-c78320b1-6f78-999a-a193-d037d40b4fdd: failed to heartbeat to: peer=7f430f46-c238-06f7-0489-5688ee577e7c backoff time=10ms error="send timed out"
2026-09-21T15:25:55.799Z [ERROR] server-c78320b1-6f78-999a-a193-d037d40b4fdd: failed to appendEntries to: peer="{Voter server-dab7a11b-1a04-197c-05a2-579b715cda8d dab7a11b-1a04-197c-05a2-579b715cda8d}" error="send timed out"
2026-09-21T15:25:55.799Z [ERROR] server-c78320b1-6f78-999a-a193-d037d40b4fdd: failed to appendEntries to: peer="{Voter server-7f430f46-c238-06f7-0489-5688ee577e7c 7f430f46-c238-06f7-0489-5688ee577e7c}" error="send timed out"
2026-09-21T15:25:55.822Z [ERROR] server-c78320b1-6f78-999a-a193-d037d40b4fdd: failed to heartbeat to: peer=dab7a11b-1a04-197c-05a2-579b715cda8d backoff time=10ms error="send timed out"
2026-09-21T15:25:55.822Z [ERROR] server-c78320b1-6f78-999a-a193-d037d40b4fdd: failed to heartbeat to: peer=7f430f46-c238-06f7-0489-5688ee577e7c backoff time=10ms error="send timed out"
2026-09-21T15:25:56.299Z [ERROR] server-c78320b1-6f78-999a-a193-d037d40b4fdd: failed to appendEntries to: peer="{Voter server-dab7a11b-1a04-197c-05a2-579b715cda8d dab7a11b-1a04-197c-05a2-579b715cda8d}" error="send timed out"
2026-09-21T15:25:56.341Z [ERROR] server-c78320b1-6f78-999a-a193-d037d40b4fdd: failed to heartbeat to: peer=dab7a11b-1a04-197c-05a2-579b715cda8d backoff time=10ms error="send timed out"
--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (2.25s)
=== RUN   TestRaft_LeadershipTransferIgnoresNonvoters
2026-09-21T15:25:56.342Z [INFO]  server-8b937c46-f8e1-ce9b-e947-de6fbd645835: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b937c46-f8e1-ce9b-e947-de6fbd645835 Address:8b937c46-f8e1-ce9b-e947-de6fbd645835} {Suffrage:Voter ID:server-9413936d-e7e4-6b15-655d-19fc027a7d6a Address:9413936d-e7e4-6b15-655d-19fc027a7d6a}]"
2026-09-21T15:25:56.342Z [INFO]  server-9413936d-e7e4-6b15-655d-19fc027a7d6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b937c46-f8e1-ce9b-e947-de6fbd645835 Address:8b937c46-f8e1-ce9b-e947-de6fbd645835} {Suffrage:Voter ID:server-9413936d-e7e4-6b15-655d-19fc027a7d6a Address:9413936d-e7e4-6b15-655d-19fc027a7d6a}]"
2026-09-21T15:25:56.342Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:56.342Z [INFO]  server-8b937c46-f8e1-ce9b-e947-de6fbd645835: entering follower state: follower="Node at 8b937c46-f8e1-ce9b-e947-de6fbd645835 [Follower]" leader-address= leader-id=
2026-09-21T15:25:56.342Z [INFO]  server-9413936d-e7e4-6b15-655d-19fc027a7d6a: entering follower state: follower="Node at 9413936d-e7e4-6b15-655d-19fc027a7d6a [Follower]" leader-address= leader-id=
2026-09-21T15:25:56.396Z [WARN]  server-8b937c46-f8e1-ce9b-e947-de6fbd645835: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:56.396Z [INFO]  server-8b937c46-f8e1-ce9b-e947-de6fbd645835: entering candidate state: node="Node at 8b937c46-f8e1-ce9b-e947-de6fbd645835 [Candidate]" term=2
2026-09-21T15:25:56.396Z [INFO]  server-8b937c46-f8e1-ce9b-e947-de6fbd645835: election won: term=2 tally=2
2026-09-21T15:25:56.396Z [INFO]  server-8b937c46-f8e1-ce9b-e947-de6fbd645835: entering leader state: leader="Node at 8b937c46-f8e1-ce9b-e947-de6fbd645835 [Leader]"
2026-09-21T15:25:56.396Z [INFO]  server-8b937c46-f8e1-ce9b-e947-de6fbd645835: added peer, starting replication: peer=server-9413936d-e7e4-6b15-655d-19fc027a7d6a
2026-09-21T15:25:56.396Z [INFO]  server-8b937c46-f8e1-ce9b-e947-de6fbd645835: pipelining replication: peer="{Voter server-9413936d-e7e4-6b15-655d-19fc027a7d6a 9413936d-e7e4-6b15-655d-19fc027a7d6a}"
2026-09-21T15:25:56.502Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:56.396541428 +0000 UTC m=+108.337711601 (1 nodes), 53.70508ms from start of poll, 53.98568ms from cluster start. Timeout at 2026-09-21 15:25:56.501545108 +0000 UTC m=+108.442715281, 105.00368ms after stability
2026-09-21T15:25:56.502Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:56.608Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:56.502840268 +0000 UTC m=+108.444010441 (1 nodes), 840ns from start of poll, 160.28452ms from cluster start. Timeout at 2026-09-21 15:25:56.607856388 +0000 UTC m=+108.549026561, 105.01612ms after stability
2026-09-21T15:25:56.608Z [INFO]  server-8b937c46-f8e1-ce9b-e947-de6fbd645835: updating configuration: command=DemoteVoter server-id=server-9413936d-e7e4-6b15-655d-19fc027a7d6a server-addr= servers="[{Suffrage:Voter ID:server-8b937c46-f8e1-ce9b-e947-de6fbd645835 Address:8b937c46-f8e1-ce9b-e947-de6fbd645835} {Suffrage:Nonvoter ID:server-9413936d-e7e4-6b15-655d-19fc027a7d6a Address:9413936d-e7e4-6b15-655d-19fc027a7d6a}]"
2026-09-21T15:25:56.608Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:56.713Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:56.608285108 +0000 UTC m=+108.549455281 (1 nodes), 840ns from start of poll, 265.72936ms from cluster start. Timeout at 2026-09-21 15:25:56.713305548 +0000 UTC m=+108.654475721, 105.02044ms after stability
2026-09-21T15:25:56.713Z [INFO]  server-8b937c46-f8e1-ce9b-e947-de6fbd645835: aborting pipeline replication: peer="{Voter server-9413936d-e7e4-6b15-655d-19fc027a7d6a 9413936d-e7e4-6b15-655d-19fc027a7d6a}"
--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.37s)
=== RUN   TestRaft_LeadershipTransferStopRightAway
--- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s)
=== RUN   TestRaft_GetConfigurationNoBootstrap
2026-09-21T15:25:56.714Z [INFO]  server-ef4fe9c7-7d35-d722-b839-fec559c58ac8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef4fe9c7-7d35-d722-b839-fec559c58ac8 Address:ef4fe9c7-7d35-d722-b839-fec559c58ac8} {Suffrage:Voter ID:server-3fbb9fe4-0b58-36c9-db22-db80782c9d45 Address:3fbb9fe4-0b58-36c9-db22-db80782c9d45}]"
2026-09-21T15:25:56.714Z [INFO]  server-3fbb9fe4-0b58-36c9-db22-db80782c9d45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef4fe9c7-7d35-d722-b839-fec559c58ac8 Address:ef4fe9c7-7d35-d722-b839-fec559c58ac8} {Suffrage:Voter ID:server-3fbb9fe4-0b58-36c9-db22-db80782c9d45 Address:3fbb9fe4-0b58-36c9-db22-db80782c9d45}]"
2026-09-21T15:25:56.714Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:56.714Z [INFO]  server-ef4fe9c7-7d35-d722-b839-fec559c58ac8: entering follower state: follower="Node at ef4fe9c7-7d35-d722-b839-fec559c58ac8 [Follower]" leader-address= leader-id=
2026-09-21T15:25:56.714Z [INFO]  server-3fbb9fe4-0b58-36c9-db22-db80782c9d45: entering follower state: follower="Node at 3fbb9fe4-0b58-36c9-db22-db80782c9d45 [Follower]" leader-address= leader-id=
2026-09-21T15:25:56.766Z [WARN]  server-3fbb9fe4-0b58-36c9-db22-db80782c9d45: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:56.766Z [INFO]  server-3fbb9fe4-0b58-36c9-db22-db80782c9d45: entering candidate state: node="Node at 3fbb9fe4-0b58-36c9-db22-db80782c9d45 [Candidate]" term=2
2026-09-21T15:25:56.766Z [INFO]  server-3fbb9fe4-0b58-36c9-db22-db80782c9d45: election won: term=2 tally=2
2026-09-21T15:25:56.766Z [INFO]  server-3fbb9fe4-0b58-36c9-db22-db80782c9d45: entering leader state: leader="Node at 3fbb9fe4-0b58-36c9-db22-db80782c9d45 [Leader]"
2026-09-21T15:25:56.766Z [INFO]  server-3fbb9fe4-0b58-36c9-db22-db80782c9d45: added peer, starting replication: peer=server-ef4fe9c7-7d35-d722-b839-fec559c58ac8
2026-09-21T15:25:56.767Z [INFO]  server-3fbb9fe4-0b58-36c9-db22-db80782c9d45: pipelining replication: peer="{Voter server-ef4fe9c7-7d35-d722-b839-fec559c58ac8 ef4fe9c7-7d35-d722-b839-fec559c58ac8}"
2026-09-21T15:25:56.872Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:56.766967148 +0000 UTC m=+108.708137361 (1 nodes), 52.30948ms from start of poll, 52.59748ms from cluster start. Timeout at 2026-09-21 15:25:56.871969908 +0000 UTC m=+108.813140081, 105.00272ms after stability
2026-09-21T15:25:56.873Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:56.978Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:56.873064028 +0000 UTC m=+108.814234201 (1 nodes), 2.36µs from start of poll, 158.69432ms from cluster start. Timeout at 2026-09-21 15:25:56.978099748 +0000 UTC m=+108.919269921, 105.03572ms after stability
2026-09-21T15:25:56.989Z [INFO]  server-3fbb9fe4-0b58-36c9-db22-db80782c9d45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef4fe9c7-7d35-d722-b839-fec559c58ac8 Address:ef4fe9c7-7d35-d722-b839-fec559c58ac8} {Suffrage:Voter ID:server-3fbb9fe4-0b58-36c9-db22-db80782c9d45 Address:3fbb9fe4-0b58-36c9-db22-db80782c9d45}]"
2026-09-21T15:25:56.989Z [INFO]  server-3fbb9fe4-0b58-36c9-db22-db80782c9d45: aborting pipeline replication: peer="{Voter server-ef4fe9c7-7d35-d722-b839-fec559c58ac8 ef4fe9c7-7d35-d722-b839-fec559c58ac8}"
--- PASS: TestRaft_GetConfigurationNoBootstrap (0.28s)
=== RUN   TestRaft_LogStoreIsMonotonic
2026-09-21T15:25:56.990Z [INFO]  server-4810461d-f24f-65d6-7095-3f8dfc25fdac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4810461d-f24f-65d6-7095-3f8dfc25fdac Address:4810461d-f24f-65d6-7095-3f8dfc25fdac}]"
2026-09-21T15:25:56.990Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:56.990Z [INFO]  server-4810461d-f24f-65d6-7095-3f8dfc25fdac: entering follower state: follower="Node at 4810461d-f24f-65d6-7095-3f8dfc25fdac [Follower]" leader-address= leader-id=
2026-09-21T15:25:57.067Z [WARN]  server-4810461d-f24f-65d6-7095-3f8dfc25fdac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:57.067Z [INFO]  server-4810461d-f24f-65d6-7095-3f8dfc25fdac: entering candidate state: node="Node at 4810461d-f24f-65d6-7095-3f8dfc25fdac [Candidate]" term=2
2026-09-21T15:25:57.067Z [INFO]  server-4810461d-f24f-65d6-7095-3f8dfc25fdac: election won: term=2 tally=1
2026-09-21T15:25:57.067Z [INFO]  server-4810461d-f24f-65d6-7095-3f8dfc25fdac: entering leader state: leader="Node at 4810461d-f24f-65d6-7095-3f8dfc25fdac [Leader]"
2026-09-21T15:25:57.172Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:57.067671548 +0000 UTC m=+109.008841721 (1 nodes), 77.5474ms from start of poll, 77.712ms from cluster start. Timeout at 2026-09-21 15:25:57.172674628 +0000 UTC m=+109.113844841, 105.00312ms after stability
--- PASS: TestRaft_LogStoreIsMonotonic (0.18s)
=== RUN   TestRaft_CacheLogWithStoreError
2026-09-21T15:25:57.173Z [INFO]  server-68717c2d-35d0-686d-a9e5-d6aa0fa536ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68717c2d-35d0-686d-a9e5-d6aa0fa536ee Address:68717c2d-35d0-686d-a9e5-d6aa0fa536ee} {Suffrage:Voter ID:server-2fb77f7e-f463-4fa0-ba07-5608a45d7655 Address:2fb77f7e-f463-4fa0-ba07-5608a45d7655}]"
2026-09-21T15:25:57.173Z [INFO]  server-2fb77f7e-f463-4fa0-ba07-5608a45d7655: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68717c2d-35d0-686d-a9e5-d6aa0fa536ee Address:68717c2d-35d0-686d-a9e5-d6aa0fa536ee} {Suffrage:Voter ID:server-2fb77f7e-f463-4fa0-ba07-5608a45d7655 Address:2fb77f7e-f463-4fa0-ba07-5608a45d7655}]"
2026-09-21T15:25:57.173Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:57.173Z [INFO]  server-68717c2d-35d0-686d-a9e5-d6aa0fa536ee: entering follower state: follower="Node at 68717c2d-35d0-686d-a9e5-d6aa0fa536ee [Follower]" leader-address= leader-id=
2026-09-21T15:25:57.173Z [INFO]  server-2fb77f7e-f463-4fa0-ba07-5608a45d7655: entering follower state: follower="Node at 2fb77f7e-f463-4fa0-ba07-5608a45d7655 [Follower]" leader-address= leader-id=
2026-09-21T15:25:57.267Z [WARN]  server-2fb77f7e-f463-4fa0-ba07-5608a45d7655: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:57.267Z [INFO]  server-2fb77f7e-f463-4fa0-ba07-5608a45d7655: entering candidate state: node="Node at 2fb77f7e-f463-4fa0-ba07-5608a45d7655 [Candidate]" term=2
2026-09-21T15:25:57.267Z [INFO]  server-2fb77f7e-f463-4fa0-ba07-5608a45d7655: election won: term=2 tally=2
2026-09-21T15:25:57.267Z [INFO]  server-2fb77f7e-f463-4fa0-ba07-5608a45d7655: entering leader state: leader="Node at 2fb77f7e-f463-4fa0-ba07-5608a45d7655 [Leader]"
2026-09-21T15:25:57.267Z [INFO]  server-2fb77f7e-f463-4fa0-ba07-5608a45d7655: added peer, starting replication: peer=server-68717c2d-35d0-686d-a9e5-d6aa0fa536ee
2026-09-21T15:25:57.267Z [INFO]  server-2fb77f7e-f463-4fa0-ba07-5608a45d7655: pipelining replication: peer="{Voter server-68717c2d-35d0-686d-a9e5-d6aa0fa536ee 68717c2d-35d0-686d-a9e5-d6aa0fa536ee}"
2026-09-21T15:25:57.373Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:57.267644268 +0000 UTC m=+109.208814441 (1 nodes), 93.94828ms from start of poll, 94.252ms from cluster start. Timeout at 2026-09-21 15:25:57.372648828 +0000 UTC m=+109.313819001, 105.00456ms after stability
2026-09-21T15:25:57.373Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:57.478Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:57.373137868 +0000 UTC m=+109.314308041 (1 nodes), 2.52µs from start of poll, 199.7456ms from cluster start. Timeout at 2026-09-21 15:25:57.478158868 +0000 UTC m=+109.419329041, 105.021ms after stability
2026-09-21T15:25:57.479Z [ERROR] server-68717c2d-35d0-686d-a9e5-d6aa0fa536ee: failed to append to logs: error="unable to store logs within log store, err: "some error""
2026-09-21T15:25:57.479Z [WARN]  server-68717c2d-35d0-686d-a9e5-d6aa0fa536ee: failed to get previous log: previous-index=9 last-index=7 error="log not found"
2026-09-21T15:25:57.479Z [INFO]  server-2fb77f7e-f463-4fa0-ba07-5608a45d7655: aborting pipeline replication: peer="{Voter server-68717c2d-35d0-686d-a9e5-d6aa0fa536ee 68717c2d-35d0-686d-a9e5-d6aa0fa536ee}"
2026-09-21T15:25:57.487Z [INFO]  server-2fb77f7e-f463-4fa0-ba07-5608a45d7655: pipelining replication: peer="{Voter server-68717c2d-35d0-686d-a9e5-d6aa0fa536ee 68717c2d-35d0-686d-a9e5-d6aa0fa536ee}"
2026-09-21T15:25:57.487Z [INFO]  server-68717c2d-35d0-686d-a9e5-d6aa0fa536ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68717c2d-35d0-686d-a9e5-d6aa0fa536ee Address:68717c2d-35d0-686d-a9e5-d6aa0fa536ee} {Suffrage:Voter ID:server-2fb77f7e-f463-4fa0-ba07-5608a45d7655 Address:2fb77f7e-f463-4fa0-ba07-5608a45d7655}]"
2026-09-21T15:25:57.488Z [INFO]  server-2fb77f7e-f463-4fa0-ba07-5608a45d7655: aborting pipeline replication: peer="{Voter server-68717c2d-35d0-686d-a9e5-d6aa0fa536ee 68717c2d-35d0-686d-a9e5-d6aa0fa536ee}"
--- PASS: TestRaft_CacheLogWithStoreError (0.32s)
=== RUN   TestRaft_ReloadConfig
2026-09-21T15:25:57.488Z [INFO]  server-9d4e7cfa-7266-b2b4-3e3c-e6156689b991: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d4e7cfa-7266-b2b4-3e3c-e6156689b991 Address:9d4e7cfa-7266-b2b4-3e3c-e6156689b991}]"
--- PASS: TestRaft_ReloadConfig (0.00s)
=== RUN   TestRaft_ReloadConfigValidates
2026-09-21T15:25:57.489Z [INFO]  server-3607c964-be22-e67e-a476-a2ae9813ff28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3607c964-be22-e67e-a476-a2ae9813ff28 Address:3607c964-be22-e67e-a476-a2ae9813ff28}]"
--- PASS: TestRaft_ReloadConfigValidates (0.00s)
=== RUN   TestRaft_InstallSnapshot_InvalidPeers
2026-09-21T15:25:57.489Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)"
--- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s)
=== RUN   TestRaft_VoteNotGranted_WhenNodeNotInCluster
2026-09-21T15:25:57.489Z [INFO]  server-b042f783-4529-51e1-8824-f0cc102502ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b042f783-4529-51e1-8824-f0cc102502ca Address:b042f783-4529-51e1-8824-f0cc102502ca} {Suffrage:Voter ID:server-c3672c88-8aab-a920-7cc8-4b0a7b3a72d3 Address:c3672c88-8aab-a920-7cc8-4b0a7b3a72d3} {Suffrage:Voter ID:server-d82bbd3e-0995-78a1-05cc-c00e0072ac28 Address:d82bbd3e-0995-78a1-05cc-c00e0072ac28}]"
2026-09-21T15:25:57.490Z [INFO]  server-b042f783-4529-51e1-8824-f0cc102502ca: entering follower state: follower="Node at b042f783-4529-51e1-8824-f0cc102502ca [Follower]" leader-address= leader-id=
2026-09-21T15:25:57.490Z [INFO]  server-c3672c88-8aab-a920-7cc8-4b0a7b3a72d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b042f783-4529-51e1-8824-f0cc102502ca Address:b042f783-4529-51e1-8824-f0cc102502ca} {Suffrage:Voter ID:server-c3672c88-8aab-a920-7cc8-4b0a7b3a72d3 Address:c3672c88-8aab-a920-7cc8-4b0a7b3a72d3} {Suffrage:Voter ID:server-d82bbd3e-0995-78a1-05cc-c00e0072ac28 Address:d82bbd3e-0995-78a1-05cc-c00e0072ac28}]"
2026-09-21T15:25:57.490Z [INFO]  server-c3672c88-8aab-a920-7cc8-4b0a7b3a72d3: entering follower state: follower="Node at c3672c88-8aab-a920-7cc8-4b0a7b3a72d3 [Follower]" leader-address= leader-id=
2026-09-21T15:25:57.490Z [INFO]  server-d82bbd3e-0995-78a1-05cc-c00e0072ac28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b042f783-4529-51e1-8824-f0cc102502ca Address:b042f783-4529-51e1-8824-f0cc102502ca} {Suffrage:Voter ID:server-c3672c88-8aab-a920-7cc8-4b0a7b3a72d3 Address:c3672c88-8aab-a920-7cc8-4b0a7b3a72d3} {Suffrage:Voter ID:server-d82bbd3e-0995-78a1-05cc-c00e0072ac28 Address:d82bbd3e-0995-78a1-05cc-c00e0072ac28}]"
2026-09-21T15:25:57.490Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:57.490Z [INFO]  server-d82bbd3e-0995-78a1-05cc-c00e0072ac28: entering follower state: follower="Node at d82bbd3e-0995-78a1-05cc-c00e0072ac28 [Follower]" leader-address= leader-id=
2026-09-21T15:25:57.554Z [WARN]  server-b042f783-4529-51e1-8824-f0cc102502ca: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:57.554Z [INFO]  server-b042f783-4529-51e1-8824-f0cc102502ca: entering candidate state: node="Node at b042f783-4529-51e1-8824-f0cc102502ca [Candidate]" term=2
2026-09-21T15:25:57.554Z [INFO]  server-b042f783-4529-51e1-8824-f0cc102502ca: election won: term=2 tally=2
2026-09-21T15:25:57.554Z [INFO]  server-b042f783-4529-51e1-8824-f0cc102502ca: entering leader state: leader="Node at b042f783-4529-51e1-8824-f0cc102502ca [Leader]"
2026-09-21T15:25:57.554Z [INFO]  server-b042f783-4529-51e1-8824-f0cc102502ca: added peer, starting replication: peer=server-c3672c88-8aab-a920-7cc8-4b0a7b3a72d3
2026-09-21T15:25:57.554Z [INFO]  server-b042f783-4529-51e1-8824-f0cc102502ca: added peer, starting replication: peer=server-d82bbd3e-0995-78a1-05cc-c00e0072ac28
2026-09-21T15:25:57.554Z [INFO]  server-b042f783-4529-51e1-8824-f0cc102502ca: pipelining replication: peer="{Voter server-c3672c88-8aab-a920-7cc8-4b0a7b3a72d3 c3672c88-8aab-a920-7cc8-4b0a7b3a72d3}"
2026-09-21T15:25:57.554Z [INFO]  server-b042f783-4529-51e1-8824-f0cc102502ca: pipelining replication: peer="{Voter server-d82bbd3e-0995-78a1-05cc-c00e0072ac28 d82bbd3e-0995-78a1-05cc-c00e0072ac28}"
2026-09-21T15:25:57.660Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:57.554510948 +0000 UTC m=+109.495681161 (1 nodes), 64.09112ms from start of poll, 64.6432ms from cluster start. Timeout at 2026-09-21 15:25:57.659512788 +0000 UTC m=+109.600682961, 105.0018ms after stability
2026-09-21T15:25:57.665Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:57.770Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:57.665519308 +0000 UTC m=+109.606689481 (2 nodes), 1.6µs from start of poll, 175.65152ms from cluster start. Timeout at 2026-09-21 15:25:57.770573948 +0000 UTC m=+109.711744121, 105.05464ms after stability
2026-09-21T15:25:57.770Z [INFO]  server-b042f783-4529-51e1-8824-f0cc102502ca: updating configuration: command=RemoveServer server-id=server-c3672c88-8aab-a920-7cc8-4b0a7b3a72d3 server-addr= servers="[{Suffrage:Voter ID:server-b042f783-4529-51e1-8824-f0cc102502ca Address:b042f783-4529-51e1-8824-f0cc102502ca} {Suffrage:Voter ID:server-d82bbd3e-0995-78a1-05cc-c00e0072ac28 Address:d82bbd3e-0995-78a1-05cc-c00e0072ac28}]"
2026-09-21T15:25:57.770Z [INFO]  server-b042f783-4529-51e1-8824-f0cc102502ca: removed peer, stopping replication: peer=server-c3672c88-8aab-a920-7cc8-4b0a7b3a72d3 last-index=3
2026-09-21T15:25:57.770Z [INFO]  server-b042f783-4529-51e1-8824-f0cc102502ca: aborting pipeline replication: peer="{Voter server-c3672c88-8aab-a920-7cc8-4b0a7b3a72d3 c3672c88-8aab-a920-7cc8-4b0a7b3a72d3}"
2026-09-21T15:25:57.881Z [ERROR] server-b042f783-4529-51e1-8824-f0cc102502ca: failed to pipeline appendEntries: peer="{Voter server-d82bbd3e-0995-78a1-05cc-c00e0072ac28 d82bbd3e-0995-78a1-05cc-c00e0072ac28}" error="append pipeline closed"
2026-09-21T15:25:57.881Z [INFO]  server-b042f783-4529-51e1-8824-f0cc102502ca: aborting pipeline replication: peer="{Voter server-d82bbd3e-0995-78a1-05cc-c00e0072ac28 d82bbd3e-0995-78a1-05cc-c00e0072ac28}"
2026-09-21T15:25:57.881Z [WARN]  server-c3672c88-8aab-a920-7cc8-4b0a7b3a72d3: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2026-09-21T15:25:57.885Z [ERROR] server-b042f783-4529-51e1-8824-f0cc102502ca: failed to heartbeat to: peer=d82bbd3e-0995-78a1-05cc-c00e0072ac28 backoff time=10ms error="failed to connect to peer: d82bbd3e-0995-78a1-05cc-c00e0072ac28"
2026-09-21T15:25:57.890Z [ERROR] server-b042f783-4529-51e1-8824-f0cc102502ca: failed to appendEntries to: peer="{Voter server-d82bbd3e-0995-78a1-05cc-c00e0072ac28 d82bbd3e-0995-78a1-05cc-c00e0072ac28}" error="failed to connect to peer: d82bbd3e-0995-78a1-05cc-c00e0072ac28"
2026-09-21T15:25:57.903Z [ERROR] server-b042f783-4529-51e1-8824-f0cc102502ca: failed to heartbeat to: peer=d82bbd3e-0995-78a1-05cc-c00e0072ac28 backoff time=10ms error="failed to connect to peer: d82bbd3e-0995-78a1-05cc-c00e0072ac28"
2026-09-21T15:25:57.908Z [ERROR] server-b042f783-4529-51e1-8824-f0cc102502ca: failed to appendEntries to: peer="{Voter server-d82bbd3e-0995-78a1-05cc-c00e0072ac28 d82bbd3e-0995-78a1-05cc-c00e0072ac28}" error="failed to connect to peer: d82bbd3e-0995-78a1-05cc-c00e0072ac28"
2026-09-21T15:25:57.921Z [ERROR] server-b042f783-4529-51e1-8824-f0cc102502ca: failed to heartbeat to: peer=d82bbd3e-0995-78a1-05cc-c00e0072ac28 backoff time=10ms error="failed to connect to peer: d82bbd3e-0995-78a1-05cc-c00e0072ac28"
2026-09-21T15:25:57.924Z [ERROR] server-b042f783-4529-51e1-8824-f0cc102502ca: failed to appendEntries to: peer="{Voter server-d82bbd3e-0995-78a1-05cc-c00e0072ac28 d82bbd3e-0995-78a1-05cc-c00e0072ac28}" error="failed to connect to peer: d82bbd3e-0995-78a1-05cc-c00e0072ac28"
2026-09-21T15:25:57.924Z [WARN]  server-b042f783-4529-51e1-8824-f0cc102502ca: failed to contact: server-id=server-d82bbd3e-0995-78a1-05cc-c00e0072ac28 time=50.04744ms
2026-09-21T15:25:57.925Z [WARN]  server-b042f783-4529-51e1-8824-f0cc102502ca: failed to contact quorum of nodes, stepping down
2026-09-21T15:25:57.925Z [INFO]  server-b042f783-4529-51e1-8824-f0cc102502ca: entering follower state: follower="Node at b042f783-4529-51e1-8824-f0cc102502ca [Follower]" leader-address= leader-id=
2026-09-21T15:25:57.972Z [WARN]  server-d82bbd3e-0995-78a1-05cc-c00e0072ac28: heartbeat timeout reached, starting election: last-leader-addr=b042f783-4529-51e1-8824-f0cc102502ca last-leader-id=server-b042f783-4529-51e1-8824-f0cc102502ca
2026-09-21T15:25:57.972Z [INFO]  server-d82bbd3e-0995-78a1-05cc-c00e0072ac28: entering candidate state: node="Node at d82bbd3e-0995-78a1-05cc-c00e0072ac28 [Candidate]" term=3
2026-09-21T15:25:57.972Z [ERROR] server-d82bbd3e-0995-78a1-05cc-c00e0072ac28: failed to make requestVote RPC: target="{Voter server-b042f783-4529-51e1-8824-f0cc102502ca b042f783-4529-51e1-8824-f0cc102502ca}" error="failed to connect to peer: b042f783-4529-51e1-8824-f0cc102502ca" term=3
2026-09-21T15:25:57.981Z [WARN]  server-d82bbd3e-0995-78a1-05cc-c00e0072ac28: rejecting vote request since node is not in configuration: from=c3672c88-8aab-a920-7cc8-4b0a7b3a72d3
--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.49s)
=== RUN   TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin
2026-09-21T15:25:57.982Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44d0cd83-7d24-2216-387a-e83cbb006b6b Address:44d0cd83-7d24-2216-387a-e83cbb006b6b} {Suffrage:Voter ID:server-8812654b-fe3e-d875-21ca-48c6800de749 Address:8812654b-fe3e-d875-21ca-48c6800de749} {Suffrage:Voter ID:server-659c051d-7a40-9e53-a4fa-5006f9d7b62c Address:659c051d-7a40-9e53-a4fa-5006f9d7b62c}]"
2026-09-21T15:25:57.982Z [INFO]  server-8812654b-fe3e-d875-21ca-48c6800de749: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44d0cd83-7d24-2216-387a-e83cbb006b6b Address:44d0cd83-7d24-2216-387a-e83cbb006b6b} {Suffrage:Voter ID:server-8812654b-fe3e-d875-21ca-48c6800de749 Address:8812654b-fe3e-d875-21ca-48c6800de749} {Suffrage:Voter ID:server-659c051d-7a40-9e53-a4fa-5006f9d7b62c Address:659c051d-7a40-9e53-a4fa-5006f9d7b62c}]"
2026-09-21T15:25:57.982Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: entering follower state: follower="Node at 44d0cd83-7d24-2216-387a-e83cbb006b6b [Follower]" leader-address= leader-id=
2026-09-21T15:25:57.982Z [INFO]  server-8812654b-fe3e-d875-21ca-48c6800de749: entering follower state: follower="Node at 8812654b-fe3e-d875-21ca-48c6800de749 [Follower]" leader-address= leader-id=
2026-09-21T15:25:57.982Z [INFO]  server-659c051d-7a40-9e53-a4fa-5006f9d7b62c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44d0cd83-7d24-2216-387a-e83cbb006b6b Address:44d0cd83-7d24-2216-387a-e83cbb006b6b} {Suffrage:Voter ID:server-8812654b-fe3e-d875-21ca-48c6800de749 Address:8812654b-fe3e-d875-21ca-48c6800de749} {Suffrage:Voter ID:server-659c051d-7a40-9e53-a4fa-5006f9d7b62c Address:659c051d-7a40-9e53-a4fa-5006f9d7b62c}]"
2026-09-21T15:25:57.982Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:57.982Z [INFO]  server-659c051d-7a40-9e53-a4fa-5006f9d7b62c: entering follower state: follower="Node at 659c051d-7a40-9e53-a4fa-5006f9d7b62c [Follower]" leader-address= leader-id=
2026-09-21T15:25:58.060Z [WARN]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:58.060Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: entering candidate state: node="Node at 44d0cd83-7d24-2216-387a-e83cbb006b6b [Candidate]" term=2
2026-09-21T15:25:58.060Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: election won: term=2 tally=2
2026-09-21T15:25:58.060Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: entering leader state: leader="Node at 44d0cd83-7d24-2216-387a-e83cbb006b6b [Leader]"
2026-09-21T15:25:58.060Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: added peer, starting replication: peer=server-8812654b-fe3e-d875-21ca-48c6800de749
2026-09-21T15:25:58.060Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: added peer, starting replication: peer=server-659c051d-7a40-9e53-a4fa-5006f9d7b62c
2026-09-21T15:25:58.060Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: pipelining replication: peer="{Voter server-659c051d-7a40-9e53-a4fa-5006f9d7b62c 659c051d-7a40-9e53-a4fa-5006f9d7b62c}"
2026-09-21T15:25:58.061Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: pipelining replication: peer="{Voter server-8812654b-fe3e-d875-21ca-48c6800de749 8812654b-fe3e-d875-21ca-48c6800de749}"
2026-09-21T15:25:58.166Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:58.060775148 +0000 UTC m=+110.001945321 (1 nodes), 78.01928ms from start of poll, 78.44492ms from cluster start. Timeout at 2026-09-21 15:25:58.165808068 +0000 UTC m=+110.106978241, 105.03292ms after stability
2026-09-21T15:25:58.172Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:25:58.278Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:25:58.172896028 +0000 UTC m=+110.114066241 (2 nodes), 4.08µs from start of poll, 190.56584ms from cluster start. Timeout at 2026-09-21 15:25:58.277912108 +0000 UTC m=+110.219082241, 105.016ms after stability
2026-09-21T15:25:58.279Z [ERROR] server-44d0cd83-7d24-2216-387a-e83cbb006b6b: failed to heartbeat to: peer=8812654b-fe3e-d875-21ca-48c6800de749 backoff time=10ms error="failed to connect to peer: 8812654b-fe3e-d875-21ca-48c6800de749"
2026-09-21T15:25:58.279Z [ERROR] server-44d0cd83-7d24-2216-387a-e83cbb006b6b: failed to pipeline appendEntries: peer="{Voter server-8812654b-fe3e-d875-21ca-48c6800de749 8812654b-fe3e-d875-21ca-48c6800de749}" error="append pipeline closed"
2026-09-21T15:25:58.279Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: aborting pipeline replication: peer="{Voter server-8812654b-fe3e-d875-21ca-48c6800de749 8812654b-fe3e-d875-21ca-48c6800de749}"
2026-09-21T15:25:58.290Z [ERROR] server-44d0cd83-7d24-2216-387a-e83cbb006b6b: failed to appendEntries to: peer="{Voter server-8812654b-fe3e-d875-21ca-48c6800de749 8812654b-fe3e-d875-21ca-48c6800de749}" error="failed to connect to peer: 8812654b-fe3e-d875-21ca-48c6800de749"
2026-09-21T15:25:58.299Z [ERROR] server-44d0cd83-7d24-2216-387a-e83cbb006b6b: failed to heartbeat to: peer=8812654b-fe3e-d875-21ca-48c6800de749 backoff time=10ms error="failed to connect to peer: 8812654b-fe3e-d875-21ca-48c6800de749"
2026-09-21T15:25:58.307Z [ERROR] server-44d0cd83-7d24-2216-387a-e83cbb006b6b: failed to appendEntries to: peer="{Voter server-8812654b-fe3e-d875-21ca-48c6800de749 8812654b-fe3e-d875-21ca-48c6800de749}" error="failed to connect to peer: 8812654b-fe3e-d875-21ca-48c6800de749"
2026-09-21T15:25:58.320Z [ERROR] server-44d0cd83-7d24-2216-387a-e83cbb006b6b: failed to heartbeat to: peer=8812654b-fe3e-d875-21ca-48c6800de749 backoff time=10ms error="failed to connect to peer: 8812654b-fe3e-d875-21ca-48c6800de749"
2026-09-21T15:25:58.323Z [WARN]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: failed to contact: server-id=server-8812654b-fe3e-d875-21ca-48c6800de749 time=50.18944ms
2026-09-21T15:25:58.327Z [ERROR] server-44d0cd83-7d24-2216-387a-e83cbb006b6b: failed to appendEntries to: peer="{Voter server-8812654b-fe3e-d875-21ca-48c6800de749 8812654b-fe3e-d875-21ca-48c6800de749}" error="failed to connect to peer: 8812654b-fe3e-d875-21ca-48c6800de749"
2026-09-21T15:25:58.335Z [ERROR] server-44d0cd83-7d24-2216-387a-e83cbb006b6b: failed to heartbeat to: peer=8812654b-fe3e-d875-21ca-48c6800de749 backoff time=20ms error="failed to connect to peer: 8812654b-fe3e-d875-21ca-48c6800de749"
2026-09-21T15:25:58.357Z [ERROR] server-44d0cd83-7d24-2216-387a-e83cbb006b6b: failed to appendEntries to: peer="{Voter server-8812654b-fe3e-d875-21ca-48c6800de749 8812654b-fe3e-d875-21ca-48c6800de749}" error="failed to connect to peer: 8812654b-fe3e-d875-21ca-48c6800de749"
2026-09-21T15:25:58.359Z [WARN]  server-8812654b-fe3e-d875-21ca-48c6800de749: heartbeat timeout reached, starting election: last-leader-addr=44d0cd83-7d24-2216-387a-e83cbb006b6b last-leader-id=server-44d0cd83-7d24-2216-387a-e83cbb006b6b
2026-09-21T15:25:58.359Z [INFO]  server-8812654b-fe3e-d875-21ca-48c6800de749: entering candidate state: node="Node at 8812654b-fe3e-d875-21ca-48c6800de749 [Candidate]" term=3
2026-09-21T15:25:58.359Z [ERROR] server-8812654b-fe3e-d875-21ca-48c6800de749: failed to make requestVote RPC: target="{Voter server-659c051d-7a40-9e53-a4fa-5006f9d7b62c 659c051d-7a40-9e53-a4fa-5006f9d7b62c}" error="failed to connect to peer: 659c051d-7a40-9e53-a4fa-5006f9d7b62c" term=3
2026-09-21T15:25:58.359Z [ERROR] server-8812654b-fe3e-d875-21ca-48c6800de749: failed to make requestVote RPC: target="{Voter server-44d0cd83-7d24-2216-387a-e83cbb006b6b 44d0cd83-7d24-2216-387a-e83cbb006b6b}" error="failed to connect to peer: 44d0cd83-7d24-2216-387a-e83cbb006b6b" term=3
2026-09-21T15:25:58.365Z [ERROR] server-44d0cd83-7d24-2216-387a-e83cbb006b6b: failed to heartbeat to: peer=8812654b-fe3e-d875-21ca-48c6800de749 backoff time=25ms error="failed to connect to peer: 8812654b-fe3e-d875-21ca-48c6800de749"
2026-09-21T15:25:58.371Z [WARN]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: failed to contact: server-id=server-8812654b-fe3e-d875-21ca-48c6800de749 time=98.02552ms
2026-09-21T15:25:58.384Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: updating configuration: command=RemoveServer server-id=server-8812654b-fe3e-d875-21ca-48c6800de749 server-addr= servers="[{Suffrage:Voter ID:server-44d0cd83-7d24-2216-387a-e83cbb006b6b Address:44d0cd83-7d24-2216-387a-e83cbb006b6b} {Suffrage:Voter ID:server-659c051d-7a40-9e53-a4fa-5006f9d7b62c Address:659c051d-7a40-9e53-a4fa-5006f9d7b62c}]"
2026-09-21T15:25:58.384Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: removed peer, stopping replication: peer=server-8812654b-fe3e-d875-21ca-48c6800de749 last-index=3
2026-09-21T15:25:58.385Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: updating configuration: command=AddNonvoter server-id=server-8812654b-fe3e-d875-21ca-48c6800de749 server-addr=8812654b-fe3e-d875-21ca-48c6800de749 servers="[{Suffrage:Voter ID:server-44d0cd83-7d24-2216-387a-e83cbb006b6b Address:44d0cd83-7d24-2216-387a-e83cbb006b6b} {Suffrage:Voter ID:server-659c051d-7a40-9e53-a4fa-5006f9d7b62c Address:659c051d-7a40-9e53-a4fa-5006f9d7b62c} {Suffrage:Nonvoter ID:server-8812654b-fe3e-d875-21ca-48c6800de749 Address:8812654b-fe3e-d875-21ca-48c6800de749}]"
2026-09-21T15:25:58.385Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: added peer, starting replication: peer=server-8812654b-fe3e-d875-21ca-48c6800de749
2026-09-21T15:25:58.385Z [ERROR] server-44d0cd83-7d24-2216-387a-e83cbb006b6b: failed to appendEntries to: peer="{Nonvoter server-8812654b-fe3e-d875-21ca-48c6800de749 8812654b-fe3e-d875-21ca-48c6800de749}" error="failed to connect to peer: 8812654b-fe3e-d875-21ca-48c6800de749"
2026-09-21T15:25:58.404Z [ERROR] server-44d0cd83-7d24-2216-387a-e83cbb006b6b: peer has newer term, stopping replication: peer="{Nonvoter server-8812654b-fe3e-d875-21ca-48c6800de749 8812654b-fe3e-d875-21ca-48c6800de749}"
2026-09-21T15:25:58.404Z [ERROR] server-44d0cd83-7d24-2216-387a-e83cbb006b6b: peer has newer term, stopping replication: peer="{Voter server-8812654b-fe3e-d875-21ca-48c6800de749 8812654b-fe3e-d875-21ca-48c6800de749}"
2026-09-21T15:25:58.404Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: entering follower state: follower="Node at 44d0cd83-7d24-2216-387a-e83cbb006b6b [Follower]" leader-address= leader-id=
2026-09-21T15:25:58.404Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: aborting pipeline replication: peer="{Voter server-659c051d-7a40-9e53-a4fa-5006f9d7b62c 659c051d-7a40-9e53-a4fa-5006f9d7b62c}"
2026-09-21T15:25:58.434Z [WARN]  server-8812654b-fe3e-d875-21ca-48c6800de749: Election timeout reached, restarting election
2026-09-21T15:25:58.434Z [INFO]  server-8812654b-fe3e-d875-21ca-48c6800de749: entering candidate state: node="Node at 8812654b-fe3e-d875-21ca-48c6800de749 [Candidate]" term=23
2026-09-21T15:25:58.434Z [WARN]  server-659c051d-7a40-9e53-a4fa-5006f9d7b62c: rejecting vote request since we have a leader: from=8812654b-fe3e-d875-21ca-48c6800de749 leader=44d0cd83-7d24-2216-387a-e83cbb006b6b leader-id=server-44d0cd83-7d24-2216-387a-e83cbb006b6b
2026-09-21T15:25:58.434Z [WARN]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: rejecting vote request since node is not a voter: from=8812654b-fe3e-d875-21ca-48c6800de749
2026-09-21T15:25:58.487Z [WARN]  server-8812654b-fe3e-d875-21ca-48c6800de749: Election timeout reached, restarting election
2026-09-21T15:25:58.487Z [INFO]  server-8812654b-fe3e-d875-21ca-48c6800de749: entering candidate state: node="Node at 8812654b-fe3e-d875-21ca-48c6800de749 [Candidate]" term=24
2026-09-21T15:25:58.487Z [WARN]  server-659c051d-7a40-9e53-a4fa-5006f9d7b62c: rejecting vote request since we have a leader: from=8812654b-fe3e-d875-21ca-48c6800de749 leader=44d0cd83-7d24-2216-387a-e83cbb006b6b leader-id=server-44d0cd83-7d24-2216-387a-e83cbb006b6b
2026-09-21T15:25:58.487Z [WARN]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: rejecting vote request since node is not a voter: from=8812654b-fe3e-d875-21ca-48c6800de749
2026-09-21T15:25:58.491Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:58.499Z [WARN]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:58.499Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: entering candidate state: node="Node at 44d0cd83-7d24-2216-387a-e83cbb006b6b [Candidate]" term=25
2026-09-21T15:25:58.499Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: election won: term=25 tally=2
2026-09-21T15:25:58.499Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: entering leader state: leader="Node at 44d0cd83-7d24-2216-387a-e83cbb006b6b [Leader]"
2026-09-21T15:25:58.499Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: added peer, starting replication: peer=server-659c051d-7a40-9e53-a4fa-5006f9d7b62c
2026-09-21T15:25:58.499Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: added peer, starting replication: peer=server-8812654b-fe3e-d875-21ca-48c6800de749
2026-09-21T15:25:58.499Z [WARN]  server-8812654b-fe3e-d875-21ca-48c6800de749: failed to get previous log: previous-index=4 last-index=2 error="log not found"
2026-09-21T15:25:58.499Z [INFO]  server-8812654b-fe3e-d875-21ca-48c6800de749: entering follower state: follower="Node at 8812654b-fe3e-d875-21ca-48c6800de749 [Follower]" leader-address=44d0cd83-7d24-2216-387a-e83cbb006b6b leader-id=server-44d0cd83-7d24-2216-387a-e83cbb006b6b
2026-09-21T15:25:58.499Z [WARN]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: appendEntries rejected, sending older logs: peer="{Nonvoter server-8812654b-fe3e-d875-21ca-48c6800de749 8812654b-fe3e-d875-21ca-48c6800de749}" next=3
2026-09-21T15:25:58.499Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: pipelining replication: peer="{Voter server-659c051d-7a40-9e53-a4fa-5006f9d7b62c 659c051d-7a40-9e53-a4fa-5006f9d7b62c}"
2026-09-21T15:25:58.499Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: pipelining replication: peer="{Nonvoter server-8812654b-fe3e-d875-21ca-48c6800de749 8812654b-fe3e-d875-21ca-48c6800de749}"
2026-09-21T15:25:58.605Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:58.499807508 +0000 UTC m=+110.440977681 (1 nodes), 8.219ms from start of poll, 517.47728ms from cluster start. Timeout at 2026-09-21 15:25:58.604812068 +0000 UTC m=+110.545982241, 105.00456ms after stability
2026-09-21T15:25:58.640Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: updating configuration: command=RemoveServer server-id=server-8812654b-fe3e-d875-21ca-48c6800de749 server-addr= servers="[{Suffrage:Voter ID:server-44d0cd83-7d24-2216-387a-e83cbb006b6b Address:44d0cd83-7d24-2216-387a-e83cbb006b6b} {Suffrage:Voter ID:server-659c051d-7a40-9e53-a4fa-5006f9d7b62c Address:659c051d-7a40-9e53-a4fa-5006f9d7b62c}]"
2026-09-21T15:25:58.641Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: removed peer, stopping replication: peer=server-8812654b-fe3e-d875-21ca-48c6800de749 last-index=106
2026-09-21T15:25:58.641Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: aborting pipeline replication: peer="{Nonvoter server-8812654b-fe3e-d875-21ca-48c6800de749 8812654b-fe3e-d875-21ca-48c6800de749}"
2026-09-21T15:25:58.641Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: updating configuration: command=AddVoter server-id=server-8812654b-fe3e-d875-21ca-48c6800de749 server-addr=8812654b-fe3e-d875-21ca-48c6800de749 servers="[{Suffrage:Voter ID:server-44d0cd83-7d24-2216-387a-e83cbb006b6b Address:44d0cd83-7d24-2216-387a-e83cbb006b6b} {Suffrage:Voter ID:server-659c051d-7a40-9e53-a4fa-5006f9d7b62c Address:659c051d-7a40-9e53-a4fa-5006f9d7b62c} {Suffrage:Voter ID:server-8812654b-fe3e-d875-21ca-48c6800de749 Address:8812654b-fe3e-d875-21ca-48c6800de749}]"
2026-09-21T15:25:58.641Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: added peer, starting replication: peer=server-8812654b-fe3e-d875-21ca-48c6800de749
2026-09-21T15:25:58.641Z [WARN]  server-8812654b-fe3e-d875-21ca-48c6800de749: failed to get previous log: previous-index=107 last-index=106 error="log not found"
2026-09-21T15:25:58.641Z [WARN]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: appendEntries rejected, sending older logs: peer="{Voter server-8812654b-fe3e-d875-21ca-48c6800de749 8812654b-fe3e-d875-21ca-48c6800de749}" next=107
2026-09-21T15:25:58.641Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: pipelining replication: peer="{Voter server-8812654b-fe3e-d875-21ca-48c6800de749 8812654b-fe3e-d875-21ca-48c6800de749}"
2026-09-21T15:25:59.723Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:59.829Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:59.723591788 +0000 UTC m=+111.664761961 (1 nodes), 1.16µs from start of poll, 1.74126156s from cluster start. Timeout at 2026-09-21 15:25:59.828631988 +0000 UTC m=+111.769802161, 105.0402ms after stability
2026-09-21T15:25:59.829Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: aborting pipeline replication: peer="{Voter server-659c051d-7a40-9e53-a4fa-5006f9d7b62c 659c051d-7a40-9e53-a4fa-5006f9d7b62c}"
2026-09-21T15:25:59.829Z [INFO]  server-44d0cd83-7d24-2216-387a-e83cbb006b6b: aborting pipeline replication: peer="{Voter server-8812654b-fe3e-d875-21ca-48c6800de749 8812654b-fe3e-d875-21ca-48c6800de749}"
--- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.85s)
=== RUN   TestRaft_FollowerRemovalNoElection
2026-09-21T15:25:59.830Z [INFO]  server-aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d Address:aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d} {Suffrage:Voter ID:server-980d2c6d-8ab1-445f-bc39-0475e0005a6c Address:980d2c6d-8ab1-445f-bc39-0475e0005a6c} {Suffrage:Voter ID:server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047 Address:2be3228b-01d1-b5bd-0b69-5cf3dc3db047}]"
2026-09-21T15:25:59.830Z [INFO]  server-980d2c6d-8ab1-445f-bc39-0475e0005a6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d Address:aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d} {Suffrage:Voter ID:server-980d2c6d-8ab1-445f-bc39-0475e0005a6c Address:980d2c6d-8ab1-445f-bc39-0475e0005a6c} {Suffrage:Voter ID:server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047 Address:2be3228b-01d1-b5bd-0b69-5cf3dc3db047}]"
2026-09-21T15:25:59.830Z [INFO]  server-aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d: entering follower state: follower="Node at aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d [Follower]" leader-address= leader-id=
2026-09-21T15:25:59.830Z [INFO]  server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d Address:aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d} {Suffrage:Voter ID:server-980d2c6d-8ab1-445f-bc39-0475e0005a6c Address:980d2c6d-8ab1-445f-bc39-0475e0005a6c} {Suffrage:Voter ID:server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047 Address:2be3228b-01d1-b5bd-0b69-5cf3dc3db047}]"
2026-09-21T15:25:59.830Z [INFO]  server-980d2c6d-8ab1-445f-bc39-0475e0005a6c: entering follower state: follower="Node at 980d2c6d-8ab1-445f-bc39-0475e0005a6c [Follower]" leader-address= leader-id=
2026-09-21T15:25:59.830Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:25:59.830Z [INFO]  server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: entering follower state: follower="Node at 2be3228b-01d1-b5bd-0b69-5cf3dc3db047 [Follower]" leader-address= leader-id=
2026-09-21T15:25:59.973Z [WARN]  server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:25:59.973Z [INFO]  server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: entering candidate state: node="Node at 2be3228b-01d1-b5bd-0b69-5cf3dc3db047 [Candidate]" term=2
2026-09-21T15:25:59.973Z [INFO]  server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: election won: term=2 tally=2
2026-09-21T15:25:59.973Z [INFO]  server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: entering leader state: leader="Node at 2be3228b-01d1-b5bd-0b69-5cf3dc3db047 [Leader]"
2026-09-21T15:25:59.973Z [INFO]  server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: added peer, starting replication: peer=server-aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d
2026-09-21T15:25:59.973Z [INFO]  server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: added peer, starting replication: peer=server-980d2c6d-8ab1-445f-bc39-0475e0005a6c
2026-09-21T15:25:59.973Z [INFO]  server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: pipelining replication: peer="{Voter server-980d2c6d-8ab1-445f-bc39-0475e0005a6c 980d2c6d-8ab1-445f-bc39-0475e0005a6c}"
2026-09-21T15:25:59.973Z [INFO]  server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: pipelining replication: peer="{Voter server-aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d}"
2026-09-21T15:26:00.178Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:25:59.973454628 +0000 UTC m=+111.914624801 (1 nodes), 142.86216ms from start of poll, 143.29104ms from cluster start. Timeout at 2026-09-21 15:26:00.178472468 +0000 UTC m=+112.119642641, 205.01784ms after stability
2026-09-21T15:26:00.178Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:26:00.384Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:26:00.178871028 +0000 UTC m=+112.120041201 (1 nodes), 2.2µs from start of poll, 348.70744ms from cluster start. Timeout at 2026-09-21 15:26:00.383895508 +0000 UTC m=+112.325065721, 205.02452ms after stability
2026-09-21T15:26:00.389Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:26:00.595Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:26:00.390033388 +0000 UTC m=+112.331203561 (2 nodes), 3.08µs from start of poll, 559.8698ms from cluster start. Timeout at 2026-09-21 15:26:00.595070628 +0000 UTC m=+112.536240801, 205.03724ms after stability
2026-09-21T15:26:00.595Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:26:00.800Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:26:00.595211828 +0000 UTC m=+112.536382001 (1 nodes), 2.68µs from start of poll, 765.04824ms from cluster start. Timeout at 2026-09-21 15:26:00.800237748 +0000 UTC m=+112.741407921, 205.02592ms after stability
    raft_test.go:2961: [INFO] restarting Node at aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d [Follower]
2026-09-21T15:26:00.801Z [INFO]  server-aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d Address:aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d} {Suffrage:Voter ID:server-980d2c6d-8ab1-445f-bc39-0475e0005a6c Address:980d2c6d-8ab1-445f-bc39-0475e0005a6c} {Suffrage:Voter ID:server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047 Address:2be3228b-01d1-b5bd-0b69-5cf3dc3db047}]"
2026-09-21T15:26:00.801Z [INFO]  server-aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d: entering follower state: follower="Node at aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d [Follower]" leader-address= leader-id=
2026-09-21T15:26:00.844Z [WARN]  server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: failed to contact: server-id=server-aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d time=50.23288ms
2026-09-21T15:26:00.893Z [WARN]  server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: failed to contact: server-id=server-aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d time=99.37448ms
2026-09-21T15:26:00.939Z [WARN]  server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: failed to contact: server-id=server-aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d time=144.75556ms
2026-09-21T15:26:00.973Z [WARN]  server-aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:26:00.973Z [INFO]  server-aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d: entering candidate state: node="Node at aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d [Candidate]" term=3
2026-09-21T15:26:00.974Z [WARN]  server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: rejecting vote request since we have a leader: from=aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d leader=2be3228b-01d1-b5bd-0b69-5cf3dc3db047 leader-id=server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047
2026-09-21T15:26:00.974Z [WARN]  server-980d2c6d-8ab1-445f-bc39-0475e0005a6c: rejecting vote request since we have a leader: from=aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d leader=2be3228b-01d1-b5bd-0b69-5cf3dc3db047 leader-id=server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047
2026-09-21T15:26:01.051Z [ERROR] cluster: peer sees incorrect leader: peer="Node at aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d [Candidate]" leader=[none] expected-leader=2be3228b-01d1-b5bd-0b69-5cf3dc3db047
    testing.go:604: at least one peer has the wrong notion of leader
2026-09-21T15:26:01.052Z [INFO]  server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: aborting pipeline replication: peer="{Voter server-980d2c6d-8ab1-445f-bc39-0475e0005a6c 980d2c6d-8ab1-445f-bc39-0475e0005a6c}"
2026-09-21T15:26:01.301Z [ERROR] server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: failed to heartbeat to: peer=aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d backoff time=10ms error="command timed out"
2026-09-21T15:26:01.422Z [ERROR] server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: failed to pipeline appendEntries: peer="{Voter server-aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d}" error="command enqueue timeout"
2026-09-21T15:26:01.422Z [INFO]  server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: aborting pipeline replication: peer="{Voter server-aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d}"
2026-09-21T15:26:01.829Z [ERROR] server-2be3228b-01d1-b5bd-0b69-5cf3dc3db047: failed to heartbeat to: peer=aaee8f31-dfaa-37d4-c4e8-bd5d7bc4055d backoff time=10ms error="command timed out"
--- FAIL: TestRaft_FollowerRemovalNoElection (2.00s)
=== RUN   TestRaft_VoteWithNoIDNoAddr
2026-09-21T15:26:01.830Z [INFO]  server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9 Address:6d93a6d1-bfc3-e126-0345-f1bb5a1479c9} {Suffrage:Voter ID:server-dc94e20e-313f-f4bc-4fff-dd2611a58f85 Address:dc94e20e-313f-f4bc-4fff-dd2611a58f85} {Suffrage:Voter ID:server-85408534-3cdc-2063-bf69-09aa5c4b1b23 Address:85408534-3cdc-2063-bf69-09aa5c4b1b23}]"
2026-09-21T15:26:01.830Z [INFO]  server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9: entering follower state: follower="Node at 6d93a6d1-bfc3-e126-0345-f1bb5a1479c9 [Follower]" leader-address= leader-id=
2026-09-21T15:26:01.831Z [INFO]  server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9 Address:6d93a6d1-bfc3-e126-0345-f1bb5a1479c9} {Suffrage:Voter ID:server-dc94e20e-313f-f4bc-4fff-dd2611a58f85 Address:dc94e20e-313f-f4bc-4fff-dd2611a58f85} {Suffrage:Voter ID:server-85408534-3cdc-2063-bf69-09aa5c4b1b23 Address:85408534-3cdc-2063-bf69-09aa5c4b1b23}]"
2026-09-21T15:26:01.831Z [INFO]  server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: entering follower state: follower="Node at dc94e20e-313f-f4bc-4fff-dd2611a58f85 [Follower]" leader-address= leader-id=
2026-09-21T15:26:01.832Z [INFO]  server-85408534-3cdc-2063-bf69-09aa5c4b1b23: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9 Address:6d93a6d1-bfc3-e126-0345-f1bb5a1479c9} {Suffrage:Voter ID:server-dc94e20e-313f-f4bc-4fff-dd2611a58f85 Address:dc94e20e-313f-f4bc-4fff-dd2611a58f85} {Suffrage:Voter ID:server-85408534-3cdc-2063-bf69-09aa5c4b1b23 Address:85408534-3cdc-2063-bf69-09aa5c4b1b23}]"
2026-09-21T15:26:01.832Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:26:01.832Z [INFO]  server-85408534-3cdc-2063-bf69-09aa5c4b1b23: entering follower state: follower="Node at 85408534-3cdc-2063-bf69-09aa5c4b1b23 [Follower]" leader-address= leader-id=
2026-09-21T15:26:01.891Z [WARN]  server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-09-21T15:26:01.891Z [INFO]  server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: entering candidate state: node="Node at dc94e20e-313f-f4bc-4fff-dd2611a58f85 [Candidate]" term=2
2026-09-21T15:26:01.891Z [INFO]  server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: election won: term=2 tally=2
2026-09-21T15:26:01.891Z [INFO]  server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: entering leader state: leader="Node at dc94e20e-313f-f4bc-4fff-dd2611a58f85 [Leader]"
2026-09-21T15:26:01.891Z [INFO]  server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: added peer, starting replication: peer=server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9
2026-09-21T15:26:01.891Z [INFO]  server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: added peer, starting replication: peer=server-85408534-3cdc-2063-bf69-09aa5c4b1b23
2026-09-21T15:26:01.892Z [INFO]  server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: pipelining replication: peer="{Voter server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9 6d93a6d1-bfc3-e126-0345-f1bb5a1479c9}"
2026-09-21T15:26:01.892Z [INFO]  server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: pipelining replication: peer="{Voter server-85408534-3cdc-2063-bf69-09aa5c4b1b23 85408534-3cdc-2063-bf69-09aa5c4b1b23}"
2026-09-21T15:26:01.997Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:26:01.891904508 +0000 UTC m=+113.833074681 (1 nodes), 59.42684ms from start of poll, 61.21256ms from cluster start. Timeout at 2026-09-21 15:26:01.996912428 +0000 UTC m=+113.938082601, 105.00792ms after stability
2026-09-21T15:26:01.997Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-09-21T15:26:02.103Z [INFO]  cluster: stable state for Leader reached at 2026-09-21 15:26:01.997093868 +0000 UTC m=+113.938264041 (1 nodes), 2.08µs from start of poll, 166.40192ms from cluster start. Timeout at 2026-09-21 15:26:02.102117268 +0000 UTC m=+114.043287441, 105.0234ms after stability
2026-09-21T15:26:02.108Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-09-21T15:26:02.215Z [INFO]  cluster: stable state for Follower reached at 2026-09-21 15:26:02.109040308 +0000 UTC m=+114.050210481 (2 nodes), 2.48µs from start of poll, 278.34836ms from cluster start. Timeout at 2026-09-21 15:26:02.214065668 +0000 UTC m=+114.155235841, 105.02536ms after stability
2026-09-21T15:26:02.216Z [ERROR] server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to pipeline appendEntries: peer="{Voter server-85408534-3cdc-2063-bf69-09aa5c4b1b23 85408534-3cdc-2063-bf69-09aa5c4b1b23}" error="append pipeline closed"
2026-09-21T15:26:02.216Z [INFO]  server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: aborting pipeline replication: peer="{Voter server-85408534-3cdc-2063-bf69-09aa5c4b1b23 85408534-3cdc-2063-bf69-09aa5c4b1b23}"
2026-09-21T15:26:02.216Z [ERROR] server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to heartbeat to: peer=85408534-3cdc-2063-bf69-09aa5c4b1b23 backoff time=10ms error="failed to connect to peer: 85408534-3cdc-2063-bf69-09aa5c4b1b23"
2026-09-21T15:26:02.220Z [ERROR] server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to pipeline appendEntries: peer="{Voter server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9 6d93a6d1-bfc3-e126-0345-f1bb5a1479c9}" error="append pipeline closed"
2026-09-21T15:26:02.220Z [INFO]  server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: aborting pipeline replication: peer="{Voter server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9 6d93a6d1-bfc3-e126-0345-f1bb5a1479c9}"
2026-09-21T15:26:02.223Z [ERROR] server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to heartbeat to: peer=6d93a6d1-bfc3-e126-0345-f1bb5a1479c9 backoff time=10ms error="failed to connect to peer: 6d93a6d1-bfc3-e126-0345-f1bb5a1479c9"
2026-09-21T15:26:02.226Z [ERROR] server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to appendEntries to: peer="{Voter server-85408534-3cdc-2063-bf69-09aa5c4b1b23 85408534-3cdc-2063-bf69-09aa5c4b1b23}" error="failed to connect to peer: 85408534-3cdc-2063-bf69-09aa5c4b1b23"
2026-09-21T15:26:02.229Z [ERROR] server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to appendEntries to: peer="{Voter server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9 6d93a6d1-bfc3-e126-0345-f1bb5a1479c9}" error="failed to connect to peer: 6d93a6d1-bfc3-e126-0345-f1bb5a1479c9"
2026-09-21T15:26:02.233Z [ERROR] server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to heartbeat to: peer=85408534-3cdc-2063-bf69-09aa5c4b1b23 backoff time=10ms error="failed to connect to peer: 85408534-3cdc-2063-bf69-09aa5c4b1b23"
2026-09-21T15:26:02.242Z [ERROR] server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to heartbeat to: peer=6d93a6d1-bfc3-e126-0345-f1bb5a1479c9 backoff time=10ms error="failed to connect to peer: 6d93a6d1-bfc3-e126-0345-f1bb5a1479c9"
2026-09-21T15:26:02.247Z [ERROR] server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to appendEntries to: peer="{Voter server-85408534-3cdc-2063-bf69-09aa5c4b1b23 85408534-3cdc-2063-bf69-09aa5c4b1b23}" error="failed to connect to peer: 85408534-3cdc-2063-bf69-09aa5c4b1b23"
2026-09-21T15:26:02.249Z [ERROR] server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to appendEntries to: peer="{Voter server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9 6d93a6d1-bfc3-e126-0345-f1bb5a1479c9}" error="failed to connect to peer: 6d93a6d1-bfc3-e126-0345-f1bb5a1479c9"
2026-09-21T15:26:02.251Z [ERROR] server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to heartbeat to: peer=85408534-3cdc-2063-bf69-09aa5c4b1b23 backoff time=10ms error="failed to connect to peer: 85408534-3cdc-2063-bf69-09aa5c4b1b23"
2026-09-21T15:26:02.258Z [ERROR] server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to heartbeat to: peer=6d93a6d1-bfc3-e126-0345-f1bb5a1479c9 backoff time=10ms error="failed to connect to peer: 6d93a6d1-bfc3-e126-0345-f1bb5a1479c9"
2026-09-21T15:26:02.260Z [WARN]  server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to contact: server-id=server-85408534-3cdc-2063-bf69-09aa5c4b1b23 time=50.07312ms
2026-09-21T15:26:02.263Z [ERROR] server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to appendEntries to: peer="{Voter server-85408534-3cdc-2063-bf69-09aa5c4b1b23 85408534-3cdc-2063-bf69-09aa5c4b1b23}" error="failed to connect to peer: 85408534-3cdc-2063-bf69-09aa5c4b1b23"
2026-09-21T15:26:02.267Z [ERROR] server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to appendEntries to: peer="{Voter server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9 6d93a6d1-bfc3-e126-0345-f1bb5a1479c9}" error="failed to connect to peer: 6d93a6d1-bfc3-e126-0345-f1bb5a1479c9"
2026-09-21T15:26:02.268Z [ERROR] server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to heartbeat to: peer=85408534-3cdc-2063-bf69-09aa5c4b1b23 backoff time=20ms error="failed to connect to peer: 85408534-3cdc-2063-bf69-09aa5c4b1b23"
2026-09-21T15:26:02.270Z [WARN]  server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to contact: server-id=server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9 time=56.4084ms
2026-09-21T15:26:02.270Z [WARN]  server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to contact: server-id=server-85408534-3cdc-2063-bf69-09aa5c4b1b23 time=60.33856ms
2026-09-21T15:26:02.270Z [WARN]  server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: failed to contact quorum of nodes, stepping down
2026-09-21T15:26:02.270Z [INFO]  server-dc94e20e-313f-f4bc-4fff-dd2611a58f85: entering follower state: follower="Node at dc94e20e-313f-f4bc-4fff-dd2611a58f85 [Follower]" leader-address= leader-id=
2026-09-21T15:26:02.283Z [WARN]  server-85408534-3cdc-2063-bf69-09aa5c4b1b23: heartbeat timeout reached, starting election: last-leader-addr=dc94e20e-313f-f4bc-4fff-dd2611a58f85 last-leader-id=server-dc94e20e-313f-f4bc-4fff-dd2611a58f85
2026-09-21T15:26:02.283Z [INFO]  server-85408534-3cdc-2063-bf69-09aa5c4b1b23: entering candidate state: node="Node at 85408534-3cdc-2063-bf69-09aa5c4b1b23 [Candidate]" term=3
2026-09-21T15:26:02.283Z [WARN]  server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9: rejecting vote request since we have a leader: from=85408534-3cdc-2063-bf69-09aa5c4b1b23 leader=dc94e20e-313f-f4bc-4fff-dd2611a58f85 leader-id=server-dc94e20e-313f-f4bc-4fff-dd2611a58f85
2026-09-21T15:26:02.283Z [ERROR] server-85408534-3cdc-2063-bf69-09aa5c4b1b23: failed to make requestVote RPC: target="{Voter server-dc94e20e-313f-f4bc-4fff-dd2611a58f85 dc94e20e-313f-f4bc-4fff-dd2611a58f85}" error="failed to connect to peer: dc94e20e-313f-f4bc-4fff-dd2611a58f85" term=3
2026-09-21T15:26:02.329Z [WARN]  server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9: heartbeat timeout reached, starting election: last-leader-addr=dc94e20e-313f-f4bc-4fff-dd2611a58f85 last-leader-id=server-dc94e20e-313f-f4bc-4fff-dd2611a58f85
2026-09-21T15:26:02.329Z [INFO]  server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9: entering candidate state: node="Node at 6d93a6d1-bfc3-e126-0345-f1bb5a1479c9 [Candidate]" term=3
2026-09-21T15:26:02.329Z [INFO]  server-85408534-3cdc-2063-bf69-09aa5c4b1b23: duplicate requestVote for same term: term=3
2026-09-21T15:26:02.329Z [ERROR] server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9: failed to make requestVote RPC: target="{Voter server-dc94e20e-313f-f4bc-4fff-dd2611a58f85 dc94e20e-313f-f4bc-4fff-dd2611a58f85}" error="failed to connect to peer: dc94e20e-313f-f4bc-4fff-dd2611a58f85" term=3
2026-09-21T15:26:02.331Z [INFO]  server-6d93a6d1-bfc3-e126-0345-f1bb5a1479c9: entering follower state: follower="Node at 6d93a6d1-bfc3-e126-0345-f1bb5a1479c9 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VoteWithNoIDNoAddr (0.50s)
=== RUN   TestRaft_runFollower_State_Transition
=== RUN   TestRaft_runFollower_State_Transition/NonVoter
2026-09-21T15:26:02.332Z [INFO]  127.0.0.1:39361 :: starting node: addr=127.0.0.1:39361
2026-09-21T15:26:02.332Z [INFO]  127.0.0.1:39361 :: initial configuration: index=0 servers=[]
2026-09-21T15:26:02.332Z [INFO]  127.0.0.1:39361 :: entering follower state: follower="Node at 127.0.0.1:39361 [Follower]" leader-address= leader-id=
2026-09-21T15:26:02.426Z [WARN]  127.0.0.1:39361 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
=== RUN   TestRaft_runFollower_State_Transition/Voter
2026-09-21T15:26:02.483Z [INFO]  127.0.0.1:42213 :: starting node: addr=127.0.0.1:42213
2026-09-21T15:26:02.483Z [INFO]  127.0.0.1:42213 :: initial configuration: index=0 servers=[]
2026-09-21T15:26:02.483Z [INFO]  127.0.0.1:42213 :: entering follower state: follower="Node at 127.0.0.1:42213 [Follower]" leader-address= leader-id=
2026-09-21T15:26:02.542Z [WARN]  127.0.0.1:42213 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
=== RUN   TestRaft_runFollower_State_Transition/Not_in_Config
2026-09-21T15:26:02.634Z [INFO]  127.0.0.1:36777 :: starting node: addr=127.0.0.1:36777
2026-09-21T15:26:02.634Z [INFO]  127.0.0.1:36777 :: initial configuration: index=0 servers=[]
2026-09-21T15:26:02.634Z [INFO]  127.0.0.1:36777 :: entering follower state: follower="Node at 127.0.0.1:36777 [Follower]" leader-address= leader-id=
2026-09-21T15:26:02.709Z [WARN]  127.0.0.1:36777 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
--- PASS: TestRaft_runFollower_State_Transition (0.45s)
    --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s)
=== RUN   TestRaft_runFollower_ReloadTimeoutConfigs
2026-09-21T15:26:02.784Z [INFO]  127.0.0.1:32865 :: starting node: addr=127.0.0.1:32865
2026-09-21T15:26:02.784Z [INFO]  127.0.0.1:32865 :: initial configuration: index=0 servers=[]
2026-09-21T15:26:02.784Z [INFO]  127.0.0.1:32865 :: entering follower state: follower="Node at 127.0.0.1:32865 [Follower]" leader-address= leader-id=
2026-09-21T15:26:02.784Z [WARN]  127.0.0.1:32865 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
--- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s)
=== RUN   TestSaturationMetric
=== RUN   TestSaturationMetric/without_smoothing
--- PASS: TestSaturationMetric (0.00s)
    --- PASS: TestSaturationMetric/without_smoothing (0.00s)
=== RUN   TestSaturationMetric_IncorrectUsage
=== RUN   TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_first
--- PASS: TestSaturationMetric_IncorrectUsage (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s)
=== RUN   TestTCPTransport_BadAddr
--- PASS: TestTCPTransport_BadAddr (0.00s)
=== RUN   TestTCPTransport_EmptyAddr
--- PASS: TestTCPTransport_EmptyAddr (0.00s)
=== RUN   TestTCPTransport_WithAdvertise
--- PASS: TestTCPTransport_WithAdvertise (0.00s)
=== RUN   TestTransport_StartStop
--- PASS: TestTransport_StartStop (0.00s)
=== RUN   TestTransport_AppendEntries
--- PASS: TestTransport_AppendEntries (0.00s)
=== RUN   TestTransport_AppendEntriesPipeline
--- PASS: TestTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestTransport_RequestVote
--- PASS: TestTransport_RequestVote (0.00s)
=== RUN   TestTransport_InstallSnapshot
--- PASS: TestTransport_InstallSnapshot (0.00s)
=== RUN   TestTransport_EncodeDecode
--- PASS: TestTransport_EncodeDecode (0.00s)
=== RUN   TestRandomTimeout
--- PASS: TestRandomTimeout (0.00s)
=== RUN   TestNewSeed
--- PASS: TestNewSeed (0.00s)
=== RUN   TestRandomTimeout_NoTime
--- PASS: TestRandomTimeout_NoTime (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
=== RUN   TestMax
--- PASS: TestMax (0.00s)
=== RUN   TestGenerateUUID
--- PASS: TestGenerateUUID (0.00s)
=== RUN   TestBackoff
--- PASS: TestBackoff (0.00s)
=== RUN   TestOverrideNotifyBool
--- PASS: TestOverrideNotifyBool (0.00s)
FAIL
FAIL	github.com/hashicorp/raft	114.894s
?   	github.com/hashicorp/raft/bench	[no test files]
FAIL
dh_auto_test: error: cd obj-aarch64-linux-gnu && go test -vet=off -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench returned exit code 1
make[1]: *** [debian/rules:17: override_dh_auto_test] Error 25
make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
make: *** [debian/rules:14: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
I: copying local configuration
E: Failed autobuilding of package
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /srv/workspace/pbuilder/3202066 and its subdirectories
Tue Aug 19 09:03:06 UTC 2025  W: No second build log, what happened?