Thu Mar 20 17:50:07 UTC 2025  I: starting to build golang-github-hashicorp-raft/trixie/amd64 on jenkins on '2025-03-20 17:49'
Thu Mar 20 17:50:07 UTC 2025  I: The jenkins build log is/was available at https://jenkins.debian.net/userContent/reproducible/debian/build_service/amd64_24/52028/console.log
Thu Mar 20 17:50:07 UTC 2025  I: Downloading source for trixie/golang-github-hashicorp-raft=1.5.0-1
--2025-03-20 17:50:07--  http://deb.debian.org/debian/pool/main/g/golang-github-hashicorp-raft/golang-github-hashicorp-raft_1.5.0-1.dsc
Connecting to 46.16.76.132:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 2480 (2.4K) [text/prs.lines.tag]
Saving to: ‘golang-github-hashicorp-raft_1.5.0-1.dsc’

     0K ..                                                    100%  247M=0s

2025-03-20 17:50:07 (247 MB/s) - ‘golang-github-hashicorp-raft_1.5.0-1.dsc’ saved [2480/2480]

Thu Mar 20 17:50:07 UTC 2025  I: golang-github-hashicorp-raft_1.5.0-1.dsc
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 3.0 (quilt)
Source: golang-github-hashicorp-raft
Binary: golang-github-hashicorp-raft-dev
Architecture: all
Version: 1.5.0-1
Maintainer: Debian Go Packaging Team <team+pkg-go@tracker.debian.org>
Uploaders: Tim Potter <tpot@hpe.com>, Dmitry Smirnov <onlyjob@debian.org>,
Homepage: https://github.com/hashicorp/raft
Standards-Version: 4.6.2
Vcs-Browser: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft
Vcs-Git: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git
Testsuite: autopkgtest-pkg-go
Build-Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
Package-List:
 golang-github-hashicorp-raft-dev deb golang optional arch=all
Checksums-Sha1:
 cea1c93d584c6f8dfbf53aef3438ab18ec7e7222 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 015fac87182a135b38ce31d1eb99f68ca251ae23 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Checksums-Sha256:
 f4dc9aabc7784a1b9b5da36b85a3797ac37911f2a6dc54a6fd6cb60a6fd643e9 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 0d95ee03abc0b447cfb92e400d914ecee5113fd181a4ea1d51fa5b6c37288658 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Files:
 7ebc703c8f7e2d7fc02568216f2e3b0a 159293 golang-github-hashicorp-raft_1.5.0.orig.tar.gz
 1f0ee65ab76bca0e05a75b8f9ad32d8d 3964 golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
Go-Import-Path: github.com/hashicorp/raft

-----BEGIN PGP SIGNATURE-----

iQJKBAEBCgA0FiEEMD9oek78sa58GjWjtwAXP7uAWikFAmTCiiIWHGRzd2FyYnJp
Y2tAZGViaWFuLm9yZwAKCRC3ABc/u4BaKYS6D/wN81mTx6pt43be/9H7Ovy6z+5p
nv0+kgrOLtH5y4KUjNLZJuPljAp+uoELmotxp1wqkoLSYs1AwCZ11X4NVs8YoLXj
sLd+Ub+QcWb0Gejqjw7Nk9VZqqXjwfFJW54n8NC2Zy8D6OelCl47sDPmiu03v3uz
oRu4dp2DW/uKkC0vb4RczT81GBmJ19Tu2ZdDvtk7kwnvF48cqDhXMhUTQAEjoyVU
Q8kfY13blza+uXZk6xhWyYsc+wpTh6RHZIWsbpgejKmEYnZCoCdaR0UqOoau33Bs
T1OXVNM0P6vrmeXyoiEy66Q2XyHeei+HYRUgED16Ujg0qxPxSEVE5timSwcXHa/u
xb5ZcKQwqEoJISjWTFDG8gww8MBPOKNGW5BKRdQSdtDhNEsnmn58jl1lsRs5L6Xg
+DrufjU6KKqqYyOBS+H+s68KXleg2t/qP+K8GZjJl/eX3+Tw5xOdt3ZWDMWxB2qF
gS+ASmxYV0Uu/pd0MLWXjAFcllnl93MzPVkF14A8eYrghujLjN3+oUOD0RQzxpAb
LHc5b0iPnN4/YQtJX3fZi2y6c62BCZ5Slb5sB0bsgNPbk1T3qQrtMPWJXUfr32ys
VOow6eNA+VDxNxHZ3aSUiVC0lcUm6dJDYqcfk/PgDdBR/E7iigkTi3O/2rgW0Y7Z
14FQMVvbK31veGXJjA==
=okTI
-----END PGP SIGNATURE-----
Thu Mar 20 17:50:07 UTC 2025  I: Checking whether the package is not for us
Thu Mar 20 17:50:07 UTC 2025  I: Starting 1st build on remote node ionos15-amd64.debian.net.
Thu Mar 20 17:50:07 UTC 2025  I: Preparing to do remote build '1' on ionos15-amd64.debian.net.
Thu Mar 20 17:53:43 UTC 2025  I: Deleting $TMPDIR on ionos15-amd64.debian.net.
I: pbuilder: network access will be disabled during build
I: Current time: Wed Apr 22 12:13:07 -12 2026
I: pbuilder-time-stamp: 1776903187
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/trixie-reproducible-base.tgz]
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: policy-rc.d already exists
I: Copying source file
I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc]
I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz]
I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz]
I: Extracting source
dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: unsupported subcommand
dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying 0001-Skip-unreliable-tests.patch
I: Not using root during the build.
I: Installing the build-deps
I: user script /srv/workspace/pbuilder/598648/tmp/hooks/D02_print_environment starting
I: set
  BUILDDIR='/build/reproducible-path'
  BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other'
  BUILDUSERNAME='pbuilder1'
  BUILD_ARCH='amd64'
  DEBIAN_FRONTEND='noninteractive'
  DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=42 '
  DISTRIBUTION='trixie'
  HOME='/root'
  HOST_ARCH='amd64'
  IFS=' 	
  '
  INVOCATION_ID='d5101bcc33ef401db56e91f03a166507'
  LANG='C'
  LANGUAGE='en_US:en'
  LC_ALL='C'
  MAIL='/var/mail/root'
  OPTIND='1'
  PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games'
  PBCURRENTCOMMANDLINEOPERATION='build'
  PBUILDER_OPERATION='build'
  PBUILDER_PKGDATADIR='/usr/share/pbuilder'
  PBUILDER_PKGLIBDIR='/usr/lib/pbuilder'
  PBUILDER_SYSCONFDIR='/etc'
  PPID='598648'
  PS1='# '
  PS2='> '
  PS4='+ '
  PWD='/'
  SHELL='/bin/bash'
  SHLVL='2'
  SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.3GHnaTNl/pbuilderrc_j3Ok --distribution trixie --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/trixie-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.3GHnaTNl/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc'
  SUDO_GID='111'
  SUDO_UID='106'
  SUDO_USER='jenkins'
  TERM='unknown'
  TZ='/usr/share/zoneinfo/Etc/GMT+12'
  USER='root'
  _='/usr/bin/systemd-run'
  http_proxy='http://213.165.73.152:3128'
I: uname -a
  Linux ionos15-amd64 6.12.12+bpo-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.12-1~bpo12+1 (2025-02-23) x86_64 GNU/Linux
I: ls -l /bin
  lrwxrwxrwx 1 root root 7 Mar  4  2025 /bin -> usr/bin
I: user script /srv/workspace/pbuilder/598648/tmp/hooks/D02_print_environment finished
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: amd64
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 19785 files and directories currently installed.)
Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ...
Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however:
  Package debhelper-compat is not installed.
 pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however:
  Package dh-sequence-golang is not installed.
 pbuilder-satisfydepends-dummy depends on golang-any; however:
  Package golang-any is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however:
  Package golang-github-armon-go-metrics-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however:
  Package golang-github-hashicorp-go-hclog-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however:
  Package golang-github-hashicorp-go-msgpack-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however:
  Package golang-github-stretchr-testify-dev is not installed.

Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
Building tag database...
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
The following NEW packages will be installed:
  autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.24-go{a} golang-1.24-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-prometheus-sigv4-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libunistring5{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} 
The following packages are RECOMMENDED but will NOT be installed:
  curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 
0 packages upgraded, 107 newly installed, 0 to remove and 0 not upgraded.
Need to get 132 MB of archives. After unpacking 714 MB will be used.
Writing extended state information...
Get: 1 http://deb.debian.org/debian trixie/main amd64 sensible-utils all 0.0.24 [24.8 kB]
Get: 2 http://deb.debian.org/debian trixie/main amd64 libmagic-mgc amd64 1:5.45-3+b1 [314 kB]
Get: 3 http://deb.debian.org/debian trixie/main amd64 libmagic1t64 amd64 1:5.45-3+b1 [108 kB]
Get: 4 http://deb.debian.org/debian trixie/main amd64 file amd64 1:5.45-3+b1 [43.3 kB]
Get: 5 http://deb.debian.org/debian trixie/main amd64 gettext-base amd64 0.23.1-1 [243 kB]
Get: 6 http://deb.debian.org/debian trixie/main amd64 libuchardet0 amd64 0.0.8-1+b2 [68.9 kB]
Get: 7 http://deb.debian.org/debian trixie/main amd64 groff-base amd64 1.23.0-7 [1185 kB]
Get: 8 http://deb.debian.org/debian trixie/main amd64 bsdextrautils amd64 2.40.4-5 [92.4 kB]
Get: 9 http://deb.debian.org/debian trixie/main amd64 libpipeline1 amd64 1.5.8-1 [42.0 kB]
Get: 10 http://deb.debian.org/debian trixie/main amd64 man-db amd64 2.13.0-1 [1420 kB]
Get: 11 http://deb.debian.org/debian trixie/main amd64 m4 amd64 1.4.19-7 [294 kB]
Get: 12 http://deb.debian.org/debian trixie/main amd64 autoconf all 2.72-3 [493 kB]
Get: 13 http://deb.debian.org/debian trixie/main amd64 autotools-dev all 20220109.1 [51.6 kB]
Get: 14 http://deb.debian.org/debian trixie/main amd64 automake all 1:1.17-3 [862 kB]
Get: 15 http://deb.debian.org/debian trixie/main amd64 autopoint all 0.23.1-1 [770 kB]
Get: 16 http://deb.debian.org/debian trixie/main amd64 libdebhelper-perl all 13.24.1 [90.9 kB]
Get: 17 http://deb.debian.org/debian trixie/main amd64 libtool all 2.5.4-4 [539 kB]
Get: 18 http://deb.debian.org/debian trixie/main amd64 dh-autoreconf all 20 [17.1 kB]
Get: 19 http://deb.debian.org/debian trixie/main amd64 libarchive-zip-perl all 1.68-1 [104 kB]
Get: 20 http://deb.debian.org/debian trixie/main amd64 libfile-stripnondeterminism-perl all 1.14.1-2 [19.7 kB]
Get: 21 http://deb.debian.org/debian trixie/main amd64 dh-strip-nondeterminism all 1.14.1-2 [8620 B]
Get: 22 http://deb.debian.org/debian trixie/main amd64 libelf1t64 amd64 0.192-4 [189 kB]
Get: 23 http://deb.debian.org/debian trixie/main amd64 dwz amd64 0.15-1+b1 [110 kB]
Get: 24 http://deb.debian.org/debian trixie/main amd64 libunistring5 amd64 1.3-1 [476 kB]
Get: 25 http://deb.debian.org/debian trixie/main amd64 libicu72 amd64 72.1-6 [9421 kB]
Get: 26 http://deb.debian.org/debian trixie/main amd64 libxml2 amd64 2.12.7+dfsg+really2.9.14-0.2+b2 [699 kB]
Get: 27 http://deb.debian.org/debian trixie/main amd64 gettext amd64 0.23.1-1 [1680 kB]
Get: 28 http://deb.debian.org/debian trixie/main amd64 intltool-debian all 0.35.0+20060710.6 [22.9 kB]
Get: 29 http://deb.debian.org/debian trixie/main amd64 po-debconf all 1.0.21+nmu1 [248 kB]
Get: 30 http://deb.debian.org/debian trixie/main amd64 debhelper all 13.24.1 [920 kB]
Get: 31 http://deb.debian.org/debian trixie/main amd64 dh-golang all 1.63 [24.4 kB]
Get: 32 http://deb.debian.org/debian trixie/main amd64 golang-1.24-src all 1.24.1-1 [21.2 MB]
Get: 33 http://deb.debian.org/debian trixie/main amd64 golang-1.24-go amd64 1.24.1-1 [28.6 MB]
Get: 34 http://deb.debian.org/debian trixie/main amd64 golang-src all 2:1.24~2 [5136 B]
Get: 35 http://deb.debian.org/debian trixie/main amd64 golang-go amd64 2:1.24~2 [44.3 kB]
Get: 36 http://deb.debian.org/debian trixie/main amd64 golang-any amd64 2:1.24~2 [5216 B]
Get: 37 http://deb.debian.org/debian trixie/main amd64 golang-github-alecthomas-units-dev all 0.0~git20240927.0f3dac3-1 [8880 B]
Get: 38 http://deb.debian.org/debian trixie/main amd64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB]
Get: 39 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB]
Get: 40 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-sys-dev all 0.22.0-1 [431 kB]
Get: 41 http://deb.debian.org/debian trixie/main amd64 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B]
Get: 42 http://deb.debian.org/debian trixie/main amd64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB]
Get: 43 http://deb.debian.org/debian trixie/main amd64 golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB]
Get: 44 http://deb.debian.org/debian trixie/main amd64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB]
Get: 45 http://deb.debian.org/debian trixie/main amd64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB]
Get: 46 http://deb.debian.org/debian trixie/main amd64 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB]
Get: 47 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB]
Get: 48 http://deb.debian.org/debian trixie/main amd64 golang-github-stretchr-testify-dev all 1.10.0-1 [78.4 kB]
Get: 49 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB]
Get: 50 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB]
Get: 51 http://deb.debian.org/debian trixie/main amd64 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB]
Get: 52 http://deb.debian.org/debian trixie/main amd64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B]
Get: 53 http://deb.debian.org/debian trixie/main amd64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB]
Get: 54 http://deb.debian.org/debian trixie/main amd64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB]
Get: 55 http://deb.debian.org/debian trixie/main amd64 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB]
Get: 56 http://deb.debian.org/debian trixie/main amd64 golang-google-protobuf-dev all 1.36.5-1 [796 kB]
Get: 57 http://deb.debian.org/debian trixie/main amd64 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB]
Get: 58 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B]
Get: 59 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB]
Get: 60 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB]
Get: 61 http://deb.debian.org/debian trixie/main amd64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB]
Get: 62 http://deb.debian.org/debian trixie/main amd64 golang-github-beorn7-perks-dev all 1.0.1-3 [12.2 kB]
Get: 63 http://deb.debian.org/debian trixie/main amd64 golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB]
Get: 64 http://deb.debian.org/debian trixie/main amd64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B]
Get: 65 http://deb.debian.org/debian trixie/main amd64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB]
Get: 66 http://deb.debian.org/debian trixie/main amd64 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB]
Get: 67 http://deb.debian.org/debian trixie/main amd64 golang-github-klauspost-compress-dev all 1.18.0+ds1-1 [33.3 MB]
Get: 68 http://deb.debian.org/debian trixie/main amd64 golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB]
Get: 69 http://deb.debian.org/debian trixie/main amd64 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB]
Get: 70 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-yaml.v2-dev all 2.4.0-5 [60.5 kB]
Get: 71 http://deb.debian.org/debian trixie/main amd64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB]
Get: 72 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-term-dev all 0.22.0-1 [14.8 kB]
Get: 73 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-text-dev all 0.22.0-1 [3995 kB]
Get: 74 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-crypto-dev all 1:0.25.0-1 [1682 kB]
Get: 75 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-net-dev all 1:0.27.0-1 [898 kB]
Get: 76 http://deb.debian.org/debian trixie/main amd64 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB]
Get: 77 http://deb.debian.org/debian trixie/main amd64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB]
Get: 78 http://deb.debian.org/debian trixie/main amd64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB]
Get: 79 http://deb.debian.org/debian trixie/main amd64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB]
Get: 80 http://deb.debian.org/debian trixie/main amd64 golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B]
Get: 81 http://deb.debian.org/debian trixie/main amd64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B]
Get: 82 http://deb.debian.org/debian trixie/main amd64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-4 [14.6 kB]
Get: 83 http://deb.debian.org/debian trixie/main amd64 golang-github-prometheus-sigv4-dev all 0.1.2-2 [8484 B]
Get: 84 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-oauth2-dev all 0.27.0-1 [37.2 kB]
Get: 85 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB]
Get: 86 http://deb.debian.org/debian trixie/main amd64 golang-github-prometheus-common-dev all 0.62.0-3 [118 kB]
Get: 87 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB]
Get: 88 http://deb.debian.org/debian trixie/main amd64 golang-github-prometheus-procfs-dev all 0.15.1-1 [184 kB]
Get: 89 http://deb.debian.org/debian trixie/main amd64 golang-github-prometheus-client-golang-dev all 1.21.1-1 [194 kB]
Get: 90 http://deb.debian.org/debian trixie/main amd64 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB]
Get: 91 http://deb.debian.org/debian trixie/main amd64 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB]
Get: 92 http://deb.debian.org/debian trixie/main amd64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B]
Get: 93 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB]
Get: 94 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B]
Get: 95 http://deb.debian.org/debian trixie/main amd64 libsasl2-modules-db amd64 2.1.28+dfsg1-9 [19.8 kB]
Get: 96 http://deb.debian.org/debian trixie/main amd64 libsasl2-2 amd64 2.1.28+dfsg1-9 [57.5 kB]
Get: 97 http://deb.debian.org/debian trixie/main amd64 libssl-dev amd64 3.4.1-1 [2787 kB]
Get: 98 http://deb.debian.org/debian trixie/main amd64 libsasl2-dev amd64 2.1.28+dfsg1-9 [234 kB]
Get: 99 http://deb.debian.org/debian trixie/main amd64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB]
Get: 100 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-mod-dev all 0.19.0-1 [104 kB]
Get: 101 http://deb.debian.org/debian trixie/main amd64 golang-github-yuin-goldmark-dev all 1.7.4-1 [113 kB]
Get: 102 http://deb.debian.org/debian trixie/main amd64 golang-golang-x-tools-dev all 1:0.25.0+ds-1 [1608 kB]
Get: 103 http://deb.debian.org/debian trixie/main amd64 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB]
Get: 104 http://deb.debian.org/debian trixie/main amd64 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB]
Get: 105 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-2 [31.2 kB]
Get: 106 http://deb.debian.org/debian trixie/main amd64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB]
Get: 107 http://deb.debian.org/debian trixie/main amd64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB]
Fetched 132 MB in 2s (55.1 MB/s)
Preconfiguring packages ...
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 19785 files and directories currently installed.)
Preparing to unpack .../000-sensible-utils_0.0.24_all.deb ...
Unpacking sensible-utils (0.0.24) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3+b1_amd64.deb ...
Unpacking libmagic-mgc (1:5.45-3+b1) ...
Selecting previously unselected package libmagic1t64:amd64.
Preparing to unpack .../002-libmagic1t64_1%3a5.45-3+b1_amd64.deb ...
Unpacking libmagic1t64:amd64 (1:5.45-3+b1) ...
Selecting previously unselected package file.
Preparing to unpack .../003-file_1%3a5.45-3+b1_amd64.deb ...
Unpacking file (1:5.45-3+b1) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../004-gettext-base_0.23.1-1_amd64.deb ...
Unpacking gettext-base (0.23.1-1) ...
Selecting previously unselected package libuchardet0:amd64.
Preparing to unpack .../005-libuchardet0_0.0.8-1+b2_amd64.deb ...
Unpacking libuchardet0:amd64 (0.0.8-1+b2) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../006-groff-base_1.23.0-7_amd64.deb ...
Unpacking groff-base (1.23.0-7) ...
Selecting previously unselected package bsdextrautils.
Preparing to unpack .../007-bsdextrautils_2.40.4-5_amd64.deb ...
Unpacking bsdextrautils (2.40.4-5) ...
Selecting previously unselected package libpipeline1:amd64.
Preparing to unpack .../008-libpipeline1_1.5.8-1_amd64.deb ...
Unpacking libpipeline1:amd64 (1.5.8-1) ...
Selecting previously unselected package man-db.
Preparing to unpack .../009-man-db_2.13.0-1_amd64.deb ...
Unpacking man-db (2.13.0-1) ...
Selecting previously unselected package m4.
Preparing to unpack .../010-m4_1.4.19-7_amd64.deb ...
Unpacking m4 (1.4.19-7) ...
Selecting previously unselected package autoconf.
Preparing to unpack .../011-autoconf_2.72-3_all.deb ...
Unpacking autoconf (2.72-3) ...
Selecting previously unselected package autotools-dev.
Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ...
Unpacking autotools-dev (20220109.1) ...
Selecting previously unselected package automake.
Preparing to unpack .../013-automake_1%3a1.17-3_all.deb ...
Unpacking automake (1:1.17-3) ...
Selecting previously unselected package autopoint.
Preparing to unpack .../014-autopoint_0.23.1-1_all.deb ...
Unpacking autopoint (0.23.1-1) ...
Selecting previously unselected package libdebhelper-perl.
Preparing to unpack .../015-libdebhelper-perl_13.24.1_all.deb ...
Unpacking libdebhelper-perl (13.24.1) ...
Selecting previously unselected package libtool.
Preparing to unpack .../016-libtool_2.5.4-4_all.deb ...
Unpacking libtool (2.5.4-4) ...
Selecting previously unselected package dh-autoreconf.
Preparing to unpack .../017-dh-autoreconf_20_all.deb ...
Unpacking dh-autoreconf (20) ...
Selecting previously unselected package libarchive-zip-perl.
Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ...
Unpacking libarchive-zip-perl (1.68-1) ...
Selecting previously unselected package libfile-stripnondeterminism-perl.
Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.1-2_all.deb ...
Unpacking libfile-stripnondeterminism-perl (1.14.1-2) ...
Selecting previously unselected package dh-strip-nondeterminism.
Preparing to unpack .../020-dh-strip-nondeterminism_1.14.1-2_all.deb ...
Unpacking dh-strip-nondeterminism (1.14.1-2) ...
Selecting previously unselected package libelf1t64:amd64.
Preparing to unpack .../021-libelf1t64_0.192-4_amd64.deb ...
Unpacking libelf1t64:amd64 (0.192-4) ...
Selecting previously unselected package dwz.
Preparing to unpack .../022-dwz_0.15-1+b1_amd64.deb ...
Unpacking dwz (0.15-1+b1) ...
Selecting previously unselected package libunistring5:amd64.
Preparing to unpack .../023-libunistring5_1.3-1_amd64.deb ...
Unpacking libunistring5:amd64 (1.3-1) ...
Selecting previously unselected package libicu72:amd64.
Preparing to unpack .../024-libicu72_72.1-6_amd64.deb ...
Unpacking libicu72:amd64 (72.1-6) ...
Selecting previously unselected package libxml2:amd64.
Preparing to unpack .../025-libxml2_2.12.7+dfsg+really2.9.14-0.2+b2_amd64.deb ...
Unpacking libxml2:amd64 (2.12.7+dfsg+really2.9.14-0.2+b2) ...
Selecting previously unselected package gettext.
Preparing to unpack .../026-gettext_0.23.1-1_amd64.deb ...
Unpacking gettext (0.23.1-1) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../027-intltool-debian_0.35.0+20060710.6_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.6) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../028-po-debconf_1.0.21+nmu1_all.deb ...
Unpacking po-debconf (1.0.21+nmu1) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../029-debhelper_13.24.1_all.deb ...
Unpacking debhelper (13.24.1) ...
Selecting previously unselected package dh-golang.
Preparing to unpack .../030-dh-golang_1.63_all.deb ...
Unpacking dh-golang (1.63) ...
Selecting previously unselected package golang-1.24-src.
Preparing to unpack .../031-golang-1.24-src_1.24.1-1_all.deb ...
Unpacking golang-1.24-src (1.24.1-1) ...
Selecting previously unselected package golang-1.24-go.
Preparing to unpack .../032-golang-1.24-go_1.24.1-1_amd64.deb ...
Unpacking golang-1.24-go (1.24.1-1) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../033-golang-src_2%3a1.24~2_all.deb ...
Unpacking golang-src (2:1.24~2) ...
Selecting previously unselected package golang-go:amd64.
Preparing to unpack .../034-golang-go_2%3a1.24~2_amd64.deb ...
Unpacking golang-go:amd64 (2:1.24~2) ...
Selecting previously unselected package golang-any:amd64.
Preparing to unpack .../035-golang-any_2%3a1.24~2_amd64.deb ...
Unpacking golang-any:amd64 (2:1.24~2) ...
Selecting previously unselected package golang-github-alecthomas-units-dev.
Preparing to unpack .../036-golang-github-alecthomas-units-dev_0.0~git20240927.0f3dac3-1_all.deb ...
Unpacking golang-github-alecthomas-units-dev (0.0~git20240927.0f3dac3-1) ...
Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev.
Preparing to unpack .../037-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ...
Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev.
Preparing to unpack .../038-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ...
Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Selecting previously unselected package golang-golang-x-sys-dev.
Preparing to unpack .../039-golang-golang-x-sys-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-sys-dev (0.22.0-1) ...
Selecting previously unselected package golang-github-mattn-go-isatty-dev.
Preparing to unpack .../040-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ...
Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Selecting previously unselected package golang-github-mattn-go-colorable-dev.
Preparing to unpack .../041-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ...
Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Selecting previously unselected package golang-github-fatih-color-dev.
Preparing to unpack .../042-golang-github-fatih-color-dev_1.17.0-1_all.deb ...
Unpacking golang-github-fatih-color-dev (1.17.0-1) ...
Selecting previously unselected package golang-github-davecgh-go-spew-dev.
Preparing to unpack .../043-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ...
Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Selecting previously unselected package golang-github-pmezard-go-difflib-dev.
Preparing to unpack .../044-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ...
Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Selecting previously unselected package golang-github-stretchr-objx-dev.
Preparing to unpack .../045-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ...
Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ...
Selecting previously unselected package golang-gopkg-yaml.v3-dev.
Preparing to unpack .../046-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ...
Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Selecting previously unselected package golang-github-stretchr-testify-dev.
Preparing to unpack .../047-golang-github-stretchr-testify-dev_1.10.0-1_all.deb ...
Unpacking golang-github-stretchr-testify-dev (1.10.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-hclog-dev.
Preparing to unpack .../048-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ...
Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev.
Preparing to unpack .../049-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ...
Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Selecting previously unselected package golang-github-pkg-errors-dev.
Preparing to unpack .../050-golang-github-pkg-errors-dev_0.9.1-3_all.deb ...
Unpacking golang-github-pkg-errors-dev (0.9.1-3) ...
Selecting previously unselected package golang-github-tv42-httpunix-dev.
Preparing to unpack .../051-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ...
Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev.
Preparing to unpack .../052-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ...
Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Selecting previously unselected package golang-github-datadog-datadog-go-dev.
Preparing to unpack .../053-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ...
Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Selecting previously unselected package golang-github-google-go-cmp-dev.
Preparing to unpack .../054-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ...
Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ...
Selecting previously unselected package golang-google-protobuf-dev.
Preparing to unpack .../055-golang-google-protobuf-dev_1.36.5-1_all.deb ...
Unpacking golang-google-protobuf-dev (1.36.5-1) ...
Selecting previously unselected package golang-github-golang-protobuf-1-5-dev.
Preparing to unpack .../056-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ...
Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Selecting previously unselected package golang-github-hashicorp-go-uuid-dev.
Preparing to unpack .../057-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ...
Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Selecting previously unselected package golang-github-hashicorp-golang-lru-dev.
Preparing to unpack .../058-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ...
Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev.
Preparing to unpack .../059-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ...
Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Selecting previously unselected package golang-github-pascaldekloe-goe-dev.
Preparing to unpack .../060-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ...
Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Selecting previously unselected package golang-github-beorn7-perks-dev.
Preparing to unpack .../061-golang-github-beorn7-perks-dev_1.0.1-3_all.deb ...
Unpacking golang-github-beorn7-perks-dev (1.0.1-3) ...
Selecting previously unselected package golang-github-cespare-xxhash-dev.
Preparing to unpack .../062-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ...
Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ...
Selecting previously unselected package golang-github-modern-go-concurrent-dev.
Preparing to unpack .../063-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ...
Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Selecting previously unselected package golang-github-modern-go-reflect2-dev.
Preparing to unpack .../064-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ...
Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-json-iterator-go-dev.
Preparing to unpack .../065-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ...
Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ...
Selecting previously unselected package golang-github-klauspost-compress-dev.
Preparing to unpack .../066-golang-github-klauspost-compress-dev_1.18.0+ds1-1_all.deb ...
Unpacking golang-github-klauspost-compress-dev (1.18.0+ds1-1) ...
Selecting previously unselected package golang-github-kylelemons-godebug-dev.
Preparing to unpack .../067-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ...
Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Selecting previously unselected package golang-github-prometheus-client-model-dev.
Preparing to unpack .../068-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ...
Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ...
Selecting previously unselected package golang-gopkg-yaml.v2-dev.
Preparing to unpack .../069-golang-gopkg-yaml.v2-dev_2.4.0-5_all.deb ...
Unpacking golang-gopkg-yaml.v2-dev (2.4.0-5) ...
Selecting previously unselected package golang-github-jmespath-go-jmespath-dev.
Preparing to unpack .../070-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ...
Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Selecting previously unselected package golang-golang-x-term-dev.
Preparing to unpack .../071-golang-golang-x-term-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-term-dev (0.22.0-1) ...
Selecting previously unselected package golang-golang-x-text-dev.
Preparing to unpack .../072-golang-golang-x-text-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-text-dev (0.22.0-1) ...
Selecting previously unselected package golang-golang-x-crypto-dev.
Preparing to unpack .../073-golang-golang-x-crypto-dev_1%3a0.25.0-1_all.deb ...
Unpacking golang-golang-x-crypto-dev (1:0.25.0-1) ...
Selecting previously unselected package golang-golang-x-net-dev.
Preparing to unpack .../074-golang-golang-x-net-dev_1%3a0.27.0-1_all.deb ...
Unpacking golang-golang-x-net-dev (1:0.27.0-1) ...
Selecting previously unselected package golang-github-aws-aws-sdk-go-dev.
Preparing to unpack .../075-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ...
Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Selecting previously unselected package golang-github-go-logfmt-logfmt-dev.
Preparing to unpack .../076-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ...
Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Selecting previously unselected package golang-github-go-kit-log-dev.
Preparing to unpack .../077-golang-github-go-kit-log-dev_0.2.1-1_all.deb ...
Unpacking golang-github-go-kit-log-dev (0.2.1-1) ...
Selecting previously unselected package golang-github-julienschmidt-httprouter-dev.
Preparing to unpack .../078-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ...
Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Selecting previously unselected package golang-github-munnerz-goautoneg-dev.
Preparing to unpack .../079-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ...
Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Selecting previously unselected package golang-github-jpillora-backoff-dev.
Preparing to unpack .../080-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ...
Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev.
Preparing to unpack .../081-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-4_all.deb ...
Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ...
Selecting previously unselected package golang-github-prometheus-sigv4-dev.
Preparing to unpack .../082-golang-github-prometheus-sigv4-dev_0.1.2-2_all.deb ...
Unpacking golang-github-prometheus-sigv4-dev (0.1.2-2) ...
Selecting previously unselected package golang-golang-x-oauth2-dev.
Preparing to unpack .../083-golang-golang-x-oauth2-dev_0.27.0-1_all.deb ...
Unpacking golang-golang-x-oauth2-dev (0.27.0-1) ...
Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev.
Preparing to unpack .../084-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ...
Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Selecting previously unselected package golang-github-prometheus-common-dev.
Preparing to unpack .../085-golang-github-prometheus-common-dev_0.62.0-3_all.deb ...
Unpacking golang-github-prometheus-common-dev (0.62.0-3) ...
Selecting previously unselected package golang-golang-x-sync-dev.
Preparing to unpack .../086-golang-golang-x-sync-dev_0.7.0-1_all.deb ...
Unpacking golang-golang-x-sync-dev (0.7.0-1) ...
Selecting previously unselected package golang-github-prometheus-procfs-dev.
Preparing to unpack .../087-golang-github-prometheus-procfs-dev_0.15.1-1_all.deb ...
Unpacking golang-github-prometheus-procfs-dev (0.15.1-1) ...
Selecting previously unselected package golang-github-prometheus-client-golang-dev.
Preparing to unpack .../088-golang-github-prometheus-client-golang-dev_1.21.1-1_all.deb ...
Unpacking golang-github-prometheus-client-golang-dev (1.21.1-1) ...
Selecting previously unselected package golang-github-armon-go-metrics-dev.
Preparing to unpack .../089-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ...
Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ...
Selecting previously unselected package golang-github-creack-pty-dev.
Preparing to unpack .../090-golang-github-creack-pty-dev_1.1.21-1_all.deb ...
Unpacking golang-github-creack-pty-dev (1.1.21-1) ...
Selecting previously unselected package golang-github-vmihailenco-tagparser-dev.
Preparing to unpack .../091-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ...
Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev.
Preparing to unpack .../092-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ...
Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Selecting previously unselected package golang-gopkg-tomb.v2-dev.
Preparing to unpack .../093-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ...
Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../094-libsasl2-modules-db_2.1.28+dfsg1-9_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.28+dfsg1-9) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../095-libsasl2-2_2.1.28+dfsg1-9_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.28+dfsg1-9) ...
Selecting previously unselected package libssl-dev:amd64.
Preparing to unpack .../096-libssl-dev_3.4.1-1_amd64.deb ...
Unpacking libssl-dev:amd64 (3.4.1-1) ...
Selecting previously unselected package libsasl2-dev.
Preparing to unpack .../097-libsasl2-dev_2.1.28+dfsg1-9_amd64.deb ...
Unpacking libsasl2-dev (2.1.28+dfsg1-9) ...
Selecting previously unselected package golang-github-kr-text-dev.
Preparing to unpack .../098-golang-github-kr-text-dev_0.2.0-2_all.deb ...
Unpacking golang-github-kr-text-dev (0.2.0-2) ...
Selecting previously unselected package golang-golang-x-mod-dev.
Preparing to unpack .../099-golang-golang-x-mod-dev_0.19.0-1_all.deb ...
Unpacking golang-golang-x-mod-dev (0.19.0-1) ...
Selecting previously unselected package golang-github-yuin-goldmark-dev.
Preparing to unpack .../100-golang-github-yuin-goldmark-dev_1.7.4-1_all.deb ...
Unpacking golang-github-yuin-goldmark-dev (1.7.4-1) ...
Selecting previously unselected package golang-golang-x-tools-dev.
Preparing to unpack .../101-golang-golang-x-tools-dev_1%3a0.25.0+ds-1_all.deb ...
Unpacking golang-golang-x-tools-dev (1:0.25.0+ds-1) ...
Selecting previously unselected package golang-github-rogpeppe-go-internal-dev.
Preparing to unpack .../102-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ...
Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Selecting previously unselected package golang-github-kr-pretty-dev.
Preparing to unpack .../103-golang-github-kr-pretty-dev_0.3.1-1_all.deb ...
Unpacking golang-github-kr-pretty-dev (0.3.1-1) ...
Selecting previously unselected package golang-gopkg-check.v1-dev.
Preparing to unpack .../104-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-2_all.deb ...
Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ...
Selecting previously unselected package golang-gopkg-mgo.v2-dev.
Preparing to unpack .../105-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ...
Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev.
Preparing to unpack .../106-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ...
Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-1.24-src (1.24.1-1) ...
Setting up libpipeline1:amd64 (1.5.8-1) ...
Setting up golang-github-google-go-cmp-dev (0.6.0-1) ...
Setting up golang-gopkg-yaml.v2-dev (2.4.0-5) ...
Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Setting up libicu72:amd64 (72.1-6) ...
Setting up bsdextrautils (2.40.4-5) ...
Setting up golang-github-creack-pty-dev (1.1.21-1) ...
Setting up golang-github-stretchr-objx-dev (0.5.2-1) ...
Setting up libmagic-mgc (1:5.45-3+b1) ...
Setting up golang-github-pkg-errors-dev (0.9.1-3) ...
Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Setting up libarchive-zip-perl (1.68-1) ...
Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Setting up libdebhelper-perl (13.24.1) ...
Setting up golang-golang-x-sys-dev (0.22.0-1) ...
Setting up libmagic1t64:amd64 (1:5.45-3+b1) ...
Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Setting up gettext-base (0.23.1-1) ...
Setting up m4 (1.4.19-7) ...
Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Setting up golang-golang-x-term-dev (0.22.0-1) ...
Setting up file (1:5.45-3+b1) ...
Setting up golang-github-beorn7-perks-dev (1.0.1-3) ...
Setting up libelf1t64:amd64 (0.192-4) ...
Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Setting up libsasl2-modules-db:amd64 (2.1.28+dfsg1-9) ...
Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ...
Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Setting up autotools-dev (20220109.1) ...
Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Setting up golang-google-protobuf-dev (1.36.5-1) ...
Setting up golang-github-kr-text-dev (0.2.0-2) ...
Setting up golang-github-stretchr-testify-dev (1.10.0-1) ...
Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Setting up golang-github-yuin-goldmark-dev (1.7.4-1) ...
Setting up libunistring5:amd64 (1.3-1) ...
Setting up libssl-dev:amd64 (3.4.1-1) ...
Setting up golang-golang-x-sync-dev (0.7.0-1) ...
Setting up golang-golang-x-mod-dev (0.19.0-1) ...
Setting up autopoint (0.23.1-1) ...
Setting up golang-1.24-go (1.24.1-1) ...
Setting up golang-github-klauspost-compress-dev (1.18.0+ds1-1) ...
Setting up libsasl2-2:amd64 (2.1.28+dfsg1-9) ...
Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Setting up autoconf (2.72-3) ...
Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Setting up dwz (0.15-1+b1) ...
Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Setting up sensible-utils (0.0.24) ...
Setting up golang-golang-x-text-dev (0.22.0-1) ...
Setting up libuchardet0:amd64 (0.0.8-1+b2) ...
Setting up golang-github-prometheus-procfs-dev (0.15.1-1) ...
Setting up golang-github-alecthomas-units-dev (0.0~git20240927.0f3dac3-1) ...
Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Setting up golang-golang-x-crypto-dev (1:0.25.0-1) ...
Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ...
Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Setting up golang-golang-x-oauth2-dev (0.27.0-1) ...
Setting up golang-src (2:1.24~2) ...
Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Setting up libxml2:amd64 (2.12.7+dfsg+really2.9.14-0.2+b2) ...
Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Setting up automake (1:1.17-3) ...
update-alternatives: using /usr/bin/automake-1.17 to provide /usr/bin/automake (automake) in auto mode
Setting up libfile-stripnondeterminism-perl (1.14.1-2) ...
Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Setting up golang-github-go-kit-log-dev (0.2.1-1) ...
Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Setting up gettext (0.23.1-1) ...
Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Setting up libtool (2.5.4-4) ...
Setting up golang-go:amd64 (2:1.24~2) ...
Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Setting up golang-github-json-iterator-go-dev (1.1.12-2) ...
Setting up golang-any:amd64 (2:1.24~2) ...
Setting up intltool-debian (0.35.0+20060710.6) ...
Setting up dh-autoreconf (20) ...
Setting up libsasl2-dev (2.1.28+dfsg1-9) ...
Setting up golang-golang-x-net-dev (1:0.27.0-1) ...
Setting up dh-strip-nondeterminism (1.14.1-2) ...
Setting up groff-base (1.23.0-7) ...
Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Setting up golang-github-fatih-color-dev (1.17.0-1) ...
Setting up po-debconf (1.0.21+nmu1) ...
Setting up golang-golang-x-tools-dev (1:0.25.0+ds-1) ...
Setting up man-db (2.13.0-1) ...
Not building database; man-db/auto-update is not 'true'.
Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Setting up golang-github-kr-pretty-dev (0.3.1-1) ...
Setting up debhelper (13.24.1) ...
Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ...
Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Setting up dh-golang (1.63) ...
Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ...
Setting up golang-github-prometheus-sigv4-dev (0.1.2-2) ...
Setting up golang-github-prometheus-common-dev (0.62.0-3) ...
Setting up golang-github-prometheus-client-golang-dev (1.21.1-1) ...
Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ...
Processing triggers for libc-bin (2.41-4) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Building tag database...
 -> Finished parsing the build-deps
I: Building the package
I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S  > ../golang-github-hashicorp-raft_1.5.0-1_source.changes
dpkg-buildpackage: info: source package golang-github-hashicorp-raft
dpkg-buildpackage: info: source version 1.5.0-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Daniel Swarbrick <dswarbrick@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture amd64
 debian/rules clean
dh clean --buildsystem=golang
   dh_auto_clean -O--buildsystem=golang
   dh_autoreconf_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules binary
dh binary --buildsystem=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-x86_64-linux-gnu && go install -trimpath -v -p 42 github.com/hashicorp/raft github.com/hashicorp/raft/bench
internal/goarch
internal/goos
internal/profilerecord
vendor/golang.org/x/crypto/cryptobyte/asn1
vendor/golang.org/x/crypto/internal/alias
unicode/utf8
internal/godebugs
internal/msan
internal/unsafeheader
internal/asan
internal/coverage/rtcov
internal/goexperiment
internal/byteorder
unicode/utf16
internal/nettrace
math/bits
unicode
cmp
encoding
crypto/internal/fips140/alias
internal/itoa
log/internal
internal/cpu
container/list
internal/runtime/syscall
crypto/internal/boring/sig
sync/atomic
internal/runtime/atomic
internal/runtime/math
crypto/internal/fips140/subtle
internal/abi
internal/runtime/sys
crypto/internal/fips140deps/byteorder
internal/chacha8rand
crypto/internal/fips140deps/cpu
internal/bytealg
math
internal/runtime/exithook
internal/stringslite
internal/race
internal/sync
internal/runtime/maps
runtime
iter
weak
crypto/subtle
internal/reflectlite
sync
maps
slices
internal/singleflight
internal/testlog
internal/bisect
unique
runtime/cgo
errors
sort
internal/oserror
internal/godebug
io
github.com/hashicorp/golang-lru/simplelru
math/rand/v2
path
vendor/golang.org/x/net/dns/dnsmessage
strconv
hash
bytes
strings
internal/saferio
syscall
hash/crc64
hash/crc32
crypto/internal/fips140deps/godebug
math/rand
crypto/internal/randutil
crypto/internal/impl
net/http/internal/ascii
crypto/internal/fips140
crypto
net/netip
regexp/syntax
html
reflect
vendor/golang.org/x/text/transform
bufio
github.com/hashicorp/go-immutable-radix
crypto/tls/internal/fips140tls
crypto/internal/fips140/sha512
crypto/internal/fips140/sha256
crypto/internal/fips140/sha3
internal/syscall/execenv
internal/syscall/unix
time
crypto/sha3
crypto/internal/fips140/hmac
crypto/internal/fips140/check
crypto/internal/fips140hash
crypto/internal/fips140/hkdf
crypto/internal/fips140/tls12
crypto/internal/fips140/edwards25519/field
crypto/internal/fips140/bigmod
crypto/internal/fips140/aes
crypto/internal/fips140/nistec/fiat
crypto/internal/fips140/tls13
crypto/internal/fips140/edwards25519
regexp
context
io/fs
internal/poll
internal/filepathlite
crypto/internal/fips140/nistec
os
internal/fmtsort
encoding/binary
crypto/internal/sysrand
os/signal
internal/sysinfo
path/filepath
io/ioutil
vendor/golang.org/x/sys/cpu
fmt
crypto/internal/entropy
crypto/internal/fips140/drbg
net
encoding/base64
vendor/golang.org/x/crypto/internal/poly1305
encoding/hex
net/http/internal
net/url
go/token
mime/quotedprintable
text/template/parse
log
runtime/trace
vendor/golang.org/x/net/http2/hpack
vendor/golang.org/x/text/unicode/norm
runtime/debug
flag
encoding/gob
compress/flate
crypto/internal/fips140only
crypto/internal/fips140/ecdh
crypto/internal/fips140/ed25519
crypto/internal/fips140/ecdsa
math/big
crypto/internal/fips140/mlkem
crypto/internal/fips140/aes/gcm
crypto/internal/fips140/rsa
golang.org/x/sys/unix
encoding/pem
mime
encoding/json
crypto/rc4
crypto/md5
vendor/golang.org/x/text/unicode/bidi
testing
crypto/cipher
compress/gzip
text/template
crypto/internal/boring
vendor/golang.org/x/crypto/chacha20
crypto/des
vendor/golang.org/x/text/secure/bidirule
crypto/dsa
crypto/elliptic
encoding/asn1
vendor/golang.org/x/crypto/chacha20poly1305
vendor/golang.org/x/net/idna
crypto/sha256
crypto/sha512
crypto/rand
crypto/hmac
crypto/aes
crypto/internal/boring/bbig
crypto/ecdh
crypto/sha1
crypto/ed25519
crypto/rsa
html/template
crypto/x509/pkix
vendor/golang.org/x/crypto/cryptobyte
github.com/mattn/go-isatty
github.com/mattn/go-colorable
github.com/fatih/color
crypto/ecdsa
github.com/hashicorp/go-hclog
crypto/internal/hpke
vendor/golang.org/x/net/http/httpproxy
net/textproto
crypto/x509
vendor/golang.org/x/net/http/httpguts
mime/multipart
crypto/tls
net/http/httptrace
net/http
net/rpc
github.com/armon/go-metrics
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/raft
github.com/hashicorp/raft/bench
   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
dh_auto_test
	cd obj-x86_64-linux-gnu && go test -vet=off -v -p 42 github.com/hashicorp/raft github.com/hashicorp/raft/bench
=== RUN   TestCommitment_setVoters
--- PASS: TestCommitment_setVoters (0.00s)
=== RUN   TestCommitment_match_max
--- PASS: TestCommitment_match_max (0.00s)
=== RUN   TestCommitment_match_nonVoting
--- PASS: TestCommitment_match_nonVoting (0.00s)
=== RUN   TestCommitment_recalculate
--- PASS: TestCommitment_recalculate (0.00s)
=== RUN   TestCommitment_recalculate_startIndex
--- PASS: TestCommitment_recalculate_startIndex (0.00s)
=== RUN   TestCommitment_noVoterSanity
--- PASS: TestCommitment_noVoterSanity (0.00s)
=== RUN   TestCommitment_singleVoter
--- PASS: TestCommitment_singleVoter (0.00s)
=== RUN   TestConfiguration_Configuration_Clone
--- PASS: TestConfiguration_Configuration_Clone (0.00s)
=== RUN   TestConfiguration_configurations_Clone
--- PASS: TestConfiguration_configurations_Clone (0.00s)
=== RUN   TestConfiguration_hasVote
--- PASS: TestConfiguration_hasVote (0.00s)
=== RUN   TestConfiguration_checkConfiguration
--- PASS: TestConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_nextConfiguration_table
--- PASS: TestConfiguration_nextConfiguration_table (0.00s)
=== RUN   TestConfiguration_nextConfiguration_prevIndex
--- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
=== RUN   TestConfiguration_nextConfiguration_checkConfiguration
--- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_encodeDecodePeers
--- PASS: TestConfiguration_encodeDecodePeers (0.00s)
=== RUN   TestConfiguration_encodeDecodeConfiguration
--- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
=== RUN   TestDiscardSnapshotStoreImpl
--- PASS: TestDiscardSnapshotStoreImpl (0.00s)
=== RUN   TestDiscardSnapshotSinkImpl
--- PASS: TestDiscardSnapshotSinkImpl (0.00s)
=== RUN   TestFileSnapshotStoreImpl
--- PASS: TestFileSnapshotStoreImpl (0.00s)
=== RUN   TestFileSnapshotSinkImpl
--- PASS: TestFileSnapshotSinkImpl (0.00s)
=== RUN   TestFileSS_CreateSnapshotMissingParentDir
2026-04-23T00:14:42.963Z [INFO]  creating new snapshot: path=/tmp/raft1016634406/raft2363040994/snapshots/3-10-1776903282963.tmp
--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.01s)
=== RUN   TestFileSS_CreateSnapshot
2026-04-23T00:14:42.969Z [INFO]  creating new snapshot: path=/tmp/raft3857052845/snapshots/3-10-1776903282969.tmp
2026-04-23T00:14:42.971Z [WARN]  found temporary snapshot: name=3-10-1776903282969.tmp
--- PASS: TestFileSS_CreateSnapshot (0.01s)
=== RUN   TestFileSS_CancelSnapshot
2026-04-23T00:14:42.976Z [INFO]  creating new snapshot: path=/tmp/raft3267141696/snapshots/3-10-1776903282976.tmp
--- PASS: TestFileSS_CancelSnapshot (0.00s)
=== RUN   TestFileSS_Retention
2026-04-23T00:14:42.980Z [INFO]  creating new snapshot: path=/tmp/raft594419916/snapshots/3-10-1776903282980.tmp
2026-04-23T00:14:42.991Z [INFO]  creating new snapshot: path=/tmp/raft594419916/snapshots/3-11-1776903282991.tmp
2026-04-23T00:14:43.002Z [INFO]  creating new snapshot: path=/tmp/raft594419916/snapshots/3-12-1776903283002.tmp
2026-04-23T00:14:43.018Z [INFO]  reaping snapshot: path=/tmp/raft594419916/snapshots/3-10-1776903282980
2026-04-23T00:14:43.018Z [INFO]  creating new snapshot: path=/tmp/raft594419916/snapshots/3-13-1776903283018.tmp
2026-04-23T00:14:43.025Z [INFO]  reaping snapshot: path=/tmp/raft594419916/snapshots/3-11-1776903282991
2026-04-23T00:14:43.025Z [INFO]  creating new snapshot: path=/tmp/raft594419916/snapshots/3-14-1776903283025.tmp
2026-04-23T00:14:43.032Z [INFO]  reaping snapshot: path=/tmp/raft594419916/snapshots/3-12-1776903283002
--- PASS: TestFileSS_Retention (0.05s)
=== RUN   TestFileSS_BadPerm
--- PASS: TestFileSS_BadPerm (0.00s)
=== RUN   TestFileSS_MissingParentDir
--- PASS: TestFileSS_MissingParentDir (0.00s)
=== RUN   TestFileSS_Ordering
2026-04-23T00:14:43.033Z [INFO]  creating new snapshot: path=/tmp/raft1573997393/snapshots/5-130350-1776903283033.tmp
2026-04-23T00:14:43.041Z [INFO]  creating new snapshot: path=/tmp/raft1573997393/snapshots/36-204917-1776903283041.tmp
--- PASS: TestFileSS_Ordering (0.01s)
=== RUN   TestDeferFutureSuccess
--- PASS: TestDeferFutureSuccess (0.00s)
=== RUN   TestDeferFutureError
--- PASS: TestDeferFutureError (0.00s)
=== RUN   TestDeferFutureConcurrent
--- PASS: TestDeferFutureConcurrent (0.00s)
=== RUN   TestInmemSnapshotStoreImpl
--- PASS: TestInmemSnapshotStoreImpl (0.00s)
=== RUN   TestInmemSnapshotSinkImpl
--- PASS: TestInmemSnapshotSinkImpl (0.00s)
=== RUN   TestInmemSS_CreateSnapshot
--- PASS: TestInmemSS_CreateSnapshot (0.00s)
=== RUN   TestInmemSS_OpenSnapshotTwice
--- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
=== RUN   TestInmemTransportImpl
--- PASS: TestInmemTransportImpl (0.00s)
=== RUN   TestInmemTransportWriteTimeout
--- PASS: TestInmemTransportWriteTimeout (0.01s)
=== RUN   TestRaft_Integ
--- SKIP: TestRaft_Integ (0.00s)
=== RUN   TestRaft_RestartFollower_LongInitialHeartbeat
--- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s)
=== RUN   TestLogCache
--- PASS: TestLogCache (0.00s)
=== RUN   TestLogCacheWithBackendStoreError
--- PASS: TestLogCacheWithBackendStoreError (0.00s)
=== RUN   TestOldestLog
=== RUN   TestOldestLog/empty_logs
=== RUN   TestOldestLog/simple_case
--- PASS: TestOldestLog (0.00s)
    --- PASS: TestOldestLog/empty_logs (0.00s)
    --- PASS: TestOldestLog/simple_case (0.00s)
=== RUN   TestEmitsLogStoreMetrics
--- PASS: TestEmitsLogStoreMetrics (0.01s)
=== RUN   TestNetworkTransport_CloseStreams
--- PASS: TestNetworkTransport_CloseStreams (0.00s)
=== RUN   TestNetworkTransport_StartStop
--- PASS: TestNetworkTransport_StartStop (0.00s)
=== RUN   TestNetworkTransport_Heartbeat_FastPath
--- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s)
=== RUN   TestNetworkTransport_AppendEntries
--- PASS: TestNetworkTransport_AppendEntries (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline
--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_CloseStreams
--- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.02s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0
2026-04-23T00:14:43.146Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2
2026-04-23T00:14:43.198Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10
2026-04-23T00:14:43.250Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s)
=== RUN   TestNetworkTransport_RequestVote
--- PASS: TestNetworkTransport_RequestVote (0.00s)
=== RUN   TestNetworkTransport_InstallSnapshot
--- PASS: TestNetworkTransport_InstallSnapshot (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode
--- PASS: TestNetworkTransport_EncodeDecode (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode_AddressProvider
--- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
=== RUN   TestNetworkTransport_PooledConn
--- PASS: TestNetworkTransport_PooledConn (0.00s)
=== RUN   TestNetworkTransport_ListenBackoff
    net_transport_test.go:839: countingWriter: 2026-04-23T00:14:43.258Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-04-23T00:14:43.263Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-04-23T00:14:43.274Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-04-23T00:14:43.295Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-04-23T00:14:43.335Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-04-23T00:14:43.416Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-04-23T00:14:43.576Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-04-23T00:14:43.897Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-04-23T00:14:44.538Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-04-23T00:14:45.540Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-04-23T00:14:46.542Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
--- PASS: TestNetworkTransport_ListenBackoff (4.00s)
=== RUN   TestPeersJSON_BadConfiguration
--- PASS: TestPeersJSON_BadConfiguration (0.03s)
=== RUN   TestPeersJSON_ReadPeersJSON
--- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
=== RUN   TestPeersJSON_ReadConfigJSON
--- PASS: TestPeersJSON_ReadConfigJSON (0.00s)
=== RUN   TestRaft_StartStop
2026-04-23T00:14:47.288Z [INFO]  server-fa19d8c5-7b04-902d-d416-c2bbd4df6ce5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa19d8c5-7b04-902d-d416-c2bbd4df6ce5 Address:fa19d8c5-7b04-902d-d416-c2bbd4df6ce5}]"
--- PASS: TestRaft_StartStop (0.00s)
=== RUN   TestRaft_AfterShutdown
2026-04-23T00:14:47.288Z [INFO]  server-4370fcd5-179b-87c2-c8be-f88a3f8abec4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4370fcd5-179b-87c2-c8be-f88a3f8abec4 Address:4370fcd5-179b-87c2-c8be-f88a3f8abec4}]"
--- PASS: TestRaft_AfterShutdown (0.00s)
=== RUN   TestRaft_LiveBootstrap
2026-04-23T00:14:47.289Z [INFO]  server-305c840c-b1b1-ab56-3b53-275096f72731: initial configuration: index=0 servers=[]
2026-04-23T00:14:47.289Z [INFO]  server-d4077b50-ca40-27e1-580f-b6d1a9d64736: initial configuration: index=0 servers=[]
2026-04-23T00:14:47.289Z [INFO]  server-82c31f6a-9817-a13c-ce1b-8f6e06632e9d: initial configuration: index=0 servers=[]
2026-04-23T00:14:47.289Z [INFO]  server-305c840c-b1b1-ab56-3b53-275096f72731: entering follower state: follower="Node at 305c840c-b1b1-ab56-3b53-275096f72731 [Follower]" leader-address= leader-id=
2026-04-23T00:14:47.289Z [INFO]  server-d4077b50-ca40-27e1-580f-b6d1a9d64736: entering follower state: follower="Node at d4077b50-ca40-27e1-580f-b6d1a9d64736 [Follower]" leader-address= leader-id=
2026-04-23T00:14:47.289Z [INFO]  server-82c31f6a-9817-a13c-ce1b-8f6e06632e9d: entering follower state: follower="Node at 82c31f6a-9817-a13c-ce1b-8f6e06632e9d [Follower]" leader-address= leader-id=
2026-04-23T00:14:47.289Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:14:47.378Z [WARN]  server-82c31f6a-9817-a13c-ce1b-8f6e06632e9d: no known peers, aborting election
2026-04-23T00:14:47.379Z [WARN]  server-d4077b50-ca40-27e1-580f-b6d1a9d64736: no known peers, aborting election
2026-04-23T00:14:47.382Z [WARN]  server-305c840c-b1b1-ab56-3b53-275096f72731: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:47.382Z [INFO]  server-305c840c-b1b1-ab56-3b53-275096f72731: entering candidate state: node="Node at 305c840c-b1b1-ab56-3b53-275096f72731 [Candidate]" term=2
2026-04-23T00:14:47.382Z [INFO]  server-305c840c-b1b1-ab56-3b53-275096f72731: election won: term=2 tally=2
2026-04-23T00:14:47.382Z [INFO]  server-305c840c-b1b1-ab56-3b53-275096f72731: entering leader state: leader="Node at 305c840c-b1b1-ab56-3b53-275096f72731 [Leader]"
2026-04-23T00:14:47.382Z [INFO]  server-305c840c-b1b1-ab56-3b53-275096f72731: added peer, starting replication: peer=server-d4077b50-ca40-27e1-580f-b6d1a9d64736
2026-04-23T00:14:47.382Z [INFO]  server-305c840c-b1b1-ab56-3b53-275096f72731: added peer, starting replication: peer=server-82c31f6a-9817-a13c-ce1b-8f6e06632e9d
2026-04-23T00:14:47.382Z [WARN]  server-82c31f6a-9817-a13c-ce1b-8f6e06632e9d: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2026-04-23T00:14:47.382Z [WARN]  server-d4077b50-ca40-27e1-580f-b6d1a9d64736: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2026-04-23T00:14:47.382Z [WARN]  server-305c840c-b1b1-ab56-3b53-275096f72731: appendEntries rejected, sending older logs: peer="{Voter server-82c31f6a-9817-a13c-ce1b-8f6e06632e9d 82c31f6a-9817-a13c-ce1b-8f6e06632e9d}" next=1
2026-04-23T00:14:47.382Z [WARN]  server-305c840c-b1b1-ab56-3b53-275096f72731: appendEntries rejected, sending older logs: peer="{Voter server-d4077b50-ca40-27e1-580f-b6d1a9d64736 d4077b50-ca40-27e1-580f-b6d1a9d64736}" next=1
2026-04-23T00:14:47.382Z [INFO]  server-305c840c-b1b1-ab56-3b53-275096f72731: pipelining replication: peer="{Voter server-82c31f6a-9817-a13c-ce1b-8f6e06632e9d 82c31f6a-9817-a13c-ce1b-8f6e06632e9d}"
2026-04-23T00:14:47.382Z [INFO]  server-305c840c-b1b1-ab56-3b53-275096f72731: pipelining replication: peer="{Voter server-d4077b50-ca40-27e1-580f-b6d1a9d64736 d4077b50-ca40-27e1-580f-b6d1a9d64736}"
2026-04-23T00:14:47.488Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:14:47.382402129 +0000 UTC m=+4.422003481 (2 nodes), 92.969403ms from start of poll, 93.30735ms from cluster start. Timeout at 2026-04-23 00:14:47.487406768 +0000 UTC m=+4.527008142, 105.004661ms after stability
2026-04-23T00:14:47.488Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:47.593Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:47.488089759 +0000 UTC m=+4.527691117 (1 nodes), 1.171µs from start of poll, 198.994986ms from cluster start. Timeout at 2026-04-23 00:14:47.593092049 +0000 UTC m=+4.632693406, 105.002289ms after stability
2026-04-23T00:14:47.604Z [INFO]  server-305c840c-b1b1-ab56-3b53-275096f72731: aborting pipeline replication: peer="{Voter server-82c31f6a-9817-a13c-ce1b-8f6e06632e9d 82c31f6a-9817-a13c-ce1b-8f6e06632e9d}"
2026-04-23T00:14:47.604Z [INFO]  server-305c840c-b1b1-ab56-3b53-275096f72731: aborting pipeline replication: peer="{Voter server-d4077b50-ca40-27e1-580f-b6d1a9d64736 d4077b50-ca40-27e1-580f-b6d1a9d64736}"
--- PASS: TestRaft_LiveBootstrap (0.32s)
=== RUN   TestRaft_LiveBootstrap_From_NonVoter
2026-04-23T00:14:47.611Z [INFO]  server-e0838e10-43ee-5af3-c015-103cd3c3a4e0: initial configuration: index=0 servers=[]
2026-04-23T00:14:47.611Z [INFO]  server-ad136ade-bdb0-154e-0cdd-a34a427438e2: initial configuration: index=0 servers=[]
2026-04-23T00:14:47.611Z [INFO]  server-e0838e10-43ee-5af3-c015-103cd3c3a4e0: entering follower state: follower="Node at e0838e10-43ee-5af3-c015-103cd3c3a4e0 [Follower]" leader-address= leader-id=
2026-04-23T00:14:47.611Z [INFO]  server-ad136ade-bdb0-154e-0cdd-a34a427438e2: entering follower state: follower="Node at ad136ade-bdb0-154e-0cdd-a34a427438e2 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s)
=== RUN   TestRaft_RecoverCluster_NoState
2026-04-23T00:14:47.612Z [INFO]  server-a28828a0-c7da-0880-42e3-f1466a9ddc40: initial configuration: index=0 servers=[]
2026-04-23T00:14:47.612Z [INFO]  server-a28828a0-c7da-0880-42e3-f1466a9ddc40: entering follower state: follower="Node at a28828a0-c7da-0880-42e3-f1466a9ddc40 [Shutdown]" leader-address= leader-id=
--- PASS: TestRaft_RecoverCluster_NoState (0.00s)
=== RUN   TestRaft_RecoverCluster
=== RUN   TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs
2026-04-23T00:14:47.613Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecd69b62-9771-72be-9498-26e934fe1a79 Address:ecd69b62-9771-72be-9498-26e934fe1a79} {Suffrage:Voter ID:server-9bd52eea-8552-e58b-d9f9-5e86174914bb Address:9bd52eea-8552-e58b-d9f9-5e86174914bb} {Suffrage:Voter ID:server-65b6edac-cafb-cda0-8f6f-21a4538fd088 Address:65b6edac-cafb-cda0-8f6f-21a4538fd088}]"
2026-04-23T00:14:47.613Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: entering follower state: follower="Node at ecd69b62-9771-72be-9498-26e934fe1a79 [Follower]" leader-address= leader-id=
2026-04-23T00:14:47.613Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecd69b62-9771-72be-9498-26e934fe1a79 Address:ecd69b62-9771-72be-9498-26e934fe1a79} {Suffrage:Voter ID:server-9bd52eea-8552-e58b-d9f9-5e86174914bb Address:9bd52eea-8552-e58b-d9f9-5e86174914bb} {Suffrage:Voter ID:server-65b6edac-cafb-cda0-8f6f-21a4538fd088 Address:65b6edac-cafb-cda0-8f6f-21a4538fd088}]"
2026-04-23T00:14:47.613Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: entering follower state: follower="Node at 9bd52eea-8552-e58b-d9f9-5e86174914bb [Follower]" leader-address= leader-id=
2026-04-23T00:14:47.613Z [INFO]  server-65b6edac-cafb-cda0-8f6f-21a4538fd088: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecd69b62-9771-72be-9498-26e934fe1a79 Address:ecd69b62-9771-72be-9498-26e934fe1a79} {Suffrage:Voter ID:server-9bd52eea-8552-e58b-d9f9-5e86174914bb Address:9bd52eea-8552-e58b-d9f9-5e86174914bb} {Suffrage:Voter ID:server-65b6edac-cafb-cda0-8f6f-21a4538fd088 Address:65b6edac-cafb-cda0-8f6f-21a4538fd088}]"
2026-04-23T00:14:47.613Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:47.613Z [INFO]  server-65b6edac-cafb-cda0-8f6f-21a4538fd088: entering follower state: follower="Node at 65b6edac-cafb-cda0-8f6f-21a4538fd088 [Follower]" leader-address= leader-id=
2026-04-23T00:14:47.666Z [WARN]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:47.666Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: entering candidate state: node="Node at 9bd52eea-8552-e58b-d9f9-5e86174914bb [Candidate]" term=2
2026-04-23T00:14:47.666Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: election won: term=2 tally=2
2026-04-23T00:14:47.666Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: entering leader state: leader="Node at 9bd52eea-8552-e58b-d9f9-5e86174914bb [Leader]"
2026-04-23T00:14:47.666Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: added peer, starting replication: peer=server-ecd69b62-9771-72be-9498-26e934fe1a79
2026-04-23T00:14:47.666Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: added peer, starting replication: peer=server-65b6edac-cafb-cda0-8f6f-21a4538fd088
2026-04-23T00:14:47.666Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: pipelining replication: peer="{Voter server-65b6edac-cafb-cda0-8f6f-21a4538fd088 65b6edac-cafb-cda0-8f6f-21a4538fd088}"
2026-04-23T00:14:47.666Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: pipelining replication: peer="{Voter server-ecd69b62-9771-72be-9498-26e934fe1a79 ecd69b62-9771-72be-9498-26e934fe1a79}"
2026-04-23T00:14:47.771Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:47.666295372 +0000 UTC m=+4.705896795 (1 nodes), 52.568996ms from start of poll, 52.930054ms from cluster start. Timeout at 2026-04-23 00:14:47.771296873 +0000 UTC m=+4.810898262, 105.001467ms after stability
2026-04-23T00:14:47.771Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: aborting pipeline replication: peer="{Voter server-65b6edac-cafb-cda0-8f6f-21a4538fd088 65b6edac-cafb-cda0-8f6f-21a4538fd088}"
2026-04-23T00:14:47.771Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: aborting pipeline replication: peer="{Voter server-ecd69b62-9771-72be-9498-26e934fe1a79 ecd69b62-9771-72be-9498-26e934fe1a79}"
2026-04-23T00:14:47.794Z [INFO]  creating new snapshot: path=/tmp/raft4020182433/snapshots/2-2-1776903287794.tmp
2026-04-23T00:14:47.795Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: starting restore from snapshot: id=2-2-1776903287794 last-index=2 last-term=2 size-in-bytes=1
2026-04-23T00:14:47.795Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: snapshot restore progress: id=2-2-1776903287794 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2026-04-23T00:14:47.795Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: restored from snapshot: id=2-2-1776903287794 last-index=2 last-term=2 size-in-bytes=1
2026-04-23T00:14:47.795Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecd69b62-9771-72be-9498-26e934fe1a79 Address:ecd69b62-9771-72be-9498-26e934fe1a79} {Suffrage:Voter ID:server-9bd52eea-8552-e58b-d9f9-5e86174914bb Address:9bd52eea-8552-e58b-d9f9-5e86174914bb} {Suffrage:Voter ID:server-65b6edac-cafb-cda0-8f6f-21a4538fd088 Address:65b6edac-cafb-cda0-8f6f-21a4538fd088}]"
2026-04-23T00:14:47.795Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: entering follower state: follower="Node at ecd69b62-9771-72be-9498-26e934fe1a79 [Follower]" leader-address= leader-id=
2026-04-23T00:14:47.795Z [INFO]  creating new snapshot: path=/tmp/raft1978049901/snapshots/2-2-1776903287795.tmp
2026-04-23T00:14:47.863Z [WARN]  server-ecd69b62-9771-72be-9498-26e934fe1a79: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:47.863Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: entering candidate state: node="Node at ecd69b62-9771-72be-9498-26e934fe1a79 [Candidate]" term=3
2026-04-23T00:14:47.863Z [ERROR] server-ecd69b62-9771-72be-9498-26e934fe1a79: failed to make requestVote RPC: target="{Voter server-9bd52eea-8552-e58b-d9f9-5e86174914bb 9bd52eea-8552-e58b-d9f9-5e86174914bb}" error="failed to connect to peer: 9bd52eea-8552-e58b-d9f9-5e86174914bb" term=3
2026-04-23T00:14:47.863Z [ERROR] server-ecd69b62-9771-72be-9498-26e934fe1a79: failed to make requestVote RPC: target="{Voter server-65b6edac-cafb-cda0-8f6f-21a4538fd088 65b6edac-cafb-cda0-8f6f-21a4538fd088}" error="failed to connect to peer: 65b6edac-cafb-cda0-8f6f-21a4538fd088" term=3
2026-04-23T00:14:47.903Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: starting restore from snapshot: id=2-2-1776903287795 last-index=2 last-term=2 size-in-bytes=1
2026-04-23T00:14:47.903Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: snapshot restore progress: id=2-2-1776903287795 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2026-04-23T00:14:47.903Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: restored from snapshot: id=2-2-1776903287795 last-index=2 last-term=2 size-in-bytes=1
2026-04-23T00:14:47.903Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecd69b62-9771-72be-9498-26e934fe1a79 Address:ecd69b62-9771-72be-9498-26e934fe1a79} {Suffrage:Voter ID:server-9bd52eea-8552-e58b-d9f9-5e86174914bb Address:9bd52eea-8552-e58b-d9f9-5e86174914bb} {Suffrage:Voter ID:server-65b6edac-cafb-cda0-8f6f-21a4538fd088 Address:65b6edac-cafb-cda0-8f6f-21a4538fd088}]"
2026-04-23T00:14:47.903Z [INFO]  creating new snapshot: path=/tmp/raft4206728895/snapshots/2-2-1776903287903.tmp
2026-04-23T00:14:47.903Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: entering follower state: follower="Node at 9bd52eea-8552-e58b-d9f9-5e86174914bb [Follower]" leader-address= leader-id=
2026-04-23T00:14:47.955Z [WARN]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:47.955Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: entering candidate state: node="Node at 9bd52eea-8552-e58b-d9f9-5e86174914bb [Candidate]" term=3
2026-04-23T00:14:47.955Z [ERROR] server-9bd52eea-8552-e58b-d9f9-5e86174914bb: failed to make requestVote RPC: target="{Voter server-65b6edac-cafb-cda0-8f6f-21a4538fd088 65b6edac-cafb-cda0-8f6f-21a4538fd088}" error="failed to connect to peer: 65b6edac-cafb-cda0-8f6f-21a4538fd088" term=3
2026-04-23T00:14:47.955Z [ERROR] server-9bd52eea-8552-e58b-d9f9-5e86174914bb: failed to make requestVote RPC: target="{Voter server-ecd69b62-9771-72be-9498-26e934fe1a79 ecd69b62-9771-72be-9498-26e934fe1a79}" error="failed to connect to peer: ecd69b62-9771-72be-9498-26e934fe1a79" term=3
2026-04-23T00:14:47.962Z [WARN]  server-ecd69b62-9771-72be-9498-26e934fe1a79: Election timeout reached, restarting election
2026-04-23T00:14:47.962Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: entering candidate state: node="Node at ecd69b62-9771-72be-9498-26e934fe1a79 [Candidate]" term=4
2026-04-23T00:14:47.962Z [ERROR] server-ecd69b62-9771-72be-9498-26e934fe1a79: failed to make requestVote RPC: target="{Voter server-65b6edac-cafb-cda0-8f6f-21a4538fd088 65b6edac-cafb-cda0-8f6f-21a4538fd088}" error="failed to connect to peer: 65b6edac-cafb-cda0-8f6f-21a4538fd088" term=4
2026-04-23T00:14:47.962Z [ERROR] server-ecd69b62-9771-72be-9498-26e934fe1a79: failed to make requestVote RPC: target="{Voter server-9bd52eea-8552-e58b-d9f9-5e86174914bb 9bd52eea-8552-e58b-d9f9-5e86174914bb}" error="failed to connect to peer: 9bd52eea-8552-e58b-d9f9-5e86174914bb" term=4
2026-04-23T00:14:48.030Z [WARN]  server-ecd69b62-9771-72be-9498-26e934fe1a79: Election timeout reached, restarting election
2026-04-23T00:14:48.030Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: entering candidate state: node="Node at ecd69b62-9771-72be-9498-26e934fe1a79 [Candidate]" term=5
2026-04-23T00:14:48.030Z [ERROR] server-ecd69b62-9771-72be-9498-26e934fe1a79: failed to make requestVote RPC: target="{Voter server-65b6edac-cafb-cda0-8f6f-21a4538fd088 65b6edac-cafb-cda0-8f6f-21a4538fd088}" error="failed to connect to peer: 65b6edac-cafb-cda0-8f6f-21a4538fd088" term=5
2026-04-23T00:14:48.030Z [ERROR] server-ecd69b62-9771-72be-9498-26e934fe1a79: failed to make requestVote RPC: target="{Voter server-9bd52eea-8552-e58b-d9f9-5e86174914bb 9bd52eea-8552-e58b-d9f9-5e86174914bb}" error="failed to connect to peer: 9bd52eea-8552-e58b-d9f9-5e86174914bb" term=5
2026-04-23T00:14:48.036Z [WARN]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: Election timeout reached, restarting election
2026-04-23T00:14:48.036Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: entering candidate state: node="Node at 9bd52eea-8552-e58b-d9f9-5e86174914bb [Candidate]" term=4
2026-04-23T00:14:48.036Z [ERROR] server-9bd52eea-8552-e58b-d9f9-5e86174914bb: failed to make requestVote RPC: target="{Voter server-65b6edac-cafb-cda0-8f6f-21a4538fd088 65b6edac-cafb-cda0-8f6f-21a4538fd088}" error="failed to connect to peer: 65b6edac-cafb-cda0-8f6f-21a4538fd088" term=4
2026-04-23T00:14:48.036Z [ERROR] server-9bd52eea-8552-e58b-d9f9-5e86174914bb: failed to make requestVote RPC: target="{Voter server-ecd69b62-9771-72be-9498-26e934fe1a79 ecd69b62-9771-72be-9498-26e934fe1a79}" error="failed to connect to peer: ecd69b62-9771-72be-9498-26e934fe1a79" term=4
2026-04-23T00:14:48.082Z [WARN]  server-ecd69b62-9771-72be-9498-26e934fe1a79: Election timeout reached, restarting election
2026-04-23T00:14:48.082Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: entering candidate state: node="Node at ecd69b62-9771-72be-9498-26e934fe1a79 [Candidate]" term=6
2026-04-23T00:14:48.082Z [ERROR] server-ecd69b62-9771-72be-9498-26e934fe1a79: failed to make requestVote RPC: target="{Voter server-65b6edac-cafb-cda0-8f6f-21a4538fd088 65b6edac-cafb-cda0-8f6f-21a4538fd088}" error="failed to connect to peer: 65b6edac-cafb-cda0-8f6f-21a4538fd088" term=6
2026-04-23T00:14:48.082Z [ERROR] server-ecd69b62-9771-72be-9498-26e934fe1a79: failed to make requestVote RPC: target="{Voter server-9bd52eea-8552-e58b-d9f9-5e86174914bb 9bd52eea-8552-e58b-d9f9-5e86174914bb}" error="failed to connect to peer: 9bd52eea-8552-e58b-d9f9-5e86174914bb" term=6
2026-04-23T00:14:48.131Z [WARN]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: Election timeout reached, restarting election
2026-04-23T00:14:48.131Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: entering candidate state: node="Node at 9bd52eea-8552-e58b-d9f9-5e86174914bb [Candidate]" term=5
2026-04-23T00:14:48.131Z [ERROR] server-9bd52eea-8552-e58b-d9f9-5e86174914bb: failed to make requestVote RPC: target="{Voter server-65b6edac-cafb-cda0-8f6f-21a4538fd088 65b6edac-cafb-cda0-8f6f-21a4538fd088}" error="failed to connect to peer: 65b6edac-cafb-cda0-8f6f-21a4538fd088" term=5
2026-04-23T00:14:48.131Z [ERROR] server-9bd52eea-8552-e58b-d9f9-5e86174914bb: failed to make requestVote RPC: target="{Voter server-ecd69b62-9771-72be-9498-26e934fe1a79 ecd69b62-9771-72be-9498-26e934fe1a79}" error="failed to connect to peer: ecd69b62-9771-72be-9498-26e934fe1a79" term=5
2026-04-23T00:14:48.133Z [INFO]  server-65b6edac-cafb-cda0-8f6f-21a4538fd088: starting restore from snapshot: id=2-2-1776903287903 last-index=2 last-term=2 size-in-bytes=1
2026-04-23T00:14:48.133Z [INFO]  server-65b6edac-cafb-cda0-8f6f-21a4538fd088: snapshot restore progress: id=2-2-1776903287903 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2026-04-23T00:14:48.133Z [INFO]  server-65b6edac-cafb-cda0-8f6f-21a4538fd088: restored from snapshot: id=2-2-1776903287903 last-index=2 last-term=2 size-in-bytes=1
2026-04-23T00:14:48.133Z [INFO]  server-65b6edac-cafb-cda0-8f6f-21a4538fd088: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ecd69b62-9771-72be-9498-26e934fe1a79 Address:ecd69b62-9771-72be-9498-26e934fe1a79} {Suffrage:Voter ID:server-9bd52eea-8552-e58b-d9f9-5e86174914bb Address:9bd52eea-8552-e58b-d9f9-5e86174914bb} {Suffrage:Voter ID:server-65b6edac-cafb-cda0-8f6f-21a4538fd088 Address:65b6edac-cafb-cda0-8f6f-21a4538fd088}]"
2026-04-23T00:14:48.134Z [INFO]  server-65b6edac-cafb-cda0-8f6f-21a4538fd088: entering follower state: follower="Node at 65b6edac-cafb-cda0-8f6f-21a4538fd088 [Follower]" leader-address= leader-id=
2026-04-23T00:14:48.141Z [WARN]  server-ecd69b62-9771-72be-9498-26e934fe1a79: Election timeout reached, restarting election
2026-04-23T00:14:48.141Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: entering candidate state: node="Node at ecd69b62-9771-72be-9498-26e934fe1a79 [Candidate]" term=7
2026-04-23T00:14:48.141Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: election won: term=7 tally=2
2026-04-23T00:14:48.141Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: entering leader state: leader="Node at ecd69b62-9771-72be-9498-26e934fe1a79 [Leader]"
2026-04-23T00:14:48.141Z [INFO]  server-9bd52eea-8552-e58b-d9f9-5e86174914bb: entering follower state: follower="Node at 9bd52eea-8552-e58b-d9f9-5e86174914bb [Follower]" leader-address= leader-id=
2026-04-23T00:14:48.141Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: added peer, starting replication: peer=server-9bd52eea-8552-e58b-d9f9-5e86174914bb
2026-04-23T00:14:48.141Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: added peer, starting replication: peer=server-65b6edac-cafb-cda0-8f6f-21a4538fd088
2026-04-23T00:14:48.141Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: pipelining replication: peer="{Voter server-9bd52eea-8552-e58b-d9f9-5e86174914bb 9bd52eea-8552-e58b-d9f9-5e86174914bb}"
2026-04-23T00:14:48.141Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: pipelining replication: peer="{Voter server-65b6edac-cafb-cda0-8f6f-21a4538fd088 65b6edac-cafb-cda0-8f6f-21a4538fd088}"
2026-04-23T00:14:48.449Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:48.554Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:48.449100946 +0000 UTC m=+5.488702302 (1 nodes), 27.669µs from start of poll, 835.735561ms from cluster start. Timeout at 2026-04-23 00:14:48.554107791 +0000 UTC m=+5.593709146, 105.006844ms after stability
2026-04-23T00:14:48.554Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: aborting pipeline replication: peer="{Voter server-65b6edac-cafb-cda0-8f6f-21a4538fd088 65b6edac-cafb-cda0-8f6f-21a4538fd088}"
2026-04-23T00:14:48.554Z [INFO]  server-ecd69b62-9771-72be-9498-26e934fe1a79: aborting pipeline replication: peer="{Voter server-9bd52eea-8552-e58b-d9f9-5e86174914bb 9bd52eea-8552-e58b-d9f9-5e86174914bb}"
=== RUN   TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs
2026-04-23T00:14:48.628Z [INFO]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c3f3527-0d85-78fc-5054-25d0e946d59a Address:2c3f3527-0d85-78fc-5054-25d0e946d59a} {Suffrage:Voter ID:server-9388bffb-20ad-7dfc-52db-41589fcaf562 Address:9388bffb-20ad-7dfc-52db-41589fcaf562} {Suffrage:Voter ID:server-38d5edb8-5cb0-176f-bca2-3af8c8a51541 Address:38d5edb8-5cb0-176f-bca2-3af8c8a51541}]"
2026-04-23T00:14:48.628Z [INFO]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: entering follower state: follower="Node at 2c3f3527-0d85-78fc-5054-25d0e946d59a [Follower]" leader-address= leader-id=
2026-04-23T00:14:48.628Z [INFO]  server-9388bffb-20ad-7dfc-52db-41589fcaf562: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c3f3527-0d85-78fc-5054-25d0e946d59a Address:2c3f3527-0d85-78fc-5054-25d0e946d59a} {Suffrage:Voter ID:server-9388bffb-20ad-7dfc-52db-41589fcaf562 Address:9388bffb-20ad-7dfc-52db-41589fcaf562} {Suffrage:Voter ID:server-38d5edb8-5cb0-176f-bca2-3af8c8a51541 Address:38d5edb8-5cb0-176f-bca2-3af8c8a51541}]"
2026-04-23T00:14:48.628Z [INFO]  server-9388bffb-20ad-7dfc-52db-41589fcaf562: entering follower state: follower="Node at 9388bffb-20ad-7dfc-52db-41589fcaf562 [Follower]" leader-address= leader-id=
2026-04-23T00:14:48.628Z [INFO]  server-38d5edb8-5cb0-176f-bca2-3af8c8a51541: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c3f3527-0d85-78fc-5054-25d0e946d59a Address:2c3f3527-0d85-78fc-5054-25d0e946d59a} {Suffrage:Voter ID:server-9388bffb-20ad-7dfc-52db-41589fcaf562 Address:9388bffb-20ad-7dfc-52db-41589fcaf562} {Suffrage:Voter ID:server-38d5edb8-5cb0-176f-bca2-3af8c8a51541 Address:38d5edb8-5cb0-176f-bca2-3af8c8a51541}]"
2026-04-23T00:14:48.628Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:48.628Z [INFO]  server-38d5edb8-5cb0-176f-bca2-3af8c8a51541: entering follower state: follower="Node at 38d5edb8-5cb0-176f-bca2-3af8c8a51541 [Follower]" leader-address= leader-id=
2026-04-23T00:14:48.709Z [WARN]  server-38d5edb8-5cb0-176f-bca2-3af8c8a51541: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:48.709Z [INFO]  server-38d5edb8-5cb0-176f-bca2-3af8c8a51541: entering candidate state: node="Node at 38d5edb8-5cb0-176f-bca2-3af8c8a51541 [Candidate]" term=2
2026-04-23T00:14:48.709Z [INFO]  server-38d5edb8-5cb0-176f-bca2-3af8c8a51541: election won: term=2 tally=2
2026-04-23T00:14:48.709Z [INFO]  server-38d5edb8-5cb0-176f-bca2-3af8c8a51541: entering leader state: leader="Node at 38d5edb8-5cb0-176f-bca2-3af8c8a51541 [Leader]"
2026-04-23T00:14:48.709Z [INFO]  server-38d5edb8-5cb0-176f-bca2-3af8c8a51541: added peer, starting replication: peer=server-2c3f3527-0d85-78fc-5054-25d0e946d59a
2026-04-23T00:14:48.709Z [INFO]  server-38d5edb8-5cb0-176f-bca2-3af8c8a51541: added peer, starting replication: peer=server-9388bffb-20ad-7dfc-52db-41589fcaf562
2026-04-23T00:14:48.709Z [INFO]  server-38d5edb8-5cb0-176f-bca2-3af8c8a51541: pipelining replication: peer="{Voter server-9388bffb-20ad-7dfc-52db-41589fcaf562 9388bffb-20ad-7dfc-52db-41589fcaf562}"
2026-04-23T00:14:48.709Z [INFO]  server-38d5edb8-5cb0-176f-bca2-3af8c8a51541: pipelining replication: peer="{Voter server-2c3f3527-0d85-78fc-5054-25d0e946d59a 2c3f3527-0d85-78fc-5054-25d0e946d59a}"
2026-04-23T00:14:48.814Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:48.709373717 +0000 UTC m=+5.748975086 (1 nodes), 80.760588ms from start of poll, 81.169164ms from cluster start. Timeout at 2026-04-23 00:14:48.814374946 +0000 UTC m=+5.853976335, 105.001249ms after stability
2026-04-23T00:14:48.815Z [INFO]  server-38d5edb8-5cb0-176f-bca2-3af8c8a51541: aborting pipeline replication: peer="{Voter server-2c3f3527-0d85-78fc-5054-25d0e946d59a 2c3f3527-0d85-78fc-5054-25d0e946d59a}"
2026-04-23T00:14:48.815Z [INFO]  server-38d5edb8-5cb0-176f-bca2-3af8c8a51541: aborting pipeline replication: peer="{Voter server-9388bffb-20ad-7dfc-52db-41589fcaf562 9388bffb-20ad-7dfc-52db-41589fcaf562}"
2026-04-23T00:14:48.825Z [INFO]  creating new snapshot: path=/tmp/raft2725668366/snapshots/2-6-1776903288825.tmp
2026-04-23T00:14:48.845Z [INFO]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: starting restore from snapshot: id=2-6-1776903288825 last-index=6 last-term=2 size-in-bytes=25
2026-04-23T00:14:48.845Z [INFO]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: snapshot restore progress: id=2-6-1776903288825 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2026-04-23T00:14:48.845Z [INFO]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: restored from snapshot: id=2-6-1776903288825 last-index=6 last-term=2 size-in-bytes=25
2026-04-23T00:14:48.845Z [INFO]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c3f3527-0d85-78fc-5054-25d0e946d59a Address:2c3f3527-0d85-78fc-5054-25d0e946d59a} {Suffrage:Voter ID:server-9388bffb-20ad-7dfc-52db-41589fcaf562 Address:9388bffb-20ad-7dfc-52db-41589fcaf562} {Suffrage:Voter ID:server-38d5edb8-5cb0-176f-bca2-3af8c8a51541 Address:38d5edb8-5cb0-176f-bca2-3af8c8a51541}]"
2026-04-23T00:14:48.845Z [INFO]  creating new snapshot: path=/tmp/raft1036901846/snapshots/2-6-1776903288845.tmp
2026-04-23T00:14:48.845Z [INFO]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: entering follower state: follower="Node at 2c3f3527-0d85-78fc-5054-25d0e946d59a [Follower]" leader-address= leader-id=
2026-04-23T00:14:48.845Z [INFO]  server-9388bffb-20ad-7dfc-52db-41589fcaf562: starting restore from snapshot: id=2-6-1776903288845 last-index=6 last-term=2 size-in-bytes=25
2026-04-23T00:14:48.846Z [INFO]  server-9388bffb-20ad-7dfc-52db-41589fcaf562: snapshot restore progress: id=2-6-1776903288845 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2026-04-23T00:14:48.846Z [INFO]  server-9388bffb-20ad-7dfc-52db-41589fcaf562: restored from snapshot: id=2-6-1776903288845 last-index=6 last-term=2 size-in-bytes=25
2026-04-23T00:14:48.846Z [INFO]  server-9388bffb-20ad-7dfc-52db-41589fcaf562: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c3f3527-0d85-78fc-5054-25d0e946d59a Address:2c3f3527-0d85-78fc-5054-25d0e946d59a} {Suffrage:Voter ID:server-9388bffb-20ad-7dfc-52db-41589fcaf562 Address:9388bffb-20ad-7dfc-52db-41589fcaf562} {Suffrage:Voter ID:server-38d5edb8-5cb0-176f-bca2-3af8c8a51541 Address:38d5edb8-5cb0-176f-bca2-3af8c8a51541}]"
2026-04-23T00:14:48.846Z [INFO]  creating new snapshot: path=/tmp/raft922889198/snapshots/2-6-1776903288846.tmp
2026-04-23T00:14:48.846Z [INFO]  server-9388bffb-20ad-7dfc-52db-41589fcaf562: entering follower state: follower="Node at 9388bffb-20ad-7dfc-52db-41589fcaf562 [Follower]" leader-address= leader-id=
2026-04-23T00:14:48.913Z [WARN]  server-9388bffb-20ad-7dfc-52db-41589fcaf562: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:48.913Z [INFO]  server-9388bffb-20ad-7dfc-52db-41589fcaf562: entering candidate state: node="Node at 9388bffb-20ad-7dfc-52db-41589fcaf562 [Candidate]" term=3
2026-04-23T00:14:48.914Z [ERROR] server-9388bffb-20ad-7dfc-52db-41589fcaf562: failed to make requestVote RPC: target="{Voter server-38d5edb8-5cb0-176f-bca2-3af8c8a51541 38d5edb8-5cb0-176f-bca2-3af8c8a51541}" error="failed to connect to peer: 38d5edb8-5cb0-176f-bca2-3af8c8a51541" term=3
2026-04-23T00:14:48.914Z [ERROR] server-9388bffb-20ad-7dfc-52db-41589fcaf562: failed to make requestVote RPC: target="{Voter server-2c3f3527-0d85-78fc-5054-25d0e946d59a 2c3f3527-0d85-78fc-5054-25d0e946d59a}" error="failed to connect to peer: 2c3f3527-0d85-78fc-5054-25d0e946d59a" term=3
2026-04-23T00:14:48.934Z [WARN]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:48.934Z [INFO]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: entering candidate state: node="Node at 2c3f3527-0d85-78fc-5054-25d0e946d59a [Candidate]" term=3
2026-04-23T00:14:48.934Z [ERROR] server-2c3f3527-0d85-78fc-5054-25d0e946d59a: failed to make requestVote RPC: target="{Voter server-9388bffb-20ad-7dfc-52db-41589fcaf562 9388bffb-20ad-7dfc-52db-41589fcaf562}" error="failed to connect to peer: 9388bffb-20ad-7dfc-52db-41589fcaf562" term=3
2026-04-23T00:14:48.934Z [ERROR] server-2c3f3527-0d85-78fc-5054-25d0e946d59a: failed to make requestVote RPC: target="{Voter server-38d5edb8-5cb0-176f-bca2-3af8c8a51541 38d5edb8-5cb0-176f-bca2-3af8c8a51541}" error="failed to connect to peer: 38d5edb8-5cb0-176f-bca2-3af8c8a51541" term=3
2026-04-23T00:14:48.976Z [WARN]  server-9388bffb-20ad-7dfc-52db-41589fcaf562: Election timeout reached, restarting election
2026-04-23T00:14:48.976Z [INFO]  server-9388bffb-20ad-7dfc-52db-41589fcaf562: entering candidate state: node="Node at 9388bffb-20ad-7dfc-52db-41589fcaf562 [Candidate]" term=4
2026-04-23T00:14:48.977Z [ERROR] server-9388bffb-20ad-7dfc-52db-41589fcaf562: failed to make requestVote RPC: target="{Voter server-38d5edb8-5cb0-176f-bca2-3af8c8a51541 38d5edb8-5cb0-176f-bca2-3af8c8a51541}" error="failed to connect to peer: 38d5edb8-5cb0-176f-bca2-3af8c8a51541" term=4
2026-04-23T00:14:48.977Z [ERROR] server-9388bffb-20ad-7dfc-52db-41589fcaf562: failed to make requestVote RPC: target="{Voter server-2c3f3527-0d85-78fc-5054-25d0e946d59a 2c3f3527-0d85-78fc-5054-25d0e946d59a}" error="failed to connect to peer: 2c3f3527-0d85-78fc-5054-25d0e946d59a" term=4
2026-04-23T00:14:48.985Z [INFO]  server-38d5edb8-5cb0-176f-bca2-3af8c8a51541: starting restore from snapshot: id=2-6-1776903288846 last-index=6 last-term=2 size-in-bytes=25
2026-04-23T00:14:48.986Z [INFO]  server-38d5edb8-5cb0-176f-bca2-3af8c8a51541: snapshot restore progress: id=2-6-1776903288846 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2026-04-23T00:14:48.986Z [INFO]  server-38d5edb8-5cb0-176f-bca2-3af8c8a51541: restored from snapshot: id=2-6-1776903288846 last-index=6 last-term=2 size-in-bytes=25
2026-04-23T00:14:48.986Z [INFO]  server-38d5edb8-5cb0-176f-bca2-3af8c8a51541: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c3f3527-0d85-78fc-5054-25d0e946d59a Address:2c3f3527-0d85-78fc-5054-25d0e946d59a} {Suffrage:Voter ID:server-9388bffb-20ad-7dfc-52db-41589fcaf562 Address:9388bffb-20ad-7dfc-52db-41589fcaf562} {Suffrage:Voter ID:server-38d5edb8-5cb0-176f-bca2-3af8c8a51541 Address:38d5edb8-5cb0-176f-bca2-3af8c8a51541}]"
2026-04-23T00:14:48.986Z [INFO]  server-38d5edb8-5cb0-176f-bca2-3af8c8a51541: entering follower state: follower="Node at 38d5edb8-5cb0-176f-bca2-3af8c8a51541 [Follower]" leader-address= leader-id=
2026-04-23T00:14:49.001Z [WARN]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: Election timeout reached, restarting election
2026-04-23T00:14:49.001Z [INFO]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: entering candidate state: node="Node at 2c3f3527-0d85-78fc-5054-25d0e946d59a [Candidate]" term=4
2026-04-23T00:14:49.001Z [INFO]  server-9388bffb-20ad-7dfc-52db-41589fcaf562: duplicate requestVote for same term: term=4
2026-04-23T00:14:49.001Z [INFO]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: election won: term=4 tally=2
2026-04-23T00:14:49.001Z [INFO]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: entering leader state: leader="Node at 2c3f3527-0d85-78fc-5054-25d0e946d59a [Leader]"
2026-04-23T00:14:49.001Z [INFO]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: added peer, starting replication: peer=server-9388bffb-20ad-7dfc-52db-41589fcaf562
2026-04-23T00:14:49.001Z [INFO]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: added peer, starting replication: peer=server-38d5edb8-5cb0-176f-bca2-3af8c8a51541
2026-04-23T00:14:49.001Z [INFO]  server-9388bffb-20ad-7dfc-52db-41589fcaf562: entering follower state: follower="Node at 9388bffb-20ad-7dfc-52db-41589fcaf562 [Follower]" leader-address=2c3f3527-0d85-78fc-5054-25d0e946d59a leader-id=server-2c3f3527-0d85-78fc-5054-25d0e946d59a
2026-04-23T00:14:49.001Z [INFO]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: pipelining replication: peer="{Voter server-38d5edb8-5cb0-176f-bca2-3af8c8a51541 38d5edb8-5cb0-176f-bca2-3af8c8a51541}"
2026-04-23T00:14:49.001Z [INFO]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: pipelining replication: peer="{Voter server-9388bffb-20ad-7dfc-52db-41589fcaf562 9388bffb-20ad-7dfc-52db-41589fcaf562}"
2026-04-23T00:14:49.301Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:49.407Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:49.301989359 +0000 UTC m=+6.341590732 (1 nodes), 41.238µs from start of poll, 673.78481ms from cluster start. Timeout at 2026-04-23 00:14:49.406998002 +0000 UTC m=+6.446599391, 105.008659ms after stability
2026-04-23T00:14:49.407Z [INFO]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: aborting pipeline replication: peer="{Voter server-9388bffb-20ad-7dfc-52db-41589fcaf562 9388bffb-20ad-7dfc-52db-41589fcaf562}"
2026-04-23T00:14:49.407Z [INFO]  server-2c3f3527-0d85-78fc-5054-25d0e946d59a: aborting pipeline replication: peer="{Voter server-38d5edb8-5cb0-176f-bca2-3af8c8a51541 38d5edb8-5cb0-176f-bca2-3af8c8a51541}"
=== RUN   TestRaft_RecoverCluster/snapshot,_with_trailing_logs
2026-04-23T00:14:49.409Z [INFO]  server-3cf08bfb-6344-2224-dda8-9d4e5548936b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3cf08bfb-6344-2224-dda8-9d4e5548936b Address:3cf08bfb-6344-2224-dda8-9d4e5548936b} {Suffrage:Voter ID:server-7088f165-9fe0-8065-15fa-415ee044129e Address:7088f165-9fe0-8065-15fa-415ee044129e} {Suffrage:Voter ID:server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe Address:7c05b4bb-d3f9-d028-1149-780b4b5f4cbe}]"
2026-04-23T00:14:49.409Z [INFO]  server-7088f165-9fe0-8065-15fa-415ee044129e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3cf08bfb-6344-2224-dda8-9d4e5548936b Address:3cf08bfb-6344-2224-dda8-9d4e5548936b} {Suffrage:Voter ID:server-7088f165-9fe0-8065-15fa-415ee044129e Address:7088f165-9fe0-8065-15fa-415ee044129e} {Suffrage:Voter ID:server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe Address:7c05b4bb-d3f9-d028-1149-780b4b5f4cbe}]"
2026-04-23T00:14:49.409Z [INFO]  server-3cf08bfb-6344-2224-dda8-9d4e5548936b: entering follower state: follower="Node at 3cf08bfb-6344-2224-dda8-9d4e5548936b [Follower]" leader-address= leader-id=
2026-04-23T00:14:49.409Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3cf08bfb-6344-2224-dda8-9d4e5548936b Address:3cf08bfb-6344-2224-dda8-9d4e5548936b} {Suffrage:Voter ID:server-7088f165-9fe0-8065-15fa-415ee044129e Address:7088f165-9fe0-8065-15fa-415ee044129e} {Suffrage:Voter ID:server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe Address:7c05b4bb-d3f9-d028-1149-780b4b5f4cbe}]"
2026-04-23T00:14:49.409Z [INFO]  server-7088f165-9fe0-8065-15fa-415ee044129e: entering follower state: follower="Node at 7088f165-9fe0-8065-15fa-415ee044129e [Follower]" leader-address= leader-id=
2026-04-23T00:14:49.409Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:49.409Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: entering follower state: follower="Node at 7c05b4bb-d3f9-d028-1149-780b4b5f4cbe [Follower]" leader-address= leader-id=
2026-04-23T00:14:49.467Z [WARN]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:49.467Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: entering candidate state: node="Node at 7c05b4bb-d3f9-d028-1149-780b4b5f4cbe [Candidate]" term=2
2026-04-23T00:14:49.467Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: election won: term=2 tally=2
2026-04-23T00:14:49.467Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: entering leader state: leader="Node at 7c05b4bb-d3f9-d028-1149-780b4b5f4cbe [Leader]"
2026-04-23T00:14:49.467Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: added peer, starting replication: peer=server-3cf08bfb-6344-2224-dda8-9d4e5548936b
2026-04-23T00:14:49.467Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: added peer, starting replication: peer=server-7088f165-9fe0-8065-15fa-415ee044129e
2026-04-23T00:14:49.467Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: pipelining replication: peer="{Voter server-3cf08bfb-6344-2224-dda8-9d4e5548936b 3cf08bfb-6344-2224-dda8-9d4e5548936b}"
2026-04-23T00:14:49.467Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: pipelining replication: peer="{Voter server-7088f165-9fe0-8065-15fa-415ee044129e 7088f165-9fe0-8065-15fa-415ee044129e}"
2026-04-23T00:14:49.572Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:49.467505311 +0000 UTC m=+6.507106693 (1 nodes), 58.225701ms from start of poll, 58.586082ms from cluster start. Timeout at 2026-04-23 00:14:49.572507464 +0000 UTC m=+6.612108816, 105.002123ms after stability
2026-04-23T00:14:49.575Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: aborting pipeline replication: peer="{Voter server-3cf08bfb-6344-2224-dda8-9d4e5548936b 3cf08bfb-6344-2224-dda8-9d4e5548936b}"
2026-04-23T00:14:49.575Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: aborting pipeline replication: peer="{Voter server-7088f165-9fe0-8065-15fa-415ee044129e 7088f165-9fe0-8065-15fa-415ee044129e}"
2026-04-23T00:14:49.615Z [INFO]  creating new snapshot: path=/tmp/raft4130294849/snapshots/2-27-1776903289615.tmp
2026-04-23T00:14:49.615Z [INFO]  server-3cf08bfb-6344-2224-dda8-9d4e5548936b: starting restore from snapshot: id=2-27-1776903289615 last-index=27 last-term=2 size-in-bytes=168
2026-04-23T00:14:49.615Z [INFO]  server-3cf08bfb-6344-2224-dda8-9d4e5548936b: snapshot restore progress: id=2-27-1776903289615 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2026-04-23T00:14:49.616Z [INFO]  server-3cf08bfb-6344-2224-dda8-9d4e5548936b: restored from snapshot: id=2-27-1776903289615 last-index=27 last-term=2 size-in-bytes=168
2026-04-23T00:14:49.616Z [INFO]  server-3cf08bfb-6344-2224-dda8-9d4e5548936b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3cf08bfb-6344-2224-dda8-9d4e5548936b Address:3cf08bfb-6344-2224-dda8-9d4e5548936b} {Suffrage:Voter ID:server-7088f165-9fe0-8065-15fa-415ee044129e Address:7088f165-9fe0-8065-15fa-415ee044129e} {Suffrage:Voter ID:server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe Address:7c05b4bb-d3f9-d028-1149-780b4b5f4cbe}]"
2026-04-23T00:14:49.616Z [INFO]  server-3cf08bfb-6344-2224-dda8-9d4e5548936b: entering follower state: follower="Node at 3cf08bfb-6344-2224-dda8-9d4e5548936b [Follower]" leader-address= leader-id=
2026-04-23T00:14:49.616Z [INFO]  creating new snapshot: path=/tmp/raft953242883/snapshots/2-27-1776903289616.tmp
2026-04-23T00:14:49.616Z [INFO]  server-7088f165-9fe0-8065-15fa-415ee044129e: starting restore from snapshot: id=2-27-1776903289616 last-index=27 last-term=2 size-in-bytes=168
2026-04-23T00:14:49.616Z [INFO]  server-7088f165-9fe0-8065-15fa-415ee044129e: snapshot restore progress: id=2-27-1776903289616 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2026-04-23T00:14:49.616Z [INFO]  server-7088f165-9fe0-8065-15fa-415ee044129e: restored from snapshot: id=2-27-1776903289616 last-index=27 last-term=2 size-in-bytes=168
2026-04-23T00:14:49.616Z [INFO]  server-7088f165-9fe0-8065-15fa-415ee044129e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3cf08bfb-6344-2224-dda8-9d4e5548936b Address:3cf08bfb-6344-2224-dda8-9d4e5548936b} {Suffrage:Voter ID:server-7088f165-9fe0-8065-15fa-415ee044129e Address:7088f165-9fe0-8065-15fa-415ee044129e} {Suffrage:Voter ID:server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe Address:7c05b4bb-d3f9-d028-1149-780b4b5f4cbe}]"
2026-04-23T00:14:49.616Z [INFO]  server-7088f165-9fe0-8065-15fa-415ee044129e: entering follower state: follower="Node at 7088f165-9fe0-8065-15fa-415ee044129e [Follower]" leader-address= leader-id=
2026-04-23T00:14:49.617Z [INFO]  creating new snapshot: path=/tmp/raft709373388/snapshots/2-27-1776903289617.tmp
2026-04-23T00:14:49.617Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: starting restore from snapshot: id=2-27-1776903289617 last-index=27 last-term=2 size-in-bytes=168
2026-04-23T00:14:49.617Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: snapshot restore progress: id=2-27-1776903289617 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2026-04-23T00:14:49.617Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: restored from snapshot: id=2-27-1776903289617 last-index=27 last-term=2 size-in-bytes=168
2026-04-23T00:14:49.617Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3cf08bfb-6344-2224-dda8-9d4e5548936b Address:3cf08bfb-6344-2224-dda8-9d4e5548936b} {Suffrage:Voter ID:server-7088f165-9fe0-8065-15fa-415ee044129e Address:7088f165-9fe0-8065-15fa-415ee044129e} {Suffrage:Voter ID:server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe Address:7c05b4bb-d3f9-d028-1149-780b4b5f4cbe}]"
2026-04-23T00:14:49.617Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: entering follower state: follower="Node at 7c05b4bb-d3f9-d028-1149-780b4b5f4cbe [Follower]" leader-address= leader-id=
2026-04-23T00:14:49.676Z [WARN]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:49.676Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: entering candidate state: node="Node at 7c05b4bb-d3f9-d028-1149-780b4b5f4cbe [Candidate]" term=3
2026-04-23T00:14:49.676Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: election won: term=3 tally=2
2026-04-23T00:14:49.676Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: entering leader state: leader="Node at 7c05b4bb-d3f9-d028-1149-780b4b5f4cbe [Leader]"
2026-04-23T00:14:49.676Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: added peer, starting replication: peer=server-3cf08bfb-6344-2224-dda8-9d4e5548936b
2026-04-23T00:14:49.676Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: added peer, starting replication: peer=server-7088f165-9fe0-8065-15fa-415ee044129e
2026-04-23T00:14:49.676Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: pipelining replication: peer="{Voter server-3cf08bfb-6344-2224-dda8-9d4e5548936b 3cf08bfb-6344-2224-dda8-9d4e5548936b}"
2026-04-23T00:14:49.676Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: pipelining replication: peer="{Voter server-7088f165-9fe0-8065-15fa-415ee044129e 7088f165-9fe0-8065-15fa-415ee044129e}"
2026-04-23T00:14:49.933Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:50.038Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:49.933697108 +0000 UTC m=+6.973298517 (1 nodes), 83.637µs from start of poll, 524.777906ms from cluster start. Timeout at 2026-04-23 00:14:50.038705574 +0000 UTC m=+7.078306945, 105.008428ms after stability
2026-04-23T00:14:50.039Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: aborting pipeline replication: peer="{Voter server-3cf08bfb-6344-2224-dda8-9d4e5548936b 3cf08bfb-6344-2224-dda8-9d4e5548936b}"
2026-04-23T00:14:50.039Z [INFO]  server-7c05b4bb-d3f9-d028-1149-780b4b5f4cbe: aborting pipeline replication: peer="{Voter server-7088f165-9fe0-8065-15fa-415ee044129e 7088f165-9fe0-8065-15fa-415ee044129e}"
--- PASS: TestRaft_RecoverCluster (2.55s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (1.01s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.78s)
    --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.75s)
=== RUN   TestRaft_HasExistingState
    raft_test.go:258: Skipping unreliable test
--- SKIP: TestRaft_HasExistingState (0.00s)
=== RUN   TestRaft_SingleNode
2026-04-23T00:14:50.160Z [INFO]  server-dd0f4c04-2220-a191-7bc0-c30d9b459fa1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd0f4c04-2220-a191-7bc0-c30d9b459fa1 Address:dd0f4c04-2220-a191-7bc0-c30d9b459fa1}]"
2026-04-23T00:14:50.160Z [INFO]  server-dd0f4c04-2220-a191-7bc0-c30d9b459fa1: entering follower state: follower="Node at dd0f4c04-2220-a191-7bc0-c30d9b459fa1 [Follower]" leader-address= leader-id=
2026-04-23T00:14:50.232Z [WARN]  server-dd0f4c04-2220-a191-7bc0-c30d9b459fa1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:50.232Z [INFO]  server-dd0f4c04-2220-a191-7bc0-c30d9b459fa1: entering candidate state: node="Node at dd0f4c04-2220-a191-7bc0-c30d9b459fa1 [Candidate]" term=2
2026-04-23T00:14:50.232Z [INFO]  server-dd0f4c04-2220-a191-7bc0-c30d9b459fa1: election won: term=2 tally=1
2026-04-23T00:14:50.232Z [INFO]  server-dd0f4c04-2220-a191-7bc0-c30d9b459fa1: entering leader state: leader="Node at dd0f4c04-2220-a191-7bc0-c30d9b459fa1 [Leader]"
--- PASS: TestRaft_SingleNode (0.22s)
=== RUN   TestRaft_TripleNode
2026-04-23T00:14:50.379Z [INFO]  server-c373c83f-6dc3-6756-a2b1-c75e04e2879b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c373c83f-6dc3-6756-a2b1-c75e04e2879b Address:c373c83f-6dc3-6756-a2b1-c75e04e2879b} {Suffrage:Voter ID:server-31257e04-4a01-9169-7fd9-c4af7b2a5a6b Address:31257e04-4a01-9169-7fd9-c4af7b2a5a6b} {Suffrage:Voter ID:server-f467a970-33a4-d166-fef4-1543c0eab946 Address:f467a970-33a4-d166-fef4-1543c0eab946}]"
2026-04-23T00:14:50.379Z [INFO]  server-c373c83f-6dc3-6756-a2b1-c75e04e2879b: entering follower state: follower="Node at c373c83f-6dc3-6756-a2b1-c75e04e2879b [Follower]" leader-address= leader-id=
2026-04-23T00:14:50.379Z [INFO]  server-31257e04-4a01-9169-7fd9-c4af7b2a5a6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c373c83f-6dc3-6756-a2b1-c75e04e2879b Address:c373c83f-6dc3-6756-a2b1-c75e04e2879b} {Suffrage:Voter ID:server-31257e04-4a01-9169-7fd9-c4af7b2a5a6b Address:31257e04-4a01-9169-7fd9-c4af7b2a5a6b} {Suffrage:Voter ID:server-f467a970-33a4-d166-fef4-1543c0eab946 Address:f467a970-33a4-d166-fef4-1543c0eab946}]"
2026-04-23T00:14:50.379Z [INFO]  server-31257e04-4a01-9169-7fd9-c4af7b2a5a6b: entering follower state: follower="Node at 31257e04-4a01-9169-7fd9-c4af7b2a5a6b [Follower]" leader-address= leader-id=
2026-04-23T00:14:50.379Z [INFO]  server-f467a970-33a4-d166-fef4-1543c0eab946: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c373c83f-6dc3-6756-a2b1-c75e04e2879b Address:c373c83f-6dc3-6756-a2b1-c75e04e2879b} {Suffrage:Voter ID:server-31257e04-4a01-9169-7fd9-c4af7b2a5a6b Address:31257e04-4a01-9169-7fd9-c4af7b2a5a6b} {Suffrage:Voter ID:server-f467a970-33a4-d166-fef4-1543c0eab946 Address:f467a970-33a4-d166-fef4-1543c0eab946}]"
2026-04-23T00:14:50.379Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:14:50.379Z [INFO]  server-f467a970-33a4-d166-fef4-1543c0eab946: entering follower state: follower="Node at f467a970-33a4-d166-fef4-1543c0eab946 [Follower]" leader-address= leader-id=
2026-04-23T00:14:50.450Z [WARN]  server-c373c83f-6dc3-6756-a2b1-c75e04e2879b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:50.450Z [INFO]  server-c373c83f-6dc3-6756-a2b1-c75e04e2879b: entering candidate state: node="Node at c373c83f-6dc3-6756-a2b1-c75e04e2879b [Candidate]" term=2
2026-04-23T00:14:50.450Z [INFO]  server-c373c83f-6dc3-6756-a2b1-c75e04e2879b: election won: term=2 tally=2
2026-04-23T00:14:50.450Z [INFO]  server-c373c83f-6dc3-6756-a2b1-c75e04e2879b: entering leader state: leader="Node at c373c83f-6dc3-6756-a2b1-c75e04e2879b [Leader]"
2026-04-23T00:14:50.450Z [INFO]  server-c373c83f-6dc3-6756-a2b1-c75e04e2879b: added peer, starting replication: peer=server-31257e04-4a01-9169-7fd9-c4af7b2a5a6b
2026-04-23T00:14:50.450Z [INFO]  server-c373c83f-6dc3-6756-a2b1-c75e04e2879b: added peer, starting replication: peer=server-f467a970-33a4-d166-fef4-1543c0eab946
2026-04-23T00:14:50.450Z [INFO]  server-c373c83f-6dc3-6756-a2b1-c75e04e2879b: pipelining replication: peer="{Voter server-31257e04-4a01-9169-7fd9-c4af7b2a5a6b 31257e04-4a01-9169-7fd9-c4af7b2a5a6b}"
2026-04-23T00:14:50.450Z [INFO]  server-c373c83f-6dc3-6756-a2b1-c75e04e2879b: pipelining replication: peer="{Voter server-f467a970-33a4-d166-fef4-1543c0eab946 f467a970-33a4-d166-fef4-1543c0eab946}"
2026-04-23T00:14:50.556Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:14:50.45060577 +0000 UTC m=+7.490207127 (2 nodes), 70.97663ms from start of poll, 71.331109ms from cluster start. Timeout at 2026-04-23 00:14:50.555608548 +0000 UTC m=+7.595209904, 105.002777ms after stability
2026-04-23T00:14:50.556Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:50.661Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:50.556127097 +0000 UTC m=+7.595728458 (1 nodes), 917ns from start of poll, 176.85244ms from cluster start. Timeout at 2026-04-23 00:14:50.661129648 +0000 UTC m=+7.700731001, 105.002543ms after stability
2026-04-23T00:14:50.672Z [INFO]  server-c373c83f-6dc3-6756-a2b1-c75e04e2879b: aborting pipeline replication: peer="{Voter server-f467a970-33a4-d166-fef4-1543c0eab946 f467a970-33a4-d166-fef4-1543c0eab946}"
2026-04-23T00:14:50.672Z [INFO]  server-c373c83f-6dc3-6756-a2b1-c75e04e2879b: aborting pipeline replication: peer="{Voter server-31257e04-4a01-9169-7fd9-c4af7b2a5a6b 31257e04-4a01-9169-7fd9-c4af7b2a5a6b}"
--- PASS: TestRaft_TripleNode (0.29s)
=== RUN   TestRaft_LeaderFail
2026-04-23T00:14:50.673Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3120537e-7b14-73a9-6363-e24fca43dfd0 Address:3120537e-7b14-73a9-6363-e24fca43dfd0} {Suffrage:Voter ID:server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 Address:323a8a44-92ff-84ef-6ca5-05a9c6b9aea4} {Suffrage:Voter ID:server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c Address:547c3f8e-1b9b-eb04-201e-c7d2ec70134c}]"
2026-04-23T00:14:50.673Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: entering follower state: follower="Node at 3120537e-7b14-73a9-6363-e24fca43dfd0 [Follower]" leader-address= leader-id=
2026-04-23T00:14:50.673Z [INFO]  server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3120537e-7b14-73a9-6363-e24fca43dfd0 Address:3120537e-7b14-73a9-6363-e24fca43dfd0} {Suffrage:Voter ID:server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 Address:323a8a44-92ff-84ef-6ca5-05a9c6b9aea4} {Suffrage:Voter ID:server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c Address:547c3f8e-1b9b-eb04-201e-c7d2ec70134c}]"
2026-04-23T00:14:50.673Z [INFO]  server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4: entering follower state: follower="Node at 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 [Follower]" leader-address= leader-id=
2026-04-23T00:14:50.673Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3120537e-7b14-73a9-6363-e24fca43dfd0 Address:3120537e-7b14-73a9-6363-e24fca43dfd0} {Suffrage:Voter ID:server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 Address:323a8a44-92ff-84ef-6ca5-05a9c6b9aea4} {Suffrage:Voter ID:server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c Address:547c3f8e-1b9b-eb04-201e-c7d2ec70134c}]"
2026-04-23T00:14:50.673Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:14:50.673Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: entering follower state: follower="Node at 547c3f8e-1b9b-eb04-201e-c7d2ec70134c [Follower]" leader-address= leader-id=
2026-04-23T00:14:50.726Z [WARN]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:50.726Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: entering candidate state: node="Node at 3120537e-7b14-73a9-6363-e24fca43dfd0 [Candidate]" term=2
2026-04-23T00:14:50.726Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: election won: term=2 tally=2
2026-04-23T00:14:50.726Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: entering leader state: leader="Node at 3120537e-7b14-73a9-6363-e24fca43dfd0 [Leader]"
2026-04-23T00:14:50.726Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: added peer, starting replication: peer=server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4
2026-04-23T00:14:50.726Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: added peer, starting replication: peer=server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c
2026-04-23T00:14:50.726Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: pipelining replication: peer="{Voter server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4}"
2026-04-23T00:14:50.726Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: pipelining replication: peer="{Voter server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c 547c3f8e-1b9b-eb04-201e-c7d2ec70134c}"
2026-04-23T00:14:50.832Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:14:50.726462205 +0000 UTC m=+7.766063556 (2 nodes), 52.985577ms from start of poll, 53.439147ms from cluster start. Timeout at 2026-04-23 00:14:50.831464681 +0000 UTC m=+7.871066052, 105.002496ms after stability
2026-04-23T00:14:50.832Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:50.937Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:50.832329855 +0000 UTC m=+7.871931209 (1 nodes), 877ns from start of poll, 159.3068ms from cluster start. Timeout at 2026-04-23 00:14:50.937335836 +0000 UTC m=+7.976937193, 105.005984ms after stability
    raft_test.go:378: [INFO] Disconnecting Node at 3120537e-7b14-73a9-6363-e24fca43dfd0 [Leader]
2026-04-23T00:14:50.950Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to heartbeat to: peer=323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 backoff time=10ms error="failed to connect to peer: 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4"
2026-04-23T00:14:50.950Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to heartbeat to: peer=547c3f8e-1b9b-eb04-201e-c7d2ec70134c backoff time=10ms error="failed to connect to peer: 547c3f8e-1b9b-eb04-201e-c7d2ec70134c"
2026-04-23T00:14:50.950Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:50.950Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to pipeline appendEntries: peer="{Voter server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c 547c3f8e-1b9b-eb04-201e-c7d2ec70134c}" error="append pipeline closed"
2026-04-23T00:14:50.950Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: aborting pipeline replication: peer="{Voter server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c 547c3f8e-1b9b-eb04-201e-c7d2ec70134c}"
2026-04-23T00:14:50.955Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to pipeline appendEntries: peer="{Voter server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4}" error="append pipeline closed"
2026-04-23T00:14:50.955Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: aborting pipeline replication: peer="{Voter server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4}"
2026-04-23T00:14:50.959Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to appendEntries to: peer="{Voter server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c 547c3f8e-1b9b-eb04-201e-c7d2ec70134c}" error="failed to connect to peer: 547c3f8e-1b9b-eb04-201e-c7d2ec70134c"
2026-04-23T00:14:50.965Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to appendEntries to: peer="{Voter server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4}" error="failed to connect to peer: 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4"
2026-04-23T00:14:50.969Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to heartbeat to: peer=323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 backoff time=10ms error="failed to connect to peer: 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4"
2026-04-23T00:14:50.969Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to heartbeat to: peer=547c3f8e-1b9b-eb04-201e-c7d2ec70134c backoff time=10ms error="failed to connect to peer: 547c3f8e-1b9b-eb04-201e-c7d2ec70134c"
2026-04-23T00:14:50.977Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to appendEntries to: peer="{Voter server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c 547c3f8e-1b9b-eb04-201e-c7d2ec70134c}" error="failed to connect to peer: 547c3f8e-1b9b-eb04-201e-c7d2ec70134c"
2026-04-23T00:14:50.984Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to appendEntries to: peer="{Voter server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4}" error="failed to connect to peer: 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4"
2026-04-23T00:14:50.988Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to heartbeat to: peer=323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 backoff time=10ms error="failed to connect to peer: 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4"
2026-04-23T00:14:50.991Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to heartbeat to: peer=547c3f8e-1b9b-eb04-201e-c7d2ec70134c backoff time=10ms error="failed to connect to peer: 547c3f8e-1b9b-eb04-201e-c7d2ec70134c"
2026-04-23T00:14:50.993Z [WARN]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to contact: server-id=server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c time=50.194584ms
2026-04-23T00:14:50.995Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to appendEntries to: peer="{Voter server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c 547c3f8e-1b9b-eb04-201e-c7d2ec70134c}" error="failed to connect to peer: 547c3f8e-1b9b-eb04-201e-c7d2ec70134c"
2026-04-23T00:14:51.004Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to appendEntries to: peer="{Voter server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4}" error="failed to connect to peer: 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4"
2026-04-23T00:14:51.004Z [WARN]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to contact: server-id=server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 time=57.975292ms
2026-04-23T00:14:51.004Z [WARN]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to contact: server-id=server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c time=61.122067ms
2026-04-23T00:14:51.004Z [WARN]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to contact quorum of nodes, stepping down
2026-04-23T00:14:51.004Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: entering follower state: follower="Node at 3120537e-7b14-73a9-6363-e24fca43dfd0 [Follower]" leader-address= leader-id=
2026-04-23T00:14:51.011Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to heartbeat to: peer=547c3f8e-1b9b-eb04-201e-c7d2ec70134c backoff time=20ms error="failed to connect to peer: 547c3f8e-1b9b-eb04-201e-c7d2ec70134c"
2026-04-23T00:14:51.023Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to appendEntries to: peer="{Voter server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c 547c3f8e-1b9b-eb04-201e-c7d2ec70134c}" error="failed to connect to peer: 547c3f8e-1b9b-eb04-201e-c7d2ec70134c"
2026-04-23T00:14:51.059Z [WARN]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:51.060Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: entering candidate state: node="Node at 3120537e-7b14-73a9-6363-e24fca43dfd0 [Candidate]" term=3
2026-04-23T00:14:51.060Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to make requestVote RPC: target="{Voter server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c 547c3f8e-1b9b-eb04-201e-c7d2ec70134c}" error="failed to connect to peer: 547c3f8e-1b9b-eb04-201e-c7d2ec70134c" term=3
2026-04-23T00:14:51.060Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to make requestVote RPC: target="{Voter server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4}" error="failed to connect to peer: 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4" term=3
2026-04-23T00:14:51.065Z [WARN]  server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4: heartbeat timeout reached, starting election: last-leader-addr=3120537e-7b14-73a9-6363-e24fca43dfd0 last-leader-id=server-3120537e-7b14-73a9-6363-e24fca43dfd0
2026-04-23T00:14:51.065Z [INFO]  server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4: entering candidate state: node="Node at 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 [Candidate]" term=3
2026-04-23T00:14:51.065Z [WARN]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: rejecting vote request since we have a leader: from=323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 leader=3120537e-7b14-73a9-6363-e24fca43dfd0 leader-id=server-3120537e-7b14-73a9-6363-e24fca43dfd0
2026-04-23T00:14:51.065Z [ERROR] server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4: failed to make requestVote RPC: target="{Voter server-3120537e-7b14-73a9-6363-e24fca43dfd0 3120537e-7b14-73a9-6363-e24fca43dfd0}" error="failed to connect to peer: 3120537e-7b14-73a9-6363-e24fca43dfd0" term=3
2026-04-23T00:14:51.069Z [WARN]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: heartbeat timeout reached, starting election: last-leader-addr=3120537e-7b14-73a9-6363-e24fca43dfd0 last-leader-id=server-3120537e-7b14-73a9-6363-e24fca43dfd0
2026-04-23T00:14:51.069Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: entering candidate state: node="Node at 547c3f8e-1b9b-eb04-201e-c7d2ec70134c [Candidate]" term=3
2026-04-23T00:14:51.069Z [INFO]  server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4: duplicate requestVote for same term: term=3
2026-04-23T00:14:51.069Z [ERROR] server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: failed to make requestVote RPC: target="{Voter server-3120537e-7b14-73a9-6363-e24fca43dfd0 3120537e-7b14-73a9-6363-e24fca43dfd0}" error="failed to connect to peer: 3120537e-7b14-73a9-6363-e24fca43dfd0" term=3
2026-04-23T00:14:51.121Z [WARN]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: Election timeout reached, restarting election
2026-04-23T00:14:51.121Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: entering candidate state: node="Node at 3120537e-7b14-73a9-6363-e24fca43dfd0 [Candidate]" term=4
2026-04-23T00:14:51.121Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to make requestVote RPC: target="{Voter server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c 547c3f8e-1b9b-eb04-201e-c7d2ec70134c}" error="failed to connect to peer: 547c3f8e-1b9b-eb04-201e-c7d2ec70134c" term=4
2026-04-23T00:14:51.121Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to make requestVote RPC: target="{Voter server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4}" error="failed to connect to peer: 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4" term=4
2026-04-23T00:14:51.127Z [WARN]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: Election timeout reached, restarting election
2026-04-23T00:14:51.127Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: entering candidate state: node="Node at 547c3f8e-1b9b-eb04-201e-c7d2ec70134c [Candidate]" term=4
2026-04-23T00:14:51.127Z [ERROR] server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: failed to make requestVote RPC: target="{Voter server-3120537e-7b14-73a9-6363-e24fca43dfd0 3120537e-7b14-73a9-6363-e24fca43dfd0}" error="failed to connect to peer: 3120537e-7b14-73a9-6363-e24fca43dfd0" term=4
2026-04-23T00:14:51.127Z [INFO]  server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4: entering follower state: follower="Node at 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 [Follower]" leader-address= leader-id=
2026-04-23T00:14:51.127Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: election won: term=4 tally=2
2026-04-23T00:14:51.128Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: entering leader state: leader="Node at 547c3f8e-1b9b-eb04-201e-c7d2ec70134c [Leader]"
2026-04-23T00:14:51.128Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: added peer, starting replication: peer=server-3120537e-7b14-73a9-6363-e24fca43dfd0
2026-04-23T00:14:51.128Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: added peer, starting replication: peer=server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4
2026-04-23T00:14:51.128Z [ERROR] server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: failed to appendEntries to: peer="{Voter server-3120537e-7b14-73a9-6363-e24fca43dfd0 3120537e-7b14-73a9-6363-e24fca43dfd0}" error="failed to connect to peer: 3120537e-7b14-73a9-6363-e24fca43dfd0"
2026-04-23T00:14:51.128Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: pipelining replication: peer="{Voter server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4}"
2026-04-23T00:14:51.137Z [ERROR] server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: failed to heartbeat to: peer=3120537e-7b14-73a9-6363-e24fca43dfd0 backoff time=10ms error="failed to connect to peer: 3120537e-7b14-73a9-6363-e24fca43dfd0"
2026-04-23T00:14:51.148Z [ERROR] server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: failed to appendEntries to: peer="{Voter server-3120537e-7b14-73a9-6363-e24fca43dfd0 3120537e-7b14-73a9-6363-e24fca43dfd0}" error="failed to connect to peer: 3120537e-7b14-73a9-6363-e24fca43dfd0"
2026-04-23T00:14:51.156Z [ERROR] server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: failed to heartbeat to: peer=3120537e-7b14-73a9-6363-e24fca43dfd0 backoff time=10ms error="failed to connect to peer: 3120537e-7b14-73a9-6363-e24fca43dfd0"
2026-04-23T00:14:51.165Z [ERROR] server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: failed to appendEntries to: peer="{Voter server-3120537e-7b14-73a9-6363-e24fca43dfd0 3120537e-7b14-73a9-6363-e24fca43dfd0}" error="failed to connect to peer: 3120537e-7b14-73a9-6363-e24fca43dfd0"
2026-04-23T00:14:51.177Z [ERROR] server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: failed to heartbeat to: peer=3120537e-7b14-73a9-6363-e24fca43dfd0 backoff time=10ms error="failed to connect to peer: 3120537e-7b14-73a9-6363-e24fca43dfd0"
2026-04-23T00:14:51.178Z [WARN]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: failed to contact: server-id=server-3120537e-7b14-73a9-6363-e24fca43dfd0 time=50.950166ms
2026-04-23T00:14:51.192Z [ERROR] server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: failed to appendEntries to: peer="{Voter server-3120537e-7b14-73a9-6363-e24fca43dfd0 3120537e-7b14-73a9-6363-e24fca43dfd0}" error="failed to connect to peer: 3120537e-7b14-73a9-6363-e24fca43dfd0"
2026-04-23T00:14:51.195Z [WARN]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: Election timeout reached, restarting election
2026-04-23T00:14:51.195Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: entering candidate state: node="Node at 3120537e-7b14-73a9-6363-e24fca43dfd0 [Candidate]" term=5
2026-04-23T00:14:51.195Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to make requestVote RPC: target="{Voter server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c 547c3f8e-1b9b-eb04-201e-c7d2ec70134c}" error="failed to connect to peer: 547c3f8e-1b9b-eb04-201e-c7d2ec70134c" term=5
2026-04-23T00:14:51.195Z [ERROR] server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to make requestVote RPC: target="{Voter server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4}" error="failed to connect to peer: 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4" term=5
2026-04-23T00:14:51.198Z [ERROR] server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: failed to heartbeat to: peer=3120537e-7b14-73a9-6363-e24fca43dfd0 backoff time=20ms error="failed to connect to peer: 3120537e-7b14-73a9-6363-e24fca43dfd0"
2026-04-23T00:14:51.225Z [ERROR] server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: failed to heartbeat to: peer=3120537e-7b14-73a9-6363-e24fca43dfd0 backoff time=25ms error="failed to connect to peer: 3120537e-7b14-73a9-6363-e24fca43dfd0"
2026-04-23T00:14:51.229Z [WARN]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: failed to contact: server-id=server-3120537e-7b14-73a9-6363-e24fca43dfd0 time=100.972964ms
2026-04-23T00:14:51.233Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:51.128041985 +0000 UTC m=+8.167643353 (1 nodes), 177.807212ms from start of poll, 455.018944ms from cluster start. Timeout at 2026-04-23 00:14:51.233045225 +0000 UTC m=+8.272646580, 105.003227ms after stability
    raft_test.go:413: [INFO] Reconnecting Node at 3120537e-7b14-73a9-6363-e24fca43dfd0 [Candidate]
2026-04-23T00:14:51.239Z [ERROR] server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: peer has newer term, stopping replication: peer="{Voter server-3120537e-7b14-73a9-6363-e24fca43dfd0 3120537e-7b14-73a9-6363-e24fca43dfd0}"
2026-04-23T00:14:51.239Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: entering follower state: follower="Node at 547c3f8e-1b9b-eb04-201e-c7d2ec70134c [Follower]" leader-address= leader-id=
2026-04-23T00:14:51.239Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: aborting pipeline replication: peer="{Voter server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4}"
2026-04-23T00:14:51.282Z [WARN]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: Election timeout reached, restarting election
2026-04-23T00:14:51.282Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: entering candidate state: node="Node at 3120537e-7b14-73a9-6363-e24fca43dfd0 [Candidate]" term=6
2026-04-23T00:14:51.282Z [WARN]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: rejecting vote request since our last term is greater: candidate=3120537e-7b14-73a9-6363-e24fca43dfd0 last-term=4 last-candidate-term=2
2026-04-23T00:14:51.282Z [WARN]  server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4: rejecting vote request since we have a leader: from=3120537e-7b14-73a9-6363-e24fca43dfd0 leader=547c3f8e-1b9b-eb04-201e-c7d2ec70134c leader-id=server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c
2026-04-23T00:14:51.336Z [WARN]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: Election timeout reached, restarting election
2026-04-23T00:14:51.336Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: entering candidate state: node="Node at 3120537e-7b14-73a9-6363-e24fca43dfd0 [Candidate]" term=7
2026-04-23T00:14:51.336Z [WARN]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: rejecting vote request since our last term is greater: candidate=3120537e-7b14-73a9-6363-e24fca43dfd0 last-term=4 last-candidate-term=2
2026-04-23T00:14:51.336Z [WARN]  server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4: rejecting vote request since we have a leader: from=3120537e-7b14-73a9-6363-e24fca43dfd0 leader=547c3f8e-1b9b-eb04-201e-c7d2ec70134c leader-id=server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c
2026-04-23T00:14:51.338Z [WARN]  server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4: heartbeat timeout reached, starting election: last-leader-addr=547c3f8e-1b9b-eb04-201e-c7d2ec70134c last-leader-id=server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c
2026-04-23T00:14:51.338Z [INFO]  server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4: entering candidate state: node="Node at 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 [Candidate]" term=5
2026-04-23T00:14:51.338Z [INFO]  server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4: entering follower state: follower="Node at 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 [Follower]" leader-address= leader-id=
2026-04-23T00:14:51.338Z [WARN]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:51.338Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: entering candidate state: node="Node at 547c3f8e-1b9b-eb04-201e-c7d2ec70134c [Candidate]" term=8
2026-04-23T00:14:51.338Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: election won: term=8 tally=2
2026-04-23T00:14:51.338Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: entering leader state: leader="Node at 547c3f8e-1b9b-eb04-201e-c7d2ec70134c [Leader]"
2026-04-23T00:14:51.338Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: added peer, starting replication: peer=server-3120537e-7b14-73a9-6363-e24fca43dfd0
2026-04-23T00:14:51.338Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: added peer, starting replication: peer=server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4
2026-04-23T00:14:51.338Z [INFO]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: entering follower state: follower="Node at 3120537e-7b14-73a9-6363-e24fca43dfd0 [Follower]" leader-address= leader-id=
2026-04-23T00:14:51.338Z [WARN]  server-3120537e-7b14-73a9-6363-e24fca43dfd0: failed to get previous log: previous-index=5 last-index=3 error="log not found"
2026-04-23T00:14:51.338Z [WARN]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: appendEntries rejected, sending older logs: peer="{Voter server-3120537e-7b14-73a9-6363-e24fca43dfd0 3120537e-7b14-73a9-6363-e24fca43dfd0}" next=4
2026-04-23T00:14:51.338Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: pipelining replication: peer="{Voter server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4}"
2026-04-23T00:14:51.339Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: pipelining replication: peer="{Voter server-3120537e-7b14-73a9-6363-e24fca43dfd0 3120537e-7b14-73a9-6363-e24fca43dfd0}"
2026-04-23T00:14:51.345Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: aborting pipeline replication: peer="{Voter server-323a8a44-92ff-84ef-6ca5-05a9c6b9aea4 323a8a44-92ff-84ef-6ca5-05a9c6b9aea4}"
2026-04-23T00:14:51.345Z [INFO]  server-547c3f8e-1b9b-eb04-201e-c7d2ec70134c: aborting pipeline replication: peer="{Voter server-3120537e-7b14-73a9-6363-e24fca43dfd0 3120537e-7b14-73a9-6363-e24fca43dfd0}"
--- PASS: TestRaft_LeaderFail (0.67s)
=== RUN   TestRaft_BehindFollower
2026-04-23T00:14:51.347Z [INFO]  server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f Address:807a4e18-1c49-3e1f-2d72-b61a5b2e112f} {Suffrage:Voter ID:server-04c65549-9166-8a86-6ad4-e71543163513 Address:04c65549-9166-8a86-6ad4-e71543163513} {Suffrage:Voter ID:server-dd187511-d444-8823-a762-192d7edbb1fc Address:dd187511-d444-8823-a762-192d7edbb1fc}]"
2026-04-23T00:14:51.347Z [INFO]  server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f: entering follower state: follower="Node at 807a4e18-1c49-3e1f-2d72-b61a5b2e112f [Follower]" leader-address= leader-id=
2026-04-23T00:14:51.347Z [INFO]  server-04c65549-9166-8a86-6ad4-e71543163513: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f Address:807a4e18-1c49-3e1f-2d72-b61a5b2e112f} {Suffrage:Voter ID:server-04c65549-9166-8a86-6ad4-e71543163513 Address:04c65549-9166-8a86-6ad4-e71543163513} {Suffrage:Voter ID:server-dd187511-d444-8823-a762-192d7edbb1fc Address:dd187511-d444-8823-a762-192d7edbb1fc}]"
2026-04-23T00:14:51.347Z [INFO]  server-04c65549-9166-8a86-6ad4-e71543163513: entering follower state: follower="Node at 04c65549-9166-8a86-6ad4-e71543163513 [Follower]" leader-address= leader-id=
2026-04-23T00:14:51.347Z [INFO]  server-dd187511-d444-8823-a762-192d7edbb1fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f Address:807a4e18-1c49-3e1f-2d72-b61a5b2e112f} {Suffrage:Voter ID:server-04c65549-9166-8a86-6ad4-e71543163513 Address:04c65549-9166-8a86-6ad4-e71543163513} {Suffrage:Voter ID:server-dd187511-d444-8823-a762-192d7edbb1fc Address:dd187511-d444-8823-a762-192d7edbb1fc}]"
2026-04-23T00:14:51.347Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:51.347Z [INFO]  server-dd187511-d444-8823-a762-192d7edbb1fc: entering follower state: follower="Node at dd187511-d444-8823-a762-192d7edbb1fc [Follower]" leader-address= leader-id=
2026-04-23T00:14:51.427Z [WARN]  server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:51.427Z [INFO]  server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f: entering candidate state: node="Node at 807a4e18-1c49-3e1f-2d72-b61a5b2e112f [Candidate]" term=2
2026-04-23T00:14:51.428Z [INFO]  server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f: election won: term=2 tally=2
2026-04-23T00:14:51.428Z [INFO]  server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f: entering leader state: leader="Node at 807a4e18-1c49-3e1f-2d72-b61a5b2e112f [Leader]"
2026-04-23T00:14:51.428Z [INFO]  server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f: added peer, starting replication: peer=server-04c65549-9166-8a86-6ad4-e71543163513
2026-04-23T00:14:51.428Z [INFO]  server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f: added peer, starting replication: peer=server-dd187511-d444-8823-a762-192d7edbb1fc
2026-04-23T00:14:51.428Z [INFO]  server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f: pipelining replication: peer="{Voter server-dd187511-d444-8823-a762-192d7edbb1fc dd187511-d444-8823-a762-192d7edbb1fc}"
2026-04-23T00:14:51.428Z [INFO]  server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f: pipelining replication: peer="{Voter server-04c65549-9166-8a86-6ad4-e71543163513 04c65549-9166-8a86-6ad4-e71543163513}"
2026-04-23T00:14:51.533Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:51.428141154 +0000 UTC m=+8.467742521 (1 nodes), 80.654949ms from start of poll, 81.150294ms from cluster start. Timeout at 2026-04-23 00:14:51.533143624 +0000 UTC m=+8.572744981, 105.00246ms after stability
2026-04-23T00:14:51.533Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:14:51.639Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:14:51.533727316 +0000 UTC m=+8.573328672 (2 nodes), 416ns from start of poll, 186.736445ms from cluster start. Timeout at 2026-04-23 00:14:51.638732462 +0000 UTC m=+8.678333818, 105.005146ms after stability
2026-04-23T00:14:51.639Z [ERROR] server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f: failed to pipeline appendEntries: peer="{Voter server-04c65549-9166-8a86-6ad4-e71543163513 04c65549-9166-8a86-6ad4-e71543163513}" error="append pipeline closed"
2026-04-23T00:14:51.639Z [INFO]  server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f: aborting pipeline replication: peer="{Voter server-04c65549-9166-8a86-6ad4-e71543163513 04c65549-9166-8a86-6ad4-e71543163513}"
2026-04-23T00:14:51.639Z [ERROR] server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f: failed to appendEntries to: peer="{Voter server-04c65549-9166-8a86-6ad4-e71543163513 04c65549-9166-8a86-6ad4-e71543163513}" error="failed to connect to peer: 04c65549-9166-8a86-6ad4-e71543163513"
    raft_test.go:459: [INFO] Finished apply without behind follower
2026-04-23T00:14:51.650Z [INFO]  server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f: pipelining replication: peer="{Voter server-04c65549-9166-8a86-6ad4-e71543163513 04c65549-9166-8a86-6ad4-e71543163513}"
2026-04-23T00:14:51.664Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:51.769Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:51.664271835 +0000 UTC m=+8.703873194 (1 nodes), 729ns from start of poll, 317.280967ms from cluster start. Timeout at 2026-04-23 00:14:51.769277872 +0000 UTC m=+8.808879228, 105.006034ms after stability
2026-04-23T00:14:51.769Z [INFO]  server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f: aborting pipeline replication: peer="{Voter server-04c65549-9166-8a86-6ad4-e71543163513 04c65549-9166-8a86-6ad4-e71543163513}"
2026-04-23T00:14:51.769Z [INFO]  server-807a4e18-1c49-3e1f-2d72-b61a5b2e112f: aborting pipeline replication: peer="{Voter server-dd187511-d444-8823-a762-192d7edbb1fc dd187511-d444-8823-a762-192d7edbb1fc}"
--- PASS: TestRaft_BehindFollower (0.42s)
=== RUN   TestRaft_ApplyNonLeader
2026-04-23T00:14:51.771Z [INFO]  server-40c99d01-9195-bb88-24b2-d166d3840bb3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-40c99d01-9195-bb88-24b2-d166d3840bb3 Address:40c99d01-9195-bb88-24b2-d166d3840bb3} {Suffrage:Voter ID:server-0eded02f-ba1f-14bc-77f6-479509dd3bd8 Address:0eded02f-ba1f-14bc-77f6-479509dd3bd8} {Suffrage:Voter ID:server-474a6d10-2aa6-d0d0-a4e6-0691f9f3e562 Address:474a6d10-2aa6-d0d0-a4e6-0691f9f3e562}]"
2026-04-23T00:14:51.771Z [INFO]  server-40c99d01-9195-bb88-24b2-d166d3840bb3: entering follower state: follower="Node at 40c99d01-9195-bb88-24b2-d166d3840bb3 [Follower]" leader-address= leader-id=
2026-04-23T00:14:51.771Z [INFO]  server-0eded02f-ba1f-14bc-77f6-479509dd3bd8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-40c99d01-9195-bb88-24b2-d166d3840bb3 Address:40c99d01-9195-bb88-24b2-d166d3840bb3} {Suffrage:Voter ID:server-0eded02f-ba1f-14bc-77f6-479509dd3bd8 Address:0eded02f-ba1f-14bc-77f6-479509dd3bd8} {Suffrage:Voter ID:server-474a6d10-2aa6-d0d0-a4e6-0691f9f3e562 Address:474a6d10-2aa6-d0d0-a4e6-0691f9f3e562}]"
2026-04-23T00:14:51.771Z [INFO]  server-0eded02f-ba1f-14bc-77f6-479509dd3bd8: entering follower state: follower="Node at 0eded02f-ba1f-14bc-77f6-479509dd3bd8 [Follower]" leader-address= leader-id=
2026-04-23T00:14:51.771Z [INFO]  server-474a6d10-2aa6-d0d0-a4e6-0691f9f3e562: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-40c99d01-9195-bb88-24b2-d166d3840bb3 Address:40c99d01-9195-bb88-24b2-d166d3840bb3} {Suffrage:Voter ID:server-0eded02f-ba1f-14bc-77f6-479509dd3bd8 Address:0eded02f-ba1f-14bc-77f6-479509dd3bd8} {Suffrage:Voter ID:server-474a6d10-2aa6-d0d0-a4e6-0691f9f3e562 Address:474a6d10-2aa6-d0d0-a4e6-0691f9f3e562}]"
2026-04-23T00:14:51.771Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:51.771Z [INFO]  server-474a6d10-2aa6-d0d0-a4e6-0691f9f3e562: entering follower state: follower="Node at 474a6d10-2aa6-d0d0-a4e6-0691f9f3e562 [Follower]" leader-address= leader-id=
2026-04-23T00:14:51.835Z [WARN]  server-474a6d10-2aa6-d0d0-a4e6-0691f9f3e562: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:51.835Z [INFO]  server-474a6d10-2aa6-d0d0-a4e6-0691f9f3e562: entering candidate state: node="Node at 474a6d10-2aa6-d0d0-a4e6-0691f9f3e562 [Candidate]" term=2
2026-04-23T00:14:51.836Z [INFO]  server-474a6d10-2aa6-d0d0-a4e6-0691f9f3e562: election won: term=2 tally=2
2026-04-23T00:14:51.836Z [INFO]  server-474a6d10-2aa6-d0d0-a4e6-0691f9f3e562: entering leader state: leader="Node at 474a6d10-2aa6-d0d0-a4e6-0691f9f3e562 [Leader]"
2026-04-23T00:14:51.836Z [INFO]  server-474a6d10-2aa6-d0d0-a4e6-0691f9f3e562: added peer, starting replication: peer=server-40c99d01-9195-bb88-24b2-d166d3840bb3
2026-04-23T00:14:51.836Z [INFO]  server-474a6d10-2aa6-d0d0-a4e6-0691f9f3e562: added peer, starting replication: peer=server-0eded02f-ba1f-14bc-77f6-479509dd3bd8
2026-04-23T00:14:51.836Z [INFO]  server-474a6d10-2aa6-d0d0-a4e6-0691f9f3e562: pipelining replication: peer="{Voter server-40c99d01-9195-bb88-24b2-d166d3840bb3 40c99d01-9195-bb88-24b2-d166d3840bb3}"
2026-04-23T00:14:51.836Z [INFO]  server-474a6d10-2aa6-d0d0-a4e6-0691f9f3e562: pipelining replication: peer="{Voter server-0eded02f-ba1f-14bc-77f6-479509dd3bd8 0eded02f-ba1f-14bc-77f6-479509dd3bd8}"
2026-04-23T00:14:51.942Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:51.836094499 +0000 UTC m=+8.875695856 (1 nodes), 64.203484ms from start of poll, 64.654451ms from cluster start. Timeout at 2026-04-23 00:14:51.94109565 +0000 UTC m=+8.980697004, 105.001148ms after stability
2026-04-23T00:14:51.942Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:14:52.048Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:14:51.942170432 +0000 UTC m=+8.981771799 (2 nodes), 1.018µs from start of poll, 170.730394ms from cluster start. Timeout at 2026-04-23 00:14:52.047177032 +0000 UTC m=+9.086778384, 105.006585ms after stability
2026-04-23T00:14:52.048Z [INFO]  server-474a6d10-2aa6-d0d0-a4e6-0691f9f3e562: aborting pipeline replication: peer="{Voter server-0eded02f-ba1f-14bc-77f6-479509dd3bd8 0eded02f-ba1f-14bc-77f6-479509dd3bd8}"
2026-04-23T00:14:52.048Z [INFO]  server-474a6d10-2aa6-d0d0-a4e6-0691f9f3e562: aborting pipeline replication: peer="{Voter server-40c99d01-9195-bb88-24b2-d166d3840bb3 40c99d01-9195-bb88-24b2-d166d3840bb3}"
--- PASS: TestRaft_ApplyNonLeader (0.28s)
=== RUN   TestRaft_ApplyConcurrent
2026-04-23T00:14:52.049Z [INFO]  server-d41969cf-f33f-5f7c-9227-e72cc2d1ab0a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d41969cf-f33f-5f7c-9227-e72cc2d1ab0a Address:d41969cf-f33f-5f7c-9227-e72cc2d1ab0a} {Suffrage:Voter ID:server-9e5fb19d-09f6-4cf8-7f07-38397be2b323 Address:9e5fb19d-09f6-4cf8-7f07-38397be2b323} {Suffrage:Voter ID:server-2b961408-98fe-8f76-8795-8b3237410b3c Address:2b961408-98fe-8f76-8795-8b3237410b3c}]"
2026-04-23T00:14:52.049Z [INFO]  server-9e5fb19d-09f6-4cf8-7f07-38397be2b323: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d41969cf-f33f-5f7c-9227-e72cc2d1ab0a Address:d41969cf-f33f-5f7c-9227-e72cc2d1ab0a} {Suffrage:Voter ID:server-9e5fb19d-09f6-4cf8-7f07-38397be2b323 Address:9e5fb19d-09f6-4cf8-7f07-38397be2b323} {Suffrage:Voter ID:server-2b961408-98fe-8f76-8795-8b3237410b3c Address:2b961408-98fe-8f76-8795-8b3237410b3c}]"
2026-04-23T00:14:52.049Z [INFO]  server-d41969cf-f33f-5f7c-9227-e72cc2d1ab0a: entering follower state: follower="Node at d41969cf-f33f-5f7c-9227-e72cc2d1ab0a [Follower]" leader-address= leader-id=
2026-04-23T00:14:52.049Z [INFO]  server-9e5fb19d-09f6-4cf8-7f07-38397be2b323: entering follower state: follower="Node at 9e5fb19d-09f6-4cf8-7f07-38397be2b323 [Follower]" leader-address= leader-id=
2026-04-23T00:14:52.049Z [INFO]  server-2b961408-98fe-8f76-8795-8b3237410b3c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d41969cf-f33f-5f7c-9227-e72cc2d1ab0a Address:d41969cf-f33f-5f7c-9227-e72cc2d1ab0a} {Suffrage:Voter ID:server-9e5fb19d-09f6-4cf8-7f07-38397be2b323 Address:9e5fb19d-09f6-4cf8-7f07-38397be2b323} {Suffrage:Voter ID:server-2b961408-98fe-8f76-8795-8b3237410b3c Address:2b961408-98fe-8f76-8795-8b3237410b3c}]"
2026-04-23T00:14:52.049Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:52.049Z [INFO]  server-2b961408-98fe-8f76-8795-8b3237410b3c: entering follower state: follower="Node at 2b961408-98fe-8f76-8795-8b3237410b3c [Follower]" leader-address= leader-id=
2026-04-23T00:14:52.187Z [WARN]  server-2b961408-98fe-8f76-8795-8b3237410b3c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:52.187Z [INFO]  server-2b961408-98fe-8f76-8795-8b3237410b3c: entering candidate state: node="Node at 2b961408-98fe-8f76-8795-8b3237410b3c [Candidate]" term=2
2026-04-23T00:14:52.187Z [INFO]  server-2b961408-98fe-8f76-8795-8b3237410b3c: election won: term=2 tally=2
2026-04-23T00:14:52.187Z [INFO]  server-2b961408-98fe-8f76-8795-8b3237410b3c: entering leader state: leader="Node at 2b961408-98fe-8f76-8795-8b3237410b3c [Leader]"
2026-04-23T00:14:52.187Z [INFO]  server-2b961408-98fe-8f76-8795-8b3237410b3c: added peer, starting replication: peer=server-d41969cf-f33f-5f7c-9227-e72cc2d1ab0a
2026-04-23T00:14:52.187Z [INFO]  server-2b961408-98fe-8f76-8795-8b3237410b3c: added peer, starting replication: peer=server-9e5fb19d-09f6-4cf8-7f07-38397be2b323
2026-04-23T00:14:52.188Z [INFO]  server-2b961408-98fe-8f76-8795-8b3237410b3c: pipelining replication: peer="{Voter server-9e5fb19d-09f6-4cf8-7f07-38397be2b323 9e5fb19d-09f6-4cf8-7f07-38397be2b323}"
2026-04-23T00:14:52.188Z [INFO]  server-2b961408-98fe-8f76-8795-8b3237410b3c: pipelining replication: peer="{Voter server-d41969cf-f33f-5f7c-9227-e72cc2d1ab0a d41969cf-f33f-5f7c-9227-e72cc2d1ab0a}"
2026-04-23T00:14:52.394Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:52.187951636 +0000 UTC m=+9.227552992 (1 nodes), 138.410436ms from start of poll, 138.76305ms from cluster start. Timeout at 2026-04-23 00:14:52.392955821 +0000 UTC m=+9.432557177, 205.004185ms after stability
2026-04-23T00:14:52.413Z [INFO]  server-2b961408-98fe-8f76-8795-8b3237410b3c: aborting pipeline replication: peer="{Voter server-9e5fb19d-09f6-4cf8-7f07-38397be2b323 9e5fb19d-09f6-4cf8-7f07-38397be2b323}"
2026-04-23T00:14:52.413Z [INFO]  server-2b961408-98fe-8f76-8795-8b3237410b3c: aborting pipeline replication: peer="{Voter server-d41969cf-f33f-5f7c-9227-e72cc2d1ab0a d41969cf-f33f-5f7c-9227-e72cc2d1ab0a}"
--- PASS: TestRaft_ApplyConcurrent (0.37s)
=== RUN   TestRaft_ApplyConcurrent_Timeout
2026-04-23T00:14:52.414Z [INFO]  server-76773c04-b3fa-b764-9e5a-09d03f03ed67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-76773c04-b3fa-b764-9e5a-09d03f03ed67 Address:76773c04-b3fa-b764-9e5a-09d03f03ed67}]"
2026-04-23T00:14:52.414Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:52.414Z [INFO]  server-76773c04-b3fa-b764-9e5a-09d03f03ed67: entering follower state: follower="Node at 76773c04-b3fa-b764-9e5a-09d03f03ed67 [Follower]" leader-address= leader-id=
2026-04-23T00:14:52.566Z [WARN]  server-76773c04-b3fa-b764-9e5a-09d03f03ed67: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:52.567Z [INFO]  server-76773c04-b3fa-b764-9e5a-09d03f03ed67: entering candidate state: node="Node at 76773c04-b3fa-b764-9e5a-09d03f03ed67 [Candidate]" term=2
2026-04-23T00:14:52.567Z [INFO]  server-76773c04-b3fa-b764-9e5a-09d03f03ed67: election won: term=2 tally=1
2026-04-23T00:14:52.567Z [INFO]  server-76773c04-b3fa-b764-9e5a-09d03f03ed67: entering leader state: leader="Node at 76773c04-b3fa-b764-9e5a-09d03f03ed67 [Leader]"
2026-04-23T00:14:52.768Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:52.567130384 +0000 UTC m=+9.606731739 (1 nodes), 152.79995ms from start of poll, 152.936646ms from cluster start. Timeout at 2026-04-23 00:14:52.768134539 +0000 UTC m=+9.807735892, 201.004153ms after stability
--- PASS: TestRaft_ApplyConcurrent_Timeout (0.35s)
=== RUN   TestRaft_JoinNode
2026-04-23T00:14:52.769Z [INFO]  server-afb5e2ec-ca12-ec83-7064-c0cf68ed0b3c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afb5e2ec-ca12-ec83-7064-c0cf68ed0b3c Address:afb5e2ec-ca12-ec83-7064-c0cf68ed0b3c} {Suffrage:Voter ID:server-694516f0-1e07-b31e-8d34-a2a98707d20f Address:694516f0-1e07-b31e-8d34-a2a98707d20f}]"
2026-04-23T00:14:52.769Z [INFO]  server-694516f0-1e07-b31e-8d34-a2a98707d20f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afb5e2ec-ca12-ec83-7064-c0cf68ed0b3c Address:afb5e2ec-ca12-ec83-7064-c0cf68ed0b3c} {Suffrage:Voter ID:server-694516f0-1e07-b31e-8d34-a2a98707d20f Address:694516f0-1e07-b31e-8d34-a2a98707d20f}]"
2026-04-23T00:14:52.769Z [INFO]  server-afb5e2ec-ca12-ec83-7064-c0cf68ed0b3c: entering follower state: follower="Node at afb5e2ec-ca12-ec83-7064-c0cf68ed0b3c [Follower]" leader-address= leader-id=
2026-04-23T00:14:52.769Z [INFO]  server-694516f0-1e07-b31e-8d34-a2a98707d20f: entering follower state: follower="Node at 694516f0-1e07-b31e-8d34-a2a98707d20f [Follower]" leader-address= leader-id=
2026-04-23T00:14:52.769Z [INFO]  server-d8910caf-ecab-eb39-22e6-b7ed732d2d1c: initial configuration: index=0 servers=[]
2026-04-23T00:14:52.769Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:52.769Z [INFO]  server-d8910caf-ecab-eb39-22e6-b7ed732d2d1c: entering follower state: follower="Node at d8910caf-ecab-eb39-22e6-b7ed732d2d1c [Follower]" leader-address= leader-id=
2026-04-23T00:14:52.822Z [WARN]  server-694516f0-1e07-b31e-8d34-a2a98707d20f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:52.822Z [INFO]  server-694516f0-1e07-b31e-8d34-a2a98707d20f: entering candidate state: node="Node at 694516f0-1e07-b31e-8d34-a2a98707d20f [Candidate]" term=2
2026-04-23T00:14:52.823Z [INFO]  server-694516f0-1e07-b31e-8d34-a2a98707d20f: election won: term=2 tally=2
2026-04-23T00:14:52.823Z [INFO]  server-694516f0-1e07-b31e-8d34-a2a98707d20f: entering leader state: leader="Node at 694516f0-1e07-b31e-8d34-a2a98707d20f [Leader]"
2026-04-23T00:14:52.823Z [INFO]  server-694516f0-1e07-b31e-8d34-a2a98707d20f: added peer, starting replication: peer=server-afb5e2ec-ca12-ec83-7064-c0cf68ed0b3c
2026-04-23T00:14:52.823Z [INFO]  server-694516f0-1e07-b31e-8d34-a2a98707d20f: pipelining replication: peer="{Voter server-afb5e2ec-ca12-ec83-7064-c0cf68ed0b3c afb5e2ec-ca12-ec83-7064-c0cf68ed0b3c}"
2026-04-23T00:14:52.857Z [WARN]  server-d8910caf-ecab-eb39-22e6-b7ed732d2d1c: no known peers, aborting election
2026-04-23T00:14:52.928Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:52.82315348 +0000 UTC m=+9.862754838 (1 nodes), 53.621868ms from start of poll, 54.075855ms from cluster start. Timeout at 2026-04-23 00:14:52.928155712 +0000 UTC m=+9.967757063, 105.002225ms after stability
2026-04-23T00:14:52.928Z [INFO]  server-694516f0-1e07-b31e-8d34-a2a98707d20f: updating configuration: command=AddVoter server-id=server-d8910caf-ecab-eb39-22e6-b7ed732d2d1c server-addr=d8910caf-ecab-eb39-22e6-b7ed732d2d1c servers="[{Suffrage:Voter ID:server-afb5e2ec-ca12-ec83-7064-c0cf68ed0b3c Address:afb5e2ec-ca12-ec83-7064-c0cf68ed0b3c} {Suffrage:Voter ID:server-694516f0-1e07-b31e-8d34-a2a98707d20f Address:694516f0-1e07-b31e-8d34-a2a98707d20f} {Suffrage:Voter ID:server-d8910caf-ecab-eb39-22e6-b7ed732d2d1c Address:d8910caf-ecab-eb39-22e6-b7ed732d2d1c}]"
2026-04-23T00:14:52.929Z [INFO]  server-694516f0-1e07-b31e-8d34-a2a98707d20f: added peer, starting replication: peer=server-d8910caf-ecab-eb39-22e6-b7ed732d2d1c
2026-04-23T00:14:52.929Z [WARN]  server-d8910caf-ecab-eb39-22e6-b7ed732d2d1c: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2026-04-23T00:14:52.929Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:52.929Z [WARN]  server-694516f0-1e07-b31e-8d34-a2a98707d20f: appendEntries rejected, sending older logs: peer="{Voter server-d8910caf-ecab-eb39-22e6-b7ed732d2d1c d8910caf-ecab-eb39-22e6-b7ed732d2d1c}" next=1
2026-04-23T00:14:52.929Z [INFO]  server-694516f0-1e07-b31e-8d34-a2a98707d20f: pipelining replication: peer="{Voter server-d8910caf-ecab-eb39-22e6-b7ed732d2d1c d8910caf-ecab-eb39-22e6-b7ed732d2d1c}"
2026-04-23T00:14:53.035Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:52.929229537 +0000 UTC m=+9.968830894 (1 nodes), 657ns from start of poll, 160.151911ms from cluster start. Timeout at 2026-04-23 00:14:53.034233359 +0000 UTC m=+10.073834712, 105.003818ms after stability
2026-04-23T00:14:53.035Z [INFO]  server-694516f0-1e07-b31e-8d34-a2a98707d20f: aborting pipeline replication: peer="{Voter server-d8910caf-ecab-eb39-22e6-b7ed732d2d1c d8910caf-ecab-eb39-22e6-b7ed732d2d1c}"
2026-04-23T00:14:53.035Z [INFO]  server-694516f0-1e07-b31e-8d34-a2a98707d20f: aborting pipeline replication: peer="{Voter server-afb5e2ec-ca12-ec83-7064-c0cf68ed0b3c afb5e2ec-ca12-ec83-7064-c0cf68ed0b3c}"
--- PASS: TestRaft_JoinNode (0.27s)
=== RUN   TestRaft_JoinNode_ConfigStore
2026-04-23T00:14:53.035Z [INFO]  server-265271a2-e895-85c7-7998-ef000f10f14c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-265271a2-e895-85c7-7998-ef000f10f14c Address:265271a2-e895-85c7-7998-ef000f10f14c}]"
2026-04-23T00:14:53.036Z [INFO]  server-265271a2-e895-85c7-7998-ef000f10f14c: entering follower state: follower="Node at 265271a2-e895-85c7-7998-ef000f10f14c [Follower]" leader-address= leader-id=
2026-04-23T00:14:53.036Z [INFO]  server-a69ab326-8a81-f51a-a89f-7a74c55ec1a4: initial configuration: index=0 servers=[]
2026-04-23T00:14:53.036Z [INFO]  server-a69ab326-8a81-f51a-a89f-7a74c55ec1a4: entering follower state: follower="Node at a69ab326-8a81-f51a-a89f-7a74c55ec1a4 [Follower]" leader-address= leader-id=
2026-04-23T00:14:53.036Z [INFO]  server-f9e0c4b9-bbcb-2a1d-e267-fa7f6941a34b: initial configuration: index=0 servers=[]
2026-04-23T00:14:53.036Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:53.036Z [INFO]  server-f9e0c4b9-bbcb-2a1d-e267-fa7f6941a34b: entering follower state: follower="Node at f9e0c4b9-bbcb-2a1d-e267-fa7f6941a34b [Follower]" leader-address= leader-id=
2026-04-23T00:14:53.104Z [WARN]  server-a69ab326-8a81-f51a-a89f-7a74c55ec1a4: no known peers, aborting election
2026-04-23T00:14:53.111Z [WARN]  server-f9e0c4b9-bbcb-2a1d-e267-fa7f6941a34b: no known peers, aborting election
2026-04-23T00:14:53.114Z [WARN]  server-265271a2-e895-85c7-7998-ef000f10f14c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:53.114Z [INFO]  server-265271a2-e895-85c7-7998-ef000f10f14c: entering candidate state: node="Node at 265271a2-e895-85c7-7998-ef000f10f14c [Candidate]" term=2
2026-04-23T00:14:53.115Z [INFO]  server-265271a2-e895-85c7-7998-ef000f10f14c: election won: term=2 tally=1
2026-04-23T00:14:53.115Z [INFO]  server-265271a2-e895-85c7-7998-ef000f10f14c: entering leader state: leader="Node at 265271a2-e895-85c7-7998-ef000f10f14c [Leader]"
2026-04-23T00:14:53.220Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:53.115112787 +0000 UTC m=+10.154714139 (1 nodes), 78.685356ms from start of poll, 79.246476ms from cluster start. Timeout at 2026-04-23 00:14:53.220118522 +0000 UTC m=+10.259719911, 105.005772ms after stability
2026-04-23T00:14:53.220Z [INFO]  server-265271a2-e895-85c7-7998-ef000f10f14c: updating configuration: command=AddVoter server-id=server-a69ab326-8a81-f51a-a89f-7a74c55ec1a4 server-addr=a69ab326-8a81-f51a-a89f-7a74c55ec1a4 servers="[{Suffrage:Voter ID:server-265271a2-e895-85c7-7998-ef000f10f14c Address:265271a2-e895-85c7-7998-ef000f10f14c} {Suffrage:Voter ID:server-a69ab326-8a81-f51a-a89f-7a74c55ec1a4 Address:a69ab326-8a81-f51a-a89f-7a74c55ec1a4}]"
2026-04-23T00:14:53.220Z [INFO]  server-265271a2-e895-85c7-7998-ef000f10f14c: added peer, starting replication: peer=server-a69ab326-8a81-f51a-a89f-7a74c55ec1a4
2026-04-23T00:14:53.220Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:53.220Z [WARN]  server-a69ab326-8a81-f51a-a89f-7a74c55ec1a4: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2026-04-23T00:14:53.220Z [WARN]  server-265271a2-e895-85c7-7998-ef000f10f14c: appendEntries rejected, sending older logs: peer="{Voter server-a69ab326-8a81-f51a-a89f-7a74c55ec1a4 a69ab326-8a81-f51a-a89f-7a74c55ec1a4}" next=1
2026-04-23T00:14:53.221Z [INFO]  server-265271a2-e895-85c7-7998-ef000f10f14c: pipelining replication: peer="{Voter server-a69ab326-8a81-f51a-a89f-7a74c55ec1a4 a69ab326-8a81-f51a-a89f-7a74c55ec1a4}"
2026-04-23T00:14:53.325Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:53.220615395 +0000 UTC m=+10.260216761 (1 nodes), 729ns from start of poll, 184.749098ms from cluster start. Timeout at 2026-04-23 00:14:53.325618401 +0000 UTC m=+10.365219758, 105.002997ms after stability
2026-04-23T00:14:53.325Z [INFO]  server-265271a2-e895-85c7-7998-ef000f10f14c: updating configuration: command=AddVoter server-id=server-f9e0c4b9-bbcb-2a1d-e267-fa7f6941a34b server-addr=f9e0c4b9-bbcb-2a1d-e267-fa7f6941a34b servers="[{Suffrage:Voter ID:server-265271a2-e895-85c7-7998-ef000f10f14c Address:265271a2-e895-85c7-7998-ef000f10f14c} {Suffrage:Voter ID:server-a69ab326-8a81-f51a-a89f-7a74c55ec1a4 Address:a69ab326-8a81-f51a-a89f-7a74c55ec1a4} {Suffrage:Voter ID:server-f9e0c4b9-bbcb-2a1d-e267-fa7f6941a34b Address:f9e0c4b9-bbcb-2a1d-e267-fa7f6941a34b}]"
2026-04-23T00:14:53.326Z [INFO]  server-265271a2-e895-85c7-7998-ef000f10f14c: added peer, starting replication: peer=server-f9e0c4b9-bbcb-2a1d-e267-fa7f6941a34b
2026-04-23T00:14:53.326Z [WARN]  server-f9e0c4b9-bbcb-2a1d-e267-fa7f6941a34b: failed to get previous log: previous-index=4 last-index=0 error="log not found"
2026-04-23T00:14:53.326Z [WARN]  server-265271a2-e895-85c7-7998-ef000f10f14c: appendEntries rejected, sending older logs: peer="{Voter server-f9e0c4b9-bbcb-2a1d-e267-fa7f6941a34b f9e0c4b9-bbcb-2a1d-e267-fa7f6941a34b}" next=1
2026-04-23T00:14:53.326Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:53.326Z [INFO]  server-265271a2-e895-85c7-7998-ef000f10f14c: pipelining replication: peer="{Voter server-f9e0c4b9-bbcb-2a1d-e267-fa7f6941a34b f9e0c4b9-bbcb-2a1d-e267-fa7f6941a34b}"
2026-04-23T00:14:53.431Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:53.326285452 +0000 UTC m=+10.365886818 (1 nodes), 898ns from start of poll, 290.419155ms from cluster start. Timeout at 2026-04-23 00:14:53.43129141 +0000 UTC m=+10.470892762, 105.005944ms after stability
2026-04-23T00:14:53.431Z [INFO]  server-265271a2-e895-85c7-7998-ef000f10f14c: aborting pipeline replication: peer="{Voter server-f9e0c4b9-bbcb-2a1d-e267-fa7f6941a34b f9e0c4b9-bbcb-2a1d-e267-fa7f6941a34b}"
2026-04-23T00:14:53.431Z [INFO]  server-265271a2-e895-85c7-7998-ef000f10f14c: aborting pipeline replication: peer="{Voter server-a69ab326-8a81-f51a-a89f-7a74c55ec1a4 a69ab326-8a81-f51a-a89f-7a74c55ec1a4}"
--- PASS: TestRaft_JoinNode_ConfigStore (0.40s)
=== RUN   TestRaft_RemoveFollower
2026-04-23T00:14:53.432Z [INFO]  server-37253f3d-becd-5e5a-043f-bec09e6eb798: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37253f3d-becd-5e5a-043f-bec09e6eb798 Address:37253f3d-becd-5e5a-043f-bec09e6eb798} {Suffrage:Voter ID:server-c71669d1-cd65-fdfe-0cfd-05e5a8028c33 Address:c71669d1-cd65-fdfe-0cfd-05e5a8028c33} {Suffrage:Voter ID:server-241ec3de-aef1-59f2-71dd-db0bf0339120 Address:241ec3de-aef1-59f2-71dd-db0bf0339120}]"
2026-04-23T00:14:53.432Z [INFO]  server-37253f3d-becd-5e5a-043f-bec09e6eb798: entering follower state: follower="Node at 37253f3d-becd-5e5a-043f-bec09e6eb798 [Follower]" leader-address= leader-id=
2026-04-23T00:14:53.432Z [INFO]  server-c71669d1-cd65-fdfe-0cfd-05e5a8028c33: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37253f3d-becd-5e5a-043f-bec09e6eb798 Address:37253f3d-becd-5e5a-043f-bec09e6eb798} {Suffrage:Voter ID:server-c71669d1-cd65-fdfe-0cfd-05e5a8028c33 Address:c71669d1-cd65-fdfe-0cfd-05e5a8028c33} {Suffrage:Voter ID:server-241ec3de-aef1-59f2-71dd-db0bf0339120 Address:241ec3de-aef1-59f2-71dd-db0bf0339120}]"
2026-04-23T00:14:53.432Z [INFO]  server-241ec3de-aef1-59f2-71dd-db0bf0339120: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37253f3d-becd-5e5a-043f-bec09e6eb798 Address:37253f3d-becd-5e5a-043f-bec09e6eb798} {Suffrage:Voter ID:server-c71669d1-cd65-fdfe-0cfd-05e5a8028c33 Address:c71669d1-cd65-fdfe-0cfd-05e5a8028c33} {Suffrage:Voter ID:server-241ec3de-aef1-59f2-71dd-db0bf0339120 Address:241ec3de-aef1-59f2-71dd-db0bf0339120}]"
2026-04-23T00:14:53.432Z [INFO]  server-c71669d1-cd65-fdfe-0cfd-05e5a8028c33: entering follower state: follower="Node at c71669d1-cd65-fdfe-0cfd-05e5a8028c33 [Follower]" leader-address= leader-id=
2026-04-23T00:14:53.432Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:53.432Z [INFO]  server-241ec3de-aef1-59f2-71dd-db0bf0339120: entering follower state: follower="Node at 241ec3de-aef1-59f2-71dd-db0bf0339120 [Follower]" leader-address= leader-id=
2026-04-23T00:14:53.485Z [WARN]  server-241ec3de-aef1-59f2-71dd-db0bf0339120: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:53.485Z [INFO]  server-241ec3de-aef1-59f2-71dd-db0bf0339120: entering candidate state: node="Node at 241ec3de-aef1-59f2-71dd-db0bf0339120 [Candidate]" term=2
2026-04-23T00:14:53.485Z [INFO]  server-241ec3de-aef1-59f2-71dd-db0bf0339120: election won: term=2 tally=2
2026-04-23T00:14:53.485Z [INFO]  server-241ec3de-aef1-59f2-71dd-db0bf0339120: entering leader state: leader="Node at 241ec3de-aef1-59f2-71dd-db0bf0339120 [Leader]"
2026-04-23T00:14:53.485Z [INFO]  server-241ec3de-aef1-59f2-71dd-db0bf0339120: added peer, starting replication: peer=server-37253f3d-becd-5e5a-043f-bec09e6eb798
2026-04-23T00:14:53.485Z [INFO]  server-241ec3de-aef1-59f2-71dd-db0bf0339120: added peer, starting replication: peer=server-c71669d1-cd65-fdfe-0cfd-05e5a8028c33
2026-04-23T00:14:53.485Z [INFO]  server-241ec3de-aef1-59f2-71dd-db0bf0339120: pipelining replication: peer="{Voter server-37253f3d-becd-5e5a-043f-bec09e6eb798 37253f3d-becd-5e5a-043f-bec09e6eb798}"
2026-04-23T00:14:53.485Z [INFO]  server-241ec3de-aef1-59f2-71dd-db0bf0339120: pipelining replication: peer="{Voter server-c71669d1-cd65-fdfe-0cfd-05e5a8028c33 c71669d1-cd65-fdfe-0cfd-05e5a8028c33}"
2026-04-23T00:14:53.591Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:53.485553866 +0000 UTC m=+10.525155227 (1 nodes), 52.749203ms from start of poll, 53.129982ms from cluster start. Timeout at 2026-04-23 00:14:53.590554828 +0000 UTC m=+10.630156190, 105.000963ms after stability
2026-04-23T00:14:53.596Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:14:53.702Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:14:53.596714388 +0000 UTC m=+10.636315744 (2 nodes), 1.122µs from start of poll, 164.290499ms from cluster start. Timeout at 2026-04-23 00:14:53.701719583 +0000 UTC m=+10.741320935, 105.005191ms after stability
2026-04-23T00:14:53.702Z [INFO]  server-241ec3de-aef1-59f2-71dd-db0bf0339120: updating configuration: command=RemoveServer server-id=server-37253f3d-becd-5e5a-043f-bec09e6eb798 server-addr= servers="[{Suffrage:Voter ID:server-c71669d1-cd65-fdfe-0cfd-05e5a8028c33 Address:c71669d1-cd65-fdfe-0cfd-05e5a8028c33} {Suffrage:Voter ID:server-241ec3de-aef1-59f2-71dd-db0bf0339120 Address:241ec3de-aef1-59f2-71dd-db0bf0339120}]"
2026-04-23T00:14:53.702Z [INFO]  server-241ec3de-aef1-59f2-71dd-db0bf0339120: removed peer, stopping replication: peer=server-37253f3d-becd-5e5a-043f-bec09e6eb798 last-index=3
2026-04-23T00:14:53.702Z [INFO]  server-241ec3de-aef1-59f2-71dd-db0bf0339120: aborting pipeline replication: peer="{Voter server-37253f3d-becd-5e5a-043f-bec09e6eb798 37253f3d-becd-5e5a-043f-bec09e6eb798}"
2026-04-23T00:14:53.777Z [WARN]  server-37253f3d-becd-5e5a-043f-bec09e6eb798: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2026-04-23T00:14:53.807Z [INFO]  server-241ec3de-aef1-59f2-71dd-db0bf0339120: aborting pipeline replication: peer="{Voter server-c71669d1-cd65-fdfe-0cfd-05e5a8028c33 c71669d1-cd65-fdfe-0cfd-05e5a8028c33}"
--- PASS: TestRaft_RemoveFollower (0.38s)
=== RUN   TestRaft_RemoveLeader
2026-04-23T00:14:53.808Z [INFO]  server-114d184c-8c08-8203-3c03-ac980b466814: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-114d184c-8c08-8203-3c03-ac980b466814 Address:114d184c-8c08-8203-3c03-ac980b466814} {Suffrage:Voter ID:server-3d49d084-c7ba-0bbd-17c8-905f20e25bea Address:3d49d084-c7ba-0bbd-17c8-905f20e25bea} {Suffrage:Voter ID:server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256 Address:39f5f4f9-b710-4bbe-22fc-4a70e10d4256}]"
2026-04-23T00:14:53.809Z [INFO]  server-114d184c-8c08-8203-3c03-ac980b466814: entering follower state: follower="Node at 114d184c-8c08-8203-3c03-ac980b466814 [Follower]" leader-address= leader-id=
2026-04-23T00:14:53.809Z [INFO]  server-3d49d084-c7ba-0bbd-17c8-905f20e25bea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-114d184c-8c08-8203-3c03-ac980b466814 Address:114d184c-8c08-8203-3c03-ac980b466814} {Suffrage:Voter ID:server-3d49d084-c7ba-0bbd-17c8-905f20e25bea Address:3d49d084-c7ba-0bbd-17c8-905f20e25bea} {Suffrage:Voter ID:server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256 Address:39f5f4f9-b710-4bbe-22fc-4a70e10d4256}]"
2026-04-23T00:14:53.809Z [INFO]  server-3d49d084-c7ba-0bbd-17c8-905f20e25bea: entering follower state: follower="Node at 3d49d084-c7ba-0bbd-17c8-905f20e25bea [Follower]" leader-address= leader-id=
2026-04-23T00:14:53.809Z [INFO]  server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-114d184c-8c08-8203-3c03-ac980b466814 Address:114d184c-8c08-8203-3c03-ac980b466814} {Suffrage:Voter ID:server-3d49d084-c7ba-0bbd-17c8-905f20e25bea Address:3d49d084-c7ba-0bbd-17c8-905f20e25bea} {Suffrage:Voter ID:server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256 Address:39f5f4f9-b710-4bbe-22fc-4a70e10d4256}]"
2026-04-23T00:14:53.809Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:53.809Z [INFO]  server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256: entering follower state: follower="Node at 39f5f4f9-b710-4bbe-22fc-4a70e10d4256 [Follower]" leader-address= leader-id=
2026-04-23T00:14:53.872Z [WARN]  server-3d49d084-c7ba-0bbd-17c8-905f20e25bea: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:53.872Z [INFO]  server-3d49d084-c7ba-0bbd-17c8-905f20e25bea: entering candidate state: node="Node at 3d49d084-c7ba-0bbd-17c8-905f20e25bea [Candidate]" term=2
2026-04-23T00:14:53.872Z [INFO]  server-3d49d084-c7ba-0bbd-17c8-905f20e25bea: election won: term=2 tally=2
2026-04-23T00:14:53.872Z [INFO]  server-3d49d084-c7ba-0bbd-17c8-905f20e25bea: entering leader state: leader="Node at 3d49d084-c7ba-0bbd-17c8-905f20e25bea [Leader]"
2026-04-23T00:14:53.872Z [INFO]  server-3d49d084-c7ba-0bbd-17c8-905f20e25bea: added peer, starting replication: peer=server-114d184c-8c08-8203-3c03-ac980b466814
2026-04-23T00:14:53.872Z [INFO]  server-3d49d084-c7ba-0bbd-17c8-905f20e25bea: added peer, starting replication: peer=server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256
2026-04-23T00:14:53.872Z [INFO]  server-3d49d084-c7ba-0bbd-17c8-905f20e25bea: pipelining replication: peer="{Voter server-114d184c-8c08-8203-3c03-ac980b466814 114d184c-8c08-8203-3c03-ac980b466814}"
2026-04-23T00:14:53.873Z [INFO]  server-3d49d084-c7ba-0bbd-17c8-905f20e25bea: pipelining replication: peer="{Voter server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256 39f5f4f9-b710-4bbe-22fc-4a70e10d4256}"
2026-04-23T00:14:53.978Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:53.872898121 +0000 UTC m=+10.912499488 (1 nodes), 63.707193ms from start of poll, 64.014971ms from cluster start. Timeout at 2026-04-23 00:14:53.977898995 +0000 UTC m=+11.017500346, 105.000858ms after stability
2026-04-23T00:14:53.983Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:14:54.089Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:14:53.983942343 +0000 UTC m=+11.023543722 (2 nodes), 1.869µs from start of poll, 175.059205ms from cluster start. Timeout at 2026-04-23 00:14:54.088948462 +0000 UTC m=+11.128549815, 105.006093ms after stability
2026-04-23T00:14:54.089Z [INFO]  server-3d49d084-c7ba-0bbd-17c8-905f20e25bea: updating configuration: command=RemoveServer server-id=server-3d49d084-c7ba-0bbd-17c8-905f20e25bea server-addr= servers="[{Suffrage:Voter ID:server-114d184c-8c08-8203-3c03-ac980b466814 Address:114d184c-8c08-8203-3c03-ac980b466814} {Suffrage:Voter ID:server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256 Address:39f5f4f9-b710-4bbe-22fc-4a70e10d4256}]"
2026-04-23T00:14:54.089Z [INFO]  server-3d49d084-c7ba-0bbd-17c8-905f20e25bea: removed ourself, shutting down
2026-04-23T00:14:54.089Z [INFO]  server-3d49d084-c7ba-0bbd-17c8-905f20e25bea: aborting pipeline replication: peer="{Voter server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256 39f5f4f9-b710-4bbe-22fc-4a70e10d4256}"
2026-04-23T00:14:54.089Z [INFO]  server-3d49d084-c7ba-0bbd-17c8-905f20e25bea: aborting pipeline replication: peer="{Voter server-114d184c-8c08-8203-3c03-ac980b466814 114d184c-8c08-8203-3c03-ac980b466814}"
2026-04-23T00:14:54.170Z [WARN]  server-114d184c-8c08-8203-3c03-ac980b466814: heartbeat timeout reached, starting election: last-leader-addr=3d49d084-c7ba-0bbd-17c8-905f20e25bea last-leader-id=server-3d49d084-c7ba-0bbd-17c8-905f20e25bea
2026-04-23T00:14:54.170Z [INFO]  server-114d184c-8c08-8203-3c03-ac980b466814: entering candidate state: node="Node at 114d184c-8c08-8203-3c03-ac980b466814 [Candidate]" term=3
2026-04-23T00:14:54.171Z [WARN]  server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256: rejecting vote request since we have a leader: from=114d184c-8c08-8203-3c03-ac980b466814 leader=3d49d084-c7ba-0bbd-17c8-905f20e25bea leader-id=server-3d49d084-c7ba-0bbd-17c8-905f20e25bea
2026-04-23T00:14:54.189Z [WARN]  server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256: heartbeat timeout reached, starting election: last-leader-addr=3d49d084-c7ba-0bbd-17c8-905f20e25bea last-leader-id=server-3d49d084-c7ba-0bbd-17c8-905f20e25bea
2026-04-23T00:14:54.189Z [INFO]  server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256: entering candidate state: node="Node at 39f5f4f9-b710-4bbe-22fc-4a70e10d4256 [Candidate]" term=3
2026-04-23T00:14:54.189Z [INFO]  server-114d184c-8c08-8203-3c03-ac980b466814: duplicate requestVote for same term: term=3
2026-04-23T00:14:54.250Z [WARN]  server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256: Election timeout reached, restarting election
2026-04-23T00:14:54.250Z [INFO]  server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256: entering candidate state: node="Node at 39f5f4f9-b710-4bbe-22fc-4a70e10d4256 [Candidate]" term=4
2026-04-23T00:14:54.250Z [INFO]  server-114d184c-8c08-8203-3c03-ac980b466814: entering follower state: follower="Node at 114d184c-8c08-8203-3c03-ac980b466814 [Follower]" leader-address= leader-id=
2026-04-23T00:14:54.251Z [INFO]  server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256: election won: term=4 tally=2
2026-04-23T00:14:54.251Z [INFO]  server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256: entering leader state: leader="Node at 39f5f4f9-b710-4bbe-22fc-4a70e10d4256 [Leader]"
2026-04-23T00:14:54.251Z [INFO]  server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256: added peer, starting replication: peer=server-114d184c-8c08-8203-3c03-ac980b466814
2026-04-23T00:14:54.251Z [INFO]  server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256: pipelining replication: peer="{Voter server-114d184c-8c08-8203-3c03-ac980b466814 114d184c-8c08-8203-3c03-ac980b466814}"
2026-04-23T00:14:54.300Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:54.406Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:54.30030625 +0000 UTC m=+11.339907619 (1 nodes), 63.064µs from start of poll, 491.423102ms from cluster start. Timeout at 2026-04-23 00:14:54.405311088 +0000 UTC m=+11.444912444, 105.004825ms after stability
2026-04-23T00:14:54.406Z [INFO]  server-39f5f4f9-b710-4bbe-22fc-4a70e10d4256: aborting pipeline replication: peer="{Voter server-114d184c-8c08-8203-3c03-ac980b466814 114d184c-8c08-8203-3c03-ac980b466814}"
--- PASS: TestRaft_RemoveLeader (0.60s)
=== RUN   TestRaft_RemoveLeader_NoShutdown
2026-04-23T00:14:54.407Z [INFO]  server-b371738e-2398-fce2-5614-a74a84504a8a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b371738e-2398-fce2-5614-a74a84504a8a Address:b371738e-2398-fce2-5614-a74a84504a8a} {Suffrage:Voter ID:server-dad13e95-4ee8-63c8-d82d-27bdd48126e7 Address:dad13e95-4ee8-63c8-d82d-27bdd48126e7} {Suffrage:Voter ID:server-45198342-fc91-5c1b-263a-330ce576249f Address:45198342-fc91-5c1b-263a-330ce576249f}]"
2026-04-23T00:14:54.407Z [INFO]  server-b371738e-2398-fce2-5614-a74a84504a8a: entering follower state: follower="Node at b371738e-2398-fce2-5614-a74a84504a8a [Follower]" leader-address= leader-id=
2026-04-23T00:14:54.407Z [INFO]  server-dad13e95-4ee8-63c8-d82d-27bdd48126e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b371738e-2398-fce2-5614-a74a84504a8a Address:b371738e-2398-fce2-5614-a74a84504a8a} {Suffrage:Voter ID:server-dad13e95-4ee8-63c8-d82d-27bdd48126e7 Address:dad13e95-4ee8-63c8-d82d-27bdd48126e7} {Suffrage:Voter ID:server-45198342-fc91-5c1b-263a-330ce576249f Address:45198342-fc91-5c1b-263a-330ce576249f}]"
2026-04-23T00:14:54.407Z [INFO]  server-dad13e95-4ee8-63c8-d82d-27bdd48126e7: entering follower state: follower="Node at dad13e95-4ee8-63c8-d82d-27bdd48126e7 [Follower]" leader-address= leader-id=
2026-04-23T00:14:54.407Z [INFO]  server-45198342-fc91-5c1b-263a-330ce576249f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b371738e-2398-fce2-5614-a74a84504a8a Address:b371738e-2398-fce2-5614-a74a84504a8a} {Suffrage:Voter ID:server-dad13e95-4ee8-63c8-d82d-27bdd48126e7 Address:dad13e95-4ee8-63c8-d82d-27bdd48126e7} {Suffrage:Voter ID:server-45198342-fc91-5c1b-263a-330ce576249f Address:45198342-fc91-5c1b-263a-330ce576249f}]"
2026-04-23T00:14:54.407Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:14:54.407Z [INFO]  server-45198342-fc91-5c1b-263a-330ce576249f: entering follower state: follower="Node at 45198342-fc91-5c1b-263a-330ce576249f [Follower]" leader-address= leader-id=
2026-04-23T00:14:54.474Z [WARN]  server-dad13e95-4ee8-63c8-d82d-27bdd48126e7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:54.474Z [INFO]  server-dad13e95-4ee8-63c8-d82d-27bdd48126e7: entering candidate state: node="Node at dad13e95-4ee8-63c8-d82d-27bdd48126e7 [Candidate]" term=2
2026-04-23T00:14:54.474Z [INFO]  server-dad13e95-4ee8-63c8-d82d-27bdd48126e7: election won: term=2 tally=2
2026-04-23T00:14:54.474Z [INFO]  server-dad13e95-4ee8-63c8-d82d-27bdd48126e7: entering leader state: leader="Node at dad13e95-4ee8-63c8-d82d-27bdd48126e7 [Leader]"
2026-04-23T00:14:54.474Z [INFO]  server-dad13e95-4ee8-63c8-d82d-27bdd48126e7: added peer, starting replication: peer=server-b371738e-2398-fce2-5614-a74a84504a8a
2026-04-23T00:14:54.474Z [INFO]  server-dad13e95-4ee8-63c8-d82d-27bdd48126e7: added peer, starting replication: peer=server-45198342-fc91-5c1b-263a-330ce576249f
2026-04-23T00:14:54.475Z [INFO]  server-dad13e95-4ee8-63c8-d82d-27bdd48126e7: pipelining replication: peer="{Voter server-b371738e-2398-fce2-5614-a74a84504a8a b371738e-2398-fce2-5614-a74a84504a8a}"
2026-04-23T00:14:54.475Z [INFO]  server-dad13e95-4ee8-63c8-d82d-27bdd48126e7: pipelining replication: peer="{Voter server-45198342-fc91-5c1b-263a-330ce576249f 45198342-fc91-5c1b-263a-330ce576249f}"
2026-04-23T00:14:54.580Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:14:54.47496206 +0000 UTC m=+11.514563469 (2 nodes), 67.241867ms from start of poll, 67.538276ms from cluster start. Timeout at 2026-04-23 00:14:54.57996603 +0000 UTC m=+11.619567381, 105.003912ms after stability
2026-04-23T00:14:54.580Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:54.686Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:54.58049185 +0000 UTC m=+11.620093207 (1 nodes), 1.998µs from start of poll, 173.068014ms from cluster start. Timeout at 2026-04-23 00:14:54.685495296 +0000 UTC m=+11.725096664, 105.003457ms after stability
2026-04-23T00:14:54.689Z [INFO]  server-dad13e95-4ee8-63c8-d82d-27bdd48126e7: updating configuration: command=RemoveServer server-id=server-dad13e95-4ee8-63c8-d82d-27bdd48126e7 server-addr= servers="[{Suffrage:Voter ID:server-b371738e-2398-fce2-5614-a74a84504a8a Address:b371738e-2398-fce2-5614-a74a84504a8a} {Suffrage:Voter ID:server-45198342-fc91-5c1b-263a-330ce576249f Address:45198342-fc91-5c1b-263a-330ce576249f}]"
2026-04-23T00:14:54.689Z [INFO]  server-dad13e95-4ee8-63c8-d82d-27bdd48126e7: removed ourself, transitioning to follower
2026-04-23T00:14:54.689Z [INFO]  server-dad13e95-4ee8-63c8-d82d-27bdd48126e7: entering follower state: follower="Node at dad13e95-4ee8-63c8-d82d-27bdd48126e7 [Follower]" leader-address= leader-id=
2026-04-23T00:14:54.689Z [INFO]  server-dad13e95-4ee8-63c8-d82d-27bdd48126e7: aborting pipeline replication: peer="{Voter server-45198342-fc91-5c1b-263a-330ce576249f 45198342-fc91-5c1b-263a-330ce576249f}"
2026-04-23T00:14:54.689Z [INFO]  server-dad13e95-4ee8-63c8-d82d-27bdd48126e7: aborting pipeline replication: peer="{Voter server-b371738e-2398-fce2-5614-a74a84504a8a b371738e-2398-fce2-5614-a74a84504a8a}"
2026-04-23T00:14:54.740Z [WARN]  server-b371738e-2398-fce2-5614-a74a84504a8a: heartbeat timeout reached, starting election: last-leader-addr=dad13e95-4ee8-63c8-d82d-27bdd48126e7 last-leader-id=server-dad13e95-4ee8-63c8-d82d-27bdd48126e7
2026-04-23T00:14:54.740Z [INFO]  server-b371738e-2398-fce2-5614-a74a84504a8a: entering candidate state: node="Node at b371738e-2398-fce2-5614-a74a84504a8a [Candidate]" term=3
2026-04-23T00:14:54.740Z [WARN]  server-45198342-fc91-5c1b-263a-330ce576249f: rejecting vote request since we have a leader: from=b371738e-2398-fce2-5614-a74a84504a8a leader=dad13e95-4ee8-63c8-d82d-27bdd48126e7 leader-id=server-dad13e95-4ee8-63c8-d82d-27bdd48126e7
2026-04-23T00:14:54.741Z [WARN]  server-45198342-fc91-5c1b-263a-330ce576249f: heartbeat timeout reached, starting election: last-leader-addr=dad13e95-4ee8-63c8-d82d-27bdd48126e7 last-leader-id=server-dad13e95-4ee8-63c8-d82d-27bdd48126e7
2026-04-23T00:14:54.741Z [INFO]  server-45198342-fc91-5c1b-263a-330ce576249f: entering candidate state: node="Node at 45198342-fc91-5c1b-263a-330ce576249f [Candidate]" term=3
2026-04-23T00:14:54.742Z [INFO]  server-b371738e-2398-fce2-5614-a74a84504a8a: duplicate requestVote for same term: term=3
2026-04-23T00:14:54.783Z [WARN]  server-dad13e95-4ee8-63c8-d82d-27bdd48126e7: not part of stable configuration, aborting election
2026-04-23T00:14:54.795Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:54.798Z [WARN]  server-b371738e-2398-fce2-5614-a74a84504a8a: Election timeout reached, restarting election
2026-04-23T00:14:54.798Z [INFO]  server-b371738e-2398-fce2-5614-a74a84504a8a: entering candidate state: node="Node at b371738e-2398-fce2-5614-a74a84504a8a [Candidate]" term=4
2026-04-23T00:14:54.798Z [INFO]  server-45198342-fc91-5c1b-263a-330ce576249f: entering follower state: follower="Node at 45198342-fc91-5c1b-263a-330ce576249f [Follower]" leader-address= leader-id=
2026-04-23T00:14:54.798Z [INFO]  server-b371738e-2398-fce2-5614-a74a84504a8a: election won: term=4 tally=2
2026-04-23T00:14:54.799Z [INFO]  server-b371738e-2398-fce2-5614-a74a84504a8a: entering leader state: leader="Node at b371738e-2398-fce2-5614-a74a84504a8a [Leader]"
2026-04-23T00:14:54.799Z [INFO]  server-b371738e-2398-fce2-5614-a74a84504a8a: added peer, starting replication: peer=server-45198342-fc91-5c1b-263a-330ce576249f
2026-04-23T00:14:54.799Z [INFO]  server-b371738e-2398-fce2-5614-a74a84504a8a: pipelining replication: peer="{Voter server-45198342-fc91-5c1b-263a-330ce576249f 45198342-fc91-5c1b-263a-330ce576249f}"
2026-04-23T00:14:54.904Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:54.799100505 +0000 UTC m=+11.838701857 (1 nodes), 3.418827ms from start of poll, 391.676664ms from cluster start. Timeout at 2026-04-23 00:14:54.904102823 +0000 UTC m=+11.943704179, 105.002322ms after stability
2026-04-23T00:14:55.010Z [INFO]  server-b371738e-2398-fce2-5614-a74a84504a8a: aborting pipeline replication: peer="{Voter server-45198342-fc91-5c1b-263a-330ce576249f 45198342-fc91-5c1b-263a-330ce576249f}"
--- PASS: TestRaft_RemoveLeader_NoShutdown (0.60s)
=== RUN   TestRaft_RemoveFollower_SplitCluster
2026-04-23T00:14:55.012Z [INFO]  server-c7d8d08e-b9ad-6700-140e-042977d76505: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7d8d08e-b9ad-6700-140e-042977d76505 Address:c7d8d08e-b9ad-6700-140e-042977d76505} {Suffrage:Voter ID:server-39f0e1ee-d7dd-16cd-90fb-338521924292 Address:39f0e1ee-d7dd-16cd-90fb-338521924292} {Suffrage:Voter ID:server-b3a1f851-421e-27d5-05ef-c3abb828d08c Address:b3a1f851-421e-27d5-05ef-c3abb828d08c} {Suffrage:Voter ID:server-5b74cfe4-9272-1af1-6cec-d3ee5e67263b Address:5b74cfe4-9272-1af1-6cec-d3ee5e67263b}]"
2026-04-23T00:14:55.012Z [INFO]  server-c7d8d08e-b9ad-6700-140e-042977d76505: entering follower state: follower="Node at c7d8d08e-b9ad-6700-140e-042977d76505 [Follower]" leader-address= leader-id=
2026-04-23T00:14:55.012Z [INFO]  server-39f0e1ee-d7dd-16cd-90fb-338521924292: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7d8d08e-b9ad-6700-140e-042977d76505 Address:c7d8d08e-b9ad-6700-140e-042977d76505} {Suffrage:Voter ID:server-39f0e1ee-d7dd-16cd-90fb-338521924292 Address:39f0e1ee-d7dd-16cd-90fb-338521924292} {Suffrage:Voter ID:server-b3a1f851-421e-27d5-05ef-c3abb828d08c Address:b3a1f851-421e-27d5-05ef-c3abb828d08c} {Suffrage:Voter ID:server-5b74cfe4-9272-1af1-6cec-d3ee5e67263b Address:5b74cfe4-9272-1af1-6cec-d3ee5e67263b}]"
2026-04-23T00:14:55.012Z [INFO]  server-39f0e1ee-d7dd-16cd-90fb-338521924292: entering follower state: follower="Node at 39f0e1ee-d7dd-16cd-90fb-338521924292 [Follower]" leader-address= leader-id=
2026-04-23T00:14:55.012Z [INFO]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7d8d08e-b9ad-6700-140e-042977d76505 Address:c7d8d08e-b9ad-6700-140e-042977d76505} {Suffrage:Voter ID:server-39f0e1ee-d7dd-16cd-90fb-338521924292 Address:39f0e1ee-d7dd-16cd-90fb-338521924292} {Suffrage:Voter ID:server-b3a1f851-421e-27d5-05ef-c3abb828d08c Address:b3a1f851-421e-27d5-05ef-c3abb828d08c} {Suffrage:Voter ID:server-5b74cfe4-9272-1af1-6cec-d3ee5e67263b Address:5b74cfe4-9272-1af1-6cec-d3ee5e67263b}]"
2026-04-23T00:14:55.012Z [INFO]  server-5b74cfe4-9272-1af1-6cec-d3ee5e67263b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7d8d08e-b9ad-6700-140e-042977d76505 Address:c7d8d08e-b9ad-6700-140e-042977d76505} {Suffrage:Voter ID:server-39f0e1ee-d7dd-16cd-90fb-338521924292 Address:39f0e1ee-d7dd-16cd-90fb-338521924292} {Suffrage:Voter ID:server-b3a1f851-421e-27d5-05ef-c3abb828d08c Address:b3a1f851-421e-27d5-05ef-c3abb828d08c} {Suffrage:Voter ID:server-5b74cfe4-9272-1af1-6cec-d3ee5e67263b Address:5b74cfe4-9272-1af1-6cec-d3ee5e67263b}]"
2026-04-23T00:14:55.012Z [INFO]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: entering follower state: follower="Node at b3a1f851-421e-27d5-05ef-c3abb828d08c [Follower]" leader-address= leader-id=
2026-04-23T00:14:55.013Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:55.013Z [INFO]  server-5b74cfe4-9272-1af1-6cec-d3ee5e67263b: entering follower state: follower="Node at 5b74cfe4-9272-1af1-6cec-d3ee5e67263b [Follower]" leader-address= leader-id=
2026-04-23T00:14:55.078Z [WARN]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:55.078Z [INFO]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: entering candidate state: node="Node at b3a1f851-421e-27d5-05ef-c3abb828d08c [Candidate]" term=2
2026-04-23T00:14:55.079Z [INFO]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: election won: term=2 tally=3
2026-04-23T00:14:55.079Z [INFO]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: entering leader state: leader="Node at b3a1f851-421e-27d5-05ef-c3abb828d08c [Leader]"
2026-04-23T00:14:55.079Z [INFO]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: added peer, starting replication: peer=server-c7d8d08e-b9ad-6700-140e-042977d76505
2026-04-23T00:14:55.079Z [INFO]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: added peer, starting replication: peer=server-39f0e1ee-d7dd-16cd-90fb-338521924292
2026-04-23T00:14:55.079Z [INFO]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: added peer, starting replication: peer=server-5b74cfe4-9272-1af1-6cec-d3ee5e67263b
2026-04-23T00:14:55.079Z [INFO]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: pipelining replication: peer="{Voter server-c7d8d08e-b9ad-6700-140e-042977d76505 c7d8d08e-b9ad-6700-140e-042977d76505}"
2026-04-23T00:14:55.079Z [INFO]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: pipelining replication: peer="{Voter server-5b74cfe4-9272-1af1-6cec-d3ee5e67263b 5b74cfe4-9272-1af1-6cec-d3ee5e67263b}"
2026-04-23T00:14:55.079Z [INFO]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: pipelining replication: peer="{Voter server-39f0e1ee-d7dd-16cd-90fb-338521924292 39f0e1ee-d7dd-16cd-90fb-338521924292}"
2026-04-23T00:14:55.184Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:55.079095492 +0000 UTC m=+12.118696847 (1 nodes), 66.078286ms from start of poll, 66.555995ms from cluster start. Timeout at 2026-04-23 00:14:55.184098445 +0000 UTC m=+12.223699802, 105.002955ms after stability
2026-04-23T00:14:55.289Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:14:55.395Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:14:55.289978896 +0000 UTC m=+12.329580258 (3 nodes), 1.453µs from start of poll, 277.439406ms from cluster start. Timeout at 2026-04-23 00:14:55.394984817 +0000 UTC m=+12.434586176, 105.005918ms after stability
2026-04-23T00:14:55.395Z [INFO]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: updating configuration: command=RemoveServer server-id=server-5b74cfe4-9272-1af1-6cec-d3ee5e67263b server-addr= servers="[{Suffrage:Voter ID:server-c7d8d08e-b9ad-6700-140e-042977d76505 Address:c7d8d08e-b9ad-6700-140e-042977d76505} {Suffrage:Voter ID:server-39f0e1ee-d7dd-16cd-90fb-338521924292 Address:39f0e1ee-d7dd-16cd-90fb-338521924292} {Suffrage:Voter ID:server-b3a1f851-421e-27d5-05ef-c3abb828d08c Address:b3a1f851-421e-27d5-05ef-c3abb828d08c}]"
2026-04-23T00:14:55.395Z [INFO]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: removed peer, stopping replication: peer=server-5b74cfe4-9272-1af1-6cec-d3ee5e67263b last-index=3
2026-04-23T00:14:55.395Z [ERROR] server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to pipeline appendEntries: peer="{Voter server-39f0e1ee-d7dd-16cd-90fb-338521924292 39f0e1ee-d7dd-16cd-90fb-338521924292}" error="append pipeline closed"
2026-04-23T00:14:55.395Z [INFO]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: aborting pipeline replication: peer="{Voter server-39f0e1ee-d7dd-16cd-90fb-338521924292 39f0e1ee-d7dd-16cd-90fb-338521924292}"
2026-04-23T00:14:55.395Z [ERROR] server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to pipeline appendEntries: peer="{Voter server-c7d8d08e-b9ad-6700-140e-042977d76505 c7d8d08e-b9ad-6700-140e-042977d76505}" error="append pipeline closed"
2026-04-23T00:14:55.395Z [INFO]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: aborting pipeline replication: peer="{Voter server-c7d8d08e-b9ad-6700-140e-042977d76505 c7d8d08e-b9ad-6700-140e-042977d76505}"
2026-04-23T00:14:55.395Z [INFO]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: aborting pipeline replication: peer="{Voter server-5b74cfe4-9272-1af1-6cec-d3ee5e67263b 5b74cfe4-9272-1af1-6cec-d3ee5e67263b}"
2026-04-23T00:14:55.397Z [ERROR] server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to heartbeat to: peer=39f0e1ee-d7dd-16cd-90fb-338521924292 backoff time=10ms error="failed to connect to peer: 39f0e1ee-d7dd-16cd-90fb-338521924292"
2026-04-23T00:14:55.398Z [ERROR] server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to heartbeat to: peer=c7d8d08e-b9ad-6700-140e-042977d76505 backoff time=10ms error="failed to connect to peer: c7d8d08e-b9ad-6700-140e-042977d76505"
2026-04-23T00:14:55.402Z [ERROR] server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to appendEntries to: peer="{Voter server-39f0e1ee-d7dd-16cd-90fb-338521924292 39f0e1ee-d7dd-16cd-90fb-338521924292}" error="failed to connect to peer: 39f0e1ee-d7dd-16cd-90fb-338521924292"
2026-04-23T00:14:55.403Z [ERROR] server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to appendEntries to: peer="{Voter server-c7d8d08e-b9ad-6700-140e-042977d76505 c7d8d08e-b9ad-6700-140e-042977d76505}" error="failed to connect to peer: c7d8d08e-b9ad-6700-140e-042977d76505"
2026-04-23T00:14:55.418Z [ERROR] server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to heartbeat to: peer=39f0e1ee-d7dd-16cd-90fb-338521924292 backoff time=10ms error="failed to connect to peer: 39f0e1ee-d7dd-16cd-90fb-338521924292"
2026-04-23T00:14:55.419Z [ERROR] server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to appendEntries to: peer="{Voter server-39f0e1ee-d7dd-16cd-90fb-338521924292 39f0e1ee-d7dd-16cd-90fb-338521924292}" error="failed to connect to peer: 39f0e1ee-d7dd-16cd-90fb-338521924292"
2026-04-23T00:14:55.419Z [ERROR] server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to heartbeat to: peer=c7d8d08e-b9ad-6700-140e-042977d76505 backoff time=10ms error="failed to connect to peer: c7d8d08e-b9ad-6700-140e-042977d76505"
2026-04-23T00:14:55.422Z [ERROR] server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to appendEntries to: peer="{Voter server-c7d8d08e-b9ad-6700-140e-042977d76505 c7d8d08e-b9ad-6700-140e-042977d76505}" error="failed to connect to peer: c7d8d08e-b9ad-6700-140e-042977d76505"
2026-04-23T00:14:55.437Z [ERROR] server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to heartbeat to: peer=39f0e1ee-d7dd-16cd-90fb-338521924292 backoff time=10ms error="failed to connect to peer: 39f0e1ee-d7dd-16cd-90fb-338521924292"
2026-04-23T00:14:55.437Z [ERROR] server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to appendEntries to: peer="{Voter server-39f0e1ee-d7dd-16cd-90fb-338521924292 39f0e1ee-d7dd-16cd-90fb-338521924292}" error="failed to connect to peer: 39f0e1ee-d7dd-16cd-90fb-338521924292"
2026-04-23T00:14:55.438Z [ERROR] server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to appendEntries to: peer="{Voter server-c7d8d08e-b9ad-6700-140e-042977d76505 c7d8d08e-b9ad-6700-140e-042977d76505}" error="failed to connect to peer: c7d8d08e-b9ad-6700-140e-042977d76505"
2026-04-23T00:14:55.439Z [ERROR] server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to heartbeat to: peer=c7d8d08e-b9ad-6700-140e-042977d76505 backoff time=10ms error="failed to connect to peer: c7d8d08e-b9ad-6700-140e-042977d76505"
2026-04-23T00:14:55.442Z [WARN]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to contact: server-id=server-39f0e1ee-d7dd-16cd-90fb-338521924292 time=50.9124ms
2026-04-23T00:14:55.453Z [WARN]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to contact: server-id=server-c7d8d08e-b9ad-6700-140e-042977d76505 time=57.324781ms
2026-04-23T00:14:55.453Z [WARN]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to contact: server-id=server-39f0e1ee-d7dd-16cd-90fb-338521924292 time=61.57886ms
2026-04-23T00:14:55.453Z [WARN]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to contact quorum of nodes, stepping down
2026-04-23T00:14:55.453Z [INFO]  server-b3a1f851-421e-27d5-05ef-c3abb828d08c: entering follower state: follower="Node at b3a1f851-421e-27d5-05ef-c3abb828d08c [Follower]" leader-address= leader-id=
2026-04-23T00:14:55.453Z [ERROR] server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to appendEntries to: peer="{Voter server-c7d8d08e-b9ad-6700-140e-042977d76505 c7d8d08e-b9ad-6700-140e-042977d76505}" error="failed to connect to peer: c7d8d08e-b9ad-6700-140e-042977d76505"
2026-04-23T00:14:55.453Z [ERROR] server-b3a1f851-421e-27d5-05ef-c3abb828d08c: failed to appendEntries to: peer="{Voter server-39f0e1ee-d7dd-16cd-90fb-338521924292 39f0e1ee-d7dd-16cd-90fb-338521924292}" error="failed to connect to peer: 39f0e1ee-d7dd-16cd-90fb-338521924292"
--- PASS: TestRaft_RemoveFollower_SplitCluster (0.44s)
=== RUN   TestRaft_AddKnownPeer
2026-04-23T00:14:55.454Z [INFO]  server-222a76c0-2c3a-95bc-3935-5e3da00d5d3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-222a76c0-2c3a-95bc-3935-5e3da00d5d3e Address:222a76c0-2c3a-95bc-3935-5e3da00d5d3e} {Suffrage:Voter ID:server-74f4d427-57d8-8cf3-c945-42b30df79265 Address:74f4d427-57d8-8cf3-c945-42b30df79265} {Suffrage:Voter ID:server-ebe37aef-95af-39fa-87d3-0693ceca3615 Address:ebe37aef-95af-39fa-87d3-0693ceca3615}]"
2026-04-23T00:14:55.454Z [INFO]  server-74f4d427-57d8-8cf3-c945-42b30df79265: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-222a76c0-2c3a-95bc-3935-5e3da00d5d3e Address:222a76c0-2c3a-95bc-3935-5e3da00d5d3e} {Suffrage:Voter ID:server-74f4d427-57d8-8cf3-c945-42b30df79265 Address:74f4d427-57d8-8cf3-c945-42b30df79265} {Suffrage:Voter ID:server-ebe37aef-95af-39fa-87d3-0693ceca3615 Address:ebe37aef-95af-39fa-87d3-0693ceca3615}]"
2026-04-23T00:14:55.454Z [INFO]  server-222a76c0-2c3a-95bc-3935-5e3da00d5d3e: entering follower state: follower="Node at 222a76c0-2c3a-95bc-3935-5e3da00d5d3e [Follower]" leader-address= leader-id=
2026-04-23T00:14:55.454Z [INFO]  server-ebe37aef-95af-39fa-87d3-0693ceca3615: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-222a76c0-2c3a-95bc-3935-5e3da00d5d3e Address:222a76c0-2c3a-95bc-3935-5e3da00d5d3e} {Suffrage:Voter ID:server-74f4d427-57d8-8cf3-c945-42b30df79265 Address:74f4d427-57d8-8cf3-c945-42b30df79265} {Suffrage:Voter ID:server-ebe37aef-95af-39fa-87d3-0693ceca3615 Address:ebe37aef-95af-39fa-87d3-0693ceca3615}]"
2026-04-23T00:14:55.454Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:55.454Z [INFO]  server-74f4d427-57d8-8cf3-c945-42b30df79265: entering follower state: follower="Node at 74f4d427-57d8-8cf3-c945-42b30df79265 [Follower]" leader-address= leader-id=
2026-04-23T00:14:55.454Z [INFO]  server-ebe37aef-95af-39fa-87d3-0693ceca3615: entering follower state: follower="Node at ebe37aef-95af-39fa-87d3-0693ceca3615 [Follower]" leader-address= leader-id=
2026-04-23T00:14:55.518Z [WARN]  server-ebe37aef-95af-39fa-87d3-0693ceca3615: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:55.518Z [INFO]  server-ebe37aef-95af-39fa-87d3-0693ceca3615: entering candidate state: node="Node at ebe37aef-95af-39fa-87d3-0693ceca3615 [Candidate]" term=2
2026-04-23T00:14:55.518Z [INFO]  server-ebe37aef-95af-39fa-87d3-0693ceca3615: election won: term=2 tally=2
2026-04-23T00:14:55.518Z [INFO]  server-ebe37aef-95af-39fa-87d3-0693ceca3615: entering leader state: leader="Node at ebe37aef-95af-39fa-87d3-0693ceca3615 [Leader]"
2026-04-23T00:14:55.518Z [INFO]  server-ebe37aef-95af-39fa-87d3-0693ceca3615: added peer, starting replication: peer=server-222a76c0-2c3a-95bc-3935-5e3da00d5d3e
2026-04-23T00:14:55.518Z [INFO]  server-ebe37aef-95af-39fa-87d3-0693ceca3615: added peer, starting replication: peer=server-74f4d427-57d8-8cf3-c945-42b30df79265
2026-04-23T00:14:55.518Z [INFO]  server-ebe37aef-95af-39fa-87d3-0693ceca3615: pipelining replication: peer="{Voter server-74f4d427-57d8-8cf3-c945-42b30df79265 74f4d427-57d8-8cf3-c945-42b30df79265}"
2026-04-23T00:14:55.518Z [INFO]  server-ebe37aef-95af-39fa-87d3-0693ceca3615: pipelining replication: peer="{Voter server-222a76c0-2c3a-95bc-3935-5e3da00d5d3e 222a76c0-2c3a-95bc-3935-5e3da00d5d3e}"
2026-04-23T00:14:55.624Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:55.518890697 +0000 UTC m=+12.558492065 (1 nodes), 64.346699ms from start of poll, 64.650502ms from cluster start. Timeout at 2026-04-23 00:14:55.623896043 +0000 UTC m=+12.663497400, 105.005335ms after stability
2026-04-23T00:14:55.624Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:14:55.729Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:14:55.624639037 +0000 UTC m=+12.664240407 (2 nodes), 1.167µs from start of poll, 170.398844ms from cluster start. Timeout at 2026-04-23 00:14:55.729647928 +0000 UTC m=+12.769249284, 105.008877ms after stability
2026-04-23T00:14:55.729Z [INFO]  server-ebe37aef-95af-39fa-87d3-0693ceca3615: updating configuration: command=AddVoter server-id=server-222a76c0-2c3a-95bc-3935-5e3da00d5d3e server-addr=222a76c0-2c3a-95bc-3935-5e3da00d5d3e servers="[{Suffrage:Voter ID:server-222a76c0-2c3a-95bc-3935-5e3da00d5d3e Address:222a76c0-2c3a-95bc-3935-5e3da00d5d3e} {Suffrage:Voter ID:server-74f4d427-57d8-8cf3-c945-42b30df79265 Address:74f4d427-57d8-8cf3-c945-42b30df79265} {Suffrage:Voter ID:server-ebe37aef-95af-39fa-87d3-0693ceca3615 Address:ebe37aef-95af-39fa-87d3-0693ceca3615}]"
2026-04-23T00:14:55.730Z [INFO]  server-ebe37aef-95af-39fa-87d3-0693ceca3615: aborting pipeline replication: peer="{Voter server-74f4d427-57d8-8cf3-c945-42b30df79265 74f4d427-57d8-8cf3-c945-42b30df79265}"
2026-04-23T00:14:55.730Z [INFO]  server-ebe37aef-95af-39fa-87d3-0693ceca3615: aborting pipeline replication: peer="{Voter server-222a76c0-2c3a-95bc-3935-5e3da00d5d3e 222a76c0-2c3a-95bc-3935-5e3da00d5d3e}"
--- PASS: TestRaft_AddKnownPeer (0.28s)
=== RUN   TestRaft_RemoveUnknownPeer
2026-04-23T00:14:55.731Z [INFO]  server-5af6a6db-9d33-455e-ef32-669190bbf1ae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5af6a6db-9d33-455e-ef32-669190bbf1ae Address:5af6a6db-9d33-455e-ef32-669190bbf1ae} {Suffrage:Voter ID:server-e44da95c-bc79-aaf9-abf3-4947fc278020 Address:e44da95c-bc79-aaf9-abf3-4947fc278020} {Suffrage:Voter ID:server-b8f8a869-0163-9c5a-6060-e926c19e02b8 Address:b8f8a869-0163-9c5a-6060-e926c19e02b8}]"
2026-04-23T00:14:55.731Z [INFO]  server-5af6a6db-9d33-455e-ef32-669190bbf1ae: entering follower state: follower="Node at 5af6a6db-9d33-455e-ef32-669190bbf1ae [Follower]" leader-address= leader-id=
2026-04-23T00:14:55.731Z [INFO]  server-e44da95c-bc79-aaf9-abf3-4947fc278020: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5af6a6db-9d33-455e-ef32-669190bbf1ae Address:5af6a6db-9d33-455e-ef32-669190bbf1ae} {Suffrage:Voter ID:server-e44da95c-bc79-aaf9-abf3-4947fc278020 Address:e44da95c-bc79-aaf9-abf3-4947fc278020} {Suffrage:Voter ID:server-b8f8a869-0163-9c5a-6060-e926c19e02b8 Address:b8f8a869-0163-9c5a-6060-e926c19e02b8}]"
2026-04-23T00:14:55.731Z [INFO]  server-e44da95c-bc79-aaf9-abf3-4947fc278020: entering follower state: follower="Node at e44da95c-bc79-aaf9-abf3-4947fc278020 [Follower]" leader-address= leader-id=
2026-04-23T00:14:55.731Z [INFO]  server-b8f8a869-0163-9c5a-6060-e926c19e02b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5af6a6db-9d33-455e-ef32-669190bbf1ae Address:5af6a6db-9d33-455e-ef32-669190bbf1ae} {Suffrage:Voter ID:server-e44da95c-bc79-aaf9-abf3-4947fc278020 Address:e44da95c-bc79-aaf9-abf3-4947fc278020} {Suffrage:Voter ID:server-b8f8a869-0163-9c5a-6060-e926c19e02b8 Address:b8f8a869-0163-9c5a-6060-e926c19e02b8}]"
2026-04-23T00:14:55.731Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:55.731Z [INFO]  server-b8f8a869-0163-9c5a-6060-e926c19e02b8: entering follower state: follower="Node at b8f8a869-0163-9c5a-6060-e926c19e02b8 [Follower]" leader-address= leader-id=
2026-04-23T00:14:55.788Z [WARN]  server-b8f8a869-0163-9c5a-6060-e926c19e02b8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:55.788Z [INFO]  server-b8f8a869-0163-9c5a-6060-e926c19e02b8: entering candidate state: node="Node at b8f8a869-0163-9c5a-6060-e926c19e02b8 [Candidate]" term=2
2026-04-23T00:14:55.788Z [INFO]  server-b8f8a869-0163-9c5a-6060-e926c19e02b8: election won: term=2 tally=2
2026-04-23T00:14:55.788Z [INFO]  server-b8f8a869-0163-9c5a-6060-e926c19e02b8: entering leader state: leader="Node at b8f8a869-0163-9c5a-6060-e926c19e02b8 [Leader]"
2026-04-23T00:14:55.788Z [INFO]  server-b8f8a869-0163-9c5a-6060-e926c19e02b8: added peer, starting replication: peer=server-5af6a6db-9d33-455e-ef32-669190bbf1ae
2026-04-23T00:14:55.788Z [INFO]  server-b8f8a869-0163-9c5a-6060-e926c19e02b8: added peer, starting replication: peer=server-e44da95c-bc79-aaf9-abf3-4947fc278020
2026-04-23T00:14:55.788Z [INFO]  server-b8f8a869-0163-9c5a-6060-e926c19e02b8: pipelining replication: peer="{Voter server-5af6a6db-9d33-455e-ef32-669190bbf1ae 5af6a6db-9d33-455e-ef32-669190bbf1ae}"
2026-04-23T00:14:55.788Z [INFO]  server-b8f8a869-0163-9c5a-6060-e926c19e02b8: pipelining replication: peer="{Voter server-e44da95c-bc79-aaf9-abf3-4947fc278020 e44da95c-bc79-aaf9-abf3-4947fc278020}"
2026-04-23T00:14:55.894Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:55.788246465 +0000 UTC m=+12.827847836 (1 nodes), 56.860414ms from start of poll, 57.2463ms from cluster start. Timeout at 2026-04-23 00:14:55.89324713 +0000 UTC m=+12.932848506, 105.00067ms after stability
2026-04-23T00:14:55.894Z [INFO]  server-b8f8a869-0163-9c5a-6060-e926c19e02b8: updating configuration: command=RemoveServer server-id=93e0c912-4485-808a-fd48-ddf74e22e010 server-addr= servers="[{Suffrage:Voter ID:server-5af6a6db-9d33-455e-ef32-669190bbf1ae Address:5af6a6db-9d33-455e-ef32-669190bbf1ae} {Suffrage:Voter ID:server-e44da95c-bc79-aaf9-abf3-4947fc278020 Address:e44da95c-bc79-aaf9-abf3-4947fc278020} {Suffrage:Voter ID:server-b8f8a869-0163-9c5a-6060-e926c19e02b8 Address:b8f8a869-0163-9c5a-6060-e926c19e02b8}]"
2026-04-23T00:14:55.894Z [INFO]  server-b8f8a869-0163-9c5a-6060-e926c19e02b8: aborting pipeline replication: peer="{Voter server-e44da95c-bc79-aaf9-abf3-4947fc278020 e44da95c-bc79-aaf9-abf3-4947fc278020}"
2026-04-23T00:14:55.894Z [INFO]  server-b8f8a869-0163-9c5a-6060-e926c19e02b8: aborting pipeline replication: peer="{Voter server-5af6a6db-9d33-455e-ef32-669190bbf1ae 5af6a6db-9d33-455e-ef32-669190bbf1ae}"
--- PASS: TestRaft_RemoveUnknownPeer (0.16s)
=== RUN   TestRaft_SnapshotRestore
2026-04-23T00:14:55.895Z [INFO]  server-78f355a7-d2a0-860f-64b1-1deccd9dd06b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78f355a7-d2a0-860f-64b1-1deccd9dd06b Address:78f355a7-d2a0-860f-64b1-1deccd9dd06b}]"
2026-04-23T00:14:55.895Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:55.895Z [INFO]  server-78f355a7-d2a0-860f-64b1-1deccd9dd06b: entering follower state: follower="Node at 78f355a7-d2a0-860f-64b1-1deccd9dd06b [Follower]" leader-address= leader-id=
2026-04-23T00:14:55.993Z [WARN]  server-78f355a7-d2a0-860f-64b1-1deccd9dd06b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:55.993Z [INFO]  server-78f355a7-d2a0-860f-64b1-1deccd9dd06b: entering candidate state: node="Node at 78f355a7-d2a0-860f-64b1-1deccd9dd06b [Candidate]" term=2
2026-04-23T00:14:55.993Z [INFO]  server-78f355a7-d2a0-860f-64b1-1deccd9dd06b: election won: term=2 tally=1
2026-04-23T00:14:55.993Z [INFO]  server-78f355a7-d2a0-860f-64b1-1deccd9dd06b: entering leader state: leader="Node at 78f355a7-d2a0-860f-64b1-1deccd9dd06b [Leader]"
2026-04-23T00:14:56.099Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:55.993658449 +0000 UTC m=+13.033259803 (1 nodes), 98.376574ms from start of poll, 98.494185ms from cluster start. Timeout at 2026-04-23 00:14:56.098658919 +0000 UTC m=+13.138260276, 105.000473ms after stability
2026-04-23T00:14:56.100Z [INFO]  server-78f355a7-d2a0-860f-64b1-1deccd9dd06b: starting snapshot up to: index=102
2026-04-23T00:14:56.100Z [INFO]  creating new snapshot: path=/tmp/raft1629870305/snapshots/2-102-1776903296100.tmp
2026-04-23T00:14:56.101Z [INFO]  server-78f355a7-d2a0-860f-64b1-1deccd9dd06b: compacting logs: from=1 to=92
2026-04-23T00:14:56.101Z [INFO]  server-78f355a7-d2a0-860f-64b1-1deccd9dd06b: snapshot complete up to: index=102
2026-04-23T00:14:56.101Z [INFO]  server-78f355a7-d2a0-860f-64b1-1deccd9dd06b: starting restore from snapshot: id=2-102-1776903296100 last-index=102 last-term=2 size-in-bytes=693
2026-04-23T00:14:56.101Z [INFO]  server-78f355a7-d2a0-860f-64b1-1deccd9dd06b: snapshot restore progress: id=2-102-1776903296100 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-04-23T00:14:56.101Z [INFO]  server-78f355a7-d2a0-860f-64b1-1deccd9dd06b: restored from snapshot: id=2-102-1776903296100 last-index=102 last-term=2 size-in-bytes=693
2026-04-23T00:14:56.101Z [INFO]  server-78f355a7-d2a0-860f-64b1-1deccd9dd06b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-78f355a7-d2a0-860f-64b1-1deccd9dd06b Address:78f355a7-d2a0-860f-64b1-1deccd9dd06b}]"
--- PASS: TestRaft_SnapshotRestore (0.21s)
=== RUN   TestRaft_RestoreSnapshotOnStartup_Monotonic
2026-04-23T00:14:56.102Z [INFO]  server-458dfe56-f53f-0243-1ff8-5eb5a604462f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-458dfe56-f53f-0243-1ff8-5eb5a604462f Address:458dfe56-f53f-0243-1ff8-5eb5a604462f}]"
2026-04-23T00:14:56.102Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:56.102Z [INFO]  server-458dfe56-f53f-0243-1ff8-5eb5a604462f: entering follower state: follower="Node at 458dfe56-f53f-0243-1ff8-5eb5a604462f [Follower]" leader-address= leader-id=
2026-04-23T00:14:56.169Z [WARN]  server-458dfe56-f53f-0243-1ff8-5eb5a604462f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:56.169Z [INFO]  server-458dfe56-f53f-0243-1ff8-5eb5a604462f: entering candidate state: node="Node at 458dfe56-f53f-0243-1ff8-5eb5a604462f [Candidate]" term=2
2026-04-23T00:14:56.169Z [INFO]  server-458dfe56-f53f-0243-1ff8-5eb5a604462f: election won: term=2 tally=1
2026-04-23T00:14:56.169Z [INFO]  server-458dfe56-f53f-0243-1ff8-5eb5a604462f: entering leader state: leader="Node at 458dfe56-f53f-0243-1ff8-5eb5a604462f [Leader]"
2026-04-23T00:14:56.275Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:56.169707879 +0000 UTC m=+13.209309236 (1 nodes), 67.071449ms from start of poll, 67.186597ms from cluster start. Timeout at 2026-04-23 00:14:56.274711102 +0000 UTC m=+13.314312454, 105.003218ms after stability
2026-04-23T00:14:56.276Z [INFO]  server-458dfe56-f53f-0243-1ff8-5eb5a604462f: starting snapshot up to: index=102
2026-04-23T00:14:56.276Z [INFO]  creating new snapshot: path=/tmp/raft3922838275/snapshots/2-102-1776903296276.tmp
2026-04-23T00:14:56.277Z [INFO]  server-458dfe56-f53f-0243-1ff8-5eb5a604462f: compacting logs: from=1 to=92
2026-04-23T00:14:56.277Z [INFO]  server-458dfe56-f53f-0243-1ff8-5eb5a604462f: snapshot complete up to: index=102
2026-04-23T00:14:56.277Z [INFO]  server-458dfe56-f53f-0243-1ff8-5eb5a604462f: starting restore from snapshot: id=2-102-1776903296276 last-index=102 last-term=2 size-in-bytes=693
2026-04-23T00:14:56.277Z [INFO]  server-458dfe56-f53f-0243-1ff8-5eb5a604462f: snapshot restore progress: id=2-102-1776903296276 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-04-23T00:14:56.277Z [INFO]  server-458dfe56-f53f-0243-1ff8-5eb5a604462f: restored from snapshot: id=2-102-1776903296276 last-index=102 last-term=2 size-in-bytes=693
2026-04-23T00:14:56.277Z [INFO]  server-458dfe56-f53f-0243-1ff8-5eb5a604462f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-458dfe56-f53f-0243-1ff8-5eb5a604462f Address:458dfe56-f53f-0243-1ff8-5eb5a604462f}]"
--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.56s)
=== RUN   TestRaft_SnapshotRestore_Progress
2026-04-23T00:14:56.666Z [INFO]  server-aa26a52c-2a00-5d8d-bc0b-e7d566660e4f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa26a52c-2a00-5d8d-bc0b-e7d566660e4f Address:aa26a52c-2a00-5d8d-bc0b-e7d566660e4f}]"
2026-04-23T00:14:56.666Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:56.666Z [INFO]  server-aa26a52c-2a00-5d8d-bc0b-e7d566660e4f: entering follower state: follower="Node at aa26a52c-2a00-5d8d-bc0b-e7d566660e4f [Follower]" leader-address= leader-id=
2026-04-23T00:14:56.728Z [WARN]  server-aa26a52c-2a00-5d8d-bc0b-e7d566660e4f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:56.728Z [INFO]  server-aa26a52c-2a00-5d8d-bc0b-e7d566660e4f: entering candidate state: node="Node at aa26a52c-2a00-5d8d-bc0b-e7d566660e4f [Candidate]" term=2
2026-04-23T00:14:56.728Z [INFO]  server-aa26a52c-2a00-5d8d-bc0b-e7d566660e4f: election won: term=2 tally=1
2026-04-23T00:14:56.728Z [INFO]  server-aa26a52c-2a00-5d8d-bc0b-e7d566660e4f: entering leader state: leader="Node at aa26a52c-2a00-5d8d-bc0b-e7d566660e4f [Leader]"
2026-04-23T00:14:56.834Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:56.728570217 +0000 UTC m=+13.768171636 (1 nodes), 61.767928ms from start of poll, 61.986094ms from cluster start. Timeout at 2026-04-23 00:14:56.833572877 +0000 UTC m=+13.873174237, 105.002601ms after stability
2026-04-23T00:14:56.837Z [INFO]  server-aa26a52c-2a00-5d8d-bc0b-e7d566660e4f: starting snapshot up to: index=102
2026-04-23T00:14:56.837Z [INFO]  creating new snapshot: path=/tmp/raft655925955/snapshots/2-102-1776903296837.tmp
2026-04-23T00:14:56.837Z [INFO]  server-aa26a52c-2a00-5d8d-bc0b-e7d566660e4f: compacting logs: from=1 to=92
2026-04-23T00:14:56.837Z [INFO]  server-aa26a52c-2a00-5d8d-bc0b-e7d566660e4f: snapshot complete up to: index=102
--- PASS: TestRaft_SnapshotRestore_Progress (0.17s)
=== RUN   TestRaft_NoRestoreOnStart
2026-04-23T00:14:56.838Z [INFO]  server-6c8f8614-a415-370d-1167-f74ffdfe2ec6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c8f8614-a415-370d-1167-f74ffdfe2ec6 Address:6c8f8614-a415-370d-1167-f74ffdfe2ec6}]"
2026-04-23T00:14:56.838Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:56.838Z [INFO]  server-6c8f8614-a415-370d-1167-f74ffdfe2ec6: entering follower state: follower="Node at 6c8f8614-a415-370d-1167-f74ffdfe2ec6 [Follower]" leader-address= leader-id=
2026-04-23T00:14:56.934Z [WARN]  server-6c8f8614-a415-370d-1167-f74ffdfe2ec6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:56.934Z [INFO]  server-6c8f8614-a415-370d-1167-f74ffdfe2ec6: entering candidate state: node="Node at 6c8f8614-a415-370d-1167-f74ffdfe2ec6 [Candidate]" term=2
2026-04-23T00:14:56.934Z [INFO]  server-6c8f8614-a415-370d-1167-f74ffdfe2ec6: election won: term=2 tally=1
2026-04-23T00:14:56.934Z [INFO]  server-6c8f8614-a415-370d-1167-f74ffdfe2ec6: entering leader state: leader="Node at 6c8f8614-a415-370d-1167-f74ffdfe2ec6 [Leader]"
2026-04-23T00:14:57.039Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:56.93447537 +0000 UTC m=+13.974076727 (1 nodes), 95.684606ms from start of poll, 95.840858ms from cluster start. Timeout at 2026-04-23 00:14:57.03947809 +0000 UTC m=+14.079079446, 105.002719ms after stability
2026-04-23T00:14:57.041Z [INFO]  server-6c8f8614-a415-370d-1167-f74ffdfe2ec6: starting snapshot up to: index=102
2026-04-23T00:14:57.041Z [INFO]  creating new snapshot: path=/tmp/raft4279191849/snapshots/2-102-1776903297041.tmp
2026-04-23T00:14:57.042Z [INFO]  server-6c8f8614-a415-370d-1167-f74ffdfe2ec6: compacting logs: from=1 to=92
2026-04-23T00:14:57.042Z [INFO]  server-6c8f8614-a415-370d-1167-f74ffdfe2ec6: snapshot complete up to: index=102
2026-04-23T00:14:57.042Z [INFO]  server-6c8f8614-a415-370d-1167-f74ffdfe2ec6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c8f8614-a415-370d-1167-f74ffdfe2ec6 Address:6c8f8614-a415-370d-1167-f74ffdfe2ec6}]"
--- PASS: TestRaft_NoRestoreOnStart (0.20s)
=== RUN   TestRaft_SnapshotRestore_PeerChange
2026-04-23T00:14:57.042Z [INFO]  server-6c8f8614-a415-370d-1167-f74ffdfe2ec6: entering follower state: follower="Node at 6c8f8614-a415-370d-1167-f74ffdfe2ec6 [Follower]" leader-address= leader-id=
    raft_test.go:1266: Skipping unreliable test
--- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
=== RUN   TestRaft_AutoSnapshot
2026-04-23T00:14:57.042Z [INFO]  server-62802f77-8bbd-cdc1-2fae-817894b2ce75: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62802f77-8bbd-cdc1-2fae-817894b2ce75 Address:62802f77-8bbd-cdc1-2fae-817894b2ce75}]"
2026-04-23T00:14:57.042Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:57.042Z [INFO]  server-62802f77-8bbd-cdc1-2fae-817894b2ce75: entering follower state: follower="Node at 62802f77-8bbd-cdc1-2fae-817894b2ce75 [Follower]" leader-address= leader-id=
2026-04-23T00:14:57.118Z [WARN]  server-62802f77-8bbd-cdc1-2fae-817894b2ce75: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:57.118Z [INFO]  server-62802f77-8bbd-cdc1-2fae-817894b2ce75: entering candidate state: node="Node at 62802f77-8bbd-cdc1-2fae-817894b2ce75 [Candidate]" term=2
2026-04-23T00:14:57.118Z [INFO]  server-62802f77-8bbd-cdc1-2fae-817894b2ce75: election won: term=2 tally=1
2026-04-23T00:14:57.118Z [INFO]  server-62802f77-8bbd-cdc1-2fae-817894b2ce75: entering leader state: leader="Node at 62802f77-8bbd-cdc1-2fae-817894b2ce75 [Leader]"
2026-04-23T00:14:57.140Z [WARN]  server-6c8f8614-a415-370d-1167-f74ffdfe2ec6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:57.140Z [INFO]  server-6c8f8614-a415-370d-1167-f74ffdfe2ec6: entering candidate state: node="Node at 6c8f8614-a415-370d-1167-f74ffdfe2ec6 [Candidate]" term=3
2026-04-23T00:14:57.140Z [INFO]  server-6c8f8614-a415-370d-1167-f74ffdfe2ec6: election won: term=3 tally=1
2026-04-23T00:14:57.140Z [INFO]  server-6c8f8614-a415-370d-1167-f74ffdfe2ec6: entering leader state: leader="Node at 6c8f8614-a415-370d-1167-f74ffdfe2ec6 [Leader]"
2026-04-23T00:14:57.223Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:57.118508325 +0000 UTC m=+14.158109682 (1 nodes), 75.654945ms from start of poll, 75.799859ms from cluster start. Timeout at 2026-04-23 00:14:57.223510754 +0000 UTC m=+14.263112109, 105.002427ms after stability
2026-04-23T00:14:57.236Z [INFO]  server-62802f77-8bbd-cdc1-2fae-817894b2ce75: starting snapshot up to: index=102
2026-04-23T00:14:57.236Z [INFO]  creating new snapshot: path=/tmp/raft4245598050/snapshots/2-102-1776903297236.tmp
2026-04-23T00:14:57.237Z [INFO]  server-62802f77-8bbd-cdc1-2fae-817894b2ce75: compacting logs: from=1 to=92
2026-04-23T00:14:57.237Z [INFO]  server-62802f77-8bbd-cdc1-2fae-817894b2ce75: snapshot complete up to: index=102
--- PASS: TestRaft_AutoSnapshot (0.29s)
=== RUN   TestRaft_UserSnapshot
2026-04-23T00:14:57.331Z [INFO]  server-ca81983f-91fd-2a57-96d9-32d07a269a23: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca81983f-91fd-2a57-96d9-32d07a269a23 Address:ca81983f-91fd-2a57-96d9-32d07a269a23}]"
2026-04-23T00:14:57.331Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:57.331Z [INFO]  server-ca81983f-91fd-2a57-96d9-32d07a269a23: entering follower state: follower="Node at ca81983f-91fd-2a57-96d9-32d07a269a23 [Follower]" leader-address= leader-id=
2026-04-23T00:14:57.386Z [WARN]  server-ca81983f-91fd-2a57-96d9-32d07a269a23: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:57.386Z [INFO]  server-ca81983f-91fd-2a57-96d9-32d07a269a23: entering candidate state: node="Node at ca81983f-91fd-2a57-96d9-32d07a269a23 [Candidate]" term=2
2026-04-23T00:14:57.386Z [INFO]  server-ca81983f-91fd-2a57-96d9-32d07a269a23: election won: term=2 tally=1
2026-04-23T00:14:57.386Z [INFO]  server-ca81983f-91fd-2a57-96d9-32d07a269a23: entering leader state: leader="Node at ca81983f-91fd-2a57-96d9-32d07a269a23 [Leader]"
2026-04-23T00:14:57.492Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:57.386528264 +0000 UTC m=+14.426129645 (1 nodes), 55.081669ms from start of poll, 55.241777ms from cluster start. Timeout at 2026-04-23 00:14:57.491533976 +0000 UTC m=+14.531135341, 105.005696ms after stability
2026-04-23T00:14:57.492Z [ERROR] server-ca81983f-91fd-2a57-96d9-32d07a269a23: failed to take snapshot: error="nothing new to snapshot"
2026-04-23T00:14:57.492Z [INFO]  server-ca81983f-91fd-2a57-96d9-32d07a269a23: starting snapshot up to: index=12
2026-04-23T00:14:57.492Z [INFO]  creating new snapshot: path=/tmp/raft124433128/snapshots/2-12-1776903297492.tmp
2026-04-23T00:14:57.493Z [INFO]  server-ca81983f-91fd-2a57-96d9-32d07a269a23: compacting logs: from=1 to=2
2026-04-23T00:14:57.493Z [INFO]  server-ca81983f-91fd-2a57-96d9-32d07a269a23: snapshot complete up to: index=12
--- PASS: TestRaft_UserSnapshot (0.16s)
=== RUN   TestRaft_UserRestore
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false
2026-04-23T00:14:57.494Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c40841b-cccc-8009-d1d0-3f3766b20c85 Address:4c40841b-cccc-8009-d1d0-3f3766b20c85} {Suffrage:Voter ID:server-58922178-b43b-fce8-8d4a-d62ed8e42d18 Address:58922178-b43b-fce8-8d4a-d62ed8e42d18} {Suffrage:Voter ID:server-4f69fdf4-167c-9381-1f80-dbbb9b639bb0 Address:4f69fdf4-167c-9381-1f80-dbbb9b639bb0}]"
2026-04-23T00:14:57.494Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: entering follower state: follower="Node at 4c40841b-cccc-8009-d1d0-3f3766b20c85 [Follower]" leader-address= leader-id=
2026-04-23T00:14:57.494Z [INFO]  server-58922178-b43b-fce8-8d4a-d62ed8e42d18: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c40841b-cccc-8009-d1d0-3f3766b20c85 Address:4c40841b-cccc-8009-d1d0-3f3766b20c85} {Suffrage:Voter ID:server-58922178-b43b-fce8-8d4a-d62ed8e42d18 Address:58922178-b43b-fce8-8d4a-d62ed8e42d18} {Suffrage:Voter ID:server-4f69fdf4-167c-9381-1f80-dbbb9b639bb0 Address:4f69fdf4-167c-9381-1f80-dbbb9b639bb0}]"
2026-04-23T00:14:57.494Z [INFO]  server-58922178-b43b-fce8-8d4a-d62ed8e42d18: entering follower state: follower="Node at 58922178-b43b-fce8-8d4a-d62ed8e42d18 [Follower]" leader-address= leader-id=
2026-04-23T00:14:57.494Z [INFO]  server-4f69fdf4-167c-9381-1f80-dbbb9b639bb0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c40841b-cccc-8009-d1d0-3f3766b20c85 Address:4c40841b-cccc-8009-d1d0-3f3766b20c85} {Suffrage:Voter ID:server-58922178-b43b-fce8-8d4a-d62ed8e42d18 Address:58922178-b43b-fce8-8d4a-d62ed8e42d18} {Suffrage:Voter ID:server-4f69fdf4-167c-9381-1f80-dbbb9b639bb0 Address:4f69fdf4-167c-9381-1f80-dbbb9b639bb0}]"
2026-04-23T00:14:57.494Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:14:57.494Z [INFO]  server-4f69fdf4-167c-9381-1f80-dbbb9b639bb0: entering follower state: follower="Node at 4f69fdf4-167c-9381-1f80-dbbb9b639bb0 [Follower]" leader-address= leader-id=
2026-04-23T00:14:58.107Z [WARN]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:14:58.107Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: entering candidate state: node="Node at 4c40841b-cccc-8009-d1d0-3f3766b20c85 [Candidate]" term=2
2026-04-23T00:14:58.107Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: election won: term=2 tally=2
2026-04-23T00:14:58.107Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: entering leader state: leader="Node at 4c40841b-cccc-8009-d1d0-3f3766b20c85 [Leader]"
2026-04-23T00:14:58.108Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: added peer, starting replication: peer=server-58922178-b43b-fce8-8d4a-d62ed8e42d18
2026-04-23T00:14:58.108Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: added peer, starting replication: peer=server-4f69fdf4-167c-9381-1f80-dbbb9b639bb0
2026-04-23T00:14:58.108Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: pipelining replication: peer="{Voter server-4f69fdf4-167c-9381-1f80-dbbb9b639bb0 4f69fdf4-167c-9381-1f80-dbbb9b639bb0}"
2026-04-23T00:14:58.108Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: pipelining replication: peer="{Voter server-58922178-b43b-fce8-8d4a-d62ed8e42d18 58922178-b43b-fce8-8d4a-d62ed8e42d18}"
2026-04-23T00:14:59.113Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:14:58.108084777 +0000 UTC m=+15.147686134 (1 nodes), 613.558295ms from start of poll, 613.949983ms from cluster start. Timeout at 2026-04-23 00:14:59.113089796 +0000 UTC m=+16.152691173, 1.005005039s after stability
2026-04-23T00:14:59.114Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: starting snapshot up to: index=12
2026-04-23T00:14:59.114Z [INFO]  creating new snapshot: path=/tmp/raft2083530919/snapshots/2-12-1776903299114.tmp
2026-04-23T00:14:59.115Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: snapshot complete up to: index=12
2026-04-23T00:14:59.115Z [INFO]  creating new snapshot: path=/tmp/raft2083530919/snapshots/2-23-1776903299115.tmp
2026-04-23T00:14:59.119Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: copied to local snapshot: bytes=71
2026-04-23T00:14:59.121Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: snapshot restore progress: id=2-23-1776903299115 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:14:59.121Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: restored user snapshot: index=23
2026-04-23T00:14:59.121Z [ERROR] server-4c40841b-cccc-8009-d1d0-3f3766b20c85: failed to get log: index=23 error="log not found"
2026-04-23T00:14:59.121Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: aborting pipeline replication: peer="{Voter server-58922178-b43b-fce8-8d4a-d62ed8e42d18 58922178-b43b-fce8-8d4a-d62ed8e42d18}"
2026-04-23T00:14:59.121Z [ERROR] server-4c40841b-cccc-8009-d1d0-3f3766b20c85: failed to get log: index=23 error="log not found"
2026-04-23T00:14:59.121Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: aborting pipeline replication: peer="{Voter server-4f69fdf4-167c-9381-1f80-dbbb9b639bb0 4f69fdf4-167c-9381-1f80-dbbb9b639bb0}"
2026-04-23T00:14:59.129Z [ERROR] server-4c40841b-cccc-8009-d1d0-3f3766b20c85: failed to get log: index=23 error="log not found"
2026-04-23T00:14:59.129Z [INFO]  creating new snapshot: path=/tmp/raft1530646176/snapshots/2-23-1776903299129.tmp
2026-04-23T00:14:59.130Z [INFO]  server-4f69fdf4-167c-9381-1f80-dbbb9b639bb0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:14:59.130Z [ERROR] server-4c40841b-cccc-8009-d1d0-3f3766b20c85: failed to get log: index=23 error="log not found"
2026-04-23T00:14:59.130Z [INFO]  server-4f69fdf4-167c-9381-1f80-dbbb9b639bb0: copied to local snapshot: bytes=71
2026-04-23T00:14:59.131Z [INFO]  server-4f69fdf4-167c-9381-1f80-dbbb9b639bb0: snapshot restore progress: id=2-23-1776903299129 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:14:59.131Z [INFO]  server-4f69fdf4-167c-9381-1f80-dbbb9b639bb0: Installed remote snapshot
2026-04-23T00:14:59.131Z [WARN]  server-4f69fdf4-167c-9381-1f80-dbbb9b639bb0: skipping application of old log: index=22
2026-04-23T00:14:59.131Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: pipelining replication: peer="{Voter server-4f69fdf4-167c-9381-1f80-dbbb9b639bb0 4f69fdf4-167c-9381-1f80-dbbb9b639bb0}"
2026-04-23T00:14:59.131Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:14:59.138Z [INFO]  creating new snapshot: path=/tmp/raft3511459993/snapshots/2-23-1776903299138.tmp
2026-04-23T00:14:59.139Z [INFO]  server-58922178-b43b-fce8-8d4a-d62ed8e42d18: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:14:59.140Z [INFO]  server-58922178-b43b-fce8-8d4a-d62ed8e42d18: copied to local snapshot: bytes=71
2026-04-23T00:14:59.140Z [INFO]  server-58922178-b43b-fce8-8d4a-d62ed8e42d18: snapshot restore progress: id=2-23-1776903299138 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:14:59.140Z [INFO]  server-58922178-b43b-fce8-8d4a-d62ed8e42d18: Installed remote snapshot
2026-04-23T00:14:59.140Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: pipelining replication: peer="{Voter server-58922178-b43b-fce8-8d4a-d62ed8e42d18 58922178-b43b-fce8-8d4a-d62ed8e42d18}"
2026-04-23T00:15:00.136Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:14:59.13130022 +0000 UTC m=+16.170901597 (2 nodes), 926ns from start of poll, 1.637165446s from cluster start. Timeout at 2026-04-23 00:15:00.136306885 +0000 UTC m=+17.175908302, 1.005006705s after stability
2026-04-23T00:15:00.148Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: aborting pipeline replication: peer="{Voter server-4f69fdf4-167c-9381-1f80-dbbb9b639bb0 4f69fdf4-167c-9381-1f80-dbbb9b639bb0}"
2026-04-23T00:15:00.148Z [INFO]  server-4c40841b-cccc-8009-d1d0-3f3766b20c85: aborting pipeline replication: peer="{Voter server-58922178-b43b-fce8-8d4a-d62ed8e42d18 58922178-b43b-fce8-8d4a-d62ed8e42d18}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false
2026-04-23T00:15:00.155Z [INFO]  server-8886d166-9a44-97f6-0324-06f84ef852c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8886d166-9a44-97f6-0324-06f84ef852c7 Address:8886d166-9a44-97f6-0324-06f84ef852c7} {Suffrage:Voter ID:server-134be318-90e4-0910-c61f-57e2d0c53356 Address:134be318-90e4-0910-c61f-57e2d0c53356} {Suffrage:Voter ID:server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1 Address:18e6bbbc-5e97-dd58-1dc3-295b2046a5a1}]"
2026-04-23T00:15:00.155Z [INFO]  server-134be318-90e4-0910-c61f-57e2d0c53356: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8886d166-9a44-97f6-0324-06f84ef852c7 Address:8886d166-9a44-97f6-0324-06f84ef852c7} {Suffrage:Voter ID:server-134be318-90e4-0910-c61f-57e2d0c53356 Address:134be318-90e4-0910-c61f-57e2d0c53356} {Suffrage:Voter ID:server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1 Address:18e6bbbc-5e97-dd58-1dc3-295b2046a5a1}]"
2026-04-23T00:15:00.155Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8886d166-9a44-97f6-0324-06f84ef852c7 Address:8886d166-9a44-97f6-0324-06f84ef852c7} {Suffrage:Voter ID:server-134be318-90e4-0910-c61f-57e2d0c53356 Address:134be318-90e4-0910-c61f-57e2d0c53356} {Suffrage:Voter ID:server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1 Address:18e6bbbc-5e97-dd58-1dc3-295b2046a5a1}]"
2026-04-23T00:15:00.155Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:00.155Z [INFO]  server-8886d166-9a44-97f6-0324-06f84ef852c7: entering follower state: follower="Node at 8886d166-9a44-97f6-0324-06f84ef852c7 [Follower]" leader-address= leader-id=
2026-04-23T00:15:00.155Z [INFO]  server-134be318-90e4-0910-c61f-57e2d0c53356: entering follower state: follower="Node at 134be318-90e4-0910-c61f-57e2d0c53356 [Follower]" leader-address= leader-id=
2026-04-23T00:15:00.155Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: entering follower state: follower="Node at 18e6bbbc-5e97-dd58-1dc3-295b2046a5a1 [Follower]" leader-address= leader-id=
2026-04-23T00:15:00.713Z [WARN]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:00.714Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: entering candidate state: node="Node at 18e6bbbc-5e97-dd58-1dc3-295b2046a5a1 [Candidate]" term=2
2026-04-23T00:15:00.714Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: election won: term=2 tally=2
2026-04-23T00:15:00.714Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: entering leader state: leader="Node at 18e6bbbc-5e97-dd58-1dc3-295b2046a5a1 [Leader]"
2026-04-23T00:15:00.714Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: added peer, starting replication: peer=server-8886d166-9a44-97f6-0324-06f84ef852c7
2026-04-23T00:15:00.714Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: added peer, starting replication: peer=server-134be318-90e4-0910-c61f-57e2d0c53356
2026-04-23T00:15:00.714Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: pipelining replication: peer="{Voter server-8886d166-9a44-97f6-0324-06f84ef852c7 8886d166-9a44-97f6-0324-06f84ef852c7}"
2026-04-23T00:15:00.717Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: pipelining replication: peer="{Voter server-134be318-90e4-0910-c61f-57e2d0c53356 134be318-90e4-0910-c61f-57e2d0c53356}"
2026-04-23T00:15:01.720Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:00.714373775 +0000 UTC m=+17.753975144 (1 nodes), 558.80747ms from start of poll, 559.30496ms from cluster start. Timeout at 2026-04-23 00:15:01.719378086 +0000 UTC m=+18.758979461, 1.005004317s after stability
2026-04-23T00:15:01.720Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: starting snapshot up to: index=12
2026-04-23T00:15:01.720Z [INFO]  creating new snapshot: path=/tmp/raft2508865135/snapshots/2-12-1776903301720.tmp
2026-04-23T00:15:01.721Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: snapshot complete up to: index=12
2026-04-23T00:15:01.723Z [INFO]  creating new snapshot: path=/tmp/raft2508865135/snapshots/2-23-1776903301723.tmp
2026-04-23T00:15:01.723Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: copied to local snapshot: bytes=71
2026-04-23T00:15:01.723Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: snapshot restore progress: id=2-23-1776903301723 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:01.723Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: removing all old logs from log store
2026-04-23T00:15:01.724Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: compacting logs: from=1 to=22
2026-04-23T00:15:01.724Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: restored user snapshot: index=23
2026-04-23T00:15:01.724Z [ERROR] server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: failed to get log: index=22 error="log not found"
2026-04-23T00:15:01.724Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: aborting pipeline replication: peer="{Voter server-134be318-90e4-0910-c61f-57e2d0c53356 134be318-90e4-0910-c61f-57e2d0c53356}"
2026-04-23T00:15:01.724Z [ERROR] server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: failed to get log: index=20 error="log not found"
2026-04-23T00:15:01.724Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: aborting pipeline replication: peer="{Voter server-8886d166-9a44-97f6-0324-06f84ef852c7 8886d166-9a44-97f6-0324-06f84ef852c7}"
2026-04-23T00:15:01.724Z [ERROR] server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: failed to get log: index=16 error="log not found"
2026-04-23T00:15:01.724Z [INFO]  creating new snapshot: path=/tmp/raft4273276197/snapshots/2-23-1776903301724.tmp
2026-04-23T00:15:01.724Z [INFO]  server-8886d166-9a44-97f6-0324-06f84ef852c7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:01.725Z [INFO]  server-8886d166-9a44-97f6-0324-06f84ef852c7: copied to local snapshot: bytes=71
2026-04-23T00:15:01.727Z [INFO]  server-8886d166-9a44-97f6-0324-06f84ef852c7: snapshot restore progress: id=2-23-1776903301724 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:01.727Z [INFO]  server-8886d166-9a44-97f6-0324-06f84ef852c7: removing all old logs from log store
2026-04-23T00:15:01.727Z [INFO]  server-8886d166-9a44-97f6-0324-06f84ef852c7: compacting logs: from=1 to=20
2026-04-23T00:15:01.727Z [INFO]  server-8886d166-9a44-97f6-0324-06f84ef852c7: Installed remote snapshot
2026-04-23T00:15:01.727Z [WARN]  server-8886d166-9a44-97f6-0324-06f84ef852c7: skipping application of old log: index=22
2026-04-23T00:15:01.727Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:15:01.727Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: pipelining replication: peer="{Voter server-8886d166-9a44-97f6-0324-06f84ef852c7 8886d166-9a44-97f6-0324-06f84ef852c7}"
2026-04-23T00:15:01.732Z [ERROR] server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: failed to get log: index=22 error="log not found"
2026-04-23T00:15:01.732Z [INFO]  creating new snapshot: path=/tmp/raft3150559517/snapshots/2-23-1776903301732.tmp
2026-04-23T00:15:01.732Z [INFO]  server-134be318-90e4-0910-c61f-57e2d0c53356: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:01.733Z [INFO]  server-134be318-90e4-0910-c61f-57e2d0c53356: copied to local snapshot: bytes=71
2026-04-23T00:15:01.738Z [INFO]  server-134be318-90e4-0910-c61f-57e2d0c53356: snapshot restore progress: id=2-23-1776903301732 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:01.738Z [INFO]  server-134be318-90e4-0910-c61f-57e2d0c53356: removing all old logs from log store
2026-04-23T00:15:01.738Z [INFO]  server-134be318-90e4-0910-c61f-57e2d0c53356: compacting logs: from=1 to=22
2026-04-23T00:15:01.738Z [INFO]  server-134be318-90e4-0910-c61f-57e2d0c53356: Installed remote snapshot
2026-04-23T00:15:01.738Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: pipelining replication: peer="{Voter server-134be318-90e4-0910-c61f-57e2d0c53356 134be318-90e4-0910-c61f-57e2d0c53356}"
2026-04-23T00:15:02.733Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:15:01.727472555 +0000 UTC m=+18.767073929 (2 nodes), 2.029µs from start of poll, 1.572403745s from cluster start. Timeout at 2026-04-23 00:15:02.732484916 +0000 UTC m=+19.772086305, 1.005012376s after stability
2026-04-23T00:15:02.745Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: aborting pipeline replication: peer="{Voter server-134be318-90e4-0910-c61f-57e2d0c53356 134be318-90e4-0910-c61f-57e2d0c53356}"
2026-04-23T00:15:02.745Z [INFO]  server-18e6bbbc-5e97-dd58-1dc3-295b2046a5a1: aborting pipeline replication: peer="{Voter server-8886d166-9a44-97f6-0324-06f84ef852c7 8886d166-9a44-97f6-0324-06f84ef852c7}"
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true
2026-04-23T00:15:02.750Z [INFO]  server-06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4 Address:06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4} {Suffrage:Voter ID:server-1d56dbcd-171e-8168-6c49-856b56f12120 Address:1d56dbcd-171e-8168-6c49-856b56f12120} {Suffrage:Voter ID:server-f9e84588-cb86-2d48-4fdf-315cb172e745 Address:f9e84588-cb86-2d48-4fdf-315cb172e745}]"
2026-04-23T00:15:02.751Z [INFO]  server-1d56dbcd-171e-8168-6c49-856b56f12120: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4 Address:06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4} {Suffrage:Voter ID:server-1d56dbcd-171e-8168-6c49-856b56f12120 Address:1d56dbcd-171e-8168-6c49-856b56f12120} {Suffrage:Voter ID:server-f9e84588-cb86-2d48-4fdf-315cb172e745 Address:f9e84588-cb86-2d48-4fdf-315cb172e745}]"
2026-04-23T00:15:02.751Z [INFO]  server-f9e84588-cb86-2d48-4fdf-315cb172e745: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4 Address:06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4} {Suffrage:Voter ID:server-1d56dbcd-171e-8168-6c49-856b56f12120 Address:1d56dbcd-171e-8168-6c49-856b56f12120} {Suffrage:Voter ID:server-f9e84588-cb86-2d48-4fdf-315cb172e745 Address:f9e84588-cb86-2d48-4fdf-315cb172e745}]"
2026-04-23T00:15:02.751Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:02.751Z [INFO]  server-06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4: entering follower state: follower="Node at 06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4 [Follower]" leader-address= leader-id=
2026-04-23T00:15:02.751Z [INFO]  server-1d56dbcd-171e-8168-6c49-856b56f12120: entering follower state: follower="Node at 1d56dbcd-171e-8168-6c49-856b56f12120 [Follower]" leader-address= leader-id=
2026-04-23T00:15:02.751Z [INFO]  server-f9e84588-cb86-2d48-4fdf-315cb172e745: entering follower state: follower="Node at f9e84588-cb86-2d48-4fdf-315cb172e745 [Follower]" leader-address= leader-id=
2026-04-23T00:15:03.483Z [WARN]  server-06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:03.483Z [INFO]  server-06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4: entering candidate state: node="Node at 06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4 [Candidate]" term=2
2026-04-23T00:15:03.483Z [INFO]  server-06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4: election won: term=2 tally=2
2026-04-23T00:15:03.483Z [INFO]  server-06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4: entering leader state: leader="Node at 06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4 [Leader]"
2026-04-23T00:15:03.483Z [INFO]  server-06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4: added peer, starting replication: peer=server-1d56dbcd-171e-8168-6c49-856b56f12120
2026-04-23T00:15:03.483Z [INFO]  server-06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4: added peer, starting replication: peer=server-f9e84588-cb86-2d48-4fdf-315cb172e745
2026-04-23T00:15:03.483Z [INFO]  server-06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4: pipelining replication: peer="{Voter server-f9e84588-cb86-2d48-4fdf-315cb172e745 f9e84588-cb86-2d48-4fdf-315cb172e745}"
2026-04-23T00:15:03.484Z [INFO]  server-06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4: pipelining replication: peer="{Voter server-1d56dbcd-171e-8168-6c49-856b56f12120 1d56dbcd-171e-8168-6c49-856b56f12120}"
2026-04-23T00:15:04.489Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:03.484225626 +0000 UTC m=+20.523826990 (1 nodes), 733.072069ms from start of poll, 735.551449ms from cluster start. Timeout at 2026-04-23 00:15:04.489230946 +0000 UTC m=+21.528832325, 1.005005335s after stability
2026-04-23T00:15:04.490Z [INFO]  server-06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4: starting snapshot up to: index=12
2026-04-23T00:15:04.490Z [INFO]  creating new snapshot: path=/tmp/raft3049396002/snapshots/2-12-1776903304490.tmp
2026-04-23T00:15:04.491Z [INFO]  server-06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4: snapshot complete up to: index=12
2026-04-23T00:15:04.493Z [INFO]  server-8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b Address:8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b} {Suffrage:Voter ID:server-912ab360-8a3f-475f-8760-ed382d71e3c8 Address:912ab360-8a3f-475f-8760-ed382d71e3c8} {Suffrage:Voter ID:server-e2627e83-ee3d-038d-dacc-bc36c8c8e19b Address:e2627e83-ee3d-038d-dacc-bc36c8c8e19b}]"
2026-04-23T00:15:04.493Z [INFO]  server-912ab360-8a3f-475f-8760-ed382d71e3c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b Address:8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b} {Suffrage:Voter ID:server-912ab360-8a3f-475f-8760-ed382d71e3c8 Address:912ab360-8a3f-475f-8760-ed382d71e3c8} {Suffrage:Voter ID:server-e2627e83-ee3d-038d-dacc-bc36c8c8e19b Address:e2627e83-ee3d-038d-dacc-bc36c8c8e19b}]"
2026-04-23T00:15:04.493Z [INFO]  server-e2627e83-ee3d-038d-dacc-bc36c8c8e19b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b Address:8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b} {Suffrage:Voter ID:server-912ab360-8a3f-475f-8760-ed382d71e3c8 Address:912ab360-8a3f-475f-8760-ed382d71e3c8} {Suffrage:Voter ID:server-e2627e83-ee3d-038d-dacc-bc36c8c8e19b Address:e2627e83-ee3d-038d-dacc-bc36c8c8e19b}]"
2026-04-23T00:15:04.493Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:04.494Z [INFO]  server-8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b: entering follower state: follower="Node at 8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b [Follower]" leader-address= leader-id=
2026-04-23T00:15:04.494Z [INFO]  server-912ab360-8a3f-475f-8760-ed382d71e3c8: entering follower state: follower="Node at 912ab360-8a3f-475f-8760-ed382d71e3c8 [Follower]" leader-address= leader-id=
2026-04-23T00:15:04.494Z [INFO]  server-e2627e83-ee3d-038d-dacc-bc36c8c8e19b: entering follower state: follower="Node at e2627e83-ee3d-038d-dacc-bc36c8c8e19b [Follower]" leader-address= leader-id=
2026-04-23T00:15:05.028Z [WARN]  server-912ab360-8a3f-475f-8760-ed382d71e3c8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:05.028Z [INFO]  server-912ab360-8a3f-475f-8760-ed382d71e3c8: entering candidate state: node="Node at 912ab360-8a3f-475f-8760-ed382d71e3c8 [Candidate]" term=2
2026-04-23T00:15:05.028Z [INFO]  server-912ab360-8a3f-475f-8760-ed382d71e3c8: election won: term=2 tally=2
2026-04-23T00:15:05.028Z [INFO]  server-912ab360-8a3f-475f-8760-ed382d71e3c8: entering leader state: leader="Node at 912ab360-8a3f-475f-8760-ed382d71e3c8 [Leader]"
2026-04-23T00:15:05.028Z [INFO]  server-912ab360-8a3f-475f-8760-ed382d71e3c8: added peer, starting replication: peer=server-8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b
2026-04-23T00:15:05.028Z [INFO]  server-912ab360-8a3f-475f-8760-ed382d71e3c8: added peer, starting replication: peer=server-e2627e83-ee3d-038d-dacc-bc36c8c8e19b
2026-04-23T00:15:05.028Z [INFO]  server-912ab360-8a3f-475f-8760-ed382d71e3c8: pipelining replication: peer="{Voter server-e2627e83-ee3d-038d-dacc-bc36c8c8e19b e2627e83-ee3d-038d-dacc-bc36c8c8e19b}"
2026-04-23T00:15:05.028Z [INFO]  server-912ab360-8a3f-475f-8760-ed382d71e3c8: pipelining replication: peer="{Voter server-8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b 8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b}"
2026-04-23T00:15:06.035Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:05.029064333 +0000 UTC m=+22.068665692 (1 nodes), 535.061249ms from start of poll, 535.513159ms from cluster start. Timeout at 2026-04-23 00:15:06.034065829 +0000 UTC m=+23.073667190, 1.005001498s after stability
2026-04-23T00:15:06.035Z [INFO]  creating new snapshot: path=/tmp/raft2228703719/snapshots/2-13-1776903306035.tmp
2026-04-23T00:15:06.036Z [INFO]  server-912ab360-8a3f-475f-8760-ed382d71e3c8: copied to local snapshot: bytes=71
2026-04-23T00:15:06.036Z [INFO]  server-912ab360-8a3f-475f-8760-ed382d71e3c8: snapshot restore progress: id=2-13-1776903306035 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:06.036Z [INFO]  server-912ab360-8a3f-475f-8760-ed382d71e3c8: restored user snapshot: index=13
2026-04-23T00:15:06.036Z [ERROR] server-912ab360-8a3f-475f-8760-ed382d71e3c8: failed to get log: index=3 error="log not found"
2026-04-23T00:15:06.036Z [INFO]  server-912ab360-8a3f-475f-8760-ed382d71e3c8: aborting pipeline replication: peer="{Voter server-8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b 8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b}"
2026-04-23T00:15:06.037Z [ERROR] server-912ab360-8a3f-475f-8760-ed382d71e3c8: failed to get log: index=3 error="log not found"
2026-04-23T00:15:06.037Z [INFO]  server-912ab360-8a3f-475f-8760-ed382d71e3c8: aborting pipeline replication: peer="{Voter server-e2627e83-ee3d-038d-dacc-bc36c8c8e19b e2627e83-ee3d-038d-dacc-bc36c8c8e19b}"
2026-04-23T00:15:06.046Z [ERROR] server-912ab360-8a3f-475f-8760-ed382d71e3c8: failed to get log: index=3 error="log not found"
2026-04-23T00:15:06.047Z [INFO]  creating new snapshot: path=/tmp/raft2164889629/snapshots/2-13-1776903306047.tmp
2026-04-23T00:15:06.047Z [INFO]  server-8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:06.047Z [ERROR] server-912ab360-8a3f-475f-8760-ed382d71e3c8: failed to get log: index=3 error="log not found"
2026-04-23T00:15:06.048Z [INFO]  server-8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b: copied to local snapshot: bytes=71
2026-04-23T00:15:06.048Z [INFO]  creating new snapshot: path=/tmp/raft23559525/snapshots/2-13-1776903306048.tmp
2026-04-23T00:15:06.048Z [INFO]  server-8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b: snapshot restore progress: id=2-13-1776903306047 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:06.048Z [INFO]  server-8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b: Installed remote snapshot
2026-04-23T00:15:06.048Z [INFO]  server-912ab360-8a3f-475f-8760-ed382d71e3c8: pipelining replication: peer="{Voter server-8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b 8b0dfbc0-7dbf-d964-f2e7-b18edb8b398b}"
2026-04-23T00:15:06.048Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:15:06.048Z [INFO]  server-e2627e83-ee3d-038d-dacc-bc36c8c8e19b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:06.049Z [INFO]  server-e2627e83-ee3d-038d-dacc-bc36c8c8e19b: copied to local snapshot: bytes=71
2026-04-23T00:15:06.049Z [INFO]  server-e2627e83-ee3d-038d-dacc-bc36c8c8e19b: snapshot restore progress: id=2-13-1776903306048 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:06.049Z [INFO]  server-e2627e83-ee3d-038d-dacc-bc36c8c8e19b: Installed remote snapshot
2026-04-23T00:15:06.049Z [INFO]  server-912ab360-8a3f-475f-8760-ed382d71e3c8: pipelining replication: peer="{Voter server-e2627e83-ee3d-038d-dacc-bc36c8c8e19b e2627e83-ee3d-038d-dacc-bc36c8c8e19b}"
2026-04-23T00:15:07.053Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:15:06.048856558 +0000 UTC m=+23.088457929 (2 nodes), 1.181µs from start of poll, 1.555305396s from cluster start. Timeout at 2026-04-23 00:15:07.053869061 +0000 UTC m=+24.093470421, 1.005012492s after stability
2026-04-23T00:15:07.066Z [INFO]  server-06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4: aborting pipeline replication: peer="{Voter server-f9e84588-cb86-2d48-4fdf-315cb172e745 f9e84588-cb86-2d48-4fdf-315cb172e745}"
2026-04-23T00:15:07.066Z [INFO]  server-06e7dfbd-45b6-a652-4d59-b2a94a0cb4c4: aborting pipeline replication: peer="{Voter server-1d56dbcd-171e-8168-6c49-856b56f12120 1d56dbcd-171e-8168-6c49-856b56f12120}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true
2026-04-23T00:15:07.068Z [INFO]  server-7b7cf9b8-9da2-03fb-ba13-fd4e46a7f088: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b7cf9b8-9da2-03fb-ba13-fd4e46a7f088 Address:7b7cf9b8-9da2-03fb-ba13-fd4e46a7f088} {Suffrage:Voter ID:server-92b223b8-0163-6c12-67ce-2fcd89d619fe Address:92b223b8-0163-6c12-67ce-2fcd89d619fe} {Suffrage:Voter ID:server-d78758e2-2f8d-a11c-0d51-48aa2f68a7ff Address:d78758e2-2f8d-a11c-0d51-48aa2f68a7ff}]"
2026-04-23T00:15:07.068Z [INFO]  server-92b223b8-0163-6c12-67ce-2fcd89d619fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b7cf9b8-9da2-03fb-ba13-fd4e46a7f088 Address:7b7cf9b8-9da2-03fb-ba13-fd4e46a7f088} {Suffrage:Voter ID:server-92b223b8-0163-6c12-67ce-2fcd89d619fe Address:92b223b8-0163-6c12-67ce-2fcd89d619fe} {Suffrage:Voter ID:server-d78758e2-2f8d-a11c-0d51-48aa2f68a7ff Address:d78758e2-2f8d-a11c-0d51-48aa2f68a7ff}]"
2026-04-23T00:15:07.068Z [INFO]  server-7b7cf9b8-9da2-03fb-ba13-fd4e46a7f088: entering follower state: follower="Node at 7b7cf9b8-9da2-03fb-ba13-fd4e46a7f088 [Follower]" leader-address= leader-id=
2026-04-23T00:15:07.068Z [INFO]  server-d78758e2-2f8d-a11c-0d51-48aa2f68a7ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7b7cf9b8-9da2-03fb-ba13-fd4e46a7f088 Address:7b7cf9b8-9da2-03fb-ba13-fd4e46a7f088} {Suffrage:Voter ID:server-92b223b8-0163-6c12-67ce-2fcd89d619fe Address:92b223b8-0163-6c12-67ce-2fcd89d619fe} {Suffrage:Voter ID:server-d78758e2-2f8d-a11c-0d51-48aa2f68a7ff Address:d78758e2-2f8d-a11c-0d51-48aa2f68a7ff}]"
2026-04-23T00:15:07.068Z [INFO]  server-92b223b8-0163-6c12-67ce-2fcd89d619fe: entering follower state: follower="Node at 92b223b8-0163-6c12-67ce-2fcd89d619fe [Follower]" leader-address= leader-id=
2026-04-23T00:15:07.068Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:07.068Z [INFO]  server-d78758e2-2f8d-a11c-0d51-48aa2f68a7ff: entering follower state: follower="Node at d78758e2-2f8d-a11c-0d51-48aa2f68a7ff [Follower]" leader-address= leader-id=
2026-04-23T00:15:07.610Z [WARN]  server-d78758e2-2f8d-a11c-0d51-48aa2f68a7ff: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:07.610Z [INFO]  server-d78758e2-2f8d-a11c-0d51-48aa2f68a7ff: entering candidate state: node="Node at d78758e2-2f8d-a11c-0d51-48aa2f68a7ff [Candidate]" term=2
2026-04-23T00:15:07.610Z [INFO]  server-d78758e2-2f8d-a11c-0d51-48aa2f68a7ff: election won: term=2 tally=2
2026-04-23T00:15:07.610Z [INFO]  server-d78758e2-2f8d-a11c-0d51-48aa2f68a7ff: entering leader state: leader="Node at d78758e2-2f8d-a11c-0d51-48aa2f68a7ff [Leader]"
2026-04-23T00:15:07.610Z [INFO]  server-d78758e2-2f8d-a11c-0d51-48aa2f68a7ff: added peer, starting replication: peer=server-7b7cf9b8-9da2-03fb-ba13-fd4e46a7f088
2026-04-23T00:15:07.610Z [INFO]  server-d78758e2-2f8d-a11c-0d51-48aa2f68a7ff: added peer, starting replication: peer=server-92b223b8-0163-6c12-67ce-2fcd89d619fe
2026-04-23T00:15:07.610Z [INFO]  server-d78758e2-2f8d-a11c-0d51-48aa2f68a7ff: pipelining replication: peer="{Voter server-92b223b8-0163-6c12-67ce-2fcd89d619fe 92b223b8-0163-6c12-67ce-2fcd89d619fe}"
2026-04-23T00:15:07.610Z [INFO]  server-d78758e2-2f8d-a11c-0d51-48aa2f68a7ff: pipelining replication: peer="{Voter server-7b7cf9b8-9da2-03fb-ba13-fd4e46a7f088 7b7cf9b8-9da2-03fb-ba13-fd4e46a7f088}"
2026-04-23T00:15:08.616Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:07.610634236 +0000 UTC m=+24.650235608 (1 nodes), 542.341412ms from start of poll, 542.675962ms from cluster start. Timeout at 2026-04-23 00:15:08.615637096 +0000 UTC m=+25.655238448, 1.00500284s after stability
2026-04-23T00:15:08.616Z [INFO]  server-d78758e2-2f8d-a11c-0d51-48aa2f68a7ff: starting snapshot up to: index=12
2026-04-23T00:15:08.616Z [INFO]  creating new snapshot: path=/tmp/raft2166989727/snapshots/2-12-1776903308616.tmp
2026-04-23T00:15:08.617Z [INFO]  server-d78758e2-2f8d-a11c-0d51-48aa2f68a7ff: snapshot complete up to: index=12
2026-04-23T00:15:08.618Z [INFO]  server-e58f9c9e-ba99-1917-0137-856685ab73c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e58f9c9e-ba99-1917-0137-856685ab73c8 Address:e58f9c9e-ba99-1917-0137-856685ab73c8} {Suffrage:Voter ID:server-16807a39-25c1-5560-ef53-a661b6b6a7dc Address:16807a39-25c1-5560-ef53-a661b6b6a7dc} {Suffrage:Voter ID:server-81aaae29-4a5a-2fea-22a2-ab719a7e0e67 Address:81aaae29-4a5a-2fea-22a2-ab719a7e0e67}]"
2026-04-23T00:15:08.619Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e58f9c9e-ba99-1917-0137-856685ab73c8 Address:e58f9c9e-ba99-1917-0137-856685ab73c8} {Suffrage:Voter ID:server-16807a39-25c1-5560-ef53-a661b6b6a7dc Address:16807a39-25c1-5560-ef53-a661b6b6a7dc} {Suffrage:Voter ID:server-81aaae29-4a5a-2fea-22a2-ab719a7e0e67 Address:81aaae29-4a5a-2fea-22a2-ab719a7e0e67}]"
2026-04-23T00:15:08.619Z [INFO]  server-e58f9c9e-ba99-1917-0137-856685ab73c8: entering follower state: follower="Node at e58f9c9e-ba99-1917-0137-856685ab73c8 [Follower]" leader-address= leader-id=
2026-04-23T00:15:08.619Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: entering follower state: follower="Node at 16807a39-25c1-5560-ef53-a661b6b6a7dc [Follower]" leader-address= leader-id=
2026-04-23T00:15:08.619Z [INFO]  server-81aaae29-4a5a-2fea-22a2-ab719a7e0e67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e58f9c9e-ba99-1917-0137-856685ab73c8 Address:e58f9c9e-ba99-1917-0137-856685ab73c8} {Suffrage:Voter ID:server-16807a39-25c1-5560-ef53-a661b6b6a7dc Address:16807a39-25c1-5560-ef53-a661b6b6a7dc} {Suffrage:Voter ID:server-81aaae29-4a5a-2fea-22a2-ab719a7e0e67 Address:81aaae29-4a5a-2fea-22a2-ab719a7e0e67}]"
2026-04-23T00:15:08.619Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:08.619Z [INFO]  server-81aaae29-4a5a-2fea-22a2-ab719a7e0e67: entering follower state: follower="Node at 81aaae29-4a5a-2fea-22a2-ab719a7e0e67 [Follower]" leader-address= leader-id=
2026-04-23T00:15:09.253Z [WARN]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:09.253Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: entering candidate state: node="Node at 16807a39-25c1-5560-ef53-a661b6b6a7dc [Candidate]" term=2
2026-04-23T00:15:09.253Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: election won: term=2 tally=2
2026-04-23T00:15:09.253Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: entering leader state: leader="Node at 16807a39-25c1-5560-ef53-a661b6b6a7dc [Leader]"
2026-04-23T00:15:09.253Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: added peer, starting replication: peer=server-e58f9c9e-ba99-1917-0137-856685ab73c8
2026-04-23T00:15:09.253Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: added peer, starting replication: peer=server-81aaae29-4a5a-2fea-22a2-ab719a7e0e67
2026-04-23T00:15:09.253Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: pipelining replication: peer="{Voter server-81aaae29-4a5a-2fea-22a2-ab719a7e0e67 81aaae29-4a5a-2fea-22a2-ab719a7e0e67}"
2026-04-23T00:15:09.253Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: pipelining replication: peer="{Voter server-e58f9c9e-ba99-1917-0137-856685ab73c8 e58f9c9e-ba99-1917-0137-856685ab73c8}"
2026-04-23T00:15:10.258Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:09.253337511 +0000 UTC m=+26.292938869 (1 nodes), 634.07648ms from start of poll, 634.494186ms from cluster start. Timeout at 2026-04-23 00:15:10.258341833 +0000 UTC m=+27.297943257, 1.005004388s after stability
2026-04-23T00:15:10.259Z [INFO]  creating new snapshot: path=/tmp/raft382756064/snapshots/2-13-1776903310259.tmp
2026-04-23T00:15:10.260Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: copied to local snapshot: bytes=71
2026-04-23T00:15:10.260Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: snapshot restore progress: id=2-13-1776903310259 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:10.260Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: removing all old logs from log store
2026-04-23T00:15:10.260Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: compacting logs: from=1 to=2
2026-04-23T00:15:10.260Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: restored user snapshot: index=13
2026-04-23T00:15:10.260Z [ERROR] server-16807a39-25c1-5560-ef53-a661b6b6a7dc: failed to get log: index=2 error="log not found"
2026-04-23T00:15:10.260Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: aborting pipeline replication: peer="{Voter server-81aaae29-4a5a-2fea-22a2-ab719a7e0e67 81aaae29-4a5a-2fea-22a2-ab719a7e0e67}"
2026-04-23T00:15:10.260Z [ERROR] server-16807a39-25c1-5560-ef53-a661b6b6a7dc: failed to get log: index=2 error="log not found"
2026-04-23T00:15:10.260Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: aborting pipeline replication: peer="{Voter server-e58f9c9e-ba99-1917-0137-856685ab73c8 e58f9c9e-ba99-1917-0137-856685ab73c8}"
2026-04-23T00:15:10.266Z [ERROR] server-16807a39-25c1-5560-ef53-a661b6b6a7dc: failed to get log: index=2 error="log not found"
2026-04-23T00:15:10.266Z [INFO]  creating new snapshot: path=/tmp/raft3383082218/snapshots/2-13-1776903310266.tmp
2026-04-23T00:15:10.267Z [INFO]  server-81aaae29-4a5a-2fea-22a2-ab719a7e0e67: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:10.267Z [INFO]  server-81aaae29-4a5a-2fea-22a2-ab719a7e0e67: copied to local snapshot: bytes=71
2026-04-23T00:15:10.267Z [INFO]  server-81aaae29-4a5a-2fea-22a2-ab719a7e0e67: snapshot restore progress: id=2-13-1776903310266 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:10.267Z [INFO]  server-81aaae29-4a5a-2fea-22a2-ab719a7e0e67: removing all old logs from log store
2026-04-23T00:15:10.267Z [INFO]  server-81aaae29-4a5a-2fea-22a2-ab719a7e0e67: compacting logs: from=1 to=2
2026-04-23T00:15:10.267Z [INFO]  server-81aaae29-4a5a-2fea-22a2-ab719a7e0e67: Installed remote snapshot
2026-04-23T00:15:10.267Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: pipelining replication: peer="{Voter server-81aaae29-4a5a-2fea-22a2-ab719a7e0e67 81aaae29-4a5a-2fea-22a2-ab719a7e0e67}"
2026-04-23T00:15:10.267Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:15:10.270Z [ERROR] server-16807a39-25c1-5560-ef53-a661b6b6a7dc: failed to get log: index=2 error="log not found"
2026-04-23T00:15:10.270Z [INFO]  creating new snapshot: path=/tmp/raft985316626/snapshots/2-13-1776903310270.tmp
2026-04-23T00:15:10.270Z [INFO]  server-e58f9c9e-ba99-1917-0137-856685ab73c8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:10.271Z [INFO]  server-e58f9c9e-ba99-1917-0137-856685ab73c8: copied to local snapshot: bytes=71
2026-04-23T00:15:10.271Z [INFO]  server-e58f9c9e-ba99-1917-0137-856685ab73c8: snapshot restore progress: id=2-13-1776903310270 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:10.271Z [INFO]  server-e58f9c9e-ba99-1917-0137-856685ab73c8: removing all old logs from log store
2026-04-23T00:15:10.271Z [INFO]  server-e58f9c9e-ba99-1917-0137-856685ab73c8: compacting logs: from=1 to=2
2026-04-23T00:15:10.271Z [INFO]  server-e58f9c9e-ba99-1917-0137-856685ab73c8: Installed remote snapshot
2026-04-23T00:15:10.271Z [INFO]  server-16807a39-25c1-5560-ef53-a661b6b6a7dc: pipelining replication: peer="{Voter server-e58f9c9e-ba99-1917-0137-856685ab73c8 e58f9c9e-ba99-1917-0137-856685ab73c8}"
2026-04-23T00:15:11.273Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:15:10.267959102 +0000 UTC m=+27.307560484 (2 nodes), 729ns from start of poll, 1.649115801s from cluster start. Timeout at 2026-04-23 00:15:11.272973901 +0000 UTC m=+28.312575295, 1.005014811s after stability
2026-04-23T00:15:11.285Z [INFO]  server-d78758e2-2f8d-a11c-0d51-48aa2f68a7ff: aborting pipeline replication: peer="{Voter server-92b223b8-0163-6c12-67ce-2fcd89d619fe 92b223b8-0163-6c12-67ce-2fcd89d619fe}"
2026-04-23T00:15:11.285Z [INFO]  server-d78758e2-2f8d-a11c-0d51-48aa2f68a7ff: aborting pipeline replication: peer="{Voter server-7b7cf9b8-9da2-03fb-ba13-fd4e46a7f088 7b7cf9b8-9da2-03fb-ba13-fd4e46a7f088}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false
2026-04-23T00:15:11.287Z [INFO]  server-7ecd3a18-677f-3ec4-dc2b-c16541d00251: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ecd3a18-677f-3ec4-dc2b-c16541d00251 Address:7ecd3a18-677f-3ec4-dc2b-c16541d00251} {Suffrage:Voter ID:server-faec1d95-05d1-184a-6716-1b15d6a4c32b Address:faec1d95-05d1-184a-6716-1b15d6a4c32b} {Suffrage:Voter ID:server-d41d19de-fa25-b77a-4c61-c6d59e50563a Address:d41d19de-fa25-b77a-4c61-c6d59e50563a}]"
2026-04-23T00:15:11.287Z [INFO]  server-7ecd3a18-677f-3ec4-dc2b-c16541d00251: entering follower state: follower="Node at 7ecd3a18-677f-3ec4-dc2b-c16541d00251 [Follower]" leader-address= leader-id=
2026-04-23T00:15:11.287Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ecd3a18-677f-3ec4-dc2b-c16541d00251 Address:7ecd3a18-677f-3ec4-dc2b-c16541d00251} {Suffrage:Voter ID:server-faec1d95-05d1-184a-6716-1b15d6a4c32b Address:faec1d95-05d1-184a-6716-1b15d6a4c32b} {Suffrage:Voter ID:server-d41d19de-fa25-b77a-4c61-c6d59e50563a Address:d41d19de-fa25-b77a-4c61-c6d59e50563a}]"
2026-04-23T00:15:11.287Z [INFO]  server-d41d19de-fa25-b77a-4c61-c6d59e50563a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7ecd3a18-677f-3ec4-dc2b-c16541d00251 Address:7ecd3a18-677f-3ec4-dc2b-c16541d00251} {Suffrage:Voter ID:server-faec1d95-05d1-184a-6716-1b15d6a4c32b Address:faec1d95-05d1-184a-6716-1b15d6a4c32b} {Suffrage:Voter ID:server-d41d19de-fa25-b77a-4c61-c6d59e50563a Address:d41d19de-fa25-b77a-4c61-c6d59e50563a}]"
2026-04-23T00:15:11.287Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:11.287Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: entering follower state: follower="Node at faec1d95-05d1-184a-6716-1b15d6a4c32b [Follower]" leader-address= leader-id=
2026-04-23T00:15:11.287Z [INFO]  server-d41d19de-fa25-b77a-4c61-c6d59e50563a: entering follower state: follower="Node at d41d19de-fa25-b77a-4c61-c6d59e50563a [Follower]" leader-address= leader-id=
2026-04-23T00:15:11.821Z [WARN]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:11.821Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: entering candidate state: node="Node at faec1d95-05d1-184a-6716-1b15d6a4c32b [Candidate]" term=2
2026-04-23T00:15:11.822Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: election won: term=2 tally=2
2026-04-23T00:15:11.822Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: entering leader state: leader="Node at faec1d95-05d1-184a-6716-1b15d6a4c32b [Leader]"
2026-04-23T00:15:11.822Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: added peer, starting replication: peer=server-7ecd3a18-677f-3ec4-dc2b-c16541d00251
2026-04-23T00:15:11.822Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: added peer, starting replication: peer=server-d41d19de-fa25-b77a-4c61-c6d59e50563a
2026-04-23T00:15:11.822Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: pipelining replication: peer="{Voter server-7ecd3a18-677f-3ec4-dc2b-c16541d00251 7ecd3a18-677f-3ec4-dc2b-c16541d00251}"
2026-04-23T00:15:11.822Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: pipelining replication: peer="{Voter server-d41d19de-fa25-b77a-4c61-c6d59e50563a d41d19de-fa25-b77a-4c61-c6d59e50563a}"
2026-04-23T00:15:12.828Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:11.822166025 +0000 UTC m=+28.861767382 (1 nodes), 534.473693ms from start of poll, 534.867334ms from cluster start. Timeout at 2026-04-23 00:15:12.827167274 +0000 UTC m=+29.866768631, 1.005001249s after stability
2026-04-23T00:15:12.828Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: starting snapshot up to: index=12
2026-04-23T00:15:12.828Z [INFO]  creating new snapshot: path=/tmp/raft477441200/snapshots/2-12-1776903312828.tmp
2026-04-23T00:15:12.829Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: snapshot complete up to: index=12
2026-04-23T00:15:12.829Z [INFO]  creating new snapshot: path=/tmp/raft477441200/snapshots/2-23-1776903312829.tmp
2026-04-23T00:15:12.830Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: copied to local snapshot: bytes=71
2026-04-23T00:15:12.830Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: snapshot restore progress: id=2-23-1776903312829 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:12.830Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: restored user snapshot: index=23
2026-04-23T00:15:12.830Z [ERROR] server-faec1d95-05d1-184a-6716-1b15d6a4c32b: failed to get log: index=23 error="log not found"
2026-04-23T00:15:12.830Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: aborting pipeline replication: peer="{Voter server-d41d19de-fa25-b77a-4c61-c6d59e50563a d41d19de-fa25-b77a-4c61-c6d59e50563a}"
2026-04-23T00:15:12.830Z [ERROR] server-faec1d95-05d1-184a-6716-1b15d6a4c32b: failed to get log: index=23 error="log not found"
2026-04-23T00:15:12.830Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: aborting pipeline replication: peer="{Voter server-7ecd3a18-677f-3ec4-dc2b-c16541d00251 7ecd3a18-677f-3ec4-dc2b-c16541d00251}"
2026-04-23T00:15:12.837Z [ERROR] server-faec1d95-05d1-184a-6716-1b15d6a4c32b: failed to get log: index=23 error="log not found"
2026-04-23T00:15:12.838Z [INFO]  creating new snapshot: path=/tmp/raft3788517509/snapshots/2-23-1776903312838.tmp
2026-04-23T00:15:12.838Z [INFO]  server-7ecd3a18-677f-3ec4-dc2b-c16541d00251: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:12.838Z [INFO]  server-7ecd3a18-677f-3ec4-dc2b-c16541d00251: copied to local snapshot: bytes=71
2026-04-23T00:15:12.838Z [INFO]  server-7ecd3a18-677f-3ec4-dc2b-c16541d00251: snapshot restore progress: id=2-23-1776903312838 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:12.838Z [INFO]  server-7ecd3a18-677f-3ec4-dc2b-c16541d00251: Installed remote snapshot
2026-04-23T00:15:12.839Z [WARN]  server-7ecd3a18-677f-3ec4-dc2b-c16541d00251: skipping application of old log: index=22
2026-04-23T00:15:12.839Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: pipelining replication: peer="{Voter server-7ecd3a18-677f-3ec4-dc2b-c16541d00251 7ecd3a18-677f-3ec4-dc2b-c16541d00251}"
2026-04-23T00:15:12.839Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:15:12.839Z [ERROR] server-faec1d95-05d1-184a-6716-1b15d6a4c32b: failed to get log: index=23 error="log not found"
2026-04-23T00:15:12.840Z [INFO]  creating new snapshot: path=/tmp/raft361937270/snapshots/2-23-1776903312840.tmp
2026-04-23T00:15:12.841Z [INFO]  server-d41d19de-fa25-b77a-4c61-c6d59e50563a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:12.841Z [INFO]  server-d41d19de-fa25-b77a-4c61-c6d59e50563a: copied to local snapshot: bytes=71
2026-04-23T00:15:12.842Z [INFO]  server-d41d19de-fa25-b77a-4c61-c6d59e50563a: snapshot restore progress: id=2-23-1776903312840 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:12.842Z [INFO]  server-d41d19de-fa25-b77a-4c61-c6d59e50563a: Installed remote snapshot
2026-04-23T00:15:12.842Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: pipelining replication: peer="{Voter server-d41d19de-fa25-b77a-4c61-c6d59e50563a d41d19de-fa25-b77a-4c61-c6d59e50563a}"
2026-04-23T00:15:13.844Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:15:12.839157737 +0000 UTC m=+29.878759098 (2 nodes), 975ns from start of poll, 1.55185905s from cluster start. Timeout at 2026-04-23 00:15:13.844167773 +0000 UTC m=+30.883769144, 1.005010046s after stability
2026-04-23T00:15:13.857Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: aborting pipeline replication: peer="{Voter server-d41d19de-fa25-b77a-4c61-c6d59e50563a d41d19de-fa25-b77a-4c61-c6d59e50563a}"
2026-04-23T00:15:13.857Z [INFO]  server-faec1d95-05d1-184a-6716-1b15d6a4c32b: aborting pipeline replication: peer="{Voter server-7ecd3a18-677f-3ec4-dc2b-c16541d00251 7ecd3a18-677f-3ec4-dc2b-c16541d00251}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false
2026-04-23T00:15:13.858Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92f57bdb-bde2-7f69-c0b3-2e115a585421 Address:92f57bdb-bde2-7f69-c0b3-2e115a585421} {Suffrage:Voter ID:server-4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f Address:4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f} {Suffrage:Voter ID:server-566d7d6b-fd3c-0701-844b-bd41a83a50f8 Address:566d7d6b-fd3c-0701-844b-bd41a83a50f8}]"
2026-04-23T00:15:13.858Z [INFO]  server-4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92f57bdb-bde2-7f69-c0b3-2e115a585421 Address:92f57bdb-bde2-7f69-c0b3-2e115a585421} {Suffrage:Voter ID:server-4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f Address:4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f} {Suffrage:Voter ID:server-566d7d6b-fd3c-0701-844b-bd41a83a50f8 Address:566d7d6b-fd3c-0701-844b-bd41a83a50f8}]"
2026-04-23T00:15:13.858Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: entering follower state: follower="Node at 92f57bdb-bde2-7f69-c0b3-2e115a585421 [Follower]" leader-address= leader-id=
2026-04-23T00:15:13.858Z [INFO]  server-4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f: entering follower state: follower="Node at 4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f [Follower]" leader-address= leader-id=
2026-04-23T00:15:13.858Z [INFO]  server-566d7d6b-fd3c-0701-844b-bd41a83a50f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92f57bdb-bde2-7f69-c0b3-2e115a585421 Address:92f57bdb-bde2-7f69-c0b3-2e115a585421} {Suffrage:Voter ID:server-4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f Address:4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f} {Suffrage:Voter ID:server-566d7d6b-fd3c-0701-844b-bd41a83a50f8 Address:566d7d6b-fd3c-0701-844b-bd41a83a50f8}]"
2026-04-23T00:15:13.858Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:13.858Z [INFO]  server-566d7d6b-fd3c-0701-844b-bd41a83a50f8: entering follower state: follower="Node at 566d7d6b-fd3c-0701-844b-bd41a83a50f8 [Follower]" leader-address= leader-id=
2026-04-23T00:15:14.485Z [WARN]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:14.485Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: entering candidate state: node="Node at 92f57bdb-bde2-7f69-c0b3-2e115a585421 [Candidate]" term=2
2026-04-23T00:15:14.485Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: election won: term=2 tally=2
2026-04-23T00:15:14.485Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: entering leader state: leader="Node at 92f57bdb-bde2-7f69-c0b3-2e115a585421 [Leader]"
2026-04-23T00:15:14.485Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: added peer, starting replication: peer=server-4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f
2026-04-23T00:15:14.485Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: added peer, starting replication: peer=server-566d7d6b-fd3c-0701-844b-bd41a83a50f8
2026-04-23T00:15:14.485Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: pipelining replication: peer="{Voter server-4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f 4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f}"
2026-04-23T00:15:14.485Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: pipelining replication: peer="{Voter server-566d7d6b-fd3c-0701-844b-bd41a83a50f8 566d7d6b-fd3c-0701-844b-bd41a83a50f8}"
2026-04-23T00:15:15.491Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:14.485911082 +0000 UTC m=+31.525512437 (1 nodes), 627.026729ms from start of poll, 627.568001ms from cluster start. Timeout at 2026-04-23 00:15:15.490912633 +0000 UTC m=+32.530513985, 1.005001548s after stability
2026-04-23T00:15:15.491Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: starting snapshot up to: index=12
2026-04-23T00:15:15.491Z [INFO]  creating new snapshot: path=/tmp/raft2392116872/snapshots/2-12-1776903315491.tmp
2026-04-23T00:15:15.492Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: snapshot complete up to: index=12
2026-04-23T00:15:15.493Z [INFO]  creating new snapshot: path=/tmp/raft2392116872/snapshots/2-23-1776903315493.tmp
2026-04-23T00:15:15.493Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: copied to local snapshot: bytes=71
2026-04-23T00:15:15.493Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: snapshot restore progress: id=2-23-1776903315493 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:15.494Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: removing all old logs from log store
2026-04-23T00:15:15.494Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: compacting logs: from=1 to=22
2026-04-23T00:15:15.494Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: restored user snapshot: index=23
2026-04-23T00:15:15.494Z [ERROR] server-92f57bdb-bde2-7f69-c0b3-2e115a585421: failed to get log: index=22 error="log not found"
2026-04-23T00:15:15.494Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: aborting pipeline replication: peer="{Voter server-566d7d6b-fd3c-0701-844b-bd41a83a50f8 566d7d6b-fd3c-0701-844b-bd41a83a50f8}"
2026-04-23T00:15:15.494Z [ERROR] server-92f57bdb-bde2-7f69-c0b3-2e115a585421: failed to get log: index=22 error="log not found"
2026-04-23T00:15:15.494Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: aborting pipeline replication: peer="{Voter server-4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f 4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f}"
2026-04-23T00:15:15.500Z [ERROR] server-92f57bdb-bde2-7f69-c0b3-2e115a585421: failed to get log: index=22 error="log not found"
2026-04-23T00:15:15.500Z [INFO]  creating new snapshot: path=/tmp/raft3131951434/snapshots/2-23-1776903315500.tmp
2026-04-23T00:15:15.500Z [INFO]  server-4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:15.501Z [INFO]  server-4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f: copied to local snapshot: bytes=71
2026-04-23T00:15:15.501Z [INFO]  server-4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f: snapshot restore progress: id=2-23-1776903315500 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:15.501Z [INFO]  server-4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f: removing all old logs from log store
2026-04-23T00:15:15.501Z [INFO]  server-4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f: compacting logs: from=1 to=22
2026-04-23T00:15:15.501Z [INFO]  server-4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f: Installed remote snapshot
2026-04-23T00:15:15.501Z [WARN]  server-4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f: skipping application of old log: index=22
2026-04-23T00:15:15.501Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: pipelining replication: peer="{Voter server-4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f 4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f}"
2026-04-23T00:15:15.501Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:15:15.503Z [ERROR] server-92f57bdb-bde2-7f69-c0b3-2e115a585421: failed to get log: index=22 error="log not found"
2026-04-23T00:15:15.503Z [INFO]  creating new snapshot: path=/tmp/raft541184466/snapshots/2-23-1776903315503.tmp
2026-04-23T00:15:15.503Z [INFO]  server-566d7d6b-fd3c-0701-844b-bd41a83a50f8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:15.504Z [INFO]  server-566d7d6b-fd3c-0701-844b-bd41a83a50f8: copied to local snapshot: bytes=71
2026-04-23T00:15:15.504Z [INFO]  server-566d7d6b-fd3c-0701-844b-bd41a83a50f8: snapshot restore progress: id=2-23-1776903315503 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:15.504Z [INFO]  server-566d7d6b-fd3c-0701-844b-bd41a83a50f8: removing all old logs from log store
2026-04-23T00:15:15.504Z [INFO]  server-566d7d6b-fd3c-0701-844b-bd41a83a50f8: compacting logs: from=1 to=22
2026-04-23T00:15:15.504Z [INFO]  server-566d7d6b-fd3c-0701-844b-bd41a83a50f8: Installed remote snapshot
2026-04-23T00:15:15.504Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: pipelining replication: peer="{Voter server-566d7d6b-fd3c-0701-844b-bd41a83a50f8 566d7d6b-fd3c-0701-844b-bd41a83a50f8}"
2026-04-23T00:15:16.507Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:15:15.501372729 +0000 UTC m=+32.540974081 (2 nodes), 794ns from start of poll, 1.643029645s from cluster start. Timeout at 2026-04-23 00:15:16.506377056 +0000 UTC m=+33.545978409, 1.005004328s after stability
2026-04-23T00:15:16.520Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: aborting pipeline replication: peer="{Voter server-566d7d6b-fd3c-0701-844b-bd41a83a50f8 566d7d6b-fd3c-0701-844b-bd41a83a50f8}"
2026-04-23T00:15:16.520Z [INFO]  server-92f57bdb-bde2-7f69-c0b3-2e115a585421: aborting pipeline replication: peer="{Voter server-4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f 4b8bd85a-b52b-c953-2e5e-35b4ae2fab3f}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true
2026-04-23T00:15:16.522Z [INFO]  server-e765fdf8-56ff-0b93-ae22-7f4d9719604c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e765fdf8-56ff-0b93-ae22-7f4d9719604c Address:e765fdf8-56ff-0b93-ae22-7f4d9719604c} {Suffrage:Voter ID:server-8b5c7330-3bda-303d-0ad1-4bab8881557d Address:8b5c7330-3bda-303d-0ad1-4bab8881557d} {Suffrage:Voter ID:server-e4a98f38-798d-c27b-7faa-4567062cf790 Address:e4a98f38-798d-c27b-7faa-4567062cf790}]"
2026-04-23T00:15:16.522Z [INFO]  server-e765fdf8-56ff-0b93-ae22-7f4d9719604c: entering follower state: follower="Node at e765fdf8-56ff-0b93-ae22-7f4d9719604c [Follower]" leader-address= leader-id=
2026-04-23T00:15:16.522Z [INFO]  server-8b5c7330-3bda-303d-0ad1-4bab8881557d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e765fdf8-56ff-0b93-ae22-7f4d9719604c Address:e765fdf8-56ff-0b93-ae22-7f4d9719604c} {Suffrage:Voter ID:server-8b5c7330-3bda-303d-0ad1-4bab8881557d Address:8b5c7330-3bda-303d-0ad1-4bab8881557d} {Suffrage:Voter ID:server-e4a98f38-798d-c27b-7faa-4567062cf790 Address:e4a98f38-798d-c27b-7faa-4567062cf790}]"
2026-04-23T00:15:16.522Z [INFO]  server-8b5c7330-3bda-303d-0ad1-4bab8881557d: entering follower state: follower="Node at 8b5c7330-3bda-303d-0ad1-4bab8881557d [Follower]" leader-address= leader-id=
2026-04-23T00:15:16.522Z [INFO]  server-e4a98f38-798d-c27b-7faa-4567062cf790: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e765fdf8-56ff-0b93-ae22-7f4d9719604c Address:e765fdf8-56ff-0b93-ae22-7f4d9719604c} {Suffrage:Voter ID:server-8b5c7330-3bda-303d-0ad1-4bab8881557d Address:8b5c7330-3bda-303d-0ad1-4bab8881557d} {Suffrage:Voter ID:server-e4a98f38-798d-c27b-7faa-4567062cf790 Address:e4a98f38-798d-c27b-7faa-4567062cf790}]"
2026-04-23T00:15:16.522Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:16.522Z [INFO]  server-e4a98f38-798d-c27b-7faa-4567062cf790: entering follower state: follower="Node at e4a98f38-798d-c27b-7faa-4567062cf790 [Follower]" leader-address= leader-id=
2026-04-23T00:15:17.263Z [WARN]  server-e4a98f38-798d-c27b-7faa-4567062cf790: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:17.263Z [INFO]  server-e4a98f38-798d-c27b-7faa-4567062cf790: entering candidate state: node="Node at e4a98f38-798d-c27b-7faa-4567062cf790 [Candidate]" term=2
2026-04-23T00:15:17.263Z [INFO]  server-e4a98f38-798d-c27b-7faa-4567062cf790: election won: term=2 tally=2
2026-04-23T00:15:17.263Z [INFO]  server-e4a98f38-798d-c27b-7faa-4567062cf790: entering leader state: leader="Node at e4a98f38-798d-c27b-7faa-4567062cf790 [Leader]"
2026-04-23T00:15:17.263Z [INFO]  server-e4a98f38-798d-c27b-7faa-4567062cf790: added peer, starting replication: peer=server-e765fdf8-56ff-0b93-ae22-7f4d9719604c
2026-04-23T00:15:17.263Z [INFO]  server-e4a98f38-798d-c27b-7faa-4567062cf790: added peer, starting replication: peer=server-8b5c7330-3bda-303d-0ad1-4bab8881557d
2026-04-23T00:15:17.263Z [INFO]  server-e4a98f38-798d-c27b-7faa-4567062cf790: pipelining replication: peer="{Voter server-e765fdf8-56ff-0b93-ae22-7f4d9719604c e765fdf8-56ff-0b93-ae22-7f4d9719604c}"
2026-04-23T00:15:17.263Z [INFO]  server-e4a98f38-798d-c27b-7faa-4567062cf790: pipelining replication: peer="{Voter server-8b5c7330-3bda-303d-0ad1-4bab8881557d 8b5c7330-3bda-303d-0ad1-4bab8881557d}"
2026-04-23T00:15:18.269Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:17.263838875 +0000 UTC m=+34.303440232 (1 nodes), 741.57956ms from start of poll, 741.918401ms from cluster start. Timeout at 2026-04-23 00:15:18.268841532 +0000 UTC m=+35.308442888, 1.005002656s after stability
2026-04-23T00:15:18.270Z [INFO]  server-e4a98f38-798d-c27b-7faa-4567062cf790: starting snapshot up to: index=12
2026-04-23T00:15:18.270Z [INFO]  creating new snapshot: path=/tmp/raft1981588222/snapshots/2-12-1776903318270.tmp
2026-04-23T00:15:18.270Z [INFO]  server-e4a98f38-798d-c27b-7faa-4567062cf790: snapshot complete up to: index=12
2026-04-23T00:15:18.272Z [INFO]  server-1b56a842-8203-7c38-03df-40de43715661: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b56a842-8203-7c38-03df-40de43715661 Address:1b56a842-8203-7c38-03df-40de43715661} {Suffrage:Voter ID:server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9 Address:e24ce48f-60a3-3394-b17c-441ae7b2a8b9} {Suffrage:Voter ID:server-ec524619-6127-8462-ea71-a06367486ba1 Address:ec524619-6127-8462-ea71-a06367486ba1}]"
2026-04-23T00:15:18.272Z [INFO]  server-1b56a842-8203-7c38-03df-40de43715661: entering follower state: follower="Node at 1b56a842-8203-7c38-03df-40de43715661 [Follower]" leader-address= leader-id=
2026-04-23T00:15:18.272Z [INFO]  server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b56a842-8203-7c38-03df-40de43715661 Address:1b56a842-8203-7c38-03df-40de43715661} {Suffrage:Voter ID:server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9 Address:e24ce48f-60a3-3394-b17c-441ae7b2a8b9} {Suffrage:Voter ID:server-ec524619-6127-8462-ea71-a06367486ba1 Address:ec524619-6127-8462-ea71-a06367486ba1}]"
2026-04-23T00:15:18.272Z [INFO]  server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: entering follower state: follower="Node at e24ce48f-60a3-3394-b17c-441ae7b2a8b9 [Follower]" leader-address= leader-id=
2026-04-23T00:15:18.272Z [INFO]  server-ec524619-6127-8462-ea71-a06367486ba1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b56a842-8203-7c38-03df-40de43715661 Address:1b56a842-8203-7c38-03df-40de43715661} {Suffrage:Voter ID:server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9 Address:e24ce48f-60a3-3394-b17c-441ae7b2a8b9} {Suffrage:Voter ID:server-ec524619-6127-8462-ea71-a06367486ba1 Address:ec524619-6127-8462-ea71-a06367486ba1}]"
2026-04-23T00:15:18.273Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:18.273Z [INFO]  server-ec524619-6127-8462-ea71-a06367486ba1: entering follower state: follower="Node at ec524619-6127-8462-ea71-a06367486ba1 [Follower]" leader-address= leader-id=
2026-04-23T00:15:19.067Z [WARN]  server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:19.067Z [INFO]  server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: entering candidate state: node="Node at e24ce48f-60a3-3394-b17c-441ae7b2a8b9 [Candidate]" term=2
2026-04-23T00:15:19.067Z [INFO]  server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: election won: term=2 tally=2
2026-04-23T00:15:19.067Z [INFO]  server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: entering leader state: leader="Node at e24ce48f-60a3-3394-b17c-441ae7b2a8b9 [Leader]"
2026-04-23T00:15:19.067Z [INFO]  server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: added peer, starting replication: peer=server-1b56a842-8203-7c38-03df-40de43715661
2026-04-23T00:15:19.067Z [INFO]  server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: added peer, starting replication: peer=server-ec524619-6127-8462-ea71-a06367486ba1
2026-04-23T00:15:19.067Z [INFO]  server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: pipelining replication: peer="{Voter server-ec524619-6127-8462-ea71-a06367486ba1 ec524619-6127-8462-ea71-a06367486ba1}"
2026-04-23T00:15:19.067Z [INFO]  server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: pipelining replication: peer="{Voter server-1b56a842-8203-7c38-03df-40de43715661 1b56a842-8203-7c38-03df-40de43715661}"
2026-04-23T00:15:20.072Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:19.067837685 +0000 UTC m=+36.107439069 (1 nodes), 794.766281ms from start of poll, 795.16618ms from cluster start. Timeout at 2026-04-23 00:15:20.072839913 +0000 UTC m=+37.112441306, 1.005002237s after stability
2026-04-23T00:15:20.337Z [INFO]  creating new snapshot: path=/tmp/raft3550555479/snapshots/2-14-1776903320337.tmp
2026-04-23T00:15:20.338Z [INFO]  server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: copied to local snapshot: bytes=71
2026-04-23T00:15:20.338Z [INFO]  server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: snapshot restore progress: id=2-14-1776903320337 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:20.338Z [INFO]  server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: restored user snapshot: index=14
2026-04-23T00:15:20.338Z [ERROR] server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: failed to get log: index=3 error="log not found"
2026-04-23T00:15:20.338Z [INFO]  server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: aborting pipeline replication: peer="{Voter server-ec524619-6127-8462-ea71-a06367486ba1 ec524619-6127-8462-ea71-a06367486ba1}"
2026-04-23T00:15:20.338Z [ERROR] server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: failed to get log: index=3 error="log not found"
2026-04-23T00:15:20.338Z [INFO]  server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: aborting pipeline replication: peer="{Voter server-1b56a842-8203-7c38-03df-40de43715661 1b56a842-8203-7c38-03df-40de43715661}"
2026-04-23T00:15:20.344Z [ERROR] server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: failed to get log: index=3 error="log not found"
2026-04-23T00:15:20.344Z [INFO]  creating new snapshot: path=/tmp/raft2541828584/snapshots/2-14-1776903320344.tmp
2026-04-23T00:15:20.345Z [INFO]  server-1b56a842-8203-7c38-03df-40de43715661: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:20.345Z [INFO]  server-1b56a842-8203-7c38-03df-40de43715661: copied to local snapshot: bytes=71
2026-04-23T00:15:20.345Z [INFO]  server-1b56a842-8203-7c38-03df-40de43715661: snapshot restore progress: id=2-14-1776903320344 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:20.345Z [INFO]  server-1b56a842-8203-7c38-03df-40de43715661: Installed remote snapshot
2026-04-23T00:15:20.345Z [INFO]  server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: pipelining replication: peer="{Voter server-1b56a842-8203-7c38-03df-40de43715661 1b56a842-8203-7c38-03df-40de43715661}"
2026-04-23T00:15:20.345Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:15:20.347Z [ERROR] server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: failed to get log: index=3 error="log not found"
2026-04-23T00:15:20.348Z [INFO]  creating new snapshot: path=/tmp/raft3423356657/snapshots/2-14-1776903320348.tmp
2026-04-23T00:15:20.349Z [INFO]  server-ec524619-6127-8462-ea71-a06367486ba1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:20.349Z [INFO]  server-ec524619-6127-8462-ea71-a06367486ba1: copied to local snapshot: bytes=71
2026-04-23T00:15:20.349Z [INFO]  server-ec524619-6127-8462-ea71-a06367486ba1: snapshot restore progress: id=2-14-1776903320348 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:20.349Z [INFO]  server-ec524619-6127-8462-ea71-a06367486ba1: Installed remote snapshot
2026-04-23T00:15:20.349Z [INFO]  server-e24ce48f-60a3-3394-b17c-441ae7b2a8b9: pipelining replication: peer="{Voter server-ec524619-6127-8462-ea71-a06367486ba1 ec524619-6127-8462-ea71-a06367486ba1}"
2026-04-23T00:15:21.351Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:15:20.345966684 +0000 UTC m=+37.385568049 (2 nodes), 790ns from start of poll, 2.07329516s from cluster start. Timeout at 2026-04-23 00:15:21.350985179 +0000 UTC m=+38.390586607, 1.005018558s after stability
2026-04-23T00:15:21.363Z [INFO]  server-e4a98f38-798d-c27b-7faa-4567062cf790: aborting pipeline replication: peer="{Voter server-8b5c7330-3bda-303d-0ad1-4bab8881557d 8b5c7330-3bda-303d-0ad1-4bab8881557d}"
2026-04-23T00:15:21.363Z [INFO]  server-e4a98f38-798d-c27b-7faa-4567062cf790: aborting pipeline replication: peer="{Voter server-e765fdf8-56ff-0b93-ae22-7f4d9719604c e765fdf8-56ff-0b93-ae22-7f4d9719604c}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true
2026-04-23T00:15:21.365Z [INFO]  server-76e2ed51-e91a-2bb3-bdb9-0dfecb58e3bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-76e2ed51-e91a-2bb3-bdb9-0dfecb58e3bc Address:76e2ed51-e91a-2bb3-bdb9-0dfecb58e3bc} {Suffrage:Voter ID:server-dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af Address:dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af} {Suffrage:Voter ID:server-ddfd7e6e-3cc2-defe-91c9-d6194fb88e6f Address:ddfd7e6e-3cc2-defe-91c9-d6194fb88e6f}]"
2026-04-23T00:15:21.365Z [INFO]  server-76e2ed51-e91a-2bb3-bdb9-0dfecb58e3bc: entering follower state: follower="Node at 76e2ed51-e91a-2bb3-bdb9-0dfecb58e3bc [Follower]" leader-address= leader-id=
2026-04-23T00:15:21.365Z [INFO]  server-dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-76e2ed51-e91a-2bb3-bdb9-0dfecb58e3bc Address:76e2ed51-e91a-2bb3-bdb9-0dfecb58e3bc} {Suffrage:Voter ID:server-dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af Address:dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af} {Suffrage:Voter ID:server-ddfd7e6e-3cc2-defe-91c9-d6194fb88e6f Address:ddfd7e6e-3cc2-defe-91c9-d6194fb88e6f}]"
2026-04-23T00:15:21.370Z [INFO]  server-ddfd7e6e-3cc2-defe-91c9-d6194fb88e6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-76e2ed51-e91a-2bb3-bdb9-0dfecb58e3bc Address:76e2ed51-e91a-2bb3-bdb9-0dfecb58e3bc} {Suffrage:Voter ID:server-dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af Address:dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af} {Suffrage:Voter ID:server-ddfd7e6e-3cc2-defe-91c9-d6194fb88e6f Address:ddfd7e6e-3cc2-defe-91c9-d6194fb88e6f}]"
2026-04-23T00:15:21.370Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:21.370Z [INFO]  server-dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af: entering follower state: follower="Node at dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af [Follower]" leader-address= leader-id=
2026-04-23T00:15:21.371Z [INFO]  server-ddfd7e6e-3cc2-defe-91c9-d6194fb88e6f: entering follower state: follower="Node at ddfd7e6e-3cc2-defe-91c9-d6194fb88e6f [Follower]" leader-address= leader-id=
2026-04-23T00:15:21.994Z [WARN]  server-dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:21.994Z [INFO]  server-dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af: entering candidate state: node="Node at dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af [Candidate]" term=2
2026-04-23T00:15:21.994Z [INFO]  server-dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af: election won: term=2 tally=2
2026-04-23T00:15:21.994Z [INFO]  server-dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af: entering leader state: leader="Node at dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af [Leader]"
2026-04-23T00:15:21.994Z [INFO]  server-dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af: added peer, starting replication: peer=server-76e2ed51-e91a-2bb3-bdb9-0dfecb58e3bc
2026-04-23T00:15:21.994Z [INFO]  server-dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af: added peer, starting replication: peer=server-ddfd7e6e-3cc2-defe-91c9-d6194fb88e6f
2026-04-23T00:15:21.995Z [INFO]  server-dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af: pipelining replication: peer="{Voter server-76e2ed51-e91a-2bb3-bdb9-0dfecb58e3bc 76e2ed51-e91a-2bb3-bdb9-0dfecb58e3bc}"
2026-04-23T00:15:21.995Z [INFO]  server-dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af: pipelining replication: peer="{Voter server-ddfd7e6e-3cc2-defe-91c9-d6194fb88e6f ddfd7e6e-3cc2-defe-91c9-d6194fb88e6f}"
2026-04-23T00:15:23.000Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:21.995249618 +0000 UTC m=+39.034850991 (1 nodes), 624.315134ms from start of poll, 630.038586ms from cluster start. Timeout at 2026-04-23 00:15:23.000261059 +0000 UTC m=+40.039862428, 1.005011437s after stability
2026-04-23T00:15:23.001Z [INFO]  server-dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af: starting snapshot up to: index=12
2026-04-23T00:15:23.001Z [INFO]  creating new snapshot: path=/tmp/raft2978661465/snapshots/2-12-1776903323001.tmp
2026-04-23T00:15:23.002Z [INFO]  server-dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af: snapshot complete up to: index=12
2026-04-23T00:15:23.004Z [INFO]  server-19c11fed-b638-58ad-e67c-1c3c0bf117db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19c11fed-b638-58ad-e67c-1c3c0bf117db Address:19c11fed-b638-58ad-e67c-1c3c0bf117db} {Suffrage:Voter ID:server-57bda9a8-186d-79a3-0a68-dce9386bf12d Address:57bda9a8-186d-79a3-0a68-dce9386bf12d} {Suffrage:Voter ID:server-1a022e0b-d870-53ad-c0f1-28beca9f8545 Address:1a022e0b-d870-53ad-c0f1-28beca9f8545}]"
2026-04-23T00:15:23.004Z [INFO]  server-19c11fed-b638-58ad-e67c-1c3c0bf117db: entering follower state: follower="Node at 19c11fed-b638-58ad-e67c-1c3c0bf117db [Follower]" leader-address= leader-id=
2026-04-23T00:15:23.005Z [INFO]  server-57bda9a8-186d-79a3-0a68-dce9386bf12d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19c11fed-b638-58ad-e67c-1c3c0bf117db Address:19c11fed-b638-58ad-e67c-1c3c0bf117db} {Suffrage:Voter ID:server-57bda9a8-186d-79a3-0a68-dce9386bf12d Address:57bda9a8-186d-79a3-0a68-dce9386bf12d} {Suffrage:Voter ID:server-1a022e0b-d870-53ad-c0f1-28beca9f8545 Address:1a022e0b-d870-53ad-c0f1-28beca9f8545}]"
2026-04-23T00:15:23.005Z [INFO]  server-57bda9a8-186d-79a3-0a68-dce9386bf12d: entering follower state: follower="Node at 57bda9a8-186d-79a3-0a68-dce9386bf12d [Follower]" leader-address= leader-id=
2026-04-23T00:15:23.005Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-19c11fed-b638-58ad-e67c-1c3c0bf117db Address:19c11fed-b638-58ad-e67c-1c3c0bf117db} {Suffrage:Voter ID:server-57bda9a8-186d-79a3-0a68-dce9386bf12d Address:57bda9a8-186d-79a3-0a68-dce9386bf12d} {Suffrage:Voter ID:server-1a022e0b-d870-53ad-c0f1-28beca9f8545 Address:1a022e0b-d870-53ad-c0f1-28beca9f8545}]"
2026-04-23T00:15:23.005Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:23.005Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: entering follower state: follower="Node at 1a022e0b-d870-53ad-c0f1-28beca9f8545 [Follower]" leader-address= leader-id=
2026-04-23T00:15:23.515Z [WARN]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:23.516Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: entering candidate state: node="Node at 1a022e0b-d870-53ad-c0f1-28beca9f8545 [Candidate]" term=2
2026-04-23T00:15:23.516Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: election won: term=2 tally=2
2026-04-23T00:15:23.516Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: entering leader state: leader="Node at 1a022e0b-d870-53ad-c0f1-28beca9f8545 [Leader]"
2026-04-23T00:15:23.518Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: added peer, starting replication: peer=server-19c11fed-b638-58ad-e67c-1c3c0bf117db
2026-04-23T00:15:23.518Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: added peer, starting replication: peer=server-57bda9a8-186d-79a3-0a68-dce9386bf12d
2026-04-23T00:15:23.518Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: pipelining replication: peer="{Voter server-19c11fed-b638-58ad-e67c-1c3c0bf117db 19c11fed-b638-58ad-e67c-1c3c0bf117db}"
2026-04-23T00:15:23.521Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: pipelining replication: peer="{Voter server-57bda9a8-186d-79a3-0a68-dce9386bf12d 57bda9a8-186d-79a3-0a68-dce9386bf12d}"
2026-04-23T00:15:24.526Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:23.52147869 +0000 UTC m=+40.561080059 (1 nodes), 515.632112ms from start of poll, 517.067492ms from cluster start. Timeout at 2026-04-23 00:15:24.526483031 +0000 UTC m=+41.566084415, 1.005004356s after stability
2026-04-23T00:15:24.531Z [INFO]  creating new snapshot: path=/tmp/raft2881815807/snapshots/2-14-1776903324531.tmp
2026-04-23T00:15:24.532Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: copied to local snapshot: bytes=71
2026-04-23T00:15:24.532Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: snapshot restore progress: id=2-14-1776903324531 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:24.532Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: removing all old logs from log store
2026-04-23T00:15:24.532Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: compacting logs: from=1 to=2
2026-04-23T00:15:24.532Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: restored user snapshot: index=14
2026-04-23T00:15:24.532Z [ERROR] server-1a022e0b-d870-53ad-c0f1-28beca9f8545: failed to get log: index=2 error="log not found"
2026-04-23T00:15:24.533Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: aborting pipeline replication: peer="{Voter server-57bda9a8-186d-79a3-0a68-dce9386bf12d 57bda9a8-186d-79a3-0a68-dce9386bf12d}"
2026-04-23T00:15:24.532Z [ERROR] server-1a022e0b-d870-53ad-c0f1-28beca9f8545: failed to get log: index=2 error="log not found"
2026-04-23T00:15:24.533Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: aborting pipeline replication: peer="{Voter server-19c11fed-b638-58ad-e67c-1c3c0bf117db 19c11fed-b638-58ad-e67c-1c3c0bf117db}"
2026-04-23T00:15:24.539Z [ERROR] server-1a022e0b-d870-53ad-c0f1-28beca9f8545: failed to get log: index=2 error="log not found"
2026-04-23T00:15:24.539Z [INFO]  creating new snapshot: path=/tmp/raft3469400205/snapshots/2-14-1776903324539.tmp
2026-04-23T00:15:24.540Z [INFO]  server-19c11fed-b638-58ad-e67c-1c3c0bf117db: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:24.540Z [INFO]  server-19c11fed-b638-58ad-e67c-1c3c0bf117db: copied to local snapshot: bytes=71
2026-04-23T00:15:24.540Z [INFO]  server-19c11fed-b638-58ad-e67c-1c3c0bf117db: snapshot restore progress: id=2-14-1776903324539 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:24.541Z [INFO]  server-19c11fed-b638-58ad-e67c-1c3c0bf117db: removing all old logs from log store
2026-04-23T00:15:24.541Z [INFO]  server-19c11fed-b638-58ad-e67c-1c3c0bf117db: compacting logs: from=1 to=2
2026-04-23T00:15:24.541Z [INFO]  server-19c11fed-b638-58ad-e67c-1c3c0bf117db: Installed remote snapshot
2026-04-23T00:15:24.541Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: pipelining replication: peer="{Voter server-19c11fed-b638-58ad-e67c-1c3c0bf117db 19c11fed-b638-58ad-e67c-1c3c0bf117db}"
2026-04-23T00:15:24.541Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:15:24.541Z [ERROR] server-1a022e0b-d870-53ad-c0f1-28beca9f8545: failed to get log: index=2 error="log not found"
2026-04-23T00:15:24.541Z [INFO]  creating new snapshot: path=/tmp/raft4261535288/snapshots/2-14-1776903324541.tmp
2026-04-23T00:15:24.542Z [INFO]  server-57bda9a8-186d-79a3-0a68-dce9386bf12d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:24.542Z [INFO]  server-57bda9a8-186d-79a3-0a68-dce9386bf12d: copied to local snapshot: bytes=71
2026-04-23T00:15:24.542Z [INFO]  server-57bda9a8-186d-79a3-0a68-dce9386bf12d: snapshot restore progress: id=2-14-1776903324541 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:24.542Z [INFO]  server-57bda9a8-186d-79a3-0a68-dce9386bf12d: removing all old logs from log store
2026-04-23T00:15:24.543Z [INFO]  server-57bda9a8-186d-79a3-0a68-dce9386bf12d: compacting logs: from=1 to=2
2026-04-23T00:15:24.543Z [INFO]  server-57bda9a8-186d-79a3-0a68-dce9386bf12d: Installed remote snapshot
2026-04-23T00:15:24.543Z [INFO]  server-1a022e0b-d870-53ad-c0f1-28beca9f8545: pipelining replication: peer="{Voter server-57bda9a8-186d-79a3-0a68-dce9386bf12d 57bda9a8-186d-79a3-0a68-dce9386bf12d}"
2026-04-23T00:15:25.546Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:15:24.54121236 +0000 UTC m=+41.580813721 (2 nodes), 1.408µs from start of poll, 1.536801154s from cluster start. Timeout at 2026-04-23 00:15:25.546229195 +0000 UTC m=+42.585830603, 1.005016882s after stability
2026-04-23T00:15:25.558Z [INFO]  server-dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af: aborting pipeline replication: peer="{Voter server-76e2ed51-e91a-2bb3-bdb9-0dfecb58e3bc 76e2ed51-e91a-2bb3-bdb9-0dfecb58e3bc}"
2026-04-23T00:15:25.559Z [INFO]  server-dcfea9ad-0f92-78f9-00b1-90cdd7dfe0af: aborting pipeline replication: peer="{Voter server-ddfd7e6e-3cc2-defe-91c9-d6194fb88e6f ddfd7e6e-3cc2-defe-91c9-d6194fb88e6f}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false
2026-04-23T00:15:25.561Z [INFO]  server-0cf94550-abf2-507e-a1da-22d24decaf3f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cf94550-abf2-507e-a1da-22d24decaf3f Address:0cf94550-abf2-507e-a1da-22d24decaf3f} {Suffrage:Voter ID:server-89e152bd-22bb-6fde-e380-0c3a4bacc932 Address:89e152bd-22bb-6fde-e380-0c3a4bacc932} {Suffrage:Voter ID:server-6ed16911-0933-9f92-e5e3-a40cf390f2f8 Address:6ed16911-0933-9f92-e5e3-a40cf390f2f8}]"
2026-04-23T00:15:25.561Z [INFO]  server-0cf94550-abf2-507e-a1da-22d24decaf3f: entering follower state: follower="Node at 0cf94550-abf2-507e-a1da-22d24decaf3f [Follower]" leader-address= leader-id=
2026-04-23T00:15:25.561Z [INFO]  server-89e152bd-22bb-6fde-e380-0c3a4bacc932: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cf94550-abf2-507e-a1da-22d24decaf3f Address:0cf94550-abf2-507e-a1da-22d24decaf3f} {Suffrage:Voter ID:server-89e152bd-22bb-6fde-e380-0c3a4bacc932 Address:89e152bd-22bb-6fde-e380-0c3a4bacc932} {Suffrage:Voter ID:server-6ed16911-0933-9f92-e5e3-a40cf390f2f8 Address:6ed16911-0933-9f92-e5e3-a40cf390f2f8}]"
2026-04-23T00:15:25.561Z [INFO]  server-89e152bd-22bb-6fde-e380-0c3a4bacc932: entering follower state: follower="Node at 89e152bd-22bb-6fde-e380-0c3a4bacc932 [Follower]" leader-address= leader-id=
2026-04-23T00:15:25.561Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cf94550-abf2-507e-a1da-22d24decaf3f Address:0cf94550-abf2-507e-a1da-22d24decaf3f} {Suffrage:Voter ID:server-89e152bd-22bb-6fde-e380-0c3a4bacc932 Address:89e152bd-22bb-6fde-e380-0c3a4bacc932} {Suffrage:Voter ID:server-6ed16911-0933-9f92-e5e3-a40cf390f2f8 Address:6ed16911-0933-9f92-e5e3-a40cf390f2f8}]"
2026-04-23T00:15:25.561Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:25.561Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: entering follower state: follower="Node at 6ed16911-0933-9f92-e5e3-a40cf390f2f8 [Follower]" leader-address= leader-id=
2026-04-23T00:15:26.425Z [WARN]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:26.425Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: entering candidate state: node="Node at 6ed16911-0933-9f92-e5e3-a40cf390f2f8 [Candidate]" term=2
2026-04-23T00:15:26.425Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: election won: term=2 tally=2
2026-04-23T00:15:26.425Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: entering leader state: leader="Node at 6ed16911-0933-9f92-e5e3-a40cf390f2f8 [Leader]"
2026-04-23T00:15:26.425Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: added peer, starting replication: peer=server-0cf94550-abf2-507e-a1da-22d24decaf3f
2026-04-23T00:15:26.425Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: added peer, starting replication: peer=server-89e152bd-22bb-6fde-e380-0c3a4bacc932
2026-04-23T00:15:26.425Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: pipelining replication: peer="{Voter server-89e152bd-22bb-6fde-e380-0c3a4bacc932 89e152bd-22bb-6fde-e380-0c3a4bacc932}"
2026-04-23T00:15:26.425Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: pipelining replication: peer="{Voter server-0cf94550-abf2-507e-a1da-22d24decaf3f 0cf94550-abf2-507e-a1da-22d24decaf3f}"
2026-04-23T00:15:27.431Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:26.425845162 +0000 UTC m=+43.465446528 (1 nodes), 864.141595ms from start of poll, 864.875132ms from cluster start. Timeout at 2026-04-23 00:15:27.430850203 +0000 UTC m=+44.470451587, 1.005005059s after stability
2026-04-23T00:15:27.432Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: starting snapshot up to: index=12
2026-04-23T00:15:27.432Z [INFO]  creating new snapshot: path=/tmp/raft1365125328/snapshots/2-12-1776903327432.tmp
2026-04-23T00:15:27.433Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: snapshot complete up to: index=12
2026-04-23T00:15:27.434Z [INFO]  creating new snapshot: path=/tmp/raft1365125328/snapshots/2-23-1776903327434.tmp
2026-04-23T00:15:27.434Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: copied to local snapshot: bytes=71
2026-04-23T00:15:27.434Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: snapshot restore progress: id=2-23-1776903327434 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:27.434Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: restored user snapshot: index=23
2026-04-23T00:15:27.434Z [ERROR] server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: failed to get log: index=23 error="log not found"
2026-04-23T00:15:27.434Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: aborting pipeline replication: peer="{Voter server-89e152bd-22bb-6fde-e380-0c3a4bacc932 89e152bd-22bb-6fde-e380-0c3a4bacc932}"
2026-04-23T00:15:27.434Z [ERROR] server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: failed to get log: index=23 error="log not found"
2026-04-23T00:15:27.434Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: aborting pipeline replication: peer="{Voter server-0cf94550-abf2-507e-a1da-22d24decaf3f 0cf94550-abf2-507e-a1da-22d24decaf3f}"
2026-04-23T00:15:27.443Z [ERROR] server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: failed to get log: index=23 error="log not found"
2026-04-23T00:15:27.443Z [ERROR] server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: failed to get log: index=23 error="log not found"
2026-04-23T00:15:27.443Z [INFO]  creating new snapshot: path=/tmp/raft3384023364/snapshots/2-23-1776903327443.tmp
2026-04-23T00:15:27.443Z [INFO]  creating new snapshot: path=/tmp/raft1751877296/snapshots/2-23-1776903327443.tmp
2026-04-23T00:15:27.443Z [INFO]  server-89e152bd-22bb-6fde-e380-0c3a4bacc932: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:27.443Z [INFO]  server-0cf94550-abf2-507e-a1da-22d24decaf3f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:27.443Z [INFO]  server-89e152bd-22bb-6fde-e380-0c3a4bacc932: copied to local snapshot: bytes=71
2026-04-23T00:15:27.443Z [INFO]  server-89e152bd-22bb-6fde-e380-0c3a4bacc932: snapshot restore progress: id=2-23-1776903327443 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:27.444Z [INFO]  server-89e152bd-22bb-6fde-e380-0c3a4bacc932: Installed remote snapshot
2026-04-23T00:15:27.444Z [WARN]  server-89e152bd-22bb-6fde-e380-0c3a4bacc932: skipping application of old log: index=22
2026-04-23T00:15:27.444Z [INFO]  server-0cf94550-abf2-507e-a1da-22d24decaf3f: copied to local snapshot: bytes=71
2026-04-23T00:15:27.444Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: pipelining replication: peer="{Voter server-89e152bd-22bb-6fde-e380-0c3a4bacc932 89e152bd-22bb-6fde-e380-0c3a4bacc932}"
2026-04-23T00:15:27.444Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:15:27.444Z [INFO]  server-0cf94550-abf2-507e-a1da-22d24decaf3f: snapshot restore progress: id=2-23-1776903327443 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:27.444Z [INFO]  server-0cf94550-abf2-507e-a1da-22d24decaf3f: Installed remote snapshot
2026-04-23T00:15:27.444Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: pipelining replication: peer="{Voter server-0cf94550-abf2-507e-a1da-22d24decaf3f 0cf94550-abf2-507e-a1da-22d24decaf3f}"
2026-04-23T00:15:28.449Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:15:27.444141785 +0000 UTC m=+44.483743157 (2 nodes), 844ns from start of poll, 1.883171761s from cluster start. Timeout at 2026-04-23 00:15:28.449157562 +0000 UTC m=+45.488758956, 1.005015799s after stability
2026-04-23T00:15:28.461Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: aborting pipeline replication: peer="{Voter server-89e152bd-22bb-6fde-e380-0c3a4bacc932 89e152bd-22bb-6fde-e380-0c3a4bacc932}"
2026-04-23T00:15:28.461Z [INFO]  server-6ed16911-0933-9f92-e5e3-a40cf390f2f8: aborting pipeline replication: peer="{Voter server-0cf94550-abf2-507e-a1da-22d24decaf3f 0cf94550-abf2-507e-a1da-22d24decaf3f}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false
2026-04-23T00:15:28.463Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b Address:4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b} {Suffrage:Voter ID:server-05aa7aca-01ef-2ab1-d862-7ef65a9bea93 Address:05aa7aca-01ef-2ab1-d862-7ef65a9bea93} {Suffrage:Voter ID:server-04f4f7a9-7055-2c7d-d064-933ae966865c Address:04f4f7a9-7055-2c7d-d064-933ae966865c}]"
2026-04-23T00:15:28.463Z [INFO]  server-05aa7aca-01ef-2ab1-d862-7ef65a9bea93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b Address:4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b} {Suffrage:Voter ID:server-05aa7aca-01ef-2ab1-d862-7ef65a9bea93 Address:05aa7aca-01ef-2ab1-d862-7ef65a9bea93} {Suffrage:Voter ID:server-04f4f7a9-7055-2c7d-d064-933ae966865c Address:04f4f7a9-7055-2c7d-d064-933ae966865c}]"
2026-04-23T00:15:28.463Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: entering follower state: follower="Node at 4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b [Follower]" leader-address= leader-id=
2026-04-23T00:15:28.463Z [INFO]  server-04f4f7a9-7055-2c7d-d064-933ae966865c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b Address:4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b} {Suffrage:Voter ID:server-05aa7aca-01ef-2ab1-d862-7ef65a9bea93 Address:05aa7aca-01ef-2ab1-d862-7ef65a9bea93} {Suffrage:Voter ID:server-04f4f7a9-7055-2c7d-d064-933ae966865c Address:04f4f7a9-7055-2c7d-d064-933ae966865c}]"
2026-04-23T00:15:28.463Z [INFO]  server-05aa7aca-01ef-2ab1-d862-7ef65a9bea93: entering follower state: follower="Node at 05aa7aca-01ef-2ab1-d862-7ef65a9bea93 [Follower]" leader-address= leader-id=
2026-04-23T00:15:28.463Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:28.463Z [INFO]  server-04f4f7a9-7055-2c7d-d064-933ae966865c: entering follower state: follower="Node at 04f4f7a9-7055-2c7d-d064-933ae966865c [Follower]" leader-address= leader-id=
2026-04-23T00:15:29.036Z [WARN]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:29.036Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: entering candidate state: node="Node at 4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b [Candidate]" term=2
2026-04-23T00:15:29.036Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: election won: term=2 tally=2
2026-04-23T00:15:29.036Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: entering leader state: leader="Node at 4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b [Leader]"
2026-04-23T00:15:29.036Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: added peer, starting replication: peer=server-05aa7aca-01ef-2ab1-d862-7ef65a9bea93
2026-04-23T00:15:29.036Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: added peer, starting replication: peer=server-04f4f7a9-7055-2c7d-d064-933ae966865c
2026-04-23T00:15:29.036Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: pipelining replication: peer="{Voter server-04f4f7a9-7055-2c7d-d064-933ae966865c 04f4f7a9-7055-2c7d-d064-933ae966865c}"
2026-04-23T00:15:29.036Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: pipelining replication: peer="{Voter server-05aa7aca-01ef-2ab1-d862-7ef65a9bea93 05aa7aca-01ef-2ab1-d862-7ef65a9bea93}"
2026-04-23T00:15:30.042Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:29.036471271 +0000 UTC m=+46.076072627 (1 nodes), 572.567735ms from start of poll, 572.95925ms from cluster start. Timeout at 2026-04-23 00:15:30.041475202 +0000 UTC m=+47.081076561, 1.005003934s after stability
2026-04-23T00:15:30.042Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: starting snapshot up to: index=12
2026-04-23T00:15:30.043Z [INFO]  creating new snapshot: path=/tmp/raft1732563647/snapshots/2-12-1776903330043.tmp
2026-04-23T00:15:30.043Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: snapshot complete up to: index=12
2026-04-23T00:15:30.044Z [INFO]  creating new snapshot: path=/tmp/raft1732563647/snapshots/2-23-1776903330044.tmp
2026-04-23T00:15:30.044Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: copied to local snapshot: bytes=71
2026-04-23T00:15:30.045Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: snapshot restore progress: id=2-23-1776903330044 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:30.045Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: removing all old logs from log store
2026-04-23T00:15:30.045Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: compacting logs: from=1 to=22
2026-04-23T00:15:30.045Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: restored user snapshot: index=23
2026-04-23T00:15:30.045Z [ERROR] server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: failed to get log: index=22 error="log not found"
2026-04-23T00:15:30.045Z [ERROR] server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: failed to get log: index=22 error="log not found"
2026-04-23T00:15:30.045Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: aborting pipeline replication: peer="{Voter server-04f4f7a9-7055-2c7d-d064-933ae966865c 04f4f7a9-7055-2c7d-d064-933ae966865c}"
2026-04-23T00:15:30.045Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: aborting pipeline replication: peer="{Voter server-05aa7aca-01ef-2ab1-d862-7ef65a9bea93 05aa7aca-01ef-2ab1-d862-7ef65a9bea93}"
2026-04-23T00:15:30.052Z [ERROR] server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: failed to get log: index=22 error="log not found"
2026-04-23T00:15:30.052Z [INFO]  creating new snapshot: path=/tmp/raft1617924397/snapshots/2-23-1776903330052.tmp
2026-04-23T00:15:30.053Z [ERROR] server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: failed to get log: index=22 error="log not found"
2026-04-23T00:15:30.053Z [INFO]  server-05aa7aca-01ef-2ab1-d862-7ef65a9bea93: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:30.053Z [INFO]  creating new snapshot: path=/tmp/raft2852365036/snapshots/2-23-1776903330053.tmp
2026-04-23T00:15:30.053Z [INFO]  server-05aa7aca-01ef-2ab1-d862-7ef65a9bea93: copied to local snapshot: bytes=71
2026-04-23T00:15:30.053Z [INFO]  server-05aa7aca-01ef-2ab1-d862-7ef65a9bea93: snapshot restore progress: id=2-23-1776903330052 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:30.053Z [INFO]  server-05aa7aca-01ef-2ab1-d862-7ef65a9bea93: removing all old logs from log store
2026-04-23T00:15:30.053Z [INFO]  server-05aa7aca-01ef-2ab1-d862-7ef65a9bea93: compacting logs: from=1 to=22
2026-04-23T00:15:30.053Z [INFO]  server-05aa7aca-01ef-2ab1-d862-7ef65a9bea93: Installed remote snapshot
2026-04-23T00:15:30.053Z [INFO]  server-04f4f7a9-7055-2c7d-d064-933ae966865c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:30.053Z [WARN]  server-05aa7aca-01ef-2ab1-d862-7ef65a9bea93: skipping application of old log: index=22
2026-04-23T00:15:30.053Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: pipelining replication: peer="{Voter server-05aa7aca-01ef-2ab1-d862-7ef65a9bea93 05aa7aca-01ef-2ab1-d862-7ef65a9bea93}"
2026-04-23T00:15:30.053Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:15:30.053Z [INFO]  server-04f4f7a9-7055-2c7d-d064-933ae966865c: copied to local snapshot: bytes=71
2026-04-23T00:15:30.054Z [INFO]  server-04f4f7a9-7055-2c7d-d064-933ae966865c: snapshot restore progress: id=2-23-1776903330053 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:30.054Z [INFO]  server-04f4f7a9-7055-2c7d-d064-933ae966865c: removing all old logs from log store
2026-04-23T00:15:30.054Z [INFO]  server-04f4f7a9-7055-2c7d-d064-933ae966865c: compacting logs: from=1 to=22
2026-04-23T00:15:30.054Z [INFO]  server-04f4f7a9-7055-2c7d-d064-933ae966865c: Installed remote snapshot
2026-04-23T00:15:30.054Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: pipelining replication: peer="{Voter server-04f4f7a9-7055-2c7d-d064-933ae966865c 04f4f7a9-7055-2c7d-d064-933ae966865c}"
2026-04-23T00:15:31.059Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:15:30.053762544 +0000 UTC m=+47.093363899 (2 nodes), 700ns from start of poll, 1.590250522s from cluster start. Timeout at 2026-04-23 00:15:31.058780247 +0000 UTC m=+48.098381603, 1.005017704s after stability
2026-04-23T00:15:31.070Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: aborting pipeline replication: peer="{Voter server-04f4f7a9-7055-2c7d-d064-933ae966865c 04f4f7a9-7055-2c7d-d064-933ae966865c}"
2026-04-23T00:15:31.070Z [INFO]  server-4e287bc0-28da-65d1-e2ea-c1e4cb99cb2b: aborting pipeline replication: peer="{Voter server-05aa7aca-01ef-2ab1-d862-7ef65a9bea93 05aa7aca-01ef-2ab1-d862-7ef65a9bea93}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true
2026-04-23T00:15:31.072Z [INFO]  server-672ababd-8f37-ef2c-9e1f-f53072976442: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-672ababd-8f37-ef2c-9e1f-f53072976442 Address:672ababd-8f37-ef2c-9e1f-f53072976442} {Suffrage:Voter ID:server-260ff848-6b09-e770-e64e-a3d7bbde5ab2 Address:260ff848-6b09-e770-e64e-a3d7bbde5ab2} {Suffrage:Voter ID:server-17777bf2-a1b0-60fe-7446-910fb0a7a684 Address:17777bf2-a1b0-60fe-7446-910fb0a7a684}]"
2026-04-23T00:15:31.072Z [INFO]  server-260ff848-6b09-e770-e64e-a3d7bbde5ab2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-672ababd-8f37-ef2c-9e1f-f53072976442 Address:672ababd-8f37-ef2c-9e1f-f53072976442} {Suffrage:Voter ID:server-260ff848-6b09-e770-e64e-a3d7bbde5ab2 Address:260ff848-6b09-e770-e64e-a3d7bbde5ab2} {Suffrage:Voter ID:server-17777bf2-a1b0-60fe-7446-910fb0a7a684 Address:17777bf2-a1b0-60fe-7446-910fb0a7a684}]"
2026-04-23T00:15:31.072Z [INFO]  server-672ababd-8f37-ef2c-9e1f-f53072976442: entering follower state: follower="Node at 672ababd-8f37-ef2c-9e1f-f53072976442 [Follower]" leader-address= leader-id=
2026-04-23T00:15:31.072Z [INFO]  server-17777bf2-a1b0-60fe-7446-910fb0a7a684: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-672ababd-8f37-ef2c-9e1f-f53072976442 Address:672ababd-8f37-ef2c-9e1f-f53072976442} {Suffrage:Voter ID:server-260ff848-6b09-e770-e64e-a3d7bbde5ab2 Address:260ff848-6b09-e770-e64e-a3d7bbde5ab2} {Suffrage:Voter ID:server-17777bf2-a1b0-60fe-7446-910fb0a7a684 Address:17777bf2-a1b0-60fe-7446-910fb0a7a684}]"
2026-04-23T00:15:31.072Z [INFO]  server-260ff848-6b09-e770-e64e-a3d7bbde5ab2: entering follower state: follower="Node at 260ff848-6b09-e770-e64e-a3d7bbde5ab2 [Follower]" leader-address= leader-id=
2026-04-23T00:15:31.072Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:31.072Z [INFO]  server-17777bf2-a1b0-60fe-7446-910fb0a7a684: entering follower state: follower="Node at 17777bf2-a1b0-60fe-7446-910fb0a7a684 [Follower]" leader-address= leader-id=
2026-04-23T00:15:31.610Z [WARN]  server-672ababd-8f37-ef2c-9e1f-f53072976442: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:31.610Z [INFO]  server-672ababd-8f37-ef2c-9e1f-f53072976442: entering candidate state: node="Node at 672ababd-8f37-ef2c-9e1f-f53072976442 [Candidate]" term=2
2026-04-23T00:15:31.610Z [INFO]  server-672ababd-8f37-ef2c-9e1f-f53072976442: election won: term=2 tally=2
2026-04-23T00:15:31.610Z [INFO]  server-672ababd-8f37-ef2c-9e1f-f53072976442: entering leader state: leader="Node at 672ababd-8f37-ef2c-9e1f-f53072976442 [Leader]"
2026-04-23T00:15:31.610Z [INFO]  server-672ababd-8f37-ef2c-9e1f-f53072976442: added peer, starting replication: peer=server-260ff848-6b09-e770-e64e-a3d7bbde5ab2
2026-04-23T00:15:31.610Z [INFO]  server-672ababd-8f37-ef2c-9e1f-f53072976442: added peer, starting replication: peer=server-17777bf2-a1b0-60fe-7446-910fb0a7a684
2026-04-23T00:15:31.610Z [INFO]  server-672ababd-8f37-ef2c-9e1f-f53072976442: pipelining replication: peer="{Voter server-260ff848-6b09-e770-e64e-a3d7bbde5ab2 260ff848-6b09-e770-e64e-a3d7bbde5ab2}"
2026-04-23T00:15:31.610Z [INFO]  server-672ababd-8f37-ef2c-9e1f-f53072976442: pipelining replication: peer="{Voter server-17777bf2-a1b0-60fe-7446-910fb0a7a684 17777bf2-a1b0-60fe-7446-910fb0a7a684}"
2026-04-23T00:15:32.615Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:31.61024282 +0000 UTC m=+48.649844184 (1 nodes), 537.853271ms from start of poll, 538.271587ms from cluster start. Timeout at 2026-04-23 00:15:32.615245516 +0000 UTC m=+49.654846871, 1.005002687s after stability
2026-04-23T00:15:32.616Z [INFO]  server-672ababd-8f37-ef2c-9e1f-f53072976442: starting snapshot up to: index=12
2026-04-23T00:15:32.616Z [INFO]  creating new snapshot: path=/tmp/raft3309252360/snapshots/2-12-1776903332616.tmp
2026-04-23T00:15:32.617Z [INFO]  server-672ababd-8f37-ef2c-9e1f-f53072976442: snapshot complete up to: index=12
2026-04-23T00:15:32.617Z [INFO]  server-08ef5b33-be82-c9ed-cdc8-0260c3752bf0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-08ef5b33-be82-c9ed-cdc8-0260c3752bf0 Address:08ef5b33-be82-c9ed-cdc8-0260c3752bf0} {Suffrage:Voter ID:server-79571824-3817-41f5-de1f-12ab28d79003 Address:79571824-3817-41f5-de1f-12ab28d79003} {Suffrage:Voter ID:server-a2008b58-f358-2196-4475-bb09ec4614df Address:a2008b58-f358-2196-4475-bb09ec4614df}]"
2026-04-23T00:15:32.618Z [INFO]  server-08ef5b33-be82-c9ed-cdc8-0260c3752bf0: entering follower state: follower="Node at 08ef5b33-be82-c9ed-cdc8-0260c3752bf0 [Follower]" leader-address= leader-id=
2026-04-23T00:15:32.618Z [INFO]  server-79571824-3817-41f5-de1f-12ab28d79003: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-08ef5b33-be82-c9ed-cdc8-0260c3752bf0 Address:08ef5b33-be82-c9ed-cdc8-0260c3752bf0} {Suffrage:Voter ID:server-79571824-3817-41f5-de1f-12ab28d79003 Address:79571824-3817-41f5-de1f-12ab28d79003} {Suffrage:Voter ID:server-a2008b58-f358-2196-4475-bb09ec4614df Address:a2008b58-f358-2196-4475-bb09ec4614df}]"
2026-04-23T00:15:32.618Z [INFO]  server-79571824-3817-41f5-de1f-12ab28d79003: entering follower state: follower="Node at 79571824-3817-41f5-de1f-12ab28d79003 [Follower]" leader-address= leader-id=
2026-04-23T00:15:32.618Z [INFO]  server-a2008b58-f358-2196-4475-bb09ec4614df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-08ef5b33-be82-c9ed-cdc8-0260c3752bf0 Address:08ef5b33-be82-c9ed-cdc8-0260c3752bf0} {Suffrage:Voter ID:server-79571824-3817-41f5-de1f-12ab28d79003 Address:79571824-3817-41f5-de1f-12ab28d79003} {Suffrage:Voter ID:server-a2008b58-f358-2196-4475-bb09ec4614df Address:a2008b58-f358-2196-4475-bb09ec4614df}]"
2026-04-23T00:15:32.618Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:32.618Z [INFO]  server-a2008b58-f358-2196-4475-bb09ec4614df: entering follower state: follower="Node at a2008b58-f358-2196-4475-bb09ec4614df [Follower]" leader-address= leader-id=
2026-04-23T00:15:33.377Z [WARN]  server-a2008b58-f358-2196-4475-bb09ec4614df: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:33.377Z [INFO]  server-a2008b58-f358-2196-4475-bb09ec4614df: entering candidate state: node="Node at a2008b58-f358-2196-4475-bb09ec4614df [Candidate]" term=2
2026-04-23T00:15:33.377Z [INFO]  server-a2008b58-f358-2196-4475-bb09ec4614df: election won: term=2 tally=2
2026-04-23T00:15:33.377Z [INFO]  server-a2008b58-f358-2196-4475-bb09ec4614df: entering leader state: leader="Node at a2008b58-f358-2196-4475-bb09ec4614df [Leader]"
2026-04-23T00:15:33.378Z [INFO]  server-a2008b58-f358-2196-4475-bb09ec4614df: added peer, starting replication: peer=server-08ef5b33-be82-c9ed-cdc8-0260c3752bf0
2026-04-23T00:15:33.378Z [INFO]  server-a2008b58-f358-2196-4475-bb09ec4614df: added peer, starting replication: peer=server-79571824-3817-41f5-de1f-12ab28d79003
2026-04-23T00:15:33.378Z [INFO]  server-a2008b58-f358-2196-4475-bb09ec4614df: pipelining replication: peer="{Voter server-79571824-3817-41f5-de1f-12ab28d79003 79571824-3817-41f5-de1f-12ab28d79003}"
2026-04-23T00:15:33.379Z [INFO]  server-a2008b58-f358-2196-4475-bb09ec4614df: pipelining replication: peer="{Voter server-08ef5b33-be82-c9ed-cdc8-0260c3752bf0 08ef5b33-be82-c9ed-cdc8-0260c3752bf0}"
2026-04-23T00:15:34.383Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:33.378022599 +0000 UTC m=+50.417624047 (1 nodes), 759.789972ms from start of poll, 760.1519ms from cluster start. Timeout at 2026-04-23 00:15:34.383027436 +0000 UTC m=+51.422628792, 1.005004745s after stability
2026-04-23T00:15:34.383Z [INFO]  creating new snapshot: path=/tmp/raft3955679718/snapshots/2-15-1776903334383.tmp
2026-04-23T00:15:34.383Z [INFO]  server-a2008b58-f358-2196-4475-bb09ec4614df: copied to local snapshot: bytes=71
2026-04-23T00:15:34.384Z [INFO]  server-a2008b58-f358-2196-4475-bb09ec4614df: snapshot restore progress: id=2-15-1776903334383 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:34.384Z [INFO]  server-a2008b58-f358-2196-4475-bb09ec4614df: restored user snapshot: index=15
2026-04-23T00:15:34.384Z [ERROR] server-a2008b58-f358-2196-4475-bb09ec4614df: failed to get log: index=3 error="log not found"
2026-04-23T00:15:34.384Z [INFO]  server-a2008b58-f358-2196-4475-bb09ec4614df: aborting pipeline replication: peer="{Voter server-79571824-3817-41f5-de1f-12ab28d79003 79571824-3817-41f5-de1f-12ab28d79003}"
2026-04-23T00:15:34.384Z [ERROR] server-a2008b58-f358-2196-4475-bb09ec4614df: failed to get log: index=3 error="log not found"
2026-04-23T00:15:34.384Z [INFO]  server-a2008b58-f358-2196-4475-bb09ec4614df: aborting pipeline replication: peer="{Voter server-08ef5b33-be82-c9ed-cdc8-0260c3752bf0 08ef5b33-be82-c9ed-cdc8-0260c3752bf0}"
2026-04-23T00:15:34.390Z [ERROR] server-a2008b58-f358-2196-4475-bb09ec4614df: failed to get log: index=3 error="log not found"
2026-04-23T00:15:34.391Z [ERROR] server-a2008b58-f358-2196-4475-bb09ec4614df: failed to get log: index=3 error="log not found"
2026-04-23T00:15:34.391Z [INFO]  creating new snapshot: path=/tmp/raft1882779507/snapshots/2-15-1776903334391.tmp
2026-04-23T00:15:34.391Z [INFO]  server-79571824-3817-41f5-de1f-12ab28d79003: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:34.391Z [INFO]  creating new snapshot: path=/tmp/raft296017072/snapshots/2-15-1776903334391.tmp
2026-04-23T00:15:34.391Z [INFO]  server-79571824-3817-41f5-de1f-12ab28d79003: copied to local snapshot: bytes=71
2026-04-23T00:15:34.391Z [INFO]  server-08ef5b33-be82-c9ed-cdc8-0260c3752bf0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:34.391Z [INFO]  server-79571824-3817-41f5-de1f-12ab28d79003: snapshot restore progress: id=2-15-1776903334391 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:34.391Z [INFO]  server-79571824-3817-41f5-de1f-12ab28d79003: Installed remote snapshot
2026-04-23T00:15:34.392Z [INFO]  server-a2008b58-f358-2196-4475-bb09ec4614df: pipelining replication: peer="{Voter server-79571824-3817-41f5-de1f-12ab28d79003 79571824-3817-41f5-de1f-12ab28d79003}"
2026-04-23T00:15:34.392Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:15:34.392Z [INFO]  server-08ef5b33-be82-c9ed-cdc8-0260c3752bf0: copied to local snapshot: bytes=71
2026-04-23T00:15:34.392Z [INFO]  server-08ef5b33-be82-c9ed-cdc8-0260c3752bf0: snapshot restore progress: id=2-15-1776903334391 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:34.392Z [INFO]  server-08ef5b33-be82-c9ed-cdc8-0260c3752bf0: Installed remote snapshot
2026-04-23T00:15:34.392Z [INFO]  server-a2008b58-f358-2196-4475-bb09ec4614df: pipelining replication: peer="{Voter server-08ef5b33-be82-c9ed-cdc8-0260c3752bf0 08ef5b33-be82-c9ed-cdc8-0260c3752bf0}"
2026-04-23T00:15:35.397Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:15:34.392162273 +0000 UTC m=+51.431763631 (2 nodes), 870ns from start of poll, 1.774291484s from cluster start. Timeout at 2026-04-23 00:15:35.397165991 +0000 UTC m=+52.436767347, 1.005003716s after stability
2026-04-23T00:15:35.410Z [INFO]  server-672ababd-8f37-ef2c-9e1f-f53072976442: aborting pipeline replication: peer="{Voter server-17777bf2-a1b0-60fe-7446-910fb0a7a684 17777bf2-a1b0-60fe-7446-910fb0a7a684}"
2026-04-23T00:15:35.410Z [INFO]  server-672ababd-8f37-ef2c-9e1f-f53072976442: aborting pipeline replication: peer="{Voter server-260ff848-6b09-e770-e64e-a3d7bbde5ab2 260ff848-6b09-e770-e64e-a3d7bbde5ab2}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true
2026-04-23T00:15:35.411Z [INFO]  server-724b8fe5-dc70-e834-12be-0353675967eb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-724b8fe5-dc70-e834-12be-0353675967eb Address:724b8fe5-dc70-e834-12be-0353675967eb} {Suffrage:Voter ID:server-cecf1882-2645-699c-fd63-4100589bd5af Address:cecf1882-2645-699c-fd63-4100589bd5af} {Suffrage:Voter ID:server-59ad74b9-38d1-b8ff-1a86-f3b27524db6b Address:59ad74b9-38d1-b8ff-1a86-f3b27524db6b}]"
2026-04-23T00:15:35.411Z [INFO]  server-724b8fe5-dc70-e834-12be-0353675967eb: entering follower state: follower="Node at 724b8fe5-dc70-e834-12be-0353675967eb [Follower]" leader-address= leader-id=
2026-04-23T00:15:35.411Z [INFO]  server-cecf1882-2645-699c-fd63-4100589bd5af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-724b8fe5-dc70-e834-12be-0353675967eb Address:724b8fe5-dc70-e834-12be-0353675967eb} {Suffrage:Voter ID:server-cecf1882-2645-699c-fd63-4100589bd5af Address:cecf1882-2645-699c-fd63-4100589bd5af} {Suffrage:Voter ID:server-59ad74b9-38d1-b8ff-1a86-f3b27524db6b Address:59ad74b9-38d1-b8ff-1a86-f3b27524db6b}]"
2026-04-23T00:15:35.411Z [INFO]  server-cecf1882-2645-699c-fd63-4100589bd5af: entering follower state: follower="Node at cecf1882-2645-699c-fd63-4100589bd5af [Follower]" leader-address= leader-id=
2026-04-23T00:15:35.411Z [INFO]  server-59ad74b9-38d1-b8ff-1a86-f3b27524db6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-724b8fe5-dc70-e834-12be-0353675967eb Address:724b8fe5-dc70-e834-12be-0353675967eb} {Suffrage:Voter ID:server-cecf1882-2645-699c-fd63-4100589bd5af Address:cecf1882-2645-699c-fd63-4100589bd5af} {Suffrage:Voter ID:server-59ad74b9-38d1-b8ff-1a86-f3b27524db6b Address:59ad74b9-38d1-b8ff-1a86-f3b27524db6b}]"
2026-04-23T00:15:35.411Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:35.411Z [INFO]  server-59ad74b9-38d1-b8ff-1a86-f3b27524db6b: entering follower state: follower="Node at 59ad74b9-38d1-b8ff-1a86-f3b27524db6b [Follower]" leader-address= leader-id=
2026-04-23T00:15:35.980Z [WARN]  server-724b8fe5-dc70-e834-12be-0353675967eb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:35.980Z [INFO]  server-724b8fe5-dc70-e834-12be-0353675967eb: entering candidate state: node="Node at 724b8fe5-dc70-e834-12be-0353675967eb [Candidate]" term=2
2026-04-23T00:15:35.980Z [INFO]  server-724b8fe5-dc70-e834-12be-0353675967eb: election won: term=2 tally=2
2026-04-23T00:15:35.980Z [INFO]  server-724b8fe5-dc70-e834-12be-0353675967eb: entering leader state: leader="Node at 724b8fe5-dc70-e834-12be-0353675967eb [Leader]"
2026-04-23T00:15:35.980Z [INFO]  server-724b8fe5-dc70-e834-12be-0353675967eb: added peer, starting replication: peer=server-cecf1882-2645-699c-fd63-4100589bd5af
2026-04-23T00:15:35.980Z [INFO]  server-724b8fe5-dc70-e834-12be-0353675967eb: added peer, starting replication: peer=server-59ad74b9-38d1-b8ff-1a86-f3b27524db6b
2026-04-23T00:15:35.980Z [INFO]  server-724b8fe5-dc70-e834-12be-0353675967eb: pipelining replication: peer="{Voter server-cecf1882-2645-699c-fd63-4100589bd5af cecf1882-2645-699c-fd63-4100589bd5af}"
2026-04-23T00:15:35.980Z [INFO]  server-724b8fe5-dc70-e834-12be-0353675967eb: pipelining replication: peer="{Voter server-59ad74b9-38d1-b8ff-1a86-f3b27524db6b 59ad74b9-38d1-b8ff-1a86-f3b27524db6b}"
2026-04-23T00:15:36.985Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:35.980658095 +0000 UTC m=+53.020259517 (1 nodes), 568.840285ms from start of poll, 569.328307ms from cluster start. Timeout at 2026-04-23 00:15:36.985660946 +0000 UTC m=+54.025262298, 1.005002781s after stability
2026-04-23T00:15:36.986Z [INFO]  server-724b8fe5-dc70-e834-12be-0353675967eb: starting snapshot up to: index=12
2026-04-23T00:15:36.986Z [INFO]  creating new snapshot: path=/tmp/raft2825442641/snapshots/2-12-1776903336986.tmp
2026-04-23T00:15:36.986Z [INFO]  server-724b8fe5-dc70-e834-12be-0353675967eb: snapshot complete up to: index=12
2026-04-23T00:15:36.987Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85 Address:5296228a-f227-b68e-c4f4-4f5d8a0ecb85} {Suffrage:Voter ID:server-48c53d64-11d7-eecd-db69-95d53bc35ab9 Address:48c53d64-11d7-eecd-db69-95d53bc35ab9} {Suffrage:Voter ID:server-35ab28ee-7a86-375f-eed1-44dedb19530d Address:35ab28ee-7a86-375f-eed1-44dedb19530d}]"
2026-04-23T00:15:36.987Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: entering follower state: follower="Node at 5296228a-f227-b68e-c4f4-4f5d8a0ecb85 [Follower]" leader-address= leader-id=
2026-04-23T00:15:36.987Z [INFO]  server-48c53d64-11d7-eecd-db69-95d53bc35ab9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85 Address:5296228a-f227-b68e-c4f4-4f5d8a0ecb85} {Suffrage:Voter ID:server-48c53d64-11d7-eecd-db69-95d53bc35ab9 Address:48c53d64-11d7-eecd-db69-95d53bc35ab9} {Suffrage:Voter ID:server-35ab28ee-7a86-375f-eed1-44dedb19530d Address:35ab28ee-7a86-375f-eed1-44dedb19530d}]"
2026-04-23T00:15:36.987Z [INFO]  server-48c53d64-11d7-eecd-db69-95d53bc35ab9: entering follower state: follower="Node at 48c53d64-11d7-eecd-db69-95d53bc35ab9 [Follower]" leader-address= leader-id=
2026-04-23T00:15:36.987Z [INFO]  server-35ab28ee-7a86-375f-eed1-44dedb19530d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85 Address:5296228a-f227-b68e-c4f4-4f5d8a0ecb85} {Suffrage:Voter ID:server-48c53d64-11d7-eecd-db69-95d53bc35ab9 Address:48c53d64-11d7-eecd-db69-95d53bc35ab9} {Suffrage:Voter ID:server-35ab28ee-7a86-375f-eed1-44dedb19530d Address:35ab28ee-7a86-375f-eed1-44dedb19530d}]"
2026-04-23T00:15:36.987Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:36.987Z [INFO]  server-35ab28ee-7a86-375f-eed1-44dedb19530d: entering follower state: follower="Node at 35ab28ee-7a86-375f-eed1-44dedb19530d [Follower]" leader-address= leader-id=
2026-04-23T00:15:37.566Z [WARN]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:37.566Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: entering candidate state: node="Node at 5296228a-f227-b68e-c4f4-4f5d8a0ecb85 [Candidate]" term=2
2026-04-23T00:15:37.566Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: election won: term=2 tally=2
2026-04-23T00:15:37.566Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: entering leader state: leader="Node at 5296228a-f227-b68e-c4f4-4f5d8a0ecb85 [Leader]"
2026-04-23T00:15:37.567Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: added peer, starting replication: peer=server-48c53d64-11d7-eecd-db69-95d53bc35ab9
2026-04-23T00:15:37.567Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: added peer, starting replication: peer=server-35ab28ee-7a86-375f-eed1-44dedb19530d
2026-04-23T00:15:37.567Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: pipelining replication: peer="{Voter server-35ab28ee-7a86-375f-eed1-44dedb19530d 35ab28ee-7a86-375f-eed1-44dedb19530d}"
2026-04-23T00:15:37.567Z [WARN]  server-48c53d64-11d7-eecd-db69-95d53bc35ab9: rejecting vote request since our last term is greater: candidate=5296228a-f227-b68e-c4f4-4f5d8a0ecb85 last-term=2 last-candidate-term=1
2026-04-23T00:15:37.567Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: pipelining replication: peer="{Voter server-48c53d64-11d7-eecd-db69-95d53bc35ab9 48c53d64-11d7-eecd-db69-95d53bc35ab9}"
2026-04-23T00:15:38.573Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:37.567263344 +0000 UTC m=+54.606864714 (1 nodes), 579.304499ms from start of poll, 579.656237ms from cluster start. Timeout at 2026-04-23 00:15:38.572268091 +0000 UTC m=+55.611869448, 1.005004734s after stability
2026-04-23T00:15:38.573Z [INFO]  creating new snapshot: path=/tmp/raft3909810100/snapshots/2-15-1776903338573.tmp
2026-04-23T00:15:38.573Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: copied to local snapshot: bytes=71
2026-04-23T00:15:38.574Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: snapshot restore progress: id=2-15-1776903338573 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:38.574Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: removing all old logs from log store
2026-04-23T00:15:38.574Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: compacting logs: from=1 to=2
2026-04-23T00:15:38.574Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: restored user snapshot: index=15
2026-04-23T00:15:38.574Z [ERROR] server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: failed to get log: index=2 error="log not found"
2026-04-23T00:15:38.574Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: aborting pipeline replication: peer="{Voter server-48c53d64-11d7-eecd-db69-95d53bc35ab9 48c53d64-11d7-eecd-db69-95d53bc35ab9}"
2026-04-23T00:15:38.574Z [ERROR] server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: failed to get log: index=2 error="log not found"
2026-04-23T00:15:38.574Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: aborting pipeline replication: peer="{Voter server-35ab28ee-7a86-375f-eed1-44dedb19530d 35ab28ee-7a86-375f-eed1-44dedb19530d}"
2026-04-23T00:15:38.580Z [ERROR] server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: failed to get log: index=2 error="log not found"
2026-04-23T00:15:38.581Z [INFO]  creating new snapshot: path=/tmp/raft948687380/snapshots/2-15-1776903338581.tmp
2026-04-23T00:15:38.581Z [INFO]  server-35ab28ee-7a86-375f-eed1-44dedb19530d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:38.581Z [INFO]  server-35ab28ee-7a86-375f-eed1-44dedb19530d: copied to local snapshot: bytes=71
2026-04-23T00:15:38.581Z [INFO]  server-35ab28ee-7a86-375f-eed1-44dedb19530d: snapshot restore progress: id=2-15-1776903338581 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:38.581Z [INFO]  server-35ab28ee-7a86-375f-eed1-44dedb19530d: removing all old logs from log store
2026-04-23T00:15:38.581Z [INFO]  server-35ab28ee-7a86-375f-eed1-44dedb19530d: compacting logs: from=1 to=2
2026-04-23T00:15:38.581Z [INFO]  server-35ab28ee-7a86-375f-eed1-44dedb19530d: Installed remote snapshot
2026-04-23T00:15:38.581Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: pipelining replication: peer="{Voter server-35ab28ee-7a86-375f-eed1-44dedb19530d 35ab28ee-7a86-375f-eed1-44dedb19530d}"
2026-04-23T00:15:38.581Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:15:38.584Z [ERROR] server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: failed to get log: index=2 error="log not found"
2026-04-23T00:15:38.584Z [INFO]  creating new snapshot: path=/tmp/raft66257850/snapshots/2-15-1776903338584.tmp
2026-04-23T00:15:38.584Z [INFO]  server-48c53d64-11d7-eecd-db69-95d53bc35ab9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:38.585Z [INFO]  server-48c53d64-11d7-eecd-db69-95d53bc35ab9: copied to local snapshot: bytes=71
2026-04-23T00:15:38.585Z [INFO]  server-48c53d64-11d7-eecd-db69-95d53bc35ab9: snapshot restore progress: id=2-15-1776903338584 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:38.585Z [INFO]  server-48c53d64-11d7-eecd-db69-95d53bc35ab9: removing all old logs from log store
2026-04-23T00:15:38.585Z [INFO]  server-48c53d64-11d7-eecd-db69-95d53bc35ab9: compacting logs: from=1 to=2
2026-04-23T00:15:38.585Z [INFO]  server-48c53d64-11d7-eecd-db69-95d53bc35ab9: Installed remote snapshot
2026-04-23T00:15:38.585Z [INFO]  server-5296228a-f227-b68e-c4f4-4f5d8a0ecb85: pipelining replication: peer="{Voter server-48c53d64-11d7-eecd-db69-95d53bc35ab9 48c53d64-11d7-eecd-db69-95d53bc35ab9}"
2026-04-23T00:15:39.587Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:15:38.581962528 +0000 UTC m=+55.621563885 (2 nodes), 544ns from start of poll, 1.594355408s from cluster start. Timeout at 2026-04-23 00:15:39.586985284 +0000 UTC m=+56.626586638, 1.005022753s after stability
2026-04-23T00:15:39.599Z [INFO]  server-724b8fe5-dc70-e834-12be-0353675967eb: aborting pipeline replication: peer="{Voter server-59ad74b9-38d1-b8ff-1a86-f3b27524db6b 59ad74b9-38d1-b8ff-1a86-f3b27524db6b}"
2026-04-23T00:15:39.599Z [INFO]  server-724b8fe5-dc70-e834-12be-0353675967eb: aborting pipeline replication: peer="{Voter server-cecf1882-2645-699c-fd63-4100589bd5af cecf1882-2645-699c-fd63-4100589bd5af}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false
2026-04-23T00:15:39.600Z [INFO]  server-cab0e6f3-bce0-e388-d223-fa1c3b70b3f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cab0e6f3-bce0-e388-d223-fa1c3b70b3f5 Address:cab0e6f3-bce0-e388-d223-fa1c3b70b3f5} {Suffrage:Voter ID:server-9092a628-041e-b308-65a1-765bf4b55497 Address:9092a628-041e-b308-65a1-765bf4b55497} {Suffrage:Voter ID:server-42e9ab5d-1008-f004-0930-38db6dfa1a05 Address:42e9ab5d-1008-f004-0930-38db6dfa1a05}]"
2026-04-23T00:15:39.600Z [INFO]  server-cab0e6f3-bce0-e388-d223-fa1c3b70b3f5: entering follower state: follower="Node at cab0e6f3-bce0-e388-d223-fa1c3b70b3f5 [Follower]" leader-address= leader-id=
2026-04-23T00:15:39.600Z [INFO]  server-9092a628-041e-b308-65a1-765bf4b55497: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cab0e6f3-bce0-e388-d223-fa1c3b70b3f5 Address:cab0e6f3-bce0-e388-d223-fa1c3b70b3f5} {Suffrage:Voter ID:server-9092a628-041e-b308-65a1-765bf4b55497 Address:9092a628-041e-b308-65a1-765bf4b55497} {Suffrage:Voter ID:server-42e9ab5d-1008-f004-0930-38db6dfa1a05 Address:42e9ab5d-1008-f004-0930-38db6dfa1a05}]"
2026-04-23T00:15:39.600Z [INFO]  server-9092a628-041e-b308-65a1-765bf4b55497: entering follower state: follower="Node at 9092a628-041e-b308-65a1-765bf4b55497 [Follower]" leader-address= leader-id=
2026-04-23T00:15:39.600Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cab0e6f3-bce0-e388-d223-fa1c3b70b3f5 Address:cab0e6f3-bce0-e388-d223-fa1c3b70b3f5} {Suffrage:Voter ID:server-9092a628-041e-b308-65a1-765bf4b55497 Address:9092a628-041e-b308-65a1-765bf4b55497} {Suffrage:Voter ID:server-42e9ab5d-1008-f004-0930-38db6dfa1a05 Address:42e9ab5d-1008-f004-0930-38db6dfa1a05}]"
2026-04-23T00:15:39.600Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:39.600Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: entering follower state: follower="Node at 42e9ab5d-1008-f004-0930-38db6dfa1a05 [Follower]" leader-address= leader-id=
2026-04-23T00:15:40.347Z [WARN]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:40.347Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: entering candidate state: node="Node at 42e9ab5d-1008-f004-0930-38db6dfa1a05 [Candidate]" term=2
2026-04-23T00:15:40.347Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: election won: term=2 tally=2
2026-04-23T00:15:40.347Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: entering leader state: leader="Node at 42e9ab5d-1008-f004-0930-38db6dfa1a05 [Leader]"
2026-04-23T00:15:40.347Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: added peer, starting replication: peer=server-cab0e6f3-bce0-e388-d223-fa1c3b70b3f5
2026-04-23T00:15:40.347Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: added peer, starting replication: peer=server-9092a628-041e-b308-65a1-765bf4b55497
2026-04-23T00:15:40.348Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: pipelining replication: peer="{Voter server-9092a628-041e-b308-65a1-765bf4b55497 9092a628-041e-b308-65a1-765bf4b55497}"
2026-04-23T00:15:40.348Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: pipelining replication: peer="{Voter server-cab0e6f3-bce0-e388-d223-fa1c3b70b3f5 cab0e6f3-bce0-e388-d223-fa1c3b70b3f5}"
2026-04-23T00:15:41.353Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:40.347242315 +0000 UTC m=+57.386843752 (1 nodes), 746.416951ms from start of poll, 746.69321ms from cluster start. Timeout at 2026-04-23 00:15:41.35224835 +0000 UTC m=+58.391849702, 1.00500595s after stability
2026-04-23T00:15:41.353Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: starting snapshot up to: index=12
2026-04-23T00:15:41.353Z [INFO]  creating new snapshot: path=/tmp/raft1424757365/snapshots/2-12-1776903341353.tmp
2026-04-23T00:15:41.354Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: snapshot complete up to: index=12
2026-04-23T00:15:41.354Z [INFO]  creating new snapshot: path=/tmp/raft1424757365/snapshots/2-113-1776903341354.tmp
2026-04-23T00:15:41.355Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: copied to local snapshot: bytes=71
2026-04-23T00:15:41.355Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: snapshot restore progress: id=2-113-1776903341354 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:41.355Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: restored user snapshot: index=113
2026-04-23T00:15:41.355Z [ERROR] server-42e9ab5d-1008-f004-0930-38db6dfa1a05: failed to get log: index=23 error="log not found"
2026-04-23T00:15:41.355Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: aborting pipeline replication: peer="{Voter server-9092a628-041e-b308-65a1-765bf4b55497 9092a628-041e-b308-65a1-765bf4b55497}"
2026-04-23T00:15:41.355Z [ERROR] server-42e9ab5d-1008-f004-0930-38db6dfa1a05: failed to get log: index=23 error="log not found"
2026-04-23T00:15:41.355Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: aborting pipeline replication: peer="{Voter server-cab0e6f3-bce0-e388-d223-fa1c3b70b3f5 cab0e6f3-bce0-e388-d223-fa1c3b70b3f5}"
2026-04-23T00:15:41.365Z [ERROR] server-42e9ab5d-1008-f004-0930-38db6dfa1a05: failed to get log: index=23 error="log not found"
2026-04-23T00:15:41.365Z [ERROR] server-42e9ab5d-1008-f004-0930-38db6dfa1a05: failed to get log: index=23 error="log not found"
2026-04-23T00:15:41.365Z [INFO]  creating new snapshot: path=/tmp/raft3132207168/snapshots/2-113-1776903341365.tmp
2026-04-23T00:15:41.365Z [INFO]  creating new snapshot: path=/tmp/raft2904318219/snapshots/2-113-1776903341365.tmp
2026-04-23T00:15:41.366Z [INFO]  server-9092a628-041e-b308-65a1-765bf4b55497: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:41.366Z [INFO]  server-cab0e6f3-bce0-e388-d223-fa1c3b70b3f5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:41.366Z [INFO]  server-9092a628-041e-b308-65a1-765bf4b55497: copied to local snapshot: bytes=71
2026-04-23T00:15:41.366Z [INFO]  server-9092a628-041e-b308-65a1-765bf4b55497: snapshot restore progress: id=2-113-1776903341365 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:41.366Z [INFO]  server-cab0e6f3-bce0-e388-d223-fa1c3b70b3f5: copied to local snapshot: bytes=71
2026-04-23T00:15:41.366Z [INFO]  server-9092a628-041e-b308-65a1-765bf4b55497: Installed remote snapshot
2026-04-23T00:15:41.366Z [WARN]  server-9092a628-041e-b308-65a1-765bf4b55497: skipping application of old log: index=22
2026-04-23T00:15:41.366Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: pipelining replication: peer="{Voter server-9092a628-041e-b308-65a1-765bf4b55497 9092a628-041e-b308-65a1-765bf4b55497}"
2026-04-23T00:15:41.366Z [INFO]  server-cab0e6f3-bce0-e388-d223-fa1c3b70b3f5: snapshot restore progress: id=2-113-1776903341365 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:41.366Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:15:41.366Z [INFO]  server-cab0e6f3-bce0-e388-d223-fa1c3b70b3f5: Installed remote snapshot
2026-04-23T00:15:41.366Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: pipelining replication: peer="{Voter server-cab0e6f3-bce0-e388-d223-fa1c3b70b3f5 cab0e6f3-bce0-e388-d223-fa1c3b70b3f5}"
2026-04-23T00:15:42.372Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:15:41.366824798 +0000 UTC m=+58.406426166 (2 nodes), 878ns from start of poll, 1.766275624s from cluster start. Timeout at 2026-04-23 00:15:42.371834272 +0000 UTC m=+59.411435631, 1.005009465s after stability
2026-04-23T00:15:42.384Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: aborting pipeline replication: peer="{Voter server-9092a628-041e-b308-65a1-765bf4b55497 9092a628-041e-b308-65a1-765bf4b55497}"
2026-04-23T00:15:42.384Z [INFO]  server-42e9ab5d-1008-f004-0930-38db6dfa1a05: aborting pipeline replication: peer="{Voter server-cab0e6f3-bce0-e388-d223-fa1c3b70b3f5 cab0e6f3-bce0-e388-d223-fa1c3b70b3f5}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false
2026-04-23T00:15:42.385Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0 Address:aa51b7d8-72eb-7f98-f2d6-f47cf54269d0} {Suffrage:Voter ID:server-c45a1ba2-ae1b-3d3b-b850-689fbc87656d Address:c45a1ba2-ae1b-3d3b-b850-689fbc87656d} {Suffrage:Voter ID:server-823c96dd-fd5f-e4d0-20a3-55bf23f323f0 Address:823c96dd-fd5f-e4d0-20a3-55bf23f323f0}]"
2026-04-23T00:15:42.385Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: entering follower state: follower="Node at aa51b7d8-72eb-7f98-f2d6-f47cf54269d0 [Follower]" leader-address= leader-id=
2026-04-23T00:15:42.385Z [INFO]  server-c45a1ba2-ae1b-3d3b-b850-689fbc87656d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0 Address:aa51b7d8-72eb-7f98-f2d6-f47cf54269d0} {Suffrage:Voter ID:server-c45a1ba2-ae1b-3d3b-b850-689fbc87656d Address:c45a1ba2-ae1b-3d3b-b850-689fbc87656d} {Suffrage:Voter ID:server-823c96dd-fd5f-e4d0-20a3-55bf23f323f0 Address:823c96dd-fd5f-e4d0-20a3-55bf23f323f0}]"
2026-04-23T00:15:42.385Z [INFO]  server-c45a1ba2-ae1b-3d3b-b850-689fbc87656d: entering follower state: follower="Node at c45a1ba2-ae1b-3d3b-b850-689fbc87656d [Follower]" leader-address= leader-id=
2026-04-23T00:15:42.385Z [INFO]  server-823c96dd-fd5f-e4d0-20a3-55bf23f323f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0 Address:aa51b7d8-72eb-7f98-f2d6-f47cf54269d0} {Suffrage:Voter ID:server-c45a1ba2-ae1b-3d3b-b850-689fbc87656d Address:c45a1ba2-ae1b-3d3b-b850-689fbc87656d} {Suffrage:Voter ID:server-823c96dd-fd5f-e4d0-20a3-55bf23f323f0 Address:823c96dd-fd5f-e4d0-20a3-55bf23f323f0}]"
2026-04-23T00:15:42.385Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:42.385Z [INFO]  server-823c96dd-fd5f-e4d0-20a3-55bf23f323f0: entering follower state: follower="Node at 823c96dd-fd5f-e4d0-20a3-55bf23f323f0 [Follower]" leader-address= leader-id=
2026-04-23T00:15:42.911Z [WARN]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:42.911Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: entering candidate state: node="Node at aa51b7d8-72eb-7f98-f2d6-f47cf54269d0 [Candidate]" term=2
2026-04-23T00:15:42.911Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: election won: term=2 tally=2
2026-04-23T00:15:42.911Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: entering leader state: leader="Node at aa51b7d8-72eb-7f98-f2d6-f47cf54269d0 [Leader]"
2026-04-23T00:15:42.911Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: added peer, starting replication: peer=server-c45a1ba2-ae1b-3d3b-b850-689fbc87656d
2026-04-23T00:15:42.911Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: added peer, starting replication: peer=server-823c96dd-fd5f-e4d0-20a3-55bf23f323f0
2026-04-23T00:15:42.911Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: pipelining replication: peer="{Voter server-c45a1ba2-ae1b-3d3b-b850-689fbc87656d c45a1ba2-ae1b-3d3b-b850-689fbc87656d}"
2026-04-23T00:15:42.911Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: pipelining replication: peer="{Voter server-823c96dd-fd5f-e4d0-20a3-55bf23f323f0 823c96dd-fd5f-e4d0-20a3-55bf23f323f0}"
2026-04-23T00:15:43.917Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:42.911701427 +0000 UTC m=+59.951302800 (1 nodes), 525.932862ms from start of poll, 526.252913ms from cluster start. Timeout at 2026-04-23 00:15:43.916704206 +0000 UTC m=+60.956305559, 1.005002759s after stability
2026-04-23T00:15:43.917Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: starting snapshot up to: index=12
2026-04-23T00:15:43.917Z [INFO]  creating new snapshot: path=/tmp/raft2254030808/snapshots/2-12-1776903343917.tmp
2026-04-23T00:15:43.918Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: snapshot complete up to: index=12
2026-04-23T00:15:43.920Z [INFO]  creating new snapshot: path=/tmp/raft2254030808/snapshots/2-113-1776903343920.tmp
2026-04-23T00:15:43.920Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: copied to local snapshot: bytes=71
2026-04-23T00:15:43.920Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: snapshot restore progress: id=2-113-1776903343920 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:43.920Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: removing all old logs from log store
2026-04-23T00:15:43.920Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: compacting logs: from=1 to=22
2026-04-23T00:15:43.920Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: restored user snapshot: index=113
2026-04-23T00:15:43.920Z [ERROR] server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: failed to get log: index=22 error="log not found"
2026-04-23T00:15:43.920Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: aborting pipeline replication: peer="{Voter server-823c96dd-fd5f-e4d0-20a3-55bf23f323f0 823c96dd-fd5f-e4d0-20a3-55bf23f323f0}"
2026-04-23T00:15:43.920Z [ERROR] server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: failed to get log: index=22 error="log not found"
2026-04-23T00:15:43.920Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: aborting pipeline replication: peer="{Voter server-c45a1ba2-ae1b-3d3b-b850-689fbc87656d c45a1ba2-ae1b-3d3b-b850-689fbc87656d}"
2026-04-23T00:15:43.926Z [ERROR] server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: failed to get log: index=22 error="log not found"
2026-04-23T00:15:43.926Z [INFO]  creating new snapshot: path=/tmp/raft1507396341/snapshots/2-113-1776903343926.tmp
2026-04-23T00:15:43.926Z [INFO]  server-823c96dd-fd5f-e4d0-20a3-55bf23f323f0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:43.926Z [INFO]  server-823c96dd-fd5f-e4d0-20a3-55bf23f323f0: copied to local snapshot: bytes=71
2026-04-23T00:15:43.927Z [INFO]  server-823c96dd-fd5f-e4d0-20a3-55bf23f323f0: snapshot restore progress: id=2-113-1776903343926 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:43.927Z [INFO]  server-823c96dd-fd5f-e4d0-20a3-55bf23f323f0: removing all old logs from log store
2026-04-23T00:15:43.927Z [INFO]  server-823c96dd-fd5f-e4d0-20a3-55bf23f323f0: compacting logs: from=1 to=22
2026-04-23T00:15:43.927Z [INFO]  server-823c96dd-fd5f-e4d0-20a3-55bf23f323f0: Installed remote snapshot
2026-04-23T00:15:43.927Z [WARN]  server-823c96dd-fd5f-e4d0-20a3-55bf23f323f0: skipping application of old log: index=22
2026-04-23T00:15:43.927Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: pipelining replication: peer="{Voter server-823c96dd-fd5f-e4d0-20a3-55bf23f323f0 823c96dd-fd5f-e4d0-20a3-55bf23f323f0}"
2026-04-23T00:15:43.927Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:15:43.928Z [ERROR] server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: failed to get log: index=22 error="log not found"
2026-04-23T00:15:43.928Z [INFO]  creating new snapshot: path=/tmp/raft4230282724/snapshots/2-113-1776903343928.tmp
2026-04-23T00:15:43.928Z [INFO]  server-c45a1ba2-ae1b-3d3b-b850-689fbc87656d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:43.929Z [INFO]  server-c45a1ba2-ae1b-3d3b-b850-689fbc87656d: copied to local snapshot: bytes=71
2026-04-23T00:15:43.929Z [INFO]  server-c45a1ba2-ae1b-3d3b-b850-689fbc87656d: snapshot restore progress: id=2-113-1776903343928 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:43.929Z [INFO]  server-c45a1ba2-ae1b-3d3b-b850-689fbc87656d: removing all old logs from log store
2026-04-23T00:15:43.929Z [INFO]  server-c45a1ba2-ae1b-3d3b-b850-689fbc87656d: compacting logs: from=1 to=22
2026-04-23T00:15:43.929Z [INFO]  server-c45a1ba2-ae1b-3d3b-b850-689fbc87656d: Installed remote snapshot
2026-04-23T00:15:43.929Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: pipelining replication: peer="{Voter server-c45a1ba2-ae1b-3d3b-b850-689fbc87656d c45a1ba2-ae1b-3d3b-b850-689fbc87656d}"
2026-04-23T00:15:44.932Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:15:43.927348078 +0000 UTC m=+60.966949452 (2 nodes), 603ns from start of poll, 1.541899565s from cluster start. Timeout at 2026-04-23 00:15:44.932360494 +0000 UTC m=+61.971961850, 1.005012398s after stability
2026-04-23T00:15:44.943Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: aborting pipeline replication: peer="{Voter server-823c96dd-fd5f-e4d0-20a3-55bf23f323f0 823c96dd-fd5f-e4d0-20a3-55bf23f323f0}"
2026-04-23T00:15:44.943Z [INFO]  server-aa51b7d8-72eb-7f98-f2d6-f47cf54269d0: aborting pipeline replication: peer="{Voter server-c45a1ba2-ae1b-3d3b-b850-689fbc87656d c45a1ba2-ae1b-3d3b-b850-689fbc87656d}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true
2026-04-23T00:15:44.947Z [INFO]  server-0921443b-1e0f-8166-6688-24272b9171cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0921443b-1e0f-8166-6688-24272b9171cd Address:0921443b-1e0f-8166-6688-24272b9171cd} {Suffrage:Voter ID:server-b7ff5aec-4326-7578-31f9-70e33ebf244e Address:b7ff5aec-4326-7578-31f9-70e33ebf244e} {Suffrage:Voter ID:server-3caa13cf-d96f-0c08-e00e-2fb392552276 Address:3caa13cf-d96f-0c08-e00e-2fb392552276}]"
2026-04-23T00:15:44.948Z [INFO]  server-0921443b-1e0f-8166-6688-24272b9171cd: entering follower state: follower="Node at 0921443b-1e0f-8166-6688-24272b9171cd [Follower]" leader-address= leader-id=
2026-04-23T00:15:44.948Z [INFO]  server-b7ff5aec-4326-7578-31f9-70e33ebf244e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0921443b-1e0f-8166-6688-24272b9171cd Address:0921443b-1e0f-8166-6688-24272b9171cd} {Suffrage:Voter ID:server-b7ff5aec-4326-7578-31f9-70e33ebf244e Address:b7ff5aec-4326-7578-31f9-70e33ebf244e} {Suffrage:Voter ID:server-3caa13cf-d96f-0c08-e00e-2fb392552276 Address:3caa13cf-d96f-0c08-e00e-2fb392552276}]"
2026-04-23T00:15:44.948Z [INFO]  server-3caa13cf-d96f-0c08-e00e-2fb392552276: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0921443b-1e0f-8166-6688-24272b9171cd Address:0921443b-1e0f-8166-6688-24272b9171cd} {Suffrage:Voter ID:server-b7ff5aec-4326-7578-31f9-70e33ebf244e Address:b7ff5aec-4326-7578-31f9-70e33ebf244e} {Suffrage:Voter ID:server-3caa13cf-d96f-0c08-e00e-2fb392552276 Address:3caa13cf-d96f-0c08-e00e-2fb392552276}]"
2026-04-23T00:15:44.948Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:44.948Z [INFO]  server-b7ff5aec-4326-7578-31f9-70e33ebf244e: entering follower state: follower="Node at b7ff5aec-4326-7578-31f9-70e33ebf244e [Follower]" leader-address= leader-id=
2026-04-23T00:15:44.948Z [INFO]  server-3caa13cf-d96f-0c08-e00e-2fb392552276: entering follower state: follower="Node at 3caa13cf-d96f-0c08-e00e-2fb392552276 [Follower]" leader-address= leader-id=
2026-04-23T00:15:45.749Z [WARN]  server-b7ff5aec-4326-7578-31f9-70e33ebf244e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:45.749Z [INFO]  server-b7ff5aec-4326-7578-31f9-70e33ebf244e: entering candidate state: node="Node at b7ff5aec-4326-7578-31f9-70e33ebf244e [Candidate]" term=2
2026-04-23T00:15:45.749Z [INFO]  server-b7ff5aec-4326-7578-31f9-70e33ebf244e: election won: term=2 tally=2
2026-04-23T00:15:45.749Z [INFO]  server-b7ff5aec-4326-7578-31f9-70e33ebf244e: entering leader state: leader="Node at b7ff5aec-4326-7578-31f9-70e33ebf244e [Leader]"
2026-04-23T00:15:45.749Z [INFO]  server-b7ff5aec-4326-7578-31f9-70e33ebf244e: added peer, starting replication: peer=server-0921443b-1e0f-8166-6688-24272b9171cd
2026-04-23T00:15:45.749Z [INFO]  server-b7ff5aec-4326-7578-31f9-70e33ebf244e: added peer, starting replication: peer=server-3caa13cf-d96f-0c08-e00e-2fb392552276
2026-04-23T00:15:45.749Z [INFO]  server-b7ff5aec-4326-7578-31f9-70e33ebf244e: pipelining replication: peer="{Voter server-3caa13cf-d96f-0c08-e00e-2fb392552276 3caa13cf-d96f-0c08-e00e-2fb392552276}"
2026-04-23T00:15:45.749Z [INFO]  server-b7ff5aec-4326-7578-31f9-70e33ebf244e: pipelining replication: peer="{Voter server-0921443b-1e0f-8166-6688-24272b9171cd 0921443b-1e0f-8166-6688-24272b9171cd}"
2026-04-23T00:15:46.754Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:45.749639788 +0000 UTC m=+62.789241155 (1 nodes), 801.483074ms from start of poll, 801.774802ms from cluster start. Timeout at 2026-04-23 00:15:46.754641174 +0000 UTC m=+63.794242529, 1.005001374s after stability
2026-04-23T00:15:46.754Z [INFO]  server-b7ff5aec-4326-7578-31f9-70e33ebf244e: starting snapshot up to: index=12
2026-04-23T00:15:46.755Z [INFO]  creating new snapshot: path=/tmp/raft1800259483/snapshots/2-12-1776903346755.tmp
2026-04-23T00:15:46.755Z [INFO]  server-b7ff5aec-4326-7578-31f9-70e33ebf244e: snapshot complete up to: index=12
2026-04-23T00:15:46.756Z [INFO]  server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95 Address:d3ffce61-a79f-5cb8-9b14-cb01a4287f95} {Suffrage:Voter ID:server-006520d0-fedd-ff3e-6d60-72b98b314384 Address:006520d0-fedd-ff3e-6d60-72b98b314384} {Suffrage:Voter ID:server-16072cba-ec87-672e-2e30-65af8e225f1e Address:16072cba-ec87-672e-2e30-65af8e225f1e}]"
2026-04-23T00:15:46.756Z [INFO]  server-006520d0-fedd-ff3e-6d60-72b98b314384: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95 Address:d3ffce61-a79f-5cb8-9b14-cb01a4287f95} {Suffrage:Voter ID:server-006520d0-fedd-ff3e-6d60-72b98b314384 Address:006520d0-fedd-ff3e-6d60-72b98b314384} {Suffrage:Voter ID:server-16072cba-ec87-672e-2e30-65af8e225f1e Address:16072cba-ec87-672e-2e30-65af8e225f1e}]"
2026-04-23T00:15:46.756Z [INFO]  server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: entering follower state: follower="Node at d3ffce61-a79f-5cb8-9b14-cb01a4287f95 [Follower]" leader-address= leader-id=
2026-04-23T00:15:46.756Z [INFO]  server-006520d0-fedd-ff3e-6d60-72b98b314384: entering follower state: follower="Node at 006520d0-fedd-ff3e-6d60-72b98b314384 [Follower]" leader-address= leader-id=
2026-04-23T00:15:46.756Z [INFO]  server-16072cba-ec87-672e-2e30-65af8e225f1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95 Address:d3ffce61-a79f-5cb8-9b14-cb01a4287f95} {Suffrage:Voter ID:server-006520d0-fedd-ff3e-6d60-72b98b314384 Address:006520d0-fedd-ff3e-6d60-72b98b314384} {Suffrage:Voter ID:server-16072cba-ec87-672e-2e30-65af8e225f1e Address:16072cba-ec87-672e-2e30-65af8e225f1e}]"
2026-04-23T00:15:46.756Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:46.756Z [INFO]  server-16072cba-ec87-672e-2e30-65af8e225f1e: entering follower state: follower="Node at 16072cba-ec87-672e-2e30-65af8e225f1e [Follower]" leader-address= leader-id=
2026-04-23T00:15:47.303Z [WARN]  server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:47.303Z [INFO]  server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: entering candidate state: node="Node at d3ffce61-a79f-5cb8-9b14-cb01a4287f95 [Candidate]" term=2
2026-04-23T00:15:47.303Z [INFO]  server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: election won: term=2 tally=2
2026-04-23T00:15:47.303Z [INFO]  server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: entering leader state: leader="Node at d3ffce61-a79f-5cb8-9b14-cb01a4287f95 [Leader]"
2026-04-23T00:15:47.303Z [INFO]  server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: added peer, starting replication: peer=server-006520d0-fedd-ff3e-6d60-72b98b314384
2026-04-23T00:15:47.303Z [INFO]  server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: added peer, starting replication: peer=server-16072cba-ec87-672e-2e30-65af8e225f1e
2026-04-23T00:15:47.303Z [INFO]  server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: pipelining replication: peer="{Voter server-006520d0-fedd-ff3e-6d60-72b98b314384 006520d0-fedd-ff3e-6d60-72b98b314384}"
2026-04-23T00:15:47.304Z [INFO]  server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: pipelining replication: peer="{Voter server-16072cba-ec87-672e-2e30-65af8e225f1e 16072cba-ec87-672e-2e30-65af8e225f1e}"
2026-04-23T00:15:48.309Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:47.303875538 +0000 UTC m=+64.343476889 (1 nodes), 547.185247ms from start of poll, 547.520984ms from cluster start. Timeout at 2026-04-23 00:15:48.308878297 +0000 UTC m=+65.348479654, 1.005002765s after stability
2026-04-23T00:15:48.309Z [INFO]  creating new snapshot: path=/tmp/raft3916859451/snapshots/2-113-1776903348309.tmp
2026-04-23T00:15:48.309Z [INFO]  server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: copied to local snapshot: bytes=71
2026-04-23T00:15:48.310Z [INFO]  server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: snapshot restore progress: id=2-113-1776903348309 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:48.310Z [INFO]  server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: restored user snapshot: index=113
2026-04-23T00:15:48.310Z [ERROR] server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: failed to get log: index=3 error="log not found"
2026-04-23T00:15:48.310Z [INFO]  server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: aborting pipeline replication: peer="{Voter server-16072cba-ec87-672e-2e30-65af8e225f1e 16072cba-ec87-672e-2e30-65af8e225f1e}"
2026-04-23T00:15:48.310Z [ERROR] server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: failed to get log: index=3 error="log not found"
2026-04-23T00:15:48.310Z [INFO]  server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: aborting pipeline replication: peer="{Voter server-006520d0-fedd-ff3e-6d60-72b98b314384 006520d0-fedd-ff3e-6d60-72b98b314384}"
2026-04-23T00:15:48.316Z [ERROR] server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: failed to get log: index=3 error="log not found"
2026-04-23T00:15:48.317Z [INFO]  creating new snapshot: path=/tmp/raft1181701945/snapshots/2-113-1776903348317.tmp
2026-04-23T00:15:48.317Z [INFO]  server-006520d0-fedd-ff3e-6d60-72b98b314384: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:48.317Z [INFO]  server-006520d0-fedd-ff3e-6d60-72b98b314384: copied to local snapshot: bytes=71
2026-04-23T00:15:48.317Z [INFO]  server-006520d0-fedd-ff3e-6d60-72b98b314384: snapshot restore progress: id=2-113-1776903348317 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:48.317Z [INFO]  server-006520d0-fedd-ff3e-6d60-72b98b314384: Installed remote snapshot
2026-04-23T00:15:48.317Z [INFO]  server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: pipelining replication: peer="{Voter server-006520d0-fedd-ff3e-6d60-72b98b314384 006520d0-fedd-ff3e-6d60-72b98b314384}"
2026-04-23T00:15:48.317Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:15:48.320Z [ERROR] server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: failed to get log: index=3 error="log not found"
2026-04-23T00:15:48.320Z [INFO]  creating new snapshot: path=/tmp/raft2298133010/snapshots/2-113-1776903348320.tmp
2026-04-23T00:15:48.320Z [INFO]  server-16072cba-ec87-672e-2e30-65af8e225f1e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:48.321Z [INFO]  server-16072cba-ec87-672e-2e30-65af8e225f1e: copied to local snapshot: bytes=71
2026-04-23T00:15:48.321Z [INFO]  server-16072cba-ec87-672e-2e30-65af8e225f1e: snapshot restore progress: id=2-113-1776903348320 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:48.321Z [INFO]  server-16072cba-ec87-672e-2e30-65af8e225f1e: Installed remote snapshot
2026-04-23T00:15:48.321Z [INFO]  server-d3ffce61-a79f-5cb8-9b14-cb01a4287f95: pipelining replication: peer="{Voter server-16072cba-ec87-672e-2e30-65af8e225f1e 16072cba-ec87-672e-2e30-65af8e225f1e}"
2026-04-23T00:15:49.323Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:15:48.317947209 +0000 UTC m=+65.357548560 (2 nodes), 849ns from start of poll, 1.561592655s from cluster start. Timeout at 2026-04-23 00:15:49.322964759 +0000 UTC m=+66.362566113, 1.005017553s after stability
2026-04-23T00:15:49.334Z [INFO]  server-b7ff5aec-4326-7578-31f9-70e33ebf244e: aborting pipeline replication: peer="{Voter server-3caa13cf-d96f-0c08-e00e-2fb392552276 3caa13cf-d96f-0c08-e00e-2fb392552276}"
2026-04-23T00:15:49.334Z [INFO]  server-b7ff5aec-4326-7578-31f9-70e33ebf244e: aborting pipeline replication: peer="{Voter server-0921443b-1e0f-8166-6688-24272b9171cd 0921443b-1e0f-8166-6688-24272b9171cd}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true
2026-04-23T00:15:49.335Z [INFO]  server-e7df154c-0bd0-9973-bf58-381479170cb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7df154c-0bd0-9973-bf58-381479170cb2 Address:e7df154c-0bd0-9973-bf58-381479170cb2} {Suffrage:Voter ID:server-1a621df2-4089-f41b-123c-2db018c1cd96 Address:1a621df2-4089-f41b-123c-2db018c1cd96} {Suffrage:Voter ID:server-8c74f8df-999d-11b5-7c6d-5b379fd67607 Address:8c74f8df-999d-11b5-7c6d-5b379fd67607}]"
2026-04-23T00:15:49.335Z [INFO]  server-1a621df2-4089-f41b-123c-2db018c1cd96: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7df154c-0bd0-9973-bf58-381479170cb2 Address:e7df154c-0bd0-9973-bf58-381479170cb2} {Suffrage:Voter ID:server-1a621df2-4089-f41b-123c-2db018c1cd96 Address:1a621df2-4089-f41b-123c-2db018c1cd96} {Suffrage:Voter ID:server-8c74f8df-999d-11b5-7c6d-5b379fd67607 Address:8c74f8df-999d-11b5-7c6d-5b379fd67607}]"
2026-04-23T00:15:49.335Z [INFO]  server-e7df154c-0bd0-9973-bf58-381479170cb2: entering follower state: follower="Node at e7df154c-0bd0-9973-bf58-381479170cb2 [Follower]" leader-address= leader-id=
2026-04-23T00:15:49.335Z [INFO]  server-1a621df2-4089-f41b-123c-2db018c1cd96: entering follower state: follower="Node at 1a621df2-4089-f41b-123c-2db018c1cd96 [Follower]" leader-address= leader-id=
2026-04-23T00:15:49.335Z [INFO]  server-8c74f8df-999d-11b5-7c6d-5b379fd67607: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7df154c-0bd0-9973-bf58-381479170cb2 Address:e7df154c-0bd0-9973-bf58-381479170cb2} {Suffrage:Voter ID:server-1a621df2-4089-f41b-123c-2db018c1cd96 Address:1a621df2-4089-f41b-123c-2db018c1cd96} {Suffrage:Voter ID:server-8c74f8df-999d-11b5-7c6d-5b379fd67607 Address:8c74f8df-999d-11b5-7c6d-5b379fd67607}]"
2026-04-23T00:15:49.335Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:49.335Z [INFO]  server-8c74f8df-999d-11b5-7c6d-5b379fd67607: entering follower state: follower="Node at 8c74f8df-999d-11b5-7c6d-5b379fd67607 [Follower]" leader-address= leader-id=
2026-04-23T00:15:49.888Z [WARN]  server-e7df154c-0bd0-9973-bf58-381479170cb2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:49.888Z [INFO]  server-e7df154c-0bd0-9973-bf58-381479170cb2: entering candidate state: node="Node at e7df154c-0bd0-9973-bf58-381479170cb2 [Candidate]" term=2
2026-04-23T00:15:49.888Z [INFO]  server-e7df154c-0bd0-9973-bf58-381479170cb2: election won: term=2 tally=2
2026-04-23T00:15:49.888Z [INFO]  server-e7df154c-0bd0-9973-bf58-381479170cb2: entering leader state: leader="Node at e7df154c-0bd0-9973-bf58-381479170cb2 [Leader]"
2026-04-23T00:15:49.888Z [INFO]  server-e7df154c-0bd0-9973-bf58-381479170cb2: added peer, starting replication: peer=server-1a621df2-4089-f41b-123c-2db018c1cd96
2026-04-23T00:15:49.888Z [INFO]  server-e7df154c-0bd0-9973-bf58-381479170cb2: added peer, starting replication: peer=server-8c74f8df-999d-11b5-7c6d-5b379fd67607
2026-04-23T00:15:49.888Z [INFO]  server-e7df154c-0bd0-9973-bf58-381479170cb2: pipelining replication: peer="{Voter server-1a621df2-4089-f41b-123c-2db018c1cd96 1a621df2-4089-f41b-123c-2db018c1cd96}"
2026-04-23T00:15:49.888Z [INFO]  server-e7df154c-0bd0-9973-bf58-381479170cb2: pipelining replication: peer="{Voter server-8c74f8df-999d-11b5-7c6d-5b379fd67607 8c74f8df-999d-11b5-7c6d-5b379fd67607}"
2026-04-23T00:15:50.893Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:49.88871659 +0000 UTC m=+66.928317962 (1 nodes), 552.881231ms from start of poll, 553.193714ms from cluster start. Timeout at 2026-04-23 00:15:50.893719801 +0000 UTC m=+67.933321152, 1.00500319s after stability
2026-04-23T00:15:50.894Z [INFO]  server-e7df154c-0bd0-9973-bf58-381479170cb2: starting snapshot up to: index=12
2026-04-23T00:15:50.894Z [INFO]  creating new snapshot: path=/tmp/raft3201392458/snapshots/2-12-1776903350894.tmp
2026-04-23T00:15:50.894Z [INFO]  server-e7df154c-0bd0-9973-bf58-381479170cb2: snapshot complete up to: index=12
2026-04-23T00:15:50.895Z [INFO]  server-12730079-5a51-ddcf-1045-1c842b77b991: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12730079-5a51-ddcf-1045-1c842b77b991 Address:12730079-5a51-ddcf-1045-1c842b77b991} {Suffrage:Voter ID:server-3f765b4c-e903-2495-c1f6-1b59fb113b41 Address:3f765b4c-e903-2495-c1f6-1b59fb113b41} {Suffrage:Voter ID:server-70c7ede9-6eb8-3233-88ec-ebdca78baf35 Address:70c7ede9-6eb8-3233-88ec-ebdca78baf35}]"
2026-04-23T00:15:50.895Z [INFO]  server-12730079-5a51-ddcf-1045-1c842b77b991: entering follower state: follower="Node at 12730079-5a51-ddcf-1045-1c842b77b991 [Follower]" leader-address= leader-id=
2026-04-23T00:15:50.895Z [INFO]  server-3f765b4c-e903-2495-c1f6-1b59fb113b41: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12730079-5a51-ddcf-1045-1c842b77b991 Address:12730079-5a51-ddcf-1045-1c842b77b991} {Suffrage:Voter ID:server-3f765b4c-e903-2495-c1f6-1b59fb113b41 Address:3f765b4c-e903-2495-c1f6-1b59fb113b41} {Suffrage:Voter ID:server-70c7ede9-6eb8-3233-88ec-ebdca78baf35 Address:70c7ede9-6eb8-3233-88ec-ebdca78baf35}]"
2026-04-23T00:15:50.895Z [INFO]  server-3f765b4c-e903-2495-c1f6-1b59fb113b41: entering follower state: follower="Node at 3f765b4c-e903-2495-c1f6-1b59fb113b41 [Follower]" leader-address= leader-id=
2026-04-23T00:15:50.896Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12730079-5a51-ddcf-1045-1c842b77b991 Address:12730079-5a51-ddcf-1045-1c842b77b991} {Suffrage:Voter ID:server-3f765b4c-e903-2495-c1f6-1b59fb113b41 Address:3f765b4c-e903-2495-c1f6-1b59fb113b41} {Suffrage:Voter ID:server-70c7ede9-6eb8-3233-88ec-ebdca78baf35 Address:70c7ede9-6eb8-3233-88ec-ebdca78baf35}]"
2026-04-23T00:15:50.896Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:50.896Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: entering follower state: follower="Node at 70c7ede9-6eb8-3233-88ec-ebdca78baf35 [Follower]" leader-address= leader-id=
2026-04-23T00:15:51.401Z [WARN]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:51.402Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: entering candidate state: node="Node at 70c7ede9-6eb8-3233-88ec-ebdca78baf35 [Candidate]" term=2
2026-04-23T00:15:51.402Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: election won: term=2 tally=2
2026-04-23T00:15:51.402Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: entering leader state: leader="Node at 70c7ede9-6eb8-3233-88ec-ebdca78baf35 [Leader]"
2026-04-23T00:15:51.402Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: added peer, starting replication: peer=server-12730079-5a51-ddcf-1045-1c842b77b991
2026-04-23T00:15:51.402Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: added peer, starting replication: peer=server-3f765b4c-e903-2495-c1f6-1b59fb113b41
2026-04-23T00:15:51.402Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: pipelining replication: peer="{Voter server-3f765b4c-e903-2495-c1f6-1b59fb113b41 3f765b4c-e903-2495-c1f6-1b59fb113b41}"
2026-04-23T00:15:51.402Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: pipelining replication: peer="{Voter server-12730079-5a51-ddcf-1045-1c842b77b991 12730079-5a51-ddcf-1045-1c842b77b991}"
2026-04-23T00:15:52.407Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:51.402183409 +0000 UTC m=+68.441784761 (1 nodes), 505.210823ms from start of poll, 506.601433ms from cluster start. Timeout at 2026-04-23 00:15:52.407190426 +0000 UTC m=+69.446791781, 1.00500702s after stability
2026-04-23T00:15:52.407Z [INFO]  creating new snapshot: path=/tmp/raft1390106172/snapshots/2-113-1776903352407.tmp
2026-04-23T00:15:52.407Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: copied to local snapshot: bytes=71
2026-04-23T00:15:52.408Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: snapshot restore progress: id=2-113-1776903352407 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:52.408Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: removing all old logs from log store
2026-04-23T00:15:52.408Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: compacting logs: from=1 to=2
2026-04-23T00:15:52.408Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: restored user snapshot: index=113
2026-04-23T00:15:52.408Z [ERROR] server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: failed to get log: index=2 error="log not found"
2026-04-23T00:15:52.408Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: aborting pipeline replication: peer="{Voter server-3f765b4c-e903-2495-c1f6-1b59fb113b41 3f765b4c-e903-2495-c1f6-1b59fb113b41}"
2026-04-23T00:15:52.408Z [ERROR] server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: failed to get log: index=2 error="log not found"
2026-04-23T00:15:52.408Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: aborting pipeline replication: peer="{Voter server-12730079-5a51-ddcf-1045-1c842b77b991 12730079-5a51-ddcf-1045-1c842b77b991}"
2026-04-23T00:15:52.415Z [ERROR] server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: failed to get log: index=2 error="log not found"
2026-04-23T00:15:52.415Z [INFO]  creating new snapshot: path=/tmp/raft1843170528/snapshots/2-113-1776903352415.tmp
2026-04-23T00:15:52.415Z [INFO]  server-12730079-5a51-ddcf-1045-1c842b77b991: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:52.416Z [INFO]  server-12730079-5a51-ddcf-1045-1c842b77b991: copied to local snapshot: bytes=71
2026-04-23T00:15:52.416Z [ERROR] server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: failed to get log: index=2 error="log not found"
2026-04-23T00:15:52.416Z [INFO]  server-12730079-5a51-ddcf-1045-1c842b77b991: snapshot restore progress: id=2-113-1776903352415 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:52.416Z [INFO]  server-12730079-5a51-ddcf-1045-1c842b77b991: removing all old logs from log store
2026-04-23T00:15:52.416Z [INFO]  server-12730079-5a51-ddcf-1045-1c842b77b991: compacting logs: from=1 to=2
2026-04-23T00:15:52.416Z [INFO]  server-12730079-5a51-ddcf-1045-1c842b77b991: Installed remote snapshot
2026-04-23T00:15:52.416Z [INFO]  creating new snapshot: path=/tmp/raft4264012576/snapshots/2-113-1776903352416.tmp
2026-04-23T00:15:52.416Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: pipelining replication: peer="{Voter server-12730079-5a51-ddcf-1045-1c842b77b991 12730079-5a51-ddcf-1045-1c842b77b991}"
2026-04-23T00:15:52.416Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:15:52.416Z [INFO]  server-3f765b4c-e903-2495-c1f6-1b59fb113b41: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:52.416Z [INFO]  server-3f765b4c-e903-2495-c1f6-1b59fb113b41: copied to local snapshot: bytes=71
2026-04-23T00:15:52.416Z [INFO]  server-3f765b4c-e903-2495-c1f6-1b59fb113b41: snapshot restore progress: id=2-113-1776903352416 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:52.416Z [INFO]  server-3f765b4c-e903-2495-c1f6-1b59fb113b41: removing all old logs from log store
2026-04-23T00:15:52.416Z [INFO]  server-3f765b4c-e903-2495-c1f6-1b59fb113b41: compacting logs: from=1 to=2
2026-04-23T00:15:52.416Z [INFO]  server-3f765b4c-e903-2495-c1f6-1b59fb113b41: Installed remote snapshot
2026-04-23T00:15:52.416Z [INFO]  server-70c7ede9-6eb8-3233-88ec-ebdca78baf35: pipelining replication: peer="{Voter server-3f765b4c-e903-2495-c1f6-1b59fb113b41 3f765b4c-e903-2495-c1f6-1b59fb113b41}"
2026-04-23T00:15:53.422Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:15:52.416483683 +0000 UTC m=+69.456085051 (2 nodes), 931ns from start of poll, 1.520901723s from cluster start. Timeout at 2026-04-23 00:15:53.421498824 +0000 UTC m=+70.461100178, 1.005015127s after stability
2026-04-23T00:15:53.434Z [INFO]  server-e7df154c-0bd0-9973-bf58-381479170cb2: aborting pipeline replication: peer="{Voter server-8c74f8df-999d-11b5-7c6d-5b379fd67607 8c74f8df-999d-11b5-7c6d-5b379fd67607}"
2026-04-23T00:15:53.434Z [INFO]  server-e7df154c-0bd0-9973-bf58-381479170cb2: aborting pipeline replication: peer="{Voter server-1a621df2-4089-f41b-123c-2db018c1cd96 1a621df2-4089-f41b-123c-2db018c1cd96}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false
2026-04-23T00:15:53.435Z [INFO]  server-0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8 Address:0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8} {Suffrage:Voter ID:server-668e6088-ec5c-7751-2d9b-9bc2f46d3527 Address:668e6088-ec5c-7751-2d9b-9bc2f46d3527} {Suffrage:Voter ID:server-202f05b2-012d-2488-53e6-c673450095b4 Address:202f05b2-012d-2488-53e6-c673450095b4}]"
2026-04-23T00:15:53.435Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8 Address:0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8} {Suffrage:Voter ID:server-668e6088-ec5c-7751-2d9b-9bc2f46d3527 Address:668e6088-ec5c-7751-2d9b-9bc2f46d3527} {Suffrage:Voter ID:server-202f05b2-012d-2488-53e6-c673450095b4 Address:202f05b2-012d-2488-53e6-c673450095b4}]"
2026-04-23T00:15:53.435Z [INFO]  server-0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8: entering follower state: follower="Node at 0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8 [Follower]" leader-address= leader-id=
2026-04-23T00:15:53.435Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: entering follower state: follower="Node at 668e6088-ec5c-7751-2d9b-9bc2f46d3527 [Follower]" leader-address= leader-id=
2026-04-23T00:15:53.435Z [INFO]  server-202f05b2-012d-2488-53e6-c673450095b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8 Address:0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8} {Suffrage:Voter ID:server-668e6088-ec5c-7751-2d9b-9bc2f46d3527 Address:668e6088-ec5c-7751-2d9b-9bc2f46d3527} {Suffrage:Voter ID:server-202f05b2-012d-2488-53e6-c673450095b4 Address:202f05b2-012d-2488-53e6-c673450095b4}]"
2026-04-23T00:15:53.435Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:53.435Z [INFO]  server-202f05b2-012d-2488-53e6-c673450095b4: entering follower state: follower="Node at 202f05b2-012d-2488-53e6-c673450095b4 [Follower]" leader-address= leader-id=
2026-04-23T00:15:54.300Z [WARN]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:54.300Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: entering candidate state: node="Node at 668e6088-ec5c-7751-2d9b-9bc2f46d3527 [Candidate]" term=2
2026-04-23T00:15:54.300Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: election won: term=2 tally=2
2026-04-23T00:15:54.300Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: entering leader state: leader="Node at 668e6088-ec5c-7751-2d9b-9bc2f46d3527 [Leader]"
2026-04-23T00:15:54.300Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: added peer, starting replication: peer=server-0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8
2026-04-23T00:15:54.300Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: added peer, starting replication: peer=server-202f05b2-012d-2488-53e6-c673450095b4
2026-04-23T00:15:54.300Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: pipelining replication: peer="{Voter server-0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8 0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8}"
2026-04-23T00:15:54.300Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: pipelining replication: peer="{Voter server-202f05b2-012d-2488-53e6-c673450095b4 202f05b2-012d-2488-53e6-c673450095b4}"
2026-04-23T00:15:55.305Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:54.300843489 +0000 UTC m=+71.340444852 (1 nodes), 865.056794ms from start of poll, 865.413019ms from cluster start. Timeout at 2026-04-23 00:15:55.305844137 +0000 UTC m=+72.345445489, 1.005000637s after stability
2026-04-23T00:15:55.306Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: starting snapshot up to: index=12
2026-04-23T00:15:55.306Z [INFO]  creating new snapshot: path=/tmp/raft3083247712/snapshots/2-12-1776903355306.tmp
2026-04-23T00:15:55.306Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: snapshot complete up to: index=12
2026-04-23T00:15:55.307Z [INFO]  creating new snapshot: path=/tmp/raft3083247712/snapshots/2-1013-1776903355307.tmp
2026-04-23T00:15:55.307Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: copied to local snapshot: bytes=71
2026-04-23T00:15:55.307Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: snapshot restore progress: id=2-1013-1776903355307 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:55.307Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: restored user snapshot: index=1013
2026-04-23T00:15:55.307Z [ERROR] server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: failed to get log: index=23 error="log not found"
2026-04-23T00:15:55.307Z [ERROR] server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: failed to get log: index=23 error="log not found"
2026-04-23T00:15:55.308Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: aborting pipeline replication: peer="{Voter server-0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8 0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8}"
2026-04-23T00:15:55.307Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: aborting pipeline replication: peer="{Voter server-202f05b2-012d-2488-53e6-c673450095b4 202f05b2-012d-2488-53e6-c673450095b4}"
2026-04-23T00:15:55.316Z [ERROR] server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: failed to get log: index=23 error="log not found"
2026-04-23T00:15:55.316Z [INFO]  creating new snapshot: path=/tmp/raft3826987854/snapshots/2-1013-1776903355316.tmp
2026-04-23T00:15:55.317Z [INFO]  server-0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:55.317Z [INFO]  server-0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8: copied to local snapshot: bytes=71
2026-04-23T00:15:55.317Z [INFO]  server-0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8: snapshot restore progress: id=2-1013-1776903355316 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:55.317Z [INFO]  server-0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8: Installed remote snapshot
2026-04-23T00:15:55.317Z [WARN]  server-0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8: skipping application of old log: index=22
2026-04-23T00:15:55.317Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: pipelining replication: peer="{Voter server-0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8 0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8}"
2026-04-23T00:15:55.317Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:15:55.318Z [ERROR] server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: failed to get log: index=23 error="log not found"
2026-04-23T00:15:55.318Z [INFO]  creating new snapshot: path=/tmp/raft2237718644/snapshots/2-1013-1776903355318.tmp
2026-04-23T00:15:55.319Z [INFO]  server-202f05b2-012d-2488-53e6-c673450095b4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:55.319Z [INFO]  server-202f05b2-012d-2488-53e6-c673450095b4: copied to local snapshot: bytes=71
2026-04-23T00:15:55.319Z [INFO]  server-202f05b2-012d-2488-53e6-c673450095b4: snapshot restore progress: id=2-1013-1776903355318 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:55.319Z [INFO]  server-202f05b2-012d-2488-53e6-c673450095b4: Installed remote snapshot
2026-04-23T00:15:55.319Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: pipelining replication: peer="{Voter server-202f05b2-012d-2488-53e6-c673450095b4 202f05b2-012d-2488-53e6-c673450095b4}"
2026-04-23T00:15:56.323Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:15:55.317632051 +0000 UTC m=+72.357233407 (2 nodes), 1.089µs from start of poll, 1.882201574s from cluster start. Timeout at 2026-04-23 00:15:56.322635592 +0000 UTC m=+73.362236944, 1.005003537s after stability
2026-04-23T00:15:56.334Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: aborting pipeline replication: peer="{Voter server-0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8 0ce2fd55-0b5b-8ce2-4bf1-e64af3ef2bb8}"
2026-04-23T00:15:56.334Z [INFO]  server-668e6088-ec5c-7751-2d9b-9bc2f46d3527: aborting pipeline replication: peer="{Voter server-202f05b2-012d-2488-53e6-c673450095b4 202f05b2-012d-2488-53e6-c673450095b4}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false
2026-04-23T00:15:56.336Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a Address:f618c9eb-66b8-f11d-bbb9-e8cd04c2827a} {Suffrage:Voter ID:server-f130210b-d846-a339-da61-281b9c4bc038 Address:f130210b-d846-a339-da61-281b9c4bc038} {Suffrage:Voter ID:server-113671d7-9dff-c706-06f6-4cc61057c728 Address:113671d7-9dff-c706-06f6-4cc61057c728}]"
2026-04-23T00:15:56.336Z [INFO]  server-f130210b-d846-a339-da61-281b9c4bc038: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a Address:f618c9eb-66b8-f11d-bbb9-e8cd04c2827a} {Suffrage:Voter ID:server-f130210b-d846-a339-da61-281b9c4bc038 Address:f130210b-d846-a339-da61-281b9c4bc038} {Suffrage:Voter ID:server-113671d7-9dff-c706-06f6-4cc61057c728 Address:113671d7-9dff-c706-06f6-4cc61057c728}]"
2026-04-23T00:15:56.336Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: entering follower state: follower="Node at f618c9eb-66b8-f11d-bbb9-e8cd04c2827a [Follower]" leader-address= leader-id=
2026-04-23T00:15:56.336Z [INFO]  server-f130210b-d846-a339-da61-281b9c4bc038: entering follower state: follower="Node at f130210b-d846-a339-da61-281b9c4bc038 [Follower]" leader-address= leader-id=
2026-04-23T00:15:56.336Z [INFO]  server-113671d7-9dff-c706-06f6-4cc61057c728: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a Address:f618c9eb-66b8-f11d-bbb9-e8cd04c2827a} {Suffrage:Voter ID:server-f130210b-d846-a339-da61-281b9c4bc038 Address:f130210b-d846-a339-da61-281b9c4bc038} {Suffrage:Voter ID:server-113671d7-9dff-c706-06f6-4cc61057c728 Address:113671d7-9dff-c706-06f6-4cc61057c728}]"
2026-04-23T00:15:56.336Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:56.336Z [INFO]  server-113671d7-9dff-c706-06f6-4cc61057c728: entering follower state: follower="Node at 113671d7-9dff-c706-06f6-4cc61057c728 [Follower]" leader-address= leader-id=
2026-04-23T00:15:56.919Z [WARN]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:56.919Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: entering candidate state: node="Node at f618c9eb-66b8-f11d-bbb9-e8cd04c2827a [Candidate]" term=2
2026-04-23T00:15:56.919Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: election won: term=2 tally=2
2026-04-23T00:15:56.919Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: entering leader state: leader="Node at f618c9eb-66b8-f11d-bbb9-e8cd04c2827a [Leader]"
2026-04-23T00:15:56.919Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: added peer, starting replication: peer=server-f130210b-d846-a339-da61-281b9c4bc038
2026-04-23T00:15:56.919Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: added peer, starting replication: peer=server-113671d7-9dff-c706-06f6-4cc61057c728
2026-04-23T00:15:56.920Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: pipelining replication: peer="{Voter server-f130210b-d846-a339-da61-281b9c4bc038 f130210b-d846-a339-da61-281b9c4bc038}"
2026-04-23T00:15:56.920Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: pipelining replication: peer="{Voter server-113671d7-9dff-c706-06f6-4cc61057c728 113671d7-9dff-c706-06f6-4cc61057c728}"
2026-04-23T00:15:57.925Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:56.919977699 +0000 UTC m=+73.959579068 (1 nodes), 583.104481ms from start of poll, 583.473453ms from cluster start. Timeout at 2026-04-23 00:15:57.924978551 +0000 UTC m=+74.964579908, 1.00500084s after stability
2026-04-23T00:15:57.926Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: starting snapshot up to: index=12
2026-04-23T00:15:57.926Z [INFO]  creating new snapshot: path=/tmp/raft3735202450/snapshots/2-12-1776903357926.tmp
2026-04-23T00:15:57.926Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: snapshot complete up to: index=12
2026-04-23T00:15:57.927Z [INFO]  creating new snapshot: path=/tmp/raft3735202450/snapshots/2-1013-1776903357927.tmp
2026-04-23T00:15:57.927Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: copied to local snapshot: bytes=71
2026-04-23T00:15:57.927Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: snapshot restore progress: id=2-1013-1776903357927 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:57.928Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: removing all old logs from log store
2026-04-23T00:15:57.928Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: compacting logs: from=1 to=22
2026-04-23T00:15:57.928Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: restored user snapshot: index=1013
2026-04-23T00:15:57.928Z [ERROR] server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: failed to get log: index=22 error="log not found"
2026-04-23T00:15:57.928Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: aborting pipeline replication: peer="{Voter server-113671d7-9dff-c706-06f6-4cc61057c728 113671d7-9dff-c706-06f6-4cc61057c728}"
2026-04-23T00:15:57.928Z [ERROR] server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: failed to get log: index=22 error="log not found"
2026-04-23T00:15:57.928Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: aborting pipeline replication: peer="{Voter server-f130210b-d846-a339-da61-281b9c4bc038 f130210b-d846-a339-da61-281b9c4bc038}"
2026-04-23T00:15:57.936Z [ERROR] server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: failed to get log: index=22 error="log not found"
2026-04-23T00:15:57.936Z [ERROR] server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: failed to get log: index=22 error="log not found"
2026-04-23T00:15:57.936Z [INFO]  creating new snapshot: path=/tmp/raft1337690870/snapshots/2-1013-1776903357936.tmp
2026-04-23T00:15:57.936Z [INFO]  creating new snapshot: path=/tmp/raft2377640302/snapshots/2-1013-1776903357936.tmp
2026-04-23T00:15:57.936Z [INFO]  server-f130210b-d846-a339-da61-281b9c4bc038: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:57.936Z [INFO]  server-113671d7-9dff-c706-06f6-4cc61057c728: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:57.937Z [INFO]  server-f130210b-d846-a339-da61-281b9c4bc038: copied to local snapshot: bytes=71
2026-04-23T00:15:57.937Z [INFO]  server-f130210b-d846-a339-da61-281b9c4bc038: snapshot restore progress: id=2-1013-1776903357936 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:57.937Z [INFO]  server-f130210b-d846-a339-da61-281b9c4bc038: removing all old logs from log store
2026-04-23T00:15:57.937Z [INFO]  server-f130210b-d846-a339-da61-281b9c4bc038: compacting logs: from=1 to=22
2026-04-23T00:15:57.937Z [INFO]  server-f130210b-d846-a339-da61-281b9c4bc038: Installed remote snapshot
2026-04-23T00:15:57.937Z [INFO]  server-113671d7-9dff-c706-06f6-4cc61057c728: copied to local snapshot: bytes=71
2026-04-23T00:15:57.937Z [WARN]  server-f130210b-d846-a339-da61-281b9c4bc038: skipping application of old log: index=22
2026-04-23T00:15:57.937Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: pipelining replication: peer="{Voter server-f130210b-d846-a339-da61-281b9c4bc038 f130210b-d846-a339-da61-281b9c4bc038}"
2026-04-23T00:15:57.937Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:15:57.937Z [INFO]  server-113671d7-9dff-c706-06f6-4cc61057c728: snapshot restore progress: id=2-1013-1776903357936 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:15:57.937Z [INFO]  server-113671d7-9dff-c706-06f6-4cc61057c728: removing all old logs from log store
2026-04-23T00:15:57.937Z [INFO]  server-113671d7-9dff-c706-06f6-4cc61057c728: compacting logs: from=1 to=22
2026-04-23T00:15:57.937Z [INFO]  server-113671d7-9dff-c706-06f6-4cc61057c728: Installed remote snapshot
2026-04-23T00:15:57.937Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: pipelining replication: peer="{Voter server-113671d7-9dff-c706-06f6-4cc61057c728 113671d7-9dff-c706-06f6-4cc61057c728}"
2026-04-23T00:15:58.943Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:15:57.937315602 +0000 UTC m=+74.976916962 (2 nodes), 779ns from start of poll, 1.600811347s from cluster start. Timeout at 2026-04-23 00:15:58.942326519 +0000 UTC m=+75.981927912, 1.00501095s after stability
2026-04-23T00:15:58.955Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: aborting pipeline replication: peer="{Voter server-113671d7-9dff-c706-06f6-4cc61057c728 113671d7-9dff-c706-06f6-4cc61057c728}"
2026-04-23T00:15:58.955Z [INFO]  server-f618c9eb-66b8-f11d-bbb9-e8cd04c2827a: aborting pipeline replication: peer="{Voter server-f130210b-d846-a339-da61-281b9c4bc038 f130210b-d846-a339-da61-281b9c4bc038}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true
2026-04-23T00:15:58.957Z [INFO]  server-931db8f3-194d-d45d-f4a1-6caf9b9a9ff1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-931db8f3-194d-d45d-f4a1-6caf9b9a9ff1 Address:931db8f3-194d-d45d-f4a1-6caf9b9a9ff1} {Suffrage:Voter ID:server-8fc8e6d9-8183-41bf-ce85-643b0ceeb80e Address:8fc8e6d9-8183-41bf-ce85-643b0ceeb80e} {Suffrage:Voter ID:server-95bf102c-9ec8-4eb6-7d2e-f054ed221f7a Address:95bf102c-9ec8-4eb6-7d2e-f054ed221f7a}]"
2026-04-23T00:15:58.957Z [INFO]  server-931db8f3-194d-d45d-f4a1-6caf9b9a9ff1: entering follower state: follower="Node at 931db8f3-194d-d45d-f4a1-6caf9b9a9ff1 [Follower]" leader-address= leader-id=
2026-04-23T00:15:58.957Z [INFO]  server-8fc8e6d9-8183-41bf-ce85-643b0ceeb80e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-931db8f3-194d-d45d-f4a1-6caf9b9a9ff1 Address:931db8f3-194d-d45d-f4a1-6caf9b9a9ff1} {Suffrage:Voter ID:server-8fc8e6d9-8183-41bf-ce85-643b0ceeb80e Address:8fc8e6d9-8183-41bf-ce85-643b0ceeb80e} {Suffrage:Voter ID:server-95bf102c-9ec8-4eb6-7d2e-f054ed221f7a Address:95bf102c-9ec8-4eb6-7d2e-f054ed221f7a}]"
2026-04-23T00:15:58.957Z [INFO]  server-8fc8e6d9-8183-41bf-ce85-643b0ceeb80e: entering follower state: follower="Node at 8fc8e6d9-8183-41bf-ce85-643b0ceeb80e [Follower]" leader-address= leader-id=
2026-04-23T00:15:58.957Z [INFO]  server-95bf102c-9ec8-4eb6-7d2e-f054ed221f7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-931db8f3-194d-d45d-f4a1-6caf9b9a9ff1 Address:931db8f3-194d-d45d-f4a1-6caf9b9a9ff1} {Suffrage:Voter ID:server-8fc8e6d9-8183-41bf-ce85-643b0ceeb80e Address:8fc8e6d9-8183-41bf-ce85-643b0ceeb80e} {Suffrage:Voter ID:server-95bf102c-9ec8-4eb6-7d2e-f054ed221f7a Address:95bf102c-9ec8-4eb6-7d2e-f054ed221f7a}]"
2026-04-23T00:15:58.957Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:15:58.957Z [INFO]  server-95bf102c-9ec8-4eb6-7d2e-f054ed221f7a: entering follower state: follower="Node at 95bf102c-9ec8-4eb6-7d2e-f054ed221f7a [Follower]" leader-address= leader-id=
2026-04-23T00:15:59.477Z [WARN]  server-95bf102c-9ec8-4eb6-7d2e-f054ed221f7a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:15:59.477Z [INFO]  server-95bf102c-9ec8-4eb6-7d2e-f054ed221f7a: entering candidate state: node="Node at 95bf102c-9ec8-4eb6-7d2e-f054ed221f7a [Candidate]" term=2
2026-04-23T00:15:59.477Z [INFO]  server-95bf102c-9ec8-4eb6-7d2e-f054ed221f7a: election won: term=2 tally=2
2026-04-23T00:15:59.477Z [INFO]  server-95bf102c-9ec8-4eb6-7d2e-f054ed221f7a: entering leader state: leader="Node at 95bf102c-9ec8-4eb6-7d2e-f054ed221f7a [Leader]"
2026-04-23T00:15:59.477Z [INFO]  server-95bf102c-9ec8-4eb6-7d2e-f054ed221f7a: added peer, starting replication: peer=server-931db8f3-194d-d45d-f4a1-6caf9b9a9ff1
2026-04-23T00:15:59.477Z [INFO]  server-95bf102c-9ec8-4eb6-7d2e-f054ed221f7a: added peer, starting replication: peer=server-8fc8e6d9-8183-41bf-ce85-643b0ceeb80e
2026-04-23T00:15:59.478Z [INFO]  server-95bf102c-9ec8-4eb6-7d2e-f054ed221f7a: pipelining replication: peer="{Voter server-8fc8e6d9-8183-41bf-ce85-643b0ceeb80e 8fc8e6d9-8183-41bf-ce85-643b0ceeb80e}"
2026-04-23T00:15:59.478Z [INFO]  server-95bf102c-9ec8-4eb6-7d2e-f054ed221f7a: pipelining replication: peer="{Voter server-931db8f3-194d-d45d-f4a1-6caf9b9a9ff1 931db8f3-194d-d45d-f4a1-6caf9b9a9ff1}"
2026-04-23T00:16:00.483Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:15:59.477975281 +0000 UTC m=+76.517576668 (1 nodes), 520.4424ms from start of poll, 520.895164ms from cluster start. Timeout at 2026-04-23 00:16:00.482980681 +0000 UTC m=+77.522582038, 1.00500537s after stability
2026-04-23T00:16:00.484Z [INFO]  server-95bf102c-9ec8-4eb6-7d2e-f054ed221f7a: starting snapshot up to: index=12
2026-04-23T00:16:00.484Z [INFO]  creating new snapshot: path=/tmp/raft1512358998/snapshots/2-12-1776903360484.tmp
2026-04-23T00:16:00.489Z [INFO]  server-95bf102c-9ec8-4eb6-7d2e-f054ed221f7a: snapshot complete up to: index=12
2026-04-23T00:16:00.490Z [INFO]  server-d0c0ddbc-9f8f-e299-b6dc-7947130c65e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0c0ddbc-9f8f-e299-b6dc-7947130c65e7 Address:d0c0ddbc-9f8f-e299-b6dc-7947130c65e7} {Suffrage:Voter ID:server-64765d13-9e01-24a4-4774-45e631176f41 Address:64765d13-9e01-24a4-4774-45e631176f41} {Suffrage:Voter ID:server-60d30701-37bd-9d2c-203d-4126f23ea08d Address:60d30701-37bd-9d2c-203d-4126f23ea08d}]"
2026-04-23T00:16:00.490Z [INFO]  server-64765d13-9e01-24a4-4774-45e631176f41: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0c0ddbc-9f8f-e299-b6dc-7947130c65e7 Address:d0c0ddbc-9f8f-e299-b6dc-7947130c65e7} {Suffrage:Voter ID:server-64765d13-9e01-24a4-4774-45e631176f41 Address:64765d13-9e01-24a4-4774-45e631176f41} {Suffrage:Voter ID:server-60d30701-37bd-9d2c-203d-4126f23ea08d Address:60d30701-37bd-9d2c-203d-4126f23ea08d}]"
2026-04-23T00:16:00.490Z [INFO]  server-d0c0ddbc-9f8f-e299-b6dc-7947130c65e7: entering follower state: follower="Node at d0c0ddbc-9f8f-e299-b6dc-7947130c65e7 [Follower]" leader-address= leader-id=
2026-04-23T00:16:00.490Z [INFO]  server-64765d13-9e01-24a4-4774-45e631176f41: entering follower state: follower="Node at 64765d13-9e01-24a4-4774-45e631176f41 [Follower]" leader-address= leader-id=
2026-04-23T00:16:00.490Z [INFO]  server-60d30701-37bd-9d2c-203d-4126f23ea08d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0c0ddbc-9f8f-e299-b6dc-7947130c65e7 Address:d0c0ddbc-9f8f-e299-b6dc-7947130c65e7} {Suffrage:Voter ID:server-64765d13-9e01-24a4-4774-45e631176f41 Address:64765d13-9e01-24a4-4774-45e631176f41} {Suffrage:Voter ID:server-60d30701-37bd-9d2c-203d-4126f23ea08d Address:60d30701-37bd-9d2c-203d-4126f23ea08d}]"
2026-04-23T00:16:00.490Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:00.490Z [INFO]  server-60d30701-37bd-9d2c-203d-4126f23ea08d: entering follower state: follower="Node at 60d30701-37bd-9d2c-203d-4126f23ea08d [Follower]" leader-address= leader-id=
2026-04-23T00:16:01.067Z [WARN]  server-60d30701-37bd-9d2c-203d-4126f23ea08d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:01.067Z [INFO]  server-60d30701-37bd-9d2c-203d-4126f23ea08d: entering candidate state: node="Node at 60d30701-37bd-9d2c-203d-4126f23ea08d [Candidate]" term=2
2026-04-23T00:16:01.067Z [INFO]  server-60d30701-37bd-9d2c-203d-4126f23ea08d: election won: term=2 tally=2
2026-04-23T00:16:01.067Z [INFO]  server-60d30701-37bd-9d2c-203d-4126f23ea08d: entering leader state: leader="Node at 60d30701-37bd-9d2c-203d-4126f23ea08d [Leader]"
2026-04-23T00:16:01.067Z [INFO]  server-60d30701-37bd-9d2c-203d-4126f23ea08d: added peer, starting replication: peer=server-d0c0ddbc-9f8f-e299-b6dc-7947130c65e7
2026-04-23T00:16:01.067Z [INFO]  server-60d30701-37bd-9d2c-203d-4126f23ea08d: added peer, starting replication: peer=server-64765d13-9e01-24a4-4774-45e631176f41
2026-04-23T00:16:01.067Z [INFO]  server-60d30701-37bd-9d2c-203d-4126f23ea08d: pipelining replication: peer="{Voter server-64765d13-9e01-24a4-4774-45e631176f41 64765d13-9e01-24a4-4774-45e631176f41}"
2026-04-23T00:16:01.067Z [INFO]  server-60d30701-37bd-9d2c-203d-4126f23ea08d: pipelining replication: peer="{Voter server-d0c0ddbc-9f8f-e299-b6dc-7947130c65e7 d0c0ddbc-9f8f-e299-b6dc-7947130c65e7}"
2026-04-23T00:16:02.072Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:01.067823123 +0000 UTC m=+78.107424486 (1 nodes), 577.504863ms from start of poll, 577.820956ms from cluster start. Timeout at 2026-04-23 00:16:02.072824415 +0000 UTC m=+79.112425767, 1.005001281s after stability
2026-04-23T00:16:02.073Z [INFO]  creating new snapshot: path=/tmp/raft3900886932/snapshots/2-1013-1776903362073.tmp
2026-04-23T00:16:02.219Z [INFO]  server-60d30701-37bd-9d2c-203d-4126f23ea08d: copied to local snapshot: bytes=71
2026-04-23T00:16:02.219Z [INFO]  server-60d30701-37bd-9d2c-203d-4126f23ea08d: snapshot restore progress: id=2-1013-1776903362073 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:02.219Z [INFO]  server-60d30701-37bd-9d2c-203d-4126f23ea08d: restored user snapshot: index=1013
2026-04-23T00:16:02.219Z [ERROR] server-60d30701-37bd-9d2c-203d-4126f23ea08d: failed to get log: index=3 error="log not found"
2026-04-23T00:16:02.219Z [INFO]  server-60d30701-37bd-9d2c-203d-4126f23ea08d: aborting pipeline replication: peer="{Voter server-64765d13-9e01-24a4-4774-45e631176f41 64765d13-9e01-24a4-4774-45e631176f41}"
2026-04-23T00:16:02.219Z [ERROR] server-60d30701-37bd-9d2c-203d-4126f23ea08d: failed to get log: index=3 error="log not found"
2026-04-23T00:16:02.219Z [INFO]  server-60d30701-37bd-9d2c-203d-4126f23ea08d: aborting pipeline replication: peer="{Voter server-d0c0ddbc-9f8f-e299-b6dc-7947130c65e7 d0c0ddbc-9f8f-e299-b6dc-7947130c65e7}"
2026-04-23T00:16:02.229Z [ERROR] server-60d30701-37bd-9d2c-203d-4126f23ea08d: failed to get log: index=3 error="log not found"
2026-04-23T00:16:02.229Z [ERROR] server-60d30701-37bd-9d2c-203d-4126f23ea08d: failed to get log: index=3 error="log not found"
2026-04-23T00:16:02.230Z [INFO]  creating new snapshot: path=/tmp/raft1653080315/snapshots/2-1013-1776903362230.tmp
2026-04-23T00:16:02.230Z [INFO]  creating new snapshot: path=/tmp/raft2143873275/snapshots/2-1013-1776903362230.tmp
2026-04-23T00:16:02.230Z [INFO]  server-d0c0ddbc-9f8f-e299-b6dc-7947130c65e7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:02.230Z [INFO]  server-64765d13-9e01-24a4-4774-45e631176f41: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:02.337Z [INFO]  server-64765d13-9e01-24a4-4774-45e631176f41: copied to local snapshot: bytes=71
2026-04-23T00:16:02.337Z [INFO]  server-d0c0ddbc-9f8f-e299-b6dc-7947130c65e7: copied to local snapshot: bytes=71
2026-04-23T00:16:02.337Z [INFO]  server-d0c0ddbc-9f8f-e299-b6dc-7947130c65e7: snapshot restore progress: id=2-1013-1776903362230 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:02.337Z [INFO]  server-d0c0ddbc-9f8f-e299-b6dc-7947130c65e7: Installed remote snapshot
2026-04-23T00:16:02.337Z [INFO]  server-64765d13-9e01-24a4-4774-45e631176f41: snapshot restore progress: id=2-1013-1776903362230 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:02.337Z [INFO]  server-64765d13-9e01-24a4-4774-45e631176f41: Installed remote snapshot
2026-04-23T00:16:02.337Z [INFO]  server-60d30701-37bd-9d2c-203d-4126f23ea08d: pipelining replication: peer="{Voter server-64765d13-9e01-24a4-4774-45e631176f41 64765d13-9e01-24a4-4774-45e631176f41}"
2026-04-23T00:16:02.337Z [INFO]  server-60d30701-37bd-9d2c-203d-4126f23ea08d: pipelining replication: peer="{Voter server-d0c0ddbc-9f8f-e299-b6dc-7947130c65e7 d0c0ddbc-9f8f-e299-b6dc-7947130c65e7}"
2026-04-23T00:16:02.337Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:03.343Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:02.337910702 +0000 UTC m=+79.377512141 (2 nodes), 1.108µs from start of poll, 1.847908611s from cluster start. Timeout at 2026-04-23 00:16:03.342926999 +0000 UTC m=+80.382528362, 1.005016221s after stability
2026-04-23T00:16:03.354Z [INFO]  server-95bf102c-9ec8-4eb6-7d2e-f054ed221f7a: aborting pipeline replication: peer="{Voter server-931db8f3-194d-d45d-f4a1-6caf9b9a9ff1 931db8f3-194d-d45d-f4a1-6caf9b9a9ff1}"
2026-04-23T00:16:03.354Z [INFO]  server-95bf102c-9ec8-4eb6-7d2e-f054ed221f7a: aborting pipeline replication: peer="{Voter server-8fc8e6d9-8183-41bf-ce85-643b0ceeb80e 8fc8e6d9-8183-41bf-ce85-643b0ceeb80e}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true
2026-04-23T00:16:03.356Z [INFO]  server-a24f3239-df5a-0218-4a79-0a1bf472cde3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a24f3239-df5a-0218-4a79-0a1bf472cde3 Address:a24f3239-df5a-0218-4a79-0a1bf472cde3} {Suffrage:Voter ID:server-da6e5e02-bf7a-e644-ba4a-2dfe1976fb29 Address:da6e5e02-bf7a-e644-ba4a-2dfe1976fb29} {Suffrage:Voter ID:server-ae37dee0-3cd3-1e3c-afe7-a07aa0e3fe22 Address:ae37dee0-3cd3-1e3c-afe7-a07aa0e3fe22}]"
2026-04-23T00:16:03.356Z [INFO]  server-a24f3239-df5a-0218-4a79-0a1bf472cde3: entering follower state: follower="Node at a24f3239-df5a-0218-4a79-0a1bf472cde3 [Follower]" leader-address= leader-id=
2026-04-23T00:16:03.356Z [INFO]  server-da6e5e02-bf7a-e644-ba4a-2dfe1976fb29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a24f3239-df5a-0218-4a79-0a1bf472cde3 Address:a24f3239-df5a-0218-4a79-0a1bf472cde3} {Suffrage:Voter ID:server-da6e5e02-bf7a-e644-ba4a-2dfe1976fb29 Address:da6e5e02-bf7a-e644-ba4a-2dfe1976fb29} {Suffrage:Voter ID:server-ae37dee0-3cd3-1e3c-afe7-a07aa0e3fe22 Address:ae37dee0-3cd3-1e3c-afe7-a07aa0e3fe22}]"
2026-04-23T00:16:03.356Z [INFO]  server-da6e5e02-bf7a-e644-ba4a-2dfe1976fb29: entering follower state: follower="Node at da6e5e02-bf7a-e644-ba4a-2dfe1976fb29 [Follower]" leader-address= leader-id=
2026-04-23T00:16:03.356Z [INFO]  server-ae37dee0-3cd3-1e3c-afe7-a07aa0e3fe22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a24f3239-df5a-0218-4a79-0a1bf472cde3 Address:a24f3239-df5a-0218-4a79-0a1bf472cde3} {Suffrage:Voter ID:server-da6e5e02-bf7a-e644-ba4a-2dfe1976fb29 Address:da6e5e02-bf7a-e644-ba4a-2dfe1976fb29} {Suffrage:Voter ID:server-ae37dee0-3cd3-1e3c-afe7-a07aa0e3fe22 Address:ae37dee0-3cd3-1e3c-afe7-a07aa0e3fe22}]"
2026-04-23T00:16:03.356Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:03.356Z [INFO]  server-ae37dee0-3cd3-1e3c-afe7-a07aa0e3fe22: entering follower state: follower="Node at ae37dee0-3cd3-1e3c-afe7-a07aa0e3fe22 [Follower]" leader-address= leader-id=
2026-04-23T00:16:03.939Z [WARN]  server-da6e5e02-bf7a-e644-ba4a-2dfe1976fb29: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:03.939Z [INFO]  server-da6e5e02-bf7a-e644-ba4a-2dfe1976fb29: entering candidate state: node="Node at da6e5e02-bf7a-e644-ba4a-2dfe1976fb29 [Candidate]" term=2
2026-04-23T00:16:03.940Z [INFO]  server-da6e5e02-bf7a-e644-ba4a-2dfe1976fb29: election won: term=2 tally=2
2026-04-23T00:16:03.940Z [INFO]  server-da6e5e02-bf7a-e644-ba4a-2dfe1976fb29: entering leader state: leader="Node at da6e5e02-bf7a-e644-ba4a-2dfe1976fb29 [Leader]"
2026-04-23T00:16:03.940Z [INFO]  server-da6e5e02-bf7a-e644-ba4a-2dfe1976fb29: added peer, starting replication: peer=server-a24f3239-df5a-0218-4a79-0a1bf472cde3
2026-04-23T00:16:03.940Z [INFO]  server-da6e5e02-bf7a-e644-ba4a-2dfe1976fb29: added peer, starting replication: peer=server-ae37dee0-3cd3-1e3c-afe7-a07aa0e3fe22
2026-04-23T00:16:03.940Z [INFO]  server-da6e5e02-bf7a-e644-ba4a-2dfe1976fb29: pipelining replication: peer="{Voter server-ae37dee0-3cd3-1e3c-afe7-a07aa0e3fe22 ae37dee0-3cd3-1e3c-afe7-a07aa0e3fe22}"
2026-04-23T00:16:03.940Z [INFO]  server-da6e5e02-bf7a-e644-ba4a-2dfe1976fb29: pipelining replication: peer="{Voter server-a24f3239-df5a-0218-4a79-0a1bf472cde3 a24f3239-df5a-0218-4a79-0a1bf472cde3}"
2026-04-23T00:16:04.945Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:03.94021128 +0000 UTC m=+80.979812639 (1 nodes), 583.934568ms from start of poll, 584.284731ms from cluster start. Timeout at 2026-04-23 00:16:04.9452179 +0000 UTC m=+81.984819256, 1.005006617s after stability
2026-04-23T00:16:04.946Z [INFO]  server-da6e5e02-bf7a-e644-ba4a-2dfe1976fb29: starting snapshot up to: index=12
2026-04-23T00:16:04.946Z [INFO]  creating new snapshot: path=/tmp/raft3658569726/snapshots/2-12-1776903364946.tmp
2026-04-23T00:16:05.185Z [INFO]  server-da6e5e02-bf7a-e644-ba4a-2dfe1976fb29: snapshot complete up to: index=12
2026-04-23T00:16:05.186Z [INFO]  server-8c9dbea7-85a8-e94a-2d5a-52c804090a3d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c9dbea7-85a8-e94a-2d5a-52c804090a3d Address:8c9dbea7-85a8-e94a-2d5a-52c804090a3d} {Suffrage:Voter ID:server-5361306b-a151-e82d-32c4-f1c2df6fd2a3 Address:5361306b-a151-e82d-32c4-f1c2df6fd2a3} {Suffrage:Voter ID:server-dadefc40-6bdd-4f38-fe6f-a71a97129268 Address:dadefc40-6bdd-4f38-fe6f-a71a97129268}]"
2026-04-23T00:16:05.186Z [INFO]  server-8c9dbea7-85a8-e94a-2d5a-52c804090a3d: entering follower state: follower="Node at 8c9dbea7-85a8-e94a-2d5a-52c804090a3d [Follower]" leader-address= leader-id=
2026-04-23T00:16:05.186Z [INFO]  server-5361306b-a151-e82d-32c4-f1c2df6fd2a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c9dbea7-85a8-e94a-2d5a-52c804090a3d Address:8c9dbea7-85a8-e94a-2d5a-52c804090a3d} {Suffrage:Voter ID:server-5361306b-a151-e82d-32c4-f1c2df6fd2a3 Address:5361306b-a151-e82d-32c4-f1c2df6fd2a3} {Suffrage:Voter ID:server-dadefc40-6bdd-4f38-fe6f-a71a97129268 Address:dadefc40-6bdd-4f38-fe6f-a71a97129268}]"
2026-04-23T00:16:05.186Z [INFO]  server-5361306b-a151-e82d-32c4-f1c2df6fd2a3: entering follower state: follower="Node at 5361306b-a151-e82d-32c4-f1c2df6fd2a3 [Follower]" leader-address= leader-id=
2026-04-23T00:16:05.186Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c9dbea7-85a8-e94a-2d5a-52c804090a3d Address:8c9dbea7-85a8-e94a-2d5a-52c804090a3d} {Suffrage:Voter ID:server-5361306b-a151-e82d-32c4-f1c2df6fd2a3 Address:5361306b-a151-e82d-32c4-f1c2df6fd2a3} {Suffrage:Voter ID:server-dadefc40-6bdd-4f38-fe6f-a71a97129268 Address:dadefc40-6bdd-4f38-fe6f-a71a97129268}]"
2026-04-23T00:16:05.186Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:05.186Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: entering follower state: follower="Node at dadefc40-6bdd-4f38-fe6f-a71a97129268 [Follower]" leader-address= leader-id=
2026-04-23T00:16:05.713Z [WARN]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:05.713Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: entering candidate state: node="Node at dadefc40-6bdd-4f38-fe6f-a71a97129268 [Candidate]" term=2
2026-04-23T00:16:05.713Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: election won: term=2 tally=2
2026-04-23T00:16:05.713Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: entering leader state: leader="Node at dadefc40-6bdd-4f38-fe6f-a71a97129268 [Leader]"
2026-04-23T00:16:05.713Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: added peer, starting replication: peer=server-8c9dbea7-85a8-e94a-2d5a-52c804090a3d
2026-04-23T00:16:05.713Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: added peer, starting replication: peer=server-5361306b-a151-e82d-32c4-f1c2df6fd2a3
2026-04-23T00:16:05.713Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: pipelining replication: peer="{Voter server-5361306b-a151-e82d-32c4-f1c2df6fd2a3 5361306b-a151-e82d-32c4-f1c2df6fd2a3}"
2026-04-23T00:16:05.713Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: pipelining replication: peer="{Voter server-8c9dbea7-85a8-e94a-2d5a-52c804090a3d 8c9dbea7-85a8-e94a-2d5a-52c804090a3d}"
2026-04-23T00:16:06.719Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:05.713608622 +0000 UTC m=+82.753209982 (1 nodes), 526.6745ms from start of poll, 527.033665ms from cluster start. Timeout at 2026-04-23 00:16:06.718609666 +0000 UTC m=+83.758211019, 1.005001037s after stability
2026-04-23T00:16:06.719Z [INFO]  creating new snapshot: path=/tmp/raft3574134343/snapshots/2-1013-1776903366719.tmp
2026-04-23T00:16:06.720Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: copied to local snapshot: bytes=71
2026-04-23T00:16:06.720Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: snapshot restore progress: id=2-1013-1776903366719 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:06.720Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: removing all old logs from log store
2026-04-23T00:16:06.720Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: compacting logs: from=1 to=2
2026-04-23T00:16:06.720Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: restored user snapshot: index=1013
2026-04-23T00:16:06.720Z [ERROR] server-dadefc40-6bdd-4f38-fe6f-a71a97129268: failed to get log: index=2 error="log not found"
2026-04-23T00:16:06.720Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: aborting pipeline replication: peer="{Voter server-5361306b-a151-e82d-32c4-f1c2df6fd2a3 5361306b-a151-e82d-32c4-f1c2df6fd2a3}"
2026-04-23T00:16:06.720Z [ERROR] server-dadefc40-6bdd-4f38-fe6f-a71a97129268: failed to get log: index=2 error="log not found"
2026-04-23T00:16:06.720Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: aborting pipeline replication: peer="{Voter server-8c9dbea7-85a8-e94a-2d5a-52c804090a3d 8c9dbea7-85a8-e94a-2d5a-52c804090a3d}"
2026-04-23T00:16:06.726Z [ERROR] server-dadefc40-6bdd-4f38-fe6f-a71a97129268: failed to get log: index=2 error="log not found"
2026-04-23T00:16:06.726Z [ERROR] server-dadefc40-6bdd-4f38-fe6f-a71a97129268: failed to get log: index=2 error="log not found"
2026-04-23T00:16:06.726Z [INFO]  creating new snapshot: path=/tmp/raft1406484173/snapshots/2-1013-1776903366726.tmp
2026-04-23T00:16:06.726Z [INFO]  creating new snapshot: path=/tmp/raft2100715692/snapshots/2-1013-1776903366726.tmp
2026-04-23T00:16:06.726Z [INFO]  server-8c9dbea7-85a8-e94a-2d5a-52c804090a3d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:06.726Z [INFO]  server-5361306b-a151-e82d-32c4-f1c2df6fd2a3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:06.726Z [INFO]  server-8c9dbea7-85a8-e94a-2d5a-52c804090a3d: copied to local snapshot: bytes=71
2026-04-23T00:16:06.727Z [INFO]  server-8c9dbea7-85a8-e94a-2d5a-52c804090a3d: snapshot restore progress: id=2-1013-1776903366726 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:06.727Z [INFO]  server-8c9dbea7-85a8-e94a-2d5a-52c804090a3d: removing all old logs from log store
2026-04-23T00:16:06.727Z [INFO]  server-8c9dbea7-85a8-e94a-2d5a-52c804090a3d: compacting logs: from=1 to=2
2026-04-23T00:16:06.727Z [INFO]  server-8c9dbea7-85a8-e94a-2d5a-52c804090a3d: Installed remote snapshot
2026-04-23T00:16:06.727Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: pipelining replication: peer="{Voter server-8c9dbea7-85a8-e94a-2d5a-52c804090a3d 8c9dbea7-85a8-e94a-2d5a-52c804090a3d}"
2026-04-23T00:16:06.727Z [INFO]  server-5361306b-a151-e82d-32c4-f1c2df6fd2a3: copied to local snapshot: bytes=71
2026-04-23T00:16:06.727Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:06.727Z [INFO]  server-5361306b-a151-e82d-32c4-f1c2df6fd2a3: snapshot restore progress: id=2-1013-1776903366726 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:06.727Z [INFO]  server-5361306b-a151-e82d-32c4-f1c2df6fd2a3: removing all old logs from log store
2026-04-23T00:16:06.727Z [INFO]  server-5361306b-a151-e82d-32c4-f1c2df6fd2a3: compacting logs: from=1 to=2
2026-04-23T00:16:06.727Z [INFO]  server-5361306b-a151-e82d-32c4-f1c2df6fd2a3: Installed remote snapshot
2026-04-23T00:16:06.727Z [INFO]  server-dadefc40-6bdd-4f38-fe6f-a71a97129268: pipelining replication: peer="{Voter server-5361306b-a151-e82d-32c4-f1c2df6fd2a3 5361306b-a151-e82d-32c4-f1c2df6fd2a3}"
2026-04-23T00:16:07.732Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:06.727262655 +0000 UTC m=+83.766864017 (2 nodes), 659ns from start of poll, 1.5406877s from cluster start. Timeout at 2026-04-23 00:16:07.732291178 +0000 UTC m=+84.771892531, 1.005028514s after stability
2026-04-23T00:16:07.744Z [INFO]  server-da6e5e02-bf7a-e644-ba4a-2dfe1976fb29: aborting pipeline replication: peer="{Voter server-ae37dee0-3cd3-1e3c-afe7-a07aa0e3fe22 ae37dee0-3cd3-1e3c-afe7-a07aa0e3fe22}"
2026-04-23T00:16:07.744Z [INFO]  server-da6e5e02-bf7a-e644-ba4a-2dfe1976fb29: aborting pipeline replication: peer="{Voter server-a24f3239-df5a-0218-4a79-0a1bf472cde3 a24f3239-df5a-0218-4a79-0a1bf472cde3}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false
2026-04-23T00:16:07.745Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-efb13812-ada6-4348-a598-5d07f711f67f Address:efb13812-ada6-4348-a598-5d07f711f67f} {Suffrage:Voter ID:server-af2f3b5e-0cd6-4c9a-25dd-b40f595efe17 Address:af2f3b5e-0cd6-4c9a-25dd-b40f595efe17} {Suffrage:Voter ID:server-1d0f1471-d024-d5f1-fe51-8e313677e4bd Address:1d0f1471-d024-d5f1-fe51-8e313677e4bd}]"
2026-04-23T00:16:07.745Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: entering follower state: follower="Node at efb13812-ada6-4348-a598-5d07f711f67f [Follower]" leader-address= leader-id=
2026-04-23T00:16:07.745Z [INFO]  server-af2f3b5e-0cd6-4c9a-25dd-b40f595efe17: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-efb13812-ada6-4348-a598-5d07f711f67f Address:efb13812-ada6-4348-a598-5d07f711f67f} {Suffrage:Voter ID:server-af2f3b5e-0cd6-4c9a-25dd-b40f595efe17 Address:af2f3b5e-0cd6-4c9a-25dd-b40f595efe17} {Suffrage:Voter ID:server-1d0f1471-d024-d5f1-fe51-8e313677e4bd Address:1d0f1471-d024-d5f1-fe51-8e313677e4bd}]"
2026-04-23T00:16:07.745Z [INFO]  server-af2f3b5e-0cd6-4c9a-25dd-b40f595efe17: entering follower state: follower="Node at af2f3b5e-0cd6-4c9a-25dd-b40f595efe17 [Follower]" leader-address= leader-id=
2026-04-23T00:16:07.745Z [INFO]  server-1d0f1471-d024-d5f1-fe51-8e313677e4bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-efb13812-ada6-4348-a598-5d07f711f67f Address:efb13812-ada6-4348-a598-5d07f711f67f} {Suffrage:Voter ID:server-af2f3b5e-0cd6-4c9a-25dd-b40f595efe17 Address:af2f3b5e-0cd6-4c9a-25dd-b40f595efe17} {Suffrage:Voter ID:server-1d0f1471-d024-d5f1-fe51-8e313677e4bd Address:1d0f1471-d024-d5f1-fe51-8e313677e4bd}]"
2026-04-23T00:16:07.745Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:07.745Z [INFO]  server-1d0f1471-d024-d5f1-fe51-8e313677e4bd: entering follower state: follower="Node at 1d0f1471-d024-d5f1-fe51-8e313677e4bd [Follower]" leader-address= leader-id=
2026-04-23T00:16:08.388Z [WARN]  server-efb13812-ada6-4348-a598-5d07f711f67f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:08.388Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: entering candidate state: node="Node at efb13812-ada6-4348-a598-5d07f711f67f [Candidate]" term=2
2026-04-23T00:16:08.388Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: election won: term=2 tally=2
2026-04-23T00:16:08.388Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: entering leader state: leader="Node at efb13812-ada6-4348-a598-5d07f711f67f [Leader]"
2026-04-23T00:16:08.388Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: added peer, starting replication: peer=server-af2f3b5e-0cd6-4c9a-25dd-b40f595efe17
2026-04-23T00:16:08.388Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: added peer, starting replication: peer=server-1d0f1471-d024-d5f1-fe51-8e313677e4bd
2026-04-23T00:16:08.389Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: pipelining replication: peer="{Voter server-1d0f1471-d024-d5f1-fe51-8e313677e4bd 1d0f1471-d024-d5f1-fe51-8e313677e4bd}"
2026-04-23T00:16:08.389Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: pipelining replication: peer="{Voter server-af2f3b5e-0cd6-4c9a-25dd-b40f595efe17 af2f3b5e-0cd6-4c9a-25dd-b40f595efe17}"
2026-04-23T00:16:09.394Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:08.388967991 +0000 UTC m=+85.428569366 (1 nodes), 643.223088ms from start of poll, 643.542532ms from cluster start. Timeout at 2026-04-23 00:16:09.393971664 +0000 UTC m=+86.433573016, 1.00500365s after stability
2026-04-23T00:16:09.394Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: starting snapshot up to: index=12
2026-04-23T00:16:09.394Z [INFO]  creating new snapshot: path=/tmp/raft878494470/snapshots/2-12-1776903369394.tmp
2026-04-23T00:16:09.459Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: snapshot complete up to: index=12
2026-04-23T00:16:09.459Z [INFO]  creating new snapshot: path=/tmp/raft878494470/snapshots/2-10013-1776903369459.tmp
2026-04-23T00:16:09.484Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: copied to local snapshot: bytes=71
2026-04-23T00:16:09.484Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: snapshot restore progress: id=2-10013-1776903369459 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:09.485Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: restored user snapshot: index=10013
2026-04-23T00:16:09.485Z [ERROR] server-efb13812-ada6-4348-a598-5d07f711f67f: failed to get log: index=23 error="log not found"
2026-04-23T00:16:09.485Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: aborting pipeline replication: peer="{Voter server-1d0f1471-d024-d5f1-fe51-8e313677e4bd 1d0f1471-d024-d5f1-fe51-8e313677e4bd}"
2026-04-23T00:16:09.485Z [ERROR] server-efb13812-ada6-4348-a598-5d07f711f67f: failed to get log: index=23 error="log not found"
2026-04-23T00:16:09.485Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: aborting pipeline replication: peer="{Voter server-af2f3b5e-0cd6-4c9a-25dd-b40f595efe17 af2f3b5e-0cd6-4c9a-25dd-b40f595efe17}"
2026-04-23T00:16:09.490Z [ERROR] server-efb13812-ada6-4348-a598-5d07f711f67f: failed to get log: index=23 error="log not found"
2026-04-23T00:16:09.490Z [INFO]  creating new snapshot: path=/tmp/raft327374047/snapshots/2-10013-1776903369490.tmp
2026-04-23T00:16:09.491Z [INFO]  server-1d0f1471-d024-d5f1-fe51-8e313677e4bd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:09.491Z [INFO]  server-1d0f1471-d024-d5f1-fe51-8e313677e4bd: copied to local snapshot: bytes=71
2026-04-23T00:16:09.491Z [INFO]  server-1d0f1471-d024-d5f1-fe51-8e313677e4bd: snapshot restore progress: id=2-10013-1776903369490 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:09.491Z [INFO]  server-1d0f1471-d024-d5f1-fe51-8e313677e4bd: Installed remote snapshot
2026-04-23T00:16:09.491Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: pipelining replication: peer="{Voter server-1d0f1471-d024-d5f1-fe51-8e313677e4bd 1d0f1471-d024-d5f1-fe51-8e313677e4bd}"
2026-04-23T00:16:09.491Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:09.492Z [ERROR] server-efb13812-ada6-4348-a598-5d07f711f67f: failed to get log: index=23 error="log not found"
2026-04-23T00:16:09.493Z [INFO]  creating new snapshot: path=/tmp/raft1022366385/snapshots/2-10013-1776903369493.tmp
2026-04-23T00:16:09.493Z [INFO]  server-af2f3b5e-0cd6-4c9a-25dd-b40f595efe17: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:09.775Z [INFO]  server-af2f3b5e-0cd6-4c9a-25dd-b40f595efe17: copied to local snapshot: bytes=71
2026-04-23T00:16:09.775Z [INFO]  server-af2f3b5e-0cd6-4c9a-25dd-b40f595efe17: snapshot restore progress: id=2-10013-1776903369493 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:09.776Z [INFO]  server-af2f3b5e-0cd6-4c9a-25dd-b40f595efe17: Installed remote snapshot
2026-04-23T00:16:09.776Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: pipelining replication: peer="{Voter server-af2f3b5e-0cd6-4c9a-25dd-b40f595efe17 af2f3b5e-0cd6-4c9a-25dd-b40f595efe17}"
2026-04-23T00:16:10.496Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:09.491714075 +0000 UTC m=+86.531315427 (2 nodes), 838ns from start of poll, 1.746288593s from cluster start. Timeout at 2026-04-23 00:16:10.49672834 +0000 UTC m=+87.536329706, 1.005014279s after stability
2026-04-23T00:16:10.507Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: aborting pipeline replication: peer="{Voter server-1d0f1471-d024-d5f1-fe51-8e313677e4bd 1d0f1471-d024-d5f1-fe51-8e313677e4bd}"
2026-04-23T00:16:10.507Z [INFO]  server-efb13812-ada6-4348-a598-5d07f711f67f: aborting pipeline replication: peer="{Voter server-af2f3b5e-0cd6-4c9a-25dd-b40f595efe17 af2f3b5e-0cd6-4c9a-25dd-b40f595efe17}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false
2026-04-23T00:16:10.509Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a32a2001-2398-f001-e55e-58aa6293bc47 Address:a32a2001-2398-f001-e55e-58aa6293bc47} {Suffrage:Voter ID:server-769cc2f6-d109-a3d1-86cd-9b238bf2904d Address:769cc2f6-d109-a3d1-86cd-9b238bf2904d} {Suffrage:Voter ID:server-e86f2db6-28a4-e7bf-b822-6963fbc4dc0b Address:e86f2db6-28a4-e7bf-b822-6963fbc4dc0b}]"
2026-04-23T00:16:10.509Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: entering follower state: follower="Node at a32a2001-2398-f001-e55e-58aa6293bc47 [Follower]" leader-address= leader-id=
2026-04-23T00:16:10.509Z [INFO]  server-769cc2f6-d109-a3d1-86cd-9b238bf2904d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a32a2001-2398-f001-e55e-58aa6293bc47 Address:a32a2001-2398-f001-e55e-58aa6293bc47} {Suffrage:Voter ID:server-769cc2f6-d109-a3d1-86cd-9b238bf2904d Address:769cc2f6-d109-a3d1-86cd-9b238bf2904d} {Suffrage:Voter ID:server-e86f2db6-28a4-e7bf-b822-6963fbc4dc0b Address:e86f2db6-28a4-e7bf-b822-6963fbc4dc0b}]"
2026-04-23T00:16:10.509Z [INFO]  server-769cc2f6-d109-a3d1-86cd-9b238bf2904d: entering follower state: follower="Node at 769cc2f6-d109-a3d1-86cd-9b238bf2904d [Follower]" leader-address= leader-id=
2026-04-23T00:16:10.509Z [INFO]  server-e86f2db6-28a4-e7bf-b822-6963fbc4dc0b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a32a2001-2398-f001-e55e-58aa6293bc47 Address:a32a2001-2398-f001-e55e-58aa6293bc47} {Suffrage:Voter ID:server-769cc2f6-d109-a3d1-86cd-9b238bf2904d Address:769cc2f6-d109-a3d1-86cd-9b238bf2904d} {Suffrage:Voter ID:server-e86f2db6-28a4-e7bf-b822-6963fbc4dc0b Address:e86f2db6-28a4-e7bf-b822-6963fbc4dc0b}]"
2026-04-23T00:16:10.509Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:10.509Z [INFO]  server-e86f2db6-28a4-e7bf-b822-6963fbc4dc0b: entering follower state: follower="Node at e86f2db6-28a4-e7bf-b822-6963fbc4dc0b [Follower]" leader-address= leader-id=
2026-04-23T00:16:11.059Z [WARN]  server-a32a2001-2398-f001-e55e-58aa6293bc47: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:11.059Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: entering candidate state: node="Node at a32a2001-2398-f001-e55e-58aa6293bc47 [Candidate]" term=2
2026-04-23T00:16:11.060Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: election won: term=2 tally=2
2026-04-23T00:16:11.060Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: entering leader state: leader="Node at a32a2001-2398-f001-e55e-58aa6293bc47 [Leader]"
2026-04-23T00:16:11.060Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: added peer, starting replication: peer=server-769cc2f6-d109-a3d1-86cd-9b238bf2904d
2026-04-23T00:16:11.060Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: added peer, starting replication: peer=server-e86f2db6-28a4-e7bf-b822-6963fbc4dc0b
2026-04-23T00:16:11.060Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: pipelining replication: peer="{Voter server-769cc2f6-d109-a3d1-86cd-9b238bf2904d 769cc2f6-d109-a3d1-86cd-9b238bf2904d}"
2026-04-23T00:16:11.060Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: pipelining replication: peer="{Voter server-e86f2db6-28a4-e7bf-b822-6963fbc4dc0b e86f2db6-28a4-e7bf-b822-6963fbc4dc0b}"
2026-04-23T00:16:12.065Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:11.060105013 +0000 UTC m=+88.099706368 (1 nodes), 550.714453ms from start of poll, 551.001742ms from cluster start. Timeout at 2026-04-23 00:16:12.065179119 +0000 UTC m=+89.104780475, 1.005074107s after stability
2026-04-23T00:16:12.066Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: starting snapshot up to: index=12
2026-04-23T00:16:12.066Z [INFO]  creating new snapshot: path=/tmp/raft3375878444/snapshots/2-12-1776903372066.tmp
2026-04-23T00:16:12.189Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: snapshot complete up to: index=12
2026-04-23T00:16:12.191Z [INFO]  creating new snapshot: path=/tmp/raft3375878444/snapshots/2-10013-1776903372191.tmp
2026-04-23T00:16:12.283Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: copied to local snapshot: bytes=71
2026-04-23T00:16:12.283Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: snapshot restore progress: id=2-10013-1776903372191 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:12.283Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: removing all old logs from log store
2026-04-23T00:16:12.283Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: compacting logs: from=1 to=22
2026-04-23T00:16:12.283Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: restored user snapshot: index=10013
2026-04-23T00:16:12.283Z [ERROR] server-a32a2001-2398-f001-e55e-58aa6293bc47: failed to get log: index=22 error="log not found"
2026-04-23T00:16:12.283Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: aborting pipeline replication: peer="{Voter server-e86f2db6-28a4-e7bf-b822-6963fbc4dc0b e86f2db6-28a4-e7bf-b822-6963fbc4dc0b}"
2026-04-23T00:16:12.283Z [ERROR] server-a32a2001-2398-f001-e55e-58aa6293bc47: failed to get log: index=22 error="log not found"
2026-04-23T00:16:12.283Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: aborting pipeline replication: peer="{Voter server-769cc2f6-d109-a3d1-86cd-9b238bf2904d 769cc2f6-d109-a3d1-86cd-9b238bf2904d}"
2026-04-23T00:16:12.290Z [ERROR] server-a32a2001-2398-f001-e55e-58aa6293bc47: failed to get log: index=22 error="log not found"
2026-04-23T00:16:12.290Z [INFO]  creating new snapshot: path=/tmp/raft1442360595/snapshots/2-10013-1776903372290.tmp
2026-04-23T00:16:12.291Z [INFO]  server-e86f2db6-28a4-e7bf-b822-6963fbc4dc0b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:12.292Z [ERROR] server-a32a2001-2398-f001-e55e-58aa6293bc47: failed to get log: index=22 error="log not found"
2026-04-23T00:16:12.293Z [INFO]  creating new snapshot: path=/tmp/raft1224921610/snapshots/2-10013-1776903372293.tmp
2026-04-23T00:16:12.293Z [INFO]  server-769cc2f6-d109-a3d1-86cd-9b238bf2904d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:12.521Z [INFO]  server-769cc2f6-d109-a3d1-86cd-9b238bf2904d: copied to local snapshot: bytes=71
2026-04-23T00:16:12.521Z [INFO]  server-e86f2db6-28a4-e7bf-b822-6963fbc4dc0b: copied to local snapshot: bytes=71
2026-04-23T00:16:12.521Z [INFO]  server-769cc2f6-d109-a3d1-86cd-9b238bf2904d: snapshot restore progress: id=2-10013-1776903372293 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:12.521Z [INFO]  server-769cc2f6-d109-a3d1-86cd-9b238bf2904d: removing all old logs from log store
2026-04-23T00:16:12.521Z [INFO]  server-769cc2f6-d109-a3d1-86cd-9b238bf2904d: compacting logs: from=1 to=22
2026-04-23T00:16:12.521Z [INFO]  server-769cc2f6-d109-a3d1-86cd-9b238bf2904d: Installed remote snapshot
2026-04-23T00:16:12.521Z [INFO]  server-e86f2db6-28a4-e7bf-b822-6963fbc4dc0b: snapshot restore progress: id=2-10013-1776903372290 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:12.521Z [INFO]  server-e86f2db6-28a4-e7bf-b822-6963fbc4dc0b: removing all old logs from log store
2026-04-23T00:16:12.521Z [INFO]  server-e86f2db6-28a4-e7bf-b822-6963fbc4dc0b: compacting logs: from=1 to=22
2026-04-23T00:16:12.521Z [INFO]  server-e86f2db6-28a4-e7bf-b822-6963fbc4dc0b: Installed remote snapshot
2026-04-23T00:16:12.521Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: pipelining replication: peer="{Voter server-769cc2f6-d109-a3d1-86cd-9b238bf2904d 769cc2f6-d109-a3d1-86cd-9b238bf2904d}"
2026-04-23T00:16:12.521Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:12.521Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: pipelining replication: peer="{Voter server-e86f2db6-28a4-e7bf-b822-6963fbc4dc0b e86f2db6-28a4-e7bf-b822-6963fbc4dc0b}"
2026-04-23T00:16:13.527Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:12.521603289 +0000 UTC m=+89.561204659 (2 nodes), 731ns from start of poll, 2.012500033s from cluster start. Timeout at 2026-04-23 00:16:13.526631048 +0000 UTC m=+90.566232441, 1.005027782s after stability
2026-04-23T00:16:13.539Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: aborting pipeline replication: peer="{Voter server-e86f2db6-28a4-e7bf-b822-6963fbc4dc0b e86f2db6-28a4-e7bf-b822-6963fbc4dc0b}"
2026-04-23T00:16:13.539Z [INFO]  server-a32a2001-2398-f001-e55e-58aa6293bc47: aborting pipeline replication: peer="{Voter server-769cc2f6-d109-a3d1-86cd-9b238bf2904d 769cc2f6-d109-a3d1-86cd-9b238bf2904d}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true
2026-04-23T00:16:13.540Z [INFO]  server-61315ea8-663f-c9f3-12b0-ad01aa787ef8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-61315ea8-663f-c9f3-12b0-ad01aa787ef8 Address:61315ea8-663f-c9f3-12b0-ad01aa787ef8} {Suffrage:Voter ID:server-c913caca-af1e-2df8-80f0-a95271d71a87 Address:c913caca-af1e-2df8-80f0-a95271d71a87} {Suffrage:Voter ID:server-57feb8a3-2f16-b71f-7cf2-9f4654571fa9 Address:57feb8a3-2f16-b71f-7cf2-9f4654571fa9}]"
2026-04-23T00:16:13.540Z [INFO]  server-61315ea8-663f-c9f3-12b0-ad01aa787ef8: entering follower state: follower="Node at 61315ea8-663f-c9f3-12b0-ad01aa787ef8 [Follower]" leader-address= leader-id=
2026-04-23T00:16:13.540Z [INFO]  server-c913caca-af1e-2df8-80f0-a95271d71a87: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-61315ea8-663f-c9f3-12b0-ad01aa787ef8 Address:61315ea8-663f-c9f3-12b0-ad01aa787ef8} {Suffrage:Voter ID:server-c913caca-af1e-2df8-80f0-a95271d71a87 Address:c913caca-af1e-2df8-80f0-a95271d71a87} {Suffrage:Voter ID:server-57feb8a3-2f16-b71f-7cf2-9f4654571fa9 Address:57feb8a3-2f16-b71f-7cf2-9f4654571fa9}]"
2026-04-23T00:16:13.540Z [INFO]  server-57feb8a3-2f16-b71f-7cf2-9f4654571fa9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-61315ea8-663f-c9f3-12b0-ad01aa787ef8 Address:61315ea8-663f-c9f3-12b0-ad01aa787ef8} {Suffrage:Voter ID:server-c913caca-af1e-2df8-80f0-a95271d71a87 Address:c913caca-af1e-2df8-80f0-a95271d71a87} {Suffrage:Voter ID:server-57feb8a3-2f16-b71f-7cf2-9f4654571fa9 Address:57feb8a3-2f16-b71f-7cf2-9f4654571fa9}]"
2026-04-23T00:16:13.540Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:13.540Z [INFO]  server-c913caca-af1e-2df8-80f0-a95271d71a87: entering follower state: follower="Node at c913caca-af1e-2df8-80f0-a95271d71a87 [Follower]" leader-address= leader-id=
2026-04-23T00:16:13.540Z [INFO]  server-57feb8a3-2f16-b71f-7cf2-9f4654571fa9: entering follower state: follower="Node at 57feb8a3-2f16-b71f-7cf2-9f4654571fa9 [Follower]" leader-address= leader-id=
2026-04-23T00:16:14.242Z [WARN]  server-61315ea8-663f-c9f3-12b0-ad01aa787ef8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:14.242Z [INFO]  server-61315ea8-663f-c9f3-12b0-ad01aa787ef8: entering candidate state: node="Node at 61315ea8-663f-c9f3-12b0-ad01aa787ef8 [Candidate]" term=2
2026-04-23T00:16:14.242Z [INFO]  server-61315ea8-663f-c9f3-12b0-ad01aa787ef8: election won: term=2 tally=2
2026-04-23T00:16:14.242Z [INFO]  server-61315ea8-663f-c9f3-12b0-ad01aa787ef8: entering leader state: leader="Node at 61315ea8-663f-c9f3-12b0-ad01aa787ef8 [Leader]"
2026-04-23T00:16:14.242Z [INFO]  server-61315ea8-663f-c9f3-12b0-ad01aa787ef8: added peer, starting replication: peer=server-c913caca-af1e-2df8-80f0-a95271d71a87
2026-04-23T00:16:14.242Z [INFO]  server-61315ea8-663f-c9f3-12b0-ad01aa787ef8: added peer, starting replication: peer=server-57feb8a3-2f16-b71f-7cf2-9f4654571fa9
2026-04-23T00:16:14.242Z [INFO]  server-61315ea8-663f-c9f3-12b0-ad01aa787ef8: pipelining replication: peer="{Voter server-c913caca-af1e-2df8-80f0-a95271d71a87 c913caca-af1e-2df8-80f0-a95271d71a87}"
2026-04-23T00:16:14.242Z [INFO]  server-61315ea8-663f-c9f3-12b0-ad01aa787ef8: pipelining replication: peer="{Voter server-57feb8a3-2f16-b71f-7cf2-9f4654571fa9 57feb8a3-2f16-b71f-7cf2-9f4654571fa9}"
2026-04-23T00:16:15.247Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:14.242374321 +0000 UTC m=+91.281975743 (1 nodes), 701.667278ms from start of poll, 701.996068ms from cluster start. Timeout at 2026-04-23 00:16:15.24737512 +0000 UTC m=+92.286976472, 1.005000729s after stability
2026-04-23T00:16:15.248Z [INFO]  server-61315ea8-663f-c9f3-12b0-ad01aa787ef8: starting snapshot up to: index=12
2026-04-23T00:16:15.248Z [INFO]  creating new snapshot: path=/tmp/raft4241974856/snapshots/2-12-1776903375248.tmp
2026-04-23T00:16:15.248Z [INFO]  server-61315ea8-663f-c9f3-12b0-ad01aa787ef8: snapshot complete up to: index=12
2026-04-23T00:16:15.250Z [INFO]  server-02ce5d4c-700d-ad4a-1078-1ad4f0157325: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02ce5d4c-700d-ad4a-1078-1ad4f0157325 Address:02ce5d4c-700d-ad4a-1078-1ad4f0157325} {Suffrage:Voter ID:server-b221ed00-ae1f-fe39-c705-796d86e86e70 Address:b221ed00-ae1f-fe39-c705-796d86e86e70} {Suffrage:Voter ID:server-438a5059-4fd7-449a-50cf-09515be6101c Address:438a5059-4fd7-449a-50cf-09515be6101c}]"
2026-04-23T00:16:15.250Z [INFO]  server-b221ed00-ae1f-fe39-c705-796d86e86e70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02ce5d4c-700d-ad4a-1078-1ad4f0157325 Address:02ce5d4c-700d-ad4a-1078-1ad4f0157325} {Suffrage:Voter ID:server-b221ed00-ae1f-fe39-c705-796d86e86e70 Address:b221ed00-ae1f-fe39-c705-796d86e86e70} {Suffrage:Voter ID:server-438a5059-4fd7-449a-50cf-09515be6101c Address:438a5059-4fd7-449a-50cf-09515be6101c}]"
2026-04-23T00:16:15.250Z [INFO]  server-02ce5d4c-700d-ad4a-1078-1ad4f0157325: entering follower state: follower="Node at 02ce5d4c-700d-ad4a-1078-1ad4f0157325 [Follower]" leader-address= leader-id=
2026-04-23T00:16:15.250Z [INFO]  server-b221ed00-ae1f-fe39-c705-796d86e86e70: entering follower state: follower="Node at b221ed00-ae1f-fe39-c705-796d86e86e70 [Follower]" leader-address= leader-id=
2026-04-23T00:16:15.250Z [INFO]  server-438a5059-4fd7-449a-50cf-09515be6101c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02ce5d4c-700d-ad4a-1078-1ad4f0157325 Address:02ce5d4c-700d-ad4a-1078-1ad4f0157325} {Suffrage:Voter ID:server-b221ed00-ae1f-fe39-c705-796d86e86e70 Address:b221ed00-ae1f-fe39-c705-796d86e86e70} {Suffrage:Voter ID:server-438a5059-4fd7-449a-50cf-09515be6101c Address:438a5059-4fd7-449a-50cf-09515be6101c}]"
2026-04-23T00:16:15.250Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:15.250Z [INFO]  server-438a5059-4fd7-449a-50cf-09515be6101c: entering follower state: follower="Node at 438a5059-4fd7-449a-50cf-09515be6101c [Follower]" leader-address= leader-id=
2026-04-23T00:16:15.800Z [WARN]  server-b221ed00-ae1f-fe39-c705-796d86e86e70: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:15.800Z [INFO]  server-b221ed00-ae1f-fe39-c705-796d86e86e70: entering candidate state: node="Node at b221ed00-ae1f-fe39-c705-796d86e86e70 [Candidate]" term=2
2026-04-23T00:16:15.800Z [INFO]  server-b221ed00-ae1f-fe39-c705-796d86e86e70: election won: term=2 tally=2
2026-04-23T00:16:15.800Z [INFO]  server-b221ed00-ae1f-fe39-c705-796d86e86e70: entering leader state: leader="Node at b221ed00-ae1f-fe39-c705-796d86e86e70 [Leader]"
2026-04-23T00:16:15.800Z [INFO]  server-b221ed00-ae1f-fe39-c705-796d86e86e70: added peer, starting replication: peer=server-02ce5d4c-700d-ad4a-1078-1ad4f0157325
2026-04-23T00:16:15.800Z [INFO]  server-b221ed00-ae1f-fe39-c705-796d86e86e70: added peer, starting replication: peer=server-438a5059-4fd7-449a-50cf-09515be6101c
2026-04-23T00:16:15.800Z [INFO]  server-b221ed00-ae1f-fe39-c705-796d86e86e70: pipelining replication: peer="{Voter server-438a5059-4fd7-449a-50cf-09515be6101c 438a5059-4fd7-449a-50cf-09515be6101c}"
2026-04-23T00:16:15.800Z [INFO]  server-b221ed00-ae1f-fe39-c705-796d86e86e70: pipelining replication: peer="{Voter server-02ce5d4c-700d-ad4a-1078-1ad4f0157325 02ce5d4c-700d-ad4a-1078-1ad4f0157325}"
2026-04-23T00:16:16.805Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:15.800362696 +0000 UTC m=+92.839964068 (1 nodes), 549.487281ms from start of poll, 549.807708ms from cluster start. Timeout at 2026-04-23 00:16:16.805365805 +0000 UTC m=+93.844967161, 1.005003093s after stability
2026-04-23T00:16:16.805Z [INFO]  creating new snapshot: path=/tmp/raft2638448657/snapshots/2-10013-1776903376805.tmp
2026-04-23T00:16:16.806Z [INFO]  server-b221ed00-ae1f-fe39-c705-796d86e86e70: copied to local snapshot: bytes=71
2026-04-23T00:16:16.806Z [INFO]  server-b221ed00-ae1f-fe39-c705-796d86e86e70: snapshot restore progress: id=2-10013-1776903376805 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:16.806Z [INFO]  server-b221ed00-ae1f-fe39-c705-796d86e86e70: restored user snapshot: index=10013
2026-04-23T00:16:16.806Z [ERROR] server-b221ed00-ae1f-fe39-c705-796d86e86e70: failed to get log: index=3 error="log not found"
2026-04-23T00:16:16.806Z [ERROR] server-b221ed00-ae1f-fe39-c705-796d86e86e70: failed to get log: index=3 error="log not found"
2026-04-23T00:16:16.806Z [INFO]  server-b221ed00-ae1f-fe39-c705-796d86e86e70: aborting pipeline replication: peer="{Voter server-02ce5d4c-700d-ad4a-1078-1ad4f0157325 02ce5d4c-700d-ad4a-1078-1ad4f0157325}"
2026-04-23T00:16:16.806Z [INFO]  server-b221ed00-ae1f-fe39-c705-796d86e86e70: aborting pipeline replication: peer="{Voter server-438a5059-4fd7-449a-50cf-09515be6101c 438a5059-4fd7-449a-50cf-09515be6101c}"
2026-04-23T00:16:16.814Z [ERROR] server-b221ed00-ae1f-fe39-c705-796d86e86e70: failed to get log: index=3 error="log not found"
2026-04-23T00:16:16.814Z [INFO]  creating new snapshot: path=/tmp/raft485460003/snapshots/2-10013-1776903376814.tmp
2026-04-23T00:16:16.814Z [INFO]  server-438a5059-4fd7-449a-50cf-09515be6101c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:16.815Z [INFO]  server-438a5059-4fd7-449a-50cf-09515be6101c: copied to local snapshot: bytes=71
2026-04-23T00:16:16.815Z [INFO]  server-438a5059-4fd7-449a-50cf-09515be6101c: snapshot restore progress: id=2-10013-1776903376814 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:16.815Z [INFO]  server-438a5059-4fd7-449a-50cf-09515be6101c: Installed remote snapshot
2026-04-23T00:16:16.815Z [INFO]  server-b221ed00-ae1f-fe39-c705-796d86e86e70: pipelining replication: peer="{Voter server-438a5059-4fd7-449a-50cf-09515be6101c 438a5059-4fd7-449a-50cf-09515be6101c}"
2026-04-23T00:16:16.815Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:16.816Z [ERROR] server-b221ed00-ae1f-fe39-c705-796d86e86e70: failed to get log: index=3 error="log not found"
2026-04-23T00:16:16.816Z [INFO]  creating new snapshot: path=/tmp/raft4084642614/snapshots/2-10013-1776903376816.tmp
2026-04-23T00:16:16.817Z [INFO]  server-02ce5d4c-700d-ad4a-1078-1ad4f0157325: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:16.817Z [INFO]  server-02ce5d4c-700d-ad4a-1078-1ad4f0157325: copied to local snapshot: bytes=71
2026-04-23T00:16:16.817Z [INFO]  server-02ce5d4c-700d-ad4a-1078-1ad4f0157325: snapshot restore progress: id=2-10013-1776903376816 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:16.817Z [INFO]  server-02ce5d4c-700d-ad4a-1078-1ad4f0157325: Installed remote snapshot
2026-04-23T00:16:16.817Z [INFO]  server-b221ed00-ae1f-fe39-c705-796d86e86e70: pipelining replication: peer="{Voter server-02ce5d4c-700d-ad4a-1078-1ad4f0157325 02ce5d4c-700d-ad4a-1078-1ad4f0157325}"
2026-04-23T00:16:17.821Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:16.815396427 +0000 UTC m=+93.854997780 (2 nodes), 521ns from start of poll, 1.56484142s from cluster start. Timeout at 2026-04-23 00:16:17.820417527 +0000 UTC m=+94.860018901, 1.005021121s after stability
2026-04-23T00:16:17.832Z [INFO]  server-61315ea8-663f-c9f3-12b0-ad01aa787ef8: aborting pipeline replication: peer="{Voter server-c913caca-af1e-2df8-80f0-a95271d71a87 c913caca-af1e-2df8-80f0-a95271d71a87}"
2026-04-23T00:16:17.832Z [INFO]  server-61315ea8-663f-c9f3-12b0-ad01aa787ef8: aborting pipeline replication: peer="{Voter server-57feb8a3-2f16-b71f-7cf2-9f4654571fa9 57feb8a3-2f16-b71f-7cf2-9f4654571fa9}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true
2026-04-23T00:16:17.834Z [INFO]  server-62f55914-c9dd-810d-a72d-f1c9ca6d0efd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62f55914-c9dd-810d-a72d-f1c9ca6d0efd Address:62f55914-c9dd-810d-a72d-f1c9ca6d0efd} {Suffrage:Voter ID:server-bf535615-ad1e-ac89-9c3f-2f95b3bcaefd Address:bf535615-ad1e-ac89-9c3f-2f95b3bcaefd} {Suffrage:Voter ID:server-989356df-05bd-679e-1b7b-94de4cf9d4c6 Address:989356df-05bd-679e-1b7b-94de4cf9d4c6}]"
2026-04-23T00:16:17.834Z [INFO]  server-62f55914-c9dd-810d-a72d-f1c9ca6d0efd: entering follower state: follower="Node at 62f55914-c9dd-810d-a72d-f1c9ca6d0efd [Follower]" leader-address= leader-id=
2026-04-23T00:16:17.834Z [INFO]  server-bf535615-ad1e-ac89-9c3f-2f95b3bcaefd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62f55914-c9dd-810d-a72d-f1c9ca6d0efd Address:62f55914-c9dd-810d-a72d-f1c9ca6d0efd} {Suffrage:Voter ID:server-bf535615-ad1e-ac89-9c3f-2f95b3bcaefd Address:bf535615-ad1e-ac89-9c3f-2f95b3bcaefd} {Suffrage:Voter ID:server-989356df-05bd-679e-1b7b-94de4cf9d4c6 Address:989356df-05bd-679e-1b7b-94de4cf9d4c6}]"
2026-04-23T00:16:17.834Z [INFO]  server-989356df-05bd-679e-1b7b-94de4cf9d4c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62f55914-c9dd-810d-a72d-f1c9ca6d0efd Address:62f55914-c9dd-810d-a72d-f1c9ca6d0efd} {Suffrage:Voter ID:server-bf535615-ad1e-ac89-9c3f-2f95b3bcaefd Address:bf535615-ad1e-ac89-9c3f-2f95b3bcaefd} {Suffrage:Voter ID:server-989356df-05bd-679e-1b7b-94de4cf9d4c6 Address:989356df-05bd-679e-1b7b-94de4cf9d4c6}]"
2026-04-23T00:16:17.834Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:17.834Z [INFO]  server-bf535615-ad1e-ac89-9c3f-2f95b3bcaefd: entering follower state: follower="Node at bf535615-ad1e-ac89-9c3f-2f95b3bcaefd [Follower]" leader-address= leader-id=
2026-04-23T00:16:17.834Z [INFO]  server-989356df-05bd-679e-1b7b-94de4cf9d4c6: entering follower state: follower="Node at 989356df-05bd-679e-1b7b-94de4cf9d4c6 [Follower]" leader-address= leader-id=
2026-04-23T00:16:18.405Z [WARN]  server-62f55914-c9dd-810d-a72d-f1c9ca6d0efd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:18.405Z [INFO]  server-62f55914-c9dd-810d-a72d-f1c9ca6d0efd: entering candidate state: node="Node at 62f55914-c9dd-810d-a72d-f1c9ca6d0efd [Candidate]" term=2
2026-04-23T00:16:18.406Z [INFO]  server-62f55914-c9dd-810d-a72d-f1c9ca6d0efd: election won: term=2 tally=2
2026-04-23T00:16:18.406Z [INFO]  server-62f55914-c9dd-810d-a72d-f1c9ca6d0efd: entering leader state: leader="Node at 62f55914-c9dd-810d-a72d-f1c9ca6d0efd [Leader]"
2026-04-23T00:16:18.406Z [INFO]  server-62f55914-c9dd-810d-a72d-f1c9ca6d0efd: added peer, starting replication: peer=server-bf535615-ad1e-ac89-9c3f-2f95b3bcaefd
2026-04-23T00:16:18.406Z [INFO]  server-62f55914-c9dd-810d-a72d-f1c9ca6d0efd: added peer, starting replication: peer=server-989356df-05bd-679e-1b7b-94de4cf9d4c6
2026-04-23T00:16:18.406Z [INFO]  server-62f55914-c9dd-810d-a72d-f1c9ca6d0efd: pipelining replication: peer="{Voter server-bf535615-ad1e-ac89-9c3f-2f95b3bcaefd bf535615-ad1e-ac89-9c3f-2f95b3bcaefd}"
2026-04-23T00:16:18.406Z [INFO]  server-62f55914-c9dd-810d-a72d-f1c9ca6d0efd: pipelining replication: peer="{Voter server-989356df-05bd-679e-1b7b-94de4cf9d4c6 989356df-05bd-679e-1b7b-94de4cf9d4c6}"
2026-04-23T00:16:19.411Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:18.406074203 +0000 UTC m=+95.445675574 (1 nodes), 571.631467ms from start of poll, 571.954285ms from cluster start. Timeout at 2026-04-23 00:16:19.411074986 +0000 UTC m=+96.450676338, 1.005000764s after stability
2026-04-23T00:16:19.411Z [INFO]  server-62f55914-c9dd-810d-a72d-f1c9ca6d0efd: starting snapshot up to: index=12
2026-04-23T00:16:19.411Z [INFO]  creating new snapshot: path=/tmp/raft2854866299/snapshots/2-12-1776903379411.tmp
2026-04-23T00:16:19.412Z [INFO]  server-62f55914-c9dd-810d-a72d-f1c9ca6d0efd: snapshot complete up to: index=12
2026-04-23T00:16:19.413Z [INFO]  server-c7390b9b-baa2-3d4a-4b45-dd191dfd29e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7390b9b-baa2-3d4a-4b45-dd191dfd29e8 Address:c7390b9b-baa2-3d4a-4b45-dd191dfd29e8} {Suffrage:Voter ID:server-5df76976-a46f-c6ea-5a1d-32102d0595d6 Address:5df76976-a46f-c6ea-5a1d-32102d0595d6} {Suffrage:Voter ID:server-b0215080-6286-a625-b640-056096099819 Address:b0215080-6286-a625-b640-056096099819}]"
2026-04-23T00:16:19.413Z [INFO]  server-c7390b9b-baa2-3d4a-4b45-dd191dfd29e8: entering follower state: follower="Node at c7390b9b-baa2-3d4a-4b45-dd191dfd29e8 [Follower]" leader-address= leader-id=
2026-04-23T00:16:19.413Z [INFO]  server-5df76976-a46f-c6ea-5a1d-32102d0595d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7390b9b-baa2-3d4a-4b45-dd191dfd29e8 Address:c7390b9b-baa2-3d4a-4b45-dd191dfd29e8} {Suffrage:Voter ID:server-5df76976-a46f-c6ea-5a1d-32102d0595d6 Address:5df76976-a46f-c6ea-5a1d-32102d0595d6} {Suffrage:Voter ID:server-b0215080-6286-a625-b640-056096099819 Address:b0215080-6286-a625-b640-056096099819}]"
2026-04-23T00:16:19.414Z [INFO]  server-5df76976-a46f-c6ea-5a1d-32102d0595d6: entering follower state: follower="Node at 5df76976-a46f-c6ea-5a1d-32102d0595d6 [Follower]" leader-address= leader-id=
2026-04-23T00:16:19.414Z [INFO]  server-b0215080-6286-a625-b640-056096099819: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7390b9b-baa2-3d4a-4b45-dd191dfd29e8 Address:c7390b9b-baa2-3d4a-4b45-dd191dfd29e8} {Suffrage:Voter ID:server-5df76976-a46f-c6ea-5a1d-32102d0595d6 Address:5df76976-a46f-c6ea-5a1d-32102d0595d6} {Suffrage:Voter ID:server-b0215080-6286-a625-b640-056096099819 Address:b0215080-6286-a625-b640-056096099819}]"
2026-04-23T00:16:19.414Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:19.414Z [INFO]  server-b0215080-6286-a625-b640-056096099819: entering follower state: follower="Node at b0215080-6286-a625-b640-056096099819 [Follower]" leader-address= leader-id=
2026-04-23T00:16:19.930Z [WARN]  server-b0215080-6286-a625-b640-056096099819: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:19.930Z [INFO]  server-b0215080-6286-a625-b640-056096099819: entering candidate state: node="Node at b0215080-6286-a625-b640-056096099819 [Candidate]" term=2
2026-04-23T00:16:19.930Z [INFO]  server-b0215080-6286-a625-b640-056096099819: election won: term=2 tally=2
2026-04-23T00:16:19.930Z [INFO]  server-b0215080-6286-a625-b640-056096099819: entering leader state: leader="Node at b0215080-6286-a625-b640-056096099819 [Leader]"
2026-04-23T00:16:19.930Z [INFO]  server-b0215080-6286-a625-b640-056096099819: added peer, starting replication: peer=server-c7390b9b-baa2-3d4a-4b45-dd191dfd29e8
2026-04-23T00:16:19.930Z [INFO]  server-b0215080-6286-a625-b640-056096099819: added peer, starting replication: peer=server-5df76976-a46f-c6ea-5a1d-32102d0595d6
2026-04-23T00:16:19.931Z [INFO]  server-b0215080-6286-a625-b640-056096099819: pipelining replication: peer="{Voter server-5df76976-a46f-c6ea-5a1d-32102d0595d6 5df76976-a46f-c6ea-5a1d-32102d0595d6}"
2026-04-23T00:16:19.931Z [INFO]  server-b0215080-6286-a625-b640-056096099819: pipelining replication: peer="{Voter server-c7390b9b-baa2-3d4a-4b45-dd191dfd29e8 c7390b9b-baa2-3d4a-4b45-dd191dfd29e8}"
2026-04-23T00:16:20.936Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:19.930971076 +0000 UTC m=+96.970572435 (1 nodes), 516.590765ms from start of poll, 517.582834ms from cluster start. Timeout at 2026-04-23 00:16:20.935978162 +0000 UTC m=+97.975579514, 1.005007079s after stability
2026-04-23T00:16:20.936Z [INFO]  creating new snapshot: path=/tmp/raft3106898519/snapshots/2-10013-1776903380936.tmp
2026-04-23T00:16:20.937Z [INFO]  server-b0215080-6286-a625-b640-056096099819: copied to local snapshot: bytes=71
2026-04-23T00:16:20.937Z [INFO]  server-b0215080-6286-a625-b640-056096099819: snapshot restore progress: id=2-10013-1776903380936 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:20.937Z [INFO]  server-b0215080-6286-a625-b640-056096099819: removing all old logs from log store
2026-04-23T00:16:20.937Z [INFO]  server-b0215080-6286-a625-b640-056096099819: compacting logs: from=1 to=2
2026-04-23T00:16:20.937Z [INFO]  server-b0215080-6286-a625-b640-056096099819: restored user snapshot: index=10013
2026-04-23T00:16:20.937Z [ERROR] server-b0215080-6286-a625-b640-056096099819: failed to get log: index=2 error="log not found"
2026-04-23T00:16:20.937Z [ERROR] server-b0215080-6286-a625-b640-056096099819: failed to get log: index=2 error="log not found"
2026-04-23T00:16:20.937Z [INFO]  server-b0215080-6286-a625-b640-056096099819: aborting pipeline replication: peer="{Voter server-c7390b9b-baa2-3d4a-4b45-dd191dfd29e8 c7390b9b-baa2-3d4a-4b45-dd191dfd29e8}"
2026-04-23T00:16:20.937Z [INFO]  server-b0215080-6286-a625-b640-056096099819: aborting pipeline replication: peer="{Voter server-5df76976-a46f-c6ea-5a1d-32102d0595d6 5df76976-a46f-c6ea-5a1d-32102d0595d6}"
2026-04-23T00:16:20.943Z [ERROR] server-b0215080-6286-a625-b640-056096099819: failed to get log: index=2 error="log not found"
2026-04-23T00:16:20.944Z [INFO]  creating new snapshot: path=/tmp/raft2165483286/snapshots/2-10013-1776903380944.tmp
2026-04-23T00:16:20.944Z [INFO]  server-c7390b9b-baa2-3d4a-4b45-dd191dfd29e8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:20.945Z [INFO]  server-c7390b9b-baa2-3d4a-4b45-dd191dfd29e8: copied to local snapshot: bytes=71
2026-04-23T00:16:20.945Z [INFO]  server-c7390b9b-baa2-3d4a-4b45-dd191dfd29e8: snapshot restore progress: id=2-10013-1776903380944 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:20.945Z [INFO]  server-c7390b9b-baa2-3d4a-4b45-dd191dfd29e8: removing all old logs from log store
2026-04-23T00:16:20.945Z [INFO]  server-c7390b9b-baa2-3d4a-4b45-dd191dfd29e8: compacting logs: from=1 to=2
2026-04-23T00:16:20.945Z [INFO]  server-c7390b9b-baa2-3d4a-4b45-dd191dfd29e8: Installed remote snapshot
2026-04-23T00:16:20.945Z [INFO]  server-b0215080-6286-a625-b640-056096099819: pipelining replication: peer="{Voter server-c7390b9b-baa2-3d4a-4b45-dd191dfd29e8 c7390b9b-baa2-3d4a-4b45-dd191dfd29e8}"
2026-04-23T00:16:20.945Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:20.946Z [ERROR] server-b0215080-6286-a625-b640-056096099819: failed to get log: index=2 error="log not found"
2026-04-23T00:16:20.947Z [INFO]  creating new snapshot: path=/tmp/raft2988008774/snapshots/2-10013-1776903380947.tmp
2026-04-23T00:16:20.947Z [INFO]  server-5df76976-a46f-c6ea-5a1d-32102d0595d6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:20.947Z [INFO]  server-5df76976-a46f-c6ea-5a1d-32102d0595d6: copied to local snapshot: bytes=71
2026-04-23T00:16:20.947Z [INFO]  server-5df76976-a46f-c6ea-5a1d-32102d0595d6: snapshot restore progress: id=2-10013-1776903380947 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-04-23T00:16:20.948Z [INFO]  server-5df76976-a46f-c6ea-5a1d-32102d0595d6: removing all old logs from log store
2026-04-23T00:16:20.948Z [INFO]  server-5df76976-a46f-c6ea-5a1d-32102d0595d6: compacting logs: from=1 to=2
2026-04-23T00:16:20.948Z [INFO]  server-5df76976-a46f-c6ea-5a1d-32102d0595d6: Installed remote snapshot
2026-04-23T00:16:20.948Z [INFO]  server-b0215080-6286-a625-b640-056096099819: pipelining replication: peer="{Voter server-5df76976-a46f-c6ea-5a1d-32102d0595d6 5df76976-a46f-c6ea-5a1d-32102d0595d6}"
2026-04-23T00:16:21.950Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:20.945516347 +0000 UTC m=+97.985117720 (2 nodes), 784ns from start of poll, 1.532128119s from cluster start. Timeout at 2026-04-23 00:16:21.950530668 +0000 UTC m=+98.990132070, 1.00501435s after stability
2026-04-23T00:16:21.961Z [INFO]  server-62f55914-c9dd-810d-a72d-f1c9ca6d0efd: aborting pipeline replication: peer="{Voter server-989356df-05bd-679e-1b7b-94de4cf9d4c6 989356df-05bd-679e-1b7b-94de4cf9d4c6}"
2026-04-23T00:16:21.961Z [INFO]  server-62f55914-c9dd-810d-a72d-f1c9ca6d0efd: aborting pipeline replication: peer="{Voter server-bf535615-ad1e-ac89-9c3f-2f95b3bcaefd bf535615-ad1e-ac89-9c3f-2f95b3bcaefd}"
--- PASS: TestRaft_UserRestore (84.47s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.66s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.60s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.32s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.22s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.57s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.66s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.84s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.20s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.90s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.34s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.19s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.78s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.56s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.39s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.10s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.90s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.62s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.40s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.39s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.76s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (3.03s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.29s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.13s)
=== RUN   TestRaft_SendSnapshotFollower
2026-04-23T00:16:21.962Z [INFO]  server-6f082d2f-811f-2481-8fea-a5fffdf65996: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f082d2f-811f-2481-8fea-a5fffdf65996 Address:6f082d2f-811f-2481-8fea-a5fffdf65996} {Suffrage:Voter ID:server-e5cd35e2-ca79-2f36-6068-46c483a7a21d Address:e5cd35e2-ca79-2f36-6068-46c483a7a21d} {Suffrage:Voter ID:server-3cdc0e9d-be8f-ae7c-b3fe-44e48ec82f7f Address:3cdc0e9d-be8f-ae7c-b3fe-44e48ec82f7f}]"
2026-04-23T00:16:21.963Z [INFO]  server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f082d2f-811f-2481-8fea-a5fffdf65996 Address:6f082d2f-811f-2481-8fea-a5fffdf65996} {Suffrage:Voter ID:server-e5cd35e2-ca79-2f36-6068-46c483a7a21d Address:e5cd35e2-ca79-2f36-6068-46c483a7a21d} {Suffrage:Voter ID:server-3cdc0e9d-be8f-ae7c-b3fe-44e48ec82f7f Address:3cdc0e9d-be8f-ae7c-b3fe-44e48ec82f7f}]"
2026-04-23T00:16:21.963Z [INFO]  server-6f082d2f-811f-2481-8fea-a5fffdf65996: entering follower state: follower="Node at 6f082d2f-811f-2481-8fea-a5fffdf65996 [Follower]" leader-address= leader-id=
2026-04-23T00:16:21.963Z [INFO]  server-3cdc0e9d-be8f-ae7c-b3fe-44e48ec82f7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f082d2f-811f-2481-8fea-a5fffdf65996 Address:6f082d2f-811f-2481-8fea-a5fffdf65996} {Suffrage:Voter ID:server-e5cd35e2-ca79-2f36-6068-46c483a7a21d Address:e5cd35e2-ca79-2f36-6068-46c483a7a21d} {Suffrage:Voter ID:server-3cdc0e9d-be8f-ae7c-b3fe-44e48ec82f7f Address:3cdc0e9d-be8f-ae7c-b3fe-44e48ec82f7f}]"
2026-04-23T00:16:21.963Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:21.963Z [INFO]  server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: entering follower state: follower="Node at e5cd35e2-ca79-2f36-6068-46c483a7a21d [Follower]" leader-address= leader-id=
2026-04-23T00:16:21.963Z [INFO]  server-3cdc0e9d-be8f-ae7c-b3fe-44e48ec82f7f: entering follower state: follower="Node at 3cdc0e9d-be8f-ae7c-b3fe-44e48ec82f7f [Follower]" leader-address= leader-id=
2026-04-23T00:16:22.020Z [WARN]  server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:22.020Z [INFO]  server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: entering candidate state: node="Node at e5cd35e2-ca79-2f36-6068-46c483a7a21d [Candidate]" term=2
2026-04-23T00:16:22.020Z [INFO]  server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: election won: term=2 tally=2
2026-04-23T00:16:22.020Z [INFO]  server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: entering leader state: leader="Node at e5cd35e2-ca79-2f36-6068-46c483a7a21d [Leader]"
2026-04-23T00:16:22.020Z [INFO]  server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: added peer, starting replication: peer=server-6f082d2f-811f-2481-8fea-a5fffdf65996
2026-04-23T00:16:22.020Z [INFO]  server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: added peer, starting replication: peer=server-3cdc0e9d-be8f-ae7c-b3fe-44e48ec82f7f
2026-04-23T00:16:22.020Z [INFO]  server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: pipelining replication: peer="{Voter server-6f082d2f-811f-2481-8fea-a5fffdf65996 6f082d2f-811f-2481-8fea-a5fffdf65996}"
2026-04-23T00:16:22.020Z [INFO]  server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: pipelining replication: peer="{Voter server-3cdc0e9d-be8f-ae7c-b3fe-44e48ec82f7f 3cdc0e9d-be8f-ae7c-b3fe-44e48ec82f7f}"
2026-04-23T00:16:22.126Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:22.020404854 +0000 UTC m=+99.060006223 (2 nodes), 57.220418ms from start of poll, 57.593114ms from cluster start. Timeout at 2026-04-23 00:16:22.125408269 +0000 UTC m=+99.165009624, 105.003401ms after stability
2026-04-23T00:16:22.126Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:22.232Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:22.126572746 +0000 UTC m=+99.166174098 (1 nodes), 991ns from start of poll, 163.760989ms from cluster start. Timeout at 2026-04-23 00:16:22.231601557 +0000 UTC m=+99.271202913, 105.028815ms after stability
2026-04-23T00:16:22.232Z [ERROR] server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: failed to pipeline appendEntries: peer="{Voter server-6f082d2f-811f-2481-8fea-a5fffdf65996 6f082d2f-811f-2481-8fea-a5fffdf65996}" error="append pipeline closed"
2026-04-23T00:16:22.232Z [INFO]  server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: aborting pipeline replication: peer="{Voter server-6f082d2f-811f-2481-8fea-a5fffdf65996 6f082d2f-811f-2481-8fea-a5fffdf65996}"
2026-04-23T00:16:22.232Z [ERROR] server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: failed to appendEntries to: peer="{Voter server-6f082d2f-811f-2481-8fea-a5fffdf65996 6f082d2f-811f-2481-8fea-a5fffdf65996}" error="failed to connect to peer: 6f082d2f-811f-2481-8fea-a5fffdf65996"
    raft_test.go:1627: [INFO] Finished apply without behind follower
2026-04-23T00:16:22.234Z [ERROR] server-6f082d2f-811f-2481-8fea-a5fffdf65996: failed to take snapshot: error="nothing new to snapshot"
2026-04-23T00:16:22.234Z [INFO]  server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: starting snapshot up to: index=102
2026-04-23T00:16:22.234Z [INFO]  creating new snapshot: path=/tmp/raft1146983368/snapshots/2-102-1776903382234.tmp
2026-04-23T00:16:22.234Z [INFO]  server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: compacting logs: from=1 to=92
2026-04-23T00:16:22.234Z [INFO]  server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: snapshot complete up to: index=102
2026-04-23T00:16:22.234Z [INFO]  server-3cdc0e9d-be8f-ae7c-b3fe-44e48ec82f7f: starting snapshot up to: index=101
2026-04-23T00:16:22.234Z [INFO]  creating new snapshot: path=/tmp/raft2035103795/snapshots/2-101-1776903382234.tmp
2026-04-23T00:16:22.235Z [INFO]  server-3cdc0e9d-be8f-ae7c-b3fe-44e48ec82f7f: compacting logs: from=1 to=92
2026-04-23T00:16:22.235Z [INFO]  server-3cdc0e9d-be8f-ae7c-b3fe-44e48ec82f7f: snapshot complete up to: index=101
2026-04-23T00:16:22.242Z [ERROR] server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: failed to get log: index=2 error="log not found"
2026-04-23T00:16:22.242Z [INFO]  creating new snapshot: path=/tmp/raft52127671/snapshots/2-102-1776903382242.tmp
2026-04-23T00:16:22.242Z [INFO]  server-6f082d2f-811f-2481-8fea-a5fffdf65996: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2026-04-23T00:16:22.243Z [INFO]  server-6f082d2f-811f-2481-8fea-a5fffdf65996: copied to local snapshot: bytes=693
2026-04-23T00:16:22.243Z [INFO]  server-6f082d2f-811f-2481-8fea-a5fffdf65996: snapshot restore progress: id=2-102-1776903382242 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-04-23T00:16:22.243Z [INFO]  server-6f082d2f-811f-2481-8fea-a5fffdf65996: Installed remote snapshot
2026-04-23T00:16:22.243Z [WARN]  server-6f082d2f-811f-2481-8fea-a5fffdf65996: skipping application of old log: index=102
2026-04-23T00:16:22.243Z [INFO]  server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: pipelining replication: peer="{Voter server-6f082d2f-811f-2481-8fea-a5fffdf65996 6f082d2f-811f-2481-8fea-a5fffdf65996}"
2026-04-23T00:16:22.247Z [INFO]  server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: aborting pipeline replication: peer="{Voter server-3cdc0e9d-be8f-ae7c-b3fe-44e48ec82f7f 3cdc0e9d-be8f-ae7c-b3fe-44e48ec82f7f}"
2026-04-23T00:16:22.247Z [INFO]  server-e5cd35e2-ca79-2f36-6068-46c483a7a21d: aborting pipeline replication: peer="{Voter server-6f082d2f-811f-2481-8fea-a5fffdf65996 6f082d2f-811f-2481-8fea-a5fffdf65996}"
--- PASS: TestRaft_SendSnapshotFollower (0.29s)
=== RUN   TestRaft_SendSnapshotAndLogsFollower
2026-04-23T00:16:22.248Z [INFO]  server-a16d3de8-a1d9-917f-cde5-30c14a1ced16: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a16d3de8-a1d9-917f-cde5-30c14a1ced16 Address:a16d3de8-a1d9-917f-cde5-30c14a1ced16} {Suffrage:Voter ID:server-83b72e99-a6d5-1143-1e9c-289510be7e6b Address:83b72e99-a6d5-1143-1e9c-289510be7e6b} {Suffrage:Voter ID:server-f44a65d1-6776-fe09-e542-140c56806599 Address:f44a65d1-6776-fe09-e542-140c56806599}]"
2026-04-23T00:16:22.248Z [INFO]  server-83b72e99-a6d5-1143-1e9c-289510be7e6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a16d3de8-a1d9-917f-cde5-30c14a1ced16 Address:a16d3de8-a1d9-917f-cde5-30c14a1ced16} {Suffrage:Voter ID:server-83b72e99-a6d5-1143-1e9c-289510be7e6b Address:83b72e99-a6d5-1143-1e9c-289510be7e6b} {Suffrage:Voter ID:server-f44a65d1-6776-fe09-e542-140c56806599 Address:f44a65d1-6776-fe09-e542-140c56806599}]"
2026-04-23T00:16:22.248Z [INFO]  server-a16d3de8-a1d9-917f-cde5-30c14a1ced16: entering follower state: follower="Node at a16d3de8-a1d9-917f-cde5-30c14a1ced16 [Follower]" leader-address= leader-id=
2026-04-23T00:16:22.248Z [INFO]  server-f44a65d1-6776-fe09-e542-140c56806599: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a16d3de8-a1d9-917f-cde5-30c14a1ced16 Address:a16d3de8-a1d9-917f-cde5-30c14a1ced16} {Suffrage:Voter ID:server-83b72e99-a6d5-1143-1e9c-289510be7e6b Address:83b72e99-a6d5-1143-1e9c-289510be7e6b} {Suffrage:Voter ID:server-f44a65d1-6776-fe09-e542-140c56806599 Address:f44a65d1-6776-fe09-e542-140c56806599}]"
2026-04-23T00:16:22.248Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:22.248Z [INFO]  server-83b72e99-a6d5-1143-1e9c-289510be7e6b: entering follower state: follower="Node at 83b72e99-a6d5-1143-1e9c-289510be7e6b [Follower]" leader-address= leader-id=
2026-04-23T00:16:22.248Z [INFO]  server-f44a65d1-6776-fe09-e542-140c56806599: entering follower state: follower="Node at f44a65d1-6776-fe09-e542-140c56806599 [Follower]" leader-address= leader-id=
2026-04-23T00:16:22.302Z [WARN]  server-f44a65d1-6776-fe09-e542-140c56806599: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:22.302Z [INFO]  server-f44a65d1-6776-fe09-e542-140c56806599: entering candidate state: node="Node at f44a65d1-6776-fe09-e542-140c56806599 [Candidate]" term=2
2026-04-23T00:16:22.302Z [INFO]  server-f44a65d1-6776-fe09-e542-140c56806599: election won: term=2 tally=2
2026-04-23T00:16:22.302Z [INFO]  server-f44a65d1-6776-fe09-e542-140c56806599: entering leader state: leader="Node at f44a65d1-6776-fe09-e542-140c56806599 [Leader]"
2026-04-23T00:16:22.302Z [INFO]  server-f44a65d1-6776-fe09-e542-140c56806599: added peer, starting replication: peer=server-a16d3de8-a1d9-917f-cde5-30c14a1ced16
2026-04-23T00:16:22.302Z [INFO]  server-f44a65d1-6776-fe09-e542-140c56806599: added peer, starting replication: peer=server-83b72e99-a6d5-1143-1e9c-289510be7e6b
2026-04-23T00:16:22.302Z [INFO]  server-f44a65d1-6776-fe09-e542-140c56806599: pipelining replication: peer="{Voter server-83b72e99-a6d5-1143-1e9c-289510be7e6b 83b72e99-a6d5-1143-1e9c-289510be7e6b}"
2026-04-23T00:16:22.302Z [INFO]  server-f44a65d1-6776-fe09-e542-140c56806599: pipelining replication: peer="{Voter server-a16d3de8-a1d9-917f-cde5-30c14a1ced16 a16d3de8-a1d9-917f-cde5-30c14a1ced16}"
2026-04-23T00:16:22.408Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:22.302733175 +0000 UTC m=+99.342334527 (2 nodes), 54.474011ms from start of poll, 54.720716ms from cluster start. Timeout at 2026-04-23 00:16:22.407737429 +0000 UTC m=+99.447338785, 105.004258ms after stability
2026-04-23T00:16:22.408Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:22.514Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:22.408378037 +0000 UTC m=+99.447979392 (1 nodes), 749ns from start of poll, 160.365581ms from cluster start. Timeout at 2026-04-23 00:16:22.513388803 +0000 UTC m=+99.552990178, 105.010786ms after stability
2026-04-23T00:16:22.514Z [ERROR] server-f44a65d1-6776-fe09-e542-140c56806599: failed to pipeline appendEntries: peer="{Voter server-a16d3de8-a1d9-917f-cde5-30c14a1ced16 a16d3de8-a1d9-917f-cde5-30c14a1ced16}" error="append pipeline closed"
2026-04-23T00:16:22.514Z [INFO]  server-f44a65d1-6776-fe09-e542-140c56806599: aborting pipeline replication: peer="{Voter server-a16d3de8-a1d9-917f-cde5-30c14a1ced16 a16d3de8-a1d9-917f-cde5-30c14a1ced16}"
2026-04-23T00:16:22.514Z [ERROR] server-f44a65d1-6776-fe09-e542-140c56806599: failed to appendEntries to: peer="{Voter server-a16d3de8-a1d9-917f-cde5-30c14a1ced16 a16d3de8-a1d9-917f-cde5-30c14a1ced16}" error="failed to connect to peer: a16d3de8-a1d9-917f-cde5-30c14a1ced16"
    raft_test.go:1669: [INFO] Finished apply without behind follower
2026-04-23T00:16:22.516Z [ERROR] server-a16d3de8-a1d9-917f-cde5-30c14a1ced16: failed to take snapshot: error="nothing new to snapshot"
2026-04-23T00:16:22.516Z [INFO]  server-83b72e99-a6d5-1143-1e9c-289510be7e6b: starting snapshot up to: index=96
2026-04-23T00:16:22.516Z [INFO]  creating new snapshot: path=/tmp/raft406067802/snapshots/2-96-1776903382516.tmp
2026-04-23T00:16:22.521Z [ERROR] server-f44a65d1-6776-fe09-e542-140c56806599: failed to heartbeat to: peer=a16d3de8-a1d9-917f-cde5-30c14a1ced16 backoff time=10ms error="failed to connect to peer: a16d3de8-a1d9-917f-cde5-30c14a1ced16"
2026-04-23T00:16:22.525Z [ERROR] server-f44a65d1-6776-fe09-e542-140c56806599: failed to appendEntries to: peer="{Voter server-a16d3de8-a1d9-917f-cde5-30c14a1ced16 a16d3de8-a1d9-917f-cde5-30c14a1ced16}" error="failed to connect to peer: a16d3de8-a1d9-917f-cde5-30c14a1ced16"
2026-04-23T00:16:22.535Z [ERROR] server-f44a65d1-6776-fe09-e542-140c56806599: failed to appendEntries to: peer="{Voter server-a16d3de8-a1d9-917f-cde5-30c14a1ced16 a16d3de8-a1d9-917f-cde5-30c14a1ced16}" error="failed to connect to peer: a16d3de8-a1d9-917f-cde5-30c14a1ced16"
2026-04-23T00:16:22.537Z [INFO]  server-83b72e99-a6d5-1143-1e9c-289510be7e6b: compacting logs: from=1 to=92
2026-04-23T00:16:22.537Z [INFO]  server-83b72e99-a6d5-1143-1e9c-289510be7e6b: snapshot complete up to: index=96
2026-04-23T00:16:22.537Z [INFO]  server-f44a65d1-6776-fe09-e542-140c56806599: starting snapshot up to: index=102
2026-04-23T00:16:22.537Z [INFO]  creating new snapshot: path=/tmp/raft1323839456/snapshots/2-102-1776903382537.tmp
2026-04-23T00:16:22.538Z [INFO]  server-f44a65d1-6776-fe09-e542-140c56806599: compacting logs: from=1 to=92
2026-04-23T00:16:22.538Z [INFO]  server-f44a65d1-6776-fe09-e542-140c56806599: snapshot complete up to: index=102
2026-04-23T00:16:22.538Z [ERROR] server-f44a65d1-6776-fe09-e542-140c56806599: failed to heartbeat to: peer=a16d3de8-a1d9-917f-cde5-30c14a1ced16 backoff time=10ms error="failed to connect to peer: a16d3de8-a1d9-917f-cde5-30c14a1ced16"
    raft_test.go:1690: [INFO] Finished apply without behind follower
2026-04-23T00:16:22.558Z [ERROR] server-f44a65d1-6776-fe09-e542-140c56806599: failed to get log: index=2 error="log not found"
2026-04-23T00:16:22.559Z [INFO]  creating new snapshot: path=/tmp/raft4071854722/snapshots/2-102-1776903382559.tmp
2026-04-23T00:16:22.559Z [INFO]  server-a16d3de8-a1d9-917f-cde5-30c14a1ced16: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2026-04-23T00:16:22.559Z [INFO]  server-a16d3de8-a1d9-917f-cde5-30c14a1ced16: copied to local snapshot: bytes=693
2026-04-23T00:16:22.559Z [INFO]  server-a16d3de8-a1d9-917f-cde5-30c14a1ced16: snapshot restore progress: id=2-102-1776903382559 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-04-23T00:16:22.559Z [INFO]  server-a16d3de8-a1d9-917f-cde5-30c14a1ced16: Installed remote snapshot
2026-04-23T00:16:22.559Z [INFO]  server-f44a65d1-6776-fe09-e542-140c56806599: pipelining replication: peer="{Voter server-a16d3de8-a1d9-917f-cde5-30c14a1ced16 a16d3de8-a1d9-917f-cde5-30c14a1ced16}"
2026-04-23T00:16:22.570Z [INFO]  server-f44a65d1-6776-fe09-e542-140c56806599: aborting pipeline replication: peer="{Voter server-a16d3de8-a1d9-917f-cde5-30c14a1ced16 a16d3de8-a1d9-917f-cde5-30c14a1ced16}"
2026-04-23T00:16:22.570Z [INFO]  server-f44a65d1-6776-fe09-e542-140c56806599: aborting pipeline replication: peer="{Voter server-83b72e99-a6d5-1143-1e9c-289510be7e6b 83b72e99-a6d5-1143-1e9c-289510be7e6b}"
2026-04-23T00:16:23.070Z [ERROR] server-f44a65d1-6776-fe09-e542-140c56806599: failed to heartbeat to: peer=a16d3de8-a1d9-917f-cde5-30c14a1ced16 backoff time=10ms error="command timed out"
--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.82s)
=== RUN   TestRaft_ReJoinFollower
2026-04-23T00:16:23.071Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aedba1c2-728f-cb97-f543-91ac59167337 Address:aedba1c2-728f-cb97-f543-91ac59167337} {Suffrage:Voter ID:server-edf42206-2278-6bb6-a87d-798f4be563c6 Address:edf42206-2278-6bb6-a87d-798f4be563c6} {Suffrage:Voter ID:server-8ad71f1d-d5e4-6316-dee7-e50cf2cfe428 Address:8ad71f1d-d5e4-6316-dee7-e50cf2cfe428}]"
2026-04-23T00:16:23.072Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: entering follower state: follower="Node at aedba1c2-728f-cb97-f543-91ac59167337 [Follower]" leader-address= leader-id=
2026-04-23T00:16:23.072Z [INFO]  server-edf42206-2278-6bb6-a87d-798f4be563c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aedba1c2-728f-cb97-f543-91ac59167337 Address:aedba1c2-728f-cb97-f543-91ac59167337} {Suffrage:Voter ID:server-edf42206-2278-6bb6-a87d-798f4be563c6 Address:edf42206-2278-6bb6-a87d-798f4be563c6} {Suffrage:Voter ID:server-8ad71f1d-d5e4-6316-dee7-e50cf2cfe428 Address:8ad71f1d-d5e4-6316-dee7-e50cf2cfe428}]"
2026-04-23T00:16:23.072Z [INFO]  server-8ad71f1d-d5e4-6316-dee7-e50cf2cfe428: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aedba1c2-728f-cb97-f543-91ac59167337 Address:aedba1c2-728f-cb97-f543-91ac59167337} {Suffrage:Voter ID:server-edf42206-2278-6bb6-a87d-798f4be563c6 Address:edf42206-2278-6bb6-a87d-798f4be563c6} {Suffrage:Voter ID:server-8ad71f1d-d5e4-6316-dee7-e50cf2cfe428 Address:8ad71f1d-d5e4-6316-dee7-e50cf2cfe428}]"
2026-04-23T00:16:23.072Z [INFO]  server-edf42206-2278-6bb6-a87d-798f4be563c6: entering follower state: follower="Node at edf42206-2278-6bb6-a87d-798f4be563c6 [Follower]" leader-address= leader-id=
2026-04-23T00:16:23.072Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:23.072Z [INFO]  server-8ad71f1d-d5e4-6316-dee7-e50cf2cfe428: entering follower state: follower="Node at 8ad71f1d-d5e4-6316-dee7-e50cf2cfe428 [Follower]" leader-address= leader-id=
2026-04-23T00:16:23.125Z [WARN]  server-aedba1c2-728f-cb97-f543-91ac59167337: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:23.125Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: entering candidate state: node="Node at aedba1c2-728f-cb97-f543-91ac59167337 [Candidate]" term=2
2026-04-23T00:16:23.125Z [WARN]  server-8ad71f1d-d5e4-6316-dee7-e50cf2cfe428: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:23.125Z [INFO]  server-8ad71f1d-d5e4-6316-dee7-e50cf2cfe428: entering candidate state: node="Node at 8ad71f1d-d5e4-6316-dee7-e50cf2cfe428 [Candidate]" term=2
2026-04-23T00:16:23.125Z [INFO]  server-8ad71f1d-d5e4-6316-dee7-e50cf2cfe428: duplicate requestVote for same term: term=2
2026-04-23T00:16:23.125Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: duplicate requestVote for same term: term=2
2026-04-23T00:16:23.125Z [INFO]  server-edf42206-2278-6bb6-a87d-798f4be563c6: duplicate requestVote for same term: term=2
2026-04-23T00:16:23.125Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: election won: term=2 tally=2
2026-04-23T00:16:23.125Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: entering leader state: leader="Node at aedba1c2-728f-cb97-f543-91ac59167337 [Leader]"
2026-04-23T00:16:23.125Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: added peer, starting replication: peer=server-edf42206-2278-6bb6-a87d-798f4be563c6
2026-04-23T00:16:23.125Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: added peer, starting replication: peer=server-8ad71f1d-d5e4-6316-dee7-e50cf2cfe428
2026-04-23T00:16:23.125Z [INFO]  server-8ad71f1d-d5e4-6316-dee7-e50cf2cfe428: entering follower state: follower="Node at 8ad71f1d-d5e4-6316-dee7-e50cf2cfe428 [Follower]" leader-address=aedba1c2-728f-cb97-f543-91ac59167337 leader-id=server-aedba1c2-728f-cb97-f543-91ac59167337
2026-04-23T00:16:23.125Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: pipelining replication: peer="{Voter server-8ad71f1d-d5e4-6316-dee7-e50cf2cfe428 8ad71f1d-d5e4-6316-dee7-e50cf2cfe428}"
2026-04-23T00:16:23.125Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: pipelining replication: peer="{Voter server-edf42206-2278-6bb6-a87d-798f4be563c6 edf42206-2278-6bb6-a87d-798f4be563c6}"
2026-04-23T00:16:23.231Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:23.125605104 +0000 UTC m=+100.165206466 (1 nodes), 53.446781ms from start of poll, 53.748462ms from cluster start. Timeout at 2026-04-23 00:16:23.230606156 +0000 UTC m=+100.270207512, 105.001046ms after stability
2026-04-23T00:16:23.236Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:23.341Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:23.236343469 +0000 UTC m=+100.275944826 (2 nodes), 935ns from start of poll, 164.486822ms from cluster start. Timeout at 2026-04-23 00:16:23.34137504 +0000 UTC m=+100.380976396, 105.03157ms after stability
2026-04-23T00:16:23.341Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: updating configuration: command=RemoveServer server-id=server-edf42206-2278-6bb6-a87d-798f4be563c6 server-addr= servers="[{Suffrage:Voter ID:server-aedba1c2-728f-cb97-f543-91ac59167337 Address:aedba1c2-728f-cb97-f543-91ac59167337} {Suffrage:Voter ID:server-8ad71f1d-d5e4-6316-dee7-e50cf2cfe428 Address:8ad71f1d-d5e4-6316-dee7-e50cf2cfe428}]"
2026-04-23T00:16:23.341Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: removed peer, stopping replication: peer=server-edf42206-2278-6bb6-a87d-798f4be563c6 last-index=3
2026-04-23T00:16:23.341Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: aborting pipeline replication: peer="{Voter server-edf42206-2278-6bb6-a87d-798f4be563c6 edf42206-2278-6bb6-a87d-798f4be563c6}"
2026-04-23T00:16:23.447Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: updating configuration: command=AddVoter server-id=server-edf42206-2278-6bb6-a87d-798f4be563c6 server-addr=edf42206-2278-6bb6-a87d-798f4be563c6 servers="[{Suffrage:Voter ID:server-aedba1c2-728f-cb97-f543-91ac59167337 Address:aedba1c2-728f-cb97-f543-91ac59167337} {Suffrage:Voter ID:server-8ad71f1d-d5e4-6316-dee7-e50cf2cfe428 Address:8ad71f1d-d5e4-6316-dee7-e50cf2cfe428} {Suffrage:Voter ID:server-edf42206-2278-6bb6-a87d-798f4be563c6 Address:edf42206-2278-6bb6-a87d-798f4be563c6}]"
2026-04-23T00:16:23.448Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: added peer, starting replication: peer=server-edf42206-2278-6bb6-a87d-798f4be563c6
2026-04-23T00:16:23.448Z [WARN]  server-edf42206-2278-6bb6-a87d-798f4be563c6: failed to get previous log: previous-index=4 last-index=3 error="log not found"
2026-04-23T00:16:23.448Z [WARN]  server-aedba1c2-728f-cb97-f543-91ac59167337: appendEntries rejected, sending older logs: peer="{Voter server-edf42206-2278-6bb6-a87d-798f4be563c6 edf42206-2278-6bb6-a87d-798f4be563c6}" next=4
2026-04-23T00:16:23.448Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: pipelining replication: peer="{Voter server-edf42206-2278-6bb6-a87d-798f4be563c6 edf42206-2278-6bb6-a87d-798f4be563c6}"
2026-04-23T00:16:23.448Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:23.553Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:23.448184414 +0000 UTC m=+100.487785766 (1 nodes), 747ns from start of poll, 376.327762ms from cluster start. Timeout at 2026-04-23 00:16:23.55320946 +0000 UTC m=+100.592810813, 105.025047ms after stability
2026-04-23T00:16:23.553Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: aborting pipeline replication: peer="{Voter server-8ad71f1d-d5e4-6316-dee7-e50cf2cfe428 8ad71f1d-d5e4-6316-dee7-e50cf2cfe428}"
2026-04-23T00:16:23.553Z [INFO]  server-aedba1c2-728f-cb97-f543-91ac59167337: aborting pipeline replication: peer="{Voter server-edf42206-2278-6bb6-a87d-798f4be563c6 edf42206-2278-6bb6-a87d-798f4be563c6}"
--- PASS: TestRaft_ReJoinFollower (0.48s)
=== RUN   TestRaft_LeaderLeaseExpire
2026-04-23T00:16:23.554Z [INFO]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f4bb7bdf-9772-7e06-af10-db3151616b24 Address:f4bb7bdf-9772-7e06-af10-db3151616b24} {Suffrage:Voter ID:server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 Address:b5d0d83e-c7fe-b41f-6f3d-eab0191314d0}]"
2026-04-23T00:16:23.554Z [INFO]  server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f4bb7bdf-9772-7e06-af10-db3151616b24 Address:f4bb7bdf-9772-7e06-af10-db3151616b24} {Suffrage:Voter ID:server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 Address:b5d0d83e-c7fe-b41f-6f3d-eab0191314d0}]"
2026-04-23T00:16:23.554Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:23.554Z [INFO]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: entering follower state: follower="Node at f4bb7bdf-9772-7e06-af10-db3151616b24 [Follower]" leader-address= leader-id=
2026-04-23T00:16:23.554Z [INFO]  server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0: entering follower state: follower="Node at b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 [Follower]" leader-address= leader-id=
2026-04-23T00:16:23.610Z [WARN]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:23.610Z [INFO]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: entering candidate state: node="Node at f4bb7bdf-9772-7e06-af10-db3151616b24 [Candidate]" term=2
2026-04-23T00:16:23.611Z [INFO]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: election won: term=2 tally=2
2026-04-23T00:16:23.611Z [INFO]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: entering leader state: leader="Node at f4bb7bdf-9772-7e06-af10-db3151616b24 [Leader]"
2026-04-23T00:16:23.611Z [INFO]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: added peer, starting replication: peer=server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0
2026-04-23T00:16:23.611Z [INFO]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: pipelining replication: peer="{Voter server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 b5d0d83e-c7fe-b41f-6f3d-eab0191314d0}"
2026-04-23T00:16:23.716Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:23.61107836 +0000 UTC m=+100.650679716 (1 nodes), 56.099859ms from start of poll, 56.304378ms from cluster start. Timeout at 2026-04-23 00:16:23.716081408 +0000 UTC m=+100.755682764, 105.003048ms after stability
2026-04-23T00:16:23.722Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:23.828Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:23.722555691 +0000 UTC m=+100.762157048 (1 nodes), 1.719µs from start of poll, 167.78171ms from cluster start. Timeout at 2026-04-23 00:16:23.827578376 +0000 UTC m=+100.867179730, 105.022682ms after stability
    raft_test.go:1799: [INFO] Disconnecting Node at b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 [Follower]
2026-04-23T00:16:23.830Z [ERROR] server-f4bb7bdf-9772-7e06-af10-db3151616b24: failed to pipeline appendEntries: peer="{Voter server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 b5d0d83e-c7fe-b41f-6f3d-eab0191314d0}" error="append pipeline closed"
2026-04-23T00:16:23.830Z [INFO]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: aborting pipeline replication: peer="{Voter server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 b5d0d83e-c7fe-b41f-6f3d-eab0191314d0}"
2026-04-23T00:16:23.838Z [ERROR] server-f4bb7bdf-9772-7e06-af10-db3151616b24: failed to heartbeat to: peer=b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 backoff time=10ms error="failed to connect to peer: b5d0d83e-c7fe-b41f-6f3d-eab0191314d0"
2026-04-23T00:16:23.839Z [ERROR] server-f4bb7bdf-9772-7e06-af10-db3151616b24: failed to appendEntries to: peer="{Voter server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 b5d0d83e-c7fe-b41f-6f3d-eab0191314d0}" error="failed to connect to peer: b5d0d83e-c7fe-b41f-6f3d-eab0191314d0"
2026-04-23T00:16:23.856Z [ERROR] server-f4bb7bdf-9772-7e06-af10-db3151616b24: failed to appendEntries to: peer="{Voter server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 b5d0d83e-c7fe-b41f-6f3d-eab0191314d0}" error="failed to connect to peer: b5d0d83e-c7fe-b41f-6f3d-eab0191314d0"
2026-04-23T00:16:23.856Z [ERROR] server-f4bb7bdf-9772-7e06-af10-db3151616b24: failed to heartbeat to: peer=b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 backoff time=10ms error="failed to connect to peer: b5d0d83e-c7fe-b41f-6f3d-eab0191314d0"
2026-04-23T00:16:23.872Z [ERROR] server-f4bb7bdf-9772-7e06-af10-db3151616b24: failed to appendEntries to: peer="{Voter server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 b5d0d83e-c7fe-b41f-6f3d-eab0191314d0}" error="failed to connect to peer: b5d0d83e-c7fe-b41f-6f3d-eab0191314d0"
2026-04-23T00:16:23.874Z [ERROR] server-f4bb7bdf-9772-7e06-af10-db3151616b24: failed to heartbeat to: peer=b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 backoff time=10ms error="failed to connect to peer: b5d0d83e-c7fe-b41f-6f3d-eab0191314d0"
2026-04-23T00:16:23.878Z [WARN]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: failed to contact: server-id=server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 time=50.377683ms
2026-04-23T00:16:23.878Z [WARN]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: failed to contact quorum of nodes, stepping down
2026-04-23T00:16:23.878Z [INFO]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: entering follower state: follower="Node at f4bb7bdf-9772-7e06-af10-db3151616b24 [Follower]" leader-address= leader-id=
2026-04-23T00:16:23.878Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:23.942Z [WARN]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:23.942Z [INFO]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: entering candidate state: node="Node at f4bb7bdf-9772-7e06-af10-db3151616b24 [Candidate]" term=3
2026-04-23T00:16:23.942Z [ERROR] server-f4bb7bdf-9772-7e06-af10-db3151616b24: failed to make requestVote RPC: target="{Voter server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 b5d0d83e-c7fe-b41f-6f3d-eab0191314d0}" error="failed to connect to peer: b5d0d83e-c7fe-b41f-6f3d-eab0191314d0" term=3
2026-04-23T00:16:23.960Z [WARN]  server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0: heartbeat timeout reached, starting election: last-leader-addr=f4bb7bdf-9772-7e06-af10-db3151616b24 last-leader-id=server-f4bb7bdf-9772-7e06-af10-db3151616b24
2026-04-23T00:16:23.960Z [INFO]  server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0: entering candidate state: node="Node at b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 [Candidate]" term=3
2026-04-23T00:16:23.960Z [ERROR] server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0: failed to make requestVote RPC: target="{Voter server-f4bb7bdf-9772-7e06-af10-db3151616b24 f4bb7bdf-9772-7e06-af10-db3151616b24}" error="failed to connect to peer: f4bb7bdf-9772-7e06-af10-db3151616b24" term=3
2026-04-23T00:16:24.008Z [WARN]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: Election timeout reached, restarting election
2026-04-23T00:16:24.008Z [INFO]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: entering candidate state: node="Node at f4bb7bdf-9772-7e06-af10-db3151616b24 [Candidate]" term=4
2026-04-23T00:16:24.008Z [ERROR] server-f4bb7bdf-9772-7e06-af10-db3151616b24: failed to make requestVote RPC: target="{Voter server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 b5d0d83e-c7fe-b41f-6f3d-eab0191314d0}" error="failed to connect to peer: b5d0d83e-c7fe-b41f-6f3d-eab0191314d0" term=4
2026-04-23T00:16:24.024Z [WARN]  server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0: Election timeout reached, restarting election
2026-04-23T00:16:24.024Z [INFO]  server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0: entering candidate state: node="Node at b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 [Candidate]" term=4
2026-04-23T00:16:24.024Z [ERROR] server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0: failed to make requestVote RPC: target="{Voter server-f4bb7bdf-9772-7e06-af10-db3151616b24 f4bb7bdf-9772-7e06-af10-db3151616b24}" error="failed to connect to peer: f4bb7bdf-9772-7e06-af10-db3151616b24" term=4
2026-04-23T00:16:24.065Z [WARN]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: Election timeout reached, restarting election
2026-04-23T00:16:24.065Z [INFO]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: entering candidate state: node="Node at f4bb7bdf-9772-7e06-af10-db3151616b24 [Candidate]" term=5
2026-04-23T00:16:24.065Z [ERROR] server-f4bb7bdf-9772-7e06-af10-db3151616b24: failed to make requestVote RPC: target="{Voter server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 b5d0d83e-c7fe-b41f-6f3d-eab0191314d0}" error="failed to connect to peer: b5d0d83e-c7fe-b41f-6f3d-eab0191314d0" term=5
2026-04-23T00:16:24.067Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:23.960932324 +0000 UTC m=+101.000533677 (0 nodes), 82.212547ms from start of poll, 406.158339ms from cluster start. Timeout at 2026-04-23 00:16:24.065935313 +0000 UTC m=+101.105536670, 105.002993ms after stability
2026-04-23T00:16:24.122Z [WARN]  server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0: Election timeout reached, restarting election
2026-04-23T00:16:24.122Z [INFO]  server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0: entering candidate state: node="Node at b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 [Candidate]" term=5
2026-04-23T00:16:24.122Z [ERROR] server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0: failed to make requestVote RPC: target="{Voter server-f4bb7bdf-9772-7e06-af10-db3151616b24 f4bb7bdf-9772-7e06-af10-db3151616b24}" error="failed to connect to peer: f4bb7bdf-9772-7e06-af10-db3151616b24" term=5
2026-04-23T00:16:24.134Z [WARN]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: Election timeout reached, restarting election
2026-04-23T00:16:24.134Z [INFO]  server-f4bb7bdf-9772-7e06-af10-db3151616b24: entering candidate state: node="Node at f4bb7bdf-9772-7e06-af10-db3151616b24 [Candidate]" term=6
2026-04-23T00:16:24.134Z [ERROR] server-f4bb7bdf-9772-7e06-af10-db3151616b24: failed to make requestVote RPC: target="{Voter server-b5d0d83e-c7fe-b41f-6f3d-eab0191314d0 b5d0d83e-c7fe-b41f-6f3d-eab0191314d0}" error="failed to connect to peer: b5d0d83e-c7fe-b41f-6f3d-eab0191314d0" term=6
--- PASS: TestRaft_LeaderLeaseExpire (0.62s)
=== RUN   TestRaft_Barrier
2026-04-23T00:16:24.173Z [INFO]  server-cb219ef7-dd5f-e841-0041-abdeb1690b9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb219ef7-dd5f-e841-0041-abdeb1690b9f Address:cb219ef7-dd5f-e841-0041-abdeb1690b9f} {Suffrage:Voter ID:server-eb4d95c7-a2a0-2b93-57ee-ac2a177905b4 Address:eb4d95c7-a2a0-2b93-57ee-ac2a177905b4} {Suffrage:Voter ID:server-2b1a89eb-ba44-b3d4-53d2-74fd744b8910 Address:2b1a89eb-ba44-b3d4-53d2-74fd744b8910}]"
2026-04-23T00:16:24.173Z [INFO]  server-cb219ef7-dd5f-e841-0041-abdeb1690b9f: entering follower state: follower="Node at cb219ef7-dd5f-e841-0041-abdeb1690b9f [Follower]" leader-address= leader-id=
2026-04-23T00:16:24.173Z [INFO]  server-eb4d95c7-a2a0-2b93-57ee-ac2a177905b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb219ef7-dd5f-e841-0041-abdeb1690b9f Address:cb219ef7-dd5f-e841-0041-abdeb1690b9f} {Suffrage:Voter ID:server-eb4d95c7-a2a0-2b93-57ee-ac2a177905b4 Address:eb4d95c7-a2a0-2b93-57ee-ac2a177905b4} {Suffrage:Voter ID:server-2b1a89eb-ba44-b3d4-53d2-74fd744b8910 Address:2b1a89eb-ba44-b3d4-53d2-74fd744b8910}]"
2026-04-23T00:16:24.174Z [INFO]  server-eb4d95c7-a2a0-2b93-57ee-ac2a177905b4: entering follower state: follower="Node at eb4d95c7-a2a0-2b93-57ee-ac2a177905b4 [Follower]" leader-address= leader-id=
2026-04-23T00:16:24.174Z [INFO]  server-2b1a89eb-ba44-b3d4-53d2-74fd744b8910: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb219ef7-dd5f-e841-0041-abdeb1690b9f Address:cb219ef7-dd5f-e841-0041-abdeb1690b9f} {Suffrage:Voter ID:server-eb4d95c7-a2a0-2b93-57ee-ac2a177905b4 Address:eb4d95c7-a2a0-2b93-57ee-ac2a177905b4} {Suffrage:Voter ID:server-2b1a89eb-ba44-b3d4-53d2-74fd744b8910 Address:2b1a89eb-ba44-b3d4-53d2-74fd744b8910}]"
2026-04-23T00:16:24.174Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:24.174Z [INFO]  server-2b1a89eb-ba44-b3d4-53d2-74fd744b8910: entering follower state: follower="Node at 2b1a89eb-ba44-b3d4-53d2-74fd744b8910 [Follower]" leader-address= leader-id=
2026-04-23T00:16:24.229Z [WARN]  server-eb4d95c7-a2a0-2b93-57ee-ac2a177905b4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:24.229Z [INFO]  server-eb4d95c7-a2a0-2b93-57ee-ac2a177905b4: entering candidate state: node="Node at eb4d95c7-a2a0-2b93-57ee-ac2a177905b4 [Candidate]" term=2
2026-04-23T00:16:24.229Z [INFO]  server-eb4d95c7-a2a0-2b93-57ee-ac2a177905b4: election won: term=2 tally=2
2026-04-23T00:16:24.229Z [INFO]  server-eb4d95c7-a2a0-2b93-57ee-ac2a177905b4: entering leader state: leader="Node at eb4d95c7-a2a0-2b93-57ee-ac2a177905b4 [Leader]"
2026-04-23T00:16:24.229Z [INFO]  server-eb4d95c7-a2a0-2b93-57ee-ac2a177905b4: added peer, starting replication: peer=server-cb219ef7-dd5f-e841-0041-abdeb1690b9f
2026-04-23T00:16:24.229Z [INFO]  server-eb4d95c7-a2a0-2b93-57ee-ac2a177905b4: added peer, starting replication: peer=server-2b1a89eb-ba44-b3d4-53d2-74fd744b8910
2026-04-23T00:16:24.229Z [INFO]  server-eb4d95c7-a2a0-2b93-57ee-ac2a177905b4: pipelining replication: peer="{Voter server-cb219ef7-dd5f-e841-0041-abdeb1690b9f cb219ef7-dd5f-e841-0041-abdeb1690b9f}"
2026-04-23T00:16:24.229Z [INFO]  server-eb4d95c7-a2a0-2b93-57ee-ac2a177905b4: pipelining replication: peer="{Voter server-2b1a89eb-ba44-b3d4-53d2-74fd744b8910 2b1a89eb-ba44-b3d4-53d2-74fd744b8910}"
2026-04-23T00:16:24.335Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:24.229834163 +0000 UTC m=+101.269435528 (1 nodes), 55.728144ms from start of poll, 56.091746ms from cluster start. Timeout at 2026-04-23 00:16:24.334844797 +0000 UTC m=+101.374446154, 105.010626ms after stability
2026-04-23T00:16:24.337Z [INFO]  server-eb4d95c7-a2a0-2b93-57ee-ac2a177905b4: aborting pipeline replication: peer="{Voter server-2b1a89eb-ba44-b3d4-53d2-74fd744b8910 2b1a89eb-ba44-b3d4-53d2-74fd744b8910}"
2026-04-23T00:16:24.337Z [INFO]  server-eb4d95c7-a2a0-2b93-57ee-ac2a177905b4: aborting pipeline replication: peer="{Voter server-cb219ef7-dd5f-e841-0041-abdeb1690b9f cb219ef7-dd5f-e841-0041-abdeb1690b9f}"
--- PASS: TestRaft_Barrier (0.16s)
=== RUN   TestRaft_VerifyLeader
2026-04-23T00:16:24.338Z [INFO]  server-b0523c6c-41e8-ec5e-7f23-536e90e0b4d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b0523c6c-41e8-ec5e-7f23-536e90e0b4d1 Address:b0523c6c-41e8-ec5e-7f23-536e90e0b4d1} {Suffrage:Voter ID:server-c6f5b424-61dd-9cb5-c0b3-e04d579b131c Address:c6f5b424-61dd-9cb5-c0b3-e04d579b131c} {Suffrage:Voter ID:server-75fdb930-1df4-802b-6f9e-4c9b7c8ee66a Address:75fdb930-1df4-802b-6f9e-4c9b7c8ee66a}]"
2026-04-23T00:16:24.338Z [INFO]  server-b0523c6c-41e8-ec5e-7f23-536e90e0b4d1: entering follower state: follower="Node at b0523c6c-41e8-ec5e-7f23-536e90e0b4d1 [Follower]" leader-address= leader-id=
2026-04-23T00:16:24.338Z [INFO]  server-c6f5b424-61dd-9cb5-c0b3-e04d579b131c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b0523c6c-41e8-ec5e-7f23-536e90e0b4d1 Address:b0523c6c-41e8-ec5e-7f23-536e90e0b4d1} {Suffrage:Voter ID:server-c6f5b424-61dd-9cb5-c0b3-e04d579b131c Address:c6f5b424-61dd-9cb5-c0b3-e04d579b131c} {Suffrage:Voter ID:server-75fdb930-1df4-802b-6f9e-4c9b7c8ee66a Address:75fdb930-1df4-802b-6f9e-4c9b7c8ee66a}]"
2026-04-23T00:16:24.338Z [INFO]  server-c6f5b424-61dd-9cb5-c0b3-e04d579b131c: entering follower state: follower="Node at c6f5b424-61dd-9cb5-c0b3-e04d579b131c [Follower]" leader-address= leader-id=
2026-04-23T00:16:24.338Z [INFO]  server-75fdb930-1df4-802b-6f9e-4c9b7c8ee66a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b0523c6c-41e8-ec5e-7f23-536e90e0b4d1 Address:b0523c6c-41e8-ec5e-7f23-536e90e0b4d1} {Suffrage:Voter ID:server-c6f5b424-61dd-9cb5-c0b3-e04d579b131c Address:c6f5b424-61dd-9cb5-c0b3-e04d579b131c} {Suffrage:Voter ID:server-75fdb930-1df4-802b-6f9e-4c9b7c8ee66a Address:75fdb930-1df4-802b-6f9e-4c9b7c8ee66a}]"
2026-04-23T00:16:24.338Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:24.338Z [INFO]  server-75fdb930-1df4-802b-6f9e-4c9b7c8ee66a: entering follower state: follower="Node at 75fdb930-1df4-802b-6f9e-4c9b7c8ee66a [Follower]" leader-address= leader-id=
2026-04-23T00:16:24.417Z [WARN]  server-b0523c6c-41e8-ec5e-7f23-536e90e0b4d1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:24.417Z [INFO]  server-b0523c6c-41e8-ec5e-7f23-536e90e0b4d1: entering candidate state: node="Node at b0523c6c-41e8-ec5e-7f23-536e90e0b4d1 [Candidate]" term=2
2026-04-23T00:16:24.417Z [INFO]  server-b0523c6c-41e8-ec5e-7f23-536e90e0b4d1: election won: term=2 tally=2
2026-04-23T00:16:24.417Z [INFO]  server-b0523c6c-41e8-ec5e-7f23-536e90e0b4d1: entering leader state: leader="Node at b0523c6c-41e8-ec5e-7f23-536e90e0b4d1 [Leader]"
2026-04-23T00:16:24.417Z [INFO]  server-b0523c6c-41e8-ec5e-7f23-536e90e0b4d1: added peer, starting replication: peer=server-c6f5b424-61dd-9cb5-c0b3-e04d579b131c
2026-04-23T00:16:24.417Z [INFO]  server-b0523c6c-41e8-ec5e-7f23-536e90e0b4d1: added peer, starting replication: peer=server-75fdb930-1df4-802b-6f9e-4c9b7c8ee66a
2026-04-23T00:16:24.417Z [INFO]  server-b0523c6c-41e8-ec5e-7f23-536e90e0b4d1: pipelining replication: peer="{Voter server-c6f5b424-61dd-9cb5-c0b3-e04d579b131c c6f5b424-61dd-9cb5-c0b3-e04d579b131c}"
2026-04-23T00:16:24.418Z [INFO]  server-b0523c6c-41e8-ec5e-7f23-536e90e0b4d1: pipelining replication: peer="{Voter server-75fdb930-1df4-802b-6f9e-4c9b7c8ee66a 75fdb930-1df4-802b-6f9e-4c9b7c8ee66a}"
2026-04-23T00:16:24.523Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:24.417298209 +0000 UTC m=+101.456899570 (1 nodes), 78.677492ms from start of poll, 79.002333ms from cluster start. Timeout at 2026-04-23 00:16:24.522306041 +0000 UTC m=+101.561907394, 105.007824ms after stability
2026-04-23T00:16:24.523Z [INFO]  server-b0523c6c-41e8-ec5e-7f23-536e90e0b4d1: aborting pipeline replication: peer="{Voter server-75fdb930-1df4-802b-6f9e-4c9b7c8ee66a 75fdb930-1df4-802b-6f9e-4c9b7c8ee66a}"
2026-04-23T00:16:24.523Z [INFO]  server-b0523c6c-41e8-ec5e-7f23-536e90e0b4d1: aborting pipeline replication: peer="{Voter server-c6f5b424-61dd-9cb5-c0b3-e04d579b131c c6f5b424-61dd-9cb5-c0b3-e04d579b131c}"
--- PASS: TestRaft_VerifyLeader (0.19s)
=== RUN   TestRaft_VerifyLeader_Single
2026-04-23T00:16:24.524Z [INFO]  server-8725e2d7-c35d-bca1-55b4-a0251f2d4917: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8725e2d7-c35d-bca1-55b4-a0251f2d4917 Address:8725e2d7-c35d-bca1-55b4-a0251f2d4917}]"
2026-04-23T00:16:24.524Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:24.524Z [INFO]  server-8725e2d7-c35d-bca1-55b4-a0251f2d4917: entering follower state: follower="Node at 8725e2d7-c35d-bca1-55b4-a0251f2d4917 [Follower]" leader-address= leader-id=
2026-04-23T00:16:24.617Z [WARN]  server-8725e2d7-c35d-bca1-55b4-a0251f2d4917: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:24.617Z [INFO]  server-8725e2d7-c35d-bca1-55b4-a0251f2d4917: entering candidate state: node="Node at 8725e2d7-c35d-bca1-55b4-a0251f2d4917 [Candidate]" term=2
2026-04-23T00:16:24.617Z [INFO]  server-8725e2d7-c35d-bca1-55b4-a0251f2d4917: election won: term=2 tally=1
2026-04-23T00:16:24.617Z [INFO]  server-8725e2d7-c35d-bca1-55b4-a0251f2d4917: entering leader state: leader="Node at 8725e2d7-c35d-bca1-55b4-a0251f2d4917 [Leader]"
2026-04-23T00:16:24.722Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:24.617350615 +0000 UTC m=+101.656951972 (1 nodes), 92.649246ms from start of poll, 92.783386ms from cluster start. Timeout at 2026-04-23 00:16:24.722354287 +0000 UTC m=+101.761955659, 105.003687ms after stability
--- PASS: TestRaft_VerifyLeader_Single (0.20s)
=== RUN   TestRaft_VerifyLeader_Fail
2026-04-23T00:16:24.723Z [INFO]  server-22a43c3e-b78e-bb3e-59a3-3c9250558366: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22a43c3e-b78e-bb3e-59a3-3c9250558366 Address:22a43c3e-b78e-bb3e-59a3-3c9250558366} {Suffrage:Voter ID:server-7fd882bc-8fc2-9950-9b04-00e37a8ac746 Address:7fd882bc-8fc2-9950-9b04-00e37a8ac746}]"
2026-04-23T00:16:24.723Z [INFO]  server-7fd882bc-8fc2-9950-9b04-00e37a8ac746: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22a43c3e-b78e-bb3e-59a3-3c9250558366 Address:22a43c3e-b78e-bb3e-59a3-3c9250558366} {Suffrage:Voter ID:server-7fd882bc-8fc2-9950-9b04-00e37a8ac746 Address:7fd882bc-8fc2-9950-9b04-00e37a8ac746}]"
2026-04-23T00:16:24.723Z [INFO]  server-22a43c3e-b78e-bb3e-59a3-3c9250558366: entering follower state: follower="Node at 22a43c3e-b78e-bb3e-59a3-3c9250558366 [Follower]" leader-address= leader-id=
2026-04-23T00:16:24.723Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:24.723Z [INFO]  server-7fd882bc-8fc2-9950-9b04-00e37a8ac746: entering follower state: follower="Node at 7fd882bc-8fc2-9950-9b04-00e37a8ac746 [Follower]" leader-address= leader-id=
2026-04-23T00:16:24.800Z [WARN]  server-22a43c3e-b78e-bb3e-59a3-3c9250558366: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:24.800Z [INFO]  server-22a43c3e-b78e-bb3e-59a3-3c9250558366: entering candidate state: node="Node at 22a43c3e-b78e-bb3e-59a3-3c9250558366 [Candidate]" term=2
2026-04-23T00:16:24.800Z [INFO]  server-22a43c3e-b78e-bb3e-59a3-3c9250558366: election won: term=2 tally=2
2026-04-23T00:16:24.800Z [INFO]  server-22a43c3e-b78e-bb3e-59a3-3c9250558366: entering leader state: leader="Node at 22a43c3e-b78e-bb3e-59a3-3c9250558366 [Leader]"
2026-04-23T00:16:24.800Z [INFO]  server-22a43c3e-b78e-bb3e-59a3-3c9250558366: added peer, starting replication: peer=server-7fd882bc-8fc2-9950-9b04-00e37a8ac746
2026-04-23T00:16:24.800Z [INFO]  server-22a43c3e-b78e-bb3e-59a3-3c9250558366: pipelining replication: peer="{Voter server-7fd882bc-8fc2-9950-9b04-00e37a8ac746 7fd882bc-8fc2-9950-9b04-00e37a8ac746}"
2026-04-23T00:16:24.905Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:24.800414715 +0000 UTC m=+101.840016072 (1 nodes), 77.116059ms from start of poll, 77.322019ms from cluster start. Timeout at 2026-04-23 00:16:24.905417563 +0000 UTC m=+101.945018920, 105.002848ms after stability
2026-04-23T00:16:24.905Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:25.011Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:24.905788538 +0000 UTC m=+101.945389898 (1 nodes), 1.325µs from start of poll, 182.695845ms from cluster start. Timeout at 2026-04-23 00:16:25.010817952 +0000 UTC m=+102.050419313, 105.029415ms after stability
2026-04-23T00:16:25.016Z [ERROR] server-22a43c3e-b78e-bb3e-59a3-3c9250558366: peer has newer term, stopping replication: peer="{Voter server-7fd882bc-8fc2-9950-9b04-00e37a8ac746 7fd882bc-8fc2-9950-9b04-00e37a8ac746}"
2026-04-23T00:16:25.016Z [INFO]  server-22a43c3e-b78e-bb3e-59a3-3c9250558366: aborting pipeline replication: peer="{Voter server-7fd882bc-8fc2-9950-9b04-00e37a8ac746 7fd882bc-8fc2-9950-9b04-00e37a8ac746}"
2026-04-23T00:16:25.016Z [INFO]  server-22a43c3e-b78e-bb3e-59a3-3c9250558366: entering follower state: follower="Node at 22a43c3e-b78e-bb3e-59a3-3c9250558366 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VerifyLeader_Fail (0.29s)
=== RUN   TestRaft_VerifyLeader_PartialConnect
2026-04-23T00:16:25.017Z [INFO]  server-5f09dcc2-f198-a932-8823-254cc197351e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f09dcc2-f198-a932-8823-254cc197351e Address:5f09dcc2-f198-a932-8823-254cc197351e} {Suffrage:Voter ID:server-a07c0edb-20a6-57ba-e0f0-46c7d6967cdd Address:a07c0edb-20a6-57ba-e0f0-46c7d6967cdd} {Suffrage:Voter ID:server-a4238f92-cc01-9498-a5c5-17fc6040ba3d Address:a4238f92-cc01-9498-a5c5-17fc6040ba3d}]"
2026-04-23T00:16:25.017Z [INFO]  server-5f09dcc2-f198-a932-8823-254cc197351e: entering follower state: follower="Node at 5f09dcc2-f198-a932-8823-254cc197351e [Follower]" leader-address= leader-id=
2026-04-23T00:16:25.017Z [INFO]  server-a07c0edb-20a6-57ba-e0f0-46c7d6967cdd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f09dcc2-f198-a932-8823-254cc197351e Address:5f09dcc2-f198-a932-8823-254cc197351e} {Suffrage:Voter ID:server-a07c0edb-20a6-57ba-e0f0-46c7d6967cdd Address:a07c0edb-20a6-57ba-e0f0-46c7d6967cdd} {Suffrage:Voter ID:server-a4238f92-cc01-9498-a5c5-17fc6040ba3d Address:a4238f92-cc01-9498-a5c5-17fc6040ba3d}]"
2026-04-23T00:16:25.017Z [INFO]  server-a07c0edb-20a6-57ba-e0f0-46c7d6967cdd: entering follower state: follower="Node at a07c0edb-20a6-57ba-e0f0-46c7d6967cdd [Follower]" leader-address= leader-id=
2026-04-23T00:16:25.017Z [INFO]  server-a4238f92-cc01-9498-a5c5-17fc6040ba3d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f09dcc2-f198-a932-8823-254cc197351e Address:5f09dcc2-f198-a932-8823-254cc197351e} {Suffrage:Voter ID:server-a07c0edb-20a6-57ba-e0f0-46c7d6967cdd Address:a07c0edb-20a6-57ba-e0f0-46c7d6967cdd} {Suffrage:Voter ID:server-a4238f92-cc01-9498-a5c5-17fc6040ba3d Address:a4238f92-cc01-9498-a5c5-17fc6040ba3d}]"
2026-04-23T00:16:25.017Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:25.017Z [INFO]  server-a4238f92-cc01-9498-a5c5-17fc6040ba3d: entering follower state: follower="Node at a4238f92-cc01-9498-a5c5-17fc6040ba3d [Follower]" leader-address= leader-id=
2026-04-23T00:16:25.095Z [WARN]  server-5f09dcc2-f198-a932-8823-254cc197351e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:25.095Z [INFO]  server-5f09dcc2-f198-a932-8823-254cc197351e: entering candidate state: node="Node at 5f09dcc2-f198-a932-8823-254cc197351e [Candidate]" term=2
2026-04-23T00:16:25.095Z [INFO]  server-5f09dcc2-f198-a932-8823-254cc197351e: election won: term=2 tally=2
2026-04-23T00:16:25.095Z [INFO]  server-5f09dcc2-f198-a932-8823-254cc197351e: entering leader state: leader="Node at 5f09dcc2-f198-a932-8823-254cc197351e [Leader]"
2026-04-23T00:16:25.095Z [INFO]  server-5f09dcc2-f198-a932-8823-254cc197351e: added peer, starting replication: peer=server-a07c0edb-20a6-57ba-e0f0-46c7d6967cdd
2026-04-23T00:16:25.095Z [INFO]  server-5f09dcc2-f198-a932-8823-254cc197351e: added peer, starting replication: peer=server-a4238f92-cc01-9498-a5c5-17fc6040ba3d
2026-04-23T00:16:25.095Z [INFO]  server-5f09dcc2-f198-a932-8823-254cc197351e: pipelining replication: peer="{Voter server-a4238f92-cc01-9498-a5c5-17fc6040ba3d a4238f92-cc01-9498-a5c5-17fc6040ba3d}"
2026-04-23T00:16:25.095Z [INFO]  server-5f09dcc2-f198-a932-8823-254cc197351e: pipelining replication: peer="{Voter server-a07c0edb-20a6-57ba-e0f0-46c7d6967cdd a07c0edb-20a6-57ba-e0f0-46c7d6967cdd}"
2026-04-23T00:16:25.200Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:25.095496961 +0000 UTC m=+102.135098329 (1 nodes), 77.978478ms from start of poll, 78.278358ms from cluster start. Timeout at 2026-04-23 00:16:25.200500237 +0000 UTC m=+102.240101589, 105.00326ms after stability
2026-04-23T00:16:25.206Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:25.312Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:25.206638338 +0000 UTC m=+102.246239704 (2 nodes), 1.7µs from start of poll, 189.419733ms from cluster start. Timeout at 2026-04-23 00:16:25.311653996 +0000 UTC m=+102.351255373, 105.015669ms after stability
    raft_test.go:1968: [INFO] Disconnecting Node at a07c0edb-20a6-57ba-e0f0-46c7d6967cdd [Follower]
2026-04-23T00:16:25.312Z [ERROR] server-5f09dcc2-f198-a932-8823-254cc197351e: failed to heartbeat to: peer=a07c0edb-20a6-57ba-e0f0-46c7d6967cdd backoff time=10ms error="failed to connect to peer: a07c0edb-20a6-57ba-e0f0-46c7d6967cdd"
2026-04-23T00:16:25.312Z [INFO]  server-5f09dcc2-f198-a932-8823-254cc197351e: aborting pipeline replication: peer="{Voter server-a07c0edb-20a6-57ba-e0f0-46c7d6967cdd a07c0edb-20a6-57ba-e0f0-46c7d6967cdd}"
2026-04-23T00:16:25.312Z [INFO]  server-5f09dcc2-f198-a932-8823-254cc197351e: aborting pipeline replication: peer="{Voter server-a4238f92-cc01-9498-a5c5-17fc6040ba3d a4238f92-cc01-9498-a5c5-17fc6040ba3d}"
--- PASS: TestRaft_VerifyLeader_PartialConnect (0.30s)
=== RUN   TestRaft_NotifyCh
2026-04-23T00:16:25.314Z [INFO]  server-6b7046af-f404-3967-54c5-345f05743e46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b7046af-f404-3967-54c5-345f05743e46 Address:6b7046af-f404-3967-54c5-345f05743e46}]"
2026-04-23T00:16:25.314Z [INFO]  server-6b7046af-f404-3967-54c5-345f05743e46: entering follower state: follower="Node at 6b7046af-f404-3967-54c5-345f05743e46 [Follower]" leader-address= leader-id=
2026-04-23T00:16:25.413Z [WARN]  server-6b7046af-f404-3967-54c5-345f05743e46: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:25.413Z [INFO]  server-6b7046af-f404-3967-54c5-345f05743e46: entering candidate state: node="Node at 6b7046af-f404-3967-54c5-345f05743e46 [Candidate]" term=2
2026-04-23T00:16:25.413Z [INFO]  server-6b7046af-f404-3967-54c5-345f05743e46: election won: term=2 tally=1
2026-04-23T00:16:25.413Z [INFO]  server-6b7046af-f404-3967-54c5-345f05743e46: entering leader state: leader="Node at 6b7046af-f404-3967-54c5-345f05743e46 [Leader]"
--- PASS: TestRaft_NotifyCh (0.10s)
=== RUN   TestRaft_AppendEntry
2026-04-23T00:16:25.414Z [INFO]  server-26558b9c-b86a-2f34-7313-b16a8527bb82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26558b9c-b86a-2f34-7313-b16a8527bb82 Address:26558b9c-b86a-2f34-7313-b16a8527bb82} {Suffrage:Voter ID:server-c8ffbc74-44ce-2f02-b381-de74f69e97a5 Address:c8ffbc74-44ce-2f02-b381-de74f69e97a5} {Suffrage:Voter ID:server-061c8f8b-88f7-ac7b-533a-91b5aae32938 Address:061c8f8b-88f7-ac7b-533a-91b5aae32938}]"
2026-04-23T00:16:25.414Z [INFO]  server-26558b9c-b86a-2f34-7313-b16a8527bb82: entering follower state: follower="Node at 26558b9c-b86a-2f34-7313-b16a8527bb82 [Follower]" leader-address= leader-id=
2026-04-23T00:16:25.414Z [INFO]  server-c8ffbc74-44ce-2f02-b381-de74f69e97a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26558b9c-b86a-2f34-7313-b16a8527bb82 Address:26558b9c-b86a-2f34-7313-b16a8527bb82} {Suffrage:Voter ID:server-c8ffbc74-44ce-2f02-b381-de74f69e97a5 Address:c8ffbc74-44ce-2f02-b381-de74f69e97a5} {Suffrage:Voter ID:server-061c8f8b-88f7-ac7b-533a-91b5aae32938 Address:061c8f8b-88f7-ac7b-533a-91b5aae32938}]"
2026-04-23T00:16:25.414Z [INFO]  server-c8ffbc74-44ce-2f02-b381-de74f69e97a5: entering follower state: follower="Node at c8ffbc74-44ce-2f02-b381-de74f69e97a5 [Follower]" leader-address= leader-id=
2026-04-23T00:16:25.414Z [INFO]  server-061c8f8b-88f7-ac7b-533a-91b5aae32938: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26558b9c-b86a-2f34-7313-b16a8527bb82 Address:26558b9c-b86a-2f34-7313-b16a8527bb82} {Suffrage:Voter ID:server-c8ffbc74-44ce-2f02-b381-de74f69e97a5 Address:c8ffbc74-44ce-2f02-b381-de74f69e97a5} {Suffrage:Voter ID:server-061c8f8b-88f7-ac7b-533a-91b5aae32938 Address:061c8f8b-88f7-ac7b-533a-91b5aae32938}]"
2026-04-23T00:16:25.414Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:25.414Z [INFO]  server-061c8f8b-88f7-ac7b-533a-91b5aae32938: entering follower state: follower="Node at 061c8f8b-88f7-ac7b-533a-91b5aae32938 [Follower]" leader-address= leader-id=
2026-04-23T00:16:25.481Z [WARN]  server-26558b9c-b86a-2f34-7313-b16a8527bb82: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:25.481Z [INFO]  server-26558b9c-b86a-2f34-7313-b16a8527bb82: entering candidate state: node="Node at 26558b9c-b86a-2f34-7313-b16a8527bb82 [Candidate]" term=2
2026-04-23T00:16:25.481Z [INFO]  server-26558b9c-b86a-2f34-7313-b16a8527bb82: election won: term=2 tally=2
2026-04-23T00:16:25.481Z [INFO]  server-26558b9c-b86a-2f34-7313-b16a8527bb82: entering leader state: leader="Node at 26558b9c-b86a-2f34-7313-b16a8527bb82 [Leader]"
2026-04-23T00:16:25.481Z [INFO]  server-26558b9c-b86a-2f34-7313-b16a8527bb82: added peer, starting replication: peer=server-c8ffbc74-44ce-2f02-b381-de74f69e97a5
2026-04-23T00:16:25.481Z [INFO]  server-26558b9c-b86a-2f34-7313-b16a8527bb82: added peer, starting replication: peer=server-061c8f8b-88f7-ac7b-533a-91b5aae32938
2026-04-23T00:16:25.481Z [INFO]  server-26558b9c-b86a-2f34-7313-b16a8527bb82: pipelining replication: peer="{Voter server-c8ffbc74-44ce-2f02-b381-de74f69e97a5 c8ffbc74-44ce-2f02-b381-de74f69e97a5}"
2026-04-23T00:16:25.481Z [INFO]  server-26558b9c-b86a-2f34-7313-b16a8527bb82: pipelining replication: peer="{Voter server-061c8f8b-88f7-ac7b-533a-91b5aae32938 061c8f8b-88f7-ac7b-533a-91b5aae32938}"
2026-04-23T00:16:25.586Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:25.481684025 +0000 UTC m=+102.521285397 (2 nodes), 66.849056ms from start of poll, 67.198944ms from cluster start. Timeout at 2026-04-23 00:16:25.586713484 +0000 UTC m=+102.626314848, 105.029451ms after stability
2026-04-23T00:16:25.586Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:25.692Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:25.586809995 +0000 UTC m=+102.626411354 (1 nodes), 1.373µs from start of poll, 172.324901ms from cluster start. Timeout at 2026-04-23 00:16:25.691831802 +0000 UTC m=+102.731433156, 105.021802ms after stability
2026-04-23T00:16:25.692Z [WARN]  server-c8ffbc74-44ce-2f02-b381-de74f69e97a5: clearing log suffix: from=1 to=2
2026-04-23T00:16:25.692Z [WARN]  server-c8ffbc74-44ce-2f02-b381-de74f69e97a5: skipping application of old log: index=1
2026-04-23T00:16:25.692Z [WARN]  server-c8ffbc74-44ce-2f02-b381-de74f69e97a5: skipping application of old log: index=1
2026-04-23T00:16:25.692Z [INFO]  server-26558b9c-b86a-2f34-7313-b16a8527bb82: aborting pipeline replication: peer="{Voter server-061c8f8b-88f7-ac7b-533a-91b5aae32938 061c8f8b-88f7-ac7b-533a-91b5aae32938}"
2026-04-23T00:16:25.692Z [INFO]  server-26558b9c-b86a-2f34-7313-b16a8527bb82: aborting pipeline replication: peer="{Voter server-c8ffbc74-44ce-2f02-b381-de74f69e97a5 c8ffbc74-44ce-2f02-b381-de74f69e97a5}"
--- PASS: TestRaft_AppendEntry (0.28s)
=== RUN   TestRaft_VotingGrant_WhenLeaderAvailable
2026-04-23T00:16:25.693Z [INFO]  server-4aa0532c-bdb2-f140-bf1f-00b75ed0ee0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4aa0532c-bdb2-f140-bf1f-00b75ed0ee0c Address:4aa0532c-bdb2-f140-bf1f-00b75ed0ee0c} {Suffrage:Voter ID:server-abdd2678-1732-1051-3f9d-5e0e1e49bd45 Address:abdd2678-1732-1051-3f9d-5e0e1e49bd45} {Suffrage:Voter ID:server-90789be2-2e4c-745a-5cd7-ae471d4c5c45 Address:90789be2-2e4c-745a-5cd7-ae471d4c5c45}]"
2026-04-23T00:16:25.693Z [INFO]  server-4aa0532c-bdb2-f140-bf1f-00b75ed0ee0c: entering follower state: follower="Node at 4aa0532c-bdb2-f140-bf1f-00b75ed0ee0c [Follower]" leader-address= leader-id=
2026-04-23T00:16:25.693Z [INFO]  server-abdd2678-1732-1051-3f9d-5e0e1e49bd45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4aa0532c-bdb2-f140-bf1f-00b75ed0ee0c Address:4aa0532c-bdb2-f140-bf1f-00b75ed0ee0c} {Suffrage:Voter ID:server-abdd2678-1732-1051-3f9d-5e0e1e49bd45 Address:abdd2678-1732-1051-3f9d-5e0e1e49bd45} {Suffrage:Voter ID:server-90789be2-2e4c-745a-5cd7-ae471d4c5c45 Address:90789be2-2e4c-745a-5cd7-ae471d4c5c45}]"
2026-04-23T00:16:25.693Z [INFO]  server-abdd2678-1732-1051-3f9d-5e0e1e49bd45: entering follower state: follower="Node at abdd2678-1732-1051-3f9d-5e0e1e49bd45 [Follower]" leader-address= leader-id=
2026-04-23T00:16:25.693Z [INFO]  server-90789be2-2e4c-745a-5cd7-ae471d4c5c45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4aa0532c-bdb2-f140-bf1f-00b75ed0ee0c Address:4aa0532c-bdb2-f140-bf1f-00b75ed0ee0c} {Suffrage:Voter ID:server-abdd2678-1732-1051-3f9d-5e0e1e49bd45 Address:abdd2678-1732-1051-3f9d-5e0e1e49bd45} {Suffrage:Voter ID:server-90789be2-2e4c-745a-5cd7-ae471d4c5c45 Address:90789be2-2e4c-745a-5cd7-ae471d4c5c45}]"
2026-04-23T00:16:25.694Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:25.694Z [INFO]  server-90789be2-2e4c-745a-5cd7-ae471d4c5c45: entering follower state: follower="Node at 90789be2-2e4c-745a-5cd7-ae471d4c5c45 [Follower]" leader-address= leader-id=
2026-04-23T00:16:25.774Z [WARN]  server-90789be2-2e4c-745a-5cd7-ae471d4c5c45: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:25.774Z [INFO]  server-90789be2-2e4c-745a-5cd7-ae471d4c5c45: entering candidate state: node="Node at 90789be2-2e4c-745a-5cd7-ae471d4c5c45 [Candidate]" term=2
2026-04-23T00:16:25.774Z [INFO]  server-90789be2-2e4c-745a-5cd7-ae471d4c5c45: election won: term=2 tally=2
2026-04-23T00:16:25.774Z [INFO]  server-90789be2-2e4c-745a-5cd7-ae471d4c5c45: entering leader state: leader="Node at 90789be2-2e4c-745a-5cd7-ae471d4c5c45 [Leader]"
2026-04-23T00:16:25.774Z [INFO]  server-90789be2-2e4c-745a-5cd7-ae471d4c5c45: added peer, starting replication: peer=server-4aa0532c-bdb2-f140-bf1f-00b75ed0ee0c
2026-04-23T00:16:25.774Z [INFO]  server-90789be2-2e4c-745a-5cd7-ae471d4c5c45: added peer, starting replication: peer=server-abdd2678-1732-1051-3f9d-5e0e1e49bd45
2026-04-23T00:16:25.774Z [INFO]  server-90789be2-2e4c-745a-5cd7-ae471d4c5c45: pipelining replication: peer="{Voter server-abdd2678-1732-1051-3f9d-5e0e1e49bd45 abdd2678-1732-1051-3f9d-5e0e1e49bd45}"
2026-04-23T00:16:25.774Z [INFO]  server-90789be2-2e4c-745a-5cd7-ae471d4c5c45: pipelining replication: peer="{Voter server-4aa0532c-bdb2-f140-bf1f-00b75ed0ee0c 4aa0532c-bdb2-f140-bf1f-00b75ed0ee0c}"
2026-04-23T00:16:25.879Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:25.77456361 +0000 UTC m=+102.814164966 (2 nodes), 80.535852ms from start of poll, 80.893953ms from cluster start. Timeout at 2026-04-23 00:16:25.879567467 +0000 UTC m=+102.919168822, 105.003856ms after stability
2026-04-23T00:16:25.879Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:25.985Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:25.879937678 +0000 UTC m=+102.919539034 (1 nodes), 544ns from start of poll, 186.268021ms from cluster start. Timeout at 2026-04-23 00:16:25.984972311 +0000 UTC m=+103.024573663, 105.034629ms after stability
2026-04-23T00:16:25.985Z [WARN]  server-abdd2678-1732-1051-3f9d-5e0e1e49bd45: rejecting vote request since we have a leader: from=4aa0532c-bdb2-f140-bf1f-00b75ed0ee0c leader=90789be2-2e4c-745a-5cd7-ae471d4c5c45 leader-id=server-90789be2-2e4c-745a-5cd7-ae471d4c5c45
2026-04-23T00:16:25.985Z [INFO]  server-90789be2-2e4c-745a-5cd7-ae471d4c5c45: aborting pipeline replication: peer="{Voter server-abdd2678-1732-1051-3f9d-5e0e1e49bd45 abdd2678-1732-1051-3f9d-5e0e1e49bd45}"
2026-04-23T00:16:25.985Z [INFO]  server-90789be2-2e4c-745a-5cd7-ae471d4c5c45: aborting pipeline replication: peer="{Voter server-4aa0532c-bdb2-f140-bf1f-00b75ed0ee0c 4aa0532c-bdb2-f140-bf1f-00b75ed0ee0c}"
--- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.29s)
=== RUN   TestRaft_ProtocolVersion_RejectRPC
2026-04-23T00:16:25.986Z [INFO]  server-34a4279b-2412-fd77-cea9-9fb4f13a9fe3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34a4279b-2412-fd77-cea9-9fb4f13a9fe3 Address:34a4279b-2412-fd77-cea9-9fb4f13a9fe3} {Suffrage:Voter ID:server-454116d2-dfe8-8015-192d-11a458edae7b Address:454116d2-dfe8-8015-192d-11a458edae7b} {Suffrage:Voter ID:server-124a1edf-682f-bce9-7c11-24d1a0979cb5 Address:124a1edf-682f-bce9-7c11-24d1a0979cb5}]"
2026-04-23T00:16:25.986Z [INFO]  server-454116d2-dfe8-8015-192d-11a458edae7b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34a4279b-2412-fd77-cea9-9fb4f13a9fe3 Address:34a4279b-2412-fd77-cea9-9fb4f13a9fe3} {Suffrage:Voter ID:server-454116d2-dfe8-8015-192d-11a458edae7b Address:454116d2-dfe8-8015-192d-11a458edae7b} {Suffrage:Voter ID:server-124a1edf-682f-bce9-7c11-24d1a0979cb5 Address:124a1edf-682f-bce9-7c11-24d1a0979cb5}]"
2026-04-23T00:16:25.986Z [INFO]  server-454116d2-dfe8-8015-192d-11a458edae7b: entering follower state: follower="Node at 454116d2-dfe8-8015-192d-11a458edae7b [Follower]" leader-address= leader-id=
2026-04-23T00:16:25.986Z [INFO]  server-34a4279b-2412-fd77-cea9-9fb4f13a9fe3: entering follower state: follower="Node at 34a4279b-2412-fd77-cea9-9fb4f13a9fe3 [Follower]" leader-address= leader-id=
2026-04-23T00:16:25.986Z [INFO]  server-124a1edf-682f-bce9-7c11-24d1a0979cb5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34a4279b-2412-fd77-cea9-9fb4f13a9fe3 Address:34a4279b-2412-fd77-cea9-9fb4f13a9fe3} {Suffrage:Voter ID:server-454116d2-dfe8-8015-192d-11a458edae7b Address:454116d2-dfe8-8015-192d-11a458edae7b} {Suffrage:Voter ID:server-124a1edf-682f-bce9-7c11-24d1a0979cb5 Address:124a1edf-682f-bce9-7c11-24d1a0979cb5}]"
2026-04-23T00:16:25.986Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:25.986Z [INFO]  server-124a1edf-682f-bce9-7c11-24d1a0979cb5: entering follower state: follower="Node at 124a1edf-682f-bce9-7c11-24d1a0979cb5 [Follower]" leader-address= leader-id=
2026-04-23T00:16:26.068Z [WARN]  server-34a4279b-2412-fd77-cea9-9fb4f13a9fe3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:26.068Z [INFO]  server-34a4279b-2412-fd77-cea9-9fb4f13a9fe3: entering candidate state: node="Node at 34a4279b-2412-fd77-cea9-9fb4f13a9fe3 [Candidate]" term=2
2026-04-23T00:16:26.068Z [INFO]  server-34a4279b-2412-fd77-cea9-9fb4f13a9fe3: election won: term=2 tally=2
2026-04-23T00:16:26.068Z [INFO]  server-34a4279b-2412-fd77-cea9-9fb4f13a9fe3: entering leader state: leader="Node at 34a4279b-2412-fd77-cea9-9fb4f13a9fe3 [Leader]"
2026-04-23T00:16:26.068Z [INFO]  server-34a4279b-2412-fd77-cea9-9fb4f13a9fe3: added peer, starting replication: peer=server-454116d2-dfe8-8015-192d-11a458edae7b
2026-04-23T00:16:26.068Z [INFO]  server-34a4279b-2412-fd77-cea9-9fb4f13a9fe3: added peer, starting replication: peer=server-124a1edf-682f-bce9-7c11-24d1a0979cb5
2026-04-23T00:16:26.069Z [INFO]  server-34a4279b-2412-fd77-cea9-9fb4f13a9fe3: pipelining replication: peer="{Voter server-124a1edf-682f-bce9-7c11-24d1a0979cb5 124a1edf-682f-bce9-7c11-24d1a0979cb5}"
2026-04-23T00:16:26.069Z [INFO]  server-34a4279b-2412-fd77-cea9-9fb4f13a9fe3: pipelining replication: peer="{Voter server-454116d2-dfe8-8015-192d-11a458edae7b 454116d2-dfe8-8015-192d-11a458edae7b}"
2026-04-23T00:16:26.174Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:26.068928626 +0000 UTC m=+103.108529979 (2 nodes), 82.211923ms from start of poll, 82.484138ms from cluster start. Timeout at 2026-04-23 00:16:26.173931721 +0000 UTC m=+103.213533074, 105.003095ms after stability
2026-04-23T00:16:26.174Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:26.280Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:26.174476656 +0000 UTC m=+103.214078009 (1 nodes), 1.398µs from start of poll, 188.032168ms from cluster start. Timeout at 2026-04-23 00:16:26.27948385 +0000 UTC m=+103.319085232, 105.007223ms after stability
2026-04-23T00:16:26.280Z [INFO]  server-34a4279b-2412-fd77-cea9-9fb4f13a9fe3: aborting pipeline replication: peer="{Voter server-454116d2-dfe8-8015-192d-11a458edae7b 454116d2-dfe8-8015-192d-11a458edae7b}"
2026-04-23T00:16:26.280Z [INFO]  server-34a4279b-2412-fd77-cea9-9fb4f13a9fe3: aborting pipeline replication: peer="{Voter server-124a1edf-682f-bce9-7c11-24d1a0979cb5 124a1edf-682f-bce9-7c11-24d1a0979cb5}"
--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.29s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_1_2
    raft_test.go:2151: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_2_3
    raft_test.go:2196: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s)
=== RUN   TestRaft_LeaderID_Propagated
2026-04-23T00:16:26.281Z [INFO]  server-ea4c61e9-daa8-fefd-ce97-057b7044b4f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea4c61e9-daa8-fefd-ce97-057b7044b4f7 Address:ea4c61e9-daa8-fefd-ce97-057b7044b4f7} {Suffrage:Voter ID:server-32942ab9-9d7b-c52d-11e9-d1fa4a89fd67 Address:32942ab9-9d7b-c52d-11e9-d1fa4a89fd67} {Suffrage:Voter ID:server-ef4c31e8-a707-138e-dc57-f56c9e8e9f61 Address:ef4c31e8-a707-138e-dc57-f56c9e8e9f61}]"
2026-04-23T00:16:26.281Z [INFO]  server-ea4c61e9-daa8-fefd-ce97-057b7044b4f7: entering follower state: follower="Node at ea4c61e9-daa8-fefd-ce97-057b7044b4f7 [Follower]" leader-address= leader-id=
2026-04-23T00:16:26.282Z [INFO]  server-32942ab9-9d7b-c52d-11e9-d1fa4a89fd67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea4c61e9-daa8-fefd-ce97-057b7044b4f7 Address:ea4c61e9-daa8-fefd-ce97-057b7044b4f7} {Suffrage:Voter ID:server-32942ab9-9d7b-c52d-11e9-d1fa4a89fd67 Address:32942ab9-9d7b-c52d-11e9-d1fa4a89fd67} {Suffrage:Voter ID:server-ef4c31e8-a707-138e-dc57-f56c9e8e9f61 Address:ef4c31e8-a707-138e-dc57-f56c9e8e9f61}]"
2026-04-23T00:16:26.282Z [INFO]  server-32942ab9-9d7b-c52d-11e9-d1fa4a89fd67: entering follower state: follower="Node at 32942ab9-9d7b-c52d-11e9-d1fa4a89fd67 [Follower]" leader-address= leader-id=
2026-04-23T00:16:26.282Z [INFO]  server-ef4c31e8-a707-138e-dc57-f56c9e8e9f61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea4c61e9-daa8-fefd-ce97-057b7044b4f7 Address:ea4c61e9-daa8-fefd-ce97-057b7044b4f7} {Suffrage:Voter ID:server-32942ab9-9d7b-c52d-11e9-d1fa4a89fd67 Address:32942ab9-9d7b-c52d-11e9-d1fa4a89fd67} {Suffrage:Voter ID:server-ef4c31e8-a707-138e-dc57-f56c9e8e9f61 Address:ef4c31e8-a707-138e-dc57-f56c9e8e9f61}]"
2026-04-23T00:16:26.282Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:26.282Z [INFO]  server-ef4c31e8-a707-138e-dc57-f56c9e8e9f61: entering follower state: follower="Node at ef4c31e8-a707-138e-dc57-f56c9e8e9f61 [Follower]" leader-address= leader-id=
2026-04-23T00:16:26.336Z [WARN]  server-ea4c61e9-daa8-fefd-ce97-057b7044b4f7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:26.336Z [INFO]  server-ea4c61e9-daa8-fefd-ce97-057b7044b4f7: entering candidate state: node="Node at ea4c61e9-daa8-fefd-ce97-057b7044b4f7 [Candidate]" term=2
2026-04-23T00:16:26.336Z [INFO]  server-ea4c61e9-daa8-fefd-ce97-057b7044b4f7: election won: term=2 tally=2
2026-04-23T00:16:26.336Z [INFO]  server-ea4c61e9-daa8-fefd-ce97-057b7044b4f7: entering leader state: leader="Node at ea4c61e9-daa8-fefd-ce97-057b7044b4f7 [Leader]"
2026-04-23T00:16:26.336Z [INFO]  server-ea4c61e9-daa8-fefd-ce97-057b7044b4f7: added peer, starting replication: peer=server-32942ab9-9d7b-c52d-11e9-d1fa4a89fd67
2026-04-23T00:16:26.336Z [INFO]  server-ea4c61e9-daa8-fefd-ce97-057b7044b4f7: added peer, starting replication: peer=server-ef4c31e8-a707-138e-dc57-f56c9e8e9f61
2026-04-23T00:16:26.336Z [INFO]  server-ea4c61e9-daa8-fefd-ce97-057b7044b4f7: pipelining replication: peer="{Voter server-ef4c31e8-a707-138e-dc57-f56c9e8e9f61 ef4c31e8-a707-138e-dc57-f56c9e8e9f61}"
2026-04-23T00:16:26.336Z [INFO]  server-ea4c61e9-daa8-fefd-ce97-057b7044b4f7: pipelining replication: peer="{Voter server-32942ab9-9d7b-c52d-11e9-d1fa4a89fd67 32942ab9-9d7b-c52d-11e9-d1fa4a89fd67}"
2026-04-23T00:16:26.441Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:26.336428315 +0000 UTC m=+103.376029672 (1 nodes), 54.224245ms from start of poll, 54.593133ms from cluster start. Timeout at 2026-04-23 00:16:26.441432046 +0000 UTC m=+103.481033406, 105.003734ms after stability
2026-04-23T00:16:26.442Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:26.547Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:26.442051682 +0000 UTC m=+103.481653054 (1 nodes), 870ns from start of poll, 160.216515ms from cluster start. Timeout at 2026-04-23 00:16:26.54706459 +0000 UTC m=+103.586665965, 105.012911ms after stability
2026-04-23T00:16:26.547Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:26.652Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:26.547245674 +0000 UTC m=+103.586847038 (1 nodes), 504ns from start of poll, 265.410499ms from cluster start. Timeout at 2026-04-23 00:16:26.652262581 +0000 UTC m=+103.691863964, 105.016926ms after stability
2026-04-23T00:16:26.653Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:26.758Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:26.653108251 +0000 UTC m=+103.692709603 (1 nodes), 665ns from start of poll, 371.273064ms from cluster start. Timeout at 2026-04-23 00:16:26.758129924 +0000 UTC m=+103.797731317, 105.021714ms after stability
2026-04-23T00:16:26.758Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:26.864Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:26.758849316 +0000 UTC m=+103.798450673 (1 nodes), 634ns from start of poll, 477.014134ms from cluster start. Timeout at 2026-04-23 00:16:26.863871882 +0000 UTC m=+103.903473234, 105.022561ms after stability
2026-04-23T00:16:26.864Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:26.969Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:26.864207546 +0000 UTC m=+103.903808904 (1 nodes), 711ns from start of poll, 582.372365ms from cluster start. Timeout at 2026-04-23 00:16:26.969227084 +0000 UTC m=+104.008828439, 105.019535ms after stability
2026-04-23T00:16:27.075Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:27.181Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:27.075246669 +0000 UTC m=+104.114848026 (1 nodes), 1.258µs from start of poll, 793.411487ms from cluster start. Timeout at 2026-04-23 00:16:27.180258415 +0000 UTC m=+104.219859826, 105.0118ms after stability
2026-04-23T00:16:27.181Z [INFO]  server-ea4c61e9-daa8-fefd-ce97-057b7044b4f7: aborting pipeline replication: peer="{Voter server-ef4c31e8-a707-138e-dc57-f56c9e8e9f61 ef4c31e8-a707-138e-dc57-f56c9e8e9f61}"
2026-04-23T00:16:27.181Z [INFO]  server-ea4c61e9-daa8-fefd-ce97-057b7044b4f7: aborting pipeline replication: peer="{Voter server-32942ab9-9d7b-c52d-11e9-d1fa4a89fd67 32942ab9-9d7b-c52d-11e9-d1fa4a89fd67}"
--- PASS: TestRaft_LeaderID_Propagated (0.90s)
=== RUN   TestRaft_LeadershipTransferInProgress
--- PASS: TestRaft_LeadershipTransferInProgress (0.00s)
=== RUN   TestRaft_LeadershipTransferPickServer
--- PASS: TestRaft_LeadershipTransferPickServer (0.00s)
=== RUN   TestRaft_LeadershipTransfer
2026-04-23T00:16:27.182Z [INFO]  server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed Address:a12c451c-4ee7-7d8b-5b8b-694dfd9894ed} {Suffrage:Voter ID:server-71919961-e808-80a9-a8a7-e3b977120484 Address:71919961-e808-80a9-a8a7-e3b977120484} {Suffrage:Voter ID:server-190ba0fa-778a-7c8d-9237-f4bb0421cfd1 Address:190ba0fa-778a-7c8d-9237-f4bb0421cfd1}]"
2026-04-23T00:16:27.182Z [INFO]  server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed: entering follower state: follower="Node at a12c451c-4ee7-7d8b-5b8b-694dfd9894ed [Follower]" leader-address= leader-id=
2026-04-23T00:16:27.182Z [INFO]  server-71919961-e808-80a9-a8a7-e3b977120484: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed Address:a12c451c-4ee7-7d8b-5b8b-694dfd9894ed} {Suffrage:Voter ID:server-71919961-e808-80a9-a8a7-e3b977120484 Address:71919961-e808-80a9-a8a7-e3b977120484} {Suffrage:Voter ID:server-190ba0fa-778a-7c8d-9237-f4bb0421cfd1 Address:190ba0fa-778a-7c8d-9237-f4bb0421cfd1}]"
2026-04-23T00:16:27.182Z [INFO]  server-71919961-e808-80a9-a8a7-e3b977120484: entering follower state: follower="Node at 71919961-e808-80a9-a8a7-e3b977120484 [Follower]" leader-address= leader-id=
2026-04-23T00:16:27.182Z [INFO]  server-190ba0fa-778a-7c8d-9237-f4bb0421cfd1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed Address:a12c451c-4ee7-7d8b-5b8b-694dfd9894ed} {Suffrage:Voter ID:server-71919961-e808-80a9-a8a7-e3b977120484 Address:71919961-e808-80a9-a8a7-e3b977120484} {Suffrage:Voter ID:server-190ba0fa-778a-7c8d-9237-f4bb0421cfd1 Address:190ba0fa-778a-7c8d-9237-f4bb0421cfd1}]"
2026-04-23T00:16:27.182Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:27.182Z [INFO]  server-190ba0fa-778a-7c8d-9237-f4bb0421cfd1: entering follower state: follower="Node at 190ba0fa-778a-7c8d-9237-f4bb0421cfd1 [Follower]" leader-address= leader-id=
2026-04-23T00:16:27.236Z [WARN]  server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:27.236Z [INFO]  server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed: entering candidate state: node="Node at a12c451c-4ee7-7d8b-5b8b-694dfd9894ed [Candidate]" term=2
2026-04-23T00:16:27.236Z [INFO]  server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed: election won: term=2 tally=2
2026-04-23T00:16:27.236Z [INFO]  server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed: entering leader state: leader="Node at a12c451c-4ee7-7d8b-5b8b-694dfd9894ed [Leader]"
2026-04-23T00:16:27.236Z [INFO]  server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed: added peer, starting replication: peer=server-71919961-e808-80a9-a8a7-e3b977120484
2026-04-23T00:16:27.236Z [INFO]  server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed: added peer, starting replication: peer=server-190ba0fa-778a-7c8d-9237-f4bb0421cfd1
2026-04-23T00:16:27.236Z [INFO]  server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed: pipelining replication: peer="{Voter server-190ba0fa-778a-7c8d-9237-f4bb0421cfd1 190ba0fa-778a-7c8d-9237-f4bb0421cfd1}"
2026-04-23T00:16:27.236Z [INFO]  server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed: pipelining replication: peer="{Voter server-71919961-e808-80a9-a8a7-e3b977120484 71919961-e808-80a9-a8a7-e3b977120484}"
2026-04-23T00:16:27.342Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:27.236761959 +0000 UTC m=+104.276363311 (1 nodes), 53.980933ms from start of poll, 54.371329ms from cluster start. Timeout at 2026-04-23 00:16:27.341766674 +0000 UTC m=+104.381368031, 105.00472ms after stability
2026-04-23T00:16:27.342Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:27.447Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:27.342244516 +0000 UTC m=+104.381845867 (1 nodes), 1.092µs from start of poll, 159.853885ms from cluster start. Timeout at 2026-04-23 00:16:27.447267332 +0000 UTC m=+104.486868689, 105.022822ms after stability
2026-04-23T00:16:27.447Z [INFO]  server-71919961-e808-80a9-a8a7-e3b977120484: entering candidate state: node="Node at 71919961-e808-80a9-a8a7-e3b977120484 [Candidate]" term=3
2026-04-23T00:16:27.448Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:27.448Z [INFO]  server-71919961-e808-80a9-a8a7-e3b977120484: election won: term=3 tally=2
2026-04-23T00:16:27.448Z [INFO]  server-71919961-e808-80a9-a8a7-e3b977120484: entering leader state: leader="Node at 71919961-e808-80a9-a8a7-e3b977120484 [Leader]"
2026-04-23T00:16:27.448Z [INFO]  server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed: entering follower state: follower="Node at a12c451c-4ee7-7d8b-5b8b-694dfd9894ed [Follower]" leader-address= leader-id=
2026-04-23T00:16:27.448Z [INFO]  server-71919961-e808-80a9-a8a7-e3b977120484: added peer, starting replication: peer=server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed
2026-04-23T00:16:27.448Z [INFO]  server-71919961-e808-80a9-a8a7-e3b977120484: added peer, starting replication: peer=server-190ba0fa-778a-7c8d-9237-f4bb0421cfd1
2026-04-23T00:16:27.448Z [INFO]  server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed: aborting pipeline replication: peer="{Voter server-190ba0fa-778a-7c8d-9237-f4bb0421cfd1 190ba0fa-778a-7c8d-9237-f4bb0421cfd1}"
2026-04-23T00:16:27.448Z [INFO]  server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed: aborting pipeline replication: peer="{Voter server-71919961-e808-80a9-a8a7-e3b977120484 71919961-e808-80a9-a8a7-e3b977120484}"
2026-04-23T00:16:27.448Z [INFO]  server-71919961-e808-80a9-a8a7-e3b977120484: pipelining replication: peer="{Voter server-190ba0fa-778a-7c8d-9237-f4bb0421cfd1 190ba0fa-778a-7c8d-9237-f4bb0421cfd1}"
2026-04-23T00:16:27.448Z [INFO]  server-71919961-e808-80a9-a8a7-e3b977120484: pipelining replication: peer="{Voter server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed a12c451c-4ee7-7d8b-5b8b-694dfd9894ed}"
2026-04-23T00:16:27.553Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:27.44809941 +0000 UTC m=+104.487700773 (1 nodes), 59.409µs from start of poll, 265.708791ms from cluster start. Timeout at 2026-04-23 00:16:27.553116842 +0000 UTC m=+104.592718198, 105.017425ms after stability
2026-04-23T00:16:27.553Z [INFO]  server-71919961-e808-80a9-a8a7-e3b977120484: aborting pipeline replication: peer="{Voter server-190ba0fa-778a-7c8d-9237-f4bb0421cfd1 190ba0fa-778a-7c8d-9237-f4bb0421cfd1}"
2026-04-23T00:16:27.553Z [INFO]  server-71919961-e808-80a9-a8a7-e3b977120484: aborting pipeline replication: peer="{Voter server-a12c451c-4ee7-7d8b-5b8b-694dfd9894ed a12c451c-4ee7-7d8b-5b8b-694dfd9894ed}"
--- PASS: TestRaft_LeadershipTransfer (0.37s)
=== RUN   TestRaft_LeadershipTransferWithOneNode
2026-04-23T00:16:27.554Z [INFO]  server-eff75ed1-07ea-83e9-7cf1-0db9048802c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eff75ed1-07ea-83e9-7cf1-0db9048802c1 Address:eff75ed1-07ea-83e9-7cf1-0db9048802c1}]"
2026-04-23T00:16:27.554Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:27.554Z [INFO]  server-eff75ed1-07ea-83e9-7cf1-0db9048802c1: entering follower state: follower="Node at eff75ed1-07ea-83e9-7cf1-0db9048802c1 [Follower]" leader-address= leader-id=
2026-04-23T00:16:27.620Z [WARN]  server-eff75ed1-07ea-83e9-7cf1-0db9048802c1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:27.620Z [INFO]  server-eff75ed1-07ea-83e9-7cf1-0db9048802c1: entering candidate state: node="Node at eff75ed1-07ea-83e9-7cf1-0db9048802c1 [Candidate]" term=2
2026-04-23T00:16:27.620Z [INFO]  server-eff75ed1-07ea-83e9-7cf1-0db9048802c1: election won: term=2 tally=1
2026-04-23T00:16:27.620Z [INFO]  server-eff75ed1-07ea-83e9-7cf1-0db9048802c1: entering leader state: leader="Node at eff75ed1-07ea-83e9-7cf1-0db9048802c1 [Leader]"
2026-04-23T00:16:27.725Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:27.620230221 +0000 UTC m=+104.659831573 (1 nodes), 65.448094ms from start of poll, 65.6097ms from cluster start. Timeout at 2026-04-23 00:16:27.725236016 +0000 UTC m=+104.764837372, 105.005799ms after stability
--- PASS: TestRaft_LeadershipTransferWithOneNode (0.17s)
=== RUN   TestRaft_LeadershipTransferWithSevenNodes
2026-04-23T00:16:27.726Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eba17a29-fbc3-9cbf-db99-643a8a4cc165 Address:eba17a29-fbc3-9cbf-db99-643a8a4cc165} {Suffrage:Voter ID:server-1dcfdfc0-79b0-a21f-3d2e-38b7093129cd Address:1dcfdfc0-79b0-a21f-3d2e-38b7093129cd} {Suffrage:Voter ID:server-b2de5c57-1774-89ec-9f1e-7b1bad32844d Address:b2de5c57-1774-89ec-9f1e-7b1bad32844d} {Suffrage:Voter ID:server-39d93644-0a61-36f8-5ac2-cf6ae494111f Address:39d93644-0a61-36f8-5ac2-cf6ae494111f} {Suffrage:Voter ID:server-f8eeeebf-5301-fdd8-a0e2-8996774a03e2 Address:f8eeeebf-5301-fdd8-a0e2-8996774a03e2} {Suffrage:Voter ID:server-654c0210-9fd1-e53e-f095-e4d1f250b73e Address:654c0210-9fd1-e53e-f095-e4d1f250b73e} {Suffrage:Voter ID:server-db273d6d-906e-5f88-0a78-ce4f3f339ffb Address:db273d6d-906e-5f88-0a78-ce4f3f339ffb}]"
2026-04-23T00:16:27.727Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: entering follower state: follower="Node at eba17a29-fbc3-9cbf-db99-643a8a4cc165 [Follower]" leader-address= leader-id=
2026-04-23T00:16:27.727Z [INFO]  server-1dcfdfc0-79b0-a21f-3d2e-38b7093129cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eba17a29-fbc3-9cbf-db99-643a8a4cc165 Address:eba17a29-fbc3-9cbf-db99-643a8a4cc165} {Suffrage:Voter ID:server-1dcfdfc0-79b0-a21f-3d2e-38b7093129cd Address:1dcfdfc0-79b0-a21f-3d2e-38b7093129cd} {Suffrage:Voter ID:server-b2de5c57-1774-89ec-9f1e-7b1bad32844d Address:b2de5c57-1774-89ec-9f1e-7b1bad32844d} {Suffrage:Voter ID:server-39d93644-0a61-36f8-5ac2-cf6ae494111f Address:39d93644-0a61-36f8-5ac2-cf6ae494111f} {Suffrage:Voter ID:server-f8eeeebf-5301-fdd8-a0e2-8996774a03e2 Address:f8eeeebf-5301-fdd8-a0e2-8996774a03e2} {Suffrage:Voter ID:server-654c0210-9fd1-e53e-f095-e4d1f250b73e Address:654c0210-9fd1-e53e-f095-e4d1f250b73e} {Suffrage:Voter ID:server-db273d6d-906e-5f88-0a78-ce4f3f339ffb Address:db273d6d-906e-5f88-0a78-ce4f3f339ffb}]"
2026-04-23T00:16:27.727Z [INFO]  server-1dcfdfc0-79b0-a21f-3d2e-38b7093129cd: entering follower state: follower="Node at 1dcfdfc0-79b0-a21f-3d2e-38b7093129cd [Follower]" leader-address= leader-id=
2026-04-23T00:16:27.727Z [INFO]  server-b2de5c57-1774-89ec-9f1e-7b1bad32844d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eba17a29-fbc3-9cbf-db99-643a8a4cc165 Address:eba17a29-fbc3-9cbf-db99-643a8a4cc165} {Suffrage:Voter ID:server-1dcfdfc0-79b0-a21f-3d2e-38b7093129cd Address:1dcfdfc0-79b0-a21f-3d2e-38b7093129cd} {Suffrage:Voter ID:server-b2de5c57-1774-89ec-9f1e-7b1bad32844d Address:b2de5c57-1774-89ec-9f1e-7b1bad32844d} {Suffrage:Voter ID:server-39d93644-0a61-36f8-5ac2-cf6ae494111f Address:39d93644-0a61-36f8-5ac2-cf6ae494111f} {Suffrage:Voter ID:server-f8eeeebf-5301-fdd8-a0e2-8996774a03e2 Address:f8eeeebf-5301-fdd8-a0e2-8996774a03e2} {Suffrage:Voter ID:server-654c0210-9fd1-e53e-f095-e4d1f250b73e Address:654c0210-9fd1-e53e-f095-e4d1f250b73e} {Suffrage:Voter ID:server-db273d6d-906e-5f88-0a78-ce4f3f339ffb Address:db273d6d-906e-5f88-0a78-ce4f3f339ffb}]"
2026-04-23T00:16:27.727Z [INFO]  server-b2de5c57-1774-89ec-9f1e-7b1bad32844d: entering follower state: follower="Node at b2de5c57-1774-89ec-9f1e-7b1bad32844d [Follower]" leader-address= leader-id=
2026-04-23T00:16:27.727Z [INFO]  server-39d93644-0a61-36f8-5ac2-cf6ae494111f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eba17a29-fbc3-9cbf-db99-643a8a4cc165 Address:eba17a29-fbc3-9cbf-db99-643a8a4cc165} {Suffrage:Voter ID:server-1dcfdfc0-79b0-a21f-3d2e-38b7093129cd Address:1dcfdfc0-79b0-a21f-3d2e-38b7093129cd} {Suffrage:Voter ID:server-b2de5c57-1774-89ec-9f1e-7b1bad32844d Address:b2de5c57-1774-89ec-9f1e-7b1bad32844d} {Suffrage:Voter ID:server-39d93644-0a61-36f8-5ac2-cf6ae494111f Address:39d93644-0a61-36f8-5ac2-cf6ae494111f} {Suffrage:Voter ID:server-f8eeeebf-5301-fdd8-a0e2-8996774a03e2 Address:f8eeeebf-5301-fdd8-a0e2-8996774a03e2} {Suffrage:Voter ID:server-654c0210-9fd1-e53e-f095-e4d1f250b73e Address:654c0210-9fd1-e53e-f095-e4d1f250b73e} {Suffrage:Voter ID:server-db273d6d-906e-5f88-0a78-ce4f3f339ffb Address:db273d6d-906e-5f88-0a78-ce4f3f339ffb}]"
2026-04-23T00:16:27.727Z [INFO]  server-39d93644-0a61-36f8-5ac2-cf6ae494111f: entering follower state: follower="Node at 39d93644-0a61-36f8-5ac2-cf6ae494111f [Follower]" leader-address= leader-id=
2026-04-23T00:16:27.727Z [INFO]  server-f8eeeebf-5301-fdd8-a0e2-8996774a03e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eba17a29-fbc3-9cbf-db99-643a8a4cc165 Address:eba17a29-fbc3-9cbf-db99-643a8a4cc165} {Suffrage:Voter ID:server-1dcfdfc0-79b0-a21f-3d2e-38b7093129cd Address:1dcfdfc0-79b0-a21f-3d2e-38b7093129cd} {Suffrage:Voter ID:server-b2de5c57-1774-89ec-9f1e-7b1bad32844d Address:b2de5c57-1774-89ec-9f1e-7b1bad32844d} {Suffrage:Voter ID:server-39d93644-0a61-36f8-5ac2-cf6ae494111f Address:39d93644-0a61-36f8-5ac2-cf6ae494111f} {Suffrage:Voter ID:server-f8eeeebf-5301-fdd8-a0e2-8996774a03e2 Address:f8eeeebf-5301-fdd8-a0e2-8996774a03e2} {Suffrage:Voter ID:server-654c0210-9fd1-e53e-f095-e4d1f250b73e Address:654c0210-9fd1-e53e-f095-e4d1f250b73e} {Suffrage:Voter ID:server-db273d6d-906e-5f88-0a78-ce4f3f339ffb Address:db273d6d-906e-5f88-0a78-ce4f3f339ffb}]"
2026-04-23T00:16:27.727Z [INFO]  server-f8eeeebf-5301-fdd8-a0e2-8996774a03e2: entering follower state: follower="Node at f8eeeebf-5301-fdd8-a0e2-8996774a03e2 [Follower]" leader-address= leader-id=
2026-04-23T00:16:27.727Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eba17a29-fbc3-9cbf-db99-643a8a4cc165 Address:eba17a29-fbc3-9cbf-db99-643a8a4cc165} {Suffrage:Voter ID:server-1dcfdfc0-79b0-a21f-3d2e-38b7093129cd Address:1dcfdfc0-79b0-a21f-3d2e-38b7093129cd} {Suffrage:Voter ID:server-b2de5c57-1774-89ec-9f1e-7b1bad32844d Address:b2de5c57-1774-89ec-9f1e-7b1bad32844d} {Suffrage:Voter ID:server-39d93644-0a61-36f8-5ac2-cf6ae494111f Address:39d93644-0a61-36f8-5ac2-cf6ae494111f} {Suffrage:Voter ID:server-f8eeeebf-5301-fdd8-a0e2-8996774a03e2 Address:f8eeeebf-5301-fdd8-a0e2-8996774a03e2} {Suffrage:Voter ID:server-654c0210-9fd1-e53e-f095-e4d1f250b73e Address:654c0210-9fd1-e53e-f095-e4d1f250b73e} {Suffrage:Voter ID:server-db273d6d-906e-5f88-0a78-ce4f3f339ffb Address:db273d6d-906e-5f88-0a78-ce4f3f339ffb}]"
2026-04-23T00:16:27.727Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: entering follower state: follower="Node at 654c0210-9fd1-e53e-f095-e4d1f250b73e [Follower]" leader-address= leader-id=
2026-04-23T00:16:27.727Z [INFO]  server-db273d6d-906e-5f88-0a78-ce4f3f339ffb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eba17a29-fbc3-9cbf-db99-643a8a4cc165 Address:eba17a29-fbc3-9cbf-db99-643a8a4cc165} {Suffrage:Voter ID:server-1dcfdfc0-79b0-a21f-3d2e-38b7093129cd Address:1dcfdfc0-79b0-a21f-3d2e-38b7093129cd} {Suffrage:Voter ID:server-b2de5c57-1774-89ec-9f1e-7b1bad32844d Address:b2de5c57-1774-89ec-9f1e-7b1bad32844d} {Suffrage:Voter ID:server-39d93644-0a61-36f8-5ac2-cf6ae494111f Address:39d93644-0a61-36f8-5ac2-cf6ae494111f} {Suffrage:Voter ID:server-f8eeeebf-5301-fdd8-a0e2-8996774a03e2 Address:f8eeeebf-5301-fdd8-a0e2-8996774a03e2} {Suffrage:Voter ID:server-654c0210-9fd1-e53e-f095-e4d1f250b73e Address:654c0210-9fd1-e53e-f095-e4d1f250b73e} {Suffrage:Voter ID:server-db273d6d-906e-5f88-0a78-ce4f3f339ffb Address:db273d6d-906e-5f88-0a78-ce4f3f339ffb}]"
2026-04-23T00:16:27.727Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:27.727Z [INFO]  server-db273d6d-906e-5f88-0a78-ce4f3f339ffb: entering follower state: follower="Node at db273d6d-906e-5f88-0a78-ce4f3f339ffb [Follower]" leader-address= leader-id=
2026-04-23T00:16:27.784Z [WARN]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:27.784Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: entering candidate state: node="Node at 654c0210-9fd1-e53e-f095-e4d1f250b73e [Candidate]" term=2
2026-04-23T00:16:27.784Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: election won: term=2 tally=4
2026-04-23T00:16:27.784Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: entering leader state: leader="Node at 654c0210-9fd1-e53e-f095-e4d1f250b73e [Leader]"
2026-04-23T00:16:27.784Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: added peer, starting replication: peer=server-eba17a29-fbc3-9cbf-db99-643a8a4cc165
2026-04-23T00:16:27.784Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: added peer, starting replication: peer=server-1dcfdfc0-79b0-a21f-3d2e-38b7093129cd
2026-04-23T00:16:27.784Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: added peer, starting replication: peer=server-b2de5c57-1774-89ec-9f1e-7b1bad32844d
2026-04-23T00:16:27.784Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: added peer, starting replication: peer=server-39d93644-0a61-36f8-5ac2-cf6ae494111f
2026-04-23T00:16:27.784Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: added peer, starting replication: peer=server-f8eeeebf-5301-fdd8-a0e2-8996774a03e2
2026-04-23T00:16:27.784Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: added peer, starting replication: peer=server-db273d6d-906e-5f88-0a78-ce4f3f339ffb
2026-04-23T00:16:27.785Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: pipelining replication: peer="{Voter server-db273d6d-906e-5f88-0a78-ce4f3f339ffb db273d6d-906e-5f88-0a78-ce4f3f339ffb}"
2026-04-23T00:16:27.785Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: pipelining replication: peer="{Voter server-1dcfdfc0-79b0-a21f-3d2e-38b7093129cd 1dcfdfc0-79b0-a21f-3d2e-38b7093129cd}"
2026-04-23T00:16:27.785Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: pipelining replication: peer="{Voter server-b2de5c57-1774-89ec-9f1e-7b1bad32844d b2de5c57-1774-89ec-9f1e-7b1bad32844d}"
2026-04-23T00:16:27.785Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: pipelining replication: peer="{Voter server-eba17a29-fbc3-9cbf-db99-643a8a4cc165 eba17a29-fbc3-9cbf-db99-643a8a4cc165}"
2026-04-23T00:16:27.785Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: pipelining replication: peer="{Voter server-39d93644-0a61-36f8-5ac2-cf6ae494111f 39d93644-0a61-36f8-5ac2-cf6ae494111f}"
2026-04-23T00:16:27.785Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: pipelining replication: peer="{Voter server-f8eeeebf-5301-fdd8-a0e2-8996774a03e2 f8eeeebf-5301-fdd8-a0e2-8996774a03e2}"
2026-04-23T00:16:27.890Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:27.784950969 +0000 UTC m=+104.824552328 (1 nodes), 57.412484ms from start of poll, 58.127511ms from cluster start. Timeout at 2026-04-23 00:16:27.889953751 +0000 UTC m=+104.929555107, 105.002779ms after stability
2026-04-23T00:16:27.890Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:27.996Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:27.890662861 +0000 UTC m=+104.930264217 (6 nodes), 1.017µs from start of poll, 163.8394ms from cluster start. Timeout at 2026-04-23 00:16:27.995690369 +0000 UTC m=+105.035291754, 105.027537ms after stability
2026-04-23T00:16:27.996Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:28.102Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:27.996836977 +0000 UTC m=+105.036438329 (1 nodes), 1.876µs from start of poll, 270.013512ms from cluster start. Timeout at 2026-04-23 00:16:28.101871375 +0000 UTC m=+105.141472729, 105.0344ms after stability
2026-04-23T00:16:28.102Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: entering candidate state: node="Node at eba17a29-fbc3-9cbf-db99-643a8a4cc165 [Candidate]" term=3
2026-04-23T00:16:28.102Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: election won: term=3 tally=4
2026-04-23T00:16:28.102Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: entering leader state: leader="Node at eba17a29-fbc3-9cbf-db99-643a8a4cc165 [Leader]"
2026-04-23T00:16:28.102Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:28.102Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: added peer, starting replication: peer=server-1dcfdfc0-79b0-a21f-3d2e-38b7093129cd
2026-04-23T00:16:28.102Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: added peer, starting replication: peer=server-b2de5c57-1774-89ec-9f1e-7b1bad32844d
2026-04-23T00:16:28.102Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: added peer, starting replication: peer=server-39d93644-0a61-36f8-5ac2-cf6ae494111f
2026-04-23T00:16:28.102Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: added peer, starting replication: peer=server-f8eeeebf-5301-fdd8-a0e2-8996774a03e2
2026-04-23T00:16:28.102Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: entering follower state: follower="Node at 654c0210-9fd1-e53e-f095-e4d1f250b73e [Follower]" leader-address= leader-id=
2026-04-23T00:16:28.102Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: added peer, starting replication: peer=server-654c0210-9fd1-e53e-f095-e4d1f250b73e
2026-04-23T00:16:28.102Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: added peer, starting replication: peer=server-db273d6d-906e-5f88-0a78-ce4f3f339ffb
2026-04-23T00:16:28.102Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: aborting pipeline replication: peer="{Voter server-db273d6d-906e-5f88-0a78-ce4f3f339ffb db273d6d-906e-5f88-0a78-ce4f3f339ffb}"
2026-04-23T00:16:28.102Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: aborting pipeline replication: peer="{Voter server-eba17a29-fbc3-9cbf-db99-643a8a4cc165 eba17a29-fbc3-9cbf-db99-643a8a4cc165}"
2026-04-23T00:16:28.102Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: aborting pipeline replication: peer="{Voter server-1dcfdfc0-79b0-a21f-3d2e-38b7093129cd 1dcfdfc0-79b0-a21f-3d2e-38b7093129cd}"
2026-04-23T00:16:28.102Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: aborting pipeline replication: peer="{Voter server-b2de5c57-1774-89ec-9f1e-7b1bad32844d b2de5c57-1774-89ec-9f1e-7b1bad32844d}"
2026-04-23T00:16:28.102Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: aborting pipeline replication: peer="{Voter server-39d93644-0a61-36f8-5ac2-cf6ae494111f 39d93644-0a61-36f8-5ac2-cf6ae494111f}"
2026-04-23T00:16:28.102Z [INFO]  server-654c0210-9fd1-e53e-f095-e4d1f250b73e: aborting pipeline replication: peer="{Voter server-f8eeeebf-5301-fdd8-a0e2-8996774a03e2 f8eeeebf-5301-fdd8-a0e2-8996774a03e2}"
2026-04-23T00:16:28.102Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: pipelining replication: peer="{Voter server-1dcfdfc0-79b0-a21f-3d2e-38b7093129cd 1dcfdfc0-79b0-a21f-3d2e-38b7093129cd}"
2026-04-23T00:16:28.102Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: pipelining replication: peer="{Voter server-39d93644-0a61-36f8-5ac2-cf6ae494111f 39d93644-0a61-36f8-5ac2-cf6ae494111f}"
2026-04-23T00:16:28.102Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: pipelining replication: peer="{Voter server-654c0210-9fd1-e53e-f095-e4d1f250b73e 654c0210-9fd1-e53e-f095-e4d1f250b73e}"
2026-04-23T00:16:28.102Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: pipelining replication: peer="{Voter server-db273d6d-906e-5f88-0a78-ce4f3f339ffb db273d6d-906e-5f88-0a78-ce4f3f339ffb}"
2026-04-23T00:16:28.104Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: pipelining replication: peer="{Voter server-f8eeeebf-5301-fdd8-a0e2-8996774a03e2 f8eeeebf-5301-fdd8-a0e2-8996774a03e2}"
2026-04-23T00:16:28.104Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: pipelining replication: peer="{Voter server-b2de5c57-1774-89ec-9f1e-7b1bad32844d b2de5c57-1774-89ec-9f1e-7b1bad32844d}"
2026-04-23T00:16:28.207Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:28.102558838 +0000 UTC m=+105.142160272 (1 nodes), 30.38µs from start of poll, 375.735455ms from cluster start. Timeout at 2026-04-23 00:16:28.207582743 +0000 UTC m=+105.247184118, 105.023846ms after stability
2026-04-23T00:16:28.207Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: aborting pipeline replication: peer="{Voter server-db273d6d-906e-5f88-0a78-ce4f3f339ffb db273d6d-906e-5f88-0a78-ce4f3f339ffb}"
2026-04-23T00:16:28.207Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: aborting pipeline replication: peer="{Voter server-654c0210-9fd1-e53e-f095-e4d1f250b73e 654c0210-9fd1-e53e-f095-e4d1f250b73e}"
2026-04-23T00:16:28.207Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: aborting pipeline replication: peer="{Voter server-f8eeeebf-5301-fdd8-a0e2-8996774a03e2 f8eeeebf-5301-fdd8-a0e2-8996774a03e2}"
2026-04-23T00:16:28.207Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: aborting pipeline replication: peer="{Voter server-1dcfdfc0-79b0-a21f-3d2e-38b7093129cd 1dcfdfc0-79b0-a21f-3d2e-38b7093129cd}"
2026-04-23T00:16:28.207Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: aborting pipeline replication: peer="{Voter server-b2de5c57-1774-89ec-9f1e-7b1bad32844d b2de5c57-1774-89ec-9f1e-7b1bad32844d}"
2026-04-23T00:16:28.207Z [INFO]  server-eba17a29-fbc3-9cbf-db99-643a8a4cc165: aborting pipeline replication: peer="{Voter server-39d93644-0a61-36f8-5ac2-cf6ae494111f 39d93644-0a61-36f8-5ac2-cf6ae494111f}"
--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.48s)
=== RUN   TestRaft_LeadershipTransferToInvalidID
2026-04-23T00:16:28.209Z [INFO]  server-ed9665e5-b724-2e27-b766-38f7639c506c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed9665e5-b724-2e27-b766-38f7639c506c Address:ed9665e5-b724-2e27-b766-38f7639c506c} {Suffrage:Voter ID:server-c2784ea3-27d8-a99f-c46b-d18b76517333 Address:c2784ea3-27d8-a99f-c46b-d18b76517333} {Suffrage:Voter ID:server-b6e43e33-90be-268c-3669-de06d18531f7 Address:b6e43e33-90be-268c-3669-de06d18531f7}]"
2026-04-23T00:16:28.209Z [INFO]  server-c2784ea3-27d8-a99f-c46b-d18b76517333: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed9665e5-b724-2e27-b766-38f7639c506c Address:ed9665e5-b724-2e27-b766-38f7639c506c} {Suffrage:Voter ID:server-c2784ea3-27d8-a99f-c46b-d18b76517333 Address:c2784ea3-27d8-a99f-c46b-d18b76517333} {Suffrage:Voter ID:server-b6e43e33-90be-268c-3669-de06d18531f7 Address:b6e43e33-90be-268c-3669-de06d18531f7}]"
2026-04-23T00:16:28.209Z [INFO]  server-ed9665e5-b724-2e27-b766-38f7639c506c: entering follower state: follower="Node at ed9665e5-b724-2e27-b766-38f7639c506c [Follower]" leader-address= leader-id=
2026-04-23T00:16:28.209Z [INFO]  server-c2784ea3-27d8-a99f-c46b-d18b76517333: entering follower state: follower="Node at c2784ea3-27d8-a99f-c46b-d18b76517333 [Follower]" leader-address= leader-id=
2026-04-23T00:16:28.209Z [INFO]  server-b6e43e33-90be-268c-3669-de06d18531f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed9665e5-b724-2e27-b766-38f7639c506c Address:ed9665e5-b724-2e27-b766-38f7639c506c} {Suffrage:Voter ID:server-c2784ea3-27d8-a99f-c46b-d18b76517333 Address:c2784ea3-27d8-a99f-c46b-d18b76517333} {Suffrage:Voter ID:server-b6e43e33-90be-268c-3669-de06d18531f7 Address:b6e43e33-90be-268c-3669-de06d18531f7}]"
2026-04-23T00:16:28.209Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:28.209Z [INFO]  server-b6e43e33-90be-268c-3669-de06d18531f7: entering follower state: follower="Node at b6e43e33-90be-268c-3669-de06d18531f7 [Follower]" leader-address= leader-id=
2026-04-23T00:16:28.261Z [WARN]  server-ed9665e5-b724-2e27-b766-38f7639c506c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:28.261Z [INFO]  server-ed9665e5-b724-2e27-b766-38f7639c506c: entering candidate state: node="Node at ed9665e5-b724-2e27-b766-38f7639c506c [Candidate]" term=2
2026-04-23T00:16:28.261Z [INFO]  server-ed9665e5-b724-2e27-b766-38f7639c506c: election won: term=2 tally=2
2026-04-23T00:16:28.261Z [INFO]  server-ed9665e5-b724-2e27-b766-38f7639c506c: entering leader state: leader="Node at ed9665e5-b724-2e27-b766-38f7639c506c [Leader]"
2026-04-23T00:16:28.261Z [INFO]  server-ed9665e5-b724-2e27-b766-38f7639c506c: added peer, starting replication: peer=server-c2784ea3-27d8-a99f-c46b-d18b76517333
2026-04-23T00:16:28.261Z [INFO]  server-ed9665e5-b724-2e27-b766-38f7639c506c: added peer, starting replication: peer=server-b6e43e33-90be-268c-3669-de06d18531f7
2026-04-23T00:16:28.261Z [INFO]  server-ed9665e5-b724-2e27-b766-38f7639c506c: pipelining replication: peer="{Voter server-b6e43e33-90be-268c-3669-de06d18531f7 b6e43e33-90be-268c-3669-de06d18531f7}"
2026-04-23T00:16:28.261Z [INFO]  server-ed9665e5-b724-2e27-b766-38f7639c506c: pipelining replication: peer="{Voter server-c2784ea3-27d8-a99f-c46b-d18b76517333 c2784ea3-27d8-a99f-c46b-d18b76517333}"
2026-04-23T00:16:28.367Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:28.261545306 +0000 UTC m=+105.301146667 (1 nodes), 51.817054ms from start of poll, 52.168884ms from cluster start. Timeout at 2026-04-23 00:16:28.366546618 +0000 UTC m=+105.406147972, 105.001305ms after stability
2026-04-23T00:16:28.367Z [INFO]  server-ed9665e5-b724-2e27-b766-38f7639c506c: aborting pipeline replication: peer="{Voter server-b6e43e33-90be-268c-3669-de06d18531f7 b6e43e33-90be-268c-3669-de06d18531f7}"
2026-04-23T00:16:28.367Z [INFO]  server-ed9665e5-b724-2e27-b766-38f7639c506c: aborting pipeline replication: peer="{Voter server-c2784ea3-27d8-a99f-c46b-d18b76517333 c2784ea3-27d8-a99f-c46b-d18b76517333}"
--- PASS: TestRaft_LeadershipTransferToInvalidID (0.16s)
=== RUN   TestRaft_LeadershipTransferToInvalidAddress
2026-04-23T00:16:28.368Z [INFO]  server-ef720151-ad35-e63a-6b73-4859755bb274: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef720151-ad35-e63a-6b73-4859755bb274 Address:ef720151-ad35-e63a-6b73-4859755bb274} {Suffrage:Voter ID:server-c668814a-df89-23a5-67f3-1c1576310f8f Address:c668814a-df89-23a5-67f3-1c1576310f8f} {Suffrage:Voter ID:server-cda5f80a-950e-bfb3-4418-c41fc85e1fc0 Address:cda5f80a-950e-bfb3-4418-c41fc85e1fc0}]"
2026-04-23T00:16:28.368Z [INFO]  server-c668814a-df89-23a5-67f3-1c1576310f8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef720151-ad35-e63a-6b73-4859755bb274 Address:ef720151-ad35-e63a-6b73-4859755bb274} {Suffrage:Voter ID:server-c668814a-df89-23a5-67f3-1c1576310f8f Address:c668814a-df89-23a5-67f3-1c1576310f8f} {Suffrage:Voter ID:server-cda5f80a-950e-bfb3-4418-c41fc85e1fc0 Address:cda5f80a-950e-bfb3-4418-c41fc85e1fc0}]"
2026-04-23T00:16:28.368Z [INFO]  server-c668814a-df89-23a5-67f3-1c1576310f8f: entering follower state: follower="Node at c668814a-df89-23a5-67f3-1c1576310f8f [Follower]" leader-address= leader-id=
2026-04-23T00:16:28.368Z [INFO]  server-ef720151-ad35-e63a-6b73-4859755bb274: entering follower state: follower="Node at ef720151-ad35-e63a-6b73-4859755bb274 [Follower]" leader-address= leader-id=
2026-04-23T00:16:28.368Z [INFO]  server-cda5f80a-950e-bfb3-4418-c41fc85e1fc0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef720151-ad35-e63a-6b73-4859755bb274 Address:ef720151-ad35-e63a-6b73-4859755bb274} {Suffrage:Voter ID:server-c668814a-df89-23a5-67f3-1c1576310f8f Address:c668814a-df89-23a5-67f3-1c1576310f8f} {Suffrage:Voter ID:server-cda5f80a-950e-bfb3-4418-c41fc85e1fc0 Address:cda5f80a-950e-bfb3-4418-c41fc85e1fc0}]"
2026-04-23T00:16:28.368Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:28.368Z [INFO]  server-cda5f80a-950e-bfb3-4418-c41fc85e1fc0: entering follower state: follower="Node at cda5f80a-950e-bfb3-4418-c41fc85e1fc0 [Follower]" leader-address= leader-id=
2026-04-23T00:16:28.432Z [WARN]  server-cda5f80a-950e-bfb3-4418-c41fc85e1fc0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:28.432Z [INFO]  server-cda5f80a-950e-bfb3-4418-c41fc85e1fc0: entering candidate state: node="Node at cda5f80a-950e-bfb3-4418-c41fc85e1fc0 [Candidate]" term=2
2026-04-23T00:16:28.432Z [INFO]  server-cda5f80a-950e-bfb3-4418-c41fc85e1fc0: election won: term=2 tally=2
2026-04-23T00:16:28.432Z [INFO]  server-cda5f80a-950e-bfb3-4418-c41fc85e1fc0: entering leader state: leader="Node at cda5f80a-950e-bfb3-4418-c41fc85e1fc0 [Leader]"
2026-04-23T00:16:28.432Z [INFO]  server-cda5f80a-950e-bfb3-4418-c41fc85e1fc0: added peer, starting replication: peer=server-ef720151-ad35-e63a-6b73-4859755bb274
2026-04-23T00:16:28.432Z [INFO]  server-cda5f80a-950e-bfb3-4418-c41fc85e1fc0: added peer, starting replication: peer=server-c668814a-df89-23a5-67f3-1c1576310f8f
2026-04-23T00:16:28.432Z [INFO]  server-cda5f80a-950e-bfb3-4418-c41fc85e1fc0: pipelining replication: peer="{Voter server-ef720151-ad35-e63a-6b73-4859755bb274 ef720151-ad35-e63a-6b73-4859755bb274}"
2026-04-23T00:16:28.432Z [INFO]  server-cda5f80a-950e-bfb3-4418-c41fc85e1fc0: pipelining replication: peer="{Voter server-c668814a-df89-23a5-67f3-1c1576310f8f c668814a-df89-23a5-67f3-1c1576310f8f}"
2026-04-23T00:16:28.537Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:28.432593574 +0000 UTC m=+105.472194930 (2 nodes), 64.087904ms from start of poll, 64.420747ms from cluster start. Timeout at 2026-04-23 00:16:28.537596515 +0000 UTC m=+105.577197867, 105.002937ms after stability
2026-04-23T00:16:28.537Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:28.642Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:28.537745009 +0000 UTC m=+105.577346361 (1 nodes), 1.577µs from start of poll, 169.572178ms from cluster start. Timeout at 2026-04-23 00:16:28.642769542 +0000 UTC m=+105.682370934, 105.024573ms after stability
2026-04-23T00:16:28.642Z [INFO]  server-cda5f80a-950e-bfb3-4418-c41fc85e1fc0: aborting pipeline replication: peer="{Voter server-ef720151-ad35-e63a-6b73-4859755bb274 ef720151-ad35-e63a-6b73-4859755bb274}"
2026-04-23T00:16:28.642Z [INFO]  server-cda5f80a-950e-bfb3-4418-c41fc85e1fc0: aborting pipeline replication: peer="{Voter server-c668814a-df89-23a5-67f3-1c1576310f8f c668814a-df89-23a5-67f3-1c1576310f8f}"
--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.28s)
=== RUN   TestRaft_LeadershipTransferToBehindServer
2026-04-23T00:16:28.643Z [INFO]  server-a21e279a-bed7-bd02-2dc8-b1f4b4fb862b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a21e279a-bed7-bd02-2dc8-b1f4b4fb862b Address:a21e279a-bed7-bd02-2dc8-b1f4b4fb862b} {Suffrage:Voter ID:server-37cdd95d-c0be-d38a-9675-0cb926c9c528 Address:37cdd95d-c0be-d38a-9675-0cb926c9c528} {Suffrage:Voter ID:server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5 Address:e2bfe3e4-0107-554d-a2e9-39018ba9f9d5}]"
2026-04-23T00:16:28.643Z [INFO]  server-37cdd95d-c0be-d38a-9675-0cb926c9c528: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a21e279a-bed7-bd02-2dc8-b1f4b4fb862b Address:a21e279a-bed7-bd02-2dc8-b1f4b4fb862b} {Suffrage:Voter ID:server-37cdd95d-c0be-d38a-9675-0cb926c9c528 Address:37cdd95d-c0be-d38a-9675-0cb926c9c528} {Suffrage:Voter ID:server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5 Address:e2bfe3e4-0107-554d-a2e9-39018ba9f9d5}]"
2026-04-23T00:16:28.643Z [INFO]  server-a21e279a-bed7-bd02-2dc8-b1f4b4fb862b: entering follower state: follower="Node at a21e279a-bed7-bd02-2dc8-b1f4b4fb862b [Follower]" leader-address= leader-id=
2026-04-23T00:16:28.644Z [INFO]  server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a21e279a-bed7-bd02-2dc8-b1f4b4fb862b Address:a21e279a-bed7-bd02-2dc8-b1f4b4fb862b} {Suffrage:Voter ID:server-37cdd95d-c0be-d38a-9675-0cb926c9c528 Address:37cdd95d-c0be-d38a-9675-0cb926c9c528} {Suffrage:Voter ID:server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5 Address:e2bfe3e4-0107-554d-a2e9-39018ba9f9d5}]"
2026-04-23T00:16:28.644Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:28.644Z [INFO]  server-37cdd95d-c0be-d38a-9675-0cb926c9c528: entering follower state: follower="Node at 37cdd95d-c0be-d38a-9675-0cb926c9c528 [Follower]" leader-address= leader-id=
2026-04-23T00:16:28.644Z [INFO]  server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5: entering follower state: follower="Node at e2bfe3e4-0107-554d-a2e9-39018ba9f9d5 [Follower]" leader-address= leader-id=
2026-04-23T00:16:28.703Z [WARN]  server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:28.703Z [INFO]  server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5: entering candidate state: node="Node at e2bfe3e4-0107-554d-a2e9-39018ba9f9d5 [Candidate]" term=2
2026-04-23T00:16:28.703Z [INFO]  server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5: election won: term=2 tally=2
2026-04-23T00:16:28.703Z [INFO]  server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5: entering leader state: leader="Node at e2bfe3e4-0107-554d-a2e9-39018ba9f9d5 [Leader]"
2026-04-23T00:16:28.703Z [INFO]  server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5: added peer, starting replication: peer=server-a21e279a-bed7-bd02-2dc8-b1f4b4fb862b
2026-04-23T00:16:28.703Z [INFO]  server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5: added peer, starting replication: peer=server-37cdd95d-c0be-d38a-9675-0cb926c9c528
2026-04-23T00:16:28.704Z [INFO]  server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5: pipelining replication: peer="{Voter server-37cdd95d-c0be-d38a-9675-0cb926c9c528 37cdd95d-c0be-d38a-9675-0cb926c9c528}"
2026-04-23T00:16:28.704Z [INFO]  server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5: pipelining replication: peer="{Voter server-a21e279a-bed7-bd02-2dc8-b1f4b4fb862b a21e279a-bed7-bd02-2dc8-b1f4b4fb862b}"
2026-04-23T00:16:28.809Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:28.703990075 +0000 UTC m=+105.743591427 (1 nodes), 59.944806ms from start of poll, 60.17741ms from cluster start. Timeout at 2026-04-23 00:16:28.809000151 +0000 UTC m=+105.848601544, 105.010117ms after stability
2026-04-23T00:16:28.809Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:28.915Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:28.809925153 +0000 UTC m=+105.849526543 (2 nodes), 2.429µs from start of poll, 166.112526ms from cluster start. Timeout at 2026-04-23 00:16:28.914955924 +0000 UTC m=+105.954557313, 105.03077ms after stability
2026-04-23T00:16:28.944Z [INFO]  server-a21e279a-bed7-bd02-2dc8-b1f4b4fb862b: entering candidate state: node="Node at a21e279a-bed7-bd02-2dc8-b1f4b4fb862b [Candidate]" term=3
2026-04-23T00:16:28.944Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:28.944Z [INFO]  server-a21e279a-bed7-bd02-2dc8-b1f4b4fb862b: election won: term=3 tally=2
2026-04-23T00:16:28.944Z [INFO]  server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5: entering follower state: follower="Node at e2bfe3e4-0107-554d-a2e9-39018ba9f9d5 [Follower]" leader-address= leader-id=
2026-04-23T00:16:28.944Z [INFO]  server-a21e279a-bed7-bd02-2dc8-b1f4b4fb862b: entering leader state: leader="Node at a21e279a-bed7-bd02-2dc8-b1f4b4fb862b [Leader]"
2026-04-23T00:16:28.944Z [INFO]  server-a21e279a-bed7-bd02-2dc8-b1f4b4fb862b: added peer, starting replication: peer=server-37cdd95d-c0be-d38a-9675-0cb926c9c528
2026-04-23T00:16:28.944Z [INFO]  server-a21e279a-bed7-bd02-2dc8-b1f4b4fb862b: added peer, starting replication: peer=server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5
2026-04-23T00:16:28.944Z [INFO]  server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5: aborting pipeline replication: peer="{Voter server-a21e279a-bed7-bd02-2dc8-b1f4b4fb862b a21e279a-bed7-bd02-2dc8-b1f4b4fb862b}"
2026-04-23T00:16:28.944Z [INFO]  server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5: aborting pipeline replication: peer="{Voter server-37cdd95d-c0be-d38a-9675-0cb926c9c528 37cdd95d-c0be-d38a-9675-0cb926c9c528}"
2026-04-23T00:16:28.944Z [INFO]  server-a21e279a-bed7-bd02-2dc8-b1f4b4fb862b: pipelining replication: peer="{Voter server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5 e2bfe3e4-0107-554d-a2e9-39018ba9f9d5}"
2026-04-23T00:16:28.945Z [INFO]  server-a21e279a-bed7-bd02-2dc8-b1f4b4fb862b: pipelining replication: peer="{Voter server-37cdd95d-c0be-d38a-9675-0cb926c9c528 37cdd95d-c0be-d38a-9675-0cb926c9c528}"
2026-04-23T00:16:29.050Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:28.944881454 +0000 UTC m=+105.984482810 (1 nodes), 35.043µs from start of poll, 301.068793ms from cluster start. Timeout at 2026-04-23 00:16:29.049908812 +0000 UTC m=+106.089510187, 105.027377ms after stability
2026-04-23T00:16:29.050Z [INFO]  server-a21e279a-bed7-bd02-2dc8-b1f4b4fb862b: aborting pipeline replication: peer="{Voter server-e2bfe3e4-0107-554d-a2e9-39018ba9f9d5 e2bfe3e4-0107-554d-a2e9-39018ba9f9d5}"
2026-04-23T00:16:29.050Z [INFO]  server-a21e279a-bed7-bd02-2dc8-b1f4b4fb862b: aborting pipeline replication: peer="{Voter server-37cdd95d-c0be-d38a-9675-0cb926c9c528 37cdd95d-c0be-d38a-9675-0cb926c9c528}"
--- PASS: TestRaft_LeadershipTransferToBehindServer (0.41s)
=== RUN   TestRaft_LeadershipTransferToItself
2026-04-23T00:16:29.051Z [INFO]  server-10702ef9-c5a4-41fd-8e83-387fc2bfd21e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-10702ef9-c5a4-41fd-8e83-387fc2bfd21e Address:10702ef9-c5a4-41fd-8e83-387fc2bfd21e} {Suffrage:Voter ID:server-0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272 Address:0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272} {Suffrage:Voter ID:server-a38eb36f-f9c2-bb38-8a25-8d2bbc8f45f6 Address:a38eb36f-f9c2-bb38-8a25-8d2bbc8f45f6}]"
2026-04-23T00:16:29.051Z [INFO]  server-0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-10702ef9-c5a4-41fd-8e83-387fc2bfd21e Address:10702ef9-c5a4-41fd-8e83-387fc2bfd21e} {Suffrage:Voter ID:server-0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272 Address:0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272} {Suffrage:Voter ID:server-a38eb36f-f9c2-bb38-8a25-8d2bbc8f45f6 Address:a38eb36f-f9c2-bb38-8a25-8d2bbc8f45f6}]"
2026-04-23T00:16:29.051Z [INFO]  server-10702ef9-c5a4-41fd-8e83-387fc2bfd21e: entering follower state: follower="Node at 10702ef9-c5a4-41fd-8e83-387fc2bfd21e [Follower]" leader-address= leader-id=
2026-04-23T00:16:29.051Z [INFO]  server-0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272: entering follower state: follower="Node at 0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272 [Follower]" leader-address= leader-id=
2026-04-23T00:16:29.051Z [INFO]  server-a38eb36f-f9c2-bb38-8a25-8d2bbc8f45f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-10702ef9-c5a4-41fd-8e83-387fc2bfd21e Address:10702ef9-c5a4-41fd-8e83-387fc2bfd21e} {Suffrage:Voter ID:server-0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272 Address:0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272} {Suffrage:Voter ID:server-a38eb36f-f9c2-bb38-8a25-8d2bbc8f45f6 Address:a38eb36f-f9c2-bb38-8a25-8d2bbc8f45f6}]"
2026-04-23T00:16:29.051Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:29.051Z [INFO]  server-a38eb36f-f9c2-bb38-8a25-8d2bbc8f45f6: entering follower state: follower="Node at a38eb36f-f9c2-bb38-8a25-8d2bbc8f45f6 [Follower]" leader-address= leader-id=
2026-04-23T00:16:29.128Z [WARN]  server-0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:29.128Z [INFO]  server-0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272: entering candidate state: node="Node at 0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272 [Candidate]" term=2
2026-04-23T00:16:29.128Z [INFO]  server-0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272: election won: term=2 tally=2
2026-04-23T00:16:29.128Z [INFO]  server-0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272: entering leader state: leader="Node at 0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272 [Leader]"
2026-04-23T00:16:29.128Z [INFO]  server-0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272: added peer, starting replication: peer=server-10702ef9-c5a4-41fd-8e83-387fc2bfd21e
2026-04-23T00:16:29.128Z [INFO]  server-0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272: added peer, starting replication: peer=server-a38eb36f-f9c2-bb38-8a25-8d2bbc8f45f6
2026-04-23T00:16:29.128Z [WARN]  server-10702ef9-c5a4-41fd-8e83-387fc2bfd21e: rejecting vote request since our last term is greater: candidate=0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272 last-term=2 last-candidate-term=1
2026-04-23T00:16:29.128Z [INFO]  server-0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272: pipelining replication: peer="{Voter server-a38eb36f-f9c2-bb38-8a25-8d2bbc8f45f6 a38eb36f-f9c2-bb38-8a25-8d2bbc8f45f6}"
2026-04-23T00:16:29.128Z [INFO]  server-0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272: pipelining replication: peer="{Voter server-10702ef9-c5a4-41fd-8e83-387fc2bfd21e 10702ef9-c5a4-41fd-8e83-387fc2bfd21e}"
2026-04-23T00:16:29.234Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:29.128672159 +0000 UTC m=+106.168273516 (1 nodes), 76.890005ms from start of poll, 77.220341ms from cluster start. Timeout at 2026-04-23 00:16:29.233680584 +0000 UTC m=+106.273281941, 105.008425ms after stability
2026-04-23T00:16:29.234Z [INFO]  server-0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272: cannot transfer leadership to itself
2026-04-23T00:16:29.234Z [INFO]  server-0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272: aborting pipeline replication: peer="{Voter server-a38eb36f-f9c2-bb38-8a25-8d2bbc8f45f6 a38eb36f-f9c2-bb38-8a25-8d2bbc8f45f6}"
2026-04-23T00:16:29.234Z [INFO]  server-0b3d8844-2a6f-ffc4-7c4a-3e78cf75b272: aborting pipeline replication: peer="{Voter server-10702ef9-c5a4-41fd-8e83-387fc2bfd21e 10702ef9-c5a4-41fd-8e83-387fc2bfd21e}"
--- PASS: TestRaft_LeadershipTransferToItself (0.18s)
=== RUN   TestRaft_LeadershipTransferLeaderRejectsClientRequests
2026-04-23T00:16:29.235Z [INFO]  server-b5985bb8-cf37-ec01-510d-85c99196c80c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5985bb8-cf37-ec01-510d-85c99196c80c Address:b5985bb8-cf37-ec01-510d-85c99196c80c} {Suffrage:Voter ID:server-19981d91-366d-16b5-c1fb-d766399435fc Address:19981d91-366d-16b5-c1fb-d766399435fc} {Suffrage:Voter ID:server-3d09d17a-3240-dd97-9714-c4423eb6d13b Address:3d09d17a-3240-dd97-9714-c4423eb6d13b}]"
2026-04-23T00:16:29.235Z [INFO]  server-19981d91-366d-16b5-c1fb-d766399435fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5985bb8-cf37-ec01-510d-85c99196c80c Address:b5985bb8-cf37-ec01-510d-85c99196c80c} {Suffrage:Voter ID:server-19981d91-366d-16b5-c1fb-d766399435fc Address:19981d91-366d-16b5-c1fb-d766399435fc} {Suffrage:Voter ID:server-3d09d17a-3240-dd97-9714-c4423eb6d13b Address:3d09d17a-3240-dd97-9714-c4423eb6d13b}]"
2026-04-23T00:16:29.235Z [INFO]  server-b5985bb8-cf37-ec01-510d-85c99196c80c: entering follower state: follower="Node at b5985bb8-cf37-ec01-510d-85c99196c80c [Follower]" leader-address= leader-id=
2026-04-23T00:16:29.235Z [INFO]  server-19981d91-366d-16b5-c1fb-d766399435fc: entering follower state: follower="Node at 19981d91-366d-16b5-c1fb-d766399435fc [Follower]" leader-address= leader-id=
2026-04-23T00:16:29.235Z [INFO]  server-3d09d17a-3240-dd97-9714-c4423eb6d13b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5985bb8-cf37-ec01-510d-85c99196c80c Address:b5985bb8-cf37-ec01-510d-85c99196c80c} {Suffrage:Voter ID:server-19981d91-366d-16b5-c1fb-d766399435fc Address:19981d91-366d-16b5-c1fb-d766399435fc} {Suffrage:Voter ID:server-3d09d17a-3240-dd97-9714-c4423eb6d13b Address:3d09d17a-3240-dd97-9714-c4423eb6d13b}]"
2026-04-23T00:16:29.235Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:29.235Z [INFO]  server-3d09d17a-3240-dd97-9714-c4423eb6d13b: entering follower state: follower="Node at 3d09d17a-3240-dd97-9714-c4423eb6d13b [Follower]" leader-address= leader-id=
2026-04-23T00:16:29.307Z [WARN]  server-b5985bb8-cf37-ec01-510d-85c99196c80c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:29.307Z [INFO]  server-b5985bb8-cf37-ec01-510d-85c99196c80c: entering candidate state: node="Node at b5985bb8-cf37-ec01-510d-85c99196c80c [Candidate]" term=2
2026-04-23T00:16:29.307Z [INFO]  server-b5985bb8-cf37-ec01-510d-85c99196c80c: election won: term=2 tally=2
2026-04-23T00:16:29.307Z [INFO]  server-b5985bb8-cf37-ec01-510d-85c99196c80c: entering leader state: leader="Node at b5985bb8-cf37-ec01-510d-85c99196c80c [Leader]"
2026-04-23T00:16:29.307Z [INFO]  server-b5985bb8-cf37-ec01-510d-85c99196c80c: added peer, starting replication: peer=server-19981d91-366d-16b5-c1fb-d766399435fc
2026-04-23T00:16:29.307Z [INFO]  server-b5985bb8-cf37-ec01-510d-85c99196c80c: added peer, starting replication: peer=server-3d09d17a-3240-dd97-9714-c4423eb6d13b
2026-04-23T00:16:29.307Z [INFO]  server-b5985bb8-cf37-ec01-510d-85c99196c80c: pipelining replication: peer="{Voter server-3d09d17a-3240-dd97-9714-c4423eb6d13b 3d09d17a-3240-dd97-9714-c4423eb6d13b}"
2026-04-23T00:16:29.307Z [INFO]  server-b5985bb8-cf37-ec01-510d-85c99196c80c: pipelining replication: peer="{Voter server-19981d91-366d-16b5-c1fb-d766399435fc 19981d91-366d-16b5-c1fb-d766399435fc}"
2026-04-23T00:16:29.413Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:29.307712886 +0000 UTC m=+106.347314242 (1 nodes), 72.039013ms from start of poll, 72.368849ms from cluster start. Timeout at 2026-04-23 00:16:29.412714374 +0000 UTC m=+106.452315726, 105.001484ms after stability
    raft_test.go:2457: waiting on future 0
    raft_test.go:2457: waiting on future 1
    raft_test.go:2457: waiting on future 2
    raft_test.go:2457: waiting on future 3
    raft_test.go:2457: waiting on future 4
    raft_test.go:2457: waiting on future 5
    raft_test.go:2457: waiting on future 6
2026-04-23T00:16:29.413Z [INFO]  server-b5985bb8-cf37-ec01-510d-85c99196c80c: aborting pipeline replication: peer="{Voter server-3d09d17a-3240-dd97-9714-c4423eb6d13b 3d09d17a-3240-dd97-9714-c4423eb6d13b}"
2026-04-23T00:16:29.413Z [INFO]  server-b5985bb8-cf37-ec01-510d-85c99196c80c: aborting pipeline replication: peer="{Voter server-19981d91-366d-16b5-c1fb-d766399435fc 19981d91-366d-16b5-c1fb-d766399435fc}"
--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.18s)
=== RUN   TestRaft_LeadershipTransferLeaderReplicationTimeout
2026-04-23T00:16:29.415Z [INFO]  server-26f82934-cd34-a4f1-db18-e1c5bbcb4e4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26f82934-cd34-a4f1-db18-e1c5bbcb4e4b Address:26f82934-cd34-a4f1-db18-e1c5bbcb4e4b} {Suffrage:Voter ID:server-38cc519e-539e-0ff6-257e-a75592a9dc70 Address:38cc519e-539e-0ff6-257e-a75592a9dc70} {Suffrage:Voter ID:server-2aff7b4d-3db4-0877-b8a0-fc275b68ed46 Address:2aff7b4d-3db4-0877-b8a0-fc275b68ed46}]"
2026-04-23T00:16:29.415Z [INFO]  server-38cc519e-539e-0ff6-257e-a75592a9dc70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26f82934-cd34-a4f1-db18-e1c5bbcb4e4b Address:26f82934-cd34-a4f1-db18-e1c5bbcb4e4b} {Suffrage:Voter ID:server-38cc519e-539e-0ff6-257e-a75592a9dc70 Address:38cc519e-539e-0ff6-257e-a75592a9dc70} {Suffrage:Voter ID:server-2aff7b4d-3db4-0877-b8a0-fc275b68ed46 Address:2aff7b4d-3db4-0877-b8a0-fc275b68ed46}]"
2026-04-23T00:16:29.415Z [INFO]  server-26f82934-cd34-a4f1-db18-e1c5bbcb4e4b: entering follower state: follower="Node at 26f82934-cd34-a4f1-db18-e1c5bbcb4e4b [Follower]" leader-address= leader-id=
2026-04-23T00:16:29.415Z [INFO]  server-38cc519e-539e-0ff6-257e-a75592a9dc70: entering follower state: follower="Node at 38cc519e-539e-0ff6-257e-a75592a9dc70 [Follower]" leader-address= leader-id=
2026-04-23T00:16:29.415Z [INFO]  server-2aff7b4d-3db4-0877-b8a0-fc275b68ed46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-26f82934-cd34-a4f1-db18-e1c5bbcb4e4b Address:26f82934-cd34-a4f1-db18-e1c5bbcb4e4b} {Suffrage:Voter ID:server-38cc519e-539e-0ff6-257e-a75592a9dc70 Address:38cc519e-539e-0ff6-257e-a75592a9dc70} {Suffrage:Voter ID:server-2aff7b4d-3db4-0877-b8a0-fc275b68ed46 Address:2aff7b4d-3db4-0877-b8a0-fc275b68ed46}]"
2026-04-23T00:16:29.415Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:29.415Z [INFO]  server-2aff7b4d-3db4-0877-b8a0-fc275b68ed46: entering follower state: follower="Node at 2aff7b4d-3db4-0877-b8a0-fc275b68ed46 [Follower]" leader-address= leader-id=
2026-04-23T00:16:29.479Z [WARN]  server-26f82934-cd34-a4f1-db18-e1c5bbcb4e4b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:29.479Z [INFO]  server-26f82934-cd34-a4f1-db18-e1c5bbcb4e4b: entering candidate state: node="Node at 26f82934-cd34-a4f1-db18-e1c5bbcb4e4b [Candidate]" term=2
2026-04-23T00:16:29.479Z [INFO]  server-26f82934-cd34-a4f1-db18-e1c5bbcb4e4b: election won: term=2 tally=2
2026-04-23T00:16:29.479Z [INFO]  server-26f82934-cd34-a4f1-db18-e1c5bbcb4e4b: entering leader state: leader="Node at 26f82934-cd34-a4f1-db18-e1c5bbcb4e4b [Leader]"
2026-04-23T00:16:29.479Z [INFO]  server-26f82934-cd34-a4f1-db18-e1c5bbcb4e4b: added peer, starting replication: peer=server-38cc519e-539e-0ff6-257e-a75592a9dc70
2026-04-23T00:16:29.479Z [INFO]  server-26f82934-cd34-a4f1-db18-e1c5bbcb4e4b: added peer, starting replication: peer=server-2aff7b4d-3db4-0877-b8a0-fc275b68ed46
2026-04-23T00:16:29.480Z [INFO]  server-26f82934-cd34-a4f1-db18-e1c5bbcb4e4b: pipelining replication: peer="{Voter server-2aff7b4d-3db4-0877-b8a0-fc275b68ed46 2aff7b4d-3db4-0877-b8a0-fc275b68ed46}"
2026-04-23T00:16:29.480Z [INFO]  server-26f82934-cd34-a4f1-db18-e1c5bbcb4e4b: pipelining replication: peer="{Voter server-38cc519e-539e-0ff6-257e-a75592a9dc70 38cc519e-539e-0ff6-257e-a75592a9dc70}"
2026-04-23T00:16:29.585Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:29.480018933 +0000 UTC m=+106.519620306 (1 nodes), 64.625537ms from start of poll, 64.984529ms from cluster start. Timeout at 2026-04-23 00:16:29.58502044 +0000 UTC m=+106.624621813, 105.001507ms after stability
2026-04-23T00:16:29.585Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:29.691Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:29.585828265 +0000 UTC m=+106.625429622 (2 nodes), 1.506µs from start of poll, 170.793845ms from cluster start. Timeout at 2026-04-23 00:16:29.690848234 +0000 UTC m=+106.730449613, 105.019991ms after stability
2026-04-23T00:16:29.942Z [INFO]  server-26f82934-cd34-a4f1-db18-e1c5bbcb4e4b: aborting pipeline replication: peer="{Voter server-2aff7b4d-3db4-0877-b8a0-fc275b68ed46 2aff7b4d-3db4-0877-b8a0-fc275b68ed46}"
2026-04-23T00:16:29.942Z [INFO]  server-26f82934-cd34-a4f1-db18-e1c5bbcb4e4b: aborting pipeline replication: peer="{Voter server-38cc519e-539e-0ff6-257e-a75592a9dc70 38cc519e-539e-0ff6-257e-a75592a9dc70}"
--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.53s)
=== RUN   TestRaft_LeadershipTransferIgnoresNonvoters
2026-04-23T00:16:29.943Z [INFO]  server-4f02c11a-f57b-447d-5e7b-a5fdbd7a4a16: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f02c11a-f57b-447d-5e7b-a5fdbd7a4a16 Address:4f02c11a-f57b-447d-5e7b-a5fdbd7a4a16} {Suffrage:Voter ID:server-52ead5f9-af38-b2d6-77d9-b9fa5888b5ab Address:52ead5f9-af38-b2d6-77d9-b9fa5888b5ab}]"
2026-04-23T00:16:29.943Z [INFO]  server-52ead5f9-af38-b2d6-77d9-b9fa5888b5ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f02c11a-f57b-447d-5e7b-a5fdbd7a4a16 Address:4f02c11a-f57b-447d-5e7b-a5fdbd7a4a16} {Suffrage:Voter ID:server-52ead5f9-af38-b2d6-77d9-b9fa5888b5ab Address:52ead5f9-af38-b2d6-77d9-b9fa5888b5ab}]"
2026-04-23T00:16:29.943Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:29.943Z [INFO]  server-52ead5f9-af38-b2d6-77d9-b9fa5888b5ab: entering follower state: follower="Node at 52ead5f9-af38-b2d6-77d9-b9fa5888b5ab [Follower]" leader-address= leader-id=
2026-04-23T00:16:29.943Z [INFO]  server-4f02c11a-f57b-447d-5e7b-a5fdbd7a4a16: entering follower state: follower="Node at 4f02c11a-f57b-447d-5e7b-a5fdbd7a4a16 [Follower]" leader-address= leader-id=
2026-04-23T00:16:30.013Z [WARN]  server-52ead5f9-af38-b2d6-77d9-b9fa5888b5ab: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:30.013Z [INFO]  server-52ead5f9-af38-b2d6-77d9-b9fa5888b5ab: entering candidate state: node="Node at 52ead5f9-af38-b2d6-77d9-b9fa5888b5ab [Candidate]" term=2
2026-04-23T00:16:30.013Z [INFO]  server-52ead5f9-af38-b2d6-77d9-b9fa5888b5ab: election won: term=2 tally=2
2026-04-23T00:16:30.013Z [INFO]  server-52ead5f9-af38-b2d6-77d9-b9fa5888b5ab: entering leader state: leader="Node at 52ead5f9-af38-b2d6-77d9-b9fa5888b5ab [Leader]"
2026-04-23T00:16:30.013Z [INFO]  server-52ead5f9-af38-b2d6-77d9-b9fa5888b5ab: added peer, starting replication: peer=server-4f02c11a-f57b-447d-5e7b-a5fdbd7a4a16
2026-04-23T00:16:30.013Z [INFO]  server-52ead5f9-af38-b2d6-77d9-b9fa5888b5ab: pipelining replication: peer="{Voter server-4f02c11a-f57b-447d-5e7b-a5fdbd7a4a16 4f02c11a-f57b-447d-5e7b-a5fdbd7a4a16}"
2026-04-23T00:16:30.118Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:30.013370319 +0000 UTC m=+107.052971675 (1 nodes), 69.965615ms from start of poll, 70.258379ms from cluster start. Timeout at 2026-04-23 00:16:30.118372892 +0000 UTC m=+107.157974248, 105.002573ms after stability
2026-04-23T00:16:30.118Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:30.223Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:30.11852828 +0000 UTC m=+107.158129637 (1 nodes), 792ns from start of poll, 175.416341ms from cluster start. Timeout at 2026-04-23 00:16:30.223585601 +0000 UTC m=+107.263186957, 105.05732ms after stability
2026-04-23T00:16:30.223Z [INFO]  server-52ead5f9-af38-b2d6-77d9-b9fa5888b5ab: updating configuration: command=DemoteVoter server-id=server-4f02c11a-f57b-447d-5e7b-a5fdbd7a4a16 server-addr= servers="[{Suffrage:Nonvoter ID:server-4f02c11a-f57b-447d-5e7b-a5fdbd7a4a16 Address:4f02c11a-f57b-447d-5e7b-a5fdbd7a4a16} {Suffrage:Voter ID:server-52ead5f9-af38-b2d6-77d9-b9fa5888b5ab Address:52ead5f9-af38-b2d6-77d9-b9fa5888b5ab}]"
2026-04-23T00:16:30.223Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:30.329Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:30.223753325 +0000 UTC m=+107.263354678 (1 nodes), 595ns from start of poll, 280.641382ms from cluster start. Timeout at 2026-04-23 00:16:30.328790215 +0000 UTC m=+107.368391596, 105.036918ms after stability
2026-04-23T00:16:30.329Z [INFO]  server-52ead5f9-af38-b2d6-77d9-b9fa5888b5ab: aborting pipeline replication: peer="{Voter server-4f02c11a-f57b-447d-5e7b-a5fdbd7a4a16 4f02c11a-f57b-447d-5e7b-a5fdbd7a4a16}"
--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.39s)
=== RUN   TestRaft_LeadershipTransferStopRightAway
--- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s)
=== RUN   TestRaft_GetConfigurationNoBootstrap
2026-04-23T00:16:30.330Z [INFO]  server-44a2cee4-2351-3fa4-3837-aa7ec9bfb09c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44a2cee4-2351-3fa4-3837-aa7ec9bfb09c Address:44a2cee4-2351-3fa4-3837-aa7ec9bfb09c} {Suffrage:Voter ID:server-25e03f72-e33e-144a-acb2-a6a63a4e5832 Address:25e03f72-e33e-144a-acb2-a6a63a4e5832}]"
2026-04-23T00:16:30.330Z [INFO]  server-25e03f72-e33e-144a-acb2-a6a63a4e5832: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44a2cee4-2351-3fa4-3837-aa7ec9bfb09c Address:44a2cee4-2351-3fa4-3837-aa7ec9bfb09c} {Suffrage:Voter ID:server-25e03f72-e33e-144a-acb2-a6a63a4e5832 Address:25e03f72-e33e-144a-acb2-a6a63a4e5832}]"
2026-04-23T00:16:30.330Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:30.330Z [INFO]  server-44a2cee4-2351-3fa4-3837-aa7ec9bfb09c: entering follower state: follower="Node at 44a2cee4-2351-3fa4-3837-aa7ec9bfb09c [Follower]" leader-address= leader-id=
2026-04-23T00:16:30.330Z [INFO]  server-25e03f72-e33e-144a-acb2-a6a63a4e5832: entering follower state: follower="Node at 25e03f72-e33e-144a-acb2-a6a63a4e5832 [Follower]" leader-address= leader-id=
2026-04-23T00:16:30.414Z [WARN]  server-44a2cee4-2351-3fa4-3837-aa7ec9bfb09c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:30.414Z [INFO]  server-44a2cee4-2351-3fa4-3837-aa7ec9bfb09c: entering candidate state: node="Node at 44a2cee4-2351-3fa4-3837-aa7ec9bfb09c [Candidate]" term=2
2026-04-23T00:16:30.414Z [INFO]  server-44a2cee4-2351-3fa4-3837-aa7ec9bfb09c: election won: term=2 tally=2
2026-04-23T00:16:30.414Z [INFO]  server-44a2cee4-2351-3fa4-3837-aa7ec9bfb09c: entering leader state: leader="Node at 44a2cee4-2351-3fa4-3837-aa7ec9bfb09c [Leader]"
2026-04-23T00:16:30.414Z [INFO]  server-44a2cee4-2351-3fa4-3837-aa7ec9bfb09c: added peer, starting replication: peer=server-25e03f72-e33e-144a-acb2-a6a63a4e5832
2026-04-23T00:16:30.414Z [INFO]  server-44a2cee4-2351-3fa4-3837-aa7ec9bfb09c: pipelining replication: peer="{Voter server-25e03f72-e33e-144a-acb2-a6a63a4e5832 25e03f72-e33e-144a-acb2-a6a63a4e5832}"
2026-04-23T00:16:30.520Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:30.414909581 +0000 UTC m=+107.454510933 (1 nodes), 84.127143ms from start of poll, 84.339083ms from cluster start. Timeout at 2026-04-23 00:16:30.519913901 +0000 UTC m=+107.559515258, 105.004325ms after stability
2026-04-23T00:16:30.520Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:30.627Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:30.520990796 +0000 UTC m=+107.560592154 (1 nodes), 531ns from start of poll, 190.420304ms from cluster start. Timeout at 2026-04-23 00:16:30.626015013 +0000 UTC m=+107.665616404, 105.02425ms after stability
2026-04-23T00:16:30.638Z [INFO]  server-25e03f72-e33e-144a-acb2-a6a63a4e5832: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44a2cee4-2351-3fa4-3837-aa7ec9bfb09c Address:44a2cee4-2351-3fa4-3837-aa7ec9bfb09c} {Suffrage:Voter ID:server-25e03f72-e33e-144a-acb2-a6a63a4e5832 Address:25e03f72-e33e-144a-acb2-a6a63a4e5832}]"
2026-04-23T00:16:30.638Z [INFO]  server-44a2cee4-2351-3fa4-3837-aa7ec9bfb09c: aborting pipeline replication: peer="{Voter server-25e03f72-e33e-144a-acb2-a6a63a4e5832 25e03f72-e33e-144a-acb2-a6a63a4e5832}"
--- PASS: TestRaft_GetConfigurationNoBootstrap (0.31s)
=== RUN   TestRaft_LogStoreIsMonotonic
2026-04-23T00:16:30.639Z [INFO]  server-aaebf406-1e25-a8eb-ab08-e5b6a7d37356: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aaebf406-1e25-a8eb-ab08-e5b6a7d37356 Address:aaebf406-1e25-a8eb-ab08-e5b6a7d37356}]"
2026-04-23T00:16:30.639Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:30.639Z [INFO]  server-aaebf406-1e25-a8eb-ab08-e5b6a7d37356: entering follower state: follower="Node at aaebf406-1e25-a8eb-ab08-e5b6a7d37356 [Follower]" leader-address= leader-id=
2026-04-23T00:16:30.710Z [WARN]  server-aaebf406-1e25-a8eb-ab08-e5b6a7d37356: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:30.710Z [INFO]  server-aaebf406-1e25-a8eb-ab08-e5b6a7d37356: entering candidate state: node="Node at aaebf406-1e25-a8eb-ab08-e5b6a7d37356 [Candidate]" term=2
2026-04-23T00:16:30.712Z [INFO]  server-aaebf406-1e25-a8eb-ab08-e5b6a7d37356: election won: term=2 tally=1
2026-04-23T00:16:30.712Z [INFO]  server-aaebf406-1e25-a8eb-ab08-e5b6a7d37356: entering leader state: leader="Node at aaebf406-1e25-a8eb-ab08-e5b6a7d37356 [Leader]"
2026-04-23T00:16:30.817Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:30.712626021 +0000 UTC m=+107.752227374 (1 nodes), 73.079098ms from start of poll, 73.206076ms from cluster start. Timeout at 2026-04-23 00:16:30.817630276 +0000 UTC m=+107.857231631, 105.004257ms after stability
--- PASS: TestRaft_LogStoreIsMonotonic (0.18s)
=== RUN   TestRaft_CacheLogWithStoreError
2026-04-23T00:16:30.818Z [INFO]  server-5528b28d-58e1-8d3a-c568-50a7b1843615: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5528b28d-58e1-8d3a-c568-50a7b1843615 Address:5528b28d-58e1-8d3a-c568-50a7b1843615} {Suffrage:Voter ID:server-51db00a8-0d73-438d-5bfa-1e39823b6fd4 Address:51db00a8-0d73-438d-5bfa-1e39823b6fd4}]"
2026-04-23T00:16:30.818Z [INFO]  server-51db00a8-0d73-438d-5bfa-1e39823b6fd4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5528b28d-58e1-8d3a-c568-50a7b1843615 Address:5528b28d-58e1-8d3a-c568-50a7b1843615} {Suffrage:Voter ID:server-51db00a8-0d73-438d-5bfa-1e39823b6fd4 Address:51db00a8-0d73-438d-5bfa-1e39823b6fd4}]"
2026-04-23T00:16:30.818Z [INFO]  server-5528b28d-58e1-8d3a-c568-50a7b1843615: entering follower state: follower="Node at 5528b28d-58e1-8d3a-c568-50a7b1843615 [Follower]" leader-address= leader-id=
2026-04-23T00:16:30.818Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:30.818Z [INFO]  server-51db00a8-0d73-438d-5bfa-1e39823b6fd4: entering follower state: follower="Node at 51db00a8-0d73-438d-5bfa-1e39823b6fd4 [Follower]" leader-address= leader-id=
2026-04-23T00:16:30.876Z [WARN]  server-5528b28d-58e1-8d3a-c568-50a7b1843615: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:30.876Z [INFO]  server-5528b28d-58e1-8d3a-c568-50a7b1843615: entering candidate state: node="Node at 5528b28d-58e1-8d3a-c568-50a7b1843615 [Candidate]" term=2
2026-04-23T00:16:30.876Z [INFO]  server-5528b28d-58e1-8d3a-c568-50a7b1843615: election won: term=2 tally=2
2026-04-23T00:16:30.876Z [INFO]  server-5528b28d-58e1-8d3a-c568-50a7b1843615: entering leader state: leader="Node at 5528b28d-58e1-8d3a-c568-50a7b1843615 [Leader]"
2026-04-23T00:16:30.876Z [INFO]  server-5528b28d-58e1-8d3a-c568-50a7b1843615: added peer, starting replication: peer=server-51db00a8-0d73-438d-5bfa-1e39823b6fd4
2026-04-23T00:16:30.876Z [INFO]  server-5528b28d-58e1-8d3a-c568-50a7b1843615: pipelining replication: peer="{Voter server-51db00a8-0d73-438d-5bfa-1e39823b6fd4 51db00a8-0d73-438d-5bfa-1e39823b6fd4}"
2026-04-23T00:16:30.982Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:30.876257533 +0000 UTC m=+107.915858889 (1 nodes), 57.628854ms from start of poll, 57.827904ms from cluster start. Timeout at 2026-04-23 00:16:30.981269366 +0000 UTC m=+108.020870721, 105.011832ms after stability
2026-04-23T00:16:30.982Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:31.088Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:30.982058026 +0000 UTC m=+108.021659383 (1 nodes), 892ns from start of poll, 163.628398ms from cluster start. Timeout at 2026-04-23 00:16:31.08708006 +0000 UTC m=+108.126681415, 105.022032ms after stability
2026-04-23T00:16:31.088Z [ERROR] server-51db00a8-0d73-438d-5bfa-1e39823b6fd4: failed to append to logs: error="unable to store logs within log store, err: "some error""
2026-04-23T00:16:31.088Z [WARN]  server-51db00a8-0d73-438d-5bfa-1e39823b6fd4: failed to get previous log: previous-index=9 last-index=7 error="log not found"
2026-04-23T00:16:31.088Z [INFO]  server-5528b28d-58e1-8d3a-c568-50a7b1843615: aborting pipeline replication: peer="{Voter server-51db00a8-0d73-438d-5bfa-1e39823b6fd4 51db00a8-0d73-438d-5bfa-1e39823b6fd4}"
2026-04-23T00:16:31.097Z [INFO]  server-5528b28d-58e1-8d3a-c568-50a7b1843615: pipelining replication: peer="{Voter server-51db00a8-0d73-438d-5bfa-1e39823b6fd4 51db00a8-0d73-438d-5bfa-1e39823b6fd4}"
2026-04-23T00:16:31.097Z [INFO]  server-51db00a8-0d73-438d-5bfa-1e39823b6fd4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5528b28d-58e1-8d3a-c568-50a7b1843615 Address:5528b28d-58e1-8d3a-c568-50a7b1843615} {Suffrage:Voter ID:server-51db00a8-0d73-438d-5bfa-1e39823b6fd4 Address:51db00a8-0d73-438d-5bfa-1e39823b6fd4}]"
2026-04-23T00:16:31.097Z [INFO]  server-5528b28d-58e1-8d3a-c568-50a7b1843615: aborting pipeline replication: peer="{Voter server-51db00a8-0d73-438d-5bfa-1e39823b6fd4 51db00a8-0d73-438d-5bfa-1e39823b6fd4}"
--- PASS: TestRaft_CacheLogWithStoreError (0.28s)
=== RUN   TestRaft_ReloadConfig
2026-04-23T00:16:31.098Z [INFO]  server-de27dfc7-a841-c195-d76f-7ece233b80b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de27dfc7-a841-c195-d76f-7ece233b80b8 Address:de27dfc7-a841-c195-d76f-7ece233b80b8}]"
2026-04-23T00:16:31.098Z [INFO]  server-de27dfc7-a841-c195-d76f-7ece233b80b8: entering follower state: follower="Node at de27dfc7-a841-c195-d76f-7ece233b80b8 [Shutdown]" leader-address= leader-id=
--- PASS: TestRaft_ReloadConfig (0.00s)
=== RUN   TestRaft_ReloadConfigValidates
2026-04-23T00:16:31.099Z [INFO]  server-76f94e0b-bb31-8662-98a7-0483e3e5f1fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-76f94e0b-bb31-8662-98a7-0483e3e5f1fa Address:76f94e0b-bb31-8662-98a7-0483e3e5f1fa}]"
2026-04-23T00:16:31.099Z [INFO]  server-76f94e0b-bb31-8662-98a7-0483e3e5f1fa: entering follower state: follower="Node at 76f94e0b-bb31-8662-98a7-0483e3e5f1fa [Shutdown]" leader-address= leader-id=
--- PASS: TestRaft_ReloadConfigValidates (0.00s)
=== RUN   TestRaft_InstallSnapshot_InvalidPeers
2026-04-23T00:16:31.099Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)"
--- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s)
=== RUN   TestRaft_VoteNotGranted_WhenNodeNotInCluster
2026-04-23T00:16:31.100Z [INFO]  server-f3c478a9-c814-8ee2-2725-6948283951d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f3c478a9-c814-8ee2-2725-6948283951d3 Address:f3c478a9-c814-8ee2-2725-6948283951d3} {Suffrage:Voter ID:server-f19680ac-a5bb-76bb-03d9-3d9635c54276 Address:f19680ac-a5bb-76bb-03d9-3d9635c54276} {Suffrage:Voter ID:server-8700683a-09a0-cbe0-f181-fed719182411 Address:8700683a-09a0-cbe0-f181-fed719182411}]"
2026-04-23T00:16:31.100Z [INFO]  server-f3c478a9-c814-8ee2-2725-6948283951d3: entering follower state: follower="Node at f3c478a9-c814-8ee2-2725-6948283951d3 [Follower]" leader-address= leader-id=
2026-04-23T00:16:31.100Z [INFO]  server-f19680ac-a5bb-76bb-03d9-3d9635c54276: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f3c478a9-c814-8ee2-2725-6948283951d3 Address:f3c478a9-c814-8ee2-2725-6948283951d3} {Suffrage:Voter ID:server-f19680ac-a5bb-76bb-03d9-3d9635c54276 Address:f19680ac-a5bb-76bb-03d9-3d9635c54276} {Suffrage:Voter ID:server-8700683a-09a0-cbe0-f181-fed719182411 Address:8700683a-09a0-cbe0-f181-fed719182411}]"
2026-04-23T00:16:31.100Z [INFO]  server-8700683a-09a0-cbe0-f181-fed719182411: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f3c478a9-c814-8ee2-2725-6948283951d3 Address:f3c478a9-c814-8ee2-2725-6948283951d3} {Suffrage:Voter ID:server-f19680ac-a5bb-76bb-03d9-3d9635c54276 Address:f19680ac-a5bb-76bb-03d9-3d9635c54276} {Suffrage:Voter ID:server-8700683a-09a0-cbe0-f181-fed719182411 Address:8700683a-09a0-cbe0-f181-fed719182411}]"
2026-04-23T00:16:31.100Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:31.100Z [INFO]  server-f19680ac-a5bb-76bb-03d9-3d9635c54276: entering follower state: follower="Node at f19680ac-a5bb-76bb-03d9-3d9635c54276 [Follower]" leader-address= leader-id=
2026-04-23T00:16:31.100Z [INFO]  server-8700683a-09a0-cbe0-f181-fed719182411: entering follower state: follower="Node at 8700683a-09a0-cbe0-f181-fed719182411 [Follower]" leader-address= leader-id=
2026-04-23T00:16:31.152Z [WARN]  server-f3c478a9-c814-8ee2-2725-6948283951d3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:31.152Z [INFO]  server-f3c478a9-c814-8ee2-2725-6948283951d3: entering candidate state: node="Node at f3c478a9-c814-8ee2-2725-6948283951d3 [Candidate]" term=2
2026-04-23T00:16:31.152Z [INFO]  server-f3c478a9-c814-8ee2-2725-6948283951d3: election won: term=2 tally=2
2026-04-23T00:16:31.152Z [INFO]  server-f3c478a9-c814-8ee2-2725-6948283951d3: entering leader state: leader="Node at f3c478a9-c814-8ee2-2725-6948283951d3 [Leader]"
2026-04-23T00:16:31.152Z [INFO]  server-f3c478a9-c814-8ee2-2725-6948283951d3: added peer, starting replication: peer=server-f19680ac-a5bb-76bb-03d9-3d9635c54276
2026-04-23T00:16:31.152Z [INFO]  server-f3c478a9-c814-8ee2-2725-6948283951d3: added peer, starting replication: peer=server-8700683a-09a0-cbe0-f181-fed719182411
2026-04-23T00:16:31.152Z [INFO]  server-f3c478a9-c814-8ee2-2725-6948283951d3: pipelining replication: peer="{Voter server-8700683a-09a0-cbe0-f181-fed719182411 8700683a-09a0-cbe0-f181-fed719182411}"
2026-04-23T00:16:31.152Z [INFO]  server-f3c478a9-c814-8ee2-2725-6948283951d3: pipelining replication: peer="{Voter server-f19680ac-a5bb-76bb-03d9-3d9635c54276 f19680ac-a5bb-76bb-03d9-3d9635c54276}"
2026-04-23T00:16:31.258Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:31.152750392 +0000 UTC m=+108.192351818 (1 nodes), 52.59511ms from start of poll, 52.808281ms from cluster start. Timeout at 2026-04-23 00:16:31.2577518 +0000 UTC m=+108.297353155, 105.001337ms after stability
2026-04-23T00:16:31.264Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:31.369Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:31.264408996 +0000 UTC m=+108.304010365 (2 nodes), 1.406µs from start of poll, 164.466828ms from cluster start. Timeout at 2026-04-23 00:16:31.369434939 +0000 UTC m=+108.409036322, 105.025957ms after stability
2026-04-23T00:16:31.369Z [INFO]  server-f3c478a9-c814-8ee2-2725-6948283951d3: updating configuration: command=RemoveServer server-id=server-f19680ac-a5bb-76bb-03d9-3d9635c54276 server-addr= servers="[{Suffrage:Voter ID:server-f3c478a9-c814-8ee2-2725-6948283951d3 Address:f3c478a9-c814-8ee2-2725-6948283951d3} {Suffrage:Voter ID:server-8700683a-09a0-cbe0-f181-fed719182411 Address:8700683a-09a0-cbe0-f181-fed719182411}]"
2026-04-23T00:16:31.369Z [INFO]  server-f3c478a9-c814-8ee2-2725-6948283951d3: removed peer, stopping replication: peer=server-f19680ac-a5bb-76bb-03d9-3d9635c54276 last-index=3
2026-04-23T00:16:31.369Z [INFO]  server-f3c478a9-c814-8ee2-2725-6948283951d3: aborting pipeline replication: peer="{Voter server-f19680ac-a5bb-76bb-03d9-3d9635c54276 f19680ac-a5bb-76bb-03d9-3d9635c54276}"
2026-04-23T00:16:31.475Z [ERROR] server-f3c478a9-c814-8ee2-2725-6948283951d3: failed to heartbeat to: peer=8700683a-09a0-cbe0-f181-fed719182411 backoff time=10ms error="failed to connect to peer: 8700683a-09a0-cbe0-f181-fed719182411"
2026-04-23T00:16:31.481Z [ERROR] server-f3c478a9-c814-8ee2-2725-6948283951d3: failed to pipeline appendEntries: peer="{Voter server-8700683a-09a0-cbe0-f181-fed719182411 8700683a-09a0-cbe0-f181-fed719182411}" error="append pipeline closed"
2026-04-23T00:16:31.481Z [INFO]  server-f3c478a9-c814-8ee2-2725-6948283951d3: aborting pipeline replication: peer="{Voter server-8700683a-09a0-cbe0-f181-fed719182411 8700683a-09a0-cbe0-f181-fed719182411}"
2026-04-23T00:16:31.491Z [ERROR] server-f3c478a9-c814-8ee2-2725-6948283951d3: failed to appendEntries to: peer="{Voter server-8700683a-09a0-cbe0-f181-fed719182411 8700683a-09a0-cbe0-f181-fed719182411}" error="failed to connect to peer: 8700683a-09a0-cbe0-f181-fed719182411"
2026-04-23T00:16:31.494Z [WARN]  server-f19680ac-a5bb-76bb-03d9-3d9635c54276: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2026-04-23T00:16:31.495Z [ERROR] server-f3c478a9-c814-8ee2-2725-6948283951d3: failed to heartbeat to: peer=8700683a-09a0-cbe0-f181-fed719182411 backoff time=10ms error="failed to connect to peer: 8700683a-09a0-cbe0-f181-fed719182411"
2026-04-23T00:16:31.511Z [ERROR] server-f3c478a9-c814-8ee2-2725-6948283951d3: failed to appendEntries to: peer="{Voter server-8700683a-09a0-cbe0-f181-fed719182411 8700683a-09a0-cbe0-f181-fed719182411}" error="failed to connect to peer: 8700683a-09a0-cbe0-f181-fed719182411"
2026-04-23T00:16:31.514Z [ERROR] server-f3c478a9-c814-8ee2-2725-6948283951d3: failed to heartbeat to: peer=8700683a-09a0-cbe0-f181-fed719182411 backoff time=10ms error="failed to connect to peer: 8700683a-09a0-cbe0-f181-fed719182411"
2026-04-23T00:16:31.520Z [WARN]  server-f3c478a9-c814-8ee2-2725-6948283951d3: failed to contact: server-id=server-8700683a-09a0-cbe0-f181-fed719182411 time=50.013032ms
2026-04-23T00:16:31.520Z [WARN]  server-f3c478a9-c814-8ee2-2725-6948283951d3: failed to contact quorum of nodes, stepping down
2026-04-23T00:16:31.520Z [INFO]  server-f3c478a9-c814-8ee2-2725-6948283951d3: entering follower state: follower="Node at f3c478a9-c814-8ee2-2725-6948283951d3 [Follower]" leader-address= leader-id=
2026-04-23T00:16:31.529Z [ERROR] server-f3c478a9-c814-8ee2-2725-6948283951d3: failed to appendEntries to: peer="{Voter server-8700683a-09a0-cbe0-f181-fed719182411 8700683a-09a0-cbe0-f181-fed719182411}" error="failed to connect to peer: 8700683a-09a0-cbe0-f181-fed719182411"
2026-04-23T00:16:31.585Z [WARN]  server-f3c478a9-c814-8ee2-2725-6948283951d3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:31.586Z [INFO]  server-f3c478a9-c814-8ee2-2725-6948283951d3: entering candidate state: node="Node at f3c478a9-c814-8ee2-2725-6948283951d3 [Candidate]" term=3
2026-04-23T00:16:31.586Z [ERROR] server-f3c478a9-c814-8ee2-2725-6948283951d3: failed to make requestVote RPC: target="{Voter server-8700683a-09a0-cbe0-f181-fed719182411 8700683a-09a0-cbe0-f181-fed719182411}" error="failed to connect to peer: 8700683a-09a0-cbe0-f181-fed719182411" term=3
2026-04-23T00:16:31.611Z [WARN]  server-8700683a-09a0-cbe0-f181-fed719182411: heartbeat timeout reached, starting election: last-leader-addr=f3c478a9-c814-8ee2-2725-6948283951d3 last-leader-id=server-f3c478a9-c814-8ee2-2725-6948283951d3
2026-04-23T00:16:31.611Z [INFO]  server-8700683a-09a0-cbe0-f181-fed719182411: entering candidate state: node="Node at 8700683a-09a0-cbe0-f181-fed719182411 [Candidate]" term=3
2026-04-23T00:16:31.611Z [ERROR] server-8700683a-09a0-cbe0-f181-fed719182411: failed to make requestVote RPC: target="{Voter server-f3c478a9-c814-8ee2-2725-6948283951d3 f3c478a9-c814-8ee2-2725-6948283951d3}" error="failed to connect to peer: f3c478a9-c814-8ee2-2725-6948283951d3" term=3
2026-04-23T00:16:31.612Z [WARN]  server-8700683a-09a0-cbe0-f181-fed719182411: rejecting vote request since node is not in configuration: from=f19680ac-a5bb-76bb-03d9-3d9635c54276
--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.51s)
=== RUN   TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin
2026-04-23T00:16:31.613Z [INFO]  server-30a795dd-b709-7187-766a-aeec9b2e812c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30a795dd-b709-7187-766a-aeec9b2e812c Address:30a795dd-b709-7187-766a-aeec9b2e812c} {Suffrage:Voter ID:server-930f8708-05c4-3715-d7be-0ee6308f036c Address:930f8708-05c4-3715-d7be-0ee6308f036c} {Suffrage:Voter ID:server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 Address:845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2}]"
2026-04-23T00:16:31.614Z [INFO]  server-930f8708-05c4-3715-d7be-0ee6308f036c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30a795dd-b709-7187-766a-aeec9b2e812c Address:30a795dd-b709-7187-766a-aeec9b2e812c} {Suffrage:Voter ID:server-930f8708-05c4-3715-d7be-0ee6308f036c Address:930f8708-05c4-3715-d7be-0ee6308f036c} {Suffrage:Voter ID:server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 Address:845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2}]"
2026-04-23T00:16:31.613Z [INFO]  server-30a795dd-b709-7187-766a-aeec9b2e812c: entering follower state: follower="Node at 30a795dd-b709-7187-766a-aeec9b2e812c [Follower]" leader-address= leader-id=
2026-04-23T00:16:31.614Z [INFO]  server-930f8708-05c4-3715-d7be-0ee6308f036c: entering follower state: follower="Node at 930f8708-05c4-3715-d7be-0ee6308f036c [Follower]" leader-address= leader-id=
2026-04-23T00:16:31.614Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30a795dd-b709-7187-766a-aeec9b2e812c Address:30a795dd-b709-7187-766a-aeec9b2e812c} {Suffrage:Voter ID:server-930f8708-05c4-3715-d7be-0ee6308f036c Address:930f8708-05c4-3715-d7be-0ee6308f036c} {Suffrage:Voter ID:server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 Address:845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2}]"
2026-04-23T00:16:31.614Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:31.614Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: entering follower state: follower="Node at 845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 [Follower]" leader-address= leader-id=
2026-04-23T00:16:31.669Z [WARN]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:31.669Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: entering candidate state: node="Node at 845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 [Candidate]" term=2
2026-04-23T00:16:31.669Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: election won: term=2 tally=2
2026-04-23T00:16:31.669Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: entering leader state: leader="Node at 845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 [Leader]"
2026-04-23T00:16:31.669Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: added peer, starting replication: peer=server-30a795dd-b709-7187-766a-aeec9b2e812c
2026-04-23T00:16:31.669Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: added peer, starting replication: peer=server-930f8708-05c4-3715-d7be-0ee6308f036c
2026-04-23T00:16:31.669Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: pipelining replication: peer="{Voter server-930f8708-05c4-3715-d7be-0ee6308f036c 930f8708-05c4-3715-d7be-0ee6308f036c}"
2026-04-23T00:16:31.670Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: pipelining replication: peer="{Voter server-30a795dd-b709-7187-766a-aeec9b2e812c 30a795dd-b709-7187-766a-aeec9b2e812c}"
2026-04-23T00:16:31.775Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:31.669832117 +0000 UTC m=+108.709433487 (1 nodes), 55.676413ms from start of poll, 56.029464ms from cluster start. Timeout at 2026-04-23 00:16:31.774835091 +0000 UTC m=+108.814436447, 105.00296ms after stability
2026-04-23T00:16:31.781Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:31.887Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:31.781642764 +0000 UTC m=+108.821244131 (2 nodes), 1.727µs from start of poll, 167.840108ms from cluster start. Timeout at 2026-04-23 00:16:31.886666511 +0000 UTC m=+108.926267885, 105.023754ms after stability
2026-04-23T00:16:31.890Z [ERROR] server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: failed to pipeline appendEntries: peer="{Voter server-30a795dd-b709-7187-766a-aeec9b2e812c 30a795dd-b709-7187-766a-aeec9b2e812c}" error="append pipeline closed"
2026-04-23T00:16:31.890Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: aborting pipeline replication: peer="{Voter server-30a795dd-b709-7187-766a-aeec9b2e812c 30a795dd-b709-7187-766a-aeec9b2e812c}"
2026-04-23T00:16:31.891Z [ERROR] server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: failed to heartbeat to: peer=30a795dd-b709-7187-766a-aeec9b2e812c backoff time=10ms error="failed to connect to peer: 30a795dd-b709-7187-766a-aeec9b2e812c"
2026-04-23T00:16:31.899Z [ERROR] server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: failed to appendEntries to: peer="{Voter server-30a795dd-b709-7187-766a-aeec9b2e812c 30a795dd-b709-7187-766a-aeec9b2e812c}" error="failed to connect to peer: 30a795dd-b709-7187-766a-aeec9b2e812c"
2026-04-23T00:16:31.908Z [ERROR] server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: failed to heartbeat to: peer=30a795dd-b709-7187-766a-aeec9b2e812c backoff time=10ms error="failed to connect to peer: 30a795dd-b709-7187-766a-aeec9b2e812c"
2026-04-23T00:16:31.916Z [ERROR] server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: failed to appendEntries to: peer="{Voter server-30a795dd-b709-7187-766a-aeec9b2e812c 30a795dd-b709-7187-766a-aeec9b2e812c}" error="failed to connect to peer: 30a795dd-b709-7187-766a-aeec9b2e812c"
2026-04-23T00:16:31.925Z [ERROR] server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: failed to heartbeat to: peer=30a795dd-b709-7187-766a-aeec9b2e812c backoff time=10ms error="failed to connect to peer: 30a795dd-b709-7187-766a-aeec9b2e812c"
2026-04-23T00:16:31.933Z [ERROR] server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: failed to appendEntries to: peer="{Voter server-30a795dd-b709-7187-766a-aeec9b2e812c 30a795dd-b709-7187-766a-aeec9b2e812c}" error="failed to connect to peer: 30a795dd-b709-7187-766a-aeec9b2e812c"
2026-04-23T00:16:31.933Z [WARN]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: failed to contact: server-id=server-30a795dd-b709-7187-766a-aeec9b2e812c time=50.934466ms
2026-04-23T00:16:31.936Z [WARN]  server-30a795dd-b709-7187-766a-aeec9b2e812c: heartbeat timeout reached, starting election: last-leader-addr=845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 last-leader-id=server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2
2026-04-23T00:16:31.936Z [INFO]  server-30a795dd-b709-7187-766a-aeec9b2e812c: entering candidate state: node="Node at 30a795dd-b709-7187-766a-aeec9b2e812c [Candidate]" term=3
2026-04-23T00:16:31.936Z [ERROR] server-30a795dd-b709-7187-766a-aeec9b2e812c: failed to make requestVote RPC: target="{Voter server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2}" error="failed to connect to peer: 845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2" term=3
2026-04-23T00:16:31.936Z [ERROR] server-30a795dd-b709-7187-766a-aeec9b2e812c: failed to make requestVote RPC: target="{Voter server-930f8708-05c4-3715-d7be-0ee6308f036c 930f8708-05c4-3715-d7be-0ee6308f036c}" error="failed to connect to peer: 930f8708-05c4-3715-d7be-0ee6308f036c" term=3
2026-04-23T00:16:31.943Z [ERROR] server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: failed to heartbeat to: peer=30a795dd-b709-7187-766a-aeec9b2e812c backoff time=20ms error="failed to connect to peer: 30a795dd-b709-7187-766a-aeec9b2e812c"
2026-04-23T00:16:31.961Z [ERROR] server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: failed to appendEntries to: peer="{Voter server-30a795dd-b709-7187-766a-aeec9b2e812c 30a795dd-b709-7187-766a-aeec9b2e812c}" error="failed to connect to peer: 30a795dd-b709-7187-766a-aeec9b2e812c"
2026-04-23T00:16:31.972Z [ERROR] server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: failed to heartbeat to: peer=30a795dd-b709-7187-766a-aeec9b2e812c backoff time=25ms error="failed to connect to peer: 30a795dd-b709-7187-766a-aeec9b2e812c"
2026-04-23T00:16:31.977Z [WARN]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: failed to contact: server-id=server-30a795dd-b709-7187-766a-aeec9b2e812c time=94.40065ms
2026-04-23T00:16:31.988Z [WARN]  server-30a795dd-b709-7187-766a-aeec9b2e812c: Election timeout reached, restarting election
2026-04-23T00:16:31.988Z [INFO]  server-30a795dd-b709-7187-766a-aeec9b2e812c: entering candidate state: node="Node at 30a795dd-b709-7187-766a-aeec9b2e812c [Candidate]" term=4
2026-04-23T00:16:31.989Z [ERROR] server-30a795dd-b709-7187-766a-aeec9b2e812c: failed to make requestVote RPC: target="{Voter server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2}" error="failed to connect to peer: 845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2" term=4
2026-04-23T00:16:31.989Z [ERROR] server-30a795dd-b709-7187-766a-aeec9b2e812c: failed to make requestVote RPC: target="{Voter server-930f8708-05c4-3715-d7be-0ee6308f036c 930f8708-05c4-3715-d7be-0ee6308f036c}" error="failed to connect to peer: 930f8708-05c4-3715-d7be-0ee6308f036c" term=4
2026-04-23T00:16:31.993Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: updating configuration: command=RemoveServer server-id=server-30a795dd-b709-7187-766a-aeec9b2e812c server-addr= servers="[{Suffrage:Voter ID:server-930f8708-05c4-3715-d7be-0ee6308f036c Address:930f8708-05c4-3715-d7be-0ee6308f036c} {Suffrage:Voter ID:server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 Address:845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2}]"
2026-04-23T00:16:31.993Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: removed peer, stopping replication: peer=server-30a795dd-b709-7187-766a-aeec9b2e812c last-index=3
2026-04-23T00:16:31.993Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: updating configuration: command=AddNonvoter server-id=server-30a795dd-b709-7187-766a-aeec9b2e812c server-addr=30a795dd-b709-7187-766a-aeec9b2e812c servers="[{Suffrage:Voter ID:server-930f8708-05c4-3715-d7be-0ee6308f036c Address:930f8708-05c4-3715-d7be-0ee6308f036c} {Suffrage:Voter ID:server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 Address:845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2} {Suffrage:Nonvoter ID:server-30a795dd-b709-7187-766a-aeec9b2e812c Address:30a795dd-b709-7187-766a-aeec9b2e812c}]"
2026-04-23T00:16:31.993Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: added peer, starting replication: peer=server-30a795dd-b709-7187-766a-aeec9b2e812c
2026-04-23T00:16:31.993Z [ERROR] server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: failed to appendEntries to: peer="{Nonvoter server-30a795dd-b709-7187-766a-aeec9b2e812c 30a795dd-b709-7187-766a-aeec9b2e812c}" error="failed to connect to peer: 30a795dd-b709-7187-766a-aeec9b2e812c"
2026-04-23T00:16:32.008Z [ERROR] server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: peer has newer term, stopping replication: peer="{Voter server-30a795dd-b709-7187-766a-aeec9b2e812c 30a795dd-b709-7187-766a-aeec9b2e812c}"
2026-04-23T00:16:32.008Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: entering follower state: follower="Node at 845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 [Follower]" leader-address= leader-id=
2026-04-23T00:16:32.008Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: aborting pipeline replication: peer="{Voter server-930f8708-05c4-3715-d7be-0ee6308f036c 930f8708-05c4-3715-d7be-0ee6308f036c}"
2026-04-23T00:16:32.013Z [ERROR] server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: peer has newer term, stopping replication: peer="{Nonvoter server-30a795dd-b709-7187-766a-aeec9b2e812c 30a795dd-b709-7187-766a-aeec9b2e812c}"
2026-04-23T00:16:32.081Z [WARN]  server-30a795dd-b709-7187-766a-aeec9b2e812c: Election timeout reached, restarting election
2026-04-23T00:16:32.081Z [INFO]  server-30a795dd-b709-7187-766a-aeec9b2e812c: entering candidate state: node="Node at 30a795dd-b709-7187-766a-aeec9b2e812c [Candidate]" term=23
2026-04-23T00:16:32.081Z [WARN]  server-930f8708-05c4-3715-d7be-0ee6308f036c: rejecting vote request since we have a leader: from=30a795dd-b709-7187-766a-aeec9b2e812c leader=845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 leader-id=server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2
2026-04-23T00:16:32.081Z [WARN]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: rejecting vote request since node is not a voter: from=30a795dd-b709-7187-766a-aeec9b2e812c
2026-04-23T00:16:32.099Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:32.101Z [WARN]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:32.101Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: entering candidate state: node="Node at 845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 [Candidate]" term=24
2026-04-23T00:16:32.101Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: election won: term=24 tally=2
2026-04-23T00:16:32.101Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: entering leader state: leader="Node at 845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 [Leader]"
2026-04-23T00:16:32.101Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: added peer, starting replication: peer=server-930f8708-05c4-3715-d7be-0ee6308f036c
2026-04-23T00:16:32.101Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: added peer, starting replication: peer=server-30a795dd-b709-7187-766a-aeec9b2e812c
2026-04-23T00:16:32.101Z [WARN]  server-30a795dd-b709-7187-766a-aeec9b2e812c: failed to get previous log: previous-index=4 last-index=2 error="log not found"
2026-04-23T00:16:32.101Z [INFO]  server-30a795dd-b709-7187-766a-aeec9b2e812c: entering follower state: follower="Node at 30a795dd-b709-7187-766a-aeec9b2e812c [Follower]" leader-address=845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 leader-id=server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2
2026-04-23T00:16:32.101Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: pipelining replication: peer="{Voter server-930f8708-05c4-3715-d7be-0ee6308f036c 930f8708-05c4-3715-d7be-0ee6308f036c}"
2026-04-23T00:16:32.101Z [WARN]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: appendEntries rejected, sending older logs: peer="{Nonvoter server-30a795dd-b709-7187-766a-aeec9b2e812c 30a795dd-b709-7187-766a-aeec9b2e812c}" next=3
2026-04-23T00:16:32.102Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: pipelining replication: peer="{Nonvoter server-30a795dd-b709-7187-766a-aeec9b2e812c 30a795dd-b709-7187-766a-aeec9b2e812c}"
2026-04-23T00:16:32.207Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:32.10190335 +0000 UTC m=+109.141504703 (1 nodes), 2.495119ms from start of poll, 488.10068ms from cluster start. Timeout at 2026-04-23 00:16:32.206906792 +0000 UTC m=+109.246508145, 105.003442ms after stability
2026-04-23T00:16:32.225Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: updating configuration: command=RemoveServer server-id=server-30a795dd-b709-7187-766a-aeec9b2e812c server-addr= servers="[{Suffrage:Voter ID:server-930f8708-05c4-3715-d7be-0ee6308f036c Address:930f8708-05c4-3715-d7be-0ee6308f036c} {Suffrage:Voter ID:server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 Address:845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2}]"
2026-04-23T00:16:32.225Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: removed peer, stopping replication: peer=server-30a795dd-b709-7187-766a-aeec9b2e812c last-index=106
2026-04-23T00:16:32.225Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: aborting pipeline replication: peer="{Nonvoter server-30a795dd-b709-7187-766a-aeec9b2e812c 30a795dd-b709-7187-766a-aeec9b2e812c}"
2026-04-23T00:16:32.226Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: updating configuration: command=AddVoter server-id=server-30a795dd-b709-7187-766a-aeec9b2e812c server-addr=30a795dd-b709-7187-766a-aeec9b2e812c servers="[{Suffrage:Voter ID:server-930f8708-05c4-3715-d7be-0ee6308f036c Address:930f8708-05c4-3715-d7be-0ee6308f036c} {Suffrage:Voter ID:server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2 Address:845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2} {Suffrage:Voter ID:server-30a795dd-b709-7187-766a-aeec9b2e812c Address:30a795dd-b709-7187-766a-aeec9b2e812c}]"
2026-04-23T00:16:32.226Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: added peer, starting replication: peer=server-30a795dd-b709-7187-766a-aeec9b2e812c
2026-04-23T00:16:32.226Z [WARN]  server-30a795dd-b709-7187-766a-aeec9b2e812c: failed to get previous log: previous-index=107 last-index=106 error="log not found"
2026-04-23T00:16:32.226Z [WARN]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: appendEntries rejected, sending older logs: peer="{Voter server-30a795dd-b709-7187-766a-aeec9b2e812c 30a795dd-b709-7187-766a-aeec9b2e812c}" next=107
2026-04-23T00:16:32.226Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: pipelining replication: peer="{Voter server-30a795dd-b709-7187-766a-aeec9b2e812c 30a795dd-b709-7187-766a-aeec9b2e812c}"
2026-04-23T00:16:33.294Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:33.400Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:33.294594936 +0000 UTC m=+110.334196300 (1 nodes), 719ns from start of poll, 1.680792277s from cluster start. Timeout at 2026-04-23 00:16:33.399614103 +0000 UTC m=+110.439215502, 105.019202ms after stability
2026-04-23T00:16:33.400Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: aborting pipeline replication: peer="{Voter server-30a795dd-b709-7187-766a-aeec9b2e812c 30a795dd-b709-7187-766a-aeec9b2e812c}"
2026-04-23T00:16:33.400Z [INFO]  server-845d8fe3-7417-9a1d-6132-d1c0d8dfd9d2: aborting pipeline replication: peer="{Voter server-930f8708-05c4-3715-d7be-0ee6308f036c 930f8708-05c4-3715-d7be-0ee6308f036c}"
--- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.79s)
=== RUN   TestRaft_FollowerRemovalNoElection
2026-04-23T00:16:33.402Z [INFO]  server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9 Address:0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9} {Suffrage:Voter ID:server-e034c8f0-9008-0324-206f-52fe9e46a74b Address:e034c8f0-9008-0324-206f-52fe9e46a74b} {Suffrage:Voter ID:server-022291e9-502e-d413-59cf-18f1a60137c0 Address:022291e9-502e-d413-59cf-18f1a60137c0}]"
2026-04-23T00:16:33.402Z [INFO]  server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9: entering follower state: follower="Node at 0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9 [Follower]" leader-address= leader-id=
2026-04-23T00:16:33.402Z [INFO]  server-e034c8f0-9008-0324-206f-52fe9e46a74b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9 Address:0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9} {Suffrage:Voter ID:server-e034c8f0-9008-0324-206f-52fe9e46a74b Address:e034c8f0-9008-0324-206f-52fe9e46a74b} {Suffrage:Voter ID:server-022291e9-502e-d413-59cf-18f1a60137c0 Address:022291e9-502e-d413-59cf-18f1a60137c0}]"
2026-04-23T00:16:33.402Z [INFO]  server-e034c8f0-9008-0324-206f-52fe9e46a74b: entering follower state: follower="Node at e034c8f0-9008-0324-206f-52fe9e46a74b [Follower]" leader-address= leader-id=
2026-04-23T00:16:33.402Z [INFO]  server-022291e9-502e-d413-59cf-18f1a60137c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9 Address:0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9} {Suffrage:Voter ID:server-e034c8f0-9008-0324-206f-52fe9e46a74b Address:e034c8f0-9008-0324-206f-52fe9e46a74b} {Suffrage:Voter ID:server-022291e9-502e-d413-59cf-18f1a60137c0 Address:022291e9-502e-d413-59cf-18f1a60137c0}]"
2026-04-23T00:16:33.402Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:33.402Z [INFO]  server-022291e9-502e-d413-59cf-18f1a60137c0: entering follower state: follower="Node at 022291e9-502e-d413-59cf-18f1a60137c0 [Follower]" leader-address= leader-id=
2026-04-23T00:16:33.548Z [WARN]  server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:33.548Z [INFO]  server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9: entering candidate state: node="Node at 0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9 [Candidate]" term=2
2026-04-23T00:16:33.548Z [INFO]  server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9: election won: term=2 tally=2
2026-04-23T00:16:33.548Z [INFO]  server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9: entering leader state: leader="Node at 0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9 [Leader]"
2026-04-23T00:16:33.548Z [INFO]  server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9: added peer, starting replication: peer=server-e034c8f0-9008-0324-206f-52fe9e46a74b
2026-04-23T00:16:33.548Z [INFO]  server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9: added peer, starting replication: peer=server-022291e9-502e-d413-59cf-18f1a60137c0
2026-04-23T00:16:33.549Z [INFO]  server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9: pipelining replication: peer="{Voter server-e034c8f0-9008-0324-206f-52fe9e46a74b e034c8f0-9008-0324-206f-52fe9e46a74b}"
2026-04-23T00:16:33.549Z [INFO]  server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9: pipelining replication: peer="{Voter server-022291e9-502e-d413-59cf-18f1a60137c0 022291e9-502e-d413-59cf-18f1a60137c0}"
2026-04-23T00:16:33.754Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:33.548944866 +0000 UTC m=+110.588546233 (1 nodes), 145.984605ms from start of poll, 146.380743ms from cluster start. Timeout at 2026-04-23 00:16:33.753948429 +0000 UTC m=+110.793549781, 205.003548ms after stability
2026-04-23T00:16:33.754Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:33.959Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:33.754323978 +0000 UTC m=+110.793925333 (1 nodes), 1.638µs from start of poll, 351.759843ms from cluster start. Timeout at 2026-04-23 00:16:33.959343894 +0000 UTC m=+110.998945251, 205.019918ms after stability
2026-04-23T00:16:33.965Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:34.170Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:33.965506246 +0000 UTC m=+111.005107600 (2 nodes), 2.5µs from start of poll, 562.94211ms from cluster start. Timeout at 2026-04-23 00:16:34.17060912 +0000 UTC m=+111.210210475, 205.102875ms after stability
2026-04-23T00:16:34.170Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:34.376Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:34.170770102 +0000 UTC m=+111.210371457 (1 nodes), 2.076µs from start of poll, 768.205967ms from cluster start. Timeout at 2026-04-23 00:16:34.375792321 +0000 UTC m=+111.415393676, 205.022219ms after stability
    raft_test.go:2961: [INFO] restarting Node at e034c8f0-9008-0324-206f-52fe9e46a74b [Follower]
2026-04-23T00:16:34.376Z [INFO]  server-e034c8f0-9008-0324-206f-52fe9e46a74b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9 Address:0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9} {Suffrage:Voter ID:server-e034c8f0-9008-0324-206f-52fe9e46a74b Address:e034c8f0-9008-0324-206f-52fe9e46a74b} {Suffrage:Voter ID:server-022291e9-502e-d413-59cf-18f1a60137c0 Address:022291e9-502e-d413-59cf-18f1a60137c0}]"
2026-04-23T00:16:34.376Z [INFO]  server-e034c8f0-9008-0324-206f-52fe9e46a74b: entering follower state: follower="Node at e034c8f0-9008-0324-206f-52fe9e46a74b [Follower]" leader-address= leader-id=
2026-04-23T00:16:34.627Z [INFO]  server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9: aborting pipeline replication: peer="{Voter server-022291e9-502e-d413-59cf-18f1a60137c0 022291e9-502e-d413-59cf-18f1a60137c0}"
2026-04-23T00:16:35.005Z [ERROR] server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9: failed to pipeline appendEntries: peer="{Voter server-e034c8f0-9008-0324-206f-52fe9e46a74b e034c8f0-9008-0324-206f-52fe9e46a74b}" error="command enqueue timeout"
2026-04-23T00:16:35.005Z [INFO]  server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9: aborting pipeline replication: peer="{Voter server-e034c8f0-9008-0324-206f-52fe9e46a74b e034c8f0-9008-0324-206f-52fe9e46a74b}"
2026-04-23T00:16:35.145Z [ERROR] server-0f7998d6-5bdf-ec8a-1cf8-274a6f16efd9: failed to heartbeat to: peer=e034c8f0-9008-0324-206f-52fe9e46a74b backoff time=10ms error="command timed out"
--- PASS: TestRaft_FollowerRemovalNoElection (1.75s)
=== RUN   TestRaft_VoteWithNoIDNoAddr
2026-04-23T00:16:35.147Z [INFO]  server-988f188e-70d7-81ae-4807-83aa2a01d193: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-988f188e-70d7-81ae-4807-83aa2a01d193 Address:988f188e-70d7-81ae-4807-83aa2a01d193} {Suffrage:Voter ID:server-03b797ef-05f5-245f-38b7-872ec65171f1 Address:03b797ef-05f5-245f-38b7-872ec65171f1} {Suffrage:Voter ID:server-cae752ff-25be-ac93-57b4-726e0dd543ec Address:cae752ff-25be-ac93-57b4-726e0dd543ec}]"
2026-04-23T00:16:35.147Z [INFO]  server-988f188e-70d7-81ae-4807-83aa2a01d193: entering follower state: follower="Node at 988f188e-70d7-81ae-4807-83aa2a01d193 [Follower]" leader-address= leader-id=
2026-04-23T00:16:35.147Z [INFO]  server-03b797ef-05f5-245f-38b7-872ec65171f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-988f188e-70d7-81ae-4807-83aa2a01d193 Address:988f188e-70d7-81ae-4807-83aa2a01d193} {Suffrage:Voter ID:server-03b797ef-05f5-245f-38b7-872ec65171f1 Address:03b797ef-05f5-245f-38b7-872ec65171f1} {Suffrage:Voter ID:server-cae752ff-25be-ac93-57b4-726e0dd543ec Address:cae752ff-25be-ac93-57b4-726e0dd543ec}]"
2026-04-23T00:16:35.147Z [INFO]  server-03b797ef-05f5-245f-38b7-872ec65171f1: entering follower state: follower="Node at 03b797ef-05f5-245f-38b7-872ec65171f1 [Follower]" leader-address= leader-id=
2026-04-23T00:16:35.147Z [INFO]  server-cae752ff-25be-ac93-57b4-726e0dd543ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-988f188e-70d7-81ae-4807-83aa2a01d193 Address:988f188e-70d7-81ae-4807-83aa2a01d193} {Suffrage:Voter ID:server-03b797ef-05f5-245f-38b7-872ec65171f1 Address:03b797ef-05f5-245f-38b7-872ec65171f1} {Suffrage:Voter ID:server-cae752ff-25be-ac93-57b4-726e0dd543ec Address:cae752ff-25be-ac93-57b4-726e0dd543ec}]"
2026-04-23T00:16:35.147Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:35.147Z [INFO]  server-cae752ff-25be-ac93-57b4-726e0dd543ec: entering follower state: follower="Node at cae752ff-25be-ac93-57b4-726e0dd543ec [Follower]" leader-address= leader-id=
2026-04-23T00:16:35.214Z [WARN]  server-cae752ff-25be-ac93-57b4-726e0dd543ec: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-04-23T00:16:35.214Z [INFO]  server-cae752ff-25be-ac93-57b4-726e0dd543ec: entering candidate state: node="Node at cae752ff-25be-ac93-57b4-726e0dd543ec [Candidate]" term=2
2026-04-23T00:16:35.214Z [INFO]  server-cae752ff-25be-ac93-57b4-726e0dd543ec: election won: term=2 tally=2
2026-04-23T00:16:35.214Z [INFO]  server-cae752ff-25be-ac93-57b4-726e0dd543ec: entering leader state: leader="Node at cae752ff-25be-ac93-57b4-726e0dd543ec [Leader]"
2026-04-23T00:16:35.214Z [INFO]  server-cae752ff-25be-ac93-57b4-726e0dd543ec: added peer, starting replication: peer=server-988f188e-70d7-81ae-4807-83aa2a01d193
2026-04-23T00:16:35.214Z [INFO]  server-cae752ff-25be-ac93-57b4-726e0dd543ec: added peer, starting replication: peer=server-03b797ef-05f5-245f-38b7-872ec65171f1
2026-04-23T00:16:35.215Z [INFO]  server-cae752ff-25be-ac93-57b4-726e0dd543ec: pipelining replication: peer="{Voter server-988f188e-70d7-81ae-4807-83aa2a01d193 988f188e-70d7-81ae-4807-83aa2a01d193}"
2026-04-23T00:16:35.215Z [INFO]  server-cae752ff-25be-ac93-57b4-726e0dd543ec: pipelining replication: peer="{Voter server-03b797ef-05f5-245f-38b7-872ec65171f1 03b797ef-05f5-245f-38b7-872ec65171f1}"
2026-04-23T00:16:35.320Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:35.214987429 +0000 UTC m=+112.254588821 (1 nodes), 67.58668ms from start of poll, 67.908713ms from cluster start. Timeout at 2026-04-23 00:16:35.31999112 +0000 UTC m=+112.359592477, 105.003656ms after stability
2026-04-23T00:16:35.320Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-04-23T00:16:35.426Z [INFO]  cluster: stable state for Leader reached at 2026-04-23 00:16:35.32021409 +0000 UTC m=+112.359815460 (1 nodes), 1.264µs from start of poll, 173.135352ms from cluster start. Timeout at 2026-04-23 00:16:35.425239451 +0000 UTC m=+112.464840941, 105.025481ms after stability
2026-04-23T00:16:35.431Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-04-23T00:16:35.537Z [INFO]  cluster: stable state for Follower reached at 2026-04-23 00:16:35.431510075 +0000 UTC m=+112.471111440 (2 nodes), 1.385µs from start of poll, 284.431332ms from cluster start. Timeout at 2026-04-23 00:16:35.536525311 +0000 UTC m=+112.576126764, 105.015324ms after stability
2026-04-23T00:16:35.538Z [ERROR] server-cae752ff-25be-ac93-57b4-726e0dd543ec: failed to heartbeat to: peer=03b797ef-05f5-245f-38b7-872ec65171f1 backoff time=10ms error="failed to connect to peer: 03b797ef-05f5-245f-38b7-872ec65171f1"
2026-04-23T00:16:35.538Z [ERROR] server-cae752ff-25be-ac93-57b4-726e0dd543ec: failed to pipeline appendEntries: peer="{Voter server-988f188e-70d7-81ae-4807-83aa2a01d193 988f188e-70d7-81ae-4807-83aa2a01d193}" error="append pipeline closed"
2026-04-23T00:16:35.538Z [INFO]  server-cae752ff-25be-ac93-57b4-726e0dd543ec: aborting pipeline replication: peer="{Voter server-988f188e-70d7-81ae-4807-83aa2a01d193 988f188e-70d7-81ae-4807-83aa2a01d193}"
2026-04-23T00:16:35.541Z [ERROR] server-cae752ff-25be-ac93-57b4-726e0dd543ec: failed to pipeline appendEntries: peer="{Voter server-03b797ef-05f5-245f-38b7-872ec65171f1 03b797ef-05f5-245f-38b7-872ec65171f1}" error="append pipeline closed"
2026-04-23T00:16:35.541Z [INFO]  server-cae752ff-25be-ac93-57b4-726e0dd543ec: aborting pipeline replication: peer="{Voter server-03b797ef-05f5-245f-38b7-872ec65171f1 03b797ef-05f5-245f-38b7-872ec65171f1}"
2026-04-23T00:16:35.541Z [ERROR] server-cae752ff-25be-ac93-57b4-726e0dd543ec: failed to heartbeat to: peer=988f188e-70d7-81ae-4807-83aa2a01d193 backoff time=10ms error="failed to connect to peer: 988f188e-70d7-81ae-4807-83aa2a01d193"
2026-04-23T00:16:35.545Z [ERROR] server-cae752ff-25be-ac93-57b4-726e0dd543ec: failed to appendEntries to: peer="{Voter server-988f188e-70d7-81ae-4807-83aa2a01d193 988f188e-70d7-81ae-4807-83aa2a01d193}" error="failed to connect to peer: 988f188e-70d7-81ae-4807-83aa2a01d193"
2026-04-23T00:16:35.551Z [ERROR] server-cae752ff-25be-ac93-57b4-726e0dd543ec: failed to appendEntries to: peer="{Voter server-03b797ef-05f5-245f-38b7-872ec65171f1 03b797ef-05f5-245f-38b7-872ec65171f1}" error="failed to connect to peer: 03b797ef-05f5-245f-38b7-872ec65171f1"
2026-04-23T00:16:35.554Z [ERROR] server-cae752ff-25be-ac93-57b4-726e0dd543ec: failed to heartbeat to: peer=03b797ef-05f5-245f-38b7-872ec65171f1 backoff time=10ms error="failed to connect to peer: 03b797ef-05f5-245f-38b7-872ec65171f1"
2026-04-23T00:16:35.562Z [ERROR] server-cae752ff-25be-ac93-57b4-726e0dd543ec: failed to heartbeat to: peer=988f188e-70d7-81ae-4807-83aa2a01d193 backoff time=10ms error="failed to connect to peer: 988f188e-70d7-81ae-4807-83aa2a01d193"
2026-04-23T00:16:35.563Z [ERROR] server-cae752ff-25be-ac93-57b4-726e0dd543ec: failed to appendEntries to: peer="{Voter server-988f188e-70d7-81ae-4807-83aa2a01d193 988f188e-70d7-81ae-4807-83aa2a01d193}" error="failed to connect to peer: 988f188e-70d7-81ae-4807-83aa2a01d193"
2026-04-23T00:16:35.570Z [ERROR] server-cae752ff-25be-ac93-57b4-726e0dd543ec: failed to appendEntries to: peer="{Voter server-03b797ef-05f5-245f-38b7-872ec65171f1 03b797ef-05f5-245f-38b7-872ec65171f1}" error="failed to connect to peer: 03b797ef-05f5-245f-38b7-872ec65171f1"
2026-04-23T00:16:35.572Z [ERROR] server-cae752ff-25be-ac93-57b4-726e0dd543ec: failed to heartbeat to: peer=03b797ef-05f5-245f-38b7-872ec65171f1 backoff time=10ms error="failed to connect to peer: 03b797ef-05f5-245f-38b7-872ec65171f1"
2026-04-23T00:16:35.582Z [ERROR] server-cae752ff-25be-ac93-57b4-726e0dd543ec: failed to heartbeat to: peer=988f188e-70d7-81ae-4807-83aa2a01d193 backoff time=10ms error="failed to connect to peer: 988f188e-70d7-81ae-4807-83aa2a01d193"
2026-04-23T00:16:35.583Z [ERROR] server-cae752ff-25be-ac93-57b4-726e0dd543ec: failed to appendEntries to: peer="{Voter server-988f188e-70d7-81ae-4807-83aa2a01d193 988f188e-70d7-81ae-4807-83aa2a01d193}" error="failed to connect to peer: 988f188e-70d7-81ae-4807-83aa2a01d193"
2026-04-23T00:16:35.585Z [WARN]  server-cae752ff-25be-ac93-57b4-726e0dd543ec: failed to contact: server-id=server-988f188e-70d7-81ae-4807-83aa2a01d193 time=50.398215ms
2026-04-23T00:16:35.585Z [WARN]  server-cae752ff-25be-ac93-57b4-726e0dd543ec: failed to contact: server-id=server-03b797ef-05f5-245f-38b7-872ec65171f1 time=50.407493ms
2026-04-23T00:16:35.585Z [WARN]  server-cae752ff-25be-ac93-57b4-726e0dd543ec: failed to contact quorum of nodes, stepping down
2026-04-23T00:16:35.585Z [INFO]  server-cae752ff-25be-ac93-57b4-726e0dd543ec: entering follower state: follower="Node at cae752ff-25be-ac93-57b4-726e0dd543ec [Follower]" leader-address= leader-id=
2026-04-23T00:16:35.586Z [ERROR] server-cae752ff-25be-ac93-57b4-726e0dd543ec: failed to appendEntries to: peer="{Voter server-03b797ef-05f5-245f-38b7-872ec65171f1 03b797ef-05f5-245f-38b7-872ec65171f1}" error="failed to connect to peer: 03b797ef-05f5-245f-38b7-872ec65171f1"
2026-04-23T00:16:35.629Z [WARN]  server-03b797ef-05f5-245f-38b7-872ec65171f1: heartbeat timeout reached, starting election: last-leader-addr=cae752ff-25be-ac93-57b4-726e0dd543ec last-leader-id=server-cae752ff-25be-ac93-57b4-726e0dd543ec
2026-04-23T00:16:35.629Z [INFO]  server-03b797ef-05f5-245f-38b7-872ec65171f1: entering candidate state: node="Node at 03b797ef-05f5-245f-38b7-872ec65171f1 [Candidate]" term=3
2026-04-23T00:16:35.629Z [ERROR] server-03b797ef-05f5-245f-38b7-872ec65171f1: failed to make requestVote RPC: target="{Voter server-cae752ff-25be-ac93-57b4-726e0dd543ec cae752ff-25be-ac93-57b4-726e0dd543ec}" error="failed to connect to peer: cae752ff-25be-ac93-57b4-726e0dd543ec" term=3
2026-04-23T00:16:35.629Z [WARN]  server-988f188e-70d7-81ae-4807-83aa2a01d193: rejecting vote request since we have a leader: from=03b797ef-05f5-245f-38b7-872ec65171f1 leader=cae752ff-25be-ac93-57b4-726e0dd543ec leader-id=server-cae752ff-25be-ac93-57b4-726e0dd543ec
2026-04-23T00:16:35.651Z [WARN]  server-988f188e-70d7-81ae-4807-83aa2a01d193: heartbeat timeout reached, starting election: last-leader-addr=cae752ff-25be-ac93-57b4-726e0dd543ec last-leader-id=server-cae752ff-25be-ac93-57b4-726e0dd543ec
2026-04-23T00:16:35.651Z [INFO]  server-988f188e-70d7-81ae-4807-83aa2a01d193: entering candidate state: node="Node at 988f188e-70d7-81ae-4807-83aa2a01d193 [Candidate]" term=3
2026-04-23T00:16:35.651Z [INFO]  server-988f188e-70d7-81ae-4807-83aa2a01d193: entering follower state: follower="Node at 988f188e-70d7-81ae-4807-83aa2a01d193 [Follower]" leader-address= leader-id=
2026-04-23T00:16:35.651Z [INFO]  server-03b797ef-05f5-245f-38b7-872ec65171f1: duplicate requestVote for same term: term=3
2026-04-23T00:16:35.651Z [ERROR] server-988f188e-70d7-81ae-4807-83aa2a01d193: failed to make requestVote RPC: target="{Voter server-cae752ff-25be-ac93-57b4-726e0dd543ec cae752ff-25be-ac93-57b4-726e0dd543ec}" error="failed to connect to peer: cae752ff-25be-ac93-57b4-726e0dd543ec" term=3
--- PASS: TestRaft_VoteWithNoIDNoAddr (0.51s)
=== RUN   TestRaft_runFollower_State_Transition
=== RUN   TestRaft_runFollower_State_Transition/NonVoter
2026-04-23T00:16:35.652Z [INFO]  127.0.0.1:34847 :: starting node: addr=127.0.0.1:34847
2026-04-23T00:16:35.652Z [INFO]  127.0.0.1:34847 :: initial configuration: index=0 servers=[]
2026-04-23T00:16:35.652Z [INFO]  127.0.0.1:34847 :: entering follower state: follower="Node at 127.0.0.1:34847 [Follower]" leader-address= leader-id=
2026-04-23T00:16:35.723Z [WARN]  127.0.0.1:34847 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
=== RUN   TestRaft_runFollower_State_Transition/Voter
2026-04-23T00:16:35.803Z [INFO]  127.0.0.1:45637 :: starting node: addr=127.0.0.1:45637
2026-04-23T00:16:35.803Z [INFO]  127.0.0.1:45637 :: initial configuration: index=0 servers=[]
2026-04-23T00:16:35.803Z [INFO]  127.0.0.1:45637 :: entering follower state: follower="Node at 127.0.0.1:45637 [Follower]" leader-address= leader-id=
2026-04-23T00:16:35.881Z [WARN]  127.0.0.1:45637 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
=== RUN   TestRaft_runFollower_State_Transition/Not_in_Config
2026-04-23T00:16:35.970Z [INFO]  127.0.0.1:41959 :: starting node: addr=127.0.0.1:41959
2026-04-23T00:16:35.970Z [INFO]  127.0.0.1:41959 :: initial configuration: index=0 servers=[]
2026-04-23T00:16:35.970Z [INFO]  127.0.0.1:41959 :: entering follower state: follower="Node at 127.0.0.1:41959 [Follower]" leader-address= leader-id=
2026-04-23T00:16:36.045Z [WARN]  127.0.0.1:41959 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
--- PASS: TestRaft_runFollower_State_Transition (0.47s)
    --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.17s)
=== RUN   TestRaft_runFollower_ReloadTimeoutConfigs
2026-04-23T00:16:36.120Z [INFO]  127.0.0.1:35109 :: starting node: addr=127.0.0.1:35109
2026-04-23T00:16:36.120Z [INFO]  127.0.0.1:35109 :: initial configuration: index=0 servers=[]
2026-04-23T00:16:36.120Z [INFO]  127.0.0.1:35109 :: entering follower state: follower="Node at 127.0.0.1:35109 [Follower]" leader-address= leader-id=
2026-04-23T00:16:36.120Z [WARN]  127.0.0.1:35109 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
--- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s)
=== RUN   TestSaturationMetric
=== RUN   TestSaturationMetric/without_smoothing
--- PASS: TestSaturationMetric (0.00s)
    --- PASS: TestSaturationMetric/without_smoothing (0.00s)
=== RUN   TestSaturationMetric_IncorrectUsage
=== RUN   TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_first
--- PASS: TestSaturationMetric_IncorrectUsage (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s)
=== RUN   TestTCPTransport_BadAddr
--- PASS: TestTCPTransport_BadAddr (0.00s)
=== RUN   TestTCPTransport_EmptyAddr
--- PASS: TestTCPTransport_EmptyAddr (0.00s)
=== RUN   TestTCPTransport_WithAdvertise
--- PASS: TestTCPTransport_WithAdvertise (0.00s)
=== RUN   TestTransport_StartStop
--- PASS: TestTransport_StartStop (0.00s)
=== RUN   TestTransport_AppendEntries
--- PASS: TestTransport_AppendEntries (0.00s)
=== RUN   TestTransport_AppendEntriesPipeline
--- PASS: TestTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestTransport_RequestVote
--- PASS: TestTransport_RequestVote (0.00s)
=== RUN   TestTransport_InstallSnapshot
--- PASS: TestTransport_InstallSnapshot (0.00s)
=== RUN   TestTransport_EncodeDecode
--- PASS: TestTransport_EncodeDecode (0.00s)
=== RUN   TestRandomTimeout
--- PASS: TestRandomTimeout (0.00s)
=== RUN   TestNewSeed
--- PASS: TestNewSeed (0.00s)
=== RUN   TestRandomTimeout_NoTime
--- PASS: TestRandomTimeout_NoTime (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
=== RUN   TestMax
--- PASS: TestMax (0.00s)
=== RUN   TestGenerateUUID
--- PASS: TestGenerateUUID (0.00s)
=== RUN   TestBackoff
--- PASS: TestBackoff (0.00s)
=== RUN   TestOverrideNotifyBool
--- PASS: TestOverrideNotifyBool (0.00s)
PASS
ok  	github.com/hashicorp/raft	113.323s
?   	github.com/hashicorp/raft/bench	[no test files]
make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
   create-stamp debian/debhelper-build-stamp
   dh_testroot -O--buildsystem=golang
   dh_prep -O--buildsystem=golang
   dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang
   dh_installdocs -O--buildsystem=golang
   dh_installchangelogs -O--buildsystem=golang
   dh_installsystemduser -O--buildsystem=golang
   dh_perl -O--buildsystem=golang
   dh_link -O--buildsystem=golang
   dh_strip_nondeterminism -O--buildsystem=golang
   dh_compress -O--buildsystem=golang
   dh_fixperms -O--buildsystem=golang
   dh_missing -O--buildsystem=golang
   dh_installdeb -O--buildsystem=golang
   dh_golang -O--buildsystem=golang
   dh_gencontrol -O--buildsystem=golang
   dh_md5sums -O--buildsystem=golang
   dh_builddeb -O--buildsystem=golang
dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'.
 dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_amd64.buildinfo
 dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_amd64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-buildpackage: info: binary-only upload (no source included)
dpkg-genchanges: info: including full source code in upload
I: copying local configuration
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /srv/workspace/pbuilder/598648 and its subdirectories
I: Current time: Wed Apr 22 12:16:41 -12 2026
I: pbuilder-time-stamp: 1776903401
Thu Mar 20 17:53:44 UTC 2025  I: 1st build successful. Starting 2nd build on remote node ionos11-amd64.debian.net.
Thu Mar 20 17:53:44 UTC 2025  I: Preparing to do remote build '2' on ionos11-amd64.debian.net.
Thu Mar 20 18:03:21 UTC 2025  I: Deleting $TMPDIR on ionos11-amd64.debian.net.