I: pbuilder: network access will be disabled during build
I: Current time: Sun Mar 29 15:06:38 +14 2026
I: pbuilder-time-stamp: 1774746398
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/trixie-reproducible-base.tgz]
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: policy-rc.d already exists
I: using eatmydata during job
I: Copying source file
I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc]
I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz]
I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz]
I: Extracting source
dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: unsupported subcommand
dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying 0001-Skip-unreliable-tests.patch
I: Not using root during the build.
I: Installing the build-deps
I: user script /srv/workspace/pbuilder/34756/tmp/hooks/D01_modify_environment starting
debug: Running on ionos16-i386.
I: Changing host+domainname to test build reproducibility
I: Adding a custom variable just for the fun of it...
I: Changing /bin/sh to bash
'/bin/sh' -> '/bin/bash'
lrwxrwxrwx 1 root root 9 Mar 29 01:06 /bin/sh -> /bin/bash
I: Setting pbuilder2's login shell to /bin/bash
I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other
I: user script /srv/workspace/pbuilder/34756/tmp/hooks/D01_modify_environment finished
I: user script /srv/workspace/pbuilder/34756/tmp/hooks/D02_print_environment starting
I: set
  BASH=/bin/sh
  BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:globskipdots:hostcomplete:interactive_comments:patsub_replacement:progcomp:promptvars:sourcepath
  BASH_ALIASES=()
  BASH_ARGC=()
  BASH_ARGV=()
  BASH_CMDS=()
  BASH_LINENO=([0]="12" [1]="0")
  BASH_LOADABLES_PATH=/usr/local/lib/bash:/usr/lib/bash:/opt/local/lib/bash:/usr/pkg/lib/bash:/opt/pkg/lib/bash:.
  BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment")
  BASH_VERSINFO=([0]="5" [1]="2" [2]="37" [3]="1" [4]="release" [5]="i686-pc-linux-gnu")
  BASH_VERSION='5.2.37(1)-release'
  BUILDDIR=/build/reproducible-path
  BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other'
  BUILDUSERNAME=pbuilder2
  BUILD_ARCH=i386
  DEBIAN_FRONTEND=noninteractive
  DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=21 '
  DIRSTACK=()
  DISTRIBUTION=trixie
  EUID=0
  FUNCNAME=([0]="Echo" [1]="main")
  GROUPS=()
  HOME=/root
  HOSTNAME=i-capture-the-hostname
  HOSTTYPE=i686
  HOST_ARCH=i386
  IFS=' 	
  '
  INVOCATION_ID=01981242902c44c8919bb3a2469bbbfd
  LANG=C
  LANGUAGE=de_CH:de
  LC_ALL=C
  LD_LIBRARY_PATH=/usr/lib/libeatmydata
  LD_PRELOAD=libeatmydata.so
  MACHTYPE=i686-pc-linux-gnu
  MAIL=/var/mail/root
  OPTERR=1
  OPTIND=1
  OSTYPE=linux-gnu
  PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path
  PBCURRENTCOMMANDLINEOPERATION=build
  PBUILDER_OPERATION=build
  PBUILDER_PKGDATADIR=/usr/share/pbuilder
  PBUILDER_PKGLIBDIR=/usr/lib/pbuilder
  PBUILDER_SYSCONFDIR=/etc
  PIPESTATUS=([0]="0")
  POSIXLY_CORRECT=y
  PPID=34756
  PS4='+ '
  PWD=/
  SHELL=/bin/bash
  SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix
  SHLVL=3
  SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.9TGEveym/pbuilderrc_dAhT --distribution trixie --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/trixie-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.9TGEveym/b2 --logfile b2/build.log golang-github-hashicorp-raft_1.5.0-1.dsc'
  SUDO_GID=112
  SUDO_UID=107
  SUDO_USER=jenkins
  TERM=unknown
  TZ=/usr/share/zoneinfo/Etc/GMT-14
  UID=0
  USER=root
  _='I: set'
  http_proxy=http://213.165.73.152:3128
I: uname -a
  Linux i-capture-the-hostname 6.1.0-31-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.128-1 (2025-02-07) x86_64 GNU/Linux
I: ls -l /bin
  lrwxrwxrwx 1 root root 7 Nov 22  2024 /bin -> usr/bin
I: user script /srv/workspace/pbuilder/34756/tmp/hooks/D02_print_environment finished
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: i386
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 19841 files and directories currently installed.)
Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ...
Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however:
  Package debhelper-compat is not installed.
 pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however:
  Package dh-sequence-golang is not installed.
 pbuilder-satisfydepends-dummy depends on golang-any; however:
  Package golang-any is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however:
  Package golang-github-armon-go-metrics-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however:
  Package golang-github-hashicorp-go-hclog-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however:
  Package golang-github-hashicorp-go-msgpack-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however:
  Package golang-github-stretchr-testify-dev is not installed.

Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
Building tag database...
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
The following NEW packages will be installed:
  autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.24-go{a} golang-1.24-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-prometheus-sigv4-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libunistring5{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} 
The following packages are RECOMMENDED but will NOT be installed:
  curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 
0 packages upgraded, 107 newly installed, 0 to remove and 0 not upgraded.
Need to get 129 MB of archives. After unpacking 699 MB will be used.
Writing extended state information...
Get: 1 http://deb.debian.org/debian trixie/main i386 sensible-utils all 0.0.24 [24.8 kB]
Get: 2 http://deb.debian.org/debian trixie/main i386 libmagic-mgc i386 1:5.45-3+b1 [314 kB]
Get: 3 http://deb.debian.org/debian trixie/main i386 libmagic1t64 i386 1:5.45-3+b1 [115 kB]
Get: 4 http://deb.debian.org/debian trixie/main i386 file i386 1:5.45-3+b1 [43.2 kB]
Get: 5 http://deb.debian.org/debian trixie/main i386 gettext-base i386 0.23.1-1 [245 kB]
Get: 6 http://deb.debian.org/debian trixie/main i386 libuchardet0 i386 0.0.8-1+b2 [69.2 kB]
Get: 7 http://deb.debian.org/debian trixie/main i386 groff-base i386 1.23.0-7 [1199 kB]
Get: 8 http://deb.debian.org/debian trixie/main i386 bsdextrautils i386 2.40.4-3 [96.2 kB]
Get: 9 http://deb.debian.org/debian trixie/main i386 libpipeline1 i386 1.5.8-1 [41.2 kB]
Get: 10 http://deb.debian.org/debian trixie/main i386 man-db i386 2.13.0-1 [1428 kB]
Get: 11 http://deb.debian.org/debian trixie/main i386 m4 i386 1.4.19-5 [301 kB]
Get: 12 http://deb.debian.org/debian trixie/main i386 autoconf all 2.72-3 [493 kB]
Get: 13 http://deb.debian.org/debian trixie/main i386 autotools-dev all 20220109.1 [51.6 kB]
Get: 14 http://deb.debian.org/debian trixie/main i386 automake all 1:1.17-3 [862 kB]
Get: 15 http://deb.debian.org/debian trixie/main i386 autopoint all 0.23.1-1 [770 kB]
Get: 16 http://deb.debian.org/debian trixie/main i386 libdebhelper-perl all 13.24.1 [90.9 kB]
Get: 17 http://deb.debian.org/debian trixie/main i386 libtool all 2.5.4-3 [539 kB]
Get: 18 http://deb.debian.org/debian trixie/main i386 dh-autoreconf all 20 [17.1 kB]
Get: 19 http://deb.debian.org/debian trixie/main i386 libarchive-zip-perl all 1.68-1 [104 kB]
Get: 20 http://deb.debian.org/debian trixie/main i386 libfile-stripnondeterminism-perl all 1.14.1-2 [19.7 kB]
Get: 21 http://deb.debian.org/debian trixie/main i386 dh-strip-nondeterminism all 1.14.1-2 [8620 B]
Get: 22 http://deb.debian.org/debian trixie/main i386 libelf1t64 i386 0.192-4 [195 kB]
Get: 23 http://deb.debian.org/debian trixie/main i386 dwz i386 0.15-1+b1 [116 kB]
Get: 24 http://deb.debian.org/debian trixie/main i386 libunistring5 i386 1.3-1 [458 kB]
Get: 25 http://deb.debian.org/debian trixie/main i386 libicu72 i386 72.1-6 [9582 kB]
Get: 26 http://deb.debian.org/debian trixie/main i386 libxml2 i386 2.12.7+dfsg+really2.9.14-0.2+b1 [734 kB]
Get: 27 http://deb.debian.org/debian trixie/main i386 gettext i386 0.23.1-1 [1714 kB]
Get: 28 http://deb.debian.org/debian trixie/main i386 intltool-debian all 0.35.0+20060710.6 [22.9 kB]
Get: 29 http://deb.debian.org/debian trixie/main i386 po-debconf all 1.0.21+nmu1 [248 kB]
Get: 30 http://deb.debian.org/debian trixie/main i386 debhelper all 13.24.1 [920 kB]
Get: 31 http://deb.debian.org/debian trixie/main i386 dh-golang all 1.63 [24.4 kB]
Get: 32 http://deb.debian.org/debian trixie/main i386 golang-1.24-src all 1.24.0-1 [20.3 MB]
Get: 33 http://deb.debian.org/debian trixie/main i386 golang-1.24-go i386 1.24.0-1 [26.8 MB]
Get: 34 http://deb.debian.org/debian trixie/main i386 golang-src all 2:1.24~2 [5136 B]
Get: 35 http://deb.debian.org/debian trixie/main i386 golang-go i386 2:1.24~2 [44.3 kB]
Get: 36 http://deb.debian.org/debian trixie/main i386 golang-any i386 2:1.24~2 [5216 B]
Get: 37 http://deb.debian.org/debian trixie/main i386 golang-github-alecthomas-units-dev all 0.0~git20240626.61d1e34-1 [9032 B]
Get: 38 http://deb.debian.org/debian trixie/main i386 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB]
Get: 39 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB]
Get: 40 http://deb.debian.org/debian trixie/main i386 golang-golang-x-sys-dev all 0.22.0-1 [431 kB]
Get: 41 http://deb.debian.org/debian trixie/main i386 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B]
Get: 42 http://deb.debian.org/debian trixie/main i386 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB]
Get: 43 http://deb.debian.org/debian trixie/main i386 golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB]
Get: 44 http://deb.debian.org/debian trixie/main i386 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB]
Get: 45 http://deb.debian.org/debian trixie/main i386 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB]
Get: 46 http://deb.debian.org/debian trixie/main i386 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB]
Get: 47 http://deb.debian.org/debian trixie/main i386 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB]
Get: 48 http://deb.debian.org/debian trixie/main i386 golang-github-stretchr-testify-dev all 1.9.0-1 [74.7 kB]
Get: 49 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB]
Get: 50 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB]
Get: 51 http://deb.debian.org/debian trixie/main i386 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB]
Get: 52 http://deb.debian.org/debian trixie/main i386 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B]
Get: 53 http://deb.debian.org/debian trixie/main i386 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB]
Get: 54 http://deb.debian.org/debian trixie/main i386 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB]
Get: 55 http://deb.debian.org/debian trixie/main i386 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB]
Get: 56 http://deb.debian.org/debian trixie/main i386 golang-google-protobuf-dev all 1.33.0-1 [625 kB]
Get: 57 http://deb.debian.org/debian trixie/main i386 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB]
Get: 58 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B]
Get: 59 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB]
Get: 60 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB]
Get: 61 http://deb.debian.org/debian trixie/main i386 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB]
Get: 62 http://deb.debian.org/debian trixie/main i386 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB]
Get: 63 http://deb.debian.org/debian trixie/main i386 golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB]
Get: 64 http://deb.debian.org/debian trixie/main i386 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B]
Get: 65 http://deb.debian.org/debian trixie/main i386 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB]
Get: 66 http://deb.debian.org/debian trixie/main i386 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB]
Get: 67 http://deb.debian.org/debian trixie/main i386 golang-github-klauspost-compress-dev all 1.17.11+ds1-1 [33.3 MB]
Get: 68 http://deb.debian.org/debian trixie/main i386 golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB]
Get: 69 http://deb.debian.org/debian trixie/main i386 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB]
Get: 70 http://deb.debian.org/debian trixie/main i386 golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB]
Get: 71 http://deb.debian.org/debian trixie/main i386 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB]
Get: 72 http://deb.debian.org/debian trixie/main i386 golang-golang-x-term-dev all 0.22.0-1 [14.8 kB]
Get: 73 http://deb.debian.org/debian trixie/main i386 golang-golang-x-text-dev all 0.22.0-1 [3995 kB]
Get: 74 http://deb.debian.org/debian trixie/main i386 golang-golang-x-crypto-dev all 1:0.25.0-1 [1682 kB]
Get: 75 http://deb.debian.org/debian trixie/main i386 golang-golang-x-net-dev all 1:0.27.0-1 [898 kB]
Get: 76 http://deb.debian.org/debian trixie/main i386 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB]
Get: 77 http://deb.debian.org/debian trixie/main i386 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB]
Get: 78 http://deb.debian.org/debian trixie/main i386 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB]
Get: 79 http://deb.debian.org/debian trixie/main i386 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB]
Get: 80 http://deb.debian.org/debian trixie/main i386 golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B]
Get: 81 http://deb.debian.org/debian trixie/main i386 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B]
Get: 82 http://deb.debian.org/debian trixie/main i386 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB]
Get: 83 http://deb.debian.org/debian trixie/main i386 golang-github-prometheus-sigv4-dev all 0.1.2-2 [8484 B]
Get: 84 http://deb.debian.org/debian trixie/main i386 golang-golang-x-oauth2-dev all 0.15.0-1 [38.0 kB]
Get: 85 http://deb.debian.org/debian trixie/main i386 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB]
Get: 86 http://deb.debian.org/debian trixie/main i386 golang-github-prometheus-common-dev all 0.62.0-3 [118 kB]
Get: 87 http://deb.debian.org/debian trixie/main i386 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB]
Get: 88 http://deb.debian.org/debian trixie/main i386 golang-github-prometheus-procfs-dev all 0.15.1-1 [184 kB]
Get: 89 http://deb.debian.org/debian trixie/main i386 golang-github-prometheus-client-golang-dev all 1.21.0-1 [195 kB]
Get: 90 http://deb.debian.org/debian trixie/main i386 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB]
Get: 91 http://deb.debian.org/debian trixie/main i386 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB]
Get: 92 http://deb.debian.org/debian trixie/main i386 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B]
Get: 93 http://deb.debian.org/debian trixie/main i386 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB]
Get: 94 http://deb.debian.org/debian trixie/main i386 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B]
Get: 95 http://deb.debian.org/debian trixie/main i386 libsasl2-modules-db i386 2.1.28+dfsg1-8+b1 [20.9 kB]
Get: 96 http://deb.debian.org/debian trixie/main i386 libsasl2-2 i386 2.1.28+dfsg1-8+b1 [61.3 kB]
Get: 97 http://deb.debian.org/debian trixie/main i386 libssl-dev i386 3.4.1-1 [2837 kB]
Get: 98 http://deb.debian.org/debian trixie/main i386 libsasl2-dev i386 2.1.28+dfsg1-8+b1 [248 kB]
Get: 99 http://deb.debian.org/debian trixie/main i386 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB]
Get: 100 http://deb.debian.org/debian trixie/main i386 golang-golang-x-mod-dev all 0.19.0-1 [104 kB]
Get: 101 http://deb.debian.org/debian trixie/main i386 golang-github-yuin-goldmark-dev all 1.7.4-1 [113 kB]
Get: 102 http://deb.debian.org/debian trixie/main i386 golang-golang-x-tools-dev all 1:0.23.0+ds-1 [1593 kB]
Get: 103 http://deb.debian.org/debian trixie/main i386 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB]
Get: 104 http://deb.debian.org/debian trixie/main i386 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB]
Get: 105 http://deb.debian.org/debian trixie/main i386 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB]
Get: 106 http://deb.debian.org/debian trixie/main i386 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB]
Get: 107 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB]
Fetched 129 MB in 1s (95.9 MB/s)
Preconfiguring packages ...
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 19841 files and directories currently installed.)
Preparing to unpack .../000-sensible-utils_0.0.24_all.deb ...
Unpacking sensible-utils (0.0.24) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3+b1_i386.deb ...
Unpacking libmagic-mgc (1:5.45-3+b1) ...
Selecting previously unselected package libmagic1t64:i386.
Preparing to unpack .../002-libmagic1t64_1%3a5.45-3+b1_i386.deb ...
Unpacking libmagic1t64:i386 (1:5.45-3+b1) ...
Selecting previously unselected package file.
Preparing to unpack .../003-file_1%3a5.45-3+b1_i386.deb ...
Unpacking file (1:5.45-3+b1) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../004-gettext-base_0.23.1-1_i386.deb ...
Unpacking gettext-base (0.23.1-1) ...
Selecting previously unselected package libuchardet0:i386.
Preparing to unpack .../005-libuchardet0_0.0.8-1+b2_i386.deb ...
Unpacking libuchardet0:i386 (0.0.8-1+b2) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../006-groff-base_1.23.0-7_i386.deb ...
Unpacking groff-base (1.23.0-7) ...
Selecting previously unselected package bsdextrautils.
Preparing to unpack .../007-bsdextrautils_2.40.4-3_i386.deb ...
Unpacking bsdextrautils (2.40.4-3) ...
Selecting previously unselected package libpipeline1:i386.
Preparing to unpack .../008-libpipeline1_1.5.8-1_i386.deb ...
Unpacking libpipeline1:i386 (1.5.8-1) ...
Selecting previously unselected package man-db.
Preparing to unpack .../009-man-db_2.13.0-1_i386.deb ...
Unpacking man-db (2.13.0-1) ...
Selecting previously unselected package m4.
Preparing to unpack .../010-m4_1.4.19-5_i386.deb ...
Unpacking m4 (1.4.19-5) ...
Selecting previously unselected package autoconf.
Preparing to unpack .../011-autoconf_2.72-3_all.deb ...
Unpacking autoconf (2.72-3) ...
Selecting previously unselected package autotools-dev.
Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ...
Unpacking autotools-dev (20220109.1) ...
Selecting previously unselected package automake.
Preparing to unpack .../013-automake_1%3a1.17-3_all.deb ...
Unpacking automake (1:1.17-3) ...
Selecting previously unselected package autopoint.
Preparing to unpack .../014-autopoint_0.23.1-1_all.deb ...
Unpacking autopoint (0.23.1-1) ...
Selecting previously unselected package libdebhelper-perl.
Preparing to unpack .../015-libdebhelper-perl_13.24.1_all.deb ...
Unpacking libdebhelper-perl (13.24.1) ...
Selecting previously unselected package libtool.
Preparing to unpack .../016-libtool_2.5.4-3_all.deb ...
Unpacking libtool (2.5.4-3) ...
Selecting previously unselected package dh-autoreconf.
Preparing to unpack .../017-dh-autoreconf_20_all.deb ...
Unpacking dh-autoreconf (20) ...
Selecting previously unselected package libarchive-zip-perl.
Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ...
Unpacking libarchive-zip-perl (1.68-1) ...
Selecting previously unselected package libfile-stripnondeterminism-perl.
Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.1-2_all.deb ...
Unpacking libfile-stripnondeterminism-perl (1.14.1-2) ...
Selecting previously unselected package dh-strip-nondeterminism.
Preparing to unpack .../020-dh-strip-nondeterminism_1.14.1-2_all.deb ...
Unpacking dh-strip-nondeterminism (1.14.1-2) ...
Selecting previously unselected package libelf1t64:i386.
Preparing to unpack .../021-libelf1t64_0.192-4_i386.deb ...
Unpacking libelf1t64:i386 (0.192-4) ...
Selecting previously unselected package dwz.
Preparing to unpack .../022-dwz_0.15-1+b1_i386.deb ...
Unpacking dwz (0.15-1+b1) ...
Selecting previously unselected package libunistring5:i386.
Preparing to unpack .../023-libunistring5_1.3-1_i386.deb ...
Unpacking libunistring5:i386 (1.3-1) ...
Selecting previously unselected package libicu72:i386.
Preparing to unpack .../024-libicu72_72.1-6_i386.deb ...
Unpacking libicu72:i386 (72.1-6) ...
Selecting previously unselected package libxml2:i386.
Preparing to unpack .../025-libxml2_2.12.7+dfsg+really2.9.14-0.2+b1_i386.deb ...
Unpacking libxml2:i386 (2.12.7+dfsg+really2.9.14-0.2+b1) ...
Selecting previously unselected package gettext.
Preparing to unpack .../026-gettext_0.23.1-1_i386.deb ...
Unpacking gettext (0.23.1-1) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../027-intltool-debian_0.35.0+20060710.6_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.6) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../028-po-debconf_1.0.21+nmu1_all.deb ...
Unpacking po-debconf (1.0.21+nmu1) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../029-debhelper_13.24.1_all.deb ...
Unpacking debhelper (13.24.1) ...
Selecting previously unselected package dh-golang.
Preparing to unpack .../030-dh-golang_1.63_all.deb ...
Unpacking dh-golang (1.63) ...
Selecting previously unselected package golang-1.24-src.
Preparing to unpack .../031-golang-1.24-src_1.24.0-1_all.deb ...
Unpacking golang-1.24-src (1.24.0-1) ...
Selecting previously unselected package golang-1.24-go.
Preparing to unpack .../032-golang-1.24-go_1.24.0-1_i386.deb ...
Unpacking golang-1.24-go (1.24.0-1) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../033-golang-src_2%3a1.24~2_all.deb ...
Unpacking golang-src (2:1.24~2) ...
Selecting previously unselected package golang-go:i386.
Preparing to unpack .../034-golang-go_2%3a1.24~2_i386.deb ...
Unpacking golang-go:i386 (2:1.24~2) ...
Selecting previously unselected package golang-any:i386.
Preparing to unpack .../035-golang-any_2%3a1.24~2_i386.deb ...
Unpacking golang-any:i386 (2:1.24~2) ...
Selecting previously unselected package golang-github-alecthomas-units-dev.
Preparing to unpack .../036-golang-github-alecthomas-units-dev_0.0~git20240626.61d1e34-1_all.deb ...
Unpacking golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ...
Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev.
Preparing to unpack .../037-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ...
Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev.
Preparing to unpack .../038-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ...
Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Selecting previously unselected package golang-golang-x-sys-dev.
Preparing to unpack .../039-golang-golang-x-sys-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-sys-dev (0.22.0-1) ...
Selecting previously unselected package golang-github-mattn-go-isatty-dev.
Preparing to unpack .../040-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ...
Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Selecting previously unselected package golang-github-mattn-go-colorable-dev.
Preparing to unpack .../041-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ...
Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Selecting previously unselected package golang-github-fatih-color-dev.
Preparing to unpack .../042-golang-github-fatih-color-dev_1.17.0-1_all.deb ...
Unpacking golang-github-fatih-color-dev (1.17.0-1) ...
Selecting previously unselected package golang-github-davecgh-go-spew-dev.
Preparing to unpack .../043-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ...
Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Selecting previously unselected package golang-github-pmezard-go-difflib-dev.
Preparing to unpack .../044-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ...
Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Selecting previously unselected package golang-github-stretchr-objx-dev.
Preparing to unpack .../045-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ...
Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ...
Selecting previously unselected package golang-gopkg-yaml.v3-dev.
Preparing to unpack .../046-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ...
Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Selecting previously unselected package golang-github-stretchr-testify-dev.
Preparing to unpack .../047-golang-github-stretchr-testify-dev_1.9.0-1_all.deb ...
Unpacking golang-github-stretchr-testify-dev (1.9.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-hclog-dev.
Preparing to unpack .../048-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ...
Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev.
Preparing to unpack .../049-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ...
Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Selecting previously unselected package golang-github-pkg-errors-dev.
Preparing to unpack .../050-golang-github-pkg-errors-dev_0.9.1-3_all.deb ...
Unpacking golang-github-pkg-errors-dev (0.9.1-3) ...
Selecting previously unselected package golang-github-tv42-httpunix-dev.
Preparing to unpack .../051-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ...
Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev.
Preparing to unpack .../052-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ...
Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Selecting previously unselected package golang-github-datadog-datadog-go-dev.
Preparing to unpack .../053-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ...
Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Selecting previously unselected package golang-github-google-go-cmp-dev.
Preparing to unpack .../054-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ...
Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ...
Selecting previously unselected package golang-google-protobuf-dev.
Preparing to unpack .../055-golang-google-protobuf-dev_1.33.0-1_all.deb ...
Unpacking golang-google-protobuf-dev (1.33.0-1) ...
Selecting previously unselected package golang-github-golang-protobuf-1-5-dev.
Preparing to unpack .../056-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ...
Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Selecting previously unselected package golang-github-hashicorp-go-uuid-dev.
Preparing to unpack .../057-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ...
Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Selecting previously unselected package golang-github-hashicorp-golang-lru-dev.
Preparing to unpack .../058-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ...
Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev.
Preparing to unpack .../059-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ...
Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Selecting previously unselected package golang-github-pascaldekloe-goe-dev.
Preparing to unpack .../060-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ...
Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Selecting previously unselected package golang-github-beorn7-perks-dev.
Preparing to unpack .../061-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ...
Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ...
Selecting previously unselected package golang-github-cespare-xxhash-dev.
Preparing to unpack .../062-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ...
Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ...
Selecting previously unselected package golang-github-modern-go-concurrent-dev.
Preparing to unpack .../063-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ...
Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Selecting previously unselected package golang-github-modern-go-reflect2-dev.
Preparing to unpack .../064-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ...
Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-json-iterator-go-dev.
Preparing to unpack .../065-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ...
Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ...
Selecting previously unselected package golang-github-klauspost-compress-dev.
Preparing to unpack .../066-golang-github-klauspost-compress-dev_1.17.11+ds1-1_all.deb ...
Unpacking golang-github-klauspost-compress-dev (1.17.11+ds1-1) ...
Selecting previously unselected package golang-github-kylelemons-godebug-dev.
Preparing to unpack .../067-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ...
Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Selecting previously unselected package golang-github-prometheus-client-model-dev.
Preparing to unpack .../068-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ...
Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ...
Selecting previously unselected package golang-gopkg-yaml.v2-dev.
Preparing to unpack .../069-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ...
Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Selecting previously unselected package golang-github-jmespath-go-jmespath-dev.
Preparing to unpack .../070-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ...
Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Selecting previously unselected package golang-golang-x-term-dev.
Preparing to unpack .../071-golang-golang-x-term-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-term-dev (0.22.0-1) ...
Selecting previously unselected package golang-golang-x-text-dev.
Preparing to unpack .../072-golang-golang-x-text-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-text-dev (0.22.0-1) ...
Selecting previously unselected package golang-golang-x-crypto-dev.
Preparing to unpack .../073-golang-golang-x-crypto-dev_1%3a0.25.0-1_all.deb ...
Unpacking golang-golang-x-crypto-dev (1:0.25.0-1) ...
Selecting previously unselected package golang-golang-x-net-dev.
Preparing to unpack .../074-golang-golang-x-net-dev_1%3a0.27.0-1_all.deb ...
Unpacking golang-golang-x-net-dev (1:0.27.0-1) ...
Selecting previously unselected package golang-github-aws-aws-sdk-go-dev.
Preparing to unpack .../075-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ...
Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Selecting previously unselected package golang-github-go-logfmt-logfmt-dev.
Preparing to unpack .../076-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ...
Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Selecting previously unselected package golang-github-go-kit-log-dev.
Preparing to unpack .../077-golang-github-go-kit-log-dev_0.2.1-1_all.deb ...
Unpacking golang-github-go-kit-log-dev (0.2.1-1) ...
Selecting previously unselected package golang-github-julienschmidt-httprouter-dev.
Preparing to unpack .../078-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ...
Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Selecting previously unselected package golang-github-munnerz-goautoneg-dev.
Preparing to unpack .../079-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ...
Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Selecting previously unselected package golang-github-jpillora-backoff-dev.
Preparing to unpack .../080-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ...
Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev.
Preparing to unpack .../081-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ...
Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Selecting previously unselected package golang-github-prometheus-sigv4-dev.
Preparing to unpack .../082-golang-github-prometheus-sigv4-dev_0.1.2-2_all.deb ...
Unpacking golang-github-prometheus-sigv4-dev (0.1.2-2) ...
Selecting previously unselected package golang-golang-x-oauth2-dev.
Preparing to unpack .../083-golang-golang-x-oauth2-dev_0.15.0-1_all.deb ...
Unpacking golang-golang-x-oauth2-dev (0.15.0-1) ...
Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev.
Preparing to unpack .../084-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ...
Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Selecting previously unselected package golang-github-prometheus-common-dev.
Preparing to unpack .../085-golang-github-prometheus-common-dev_0.62.0-3_all.deb ...
Unpacking golang-github-prometheus-common-dev (0.62.0-3) ...
Selecting previously unselected package golang-golang-x-sync-dev.
Preparing to unpack .../086-golang-golang-x-sync-dev_0.7.0-1_all.deb ...
Unpacking golang-golang-x-sync-dev (0.7.0-1) ...
Selecting previously unselected package golang-github-prometheus-procfs-dev.
Preparing to unpack .../087-golang-github-prometheus-procfs-dev_0.15.1-1_all.deb ...
Unpacking golang-github-prometheus-procfs-dev (0.15.1-1) ...
Selecting previously unselected package golang-github-prometheus-client-golang-dev.
Preparing to unpack .../088-golang-github-prometheus-client-golang-dev_1.21.0-1_all.deb ...
Unpacking golang-github-prometheus-client-golang-dev (1.21.0-1) ...
Selecting previously unselected package golang-github-armon-go-metrics-dev.
Preparing to unpack .../089-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ...
Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ...
Selecting previously unselected package golang-github-creack-pty-dev.
Preparing to unpack .../090-golang-github-creack-pty-dev_1.1.21-1_all.deb ...
Unpacking golang-github-creack-pty-dev (1.1.21-1) ...
Selecting previously unselected package golang-github-vmihailenco-tagparser-dev.
Preparing to unpack .../091-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ...
Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev.
Preparing to unpack .../092-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ...
Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Selecting previously unselected package golang-gopkg-tomb.v2-dev.
Preparing to unpack .../093-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ...
Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Selecting previously unselected package libsasl2-modules-db:i386.
Preparing to unpack .../094-libsasl2-modules-db_2.1.28+dfsg1-8+b1_i386.deb ...
Unpacking libsasl2-modules-db:i386 (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package libsasl2-2:i386.
Preparing to unpack .../095-libsasl2-2_2.1.28+dfsg1-8+b1_i386.deb ...
Unpacking libsasl2-2:i386 (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package libssl-dev:i386.
Preparing to unpack .../096-libssl-dev_3.4.1-1_i386.deb ...
Unpacking libssl-dev:i386 (3.4.1-1) ...
Selecting previously unselected package libsasl2-dev.
Preparing to unpack .../097-libsasl2-dev_2.1.28+dfsg1-8+b1_i386.deb ...
Unpacking libsasl2-dev (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package golang-github-kr-text-dev.
Preparing to unpack .../098-golang-github-kr-text-dev_0.2.0-2_all.deb ...
Unpacking golang-github-kr-text-dev (0.2.0-2) ...
Selecting previously unselected package golang-golang-x-mod-dev.
Preparing to unpack .../099-golang-golang-x-mod-dev_0.19.0-1_all.deb ...
Unpacking golang-golang-x-mod-dev (0.19.0-1) ...
Selecting previously unselected package golang-github-yuin-goldmark-dev.
Preparing to unpack .../100-golang-github-yuin-goldmark-dev_1.7.4-1_all.deb ...
Unpacking golang-github-yuin-goldmark-dev (1.7.4-1) ...
Selecting previously unselected package golang-golang-x-tools-dev.
Preparing to unpack .../101-golang-golang-x-tools-dev_1%3a0.23.0+ds-1_all.deb ...
Unpacking golang-golang-x-tools-dev (1:0.23.0+ds-1) ...
Selecting previously unselected package golang-github-rogpeppe-go-internal-dev.
Preparing to unpack .../102-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ...
Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Selecting previously unselected package golang-github-kr-pretty-dev.
Preparing to unpack .../103-golang-github-kr-pretty-dev_0.3.1-1_all.deb ...
Unpacking golang-github-kr-pretty-dev (0.3.1-1) ...
Selecting previously unselected package golang-gopkg-check.v1-dev.
Preparing to unpack .../104-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ...
Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Selecting previously unselected package golang-gopkg-mgo.v2-dev.
Preparing to unpack .../105-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ...
Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev.
Preparing to unpack .../106-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ...
Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-1.24-src (1.24.0-1) ...
Setting up libpipeline1:i386 (1.5.8-1) ...
Setting up golang-github-google-go-cmp-dev (0.6.0-1) ...
Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Setting up libicu72:i386 (72.1-6) ...
Setting up bsdextrautils (2.40.4-3) ...
Setting up golang-github-creack-pty-dev (1.1.21-1) ...
Setting up golang-github-stretchr-objx-dev (0.5.2-1) ...
Setting up libmagic-mgc (1:5.45-3+b1) ...
Setting up golang-github-pkg-errors-dev (0.9.1-3) ...
Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Setting up libarchive-zip-perl (1.68-1) ...
Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Setting up libdebhelper-perl (13.24.1) ...
Setting up golang-golang-x-sys-dev (0.22.0-1) ...
Setting up libmagic1t64:i386 (1:5.45-3+b1) ...
Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Setting up gettext-base (0.23.1-1) ...
Setting up m4 (1.4.19-5) ...
Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Setting up golang-golang-x-term-dev (0.22.0-1) ...
Setting up file (1:5.45-3+b1) ...
Setting up golang-github-beorn7-perks-dev (1.0.1-1) ...
Setting up libelf1t64:i386 (0.192-4) ...
Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Setting up libsasl2-modules-db:i386 (2.1.28+dfsg1-8+b1) ...
Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ...
Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Setting up autotools-dev (20220109.1) ...
Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Setting up golang-google-protobuf-dev (1.33.0-1) ...
Setting up golang-github-kr-text-dev (0.2.0-2) ...
Setting up golang-github-stretchr-testify-dev (1.9.0-1) ...
Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Setting up golang-github-yuin-goldmark-dev (1.7.4-1) ...
Setting up libunistring5:i386 (1.3-1) ...
Setting up libssl-dev:i386 (3.4.1-1) ...
Setting up golang-golang-x-sync-dev (0.7.0-1) ...
Setting up golang-golang-x-mod-dev (0.19.0-1) ...
Setting up autopoint (0.23.1-1) ...
Setting up golang-1.24-go (1.24.0-1) ...
Setting up golang-github-klauspost-compress-dev (1.17.11+ds1-1) ...
Setting up libsasl2-2:i386 (2.1.28+dfsg1-8+b1) ...
Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Setting up autoconf (2.72-3) ...
Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Setting up dwz (0.15-1+b1) ...
Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Setting up sensible-utils (0.0.24) ...
Setting up golang-golang-x-text-dev (0.22.0-1) ...
Setting up libuchardet0:i386 (0.0.8-1+b2) ...
Setting up golang-github-prometheus-procfs-dev (0.15.1-1) ...
Setting up golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ...
Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Setting up golang-golang-x-crypto-dev (1:0.25.0-1) ...
Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ...
Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Setting up golang-golang-x-oauth2-dev (0.15.0-1) ...
Setting up golang-src (2:1.24~2) ...
Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Setting up libxml2:i386 (2.12.7+dfsg+really2.9.14-0.2+b1) ...
Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Setting up automake (1:1.17-3) ...
update-alternatives: using /usr/bin/automake-1.17 to provide /usr/bin/automake (automake) in auto mode
Setting up libfile-stripnondeterminism-perl (1.14.1-2) ...
Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Setting up golang-github-go-kit-log-dev (0.2.1-1) ...
Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Setting up gettext (0.23.1-1) ...
Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Setting up libtool (2.5.4-3) ...
Setting up golang-go:i386 (2:1.24~2) ...
Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Setting up golang-github-json-iterator-go-dev (1.1.12-2) ...
Setting up golang-any:i386 (2:1.24~2) ...
Setting up intltool-debian (0.35.0+20060710.6) ...
Setting up dh-autoreconf (20) ...
Setting up libsasl2-dev (2.1.28+dfsg1-8+b1) ...
Setting up golang-golang-x-net-dev (1:0.27.0-1) ...
Setting up dh-strip-nondeterminism (1.14.1-2) ...
Setting up groff-base (1.23.0-7) ...
Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Setting up golang-github-fatih-color-dev (1.17.0-1) ...
Setting up po-debconf (1.0.21+nmu1) ...
Setting up golang-golang-x-tools-dev (1:0.23.0+ds-1) ...
Setting up man-db (2.13.0-1) ...
Not building database; man-db/auto-update is not 'true'.
Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Setting up golang-github-kr-pretty-dev (0.3.1-1) ...
Setting up debhelper (13.24.1) ...
Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Setting up dh-golang (1.63) ...
Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Setting up golang-github-prometheus-sigv4-dev (0.1.2-2) ...
Setting up golang-github-prometheus-common-dev (0.62.0-3) ...
Setting up golang-github-prometheus-client-golang-dev (1.21.0-1) ...
Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ...
Processing triggers for libc-bin (2.40-7) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Building tag database...
 -> Finished parsing the build-deps
I: Building the package
I: user script /srv/workspace/pbuilder/34756/tmp/hooks/A99_set_merged_usr starting
Not re-configuring usrmerge for trixie
I: user script /srv/workspace/pbuilder/34756/tmp/hooks/A99_set_merged_usr finished
hostname: Name or service not known
I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S  > ../golang-github-hashicorp-raft_1.5.0-1_source.changes
dpkg-buildpackage: info: source package golang-github-hashicorp-raft
dpkg-buildpackage: info: source version 1.5.0-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Daniel Swarbrick <dswarbrick@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture i386
 debian/rules clean
dh clean --buildsystem=golang
   dh_auto_clean -O--buildsystem=golang
   dh_autoreconf_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules binary
dh binary --buildsystem=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-i686-linux-gnu && go install -trimpath -v -p 21 github.com/hashicorp/raft github.com/hashicorp/raft/bench
internal/godebugs
internal/goos
internal/msan
internal/profilerecord
internal/unsafeheader
internal/asan
internal/coverage/rtcov
crypto/internal/fips140/alias
container/list
unicode/utf8
cmp
internal/goarch
internal/runtime/syscall
math/bits
internal/byteorder
unicode
sync/atomic
internal/cpu
internal/runtime/atomic
internal/goexperiment
crypto/internal/boring/sig
encoding
unicode/utf16
log/internal
internal/nettrace
vendor/golang.org/x/crypto/cryptobyte/asn1
vendor/golang.org/x/crypto/internal/alias
internal/itoa
internal/runtime/math
internal/abi
internal/runtime/sys
crypto/internal/fips140deps/byteorder
internal/chacha8rand
math
crypto/internal/fips140deps/cpu
internal/bytealg
internal/runtime/exithook
internal/stringslite
internal/race
internal/sync
internal/runtime/maps
runtime
crypto/internal/fips140/subtle
weak
iter
internal/reflectlite
sync
maps
slices
crypto/subtle
errors
sort
internal/oserror
github.com/hashicorp/golang-lru/simplelru
path
vendor/golang.org/x/net/dns/dnsmessage
math/rand/v2
strconv
internal/singleflight
internal/bisect
internal/testlog
unique
io
runtime/cgo
syscall
internal/godebug
hash
internal/saferio
crypto/internal/randutil
bytes
strings
hash/crc64
hash/crc32
crypto
net/netip
reflect
crypto/internal/fips140deps/godebug
math/rand
vendor/golang.org/x/text/transform
crypto/internal/fips140
crypto/internal/impl
net/http/internal/ascii
bufio
github.com/hashicorp/go-immutable-radix
html
regexp/syntax
crypto/tls/internal/fips140tls
crypto/internal/fips140/sha512
crypto/internal/fips140/sha256
crypto/internal/fips140/sha3
internal/syscall/execenv
internal/syscall/unix
time
crypto/internal/fips140/hmac
crypto/sha3
crypto/internal/fips140/check
crypto/internal/fips140hash
regexp
crypto/internal/fips140/hkdf
crypto/internal/fips140/tls12
crypto/internal/fips140/bigmod
crypto/internal/fips140/edwards25519/field
crypto/internal/fips140/aes
crypto/internal/fips140/nistec/fiat
crypto/internal/fips140/tls13
crypto/internal/fips140/edwards25519
context
io/fs
internal/poll
internal/filepathlite
internal/fmtsort
encoding/binary
os
vendor/golang.org/x/crypto/internal/poly1305
encoding/base64
golang.org/x/sys/unix
crypto/internal/fips140/nistec
encoding/pem
crypto/internal/sysrand
internal/sysinfo
io/ioutil
os/signal
path/filepath
fmt
net
crypto/internal/entropy
crypto/internal/fips140/drbg
crypto/internal/fips140/aes/gcm
crypto/internal/fips140only
crypto/internal/fips140/ed25519
crypto/internal/fips140/rsa
crypto/internal/fips140/mlkem
crypto/rc4
crypto/md5
crypto/cipher
log
encoding/hex
net/url
encoding/json
mime/quotedprintable
net/http/internal
flag
text/template/parse
mime
vendor/golang.org/x/net/http2/hpack
go/token
math/big
vendor/golang.org/x/text/unicode/norm
compress/flate
encoding/gob
runtime/debug
runtime/trace
crypto/internal/fips140/ecdh
crypto/internal/fips140/ecdsa
crypto/internal/boring
crypto/des
vendor/golang.org/x/crypto/chacha20
vendor/golang.org/x/text/unicode/bidi
crypto/aes
crypto/ecdh
crypto/sha512
crypto/hmac
crypto/sha1
crypto/sha256
compress/gzip
vendor/golang.org/x/crypto/chacha20poly1305
testing
text/template
github.com/mattn/go-isatty
github.com/mattn/go-colorable
vendor/golang.org/x/text/secure/bidirule
github.com/fatih/color
vendor/golang.org/x/net/idna
github.com/hashicorp/go-hclog
crypto/internal/boring/bbig
crypto/rand
crypto/dsa
crypto/elliptic
encoding/asn1
crypto/ed25519
crypto/internal/hpke
crypto/rsa
html/template
crypto/x509/pkix
vendor/golang.org/x/crypto/cryptobyte
crypto/ecdsa
vendor/golang.org/x/net/http/httpproxy
net/textproto
crypto/x509
vendor/golang.org/x/net/http/httpguts
mime/multipart
crypto/tls
net/http/httptrace
net/http
net/rpc
github.com/armon/go-metrics
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/raft
github.com/hashicorp/raft/bench
   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
dh_auto_test
	cd obj-i686-linux-gnu && go test -vet=off -v -p 21 github.com/hashicorp/raft github.com/hashicorp/raft/bench
=== RUN   TestCommitment_setVoters
--- PASS: TestCommitment_setVoters (0.00s)
=== RUN   TestCommitment_match_max
--- PASS: TestCommitment_match_max (0.00s)
=== RUN   TestCommitment_match_nonVoting
--- PASS: TestCommitment_match_nonVoting (0.00s)
=== RUN   TestCommitment_recalculate
--- PASS: TestCommitment_recalculate (0.00s)
=== RUN   TestCommitment_recalculate_startIndex
--- PASS: TestCommitment_recalculate_startIndex (0.00s)
=== RUN   TestCommitment_noVoterSanity
--- PASS: TestCommitment_noVoterSanity (0.00s)
=== RUN   TestCommitment_singleVoter
--- PASS: TestCommitment_singleVoter (0.00s)
=== RUN   TestConfiguration_Configuration_Clone
--- PASS: TestConfiguration_Configuration_Clone (0.00s)
=== RUN   TestConfiguration_configurations_Clone
--- PASS: TestConfiguration_configurations_Clone (0.00s)
=== RUN   TestConfiguration_hasVote
--- PASS: TestConfiguration_hasVote (0.00s)
=== RUN   TestConfiguration_checkConfiguration
--- PASS: TestConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_nextConfiguration_table
--- PASS: TestConfiguration_nextConfiguration_table (0.00s)
=== RUN   TestConfiguration_nextConfiguration_prevIndex
--- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
=== RUN   TestConfiguration_nextConfiguration_checkConfiguration
--- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_encodeDecodePeers
--- PASS: TestConfiguration_encodeDecodePeers (0.00s)
=== RUN   TestConfiguration_encodeDecodeConfiguration
--- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
=== RUN   TestDiscardSnapshotStoreImpl
--- PASS: TestDiscardSnapshotStoreImpl (0.00s)
=== RUN   TestDiscardSnapshotSinkImpl
--- PASS: TestDiscardSnapshotSinkImpl (0.00s)
=== RUN   TestFileSnapshotStoreImpl
--- PASS: TestFileSnapshotStoreImpl (0.00s)
=== RUN   TestFileSnapshotSinkImpl
--- PASS: TestFileSnapshotSinkImpl (0.00s)
=== RUN   TestFileSS_CreateSnapshotMissingParentDir
2026-03-29T01:07:36.214Z [INFO]  creating new snapshot: path=/tmp/raft20252914/raft1881879372/snapshots/3-10-1774746456214.tmp
--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s)
=== RUN   TestFileSS_CreateSnapshot
2026-03-29T01:07:36.218Z [INFO]  creating new snapshot: path=/tmp/raft2884396987/snapshots/3-10-1774746456218.tmp
2026-03-29T01:07:36.221Z [WARN]  found temporary snapshot: name=3-10-1774746456218.tmp
--- PASS: TestFileSS_CreateSnapshot (0.01s)
=== RUN   TestFileSS_CancelSnapshot
2026-03-29T01:07:36.228Z [INFO]  creating new snapshot: path=/tmp/raft3217474716/snapshots/3-10-1774746456228.tmp
--- PASS: TestFileSS_CancelSnapshot (0.00s)
=== RUN   TestFileSS_Retention
2026-03-29T01:07:36.233Z [INFO]  creating new snapshot: path=/tmp/raft2558403220/snapshots/3-10-1774746456233.tmp
2026-03-29T01:07:36.241Z [INFO]  creating new snapshot: path=/tmp/raft2558403220/snapshots/3-11-1774746456241.tmp
2026-03-29T01:07:36.250Z [INFO]  creating new snapshot: path=/tmp/raft2558403220/snapshots/3-12-1774746456250.tmp
2026-03-29T01:07:36.259Z [INFO]  reaping snapshot: path=/tmp/raft2558403220/snapshots/3-10-1774746456233
2026-03-29T01:07:36.259Z [INFO]  creating new snapshot: path=/tmp/raft2558403220/snapshots/3-13-1774746456259.tmp
2026-03-29T01:07:36.270Z [INFO]  reaping snapshot: path=/tmp/raft2558403220/snapshots/3-11-1774746456241
2026-03-29T01:07:36.270Z [INFO]  creating new snapshot: path=/tmp/raft2558403220/snapshots/3-14-1774746456270.tmp
2026-03-29T01:07:36.278Z [INFO]  reaping snapshot: path=/tmp/raft2558403220/snapshots/3-12-1774746456250
--- PASS: TestFileSS_Retention (0.05s)
=== RUN   TestFileSS_BadPerm
--- PASS: TestFileSS_BadPerm (0.00s)
=== RUN   TestFileSS_MissingParentDir
--- PASS: TestFileSS_MissingParentDir (0.00s)
=== RUN   TestFileSS_Ordering
2026-03-29T01:07:36.279Z [INFO]  creating new snapshot: path=/tmp/raft1753175872/snapshots/5-130350-1774746456279.tmp
2026-03-29T01:07:36.289Z [INFO]  creating new snapshot: path=/tmp/raft1753175872/snapshots/36-204917-1774746456289.tmp
--- PASS: TestFileSS_Ordering (0.02s)
=== RUN   TestDeferFutureSuccess
--- PASS: TestDeferFutureSuccess (0.00s)
=== RUN   TestDeferFutureError
--- PASS: TestDeferFutureError (0.00s)
=== RUN   TestDeferFutureConcurrent
--- PASS: TestDeferFutureConcurrent (0.00s)
=== RUN   TestInmemSnapshotStoreImpl
--- PASS: TestInmemSnapshotStoreImpl (0.00s)
=== RUN   TestInmemSnapshotSinkImpl
--- PASS: TestInmemSnapshotSinkImpl (0.00s)
=== RUN   TestInmemSS_CreateSnapshot
--- PASS: TestInmemSS_CreateSnapshot (0.00s)
=== RUN   TestInmemSS_OpenSnapshotTwice
--- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
=== RUN   TestInmemTransportImpl
--- PASS: TestInmemTransportImpl (0.00s)
=== RUN   TestInmemTransportWriteTimeout
--- PASS: TestInmemTransportWriteTimeout (0.01s)
=== RUN   TestRaft_Integ
--- SKIP: TestRaft_Integ (0.00s)
=== RUN   TestRaft_RestartFollower_LongInitialHeartbeat
--- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s)
=== RUN   TestLogCache
--- PASS: TestLogCache (0.00s)
=== RUN   TestLogCacheWithBackendStoreError
--- PASS: TestLogCacheWithBackendStoreError (0.00s)
=== RUN   TestOldestLog
=== RUN   TestOldestLog/empty_logs
=== RUN   TestOldestLog/simple_case
--- PASS: TestOldestLog (0.00s)
    --- PASS: TestOldestLog/empty_logs (0.00s)
    --- PASS: TestOldestLog/simple_case (0.00s)
=== RUN   TestEmitsLogStoreMetrics
--- PASS: TestEmitsLogStoreMetrics (0.01s)
=== RUN   TestNetworkTransport_CloseStreams
--- PASS: TestNetworkTransport_CloseStreams (0.00s)
=== RUN   TestNetworkTransport_StartStop
--- PASS: TestNetworkTransport_StartStop (0.00s)
=== RUN   TestNetworkTransport_Heartbeat_FastPath
--- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s)
=== RUN   TestNetworkTransport_AppendEntries
--- PASS: TestNetworkTransport_AppendEntries (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline
--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_CloseStreams
--- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.02s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0
2026-03-29T01:07:36.393Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2
2026-03-29T01:07:36.444Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10
2026-03-29T01:07:36.496Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.15s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s)
=== RUN   TestNetworkTransport_RequestVote
--- PASS: TestNetworkTransport_RequestVote (0.00s)
=== RUN   TestNetworkTransport_InstallSnapshot
--- PASS: TestNetworkTransport_InstallSnapshot (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode
--- PASS: TestNetworkTransport_EncodeDecode (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode_AddressProvider
--- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
=== RUN   TestNetworkTransport_PooledConn
--- PASS: TestNetworkTransport_PooledConn (0.00s)
=== RUN   TestNetworkTransport_ListenBackoff
    net_transport_test.go:839: countingWriter: 2026-03-29T01:07:36.502Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-03-29T01:07:36.507Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-03-29T01:07:36.518Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-03-29T01:07:36.538Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-03-29T01:07:36.579Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-03-29T01:07:36.659Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-03-29T01:07:36.820Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-03-29T01:07:37.140Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-03-29T01:07:37.781Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-03-29T01:07:38.784Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2026-03-29T01:07:39.786Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
--- PASS: TestNetworkTransport_ListenBackoff (4.00s)
=== RUN   TestPeersJSON_BadConfiguration
--- PASS: TestPeersJSON_BadConfiguration (0.00s)
=== RUN   TestPeersJSON_ReadPeersJSON
--- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
=== RUN   TestPeersJSON_ReadConfigJSON
--- PASS: TestPeersJSON_ReadConfigJSON (0.00s)
=== RUN   TestRaft_StartStop
2026-03-29T01:07:40.504Z [INFO]  server-e7692aad-c4f2-c2f0-2481-4a0cce1dd25f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7692aad-c4f2-c2f0-2481-4a0cce1dd25f Address:e7692aad-c4f2-c2f0-2481-4a0cce1dd25f}]"
--- PASS: TestRaft_StartStop (0.00s)
=== RUN   TestRaft_AfterShutdown
2026-03-29T01:07:40.505Z [INFO]  server-a45ba92e-99d1-be35-9a1f-75e2356ca07e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a45ba92e-99d1-be35-9a1f-75e2356ca07e Address:a45ba92e-99d1-be35-9a1f-75e2356ca07e}]"
--- PASS: TestRaft_AfterShutdown (0.00s)
=== RUN   TestRaft_LiveBootstrap
2026-03-29T01:07:40.505Z [INFO]  server-694f5af6-4503-7b5b-e9e7-3add97e6d411: initial configuration: index=0 servers=[]
2026-03-29T01:07:40.505Z [INFO]  server-2007752c-2e48-0912-dd56-ba29af7d124f: initial configuration: index=0 servers=[]
2026-03-29T01:07:40.505Z [INFO]  server-694f5af6-4503-7b5b-e9e7-3add97e6d411: entering follower state: follower="Node at 694f5af6-4503-7b5b-e9e7-3add97e6d411 [Follower]" leader-address= leader-id=
2026-03-29T01:07:40.505Z [INFO]  server-2007752c-2e48-0912-dd56-ba29af7d124f: entering follower state: follower="Node at 2007752c-2e48-0912-dd56-ba29af7d124f [Follower]" leader-address= leader-id=
2026-03-29T01:07:40.505Z [INFO]  server-4e8fca20-a0e6-1ab9-1a85-684dd6c4d18e: initial configuration: index=0 servers=[]
2026-03-29T01:07:40.506Z [INFO]  server-4e8fca20-a0e6-1ab9-1a85-684dd6c4d18e: entering follower state: follower="Node at 4e8fca20-a0e6-1ab9-1a85-684dd6c4d18e [Follower]" leader-address= leader-id=
2026-03-29T01:07:40.506Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:07:40.566Z [WARN]  server-2007752c-2e48-0912-dd56-ba29af7d124f: no known peers, aborting election
2026-03-29T01:07:40.572Z [WARN]  server-694f5af6-4503-7b5b-e9e7-3add97e6d411: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:40.572Z [INFO]  server-694f5af6-4503-7b5b-e9e7-3add97e6d411: entering candidate state: node="Node at 694f5af6-4503-7b5b-e9e7-3add97e6d411 [Candidate]" term=2
2026-03-29T01:07:40.572Z [INFO]  server-694f5af6-4503-7b5b-e9e7-3add97e6d411: election won: term=2 tally=2
2026-03-29T01:07:40.572Z [INFO]  server-694f5af6-4503-7b5b-e9e7-3add97e6d411: entering leader state: leader="Node at 694f5af6-4503-7b5b-e9e7-3add97e6d411 [Leader]"
2026-03-29T01:07:40.572Z [INFO]  server-694f5af6-4503-7b5b-e9e7-3add97e6d411: added peer, starting replication: peer=server-2007752c-2e48-0912-dd56-ba29af7d124f
2026-03-29T01:07:40.572Z [INFO]  server-694f5af6-4503-7b5b-e9e7-3add97e6d411: added peer, starting replication: peer=server-4e8fca20-a0e6-1ab9-1a85-684dd6c4d18e
2026-03-29T01:07:40.572Z [WARN]  server-2007752c-2e48-0912-dd56-ba29af7d124f: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2026-03-29T01:07:40.572Z [WARN]  server-4e8fca20-a0e6-1ab9-1a85-684dd6c4d18e: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2026-03-29T01:07:40.572Z [WARN]  server-694f5af6-4503-7b5b-e9e7-3add97e6d411: appendEntries rejected, sending older logs: peer="{Voter server-2007752c-2e48-0912-dd56-ba29af7d124f 2007752c-2e48-0912-dd56-ba29af7d124f}" next=1
2026-03-29T01:07:40.572Z [WARN]  server-694f5af6-4503-7b5b-e9e7-3add97e6d411: appendEntries rejected, sending older logs: peer="{Voter server-4e8fca20-a0e6-1ab9-1a85-684dd6c4d18e 4e8fca20-a0e6-1ab9-1a85-684dd6c4d18e}" next=1
2026-03-29T01:07:40.573Z [INFO]  server-694f5af6-4503-7b5b-e9e7-3add97e6d411: pipelining replication: peer="{Voter server-2007752c-2e48-0912-dd56-ba29af7d124f 2007752c-2e48-0912-dd56-ba29af7d124f}"
2026-03-29T01:07:40.573Z [INFO]  server-694f5af6-4503-7b5b-e9e7-3add97e6d411: pipelining replication: peer="{Voter server-4e8fca20-a0e6-1ab9-1a85-684dd6c4d18e 4e8fca20-a0e6-1ab9-1a85-684dd6c4d18e}"
2026-03-29T01:07:40.678Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:07:40.572806036 +0000 UTC m=+4.360948595 (2 nodes), 66.711771ms from start of poll, 66.969133ms from cluster start. Timeout at 2026-03-29 01:07:40.677812954 +0000 UTC m=+4.465955561, 105.006966ms after stability
2026-03-29T01:07:40.678Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:40.784Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:40.678880176 +0000 UTC m=+4.467022826 (1 nodes), 1.537µs from start of poll, 173.043364ms from cluster start. Timeout at 2026-03-29 01:07:40.783883988 +0000 UTC m=+4.572026524, 105.003698ms after stability
2026-03-29T01:07:40.795Z [INFO]  server-694f5af6-4503-7b5b-e9e7-3add97e6d411: aborting pipeline replication: peer="{Voter server-4e8fca20-a0e6-1ab9-1a85-684dd6c4d18e 4e8fca20-a0e6-1ab9-1a85-684dd6c4d18e}"
2026-03-29T01:07:40.795Z [INFO]  server-694f5af6-4503-7b5b-e9e7-3add97e6d411: aborting pipeline replication: peer="{Voter server-2007752c-2e48-0912-dd56-ba29af7d124f 2007752c-2e48-0912-dd56-ba29af7d124f}"
--- PASS: TestRaft_LiveBootstrap (0.29s)
=== RUN   TestRaft_LiveBootstrap_From_NonVoter
2026-03-29T01:07:40.796Z [INFO]  server-696a58ad-2059-c0e6-605c-60e2a52dc259: initial configuration: index=0 servers=[]
2026-03-29T01:07:40.796Z [INFO]  server-2774044e-acfb-4748-9f7a-1b2c5ce669d3: initial configuration: index=0 servers=[]
2026-03-29T01:07:40.796Z [INFO]  server-696a58ad-2059-c0e6-605c-60e2a52dc259: entering follower state: follower="Node at 696a58ad-2059-c0e6-605c-60e2a52dc259 [Follower]" leader-address= leader-id=
2026-03-29T01:07:40.796Z [INFO]  server-2774044e-acfb-4748-9f7a-1b2c5ce669d3: entering follower state: follower="Node at 2774044e-acfb-4748-9f7a-1b2c5ce669d3 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s)
=== RUN   TestRaft_RecoverCluster_NoState
2026-03-29T01:07:40.797Z [INFO]  server-dc189847-0b4a-87b9-5130-9c90e9b925af: initial configuration: index=0 servers=[]
--- PASS: TestRaft_RecoverCluster_NoState (0.00s)
=== RUN   TestRaft_RecoverCluster
=== RUN   TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs
2026-03-29T01:07:40.798Z [INFO]  server-69a5b43b-3fdc-e636-218c-7b7aa3e51277: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69a5b43b-3fdc-e636-218c-7b7aa3e51277 Address:69a5b43b-3fdc-e636-218c-7b7aa3e51277} {Suffrage:Voter ID:server-aa6631ee-464f-f2c6-dd97-60b469daf451 Address:aa6631ee-464f-f2c6-dd97-60b469daf451} {Suffrage:Voter ID:server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615 Address:c4478344-52d3-e4d6-ad79-a9a8b4bf6615}]"
2026-03-29T01:07:40.798Z [INFO]  server-69a5b43b-3fdc-e636-218c-7b7aa3e51277: entering follower state: follower="Node at 69a5b43b-3fdc-e636-218c-7b7aa3e51277 [Follower]" leader-address= leader-id=
2026-03-29T01:07:40.798Z [INFO]  server-aa6631ee-464f-f2c6-dd97-60b469daf451: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69a5b43b-3fdc-e636-218c-7b7aa3e51277 Address:69a5b43b-3fdc-e636-218c-7b7aa3e51277} {Suffrage:Voter ID:server-aa6631ee-464f-f2c6-dd97-60b469daf451 Address:aa6631ee-464f-f2c6-dd97-60b469daf451} {Suffrage:Voter ID:server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615 Address:c4478344-52d3-e4d6-ad79-a9a8b4bf6615}]"
2026-03-29T01:07:40.798Z [INFO]  server-aa6631ee-464f-f2c6-dd97-60b469daf451: entering follower state: follower="Node at aa6631ee-464f-f2c6-dd97-60b469daf451 [Follower]" leader-address= leader-id=
2026-03-29T01:07:40.798Z [INFO]  server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69a5b43b-3fdc-e636-218c-7b7aa3e51277 Address:69a5b43b-3fdc-e636-218c-7b7aa3e51277} {Suffrage:Voter ID:server-aa6631ee-464f-f2c6-dd97-60b469daf451 Address:aa6631ee-464f-f2c6-dd97-60b469daf451} {Suffrage:Voter ID:server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615 Address:c4478344-52d3-e4d6-ad79-a9a8b4bf6615}]"
2026-03-29T01:07:40.798Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:40.798Z [INFO]  server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615: entering follower state: follower="Node at c4478344-52d3-e4d6-ad79-a9a8b4bf6615 [Follower]" leader-address= leader-id=
2026-03-29T01:07:40.862Z [WARN]  server-69a5b43b-3fdc-e636-218c-7b7aa3e51277: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:40.862Z [INFO]  server-69a5b43b-3fdc-e636-218c-7b7aa3e51277: entering candidate state: node="Node at 69a5b43b-3fdc-e636-218c-7b7aa3e51277 [Candidate]" term=2
2026-03-29T01:07:40.862Z [INFO]  server-69a5b43b-3fdc-e636-218c-7b7aa3e51277: election won: term=2 tally=2
2026-03-29T01:07:40.862Z [INFO]  server-69a5b43b-3fdc-e636-218c-7b7aa3e51277: entering leader state: leader="Node at 69a5b43b-3fdc-e636-218c-7b7aa3e51277 [Leader]"
2026-03-29T01:07:40.862Z [INFO]  server-69a5b43b-3fdc-e636-218c-7b7aa3e51277: added peer, starting replication: peer=server-aa6631ee-464f-f2c6-dd97-60b469daf451
2026-03-29T01:07:40.862Z [INFO]  server-69a5b43b-3fdc-e636-218c-7b7aa3e51277: added peer, starting replication: peer=server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615
2026-03-29T01:07:40.862Z [INFO]  server-69a5b43b-3fdc-e636-218c-7b7aa3e51277: pipelining replication: peer="{Voter server-aa6631ee-464f-f2c6-dd97-60b469daf451 aa6631ee-464f-f2c6-dd97-60b469daf451}"
2026-03-29T01:07:40.862Z [INFO]  server-69a5b43b-3fdc-e636-218c-7b7aa3e51277: pipelining replication: peer="{Voter server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615 c4478344-52d3-e4d6-ad79-a9a8b4bf6615}"
2026-03-29T01:07:40.967Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:40.862490212 +0000 UTC m=+4.650632775 (1 nodes), 64.119483ms from start of poll, 64.483235ms from cluster start. Timeout at 2026-03-29 01:07:40.967493472 +0000 UTC m=+4.755636010, 105.003235ms after stability
2026-03-29T01:07:40.967Z [INFO]  server-69a5b43b-3fdc-e636-218c-7b7aa3e51277: aborting pipeline replication: peer="{Voter server-aa6631ee-464f-f2c6-dd97-60b469daf451 aa6631ee-464f-f2c6-dd97-60b469daf451}"
2026-03-29T01:07:40.967Z [INFO]  server-69a5b43b-3fdc-e636-218c-7b7aa3e51277: aborting pipeline replication: peer="{Voter server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615 c4478344-52d3-e4d6-ad79-a9a8b4bf6615}"
2026-03-29T01:07:40.967Z [INFO]  creating new snapshot: path=/tmp/raft3440810888/snapshots/2-2-1774746460967.tmp
2026-03-29T01:07:40.968Z [INFO]  server-69a5b43b-3fdc-e636-218c-7b7aa3e51277: starting restore from snapshot: id=2-2-1774746460967 last-index=2 last-term=2 size-in-bytes=1
2026-03-29T01:07:40.968Z [INFO]  server-69a5b43b-3fdc-e636-218c-7b7aa3e51277: snapshot restore progress: id=2-2-1774746460967 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2026-03-29T01:07:40.968Z [INFO]  server-69a5b43b-3fdc-e636-218c-7b7aa3e51277: restored from snapshot: id=2-2-1774746460967 last-index=2 last-term=2 size-in-bytes=1
2026-03-29T01:07:40.968Z [INFO]  server-69a5b43b-3fdc-e636-218c-7b7aa3e51277: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69a5b43b-3fdc-e636-218c-7b7aa3e51277 Address:69a5b43b-3fdc-e636-218c-7b7aa3e51277} {Suffrage:Voter ID:server-aa6631ee-464f-f2c6-dd97-60b469daf451 Address:aa6631ee-464f-f2c6-dd97-60b469daf451} {Suffrage:Voter ID:server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615 Address:c4478344-52d3-e4d6-ad79-a9a8b4bf6615}]"
2026-03-29T01:07:40.968Z [INFO]  server-69a5b43b-3fdc-e636-218c-7b7aa3e51277: entering follower state: follower="Node at 69a5b43b-3fdc-e636-218c-7b7aa3e51277 [Follower]" leader-address= leader-id=
2026-03-29T01:07:40.968Z [INFO]  creating new snapshot: path=/tmp/raft1476581610/snapshots/2-2-1774746460968.tmp
2026-03-29T01:07:40.969Z [INFO]  server-aa6631ee-464f-f2c6-dd97-60b469daf451: starting restore from snapshot: id=2-2-1774746460968 last-index=2 last-term=2 size-in-bytes=1
2026-03-29T01:07:40.969Z [INFO]  server-aa6631ee-464f-f2c6-dd97-60b469daf451: snapshot restore progress: id=2-2-1774746460968 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2026-03-29T01:07:40.969Z [INFO]  server-aa6631ee-464f-f2c6-dd97-60b469daf451: restored from snapshot: id=2-2-1774746460968 last-index=2 last-term=2 size-in-bytes=1
2026-03-29T01:07:40.969Z [INFO]  server-aa6631ee-464f-f2c6-dd97-60b469daf451: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69a5b43b-3fdc-e636-218c-7b7aa3e51277 Address:69a5b43b-3fdc-e636-218c-7b7aa3e51277} {Suffrage:Voter ID:server-aa6631ee-464f-f2c6-dd97-60b469daf451 Address:aa6631ee-464f-f2c6-dd97-60b469daf451} {Suffrage:Voter ID:server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615 Address:c4478344-52d3-e4d6-ad79-a9a8b4bf6615}]"
2026-03-29T01:07:40.969Z [INFO]  server-aa6631ee-464f-f2c6-dd97-60b469daf451: entering follower state: follower="Node at aa6631ee-464f-f2c6-dd97-60b469daf451 [Follower]" leader-address= leader-id=
2026-03-29T01:07:40.969Z [INFO]  creating new snapshot: path=/tmp/raft2397179166/snapshots/2-2-1774746460969.tmp
2026-03-29T01:07:40.969Z [INFO]  server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615: starting restore from snapshot: id=2-2-1774746460969 last-index=2 last-term=2 size-in-bytes=1
2026-03-29T01:07:40.970Z [INFO]  server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615: snapshot restore progress: id=2-2-1774746460969 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2026-03-29T01:07:40.970Z [INFO]  server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615: restored from snapshot: id=2-2-1774746460969 last-index=2 last-term=2 size-in-bytes=1
2026-03-29T01:07:40.970Z [INFO]  server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-69a5b43b-3fdc-e636-218c-7b7aa3e51277 Address:69a5b43b-3fdc-e636-218c-7b7aa3e51277} {Suffrage:Voter ID:server-aa6631ee-464f-f2c6-dd97-60b469daf451 Address:aa6631ee-464f-f2c6-dd97-60b469daf451} {Suffrage:Voter ID:server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615 Address:c4478344-52d3-e4d6-ad79-a9a8b4bf6615}]"
2026-03-29T01:07:40.970Z [INFO]  server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615: entering follower state: follower="Node at c4478344-52d3-e4d6-ad79-a9a8b4bf6615 [Follower]" leader-address= leader-id=
2026-03-29T01:07:41.032Z [WARN]  server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:41.032Z [INFO]  server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615: entering candidate state: node="Node at c4478344-52d3-e4d6-ad79-a9a8b4bf6615 [Candidate]" term=3
2026-03-29T01:07:41.032Z [INFO]  server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615: election won: term=3 tally=2
2026-03-29T01:07:41.032Z [INFO]  server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615: entering leader state: leader="Node at c4478344-52d3-e4d6-ad79-a9a8b4bf6615 [Leader]"
2026-03-29T01:07:41.032Z [INFO]  server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615: added peer, starting replication: peer=server-69a5b43b-3fdc-e636-218c-7b7aa3e51277
2026-03-29T01:07:41.032Z [INFO]  server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615: added peer, starting replication: peer=server-aa6631ee-464f-f2c6-dd97-60b469daf451
2026-03-29T01:07:41.032Z [INFO]  server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615: pipelining replication: peer="{Voter server-aa6631ee-464f-f2c6-dd97-60b469daf451 aa6631ee-464f-f2c6-dd97-60b469daf451}"
2026-03-29T01:07:41.032Z [INFO]  server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615: pipelining replication: peer="{Voter server-69a5b43b-3fdc-e636-218c-7b7aa3e51277 69a5b43b-3fdc-e636-218c-7b7aa3e51277}"
2026-03-29T01:07:41.285Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:41.391Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:41.285792529 +0000 UTC m=+5.073935080 (1 nodes), 31.593µs from start of poll, 487.78554ms from cluster start. Timeout at 2026-03-29 01:07:41.390794266 +0000 UTC m=+5.178936807, 105.001727ms after stability
2026-03-29T01:07:41.391Z [INFO]  server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615: aborting pipeline replication: peer="{Voter server-aa6631ee-464f-f2c6-dd97-60b469daf451 aa6631ee-464f-f2c6-dd97-60b469daf451}"
2026-03-29T01:07:41.391Z [INFO]  server-c4478344-52d3-e4d6-ad79-a9a8b4bf6615: aborting pipeline replication: peer="{Voter server-69a5b43b-3fdc-e636-218c-7b7aa3e51277 69a5b43b-3fdc-e636-218c-7b7aa3e51277}"
=== RUN   TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs
2026-03-29T01:07:41.394Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0 Address:fe0abbef-f50b-e9d1-4753-ab5d74b43fe0} {Suffrage:Voter ID:server-f38b491a-bc23-b76c-f042-95438a377ce2 Address:f38b491a-bc23-b76c-f042-95438a377ce2} {Suffrage:Voter ID:server-49df6148-dda3-7457-6034-6b9b6954df1d Address:49df6148-dda3-7457-6034-6b9b6954df1d}]"
2026-03-29T01:07:41.394Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: entering follower state: follower="Node at fe0abbef-f50b-e9d1-4753-ab5d74b43fe0 [Follower]" leader-address= leader-id=
2026-03-29T01:07:41.394Z [INFO]  server-f38b491a-bc23-b76c-f042-95438a377ce2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0 Address:fe0abbef-f50b-e9d1-4753-ab5d74b43fe0} {Suffrage:Voter ID:server-f38b491a-bc23-b76c-f042-95438a377ce2 Address:f38b491a-bc23-b76c-f042-95438a377ce2} {Suffrage:Voter ID:server-49df6148-dda3-7457-6034-6b9b6954df1d Address:49df6148-dda3-7457-6034-6b9b6954df1d}]"
2026-03-29T01:07:41.394Z [INFO]  server-f38b491a-bc23-b76c-f042-95438a377ce2: entering follower state: follower="Node at f38b491a-bc23-b76c-f042-95438a377ce2 [Follower]" leader-address= leader-id=
2026-03-29T01:07:41.394Z [INFO]  server-49df6148-dda3-7457-6034-6b9b6954df1d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0 Address:fe0abbef-f50b-e9d1-4753-ab5d74b43fe0} {Suffrage:Voter ID:server-f38b491a-bc23-b76c-f042-95438a377ce2 Address:f38b491a-bc23-b76c-f042-95438a377ce2} {Suffrage:Voter ID:server-49df6148-dda3-7457-6034-6b9b6954df1d Address:49df6148-dda3-7457-6034-6b9b6954df1d}]"
2026-03-29T01:07:41.394Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:41.394Z [INFO]  server-49df6148-dda3-7457-6034-6b9b6954df1d: entering follower state: follower="Node at 49df6148-dda3-7457-6034-6b9b6954df1d [Follower]" leader-address= leader-id=
2026-03-29T01:07:41.454Z [WARN]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:41.454Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: entering candidate state: node="Node at fe0abbef-f50b-e9d1-4753-ab5d74b43fe0 [Candidate]" term=2
2026-03-29T01:07:41.455Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: election won: term=2 tally=2
2026-03-29T01:07:41.455Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: entering leader state: leader="Node at fe0abbef-f50b-e9d1-4753-ab5d74b43fe0 [Leader]"
2026-03-29T01:07:41.455Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: added peer, starting replication: peer=server-f38b491a-bc23-b76c-f042-95438a377ce2
2026-03-29T01:07:41.455Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: added peer, starting replication: peer=server-49df6148-dda3-7457-6034-6b9b6954df1d
2026-03-29T01:07:41.455Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: pipelining replication: peer="{Voter server-49df6148-dda3-7457-6034-6b9b6954df1d 49df6148-dda3-7457-6034-6b9b6954df1d}"
2026-03-29T01:07:41.455Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: pipelining replication: peer="{Voter server-f38b491a-bc23-b76c-f042-95438a377ce2 f38b491a-bc23-b76c-f042-95438a377ce2}"
2026-03-29T01:07:41.560Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:41.455066177 +0000 UTC m=+5.243208733 (1 nodes), 60.73127ms from start of poll, 61.060012ms from cluster start. Timeout at 2026-03-29 01:07:41.560069155 +0000 UTC m=+5.348211692, 105.002959ms after stability
2026-03-29T01:07:41.560Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: aborting pipeline replication: peer="{Voter server-49df6148-dda3-7457-6034-6b9b6954df1d 49df6148-dda3-7457-6034-6b9b6954df1d}"
2026-03-29T01:07:41.560Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: aborting pipeline replication: peer="{Voter server-f38b491a-bc23-b76c-f042-95438a377ce2 f38b491a-bc23-b76c-f042-95438a377ce2}"
2026-03-29T01:07:41.560Z [INFO]  creating new snapshot: path=/tmp/raft758959579/snapshots/2-6-1774746461560.tmp
2026-03-29T01:07:41.561Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: starting restore from snapshot: id=2-6-1774746461560 last-index=6 last-term=2 size-in-bytes=25
2026-03-29T01:07:41.561Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: snapshot restore progress: id=2-6-1774746461560 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2026-03-29T01:07:41.561Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: restored from snapshot: id=2-6-1774746461560 last-index=6 last-term=2 size-in-bytes=25
2026-03-29T01:07:41.561Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0 Address:fe0abbef-f50b-e9d1-4753-ab5d74b43fe0} {Suffrage:Voter ID:server-f38b491a-bc23-b76c-f042-95438a377ce2 Address:f38b491a-bc23-b76c-f042-95438a377ce2} {Suffrage:Voter ID:server-49df6148-dda3-7457-6034-6b9b6954df1d Address:49df6148-dda3-7457-6034-6b9b6954df1d}]"
2026-03-29T01:07:41.561Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: entering follower state: follower="Node at fe0abbef-f50b-e9d1-4753-ab5d74b43fe0 [Follower]" leader-address= leader-id=
2026-03-29T01:07:41.561Z [INFO]  creating new snapshot: path=/tmp/raft191571554/snapshots/2-6-1774746461561.tmp
2026-03-29T01:07:41.562Z [INFO]  server-f38b491a-bc23-b76c-f042-95438a377ce2: starting restore from snapshot: id=2-6-1774746461561 last-index=6 last-term=2 size-in-bytes=25
2026-03-29T01:07:41.562Z [INFO]  server-f38b491a-bc23-b76c-f042-95438a377ce2: snapshot restore progress: id=2-6-1774746461561 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2026-03-29T01:07:41.562Z [INFO]  server-f38b491a-bc23-b76c-f042-95438a377ce2: restored from snapshot: id=2-6-1774746461561 last-index=6 last-term=2 size-in-bytes=25
2026-03-29T01:07:41.562Z [INFO]  server-f38b491a-bc23-b76c-f042-95438a377ce2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0 Address:fe0abbef-f50b-e9d1-4753-ab5d74b43fe0} {Suffrage:Voter ID:server-f38b491a-bc23-b76c-f042-95438a377ce2 Address:f38b491a-bc23-b76c-f042-95438a377ce2} {Suffrage:Voter ID:server-49df6148-dda3-7457-6034-6b9b6954df1d Address:49df6148-dda3-7457-6034-6b9b6954df1d}]"
2026-03-29T01:07:41.562Z [INFO]  server-f38b491a-bc23-b76c-f042-95438a377ce2: entering follower state: follower="Node at f38b491a-bc23-b76c-f042-95438a377ce2 [Follower]" leader-address= leader-id=
2026-03-29T01:07:41.562Z [INFO]  creating new snapshot: path=/tmp/raft952773468/snapshots/2-6-1774746461562.tmp
2026-03-29T01:07:41.562Z [INFO]  server-49df6148-dda3-7457-6034-6b9b6954df1d: starting restore from snapshot: id=2-6-1774746461562 last-index=6 last-term=2 size-in-bytes=25
2026-03-29T01:07:41.562Z [INFO]  server-49df6148-dda3-7457-6034-6b9b6954df1d: snapshot restore progress: id=2-6-1774746461562 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2026-03-29T01:07:41.562Z [INFO]  server-49df6148-dda3-7457-6034-6b9b6954df1d: restored from snapshot: id=2-6-1774746461562 last-index=6 last-term=2 size-in-bytes=25
2026-03-29T01:07:41.562Z [INFO]  server-49df6148-dda3-7457-6034-6b9b6954df1d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0 Address:fe0abbef-f50b-e9d1-4753-ab5d74b43fe0} {Suffrage:Voter ID:server-f38b491a-bc23-b76c-f042-95438a377ce2 Address:f38b491a-bc23-b76c-f042-95438a377ce2} {Suffrage:Voter ID:server-49df6148-dda3-7457-6034-6b9b6954df1d Address:49df6148-dda3-7457-6034-6b9b6954df1d}]"
2026-03-29T01:07:41.562Z [INFO]  server-49df6148-dda3-7457-6034-6b9b6954df1d: entering follower state: follower="Node at 49df6148-dda3-7457-6034-6b9b6954df1d [Follower]" leader-address= leader-id=
2026-03-29T01:07:41.614Z [WARN]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:41.614Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: entering candidate state: node="Node at fe0abbef-f50b-e9d1-4753-ab5d74b43fe0 [Candidate]" term=3
2026-03-29T01:07:41.614Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: election won: term=3 tally=2
2026-03-29T01:07:41.614Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: entering leader state: leader="Node at fe0abbef-f50b-e9d1-4753-ab5d74b43fe0 [Leader]"
2026-03-29T01:07:41.614Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: added peer, starting replication: peer=server-f38b491a-bc23-b76c-f042-95438a377ce2
2026-03-29T01:07:41.614Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: added peer, starting replication: peer=server-49df6148-dda3-7457-6034-6b9b6954df1d
2026-03-29T01:07:41.615Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: pipelining replication: peer="{Voter server-f38b491a-bc23-b76c-f042-95438a377ce2 f38b491a-bc23-b76c-f042-95438a377ce2}"
2026-03-29T01:07:41.615Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: pipelining replication: peer="{Voter server-49df6148-dda3-7457-6034-6b9b6954df1d 49df6148-dda3-7457-6034-6b9b6954df1d}"
2026-03-29T01:07:41.878Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:41.984Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:41.878743933 +0000 UTC m=+5.666886468 (1 nodes), 18.882µs from start of poll, 484.737747ms from cluster start. Timeout at 2026-03-29 01:07:41.983746323 +0000 UTC m=+5.771888862, 105.002394ms after stability
2026-03-29T01:07:41.984Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: aborting pipeline replication: peer="{Voter server-49df6148-dda3-7457-6034-6b9b6954df1d 49df6148-dda3-7457-6034-6b9b6954df1d}"
2026-03-29T01:07:41.984Z [INFO]  server-fe0abbef-f50b-e9d1-4753-ab5d74b43fe0: aborting pipeline replication: peer="{Voter server-f38b491a-bc23-b76c-f042-95438a377ce2 f38b491a-bc23-b76c-f042-95438a377ce2}"
=== RUN   TestRaft_RecoverCluster/snapshot,_with_trailing_logs
2026-03-29T01:07:41.985Z [INFO]  server-4f2a6fb2-16c2-f718-3550-c9414296f3e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f2a6fb2-16c2-f718-3550-c9414296f3e3 Address:4f2a6fb2-16c2-f718-3550-c9414296f3e3} {Suffrage:Voter ID:server-09776095-c0cd-7b43-bf59-121b93bc726a Address:09776095-c0cd-7b43-bf59-121b93bc726a} {Suffrage:Voter ID:server-37b458cd-3b55-94ca-c96d-9566a9f451fa Address:37b458cd-3b55-94ca-c96d-9566a9f451fa}]"
2026-03-29T01:07:41.985Z [INFO]  server-4f2a6fb2-16c2-f718-3550-c9414296f3e3: entering follower state: follower="Node at 4f2a6fb2-16c2-f718-3550-c9414296f3e3 [Follower]" leader-address= leader-id=
2026-03-29T01:07:41.985Z [INFO]  server-09776095-c0cd-7b43-bf59-121b93bc726a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f2a6fb2-16c2-f718-3550-c9414296f3e3 Address:4f2a6fb2-16c2-f718-3550-c9414296f3e3} {Suffrage:Voter ID:server-09776095-c0cd-7b43-bf59-121b93bc726a Address:09776095-c0cd-7b43-bf59-121b93bc726a} {Suffrage:Voter ID:server-37b458cd-3b55-94ca-c96d-9566a9f451fa Address:37b458cd-3b55-94ca-c96d-9566a9f451fa}]"
2026-03-29T01:07:41.985Z [INFO]  server-09776095-c0cd-7b43-bf59-121b93bc726a: entering follower state: follower="Node at 09776095-c0cd-7b43-bf59-121b93bc726a [Follower]" leader-address= leader-id=
2026-03-29T01:07:41.985Z [INFO]  server-37b458cd-3b55-94ca-c96d-9566a9f451fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f2a6fb2-16c2-f718-3550-c9414296f3e3 Address:4f2a6fb2-16c2-f718-3550-c9414296f3e3} {Suffrage:Voter ID:server-09776095-c0cd-7b43-bf59-121b93bc726a Address:09776095-c0cd-7b43-bf59-121b93bc726a} {Suffrage:Voter ID:server-37b458cd-3b55-94ca-c96d-9566a9f451fa Address:37b458cd-3b55-94ca-c96d-9566a9f451fa}]"
2026-03-29T01:07:41.985Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:41.985Z [INFO]  server-37b458cd-3b55-94ca-c96d-9566a9f451fa: entering follower state: follower="Node at 37b458cd-3b55-94ca-c96d-9566a9f451fa [Follower]" leader-address= leader-id=
2026-03-29T01:07:42.050Z [WARN]  server-09776095-c0cd-7b43-bf59-121b93bc726a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:42.050Z [INFO]  server-09776095-c0cd-7b43-bf59-121b93bc726a: entering candidate state: node="Node at 09776095-c0cd-7b43-bf59-121b93bc726a [Candidate]" term=2
2026-03-29T01:07:42.050Z [INFO]  server-09776095-c0cd-7b43-bf59-121b93bc726a: election won: term=2 tally=2
2026-03-29T01:07:42.050Z [INFO]  server-09776095-c0cd-7b43-bf59-121b93bc726a: entering leader state: leader="Node at 09776095-c0cd-7b43-bf59-121b93bc726a [Leader]"
2026-03-29T01:07:42.050Z [INFO]  server-09776095-c0cd-7b43-bf59-121b93bc726a: added peer, starting replication: peer=server-4f2a6fb2-16c2-f718-3550-c9414296f3e3
2026-03-29T01:07:42.050Z [INFO]  server-09776095-c0cd-7b43-bf59-121b93bc726a: added peer, starting replication: peer=server-37b458cd-3b55-94ca-c96d-9566a9f451fa
2026-03-29T01:07:42.050Z [INFO]  server-09776095-c0cd-7b43-bf59-121b93bc726a: pipelining replication: peer="{Voter server-4f2a6fb2-16c2-f718-3550-c9414296f3e3 4f2a6fb2-16c2-f718-3550-c9414296f3e3}"
2026-03-29T01:07:42.050Z [INFO]  server-09776095-c0cd-7b43-bf59-121b93bc726a: pipelining replication: peer="{Voter server-37b458cd-3b55-94ca-c96d-9566a9f451fa 37b458cd-3b55-94ca-c96d-9566a9f451fa}"
2026-03-29T01:07:42.155Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:42.050739369 +0000 UTC m=+5.838881972 (1 nodes), 65.059192ms from start of poll, 65.300618ms from cluster start. Timeout at 2026-03-29 01:07:42.155742684 +0000 UTC m=+5.943885218, 105.003246ms after stability
2026-03-29T01:07:42.159Z [INFO]  server-09776095-c0cd-7b43-bf59-121b93bc726a: aborting pipeline replication: peer="{Voter server-37b458cd-3b55-94ca-c96d-9566a9f451fa 37b458cd-3b55-94ca-c96d-9566a9f451fa}"
2026-03-29T01:07:42.159Z [INFO]  server-09776095-c0cd-7b43-bf59-121b93bc726a: aborting pipeline replication: peer="{Voter server-4f2a6fb2-16c2-f718-3550-c9414296f3e3 4f2a6fb2-16c2-f718-3550-c9414296f3e3}"
2026-03-29T01:07:42.159Z [INFO]  creating new snapshot: path=/tmp/raft2929442365/snapshots/2-27-1774746462159.tmp
2026-03-29T01:07:42.160Z [INFO]  server-4f2a6fb2-16c2-f718-3550-c9414296f3e3: starting restore from snapshot: id=2-27-1774746462159 last-index=27 last-term=2 size-in-bytes=168
2026-03-29T01:07:42.160Z [INFO]  server-4f2a6fb2-16c2-f718-3550-c9414296f3e3: snapshot restore progress: id=2-27-1774746462159 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2026-03-29T01:07:42.160Z [INFO]  server-4f2a6fb2-16c2-f718-3550-c9414296f3e3: restored from snapshot: id=2-27-1774746462159 last-index=27 last-term=2 size-in-bytes=168
2026-03-29T01:07:42.160Z [INFO]  server-4f2a6fb2-16c2-f718-3550-c9414296f3e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f2a6fb2-16c2-f718-3550-c9414296f3e3 Address:4f2a6fb2-16c2-f718-3550-c9414296f3e3} {Suffrage:Voter ID:server-09776095-c0cd-7b43-bf59-121b93bc726a Address:09776095-c0cd-7b43-bf59-121b93bc726a} {Suffrage:Voter ID:server-37b458cd-3b55-94ca-c96d-9566a9f451fa Address:37b458cd-3b55-94ca-c96d-9566a9f451fa}]"
2026-03-29T01:07:42.160Z [INFO]  server-4f2a6fb2-16c2-f718-3550-c9414296f3e3: entering follower state: follower="Node at 4f2a6fb2-16c2-f718-3550-c9414296f3e3 [Follower]" leader-address= leader-id=
2026-03-29T01:07:42.160Z [INFO]  creating new snapshot: path=/tmp/raft166683543/snapshots/2-27-1774746462160.tmp
2026-03-29T01:07:42.161Z [INFO]  server-09776095-c0cd-7b43-bf59-121b93bc726a: starting restore from snapshot: id=2-27-1774746462160 last-index=27 last-term=2 size-in-bytes=168
2026-03-29T01:07:42.161Z [INFO]  server-09776095-c0cd-7b43-bf59-121b93bc726a: snapshot restore progress: id=2-27-1774746462160 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2026-03-29T01:07:42.161Z [INFO]  server-09776095-c0cd-7b43-bf59-121b93bc726a: restored from snapshot: id=2-27-1774746462160 last-index=27 last-term=2 size-in-bytes=168
2026-03-29T01:07:42.161Z [INFO]  server-09776095-c0cd-7b43-bf59-121b93bc726a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f2a6fb2-16c2-f718-3550-c9414296f3e3 Address:4f2a6fb2-16c2-f718-3550-c9414296f3e3} {Suffrage:Voter ID:server-09776095-c0cd-7b43-bf59-121b93bc726a Address:09776095-c0cd-7b43-bf59-121b93bc726a} {Suffrage:Voter ID:server-37b458cd-3b55-94ca-c96d-9566a9f451fa Address:37b458cd-3b55-94ca-c96d-9566a9f451fa}]"
2026-03-29T01:07:42.161Z [INFO]  server-09776095-c0cd-7b43-bf59-121b93bc726a: entering follower state: follower="Node at 09776095-c0cd-7b43-bf59-121b93bc726a [Follower]" leader-address= leader-id=
2026-03-29T01:07:42.161Z [INFO]  creating new snapshot: path=/tmp/raft1525074200/snapshots/2-27-1774746462161.tmp
2026-03-29T01:07:42.161Z [INFO]  server-37b458cd-3b55-94ca-c96d-9566a9f451fa: starting restore from snapshot: id=2-27-1774746462161 last-index=27 last-term=2 size-in-bytes=168
2026-03-29T01:07:42.161Z [INFO]  server-37b458cd-3b55-94ca-c96d-9566a9f451fa: snapshot restore progress: id=2-27-1774746462161 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2026-03-29T01:07:42.162Z [INFO]  server-37b458cd-3b55-94ca-c96d-9566a9f451fa: restored from snapshot: id=2-27-1774746462161 last-index=27 last-term=2 size-in-bytes=168
2026-03-29T01:07:42.162Z [INFO]  server-37b458cd-3b55-94ca-c96d-9566a9f451fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f2a6fb2-16c2-f718-3550-c9414296f3e3 Address:4f2a6fb2-16c2-f718-3550-c9414296f3e3} {Suffrage:Voter ID:server-09776095-c0cd-7b43-bf59-121b93bc726a Address:09776095-c0cd-7b43-bf59-121b93bc726a} {Suffrage:Voter ID:server-37b458cd-3b55-94ca-c96d-9566a9f451fa Address:37b458cd-3b55-94ca-c96d-9566a9f451fa}]"
2026-03-29T01:07:42.162Z [INFO]  server-37b458cd-3b55-94ca-c96d-9566a9f451fa: entering follower state: follower="Node at 37b458cd-3b55-94ca-c96d-9566a9f451fa [Follower]" leader-address= leader-id=
2026-03-29T01:07:42.229Z [WARN]  server-37b458cd-3b55-94ca-c96d-9566a9f451fa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:42.229Z [INFO]  server-37b458cd-3b55-94ca-c96d-9566a9f451fa: entering candidate state: node="Node at 37b458cd-3b55-94ca-c96d-9566a9f451fa [Candidate]" term=3
2026-03-29T01:07:42.229Z [INFO]  server-37b458cd-3b55-94ca-c96d-9566a9f451fa: election won: term=3 tally=2
2026-03-29T01:07:42.229Z [INFO]  server-37b458cd-3b55-94ca-c96d-9566a9f451fa: entering leader state: leader="Node at 37b458cd-3b55-94ca-c96d-9566a9f451fa [Leader]"
2026-03-29T01:07:42.229Z [INFO]  server-37b458cd-3b55-94ca-c96d-9566a9f451fa: added peer, starting replication: peer=server-4f2a6fb2-16c2-f718-3550-c9414296f3e3
2026-03-29T01:07:42.229Z [INFO]  server-37b458cd-3b55-94ca-c96d-9566a9f451fa: added peer, starting replication: peer=server-09776095-c0cd-7b43-bf59-121b93bc726a
2026-03-29T01:07:42.229Z [INFO]  server-37b458cd-3b55-94ca-c96d-9566a9f451fa: pipelining replication: peer="{Voter server-09776095-c0cd-7b43-bf59-121b93bc726a 09776095-c0cd-7b43-bf59-121b93bc726a}"
2026-03-29T01:07:42.229Z [INFO]  server-37b458cd-3b55-94ca-c96d-9566a9f451fa: pipelining replication: peer="{Voter server-4f2a6fb2-16c2-f718-3550-c9414296f3e3 4f2a6fb2-16c2-f718-3550-c9414296f3e3}"
2026-03-29T01:07:42.477Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:42.584Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:42.478096966 +0000 UTC m=+6.266239687 (1 nodes), 59.705µs from start of poll, 492.658333ms from cluster start. Timeout at 2026-03-29 01:07:42.583107283 +0000 UTC m=+6.371249887, 105.0102ms after stability
2026-03-29T01:07:42.584Z [INFO]  server-37b458cd-3b55-94ca-c96d-9566a9f451fa: aborting pipeline replication: peer="{Voter server-09776095-c0cd-7b43-bf59-121b93bc726a 09776095-c0cd-7b43-bf59-121b93bc726a}"
2026-03-29T01:07:42.584Z [INFO]  server-37b458cd-3b55-94ca-c96d-9566a9f451fa: aborting pipeline replication: peer="{Voter server-4f2a6fb2-16c2-f718-3550-c9414296f3e3 4f2a6fb2-16c2-f718-3550-c9414296f3e3}"
2026-03-29T01:07:43.084Z [ERROR] server-37b458cd-3b55-94ca-c96d-9566a9f451fa: failed to heartbeat to: peer=09776095-c0cd-7b43-bf59-121b93bc726a backoff time=10ms error="command timed out"
--- PASS: TestRaft_RecoverCluster (2.29s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.60s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.59s)
    --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (1.10s)
=== RUN   TestRaft_HasExistingState
    raft_test.go:258: Skipping unreliable test
--- SKIP: TestRaft_HasExistingState (0.00s)
=== RUN   TestRaft_SingleNode
2026-03-29T01:07:43.085Z [INFO]  server-15bd6d49-67e8-636c-ade7-807ec1824550: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15bd6d49-67e8-636c-ade7-807ec1824550 Address:15bd6d49-67e8-636c-ade7-807ec1824550}]"
2026-03-29T01:07:43.085Z [INFO]  server-15bd6d49-67e8-636c-ade7-807ec1824550: entering follower state: follower="Node at 15bd6d49-67e8-636c-ade7-807ec1824550 [Follower]" leader-address= leader-id=
2026-03-29T01:07:43.155Z [WARN]  server-15bd6d49-67e8-636c-ade7-807ec1824550: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:43.155Z [INFO]  server-15bd6d49-67e8-636c-ade7-807ec1824550: entering candidate state: node="Node at 15bd6d49-67e8-636c-ade7-807ec1824550 [Candidate]" term=2
2026-03-29T01:07:43.155Z [INFO]  server-15bd6d49-67e8-636c-ade7-807ec1824550: election won: term=2 tally=1
2026-03-29T01:07:43.155Z [INFO]  server-15bd6d49-67e8-636c-ade7-807ec1824550: entering leader state: leader="Node at 15bd6d49-67e8-636c-ade7-807ec1824550 [Leader]"
--- PASS: TestRaft_SingleNode (0.07s)
=== RUN   TestRaft_TripleNode
2026-03-29T01:07:43.155Z [INFO]  server-c7746d97-851b-c3a5-fbb0-1805d03326c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7746d97-851b-c3a5-fbb0-1805d03326c0 Address:c7746d97-851b-c3a5-fbb0-1805d03326c0} {Suffrage:Voter ID:server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655 Address:f2fc4d6f-c0bd-9913-2167-2c1c97a04655} {Suffrage:Voter ID:server-11b76163-06b0-dd17-131a-e1072fa4c92f Address:11b76163-06b0-dd17-131a-e1072fa4c92f}]"
2026-03-29T01:07:43.155Z [INFO]  server-c7746d97-851b-c3a5-fbb0-1805d03326c0: entering follower state: follower="Node at c7746d97-851b-c3a5-fbb0-1805d03326c0 [Follower]" leader-address= leader-id=
2026-03-29T01:07:43.156Z [INFO]  server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7746d97-851b-c3a5-fbb0-1805d03326c0 Address:c7746d97-851b-c3a5-fbb0-1805d03326c0} {Suffrage:Voter ID:server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655 Address:f2fc4d6f-c0bd-9913-2167-2c1c97a04655} {Suffrage:Voter ID:server-11b76163-06b0-dd17-131a-e1072fa4c92f Address:11b76163-06b0-dd17-131a-e1072fa4c92f}]"
2026-03-29T01:07:43.156Z [INFO]  server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655: entering follower state: follower="Node at f2fc4d6f-c0bd-9913-2167-2c1c97a04655 [Follower]" leader-address= leader-id=
2026-03-29T01:07:43.156Z [INFO]  server-11b76163-06b0-dd17-131a-e1072fa4c92f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7746d97-851b-c3a5-fbb0-1805d03326c0 Address:c7746d97-851b-c3a5-fbb0-1805d03326c0} {Suffrage:Voter ID:server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655 Address:f2fc4d6f-c0bd-9913-2167-2c1c97a04655} {Suffrage:Voter ID:server-11b76163-06b0-dd17-131a-e1072fa4c92f Address:11b76163-06b0-dd17-131a-e1072fa4c92f}]"
2026-03-29T01:07:43.156Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:07:43.156Z [INFO]  server-11b76163-06b0-dd17-131a-e1072fa4c92f: entering follower state: follower="Node at 11b76163-06b0-dd17-131a-e1072fa4c92f [Follower]" leader-address= leader-id=
2026-03-29T01:07:43.220Z [WARN]  server-c7746d97-851b-c3a5-fbb0-1805d03326c0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:43.220Z [WARN]  server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:43.220Z [INFO]  server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655: entering candidate state: node="Node at f2fc4d6f-c0bd-9913-2167-2c1c97a04655 [Candidate]" term=2
2026-03-29T01:07:43.220Z [INFO]  server-c7746d97-851b-c3a5-fbb0-1805d03326c0: entering candidate state: node="Node at c7746d97-851b-c3a5-fbb0-1805d03326c0 [Candidate]" term=2
2026-03-29T01:07:43.220Z [INFO]  server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655: election won: term=2 tally=2
2026-03-29T01:07:43.220Z [INFO]  server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655: entering leader state: leader="Node at f2fc4d6f-c0bd-9913-2167-2c1c97a04655 [Leader]"
2026-03-29T01:07:43.220Z [INFO]  server-11b76163-06b0-dd17-131a-e1072fa4c92f: duplicate requestVote for same term: term=2
2026-03-29T01:07:43.220Z [INFO]  server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655: added peer, starting replication: peer=server-c7746d97-851b-c3a5-fbb0-1805d03326c0
2026-03-29T01:07:43.220Z [INFO]  server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655: added peer, starting replication: peer=server-11b76163-06b0-dd17-131a-e1072fa4c92f
2026-03-29T01:07:43.220Z [WARN]  server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655: rejecting vote request since we have a leader: from=c7746d97-851b-c3a5-fbb0-1805d03326c0 leader=f2fc4d6f-c0bd-9913-2167-2c1c97a04655 leader-id=server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655
2026-03-29T01:07:43.220Z [INFO]  server-c7746d97-851b-c3a5-fbb0-1805d03326c0: duplicate requestVote for same term: term=2
2026-03-29T01:07:43.220Z [INFO]  server-c7746d97-851b-c3a5-fbb0-1805d03326c0: entering follower state: follower="Node at c7746d97-851b-c3a5-fbb0-1805d03326c0 [Follower]" leader-address=f2fc4d6f-c0bd-9913-2167-2c1c97a04655 leader-id=server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655
2026-03-29T01:07:43.220Z [INFO]  server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655: pipelining replication: peer="{Voter server-11b76163-06b0-dd17-131a-e1072fa4c92f 11b76163-06b0-dd17-131a-e1072fa4c92f}"
2026-03-29T01:07:43.220Z [INFO]  server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655: pipelining replication: peer="{Voter server-c7746d97-851b-c3a5-fbb0-1805d03326c0 c7746d97-851b-c3a5-fbb0-1805d03326c0}"
2026-03-29T01:07:43.325Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:07:43.220263711 +0000 UTC m=+7.008406251 (2 nodes), 64.170194ms from start of poll, 64.389633ms from cluster start. Timeout at 2026-03-29 01:07:43.325266865 +0000 UTC m=+7.113409404, 105.003153ms after stability
2026-03-29T01:07:43.325Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:43.431Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:43.325712833 +0000 UTC m=+7.113855368 (1 nodes), 606ns from start of poll, 169.83875ms from cluster start. Timeout at 2026-03-29 01:07:43.430731607 +0000 UTC m=+7.218874144, 105.018776ms after stability
2026-03-29T01:07:43.442Z [INFO]  server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655: aborting pipeline replication: peer="{Voter server-11b76163-06b0-dd17-131a-e1072fa4c92f 11b76163-06b0-dd17-131a-e1072fa4c92f}"
2026-03-29T01:07:43.442Z [INFO]  server-f2fc4d6f-c0bd-9913-2167-2c1c97a04655: aborting pipeline replication: peer="{Voter server-c7746d97-851b-c3a5-fbb0-1805d03326c0 c7746d97-851b-c3a5-fbb0-1805d03326c0}"
--- PASS: TestRaft_TripleNode (0.29s)
=== RUN   TestRaft_LeaderFail
2026-03-29T01:07:43.443Z [INFO]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a Address:6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a} {Suffrage:Voter ID:server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c Address:d35bc808-cc05-d524-2cb0-9c2ffca4f18c} {Suffrage:Voter ID:server-03929e1d-b95f-3dd0-8581-cc287026d3b4 Address:03929e1d-b95f-3dd0-8581-cc287026d3b4}]"
2026-03-29T01:07:43.443Z [INFO]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: entering follower state: follower="Node at 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a [Follower]" leader-address= leader-id=
2026-03-29T01:07:43.443Z [INFO]  server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a Address:6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a} {Suffrage:Voter ID:server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c Address:d35bc808-cc05-d524-2cb0-9c2ffca4f18c} {Suffrage:Voter ID:server-03929e1d-b95f-3dd0-8581-cc287026d3b4 Address:03929e1d-b95f-3dd0-8581-cc287026d3b4}]"
2026-03-29T01:07:43.443Z [INFO]  server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: entering follower state: follower="Node at d35bc808-cc05-d524-2cb0-9c2ffca4f18c [Follower]" leader-address= leader-id=
2026-03-29T01:07:43.444Z [INFO]  server-03929e1d-b95f-3dd0-8581-cc287026d3b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a Address:6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a} {Suffrage:Voter ID:server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c Address:d35bc808-cc05-d524-2cb0-9c2ffca4f18c} {Suffrage:Voter ID:server-03929e1d-b95f-3dd0-8581-cc287026d3b4 Address:03929e1d-b95f-3dd0-8581-cc287026d3b4}]"
2026-03-29T01:07:43.444Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:07:43.444Z [INFO]  server-03929e1d-b95f-3dd0-8581-cc287026d3b4: entering follower state: follower="Node at 03929e1d-b95f-3dd0-8581-cc287026d3b4 [Follower]" leader-address= leader-id=
2026-03-29T01:07:43.494Z [WARN]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:43.494Z [INFO]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: entering candidate state: node="Node at 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a [Candidate]" term=2
2026-03-29T01:07:43.494Z [INFO]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: election won: term=2 tally=2
2026-03-29T01:07:43.494Z [INFO]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: entering leader state: leader="Node at 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a [Leader]"
2026-03-29T01:07:43.494Z [INFO]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: added peer, starting replication: peer=server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c
2026-03-29T01:07:43.494Z [INFO]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: added peer, starting replication: peer=server-03929e1d-b95f-3dd0-8581-cc287026d3b4
2026-03-29T01:07:43.494Z [INFO]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: pipelining replication: peer="{Voter server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c d35bc808-cc05-d524-2cb0-9c2ffca4f18c}"
2026-03-29T01:07:43.494Z [INFO]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: pipelining replication: peer="{Voter server-03929e1d-b95f-3dd0-8581-cc287026d3b4 03929e1d-b95f-3dd0-8581-cc287026d3b4}"
2026-03-29T01:07:43.600Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:07:43.494479336 +0000 UTC m=+7.282621871 (2 nodes), 50.437125ms from start of poll, 50.758555ms from cluster start. Timeout at 2026-03-29 01:07:43.599482041 +0000 UTC m=+7.387624581, 105.00271ms after stability
2026-03-29T01:07:43.600Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:43.705Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:43.60010247 +0000 UTC m=+7.388245005 (1 nodes), 474ns from start of poll, 156.381689ms from cluster start. Timeout at 2026-03-29 01:07:43.705106845 +0000 UTC m=+7.493249378, 105.004373ms after stability
    raft_test.go:378: [INFO] Disconnecting Node at 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a [Leader]
2026-03-29T01:07:43.721Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to heartbeat to: peer=03929e1d-b95f-3dd0-8581-cc287026d3b4 backoff time=10ms error="failed to connect to peer: 03929e1d-b95f-3dd0-8581-cc287026d3b4"
2026-03-29T01:07:43.721Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:43.722Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to pipeline appendEntries: peer="{Voter server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c d35bc808-cc05-d524-2cb0-9c2ffca4f18c}" error="append pipeline closed"
2026-03-29T01:07:43.722Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to heartbeat to: peer=d35bc808-cc05-d524-2cb0-9c2ffca4f18c backoff time=10ms error="failed to connect to peer: d35bc808-cc05-d524-2cb0-9c2ffca4f18c"
2026-03-29T01:07:43.722Z [INFO]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: aborting pipeline replication: peer="{Voter server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c d35bc808-cc05-d524-2cb0-9c2ffca4f18c}"
2026-03-29T01:07:43.722Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to pipeline appendEntries: peer="{Voter server-03929e1d-b95f-3dd0-8581-cc287026d3b4 03929e1d-b95f-3dd0-8581-cc287026d3b4}" error="append pipeline closed"
2026-03-29T01:07:43.722Z [INFO]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: aborting pipeline replication: peer="{Voter server-03929e1d-b95f-3dd0-8581-cc287026d3b4 03929e1d-b95f-3dd0-8581-cc287026d3b4}"
2026-03-29T01:07:43.732Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to appendEntries to: peer="{Voter server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c d35bc808-cc05-d524-2cb0-9c2ffca4f18c}" error="failed to connect to peer: d35bc808-cc05-d524-2cb0-9c2ffca4f18c"
2026-03-29T01:07:43.732Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to appendEntries to: peer="{Voter server-03929e1d-b95f-3dd0-8581-cc287026d3b4 03929e1d-b95f-3dd0-8581-cc287026d3b4}" error="failed to connect to peer: 03929e1d-b95f-3dd0-8581-cc287026d3b4"
2026-03-29T01:07:43.739Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to heartbeat to: peer=03929e1d-b95f-3dd0-8581-cc287026d3b4 backoff time=10ms error="failed to connect to peer: 03929e1d-b95f-3dd0-8581-cc287026d3b4"
2026-03-29T01:07:43.743Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to heartbeat to: peer=d35bc808-cc05-d524-2cb0-9c2ffca4f18c backoff time=10ms error="failed to connect to peer: d35bc808-cc05-d524-2cb0-9c2ffca4f18c"
2026-03-29T01:07:43.749Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to appendEntries to: peer="{Voter server-03929e1d-b95f-3dd0-8581-cc287026d3b4 03929e1d-b95f-3dd0-8581-cc287026d3b4}" error="failed to connect to peer: 03929e1d-b95f-3dd0-8581-cc287026d3b4"
2026-03-29T01:07:43.750Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to appendEntries to: peer="{Voter server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c d35bc808-cc05-d524-2cb0-9c2ffca4f18c}" error="failed to connect to peer: d35bc808-cc05-d524-2cb0-9c2ffca4f18c"
2026-03-29T01:07:43.758Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to heartbeat to: peer=03929e1d-b95f-3dd0-8581-cc287026d3b4 backoff time=10ms error="failed to connect to peer: 03929e1d-b95f-3dd0-8581-cc287026d3b4"
2026-03-29T01:07:43.760Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to heartbeat to: peer=d35bc808-cc05-d524-2cb0-9c2ffca4f18c backoff time=10ms error="failed to connect to peer: d35bc808-cc05-d524-2cb0-9c2ffca4f18c"
2026-03-29T01:07:43.766Z [WARN]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to contact: server-id=server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c time=50.862034ms
2026-03-29T01:07:43.766Z [WARN]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to contact: server-id=server-03929e1d-b95f-3dd0-8581-cc287026d3b4 time=50.899552ms
2026-03-29T01:07:43.766Z [WARN]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to contact quorum of nodes, stepping down
2026-03-29T01:07:43.766Z [INFO]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: entering follower state: follower="Node at 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a [Follower]" leader-address= leader-id=
2026-03-29T01:07:43.769Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to appendEntries to: peer="{Voter server-03929e1d-b95f-3dd0-8581-cc287026d3b4 03929e1d-b95f-3dd0-8581-cc287026d3b4}" error="failed to connect to peer: 03929e1d-b95f-3dd0-8581-cc287026d3b4"
2026-03-29T01:07:43.770Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to appendEntries to: peer="{Voter server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c d35bc808-cc05-d524-2cb0-9c2ffca4f18c}" error="failed to connect to peer: d35bc808-cc05-d524-2cb0-9c2ffca4f18c"
2026-03-29T01:07:43.783Z [WARN]  server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: heartbeat timeout reached, starting election: last-leader-addr=6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a last-leader-id=server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a
2026-03-29T01:07:43.783Z [INFO]  server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: entering candidate state: node="Node at d35bc808-cc05-d524-2cb0-9c2ffca4f18c [Candidate]" term=3
2026-03-29T01:07:43.783Z [WARN]  server-03929e1d-b95f-3dd0-8581-cc287026d3b4: rejecting vote request since we have a leader: from=d35bc808-cc05-d524-2cb0-9c2ffca4f18c leader=6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a leader-id=server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a
2026-03-29T01:07:43.783Z [ERROR] server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: failed to make requestVote RPC: target="{Voter server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a}" error="failed to connect to peer: 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a" term=3
2026-03-29T01:07:43.788Z [WARN]  server-03929e1d-b95f-3dd0-8581-cc287026d3b4: heartbeat timeout reached, starting election: last-leader-addr=6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a last-leader-id=server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a
2026-03-29T01:07:43.788Z [INFO]  server-03929e1d-b95f-3dd0-8581-cc287026d3b4: entering candidate state: node="Node at 03929e1d-b95f-3dd0-8581-cc287026d3b4 [Candidate]" term=3
2026-03-29T01:07:43.788Z [INFO]  server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: duplicate requestVote for same term: term=3
2026-03-29T01:07:43.788Z [ERROR] server-03929e1d-b95f-3dd0-8581-cc287026d3b4: failed to make requestVote RPC: target="{Voter server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a}" error="failed to connect to peer: 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a" term=3
2026-03-29T01:07:43.849Z [WARN]  server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: Election timeout reached, restarting election
2026-03-29T01:07:43.849Z [INFO]  server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: entering candidate state: node="Node at d35bc808-cc05-d524-2cb0-9c2ffca4f18c [Candidate]" term=4
2026-03-29T01:07:43.849Z [ERROR] server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: failed to make requestVote RPC: target="{Voter server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a}" error="failed to connect to peer: 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a" term=4
2026-03-29T01:07:43.849Z [INFO]  server-03929e1d-b95f-3dd0-8581-cc287026d3b4: entering follower state: follower="Node at 03929e1d-b95f-3dd0-8581-cc287026d3b4 [Follower]" leader-address= leader-id=
2026-03-29T01:07:43.849Z [INFO]  server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: election won: term=4 tally=2
2026-03-29T01:07:43.850Z [INFO]  server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: entering leader state: leader="Node at d35bc808-cc05-d524-2cb0-9c2ffca4f18c [Leader]"
2026-03-29T01:07:43.850Z [INFO]  server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: added peer, starting replication: peer=server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a
2026-03-29T01:07:43.850Z [INFO]  server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: added peer, starting replication: peer=server-03929e1d-b95f-3dd0-8581-cc287026d3b4
2026-03-29T01:07:43.850Z [INFO]  server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: pipelining replication: peer="{Voter server-03929e1d-b95f-3dd0-8581-cc287026d3b4 03929e1d-b95f-3dd0-8581-cc287026d3b4}"
2026-03-29T01:07:43.850Z [ERROR] server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: failed to appendEntries to: peer="{Voter server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a}" error="failed to connect to peer: 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a"
2026-03-29T01:07:43.856Z [ERROR] server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: failed to heartbeat to: peer=6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a backoff time=10ms error="failed to connect to peer: 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a"
2026-03-29T01:07:43.856Z [WARN]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:43.856Z [INFO]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: entering candidate state: node="Node at 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a [Candidate]" term=3
2026-03-29T01:07:43.856Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to make requestVote RPC: target="{Voter server-03929e1d-b95f-3dd0-8581-cc287026d3b4 03929e1d-b95f-3dd0-8581-cc287026d3b4}" error="failed to connect to peer: 03929e1d-b95f-3dd0-8581-cc287026d3b4" term=3
2026-03-29T01:07:43.856Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to make requestVote RPC: target="{Voter server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c d35bc808-cc05-d524-2cb0-9c2ffca4f18c}" error="failed to connect to peer: d35bc808-cc05-d524-2cb0-9c2ffca4f18c" term=3
2026-03-29T01:07:43.868Z [ERROR] server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: failed to appendEntries to: peer="{Voter server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a}" error="failed to connect to peer: 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a"
2026-03-29T01:07:43.876Z [ERROR] server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: failed to heartbeat to: peer=6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a backoff time=10ms error="failed to connect to peer: 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a"
2026-03-29T01:07:43.886Z [ERROR] server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: failed to appendEntries to: peer="{Voter server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a}" error="failed to connect to peer: 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a"
2026-03-29T01:07:43.896Z [ERROR] server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: failed to heartbeat to: peer=6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a backoff time=10ms error="failed to connect to peer: 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a"
2026-03-29T01:07:43.900Z [WARN]  server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: failed to contact: server-id=server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a time=50.563073ms
2026-03-29T01:07:43.913Z [ERROR] server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: failed to heartbeat to: peer=6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a backoff time=20ms error="failed to connect to peer: 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a"
2026-03-29T01:07:43.916Z [ERROR] server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: failed to appendEntries to: peer="{Voter server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a}" error="failed to connect to peer: 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a"
2026-03-29T01:07:43.938Z [WARN]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: Election timeout reached, restarting election
2026-03-29T01:07:43.938Z [INFO]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: entering candidate state: node="Node at 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a [Candidate]" term=4
2026-03-29T01:07:43.938Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to make requestVote RPC: target="{Voter server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c d35bc808-cc05-d524-2cb0-9c2ffca4f18c}" error="failed to connect to peer: d35bc808-cc05-d524-2cb0-9c2ffca4f18c" term=4
2026-03-29T01:07:43.938Z [ERROR] server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: failed to make requestVote RPC: target="{Voter server-03929e1d-b95f-3dd0-8581-cc287026d3b4 03929e1d-b95f-3dd0-8581-cc287026d3b4}" error="failed to connect to peer: 03929e1d-b95f-3dd0-8581-cc287026d3b4" term=4
2026-03-29T01:07:43.939Z [ERROR] server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: failed to heartbeat to: peer=6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a backoff time=25ms error="failed to connect to peer: 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a"
2026-03-29T01:07:43.949Z [WARN]  server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: failed to contact: server-id=server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a time=99.148822ms
2026-03-29T01:07:43.961Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:43.856297029 +0000 UTC m=+7.644439563 (1 nodes), 134.573966ms from start of poll, 412.576247ms from cluster start. Timeout at 2026-03-29 01:07:43.961366068 +0000 UTC m=+7.749508615, 105.069052ms after stability
    raft_test.go:413: [INFO] Reconnecting Node at 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a [Candidate]
2026-03-29T01:07:43.965Z [INFO]  server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a: entering follower state: follower="Node at 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a [Follower]" leader-address=d35bc808-cc05-d524-2cb0-9c2ffca4f18c leader-id=server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c
2026-03-29T01:07:43.965Z [INFO]  server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: pipelining replication: peer="{Voter server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a}"
2026-03-29T01:07:43.971Z [INFO]  server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: aborting pipeline replication: peer="{Voter server-03929e1d-b95f-3dd0-8581-cc287026d3b4 03929e1d-b95f-3dd0-8581-cc287026d3b4}"
2026-03-29T01:07:43.971Z [INFO]  server-d35bc808-cc05-d524-2cb0-9c2ffca4f18c: aborting pipeline replication: peer="{Voter server-6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a 6e9e7d7d-8fb0-2924-ea13-6395eb9ea54a}"
--- PASS: TestRaft_LeaderFail (0.53s)
=== RUN   TestRaft_BehindFollower
2026-03-29T01:07:43.972Z [INFO]  server-60e28479-8095-8dda-2e56-9046f5969e22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60e28479-8095-8dda-2e56-9046f5969e22 Address:60e28479-8095-8dda-2e56-9046f5969e22} {Suffrage:Voter ID:server-fea9f58c-9d42-67fc-dc0f-f94e07d7e821 Address:fea9f58c-9d42-67fc-dc0f-f94e07d7e821} {Suffrage:Voter ID:server-39e9e119-5b84-ede0-1439-519461eadb3d Address:39e9e119-5b84-ede0-1439-519461eadb3d}]"
2026-03-29T01:07:43.972Z [INFO]  server-60e28479-8095-8dda-2e56-9046f5969e22: entering follower state: follower="Node at 60e28479-8095-8dda-2e56-9046f5969e22 [Follower]" leader-address= leader-id=
2026-03-29T01:07:43.972Z [INFO]  server-fea9f58c-9d42-67fc-dc0f-f94e07d7e821: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60e28479-8095-8dda-2e56-9046f5969e22 Address:60e28479-8095-8dda-2e56-9046f5969e22} {Suffrage:Voter ID:server-fea9f58c-9d42-67fc-dc0f-f94e07d7e821 Address:fea9f58c-9d42-67fc-dc0f-f94e07d7e821} {Suffrage:Voter ID:server-39e9e119-5b84-ede0-1439-519461eadb3d Address:39e9e119-5b84-ede0-1439-519461eadb3d}]"
2026-03-29T01:07:43.972Z [INFO]  server-fea9f58c-9d42-67fc-dc0f-f94e07d7e821: entering follower state: follower="Node at fea9f58c-9d42-67fc-dc0f-f94e07d7e821 [Follower]" leader-address= leader-id=
2026-03-29T01:07:43.972Z [INFO]  server-39e9e119-5b84-ede0-1439-519461eadb3d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-60e28479-8095-8dda-2e56-9046f5969e22 Address:60e28479-8095-8dda-2e56-9046f5969e22} {Suffrage:Voter ID:server-fea9f58c-9d42-67fc-dc0f-f94e07d7e821 Address:fea9f58c-9d42-67fc-dc0f-f94e07d7e821} {Suffrage:Voter ID:server-39e9e119-5b84-ede0-1439-519461eadb3d Address:39e9e119-5b84-ede0-1439-519461eadb3d}]"
2026-03-29T01:07:43.972Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:43.972Z [INFO]  server-39e9e119-5b84-ede0-1439-519461eadb3d: entering follower state: follower="Node at 39e9e119-5b84-ede0-1439-519461eadb3d [Follower]" leader-address= leader-id=
2026-03-29T01:07:44.042Z [WARN]  server-60e28479-8095-8dda-2e56-9046f5969e22: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:44.043Z [INFO]  server-60e28479-8095-8dda-2e56-9046f5969e22: entering candidate state: node="Node at 60e28479-8095-8dda-2e56-9046f5969e22 [Candidate]" term=2
2026-03-29T01:07:44.043Z [INFO]  server-60e28479-8095-8dda-2e56-9046f5969e22: election won: term=2 tally=2
2026-03-29T01:07:44.043Z [INFO]  server-60e28479-8095-8dda-2e56-9046f5969e22: entering leader state: leader="Node at 60e28479-8095-8dda-2e56-9046f5969e22 [Leader]"
2026-03-29T01:07:44.043Z [INFO]  server-60e28479-8095-8dda-2e56-9046f5969e22: added peer, starting replication: peer=server-fea9f58c-9d42-67fc-dc0f-f94e07d7e821
2026-03-29T01:07:44.043Z [INFO]  server-60e28479-8095-8dda-2e56-9046f5969e22: added peer, starting replication: peer=server-39e9e119-5b84-ede0-1439-519461eadb3d
2026-03-29T01:07:44.043Z [INFO]  server-60e28479-8095-8dda-2e56-9046f5969e22: pipelining replication: peer="{Voter server-39e9e119-5b84-ede0-1439-519461eadb3d 39e9e119-5b84-ede0-1439-519461eadb3d}"
2026-03-29T01:07:44.043Z [INFO]  server-60e28479-8095-8dda-2e56-9046f5969e22: pipelining replication: peer="{Voter server-fea9f58c-9d42-67fc-dc0f-f94e07d7e821 fea9f58c-9d42-67fc-dc0f-f94e07d7e821}"
2026-03-29T01:07:44.149Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:44.043153623 +0000 UTC m=+7.831296172 (1 nodes), 70.289517ms from start of poll, 70.605518ms from cluster start. Timeout at 2026-03-29 01:07:44.148155881 +0000 UTC m=+7.936298411, 105.002239ms after stability
2026-03-29T01:07:44.149Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:07:44.254Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:07:44.149150384 +0000 UTC m=+7.937292930 (2 nodes), 853ns from start of poll, 176.602276ms from cluster start. Timeout at 2026-03-29 01:07:44.254152816 +0000 UTC m=+8.042295355, 105.002425ms after stability
2026-03-29T01:07:44.254Z [ERROR] server-60e28479-8095-8dda-2e56-9046f5969e22: failed to pipeline appendEntries: peer="{Voter server-fea9f58c-9d42-67fc-dc0f-f94e07d7e821 fea9f58c-9d42-67fc-dc0f-f94e07d7e821}" error="append pipeline closed"
2026-03-29T01:07:44.254Z [INFO]  server-60e28479-8095-8dda-2e56-9046f5969e22: aborting pipeline replication: peer="{Voter server-fea9f58c-9d42-67fc-dc0f-f94e07d7e821 fea9f58c-9d42-67fc-dc0f-f94e07d7e821}"
2026-03-29T01:07:44.254Z [ERROR] server-60e28479-8095-8dda-2e56-9046f5969e22: failed to appendEntries to: peer="{Voter server-fea9f58c-9d42-67fc-dc0f-f94e07d7e821 fea9f58c-9d42-67fc-dc0f-f94e07d7e821}" error="failed to connect to peer: fea9f58c-9d42-67fc-dc0f-f94e07d7e821"
    raft_test.go:459: [INFO] Finished apply without behind follower
2026-03-29T01:07:44.265Z [INFO]  server-60e28479-8095-8dda-2e56-9046f5969e22: pipelining replication: peer="{Voter server-fea9f58c-9d42-67fc-dc0f-f94e07d7e821 fea9f58c-9d42-67fc-dc0f-f94e07d7e821}"
2026-03-29T01:07:44.279Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:44.385Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:44.27964963 +0000 UTC m=+8.067792247 (1 nodes), 1.36µs from start of poll, 307.101593ms from cluster start. Timeout at 2026-03-29 01:07:44.384659068 +0000 UTC m=+8.172801671, 105.009424ms after stability
2026-03-29T01:07:44.385Z [INFO]  server-60e28479-8095-8dda-2e56-9046f5969e22: aborting pipeline replication: peer="{Voter server-39e9e119-5b84-ede0-1439-519461eadb3d 39e9e119-5b84-ede0-1439-519461eadb3d}"
2026-03-29T01:07:44.385Z [INFO]  server-60e28479-8095-8dda-2e56-9046f5969e22: aborting pipeline replication: peer="{Voter server-fea9f58c-9d42-67fc-dc0f-f94e07d7e821 fea9f58c-9d42-67fc-dc0f-f94e07d7e821}"
--- PASS: TestRaft_BehindFollower (0.41s)
=== RUN   TestRaft_ApplyNonLeader
2026-03-29T01:07:44.386Z [INFO]  server-107c35cb-c275-01dd-5729-c64e7391c02d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-107c35cb-c275-01dd-5729-c64e7391c02d Address:107c35cb-c275-01dd-5729-c64e7391c02d} {Suffrage:Voter ID:server-ad15836a-661c-3de7-3bd0-2b06a08ae487 Address:ad15836a-661c-3de7-3bd0-2b06a08ae487} {Suffrage:Voter ID:server-bc0a4052-9fcd-f4f1-6ad4-3e0835c93958 Address:bc0a4052-9fcd-f4f1-6ad4-3e0835c93958}]"
2026-03-29T01:07:44.386Z [INFO]  server-107c35cb-c275-01dd-5729-c64e7391c02d: entering follower state: follower="Node at 107c35cb-c275-01dd-5729-c64e7391c02d [Follower]" leader-address= leader-id=
2026-03-29T01:07:44.386Z [INFO]  server-ad15836a-661c-3de7-3bd0-2b06a08ae487: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-107c35cb-c275-01dd-5729-c64e7391c02d Address:107c35cb-c275-01dd-5729-c64e7391c02d} {Suffrage:Voter ID:server-ad15836a-661c-3de7-3bd0-2b06a08ae487 Address:ad15836a-661c-3de7-3bd0-2b06a08ae487} {Suffrage:Voter ID:server-bc0a4052-9fcd-f4f1-6ad4-3e0835c93958 Address:bc0a4052-9fcd-f4f1-6ad4-3e0835c93958}]"
2026-03-29T01:07:44.386Z [INFO]  server-ad15836a-661c-3de7-3bd0-2b06a08ae487: entering follower state: follower="Node at ad15836a-661c-3de7-3bd0-2b06a08ae487 [Follower]" leader-address= leader-id=
2026-03-29T01:07:44.386Z [INFO]  server-bc0a4052-9fcd-f4f1-6ad4-3e0835c93958: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-107c35cb-c275-01dd-5729-c64e7391c02d Address:107c35cb-c275-01dd-5729-c64e7391c02d} {Suffrage:Voter ID:server-ad15836a-661c-3de7-3bd0-2b06a08ae487 Address:ad15836a-661c-3de7-3bd0-2b06a08ae487} {Suffrage:Voter ID:server-bc0a4052-9fcd-f4f1-6ad4-3e0835c93958 Address:bc0a4052-9fcd-f4f1-6ad4-3e0835c93958}]"
2026-03-29T01:07:44.386Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:44.386Z [INFO]  server-bc0a4052-9fcd-f4f1-6ad4-3e0835c93958: entering follower state: follower="Node at bc0a4052-9fcd-f4f1-6ad4-3e0835c93958 [Follower]" leader-address= leader-id=
2026-03-29T01:07:44.459Z [WARN]  server-ad15836a-661c-3de7-3bd0-2b06a08ae487: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:44.459Z [INFO]  server-ad15836a-661c-3de7-3bd0-2b06a08ae487: entering candidate state: node="Node at ad15836a-661c-3de7-3bd0-2b06a08ae487 [Candidate]" term=2
2026-03-29T01:07:44.459Z [INFO]  server-ad15836a-661c-3de7-3bd0-2b06a08ae487: election won: term=2 tally=2
2026-03-29T01:07:44.459Z [INFO]  server-ad15836a-661c-3de7-3bd0-2b06a08ae487: entering leader state: leader="Node at ad15836a-661c-3de7-3bd0-2b06a08ae487 [Leader]"
2026-03-29T01:07:44.460Z [INFO]  server-ad15836a-661c-3de7-3bd0-2b06a08ae487: added peer, starting replication: peer=server-107c35cb-c275-01dd-5729-c64e7391c02d
2026-03-29T01:07:44.460Z [INFO]  server-ad15836a-661c-3de7-3bd0-2b06a08ae487: added peer, starting replication: peer=server-bc0a4052-9fcd-f4f1-6ad4-3e0835c93958
2026-03-29T01:07:44.460Z [INFO]  server-ad15836a-661c-3de7-3bd0-2b06a08ae487: pipelining replication: peer="{Voter server-bc0a4052-9fcd-f4f1-6ad4-3e0835c93958 bc0a4052-9fcd-f4f1-6ad4-3e0835c93958}"
2026-03-29T01:07:44.460Z [INFO]  server-ad15836a-661c-3de7-3bd0-2b06a08ae487: pipelining replication: peer="{Voter server-107c35cb-c275-01dd-5729-c64e7391c02d 107c35cb-c275-01dd-5729-c64e7391c02d}"
2026-03-29T01:07:44.566Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:44.460088544 +0000 UTC m=+8.248231111 (1 nodes), 73.770122ms from start of poll, 74.09821ms from cluster start. Timeout at 2026-03-29 01:07:44.565095272 +0000 UTC m=+8.353237882, 105.006771ms after stability
2026-03-29T01:07:44.566Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:07:44.671Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:07:44.566210067 +0000 UTC m=+8.354352664 (2 nodes), 1.477µs from start of poll, 180.219763ms from cluster start. Timeout at 2026-03-29 01:07:44.671214626 +0000 UTC m=+8.459357165, 105.004501ms after stability
2026-03-29T01:07:44.671Z [INFO]  server-ad15836a-661c-3de7-3bd0-2b06a08ae487: aborting pipeline replication: peer="{Voter server-107c35cb-c275-01dd-5729-c64e7391c02d 107c35cb-c275-01dd-5729-c64e7391c02d}"
2026-03-29T01:07:44.671Z [INFO]  server-ad15836a-661c-3de7-3bd0-2b06a08ae487: aborting pipeline replication: peer="{Voter server-bc0a4052-9fcd-f4f1-6ad4-3e0835c93958 bc0a4052-9fcd-f4f1-6ad4-3e0835c93958}"
--- PASS: TestRaft_ApplyNonLeader (0.29s)
=== RUN   TestRaft_ApplyConcurrent
2026-03-29T01:07:44.672Z [INFO]  server-68dc8a54-1797-fe36-3bb1-6f0ca8c51bf7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68dc8a54-1797-fe36-3bb1-6f0ca8c51bf7 Address:68dc8a54-1797-fe36-3bb1-6f0ca8c51bf7} {Suffrage:Voter ID:server-95d45d33-c493-bc50-936a-3a280e63e5a9 Address:95d45d33-c493-bc50-936a-3a280e63e5a9} {Suffrage:Voter ID:server-74eacdb5-2742-5f21-129e-21626e35de28 Address:74eacdb5-2742-5f21-129e-21626e35de28}]"
2026-03-29T01:07:44.672Z [INFO]  server-68dc8a54-1797-fe36-3bb1-6f0ca8c51bf7: entering follower state: follower="Node at 68dc8a54-1797-fe36-3bb1-6f0ca8c51bf7 [Follower]" leader-address= leader-id=
2026-03-29T01:07:44.672Z [INFO]  server-95d45d33-c493-bc50-936a-3a280e63e5a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68dc8a54-1797-fe36-3bb1-6f0ca8c51bf7 Address:68dc8a54-1797-fe36-3bb1-6f0ca8c51bf7} {Suffrage:Voter ID:server-95d45d33-c493-bc50-936a-3a280e63e5a9 Address:95d45d33-c493-bc50-936a-3a280e63e5a9} {Suffrage:Voter ID:server-74eacdb5-2742-5f21-129e-21626e35de28 Address:74eacdb5-2742-5f21-129e-21626e35de28}]"
2026-03-29T01:07:44.672Z [INFO]  server-95d45d33-c493-bc50-936a-3a280e63e5a9: entering follower state: follower="Node at 95d45d33-c493-bc50-936a-3a280e63e5a9 [Follower]" leader-address= leader-id=
2026-03-29T01:07:44.672Z [INFO]  server-74eacdb5-2742-5f21-129e-21626e35de28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68dc8a54-1797-fe36-3bb1-6f0ca8c51bf7 Address:68dc8a54-1797-fe36-3bb1-6f0ca8c51bf7} {Suffrage:Voter ID:server-95d45d33-c493-bc50-936a-3a280e63e5a9 Address:95d45d33-c493-bc50-936a-3a280e63e5a9} {Suffrage:Voter ID:server-74eacdb5-2742-5f21-129e-21626e35de28 Address:74eacdb5-2742-5f21-129e-21626e35de28}]"
2026-03-29T01:07:44.672Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:44.672Z [INFO]  server-74eacdb5-2742-5f21-129e-21626e35de28: entering follower state: follower="Node at 74eacdb5-2742-5f21-129e-21626e35de28 [Follower]" leader-address= leader-id=
2026-03-29T01:07:44.779Z [WARN]  server-74eacdb5-2742-5f21-129e-21626e35de28: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:44.779Z [INFO]  server-74eacdb5-2742-5f21-129e-21626e35de28: entering candidate state: node="Node at 74eacdb5-2742-5f21-129e-21626e35de28 [Candidate]" term=2
2026-03-29T01:07:44.779Z [INFO]  server-74eacdb5-2742-5f21-129e-21626e35de28: election won: term=2 tally=2
2026-03-29T01:07:44.779Z [INFO]  server-74eacdb5-2742-5f21-129e-21626e35de28: entering leader state: leader="Node at 74eacdb5-2742-5f21-129e-21626e35de28 [Leader]"
2026-03-29T01:07:44.779Z [INFO]  server-74eacdb5-2742-5f21-129e-21626e35de28: added peer, starting replication: peer=server-68dc8a54-1797-fe36-3bb1-6f0ca8c51bf7
2026-03-29T01:07:44.779Z [INFO]  server-74eacdb5-2742-5f21-129e-21626e35de28: added peer, starting replication: peer=server-95d45d33-c493-bc50-936a-3a280e63e5a9
2026-03-29T01:07:44.779Z [INFO]  server-74eacdb5-2742-5f21-129e-21626e35de28: pipelining replication: peer="{Voter server-68dc8a54-1797-fe36-3bb1-6f0ca8c51bf7 68dc8a54-1797-fe36-3bb1-6f0ca8c51bf7}"
2026-03-29T01:07:44.779Z [INFO]  server-74eacdb5-2742-5f21-129e-21626e35de28: pipelining replication: peer="{Voter server-95d45d33-c493-bc50-936a-3a280e63e5a9 95d45d33-c493-bc50-936a-3a280e63e5a9}"
2026-03-29T01:07:44.985Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:44.779751082 +0000 UTC m=+8.567893622 (1 nodes), 106.797337ms from start of poll, 107.173485ms from cluster start. Timeout at 2026-03-29 01:07:44.984755865 +0000 UTC m=+8.772898394, 205.004772ms after stability
2026-03-29T01:07:44.997Z [INFO]  server-74eacdb5-2742-5f21-129e-21626e35de28: aborting pipeline replication: peer="{Voter server-95d45d33-c493-bc50-936a-3a280e63e5a9 95d45d33-c493-bc50-936a-3a280e63e5a9}"
2026-03-29T01:07:44.997Z [INFO]  server-74eacdb5-2742-5f21-129e-21626e35de28: aborting pipeline replication: peer="{Voter server-68dc8a54-1797-fe36-3bb1-6f0ca8c51bf7 68dc8a54-1797-fe36-3bb1-6f0ca8c51bf7}"
--- PASS: TestRaft_ApplyConcurrent (0.33s)
=== RUN   TestRaft_ApplyConcurrent_Timeout
2026-03-29T01:07:44.998Z [INFO]  server-35195044-4fca-fbd0-2dd3-c9c9834ac138: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35195044-4fca-fbd0-2dd3-c9c9834ac138 Address:35195044-4fca-fbd0-2dd3-c9c9834ac138}]"
2026-03-29T01:07:44.998Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:44.998Z [INFO]  server-35195044-4fca-fbd0-2dd3-c9c9834ac138: entering follower state: follower="Node at 35195044-4fca-fbd0-2dd3-c9c9834ac138 [Follower]" leader-address= leader-id=
2026-03-29T01:07:45.114Z [WARN]  server-35195044-4fca-fbd0-2dd3-c9c9834ac138: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:45.114Z [INFO]  server-35195044-4fca-fbd0-2dd3-c9c9834ac138: entering candidate state: node="Node at 35195044-4fca-fbd0-2dd3-c9c9834ac138 [Candidate]" term=2
2026-03-29T01:07:45.114Z [INFO]  server-35195044-4fca-fbd0-2dd3-c9c9834ac138: election won: term=2 tally=1
2026-03-29T01:07:45.114Z [INFO]  server-35195044-4fca-fbd0-2dd3-c9c9834ac138: entering leader state: leader="Node at 35195044-4fca-fbd0-2dd3-c9c9834ac138 [Leader]"
2026-03-29T01:07:45.315Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:45.114800332 +0000 UTC m=+8.902942933 (1 nodes), 116.575175ms from start of poll, 116.677847ms from cluster start. Timeout at 2026-03-29 01:07:45.315802067 +0000 UTC m=+9.103944707, 201.001774ms after stability
--- PASS: TestRaft_ApplyConcurrent_Timeout (0.32s)
=== RUN   TestRaft_JoinNode
2026-03-29T01:07:45.316Z [INFO]  server-96a28ad1-9ca1-8f53-b17b-68581e156752: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96a28ad1-9ca1-8f53-b17b-68581e156752 Address:96a28ad1-9ca1-8f53-b17b-68581e156752} {Suffrage:Voter ID:server-966246e9-34fa-1885-a6c0-0a51669d0f19 Address:966246e9-34fa-1885-a6c0-0a51669d0f19}]"
2026-03-29T01:07:45.316Z [INFO]  server-96a28ad1-9ca1-8f53-b17b-68581e156752: entering follower state: follower="Node at 96a28ad1-9ca1-8f53-b17b-68581e156752 [Follower]" leader-address= leader-id=
2026-03-29T01:07:45.316Z [INFO]  server-966246e9-34fa-1885-a6c0-0a51669d0f19: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-96a28ad1-9ca1-8f53-b17b-68581e156752 Address:96a28ad1-9ca1-8f53-b17b-68581e156752} {Suffrage:Voter ID:server-966246e9-34fa-1885-a6c0-0a51669d0f19 Address:966246e9-34fa-1885-a6c0-0a51669d0f19}]"
2026-03-29T01:07:45.316Z [INFO]  server-966246e9-34fa-1885-a6c0-0a51669d0f19: entering follower state: follower="Node at 966246e9-34fa-1885-a6c0-0a51669d0f19 [Follower]" leader-address= leader-id=
2026-03-29T01:07:45.316Z [INFO]  server-c8d671db-a2df-695c-7edb-4b7ecc9fea7d: initial configuration: index=0 servers=[]
2026-03-29T01:07:45.316Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:45.316Z [INFO]  server-c8d671db-a2df-695c-7edb-4b7ecc9fea7d: entering follower state: follower="Node at c8d671db-a2df-695c-7edb-4b7ecc9fea7d [Follower]" leader-address= leader-id=
2026-03-29T01:07:45.391Z [WARN]  server-966246e9-34fa-1885-a6c0-0a51669d0f19: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:45.391Z [INFO]  server-966246e9-34fa-1885-a6c0-0a51669d0f19: entering candidate state: node="Node at 966246e9-34fa-1885-a6c0-0a51669d0f19 [Candidate]" term=2
2026-03-29T01:07:45.392Z [INFO]  server-966246e9-34fa-1885-a6c0-0a51669d0f19: election won: term=2 tally=2
2026-03-29T01:07:45.392Z [INFO]  server-966246e9-34fa-1885-a6c0-0a51669d0f19: entering leader state: leader="Node at 966246e9-34fa-1885-a6c0-0a51669d0f19 [Leader]"
2026-03-29T01:07:45.392Z [INFO]  server-966246e9-34fa-1885-a6c0-0a51669d0f19: added peer, starting replication: peer=server-96a28ad1-9ca1-8f53-b17b-68581e156752
2026-03-29T01:07:45.392Z [INFO]  server-966246e9-34fa-1885-a6c0-0a51669d0f19: pipelining replication: peer="{Voter server-96a28ad1-9ca1-8f53-b17b-68581e156752 96a28ad1-9ca1-8f53-b17b-68581e156752}"
2026-03-29T01:07:45.404Z [WARN]  server-c8d671db-a2df-695c-7edb-4b7ecc9fea7d: no known peers, aborting election
2026-03-29T01:07:45.497Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:45.392052726 +0000 UTC m=+9.180195261 (1 nodes), 75.094069ms from start of poll, 75.514391ms from cluster start. Timeout at 2026-03-29 01:07:45.497054288 +0000 UTC m=+9.285196899, 105.001638ms after stability
2026-03-29T01:07:45.497Z [INFO]  server-966246e9-34fa-1885-a6c0-0a51669d0f19: updating configuration: command=AddVoter server-id=server-c8d671db-a2df-695c-7edb-4b7ecc9fea7d server-addr=c8d671db-a2df-695c-7edb-4b7ecc9fea7d servers="[{Suffrage:Voter ID:server-96a28ad1-9ca1-8f53-b17b-68581e156752 Address:96a28ad1-9ca1-8f53-b17b-68581e156752} {Suffrage:Voter ID:server-966246e9-34fa-1885-a6c0-0a51669d0f19 Address:966246e9-34fa-1885-a6c0-0a51669d0f19} {Suffrage:Voter ID:server-c8d671db-a2df-695c-7edb-4b7ecc9fea7d Address:c8d671db-a2df-695c-7edb-4b7ecc9fea7d}]"
2026-03-29T01:07:45.497Z [INFO]  server-966246e9-34fa-1885-a6c0-0a51669d0f19: added peer, starting replication: peer=server-c8d671db-a2df-695c-7edb-4b7ecc9fea7d
2026-03-29T01:07:45.497Z [WARN]  server-c8d671db-a2df-695c-7edb-4b7ecc9fea7d: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2026-03-29T01:07:45.497Z [WARN]  server-966246e9-34fa-1885-a6c0-0a51669d0f19: appendEntries rejected, sending older logs: peer="{Voter server-c8d671db-a2df-695c-7edb-4b7ecc9fea7d c8d671db-a2df-695c-7edb-4b7ecc9fea7d}" next=1
2026-03-29T01:07:45.497Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:45.498Z [INFO]  server-966246e9-34fa-1885-a6c0-0a51669d0f19: pipelining replication: peer="{Voter server-c8d671db-a2df-695c-7edb-4b7ecc9fea7d c8d671db-a2df-695c-7edb-4b7ecc9fea7d}"
2026-03-29T01:07:45.603Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:45.498022844 +0000 UTC m=+9.286165379 (1 nodes), 1.012µs from start of poll, 181.484509ms from cluster start. Timeout at 2026-03-29 01:07:45.603032059 +0000 UTC m=+9.391174666, 105.009287ms after stability
2026-03-29T01:07:45.603Z [INFO]  server-966246e9-34fa-1885-a6c0-0a51669d0f19: aborting pipeline replication: peer="{Voter server-c8d671db-a2df-695c-7edb-4b7ecc9fea7d c8d671db-a2df-695c-7edb-4b7ecc9fea7d}"
2026-03-29T01:07:45.603Z [INFO]  server-966246e9-34fa-1885-a6c0-0a51669d0f19: aborting pipeline replication: peer="{Voter server-96a28ad1-9ca1-8f53-b17b-68581e156752 96a28ad1-9ca1-8f53-b17b-68581e156752}"
--- PASS: TestRaft_JoinNode (0.29s)
=== RUN   TestRaft_JoinNode_ConfigStore
2026-03-29T01:07:45.604Z [INFO]  server-b89e01cf-d937-a387-5bca-aaa2e2fcea30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b89e01cf-d937-a387-5bca-aaa2e2fcea30 Address:b89e01cf-d937-a387-5bca-aaa2e2fcea30}]"
2026-03-29T01:07:45.604Z [INFO]  server-b89e01cf-d937-a387-5bca-aaa2e2fcea30: entering follower state: follower="Node at b89e01cf-d937-a387-5bca-aaa2e2fcea30 [Follower]" leader-address= leader-id=
2026-03-29T01:07:45.604Z [INFO]  server-785f4b4b-fb3e-b992-496b-be91e8fdaabe: initial configuration: index=0 servers=[]
2026-03-29T01:07:45.604Z [INFO]  server-785f4b4b-fb3e-b992-496b-be91e8fdaabe: entering follower state: follower="Node at 785f4b4b-fb3e-b992-496b-be91e8fdaabe [Follower]" leader-address= leader-id=
2026-03-29T01:07:45.604Z [INFO]  server-a70223b7-349d-7d93-ed12-3c279cf7aced: initial configuration: index=0 servers=[]
2026-03-29T01:07:45.604Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:45.604Z [INFO]  server-a70223b7-349d-7d93-ed12-3c279cf7aced: entering follower state: follower="Node at a70223b7-349d-7d93-ed12-3c279cf7aced [Follower]" leader-address= leader-id=
2026-03-29T01:07:45.658Z [WARN]  server-a70223b7-349d-7d93-ed12-3c279cf7aced: no known peers, aborting election
2026-03-29T01:07:45.695Z [WARN]  server-b89e01cf-d937-a387-5bca-aaa2e2fcea30: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:45.695Z [INFO]  server-b89e01cf-d937-a387-5bca-aaa2e2fcea30: entering candidate state: node="Node at b89e01cf-d937-a387-5bca-aaa2e2fcea30 [Candidate]" term=2
2026-03-29T01:07:45.695Z [INFO]  server-b89e01cf-d937-a387-5bca-aaa2e2fcea30: election won: term=2 tally=1
2026-03-29T01:07:45.695Z [INFO]  server-b89e01cf-d937-a387-5bca-aaa2e2fcea30: entering leader state: leader="Node at b89e01cf-d937-a387-5bca-aaa2e2fcea30 [Leader]"
2026-03-29T01:07:45.699Z [WARN]  server-785f4b4b-fb3e-b992-496b-be91e8fdaabe: no known peers, aborting election
2026-03-29T01:07:45.801Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:45.695471347 +0000 UTC m=+9.483613888 (1 nodes), 90.678678ms from start of poll, 91.207372ms from cluster start. Timeout at 2026-03-29 01:07:45.80047879 +0000 UTC m=+9.588621377, 105.007489ms after stability
2026-03-29T01:07:45.801Z [INFO]  server-b89e01cf-d937-a387-5bca-aaa2e2fcea30: updating configuration: command=AddVoter server-id=server-785f4b4b-fb3e-b992-496b-be91e8fdaabe server-addr=785f4b4b-fb3e-b992-496b-be91e8fdaabe servers="[{Suffrage:Voter ID:server-b89e01cf-d937-a387-5bca-aaa2e2fcea30 Address:b89e01cf-d937-a387-5bca-aaa2e2fcea30} {Suffrage:Voter ID:server-785f4b4b-fb3e-b992-496b-be91e8fdaabe Address:785f4b4b-fb3e-b992-496b-be91e8fdaabe}]"
2026-03-29T01:07:45.801Z [INFO]  server-b89e01cf-d937-a387-5bca-aaa2e2fcea30: added peer, starting replication: peer=server-785f4b4b-fb3e-b992-496b-be91e8fdaabe
2026-03-29T01:07:45.801Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:45.801Z [WARN]  server-785f4b4b-fb3e-b992-496b-be91e8fdaabe: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2026-03-29T01:07:45.801Z [WARN]  server-b89e01cf-d937-a387-5bca-aaa2e2fcea30: appendEntries rejected, sending older logs: peer="{Voter server-785f4b4b-fb3e-b992-496b-be91e8fdaabe 785f4b4b-fb3e-b992-496b-be91e8fdaabe}" next=1
2026-03-29T01:07:45.802Z [INFO]  server-b89e01cf-d937-a387-5bca-aaa2e2fcea30: pipelining replication: peer="{Voter server-785f4b4b-fb3e-b992-496b-be91e8fdaabe 785f4b4b-fb3e-b992-496b-be91e8fdaabe}"
2026-03-29T01:07:45.907Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:45.80191512 +0000 UTC m=+9.590057665 (1 nodes), 564ns from start of poll, 197.651149ms from cluster start. Timeout at 2026-03-29 01:07:45.906928253 +0000 UTC m=+9.695070837, 105.013172ms after stability
2026-03-29T01:07:45.907Z [INFO]  server-b89e01cf-d937-a387-5bca-aaa2e2fcea30: updating configuration: command=AddVoter server-id=server-a70223b7-349d-7d93-ed12-3c279cf7aced server-addr=a70223b7-349d-7d93-ed12-3c279cf7aced servers="[{Suffrage:Voter ID:server-b89e01cf-d937-a387-5bca-aaa2e2fcea30 Address:b89e01cf-d937-a387-5bca-aaa2e2fcea30} {Suffrage:Voter ID:server-785f4b4b-fb3e-b992-496b-be91e8fdaabe Address:785f4b4b-fb3e-b992-496b-be91e8fdaabe} {Suffrage:Voter ID:server-a70223b7-349d-7d93-ed12-3c279cf7aced Address:a70223b7-349d-7d93-ed12-3c279cf7aced}]"
2026-03-29T01:07:45.907Z [INFO]  server-b89e01cf-d937-a387-5bca-aaa2e2fcea30: added peer, starting replication: peer=server-a70223b7-349d-7d93-ed12-3c279cf7aced
2026-03-29T01:07:45.907Z [WARN]  server-a70223b7-349d-7d93-ed12-3c279cf7aced: failed to get previous log: previous-index=4 last-index=0 error="log not found"
2026-03-29T01:07:45.907Z [WARN]  server-b89e01cf-d937-a387-5bca-aaa2e2fcea30: appendEntries rejected, sending older logs: peer="{Voter server-a70223b7-349d-7d93-ed12-3c279cf7aced a70223b7-349d-7d93-ed12-3c279cf7aced}" next=1
2026-03-29T01:07:45.907Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:45.907Z [INFO]  server-b89e01cf-d937-a387-5bca-aaa2e2fcea30: pipelining replication: peer="{Voter server-a70223b7-349d-7d93-ed12-3c279cf7aced a70223b7-349d-7d93-ed12-3c279cf7aced}"
2026-03-29T01:07:46.013Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:45.907668836 +0000 UTC m=+9.695811392 (1 nodes), 675ns from start of poll, 303.404876ms from cluster start. Timeout at 2026-03-29 01:07:46.012677222 +0000 UTC m=+9.800819805, 105.008413ms after stability
2026-03-29T01:07:46.013Z [INFO]  server-b89e01cf-d937-a387-5bca-aaa2e2fcea30: aborting pipeline replication: peer="{Voter server-a70223b7-349d-7d93-ed12-3c279cf7aced a70223b7-349d-7d93-ed12-3c279cf7aced}"
2026-03-29T01:07:46.013Z [INFO]  server-b89e01cf-d937-a387-5bca-aaa2e2fcea30: aborting pipeline replication: peer="{Voter server-785f4b4b-fb3e-b992-496b-be91e8fdaabe 785f4b4b-fb3e-b992-496b-be91e8fdaabe}"
--- PASS: TestRaft_JoinNode_ConfigStore (0.41s)
=== RUN   TestRaft_RemoveFollower
2026-03-29T01:07:46.014Z [INFO]  server-1316e6a5-c75d-37a1-182a-f3bddc01c69b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1316e6a5-c75d-37a1-182a-f3bddc01c69b Address:1316e6a5-c75d-37a1-182a-f3bddc01c69b} {Suffrage:Voter ID:server-0879fd21-80a1-f382-15ef-a276975cc5b1 Address:0879fd21-80a1-f382-15ef-a276975cc5b1} {Suffrage:Voter ID:server-65ba9c2e-8f42-71c6-d591-fb285847da91 Address:65ba9c2e-8f42-71c6-d591-fb285847da91}]"
2026-03-29T01:07:46.014Z [INFO]  server-1316e6a5-c75d-37a1-182a-f3bddc01c69b: entering follower state: follower="Node at 1316e6a5-c75d-37a1-182a-f3bddc01c69b [Follower]" leader-address= leader-id=
2026-03-29T01:07:46.014Z [INFO]  server-0879fd21-80a1-f382-15ef-a276975cc5b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1316e6a5-c75d-37a1-182a-f3bddc01c69b Address:1316e6a5-c75d-37a1-182a-f3bddc01c69b} {Suffrage:Voter ID:server-0879fd21-80a1-f382-15ef-a276975cc5b1 Address:0879fd21-80a1-f382-15ef-a276975cc5b1} {Suffrage:Voter ID:server-65ba9c2e-8f42-71c6-d591-fb285847da91 Address:65ba9c2e-8f42-71c6-d591-fb285847da91}]"
2026-03-29T01:07:46.014Z [INFO]  server-0879fd21-80a1-f382-15ef-a276975cc5b1: entering follower state: follower="Node at 0879fd21-80a1-f382-15ef-a276975cc5b1 [Follower]" leader-address= leader-id=
2026-03-29T01:07:46.014Z [INFO]  server-65ba9c2e-8f42-71c6-d591-fb285847da91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1316e6a5-c75d-37a1-182a-f3bddc01c69b Address:1316e6a5-c75d-37a1-182a-f3bddc01c69b} {Suffrage:Voter ID:server-0879fd21-80a1-f382-15ef-a276975cc5b1 Address:0879fd21-80a1-f382-15ef-a276975cc5b1} {Suffrage:Voter ID:server-65ba9c2e-8f42-71c6-d591-fb285847da91 Address:65ba9c2e-8f42-71c6-d591-fb285847da91}]"
2026-03-29T01:07:46.014Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:46.014Z [INFO]  server-65ba9c2e-8f42-71c6-d591-fb285847da91: entering follower state: follower="Node at 65ba9c2e-8f42-71c6-d591-fb285847da91 [Follower]" leader-address= leader-id=
2026-03-29T01:07:46.073Z [WARN]  server-65ba9c2e-8f42-71c6-d591-fb285847da91: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:46.073Z [INFO]  server-65ba9c2e-8f42-71c6-d591-fb285847da91: entering candidate state: node="Node at 65ba9c2e-8f42-71c6-d591-fb285847da91 [Candidate]" term=2
2026-03-29T01:07:46.073Z [INFO]  server-65ba9c2e-8f42-71c6-d591-fb285847da91: election won: term=2 tally=2
2026-03-29T01:07:46.073Z [INFO]  server-65ba9c2e-8f42-71c6-d591-fb285847da91: entering leader state: leader="Node at 65ba9c2e-8f42-71c6-d591-fb285847da91 [Leader]"
2026-03-29T01:07:46.073Z [INFO]  server-65ba9c2e-8f42-71c6-d591-fb285847da91: added peer, starting replication: peer=server-1316e6a5-c75d-37a1-182a-f3bddc01c69b
2026-03-29T01:07:46.073Z [INFO]  server-65ba9c2e-8f42-71c6-d591-fb285847da91: added peer, starting replication: peer=server-0879fd21-80a1-f382-15ef-a276975cc5b1
2026-03-29T01:07:46.073Z [INFO]  server-65ba9c2e-8f42-71c6-d591-fb285847da91: pipelining replication: peer="{Voter server-1316e6a5-c75d-37a1-182a-f3bddc01c69b 1316e6a5-c75d-37a1-182a-f3bddc01c69b}"
2026-03-29T01:07:46.073Z [INFO]  server-65ba9c2e-8f42-71c6-d591-fb285847da91: pipelining replication: peer="{Voter server-0879fd21-80a1-f382-15ef-a276975cc5b1 0879fd21-80a1-f382-15ef-a276975cc5b1}"
2026-03-29T01:07:46.178Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:46.073164426 +0000 UTC m=+9.861306969 (1 nodes), 58.342832ms from start of poll, 58.711493ms from cluster start. Timeout at 2026-03-29 01:07:46.178171108 +0000 UTC m=+9.966313712, 105.006743ms after stability
2026-03-29T01:07:46.183Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:07:46.289Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:07:46.18372353 +0000 UTC m=+9.971866157 (2 nodes), 2.175µs from start of poll, 169.270681ms from cluster start. Timeout at 2026-03-29 01:07:46.288731736 +0000 UTC m=+10.076874336, 105.008179ms after stability
2026-03-29T01:07:46.289Z [INFO]  server-65ba9c2e-8f42-71c6-d591-fb285847da91: updating configuration: command=RemoveServer server-id=server-1316e6a5-c75d-37a1-182a-f3bddc01c69b server-addr= servers="[{Suffrage:Voter ID:server-0879fd21-80a1-f382-15ef-a276975cc5b1 Address:0879fd21-80a1-f382-15ef-a276975cc5b1} {Suffrage:Voter ID:server-65ba9c2e-8f42-71c6-d591-fb285847da91 Address:65ba9c2e-8f42-71c6-d591-fb285847da91}]"
2026-03-29T01:07:46.289Z [INFO]  server-65ba9c2e-8f42-71c6-d591-fb285847da91: removed peer, stopping replication: peer=server-1316e6a5-c75d-37a1-182a-f3bddc01c69b last-index=3
2026-03-29T01:07:46.289Z [INFO]  server-65ba9c2e-8f42-71c6-d591-fb285847da91: aborting pipeline replication: peer="{Voter server-1316e6a5-c75d-37a1-182a-f3bddc01c69b 1316e6a5-c75d-37a1-182a-f3bddc01c69b}"
2026-03-29T01:07:46.352Z [WARN]  server-1316e6a5-c75d-37a1-182a-f3bddc01c69b: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2026-03-29T01:07:46.394Z [INFO]  server-65ba9c2e-8f42-71c6-d591-fb285847da91: aborting pipeline replication: peer="{Voter server-0879fd21-80a1-f382-15ef-a276975cc5b1 0879fd21-80a1-f382-15ef-a276975cc5b1}"
--- PASS: TestRaft_RemoveFollower (0.38s)
=== RUN   TestRaft_RemoveLeader
2026-03-29T01:07:46.399Z [INFO]  server-cbcc5791-39e3-43bc-bb61-33694d99b16a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cbcc5791-39e3-43bc-bb61-33694d99b16a Address:cbcc5791-39e3-43bc-bb61-33694d99b16a} {Suffrage:Voter ID:server-de76ae0e-227e-8a7c-de37-2143a13b3e64 Address:de76ae0e-227e-8a7c-de37-2143a13b3e64} {Suffrage:Voter ID:server-7631b864-2685-cab1-d127-7e704af037c8 Address:7631b864-2685-cab1-d127-7e704af037c8}]"
2026-03-29T01:07:46.399Z [INFO]  server-cbcc5791-39e3-43bc-bb61-33694d99b16a: entering follower state: follower="Node at cbcc5791-39e3-43bc-bb61-33694d99b16a [Follower]" leader-address= leader-id=
2026-03-29T01:07:46.399Z [INFO]  server-de76ae0e-227e-8a7c-de37-2143a13b3e64: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cbcc5791-39e3-43bc-bb61-33694d99b16a Address:cbcc5791-39e3-43bc-bb61-33694d99b16a} {Suffrage:Voter ID:server-de76ae0e-227e-8a7c-de37-2143a13b3e64 Address:de76ae0e-227e-8a7c-de37-2143a13b3e64} {Suffrage:Voter ID:server-7631b864-2685-cab1-d127-7e704af037c8 Address:7631b864-2685-cab1-d127-7e704af037c8}]"
2026-03-29T01:07:46.399Z [INFO]  server-de76ae0e-227e-8a7c-de37-2143a13b3e64: entering follower state: follower="Node at de76ae0e-227e-8a7c-de37-2143a13b3e64 [Follower]" leader-address= leader-id=
2026-03-29T01:07:46.399Z [INFO]  server-7631b864-2685-cab1-d127-7e704af037c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cbcc5791-39e3-43bc-bb61-33694d99b16a Address:cbcc5791-39e3-43bc-bb61-33694d99b16a} {Suffrage:Voter ID:server-de76ae0e-227e-8a7c-de37-2143a13b3e64 Address:de76ae0e-227e-8a7c-de37-2143a13b3e64} {Suffrage:Voter ID:server-7631b864-2685-cab1-d127-7e704af037c8 Address:7631b864-2685-cab1-d127-7e704af037c8}]"
2026-03-29T01:07:46.399Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:46.399Z [INFO]  server-7631b864-2685-cab1-d127-7e704af037c8: entering follower state: follower="Node at 7631b864-2685-cab1-d127-7e704af037c8 [Follower]" leader-address= leader-id=
2026-03-29T01:07:46.453Z [WARN]  server-de76ae0e-227e-8a7c-de37-2143a13b3e64: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:46.453Z [INFO]  server-de76ae0e-227e-8a7c-de37-2143a13b3e64: entering candidate state: node="Node at de76ae0e-227e-8a7c-de37-2143a13b3e64 [Candidate]" term=2
2026-03-29T01:07:46.453Z [INFO]  server-de76ae0e-227e-8a7c-de37-2143a13b3e64: election won: term=2 tally=2
2026-03-29T01:07:46.453Z [INFO]  server-de76ae0e-227e-8a7c-de37-2143a13b3e64: entering leader state: leader="Node at de76ae0e-227e-8a7c-de37-2143a13b3e64 [Leader]"
2026-03-29T01:07:46.453Z [INFO]  server-de76ae0e-227e-8a7c-de37-2143a13b3e64: added peer, starting replication: peer=server-cbcc5791-39e3-43bc-bb61-33694d99b16a
2026-03-29T01:07:46.453Z [INFO]  server-de76ae0e-227e-8a7c-de37-2143a13b3e64: added peer, starting replication: peer=server-7631b864-2685-cab1-d127-7e704af037c8
2026-03-29T01:07:46.454Z [INFO]  server-de76ae0e-227e-8a7c-de37-2143a13b3e64: pipelining replication: peer="{Voter server-7631b864-2685-cab1-d127-7e704af037c8 7631b864-2685-cab1-d127-7e704af037c8}"
2026-03-29T01:07:46.454Z [INFO]  server-de76ae0e-227e-8a7c-de37-2143a13b3e64: pipelining replication: peer="{Voter server-cbcc5791-39e3-43bc-bb61-33694d99b16a cbcc5791-39e3-43bc-bb61-33694d99b16a}"
2026-03-29T01:07:46.559Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:46.454012681 +0000 UTC m=+10.242155278 (1 nodes), 54.728224ms from start of poll, 55.047537ms from cluster start. Timeout at 2026-03-29 01:07:46.559018079 +0000 UTC m=+10.347160607, 105.005329ms after stability
2026-03-29T01:07:46.564Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:07:46.670Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:07:46.56448815 +0000 UTC m=+10.352630709 (2 nodes), 1.055µs from start of poll, 165.522968ms from cluster start. Timeout at 2026-03-29 01:07:46.669490866 +0000 UTC m=+10.457633405, 105.002696ms after stability
2026-03-29T01:07:46.670Z [INFO]  server-de76ae0e-227e-8a7c-de37-2143a13b3e64: updating configuration: command=RemoveServer server-id=server-de76ae0e-227e-8a7c-de37-2143a13b3e64 server-addr= servers="[{Suffrage:Voter ID:server-cbcc5791-39e3-43bc-bb61-33694d99b16a Address:cbcc5791-39e3-43bc-bb61-33694d99b16a} {Suffrage:Voter ID:server-7631b864-2685-cab1-d127-7e704af037c8 Address:7631b864-2685-cab1-d127-7e704af037c8}]"
2026-03-29T01:07:46.670Z [INFO]  server-de76ae0e-227e-8a7c-de37-2143a13b3e64: removed ourself, shutting down
2026-03-29T01:07:46.670Z [INFO]  server-de76ae0e-227e-8a7c-de37-2143a13b3e64: aborting pipeline replication: peer="{Voter server-7631b864-2685-cab1-d127-7e704af037c8 7631b864-2685-cab1-d127-7e704af037c8}"
2026-03-29T01:07:46.670Z [INFO]  server-de76ae0e-227e-8a7c-de37-2143a13b3e64: aborting pipeline replication: peer="{Voter server-cbcc5791-39e3-43bc-bb61-33694d99b16a cbcc5791-39e3-43bc-bb61-33694d99b16a}"
2026-03-29T01:07:46.754Z [WARN]  server-7631b864-2685-cab1-d127-7e704af037c8: heartbeat timeout reached, starting election: last-leader-addr=de76ae0e-227e-8a7c-de37-2143a13b3e64 last-leader-id=server-de76ae0e-227e-8a7c-de37-2143a13b3e64
2026-03-29T01:07:46.754Z [INFO]  server-7631b864-2685-cab1-d127-7e704af037c8: entering candidate state: node="Node at 7631b864-2685-cab1-d127-7e704af037c8 [Candidate]" term=3
2026-03-29T01:07:46.754Z [WARN]  server-cbcc5791-39e3-43bc-bb61-33694d99b16a: rejecting vote request since we have a leader: from=7631b864-2685-cab1-d127-7e704af037c8 leader=de76ae0e-227e-8a7c-de37-2143a13b3e64 leader-id=server-de76ae0e-227e-8a7c-de37-2143a13b3e64
2026-03-29T01:07:46.761Z [WARN]  server-cbcc5791-39e3-43bc-bb61-33694d99b16a: heartbeat timeout reached, starting election: last-leader-addr=de76ae0e-227e-8a7c-de37-2143a13b3e64 last-leader-id=server-de76ae0e-227e-8a7c-de37-2143a13b3e64
2026-03-29T01:07:46.761Z [INFO]  server-cbcc5791-39e3-43bc-bb61-33694d99b16a: entering candidate state: node="Node at cbcc5791-39e3-43bc-bb61-33694d99b16a [Candidate]" term=3
2026-03-29T01:07:46.761Z [INFO]  server-7631b864-2685-cab1-d127-7e704af037c8: duplicate requestVote for same term: term=3
2026-03-29T01:07:46.813Z [WARN]  server-7631b864-2685-cab1-d127-7e704af037c8: Election timeout reached, restarting election
2026-03-29T01:07:46.813Z [INFO]  server-7631b864-2685-cab1-d127-7e704af037c8: entering candidate state: node="Node at 7631b864-2685-cab1-d127-7e704af037c8 [Candidate]" term=4
2026-03-29T01:07:46.813Z [INFO]  server-cbcc5791-39e3-43bc-bb61-33694d99b16a: entering follower state: follower="Node at cbcc5791-39e3-43bc-bb61-33694d99b16a [Follower]" leader-address= leader-id=
2026-03-29T01:07:46.813Z [INFO]  server-7631b864-2685-cab1-d127-7e704af037c8: election won: term=4 tally=2
2026-03-29T01:07:46.813Z [INFO]  server-7631b864-2685-cab1-d127-7e704af037c8: entering leader state: leader="Node at 7631b864-2685-cab1-d127-7e704af037c8 [Leader]"
2026-03-29T01:07:46.813Z [INFO]  server-7631b864-2685-cab1-d127-7e704af037c8: added peer, starting replication: peer=server-cbcc5791-39e3-43bc-bb61-33694d99b16a
2026-03-29T01:07:46.813Z [INFO]  server-7631b864-2685-cab1-d127-7e704af037c8: pipelining replication: peer="{Voter server-cbcc5791-39e3-43bc-bb61-33694d99b16a cbcc5791-39e3-43bc-bb61-33694d99b16a}"
2026-03-29T01:07:46.882Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:46.988Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:46.882079303 +0000 UTC m=+10.670221856 (1 nodes), 60.209µs from start of poll, 483.114115ms from cluster start. Timeout at 2026-03-29 01:07:46.987083052 +0000 UTC m=+10.775225656, 105.0038ms after stability
2026-03-29T01:07:46.988Z [INFO]  server-7631b864-2685-cab1-d127-7e704af037c8: aborting pipeline replication: peer="{Voter server-cbcc5791-39e3-43bc-bb61-33694d99b16a cbcc5791-39e3-43bc-bb61-33694d99b16a}"
--- PASS: TestRaft_RemoveLeader (0.59s)
=== RUN   TestRaft_RemoveLeader_NoShutdown
2026-03-29T01:07:46.989Z [INFO]  server-ca5ef020-cf24-6a49-41e5-d738253eb1b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca5ef020-cf24-6a49-41e5-d738253eb1b7 Address:ca5ef020-cf24-6a49-41e5-d738253eb1b7} {Suffrage:Voter ID:server-145af84c-cf8b-badd-35ea-2bf7ac6a791e Address:145af84c-cf8b-badd-35ea-2bf7ac6a791e} {Suffrage:Voter ID:server-57be2424-df4d-5938-3959-36848ade3e89 Address:57be2424-df4d-5938-3959-36848ade3e89}]"
2026-03-29T01:07:46.989Z [INFO]  server-ca5ef020-cf24-6a49-41e5-d738253eb1b7: entering follower state: follower="Node at ca5ef020-cf24-6a49-41e5-d738253eb1b7 [Follower]" leader-address= leader-id=
2026-03-29T01:07:46.989Z [INFO]  server-145af84c-cf8b-badd-35ea-2bf7ac6a791e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca5ef020-cf24-6a49-41e5-d738253eb1b7 Address:ca5ef020-cf24-6a49-41e5-d738253eb1b7} {Suffrage:Voter ID:server-145af84c-cf8b-badd-35ea-2bf7ac6a791e Address:145af84c-cf8b-badd-35ea-2bf7ac6a791e} {Suffrage:Voter ID:server-57be2424-df4d-5938-3959-36848ade3e89 Address:57be2424-df4d-5938-3959-36848ade3e89}]"
2026-03-29T01:07:46.989Z [INFO]  server-145af84c-cf8b-badd-35ea-2bf7ac6a791e: entering follower state: follower="Node at 145af84c-cf8b-badd-35ea-2bf7ac6a791e [Follower]" leader-address= leader-id=
2026-03-29T01:07:46.989Z [INFO]  server-57be2424-df4d-5938-3959-36848ade3e89: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ca5ef020-cf24-6a49-41e5-d738253eb1b7 Address:ca5ef020-cf24-6a49-41e5-d738253eb1b7} {Suffrage:Voter ID:server-145af84c-cf8b-badd-35ea-2bf7ac6a791e Address:145af84c-cf8b-badd-35ea-2bf7ac6a791e} {Suffrage:Voter ID:server-57be2424-df4d-5938-3959-36848ade3e89 Address:57be2424-df4d-5938-3959-36848ade3e89}]"
2026-03-29T01:07:46.989Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:07:46.989Z [INFO]  server-57be2424-df4d-5938-3959-36848ade3e89: entering follower state: follower="Node at 57be2424-df4d-5938-3959-36848ade3e89 [Follower]" leader-address= leader-id=
2026-03-29T01:07:47.048Z [WARN]  server-ca5ef020-cf24-6a49-41e5-d738253eb1b7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:47.048Z [INFO]  server-ca5ef020-cf24-6a49-41e5-d738253eb1b7: entering candidate state: node="Node at ca5ef020-cf24-6a49-41e5-d738253eb1b7 [Candidate]" term=2
2026-03-29T01:07:47.048Z [INFO]  server-ca5ef020-cf24-6a49-41e5-d738253eb1b7: election won: term=2 tally=2
2026-03-29T01:07:47.048Z [INFO]  server-ca5ef020-cf24-6a49-41e5-d738253eb1b7: entering leader state: leader="Node at ca5ef020-cf24-6a49-41e5-d738253eb1b7 [Leader]"
2026-03-29T01:07:47.049Z [INFO]  server-ca5ef020-cf24-6a49-41e5-d738253eb1b7: added peer, starting replication: peer=server-145af84c-cf8b-badd-35ea-2bf7ac6a791e
2026-03-29T01:07:47.049Z [INFO]  server-ca5ef020-cf24-6a49-41e5-d738253eb1b7: added peer, starting replication: peer=server-57be2424-df4d-5938-3959-36848ade3e89
2026-03-29T01:07:47.049Z [INFO]  server-ca5ef020-cf24-6a49-41e5-d738253eb1b7: pipelining replication: peer="{Voter server-57be2424-df4d-5938-3959-36848ade3e89 57be2424-df4d-5938-3959-36848ade3e89}"
2026-03-29T01:07:47.049Z [INFO]  server-ca5ef020-cf24-6a49-41e5-d738253eb1b7: pipelining replication: peer="{Voter server-145af84c-cf8b-badd-35ea-2bf7ac6a791e 145af84c-cf8b-badd-35ea-2bf7ac6a791e}"
2026-03-29T01:07:47.154Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:07:47.04906805 +0000 UTC m=+10.837210620 (2 nodes), 59.707234ms from start of poll, 60.017982ms from cluster start. Timeout at 2026-03-29 01:07:47.154069694 +0000 UTC m=+10.942212305, 105.001685ms after stability
2026-03-29T01:07:47.154Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:47.260Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:47.154477343 +0000 UTC m=+10.942619960 (1 nodes), 1.985µs from start of poll, 165.427322ms from cluster start. Timeout at 2026-03-29 01:07:47.25948193 +0000 UTC m=+11.047624540, 105.00458ms after stability
2026-03-29T01:07:47.264Z [INFO]  server-ca5ef020-cf24-6a49-41e5-d738253eb1b7: updating configuration: command=RemoveServer server-id=server-ca5ef020-cf24-6a49-41e5-d738253eb1b7 server-addr= servers="[{Suffrage:Voter ID:server-145af84c-cf8b-badd-35ea-2bf7ac6a791e Address:145af84c-cf8b-badd-35ea-2bf7ac6a791e} {Suffrage:Voter ID:server-57be2424-df4d-5938-3959-36848ade3e89 Address:57be2424-df4d-5938-3959-36848ade3e89}]"
2026-03-29T01:07:47.264Z [INFO]  server-ca5ef020-cf24-6a49-41e5-d738253eb1b7: removed ourself, transitioning to follower
2026-03-29T01:07:47.264Z [INFO]  server-ca5ef020-cf24-6a49-41e5-d738253eb1b7: entering follower state: follower="Node at ca5ef020-cf24-6a49-41e5-d738253eb1b7 [Follower]" leader-address= leader-id=
2026-03-29T01:07:47.264Z [INFO]  server-ca5ef020-cf24-6a49-41e5-d738253eb1b7: aborting pipeline replication: peer="{Voter server-145af84c-cf8b-badd-35ea-2bf7ac6a791e 145af84c-cf8b-badd-35ea-2bf7ac6a791e}"
2026-03-29T01:07:47.264Z [INFO]  server-ca5ef020-cf24-6a49-41e5-d738253eb1b7: aborting pipeline replication: peer="{Voter server-57be2424-df4d-5938-3959-36848ade3e89 57be2424-df4d-5938-3959-36848ade3e89}"
2026-03-29T01:07:47.335Z [WARN]  server-ca5ef020-cf24-6a49-41e5-d738253eb1b7: not part of stable configuration, aborting election
2026-03-29T01:07:47.338Z [WARN]  server-57be2424-df4d-5938-3959-36848ade3e89: heartbeat timeout reached, starting election: last-leader-addr=ca5ef020-cf24-6a49-41e5-d738253eb1b7 last-leader-id=server-ca5ef020-cf24-6a49-41e5-d738253eb1b7
2026-03-29T01:07:47.338Z [INFO]  server-57be2424-df4d-5938-3959-36848ade3e89: entering candidate state: node="Node at 57be2424-df4d-5938-3959-36848ade3e89 [Candidate]" term=3
2026-03-29T01:07:47.338Z [WARN]  server-145af84c-cf8b-badd-35ea-2bf7ac6a791e: rejecting vote request since we have a leader: from=57be2424-df4d-5938-3959-36848ade3e89 leader=ca5ef020-cf24-6a49-41e5-d738253eb1b7 leader-id=server-ca5ef020-cf24-6a49-41e5-d738253eb1b7
2026-03-29T01:07:47.346Z [WARN]  server-145af84c-cf8b-badd-35ea-2bf7ac6a791e: heartbeat timeout reached, starting election: last-leader-addr=ca5ef020-cf24-6a49-41e5-d738253eb1b7 last-leader-id=server-ca5ef020-cf24-6a49-41e5-d738253eb1b7
2026-03-29T01:07:47.347Z [INFO]  server-145af84c-cf8b-badd-35ea-2bf7ac6a791e: entering candidate state: node="Node at 145af84c-cf8b-badd-35ea-2bf7ac6a791e [Candidate]" term=3
2026-03-29T01:07:47.347Z [INFO]  server-57be2424-df4d-5938-3959-36848ade3e89: duplicate requestVote for same term: term=3
2026-03-29T01:07:47.370Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:47.404Z [WARN]  server-145af84c-cf8b-badd-35ea-2bf7ac6a791e: Election timeout reached, restarting election
2026-03-29T01:07:47.404Z [INFO]  server-145af84c-cf8b-badd-35ea-2bf7ac6a791e: entering candidate state: node="Node at 145af84c-cf8b-badd-35ea-2bf7ac6a791e [Candidate]" term=4
2026-03-29T01:07:47.404Z [INFO]  server-57be2424-df4d-5938-3959-36848ade3e89: entering follower state: follower="Node at 57be2424-df4d-5938-3959-36848ade3e89 [Follower]" leader-address= leader-id=
2026-03-29T01:07:47.404Z [INFO]  server-145af84c-cf8b-badd-35ea-2bf7ac6a791e: election won: term=4 tally=2
2026-03-29T01:07:47.404Z [INFO]  server-145af84c-cf8b-badd-35ea-2bf7ac6a791e: entering leader state: leader="Node at 145af84c-cf8b-badd-35ea-2bf7ac6a791e [Leader]"
2026-03-29T01:07:47.404Z [INFO]  server-145af84c-cf8b-badd-35ea-2bf7ac6a791e: added peer, starting replication: peer=server-57be2424-df4d-5938-3959-36848ade3e89
2026-03-29T01:07:47.404Z [INFO]  server-145af84c-cf8b-badd-35ea-2bf7ac6a791e: pipelining replication: peer="{Voter server-57be2424-df4d-5938-3959-36848ade3e89 57be2424-df4d-5938-3959-36848ade3e89}"
2026-03-29T01:07:47.510Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:47.40467064 +0000 UTC m=+11.192813190 (1 nodes), 33.798398ms from start of poll, 415.620552ms from cluster start. Timeout at 2026-03-29 01:07:47.509759265 +0000 UTC m=+11.297901902, 105.088712ms after stability
2026-03-29T01:07:47.616Z [INFO]  server-145af84c-cf8b-badd-35ea-2bf7ac6a791e: aborting pipeline replication: peer="{Voter server-57be2424-df4d-5938-3959-36848ade3e89 57be2424-df4d-5938-3959-36848ade3e89}"
--- PASS: TestRaft_RemoveLeader_NoShutdown (0.63s)
=== RUN   TestRaft_RemoveFollower_SplitCluster
2026-03-29T01:07:47.617Z [INFO]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21c5110c-095b-e3dd-b8c3-c0fc0b117490 Address:21c5110c-095b-e3dd-b8c3-c0fc0b117490} {Suffrage:Voter ID:server-bad00201-8c6c-3416-3fd9-72fb23a0710a Address:bad00201-8c6c-3416-3fd9-72fb23a0710a} {Suffrage:Voter ID:server-a2f29c39-b95d-b45b-4b03-afb148e5b60c Address:a2f29c39-b95d-b45b-4b03-afb148e5b60c} {Suffrage:Voter ID:server-714dd75b-2b44-f74e-a419-4eb54f703b36 Address:714dd75b-2b44-f74e-a419-4eb54f703b36}]"
2026-03-29T01:07:47.617Z [INFO]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: entering follower state: follower="Node at 21c5110c-095b-e3dd-b8c3-c0fc0b117490 [Follower]" leader-address= leader-id=
2026-03-29T01:07:47.617Z [INFO]  server-bad00201-8c6c-3416-3fd9-72fb23a0710a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21c5110c-095b-e3dd-b8c3-c0fc0b117490 Address:21c5110c-095b-e3dd-b8c3-c0fc0b117490} {Suffrage:Voter ID:server-bad00201-8c6c-3416-3fd9-72fb23a0710a Address:bad00201-8c6c-3416-3fd9-72fb23a0710a} {Suffrage:Voter ID:server-a2f29c39-b95d-b45b-4b03-afb148e5b60c Address:a2f29c39-b95d-b45b-4b03-afb148e5b60c} {Suffrage:Voter ID:server-714dd75b-2b44-f74e-a419-4eb54f703b36 Address:714dd75b-2b44-f74e-a419-4eb54f703b36}]"
2026-03-29T01:07:47.617Z [INFO]  server-bad00201-8c6c-3416-3fd9-72fb23a0710a: entering follower state: follower="Node at bad00201-8c6c-3416-3fd9-72fb23a0710a [Follower]" leader-address= leader-id=
2026-03-29T01:07:47.617Z [INFO]  server-a2f29c39-b95d-b45b-4b03-afb148e5b60c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21c5110c-095b-e3dd-b8c3-c0fc0b117490 Address:21c5110c-095b-e3dd-b8c3-c0fc0b117490} {Suffrage:Voter ID:server-bad00201-8c6c-3416-3fd9-72fb23a0710a Address:bad00201-8c6c-3416-3fd9-72fb23a0710a} {Suffrage:Voter ID:server-a2f29c39-b95d-b45b-4b03-afb148e5b60c Address:a2f29c39-b95d-b45b-4b03-afb148e5b60c} {Suffrage:Voter ID:server-714dd75b-2b44-f74e-a419-4eb54f703b36 Address:714dd75b-2b44-f74e-a419-4eb54f703b36}]"
2026-03-29T01:07:47.617Z [INFO]  server-a2f29c39-b95d-b45b-4b03-afb148e5b60c: entering follower state: follower="Node at a2f29c39-b95d-b45b-4b03-afb148e5b60c [Follower]" leader-address= leader-id=
2026-03-29T01:07:47.618Z [INFO]  server-714dd75b-2b44-f74e-a419-4eb54f703b36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21c5110c-095b-e3dd-b8c3-c0fc0b117490 Address:21c5110c-095b-e3dd-b8c3-c0fc0b117490} {Suffrage:Voter ID:server-bad00201-8c6c-3416-3fd9-72fb23a0710a Address:bad00201-8c6c-3416-3fd9-72fb23a0710a} {Suffrage:Voter ID:server-a2f29c39-b95d-b45b-4b03-afb148e5b60c Address:a2f29c39-b95d-b45b-4b03-afb148e5b60c} {Suffrage:Voter ID:server-714dd75b-2b44-f74e-a419-4eb54f703b36 Address:714dd75b-2b44-f74e-a419-4eb54f703b36}]"
2026-03-29T01:07:47.618Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:47.618Z [INFO]  server-714dd75b-2b44-f74e-a419-4eb54f703b36: entering follower state: follower="Node at 714dd75b-2b44-f74e-a419-4eb54f703b36 [Follower]" leader-address= leader-id=
2026-03-29T01:07:47.672Z [WARN]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:47.672Z [INFO]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: entering candidate state: node="Node at 21c5110c-095b-e3dd-b8c3-c0fc0b117490 [Candidate]" term=2
2026-03-29T01:07:47.673Z [INFO]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: election won: term=2 tally=3
2026-03-29T01:07:47.673Z [INFO]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: entering leader state: leader="Node at 21c5110c-095b-e3dd-b8c3-c0fc0b117490 [Leader]"
2026-03-29T01:07:47.673Z [INFO]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: added peer, starting replication: peer=server-bad00201-8c6c-3416-3fd9-72fb23a0710a
2026-03-29T01:07:47.673Z [INFO]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: added peer, starting replication: peer=server-a2f29c39-b95d-b45b-4b03-afb148e5b60c
2026-03-29T01:07:47.673Z [INFO]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: added peer, starting replication: peer=server-714dd75b-2b44-f74e-a419-4eb54f703b36
2026-03-29T01:07:47.673Z [INFO]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: pipelining replication: peer="{Voter server-bad00201-8c6c-3416-3fd9-72fb23a0710a bad00201-8c6c-3416-3fd9-72fb23a0710a}"
2026-03-29T01:07:47.673Z [INFO]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: pipelining replication: peer="{Voter server-714dd75b-2b44-f74e-a419-4eb54f703b36 714dd75b-2b44-f74e-a419-4eb54f703b36}"
2026-03-29T01:07:47.673Z [INFO]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: pipelining replication: peer="{Voter server-a2f29c39-b95d-b45b-4b03-afb148e5b60c a2f29c39-b95d-b45b-4b03-afb148e5b60c}"
2026-03-29T01:07:47.778Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:47.673072803 +0000 UTC m=+11.461215336 (1 nodes), 55.024962ms from start of poll, 55.453003ms from cluster start. Timeout at 2026-03-29 01:07:47.778076487 +0000 UTC m=+11.566219025, 105.003689ms after stability
2026-03-29T01:07:47.883Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:07:47.988Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:07:47.883505195 +0000 UTC m=+11.671647745 (3 nodes), 975ns from start of poll, 265.885412ms from cluster start. Timeout at 2026-03-29 01:07:47.988520455 +0000 UTC m=+11.776663038, 105.015293ms after stability
2026-03-29T01:07:47.988Z [INFO]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: updating configuration: command=RemoveServer server-id=server-714dd75b-2b44-f74e-a419-4eb54f703b36 server-addr= servers="[{Suffrage:Voter ID:server-21c5110c-095b-e3dd-b8c3-c0fc0b117490 Address:21c5110c-095b-e3dd-b8c3-c0fc0b117490} {Suffrage:Voter ID:server-bad00201-8c6c-3416-3fd9-72fb23a0710a Address:bad00201-8c6c-3416-3fd9-72fb23a0710a} {Suffrage:Voter ID:server-a2f29c39-b95d-b45b-4b03-afb148e5b60c Address:a2f29c39-b95d-b45b-4b03-afb148e5b60c}]"
2026-03-29T01:07:47.989Z [INFO]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: removed peer, stopping replication: peer=server-714dd75b-2b44-f74e-a419-4eb54f703b36 last-index=3
2026-03-29T01:07:47.989Z [INFO]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: aborting pipeline replication: peer="{Voter server-714dd75b-2b44-f74e-a419-4eb54f703b36 714dd75b-2b44-f74e-a419-4eb54f703b36}"
2026-03-29T01:07:47.989Z [ERROR] server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to pipeline appendEntries: peer="{Voter server-a2f29c39-b95d-b45b-4b03-afb148e5b60c a2f29c39-b95d-b45b-4b03-afb148e5b60c}" error="append pipeline closed"
2026-03-29T01:07:47.989Z [INFO]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: aborting pipeline replication: peer="{Voter server-a2f29c39-b95d-b45b-4b03-afb148e5b60c a2f29c39-b95d-b45b-4b03-afb148e5b60c}"
2026-03-29T01:07:47.989Z [ERROR] server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to pipeline appendEntries: peer="{Voter server-bad00201-8c6c-3416-3fd9-72fb23a0710a bad00201-8c6c-3416-3fd9-72fb23a0710a}" error="append pipeline closed"
2026-03-29T01:07:47.989Z [INFO]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: aborting pipeline replication: peer="{Voter server-bad00201-8c6c-3416-3fd9-72fb23a0710a bad00201-8c6c-3416-3fd9-72fb23a0710a}"
2026-03-29T01:07:47.989Z [ERROR] server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to heartbeat to: peer=a2f29c39-b95d-b45b-4b03-afb148e5b60c backoff time=10ms error="failed to connect to peer: a2f29c39-b95d-b45b-4b03-afb148e5b60c"
2026-03-29T01:07:47.994Z [ERROR] server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to heartbeat to: peer=bad00201-8c6c-3416-3fd9-72fb23a0710a backoff time=10ms error="failed to connect to peer: bad00201-8c6c-3416-3fd9-72fb23a0710a"
2026-03-29T01:07:47.997Z [ERROR] server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to appendEntries to: peer="{Voter server-a2f29c39-b95d-b45b-4b03-afb148e5b60c a2f29c39-b95d-b45b-4b03-afb148e5b60c}" error="failed to connect to peer: a2f29c39-b95d-b45b-4b03-afb148e5b60c"
2026-03-29T01:07:47.999Z [ERROR] server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to appendEntries to: peer="{Voter server-bad00201-8c6c-3416-3fd9-72fb23a0710a bad00201-8c6c-3416-3fd9-72fb23a0710a}" error="failed to connect to peer: bad00201-8c6c-3416-3fd9-72fb23a0710a"
2026-03-29T01:07:48.008Z [ERROR] server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to heartbeat to: peer=a2f29c39-b95d-b45b-4b03-afb148e5b60c backoff time=10ms error="failed to connect to peer: a2f29c39-b95d-b45b-4b03-afb148e5b60c"
2026-03-29T01:07:48.015Z [ERROR] server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to appendEntries to: peer="{Voter server-a2f29c39-b95d-b45b-4b03-afb148e5b60c a2f29c39-b95d-b45b-4b03-afb148e5b60c}" error="failed to connect to peer: a2f29c39-b95d-b45b-4b03-afb148e5b60c"
2026-03-29T01:07:48.015Z [ERROR] server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to appendEntries to: peer="{Voter server-bad00201-8c6c-3416-3fd9-72fb23a0710a bad00201-8c6c-3416-3fd9-72fb23a0710a}" error="failed to connect to peer: bad00201-8c6c-3416-3fd9-72fb23a0710a"
2026-03-29T01:07:48.016Z [ERROR] server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to heartbeat to: peer=bad00201-8c6c-3416-3fd9-72fb23a0710a backoff time=10ms error="failed to connect to peer: bad00201-8c6c-3416-3fd9-72fb23a0710a"
2026-03-29T01:07:48.026Z [ERROR] server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to heartbeat to: peer=a2f29c39-b95d-b45b-4b03-afb148e5b60c backoff time=10ms error="failed to connect to peer: a2f29c39-b95d-b45b-4b03-afb148e5b60c"
2026-03-29T01:07:48.032Z [ERROR] server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to appendEntries to: peer="{Voter server-bad00201-8c6c-3416-3fd9-72fb23a0710a bad00201-8c6c-3416-3fd9-72fb23a0710a}" error="failed to connect to peer: bad00201-8c6c-3416-3fd9-72fb23a0710a"
2026-03-29T01:07:48.033Z [ERROR] server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to appendEntries to: peer="{Voter server-a2f29c39-b95d-b45b-4b03-afb148e5b60c a2f29c39-b95d-b45b-4b03-afb148e5b60c}" error="failed to connect to peer: a2f29c39-b95d-b45b-4b03-afb148e5b60c"
2026-03-29T01:07:48.034Z [ERROR] server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to heartbeat to: peer=bad00201-8c6c-3416-3fd9-72fb23a0710a backoff time=10ms error="failed to connect to peer: bad00201-8c6c-3416-3fd9-72fb23a0710a"
2026-03-29T01:07:48.035Z [WARN]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to contact: server-id=server-bad00201-8c6c-3416-3fd9-72fb23a0710a time=50.314744ms
2026-03-29T01:07:48.038Z [WARN]  server-bad00201-8c6c-3416-3fd9-72fb23a0710a: heartbeat timeout reached, starting election: last-leader-addr=21c5110c-095b-e3dd-b8c3-c0fc0b117490 last-leader-id=server-21c5110c-095b-e3dd-b8c3-c0fc0b117490
2026-03-29T01:07:48.038Z [INFO]  server-bad00201-8c6c-3416-3fd9-72fb23a0710a: entering candidate state: node="Node at bad00201-8c6c-3416-3fd9-72fb23a0710a [Candidate]" term=3
2026-03-29T01:07:48.038Z [ERROR] server-bad00201-8c6c-3416-3fd9-72fb23a0710a: failed to make requestVote RPC: target="{Voter server-714dd75b-2b44-f74e-a419-4eb54f703b36 714dd75b-2b44-f74e-a419-4eb54f703b36}" error="failed to connect to peer: 714dd75b-2b44-f74e-a419-4eb54f703b36" term=3
2026-03-29T01:07:48.039Z [WARN]  server-a2f29c39-b95d-b45b-4b03-afb148e5b60c: rejecting vote request since we have a leader: from=bad00201-8c6c-3416-3fd9-72fb23a0710a leader=21c5110c-095b-e3dd-b8c3-c0fc0b117490 leader-id=server-21c5110c-095b-e3dd-b8c3-c0fc0b117490
2026-03-29T01:07:48.038Z [ERROR] server-bad00201-8c6c-3416-3fd9-72fb23a0710a: failed to make requestVote RPC: target="{Voter server-21c5110c-095b-e3dd-b8c3-c0fc0b117490 21c5110c-095b-e3dd-b8c3-c0fc0b117490}" error="failed to connect to peer: 21c5110c-095b-e3dd-b8c3-c0fc0b117490" term=3
2026-03-29T01:07:48.043Z [ERROR] server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to heartbeat to: peer=a2f29c39-b95d-b45b-4b03-afb148e5b60c backoff time=20ms error="failed to connect to peer: a2f29c39-b95d-b45b-4b03-afb148e5b60c"
2026-03-29T01:07:48.045Z [WARN]  server-714dd75b-2b44-f74e-a419-4eb54f703b36: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2026-03-29T01:07:48.046Z [WARN]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to contact: server-id=server-bad00201-8c6c-3416-3fd9-72fb23a0710a time=61.369391ms
2026-03-29T01:07:48.046Z [WARN]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to contact: server-id=server-a2f29c39-b95d-b45b-4b03-afb148e5b60c time=59.036955ms
2026-03-29T01:07:48.046Z [WARN]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to contact quorum of nodes, stepping down
2026-03-29T01:07:48.046Z [INFO]  server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: entering follower state: follower="Node at 21c5110c-095b-e3dd-b8c3-c0fc0b117490 [Follower]" leader-address= leader-id=
2026-03-29T01:07:48.046Z [ERROR] server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to appendEntries to: peer="{Voter server-a2f29c39-b95d-b45b-4b03-afb148e5b60c a2f29c39-b95d-b45b-4b03-afb148e5b60c}" error="failed to connect to peer: a2f29c39-b95d-b45b-4b03-afb148e5b60c"
2026-03-29T01:07:48.046Z [ERROR] server-21c5110c-095b-e3dd-b8c3-c0fc0b117490: failed to appendEntries to: peer="{Voter server-bad00201-8c6c-3416-3fd9-72fb23a0710a bad00201-8c6c-3416-3fd9-72fb23a0710a}" error="failed to connect to peer: bad00201-8c6c-3416-3fd9-72fb23a0710a"
--- PASS: TestRaft_RemoveFollower_SplitCluster (0.43s)
=== RUN   TestRaft_AddKnownPeer
2026-03-29T01:07:48.047Z [INFO]  server-4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b Address:4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b} {Suffrage:Voter ID:server-737517e0-ce50-7c47-305b-80456af55639 Address:737517e0-ce50-7c47-305b-80456af55639} {Suffrage:Voter ID:server-f2123922-e4f0-9b10-46e8-4f6ce8921f73 Address:f2123922-e4f0-9b10-46e8-4f6ce8921f73}]"
2026-03-29T01:07:48.047Z [INFO]  server-4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b: entering follower state: follower="Node at 4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b [Follower]" leader-address= leader-id=
2026-03-29T01:07:48.047Z [INFO]  server-737517e0-ce50-7c47-305b-80456af55639: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b Address:4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b} {Suffrage:Voter ID:server-737517e0-ce50-7c47-305b-80456af55639 Address:737517e0-ce50-7c47-305b-80456af55639} {Suffrage:Voter ID:server-f2123922-e4f0-9b10-46e8-4f6ce8921f73 Address:f2123922-e4f0-9b10-46e8-4f6ce8921f73}]"
2026-03-29T01:07:48.047Z [INFO]  server-737517e0-ce50-7c47-305b-80456af55639: entering follower state: follower="Node at 737517e0-ce50-7c47-305b-80456af55639 [Follower]" leader-address= leader-id=
2026-03-29T01:07:48.047Z [INFO]  server-f2123922-e4f0-9b10-46e8-4f6ce8921f73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b Address:4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b} {Suffrage:Voter ID:server-737517e0-ce50-7c47-305b-80456af55639 Address:737517e0-ce50-7c47-305b-80456af55639} {Suffrage:Voter ID:server-f2123922-e4f0-9b10-46e8-4f6ce8921f73 Address:f2123922-e4f0-9b10-46e8-4f6ce8921f73}]"
2026-03-29T01:07:48.047Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:48.047Z [INFO]  server-f2123922-e4f0-9b10-46e8-4f6ce8921f73: entering follower state: follower="Node at f2123922-e4f0-9b10-46e8-4f6ce8921f73 [Follower]" leader-address= leader-id=
2026-03-29T01:07:48.100Z [WARN]  server-4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:48.100Z [INFO]  server-4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b: entering candidate state: node="Node at 4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b [Candidate]" term=2
2026-03-29T01:07:48.100Z [INFO]  server-4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b: election won: term=2 tally=2
2026-03-29T01:07:48.100Z [INFO]  server-4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b: entering leader state: leader="Node at 4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b [Leader]"
2026-03-29T01:07:48.100Z [INFO]  server-4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b: added peer, starting replication: peer=server-737517e0-ce50-7c47-305b-80456af55639
2026-03-29T01:07:48.100Z [INFO]  server-4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b: added peer, starting replication: peer=server-f2123922-e4f0-9b10-46e8-4f6ce8921f73
2026-03-29T01:07:48.100Z [INFO]  server-4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b: pipelining replication: peer="{Voter server-737517e0-ce50-7c47-305b-80456af55639 737517e0-ce50-7c47-305b-80456af55639}"
2026-03-29T01:07:48.100Z [INFO]  server-4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b: pipelining replication: peer="{Voter server-f2123922-e4f0-9b10-46e8-4f6ce8921f73 f2123922-e4f0-9b10-46e8-4f6ce8921f73}"
2026-03-29T01:07:48.206Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:48.100854279 +0000 UTC m=+11.888996812 (1 nodes), 52.926486ms from start of poll, 53.240719ms from cluster start. Timeout at 2026-03-29 01:07:48.205867509 +0000 UTC m=+11.994010086, 105.013274ms after stability
2026-03-29T01:07:48.206Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:07:48.311Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:07:48.20635592 +0000 UTC m=+11.994498581 (2 nodes), 2.232µs from start of poll, 158.742488ms from cluster start. Timeout at 2026-03-29 01:07:48.311358426 +0000 UTC m=+12.099500957, 105.002376ms after stability
2026-03-29T01:07:48.311Z [INFO]  server-4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b: updating configuration: command=AddVoter server-id=server-737517e0-ce50-7c47-305b-80456af55639 server-addr=737517e0-ce50-7c47-305b-80456af55639 servers="[{Suffrage:Voter ID:server-4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b Address:4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b} {Suffrage:Voter ID:server-737517e0-ce50-7c47-305b-80456af55639 Address:737517e0-ce50-7c47-305b-80456af55639} {Suffrage:Voter ID:server-f2123922-e4f0-9b10-46e8-4f6ce8921f73 Address:f2123922-e4f0-9b10-46e8-4f6ce8921f73}]"
2026-03-29T01:07:48.312Z [INFO]  server-4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b: aborting pipeline replication: peer="{Voter server-f2123922-e4f0-9b10-46e8-4f6ce8921f73 f2123922-e4f0-9b10-46e8-4f6ce8921f73}"
2026-03-29T01:07:48.312Z [INFO]  server-4d8bd9cd-5b90-0955-4aa4-47f2a3ede35b: aborting pipeline replication: peer="{Voter server-737517e0-ce50-7c47-305b-80456af55639 737517e0-ce50-7c47-305b-80456af55639}"
--- PASS: TestRaft_AddKnownPeer (0.27s)
=== RUN   TestRaft_RemoveUnknownPeer
2026-03-29T01:07:48.313Z [INFO]  server-8e4af2e0-ef23-119f-ddb6-cfabfd413fda: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4af2e0-ef23-119f-ddb6-cfabfd413fda Address:8e4af2e0-ef23-119f-ddb6-cfabfd413fda} {Suffrage:Voter ID:server-12e40acc-00e0-e8ac-b2d6-7eec29610b79 Address:12e40acc-00e0-e8ac-b2d6-7eec29610b79} {Suffrage:Voter ID:server-ce04abf9-ac49-7b1f-c6df-886513da6ed6 Address:ce04abf9-ac49-7b1f-c6df-886513da6ed6}]"
2026-03-29T01:07:48.313Z [INFO]  server-8e4af2e0-ef23-119f-ddb6-cfabfd413fda: entering follower state: follower="Node at 8e4af2e0-ef23-119f-ddb6-cfabfd413fda [Follower]" leader-address= leader-id=
2026-03-29T01:07:48.313Z [INFO]  server-12e40acc-00e0-e8ac-b2d6-7eec29610b79: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4af2e0-ef23-119f-ddb6-cfabfd413fda Address:8e4af2e0-ef23-119f-ddb6-cfabfd413fda} {Suffrage:Voter ID:server-12e40acc-00e0-e8ac-b2d6-7eec29610b79 Address:12e40acc-00e0-e8ac-b2d6-7eec29610b79} {Suffrage:Voter ID:server-ce04abf9-ac49-7b1f-c6df-886513da6ed6 Address:ce04abf9-ac49-7b1f-c6df-886513da6ed6}]"
2026-03-29T01:07:48.313Z [INFO]  server-12e40acc-00e0-e8ac-b2d6-7eec29610b79: entering follower state: follower="Node at 12e40acc-00e0-e8ac-b2d6-7eec29610b79 [Follower]" leader-address= leader-id=
2026-03-29T01:07:48.313Z [INFO]  server-ce04abf9-ac49-7b1f-c6df-886513da6ed6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e4af2e0-ef23-119f-ddb6-cfabfd413fda Address:8e4af2e0-ef23-119f-ddb6-cfabfd413fda} {Suffrage:Voter ID:server-12e40acc-00e0-e8ac-b2d6-7eec29610b79 Address:12e40acc-00e0-e8ac-b2d6-7eec29610b79} {Suffrage:Voter ID:server-ce04abf9-ac49-7b1f-c6df-886513da6ed6 Address:ce04abf9-ac49-7b1f-c6df-886513da6ed6}]"
2026-03-29T01:07:48.313Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:48.313Z [INFO]  server-ce04abf9-ac49-7b1f-c6df-886513da6ed6: entering follower state: follower="Node at ce04abf9-ac49-7b1f-c6df-886513da6ed6 [Follower]" leader-address= leader-id=
2026-03-29T01:07:48.380Z [WARN]  server-ce04abf9-ac49-7b1f-c6df-886513da6ed6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:48.380Z [INFO]  server-ce04abf9-ac49-7b1f-c6df-886513da6ed6: entering candidate state: node="Node at ce04abf9-ac49-7b1f-c6df-886513da6ed6 [Candidate]" term=2
2026-03-29T01:07:48.380Z [INFO]  server-ce04abf9-ac49-7b1f-c6df-886513da6ed6: election won: term=2 tally=2
2026-03-29T01:07:48.381Z [INFO]  server-ce04abf9-ac49-7b1f-c6df-886513da6ed6: entering leader state: leader="Node at ce04abf9-ac49-7b1f-c6df-886513da6ed6 [Leader]"
2026-03-29T01:07:48.381Z [INFO]  server-ce04abf9-ac49-7b1f-c6df-886513da6ed6: added peer, starting replication: peer=server-8e4af2e0-ef23-119f-ddb6-cfabfd413fda
2026-03-29T01:07:48.381Z [INFO]  server-ce04abf9-ac49-7b1f-c6df-886513da6ed6: added peer, starting replication: peer=server-12e40acc-00e0-e8ac-b2d6-7eec29610b79
2026-03-29T01:07:48.381Z [INFO]  server-ce04abf9-ac49-7b1f-c6df-886513da6ed6: pipelining replication: peer="{Voter server-8e4af2e0-ef23-119f-ddb6-cfabfd413fda 8e4af2e0-ef23-119f-ddb6-cfabfd413fda}"
2026-03-29T01:07:48.381Z [INFO]  server-ce04abf9-ac49-7b1f-c6df-886513da6ed6: pipelining replication: peer="{Voter server-12e40acc-00e0-e8ac-b2d6-7eec29610b79 12e40acc-00e0-e8ac-b2d6-7eec29610b79}"
2026-03-29T01:07:48.486Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:48.381045347 +0000 UTC m=+12.169187881 (1 nodes), 67.663216ms from start of poll, 67.944339ms from cluster start. Timeout at 2026-03-29 01:07:48.486046989 +0000 UTC m=+12.274189599, 105.001718ms after stability
2026-03-29T01:07:48.486Z [INFO]  server-ce04abf9-ac49-7b1f-c6df-886513da6ed6: updating configuration: command=RemoveServer server-id=fa9508a8-9bb3-ca6d-66e5-8e25bfa051f4 server-addr= servers="[{Suffrage:Voter ID:server-8e4af2e0-ef23-119f-ddb6-cfabfd413fda Address:8e4af2e0-ef23-119f-ddb6-cfabfd413fda} {Suffrage:Voter ID:server-12e40acc-00e0-e8ac-b2d6-7eec29610b79 Address:12e40acc-00e0-e8ac-b2d6-7eec29610b79} {Suffrage:Voter ID:server-ce04abf9-ac49-7b1f-c6df-886513da6ed6 Address:ce04abf9-ac49-7b1f-c6df-886513da6ed6}]"
2026-03-29T01:07:48.487Z [INFO]  server-ce04abf9-ac49-7b1f-c6df-886513da6ed6: aborting pipeline replication: peer="{Voter server-12e40acc-00e0-e8ac-b2d6-7eec29610b79 12e40acc-00e0-e8ac-b2d6-7eec29610b79}"
2026-03-29T01:07:48.487Z [INFO]  server-ce04abf9-ac49-7b1f-c6df-886513da6ed6: aborting pipeline replication: peer="{Voter server-8e4af2e0-ef23-119f-ddb6-cfabfd413fda 8e4af2e0-ef23-119f-ddb6-cfabfd413fda}"
--- PASS: TestRaft_RemoveUnknownPeer (0.17s)
=== RUN   TestRaft_SnapshotRestore
2026-03-29T01:07:48.487Z [INFO]  server-15eaa5df-9311-5723-7698-3aae84c6130e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15eaa5df-9311-5723-7698-3aae84c6130e Address:15eaa5df-9311-5723-7698-3aae84c6130e}]"
2026-03-29T01:07:48.487Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:48.487Z [INFO]  server-15eaa5df-9311-5723-7698-3aae84c6130e: entering follower state: follower="Node at 15eaa5df-9311-5723-7698-3aae84c6130e [Follower]" leader-address= leader-id=
2026-03-29T01:07:48.566Z [WARN]  server-15eaa5df-9311-5723-7698-3aae84c6130e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:48.566Z [INFO]  server-15eaa5df-9311-5723-7698-3aae84c6130e: entering candidate state: node="Node at 15eaa5df-9311-5723-7698-3aae84c6130e [Candidate]" term=2
2026-03-29T01:07:48.566Z [INFO]  server-15eaa5df-9311-5723-7698-3aae84c6130e: election won: term=2 tally=1
2026-03-29T01:07:48.566Z [INFO]  server-15eaa5df-9311-5723-7698-3aae84c6130e: entering leader state: leader="Node at 15eaa5df-9311-5723-7698-3aae84c6130e [Leader]"
2026-03-29T01:07:48.672Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:48.566874979 +0000 UTC m=+12.355017580 (1 nodes), 79.112415ms from start of poll, 79.215505ms from cluster start. Timeout at 2026-03-29 01:07:48.671880933 +0000 UTC m=+12.460023517, 105.005937ms after stability
2026-03-29T01:07:48.673Z [INFO]  server-15eaa5df-9311-5723-7698-3aae84c6130e: starting snapshot up to: index=102
2026-03-29T01:07:48.673Z [INFO]  creating new snapshot: path=/tmp/raft2092552336/snapshots/2-102-1774746468673.tmp
2026-03-29T01:07:48.674Z [INFO]  server-15eaa5df-9311-5723-7698-3aae84c6130e: compacting logs: from=1 to=92
2026-03-29T01:07:48.674Z [INFO]  server-15eaa5df-9311-5723-7698-3aae84c6130e: snapshot complete up to: index=102
2026-03-29T01:07:48.674Z [INFO]  server-15eaa5df-9311-5723-7698-3aae84c6130e: starting restore from snapshot: id=2-102-1774746468673 last-index=102 last-term=2 size-in-bytes=693
2026-03-29T01:07:48.674Z [INFO]  server-15eaa5df-9311-5723-7698-3aae84c6130e: snapshot restore progress: id=2-102-1774746468673 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-03-29T01:07:48.674Z [INFO]  server-15eaa5df-9311-5723-7698-3aae84c6130e: restored from snapshot: id=2-102-1774746468673 last-index=102 last-term=2 size-in-bytes=693
2026-03-29T01:07:48.674Z [INFO]  server-15eaa5df-9311-5723-7698-3aae84c6130e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15eaa5df-9311-5723-7698-3aae84c6130e Address:15eaa5df-9311-5723-7698-3aae84c6130e}]"
--- PASS: TestRaft_SnapshotRestore (0.19s)
=== RUN   TestRaft_RestoreSnapshotOnStartup_Monotonic
2026-03-29T01:07:48.677Z [INFO]  server-d2404f44-cd8a-3dfe-cf2f-9a9df033bab5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2404f44-cd8a-3dfe-cf2f-9a9df033bab5 Address:d2404f44-cd8a-3dfe-cf2f-9a9df033bab5}]"
2026-03-29T01:07:48.677Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:48.677Z [INFO]  server-d2404f44-cd8a-3dfe-cf2f-9a9df033bab5: entering follower state: follower="Node at d2404f44-cd8a-3dfe-cf2f-9a9df033bab5 [Follower]" leader-address= leader-id=
2026-03-29T01:07:48.758Z [WARN]  server-d2404f44-cd8a-3dfe-cf2f-9a9df033bab5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:48.759Z [INFO]  server-d2404f44-cd8a-3dfe-cf2f-9a9df033bab5: entering candidate state: node="Node at d2404f44-cd8a-3dfe-cf2f-9a9df033bab5 [Candidate]" term=2
2026-03-29T01:07:48.759Z [INFO]  server-d2404f44-cd8a-3dfe-cf2f-9a9df033bab5: election won: term=2 tally=1
2026-03-29T01:07:48.759Z [INFO]  server-d2404f44-cd8a-3dfe-cf2f-9a9df033bab5: entering leader state: leader="Node at d2404f44-cd8a-3dfe-cf2f-9a9df033bab5 [Leader]"
2026-03-29T01:07:48.864Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:48.759094754 +0000 UTC m=+12.547237286 (1 nodes), 81.36991ms from start of poll, 81.471307ms from cluster start. Timeout at 2026-03-29 01:07:48.864096246 +0000 UTC m=+12.652238843, 105.001557ms after stability
2026-03-29T01:07:48.866Z [INFO]  server-d2404f44-cd8a-3dfe-cf2f-9a9df033bab5: starting snapshot up to: index=102
2026-03-29T01:07:48.866Z [INFO]  creating new snapshot: path=/tmp/raft2386829977/snapshots/2-102-1774746468866.tmp
2026-03-29T01:07:48.866Z [INFO]  server-d2404f44-cd8a-3dfe-cf2f-9a9df033bab5: compacting logs: from=1 to=92
2026-03-29T01:07:48.866Z [INFO]  server-d2404f44-cd8a-3dfe-cf2f-9a9df033bab5: snapshot complete up to: index=102
2026-03-29T01:07:48.867Z [INFO]  server-d2404f44-cd8a-3dfe-cf2f-9a9df033bab5: starting restore from snapshot: id=2-102-1774746468866 last-index=102 last-term=2 size-in-bytes=693
2026-03-29T01:07:48.867Z [INFO]  server-d2404f44-cd8a-3dfe-cf2f-9a9df033bab5: snapshot restore progress: id=2-102-1774746468866 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-03-29T01:07:48.867Z [INFO]  server-d2404f44-cd8a-3dfe-cf2f-9a9df033bab5: restored from snapshot: id=2-102-1774746468866 last-index=102 last-term=2 size-in-bytes=693
2026-03-29T01:07:48.867Z [INFO]  server-d2404f44-cd8a-3dfe-cf2f-9a9df033bab5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2404f44-cd8a-3dfe-cf2f-9a9df033bab5 Address:d2404f44-cd8a-3dfe-cf2f-9a9df033bab5}]"
--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.19s)
=== RUN   TestRaft_SnapshotRestore_Progress
2026-03-29T01:07:48.867Z [INFO]  server-914f9fbb-f4d7-bdc2-8c60-93a3bccee5cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-914f9fbb-f4d7-bdc2-8c60-93a3bccee5cf Address:914f9fbb-f4d7-bdc2-8c60-93a3bccee5cf}]"
2026-03-29T01:07:48.867Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:48.867Z [INFO]  server-914f9fbb-f4d7-bdc2-8c60-93a3bccee5cf: entering follower state: follower="Node at 914f9fbb-f4d7-bdc2-8c60-93a3bccee5cf [Follower]" leader-address= leader-id=
2026-03-29T01:07:48.925Z [WARN]  server-914f9fbb-f4d7-bdc2-8c60-93a3bccee5cf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:48.925Z [INFO]  server-914f9fbb-f4d7-bdc2-8c60-93a3bccee5cf: entering candidate state: node="Node at 914f9fbb-f4d7-bdc2-8c60-93a3bccee5cf [Candidate]" term=2
2026-03-29T01:07:48.926Z [INFO]  server-914f9fbb-f4d7-bdc2-8c60-93a3bccee5cf: election won: term=2 tally=1
2026-03-29T01:07:48.926Z [INFO]  server-914f9fbb-f4d7-bdc2-8c60-93a3bccee5cf: entering leader state: leader="Node at 914f9fbb-f4d7-bdc2-8c60-93a3bccee5cf [Leader]"
2026-03-29T01:07:49.031Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:48.926045787 +0000 UTC m=+12.714188325 (1 nodes), 58.24058ms from start of poll, 58.349223ms from cluster start. Timeout at 2026-03-29 01:07:49.031048737 +0000 UTC m=+12.819191267, 105.002942ms after stability
2026-03-29T01:07:49.033Z [INFO]  server-914f9fbb-f4d7-bdc2-8c60-93a3bccee5cf: starting snapshot up to: index=102
2026-03-29T01:07:49.033Z [INFO]  creating new snapshot: path=/tmp/raft4080153570/snapshots/2-102-1774746469033.tmp
2026-03-29T01:07:49.033Z [INFO]  server-914f9fbb-f4d7-bdc2-8c60-93a3bccee5cf: compacting logs: from=1 to=92
2026-03-29T01:07:49.033Z [INFO]  server-914f9fbb-f4d7-bdc2-8c60-93a3bccee5cf: snapshot complete up to: index=102
--- PASS: TestRaft_SnapshotRestore_Progress (0.17s)
=== RUN   TestRaft_NoRestoreOnStart
2026-03-29T01:07:49.034Z [INFO]  server-e88aeb0b-0029-fcef-030d-0ea94d4c422c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e88aeb0b-0029-fcef-030d-0ea94d4c422c Address:e88aeb0b-0029-fcef-030d-0ea94d4c422c}]"
2026-03-29T01:07:49.034Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:49.034Z [INFO]  server-e88aeb0b-0029-fcef-030d-0ea94d4c422c: entering follower state: follower="Node at e88aeb0b-0029-fcef-030d-0ea94d4c422c [Follower]" leader-address= leader-id=
2026-03-29T01:07:49.117Z [WARN]  server-e88aeb0b-0029-fcef-030d-0ea94d4c422c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:49.117Z [INFO]  server-e88aeb0b-0029-fcef-030d-0ea94d4c422c: entering candidate state: node="Node at e88aeb0b-0029-fcef-030d-0ea94d4c422c [Candidate]" term=2
2026-03-29T01:07:49.117Z [INFO]  server-e88aeb0b-0029-fcef-030d-0ea94d4c422c: election won: term=2 tally=1
2026-03-29T01:07:49.117Z [INFO]  server-e88aeb0b-0029-fcef-030d-0ea94d4c422c: entering leader state: leader="Node at e88aeb0b-0029-fcef-030d-0ea94d4c422c [Leader]"
2026-03-29T01:07:49.222Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:49.117451025 +0000 UTC m=+12.905593568 (1 nodes), 82.784898ms from start of poll, 82.88962ms from cluster start. Timeout at 2026-03-29 01:07:49.222458941 +0000 UTC m=+13.010601528, 105.00796ms after stability
2026-03-29T01:07:49.224Z [INFO]  server-e88aeb0b-0029-fcef-030d-0ea94d4c422c: starting snapshot up to: index=102
2026-03-29T01:07:49.224Z [INFO]  creating new snapshot: path=/tmp/raft295442555/snapshots/2-102-1774746469224.tmp
2026-03-29T01:07:49.225Z [INFO]  server-e88aeb0b-0029-fcef-030d-0ea94d4c422c: compacting logs: from=1 to=92
2026-03-29T01:07:49.225Z [INFO]  server-e88aeb0b-0029-fcef-030d-0ea94d4c422c: snapshot complete up to: index=102
2026-03-29T01:07:49.225Z [INFO]  server-e88aeb0b-0029-fcef-030d-0ea94d4c422c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e88aeb0b-0029-fcef-030d-0ea94d4c422c Address:e88aeb0b-0029-fcef-030d-0ea94d4c422c}]"
--- PASS: TestRaft_NoRestoreOnStart (0.19s)
=== RUN   TestRaft_SnapshotRestore_PeerChange
2026-03-29T01:07:49.225Z [INFO]  server-e88aeb0b-0029-fcef-030d-0ea94d4c422c: entering follower state: follower="Node at e88aeb0b-0029-fcef-030d-0ea94d4c422c [Follower]" leader-address= leader-id=
    raft_test.go:1266: Skipping unreliable test
--- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
=== RUN   TestRaft_AutoSnapshot
2026-03-29T01:07:49.226Z [INFO]  server-5f9261a6-6a46-19ca-39f4-e6ec23d35415: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f9261a6-6a46-19ca-39f4-e6ec23d35415 Address:5f9261a6-6a46-19ca-39f4-e6ec23d35415}]"
2026-03-29T01:07:49.226Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:49.226Z [INFO]  server-5f9261a6-6a46-19ca-39f4-e6ec23d35415: entering follower state: follower="Node at 5f9261a6-6a46-19ca-39f4-e6ec23d35415 [Follower]" leader-address= leader-id=
2026-03-29T01:07:49.279Z [WARN]  server-5f9261a6-6a46-19ca-39f4-e6ec23d35415: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:49.279Z [INFO]  server-5f9261a6-6a46-19ca-39f4-e6ec23d35415: entering candidate state: node="Node at 5f9261a6-6a46-19ca-39f4-e6ec23d35415 [Candidate]" term=2
2026-03-29T01:07:49.279Z [INFO]  server-5f9261a6-6a46-19ca-39f4-e6ec23d35415: election won: term=2 tally=1
2026-03-29T01:07:49.279Z [INFO]  server-5f9261a6-6a46-19ca-39f4-e6ec23d35415: entering leader state: leader="Node at 5f9261a6-6a46-19ca-39f4-e6ec23d35415 [Leader]"
2026-03-29T01:07:49.298Z [WARN]  server-e88aeb0b-0029-fcef-030d-0ea94d4c422c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:49.298Z [INFO]  server-e88aeb0b-0029-fcef-030d-0ea94d4c422c: entering candidate state: node="Node at e88aeb0b-0029-fcef-030d-0ea94d4c422c [Candidate]" term=3
2026-03-29T01:07:49.298Z [INFO]  server-e88aeb0b-0029-fcef-030d-0ea94d4c422c: election won: term=3 tally=1
2026-03-29T01:07:49.298Z [INFO]  server-e88aeb0b-0029-fcef-030d-0ea94d4c422c: entering leader state: leader="Node at e88aeb0b-0029-fcef-030d-0ea94d4c422c [Leader]"
2026-03-29T01:07:49.385Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:49.279590486 +0000 UTC m=+13.067733033 (1 nodes), 53.270419ms from start of poll, 53.374695ms from cluster start. Timeout at 2026-03-29 01:07:49.384598194 +0000 UTC m=+13.172740781, 105.007748ms after stability
2026-03-29T01:07:49.395Z [INFO]  server-5f9261a6-6a46-19ca-39f4-e6ec23d35415: starting snapshot up to: index=102
2026-03-29T01:07:49.395Z [INFO]  creating new snapshot: path=/tmp/raft3348027626/snapshots/2-102-1774746469395.tmp
2026-03-29T01:07:49.396Z [INFO]  server-5f9261a6-6a46-19ca-39f4-e6ec23d35415: compacting logs: from=1 to=92
2026-03-29T01:07:49.396Z [INFO]  server-5f9261a6-6a46-19ca-39f4-e6ec23d35415: snapshot complete up to: index=102
--- PASS: TestRaft_AutoSnapshot (0.27s)
=== RUN   TestRaft_UserSnapshot
2026-03-29T01:07:49.493Z [INFO]  server-86cea381-0dc3-291f-0b35-1400311e8120: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86cea381-0dc3-291f-0b35-1400311e8120 Address:86cea381-0dc3-291f-0b35-1400311e8120}]"
2026-03-29T01:07:49.493Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:49.493Z [INFO]  server-86cea381-0dc3-291f-0b35-1400311e8120: entering follower state: follower="Node at 86cea381-0dc3-291f-0b35-1400311e8120 [Follower]" leader-address= leader-id=
2026-03-29T01:07:49.579Z [WARN]  server-86cea381-0dc3-291f-0b35-1400311e8120: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:49.579Z [INFO]  server-86cea381-0dc3-291f-0b35-1400311e8120: entering candidate state: node="Node at 86cea381-0dc3-291f-0b35-1400311e8120 [Candidate]" term=2
2026-03-29T01:07:49.579Z [INFO]  server-86cea381-0dc3-291f-0b35-1400311e8120: election won: term=2 tally=1
2026-03-29T01:07:49.579Z [INFO]  server-86cea381-0dc3-291f-0b35-1400311e8120: entering leader state: leader="Node at 86cea381-0dc3-291f-0b35-1400311e8120 [Leader]"
2026-03-29T01:07:49.685Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:49.579586121 +0000 UTC m=+13.367728662 (1 nodes), 85.740979ms from start of poll, 85.846313ms from cluster start. Timeout at 2026-03-29 01:07:49.684592273 +0000 UTC m=+13.472734870, 105.006208ms after stability
2026-03-29T01:07:49.685Z [ERROR] server-86cea381-0dc3-291f-0b35-1400311e8120: failed to take snapshot: error="nothing new to snapshot"
2026-03-29T01:07:49.685Z [INFO]  server-86cea381-0dc3-291f-0b35-1400311e8120: starting snapshot up to: index=12
2026-03-29T01:07:49.685Z [INFO]  creating new snapshot: path=/tmp/raft4084425874/snapshots/2-12-1774746469685.tmp
2026-03-29T01:07:49.686Z [INFO]  server-86cea381-0dc3-291f-0b35-1400311e8120: compacting logs: from=1 to=2
2026-03-29T01:07:49.686Z [INFO]  server-86cea381-0dc3-291f-0b35-1400311e8120: snapshot complete up to: index=12
--- PASS: TestRaft_UserSnapshot (0.19s)
=== RUN   TestRaft_UserRestore
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false
2026-03-29T01:07:49.687Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4b8fda5-d550-5f26-c470-a1a2d912b535 Address:a4b8fda5-d550-5f26-c470-a1a2d912b535} {Suffrage:Voter ID:server-57ebaef0-5453-fc16-2e0c-22110dff9f2d Address:57ebaef0-5453-fc16-2e0c-22110dff9f2d} {Suffrage:Voter ID:server-0a125094-473f-7117-e2ba-284361aacdf9 Address:0a125094-473f-7117-e2ba-284361aacdf9}]"
2026-03-29T01:07:49.687Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: entering follower state: follower="Node at a4b8fda5-d550-5f26-c470-a1a2d912b535 [Follower]" leader-address= leader-id=
2026-03-29T01:07:49.687Z [INFO]  server-57ebaef0-5453-fc16-2e0c-22110dff9f2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4b8fda5-d550-5f26-c470-a1a2d912b535 Address:a4b8fda5-d550-5f26-c470-a1a2d912b535} {Suffrage:Voter ID:server-57ebaef0-5453-fc16-2e0c-22110dff9f2d Address:57ebaef0-5453-fc16-2e0c-22110dff9f2d} {Suffrage:Voter ID:server-0a125094-473f-7117-e2ba-284361aacdf9 Address:0a125094-473f-7117-e2ba-284361aacdf9}]"
2026-03-29T01:07:49.687Z [INFO]  server-57ebaef0-5453-fc16-2e0c-22110dff9f2d: entering follower state: follower="Node at 57ebaef0-5453-fc16-2e0c-22110dff9f2d [Follower]" leader-address= leader-id=
2026-03-29T01:07:49.687Z [INFO]  server-0a125094-473f-7117-e2ba-284361aacdf9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a4b8fda5-d550-5f26-c470-a1a2d912b535 Address:a4b8fda5-d550-5f26-c470-a1a2d912b535} {Suffrage:Voter ID:server-57ebaef0-5453-fc16-2e0c-22110dff9f2d Address:57ebaef0-5453-fc16-2e0c-22110dff9f2d} {Suffrage:Voter ID:server-0a125094-473f-7117-e2ba-284361aacdf9 Address:0a125094-473f-7117-e2ba-284361aacdf9}]"
2026-03-29T01:07:49.687Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:49.687Z [INFO]  server-0a125094-473f-7117-e2ba-284361aacdf9: entering follower state: follower="Node at 0a125094-473f-7117-e2ba-284361aacdf9 [Follower]" leader-address= leader-id=
2026-03-29T01:07:50.374Z [WARN]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:50.374Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: entering candidate state: node="Node at a4b8fda5-d550-5f26-c470-a1a2d912b535 [Candidate]" term=2
2026-03-29T01:07:50.374Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: election won: term=2 tally=2
2026-03-29T01:07:50.374Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: entering leader state: leader="Node at a4b8fda5-d550-5f26-c470-a1a2d912b535 [Leader]"
2026-03-29T01:07:50.374Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: added peer, starting replication: peer=server-57ebaef0-5453-fc16-2e0c-22110dff9f2d
2026-03-29T01:07:50.374Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: added peer, starting replication: peer=server-0a125094-473f-7117-e2ba-284361aacdf9
2026-03-29T01:07:50.374Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: pipelining replication: peer="{Voter server-57ebaef0-5453-fc16-2e0c-22110dff9f2d 57ebaef0-5453-fc16-2e0c-22110dff9f2d}"
2026-03-29T01:07:50.374Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: pipelining replication: peer="{Voter server-0a125094-473f-7117-e2ba-284361aacdf9 0a125094-473f-7117-e2ba-284361aacdf9}"
2026-03-29T01:07:51.379Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:50.374572676 +0000 UTC m=+14.162715215 (1 nodes), 686.907198ms from start of poll, 687.252985ms from cluster start. Timeout at 2026-03-29 01:07:51.379575186 +0000 UTC m=+15.167717722, 1.005002507s after stability
2026-03-29T01:07:51.380Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: starting snapshot up to: index=12
2026-03-29T01:07:51.380Z [INFO]  creating new snapshot: path=/tmp/raft2921670204/snapshots/2-12-1774746471380.tmp
2026-03-29T01:07:51.380Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: snapshot complete up to: index=12
2026-03-29T01:07:51.381Z [INFO]  creating new snapshot: path=/tmp/raft2921670204/snapshots/2-23-1774746471381.tmp
2026-03-29T01:07:51.382Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: copied to local snapshot: bytes=71
2026-03-29T01:07:51.382Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: snapshot restore progress: id=2-23-1774746471381 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:07:51.382Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: restored user snapshot: index=23
2026-03-29T01:07:51.382Z [ERROR] server-a4b8fda5-d550-5f26-c470-a1a2d912b535: failed to get log: index=23 error="log not found"
2026-03-29T01:07:51.382Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: aborting pipeline replication: peer="{Voter server-0a125094-473f-7117-e2ba-284361aacdf9 0a125094-473f-7117-e2ba-284361aacdf9}"
2026-03-29T01:07:51.382Z [ERROR] server-a4b8fda5-d550-5f26-c470-a1a2d912b535: failed to get log: index=23 error="log not found"
2026-03-29T01:07:51.383Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: aborting pipeline replication: peer="{Voter server-57ebaef0-5453-fc16-2e0c-22110dff9f2d 57ebaef0-5453-fc16-2e0c-22110dff9f2d}"
2026-03-29T01:07:51.389Z [ERROR] server-a4b8fda5-d550-5f26-c470-a1a2d912b535: failed to get log: index=23 error="log not found"
2026-03-29T01:07:51.390Z [INFO]  creating new snapshot: path=/tmp/raft2398012014/snapshots/2-23-1774746471390.tmp
2026-03-29T01:07:51.390Z [INFO]  server-0a125094-473f-7117-e2ba-284361aacdf9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:07:51.390Z [INFO]  server-0a125094-473f-7117-e2ba-284361aacdf9: copied to local snapshot: bytes=71
2026-03-29T01:07:51.391Z [INFO]  server-0a125094-473f-7117-e2ba-284361aacdf9: snapshot restore progress: id=2-23-1774746471390 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:07:51.391Z [INFO]  server-0a125094-473f-7117-e2ba-284361aacdf9: Installed remote snapshot
2026-03-29T01:07:51.391Z [WARN]  server-0a125094-473f-7117-e2ba-284361aacdf9: skipping application of old log: index=22
2026-03-29T01:07:51.391Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: pipelining replication: peer="{Voter server-0a125094-473f-7117-e2ba-284361aacdf9 0a125094-473f-7117-e2ba-284361aacdf9}"
2026-03-29T01:07:51.391Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:07:51.392Z [ERROR] server-a4b8fda5-d550-5f26-c470-a1a2d912b535: failed to get log: index=23 error="log not found"
2026-03-29T01:07:51.392Z [INFO]  creating new snapshot: path=/tmp/raft2111683346/snapshots/2-23-1774746471392.tmp
2026-03-29T01:07:51.392Z [INFO]  server-57ebaef0-5453-fc16-2e0c-22110dff9f2d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:07:51.392Z [INFO]  server-57ebaef0-5453-fc16-2e0c-22110dff9f2d: copied to local snapshot: bytes=71
2026-03-29T01:07:51.392Z [INFO]  server-57ebaef0-5453-fc16-2e0c-22110dff9f2d: snapshot restore progress: id=2-23-1774746471392 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:07:51.393Z [INFO]  server-57ebaef0-5453-fc16-2e0c-22110dff9f2d: Installed remote snapshot
2026-03-29T01:07:51.393Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: pipelining replication: peer="{Voter server-57ebaef0-5453-fc16-2e0c-22110dff9f2d 57ebaef0-5453-fc16-2e0c-22110dff9f2d}"
2026-03-29T01:07:52.396Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:07:51.391330159 +0000 UTC m=+15.179472707 (2 nodes), 866ns from start of poll, 1.704010477s from cluster start. Timeout at 2026-03-29 01:07:52.396342693 +0000 UTC m=+16.184485363, 1.005012656s after stability
2026-03-29T01:07:52.408Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: aborting pipeline replication: peer="{Voter server-0a125094-473f-7117-e2ba-284361aacdf9 0a125094-473f-7117-e2ba-284361aacdf9}"
2026-03-29T01:07:52.408Z [INFO]  server-a4b8fda5-d550-5f26-c470-a1a2d912b535: aborting pipeline replication: peer="{Voter server-57ebaef0-5453-fc16-2e0c-22110dff9f2d 57ebaef0-5453-fc16-2e0c-22110dff9f2d}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false
2026-03-29T01:07:52.409Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-000342f2-4718-37c2-7de6-0d0ff98273e8 Address:000342f2-4718-37c2-7de6-0d0ff98273e8} {Suffrage:Voter ID:server-8c88bb74-71f0-b4d7-73d6-e0acfe406c88 Address:8c88bb74-71f0-b4d7-73d6-e0acfe406c88} {Suffrage:Voter ID:server-1e7c3462-e790-bd51-0768-b36b9785a4f4 Address:1e7c3462-e790-bd51-0768-b36b9785a4f4}]"
2026-03-29T01:07:52.409Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: entering follower state: follower="Node at 000342f2-4718-37c2-7de6-0d0ff98273e8 [Follower]" leader-address= leader-id=
2026-03-29T01:07:52.409Z [INFO]  server-8c88bb74-71f0-b4d7-73d6-e0acfe406c88: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-000342f2-4718-37c2-7de6-0d0ff98273e8 Address:000342f2-4718-37c2-7de6-0d0ff98273e8} {Suffrage:Voter ID:server-8c88bb74-71f0-b4d7-73d6-e0acfe406c88 Address:8c88bb74-71f0-b4d7-73d6-e0acfe406c88} {Suffrage:Voter ID:server-1e7c3462-e790-bd51-0768-b36b9785a4f4 Address:1e7c3462-e790-bd51-0768-b36b9785a4f4}]"
2026-03-29T01:07:52.409Z [INFO]  server-8c88bb74-71f0-b4d7-73d6-e0acfe406c88: entering follower state: follower="Node at 8c88bb74-71f0-b4d7-73d6-e0acfe406c88 [Follower]" leader-address= leader-id=
2026-03-29T01:07:52.410Z [INFO]  server-1e7c3462-e790-bd51-0768-b36b9785a4f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-000342f2-4718-37c2-7de6-0d0ff98273e8 Address:000342f2-4718-37c2-7de6-0d0ff98273e8} {Suffrage:Voter ID:server-8c88bb74-71f0-b4d7-73d6-e0acfe406c88 Address:8c88bb74-71f0-b4d7-73d6-e0acfe406c88} {Suffrage:Voter ID:server-1e7c3462-e790-bd51-0768-b36b9785a4f4 Address:1e7c3462-e790-bd51-0768-b36b9785a4f4}]"
2026-03-29T01:07:52.410Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:52.410Z [INFO]  server-1e7c3462-e790-bd51-0768-b36b9785a4f4: entering follower state: follower="Node at 1e7c3462-e790-bd51-0768-b36b9785a4f4 [Follower]" leader-address= leader-id=
2026-03-29T01:07:53.057Z [WARN]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:53.057Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: entering candidate state: node="Node at 000342f2-4718-37c2-7de6-0d0ff98273e8 [Candidate]" term=2
2026-03-29T01:07:53.057Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: election won: term=2 tally=2
2026-03-29T01:07:53.057Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: entering leader state: leader="Node at 000342f2-4718-37c2-7de6-0d0ff98273e8 [Leader]"
2026-03-29T01:07:53.057Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: added peer, starting replication: peer=server-8c88bb74-71f0-b4d7-73d6-e0acfe406c88
2026-03-29T01:07:53.057Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: added peer, starting replication: peer=server-1e7c3462-e790-bd51-0768-b36b9785a4f4
2026-03-29T01:07:53.058Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: pipelining replication: peer="{Voter server-8c88bb74-71f0-b4d7-73d6-e0acfe406c88 8c88bb74-71f0-b4d7-73d6-e0acfe406c88}"
2026-03-29T01:07:53.058Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: pipelining replication: peer="{Voter server-1e7c3462-e790-bd51-0768-b36b9785a4f4 1e7c3462-e790-bd51-0768-b36b9785a4f4}"
2026-03-29T01:07:54.063Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:53.057977714 +0000 UTC m=+16.846120421 (1 nodes), 647.584967ms from start of poll, 648.665257ms from cluster start. Timeout at 2026-03-29 01:07:54.062979545 +0000 UTC m=+17.851122083, 1.005001662s after stability
2026-03-29T01:07:54.064Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: starting snapshot up to: index=12
2026-03-29T01:07:54.064Z [INFO]  creating new snapshot: path=/tmp/raft170363790/snapshots/2-12-1774746474064.tmp
2026-03-29T01:07:54.065Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: snapshot complete up to: index=12
2026-03-29T01:07:54.065Z [INFO]  creating new snapshot: path=/tmp/raft170363790/snapshots/2-23-1774746474065.tmp
2026-03-29T01:07:54.065Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: copied to local snapshot: bytes=71
2026-03-29T01:07:54.066Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: snapshot restore progress: id=2-23-1774746474065 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:07:54.066Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: removing all old logs from log store
2026-03-29T01:07:54.066Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: compacting logs: from=1 to=22
2026-03-29T01:07:54.066Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: restored user snapshot: index=23
2026-03-29T01:07:54.066Z [ERROR] server-000342f2-4718-37c2-7de6-0d0ff98273e8: failed to get log: index=22 error="log not found"
2026-03-29T01:07:54.066Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: aborting pipeline replication: peer="{Voter server-8c88bb74-71f0-b4d7-73d6-e0acfe406c88 8c88bb74-71f0-b4d7-73d6-e0acfe406c88}"
2026-03-29T01:07:54.066Z [ERROR] server-000342f2-4718-37c2-7de6-0d0ff98273e8: failed to get log: index=22 error="log not found"
2026-03-29T01:07:54.066Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: aborting pipeline replication: peer="{Voter server-1e7c3462-e790-bd51-0768-b36b9785a4f4 1e7c3462-e790-bd51-0768-b36b9785a4f4}"
2026-03-29T01:07:54.073Z [ERROR] server-000342f2-4718-37c2-7de6-0d0ff98273e8: failed to get log: index=22 error="log not found"
2026-03-29T01:07:54.073Z [INFO]  creating new snapshot: path=/tmp/raft3356263162/snapshots/2-23-1774746474073.tmp
2026-03-29T01:07:54.074Z [INFO]  server-1e7c3462-e790-bd51-0768-b36b9785a4f4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:07:54.074Z [INFO]  server-1e7c3462-e790-bd51-0768-b36b9785a4f4: copied to local snapshot: bytes=71
2026-03-29T01:07:54.074Z [INFO]  server-1e7c3462-e790-bd51-0768-b36b9785a4f4: snapshot restore progress: id=2-23-1774746474073 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:07:54.074Z [INFO]  server-1e7c3462-e790-bd51-0768-b36b9785a4f4: removing all old logs from log store
2026-03-29T01:07:54.074Z [INFO]  server-1e7c3462-e790-bd51-0768-b36b9785a4f4: compacting logs: from=1 to=22
2026-03-29T01:07:54.074Z [INFO]  server-1e7c3462-e790-bd51-0768-b36b9785a4f4: Installed remote snapshot
2026-03-29T01:07:54.074Z [WARN]  server-1e7c3462-e790-bd51-0768-b36b9785a4f4: skipping application of old log: index=22
2026-03-29T01:07:54.074Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: pipelining replication: peer="{Voter server-1e7c3462-e790-bd51-0768-b36b9785a4f4 1e7c3462-e790-bd51-0768-b36b9785a4f4}"
2026-03-29T01:07:54.075Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:07:54.075Z [ERROR] server-000342f2-4718-37c2-7de6-0d0ff98273e8: failed to get log: index=22 error="log not found"
2026-03-29T01:07:54.076Z [INFO]  creating new snapshot: path=/tmp/raft1974563103/snapshots/2-23-1774746474076.tmp
2026-03-29T01:07:54.076Z [INFO]  server-8c88bb74-71f0-b4d7-73d6-e0acfe406c88: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:07:54.076Z [INFO]  server-8c88bb74-71f0-b4d7-73d6-e0acfe406c88: copied to local snapshot: bytes=71
2026-03-29T01:07:54.077Z [INFO]  server-8c88bb74-71f0-b4d7-73d6-e0acfe406c88: snapshot restore progress: id=2-23-1774746474076 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:07:54.077Z [INFO]  server-8c88bb74-71f0-b4d7-73d6-e0acfe406c88: removing all old logs from log store
2026-03-29T01:07:54.077Z [INFO]  server-8c88bb74-71f0-b4d7-73d6-e0acfe406c88: compacting logs: from=1 to=22
2026-03-29T01:07:54.077Z [INFO]  server-8c88bb74-71f0-b4d7-73d6-e0acfe406c88: Installed remote snapshot
2026-03-29T01:07:54.077Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: pipelining replication: peer="{Voter server-8c88bb74-71f0-b4d7-73d6-e0acfe406c88 8c88bb74-71f0-b4d7-73d6-e0acfe406c88}"
2026-03-29T01:07:55.080Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:07:54.075081004 +0000 UTC m=+17.863223556 (2 nodes), 1.323µs from start of poll, 1.665768392s from cluster start. Timeout at 2026-03-29 01:07:55.080088713 +0000 UTC m=+18.868231310, 1.005007754s after stability
2026-03-29T01:07:55.092Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: aborting pipeline replication: peer="{Voter server-1e7c3462-e790-bd51-0768-b36b9785a4f4 1e7c3462-e790-bd51-0768-b36b9785a4f4}"
2026-03-29T01:07:55.092Z [INFO]  server-000342f2-4718-37c2-7de6-0d0ff98273e8: aborting pipeline replication: peer="{Voter server-8c88bb74-71f0-b4d7-73d6-e0acfe406c88 8c88bb74-71f0-b4d7-73d6-e0acfe406c88}"
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true
2026-03-29T01:07:55.093Z [INFO]  server-8c03b20d-0e82-aa9c-121d-b741732ac3c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c03b20d-0e82-aa9c-121d-b741732ac3c4 Address:8c03b20d-0e82-aa9c-121d-b741732ac3c4} {Suffrage:Voter ID:server-7861213c-7d9b-bedb-339b-026bd1b09598 Address:7861213c-7d9b-bedb-339b-026bd1b09598} {Suffrage:Voter ID:server-cea318a7-fd23-5490-796d-8cb8f6f3211a Address:cea318a7-fd23-5490-796d-8cb8f6f3211a}]"
2026-03-29T01:07:55.093Z [INFO]  server-8c03b20d-0e82-aa9c-121d-b741732ac3c4: entering follower state: follower="Node at 8c03b20d-0e82-aa9c-121d-b741732ac3c4 [Follower]" leader-address= leader-id=
2026-03-29T01:07:55.093Z [INFO]  server-7861213c-7d9b-bedb-339b-026bd1b09598: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c03b20d-0e82-aa9c-121d-b741732ac3c4 Address:8c03b20d-0e82-aa9c-121d-b741732ac3c4} {Suffrage:Voter ID:server-7861213c-7d9b-bedb-339b-026bd1b09598 Address:7861213c-7d9b-bedb-339b-026bd1b09598} {Suffrage:Voter ID:server-cea318a7-fd23-5490-796d-8cb8f6f3211a Address:cea318a7-fd23-5490-796d-8cb8f6f3211a}]"
2026-03-29T01:07:55.093Z [INFO]  server-7861213c-7d9b-bedb-339b-026bd1b09598: entering follower state: follower="Node at 7861213c-7d9b-bedb-339b-026bd1b09598 [Follower]" leader-address= leader-id=
2026-03-29T01:07:55.093Z [INFO]  server-cea318a7-fd23-5490-796d-8cb8f6f3211a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c03b20d-0e82-aa9c-121d-b741732ac3c4 Address:8c03b20d-0e82-aa9c-121d-b741732ac3c4} {Suffrage:Voter ID:server-7861213c-7d9b-bedb-339b-026bd1b09598 Address:7861213c-7d9b-bedb-339b-026bd1b09598} {Suffrage:Voter ID:server-cea318a7-fd23-5490-796d-8cb8f6f3211a Address:cea318a7-fd23-5490-796d-8cb8f6f3211a}]"
2026-03-29T01:07:55.093Z [INFO]  server-cea318a7-fd23-5490-796d-8cb8f6f3211a: entering follower state: follower="Node at cea318a7-fd23-5490-796d-8cb8f6f3211a [Follower]" leader-address= leader-id=
2026-03-29T01:07:55.093Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:55.955Z [WARN]  server-7861213c-7d9b-bedb-339b-026bd1b09598: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:55.955Z [INFO]  server-7861213c-7d9b-bedb-339b-026bd1b09598: entering candidate state: node="Node at 7861213c-7d9b-bedb-339b-026bd1b09598 [Candidate]" term=2
2026-03-29T01:07:55.955Z [INFO]  server-7861213c-7d9b-bedb-339b-026bd1b09598: election won: term=2 tally=2
2026-03-29T01:07:55.955Z [INFO]  server-7861213c-7d9b-bedb-339b-026bd1b09598: entering leader state: leader="Node at 7861213c-7d9b-bedb-339b-026bd1b09598 [Leader]"
2026-03-29T01:07:55.955Z [INFO]  server-7861213c-7d9b-bedb-339b-026bd1b09598: added peer, starting replication: peer=server-8c03b20d-0e82-aa9c-121d-b741732ac3c4
2026-03-29T01:07:55.955Z [INFO]  server-7861213c-7d9b-bedb-339b-026bd1b09598: added peer, starting replication: peer=server-cea318a7-fd23-5490-796d-8cb8f6f3211a
2026-03-29T01:07:55.955Z [INFO]  server-7861213c-7d9b-bedb-339b-026bd1b09598: pipelining replication: peer="{Voter server-cea318a7-fd23-5490-796d-8cb8f6f3211a cea318a7-fd23-5490-796d-8cb8f6f3211a}"
2026-03-29T01:07:55.955Z [INFO]  server-7861213c-7d9b-bedb-339b-026bd1b09598: pipelining replication: peer="{Voter server-8c03b20d-0e82-aa9c-121d-b741732ac3c4 8c03b20d-0e82-aa9c-121d-b741732ac3c4}"
2026-03-29T01:07:56.960Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:55.955678757 +0000 UTC m=+19.743821291 (1 nodes), 861.873577ms from start of poll, 862.222867ms from cluster start. Timeout at 2026-03-29 01:07:56.960684372 +0000 UTC m=+20.748826982, 1.005005691s after stability
2026-03-29T01:07:56.961Z [INFO]  server-7861213c-7d9b-bedb-339b-026bd1b09598: starting snapshot up to: index=12
2026-03-29T01:07:56.961Z [INFO]  creating new snapshot: path=/tmp/raft3047784489/snapshots/2-12-1774746476961.tmp
2026-03-29T01:07:56.964Z [INFO]  server-7861213c-7d9b-bedb-339b-026bd1b09598: snapshot complete up to: index=12
2026-03-29T01:07:56.965Z [INFO]  server-c5c78742-1db9-fdfa-e430-d7b58505add7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c5c78742-1db9-fdfa-e430-d7b58505add7 Address:c5c78742-1db9-fdfa-e430-d7b58505add7} {Suffrage:Voter ID:server-5b6b4948-9f6d-e772-9334-32b744069655 Address:5b6b4948-9f6d-e772-9334-32b744069655} {Suffrage:Voter ID:server-7edade2e-9df0-93a3-e8e8-bff31937ec9c Address:7edade2e-9df0-93a3-e8e8-bff31937ec9c}]"
2026-03-29T01:07:56.965Z [INFO]  server-c5c78742-1db9-fdfa-e430-d7b58505add7: entering follower state: follower="Node at c5c78742-1db9-fdfa-e430-d7b58505add7 [Follower]" leader-address= leader-id=
2026-03-29T01:07:56.965Z [INFO]  server-5b6b4948-9f6d-e772-9334-32b744069655: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c5c78742-1db9-fdfa-e430-d7b58505add7 Address:c5c78742-1db9-fdfa-e430-d7b58505add7} {Suffrage:Voter ID:server-5b6b4948-9f6d-e772-9334-32b744069655 Address:5b6b4948-9f6d-e772-9334-32b744069655} {Suffrage:Voter ID:server-7edade2e-9df0-93a3-e8e8-bff31937ec9c Address:7edade2e-9df0-93a3-e8e8-bff31937ec9c}]"
2026-03-29T01:07:56.965Z [INFO]  server-5b6b4948-9f6d-e772-9334-32b744069655: entering follower state: follower="Node at 5b6b4948-9f6d-e772-9334-32b744069655 [Follower]" leader-address= leader-id=
2026-03-29T01:07:56.965Z [INFO]  server-7edade2e-9df0-93a3-e8e8-bff31937ec9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c5c78742-1db9-fdfa-e430-d7b58505add7 Address:c5c78742-1db9-fdfa-e430-d7b58505add7} {Suffrage:Voter ID:server-5b6b4948-9f6d-e772-9334-32b744069655 Address:5b6b4948-9f6d-e772-9334-32b744069655} {Suffrage:Voter ID:server-7edade2e-9df0-93a3-e8e8-bff31937ec9c Address:7edade2e-9df0-93a3-e8e8-bff31937ec9c}]"
2026-03-29T01:07:56.965Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:56.965Z [INFO]  server-7edade2e-9df0-93a3-e8e8-bff31937ec9c: entering follower state: follower="Node at 7edade2e-9df0-93a3-e8e8-bff31937ec9c [Follower]" leader-address= leader-id=
2026-03-29T01:07:57.850Z [WARN]  server-5b6b4948-9f6d-e772-9334-32b744069655: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:07:57.850Z [INFO]  server-5b6b4948-9f6d-e772-9334-32b744069655: entering candidate state: node="Node at 5b6b4948-9f6d-e772-9334-32b744069655 [Candidate]" term=2
2026-03-29T01:07:57.850Z [INFO]  server-5b6b4948-9f6d-e772-9334-32b744069655: election won: term=2 tally=2
2026-03-29T01:07:57.850Z [INFO]  server-5b6b4948-9f6d-e772-9334-32b744069655: entering leader state: leader="Node at 5b6b4948-9f6d-e772-9334-32b744069655 [Leader]"
2026-03-29T01:07:57.851Z [INFO]  server-5b6b4948-9f6d-e772-9334-32b744069655: added peer, starting replication: peer=server-c5c78742-1db9-fdfa-e430-d7b58505add7
2026-03-29T01:07:57.851Z [INFO]  server-5b6b4948-9f6d-e772-9334-32b744069655: added peer, starting replication: peer=server-7edade2e-9df0-93a3-e8e8-bff31937ec9c
2026-03-29T01:07:57.851Z [INFO]  server-5b6b4948-9f6d-e772-9334-32b744069655: pipelining replication: peer="{Voter server-7edade2e-9df0-93a3-e8e8-bff31937ec9c 7edade2e-9df0-93a3-e8e8-bff31937ec9c}"
2026-03-29T01:07:57.851Z [INFO]  server-5b6b4948-9f6d-e772-9334-32b744069655: pipelining replication: peer="{Voter server-c5c78742-1db9-fdfa-e430-d7b58505add7 c5c78742-1db9-fdfa-e430-d7b58505add7}"
2026-03-29T01:07:58.857Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:07:57.851099326 +0000 UTC m=+21.639241897 (1 nodes), 885.734013ms from start of poll, 886.07622ms from cluster start. Timeout at 2026-03-29 01:07:58.856101485 +0000 UTC m=+22.644244095, 1.005002198s after stability
2026-03-29T01:07:58.857Z [INFO]  creating new snapshot: path=/tmp/raft1156946454/snapshots/2-13-1774746478857.tmp
2026-03-29T01:07:58.857Z [INFO]  server-5b6b4948-9f6d-e772-9334-32b744069655: copied to local snapshot: bytes=71
2026-03-29T01:07:58.858Z [INFO]  server-5b6b4948-9f6d-e772-9334-32b744069655: snapshot restore progress: id=2-13-1774746478857 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:07:58.858Z [INFO]  server-5b6b4948-9f6d-e772-9334-32b744069655: restored user snapshot: index=13
2026-03-29T01:07:58.858Z [ERROR] server-5b6b4948-9f6d-e772-9334-32b744069655: failed to get log: index=3 error="log not found"
2026-03-29T01:07:58.858Z [INFO]  server-5b6b4948-9f6d-e772-9334-32b744069655: aborting pipeline replication: peer="{Voter server-c5c78742-1db9-fdfa-e430-d7b58505add7 c5c78742-1db9-fdfa-e430-d7b58505add7}"
2026-03-29T01:07:58.858Z [ERROR] server-5b6b4948-9f6d-e772-9334-32b744069655: failed to get log: index=3 error="log not found"
2026-03-29T01:07:58.858Z [INFO]  server-5b6b4948-9f6d-e772-9334-32b744069655: aborting pipeline replication: peer="{Voter server-7edade2e-9df0-93a3-e8e8-bff31937ec9c 7edade2e-9df0-93a3-e8e8-bff31937ec9c}"
2026-03-29T01:07:58.866Z [ERROR] server-5b6b4948-9f6d-e772-9334-32b744069655: failed to get log: index=3 error="log not found"
2026-03-29T01:07:58.866Z [INFO]  creating new snapshot: path=/tmp/raft2795664586/snapshots/2-13-1774746478866.tmp
2026-03-29T01:07:58.866Z [ERROR] server-5b6b4948-9f6d-e772-9334-32b744069655: failed to get log: index=3 error="log not found"
2026-03-29T01:07:58.866Z [INFO]  server-7edade2e-9df0-93a3-e8e8-bff31937ec9c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:07:58.867Z [INFO]  creating new snapshot: path=/tmp/raft2272060080/snapshots/2-13-1774746478867.tmp
2026-03-29T01:07:58.867Z [INFO]  server-7edade2e-9df0-93a3-e8e8-bff31937ec9c: copied to local snapshot: bytes=71
2026-03-29T01:07:58.867Z [INFO]  server-c5c78742-1db9-fdfa-e430-d7b58505add7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:07:58.867Z [INFO]  server-7edade2e-9df0-93a3-e8e8-bff31937ec9c: snapshot restore progress: id=2-13-1774746478866 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:07:58.867Z [INFO]  server-7edade2e-9df0-93a3-e8e8-bff31937ec9c: Installed remote snapshot
2026-03-29T01:07:58.867Z [INFO]  server-5b6b4948-9f6d-e772-9334-32b744069655: pipelining replication: peer="{Voter server-7edade2e-9df0-93a3-e8e8-bff31937ec9c 7edade2e-9df0-93a3-e8e8-bff31937ec9c}"
2026-03-29T01:07:58.867Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:07:58.867Z [INFO]  server-c5c78742-1db9-fdfa-e430-d7b58505add7: copied to local snapshot: bytes=71
2026-03-29T01:07:58.867Z [INFO]  server-c5c78742-1db9-fdfa-e430-d7b58505add7: snapshot restore progress: id=2-13-1774746478867 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:07:58.867Z [INFO]  server-c5c78742-1db9-fdfa-e430-d7b58505add7: Installed remote snapshot
2026-03-29T01:07:58.867Z [INFO]  server-5b6b4948-9f6d-e772-9334-32b744069655: pipelining replication: peer="{Voter server-c5c78742-1db9-fdfa-e430-d7b58505add7 c5c78742-1db9-fdfa-e430-d7b58505add7}"
2026-03-29T01:07:59.873Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:07:58.867595484 +0000 UTC m=+22.655738036 (2 nodes), 927ns from start of poll, 1.902572359s from cluster start. Timeout at 2026-03-29 01:07:59.872600457 +0000 UTC m=+23.660742986, 1.00500495s after stability
2026-03-29T01:07:59.885Z [INFO]  server-7861213c-7d9b-bedb-339b-026bd1b09598: aborting pipeline replication: peer="{Voter server-cea318a7-fd23-5490-796d-8cb8f6f3211a cea318a7-fd23-5490-796d-8cb8f6f3211a}"
2026-03-29T01:07:59.885Z [INFO]  server-7861213c-7d9b-bedb-339b-026bd1b09598: aborting pipeline replication: peer="{Voter server-8c03b20d-0e82-aa9c-121d-b741732ac3c4 8c03b20d-0e82-aa9c-121d-b741732ac3c4}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true
2026-03-29T01:07:59.886Z [INFO]  server-49be0bea-89cb-3f83-ac38-b2b00479e689: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49be0bea-89cb-3f83-ac38-b2b00479e689 Address:49be0bea-89cb-3f83-ac38-b2b00479e689} {Suffrage:Voter ID:server-849db978-e23c-3e7e-47cc-7b2b74cb859f Address:849db978-e23c-3e7e-47cc-7b2b74cb859f} {Suffrage:Voter ID:server-31e42e13-9986-493f-64fb-57a20711fc22 Address:31e42e13-9986-493f-64fb-57a20711fc22}]"
2026-03-29T01:07:59.886Z [INFO]  server-49be0bea-89cb-3f83-ac38-b2b00479e689: entering follower state: follower="Node at 49be0bea-89cb-3f83-ac38-b2b00479e689 [Follower]" leader-address= leader-id=
2026-03-29T01:07:59.886Z [INFO]  server-849db978-e23c-3e7e-47cc-7b2b74cb859f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49be0bea-89cb-3f83-ac38-b2b00479e689 Address:49be0bea-89cb-3f83-ac38-b2b00479e689} {Suffrage:Voter ID:server-849db978-e23c-3e7e-47cc-7b2b74cb859f Address:849db978-e23c-3e7e-47cc-7b2b74cb859f} {Suffrage:Voter ID:server-31e42e13-9986-493f-64fb-57a20711fc22 Address:31e42e13-9986-493f-64fb-57a20711fc22}]"
2026-03-29T01:07:59.886Z [INFO]  server-849db978-e23c-3e7e-47cc-7b2b74cb859f: entering follower state: follower="Node at 849db978-e23c-3e7e-47cc-7b2b74cb859f [Follower]" leader-address= leader-id=
2026-03-29T01:07:59.886Z [INFO]  server-31e42e13-9986-493f-64fb-57a20711fc22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-49be0bea-89cb-3f83-ac38-b2b00479e689 Address:49be0bea-89cb-3f83-ac38-b2b00479e689} {Suffrage:Voter ID:server-849db978-e23c-3e7e-47cc-7b2b74cb859f Address:849db978-e23c-3e7e-47cc-7b2b74cb859f} {Suffrage:Voter ID:server-31e42e13-9986-493f-64fb-57a20711fc22 Address:31e42e13-9986-493f-64fb-57a20711fc22}]"
2026-03-29T01:07:59.886Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:07:59.886Z [INFO]  server-31e42e13-9986-493f-64fb-57a20711fc22: entering follower state: follower="Node at 31e42e13-9986-493f-64fb-57a20711fc22 [Follower]" leader-address= leader-id=
2026-03-29T01:08:00.515Z [WARN]  server-31e42e13-9986-493f-64fb-57a20711fc22: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:00.515Z [INFO]  server-31e42e13-9986-493f-64fb-57a20711fc22: entering candidate state: node="Node at 31e42e13-9986-493f-64fb-57a20711fc22 [Candidate]" term=2
2026-03-29T01:08:00.515Z [INFO]  server-31e42e13-9986-493f-64fb-57a20711fc22: election won: term=2 tally=2
2026-03-29T01:08:00.515Z [INFO]  server-31e42e13-9986-493f-64fb-57a20711fc22: entering leader state: leader="Node at 31e42e13-9986-493f-64fb-57a20711fc22 [Leader]"
2026-03-29T01:08:00.515Z [INFO]  server-31e42e13-9986-493f-64fb-57a20711fc22: added peer, starting replication: peer=server-49be0bea-89cb-3f83-ac38-b2b00479e689
2026-03-29T01:08:00.515Z [INFO]  server-31e42e13-9986-493f-64fb-57a20711fc22: added peer, starting replication: peer=server-849db978-e23c-3e7e-47cc-7b2b74cb859f
2026-03-29T01:08:00.515Z [INFO]  server-31e42e13-9986-493f-64fb-57a20711fc22: pipelining replication: peer="{Voter server-49be0bea-89cb-3f83-ac38-b2b00479e689 49be0bea-89cb-3f83-ac38-b2b00479e689}"
2026-03-29T01:08:00.515Z [INFO]  server-31e42e13-9986-493f-64fb-57a20711fc22: pipelining replication: peer="{Voter server-849db978-e23c-3e7e-47cc-7b2b74cb859f 849db978-e23c-3e7e-47cc-7b2b74cb859f}"
2026-03-29T01:08:01.520Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:00.515151692 +0000 UTC m=+24.303294230 (1 nodes), 628.427806ms from start of poll, 628.745036ms from cluster start. Timeout at 2026-03-29 01:08:01.520155895 +0000 UTC m=+25.308298482, 1.005004252s after stability
2026-03-29T01:08:01.522Z [INFO]  server-31e42e13-9986-493f-64fb-57a20711fc22: starting snapshot up to: index=12
2026-03-29T01:08:01.522Z [INFO]  creating new snapshot: path=/tmp/raft2483005070/snapshots/2-12-1774746481522.tmp
2026-03-29T01:08:01.523Z [INFO]  server-31e42e13-9986-493f-64fb-57a20711fc22: snapshot complete up to: index=12
2026-03-29T01:08:01.524Z [INFO]  server-797eed01-e289-4bf1-25b3-ce2a24c81c08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-797eed01-e289-4bf1-25b3-ce2a24c81c08 Address:797eed01-e289-4bf1-25b3-ce2a24c81c08} {Suffrage:Voter ID:server-132cba00-9a81-228b-8276-cae6c4a1a165 Address:132cba00-9a81-228b-8276-cae6c4a1a165} {Suffrage:Voter ID:server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f Address:48df6ef1-7e8b-eb14-8550-aa44c832fd2f}]"
2026-03-29T01:08:01.524Z [INFO]  server-797eed01-e289-4bf1-25b3-ce2a24c81c08: entering follower state: follower="Node at 797eed01-e289-4bf1-25b3-ce2a24c81c08 [Follower]" leader-address= leader-id=
2026-03-29T01:08:01.524Z [INFO]  server-132cba00-9a81-228b-8276-cae6c4a1a165: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-797eed01-e289-4bf1-25b3-ce2a24c81c08 Address:797eed01-e289-4bf1-25b3-ce2a24c81c08} {Suffrage:Voter ID:server-132cba00-9a81-228b-8276-cae6c4a1a165 Address:132cba00-9a81-228b-8276-cae6c4a1a165} {Suffrage:Voter ID:server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f Address:48df6ef1-7e8b-eb14-8550-aa44c832fd2f}]"
2026-03-29T01:08:01.524Z [INFO]  server-132cba00-9a81-228b-8276-cae6c4a1a165: entering follower state: follower="Node at 132cba00-9a81-228b-8276-cae6c4a1a165 [Follower]" leader-address= leader-id=
2026-03-29T01:08:01.524Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-797eed01-e289-4bf1-25b3-ce2a24c81c08 Address:797eed01-e289-4bf1-25b3-ce2a24c81c08} {Suffrage:Voter ID:server-132cba00-9a81-228b-8276-cae6c4a1a165 Address:132cba00-9a81-228b-8276-cae6c4a1a165} {Suffrage:Voter ID:server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f Address:48df6ef1-7e8b-eb14-8550-aa44c832fd2f}]"
2026-03-29T01:08:01.524Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:01.524Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: entering follower state: follower="Node at 48df6ef1-7e8b-eb14-8550-aa44c832fd2f [Follower]" leader-address= leader-id=
2026-03-29T01:08:02.072Z [WARN]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:02.072Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: entering candidate state: node="Node at 48df6ef1-7e8b-eb14-8550-aa44c832fd2f [Candidate]" term=2
2026-03-29T01:08:02.072Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: election won: term=2 tally=2
2026-03-29T01:08:02.072Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: entering leader state: leader="Node at 48df6ef1-7e8b-eb14-8550-aa44c832fd2f [Leader]"
2026-03-29T01:08:02.072Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: added peer, starting replication: peer=server-797eed01-e289-4bf1-25b3-ce2a24c81c08
2026-03-29T01:08:02.072Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: added peer, starting replication: peer=server-132cba00-9a81-228b-8276-cae6c4a1a165
2026-03-29T01:08:02.072Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: pipelining replication: peer="{Voter server-797eed01-e289-4bf1-25b3-ce2a24c81c08 797eed01-e289-4bf1-25b3-ce2a24c81c08}"
2026-03-29T01:08:02.072Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: pipelining replication: peer="{Voter server-132cba00-9a81-228b-8276-cae6c4a1a165 132cba00-9a81-228b-8276-cae6c4a1a165}"
2026-03-29T01:08:03.077Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:02.072587693 +0000 UTC m=+25.860730261 (1 nodes), 548.275491ms from start of poll, 548.684814ms from cluster start. Timeout at 2026-03-29 01:08:03.077596558 +0000 UTC m=+26.865739142, 1.005008881s after stability
2026-03-29T01:08:03.078Z [INFO]  creating new snapshot: path=/tmp/raft91803158/snapshots/2-13-1774746483078.tmp
2026-03-29T01:08:03.078Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: copied to local snapshot: bytes=71
2026-03-29T01:08:03.078Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: snapshot restore progress: id=2-13-1774746483078 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:03.078Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: removing all old logs from log store
2026-03-29T01:08:03.078Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: compacting logs: from=1 to=2
2026-03-29T01:08:03.078Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: restored user snapshot: index=13
2026-03-29T01:08:03.079Z [ERROR] server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: failed to get log: index=2 error="log not found"
2026-03-29T01:08:03.079Z [ERROR] server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: failed to get log: index=2 error="log not found"
2026-03-29T01:08:03.079Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: aborting pipeline replication: peer="{Voter server-132cba00-9a81-228b-8276-cae6c4a1a165 132cba00-9a81-228b-8276-cae6c4a1a165}"
2026-03-29T01:08:03.079Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: aborting pipeline replication: peer="{Voter server-797eed01-e289-4bf1-25b3-ce2a24c81c08 797eed01-e289-4bf1-25b3-ce2a24c81c08}"
2026-03-29T01:08:03.086Z [ERROR] server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: failed to get log: index=2 error="log not found"
2026-03-29T01:08:03.087Z [INFO]  creating new snapshot: path=/tmp/raft2959409454/snapshots/2-13-1774746483087.tmp
2026-03-29T01:08:03.087Z [INFO]  server-132cba00-9a81-228b-8276-cae6c4a1a165: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:03.087Z [INFO]  server-132cba00-9a81-228b-8276-cae6c4a1a165: copied to local snapshot: bytes=71
2026-03-29T01:08:03.088Z [INFO]  server-132cba00-9a81-228b-8276-cae6c4a1a165: snapshot restore progress: id=2-13-1774746483087 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:03.088Z [INFO]  server-132cba00-9a81-228b-8276-cae6c4a1a165: removing all old logs from log store
2026-03-29T01:08:03.088Z [INFO]  server-132cba00-9a81-228b-8276-cae6c4a1a165: compacting logs: from=1 to=2
2026-03-29T01:08:03.088Z [INFO]  server-132cba00-9a81-228b-8276-cae6c4a1a165: Installed remote snapshot
2026-03-29T01:08:03.088Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: pipelining replication: peer="{Voter server-132cba00-9a81-228b-8276-cae6c4a1a165 132cba00-9a81-228b-8276-cae6c4a1a165}"
2026-03-29T01:08:03.088Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:08:03.089Z [ERROR] server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: failed to get log: index=2 error="log not found"
2026-03-29T01:08:03.089Z [INFO]  creating new snapshot: path=/tmp/raft3612456393/snapshots/2-13-1774746483089.tmp
2026-03-29T01:08:03.089Z [INFO]  server-797eed01-e289-4bf1-25b3-ce2a24c81c08: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:03.089Z [INFO]  server-797eed01-e289-4bf1-25b3-ce2a24c81c08: copied to local snapshot: bytes=71
2026-03-29T01:08:03.090Z [INFO]  server-797eed01-e289-4bf1-25b3-ce2a24c81c08: snapshot restore progress: id=2-13-1774746483089 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:03.090Z [INFO]  server-797eed01-e289-4bf1-25b3-ce2a24c81c08: removing all old logs from log store
2026-03-29T01:08:03.090Z [INFO]  server-797eed01-e289-4bf1-25b3-ce2a24c81c08: compacting logs: from=1 to=2
2026-03-29T01:08:03.090Z [INFO]  server-797eed01-e289-4bf1-25b3-ce2a24c81c08: Installed remote snapshot
2026-03-29T01:08:03.090Z [INFO]  server-48df6ef1-7e8b-eb14-8550-aa44c832fd2f: pipelining replication: peer="{Voter server-797eed01-e289-4bf1-25b3-ce2a24c81c08 797eed01-e289-4bf1-25b3-ce2a24c81c08}"
2026-03-29T01:08:04.094Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:08:03.088308331 +0000 UTC m=+26.876450866 (2 nodes), 616ns from start of poll, 1.564405419s from cluster start. Timeout at 2026-03-29 01:08:04.093313871 +0000 UTC m=+27.881456408, 1.005005542s after stability
2026-03-29T01:08:04.105Z [INFO]  server-31e42e13-9986-493f-64fb-57a20711fc22: aborting pipeline replication: peer="{Voter server-849db978-e23c-3e7e-47cc-7b2b74cb859f 849db978-e23c-3e7e-47cc-7b2b74cb859f}"
2026-03-29T01:08:04.105Z [INFO]  server-31e42e13-9986-493f-64fb-57a20711fc22: aborting pipeline replication: peer="{Voter server-49be0bea-89cb-3f83-ac38-b2b00479e689 49be0bea-89cb-3f83-ac38-b2b00479e689}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false
2026-03-29T01:08:04.106Z [INFO]  server-30db0ada-56b0-f2a4-aa2e-f91f8919a215: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30db0ada-56b0-f2a4-aa2e-f91f8919a215 Address:30db0ada-56b0-f2a4-aa2e-f91f8919a215} {Suffrage:Voter ID:server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48 Address:bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48} {Suffrage:Voter ID:server-4090824f-c41f-6ecc-0429-3c113882cf8f Address:4090824f-c41f-6ecc-0429-3c113882cf8f}]"
2026-03-29T01:08:04.106Z [INFO]  server-30db0ada-56b0-f2a4-aa2e-f91f8919a215: entering follower state: follower="Node at 30db0ada-56b0-f2a4-aa2e-f91f8919a215 [Follower]" leader-address= leader-id=
2026-03-29T01:08:04.106Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30db0ada-56b0-f2a4-aa2e-f91f8919a215 Address:30db0ada-56b0-f2a4-aa2e-f91f8919a215} {Suffrage:Voter ID:server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48 Address:bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48} {Suffrage:Voter ID:server-4090824f-c41f-6ecc-0429-3c113882cf8f Address:4090824f-c41f-6ecc-0429-3c113882cf8f}]"
2026-03-29T01:08:04.106Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: entering follower state: follower="Node at bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48 [Follower]" leader-address= leader-id=
2026-03-29T01:08:04.106Z [INFO]  server-4090824f-c41f-6ecc-0429-3c113882cf8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30db0ada-56b0-f2a4-aa2e-f91f8919a215 Address:30db0ada-56b0-f2a4-aa2e-f91f8919a215} {Suffrage:Voter ID:server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48 Address:bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48} {Suffrage:Voter ID:server-4090824f-c41f-6ecc-0429-3c113882cf8f Address:4090824f-c41f-6ecc-0429-3c113882cf8f}]"
2026-03-29T01:08:04.106Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:04.106Z [INFO]  server-4090824f-c41f-6ecc-0429-3c113882cf8f: entering follower state: follower="Node at 4090824f-c41f-6ecc-0429-3c113882cf8f [Follower]" leader-address= leader-id=
2026-03-29T01:08:04.614Z [WARN]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:04.614Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: entering candidate state: node="Node at bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48 [Candidate]" term=2
2026-03-29T01:08:04.614Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: election won: term=2 tally=2
2026-03-29T01:08:04.614Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: entering leader state: leader="Node at bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48 [Leader]"
2026-03-29T01:08:04.614Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: added peer, starting replication: peer=server-30db0ada-56b0-f2a4-aa2e-f91f8919a215
2026-03-29T01:08:04.614Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: added peer, starting replication: peer=server-4090824f-c41f-6ecc-0429-3c113882cf8f
2026-03-29T01:08:04.614Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: pipelining replication: peer="{Voter server-30db0ada-56b0-f2a4-aa2e-f91f8919a215 30db0ada-56b0-f2a4-aa2e-f91f8919a215}"
2026-03-29T01:08:04.614Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: pipelining replication: peer="{Voter server-4090824f-c41f-6ecc-0429-3c113882cf8f 4090824f-c41f-6ecc-0429-3c113882cf8f}"
2026-03-29T01:08:05.619Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:04.614704807 +0000 UTC m=+28.402847350 (1 nodes), 507.900026ms from start of poll, 508.279597ms from cluster start. Timeout at 2026-03-29 01:08:05.619708406 +0000 UTC m=+29.407850948, 1.005003598s after stability
2026-03-29T01:08:05.620Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: starting snapshot up to: index=12
2026-03-29T01:08:05.620Z [INFO]  creating new snapshot: path=/tmp/raft3166450785/snapshots/2-12-1774746485620.tmp
2026-03-29T01:08:05.620Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: snapshot complete up to: index=12
2026-03-29T01:08:05.621Z [INFO]  creating new snapshot: path=/tmp/raft3166450785/snapshots/2-23-1774746485621.tmp
2026-03-29T01:08:05.622Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: copied to local snapshot: bytes=71
2026-03-29T01:08:05.622Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: snapshot restore progress: id=2-23-1774746485621 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:05.622Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: restored user snapshot: index=23
2026-03-29T01:08:05.622Z [ERROR] server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: failed to get log: index=23 error="log not found"
2026-03-29T01:08:05.622Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: aborting pipeline replication: peer="{Voter server-4090824f-c41f-6ecc-0429-3c113882cf8f 4090824f-c41f-6ecc-0429-3c113882cf8f}"
2026-03-29T01:08:05.622Z [ERROR] server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: failed to get log: index=23 error="log not found"
2026-03-29T01:08:05.622Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: aborting pipeline replication: peer="{Voter server-30db0ada-56b0-f2a4-aa2e-f91f8919a215 30db0ada-56b0-f2a4-aa2e-f91f8919a215}"
2026-03-29T01:08:05.628Z [ERROR] server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: failed to get log: index=23 error="log not found"
2026-03-29T01:08:05.629Z [INFO]  creating new snapshot: path=/tmp/raft4034563444/snapshots/2-23-1774746485629.tmp
2026-03-29T01:08:05.629Z [ERROR] server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: failed to get log: index=23 error="log not found"
2026-03-29T01:08:05.629Z [INFO]  server-30db0ada-56b0-f2a4-aa2e-f91f8919a215: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:05.629Z [INFO]  creating new snapshot: path=/tmp/raft762936853/snapshots/2-23-1774746485629.tmp
2026-03-29T01:08:05.629Z [INFO]  server-30db0ada-56b0-f2a4-aa2e-f91f8919a215: copied to local snapshot: bytes=71
2026-03-29T01:08:05.629Z [INFO]  server-4090824f-c41f-6ecc-0429-3c113882cf8f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:05.630Z [INFO]  server-30db0ada-56b0-f2a4-aa2e-f91f8919a215: snapshot restore progress: id=2-23-1774746485629 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:05.630Z [INFO]  server-30db0ada-56b0-f2a4-aa2e-f91f8919a215: Installed remote snapshot
2026-03-29T01:08:05.630Z [INFO]  server-4090824f-c41f-6ecc-0429-3c113882cf8f: copied to local snapshot: bytes=71
2026-03-29T01:08:05.630Z [WARN]  server-30db0ada-56b0-f2a4-aa2e-f91f8919a215: skipping application of old log: index=22
2026-03-29T01:08:05.630Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: pipelining replication: peer="{Voter server-30db0ada-56b0-f2a4-aa2e-f91f8919a215 30db0ada-56b0-f2a4-aa2e-f91f8919a215}"
2026-03-29T01:08:05.630Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:08:05.630Z [INFO]  server-4090824f-c41f-6ecc-0429-3c113882cf8f: snapshot restore progress: id=2-23-1774746485629 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:05.630Z [INFO]  server-4090824f-c41f-6ecc-0429-3c113882cf8f: Installed remote snapshot
2026-03-29T01:08:05.631Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: pipelining replication: peer="{Voter server-4090824f-c41f-6ecc-0429-3c113882cf8f 4090824f-c41f-6ecc-0429-3c113882cf8f}"
2026-03-29T01:08:06.636Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:08:05.630530219 +0000 UTC m=+29.418672755 (2 nodes), 1.256µs from start of poll, 1.524105002s from cluster start. Timeout at 2026-03-29 01:08:06.635536243 +0000 UTC m=+30.423678843, 1.005006088s after stability
2026-03-29T01:08:06.647Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: aborting pipeline replication: peer="{Voter server-30db0ada-56b0-f2a4-aa2e-f91f8919a215 30db0ada-56b0-f2a4-aa2e-f91f8919a215}"
2026-03-29T01:08:06.647Z [INFO]  server-bd0b288b-a1dc-8b9a-ed6a-74c4150e1e48: aborting pipeline replication: peer="{Voter server-4090824f-c41f-6ecc-0429-3c113882cf8f 4090824f-c41f-6ecc-0429-3c113882cf8f}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false
2026-03-29T01:08:06.649Z [INFO]  server-b23af7f7-360a-6db4-c559-ee04bdf6a2bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b23af7f7-360a-6db4-c559-ee04bdf6a2bd Address:b23af7f7-360a-6db4-c559-ee04bdf6a2bd} {Suffrage:Voter ID:server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6 Address:cebbc18a-55cb-61fe-7c54-0ae77261f4d6} {Suffrage:Voter ID:server-ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348 Address:ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348}]"
2026-03-29T01:08:06.649Z [INFO]  server-b23af7f7-360a-6db4-c559-ee04bdf6a2bd: entering follower state: follower="Node at b23af7f7-360a-6db4-c559-ee04bdf6a2bd [Follower]" leader-address= leader-id=
2026-03-29T01:08:06.649Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b23af7f7-360a-6db4-c559-ee04bdf6a2bd Address:b23af7f7-360a-6db4-c559-ee04bdf6a2bd} {Suffrage:Voter ID:server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6 Address:cebbc18a-55cb-61fe-7c54-0ae77261f4d6} {Suffrage:Voter ID:server-ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348 Address:ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348}]"
2026-03-29T01:08:06.649Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: entering follower state: follower="Node at cebbc18a-55cb-61fe-7c54-0ae77261f4d6 [Follower]" leader-address= leader-id=
2026-03-29T01:08:06.649Z [INFO]  server-ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b23af7f7-360a-6db4-c559-ee04bdf6a2bd Address:b23af7f7-360a-6db4-c559-ee04bdf6a2bd} {Suffrage:Voter ID:server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6 Address:cebbc18a-55cb-61fe-7c54-0ae77261f4d6} {Suffrage:Voter ID:server-ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348 Address:ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348}]"
2026-03-29T01:08:06.649Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:06.649Z [INFO]  server-ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348: entering follower state: follower="Node at ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348 [Follower]" leader-address= leader-id=
2026-03-29T01:08:07.275Z [WARN]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:07.275Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: entering candidate state: node="Node at cebbc18a-55cb-61fe-7c54-0ae77261f4d6 [Candidate]" term=2
2026-03-29T01:08:07.275Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: election won: term=2 tally=2
2026-03-29T01:08:07.275Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: entering leader state: leader="Node at cebbc18a-55cb-61fe-7c54-0ae77261f4d6 [Leader]"
2026-03-29T01:08:07.275Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: added peer, starting replication: peer=server-b23af7f7-360a-6db4-c559-ee04bdf6a2bd
2026-03-29T01:08:07.275Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: added peer, starting replication: peer=server-ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348
2026-03-29T01:08:07.275Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: pipelining replication: peer="{Voter server-ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348 ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348}"
2026-03-29T01:08:07.275Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: pipelining replication: peer="{Voter server-b23af7f7-360a-6db4-c559-ee04bdf6a2bd b23af7f7-360a-6db4-c559-ee04bdf6a2bd}"
2026-03-29T01:08:08.281Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:07.275630269 +0000 UTC m=+31.063772804 (1 nodes), 625.880821ms from start of poll, 626.313556ms from cluster start. Timeout at 2026-03-29 01:08:08.280641213 +0000 UTC m=+32.068783797, 1.005010993s after stability
2026-03-29T01:08:08.282Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: starting snapshot up to: index=12
2026-03-29T01:08:08.282Z [INFO]  creating new snapshot: path=/tmp/raft1270017622/snapshots/2-12-1774746488282.tmp
2026-03-29T01:08:08.285Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: snapshot complete up to: index=12
2026-03-29T01:08:08.286Z [INFO]  creating new snapshot: path=/tmp/raft1270017622/snapshots/2-23-1774746488286.tmp
2026-03-29T01:08:08.287Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: copied to local snapshot: bytes=71
2026-03-29T01:08:08.287Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: snapshot restore progress: id=2-23-1774746488286 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:08.287Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: removing all old logs from log store
2026-03-29T01:08:08.287Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: compacting logs: from=1 to=22
2026-03-29T01:08:08.287Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: restored user snapshot: index=23
2026-03-29T01:08:08.287Z [ERROR] server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: failed to get log: index=22 error="log not found"
2026-03-29T01:08:08.287Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: aborting pipeline replication: peer="{Voter server-ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348 ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348}"
2026-03-29T01:08:08.287Z [ERROR] server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: failed to get log: index=22 error="log not found"
2026-03-29T01:08:08.287Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: aborting pipeline replication: peer="{Voter server-b23af7f7-360a-6db4-c559-ee04bdf6a2bd b23af7f7-360a-6db4-c559-ee04bdf6a2bd}"
2026-03-29T01:08:08.293Z [ERROR] server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: failed to get log: index=22 error="log not found"
2026-03-29T01:08:08.293Z [INFO]  creating new snapshot: path=/tmp/raft1011205584/snapshots/2-23-1774746488293.tmp
2026-03-29T01:08:08.294Z [INFO]  server-b23af7f7-360a-6db4-c559-ee04bdf6a2bd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:08.294Z [INFO]  server-b23af7f7-360a-6db4-c559-ee04bdf6a2bd: copied to local snapshot: bytes=71
2026-03-29T01:08:08.294Z [INFO]  server-b23af7f7-360a-6db4-c559-ee04bdf6a2bd: snapshot restore progress: id=2-23-1774746488293 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:08.294Z [INFO]  server-b23af7f7-360a-6db4-c559-ee04bdf6a2bd: removing all old logs from log store
2026-03-29T01:08:08.294Z [INFO]  server-b23af7f7-360a-6db4-c559-ee04bdf6a2bd: compacting logs: from=1 to=22
2026-03-29T01:08:08.294Z [INFO]  server-b23af7f7-360a-6db4-c559-ee04bdf6a2bd: Installed remote snapshot
2026-03-29T01:08:08.294Z [WARN]  server-b23af7f7-360a-6db4-c559-ee04bdf6a2bd: skipping application of old log: index=22
2026-03-29T01:08:08.294Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: pipelining replication: peer="{Voter server-b23af7f7-360a-6db4-c559-ee04bdf6a2bd b23af7f7-360a-6db4-c559-ee04bdf6a2bd}"
2026-03-29T01:08:08.294Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:08:08.295Z [ERROR] server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: failed to get log: index=22 error="log not found"
2026-03-29T01:08:08.295Z [INFO]  creating new snapshot: path=/tmp/raft2912939351/snapshots/2-23-1774746488295.tmp
2026-03-29T01:08:08.296Z [INFO]  server-ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:08.296Z [INFO]  server-ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348: copied to local snapshot: bytes=71
2026-03-29T01:08:08.296Z [INFO]  server-ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348: snapshot restore progress: id=2-23-1774746488295 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:08.296Z [INFO]  server-ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348: removing all old logs from log store
2026-03-29T01:08:08.296Z [INFO]  server-ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348: compacting logs: from=1 to=22
2026-03-29T01:08:08.296Z [INFO]  server-ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348: Installed remote snapshot
2026-03-29T01:08:08.296Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: pipelining replication: peer="{Voter server-ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348 ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348}"
2026-03-29T01:08:09.299Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:08:08.294904211 +0000 UTC m=+32.083046746 (2 nodes), 657ns from start of poll, 1.645587498s from cluster start. Timeout at 2026-03-29 01:08:09.299911542 +0000 UTC m=+33.088054081, 1.005007335s after stability
2026-03-29T01:08:09.312Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: aborting pipeline replication: peer="{Voter server-ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348 ac55f7cd-d01b-11d1-6c57-fb7a6c8cb348}"
2026-03-29T01:08:09.312Z [INFO]  server-cebbc18a-55cb-61fe-7c54-0ae77261f4d6: aborting pipeline replication: peer="{Voter server-b23af7f7-360a-6db4-c559-ee04bdf6a2bd b23af7f7-360a-6db4-c559-ee04bdf6a2bd}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true
2026-03-29T01:08:09.314Z [INFO]  server-d3705e40-745b-b0f7-e7f6-f53422387c27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3705e40-745b-b0f7-e7f6-f53422387c27 Address:d3705e40-745b-b0f7-e7f6-f53422387c27} {Suffrage:Voter ID:server-a920efa6-e176-0caa-4d07-d68578442b74 Address:a920efa6-e176-0caa-4d07-d68578442b74} {Suffrage:Voter ID:server-f3f702b9-f890-58bc-5df8-7f0cc4b167d9 Address:f3f702b9-f890-58bc-5df8-7f0cc4b167d9}]"
2026-03-29T01:08:09.314Z [INFO]  server-d3705e40-745b-b0f7-e7f6-f53422387c27: entering follower state: follower="Node at d3705e40-745b-b0f7-e7f6-f53422387c27 [Follower]" leader-address= leader-id=
2026-03-29T01:08:09.314Z [INFO]  server-a920efa6-e176-0caa-4d07-d68578442b74: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3705e40-745b-b0f7-e7f6-f53422387c27 Address:d3705e40-745b-b0f7-e7f6-f53422387c27} {Suffrage:Voter ID:server-a920efa6-e176-0caa-4d07-d68578442b74 Address:a920efa6-e176-0caa-4d07-d68578442b74} {Suffrage:Voter ID:server-f3f702b9-f890-58bc-5df8-7f0cc4b167d9 Address:f3f702b9-f890-58bc-5df8-7f0cc4b167d9}]"
2026-03-29T01:08:09.314Z [INFO]  server-a920efa6-e176-0caa-4d07-d68578442b74: entering follower state: follower="Node at a920efa6-e176-0caa-4d07-d68578442b74 [Follower]" leader-address= leader-id=
2026-03-29T01:08:09.314Z [INFO]  server-f3f702b9-f890-58bc-5df8-7f0cc4b167d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3705e40-745b-b0f7-e7f6-f53422387c27 Address:d3705e40-745b-b0f7-e7f6-f53422387c27} {Suffrage:Voter ID:server-a920efa6-e176-0caa-4d07-d68578442b74 Address:a920efa6-e176-0caa-4d07-d68578442b74} {Suffrage:Voter ID:server-f3f702b9-f890-58bc-5df8-7f0cc4b167d9 Address:f3f702b9-f890-58bc-5df8-7f0cc4b167d9}]"
2026-03-29T01:08:09.314Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:09.314Z [INFO]  server-f3f702b9-f890-58bc-5df8-7f0cc4b167d9: entering follower state: follower="Node at f3f702b9-f890-58bc-5df8-7f0cc4b167d9 [Follower]" leader-address= leader-id=
2026-03-29T01:08:09.927Z [WARN]  server-f3f702b9-f890-58bc-5df8-7f0cc4b167d9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:09.927Z [INFO]  server-f3f702b9-f890-58bc-5df8-7f0cc4b167d9: entering candidate state: node="Node at f3f702b9-f890-58bc-5df8-7f0cc4b167d9 [Candidate]" term=2
2026-03-29T01:08:09.927Z [INFO]  server-f3f702b9-f890-58bc-5df8-7f0cc4b167d9: election won: term=2 tally=2
2026-03-29T01:08:09.927Z [INFO]  server-f3f702b9-f890-58bc-5df8-7f0cc4b167d9: entering leader state: leader="Node at f3f702b9-f890-58bc-5df8-7f0cc4b167d9 [Leader]"
2026-03-29T01:08:09.927Z [INFO]  server-f3f702b9-f890-58bc-5df8-7f0cc4b167d9: added peer, starting replication: peer=server-d3705e40-745b-b0f7-e7f6-f53422387c27
2026-03-29T01:08:09.927Z [INFO]  server-f3f702b9-f890-58bc-5df8-7f0cc4b167d9: added peer, starting replication: peer=server-a920efa6-e176-0caa-4d07-d68578442b74
2026-03-29T01:08:09.927Z [INFO]  server-f3f702b9-f890-58bc-5df8-7f0cc4b167d9: pipelining replication: peer="{Voter server-a920efa6-e176-0caa-4d07-d68578442b74 a920efa6-e176-0caa-4d07-d68578442b74}"
2026-03-29T01:08:09.927Z [INFO]  server-f3f702b9-f890-58bc-5df8-7f0cc4b167d9: pipelining replication: peer="{Voter server-d3705e40-745b-b0f7-e7f6-f53422387c27 d3705e40-745b-b0f7-e7f6-f53422387c27}"
2026-03-29T01:08:10.932Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:09.927555182 +0000 UTC m=+33.715697739 (1 nodes), 612.947964ms from start of poll, 613.254783ms from cluster start. Timeout at 2026-03-29 01:08:10.932565861 +0000 UTC m=+34.720708445, 1.005010706s after stability
2026-03-29T01:08:10.934Z [INFO]  server-f3f702b9-f890-58bc-5df8-7f0cc4b167d9: starting snapshot up to: index=12
2026-03-29T01:08:10.934Z [INFO]  creating new snapshot: path=/tmp/raft3352720181/snapshots/2-12-1774746490934.tmp
2026-03-29T01:08:10.934Z [INFO]  server-f3f702b9-f890-58bc-5df8-7f0cc4b167d9: snapshot complete up to: index=12
2026-03-29T01:08:10.935Z [INFO]  server-f7516ccb-6b12-8810-e0c3-cce5f3bde450: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f7516ccb-6b12-8810-e0c3-cce5f3bde450 Address:f7516ccb-6b12-8810-e0c3-cce5f3bde450} {Suffrage:Voter ID:server-7b02a9f6-178e-f349-ccdc-28fbdd22e5a1 Address:7b02a9f6-178e-f349-ccdc-28fbdd22e5a1} {Suffrage:Voter ID:server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412 Address:8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412}]"
2026-03-29T01:08:10.935Z [INFO]  server-f7516ccb-6b12-8810-e0c3-cce5f3bde450: entering follower state: follower="Node at f7516ccb-6b12-8810-e0c3-cce5f3bde450 [Follower]" leader-address= leader-id=
2026-03-29T01:08:10.935Z [INFO]  server-7b02a9f6-178e-f349-ccdc-28fbdd22e5a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f7516ccb-6b12-8810-e0c3-cce5f3bde450 Address:f7516ccb-6b12-8810-e0c3-cce5f3bde450} {Suffrage:Voter ID:server-7b02a9f6-178e-f349-ccdc-28fbdd22e5a1 Address:7b02a9f6-178e-f349-ccdc-28fbdd22e5a1} {Suffrage:Voter ID:server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412 Address:8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412}]"
2026-03-29T01:08:10.935Z [INFO]  server-7b02a9f6-178e-f349-ccdc-28fbdd22e5a1: entering follower state: follower="Node at 7b02a9f6-178e-f349-ccdc-28fbdd22e5a1 [Follower]" leader-address= leader-id=
2026-03-29T01:08:10.936Z [INFO]  server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f7516ccb-6b12-8810-e0c3-cce5f3bde450 Address:f7516ccb-6b12-8810-e0c3-cce5f3bde450} {Suffrage:Voter ID:server-7b02a9f6-178e-f349-ccdc-28fbdd22e5a1 Address:7b02a9f6-178e-f349-ccdc-28fbdd22e5a1} {Suffrage:Voter ID:server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412 Address:8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412}]"
2026-03-29T01:08:10.936Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:10.936Z [INFO]  server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: entering follower state: follower="Node at 8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412 [Follower]" leader-address= leader-id=
2026-03-29T01:08:11.770Z [WARN]  server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:11.770Z [INFO]  server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: entering candidate state: node="Node at 8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412 [Candidate]" term=2
2026-03-29T01:08:11.770Z [INFO]  server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: election won: term=2 tally=2
2026-03-29T01:08:11.770Z [INFO]  server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: entering leader state: leader="Node at 8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412 [Leader]"
2026-03-29T01:08:11.770Z [INFO]  server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: added peer, starting replication: peer=server-f7516ccb-6b12-8810-e0c3-cce5f3bde450
2026-03-29T01:08:11.770Z [INFO]  server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: added peer, starting replication: peer=server-7b02a9f6-178e-f349-ccdc-28fbdd22e5a1
2026-03-29T01:08:11.770Z [INFO]  server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: pipelining replication: peer="{Voter server-7b02a9f6-178e-f349-ccdc-28fbdd22e5a1 7b02a9f6-178e-f349-ccdc-28fbdd22e5a1}"
2026-03-29T01:08:11.770Z [INFO]  server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: pipelining replication: peer="{Voter server-f7516ccb-6b12-8810-e0c3-cce5f3bde450 f7516ccb-6b12-8810-e0c3-cce5f3bde450}"
2026-03-29T01:08:12.776Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:11.770327116 +0000 UTC m=+35.558469660 (1 nodes), 834.235619ms from start of poll, 834.57476ms from cluster start. Timeout at 2026-03-29 01:08:12.775333741 +0000 UTC m=+36.563476368, 1.005006708s after stability
2026-03-29T01:08:12.776Z [INFO]  creating new snapshot: path=/tmp/raft3553455875/snapshots/2-14-1774746492776.tmp
2026-03-29T01:08:12.777Z [INFO]  server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: copied to local snapshot: bytes=71
2026-03-29T01:08:12.777Z [INFO]  server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: snapshot restore progress: id=2-14-1774746492776 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:12.777Z [INFO]  server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: restored user snapshot: index=14
2026-03-29T01:08:12.777Z [ERROR] server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: failed to get log: index=3 error="log not found"
2026-03-29T01:08:12.777Z [INFO]  server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: aborting pipeline replication: peer="{Voter server-7b02a9f6-178e-f349-ccdc-28fbdd22e5a1 7b02a9f6-178e-f349-ccdc-28fbdd22e5a1}"
2026-03-29T01:08:12.777Z [ERROR] server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: failed to get log: index=3 error="log not found"
2026-03-29T01:08:12.777Z [INFO]  server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: aborting pipeline replication: peer="{Voter server-f7516ccb-6b12-8810-e0c3-cce5f3bde450 f7516ccb-6b12-8810-e0c3-cce5f3bde450}"
2026-03-29T01:08:12.785Z [ERROR] server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: failed to get log: index=3 error="log not found"
2026-03-29T01:08:12.785Z [INFO]  creating new snapshot: path=/tmp/raft2646958760/snapshots/2-14-1774746492785.tmp
2026-03-29T01:08:12.785Z [ERROR] server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: failed to get log: index=3 error="log not found"
2026-03-29T01:08:12.785Z [INFO]  server-f7516ccb-6b12-8810-e0c3-cce5f3bde450: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:12.786Z [INFO]  creating new snapshot: path=/tmp/raft2336303893/snapshots/2-14-1774746492786.tmp
2026-03-29T01:08:12.786Z [INFO]  server-7b02a9f6-178e-f349-ccdc-28fbdd22e5a1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:12.786Z [INFO]  server-f7516ccb-6b12-8810-e0c3-cce5f3bde450: copied to local snapshot: bytes=71
2026-03-29T01:08:12.786Z [INFO]  server-f7516ccb-6b12-8810-e0c3-cce5f3bde450: snapshot restore progress: id=2-14-1774746492785 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:12.786Z [INFO]  server-7b02a9f6-178e-f349-ccdc-28fbdd22e5a1: copied to local snapshot: bytes=71
2026-03-29T01:08:12.786Z [INFO]  server-f7516ccb-6b12-8810-e0c3-cce5f3bde450: Installed remote snapshot
2026-03-29T01:08:12.786Z [INFO]  server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: pipelining replication: peer="{Voter server-f7516ccb-6b12-8810-e0c3-cce5f3bde450 f7516ccb-6b12-8810-e0c3-cce5f3bde450}"
2026-03-29T01:08:12.786Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:08:12.786Z [INFO]  server-7b02a9f6-178e-f349-ccdc-28fbdd22e5a1: snapshot restore progress: id=2-14-1774746492786 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:12.786Z [INFO]  server-7b02a9f6-178e-f349-ccdc-28fbdd22e5a1: Installed remote snapshot
2026-03-29T01:08:12.786Z [INFO]  server-8c2db8f0-7fa2-0fc4-fcc9-2ffb53abd412: pipelining replication: peer="{Voter server-7b02a9f6-178e-f349-ccdc-28fbdd22e5a1 7b02a9f6-178e-f349-ccdc-28fbdd22e5a1}"
2026-03-29T01:08:13.792Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:08:12.786588494 +0000 UTC m=+36.574731029 (2 nodes), 608ns from start of poll, 1.850836129s from cluster start. Timeout at 2026-03-29 01:08:13.791596712 +0000 UTC m=+37.579739278, 1.005008249s after stability
2026-03-29T01:08:13.809Z [INFO]  server-f3f702b9-f890-58bc-5df8-7f0cc4b167d9: aborting pipeline replication: peer="{Voter server-a920efa6-e176-0caa-4d07-d68578442b74 a920efa6-e176-0caa-4d07-d68578442b74}"
2026-03-29T01:08:13.809Z [INFO]  server-f3f702b9-f890-58bc-5df8-7f0cc4b167d9: aborting pipeline replication: peer="{Voter server-d3705e40-745b-b0f7-e7f6-f53422387c27 d3705e40-745b-b0f7-e7f6-f53422387c27}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true
2026-03-29T01:08:13.811Z [INFO]  server-ed95e139-7c75-9ad3-8418-6861de23829c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed95e139-7c75-9ad3-8418-6861de23829c Address:ed95e139-7c75-9ad3-8418-6861de23829c} {Suffrage:Voter ID:server-39523131-6e78-0c43-b71e-94dd68835c6e Address:39523131-6e78-0c43-b71e-94dd68835c6e} {Suffrage:Voter ID:server-40bf57a0-d5d8-e1d4-16f5-7eaabe188aa0 Address:40bf57a0-d5d8-e1d4-16f5-7eaabe188aa0}]"
2026-03-29T01:08:13.811Z [INFO]  server-ed95e139-7c75-9ad3-8418-6861de23829c: entering follower state: follower="Node at ed95e139-7c75-9ad3-8418-6861de23829c [Follower]" leader-address= leader-id=
2026-03-29T01:08:13.811Z [INFO]  server-39523131-6e78-0c43-b71e-94dd68835c6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed95e139-7c75-9ad3-8418-6861de23829c Address:ed95e139-7c75-9ad3-8418-6861de23829c} {Suffrage:Voter ID:server-39523131-6e78-0c43-b71e-94dd68835c6e Address:39523131-6e78-0c43-b71e-94dd68835c6e} {Suffrage:Voter ID:server-40bf57a0-d5d8-e1d4-16f5-7eaabe188aa0 Address:40bf57a0-d5d8-e1d4-16f5-7eaabe188aa0}]"
2026-03-29T01:08:13.811Z [INFO]  server-39523131-6e78-0c43-b71e-94dd68835c6e: entering follower state: follower="Node at 39523131-6e78-0c43-b71e-94dd68835c6e [Follower]" leader-address= leader-id=
2026-03-29T01:08:13.811Z [INFO]  server-40bf57a0-d5d8-e1d4-16f5-7eaabe188aa0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ed95e139-7c75-9ad3-8418-6861de23829c Address:ed95e139-7c75-9ad3-8418-6861de23829c} {Suffrage:Voter ID:server-39523131-6e78-0c43-b71e-94dd68835c6e Address:39523131-6e78-0c43-b71e-94dd68835c6e} {Suffrage:Voter ID:server-40bf57a0-d5d8-e1d4-16f5-7eaabe188aa0 Address:40bf57a0-d5d8-e1d4-16f5-7eaabe188aa0}]"
2026-03-29T01:08:13.811Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:13.811Z [INFO]  server-40bf57a0-d5d8-e1d4-16f5-7eaabe188aa0: entering follower state: follower="Node at 40bf57a0-d5d8-e1d4-16f5-7eaabe188aa0 [Follower]" leader-address= leader-id=
2026-03-29T01:08:14.349Z [WARN]  server-39523131-6e78-0c43-b71e-94dd68835c6e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:14.349Z [INFO]  server-39523131-6e78-0c43-b71e-94dd68835c6e: entering candidate state: node="Node at 39523131-6e78-0c43-b71e-94dd68835c6e [Candidate]" term=2
2026-03-29T01:08:14.349Z [INFO]  server-39523131-6e78-0c43-b71e-94dd68835c6e: election won: term=2 tally=2
2026-03-29T01:08:14.349Z [INFO]  server-39523131-6e78-0c43-b71e-94dd68835c6e: entering leader state: leader="Node at 39523131-6e78-0c43-b71e-94dd68835c6e [Leader]"
2026-03-29T01:08:14.349Z [INFO]  server-39523131-6e78-0c43-b71e-94dd68835c6e: added peer, starting replication: peer=server-ed95e139-7c75-9ad3-8418-6861de23829c
2026-03-29T01:08:14.349Z [INFO]  server-39523131-6e78-0c43-b71e-94dd68835c6e: added peer, starting replication: peer=server-40bf57a0-d5d8-e1d4-16f5-7eaabe188aa0
2026-03-29T01:08:14.349Z [INFO]  server-39523131-6e78-0c43-b71e-94dd68835c6e: pipelining replication: peer="{Voter server-ed95e139-7c75-9ad3-8418-6861de23829c ed95e139-7c75-9ad3-8418-6861de23829c}"
2026-03-29T01:08:14.350Z [INFO]  server-39523131-6e78-0c43-b71e-94dd68835c6e: pipelining replication: peer="{Voter server-40bf57a0-d5d8-e1d4-16f5-7eaabe188aa0 40bf57a0-d5d8-e1d4-16f5-7eaabe188aa0}"
2026-03-29T01:08:15.355Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:14.34963067 +0000 UTC m=+38.137773208 (1 nodes), 538.132911ms from start of poll, 538.463685ms from cluster start. Timeout at 2026-03-29 01:08:15.354635231 +0000 UTC m=+39.142777782, 1.005004574s after stability
2026-03-29T01:08:15.355Z [INFO]  server-39523131-6e78-0c43-b71e-94dd68835c6e: starting snapshot up to: index=12
2026-03-29T01:08:15.355Z [INFO]  creating new snapshot: path=/tmp/raft458571571/snapshots/2-12-1774746495355.tmp
2026-03-29T01:08:15.356Z [INFO]  server-39523131-6e78-0c43-b71e-94dd68835c6e: snapshot complete up to: index=12
2026-03-29T01:08:15.357Z [INFO]  server-00525b01-3829-f1a6-2c44-3bf9693053c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00525b01-3829-f1a6-2c44-3bf9693053c5 Address:00525b01-3829-f1a6-2c44-3bf9693053c5} {Suffrage:Voter ID:server-f318b947-f0b2-c13b-8951-8b4ee4a9679a Address:f318b947-f0b2-c13b-8951-8b4ee4a9679a} {Suffrage:Voter ID:server-45eeb054-8a28-4586-1387-10bdeacb1900 Address:45eeb054-8a28-4586-1387-10bdeacb1900}]"
2026-03-29T01:08:15.357Z [INFO]  server-00525b01-3829-f1a6-2c44-3bf9693053c5: entering follower state: follower="Node at 00525b01-3829-f1a6-2c44-3bf9693053c5 [Follower]" leader-address= leader-id=
2026-03-29T01:08:15.357Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00525b01-3829-f1a6-2c44-3bf9693053c5 Address:00525b01-3829-f1a6-2c44-3bf9693053c5} {Suffrage:Voter ID:server-f318b947-f0b2-c13b-8951-8b4ee4a9679a Address:f318b947-f0b2-c13b-8951-8b4ee4a9679a} {Suffrage:Voter ID:server-45eeb054-8a28-4586-1387-10bdeacb1900 Address:45eeb054-8a28-4586-1387-10bdeacb1900}]"
2026-03-29T01:08:15.357Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: entering follower state: follower="Node at f318b947-f0b2-c13b-8951-8b4ee4a9679a [Follower]" leader-address= leader-id=
2026-03-29T01:08:15.357Z [INFO]  server-45eeb054-8a28-4586-1387-10bdeacb1900: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00525b01-3829-f1a6-2c44-3bf9693053c5 Address:00525b01-3829-f1a6-2c44-3bf9693053c5} {Suffrage:Voter ID:server-f318b947-f0b2-c13b-8951-8b4ee4a9679a Address:f318b947-f0b2-c13b-8951-8b4ee4a9679a} {Suffrage:Voter ID:server-45eeb054-8a28-4586-1387-10bdeacb1900 Address:45eeb054-8a28-4586-1387-10bdeacb1900}]"
2026-03-29T01:08:15.357Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:15.357Z [INFO]  server-45eeb054-8a28-4586-1387-10bdeacb1900: entering follower state: follower="Node at 45eeb054-8a28-4586-1387-10bdeacb1900 [Follower]" leader-address= leader-id=
2026-03-29T01:08:16.007Z [WARN]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:16.007Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: entering candidate state: node="Node at f318b947-f0b2-c13b-8951-8b4ee4a9679a [Candidate]" term=2
2026-03-29T01:08:16.008Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: election won: term=2 tally=2
2026-03-29T01:08:16.008Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: entering leader state: leader="Node at f318b947-f0b2-c13b-8951-8b4ee4a9679a [Leader]"
2026-03-29T01:08:16.008Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: added peer, starting replication: peer=server-00525b01-3829-f1a6-2c44-3bf9693053c5
2026-03-29T01:08:16.008Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: added peer, starting replication: peer=server-45eeb054-8a28-4586-1387-10bdeacb1900
2026-03-29T01:08:16.008Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: pipelining replication: peer="{Voter server-45eeb054-8a28-4586-1387-10bdeacb1900 45eeb054-8a28-4586-1387-10bdeacb1900}"
2026-03-29T01:08:16.008Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: pipelining replication: peer="{Voter server-00525b01-3829-f1a6-2c44-3bf9693053c5 00525b01-3829-f1a6-2c44-3bf9693053c5}"
2026-03-29T01:08:17.013Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:16.008032374 +0000 UTC m=+39.796174909 (1 nodes), 650.499013ms from start of poll, 650.846385ms from cluster start. Timeout at 2026-03-29 01:08:17.013037299 +0000 UTC m=+40.801179827, 1.005004918s after stability
2026-03-29T01:08:17.013Z [INFO]  creating new snapshot: path=/tmp/raft28690855/snapshots/2-14-1774746497013.tmp
2026-03-29T01:08:17.014Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: copied to local snapshot: bytes=71
2026-03-29T01:08:17.014Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: snapshot restore progress: id=2-14-1774746497013 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:17.014Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: removing all old logs from log store
2026-03-29T01:08:17.014Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: compacting logs: from=1 to=2
2026-03-29T01:08:17.014Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: restored user snapshot: index=14
2026-03-29T01:08:17.014Z [ERROR] server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: failed to get log: index=2 error="log not found"
2026-03-29T01:08:17.014Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: aborting pipeline replication: peer="{Voter server-45eeb054-8a28-4586-1387-10bdeacb1900 45eeb054-8a28-4586-1387-10bdeacb1900}"
2026-03-29T01:08:17.014Z [ERROR] server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: failed to get log: index=2 error="log not found"
2026-03-29T01:08:17.015Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: aborting pipeline replication: peer="{Voter server-00525b01-3829-f1a6-2c44-3bf9693053c5 00525b01-3829-f1a6-2c44-3bf9693053c5}"
2026-03-29T01:08:17.022Z [ERROR] server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: failed to get log: index=2 error="log not found"
2026-03-29T01:08:17.022Z [INFO]  creating new snapshot: path=/tmp/raft964313424/snapshots/2-14-1774746497022.tmp
2026-03-29T01:08:17.022Z [INFO]  server-45eeb054-8a28-4586-1387-10bdeacb1900: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:17.022Z [INFO]  server-45eeb054-8a28-4586-1387-10bdeacb1900: copied to local snapshot: bytes=71
2026-03-29T01:08:17.023Z [INFO]  server-45eeb054-8a28-4586-1387-10bdeacb1900: snapshot restore progress: id=2-14-1774746497022 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:17.023Z [INFO]  server-45eeb054-8a28-4586-1387-10bdeacb1900: removing all old logs from log store
2026-03-29T01:08:17.023Z [INFO]  server-45eeb054-8a28-4586-1387-10bdeacb1900: compacting logs: from=1 to=2
2026-03-29T01:08:17.023Z [INFO]  server-45eeb054-8a28-4586-1387-10bdeacb1900: Installed remote snapshot
2026-03-29T01:08:17.023Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: pipelining replication: peer="{Voter server-45eeb054-8a28-4586-1387-10bdeacb1900 45eeb054-8a28-4586-1387-10bdeacb1900}"
2026-03-29T01:08:17.023Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:08:17.024Z [ERROR] server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: failed to get log: index=2 error="log not found"
2026-03-29T01:08:17.024Z [INFO]  creating new snapshot: path=/tmp/raft2164124119/snapshots/2-14-1774746497024.tmp
2026-03-29T01:08:17.024Z [INFO]  server-00525b01-3829-f1a6-2c44-3bf9693053c5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:17.025Z [INFO]  server-00525b01-3829-f1a6-2c44-3bf9693053c5: copied to local snapshot: bytes=71
2026-03-29T01:08:17.025Z [INFO]  server-00525b01-3829-f1a6-2c44-3bf9693053c5: snapshot restore progress: id=2-14-1774746497024 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:17.025Z [INFO]  server-00525b01-3829-f1a6-2c44-3bf9693053c5: removing all old logs from log store
2026-03-29T01:08:17.025Z [INFO]  server-00525b01-3829-f1a6-2c44-3bf9693053c5: compacting logs: from=1 to=2
2026-03-29T01:08:17.025Z [INFO]  server-00525b01-3829-f1a6-2c44-3bf9693053c5: Installed remote snapshot
2026-03-29T01:08:17.025Z [INFO]  server-f318b947-f0b2-c13b-8951-8b4ee4a9679a: pipelining replication: peer="{Voter server-00525b01-3829-f1a6-2c44-3bf9693053c5 00525b01-3829-f1a6-2c44-3bf9693053c5}"
2026-03-29T01:08:18.029Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:08:17.023229334 +0000 UTC m=+40.811371872 (2 nodes), 1.045µs from start of poll, 1.666043348s from cluster start. Timeout at 2026-03-29 01:08:18.028246072 +0000 UTC m=+41.816388672, 1.0050168s after stability
2026-03-29T01:08:18.040Z [INFO]  server-39523131-6e78-0c43-b71e-94dd68835c6e: aborting pipeline replication: peer="{Voter server-40bf57a0-d5d8-e1d4-16f5-7eaabe188aa0 40bf57a0-d5d8-e1d4-16f5-7eaabe188aa0}"
2026-03-29T01:08:18.040Z [INFO]  server-39523131-6e78-0c43-b71e-94dd68835c6e: aborting pipeline replication: peer="{Voter server-ed95e139-7c75-9ad3-8418-6861de23829c ed95e139-7c75-9ad3-8418-6861de23829c}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false
2026-03-29T01:08:18.041Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27448ec0-be39-07db-ff45-a98773869cbc Address:27448ec0-be39-07db-ff45-a98773869cbc} {Suffrage:Voter ID:server-9cccf397-9a36-5efc-0450-ed9f7981b7d4 Address:9cccf397-9a36-5efc-0450-ed9f7981b7d4} {Suffrage:Voter ID:server-469f5b1a-d231-afca-5079-cb26621bd3c5 Address:469f5b1a-d231-afca-5079-cb26621bd3c5}]"
2026-03-29T01:08:18.041Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: entering follower state: follower="Node at 27448ec0-be39-07db-ff45-a98773869cbc [Follower]" leader-address= leader-id=
2026-03-29T01:08:18.042Z [INFO]  server-9cccf397-9a36-5efc-0450-ed9f7981b7d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27448ec0-be39-07db-ff45-a98773869cbc Address:27448ec0-be39-07db-ff45-a98773869cbc} {Suffrage:Voter ID:server-9cccf397-9a36-5efc-0450-ed9f7981b7d4 Address:9cccf397-9a36-5efc-0450-ed9f7981b7d4} {Suffrage:Voter ID:server-469f5b1a-d231-afca-5079-cb26621bd3c5 Address:469f5b1a-d231-afca-5079-cb26621bd3c5}]"
2026-03-29T01:08:18.042Z [INFO]  server-9cccf397-9a36-5efc-0450-ed9f7981b7d4: entering follower state: follower="Node at 9cccf397-9a36-5efc-0450-ed9f7981b7d4 [Follower]" leader-address= leader-id=
2026-03-29T01:08:18.042Z [INFO]  server-469f5b1a-d231-afca-5079-cb26621bd3c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-27448ec0-be39-07db-ff45-a98773869cbc Address:27448ec0-be39-07db-ff45-a98773869cbc} {Suffrage:Voter ID:server-9cccf397-9a36-5efc-0450-ed9f7981b7d4 Address:9cccf397-9a36-5efc-0450-ed9f7981b7d4} {Suffrage:Voter ID:server-469f5b1a-d231-afca-5079-cb26621bd3c5 Address:469f5b1a-d231-afca-5079-cb26621bd3c5}]"
2026-03-29T01:08:18.042Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:18.042Z [INFO]  server-469f5b1a-d231-afca-5079-cb26621bd3c5: entering follower state: follower="Node at 469f5b1a-d231-afca-5079-cb26621bd3c5 [Follower]" leader-address= leader-id=
2026-03-29T01:08:18.572Z [WARN]  server-27448ec0-be39-07db-ff45-a98773869cbc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:18.572Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: entering candidate state: node="Node at 27448ec0-be39-07db-ff45-a98773869cbc [Candidate]" term=2
2026-03-29T01:08:18.572Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: election won: term=2 tally=2
2026-03-29T01:08:18.572Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: entering leader state: leader="Node at 27448ec0-be39-07db-ff45-a98773869cbc [Leader]"
2026-03-29T01:08:18.573Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: added peer, starting replication: peer=server-9cccf397-9a36-5efc-0450-ed9f7981b7d4
2026-03-29T01:08:18.573Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: added peer, starting replication: peer=server-469f5b1a-d231-afca-5079-cb26621bd3c5
2026-03-29T01:08:18.573Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: pipelining replication: peer="{Voter server-469f5b1a-d231-afca-5079-cb26621bd3c5 469f5b1a-d231-afca-5079-cb26621bd3c5}"
2026-03-29T01:08:18.573Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: pipelining replication: peer="{Voter server-9cccf397-9a36-5efc-0450-ed9f7981b7d4 9cccf397-9a36-5efc-0450-ed9f7981b7d4}"
2026-03-29T01:08:19.578Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:18.57302584 +0000 UTC m=+42.361168440 (1 nodes), 530.866439ms from start of poll, 531.188786ms from cluster start. Timeout at 2026-03-29 01:08:19.578027894 +0000 UTC m=+43.366170505, 1.005002065s after stability
2026-03-29T01:08:19.580Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: starting snapshot up to: index=12
2026-03-29T01:08:19.580Z [INFO]  creating new snapshot: path=/tmp/raft3736267589/snapshots/2-12-1774746499580.tmp
2026-03-29T01:08:19.580Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: snapshot complete up to: index=12
2026-03-29T01:08:19.581Z [INFO]  creating new snapshot: path=/tmp/raft3736267589/snapshots/2-23-1774746499581.tmp
2026-03-29T01:08:19.581Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: copied to local snapshot: bytes=71
2026-03-29T01:08:19.581Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: snapshot restore progress: id=2-23-1774746499581 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:19.581Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: restored user snapshot: index=23
2026-03-29T01:08:19.582Z [ERROR] server-27448ec0-be39-07db-ff45-a98773869cbc: failed to get log: index=23 error="log not found"
2026-03-29T01:08:19.582Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: aborting pipeline replication: peer="{Voter server-469f5b1a-d231-afca-5079-cb26621bd3c5 469f5b1a-d231-afca-5079-cb26621bd3c5}"
2026-03-29T01:08:19.582Z [ERROR] server-27448ec0-be39-07db-ff45-a98773869cbc: failed to get log: index=23 error="log not found"
2026-03-29T01:08:19.582Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: aborting pipeline replication: peer="{Voter server-9cccf397-9a36-5efc-0450-ed9f7981b7d4 9cccf397-9a36-5efc-0450-ed9f7981b7d4}"
2026-03-29T01:08:19.589Z [ERROR] server-27448ec0-be39-07db-ff45-a98773869cbc: failed to get log: index=23 error="log not found"
2026-03-29T01:08:19.589Z [INFO]  creating new snapshot: path=/tmp/raft805927267/snapshots/2-23-1774746499589.tmp
2026-03-29T01:08:19.589Z [INFO]  server-9cccf397-9a36-5efc-0450-ed9f7981b7d4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:19.590Z [INFO]  server-9cccf397-9a36-5efc-0450-ed9f7981b7d4: copied to local snapshot: bytes=71
2026-03-29T01:08:19.590Z [INFO]  server-9cccf397-9a36-5efc-0450-ed9f7981b7d4: snapshot restore progress: id=2-23-1774746499589 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:19.590Z [INFO]  server-9cccf397-9a36-5efc-0450-ed9f7981b7d4: Installed remote snapshot
2026-03-29T01:08:19.590Z [WARN]  server-9cccf397-9a36-5efc-0450-ed9f7981b7d4: skipping application of old log: index=22
2026-03-29T01:08:19.590Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: pipelining replication: peer="{Voter server-9cccf397-9a36-5efc-0450-ed9f7981b7d4 9cccf397-9a36-5efc-0450-ed9f7981b7d4}"
2026-03-29T01:08:19.590Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:08:19.592Z [ERROR] server-27448ec0-be39-07db-ff45-a98773869cbc: failed to get log: index=23 error="log not found"
2026-03-29T01:08:19.593Z [INFO]  creating new snapshot: path=/tmp/raft783732827/snapshots/2-23-1774746499593.tmp
2026-03-29T01:08:19.593Z [INFO]  server-469f5b1a-d231-afca-5079-cb26621bd3c5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:19.593Z [INFO]  server-469f5b1a-d231-afca-5079-cb26621bd3c5: copied to local snapshot: bytes=71
2026-03-29T01:08:19.593Z [INFO]  server-469f5b1a-d231-afca-5079-cb26621bd3c5: snapshot restore progress: id=2-23-1774746499593 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:19.593Z [INFO]  server-469f5b1a-d231-afca-5079-cb26621bd3c5: Installed remote snapshot
2026-03-29T01:08:19.594Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: pipelining replication: peer="{Voter server-469f5b1a-d231-afca-5079-cb26621bd3c5 469f5b1a-d231-afca-5079-cb26621bd3c5}"
2026-03-29T01:08:20.595Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:08:19.590680636 +0000 UTC m=+43.378823172 (2 nodes), 597ns from start of poll, 1.548843518s from cluster start. Timeout at 2026-03-29 01:08:20.595684049 +0000 UTC m=+44.383826592, 1.00500342s after stability
2026-03-29T01:08:20.607Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: aborting pipeline replication: peer="{Voter server-469f5b1a-d231-afca-5079-cb26621bd3c5 469f5b1a-d231-afca-5079-cb26621bd3c5}"
2026-03-29T01:08:20.607Z [INFO]  server-27448ec0-be39-07db-ff45-a98773869cbc: aborting pipeline replication: peer="{Voter server-9cccf397-9a36-5efc-0450-ed9f7981b7d4 9cccf397-9a36-5efc-0450-ed9f7981b7d4}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false
2026-03-29T01:08:20.608Z [INFO]  server-7be4ca3e-b0f9-a161-8db3-706edad71d1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7be4ca3e-b0f9-a161-8db3-706edad71d1f Address:7be4ca3e-b0f9-a161-8db3-706edad71d1f} {Suffrage:Voter ID:server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603 Address:5924d7f7-0ea4-9956-3c7c-ba1cc7c52603} {Suffrage:Voter ID:server-872d513d-5e48-297e-500d-912e1936d27c Address:872d513d-5e48-297e-500d-912e1936d27c}]"
2026-03-29T01:08:20.608Z [INFO]  server-7be4ca3e-b0f9-a161-8db3-706edad71d1f: entering follower state: follower="Node at 7be4ca3e-b0f9-a161-8db3-706edad71d1f [Follower]" leader-address= leader-id=
2026-03-29T01:08:20.608Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7be4ca3e-b0f9-a161-8db3-706edad71d1f Address:7be4ca3e-b0f9-a161-8db3-706edad71d1f} {Suffrage:Voter ID:server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603 Address:5924d7f7-0ea4-9956-3c7c-ba1cc7c52603} {Suffrage:Voter ID:server-872d513d-5e48-297e-500d-912e1936d27c Address:872d513d-5e48-297e-500d-912e1936d27c}]"
2026-03-29T01:08:20.608Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: entering follower state: follower="Node at 5924d7f7-0ea4-9956-3c7c-ba1cc7c52603 [Follower]" leader-address= leader-id=
2026-03-29T01:08:20.608Z [INFO]  server-872d513d-5e48-297e-500d-912e1936d27c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7be4ca3e-b0f9-a161-8db3-706edad71d1f Address:7be4ca3e-b0f9-a161-8db3-706edad71d1f} {Suffrage:Voter ID:server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603 Address:5924d7f7-0ea4-9956-3c7c-ba1cc7c52603} {Suffrage:Voter ID:server-872d513d-5e48-297e-500d-912e1936d27c Address:872d513d-5e48-297e-500d-912e1936d27c}]"
2026-03-29T01:08:20.608Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:20.608Z [INFO]  server-872d513d-5e48-297e-500d-912e1936d27c: entering follower state: follower="Node at 872d513d-5e48-297e-500d-912e1936d27c [Follower]" leader-address= leader-id=
2026-03-29T01:08:21.184Z [WARN]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:21.185Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: entering candidate state: node="Node at 5924d7f7-0ea4-9956-3c7c-ba1cc7c52603 [Candidate]" term=2
2026-03-29T01:08:21.185Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: election won: term=2 tally=2
2026-03-29T01:08:21.185Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: entering leader state: leader="Node at 5924d7f7-0ea4-9956-3c7c-ba1cc7c52603 [Leader]"
2026-03-29T01:08:21.185Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: added peer, starting replication: peer=server-7be4ca3e-b0f9-a161-8db3-706edad71d1f
2026-03-29T01:08:21.185Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: added peer, starting replication: peer=server-872d513d-5e48-297e-500d-912e1936d27c
2026-03-29T01:08:21.185Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: pipelining replication: peer="{Voter server-7be4ca3e-b0f9-a161-8db3-706edad71d1f 7be4ca3e-b0f9-a161-8db3-706edad71d1f}"
2026-03-29T01:08:21.185Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: pipelining replication: peer="{Voter server-872d513d-5e48-297e-500d-912e1936d27c 872d513d-5e48-297e-500d-912e1936d27c}"
2026-03-29T01:08:22.191Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:21.185171204 +0000 UTC m=+44.973313737 (1 nodes), 576.437451ms from start of poll, 576.728365ms from cluster start. Timeout at 2026-03-29 01:08:22.190177293 +0000 UTC m=+45.978319822, 1.005006085s after stability
2026-03-29T01:08:22.191Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: starting snapshot up to: index=12
2026-03-29T01:08:22.191Z [INFO]  creating new snapshot: path=/tmp/raft1065354925/snapshots/2-12-1774746502191.tmp
2026-03-29T01:08:22.192Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: snapshot complete up to: index=12
2026-03-29T01:08:22.193Z [INFO]  creating new snapshot: path=/tmp/raft1065354925/snapshots/2-23-1774746502193.tmp
2026-03-29T01:08:22.193Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: copied to local snapshot: bytes=71
2026-03-29T01:08:22.193Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: snapshot restore progress: id=2-23-1774746502193 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:22.194Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: removing all old logs from log store
2026-03-29T01:08:22.194Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: compacting logs: from=1 to=22
2026-03-29T01:08:22.194Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: restored user snapshot: index=23
2026-03-29T01:08:22.194Z [ERROR] server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: failed to get log: index=22 error="log not found"
2026-03-29T01:08:22.194Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: aborting pipeline replication: peer="{Voter server-872d513d-5e48-297e-500d-912e1936d27c 872d513d-5e48-297e-500d-912e1936d27c}"
2026-03-29T01:08:22.194Z [ERROR] server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: failed to get log: index=22 error="log not found"
2026-03-29T01:08:22.194Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: aborting pipeline replication: peer="{Voter server-7be4ca3e-b0f9-a161-8db3-706edad71d1f 7be4ca3e-b0f9-a161-8db3-706edad71d1f}"
2026-03-29T01:08:22.199Z [ERROR] server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: failed to get log: index=22 error="log not found"
2026-03-29T01:08:22.199Z [INFO]  creating new snapshot: path=/tmp/raft2080071841/snapshots/2-23-1774746502199.tmp
2026-03-29T01:08:22.199Z [INFO]  server-872d513d-5e48-297e-500d-912e1936d27c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:22.200Z [INFO]  server-872d513d-5e48-297e-500d-912e1936d27c: copied to local snapshot: bytes=71
2026-03-29T01:08:22.200Z [INFO]  server-872d513d-5e48-297e-500d-912e1936d27c: snapshot restore progress: id=2-23-1774746502199 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:22.200Z [INFO]  server-872d513d-5e48-297e-500d-912e1936d27c: removing all old logs from log store
2026-03-29T01:08:22.200Z [INFO]  server-872d513d-5e48-297e-500d-912e1936d27c: compacting logs: from=1 to=22
2026-03-29T01:08:22.200Z [INFO]  server-872d513d-5e48-297e-500d-912e1936d27c: Installed remote snapshot
2026-03-29T01:08:22.200Z [WARN]  server-872d513d-5e48-297e-500d-912e1936d27c: skipping application of old log: index=22
2026-03-29T01:08:22.200Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: pipelining replication: peer="{Voter server-872d513d-5e48-297e-500d-912e1936d27c 872d513d-5e48-297e-500d-912e1936d27c}"
2026-03-29T01:08:22.200Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:08:22.204Z [ERROR] server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: failed to get log: index=22 error="log not found"
2026-03-29T01:08:22.205Z [INFO]  creating new snapshot: path=/tmp/raft505176377/snapshots/2-23-1774746502205.tmp
2026-03-29T01:08:22.205Z [INFO]  server-7be4ca3e-b0f9-a161-8db3-706edad71d1f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:22.205Z [INFO]  server-7be4ca3e-b0f9-a161-8db3-706edad71d1f: copied to local snapshot: bytes=71
2026-03-29T01:08:22.206Z [INFO]  server-7be4ca3e-b0f9-a161-8db3-706edad71d1f: snapshot restore progress: id=2-23-1774746502205 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:22.206Z [INFO]  server-7be4ca3e-b0f9-a161-8db3-706edad71d1f: removing all old logs from log store
2026-03-29T01:08:22.206Z [INFO]  server-7be4ca3e-b0f9-a161-8db3-706edad71d1f: compacting logs: from=1 to=22
2026-03-29T01:08:22.206Z [INFO]  server-7be4ca3e-b0f9-a161-8db3-706edad71d1f: Installed remote snapshot
2026-03-29T01:08:22.206Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: pipelining replication: peer="{Voter server-7be4ca3e-b0f9-a161-8db3-706edad71d1f 7be4ca3e-b0f9-a161-8db3-706edad71d1f}"
2026-03-29T01:08:23.206Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:08:22.200442141 +0000 UTC m=+45.988584683 (2 nodes), 577ns from start of poll, 1.591999311s from cluster start. Timeout at 2026-03-29 01:08:23.205457161 +0000 UTC m=+46.993599764, 1.005015081s after stability
2026-03-29T01:08:23.218Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: aborting pipeline replication: peer="{Voter server-7be4ca3e-b0f9-a161-8db3-706edad71d1f 7be4ca3e-b0f9-a161-8db3-706edad71d1f}"
2026-03-29T01:08:23.218Z [INFO]  server-5924d7f7-0ea4-9956-3c7c-ba1cc7c52603: aborting pipeline replication: peer="{Voter server-872d513d-5e48-297e-500d-912e1936d27c 872d513d-5e48-297e-500d-912e1936d27c}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true
2026-03-29T01:08:23.219Z [INFO]  server-35a4641a-657c-016a-a3a7-0d4d3a0b7ab0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35a4641a-657c-016a-a3a7-0d4d3a0b7ab0 Address:35a4641a-657c-016a-a3a7-0d4d3a0b7ab0} {Suffrage:Voter ID:server-baece70b-f353-ee77-2fb1-bae43a0840be Address:baece70b-f353-ee77-2fb1-bae43a0840be} {Suffrage:Voter ID:server-24ff37dd-24d9-53db-643f-8f08c5d02a7e Address:24ff37dd-24d9-53db-643f-8f08c5d02a7e}]"
2026-03-29T01:08:23.219Z [INFO]  server-35a4641a-657c-016a-a3a7-0d4d3a0b7ab0: entering follower state: follower="Node at 35a4641a-657c-016a-a3a7-0d4d3a0b7ab0 [Follower]" leader-address= leader-id=
2026-03-29T01:08:23.219Z [INFO]  server-baece70b-f353-ee77-2fb1-bae43a0840be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35a4641a-657c-016a-a3a7-0d4d3a0b7ab0 Address:35a4641a-657c-016a-a3a7-0d4d3a0b7ab0} {Suffrage:Voter ID:server-baece70b-f353-ee77-2fb1-bae43a0840be Address:baece70b-f353-ee77-2fb1-bae43a0840be} {Suffrage:Voter ID:server-24ff37dd-24d9-53db-643f-8f08c5d02a7e Address:24ff37dd-24d9-53db-643f-8f08c5d02a7e}]"
2026-03-29T01:08:23.219Z [INFO]  server-baece70b-f353-ee77-2fb1-bae43a0840be: entering follower state: follower="Node at baece70b-f353-ee77-2fb1-bae43a0840be [Follower]" leader-address= leader-id=
2026-03-29T01:08:23.219Z [INFO]  server-24ff37dd-24d9-53db-643f-8f08c5d02a7e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35a4641a-657c-016a-a3a7-0d4d3a0b7ab0 Address:35a4641a-657c-016a-a3a7-0d4d3a0b7ab0} {Suffrage:Voter ID:server-baece70b-f353-ee77-2fb1-bae43a0840be Address:baece70b-f353-ee77-2fb1-bae43a0840be} {Suffrage:Voter ID:server-24ff37dd-24d9-53db-643f-8f08c5d02a7e Address:24ff37dd-24d9-53db-643f-8f08c5d02a7e}]"
2026-03-29T01:08:23.219Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:23.219Z [INFO]  server-24ff37dd-24d9-53db-643f-8f08c5d02a7e: entering follower state: follower="Node at 24ff37dd-24d9-53db-643f-8f08c5d02a7e [Follower]" leader-address= leader-id=
2026-03-29T01:08:23.857Z [WARN]  server-35a4641a-657c-016a-a3a7-0d4d3a0b7ab0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:23.857Z [INFO]  server-35a4641a-657c-016a-a3a7-0d4d3a0b7ab0: entering candidate state: node="Node at 35a4641a-657c-016a-a3a7-0d4d3a0b7ab0 [Candidate]" term=2
2026-03-29T01:08:23.857Z [INFO]  server-35a4641a-657c-016a-a3a7-0d4d3a0b7ab0: election won: term=2 tally=2
2026-03-29T01:08:23.857Z [INFO]  server-35a4641a-657c-016a-a3a7-0d4d3a0b7ab0: entering leader state: leader="Node at 35a4641a-657c-016a-a3a7-0d4d3a0b7ab0 [Leader]"
2026-03-29T01:08:23.857Z [INFO]  server-35a4641a-657c-016a-a3a7-0d4d3a0b7ab0: added peer, starting replication: peer=server-baece70b-f353-ee77-2fb1-bae43a0840be
2026-03-29T01:08:23.857Z [INFO]  server-35a4641a-657c-016a-a3a7-0d4d3a0b7ab0: added peer, starting replication: peer=server-24ff37dd-24d9-53db-643f-8f08c5d02a7e
2026-03-29T01:08:23.857Z [INFO]  server-35a4641a-657c-016a-a3a7-0d4d3a0b7ab0: pipelining replication: peer="{Voter server-baece70b-f353-ee77-2fb1-bae43a0840be baece70b-f353-ee77-2fb1-bae43a0840be}"
2026-03-29T01:08:23.857Z [INFO]  server-35a4641a-657c-016a-a3a7-0d4d3a0b7ab0: pipelining replication: peer="{Voter server-24ff37dd-24d9-53db-643f-8f08c5d02a7e 24ff37dd-24d9-53db-643f-8f08c5d02a7e}"
2026-03-29T01:08:24.862Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:23.857560792 +0000 UTC m=+47.645703360 (1 nodes), 637.933051ms from start of poll, 638.255749ms from cluster start. Timeout at 2026-03-29 01:08:24.862571244 +0000 UTC m=+48.650713828, 1.005010468s after stability
2026-03-29T01:08:24.863Z [INFO]  server-35a4641a-657c-016a-a3a7-0d4d3a0b7ab0: starting snapshot up to: index=12
2026-03-29T01:08:24.863Z [INFO]  creating new snapshot: path=/tmp/raft2417381407/snapshots/2-12-1774746504863.tmp
2026-03-29T01:08:24.866Z [INFO]  server-35a4641a-657c-016a-a3a7-0d4d3a0b7ab0: snapshot complete up to: index=12
2026-03-29T01:08:24.867Z [INFO]  server-77018626-686a-8b5a-4ad5-4cae25e9c640: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77018626-686a-8b5a-4ad5-4cae25e9c640 Address:77018626-686a-8b5a-4ad5-4cae25e9c640} {Suffrage:Voter ID:server-382bad15-0d56-66bd-0caa-854eca55005b Address:382bad15-0d56-66bd-0caa-854eca55005b} {Suffrage:Voter ID:server-86d477dd-10f9-9a42-4e3e-42771df1a3f5 Address:86d477dd-10f9-9a42-4e3e-42771df1a3f5}]"
2026-03-29T01:08:24.867Z [INFO]  server-77018626-686a-8b5a-4ad5-4cae25e9c640: entering follower state: follower="Node at 77018626-686a-8b5a-4ad5-4cae25e9c640 [Follower]" leader-address= leader-id=
2026-03-29T01:08:24.867Z [INFO]  server-382bad15-0d56-66bd-0caa-854eca55005b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77018626-686a-8b5a-4ad5-4cae25e9c640 Address:77018626-686a-8b5a-4ad5-4cae25e9c640} {Suffrage:Voter ID:server-382bad15-0d56-66bd-0caa-854eca55005b Address:382bad15-0d56-66bd-0caa-854eca55005b} {Suffrage:Voter ID:server-86d477dd-10f9-9a42-4e3e-42771df1a3f5 Address:86d477dd-10f9-9a42-4e3e-42771df1a3f5}]"
2026-03-29T01:08:24.867Z [INFO]  server-382bad15-0d56-66bd-0caa-854eca55005b: entering follower state: follower="Node at 382bad15-0d56-66bd-0caa-854eca55005b [Follower]" leader-address= leader-id=
2026-03-29T01:08:24.867Z [INFO]  server-86d477dd-10f9-9a42-4e3e-42771df1a3f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77018626-686a-8b5a-4ad5-4cae25e9c640 Address:77018626-686a-8b5a-4ad5-4cae25e9c640} {Suffrage:Voter ID:server-382bad15-0d56-66bd-0caa-854eca55005b Address:382bad15-0d56-66bd-0caa-854eca55005b} {Suffrage:Voter ID:server-86d477dd-10f9-9a42-4e3e-42771df1a3f5 Address:86d477dd-10f9-9a42-4e3e-42771df1a3f5}]"
2026-03-29T01:08:24.867Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:24.867Z [INFO]  server-86d477dd-10f9-9a42-4e3e-42771df1a3f5: entering follower state: follower="Node at 86d477dd-10f9-9a42-4e3e-42771df1a3f5 [Follower]" leader-address= leader-id=
2026-03-29T01:08:25.507Z [WARN]  server-77018626-686a-8b5a-4ad5-4cae25e9c640: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:25.508Z [INFO]  server-77018626-686a-8b5a-4ad5-4cae25e9c640: entering candidate state: node="Node at 77018626-686a-8b5a-4ad5-4cae25e9c640 [Candidate]" term=2
2026-03-29T01:08:25.508Z [INFO]  server-77018626-686a-8b5a-4ad5-4cae25e9c640: election won: term=2 tally=2
2026-03-29T01:08:25.508Z [INFO]  server-77018626-686a-8b5a-4ad5-4cae25e9c640: entering leader state: leader="Node at 77018626-686a-8b5a-4ad5-4cae25e9c640 [Leader]"
2026-03-29T01:08:25.508Z [INFO]  server-77018626-686a-8b5a-4ad5-4cae25e9c640: added peer, starting replication: peer=server-382bad15-0d56-66bd-0caa-854eca55005b
2026-03-29T01:08:25.508Z [INFO]  server-77018626-686a-8b5a-4ad5-4cae25e9c640: added peer, starting replication: peer=server-86d477dd-10f9-9a42-4e3e-42771df1a3f5
2026-03-29T01:08:25.508Z [INFO]  server-77018626-686a-8b5a-4ad5-4cae25e9c640: pipelining replication: peer="{Voter server-86d477dd-10f9-9a42-4e3e-42771df1a3f5 86d477dd-10f9-9a42-4e3e-42771df1a3f5}"
2026-03-29T01:08:25.508Z [INFO]  server-77018626-686a-8b5a-4ad5-4cae25e9c640: pipelining replication: peer="{Voter server-382bad15-0d56-66bd-0caa-854eca55005b 382bad15-0d56-66bd-0caa-854eca55005b}"
2026-03-29T01:08:26.513Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:25.50826017 +0000 UTC m=+49.296402705 (1 nodes), 640.430094ms from start of poll, 640.845752ms from cluster start. Timeout at 2026-03-29 01:08:26.513263051 +0000 UTC m=+50.301405714, 1.005003009s after stability
2026-03-29T01:08:26.514Z [INFO]  creating new snapshot: path=/tmp/raft3668338075/snapshots/2-15-1774746506514.tmp
2026-03-29T01:08:26.514Z [INFO]  server-77018626-686a-8b5a-4ad5-4cae25e9c640: copied to local snapshot: bytes=71
2026-03-29T01:08:26.514Z [INFO]  server-77018626-686a-8b5a-4ad5-4cae25e9c640: snapshot restore progress: id=2-15-1774746506514 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:26.515Z [INFO]  server-77018626-686a-8b5a-4ad5-4cae25e9c640: restored user snapshot: index=15
2026-03-29T01:08:26.515Z [ERROR] server-77018626-686a-8b5a-4ad5-4cae25e9c640: failed to get log: index=3 error="log not found"
2026-03-29T01:08:26.515Z [INFO]  server-77018626-686a-8b5a-4ad5-4cae25e9c640: aborting pipeline replication: peer="{Voter server-86d477dd-10f9-9a42-4e3e-42771df1a3f5 86d477dd-10f9-9a42-4e3e-42771df1a3f5}"
2026-03-29T01:08:26.515Z [ERROR] server-77018626-686a-8b5a-4ad5-4cae25e9c640: failed to get log: index=3 error="log not found"
2026-03-29T01:08:26.515Z [INFO]  server-77018626-686a-8b5a-4ad5-4cae25e9c640: aborting pipeline replication: peer="{Voter server-382bad15-0d56-66bd-0caa-854eca55005b 382bad15-0d56-66bd-0caa-854eca55005b}"
2026-03-29T01:08:26.525Z [ERROR] server-77018626-686a-8b5a-4ad5-4cae25e9c640: failed to get log: index=3 error="log not found"
2026-03-29T01:08:26.525Z [ERROR] server-77018626-686a-8b5a-4ad5-4cae25e9c640: failed to get log: index=3 error="log not found"
2026-03-29T01:08:26.526Z [INFO]  creating new snapshot: path=/tmp/raft302298240/snapshots/2-15-1774746506526.tmp
2026-03-29T01:08:26.526Z [INFO]  creating new snapshot: path=/tmp/raft3165887259/snapshots/2-15-1774746506526.tmp
2026-03-29T01:08:26.526Z [INFO]  server-382bad15-0d56-66bd-0caa-854eca55005b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:26.527Z [INFO]  server-382bad15-0d56-66bd-0caa-854eca55005b: copied to local snapshot: bytes=71
2026-03-29T01:08:26.527Z [INFO]  server-86d477dd-10f9-9a42-4e3e-42771df1a3f5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:26.527Z [INFO]  server-382bad15-0d56-66bd-0caa-854eca55005b: snapshot restore progress: id=2-15-1774746506526 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:26.527Z [INFO]  server-382bad15-0d56-66bd-0caa-854eca55005b: Installed remote snapshot
2026-03-29T01:08:26.527Z [INFO]  server-86d477dd-10f9-9a42-4e3e-42771df1a3f5: copied to local snapshot: bytes=71
2026-03-29T01:08:26.527Z [INFO]  server-77018626-686a-8b5a-4ad5-4cae25e9c640: pipelining replication: peer="{Voter server-382bad15-0d56-66bd-0caa-854eca55005b 382bad15-0d56-66bd-0caa-854eca55005b}"
2026-03-29T01:08:26.527Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:08:26.527Z [INFO]  server-86d477dd-10f9-9a42-4e3e-42771df1a3f5: snapshot restore progress: id=2-15-1774746506526 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:26.527Z [INFO]  server-86d477dd-10f9-9a42-4e3e-42771df1a3f5: Installed remote snapshot
2026-03-29T01:08:26.527Z [INFO]  server-77018626-686a-8b5a-4ad5-4cae25e9c640: pipelining replication: peer="{Voter server-86d477dd-10f9-9a42-4e3e-42771df1a3f5 86d477dd-10f9-9a42-4e3e-42771df1a3f5}"
2026-03-29T01:08:27.533Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:08:26.52760571 +0000 UTC m=+50.315748250 (2 nodes), 969ns from start of poll, 1.660191297s from cluster start. Timeout at 2026-03-29 01:08:27.532613822 +0000 UTC m=+51.320756397, 1.005008147s after stability
2026-03-29T01:08:27.545Z [INFO]  server-35a4641a-657c-016a-a3a7-0d4d3a0b7ab0: aborting pipeline replication: peer="{Voter server-24ff37dd-24d9-53db-643f-8f08c5d02a7e 24ff37dd-24d9-53db-643f-8f08c5d02a7e}"
2026-03-29T01:08:27.545Z [INFO]  server-35a4641a-657c-016a-a3a7-0d4d3a0b7ab0: aborting pipeline replication: peer="{Voter server-baece70b-f353-ee77-2fb1-bae43a0840be baece70b-f353-ee77-2fb1-bae43a0840be}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true
2026-03-29T01:08:27.546Z [INFO]  server-f07e704a-b8b5-b672-181f-ebf1d3ad0ae6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f07e704a-b8b5-b672-181f-ebf1d3ad0ae6 Address:f07e704a-b8b5-b672-181f-ebf1d3ad0ae6} {Suffrage:Voter ID:server-ea621e93-e8cc-2e3d-029e-07275fe008c8 Address:ea621e93-e8cc-2e3d-029e-07275fe008c8} {Suffrage:Voter ID:server-65474ffe-4d69-b97d-45ea-3e1098e27452 Address:65474ffe-4d69-b97d-45ea-3e1098e27452}]"
2026-03-29T01:08:27.546Z [INFO]  server-f07e704a-b8b5-b672-181f-ebf1d3ad0ae6: entering follower state: follower="Node at f07e704a-b8b5-b672-181f-ebf1d3ad0ae6 [Follower]" leader-address= leader-id=
2026-03-29T01:08:27.546Z [INFO]  server-ea621e93-e8cc-2e3d-029e-07275fe008c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f07e704a-b8b5-b672-181f-ebf1d3ad0ae6 Address:f07e704a-b8b5-b672-181f-ebf1d3ad0ae6} {Suffrage:Voter ID:server-ea621e93-e8cc-2e3d-029e-07275fe008c8 Address:ea621e93-e8cc-2e3d-029e-07275fe008c8} {Suffrage:Voter ID:server-65474ffe-4d69-b97d-45ea-3e1098e27452 Address:65474ffe-4d69-b97d-45ea-3e1098e27452}]"
2026-03-29T01:08:27.546Z [INFO]  server-ea621e93-e8cc-2e3d-029e-07275fe008c8: entering follower state: follower="Node at ea621e93-e8cc-2e3d-029e-07275fe008c8 [Follower]" leader-address= leader-id=
2026-03-29T01:08:27.546Z [INFO]  server-65474ffe-4d69-b97d-45ea-3e1098e27452: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f07e704a-b8b5-b672-181f-ebf1d3ad0ae6 Address:f07e704a-b8b5-b672-181f-ebf1d3ad0ae6} {Suffrage:Voter ID:server-ea621e93-e8cc-2e3d-029e-07275fe008c8 Address:ea621e93-e8cc-2e3d-029e-07275fe008c8} {Suffrage:Voter ID:server-65474ffe-4d69-b97d-45ea-3e1098e27452 Address:65474ffe-4d69-b97d-45ea-3e1098e27452}]"
2026-03-29T01:08:27.546Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:27.546Z [INFO]  server-65474ffe-4d69-b97d-45ea-3e1098e27452: entering follower state: follower="Node at 65474ffe-4d69-b97d-45ea-3e1098e27452 [Follower]" leader-address= leader-id=
2026-03-29T01:08:28.297Z [WARN]  server-f07e704a-b8b5-b672-181f-ebf1d3ad0ae6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:28.297Z [INFO]  server-f07e704a-b8b5-b672-181f-ebf1d3ad0ae6: entering candidate state: node="Node at f07e704a-b8b5-b672-181f-ebf1d3ad0ae6 [Candidate]" term=2
2026-03-29T01:08:28.297Z [INFO]  server-f07e704a-b8b5-b672-181f-ebf1d3ad0ae6: election won: term=2 tally=2
2026-03-29T01:08:28.297Z [INFO]  server-f07e704a-b8b5-b672-181f-ebf1d3ad0ae6: entering leader state: leader="Node at f07e704a-b8b5-b672-181f-ebf1d3ad0ae6 [Leader]"
2026-03-29T01:08:28.297Z [INFO]  server-f07e704a-b8b5-b672-181f-ebf1d3ad0ae6: added peer, starting replication: peer=server-ea621e93-e8cc-2e3d-029e-07275fe008c8
2026-03-29T01:08:28.297Z [INFO]  server-f07e704a-b8b5-b672-181f-ebf1d3ad0ae6: added peer, starting replication: peer=server-65474ffe-4d69-b97d-45ea-3e1098e27452
2026-03-29T01:08:28.297Z [INFO]  server-f07e704a-b8b5-b672-181f-ebf1d3ad0ae6: pipelining replication: peer="{Voter server-65474ffe-4d69-b97d-45ea-3e1098e27452 65474ffe-4d69-b97d-45ea-3e1098e27452}"
2026-03-29T01:08:28.297Z [INFO]  server-f07e704a-b8b5-b672-181f-ebf1d3ad0ae6: pipelining replication: peer="{Voter server-ea621e93-e8cc-2e3d-029e-07275fe008c8 ea621e93-e8cc-2e3d-029e-07275fe008c8}"
2026-03-29T01:08:29.302Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:28.297668887 +0000 UTC m=+52.085811431 (1 nodes), 751.104558ms from start of poll, 751.367025ms from cluster start. Timeout at 2026-03-29 01:08:29.302671923 +0000 UTC m=+53.090814458, 1.005003027s after stability
2026-03-29T01:08:29.303Z [INFO]  server-f07e704a-b8b5-b672-181f-ebf1d3ad0ae6: starting snapshot up to: index=12
2026-03-29T01:08:29.303Z [INFO]  creating new snapshot: path=/tmp/raft4252353253/snapshots/2-12-1774746509303.tmp
2026-03-29T01:08:29.303Z [INFO]  server-f07e704a-b8b5-b672-181f-ebf1d3ad0ae6: snapshot complete up to: index=12
2026-03-29T01:08:29.304Z [INFO]  server-561156a9-c58c-a1ab-b7ee-6a2b6c027e0b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-561156a9-c58c-a1ab-b7ee-6a2b6c027e0b Address:561156a9-c58c-a1ab-b7ee-6a2b6c027e0b} {Suffrage:Voter ID:server-f39d5589-1626-1a5b-886e-58ce2317d508 Address:f39d5589-1626-1a5b-886e-58ce2317d508} {Suffrage:Voter ID:server-7997ea0b-b056-577a-2a78-d4d9667b4051 Address:7997ea0b-b056-577a-2a78-d4d9667b4051}]"
2026-03-29T01:08:29.304Z [INFO]  server-561156a9-c58c-a1ab-b7ee-6a2b6c027e0b: entering follower state: follower="Node at 561156a9-c58c-a1ab-b7ee-6a2b6c027e0b [Follower]" leader-address= leader-id=
2026-03-29T01:08:29.304Z [INFO]  server-f39d5589-1626-1a5b-886e-58ce2317d508: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-561156a9-c58c-a1ab-b7ee-6a2b6c027e0b Address:561156a9-c58c-a1ab-b7ee-6a2b6c027e0b} {Suffrage:Voter ID:server-f39d5589-1626-1a5b-886e-58ce2317d508 Address:f39d5589-1626-1a5b-886e-58ce2317d508} {Suffrage:Voter ID:server-7997ea0b-b056-577a-2a78-d4d9667b4051 Address:7997ea0b-b056-577a-2a78-d4d9667b4051}]"
2026-03-29T01:08:29.304Z [INFO]  server-f39d5589-1626-1a5b-886e-58ce2317d508: entering follower state: follower="Node at f39d5589-1626-1a5b-886e-58ce2317d508 [Follower]" leader-address= leader-id=
2026-03-29T01:08:29.304Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-561156a9-c58c-a1ab-b7ee-6a2b6c027e0b Address:561156a9-c58c-a1ab-b7ee-6a2b6c027e0b} {Suffrage:Voter ID:server-f39d5589-1626-1a5b-886e-58ce2317d508 Address:f39d5589-1626-1a5b-886e-58ce2317d508} {Suffrage:Voter ID:server-7997ea0b-b056-577a-2a78-d4d9667b4051 Address:7997ea0b-b056-577a-2a78-d4d9667b4051}]"
2026-03-29T01:08:29.304Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:29.304Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: entering follower state: follower="Node at 7997ea0b-b056-577a-2a78-d4d9667b4051 [Follower]" leader-address= leader-id=
2026-03-29T01:08:30.077Z [WARN]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:30.077Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: entering candidate state: node="Node at 7997ea0b-b056-577a-2a78-d4d9667b4051 [Candidate]" term=2
2026-03-29T01:08:30.077Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: election won: term=2 tally=2
2026-03-29T01:08:30.077Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: entering leader state: leader="Node at 7997ea0b-b056-577a-2a78-d4d9667b4051 [Leader]"
2026-03-29T01:08:30.077Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: added peer, starting replication: peer=server-561156a9-c58c-a1ab-b7ee-6a2b6c027e0b
2026-03-29T01:08:30.077Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: added peer, starting replication: peer=server-f39d5589-1626-1a5b-886e-58ce2317d508
2026-03-29T01:08:30.077Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: pipelining replication: peer="{Voter server-561156a9-c58c-a1ab-b7ee-6a2b6c027e0b 561156a9-c58c-a1ab-b7ee-6a2b6c027e0b}"
2026-03-29T01:08:30.077Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: pipelining replication: peer="{Voter server-f39d5589-1626-1a5b-886e-58ce2317d508 f39d5589-1626-1a5b-886e-58ce2317d508}"
2026-03-29T01:08:31.082Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:30.077408446 +0000 UTC m=+53.865550981 (1 nodes), 772.456747ms from start of poll, 772.794311ms from cluster start. Timeout at 2026-03-29 01:08:31.082414477 +0000 UTC m=+54.870557046, 1.005006065s after stability
2026-03-29T01:08:31.082Z [INFO]  creating new snapshot: path=/tmp/raft3683672396/snapshots/2-15-1774746511082.tmp
2026-03-29T01:08:31.083Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: copied to local snapshot: bytes=71
2026-03-29T01:08:31.083Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: snapshot restore progress: id=2-15-1774746511082 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:31.083Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: removing all old logs from log store
2026-03-29T01:08:31.083Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: compacting logs: from=1 to=2
2026-03-29T01:08:31.083Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: restored user snapshot: index=15
2026-03-29T01:08:31.083Z [ERROR] server-7997ea0b-b056-577a-2a78-d4d9667b4051: failed to get log: index=2 error="log not found"
2026-03-29T01:08:31.083Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: aborting pipeline replication: peer="{Voter server-f39d5589-1626-1a5b-886e-58ce2317d508 f39d5589-1626-1a5b-886e-58ce2317d508}"
2026-03-29T01:08:31.083Z [ERROR] server-7997ea0b-b056-577a-2a78-d4d9667b4051: failed to get log: index=2 error="log not found"
2026-03-29T01:08:31.083Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: aborting pipeline replication: peer="{Voter server-561156a9-c58c-a1ab-b7ee-6a2b6c027e0b 561156a9-c58c-a1ab-b7ee-6a2b6c027e0b}"
2026-03-29T01:08:31.092Z [ERROR] server-7997ea0b-b056-577a-2a78-d4d9667b4051: failed to get log: index=2 error="log not found"
2026-03-29T01:08:31.092Z [ERROR] server-7997ea0b-b056-577a-2a78-d4d9667b4051: failed to get log: index=2 error="log not found"
2026-03-29T01:08:31.093Z [INFO]  creating new snapshot: path=/tmp/raft849680524/snapshots/2-15-1774746511093.tmp
2026-03-29T01:08:31.093Z [INFO]  creating new snapshot: path=/tmp/raft2338037301/snapshots/2-15-1774746511093.tmp
2026-03-29T01:08:31.093Z [INFO]  server-f39d5589-1626-1a5b-886e-58ce2317d508: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:31.093Z [INFO]  server-561156a9-c58c-a1ab-b7ee-6a2b6c027e0b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:31.093Z [INFO]  server-561156a9-c58c-a1ab-b7ee-6a2b6c027e0b: copied to local snapshot: bytes=71
2026-03-29T01:08:31.093Z [INFO]  server-f39d5589-1626-1a5b-886e-58ce2317d508: copied to local snapshot: bytes=71
2026-03-29T01:08:31.094Z [INFO]  server-f39d5589-1626-1a5b-886e-58ce2317d508: snapshot restore progress: id=2-15-1774746511093 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:31.094Z [INFO]  server-561156a9-c58c-a1ab-b7ee-6a2b6c027e0b: snapshot restore progress: id=2-15-1774746511093 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:31.094Z [INFO]  server-f39d5589-1626-1a5b-886e-58ce2317d508: removing all old logs from log store
2026-03-29T01:08:31.094Z [INFO]  server-f39d5589-1626-1a5b-886e-58ce2317d508: compacting logs: from=1 to=2
2026-03-29T01:08:31.094Z [INFO]  server-f39d5589-1626-1a5b-886e-58ce2317d508: Installed remote snapshot
2026-03-29T01:08:31.094Z [INFO]  server-561156a9-c58c-a1ab-b7ee-6a2b6c027e0b: removing all old logs from log store
2026-03-29T01:08:31.094Z [INFO]  server-561156a9-c58c-a1ab-b7ee-6a2b6c027e0b: compacting logs: from=1 to=2
2026-03-29T01:08:31.094Z [INFO]  server-561156a9-c58c-a1ab-b7ee-6a2b6c027e0b: Installed remote snapshot
2026-03-29T01:08:31.094Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: pipelining replication: peer="{Voter server-f39d5589-1626-1a5b-886e-58ce2317d508 f39d5589-1626-1a5b-886e-58ce2317d508}"
2026-03-29T01:08:31.094Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:08:31.094Z [INFO]  server-7997ea0b-b056-577a-2a78-d4d9667b4051: pipelining replication: peer="{Voter server-561156a9-c58c-a1ab-b7ee-6a2b6c027e0b 561156a9-c58c-a1ab-b7ee-6a2b6c027e0b}"
2026-03-29T01:08:32.100Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:08:31.09425162 +0000 UTC m=+54.882394172 (2 nodes), 1.029µs from start of poll, 1.789637502s from cluster start. Timeout at 2026-03-29 01:08:32.099262516 +0000 UTC m=+55.887405098, 1.005010926s after stability
2026-03-29T01:08:32.111Z [INFO]  server-f07e704a-b8b5-b672-181f-ebf1d3ad0ae6: aborting pipeline replication: peer="{Voter server-65474ffe-4d69-b97d-45ea-3e1098e27452 65474ffe-4d69-b97d-45ea-3e1098e27452}"
2026-03-29T01:08:32.111Z [INFO]  server-f07e704a-b8b5-b672-181f-ebf1d3ad0ae6: aborting pipeline replication: peer="{Voter server-ea621e93-e8cc-2e3d-029e-07275fe008c8 ea621e93-e8cc-2e3d-029e-07275fe008c8}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false
2026-03-29T01:08:32.112Z [INFO]  server-782df6e9-904e-afd8-46a3-3e4be6206981: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-782df6e9-904e-afd8-46a3-3e4be6206981 Address:782df6e9-904e-afd8-46a3-3e4be6206981} {Suffrage:Voter ID:server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b Address:b7b94f27-0481-1a8f-57a5-5edb1c94ab2b} {Suffrage:Voter ID:server-7e357238-45a7-cdaa-2933-53c82a6e8b4a Address:7e357238-45a7-cdaa-2933-53c82a6e8b4a}]"
2026-03-29T01:08:32.112Z [INFO]  server-782df6e9-904e-afd8-46a3-3e4be6206981: entering follower state: follower="Node at 782df6e9-904e-afd8-46a3-3e4be6206981 [Follower]" leader-address= leader-id=
2026-03-29T01:08:32.112Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-782df6e9-904e-afd8-46a3-3e4be6206981 Address:782df6e9-904e-afd8-46a3-3e4be6206981} {Suffrage:Voter ID:server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b Address:b7b94f27-0481-1a8f-57a5-5edb1c94ab2b} {Suffrage:Voter ID:server-7e357238-45a7-cdaa-2933-53c82a6e8b4a Address:7e357238-45a7-cdaa-2933-53c82a6e8b4a}]"
2026-03-29T01:08:32.113Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: entering follower state: follower="Node at b7b94f27-0481-1a8f-57a5-5edb1c94ab2b [Follower]" leader-address= leader-id=
2026-03-29T01:08:32.113Z [INFO]  server-7e357238-45a7-cdaa-2933-53c82a6e8b4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-782df6e9-904e-afd8-46a3-3e4be6206981 Address:782df6e9-904e-afd8-46a3-3e4be6206981} {Suffrage:Voter ID:server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b Address:b7b94f27-0481-1a8f-57a5-5edb1c94ab2b} {Suffrage:Voter ID:server-7e357238-45a7-cdaa-2933-53c82a6e8b4a Address:7e357238-45a7-cdaa-2933-53c82a6e8b4a}]"
2026-03-29T01:08:32.113Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:32.113Z [INFO]  server-7e357238-45a7-cdaa-2933-53c82a6e8b4a: entering follower state: follower="Node at 7e357238-45a7-cdaa-2933-53c82a6e8b4a [Follower]" leader-address= leader-id=
2026-03-29T01:08:32.620Z [WARN]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:32.620Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: entering candidate state: node="Node at b7b94f27-0481-1a8f-57a5-5edb1c94ab2b [Candidate]" term=2
2026-03-29T01:08:32.620Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: election won: term=2 tally=2
2026-03-29T01:08:32.620Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: entering leader state: leader="Node at b7b94f27-0481-1a8f-57a5-5edb1c94ab2b [Leader]"
2026-03-29T01:08:32.620Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: added peer, starting replication: peer=server-782df6e9-904e-afd8-46a3-3e4be6206981
2026-03-29T01:08:32.620Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: added peer, starting replication: peer=server-7e357238-45a7-cdaa-2933-53c82a6e8b4a
2026-03-29T01:08:32.620Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: pipelining replication: peer="{Voter server-7e357238-45a7-cdaa-2933-53c82a6e8b4a 7e357238-45a7-cdaa-2933-53c82a6e8b4a}"
2026-03-29T01:08:32.620Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: pipelining replication: peer="{Voter server-782df6e9-904e-afd8-46a3-3e4be6206981 782df6e9-904e-afd8-46a3-3e4be6206981}"
2026-03-29T01:08:33.625Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:32.620387294 +0000 UTC m=+56.408529829 (1 nodes), 507.24657ms from start of poll, 507.620181ms from cluster start. Timeout at 2026-03-29 01:08:33.625388234 +0000 UTC m=+57.413530766, 1.005000937s after stability
2026-03-29T01:08:33.625Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: starting snapshot up to: index=12
2026-03-29T01:08:33.625Z [INFO]  creating new snapshot: path=/tmp/raft4182404507/snapshots/2-12-1774746513625.tmp
2026-03-29T01:08:33.627Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: snapshot complete up to: index=12
2026-03-29T01:08:33.628Z [INFO]  creating new snapshot: path=/tmp/raft4182404507/snapshots/2-113-1774746513628.tmp
2026-03-29T01:08:33.628Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: copied to local snapshot: bytes=71
2026-03-29T01:08:33.628Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: snapshot restore progress: id=2-113-1774746513628 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:33.628Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: restored user snapshot: index=113
2026-03-29T01:08:33.628Z [ERROR] server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: failed to get log: index=23 error="log not found"
2026-03-29T01:08:33.628Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: aborting pipeline replication: peer="{Voter server-7e357238-45a7-cdaa-2933-53c82a6e8b4a 7e357238-45a7-cdaa-2933-53c82a6e8b4a}"
2026-03-29T01:08:33.628Z [ERROR] server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: failed to get log: index=23 error="log not found"
2026-03-29T01:08:33.628Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: aborting pipeline replication: peer="{Voter server-782df6e9-904e-afd8-46a3-3e4be6206981 782df6e9-904e-afd8-46a3-3e4be6206981}"
2026-03-29T01:08:33.634Z [ERROR] server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: failed to get log: index=23 error="log not found"
2026-03-29T01:08:33.635Z [INFO]  creating new snapshot: path=/tmp/raft1152478657/snapshots/2-113-1774746513635.tmp
2026-03-29T01:08:33.635Z [INFO]  server-7e357238-45a7-cdaa-2933-53c82a6e8b4a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:33.635Z [INFO]  server-7e357238-45a7-cdaa-2933-53c82a6e8b4a: copied to local snapshot: bytes=71
2026-03-29T01:08:33.636Z [INFO]  server-7e357238-45a7-cdaa-2933-53c82a6e8b4a: snapshot restore progress: id=2-113-1774746513635 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:33.636Z [INFO]  server-7e357238-45a7-cdaa-2933-53c82a6e8b4a: Installed remote snapshot
2026-03-29T01:08:33.636Z [WARN]  server-7e357238-45a7-cdaa-2933-53c82a6e8b4a: skipping application of old log: index=22
2026-03-29T01:08:33.636Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: pipelining replication: peer="{Voter server-7e357238-45a7-cdaa-2933-53c82a6e8b4a 7e357238-45a7-cdaa-2933-53c82a6e8b4a}"
2026-03-29T01:08:33.636Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:08:33.639Z [ERROR] server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: failed to get log: index=23 error="log not found"
2026-03-29T01:08:33.639Z [INFO]  creating new snapshot: path=/tmp/raft2072516591/snapshots/2-113-1774746513639.tmp
2026-03-29T01:08:33.639Z [INFO]  server-782df6e9-904e-afd8-46a3-3e4be6206981: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:33.639Z [INFO]  server-782df6e9-904e-afd8-46a3-3e4be6206981: copied to local snapshot: bytes=71
2026-03-29T01:08:33.640Z [INFO]  server-782df6e9-904e-afd8-46a3-3e4be6206981: snapshot restore progress: id=2-113-1774746513639 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:33.640Z [INFO]  server-782df6e9-904e-afd8-46a3-3e4be6206981: Installed remote snapshot
2026-03-29T01:08:33.640Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: pipelining replication: peer="{Voter server-782df6e9-904e-afd8-46a3-3e4be6206981 782df6e9-904e-afd8-46a3-3e4be6206981}"
2026-03-29T01:08:34.641Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:08:33.636346643 +0000 UTC m=+57.424489199 (2 nodes), 1.388µs from start of poll, 1.523579551s from cluster start. Timeout at 2026-03-29 01:08:34.6413652 +0000 UTC m=+58.429507728, 1.005018529s after stability
2026-03-29T01:08:34.654Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: aborting pipeline replication: peer="{Voter server-7e357238-45a7-cdaa-2933-53c82a6e8b4a 7e357238-45a7-cdaa-2933-53c82a6e8b4a}"
2026-03-29T01:08:34.654Z [INFO]  server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: aborting pipeline replication: peer="{Voter server-782df6e9-904e-afd8-46a3-3e4be6206981 782df6e9-904e-afd8-46a3-3e4be6206981}"
2026-03-29T01:08:35.154Z [ERROR] server-b7b94f27-0481-1a8f-57a5-5edb1c94ab2b: failed to heartbeat to: peer=7e357238-45a7-cdaa-2933-53c82a6e8b4a backoff time=10ms error="command timed out"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false
2026-03-29T01:08:35.156Z [INFO]  server-4d95c672-ce3c-8550-d647-014744237c74: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d95c672-ce3c-8550-d647-014744237c74 Address:4d95c672-ce3c-8550-d647-014744237c74} {Suffrage:Voter ID:server-322aaba2-39c1-50a5-bff4-a712a5249066 Address:322aaba2-39c1-50a5-bff4-a712a5249066} {Suffrage:Voter ID:server-a304f26a-d07a-7f79-95a6-03c219c22fdc Address:a304f26a-d07a-7f79-95a6-03c219c22fdc}]"
2026-03-29T01:08:35.156Z [INFO]  server-4d95c672-ce3c-8550-d647-014744237c74: entering follower state: follower="Node at 4d95c672-ce3c-8550-d647-014744237c74 [Follower]" leader-address= leader-id=
2026-03-29T01:08:35.156Z [INFO]  server-322aaba2-39c1-50a5-bff4-a712a5249066: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d95c672-ce3c-8550-d647-014744237c74 Address:4d95c672-ce3c-8550-d647-014744237c74} {Suffrage:Voter ID:server-322aaba2-39c1-50a5-bff4-a712a5249066 Address:322aaba2-39c1-50a5-bff4-a712a5249066} {Suffrage:Voter ID:server-a304f26a-d07a-7f79-95a6-03c219c22fdc Address:a304f26a-d07a-7f79-95a6-03c219c22fdc}]"
2026-03-29T01:08:35.156Z [INFO]  server-322aaba2-39c1-50a5-bff4-a712a5249066: entering follower state: follower="Node at 322aaba2-39c1-50a5-bff4-a712a5249066 [Follower]" leader-address= leader-id=
2026-03-29T01:08:35.156Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d95c672-ce3c-8550-d647-014744237c74 Address:4d95c672-ce3c-8550-d647-014744237c74} {Suffrage:Voter ID:server-322aaba2-39c1-50a5-bff4-a712a5249066 Address:322aaba2-39c1-50a5-bff4-a712a5249066} {Suffrage:Voter ID:server-a304f26a-d07a-7f79-95a6-03c219c22fdc Address:a304f26a-d07a-7f79-95a6-03c219c22fdc}]"
2026-03-29T01:08:35.157Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:35.157Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: entering follower state: follower="Node at a304f26a-d07a-7f79-95a6-03c219c22fdc [Follower]" leader-address= leader-id=
2026-03-29T01:08:35.730Z [WARN]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:35.730Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: entering candidate state: node="Node at a304f26a-d07a-7f79-95a6-03c219c22fdc [Candidate]" term=2
2026-03-29T01:08:35.730Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: election won: term=2 tally=2
2026-03-29T01:08:35.730Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: entering leader state: leader="Node at a304f26a-d07a-7f79-95a6-03c219c22fdc [Leader]"
2026-03-29T01:08:35.731Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: added peer, starting replication: peer=server-4d95c672-ce3c-8550-d647-014744237c74
2026-03-29T01:08:35.731Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: added peer, starting replication: peer=server-322aaba2-39c1-50a5-bff4-a712a5249066
2026-03-29T01:08:35.731Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: pipelining replication: peer="{Voter server-4d95c672-ce3c-8550-d647-014744237c74 4d95c672-ce3c-8550-d647-014744237c74}"
2026-03-29T01:08:35.731Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: pipelining replication: peer="{Voter server-322aaba2-39c1-50a5-bff4-a712a5249066 322aaba2-39c1-50a5-bff4-a712a5249066}"
2026-03-29T01:08:36.736Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:35.731029717 +0000 UTC m=+59.519172287 (1 nodes), 574.002333ms from start of poll, 574.459538ms from cluster start. Timeout at 2026-03-29 01:08:36.736034963 +0000 UTC m=+60.524177489, 1.005005202s after stability
2026-03-29T01:08:36.736Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: starting snapshot up to: index=12
2026-03-29T01:08:36.736Z [INFO]  creating new snapshot: path=/tmp/raft2943499462/snapshots/2-12-1774746516736.tmp
2026-03-29T01:08:36.737Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: snapshot complete up to: index=12
2026-03-29T01:08:36.737Z [INFO]  creating new snapshot: path=/tmp/raft2943499462/snapshots/2-113-1774746516737.tmp
2026-03-29T01:08:36.738Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: copied to local snapshot: bytes=71
2026-03-29T01:08:36.738Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: snapshot restore progress: id=2-113-1774746516737 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:36.738Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: removing all old logs from log store
2026-03-29T01:08:36.738Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: compacting logs: from=1 to=22
2026-03-29T01:08:36.738Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: restored user snapshot: index=113
2026-03-29T01:08:36.738Z [ERROR] server-a304f26a-d07a-7f79-95a6-03c219c22fdc: failed to get log: index=22 error="log not found"
2026-03-29T01:08:36.738Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: aborting pipeline replication: peer="{Voter server-4d95c672-ce3c-8550-d647-014744237c74 4d95c672-ce3c-8550-d647-014744237c74}"
2026-03-29T01:08:36.738Z [ERROR] server-a304f26a-d07a-7f79-95a6-03c219c22fdc: failed to get log: index=22 error="log not found"
2026-03-29T01:08:36.738Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: aborting pipeline replication: peer="{Voter server-322aaba2-39c1-50a5-bff4-a712a5249066 322aaba2-39c1-50a5-bff4-a712a5249066}"
2026-03-29T01:08:36.744Z [ERROR] server-a304f26a-d07a-7f79-95a6-03c219c22fdc: failed to get log: index=22 error="log not found"
2026-03-29T01:08:36.744Z [INFO]  creating new snapshot: path=/tmp/raft495789042/snapshots/2-113-1774746516744.tmp
2026-03-29T01:08:36.745Z [INFO]  server-322aaba2-39c1-50a5-bff4-a712a5249066: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:36.745Z [INFO]  server-322aaba2-39c1-50a5-bff4-a712a5249066: copied to local snapshot: bytes=71
2026-03-29T01:08:36.745Z [INFO]  server-322aaba2-39c1-50a5-bff4-a712a5249066: snapshot restore progress: id=2-113-1774746516744 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:36.745Z [INFO]  server-322aaba2-39c1-50a5-bff4-a712a5249066: removing all old logs from log store
2026-03-29T01:08:36.745Z [INFO]  server-322aaba2-39c1-50a5-bff4-a712a5249066: compacting logs: from=1 to=22
2026-03-29T01:08:36.745Z [INFO]  server-322aaba2-39c1-50a5-bff4-a712a5249066: Installed remote snapshot
2026-03-29T01:08:36.745Z [WARN]  server-322aaba2-39c1-50a5-bff4-a712a5249066: skipping application of old log: index=22
2026-03-29T01:08:36.745Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: pipelining replication: peer="{Voter server-322aaba2-39c1-50a5-bff4-a712a5249066 322aaba2-39c1-50a5-bff4-a712a5249066}"
2026-03-29T01:08:36.745Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:08:36.746Z [ERROR] server-a304f26a-d07a-7f79-95a6-03c219c22fdc: failed to get log: index=22 error="log not found"
2026-03-29T01:08:36.747Z [INFO]  creating new snapshot: path=/tmp/raft2853297085/snapshots/2-113-1774746516747.tmp
2026-03-29T01:08:36.747Z [INFO]  server-4d95c672-ce3c-8550-d647-014744237c74: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:36.766Z [INFO]  server-4d95c672-ce3c-8550-d647-014744237c74: copied to local snapshot: bytes=71
2026-03-29T01:08:36.766Z [INFO]  server-4d95c672-ce3c-8550-d647-014744237c74: snapshot restore progress: id=2-113-1774746516747 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:36.766Z [INFO]  server-4d95c672-ce3c-8550-d647-014744237c74: removing all old logs from log store
2026-03-29T01:08:36.766Z [INFO]  server-4d95c672-ce3c-8550-d647-014744237c74: compacting logs: from=1 to=22
2026-03-29T01:08:36.766Z [INFO]  server-4d95c672-ce3c-8550-d647-014744237c74: Installed remote snapshot
2026-03-29T01:08:36.766Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: pipelining replication: peer="{Voter server-4d95c672-ce3c-8550-d647-014744237c74 4d95c672-ce3c-8550-d647-014744237c74}"
2026-03-29T01:08:37.751Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:08:36.745819079 +0000 UTC m=+60.533961625 (2 nodes), 897ns from start of poll, 1.589248876s from cluster start. Timeout at 2026-03-29 01:08:37.750830434 +0000 UTC m=+61.538972979, 1.005011354s after stability
2026-03-29T01:08:37.763Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: aborting pipeline replication: peer="{Voter server-322aaba2-39c1-50a5-bff4-a712a5249066 322aaba2-39c1-50a5-bff4-a712a5249066}"
2026-03-29T01:08:37.763Z [INFO]  server-a304f26a-d07a-7f79-95a6-03c219c22fdc: aborting pipeline replication: peer="{Voter server-4d95c672-ce3c-8550-d647-014744237c74 4d95c672-ce3c-8550-d647-014744237c74}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true
2026-03-29T01:08:37.764Z [INFO]  server-5fa624b1-017c-0f20-1a1b-5211848de5ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fa624b1-017c-0f20-1a1b-5211848de5ee Address:5fa624b1-017c-0f20-1a1b-5211848de5ee} {Suffrage:Voter ID:server-b3004ddb-e875-e584-b9e2-67adbb0d439e Address:b3004ddb-e875-e584-b9e2-67adbb0d439e} {Suffrage:Voter ID:server-791c61cc-d148-6177-1d1c-e8e5554bf2bc Address:791c61cc-d148-6177-1d1c-e8e5554bf2bc}]"
2026-03-29T01:08:37.764Z [INFO]  server-5fa624b1-017c-0f20-1a1b-5211848de5ee: entering follower state: follower="Node at 5fa624b1-017c-0f20-1a1b-5211848de5ee [Follower]" leader-address= leader-id=
2026-03-29T01:08:37.764Z [INFO]  server-b3004ddb-e875-e584-b9e2-67adbb0d439e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fa624b1-017c-0f20-1a1b-5211848de5ee Address:5fa624b1-017c-0f20-1a1b-5211848de5ee} {Suffrage:Voter ID:server-b3004ddb-e875-e584-b9e2-67adbb0d439e Address:b3004ddb-e875-e584-b9e2-67adbb0d439e} {Suffrage:Voter ID:server-791c61cc-d148-6177-1d1c-e8e5554bf2bc Address:791c61cc-d148-6177-1d1c-e8e5554bf2bc}]"
2026-03-29T01:08:37.764Z [INFO]  server-b3004ddb-e875-e584-b9e2-67adbb0d439e: entering follower state: follower="Node at b3004ddb-e875-e584-b9e2-67adbb0d439e [Follower]" leader-address= leader-id=
2026-03-29T01:08:37.764Z [INFO]  server-791c61cc-d148-6177-1d1c-e8e5554bf2bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fa624b1-017c-0f20-1a1b-5211848de5ee Address:5fa624b1-017c-0f20-1a1b-5211848de5ee} {Suffrage:Voter ID:server-b3004ddb-e875-e584-b9e2-67adbb0d439e Address:b3004ddb-e875-e584-b9e2-67adbb0d439e} {Suffrage:Voter ID:server-791c61cc-d148-6177-1d1c-e8e5554bf2bc Address:791c61cc-d148-6177-1d1c-e8e5554bf2bc}]"
2026-03-29T01:08:37.764Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:37.764Z [INFO]  server-791c61cc-d148-6177-1d1c-e8e5554bf2bc: entering follower state: follower="Node at 791c61cc-d148-6177-1d1c-e8e5554bf2bc [Follower]" leader-address= leader-id=
2026-03-29T01:08:38.328Z [WARN]  server-5fa624b1-017c-0f20-1a1b-5211848de5ee: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:38.328Z [INFO]  server-5fa624b1-017c-0f20-1a1b-5211848de5ee: entering candidate state: node="Node at 5fa624b1-017c-0f20-1a1b-5211848de5ee [Candidate]" term=2
2026-03-29T01:08:38.328Z [INFO]  server-5fa624b1-017c-0f20-1a1b-5211848de5ee: election won: term=2 tally=2
2026-03-29T01:08:38.328Z [INFO]  server-5fa624b1-017c-0f20-1a1b-5211848de5ee: entering leader state: leader="Node at 5fa624b1-017c-0f20-1a1b-5211848de5ee [Leader]"
2026-03-29T01:08:38.328Z [INFO]  server-5fa624b1-017c-0f20-1a1b-5211848de5ee: added peer, starting replication: peer=server-b3004ddb-e875-e584-b9e2-67adbb0d439e
2026-03-29T01:08:38.328Z [INFO]  server-5fa624b1-017c-0f20-1a1b-5211848de5ee: added peer, starting replication: peer=server-791c61cc-d148-6177-1d1c-e8e5554bf2bc
2026-03-29T01:08:38.328Z [INFO]  server-5fa624b1-017c-0f20-1a1b-5211848de5ee: pipelining replication: peer="{Voter server-791c61cc-d148-6177-1d1c-e8e5554bf2bc 791c61cc-d148-6177-1d1c-e8e5554bf2bc}"
2026-03-29T01:08:38.328Z [INFO]  server-5fa624b1-017c-0f20-1a1b-5211848de5ee: pipelining replication: peer="{Voter server-b3004ddb-e875-e584-b9e2-67adbb0d439e b3004ddb-e875-e584-b9e2-67adbb0d439e}"
2026-03-29T01:08:39.333Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:38.328578664 +0000 UTC m=+62.116721199 (1 nodes), 563.840254ms from start of poll, 564.188544ms from cluster start. Timeout at 2026-03-29 01:08:39.333586542 +0000 UTC m=+63.121729173, 1.005007974s after stability
2026-03-29T01:08:39.334Z [INFO]  server-5fa624b1-017c-0f20-1a1b-5211848de5ee: starting snapshot up to: index=12
2026-03-29T01:08:39.334Z [INFO]  creating new snapshot: path=/tmp/raft4069702913/snapshots/2-12-1774746519334.tmp
2026-03-29T01:08:39.334Z [INFO]  server-5fa624b1-017c-0f20-1a1b-5211848de5ee: snapshot complete up to: index=12
2026-03-29T01:08:39.335Z [INFO]  server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b22cf49-d29c-d024-e6a3-837f848ab5ea Address:8b22cf49-d29c-d024-e6a3-837f848ab5ea} {Suffrage:Voter ID:server-df38550f-31e3-9cac-5ae0-791d9534f735 Address:df38550f-31e3-9cac-5ae0-791d9534f735} {Suffrage:Voter ID:server-e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3 Address:e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3}]"
2026-03-29T01:08:39.335Z [INFO]  server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: entering follower state: follower="Node at 8b22cf49-d29c-d024-e6a3-837f848ab5ea [Follower]" leader-address= leader-id=
2026-03-29T01:08:39.335Z [INFO]  server-df38550f-31e3-9cac-5ae0-791d9534f735: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b22cf49-d29c-d024-e6a3-837f848ab5ea Address:8b22cf49-d29c-d024-e6a3-837f848ab5ea} {Suffrage:Voter ID:server-df38550f-31e3-9cac-5ae0-791d9534f735 Address:df38550f-31e3-9cac-5ae0-791d9534f735} {Suffrage:Voter ID:server-e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3 Address:e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3}]"
2026-03-29T01:08:39.335Z [INFO]  server-e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b22cf49-d29c-d024-e6a3-837f848ab5ea Address:8b22cf49-d29c-d024-e6a3-837f848ab5ea} {Suffrage:Voter ID:server-df38550f-31e3-9cac-5ae0-791d9534f735 Address:df38550f-31e3-9cac-5ae0-791d9534f735} {Suffrage:Voter ID:server-e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3 Address:e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3}]"
2026-03-29T01:08:39.335Z [INFO]  server-df38550f-31e3-9cac-5ae0-791d9534f735: entering follower state: follower="Node at df38550f-31e3-9cac-5ae0-791d9534f735 [Follower]" leader-address= leader-id=
2026-03-29T01:08:39.335Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:39.335Z [INFO]  server-e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3: entering follower state: follower="Node at e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3 [Follower]" leader-address= leader-id=
2026-03-29T01:08:39.945Z [WARN]  server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:39.945Z [INFO]  server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: entering candidate state: node="Node at 8b22cf49-d29c-d024-e6a3-837f848ab5ea [Candidate]" term=2
2026-03-29T01:08:39.945Z [INFO]  server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: election won: term=2 tally=2
2026-03-29T01:08:39.945Z [INFO]  server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: entering leader state: leader="Node at 8b22cf49-d29c-d024-e6a3-837f848ab5ea [Leader]"
2026-03-29T01:08:39.945Z [INFO]  server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: added peer, starting replication: peer=server-df38550f-31e3-9cac-5ae0-791d9534f735
2026-03-29T01:08:39.945Z [INFO]  server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: added peer, starting replication: peer=server-e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3
2026-03-29T01:08:39.945Z [INFO]  server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: pipelining replication: peer="{Voter server-e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3 e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3}"
2026-03-29T01:08:39.945Z [INFO]  server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: pipelining replication: peer="{Voter server-df38550f-31e3-9cac-5ae0-791d9534f735 df38550f-31e3-9cac-5ae0-791d9534f735}"
2026-03-29T01:08:40.950Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:39.945811922 +0000 UTC m=+63.733954468 (1 nodes), 609.814394ms from start of poll, 610.1405ms from cluster start. Timeout at 2026-03-29 01:08:40.950816139 +0000 UTC m=+64.738958667, 1.005004199s after stability
2026-03-29T01:08:40.951Z [INFO]  creating new snapshot: path=/tmp/raft3281915061/snapshots/2-113-1774746520951.tmp
2026-03-29T01:08:40.951Z [INFO]  server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: copied to local snapshot: bytes=71
2026-03-29T01:08:40.951Z [INFO]  server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: snapshot restore progress: id=2-113-1774746520951 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:40.951Z [INFO]  server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: restored user snapshot: index=113
2026-03-29T01:08:40.951Z [ERROR] server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: failed to get log: index=3 error="log not found"
2026-03-29T01:08:40.951Z [ERROR] server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: failed to get log: index=3 error="log not found"
2026-03-29T01:08:40.952Z [INFO]  server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: aborting pipeline replication: peer="{Voter server-e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3 e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3}"
2026-03-29T01:08:40.951Z [INFO]  server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: aborting pipeline replication: peer="{Voter server-df38550f-31e3-9cac-5ae0-791d9534f735 df38550f-31e3-9cac-5ae0-791d9534f735}"
2026-03-29T01:08:40.962Z [ERROR] server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: failed to get log: index=3 error="log not found"
2026-03-29T01:08:40.962Z [ERROR] server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: failed to get log: index=3 error="log not found"
2026-03-29T01:08:40.962Z [INFO]  creating new snapshot: path=/tmp/raft1848282728/snapshots/2-113-1774746520962.tmp
2026-03-29T01:08:40.962Z [INFO]  creating new snapshot: path=/tmp/raft3729961769/snapshots/2-113-1774746520962.tmp
2026-03-29T01:08:40.962Z [INFO]  server-e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:40.962Z [INFO]  server-df38550f-31e3-9cac-5ae0-791d9534f735: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:40.963Z [INFO]  server-e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3: copied to local snapshot: bytes=71
2026-03-29T01:08:40.963Z [INFO]  server-df38550f-31e3-9cac-5ae0-791d9534f735: copied to local snapshot: bytes=71
2026-03-29T01:08:40.963Z [INFO]  server-e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3: snapshot restore progress: id=2-113-1774746520962 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:40.963Z [INFO]  server-e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3: Installed remote snapshot
2026-03-29T01:08:40.963Z [INFO]  server-df38550f-31e3-9cac-5ae0-791d9534f735: snapshot restore progress: id=2-113-1774746520962 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:40.963Z [INFO]  server-df38550f-31e3-9cac-5ae0-791d9534f735: Installed remote snapshot
2026-03-29T01:08:40.963Z [INFO]  server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: pipelining replication: peer="{Voter server-e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3 e54288a5-50bb-e95e-a1a0-d2fa3ecadeb3}"
2026-03-29T01:08:40.963Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:08:40.963Z [INFO]  server-8b22cf49-d29c-d024-e6a3-837f848ab5ea: pipelining replication: peer="{Voter server-df38550f-31e3-9cac-5ae0-791d9534f735 df38550f-31e3-9cac-5ae0-791d9534f735}"
2026-03-29T01:08:41.969Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:08:40.963422366 +0000 UTC m=+64.751564910 (2 nodes), 881ns from start of poll, 1.627750942s from cluster start. Timeout at 2026-03-29 01:08:41.968428973 +0000 UTC m=+65.756571543, 1.005006633s after stability
2026-03-29T01:08:41.981Z [INFO]  server-5fa624b1-017c-0f20-1a1b-5211848de5ee: aborting pipeline replication: peer="{Voter server-791c61cc-d148-6177-1d1c-e8e5554bf2bc 791c61cc-d148-6177-1d1c-e8e5554bf2bc}"
2026-03-29T01:08:41.981Z [INFO]  server-5fa624b1-017c-0f20-1a1b-5211848de5ee: aborting pipeline replication: peer="{Voter server-b3004ddb-e875-e584-b9e2-67adbb0d439e b3004ddb-e875-e584-b9e2-67adbb0d439e}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true
2026-03-29T01:08:41.982Z [INFO]  server-50f4f8dc-92f3-14a6-1cd7-9cac99b8737d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50f4f8dc-92f3-14a6-1cd7-9cac99b8737d Address:50f4f8dc-92f3-14a6-1cd7-9cac99b8737d} {Suffrage:Voter ID:server-1475dab4-f271-6131-9a6a-25e5fe67c51a Address:1475dab4-f271-6131-9a6a-25e5fe67c51a} {Suffrage:Voter ID:server-d52378d7-c379-b38d-0802-4df16f6bff63 Address:d52378d7-c379-b38d-0802-4df16f6bff63}]"
2026-03-29T01:08:41.982Z [INFO]  server-50f4f8dc-92f3-14a6-1cd7-9cac99b8737d: entering follower state: follower="Node at 50f4f8dc-92f3-14a6-1cd7-9cac99b8737d [Follower]" leader-address= leader-id=
2026-03-29T01:08:41.982Z [INFO]  server-1475dab4-f271-6131-9a6a-25e5fe67c51a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50f4f8dc-92f3-14a6-1cd7-9cac99b8737d Address:50f4f8dc-92f3-14a6-1cd7-9cac99b8737d} {Suffrage:Voter ID:server-1475dab4-f271-6131-9a6a-25e5fe67c51a Address:1475dab4-f271-6131-9a6a-25e5fe67c51a} {Suffrage:Voter ID:server-d52378d7-c379-b38d-0802-4df16f6bff63 Address:d52378d7-c379-b38d-0802-4df16f6bff63}]"
2026-03-29T01:08:41.982Z [INFO]  server-1475dab4-f271-6131-9a6a-25e5fe67c51a: entering follower state: follower="Node at 1475dab4-f271-6131-9a6a-25e5fe67c51a [Follower]" leader-address= leader-id=
2026-03-29T01:08:41.982Z [INFO]  server-d52378d7-c379-b38d-0802-4df16f6bff63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50f4f8dc-92f3-14a6-1cd7-9cac99b8737d Address:50f4f8dc-92f3-14a6-1cd7-9cac99b8737d} {Suffrage:Voter ID:server-1475dab4-f271-6131-9a6a-25e5fe67c51a Address:1475dab4-f271-6131-9a6a-25e5fe67c51a} {Suffrage:Voter ID:server-d52378d7-c379-b38d-0802-4df16f6bff63 Address:d52378d7-c379-b38d-0802-4df16f6bff63}]"
2026-03-29T01:08:41.983Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:41.983Z [INFO]  server-d52378d7-c379-b38d-0802-4df16f6bff63: entering follower state: follower="Node at d52378d7-c379-b38d-0802-4df16f6bff63 [Follower]" leader-address= leader-id=
2026-03-29T01:08:42.746Z [WARN]  server-1475dab4-f271-6131-9a6a-25e5fe67c51a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:42.746Z [INFO]  server-1475dab4-f271-6131-9a6a-25e5fe67c51a: entering candidate state: node="Node at 1475dab4-f271-6131-9a6a-25e5fe67c51a [Candidate]" term=2
2026-03-29T01:08:42.746Z [INFO]  server-1475dab4-f271-6131-9a6a-25e5fe67c51a: election won: term=2 tally=2
2026-03-29T01:08:42.746Z [INFO]  server-1475dab4-f271-6131-9a6a-25e5fe67c51a: entering leader state: leader="Node at 1475dab4-f271-6131-9a6a-25e5fe67c51a [Leader]"
2026-03-29T01:08:42.746Z [INFO]  server-1475dab4-f271-6131-9a6a-25e5fe67c51a: added peer, starting replication: peer=server-50f4f8dc-92f3-14a6-1cd7-9cac99b8737d
2026-03-29T01:08:42.746Z [INFO]  server-1475dab4-f271-6131-9a6a-25e5fe67c51a: added peer, starting replication: peer=server-d52378d7-c379-b38d-0802-4df16f6bff63
2026-03-29T01:08:42.746Z [INFO]  server-1475dab4-f271-6131-9a6a-25e5fe67c51a: pipelining replication: peer="{Voter server-50f4f8dc-92f3-14a6-1cd7-9cac99b8737d 50f4f8dc-92f3-14a6-1cd7-9cac99b8737d}"
2026-03-29T01:08:42.746Z [INFO]  server-1475dab4-f271-6131-9a6a-25e5fe67c51a: pipelining replication: peer="{Voter server-d52378d7-c379-b38d-0802-4df16f6bff63 d52378d7-c379-b38d-0802-4df16f6bff63}"
2026-03-29T01:08:43.752Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:42.746654158 +0000 UTC m=+66.534796693 (1 nodes), 763.640999ms from start of poll, 763.990942ms from cluster start. Timeout at 2026-03-29 01:08:43.751657705 +0000 UTC m=+67.539800235, 1.005003542s after stability
2026-03-29T01:08:43.752Z [INFO]  server-1475dab4-f271-6131-9a6a-25e5fe67c51a: starting snapshot up to: index=12
2026-03-29T01:08:43.752Z [INFO]  creating new snapshot: path=/tmp/raft3012811022/snapshots/2-12-1774746523752.tmp
2026-03-29T01:08:43.753Z [INFO]  server-1475dab4-f271-6131-9a6a-25e5fe67c51a: snapshot complete up to: index=12
2026-03-29T01:08:43.754Z [INFO]  server-2b4e2d70-68a1-091f-15e0-2dbda2cfe995: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b4e2d70-68a1-091f-15e0-2dbda2cfe995 Address:2b4e2d70-68a1-091f-15e0-2dbda2cfe995} {Suffrage:Voter ID:server-93cd3884-3ebd-6338-2c52-33dbb34eda38 Address:93cd3884-3ebd-6338-2c52-33dbb34eda38} {Suffrage:Voter ID:server-040a7322-c7ee-dfcf-b749-01c345a6ae5e Address:040a7322-c7ee-dfcf-b749-01c345a6ae5e}]"
2026-03-29T01:08:43.754Z [INFO]  server-2b4e2d70-68a1-091f-15e0-2dbda2cfe995: entering follower state: follower="Node at 2b4e2d70-68a1-091f-15e0-2dbda2cfe995 [Follower]" leader-address= leader-id=
2026-03-29T01:08:43.754Z [INFO]  server-93cd3884-3ebd-6338-2c52-33dbb34eda38: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b4e2d70-68a1-091f-15e0-2dbda2cfe995 Address:2b4e2d70-68a1-091f-15e0-2dbda2cfe995} {Suffrage:Voter ID:server-93cd3884-3ebd-6338-2c52-33dbb34eda38 Address:93cd3884-3ebd-6338-2c52-33dbb34eda38} {Suffrage:Voter ID:server-040a7322-c7ee-dfcf-b749-01c345a6ae5e Address:040a7322-c7ee-dfcf-b749-01c345a6ae5e}]"
2026-03-29T01:08:43.754Z [INFO]  server-93cd3884-3ebd-6338-2c52-33dbb34eda38: entering follower state: follower="Node at 93cd3884-3ebd-6338-2c52-33dbb34eda38 [Follower]" leader-address= leader-id=
2026-03-29T01:08:43.754Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2b4e2d70-68a1-091f-15e0-2dbda2cfe995 Address:2b4e2d70-68a1-091f-15e0-2dbda2cfe995} {Suffrage:Voter ID:server-93cd3884-3ebd-6338-2c52-33dbb34eda38 Address:93cd3884-3ebd-6338-2c52-33dbb34eda38} {Suffrage:Voter ID:server-040a7322-c7ee-dfcf-b749-01c345a6ae5e Address:040a7322-c7ee-dfcf-b749-01c345a6ae5e}]"
2026-03-29T01:08:43.754Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:43.754Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: entering follower state: follower="Node at 040a7322-c7ee-dfcf-b749-01c345a6ae5e [Follower]" leader-address= leader-id=
2026-03-29T01:08:44.275Z [WARN]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:44.275Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: entering candidate state: node="Node at 040a7322-c7ee-dfcf-b749-01c345a6ae5e [Candidate]" term=2
2026-03-29T01:08:44.275Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: election won: term=2 tally=2
2026-03-29T01:08:44.275Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: entering leader state: leader="Node at 040a7322-c7ee-dfcf-b749-01c345a6ae5e [Leader]"
2026-03-29T01:08:44.275Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: added peer, starting replication: peer=server-2b4e2d70-68a1-091f-15e0-2dbda2cfe995
2026-03-29T01:08:44.275Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: added peer, starting replication: peer=server-93cd3884-3ebd-6338-2c52-33dbb34eda38
2026-03-29T01:08:44.275Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: pipelining replication: peer="{Voter server-2b4e2d70-68a1-091f-15e0-2dbda2cfe995 2b4e2d70-68a1-091f-15e0-2dbda2cfe995}"
2026-03-29T01:08:44.275Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: pipelining replication: peer="{Voter server-93cd3884-3ebd-6338-2c52-33dbb34eda38 93cd3884-3ebd-6338-2c52-33dbb34eda38}"
2026-03-29T01:08:45.281Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:44.275486085 +0000 UTC m=+68.063628622 (1 nodes), 520.905694ms from start of poll, 521.263957ms from cluster start. Timeout at 2026-03-29 01:08:45.280500438 +0000 UTC m=+69.068642991, 1.005014369s after stability
2026-03-29T01:08:45.282Z [INFO]  creating new snapshot: path=/tmp/raft3255134848/snapshots/2-113-1774746525282.tmp
2026-03-29T01:08:45.288Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: copied to local snapshot: bytes=71
2026-03-29T01:08:45.289Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: snapshot restore progress: id=2-113-1774746525282 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:45.289Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: removing all old logs from log store
2026-03-29T01:08:45.289Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: compacting logs: from=1 to=2
2026-03-29T01:08:45.289Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: restored user snapshot: index=113
2026-03-29T01:08:45.289Z [ERROR] server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: failed to get log: index=2 error="log not found"
2026-03-29T01:08:45.289Z [ERROR] server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: failed to get log: index=2 error="log not found"
2026-03-29T01:08:45.289Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: aborting pipeline replication: peer="{Voter server-2b4e2d70-68a1-091f-15e0-2dbda2cfe995 2b4e2d70-68a1-091f-15e0-2dbda2cfe995}"
2026-03-29T01:08:45.289Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: aborting pipeline replication: peer="{Voter server-93cd3884-3ebd-6338-2c52-33dbb34eda38 93cd3884-3ebd-6338-2c52-33dbb34eda38}"
2026-03-29T01:08:45.298Z [ERROR] server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: failed to get log: index=2 error="log not found"
2026-03-29T01:08:45.298Z [INFO]  creating new snapshot: path=/tmp/raft3792887002/snapshots/2-113-1774746525298.tmp
2026-03-29T01:08:45.299Z [ERROR] server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: failed to get log: index=2 error="log not found"
2026-03-29T01:08:45.299Z [INFO]  creating new snapshot: path=/tmp/raft3029732242/snapshots/2-113-1774746525299.tmp
2026-03-29T01:08:45.299Z [INFO]  server-2b4e2d70-68a1-091f-15e0-2dbda2cfe995: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:45.299Z [INFO]  server-93cd3884-3ebd-6338-2c52-33dbb34eda38: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:45.323Z [INFO]  server-2b4e2d70-68a1-091f-15e0-2dbda2cfe995: copied to local snapshot: bytes=71
2026-03-29T01:08:45.323Z [INFO]  server-93cd3884-3ebd-6338-2c52-33dbb34eda38: copied to local snapshot: bytes=71
2026-03-29T01:08:45.323Z [INFO]  server-2b4e2d70-68a1-091f-15e0-2dbda2cfe995: snapshot restore progress: id=2-113-1774746525298 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:45.323Z [INFO]  server-93cd3884-3ebd-6338-2c52-33dbb34eda38: snapshot restore progress: id=2-113-1774746525299 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:45.323Z [INFO]  server-2b4e2d70-68a1-091f-15e0-2dbda2cfe995: removing all old logs from log store
2026-03-29T01:08:45.323Z [INFO]  server-2b4e2d70-68a1-091f-15e0-2dbda2cfe995: compacting logs: from=1 to=2
2026-03-29T01:08:45.323Z [INFO]  server-2b4e2d70-68a1-091f-15e0-2dbda2cfe995: Installed remote snapshot
2026-03-29T01:08:45.323Z [INFO]  server-93cd3884-3ebd-6338-2c52-33dbb34eda38: removing all old logs from log store
2026-03-29T01:08:45.323Z [INFO]  server-93cd3884-3ebd-6338-2c52-33dbb34eda38: compacting logs: from=1 to=2
2026-03-29T01:08:45.323Z [INFO]  server-93cd3884-3ebd-6338-2c52-33dbb34eda38: Installed remote snapshot
2026-03-29T01:08:45.323Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: pipelining replication: peer="{Voter server-2b4e2d70-68a1-091f-15e0-2dbda2cfe995 2b4e2d70-68a1-091f-15e0-2dbda2cfe995}"
2026-03-29T01:08:45.323Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:08:45.323Z [INFO]  server-040a7322-c7ee-dfcf-b749-01c345a6ae5e: pipelining replication: peer="{Voter server-93cd3884-3ebd-6338-2c52-33dbb34eda38 93cd3884-3ebd-6338-2c52-33dbb34eda38}"
2026-03-29T01:08:46.329Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:08:45.323711547 +0000 UTC m=+69.111854082 (2 nodes), 636ns from start of poll, 1.569489417s from cluster start. Timeout at 2026-03-29 01:08:46.328725915 +0000 UTC m=+70.116868508, 1.005014426s after stability
2026-03-29T01:08:46.341Z [INFO]  server-1475dab4-f271-6131-9a6a-25e5fe67c51a: aborting pipeline replication: peer="{Voter server-d52378d7-c379-b38d-0802-4df16f6bff63 d52378d7-c379-b38d-0802-4df16f6bff63}"
2026-03-29T01:08:46.341Z [INFO]  server-1475dab4-f271-6131-9a6a-25e5fe67c51a: aborting pipeline replication: peer="{Voter server-50f4f8dc-92f3-14a6-1cd7-9cac99b8737d 50f4f8dc-92f3-14a6-1cd7-9cac99b8737d}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false
2026-03-29T01:08:46.342Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-647e6b15-957c-0bd8-4af7-042112242151 Address:647e6b15-957c-0bd8-4af7-042112242151} {Suffrage:Voter ID:server-76d13627-1938-03b6-3597-210613fd012a Address:76d13627-1938-03b6-3597-210613fd012a} {Suffrage:Voter ID:server-b4b8e9de-5e45-a844-f7b6-fb1db7f4b476 Address:b4b8e9de-5e45-a844-f7b6-fb1db7f4b476}]"
2026-03-29T01:08:46.342Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: entering follower state: follower="Node at 647e6b15-957c-0bd8-4af7-042112242151 [Follower]" leader-address= leader-id=
2026-03-29T01:08:46.342Z [INFO]  server-76d13627-1938-03b6-3597-210613fd012a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-647e6b15-957c-0bd8-4af7-042112242151 Address:647e6b15-957c-0bd8-4af7-042112242151} {Suffrage:Voter ID:server-76d13627-1938-03b6-3597-210613fd012a Address:76d13627-1938-03b6-3597-210613fd012a} {Suffrage:Voter ID:server-b4b8e9de-5e45-a844-f7b6-fb1db7f4b476 Address:b4b8e9de-5e45-a844-f7b6-fb1db7f4b476}]"
2026-03-29T01:08:46.342Z [INFO]  server-b4b8e9de-5e45-a844-f7b6-fb1db7f4b476: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-647e6b15-957c-0bd8-4af7-042112242151 Address:647e6b15-957c-0bd8-4af7-042112242151} {Suffrage:Voter ID:server-76d13627-1938-03b6-3597-210613fd012a Address:76d13627-1938-03b6-3597-210613fd012a} {Suffrage:Voter ID:server-b4b8e9de-5e45-a844-f7b6-fb1db7f4b476 Address:b4b8e9de-5e45-a844-f7b6-fb1db7f4b476}]"
2026-03-29T01:08:46.342Z [INFO]  server-76d13627-1938-03b6-3597-210613fd012a: entering follower state: follower="Node at 76d13627-1938-03b6-3597-210613fd012a [Follower]" leader-address= leader-id=
2026-03-29T01:08:46.342Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:46.342Z [INFO]  server-b4b8e9de-5e45-a844-f7b6-fb1db7f4b476: entering follower state: follower="Node at b4b8e9de-5e45-a844-f7b6-fb1db7f4b476 [Follower]" leader-address= leader-id=
2026-03-29T01:08:46.850Z [WARN]  server-647e6b15-957c-0bd8-4af7-042112242151: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:46.850Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: entering candidate state: node="Node at 647e6b15-957c-0bd8-4af7-042112242151 [Candidate]" term=2
2026-03-29T01:08:46.851Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: election won: term=2 tally=2
2026-03-29T01:08:46.851Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: entering leader state: leader="Node at 647e6b15-957c-0bd8-4af7-042112242151 [Leader]"
2026-03-29T01:08:46.851Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: added peer, starting replication: peer=server-76d13627-1938-03b6-3597-210613fd012a
2026-03-29T01:08:46.851Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: added peer, starting replication: peer=server-b4b8e9de-5e45-a844-f7b6-fb1db7f4b476
2026-03-29T01:08:46.851Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: pipelining replication: peer="{Voter server-76d13627-1938-03b6-3597-210613fd012a 76d13627-1938-03b6-3597-210613fd012a}"
2026-03-29T01:08:46.852Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: pipelining replication: peer="{Voter server-b4b8e9de-5e45-a844-f7b6-fb1db7f4b476 b4b8e9de-5e45-a844-f7b6-fb1db7f4b476}"
2026-03-29T01:08:47.856Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:46.85154561 +0000 UTC m=+70.639688172 (1 nodes), 509.013192ms from start of poll, 509.283912ms from cluster start. Timeout at 2026-03-29 01:08:47.856550489 +0000 UTC m=+71.644693048, 1.005004876s after stability
2026-03-29T01:08:47.857Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: starting snapshot up to: index=12
2026-03-29T01:08:47.857Z [INFO]  creating new snapshot: path=/tmp/raft2436288820/snapshots/2-12-1774746527857.tmp
2026-03-29T01:08:47.857Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: snapshot complete up to: index=12
2026-03-29T01:08:47.858Z [INFO]  creating new snapshot: path=/tmp/raft2436288820/snapshots/2-1013-1774746527858.tmp
2026-03-29T01:08:47.858Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: copied to local snapshot: bytes=71
2026-03-29T01:08:47.859Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: snapshot restore progress: id=2-1013-1774746527858 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:47.859Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: restored user snapshot: index=1013
2026-03-29T01:08:47.859Z [ERROR] server-647e6b15-957c-0bd8-4af7-042112242151: failed to get log: index=23 error="log not found"
2026-03-29T01:08:47.859Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: aborting pipeline replication: peer="{Voter server-b4b8e9de-5e45-a844-f7b6-fb1db7f4b476 b4b8e9de-5e45-a844-f7b6-fb1db7f4b476}"
2026-03-29T01:08:47.859Z [ERROR] server-647e6b15-957c-0bd8-4af7-042112242151: failed to get log: index=23 error="log not found"
2026-03-29T01:08:47.859Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: aborting pipeline replication: peer="{Voter server-76d13627-1938-03b6-3597-210613fd012a 76d13627-1938-03b6-3597-210613fd012a}"
2026-03-29T01:08:47.865Z [ERROR] server-647e6b15-957c-0bd8-4af7-042112242151: failed to get log: index=23 error="log not found"
2026-03-29T01:08:47.865Z [INFO]  creating new snapshot: path=/tmp/raft3139399044/snapshots/2-1013-1774746527865.tmp
2026-03-29T01:08:47.865Z [ERROR] server-647e6b15-957c-0bd8-4af7-042112242151: failed to get log: index=23 error="log not found"
2026-03-29T01:08:47.865Z [INFO]  server-76d13627-1938-03b6-3597-210613fd012a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:47.866Z [INFO]  server-76d13627-1938-03b6-3597-210613fd012a: copied to local snapshot: bytes=71
2026-03-29T01:08:47.866Z [INFO]  creating new snapshot: path=/tmp/raft3382280382/snapshots/2-1013-1774746527866.tmp
2026-03-29T01:08:47.866Z [INFO]  server-76d13627-1938-03b6-3597-210613fd012a: snapshot restore progress: id=2-1013-1774746527865 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:47.866Z [INFO]  server-76d13627-1938-03b6-3597-210613fd012a: Installed remote snapshot
2026-03-29T01:08:47.866Z [INFO]  server-b4b8e9de-5e45-a844-f7b6-fb1db7f4b476: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:47.866Z [WARN]  server-76d13627-1938-03b6-3597-210613fd012a: skipping application of old log: index=22
2026-03-29T01:08:47.866Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: pipelining replication: peer="{Voter server-76d13627-1938-03b6-3597-210613fd012a 76d13627-1938-03b6-3597-210613fd012a}"
2026-03-29T01:08:47.866Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:08:47.866Z [INFO]  server-b4b8e9de-5e45-a844-f7b6-fb1db7f4b476: copied to local snapshot: bytes=71
2026-03-29T01:08:47.866Z [INFO]  server-b4b8e9de-5e45-a844-f7b6-fb1db7f4b476: snapshot restore progress: id=2-1013-1774746527866 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:47.866Z [INFO]  server-b4b8e9de-5e45-a844-f7b6-fb1db7f4b476: Installed remote snapshot
2026-03-29T01:08:47.866Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: pipelining replication: peer="{Voter server-b4b8e9de-5e45-a844-f7b6-fb1db7f4b476 b4b8e9de-5e45-a844-f7b6-fb1db7f4b476}"
2026-03-29T01:08:48.871Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:08:47.866471565 +0000 UTC m=+71.654614114 (2 nodes), 970ns from start of poll, 1.524209854s from cluster start. Timeout at 2026-03-29 01:08:48.871484775 +0000 UTC m=+72.659627305, 1.005013191s after stability
2026-03-29T01:08:48.883Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: aborting pipeline replication: peer="{Voter server-b4b8e9de-5e45-a844-f7b6-fb1db7f4b476 b4b8e9de-5e45-a844-f7b6-fb1db7f4b476}"
2026-03-29T01:08:48.883Z [INFO]  server-647e6b15-957c-0bd8-4af7-042112242151: aborting pipeline replication: peer="{Voter server-76d13627-1938-03b6-3597-210613fd012a 76d13627-1938-03b6-3597-210613fd012a}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false
2026-03-29T01:08:48.884Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f Address:9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f} {Suffrage:Voter ID:server-387efd00-8051-e887-ac4b-70e29c3c9eb2 Address:387efd00-8051-e887-ac4b-70e29c3c9eb2} {Suffrage:Voter ID:server-7572c5df-cec5-e537-b6f5-5c5072a1c45c Address:7572c5df-cec5-e537-b6f5-5c5072a1c45c}]"
2026-03-29T01:08:48.884Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: entering follower state: follower="Node at 9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f [Follower]" leader-address= leader-id=
2026-03-29T01:08:48.884Z [INFO]  server-387efd00-8051-e887-ac4b-70e29c3c9eb2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f Address:9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f} {Suffrage:Voter ID:server-387efd00-8051-e887-ac4b-70e29c3c9eb2 Address:387efd00-8051-e887-ac4b-70e29c3c9eb2} {Suffrage:Voter ID:server-7572c5df-cec5-e537-b6f5-5c5072a1c45c Address:7572c5df-cec5-e537-b6f5-5c5072a1c45c}]"
2026-03-29T01:08:48.884Z [INFO]  server-387efd00-8051-e887-ac4b-70e29c3c9eb2: entering follower state: follower="Node at 387efd00-8051-e887-ac4b-70e29c3c9eb2 [Follower]" leader-address= leader-id=
2026-03-29T01:08:48.884Z [INFO]  server-7572c5df-cec5-e537-b6f5-5c5072a1c45c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f Address:9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f} {Suffrage:Voter ID:server-387efd00-8051-e887-ac4b-70e29c3c9eb2 Address:387efd00-8051-e887-ac4b-70e29c3c9eb2} {Suffrage:Voter ID:server-7572c5df-cec5-e537-b6f5-5c5072a1c45c Address:7572c5df-cec5-e537-b6f5-5c5072a1c45c}]"
2026-03-29T01:08:48.884Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:48.884Z [INFO]  server-7572c5df-cec5-e537-b6f5-5c5072a1c45c: entering follower state: follower="Node at 7572c5df-cec5-e537-b6f5-5c5072a1c45c [Follower]" leader-address= leader-id=
2026-03-29T01:08:49.451Z [WARN]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:49.451Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: entering candidate state: node="Node at 9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f [Candidate]" term=2
2026-03-29T01:08:49.451Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: election won: term=2 tally=2
2026-03-29T01:08:49.451Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: entering leader state: leader="Node at 9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f [Leader]"
2026-03-29T01:08:49.451Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: added peer, starting replication: peer=server-387efd00-8051-e887-ac4b-70e29c3c9eb2
2026-03-29T01:08:49.451Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: added peer, starting replication: peer=server-7572c5df-cec5-e537-b6f5-5c5072a1c45c
2026-03-29T01:08:49.451Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: pipelining replication: peer="{Voter server-387efd00-8051-e887-ac4b-70e29c3c9eb2 387efd00-8051-e887-ac4b-70e29c3c9eb2}"
2026-03-29T01:08:49.451Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: pipelining replication: peer="{Voter server-7572c5df-cec5-e537-b6f5-5c5072a1c45c 7572c5df-cec5-e537-b6f5-5c5072a1c45c}"
2026-03-29T01:08:50.456Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:49.451766408 +0000 UTC m=+73.239908966 (1 nodes), 567.156693ms from start of poll, 567.469735ms from cluster start. Timeout at 2026-03-29 01:08:50.456771675 +0000 UTC m=+74.244914203, 1.005005237s after stability
2026-03-29T01:08:50.457Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: starting snapshot up to: index=12
2026-03-29T01:08:50.457Z [INFO]  creating new snapshot: path=/tmp/raft4279028483/snapshots/2-12-1774746530457.tmp
2026-03-29T01:08:50.457Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: snapshot complete up to: index=12
2026-03-29T01:08:50.458Z [INFO]  creating new snapshot: path=/tmp/raft4279028483/snapshots/2-1013-1774746530458.tmp
2026-03-29T01:08:50.458Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: copied to local snapshot: bytes=71
2026-03-29T01:08:50.459Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: snapshot restore progress: id=2-1013-1774746530458 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:50.459Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: removing all old logs from log store
2026-03-29T01:08:50.459Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: compacting logs: from=1 to=22
2026-03-29T01:08:50.459Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: restored user snapshot: index=1013
2026-03-29T01:08:50.459Z [ERROR] server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: failed to get log: index=22 error="log not found"
2026-03-29T01:08:50.459Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: aborting pipeline replication: peer="{Voter server-7572c5df-cec5-e537-b6f5-5c5072a1c45c 7572c5df-cec5-e537-b6f5-5c5072a1c45c}"
2026-03-29T01:08:50.459Z [ERROR] server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: failed to get log: index=22 error="log not found"
2026-03-29T01:08:50.459Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: aborting pipeline replication: peer="{Voter server-387efd00-8051-e887-ac4b-70e29c3c9eb2 387efd00-8051-e887-ac4b-70e29c3c9eb2}"
2026-03-29T01:08:50.465Z [ERROR] server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: failed to get log: index=22 error="log not found"
2026-03-29T01:08:50.465Z [INFO]  creating new snapshot: path=/tmp/raft1361248840/snapshots/2-1013-1774746530465.tmp
2026-03-29T01:08:50.466Z [INFO]  server-387efd00-8051-e887-ac4b-70e29c3c9eb2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:50.466Z [INFO]  server-387efd00-8051-e887-ac4b-70e29c3c9eb2: copied to local snapshot: bytes=71
2026-03-29T01:08:50.466Z [INFO]  server-387efd00-8051-e887-ac4b-70e29c3c9eb2: snapshot restore progress: id=2-1013-1774746530465 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:50.466Z [INFO]  server-387efd00-8051-e887-ac4b-70e29c3c9eb2: removing all old logs from log store
2026-03-29T01:08:50.466Z [INFO]  server-387efd00-8051-e887-ac4b-70e29c3c9eb2: compacting logs: from=1 to=22
2026-03-29T01:08:50.466Z [INFO]  server-387efd00-8051-e887-ac4b-70e29c3c9eb2: Installed remote snapshot
2026-03-29T01:08:50.466Z [WARN]  server-387efd00-8051-e887-ac4b-70e29c3c9eb2: skipping application of old log: index=22
2026-03-29T01:08:50.466Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: pipelining replication: peer="{Voter server-387efd00-8051-e887-ac4b-70e29c3c9eb2 387efd00-8051-e887-ac4b-70e29c3c9eb2}"
2026-03-29T01:08:50.466Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:08:50.468Z [ERROR] server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: failed to get log: index=22 error="log not found"
2026-03-29T01:08:50.469Z [INFO]  creating new snapshot: path=/tmp/raft4038060769/snapshots/2-1013-1774746530469.tmp
2026-03-29T01:08:50.470Z [INFO]  server-7572c5df-cec5-e537-b6f5-5c5072a1c45c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:50.470Z [INFO]  server-7572c5df-cec5-e537-b6f5-5c5072a1c45c: copied to local snapshot: bytes=71
2026-03-29T01:08:50.471Z [INFO]  server-7572c5df-cec5-e537-b6f5-5c5072a1c45c: snapshot restore progress: id=2-1013-1774746530469 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:50.471Z [INFO]  server-7572c5df-cec5-e537-b6f5-5c5072a1c45c: removing all old logs from log store
2026-03-29T01:08:50.471Z [INFO]  server-7572c5df-cec5-e537-b6f5-5c5072a1c45c: compacting logs: from=1 to=22
2026-03-29T01:08:50.471Z [INFO]  server-7572c5df-cec5-e537-b6f5-5c5072a1c45c: Installed remote snapshot
2026-03-29T01:08:50.471Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: pipelining replication: peer="{Voter server-7572c5df-cec5-e537-b6f5-5c5072a1c45c 7572c5df-cec5-e537-b6f5-5c5072a1c45c}"
2026-03-29T01:08:51.471Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:08:50.46670077 +0000 UTC m=+74.254843305 (2 nodes), 627ns from start of poll, 1.582404074s from cluster start. Timeout at 2026-03-29 01:08:51.471712632 +0000 UTC m=+75.259855180, 1.005011875s after stability
2026-03-29T01:08:51.483Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: aborting pipeline replication: peer="{Voter server-7572c5df-cec5-e537-b6f5-5c5072a1c45c 7572c5df-cec5-e537-b6f5-5c5072a1c45c}"
2026-03-29T01:08:51.483Z [INFO]  server-9f3a8ccd-ce94-236b-b3e5-7c0c87ed3d8f: aborting pipeline replication: peer="{Voter server-387efd00-8051-e887-ac4b-70e29c3c9eb2 387efd00-8051-e887-ac4b-70e29c3c9eb2}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true
2026-03-29T01:08:51.485Z [INFO]  server-8d715cb9-46df-10dc-37cf-55e26dabefa4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d715cb9-46df-10dc-37cf-55e26dabefa4 Address:8d715cb9-46df-10dc-37cf-55e26dabefa4} {Suffrage:Voter ID:server-4fb3fa7b-3aa7-9771-dc42-2376decd2d5b Address:4fb3fa7b-3aa7-9771-dc42-2376decd2d5b} {Suffrage:Voter ID:server-893a102c-caec-05af-fc7a-f19393049f82 Address:893a102c-caec-05af-fc7a-f19393049f82}]"
2026-03-29T01:08:51.485Z [INFO]  server-8d715cb9-46df-10dc-37cf-55e26dabefa4: entering follower state: follower="Node at 8d715cb9-46df-10dc-37cf-55e26dabefa4 [Follower]" leader-address= leader-id=
2026-03-29T01:08:51.485Z [INFO]  server-4fb3fa7b-3aa7-9771-dc42-2376decd2d5b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d715cb9-46df-10dc-37cf-55e26dabefa4 Address:8d715cb9-46df-10dc-37cf-55e26dabefa4} {Suffrage:Voter ID:server-4fb3fa7b-3aa7-9771-dc42-2376decd2d5b Address:4fb3fa7b-3aa7-9771-dc42-2376decd2d5b} {Suffrage:Voter ID:server-893a102c-caec-05af-fc7a-f19393049f82 Address:893a102c-caec-05af-fc7a-f19393049f82}]"
2026-03-29T01:08:51.485Z [INFO]  server-4fb3fa7b-3aa7-9771-dc42-2376decd2d5b: entering follower state: follower="Node at 4fb3fa7b-3aa7-9771-dc42-2376decd2d5b [Follower]" leader-address= leader-id=
2026-03-29T01:08:51.485Z [INFO]  server-893a102c-caec-05af-fc7a-f19393049f82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d715cb9-46df-10dc-37cf-55e26dabefa4 Address:8d715cb9-46df-10dc-37cf-55e26dabefa4} {Suffrage:Voter ID:server-4fb3fa7b-3aa7-9771-dc42-2376decd2d5b Address:4fb3fa7b-3aa7-9771-dc42-2376decd2d5b} {Suffrage:Voter ID:server-893a102c-caec-05af-fc7a-f19393049f82 Address:893a102c-caec-05af-fc7a-f19393049f82}]"
2026-03-29T01:08:51.485Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:51.485Z [INFO]  server-893a102c-caec-05af-fc7a-f19393049f82: entering follower state: follower="Node at 893a102c-caec-05af-fc7a-f19393049f82 [Follower]" leader-address= leader-id=
2026-03-29T01:08:52.126Z [WARN]  server-4fb3fa7b-3aa7-9771-dc42-2376decd2d5b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:52.126Z [INFO]  server-4fb3fa7b-3aa7-9771-dc42-2376decd2d5b: entering candidate state: node="Node at 4fb3fa7b-3aa7-9771-dc42-2376decd2d5b [Candidate]" term=2
2026-03-29T01:08:52.126Z [INFO]  server-4fb3fa7b-3aa7-9771-dc42-2376decd2d5b: election won: term=2 tally=2
2026-03-29T01:08:52.126Z [INFO]  server-4fb3fa7b-3aa7-9771-dc42-2376decd2d5b: entering leader state: leader="Node at 4fb3fa7b-3aa7-9771-dc42-2376decd2d5b [Leader]"
2026-03-29T01:08:52.126Z [INFO]  server-4fb3fa7b-3aa7-9771-dc42-2376decd2d5b: added peer, starting replication: peer=server-8d715cb9-46df-10dc-37cf-55e26dabefa4
2026-03-29T01:08:52.126Z [INFO]  server-4fb3fa7b-3aa7-9771-dc42-2376decd2d5b: added peer, starting replication: peer=server-893a102c-caec-05af-fc7a-f19393049f82
2026-03-29T01:08:52.126Z [INFO]  server-4fb3fa7b-3aa7-9771-dc42-2376decd2d5b: pipelining replication: peer="{Voter server-8d715cb9-46df-10dc-37cf-55e26dabefa4 8d715cb9-46df-10dc-37cf-55e26dabefa4}"
2026-03-29T01:08:52.126Z [INFO]  server-4fb3fa7b-3aa7-9771-dc42-2376decd2d5b: pipelining replication: peer="{Voter server-893a102c-caec-05af-fc7a-f19393049f82 893a102c-caec-05af-fc7a-f19393049f82}"
2026-03-29T01:08:53.131Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:52.126177409 +0000 UTC m=+75.914319945 (1 nodes), 640.852574ms from start of poll, 641.183865ms from cluster start. Timeout at 2026-03-29 01:08:53.131184882 +0000 UTC m=+76.919327406, 1.005007461s after stability
2026-03-29T01:08:53.134Z [INFO]  server-4fb3fa7b-3aa7-9771-dc42-2376decd2d5b: starting snapshot up to: index=12
2026-03-29T01:08:53.134Z [INFO]  creating new snapshot: path=/tmp/raft174736366/snapshots/2-12-1774746533134.tmp
2026-03-29T01:08:53.134Z [INFO]  server-4fb3fa7b-3aa7-9771-dc42-2376decd2d5b: snapshot complete up to: index=12
2026-03-29T01:08:53.135Z [INFO]  server-92030a3d-cc00-2892-f08b-78b3d7a69fbf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92030a3d-cc00-2892-f08b-78b3d7a69fbf Address:92030a3d-cc00-2892-f08b-78b3d7a69fbf} {Suffrage:Voter ID:server-98b3c316-725a-df64-6943-619044d9fb4c Address:98b3c316-725a-df64-6943-619044d9fb4c} {Suffrage:Voter ID:server-25baca1b-16a6-cf9f-7861-894a300888d4 Address:25baca1b-16a6-cf9f-7861-894a300888d4}]"
2026-03-29T01:08:53.135Z [INFO]  server-92030a3d-cc00-2892-f08b-78b3d7a69fbf: entering follower state: follower="Node at 92030a3d-cc00-2892-f08b-78b3d7a69fbf [Follower]" leader-address= leader-id=
2026-03-29T01:08:53.135Z [INFO]  server-98b3c316-725a-df64-6943-619044d9fb4c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92030a3d-cc00-2892-f08b-78b3d7a69fbf Address:92030a3d-cc00-2892-f08b-78b3d7a69fbf} {Suffrage:Voter ID:server-98b3c316-725a-df64-6943-619044d9fb4c Address:98b3c316-725a-df64-6943-619044d9fb4c} {Suffrage:Voter ID:server-25baca1b-16a6-cf9f-7861-894a300888d4 Address:25baca1b-16a6-cf9f-7861-894a300888d4}]"
2026-03-29T01:08:53.135Z [INFO]  server-98b3c316-725a-df64-6943-619044d9fb4c: entering follower state: follower="Node at 98b3c316-725a-df64-6943-619044d9fb4c [Follower]" leader-address= leader-id=
2026-03-29T01:08:53.135Z [INFO]  server-25baca1b-16a6-cf9f-7861-894a300888d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92030a3d-cc00-2892-f08b-78b3d7a69fbf Address:92030a3d-cc00-2892-f08b-78b3d7a69fbf} {Suffrage:Voter ID:server-98b3c316-725a-df64-6943-619044d9fb4c Address:98b3c316-725a-df64-6943-619044d9fb4c} {Suffrage:Voter ID:server-25baca1b-16a6-cf9f-7861-894a300888d4 Address:25baca1b-16a6-cf9f-7861-894a300888d4}]"
2026-03-29T01:08:53.135Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:53.135Z [INFO]  server-25baca1b-16a6-cf9f-7861-894a300888d4: entering follower state: follower="Node at 25baca1b-16a6-cf9f-7861-894a300888d4 [Follower]" leader-address= leader-id=
2026-03-29T01:08:53.715Z [WARN]  server-25baca1b-16a6-cf9f-7861-894a300888d4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:53.715Z [INFO]  server-25baca1b-16a6-cf9f-7861-894a300888d4: entering candidate state: node="Node at 25baca1b-16a6-cf9f-7861-894a300888d4 [Candidate]" term=2
2026-03-29T01:08:53.715Z [INFO]  server-25baca1b-16a6-cf9f-7861-894a300888d4: election won: term=2 tally=2
2026-03-29T01:08:53.715Z [INFO]  server-25baca1b-16a6-cf9f-7861-894a300888d4: entering leader state: leader="Node at 25baca1b-16a6-cf9f-7861-894a300888d4 [Leader]"
2026-03-29T01:08:53.715Z [INFO]  server-25baca1b-16a6-cf9f-7861-894a300888d4: added peer, starting replication: peer=server-92030a3d-cc00-2892-f08b-78b3d7a69fbf
2026-03-29T01:08:53.715Z [INFO]  server-25baca1b-16a6-cf9f-7861-894a300888d4: added peer, starting replication: peer=server-98b3c316-725a-df64-6943-619044d9fb4c
2026-03-29T01:08:53.715Z [INFO]  server-25baca1b-16a6-cf9f-7861-894a300888d4: pipelining replication: peer="{Voter server-92030a3d-cc00-2892-f08b-78b3d7a69fbf 92030a3d-cc00-2892-f08b-78b3d7a69fbf}"
2026-03-29T01:08:53.715Z [INFO]  server-25baca1b-16a6-cf9f-7861-894a300888d4: pipelining replication: peer="{Voter server-98b3c316-725a-df64-6943-619044d9fb4c 98b3c316-725a-df64-6943-619044d9fb4c}"
2026-03-29T01:08:54.721Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:53.715758689 +0000 UTC m=+77.503901234 (1 nodes), 579.80261ms from start of poll, 580.185074ms from cluster start. Timeout at 2026-03-29 01:08:54.720768547 +0000 UTC m=+78.508911100, 1.005009866s after stability
2026-03-29T01:08:54.721Z [INFO]  creating new snapshot: path=/tmp/raft2354530981/snapshots/2-1013-1774746534721.tmp
2026-03-29T01:08:54.722Z [INFO]  server-25baca1b-16a6-cf9f-7861-894a300888d4: copied to local snapshot: bytes=71
2026-03-29T01:08:54.722Z [INFO]  server-25baca1b-16a6-cf9f-7861-894a300888d4: snapshot restore progress: id=2-1013-1774746534721 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:54.722Z [INFO]  server-25baca1b-16a6-cf9f-7861-894a300888d4: restored user snapshot: index=1013
2026-03-29T01:08:54.722Z [ERROR] server-25baca1b-16a6-cf9f-7861-894a300888d4: failed to get log: index=3 error="log not found"
2026-03-29T01:08:54.722Z [ERROR] server-25baca1b-16a6-cf9f-7861-894a300888d4: failed to get log: index=3 error="log not found"
2026-03-29T01:08:54.722Z [INFO]  server-25baca1b-16a6-cf9f-7861-894a300888d4: aborting pipeline replication: peer="{Voter server-92030a3d-cc00-2892-f08b-78b3d7a69fbf 92030a3d-cc00-2892-f08b-78b3d7a69fbf}"
2026-03-29T01:08:54.722Z [INFO]  server-25baca1b-16a6-cf9f-7861-894a300888d4: aborting pipeline replication: peer="{Voter server-98b3c316-725a-df64-6943-619044d9fb4c 98b3c316-725a-df64-6943-619044d9fb4c}"
2026-03-29T01:08:54.731Z [ERROR] server-25baca1b-16a6-cf9f-7861-894a300888d4: failed to get log: index=3 error="log not found"
2026-03-29T01:08:54.731Z [ERROR] server-25baca1b-16a6-cf9f-7861-894a300888d4: failed to get log: index=3 error="log not found"
2026-03-29T01:08:54.731Z [INFO]  creating new snapshot: path=/tmp/raft468214386/snapshots/2-1013-1774746534731.tmp
2026-03-29T01:08:54.731Z [INFO]  creating new snapshot: path=/tmp/raft1379297109/snapshots/2-1013-1774746534731.tmp
2026-03-29T01:08:54.731Z [INFO]  server-98b3c316-725a-df64-6943-619044d9fb4c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:54.731Z [INFO]  server-98b3c316-725a-df64-6943-619044d9fb4c: copied to local snapshot: bytes=71
2026-03-29T01:08:54.731Z [INFO]  server-92030a3d-cc00-2892-f08b-78b3d7a69fbf: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:54.731Z [INFO]  server-98b3c316-725a-df64-6943-619044d9fb4c: snapshot restore progress: id=2-1013-1774746534731 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:54.731Z [INFO]  server-98b3c316-725a-df64-6943-619044d9fb4c: Installed remote snapshot
2026-03-29T01:08:54.732Z [INFO]  server-25baca1b-16a6-cf9f-7861-894a300888d4: pipelining replication: peer="{Voter server-98b3c316-725a-df64-6943-619044d9fb4c 98b3c316-725a-df64-6943-619044d9fb4c}"
2026-03-29T01:08:54.732Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:08:54.732Z [INFO]  server-92030a3d-cc00-2892-f08b-78b3d7a69fbf: copied to local snapshot: bytes=71
2026-03-29T01:08:54.732Z [INFO]  server-92030a3d-cc00-2892-f08b-78b3d7a69fbf: snapshot restore progress: id=2-1013-1774746534731 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:54.732Z [INFO]  server-92030a3d-cc00-2892-f08b-78b3d7a69fbf: Installed remote snapshot
2026-03-29T01:08:54.732Z [INFO]  server-25baca1b-16a6-cf9f-7861-894a300888d4: pipelining replication: peer="{Voter server-92030a3d-cc00-2892-f08b-78b3d7a69fbf 92030a3d-cc00-2892-f08b-78b3d7a69fbf}"
2026-03-29T01:08:55.737Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:08:54.732081864 +0000 UTC m=+78.520224409 (2 nodes), 948ns from start of poll, 1.596508249s from cluster start. Timeout at 2026-03-29 01:08:55.737089032 +0000 UTC m=+79.525231562, 1.005007153s after stability
2026-03-29T01:08:55.749Z [INFO]  server-4fb3fa7b-3aa7-9771-dc42-2376decd2d5b: aborting pipeline replication: peer="{Voter server-893a102c-caec-05af-fc7a-f19393049f82 893a102c-caec-05af-fc7a-f19393049f82}"
2026-03-29T01:08:55.749Z [INFO]  server-4fb3fa7b-3aa7-9771-dc42-2376decd2d5b: aborting pipeline replication: peer="{Voter server-8d715cb9-46df-10dc-37cf-55e26dabefa4 8d715cb9-46df-10dc-37cf-55e26dabefa4}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true
2026-03-29T01:08:55.750Z [INFO]  server-634bc534-fce2-420f-0e3b-9c757112b3c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-634bc534-fce2-420f-0e3b-9c757112b3c4 Address:634bc534-fce2-420f-0e3b-9c757112b3c4} {Suffrage:Voter ID:server-863620bc-512c-fa66-072e-baad46334a6c Address:863620bc-512c-fa66-072e-baad46334a6c} {Suffrage:Voter ID:server-3e6ef040-95a3-f631-385c-4b4261ab432c Address:3e6ef040-95a3-f631-385c-4b4261ab432c}]"
2026-03-29T01:08:55.750Z [INFO]  server-634bc534-fce2-420f-0e3b-9c757112b3c4: entering follower state: follower="Node at 634bc534-fce2-420f-0e3b-9c757112b3c4 [Follower]" leader-address= leader-id=
2026-03-29T01:08:55.750Z [INFO]  server-863620bc-512c-fa66-072e-baad46334a6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-634bc534-fce2-420f-0e3b-9c757112b3c4 Address:634bc534-fce2-420f-0e3b-9c757112b3c4} {Suffrage:Voter ID:server-863620bc-512c-fa66-072e-baad46334a6c Address:863620bc-512c-fa66-072e-baad46334a6c} {Suffrage:Voter ID:server-3e6ef040-95a3-f631-385c-4b4261ab432c Address:3e6ef040-95a3-f631-385c-4b4261ab432c}]"
2026-03-29T01:08:55.750Z [INFO]  server-863620bc-512c-fa66-072e-baad46334a6c: entering follower state: follower="Node at 863620bc-512c-fa66-072e-baad46334a6c [Follower]" leader-address= leader-id=
2026-03-29T01:08:55.750Z [INFO]  server-3e6ef040-95a3-f631-385c-4b4261ab432c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-634bc534-fce2-420f-0e3b-9c757112b3c4 Address:634bc534-fce2-420f-0e3b-9c757112b3c4} {Suffrage:Voter ID:server-863620bc-512c-fa66-072e-baad46334a6c Address:863620bc-512c-fa66-072e-baad46334a6c} {Suffrage:Voter ID:server-3e6ef040-95a3-f631-385c-4b4261ab432c Address:3e6ef040-95a3-f631-385c-4b4261ab432c}]"
2026-03-29T01:08:55.750Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:55.750Z [INFO]  server-3e6ef040-95a3-f631-385c-4b4261ab432c: entering follower state: follower="Node at 3e6ef040-95a3-f631-385c-4b4261ab432c [Follower]" leader-address= leader-id=
2026-03-29T01:08:56.278Z [WARN]  server-3e6ef040-95a3-f631-385c-4b4261ab432c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:56.278Z [INFO]  server-3e6ef040-95a3-f631-385c-4b4261ab432c: entering candidate state: node="Node at 3e6ef040-95a3-f631-385c-4b4261ab432c [Candidate]" term=2
2026-03-29T01:08:56.278Z [INFO]  server-3e6ef040-95a3-f631-385c-4b4261ab432c: election won: term=2 tally=2
2026-03-29T01:08:56.278Z [INFO]  server-3e6ef040-95a3-f631-385c-4b4261ab432c: entering leader state: leader="Node at 3e6ef040-95a3-f631-385c-4b4261ab432c [Leader]"
2026-03-29T01:08:56.278Z [INFO]  server-3e6ef040-95a3-f631-385c-4b4261ab432c: added peer, starting replication: peer=server-634bc534-fce2-420f-0e3b-9c757112b3c4
2026-03-29T01:08:56.278Z [INFO]  server-3e6ef040-95a3-f631-385c-4b4261ab432c: added peer, starting replication: peer=server-863620bc-512c-fa66-072e-baad46334a6c
2026-03-29T01:08:56.278Z [INFO]  server-3e6ef040-95a3-f631-385c-4b4261ab432c: pipelining replication: peer="{Voter server-863620bc-512c-fa66-072e-baad46334a6c 863620bc-512c-fa66-072e-baad46334a6c}"
2026-03-29T01:08:56.278Z [INFO]  server-3e6ef040-95a3-f631-385c-4b4261ab432c: pipelining replication: peer="{Voter server-634bc534-fce2-420f-0e3b-9c757112b3c4 634bc534-fce2-420f-0e3b-9c757112b3c4}"
2026-03-29T01:08:57.284Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:56.278829824 +0000 UTC m=+80.066972378 (1 nodes), 528.430611ms from start of poll, 528.77796ms from cluster start. Timeout at 2026-03-29 01:08:57.283835772 +0000 UTC m=+81.071978341, 1.005005963s after stability
2026-03-29T01:08:57.285Z [INFO]  server-3e6ef040-95a3-f631-385c-4b4261ab432c: starting snapshot up to: index=12
2026-03-29T01:08:57.285Z [INFO]  creating new snapshot: path=/tmp/raft1595877514/snapshots/2-12-1774746537285.tmp
2026-03-29T01:08:57.286Z [INFO]  server-3e6ef040-95a3-f631-385c-4b4261ab432c: snapshot complete up to: index=12
2026-03-29T01:08:57.287Z [INFO]  server-7e093910-15c7-075a-3052-2e22ce76f503: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e093910-15c7-075a-3052-2e22ce76f503 Address:7e093910-15c7-075a-3052-2e22ce76f503} {Suffrage:Voter ID:server-3d007a97-d344-9f9e-1e6c-1380071318c1 Address:3d007a97-d344-9f9e-1e6c-1380071318c1} {Suffrage:Voter ID:server-c66b00c5-cb97-c61c-a171-451fc47d8306 Address:c66b00c5-cb97-c61c-a171-451fc47d8306}]"
2026-03-29T01:08:57.287Z [INFO]  server-7e093910-15c7-075a-3052-2e22ce76f503: entering follower state: follower="Node at 7e093910-15c7-075a-3052-2e22ce76f503 [Follower]" leader-address= leader-id=
2026-03-29T01:08:57.287Z [INFO]  server-3d007a97-d344-9f9e-1e6c-1380071318c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e093910-15c7-075a-3052-2e22ce76f503 Address:7e093910-15c7-075a-3052-2e22ce76f503} {Suffrage:Voter ID:server-3d007a97-d344-9f9e-1e6c-1380071318c1 Address:3d007a97-d344-9f9e-1e6c-1380071318c1} {Suffrage:Voter ID:server-c66b00c5-cb97-c61c-a171-451fc47d8306 Address:c66b00c5-cb97-c61c-a171-451fc47d8306}]"
2026-03-29T01:08:57.287Z [INFO]  server-3d007a97-d344-9f9e-1e6c-1380071318c1: entering follower state: follower="Node at 3d007a97-d344-9f9e-1e6c-1380071318c1 [Follower]" leader-address= leader-id=
2026-03-29T01:08:57.287Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7e093910-15c7-075a-3052-2e22ce76f503 Address:7e093910-15c7-075a-3052-2e22ce76f503} {Suffrage:Voter ID:server-3d007a97-d344-9f9e-1e6c-1380071318c1 Address:3d007a97-d344-9f9e-1e6c-1380071318c1} {Suffrage:Voter ID:server-c66b00c5-cb97-c61c-a171-451fc47d8306 Address:c66b00c5-cb97-c61c-a171-451fc47d8306}]"
2026-03-29T01:08:57.287Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:08:57.287Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: entering follower state: follower="Node at c66b00c5-cb97-c61c-a171-451fc47d8306 [Follower]" leader-address= leader-id=
2026-03-29T01:08:58.033Z [WARN]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:08:58.033Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: entering candidate state: node="Node at c66b00c5-cb97-c61c-a171-451fc47d8306 [Candidate]" term=2
2026-03-29T01:08:58.033Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: election won: term=2 tally=2
2026-03-29T01:08:58.033Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: entering leader state: leader="Node at c66b00c5-cb97-c61c-a171-451fc47d8306 [Leader]"
2026-03-29T01:08:58.033Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: added peer, starting replication: peer=server-7e093910-15c7-075a-3052-2e22ce76f503
2026-03-29T01:08:58.034Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: added peer, starting replication: peer=server-3d007a97-d344-9f9e-1e6c-1380071318c1
2026-03-29T01:08:58.034Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: pipelining replication: peer="{Voter server-7e093910-15c7-075a-3052-2e22ce76f503 7e093910-15c7-075a-3052-2e22ce76f503}"
2026-03-29T01:08:58.034Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: pipelining replication: peer="{Voter server-3d007a97-d344-9f9e-1e6c-1380071318c1 3d007a97-d344-9f9e-1e6c-1380071318c1}"
2026-03-29T01:08:59.039Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:08:58.03399564 +0000 UTC m=+81.822138173 (1 nodes), 746.158691ms from start of poll, 746.50824ms from cluster start. Timeout at 2026-03-29 01:08:59.038998666 +0000 UTC m=+82.827141204, 1.005003031s after stability
2026-03-29T01:08:59.039Z [INFO]  creating new snapshot: path=/tmp/raft347741147/snapshots/2-1013-1774746539039.tmp
2026-03-29T01:08:59.040Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: copied to local snapshot: bytes=71
2026-03-29T01:08:59.040Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: snapshot restore progress: id=2-1013-1774746539039 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:59.040Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: removing all old logs from log store
2026-03-29T01:08:59.040Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: compacting logs: from=1 to=2
2026-03-29T01:08:59.040Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: restored user snapshot: index=1013
2026-03-29T01:08:59.040Z [ERROR] server-c66b00c5-cb97-c61c-a171-451fc47d8306: failed to get log: index=2 error="log not found"
2026-03-29T01:08:59.040Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: aborting pipeline replication: peer="{Voter server-3d007a97-d344-9f9e-1e6c-1380071318c1 3d007a97-d344-9f9e-1e6c-1380071318c1}"
2026-03-29T01:08:59.040Z [ERROR] server-c66b00c5-cb97-c61c-a171-451fc47d8306: failed to get log: index=2 error="log not found"
2026-03-29T01:08:59.040Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: aborting pipeline replication: peer="{Voter server-7e093910-15c7-075a-3052-2e22ce76f503 7e093910-15c7-075a-3052-2e22ce76f503}"
2026-03-29T01:08:59.047Z [ERROR] server-c66b00c5-cb97-c61c-a171-451fc47d8306: failed to get log: index=2 error="log not found"
2026-03-29T01:08:59.048Z [INFO]  creating new snapshot: path=/tmp/raft929376960/snapshots/2-1013-1774746539048.tmp
2026-03-29T01:08:59.048Z [INFO]  server-7e093910-15c7-075a-3052-2e22ce76f503: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:59.048Z [INFO]  server-7e093910-15c7-075a-3052-2e22ce76f503: copied to local snapshot: bytes=71
2026-03-29T01:08:59.048Z [INFO]  server-7e093910-15c7-075a-3052-2e22ce76f503: snapshot restore progress: id=2-1013-1774746539048 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:59.048Z [INFO]  server-7e093910-15c7-075a-3052-2e22ce76f503: removing all old logs from log store
2026-03-29T01:08:59.048Z [INFO]  server-7e093910-15c7-075a-3052-2e22ce76f503: compacting logs: from=1 to=2
2026-03-29T01:08:59.048Z [INFO]  server-7e093910-15c7-075a-3052-2e22ce76f503: Installed remote snapshot
2026-03-29T01:08:59.049Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: pipelining replication: peer="{Voter server-7e093910-15c7-075a-3052-2e22ce76f503 7e093910-15c7-075a-3052-2e22ce76f503}"
2026-03-29T01:08:59.049Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:08:59.050Z [ERROR] server-c66b00c5-cb97-c61c-a171-451fc47d8306: failed to get log: index=2 error="log not found"
2026-03-29T01:08:59.050Z [INFO]  creating new snapshot: path=/tmp/raft453049360/snapshots/2-1013-1774746539050.tmp
2026-03-29T01:08:59.050Z [INFO]  server-3d007a97-d344-9f9e-1e6c-1380071318c1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:59.051Z [INFO]  server-3d007a97-d344-9f9e-1e6c-1380071318c1: copied to local snapshot: bytes=71
2026-03-29T01:08:59.051Z [INFO]  server-3d007a97-d344-9f9e-1e6c-1380071318c1: snapshot restore progress: id=2-1013-1774746539050 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:08:59.051Z [INFO]  server-3d007a97-d344-9f9e-1e6c-1380071318c1: removing all old logs from log store
2026-03-29T01:08:59.051Z [INFO]  server-3d007a97-d344-9f9e-1e6c-1380071318c1: compacting logs: from=1 to=2
2026-03-29T01:08:59.051Z [INFO]  server-3d007a97-d344-9f9e-1e6c-1380071318c1: Installed remote snapshot
2026-03-29T01:08:59.051Z [INFO]  server-c66b00c5-cb97-c61c-a171-451fc47d8306: pipelining replication: peer="{Voter server-3d007a97-d344-9f9e-1e6c-1380071318c1 3d007a97-d344-9f9e-1e6c-1380071318c1}"
2026-03-29T01:09:00.054Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:08:59.049130344 +0000 UTC m=+82.837272879 (2 nodes), 687ns from start of poll, 1.761642946s from cluster start. Timeout at 2026-03-29 01:09:00.05415347 +0000 UTC m=+83.842296015, 1.005023136s after stability
2026-03-29T01:09:00.066Z [INFO]  server-3e6ef040-95a3-f631-385c-4b4261ab432c: aborting pipeline replication: peer="{Voter server-863620bc-512c-fa66-072e-baad46334a6c 863620bc-512c-fa66-072e-baad46334a6c}"
2026-03-29T01:09:00.066Z [INFO]  server-3e6ef040-95a3-f631-385c-4b4261ab432c: aborting pipeline replication: peer="{Voter server-634bc534-fce2-420f-0e3b-9c757112b3c4 634bc534-fce2-420f-0e3b-9c757112b3c4}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false
2026-03-29T01:09:00.067Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff Address:8f885fd5-c39e-0ec5-1c0e-f9490ba286ff} {Suffrage:Voter ID:server-24e90832-7847-ba3f-0ab5-291d08bb716f Address:24e90832-7847-ba3f-0ab5-291d08bb716f} {Suffrage:Voter ID:server-91c76697-898c-8114-357a-c4cda056b856 Address:91c76697-898c-8114-357a-c4cda056b856}]"
2026-03-29T01:09:00.067Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: entering follower state: follower="Node at 8f885fd5-c39e-0ec5-1c0e-f9490ba286ff [Follower]" leader-address= leader-id=
2026-03-29T01:09:00.067Z [INFO]  server-24e90832-7847-ba3f-0ab5-291d08bb716f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff Address:8f885fd5-c39e-0ec5-1c0e-f9490ba286ff} {Suffrage:Voter ID:server-24e90832-7847-ba3f-0ab5-291d08bb716f Address:24e90832-7847-ba3f-0ab5-291d08bb716f} {Suffrage:Voter ID:server-91c76697-898c-8114-357a-c4cda056b856 Address:91c76697-898c-8114-357a-c4cda056b856}]"
2026-03-29T01:09:00.067Z [INFO]  server-24e90832-7847-ba3f-0ab5-291d08bb716f: entering follower state: follower="Node at 24e90832-7847-ba3f-0ab5-291d08bb716f [Follower]" leader-address= leader-id=
2026-03-29T01:09:00.067Z [INFO]  server-91c76697-898c-8114-357a-c4cda056b856: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff Address:8f885fd5-c39e-0ec5-1c0e-f9490ba286ff} {Suffrage:Voter ID:server-24e90832-7847-ba3f-0ab5-291d08bb716f Address:24e90832-7847-ba3f-0ab5-291d08bb716f} {Suffrage:Voter ID:server-91c76697-898c-8114-357a-c4cda056b856 Address:91c76697-898c-8114-357a-c4cda056b856}]"
2026-03-29T01:09:00.067Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:00.067Z [INFO]  server-91c76697-898c-8114-357a-c4cda056b856: entering follower state: follower="Node at 91c76697-898c-8114-357a-c4cda056b856 [Follower]" leader-address= leader-id=
2026-03-29T01:09:00.582Z [WARN]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:00.582Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: entering candidate state: node="Node at 8f885fd5-c39e-0ec5-1c0e-f9490ba286ff [Candidate]" term=2
2026-03-29T01:09:00.582Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: election won: term=2 tally=2
2026-03-29T01:09:00.582Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: entering leader state: leader="Node at 8f885fd5-c39e-0ec5-1c0e-f9490ba286ff [Leader]"
2026-03-29T01:09:00.582Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: added peer, starting replication: peer=server-24e90832-7847-ba3f-0ab5-291d08bb716f
2026-03-29T01:09:00.582Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: added peer, starting replication: peer=server-91c76697-898c-8114-357a-c4cda056b856
2026-03-29T01:09:00.583Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: pipelining replication: peer="{Voter server-24e90832-7847-ba3f-0ab5-291d08bb716f 24e90832-7847-ba3f-0ab5-291d08bb716f}"
2026-03-29T01:09:00.583Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: pipelining replication: peer="{Voter server-91c76697-898c-8114-357a-c4cda056b856 91c76697-898c-8114-357a-c4cda056b856}"
2026-03-29T01:09:01.588Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:00.582918111 +0000 UTC m=+84.371060673 (1 nodes), 515.272747ms from start of poll, 515.55866ms from cluster start. Timeout at 2026-03-29 01:09:01.587919476 +0000 UTC m=+85.376062015, 1.005001342s after stability
2026-03-29T01:09:01.589Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: starting snapshot up to: index=12
2026-03-29T01:09:01.589Z [INFO]  creating new snapshot: path=/tmp/raft1711055654/snapshots/2-12-1774746541589.tmp
2026-03-29T01:09:01.589Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: snapshot complete up to: index=12
2026-03-29T01:09:01.590Z [INFO]  creating new snapshot: path=/tmp/raft1711055654/snapshots/2-10013-1774746541590.tmp
2026-03-29T01:09:01.590Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: copied to local snapshot: bytes=71
2026-03-29T01:09:01.590Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: snapshot restore progress: id=2-10013-1774746541590 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:01.590Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: restored user snapshot: index=10013
2026-03-29T01:09:01.590Z [ERROR] server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: failed to get log: index=23 error="log not found"
2026-03-29T01:09:01.591Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: aborting pipeline replication: peer="{Voter server-91c76697-898c-8114-357a-c4cda056b856 91c76697-898c-8114-357a-c4cda056b856}"
2026-03-29T01:09:01.591Z [ERROR] server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: failed to get log: index=23 error="log not found"
2026-03-29T01:09:01.591Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: aborting pipeline replication: peer="{Voter server-24e90832-7847-ba3f-0ab5-291d08bb716f 24e90832-7847-ba3f-0ab5-291d08bb716f}"
2026-03-29T01:09:01.597Z [ERROR] server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: failed to get log: index=23 error="log not found"
2026-03-29T01:09:01.598Z [INFO]  creating new snapshot: path=/tmp/raft1474066645/snapshots/2-10013-1774746541598.tmp
2026-03-29T01:09:01.598Z [INFO]  server-91c76697-898c-8114-357a-c4cda056b856: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:01.598Z [INFO]  server-91c76697-898c-8114-357a-c4cda056b856: copied to local snapshot: bytes=71
2026-03-29T01:09:01.599Z [INFO]  server-91c76697-898c-8114-357a-c4cda056b856: snapshot restore progress: id=2-10013-1774746541598 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:01.599Z [INFO]  server-91c76697-898c-8114-357a-c4cda056b856: Installed remote snapshot
2026-03-29T01:09:01.599Z [WARN]  server-91c76697-898c-8114-357a-c4cda056b856: skipping application of old log: index=22
2026-03-29T01:09:01.599Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: pipelining replication: peer="{Voter server-91c76697-898c-8114-357a-c4cda056b856 91c76697-898c-8114-357a-c4cda056b856}"
2026-03-29T01:09:01.599Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:01.601Z [ERROR] server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: failed to get log: index=23 error="log not found"
2026-03-29T01:09:01.601Z [INFO]  creating new snapshot: path=/tmp/raft2201628434/snapshots/2-10013-1774746541601.tmp
2026-03-29T01:09:01.602Z [INFO]  server-24e90832-7847-ba3f-0ab5-291d08bb716f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:01.602Z [INFO]  server-24e90832-7847-ba3f-0ab5-291d08bb716f: copied to local snapshot: bytes=71
2026-03-29T01:09:01.602Z [INFO]  server-24e90832-7847-ba3f-0ab5-291d08bb716f: snapshot restore progress: id=2-10013-1774746541601 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:01.602Z [INFO]  server-24e90832-7847-ba3f-0ab5-291d08bb716f: Installed remote snapshot
2026-03-29T01:09:01.602Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: pipelining replication: peer="{Voter server-24e90832-7847-ba3f-0ab5-291d08bb716f 24e90832-7847-ba3f-0ab5-291d08bb716f}"
2026-03-29T01:09:02.605Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:01.599437014 +0000 UTC m=+85.387579562 (2 nodes), 2.034µs from start of poll, 1.532077549s from cluster start. Timeout at 2026-03-29 01:09:02.604443557 +0000 UTC m=+86.392586095, 1.005006533s after stability
2026-03-29T01:09:02.616Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: aborting pipeline replication: peer="{Voter server-91c76697-898c-8114-357a-c4cda056b856 91c76697-898c-8114-357a-c4cda056b856}"
2026-03-29T01:09:02.616Z [INFO]  server-8f885fd5-c39e-0ec5-1c0e-f9490ba286ff: aborting pipeline replication: peer="{Voter server-24e90832-7847-ba3f-0ab5-291d08bb716f 24e90832-7847-ba3f-0ab5-291d08bb716f}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false
2026-03-29T01:09:02.618Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643 Address:5606aa9f-b7ba-e5e0-8f82-1522b1bbc643} {Suffrage:Voter ID:server-ab1ba19a-5b1c-f068-62c4-508ba39e4f1a Address:ab1ba19a-5b1c-f068-62c4-508ba39e4f1a} {Suffrage:Voter ID:server-1077965c-edfc-4fba-2cc2-febd166cf984 Address:1077965c-edfc-4fba-2cc2-febd166cf984}]"
2026-03-29T01:09:02.618Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: entering follower state: follower="Node at 5606aa9f-b7ba-e5e0-8f82-1522b1bbc643 [Follower]" leader-address= leader-id=
2026-03-29T01:09:02.618Z [INFO]  server-ab1ba19a-5b1c-f068-62c4-508ba39e4f1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643 Address:5606aa9f-b7ba-e5e0-8f82-1522b1bbc643} {Suffrage:Voter ID:server-ab1ba19a-5b1c-f068-62c4-508ba39e4f1a Address:ab1ba19a-5b1c-f068-62c4-508ba39e4f1a} {Suffrage:Voter ID:server-1077965c-edfc-4fba-2cc2-febd166cf984 Address:1077965c-edfc-4fba-2cc2-febd166cf984}]"
2026-03-29T01:09:02.618Z [INFO]  server-ab1ba19a-5b1c-f068-62c4-508ba39e4f1a: entering follower state: follower="Node at ab1ba19a-5b1c-f068-62c4-508ba39e4f1a [Follower]" leader-address= leader-id=
2026-03-29T01:09:02.618Z [INFO]  server-1077965c-edfc-4fba-2cc2-febd166cf984: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643 Address:5606aa9f-b7ba-e5e0-8f82-1522b1bbc643} {Suffrage:Voter ID:server-ab1ba19a-5b1c-f068-62c4-508ba39e4f1a Address:ab1ba19a-5b1c-f068-62c4-508ba39e4f1a} {Suffrage:Voter ID:server-1077965c-edfc-4fba-2cc2-febd166cf984 Address:1077965c-edfc-4fba-2cc2-febd166cf984}]"
2026-03-29T01:09:02.618Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:02.618Z [INFO]  server-1077965c-edfc-4fba-2cc2-febd166cf984: entering follower state: follower="Node at 1077965c-edfc-4fba-2cc2-febd166cf984 [Follower]" leader-address= leader-id=
2026-03-29T01:09:03.221Z [WARN]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:03.221Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: entering candidate state: node="Node at 5606aa9f-b7ba-e5e0-8f82-1522b1bbc643 [Candidate]" term=2
2026-03-29T01:09:03.221Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: election won: term=2 tally=2
2026-03-29T01:09:03.221Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: entering leader state: leader="Node at 5606aa9f-b7ba-e5e0-8f82-1522b1bbc643 [Leader]"
2026-03-29T01:09:03.221Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: added peer, starting replication: peer=server-ab1ba19a-5b1c-f068-62c4-508ba39e4f1a
2026-03-29T01:09:03.221Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: added peer, starting replication: peer=server-1077965c-edfc-4fba-2cc2-febd166cf984
2026-03-29T01:09:03.221Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: pipelining replication: peer="{Voter server-1077965c-edfc-4fba-2cc2-febd166cf984 1077965c-edfc-4fba-2cc2-febd166cf984}"
2026-03-29T01:09:03.221Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: pipelining replication: peer="{Voter server-ab1ba19a-5b1c-f068-62c4-508ba39e4f1a ab1ba19a-5b1c-f068-62c4-508ba39e4f1a}"
2026-03-29T01:09:04.227Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:03.22177862 +0000 UTC m=+87.009921154 (1 nodes), 603.388511ms from start of poll, 603.781354ms from cluster start. Timeout at 2026-03-29 01:09:04.226785102 +0000 UTC m=+88.014927708, 1.005006554s after stability
2026-03-29T01:09:04.228Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: starting snapshot up to: index=12
2026-03-29T01:09:04.228Z [INFO]  creating new snapshot: path=/tmp/raft3091038045/snapshots/2-12-1774746544228.tmp
2026-03-29T01:09:04.229Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: snapshot complete up to: index=12
2026-03-29T01:09:04.229Z [INFO]  creating new snapshot: path=/tmp/raft3091038045/snapshots/2-10013-1774746544229.tmp
2026-03-29T01:09:04.230Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: copied to local snapshot: bytes=71
2026-03-29T01:09:04.230Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: snapshot restore progress: id=2-10013-1774746544229 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:04.230Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: removing all old logs from log store
2026-03-29T01:09:04.230Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: compacting logs: from=1 to=22
2026-03-29T01:09:04.230Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: restored user snapshot: index=10013
2026-03-29T01:09:04.230Z [ERROR] server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: failed to get log: index=22 error="log not found"
2026-03-29T01:09:04.230Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: aborting pipeline replication: peer="{Voter server-1077965c-edfc-4fba-2cc2-febd166cf984 1077965c-edfc-4fba-2cc2-febd166cf984}"
2026-03-29T01:09:04.230Z [ERROR] server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: failed to get log: index=22 error="log not found"
2026-03-29T01:09:04.230Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: aborting pipeline replication: peer="{Voter server-ab1ba19a-5b1c-f068-62c4-508ba39e4f1a ab1ba19a-5b1c-f068-62c4-508ba39e4f1a}"
2026-03-29T01:09:04.239Z [ERROR] server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: failed to get log: index=22 error="log not found"
2026-03-29T01:09:04.239Z [INFO]  creating new snapshot: path=/tmp/raft2232064666/snapshots/2-10013-1774746544239.tmp
2026-03-29T01:09:04.239Z [INFO]  server-ab1ba19a-5b1c-f068-62c4-508ba39e4f1a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:04.240Z [INFO]  server-ab1ba19a-5b1c-f068-62c4-508ba39e4f1a: copied to local snapshot: bytes=71
2026-03-29T01:09:04.240Z [INFO]  server-ab1ba19a-5b1c-f068-62c4-508ba39e4f1a: snapshot restore progress: id=2-10013-1774746544239 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:04.240Z [INFO]  server-ab1ba19a-5b1c-f068-62c4-508ba39e4f1a: removing all old logs from log store
2026-03-29T01:09:04.240Z [INFO]  server-ab1ba19a-5b1c-f068-62c4-508ba39e4f1a: compacting logs: from=1 to=22
2026-03-29T01:09:04.240Z [INFO]  server-ab1ba19a-5b1c-f068-62c4-508ba39e4f1a: Installed remote snapshot
2026-03-29T01:09:04.240Z [WARN]  server-ab1ba19a-5b1c-f068-62c4-508ba39e4f1a: skipping application of old log: index=22
2026-03-29T01:09:04.240Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: pipelining replication: peer="{Voter server-ab1ba19a-5b1c-f068-62c4-508ba39e4f1a ab1ba19a-5b1c-f068-62c4-508ba39e4f1a}"
2026-03-29T01:09:04.240Z [ERROR] server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: failed to get log: index=22 error="log not found"
2026-03-29T01:09:04.240Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:04.240Z [INFO]  creating new snapshot: path=/tmp/raft789732952/snapshots/2-10013-1774746544240.tmp
2026-03-29T01:09:04.241Z [INFO]  server-1077965c-edfc-4fba-2cc2-febd166cf984: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:04.241Z [INFO]  server-1077965c-edfc-4fba-2cc2-febd166cf984: copied to local snapshot: bytes=71
2026-03-29T01:09:04.241Z [INFO]  server-1077965c-edfc-4fba-2cc2-febd166cf984: snapshot restore progress: id=2-10013-1774746544240 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:04.241Z [INFO]  server-1077965c-edfc-4fba-2cc2-febd166cf984: removing all old logs from log store
2026-03-29T01:09:04.241Z [INFO]  server-1077965c-edfc-4fba-2cc2-febd166cf984: compacting logs: from=1 to=22
2026-03-29T01:09:04.241Z [INFO]  server-1077965c-edfc-4fba-2cc2-febd166cf984: Installed remote snapshot
2026-03-29T01:09:04.241Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: pipelining replication: peer="{Voter server-1077965c-edfc-4fba-2cc2-febd166cf984 1077965c-edfc-4fba-2cc2-febd166cf984}"
2026-03-29T01:09:05.245Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:04.240565955 +0000 UTC m=+88.028708502 (2 nodes), 1.445µs from start of poll, 1.622568702s from cluster start. Timeout at 2026-03-29 01:09:05.24557698 +0000 UTC m=+89.033719525, 1.005011023s after stability
2026-03-29T01:09:05.258Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: aborting pipeline replication: peer="{Voter server-1077965c-edfc-4fba-2cc2-febd166cf984 1077965c-edfc-4fba-2cc2-febd166cf984}"
2026-03-29T01:09:05.258Z [INFO]  server-5606aa9f-b7ba-e5e0-8f82-1522b1bbc643: aborting pipeline replication: peer="{Voter server-ab1ba19a-5b1c-f068-62c4-508ba39e4f1a ab1ba19a-5b1c-f068-62c4-508ba39e4f1a}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true
2026-03-29T01:09:05.259Z [INFO]  server-3dd430e9-302d-e80b-b190-0be04ecd04ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3dd430e9-302d-e80b-b190-0be04ecd04ed Address:3dd430e9-302d-e80b-b190-0be04ecd04ed} {Suffrage:Voter ID:server-9e0a13c2-66d4-029c-5b20-37b28c1e3611 Address:9e0a13c2-66d4-029c-5b20-37b28c1e3611} {Suffrage:Voter ID:server-7e692399-117f-aff9-1274-3c32154969d0 Address:7e692399-117f-aff9-1274-3c32154969d0}]"
2026-03-29T01:09:05.259Z [INFO]  server-9e0a13c2-66d4-029c-5b20-37b28c1e3611: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3dd430e9-302d-e80b-b190-0be04ecd04ed Address:3dd430e9-302d-e80b-b190-0be04ecd04ed} {Suffrage:Voter ID:server-9e0a13c2-66d4-029c-5b20-37b28c1e3611 Address:9e0a13c2-66d4-029c-5b20-37b28c1e3611} {Suffrage:Voter ID:server-7e692399-117f-aff9-1274-3c32154969d0 Address:7e692399-117f-aff9-1274-3c32154969d0}]"
2026-03-29T01:09:05.259Z [INFO]  server-3dd430e9-302d-e80b-b190-0be04ecd04ed: entering follower state: follower="Node at 3dd430e9-302d-e80b-b190-0be04ecd04ed [Follower]" leader-address= leader-id=
2026-03-29T01:09:05.259Z [INFO]  server-9e0a13c2-66d4-029c-5b20-37b28c1e3611: entering follower state: follower="Node at 9e0a13c2-66d4-029c-5b20-37b28c1e3611 [Follower]" leader-address= leader-id=
2026-03-29T01:09:05.259Z [INFO]  server-7e692399-117f-aff9-1274-3c32154969d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3dd430e9-302d-e80b-b190-0be04ecd04ed Address:3dd430e9-302d-e80b-b190-0be04ecd04ed} {Suffrage:Voter ID:server-9e0a13c2-66d4-029c-5b20-37b28c1e3611 Address:9e0a13c2-66d4-029c-5b20-37b28c1e3611} {Suffrage:Voter ID:server-7e692399-117f-aff9-1274-3c32154969d0 Address:7e692399-117f-aff9-1274-3c32154969d0}]"
2026-03-29T01:09:05.259Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:05.259Z [INFO]  server-7e692399-117f-aff9-1274-3c32154969d0: entering follower state: follower="Node at 7e692399-117f-aff9-1274-3c32154969d0 [Follower]" leader-address= leader-id=
2026-03-29T01:09:05.976Z [WARN]  server-3dd430e9-302d-e80b-b190-0be04ecd04ed: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:05.977Z [INFO]  server-3dd430e9-302d-e80b-b190-0be04ecd04ed: entering candidate state: node="Node at 3dd430e9-302d-e80b-b190-0be04ecd04ed [Candidate]" term=2
2026-03-29T01:09:05.977Z [INFO]  server-3dd430e9-302d-e80b-b190-0be04ecd04ed: election won: term=2 tally=2
2026-03-29T01:09:05.977Z [INFO]  server-3dd430e9-302d-e80b-b190-0be04ecd04ed: entering leader state: leader="Node at 3dd430e9-302d-e80b-b190-0be04ecd04ed [Leader]"
2026-03-29T01:09:05.977Z [INFO]  server-3dd430e9-302d-e80b-b190-0be04ecd04ed: added peer, starting replication: peer=server-9e0a13c2-66d4-029c-5b20-37b28c1e3611
2026-03-29T01:09:05.977Z [INFO]  server-3dd430e9-302d-e80b-b190-0be04ecd04ed: added peer, starting replication: peer=server-7e692399-117f-aff9-1274-3c32154969d0
2026-03-29T01:09:05.977Z [INFO]  server-3dd430e9-302d-e80b-b190-0be04ecd04ed: pipelining replication: peer="{Voter server-7e692399-117f-aff9-1274-3c32154969d0 7e692399-117f-aff9-1274-3c32154969d0}"
2026-03-29T01:09:05.977Z [INFO]  server-3dd430e9-302d-e80b-b190-0be04ecd04ed: pipelining replication: peer="{Voter server-9e0a13c2-66d4-029c-5b20-37b28c1e3611 9e0a13c2-66d4-029c-5b20-37b28c1e3611}"
2026-03-29T01:09:06.982Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:05.977092979 +0000 UTC m=+89.765235533 (1 nodes), 717.525756ms from start of poll, 717.864497ms from cluster start. Timeout at 2026-03-29 01:09:06.982093953 +0000 UTC m=+90.770236497, 1.005000964s after stability
2026-03-29T01:09:06.982Z [INFO]  server-3dd430e9-302d-e80b-b190-0be04ecd04ed: starting snapshot up to: index=12
2026-03-29T01:09:06.982Z [INFO]  creating new snapshot: path=/tmp/raft893475772/snapshots/2-12-1774746546982.tmp
2026-03-29T01:09:06.983Z [INFO]  server-3dd430e9-302d-e80b-b190-0be04ecd04ed: snapshot complete up to: index=12
2026-03-29T01:09:06.984Z [INFO]  server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b Address:e8b5889a-0e4d-5f76-99db-6f01fe6df05b} {Suffrage:Voter ID:server-b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6 Address:b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6} {Suffrage:Voter ID:server-99482ef4-51bb-ed9f-4283-e08dc7f221c5 Address:99482ef4-51bb-ed9f-4283-e08dc7f221c5}]"
2026-03-29T01:09:06.984Z [INFO]  server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: entering follower state: follower="Node at e8b5889a-0e4d-5f76-99db-6f01fe6df05b [Follower]" leader-address= leader-id=
2026-03-29T01:09:06.984Z [INFO]  server-b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b Address:e8b5889a-0e4d-5f76-99db-6f01fe6df05b} {Suffrage:Voter ID:server-b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6 Address:b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6} {Suffrage:Voter ID:server-99482ef4-51bb-ed9f-4283-e08dc7f221c5 Address:99482ef4-51bb-ed9f-4283-e08dc7f221c5}]"
2026-03-29T01:09:06.984Z [INFO]  server-b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6: entering follower state: follower="Node at b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6 [Follower]" leader-address= leader-id=
2026-03-29T01:09:06.984Z [INFO]  server-99482ef4-51bb-ed9f-4283-e08dc7f221c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b Address:e8b5889a-0e4d-5f76-99db-6f01fe6df05b} {Suffrage:Voter ID:server-b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6 Address:b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6} {Suffrage:Voter ID:server-99482ef4-51bb-ed9f-4283-e08dc7f221c5 Address:99482ef4-51bb-ed9f-4283-e08dc7f221c5}]"
2026-03-29T01:09:06.984Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:06.984Z [INFO]  server-99482ef4-51bb-ed9f-4283-e08dc7f221c5: entering follower state: follower="Node at 99482ef4-51bb-ed9f-4283-e08dc7f221c5 [Follower]" leader-address= leader-id=
2026-03-29T01:09:07.551Z [WARN]  server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:07.551Z [INFO]  server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: entering candidate state: node="Node at e8b5889a-0e4d-5f76-99db-6f01fe6df05b [Candidate]" term=2
2026-03-29T01:09:07.551Z [INFO]  server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: election won: term=2 tally=2
2026-03-29T01:09:07.551Z [INFO]  server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: entering leader state: leader="Node at e8b5889a-0e4d-5f76-99db-6f01fe6df05b [Leader]"
2026-03-29T01:09:07.551Z [INFO]  server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: added peer, starting replication: peer=server-b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6
2026-03-29T01:09:07.551Z [INFO]  server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: added peer, starting replication: peer=server-99482ef4-51bb-ed9f-4283-e08dc7f221c5
2026-03-29T01:09:07.551Z [INFO]  server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: pipelining replication: peer="{Voter server-99482ef4-51bb-ed9f-4283-e08dc7f221c5 99482ef4-51bb-ed9f-4283-e08dc7f221c5}"
2026-03-29T01:09:07.551Z [INFO]  server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: pipelining replication: peer="{Voter server-b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6 b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6}"
2026-03-29T01:09:08.557Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:07.551780308 +0000 UTC m=+91.339922849 (1 nodes), 567.309421ms from start of poll, 567.544017ms from cluster start. Timeout at 2026-03-29 01:09:08.556784404 +0000 UTC m=+92.344926934, 1.005004085s after stability
2026-03-29T01:09:08.558Z [INFO]  creating new snapshot: path=/tmp/raft3400096175/snapshots/2-10013-1774746548558.tmp
2026-03-29T01:09:08.558Z [INFO]  server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: copied to local snapshot: bytes=71
2026-03-29T01:09:08.559Z [INFO]  server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: snapshot restore progress: id=2-10013-1774746548558 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:08.559Z [INFO]  server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: restored user snapshot: index=10013
2026-03-29T01:09:08.559Z [ERROR] server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: failed to get log: index=3 error="log not found"
2026-03-29T01:09:08.559Z [ERROR] server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: failed to get log: index=3 error="log not found"
2026-03-29T01:09:08.559Z [INFO]  server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: aborting pipeline replication: peer="{Voter server-b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6 b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6}"
2026-03-29T01:09:08.559Z [ERROR] server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: failed to get log: index=3 error="log not found"
2026-03-29T01:09:08.559Z [INFO]  server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: aborting pipeline replication: peer="{Voter server-99482ef4-51bb-ed9f-4283-e08dc7f221c5 99482ef4-51bb-ed9f-4283-e08dc7f221c5}"
2026-03-29T01:09:08.559Z [INFO]  creating new snapshot: path=/tmp/raft1667584027/snapshots/2-10013-1774746548559.tmp
2026-03-29T01:09:08.559Z [INFO]  server-b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:08.560Z [INFO]  server-b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6: copied to local snapshot: bytes=71
2026-03-29T01:09:08.560Z [INFO]  server-b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6: snapshot restore progress: id=2-10013-1774746548559 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:08.560Z [INFO]  server-b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6: Installed remote snapshot
2026-03-29T01:09:08.560Z [INFO]  server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: pipelining replication: peer="{Voter server-b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6 b3eb9cf1-dd06-e6b9-ad66-c13bf19958d6}"
2026-03-29T01:09:08.560Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:08.565Z [ERROR] server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: failed to get log: index=3 error="log not found"
2026-03-29T01:09:08.566Z [INFO]  creating new snapshot: path=/tmp/raft3667011780/snapshots/2-10013-1774746548566.tmp
2026-03-29T01:09:08.566Z [INFO]  server-99482ef4-51bb-ed9f-4283-e08dc7f221c5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:08.566Z [INFO]  server-99482ef4-51bb-ed9f-4283-e08dc7f221c5: copied to local snapshot: bytes=71
2026-03-29T01:09:08.566Z [INFO]  server-99482ef4-51bb-ed9f-4283-e08dc7f221c5: snapshot restore progress: id=2-10013-1774746548566 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:08.566Z [INFO]  server-99482ef4-51bb-ed9f-4283-e08dc7f221c5: Installed remote snapshot
2026-03-29T01:09:08.567Z [INFO]  server-e8b5889a-0e4d-5f76-99db-6f01fe6df05b: pipelining replication: peer="{Voter server-99482ef4-51bb-ed9f-4283-e08dc7f221c5 99482ef4-51bb-ed9f-4283-e08dc7f221c5}"
2026-03-29T01:09:09.565Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:08.560658957 +0000 UTC m=+92.348801518 (2 nodes), 1.003µs from start of poll, 1.576422686s from cluster start. Timeout at 2026-03-29 01:09:09.565672638 +0000 UTC m=+93.353815214, 1.005013696s after stability
2026-03-29T01:09:09.578Z [INFO]  server-3dd430e9-302d-e80b-b190-0be04ecd04ed: aborting pipeline replication: peer="{Voter server-7e692399-117f-aff9-1274-3c32154969d0 7e692399-117f-aff9-1274-3c32154969d0}"
2026-03-29T01:09:09.578Z [INFO]  server-3dd430e9-302d-e80b-b190-0be04ecd04ed: aborting pipeline replication: peer="{Voter server-9e0a13c2-66d4-029c-5b20-37b28c1e3611 9e0a13c2-66d4-029c-5b20-37b28c1e3611}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true
2026-03-29T01:09:09.582Z [INFO]  server-a608c60e-5957-c0f5-512d-4fa0859bd7ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a608c60e-5957-c0f5-512d-4fa0859bd7ee Address:a608c60e-5957-c0f5-512d-4fa0859bd7ee} {Suffrage:Voter ID:server-e176ae85-f9ed-9658-dfc5-3b906f944546 Address:e176ae85-f9ed-9658-dfc5-3b906f944546} {Suffrage:Voter ID:server-87de8010-6853-e990-1d95-ef2d5765f172 Address:87de8010-6853-e990-1d95-ef2d5765f172}]"
2026-03-29T01:09:09.582Z [INFO]  server-a608c60e-5957-c0f5-512d-4fa0859bd7ee: entering follower state: follower="Node at a608c60e-5957-c0f5-512d-4fa0859bd7ee [Follower]" leader-address= leader-id=
2026-03-29T01:09:09.582Z [INFO]  server-e176ae85-f9ed-9658-dfc5-3b906f944546: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a608c60e-5957-c0f5-512d-4fa0859bd7ee Address:a608c60e-5957-c0f5-512d-4fa0859bd7ee} {Suffrage:Voter ID:server-e176ae85-f9ed-9658-dfc5-3b906f944546 Address:e176ae85-f9ed-9658-dfc5-3b906f944546} {Suffrage:Voter ID:server-87de8010-6853-e990-1d95-ef2d5765f172 Address:87de8010-6853-e990-1d95-ef2d5765f172}]"
2026-03-29T01:09:09.582Z [INFO]  server-e176ae85-f9ed-9658-dfc5-3b906f944546: entering follower state: follower="Node at e176ae85-f9ed-9658-dfc5-3b906f944546 [Follower]" leader-address= leader-id=
2026-03-29T01:09:09.583Z [INFO]  server-87de8010-6853-e990-1d95-ef2d5765f172: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a608c60e-5957-c0f5-512d-4fa0859bd7ee Address:a608c60e-5957-c0f5-512d-4fa0859bd7ee} {Suffrage:Voter ID:server-e176ae85-f9ed-9658-dfc5-3b906f944546 Address:e176ae85-f9ed-9658-dfc5-3b906f944546} {Suffrage:Voter ID:server-87de8010-6853-e990-1d95-ef2d5765f172 Address:87de8010-6853-e990-1d95-ef2d5765f172}]"
2026-03-29T01:09:09.583Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:09.583Z [INFO]  server-87de8010-6853-e990-1d95-ef2d5765f172: entering follower state: follower="Node at 87de8010-6853-e990-1d95-ef2d5765f172 [Follower]" leader-address= leader-id=
2026-03-29T01:09:10.095Z [WARN]  server-87de8010-6853-e990-1d95-ef2d5765f172: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:10.095Z [INFO]  server-87de8010-6853-e990-1d95-ef2d5765f172: entering candidate state: node="Node at 87de8010-6853-e990-1d95-ef2d5765f172 [Candidate]" term=2
2026-03-29T01:09:10.095Z [INFO]  server-87de8010-6853-e990-1d95-ef2d5765f172: election won: term=2 tally=2
2026-03-29T01:09:10.095Z [INFO]  server-87de8010-6853-e990-1d95-ef2d5765f172: entering leader state: leader="Node at 87de8010-6853-e990-1d95-ef2d5765f172 [Leader]"
2026-03-29T01:09:10.095Z [INFO]  server-87de8010-6853-e990-1d95-ef2d5765f172: added peer, starting replication: peer=server-a608c60e-5957-c0f5-512d-4fa0859bd7ee
2026-03-29T01:09:10.095Z [INFO]  server-87de8010-6853-e990-1d95-ef2d5765f172: added peer, starting replication: peer=server-e176ae85-f9ed-9658-dfc5-3b906f944546
2026-03-29T01:09:10.095Z [INFO]  server-87de8010-6853-e990-1d95-ef2d5765f172: pipelining replication: peer="{Voter server-a608c60e-5957-c0f5-512d-4fa0859bd7ee a608c60e-5957-c0f5-512d-4fa0859bd7ee}"
2026-03-29T01:09:10.095Z [INFO]  server-87de8010-6853-e990-1d95-ef2d5765f172: pipelining replication: peer="{Voter server-e176ae85-f9ed-9658-dfc5-3b906f944546 e176ae85-f9ed-9658-dfc5-3b906f944546}"
2026-03-29T01:09:11.101Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:10.095445066 +0000 UTC m=+93.883587723 (1 nodes), 512.391129ms from start of poll, 512.665556ms from cluster start. Timeout at 2026-03-29 01:09:11.100450806 +0000 UTC m=+94.888593336, 1.005005613s after stability
2026-03-29T01:09:11.101Z [INFO]  server-87de8010-6853-e990-1d95-ef2d5765f172: starting snapshot up to: index=12
2026-03-29T01:09:11.101Z [INFO]  creating new snapshot: path=/tmp/raft3951636448/snapshots/2-12-1774746551101.tmp
2026-03-29T01:09:11.102Z [INFO]  server-87de8010-6853-e990-1d95-ef2d5765f172: snapshot complete up to: index=12
2026-03-29T01:09:11.103Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a26b025-611b-e479-85c3-78dc038f0f5d Address:3a26b025-611b-e479-85c3-78dc038f0f5d} {Suffrage:Voter ID:server-58e538f5-8201-0aac-7395-a7870a04ee2a Address:58e538f5-8201-0aac-7395-a7870a04ee2a} {Suffrage:Voter ID:server-8ed38d6f-c205-74af-89ba-15b9dc2dec1a Address:8ed38d6f-c205-74af-89ba-15b9dc2dec1a}]"
2026-03-29T01:09:11.103Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: entering follower state: follower="Node at 3a26b025-611b-e479-85c3-78dc038f0f5d [Follower]" leader-address= leader-id=
2026-03-29T01:09:11.104Z [INFO]  server-58e538f5-8201-0aac-7395-a7870a04ee2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a26b025-611b-e479-85c3-78dc038f0f5d Address:3a26b025-611b-e479-85c3-78dc038f0f5d} {Suffrage:Voter ID:server-58e538f5-8201-0aac-7395-a7870a04ee2a Address:58e538f5-8201-0aac-7395-a7870a04ee2a} {Suffrage:Voter ID:server-8ed38d6f-c205-74af-89ba-15b9dc2dec1a Address:8ed38d6f-c205-74af-89ba-15b9dc2dec1a}]"
2026-03-29T01:09:11.104Z [INFO]  server-58e538f5-8201-0aac-7395-a7870a04ee2a: entering follower state: follower="Node at 58e538f5-8201-0aac-7395-a7870a04ee2a [Follower]" leader-address= leader-id=
2026-03-29T01:09:11.104Z [INFO]  server-8ed38d6f-c205-74af-89ba-15b9dc2dec1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a26b025-611b-e479-85c3-78dc038f0f5d Address:3a26b025-611b-e479-85c3-78dc038f0f5d} {Suffrage:Voter ID:server-58e538f5-8201-0aac-7395-a7870a04ee2a Address:58e538f5-8201-0aac-7395-a7870a04ee2a} {Suffrage:Voter ID:server-8ed38d6f-c205-74af-89ba-15b9dc2dec1a Address:8ed38d6f-c205-74af-89ba-15b9dc2dec1a}]"
2026-03-29T01:09:11.104Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:11.104Z [INFO]  server-8ed38d6f-c205-74af-89ba-15b9dc2dec1a: entering follower state: follower="Node at 8ed38d6f-c205-74af-89ba-15b9dc2dec1a [Follower]" leader-address= leader-id=
2026-03-29T01:09:11.676Z [WARN]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:11.676Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: entering candidate state: node="Node at 3a26b025-611b-e479-85c3-78dc038f0f5d [Candidate]" term=2
2026-03-29T01:09:11.676Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: election won: term=2 tally=2
2026-03-29T01:09:11.676Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: entering leader state: leader="Node at 3a26b025-611b-e479-85c3-78dc038f0f5d [Leader]"
2026-03-29T01:09:11.676Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: added peer, starting replication: peer=server-58e538f5-8201-0aac-7395-a7870a04ee2a
2026-03-29T01:09:11.676Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: added peer, starting replication: peer=server-8ed38d6f-c205-74af-89ba-15b9dc2dec1a
2026-03-29T01:09:11.676Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: pipelining replication: peer="{Voter server-8ed38d6f-c205-74af-89ba-15b9dc2dec1a 8ed38d6f-c205-74af-89ba-15b9dc2dec1a}"
2026-03-29T01:09:11.676Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: pipelining replication: peer="{Voter server-58e538f5-8201-0aac-7395-a7870a04ee2a 58e538f5-8201-0aac-7395-a7870a04ee2a}"
2026-03-29T01:09:12.681Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:11.676670216 +0000 UTC m=+95.464812751 (1 nodes), 572.083532ms from start of poll, 573.100409ms from cluster start. Timeout at 2026-03-29 01:09:12.68167958 +0000 UTC m=+96.469822120, 1.005009369s after stability
2026-03-29T01:09:12.681Z [INFO]  creating new snapshot: path=/tmp/raft2059941470/snapshots/2-10013-1774746552681.tmp
2026-03-29T01:09:12.682Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: copied to local snapshot: bytes=71
2026-03-29T01:09:12.682Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: snapshot restore progress: id=2-10013-1774746552681 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:12.682Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: removing all old logs from log store
2026-03-29T01:09:12.682Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: compacting logs: from=1 to=2
2026-03-29T01:09:12.682Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: restored user snapshot: index=10013
2026-03-29T01:09:12.682Z [ERROR] server-3a26b025-611b-e479-85c3-78dc038f0f5d: failed to get log: index=2 error="log not found"
2026-03-29T01:09:12.682Z [ERROR] server-3a26b025-611b-e479-85c3-78dc038f0f5d: failed to get log: index=2 error="log not found"
2026-03-29T01:09:12.682Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: aborting pipeline replication: peer="{Voter server-8ed38d6f-c205-74af-89ba-15b9dc2dec1a 8ed38d6f-c205-74af-89ba-15b9dc2dec1a}"
2026-03-29T01:09:12.682Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: aborting pipeline replication: peer="{Voter server-58e538f5-8201-0aac-7395-a7870a04ee2a 58e538f5-8201-0aac-7395-a7870a04ee2a}"
2026-03-29T01:09:12.689Z [ERROR] server-3a26b025-611b-e479-85c3-78dc038f0f5d: failed to get log: index=2 error="log not found"
2026-03-29T01:09:12.689Z [INFO]  creating new snapshot: path=/tmp/raft930894596/snapshots/2-10013-1774746552689.tmp
2026-03-29T01:09:12.689Z [INFO]  server-58e538f5-8201-0aac-7395-a7870a04ee2a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:12.690Z [INFO]  server-58e538f5-8201-0aac-7395-a7870a04ee2a: copied to local snapshot: bytes=71
2026-03-29T01:09:12.690Z [ERROR] server-3a26b025-611b-e479-85c3-78dc038f0f5d: failed to get log: index=2 error="log not found"
2026-03-29T01:09:12.690Z [INFO]  server-58e538f5-8201-0aac-7395-a7870a04ee2a: snapshot restore progress: id=2-10013-1774746552689 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:12.690Z [INFO]  server-58e538f5-8201-0aac-7395-a7870a04ee2a: removing all old logs from log store
2026-03-29T01:09:12.690Z [INFO]  server-58e538f5-8201-0aac-7395-a7870a04ee2a: compacting logs: from=1 to=2
2026-03-29T01:09:12.690Z [INFO]  server-58e538f5-8201-0aac-7395-a7870a04ee2a: Installed remote snapshot
2026-03-29T01:09:12.690Z [INFO]  creating new snapshot: path=/tmp/raft1038922223/snapshots/2-10013-1774746552690.tmp
2026-03-29T01:09:12.690Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: pipelining replication: peer="{Voter server-58e538f5-8201-0aac-7395-a7870a04ee2a 58e538f5-8201-0aac-7395-a7870a04ee2a}"
2026-03-29T01:09:12.690Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:12.690Z [INFO]  server-8ed38d6f-c205-74af-89ba-15b9dc2dec1a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:12.690Z [INFO]  server-8ed38d6f-c205-74af-89ba-15b9dc2dec1a: copied to local snapshot: bytes=71
2026-03-29T01:09:12.691Z [INFO]  server-8ed38d6f-c205-74af-89ba-15b9dc2dec1a: snapshot restore progress: id=2-10013-1774746552690 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2026-03-29T01:09:12.691Z [INFO]  server-8ed38d6f-c205-74af-89ba-15b9dc2dec1a: removing all old logs from log store
2026-03-29T01:09:12.691Z [INFO]  server-8ed38d6f-c205-74af-89ba-15b9dc2dec1a: compacting logs: from=1 to=2
2026-03-29T01:09:12.691Z [INFO]  server-8ed38d6f-c205-74af-89ba-15b9dc2dec1a: Installed remote snapshot
2026-03-29T01:09:12.691Z [INFO]  server-3a26b025-611b-e479-85c3-78dc038f0f5d: pipelining replication: peer="{Voter server-8ed38d6f-c205-74af-89ba-15b9dc2dec1a 8ed38d6f-c205-74af-89ba-15b9dc2dec1a}"
2026-03-29T01:09:13.696Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:12.690501789 +0000 UTC m=+96.478644322 (2 nodes), 677ns from start of poll, 1.58693198s from cluster start. Timeout at 2026-03-29 01:09:13.695515984 +0000 UTC m=+97.483658678, 1.005014356s after stability
2026-03-29T01:09:13.707Z [INFO]  server-87de8010-6853-e990-1d95-ef2d5765f172: aborting pipeline replication: peer="{Voter server-a608c60e-5957-c0f5-512d-4fa0859bd7ee a608c60e-5957-c0f5-512d-4fa0859bd7ee}"
2026-03-29T01:09:13.707Z [INFO]  server-87de8010-6853-e990-1d95-ef2d5765f172: aborting pipeline replication: peer="{Voter server-e176ae85-f9ed-9658-dfc5-3b906f944546 e176ae85-f9ed-9658-dfc5-3b906f944546}"
--- PASS: TestRaft_UserRestore (84.02s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.72s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.68s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.79s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.22s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.54s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.66s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.50s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.23s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.57s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.33s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.57s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (3.04s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.22s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.36s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.54s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.60s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.27s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.32s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.55s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.64s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.32s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.13s)
=== RUN   TestRaft_SendSnapshotFollower
2026-03-29T01:09:13.708Z [INFO]  server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28 Address:5fb6cee3-da5c-d6d3-365d-c2ec40c06b28} {Suffrage:Voter ID:server-e09ec35d-cd27-3708-f29f-6c06a49d483a Address:e09ec35d-cd27-3708-f29f-6c06a49d483a} {Suffrage:Voter ID:server-7975a893-9477-6507-ba30-e20e5c5e3bd6 Address:7975a893-9477-6507-ba30-e20e5c5e3bd6}]"
2026-03-29T01:09:13.709Z [INFO]  server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: entering follower state: follower="Node at 5fb6cee3-da5c-d6d3-365d-c2ec40c06b28 [Follower]" leader-address= leader-id=
2026-03-29T01:09:13.709Z [INFO]  server-e09ec35d-cd27-3708-f29f-6c06a49d483a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28 Address:5fb6cee3-da5c-d6d3-365d-c2ec40c06b28} {Suffrage:Voter ID:server-e09ec35d-cd27-3708-f29f-6c06a49d483a Address:e09ec35d-cd27-3708-f29f-6c06a49d483a} {Suffrage:Voter ID:server-7975a893-9477-6507-ba30-e20e5c5e3bd6 Address:7975a893-9477-6507-ba30-e20e5c5e3bd6}]"
2026-03-29T01:09:13.709Z [INFO]  server-e09ec35d-cd27-3708-f29f-6c06a49d483a: entering follower state: follower="Node at e09ec35d-cd27-3708-f29f-6c06a49d483a [Follower]" leader-address= leader-id=
2026-03-29T01:09:13.709Z [INFO]  server-7975a893-9477-6507-ba30-e20e5c5e3bd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28 Address:5fb6cee3-da5c-d6d3-365d-c2ec40c06b28} {Suffrage:Voter ID:server-e09ec35d-cd27-3708-f29f-6c06a49d483a Address:e09ec35d-cd27-3708-f29f-6c06a49d483a} {Suffrage:Voter ID:server-7975a893-9477-6507-ba30-e20e5c5e3bd6 Address:7975a893-9477-6507-ba30-e20e5c5e3bd6}]"
2026-03-29T01:09:13.709Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:13.709Z [INFO]  server-7975a893-9477-6507-ba30-e20e5c5e3bd6: entering follower state: follower="Node at 7975a893-9477-6507-ba30-e20e5c5e3bd6 [Follower]" leader-address= leader-id=
2026-03-29T01:09:13.769Z [WARN]  server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:13.769Z [INFO]  server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: entering candidate state: node="Node at 5fb6cee3-da5c-d6d3-365d-c2ec40c06b28 [Candidate]" term=2
2026-03-29T01:09:13.769Z [INFO]  server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: election won: term=2 tally=2
2026-03-29T01:09:13.769Z [INFO]  server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: entering leader state: leader="Node at 5fb6cee3-da5c-d6d3-365d-c2ec40c06b28 [Leader]"
2026-03-29T01:09:13.769Z [INFO]  server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: added peer, starting replication: peer=server-e09ec35d-cd27-3708-f29f-6c06a49d483a
2026-03-29T01:09:13.769Z [INFO]  server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: added peer, starting replication: peer=server-7975a893-9477-6507-ba30-e20e5c5e3bd6
2026-03-29T01:09:13.769Z [INFO]  server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: pipelining replication: peer="{Voter server-7975a893-9477-6507-ba30-e20e5c5e3bd6 7975a893-9477-6507-ba30-e20e5c5e3bd6}"
2026-03-29T01:09:13.769Z [INFO]  server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: pipelining replication: peer="{Voter server-e09ec35d-cd27-3708-f29f-6c06a49d483a e09ec35d-cd27-3708-f29f-6c06a49d483a}"
2026-03-29T01:09:13.875Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:13.769167694 +0000 UTC m=+97.557310227 (2 nodes), 60.034434ms from start of poll, 60.292105ms from cluster start. Timeout at 2026-03-29 01:09:13.874179744 +0000 UTC m=+97.662322324, 105.012097ms after stability
2026-03-29T01:09:13.875Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:13.980Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:13.875149347 +0000 UTC m=+97.663291971 (1 nodes), 1.841µs from start of poll, 166.273849ms from cluster start. Timeout at 2026-03-29 01:09:13.980255239 +0000 UTC m=+97.768397866, 105.105895ms after stability
2026-03-29T01:09:13.980Z [ERROR] server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: failed to pipeline appendEntries: peer="{Voter server-e09ec35d-cd27-3708-f29f-6c06a49d483a e09ec35d-cd27-3708-f29f-6c06a49d483a}" error="append pipeline closed"
2026-03-29T01:09:13.980Z [INFO]  server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: aborting pipeline replication: peer="{Voter server-e09ec35d-cd27-3708-f29f-6c06a49d483a e09ec35d-cd27-3708-f29f-6c06a49d483a}"
2026-03-29T01:09:13.981Z [ERROR] server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: failed to appendEntries to: peer="{Voter server-e09ec35d-cd27-3708-f29f-6c06a49d483a e09ec35d-cd27-3708-f29f-6c06a49d483a}" error="failed to connect to peer: e09ec35d-cd27-3708-f29f-6c06a49d483a"
2026-03-29T01:09:13.983Z [ERROR] server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: failed to heartbeat to: peer=e09ec35d-cd27-3708-f29f-6c06a49d483a backoff time=10ms error="failed to connect to peer: e09ec35d-cd27-3708-f29f-6c06a49d483a"
    raft_test.go:1627: [INFO] Finished apply without behind follower
2026-03-29T01:09:13.985Z [INFO]  server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: starting snapshot up to: index=102
2026-03-29T01:09:13.985Z [INFO]  creating new snapshot: path=/tmp/raft3398789342/snapshots/2-102-1774746553985.tmp
2026-03-29T01:09:13.986Z [INFO]  server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: compacting logs: from=1 to=92
2026-03-29T01:09:13.986Z [INFO]  server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: snapshot complete up to: index=102
2026-03-29T01:09:13.986Z [ERROR] server-e09ec35d-cd27-3708-f29f-6c06a49d483a: failed to take snapshot: error="nothing new to snapshot"
2026-03-29T01:09:13.986Z [INFO]  server-7975a893-9477-6507-ba30-e20e5c5e3bd6: starting snapshot up to: index=97
2026-03-29T01:09:13.986Z [INFO]  creating new snapshot: path=/tmp/raft2331626096/snapshots/2-97-1774746553986.tmp
2026-03-29T01:09:13.986Z [INFO]  server-7975a893-9477-6507-ba30-e20e5c5e3bd6: compacting logs: from=1 to=92
2026-03-29T01:09:13.986Z [INFO]  server-7975a893-9477-6507-ba30-e20e5c5e3bd6: snapshot complete up to: index=97
2026-03-29T01:09:13.991Z [ERROR] server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: failed to get log: index=2 error="log not found"
2026-03-29T01:09:13.991Z [INFO]  creating new snapshot: path=/tmp/raft80322294/snapshots/2-102-1774746553991.tmp
2026-03-29T01:09:13.991Z [INFO]  server-e09ec35d-cd27-3708-f29f-6c06a49d483a: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2026-03-29T01:09:13.992Z [INFO]  server-e09ec35d-cd27-3708-f29f-6c06a49d483a: copied to local snapshot: bytes=693
2026-03-29T01:09:13.992Z [INFO]  server-e09ec35d-cd27-3708-f29f-6c06a49d483a: snapshot restore progress: id=2-102-1774746553991 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-03-29T01:09:13.992Z [INFO]  server-e09ec35d-cd27-3708-f29f-6c06a49d483a: Installed remote snapshot
2026-03-29T01:09:13.992Z [WARN]  server-e09ec35d-cd27-3708-f29f-6c06a49d483a: skipping application of old log: index=102
2026-03-29T01:09:13.992Z [INFO]  server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: pipelining replication: peer="{Voter server-e09ec35d-cd27-3708-f29f-6c06a49d483a e09ec35d-cd27-3708-f29f-6c06a49d483a}"
2026-03-29T01:09:13.997Z [INFO]  server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: aborting pipeline replication: peer="{Voter server-7975a893-9477-6507-ba30-e20e5c5e3bd6 7975a893-9477-6507-ba30-e20e5c5e3bd6}"
2026-03-29T01:09:13.997Z [INFO]  server-5fb6cee3-da5c-d6d3-365d-c2ec40c06b28: aborting pipeline replication: peer="{Voter server-e09ec35d-cd27-3708-f29f-6c06a49d483a e09ec35d-cd27-3708-f29f-6c06a49d483a}"
--- PASS: TestRaft_SendSnapshotFollower (0.29s)
=== RUN   TestRaft_SendSnapshotAndLogsFollower
2026-03-29T01:09:13.998Z [INFO]  server-c9d5ce8a-3424-4858-a947-bf97930c4a68: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9d5ce8a-3424-4858-a947-bf97930c4a68 Address:c9d5ce8a-3424-4858-a947-bf97930c4a68} {Suffrage:Voter ID:server-e96cf64e-6458-554c-9eaf-e65cdaf05ca9 Address:e96cf64e-6458-554c-9eaf-e65cdaf05ca9} {Suffrage:Voter ID:server-15a20401-d136-f50b-ccbf-7ea73ef2132c Address:15a20401-d136-f50b-ccbf-7ea73ef2132c}]"
2026-03-29T01:09:13.998Z [INFO]  server-c9d5ce8a-3424-4858-a947-bf97930c4a68: entering follower state: follower="Node at c9d5ce8a-3424-4858-a947-bf97930c4a68 [Follower]" leader-address= leader-id=
2026-03-29T01:09:13.998Z [INFO]  server-e96cf64e-6458-554c-9eaf-e65cdaf05ca9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9d5ce8a-3424-4858-a947-bf97930c4a68 Address:c9d5ce8a-3424-4858-a947-bf97930c4a68} {Suffrage:Voter ID:server-e96cf64e-6458-554c-9eaf-e65cdaf05ca9 Address:e96cf64e-6458-554c-9eaf-e65cdaf05ca9} {Suffrage:Voter ID:server-15a20401-d136-f50b-ccbf-7ea73ef2132c Address:15a20401-d136-f50b-ccbf-7ea73ef2132c}]"
2026-03-29T01:09:13.998Z [INFO]  server-e96cf64e-6458-554c-9eaf-e65cdaf05ca9: entering follower state: follower="Node at e96cf64e-6458-554c-9eaf-e65cdaf05ca9 [Follower]" leader-address= leader-id=
2026-03-29T01:09:13.998Z [INFO]  server-15a20401-d136-f50b-ccbf-7ea73ef2132c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9d5ce8a-3424-4858-a947-bf97930c4a68 Address:c9d5ce8a-3424-4858-a947-bf97930c4a68} {Suffrage:Voter ID:server-e96cf64e-6458-554c-9eaf-e65cdaf05ca9 Address:e96cf64e-6458-554c-9eaf-e65cdaf05ca9} {Suffrage:Voter ID:server-15a20401-d136-f50b-ccbf-7ea73ef2132c Address:15a20401-d136-f50b-ccbf-7ea73ef2132c}]"
2026-03-29T01:09:13.998Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:13.998Z [INFO]  server-15a20401-d136-f50b-ccbf-7ea73ef2132c: entering follower state: follower="Node at 15a20401-d136-f50b-ccbf-7ea73ef2132c [Follower]" leader-address= leader-id=
2026-03-29T01:09:14.074Z [WARN]  server-15a20401-d136-f50b-ccbf-7ea73ef2132c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:14.074Z [INFO]  server-15a20401-d136-f50b-ccbf-7ea73ef2132c: entering candidate state: node="Node at 15a20401-d136-f50b-ccbf-7ea73ef2132c [Candidate]" term=2
2026-03-29T01:09:14.074Z [INFO]  server-15a20401-d136-f50b-ccbf-7ea73ef2132c: election won: term=2 tally=2
2026-03-29T01:09:14.074Z [INFO]  server-15a20401-d136-f50b-ccbf-7ea73ef2132c: entering leader state: leader="Node at 15a20401-d136-f50b-ccbf-7ea73ef2132c [Leader]"
2026-03-29T01:09:14.074Z [INFO]  server-15a20401-d136-f50b-ccbf-7ea73ef2132c: added peer, starting replication: peer=server-c9d5ce8a-3424-4858-a947-bf97930c4a68
2026-03-29T01:09:14.074Z [INFO]  server-15a20401-d136-f50b-ccbf-7ea73ef2132c: added peer, starting replication: peer=server-e96cf64e-6458-554c-9eaf-e65cdaf05ca9
2026-03-29T01:09:14.074Z [INFO]  server-15a20401-d136-f50b-ccbf-7ea73ef2132c: pipelining replication: peer="{Voter server-e96cf64e-6458-554c-9eaf-e65cdaf05ca9 e96cf64e-6458-554c-9eaf-e65cdaf05ca9}"
2026-03-29T01:09:14.074Z [INFO]  server-15a20401-d136-f50b-ccbf-7ea73ef2132c: pipelining replication: peer="{Voter server-c9d5ce8a-3424-4858-a947-bf97930c4a68 c9d5ce8a-3424-4858-a947-bf97930c4a68}"
2026-03-29T01:09:14.179Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:14.07467903 +0000 UTC m=+97.862821595 (2 nodes), 76.080302ms from start of poll, 76.371009ms from cluster start. Timeout at 2026-03-29 01:09:14.179682936 +0000 UTC m=+97.967825526, 105.003931ms after stability
2026-03-29T01:09:14.179Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:14.285Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:14.179921895 +0000 UTC m=+97.968064492 (1 nodes), 1.102µs from start of poll, 181.613906ms from cluster start. Timeout at 2026-03-29 01:09:14.284927239 +0000 UTC m=+98.073069810, 105.005318ms after stability
2026-03-29T01:09:14.286Z [ERROR] server-15a20401-d136-f50b-ccbf-7ea73ef2132c: failed to pipeline appendEntries: peer="{Voter server-c9d5ce8a-3424-4858-a947-bf97930c4a68 c9d5ce8a-3424-4858-a947-bf97930c4a68}" error="append pipeline closed"
2026-03-29T01:09:14.286Z [INFO]  server-15a20401-d136-f50b-ccbf-7ea73ef2132c: aborting pipeline replication: peer="{Voter server-c9d5ce8a-3424-4858-a947-bf97930c4a68 c9d5ce8a-3424-4858-a947-bf97930c4a68}"
2026-03-29T01:09:14.286Z [ERROR] server-15a20401-d136-f50b-ccbf-7ea73ef2132c: failed to appendEntries to: peer="{Voter server-c9d5ce8a-3424-4858-a947-bf97930c4a68 c9d5ce8a-3424-4858-a947-bf97930c4a68}" error="failed to connect to peer: c9d5ce8a-3424-4858-a947-bf97930c4a68"
    raft_test.go:1669: [INFO] Finished apply without behind follower
2026-03-29T01:09:14.289Z [ERROR] server-c9d5ce8a-3424-4858-a947-bf97930c4a68: failed to take snapshot: error="nothing new to snapshot"
2026-03-29T01:09:14.289Z [INFO]  server-e96cf64e-6458-554c-9eaf-e65cdaf05ca9: starting snapshot up to: index=100
2026-03-29T01:09:14.289Z [INFO]  creating new snapshot: path=/tmp/raft174928108/snapshots/2-100-1774746554289.tmp
2026-03-29T01:09:14.290Z [INFO]  server-e96cf64e-6458-554c-9eaf-e65cdaf05ca9: compacting logs: from=1 to=92
2026-03-29T01:09:14.290Z [INFO]  server-e96cf64e-6458-554c-9eaf-e65cdaf05ca9: snapshot complete up to: index=100
2026-03-29T01:09:14.290Z [INFO]  server-15a20401-d136-f50b-ccbf-7ea73ef2132c: starting snapshot up to: index=102
2026-03-29T01:09:14.290Z [INFO]  creating new snapshot: path=/tmp/raft3604440434/snapshots/2-102-1774746554290.tmp
2026-03-29T01:09:14.291Z [INFO]  server-15a20401-d136-f50b-ccbf-7ea73ef2132c: compacting logs: from=1 to=92
2026-03-29T01:09:14.291Z [INFO]  server-15a20401-d136-f50b-ccbf-7ea73ef2132c: snapshot complete up to: index=102
2026-03-29T01:09:14.292Z [ERROR] server-15a20401-d136-f50b-ccbf-7ea73ef2132c: failed to heartbeat to: peer=c9d5ce8a-3424-4858-a947-bf97930c4a68 backoff time=10ms error="failed to connect to peer: c9d5ce8a-3424-4858-a947-bf97930c4a68"
    raft_test.go:1690: [INFO] Finished apply without behind follower
2026-03-29T01:09:14.296Z [ERROR] server-15a20401-d136-f50b-ccbf-7ea73ef2132c: failed to get log: index=2 error="log not found"
2026-03-29T01:09:14.297Z [INFO]  creating new snapshot: path=/tmp/raft3281535943/snapshots/2-102-1774746554297.tmp
2026-03-29T01:09:14.297Z [INFO]  server-c9d5ce8a-3424-4858-a947-bf97930c4a68: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2026-03-29T01:09:14.297Z [INFO]  server-c9d5ce8a-3424-4858-a947-bf97930c4a68: copied to local snapshot: bytes=693
2026-03-29T01:09:14.297Z [INFO]  server-c9d5ce8a-3424-4858-a947-bf97930c4a68: snapshot restore progress: id=2-102-1774746554297 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2026-03-29T01:09:14.297Z [INFO]  server-c9d5ce8a-3424-4858-a947-bf97930c4a68: Installed remote snapshot
2026-03-29T01:09:14.298Z [INFO]  server-15a20401-d136-f50b-ccbf-7ea73ef2132c: pipelining replication: peer="{Voter server-c9d5ce8a-3424-4858-a947-bf97930c4a68 c9d5ce8a-3424-4858-a947-bf97930c4a68}"
2026-03-29T01:09:14.305Z [INFO]  server-15a20401-d136-f50b-ccbf-7ea73ef2132c: aborting pipeline replication: peer="{Voter server-e96cf64e-6458-554c-9eaf-e65cdaf05ca9 e96cf64e-6458-554c-9eaf-e65cdaf05ca9}"
2026-03-29T01:09:14.305Z [INFO]  server-15a20401-d136-f50b-ccbf-7ea73ef2132c: aborting pipeline replication: peer="{Voter server-c9d5ce8a-3424-4858-a947-bf97930c4a68 c9d5ce8a-3424-4858-a947-bf97930c4a68}"
--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.31s)
=== RUN   TestRaft_ReJoinFollower
2026-03-29T01:09:14.306Z [INFO]  server-05594d09-75af-cc96-758b-7bacf89fa498: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05594d09-75af-cc96-758b-7bacf89fa498 Address:05594d09-75af-cc96-758b-7bacf89fa498} {Suffrage:Voter ID:server-e9bdb21c-38fb-60dc-16b3-8782d2557f23 Address:e9bdb21c-38fb-60dc-16b3-8782d2557f23} {Suffrage:Voter ID:server-bf3b3edc-1189-3a9b-aa7a-31b382243f03 Address:bf3b3edc-1189-3a9b-aa7a-31b382243f03}]"
2026-03-29T01:09:14.306Z [INFO]  server-05594d09-75af-cc96-758b-7bacf89fa498: entering follower state: follower="Node at 05594d09-75af-cc96-758b-7bacf89fa498 [Follower]" leader-address= leader-id=
2026-03-29T01:09:14.306Z [INFO]  server-e9bdb21c-38fb-60dc-16b3-8782d2557f23: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05594d09-75af-cc96-758b-7bacf89fa498 Address:05594d09-75af-cc96-758b-7bacf89fa498} {Suffrage:Voter ID:server-e9bdb21c-38fb-60dc-16b3-8782d2557f23 Address:e9bdb21c-38fb-60dc-16b3-8782d2557f23} {Suffrage:Voter ID:server-bf3b3edc-1189-3a9b-aa7a-31b382243f03 Address:bf3b3edc-1189-3a9b-aa7a-31b382243f03}]"
2026-03-29T01:09:14.306Z [INFO]  server-e9bdb21c-38fb-60dc-16b3-8782d2557f23: entering follower state: follower="Node at e9bdb21c-38fb-60dc-16b3-8782d2557f23 [Follower]" leader-address= leader-id=
2026-03-29T01:09:14.306Z [INFO]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05594d09-75af-cc96-758b-7bacf89fa498 Address:05594d09-75af-cc96-758b-7bacf89fa498} {Suffrage:Voter ID:server-e9bdb21c-38fb-60dc-16b3-8782d2557f23 Address:e9bdb21c-38fb-60dc-16b3-8782d2557f23} {Suffrage:Voter ID:server-bf3b3edc-1189-3a9b-aa7a-31b382243f03 Address:bf3b3edc-1189-3a9b-aa7a-31b382243f03}]"
2026-03-29T01:09:14.306Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:14.306Z [INFO]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: entering follower state: follower="Node at bf3b3edc-1189-3a9b-aa7a-31b382243f03 [Follower]" leader-address= leader-id=
2026-03-29T01:09:14.382Z [WARN]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:14.382Z [INFO]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: entering candidate state: node="Node at bf3b3edc-1189-3a9b-aa7a-31b382243f03 [Candidate]" term=2
2026-03-29T01:09:14.382Z [INFO]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: election won: term=2 tally=2
2026-03-29T01:09:14.382Z [INFO]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: entering leader state: leader="Node at bf3b3edc-1189-3a9b-aa7a-31b382243f03 [Leader]"
2026-03-29T01:09:14.382Z [INFO]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: added peer, starting replication: peer=server-05594d09-75af-cc96-758b-7bacf89fa498
2026-03-29T01:09:14.382Z [INFO]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: added peer, starting replication: peer=server-e9bdb21c-38fb-60dc-16b3-8782d2557f23
2026-03-29T01:09:14.383Z [INFO]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: pipelining replication: peer="{Voter server-05594d09-75af-cc96-758b-7bacf89fa498 05594d09-75af-cc96-758b-7bacf89fa498}"
2026-03-29T01:09:14.383Z [INFO]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: pipelining replication: peer="{Voter server-e9bdb21c-38fb-60dc-16b3-8782d2557f23 e9bdb21c-38fb-60dc-16b3-8782d2557f23}"
2026-03-29T01:09:14.488Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:14.383007349 +0000 UTC m=+98.171149912 (1 nodes), 76.361527ms from start of poll, 76.584158ms from cluster start. Timeout at 2026-03-29 01:09:14.488013808 +0000 UTC m=+98.276156415, 105.006503ms after stability
2026-03-29T01:09:14.493Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:14.599Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:14.494023103 +0000 UTC m=+98.282165700 (2 nodes), 1.704µs from start of poll, 187.599946ms from cluster start. Timeout at 2026-03-29 01:09:14.59905095 +0000 UTC m=+98.387193571, 105.027871ms after stability
2026-03-29T01:09:14.599Z [INFO]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: updating configuration: command=RemoveServer server-id=server-05594d09-75af-cc96-758b-7bacf89fa498 server-addr= servers="[{Suffrage:Voter ID:server-e9bdb21c-38fb-60dc-16b3-8782d2557f23 Address:e9bdb21c-38fb-60dc-16b3-8782d2557f23} {Suffrage:Voter ID:server-bf3b3edc-1189-3a9b-aa7a-31b382243f03 Address:bf3b3edc-1189-3a9b-aa7a-31b382243f03}]"
2026-03-29T01:09:14.599Z [INFO]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: removed peer, stopping replication: peer=server-05594d09-75af-cc96-758b-7bacf89fa498 last-index=3
2026-03-29T01:09:14.599Z [INFO]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: aborting pipeline replication: peer="{Voter server-05594d09-75af-cc96-758b-7bacf89fa498 05594d09-75af-cc96-758b-7bacf89fa498}"
2026-03-29T01:09:14.702Z [WARN]  server-05594d09-75af-cc96-758b-7bacf89fa498: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2026-03-29T01:09:14.706Z [INFO]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: updating configuration: command=AddVoter server-id=server-05594d09-75af-cc96-758b-7bacf89fa498 server-addr=05594d09-75af-cc96-758b-7bacf89fa498 servers="[{Suffrage:Voter ID:server-e9bdb21c-38fb-60dc-16b3-8782d2557f23 Address:e9bdb21c-38fb-60dc-16b3-8782d2557f23} {Suffrage:Voter ID:server-bf3b3edc-1189-3a9b-aa7a-31b382243f03 Address:bf3b3edc-1189-3a9b-aa7a-31b382243f03} {Suffrage:Voter ID:server-05594d09-75af-cc96-758b-7bacf89fa498 Address:05594d09-75af-cc96-758b-7bacf89fa498}]"
2026-03-29T01:09:14.706Z [INFO]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: added peer, starting replication: peer=server-05594d09-75af-cc96-758b-7bacf89fa498
2026-03-29T01:09:14.706Z [WARN]  server-05594d09-75af-cc96-758b-7bacf89fa498: failed to get previous log: previous-index=4 last-index=3 error="log not found"
2026-03-29T01:09:14.706Z [WARN]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: appendEntries rejected, sending older logs: peer="{Voter server-05594d09-75af-cc96-758b-7bacf89fa498 05594d09-75af-cc96-758b-7bacf89fa498}" next=4
2026-03-29T01:09:14.706Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:14.706Z [INFO]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: pipelining replication: peer="{Voter server-05594d09-75af-cc96-758b-7bacf89fa498 05594d09-75af-cc96-758b-7bacf89fa498}"
2026-03-29T01:09:14.812Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:14.706385134 +0000 UTC m=+98.494527694 (1 nodes), 744ns from start of poll, 399.96194ms from cluster start. Timeout at 2026-03-29 01:09:14.811414191 +0000 UTC m=+98.599556911, 105.029217ms after stability
2026-03-29T01:09:14.812Z [INFO]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: aborting pipeline replication: peer="{Voter server-e9bdb21c-38fb-60dc-16b3-8782d2557f23 e9bdb21c-38fb-60dc-16b3-8782d2557f23}"
2026-03-29T01:09:14.812Z [INFO]  server-bf3b3edc-1189-3a9b-aa7a-31b382243f03: aborting pipeline replication: peer="{Voter server-05594d09-75af-cc96-758b-7bacf89fa498 05594d09-75af-cc96-758b-7bacf89fa498}"
--- PASS: TestRaft_ReJoinFollower (0.51s)
=== RUN   TestRaft_LeaderLeaseExpire
2026-03-29T01:09:14.815Z [INFO]  server-af08d1cb-146a-69ff-6fdb-d36720605164: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af08d1cb-146a-69ff-6fdb-d36720605164 Address:af08d1cb-146a-69ff-6fdb-d36720605164} {Suffrage:Voter ID:server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac Address:5542e18e-bb32-bda2-ca29-e9aa1ccb25ac}]"
2026-03-29T01:09:14.816Z [INFO]  server-af08d1cb-146a-69ff-6fdb-d36720605164: entering follower state: follower="Node at af08d1cb-146a-69ff-6fdb-d36720605164 [Follower]" leader-address= leader-id=
2026-03-29T01:09:14.816Z [INFO]  server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af08d1cb-146a-69ff-6fdb-d36720605164 Address:af08d1cb-146a-69ff-6fdb-d36720605164} {Suffrage:Voter ID:server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac Address:5542e18e-bb32-bda2-ca29-e9aa1ccb25ac}]"
2026-03-29T01:09:14.816Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:14.816Z [INFO]  server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac: entering follower state: follower="Node at 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac [Follower]" leader-address= leader-id=
2026-03-29T01:09:14.878Z [WARN]  server-af08d1cb-146a-69ff-6fdb-d36720605164: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:14.878Z [INFO]  server-af08d1cb-146a-69ff-6fdb-d36720605164: entering candidate state: node="Node at af08d1cb-146a-69ff-6fdb-d36720605164 [Candidate]" term=2
2026-03-29T01:09:14.878Z [INFO]  server-af08d1cb-146a-69ff-6fdb-d36720605164: election won: term=2 tally=2
2026-03-29T01:09:14.878Z [INFO]  server-af08d1cb-146a-69ff-6fdb-d36720605164: entering leader state: leader="Node at af08d1cb-146a-69ff-6fdb-d36720605164 [Leader]"
2026-03-29T01:09:14.878Z [INFO]  server-af08d1cb-146a-69ff-6fdb-d36720605164: added peer, starting replication: peer=server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac
2026-03-29T01:09:14.878Z [INFO]  server-af08d1cb-146a-69ff-6fdb-d36720605164: pipelining replication: peer="{Voter server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac}"
2026-03-29T01:09:14.983Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:14.878837031 +0000 UTC m=+98.666979627 (1 nodes), 62.742535ms from start of poll, 62.936756ms from cluster start. Timeout at 2026-03-29 01:09:14.983843212 +0000 UTC m=+98.771985742, 105.006115ms after stability
2026-03-29T01:09:14.988Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:15.094Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:14.989012501 +0000 UTC m=+98.777155059 (1 nodes), 634ns from start of poll, 173.112188ms from cluster start. Timeout at 2026-03-29 01:09:15.094025539 +0000 UTC m=+98.882168086, 105.013027ms after stability
    raft_test.go:1799: [INFO] Disconnecting Node at 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac [Follower]
2026-03-29T01:09:15.102Z [ERROR] server-af08d1cb-146a-69ff-6fdb-d36720605164: failed to pipeline appendEntries: peer="{Voter server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac}" error="append pipeline closed"
2026-03-29T01:09:15.102Z [INFO]  server-af08d1cb-146a-69ff-6fdb-d36720605164: aborting pipeline replication: peer="{Voter server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac}"
2026-03-29T01:09:15.103Z [ERROR] server-af08d1cb-146a-69ff-6fdb-d36720605164: failed to heartbeat to: peer=5542e18e-bb32-bda2-ca29-e9aa1ccb25ac backoff time=10ms error="failed to connect to peer: 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac"
2026-03-29T01:09:15.111Z [ERROR] server-af08d1cb-146a-69ff-6fdb-d36720605164: failed to appendEntries to: peer="{Voter server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac}" error="failed to connect to peer: 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac"
2026-03-29T01:09:15.122Z [ERROR] server-af08d1cb-146a-69ff-6fdb-d36720605164: failed to heartbeat to: peer=5542e18e-bb32-bda2-ca29-e9aa1ccb25ac backoff time=10ms error="failed to connect to peer: 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac"
2026-03-29T01:09:15.128Z [ERROR] server-af08d1cb-146a-69ff-6fdb-d36720605164: failed to appendEntries to: peer="{Voter server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac}" error="failed to connect to peer: 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac"
2026-03-29T01:09:15.138Z [ERROR] server-af08d1cb-146a-69ff-6fdb-d36720605164: failed to heartbeat to: peer=5542e18e-bb32-bda2-ca29-e9aa1ccb25ac backoff time=10ms error="failed to connect to peer: 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac"
2026-03-29T01:09:15.144Z [WARN]  server-af08d1cb-146a-69ff-6fdb-d36720605164: failed to contact: server-id=server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac time=50.415222ms
2026-03-29T01:09:15.144Z [WARN]  server-af08d1cb-146a-69ff-6fdb-d36720605164: failed to contact quorum of nodes, stepping down
2026-03-29T01:09:15.144Z [INFO]  server-af08d1cb-146a-69ff-6fdb-d36720605164: entering follower state: follower="Node at af08d1cb-146a-69ff-6fdb-d36720605164 [Follower]" leader-address= leader-id=
2026-03-29T01:09:15.144Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:15.145Z [WARN]  server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac: heartbeat timeout reached, starting election: last-leader-addr=af08d1cb-146a-69ff-6fdb-d36720605164 last-leader-id=server-af08d1cb-146a-69ff-6fdb-d36720605164
2026-03-29T01:09:15.145Z [INFO]  server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac: entering candidate state: node="Node at 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac [Candidate]" term=3
2026-03-29T01:09:15.146Z [ERROR] server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac: failed to make requestVote RPC: target="{Voter server-af08d1cb-146a-69ff-6fdb-d36720605164 af08d1cb-146a-69ff-6fdb-d36720605164}" error="failed to connect to peer: af08d1cb-146a-69ff-6fdb-d36720605164" term=3
2026-03-29T01:09:15.147Z [ERROR] server-af08d1cb-146a-69ff-6fdb-d36720605164: failed to appendEntries to: peer="{Voter server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac}" error="failed to connect to peer: 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac"
2026-03-29T01:09:15.207Z [WARN]  server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac: Election timeout reached, restarting election
2026-03-29T01:09:15.207Z [INFO]  server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac: entering candidate state: node="Node at 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac [Candidate]" term=4
2026-03-29T01:09:15.207Z [ERROR] server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac: failed to make requestVote RPC: target="{Voter server-af08d1cb-146a-69ff-6fdb-d36720605164 af08d1cb-146a-69ff-6fdb-d36720605164}" error="failed to connect to peer: af08d1cb-146a-69ff-6fdb-d36720605164" term=4
2026-03-29T01:09:15.232Z [WARN]  server-af08d1cb-146a-69ff-6fdb-d36720605164: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:15.232Z [INFO]  server-af08d1cb-146a-69ff-6fdb-d36720605164: entering candidate state: node="Node at af08d1cb-146a-69ff-6fdb-d36720605164 [Candidate]" term=3
2026-03-29T01:09:15.232Z [ERROR] server-af08d1cb-146a-69ff-6fdb-d36720605164: failed to make requestVote RPC: target="{Voter server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac}" error="failed to connect to peer: 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac" term=3
2026-03-29T01:09:15.292Z [WARN]  server-af08d1cb-146a-69ff-6fdb-d36720605164: Election timeout reached, restarting election
2026-03-29T01:09:15.292Z [INFO]  server-af08d1cb-146a-69ff-6fdb-d36720605164: entering candidate state: node="Node at af08d1cb-146a-69ff-6fdb-d36720605164 [Candidate]" term=4
2026-03-29T01:09:15.292Z [ERROR] server-af08d1cb-146a-69ff-6fdb-d36720605164: failed to make requestVote RPC: target="{Voter server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac}" error="failed to connect to peer: 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac" term=4
2026-03-29T01:09:15.298Z [WARN]  server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac: Election timeout reached, restarting election
2026-03-29T01:09:15.298Z [INFO]  server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac: entering candidate state: node="Node at 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac [Candidate]" term=5
2026-03-29T01:09:15.299Z [ERROR] server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac: failed to make requestVote RPC: target="{Voter server-af08d1cb-146a-69ff-6fdb-d36720605164 af08d1cb-146a-69ff-6fdb-d36720605164}" error="failed to connect to peer: af08d1cb-146a-69ff-6fdb-d36720605164" term=5
2026-03-29T01:09:15.337Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:15.232497747 +0000 UTC m=+99.020640286 (0 nodes), 87.895315ms from start of poll, 416.597415ms from cluster start. Timeout at 2026-03-29 01:09:15.337520086 +0000 UTC m=+99.125662670, 105.022384ms after stability
2026-03-29T01:09:15.356Z [WARN]  server-af08d1cb-146a-69ff-6fdb-d36720605164: Election timeout reached, restarting election
2026-03-29T01:09:15.356Z [INFO]  server-af08d1cb-146a-69ff-6fdb-d36720605164: entering candidate state: node="Node at af08d1cb-146a-69ff-6fdb-d36720605164 [Candidate]" term=5
2026-03-29T01:09:15.356Z [ERROR] server-af08d1cb-146a-69ff-6fdb-d36720605164: failed to make requestVote RPC: target="{Voter server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac}" error="failed to connect to peer: 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac" term=5
2026-03-29T01:09:15.377Z [WARN]  server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac: Election timeout reached, restarting election
2026-03-29T01:09:15.377Z [INFO]  server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac: entering candidate state: node="Node at 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac [Candidate]" term=6
2026-03-29T01:09:15.377Z [ERROR] server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac: failed to make requestVote RPC: target="{Voter server-af08d1cb-146a-69ff-6fdb-d36720605164 af08d1cb-146a-69ff-6fdb-d36720605164}" error="failed to connect to peer: af08d1cb-146a-69ff-6fdb-d36720605164" term=6
2026-03-29T01:09:15.434Z [WARN]  server-af08d1cb-146a-69ff-6fdb-d36720605164: Election timeout reached, restarting election
2026-03-29T01:09:15.434Z [INFO]  server-af08d1cb-146a-69ff-6fdb-d36720605164: entering candidate state: node="Node at af08d1cb-146a-69ff-6fdb-d36720605164 [Candidate]" term=6
2026-03-29T01:09:15.434Z [ERROR] server-af08d1cb-146a-69ff-6fdb-d36720605164: failed to make requestVote RPC: target="{Voter server-5542e18e-bb32-bda2-ca29-e9aa1ccb25ac 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac}" error="failed to connect to peer: 5542e18e-bb32-bda2-ca29-e9aa1ccb25ac" term=6
--- PASS: TestRaft_LeaderLeaseExpire (0.63s)
=== RUN   TestRaft_Barrier
2026-03-29T01:09:15.443Z [INFO]  server-dee550dd-c30f-7111-3944-99eeefbb4714: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dee550dd-c30f-7111-3944-99eeefbb4714 Address:dee550dd-c30f-7111-3944-99eeefbb4714} {Suffrage:Voter ID:server-1c92ca7d-d6e0-90a2-0a0c-1c78f2295e29 Address:1c92ca7d-d6e0-90a2-0a0c-1c78f2295e29} {Suffrage:Voter ID:server-cdab877b-4c7a-4e5b-d6ca-dde270bb841a Address:cdab877b-4c7a-4e5b-d6ca-dde270bb841a}]"
2026-03-29T01:09:15.443Z [INFO]  server-dee550dd-c30f-7111-3944-99eeefbb4714: entering follower state: follower="Node at dee550dd-c30f-7111-3944-99eeefbb4714 [Follower]" leader-address= leader-id=
2026-03-29T01:09:15.443Z [INFO]  server-1c92ca7d-d6e0-90a2-0a0c-1c78f2295e29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dee550dd-c30f-7111-3944-99eeefbb4714 Address:dee550dd-c30f-7111-3944-99eeefbb4714} {Suffrage:Voter ID:server-1c92ca7d-d6e0-90a2-0a0c-1c78f2295e29 Address:1c92ca7d-d6e0-90a2-0a0c-1c78f2295e29} {Suffrage:Voter ID:server-cdab877b-4c7a-4e5b-d6ca-dde270bb841a Address:cdab877b-4c7a-4e5b-d6ca-dde270bb841a}]"
2026-03-29T01:09:15.443Z [INFO]  server-1c92ca7d-d6e0-90a2-0a0c-1c78f2295e29: entering follower state: follower="Node at 1c92ca7d-d6e0-90a2-0a0c-1c78f2295e29 [Follower]" leader-address= leader-id=
2026-03-29T01:09:15.443Z [INFO]  server-cdab877b-4c7a-4e5b-d6ca-dde270bb841a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dee550dd-c30f-7111-3944-99eeefbb4714 Address:dee550dd-c30f-7111-3944-99eeefbb4714} {Suffrage:Voter ID:server-1c92ca7d-d6e0-90a2-0a0c-1c78f2295e29 Address:1c92ca7d-d6e0-90a2-0a0c-1c78f2295e29} {Suffrage:Voter ID:server-cdab877b-4c7a-4e5b-d6ca-dde270bb841a Address:cdab877b-4c7a-4e5b-d6ca-dde270bb841a}]"
2026-03-29T01:09:15.443Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:15.443Z [INFO]  server-cdab877b-4c7a-4e5b-d6ca-dde270bb841a: entering follower state: follower="Node at cdab877b-4c7a-4e5b-d6ca-dde270bb841a [Follower]" leader-address= leader-id=
2026-03-29T01:09:15.495Z [WARN]  server-dee550dd-c30f-7111-3944-99eeefbb4714: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:15.495Z [INFO]  server-dee550dd-c30f-7111-3944-99eeefbb4714: entering candidate state: node="Node at dee550dd-c30f-7111-3944-99eeefbb4714 [Candidate]" term=2
2026-03-29T01:09:15.495Z [INFO]  server-dee550dd-c30f-7111-3944-99eeefbb4714: election won: term=2 tally=2
2026-03-29T01:09:15.495Z [INFO]  server-dee550dd-c30f-7111-3944-99eeefbb4714: entering leader state: leader="Node at dee550dd-c30f-7111-3944-99eeefbb4714 [Leader]"
2026-03-29T01:09:15.495Z [INFO]  server-dee550dd-c30f-7111-3944-99eeefbb4714: added peer, starting replication: peer=server-1c92ca7d-d6e0-90a2-0a0c-1c78f2295e29
2026-03-29T01:09:15.495Z [INFO]  server-dee550dd-c30f-7111-3944-99eeefbb4714: added peer, starting replication: peer=server-cdab877b-4c7a-4e5b-d6ca-dde270bb841a
2026-03-29T01:09:15.496Z [INFO]  server-dee550dd-c30f-7111-3944-99eeefbb4714: pipelining replication: peer="{Voter server-1c92ca7d-d6e0-90a2-0a0c-1c78f2295e29 1c92ca7d-d6e0-90a2-0a0c-1c78f2295e29}"
2026-03-29T01:09:15.496Z [INFO]  server-dee550dd-c30f-7111-3944-99eeefbb4714: pipelining replication: peer="{Voter server-cdab877b-4c7a-4e5b-d6ca-dde270bb841a cdab877b-4c7a-4e5b-d6ca-dde270bb841a}"
2026-03-29T01:09:15.601Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:15.495975725 +0000 UTC m=+99.284118260 (1 nodes), 52.09572ms from start of poll, 52.40172ms from cluster start. Timeout at 2026-03-29 01:09:15.600979276 +0000 UTC m=+99.389121832, 105.003572ms after stability
2026-03-29T01:09:15.617Z [INFO]  server-dee550dd-c30f-7111-3944-99eeefbb4714: aborting pipeline replication: peer="{Voter server-1c92ca7d-d6e0-90a2-0a0c-1c78f2295e29 1c92ca7d-d6e0-90a2-0a0c-1c78f2295e29}"
2026-03-29T01:09:15.617Z [INFO]  server-dee550dd-c30f-7111-3944-99eeefbb4714: aborting pipeline replication: peer="{Voter server-cdab877b-4c7a-4e5b-d6ca-dde270bb841a cdab877b-4c7a-4e5b-d6ca-dde270bb841a}"
--- PASS: TestRaft_Barrier (0.17s)
=== RUN   TestRaft_VerifyLeader
2026-03-29T01:09:15.618Z [INFO]  server-bbcac0b6-f4b4-4b3e-b360-035e0ec62e15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bbcac0b6-f4b4-4b3e-b360-035e0ec62e15 Address:bbcac0b6-f4b4-4b3e-b360-035e0ec62e15} {Suffrage:Voter ID:server-e4050e1c-e608-67c8-6255-e282ee8ba280 Address:e4050e1c-e608-67c8-6255-e282ee8ba280} {Suffrage:Voter ID:server-b4f058a9-e9f8-142f-3ca1-d172f042631e Address:b4f058a9-e9f8-142f-3ca1-d172f042631e}]"
2026-03-29T01:09:15.618Z [INFO]  server-bbcac0b6-f4b4-4b3e-b360-035e0ec62e15: entering follower state: follower="Node at bbcac0b6-f4b4-4b3e-b360-035e0ec62e15 [Follower]" leader-address= leader-id=
2026-03-29T01:09:15.618Z [INFO]  server-e4050e1c-e608-67c8-6255-e282ee8ba280: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bbcac0b6-f4b4-4b3e-b360-035e0ec62e15 Address:bbcac0b6-f4b4-4b3e-b360-035e0ec62e15} {Suffrage:Voter ID:server-e4050e1c-e608-67c8-6255-e282ee8ba280 Address:e4050e1c-e608-67c8-6255-e282ee8ba280} {Suffrage:Voter ID:server-b4f058a9-e9f8-142f-3ca1-d172f042631e Address:b4f058a9-e9f8-142f-3ca1-d172f042631e}]"
2026-03-29T01:09:15.618Z [INFO]  server-e4050e1c-e608-67c8-6255-e282ee8ba280: entering follower state: follower="Node at e4050e1c-e608-67c8-6255-e282ee8ba280 [Follower]" leader-address= leader-id=
2026-03-29T01:09:15.618Z [INFO]  server-b4f058a9-e9f8-142f-3ca1-d172f042631e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bbcac0b6-f4b4-4b3e-b360-035e0ec62e15 Address:bbcac0b6-f4b4-4b3e-b360-035e0ec62e15} {Suffrage:Voter ID:server-e4050e1c-e608-67c8-6255-e282ee8ba280 Address:e4050e1c-e608-67c8-6255-e282ee8ba280} {Suffrage:Voter ID:server-b4f058a9-e9f8-142f-3ca1-d172f042631e Address:b4f058a9-e9f8-142f-3ca1-d172f042631e}]"
2026-03-29T01:09:15.618Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:15.618Z [INFO]  server-b4f058a9-e9f8-142f-3ca1-d172f042631e: entering follower state: follower="Node at b4f058a9-e9f8-142f-3ca1-d172f042631e [Follower]" leader-address= leader-id=
2026-03-29T01:09:15.676Z [WARN]  server-bbcac0b6-f4b4-4b3e-b360-035e0ec62e15: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:15.676Z [INFO]  server-bbcac0b6-f4b4-4b3e-b360-035e0ec62e15: entering candidate state: node="Node at bbcac0b6-f4b4-4b3e-b360-035e0ec62e15 [Candidate]" term=2
2026-03-29T01:09:15.677Z [INFO]  server-bbcac0b6-f4b4-4b3e-b360-035e0ec62e15: election won: term=2 tally=2
2026-03-29T01:09:15.677Z [INFO]  server-bbcac0b6-f4b4-4b3e-b360-035e0ec62e15: entering leader state: leader="Node at bbcac0b6-f4b4-4b3e-b360-035e0ec62e15 [Leader]"
2026-03-29T01:09:15.677Z [INFO]  server-bbcac0b6-f4b4-4b3e-b360-035e0ec62e15: added peer, starting replication: peer=server-e4050e1c-e608-67c8-6255-e282ee8ba280
2026-03-29T01:09:15.677Z [INFO]  server-bbcac0b6-f4b4-4b3e-b360-035e0ec62e15: added peer, starting replication: peer=server-b4f058a9-e9f8-142f-3ca1-d172f042631e
2026-03-29T01:09:15.677Z [INFO]  server-bbcac0b6-f4b4-4b3e-b360-035e0ec62e15: pipelining replication: peer="{Voter server-b4f058a9-e9f8-142f-3ca1-d172f042631e b4f058a9-e9f8-142f-3ca1-d172f042631e}"
2026-03-29T01:09:15.677Z [INFO]  server-bbcac0b6-f4b4-4b3e-b360-035e0ec62e15: pipelining replication: peer="{Voter server-e4050e1c-e608-67c8-6255-e282ee8ba280 e4050e1c-e608-67c8-6255-e282ee8ba280}"
2026-03-29T01:09:15.782Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:15.677125514 +0000 UTC m=+99.465268049 (1 nodes), 58.370982ms from start of poll, 58.664304ms from cluster start. Timeout at 2026-03-29 01:09:15.782130636 +0000 UTC m=+99.570273176, 105.005127ms after stability
2026-03-29T01:09:15.782Z [INFO]  server-bbcac0b6-f4b4-4b3e-b360-035e0ec62e15: aborting pipeline replication: peer="{Voter server-b4f058a9-e9f8-142f-3ca1-d172f042631e b4f058a9-e9f8-142f-3ca1-d172f042631e}"
2026-03-29T01:09:15.782Z [INFO]  server-bbcac0b6-f4b4-4b3e-b360-035e0ec62e15: aborting pipeline replication: peer="{Voter server-e4050e1c-e608-67c8-6255-e282ee8ba280 e4050e1c-e608-67c8-6255-e282ee8ba280}"
--- PASS: TestRaft_VerifyLeader (0.16s)
=== RUN   TestRaft_VerifyLeader_Single
2026-03-29T01:09:15.782Z [INFO]  server-5bbba7fe-fc73-cc12-d7a3-8d26d5afde02: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5bbba7fe-fc73-cc12-d7a3-8d26d5afde02 Address:5bbba7fe-fc73-cc12-d7a3-8d26d5afde02}]"
2026-03-29T01:09:15.782Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:15.782Z [INFO]  server-5bbba7fe-fc73-cc12-d7a3-8d26d5afde02: entering follower state: follower="Node at 5bbba7fe-fc73-cc12-d7a3-8d26d5afde02 [Follower]" leader-address= leader-id=
2026-03-29T01:09:15.853Z [WARN]  server-5bbba7fe-fc73-cc12-d7a3-8d26d5afde02: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:15.853Z [INFO]  server-5bbba7fe-fc73-cc12-d7a3-8d26d5afde02: entering candidate state: node="Node at 5bbba7fe-fc73-cc12-d7a3-8d26d5afde02 [Candidate]" term=2
2026-03-29T01:09:15.853Z [INFO]  server-5bbba7fe-fc73-cc12-d7a3-8d26d5afde02: election won: term=2 tally=1
2026-03-29T01:09:15.853Z [INFO]  server-5bbba7fe-fc73-cc12-d7a3-8d26d5afde02: entering leader state: leader="Node at 5bbba7fe-fc73-cc12-d7a3-8d26d5afde02 [Leader]"
2026-03-29T01:09:15.958Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:15.853188601 +0000 UTC m=+99.641331135 (1 nodes), 70.262046ms from start of poll, 70.384714ms from cluster start. Timeout at 2026-03-29 01:09:15.958191441 +0000 UTC m=+99.746333977, 105.002842ms after stability
--- PASS: TestRaft_VerifyLeader_Single (0.18s)
=== RUN   TestRaft_VerifyLeader_Fail
2026-03-29T01:09:15.959Z [INFO]  server-02cd4046-c0ac-81b3-ae8e-c35814ad7595: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02cd4046-c0ac-81b3-ae8e-c35814ad7595 Address:02cd4046-c0ac-81b3-ae8e-c35814ad7595} {Suffrage:Voter ID:server-be03f828-243e-85c0-08a9-bc860fd3e3ff Address:be03f828-243e-85c0-08a9-bc860fd3e3ff}]"
2026-03-29T01:09:15.960Z [INFO]  server-02cd4046-c0ac-81b3-ae8e-c35814ad7595: entering follower state: follower="Node at 02cd4046-c0ac-81b3-ae8e-c35814ad7595 [Follower]" leader-address= leader-id=
2026-03-29T01:09:15.960Z [INFO]  server-be03f828-243e-85c0-08a9-bc860fd3e3ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-02cd4046-c0ac-81b3-ae8e-c35814ad7595 Address:02cd4046-c0ac-81b3-ae8e-c35814ad7595} {Suffrage:Voter ID:server-be03f828-243e-85c0-08a9-bc860fd3e3ff Address:be03f828-243e-85c0-08a9-bc860fd3e3ff}]"
2026-03-29T01:09:15.960Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:15.960Z [INFO]  server-be03f828-243e-85c0-08a9-bc860fd3e3ff: entering follower state: follower="Node at be03f828-243e-85c0-08a9-bc860fd3e3ff [Follower]" leader-address= leader-id=
2026-03-29T01:09:16.012Z [WARN]  server-02cd4046-c0ac-81b3-ae8e-c35814ad7595: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:16.012Z [INFO]  server-02cd4046-c0ac-81b3-ae8e-c35814ad7595: entering candidate state: node="Node at 02cd4046-c0ac-81b3-ae8e-c35814ad7595 [Candidate]" term=2
2026-03-29T01:09:16.012Z [INFO]  server-02cd4046-c0ac-81b3-ae8e-c35814ad7595: election won: term=2 tally=2
2026-03-29T01:09:16.012Z [INFO]  server-02cd4046-c0ac-81b3-ae8e-c35814ad7595: entering leader state: leader="Node at 02cd4046-c0ac-81b3-ae8e-c35814ad7595 [Leader]"
2026-03-29T01:09:16.012Z [INFO]  server-02cd4046-c0ac-81b3-ae8e-c35814ad7595: added peer, starting replication: peer=server-be03f828-243e-85c0-08a9-bc860fd3e3ff
2026-03-29T01:09:16.013Z [INFO]  server-02cd4046-c0ac-81b3-ae8e-c35814ad7595: pipelining replication: peer="{Voter server-be03f828-243e-85c0-08a9-bc860fd3e3ff be03f828-243e-85c0-08a9-bc860fd3e3ff}"
2026-03-29T01:09:16.118Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:16.012986674 +0000 UTC m=+99.801129208 (1 nodes), 52.84125ms from start of poll, 53.132275ms from cluster start. Timeout at 2026-03-29 01:09:16.117989677 +0000 UTC m=+99.906132207, 105.002999ms after stability
2026-03-29T01:09:16.118Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:16.224Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:16.118599433 +0000 UTC m=+99.906741976 (1 nodes), 891ns from start of poll, 158.745043ms from cluster start. Timeout at 2026-03-29 01:09:16.223615109 +0000 UTC m=+100.011757639, 105.015663ms after stability
2026-03-29T01:09:16.227Z [ERROR] server-02cd4046-c0ac-81b3-ae8e-c35814ad7595: peer has newer term, stopping replication: peer="{Voter server-be03f828-243e-85c0-08a9-bc860fd3e3ff be03f828-243e-85c0-08a9-bc860fd3e3ff}"
2026-03-29T01:09:16.227Z [INFO]  server-02cd4046-c0ac-81b3-ae8e-c35814ad7595: aborting pipeline replication: peer="{Voter server-be03f828-243e-85c0-08a9-bc860fd3e3ff be03f828-243e-85c0-08a9-bc860fd3e3ff}"
2026-03-29T01:09:16.227Z [INFO]  server-02cd4046-c0ac-81b3-ae8e-c35814ad7595: entering follower state: follower="Node at 02cd4046-c0ac-81b3-ae8e-c35814ad7595 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VerifyLeader_Fail (0.27s)
=== RUN   TestRaft_VerifyLeader_PartialConnect
2026-03-29T01:09:16.228Z [INFO]  server-ac8ba01e-7095-c3a7-c49f-bf3ce666781c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac8ba01e-7095-c3a7-c49f-bf3ce666781c Address:ac8ba01e-7095-c3a7-c49f-bf3ce666781c} {Suffrage:Voter ID:server-6711dff4-a50f-35cf-3c66-83771174ea05 Address:6711dff4-a50f-35cf-3c66-83771174ea05} {Suffrage:Voter ID:server-d66b22de-55c1-f0e9-28ae-25280dc532c8 Address:d66b22de-55c1-f0e9-28ae-25280dc532c8}]"
2026-03-29T01:09:16.228Z [INFO]  server-ac8ba01e-7095-c3a7-c49f-bf3ce666781c: entering follower state: follower="Node at ac8ba01e-7095-c3a7-c49f-bf3ce666781c [Follower]" leader-address= leader-id=
2026-03-29T01:09:16.228Z [INFO]  server-6711dff4-a50f-35cf-3c66-83771174ea05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac8ba01e-7095-c3a7-c49f-bf3ce666781c Address:ac8ba01e-7095-c3a7-c49f-bf3ce666781c} {Suffrage:Voter ID:server-6711dff4-a50f-35cf-3c66-83771174ea05 Address:6711dff4-a50f-35cf-3c66-83771174ea05} {Suffrage:Voter ID:server-d66b22de-55c1-f0e9-28ae-25280dc532c8 Address:d66b22de-55c1-f0e9-28ae-25280dc532c8}]"
2026-03-29T01:09:16.228Z [INFO]  server-6711dff4-a50f-35cf-3c66-83771174ea05: entering follower state: follower="Node at 6711dff4-a50f-35cf-3c66-83771174ea05 [Follower]" leader-address= leader-id=
2026-03-29T01:09:16.228Z [INFO]  server-d66b22de-55c1-f0e9-28ae-25280dc532c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ac8ba01e-7095-c3a7-c49f-bf3ce666781c Address:ac8ba01e-7095-c3a7-c49f-bf3ce666781c} {Suffrage:Voter ID:server-6711dff4-a50f-35cf-3c66-83771174ea05 Address:6711dff4-a50f-35cf-3c66-83771174ea05} {Suffrage:Voter ID:server-d66b22de-55c1-f0e9-28ae-25280dc532c8 Address:d66b22de-55c1-f0e9-28ae-25280dc532c8}]"
2026-03-29T01:09:16.228Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:16.228Z [INFO]  server-d66b22de-55c1-f0e9-28ae-25280dc532c8: entering follower state: follower="Node at d66b22de-55c1-f0e9-28ae-25280dc532c8 [Follower]" leader-address= leader-id=
2026-03-29T01:09:16.288Z [WARN]  server-ac8ba01e-7095-c3a7-c49f-bf3ce666781c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:16.288Z [INFO]  server-ac8ba01e-7095-c3a7-c49f-bf3ce666781c: entering candidate state: node="Node at ac8ba01e-7095-c3a7-c49f-bf3ce666781c [Candidate]" term=2
2026-03-29T01:09:16.288Z [INFO]  server-ac8ba01e-7095-c3a7-c49f-bf3ce666781c: election won: term=2 tally=2
2026-03-29T01:09:16.288Z [INFO]  server-ac8ba01e-7095-c3a7-c49f-bf3ce666781c: entering leader state: leader="Node at ac8ba01e-7095-c3a7-c49f-bf3ce666781c [Leader]"
2026-03-29T01:09:16.289Z [INFO]  server-ac8ba01e-7095-c3a7-c49f-bf3ce666781c: added peer, starting replication: peer=server-6711dff4-a50f-35cf-3c66-83771174ea05
2026-03-29T01:09:16.289Z [INFO]  server-ac8ba01e-7095-c3a7-c49f-bf3ce666781c: added peer, starting replication: peer=server-d66b22de-55c1-f0e9-28ae-25280dc532c8
2026-03-29T01:09:16.289Z [INFO]  server-ac8ba01e-7095-c3a7-c49f-bf3ce666781c: pipelining replication: peer="{Voter server-6711dff4-a50f-35cf-3c66-83771174ea05 6711dff4-a50f-35cf-3c66-83771174ea05}"
2026-03-29T01:09:16.289Z [INFO]  server-ac8ba01e-7095-c3a7-c49f-bf3ce666781c: pipelining replication: peer="{Voter server-d66b22de-55c1-f0e9-28ae-25280dc532c8 d66b22de-55c1-f0e9-28ae-25280dc532c8}"
2026-03-29T01:09:16.394Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:16.289067751 +0000 UTC m=+100.077210288 (1 nodes), 60.422198ms from start of poll, 60.751944ms from cluster start. Timeout at 2026-03-29 01:09:16.394150188 +0000 UTC m=+100.182292770, 105.082482ms after stability
2026-03-29T01:09:16.399Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:16.505Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:16.399458863 +0000 UTC m=+100.187601407 (2 nodes), 1.205µs from start of poll, 171.143063ms from cluster start. Timeout at 2026-03-29 01:09:16.504464878 +0000 UTC m=+100.292607512, 105.006105ms after stability
    raft_test.go:1968: [INFO] Disconnecting Node at 6711dff4-a50f-35cf-3c66-83771174ea05 [Follower]
2026-03-29T01:09:16.505Z [ERROR] server-ac8ba01e-7095-c3a7-c49f-bf3ce666781c: failed to heartbeat to: peer=6711dff4-a50f-35cf-3c66-83771174ea05 backoff time=10ms error="failed to connect to peer: 6711dff4-a50f-35cf-3c66-83771174ea05"
2026-03-29T01:09:16.505Z [INFO]  server-ac8ba01e-7095-c3a7-c49f-bf3ce666781c: aborting pipeline replication: peer="{Voter server-d66b22de-55c1-f0e9-28ae-25280dc532c8 d66b22de-55c1-f0e9-28ae-25280dc532c8}"
2026-03-29T01:09:16.505Z [INFO]  server-ac8ba01e-7095-c3a7-c49f-bf3ce666781c: aborting pipeline replication: peer="{Voter server-6711dff4-a50f-35cf-3c66-83771174ea05 6711dff4-a50f-35cf-3c66-83771174ea05}"
--- PASS: TestRaft_VerifyLeader_PartialConnect (0.28s)
=== RUN   TestRaft_NotifyCh
2026-03-29T01:09:16.506Z [INFO]  server-70e1857a-8016-1a67-102e-c4db767174a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-70e1857a-8016-1a67-102e-c4db767174a8 Address:70e1857a-8016-1a67-102e-c4db767174a8}]"
2026-03-29T01:09:16.506Z [INFO]  server-70e1857a-8016-1a67-102e-c4db767174a8: entering follower state: follower="Node at 70e1857a-8016-1a67-102e-c4db767174a8 [Follower]" leader-address= leader-id=
2026-03-29T01:09:16.571Z [WARN]  server-70e1857a-8016-1a67-102e-c4db767174a8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:16.571Z [INFO]  server-70e1857a-8016-1a67-102e-c4db767174a8: entering candidate state: node="Node at 70e1857a-8016-1a67-102e-c4db767174a8 [Candidate]" term=2
2026-03-29T01:09:16.571Z [INFO]  server-70e1857a-8016-1a67-102e-c4db767174a8: election won: term=2 tally=1
2026-03-29T01:09:16.571Z [INFO]  server-70e1857a-8016-1a67-102e-c4db767174a8: entering leader state: leader="Node at 70e1857a-8016-1a67-102e-c4db767174a8 [Leader]"
--- PASS: TestRaft_NotifyCh (0.07s)
=== RUN   TestRaft_AppendEntry
2026-03-29T01:09:16.571Z [INFO]  server-a01380b1-fcd6-afec-33f1-71bfb75d5248: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a01380b1-fcd6-afec-33f1-71bfb75d5248 Address:a01380b1-fcd6-afec-33f1-71bfb75d5248} {Suffrage:Voter ID:server-d794687b-6809-858e-afd4-731d3a43bc7b Address:d794687b-6809-858e-afd4-731d3a43bc7b} {Suffrage:Voter ID:server-707f0be0-e5f2-2f64-fd5c-e7576d8845f2 Address:707f0be0-e5f2-2f64-fd5c-e7576d8845f2}]"
2026-03-29T01:09:16.572Z [INFO]  server-d794687b-6809-858e-afd4-731d3a43bc7b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a01380b1-fcd6-afec-33f1-71bfb75d5248 Address:a01380b1-fcd6-afec-33f1-71bfb75d5248} {Suffrage:Voter ID:server-d794687b-6809-858e-afd4-731d3a43bc7b Address:d794687b-6809-858e-afd4-731d3a43bc7b} {Suffrage:Voter ID:server-707f0be0-e5f2-2f64-fd5c-e7576d8845f2 Address:707f0be0-e5f2-2f64-fd5c-e7576d8845f2}]"
2026-03-29T01:09:16.572Z [INFO]  server-a01380b1-fcd6-afec-33f1-71bfb75d5248: entering follower state: follower="Node at a01380b1-fcd6-afec-33f1-71bfb75d5248 [Follower]" leader-address= leader-id=
2026-03-29T01:09:16.572Z [INFO]  server-d794687b-6809-858e-afd4-731d3a43bc7b: entering follower state: follower="Node at d794687b-6809-858e-afd4-731d3a43bc7b [Follower]" leader-address= leader-id=
2026-03-29T01:09:16.572Z [INFO]  server-707f0be0-e5f2-2f64-fd5c-e7576d8845f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a01380b1-fcd6-afec-33f1-71bfb75d5248 Address:a01380b1-fcd6-afec-33f1-71bfb75d5248} {Suffrage:Voter ID:server-d794687b-6809-858e-afd4-731d3a43bc7b Address:d794687b-6809-858e-afd4-731d3a43bc7b} {Suffrage:Voter ID:server-707f0be0-e5f2-2f64-fd5c-e7576d8845f2 Address:707f0be0-e5f2-2f64-fd5c-e7576d8845f2}]"
2026-03-29T01:09:16.572Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:16.572Z [INFO]  server-707f0be0-e5f2-2f64-fd5c-e7576d8845f2: entering follower state: follower="Node at 707f0be0-e5f2-2f64-fd5c-e7576d8845f2 [Follower]" leader-address= leader-id=
2026-03-29T01:09:16.627Z [WARN]  server-d794687b-6809-858e-afd4-731d3a43bc7b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:16.627Z [INFO]  server-d794687b-6809-858e-afd4-731d3a43bc7b: entering candidate state: node="Node at d794687b-6809-858e-afd4-731d3a43bc7b [Candidate]" term=2
2026-03-29T01:09:16.627Z [INFO]  server-d794687b-6809-858e-afd4-731d3a43bc7b: election won: term=2 tally=2
2026-03-29T01:09:16.627Z [INFO]  server-d794687b-6809-858e-afd4-731d3a43bc7b: entering leader state: leader="Node at d794687b-6809-858e-afd4-731d3a43bc7b [Leader]"
2026-03-29T01:09:16.627Z [INFO]  server-d794687b-6809-858e-afd4-731d3a43bc7b: added peer, starting replication: peer=server-a01380b1-fcd6-afec-33f1-71bfb75d5248
2026-03-29T01:09:16.627Z [INFO]  server-d794687b-6809-858e-afd4-731d3a43bc7b: added peer, starting replication: peer=server-707f0be0-e5f2-2f64-fd5c-e7576d8845f2
2026-03-29T01:09:16.627Z [INFO]  server-d794687b-6809-858e-afd4-731d3a43bc7b: pipelining replication: peer="{Voter server-a01380b1-fcd6-afec-33f1-71bfb75d5248 a01380b1-fcd6-afec-33f1-71bfb75d5248}"
2026-03-29T01:09:16.627Z [INFO]  server-d794687b-6809-858e-afd4-731d3a43bc7b: pipelining replication: peer="{Voter server-707f0be0-e5f2-2f64-fd5c-e7576d8845f2 707f0be0-e5f2-2f64-fd5c-e7576d8845f2}"
2026-03-29T01:09:16.732Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:16.627315567 +0000 UTC m=+100.415458124 (2 nodes), 55.213809ms from start of poll, 55.434546ms from cluster start. Timeout at 2026-03-29 01:09:16.732326489 +0000 UTC m=+100.520469072, 105.010948ms after stability
2026-03-29T01:09:16.732Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:16.838Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:16.732878559 +0000 UTC m=+100.521021196 (1 nodes), 1.698µs from start of poll, 160.997618ms from cluster start. Timeout at 2026-03-29 01:09:16.837889317 +0000 UTC m=+100.626031944, 105.010748ms after stability
2026-03-29T01:09:16.838Z [WARN]  server-a01380b1-fcd6-afec-33f1-71bfb75d5248: clearing log suffix: from=1 to=2
2026-03-29T01:09:16.838Z [WARN]  server-a01380b1-fcd6-afec-33f1-71bfb75d5248: skipping application of old log: index=1
2026-03-29T01:09:16.838Z [WARN]  server-a01380b1-fcd6-afec-33f1-71bfb75d5248: skipping application of old log: index=1
2026-03-29T01:09:16.838Z [INFO]  server-d794687b-6809-858e-afd4-731d3a43bc7b: aborting pipeline replication: peer="{Voter server-707f0be0-e5f2-2f64-fd5c-e7576d8845f2 707f0be0-e5f2-2f64-fd5c-e7576d8845f2}"
2026-03-29T01:09:16.838Z [INFO]  server-d794687b-6809-858e-afd4-731d3a43bc7b: aborting pipeline replication: peer="{Voter server-a01380b1-fcd6-afec-33f1-71bfb75d5248 a01380b1-fcd6-afec-33f1-71bfb75d5248}"
--- PASS: TestRaft_AppendEntry (0.27s)
=== RUN   TestRaft_VotingGrant_WhenLeaderAvailable
2026-03-29T01:09:16.839Z [INFO]  server-e3c8b029-dda0-b5f1-e0cf-ce78398f5d63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3c8b029-dda0-b5f1-e0cf-ce78398f5d63 Address:e3c8b029-dda0-b5f1-e0cf-ce78398f5d63} {Suffrage:Voter ID:server-c35793dd-c1bd-e122-36d4-c088d34b10ed Address:c35793dd-c1bd-e122-36d4-c088d34b10ed} {Suffrage:Voter ID:server-1b5c494a-59e1-90c5-b814-179fa879bef7 Address:1b5c494a-59e1-90c5-b814-179fa879bef7}]"
2026-03-29T01:09:16.839Z [INFO]  server-e3c8b029-dda0-b5f1-e0cf-ce78398f5d63: entering follower state: follower="Node at e3c8b029-dda0-b5f1-e0cf-ce78398f5d63 [Follower]" leader-address= leader-id=
2026-03-29T01:09:16.839Z [INFO]  server-c35793dd-c1bd-e122-36d4-c088d34b10ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3c8b029-dda0-b5f1-e0cf-ce78398f5d63 Address:e3c8b029-dda0-b5f1-e0cf-ce78398f5d63} {Suffrage:Voter ID:server-c35793dd-c1bd-e122-36d4-c088d34b10ed Address:c35793dd-c1bd-e122-36d4-c088d34b10ed} {Suffrage:Voter ID:server-1b5c494a-59e1-90c5-b814-179fa879bef7 Address:1b5c494a-59e1-90c5-b814-179fa879bef7}]"
2026-03-29T01:09:16.839Z [INFO]  server-c35793dd-c1bd-e122-36d4-c088d34b10ed: entering follower state: follower="Node at c35793dd-c1bd-e122-36d4-c088d34b10ed [Follower]" leader-address= leader-id=
2026-03-29T01:09:16.839Z [INFO]  server-1b5c494a-59e1-90c5-b814-179fa879bef7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3c8b029-dda0-b5f1-e0cf-ce78398f5d63 Address:e3c8b029-dda0-b5f1-e0cf-ce78398f5d63} {Suffrage:Voter ID:server-c35793dd-c1bd-e122-36d4-c088d34b10ed Address:c35793dd-c1bd-e122-36d4-c088d34b10ed} {Suffrage:Voter ID:server-1b5c494a-59e1-90c5-b814-179fa879bef7 Address:1b5c494a-59e1-90c5-b814-179fa879bef7}]"
2026-03-29T01:09:16.839Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:16.839Z [INFO]  server-1b5c494a-59e1-90c5-b814-179fa879bef7: entering follower state: follower="Node at 1b5c494a-59e1-90c5-b814-179fa879bef7 [Follower]" leader-address= leader-id=
2026-03-29T01:09:16.903Z [WARN]  server-1b5c494a-59e1-90c5-b814-179fa879bef7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:16.903Z [INFO]  server-1b5c494a-59e1-90c5-b814-179fa879bef7: entering candidate state: node="Node at 1b5c494a-59e1-90c5-b814-179fa879bef7 [Candidate]" term=2
2026-03-29T01:09:16.904Z [INFO]  server-1b5c494a-59e1-90c5-b814-179fa879bef7: election won: term=2 tally=2
2026-03-29T01:09:16.904Z [INFO]  server-1b5c494a-59e1-90c5-b814-179fa879bef7: entering leader state: leader="Node at 1b5c494a-59e1-90c5-b814-179fa879bef7 [Leader]"
2026-03-29T01:09:16.904Z [INFO]  server-1b5c494a-59e1-90c5-b814-179fa879bef7: added peer, starting replication: peer=server-e3c8b029-dda0-b5f1-e0cf-ce78398f5d63
2026-03-29T01:09:16.904Z [INFO]  server-1b5c494a-59e1-90c5-b814-179fa879bef7: added peer, starting replication: peer=server-c35793dd-c1bd-e122-36d4-c088d34b10ed
2026-03-29T01:09:16.904Z [INFO]  server-1b5c494a-59e1-90c5-b814-179fa879bef7: pipelining replication: peer="{Voter server-c35793dd-c1bd-e122-36d4-c088d34b10ed c35793dd-c1bd-e122-36d4-c088d34b10ed}"
2026-03-29T01:09:16.904Z [INFO]  server-1b5c494a-59e1-90c5-b814-179fa879bef7: pipelining replication: peer="{Voter server-e3c8b029-dda0-b5f1-e0cf-ce78398f5d63 e3c8b029-dda0-b5f1-e0cf-ce78398f5d63}"
2026-03-29T01:09:17.009Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:16.904036828 +0000 UTC m=+100.692179365 (2 nodes), 64.560422ms from start of poll, 64.88668ms from cluster start. Timeout at 2026-03-29 01:09:17.009039393 +0000 UTC m=+100.797181924, 105.002559ms after stability
2026-03-29T01:09:17.009Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:17.115Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:17.009821978 +0000 UTC m=+100.797964556 (1 nodes), 652ns from start of poll, 170.671871ms from cluster start. Timeout at 2026-03-29 01:09:17.114850722 +0000 UTC m=+100.902993306, 105.02875ms after stability
2026-03-29T01:09:17.115Z [WARN]  server-c35793dd-c1bd-e122-36d4-c088d34b10ed: rejecting vote request since we have a leader: from=e3c8b029-dda0-b5f1-e0cf-ce78398f5d63 leader=1b5c494a-59e1-90c5-b814-179fa879bef7 leader-id=server-1b5c494a-59e1-90c5-b814-179fa879bef7
2026-03-29T01:09:17.115Z [INFO]  server-1b5c494a-59e1-90c5-b814-179fa879bef7: aborting pipeline replication: peer="{Voter server-c35793dd-c1bd-e122-36d4-c088d34b10ed c35793dd-c1bd-e122-36d4-c088d34b10ed}"
2026-03-29T01:09:17.115Z [INFO]  server-1b5c494a-59e1-90c5-b814-179fa879bef7: aborting pipeline replication: peer="{Voter server-e3c8b029-dda0-b5f1-e0cf-ce78398f5d63 e3c8b029-dda0-b5f1-e0cf-ce78398f5d63}"
--- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.28s)
=== RUN   TestRaft_ProtocolVersion_RejectRPC
2026-03-29T01:09:17.116Z [INFO]  server-567bf746-8774-b305-2a9e-fa18b1089bcb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-567bf746-8774-b305-2a9e-fa18b1089bcb Address:567bf746-8774-b305-2a9e-fa18b1089bcb} {Suffrage:Voter ID:server-809fe7c3-c33d-862a-2142-62d810876ee5 Address:809fe7c3-c33d-862a-2142-62d810876ee5} {Suffrage:Voter ID:server-67cd9e88-0352-4364-7139-616cedf54512 Address:67cd9e88-0352-4364-7139-616cedf54512}]"
2026-03-29T01:09:17.116Z [INFO]  server-567bf746-8774-b305-2a9e-fa18b1089bcb: entering follower state: follower="Node at 567bf746-8774-b305-2a9e-fa18b1089bcb [Follower]" leader-address= leader-id=
2026-03-29T01:09:17.116Z [INFO]  server-809fe7c3-c33d-862a-2142-62d810876ee5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-567bf746-8774-b305-2a9e-fa18b1089bcb Address:567bf746-8774-b305-2a9e-fa18b1089bcb} {Suffrage:Voter ID:server-809fe7c3-c33d-862a-2142-62d810876ee5 Address:809fe7c3-c33d-862a-2142-62d810876ee5} {Suffrage:Voter ID:server-67cd9e88-0352-4364-7139-616cedf54512 Address:67cd9e88-0352-4364-7139-616cedf54512}]"
2026-03-29T01:09:17.116Z [INFO]  server-809fe7c3-c33d-862a-2142-62d810876ee5: entering follower state: follower="Node at 809fe7c3-c33d-862a-2142-62d810876ee5 [Follower]" leader-address= leader-id=
2026-03-29T01:09:17.116Z [INFO]  server-67cd9e88-0352-4364-7139-616cedf54512: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-567bf746-8774-b305-2a9e-fa18b1089bcb Address:567bf746-8774-b305-2a9e-fa18b1089bcb} {Suffrage:Voter ID:server-809fe7c3-c33d-862a-2142-62d810876ee5 Address:809fe7c3-c33d-862a-2142-62d810876ee5} {Suffrage:Voter ID:server-67cd9e88-0352-4364-7139-616cedf54512 Address:67cd9e88-0352-4364-7139-616cedf54512}]"
2026-03-29T01:09:17.116Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:17.116Z [INFO]  server-67cd9e88-0352-4364-7139-616cedf54512: entering follower state: follower="Node at 67cd9e88-0352-4364-7139-616cedf54512 [Follower]" leader-address= leader-id=
2026-03-29T01:09:17.177Z [WARN]  server-567bf746-8774-b305-2a9e-fa18b1089bcb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:17.177Z [INFO]  server-567bf746-8774-b305-2a9e-fa18b1089bcb: entering candidate state: node="Node at 567bf746-8774-b305-2a9e-fa18b1089bcb [Candidate]" term=2
2026-03-29T01:09:17.177Z [INFO]  server-567bf746-8774-b305-2a9e-fa18b1089bcb: election won: term=2 tally=2
2026-03-29T01:09:17.177Z [INFO]  server-567bf746-8774-b305-2a9e-fa18b1089bcb: entering leader state: leader="Node at 567bf746-8774-b305-2a9e-fa18b1089bcb [Leader]"
2026-03-29T01:09:17.177Z [INFO]  server-567bf746-8774-b305-2a9e-fa18b1089bcb: added peer, starting replication: peer=server-809fe7c3-c33d-862a-2142-62d810876ee5
2026-03-29T01:09:17.177Z [INFO]  server-567bf746-8774-b305-2a9e-fa18b1089bcb: added peer, starting replication: peer=server-67cd9e88-0352-4364-7139-616cedf54512
2026-03-29T01:09:17.177Z [INFO]  server-567bf746-8774-b305-2a9e-fa18b1089bcb: pipelining replication: peer="{Voter server-67cd9e88-0352-4364-7139-616cedf54512 67cd9e88-0352-4364-7139-616cedf54512}"
2026-03-29T01:09:17.177Z [INFO]  server-567bf746-8774-b305-2a9e-fa18b1089bcb: pipelining replication: peer="{Voter server-809fe7c3-c33d-862a-2142-62d810876ee5 809fe7c3-c33d-862a-2142-62d810876ee5}"
2026-03-29T01:09:17.282Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:17.17737908 +0000 UTC m=+100.965521620 (2 nodes), 60.56763ms from start of poll, 60.864721ms from cluster start. Timeout at 2026-03-29 01:09:17.282382475 +0000 UTC m=+101.070525006, 105.003386ms after stability
2026-03-29T01:09:17.283Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:17.388Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:17.283019881 +0000 UTC m=+101.071162427 (1 nodes), 804ns from start of poll, 166.505528ms from cluster start. Timeout at 2026-03-29 01:09:17.388038704 +0000 UTC m=+101.176181321, 105.018894ms after stability
2026-03-29T01:09:17.388Z [INFO]  server-567bf746-8774-b305-2a9e-fa18b1089bcb: aborting pipeline replication: peer="{Voter server-67cd9e88-0352-4364-7139-616cedf54512 67cd9e88-0352-4364-7139-616cedf54512}"
2026-03-29T01:09:17.388Z [INFO]  server-567bf746-8774-b305-2a9e-fa18b1089bcb: aborting pipeline replication: peer="{Voter server-809fe7c3-c33d-862a-2142-62d810876ee5 809fe7c3-c33d-862a-2142-62d810876ee5}"
--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_1_2
    raft_test.go:2151: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_2_3
    raft_test.go:2196: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s)
=== RUN   TestRaft_LeaderID_Propagated
2026-03-29T01:09:17.389Z [INFO]  server-2172d1d6-5ff8-7087-5b8a-4261630fa3d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2172d1d6-5ff8-7087-5b8a-4261630fa3d3 Address:2172d1d6-5ff8-7087-5b8a-4261630fa3d3} {Suffrage:Voter ID:server-ce9eee01-008a-2665-d838-1eee9ad4c47f Address:ce9eee01-008a-2665-d838-1eee9ad4c47f} {Suffrage:Voter ID:server-20c0b0c8-d05b-6117-a1d0-e9735cc26225 Address:20c0b0c8-d05b-6117-a1d0-e9735cc26225}]"
2026-03-29T01:09:17.389Z [INFO]  server-2172d1d6-5ff8-7087-5b8a-4261630fa3d3: entering follower state: follower="Node at 2172d1d6-5ff8-7087-5b8a-4261630fa3d3 [Follower]" leader-address= leader-id=
2026-03-29T01:09:17.389Z [INFO]  server-ce9eee01-008a-2665-d838-1eee9ad4c47f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2172d1d6-5ff8-7087-5b8a-4261630fa3d3 Address:2172d1d6-5ff8-7087-5b8a-4261630fa3d3} {Suffrage:Voter ID:server-ce9eee01-008a-2665-d838-1eee9ad4c47f Address:ce9eee01-008a-2665-d838-1eee9ad4c47f} {Suffrage:Voter ID:server-20c0b0c8-d05b-6117-a1d0-e9735cc26225 Address:20c0b0c8-d05b-6117-a1d0-e9735cc26225}]"
2026-03-29T01:09:17.389Z [INFO]  server-ce9eee01-008a-2665-d838-1eee9ad4c47f: entering follower state: follower="Node at ce9eee01-008a-2665-d838-1eee9ad4c47f [Follower]" leader-address= leader-id=
2026-03-29T01:09:17.390Z [INFO]  server-20c0b0c8-d05b-6117-a1d0-e9735cc26225: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2172d1d6-5ff8-7087-5b8a-4261630fa3d3 Address:2172d1d6-5ff8-7087-5b8a-4261630fa3d3} {Suffrage:Voter ID:server-ce9eee01-008a-2665-d838-1eee9ad4c47f Address:ce9eee01-008a-2665-d838-1eee9ad4c47f} {Suffrage:Voter ID:server-20c0b0c8-d05b-6117-a1d0-e9735cc26225 Address:20c0b0c8-d05b-6117-a1d0-e9735cc26225}]"
2026-03-29T01:09:17.390Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:17.390Z [INFO]  server-20c0b0c8-d05b-6117-a1d0-e9735cc26225: entering follower state: follower="Node at 20c0b0c8-d05b-6117-a1d0-e9735cc26225 [Follower]" leader-address= leader-id=
2026-03-29T01:09:17.457Z [WARN]  server-2172d1d6-5ff8-7087-5b8a-4261630fa3d3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:17.457Z [INFO]  server-2172d1d6-5ff8-7087-5b8a-4261630fa3d3: entering candidate state: node="Node at 2172d1d6-5ff8-7087-5b8a-4261630fa3d3 [Candidate]" term=2
2026-03-29T01:09:17.457Z [INFO]  server-2172d1d6-5ff8-7087-5b8a-4261630fa3d3: election won: term=2 tally=2
2026-03-29T01:09:17.457Z [INFO]  server-2172d1d6-5ff8-7087-5b8a-4261630fa3d3: entering leader state: leader="Node at 2172d1d6-5ff8-7087-5b8a-4261630fa3d3 [Leader]"
2026-03-29T01:09:17.457Z [INFO]  server-2172d1d6-5ff8-7087-5b8a-4261630fa3d3: added peer, starting replication: peer=server-ce9eee01-008a-2665-d838-1eee9ad4c47f
2026-03-29T01:09:17.457Z [INFO]  server-2172d1d6-5ff8-7087-5b8a-4261630fa3d3: added peer, starting replication: peer=server-20c0b0c8-d05b-6117-a1d0-e9735cc26225
2026-03-29T01:09:17.457Z [INFO]  server-2172d1d6-5ff8-7087-5b8a-4261630fa3d3: pipelining replication: peer="{Voter server-ce9eee01-008a-2665-d838-1eee9ad4c47f ce9eee01-008a-2665-d838-1eee9ad4c47f}"
2026-03-29T01:09:17.457Z [INFO]  server-2172d1d6-5ff8-7087-5b8a-4261630fa3d3: pipelining replication: peer="{Voter server-20c0b0c8-d05b-6117-a1d0-e9735cc26225 20c0b0c8-d05b-6117-a1d0-e9735cc26225}"
2026-03-29T01:09:17.563Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:17.457437131 +0000 UTC m=+101.245579768 (1 nodes), 67.341056ms from start of poll, 67.65794ms from cluster start. Timeout at 2026-03-29 01:09:17.562444604 +0000 UTC m=+101.350587188, 105.00742ms after stability
2026-03-29T01:09:17.563Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:17.668Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:17.56333783 +0000 UTC m=+101.351480382 (1 nodes), 618ns from start of poll, 173.558554ms from cluster start. Timeout at 2026-03-29 01:09:17.668349062 +0000 UTC m=+101.456491615, 105.011233ms after stability
2026-03-29T01:09:17.668Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:17.774Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:17.668672351 +0000 UTC m=+101.456814900 (1 nodes), 614ns from start of poll, 278.893072ms from cluster start. Timeout at 2026-03-29 01:09:17.773745655 +0000 UTC m=+101.561888202, 105.073302ms after stability
2026-03-29T01:09:17.774Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:17.880Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:17.774488676 +0000 UTC m=+101.562631224 (1 nodes), 647ns from start of poll, 384.709396ms from cluster start. Timeout at 2026-03-29 01:09:17.879499797 +0000 UTC m=+101.667642430, 105.011206ms after stability
2026-03-29T01:09:17.880Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:17.986Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:17.880425548 +0000 UTC m=+101.668568103 (1 nodes), 772ns from start of poll, 490.646275ms from cluster start. Timeout at 2026-03-29 01:09:17.985443109 +0000 UTC m=+101.773585790, 105.017687ms after stability
2026-03-29T01:09:17.986Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:18.092Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:17.986594562 +0000 UTC m=+101.774737199 (1 nodes), 1.45µs from start of poll, 596.815371ms from cluster start. Timeout at 2026-03-29 01:09:18.091606316 +0000 UTC m=+101.879748942, 105.011743ms after stability
2026-03-29T01:09:18.198Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:18.303Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:18.198181973 +0000 UTC m=+101.986324624 (1 nodes), 1.35µs from start of poll, 808.402796ms from cluster start. Timeout at 2026-03-29 01:09:18.303185626 +0000 UTC m=+102.091328167, 105.003543ms after stability
2026-03-29T01:09:18.303Z [INFO]  server-2172d1d6-5ff8-7087-5b8a-4261630fa3d3: aborting pipeline replication: peer="{Voter server-ce9eee01-008a-2665-d838-1eee9ad4c47f ce9eee01-008a-2665-d838-1eee9ad4c47f}"
2026-03-29T01:09:18.303Z [INFO]  server-2172d1d6-5ff8-7087-5b8a-4261630fa3d3: aborting pipeline replication: peer="{Voter server-20c0b0c8-d05b-6117-a1d0-e9735cc26225 20c0b0c8-d05b-6117-a1d0-e9735cc26225}"
--- PASS: TestRaft_LeaderID_Propagated (0.91s)
=== RUN   TestRaft_LeadershipTransferInProgress
--- PASS: TestRaft_LeadershipTransferInProgress (0.00s)
=== RUN   TestRaft_LeadershipTransferPickServer
--- PASS: TestRaft_LeadershipTransferPickServer (0.00s)
=== RUN   TestRaft_LeadershipTransfer
2026-03-29T01:09:18.304Z [INFO]  server-a662184c-6892-49a7-207a-af9dda6cde11: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a662184c-6892-49a7-207a-af9dda6cde11 Address:a662184c-6892-49a7-207a-af9dda6cde11} {Suffrage:Voter ID:server-50e48055-e879-5f38-bf2c-559c32e4323b Address:50e48055-e879-5f38-bf2c-559c32e4323b} {Suffrage:Voter ID:server-2d91f510-1e37-ac21-77c0-bb1051de37fa Address:2d91f510-1e37-ac21-77c0-bb1051de37fa}]"
2026-03-29T01:09:18.304Z [INFO]  server-a662184c-6892-49a7-207a-af9dda6cde11: entering follower state: follower="Node at a662184c-6892-49a7-207a-af9dda6cde11 [Follower]" leader-address= leader-id=
2026-03-29T01:09:18.304Z [INFO]  server-50e48055-e879-5f38-bf2c-559c32e4323b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a662184c-6892-49a7-207a-af9dda6cde11 Address:a662184c-6892-49a7-207a-af9dda6cde11} {Suffrage:Voter ID:server-50e48055-e879-5f38-bf2c-559c32e4323b Address:50e48055-e879-5f38-bf2c-559c32e4323b} {Suffrage:Voter ID:server-2d91f510-1e37-ac21-77c0-bb1051de37fa Address:2d91f510-1e37-ac21-77c0-bb1051de37fa}]"
2026-03-29T01:09:18.304Z [INFO]  server-50e48055-e879-5f38-bf2c-559c32e4323b: entering follower state: follower="Node at 50e48055-e879-5f38-bf2c-559c32e4323b [Follower]" leader-address= leader-id=
2026-03-29T01:09:18.305Z [INFO]  server-2d91f510-1e37-ac21-77c0-bb1051de37fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a662184c-6892-49a7-207a-af9dda6cde11 Address:a662184c-6892-49a7-207a-af9dda6cde11} {Suffrage:Voter ID:server-50e48055-e879-5f38-bf2c-559c32e4323b Address:50e48055-e879-5f38-bf2c-559c32e4323b} {Suffrage:Voter ID:server-2d91f510-1e37-ac21-77c0-bb1051de37fa Address:2d91f510-1e37-ac21-77c0-bb1051de37fa}]"
2026-03-29T01:09:18.305Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:18.305Z [INFO]  server-2d91f510-1e37-ac21-77c0-bb1051de37fa: entering follower state: follower="Node at 2d91f510-1e37-ac21-77c0-bb1051de37fa [Follower]" leader-address= leader-id=
2026-03-29T01:09:18.364Z [WARN]  server-50e48055-e879-5f38-bf2c-559c32e4323b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:18.364Z [INFO]  server-50e48055-e879-5f38-bf2c-559c32e4323b: entering candidate state: node="Node at 50e48055-e879-5f38-bf2c-559c32e4323b [Candidate]" term=2
2026-03-29T01:09:18.364Z [INFO]  server-50e48055-e879-5f38-bf2c-559c32e4323b: election won: term=2 tally=2
2026-03-29T01:09:18.364Z [INFO]  server-50e48055-e879-5f38-bf2c-559c32e4323b: entering leader state: leader="Node at 50e48055-e879-5f38-bf2c-559c32e4323b [Leader]"
2026-03-29T01:09:18.364Z [INFO]  server-50e48055-e879-5f38-bf2c-559c32e4323b: added peer, starting replication: peer=server-a662184c-6892-49a7-207a-af9dda6cde11
2026-03-29T01:09:18.364Z [INFO]  server-50e48055-e879-5f38-bf2c-559c32e4323b: added peer, starting replication: peer=server-2d91f510-1e37-ac21-77c0-bb1051de37fa
2026-03-29T01:09:18.364Z [INFO]  server-50e48055-e879-5f38-bf2c-559c32e4323b: pipelining replication: peer="{Voter server-2d91f510-1e37-ac21-77c0-bb1051de37fa 2d91f510-1e37-ac21-77c0-bb1051de37fa}"
2026-03-29T01:09:18.364Z [INFO]  server-50e48055-e879-5f38-bf2c-559c32e4323b: pipelining replication: peer="{Voter server-a662184c-6892-49a7-207a-af9dda6cde11 a662184c-6892-49a7-207a-af9dda6cde11}"
2026-03-29T01:09:18.470Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:18.364481764 +0000 UTC m=+102.152624311 (1 nodes), 59.428887ms from start of poll, 59.746644ms from cluster start. Timeout at 2026-03-29 01:09:18.469485022 +0000 UTC m=+102.257627552, 105.003241ms after stability
2026-03-29T01:09:18.470Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:18.576Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:18.470676257 +0000 UTC m=+102.258818808 (1 nodes), 784ns from start of poll, 165.941141ms from cluster start. Timeout at 2026-03-29 01:09:18.575714863 +0000 UTC m=+102.363857506, 105.038698ms after stability
2026-03-29T01:09:18.576Z [INFO]  server-a662184c-6892-49a7-207a-af9dda6cde11: entering candidate state: node="Node at a662184c-6892-49a7-207a-af9dda6cde11 [Candidate]" term=3
2026-03-29T01:09:18.576Z [ERROR] server-50e48055-e879-5f38-bf2c-559c32e4323b: peer has newer term, stopping replication: peer="{Voter server-a662184c-6892-49a7-207a-af9dda6cde11 a662184c-6892-49a7-207a-af9dda6cde11}"
2026-03-29T01:09:18.576Z [INFO]  server-50e48055-e879-5f38-bf2c-559c32e4323b: aborting pipeline replication: peer="{Voter server-a662184c-6892-49a7-207a-af9dda6cde11 a662184c-6892-49a7-207a-af9dda6cde11}"
2026-03-29T01:09:18.576Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:18.576Z [INFO]  server-a662184c-6892-49a7-207a-af9dda6cde11: election won: term=3 tally=2
2026-03-29T01:09:18.577Z [INFO]  server-a662184c-6892-49a7-207a-af9dda6cde11: entering leader state: leader="Node at a662184c-6892-49a7-207a-af9dda6cde11 [Leader]"
2026-03-29T01:09:18.577Z [INFO]  server-a662184c-6892-49a7-207a-af9dda6cde11: added peer, starting replication: peer=server-50e48055-e879-5f38-bf2c-559c32e4323b
2026-03-29T01:09:18.577Z [INFO]  server-a662184c-6892-49a7-207a-af9dda6cde11: added peer, starting replication: peer=server-2d91f510-1e37-ac21-77c0-bb1051de37fa
2026-03-29T01:09:18.576Z [INFO]  server-50e48055-e879-5f38-bf2c-559c32e4323b: entering follower state: follower="Node at 50e48055-e879-5f38-bf2c-559c32e4323b [Follower]" leader-address= leader-id=
2026-03-29T01:09:18.577Z [INFO]  server-50e48055-e879-5f38-bf2c-559c32e4323b: aborting pipeline replication: peer="{Voter server-2d91f510-1e37-ac21-77c0-bb1051de37fa 2d91f510-1e37-ac21-77c0-bb1051de37fa}"
2026-03-29T01:09:18.577Z [INFO]  server-a662184c-6892-49a7-207a-af9dda6cde11: pipelining replication: peer="{Voter server-2d91f510-1e37-ac21-77c0-bb1051de37fa 2d91f510-1e37-ac21-77c0-bb1051de37fa}"
2026-03-29T01:09:18.577Z [INFO]  server-a662184c-6892-49a7-207a-af9dda6cde11: pipelining replication: peer="{Voter server-50e48055-e879-5f38-bf2c-559c32e4323b 50e48055-e879-5f38-bf2c-559c32e4323b}"
2026-03-29T01:09:18.682Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:18.577110653 +0000 UTC m=+102.365253216 (1 nodes), 113.342µs from start of poll, 272.375549ms from cluster start. Timeout at 2026-03-29 01:09:18.682138039 +0000 UTC m=+102.470280622, 105.027406ms after stability
2026-03-29T01:09:18.682Z [INFO]  server-a662184c-6892-49a7-207a-af9dda6cde11: aborting pipeline replication: peer="{Voter server-2d91f510-1e37-ac21-77c0-bb1051de37fa 2d91f510-1e37-ac21-77c0-bb1051de37fa}"
2026-03-29T01:09:18.682Z [INFO]  server-a662184c-6892-49a7-207a-af9dda6cde11: aborting pipeline replication: peer="{Voter server-50e48055-e879-5f38-bf2c-559c32e4323b 50e48055-e879-5f38-bf2c-559c32e4323b}"
--- PASS: TestRaft_LeadershipTransfer (0.38s)
=== RUN   TestRaft_LeadershipTransferWithOneNode
2026-03-29T01:09:18.683Z [INFO]  server-3abc3a05-b9f8-5939-982e-caca0ae93bac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3abc3a05-b9f8-5939-982e-caca0ae93bac Address:3abc3a05-b9f8-5939-982e-caca0ae93bac}]"
2026-03-29T01:09:18.683Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:18.683Z [INFO]  server-3abc3a05-b9f8-5939-982e-caca0ae93bac: entering follower state: follower="Node at 3abc3a05-b9f8-5939-982e-caca0ae93bac [Follower]" leader-address= leader-id=
2026-03-29T01:09:18.752Z [WARN]  server-3abc3a05-b9f8-5939-982e-caca0ae93bac: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:18.752Z [INFO]  server-3abc3a05-b9f8-5939-982e-caca0ae93bac: entering candidate state: node="Node at 3abc3a05-b9f8-5939-982e-caca0ae93bac [Candidate]" term=2
2026-03-29T01:09:18.753Z [INFO]  server-3abc3a05-b9f8-5939-982e-caca0ae93bac: election won: term=2 tally=1
2026-03-29T01:09:18.753Z [INFO]  server-3abc3a05-b9f8-5939-982e-caca0ae93bac: entering leader state: leader="Node at 3abc3a05-b9f8-5939-982e-caca0ae93bac [Leader]"
2026-03-29T01:09:18.858Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:18.753094285 +0000 UTC m=+102.541236819 (1 nodes), 69.81573ms from start of poll, 69.931614ms from cluster start. Timeout at 2026-03-29 01:09:18.858169486 +0000 UTC m=+102.646312015, 105.075196ms after stability
--- PASS: TestRaft_LeadershipTransferWithOneNode (0.18s)
=== RUN   TestRaft_LeadershipTransferWithSevenNodes
2026-03-29T01:09:18.859Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa Address:4ae6ca88-2b45-b194-b6f1-d82ef49889aa} {Suffrage:Voter ID:server-839b050a-88b4-e9de-18f7-4efc7e1ca13d Address:839b050a-88b4-e9de-18f7-4efc7e1ca13d} {Suffrage:Voter ID:server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830 Address:cfcd0bcb-5bb8-294f-7394-f9203dd2c830} {Suffrage:Voter ID:server-0daf1b32-f4cf-44d8-e919-b34cc0ffe981 Address:0daf1b32-f4cf-44d8-e919-b34cc0ffe981} {Suffrage:Voter ID:server-d7912813-a69b-d63b-6f3f-14e67d403cdc Address:d7912813-a69b-d63b-6f3f-14e67d403cdc} {Suffrage:Voter ID:server-463af937-f979-2e26-ab3a-202d57f7d813 Address:463af937-f979-2e26-ab3a-202d57f7d813} {Suffrage:Voter ID:server-6e091262-fff1-ac0a-f2ca-611c43eaee2e Address:6e091262-fff1-ac0a-f2ca-611c43eaee2e}]"
2026-03-29T01:09:18.859Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: entering follower state: follower="Node at 4ae6ca88-2b45-b194-b6f1-d82ef49889aa [Follower]" leader-address= leader-id=
2026-03-29T01:09:18.859Z [INFO]  server-839b050a-88b4-e9de-18f7-4efc7e1ca13d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa Address:4ae6ca88-2b45-b194-b6f1-d82ef49889aa} {Suffrage:Voter ID:server-839b050a-88b4-e9de-18f7-4efc7e1ca13d Address:839b050a-88b4-e9de-18f7-4efc7e1ca13d} {Suffrage:Voter ID:server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830 Address:cfcd0bcb-5bb8-294f-7394-f9203dd2c830} {Suffrage:Voter ID:server-0daf1b32-f4cf-44d8-e919-b34cc0ffe981 Address:0daf1b32-f4cf-44d8-e919-b34cc0ffe981} {Suffrage:Voter ID:server-d7912813-a69b-d63b-6f3f-14e67d403cdc Address:d7912813-a69b-d63b-6f3f-14e67d403cdc} {Suffrage:Voter ID:server-463af937-f979-2e26-ab3a-202d57f7d813 Address:463af937-f979-2e26-ab3a-202d57f7d813} {Suffrage:Voter ID:server-6e091262-fff1-ac0a-f2ca-611c43eaee2e Address:6e091262-fff1-ac0a-f2ca-611c43eaee2e}]"
2026-03-29T01:09:18.859Z [INFO]  server-839b050a-88b4-e9de-18f7-4efc7e1ca13d: entering follower state: follower="Node at 839b050a-88b4-e9de-18f7-4efc7e1ca13d [Follower]" leader-address= leader-id=
2026-03-29T01:09:18.859Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa Address:4ae6ca88-2b45-b194-b6f1-d82ef49889aa} {Suffrage:Voter ID:server-839b050a-88b4-e9de-18f7-4efc7e1ca13d Address:839b050a-88b4-e9de-18f7-4efc7e1ca13d} {Suffrage:Voter ID:server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830 Address:cfcd0bcb-5bb8-294f-7394-f9203dd2c830} {Suffrage:Voter ID:server-0daf1b32-f4cf-44d8-e919-b34cc0ffe981 Address:0daf1b32-f4cf-44d8-e919-b34cc0ffe981} {Suffrage:Voter ID:server-d7912813-a69b-d63b-6f3f-14e67d403cdc Address:d7912813-a69b-d63b-6f3f-14e67d403cdc} {Suffrage:Voter ID:server-463af937-f979-2e26-ab3a-202d57f7d813 Address:463af937-f979-2e26-ab3a-202d57f7d813} {Suffrage:Voter ID:server-6e091262-fff1-ac0a-f2ca-611c43eaee2e Address:6e091262-fff1-ac0a-f2ca-611c43eaee2e}]"
2026-03-29T01:09:18.859Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: entering follower state: follower="Node at cfcd0bcb-5bb8-294f-7394-f9203dd2c830 [Follower]" leader-address= leader-id=
2026-03-29T01:09:18.859Z [INFO]  server-0daf1b32-f4cf-44d8-e919-b34cc0ffe981: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa Address:4ae6ca88-2b45-b194-b6f1-d82ef49889aa} {Suffrage:Voter ID:server-839b050a-88b4-e9de-18f7-4efc7e1ca13d Address:839b050a-88b4-e9de-18f7-4efc7e1ca13d} {Suffrage:Voter ID:server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830 Address:cfcd0bcb-5bb8-294f-7394-f9203dd2c830} {Suffrage:Voter ID:server-0daf1b32-f4cf-44d8-e919-b34cc0ffe981 Address:0daf1b32-f4cf-44d8-e919-b34cc0ffe981} {Suffrage:Voter ID:server-d7912813-a69b-d63b-6f3f-14e67d403cdc Address:d7912813-a69b-d63b-6f3f-14e67d403cdc} {Suffrage:Voter ID:server-463af937-f979-2e26-ab3a-202d57f7d813 Address:463af937-f979-2e26-ab3a-202d57f7d813} {Suffrage:Voter ID:server-6e091262-fff1-ac0a-f2ca-611c43eaee2e Address:6e091262-fff1-ac0a-f2ca-611c43eaee2e}]"
2026-03-29T01:09:18.859Z [INFO]  server-0daf1b32-f4cf-44d8-e919-b34cc0ffe981: entering follower state: follower="Node at 0daf1b32-f4cf-44d8-e919-b34cc0ffe981 [Follower]" leader-address= leader-id=
2026-03-29T01:09:18.859Z [INFO]  server-d7912813-a69b-d63b-6f3f-14e67d403cdc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa Address:4ae6ca88-2b45-b194-b6f1-d82ef49889aa} {Suffrage:Voter ID:server-839b050a-88b4-e9de-18f7-4efc7e1ca13d Address:839b050a-88b4-e9de-18f7-4efc7e1ca13d} {Suffrage:Voter ID:server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830 Address:cfcd0bcb-5bb8-294f-7394-f9203dd2c830} {Suffrage:Voter ID:server-0daf1b32-f4cf-44d8-e919-b34cc0ffe981 Address:0daf1b32-f4cf-44d8-e919-b34cc0ffe981} {Suffrage:Voter ID:server-d7912813-a69b-d63b-6f3f-14e67d403cdc Address:d7912813-a69b-d63b-6f3f-14e67d403cdc} {Suffrage:Voter ID:server-463af937-f979-2e26-ab3a-202d57f7d813 Address:463af937-f979-2e26-ab3a-202d57f7d813} {Suffrage:Voter ID:server-6e091262-fff1-ac0a-f2ca-611c43eaee2e Address:6e091262-fff1-ac0a-f2ca-611c43eaee2e}]"
2026-03-29T01:09:18.859Z [INFO]  server-d7912813-a69b-d63b-6f3f-14e67d403cdc: entering follower state: follower="Node at d7912813-a69b-d63b-6f3f-14e67d403cdc [Follower]" leader-address= leader-id=
2026-03-29T01:09:18.860Z [INFO]  server-463af937-f979-2e26-ab3a-202d57f7d813: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa Address:4ae6ca88-2b45-b194-b6f1-d82ef49889aa} {Suffrage:Voter ID:server-839b050a-88b4-e9de-18f7-4efc7e1ca13d Address:839b050a-88b4-e9de-18f7-4efc7e1ca13d} {Suffrage:Voter ID:server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830 Address:cfcd0bcb-5bb8-294f-7394-f9203dd2c830} {Suffrage:Voter ID:server-0daf1b32-f4cf-44d8-e919-b34cc0ffe981 Address:0daf1b32-f4cf-44d8-e919-b34cc0ffe981} {Suffrage:Voter ID:server-d7912813-a69b-d63b-6f3f-14e67d403cdc Address:d7912813-a69b-d63b-6f3f-14e67d403cdc} {Suffrage:Voter ID:server-463af937-f979-2e26-ab3a-202d57f7d813 Address:463af937-f979-2e26-ab3a-202d57f7d813} {Suffrage:Voter ID:server-6e091262-fff1-ac0a-f2ca-611c43eaee2e Address:6e091262-fff1-ac0a-f2ca-611c43eaee2e}]"
2026-03-29T01:09:18.860Z [INFO]  server-463af937-f979-2e26-ab3a-202d57f7d813: entering follower state: follower="Node at 463af937-f979-2e26-ab3a-202d57f7d813 [Follower]" leader-address= leader-id=
2026-03-29T01:09:18.860Z [INFO]  server-6e091262-fff1-ac0a-f2ca-611c43eaee2e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa Address:4ae6ca88-2b45-b194-b6f1-d82ef49889aa} {Suffrage:Voter ID:server-839b050a-88b4-e9de-18f7-4efc7e1ca13d Address:839b050a-88b4-e9de-18f7-4efc7e1ca13d} {Suffrage:Voter ID:server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830 Address:cfcd0bcb-5bb8-294f-7394-f9203dd2c830} {Suffrage:Voter ID:server-0daf1b32-f4cf-44d8-e919-b34cc0ffe981 Address:0daf1b32-f4cf-44d8-e919-b34cc0ffe981} {Suffrage:Voter ID:server-d7912813-a69b-d63b-6f3f-14e67d403cdc Address:d7912813-a69b-d63b-6f3f-14e67d403cdc} {Suffrage:Voter ID:server-463af937-f979-2e26-ab3a-202d57f7d813 Address:463af937-f979-2e26-ab3a-202d57f7d813} {Suffrage:Voter ID:server-6e091262-fff1-ac0a-f2ca-611c43eaee2e Address:6e091262-fff1-ac0a-f2ca-611c43eaee2e}]"
2026-03-29T01:09:18.860Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:18.860Z [INFO]  server-6e091262-fff1-ac0a-f2ca-611c43eaee2e: entering follower state: follower="Node at 6e091262-fff1-ac0a-f2ca-611c43eaee2e [Follower]" leader-address= leader-id=
2026-03-29T01:09:18.923Z [WARN]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:18.923Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: entering candidate state: node="Node at cfcd0bcb-5bb8-294f-7394-f9203dd2c830 [Candidate]" term=2
2026-03-29T01:09:18.923Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: election won: term=2 tally=4
2026-03-29T01:09:18.923Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: entering leader state: leader="Node at cfcd0bcb-5bb8-294f-7394-f9203dd2c830 [Leader]"
2026-03-29T01:09:18.923Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: added peer, starting replication: peer=server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa
2026-03-29T01:09:18.923Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: added peer, starting replication: peer=server-839b050a-88b4-e9de-18f7-4efc7e1ca13d
2026-03-29T01:09:18.923Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: added peer, starting replication: peer=server-0daf1b32-f4cf-44d8-e919-b34cc0ffe981
2026-03-29T01:09:18.923Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: added peer, starting replication: peer=server-d7912813-a69b-d63b-6f3f-14e67d403cdc
2026-03-29T01:09:18.923Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: added peer, starting replication: peer=server-463af937-f979-2e26-ab3a-202d57f7d813
2026-03-29T01:09:18.923Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: added peer, starting replication: peer=server-6e091262-fff1-ac0a-f2ca-611c43eaee2e
2026-03-29T01:09:18.923Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: pipelining replication: peer="{Voter server-0daf1b32-f4cf-44d8-e919-b34cc0ffe981 0daf1b32-f4cf-44d8-e919-b34cc0ffe981}"
2026-03-29T01:09:18.923Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: pipelining replication: peer="{Voter server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa 4ae6ca88-2b45-b194-b6f1-d82ef49889aa}"
2026-03-29T01:09:18.923Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: pipelining replication: peer="{Voter server-463af937-f979-2e26-ab3a-202d57f7d813 463af937-f979-2e26-ab3a-202d57f7d813}"
2026-03-29T01:09:18.923Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: pipelining replication: peer="{Voter server-d7912813-a69b-d63b-6f3f-14e67d403cdc d7912813-a69b-d63b-6f3f-14e67d403cdc}"
2026-03-29T01:09:18.923Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: pipelining replication: peer="{Voter server-6e091262-fff1-ac0a-f2ca-611c43eaee2e 6e091262-fff1-ac0a-f2ca-611c43eaee2e}"
2026-03-29T01:09:18.923Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: pipelining replication: peer="{Voter server-839b050a-88b4-e9de-18f7-4efc7e1ca13d 839b050a-88b4-e9de-18f7-4efc7e1ca13d}"
2026-03-29T01:09:19.028Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:18.923446222 +0000 UTC m=+102.711588760 (1 nodes), 63.204256ms from start of poll, 64.118247ms from cluster start. Timeout at 2026-03-29 01:09:19.028453913 +0000 UTC m=+102.816596442, 105.007682ms after stability
2026-03-29T01:09:19.028Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:19.133Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:19.028597264 +0000 UTC m=+102.816739805 (6 nodes), 1.584µs from start of poll, 169.269292ms from cluster start. Timeout at 2026-03-29 01:09:19.133629503 +0000 UTC m=+102.921772055, 105.03225ms after stability
2026-03-29T01:09:19.133Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:19.239Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:19.133837547 +0000 UTC m=+102.921980091 (1 nodes), 764ns from start of poll, 274.509578ms from cluster start. Timeout at 2026-03-29 01:09:19.238857541 +0000 UTC m=+103.027000204, 105.020113ms after stability
2026-03-29T01:09:19.239Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: entering candidate state: node="Node at 4ae6ca88-2b45-b194-b6f1-d82ef49889aa [Candidate]" term=3
2026-03-29T01:09:19.239Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:19.239Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: election won: term=3 tally=4
2026-03-29T01:09:19.239Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: entering leader state: leader="Node at 4ae6ca88-2b45-b194-b6f1-d82ef49889aa [Leader]"
2026-03-29T01:09:19.239Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: entering follower state: follower="Node at cfcd0bcb-5bb8-294f-7394-f9203dd2c830 [Follower]" leader-address= leader-id=
2026-03-29T01:09:19.239Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: added peer, starting replication: peer=server-839b050a-88b4-e9de-18f7-4efc7e1ca13d
2026-03-29T01:09:19.239Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: aborting pipeline replication: peer="{Voter server-d7912813-a69b-d63b-6f3f-14e67d403cdc d7912813-a69b-d63b-6f3f-14e67d403cdc}"
2026-03-29T01:09:19.239Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: added peer, starting replication: peer=server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830
2026-03-29T01:09:19.239Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: aborting pipeline replication: peer="{Voter server-0daf1b32-f4cf-44d8-e919-b34cc0ffe981 0daf1b32-f4cf-44d8-e919-b34cc0ffe981}"
2026-03-29T01:09:19.239Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: added peer, starting replication: peer=server-0daf1b32-f4cf-44d8-e919-b34cc0ffe981
2026-03-29T01:09:19.239Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: added peer, starting replication: peer=server-d7912813-a69b-d63b-6f3f-14e67d403cdc
2026-03-29T01:09:19.239Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: added peer, starting replication: peer=server-463af937-f979-2e26-ab3a-202d57f7d813
2026-03-29T01:09:19.239Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: added peer, starting replication: peer=server-6e091262-fff1-ac0a-f2ca-611c43eaee2e
2026-03-29T01:09:19.239Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: aborting pipeline replication: peer="{Voter server-839b050a-88b4-e9de-18f7-4efc7e1ca13d 839b050a-88b4-e9de-18f7-4efc7e1ca13d}"
2026-03-29T01:09:19.239Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: pipelining replication: peer="{Voter server-839b050a-88b4-e9de-18f7-4efc7e1ca13d 839b050a-88b4-e9de-18f7-4efc7e1ca13d}"
2026-03-29T01:09:19.239Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: aborting pipeline replication: peer="{Voter server-463af937-f979-2e26-ab3a-202d57f7d813 463af937-f979-2e26-ab3a-202d57f7d813}"
2026-03-29T01:09:19.239Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: pipelining replication: peer="{Voter server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830 cfcd0bcb-5bb8-294f-7394-f9203dd2c830}"
2026-03-29T01:09:19.239Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: aborting pipeline replication: peer="{Voter server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa 4ae6ca88-2b45-b194-b6f1-d82ef49889aa}"
2026-03-29T01:09:19.239Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: pipelining replication: peer="{Voter server-463af937-f979-2e26-ab3a-202d57f7d813 463af937-f979-2e26-ab3a-202d57f7d813}"
2026-03-29T01:09:19.239Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: pipelining replication: peer="{Voter server-d7912813-a69b-d63b-6f3f-14e67d403cdc d7912813-a69b-d63b-6f3f-14e67d403cdc}"
2026-03-29T01:09:19.239Z [INFO]  server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830: aborting pipeline replication: peer="{Voter server-6e091262-fff1-ac0a-f2ca-611c43eaee2e 6e091262-fff1-ac0a-f2ca-611c43eaee2e}"
2026-03-29T01:09:19.239Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: pipelining replication: peer="{Voter server-6e091262-fff1-ac0a-f2ca-611c43eaee2e 6e091262-fff1-ac0a-f2ca-611c43eaee2e}"
2026-03-29T01:09:19.239Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: pipelining replication: peer="{Voter server-0daf1b32-f4cf-44d8-e919-b34cc0ffe981 0daf1b32-f4cf-44d8-e919-b34cc0ffe981}"
2026-03-29T01:09:19.345Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:19.239760896 +0000 UTC m=+103.027903430 (1 nodes), 54.713µs from start of poll, 380.432917ms from cluster start. Timeout at 2026-03-29 01:09:19.344778614 +0000 UTC m=+103.132921144, 105.017714ms after stability
2026-03-29T01:09:19.345Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: aborting pipeline replication: peer="{Voter server-839b050a-88b4-e9de-18f7-4efc7e1ca13d 839b050a-88b4-e9de-18f7-4efc7e1ca13d}"
2026-03-29T01:09:19.345Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: aborting pipeline replication: peer="{Voter server-cfcd0bcb-5bb8-294f-7394-f9203dd2c830 cfcd0bcb-5bb8-294f-7394-f9203dd2c830}"
2026-03-29T01:09:19.345Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: aborting pipeline replication: peer="{Voter server-463af937-f979-2e26-ab3a-202d57f7d813 463af937-f979-2e26-ab3a-202d57f7d813}"
2026-03-29T01:09:19.345Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: aborting pipeline replication: peer="{Voter server-6e091262-fff1-ac0a-f2ca-611c43eaee2e 6e091262-fff1-ac0a-f2ca-611c43eaee2e}"
2026-03-29T01:09:19.345Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: aborting pipeline replication: peer="{Voter server-0daf1b32-f4cf-44d8-e919-b34cc0ffe981 0daf1b32-f4cf-44d8-e919-b34cc0ffe981}"
2026-03-29T01:09:19.345Z [INFO]  server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: aborting pipeline replication: peer="{Voter server-d7912813-a69b-d63b-6f3f-14e67d403cdc d7912813-a69b-d63b-6f3f-14e67d403cdc}"
2026-03-29T01:09:19.845Z [ERROR] server-4ae6ca88-2b45-b194-b6f1-d82ef49889aa: failed to heartbeat to: peer=463af937-f979-2e26-ab3a-202d57f7d813 backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.99s)
=== RUN   TestRaft_LeadershipTransferToInvalidID
2026-03-29T01:09:19.850Z [INFO]  server-a1d046c7-3a55-bfe4-d373-c0c5774e4cb5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1d046c7-3a55-bfe4-d373-c0c5774e4cb5 Address:a1d046c7-3a55-bfe4-d373-c0c5774e4cb5} {Suffrage:Voter ID:server-c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7 Address:c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7} {Suffrage:Voter ID:server-5f1f7769-de91-1c97-3e1c-b1489c9682de Address:5f1f7769-de91-1c97-3e1c-b1489c9682de}]"
2026-03-29T01:09:19.850Z [INFO]  server-a1d046c7-3a55-bfe4-d373-c0c5774e4cb5: entering follower state: follower="Node at a1d046c7-3a55-bfe4-d373-c0c5774e4cb5 [Follower]" leader-address= leader-id=
2026-03-29T01:09:19.851Z [INFO]  server-c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1d046c7-3a55-bfe4-d373-c0c5774e4cb5 Address:a1d046c7-3a55-bfe4-d373-c0c5774e4cb5} {Suffrage:Voter ID:server-c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7 Address:c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7} {Suffrage:Voter ID:server-5f1f7769-de91-1c97-3e1c-b1489c9682de Address:5f1f7769-de91-1c97-3e1c-b1489c9682de}]"
2026-03-29T01:09:19.851Z [INFO]  server-5f1f7769-de91-1c97-3e1c-b1489c9682de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1d046c7-3a55-bfe4-d373-c0c5774e4cb5 Address:a1d046c7-3a55-bfe4-d373-c0c5774e4cb5} {Suffrage:Voter ID:server-c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7 Address:c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7} {Suffrage:Voter ID:server-5f1f7769-de91-1c97-3e1c-b1489c9682de Address:5f1f7769-de91-1c97-3e1c-b1489c9682de}]"
2026-03-29T01:09:19.851Z [INFO]  server-c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7: entering follower state: follower="Node at c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7 [Follower]" leader-address= leader-id=
2026-03-29T01:09:19.851Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:19.851Z [INFO]  server-5f1f7769-de91-1c97-3e1c-b1489c9682de: entering follower state: follower="Node at 5f1f7769-de91-1c97-3e1c-b1489c9682de [Follower]" leader-address= leader-id=
2026-03-29T01:09:19.905Z [WARN]  server-5f1f7769-de91-1c97-3e1c-b1489c9682de: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:19.905Z [INFO]  server-5f1f7769-de91-1c97-3e1c-b1489c9682de: entering candidate state: node="Node at 5f1f7769-de91-1c97-3e1c-b1489c9682de [Candidate]" term=2
2026-03-29T01:09:19.905Z [WARN]  server-c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:19.905Z [INFO]  server-c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7: entering candidate state: node="Node at c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7 [Candidate]" term=2
2026-03-29T01:09:19.905Z [INFO]  server-5f1f7769-de91-1c97-3e1c-b1489c9682de: election won: term=2 tally=2
2026-03-29T01:09:19.905Z [INFO]  server-c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7: duplicate requestVote for same term: term=2
2026-03-29T01:09:19.905Z [INFO]  server-a1d046c7-3a55-bfe4-d373-c0c5774e4cb5: duplicate requestVote for same term: term=2
2026-03-29T01:09:19.905Z [INFO]  server-5f1f7769-de91-1c97-3e1c-b1489c9682de: entering leader state: leader="Node at 5f1f7769-de91-1c97-3e1c-b1489c9682de [Leader]"
2026-03-29T01:09:19.905Z [INFO]  server-5f1f7769-de91-1c97-3e1c-b1489c9682de: added peer, starting replication: peer=server-a1d046c7-3a55-bfe4-d373-c0c5774e4cb5
2026-03-29T01:09:19.905Z [INFO]  server-5f1f7769-de91-1c97-3e1c-b1489c9682de: added peer, starting replication: peer=server-c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7
2026-03-29T01:09:19.905Z [WARN]  server-5f1f7769-de91-1c97-3e1c-b1489c9682de: rejecting vote request since we have a leader: from=c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7 leader=5f1f7769-de91-1c97-3e1c-b1489c9682de leader-id=server-5f1f7769-de91-1c97-3e1c-b1489c9682de
2026-03-29T01:09:19.906Z [INFO]  server-5f1f7769-de91-1c97-3e1c-b1489c9682de: pipelining replication: peer="{Voter server-a1d046c7-3a55-bfe4-d373-c0c5774e4cb5 a1d046c7-3a55-bfe4-d373-c0c5774e4cb5}"
2026-03-29T01:09:19.906Z [INFO]  server-c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7: entering follower state: follower="Node at c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7 [Follower]" leader-address=5f1f7769-de91-1c97-3e1c-b1489c9682de leader-id=server-5f1f7769-de91-1c97-3e1c-b1489c9682de
2026-03-29T01:09:19.906Z [INFO]  server-5f1f7769-de91-1c97-3e1c-b1489c9682de: pipelining replication: peer="{Voter server-c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7 c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7}"
2026-03-29T01:09:20.011Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:19.906091422 +0000 UTC m=+103.694233962 (1 nodes), 54.918387ms from start of poll, 55.267225ms from cluster start. Timeout at 2026-03-29 01:09:20.011099357 +0000 UTC m=+103.799241885, 105.007923ms after stability
2026-03-29T01:09:20.011Z [INFO]  server-5f1f7769-de91-1c97-3e1c-b1489c9682de: aborting pipeline replication: peer="{Voter server-c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7 c92d165b-1752-4d6b-f2ba-4e3db4ce8cd7}"
2026-03-29T01:09:20.011Z [INFO]  server-5f1f7769-de91-1c97-3e1c-b1489c9682de: aborting pipeline replication: peer="{Voter server-a1d046c7-3a55-bfe4-d373-c0c5774e4cb5 a1d046c7-3a55-bfe4-d373-c0c5774e4cb5}"
--- PASS: TestRaft_LeadershipTransferToInvalidID (0.16s)
=== RUN   TestRaft_LeadershipTransferToInvalidAddress
2026-03-29T01:09:20.012Z [INFO]  server-a6af283b-3b10-7930-23d3-6c55174d850c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6af283b-3b10-7930-23d3-6c55174d850c Address:a6af283b-3b10-7930-23d3-6c55174d850c} {Suffrage:Voter ID:server-24333412-d96a-0b1e-cc23-123a75d627fc Address:24333412-d96a-0b1e-cc23-123a75d627fc} {Suffrage:Voter ID:server-ca5ace20-640f-718b-8226-21ad31af047b Address:ca5ace20-640f-718b-8226-21ad31af047b}]"
2026-03-29T01:09:20.012Z [INFO]  server-a6af283b-3b10-7930-23d3-6c55174d850c: entering follower state: follower="Node at a6af283b-3b10-7930-23d3-6c55174d850c [Follower]" leader-address= leader-id=
2026-03-29T01:09:20.012Z [INFO]  server-24333412-d96a-0b1e-cc23-123a75d627fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6af283b-3b10-7930-23d3-6c55174d850c Address:a6af283b-3b10-7930-23d3-6c55174d850c} {Suffrage:Voter ID:server-24333412-d96a-0b1e-cc23-123a75d627fc Address:24333412-d96a-0b1e-cc23-123a75d627fc} {Suffrage:Voter ID:server-ca5ace20-640f-718b-8226-21ad31af047b Address:ca5ace20-640f-718b-8226-21ad31af047b}]"
2026-03-29T01:09:20.012Z [INFO]  server-24333412-d96a-0b1e-cc23-123a75d627fc: entering follower state: follower="Node at 24333412-d96a-0b1e-cc23-123a75d627fc [Follower]" leader-address= leader-id=
2026-03-29T01:09:20.012Z [INFO]  server-ca5ace20-640f-718b-8226-21ad31af047b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6af283b-3b10-7930-23d3-6c55174d850c Address:a6af283b-3b10-7930-23d3-6c55174d850c} {Suffrage:Voter ID:server-24333412-d96a-0b1e-cc23-123a75d627fc Address:24333412-d96a-0b1e-cc23-123a75d627fc} {Suffrage:Voter ID:server-ca5ace20-640f-718b-8226-21ad31af047b Address:ca5ace20-640f-718b-8226-21ad31af047b}]"
2026-03-29T01:09:20.012Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:20.012Z [INFO]  server-ca5ace20-640f-718b-8226-21ad31af047b: entering follower state: follower="Node at ca5ace20-640f-718b-8226-21ad31af047b [Follower]" leader-address= leader-id=
2026-03-29T01:09:20.063Z [WARN]  server-ca5ace20-640f-718b-8226-21ad31af047b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:20.063Z [INFO]  server-ca5ace20-640f-718b-8226-21ad31af047b: entering candidate state: node="Node at ca5ace20-640f-718b-8226-21ad31af047b [Candidate]" term=2
2026-03-29T01:09:20.063Z [INFO]  server-ca5ace20-640f-718b-8226-21ad31af047b: election won: term=2 tally=2
2026-03-29T01:09:20.063Z [INFO]  server-ca5ace20-640f-718b-8226-21ad31af047b: entering leader state: leader="Node at ca5ace20-640f-718b-8226-21ad31af047b [Leader]"
2026-03-29T01:09:20.063Z [INFO]  server-ca5ace20-640f-718b-8226-21ad31af047b: added peer, starting replication: peer=server-a6af283b-3b10-7930-23d3-6c55174d850c
2026-03-29T01:09:20.063Z [INFO]  server-ca5ace20-640f-718b-8226-21ad31af047b: added peer, starting replication: peer=server-24333412-d96a-0b1e-cc23-123a75d627fc
2026-03-29T01:09:20.064Z [INFO]  server-ca5ace20-640f-718b-8226-21ad31af047b: pipelining replication: peer="{Voter server-24333412-d96a-0b1e-cc23-123a75d627fc 24333412-d96a-0b1e-cc23-123a75d627fc}"
2026-03-29T01:09:20.064Z [INFO]  server-ca5ace20-640f-718b-8226-21ad31af047b: pipelining replication: peer="{Voter server-a6af283b-3b10-7930-23d3-6c55174d850c a6af283b-3b10-7930-23d3-6c55174d850c}"
2026-03-29T01:09:20.169Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:20.063604119 +0000 UTC m=+103.851746656 (2 nodes), 50.787788ms from start of poll, 51.082609ms from cluster start. Timeout at 2026-03-29 01:09:20.168610351 +0000 UTC m=+103.956752942, 105.006286ms after stability
2026-03-29T01:09:20.169Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:20.275Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:20.16940101 +0000 UTC m=+103.957543634 (1 nodes), 1.263µs from start of poll, 156.879587ms from cluster start. Timeout at 2026-03-29 01:09:20.274475318 +0000 UTC m=+104.062617863, 105.074229ms after stability
2026-03-29T01:09:20.275Z [INFO]  server-ca5ace20-640f-718b-8226-21ad31af047b: aborting pipeline replication: peer="{Voter server-a6af283b-3b10-7930-23d3-6c55174d850c a6af283b-3b10-7930-23d3-6c55174d850c}"
2026-03-29T01:09:20.275Z [INFO]  server-ca5ace20-640f-718b-8226-21ad31af047b: aborting pipeline replication: peer="{Voter server-24333412-d96a-0b1e-cc23-123a75d627fc 24333412-d96a-0b1e-cc23-123a75d627fc}"
--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.26s)
=== RUN   TestRaft_LeadershipTransferToBehindServer
2026-03-29T01:09:20.276Z [INFO]  server-c1c2a8ea-5b35-1472-8254-638440faef0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1c2a8ea-5b35-1472-8254-638440faef0d Address:c1c2a8ea-5b35-1472-8254-638440faef0d} {Suffrage:Voter ID:server-94527806-5a3d-3662-3259-92dd9670c134 Address:94527806-5a3d-3662-3259-92dd9670c134} {Suffrage:Voter ID:server-8208dd7a-77da-6ea0-11ff-8ccc52afd1b5 Address:8208dd7a-77da-6ea0-11ff-8ccc52afd1b5}]"
2026-03-29T01:09:20.276Z [INFO]  server-c1c2a8ea-5b35-1472-8254-638440faef0d: entering follower state: follower="Node at c1c2a8ea-5b35-1472-8254-638440faef0d [Follower]" leader-address= leader-id=
2026-03-29T01:09:20.276Z [INFO]  server-94527806-5a3d-3662-3259-92dd9670c134: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1c2a8ea-5b35-1472-8254-638440faef0d Address:c1c2a8ea-5b35-1472-8254-638440faef0d} {Suffrage:Voter ID:server-94527806-5a3d-3662-3259-92dd9670c134 Address:94527806-5a3d-3662-3259-92dd9670c134} {Suffrage:Voter ID:server-8208dd7a-77da-6ea0-11ff-8ccc52afd1b5 Address:8208dd7a-77da-6ea0-11ff-8ccc52afd1b5}]"
2026-03-29T01:09:20.276Z [INFO]  server-94527806-5a3d-3662-3259-92dd9670c134: entering follower state: follower="Node at 94527806-5a3d-3662-3259-92dd9670c134 [Follower]" leader-address= leader-id=
2026-03-29T01:09:20.276Z [INFO]  server-8208dd7a-77da-6ea0-11ff-8ccc52afd1b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1c2a8ea-5b35-1472-8254-638440faef0d Address:c1c2a8ea-5b35-1472-8254-638440faef0d} {Suffrage:Voter ID:server-94527806-5a3d-3662-3259-92dd9670c134 Address:94527806-5a3d-3662-3259-92dd9670c134} {Suffrage:Voter ID:server-8208dd7a-77da-6ea0-11ff-8ccc52afd1b5 Address:8208dd7a-77da-6ea0-11ff-8ccc52afd1b5}]"
2026-03-29T01:09:20.276Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:20.276Z [INFO]  server-8208dd7a-77da-6ea0-11ff-8ccc52afd1b5: entering follower state: follower="Node at 8208dd7a-77da-6ea0-11ff-8ccc52afd1b5 [Follower]" leader-address= leader-id=
2026-03-29T01:09:20.336Z [WARN]  server-94527806-5a3d-3662-3259-92dd9670c134: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:20.336Z [INFO]  server-94527806-5a3d-3662-3259-92dd9670c134: entering candidate state: node="Node at 94527806-5a3d-3662-3259-92dd9670c134 [Candidate]" term=2
2026-03-29T01:09:20.336Z [INFO]  server-94527806-5a3d-3662-3259-92dd9670c134: election won: term=2 tally=2
2026-03-29T01:09:20.336Z [INFO]  server-94527806-5a3d-3662-3259-92dd9670c134: entering leader state: leader="Node at 94527806-5a3d-3662-3259-92dd9670c134 [Leader]"
2026-03-29T01:09:20.336Z [INFO]  server-94527806-5a3d-3662-3259-92dd9670c134: added peer, starting replication: peer=server-c1c2a8ea-5b35-1472-8254-638440faef0d
2026-03-29T01:09:20.336Z [INFO]  server-94527806-5a3d-3662-3259-92dd9670c134: added peer, starting replication: peer=server-8208dd7a-77da-6ea0-11ff-8ccc52afd1b5
2026-03-29T01:09:20.336Z [INFO]  server-94527806-5a3d-3662-3259-92dd9670c134: pipelining replication: peer="{Voter server-8208dd7a-77da-6ea0-11ff-8ccc52afd1b5 8208dd7a-77da-6ea0-11ff-8ccc52afd1b5}"
2026-03-29T01:09:20.336Z [INFO]  server-94527806-5a3d-3662-3259-92dd9670c134: pipelining replication: peer="{Voter server-c1c2a8ea-5b35-1472-8254-638440faef0d c1c2a8ea-5b35-1472-8254-638440faef0d}"
2026-03-29T01:09:20.442Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:20.336554421 +0000 UTC m=+104.124696960 (1 nodes), 60.311091ms from start of poll, 60.526519ms from cluster start. Timeout at 2026-03-29 01:09:20.441560606 +0000 UTC m=+104.229703133, 105.006173ms after stability
2026-03-29T01:09:20.442Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:20.548Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:20.442566978 +0000 UTC m=+104.230709512 (2 nodes), 663ns from start of poll, 166.539071ms from cluster start. Timeout at 2026-03-29 01:09:20.547572371 +0000 UTC m=+104.335714922, 105.00541ms after stability
2026-03-29T01:09:20.591Z [INFO]  server-c1c2a8ea-5b35-1472-8254-638440faef0d: entering candidate state: node="Node at c1c2a8ea-5b35-1472-8254-638440faef0d [Candidate]" term=3
2026-03-29T01:09:20.591Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:20.591Z [INFO]  server-c1c2a8ea-5b35-1472-8254-638440faef0d: election won: term=3 tally=2
2026-03-29T01:09:20.591Z [INFO]  server-c1c2a8ea-5b35-1472-8254-638440faef0d: entering leader state: leader="Node at c1c2a8ea-5b35-1472-8254-638440faef0d [Leader]"
2026-03-29T01:09:20.591Z [INFO]  server-94527806-5a3d-3662-3259-92dd9670c134: entering follower state: follower="Node at 94527806-5a3d-3662-3259-92dd9670c134 [Follower]" leader-address= leader-id=
2026-03-29T01:09:20.591Z [INFO]  server-94527806-5a3d-3662-3259-92dd9670c134: aborting pipeline replication: peer="{Voter server-c1c2a8ea-5b35-1472-8254-638440faef0d c1c2a8ea-5b35-1472-8254-638440faef0d}"
2026-03-29T01:09:20.591Z [INFO]  server-94527806-5a3d-3662-3259-92dd9670c134: aborting pipeline replication: peer="{Voter server-8208dd7a-77da-6ea0-11ff-8ccc52afd1b5 8208dd7a-77da-6ea0-11ff-8ccc52afd1b5}"
2026-03-29T01:09:20.591Z [INFO]  server-c1c2a8ea-5b35-1472-8254-638440faef0d: added peer, starting replication: peer=server-94527806-5a3d-3662-3259-92dd9670c134
2026-03-29T01:09:20.591Z [INFO]  server-c1c2a8ea-5b35-1472-8254-638440faef0d: added peer, starting replication: peer=server-8208dd7a-77da-6ea0-11ff-8ccc52afd1b5
2026-03-29T01:09:20.591Z [INFO]  server-c1c2a8ea-5b35-1472-8254-638440faef0d: pipelining replication: peer="{Voter server-94527806-5a3d-3662-3259-92dd9670c134 94527806-5a3d-3662-3259-92dd9670c134}"
2026-03-29T01:09:20.592Z [INFO]  server-c1c2a8ea-5b35-1472-8254-638440faef0d: pipelining replication: peer="{Voter server-8208dd7a-77da-6ea0-11ff-8ccc52afd1b5 8208dd7a-77da-6ea0-11ff-8ccc52afd1b5}"
2026-03-29T01:09:20.696Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:20.59183589 +0000 UTC m=+104.379978425 (1 nodes), 479.188µs from start of poll, 315.807984ms from cluster start. Timeout at 2026-03-29 01:09:20.696839592 +0000 UTC m=+104.484982129, 105.003704ms after stability
2026-03-29T01:09:20.696Z [INFO]  server-c1c2a8ea-5b35-1472-8254-638440faef0d: aborting pipeline replication: peer="{Voter server-8208dd7a-77da-6ea0-11ff-8ccc52afd1b5 8208dd7a-77da-6ea0-11ff-8ccc52afd1b5}"
2026-03-29T01:09:20.696Z [INFO]  server-c1c2a8ea-5b35-1472-8254-638440faef0d: aborting pipeline replication: peer="{Voter server-94527806-5a3d-3662-3259-92dd9670c134 94527806-5a3d-3662-3259-92dd9670c134}"
--- PASS: TestRaft_LeadershipTransferToBehindServer (0.42s)
=== RUN   TestRaft_LeadershipTransferToItself
2026-03-29T01:09:20.697Z [INFO]  server-6d956970-1625-7d07-962c-0215dd8f0109: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d956970-1625-7d07-962c-0215dd8f0109 Address:6d956970-1625-7d07-962c-0215dd8f0109} {Suffrage:Voter ID:server-4ebabacb-094a-6236-f1b8-68703f47534d Address:4ebabacb-094a-6236-f1b8-68703f47534d} {Suffrage:Voter ID:server-1b784e08-f6dd-0184-65e7-6de5497aa119 Address:1b784e08-f6dd-0184-65e7-6de5497aa119}]"
2026-03-29T01:09:20.697Z [INFO]  server-6d956970-1625-7d07-962c-0215dd8f0109: entering follower state: follower="Node at 6d956970-1625-7d07-962c-0215dd8f0109 [Follower]" leader-address= leader-id=
2026-03-29T01:09:20.697Z [INFO]  server-4ebabacb-094a-6236-f1b8-68703f47534d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d956970-1625-7d07-962c-0215dd8f0109 Address:6d956970-1625-7d07-962c-0215dd8f0109} {Suffrage:Voter ID:server-4ebabacb-094a-6236-f1b8-68703f47534d Address:4ebabacb-094a-6236-f1b8-68703f47534d} {Suffrage:Voter ID:server-1b784e08-f6dd-0184-65e7-6de5497aa119 Address:1b784e08-f6dd-0184-65e7-6de5497aa119}]"
2026-03-29T01:09:20.697Z [INFO]  server-4ebabacb-094a-6236-f1b8-68703f47534d: entering follower state: follower="Node at 4ebabacb-094a-6236-f1b8-68703f47534d [Follower]" leader-address= leader-id=
2026-03-29T01:09:20.697Z [INFO]  server-1b784e08-f6dd-0184-65e7-6de5497aa119: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d956970-1625-7d07-962c-0215dd8f0109 Address:6d956970-1625-7d07-962c-0215dd8f0109} {Suffrage:Voter ID:server-4ebabacb-094a-6236-f1b8-68703f47534d Address:4ebabacb-094a-6236-f1b8-68703f47534d} {Suffrage:Voter ID:server-1b784e08-f6dd-0184-65e7-6de5497aa119 Address:1b784e08-f6dd-0184-65e7-6de5497aa119}]"
2026-03-29T01:09:20.697Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:20.697Z [INFO]  server-1b784e08-f6dd-0184-65e7-6de5497aa119: entering follower state: follower="Node at 1b784e08-f6dd-0184-65e7-6de5497aa119 [Follower]" leader-address= leader-id=
2026-03-29T01:09:20.756Z [WARN]  server-6d956970-1625-7d07-962c-0215dd8f0109: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:20.756Z [INFO]  server-6d956970-1625-7d07-962c-0215dd8f0109: entering candidate state: node="Node at 6d956970-1625-7d07-962c-0215dd8f0109 [Candidate]" term=2
2026-03-29T01:09:20.756Z [INFO]  server-6d956970-1625-7d07-962c-0215dd8f0109: election won: term=2 tally=2
2026-03-29T01:09:20.756Z [INFO]  server-6d956970-1625-7d07-962c-0215dd8f0109: entering leader state: leader="Node at 6d956970-1625-7d07-962c-0215dd8f0109 [Leader]"
2026-03-29T01:09:20.756Z [INFO]  server-6d956970-1625-7d07-962c-0215dd8f0109: added peer, starting replication: peer=server-4ebabacb-094a-6236-f1b8-68703f47534d
2026-03-29T01:09:20.756Z [INFO]  server-6d956970-1625-7d07-962c-0215dd8f0109: added peer, starting replication: peer=server-1b784e08-f6dd-0184-65e7-6de5497aa119
2026-03-29T01:09:20.756Z [INFO]  server-6d956970-1625-7d07-962c-0215dd8f0109: pipelining replication: peer="{Voter server-1b784e08-f6dd-0184-65e7-6de5497aa119 1b784e08-f6dd-0184-65e7-6de5497aa119}"
2026-03-29T01:09:20.756Z [INFO]  server-6d956970-1625-7d07-962c-0215dd8f0109: pipelining replication: peer="{Voter server-4ebabacb-094a-6236-f1b8-68703f47534d 4ebabacb-094a-6236-f1b8-68703f47534d}"
2026-03-29T01:09:20.862Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:20.756457208 +0000 UTC m=+104.544599741 (1 nodes), 58.574742ms from start of poll, 58.814197ms from cluster start. Timeout at 2026-03-29 01:09:20.86153064 +0000 UTC m=+104.649673178, 105.073437ms after stability
2026-03-29T01:09:20.862Z [INFO]  server-6d956970-1625-7d07-962c-0215dd8f0109: cannot transfer leadership to itself
2026-03-29T01:09:20.862Z [INFO]  server-6d956970-1625-7d07-962c-0215dd8f0109: aborting pipeline replication: peer="{Voter server-1b784e08-f6dd-0184-65e7-6de5497aa119 1b784e08-f6dd-0184-65e7-6de5497aa119}"
2026-03-29T01:09:20.862Z [INFO]  server-6d956970-1625-7d07-962c-0215dd8f0109: aborting pipeline replication: peer="{Voter server-4ebabacb-094a-6236-f1b8-68703f47534d 4ebabacb-094a-6236-f1b8-68703f47534d}"
2026-03-29T01:09:21.363Z [ERROR] server-6d956970-1625-7d07-962c-0215dd8f0109: failed to heartbeat to: peer=4ebabacb-094a-6236-f1b8-68703f47534d backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeadershipTransferToItself (0.67s)
=== RUN   TestRaft_LeadershipTransferLeaderRejectsClientRequests
2026-03-29T01:09:21.364Z [INFO]  server-af992123-871d-2646-ddfb-b399ad989b6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af992123-871d-2646-ddfb-b399ad989b6f Address:af992123-871d-2646-ddfb-b399ad989b6f} {Suffrage:Voter ID:server-11b1452c-b0a6-1415-e236-625c1af22656 Address:11b1452c-b0a6-1415-e236-625c1af22656} {Suffrage:Voter ID:server-3f24d092-2929-fabf-d573-2af6e8ad24e6 Address:3f24d092-2929-fabf-d573-2af6e8ad24e6}]"
2026-03-29T01:09:21.364Z [INFO]  server-af992123-871d-2646-ddfb-b399ad989b6f: entering follower state: follower="Node at af992123-871d-2646-ddfb-b399ad989b6f [Follower]" leader-address= leader-id=
2026-03-29T01:09:21.364Z [INFO]  server-11b1452c-b0a6-1415-e236-625c1af22656: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af992123-871d-2646-ddfb-b399ad989b6f Address:af992123-871d-2646-ddfb-b399ad989b6f} {Suffrage:Voter ID:server-11b1452c-b0a6-1415-e236-625c1af22656 Address:11b1452c-b0a6-1415-e236-625c1af22656} {Suffrage:Voter ID:server-3f24d092-2929-fabf-d573-2af6e8ad24e6 Address:3f24d092-2929-fabf-d573-2af6e8ad24e6}]"
2026-03-29T01:09:21.364Z [INFO]  server-11b1452c-b0a6-1415-e236-625c1af22656: entering follower state: follower="Node at 11b1452c-b0a6-1415-e236-625c1af22656 [Follower]" leader-address= leader-id=
2026-03-29T01:09:21.364Z [INFO]  server-3f24d092-2929-fabf-d573-2af6e8ad24e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af992123-871d-2646-ddfb-b399ad989b6f Address:af992123-871d-2646-ddfb-b399ad989b6f} {Suffrage:Voter ID:server-11b1452c-b0a6-1415-e236-625c1af22656 Address:11b1452c-b0a6-1415-e236-625c1af22656} {Suffrage:Voter ID:server-3f24d092-2929-fabf-d573-2af6e8ad24e6 Address:3f24d092-2929-fabf-d573-2af6e8ad24e6}]"
2026-03-29T01:09:21.364Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:21.364Z [INFO]  server-3f24d092-2929-fabf-d573-2af6e8ad24e6: entering follower state: follower="Node at 3f24d092-2929-fabf-d573-2af6e8ad24e6 [Follower]" leader-address= leader-id=
2026-03-29T01:09:21.420Z [WARN]  server-3f24d092-2929-fabf-d573-2af6e8ad24e6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:21.420Z [INFO]  server-3f24d092-2929-fabf-d573-2af6e8ad24e6: entering candidate state: node="Node at 3f24d092-2929-fabf-d573-2af6e8ad24e6 [Candidate]" term=2
2026-03-29T01:09:21.420Z [INFO]  server-3f24d092-2929-fabf-d573-2af6e8ad24e6: election won: term=2 tally=2
2026-03-29T01:09:21.420Z [INFO]  server-3f24d092-2929-fabf-d573-2af6e8ad24e6: entering leader state: leader="Node at 3f24d092-2929-fabf-d573-2af6e8ad24e6 [Leader]"
2026-03-29T01:09:21.420Z [INFO]  server-3f24d092-2929-fabf-d573-2af6e8ad24e6: added peer, starting replication: peer=server-af992123-871d-2646-ddfb-b399ad989b6f
2026-03-29T01:09:21.420Z [INFO]  server-3f24d092-2929-fabf-d573-2af6e8ad24e6: added peer, starting replication: peer=server-11b1452c-b0a6-1415-e236-625c1af22656
2026-03-29T01:09:21.420Z [INFO]  server-3f24d092-2929-fabf-d573-2af6e8ad24e6: pipelining replication: peer="{Voter server-af992123-871d-2646-ddfb-b399ad989b6f af992123-871d-2646-ddfb-b399ad989b6f}"
2026-03-29T01:09:21.420Z [INFO]  server-3f24d092-2929-fabf-d573-2af6e8ad24e6: pipelining replication: peer="{Voter server-11b1452c-b0a6-1415-e236-625c1af22656 11b1452c-b0a6-1415-e236-625c1af22656}"
2026-03-29T01:09:21.525Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:21.420358194 +0000 UTC m=+105.208500733 (1 nodes), 55.733775ms from start of poll, 56.07354ms from cluster start. Timeout at 2026-03-29 01:09:21.525366278 +0000 UTC m=+105.313508805, 105.008072ms after stability
    raft_test.go:2457: waiting on future 0
    raft_test.go:2457: waiting on future 1
    raft_test.go:2457: waiting on future 2
    raft_test.go:2457: waiting on future 3
    raft_test.go:2457: waiting on future 4
    raft_test.go:2457: waiting on future 5
    raft_test.go:2457: waiting on future 6
2026-03-29T01:09:21.526Z [INFO]  server-3f24d092-2929-fabf-d573-2af6e8ad24e6: aborting pipeline replication: peer="{Voter server-11b1452c-b0a6-1415-e236-625c1af22656 11b1452c-b0a6-1415-e236-625c1af22656}"
2026-03-29T01:09:21.526Z [INFO]  server-3f24d092-2929-fabf-d573-2af6e8ad24e6: aborting pipeline replication: peer="{Voter server-af992123-871d-2646-ddfb-b399ad989b6f af992123-871d-2646-ddfb-b399ad989b6f}"
--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.16s)
=== RUN   TestRaft_LeadershipTransferLeaderReplicationTimeout
2026-03-29T01:09:21.527Z [INFO]  server-88cb1491-dcee-756b-5690-20a23aee951c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88cb1491-dcee-756b-5690-20a23aee951c Address:88cb1491-dcee-756b-5690-20a23aee951c} {Suffrage:Voter ID:server-9b94aaf5-1400-9335-1ca4-72afc3b954c5 Address:9b94aaf5-1400-9335-1ca4-72afc3b954c5} {Suffrage:Voter ID:server-9e130666-7876-f273-65bc-a56d6c778720 Address:9e130666-7876-f273-65bc-a56d6c778720}]"
2026-03-29T01:09:21.527Z [INFO]  server-88cb1491-dcee-756b-5690-20a23aee951c: entering follower state: follower="Node at 88cb1491-dcee-756b-5690-20a23aee951c [Follower]" leader-address= leader-id=
2026-03-29T01:09:21.527Z [INFO]  server-9b94aaf5-1400-9335-1ca4-72afc3b954c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88cb1491-dcee-756b-5690-20a23aee951c Address:88cb1491-dcee-756b-5690-20a23aee951c} {Suffrage:Voter ID:server-9b94aaf5-1400-9335-1ca4-72afc3b954c5 Address:9b94aaf5-1400-9335-1ca4-72afc3b954c5} {Suffrage:Voter ID:server-9e130666-7876-f273-65bc-a56d6c778720 Address:9e130666-7876-f273-65bc-a56d6c778720}]"
2026-03-29T01:09:21.527Z [INFO]  server-9b94aaf5-1400-9335-1ca4-72afc3b954c5: entering follower state: follower="Node at 9b94aaf5-1400-9335-1ca4-72afc3b954c5 [Follower]" leader-address= leader-id=
2026-03-29T01:09:21.527Z [INFO]  server-9e130666-7876-f273-65bc-a56d6c778720: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-88cb1491-dcee-756b-5690-20a23aee951c Address:88cb1491-dcee-756b-5690-20a23aee951c} {Suffrage:Voter ID:server-9b94aaf5-1400-9335-1ca4-72afc3b954c5 Address:9b94aaf5-1400-9335-1ca4-72afc3b954c5} {Suffrage:Voter ID:server-9e130666-7876-f273-65bc-a56d6c778720 Address:9e130666-7876-f273-65bc-a56d6c778720}]"
2026-03-29T01:09:21.527Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:21.527Z [INFO]  server-9e130666-7876-f273-65bc-a56d6c778720: entering follower state: follower="Node at 9e130666-7876-f273-65bc-a56d6c778720 [Follower]" leader-address= leader-id=
2026-03-29T01:09:21.585Z [WARN]  server-9e130666-7876-f273-65bc-a56d6c778720: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:21.585Z [INFO]  server-9e130666-7876-f273-65bc-a56d6c778720: entering candidate state: node="Node at 9e130666-7876-f273-65bc-a56d6c778720 [Candidate]" term=2
2026-03-29T01:09:21.585Z [INFO]  server-9e130666-7876-f273-65bc-a56d6c778720: election won: term=2 tally=2
2026-03-29T01:09:21.585Z [INFO]  server-9e130666-7876-f273-65bc-a56d6c778720: entering leader state: leader="Node at 9e130666-7876-f273-65bc-a56d6c778720 [Leader]"
2026-03-29T01:09:21.585Z [INFO]  server-9e130666-7876-f273-65bc-a56d6c778720: added peer, starting replication: peer=server-88cb1491-dcee-756b-5690-20a23aee951c
2026-03-29T01:09:21.585Z [INFO]  server-9e130666-7876-f273-65bc-a56d6c778720: added peer, starting replication: peer=server-9b94aaf5-1400-9335-1ca4-72afc3b954c5
2026-03-29T01:09:21.585Z [INFO]  server-9e130666-7876-f273-65bc-a56d6c778720: pipelining replication: peer="{Voter server-9b94aaf5-1400-9335-1ca4-72afc3b954c5 9b94aaf5-1400-9335-1ca4-72afc3b954c5}"
2026-03-29T01:09:21.585Z [INFO]  server-9e130666-7876-f273-65bc-a56d6c778720: pipelining replication: peer="{Voter server-88cb1491-dcee-756b-5690-20a23aee951c 88cb1491-dcee-756b-5690-20a23aee951c}"
2026-03-29T01:09:21.691Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:21.585876515 +0000 UTC m=+105.374019055 (1 nodes), 58.596304ms from start of poll, 58.918782ms from cluster start. Timeout at 2026-03-29 01:09:21.690883982 +0000 UTC m=+105.479026510, 105.007455ms after stability
2026-03-29T01:09:21.691Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:21.797Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:21.69157716 +0000 UTC m=+105.479719713 (2 nodes), 1.195µs from start of poll, 164.61944ms from cluster start. Timeout at 2026-03-29 01:09:21.796588629 +0000 UTC m=+105.584731226, 105.011513ms after stability
2026-03-29T01:09:22.237Z [INFO]  server-9e130666-7876-f273-65bc-a56d6c778720: aborting pipeline replication: peer="{Voter server-88cb1491-dcee-756b-5690-20a23aee951c 88cb1491-dcee-756b-5690-20a23aee951c}"
2026-03-29T01:09:22.237Z [INFO]  server-9e130666-7876-f273-65bc-a56d6c778720: aborting pipeline replication: peer="{Voter server-9b94aaf5-1400-9335-1ca4-72afc3b954c5 9b94aaf5-1400-9335-1ca4-72afc3b954c5}"
--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.71s)
=== RUN   TestRaft_LeadershipTransferIgnoresNonvoters
2026-03-29T01:09:22.238Z [INFO]  server-58123e47-b53d-2df4-f476-43b8be8ab483: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58123e47-b53d-2df4-f476-43b8be8ab483 Address:58123e47-b53d-2df4-f476-43b8be8ab483} {Suffrage:Voter ID:server-2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07 Address:2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07}]"
2026-03-29T01:09:22.238Z [INFO]  server-58123e47-b53d-2df4-f476-43b8be8ab483: entering follower state: follower="Node at 58123e47-b53d-2df4-f476-43b8be8ab483 [Follower]" leader-address= leader-id=
2026-03-29T01:09:22.238Z [INFO]  server-2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58123e47-b53d-2df4-f476-43b8be8ab483 Address:58123e47-b53d-2df4-f476-43b8be8ab483} {Suffrage:Voter ID:server-2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07 Address:2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07}]"
2026-03-29T01:09:22.238Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:22.238Z [INFO]  server-2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07: entering follower state: follower="Node at 2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07 [Follower]" leader-address= leader-id=
2026-03-29T01:09:22.315Z [WARN]  server-2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:22.315Z [INFO]  server-2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07: entering candidate state: node="Node at 2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07 [Candidate]" term=2
2026-03-29T01:09:22.315Z [INFO]  server-2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07: election won: term=2 tally=2
2026-03-29T01:09:22.315Z [INFO]  server-2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07: entering leader state: leader="Node at 2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07 [Leader]"
2026-03-29T01:09:22.315Z [INFO]  server-2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07: added peer, starting replication: peer=server-58123e47-b53d-2df4-f476-43b8be8ab483
2026-03-29T01:09:22.315Z [INFO]  server-2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07: pipelining replication: peer="{Voter server-58123e47-b53d-2df4-f476-43b8be8ab483 58123e47-b53d-2df4-f476-43b8be8ab483}"
2026-03-29T01:09:22.421Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:22.315937861 +0000 UTC m=+106.104080401 (1 nodes), 77.456053ms from start of poll, 77.727147ms from cluster start. Timeout at 2026-03-29 01:09:22.42097978 +0000 UTC m=+106.209122363, 105.041962ms after stability
2026-03-29T01:09:22.421Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:22.527Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:22.421575822 +0000 UTC m=+106.209718479 (1 nodes), 1.41µs from start of poll, 183.365225ms from cluster start. Timeout at 2026-03-29 01:09:22.526638625 +0000 UTC m=+106.314781153, 105.062674ms after stability
2026-03-29T01:09:22.527Z [INFO]  server-2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07: updating configuration: command=DemoteVoter server-id=server-58123e47-b53d-2df4-f476-43b8be8ab483 server-addr= servers="[{Suffrage:Nonvoter ID:server-58123e47-b53d-2df4-f476-43b8be8ab483 Address:58123e47-b53d-2df4-f476-43b8be8ab483} {Suffrage:Voter ID:server-2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07 Address:2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07}]"
2026-03-29T01:09:22.527Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:22.632Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:22.527311978 +0000 UTC m=+106.315454531 (1 nodes), 646ns from start of poll, 289.101277ms from cluster start. Timeout at 2026-03-29 01:09:22.632357392 +0000 UTC m=+106.420499937, 105.045406ms after stability
2026-03-29T01:09:22.632Z [INFO]  server-2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07: aborting pipeline replication: peer="{Voter server-58123e47-b53d-2df4-f476-43b8be8ab483 58123e47-b53d-2df4-f476-43b8be8ab483}"
2026-03-29T01:09:23.133Z [ERROR] server-2f58bf80-58eb-4de3-cc5b-0b2a2e7c5e07: failed to heartbeat to: peer=58123e47-b53d-2df4-f476-43b8be8ab483 backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.90s)
=== RUN   TestRaft_LeadershipTransferStopRightAway
--- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s)
=== RUN   TestRaft_GetConfigurationNoBootstrap
2026-03-29T01:09:23.134Z [INFO]  server-1a573a4b-56e8-366d-bb85-0e2fd17b58ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a573a4b-56e8-366d-bb85-0e2fd17b58ff Address:1a573a4b-56e8-366d-bb85-0e2fd17b58ff} {Suffrage:Voter ID:server-427e841b-42cf-6806-197e-28ce758bb167 Address:427e841b-42cf-6806-197e-28ce758bb167}]"
2026-03-29T01:09:23.134Z [INFO]  server-1a573a4b-56e8-366d-bb85-0e2fd17b58ff: entering follower state: follower="Node at 1a573a4b-56e8-366d-bb85-0e2fd17b58ff [Follower]" leader-address= leader-id=
2026-03-29T01:09:23.134Z [INFO]  server-427e841b-42cf-6806-197e-28ce758bb167: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a573a4b-56e8-366d-bb85-0e2fd17b58ff Address:1a573a4b-56e8-366d-bb85-0e2fd17b58ff} {Suffrage:Voter ID:server-427e841b-42cf-6806-197e-28ce758bb167 Address:427e841b-42cf-6806-197e-28ce758bb167}]"
2026-03-29T01:09:23.134Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:23.134Z [INFO]  server-427e841b-42cf-6806-197e-28ce758bb167: entering follower state: follower="Node at 427e841b-42cf-6806-197e-28ce758bb167 [Follower]" leader-address= leader-id=
2026-03-29T01:09:23.194Z [WARN]  server-1a573a4b-56e8-366d-bb85-0e2fd17b58ff: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:23.194Z [INFO]  server-1a573a4b-56e8-366d-bb85-0e2fd17b58ff: entering candidate state: node="Node at 1a573a4b-56e8-366d-bb85-0e2fd17b58ff [Candidate]" term=2
2026-03-29T01:09:23.195Z [INFO]  server-1a573a4b-56e8-366d-bb85-0e2fd17b58ff: election won: term=2 tally=2
2026-03-29T01:09:23.195Z [INFO]  server-1a573a4b-56e8-366d-bb85-0e2fd17b58ff: entering leader state: leader="Node at 1a573a4b-56e8-366d-bb85-0e2fd17b58ff [Leader]"
2026-03-29T01:09:23.195Z [INFO]  server-1a573a4b-56e8-366d-bb85-0e2fd17b58ff: added peer, starting replication: peer=server-427e841b-42cf-6806-197e-28ce758bb167
2026-03-29T01:09:23.195Z [INFO]  server-1a573a4b-56e8-366d-bb85-0e2fd17b58ff: pipelining replication: peer="{Voter server-427e841b-42cf-6806-197e-28ce758bb167 427e841b-42cf-6806-197e-28ce758bb167}"
2026-03-29T01:09:23.300Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:23.19512538 +0000 UTC m=+106.983267918 (1 nodes), 60.756433ms from start of poll, 60.991288ms from cluster start. Timeout at 2026-03-29 01:09:23.300129223 +0000 UTC m=+107.088271753, 105.003835ms after stability
2026-03-29T01:09:23.300Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:23.406Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:23.300206833 +0000 UTC m=+107.088349378 (1 nodes), 660ns from start of poll, 166.072748ms from cluster start. Timeout at 2026-03-29 01:09:23.405248557 +0000 UTC m=+107.193391241, 105.041863ms after stability
2026-03-29T01:09:23.416Z [INFO]  server-427e841b-42cf-6806-197e-28ce758bb167: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a573a4b-56e8-366d-bb85-0e2fd17b58ff Address:1a573a4b-56e8-366d-bb85-0e2fd17b58ff} {Suffrage:Voter ID:server-427e841b-42cf-6806-197e-28ce758bb167 Address:427e841b-42cf-6806-197e-28ce758bb167}]"
2026-03-29T01:09:23.416Z [INFO]  server-1a573a4b-56e8-366d-bb85-0e2fd17b58ff: aborting pipeline replication: peer="{Voter server-427e841b-42cf-6806-197e-28ce758bb167 427e841b-42cf-6806-197e-28ce758bb167}"
--- PASS: TestRaft_GetConfigurationNoBootstrap (0.28s)
=== RUN   TestRaft_LogStoreIsMonotonic
2026-03-29T01:09:23.417Z [INFO]  server-b1c8503b-9527-a59a-cfae-69b30e577515: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1c8503b-9527-a59a-cfae-69b30e577515 Address:b1c8503b-9527-a59a-cfae-69b30e577515}]"
2026-03-29T01:09:23.417Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:23.417Z [INFO]  server-b1c8503b-9527-a59a-cfae-69b30e577515: entering follower state: follower="Node at b1c8503b-9527-a59a-cfae-69b30e577515 [Follower]" leader-address= leader-id=
2026-03-29T01:09:23.499Z [WARN]  server-b1c8503b-9527-a59a-cfae-69b30e577515: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:23.499Z [INFO]  server-b1c8503b-9527-a59a-cfae-69b30e577515: entering candidate state: node="Node at b1c8503b-9527-a59a-cfae-69b30e577515 [Candidate]" term=2
2026-03-29T01:09:23.499Z [INFO]  server-b1c8503b-9527-a59a-cfae-69b30e577515: election won: term=2 tally=1
2026-03-29T01:09:23.499Z [INFO]  server-b1c8503b-9527-a59a-cfae-69b30e577515: entering leader state: leader="Node at b1c8503b-9527-a59a-cfae-69b30e577515 [Leader]"
2026-03-29T01:09:23.605Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:23.499919141 +0000 UTC m=+107.288061676 (1 nodes), 82.462398ms from start of poll, 82.577291ms from cluster start. Timeout at 2026-03-29 01:09:23.6049314 +0000 UTC m=+107.393073980, 105.012304ms after stability
--- PASS: TestRaft_LogStoreIsMonotonic (0.19s)
=== RUN   TestRaft_CacheLogWithStoreError
2026-03-29T01:09:23.606Z [INFO]  server-e30c437f-e89a-f68b-bc71-07015c313f54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e30c437f-e89a-f68b-bc71-07015c313f54 Address:e30c437f-e89a-f68b-bc71-07015c313f54} {Suffrage:Voter ID:server-6e78dc06-453b-4f53-4933-ed7354e71dab Address:6e78dc06-453b-4f53-4933-ed7354e71dab}]"
2026-03-29T01:09:23.606Z [INFO]  server-e30c437f-e89a-f68b-bc71-07015c313f54: entering follower state: follower="Node at e30c437f-e89a-f68b-bc71-07015c313f54 [Follower]" leader-address= leader-id=
2026-03-29T01:09:23.606Z [INFO]  server-6e78dc06-453b-4f53-4933-ed7354e71dab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e30c437f-e89a-f68b-bc71-07015c313f54 Address:e30c437f-e89a-f68b-bc71-07015c313f54} {Suffrage:Voter ID:server-6e78dc06-453b-4f53-4933-ed7354e71dab Address:6e78dc06-453b-4f53-4933-ed7354e71dab}]"
2026-03-29T01:09:23.606Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:23.606Z [INFO]  server-6e78dc06-453b-4f53-4933-ed7354e71dab: entering follower state: follower="Node at 6e78dc06-453b-4f53-4933-ed7354e71dab [Follower]" leader-address= leader-id=
2026-03-29T01:09:23.661Z [WARN]  server-6e78dc06-453b-4f53-4933-ed7354e71dab: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:23.661Z [INFO]  server-6e78dc06-453b-4f53-4933-ed7354e71dab: entering candidate state: node="Node at 6e78dc06-453b-4f53-4933-ed7354e71dab [Candidate]" term=2
2026-03-29T01:09:23.661Z [INFO]  server-6e78dc06-453b-4f53-4933-ed7354e71dab: election won: term=2 tally=2
2026-03-29T01:09:23.661Z [INFO]  server-6e78dc06-453b-4f53-4933-ed7354e71dab: entering leader state: leader="Node at 6e78dc06-453b-4f53-4933-ed7354e71dab [Leader]"
2026-03-29T01:09:23.661Z [INFO]  server-6e78dc06-453b-4f53-4933-ed7354e71dab: added peer, starting replication: peer=server-e30c437f-e89a-f68b-bc71-07015c313f54
2026-03-29T01:09:23.661Z [INFO]  server-6e78dc06-453b-4f53-4933-ed7354e71dab: pipelining replication: peer="{Voter server-e30c437f-e89a-f68b-bc71-07015c313f54 e30c437f-e89a-f68b-bc71-07015c313f54}"
2026-03-29T01:09:23.767Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:23.661704306 +0000 UTC m=+107.449846939 (1 nodes), 55.220241ms from start of poll, 55.432705ms from cluster start. Timeout at 2026-03-29 01:09:23.766723664 +0000 UTC m=+107.554866244, 105.019305ms after stability
2026-03-29T01:09:23.767Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:23.872Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:23.767690581 +0000 UTC m=+107.555833195 (1 nodes), 1.497µs from start of poll, 161.418961ms from cluster start. Timeout at 2026-03-29 01:09:23.872709112 +0000 UTC m=+107.660851685, 105.01849ms after stability
2026-03-29T01:09:23.873Z [ERROR] server-e30c437f-e89a-f68b-bc71-07015c313f54: failed to append to logs: error="unable to store logs within log store, err: "some error""
2026-03-29T01:09:23.873Z [WARN]  server-e30c437f-e89a-f68b-bc71-07015c313f54: failed to get previous log: previous-index=9 last-index=7 error="log not found"
2026-03-29T01:09:23.873Z [INFO]  server-6e78dc06-453b-4f53-4933-ed7354e71dab: aborting pipeline replication: peer="{Voter server-e30c437f-e89a-f68b-bc71-07015c313f54 e30c437f-e89a-f68b-bc71-07015c313f54}"
2026-03-29T01:09:23.879Z [INFO]  server-6e78dc06-453b-4f53-4933-ed7354e71dab: pipelining replication: peer="{Voter server-e30c437f-e89a-f68b-bc71-07015c313f54 e30c437f-e89a-f68b-bc71-07015c313f54}"
2026-03-29T01:09:23.879Z [INFO]  server-e30c437f-e89a-f68b-bc71-07015c313f54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e30c437f-e89a-f68b-bc71-07015c313f54 Address:e30c437f-e89a-f68b-bc71-07015c313f54} {Suffrage:Voter ID:server-6e78dc06-453b-4f53-4933-ed7354e71dab Address:6e78dc06-453b-4f53-4933-ed7354e71dab}]"
2026-03-29T01:09:23.880Z [INFO]  server-6e78dc06-453b-4f53-4933-ed7354e71dab: aborting pipeline replication: peer="{Voter server-e30c437f-e89a-f68b-bc71-07015c313f54 e30c437f-e89a-f68b-bc71-07015c313f54}"
--- PASS: TestRaft_CacheLogWithStoreError (0.27s)
=== RUN   TestRaft_ReloadConfig
2026-03-29T01:09:23.880Z [INFO]  server-52a473e9-9ea3-2c4b-c3bf-65ab5b49a4a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-52a473e9-9ea3-2c4b-c3bf-65ab5b49a4a1 Address:52a473e9-9ea3-2c4b-c3bf-65ab5b49a4a1}]"
--- PASS: TestRaft_ReloadConfig (0.00s)
=== RUN   TestRaft_ReloadConfigValidates
2026-03-29T01:09:23.881Z [INFO]  server-397d3a3e-58e7-14a1-b8c8-ec9d4662f3ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-397d3a3e-58e7-14a1-b8c8-ec9d4662f3ee Address:397d3a3e-58e7-14a1-b8c8-ec9d4662f3ee}]"
2026-03-29T01:09:23.881Z [INFO]  server-397d3a3e-58e7-14a1-b8c8-ec9d4662f3ee: entering follower state: follower="Node at 397d3a3e-58e7-14a1-b8c8-ec9d4662f3ee [Shutdown]" leader-address= leader-id=
--- PASS: TestRaft_ReloadConfigValidates (0.00s)
=== RUN   TestRaft_InstallSnapshot_InvalidPeers
2026-03-29T01:09:23.881Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)"
--- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s)
=== RUN   TestRaft_VoteNotGranted_WhenNodeNotInCluster
2026-03-29T01:09:23.881Z [INFO]  server-c67f6569-e01b-4aed-c7ca-73c454f74263: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c67f6569-e01b-4aed-c7ca-73c454f74263 Address:c67f6569-e01b-4aed-c7ca-73c454f74263} {Suffrage:Voter ID:server-dfbffc70-3bc4-edfc-787a-f0af221eadd8 Address:dfbffc70-3bc4-edfc-787a-f0af221eadd8} {Suffrage:Voter ID:server-91d05f17-7475-ceb7-7b19-78589ed01740 Address:91d05f17-7475-ceb7-7b19-78589ed01740}]"
2026-03-29T01:09:23.881Z [INFO]  server-c67f6569-e01b-4aed-c7ca-73c454f74263: entering follower state: follower="Node at c67f6569-e01b-4aed-c7ca-73c454f74263 [Follower]" leader-address= leader-id=
2026-03-29T01:09:23.881Z [INFO]  server-dfbffc70-3bc4-edfc-787a-f0af221eadd8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c67f6569-e01b-4aed-c7ca-73c454f74263 Address:c67f6569-e01b-4aed-c7ca-73c454f74263} {Suffrage:Voter ID:server-dfbffc70-3bc4-edfc-787a-f0af221eadd8 Address:dfbffc70-3bc4-edfc-787a-f0af221eadd8} {Suffrage:Voter ID:server-91d05f17-7475-ceb7-7b19-78589ed01740 Address:91d05f17-7475-ceb7-7b19-78589ed01740}]"
2026-03-29T01:09:23.882Z [INFO]  server-dfbffc70-3bc4-edfc-787a-f0af221eadd8: entering follower state: follower="Node at dfbffc70-3bc4-edfc-787a-f0af221eadd8 [Follower]" leader-address= leader-id=
2026-03-29T01:09:23.882Z [INFO]  server-91d05f17-7475-ceb7-7b19-78589ed01740: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c67f6569-e01b-4aed-c7ca-73c454f74263 Address:c67f6569-e01b-4aed-c7ca-73c454f74263} {Suffrage:Voter ID:server-dfbffc70-3bc4-edfc-787a-f0af221eadd8 Address:dfbffc70-3bc4-edfc-787a-f0af221eadd8} {Suffrage:Voter ID:server-91d05f17-7475-ceb7-7b19-78589ed01740 Address:91d05f17-7475-ceb7-7b19-78589ed01740}]"
2026-03-29T01:09:23.882Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:23.882Z [INFO]  server-91d05f17-7475-ceb7-7b19-78589ed01740: entering follower state: follower="Node at 91d05f17-7475-ceb7-7b19-78589ed01740 [Follower]" leader-address= leader-id=
2026-03-29T01:09:23.942Z [WARN]  server-c67f6569-e01b-4aed-c7ca-73c454f74263: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:23.942Z [INFO]  server-c67f6569-e01b-4aed-c7ca-73c454f74263: entering candidate state: node="Node at c67f6569-e01b-4aed-c7ca-73c454f74263 [Candidate]" term=2
2026-03-29T01:09:23.942Z [INFO]  server-c67f6569-e01b-4aed-c7ca-73c454f74263: election won: term=2 tally=2
2026-03-29T01:09:23.942Z [INFO]  server-c67f6569-e01b-4aed-c7ca-73c454f74263: entering leader state: leader="Node at c67f6569-e01b-4aed-c7ca-73c454f74263 [Leader]"
2026-03-29T01:09:23.942Z [INFO]  server-c67f6569-e01b-4aed-c7ca-73c454f74263: added peer, starting replication: peer=server-dfbffc70-3bc4-edfc-787a-f0af221eadd8
2026-03-29T01:09:23.942Z [INFO]  server-c67f6569-e01b-4aed-c7ca-73c454f74263: added peer, starting replication: peer=server-91d05f17-7475-ceb7-7b19-78589ed01740
2026-03-29T01:09:23.942Z [INFO]  server-c67f6569-e01b-4aed-c7ca-73c454f74263: pipelining replication: peer="{Voter server-dfbffc70-3bc4-edfc-787a-f0af221eadd8 dfbffc70-3bc4-edfc-787a-f0af221eadd8}"
2026-03-29T01:09:23.942Z [INFO]  server-c67f6569-e01b-4aed-c7ca-73c454f74263: pipelining replication: peer="{Voter server-91d05f17-7475-ceb7-7b19-78589ed01740 91d05f17-7475-ceb7-7b19-78589ed01740}"
2026-03-29T01:09:24.047Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:23.942347358 +0000 UTC m=+107.730489968 (1 nodes), 60.275934ms from start of poll, 60.489719ms from cluster start. Timeout at 2026-03-29 01:09:24.047349994 +0000 UTC m=+107.835492526, 105.002558ms after stability
2026-03-29T01:09:24.052Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:24.157Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:24.052606356 +0000 UTC m=+107.840748956 (2 nodes), 1.717µs from start of poll, 170.748707ms from cluster start. Timeout at 2026-03-29 01:09:24.157616379 +0000 UTC m=+107.945758936, 105.00998ms after stability
2026-03-29T01:09:24.157Z [INFO]  server-c67f6569-e01b-4aed-c7ca-73c454f74263: updating configuration: command=RemoveServer server-id=server-dfbffc70-3bc4-edfc-787a-f0af221eadd8 server-addr= servers="[{Suffrage:Voter ID:server-c67f6569-e01b-4aed-c7ca-73c454f74263 Address:c67f6569-e01b-4aed-c7ca-73c454f74263} {Suffrage:Voter ID:server-91d05f17-7475-ceb7-7b19-78589ed01740 Address:91d05f17-7475-ceb7-7b19-78589ed01740}]"
2026-03-29T01:09:24.157Z [INFO]  server-c67f6569-e01b-4aed-c7ca-73c454f74263: removed peer, stopping replication: peer=server-dfbffc70-3bc4-edfc-787a-f0af221eadd8 last-index=3
2026-03-29T01:09:24.157Z [INFO]  server-c67f6569-e01b-4aed-c7ca-73c454f74263: aborting pipeline replication: peer="{Voter server-dfbffc70-3bc4-edfc-787a-f0af221eadd8 dfbffc70-3bc4-edfc-787a-f0af221eadd8}"
2026-03-29T01:09:24.220Z [WARN]  server-dfbffc70-3bc4-edfc-787a-f0af221eadd8: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2026-03-29T01:09:24.263Z [ERROR] server-c67f6569-e01b-4aed-c7ca-73c454f74263: failed to heartbeat to: peer=91d05f17-7475-ceb7-7b19-78589ed01740 backoff time=10ms error="failed to connect to peer: 91d05f17-7475-ceb7-7b19-78589ed01740"
2026-03-29T01:09:24.265Z [ERROR] server-c67f6569-e01b-4aed-c7ca-73c454f74263: failed to pipeline appendEntries: peer="{Voter server-91d05f17-7475-ceb7-7b19-78589ed01740 91d05f17-7475-ceb7-7b19-78589ed01740}" error="append pipeline closed"
2026-03-29T01:09:24.265Z [INFO]  server-c67f6569-e01b-4aed-c7ca-73c454f74263: aborting pipeline replication: peer="{Voter server-91d05f17-7475-ceb7-7b19-78589ed01740 91d05f17-7475-ceb7-7b19-78589ed01740}"
2026-03-29T01:09:24.272Z [ERROR] server-c67f6569-e01b-4aed-c7ca-73c454f74263: failed to appendEntries to: peer="{Voter server-91d05f17-7475-ceb7-7b19-78589ed01740 91d05f17-7475-ceb7-7b19-78589ed01740}" error="failed to connect to peer: 91d05f17-7475-ceb7-7b19-78589ed01740"
2026-03-29T01:09:24.284Z [ERROR] server-c67f6569-e01b-4aed-c7ca-73c454f74263: failed to heartbeat to: peer=91d05f17-7475-ceb7-7b19-78589ed01740 backoff time=10ms error="failed to connect to peer: 91d05f17-7475-ceb7-7b19-78589ed01740"
2026-03-29T01:09:24.291Z [ERROR] server-c67f6569-e01b-4aed-c7ca-73c454f74263: failed to appendEntries to: peer="{Voter server-91d05f17-7475-ceb7-7b19-78589ed01740 91d05f17-7475-ceb7-7b19-78589ed01740}" error="failed to connect to peer: 91d05f17-7475-ceb7-7b19-78589ed01740"
2026-03-29T01:09:24.302Z [ERROR] server-c67f6569-e01b-4aed-c7ca-73c454f74263: failed to heartbeat to: peer=91d05f17-7475-ceb7-7b19-78589ed01740 backoff time=10ms error="failed to connect to peer: 91d05f17-7475-ceb7-7b19-78589ed01740"
2026-03-29T01:09:24.308Z [WARN]  server-c67f6569-e01b-4aed-c7ca-73c454f74263: failed to contact: server-id=server-91d05f17-7475-ceb7-7b19-78589ed01740 time=50.037861ms
2026-03-29T01:09:24.308Z [ERROR] server-c67f6569-e01b-4aed-c7ca-73c454f74263: failed to appendEntries to: peer="{Voter server-91d05f17-7475-ceb7-7b19-78589ed01740 91d05f17-7475-ceb7-7b19-78589ed01740}" error="failed to connect to peer: 91d05f17-7475-ceb7-7b19-78589ed01740"
2026-03-29T01:09:24.308Z [WARN]  server-c67f6569-e01b-4aed-c7ca-73c454f74263: failed to contact quorum of nodes, stepping down
2026-03-29T01:09:24.308Z [INFO]  server-c67f6569-e01b-4aed-c7ca-73c454f74263: entering follower state: follower="Node at c67f6569-e01b-4aed-c7ca-73c454f74263 [Follower]" leader-address= leader-id=
2026-03-29T01:09:24.343Z [WARN]  server-91d05f17-7475-ceb7-7b19-78589ed01740: heartbeat timeout reached, starting election: last-leader-addr=c67f6569-e01b-4aed-c7ca-73c454f74263 last-leader-id=server-c67f6569-e01b-4aed-c7ca-73c454f74263
2026-03-29T01:09:24.343Z [INFO]  server-91d05f17-7475-ceb7-7b19-78589ed01740: entering candidate state: node="Node at 91d05f17-7475-ceb7-7b19-78589ed01740 [Candidate]" term=3
2026-03-29T01:09:24.343Z [ERROR] server-91d05f17-7475-ceb7-7b19-78589ed01740: failed to make requestVote RPC: target="{Voter server-c67f6569-e01b-4aed-c7ca-73c454f74263 c67f6569-e01b-4aed-c7ca-73c454f74263}" error="failed to connect to peer: c67f6569-e01b-4aed-c7ca-73c454f74263" term=3
2026-03-29T01:09:24.368Z [WARN]  server-91d05f17-7475-ceb7-7b19-78589ed01740: rejecting vote request since node is not in configuration: from=dfbffc70-3bc4-edfc-787a-f0af221eadd8
--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.49s)
=== RUN   TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin
2026-03-29T01:09:24.369Z [INFO]  server-f030879f-6e2a-0947-a0cc-05ce4b272a6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f030879f-6e2a-0947-a0cc-05ce4b272a6c Address:f030879f-6e2a-0947-a0cc-05ce4b272a6c} {Suffrage:Voter ID:server-baea4647-6840-4afb-6908-6fe1097fd4c7 Address:baea4647-6840-4afb-6908-6fe1097fd4c7} {Suffrage:Voter ID:server-01788d2d-b919-cb8b-20b0-cee7e1da8858 Address:01788d2d-b919-cb8b-20b0-cee7e1da8858}]"
2026-03-29T01:09:24.369Z [INFO]  server-f030879f-6e2a-0947-a0cc-05ce4b272a6c: entering follower state: follower="Node at f030879f-6e2a-0947-a0cc-05ce4b272a6c [Follower]" leader-address= leader-id=
2026-03-29T01:09:24.369Z [INFO]  server-baea4647-6840-4afb-6908-6fe1097fd4c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f030879f-6e2a-0947-a0cc-05ce4b272a6c Address:f030879f-6e2a-0947-a0cc-05ce4b272a6c} {Suffrage:Voter ID:server-baea4647-6840-4afb-6908-6fe1097fd4c7 Address:baea4647-6840-4afb-6908-6fe1097fd4c7} {Suffrage:Voter ID:server-01788d2d-b919-cb8b-20b0-cee7e1da8858 Address:01788d2d-b919-cb8b-20b0-cee7e1da8858}]"
2026-03-29T01:09:24.369Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f030879f-6e2a-0947-a0cc-05ce4b272a6c Address:f030879f-6e2a-0947-a0cc-05ce4b272a6c} {Suffrage:Voter ID:server-baea4647-6840-4afb-6908-6fe1097fd4c7 Address:baea4647-6840-4afb-6908-6fe1097fd4c7} {Suffrage:Voter ID:server-01788d2d-b919-cb8b-20b0-cee7e1da8858 Address:01788d2d-b919-cb8b-20b0-cee7e1da8858}]"
2026-03-29T01:09:24.369Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:24.369Z [INFO]  server-baea4647-6840-4afb-6908-6fe1097fd4c7: entering follower state: follower="Node at baea4647-6840-4afb-6908-6fe1097fd4c7 [Follower]" leader-address= leader-id=
2026-03-29T01:09:24.369Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: entering follower state: follower="Node at 01788d2d-b919-cb8b-20b0-cee7e1da8858 [Follower]" leader-address= leader-id=
2026-03-29T01:09:24.429Z [WARN]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:24.429Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: entering candidate state: node="Node at 01788d2d-b919-cb8b-20b0-cee7e1da8858 [Candidate]" term=2
2026-03-29T01:09:24.429Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: election won: term=2 tally=2
2026-03-29T01:09:24.429Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: entering leader state: leader="Node at 01788d2d-b919-cb8b-20b0-cee7e1da8858 [Leader]"
2026-03-29T01:09:24.429Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: added peer, starting replication: peer=server-f030879f-6e2a-0947-a0cc-05ce4b272a6c
2026-03-29T01:09:24.429Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: added peer, starting replication: peer=server-baea4647-6840-4afb-6908-6fe1097fd4c7
2026-03-29T01:09:24.429Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: pipelining replication: peer="{Voter server-baea4647-6840-4afb-6908-6fe1097fd4c7 baea4647-6840-4afb-6908-6fe1097fd4c7}"
2026-03-29T01:09:24.429Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: pipelining replication: peer="{Voter server-f030879f-6e2a-0947-a0cc-05ce4b272a6c f030879f-6e2a-0947-a0cc-05ce4b272a6c}"
2026-03-29T01:09:24.534Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:24.429297544 +0000 UTC m=+108.217440083 (1 nodes), 59.436682ms from start of poll, 59.671228ms from cluster start. Timeout at 2026-03-29 01:09:24.534301247 +0000 UTC m=+108.322443777, 105.003694ms after stability
2026-03-29T01:09:24.539Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:24.645Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:24.53969528 +0000 UTC m=+108.327837824 (2 nodes), 1.238µs from start of poll, 170.068969ms from cluster start. Timeout at 2026-03-29 01:09:24.644733843 +0000 UTC m=+108.432876483, 105.038659ms after stability
2026-03-29T01:09:24.648Z [ERROR] server-01788d2d-b919-cb8b-20b0-cee7e1da8858: failed to heartbeat to: peer=f030879f-6e2a-0947-a0cc-05ce4b272a6c backoff time=10ms error="failed to connect to peer: f030879f-6e2a-0947-a0cc-05ce4b272a6c"
2026-03-29T01:09:24.652Z [ERROR] server-01788d2d-b919-cb8b-20b0-cee7e1da8858: failed to pipeline appendEntries: peer="{Voter server-f030879f-6e2a-0947-a0cc-05ce4b272a6c f030879f-6e2a-0947-a0cc-05ce4b272a6c}" error="append pipeline closed"
2026-03-29T01:09:24.652Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: aborting pipeline replication: peer="{Voter server-f030879f-6e2a-0947-a0cc-05ce4b272a6c f030879f-6e2a-0947-a0cc-05ce4b272a6c}"
2026-03-29T01:09:24.658Z [ERROR] server-01788d2d-b919-cb8b-20b0-cee7e1da8858: failed to appendEntries to: peer="{Voter server-f030879f-6e2a-0947-a0cc-05ce4b272a6c f030879f-6e2a-0947-a0cc-05ce4b272a6c}" error="failed to connect to peer: f030879f-6e2a-0947-a0cc-05ce4b272a6c"
2026-03-29T01:09:24.666Z [ERROR] server-01788d2d-b919-cb8b-20b0-cee7e1da8858: failed to heartbeat to: peer=f030879f-6e2a-0947-a0cc-05ce4b272a6c backoff time=10ms error="failed to connect to peer: f030879f-6e2a-0947-a0cc-05ce4b272a6c"
2026-03-29T01:09:24.676Z [ERROR] server-01788d2d-b919-cb8b-20b0-cee7e1da8858: failed to appendEntries to: peer="{Voter server-f030879f-6e2a-0947-a0cc-05ce4b272a6c f030879f-6e2a-0947-a0cc-05ce4b272a6c}" error="failed to connect to peer: f030879f-6e2a-0947-a0cc-05ce4b272a6c"
2026-03-29T01:09:24.686Z [ERROR] server-01788d2d-b919-cb8b-20b0-cee7e1da8858: failed to heartbeat to: peer=f030879f-6e2a-0947-a0cc-05ce4b272a6c backoff time=10ms error="failed to connect to peer: f030879f-6e2a-0947-a0cc-05ce4b272a6c"
2026-03-29T01:09:24.694Z [WARN]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: failed to contact: server-id=server-f030879f-6e2a-0947-a0cc-05ce4b272a6c time=51.022296ms
2026-03-29T01:09:24.697Z [ERROR] server-01788d2d-b919-cb8b-20b0-cee7e1da8858: failed to appendEntries to: peer="{Voter server-f030879f-6e2a-0947-a0cc-05ce4b272a6c f030879f-6e2a-0947-a0cc-05ce4b272a6c}" error="failed to connect to peer: f030879f-6e2a-0947-a0cc-05ce4b272a6c"
2026-03-29T01:09:24.703Z [WARN]  server-f030879f-6e2a-0947-a0cc-05ce4b272a6c: heartbeat timeout reached, starting election: last-leader-addr=01788d2d-b919-cb8b-20b0-cee7e1da8858 last-leader-id=server-01788d2d-b919-cb8b-20b0-cee7e1da8858
2026-03-29T01:09:24.703Z [INFO]  server-f030879f-6e2a-0947-a0cc-05ce4b272a6c: entering candidate state: node="Node at f030879f-6e2a-0947-a0cc-05ce4b272a6c [Candidate]" term=3
2026-03-29T01:09:24.703Z [ERROR] server-f030879f-6e2a-0947-a0cc-05ce4b272a6c: failed to make requestVote RPC: target="{Voter server-01788d2d-b919-cb8b-20b0-cee7e1da8858 01788d2d-b919-cb8b-20b0-cee7e1da8858}" error="failed to connect to peer: 01788d2d-b919-cb8b-20b0-cee7e1da8858" term=3
2026-03-29T01:09:24.703Z [ERROR] server-f030879f-6e2a-0947-a0cc-05ce4b272a6c: failed to make requestVote RPC: target="{Voter server-baea4647-6840-4afb-6908-6fe1097fd4c7 baea4647-6840-4afb-6908-6fe1097fd4c7}" error="failed to connect to peer: baea4647-6840-4afb-6908-6fe1097fd4c7" term=3
2026-03-29T01:09:24.705Z [ERROR] server-01788d2d-b919-cb8b-20b0-cee7e1da8858: failed to heartbeat to: peer=f030879f-6e2a-0947-a0cc-05ce4b272a6c backoff time=20ms error="failed to connect to peer: f030879f-6e2a-0947-a0cc-05ce4b272a6c"
2026-03-29T01:09:24.725Z [ERROR] server-01788d2d-b919-cb8b-20b0-cee7e1da8858: failed to appendEntries to: peer="{Voter server-f030879f-6e2a-0947-a0cc-05ce4b272a6c f030879f-6e2a-0947-a0cc-05ce4b272a6c}" error="failed to connect to peer: f030879f-6e2a-0947-a0cc-05ce4b272a6c"
2026-03-29T01:09:24.732Z [ERROR] server-01788d2d-b919-cb8b-20b0-cee7e1da8858: failed to heartbeat to: peer=f030879f-6e2a-0947-a0cc-05ce4b272a6c backoff time=25ms error="failed to connect to peer: f030879f-6e2a-0947-a0cc-05ce4b272a6c"
2026-03-29T01:09:24.743Z [WARN]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: failed to contact: server-id=server-f030879f-6e2a-0947-a0cc-05ce4b272a6c time=100.586581ms
2026-03-29T01:09:24.751Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: updating configuration: command=RemoveServer server-id=server-f030879f-6e2a-0947-a0cc-05ce4b272a6c server-addr= servers="[{Suffrage:Voter ID:server-baea4647-6840-4afb-6908-6fe1097fd4c7 Address:baea4647-6840-4afb-6908-6fe1097fd4c7} {Suffrage:Voter ID:server-01788d2d-b919-cb8b-20b0-cee7e1da8858 Address:01788d2d-b919-cb8b-20b0-cee7e1da8858}]"
2026-03-29T01:09:24.751Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: removed peer, stopping replication: peer=server-f030879f-6e2a-0947-a0cc-05ce4b272a6c last-index=3
2026-03-29T01:09:24.751Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: updating configuration: command=AddNonvoter server-id=server-f030879f-6e2a-0947-a0cc-05ce4b272a6c server-addr=f030879f-6e2a-0947-a0cc-05ce4b272a6c servers="[{Suffrage:Voter ID:server-baea4647-6840-4afb-6908-6fe1097fd4c7 Address:baea4647-6840-4afb-6908-6fe1097fd4c7} {Suffrage:Voter ID:server-01788d2d-b919-cb8b-20b0-cee7e1da8858 Address:01788d2d-b919-cb8b-20b0-cee7e1da8858} {Suffrage:Nonvoter ID:server-f030879f-6e2a-0947-a0cc-05ce4b272a6c Address:f030879f-6e2a-0947-a0cc-05ce4b272a6c}]"
2026-03-29T01:09:24.751Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: added peer, starting replication: peer=server-f030879f-6e2a-0947-a0cc-05ce4b272a6c
2026-03-29T01:09:24.751Z [ERROR] server-01788d2d-b919-cb8b-20b0-cee7e1da8858: failed to appendEntries to: peer="{Nonvoter server-f030879f-6e2a-0947-a0cc-05ce4b272a6c f030879f-6e2a-0947-a0cc-05ce4b272a6c}" error="failed to connect to peer: f030879f-6e2a-0947-a0cc-05ce4b272a6c"
2026-03-29T01:09:24.769Z [ERROR] server-01788d2d-b919-cb8b-20b0-cee7e1da8858: peer has newer term, stopping replication: peer="{Nonvoter server-f030879f-6e2a-0947-a0cc-05ce4b272a6c f030879f-6e2a-0947-a0cc-05ce4b272a6c}"
2026-03-29T01:09:24.769Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: entering follower state: follower="Node at 01788d2d-b919-cb8b-20b0-cee7e1da8858 [Follower]" leader-address= leader-id=
2026-03-29T01:09:24.769Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: aborting pipeline replication: peer="{Voter server-baea4647-6840-4afb-6908-6fe1097fd4c7 baea4647-6840-4afb-6908-6fe1097fd4c7}"
2026-03-29T01:09:24.772Z [ERROR] server-01788d2d-b919-cb8b-20b0-cee7e1da8858: peer has newer term, stopping replication: peer="{Voter server-f030879f-6e2a-0947-a0cc-05ce4b272a6c f030879f-6e2a-0947-a0cc-05ce4b272a6c}"
2026-03-29T01:09:24.774Z [WARN]  server-f030879f-6e2a-0947-a0cc-05ce4b272a6c: Election timeout reached, restarting election
2026-03-29T01:09:24.774Z [INFO]  server-f030879f-6e2a-0947-a0cc-05ce4b272a6c: entering candidate state: node="Node at f030879f-6e2a-0947-a0cc-05ce4b272a6c [Candidate]" term=23
2026-03-29T01:09:24.774Z [WARN]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: rejecting vote request since node is not a voter: from=f030879f-6e2a-0947-a0cc-05ce4b272a6c
2026-03-29T01:09:24.774Z [WARN]  server-baea4647-6840-4afb-6908-6fe1097fd4c7: rejecting vote request since we have a leader: from=f030879f-6e2a-0947-a0cc-05ce4b272a6c leader=01788d2d-b919-cb8b-20b0-cee7e1da8858 leader-id=server-01788d2d-b919-cb8b-20b0-cee7e1da8858
2026-03-29T01:09:24.844Z [WARN]  server-baea4647-6840-4afb-6908-6fe1097fd4c7: heartbeat timeout reached, starting election: last-leader-addr=01788d2d-b919-cb8b-20b0-cee7e1da8858 last-leader-id=server-01788d2d-b919-cb8b-20b0-cee7e1da8858
2026-03-29T01:09:24.844Z [INFO]  server-baea4647-6840-4afb-6908-6fe1097fd4c7: entering candidate state: node="Node at baea4647-6840-4afb-6908-6fe1097fd4c7 [Candidate]" term=3
2026-03-29T01:09:24.844Z [INFO]  server-baea4647-6840-4afb-6908-6fe1097fd4c7: entering follower state: follower="Node at baea4647-6840-4afb-6908-6fe1097fd4c7 [Follower]" leader-address= leader-id=
2026-03-29T01:09:24.845Z [WARN]  server-f030879f-6e2a-0947-a0cc-05ce4b272a6c: Election timeout reached, restarting election
2026-03-29T01:09:24.845Z [INFO]  server-f030879f-6e2a-0947-a0cc-05ce4b272a6c: entering candidate state: node="Node at f030879f-6e2a-0947-a0cc-05ce4b272a6c [Candidate]" term=24
2026-03-29T01:09:24.845Z [WARN]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: rejecting vote request since node is not a voter: from=f030879f-6e2a-0947-a0cc-05ce4b272a6c
2026-03-29T01:09:24.845Z [WARN]  server-baea4647-6840-4afb-6908-6fe1097fd4c7: rejecting vote request since node is not a voter: from=f030879f-6e2a-0947-a0cc-05ce4b272a6c
2026-03-29T01:09:24.854Z [WARN]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:24.854Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: entering candidate state: node="Node at 01788d2d-b919-cb8b-20b0-cee7e1da8858 [Candidate]" term=25
2026-03-29T01:09:24.854Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: election won: term=25 tally=2
2026-03-29T01:09:24.854Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: entering leader state: leader="Node at 01788d2d-b919-cb8b-20b0-cee7e1da8858 [Leader]"
2026-03-29T01:09:24.854Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: added peer, starting replication: peer=server-baea4647-6840-4afb-6908-6fe1097fd4c7
2026-03-29T01:09:24.854Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: added peer, starting replication: peer=server-f030879f-6e2a-0947-a0cc-05ce4b272a6c
2026-03-29T01:09:24.854Z [WARN]  server-f030879f-6e2a-0947-a0cc-05ce4b272a6c: failed to get previous log: previous-index=4 last-index=2 error="log not found"
2026-03-29T01:09:24.854Z [INFO]  server-f030879f-6e2a-0947-a0cc-05ce4b272a6c: entering follower state: follower="Node at f030879f-6e2a-0947-a0cc-05ce4b272a6c [Follower]" leader-address=01788d2d-b919-cb8b-20b0-cee7e1da8858 leader-id=server-01788d2d-b919-cb8b-20b0-cee7e1da8858
2026-03-29T01:09:24.854Z [WARN]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: appendEntries rejected, sending older logs: peer="{Nonvoter server-f030879f-6e2a-0947-a0cc-05ce4b272a6c f030879f-6e2a-0947-a0cc-05ce4b272a6c}" next=3
2026-03-29T01:09:24.854Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: pipelining replication: peer="{Voter server-baea4647-6840-4afb-6908-6fe1097fd4c7 baea4647-6840-4afb-6908-6fe1097fd4c7}"
2026-03-29T01:09:24.854Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: pipelining replication: peer="{Nonvoter server-f030879f-6e2a-0947-a0cc-05ce4b272a6c f030879f-6e2a-0947-a0cc-05ce4b272a6c}"
2026-03-29T01:09:24.858Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:24.963Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:24.858206765 +0000 UTC m=+108.646349362 (1 nodes), 97.753µs from start of poll, 488.580507ms from cluster start. Timeout at 2026-03-29 01:09:24.963215581 +0000 UTC m=+108.751358160, 105.008798ms after stability
2026-03-29T01:09:24.987Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: updating configuration: command=RemoveServer server-id=server-f030879f-6e2a-0947-a0cc-05ce4b272a6c server-addr= servers="[{Suffrage:Voter ID:server-baea4647-6840-4afb-6908-6fe1097fd4c7 Address:baea4647-6840-4afb-6908-6fe1097fd4c7} {Suffrage:Voter ID:server-01788d2d-b919-cb8b-20b0-cee7e1da8858 Address:01788d2d-b919-cb8b-20b0-cee7e1da8858}]"
2026-03-29T01:09:24.988Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: removed peer, stopping replication: peer=server-f030879f-6e2a-0947-a0cc-05ce4b272a6c last-index=106
2026-03-29T01:09:24.988Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: aborting pipeline replication: peer="{Nonvoter server-f030879f-6e2a-0947-a0cc-05ce4b272a6c f030879f-6e2a-0947-a0cc-05ce4b272a6c}"
2026-03-29T01:09:24.988Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: updating configuration: command=AddVoter server-id=server-f030879f-6e2a-0947-a0cc-05ce4b272a6c server-addr=f030879f-6e2a-0947-a0cc-05ce4b272a6c servers="[{Suffrage:Voter ID:server-baea4647-6840-4afb-6908-6fe1097fd4c7 Address:baea4647-6840-4afb-6908-6fe1097fd4c7} {Suffrage:Voter ID:server-01788d2d-b919-cb8b-20b0-cee7e1da8858 Address:01788d2d-b919-cb8b-20b0-cee7e1da8858} {Suffrage:Voter ID:server-f030879f-6e2a-0947-a0cc-05ce4b272a6c Address:f030879f-6e2a-0947-a0cc-05ce4b272a6c}]"
2026-03-29T01:09:24.988Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: added peer, starting replication: peer=server-f030879f-6e2a-0947-a0cc-05ce4b272a6c
2026-03-29T01:09:24.988Z [WARN]  server-f030879f-6e2a-0947-a0cc-05ce4b272a6c: failed to get previous log: previous-index=107 last-index=106 error="log not found"
2026-03-29T01:09:24.988Z [WARN]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: appendEntries rejected, sending older logs: peer="{Voter server-f030879f-6e2a-0947-a0cc-05ce4b272a6c f030879f-6e2a-0947-a0cc-05ce4b272a6c}" next=107
2026-03-29T01:09:24.988Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: pipelining replication: peer="{Voter server-f030879f-6e2a-0947-a0cc-05ce4b272a6c f030879f-6e2a-0947-a0cc-05ce4b272a6c}"
2026-03-29T01:09:26.058Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:26.164Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:26.058590538 +0000 UTC m=+109.846733071 (1 nodes), 548ns from start of poll, 1.688964216s from cluster start. Timeout at 2026-03-29 01:09:26.163609534 +0000 UTC m=+109.951752064, 105.018993ms after stability
2026-03-29T01:09:26.164Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: aborting pipeline replication: peer="{Voter server-f030879f-6e2a-0947-a0cc-05ce4b272a6c f030879f-6e2a-0947-a0cc-05ce4b272a6c}"
2026-03-29T01:09:26.164Z [INFO]  server-01788d2d-b919-cb8b-20b0-cee7e1da8858: aborting pipeline replication: peer="{Voter server-baea4647-6840-4afb-6908-6fe1097fd4c7 baea4647-6840-4afb-6908-6fe1097fd4c7}"
--- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.80s)
=== RUN   TestRaft_FollowerRemovalNoElection
2026-03-29T01:09:26.164Z [INFO]  server-df7a9415-4e34-e007-8681-d79ce50e7564: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df7a9415-4e34-e007-8681-d79ce50e7564 Address:df7a9415-4e34-e007-8681-d79ce50e7564} {Suffrage:Voter ID:server-129760ae-aae3-2bb6-9bab-bf23b6536537 Address:129760ae-aae3-2bb6-9bab-bf23b6536537} {Suffrage:Voter ID:server-e770a725-0ffc-0801-6e75-fd25b5b73d96 Address:e770a725-0ffc-0801-6e75-fd25b5b73d96}]"
2026-03-29T01:09:26.165Z [INFO]  server-df7a9415-4e34-e007-8681-d79ce50e7564: entering follower state: follower="Node at df7a9415-4e34-e007-8681-d79ce50e7564 [Follower]" leader-address= leader-id=
2026-03-29T01:09:26.165Z [INFO]  server-129760ae-aae3-2bb6-9bab-bf23b6536537: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df7a9415-4e34-e007-8681-d79ce50e7564 Address:df7a9415-4e34-e007-8681-d79ce50e7564} {Suffrage:Voter ID:server-129760ae-aae3-2bb6-9bab-bf23b6536537 Address:129760ae-aae3-2bb6-9bab-bf23b6536537} {Suffrage:Voter ID:server-e770a725-0ffc-0801-6e75-fd25b5b73d96 Address:e770a725-0ffc-0801-6e75-fd25b5b73d96}]"
2026-03-29T01:09:26.165Z [INFO]  server-129760ae-aae3-2bb6-9bab-bf23b6536537: entering follower state: follower="Node at 129760ae-aae3-2bb6-9bab-bf23b6536537 [Follower]" leader-address= leader-id=
2026-03-29T01:09:26.165Z [INFO]  server-e770a725-0ffc-0801-6e75-fd25b5b73d96: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df7a9415-4e34-e007-8681-d79ce50e7564 Address:df7a9415-4e34-e007-8681-d79ce50e7564} {Suffrage:Voter ID:server-129760ae-aae3-2bb6-9bab-bf23b6536537 Address:129760ae-aae3-2bb6-9bab-bf23b6536537} {Suffrage:Voter ID:server-e770a725-0ffc-0801-6e75-fd25b5b73d96 Address:e770a725-0ffc-0801-6e75-fd25b5b73d96}]"
2026-03-29T01:09:26.165Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:26.165Z [INFO]  server-e770a725-0ffc-0801-6e75-fd25b5b73d96: entering follower state: follower="Node at e770a725-0ffc-0801-6e75-fd25b5b73d96 [Follower]" leader-address= leader-id=
2026-03-29T01:09:26.296Z [WARN]  server-129760ae-aae3-2bb6-9bab-bf23b6536537: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:26.296Z [INFO]  server-129760ae-aae3-2bb6-9bab-bf23b6536537: entering candidate state: node="Node at 129760ae-aae3-2bb6-9bab-bf23b6536537 [Candidate]" term=2
2026-03-29T01:09:26.296Z [INFO]  server-129760ae-aae3-2bb6-9bab-bf23b6536537: election won: term=2 tally=2
2026-03-29T01:09:26.296Z [INFO]  server-129760ae-aae3-2bb6-9bab-bf23b6536537: entering leader state: leader="Node at 129760ae-aae3-2bb6-9bab-bf23b6536537 [Leader]"
2026-03-29T01:09:26.296Z [INFO]  server-129760ae-aae3-2bb6-9bab-bf23b6536537: added peer, starting replication: peer=server-df7a9415-4e34-e007-8681-d79ce50e7564
2026-03-29T01:09:26.296Z [INFO]  server-129760ae-aae3-2bb6-9bab-bf23b6536537: added peer, starting replication: peer=server-e770a725-0ffc-0801-6e75-fd25b5b73d96
2026-03-29T01:09:26.296Z [INFO]  server-129760ae-aae3-2bb6-9bab-bf23b6536537: pipelining replication: peer="{Voter server-e770a725-0ffc-0801-6e75-fd25b5b73d96 e770a725-0ffc-0801-6e75-fd25b5b73d96}"
2026-03-29T01:09:26.296Z [INFO]  server-129760ae-aae3-2bb6-9bab-bf23b6536537: pipelining replication: peer="{Voter server-df7a9415-4e34-e007-8681-d79ce50e7564 df7a9415-4e34-e007-8681-d79ce50e7564}"
2026-03-29T01:09:26.501Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:26.296518119 +0000 UTC m=+110.084660652 (1 nodes), 131.387969ms from start of poll, 131.632886ms from cluster start. Timeout at 2026-03-29 01:09:26.501520261 +0000 UTC m=+110.289662789, 205.002137ms after stability
2026-03-29T01:09:26.502Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:26.707Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:26.502017481 +0000 UTC m=+110.290160016 (1 nodes), 544ns from start of poll, 337.13225ms from cluster start. Timeout at 2026-03-29 01:09:26.707059994 +0000 UTC m=+110.495202577, 205.042561ms after stability
2026-03-29T01:09:26.712Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:26.918Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:26.712694003 +0000 UTC m=+110.500836636 (2 nodes), 2.084µs from start of poll, 547.80887ms from cluster start. Timeout at 2026-03-29 01:09:26.917707684 +0000 UTC m=+110.705850242, 205.013606ms after stability
2026-03-29T01:09:26.918Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:27.123Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:26.918328874 +0000 UTC m=+110.706471407 (1 nodes), 542ns from start of poll, 753.443641ms from cluster start. Timeout at 2026-03-29 01:09:27.123337758 +0000 UTC m=+110.911480307, 205.0089ms after stability
    raft_test.go:2961: [INFO] restarting Node at df7a9415-4e34-e007-8681-d79ce50e7564 [Follower]
2026-03-29T01:09:27.123Z [INFO]  server-df7a9415-4e34-e007-8681-d79ce50e7564: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df7a9415-4e34-e007-8681-d79ce50e7564 Address:df7a9415-4e34-e007-8681-d79ce50e7564} {Suffrage:Voter ID:server-129760ae-aae3-2bb6-9bab-bf23b6536537 Address:129760ae-aae3-2bb6-9bab-bf23b6536537} {Suffrage:Voter ID:server-e770a725-0ffc-0801-6e75-fd25b5b73d96 Address:e770a725-0ffc-0801-6e75-fd25b5b73d96}]"
2026-03-29T01:09:27.123Z [INFO]  server-df7a9415-4e34-e007-8681-d79ce50e7564: entering follower state: follower="Node at df7a9415-4e34-e007-8681-d79ce50e7564 [Follower]" leader-address= leader-id=
2026-03-29T01:09:27.373Z [INFO]  server-129760ae-aae3-2bb6-9bab-bf23b6536537: aborting pipeline replication: peer="{Voter server-e770a725-0ffc-0801-6e75-fd25b5b73d96 e770a725-0ffc-0801-6e75-fd25b5b73d96}"
2026-03-29T01:09:27.758Z [ERROR] server-129760ae-aae3-2bb6-9bab-bf23b6536537: failed to pipeline appendEntries: peer="{Voter server-df7a9415-4e34-e007-8681-d79ce50e7564 df7a9415-4e34-e007-8681-d79ce50e7564}" error="command enqueue timeout"
2026-03-29T01:09:27.758Z [INFO]  server-129760ae-aae3-2bb6-9bab-bf23b6536537: aborting pipeline replication: peer="{Voter server-df7a9415-4e34-e007-8681-d79ce50e7564 df7a9415-4e34-e007-8681-d79ce50e7564}"
2026-03-29T01:09:27.880Z [ERROR] server-129760ae-aae3-2bb6-9bab-bf23b6536537: failed to heartbeat to: peer=df7a9415-4e34-e007-8681-d79ce50e7564 backoff time=10ms error="command timed out"
--- PASS: TestRaft_FollowerRemovalNoElection (1.72s)
=== RUN   TestRaft_VoteWithNoIDNoAddr
2026-03-29T01:09:27.881Z [INFO]  server-a8458347-a220-f769-ba00-df4d803dae29: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8458347-a220-f769-ba00-df4d803dae29 Address:a8458347-a220-f769-ba00-df4d803dae29} {Suffrage:Voter ID:server-c716643f-246d-8c77-d271-1f47ee5c8bba Address:c716643f-246d-8c77-d271-1f47ee5c8bba} {Suffrage:Voter ID:server-3bd8108a-b0da-b639-4f5c-52daa4705472 Address:3bd8108a-b0da-b639-4f5c-52daa4705472}]"
2026-03-29T01:09:27.881Z [INFO]  server-a8458347-a220-f769-ba00-df4d803dae29: entering follower state: follower="Node at a8458347-a220-f769-ba00-df4d803dae29 [Follower]" leader-address= leader-id=
2026-03-29T01:09:27.881Z [INFO]  server-c716643f-246d-8c77-d271-1f47ee5c8bba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8458347-a220-f769-ba00-df4d803dae29 Address:a8458347-a220-f769-ba00-df4d803dae29} {Suffrage:Voter ID:server-c716643f-246d-8c77-d271-1f47ee5c8bba Address:c716643f-246d-8c77-d271-1f47ee5c8bba} {Suffrage:Voter ID:server-3bd8108a-b0da-b639-4f5c-52daa4705472 Address:3bd8108a-b0da-b639-4f5c-52daa4705472}]"
2026-03-29T01:09:27.882Z [INFO]  server-c716643f-246d-8c77-d271-1f47ee5c8bba: entering follower state: follower="Node at c716643f-246d-8c77-d271-1f47ee5c8bba [Follower]" leader-address= leader-id=
2026-03-29T01:09:27.882Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8458347-a220-f769-ba00-df4d803dae29 Address:a8458347-a220-f769-ba00-df4d803dae29} {Suffrage:Voter ID:server-c716643f-246d-8c77-d271-1f47ee5c8bba Address:c716643f-246d-8c77-d271-1f47ee5c8bba} {Suffrage:Voter ID:server-3bd8108a-b0da-b639-4f5c-52daa4705472 Address:3bd8108a-b0da-b639-4f5c-52daa4705472}]"
2026-03-29T01:09:27.882Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:27.882Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering follower state: follower="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Follower]" leader-address= leader-id=
2026-03-29T01:09:27.941Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:27.941Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering candidate state: node="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Candidate]" term=2
2026-03-29T01:09:27.941Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: election won: term=2 tally=2
2026-03-29T01:09:27.941Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering leader state: leader="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Leader]"
2026-03-29T01:09:27.941Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: added peer, starting replication: peer=server-a8458347-a220-f769-ba00-df4d803dae29
2026-03-29T01:09:27.941Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: added peer, starting replication: peer=server-c716643f-246d-8c77-d271-1f47ee5c8bba
2026-03-29T01:09:27.941Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: pipelining replication: peer="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}"
2026-03-29T01:09:27.941Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: pipelining replication: peer="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}"
2026-03-29T01:09:28.046Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:27.941583114 +0000 UTC m=+111.729725649 (1 nodes), 59.514651ms from start of poll, 59.737685ms from cluster start. Timeout at 2026-03-29 01:09:28.046586049 +0000 UTC m=+111.834728587, 105.002938ms after stability
2026-03-29T01:09:28.046Z [INFO]  cluster: starting stability test: raft-state=Leader
2026-03-29T01:09:28.152Z [INFO]  cluster: stable state for Leader reached at 2026-03-29 01:09:28.046685916 +0000 UTC m=+111.834828454 (1 nodes), 464ns from start of poll, 164.84049ms from cluster start. Timeout at 2026-03-29 01:09:28.151717275 +0000 UTC m=+111.939859837, 105.031383ms after stability
2026-03-29T01:09:28.157Z [INFO]  cluster: starting stability test: raft-state=Follower
2026-03-29T01:09:28.262Z [INFO]  cluster: stable state for Follower reached at 2026-03-29 01:09:28.157582327 +0000 UTC m=+111.945724871 (2 nodes), 875ns from start of poll, 275.736907ms from cluster start. Timeout at 2026-03-29 01:09:28.262593695 +0000 UTC m=+112.050736242, 105.011371ms after stability
2026-03-29T01:09:28.263Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to heartbeat to: peer=c716643f-246d-8c77-d271-1f47ee5c8bba backoff time=10ms error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba"
2026-03-29T01:09:28.263Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to heartbeat to: peer=a8458347-a220-f769-ba00-df4d803dae29 backoff time=10ms error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29"
2026-03-29T01:09:28.267Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to pipeline appendEntries: peer="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="append pipeline closed"
2026-03-29T01:09:28.267Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: aborting pipeline replication: peer="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}"
2026-03-29T01:09:28.268Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to pipeline appendEntries: peer="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="append pipeline closed"
2026-03-29T01:09:28.268Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: aborting pipeline replication: peer="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}"
2026-03-29T01:09:28.277Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to appendEntries to: peer="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29"
2026-03-29T01:09:28.277Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to appendEntries to: peer="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba"
2026-03-29T01:09:28.279Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to heartbeat to: peer=c716643f-246d-8c77-d271-1f47ee5c8bba backoff time=10ms error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba"
2026-03-29T01:09:28.283Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to heartbeat to: peer=a8458347-a220-f769-ba00-df4d803dae29 backoff time=10ms error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29"
2026-03-29T01:09:28.296Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to heartbeat to: peer=c716643f-246d-8c77-d271-1f47ee5c8bba backoff time=10ms error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba"
2026-03-29T01:09:28.296Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to appendEntries to: peer="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba"
2026-03-29T01:09:28.297Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to appendEntries to: peer="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29"
2026-03-29T01:09:28.301Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to heartbeat to: peer=a8458347-a220-f769-ba00-df4d803dae29 backoff time=10ms error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29"
2026-03-29T01:09:28.308Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to contact: server-id=server-a8458347-a220-f769-ba00-df4d803dae29 time=50.285183ms
2026-03-29T01:09:28.308Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to contact: server-id=server-c716643f-246d-8c77-d271-1f47ee5c8bba time=50.314176ms
2026-03-29T01:09:28.308Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to contact quorum of nodes, stepping down
2026-03-29T01:09:28.309Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering follower state: follower="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Follower]" leader-address= leader-id=
2026-03-29T01:09:28.312Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to appendEntries to: peer="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba"
2026-03-29T01:09:28.317Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to appendEntries to: peer="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29"
2026-03-29T01:09:28.319Z [WARN]  server-c716643f-246d-8c77-d271-1f47ee5c8bba: heartbeat timeout reached, starting election: last-leader-addr=3bd8108a-b0da-b639-4f5c-52daa4705472 last-leader-id=server-3bd8108a-b0da-b639-4f5c-52daa4705472
2026-03-29T01:09:28.319Z [INFO]  server-c716643f-246d-8c77-d271-1f47ee5c8bba: entering candidate state: node="Node at c716643f-246d-8c77-d271-1f47ee5c8bba [Candidate]" term=3
2026-03-29T01:09:28.319Z [WARN]  server-a8458347-a220-f769-ba00-df4d803dae29: rejecting vote request since we have a leader: from=c716643f-246d-8c77-d271-1f47ee5c8bba leader=3bd8108a-b0da-b639-4f5c-52daa4705472 leader-id=server-3bd8108a-b0da-b639-4f5c-52daa4705472
2026-03-29T01:09:28.319Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to make requestVote RPC: target="{Voter server-3bd8108a-b0da-b639-4f5c-52daa4705472 3bd8108a-b0da-b639-4f5c-52daa4705472}" error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472" term=3
2026-03-29T01:09:28.366Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2026-03-29T01:09:28.366Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering candidate state: node="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Candidate]" term=3
2026-03-29T01:09:28.366Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29" term=3
2026-03-29T01:09:28.366Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba" term=3
2026-03-29T01:09:28.370Z [WARN]  server-c716643f-246d-8c77-d271-1f47ee5c8bba: Election timeout reached, restarting election
2026-03-29T01:09:28.370Z [INFO]  server-c716643f-246d-8c77-d271-1f47ee5c8bba: entering candidate state: node="Node at c716643f-246d-8c77-d271-1f47ee5c8bba [Candidate]" term=4
2026-03-29T01:09:28.370Z [WARN]  server-a8458347-a220-f769-ba00-df4d803dae29: heartbeat timeout reached, starting election: last-leader-addr=3bd8108a-b0da-b639-4f5c-52daa4705472 last-leader-id=server-3bd8108a-b0da-b639-4f5c-52daa4705472
2026-03-29T01:09:28.370Z [INFO]  server-a8458347-a220-f769-ba00-df4d803dae29: entering candidate state: node="Node at a8458347-a220-f769-ba00-df4d803dae29 [Candidate]" term=3
2026-03-29T01:09:28.370Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to make requestVote RPC: target="{Voter server-3bd8108a-b0da-b639-4f5c-52daa4705472 3bd8108a-b0da-b639-4f5c-52daa4705472}" error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472" term=4
2026-03-29T01:09:28.370Z [INFO]  server-a8458347-a220-f769-ba00-df4d803dae29: entering follower state: follower="Node at a8458347-a220-f769-ba00-df4d803dae29 [Follower]" leader-address= leader-id=
2026-03-29T01:09:28.370Z [INFO]  server-c716643f-246d-8c77-d271-1f47ee5c8bba: election won: term=4 tally=2
2026-03-29T01:09:28.370Z [INFO]  server-c716643f-246d-8c77-d271-1f47ee5c8bba: entering leader state: leader="Node at c716643f-246d-8c77-d271-1f47ee5c8bba [Leader]"
2026-03-29T01:09:28.370Z [ERROR] server-a8458347-a220-f769-ba00-df4d803dae29: failed to make requestVote RPC: target="{Voter server-3bd8108a-b0da-b639-4f5c-52daa4705472 3bd8108a-b0da-b639-4f5c-52daa4705472}" error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472" term=3
2026-03-29T01:09:28.370Z [INFO]  server-c716643f-246d-8c77-d271-1f47ee5c8bba: added peer, starting replication: peer=server-a8458347-a220-f769-ba00-df4d803dae29
2026-03-29T01:09:28.370Z [INFO]  server-c716643f-246d-8c77-d271-1f47ee5c8bba: added peer, starting replication: peer=server-3bd8108a-b0da-b639-4f5c-52daa4705472
2026-03-29T01:09:28.370Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to appendEntries to: peer="{Voter server-3bd8108a-b0da-b639-4f5c-52daa4705472 3bd8108a-b0da-b639-4f5c-52daa4705472}" error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.370Z [INFO]  server-c716643f-246d-8c77-d271-1f47ee5c8bba: pipelining replication: peer="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}"
2026-03-29T01:09:28.378Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=10ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.387Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to appendEntries to: peer="{Voter server-3bd8108a-b0da-b639-4f5c-52daa4705472 3bd8108a-b0da-b639-4f5c-52daa4705472}" error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.394Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=10ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.408Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to appendEntries to: peer="{Voter server-3bd8108a-b0da-b639-4f5c-52daa4705472 3bd8108a-b0da-b639-4f5c-52daa4705472}" error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.414Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=10ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.420Z [WARN]  server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to contact: server-id=server-3bd8108a-b0da-b639-4f5c-52daa4705472 time=50.329613ms
2026-03-29T01:09:28.433Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=20ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.438Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to appendEntries to: peer="{Voter server-3bd8108a-b0da-b639-4f5c-52daa4705472 3bd8108a-b0da-b639-4f5c-52daa4705472}" error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.459Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.461Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: Election timeout reached, restarting election
2026-03-29T01:09:28.461Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering candidate state: node="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Candidate]" term=4
2026-03-29T01:09:28.461Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba" term=4
2026-03-29T01:09:28.461Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29" term=4
2026-03-29T01:09:28.467Z [WARN]  server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to contact: server-id=server-3bd8108a-b0da-b639-4f5c-52daa4705472 time=96.902095ms
2026-03-29T01:09:28.489Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to appendEntries to: peer="{Voter server-3bd8108a-b0da-b639-4f5c-52daa4705472 3bd8108a-b0da-b639-4f5c-52daa4705472}" error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.491Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.515Z [WARN]  server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to contact: server-id=server-3bd8108a-b0da-b639-4f5c-52daa4705472 time=144.759138ms
2026-03-29T01:09:28.524Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.529Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: Election timeout reached, restarting election
2026-03-29T01:09:28.529Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering candidate state: node="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Candidate]" term=5
2026-03-29T01:09:28.529Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba" term=5
2026-03-29T01:09:28.529Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29" term=5
2026-03-29T01:09:28.558Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.579Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to appendEntries to: peer="{Voter server-3bd8108a-b0da-b639-4f5c-52daa4705472 3bd8108a-b0da-b639-4f5c-52daa4705472}" error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.593Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.610Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: Election timeout reached, restarting election
2026-03-29T01:09:28.610Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering candidate state: node="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Candidate]" term=6
2026-03-29T01:09:28.610Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29" term=6
2026-03-29T01:09:28.610Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba" term=6
2026-03-29T01:09:28.626Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.661Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.694Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.695Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: Election timeout reached, restarting election
2026-03-29T01:09:28.695Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering candidate state: node="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Candidate]" term=7
2026-03-29T01:09:28.695Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29" term=7
2026-03-29T01:09:28.695Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba" term=7
2026-03-29T01:09:28.727Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.746Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to appendEntries to: peer="{Voter server-3bd8108a-b0da-b639-4f5c-52daa4705472 3bd8108a-b0da-b639-4f5c-52daa4705472}" error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.753Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: Election timeout reached, restarting election
2026-03-29T01:09:28.753Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering candidate state: node="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Candidate]" term=8
2026-03-29T01:09:28.753Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29" term=8
2026-03-29T01:09:28.753Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba" term=8
2026-03-29T01:09:28.759Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.794Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.828Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.830Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: Election timeout reached, restarting election
2026-03-29T01:09:28.830Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering candidate state: node="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Candidate]" term=9
2026-03-29T01:09:28.830Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba" term=9
2026-03-29T01:09:28.830Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29" term=9
2026-03-29T01:09:28.863Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.898Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.898Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: Election timeout reached, restarting election
2026-03-29T01:09:28.898Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering candidate state: node="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Candidate]" term=10
2026-03-29T01:09:28.898Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba" term=10
2026-03-29T01:09:28.898Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29" term=10
2026-03-29T01:09:28.931Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.967Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:28.977Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: Election timeout reached, restarting election
2026-03-29T01:09:28.977Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering candidate state: node="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Candidate]" term=11
2026-03-29T01:09:28.977Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba" term=11
2026-03-29T01:09:28.977Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29" term=11
2026-03-29T01:09:29.000Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.032Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.057Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: Election timeout reached, restarting election
2026-03-29T01:09:29.057Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering candidate state: node="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Candidate]" term=12
2026-03-29T01:09:29.057Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba" term=12
2026-03-29T01:09:29.057Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29" term=12
2026-03-29T01:09:29.064Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.076Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to appendEntries to: peer="{Voter server-3bd8108a-b0da-b639-4f5c-52daa4705472 3bd8108a-b0da-b639-4f5c-52daa4705472}" error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.100Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.130Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: Election timeout reached, restarting election
2026-03-29T01:09:29.130Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering candidate state: node="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Candidate]" term=13
2026-03-29T01:09:29.130Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba" term=13
2026-03-29T01:09:29.131Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29" term=13
2026-03-29T01:09:29.133Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.168Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.202Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.231Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: Election timeout reached, restarting election
2026-03-29T01:09:29.231Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering candidate state: node="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Candidate]" term=14
2026-03-29T01:09:29.231Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29" term=14
2026-03-29T01:09:29.231Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba" term=14
2026-03-29T01:09:29.233Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.269Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.303Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.314Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: Election timeout reached, restarting election
2026-03-29T01:09:29.314Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering candidate state: node="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Candidate]" term=15
2026-03-29T01:09:29.314Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29" term=15
2026-03-29T01:09:29.314Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba" term=15
2026-03-29T01:09:29.335Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.370Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.384Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: Election timeout reached, restarting election
2026-03-29T01:09:29.384Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering candidate state: node="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Candidate]" term=16
2026-03-29T01:09:29.384Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29" term=16
2026-03-29T01:09:29.384Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba" term=16
2026-03-29T01:09:29.405Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.440Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.476Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.477Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: Election timeout reached, restarting election
2026-03-29T01:09:29.477Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering candidate state: node="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Candidate]" term=17
2026-03-29T01:09:29.477Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba" term=17
2026-03-29T01:09:29.477Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29" term=17
2026-03-29T01:09:29.512Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.544Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.575Z [WARN]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: Election timeout reached, restarting election
2026-03-29T01:09:29.575Z [INFO]  server-3bd8108a-b0da-b639-4f5c-52daa4705472: entering candidate state: node="Node at 3bd8108a-b0da-b639-4f5c-52daa4705472 [Candidate]" term=18
2026-03-29T01:09:29.575Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-c716643f-246d-8c77-d271-1f47ee5c8bba c716643f-246d-8c77-d271-1f47ee5c8bba}" error="failed to connect to peer: c716643f-246d-8c77-d271-1f47ee5c8bba" term=18
2026-03-29T01:09:29.575Z [ERROR] server-3bd8108a-b0da-b639-4f5c-52daa4705472: failed to make requestVote RPC: target="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}" error="failed to connect to peer: a8458347-a220-f769-ba00-df4d803dae29" term=18
2026-03-29T01:09:29.577Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.609Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to heartbeat to: peer=3bd8108a-b0da-b639-4f5c-52daa4705472 backoff time=25ms error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
    raft_test.go:3028: 
        	Error Trace:	/build/reproducible-path/golang-github-hashicorp-raft-1.5.0/obj-i686-linux-gnu/src/github.com/hashicorp/raft/raft_test.go:3028
        	Error:      	Not equal: 
        	            	expected: 0x1
        	            	actual  : 0x0
        	Test:       	TestRaft_VoteWithNoIDNoAddr
2026-03-29T01:09:29.624Z [ERROR] server-c716643f-246d-8c77-d271-1f47ee5c8bba: failed to appendEntries to: peer="{Voter server-3bd8108a-b0da-b639-4f5c-52daa4705472 3bd8108a-b0da-b639-4f5c-52daa4705472}" error="failed to connect to peer: 3bd8108a-b0da-b639-4f5c-52daa4705472"
2026-03-29T01:09:29.624Z [INFO]  server-c716643f-246d-8c77-d271-1f47ee5c8bba: aborting pipeline replication: peer="{Voter server-a8458347-a220-f769-ba00-df4d803dae29 a8458347-a220-f769-ba00-df4d803dae29}"
--- FAIL: TestRaft_VoteWithNoIDNoAddr (1.74s)
=== RUN   TestRaft_runFollower_State_Transition
=== RUN   TestRaft_runFollower_State_Transition/NonVoter
2026-03-29T01:09:29.625Z [INFO]  127.0.0.1:35807 :: starting node: addr=127.0.0.1:35807
2026-03-29T01:09:29.625Z [INFO]  127.0.0.1:35807 :: initial configuration: index=0 servers=[]
2026-03-29T01:09:29.625Z [INFO]  127.0.0.1:35807 :: entering follower state: follower="Node at 127.0.0.1:35807 [Follower]" leader-address= leader-id=
2026-03-29T01:09:29.698Z [WARN]  127.0.0.1:35807 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
=== RUN   TestRaft_runFollower_State_Transition/Voter
2026-03-29T01:09:29.776Z [INFO]  127.0.0.1:38055 :: starting node: addr=127.0.0.1:38055
2026-03-29T01:09:29.776Z [INFO]  127.0.0.1:38055 :: initial configuration: index=0 servers=[]
2026-03-29T01:09:29.776Z [INFO]  127.0.0.1:38055 :: entering follower state: follower="Node at 127.0.0.1:38055 [Follower]" leader-address= leader-id=
2026-03-29T01:09:29.838Z [WARN]  127.0.0.1:38055 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
=== RUN   TestRaft_runFollower_State_Transition/Not_in_Config
2026-03-29T01:09:29.927Z [INFO]  127.0.0.1:41991 :: starting node: addr=127.0.0.1:41991
2026-03-29T01:09:29.927Z [INFO]  127.0.0.1:41991 :: initial configuration: index=0 servers=[]
2026-03-29T01:09:29.927Z [INFO]  127.0.0.1:41991 :: entering follower state: follower="Node at 127.0.0.1:41991 [Follower]" leader-address= leader-id=
2026-03-29T01:09:29.995Z [WARN]  127.0.0.1:41991 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
--- PASS: TestRaft_runFollower_State_Transition (0.45s)
    --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s)
=== RUN   TestRaft_runFollower_ReloadTimeoutConfigs
2026-03-29T01:09:30.077Z [INFO]  127.0.0.1:33583 :: starting node: addr=127.0.0.1:33583
2026-03-29T01:09:30.077Z [INFO]  127.0.0.1:33583 :: initial configuration: index=0 servers=[]
2026-03-29T01:09:30.077Z [INFO]  127.0.0.1:33583 :: entering follower state: follower="Node at 127.0.0.1:33583 [Follower]" leader-address= leader-id=
2026-03-29T01:09:30.077Z [WARN]  127.0.0.1:33583 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
--- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s)
=== RUN   TestSaturationMetric
=== RUN   TestSaturationMetric/without_smoothing
--- PASS: TestSaturationMetric (0.00s)
    --- PASS: TestSaturationMetric/without_smoothing (0.00s)
=== RUN   TestSaturationMetric_IncorrectUsage
=== RUN   TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_first
--- PASS: TestSaturationMetric_IncorrectUsage (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s)
=== RUN   TestTCPTransport_BadAddr
--- PASS: TestTCPTransport_BadAddr (0.00s)
=== RUN   TestTCPTransport_EmptyAddr
--- PASS: TestTCPTransport_EmptyAddr (0.00s)
=== RUN   TestTCPTransport_WithAdvertise
--- PASS: TestTCPTransport_WithAdvertise (0.00s)
=== RUN   TestTransport_StartStop
--- PASS: TestTransport_StartStop (0.00s)
=== RUN   TestTransport_AppendEntries
--- PASS: TestTransport_AppendEntries (0.00s)
=== RUN   TestTransport_AppendEntriesPipeline
--- PASS: TestTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestTransport_RequestVote
--- PASS: TestTransport_RequestVote (0.00s)
=== RUN   TestTransport_InstallSnapshot
--- PASS: TestTransport_InstallSnapshot (0.00s)
=== RUN   TestTransport_EncodeDecode
--- PASS: TestTransport_EncodeDecode (0.00s)
=== RUN   TestRandomTimeout
--- PASS: TestRandomTimeout (0.00s)
=== RUN   TestNewSeed
--- PASS: TestNewSeed (0.00s)
=== RUN   TestRandomTimeout_NoTime
--- PASS: TestRandomTimeout_NoTime (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
=== RUN   TestMax
--- PASS: TestMax (0.00s)
=== RUN   TestGenerateUUID
--- PASS: TestGenerateUUID (0.00s)
=== RUN   TestBackoff
--- PASS: TestBackoff (0.00s)
=== RUN   TestOverrideNotifyBool
--- PASS: TestOverrideNotifyBool (0.00s)
FAIL
FAIL	github.com/hashicorp/raft	114.028s
?   	github.com/hashicorp/raft/bench	[no test files]
FAIL
dh_auto_test: error: cd obj-i686-linux-gnu && go test -vet=off -v -p 21 github.com/hashicorp/raft github.com/hashicorp/raft/bench returned exit code 1
make[1]: *** [debian/rules:17: override_dh_auto_test] Error 25
make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
make: *** [debian/rules:14: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
I: copying local configuration
E: Failed autobuilding of package
I: user script /srv/workspace/pbuilder/34756/tmp/hooks/C01_cleanup starting
debug output: disk usage on i-capture-the-hostname at Sun Mar 29 01:09:30 UTC 2026
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           3.9G     0  3.9G   0% /dev/shm

I: user script /srv/workspace/pbuilder/34756/tmp/hooks/C01_cleanup finished
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /srv/workspace/pbuilder/34756 and its subdirectories