I: pbuilder: network access will be disabled during build
I: Current time: Tue Apr 22 11:20:14 +14 2025
I: pbuilder-time-stamp: 1745270414
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/trixie-reproducible-base.tgz]
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: policy-rc.d already exists
I: Copying source file
I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc]
I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz]
I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz]
I: Extracting source
dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: unsupported subcommand
dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying 0001-Skip-unreliable-tests.patch
I: Not using root during the build.
I: Installing the build-deps
I: user script /srv/workspace/pbuilder/2469999/tmp/hooks/D01_modify_environment starting
debug: Running on codethink04-arm64.
I: Changing host+domainname to test build reproducibility
I: Adding a custom variable just for the fun of it...
I: Changing /bin/sh to bash
'/bin/sh' -> '/bin/bash'
lrwxrwxrwx 1 root root 9 Apr 21 21:20 /bin/sh -> /bin/bash
I: Setting pbuilder2's login shell to /bin/bash
I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other
I: user script /srv/workspace/pbuilder/2469999/tmp/hooks/D01_modify_environment finished
I: user script /srv/workspace/pbuilder/2469999/tmp/hooks/D02_print_environment starting
I: set
  BASH=/bin/sh
  BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:globskipdots:hostcomplete:interactive_comments:patsub_replacement:progcomp:promptvars:sourcepath
  BASH_ALIASES=()
  BASH_ARGC=()
  BASH_ARGV=()
  BASH_CMDS=()
  BASH_LINENO=([0]="12" [1]="0")
  BASH_LOADABLES_PATH=/usr/local/lib/bash:/usr/lib/bash:/opt/local/lib/bash:/usr/pkg/lib/bash:/opt/pkg/lib/bash:.
  BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment")
  BASH_VERSINFO=([0]="5" [1]="2" [2]="37" [3]="1" [4]="release" [5]="aarch64-unknown-linux-gnu")
  BASH_VERSION='5.2.37(1)-release'
  BUILDDIR=/build/reproducible-path
  BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other'
  BUILDUSERNAME=pbuilder2
  BUILD_ARCH=arm64
  DEBIAN_FRONTEND=noninteractive
  DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=12 '
  DIRSTACK=()
  DISTRIBUTION=trixie
  EUID=0
  FUNCNAME=([0]="Echo" [1]="main")
  GROUPS=()
  HOME=/root
  HOSTNAME=i-capture-the-hostname
  HOSTTYPE=aarch64
  HOST_ARCH=arm64
  IFS=' 	
  '
  INVOCATION_ID=fa287825ebeb4bee842e002204432eff
  LANG=C
  LANGUAGE=nl_BE:nl
  LC_ALL=C
  MACHTYPE=aarch64-unknown-linux-gnu
  MAIL=/var/mail/root
  OPTERR=1
  OPTIND=1
  OSTYPE=linux-gnu
  PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path
  PBCURRENTCOMMANDLINEOPERATION=build
  PBUILDER_OPERATION=build
  PBUILDER_PKGDATADIR=/usr/share/pbuilder
  PBUILDER_PKGLIBDIR=/usr/lib/pbuilder
  PBUILDER_SYSCONFDIR=/etc
  PIPESTATUS=([0]="0")
  POSIXLY_CORRECT=y
  PPID=2469999
  PS4='+ '
  PWD=/
  SHELL=/bin/bash
  SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix
  SHLVL=3
  SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.qt4E2m4z/pbuilderrc_zgWJ --distribution trixie --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/trixie-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.qt4E2m4z/b2 --logfile b2/build.log golang-github-hashicorp-raft_1.5.0-1.dsc'
  SUDO_GID=109
  SUDO_UID=104
  SUDO_USER=jenkins
  TERM=unknown
  TZ=/usr/share/zoneinfo/Etc/GMT-14
  UID=0
  USER=root
  _='I: set'
  http_proxy=http://192.168.101.4:3128
I: uname -a
  Linux i-capture-the-hostname 6.1.0-33-cloud-arm64 #1 SMP Debian 6.1.133-1 (2025-04-10) aarch64 GNU/Linux
I: ls -l /bin
  lrwxrwxrwx 1 root root 7 Mar  4 11:20 /bin -> usr/bin
I: user script /srv/workspace/pbuilder/2469999/tmp/hooks/D02_print_environment finished
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: arm64
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 19954 files and directories currently installed.)
Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ...
Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however:
  Package debhelper-compat is not installed.
 pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however:
  Package dh-sequence-golang is not installed.
 pbuilder-satisfydepends-dummy depends on golang-any; however:
  Package golang-any is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however:
  Package golang-github-armon-go-metrics-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however:
  Package golang-github-hashicorp-go-hclog-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however:
  Package golang-github-hashicorp-go-msgpack-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however:
  Package golang-github-stretchr-testify-dev is not installed.

Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
Building tag database...
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
The following NEW packages will be installed:
  autoconf{a} automake{a} autopoint{a} autotools-dev{a} binutils-gold{a} binutils-gold-aarch64-linux-gnu{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.24-go{a} golang-1.24-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-prometheus-sigv4-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libunistring5{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} 
The following packages are RECOMMENDED but will NOT be installed:
  curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 
0 packages upgraded, 108 newly installed, 0 to remove and 0 not upgraded.
Need to get 120 MB of archives. After unpacking 687 MB will be used.
Writing extended state information...
Get: 1 http://deb.debian.org/debian trixie/main arm64 sensible-utils all 0.0.24 [24.8 kB]
Get: 2 http://deb.debian.org/debian trixie/main arm64 libmagic-mgc arm64 1:5.46-5 [338 kB]
Get: 3 http://deb.debian.org/debian trixie/main arm64 libmagic1t64 arm64 1:5.46-5 [103 kB]
Get: 4 http://deb.debian.org/debian trixie/main arm64 file arm64 1:5.46-5 [43.7 kB]
Get: 5 http://deb.debian.org/debian trixie/main arm64 gettext-base arm64 0.23.1-1 [241 kB]
Get: 6 http://deb.debian.org/debian trixie/main arm64 libuchardet0 arm64 0.0.8-1+b2 [69.2 kB]
Get: 7 http://deb.debian.org/debian trixie/main arm64 groff-base arm64 1.23.0-7 [1129 kB]
Get: 8 http://deb.debian.org/debian trixie/main arm64 bsdextrautils arm64 2.41-4 [93.9 kB]
Get: 9 http://deb.debian.org/debian trixie/main arm64 libpipeline1 arm64 1.5.8-1 [40.2 kB]
Get: 10 http://deb.debian.org/debian trixie/main arm64 man-db arm64 2.13.0-1 [1404 kB]
Get: 11 http://deb.debian.org/debian trixie/main arm64 m4 arm64 1.4.19-7 [285 kB]
Get: 12 http://deb.debian.org/debian trixie/main arm64 autoconf all 2.72-3 [493 kB]
Get: 13 http://deb.debian.org/debian trixie/main arm64 autotools-dev all 20240727.1 [60.2 kB]
Get: 14 http://deb.debian.org/debian trixie/main arm64 automake all 1:1.17-4 [862 kB]
Get: 15 http://deb.debian.org/debian trixie/main arm64 autopoint all 0.23.1-1 [770 kB]
Get: 16 http://deb.debian.org/debian trixie/main arm64 binutils-gold-aarch64-linux-gnu arm64 2.44-2 [1700 kB]
Get: 17 http://deb.debian.org/debian trixie/main arm64 binutils-gold arm64 2.44-2 [202 kB]
Get: 18 http://deb.debian.org/debian trixie/main arm64 libdebhelper-perl all 13.24.2 [90.9 kB]
Get: 19 http://deb.debian.org/debian trixie/main arm64 libtool all 2.5.4-4 [539 kB]
Get: 20 http://deb.debian.org/debian trixie/main arm64 dh-autoreconf all 20 [17.1 kB]
Get: 21 http://deb.debian.org/debian trixie/main arm64 libarchive-zip-perl all 1.68-1 [104 kB]
Get: 22 http://deb.debian.org/debian trixie/main arm64 libfile-stripnondeterminism-perl all 1.14.1-2 [19.7 kB]
Get: 23 http://deb.debian.org/debian trixie/main arm64 dh-strip-nondeterminism all 1.14.1-2 [8620 B]
Get: 24 http://deb.debian.org/debian trixie/main arm64 libelf1t64 arm64 0.192-4 [189 kB]
Get: 25 http://deb.debian.org/debian trixie/main arm64 dwz arm64 0.15-1+b1 [102 kB]
Get: 26 http://deb.debian.org/debian trixie/main arm64 libunistring5 arm64 1.3-2 [453 kB]
Get: 27 http://deb.debian.org/debian trixie/main arm64 libxml2 arm64 2.12.7+dfsg+really2.9.14-0.4 [629 kB]
Get: 28 http://deb.debian.org/debian trixie/main arm64 gettext arm64 0.23.1-1 [1610 kB]
Get: 29 http://deb.debian.org/debian trixie/main arm64 intltool-debian all 0.35.0+20060710.6 [22.9 kB]
Get: 30 http://deb.debian.org/debian trixie/main arm64 po-debconf all 1.0.21+nmu1 [248 kB]
Get: 31 http://deb.debian.org/debian trixie/main arm64 debhelper all 13.24.2 [919 kB]
Get: 32 http://deb.debian.org/debian trixie/main arm64 dh-golang all 1.63 [24.4 kB]
Get: 33 http://deb.debian.org/debian trixie/main arm64 golang-1.24-src all 1.24.2-2 [21.2 MB]
Get: 34 http://deb.debian.org/debian trixie/main arm64 golang-1.24-go arm64 1.24.2-2 [24.6 MB]
Get: 35 http://deb.debian.org/debian trixie/main arm64 golang-src all 2:1.24~2 [5136 B]
Get: 36 http://deb.debian.org/debian trixie/main arm64 golang-go arm64 2:1.24~2 [44.3 kB]
Get: 37 http://deb.debian.org/debian trixie/main arm64 golang-any arm64 2:1.24~2 [5216 B]
Get: 38 http://deb.debian.org/debian trixie/main arm64 golang-github-alecthomas-units-dev all 0.0~git20240927.0f3dac3-1 [8880 B]
Get: 39 http://deb.debian.org/debian trixie/main arm64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB]
Get: 40 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB]
Get: 41 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-sys-dev all 0.22.0-1 [431 kB]
Get: 42 http://deb.debian.org/debian trixie/main arm64 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B]
Get: 43 http://deb.debian.org/debian trixie/main arm64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB]
Get: 44 http://deb.debian.org/debian trixie/main arm64 golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB]
Get: 45 http://deb.debian.org/debian trixie/main arm64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB]
Get: 46 http://deb.debian.org/debian trixie/main arm64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB]
Get: 47 http://deb.debian.org/debian trixie/main arm64 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB]
Get: 48 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB]
Get: 49 http://deb.debian.org/debian trixie/main arm64 golang-github-stretchr-testify-dev all 1.10.0-1 [78.4 kB]
Get: 50 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB]
Get: 51 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB]
Get: 52 http://deb.debian.org/debian trixie/main arm64 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB]
Get: 53 http://deb.debian.org/debian trixie/main arm64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B]
Get: 54 http://deb.debian.org/debian trixie/main arm64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB]
Get: 55 http://deb.debian.org/debian trixie/main arm64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB]
Get: 56 http://deb.debian.org/debian trixie/main arm64 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB]
Get: 57 http://deb.debian.org/debian trixie/main arm64 golang-google-protobuf-dev all 1.36.5-1 [796 kB]
Get: 58 http://deb.debian.org/debian trixie/main arm64 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB]
Get: 59 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B]
Get: 60 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB]
Get: 61 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB]
Get: 62 http://deb.debian.org/debian trixie/main arm64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB]
Get: 63 http://deb.debian.org/debian trixie/main arm64 golang-github-beorn7-perks-dev all 1.0.1-3 [12.2 kB]
Get: 64 http://deb.debian.org/debian trixie/main arm64 golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB]
Get: 65 http://deb.debian.org/debian trixie/main arm64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B]
Get: 66 http://deb.debian.org/debian trixie/main arm64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB]
Get: 67 http://deb.debian.org/debian trixie/main arm64 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB]
Get: 68 http://deb.debian.org/debian trixie/main arm64 golang-github-klauspost-compress-dev all 1.18.0+ds1-1 [33.3 MB]
Get: 69 http://deb.debian.org/debian trixie/main arm64 golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB]
Get: 70 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB]
Get: 71 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-yaml.v2-dev all 2.4.0-5 [60.5 kB]
Get: 72 http://deb.debian.org/debian trixie/main arm64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB]
Get: 73 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-term-dev all 0.22.0-1 [14.8 kB]
Get: 74 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-text-dev all 0.22.0-1 [3995 kB]
Get: 75 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-crypto-dev all 1:0.25.0-1 [1682 kB]
Get: 76 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-net-dev all 1:0.27.0-1 [898 kB]
Get: 77 http://deb.debian.org/debian trixie/main arm64 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB]
Get: 78 http://deb.debian.org/debian trixie/main arm64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB]
Get: 79 http://deb.debian.org/debian trixie/main arm64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB]
Get: 80 http://deb.debian.org/debian trixie/main arm64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB]
Get: 81 http://deb.debian.org/debian trixie/main arm64 golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B]
Get: 82 http://deb.debian.org/debian trixie/main arm64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B]
Get: 83 http://deb.debian.org/debian trixie/main arm64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-4 [14.6 kB]
Get: 84 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-sigv4-dev all 0.1.2-2 [8484 B]
Get: 85 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-oauth2-dev all 0.27.0-1 [37.2 kB]
Get: 86 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB]
Get: 87 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-common-dev all 0.62.0-3 [118 kB]
Get: 88 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB]
Get: 89 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-procfs-dev all 0.15.1-1 [184 kB]
Get: 90 http://deb.debian.org/debian trixie/main arm64 golang-github-prometheus-client-golang-dev all 1.21.1-1 [194 kB]
Get: 91 http://deb.debian.org/debian trixie/main arm64 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB]
Get: 92 http://deb.debian.org/debian trixie/main arm64 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB]
Get: 93 http://deb.debian.org/debian trixie/main arm64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B]
Get: 94 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB]
Get: 95 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B]
Get: 96 http://deb.debian.org/debian trixie/main arm64 libsasl2-modules-db arm64 2.1.28+dfsg1-9 [20.1 kB]
Get: 97 http://deb.debian.org/debian trixie/main arm64 libsasl2-2 arm64 2.1.28+dfsg1-9 [55.6 kB]
Get: 98 http://deb.debian.org/debian trixie/main arm64 libssl-dev arm64 3.5.0-1 [3383 kB]
Get: 99 http://deb.debian.org/debian trixie/main arm64 libsasl2-dev arm64 2.1.28+dfsg1-9 [229 kB]
Get: 100 http://deb.debian.org/debian trixie/main arm64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB]
Get: 101 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-mod-dev all 0.19.0-1 [104 kB]
Get: 102 http://deb.debian.org/debian trixie/main arm64 golang-github-yuin-goldmark-dev all 1.7.4-1 [113 kB]
Get: 103 http://deb.debian.org/debian trixie/main arm64 golang-golang-x-tools-dev all 1:0.25.0+ds-1 [1608 kB]
Get: 104 http://deb.debian.org/debian trixie/main arm64 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB]
Get: 105 http://deb.debian.org/debian trixie/main arm64 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB]
Get: 106 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-2 [31.2 kB]
Get: 107 http://deb.debian.org/debian trixie/main arm64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB]
Get: 108 http://deb.debian.org/debian trixie/main arm64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB]
Fetched 120 MB in 1s (92.2 MB/s)
Preconfiguring packages ...
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 19954 files and directories currently installed.)
Preparing to unpack .../000-sensible-utils_0.0.24_all.deb ...
Unpacking sensible-utils (0.0.24) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../001-libmagic-mgc_1%3a5.46-5_arm64.deb ...
Unpacking libmagic-mgc (1:5.46-5) ...
Selecting previously unselected package libmagic1t64:arm64.
Preparing to unpack .../002-libmagic1t64_1%3a5.46-5_arm64.deb ...
Unpacking libmagic1t64:arm64 (1:5.46-5) ...
Selecting previously unselected package file.
Preparing to unpack .../003-file_1%3a5.46-5_arm64.deb ...
Unpacking file (1:5.46-5) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../004-gettext-base_0.23.1-1_arm64.deb ...
Unpacking gettext-base (0.23.1-1) ...
Selecting previously unselected package libuchardet0:arm64.
Preparing to unpack .../005-libuchardet0_0.0.8-1+b2_arm64.deb ...
Unpacking libuchardet0:arm64 (0.0.8-1+b2) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../006-groff-base_1.23.0-7_arm64.deb ...
Unpacking groff-base (1.23.0-7) ...
Selecting previously unselected package bsdextrautils.
Preparing to unpack .../007-bsdextrautils_2.41-4_arm64.deb ...
Unpacking bsdextrautils (2.41-4) ...
Selecting previously unselected package libpipeline1:arm64.
Preparing to unpack .../008-libpipeline1_1.5.8-1_arm64.deb ...
Unpacking libpipeline1:arm64 (1.5.8-1) ...
Selecting previously unselected package man-db.
Preparing to unpack .../009-man-db_2.13.0-1_arm64.deb ...
Unpacking man-db (2.13.0-1) ...
Selecting previously unselected package m4.
Preparing to unpack .../010-m4_1.4.19-7_arm64.deb ...
Unpacking m4 (1.4.19-7) ...
Selecting previously unselected package autoconf.
Preparing to unpack .../011-autoconf_2.72-3_all.deb ...
Unpacking autoconf (2.72-3) ...
Selecting previously unselected package autotools-dev.
Preparing to unpack .../012-autotools-dev_20240727.1_all.deb ...
Unpacking autotools-dev (20240727.1) ...
Selecting previously unselected package automake.
Preparing to unpack .../013-automake_1%3a1.17-4_all.deb ...
Unpacking automake (1:1.17-4) ...
Selecting previously unselected package autopoint.
Preparing to unpack .../014-autopoint_0.23.1-1_all.deb ...
Unpacking autopoint (0.23.1-1) ...
Selecting previously unselected package binutils-gold-aarch64-linux-gnu.
Preparing to unpack .../015-binutils-gold-aarch64-linux-gnu_2.44-2_arm64.deb ...
Unpacking binutils-gold-aarch64-linux-gnu (2.44-2) ...
Selecting previously unselected package binutils-gold.
Preparing to unpack .../016-binutils-gold_2.44-2_arm64.deb ...
Unpacking binutils-gold (2.44-2) ...
Selecting previously unselected package libdebhelper-perl.
Preparing to unpack .../017-libdebhelper-perl_13.24.2_all.deb ...
Unpacking libdebhelper-perl (13.24.2) ...
Selecting previously unselected package libtool.
Preparing to unpack .../018-libtool_2.5.4-4_all.deb ...
Unpacking libtool (2.5.4-4) ...
Selecting previously unselected package dh-autoreconf.
Preparing to unpack .../019-dh-autoreconf_20_all.deb ...
Unpacking dh-autoreconf (20) ...
Selecting previously unselected package libarchive-zip-perl.
Preparing to unpack .../020-libarchive-zip-perl_1.68-1_all.deb ...
Unpacking libarchive-zip-perl (1.68-1) ...
Selecting previously unselected package libfile-stripnondeterminism-perl.
Preparing to unpack .../021-libfile-stripnondeterminism-perl_1.14.1-2_all.deb ...
Unpacking libfile-stripnondeterminism-perl (1.14.1-2) ...
Selecting previously unselected package dh-strip-nondeterminism.
Preparing to unpack .../022-dh-strip-nondeterminism_1.14.1-2_all.deb ...
Unpacking dh-strip-nondeterminism (1.14.1-2) ...
Selecting previously unselected package libelf1t64:arm64.
Preparing to unpack .../023-libelf1t64_0.192-4_arm64.deb ...
Unpacking libelf1t64:arm64 (0.192-4) ...
Selecting previously unselected package dwz.
Preparing to unpack .../024-dwz_0.15-1+b1_arm64.deb ...
Unpacking dwz (0.15-1+b1) ...
Selecting previously unselected package libunistring5:arm64.
Preparing to unpack .../025-libunistring5_1.3-2_arm64.deb ...
Unpacking libunistring5:arm64 (1.3-2) ...
Selecting previously unselected package libxml2:arm64.
Preparing to unpack .../026-libxml2_2.12.7+dfsg+really2.9.14-0.4_arm64.deb ...
Unpacking libxml2:arm64 (2.12.7+dfsg+really2.9.14-0.4) ...
Selecting previously unselected package gettext.
Preparing to unpack .../027-gettext_0.23.1-1_arm64.deb ...
Unpacking gettext (0.23.1-1) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../028-intltool-debian_0.35.0+20060710.6_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.6) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../029-po-debconf_1.0.21+nmu1_all.deb ...
Unpacking po-debconf (1.0.21+nmu1) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../030-debhelper_13.24.2_all.deb ...
Unpacking debhelper (13.24.2) ...
Selecting previously unselected package dh-golang.
Preparing to unpack .../031-dh-golang_1.63_all.deb ...
Unpacking dh-golang (1.63) ...
Selecting previously unselected package golang-1.24-src.
Preparing to unpack .../032-golang-1.24-src_1.24.2-2_all.deb ...
Unpacking golang-1.24-src (1.24.2-2) ...
Selecting previously unselected package golang-1.24-go.
Preparing to unpack .../033-golang-1.24-go_1.24.2-2_arm64.deb ...
Unpacking golang-1.24-go (1.24.2-2) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../034-golang-src_2%3a1.24~2_all.deb ...
Unpacking golang-src (2:1.24~2) ...
Selecting previously unselected package golang-go:arm64.
Preparing to unpack .../035-golang-go_2%3a1.24~2_arm64.deb ...
Unpacking golang-go:arm64 (2:1.24~2) ...
Selecting previously unselected package golang-any:arm64.
Preparing to unpack .../036-golang-any_2%3a1.24~2_arm64.deb ...
Unpacking golang-any:arm64 (2:1.24~2) ...
Selecting previously unselected package golang-github-alecthomas-units-dev.
Preparing to unpack .../037-golang-github-alecthomas-units-dev_0.0~git20240927.0f3dac3-1_all.deb ...
Unpacking golang-github-alecthomas-units-dev (0.0~git20240927.0f3dac3-1) ...
Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev.
Preparing to unpack .../038-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ...
Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev.
Preparing to unpack .../039-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ...
Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Selecting previously unselected package golang-golang-x-sys-dev.
Preparing to unpack .../040-golang-golang-x-sys-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-sys-dev (0.22.0-1) ...
Selecting previously unselected package golang-github-mattn-go-isatty-dev.
Preparing to unpack .../041-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ...
Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Selecting previously unselected package golang-github-mattn-go-colorable-dev.
Preparing to unpack .../042-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ...
Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Selecting previously unselected package golang-github-fatih-color-dev.
Preparing to unpack .../043-golang-github-fatih-color-dev_1.17.0-1_all.deb ...
Unpacking golang-github-fatih-color-dev (1.17.0-1) ...
Selecting previously unselected package golang-github-davecgh-go-spew-dev.
Preparing to unpack .../044-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ...
Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Selecting previously unselected package golang-github-pmezard-go-difflib-dev.
Preparing to unpack .../045-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ...
Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Selecting previously unselected package golang-github-stretchr-objx-dev.
Preparing to unpack .../046-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ...
Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ...
Selecting previously unselected package golang-gopkg-yaml.v3-dev.
Preparing to unpack .../047-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ...
Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Selecting previously unselected package golang-github-stretchr-testify-dev.
Preparing to unpack .../048-golang-github-stretchr-testify-dev_1.10.0-1_all.deb ...
Unpacking golang-github-stretchr-testify-dev (1.10.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-hclog-dev.
Preparing to unpack .../049-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ...
Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev.
Preparing to unpack .../050-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ...
Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Selecting previously unselected package golang-github-pkg-errors-dev.
Preparing to unpack .../051-golang-github-pkg-errors-dev_0.9.1-3_all.deb ...
Unpacking golang-github-pkg-errors-dev (0.9.1-3) ...
Selecting previously unselected package golang-github-tv42-httpunix-dev.
Preparing to unpack .../052-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ...
Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev.
Preparing to unpack .../053-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ...
Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Selecting previously unselected package golang-github-datadog-datadog-go-dev.
Preparing to unpack .../054-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ...
Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Selecting previously unselected package golang-github-google-go-cmp-dev.
Preparing to unpack .../055-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ...
Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ...
Selecting previously unselected package golang-google-protobuf-dev.
Preparing to unpack .../056-golang-google-protobuf-dev_1.36.5-1_all.deb ...
Unpacking golang-google-protobuf-dev (1.36.5-1) ...
Selecting previously unselected package golang-github-golang-protobuf-1-5-dev.
Preparing to unpack .../057-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ...
Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Selecting previously unselected package golang-github-hashicorp-go-uuid-dev.
Preparing to unpack .../058-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ...
Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Selecting previously unselected package golang-github-hashicorp-golang-lru-dev.
Preparing to unpack .../059-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ...
Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev.
Preparing to unpack .../060-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ...
Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Selecting previously unselected package golang-github-pascaldekloe-goe-dev.
Preparing to unpack .../061-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ...
Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Selecting previously unselected package golang-github-beorn7-perks-dev.
Preparing to unpack .../062-golang-github-beorn7-perks-dev_1.0.1-3_all.deb ...
Unpacking golang-github-beorn7-perks-dev (1.0.1-3) ...
Selecting previously unselected package golang-github-cespare-xxhash-dev.
Preparing to unpack .../063-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ...
Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ...
Selecting previously unselected package golang-github-modern-go-concurrent-dev.
Preparing to unpack .../064-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ...
Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Selecting previously unselected package golang-github-modern-go-reflect2-dev.
Preparing to unpack .../065-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ...
Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-json-iterator-go-dev.
Preparing to unpack .../066-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ...
Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ...
Selecting previously unselected package golang-github-klauspost-compress-dev.
Preparing to unpack .../067-golang-github-klauspost-compress-dev_1.18.0+ds1-1_all.deb ...
Unpacking golang-github-klauspost-compress-dev (1.18.0+ds1-1) ...
Selecting previously unselected package golang-github-kylelemons-godebug-dev.
Preparing to unpack .../068-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ...
Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Selecting previously unselected package golang-github-prometheus-client-model-dev.
Preparing to unpack .../069-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ...
Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ...
Selecting previously unselected package golang-gopkg-yaml.v2-dev.
Preparing to unpack .../070-golang-gopkg-yaml.v2-dev_2.4.0-5_all.deb ...
Unpacking golang-gopkg-yaml.v2-dev (2.4.0-5) ...
Selecting previously unselected package golang-github-jmespath-go-jmespath-dev.
Preparing to unpack .../071-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ...
Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Selecting previously unselected package golang-golang-x-term-dev.
Preparing to unpack .../072-golang-golang-x-term-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-term-dev (0.22.0-1) ...
Selecting previously unselected package golang-golang-x-text-dev.
Preparing to unpack .../073-golang-golang-x-text-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-text-dev (0.22.0-1) ...
Selecting previously unselected package golang-golang-x-crypto-dev.
Preparing to unpack .../074-golang-golang-x-crypto-dev_1%3a0.25.0-1_all.deb ...
Unpacking golang-golang-x-crypto-dev (1:0.25.0-1) ...
Selecting previously unselected package golang-golang-x-net-dev.
Preparing to unpack .../075-golang-golang-x-net-dev_1%3a0.27.0-1_all.deb ...
Unpacking golang-golang-x-net-dev (1:0.27.0-1) ...
Selecting previously unselected package golang-github-aws-aws-sdk-go-dev.
Preparing to unpack .../076-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ...
Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Selecting previously unselected package golang-github-go-logfmt-logfmt-dev.
Preparing to unpack .../077-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ...
Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Selecting previously unselected package golang-github-go-kit-log-dev.
Preparing to unpack .../078-golang-github-go-kit-log-dev_0.2.1-1_all.deb ...
Unpacking golang-github-go-kit-log-dev (0.2.1-1) ...
Selecting previously unselected package golang-github-julienschmidt-httprouter-dev.
Preparing to unpack .../079-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ...
Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Selecting previously unselected package golang-github-munnerz-goautoneg-dev.
Preparing to unpack .../080-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ...
Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Selecting previously unselected package golang-github-jpillora-backoff-dev.
Preparing to unpack .../081-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ...
Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev.
Preparing to unpack .../082-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-4_all.deb ...
Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ...
Selecting previously unselected package golang-github-prometheus-sigv4-dev.
Preparing to unpack .../083-golang-github-prometheus-sigv4-dev_0.1.2-2_all.deb ...
Unpacking golang-github-prometheus-sigv4-dev (0.1.2-2) ...
Selecting previously unselected package golang-golang-x-oauth2-dev.
Preparing to unpack .../084-golang-golang-x-oauth2-dev_0.27.0-1_all.deb ...
Unpacking golang-golang-x-oauth2-dev (0.27.0-1) ...
Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev.
Preparing to unpack .../085-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ...
Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Selecting previously unselected package golang-github-prometheus-common-dev.
Preparing to unpack .../086-golang-github-prometheus-common-dev_0.62.0-3_all.deb ...
Unpacking golang-github-prometheus-common-dev (0.62.0-3) ...
Selecting previously unselected package golang-golang-x-sync-dev.
Preparing to unpack .../087-golang-golang-x-sync-dev_0.7.0-1_all.deb ...
Unpacking golang-golang-x-sync-dev (0.7.0-1) ...
Selecting previously unselected package golang-github-prometheus-procfs-dev.
Preparing to unpack .../088-golang-github-prometheus-procfs-dev_0.15.1-1_all.deb ...
Unpacking golang-github-prometheus-procfs-dev (0.15.1-1) ...
Selecting previously unselected package golang-github-prometheus-client-golang-dev.
Preparing to unpack .../089-golang-github-prometheus-client-golang-dev_1.21.1-1_all.deb ...
Unpacking golang-github-prometheus-client-golang-dev (1.21.1-1) ...
Selecting previously unselected package golang-github-armon-go-metrics-dev.
Preparing to unpack .../090-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ...
Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ...
Selecting previously unselected package golang-github-creack-pty-dev.
Preparing to unpack .../091-golang-github-creack-pty-dev_1.1.21-1_all.deb ...
Unpacking golang-github-creack-pty-dev (1.1.21-1) ...
Selecting previously unselected package golang-github-vmihailenco-tagparser-dev.
Preparing to unpack .../092-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ...
Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev.
Preparing to unpack .../093-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ...
Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Selecting previously unselected package golang-gopkg-tomb.v2-dev.
Preparing to unpack .../094-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ...
Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Selecting previously unselected package libsasl2-modules-db:arm64.
Preparing to unpack .../095-libsasl2-modules-db_2.1.28+dfsg1-9_arm64.deb ...
Unpacking libsasl2-modules-db:arm64 (2.1.28+dfsg1-9) ...
Selecting previously unselected package libsasl2-2:arm64.
Preparing to unpack .../096-libsasl2-2_2.1.28+dfsg1-9_arm64.deb ...
Unpacking libsasl2-2:arm64 (2.1.28+dfsg1-9) ...
Selecting previously unselected package libssl-dev:arm64.
Preparing to unpack .../097-libssl-dev_3.5.0-1_arm64.deb ...
Unpacking libssl-dev:arm64 (3.5.0-1) ...
Selecting previously unselected package libsasl2-dev.
Preparing to unpack .../098-libsasl2-dev_2.1.28+dfsg1-9_arm64.deb ...
Unpacking libsasl2-dev (2.1.28+dfsg1-9) ...
Selecting previously unselected package golang-github-kr-text-dev.
Preparing to unpack .../099-golang-github-kr-text-dev_0.2.0-2_all.deb ...
Unpacking golang-github-kr-text-dev (0.2.0-2) ...
Selecting previously unselected package golang-golang-x-mod-dev.
Preparing to unpack .../100-golang-golang-x-mod-dev_0.19.0-1_all.deb ...
Unpacking golang-golang-x-mod-dev (0.19.0-1) ...
Selecting previously unselected package golang-github-yuin-goldmark-dev.
Preparing to unpack .../101-golang-github-yuin-goldmark-dev_1.7.4-1_all.deb ...
Unpacking golang-github-yuin-goldmark-dev (1.7.4-1) ...
Selecting previously unselected package golang-golang-x-tools-dev.
Preparing to unpack .../102-golang-golang-x-tools-dev_1%3a0.25.0+ds-1_all.deb ...
Unpacking golang-golang-x-tools-dev (1:0.25.0+ds-1) ...
Selecting previously unselected package golang-github-rogpeppe-go-internal-dev.
Preparing to unpack .../103-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ...
Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Selecting previously unselected package golang-github-kr-pretty-dev.
Preparing to unpack .../104-golang-github-kr-pretty-dev_0.3.1-1_all.deb ...
Unpacking golang-github-kr-pretty-dev (0.3.1-1) ...
Selecting previously unselected package golang-gopkg-check.v1-dev.
Preparing to unpack .../105-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-2_all.deb ...
Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ...
Selecting previously unselected package golang-gopkg-mgo.v2-dev.
Preparing to unpack .../106-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ...
Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev.
Preparing to unpack .../107-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ...
Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-1.24-src (1.24.2-2) ...
Setting up libpipeline1:arm64 (1.5.8-1) ...
Setting up golang-github-google-go-cmp-dev (0.6.0-1) ...
Setting up golang-gopkg-yaml.v2-dev (2.4.0-5) ...
Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Setting up bsdextrautils (2.41-4) ...
Setting up golang-github-creack-pty-dev (1.1.21-1) ...
Setting up golang-github-stretchr-objx-dev (0.5.2-1) ...
Setting up libmagic-mgc (1:5.46-5) ...
Setting up golang-github-pkg-errors-dev (0.9.1-3) ...
Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Setting up libarchive-zip-perl (1.68-1) ...
Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Setting up libdebhelper-perl (13.24.2) ...
Setting up golang-golang-x-sys-dev (0.22.0-1) ...
Setting up libmagic1t64:arm64 (1:5.46-5) ...
Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Setting up gettext-base (0.23.1-1) ...
Setting up m4 (1.4.19-7) ...
Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Setting up binutils-gold-aarch64-linux-gnu (2.44-2) ...
Setting up golang-golang-x-term-dev (0.22.0-1) ...
Setting up file (1:5.46-5) ...
Setting up golang-github-beorn7-perks-dev (1.0.1-3) ...
Setting up libelf1t64:arm64 (0.192-4) ...
Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Setting up libsasl2-modules-db:arm64 (2.1.28+dfsg1-9) ...
Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ...
Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Setting up autotools-dev (20240727.1) ...
Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Setting up golang-google-protobuf-dev (1.36.5-1) ...
Setting up golang-github-kr-text-dev (0.2.0-2) ...
Setting up golang-github-stretchr-testify-dev (1.10.0-1) ...
Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Setting up golang-github-yuin-goldmark-dev (1.7.4-1) ...
Setting up libunistring5:arm64 (1.3-2) ...
Setting up libssl-dev:arm64 (3.5.0-1) ...
Setting up golang-golang-x-sync-dev (0.7.0-1) ...
Setting up golang-golang-x-mod-dev (0.19.0-1) ...
Setting up autopoint (0.23.1-1) ...
Setting up golang-github-klauspost-compress-dev (1.18.0+ds1-1) ...
Setting up libsasl2-2:arm64 (2.1.28+dfsg1-9) ...
Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Setting up autoconf (2.72-3) ...
Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Setting up dwz (0.15-1+b1) ...
Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Setting up sensible-utils (0.0.24) ...
Setting up golang-golang-x-text-dev (0.22.0-1) ...
Setting up libuchardet0:arm64 (0.0.8-1+b2) ...
Setting up golang-github-prometheus-procfs-dev (0.15.1-1) ...
Setting up golang-github-alecthomas-units-dev (0.0~git20240927.0f3dac3-1) ...
Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Setting up golang-golang-x-crypto-dev (1:0.25.0-1) ...
Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ...
Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Setting up golang-golang-x-oauth2-dev (0.27.0-1) ...
Setting up golang-src (2:1.24~2) ...
Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Setting up libxml2:arm64 (2.12.7+dfsg+really2.9.14-0.4) ...
Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Setting up automake (1:1.17-4) ...
update-alternatives: using /usr/bin/automake-1.17 to provide /usr/bin/automake (automake) in auto mode
Setting up libfile-stripnondeterminism-perl (1.14.1-2) ...
Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Setting up golang-github-go-kit-log-dev (0.2.1-1) ...
Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Setting up gettext (0.23.1-1) ...
Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Setting up libtool (2.5.4-4) ...
Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Setting up binutils-gold (2.44-2) ...
Setting up golang-github-json-iterator-go-dev (1.1.12-2) ...
Setting up intltool-debian (0.35.0+20060710.6) ...
Setting up dh-autoreconf (20) ...
Setting up libsasl2-dev (2.1.28+dfsg1-9) ...
Setting up golang-1.24-go (1.24.2-2) ...
Setting up golang-golang-x-net-dev (1:0.27.0-1) ...
Setting up dh-strip-nondeterminism (1.14.1-2) ...
Setting up groff-base (1.23.0-7) ...
Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Setting up golang-github-fatih-color-dev (1.17.0-1) ...
Setting up golang-go:arm64 (2:1.24~2) ...
Setting up po-debconf (1.0.21+nmu1) ...
Setting up golang-golang-x-tools-dev (1:0.25.0+ds-1) ...
Setting up man-db (2.13.0-1) ...
Not building database; man-db/auto-update is not 'true'.
Setting up golang-any:arm64 (2:1.24~2) ...
Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Setting up golang-github-kr-pretty-dev (0.3.1-1) ...
Setting up debhelper (13.24.2) ...
Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-2) ...
Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Setting up dh-golang (1.63) ...
Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-4) ...
Setting up golang-github-prometheus-sigv4-dev (0.1.2-2) ...
Setting up golang-github-prometheus-common-dev (0.62.0-3) ...
Setting up golang-github-prometheus-client-golang-dev (1.21.1-1) ...
Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ...
Processing triggers for libc-bin (2.41-7) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Building tag database...
 -> Finished parsing the build-deps
I: Building the package
I: user script /srv/workspace/pbuilder/2469999/tmp/hooks/A99_set_merged_usr starting
Not re-configuring usrmerge for trixie
I: user script /srv/workspace/pbuilder/2469999/tmp/hooks/A99_set_merged_usr finished
hostname: Name or service not known
I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S  > ../golang-github-hashicorp-raft_1.5.0-1_source.changes
dpkg-buildpackage: info: source package golang-github-hashicorp-raft
dpkg-buildpackage: info: source version 1.5.0-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Daniel Swarbrick <dswarbrick@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture arm64
 debian/rules clean
dh clean --buildsystem=golang
   dh_auto_clean -O--buildsystem=golang
   dh_autoreconf_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules binary
dh binary --buildsystem=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-aarch64-linux-gnu && go install -trimpath -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench
internal/coverage/rtcov
internal/unsafeheader
internal/byteorder
internal/godebugs
internal/goos
internal/msan
internal/profilerecord
internal/goexperiment
internal/cpu
internal/asan
sync/atomic
internal/goarch
math/bits
internal/runtime/syscall
unicode/utf8
internal/itoa
container/list
unicode
crypto/internal/fips140/alias
crypto/internal/fips140deps/byteorder
cmp
crypto/internal/boring/sig
encoding
internal/abi
internal/chacha8rand
internal/runtime/math
crypto/internal/fips140/subtle
unicode/utf16
log/internal
math
internal/nettrace
vendor/golang.org/x/crypto/cryptobyte/asn1
vendor/golang.org/x/crypto/internal/alias
internal/bytealg
internal/runtime/atomic
crypto/internal/fips140deps/cpu
internal/runtime/sys
internal/runtime/exithook
internal/stringslite
internal/race
internal/runtime/maps
internal/sync
runtime
internal/reflectlite
sync
iter
crypto/subtle
weak
slices
maps
internal/bisect
internal/testlog
internal/singleflight
unique
errors
sort
runtime/cgo
internal/godebug
io
internal/oserror
strconv
path
math/rand/v2
github.com/hashicorp/golang-lru/simplelru
vendor/golang.org/x/net/dns/dnsmessage
syscall
bytes
strings
math/rand
hash
crypto/internal/randutil
crypto/internal/fips140deps/godebug
reflect
net/netip
internal/saferio
crypto
hash/crc32
hash/crc64
vendor/golang.org/x/text/transform
bufio
crypto/internal/fips140
crypto/internal/impl
github.com/hashicorp/go-immutable-radix
net/http/internal/ascii
crypto/internal/fips140/sha3
crypto/internal/fips140/sha256
crypto/internal/fips140/sha512
crypto/tls/internal/fips140tls
regexp/syntax
html
crypto/sha3
time
internal/syscall/unix
internal/syscall/execenv
crypto/internal/fips140/hmac
crypto/internal/fips140hash
crypto/internal/fips140/check
crypto/internal/fips140/aes
crypto/internal/fips140/edwards25519/field
crypto/internal/fips140/nistec/fiat
crypto/internal/fips140/hkdf
crypto/internal/fips140/tls12
crypto/internal/fips140/bigmod
crypto/internal/fips140/tls13
crypto/internal/fips140/edwards25519
regexp
context
io/fs
internal/poll
crypto/internal/fips140/nistec
internal/filepathlite
os
internal/fmtsort
encoding/binary
crypto/internal/sysrand
fmt
path/filepath
os/signal
io/ioutil
internal/sysinfo
crypto/internal/entropy
crypto/internal/fips140/drbg
crypto/internal/fips140/aes/gcm
crypto/internal/fips140only
crypto/internal/fips140/ed25519
crypto/internal/fips140/mlkem
crypto/internal/fips140/rsa
crypto/internal/fips140/ecdh
crypto/internal/fips140/ecdsa
crypto/md5
crypto/rc4
net
encoding/base64
vendor/golang.org/x/crypto/internal/poly1305
golang.org/x/sys/unix
crypto/cipher
encoding/pem
crypto/internal/boring
crypto/des
vendor/golang.org/x/crypto/chacha20
math/big
log
compress/flate
net/url
crypto/aes
vendor/golang.org/x/text/unicode/norm
encoding/json
encoding/hex
crypto/ecdh
crypto/sha512
crypto/hmac
crypto/sha1
crypto/sha256
vendor/golang.org/x/text/unicode/bidi
vendor/golang.org/x/net/http2/hpack
vendor/golang.org/x/crypto/chacha20poly1305
mime
mime/quotedprintable
net/http/internal
encoding/gob
go/token
text/template/parse
flag
compress/gzip
runtime/debug
vendor/golang.org/x/text/secure/bidirule
runtime/trace
testing
text/template
vendor/golang.org/x/net/idna
crypto/rand
crypto/elliptic
crypto/internal/boring/bbig
encoding/asn1
crypto/dsa
crypto/ed25519
crypto/internal/hpke
crypto/rsa
vendor/golang.org/x/crypto/cryptobyte
crypto/x509/pkix
html/template
github.com/mattn/go-isatty
github.com/mattn/go-colorable
github.com/fatih/color
crypto/ecdsa
github.com/hashicorp/go-hclog
crypto/x509
net/textproto
vendor/golang.org/x/net/http/httpproxy
vendor/golang.org/x/net/http/httpguts
mime/multipart
crypto/tls
net/http/httptrace
net/http
github.com/armon/go-metrics
net/rpc
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/raft
github.com/hashicorp/raft/bench
   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
dh_auto_test
	cd obj-aarch64-linux-gnu && go test -vet=off -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench
=== RUN   TestCommitment_setVoters
--- PASS: TestCommitment_setVoters (0.00s)
=== RUN   TestCommitment_match_max
--- PASS: TestCommitment_match_max (0.00s)
=== RUN   TestCommitment_match_nonVoting
--- PASS: TestCommitment_match_nonVoting (0.00s)
=== RUN   TestCommitment_recalculate
--- PASS: TestCommitment_recalculate (0.00s)
=== RUN   TestCommitment_recalculate_startIndex
--- PASS: TestCommitment_recalculate_startIndex (0.00s)
=== RUN   TestCommitment_noVoterSanity
--- PASS: TestCommitment_noVoterSanity (0.00s)
=== RUN   TestCommitment_singleVoter
--- PASS: TestCommitment_singleVoter (0.00s)
=== RUN   TestConfiguration_Configuration_Clone
--- PASS: TestConfiguration_Configuration_Clone (0.00s)
=== RUN   TestConfiguration_configurations_Clone
--- PASS: TestConfiguration_configurations_Clone (0.00s)
=== RUN   TestConfiguration_hasVote
--- PASS: TestConfiguration_hasVote (0.00s)
=== RUN   TestConfiguration_checkConfiguration
--- PASS: TestConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_nextConfiguration_table
--- PASS: TestConfiguration_nextConfiguration_table (0.00s)
=== RUN   TestConfiguration_nextConfiguration_prevIndex
--- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
=== RUN   TestConfiguration_nextConfiguration_checkConfiguration
--- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_encodeDecodePeers
--- PASS: TestConfiguration_encodeDecodePeers (0.00s)
=== RUN   TestConfiguration_encodeDecodeConfiguration
--- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
=== RUN   TestDiscardSnapshotStoreImpl
--- PASS: TestDiscardSnapshotStoreImpl (0.00s)
=== RUN   TestDiscardSnapshotSinkImpl
--- PASS: TestDiscardSnapshotSinkImpl (0.00s)
=== RUN   TestFileSnapshotStoreImpl
--- PASS: TestFileSnapshotStoreImpl (0.00s)
=== RUN   TestFileSnapshotSinkImpl
--- PASS: TestFileSnapshotSinkImpl (0.00s)
=== RUN   TestFileSS_CreateSnapshotMissingParentDir
2025-04-21T21:22:41.487Z [INFO]  creating new snapshot: path=/tmp/raft1410015381/raft897924292/snapshots/3-10-1745270561487.tmp
--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s)
=== RUN   TestFileSS_CreateSnapshot
2025-04-21T21:22:41.487Z [INFO]  creating new snapshot: path=/tmp/raft1037902866/snapshots/3-10-1745270561487.tmp
2025-04-21T21:22:41.487Z [WARN]  found temporary snapshot: name=3-10-1745270561487.tmp
--- PASS: TestFileSS_CreateSnapshot (0.00s)
=== RUN   TestFileSS_CancelSnapshot
2025-04-21T21:22:41.487Z [INFO]  creating new snapshot: path=/tmp/raft2680742506/snapshots/3-10-1745270561487.tmp
--- PASS: TestFileSS_CancelSnapshot (0.00s)
=== RUN   TestFileSS_Retention
2025-04-21T21:22:41.488Z [INFO]  creating new snapshot: path=/tmp/raft3830412924/snapshots/3-10-1745270561488.tmp
2025-04-21T21:22:41.488Z [INFO]  creating new snapshot: path=/tmp/raft3830412924/snapshots/3-11-1745270561488.tmp
2025-04-21T21:22:41.488Z [INFO]  creating new snapshot: path=/tmp/raft3830412924/snapshots/3-12-1745270561488.tmp
2025-04-21T21:22:41.488Z [INFO]  reaping snapshot: path=/tmp/raft3830412924/snapshots/3-10-1745270561488
2025-04-21T21:22:41.488Z [INFO]  creating new snapshot: path=/tmp/raft3830412924/snapshots/3-13-1745270561488.tmp
2025-04-21T21:22:41.488Z [INFO]  reaping snapshot: path=/tmp/raft3830412924/snapshots/3-11-1745270561488
2025-04-21T21:22:41.488Z [INFO]  creating new snapshot: path=/tmp/raft3830412924/snapshots/3-14-1745270561488.tmp
2025-04-21T21:22:41.488Z [INFO]  reaping snapshot: path=/tmp/raft3830412924/snapshots/3-12-1745270561488
--- PASS: TestFileSS_Retention (0.00s)
=== RUN   TestFileSS_BadPerm
--- PASS: TestFileSS_BadPerm (0.00s)
=== RUN   TestFileSS_MissingParentDir
--- PASS: TestFileSS_MissingParentDir (0.00s)
=== RUN   TestFileSS_Ordering
2025-04-21T21:22:41.493Z [INFO]  creating new snapshot: path=/tmp/raft2180398292/snapshots/5-130350-1745270561493.tmp
2025-04-21T21:22:41.493Z [INFO]  creating new snapshot: path=/tmp/raft2180398292/snapshots/36-204917-1745270561493.tmp
--- PASS: TestFileSS_Ordering (0.00s)
=== RUN   TestDeferFutureSuccess
--- PASS: TestDeferFutureSuccess (0.00s)
=== RUN   TestDeferFutureError
--- PASS: TestDeferFutureError (0.00s)
=== RUN   TestDeferFutureConcurrent
--- PASS: TestDeferFutureConcurrent (0.00s)
=== RUN   TestInmemSnapshotStoreImpl
--- PASS: TestInmemSnapshotStoreImpl (0.00s)
=== RUN   TestInmemSnapshotSinkImpl
--- PASS: TestInmemSnapshotSinkImpl (0.00s)
=== RUN   TestInmemSS_CreateSnapshot
--- PASS: TestInmemSS_CreateSnapshot (0.00s)
=== RUN   TestInmemSS_OpenSnapshotTwice
--- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
=== RUN   TestInmemTransportImpl
--- PASS: TestInmemTransportImpl (0.00s)
=== RUN   TestInmemTransportWriteTimeout
--- PASS: TestInmemTransportWriteTimeout (0.01s)
=== RUN   TestRaft_Integ
--- SKIP: TestRaft_Integ (0.00s)
=== RUN   TestRaft_RestartFollower_LongInitialHeartbeat
--- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s)
=== RUN   TestLogCache
--- PASS: TestLogCache (0.00s)
=== RUN   TestLogCacheWithBackendStoreError
--- PASS: TestLogCacheWithBackendStoreError (0.00s)
=== RUN   TestOldestLog
=== RUN   TestOldestLog/empty_logs
=== RUN   TestOldestLog/simple_case
--- PASS: TestOldestLog (0.00s)
    --- PASS: TestOldestLog/empty_logs (0.00s)
    --- PASS: TestOldestLog/simple_case (0.00s)
=== RUN   TestEmitsLogStoreMetrics
--- PASS: TestEmitsLogStoreMetrics (0.01s)
=== RUN   TestNetworkTransport_CloseStreams
--- PASS: TestNetworkTransport_CloseStreams (0.06s)
=== RUN   TestNetworkTransport_StartStop
--- PASS: TestNetworkTransport_StartStop (0.00s)
=== RUN   TestNetworkTransport_Heartbeat_FastPath
--- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s)
=== RUN   TestNetworkTransport_AppendEntries
--- PASS: TestNetworkTransport_AppendEntries (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline
--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.03s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_CloseStreams
--- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.03s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2
2025-04-21T21:22:41.693Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10
2025-04-21T21:22:41.745Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s)
=== RUN   TestNetworkTransport_RequestVote
2025-04-21T21:22:41.797Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_RequestVote (0.02s)
=== RUN   TestNetworkTransport_InstallSnapshot
--- PASS: TestNetworkTransport_InstallSnapshot (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode
--- PASS: TestNetworkTransport_EncodeDecode (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode_AddressProvider
--- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
=== RUN   TestNetworkTransport_PooledConn
--- PASS: TestNetworkTransport_PooledConn (0.04s)
=== RUN   TestNetworkTransport_ListenBackoff
    net_transport_test.go:839: countingWriter: 2025-04-21T21:22:41.854Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-04-21T21:22:41.861Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-04-21T21:22:41.873Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-04-21T21:22:41.897Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-04-21T21:22:41.941Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-04-21T21:22:42.025Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-04-21T21:22:42.189Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-04-21T21:22:42.509Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-04-21T21:22:43.153Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-04-21T21:22:44.157Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-04-21T21:22:45.161Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
--- PASS: TestNetworkTransport_ListenBackoff (4.00s)
=== RUN   TestPeersJSON_BadConfiguration
--- PASS: TestPeersJSON_BadConfiguration (0.00s)
=== RUN   TestPeersJSON_ReadPeersJSON
--- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
=== RUN   TestPeersJSON_ReadConfigJSON
--- PASS: TestPeersJSON_ReadConfigJSON (0.00s)
=== RUN   TestRaft_StartStop
2025-04-21T21:22:45.857Z [INFO]  server-1157bff9-f687-e4e8-84ef-b8bac7db5318: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1157bff9-f687-e4e8-84ef-b8bac7db5318 Address:1157bff9-f687-e4e8-84ef-b8bac7db5318}]"
--- PASS: TestRaft_StartStop (0.00s)
=== RUN   TestRaft_AfterShutdown
2025-04-21T21:22:45.858Z [INFO]  server-198ea465-b269-d7d9-493a-bb204edc550f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-198ea465-b269-d7d9-493a-bb204edc550f Address:198ea465-b269-d7d9-493a-bb204edc550f}]"
--- PASS: TestRaft_AfterShutdown (0.00s)
=== RUN   TestRaft_LiveBootstrap
2025-04-21T21:22:45.858Z [INFO]  server-35d79e95-703f-f2dd-37e4-041d77ae783f: initial configuration: index=0 servers=[]
2025-04-21T21:22:45.858Z [INFO]  server-66a66cbd-3a3a-8427-9534-79ff66376b03: initial configuration: index=0 servers=[]
2025-04-21T21:22:45.858Z [INFO]  server-035a00da-2276-330a-347c-08889ca60557: initial configuration: index=0 servers=[]
2025-04-21T21:22:45.858Z [INFO]  server-35d79e95-703f-f2dd-37e4-041d77ae783f: entering follower state: follower="Node at 35d79e95-703f-f2dd-37e4-041d77ae783f [Follower]" leader-address= leader-id=
2025-04-21T21:22:45.858Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:22:45.858Z [INFO]  server-66a66cbd-3a3a-8427-9534-79ff66376b03: entering follower state: follower="Node at 66a66cbd-3a3a-8427-9534-79ff66376b03 [Follower]" leader-address= leader-id=
2025-04-21T21:22:45.858Z [INFO]  server-035a00da-2276-330a-347c-08889ca60557: entering follower state: follower="Node at 035a00da-2276-330a-347c-08889ca60557 [Follower]" leader-address= leader-id=
2025-04-21T21:22:45.913Z [WARN]  server-35d79e95-703f-f2dd-37e4-041d77ae783f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:45.913Z [INFO]  server-35d79e95-703f-f2dd-37e4-041d77ae783f: entering candidate state: node="Node at 35d79e95-703f-f2dd-37e4-041d77ae783f [Candidate]" term=2
2025-04-21T21:22:45.913Z [INFO]  server-35d79e95-703f-f2dd-37e4-041d77ae783f: election won: term=2 tally=2
2025-04-21T21:22:45.913Z [INFO]  server-35d79e95-703f-f2dd-37e4-041d77ae783f: entering leader state: leader="Node at 35d79e95-703f-f2dd-37e4-041d77ae783f [Leader]"
2025-04-21T21:22:45.913Z [INFO]  server-35d79e95-703f-f2dd-37e4-041d77ae783f: added peer, starting replication: peer=server-66a66cbd-3a3a-8427-9534-79ff66376b03
2025-04-21T21:22:45.913Z [INFO]  server-35d79e95-703f-f2dd-37e4-041d77ae783f: added peer, starting replication: peer=server-035a00da-2276-330a-347c-08889ca60557
2025-04-21T21:22:45.913Z [WARN]  server-035a00da-2276-330a-347c-08889ca60557: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-04-21T21:22:45.913Z [WARN]  server-35d79e95-703f-f2dd-37e4-041d77ae783f: appendEntries rejected, sending older logs: peer="{Voter server-035a00da-2276-330a-347c-08889ca60557 035a00da-2276-330a-347c-08889ca60557}" next=1
2025-04-21T21:22:45.913Z [WARN]  server-66a66cbd-3a3a-8427-9534-79ff66376b03: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-04-21T21:22:45.913Z [WARN]  server-35d79e95-703f-f2dd-37e4-041d77ae783f: appendEntries rejected, sending older logs: peer="{Voter server-66a66cbd-3a3a-8427-9534-79ff66376b03 66a66cbd-3a3a-8427-9534-79ff66376b03}" next=1
2025-04-21T21:22:45.914Z [INFO]  server-35d79e95-703f-f2dd-37e4-041d77ae783f: pipelining replication: peer="{Voter server-66a66cbd-3a3a-8427-9534-79ff66376b03 66a66cbd-3a3a-8427-9534-79ff66376b03}"
2025-04-21T21:22:45.914Z [WARN]  server-66a66cbd-3a3a-8427-9534-79ff66376b03: rejecting vote request since our last term is greater: candidate=35d79e95-703f-f2dd-37e4-041d77ae783f last-term=2 last-candidate-term=1
2025-04-21T21:22:45.929Z [INFO]  server-35d79e95-703f-f2dd-37e4-041d77ae783f: pipelining replication: peer="{Voter server-035a00da-2276-330a-347c-08889ca60557 035a00da-2276-330a-347c-08889ca60557}"
2025-04-21T21:22:46.021Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:22:45.914176686 +0000 UTC m=+4.435835969 (2 nodes), 55.653214ms from start of poll, 55.877375ms from cluster start. Timeout at 2025-04-21 21:22:46.019178926 +0000 UTC m=+4.540838249, 105.00228ms after stability
2025-04-21T21:22:46.021Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:46.129Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:46.021299646 +0000 UTC m=+4.542958969 (1 nodes), 1.12µs from start of poll, 163.000375ms from cluster start. Timeout at 2025-04-21 21:22:46.126302806 +0000 UTC m=+4.647962129, 105.00316ms after stability
2025-04-21T21:22:46.145Z [INFO]  server-35d79e95-703f-f2dd-37e4-041d77ae783f: aborting pipeline replication: peer="{Voter server-035a00da-2276-330a-347c-08889ca60557 035a00da-2276-330a-347c-08889ca60557}"
2025-04-21T21:22:46.145Z [INFO]  server-35d79e95-703f-f2dd-37e4-041d77ae783f: aborting pipeline replication: peer="{Voter server-66a66cbd-3a3a-8427-9534-79ff66376b03 66a66cbd-3a3a-8427-9534-79ff66376b03}"
2025-04-21T21:22:46.649Z [ERROR] server-35d79e95-703f-f2dd-37e4-041d77ae783f: failed to heartbeat to: peer=035a00da-2276-330a-347c-08889ca60557 backoff time=10ms error="command timed out"
--- PASS: TestRaft_LiveBootstrap (0.79s)
=== RUN   TestRaft_LiveBootstrap_From_NonVoter
2025-04-21T21:22:46.649Z [INFO]  server-2ae71a9b-2b51-ed9e-34eb-b86d5afeb9c7: initial configuration: index=0 servers=[]
2025-04-21T21:22:46.649Z [INFO]  server-9656ef50-5cc8-93bb-29ac-275b8521ff1e: initial configuration: index=0 servers=[]
2025-04-21T21:22:46.649Z [INFO]  server-2ae71a9b-2b51-ed9e-34eb-b86d5afeb9c7: entering follower state: follower="Node at 2ae71a9b-2b51-ed9e-34eb-b86d5afeb9c7 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s)
=== RUN   TestRaft_RecoverCluster_NoState
2025-04-21T21:22:46.649Z [INFO]  server-5de81c02-5b53-ab6b-fc0f-c1401f968108: initial configuration: index=0 servers=[]
--- PASS: TestRaft_RecoverCluster_NoState (0.00s)
=== RUN   TestRaft_RecoverCluster
=== RUN   TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs
2025-04-21T21:22:46.650Z [INFO]  server-946655f3-3118-6a41-37dd-caf363304da9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-946655f3-3118-6a41-37dd-caf363304da9 Address:946655f3-3118-6a41-37dd-caf363304da9} {Suffrage:Voter ID:server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77 Address:f72effc1-ce8a-8ab2-bd70-61bfefb40d77} {Suffrage:Voter ID:server-0828ab42-0da7-deda-d037-153eafa64158 Address:0828ab42-0da7-deda-d037-153eafa64158}]"
2025-04-21T21:22:46.650Z [INFO]  server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-946655f3-3118-6a41-37dd-caf363304da9 Address:946655f3-3118-6a41-37dd-caf363304da9} {Suffrage:Voter ID:server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77 Address:f72effc1-ce8a-8ab2-bd70-61bfefb40d77} {Suffrage:Voter ID:server-0828ab42-0da7-deda-d037-153eafa64158 Address:0828ab42-0da7-deda-d037-153eafa64158}]"
2025-04-21T21:22:46.650Z [INFO]  server-0828ab42-0da7-deda-d037-153eafa64158: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-946655f3-3118-6a41-37dd-caf363304da9 Address:946655f3-3118-6a41-37dd-caf363304da9} {Suffrage:Voter ID:server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77 Address:f72effc1-ce8a-8ab2-bd70-61bfefb40d77} {Suffrage:Voter ID:server-0828ab42-0da7-deda-d037-153eafa64158 Address:0828ab42-0da7-deda-d037-153eafa64158}]"
2025-04-21T21:22:46.650Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:46.650Z [INFO]  server-946655f3-3118-6a41-37dd-caf363304da9: entering follower state: follower="Node at 946655f3-3118-6a41-37dd-caf363304da9 [Follower]" leader-address= leader-id=
2025-04-21T21:22:46.650Z [INFO]  server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77: entering follower state: follower="Node at f72effc1-ce8a-8ab2-bd70-61bfefb40d77 [Follower]" leader-address= leader-id=
2025-04-21T21:22:46.650Z [INFO]  server-0828ab42-0da7-deda-d037-153eafa64158: entering follower state: follower="Node at 0828ab42-0da7-deda-d037-153eafa64158 [Follower]" leader-address= leader-id=
2025-04-21T21:22:46.717Z [WARN]  server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:46.717Z [INFO]  server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77: entering candidate state: node="Node at f72effc1-ce8a-8ab2-bd70-61bfefb40d77 [Candidate]" term=2
2025-04-21T21:22:46.717Z [INFO]  server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77: election won: term=2 tally=2
2025-04-21T21:22:46.717Z [INFO]  server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77: entering leader state: leader="Node at f72effc1-ce8a-8ab2-bd70-61bfefb40d77 [Leader]"
2025-04-21T21:22:46.717Z [INFO]  server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77: added peer, starting replication: peer=server-946655f3-3118-6a41-37dd-caf363304da9
2025-04-21T21:22:46.717Z [INFO]  server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77: added peer, starting replication: peer=server-0828ab42-0da7-deda-d037-153eafa64158
2025-04-21T21:22:46.717Z [INFO]  server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77: pipelining replication: peer="{Voter server-0828ab42-0da7-deda-d037-153eafa64158 0828ab42-0da7-deda-d037-153eafa64158}"
2025-04-21T21:22:46.717Z [INFO]  server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77: pipelining replication: peer="{Voter server-946655f3-3118-6a41-37dd-caf363304da9 946655f3-3118-6a41-37dd-caf363304da9}"
2025-04-21T21:22:46.825Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:46.717423244 +0000 UTC m=+5.239082567 (1 nodes), 67.309773ms from start of poll, 67.501374ms from cluster start. Timeout at 2025-04-21 21:22:46.822426484 +0000 UTC m=+5.344085807, 105.00324ms after stability
2025-04-21T21:22:46.825Z [INFO]  server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77: aborting pipeline replication: peer="{Voter server-0828ab42-0da7-deda-d037-153eafa64158 0828ab42-0da7-deda-d037-153eafa64158}"
2025-04-21T21:22:46.825Z [INFO]  server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77: aborting pipeline replication: peer="{Voter server-946655f3-3118-6a41-37dd-caf363304da9 946655f3-3118-6a41-37dd-caf363304da9}"
2025-04-21T21:22:46.825Z [INFO]  creating new snapshot: path=/tmp/raft3906723348/snapshots/2-2-1745270566825.tmp
2025-04-21T21:22:46.825Z [INFO]  server-946655f3-3118-6a41-37dd-caf363304da9: starting restore from snapshot: id=2-2-1745270566825 last-index=2 last-term=2 size-in-bytes=1
2025-04-21T21:22:46.825Z [INFO]  server-946655f3-3118-6a41-37dd-caf363304da9: snapshot restore progress: id=2-2-1745270566825 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-04-21T21:22:46.825Z [INFO]  server-946655f3-3118-6a41-37dd-caf363304da9: restored from snapshot: id=2-2-1745270566825 last-index=2 last-term=2 size-in-bytes=1
2025-04-21T21:22:46.825Z [INFO]  server-946655f3-3118-6a41-37dd-caf363304da9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-946655f3-3118-6a41-37dd-caf363304da9 Address:946655f3-3118-6a41-37dd-caf363304da9} {Suffrage:Voter ID:server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77 Address:f72effc1-ce8a-8ab2-bd70-61bfefb40d77} {Suffrage:Voter ID:server-0828ab42-0da7-deda-d037-153eafa64158 Address:0828ab42-0da7-deda-d037-153eafa64158}]"
2025-04-21T21:22:46.825Z [INFO]  creating new snapshot: path=/tmp/raft3146756270/snapshots/2-2-1745270566825.tmp
2025-04-21T21:22:46.825Z [INFO]  server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77: starting restore from snapshot: id=2-2-1745270566825 last-index=2 last-term=2 size-in-bytes=1
2025-04-21T21:22:46.826Z [INFO]  server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77: snapshot restore progress: id=2-2-1745270566825 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-04-21T21:22:46.826Z [INFO]  server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77: restored from snapshot: id=2-2-1745270566825 last-index=2 last-term=2 size-in-bytes=1
2025-04-21T21:22:46.826Z [INFO]  server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-946655f3-3118-6a41-37dd-caf363304da9 Address:946655f3-3118-6a41-37dd-caf363304da9} {Suffrage:Voter ID:server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77 Address:f72effc1-ce8a-8ab2-bd70-61bfefb40d77} {Suffrage:Voter ID:server-0828ab42-0da7-deda-d037-153eafa64158 Address:0828ab42-0da7-deda-d037-153eafa64158}]"
2025-04-21T21:22:46.826Z [INFO]  creating new snapshot: path=/tmp/raft2423863093/snapshots/2-2-1745270566826.tmp
2025-04-21T21:22:46.826Z [INFO]  server-0828ab42-0da7-deda-d037-153eafa64158: starting restore from snapshot: id=2-2-1745270566826 last-index=2 last-term=2 size-in-bytes=1
2025-04-21T21:22:46.826Z [INFO]  server-0828ab42-0da7-deda-d037-153eafa64158: snapshot restore progress: id=2-2-1745270566826 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-04-21T21:22:46.826Z [INFO]  server-0828ab42-0da7-deda-d037-153eafa64158: restored from snapshot: id=2-2-1745270566826 last-index=2 last-term=2 size-in-bytes=1
2025-04-21T21:22:46.826Z [INFO]  server-0828ab42-0da7-deda-d037-153eafa64158: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-946655f3-3118-6a41-37dd-caf363304da9 Address:946655f3-3118-6a41-37dd-caf363304da9} {Suffrage:Voter ID:server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77 Address:f72effc1-ce8a-8ab2-bd70-61bfefb40d77} {Suffrage:Voter ID:server-0828ab42-0da7-deda-d037-153eafa64158 Address:0828ab42-0da7-deda-d037-153eafa64158}]"
2025-04-21T21:22:46.826Z [INFO]  server-946655f3-3118-6a41-37dd-caf363304da9: entering follower state: follower="Node at 946655f3-3118-6a41-37dd-caf363304da9 [Follower]" leader-address= leader-id=
2025-04-21T21:22:46.826Z [INFO]  server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77: entering follower state: follower="Node at f72effc1-ce8a-8ab2-bd70-61bfefb40d77 [Follower]" leader-address= leader-id=
2025-04-21T21:22:46.826Z [INFO]  server-0828ab42-0da7-deda-d037-153eafa64158: entering follower state: follower="Node at 0828ab42-0da7-deda-d037-153eafa64158 [Follower]" leader-address= leader-id=
2025-04-21T21:22:46.901Z [WARN]  server-0828ab42-0da7-deda-d037-153eafa64158: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:46.901Z [INFO]  server-0828ab42-0da7-deda-d037-153eafa64158: entering candidate state: node="Node at 0828ab42-0da7-deda-d037-153eafa64158 [Candidate]" term=3
2025-04-21T21:22:46.901Z [INFO]  server-0828ab42-0da7-deda-d037-153eafa64158: election won: term=3 tally=2
2025-04-21T21:22:46.901Z [INFO]  server-0828ab42-0da7-deda-d037-153eafa64158: entering leader state: leader="Node at 0828ab42-0da7-deda-d037-153eafa64158 [Leader]"
2025-04-21T21:22:46.901Z [INFO]  server-0828ab42-0da7-deda-d037-153eafa64158: added peer, starting replication: peer=server-946655f3-3118-6a41-37dd-caf363304da9
2025-04-21T21:22:46.901Z [INFO]  server-0828ab42-0da7-deda-d037-153eafa64158: added peer, starting replication: peer=server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77
2025-04-21T21:22:46.901Z [INFO]  server-0828ab42-0da7-deda-d037-153eafa64158: pipelining replication: peer="{Voter server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77 f72effc1-ce8a-8ab2-bd70-61bfefb40d77}"
2025-04-21T21:22:46.901Z [INFO]  server-0828ab42-0da7-deda-d037-153eafa64158: pipelining replication: peer="{Voter server-946655f3-3118-6a41-37dd-caf363304da9 946655f3-3118-6a41-37dd-caf363304da9}"
2025-04-21T21:22:47.145Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:47.253Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:47.145271879 +0000 UTC m=+5.666931202 (1 nodes), 15.16µs from start of poll, 495.350009ms from cluster start. Timeout at 2025-04-21 21:22:47.250280719 +0000 UTC m=+5.771940042, 105.00884ms after stability
2025-04-21T21:22:47.253Z [INFO]  server-0828ab42-0da7-deda-d037-153eafa64158: aborting pipeline replication: peer="{Voter server-f72effc1-ce8a-8ab2-bd70-61bfefb40d77 f72effc1-ce8a-8ab2-bd70-61bfefb40d77}"
2025-04-21T21:22:47.253Z [INFO]  server-0828ab42-0da7-deda-d037-153eafa64158: aborting pipeline replication: peer="{Voter server-946655f3-3118-6a41-37dd-caf363304da9 946655f3-3118-6a41-37dd-caf363304da9}"
2025-04-21T21:22:47.757Z [ERROR] server-0828ab42-0da7-deda-d037-153eafa64158: failed to heartbeat to: peer=f72effc1-ce8a-8ab2-bd70-61bfefb40d77 backoff time=10ms error="command timed out"
=== RUN   TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs
2025-04-21T21:22:47.757Z [INFO]  server-af1eccbd-b541-e759-d552-5ec329322d2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af1eccbd-b541-e759-d552-5ec329322d2d Address:af1eccbd-b541-e759-d552-5ec329322d2d} {Suffrage:Voter ID:server-b016c8e3-276f-ad1c-80ed-c650bb16345f Address:b016c8e3-276f-ad1c-80ed-c650bb16345f} {Suffrage:Voter ID:server-782181c2-c4f5-9923-0058-005b60aa5e61 Address:782181c2-c4f5-9923-0058-005b60aa5e61}]"
2025-04-21T21:22:47.757Z [INFO]  server-b016c8e3-276f-ad1c-80ed-c650bb16345f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af1eccbd-b541-e759-d552-5ec329322d2d Address:af1eccbd-b541-e759-d552-5ec329322d2d} {Suffrage:Voter ID:server-b016c8e3-276f-ad1c-80ed-c650bb16345f Address:b016c8e3-276f-ad1c-80ed-c650bb16345f} {Suffrage:Voter ID:server-782181c2-c4f5-9923-0058-005b60aa5e61 Address:782181c2-c4f5-9923-0058-005b60aa5e61}]"
2025-04-21T21:22:47.757Z [INFO]  server-782181c2-c4f5-9923-0058-005b60aa5e61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af1eccbd-b541-e759-d552-5ec329322d2d Address:af1eccbd-b541-e759-d552-5ec329322d2d} {Suffrage:Voter ID:server-b016c8e3-276f-ad1c-80ed-c650bb16345f Address:b016c8e3-276f-ad1c-80ed-c650bb16345f} {Suffrage:Voter ID:server-782181c2-c4f5-9923-0058-005b60aa5e61 Address:782181c2-c4f5-9923-0058-005b60aa5e61}]"
2025-04-21T21:22:47.757Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:47.757Z [INFO]  server-af1eccbd-b541-e759-d552-5ec329322d2d: entering follower state: follower="Node at af1eccbd-b541-e759-d552-5ec329322d2d [Follower]" leader-address= leader-id=
2025-04-21T21:22:47.757Z [INFO]  server-b016c8e3-276f-ad1c-80ed-c650bb16345f: entering follower state: follower="Node at b016c8e3-276f-ad1c-80ed-c650bb16345f [Follower]" leader-address= leader-id=
2025-04-21T21:22:47.757Z [INFO]  server-782181c2-c4f5-9923-0058-005b60aa5e61: entering follower state: follower="Node at 782181c2-c4f5-9923-0058-005b60aa5e61 [Follower]" leader-address= leader-id=
2025-04-21T21:22:47.845Z [WARN]  server-b016c8e3-276f-ad1c-80ed-c650bb16345f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:47.845Z [INFO]  server-b016c8e3-276f-ad1c-80ed-c650bb16345f: entering candidate state: node="Node at b016c8e3-276f-ad1c-80ed-c650bb16345f [Candidate]" term=2
2025-04-21T21:22:47.845Z [INFO]  server-b016c8e3-276f-ad1c-80ed-c650bb16345f: election won: term=2 tally=2
2025-04-21T21:22:47.845Z [INFO]  server-b016c8e3-276f-ad1c-80ed-c650bb16345f: entering leader state: leader="Node at b016c8e3-276f-ad1c-80ed-c650bb16345f [Leader]"
2025-04-21T21:22:47.845Z [INFO]  server-b016c8e3-276f-ad1c-80ed-c650bb16345f: added peer, starting replication: peer=server-af1eccbd-b541-e759-d552-5ec329322d2d
2025-04-21T21:22:47.845Z [INFO]  server-b016c8e3-276f-ad1c-80ed-c650bb16345f: added peer, starting replication: peer=server-782181c2-c4f5-9923-0058-005b60aa5e61
2025-04-21T21:22:47.845Z [INFO]  server-b016c8e3-276f-ad1c-80ed-c650bb16345f: pipelining replication: peer="{Voter server-782181c2-c4f5-9923-0058-005b60aa5e61 782181c2-c4f5-9923-0058-005b60aa5e61}"
2025-04-21T21:22:47.845Z [INFO]  server-b016c8e3-276f-ad1c-80ed-c650bb16345f: pipelining replication: peer="{Voter server-af1eccbd-b541-e759-d552-5ec329322d2d af1eccbd-b541-e759-d552-5ec329322d2d}"
2025-04-21T21:22:47.953Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:47.845413904 +0000 UTC m=+6.367073227 (1 nodes), 87.68207ms from start of poll, 87.868711ms from cluster start. Timeout at 2025-04-21 21:22:47.950416424 +0000 UTC m=+6.472075747, 105.00252ms after stability
2025-04-21T21:22:47.953Z [INFO]  server-b016c8e3-276f-ad1c-80ed-c650bb16345f: aborting pipeline replication: peer="{Voter server-782181c2-c4f5-9923-0058-005b60aa5e61 782181c2-c4f5-9923-0058-005b60aa5e61}"
2025-04-21T21:22:47.953Z [INFO]  server-b016c8e3-276f-ad1c-80ed-c650bb16345f: aborting pipeline replication: peer="{Voter server-af1eccbd-b541-e759-d552-5ec329322d2d af1eccbd-b541-e759-d552-5ec329322d2d}"
2025-04-21T21:22:47.953Z [INFO]  creating new snapshot: path=/tmp/raft3685071958/snapshots/2-6-1745270567953.tmp
2025-04-21T21:22:47.954Z [INFO]  server-af1eccbd-b541-e759-d552-5ec329322d2d: starting restore from snapshot: id=2-6-1745270567953 last-index=6 last-term=2 size-in-bytes=25
2025-04-21T21:22:47.954Z [INFO]  server-af1eccbd-b541-e759-d552-5ec329322d2d: snapshot restore progress: id=2-6-1745270567953 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-04-21T21:22:47.954Z [INFO]  server-af1eccbd-b541-e759-d552-5ec329322d2d: restored from snapshot: id=2-6-1745270567953 last-index=6 last-term=2 size-in-bytes=25
2025-04-21T21:22:47.954Z [INFO]  server-af1eccbd-b541-e759-d552-5ec329322d2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af1eccbd-b541-e759-d552-5ec329322d2d Address:af1eccbd-b541-e759-d552-5ec329322d2d} {Suffrage:Voter ID:server-b016c8e3-276f-ad1c-80ed-c650bb16345f Address:b016c8e3-276f-ad1c-80ed-c650bb16345f} {Suffrage:Voter ID:server-782181c2-c4f5-9923-0058-005b60aa5e61 Address:782181c2-c4f5-9923-0058-005b60aa5e61}]"
2025-04-21T21:22:47.954Z [INFO]  creating new snapshot: path=/tmp/raft1762132323/snapshots/2-6-1745270567954.tmp
2025-04-21T21:22:47.954Z [INFO]  server-b016c8e3-276f-ad1c-80ed-c650bb16345f: starting restore from snapshot: id=2-6-1745270567954 last-index=6 last-term=2 size-in-bytes=25
2025-04-21T21:22:47.954Z [INFO]  server-b016c8e3-276f-ad1c-80ed-c650bb16345f: snapshot restore progress: id=2-6-1745270567954 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-04-21T21:22:47.954Z [INFO]  server-b016c8e3-276f-ad1c-80ed-c650bb16345f: restored from snapshot: id=2-6-1745270567954 last-index=6 last-term=2 size-in-bytes=25
2025-04-21T21:22:47.954Z [INFO]  server-b016c8e3-276f-ad1c-80ed-c650bb16345f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af1eccbd-b541-e759-d552-5ec329322d2d Address:af1eccbd-b541-e759-d552-5ec329322d2d} {Suffrage:Voter ID:server-b016c8e3-276f-ad1c-80ed-c650bb16345f Address:b016c8e3-276f-ad1c-80ed-c650bb16345f} {Suffrage:Voter ID:server-782181c2-c4f5-9923-0058-005b60aa5e61 Address:782181c2-c4f5-9923-0058-005b60aa5e61}]"
2025-04-21T21:22:47.954Z [INFO]  creating new snapshot: path=/tmp/raft1758263027/snapshots/2-6-1745270567954.tmp
2025-04-21T21:22:47.954Z [INFO]  server-782181c2-c4f5-9923-0058-005b60aa5e61: starting restore from snapshot: id=2-6-1745270567954 last-index=6 last-term=2 size-in-bytes=25
2025-04-21T21:22:47.954Z [INFO]  server-782181c2-c4f5-9923-0058-005b60aa5e61: snapshot restore progress: id=2-6-1745270567954 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-04-21T21:22:47.954Z [INFO]  server-782181c2-c4f5-9923-0058-005b60aa5e61: restored from snapshot: id=2-6-1745270567954 last-index=6 last-term=2 size-in-bytes=25
2025-04-21T21:22:47.954Z [INFO]  server-782181c2-c4f5-9923-0058-005b60aa5e61: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-af1eccbd-b541-e759-d552-5ec329322d2d Address:af1eccbd-b541-e759-d552-5ec329322d2d} {Suffrage:Voter ID:server-b016c8e3-276f-ad1c-80ed-c650bb16345f Address:b016c8e3-276f-ad1c-80ed-c650bb16345f} {Suffrage:Voter ID:server-782181c2-c4f5-9923-0058-005b60aa5e61 Address:782181c2-c4f5-9923-0058-005b60aa5e61}]"
2025-04-21T21:22:47.954Z [INFO]  server-af1eccbd-b541-e759-d552-5ec329322d2d: entering follower state: follower="Node at af1eccbd-b541-e759-d552-5ec329322d2d [Follower]" leader-address= leader-id=
2025-04-21T21:22:47.954Z [INFO]  server-b016c8e3-276f-ad1c-80ed-c650bb16345f: entering follower state: follower="Node at b016c8e3-276f-ad1c-80ed-c650bb16345f [Follower]" leader-address= leader-id=
2025-04-21T21:22:47.954Z [INFO]  server-782181c2-c4f5-9923-0058-005b60aa5e61: entering follower state: follower="Node at 782181c2-c4f5-9923-0058-005b60aa5e61 [Follower]" leader-address= leader-id=
2025-04-21T21:22:48.009Z [WARN]  server-782181c2-c4f5-9923-0058-005b60aa5e61: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:48.009Z [INFO]  server-782181c2-c4f5-9923-0058-005b60aa5e61: entering candidate state: node="Node at 782181c2-c4f5-9923-0058-005b60aa5e61 [Candidate]" term=3
2025-04-21T21:22:48.009Z [INFO]  server-782181c2-c4f5-9923-0058-005b60aa5e61: election won: term=3 tally=2
2025-04-21T21:22:48.009Z [INFO]  server-782181c2-c4f5-9923-0058-005b60aa5e61: entering leader state: leader="Node at 782181c2-c4f5-9923-0058-005b60aa5e61 [Leader]"
2025-04-21T21:22:48.009Z [INFO]  server-782181c2-c4f5-9923-0058-005b60aa5e61: added peer, starting replication: peer=server-af1eccbd-b541-e759-d552-5ec329322d2d
2025-04-21T21:22:48.009Z [INFO]  server-782181c2-c4f5-9923-0058-005b60aa5e61: added peer, starting replication: peer=server-b016c8e3-276f-ad1c-80ed-c650bb16345f
2025-04-21T21:22:48.009Z [INFO]  server-782181c2-c4f5-9923-0058-005b60aa5e61: pipelining replication: peer="{Voter server-b016c8e3-276f-ad1c-80ed-c650bb16345f b016c8e3-276f-ad1c-80ed-c650bb16345f}"
2025-04-21T21:22:48.009Z [INFO]  server-782181c2-c4f5-9923-0058-005b60aa5e61: pipelining replication: peer="{Voter server-af1eccbd-b541-e759-d552-5ec329322d2d af1eccbd-b541-e759-d552-5ec329322d2d}"
2025-04-21T21:22:48.273Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:48.381Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:48.273197858 +0000 UTC m=+6.794857141 (1 nodes), 31.8µs from start of poll, 515.652625ms from cluster start. Timeout at 2025-04-21 21:22:48.378212258 +0000 UTC m=+6.899871541, 105.0144ms after stability
2025-04-21T21:22:48.381Z [INFO]  server-782181c2-c4f5-9923-0058-005b60aa5e61: aborting pipeline replication: peer="{Voter server-b016c8e3-276f-ad1c-80ed-c650bb16345f b016c8e3-276f-ad1c-80ed-c650bb16345f}"
2025-04-21T21:22:48.381Z [INFO]  server-782181c2-c4f5-9923-0058-005b60aa5e61: aborting pipeline replication: peer="{Voter server-af1eccbd-b541-e759-d552-5ec329322d2d af1eccbd-b541-e759-d552-5ec329322d2d}"
=== RUN   TestRaft_RecoverCluster/snapshot,_with_trailing_logs
2025-04-21T21:22:48.381Z [INFO]  server-d893bf19-1624-c28b-6f72-7c34ea58a624: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d893bf19-1624-c28b-6f72-7c34ea58a624 Address:d893bf19-1624-c28b-6f72-7c34ea58a624} {Suffrage:Voter ID:server-f9be0170-2231-8335-9b9d-a68345174471 Address:f9be0170-2231-8335-9b9d-a68345174471} {Suffrage:Voter ID:server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f Address:022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f}]"
2025-04-21T21:22:48.381Z [INFO]  server-f9be0170-2231-8335-9b9d-a68345174471: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d893bf19-1624-c28b-6f72-7c34ea58a624 Address:d893bf19-1624-c28b-6f72-7c34ea58a624} {Suffrage:Voter ID:server-f9be0170-2231-8335-9b9d-a68345174471 Address:f9be0170-2231-8335-9b9d-a68345174471} {Suffrage:Voter ID:server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f Address:022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f}]"
2025-04-21T21:22:48.381Z [INFO]  server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d893bf19-1624-c28b-6f72-7c34ea58a624 Address:d893bf19-1624-c28b-6f72-7c34ea58a624} {Suffrage:Voter ID:server-f9be0170-2231-8335-9b9d-a68345174471 Address:f9be0170-2231-8335-9b9d-a68345174471} {Suffrage:Voter ID:server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f Address:022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f}]"
2025-04-21T21:22:48.381Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:48.381Z [INFO]  server-d893bf19-1624-c28b-6f72-7c34ea58a624: entering follower state: follower="Node at d893bf19-1624-c28b-6f72-7c34ea58a624 [Follower]" leader-address= leader-id=
2025-04-21T21:22:48.381Z [INFO]  server-f9be0170-2231-8335-9b9d-a68345174471: entering follower state: follower="Node at f9be0170-2231-8335-9b9d-a68345174471 [Follower]" leader-address= leader-id=
2025-04-21T21:22:48.381Z [INFO]  server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f: entering follower state: follower="Node at 022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f [Follower]" leader-address= leader-id=
2025-04-21T21:22:48.445Z [WARN]  server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:48.445Z [INFO]  server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f: entering candidate state: node="Node at 022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f [Candidate]" term=2
2025-04-21T21:22:48.445Z [INFO]  server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f: election won: term=2 tally=2
2025-04-21T21:22:48.445Z [INFO]  server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f: entering leader state: leader="Node at 022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f [Leader]"
2025-04-21T21:22:48.445Z [INFO]  server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f: added peer, starting replication: peer=server-d893bf19-1624-c28b-6f72-7c34ea58a624
2025-04-21T21:22:48.445Z [INFO]  server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f: added peer, starting replication: peer=server-f9be0170-2231-8335-9b9d-a68345174471
2025-04-21T21:22:48.445Z [INFO]  server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f: pipelining replication: peer="{Voter server-f9be0170-2231-8335-9b9d-a68345174471 f9be0170-2231-8335-9b9d-a68345174471}"
2025-04-21T21:22:48.445Z [INFO]  server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f: pipelining replication: peer="{Voter server-d893bf19-1624-c28b-6f72-7c34ea58a624 d893bf19-1624-c28b-6f72-7c34ea58a624}"
2025-04-21T21:22:48.553Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:48.445423016 +0000 UTC m=+6.967082339 (1 nodes), 63.587228ms from start of poll, 63.770869ms from cluster start. Timeout at 2025-04-21 21:22:48.550426936 +0000 UTC m=+7.072086299, 105.00396ms after stability
2025-04-21T21:22:48.554Z [INFO]  server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f: aborting pipeline replication: peer="{Voter server-f9be0170-2231-8335-9b9d-a68345174471 f9be0170-2231-8335-9b9d-a68345174471}"
2025-04-21T21:22:48.554Z [INFO]  server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f: aborting pipeline replication: peer="{Voter server-d893bf19-1624-c28b-6f72-7c34ea58a624 d893bf19-1624-c28b-6f72-7c34ea58a624}"
2025-04-21T21:22:48.554Z [INFO]  creating new snapshot: path=/tmp/raft50519934/snapshots/2-27-1745270568554.tmp
2025-04-21T21:22:48.555Z [INFO]  server-d893bf19-1624-c28b-6f72-7c34ea58a624: starting restore from snapshot: id=2-27-1745270568554 last-index=27 last-term=2 size-in-bytes=168
2025-04-21T21:22:48.555Z [INFO]  server-d893bf19-1624-c28b-6f72-7c34ea58a624: snapshot restore progress: id=2-27-1745270568554 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-04-21T21:22:48.555Z [INFO]  server-d893bf19-1624-c28b-6f72-7c34ea58a624: restored from snapshot: id=2-27-1745270568554 last-index=27 last-term=2 size-in-bytes=168
2025-04-21T21:22:48.555Z [INFO]  server-d893bf19-1624-c28b-6f72-7c34ea58a624: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d893bf19-1624-c28b-6f72-7c34ea58a624 Address:d893bf19-1624-c28b-6f72-7c34ea58a624} {Suffrage:Voter ID:server-f9be0170-2231-8335-9b9d-a68345174471 Address:f9be0170-2231-8335-9b9d-a68345174471} {Suffrage:Voter ID:server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f Address:022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f}]"
2025-04-21T21:22:48.555Z [INFO]  creating new snapshot: path=/tmp/raft4122844227/snapshots/2-27-1745270568555.tmp
2025-04-21T21:22:48.556Z [INFO]  server-f9be0170-2231-8335-9b9d-a68345174471: starting restore from snapshot: id=2-27-1745270568555 last-index=27 last-term=2 size-in-bytes=168
2025-04-21T21:22:48.556Z [INFO]  server-d893bf19-1624-c28b-6f72-7c34ea58a624: entering follower state: follower="Node at d893bf19-1624-c28b-6f72-7c34ea58a624 [Follower]" leader-address= leader-id=
2025-04-21T21:22:48.556Z [INFO]  server-f9be0170-2231-8335-9b9d-a68345174471: snapshot restore progress: id=2-27-1745270568555 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-04-21T21:22:48.556Z [INFO]  server-f9be0170-2231-8335-9b9d-a68345174471: restored from snapshot: id=2-27-1745270568555 last-index=27 last-term=2 size-in-bytes=168
2025-04-21T21:22:48.556Z [INFO]  server-f9be0170-2231-8335-9b9d-a68345174471: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d893bf19-1624-c28b-6f72-7c34ea58a624 Address:d893bf19-1624-c28b-6f72-7c34ea58a624} {Suffrage:Voter ID:server-f9be0170-2231-8335-9b9d-a68345174471 Address:f9be0170-2231-8335-9b9d-a68345174471} {Suffrage:Voter ID:server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f Address:022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f}]"
2025-04-21T21:22:48.556Z [INFO]  creating new snapshot: path=/tmp/raft1714867606/snapshots/2-27-1745270568556.tmp
2025-04-21T21:22:48.557Z [INFO]  server-f9be0170-2231-8335-9b9d-a68345174471: entering follower state: follower="Node at f9be0170-2231-8335-9b9d-a68345174471 [Follower]" leader-address= leader-id=
2025-04-21T21:22:48.569Z [INFO]  server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f: starting restore from snapshot: id=2-27-1745270568556 last-index=27 last-term=2 size-in-bytes=168
2025-04-21T21:22:48.569Z [INFO]  server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f: snapshot restore progress: id=2-27-1745270568556 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-04-21T21:22:48.569Z [INFO]  server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f: restored from snapshot: id=2-27-1745270568556 last-index=27 last-term=2 size-in-bytes=168
2025-04-21T21:22:48.569Z [INFO]  server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d893bf19-1624-c28b-6f72-7c34ea58a624 Address:d893bf19-1624-c28b-6f72-7c34ea58a624} {Suffrage:Voter ID:server-f9be0170-2231-8335-9b9d-a68345174471 Address:f9be0170-2231-8335-9b9d-a68345174471} {Suffrage:Voter ID:server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f Address:022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f}]"
2025-04-21T21:22:48.569Z [INFO]  server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f: entering follower state: follower="Node at 022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f [Follower]" leader-address= leader-id=
2025-04-21T21:22:48.617Z [WARN]  server-f9be0170-2231-8335-9b9d-a68345174471: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:48.617Z [INFO]  server-f9be0170-2231-8335-9b9d-a68345174471: entering candidate state: node="Node at f9be0170-2231-8335-9b9d-a68345174471 [Candidate]" term=3
2025-04-21T21:22:48.617Z [INFO]  server-f9be0170-2231-8335-9b9d-a68345174471: election won: term=3 tally=2
2025-04-21T21:22:48.617Z [INFO]  server-f9be0170-2231-8335-9b9d-a68345174471: entering leader state: leader="Node at f9be0170-2231-8335-9b9d-a68345174471 [Leader]"
2025-04-21T21:22:48.617Z [INFO]  server-f9be0170-2231-8335-9b9d-a68345174471: added peer, starting replication: peer=server-d893bf19-1624-c28b-6f72-7c34ea58a624
2025-04-21T21:22:48.617Z [INFO]  server-f9be0170-2231-8335-9b9d-a68345174471: added peer, starting replication: peer=server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f
2025-04-21T21:22:48.617Z [INFO]  server-f9be0170-2231-8335-9b9d-a68345174471: pipelining replication: peer="{Voter server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f 022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f}"
2025-04-21T21:22:48.617Z [INFO]  server-f9be0170-2231-8335-9b9d-a68345174471: pipelining replication: peer="{Voter server-d893bf19-1624-c28b-6f72-7c34ea58a624 d893bf19-1624-c28b-6f72-7c34ea58a624}"
2025-04-21T21:22:48.885Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:48.993Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:48.885249091 +0000 UTC m=+7.406908374 (1 nodes), 20.6µs from start of poll, 503.596904ms from cluster start. Timeout at 2025-04-21 21:22:48.990269251 +0000 UTC m=+7.511928574, 105.0202ms after stability
2025-04-21T21:22:48.993Z [INFO]  server-f9be0170-2231-8335-9b9d-a68345174471: aborting pipeline replication: peer="{Voter server-022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f 022f0077-a1aa-6f53-4b1d-a1c2e70c1c8f}"
2025-04-21T21:22:48.993Z [INFO]  server-f9be0170-2231-8335-9b9d-a68345174471: aborting pipeline replication: peer="{Voter server-d893bf19-1624-c28b-6f72-7c34ea58a624 d893bf19-1624-c28b-6f72-7c34ea58a624}"
--- PASS: TestRaft_RecoverCluster (2.34s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (1.11s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.62s)
    --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.61s)
=== RUN   TestRaft_HasExistingState
    raft_test.go:258: Skipping unreliable test
--- SKIP: TestRaft_HasExistingState (0.00s)
=== RUN   TestRaft_SingleNode
2025-04-21T21:22:48.994Z [INFO]  server-8f85fa7c-4b52-96fc-bf98-041198fc78d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f85fa7c-4b52-96fc-bf98-041198fc78d0 Address:8f85fa7c-4b52-96fc-bf98-041198fc78d0}]"
2025-04-21T21:22:48.994Z [INFO]  server-8f85fa7c-4b52-96fc-bf98-041198fc78d0: entering follower state: follower="Node at 8f85fa7c-4b52-96fc-bf98-041198fc78d0 [Follower]" leader-address= leader-id=
2025-04-21T21:22:49.057Z [WARN]  server-8f85fa7c-4b52-96fc-bf98-041198fc78d0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:49.057Z [INFO]  server-8f85fa7c-4b52-96fc-bf98-041198fc78d0: entering candidate state: node="Node at 8f85fa7c-4b52-96fc-bf98-041198fc78d0 [Candidate]" term=2
2025-04-21T21:22:49.057Z [INFO]  server-8f85fa7c-4b52-96fc-bf98-041198fc78d0: election won: term=2 tally=1
2025-04-21T21:22:49.057Z [INFO]  server-8f85fa7c-4b52-96fc-bf98-041198fc78d0: entering leader state: leader="Node at 8f85fa7c-4b52-96fc-bf98-041198fc78d0 [Leader]"
--- PASS: TestRaft_SingleNode (0.06s)
=== RUN   TestRaft_TripleNode
2025-04-21T21:22:49.057Z [INFO]  server-22b1379d-2abf-ad6b-64d1-95357d578517: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22b1379d-2abf-ad6b-64d1-95357d578517 Address:22b1379d-2abf-ad6b-64d1-95357d578517} {Suffrage:Voter ID:server-2c70823a-4e42-0ee7-1784-bd81e9a8d1b2 Address:2c70823a-4e42-0ee7-1784-bd81e9a8d1b2} {Suffrage:Voter ID:server-34c51e97-7fed-9e9a-aa8f-6e397828e23b Address:34c51e97-7fed-9e9a-aa8f-6e397828e23b}]"
2025-04-21T21:22:49.057Z [INFO]  server-2c70823a-4e42-0ee7-1784-bd81e9a8d1b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22b1379d-2abf-ad6b-64d1-95357d578517 Address:22b1379d-2abf-ad6b-64d1-95357d578517} {Suffrage:Voter ID:server-2c70823a-4e42-0ee7-1784-bd81e9a8d1b2 Address:2c70823a-4e42-0ee7-1784-bd81e9a8d1b2} {Suffrage:Voter ID:server-34c51e97-7fed-9e9a-aa8f-6e397828e23b Address:34c51e97-7fed-9e9a-aa8f-6e397828e23b}]"
2025-04-21T21:22:49.057Z [INFO]  server-34c51e97-7fed-9e9a-aa8f-6e397828e23b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-22b1379d-2abf-ad6b-64d1-95357d578517 Address:22b1379d-2abf-ad6b-64d1-95357d578517} {Suffrage:Voter ID:server-2c70823a-4e42-0ee7-1784-bd81e9a8d1b2 Address:2c70823a-4e42-0ee7-1784-bd81e9a8d1b2} {Suffrage:Voter ID:server-34c51e97-7fed-9e9a-aa8f-6e397828e23b Address:34c51e97-7fed-9e9a-aa8f-6e397828e23b}]"
2025-04-21T21:22:49.057Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:22:49.057Z [INFO]  server-22b1379d-2abf-ad6b-64d1-95357d578517: entering follower state: follower="Node at 22b1379d-2abf-ad6b-64d1-95357d578517 [Follower]" leader-address= leader-id=
2025-04-21T21:22:49.057Z [INFO]  server-2c70823a-4e42-0ee7-1784-bd81e9a8d1b2: entering follower state: follower="Node at 2c70823a-4e42-0ee7-1784-bd81e9a8d1b2 [Follower]" leader-address= leader-id=
2025-04-21T21:22:49.057Z [INFO]  server-34c51e97-7fed-9e9a-aa8f-6e397828e23b: entering follower state: follower="Node at 34c51e97-7fed-9e9a-aa8f-6e397828e23b [Follower]" leader-address= leader-id=
2025-04-21T21:22:49.113Z [WARN]  server-22b1379d-2abf-ad6b-64d1-95357d578517: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:49.113Z [INFO]  server-22b1379d-2abf-ad6b-64d1-95357d578517: entering candidate state: node="Node at 22b1379d-2abf-ad6b-64d1-95357d578517 [Candidate]" term=2
2025-04-21T21:22:49.113Z [INFO]  server-22b1379d-2abf-ad6b-64d1-95357d578517: election won: term=2 tally=2
2025-04-21T21:22:49.113Z [INFO]  server-22b1379d-2abf-ad6b-64d1-95357d578517: entering leader state: leader="Node at 22b1379d-2abf-ad6b-64d1-95357d578517 [Leader]"
2025-04-21T21:22:49.113Z [INFO]  server-22b1379d-2abf-ad6b-64d1-95357d578517: added peer, starting replication: peer=server-2c70823a-4e42-0ee7-1784-bd81e9a8d1b2
2025-04-21T21:22:49.113Z [INFO]  server-22b1379d-2abf-ad6b-64d1-95357d578517: added peer, starting replication: peer=server-34c51e97-7fed-9e9a-aa8f-6e397828e23b
2025-04-21T21:22:49.113Z [INFO]  server-22b1379d-2abf-ad6b-64d1-95357d578517: pipelining replication: peer="{Voter server-34c51e97-7fed-9e9a-aa8f-6e397828e23b 34c51e97-7fed-9e9a-aa8f-6e397828e23b}"
2025-04-21T21:22:49.113Z [INFO]  server-22b1379d-2abf-ad6b-64d1-95357d578517: pipelining replication: peer="{Voter server-2c70823a-4e42-0ee7-1784-bd81e9a8d1b2 2c70823a-4e42-0ee7-1784-bd81e9a8d1b2}"
2025-04-21T21:22:49.221Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:22:49.113437465 +0000 UTC m=+7.635096788 (2 nodes), 55.733375ms from start of poll, 55.905376ms from cluster start. Timeout at 2025-04-21 21:22:49.218440305 +0000 UTC m=+7.740099588, 105.0028ms after stability
2025-04-21T21:22:49.221Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:49.326Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:49.221282989 +0000 UTC m=+7.742942272 (1 nodes), 640ns from start of poll, 163.75086ms from cluster start. Timeout at 2025-04-21 21:22:49.326290509 +0000 UTC m=+7.847949832, 105.00756ms after stability
2025-04-21T21:22:49.339Z [INFO]  server-22b1379d-2abf-ad6b-64d1-95357d578517: aborting pipeline replication: peer="{Voter server-34c51e97-7fed-9e9a-aa8f-6e397828e23b 34c51e97-7fed-9e9a-aa8f-6e397828e23b}"
2025-04-21T21:22:49.339Z [INFO]  server-22b1379d-2abf-ad6b-64d1-95357d578517: aborting pipeline replication: peer="{Voter server-2c70823a-4e42-0ee7-1784-bd81e9a8d1b2 2c70823a-4e42-0ee7-1784-bd81e9a8d1b2}"
--- PASS: TestRaft_TripleNode (0.28s)
=== RUN   TestRaft_LeaderFail
2025-04-21T21:22:49.339Z [INFO]  server-636225f9-9964-7f2d-cff6-f868de129993: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-636225f9-9964-7f2d-cff6-f868de129993 Address:636225f9-9964-7f2d-cff6-f868de129993} {Suffrage:Voter ID:server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a Address:19e88350-a755-7eff-0eb1-ba6d7d7b6d8a} {Suffrage:Voter ID:server-2b1bc52d-d62b-2345-e351-d2492084df9a Address:2b1bc52d-d62b-2345-e351-d2492084df9a}]"
2025-04-21T21:22:49.340Z [INFO]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-636225f9-9964-7f2d-cff6-f868de129993 Address:636225f9-9964-7f2d-cff6-f868de129993} {Suffrage:Voter ID:server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a Address:19e88350-a755-7eff-0eb1-ba6d7d7b6d8a} {Suffrage:Voter ID:server-2b1bc52d-d62b-2345-e351-d2492084df9a Address:2b1bc52d-d62b-2345-e351-d2492084df9a}]"
2025-04-21T21:22:49.340Z [INFO]  server-2b1bc52d-d62b-2345-e351-d2492084df9a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-636225f9-9964-7f2d-cff6-f868de129993 Address:636225f9-9964-7f2d-cff6-f868de129993} {Suffrage:Voter ID:server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a Address:19e88350-a755-7eff-0eb1-ba6d7d7b6d8a} {Suffrage:Voter ID:server-2b1bc52d-d62b-2345-e351-d2492084df9a Address:2b1bc52d-d62b-2345-e351-d2492084df9a}]"
2025-04-21T21:22:49.340Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:22:49.340Z [INFO]  server-636225f9-9964-7f2d-cff6-f868de129993: entering follower state: follower="Node at 636225f9-9964-7f2d-cff6-f868de129993 [Follower]" leader-address= leader-id=
2025-04-21T21:22:49.340Z [INFO]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: entering follower state: follower="Node at 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a [Follower]" leader-address= leader-id=
2025-04-21T21:22:49.340Z [INFO]  server-2b1bc52d-d62b-2345-e351-d2492084df9a: entering follower state: follower="Node at 2b1bc52d-d62b-2345-e351-d2492084df9a [Follower]" leader-address= leader-id=
2025-04-21T21:22:49.413Z [WARN]  server-636225f9-9964-7f2d-cff6-f868de129993: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:49.413Z [INFO]  server-636225f9-9964-7f2d-cff6-f868de129993: entering candidate state: node="Node at 636225f9-9964-7f2d-cff6-f868de129993 [Candidate]" term=2
2025-04-21T21:22:49.413Z [WARN]  server-2b1bc52d-d62b-2345-e351-d2492084df9a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:49.413Z [INFO]  server-2b1bc52d-d62b-2345-e351-d2492084df9a: entering candidate state: node="Node at 2b1bc52d-d62b-2345-e351-d2492084df9a [Candidate]" term=2
2025-04-21T21:22:49.413Z [INFO]  server-2b1bc52d-d62b-2345-e351-d2492084df9a: duplicate requestVote for same term: term=2
2025-04-21T21:22:49.413Z [INFO]  server-636225f9-9964-7f2d-cff6-f868de129993: election won: term=2 tally=2
2025-04-21T21:22:49.413Z [INFO]  server-636225f9-9964-7f2d-cff6-f868de129993: entering leader state: leader="Node at 636225f9-9964-7f2d-cff6-f868de129993 [Leader]"
2025-04-21T21:22:49.413Z [INFO]  server-636225f9-9964-7f2d-cff6-f868de129993: added peer, starting replication: peer=server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a
2025-04-21T21:22:49.413Z [INFO]  server-636225f9-9964-7f2d-cff6-f868de129993: added peer, starting replication: peer=server-2b1bc52d-d62b-2345-e351-d2492084df9a
2025-04-21T21:22:49.413Z [INFO]  server-2b1bc52d-d62b-2345-e351-d2492084df9a: entering follower state: follower="Node at 2b1bc52d-d62b-2345-e351-d2492084df9a [Follower]" leader-address=636225f9-9964-7f2d-cff6-f868de129993 leader-id=server-636225f9-9964-7f2d-cff6-f868de129993
2025-04-21T21:22:49.413Z [INFO]  server-636225f9-9964-7f2d-cff6-f868de129993: pipelining replication: peer="{Voter server-2b1bc52d-d62b-2345-e351-d2492084df9a 2b1bc52d-d62b-2345-e351-d2492084df9a}"
2025-04-21T21:22:49.413Z [INFO]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: duplicate requestVote for same term: term=2
2025-04-21T21:22:49.413Z [INFO]  server-636225f9-9964-7f2d-cff6-f868de129993: pipelining replication: peer="{Voter server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a}"
2025-04-21T21:22:49.413Z [WARN]  server-636225f9-9964-7f2d-cff6-f868de129993: rejecting vote request since we have a leader: from=2b1bc52d-d62b-2345-e351-d2492084df9a leader=636225f9-9964-7f2d-cff6-f868de129993 leader-id=server-636225f9-9964-7f2d-cff6-f868de129993
2025-04-21T21:22:49.521Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:22:49.413526241 +0000 UTC m=+7.935185564 (2 nodes), 73.458653ms from start of poll, 73.649535ms from cluster start. Timeout at 2025-04-21 21:22:49.518530361 +0000 UTC m=+8.040189684, 105.00412ms after stability
2025-04-21T21:22:49.521Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:49.629Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:49.521150844 +0000 UTC m=+8.042810127 (1 nodes), 2.24µs from start of poll, 181.274098ms from cluster start. Timeout at 2025-04-21 21:22:49.626162884 +0000 UTC m=+8.147822207, 105.01208ms after stability
    raft_test.go:378: [INFO] Disconnecting Node at 636225f9-9964-7f2d-cff6-f868de129993 [Leader]
2025-04-21T21:22:49.661Z [ERROR] server-636225f9-9964-7f2d-cff6-f868de129993: failed to heartbeat to: peer=19e88350-a755-7eff-0eb1-ba6d7d7b6d8a backoff time=10ms error="failed to connect to peer: 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a"
2025-04-21T21:22:49.661Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:49.665Z [ERROR] server-636225f9-9964-7f2d-cff6-f868de129993: failed to pipeline appendEntries: peer="{Voter server-2b1bc52d-d62b-2345-e351-d2492084df9a 2b1bc52d-d62b-2345-e351-d2492084df9a}" error="append pipeline closed"
2025-04-21T21:22:49.665Z [INFO]  server-636225f9-9964-7f2d-cff6-f868de129993: aborting pipeline replication: peer="{Voter server-2b1bc52d-d62b-2345-e351-d2492084df9a 2b1bc52d-d62b-2345-e351-d2492084df9a}"
2025-04-21T21:22:49.673Z [ERROR] server-636225f9-9964-7f2d-cff6-f868de129993: failed to pipeline appendEntries: peer="{Voter server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a}" error="append pipeline closed"
2025-04-21T21:22:49.673Z [INFO]  server-636225f9-9964-7f2d-cff6-f868de129993: aborting pipeline replication: peer="{Voter server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a}"
2025-04-21T21:22:49.673Z [ERROR] server-636225f9-9964-7f2d-cff6-f868de129993: failed to heartbeat to: peer=2b1bc52d-d62b-2345-e351-d2492084df9a backoff time=10ms error="failed to connect to peer: 2b1bc52d-d62b-2345-e351-d2492084df9a"
2025-04-21T21:22:49.677Z [ERROR] server-636225f9-9964-7f2d-cff6-f868de129993: failed to appendEntries to: peer="{Voter server-2b1bc52d-d62b-2345-e351-d2492084df9a 2b1bc52d-d62b-2345-e351-d2492084df9a}" error="failed to connect to peer: 2b1bc52d-d62b-2345-e351-d2492084df9a"
2025-04-21T21:22:49.681Z [ERROR] server-636225f9-9964-7f2d-cff6-f868de129993: failed to heartbeat to: peer=19e88350-a755-7eff-0eb1-ba6d7d7b6d8a backoff time=10ms error="failed to connect to peer: 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a"
2025-04-21T21:22:49.681Z [ERROR] server-636225f9-9964-7f2d-cff6-f868de129993: failed to appendEntries to: peer="{Voter server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a}" error="failed to connect to peer: 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a"
2025-04-21T21:22:49.697Z [ERROR] server-636225f9-9964-7f2d-cff6-f868de129993: failed to appendEntries to: peer="{Voter server-2b1bc52d-d62b-2345-e351-d2492084df9a 2b1bc52d-d62b-2345-e351-d2492084df9a}" error="failed to connect to peer: 2b1bc52d-d62b-2345-e351-d2492084df9a"
2025-04-21T21:22:49.697Z [ERROR] server-636225f9-9964-7f2d-cff6-f868de129993: failed to heartbeat to: peer=2b1bc52d-d62b-2345-e351-d2492084df9a backoff time=10ms error="failed to connect to peer: 2b1bc52d-d62b-2345-e351-d2492084df9a"
2025-04-21T21:22:49.701Z [ERROR] server-636225f9-9964-7f2d-cff6-f868de129993: failed to heartbeat to: peer=19e88350-a755-7eff-0eb1-ba6d7d7b6d8a backoff time=10ms error="failed to connect to peer: 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a"
2025-04-21T21:22:49.704Z [ERROR] server-636225f9-9964-7f2d-cff6-f868de129993: failed to appendEntries to: peer="{Voter server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a}" error="failed to connect to peer: 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a"
2025-04-21T21:22:49.704Z [WARN]  server-636225f9-9964-7f2d-cff6-f868de129993: failed to contact: server-id=server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a time=51.754547ms
2025-04-21T21:22:49.717Z [WARN]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: heartbeat timeout reached, starting election: last-leader-addr=636225f9-9964-7f2d-cff6-f868de129993 last-leader-id=server-636225f9-9964-7f2d-cff6-f868de129993
2025-04-21T21:22:49.717Z [INFO]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: entering candidate state: node="Node at 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a [Candidate]" term=3
2025-04-21T21:22:49.717Z [WARN]  server-2b1bc52d-d62b-2345-e351-d2492084df9a: rejecting vote request since we have a leader: from=19e88350-a755-7eff-0eb1-ba6d7d7b6d8a leader=636225f9-9964-7f2d-cff6-f868de129993 leader-id=server-636225f9-9964-7f2d-cff6-f868de129993
2025-04-21T21:22:49.717Z [WARN]  server-636225f9-9964-7f2d-cff6-f868de129993: failed to contact: server-id=server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a time=64.141111ms
2025-04-21T21:22:49.717Z [WARN]  server-636225f9-9964-7f2d-cff6-f868de129993: failed to contact: server-id=server-2b1bc52d-d62b-2345-e351-d2492084df9a time=56.199018ms
2025-04-21T21:22:49.717Z [WARN]  server-636225f9-9964-7f2d-cff6-f868de129993: failed to contact quorum of nodes, stepping down
2025-04-21T21:22:49.717Z [INFO]  server-636225f9-9964-7f2d-cff6-f868de129993: entering follower state: follower="Node at 636225f9-9964-7f2d-cff6-f868de129993 [Follower]" leader-address= leader-id=
2025-04-21T21:22:49.717Z [ERROR] server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: failed to make requestVote RPC: target="{Voter server-636225f9-9964-7f2d-cff6-f868de129993 636225f9-9964-7f2d-cff6-f868de129993}" error="failed to connect to peer: 636225f9-9964-7f2d-cff6-f868de129993" term=3
2025-04-21T21:22:49.717Z [ERROR] server-636225f9-9964-7f2d-cff6-f868de129993: failed to heartbeat to: peer=2b1bc52d-d62b-2345-e351-d2492084df9a backoff time=10ms error="failed to connect to peer: 2b1bc52d-d62b-2345-e351-d2492084df9a"
2025-04-21T21:22:49.721Z [ERROR] server-636225f9-9964-7f2d-cff6-f868de129993: failed to appendEntries to: peer="{Voter server-2b1bc52d-d62b-2345-e351-d2492084df9a 2b1bc52d-d62b-2345-e351-d2492084df9a}" error="failed to connect to peer: 2b1bc52d-d62b-2345-e351-d2492084df9a"
2025-04-21T21:22:49.769Z [WARN]  server-2b1bc52d-d62b-2345-e351-d2492084df9a: heartbeat timeout reached, starting election: last-leader-addr=636225f9-9964-7f2d-cff6-f868de129993 last-leader-id=server-636225f9-9964-7f2d-cff6-f868de129993
2025-04-21T21:22:49.769Z [INFO]  server-2b1bc52d-d62b-2345-e351-d2492084df9a: entering candidate state: node="Node at 2b1bc52d-d62b-2345-e351-d2492084df9a [Candidate]" term=3
2025-04-21T21:22:49.769Z [INFO]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: duplicate requestVote for same term: term=3
2025-04-21T21:22:49.769Z [ERROR] server-2b1bc52d-d62b-2345-e351-d2492084df9a: failed to make requestVote RPC: target="{Voter server-636225f9-9964-7f2d-cff6-f868de129993 636225f9-9964-7f2d-cff6-f868de129993}" error="failed to connect to peer: 636225f9-9964-7f2d-cff6-f868de129993" term=3
2025-04-21T21:22:49.773Z [WARN]  server-636225f9-9964-7f2d-cff6-f868de129993: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:49.773Z [INFO]  server-636225f9-9964-7f2d-cff6-f868de129993: entering candidate state: node="Node at 636225f9-9964-7f2d-cff6-f868de129993 [Candidate]" term=3
2025-04-21T21:22:49.773Z [ERROR] server-636225f9-9964-7f2d-cff6-f868de129993: failed to make requestVote RPC: target="{Voter server-2b1bc52d-d62b-2345-e351-d2492084df9a 2b1bc52d-d62b-2345-e351-d2492084df9a}" error="failed to connect to peer: 2b1bc52d-d62b-2345-e351-d2492084df9a" term=3
2025-04-21T21:22:49.773Z [ERROR] server-636225f9-9964-7f2d-cff6-f868de129993: failed to make requestVote RPC: target="{Voter server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a}" error="failed to connect to peer: 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a" term=3
2025-04-21T21:22:49.797Z [WARN]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: Election timeout reached, restarting election
2025-04-21T21:22:49.797Z [INFO]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: entering candidate state: node="Node at 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a [Candidate]" term=4
2025-04-21T21:22:49.797Z [INFO]  server-2b1bc52d-d62b-2345-e351-d2492084df9a: entering follower state: follower="Node at 2b1bc52d-d62b-2345-e351-d2492084df9a [Follower]" leader-address= leader-id=
2025-04-21T21:22:49.797Z [INFO]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: election won: term=4 tally=2
2025-04-21T21:22:49.797Z [INFO]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: entering leader state: leader="Node at 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a [Leader]"
2025-04-21T21:22:49.797Z [INFO]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: added peer, starting replication: peer=server-636225f9-9964-7f2d-cff6-f868de129993
2025-04-21T21:22:49.797Z [INFO]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: added peer, starting replication: peer=server-2b1bc52d-d62b-2345-e351-d2492084df9a
2025-04-21T21:22:49.797Z [INFO]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: pipelining replication: peer="{Voter server-2b1bc52d-d62b-2345-e351-d2492084df9a 2b1bc52d-d62b-2345-e351-d2492084df9a}"
2025-04-21T21:22:49.797Z [ERROR] server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: failed to make requestVote RPC: target="{Voter server-636225f9-9964-7f2d-cff6-f868de129993 636225f9-9964-7f2d-cff6-f868de129993}" error="failed to connect to peer: 636225f9-9964-7f2d-cff6-f868de129993" term=4
2025-04-21T21:22:49.797Z [ERROR] server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: failed to appendEntries to: peer="{Voter server-636225f9-9964-7f2d-cff6-f868de129993 636225f9-9964-7f2d-cff6-f868de129993}" error="failed to connect to peer: 636225f9-9964-7f2d-cff6-f868de129993"
2025-04-21T21:22:49.805Z [ERROR] server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: failed to heartbeat to: peer=636225f9-9964-7f2d-cff6-f868de129993 backoff time=10ms error="failed to connect to peer: 636225f9-9964-7f2d-cff6-f868de129993"
2025-04-21T21:22:49.833Z [ERROR] server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: failed to appendEntries to: peer="{Voter server-636225f9-9964-7f2d-cff6-f868de129993 636225f9-9964-7f2d-cff6-f868de129993}" error="failed to connect to peer: 636225f9-9964-7f2d-cff6-f868de129993"
2025-04-21T21:22:49.841Z [ERROR] server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: failed to heartbeat to: peer=636225f9-9964-7f2d-cff6-f868de129993 backoff time=10ms error="failed to connect to peer: 636225f9-9964-7f2d-cff6-f868de129993"
2025-04-21T21:22:49.849Z [WARN]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: failed to contact: server-id=server-636225f9-9964-7f2d-cff6-f868de129993 time=51.733227ms
2025-04-21T21:22:49.853Z [ERROR] server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: failed to appendEntries to: peer="{Voter server-636225f9-9964-7f2d-cff6-f868de129993 636225f9-9964-7f2d-cff6-f868de129993}" error="failed to connect to peer: 636225f9-9964-7f2d-cff6-f868de129993"
2025-04-21T21:22:49.857Z [WARN]  server-636225f9-9964-7f2d-cff6-f868de129993: Election timeout reached, restarting election
2025-04-21T21:22:49.857Z [INFO]  server-636225f9-9964-7f2d-cff6-f868de129993: entering candidate state: node="Node at 636225f9-9964-7f2d-cff6-f868de129993 [Candidate]" term=4
2025-04-21T21:22:49.857Z [ERROR] server-636225f9-9964-7f2d-cff6-f868de129993: failed to make requestVote RPC: target="{Voter server-2b1bc52d-d62b-2345-e351-d2492084df9a 2b1bc52d-d62b-2345-e351-d2492084df9a}" error="failed to connect to peer: 2b1bc52d-d62b-2345-e351-d2492084df9a" term=4
2025-04-21T21:22:49.857Z [ERROR] server-636225f9-9964-7f2d-cff6-f868de129993: failed to make requestVote RPC: target="{Voter server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a}" error="failed to connect to peer: 19e88350-a755-7eff-0eb1-ba6d7d7b6d8a" term=4
2025-04-21T21:22:49.861Z [ERROR] server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: failed to heartbeat to: peer=636225f9-9964-7f2d-cff6-f868de129993 backoff time=10ms error="failed to connect to peer: 636225f9-9964-7f2d-cff6-f868de129993"
2025-04-21T21:22:49.885Z [ERROR] server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: failed to appendEntries to: peer="{Voter server-636225f9-9964-7f2d-cff6-f868de129993 636225f9-9964-7f2d-cff6-f868de129993}" error="failed to connect to peer: 636225f9-9964-7f2d-cff6-f868de129993"
2025-04-21T21:22:49.885Z [ERROR] server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: failed to heartbeat to: peer=636225f9-9964-7f2d-cff6-f868de129993 backoff time=20ms error="failed to connect to peer: 636225f9-9964-7f2d-cff6-f868de129993"
2025-04-21T21:22:49.901Z [WARN]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: failed to contact: server-id=server-636225f9-9964-7f2d-cff6-f868de129993 time=103.686296ms
2025-04-21T21:22:49.905Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:49.797512542 +0000 UTC m=+8.319171825 (1 nodes), 136.224556ms from start of poll, 457.635796ms from cluster start. Timeout at 2025-04-21 21:22:49.902522063 +0000 UTC m=+8.424181386, 105.009561ms after stability
    raft_test.go:413: [INFO] Reconnecting Node at 636225f9-9964-7f2d-cff6-f868de129993 [Candidate]
2025-04-21T21:22:49.913Z [INFO]  server-636225f9-9964-7f2d-cff6-f868de129993: entering follower state: follower="Node at 636225f9-9964-7f2d-cff6-f868de129993 [Follower]" leader-address=19e88350-a755-7eff-0eb1-ba6d7d7b6d8a leader-id=server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a
2025-04-21T21:22:49.937Z [INFO]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: pipelining replication: peer="{Voter server-636225f9-9964-7f2d-cff6-f868de129993 636225f9-9964-7f2d-cff6-f868de129993}"
2025-04-21T21:22:49.945Z [INFO]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: aborting pipeline replication: peer="{Voter server-2b1bc52d-d62b-2345-e351-d2492084df9a 2b1bc52d-d62b-2345-e351-d2492084df9a}"
2025-04-21T21:22:49.945Z [INFO]  server-19e88350-a755-7eff-0eb1-ba6d7d7b6d8a: aborting pipeline replication: peer="{Voter server-636225f9-9964-7f2d-cff6-f868de129993 636225f9-9964-7f2d-cff6-f868de129993}"
--- PASS: TestRaft_LeaderFail (0.61s)
=== RUN   TestRaft_BehindFollower
2025-04-21T21:22:49.945Z [INFO]  server-ec5a8b64-b512-b633-2359-3fca9e728b3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec5a8b64-b512-b633-2359-3fca9e728b3e Address:ec5a8b64-b512-b633-2359-3fca9e728b3e} {Suffrage:Voter ID:server-9e702c02-0bbe-ad41-9f76-47a624ac7c6e Address:9e702c02-0bbe-ad41-9f76-47a624ac7c6e} {Suffrage:Voter ID:server-20a18f6b-1440-5499-01e4-48ca9a2a6226 Address:20a18f6b-1440-5499-01e4-48ca9a2a6226}]"
2025-04-21T21:22:49.945Z [INFO]  server-9e702c02-0bbe-ad41-9f76-47a624ac7c6e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec5a8b64-b512-b633-2359-3fca9e728b3e Address:ec5a8b64-b512-b633-2359-3fca9e728b3e} {Suffrage:Voter ID:server-9e702c02-0bbe-ad41-9f76-47a624ac7c6e Address:9e702c02-0bbe-ad41-9f76-47a624ac7c6e} {Suffrage:Voter ID:server-20a18f6b-1440-5499-01e4-48ca9a2a6226 Address:20a18f6b-1440-5499-01e4-48ca9a2a6226}]"
2025-04-21T21:22:49.945Z [INFO]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec5a8b64-b512-b633-2359-3fca9e728b3e Address:ec5a8b64-b512-b633-2359-3fca9e728b3e} {Suffrage:Voter ID:server-9e702c02-0bbe-ad41-9f76-47a624ac7c6e Address:9e702c02-0bbe-ad41-9f76-47a624ac7c6e} {Suffrage:Voter ID:server-20a18f6b-1440-5499-01e4-48ca9a2a6226 Address:20a18f6b-1440-5499-01e4-48ca9a2a6226}]"
2025-04-21T21:22:49.945Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:49.945Z [INFO]  server-ec5a8b64-b512-b633-2359-3fca9e728b3e: entering follower state: follower="Node at ec5a8b64-b512-b633-2359-3fca9e728b3e [Follower]" leader-address= leader-id=
2025-04-21T21:22:49.945Z [INFO]  server-9e702c02-0bbe-ad41-9f76-47a624ac7c6e: entering follower state: follower="Node at 9e702c02-0bbe-ad41-9f76-47a624ac7c6e [Follower]" leader-address= leader-id=
2025-04-21T21:22:49.945Z [INFO]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: entering follower state: follower="Node at 20a18f6b-1440-5499-01e4-48ca9a2a6226 [Follower]" leader-address= leader-id=
2025-04-21T21:22:50.045Z [WARN]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:50.045Z [INFO]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: entering candidate state: node="Node at 20a18f6b-1440-5499-01e4-48ca9a2a6226 [Candidate]" term=2
2025-04-21T21:22:50.045Z [WARN]  server-ec5a8b64-b512-b633-2359-3fca9e728b3e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:50.045Z [INFO]  server-ec5a8b64-b512-b633-2359-3fca9e728b3e: entering candidate state: node="Node at ec5a8b64-b512-b633-2359-3fca9e728b3e [Candidate]" term=2
2025-04-21T21:22:50.045Z [INFO]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: duplicate requestVote for same term: term=2
2025-04-21T21:22:50.045Z [WARN]  server-9e702c02-0bbe-ad41-9f76-47a624ac7c6e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:50.045Z [INFO]  server-9e702c02-0bbe-ad41-9f76-47a624ac7c6e: entering candidate state: node="Node at 9e702c02-0bbe-ad41-9f76-47a624ac7c6e [Candidate]" term=2
2025-04-21T21:22:50.045Z [INFO]  server-9e702c02-0bbe-ad41-9f76-47a624ac7c6e: duplicate requestVote for same term: term=2
2025-04-21T21:22:50.045Z [INFO]  server-ec5a8b64-b512-b633-2359-3fca9e728b3e: duplicate requestVote for same term: term=2
2025-04-21T21:22:50.045Z [INFO]  server-9e702c02-0bbe-ad41-9f76-47a624ac7c6e: duplicate requestVote for same term: term=2
2025-04-21T21:22:50.045Z [INFO]  server-ec5a8b64-b512-b633-2359-3fca9e728b3e: duplicate requestVote for same term: term=2
2025-04-21T21:22:50.045Z [INFO]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: duplicate requestVote for same term: term=2
2025-04-21T21:22:50.117Z [WARN]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: Election timeout reached, restarting election
2025-04-21T21:22:50.117Z [INFO]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: entering candidate state: node="Node at 20a18f6b-1440-5499-01e4-48ca9a2a6226 [Candidate]" term=3
2025-04-21T21:22:50.117Z [INFO]  server-9e702c02-0bbe-ad41-9f76-47a624ac7c6e: entering follower state: follower="Node at 9e702c02-0bbe-ad41-9f76-47a624ac7c6e [Follower]" leader-address= leader-id=
2025-04-21T21:22:50.117Z [INFO]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: election won: term=3 tally=2
2025-04-21T21:22:50.117Z [INFO]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: entering leader state: leader="Node at 20a18f6b-1440-5499-01e4-48ca9a2a6226 [Leader]"
2025-04-21T21:22:50.117Z [INFO]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: added peer, starting replication: peer=server-ec5a8b64-b512-b633-2359-3fca9e728b3e
2025-04-21T21:22:50.117Z [INFO]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: added peer, starting replication: peer=server-9e702c02-0bbe-ad41-9f76-47a624ac7c6e
2025-04-21T21:22:50.117Z [INFO]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: pipelining replication: peer="{Voter server-9e702c02-0bbe-ad41-9f76-47a624ac7c6e 9e702c02-0bbe-ad41-9f76-47a624ac7c6e}"
2025-04-21T21:22:50.117Z [INFO]  server-ec5a8b64-b512-b633-2359-3fca9e728b3e: entering follower state: follower="Node at ec5a8b64-b512-b633-2359-3fca9e728b3e [Follower]" leader-address= leader-id=
2025-04-21T21:22:50.117Z [INFO]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: pipelining replication: peer="{Voter server-ec5a8b64-b512-b633-2359-3fca9e728b3e ec5a8b64-b512-b633-2359-3fca9e728b3e}"
2025-04-21T21:22:50.225Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:50.117364331 +0000 UTC m=+8.639023654 (1 nodes), 171.575553ms from start of poll, 171.742274ms from cluster start. Timeout at 2025-04-21 21:22:50.222374411 +0000 UTC m=+8.744033734, 105.01008ms after stability
2025-04-21T21:22:50.225Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:22:50.333Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:22:50.225173655 +0000 UTC m=+8.746832938 (2 nodes), 920ns from start of poll, 279.551558ms from cluster start. Timeout at 2025-04-21 21:22:50.330181375 +0000 UTC m=+8.851840698, 105.00776ms after stability
2025-04-21T21:22:50.333Z [ERROR] server-20a18f6b-1440-5499-01e4-48ca9a2a6226: failed to pipeline appendEntries: peer="{Voter server-ec5a8b64-b512-b633-2359-3fca9e728b3e ec5a8b64-b512-b633-2359-3fca9e728b3e}" error="append pipeline closed"
2025-04-21T21:22:50.333Z [INFO]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: aborting pipeline replication: peer="{Voter server-ec5a8b64-b512-b633-2359-3fca9e728b3e ec5a8b64-b512-b633-2359-3fca9e728b3e}"
2025-04-21T21:22:50.333Z [ERROR] server-20a18f6b-1440-5499-01e4-48ca9a2a6226: failed to appendEntries to: peer="{Voter server-ec5a8b64-b512-b633-2359-3fca9e728b3e ec5a8b64-b512-b633-2359-3fca9e728b3e}" error="failed to connect to peer: ec5a8b64-b512-b633-2359-3fca9e728b3e"
    raft_test.go:459: [INFO] Finished apply without behind follower
2025-04-21T21:22:50.346Z [INFO]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: pipelining replication: peer="{Voter server-ec5a8b64-b512-b633-2359-3fca9e728b3e ec5a8b64-b512-b633-2359-3fca9e728b3e}"
2025-04-21T21:22:50.357Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:50.465Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:50.357134622 +0000 UTC m=+8.878793945 (1 nodes), 1.28µs from start of poll, 411.512565ms from cluster start. Timeout at 2025-04-21 21:22:50.462142262 +0000 UTC m=+8.983801545, 105.0076ms after stability
2025-04-21T21:22:50.465Z [INFO]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: aborting pipeline replication: peer="{Voter server-9e702c02-0bbe-ad41-9f76-47a624ac7c6e 9e702c02-0bbe-ad41-9f76-47a624ac7c6e}"
2025-04-21T21:22:50.465Z [INFO]  server-20a18f6b-1440-5499-01e4-48ca9a2a6226: aborting pipeline replication: peer="{Voter server-ec5a8b64-b512-b633-2359-3fca9e728b3e ec5a8b64-b512-b633-2359-3fca9e728b3e}"
--- PASS: TestRaft_BehindFollower (0.52s)
=== RUN   TestRaft_ApplyNonLeader
2025-04-21T21:22:50.465Z [INFO]  server-ccf47745-b22c-02c0-3d2d-0d130a114d72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccf47745-b22c-02c0-3d2d-0d130a114d72 Address:ccf47745-b22c-02c0-3d2d-0d130a114d72} {Suffrage:Voter ID:server-2c2e24e5-6d3c-113b-44c6-7ac914360afe Address:2c2e24e5-6d3c-113b-44c6-7ac914360afe} {Suffrage:Voter ID:server-da0f9d8d-520e-b83e-004c-4ad106e953dd Address:da0f9d8d-520e-b83e-004c-4ad106e953dd}]"
2025-04-21T21:22:50.465Z [INFO]  server-2c2e24e5-6d3c-113b-44c6-7ac914360afe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccf47745-b22c-02c0-3d2d-0d130a114d72 Address:ccf47745-b22c-02c0-3d2d-0d130a114d72} {Suffrage:Voter ID:server-2c2e24e5-6d3c-113b-44c6-7ac914360afe Address:2c2e24e5-6d3c-113b-44c6-7ac914360afe} {Suffrage:Voter ID:server-da0f9d8d-520e-b83e-004c-4ad106e953dd Address:da0f9d8d-520e-b83e-004c-4ad106e953dd}]"
2025-04-21T21:22:50.465Z [INFO]  server-da0f9d8d-520e-b83e-004c-4ad106e953dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ccf47745-b22c-02c0-3d2d-0d130a114d72 Address:ccf47745-b22c-02c0-3d2d-0d130a114d72} {Suffrage:Voter ID:server-2c2e24e5-6d3c-113b-44c6-7ac914360afe Address:2c2e24e5-6d3c-113b-44c6-7ac914360afe} {Suffrage:Voter ID:server-da0f9d8d-520e-b83e-004c-4ad106e953dd Address:da0f9d8d-520e-b83e-004c-4ad106e953dd}]"
2025-04-21T21:22:50.465Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:50.465Z [INFO]  server-ccf47745-b22c-02c0-3d2d-0d130a114d72: entering follower state: follower="Node at ccf47745-b22c-02c0-3d2d-0d130a114d72 [Follower]" leader-address= leader-id=
2025-04-21T21:22:50.465Z [INFO]  server-2c2e24e5-6d3c-113b-44c6-7ac914360afe: entering follower state: follower="Node at 2c2e24e5-6d3c-113b-44c6-7ac914360afe [Follower]" leader-address= leader-id=
2025-04-21T21:22:50.465Z [INFO]  server-da0f9d8d-520e-b83e-004c-4ad106e953dd: entering follower state: follower="Node at da0f9d8d-520e-b83e-004c-4ad106e953dd [Follower]" leader-address= leader-id=
2025-04-21T21:22:50.549Z [WARN]  server-ccf47745-b22c-02c0-3d2d-0d130a114d72: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:50.549Z [INFO]  server-ccf47745-b22c-02c0-3d2d-0d130a114d72: entering candidate state: node="Node at ccf47745-b22c-02c0-3d2d-0d130a114d72 [Candidate]" term=2
2025-04-21T21:22:50.549Z [INFO]  server-ccf47745-b22c-02c0-3d2d-0d130a114d72: election won: term=2 tally=2
2025-04-21T21:22:50.549Z [INFO]  server-ccf47745-b22c-02c0-3d2d-0d130a114d72: entering leader state: leader="Node at ccf47745-b22c-02c0-3d2d-0d130a114d72 [Leader]"
2025-04-21T21:22:50.549Z [INFO]  server-ccf47745-b22c-02c0-3d2d-0d130a114d72: added peer, starting replication: peer=server-2c2e24e5-6d3c-113b-44c6-7ac914360afe
2025-04-21T21:22:50.549Z [INFO]  server-ccf47745-b22c-02c0-3d2d-0d130a114d72: added peer, starting replication: peer=server-da0f9d8d-520e-b83e-004c-4ad106e953dd
2025-04-21T21:22:50.549Z [INFO]  server-ccf47745-b22c-02c0-3d2d-0d130a114d72: pipelining replication: peer="{Voter server-da0f9d8d-520e-b83e-004c-4ad106e953dd da0f9d8d-520e-b83e-004c-4ad106e953dd}"
2025-04-21T21:22:50.549Z [WARN]  server-2c2e24e5-6d3c-113b-44c6-7ac914360afe: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:50.549Z [INFO]  server-2c2e24e5-6d3c-113b-44c6-7ac914360afe: entering candidate state: node="Node at 2c2e24e5-6d3c-113b-44c6-7ac914360afe [Candidate]" term=2
2025-04-21T21:22:50.549Z [WARN]  server-da0f9d8d-520e-b83e-004c-4ad106e953dd: rejecting vote request since we have a leader: from=2c2e24e5-6d3c-113b-44c6-7ac914360afe leader=ccf47745-b22c-02c0-3d2d-0d130a114d72 leader-id=server-ccf47745-b22c-02c0-3d2d-0d130a114d72
2025-04-21T21:22:50.549Z [INFO]  server-2c2e24e5-6d3c-113b-44c6-7ac914360afe: duplicate requestVote for same term: term=2
2025-04-21T21:22:50.549Z [INFO]  server-2c2e24e5-6d3c-113b-44c6-7ac914360afe: entering follower state: follower="Node at 2c2e24e5-6d3c-113b-44c6-7ac914360afe [Follower]" leader-address=ccf47745-b22c-02c0-3d2d-0d130a114d72 leader-id=server-ccf47745-b22c-02c0-3d2d-0d130a114d72
2025-04-21T21:22:50.549Z [INFO]  server-ccf47745-b22c-02c0-3d2d-0d130a114d72: pipelining replication: peer="{Voter server-2c2e24e5-6d3c-113b-44c6-7ac914360afe 2c2e24e5-6d3c-113b-44c6-7ac914360afe}"
2025-04-21T21:22:50.549Z [WARN]  server-ccf47745-b22c-02c0-3d2d-0d130a114d72: rejecting vote request since we have a leader: from=2c2e24e5-6d3c-113b-44c6-7ac914360afe leader=ccf47745-b22c-02c0-3d2d-0d130a114d72 leader-id=server-ccf47745-b22c-02c0-3d2d-0d130a114d72
2025-04-21T21:22:50.657Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:50.549492995 +0000 UTC m=+9.071152318 (1 nodes), 83.698123ms from start of poll, 83.873364ms from cluster start. Timeout at 2025-04-21 21:22:50.654499235 +0000 UTC m=+9.176158558, 105.00624ms after stability
2025-04-21T21:22:50.657Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:22:50.765Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:22:50.657184878 +0000 UTC m=+9.178844201 (2 nodes), 840ns from start of poll, 191.565247ms from cluster start. Timeout at 2025-04-21 21:22:50.762196878 +0000 UTC m=+9.283856201, 105.012ms after stability
2025-04-21T21:22:50.765Z [INFO]  server-ccf47745-b22c-02c0-3d2d-0d130a114d72: aborting pipeline replication: peer="{Voter server-da0f9d8d-520e-b83e-004c-4ad106e953dd da0f9d8d-520e-b83e-004c-4ad106e953dd}"
2025-04-21T21:22:50.765Z [INFO]  server-ccf47745-b22c-02c0-3d2d-0d130a114d72: aborting pipeline replication: peer="{Voter server-2c2e24e5-6d3c-113b-44c6-7ac914360afe 2c2e24e5-6d3c-113b-44c6-7ac914360afe}"
--- PASS: TestRaft_ApplyNonLeader (0.30s)
=== RUN   TestRaft_ApplyConcurrent
2025-04-21T21:22:50.766Z [INFO]  server-850fc9e5-3004-7674-161e-b802f0844dce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-850fc9e5-3004-7674-161e-b802f0844dce Address:850fc9e5-3004-7674-161e-b802f0844dce} {Suffrage:Voter ID:server-ea1b45c7-adb4-6a98-9d3c-da658a3fab7c Address:ea1b45c7-adb4-6a98-9d3c-da658a3fab7c} {Suffrage:Voter ID:server-2e4de166-20a3-e074-39bf-4853e98de9da Address:2e4de166-20a3-e074-39bf-4853e98de9da}]"
2025-04-21T21:22:50.766Z [INFO]  server-ea1b45c7-adb4-6a98-9d3c-da658a3fab7c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-850fc9e5-3004-7674-161e-b802f0844dce Address:850fc9e5-3004-7674-161e-b802f0844dce} {Suffrage:Voter ID:server-ea1b45c7-adb4-6a98-9d3c-da658a3fab7c Address:ea1b45c7-adb4-6a98-9d3c-da658a3fab7c} {Suffrage:Voter ID:server-2e4de166-20a3-e074-39bf-4853e98de9da Address:2e4de166-20a3-e074-39bf-4853e98de9da}]"
2025-04-21T21:22:50.766Z [INFO]  server-2e4de166-20a3-e074-39bf-4853e98de9da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-850fc9e5-3004-7674-161e-b802f0844dce Address:850fc9e5-3004-7674-161e-b802f0844dce} {Suffrage:Voter ID:server-ea1b45c7-adb4-6a98-9d3c-da658a3fab7c Address:ea1b45c7-adb4-6a98-9d3c-da658a3fab7c} {Suffrage:Voter ID:server-2e4de166-20a3-e074-39bf-4853e98de9da Address:2e4de166-20a3-e074-39bf-4853e98de9da}]"
2025-04-21T21:22:50.766Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:50.766Z [INFO]  server-850fc9e5-3004-7674-161e-b802f0844dce: entering follower state: follower="Node at 850fc9e5-3004-7674-161e-b802f0844dce [Follower]" leader-address= leader-id=
2025-04-21T21:22:50.766Z [INFO]  server-ea1b45c7-adb4-6a98-9d3c-da658a3fab7c: entering follower state: follower="Node at ea1b45c7-adb4-6a98-9d3c-da658a3fab7c [Follower]" leader-address= leader-id=
2025-04-21T21:22:50.766Z [INFO]  server-2e4de166-20a3-e074-39bf-4853e98de9da: entering follower state: follower="Node at 2e4de166-20a3-e074-39bf-4853e98de9da [Follower]" leader-address= leader-id=
2025-04-21T21:22:50.901Z [WARN]  server-2e4de166-20a3-e074-39bf-4853e98de9da: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:50.901Z [INFO]  server-2e4de166-20a3-e074-39bf-4853e98de9da: entering candidate state: node="Node at 2e4de166-20a3-e074-39bf-4853e98de9da [Candidate]" term=2
2025-04-21T21:22:50.901Z [WARN]  server-ea1b45c7-adb4-6a98-9d3c-da658a3fab7c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:50.901Z [INFO]  server-ea1b45c7-adb4-6a98-9d3c-da658a3fab7c: entering candidate state: node="Node at ea1b45c7-adb4-6a98-9d3c-da658a3fab7c [Candidate]" term=2
2025-04-21T21:22:50.901Z [INFO]  server-ea1b45c7-adb4-6a98-9d3c-da658a3fab7c: duplicate requestVote for same term: term=2
2025-04-21T21:22:50.901Z [INFO]  server-2e4de166-20a3-e074-39bf-4853e98de9da: election won: term=2 tally=2
2025-04-21T21:22:50.901Z [INFO]  server-2e4de166-20a3-e074-39bf-4853e98de9da: entering leader state: leader="Node at 2e4de166-20a3-e074-39bf-4853e98de9da [Leader]"
2025-04-21T21:22:50.901Z [INFO]  server-2e4de166-20a3-e074-39bf-4853e98de9da: added peer, starting replication: peer=server-850fc9e5-3004-7674-161e-b802f0844dce
2025-04-21T21:22:50.901Z [INFO]  server-2e4de166-20a3-e074-39bf-4853e98de9da: added peer, starting replication: peer=server-ea1b45c7-adb4-6a98-9d3c-da658a3fab7c
2025-04-21T21:22:50.901Z [INFO]  server-ea1b45c7-adb4-6a98-9d3c-da658a3fab7c: entering follower state: follower="Node at ea1b45c7-adb4-6a98-9d3c-da658a3fab7c [Follower]" leader-address=2e4de166-20a3-e074-39bf-4853e98de9da leader-id=server-2e4de166-20a3-e074-39bf-4853e98de9da
2025-04-21T21:22:50.901Z [INFO]  server-2e4de166-20a3-e074-39bf-4853e98de9da: pipelining replication: peer="{Voter server-ea1b45c7-adb4-6a98-9d3c-da658a3fab7c ea1b45c7-adb4-6a98-9d3c-da658a3fab7c}"
2025-04-21T21:22:50.901Z [INFO]  server-850fc9e5-3004-7674-161e-b802f0844dce: duplicate requestVote for same term: term=2
2025-04-21T21:22:50.901Z [INFO]  server-2e4de166-20a3-e074-39bf-4853e98de9da: pipelining replication: peer="{Voter server-850fc9e5-3004-7674-161e-b802f0844dce 850fc9e5-3004-7674-161e-b802f0844dce}"
2025-04-21T21:22:50.901Z [WARN]  server-2e4de166-20a3-e074-39bf-4853e98de9da: rejecting vote request since we have a leader: from=ea1b45c7-adb4-6a98-9d3c-da658a3fab7c leader=2e4de166-20a3-e074-39bf-4853e98de9da leader-id=server-2e4de166-20a3-e074-39bf-4853e98de9da
2025-04-21T21:22:51.109Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:50.901487 +0000 UTC m=+9.423146323 (1 nodes), 135.123468ms from start of poll, 135.348549ms from cluster start. Timeout at 2025-04-21 21:22:51.10648824 +0000 UTC m=+9.628147563, 205.00124ms after stability
2025-04-21T21:22:51.123Z [INFO]  server-2e4de166-20a3-e074-39bf-4853e98de9da: aborting pipeline replication: peer="{Voter server-ea1b45c7-adb4-6a98-9d3c-da658a3fab7c ea1b45c7-adb4-6a98-9d3c-da658a3fab7c}"
2025-04-21T21:22:51.123Z [INFO]  server-2e4de166-20a3-e074-39bf-4853e98de9da: aborting pipeline replication: peer="{Voter server-850fc9e5-3004-7674-161e-b802f0844dce 850fc9e5-3004-7674-161e-b802f0844dce}"
--- PASS: TestRaft_ApplyConcurrent (0.36s)
=== RUN   TestRaft_ApplyConcurrent_Timeout
2025-04-21T21:22:51.123Z [INFO]  server-edd10a3c-e04f-2250-b96f-5f0e3951deb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edd10a3c-e04f-2250-b96f-5f0e3951deb8 Address:edd10a3c-e04f-2250-b96f-5f0e3951deb8}]"
2025-04-21T21:22:51.123Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:51.123Z [INFO]  server-edd10a3c-e04f-2250-b96f-5f0e3951deb8: entering follower state: follower="Node at edd10a3c-e04f-2250-b96f-5f0e3951deb8 [Follower]" leader-address= leader-id=
2025-04-21T21:22:51.269Z [WARN]  server-edd10a3c-e04f-2250-b96f-5f0e3951deb8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:51.269Z [INFO]  server-edd10a3c-e04f-2250-b96f-5f0e3951deb8: entering candidate state: node="Node at edd10a3c-e04f-2250-b96f-5f0e3951deb8 [Candidate]" term=2
2025-04-21T21:22:51.269Z [INFO]  server-edd10a3c-e04f-2250-b96f-5f0e3951deb8: election won: term=2 tally=1
2025-04-21T21:22:51.269Z [INFO]  server-edd10a3c-e04f-2250-b96f-5f0e3951deb8: entering leader state: leader="Node at edd10a3c-e04f-2250-b96f-5f0e3951deb8 [Leader]"
2025-04-21T21:22:51.473Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:51.269237431 +0000 UTC m=+9.790896754 (1 nodes), 145.809019ms from start of poll, 145.90598ms from cluster start. Timeout at 2025-04-21 21:22:51.470241431 +0000 UTC m=+9.991900754, 201.004ms after stability
--- PASS: TestRaft_ApplyConcurrent_Timeout (0.35s)
=== RUN   TestRaft_JoinNode
2025-04-21T21:22:51.474Z [INFO]  server-e7380d10-5489-73c0-68d6-cd9cb7162991: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7380d10-5489-73c0-68d6-cd9cb7162991 Address:e7380d10-5489-73c0-68d6-cd9cb7162991} {Suffrage:Voter ID:server-b1332067-ac5f-26ce-13e7-5cc1ce855b1a Address:b1332067-ac5f-26ce-13e7-5cc1ce855b1a}]"
2025-04-21T21:22:51.474Z [INFO]  server-b1332067-ac5f-26ce-13e7-5cc1ce855b1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e7380d10-5489-73c0-68d6-cd9cb7162991 Address:e7380d10-5489-73c0-68d6-cd9cb7162991} {Suffrage:Voter ID:server-b1332067-ac5f-26ce-13e7-5cc1ce855b1a Address:b1332067-ac5f-26ce-13e7-5cc1ce855b1a}]"
2025-04-21T21:22:51.474Z [INFO]  server-c9c4fecd-1c22-4554-c2bd-3315ca1bf614: initial configuration: index=0 servers=[]
2025-04-21T21:22:51.475Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:51.475Z [INFO]  server-e7380d10-5489-73c0-68d6-cd9cb7162991: entering follower state: follower="Node at e7380d10-5489-73c0-68d6-cd9cb7162991 [Follower]" leader-address= leader-id=
2025-04-21T21:22:51.475Z [INFO]  server-b1332067-ac5f-26ce-13e7-5cc1ce855b1a: entering follower state: follower="Node at b1332067-ac5f-26ce-13e7-5cc1ce855b1a [Follower]" leader-address= leader-id=
2025-04-21T21:22:51.475Z [INFO]  server-c9c4fecd-1c22-4554-c2bd-3315ca1bf614: entering follower state: follower="Node at c9c4fecd-1c22-4554-c2bd-3315ca1bf614 [Follower]" leader-address= leader-id=
2025-04-21T21:22:51.529Z [WARN]  server-c9c4fecd-1c22-4554-c2bd-3315ca1bf614: no known peers, aborting election
2025-04-21T21:22:51.533Z [WARN]  server-b1332067-ac5f-26ce-13e7-5cc1ce855b1a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:51.533Z [INFO]  server-b1332067-ac5f-26ce-13e7-5cc1ce855b1a: entering candidate state: node="Node at b1332067-ac5f-26ce-13e7-5cc1ce855b1a [Candidate]" term=2
2025-04-21T21:22:51.533Z [INFO]  server-b1332067-ac5f-26ce-13e7-5cc1ce855b1a: election won: term=2 tally=2
2025-04-21T21:22:51.533Z [INFO]  server-b1332067-ac5f-26ce-13e7-5cc1ce855b1a: entering leader state: leader="Node at b1332067-ac5f-26ce-13e7-5cc1ce855b1a [Leader]"
2025-04-21T21:22:51.533Z [INFO]  server-b1332067-ac5f-26ce-13e7-5cc1ce855b1a: added peer, starting replication: peer=server-e7380d10-5489-73c0-68d6-cd9cb7162991
2025-04-21T21:22:51.533Z [INFO]  server-b1332067-ac5f-26ce-13e7-5cc1ce855b1a: pipelining replication: peer="{Voter server-e7380d10-5489-73c0-68d6-cd9cb7162991 e7380d10-5489-73c0-68d6-cd9cb7162991}"
2025-04-21T21:22:51.641Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:51.533328566 +0000 UTC m=+10.054987849 (1 nodes), 58.314552ms from start of poll, 58.496393ms from cluster start. Timeout at 2025-04-21 21:22:51.638330926 +0000 UTC m=+10.159990409, 105.00256ms after stability
2025-04-21T21:22:51.641Z [INFO]  server-b1332067-ac5f-26ce-13e7-5cc1ce855b1a: updating configuration: command=AddVoter server-id=server-c9c4fecd-1c22-4554-c2bd-3315ca1bf614 server-addr=c9c4fecd-1c22-4554-c2bd-3315ca1bf614 servers="[{Suffrage:Voter ID:server-e7380d10-5489-73c0-68d6-cd9cb7162991 Address:e7380d10-5489-73c0-68d6-cd9cb7162991} {Suffrage:Voter ID:server-b1332067-ac5f-26ce-13e7-5cc1ce855b1a Address:b1332067-ac5f-26ce-13e7-5cc1ce855b1a} {Suffrage:Voter ID:server-c9c4fecd-1c22-4554-c2bd-3315ca1bf614 Address:c9c4fecd-1c22-4554-c2bd-3315ca1bf614}]"
2025-04-21T21:22:51.641Z [INFO]  server-b1332067-ac5f-26ce-13e7-5cc1ce855b1a: added peer, starting replication: peer=server-c9c4fecd-1c22-4554-c2bd-3315ca1bf614
2025-04-21T21:22:51.641Z [WARN]  server-c9c4fecd-1c22-4554-c2bd-3315ca1bf614: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-04-21T21:22:51.641Z [WARN]  server-b1332067-ac5f-26ce-13e7-5cc1ce855b1a: appendEntries rejected, sending older logs: peer="{Voter server-c9c4fecd-1c22-4554-c2bd-3315ca1bf614 c9c4fecd-1c22-4554-c2bd-3315ca1bf614}" next=1
2025-04-21T21:22:51.641Z [INFO]  server-b1332067-ac5f-26ce-13e7-5cc1ce855b1a: pipelining replication: peer="{Voter server-c9c4fecd-1c22-4554-c2bd-3315ca1bf614 c9c4fecd-1c22-4554-c2bd-3315ca1bf614}"
2025-04-21T21:22:51.641Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:51.749Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:51.641412572 +0000 UTC m=+10.163071895 (1 nodes), 720ns from start of poll, 166.580439ms from cluster start. Timeout at 2025-04-21 21:22:51.746417572 +0000 UTC m=+10.268076895, 105.005ms after stability
2025-04-21T21:22:51.749Z [INFO]  server-b1332067-ac5f-26ce-13e7-5cc1ce855b1a: aborting pipeline replication: peer="{Voter server-c9c4fecd-1c22-4554-c2bd-3315ca1bf614 c9c4fecd-1c22-4554-c2bd-3315ca1bf614}"
2025-04-21T21:22:51.749Z [INFO]  server-b1332067-ac5f-26ce-13e7-5cc1ce855b1a: aborting pipeline replication: peer="{Voter server-e7380d10-5489-73c0-68d6-cd9cb7162991 e7380d10-5489-73c0-68d6-cd9cb7162991}"
--- PASS: TestRaft_JoinNode (0.27s)
=== RUN   TestRaft_JoinNode_ConfigStore
2025-04-21T21:22:51.749Z [INFO]  server-47dafed8-c754-6e4f-84e2-92397d0aa792: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47dafed8-c754-6e4f-84e2-92397d0aa792 Address:47dafed8-c754-6e4f-84e2-92397d0aa792}]"
2025-04-21T21:22:51.749Z [INFO]  server-d2574bdc-3fc3-e01f-e1c6-18b616aa5235: initial configuration: index=0 servers=[]
2025-04-21T21:22:51.749Z [INFO]  server-efc11cbc-4d94-0bef-c9e6-d32ddab48f8c: initial configuration: index=0 servers=[]
2025-04-21T21:22:51.749Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:51.749Z [INFO]  server-47dafed8-c754-6e4f-84e2-92397d0aa792: entering follower state: follower="Node at 47dafed8-c754-6e4f-84e2-92397d0aa792 [Follower]" leader-address= leader-id=
2025-04-21T21:22:51.749Z [INFO]  server-d2574bdc-3fc3-e01f-e1c6-18b616aa5235: entering follower state: follower="Node at d2574bdc-3fc3-e01f-e1c6-18b616aa5235 [Follower]" leader-address= leader-id=
2025-04-21T21:22:51.749Z [INFO]  server-efc11cbc-4d94-0bef-c9e6-d32ddab48f8c: entering follower state: follower="Node at efc11cbc-4d94-0bef-c9e6-d32ddab48f8c [Follower]" leader-address= leader-id=
2025-04-21T21:22:51.801Z [WARN]  server-efc11cbc-4d94-0bef-c9e6-d32ddab48f8c: no known peers, aborting election
2025-04-21T21:22:51.809Z [WARN]  server-47dafed8-c754-6e4f-84e2-92397d0aa792: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:51.809Z [INFO]  server-47dafed8-c754-6e4f-84e2-92397d0aa792: entering candidate state: node="Node at 47dafed8-c754-6e4f-84e2-92397d0aa792 [Candidate]" term=2
2025-04-21T21:22:51.809Z [INFO]  server-47dafed8-c754-6e4f-84e2-92397d0aa792: election won: term=2 tally=1
2025-04-21T21:22:51.809Z [INFO]  server-47dafed8-c754-6e4f-84e2-92397d0aa792: entering leader state: leader="Node at 47dafed8-c754-6e4f-84e2-92397d0aa792 [Leader]"
2025-04-21T21:22:51.825Z [WARN]  server-d2574bdc-3fc3-e01f-e1c6-18b616aa5235: no known peers, aborting election
2025-04-21T21:22:51.917Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:51.80925278 +0000 UTC m=+10.330912103 (1 nodes), 59.454399ms from start of poll, 59.686721ms from cluster start. Timeout at 2025-04-21 21:22:51.91425618 +0000 UTC m=+10.435915503, 105.0034ms after stability
2025-04-21T21:22:51.917Z [INFO]  server-47dafed8-c754-6e4f-84e2-92397d0aa792: updating configuration: command=AddVoter server-id=server-d2574bdc-3fc3-e01f-e1c6-18b616aa5235 server-addr=d2574bdc-3fc3-e01f-e1c6-18b616aa5235 servers="[{Suffrage:Voter ID:server-47dafed8-c754-6e4f-84e2-92397d0aa792 Address:47dafed8-c754-6e4f-84e2-92397d0aa792} {Suffrage:Voter ID:server-d2574bdc-3fc3-e01f-e1c6-18b616aa5235 Address:d2574bdc-3fc3-e01f-e1c6-18b616aa5235}]"
2025-04-21T21:22:51.917Z [INFO]  server-47dafed8-c754-6e4f-84e2-92397d0aa792: added peer, starting replication: peer=server-d2574bdc-3fc3-e01f-e1c6-18b616aa5235
2025-04-21T21:22:51.917Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:51.917Z [WARN]  server-d2574bdc-3fc3-e01f-e1c6-18b616aa5235: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-04-21T21:22:51.917Z [WARN]  server-47dafed8-c754-6e4f-84e2-92397d0aa792: appendEntries rejected, sending older logs: peer="{Voter server-d2574bdc-3fc3-e01f-e1c6-18b616aa5235 d2574bdc-3fc3-e01f-e1c6-18b616aa5235}" next=1
2025-04-21T21:22:51.917Z [INFO]  server-47dafed8-c754-6e4f-84e2-92397d0aa792: pipelining replication: peer="{Voter server-d2574bdc-3fc3-e01f-e1c6-18b616aa5235 d2574bdc-3fc3-e01f-e1c6-18b616aa5235}"
2025-04-21T21:22:52.033Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:51.917378387 +0000 UTC m=+10.439037710 (1 nodes), 640ns from start of poll, 167.812328ms from cluster start. Timeout at 2025-04-21 21:22:52.022388787 +0000 UTC m=+10.544048110, 105.0104ms after stability
2025-04-21T21:22:52.033Z [INFO]  server-47dafed8-c754-6e4f-84e2-92397d0aa792: updating configuration: command=AddVoter server-id=server-efc11cbc-4d94-0bef-c9e6-d32ddab48f8c server-addr=efc11cbc-4d94-0bef-c9e6-d32ddab48f8c servers="[{Suffrage:Voter ID:server-47dafed8-c754-6e4f-84e2-92397d0aa792 Address:47dafed8-c754-6e4f-84e2-92397d0aa792} {Suffrage:Voter ID:server-d2574bdc-3fc3-e01f-e1c6-18b616aa5235 Address:d2574bdc-3fc3-e01f-e1c6-18b616aa5235} {Suffrage:Voter ID:server-efc11cbc-4d94-0bef-c9e6-d32ddab48f8c Address:efc11cbc-4d94-0bef-c9e6-d32ddab48f8c}]"
2025-04-21T21:22:52.033Z [INFO]  server-47dafed8-c754-6e4f-84e2-92397d0aa792: added peer, starting replication: peer=server-efc11cbc-4d94-0bef-c9e6-d32ddab48f8c
2025-04-21T21:22:52.033Z [WARN]  server-efc11cbc-4d94-0bef-c9e6-d32ddab48f8c: failed to get previous log: previous-index=4 last-index=0 error="log not found"
2025-04-21T21:22:52.033Z [WARN]  server-47dafed8-c754-6e4f-84e2-92397d0aa792: appendEntries rejected, sending older logs: peer="{Voter server-efc11cbc-4d94-0bef-c9e6-d32ddab48f8c efc11cbc-4d94-0bef-c9e6-d32ddab48f8c}" next=1
2025-04-21T21:22:52.033Z [INFO]  server-47dafed8-c754-6e4f-84e2-92397d0aa792: pipelining replication: peer="{Voter server-efc11cbc-4d94-0bef-c9e6-d32ddab48f8c efc11cbc-4d94-0bef-c9e6-d32ddab48f8c}"
2025-04-21T21:22:52.033Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:52.145Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:52.033469207 +0000 UTC m=+10.555128530 (1 nodes), 440ns from start of poll, 283.903148ms from cluster start. Timeout at 2025-04-21 21:22:52.138474687 +0000 UTC m=+10.660134010, 105.00548ms after stability
2025-04-21T21:22:52.145Z [INFO]  server-47dafed8-c754-6e4f-84e2-92397d0aa792: aborting pipeline replication: peer="{Voter server-d2574bdc-3fc3-e01f-e1c6-18b616aa5235 d2574bdc-3fc3-e01f-e1c6-18b616aa5235}"
2025-04-21T21:22:52.145Z [INFO]  server-47dafed8-c754-6e4f-84e2-92397d0aa792: aborting pipeline replication: peer="{Voter server-efc11cbc-4d94-0bef-c9e6-d32ddab48f8c efc11cbc-4d94-0bef-c9e6-d32ddab48f8c}"
--- PASS: TestRaft_JoinNode_ConfigStore (0.40s)
=== RUN   TestRaft_RemoveFollower
2025-04-21T21:22:52.145Z [INFO]  server-489ee399-57cd-58f1-6200-eacddcef6326: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-489ee399-57cd-58f1-6200-eacddcef6326 Address:489ee399-57cd-58f1-6200-eacddcef6326} {Suffrage:Voter ID:server-fbb138a9-f330-a1df-2c9c-0905d01eeb27 Address:fbb138a9-f330-a1df-2c9c-0905d01eeb27} {Suffrage:Voter ID:server-54b2ab10-4b36-af48-f1d6-3251aaac509d Address:54b2ab10-4b36-af48-f1d6-3251aaac509d}]"
2025-04-21T21:22:52.145Z [INFO]  server-fbb138a9-f330-a1df-2c9c-0905d01eeb27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-489ee399-57cd-58f1-6200-eacddcef6326 Address:489ee399-57cd-58f1-6200-eacddcef6326} {Suffrage:Voter ID:server-fbb138a9-f330-a1df-2c9c-0905d01eeb27 Address:fbb138a9-f330-a1df-2c9c-0905d01eeb27} {Suffrage:Voter ID:server-54b2ab10-4b36-af48-f1d6-3251aaac509d Address:54b2ab10-4b36-af48-f1d6-3251aaac509d}]"
2025-04-21T21:22:52.145Z [INFO]  server-54b2ab10-4b36-af48-f1d6-3251aaac509d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-489ee399-57cd-58f1-6200-eacddcef6326 Address:489ee399-57cd-58f1-6200-eacddcef6326} {Suffrage:Voter ID:server-fbb138a9-f330-a1df-2c9c-0905d01eeb27 Address:fbb138a9-f330-a1df-2c9c-0905d01eeb27} {Suffrage:Voter ID:server-54b2ab10-4b36-af48-f1d6-3251aaac509d Address:54b2ab10-4b36-af48-f1d6-3251aaac509d}]"
2025-04-21T21:22:52.145Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:52.145Z [INFO]  server-489ee399-57cd-58f1-6200-eacddcef6326: entering follower state: follower="Node at 489ee399-57cd-58f1-6200-eacddcef6326 [Follower]" leader-address= leader-id=
2025-04-21T21:22:52.145Z [INFO]  server-fbb138a9-f330-a1df-2c9c-0905d01eeb27: entering follower state: follower="Node at fbb138a9-f330-a1df-2c9c-0905d01eeb27 [Follower]" leader-address= leader-id=
2025-04-21T21:22:52.145Z [INFO]  server-54b2ab10-4b36-af48-f1d6-3251aaac509d: entering follower state: follower="Node at 54b2ab10-4b36-af48-f1d6-3251aaac509d [Follower]" leader-address= leader-id=
2025-04-21T21:22:52.205Z [WARN]  server-fbb138a9-f330-a1df-2c9c-0905d01eeb27: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:52.205Z [INFO]  server-fbb138a9-f330-a1df-2c9c-0905d01eeb27: entering candidate state: node="Node at fbb138a9-f330-a1df-2c9c-0905d01eeb27 [Candidate]" term=2
2025-04-21T21:22:52.205Z [INFO]  server-fbb138a9-f330-a1df-2c9c-0905d01eeb27: election won: term=2 tally=2
2025-04-21T21:22:52.205Z [INFO]  server-fbb138a9-f330-a1df-2c9c-0905d01eeb27: entering leader state: leader="Node at fbb138a9-f330-a1df-2c9c-0905d01eeb27 [Leader]"
2025-04-21T21:22:52.205Z [INFO]  server-fbb138a9-f330-a1df-2c9c-0905d01eeb27: added peer, starting replication: peer=server-489ee399-57cd-58f1-6200-eacddcef6326
2025-04-21T21:22:52.205Z [INFO]  server-fbb138a9-f330-a1df-2c9c-0905d01eeb27: added peer, starting replication: peer=server-54b2ab10-4b36-af48-f1d6-3251aaac509d
2025-04-21T21:22:52.205Z [INFO]  server-fbb138a9-f330-a1df-2c9c-0905d01eeb27: pipelining replication: peer="{Voter server-54b2ab10-4b36-af48-f1d6-3251aaac509d 54b2ab10-4b36-af48-f1d6-3251aaac509d}"
2025-04-21T21:22:52.205Z [INFO]  server-fbb138a9-f330-a1df-2c9c-0905d01eeb27: pipelining replication: peer="{Voter server-489ee399-57cd-58f1-6200-eacddcef6326 489ee399-57cd-58f1-6200-eacddcef6326}"
2025-04-21T21:22:52.321Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:52.205438123 +0000 UTC m=+10.727097446 (1 nodes), 59.635081ms from start of poll, 59.816602ms from cluster start. Timeout at 2025-04-21 21:22:52.310441243 +0000 UTC m=+10.832100526, 105.00308ms after stability
2025-04-21T21:22:52.329Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:22:52.437Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:22:52.329216714 +0000 UTC m=+10.850875997 (2 nodes), 840ns from start of poll, 183.595153ms from cluster start. Timeout at 2025-04-21 21:22:52.434229914 +0000 UTC m=+10.955889237, 105.01324ms after stability
2025-04-21T21:22:52.437Z [INFO]  server-fbb138a9-f330-a1df-2c9c-0905d01eeb27: updating configuration: command=RemoveServer server-id=server-489ee399-57cd-58f1-6200-eacddcef6326 server-addr= servers="[{Suffrage:Voter ID:server-fbb138a9-f330-a1df-2c9c-0905d01eeb27 Address:fbb138a9-f330-a1df-2c9c-0905d01eeb27} {Suffrage:Voter ID:server-54b2ab10-4b36-af48-f1d6-3251aaac509d Address:54b2ab10-4b36-af48-f1d6-3251aaac509d}]"
2025-04-21T21:22:52.437Z [INFO]  server-fbb138a9-f330-a1df-2c9c-0905d01eeb27: removed peer, stopping replication: peer=server-489ee399-57cd-58f1-6200-eacddcef6326 last-index=3
2025-04-21T21:22:52.437Z [INFO]  server-fbb138a9-f330-a1df-2c9c-0905d01eeb27: aborting pipeline replication: peer="{Voter server-489ee399-57cd-58f1-6200-eacddcef6326 489ee399-57cd-58f1-6200-eacddcef6326}"
2025-04-21T21:22:52.529Z [WARN]  server-489ee399-57cd-58f1-6200-eacddcef6326: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-04-21T21:22:52.545Z [INFO]  server-fbb138a9-f330-a1df-2c9c-0905d01eeb27: aborting pipeline replication: peer="{Voter server-54b2ab10-4b36-af48-f1d6-3251aaac509d 54b2ab10-4b36-af48-f1d6-3251aaac509d}"
--- PASS: TestRaft_RemoveFollower (0.40s)
=== RUN   TestRaft_RemoveLeader
2025-04-21T21:22:52.545Z [INFO]  server-5e7f5966-b880-0d84-d244-c725ee24a960: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e7f5966-b880-0d84-d244-c725ee24a960 Address:5e7f5966-b880-0d84-d244-c725ee24a960} {Suffrage:Voter ID:server-f37280b4-dc53-1fb7-e6dc-243892a42d3b Address:f37280b4-dc53-1fb7-e6dc-243892a42d3b} {Suffrage:Voter ID:server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380 Address:ab5ede7f-a93a-bb8b-25e5-df2f68fe2380}]"
2025-04-21T21:22:52.545Z [INFO]  server-f37280b4-dc53-1fb7-e6dc-243892a42d3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e7f5966-b880-0d84-d244-c725ee24a960 Address:5e7f5966-b880-0d84-d244-c725ee24a960} {Suffrage:Voter ID:server-f37280b4-dc53-1fb7-e6dc-243892a42d3b Address:f37280b4-dc53-1fb7-e6dc-243892a42d3b} {Suffrage:Voter ID:server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380 Address:ab5ede7f-a93a-bb8b-25e5-df2f68fe2380}]"
2025-04-21T21:22:52.545Z [INFO]  server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5e7f5966-b880-0d84-d244-c725ee24a960 Address:5e7f5966-b880-0d84-d244-c725ee24a960} {Suffrage:Voter ID:server-f37280b4-dc53-1fb7-e6dc-243892a42d3b Address:f37280b4-dc53-1fb7-e6dc-243892a42d3b} {Suffrage:Voter ID:server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380 Address:ab5ede7f-a93a-bb8b-25e5-df2f68fe2380}]"
2025-04-21T21:22:52.545Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:52.545Z [INFO]  server-5e7f5966-b880-0d84-d244-c725ee24a960: entering follower state: follower="Node at 5e7f5966-b880-0d84-d244-c725ee24a960 [Follower]" leader-address= leader-id=
2025-04-21T21:22:52.545Z [INFO]  server-f37280b4-dc53-1fb7-e6dc-243892a42d3b: entering follower state: follower="Node at f37280b4-dc53-1fb7-e6dc-243892a42d3b [Follower]" leader-address= leader-id=
2025-04-21T21:22:52.545Z [INFO]  server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380: entering follower state: follower="Node at ab5ede7f-a93a-bb8b-25e5-df2f68fe2380 [Follower]" leader-address= leader-id=
2025-04-21T21:22:52.601Z [WARN]  server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:52.601Z [INFO]  server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380: entering candidate state: node="Node at ab5ede7f-a93a-bb8b-25e5-df2f68fe2380 [Candidate]" term=2
2025-04-21T21:22:52.601Z [INFO]  server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380: election won: term=2 tally=2
2025-04-21T21:22:52.601Z [INFO]  server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380: entering leader state: leader="Node at ab5ede7f-a93a-bb8b-25e5-df2f68fe2380 [Leader]"
2025-04-21T21:22:52.601Z [INFO]  server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380: added peer, starting replication: peer=server-5e7f5966-b880-0d84-d244-c725ee24a960
2025-04-21T21:22:52.601Z [INFO]  server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380: added peer, starting replication: peer=server-f37280b4-dc53-1fb7-e6dc-243892a42d3b
2025-04-21T21:22:52.601Z [INFO]  server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380: pipelining replication: peer="{Voter server-f37280b4-dc53-1fb7-e6dc-243892a42d3b f37280b4-dc53-1fb7-e6dc-243892a42d3b}"
2025-04-21T21:22:52.601Z [INFO]  server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380: pipelining replication: peer="{Voter server-5e7f5966-b880-0d84-d244-c725ee24a960 5e7f5966-b880-0d84-d244-c725ee24a960}"
2025-04-21T21:22:52.709Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:52.601710065 +0000 UTC m=+11.123369388 (1 nodes), 55.980136ms from start of poll, 56.146497ms from cluster start. Timeout at 2025-04-21 21:22:52.706721385 +0000 UTC m=+11.228380708, 105.01132ms after stability
2025-04-21T21:22:52.717Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:22:52.829Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:22:52.717232562 +0000 UTC m=+11.238891885 (2 nodes), 1.4µs from start of poll, 171.668994ms from cluster start. Timeout at 2025-04-21 21:22:52.822246282 +0000 UTC m=+11.343905605, 105.01372ms after stability
2025-04-21T21:22:52.829Z [INFO]  server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380: updating configuration: command=RemoveServer server-id=server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380 server-addr= servers="[{Suffrage:Voter ID:server-5e7f5966-b880-0d84-d244-c725ee24a960 Address:5e7f5966-b880-0d84-d244-c725ee24a960} {Suffrage:Voter ID:server-f37280b4-dc53-1fb7-e6dc-243892a42d3b Address:f37280b4-dc53-1fb7-e6dc-243892a42d3b}]"
2025-04-21T21:22:52.829Z [INFO]  server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380: removed ourself, shutting down
2025-04-21T21:22:52.829Z [INFO]  server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380: aborting pipeline replication: peer="{Voter server-f37280b4-dc53-1fb7-e6dc-243892a42d3b f37280b4-dc53-1fb7-e6dc-243892a42d3b}"
2025-04-21T21:22:52.829Z [INFO]  server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380: aborting pipeline replication: peer="{Voter server-5e7f5966-b880-0d84-d244-c725ee24a960 5e7f5966-b880-0d84-d244-c725ee24a960}"
2025-04-21T21:22:52.937Z [WARN]  server-5e7f5966-b880-0d84-d244-c725ee24a960: heartbeat timeout reached, starting election: last-leader-addr=ab5ede7f-a93a-bb8b-25e5-df2f68fe2380 last-leader-id=server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380
2025-04-21T21:22:52.937Z [INFO]  server-5e7f5966-b880-0d84-d244-c725ee24a960: entering candidate state: node="Node at 5e7f5966-b880-0d84-d244-c725ee24a960 [Candidate]" term=3
2025-04-21T21:22:52.937Z [WARN]  server-f37280b4-dc53-1fb7-e6dc-243892a42d3b: rejecting vote request since we have a leader: from=5e7f5966-b880-0d84-d244-c725ee24a960 leader=ab5ede7f-a93a-bb8b-25e5-df2f68fe2380 leader-id=server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380
2025-04-21T21:22:52.948Z [WARN]  server-f37280b4-dc53-1fb7-e6dc-243892a42d3b: heartbeat timeout reached, starting election: last-leader-addr=ab5ede7f-a93a-bb8b-25e5-df2f68fe2380 last-leader-id=server-ab5ede7f-a93a-bb8b-25e5-df2f68fe2380
2025-04-21T21:22:52.948Z [INFO]  server-f37280b4-dc53-1fb7-e6dc-243892a42d3b: entering candidate state: node="Node at f37280b4-dc53-1fb7-e6dc-243892a42d3b [Candidate]" term=3
2025-04-21T21:22:52.948Z [INFO]  server-5e7f5966-b880-0d84-d244-c725ee24a960: duplicate requestVote for same term: term=3
2025-04-21T21:22:53.017Z [WARN]  server-f37280b4-dc53-1fb7-e6dc-243892a42d3b: Election timeout reached, restarting election
2025-04-21T21:22:53.017Z [INFO]  server-f37280b4-dc53-1fb7-e6dc-243892a42d3b: entering candidate state: node="Node at f37280b4-dc53-1fb7-e6dc-243892a42d3b [Candidate]" term=4
2025-04-21T21:22:53.017Z [INFO]  server-5e7f5966-b880-0d84-d244-c725ee24a960: entering follower state: follower="Node at 5e7f5966-b880-0d84-d244-c725ee24a960 [Follower]" leader-address= leader-id=
2025-04-21T21:22:53.017Z [INFO]  server-f37280b4-dc53-1fb7-e6dc-243892a42d3b: election won: term=4 tally=2
2025-04-21T21:22:53.017Z [INFO]  server-f37280b4-dc53-1fb7-e6dc-243892a42d3b: entering leader state: leader="Node at f37280b4-dc53-1fb7-e6dc-243892a42d3b [Leader]"
2025-04-21T21:22:53.017Z [INFO]  server-f37280b4-dc53-1fb7-e6dc-243892a42d3b: added peer, starting replication: peer=server-5e7f5966-b880-0d84-d244-c725ee24a960
2025-04-21T21:22:53.017Z [INFO]  server-f37280b4-dc53-1fb7-e6dc-243892a42d3b: pipelining replication: peer="{Voter server-5e7f5966-b880-0d84-d244-c725ee24a960 5e7f5966-b880-0d84-d244-c725ee24a960}"
2025-04-21T21:22:53.049Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:53.157Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:53.049137392 +0000 UTC m=+11.570796675 (1 nodes), 30.36µs from start of poll, 503.573784ms from cluster start. Timeout at 2025-04-21 21:22:53.154144472 +0000 UTC m=+11.675803795, 105.00712ms after stability
2025-04-21T21:22:53.157Z [INFO]  server-f37280b4-dc53-1fb7-e6dc-243892a42d3b: aborting pipeline replication: peer="{Voter server-5e7f5966-b880-0d84-d244-c725ee24a960 5e7f5966-b880-0d84-d244-c725ee24a960}"
--- PASS: TestRaft_RemoveLeader (0.61s)
=== RUN   TestRaft_RemoveLeader_NoShutdown
2025-04-21T21:22:53.157Z [INFO]  server-6048cf11-b187-40c3-2acc-cae3cb671dc9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6048cf11-b187-40c3-2acc-cae3cb671dc9 Address:6048cf11-b187-40c3-2acc-cae3cb671dc9} {Suffrage:Voter ID:server-bcb6f4a9-4080-0225-58eb-b1cff548c7f2 Address:bcb6f4a9-4080-0225-58eb-b1cff548c7f2} {Suffrage:Voter ID:server-fe356d85-c0db-94d9-6d91-f257b9fd8021 Address:fe356d85-c0db-94d9-6d91-f257b9fd8021}]"
2025-04-21T21:22:53.157Z [INFO]  server-bcb6f4a9-4080-0225-58eb-b1cff548c7f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6048cf11-b187-40c3-2acc-cae3cb671dc9 Address:6048cf11-b187-40c3-2acc-cae3cb671dc9} {Suffrage:Voter ID:server-bcb6f4a9-4080-0225-58eb-b1cff548c7f2 Address:bcb6f4a9-4080-0225-58eb-b1cff548c7f2} {Suffrage:Voter ID:server-fe356d85-c0db-94d9-6d91-f257b9fd8021 Address:fe356d85-c0db-94d9-6d91-f257b9fd8021}]"
2025-04-21T21:22:53.157Z [INFO]  server-fe356d85-c0db-94d9-6d91-f257b9fd8021: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6048cf11-b187-40c3-2acc-cae3cb671dc9 Address:6048cf11-b187-40c3-2acc-cae3cb671dc9} {Suffrage:Voter ID:server-bcb6f4a9-4080-0225-58eb-b1cff548c7f2 Address:bcb6f4a9-4080-0225-58eb-b1cff548c7f2} {Suffrage:Voter ID:server-fe356d85-c0db-94d9-6d91-f257b9fd8021 Address:fe356d85-c0db-94d9-6d91-f257b9fd8021}]"
2025-04-21T21:22:53.157Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:22:53.157Z [INFO]  server-6048cf11-b187-40c3-2acc-cae3cb671dc9: entering follower state: follower="Node at 6048cf11-b187-40c3-2acc-cae3cb671dc9 [Follower]" leader-address= leader-id=
2025-04-21T21:22:53.157Z [INFO]  server-bcb6f4a9-4080-0225-58eb-b1cff548c7f2: entering follower state: follower="Node at bcb6f4a9-4080-0225-58eb-b1cff548c7f2 [Follower]" leader-address= leader-id=
2025-04-21T21:22:53.157Z [INFO]  server-fe356d85-c0db-94d9-6d91-f257b9fd8021: entering follower state: follower="Node at fe356d85-c0db-94d9-6d91-f257b9fd8021 [Follower]" leader-address= leader-id=
2025-04-21T21:22:53.225Z [WARN]  server-fe356d85-c0db-94d9-6d91-f257b9fd8021: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:53.225Z [INFO]  server-fe356d85-c0db-94d9-6d91-f257b9fd8021: entering candidate state: node="Node at fe356d85-c0db-94d9-6d91-f257b9fd8021 [Candidate]" term=2
2025-04-21T21:22:53.225Z [INFO]  server-fe356d85-c0db-94d9-6d91-f257b9fd8021: election won: term=2 tally=2
2025-04-21T21:22:53.225Z [INFO]  server-fe356d85-c0db-94d9-6d91-f257b9fd8021: entering leader state: leader="Node at fe356d85-c0db-94d9-6d91-f257b9fd8021 [Leader]"
2025-04-21T21:22:53.225Z [INFO]  server-fe356d85-c0db-94d9-6d91-f257b9fd8021: added peer, starting replication: peer=server-6048cf11-b187-40c3-2acc-cae3cb671dc9
2025-04-21T21:22:53.225Z [INFO]  server-fe356d85-c0db-94d9-6d91-f257b9fd8021: added peer, starting replication: peer=server-bcb6f4a9-4080-0225-58eb-b1cff548c7f2
2025-04-21T21:22:53.229Z [INFO]  server-fe356d85-c0db-94d9-6d91-f257b9fd8021: pipelining replication: peer="{Voter server-6048cf11-b187-40c3-2acc-cae3cb671dc9 6048cf11-b187-40c3-2acc-cae3cb671dc9}"
2025-04-21T21:22:53.229Z [INFO]  server-fe356d85-c0db-94d9-6d91-f257b9fd8021: pipelining replication: peer="{Voter server-bcb6f4a9-4080-0225-58eb-b1cff548c7f2 bcb6f4a9-4080-0225-58eb-b1cff548c7f2}"
2025-04-21T21:22:53.338Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:22:53.229164962 +0000 UTC m=+11.750824285 (2 nodes), 71.41664ms from start of poll, 71.595961ms from cluster start. Timeout at 2025-04-21 21:22:53.334187282 +0000 UTC m=+11.855846565, 105.02228ms after stability
2025-04-21T21:22:53.338Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:53.445Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:53.338208895 +0000 UTC m=+11.859868218 (1 nodes), 2.08µs from start of poll, 180.639894ms from cluster start. Timeout at 2025-04-21 21:22:53.443214175 +0000 UTC m=+11.964873498, 105.00528ms after stability
2025-04-21T21:22:53.447Z [INFO]  server-fe356d85-c0db-94d9-6d91-f257b9fd8021: updating configuration: command=RemoveServer server-id=server-fe356d85-c0db-94d9-6d91-f257b9fd8021 server-addr= servers="[{Suffrage:Voter ID:server-6048cf11-b187-40c3-2acc-cae3cb671dc9 Address:6048cf11-b187-40c3-2acc-cae3cb671dc9} {Suffrage:Voter ID:server-bcb6f4a9-4080-0225-58eb-b1cff548c7f2 Address:bcb6f4a9-4080-0225-58eb-b1cff548c7f2}]"
2025-04-21T21:22:53.447Z [INFO]  server-fe356d85-c0db-94d9-6d91-f257b9fd8021: removed ourself, transitioning to follower
2025-04-21T21:22:53.447Z [INFO]  server-fe356d85-c0db-94d9-6d91-f257b9fd8021: entering follower state: follower="Node at fe356d85-c0db-94d9-6d91-f257b9fd8021 [Follower]" leader-address= leader-id=
2025-04-21T21:22:53.447Z [INFO]  server-fe356d85-c0db-94d9-6d91-f257b9fd8021: aborting pipeline replication: peer="{Voter server-bcb6f4a9-4080-0225-58eb-b1cff548c7f2 bcb6f4a9-4080-0225-58eb-b1cff548c7f2}"
2025-04-21T21:22:53.448Z [INFO]  server-fe356d85-c0db-94d9-6d91-f257b9fd8021: aborting pipeline replication: peer="{Voter server-6048cf11-b187-40c3-2acc-cae3cb671dc9 6048cf11-b187-40c3-2acc-cae3cb671dc9}"
2025-04-21T21:22:53.549Z [WARN]  server-fe356d85-c0db-94d9-6d91-f257b9fd8021: not part of stable configuration, aborting election
2025-04-21T21:22:53.557Z [WARN]  server-bcb6f4a9-4080-0225-58eb-b1cff548c7f2: heartbeat timeout reached, starting election: last-leader-addr=fe356d85-c0db-94d9-6d91-f257b9fd8021 last-leader-id=server-fe356d85-c0db-94d9-6d91-f257b9fd8021
2025-04-21T21:22:53.557Z [INFO]  server-bcb6f4a9-4080-0225-58eb-b1cff548c7f2: entering candidate state: node="Node at bcb6f4a9-4080-0225-58eb-b1cff548c7f2 [Candidate]" term=3
2025-04-21T21:22:53.557Z [WARN]  server-6048cf11-b187-40c3-2acc-cae3cb671dc9: rejecting vote request since we have a leader: from=bcb6f4a9-4080-0225-58eb-b1cff548c7f2 leader=fe356d85-c0db-94d9-6d91-f257b9fd8021 leader-id=server-fe356d85-c0db-94d9-6d91-f257b9fd8021
2025-04-21T21:22:53.557Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:53.573Z [WARN]  server-6048cf11-b187-40c3-2acc-cae3cb671dc9: heartbeat timeout reached, starting election: last-leader-addr=fe356d85-c0db-94d9-6d91-f257b9fd8021 last-leader-id=server-fe356d85-c0db-94d9-6d91-f257b9fd8021
2025-04-21T21:22:53.573Z [INFO]  server-6048cf11-b187-40c3-2acc-cae3cb671dc9: entering candidate state: node="Node at 6048cf11-b187-40c3-2acc-cae3cb671dc9 [Candidate]" term=3
2025-04-21T21:22:53.573Z [INFO]  server-bcb6f4a9-4080-0225-58eb-b1cff548c7f2: duplicate requestVote for same term: term=3
2025-04-21T21:22:53.633Z [WARN]  server-6048cf11-b187-40c3-2acc-cae3cb671dc9: Election timeout reached, restarting election
2025-04-21T21:22:53.633Z [INFO]  server-6048cf11-b187-40c3-2acc-cae3cb671dc9: entering candidate state: node="Node at 6048cf11-b187-40c3-2acc-cae3cb671dc9 [Candidate]" term=4
2025-04-21T21:22:53.633Z [INFO]  server-bcb6f4a9-4080-0225-58eb-b1cff548c7f2: entering follower state: follower="Node at bcb6f4a9-4080-0225-58eb-b1cff548c7f2 [Follower]" leader-address= leader-id=
2025-04-21T21:22:53.633Z [INFO]  server-6048cf11-b187-40c3-2acc-cae3cb671dc9: election won: term=4 tally=2
2025-04-21T21:22:53.633Z [INFO]  server-6048cf11-b187-40c3-2acc-cae3cb671dc9: entering leader state: leader="Node at 6048cf11-b187-40c3-2acc-cae3cb671dc9 [Leader]"
2025-04-21T21:22:53.633Z [INFO]  server-6048cf11-b187-40c3-2acc-cae3cb671dc9: added peer, starting replication: peer=server-bcb6f4a9-4080-0225-58eb-b1cff548c7f2
2025-04-21T21:22:53.633Z [INFO]  server-6048cf11-b187-40c3-2acc-cae3cb671dc9: pipelining replication: peer="{Voter server-bcb6f4a9-4080-0225-58eb-b1cff548c7f2 bcb6f4a9-4080-0225-58eb-b1cff548c7f2}"
2025-04-21T21:22:53.738Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:53.633318758 +0000 UTC m=+12.154978081 (1 nodes), 76.062231ms from start of poll, 475.749757ms from cluster start. Timeout at 2025-04-21 21:22:53.738319278 +0000 UTC m=+12.259978601, 105.00052ms after stability
2025-04-21T21:22:53.845Z [INFO]  server-6048cf11-b187-40c3-2acc-cae3cb671dc9: aborting pipeline replication: peer="{Voter server-bcb6f4a9-4080-0225-58eb-b1cff548c7f2 bcb6f4a9-4080-0225-58eb-b1cff548c7f2}"
--- PASS: TestRaft_RemoveLeader_NoShutdown (0.69s)
=== RUN   TestRaft_RemoveFollower_SplitCluster
2025-04-21T21:22:53.845Z [INFO]  server-915075bd-bf37-adf5-a835-697559e976ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-915075bd-bf37-adf5-a835-697559e976ea Address:915075bd-bf37-adf5-a835-697559e976ea} {Suffrage:Voter ID:server-a1f185b7-f75d-4a3b-36bd-06ea288ad104 Address:a1f185b7-f75d-4a3b-36bd-06ea288ad104} {Suffrage:Voter ID:server-04baf6f2-9ee6-1148-06ff-554a279894f3 Address:04baf6f2-9ee6-1148-06ff-554a279894f3} {Suffrage:Voter ID:server-b15dfdd9-9035-81a1-2118-5f2aefbe17f3 Address:b15dfdd9-9035-81a1-2118-5f2aefbe17f3}]"
2025-04-21T21:22:53.845Z [INFO]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-915075bd-bf37-adf5-a835-697559e976ea Address:915075bd-bf37-adf5-a835-697559e976ea} {Suffrage:Voter ID:server-a1f185b7-f75d-4a3b-36bd-06ea288ad104 Address:a1f185b7-f75d-4a3b-36bd-06ea288ad104} {Suffrage:Voter ID:server-04baf6f2-9ee6-1148-06ff-554a279894f3 Address:04baf6f2-9ee6-1148-06ff-554a279894f3} {Suffrage:Voter ID:server-b15dfdd9-9035-81a1-2118-5f2aefbe17f3 Address:b15dfdd9-9035-81a1-2118-5f2aefbe17f3}]"
2025-04-21T21:22:53.845Z [INFO]  server-04baf6f2-9ee6-1148-06ff-554a279894f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-915075bd-bf37-adf5-a835-697559e976ea Address:915075bd-bf37-adf5-a835-697559e976ea} {Suffrage:Voter ID:server-a1f185b7-f75d-4a3b-36bd-06ea288ad104 Address:a1f185b7-f75d-4a3b-36bd-06ea288ad104} {Suffrage:Voter ID:server-04baf6f2-9ee6-1148-06ff-554a279894f3 Address:04baf6f2-9ee6-1148-06ff-554a279894f3} {Suffrage:Voter ID:server-b15dfdd9-9035-81a1-2118-5f2aefbe17f3 Address:b15dfdd9-9035-81a1-2118-5f2aefbe17f3}]"
2025-04-21T21:22:53.845Z [INFO]  server-b15dfdd9-9035-81a1-2118-5f2aefbe17f3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-915075bd-bf37-adf5-a835-697559e976ea Address:915075bd-bf37-adf5-a835-697559e976ea} {Suffrage:Voter ID:server-a1f185b7-f75d-4a3b-36bd-06ea288ad104 Address:a1f185b7-f75d-4a3b-36bd-06ea288ad104} {Suffrage:Voter ID:server-04baf6f2-9ee6-1148-06ff-554a279894f3 Address:04baf6f2-9ee6-1148-06ff-554a279894f3} {Suffrage:Voter ID:server-b15dfdd9-9035-81a1-2118-5f2aefbe17f3 Address:b15dfdd9-9035-81a1-2118-5f2aefbe17f3}]"
2025-04-21T21:22:53.845Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:53.845Z [INFO]  server-915075bd-bf37-adf5-a835-697559e976ea: entering follower state: follower="Node at 915075bd-bf37-adf5-a835-697559e976ea [Follower]" leader-address= leader-id=
2025-04-21T21:22:53.845Z [INFO]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: entering follower state: follower="Node at a1f185b7-f75d-4a3b-36bd-06ea288ad104 [Follower]" leader-address= leader-id=
2025-04-21T21:22:53.845Z [INFO]  server-04baf6f2-9ee6-1148-06ff-554a279894f3: entering follower state: follower="Node at 04baf6f2-9ee6-1148-06ff-554a279894f3 [Follower]" leader-address= leader-id=
2025-04-21T21:22:53.845Z [INFO]  server-b15dfdd9-9035-81a1-2118-5f2aefbe17f3: entering follower state: follower="Node at b15dfdd9-9035-81a1-2118-5f2aefbe17f3 [Follower]" leader-address= leader-id=
2025-04-21T21:22:53.909Z [WARN]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:53.909Z [INFO]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: entering candidate state: node="Node at a1f185b7-f75d-4a3b-36bd-06ea288ad104 [Candidate]" term=2
2025-04-21T21:22:53.909Z [WARN]  server-04baf6f2-9ee6-1148-06ff-554a279894f3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:53.909Z [INFO]  server-04baf6f2-9ee6-1148-06ff-554a279894f3: entering candidate state: node="Node at 04baf6f2-9ee6-1148-06ff-554a279894f3 [Candidate]" term=2
2025-04-21T21:22:53.909Z [INFO]  server-b15dfdd9-9035-81a1-2118-5f2aefbe17f3: duplicate requestVote for same term: term=2
2025-04-21T21:22:53.909Z [INFO]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: election won: term=2 tally=3
2025-04-21T21:22:53.909Z [INFO]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: entering leader state: leader="Node at a1f185b7-f75d-4a3b-36bd-06ea288ad104 [Leader]"
2025-04-21T21:22:53.909Z [INFO]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: added peer, starting replication: peer=server-915075bd-bf37-adf5-a835-697559e976ea
2025-04-21T21:22:53.909Z [INFO]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: added peer, starting replication: peer=server-04baf6f2-9ee6-1148-06ff-554a279894f3
2025-04-21T21:22:53.909Z [INFO]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: added peer, starting replication: peer=server-b15dfdd9-9035-81a1-2118-5f2aefbe17f3
2025-04-21T21:22:53.909Z [INFO]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: pipelining replication: peer="{Voter server-b15dfdd9-9035-81a1-2118-5f2aefbe17f3 b15dfdd9-9035-81a1-2118-5f2aefbe17f3}"
2025-04-21T21:22:53.909Z [INFO]  server-04baf6f2-9ee6-1148-06ff-554a279894f3: duplicate requestVote for same term: term=2
2025-04-21T21:22:53.909Z [INFO]  server-915075bd-bf37-adf5-a835-697559e976ea: duplicate requestVote for same term: term=2
2025-04-21T21:22:53.909Z [WARN]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: rejecting vote request since we have a leader: from=04baf6f2-9ee6-1148-06ff-554a279894f3 leader=a1f185b7-f75d-4a3b-36bd-06ea288ad104 leader-id=server-a1f185b7-f75d-4a3b-36bd-06ea288ad104
2025-04-21T21:22:53.909Z [INFO]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: pipelining replication: peer="{Voter server-915075bd-bf37-adf5-a835-697559e976ea 915075bd-bf37-adf5-a835-697559e976ea}"
2025-04-21T21:22:53.909Z [INFO]  server-04baf6f2-9ee6-1148-06ff-554a279894f3: entering follower state: follower="Node at 04baf6f2-9ee6-1148-06ff-554a279894f3 [Follower]" leader-address=a1f185b7-f75d-4a3b-36bd-06ea288ad104 leader-id=server-a1f185b7-f75d-4a3b-36bd-06ea288ad104
2025-04-21T21:22:53.909Z [INFO]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: pipelining replication: peer="{Voter server-04baf6f2-9ee6-1148-06ff-554a279894f3 04baf6f2-9ee6-1148-06ff-554a279894f3}"
2025-04-21T21:22:54.017Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:53.909587454 +0000 UTC m=+12.431246737 (1 nodes), 63.770188ms from start of poll, 64.01171ms from cluster start. Timeout at 2025-04-21 21:22:54.014589534 +0000 UTC m=+12.536248857, 105.00212ms after stability
2025-04-21T21:22:54.129Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:22:54.238Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:22:54.12913501 +0000 UTC m=+12.650794333 (3 nodes), 4.16µs from start of poll, 283.559306ms from cluster start. Timeout at 2025-04-21 21:22:54.23414533 +0000 UTC m=+12.755804693, 105.01036ms after stability
2025-04-21T21:22:54.238Z [INFO]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: updating configuration: command=RemoveServer server-id=server-b15dfdd9-9035-81a1-2118-5f2aefbe17f3 server-addr= servers="[{Suffrage:Voter ID:server-915075bd-bf37-adf5-a835-697559e976ea Address:915075bd-bf37-adf5-a835-697559e976ea} {Suffrage:Voter ID:server-a1f185b7-f75d-4a3b-36bd-06ea288ad104 Address:a1f185b7-f75d-4a3b-36bd-06ea288ad104} {Suffrage:Voter ID:server-04baf6f2-9ee6-1148-06ff-554a279894f3 Address:04baf6f2-9ee6-1148-06ff-554a279894f3}]"
2025-04-21T21:22:54.238Z [INFO]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: removed peer, stopping replication: peer=server-b15dfdd9-9035-81a1-2118-5f2aefbe17f3 last-index=3
2025-04-21T21:22:54.238Z [ERROR] server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to pipeline appendEntries: peer="{Voter server-04baf6f2-9ee6-1148-06ff-554a279894f3 04baf6f2-9ee6-1148-06ff-554a279894f3}" error="append pipeline closed"
2025-04-21T21:22:54.238Z [INFO]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: aborting pipeline replication: peer="{Voter server-04baf6f2-9ee6-1148-06ff-554a279894f3 04baf6f2-9ee6-1148-06ff-554a279894f3}"
2025-04-21T21:22:54.238Z [INFO]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: aborting pipeline replication: peer="{Voter server-b15dfdd9-9035-81a1-2118-5f2aefbe17f3 b15dfdd9-9035-81a1-2118-5f2aefbe17f3}"
2025-04-21T21:22:54.238Z [ERROR] server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to pipeline appendEntries: peer="{Voter server-915075bd-bf37-adf5-a835-697559e976ea 915075bd-bf37-adf5-a835-697559e976ea}" error="append pipeline closed"
2025-04-21T21:22:54.238Z [INFO]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: aborting pipeline replication: peer="{Voter server-915075bd-bf37-adf5-a835-697559e976ea 915075bd-bf37-adf5-a835-697559e976ea}"
2025-04-21T21:22:54.238Z [ERROR] server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to appendEntries to: peer="{Voter server-915075bd-bf37-adf5-a835-697559e976ea 915075bd-bf37-adf5-a835-697559e976ea}" error="failed to connect to peer: 915075bd-bf37-adf5-a835-697559e976ea"
2025-04-21T21:22:54.245Z [ERROR] server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to heartbeat to: peer=915075bd-bf37-adf5-a835-697559e976ea backoff time=10ms error="failed to connect to peer: 915075bd-bf37-adf5-a835-697559e976ea"
2025-04-21T21:22:54.249Z [ERROR] server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to appendEntries to: peer="{Voter server-04baf6f2-9ee6-1148-06ff-554a279894f3 04baf6f2-9ee6-1148-06ff-554a279894f3}" error="failed to connect to peer: 04baf6f2-9ee6-1148-06ff-554a279894f3"
2025-04-21T21:22:54.249Z [ERROR] server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to heartbeat to: peer=04baf6f2-9ee6-1148-06ff-554a279894f3 backoff time=10ms error="failed to connect to peer: 04baf6f2-9ee6-1148-06ff-554a279894f3"
2025-04-21T21:22:54.261Z [ERROR] server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to appendEntries to: peer="{Voter server-915075bd-bf37-adf5-a835-697559e976ea 915075bd-bf37-adf5-a835-697559e976ea}" error="failed to connect to peer: 915075bd-bf37-adf5-a835-697559e976ea"
2025-04-21T21:22:54.265Z [ERROR] server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to heartbeat to: peer=915075bd-bf37-adf5-a835-697559e976ea backoff time=10ms error="failed to connect to peer: 915075bd-bf37-adf5-a835-697559e976ea"
2025-04-21T21:22:54.285Z [WARN]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to contact: server-id=server-915075bd-bf37-adf5-a835-697559e976ea time=52.247671ms
2025-04-21T21:22:54.285Z [ERROR] server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to heartbeat to: peer=04baf6f2-9ee6-1148-06ff-554a279894f3 backoff time=10ms error="failed to connect to peer: 04baf6f2-9ee6-1148-06ff-554a279894f3"
2025-04-21T21:22:54.285Z [ERROR] server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to appendEntries to: peer="{Voter server-04baf6f2-9ee6-1148-06ff-554a279894f3 04baf6f2-9ee6-1148-06ff-554a279894f3}" error="failed to connect to peer: 04baf6f2-9ee6-1148-06ff-554a279894f3"
2025-04-21T21:22:54.285Z [ERROR] server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to appendEntries to: peer="{Voter server-915075bd-bf37-adf5-a835-697559e976ea 915075bd-bf37-adf5-a835-697559e976ea}" error="failed to connect to peer: 915075bd-bf37-adf5-a835-697559e976ea"
2025-04-21T21:22:54.297Z [ERROR] server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to heartbeat to: peer=915075bd-bf37-adf5-a835-697559e976ea backoff time=10ms error="failed to connect to peer: 915075bd-bf37-adf5-a835-697559e976ea"
2025-04-21T21:22:54.297Z [WARN]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to contact: server-id=server-915075bd-bf37-adf5-a835-697559e976ea time=64.043591ms
2025-04-21T21:22:54.297Z [WARN]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to contact: server-id=server-04baf6f2-9ee6-1148-06ff-554a279894f3 time=59.064517ms
2025-04-21T21:22:54.297Z [WARN]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to contact quorum of nodes, stepping down
2025-04-21T21:22:54.297Z [INFO]  server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: entering follower state: follower="Node at a1f185b7-f75d-4a3b-36bd-06ea288ad104 [Follower]" leader-address= leader-id=
2025-04-21T21:22:54.297Z [ERROR] server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to appendEntries to: peer="{Voter server-04baf6f2-9ee6-1148-06ff-554a279894f3 04baf6f2-9ee6-1148-06ff-554a279894f3}" error="failed to connect to peer: 04baf6f2-9ee6-1148-06ff-554a279894f3"
2025-04-21T21:22:54.297Z [ERROR] server-a1f185b7-f75d-4a3b-36bd-06ea288ad104: failed to appendEntries to: peer="{Voter server-915075bd-bf37-adf5-a835-697559e976ea 915075bd-bf37-adf5-a835-697559e976ea}" error="failed to connect to peer: 915075bd-bf37-adf5-a835-697559e976ea"
--- PASS: TestRaft_RemoveFollower_SplitCluster (0.45s)
=== RUN   TestRaft_AddKnownPeer
2025-04-21T21:22:54.297Z [INFO]  server-93990964-e698-eeff-fc73-54f044821750: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93990964-e698-eeff-fc73-54f044821750 Address:93990964-e698-eeff-fc73-54f044821750} {Suffrage:Voter ID:server-074bb69c-3208-9941-5aa1-be30d7da7add Address:074bb69c-3208-9941-5aa1-be30d7da7add} {Suffrage:Voter ID:server-d3bfdada-7db1-c083-67a6-319db545b3ea Address:d3bfdada-7db1-c083-67a6-319db545b3ea}]"
2025-04-21T21:22:54.297Z [INFO]  server-074bb69c-3208-9941-5aa1-be30d7da7add: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93990964-e698-eeff-fc73-54f044821750 Address:93990964-e698-eeff-fc73-54f044821750} {Suffrage:Voter ID:server-074bb69c-3208-9941-5aa1-be30d7da7add Address:074bb69c-3208-9941-5aa1-be30d7da7add} {Suffrage:Voter ID:server-d3bfdada-7db1-c083-67a6-319db545b3ea Address:d3bfdada-7db1-c083-67a6-319db545b3ea}]"
2025-04-21T21:22:54.297Z [INFO]  server-d3bfdada-7db1-c083-67a6-319db545b3ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93990964-e698-eeff-fc73-54f044821750 Address:93990964-e698-eeff-fc73-54f044821750} {Suffrage:Voter ID:server-074bb69c-3208-9941-5aa1-be30d7da7add Address:074bb69c-3208-9941-5aa1-be30d7da7add} {Suffrage:Voter ID:server-d3bfdada-7db1-c083-67a6-319db545b3ea Address:d3bfdada-7db1-c083-67a6-319db545b3ea}]"
2025-04-21T21:22:54.297Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:54.297Z [INFO]  server-93990964-e698-eeff-fc73-54f044821750: entering follower state: follower="Node at 93990964-e698-eeff-fc73-54f044821750 [Follower]" leader-address= leader-id=
2025-04-21T21:22:54.297Z [INFO]  server-074bb69c-3208-9941-5aa1-be30d7da7add: entering follower state: follower="Node at 074bb69c-3208-9941-5aa1-be30d7da7add [Follower]" leader-address= leader-id=
2025-04-21T21:22:54.297Z [INFO]  server-d3bfdada-7db1-c083-67a6-319db545b3ea: entering follower state: follower="Node at d3bfdada-7db1-c083-67a6-319db545b3ea [Follower]" leader-address= leader-id=
2025-04-21T21:22:54.361Z [WARN]  server-93990964-e698-eeff-fc73-54f044821750: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:54.361Z [INFO]  server-93990964-e698-eeff-fc73-54f044821750: entering candidate state: node="Node at 93990964-e698-eeff-fc73-54f044821750 [Candidate]" term=2
2025-04-21T21:22:54.361Z [INFO]  server-93990964-e698-eeff-fc73-54f044821750: election won: term=2 tally=2
2025-04-21T21:22:54.361Z [INFO]  server-93990964-e698-eeff-fc73-54f044821750: entering leader state: leader="Node at 93990964-e698-eeff-fc73-54f044821750 [Leader]"
2025-04-21T21:22:54.361Z [INFO]  server-93990964-e698-eeff-fc73-54f044821750: added peer, starting replication: peer=server-074bb69c-3208-9941-5aa1-be30d7da7add
2025-04-21T21:22:54.361Z [INFO]  server-93990964-e698-eeff-fc73-54f044821750: added peer, starting replication: peer=server-d3bfdada-7db1-c083-67a6-319db545b3ea
2025-04-21T21:22:54.361Z [INFO]  server-93990964-e698-eeff-fc73-54f044821750: pipelining replication: peer="{Voter server-d3bfdada-7db1-c083-67a6-319db545b3ea d3bfdada-7db1-c083-67a6-319db545b3ea}"
2025-04-21T21:22:54.361Z [INFO]  server-93990964-e698-eeff-fc73-54f044821750: pipelining replication: peer="{Voter server-074bb69c-3208-9941-5aa1-be30d7da7add 074bb69c-3208-9941-5aa1-be30d7da7add}"
2025-04-21T21:22:54.469Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:54.361436931 +0000 UTC m=+12.883096214 (1 nodes), 63.638908ms from start of poll, 63.818829ms from cluster start. Timeout at 2025-04-21 21:22:54.466442251 +0000 UTC m=+12.988101574, 105.00536ms after stability
2025-04-21T21:22:54.469Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:22:54.578Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:22:54.469145174 +0000 UTC m=+12.990804497 (2 nodes), 2.6µs from start of poll, 171.527112ms from cluster start. Timeout at 2025-04-21 21:22:54.574149854 +0000 UTC m=+13.095809217, 105.00472ms after stability
2025-04-21T21:22:54.578Z [INFO]  server-93990964-e698-eeff-fc73-54f044821750: updating configuration: command=AddVoter server-id=server-074bb69c-3208-9941-5aa1-be30d7da7add server-addr=074bb69c-3208-9941-5aa1-be30d7da7add servers="[{Suffrage:Voter ID:server-93990964-e698-eeff-fc73-54f044821750 Address:93990964-e698-eeff-fc73-54f044821750} {Suffrage:Voter ID:server-074bb69c-3208-9941-5aa1-be30d7da7add Address:074bb69c-3208-9941-5aa1-be30d7da7add} {Suffrage:Voter ID:server-d3bfdada-7db1-c083-67a6-319db545b3ea Address:d3bfdada-7db1-c083-67a6-319db545b3ea}]"
2025-04-21T21:22:54.578Z [INFO]  server-93990964-e698-eeff-fc73-54f044821750: aborting pipeline replication: peer="{Voter server-d3bfdada-7db1-c083-67a6-319db545b3ea d3bfdada-7db1-c083-67a6-319db545b3ea}"
2025-04-21T21:22:54.578Z [INFO]  server-93990964-e698-eeff-fc73-54f044821750: aborting pipeline replication: peer="{Voter server-074bb69c-3208-9941-5aa1-be30d7da7add 074bb69c-3208-9941-5aa1-be30d7da7add}"
--- PASS: TestRaft_AddKnownPeer (0.28s)
=== RUN   TestRaft_RemoveUnknownPeer
2025-04-21T21:22:54.579Z [INFO]  server-acce2ed7-8856-a82c-904f-bbf1543f3533: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acce2ed7-8856-a82c-904f-bbf1543f3533 Address:acce2ed7-8856-a82c-904f-bbf1543f3533} {Suffrage:Voter ID:server-52926500-5904-2f56-80fe-b777aed8b4ef Address:52926500-5904-2f56-80fe-b777aed8b4ef} {Suffrage:Voter ID:server-f82b1f21-9427-67f1-b043-f411428f97c9 Address:f82b1f21-9427-67f1-b043-f411428f97c9}]"
2025-04-21T21:22:54.579Z [INFO]  server-52926500-5904-2f56-80fe-b777aed8b4ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acce2ed7-8856-a82c-904f-bbf1543f3533 Address:acce2ed7-8856-a82c-904f-bbf1543f3533} {Suffrage:Voter ID:server-52926500-5904-2f56-80fe-b777aed8b4ef Address:52926500-5904-2f56-80fe-b777aed8b4ef} {Suffrage:Voter ID:server-f82b1f21-9427-67f1-b043-f411428f97c9 Address:f82b1f21-9427-67f1-b043-f411428f97c9}]"
2025-04-21T21:22:54.579Z [INFO]  server-f82b1f21-9427-67f1-b043-f411428f97c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-acce2ed7-8856-a82c-904f-bbf1543f3533 Address:acce2ed7-8856-a82c-904f-bbf1543f3533} {Suffrage:Voter ID:server-52926500-5904-2f56-80fe-b777aed8b4ef Address:52926500-5904-2f56-80fe-b777aed8b4ef} {Suffrage:Voter ID:server-f82b1f21-9427-67f1-b043-f411428f97c9 Address:f82b1f21-9427-67f1-b043-f411428f97c9}]"
2025-04-21T21:22:54.579Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:54.579Z [INFO]  server-acce2ed7-8856-a82c-904f-bbf1543f3533: entering follower state: follower="Node at acce2ed7-8856-a82c-904f-bbf1543f3533 [Follower]" leader-address= leader-id=
2025-04-21T21:22:54.579Z [INFO]  server-52926500-5904-2f56-80fe-b777aed8b4ef: entering follower state: follower="Node at 52926500-5904-2f56-80fe-b777aed8b4ef [Follower]" leader-address= leader-id=
2025-04-21T21:22:54.579Z [INFO]  server-f82b1f21-9427-67f1-b043-f411428f97c9: entering follower state: follower="Node at f82b1f21-9427-67f1-b043-f411428f97c9 [Follower]" leader-address= leader-id=
2025-04-21T21:22:54.641Z [WARN]  server-f82b1f21-9427-67f1-b043-f411428f97c9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:54.641Z [INFO]  server-f82b1f21-9427-67f1-b043-f411428f97c9: entering candidate state: node="Node at f82b1f21-9427-67f1-b043-f411428f97c9 [Candidate]" term=2
2025-04-21T21:22:54.641Z [INFO]  server-f82b1f21-9427-67f1-b043-f411428f97c9: election won: term=2 tally=2
2025-04-21T21:22:54.641Z [INFO]  server-f82b1f21-9427-67f1-b043-f411428f97c9: entering leader state: leader="Node at f82b1f21-9427-67f1-b043-f411428f97c9 [Leader]"
2025-04-21T21:22:54.641Z [INFO]  server-f82b1f21-9427-67f1-b043-f411428f97c9: added peer, starting replication: peer=server-acce2ed7-8856-a82c-904f-bbf1543f3533
2025-04-21T21:22:54.641Z [INFO]  server-f82b1f21-9427-67f1-b043-f411428f97c9: added peer, starting replication: peer=server-52926500-5904-2f56-80fe-b777aed8b4ef
2025-04-21T21:22:54.641Z [INFO]  server-f82b1f21-9427-67f1-b043-f411428f97c9: pipelining replication: peer="{Voter server-52926500-5904-2f56-80fe-b777aed8b4ef 52926500-5904-2f56-80fe-b777aed8b4ef}"
2025-04-21T21:22:54.641Z [INFO]  server-f82b1f21-9427-67f1-b043-f411428f97c9: pipelining replication: peer="{Voter server-acce2ed7-8856-a82c-904f-bbf1543f3533 acce2ed7-8856-a82c-904f-bbf1543f3533}"
2025-04-21T21:22:54.749Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:54.641401492 +0000 UTC m=+13.163060815 (1 nodes), 61.987617ms from start of poll, 62.139418ms from cluster start. Timeout at 2025-04-21 21:22:54.746405492 +0000 UTC m=+13.268064815, 105.004ms after stability
2025-04-21T21:22:54.749Z [INFO]  server-f82b1f21-9427-67f1-b043-f411428f97c9: updating configuration: command=RemoveServer server-id=a62ddea9-b538-c795-efe2-b338e19ea4be server-addr= servers="[{Suffrage:Voter ID:server-acce2ed7-8856-a82c-904f-bbf1543f3533 Address:acce2ed7-8856-a82c-904f-bbf1543f3533} {Suffrage:Voter ID:server-52926500-5904-2f56-80fe-b777aed8b4ef Address:52926500-5904-2f56-80fe-b777aed8b4ef} {Suffrage:Voter ID:server-f82b1f21-9427-67f1-b043-f411428f97c9 Address:f82b1f21-9427-67f1-b043-f411428f97c9}]"
2025-04-21T21:22:54.749Z [INFO]  server-f82b1f21-9427-67f1-b043-f411428f97c9: aborting pipeline replication: peer="{Voter server-52926500-5904-2f56-80fe-b777aed8b4ef 52926500-5904-2f56-80fe-b777aed8b4ef}"
2025-04-21T21:22:54.749Z [INFO]  server-f82b1f21-9427-67f1-b043-f411428f97c9: aborting pipeline replication: peer="{Voter server-acce2ed7-8856-a82c-904f-bbf1543f3533 acce2ed7-8856-a82c-904f-bbf1543f3533}"
--- PASS: TestRaft_RemoveUnknownPeer (0.17s)
=== RUN   TestRaft_SnapshotRestore
2025-04-21T21:22:54.749Z [INFO]  server-92ac8b30-20c5-2d90-baeb-2861796b04b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92ac8b30-20c5-2d90-baeb-2861796b04b7 Address:92ac8b30-20c5-2d90-baeb-2861796b04b7}]"
2025-04-21T21:22:54.749Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:54.749Z [INFO]  server-92ac8b30-20c5-2d90-baeb-2861796b04b7: entering follower state: follower="Node at 92ac8b30-20c5-2d90-baeb-2861796b04b7 [Follower]" leader-address= leader-id=
2025-04-21T21:22:54.833Z [WARN]  server-92ac8b30-20c5-2d90-baeb-2861796b04b7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:54.833Z [INFO]  server-92ac8b30-20c5-2d90-baeb-2861796b04b7: entering candidate state: node="Node at 92ac8b30-20c5-2d90-baeb-2861796b04b7 [Candidate]" term=2
2025-04-21T21:22:54.833Z [INFO]  server-92ac8b30-20c5-2d90-baeb-2861796b04b7: election won: term=2 tally=1
2025-04-21T21:22:54.833Z [INFO]  server-92ac8b30-20c5-2d90-baeb-2861796b04b7: entering leader state: leader="Node at 92ac8b30-20c5-2d90-baeb-2861796b04b7 [Leader]"
2025-04-21T21:22:54.941Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:54.833200901 +0000 UTC m=+13.354860184 (1 nodes), 83.458121ms from start of poll, 83.528641ms from cluster start. Timeout at 2025-04-21 21:22:54.938204141 +0000 UTC m=+13.459863464, 105.00328ms after stability
2025-04-21T21:22:54.949Z [INFO]  server-92ac8b30-20c5-2d90-baeb-2861796b04b7: starting snapshot up to: index=102
2025-04-21T21:22:54.949Z [INFO]  creating new snapshot: path=/tmp/raft396283342/snapshots/2-102-1745270574949.tmp
2025-04-21T21:22:54.949Z [INFO]  server-92ac8b30-20c5-2d90-baeb-2861796b04b7: compacting logs: from=1 to=92
2025-04-21T21:22:54.949Z [INFO]  server-92ac8b30-20c5-2d90-baeb-2861796b04b7: snapshot complete up to: index=102
2025-04-21T21:22:54.949Z [INFO]  server-92ac8b30-20c5-2d90-baeb-2861796b04b7: starting restore from snapshot: id=2-102-1745270574949 last-index=102 last-term=2 size-in-bytes=693
2025-04-21T21:22:54.949Z [INFO]  server-92ac8b30-20c5-2d90-baeb-2861796b04b7: snapshot restore progress: id=2-102-1745270574949 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-04-21T21:22:54.949Z [INFO]  server-92ac8b30-20c5-2d90-baeb-2861796b04b7: restored from snapshot: id=2-102-1745270574949 last-index=102 last-term=2 size-in-bytes=693
2025-04-21T21:22:54.949Z [INFO]  server-92ac8b30-20c5-2d90-baeb-2861796b04b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92ac8b30-20c5-2d90-baeb-2861796b04b7 Address:92ac8b30-20c5-2d90-baeb-2861796b04b7}]"
--- PASS: TestRaft_SnapshotRestore (0.20s)
=== RUN   TestRaft_RestoreSnapshotOnStartup_Monotonic
2025-04-21T21:22:54.950Z [INFO]  server-d7dd33bd-9509-f810-7eec-87e44ff39c82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d7dd33bd-9509-f810-7eec-87e44ff39c82 Address:d7dd33bd-9509-f810-7eec-87e44ff39c82}]"
2025-04-21T21:22:54.950Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:54.950Z [INFO]  server-d7dd33bd-9509-f810-7eec-87e44ff39c82: entering follower state: follower="Node at d7dd33bd-9509-f810-7eec-87e44ff39c82 [Follower]" leader-address= leader-id=
2025-04-21T21:22:55.025Z [WARN]  server-d7dd33bd-9509-f810-7eec-87e44ff39c82: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:55.025Z [INFO]  server-d7dd33bd-9509-f810-7eec-87e44ff39c82: entering candidate state: node="Node at d7dd33bd-9509-f810-7eec-87e44ff39c82 [Candidate]" term=2
2025-04-21T21:22:55.025Z [INFO]  server-d7dd33bd-9509-f810-7eec-87e44ff39c82: election won: term=2 tally=1
2025-04-21T21:22:55.025Z [INFO]  server-d7dd33bd-9509-f810-7eec-87e44ff39c82: entering leader state: leader="Node at d7dd33bd-9509-f810-7eec-87e44ff39c82 [Leader]"
2025-04-21T21:22:55.133Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:55.025238471 +0000 UTC m=+13.546897794 (1 nodes), 75.008864ms from start of poll, 75.102344ms from cluster start. Timeout at 2025-04-21 21:22:55.130245111 +0000 UTC m=+13.651904434, 105.00664ms after stability
2025-04-21T21:22:55.134Z [INFO]  server-d7dd33bd-9509-f810-7eec-87e44ff39c82: starting snapshot up to: index=102
2025-04-21T21:22:55.134Z [INFO]  creating new snapshot: path=/tmp/raft1903111731/snapshots/2-102-1745270575134.tmp
2025-04-21T21:22:55.134Z [INFO]  server-d7dd33bd-9509-f810-7eec-87e44ff39c82: compacting logs: from=1 to=92
2025-04-21T21:22:55.134Z [INFO]  server-d7dd33bd-9509-f810-7eec-87e44ff39c82: snapshot complete up to: index=102
2025-04-21T21:22:55.134Z [INFO]  server-d7dd33bd-9509-f810-7eec-87e44ff39c82: starting restore from snapshot: id=2-102-1745270575134 last-index=102 last-term=2 size-in-bytes=693
2025-04-21T21:22:55.134Z [INFO]  server-d7dd33bd-9509-f810-7eec-87e44ff39c82: snapshot restore progress: id=2-102-1745270575134 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-04-21T21:22:55.134Z [INFO]  server-d7dd33bd-9509-f810-7eec-87e44ff39c82: restored from snapshot: id=2-102-1745270575134 last-index=102 last-term=2 size-in-bytes=693
2025-04-21T21:22:55.134Z [INFO]  server-d7dd33bd-9509-f810-7eec-87e44ff39c82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d7dd33bd-9509-f810-7eec-87e44ff39c82 Address:d7dd33bd-9509-f810-7eec-87e44ff39c82}]"
--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.18s)
=== RUN   TestRaft_SnapshotRestore_Progress
2025-04-21T21:22:55.135Z [INFO]  server-c6753f9c-864a-7df5-a7b2-e861209f1410: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c6753f9c-864a-7df5-a7b2-e861209f1410 Address:c6753f9c-864a-7df5-a7b2-e861209f1410}]"
2025-04-21T21:22:55.135Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:55.135Z [INFO]  server-c6753f9c-864a-7df5-a7b2-e861209f1410: entering follower state: follower="Node at c6753f9c-864a-7df5-a7b2-e861209f1410 [Follower]" leader-address= leader-id=
2025-04-21T21:22:55.217Z [WARN]  server-c6753f9c-864a-7df5-a7b2-e861209f1410: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:55.217Z [INFO]  server-c6753f9c-864a-7df5-a7b2-e861209f1410: entering candidate state: node="Node at c6753f9c-864a-7df5-a7b2-e861209f1410 [Candidate]" term=2
2025-04-21T21:22:55.217Z [INFO]  server-c6753f9c-864a-7df5-a7b2-e861209f1410: election won: term=2 tally=1
2025-04-21T21:22:55.217Z [INFO]  server-c6753f9c-864a-7df5-a7b2-e861209f1410: entering leader state: leader="Node at c6753f9c-864a-7df5-a7b2-e861209f1410 [Leader]"
2025-04-21T21:22:55.325Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:55.217219761 +0000 UTC m=+13.738879044 (1 nodes), 82.143672ms from start of poll, 82.214592ms from cluster start. Timeout at 2025-04-21 21:22:55.322222801 +0000 UTC m=+13.843882164, 105.00312ms after stability
2025-04-21T21:22:55.326Z [INFO]  server-c6753f9c-864a-7df5-a7b2-e861209f1410: starting snapshot up to: index=102
2025-04-21T21:22:55.326Z [INFO]  creating new snapshot: path=/tmp/raft809611107/snapshots/2-102-1745270575326.tmp
2025-04-21T21:22:55.326Z [INFO]  server-c6753f9c-864a-7df5-a7b2-e861209f1410: compacting logs: from=1 to=92
2025-04-21T21:22:55.326Z [INFO]  server-c6753f9c-864a-7df5-a7b2-e861209f1410: snapshot complete up to: index=102
--- PASS: TestRaft_SnapshotRestore_Progress (0.19s)
=== RUN   TestRaft_NoRestoreOnStart
2025-04-21T21:22:55.326Z [INFO]  server-6893eac9-d76c-5443-16cd-0e113300804a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6893eac9-d76c-5443-16cd-0e113300804a Address:6893eac9-d76c-5443-16cd-0e113300804a}]"
2025-04-21T21:22:55.326Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:55.327Z [INFO]  server-6893eac9-d76c-5443-16cd-0e113300804a: entering follower state: follower="Node at 6893eac9-d76c-5443-16cd-0e113300804a [Follower]" leader-address= leader-id=
2025-04-21T21:22:55.413Z [WARN]  server-6893eac9-d76c-5443-16cd-0e113300804a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:55.413Z [INFO]  server-6893eac9-d76c-5443-16cd-0e113300804a: entering candidate state: node="Node at 6893eac9-d76c-5443-16cd-0e113300804a [Candidate]" term=2
2025-04-21T21:22:55.413Z [INFO]  server-6893eac9-d76c-5443-16cd-0e113300804a: election won: term=2 tally=1
2025-04-21T21:22:55.413Z [INFO]  server-6893eac9-d76c-5443-16cd-0e113300804a: entering leader state: leader="Node at 6893eac9-d76c-5443-16cd-0e113300804a [Leader]"
2025-04-21T21:22:55.521Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:55.413215999 +0000 UTC m=+13.934875322 (1 nodes), 86.22414ms from start of poll, 86.29198ms from cluster start. Timeout at 2025-04-21 21:22:55.518219799 +0000 UTC m=+14.039879122, 105.0038ms after stability
2025-04-21T21:22:55.522Z [INFO]  server-6893eac9-d76c-5443-16cd-0e113300804a: starting snapshot up to: index=102
2025-04-21T21:22:55.522Z [INFO]  creating new snapshot: path=/tmp/raft2597062496/snapshots/2-102-1745270575522.tmp
2025-04-21T21:22:55.522Z [INFO]  server-6893eac9-d76c-5443-16cd-0e113300804a: compacting logs: from=1 to=92
2025-04-21T21:22:55.522Z [INFO]  server-6893eac9-d76c-5443-16cd-0e113300804a: snapshot complete up to: index=102
2025-04-21T21:22:55.522Z [INFO]  server-6893eac9-d76c-5443-16cd-0e113300804a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6893eac9-d76c-5443-16cd-0e113300804a Address:6893eac9-d76c-5443-16cd-0e113300804a}]"
--- PASS: TestRaft_NoRestoreOnStart (0.20s)
=== RUN   TestRaft_SnapshotRestore_PeerChange
    raft_test.go:1266: Skipping unreliable test
--- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
=== RUN   TestRaft_AutoSnapshot
2025-04-21T21:22:55.522Z [INFO]  server-8d5befa5-ddcb-9999-eef1-c88e24c3fd45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d5befa5-ddcb-9999-eef1-c88e24c3fd45 Address:8d5befa5-ddcb-9999-eef1-c88e24c3fd45}]"
2025-04-21T21:22:55.522Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:55.522Z [INFO]  server-6893eac9-d76c-5443-16cd-0e113300804a: entering follower state: follower="Node at 6893eac9-d76c-5443-16cd-0e113300804a [Follower]" leader-address= leader-id=
2025-04-21T21:22:55.522Z [INFO]  server-8d5befa5-ddcb-9999-eef1-c88e24c3fd45: entering follower state: follower="Node at 8d5befa5-ddcb-9999-eef1-c88e24c3fd45 [Follower]" leader-address= leader-id=
2025-04-21T21:22:55.621Z [WARN]  server-6893eac9-d76c-5443-16cd-0e113300804a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:55.621Z [INFO]  server-6893eac9-d76c-5443-16cd-0e113300804a: entering candidate state: node="Node at 6893eac9-d76c-5443-16cd-0e113300804a [Candidate]" term=3
2025-04-21T21:22:55.621Z [INFO]  server-6893eac9-d76c-5443-16cd-0e113300804a: election won: term=3 tally=1
2025-04-21T21:22:55.621Z [INFO]  server-6893eac9-d76c-5443-16cd-0e113300804a: entering leader state: leader="Node at 6893eac9-d76c-5443-16cd-0e113300804a [Leader]"
2025-04-21T21:22:55.621Z [WARN]  server-8d5befa5-ddcb-9999-eef1-c88e24c3fd45: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:55.621Z [INFO]  server-8d5befa5-ddcb-9999-eef1-c88e24c3fd45: entering candidate state: node="Node at 8d5befa5-ddcb-9999-eef1-c88e24c3fd45 [Candidate]" term=2
2025-04-21T21:22:55.621Z [INFO]  server-8d5befa5-ddcb-9999-eef1-c88e24c3fd45: election won: term=2 tally=1
2025-04-21T21:22:55.621Z [INFO]  server-8d5befa5-ddcb-9999-eef1-c88e24c3fd45: entering leader state: leader="Node at 8d5befa5-ddcb-9999-eef1-c88e24c3fd45 [Leader]"
2025-04-21T21:22:55.729Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:55.621305117 +0000 UTC m=+14.142964440 (1 nodes), 98.511742ms from start of poll, 98.583743ms from cluster start. Timeout at 2025-04-21 21:22:55.726306997 +0000 UTC m=+14.247966320, 105.00188ms after stability
2025-04-21T21:22:55.749Z [INFO]  server-8d5befa5-ddcb-9999-eef1-c88e24c3fd45: starting snapshot up to: index=102
2025-04-21T21:22:55.749Z [INFO]  creating new snapshot: path=/tmp/raft4063039906/snapshots/2-102-1745270575749.tmp
2025-04-21T21:22:55.749Z [INFO]  server-8d5befa5-ddcb-9999-eef1-c88e24c3fd45: compacting logs: from=1 to=92
2025-04-21T21:22:55.749Z [INFO]  server-8d5befa5-ddcb-9999-eef1-c88e24c3fd45: snapshot complete up to: index=102
--- PASS: TestRaft_AutoSnapshot (0.31s)
=== RUN   TestRaft_UserSnapshot
2025-04-21T21:22:55.837Z [INFO]  server-cf41f4be-ca56-c060-60ba-569054d7d17b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf41f4be-ca56-c060-60ba-569054d7d17b Address:cf41f4be-ca56-c060-60ba-569054d7d17b}]"
2025-04-21T21:22:55.837Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:55.837Z [INFO]  server-cf41f4be-ca56-c060-60ba-569054d7d17b: entering follower state: follower="Node at cf41f4be-ca56-c060-60ba-569054d7d17b [Follower]" leader-address= leader-id=
2025-04-21T21:22:55.937Z [WARN]  server-cf41f4be-ca56-c060-60ba-569054d7d17b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:55.937Z [INFO]  server-cf41f4be-ca56-c060-60ba-569054d7d17b: entering candidate state: node="Node at cf41f4be-ca56-c060-60ba-569054d7d17b [Candidate]" term=2
2025-04-21T21:22:55.937Z [INFO]  server-cf41f4be-ca56-c060-60ba-569054d7d17b: election won: term=2 tally=1
2025-04-21T21:22:55.937Z [INFO]  server-cf41f4be-ca56-c060-60ba-569054d7d17b: entering leader state: leader="Node at cf41f4be-ca56-c060-60ba-569054d7d17b [Leader]"
2025-04-21T21:22:56.045Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:55.93721904 +0000 UTC m=+14.458878363 (1 nodes), 99.438468ms from start of poll, 99.539109ms from cluster start. Timeout at 2025-04-21 21:22:56.04222232 +0000 UTC m=+14.563881643, 105.00328ms after stability
2025-04-21T21:22:56.045Z [ERROR] server-cf41f4be-ca56-c060-60ba-569054d7d17b: failed to take snapshot: error="nothing new to snapshot"
2025-04-21T21:22:56.045Z [INFO]  server-cf41f4be-ca56-c060-60ba-569054d7d17b: starting snapshot up to: index=12
2025-04-21T21:22:56.045Z [INFO]  creating new snapshot: path=/tmp/raft2916690370/snapshots/2-12-1745270576045.tmp
2025-04-21T21:22:56.045Z [INFO]  server-cf41f4be-ca56-c060-60ba-569054d7d17b: compacting logs: from=1 to=2
2025-04-21T21:22:56.045Z [INFO]  server-cf41f4be-ca56-c060-60ba-569054d7d17b: snapshot complete up to: index=12
--- PASS: TestRaft_UserSnapshot (0.21s)
=== RUN   TestRaft_UserRestore
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false
2025-04-21T21:22:56.045Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-deb9ead5-a91a-b02d-5287-000207a0b459 Address:deb9ead5-a91a-b02d-5287-000207a0b459} {Suffrage:Voter ID:server-06bb5b64-e38e-3121-6f4a-7914b59bfcab Address:06bb5b64-e38e-3121-6f4a-7914b59bfcab} {Suffrage:Voter ID:server-8d465764-3978-4e64-092e-90971f2e20e5 Address:8d465764-3978-4e64-092e-90971f2e20e5}]"
2025-04-21T21:22:56.046Z [INFO]  server-06bb5b64-e38e-3121-6f4a-7914b59bfcab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-deb9ead5-a91a-b02d-5287-000207a0b459 Address:deb9ead5-a91a-b02d-5287-000207a0b459} {Suffrage:Voter ID:server-06bb5b64-e38e-3121-6f4a-7914b59bfcab Address:06bb5b64-e38e-3121-6f4a-7914b59bfcab} {Suffrage:Voter ID:server-8d465764-3978-4e64-092e-90971f2e20e5 Address:8d465764-3978-4e64-092e-90971f2e20e5}]"
2025-04-21T21:22:56.046Z [INFO]  server-8d465764-3978-4e64-092e-90971f2e20e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-deb9ead5-a91a-b02d-5287-000207a0b459 Address:deb9ead5-a91a-b02d-5287-000207a0b459} {Suffrage:Voter ID:server-06bb5b64-e38e-3121-6f4a-7914b59bfcab Address:06bb5b64-e38e-3121-6f4a-7914b59bfcab} {Suffrage:Voter ID:server-8d465764-3978-4e64-092e-90971f2e20e5 Address:8d465764-3978-4e64-092e-90971f2e20e5}]"
2025-04-21T21:22:56.046Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:56.046Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: entering follower state: follower="Node at deb9ead5-a91a-b02d-5287-000207a0b459 [Follower]" leader-address= leader-id=
2025-04-21T21:22:56.046Z [INFO]  server-06bb5b64-e38e-3121-6f4a-7914b59bfcab: entering follower state: follower="Node at 06bb5b64-e38e-3121-6f4a-7914b59bfcab [Follower]" leader-address= leader-id=
2025-04-21T21:22:56.046Z [INFO]  server-8d465764-3978-4e64-092e-90971f2e20e5: entering follower state: follower="Node at 8d465764-3978-4e64-092e-90971f2e20e5 [Follower]" leader-address= leader-id=
2025-04-21T21:22:56.577Z [WARN]  server-deb9ead5-a91a-b02d-5287-000207a0b459: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:56.577Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: entering candidate state: node="Node at deb9ead5-a91a-b02d-5287-000207a0b459 [Candidate]" term=2
2025-04-21T21:22:56.577Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: election won: term=2 tally=2
2025-04-21T21:22:56.577Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: entering leader state: leader="Node at deb9ead5-a91a-b02d-5287-000207a0b459 [Leader]"
2025-04-21T21:22:56.577Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: added peer, starting replication: peer=server-06bb5b64-e38e-3121-6f4a-7914b59bfcab
2025-04-21T21:22:56.577Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: added peer, starting replication: peer=server-8d465764-3978-4e64-092e-90971f2e20e5
2025-04-21T21:22:56.577Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: pipelining replication: peer="{Voter server-8d465764-3978-4e64-092e-90971f2e20e5 8d465764-3978-4e64-092e-90971f2e20e5}"
2025-04-21T21:22:56.577Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: pipelining replication: peer="{Voter server-06bb5b64-e38e-3121-6f4a-7914b59bfcab 06bb5b64-e38e-3121-6f4a-7914b59bfcab}"
2025-04-21T21:22:57.585Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:56.577452942 +0000 UTC m=+15.099112265 (1 nodes), 531.389611ms from start of poll, 531.578852ms from cluster start. Timeout at 2025-04-21 21:22:57.582456622 +0000 UTC m=+16.104115945, 1.00500368s after stability
2025-04-21T21:22:57.585Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: starting snapshot up to: index=12
2025-04-21T21:22:57.585Z [INFO]  creating new snapshot: path=/tmp/raft2155001044/snapshots/2-12-1745270577585.tmp
2025-04-21T21:22:57.585Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: snapshot complete up to: index=12
2025-04-21T21:22:57.587Z [INFO]  creating new snapshot: path=/tmp/raft2155001044/snapshots/2-23-1745270577587.tmp
2025-04-21T21:22:57.587Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: copied to local snapshot: bytes=71
2025-04-21T21:22:57.587Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: snapshot restore progress: id=2-23-1745270577587 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:22:57.587Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: restored user snapshot: index=23
2025-04-21T21:22:57.588Z [ERROR] server-deb9ead5-a91a-b02d-5287-000207a0b459: failed to get log: index=23 error="log not found"
2025-04-21T21:22:57.588Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: aborting pipeline replication: peer="{Voter server-8d465764-3978-4e64-092e-90971f2e20e5 8d465764-3978-4e64-092e-90971f2e20e5}"
2025-04-21T21:22:57.588Z [ERROR] server-deb9ead5-a91a-b02d-5287-000207a0b459: failed to get log: index=23 error="log not found"
2025-04-21T21:22:57.588Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: aborting pipeline replication: peer="{Voter server-06bb5b64-e38e-3121-6f4a-7914b59bfcab 06bb5b64-e38e-3121-6f4a-7914b59bfcab}"
2025-04-21T21:22:57.588Z [ERROR] server-deb9ead5-a91a-b02d-5287-000207a0b459: failed to get log: index=23 error="log not found"
2025-04-21T21:22:57.588Z [INFO]  creating new snapshot: path=/tmp/raft339966401/snapshots/2-23-1745270577588.tmp
2025-04-21T21:22:57.588Z [INFO]  server-06bb5b64-e38e-3121-6f4a-7914b59bfcab: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:22:57.588Z [INFO]  server-06bb5b64-e38e-3121-6f4a-7914b59bfcab: copied to local snapshot: bytes=71
2025-04-21T21:22:57.588Z [INFO]  server-06bb5b64-e38e-3121-6f4a-7914b59bfcab: snapshot restore progress: id=2-23-1745270577588 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:22:57.588Z [INFO]  server-06bb5b64-e38e-3121-6f4a-7914b59bfcab: Installed remote snapshot
2025-04-21T21:22:57.588Z [WARN]  server-06bb5b64-e38e-3121-6f4a-7914b59bfcab: skipping application of old log: index=22
2025-04-21T21:22:57.588Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: pipelining replication: peer="{Voter server-06bb5b64-e38e-3121-6f4a-7914b59bfcab 06bb5b64-e38e-3121-6f4a-7914b59bfcab}"
2025-04-21T21:22:57.588Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:22:57.601Z [ERROR] server-deb9ead5-a91a-b02d-5287-000207a0b459: failed to get log: index=23 error="log not found"
2025-04-21T21:22:57.605Z [INFO]  creating new snapshot: path=/tmp/raft3411588334/snapshots/2-23-1745270577605.tmp
2025-04-21T21:22:57.605Z [INFO]  server-8d465764-3978-4e64-092e-90971f2e20e5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:22:57.605Z [INFO]  server-8d465764-3978-4e64-092e-90971f2e20e5: copied to local snapshot: bytes=71
2025-04-21T21:22:57.605Z [INFO]  server-8d465764-3978-4e64-092e-90971f2e20e5: snapshot restore progress: id=2-23-1745270577605 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:22:57.605Z [INFO]  server-8d465764-3978-4e64-092e-90971f2e20e5: Installed remote snapshot
2025-04-21T21:22:57.605Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: pipelining replication: peer="{Voter server-8d465764-3978-4e64-092e-90971f2e20e5 8d465764-3978-4e64-092e-90971f2e20e5}"
2025-04-21T21:22:58.597Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:22:57.588699098 +0000 UTC m=+16.110358381 (2 nodes), 520ns from start of poll, 1.542824968s from cluster start. Timeout at 2025-04-21 21:22:58.593704538 +0000 UTC m=+17.115363821, 1.00500544s after stability
2025-04-21T21:22:58.625Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: aborting pipeline replication: peer="{Voter server-8d465764-3978-4e64-092e-90971f2e20e5 8d465764-3978-4e64-092e-90971f2e20e5}"
2025-04-21T21:22:58.625Z [INFO]  server-deb9ead5-a91a-b02d-5287-000207a0b459: aborting pipeline replication: peer="{Voter server-06bb5b64-e38e-3121-6f4a-7914b59bfcab 06bb5b64-e38e-3121-6f4a-7914b59bfcab}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false
2025-04-21T21:22:58.625Z [INFO]  server-eb742b4c-d623-7a73-78be-87f032ab347b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eb742b4c-d623-7a73-78be-87f032ab347b Address:eb742b4c-d623-7a73-78be-87f032ab347b} {Suffrage:Voter ID:server-2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0 Address:2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0} {Suffrage:Voter ID:server-47958032-1730-7c9d-9581-483447cbb0ca Address:47958032-1730-7c9d-9581-483447cbb0ca}]"
2025-04-21T21:22:58.626Z [INFO]  server-2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eb742b4c-d623-7a73-78be-87f032ab347b Address:eb742b4c-d623-7a73-78be-87f032ab347b} {Suffrage:Voter ID:server-2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0 Address:2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0} {Suffrage:Voter ID:server-47958032-1730-7c9d-9581-483447cbb0ca Address:47958032-1730-7c9d-9581-483447cbb0ca}]"
2025-04-21T21:22:58.626Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eb742b4c-d623-7a73-78be-87f032ab347b Address:eb742b4c-d623-7a73-78be-87f032ab347b} {Suffrage:Voter ID:server-2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0 Address:2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0} {Suffrage:Voter ID:server-47958032-1730-7c9d-9581-483447cbb0ca Address:47958032-1730-7c9d-9581-483447cbb0ca}]"
2025-04-21T21:22:58.626Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:22:58.626Z [INFO]  server-eb742b4c-d623-7a73-78be-87f032ab347b: entering follower state: follower="Node at eb742b4c-d623-7a73-78be-87f032ab347b [Follower]" leader-address= leader-id=
2025-04-21T21:22:58.626Z [INFO]  server-2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0: entering follower state: follower="Node at 2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0 [Follower]" leader-address= leader-id=
2025-04-21T21:22:58.626Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: entering follower state: follower="Node at 47958032-1730-7c9d-9581-483447cbb0ca [Follower]" leader-address= leader-id=
2025-04-21T21:22:59.297Z [WARN]  server-47958032-1730-7c9d-9581-483447cbb0ca: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:22:59.297Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: entering candidate state: node="Node at 47958032-1730-7c9d-9581-483447cbb0ca [Candidate]" term=2
2025-04-21T21:22:59.297Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: election won: term=2 tally=2
2025-04-21T21:22:59.297Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: entering leader state: leader="Node at 47958032-1730-7c9d-9581-483447cbb0ca [Leader]"
2025-04-21T21:22:59.297Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: added peer, starting replication: peer=server-eb742b4c-d623-7a73-78be-87f032ab347b
2025-04-21T21:22:59.297Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: added peer, starting replication: peer=server-2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0
2025-04-21T21:22:59.297Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: pipelining replication: peer="{Voter server-2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0 2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0}"
2025-04-21T21:22:59.297Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: pipelining replication: peer="{Voter server-eb742b4c-d623-7a73-78be-87f032ab347b eb742b4c-d623-7a73-78be-87f032ab347b}"
2025-04-21T21:23:00.305Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:22:59.297478141 +0000 UTC m=+17.819137464 (1 nodes), 671.349512ms from start of poll, 671.645434ms from cluster start. Timeout at 2025-04-21 21:23:00.302481741 +0000 UTC m=+18.824141064, 1.0050036s after stability
2025-04-21T21:23:00.313Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: starting snapshot up to: index=12
2025-04-21T21:23:00.313Z [INFO]  creating new snapshot: path=/tmp/raft2043633872/snapshots/2-12-1745270580313.tmp
2025-04-21T21:23:00.313Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: snapshot complete up to: index=12
2025-04-21T21:23:00.314Z [INFO]  creating new snapshot: path=/tmp/raft2043633872/snapshots/2-23-1745270580314.tmp
2025-04-21T21:23:00.314Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: copied to local snapshot: bytes=71
2025-04-21T21:23:00.314Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: snapshot restore progress: id=2-23-1745270580314 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:00.314Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: removing all old logs from log store
2025-04-21T21:23:00.314Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: compacting logs: from=1 to=22
2025-04-21T21:23:00.314Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: restored user snapshot: index=23
2025-04-21T21:23:00.314Z [ERROR] server-47958032-1730-7c9d-9581-483447cbb0ca: failed to get log: index=22 error="log not found"
2025-04-21T21:23:00.314Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: aborting pipeline replication: peer="{Voter server-2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0 2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0}"
2025-04-21T21:23:00.314Z [ERROR] server-47958032-1730-7c9d-9581-483447cbb0ca: failed to get log: index=22 error="log not found"
2025-04-21T21:23:00.314Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: aborting pipeline replication: peer="{Voter server-eb742b4c-d623-7a73-78be-87f032ab347b eb742b4c-d623-7a73-78be-87f032ab347b}"
2025-04-21T21:23:00.325Z [ERROR] server-47958032-1730-7c9d-9581-483447cbb0ca: failed to get log: index=22 error="log not found"
2025-04-21T21:23:00.325Z [INFO]  creating new snapshot: path=/tmp/raft1973549031/snapshots/2-23-1745270580325.tmp
2025-04-21T21:23:00.325Z [INFO]  server-2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:00.325Z [INFO]  server-2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0: copied to local snapshot: bytes=71
2025-04-21T21:23:00.325Z [INFO]  server-2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0: snapshot restore progress: id=2-23-1745270580325 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:00.325Z [INFO]  server-2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0: removing all old logs from log store
2025-04-21T21:23:00.325Z [INFO]  server-2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0: compacting logs: from=1 to=22
2025-04-21T21:23:00.325Z [INFO]  server-2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0: Installed remote snapshot
2025-04-21T21:23:00.325Z [WARN]  server-2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0: skipping application of old log: index=22
2025-04-21T21:23:00.325Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: pipelining replication: peer="{Voter server-2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0 2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0}"
2025-04-21T21:23:00.325Z [ERROR] server-47958032-1730-7c9d-9581-483447cbb0ca: failed to get log: index=22 error="log not found"
2025-04-21T21:23:00.326Z [INFO]  creating new snapshot: path=/tmp/raft749788965/snapshots/2-23-1745270580326.tmp
2025-04-21T21:23:00.326Z [INFO]  server-eb742b4c-d623-7a73-78be-87f032ab347b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:00.326Z [INFO]  server-eb742b4c-d623-7a73-78be-87f032ab347b: copied to local snapshot: bytes=71
2025-04-21T21:23:00.326Z [INFO]  server-eb742b4c-d623-7a73-78be-87f032ab347b: snapshot restore progress: id=2-23-1745270580326 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:00.326Z [INFO]  server-eb742b4c-d623-7a73-78be-87f032ab347b: removing all old logs from log store
2025-04-21T21:23:00.326Z [INFO]  server-eb742b4c-d623-7a73-78be-87f032ab347b: compacting logs: from=1 to=22
2025-04-21T21:23:00.326Z [INFO]  server-eb742b4c-d623-7a73-78be-87f032ab347b: Installed remote snapshot
2025-04-21T21:23:00.326Z [WARN]  server-eb742b4c-d623-7a73-78be-87f032ab347b: skipping application of old log: index=22
2025-04-21T21:23:00.326Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: pipelining replication: peer="{Voter server-eb742b4c-d623-7a73-78be-87f032ab347b eb742b4c-d623-7a73-78be-87f032ab347b}"
2025-04-21T21:23:00.326Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:23:01.333Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:23:00.326389855 +0000 UTC m=+18.848049178 (2 nodes), 480ns from start of poll, 1.700557148s from cluster start. Timeout at 2025-04-21 21:23:01.331397855 +0000 UTC m=+19.853057178, 1.005008s after stability
2025-04-21T21:23:01.345Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: aborting pipeline replication: peer="{Voter server-2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0 2ed9ca77-499c-1d4c-e989-1c9cb75ba7a0}"
2025-04-21T21:23:01.345Z [INFO]  server-47958032-1730-7c9d-9581-483447cbb0ca: aborting pipeline replication: peer="{Voter server-eb742b4c-d623-7a73-78be-87f032ab347b eb742b4c-d623-7a73-78be-87f032ab347b}"
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true
2025-04-21T21:23:01.345Z [INFO]  server-bb4fd4e5-a1a9-06eb-21ed-b61048a49e94: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb4fd4e5-a1a9-06eb-21ed-b61048a49e94 Address:bb4fd4e5-a1a9-06eb-21ed-b61048a49e94} {Suffrage:Voter ID:server-b9ca2187-847e-7c66-67f7-85c62e381d98 Address:b9ca2187-847e-7c66-67f7-85c62e381d98} {Suffrage:Voter ID:server-c6920d12-7286-f438-91e6-3d06739ce14a Address:c6920d12-7286-f438-91e6-3d06739ce14a}]"
2025-04-21T21:23:01.345Z [INFO]  server-b9ca2187-847e-7c66-67f7-85c62e381d98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb4fd4e5-a1a9-06eb-21ed-b61048a49e94 Address:bb4fd4e5-a1a9-06eb-21ed-b61048a49e94} {Suffrage:Voter ID:server-b9ca2187-847e-7c66-67f7-85c62e381d98 Address:b9ca2187-847e-7c66-67f7-85c62e381d98} {Suffrage:Voter ID:server-c6920d12-7286-f438-91e6-3d06739ce14a Address:c6920d12-7286-f438-91e6-3d06739ce14a}]"
2025-04-21T21:23:01.345Z [INFO]  server-c6920d12-7286-f438-91e6-3d06739ce14a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb4fd4e5-a1a9-06eb-21ed-b61048a49e94 Address:bb4fd4e5-a1a9-06eb-21ed-b61048a49e94} {Suffrage:Voter ID:server-b9ca2187-847e-7c66-67f7-85c62e381d98 Address:b9ca2187-847e-7c66-67f7-85c62e381d98} {Suffrage:Voter ID:server-c6920d12-7286-f438-91e6-3d06739ce14a Address:c6920d12-7286-f438-91e6-3d06739ce14a}]"
2025-04-21T21:23:01.345Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:01.345Z [INFO]  server-bb4fd4e5-a1a9-06eb-21ed-b61048a49e94: entering follower state: follower="Node at bb4fd4e5-a1a9-06eb-21ed-b61048a49e94 [Follower]" leader-address= leader-id=
2025-04-21T21:23:01.345Z [INFO]  server-b9ca2187-847e-7c66-67f7-85c62e381d98: entering follower state: follower="Node at b9ca2187-847e-7c66-67f7-85c62e381d98 [Follower]" leader-address= leader-id=
2025-04-21T21:23:01.345Z [INFO]  server-c6920d12-7286-f438-91e6-3d06739ce14a: entering follower state: follower="Node at c6920d12-7286-f438-91e6-3d06739ce14a [Follower]" leader-address= leader-id=
2025-04-21T21:23:02.193Z [WARN]  server-bb4fd4e5-a1a9-06eb-21ed-b61048a49e94: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:02.193Z [INFO]  server-bb4fd4e5-a1a9-06eb-21ed-b61048a49e94: entering candidate state: node="Node at bb4fd4e5-a1a9-06eb-21ed-b61048a49e94 [Candidate]" term=2
2025-04-21T21:23:02.193Z [INFO]  server-bb4fd4e5-a1a9-06eb-21ed-b61048a49e94: election won: term=2 tally=2
2025-04-21T21:23:02.193Z [INFO]  server-bb4fd4e5-a1a9-06eb-21ed-b61048a49e94: entering leader state: leader="Node at bb4fd4e5-a1a9-06eb-21ed-b61048a49e94 [Leader]"
2025-04-21T21:23:02.193Z [INFO]  server-bb4fd4e5-a1a9-06eb-21ed-b61048a49e94: added peer, starting replication: peer=server-b9ca2187-847e-7c66-67f7-85c62e381d98
2025-04-21T21:23:02.193Z [INFO]  server-bb4fd4e5-a1a9-06eb-21ed-b61048a49e94: added peer, starting replication: peer=server-c6920d12-7286-f438-91e6-3d06739ce14a
2025-04-21T21:23:02.193Z [INFO]  server-bb4fd4e5-a1a9-06eb-21ed-b61048a49e94: pipelining replication: peer="{Voter server-c6920d12-7286-f438-91e6-3d06739ce14a c6920d12-7286-f438-91e6-3d06739ce14a}"
2025-04-21T21:23:02.193Z [INFO]  server-bb4fd4e5-a1a9-06eb-21ed-b61048a49e94: pipelining replication: peer="{Voter server-b9ca2187-847e-7c66-67f7-85c62e381d98 b9ca2187-847e-7c66-67f7-85c62e381d98}"
2025-04-21T21:23:03.201Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:02.193457402 +0000 UTC m=+20.715116725 (1 nodes), 847.545376ms from start of poll, 847.723177ms from cluster start. Timeout at 2025-04-21 21:23:03.198462682 +0000 UTC m=+21.720122005, 1.00500528s after stability
2025-04-21T21:23:03.211Z [INFO]  server-bb4fd4e5-a1a9-06eb-21ed-b61048a49e94: starting snapshot up to: index=12
2025-04-21T21:23:03.211Z [INFO]  creating new snapshot: path=/tmp/raft3815540233/snapshots/2-12-1745270583211.tmp
2025-04-21T21:23:03.212Z [INFO]  server-bb4fd4e5-a1a9-06eb-21ed-b61048a49e94: snapshot complete up to: index=12
2025-04-21T21:23:03.212Z [INFO]  server-3489c041-8b66-710f-3f7f-3cb2dcd78792: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3489c041-8b66-710f-3f7f-3cb2dcd78792 Address:3489c041-8b66-710f-3f7f-3cb2dcd78792} {Suffrage:Voter ID:server-4c596d81-a6fa-a39d-cd3b-88be940a2f8e Address:4c596d81-a6fa-a39d-cd3b-88be940a2f8e} {Suffrage:Voter ID:server-e82b4703-11f7-7718-c910-732f8de91592 Address:e82b4703-11f7-7718-c910-732f8de91592}]"
2025-04-21T21:23:03.212Z [INFO]  server-4c596d81-a6fa-a39d-cd3b-88be940a2f8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3489c041-8b66-710f-3f7f-3cb2dcd78792 Address:3489c041-8b66-710f-3f7f-3cb2dcd78792} {Suffrage:Voter ID:server-4c596d81-a6fa-a39d-cd3b-88be940a2f8e Address:4c596d81-a6fa-a39d-cd3b-88be940a2f8e} {Suffrage:Voter ID:server-e82b4703-11f7-7718-c910-732f8de91592 Address:e82b4703-11f7-7718-c910-732f8de91592}]"
2025-04-21T21:23:03.212Z [INFO]  server-e82b4703-11f7-7718-c910-732f8de91592: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3489c041-8b66-710f-3f7f-3cb2dcd78792 Address:3489c041-8b66-710f-3f7f-3cb2dcd78792} {Suffrage:Voter ID:server-4c596d81-a6fa-a39d-cd3b-88be940a2f8e Address:4c596d81-a6fa-a39d-cd3b-88be940a2f8e} {Suffrage:Voter ID:server-e82b4703-11f7-7718-c910-732f8de91592 Address:e82b4703-11f7-7718-c910-732f8de91592}]"
2025-04-21T21:23:03.212Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:03.212Z [INFO]  server-3489c041-8b66-710f-3f7f-3cb2dcd78792: entering follower state: follower="Node at 3489c041-8b66-710f-3f7f-3cb2dcd78792 [Follower]" leader-address= leader-id=
2025-04-21T21:23:03.212Z [INFO]  server-4c596d81-a6fa-a39d-cd3b-88be940a2f8e: entering follower state: follower="Node at 4c596d81-a6fa-a39d-cd3b-88be940a2f8e [Follower]" leader-address= leader-id=
2025-04-21T21:23:03.212Z [INFO]  server-e82b4703-11f7-7718-c910-732f8de91592: entering follower state: follower="Node at e82b4703-11f7-7718-c910-732f8de91592 [Follower]" leader-address= leader-id=
2025-04-21T21:23:03.845Z [WARN]  server-e82b4703-11f7-7718-c910-732f8de91592: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:03.845Z [INFO]  server-e82b4703-11f7-7718-c910-732f8de91592: entering candidate state: node="Node at e82b4703-11f7-7718-c910-732f8de91592 [Candidate]" term=2
2025-04-21T21:23:03.845Z [INFO]  server-e82b4703-11f7-7718-c910-732f8de91592: election won: term=2 tally=2
2025-04-21T21:23:03.845Z [INFO]  server-e82b4703-11f7-7718-c910-732f8de91592: entering leader state: leader="Node at e82b4703-11f7-7718-c910-732f8de91592 [Leader]"
2025-04-21T21:23:03.845Z [INFO]  server-e82b4703-11f7-7718-c910-732f8de91592: added peer, starting replication: peer=server-3489c041-8b66-710f-3f7f-3cb2dcd78792
2025-04-21T21:23:03.845Z [INFO]  server-e82b4703-11f7-7718-c910-732f8de91592: added peer, starting replication: peer=server-4c596d81-a6fa-a39d-cd3b-88be940a2f8e
2025-04-21T21:23:03.845Z [INFO]  server-e82b4703-11f7-7718-c910-732f8de91592: pipelining replication: peer="{Voter server-4c596d81-a6fa-a39d-cd3b-88be940a2f8e 4c596d81-a6fa-a39d-cd3b-88be940a2f8e}"
2025-04-21T21:23:03.845Z [INFO]  server-e82b4703-11f7-7718-c910-732f8de91592: pipelining replication: peer="{Voter server-3489c041-8b66-710f-3f7f-3cb2dcd78792 3489c041-8b66-710f-3f7f-3cb2dcd78792}"
2025-04-21T21:23:04.853Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:03.845514784 +0000 UTC m=+22.367174107 (1 nodes), 632.835253ms from start of poll, 632.988654ms from cluster start. Timeout at 2025-04-21 21:23:04.850519264 +0000 UTC m=+23.372178587, 1.00500448s after stability
2025-04-21T21:23:04.853Z [INFO]  creating new snapshot: path=/tmp/raft4174391442/snapshots/2-13-1745270584853.tmp
2025-04-21T21:23:04.853Z [INFO]  server-e82b4703-11f7-7718-c910-732f8de91592: copied to local snapshot: bytes=71
2025-04-21T21:23:04.853Z [INFO]  server-e82b4703-11f7-7718-c910-732f8de91592: snapshot restore progress: id=2-13-1745270584853 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:04.853Z [INFO]  server-e82b4703-11f7-7718-c910-732f8de91592: restored user snapshot: index=13
2025-04-21T21:23:04.853Z [ERROR] server-e82b4703-11f7-7718-c910-732f8de91592: failed to get log: index=3 error="log not found"
2025-04-21T21:23:04.853Z [INFO]  server-e82b4703-11f7-7718-c910-732f8de91592: aborting pipeline replication: peer="{Voter server-4c596d81-a6fa-a39d-cd3b-88be940a2f8e 4c596d81-a6fa-a39d-cd3b-88be940a2f8e}"
2025-04-21T21:23:04.853Z [ERROR] server-e82b4703-11f7-7718-c910-732f8de91592: failed to get log: index=3 error="log not found"
2025-04-21T21:23:04.853Z [INFO]  server-e82b4703-11f7-7718-c910-732f8de91592: aborting pipeline replication: peer="{Voter server-3489c041-8b66-710f-3f7f-3cb2dcd78792 3489c041-8b66-710f-3f7f-3cb2dcd78792}"
2025-04-21T21:23:04.861Z [ERROR] server-e82b4703-11f7-7718-c910-732f8de91592: failed to get log: index=3 error="log not found"
2025-04-21T21:23:04.861Z [INFO]  creating new snapshot: path=/tmp/raft2238323721/snapshots/2-13-1745270584861.tmp
2025-04-21T21:23:04.861Z [INFO]  server-4c596d81-a6fa-a39d-cd3b-88be940a2f8e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:04.861Z [INFO]  server-4c596d81-a6fa-a39d-cd3b-88be940a2f8e: copied to local snapshot: bytes=71
2025-04-21T21:23:04.861Z [INFO]  server-4c596d81-a6fa-a39d-cd3b-88be940a2f8e: snapshot restore progress: id=2-13-1745270584861 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:04.861Z [INFO]  server-4c596d81-a6fa-a39d-cd3b-88be940a2f8e: Installed remote snapshot
2025-04-21T21:23:04.861Z [INFO]  server-e82b4703-11f7-7718-c910-732f8de91592: pipelining replication: peer="{Voter server-4c596d81-a6fa-a39d-cd3b-88be940a2f8e 4c596d81-a6fa-a39d-cd3b-88be940a2f8e}"
2025-04-21T21:23:04.861Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:23:04.861Z [ERROR] server-e82b4703-11f7-7718-c910-732f8de91592: failed to get log: index=3 error="log not found"
2025-04-21T21:23:04.862Z [INFO]  creating new snapshot: path=/tmp/raft2161559371/snapshots/2-13-1745270584862.tmp
2025-04-21T21:23:04.862Z [INFO]  server-3489c041-8b66-710f-3f7f-3cb2dcd78792: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:04.862Z [INFO]  server-3489c041-8b66-710f-3f7f-3cb2dcd78792: copied to local snapshot: bytes=71
2025-04-21T21:23:04.862Z [INFO]  server-3489c041-8b66-710f-3f7f-3cb2dcd78792: snapshot restore progress: id=2-13-1745270584862 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:04.862Z [INFO]  server-3489c041-8b66-710f-3f7f-3cb2dcd78792: Installed remote snapshot
2025-04-21T21:23:04.862Z [INFO]  server-e82b4703-11f7-7718-c910-732f8de91592: pipelining replication: peer="{Voter server-3489c041-8b66-710f-3f7f-3cb2dcd78792 3489c041-8b66-710f-3f7f-3cb2dcd78792}"
2025-04-21T21:23:05.869Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:23:04.861924654 +0000 UTC m=+23.383583977 (2 nodes), 480ns from start of poll, 1.649398524s from cluster start. Timeout at 2025-04-21 21:23:05.866931414 +0000 UTC m=+24.388590737, 1.00500676s after stability
2025-04-21T21:23:05.885Z [INFO]  server-bb4fd4e5-a1a9-06eb-21ed-b61048a49e94: aborting pipeline replication: peer="{Voter server-c6920d12-7286-f438-91e6-3d06739ce14a c6920d12-7286-f438-91e6-3d06739ce14a}"
2025-04-21T21:23:05.885Z [INFO]  server-bb4fd4e5-a1a9-06eb-21ed-b61048a49e94: aborting pipeline replication: peer="{Voter server-b9ca2187-847e-7c66-67f7-85c62e381d98 b9ca2187-847e-7c66-67f7-85c62e381d98}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true
2025-04-21T21:23:05.885Z [INFO]  server-6adc9373-a509-2efa-422e-de18d678b607: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6adc9373-a509-2efa-422e-de18d678b607 Address:6adc9373-a509-2efa-422e-de18d678b607} {Suffrage:Voter ID:server-621e9aab-1d0a-df57-293b-ce28c8ab7197 Address:621e9aab-1d0a-df57-293b-ce28c8ab7197} {Suffrage:Voter ID:server-6270f268-ef75-2608-6976-2e1743dd49f6 Address:6270f268-ef75-2608-6976-2e1743dd49f6}]"
2025-04-21T21:23:05.885Z [INFO]  server-621e9aab-1d0a-df57-293b-ce28c8ab7197: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6adc9373-a509-2efa-422e-de18d678b607 Address:6adc9373-a509-2efa-422e-de18d678b607} {Suffrage:Voter ID:server-621e9aab-1d0a-df57-293b-ce28c8ab7197 Address:621e9aab-1d0a-df57-293b-ce28c8ab7197} {Suffrage:Voter ID:server-6270f268-ef75-2608-6976-2e1743dd49f6 Address:6270f268-ef75-2608-6976-2e1743dd49f6}]"
2025-04-21T21:23:05.885Z [INFO]  server-6270f268-ef75-2608-6976-2e1743dd49f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6adc9373-a509-2efa-422e-de18d678b607 Address:6adc9373-a509-2efa-422e-de18d678b607} {Suffrage:Voter ID:server-621e9aab-1d0a-df57-293b-ce28c8ab7197 Address:621e9aab-1d0a-df57-293b-ce28c8ab7197} {Suffrage:Voter ID:server-6270f268-ef75-2608-6976-2e1743dd49f6 Address:6270f268-ef75-2608-6976-2e1743dd49f6}]"
2025-04-21T21:23:05.885Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:05.885Z [INFO]  server-6adc9373-a509-2efa-422e-de18d678b607: entering follower state: follower="Node at 6adc9373-a509-2efa-422e-de18d678b607 [Follower]" leader-address= leader-id=
2025-04-21T21:23:05.885Z [INFO]  server-621e9aab-1d0a-df57-293b-ce28c8ab7197: entering follower state: follower="Node at 621e9aab-1d0a-df57-293b-ce28c8ab7197 [Follower]" leader-address= leader-id=
2025-04-21T21:23:05.885Z [INFO]  server-6270f268-ef75-2608-6976-2e1743dd49f6: entering follower state: follower="Node at 6270f268-ef75-2608-6976-2e1743dd49f6 [Follower]" leader-address= leader-id=
2025-04-21T21:23:06.401Z [WARN]  server-621e9aab-1d0a-df57-293b-ce28c8ab7197: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:06.401Z [INFO]  server-621e9aab-1d0a-df57-293b-ce28c8ab7197: entering candidate state: node="Node at 621e9aab-1d0a-df57-293b-ce28c8ab7197 [Candidate]" term=2
2025-04-21T21:23:06.401Z [INFO]  server-621e9aab-1d0a-df57-293b-ce28c8ab7197: election won: term=2 tally=2
2025-04-21T21:23:06.401Z [INFO]  server-621e9aab-1d0a-df57-293b-ce28c8ab7197: entering leader state: leader="Node at 621e9aab-1d0a-df57-293b-ce28c8ab7197 [Leader]"
2025-04-21T21:23:06.401Z [INFO]  server-621e9aab-1d0a-df57-293b-ce28c8ab7197: added peer, starting replication: peer=server-6adc9373-a509-2efa-422e-de18d678b607
2025-04-21T21:23:06.401Z [INFO]  server-621e9aab-1d0a-df57-293b-ce28c8ab7197: added peer, starting replication: peer=server-6270f268-ef75-2608-6976-2e1743dd49f6
2025-04-21T21:23:06.401Z [INFO]  server-621e9aab-1d0a-df57-293b-ce28c8ab7197: pipelining replication: peer="{Voter server-6270f268-ef75-2608-6976-2e1743dd49f6 6270f268-ef75-2608-6976-2e1743dd49f6}"
2025-04-21T21:23:06.401Z [INFO]  server-621e9aab-1d0a-df57-293b-ce28c8ab7197: pipelining replication: peer="{Voter server-6adc9373-a509-2efa-422e-de18d678b607 6adc9373-a509-2efa-422e-de18d678b607}"
2025-04-21T21:23:07.409Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:06.40140996 +0000 UTC m=+24.923069243 (1 nodes), 515.578985ms from start of poll, 515.750026ms from cluster start. Timeout at 2025-04-21 21:23:07.40641928 +0000 UTC m=+25.928078603, 1.00500936s after stability
2025-04-21T21:23:07.409Z [INFO]  server-621e9aab-1d0a-df57-293b-ce28c8ab7197: starting snapshot up to: index=12
2025-04-21T21:23:07.409Z [INFO]  creating new snapshot: path=/tmp/raft3054004664/snapshots/2-12-1745270587409.tmp
2025-04-21T21:23:07.409Z [INFO]  server-621e9aab-1d0a-df57-293b-ce28c8ab7197: snapshot complete up to: index=12
2025-04-21T21:23:07.410Z [INFO]  server-61f1a5d3-04cc-7dd6-5710-109ee31f9a4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-61f1a5d3-04cc-7dd6-5710-109ee31f9a4b Address:61f1a5d3-04cc-7dd6-5710-109ee31f9a4b} {Suffrage:Voter ID:server-9d36e859-930e-bc33-bfc1-239cef154c68 Address:9d36e859-930e-bc33-bfc1-239cef154c68} {Suffrage:Voter ID:server-a7f6729c-195a-79a1-a6e5-ae061bb9dd12 Address:a7f6729c-195a-79a1-a6e5-ae061bb9dd12}]"
2025-04-21T21:23:07.410Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-61f1a5d3-04cc-7dd6-5710-109ee31f9a4b Address:61f1a5d3-04cc-7dd6-5710-109ee31f9a4b} {Suffrage:Voter ID:server-9d36e859-930e-bc33-bfc1-239cef154c68 Address:9d36e859-930e-bc33-bfc1-239cef154c68} {Suffrage:Voter ID:server-a7f6729c-195a-79a1-a6e5-ae061bb9dd12 Address:a7f6729c-195a-79a1-a6e5-ae061bb9dd12}]"
2025-04-21T21:23:07.410Z [INFO]  server-a7f6729c-195a-79a1-a6e5-ae061bb9dd12: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-61f1a5d3-04cc-7dd6-5710-109ee31f9a4b Address:61f1a5d3-04cc-7dd6-5710-109ee31f9a4b} {Suffrage:Voter ID:server-9d36e859-930e-bc33-bfc1-239cef154c68 Address:9d36e859-930e-bc33-bfc1-239cef154c68} {Suffrage:Voter ID:server-a7f6729c-195a-79a1-a6e5-ae061bb9dd12 Address:a7f6729c-195a-79a1-a6e5-ae061bb9dd12}]"
2025-04-21T21:23:07.410Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:07.410Z [INFO]  server-61f1a5d3-04cc-7dd6-5710-109ee31f9a4b: entering follower state: follower="Node at 61f1a5d3-04cc-7dd6-5710-109ee31f9a4b [Follower]" leader-address= leader-id=
2025-04-21T21:23:07.410Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: entering follower state: follower="Node at 9d36e859-930e-bc33-bfc1-239cef154c68 [Follower]" leader-address= leader-id=
2025-04-21T21:23:07.410Z [INFO]  server-a7f6729c-195a-79a1-a6e5-ae061bb9dd12: entering follower state: follower="Node at a7f6729c-195a-79a1-a6e5-ae061bb9dd12 [Follower]" leader-address= leader-id=
2025-04-21T21:23:08.137Z [WARN]  server-9d36e859-930e-bc33-bfc1-239cef154c68: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:08.137Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: entering candidate state: node="Node at 9d36e859-930e-bc33-bfc1-239cef154c68 [Candidate]" term=2
2025-04-21T21:23:08.137Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: election won: term=2 tally=2
2025-04-21T21:23:08.137Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: entering leader state: leader="Node at 9d36e859-930e-bc33-bfc1-239cef154c68 [Leader]"
2025-04-21T21:23:08.137Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: added peer, starting replication: peer=server-61f1a5d3-04cc-7dd6-5710-109ee31f9a4b
2025-04-21T21:23:08.137Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: added peer, starting replication: peer=server-a7f6729c-195a-79a1-a6e5-ae061bb9dd12
2025-04-21T21:23:08.137Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: pipelining replication: peer="{Voter server-a7f6729c-195a-79a1-a6e5-ae061bb9dd12 a7f6729c-195a-79a1-a6e5-ae061bb9dd12}"
2025-04-21T21:23:08.137Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: pipelining replication: peer="{Voter server-61f1a5d3-04cc-7dd6-5710-109ee31f9a4b 61f1a5d3-04cc-7dd6-5710-109ee31f9a4b}"
2025-04-21T21:23:09.145Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:08.137448866 +0000 UTC m=+26.659108189 (1 nodes), 726.943325ms from start of poll, 727.124086ms from cluster start. Timeout at 2025-04-21 21:23:09.142450986 +0000 UTC m=+27.664110309, 1.00500212s after stability
2025-04-21T21:23:09.145Z [INFO]  creating new snapshot: path=/tmp/raft3102438059/snapshots/2-13-1745270589145.tmp
2025-04-21T21:23:09.145Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: copied to local snapshot: bytes=71
2025-04-21T21:23:09.145Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: snapshot restore progress: id=2-13-1745270589145 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:09.145Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: removing all old logs from log store
2025-04-21T21:23:09.145Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: compacting logs: from=1 to=2
2025-04-21T21:23:09.145Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: restored user snapshot: index=13
2025-04-21T21:23:09.145Z [ERROR] server-9d36e859-930e-bc33-bfc1-239cef154c68: failed to get log: index=2 error="log not found"
2025-04-21T21:23:09.145Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: aborting pipeline replication: peer="{Voter server-a7f6729c-195a-79a1-a6e5-ae061bb9dd12 a7f6729c-195a-79a1-a6e5-ae061bb9dd12}"
2025-04-21T21:23:09.145Z [ERROR] server-9d36e859-930e-bc33-bfc1-239cef154c68: failed to get log: index=2 error="log not found"
2025-04-21T21:23:09.145Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: aborting pipeline replication: peer="{Voter server-61f1a5d3-04cc-7dd6-5710-109ee31f9a4b 61f1a5d3-04cc-7dd6-5710-109ee31f9a4b}"
2025-04-21T21:23:09.153Z [ERROR] server-9d36e859-930e-bc33-bfc1-239cef154c68: failed to get log: index=2 error="log not found"
2025-04-21T21:23:09.153Z [INFO]  creating new snapshot: path=/tmp/raft4111532424/snapshots/2-13-1745270589153.tmp
2025-04-21T21:23:09.153Z [INFO]  server-61f1a5d3-04cc-7dd6-5710-109ee31f9a4b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:09.153Z [INFO]  server-61f1a5d3-04cc-7dd6-5710-109ee31f9a4b: copied to local snapshot: bytes=71
2025-04-21T21:23:09.153Z [INFO]  server-61f1a5d3-04cc-7dd6-5710-109ee31f9a4b: snapshot restore progress: id=2-13-1745270589153 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:09.153Z [INFO]  server-61f1a5d3-04cc-7dd6-5710-109ee31f9a4b: removing all old logs from log store
2025-04-21T21:23:09.153Z [INFO]  server-61f1a5d3-04cc-7dd6-5710-109ee31f9a4b: compacting logs: from=1 to=2
2025-04-21T21:23:09.153Z [INFO]  server-61f1a5d3-04cc-7dd6-5710-109ee31f9a4b: Installed remote snapshot
2025-04-21T21:23:09.153Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: pipelining replication: peer="{Voter server-61f1a5d3-04cc-7dd6-5710-109ee31f9a4b 61f1a5d3-04cc-7dd6-5710-109ee31f9a4b}"
2025-04-21T21:23:09.153Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:23:09.161Z [ERROR] server-9d36e859-930e-bc33-bfc1-239cef154c68: failed to get log: index=2 error="log not found"
2025-04-21T21:23:09.161Z [INFO]  creating new snapshot: path=/tmp/raft2293766632/snapshots/2-13-1745270589161.tmp
2025-04-21T21:23:09.161Z [INFO]  server-a7f6729c-195a-79a1-a6e5-ae061bb9dd12: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:09.161Z [INFO]  server-a7f6729c-195a-79a1-a6e5-ae061bb9dd12: copied to local snapshot: bytes=71
2025-04-21T21:23:09.161Z [INFO]  server-a7f6729c-195a-79a1-a6e5-ae061bb9dd12: snapshot restore progress: id=2-13-1745270589161 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:09.161Z [INFO]  server-a7f6729c-195a-79a1-a6e5-ae061bb9dd12: removing all old logs from log store
2025-04-21T21:23:09.161Z [INFO]  server-a7f6729c-195a-79a1-a6e5-ae061bb9dd12: compacting logs: from=1 to=2
2025-04-21T21:23:09.161Z [INFO]  server-a7f6729c-195a-79a1-a6e5-ae061bb9dd12: Installed remote snapshot
2025-04-21T21:23:09.161Z [INFO]  server-9d36e859-930e-bc33-bfc1-239cef154c68: pipelining replication: peer="{Voter server-a7f6729c-195a-79a1-a6e5-ae061bb9dd12 a7f6729c-195a-79a1-a6e5-ae061bb9dd12}"
2025-04-21T21:23:10.161Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:23:09.153936217 +0000 UTC m=+27.675595540 (2 nodes), 520ns from start of poll, 1.743611437s from cluster start. Timeout at 2025-04-21 21:23:10.158954137 +0000 UTC m=+28.680613460, 1.00501792s after stability
2025-04-21T21:23:10.177Z [INFO]  server-621e9aab-1d0a-df57-293b-ce28c8ab7197: aborting pipeline replication: peer="{Voter server-6270f268-ef75-2608-6976-2e1743dd49f6 6270f268-ef75-2608-6976-2e1743dd49f6}"
2025-04-21T21:23:10.177Z [INFO]  server-621e9aab-1d0a-df57-293b-ce28c8ab7197: aborting pipeline replication: peer="{Voter server-6adc9373-a509-2efa-422e-de18d678b607 6adc9373-a509-2efa-422e-de18d678b607}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false
2025-04-21T21:23:10.177Z [INFO]  server-55ed3d27-9f4a-19de-336d-96c007a1d494: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-55ed3d27-9f4a-19de-336d-96c007a1d494 Address:55ed3d27-9f4a-19de-336d-96c007a1d494} {Suffrage:Voter ID:server-460ea33a-1b50-9661-8e0c-5768362a47f5 Address:460ea33a-1b50-9661-8e0c-5768362a47f5} {Suffrage:Voter ID:server-1b419410-9e9a-4fb8-a45c-903d8bd70796 Address:1b419410-9e9a-4fb8-a45c-903d8bd70796}]"
2025-04-21T21:23:10.177Z [INFO]  server-460ea33a-1b50-9661-8e0c-5768362a47f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-55ed3d27-9f4a-19de-336d-96c007a1d494 Address:55ed3d27-9f4a-19de-336d-96c007a1d494} {Suffrage:Voter ID:server-460ea33a-1b50-9661-8e0c-5768362a47f5 Address:460ea33a-1b50-9661-8e0c-5768362a47f5} {Suffrage:Voter ID:server-1b419410-9e9a-4fb8-a45c-903d8bd70796 Address:1b419410-9e9a-4fb8-a45c-903d8bd70796}]"
2025-04-21T21:23:10.178Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-55ed3d27-9f4a-19de-336d-96c007a1d494 Address:55ed3d27-9f4a-19de-336d-96c007a1d494} {Suffrage:Voter ID:server-460ea33a-1b50-9661-8e0c-5768362a47f5 Address:460ea33a-1b50-9661-8e0c-5768362a47f5} {Suffrage:Voter ID:server-1b419410-9e9a-4fb8-a45c-903d8bd70796 Address:1b419410-9e9a-4fb8-a45c-903d8bd70796}]"
2025-04-21T21:23:10.178Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:10.178Z [INFO]  server-55ed3d27-9f4a-19de-336d-96c007a1d494: entering follower state: follower="Node at 55ed3d27-9f4a-19de-336d-96c007a1d494 [Follower]" leader-address= leader-id=
2025-04-21T21:23:10.178Z [INFO]  server-460ea33a-1b50-9661-8e0c-5768362a47f5: entering follower state: follower="Node at 460ea33a-1b50-9661-8e0c-5768362a47f5 [Follower]" leader-address= leader-id=
2025-04-21T21:23:10.178Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: entering follower state: follower="Node at 1b419410-9e9a-4fb8-a45c-903d8bd70796 [Follower]" leader-address= leader-id=
2025-04-21T21:23:10.833Z [WARN]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:10.833Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: entering candidate state: node="Node at 1b419410-9e9a-4fb8-a45c-903d8bd70796 [Candidate]" term=2
2025-04-21T21:23:10.833Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: election won: term=2 tally=2
2025-04-21T21:23:10.833Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: entering leader state: leader="Node at 1b419410-9e9a-4fb8-a45c-903d8bd70796 [Leader]"
2025-04-21T21:23:10.833Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: added peer, starting replication: peer=server-55ed3d27-9f4a-19de-336d-96c007a1d494
2025-04-21T21:23:10.833Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: added peer, starting replication: peer=server-460ea33a-1b50-9661-8e0c-5768362a47f5
2025-04-21T21:23:10.833Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: pipelining replication: peer="{Voter server-460ea33a-1b50-9661-8e0c-5768362a47f5 460ea33a-1b50-9661-8e0c-5768362a47f5}"
2025-04-21T21:23:10.833Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: pipelining replication: peer="{Voter server-55ed3d27-9f4a-19de-336d-96c007a1d494 55ed3d27-9f4a-19de-336d-96c007a1d494}"
2025-04-21T21:23:11.841Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:10.833454144 +0000 UTC m=+29.355113467 (1 nodes), 655.433405ms from start of poll, 655.643166ms from cluster start. Timeout at 2025-04-21 21:23:11.838456184 +0000 UTC m=+30.360115507, 1.00500204s after stability
2025-04-21T21:23:11.841Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: starting snapshot up to: index=12
2025-04-21T21:23:11.841Z [INFO]  creating new snapshot: path=/tmp/raft3695876498/snapshots/2-12-1745270591841.tmp
2025-04-21T21:23:11.841Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: snapshot complete up to: index=12
2025-04-21T21:23:11.850Z [INFO]  creating new snapshot: path=/tmp/raft3695876498/snapshots/2-23-1745270591850.tmp
2025-04-21T21:23:11.861Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: copied to local snapshot: bytes=71
2025-04-21T21:23:11.861Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: snapshot restore progress: id=2-23-1745270591850 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:11.861Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: restored user snapshot: index=23
2025-04-21T21:23:11.861Z [ERROR] server-1b419410-9e9a-4fb8-a45c-903d8bd70796: failed to get log: index=23 error="log not found"
2025-04-21T21:23:11.861Z [ERROR] server-1b419410-9e9a-4fb8-a45c-903d8bd70796: failed to get log: index=23 error="log not found"
2025-04-21T21:23:11.861Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: aborting pipeline replication: peer="{Voter server-55ed3d27-9f4a-19de-336d-96c007a1d494 55ed3d27-9f4a-19de-336d-96c007a1d494}"
2025-04-21T21:23:11.861Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: aborting pipeline replication: peer="{Voter server-460ea33a-1b50-9661-8e0c-5768362a47f5 460ea33a-1b50-9661-8e0c-5768362a47f5}"
2025-04-21T21:23:11.869Z [ERROR] server-1b419410-9e9a-4fb8-a45c-903d8bd70796: failed to get log: index=23 error="log not found"
2025-04-21T21:23:11.869Z [INFO]  creating new snapshot: path=/tmp/raft2793233625/snapshots/2-23-1745270591869.tmp
2025-04-21T21:23:11.869Z [INFO]  server-460ea33a-1b50-9661-8e0c-5768362a47f5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:11.869Z [INFO]  server-460ea33a-1b50-9661-8e0c-5768362a47f5: copied to local snapshot: bytes=71
2025-04-21T21:23:11.869Z [INFO]  server-460ea33a-1b50-9661-8e0c-5768362a47f5: snapshot restore progress: id=2-23-1745270591869 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:11.869Z [INFO]  server-460ea33a-1b50-9661-8e0c-5768362a47f5: Installed remote snapshot
2025-04-21T21:23:11.869Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: pipelining replication: peer="{Voter server-460ea33a-1b50-9661-8e0c-5768362a47f5 460ea33a-1b50-9661-8e0c-5768362a47f5}"
2025-04-21T21:23:11.869Z [ERROR] server-1b419410-9e9a-4fb8-a45c-903d8bd70796: failed to get log: index=23 error="log not found"
2025-04-21T21:23:11.870Z [INFO]  creating new snapshot: path=/tmp/raft2591707446/snapshots/2-23-1745270591870.tmp
2025-04-21T21:23:11.870Z [INFO]  server-55ed3d27-9f4a-19de-336d-96c007a1d494: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:11.870Z [INFO]  server-55ed3d27-9f4a-19de-336d-96c007a1d494: copied to local snapshot: bytes=71
2025-04-21T21:23:11.870Z [INFO]  server-55ed3d27-9f4a-19de-336d-96c007a1d494: snapshot restore progress: id=2-23-1745270591870 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:11.870Z [INFO]  server-55ed3d27-9f4a-19de-336d-96c007a1d494: Installed remote snapshot
2025-04-21T21:23:11.870Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: pipelining replication: peer="{Voter server-55ed3d27-9f4a-19de-336d-96c007a1d494 55ed3d27-9f4a-19de-336d-96c007a1d494}"
2025-04-21T21:23:11.870Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:23:12.881Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:23:11.870563513 +0000 UTC m=+30.392222836 (2 nodes), 440ns from start of poll, 1.692752535s from cluster start. Timeout at 2025-04-21 21:23:12.875570353 +0000 UTC m=+31.397229636, 1.0050068s after stability
2025-04-21T21:23:12.897Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: aborting pipeline replication: peer="{Voter server-460ea33a-1b50-9661-8e0c-5768362a47f5 460ea33a-1b50-9661-8e0c-5768362a47f5}"
2025-04-21T21:23:12.897Z [INFO]  server-1b419410-9e9a-4fb8-a45c-903d8bd70796: aborting pipeline replication: peer="{Voter server-55ed3d27-9f4a-19de-336d-96c007a1d494 55ed3d27-9f4a-19de-336d-96c007a1d494}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false
2025-04-21T21:23:12.897Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8737dce-8d8e-f1af-9010-0072508a6b16 Address:a8737dce-8d8e-f1af-9010-0072508a6b16} {Suffrage:Voter ID:server-8e241264-9fe3-5d4e-e8d3-4721aa972be0 Address:8e241264-9fe3-5d4e-e8d3-4721aa972be0} {Suffrage:Voter ID:server-3252a5e1-ca2f-6996-e2b7-6c35e52bd47c Address:3252a5e1-ca2f-6996-e2b7-6c35e52bd47c}]"
2025-04-21T21:23:12.897Z [INFO]  server-8e241264-9fe3-5d4e-e8d3-4721aa972be0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8737dce-8d8e-f1af-9010-0072508a6b16 Address:a8737dce-8d8e-f1af-9010-0072508a6b16} {Suffrage:Voter ID:server-8e241264-9fe3-5d4e-e8d3-4721aa972be0 Address:8e241264-9fe3-5d4e-e8d3-4721aa972be0} {Suffrage:Voter ID:server-3252a5e1-ca2f-6996-e2b7-6c35e52bd47c Address:3252a5e1-ca2f-6996-e2b7-6c35e52bd47c}]"
2025-04-21T21:23:12.897Z [INFO]  server-3252a5e1-ca2f-6996-e2b7-6c35e52bd47c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a8737dce-8d8e-f1af-9010-0072508a6b16 Address:a8737dce-8d8e-f1af-9010-0072508a6b16} {Suffrage:Voter ID:server-8e241264-9fe3-5d4e-e8d3-4721aa972be0 Address:8e241264-9fe3-5d4e-e8d3-4721aa972be0} {Suffrage:Voter ID:server-3252a5e1-ca2f-6996-e2b7-6c35e52bd47c Address:3252a5e1-ca2f-6996-e2b7-6c35e52bd47c}]"
2025-04-21T21:23:12.897Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:12.897Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: entering follower state: follower="Node at a8737dce-8d8e-f1af-9010-0072508a6b16 [Follower]" leader-address= leader-id=
2025-04-21T21:23:12.897Z [INFO]  server-8e241264-9fe3-5d4e-e8d3-4721aa972be0: entering follower state: follower="Node at 8e241264-9fe3-5d4e-e8d3-4721aa972be0 [Follower]" leader-address= leader-id=
2025-04-21T21:23:12.897Z [INFO]  server-3252a5e1-ca2f-6996-e2b7-6c35e52bd47c: entering follower state: follower="Node at 3252a5e1-ca2f-6996-e2b7-6c35e52bd47c [Follower]" leader-address= leader-id=
2025-04-21T21:23:13.485Z [WARN]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:13.485Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: entering candidate state: node="Node at a8737dce-8d8e-f1af-9010-0072508a6b16 [Candidate]" term=2
2025-04-21T21:23:13.485Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: election won: term=2 tally=2
2025-04-21T21:23:13.485Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: entering leader state: leader="Node at a8737dce-8d8e-f1af-9010-0072508a6b16 [Leader]"
2025-04-21T21:23:13.485Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: added peer, starting replication: peer=server-8e241264-9fe3-5d4e-e8d3-4721aa972be0
2025-04-21T21:23:13.485Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: added peer, starting replication: peer=server-3252a5e1-ca2f-6996-e2b7-6c35e52bd47c
2025-04-21T21:23:13.485Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: pipelining replication: peer="{Voter server-3252a5e1-ca2f-6996-e2b7-6c35e52bd47c 3252a5e1-ca2f-6996-e2b7-6c35e52bd47c}"
2025-04-21T21:23:13.485Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: pipelining replication: peer="{Voter server-8e241264-9fe3-5d4e-e8d3-4721aa972be0 8e241264-9fe3-5d4e-e8d3-4721aa972be0}"
2025-04-21T21:23:14.493Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:13.485568886 +0000 UTC m=+32.007228169 (1 nodes), 587.689349ms from start of poll, 587.85499ms from cluster start. Timeout at 2025-04-21 21:23:14.490578886 +0000 UTC m=+33.012238209, 1.00501004s after stability
2025-04-21T21:23:14.493Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: starting snapshot up to: index=12
2025-04-21T21:23:14.493Z [INFO]  creating new snapshot: path=/tmp/raft1231225190/snapshots/2-12-1745270594493.tmp
2025-04-21T21:23:14.493Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: snapshot complete up to: index=12
2025-04-21T21:23:14.494Z [INFO]  creating new snapshot: path=/tmp/raft1231225190/snapshots/2-23-1745270594494.tmp
2025-04-21T21:23:14.494Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: copied to local snapshot: bytes=71
2025-04-21T21:23:14.494Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: snapshot restore progress: id=2-23-1745270594494 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:14.494Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: removing all old logs from log store
2025-04-21T21:23:14.494Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: compacting logs: from=1 to=22
2025-04-21T21:23:14.494Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: restored user snapshot: index=23
2025-04-21T21:23:14.494Z [ERROR] server-a8737dce-8d8e-f1af-9010-0072508a6b16: failed to get log: index=22 error="log not found"
2025-04-21T21:23:14.494Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: aborting pipeline replication: peer="{Voter server-8e241264-9fe3-5d4e-e8d3-4721aa972be0 8e241264-9fe3-5d4e-e8d3-4721aa972be0}"
2025-04-21T21:23:14.494Z [ERROR] server-a8737dce-8d8e-f1af-9010-0072508a6b16: failed to get log: index=22 error="log not found"
2025-04-21T21:23:14.494Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: aborting pipeline replication: peer="{Voter server-3252a5e1-ca2f-6996-e2b7-6c35e52bd47c 3252a5e1-ca2f-6996-e2b7-6c35e52bd47c}"
2025-04-21T21:23:14.501Z [ERROR] server-a8737dce-8d8e-f1af-9010-0072508a6b16: failed to get log: index=22 error="log not found"
2025-04-21T21:23:14.501Z [INFO]  creating new snapshot: path=/tmp/raft2169334525/snapshots/2-23-1745270594501.tmp
2025-04-21T21:23:14.509Z [INFO]  server-3252a5e1-ca2f-6996-e2b7-6c35e52bd47c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:14.509Z [INFO]  server-3252a5e1-ca2f-6996-e2b7-6c35e52bd47c: copied to local snapshot: bytes=71
2025-04-21T21:23:14.509Z [INFO]  server-3252a5e1-ca2f-6996-e2b7-6c35e52bd47c: snapshot restore progress: id=2-23-1745270594501 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:14.509Z [INFO]  server-3252a5e1-ca2f-6996-e2b7-6c35e52bd47c: removing all old logs from log store
2025-04-21T21:23:14.509Z [INFO]  server-3252a5e1-ca2f-6996-e2b7-6c35e52bd47c: compacting logs: from=1 to=22
2025-04-21T21:23:14.509Z [INFO]  server-3252a5e1-ca2f-6996-e2b7-6c35e52bd47c: Installed remote snapshot
2025-04-21T21:23:14.509Z [WARN]  server-3252a5e1-ca2f-6996-e2b7-6c35e52bd47c: skipping application of old log: index=22
2025-04-21T21:23:14.509Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: pipelining replication: peer="{Voter server-3252a5e1-ca2f-6996-e2b7-6c35e52bd47c 3252a5e1-ca2f-6996-e2b7-6c35e52bd47c}"
2025-04-21T21:23:14.509Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:23:14.525Z [ERROR] server-a8737dce-8d8e-f1af-9010-0072508a6b16: failed to get log: index=22 error="log not found"
2025-04-21T21:23:14.525Z [INFO]  creating new snapshot: path=/tmp/raft3360665010/snapshots/2-23-1745270594525.tmp
2025-04-21T21:23:14.525Z [INFO]  server-8e241264-9fe3-5d4e-e8d3-4721aa972be0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:14.525Z [INFO]  server-8e241264-9fe3-5d4e-e8d3-4721aa972be0: copied to local snapshot: bytes=71
2025-04-21T21:23:14.525Z [INFO]  server-8e241264-9fe3-5d4e-e8d3-4721aa972be0: snapshot restore progress: id=2-23-1745270594525 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:14.525Z [INFO]  server-8e241264-9fe3-5d4e-e8d3-4721aa972be0: removing all old logs from log store
2025-04-21T21:23:14.525Z [INFO]  server-8e241264-9fe3-5d4e-e8d3-4721aa972be0: compacting logs: from=1 to=22
2025-04-21T21:23:14.525Z [INFO]  server-8e241264-9fe3-5d4e-e8d3-4721aa972be0: Installed remote snapshot
2025-04-21T21:23:14.526Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: pipelining replication: peer="{Voter server-8e241264-9fe3-5d4e-e8d3-4721aa972be0 8e241264-9fe3-5d4e-e8d3-4721aa972be0}"
2025-04-21T21:23:15.517Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:23:14.509734889 +0000 UTC m=+33.031394212 (2 nodes), 760ns from start of poll, 1.612021033s from cluster start. Timeout at 2025-04-21 21:23:15.514748769 +0000 UTC m=+34.036408092, 1.00501388s after stability
2025-04-21T21:23:15.533Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: aborting pipeline replication: peer="{Voter server-3252a5e1-ca2f-6996-e2b7-6c35e52bd47c 3252a5e1-ca2f-6996-e2b7-6c35e52bd47c}"
2025-04-21T21:23:15.533Z [INFO]  server-a8737dce-8d8e-f1af-9010-0072508a6b16: aborting pipeline replication: peer="{Voter server-8e241264-9fe3-5d4e-e8d3-4721aa972be0 8e241264-9fe3-5d4e-e8d3-4721aa972be0}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true
2025-04-21T21:23:15.533Z [INFO]  server-4416e9d4-f6a8-c049-2a7b-6191beb5abcb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4416e9d4-f6a8-c049-2a7b-6191beb5abcb Address:4416e9d4-f6a8-c049-2a7b-6191beb5abcb} {Suffrage:Voter ID:server-d65668bf-8b99-dc82-a75c-6a0866269ff7 Address:d65668bf-8b99-dc82-a75c-6a0866269ff7} {Suffrage:Voter ID:server-8a9fd009-ac0c-be32-7842-7bc60f8a5157 Address:8a9fd009-ac0c-be32-7842-7bc60f8a5157}]"
2025-04-21T21:23:15.533Z [INFO]  server-d65668bf-8b99-dc82-a75c-6a0866269ff7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4416e9d4-f6a8-c049-2a7b-6191beb5abcb Address:4416e9d4-f6a8-c049-2a7b-6191beb5abcb} {Suffrage:Voter ID:server-d65668bf-8b99-dc82-a75c-6a0866269ff7 Address:d65668bf-8b99-dc82-a75c-6a0866269ff7} {Suffrage:Voter ID:server-8a9fd009-ac0c-be32-7842-7bc60f8a5157 Address:8a9fd009-ac0c-be32-7842-7bc60f8a5157}]"
2025-04-21T21:23:15.533Z [INFO]  server-8a9fd009-ac0c-be32-7842-7bc60f8a5157: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4416e9d4-f6a8-c049-2a7b-6191beb5abcb Address:4416e9d4-f6a8-c049-2a7b-6191beb5abcb} {Suffrage:Voter ID:server-d65668bf-8b99-dc82-a75c-6a0866269ff7 Address:d65668bf-8b99-dc82-a75c-6a0866269ff7} {Suffrage:Voter ID:server-8a9fd009-ac0c-be32-7842-7bc60f8a5157 Address:8a9fd009-ac0c-be32-7842-7bc60f8a5157}]"
2025-04-21T21:23:15.533Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:15.533Z [INFO]  server-4416e9d4-f6a8-c049-2a7b-6191beb5abcb: entering follower state: follower="Node at 4416e9d4-f6a8-c049-2a7b-6191beb5abcb [Follower]" leader-address= leader-id=
2025-04-21T21:23:15.534Z [INFO]  server-d65668bf-8b99-dc82-a75c-6a0866269ff7: entering follower state: follower="Node at d65668bf-8b99-dc82-a75c-6a0866269ff7 [Follower]" leader-address= leader-id=
2025-04-21T21:23:15.534Z [INFO]  server-8a9fd009-ac0c-be32-7842-7bc60f8a5157: entering follower state: follower="Node at 8a9fd009-ac0c-be32-7842-7bc60f8a5157 [Follower]" leader-address= leader-id=
2025-04-21T21:23:16.281Z [WARN]  server-d65668bf-8b99-dc82-a75c-6a0866269ff7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:16.281Z [INFO]  server-d65668bf-8b99-dc82-a75c-6a0866269ff7: entering candidate state: node="Node at d65668bf-8b99-dc82-a75c-6a0866269ff7 [Candidate]" term=2
2025-04-21T21:23:16.281Z [INFO]  server-d65668bf-8b99-dc82-a75c-6a0866269ff7: election won: term=2 tally=2
2025-04-21T21:23:16.281Z [INFO]  server-d65668bf-8b99-dc82-a75c-6a0866269ff7: entering leader state: leader="Node at d65668bf-8b99-dc82-a75c-6a0866269ff7 [Leader]"
2025-04-21T21:23:16.281Z [INFO]  server-d65668bf-8b99-dc82-a75c-6a0866269ff7: added peer, starting replication: peer=server-4416e9d4-f6a8-c049-2a7b-6191beb5abcb
2025-04-21T21:23:16.281Z [INFO]  server-d65668bf-8b99-dc82-a75c-6a0866269ff7: added peer, starting replication: peer=server-8a9fd009-ac0c-be32-7842-7bc60f8a5157
2025-04-21T21:23:16.281Z [INFO]  server-d65668bf-8b99-dc82-a75c-6a0866269ff7: pipelining replication: peer="{Voter server-4416e9d4-f6a8-c049-2a7b-6191beb5abcb 4416e9d4-f6a8-c049-2a7b-6191beb5abcb}"
2025-04-21T21:23:16.281Z [INFO]  server-d65668bf-8b99-dc82-a75c-6a0866269ff7: pipelining replication: peer="{Voter server-8a9fd009-ac0c-be32-7842-7bc60f8a5157 8a9fd009-ac0c-be32-7842-7bc60f8a5157}"
2025-04-21T21:23:17.289Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:16.281226434 +0000 UTC m=+34.802885717 (1 nodes), 747.245742ms from start of poll, 747.438343ms from cluster start. Timeout at 2025-04-21 21:23:17.286231434 +0000 UTC m=+35.807890757, 1.00500504s after stability
2025-04-21T21:23:17.289Z [INFO]  server-d65668bf-8b99-dc82-a75c-6a0866269ff7: starting snapshot up to: index=12
2025-04-21T21:23:17.289Z [INFO]  creating new snapshot: path=/tmp/raft1845525604/snapshots/2-12-1745270597289.tmp
2025-04-21T21:23:17.289Z [INFO]  server-d65668bf-8b99-dc82-a75c-6a0866269ff7: snapshot complete up to: index=12
2025-04-21T21:23:17.290Z [INFO]  server-7c858cc2-b1c9-6b4c-9401-ed02cf8498d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c858cc2-b1c9-6b4c-9401-ed02cf8498d0 Address:7c858cc2-b1c9-6b4c-9401-ed02cf8498d0} {Suffrage:Voter ID:server-e806fdd8-3c76-e0f2-538b-db56e7f5c976 Address:e806fdd8-3c76-e0f2-538b-db56e7f5c976} {Suffrage:Voter ID:server-6e13ad29-8dda-2298-0ef1-dab3b401978a Address:6e13ad29-8dda-2298-0ef1-dab3b401978a}]"
2025-04-21T21:23:17.290Z [INFO]  server-e806fdd8-3c76-e0f2-538b-db56e7f5c976: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c858cc2-b1c9-6b4c-9401-ed02cf8498d0 Address:7c858cc2-b1c9-6b4c-9401-ed02cf8498d0} {Suffrage:Voter ID:server-e806fdd8-3c76-e0f2-538b-db56e7f5c976 Address:e806fdd8-3c76-e0f2-538b-db56e7f5c976} {Suffrage:Voter ID:server-6e13ad29-8dda-2298-0ef1-dab3b401978a Address:6e13ad29-8dda-2298-0ef1-dab3b401978a}]"
2025-04-21T21:23:17.290Z [INFO]  server-6e13ad29-8dda-2298-0ef1-dab3b401978a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c858cc2-b1c9-6b4c-9401-ed02cf8498d0 Address:7c858cc2-b1c9-6b4c-9401-ed02cf8498d0} {Suffrage:Voter ID:server-e806fdd8-3c76-e0f2-538b-db56e7f5c976 Address:e806fdd8-3c76-e0f2-538b-db56e7f5c976} {Suffrage:Voter ID:server-6e13ad29-8dda-2298-0ef1-dab3b401978a Address:6e13ad29-8dda-2298-0ef1-dab3b401978a}]"
2025-04-21T21:23:17.290Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:17.290Z [INFO]  server-7c858cc2-b1c9-6b4c-9401-ed02cf8498d0: entering follower state: follower="Node at 7c858cc2-b1c9-6b4c-9401-ed02cf8498d0 [Follower]" leader-address= leader-id=
2025-04-21T21:23:17.290Z [INFO]  server-e806fdd8-3c76-e0f2-538b-db56e7f5c976: entering follower state: follower="Node at e806fdd8-3c76-e0f2-538b-db56e7f5c976 [Follower]" leader-address= leader-id=
2025-04-21T21:23:17.290Z [INFO]  server-6e13ad29-8dda-2298-0ef1-dab3b401978a: entering follower state: follower="Node at 6e13ad29-8dda-2298-0ef1-dab3b401978a [Follower]" leader-address= leader-id=
2025-04-21T21:23:18.017Z [WARN]  server-6e13ad29-8dda-2298-0ef1-dab3b401978a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:18.017Z [INFO]  server-6e13ad29-8dda-2298-0ef1-dab3b401978a: entering candidate state: node="Node at 6e13ad29-8dda-2298-0ef1-dab3b401978a [Candidate]" term=2
2025-04-21T21:23:18.017Z [INFO]  server-6e13ad29-8dda-2298-0ef1-dab3b401978a: election won: term=2 tally=2
2025-04-21T21:23:18.017Z [INFO]  server-6e13ad29-8dda-2298-0ef1-dab3b401978a: entering leader state: leader="Node at 6e13ad29-8dda-2298-0ef1-dab3b401978a [Leader]"
2025-04-21T21:23:18.017Z [INFO]  server-6e13ad29-8dda-2298-0ef1-dab3b401978a: added peer, starting replication: peer=server-7c858cc2-b1c9-6b4c-9401-ed02cf8498d0
2025-04-21T21:23:18.017Z [INFO]  server-6e13ad29-8dda-2298-0ef1-dab3b401978a: added peer, starting replication: peer=server-e806fdd8-3c76-e0f2-538b-db56e7f5c976
2025-04-21T21:23:18.017Z [INFO]  server-6e13ad29-8dda-2298-0ef1-dab3b401978a: pipelining replication: peer="{Voter server-e806fdd8-3c76-e0f2-538b-db56e7f5c976 e806fdd8-3c76-e0f2-538b-db56e7f5c976}"
2025-04-21T21:23:18.017Z [INFO]  server-6e13ad29-8dda-2298-0ef1-dab3b401978a: pipelining replication: peer="{Voter server-7c858cc2-b1c9-6b4c-9401-ed02cf8498d0 7c858cc2-b1c9-6b4c-9401-ed02cf8498d0}"
2025-04-21T21:23:19.025Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:18.017498702 +0000 UTC m=+36.539158025 (1 nodes), 726.963005ms from start of poll, 727.119966ms from cluster start. Timeout at 2025-04-21 21:23:19.022500702 +0000 UTC m=+37.544160025, 1.005002s after stability
2025-04-21T21:23:19.025Z [INFO]  creating new snapshot: path=/tmp/raft3640317177/snapshots/2-14-1745270599025.tmp
2025-04-21T21:23:19.025Z [INFO]  server-6e13ad29-8dda-2298-0ef1-dab3b401978a: copied to local snapshot: bytes=71
2025-04-21T21:23:19.025Z [INFO]  server-6e13ad29-8dda-2298-0ef1-dab3b401978a: snapshot restore progress: id=2-14-1745270599025 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:19.025Z [INFO]  server-6e13ad29-8dda-2298-0ef1-dab3b401978a: restored user snapshot: index=14
2025-04-21T21:23:19.025Z [ERROR] server-6e13ad29-8dda-2298-0ef1-dab3b401978a: failed to get log: index=3 error="log not found"
2025-04-21T21:23:19.025Z [INFO]  server-6e13ad29-8dda-2298-0ef1-dab3b401978a: aborting pipeline replication: peer="{Voter server-7c858cc2-b1c9-6b4c-9401-ed02cf8498d0 7c858cc2-b1c9-6b4c-9401-ed02cf8498d0}"
2025-04-21T21:23:19.025Z [ERROR] server-6e13ad29-8dda-2298-0ef1-dab3b401978a: failed to get log: index=3 error="log not found"
2025-04-21T21:23:19.025Z [INFO]  server-6e13ad29-8dda-2298-0ef1-dab3b401978a: aborting pipeline replication: peer="{Voter server-e806fdd8-3c76-e0f2-538b-db56e7f5c976 e806fdd8-3c76-e0f2-538b-db56e7f5c976}"
2025-04-21T21:23:19.033Z [ERROR] server-6e13ad29-8dda-2298-0ef1-dab3b401978a: failed to get log: index=3 error="log not found"
2025-04-21T21:23:19.033Z [INFO]  creating new snapshot: path=/tmp/raft1419520973/snapshots/2-14-1745270599033.tmp
2025-04-21T21:23:19.033Z [INFO]  server-e806fdd8-3c76-e0f2-538b-db56e7f5c976: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:19.033Z [INFO]  server-e806fdd8-3c76-e0f2-538b-db56e7f5c976: copied to local snapshot: bytes=71
2025-04-21T21:23:19.033Z [INFO]  server-e806fdd8-3c76-e0f2-538b-db56e7f5c976: snapshot restore progress: id=2-14-1745270599033 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:19.033Z [INFO]  server-e806fdd8-3c76-e0f2-538b-db56e7f5c976: Installed remote snapshot
2025-04-21T21:23:19.033Z [INFO]  server-6e13ad29-8dda-2298-0ef1-dab3b401978a: pipelining replication: peer="{Voter server-e806fdd8-3c76-e0f2-538b-db56e7f5c976 e806fdd8-3c76-e0f2-538b-db56e7f5c976}"
2025-04-21T21:23:19.034Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:23:19.042Z [ERROR] server-6e13ad29-8dda-2298-0ef1-dab3b401978a: failed to get log: index=3 error="log not found"
2025-04-21T21:23:19.043Z [INFO]  creating new snapshot: path=/tmp/raft2448979527/snapshots/2-14-1745270599043.tmp
2025-04-21T21:23:19.043Z [INFO]  server-7c858cc2-b1c9-6b4c-9401-ed02cf8498d0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:19.043Z [INFO]  server-7c858cc2-b1c9-6b4c-9401-ed02cf8498d0: copied to local snapshot: bytes=71
2025-04-21T21:23:19.043Z [INFO]  server-7c858cc2-b1c9-6b4c-9401-ed02cf8498d0: snapshot restore progress: id=2-14-1745270599043 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:19.043Z [INFO]  server-7c858cc2-b1c9-6b4c-9401-ed02cf8498d0: Installed remote snapshot
2025-04-21T21:23:19.043Z [INFO]  server-6e13ad29-8dda-2298-0ef1-dab3b401978a: pipelining replication: peer="{Voter server-7c858cc2-b1c9-6b4c-9401-ed02cf8498d0 7c858cc2-b1c9-6b4c-9401-ed02cf8498d0}"
2025-04-21T21:23:20.041Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:23:19.034057374 +0000 UTC m=+37.555716697 (2 nodes), 480ns from start of poll, 1.743678638s from cluster start. Timeout at 2025-04-21 21:23:20.039066214 +0000 UTC m=+38.560725537, 1.00500884s after stability
2025-04-21T21:23:20.057Z [INFO]  server-d65668bf-8b99-dc82-a75c-6a0866269ff7: aborting pipeline replication: peer="{Voter server-8a9fd009-ac0c-be32-7842-7bc60f8a5157 8a9fd009-ac0c-be32-7842-7bc60f8a5157}"
2025-04-21T21:23:20.057Z [INFO]  server-d65668bf-8b99-dc82-a75c-6a0866269ff7: aborting pipeline replication: peer="{Voter server-4416e9d4-f6a8-c049-2a7b-6191beb5abcb 4416e9d4-f6a8-c049-2a7b-6191beb5abcb}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true
2025-04-21T21:23:20.057Z [INFO]  server-229b7814-00a2-4356-ada0-3999cd93b29e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-229b7814-00a2-4356-ada0-3999cd93b29e Address:229b7814-00a2-4356-ada0-3999cd93b29e} {Suffrage:Voter ID:server-ee3b47f8-bc0c-985e-187e-159a80a2f4c8 Address:ee3b47f8-bc0c-985e-187e-159a80a2f4c8} {Suffrage:Voter ID:server-b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed Address:b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed}]"
2025-04-21T21:23:20.057Z [INFO]  server-ee3b47f8-bc0c-985e-187e-159a80a2f4c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-229b7814-00a2-4356-ada0-3999cd93b29e Address:229b7814-00a2-4356-ada0-3999cd93b29e} {Suffrage:Voter ID:server-ee3b47f8-bc0c-985e-187e-159a80a2f4c8 Address:ee3b47f8-bc0c-985e-187e-159a80a2f4c8} {Suffrage:Voter ID:server-b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed Address:b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed}]"
2025-04-21T21:23:20.057Z [INFO]  server-b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-229b7814-00a2-4356-ada0-3999cd93b29e Address:229b7814-00a2-4356-ada0-3999cd93b29e} {Suffrage:Voter ID:server-ee3b47f8-bc0c-985e-187e-159a80a2f4c8 Address:ee3b47f8-bc0c-985e-187e-159a80a2f4c8} {Suffrage:Voter ID:server-b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed Address:b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed}]"
2025-04-21T21:23:20.057Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:20.057Z [INFO]  server-229b7814-00a2-4356-ada0-3999cd93b29e: entering follower state: follower="Node at 229b7814-00a2-4356-ada0-3999cd93b29e [Follower]" leader-address= leader-id=
2025-04-21T21:23:20.057Z [INFO]  server-ee3b47f8-bc0c-985e-187e-159a80a2f4c8: entering follower state: follower="Node at ee3b47f8-bc0c-985e-187e-159a80a2f4c8 [Follower]" leader-address= leader-id=
2025-04-21T21:23:20.057Z [INFO]  server-b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed: entering follower state: follower="Node at b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed [Follower]" leader-address= leader-id=
2025-04-21T21:23:20.641Z [WARN]  server-b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:20.641Z [INFO]  server-b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed: entering candidate state: node="Node at b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed [Candidate]" term=2
2025-04-21T21:23:20.641Z [INFO]  server-b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed: election won: term=2 tally=2
2025-04-21T21:23:20.641Z [INFO]  server-b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed: entering leader state: leader="Node at b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed [Leader]"
2025-04-21T21:23:20.641Z [INFO]  server-b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed: added peer, starting replication: peer=server-229b7814-00a2-4356-ada0-3999cd93b29e
2025-04-21T21:23:20.641Z [INFO]  server-b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed: added peer, starting replication: peer=server-ee3b47f8-bc0c-985e-187e-159a80a2f4c8
2025-04-21T21:23:20.641Z [INFO]  server-b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed: pipelining replication: peer="{Voter server-ee3b47f8-bc0c-985e-187e-159a80a2f4c8 ee3b47f8-bc0c-985e-187e-159a80a2f4c8}"
2025-04-21T21:23:20.641Z [INFO]  server-b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed: pipelining replication: peer="{Voter server-229b7814-00a2-4356-ada0-3999cd93b29e 229b7814-00a2-4356-ada0-3999cd93b29e}"
2025-04-21T21:23:21.649Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:20.641570417 +0000 UTC m=+39.163229740 (1 nodes), 583.735283ms from start of poll, 583.923724ms from cluster start. Timeout at 2025-04-21 21:23:21.646586538 +0000 UTC m=+40.168245861, 1.005016121s after stability
2025-04-21T21:23:21.649Z [INFO]  server-b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed: starting snapshot up to: index=12
2025-04-21T21:23:21.649Z [INFO]  creating new snapshot: path=/tmp/raft2003438223/snapshots/2-12-1745270601649.tmp
2025-04-21T21:23:21.649Z [INFO]  server-b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed: snapshot complete up to: index=12
2025-04-21T21:23:21.653Z [INFO]  server-3a5d0f73-34fb-d929-90a4-1037a989b7a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a5d0f73-34fb-d929-90a4-1037a989b7a5 Address:3a5d0f73-34fb-d929-90a4-1037a989b7a5} {Suffrage:Voter ID:server-eacef592-472c-4d9b-fe83-c2d117a8b8a1 Address:eacef592-472c-4d9b-fe83-c2d117a8b8a1} {Suffrage:Voter ID:server-459bae43-3b83-d16d-cd83-1281277dd996 Address:459bae43-3b83-d16d-cd83-1281277dd996}]"
2025-04-21T21:23:21.657Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a5d0f73-34fb-d929-90a4-1037a989b7a5 Address:3a5d0f73-34fb-d929-90a4-1037a989b7a5} {Suffrage:Voter ID:server-eacef592-472c-4d9b-fe83-c2d117a8b8a1 Address:eacef592-472c-4d9b-fe83-c2d117a8b8a1} {Suffrage:Voter ID:server-459bae43-3b83-d16d-cd83-1281277dd996 Address:459bae43-3b83-d16d-cd83-1281277dd996}]"
2025-04-21T21:23:21.657Z [INFO]  server-459bae43-3b83-d16d-cd83-1281277dd996: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a5d0f73-34fb-d929-90a4-1037a989b7a5 Address:3a5d0f73-34fb-d929-90a4-1037a989b7a5} {Suffrage:Voter ID:server-eacef592-472c-4d9b-fe83-c2d117a8b8a1 Address:eacef592-472c-4d9b-fe83-c2d117a8b8a1} {Suffrage:Voter ID:server-459bae43-3b83-d16d-cd83-1281277dd996 Address:459bae43-3b83-d16d-cd83-1281277dd996}]"
2025-04-21T21:23:21.657Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:21.657Z [INFO]  server-3a5d0f73-34fb-d929-90a4-1037a989b7a5: entering follower state: follower="Node at 3a5d0f73-34fb-d929-90a4-1037a989b7a5 [Follower]" leader-address= leader-id=
2025-04-21T21:23:21.657Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: entering follower state: follower="Node at eacef592-472c-4d9b-fe83-c2d117a8b8a1 [Follower]" leader-address= leader-id=
2025-04-21T21:23:21.657Z [INFO]  server-459bae43-3b83-d16d-cd83-1281277dd996: entering follower state: follower="Node at 459bae43-3b83-d16d-cd83-1281277dd996 [Follower]" leader-address= leader-id=
2025-04-21T21:23:22.193Z [WARN]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:22.193Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: entering candidate state: node="Node at eacef592-472c-4d9b-fe83-c2d117a8b8a1 [Candidate]" term=2
2025-04-21T21:23:22.193Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: election won: term=2 tally=2
2025-04-21T21:23:22.193Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: entering leader state: leader="Node at eacef592-472c-4d9b-fe83-c2d117a8b8a1 [Leader]"
2025-04-21T21:23:22.193Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: added peer, starting replication: peer=server-3a5d0f73-34fb-d929-90a4-1037a989b7a5
2025-04-21T21:23:22.193Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: added peer, starting replication: peer=server-459bae43-3b83-d16d-cd83-1281277dd996
2025-04-21T21:23:22.193Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: pipelining replication: peer="{Voter server-459bae43-3b83-d16d-cd83-1281277dd996 459bae43-3b83-d16d-cd83-1281277dd996}"
2025-04-21T21:23:22.193Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: pipelining replication: peer="{Voter server-3a5d0f73-34fb-d929-90a4-1037a989b7a5 3a5d0f73-34fb-d929-90a4-1037a989b7a5}"
2025-04-21T21:23:23.209Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:22.193581607 +0000 UTC m=+40.715240930 (1 nodes), 536.085323ms from start of poll, 539.937749ms from cluster start. Timeout at 2025-04-21 21:23:23.198585247 +0000 UTC m=+41.720244570, 1.00500364s after stability
2025-04-21T21:23:23.209Z [INFO]  creating new snapshot: path=/tmp/raft2577453743/snapshots/2-14-1745270603209.tmp
2025-04-21T21:23:23.209Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: copied to local snapshot: bytes=71
2025-04-21T21:23:23.209Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: snapshot restore progress: id=2-14-1745270603209 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:23.209Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: removing all old logs from log store
2025-04-21T21:23:23.209Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: compacting logs: from=1 to=2
2025-04-21T21:23:23.209Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: restored user snapshot: index=14
2025-04-21T21:23:23.209Z [ERROR] server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: failed to get log: index=2 error="log not found"
2025-04-21T21:23:23.209Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: aborting pipeline replication: peer="{Voter server-3a5d0f73-34fb-d929-90a4-1037a989b7a5 3a5d0f73-34fb-d929-90a4-1037a989b7a5}"
2025-04-21T21:23:23.209Z [ERROR] server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: failed to get log: index=2 error="log not found"
2025-04-21T21:23:23.209Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: aborting pipeline replication: peer="{Voter server-459bae43-3b83-d16d-cd83-1281277dd996 459bae43-3b83-d16d-cd83-1281277dd996}"
2025-04-21T21:23:23.209Z [ERROR] server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: failed to get log: index=2 error="log not found"
2025-04-21T21:23:23.209Z [INFO]  creating new snapshot: path=/tmp/raft3215364620/snapshots/2-14-1745270603209.tmp
2025-04-21T21:23:23.210Z [INFO]  server-459bae43-3b83-d16d-cd83-1281277dd996: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:23.210Z [INFO]  server-459bae43-3b83-d16d-cd83-1281277dd996: copied to local snapshot: bytes=71
2025-04-21T21:23:23.213Z [INFO]  server-459bae43-3b83-d16d-cd83-1281277dd996: snapshot restore progress: id=2-14-1745270603209 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:23.213Z [INFO]  server-459bae43-3b83-d16d-cd83-1281277dd996: removing all old logs from log store
2025-04-21T21:23:23.213Z [INFO]  server-459bae43-3b83-d16d-cd83-1281277dd996: compacting logs: from=1 to=2
2025-04-21T21:23:23.213Z [INFO]  server-459bae43-3b83-d16d-cd83-1281277dd996: Installed remote snapshot
2025-04-21T21:23:23.213Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: pipelining replication: peer="{Voter server-459bae43-3b83-d16d-cd83-1281277dd996 459bae43-3b83-d16d-cd83-1281277dd996}"
2025-04-21T21:23:23.213Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:23:23.221Z [ERROR] server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: failed to get log: index=2 error="log not found"
2025-04-21T21:23:23.221Z [INFO]  creating new snapshot: path=/tmp/raft1403404619/snapshots/2-14-1745270603221.tmp
2025-04-21T21:23:23.221Z [INFO]  server-3a5d0f73-34fb-d929-90a4-1037a989b7a5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:23.221Z [INFO]  server-3a5d0f73-34fb-d929-90a4-1037a989b7a5: copied to local snapshot: bytes=71
2025-04-21T21:23:23.221Z [INFO]  server-3a5d0f73-34fb-d929-90a4-1037a989b7a5: snapshot restore progress: id=2-14-1745270603221 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:23.221Z [INFO]  server-3a5d0f73-34fb-d929-90a4-1037a989b7a5: removing all old logs from log store
2025-04-21T21:23:23.221Z [INFO]  server-3a5d0f73-34fb-d929-90a4-1037a989b7a5: compacting logs: from=1 to=2
2025-04-21T21:23:23.221Z [INFO]  server-3a5d0f73-34fb-d929-90a4-1037a989b7a5: Installed remote snapshot
2025-04-21T21:23:23.221Z [INFO]  server-eacef592-472c-4d9b-fe83-c2d117a8b8a1: pipelining replication: peer="{Voter server-3a5d0f73-34fb-d929-90a4-1037a989b7a5 3a5d0f73-34fb-d929-90a4-1037a989b7a5}"
2025-04-21T21:23:24.225Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:23:23.213219419 +0000 UTC m=+41.734878702 (2 nodes), 880ns from start of poll, 1.559575521s from cluster start. Timeout at 2025-04-21 21:23:24.218236819 +0000 UTC m=+42.739896142, 1.00501744s after stability
2025-04-21T21:23:24.245Z [INFO]  server-b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed: aborting pipeline replication: peer="{Voter server-ee3b47f8-bc0c-985e-187e-159a80a2f4c8 ee3b47f8-bc0c-985e-187e-159a80a2f4c8}"
2025-04-21T21:23:24.245Z [INFO]  server-b62e4b08-2dcb-ae68-b8cd-fa480f81e2ed: aborting pipeline replication: peer="{Voter server-229b7814-00a2-4356-ada0-3999cd93b29e 229b7814-00a2-4356-ada0-3999cd93b29e}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false
2025-04-21T21:23:24.245Z [INFO]  server-1b5bf923-bae4-f99c-7427-e3854649baa8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b5bf923-bae4-f99c-7427-e3854649baa8 Address:1b5bf923-bae4-f99c-7427-e3854649baa8} {Suffrage:Voter ID:server-563a7ba1-8ad4-a020-664f-2634554a2440 Address:563a7ba1-8ad4-a020-664f-2634554a2440} {Suffrage:Voter ID:server-b919c1b1-ad54-9d39-7aa3-78728aec9190 Address:b919c1b1-ad54-9d39-7aa3-78728aec9190}]"
2025-04-21T21:23:24.245Z [INFO]  server-563a7ba1-8ad4-a020-664f-2634554a2440: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b5bf923-bae4-f99c-7427-e3854649baa8 Address:1b5bf923-bae4-f99c-7427-e3854649baa8} {Suffrage:Voter ID:server-563a7ba1-8ad4-a020-664f-2634554a2440 Address:563a7ba1-8ad4-a020-664f-2634554a2440} {Suffrage:Voter ID:server-b919c1b1-ad54-9d39-7aa3-78728aec9190 Address:b919c1b1-ad54-9d39-7aa3-78728aec9190}]"
2025-04-21T21:23:24.245Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b5bf923-bae4-f99c-7427-e3854649baa8 Address:1b5bf923-bae4-f99c-7427-e3854649baa8} {Suffrage:Voter ID:server-563a7ba1-8ad4-a020-664f-2634554a2440 Address:563a7ba1-8ad4-a020-664f-2634554a2440} {Suffrage:Voter ID:server-b919c1b1-ad54-9d39-7aa3-78728aec9190 Address:b919c1b1-ad54-9d39-7aa3-78728aec9190}]"
2025-04-21T21:23:24.245Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:24.245Z [INFO]  server-1b5bf923-bae4-f99c-7427-e3854649baa8: entering follower state: follower="Node at 1b5bf923-bae4-f99c-7427-e3854649baa8 [Follower]" leader-address= leader-id=
2025-04-21T21:23:24.245Z [INFO]  server-563a7ba1-8ad4-a020-664f-2634554a2440: entering follower state: follower="Node at 563a7ba1-8ad4-a020-664f-2634554a2440 [Follower]" leader-address= leader-id=
2025-04-21T21:23:24.245Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: entering follower state: follower="Node at b919c1b1-ad54-9d39-7aa3-78728aec9190 [Follower]" leader-address= leader-id=
2025-04-21T21:23:25.061Z [WARN]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:25.061Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: entering candidate state: node="Node at b919c1b1-ad54-9d39-7aa3-78728aec9190 [Candidate]" term=2
2025-04-21T21:23:25.061Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: election won: term=2 tally=2
2025-04-21T21:23:25.061Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: entering leader state: leader="Node at b919c1b1-ad54-9d39-7aa3-78728aec9190 [Leader]"
2025-04-21T21:23:25.061Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: added peer, starting replication: peer=server-1b5bf923-bae4-f99c-7427-e3854649baa8
2025-04-21T21:23:25.061Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: added peer, starting replication: peer=server-563a7ba1-8ad4-a020-664f-2634554a2440
2025-04-21T21:23:25.061Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: pipelining replication: peer="{Voter server-563a7ba1-8ad4-a020-664f-2634554a2440 563a7ba1-8ad4-a020-664f-2634554a2440}"
2025-04-21T21:23:25.061Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: pipelining replication: peer="{Voter server-1b5bf923-bae4-f99c-7427-e3854649baa8 1b5bf923-bae4-f99c-7427-e3854649baa8}"
2025-04-21T21:23:26.069Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:25.061562641 +0000 UTC m=+43.583221924 (1 nodes), 815.805083ms from start of poll, 815.983684ms from cluster start. Timeout at 2025-04-21 21:23:26.066565521 +0000 UTC m=+44.588224844, 1.00500292s after stability
2025-04-21T21:23:26.069Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: starting snapshot up to: index=12
2025-04-21T21:23:26.069Z [INFO]  creating new snapshot: path=/tmp/raft3930760737/snapshots/2-12-1745270606069.tmp
2025-04-21T21:23:26.069Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: snapshot complete up to: index=12
2025-04-21T21:23:26.070Z [INFO]  creating new snapshot: path=/tmp/raft3930760737/snapshots/2-23-1745270606070.tmp
2025-04-21T21:23:26.070Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: copied to local snapshot: bytes=71
2025-04-21T21:23:26.070Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: snapshot restore progress: id=2-23-1745270606070 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:26.070Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: restored user snapshot: index=23
2025-04-21T21:23:26.070Z [ERROR] server-b919c1b1-ad54-9d39-7aa3-78728aec9190: failed to get log: index=23 error="log not found"
2025-04-21T21:23:26.070Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: aborting pipeline replication: peer="{Voter server-563a7ba1-8ad4-a020-664f-2634554a2440 563a7ba1-8ad4-a020-664f-2634554a2440}"
2025-04-21T21:23:26.070Z [ERROR] server-b919c1b1-ad54-9d39-7aa3-78728aec9190: failed to get log: index=23 error="log not found"
2025-04-21T21:23:26.070Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: aborting pipeline replication: peer="{Voter server-1b5bf923-bae4-f99c-7427-e3854649baa8 1b5bf923-bae4-f99c-7427-e3854649baa8}"
2025-04-21T21:23:26.081Z [ERROR] server-b919c1b1-ad54-9d39-7aa3-78728aec9190: failed to get log: index=23 error="log not found"
2025-04-21T21:23:26.081Z [ERROR] server-b919c1b1-ad54-9d39-7aa3-78728aec9190: failed to get log: index=23 error="log not found"
2025-04-21T21:23:26.081Z [INFO]  creating new snapshot: path=/tmp/raft1593825095/snapshots/2-23-1745270606081.tmp
2025-04-21T21:23:26.081Z [INFO]  creating new snapshot: path=/tmp/raft2792339679/snapshots/2-23-1745270606081.tmp
2025-04-21T21:23:26.081Z [INFO]  server-563a7ba1-8ad4-a020-664f-2634554a2440: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:26.081Z [INFO]  server-1b5bf923-bae4-f99c-7427-e3854649baa8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:26.081Z [INFO]  server-563a7ba1-8ad4-a020-664f-2634554a2440: copied to local snapshot: bytes=71
2025-04-21T21:23:26.081Z [INFO]  server-1b5bf923-bae4-f99c-7427-e3854649baa8: copied to local snapshot: bytes=71
2025-04-21T21:23:26.089Z [INFO]  server-563a7ba1-8ad4-a020-664f-2634554a2440: snapshot restore progress: id=2-23-1745270606081 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:26.089Z [INFO]  server-563a7ba1-8ad4-a020-664f-2634554a2440: Installed remote snapshot
2025-04-21T21:23:26.089Z [WARN]  server-563a7ba1-8ad4-a020-664f-2634554a2440: skipping application of old log: index=22
2025-04-21T21:23:26.089Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: pipelining replication: peer="{Voter server-563a7ba1-8ad4-a020-664f-2634554a2440 563a7ba1-8ad4-a020-664f-2634554a2440}"
2025-04-21T21:23:26.089Z [INFO]  server-1b5bf923-bae4-f99c-7427-e3854649baa8: snapshot restore progress: id=2-23-1745270606081 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:26.089Z [INFO]  server-1b5bf923-bae4-f99c-7427-e3854649baa8: Installed remote snapshot
2025-04-21T21:23:26.089Z [WARN]  server-1b5bf923-bae4-f99c-7427-e3854649baa8: skipping application of old log: index=22
2025-04-21T21:23:26.089Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: pipelining replication: peer="{Voter server-1b5bf923-bae4-f99c-7427-e3854649baa8 1b5bf923-bae4-f99c-7427-e3854649baa8}"
2025-04-21T21:23:26.089Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:23:27.097Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:23:26.089536269 +0000 UTC m=+44.611195592 (2 nodes), 680ns from start of poll, 1.843957352s from cluster start. Timeout at 2025-04-21 21:23:27.094552509 +0000 UTC m=+45.616211832, 1.00501624s after stability
2025-04-21T21:23:27.113Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: aborting pipeline replication: peer="{Voter server-563a7ba1-8ad4-a020-664f-2634554a2440 563a7ba1-8ad4-a020-664f-2634554a2440}"
2025-04-21T21:23:27.113Z [INFO]  server-b919c1b1-ad54-9d39-7aa3-78728aec9190: aborting pipeline replication: peer="{Voter server-1b5bf923-bae4-f99c-7427-e3854649baa8 1b5bf923-bae4-f99c-7427-e3854649baa8}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false
2025-04-21T21:23:27.113Z [INFO]  server-4315932c-8d1e-a068-823d-6cd983c90138: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4315932c-8d1e-a068-823d-6cd983c90138 Address:4315932c-8d1e-a068-823d-6cd983c90138} {Suffrage:Voter ID:server-4509971a-838f-ad40-cf11-c85227c9d184 Address:4509971a-838f-ad40-cf11-c85227c9d184} {Suffrage:Voter ID:server-89a25f68-d39b-7d82-b050-679c081a35f9 Address:89a25f68-d39b-7d82-b050-679c081a35f9}]"
2025-04-21T21:23:27.113Z [INFO]  server-4509971a-838f-ad40-cf11-c85227c9d184: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4315932c-8d1e-a068-823d-6cd983c90138 Address:4315932c-8d1e-a068-823d-6cd983c90138} {Suffrage:Voter ID:server-4509971a-838f-ad40-cf11-c85227c9d184 Address:4509971a-838f-ad40-cf11-c85227c9d184} {Suffrage:Voter ID:server-89a25f68-d39b-7d82-b050-679c081a35f9 Address:89a25f68-d39b-7d82-b050-679c081a35f9}]"
2025-04-21T21:23:27.113Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4315932c-8d1e-a068-823d-6cd983c90138 Address:4315932c-8d1e-a068-823d-6cd983c90138} {Suffrage:Voter ID:server-4509971a-838f-ad40-cf11-c85227c9d184 Address:4509971a-838f-ad40-cf11-c85227c9d184} {Suffrage:Voter ID:server-89a25f68-d39b-7d82-b050-679c081a35f9 Address:89a25f68-d39b-7d82-b050-679c081a35f9}]"
2025-04-21T21:23:27.113Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:27.113Z [INFO]  server-4315932c-8d1e-a068-823d-6cd983c90138: entering follower state: follower="Node at 4315932c-8d1e-a068-823d-6cd983c90138 [Follower]" leader-address= leader-id=
2025-04-21T21:23:27.113Z [INFO]  server-4509971a-838f-ad40-cf11-c85227c9d184: entering follower state: follower="Node at 4509971a-838f-ad40-cf11-c85227c9d184 [Follower]" leader-address= leader-id=
2025-04-21T21:23:27.113Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: entering follower state: follower="Node at 89a25f68-d39b-7d82-b050-679c081a35f9 [Follower]" leader-address= leader-id=
2025-04-21T21:23:27.657Z [WARN]  server-89a25f68-d39b-7d82-b050-679c081a35f9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:27.657Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: entering candidate state: node="Node at 89a25f68-d39b-7d82-b050-679c081a35f9 [Candidate]" term=2
2025-04-21T21:23:27.657Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: election won: term=2 tally=2
2025-04-21T21:23:27.657Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: entering leader state: leader="Node at 89a25f68-d39b-7d82-b050-679c081a35f9 [Leader]"
2025-04-21T21:23:27.657Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: added peer, starting replication: peer=server-4315932c-8d1e-a068-823d-6cd983c90138
2025-04-21T21:23:27.657Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: added peer, starting replication: peer=server-4509971a-838f-ad40-cf11-c85227c9d184
2025-04-21T21:23:27.657Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: pipelining replication: peer="{Voter server-4509971a-838f-ad40-cf11-c85227c9d184 4509971a-838f-ad40-cf11-c85227c9d184}"
2025-04-21T21:23:27.657Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: pipelining replication: peer="{Voter server-4315932c-8d1e-a068-823d-6cd983c90138 4315932c-8d1e-a068-823d-6cd983c90138}"
2025-04-21T21:23:28.665Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:27.657554807 +0000 UTC m=+46.179214130 (1 nodes), 543.681694ms from start of poll, 543.849495ms from cluster start. Timeout at 2025-04-21 21:23:28.662556727 +0000 UTC m=+47.184216050, 1.00500192s after stability
2025-04-21T21:23:28.665Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: starting snapshot up to: index=12
2025-04-21T21:23:28.665Z [INFO]  creating new snapshot: path=/tmp/raft2138050989/snapshots/2-12-1745270608665.tmp
2025-04-21T21:23:28.666Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: snapshot complete up to: index=12
2025-04-21T21:23:28.666Z [INFO]  creating new snapshot: path=/tmp/raft2138050989/snapshots/2-23-1745270608666.tmp
2025-04-21T21:23:28.666Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: copied to local snapshot: bytes=71
2025-04-21T21:23:28.666Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: snapshot restore progress: id=2-23-1745270608666 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:28.666Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: removing all old logs from log store
2025-04-21T21:23:28.666Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: compacting logs: from=1 to=22
2025-04-21T21:23:28.666Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: restored user snapshot: index=23
2025-04-21T21:23:28.666Z [ERROR] server-89a25f68-d39b-7d82-b050-679c081a35f9: failed to get log: index=22 error="log not found"
2025-04-21T21:23:28.666Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: aborting pipeline replication: peer="{Voter server-4509971a-838f-ad40-cf11-c85227c9d184 4509971a-838f-ad40-cf11-c85227c9d184}"
2025-04-21T21:23:28.666Z [ERROR] server-89a25f68-d39b-7d82-b050-679c081a35f9: failed to get log: index=22 error="log not found"
2025-04-21T21:23:28.666Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: aborting pipeline replication: peer="{Voter server-4315932c-8d1e-a068-823d-6cd983c90138 4315932c-8d1e-a068-823d-6cd983c90138}"
2025-04-21T21:23:28.673Z [ERROR] server-89a25f68-d39b-7d82-b050-679c081a35f9: failed to get log: index=22 error="log not found"
2025-04-21T21:23:28.673Z [INFO]  creating new snapshot: path=/tmp/raft2246312901/snapshots/2-23-1745270608673.tmp
2025-04-21T21:23:28.673Z [INFO]  server-4315932c-8d1e-a068-823d-6cd983c90138: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:28.673Z [INFO]  server-4315932c-8d1e-a068-823d-6cd983c90138: copied to local snapshot: bytes=71
2025-04-21T21:23:28.673Z [INFO]  server-4315932c-8d1e-a068-823d-6cd983c90138: snapshot restore progress: id=2-23-1745270608673 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:28.673Z [INFO]  server-4315932c-8d1e-a068-823d-6cd983c90138: removing all old logs from log store
2025-04-21T21:23:28.673Z [INFO]  server-4315932c-8d1e-a068-823d-6cd983c90138: compacting logs: from=1 to=22
2025-04-21T21:23:28.673Z [INFO]  server-4315932c-8d1e-a068-823d-6cd983c90138: Installed remote snapshot
2025-04-21T21:23:28.673Z [WARN]  server-4315932c-8d1e-a068-823d-6cd983c90138: skipping application of old log: index=22
2025-04-21T21:23:28.673Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: pipelining replication: peer="{Voter server-4315932c-8d1e-a068-823d-6cd983c90138 4315932c-8d1e-a068-823d-6cd983c90138}"
2025-04-21T21:23:28.673Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:23:28.681Z [ERROR] server-89a25f68-d39b-7d82-b050-679c081a35f9: failed to get log: index=22 error="log not found"
2025-04-21T21:23:28.681Z [INFO]  creating new snapshot: path=/tmp/raft1283333238/snapshots/2-23-1745270608681.tmp
2025-04-21T21:23:28.681Z [INFO]  server-4509971a-838f-ad40-cf11-c85227c9d184: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:28.681Z [INFO]  server-4509971a-838f-ad40-cf11-c85227c9d184: copied to local snapshot: bytes=71
2025-04-21T21:23:28.681Z [INFO]  server-4509971a-838f-ad40-cf11-c85227c9d184: snapshot restore progress: id=2-23-1745270608681 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:28.682Z [INFO]  server-4509971a-838f-ad40-cf11-c85227c9d184: removing all old logs from log store
2025-04-21T21:23:28.682Z [INFO]  server-4509971a-838f-ad40-cf11-c85227c9d184: compacting logs: from=1 to=22
2025-04-21T21:23:28.682Z [INFO]  server-4509971a-838f-ad40-cf11-c85227c9d184: Installed remote snapshot
2025-04-21T21:23:28.682Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: pipelining replication: peer="{Voter server-4509971a-838f-ad40-cf11-c85227c9d184 4509971a-838f-ad40-cf11-c85227c9d184}"
2025-04-21T21:23:29.681Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:23:28.673987838 +0000 UTC m=+47.195647161 (2 nodes), 520ns from start of poll, 1.560282526s from cluster start. Timeout at 2025-04-21 21:23:29.679006278 +0000 UTC m=+48.200665601, 1.00501844s after stability
2025-04-21T21:23:29.697Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: aborting pipeline replication: peer="{Voter server-4315932c-8d1e-a068-823d-6cd983c90138 4315932c-8d1e-a068-823d-6cd983c90138}"
2025-04-21T21:23:29.697Z [INFO]  server-89a25f68-d39b-7d82-b050-679c081a35f9: aborting pipeline replication: peer="{Voter server-4509971a-838f-ad40-cf11-c85227c9d184 4509971a-838f-ad40-cf11-c85227c9d184}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true
2025-04-21T21:23:29.697Z [INFO]  server-5c2d00dc-a3e4-3f8d-f847-5d17f3704d63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c2d00dc-a3e4-3f8d-f847-5d17f3704d63 Address:5c2d00dc-a3e4-3f8d-f847-5d17f3704d63} {Suffrage:Voter ID:server-b7f631ee-153f-bb85-404d-5e548fa2f360 Address:b7f631ee-153f-bb85-404d-5e548fa2f360} {Suffrage:Voter ID:server-41e1ea2a-a306-7925-9b17-d7e1ce2dd045 Address:41e1ea2a-a306-7925-9b17-d7e1ce2dd045}]"
2025-04-21T21:23:29.697Z [INFO]  server-b7f631ee-153f-bb85-404d-5e548fa2f360: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c2d00dc-a3e4-3f8d-f847-5d17f3704d63 Address:5c2d00dc-a3e4-3f8d-f847-5d17f3704d63} {Suffrage:Voter ID:server-b7f631ee-153f-bb85-404d-5e548fa2f360 Address:b7f631ee-153f-bb85-404d-5e548fa2f360} {Suffrage:Voter ID:server-41e1ea2a-a306-7925-9b17-d7e1ce2dd045 Address:41e1ea2a-a306-7925-9b17-d7e1ce2dd045}]"
2025-04-21T21:23:29.697Z [INFO]  server-41e1ea2a-a306-7925-9b17-d7e1ce2dd045: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5c2d00dc-a3e4-3f8d-f847-5d17f3704d63 Address:5c2d00dc-a3e4-3f8d-f847-5d17f3704d63} {Suffrage:Voter ID:server-b7f631ee-153f-bb85-404d-5e548fa2f360 Address:b7f631ee-153f-bb85-404d-5e548fa2f360} {Suffrage:Voter ID:server-41e1ea2a-a306-7925-9b17-d7e1ce2dd045 Address:41e1ea2a-a306-7925-9b17-d7e1ce2dd045}]"
2025-04-21T21:23:29.697Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:29.697Z [INFO]  server-5c2d00dc-a3e4-3f8d-f847-5d17f3704d63: entering follower state: follower="Node at 5c2d00dc-a3e4-3f8d-f847-5d17f3704d63 [Follower]" leader-address= leader-id=
2025-04-21T21:23:29.697Z [INFO]  server-b7f631ee-153f-bb85-404d-5e548fa2f360: entering follower state: follower="Node at b7f631ee-153f-bb85-404d-5e548fa2f360 [Follower]" leader-address= leader-id=
2025-04-21T21:23:29.697Z [INFO]  server-41e1ea2a-a306-7925-9b17-d7e1ce2dd045: entering follower state: follower="Node at 41e1ea2a-a306-7925-9b17-d7e1ce2dd045 [Follower]" leader-address= leader-id=
2025-04-21T21:23:30.321Z [WARN]  server-b7f631ee-153f-bb85-404d-5e548fa2f360: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:30.321Z [INFO]  server-b7f631ee-153f-bb85-404d-5e548fa2f360: entering candidate state: node="Node at b7f631ee-153f-bb85-404d-5e548fa2f360 [Candidate]" term=2
2025-04-21T21:23:30.321Z [INFO]  server-b7f631ee-153f-bb85-404d-5e548fa2f360: election won: term=2 tally=2
2025-04-21T21:23:30.321Z [INFO]  server-b7f631ee-153f-bb85-404d-5e548fa2f360: entering leader state: leader="Node at b7f631ee-153f-bb85-404d-5e548fa2f360 [Leader]"
2025-04-21T21:23:30.321Z [INFO]  server-b7f631ee-153f-bb85-404d-5e548fa2f360: added peer, starting replication: peer=server-5c2d00dc-a3e4-3f8d-f847-5d17f3704d63
2025-04-21T21:23:30.321Z [INFO]  server-b7f631ee-153f-bb85-404d-5e548fa2f360: added peer, starting replication: peer=server-41e1ea2a-a306-7925-9b17-d7e1ce2dd045
2025-04-21T21:23:30.321Z [INFO]  server-b7f631ee-153f-bb85-404d-5e548fa2f360: pipelining replication: peer="{Voter server-41e1ea2a-a306-7925-9b17-d7e1ce2dd045 41e1ea2a-a306-7925-9b17-d7e1ce2dd045}"
2025-04-21T21:23:30.321Z [INFO]  server-b7f631ee-153f-bb85-404d-5e548fa2f360: pipelining replication: peer="{Voter server-5c2d00dc-a3e4-3f8d-f847-5d17f3704d63 5c2d00dc-a3e4-3f8d-f847-5d17f3704d63}"
2025-04-21T21:23:31.329Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:30.321277228 +0000 UTC m=+48.842936551 (1 nodes), 623.423229ms from start of poll, 623.595231ms from cluster start. Timeout at 2025-04-21 21:23:31.326278708 +0000 UTC m=+49.847938031, 1.00500148s after stability
2025-04-21T21:23:31.329Z [INFO]  server-b7f631ee-153f-bb85-404d-5e548fa2f360: starting snapshot up to: index=12
2025-04-21T21:23:31.329Z [INFO]  creating new snapshot: path=/tmp/raft2989742958/snapshots/2-12-1745270611329.tmp
2025-04-21T21:23:31.330Z [INFO]  server-b7f631ee-153f-bb85-404d-5e548fa2f360: snapshot complete up to: index=12
2025-04-21T21:23:31.330Z [INFO]  server-8f6f1638-8bed-d595-56ed-588e7d11749a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f6f1638-8bed-d595-56ed-588e7d11749a Address:8f6f1638-8bed-d595-56ed-588e7d11749a} {Suffrage:Voter ID:server-7b5a4829-2b04-c01a-11dc-ea1426650267 Address:7b5a4829-2b04-c01a-11dc-ea1426650267} {Suffrage:Voter ID:server-6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3 Address:6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3}]"
2025-04-21T21:23:31.330Z [INFO]  server-7b5a4829-2b04-c01a-11dc-ea1426650267: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f6f1638-8bed-d595-56ed-588e7d11749a Address:8f6f1638-8bed-d595-56ed-588e7d11749a} {Suffrage:Voter ID:server-7b5a4829-2b04-c01a-11dc-ea1426650267 Address:7b5a4829-2b04-c01a-11dc-ea1426650267} {Suffrage:Voter ID:server-6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3 Address:6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3}]"
2025-04-21T21:23:31.330Z [INFO]  server-6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f6f1638-8bed-d595-56ed-588e7d11749a Address:8f6f1638-8bed-d595-56ed-588e7d11749a} {Suffrage:Voter ID:server-7b5a4829-2b04-c01a-11dc-ea1426650267 Address:7b5a4829-2b04-c01a-11dc-ea1426650267} {Suffrage:Voter ID:server-6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3 Address:6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3}]"
2025-04-21T21:23:31.330Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:31.330Z [INFO]  server-8f6f1638-8bed-d595-56ed-588e7d11749a: entering follower state: follower="Node at 8f6f1638-8bed-d595-56ed-588e7d11749a [Follower]" leader-address= leader-id=
2025-04-21T21:23:31.330Z [INFO]  server-7b5a4829-2b04-c01a-11dc-ea1426650267: entering follower state: follower="Node at 7b5a4829-2b04-c01a-11dc-ea1426650267 [Follower]" leader-address= leader-id=
2025-04-21T21:23:31.330Z [INFO]  server-6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3: entering follower state: follower="Node at 6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3 [Follower]" leader-address= leader-id=
2025-04-21T21:23:32.017Z [WARN]  server-7b5a4829-2b04-c01a-11dc-ea1426650267: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:32.017Z [INFO]  server-7b5a4829-2b04-c01a-11dc-ea1426650267: entering candidate state: node="Node at 7b5a4829-2b04-c01a-11dc-ea1426650267 [Candidate]" term=2
2025-04-21T21:23:32.017Z [INFO]  server-7b5a4829-2b04-c01a-11dc-ea1426650267: election won: term=2 tally=2
2025-04-21T21:23:32.017Z [INFO]  server-7b5a4829-2b04-c01a-11dc-ea1426650267: entering leader state: leader="Node at 7b5a4829-2b04-c01a-11dc-ea1426650267 [Leader]"
2025-04-21T21:23:32.017Z [INFO]  server-7b5a4829-2b04-c01a-11dc-ea1426650267: added peer, starting replication: peer=server-8f6f1638-8bed-d595-56ed-588e7d11749a
2025-04-21T21:23:32.017Z [INFO]  server-7b5a4829-2b04-c01a-11dc-ea1426650267: added peer, starting replication: peer=server-6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3
2025-04-21T21:23:32.017Z [INFO]  server-7b5a4829-2b04-c01a-11dc-ea1426650267: pipelining replication: peer="{Voter server-6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3 6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3}"
2025-04-21T21:23:32.017Z [INFO]  server-7b5a4829-2b04-c01a-11dc-ea1426650267: pipelining replication: peer="{Voter server-8f6f1638-8bed-d595-56ed-588e7d11749a 8f6f1638-8bed-d595-56ed-588e7d11749a}"
2025-04-21T21:23:33.025Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:32.017573348 +0000 UTC m=+50.539232671 (1 nodes), 686.863496ms from start of poll, 687.027897ms from cluster start. Timeout at 2025-04-21 21:23:33.022575428 +0000 UTC m=+51.544234751, 1.00500208s after stability
2025-04-21T21:23:33.025Z [INFO]  creating new snapshot: path=/tmp/raft3983305347/snapshots/2-15-1745270613025.tmp
2025-04-21T21:23:33.025Z [INFO]  server-7b5a4829-2b04-c01a-11dc-ea1426650267: copied to local snapshot: bytes=71
2025-04-21T21:23:33.025Z [INFO]  server-7b5a4829-2b04-c01a-11dc-ea1426650267: snapshot restore progress: id=2-15-1745270613025 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:33.025Z [INFO]  server-7b5a4829-2b04-c01a-11dc-ea1426650267: restored user snapshot: index=15
2025-04-21T21:23:33.025Z [ERROR] server-7b5a4829-2b04-c01a-11dc-ea1426650267: failed to get log: index=3 error="log not found"
2025-04-21T21:23:33.025Z [INFO]  server-7b5a4829-2b04-c01a-11dc-ea1426650267: aborting pipeline replication: peer="{Voter server-6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3 6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3}"
2025-04-21T21:23:33.025Z [ERROR] server-7b5a4829-2b04-c01a-11dc-ea1426650267: failed to get log: index=3 error="log not found"
2025-04-21T21:23:33.025Z [INFO]  server-7b5a4829-2b04-c01a-11dc-ea1426650267: aborting pipeline replication: peer="{Voter server-8f6f1638-8bed-d595-56ed-588e7d11749a 8f6f1638-8bed-d595-56ed-588e7d11749a}"
2025-04-21T21:23:33.037Z [ERROR] server-7b5a4829-2b04-c01a-11dc-ea1426650267: failed to get log: index=3 error="log not found"
2025-04-21T21:23:33.037Z [INFO]  creating new snapshot: path=/tmp/raft848928778/snapshots/2-15-1745270613037.tmp
2025-04-21T21:23:33.037Z [INFO]  server-8f6f1638-8bed-d595-56ed-588e7d11749a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:33.037Z [INFO]  server-8f6f1638-8bed-d595-56ed-588e7d11749a: copied to local snapshot: bytes=71
2025-04-21T21:23:33.037Z [INFO]  server-8f6f1638-8bed-d595-56ed-588e7d11749a: snapshot restore progress: id=2-15-1745270613037 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:33.037Z [INFO]  server-8f6f1638-8bed-d595-56ed-588e7d11749a: Installed remote snapshot
2025-04-21T21:23:33.037Z [INFO]  server-7b5a4829-2b04-c01a-11dc-ea1426650267: pipelining replication: peer="{Voter server-8f6f1638-8bed-d595-56ed-588e7d11749a 8f6f1638-8bed-d595-56ed-588e7d11749a}"
2025-04-21T21:23:33.037Z [ERROR] server-7b5a4829-2b04-c01a-11dc-ea1426650267: failed to get log: index=3 error="log not found"
2025-04-21T21:23:33.038Z [INFO]  creating new snapshot: path=/tmp/raft3218809791/snapshots/2-15-1745270613038.tmp
2025-04-21T21:23:33.038Z [INFO]  server-6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:33.038Z [INFO]  server-6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3: copied to local snapshot: bytes=71
2025-04-21T21:23:33.038Z [INFO]  server-6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3: snapshot restore progress: id=2-15-1745270613038 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:33.038Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:23:33.038Z [INFO]  server-6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3: Installed remote snapshot
2025-04-21T21:23:33.038Z [INFO]  server-7b5a4829-2b04-c01a-11dc-ea1426650267: pipelining replication: peer="{Voter server-6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3 6bd892b9-1fba-3a46-3c6e-0f8107a2f7d3}"
2025-04-21T21:23:34.045Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:23:33.038377929 +0000 UTC m=+51.560037252 (2 nodes), 400ns from start of poll, 1.707832478s from cluster start. Timeout at 2025-04-21 21:23:34.043402609 +0000 UTC m=+52.565061932, 1.00502468s after stability
2025-04-21T21:23:34.061Z [INFO]  server-b7f631ee-153f-bb85-404d-5e548fa2f360: aborting pipeline replication: peer="{Voter server-41e1ea2a-a306-7925-9b17-d7e1ce2dd045 41e1ea2a-a306-7925-9b17-d7e1ce2dd045}"
2025-04-21T21:23:34.061Z [INFO]  server-b7f631ee-153f-bb85-404d-5e548fa2f360: aborting pipeline replication: peer="{Voter server-5c2d00dc-a3e4-3f8d-f847-5d17f3704d63 5c2d00dc-a3e4-3f8d-f847-5d17f3704d63}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true
2025-04-21T21:23:34.061Z [INFO]  server-5384fba3-2ac5-1ad2-4b16-5a8a751021d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5384fba3-2ac5-1ad2-4b16-5a8a751021d3 Address:5384fba3-2ac5-1ad2-4b16-5a8a751021d3} {Suffrage:Voter ID:server-95cc252f-49f7-3920-d082-34689e8f5bd6 Address:95cc252f-49f7-3920-d082-34689e8f5bd6} {Suffrage:Voter ID:server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff Address:c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff}]"
2025-04-21T21:23:34.061Z [INFO]  server-95cc252f-49f7-3920-d082-34689e8f5bd6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5384fba3-2ac5-1ad2-4b16-5a8a751021d3 Address:5384fba3-2ac5-1ad2-4b16-5a8a751021d3} {Suffrage:Voter ID:server-95cc252f-49f7-3920-d082-34689e8f5bd6 Address:95cc252f-49f7-3920-d082-34689e8f5bd6} {Suffrage:Voter ID:server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff Address:c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff}]"
2025-04-21T21:23:34.061Z [INFO]  server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5384fba3-2ac5-1ad2-4b16-5a8a751021d3 Address:5384fba3-2ac5-1ad2-4b16-5a8a751021d3} {Suffrage:Voter ID:server-95cc252f-49f7-3920-d082-34689e8f5bd6 Address:95cc252f-49f7-3920-d082-34689e8f5bd6} {Suffrage:Voter ID:server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff Address:c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff}]"
2025-04-21T21:23:34.061Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:34.061Z [INFO]  server-5384fba3-2ac5-1ad2-4b16-5a8a751021d3: entering follower state: follower="Node at 5384fba3-2ac5-1ad2-4b16-5a8a751021d3 [Follower]" leader-address= leader-id=
2025-04-21T21:23:34.061Z [INFO]  server-95cc252f-49f7-3920-d082-34689e8f5bd6: entering follower state: follower="Node at 95cc252f-49f7-3920-d082-34689e8f5bd6 [Follower]" leader-address= leader-id=
2025-04-21T21:23:34.061Z [INFO]  server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff: entering follower state: follower="Node at c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff [Follower]" leader-address= leader-id=
2025-04-21T21:23:34.829Z [WARN]  server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:34.829Z [INFO]  server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff: entering candidate state: node="Node at c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff [Candidate]" term=2
2025-04-21T21:23:34.829Z [WARN]  server-95cc252f-49f7-3920-d082-34689e8f5bd6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:34.829Z [INFO]  server-95cc252f-49f7-3920-d082-34689e8f5bd6: entering candidate state: node="Node at 95cc252f-49f7-3920-d082-34689e8f5bd6 [Candidate]" term=2
2025-04-21T21:23:34.829Z [INFO]  server-95cc252f-49f7-3920-d082-34689e8f5bd6: duplicate requestVote for same term: term=2
2025-04-21T21:23:34.829Z [INFO]  server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff: election won: term=2 tally=2
2025-04-21T21:23:34.829Z [INFO]  server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff: entering leader state: leader="Node at c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff [Leader]"
2025-04-21T21:23:34.829Z [INFO]  server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff: added peer, starting replication: peer=server-5384fba3-2ac5-1ad2-4b16-5a8a751021d3
2025-04-21T21:23:34.829Z [INFO]  server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff: added peer, starting replication: peer=server-95cc252f-49f7-3920-d082-34689e8f5bd6
2025-04-21T21:23:34.829Z [INFO]  server-95cc252f-49f7-3920-d082-34689e8f5bd6: entering follower state: follower="Node at 95cc252f-49f7-3920-d082-34689e8f5bd6 [Follower]" leader-address=c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff leader-id=server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff
2025-04-21T21:23:34.829Z [INFO]  server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff: pipelining replication: peer="{Voter server-95cc252f-49f7-3920-d082-34689e8f5bd6 95cc252f-49f7-3920-d082-34689e8f5bd6}"
2025-04-21T21:23:34.829Z [INFO]  server-5384fba3-2ac5-1ad2-4b16-5a8a751021d3: duplicate requestVote for same term: term=2
2025-04-21T21:23:34.829Z [INFO]  server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff: pipelining replication: peer="{Voter server-5384fba3-2ac5-1ad2-4b16-5a8a751021d3 5384fba3-2ac5-1ad2-4b16-5a8a751021d3}"
2025-04-21T21:23:34.829Z [WARN]  server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff: rejecting vote request since we have a leader: from=95cc252f-49f7-3920-d082-34689e8f5bd6 leader=c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff leader-id=server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff
2025-04-21T21:23:35.837Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:34.829480766 +0000 UTC m=+53.351140089 (1 nodes), 767.673479ms from start of poll, 767.849561ms from cluster start. Timeout at 2025-04-21 21:23:35.834485926 +0000 UTC m=+54.356145209, 1.00500512s after stability
2025-04-21T21:23:35.837Z [INFO]  server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff: starting snapshot up to: index=12
2025-04-21T21:23:35.837Z [INFO]  creating new snapshot: path=/tmp/raft1512351852/snapshots/2-12-1745270615837.tmp
2025-04-21T21:23:35.838Z [INFO]  server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff: snapshot complete up to: index=12
2025-04-21T21:23:35.838Z [INFO]  server-edc31bce-88a8-3e28-6529-a6e9776590f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edc31bce-88a8-3e28-6529-a6e9776590f5 Address:edc31bce-88a8-3e28-6529-a6e9776590f5} {Suffrage:Voter ID:server-b6ffd248-2038-44b1-db08-59ebbb018f97 Address:b6ffd248-2038-44b1-db08-59ebbb018f97} {Suffrage:Voter ID:server-28c11447-9f4d-f228-4a99-5686812a8d23 Address:28c11447-9f4d-f228-4a99-5686812a8d23}]"
2025-04-21T21:23:35.838Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edc31bce-88a8-3e28-6529-a6e9776590f5 Address:edc31bce-88a8-3e28-6529-a6e9776590f5} {Suffrage:Voter ID:server-b6ffd248-2038-44b1-db08-59ebbb018f97 Address:b6ffd248-2038-44b1-db08-59ebbb018f97} {Suffrage:Voter ID:server-28c11447-9f4d-f228-4a99-5686812a8d23 Address:28c11447-9f4d-f228-4a99-5686812a8d23}]"
2025-04-21T21:23:35.838Z [INFO]  server-28c11447-9f4d-f228-4a99-5686812a8d23: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-edc31bce-88a8-3e28-6529-a6e9776590f5 Address:edc31bce-88a8-3e28-6529-a6e9776590f5} {Suffrage:Voter ID:server-b6ffd248-2038-44b1-db08-59ebbb018f97 Address:b6ffd248-2038-44b1-db08-59ebbb018f97} {Suffrage:Voter ID:server-28c11447-9f4d-f228-4a99-5686812a8d23 Address:28c11447-9f4d-f228-4a99-5686812a8d23}]"
2025-04-21T21:23:35.838Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:35.838Z [INFO]  server-edc31bce-88a8-3e28-6529-a6e9776590f5: entering follower state: follower="Node at edc31bce-88a8-3e28-6529-a6e9776590f5 [Follower]" leader-address= leader-id=
2025-04-21T21:23:35.838Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: entering follower state: follower="Node at b6ffd248-2038-44b1-db08-59ebbb018f97 [Follower]" leader-address= leader-id=
2025-04-21T21:23:35.838Z [INFO]  server-28c11447-9f4d-f228-4a99-5686812a8d23: entering follower state: follower="Node at 28c11447-9f4d-f228-4a99-5686812a8d23 [Follower]" leader-address= leader-id=
2025-04-21T21:23:36.349Z [WARN]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:36.349Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: entering candidate state: node="Node at b6ffd248-2038-44b1-db08-59ebbb018f97 [Candidate]" term=2
2025-04-21T21:23:36.349Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: election won: term=2 tally=2
2025-04-21T21:23:36.349Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: entering leader state: leader="Node at b6ffd248-2038-44b1-db08-59ebbb018f97 [Leader]"
2025-04-21T21:23:36.349Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: added peer, starting replication: peer=server-edc31bce-88a8-3e28-6529-a6e9776590f5
2025-04-21T21:23:36.349Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: added peer, starting replication: peer=server-28c11447-9f4d-f228-4a99-5686812a8d23
2025-04-21T21:23:36.349Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: pipelining replication: peer="{Voter server-28c11447-9f4d-f228-4a99-5686812a8d23 28c11447-9f4d-f228-4a99-5686812a8d23}"
2025-04-21T21:23:36.349Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: pipelining replication: peer="{Voter server-edc31bce-88a8-3e28-6529-a6e9776590f5 edc31bce-88a8-3e28-6529-a6e9776590f5}"
2025-04-21T21:23:37.357Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:36.349521181 +0000 UTC m=+54.871180464 (1 nodes), 510.737792ms from start of poll, 510.891113ms from cluster start. Timeout at 2025-04-21 21:23:37.354530221 +0000 UTC m=+55.876189544, 1.00500908s after stability
2025-04-21T21:23:37.357Z [INFO]  creating new snapshot: path=/tmp/raft285512874/snapshots/2-15-1745270617357.tmp
2025-04-21T21:23:37.357Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: copied to local snapshot: bytes=71
2025-04-21T21:23:37.357Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: snapshot restore progress: id=2-15-1745270617357 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:37.357Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: removing all old logs from log store
2025-04-21T21:23:37.357Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: compacting logs: from=1 to=2
2025-04-21T21:23:37.357Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: restored user snapshot: index=15
2025-04-21T21:23:37.357Z [ERROR] server-b6ffd248-2038-44b1-db08-59ebbb018f97: failed to get log: index=2 error="log not found"
2025-04-21T21:23:37.357Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: aborting pipeline replication: peer="{Voter server-edc31bce-88a8-3e28-6529-a6e9776590f5 edc31bce-88a8-3e28-6529-a6e9776590f5}"
2025-04-21T21:23:37.357Z [ERROR] server-b6ffd248-2038-44b1-db08-59ebbb018f97: failed to get log: index=2 error="log not found"
2025-04-21T21:23:37.357Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: aborting pipeline replication: peer="{Voter server-28c11447-9f4d-f228-4a99-5686812a8d23 28c11447-9f4d-f228-4a99-5686812a8d23}"
2025-04-21T21:23:37.369Z [ERROR] server-b6ffd248-2038-44b1-db08-59ebbb018f97: failed to get log: index=2 error="log not found"
2025-04-21T21:23:37.369Z [INFO]  creating new snapshot: path=/tmp/raft4247042304/snapshots/2-15-1745270617369.tmp
2025-04-21T21:23:37.369Z [INFO]  server-edc31bce-88a8-3e28-6529-a6e9776590f5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:37.370Z [INFO]  server-edc31bce-88a8-3e28-6529-a6e9776590f5: copied to local snapshot: bytes=71
2025-04-21T21:23:37.370Z [INFO]  server-edc31bce-88a8-3e28-6529-a6e9776590f5: snapshot restore progress: id=2-15-1745270617369 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:37.370Z [INFO]  server-edc31bce-88a8-3e28-6529-a6e9776590f5: removing all old logs from log store
2025-04-21T21:23:37.370Z [INFO]  server-edc31bce-88a8-3e28-6529-a6e9776590f5: compacting logs: from=1 to=2
2025-04-21T21:23:37.370Z [INFO]  server-edc31bce-88a8-3e28-6529-a6e9776590f5: Installed remote snapshot
2025-04-21T21:23:37.370Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: pipelining replication: peer="{Voter server-edc31bce-88a8-3e28-6529-a6e9776590f5 edc31bce-88a8-3e28-6529-a6e9776590f5}"
2025-04-21T21:23:37.370Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:23:37.370Z [ERROR] server-b6ffd248-2038-44b1-db08-59ebbb018f97: failed to get log: index=2 error="log not found"
2025-04-21T21:23:37.370Z [INFO]  creating new snapshot: path=/tmp/raft2707025078/snapshots/2-15-1745270617370.tmp
2025-04-21T21:23:37.370Z [INFO]  server-28c11447-9f4d-f228-4a99-5686812a8d23: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:37.370Z [INFO]  server-28c11447-9f4d-f228-4a99-5686812a8d23: copied to local snapshot: bytes=71
2025-04-21T21:23:37.370Z [INFO]  server-28c11447-9f4d-f228-4a99-5686812a8d23: snapshot restore progress: id=2-15-1745270617370 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:37.370Z [INFO]  server-28c11447-9f4d-f228-4a99-5686812a8d23: removing all old logs from log store
2025-04-21T21:23:37.370Z [INFO]  server-28c11447-9f4d-f228-4a99-5686812a8d23: compacting logs: from=1 to=2
2025-04-21T21:23:37.370Z [INFO]  server-28c11447-9f4d-f228-4a99-5686812a8d23: Installed remote snapshot
2025-04-21T21:23:37.370Z [INFO]  server-b6ffd248-2038-44b1-db08-59ebbb018f97: pipelining replication: peer="{Voter server-28c11447-9f4d-f228-4a99-5686812a8d23 28c11447-9f4d-f228-4a99-5686812a8d23}"
2025-04-21T21:23:38.377Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:23:37.370305721 +0000 UTC m=+55.891965044 (2 nodes), 400ns from start of poll, 1.531675693s from cluster start. Timeout at 2025-04-21 21:23:38.375332482 +0000 UTC m=+56.896991805, 1.005026761s after stability
2025-04-21T21:23:38.393Z [INFO]  server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff: aborting pipeline replication: peer="{Voter server-95cc252f-49f7-3920-d082-34689e8f5bd6 95cc252f-49f7-3920-d082-34689e8f5bd6}"
2025-04-21T21:23:38.393Z [INFO]  server-c0b6b4e5-7d78-3bf1-0123-5b16ff3e41ff: aborting pipeline replication: peer="{Voter server-5384fba3-2ac5-1ad2-4b16-5a8a751021d3 5384fba3-2ac5-1ad2-4b16-5a8a751021d3}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false
2025-04-21T21:23:38.393Z [INFO]  server-dc953ea1-c846-efb6-367a-92d4558f032c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc953ea1-c846-efb6-367a-92d4558f032c Address:dc953ea1-c846-efb6-367a-92d4558f032c} {Suffrage:Voter ID:server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e Address:ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e} {Suffrage:Voter ID:server-2d460404-0f7f-26f0-cc5d-94fee0ccc2bf Address:2d460404-0f7f-26f0-cc5d-94fee0ccc2bf}]"
2025-04-21T21:23:38.393Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc953ea1-c846-efb6-367a-92d4558f032c Address:dc953ea1-c846-efb6-367a-92d4558f032c} {Suffrage:Voter ID:server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e Address:ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e} {Suffrage:Voter ID:server-2d460404-0f7f-26f0-cc5d-94fee0ccc2bf Address:2d460404-0f7f-26f0-cc5d-94fee0ccc2bf}]"
2025-04-21T21:23:38.393Z [INFO]  server-2d460404-0f7f-26f0-cc5d-94fee0ccc2bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc953ea1-c846-efb6-367a-92d4558f032c Address:dc953ea1-c846-efb6-367a-92d4558f032c} {Suffrage:Voter ID:server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e Address:ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e} {Suffrage:Voter ID:server-2d460404-0f7f-26f0-cc5d-94fee0ccc2bf Address:2d460404-0f7f-26f0-cc5d-94fee0ccc2bf}]"
2025-04-21T21:23:38.393Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:38.393Z [INFO]  server-dc953ea1-c846-efb6-367a-92d4558f032c: entering follower state: follower="Node at dc953ea1-c846-efb6-367a-92d4558f032c [Follower]" leader-address= leader-id=
2025-04-21T21:23:38.393Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: entering follower state: follower="Node at ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e [Follower]" leader-address= leader-id=
2025-04-21T21:23:38.393Z [INFO]  server-2d460404-0f7f-26f0-cc5d-94fee0ccc2bf: entering follower state: follower="Node at 2d460404-0f7f-26f0-cc5d-94fee0ccc2bf [Follower]" leader-address= leader-id=
2025-04-21T21:23:39.253Z [WARN]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:39.253Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: entering candidate state: node="Node at ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e [Candidate]" term=2
2025-04-21T21:23:39.253Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: election won: term=2 tally=2
2025-04-21T21:23:39.253Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: entering leader state: leader="Node at ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e [Leader]"
2025-04-21T21:23:39.253Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: added peer, starting replication: peer=server-dc953ea1-c846-efb6-367a-92d4558f032c
2025-04-21T21:23:39.253Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: added peer, starting replication: peer=server-2d460404-0f7f-26f0-cc5d-94fee0ccc2bf
2025-04-21T21:23:39.253Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: pipelining replication: peer="{Voter server-2d460404-0f7f-26f0-cc5d-94fee0ccc2bf 2d460404-0f7f-26f0-cc5d-94fee0ccc2bf}"
2025-04-21T21:23:39.253Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: pipelining replication: peer="{Voter server-dc953ea1-c846-efb6-367a-92d4558f032c dc953ea1-c846-efb6-367a-92d4558f032c}"
2025-04-21T21:23:40.261Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:39.253615018 +0000 UTC m=+57.775274301 (1 nodes), 859.828738ms from start of poll, 859.995419ms from cluster start. Timeout at 2025-04-21 21:23:40.258617338 +0000 UTC m=+58.780276621, 1.00500232s after stability
2025-04-21T21:23:40.261Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: starting snapshot up to: index=12
2025-04-21T21:23:40.261Z [INFO]  creating new snapshot: path=/tmp/raft4225486287/snapshots/2-12-1745270620261.tmp
2025-04-21T21:23:40.262Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: snapshot complete up to: index=12
2025-04-21T21:23:40.262Z [INFO]  creating new snapshot: path=/tmp/raft4225486287/snapshots/2-113-1745270620262.tmp
2025-04-21T21:23:40.262Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: copied to local snapshot: bytes=71
2025-04-21T21:23:40.263Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: snapshot restore progress: id=2-113-1745270620262 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:40.263Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: restored user snapshot: index=113
2025-04-21T21:23:40.263Z [ERROR] server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: failed to get log: index=23 error="log not found"
2025-04-21T21:23:40.263Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: aborting pipeline replication: peer="{Voter server-2d460404-0f7f-26f0-cc5d-94fee0ccc2bf 2d460404-0f7f-26f0-cc5d-94fee0ccc2bf}"
2025-04-21T21:23:40.263Z [ERROR] server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: failed to get log: index=23 error="log not found"
2025-04-21T21:23:40.263Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: aborting pipeline replication: peer="{Voter server-dc953ea1-c846-efb6-367a-92d4558f032c dc953ea1-c846-efb6-367a-92d4558f032c}"
2025-04-21T21:23:40.273Z [ERROR] server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: failed to get log: index=23 error="log not found"
2025-04-21T21:23:40.274Z [INFO]  creating new snapshot: path=/tmp/raft535478602/snapshots/2-113-1745270620274.tmp
2025-04-21T21:23:40.274Z [INFO]  server-dc953ea1-c846-efb6-367a-92d4558f032c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:40.274Z [INFO]  server-dc953ea1-c846-efb6-367a-92d4558f032c: copied to local snapshot: bytes=71
2025-04-21T21:23:40.274Z [INFO]  server-dc953ea1-c846-efb6-367a-92d4558f032c: snapshot restore progress: id=2-113-1745270620274 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:40.274Z [INFO]  server-dc953ea1-c846-efb6-367a-92d4558f032c: Installed remote snapshot
2025-04-21T21:23:40.274Z [WARN]  server-dc953ea1-c846-efb6-367a-92d4558f032c: skipping application of old log: index=22
2025-04-21T21:23:40.274Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: pipelining replication: peer="{Voter server-dc953ea1-c846-efb6-367a-92d4558f032c dc953ea1-c846-efb6-367a-92d4558f032c}"
2025-04-21T21:23:40.274Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:23:40.274Z [ERROR] server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: failed to get log: index=23 error="log not found"
2025-04-21T21:23:40.274Z [INFO]  creating new snapshot: path=/tmp/raft3941075236/snapshots/2-113-1745270620274.tmp
2025-04-21T21:23:40.274Z [INFO]  server-2d460404-0f7f-26f0-cc5d-94fee0ccc2bf: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:40.274Z [INFO]  server-2d460404-0f7f-26f0-cc5d-94fee0ccc2bf: copied to local snapshot: bytes=71
2025-04-21T21:23:40.275Z [INFO]  server-2d460404-0f7f-26f0-cc5d-94fee0ccc2bf: snapshot restore progress: id=2-113-1745270620274 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:40.275Z [INFO]  server-2d460404-0f7f-26f0-cc5d-94fee0ccc2bf: Installed remote snapshot
2025-04-21T21:23:40.275Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: pipelining replication: peer="{Voter server-2d460404-0f7f-26f0-cc5d-94fee0ccc2bf 2d460404-0f7f-26f0-cc5d-94fee0ccc2bf}"
2025-04-21T21:23:41.284Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:23:40.274497239 +0000 UTC m=+58.796156562 (2 nodes), 680ns from start of poll, 1.88087768s from cluster start. Timeout at 2025-04-21 21:23:41.279530479 +0000 UTC m=+59.801189802, 1.00503324s after stability
2025-04-21T21:23:41.301Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: aborting pipeline replication: peer="{Voter server-2d460404-0f7f-26f0-cc5d-94fee0ccc2bf 2d460404-0f7f-26f0-cc5d-94fee0ccc2bf}"
2025-04-21T21:23:41.301Z [INFO]  server-ab25d89c-a1ed-6eb2-ed3d-4571ebdbfb3e: aborting pipeline replication: peer="{Voter server-dc953ea1-c846-efb6-367a-92d4558f032c dc953ea1-c846-efb6-367a-92d4558f032c}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false
2025-04-21T21:23:41.301Z [INFO]  server-320ebbf4-4d6f-309a-cf78-e38084399e18: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-320ebbf4-4d6f-309a-cf78-e38084399e18 Address:320ebbf4-4d6f-309a-cf78-e38084399e18} {Suffrage:Voter ID:server-4dffb194-575c-1347-207b-3312f969effc Address:4dffb194-575c-1347-207b-3312f969effc} {Suffrage:Voter ID:server-ca0de73d-0ec2-7288-4518-4e689693e95a Address:ca0de73d-0ec2-7288-4518-4e689693e95a}]"
2025-04-21T21:23:41.301Z [INFO]  server-4dffb194-575c-1347-207b-3312f969effc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-320ebbf4-4d6f-309a-cf78-e38084399e18 Address:320ebbf4-4d6f-309a-cf78-e38084399e18} {Suffrage:Voter ID:server-4dffb194-575c-1347-207b-3312f969effc Address:4dffb194-575c-1347-207b-3312f969effc} {Suffrage:Voter ID:server-ca0de73d-0ec2-7288-4518-4e689693e95a Address:ca0de73d-0ec2-7288-4518-4e689693e95a}]"
2025-04-21T21:23:41.301Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-320ebbf4-4d6f-309a-cf78-e38084399e18 Address:320ebbf4-4d6f-309a-cf78-e38084399e18} {Suffrage:Voter ID:server-4dffb194-575c-1347-207b-3312f969effc Address:4dffb194-575c-1347-207b-3312f969effc} {Suffrage:Voter ID:server-ca0de73d-0ec2-7288-4518-4e689693e95a Address:ca0de73d-0ec2-7288-4518-4e689693e95a}]"
2025-04-21T21:23:41.301Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:41.301Z [INFO]  server-320ebbf4-4d6f-309a-cf78-e38084399e18: entering follower state: follower="Node at 320ebbf4-4d6f-309a-cf78-e38084399e18 [Follower]" leader-address= leader-id=
2025-04-21T21:23:41.301Z [INFO]  server-4dffb194-575c-1347-207b-3312f969effc: entering follower state: follower="Node at 4dffb194-575c-1347-207b-3312f969effc [Follower]" leader-address= leader-id=
2025-04-21T21:23:41.301Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: entering follower state: follower="Node at ca0de73d-0ec2-7288-4518-4e689693e95a [Follower]" leader-address= leader-id=
2025-04-21T21:23:41.821Z [WARN]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:41.821Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: entering candidate state: node="Node at ca0de73d-0ec2-7288-4518-4e689693e95a [Candidate]" term=2
2025-04-21T21:23:41.821Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: election won: term=2 tally=2
2025-04-21T21:23:41.821Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: entering leader state: leader="Node at ca0de73d-0ec2-7288-4518-4e689693e95a [Leader]"
2025-04-21T21:23:41.821Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: added peer, starting replication: peer=server-320ebbf4-4d6f-309a-cf78-e38084399e18
2025-04-21T21:23:41.821Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: added peer, starting replication: peer=server-4dffb194-575c-1347-207b-3312f969effc
2025-04-21T21:23:41.821Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: pipelining replication: peer="{Voter server-4dffb194-575c-1347-207b-3312f969effc 4dffb194-575c-1347-207b-3312f969effc}"
2025-04-21T21:23:41.821Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: pipelining replication: peer="{Voter server-320ebbf4-4d6f-309a-cf78-e38084399e18 320ebbf4-4d6f-309a-cf78-e38084399e18}"
2025-04-21T21:23:42.829Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:41.821660837 +0000 UTC m=+60.343320160 (1 nodes), 519.780053ms from start of poll, 519.967694ms from cluster start. Timeout at 2025-04-21 21:23:42.826662117 +0000 UTC m=+61.348321440, 1.00500128s after stability
2025-04-21T21:23:42.829Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: starting snapshot up to: index=12
2025-04-21T21:23:42.829Z [INFO]  creating new snapshot: path=/tmp/raft373588142/snapshots/2-12-1745270622829.tmp
2025-04-21T21:23:42.829Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: snapshot complete up to: index=12
2025-04-21T21:23:42.830Z [INFO]  creating new snapshot: path=/tmp/raft373588142/snapshots/2-113-1745270622830.tmp
2025-04-21T21:23:42.830Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: copied to local snapshot: bytes=71
2025-04-21T21:23:42.830Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: snapshot restore progress: id=2-113-1745270622830 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:42.830Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: removing all old logs from log store
2025-04-21T21:23:42.830Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: compacting logs: from=1 to=22
2025-04-21T21:23:42.830Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: restored user snapshot: index=113
2025-04-21T21:23:42.830Z [ERROR] server-ca0de73d-0ec2-7288-4518-4e689693e95a: failed to get log: index=22 error="log not found"
2025-04-21T21:23:42.830Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: aborting pipeline replication: peer="{Voter server-4dffb194-575c-1347-207b-3312f969effc 4dffb194-575c-1347-207b-3312f969effc}"
2025-04-21T21:23:42.830Z [ERROR] server-ca0de73d-0ec2-7288-4518-4e689693e95a: failed to get log: index=22 error="log not found"
2025-04-21T21:23:42.830Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: aborting pipeline replication: peer="{Voter server-320ebbf4-4d6f-309a-cf78-e38084399e18 320ebbf4-4d6f-309a-cf78-e38084399e18}"
2025-04-21T21:23:42.841Z [ERROR] server-ca0de73d-0ec2-7288-4518-4e689693e95a: failed to get log: index=22 error="log not found"
2025-04-21T21:23:42.841Z [INFO]  creating new snapshot: path=/tmp/raft1144294472/snapshots/2-113-1745270622841.tmp
2025-04-21T21:23:42.841Z [INFO]  server-4dffb194-575c-1347-207b-3312f969effc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:42.841Z [INFO]  server-4dffb194-575c-1347-207b-3312f969effc: copied to local snapshot: bytes=71
2025-04-21T21:23:42.841Z [INFO]  server-4dffb194-575c-1347-207b-3312f969effc: snapshot restore progress: id=2-113-1745270622841 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:42.841Z [INFO]  server-4dffb194-575c-1347-207b-3312f969effc: removing all old logs from log store
2025-04-21T21:23:42.841Z [INFO]  server-4dffb194-575c-1347-207b-3312f969effc: compacting logs: from=1 to=22
2025-04-21T21:23:42.841Z [INFO]  server-4dffb194-575c-1347-207b-3312f969effc: Installed remote snapshot
2025-04-21T21:23:42.841Z [WARN]  server-4dffb194-575c-1347-207b-3312f969effc: skipping application of old log: index=22
2025-04-21T21:23:42.841Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: pipelining replication: peer="{Voter server-4dffb194-575c-1347-207b-3312f969effc 4dffb194-575c-1347-207b-3312f969effc}"
2025-04-21T21:23:42.841Z [ERROR] server-ca0de73d-0ec2-7288-4518-4e689693e95a: failed to get log: index=22 error="log not found"
2025-04-21T21:23:42.842Z [INFO]  creating new snapshot: path=/tmp/raft1710005750/snapshots/2-113-1745270622842.tmp
2025-04-21T21:23:42.842Z [INFO]  server-320ebbf4-4d6f-309a-cf78-e38084399e18: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:42.842Z [INFO]  server-320ebbf4-4d6f-309a-cf78-e38084399e18: copied to local snapshot: bytes=71
2025-04-21T21:23:42.842Z [INFO]  server-320ebbf4-4d6f-309a-cf78-e38084399e18: snapshot restore progress: id=2-113-1745270622842 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:42.842Z [INFO]  server-320ebbf4-4d6f-309a-cf78-e38084399e18: removing all old logs from log store
2025-04-21T21:23:42.842Z [INFO]  server-320ebbf4-4d6f-309a-cf78-e38084399e18: compacting logs: from=1 to=22
2025-04-21T21:23:42.842Z [INFO]  server-320ebbf4-4d6f-309a-cf78-e38084399e18: Installed remote snapshot
2025-04-21T21:23:42.842Z [WARN]  server-320ebbf4-4d6f-309a-cf78-e38084399e18: skipping application of old log: index=22
2025-04-21T21:23:42.842Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: pipelining replication: peer="{Voter server-320ebbf4-4d6f-309a-cf78-e38084399e18 320ebbf4-4d6f-309a-cf78-e38084399e18}"
2025-04-21T21:23:42.842Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:23:43.849Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:23:42.842485378 +0000 UTC m=+61.364144701 (2 nodes), 480ns from start of poll, 1.540792235s from cluster start. Timeout at 2025-04-21 21:23:43.847516018 +0000 UTC m=+62.369175341, 1.00503064s after stability
2025-04-21T21:23:43.865Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: aborting pipeline replication: peer="{Voter server-4dffb194-575c-1347-207b-3312f969effc 4dffb194-575c-1347-207b-3312f969effc}"
2025-04-21T21:23:43.865Z [INFO]  server-ca0de73d-0ec2-7288-4518-4e689693e95a: aborting pipeline replication: peer="{Voter server-320ebbf4-4d6f-309a-cf78-e38084399e18 320ebbf4-4d6f-309a-cf78-e38084399e18}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true
2025-04-21T21:23:43.865Z [INFO]  server-a6fd2914-8169-9b64-309b-34f4174728db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6fd2914-8169-9b64-309b-34f4174728db Address:a6fd2914-8169-9b64-309b-34f4174728db} {Suffrage:Voter ID:server-01f8e7a6-ee67-8078-f279-703dbbd9c235 Address:01f8e7a6-ee67-8078-f279-703dbbd9c235} {Suffrage:Voter ID:server-49c85636-8384-c6bd-ae22-7d152ec1e0fb Address:49c85636-8384-c6bd-ae22-7d152ec1e0fb}]"
2025-04-21T21:23:43.865Z [INFO]  server-01f8e7a6-ee67-8078-f279-703dbbd9c235: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6fd2914-8169-9b64-309b-34f4174728db Address:a6fd2914-8169-9b64-309b-34f4174728db} {Suffrage:Voter ID:server-01f8e7a6-ee67-8078-f279-703dbbd9c235 Address:01f8e7a6-ee67-8078-f279-703dbbd9c235} {Suffrage:Voter ID:server-49c85636-8384-c6bd-ae22-7d152ec1e0fb Address:49c85636-8384-c6bd-ae22-7d152ec1e0fb}]"
2025-04-21T21:23:43.865Z [INFO]  server-49c85636-8384-c6bd-ae22-7d152ec1e0fb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6fd2914-8169-9b64-309b-34f4174728db Address:a6fd2914-8169-9b64-309b-34f4174728db} {Suffrage:Voter ID:server-01f8e7a6-ee67-8078-f279-703dbbd9c235 Address:01f8e7a6-ee67-8078-f279-703dbbd9c235} {Suffrage:Voter ID:server-49c85636-8384-c6bd-ae22-7d152ec1e0fb Address:49c85636-8384-c6bd-ae22-7d152ec1e0fb}]"
2025-04-21T21:23:43.865Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:43.865Z [INFO]  server-a6fd2914-8169-9b64-309b-34f4174728db: entering follower state: follower="Node at a6fd2914-8169-9b64-309b-34f4174728db [Follower]" leader-address= leader-id=
2025-04-21T21:23:43.865Z [INFO]  server-01f8e7a6-ee67-8078-f279-703dbbd9c235: entering follower state: follower="Node at 01f8e7a6-ee67-8078-f279-703dbbd9c235 [Follower]" leader-address= leader-id=
2025-04-21T21:23:43.865Z [INFO]  server-49c85636-8384-c6bd-ae22-7d152ec1e0fb: entering follower state: follower="Node at 49c85636-8384-c6bd-ae22-7d152ec1e0fb [Follower]" leader-address= leader-id=
2025-04-21T21:23:44.397Z [WARN]  server-a6fd2914-8169-9b64-309b-34f4174728db: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:44.397Z [INFO]  server-a6fd2914-8169-9b64-309b-34f4174728db: entering candidate state: node="Node at a6fd2914-8169-9b64-309b-34f4174728db [Candidate]" term=2
2025-04-21T21:23:44.397Z [INFO]  server-a6fd2914-8169-9b64-309b-34f4174728db: election won: term=2 tally=2
2025-04-21T21:23:44.397Z [INFO]  server-a6fd2914-8169-9b64-309b-34f4174728db: entering leader state: leader="Node at a6fd2914-8169-9b64-309b-34f4174728db [Leader]"
2025-04-21T21:23:44.397Z [INFO]  server-a6fd2914-8169-9b64-309b-34f4174728db: added peer, starting replication: peer=server-01f8e7a6-ee67-8078-f279-703dbbd9c235
2025-04-21T21:23:44.397Z [INFO]  server-a6fd2914-8169-9b64-309b-34f4174728db: added peer, starting replication: peer=server-49c85636-8384-c6bd-ae22-7d152ec1e0fb
2025-04-21T21:23:44.397Z [INFO]  server-a6fd2914-8169-9b64-309b-34f4174728db: pipelining replication: peer="{Voter server-49c85636-8384-c6bd-ae22-7d152ec1e0fb 49c85636-8384-c6bd-ae22-7d152ec1e0fb}"
2025-04-21T21:23:44.397Z [INFO]  server-a6fd2914-8169-9b64-309b-34f4174728db: pipelining replication: peer="{Voter server-01f8e7a6-ee67-8078-f279-703dbbd9c235 01f8e7a6-ee67-8078-f279-703dbbd9c235}"
2025-04-21T21:23:45.405Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:44.397420868 +0000 UTC m=+62.919080151 (1 nodes), 531.528452ms from start of poll, 531.712693ms from cluster start. Timeout at 2025-04-21 21:23:45.402423708 +0000 UTC m=+63.924083031, 1.00500288s after stability
2025-04-21T21:23:45.406Z [INFO]  server-a6fd2914-8169-9b64-309b-34f4174728db: starting snapshot up to: index=12
2025-04-21T21:23:45.406Z [INFO]  creating new snapshot: path=/tmp/raft313961366/snapshots/2-12-1745270625406.tmp
2025-04-21T21:23:45.406Z [INFO]  server-a6fd2914-8169-9b64-309b-34f4174728db: snapshot complete up to: index=12
2025-04-21T21:23:45.406Z [INFO]  server-ad225cf3-6898-65fa-da52-015cef1225e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad225cf3-6898-65fa-da52-015cef1225e5 Address:ad225cf3-6898-65fa-da52-015cef1225e5} {Suffrage:Voter ID:server-5551296f-b906-ef63-3801-99b468c45ad6 Address:5551296f-b906-ef63-3801-99b468c45ad6} {Suffrage:Voter ID:server-b79fbaf2-89f2-141f-ade2-9324d60221ba Address:b79fbaf2-89f2-141f-ade2-9324d60221ba}]"
2025-04-21T21:23:45.406Z [INFO]  server-5551296f-b906-ef63-3801-99b468c45ad6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad225cf3-6898-65fa-da52-015cef1225e5 Address:ad225cf3-6898-65fa-da52-015cef1225e5} {Suffrage:Voter ID:server-5551296f-b906-ef63-3801-99b468c45ad6 Address:5551296f-b906-ef63-3801-99b468c45ad6} {Suffrage:Voter ID:server-b79fbaf2-89f2-141f-ade2-9324d60221ba Address:b79fbaf2-89f2-141f-ade2-9324d60221ba}]"
2025-04-21T21:23:45.407Z [INFO]  server-b79fbaf2-89f2-141f-ade2-9324d60221ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ad225cf3-6898-65fa-da52-015cef1225e5 Address:ad225cf3-6898-65fa-da52-015cef1225e5} {Suffrage:Voter ID:server-5551296f-b906-ef63-3801-99b468c45ad6 Address:5551296f-b906-ef63-3801-99b468c45ad6} {Suffrage:Voter ID:server-b79fbaf2-89f2-141f-ade2-9324d60221ba Address:b79fbaf2-89f2-141f-ade2-9324d60221ba}]"
2025-04-21T21:23:45.407Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:45.407Z [INFO]  server-ad225cf3-6898-65fa-da52-015cef1225e5: entering follower state: follower="Node at ad225cf3-6898-65fa-da52-015cef1225e5 [Follower]" leader-address= leader-id=
2025-04-21T21:23:45.407Z [INFO]  server-5551296f-b906-ef63-3801-99b468c45ad6: entering follower state: follower="Node at 5551296f-b906-ef63-3801-99b468c45ad6 [Follower]" leader-address= leader-id=
2025-04-21T21:23:45.407Z [INFO]  server-b79fbaf2-89f2-141f-ade2-9324d60221ba: entering follower state: follower="Node at b79fbaf2-89f2-141f-ade2-9324d60221ba [Follower]" leader-address= leader-id=
2025-04-21T21:23:46.005Z [WARN]  server-5551296f-b906-ef63-3801-99b468c45ad6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:46.005Z [INFO]  server-5551296f-b906-ef63-3801-99b468c45ad6: entering candidate state: node="Node at 5551296f-b906-ef63-3801-99b468c45ad6 [Candidate]" term=2
2025-04-21T21:23:46.005Z [INFO]  server-5551296f-b906-ef63-3801-99b468c45ad6: election won: term=2 tally=2
2025-04-21T21:23:46.005Z [INFO]  server-5551296f-b906-ef63-3801-99b468c45ad6: entering leader state: leader="Node at 5551296f-b906-ef63-3801-99b468c45ad6 [Leader]"
2025-04-21T21:23:46.005Z [INFO]  server-5551296f-b906-ef63-3801-99b468c45ad6: added peer, starting replication: peer=server-ad225cf3-6898-65fa-da52-015cef1225e5
2025-04-21T21:23:46.005Z [INFO]  server-5551296f-b906-ef63-3801-99b468c45ad6: added peer, starting replication: peer=server-b79fbaf2-89f2-141f-ade2-9324d60221ba
2025-04-21T21:23:46.005Z [INFO]  server-5551296f-b906-ef63-3801-99b468c45ad6: pipelining replication: peer="{Voter server-b79fbaf2-89f2-141f-ade2-9324d60221ba b79fbaf2-89f2-141f-ade2-9324d60221ba}"
2025-04-21T21:23:46.005Z [INFO]  server-5551296f-b906-ef63-3801-99b468c45ad6: pipelining replication: peer="{Voter server-ad225cf3-6898-65fa-da52-015cef1225e5 ad225cf3-6898-65fa-da52-015cef1225e5}"
2025-04-21T21:23:47.017Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:46.005365875 +0000 UTC m=+64.527025198 (1 nodes), 598.346542ms from start of poll, 598.502343ms from cluster start. Timeout at 2025-04-21 21:23:47.010368995 +0000 UTC m=+65.532028318, 1.00500312s after stability
2025-04-21T21:23:47.017Z [INFO]  creating new snapshot: path=/tmp/raft76852227/snapshots/2-113-1745270627017.tmp
2025-04-21T21:23:47.017Z [INFO]  server-5551296f-b906-ef63-3801-99b468c45ad6: copied to local snapshot: bytes=71
2025-04-21T21:23:47.018Z [INFO]  server-5551296f-b906-ef63-3801-99b468c45ad6: snapshot restore progress: id=2-113-1745270627017 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:47.018Z [INFO]  server-5551296f-b906-ef63-3801-99b468c45ad6: restored user snapshot: index=113
2025-04-21T21:23:47.018Z [ERROR] server-5551296f-b906-ef63-3801-99b468c45ad6: failed to get log: index=3 error="log not found"
2025-04-21T21:23:47.018Z [INFO]  server-5551296f-b906-ef63-3801-99b468c45ad6: aborting pipeline replication: peer="{Voter server-b79fbaf2-89f2-141f-ade2-9324d60221ba b79fbaf2-89f2-141f-ade2-9324d60221ba}"
2025-04-21T21:23:47.018Z [ERROR] server-5551296f-b906-ef63-3801-99b468c45ad6: failed to get log: index=3 error="log not found"
2025-04-21T21:23:47.018Z [INFO]  server-5551296f-b906-ef63-3801-99b468c45ad6: aborting pipeline replication: peer="{Voter server-ad225cf3-6898-65fa-da52-015cef1225e5 ad225cf3-6898-65fa-da52-015cef1225e5}"
2025-04-21T21:23:47.029Z [ERROR] server-5551296f-b906-ef63-3801-99b468c45ad6: failed to get log: index=3 error="log not found"
2025-04-21T21:23:47.029Z [INFO]  creating new snapshot: path=/tmp/raft317176660/snapshots/2-113-1745270627029.tmp
2025-04-21T21:23:47.029Z [INFO]  server-ad225cf3-6898-65fa-da52-015cef1225e5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:47.029Z [INFO]  server-ad225cf3-6898-65fa-da52-015cef1225e5: copied to local snapshot: bytes=71
2025-04-21T21:23:47.029Z [INFO]  server-ad225cf3-6898-65fa-da52-015cef1225e5: snapshot restore progress: id=2-113-1745270627029 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:47.029Z [INFO]  server-ad225cf3-6898-65fa-da52-015cef1225e5: Installed remote snapshot
2025-04-21T21:23:47.029Z [INFO]  server-5551296f-b906-ef63-3801-99b468c45ad6: pipelining replication: peer="{Voter server-ad225cf3-6898-65fa-da52-015cef1225e5 ad225cf3-6898-65fa-da52-015cef1225e5}"
2025-04-21T21:23:47.030Z [ERROR] server-5551296f-b906-ef63-3801-99b468c45ad6: failed to get log: index=3 error="log not found"
2025-04-21T21:23:47.030Z [INFO]  creating new snapshot: path=/tmp/raft2446270161/snapshots/2-113-1745270627030.tmp
2025-04-21T21:23:47.030Z [INFO]  server-b79fbaf2-89f2-141f-ade2-9324d60221ba: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:47.030Z [INFO]  server-b79fbaf2-89f2-141f-ade2-9324d60221ba: copied to local snapshot: bytes=71
2025-04-21T21:23:47.030Z [INFO]  server-b79fbaf2-89f2-141f-ade2-9324d60221ba: snapshot restore progress: id=2-113-1745270627030 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:47.030Z [INFO]  server-b79fbaf2-89f2-141f-ade2-9324d60221ba: Installed remote snapshot
2025-04-21T21:23:47.030Z [INFO]  server-5551296f-b906-ef63-3801-99b468c45ad6: pipelining replication: peer="{Voter server-b79fbaf2-89f2-141f-ade2-9324d60221ba b79fbaf2-89f2-141f-ade2-9324d60221ba}"
2025-04-21T21:23:47.030Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:23:48.065Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:23:47.030574245 +0000 UTC m=+65.552233568 (2 nodes), 440ns from start of poll, 1.623710713s from cluster start. Timeout at 2025-04-21 21:23:48.035597645 +0000 UTC m=+66.557256968, 1.0050234s after stability
2025-04-21T21:23:48.089Z [INFO]  server-a6fd2914-8169-9b64-309b-34f4174728db: aborting pipeline replication: peer="{Voter server-49c85636-8384-c6bd-ae22-7d152ec1e0fb 49c85636-8384-c6bd-ae22-7d152ec1e0fb}"
2025-04-21T21:23:48.089Z [INFO]  server-a6fd2914-8169-9b64-309b-34f4174728db: aborting pipeline replication: peer="{Voter server-01f8e7a6-ee67-8078-f279-703dbbd9c235 01f8e7a6-ee67-8078-f279-703dbbd9c235}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true
2025-04-21T21:23:48.090Z [INFO]  server-05122540-e82e-374c-2b80-829fa5d82cc2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05122540-e82e-374c-2b80-829fa5d82cc2 Address:05122540-e82e-374c-2b80-829fa5d82cc2} {Suffrage:Voter ID:server-920105a4-76ce-0c61-2f11-4861e781a78f Address:920105a4-76ce-0c61-2f11-4861e781a78f} {Suffrage:Voter ID:server-29bb517f-ba48-7bb9-abc7-bd008812dc3c Address:29bb517f-ba48-7bb9-abc7-bd008812dc3c}]"
2025-04-21T21:23:48.090Z [INFO]  server-920105a4-76ce-0c61-2f11-4861e781a78f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05122540-e82e-374c-2b80-829fa5d82cc2 Address:05122540-e82e-374c-2b80-829fa5d82cc2} {Suffrage:Voter ID:server-920105a4-76ce-0c61-2f11-4861e781a78f Address:920105a4-76ce-0c61-2f11-4861e781a78f} {Suffrage:Voter ID:server-29bb517f-ba48-7bb9-abc7-bd008812dc3c Address:29bb517f-ba48-7bb9-abc7-bd008812dc3c}]"
2025-04-21T21:23:48.090Z [INFO]  server-29bb517f-ba48-7bb9-abc7-bd008812dc3c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05122540-e82e-374c-2b80-829fa5d82cc2 Address:05122540-e82e-374c-2b80-829fa5d82cc2} {Suffrage:Voter ID:server-920105a4-76ce-0c61-2f11-4861e781a78f Address:920105a4-76ce-0c61-2f11-4861e781a78f} {Suffrage:Voter ID:server-29bb517f-ba48-7bb9-abc7-bd008812dc3c Address:29bb517f-ba48-7bb9-abc7-bd008812dc3c}]"
2025-04-21T21:23:48.090Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:48.090Z [INFO]  server-05122540-e82e-374c-2b80-829fa5d82cc2: entering follower state: follower="Node at 05122540-e82e-374c-2b80-829fa5d82cc2 [Follower]" leader-address= leader-id=
2025-04-21T21:23:48.090Z [INFO]  server-920105a4-76ce-0c61-2f11-4861e781a78f: entering follower state: follower="Node at 920105a4-76ce-0c61-2f11-4861e781a78f [Follower]" leader-address= leader-id=
2025-04-21T21:23:48.090Z [INFO]  server-29bb517f-ba48-7bb9-abc7-bd008812dc3c: entering follower state: follower="Node at 29bb517f-ba48-7bb9-abc7-bd008812dc3c [Follower]" leader-address= leader-id=
2025-04-21T21:23:48.709Z [WARN]  server-29bb517f-ba48-7bb9-abc7-bd008812dc3c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:48.709Z [INFO]  server-29bb517f-ba48-7bb9-abc7-bd008812dc3c: entering candidate state: node="Node at 29bb517f-ba48-7bb9-abc7-bd008812dc3c [Candidate]" term=2
2025-04-21T21:23:48.709Z [INFO]  server-29bb517f-ba48-7bb9-abc7-bd008812dc3c: election won: term=2 tally=2
2025-04-21T21:23:48.709Z [INFO]  server-29bb517f-ba48-7bb9-abc7-bd008812dc3c: entering leader state: leader="Node at 29bb517f-ba48-7bb9-abc7-bd008812dc3c [Leader]"
2025-04-21T21:23:48.709Z [INFO]  server-29bb517f-ba48-7bb9-abc7-bd008812dc3c: added peer, starting replication: peer=server-05122540-e82e-374c-2b80-829fa5d82cc2
2025-04-21T21:23:48.709Z [INFO]  server-29bb517f-ba48-7bb9-abc7-bd008812dc3c: added peer, starting replication: peer=server-920105a4-76ce-0c61-2f11-4861e781a78f
2025-04-21T21:23:48.709Z [INFO]  server-29bb517f-ba48-7bb9-abc7-bd008812dc3c: pipelining replication: peer="{Voter server-920105a4-76ce-0c61-2f11-4861e781a78f 920105a4-76ce-0c61-2f11-4861e781a78f}"
2025-04-21T21:23:48.709Z [INFO]  server-29bb517f-ba48-7bb9-abc7-bd008812dc3c: pipelining replication: peer="{Voter server-05122540-e82e-374c-2b80-829fa5d82cc2 05122540-e82e-374c-2b80-829fa5d82cc2}"
2025-04-21T21:23:49.717Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:48.70975461 +0000 UTC m=+67.231413933 (1 nodes), 619.616604ms from start of poll, 619.822846ms from cluster start. Timeout at 2025-04-21 21:23:49.71475937 +0000 UTC m=+68.236418693, 1.00500476s after stability
2025-04-21T21:23:49.717Z [INFO]  server-29bb517f-ba48-7bb9-abc7-bd008812dc3c: starting snapshot up to: index=12
2025-04-21T21:23:49.717Z [INFO]  creating new snapshot: path=/tmp/raft348497053/snapshots/2-12-1745270629717.tmp
2025-04-21T21:23:49.717Z [INFO]  server-29bb517f-ba48-7bb9-abc7-bd008812dc3c: snapshot complete up to: index=12
2025-04-21T21:23:49.718Z [INFO]  server-a627613c-232e-58d0-76b9-c8af02f9e6e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a627613c-232e-58d0-76b9-c8af02f9e6e8 Address:a627613c-232e-58d0-76b9-c8af02f9e6e8} {Suffrage:Voter ID:server-a9ae0dc7-3a61-bd7b-7389-8eb8cd580582 Address:a9ae0dc7-3a61-bd7b-7389-8eb8cd580582} {Suffrage:Voter ID:server-5edeff14-5674-f6ec-a845-9aefd5f9d33e Address:5edeff14-5674-f6ec-a845-9aefd5f9d33e}]"
2025-04-21T21:23:49.718Z [INFO]  server-a9ae0dc7-3a61-bd7b-7389-8eb8cd580582: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a627613c-232e-58d0-76b9-c8af02f9e6e8 Address:a627613c-232e-58d0-76b9-c8af02f9e6e8} {Suffrage:Voter ID:server-a9ae0dc7-3a61-bd7b-7389-8eb8cd580582 Address:a9ae0dc7-3a61-bd7b-7389-8eb8cd580582} {Suffrage:Voter ID:server-5edeff14-5674-f6ec-a845-9aefd5f9d33e Address:5edeff14-5674-f6ec-a845-9aefd5f9d33e}]"
2025-04-21T21:23:49.718Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a627613c-232e-58d0-76b9-c8af02f9e6e8 Address:a627613c-232e-58d0-76b9-c8af02f9e6e8} {Suffrage:Voter ID:server-a9ae0dc7-3a61-bd7b-7389-8eb8cd580582 Address:a9ae0dc7-3a61-bd7b-7389-8eb8cd580582} {Suffrage:Voter ID:server-5edeff14-5674-f6ec-a845-9aefd5f9d33e Address:5edeff14-5674-f6ec-a845-9aefd5f9d33e}]"
2025-04-21T21:23:49.718Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:49.718Z [INFO]  server-a627613c-232e-58d0-76b9-c8af02f9e6e8: entering follower state: follower="Node at a627613c-232e-58d0-76b9-c8af02f9e6e8 [Follower]" leader-address= leader-id=
2025-04-21T21:23:49.718Z [INFO]  server-a9ae0dc7-3a61-bd7b-7389-8eb8cd580582: entering follower state: follower="Node at a9ae0dc7-3a61-bd7b-7389-8eb8cd580582 [Follower]" leader-address= leader-id=
2025-04-21T21:23:49.718Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: entering follower state: follower="Node at 5edeff14-5674-f6ec-a845-9aefd5f9d33e [Follower]" leader-address= leader-id=
2025-04-21T21:23:50.421Z [WARN]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:50.421Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: entering candidate state: node="Node at 5edeff14-5674-f6ec-a845-9aefd5f9d33e [Candidate]" term=2
2025-04-21T21:23:50.421Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: election won: term=2 tally=2
2025-04-21T21:23:50.421Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: entering leader state: leader="Node at 5edeff14-5674-f6ec-a845-9aefd5f9d33e [Leader]"
2025-04-21T21:23:50.421Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: added peer, starting replication: peer=server-a627613c-232e-58d0-76b9-c8af02f9e6e8
2025-04-21T21:23:50.421Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: added peer, starting replication: peer=server-a9ae0dc7-3a61-bd7b-7389-8eb8cd580582
2025-04-21T21:23:50.421Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: pipelining replication: peer="{Voter server-a9ae0dc7-3a61-bd7b-7389-8eb8cd580582 a9ae0dc7-3a61-bd7b-7389-8eb8cd580582}"
2025-04-21T21:23:50.421Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: pipelining replication: peer="{Voter server-a627613c-232e-58d0-76b9-c8af02f9e6e8 a627613c-232e-58d0-76b9-c8af02f9e6e8}"
2025-04-21T21:23:51.429Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:50.421469954 +0000 UTC m=+68.943129277 (1 nodes), 702.922324ms from start of poll, 703.083165ms from cluster start. Timeout at 2025-04-21 21:23:51.426486954 +0000 UTC m=+69.948146277, 1.005017s after stability
2025-04-21T21:23:51.429Z [INFO]  creating new snapshot: path=/tmp/raft3526128985/snapshots/2-113-1745270631429.tmp
2025-04-21T21:23:51.430Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: copied to local snapshot: bytes=71
2025-04-21T21:23:51.445Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: snapshot restore progress: id=2-113-1745270631429 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:51.445Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: removing all old logs from log store
2025-04-21T21:23:51.445Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: compacting logs: from=1 to=2
2025-04-21T21:23:51.445Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: restored user snapshot: index=113
2025-04-21T21:23:51.445Z [ERROR] server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: failed to get log: index=2 error="log not found"
2025-04-21T21:23:51.445Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: aborting pipeline replication: peer="{Voter server-a9ae0dc7-3a61-bd7b-7389-8eb8cd580582 a9ae0dc7-3a61-bd7b-7389-8eb8cd580582}"
2025-04-21T21:23:51.445Z [ERROR] server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: failed to get log: index=2 error="log not found"
2025-04-21T21:23:51.445Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: aborting pipeline replication: peer="{Voter server-a627613c-232e-58d0-76b9-c8af02f9e6e8 a627613c-232e-58d0-76b9-c8af02f9e6e8}"
2025-04-21T21:23:51.445Z [ERROR] server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: failed to get log: index=2 error="log not found"
2025-04-21T21:23:51.446Z [INFO]  creating new snapshot: path=/tmp/raft479915419/snapshots/2-113-1745270631446.tmp
2025-04-21T21:23:51.446Z [INFO]  server-a627613c-232e-58d0-76b9-c8af02f9e6e8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:51.446Z [INFO]  server-a627613c-232e-58d0-76b9-c8af02f9e6e8: copied to local snapshot: bytes=71
2025-04-21T21:23:51.446Z [INFO]  server-a627613c-232e-58d0-76b9-c8af02f9e6e8: snapshot restore progress: id=2-113-1745270631446 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:51.446Z [INFO]  server-a627613c-232e-58d0-76b9-c8af02f9e6e8: removing all old logs from log store
2025-04-21T21:23:51.446Z [INFO]  server-a627613c-232e-58d0-76b9-c8af02f9e6e8: compacting logs: from=1 to=2
2025-04-21T21:23:51.446Z [INFO]  server-a627613c-232e-58d0-76b9-c8af02f9e6e8: Installed remote snapshot
2025-04-21T21:23:51.446Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: pipelining replication: peer="{Voter server-a627613c-232e-58d0-76b9-c8af02f9e6e8 a627613c-232e-58d0-76b9-c8af02f9e6e8}"
2025-04-21T21:23:51.446Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:23:51.453Z [ERROR] server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: failed to get log: index=2 error="log not found"
2025-04-21T21:23:51.453Z [INFO]  creating new snapshot: path=/tmp/raft1690961956/snapshots/2-113-1745270631453.tmp
2025-04-21T21:23:51.453Z [INFO]  server-a9ae0dc7-3a61-bd7b-7389-8eb8cd580582: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:51.453Z [INFO]  server-a9ae0dc7-3a61-bd7b-7389-8eb8cd580582: copied to local snapshot: bytes=71
2025-04-21T21:23:51.453Z [INFO]  server-a9ae0dc7-3a61-bd7b-7389-8eb8cd580582: snapshot restore progress: id=2-113-1745270631453 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:51.454Z [INFO]  server-a9ae0dc7-3a61-bd7b-7389-8eb8cd580582: removing all old logs from log store
2025-04-21T21:23:51.454Z [INFO]  server-a9ae0dc7-3a61-bd7b-7389-8eb8cd580582: compacting logs: from=1 to=2
2025-04-21T21:23:51.454Z [INFO]  server-a9ae0dc7-3a61-bd7b-7389-8eb8cd580582: Installed remote snapshot
2025-04-21T21:23:51.454Z [INFO]  server-5edeff14-5674-f6ec-a845-9aefd5f9d33e: pipelining replication: peer="{Voter server-a9ae0dc7-3a61-bd7b-7389-8eb8cd580582 a9ae0dc7-3a61-bd7b-7389-8eb8cd580582}"
2025-04-21T21:23:52.453Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:23:51.446743365 +0000 UTC m=+69.968402688 (2 nodes), 400ns from start of poll, 1.728356576s from cluster start. Timeout at 2025-04-21 21:23:52.451753846 +0000 UTC m=+70.973413169, 1.005010481s after stability
2025-04-21T21:23:52.473Z [INFO]  server-29bb517f-ba48-7bb9-abc7-bd008812dc3c: aborting pipeline replication: peer="{Voter server-920105a4-76ce-0c61-2f11-4861e781a78f 920105a4-76ce-0c61-2f11-4861e781a78f}"
2025-04-21T21:23:52.473Z [INFO]  server-29bb517f-ba48-7bb9-abc7-bd008812dc3c: aborting pipeline replication: peer="{Voter server-05122540-e82e-374c-2b80-829fa5d82cc2 05122540-e82e-374c-2b80-829fa5d82cc2}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false
2025-04-21T21:23:52.473Z [INFO]  server-039804a4-38f7-a8ef-1b2d-081c90c22698: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-039804a4-38f7-a8ef-1b2d-081c90c22698 Address:039804a4-38f7-a8ef-1b2d-081c90c22698} {Suffrage:Voter ID:server-8d643c94-9ccd-3783-b700-01188e6e6763 Address:8d643c94-9ccd-3783-b700-01188e6e6763} {Suffrage:Voter ID:server-5f14858d-0ab8-63cd-6332-734c129cf315 Address:5f14858d-0ab8-63cd-6332-734c129cf315}]"
2025-04-21T21:23:52.473Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-039804a4-38f7-a8ef-1b2d-081c90c22698 Address:039804a4-38f7-a8ef-1b2d-081c90c22698} {Suffrage:Voter ID:server-8d643c94-9ccd-3783-b700-01188e6e6763 Address:8d643c94-9ccd-3783-b700-01188e6e6763} {Suffrage:Voter ID:server-5f14858d-0ab8-63cd-6332-734c129cf315 Address:5f14858d-0ab8-63cd-6332-734c129cf315}]"
2025-04-21T21:23:52.473Z [INFO]  server-5f14858d-0ab8-63cd-6332-734c129cf315: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-039804a4-38f7-a8ef-1b2d-081c90c22698 Address:039804a4-38f7-a8ef-1b2d-081c90c22698} {Suffrage:Voter ID:server-8d643c94-9ccd-3783-b700-01188e6e6763 Address:8d643c94-9ccd-3783-b700-01188e6e6763} {Suffrage:Voter ID:server-5f14858d-0ab8-63cd-6332-734c129cf315 Address:5f14858d-0ab8-63cd-6332-734c129cf315}]"
2025-04-21T21:23:52.473Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:52.473Z [INFO]  server-039804a4-38f7-a8ef-1b2d-081c90c22698: entering follower state: follower="Node at 039804a4-38f7-a8ef-1b2d-081c90c22698 [Follower]" leader-address= leader-id=
2025-04-21T21:23:52.474Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: entering follower state: follower="Node at 8d643c94-9ccd-3783-b700-01188e6e6763 [Follower]" leader-address= leader-id=
2025-04-21T21:23:52.474Z [INFO]  server-5f14858d-0ab8-63cd-6332-734c129cf315: entering follower state: follower="Node at 5f14858d-0ab8-63cd-6332-734c129cf315 [Follower]" leader-address= leader-id=
2025-04-21T21:23:53.077Z [WARN]  server-8d643c94-9ccd-3783-b700-01188e6e6763: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:53.077Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: entering candidate state: node="Node at 8d643c94-9ccd-3783-b700-01188e6e6763 [Candidate]" term=2
2025-04-21T21:23:53.077Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: election won: term=2 tally=2
2025-04-21T21:23:53.077Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: entering leader state: leader="Node at 8d643c94-9ccd-3783-b700-01188e6e6763 [Leader]"
2025-04-21T21:23:53.077Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: added peer, starting replication: peer=server-039804a4-38f7-a8ef-1b2d-081c90c22698
2025-04-21T21:23:53.077Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: added peer, starting replication: peer=server-5f14858d-0ab8-63cd-6332-734c129cf315
2025-04-21T21:23:53.077Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: pipelining replication: peer="{Voter server-5f14858d-0ab8-63cd-6332-734c129cf315 5f14858d-0ab8-63cd-6332-734c129cf315}"
2025-04-21T21:23:53.077Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: pipelining replication: peer="{Voter server-039804a4-38f7-a8ef-1b2d-081c90c22698 039804a4-38f7-a8ef-1b2d-081c90c22698}"
2025-04-21T21:23:54.085Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:53.077648966 +0000 UTC m=+71.599308289 (1 nodes), 603.733578ms from start of poll, 603.893299ms from cluster start. Timeout at 2025-04-21 21:23:54.082658006 +0000 UTC m=+72.604317329, 1.00500904s after stability
2025-04-21T21:23:54.085Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: starting snapshot up to: index=12
2025-04-21T21:23:54.085Z [INFO]  creating new snapshot: path=/tmp/raft2015908550/snapshots/2-12-1745270634085.tmp
2025-04-21T21:23:54.085Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: snapshot complete up to: index=12
2025-04-21T21:23:54.086Z [INFO]  creating new snapshot: path=/tmp/raft2015908550/snapshots/2-1013-1745270634086.tmp
2025-04-21T21:23:54.086Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: copied to local snapshot: bytes=71
2025-04-21T21:23:54.095Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: snapshot restore progress: id=2-1013-1745270634086 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:54.095Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: restored user snapshot: index=1013
2025-04-21T21:23:54.096Z [ERROR] server-8d643c94-9ccd-3783-b700-01188e6e6763: failed to get log: index=23 error="log not found"
2025-04-21T21:23:54.096Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: aborting pipeline replication: peer="{Voter server-039804a4-38f7-a8ef-1b2d-081c90c22698 039804a4-38f7-a8ef-1b2d-081c90c22698}"
2025-04-21T21:23:54.096Z [ERROR] server-8d643c94-9ccd-3783-b700-01188e6e6763: failed to get log: index=23 error="log not found"
2025-04-21T21:23:54.096Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: aborting pipeline replication: peer="{Voter server-5f14858d-0ab8-63cd-6332-734c129cf315 5f14858d-0ab8-63cd-6332-734c129cf315}"
2025-04-21T21:23:54.105Z [ERROR] server-8d643c94-9ccd-3783-b700-01188e6e6763: failed to get log: index=23 error="log not found"
2025-04-21T21:23:54.105Z [ERROR] server-8d643c94-9ccd-3783-b700-01188e6e6763: failed to get log: index=23 error="log not found"
2025-04-21T21:23:54.110Z [INFO]  creating new snapshot: path=/tmp/raft2728976888/snapshots/2-1013-1745270634110.tmp
2025-04-21T21:23:54.110Z [INFO]  server-039804a4-38f7-a8ef-1b2d-081c90c22698: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:54.110Z [INFO]  server-039804a4-38f7-a8ef-1b2d-081c90c22698: copied to local snapshot: bytes=71
2025-04-21T21:23:54.110Z [INFO]  server-039804a4-38f7-a8ef-1b2d-081c90c22698: snapshot restore progress: id=2-1013-1745270634110 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:54.110Z [INFO]  server-039804a4-38f7-a8ef-1b2d-081c90c22698: Installed remote snapshot
2025-04-21T21:23:54.110Z [WARN]  server-039804a4-38f7-a8ef-1b2d-081c90c22698: skipping application of old log: index=22
2025-04-21T21:23:54.110Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: pipelining replication: peer="{Voter server-039804a4-38f7-a8ef-1b2d-081c90c22698 039804a4-38f7-a8ef-1b2d-081c90c22698}"
2025-04-21T21:23:54.110Z [INFO]  creating new snapshot: path=/tmp/raft1934921502/snapshots/2-1013-1745270634110.tmp
2025-04-21T21:23:54.110Z [INFO]  server-5f14858d-0ab8-63cd-6332-734c129cf315: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:54.110Z [INFO]  server-5f14858d-0ab8-63cd-6332-734c129cf315: copied to local snapshot: bytes=71
2025-04-21T21:23:54.110Z [INFO]  server-5f14858d-0ab8-63cd-6332-734c129cf315: snapshot restore progress: id=2-1013-1745270634110 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:54.110Z [INFO]  server-5f14858d-0ab8-63cd-6332-734c129cf315: Installed remote snapshot
2025-04-21T21:23:54.110Z [WARN]  server-5f14858d-0ab8-63cd-6332-734c129cf315: skipping application of old log: index=22
2025-04-21T21:23:54.110Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: pipelining replication: peer="{Voter server-5f14858d-0ab8-63cd-6332-734c129cf315 5f14858d-0ab8-63cd-6332-734c129cf315}"
2025-04-21T21:23:54.110Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:23:55.117Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:23:54.11097579 +0000 UTC m=+72.632635113 (2 nodes), 440ns from start of poll, 1.637220123s from cluster start. Timeout at 2025-04-21 21:23:55.11598251 +0000 UTC m=+73.637641793, 1.00500668s after stability
2025-04-21T21:23:55.133Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: aborting pipeline replication: peer="{Voter server-5f14858d-0ab8-63cd-6332-734c129cf315 5f14858d-0ab8-63cd-6332-734c129cf315}"
2025-04-21T21:23:55.133Z [INFO]  server-8d643c94-9ccd-3783-b700-01188e6e6763: aborting pipeline replication: peer="{Voter server-039804a4-38f7-a8ef-1b2d-081c90c22698 039804a4-38f7-a8ef-1b2d-081c90c22698}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false
2025-04-21T21:23:55.133Z [INFO]  server-969a00b5-4153-425a-6f02-d2245c3f4100: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-969a00b5-4153-425a-6f02-d2245c3f4100 Address:969a00b5-4153-425a-6f02-d2245c3f4100} {Suffrage:Voter ID:server-784658f4-e814-7ad2-d226-71d0aefcc07b Address:784658f4-e814-7ad2-d226-71d0aefcc07b} {Suffrage:Voter ID:server-e5be7552-0993-6731-0a59-568f56151a93 Address:e5be7552-0993-6731-0a59-568f56151a93}]"
2025-04-21T21:23:55.133Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-969a00b5-4153-425a-6f02-d2245c3f4100 Address:969a00b5-4153-425a-6f02-d2245c3f4100} {Suffrage:Voter ID:server-784658f4-e814-7ad2-d226-71d0aefcc07b Address:784658f4-e814-7ad2-d226-71d0aefcc07b} {Suffrage:Voter ID:server-e5be7552-0993-6731-0a59-568f56151a93 Address:e5be7552-0993-6731-0a59-568f56151a93}]"
2025-04-21T21:23:55.133Z [INFO]  server-e5be7552-0993-6731-0a59-568f56151a93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-969a00b5-4153-425a-6f02-d2245c3f4100 Address:969a00b5-4153-425a-6f02-d2245c3f4100} {Suffrage:Voter ID:server-784658f4-e814-7ad2-d226-71d0aefcc07b Address:784658f4-e814-7ad2-d226-71d0aefcc07b} {Suffrage:Voter ID:server-e5be7552-0993-6731-0a59-568f56151a93 Address:e5be7552-0993-6731-0a59-568f56151a93}]"
2025-04-21T21:23:55.133Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:55.133Z [INFO]  server-969a00b5-4153-425a-6f02-d2245c3f4100: entering follower state: follower="Node at 969a00b5-4153-425a-6f02-d2245c3f4100 [Follower]" leader-address= leader-id=
2025-04-21T21:23:55.133Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: entering follower state: follower="Node at 784658f4-e814-7ad2-d226-71d0aefcc07b [Follower]" leader-address= leader-id=
2025-04-21T21:23:55.134Z [INFO]  server-e5be7552-0993-6731-0a59-568f56151a93: entering follower state: follower="Node at e5be7552-0993-6731-0a59-568f56151a93 [Follower]" leader-address= leader-id=
2025-04-21T21:23:55.713Z [WARN]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:55.713Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: entering candidate state: node="Node at 784658f4-e814-7ad2-d226-71d0aefcc07b [Candidate]" term=2
2025-04-21T21:23:55.713Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: election won: term=2 tally=2
2025-04-21T21:23:55.713Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: entering leader state: leader="Node at 784658f4-e814-7ad2-d226-71d0aefcc07b [Leader]"
2025-04-21T21:23:55.713Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: added peer, starting replication: peer=server-969a00b5-4153-425a-6f02-d2245c3f4100
2025-04-21T21:23:55.713Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: added peer, starting replication: peer=server-e5be7552-0993-6731-0a59-568f56151a93
2025-04-21T21:23:55.713Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: pipelining replication: peer="{Voter server-e5be7552-0993-6731-0a59-568f56151a93 e5be7552-0993-6731-0a59-568f56151a93}"
2025-04-21T21:23:55.713Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: pipelining replication: peer="{Voter server-969a00b5-4153-425a-6f02-d2245c3f4100 969a00b5-4153-425a-6f02-d2245c3f4100}"
2025-04-21T21:23:56.721Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:55.713658242 +0000 UTC m=+74.235317525 (1 nodes), 579.719617ms from start of poll, 579.886778ms from cluster start. Timeout at 2025-04-21 21:23:56.718671322 +0000 UTC m=+75.240330605, 1.00501308s after stability
2025-04-21T21:23:56.721Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: starting snapshot up to: index=12
2025-04-21T21:23:56.721Z [INFO]  creating new snapshot: path=/tmp/raft590832085/snapshots/2-12-1745270636721.tmp
2025-04-21T21:23:56.721Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: snapshot complete up to: index=12
2025-04-21T21:23:56.722Z [INFO]  creating new snapshot: path=/tmp/raft590832085/snapshots/2-1013-1745270636722.tmp
2025-04-21T21:23:56.722Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: copied to local snapshot: bytes=71
2025-04-21T21:23:56.722Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: snapshot restore progress: id=2-1013-1745270636722 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:56.722Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: removing all old logs from log store
2025-04-21T21:23:56.722Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: compacting logs: from=1 to=22
2025-04-21T21:23:56.722Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: restored user snapshot: index=1013
2025-04-21T21:23:56.722Z [ERROR] server-784658f4-e814-7ad2-d226-71d0aefcc07b: failed to get log: index=22 error="log not found"
2025-04-21T21:23:56.722Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: aborting pipeline replication: peer="{Voter server-969a00b5-4153-425a-6f02-d2245c3f4100 969a00b5-4153-425a-6f02-d2245c3f4100}"
2025-04-21T21:23:56.722Z [ERROR] server-784658f4-e814-7ad2-d226-71d0aefcc07b: failed to get log: index=20 error="log not found"
2025-04-21T21:23:56.722Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: aborting pipeline replication: peer="{Voter server-e5be7552-0993-6731-0a59-568f56151a93 e5be7552-0993-6731-0a59-568f56151a93}"
2025-04-21T21:23:56.722Z [ERROR] server-784658f4-e814-7ad2-d226-71d0aefcc07b: failed to get log: index=20 error="log not found"
2025-04-21T21:23:56.722Z [INFO]  creating new snapshot: path=/tmp/raft1297186581/snapshots/2-1013-1745270636722.tmp
2025-04-21T21:23:56.722Z [INFO]  server-e5be7552-0993-6731-0a59-568f56151a93: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:56.723Z [INFO]  server-e5be7552-0993-6731-0a59-568f56151a93: copied to local snapshot: bytes=71
2025-04-21T21:23:56.723Z [INFO]  server-e5be7552-0993-6731-0a59-568f56151a93: snapshot restore progress: id=2-1013-1745270636722 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:56.723Z [INFO]  server-e5be7552-0993-6731-0a59-568f56151a93: removing all old logs from log store
2025-04-21T21:23:56.723Z [INFO]  server-e5be7552-0993-6731-0a59-568f56151a93: compacting logs: from=1 to=20
2025-04-21T21:23:56.723Z [INFO]  server-e5be7552-0993-6731-0a59-568f56151a93: Installed remote snapshot
2025-04-21T21:23:56.723Z [WARN]  server-e5be7552-0993-6731-0a59-568f56151a93: skipping application of old log: index=22
2025-04-21T21:23:56.723Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: pipelining replication: peer="{Voter server-e5be7552-0993-6731-0a59-568f56151a93 e5be7552-0993-6731-0a59-568f56151a93}"
2025-04-21T21:23:56.723Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:23:56.733Z [ERROR] server-784658f4-e814-7ad2-d226-71d0aefcc07b: failed to get log: index=22 error="log not found"
2025-04-21T21:23:56.733Z [INFO]  creating new snapshot: path=/tmp/raft2178805011/snapshots/2-1013-1745270636733.tmp
2025-04-21T21:23:56.733Z [INFO]  server-969a00b5-4153-425a-6f02-d2245c3f4100: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:56.733Z [INFO]  server-969a00b5-4153-425a-6f02-d2245c3f4100: copied to local snapshot: bytes=71
2025-04-21T21:23:56.733Z [INFO]  server-969a00b5-4153-425a-6f02-d2245c3f4100: snapshot restore progress: id=2-1013-1745270636733 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:23:56.733Z [INFO]  server-969a00b5-4153-425a-6f02-d2245c3f4100: removing all old logs from log store
2025-04-21T21:23:56.733Z [INFO]  server-969a00b5-4153-425a-6f02-d2245c3f4100: compacting logs: from=1 to=22
2025-04-21T21:23:56.733Z [INFO]  server-969a00b5-4153-425a-6f02-d2245c3f4100: Installed remote snapshot
2025-04-21T21:23:56.733Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: pipelining replication: peer="{Voter server-969a00b5-4153-425a-6f02-d2245c3f4100 969a00b5-4153-425a-6f02-d2245c3f4100}"
2025-04-21T21:23:57.729Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:23:56.723312707 +0000 UTC m=+75.244972030 (2 nodes), 520ns from start of poll, 1.589541283s from cluster start. Timeout at 2025-04-21 21:23:57.728354388 +0000 UTC m=+76.250013711, 1.005041681s after stability
2025-04-21T21:23:57.749Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: aborting pipeline replication: peer="{Voter server-e5be7552-0993-6731-0a59-568f56151a93 e5be7552-0993-6731-0a59-568f56151a93}"
2025-04-21T21:23:57.749Z [INFO]  server-784658f4-e814-7ad2-d226-71d0aefcc07b: aborting pipeline replication: peer="{Voter server-969a00b5-4153-425a-6f02-d2245c3f4100 969a00b5-4153-425a-6f02-d2245c3f4100}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true
2025-04-21T21:23:57.749Z [INFO]  server-12f5ce10-4f9f-8c92-b126-b5a5da34b077: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12f5ce10-4f9f-8c92-b126-b5a5da34b077 Address:12f5ce10-4f9f-8c92-b126-b5a5da34b077} {Suffrage:Voter ID:server-78485576-14b8-3a93-b7ca-6424e74709c0 Address:78485576-14b8-3a93-b7ca-6424e74709c0} {Suffrage:Voter ID:server-5f41a4fd-ed67-d8e7-6a57-a377fa864998 Address:5f41a4fd-ed67-d8e7-6a57-a377fa864998}]"
2025-04-21T21:23:57.749Z [INFO]  server-78485576-14b8-3a93-b7ca-6424e74709c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12f5ce10-4f9f-8c92-b126-b5a5da34b077 Address:12f5ce10-4f9f-8c92-b126-b5a5da34b077} {Suffrage:Voter ID:server-78485576-14b8-3a93-b7ca-6424e74709c0 Address:78485576-14b8-3a93-b7ca-6424e74709c0} {Suffrage:Voter ID:server-5f41a4fd-ed67-d8e7-6a57-a377fa864998 Address:5f41a4fd-ed67-d8e7-6a57-a377fa864998}]"
2025-04-21T21:23:57.750Z [INFO]  server-5f41a4fd-ed67-d8e7-6a57-a377fa864998: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12f5ce10-4f9f-8c92-b126-b5a5da34b077 Address:12f5ce10-4f9f-8c92-b126-b5a5da34b077} {Suffrage:Voter ID:server-78485576-14b8-3a93-b7ca-6424e74709c0 Address:78485576-14b8-3a93-b7ca-6424e74709c0} {Suffrage:Voter ID:server-5f41a4fd-ed67-d8e7-6a57-a377fa864998 Address:5f41a4fd-ed67-d8e7-6a57-a377fa864998}]"
2025-04-21T21:23:57.750Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:57.750Z [INFO]  server-12f5ce10-4f9f-8c92-b126-b5a5da34b077: entering follower state: follower="Node at 12f5ce10-4f9f-8c92-b126-b5a5da34b077 [Follower]" leader-address= leader-id=
2025-04-21T21:23:57.750Z [INFO]  server-78485576-14b8-3a93-b7ca-6424e74709c0: entering follower state: follower="Node at 78485576-14b8-3a93-b7ca-6424e74709c0 [Follower]" leader-address= leader-id=
2025-04-21T21:23:57.750Z [INFO]  server-5f41a4fd-ed67-d8e7-6a57-a377fa864998: entering follower state: follower="Node at 5f41a4fd-ed67-d8e7-6a57-a377fa864998 [Follower]" leader-address= leader-id=
2025-04-21T21:23:58.485Z [WARN]  server-78485576-14b8-3a93-b7ca-6424e74709c0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:23:58.485Z [INFO]  server-78485576-14b8-3a93-b7ca-6424e74709c0: entering candidate state: node="Node at 78485576-14b8-3a93-b7ca-6424e74709c0 [Candidate]" term=2
2025-04-21T21:23:58.485Z [INFO]  server-78485576-14b8-3a93-b7ca-6424e74709c0: election won: term=2 tally=2
2025-04-21T21:23:58.485Z [INFO]  server-78485576-14b8-3a93-b7ca-6424e74709c0: entering leader state: leader="Node at 78485576-14b8-3a93-b7ca-6424e74709c0 [Leader]"
2025-04-21T21:23:58.485Z [INFO]  server-78485576-14b8-3a93-b7ca-6424e74709c0: added peer, starting replication: peer=server-12f5ce10-4f9f-8c92-b126-b5a5da34b077
2025-04-21T21:23:58.485Z [INFO]  server-78485576-14b8-3a93-b7ca-6424e74709c0: added peer, starting replication: peer=server-5f41a4fd-ed67-d8e7-6a57-a377fa864998
2025-04-21T21:23:58.485Z [INFO]  server-78485576-14b8-3a93-b7ca-6424e74709c0: pipelining replication: peer="{Voter server-5f41a4fd-ed67-d8e7-6a57-a377fa864998 5f41a4fd-ed67-d8e7-6a57-a377fa864998}"
2025-04-21T21:23:58.485Z [INFO]  server-78485576-14b8-3a93-b7ca-6424e74709c0: pipelining replication: peer="{Voter server-12f5ce10-4f9f-8c92-b126-b5a5da34b077 12f5ce10-4f9f-8c92-b126-b5a5da34b077}"
2025-04-21T21:23:59.493Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:23:58.485982074 +0000 UTC m=+77.007641397 (1 nodes), 735.948826ms from start of poll, 736.145147ms from cluster start. Timeout at 2025-04-21 21:23:59.490990354 +0000 UTC m=+78.012649677, 1.00500828s after stability
2025-04-21T21:23:59.493Z [INFO]  server-78485576-14b8-3a93-b7ca-6424e74709c0: starting snapshot up to: index=12
2025-04-21T21:23:59.493Z [INFO]  creating new snapshot: path=/tmp/raft770800215/snapshots/2-12-1745270639493.tmp
2025-04-21T21:23:59.494Z [INFO]  server-78485576-14b8-3a93-b7ca-6424e74709c0: snapshot complete up to: index=12
2025-04-21T21:23:59.494Z [INFO]  server-8d67e47e-b7e3-e880-d978-3f3ef3b28803: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d67e47e-b7e3-e880-d978-3f3ef3b28803 Address:8d67e47e-b7e3-e880-d978-3f3ef3b28803} {Suffrage:Voter ID:server-d2b3ce7c-0861-666b-094e-8202c62790fe Address:d2b3ce7c-0861-666b-094e-8202c62790fe} {Suffrage:Voter ID:server-28fe4499-ceb2-da83-d86c-23b753aed228 Address:28fe4499-ceb2-da83-d86c-23b753aed228}]"
2025-04-21T21:23:59.494Z [INFO]  server-d2b3ce7c-0861-666b-094e-8202c62790fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d67e47e-b7e3-e880-d978-3f3ef3b28803 Address:8d67e47e-b7e3-e880-d978-3f3ef3b28803} {Suffrage:Voter ID:server-d2b3ce7c-0861-666b-094e-8202c62790fe Address:d2b3ce7c-0861-666b-094e-8202c62790fe} {Suffrage:Voter ID:server-28fe4499-ceb2-da83-d86c-23b753aed228 Address:28fe4499-ceb2-da83-d86c-23b753aed228}]"
2025-04-21T21:23:59.494Z [INFO]  server-28fe4499-ceb2-da83-d86c-23b753aed228: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d67e47e-b7e3-e880-d978-3f3ef3b28803 Address:8d67e47e-b7e3-e880-d978-3f3ef3b28803} {Suffrage:Voter ID:server-d2b3ce7c-0861-666b-094e-8202c62790fe Address:d2b3ce7c-0861-666b-094e-8202c62790fe} {Suffrage:Voter ID:server-28fe4499-ceb2-da83-d86c-23b753aed228 Address:28fe4499-ceb2-da83-d86c-23b753aed228}]"
2025-04-21T21:23:59.494Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:23:59.495Z [INFO]  server-8d67e47e-b7e3-e880-d978-3f3ef3b28803: entering follower state: follower="Node at 8d67e47e-b7e3-e880-d978-3f3ef3b28803 [Follower]" leader-address= leader-id=
2025-04-21T21:23:59.495Z [INFO]  server-d2b3ce7c-0861-666b-094e-8202c62790fe: entering follower state: follower="Node at d2b3ce7c-0861-666b-094e-8202c62790fe [Follower]" leader-address= leader-id=
2025-04-21T21:23:59.495Z [INFO]  server-28fe4499-ceb2-da83-d86c-23b753aed228: entering follower state: follower="Node at 28fe4499-ceb2-da83-d86c-23b753aed228 [Follower]" leader-address= leader-id=
2025-04-21T21:24:00.289Z [WARN]  server-d2b3ce7c-0861-666b-094e-8202c62790fe: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:00.289Z [INFO]  server-d2b3ce7c-0861-666b-094e-8202c62790fe: entering candidate state: node="Node at d2b3ce7c-0861-666b-094e-8202c62790fe [Candidate]" term=2
2025-04-21T21:24:00.289Z [INFO]  server-d2b3ce7c-0861-666b-094e-8202c62790fe: election won: term=2 tally=2
2025-04-21T21:24:00.289Z [INFO]  server-d2b3ce7c-0861-666b-094e-8202c62790fe: entering leader state: leader="Node at d2b3ce7c-0861-666b-094e-8202c62790fe [Leader]"
2025-04-21T21:24:00.289Z [INFO]  server-d2b3ce7c-0861-666b-094e-8202c62790fe: added peer, starting replication: peer=server-8d67e47e-b7e3-e880-d978-3f3ef3b28803
2025-04-21T21:24:00.289Z [INFO]  server-d2b3ce7c-0861-666b-094e-8202c62790fe: added peer, starting replication: peer=server-28fe4499-ceb2-da83-d86c-23b753aed228
2025-04-21T21:24:00.289Z [INFO]  server-d2b3ce7c-0861-666b-094e-8202c62790fe: pipelining replication: peer="{Voter server-28fe4499-ceb2-da83-d86c-23b753aed228 28fe4499-ceb2-da83-d86c-23b753aed228}"
2025-04-21T21:24:00.289Z [INFO]  server-d2b3ce7c-0861-666b-094e-8202c62790fe: pipelining replication: peer="{Voter server-8d67e47e-b7e3-e880-d978-3f3ef3b28803 8d67e47e-b7e3-e880-d978-3f3ef3b28803}"
2025-04-21T21:24:01.297Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:00.289803357 +0000 UTC m=+78.811462680 (1 nodes), 794.806061ms from start of poll, 794.977783ms from cluster start. Timeout at 2025-04-21 21:24:01.294806677 +0000 UTC m=+79.816465960, 1.00500328s after stability
2025-04-21T21:24:01.297Z [INFO]  creating new snapshot: path=/tmp/raft3721387209/snapshots/2-1013-1745270641297.tmp
2025-04-21T21:24:01.297Z [INFO]  server-d2b3ce7c-0861-666b-094e-8202c62790fe: copied to local snapshot: bytes=71
2025-04-21T21:24:01.297Z [INFO]  server-d2b3ce7c-0861-666b-094e-8202c62790fe: snapshot restore progress: id=2-1013-1745270641297 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:01.297Z [INFO]  server-d2b3ce7c-0861-666b-094e-8202c62790fe: restored user snapshot: index=1013
2025-04-21T21:24:01.297Z [ERROR] server-d2b3ce7c-0861-666b-094e-8202c62790fe: failed to get log: index=3 error="log not found"
2025-04-21T21:24:01.297Z [INFO]  server-d2b3ce7c-0861-666b-094e-8202c62790fe: aborting pipeline replication: peer="{Voter server-28fe4499-ceb2-da83-d86c-23b753aed228 28fe4499-ceb2-da83-d86c-23b753aed228}"
2025-04-21T21:24:01.298Z [ERROR] server-d2b3ce7c-0861-666b-094e-8202c62790fe: failed to get log: index=3 error="log not found"
2025-04-21T21:24:01.298Z [INFO]  server-d2b3ce7c-0861-666b-094e-8202c62790fe: aborting pipeline replication: peer="{Voter server-8d67e47e-b7e3-e880-d978-3f3ef3b28803 8d67e47e-b7e3-e880-d978-3f3ef3b28803}"
2025-04-21T21:24:01.305Z [ERROR] server-d2b3ce7c-0861-666b-094e-8202c62790fe: failed to get log: index=3 error="log not found"
2025-04-21T21:24:01.305Z [INFO]  creating new snapshot: path=/tmp/raft917637662/snapshots/2-1013-1745270641305.tmp
2025-04-21T21:24:01.305Z [INFO]  server-8d67e47e-b7e3-e880-d978-3f3ef3b28803: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:01.305Z [INFO]  server-8d67e47e-b7e3-e880-d978-3f3ef3b28803: copied to local snapshot: bytes=71
2025-04-21T21:24:01.305Z [INFO]  server-8d67e47e-b7e3-e880-d978-3f3ef3b28803: snapshot restore progress: id=2-1013-1745270641305 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:01.305Z [INFO]  server-8d67e47e-b7e3-e880-d978-3f3ef3b28803: Installed remote snapshot
2025-04-21T21:24:01.305Z [INFO]  server-d2b3ce7c-0861-666b-094e-8202c62790fe: pipelining replication: peer="{Voter server-8d67e47e-b7e3-e880-d978-3f3ef3b28803 8d67e47e-b7e3-e880-d978-3f3ef3b28803}"
2025-04-21T21:24:01.306Z [ERROR] server-d2b3ce7c-0861-666b-094e-8202c62790fe: failed to get log: index=3 error="log not found"
2025-04-21T21:24:01.306Z [INFO]  creating new snapshot: path=/tmp/raft175885738/snapshots/2-1013-1745270641306.tmp
2025-04-21T21:24:01.306Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:01.306Z [INFO]  server-28fe4499-ceb2-da83-d86c-23b753aed228: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:01.306Z [INFO]  server-28fe4499-ceb2-da83-d86c-23b753aed228: copied to local snapshot: bytes=71
2025-04-21T21:24:01.306Z [INFO]  server-28fe4499-ceb2-da83-d86c-23b753aed228: snapshot restore progress: id=2-1013-1745270641306 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:01.306Z [INFO]  server-28fe4499-ceb2-da83-d86c-23b753aed228: Installed remote snapshot
2025-04-21T21:24:01.306Z [INFO]  server-d2b3ce7c-0861-666b-094e-8202c62790fe: pipelining replication: peer="{Voter server-28fe4499-ceb2-da83-d86c-23b753aed228 28fe4499-ceb2-da83-d86c-23b753aed228}"
2025-04-21T21:24:02.321Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:01.306523031 +0000 UTC m=+79.828182354 (2 nodes), 440ns from start of poll, 1.811697457s from cluster start. Timeout at 2025-04-21 21:24:02.311557872 +0000 UTC m=+80.833217155, 1.005034801s after stability
2025-04-21T21:24:02.337Z [INFO]  server-78485576-14b8-3a93-b7ca-6424e74709c0: aborting pipeline replication: peer="{Voter server-5f41a4fd-ed67-d8e7-6a57-a377fa864998 5f41a4fd-ed67-d8e7-6a57-a377fa864998}"
2025-04-21T21:24:02.337Z [INFO]  server-78485576-14b8-3a93-b7ca-6424e74709c0: aborting pipeline replication: peer="{Voter server-12f5ce10-4f9f-8c92-b126-b5a5da34b077 12f5ce10-4f9f-8c92-b126-b5a5da34b077}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true
2025-04-21T21:24:02.337Z [INFO]  server-faae3b55-1ee3-586f-c712-62275f16df43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-faae3b55-1ee3-586f-c712-62275f16df43 Address:faae3b55-1ee3-586f-c712-62275f16df43} {Suffrage:Voter ID:server-3b5d7097-afed-7765-41ec-724ee6bfd3c1 Address:3b5d7097-afed-7765-41ec-724ee6bfd3c1} {Suffrage:Voter ID:server-1e588b5d-88b4-d915-90ad-562c0a1087d1 Address:1e588b5d-88b4-d915-90ad-562c0a1087d1}]"
2025-04-21T21:24:02.338Z [INFO]  server-3b5d7097-afed-7765-41ec-724ee6bfd3c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-faae3b55-1ee3-586f-c712-62275f16df43 Address:faae3b55-1ee3-586f-c712-62275f16df43} {Suffrage:Voter ID:server-3b5d7097-afed-7765-41ec-724ee6bfd3c1 Address:3b5d7097-afed-7765-41ec-724ee6bfd3c1} {Suffrage:Voter ID:server-1e588b5d-88b4-d915-90ad-562c0a1087d1 Address:1e588b5d-88b4-d915-90ad-562c0a1087d1}]"
2025-04-21T21:24:02.338Z [INFO]  server-1e588b5d-88b4-d915-90ad-562c0a1087d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-faae3b55-1ee3-586f-c712-62275f16df43 Address:faae3b55-1ee3-586f-c712-62275f16df43} {Suffrage:Voter ID:server-3b5d7097-afed-7765-41ec-724ee6bfd3c1 Address:3b5d7097-afed-7765-41ec-724ee6bfd3c1} {Suffrage:Voter ID:server-1e588b5d-88b4-d915-90ad-562c0a1087d1 Address:1e588b5d-88b4-d915-90ad-562c0a1087d1}]"
2025-04-21T21:24:02.338Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:02.338Z [INFO]  server-faae3b55-1ee3-586f-c712-62275f16df43: entering follower state: follower="Node at faae3b55-1ee3-586f-c712-62275f16df43 [Follower]" leader-address= leader-id=
2025-04-21T21:24:02.338Z [INFO]  server-3b5d7097-afed-7765-41ec-724ee6bfd3c1: entering follower state: follower="Node at 3b5d7097-afed-7765-41ec-724ee6bfd3c1 [Follower]" leader-address= leader-id=
2025-04-21T21:24:02.338Z [INFO]  server-1e588b5d-88b4-d915-90ad-562c0a1087d1: entering follower state: follower="Node at 1e588b5d-88b4-d915-90ad-562c0a1087d1 [Follower]" leader-address= leader-id=
2025-04-21T21:24:02.877Z [WARN]  server-1e588b5d-88b4-d915-90ad-562c0a1087d1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:02.877Z [INFO]  server-1e588b5d-88b4-d915-90ad-562c0a1087d1: entering candidate state: node="Node at 1e588b5d-88b4-d915-90ad-562c0a1087d1 [Candidate]" term=2
2025-04-21T21:24:02.877Z [INFO]  server-1e588b5d-88b4-d915-90ad-562c0a1087d1: election won: term=2 tally=2
2025-04-21T21:24:02.877Z [INFO]  server-1e588b5d-88b4-d915-90ad-562c0a1087d1: entering leader state: leader="Node at 1e588b5d-88b4-d915-90ad-562c0a1087d1 [Leader]"
2025-04-21T21:24:02.877Z [INFO]  server-1e588b5d-88b4-d915-90ad-562c0a1087d1: added peer, starting replication: peer=server-faae3b55-1ee3-586f-c712-62275f16df43
2025-04-21T21:24:02.877Z [INFO]  server-1e588b5d-88b4-d915-90ad-562c0a1087d1: added peer, starting replication: peer=server-3b5d7097-afed-7765-41ec-724ee6bfd3c1
2025-04-21T21:24:02.877Z [INFO]  server-1e588b5d-88b4-d915-90ad-562c0a1087d1: pipelining replication: peer="{Voter server-3b5d7097-afed-7765-41ec-724ee6bfd3c1 3b5d7097-afed-7765-41ec-724ee6bfd3c1}"
2025-04-21T21:24:02.877Z [INFO]  server-1e588b5d-88b4-d915-90ad-562c0a1087d1: pipelining replication: peer="{Voter server-faae3b55-1ee3-586f-c712-62275f16df43 faae3b55-1ee3-586f-c712-62275f16df43}"
2025-04-21T21:24:03.885Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:02.877758431 +0000 UTC m=+81.399417754 (1 nodes), 539.677027ms from start of poll, 539.884669ms from cluster start. Timeout at 2025-04-21 21:24:03.882793511 +0000 UTC m=+82.404452794, 1.00503504s after stability
2025-04-21T21:24:03.885Z [INFO]  server-1e588b5d-88b4-d915-90ad-562c0a1087d1: starting snapshot up to: index=12
2025-04-21T21:24:03.885Z [INFO]  creating new snapshot: path=/tmp/raft168516665/snapshots/2-12-1745270643885.tmp
2025-04-21T21:24:03.886Z [INFO]  server-1e588b5d-88b4-d915-90ad-562c0a1087d1: snapshot complete up to: index=12
2025-04-21T21:24:03.886Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90d49745-855a-0054-7b00-ba6223225344 Address:90d49745-855a-0054-7b00-ba6223225344} {Suffrage:Voter ID:server-6d97a600-0e63-578c-ba30-912072488854 Address:6d97a600-0e63-578c-ba30-912072488854} {Suffrage:Voter ID:server-79621e7d-cb71-12cd-53f0-273d1b28ef25 Address:79621e7d-cb71-12cd-53f0-273d1b28ef25}]"
2025-04-21T21:24:03.886Z [INFO]  server-6d97a600-0e63-578c-ba30-912072488854: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90d49745-855a-0054-7b00-ba6223225344 Address:90d49745-855a-0054-7b00-ba6223225344} {Suffrage:Voter ID:server-6d97a600-0e63-578c-ba30-912072488854 Address:6d97a600-0e63-578c-ba30-912072488854} {Suffrage:Voter ID:server-79621e7d-cb71-12cd-53f0-273d1b28ef25 Address:79621e7d-cb71-12cd-53f0-273d1b28ef25}]"
2025-04-21T21:24:03.893Z [INFO]  server-79621e7d-cb71-12cd-53f0-273d1b28ef25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-90d49745-855a-0054-7b00-ba6223225344 Address:90d49745-855a-0054-7b00-ba6223225344} {Suffrage:Voter ID:server-6d97a600-0e63-578c-ba30-912072488854 Address:6d97a600-0e63-578c-ba30-912072488854} {Suffrage:Voter ID:server-79621e7d-cb71-12cd-53f0-273d1b28ef25 Address:79621e7d-cb71-12cd-53f0-273d1b28ef25}]"
2025-04-21T21:24:03.893Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:03.897Z [INFO]  server-79621e7d-cb71-12cd-53f0-273d1b28ef25: entering follower state: follower="Node at 79621e7d-cb71-12cd-53f0-273d1b28ef25 [Follower]" leader-address= leader-id=
2025-04-21T21:24:03.897Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: entering follower state: follower="Node at 90d49745-855a-0054-7b00-ba6223225344 [Follower]" leader-address= leader-id=
2025-04-21T21:24:03.897Z [INFO]  server-6d97a600-0e63-578c-ba30-912072488854: entering follower state: follower="Node at 6d97a600-0e63-578c-ba30-912072488854 [Follower]" leader-address= leader-id=
2025-04-21T21:24:04.461Z [WARN]  server-90d49745-855a-0054-7b00-ba6223225344: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:04.461Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: entering candidate state: node="Node at 90d49745-855a-0054-7b00-ba6223225344 [Candidate]" term=2
2025-04-21T21:24:04.461Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: election won: term=2 tally=2
2025-04-21T21:24:04.461Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: entering leader state: leader="Node at 90d49745-855a-0054-7b00-ba6223225344 [Leader]"
2025-04-21T21:24:04.461Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: added peer, starting replication: peer=server-6d97a600-0e63-578c-ba30-912072488854
2025-04-21T21:24:04.461Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: added peer, starting replication: peer=server-79621e7d-cb71-12cd-53f0-273d1b28ef25
2025-04-21T21:24:04.461Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: pipelining replication: peer="{Voter server-79621e7d-cb71-12cd-53f0-273d1b28ef25 79621e7d-cb71-12cd-53f0-273d1b28ef25}"
2025-04-21T21:24:04.461Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: pipelining replication: peer="{Voter server-6d97a600-0e63-578c-ba30-912072488854 6d97a600-0e63-578c-ba30-912072488854}"
2025-04-21T21:24:05.469Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:04.461648716 +0000 UTC m=+82.983308039 (1 nodes), 568.079978ms from start of poll, 574.915144ms from cluster start. Timeout at 2025-04-21 21:24:05.466649956 +0000 UTC m=+83.988309279, 1.00500124s after stability
2025-04-21T21:24:05.469Z [INFO]  creating new snapshot: path=/tmp/raft3505392412/snapshots/2-1013-1745270645469.tmp
2025-04-21T21:24:05.470Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: copied to local snapshot: bytes=71
2025-04-21T21:24:05.470Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: snapshot restore progress: id=2-1013-1745270645469 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:05.470Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: removing all old logs from log store
2025-04-21T21:24:05.470Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: compacting logs: from=1 to=2
2025-04-21T21:24:05.470Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: restored user snapshot: index=1013
2025-04-21T21:24:05.470Z [ERROR] server-90d49745-855a-0054-7b00-ba6223225344: failed to get log: index=2 error="log not found"
2025-04-21T21:24:05.470Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: aborting pipeline replication: peer="{Voter server-6d97a600-0e63-578c-ba30-912072488854 6d97a600-0e63-578c-ba30-912072488854}"
2025-04-21T21:24:05.470Z [ERROR] server-90d49745-855a-0054-7b00-ba6223225344: failed to get log: index=2 error="log not found"
2025-04-21T21:24:05.470Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: aborting pipeline replication: peer="{Voter server-79621e7d-cb71-12cd-53f0-273d1b28ef25 79621e7d-cb71-12cd-53f0-273d1b28ef25}"
2025-04-21T21:24:05.477Z [ERROR] server-90d49745-855a-0054-7b00-ba6223225344: failed to get log: index=2 error="log not found"
2025-04-21T21:24:05.477Z [ERROR] server-90d49745-855a-0054-7b00-ba6223225344: failed to get log: index=2 error="log not found"
2025-04-21T21:24:05.477Z [INFO]  creating new snapshot: path=/tmp/raft3017953712/snapshots/2-1013-1745270645477.tmp
2025-04-21T21:24:05.477Z [INFO]  creating new snapshot: path=/tmp/raft1085773543/snapshots/2-1013-1745270645477.tmp
2025-04-21T21:24:05.477Z [INFO]  server-79621e7d-cb71-12cd-53f0-273d1b28ef25: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:05.477Z [INFO]  server-6d97a600-0e63-578c-ba30-912072488854: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:05.477Z [INFO]  server-79621e7d-cb71-12cd-53f0-273d1b28ef25: copied to local snapshot: bytes=71
2025-04-21T21:24:05.477Z [INFO]  server-6d97a600-0e63-578c-ba30-912072488854: copied to local snapshot: bytes=71
2025-04-21T21:24:05.477Z [INFO]  server-79621e7d-cb71-12cd-53f0-273d1b28ef25: snapshot restore progress: id=2-1013-1745270645477 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:05.477Z [INFO]  server-6d97a600-0e63-578c-ba30-912072488854: snapshot restore progress: id=2-1013-1745270645477 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:05.477Z [INFO]  server-79621e7d-cb71-12cd-53f0-273d1b28ef25: removing all old logs from log store
2025-04-21T21:24:05.477Z [INFO]  server-79621e7d-cb71-12cd-53f0-273d1b28ef25: compacting logs: from=1 to=2
2025-04-21T21:24:05.477Z [INFO]  server-79621e7d-cb71-12cd-53f0-273d1b28ef25: Installed remote snapshot
2025-04-21T21:24:05.477Z [INFO]  server-6d97a600-0e63-578c-ba30-912072488854: removing all old logs from log store
2025-04-21T21:24:05.477Z [INFO]  server-6d97a600-0e63-578c-ba30-912072488854: compacting logs: from=1 to=2
2025-04-21T21:24:05.477Z [INFO]  server-6d97a600-0e63-578c-ba30-912072488854: Installed remote snapshot
2025-04-21T21:24:05.477Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: pipelining replication: peer="{Voter server-6d97a600-0e63-578c-ba30-912072488854 6d97a600-0e63-578c-ba30-912072488854}"
2025-04-21T21:24:05.477Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:05.478Z [INFO]  server-90d49745-855a-0054-7b00-ba6223225344: pipelining replication: peer="{Voter server-79621e7d-cb71-12cd-53f0-273d1b28ef25 79621e7d-cb71-12cd-53f0-273d1b28ef25}"
2025-04-21T21:24:06.485Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:05.478007107 +0000 UTC m=+83.999666430 (2 nodes), 680ns from start of poll, 1.591273535s from cluster start. Timeout at 2025-04-21 21:24:06.483041588 +0000 UTC m=+85.004700911, 1.005034481s after stability
2025-04-21T21:24:06.501Z [INFO]  server-1e588b5d-88b4-d915-90ad-562c0a1087d1: aborting pipeline replication: peer="{Voter server-3b5d7097-afed-7765-41ec-724ee6bfd3c1 3b5d7097-afed-7765-41ec-724ee6bfd3c1}"
2025-04-21T21:24:06.501Z [INFO]  server-1e588b5d-88b4-d915-90ad-562c0a1087d1: aborting pipeline replication: peer="{Voter server-faae3b55-1ee3-586f-c712-62275f16df43 faae3b55-1ee3-586f-c712-62275f16df43}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false
2025-04-21T21:24:06.501Z [INFO]  server-2eeb5164-93d0-ac37-d184-b5aae1ad9ec2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eeb5164-93d0-ac37-d184-b5aae1ad9ec2 Address:2eeb5164-93d0-ac37-d184-b5aae1ad9ec2} {Suffrage:Voter ID:server-47ae1bf2-c261-600f-e912-c3ae463519f6 Address:47ae1bf2-c261-600f-e912-c3ae463519f6} {Suffrage:Voter ID:server-08553fe5-3052-431e-f79a-16f8be12d1cd Address:08553fe5-3052-431e-f79a-16f8be12d1cd}]"
2025-04-21T21:24:06.501Z [INFO]  server-47ae1bf2-c261-600f-e912-c3ae463519f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eeb5164-93d0-ac37-d184-b5aae1ad9ec2 Address:2eeb5164-93d0-ac37-d184-b5aae1ad9ec2} {Suffrage:Voter ID:server-47ae1bf2-c261-600f-e912-c3ae463519f6 Address:47ae1bf2-c261-600f-e912-c3ae463519f6} {Suffrage:Voter ID:server-08553fe5-3052-431e-f79a-16f8be12d1cd Address:08553fe5-3052-431e-f79a-16f8be12d1cd}]"
2025-04-21T21:24:06.501Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2eeb5164-93d0-ac37-d184-b5aae1ad9ec2 Address:2eeb5164-93d0-ac37-d184-b5aae1ad9ec2} {Suffrage:Voter ID:server-47ae1bf2-c261-600f-e912-c3ae463519f6 Address:47ae1bf2-c261-600f-e912-c3ae463519f6} {Suffrage:Voter ID:server-08553fe5-3052-431e-f79a-16f8be12d1cd Address:08553fe5-3052-431e-f79a-16f8be12d1cd}]"
2025-04-21T21:24:06.501Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:06.501Z [INFO]  server-2eeb5164-93d0-ac37-d184-b5aae1ad9ec2: entering follower state: follower="Node at 2eeb5164-93d0-ac37-d184-b5aae1ad9ec2 [Follower]" leader-address= leader-id=
2025-04-21T21:24:06.501Z [INFO]  server-47ae1bf2-c261-600f-e912-c3ae463519f6: entering follower state: follower="Node at 47ae1bf2-c261-600f-e912-c3ae463519f6 [Follower]" leader-address= leader-id=
2025-04-21T21:24:06.501Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: entering follower state: follower="Node at 08553fe5-3052-431e-f79a-16f8be12d1cd [Follower]" leader-address= leader-id=
2025-04-21T21:24:07.201Z [WARN]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:07.201Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: entering candidate state: node="Node at 08553fe5-3052-431e-f79a-16f8be12d1cd [Candidate]" term=2
2025-04-21T21:24:07.201Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: election won: term=2 tally=2
2025-04-21T21:24:07.201Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: entering leader state: leader="Node at 08553fe5-3052-431e-f79a-16f8be12d1cd [Leader]"
2025-04-21T21:24:07.201Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: added peer, starting replication: peer=server-2eeb5164-93d0-ac37-d184-b5aae1ad9ec2
2025-04-21T21:24:07.201Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: added peer, starting replication: peer=server-47ae1bf2-c261-600f-e912-c3ae463519f6
2025-04-21T21:24:07.201Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: pipelining replication: peer="{Voter server-47ae1bf2-c261-600f-e912-c3ae463519f6 47ae1bf2-c261-600f-e912-c3ae463519f6}"
2025-04-21T21:24:07.201Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: pipelining replication: peer="{Voter server-2eeb5164-93d0-ac37-d184-b5aae1ad9ec2 2eeb5164-93d0-ac37-d184-b5aae1ad9ec2}"
2025-04-21T21:24:08.209Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:07.201957374 +0000 UTC m=+85.723616697 (1 nodes), 700.078306ms from start of poll, 700.281627ms from cluster start. Timeout at 2025-04-21 21:24:08.206961415 +0000 UTC m=+86.728620698, 1.005004001s after stability
2025-04-21T21:24:08.209Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: starting snapshot up to: index=12
2025-04-21T21:24:08.209Z [INFO]  creating new snapshot: path=/tmp/raft1912386223/snapshots/2-12-1745270648209.tmp
2025-04-21T21:24:08.210Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: snapshot complete up to: index=12
2025-04-21T21:24:08.210Z [INFO]  creating new snapshot: path=/tmp/raft1912386223/snapshots/2-10013-1745270648210.tmp
2025-04-21T21:24:08.210Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: copied to local snapshot: bytes=71
2025-04-21T21:24:08.210Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: snapshot restore progress: id=2-10013-1745270648210 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:08.210Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: restored user snapshot: index=10013
2025-04-21T21:24:08.210Z [ERROR] server-08553fe5-3052-431e-f79a-16f8be12d1cd: failed to get log: index=23 error="log not found"
2025-04-21T21:24:08.210Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: aborting pipeline replication: peer="{Voter server-47ae1bf2-c261-600f-e912-c3ae463519f6 47ae1bf2-c261-600f-e912-c3ae463519f6}"
2025-04-21T21:24:08.211Z [ERROR] server-08553fe5-3052-431e-f79a-16f8be12d1cd: failed to get log: index=23 error="log not found"
2025-04-21T21:24:08.211Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: aborting pipeline replication: peer="{Voter server-2eeb5164-93d0-ac37-d184-b5aae1ad9ec2 2eeb5164-93d0-ac37-d184-b5aae1ad9ec2}"
2025-04-21T21:24:08.211Z [ERROR] server-08553fe5-3052-431e-f79a-16f8be12d1cd: failed to get log: index=23 error="log not found"
2025-04-21T21:24:08.211Z [INFO]  creating new snapshot: path=/tmp/raft3451447166/snapshots/2-10013-1745270648211.tmp
2025-04-21T21:24:08.211Z [INFO]  server-2eeb5164-93d0-ac37-d184-b5aae1ad9ec2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:08.211Z [INFO]  server-2eeb5164-93d0-ac37-d184-b5aae1ad9ec2: copied to local snapshot: bytes=71
2025-04-21T21:24:08.211Z [INFO]  server-2eeb5164-93d0-ac37-d184-b5aae1ad9ec2: snapshot restore progress: id=2-10013-1745270648211 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:08.211Z [INFO]  server-2eeb5164-93d0-ac37-d184-b5aae1ad9ec2: Installed remote snapshot
2025-04-21T21:24:08.211Z [WARN]  server-2eeb5164-93d0-ac37-d184-b5aae1ad9ec2: skipping application of old log: index=22
2025-04-21T21:24:08.211Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: pipelining replication: peer="{Voter server-2eeb5164-93d0-ac37-d184-b5aae1ad9ec2 2eeb5164-93d0-ac37-d184-b5aae1ad9ec2}"
2025-04-21T21:24:08.211Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:08.221Z [ERROR] server-08553fe5-3052-431e-f79a-16f8be12d1cd: failed to get log: index=23 error="log not found"
2025-04-21T21:24:08.221Z [INFO]  creating new snapshot: path=/tmp/raft13165506/snapshots/2-10013-1745270648221.tmp
2025-04-21T21:24:08.221Z [INFO]  server-47ae1bf2-c261-600f-e912-c3ae463519f6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:08.221Z [INFO]  server-47ae1bf2-c261-600f-e912-c3ae463519f6: copied to local snapshot: bytes=71
2025-04-21T21:24:08.221Z [INFO]  server-47ae1bf2-c261-600f-e912-c3ae463519f6: snapshot restore progress: id=2-10013-1745270648221 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:08.221Z [INFO]  server-47ae1bf2-c261-600f-e912-c3ae463519f6: Installed remote snapshot
2025-04-21T21:24:08.221Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: pipelining replication: peer="{Voter server-47ae1bf2-c261-600f-e912-c3ae463519f6 47ae1bf2-c261-600f-e912-c3ae463519f6}"
2025-04-21T21:24:09.217Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:08.21171356 +0000 UTC m=+86.733372843 (2 nodes), 360ns from start of poll, 1.710037773s from cluster start. Timeout at 2025-04-21 21:24:09.21673828 +0000 UTC m=+87.738397603, 1.00502476s after stability
2025-04-21T21:24:09.233Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: aborting pipeline replication: peer="{Voter server-47ae1bf2-c261-600f-e912-c3ae463519f6 47ae1bf2-c261-600f-e912-c3ae463519f6}"
2025-04-21T21:24:09.233Z [INFO]  server-08553fe5-3052-431e-f79a-16f8be12d1cd: aborting pipeline replication: peer="{Voter server-2eeb5164-93d0-ac37-d184-b5aae1ad9ec2 2eeb5164-93d0-ac37-d184-b5aae1ad9ec2}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false
2025-04-21T21:24:09.233Z [INFO]  server-0f072092-dd08-67a7-964e-6e262fecfb82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f072092-dd08-67a7-964e-6e262fecfb82 Address:0f072092-dd08-67a7-964e-6e262fecfb82} {Suffrage:Voter ID:server-8aced771-bc4e-a7d7-45e6-58793d561629 Address:8aced771-bc4e-a7d7-45e6-58793d561629} {Suffrage:Voter ID:server-89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0 Address:89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0}]"
2025-04-21T21:24:09.233Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f072092-dd08-67a7-964e-6e262fecfb82 Address:0f072092-dd08-67a7-964e-6e262fecfb82} {Suffrage:Voter ID:server-8aced771-bc4e-a7d7-45e6-58793d561629 Address:8aced771-bc4e-a7d7-45e6-58793d561629} {Suffrage:Voter ID:server-89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0 Address:89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0}]"
2025-04-21T21:24:09.233Z [INFO]  server-89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f072092-dd08-67a7-964e-6e262fecfb82 Address:0f072092-dd08-67a7-964e-6e262fecfb82} {Suffrage:Voter ID:server-8aced771-bc4e-a7d7-45e6-58793d561629 Address:8aced771-bc4e-a7d7-45e6-58793d561629} {Suffrage:Voter ID:server-89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0 Address:89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0}]"
2025-04-21T21:24:09.233Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:09.233Z [INFO]  server-0f072092-dd08-67a7-964e-6e262fecfb82: entering follower state: follower="Node at 0f072092-dd08-67a7-964e-6e262fecfb82 [Follower]" leader-address= leader-id=
2025-04-21T21:24:09.233Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: entering follower state: follower="Node at 8aced771-bc4e-a7d7-45e6-58793d561629 [Follower]" leader-address= leader-id=
2025-04-21T21:24:09.233Z [INFO]  server-89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0: entering follower state: follower="Node at 89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0 [Follower]" leader-address= leader-id=
2025-04-21T21:24:09.749Z [WARN]  server-8aced771-bc4e-a7d7-45e6-58793d561629: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:09.749Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: entering candidate state: node="Node at 8aced771-bc4e-a7d7-45e6-58793d561629 [Candidate]" term=2
2025-04-21T21:24:09.749Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: election won: term=2 tally=2
2025-04-21T21:24:09.749Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: entering leader state: leader="Node at 8aced771-bc4e-a7d7-45e6-58793d561629 [Leader]"
2025-04-21T21:24:09.749Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: added peer, starting replication: peer=server-0f072092-dd08-67a7-964e-6e262fecfb82
2025-04-21T21:24:09.749Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: added peer, starting replication: peer=server-89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0
2025-04-21T21:24:09.749Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: pipelining replication: peer="{Voter server-89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0 89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0}"
2025-04-21T21:24:09.749Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: pipelining replication: peer="{Voter server-0f072092-dd08-67a7-964e-6e262fecfb82 0f072092-dd08-67a7-964e-6e262fecfb82}"
2025-04-21T21:24:10.755Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:09.749573736 +0000 UTC m=+88.271233059 (1 nodes), 515.734266ms from start of poll, 515.929427ms from cluster start. Timeout at 2025-04-21 21:24:10.754577456 +0000 UTC m=+89.276236779, 1.00500372s after stability
2025-04-21T21:24:10.755Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: starting snapshot up to: index=12
2025-04-21T21:24:10.755Z [INFO]  creating new snapshot: path=/tmp/raft2578284326/snapshots/2-12-1745270650755.tmp
2025-04-21T21:24:10.756Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: snapshot complete up to: index=12
2025-04-21T21:24:10.760Z [INFO]  creating new snapshot: path=/tmp/raft2578284326/snapshots/2-10013-1745270650760.tmp
2025-04-21T21:24:10.769Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: copied to local snapshot: bytes=71
2025-04-21T21:24:10.769Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: snapshot restore progress: id=2-10013-1745270650760 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:10.769Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: removing all old logs from log store
2025-04-21T21:24:10.769Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: compacting logs: from=1 to=22
2025-04-21T21:24:10.769Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: restored user snapshot: index=10013
2025-04-21T21:24:10.769Z [ERROR] server-8aced771-bc4e-a7d7-45e6-58793d561629: failed to get log: index=22 error="log not found"
2025-04-21T21:24:10.769Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: aborting pipeline replication: peer="{Voter server-0f072092-dd08-67a7-964e-6e262fecfb82 0f072092-dd08-67a7-964e-6e262fecfb82}"
2025-04-21T21:24:10.770Z [ERROR] server-8aced771-bc4e-a7d7-45e6-58793d561629: failed to get log: index=22 error="log not found"
2025-04-21T21:24:10.770Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: aborting pipeline replication: peer="{Voter server-89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0 89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0}"
2025-04-21T21:24:10.777Z [ERROR] server-8aced771-bc4e-a7d7-45e6-58793d561629: failed to get log: index=22 error="log not found"
2025-04-21T21:24:10.777Z [INFO]  creating new snapshot: path=/tmp/raft895143433/snapshots/2-10013-1745270650777.tmp
2025-04-21T21:24:10.777Z [INFO]  server-89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:10.777Z [INFO]  server-89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0: copied to local snapshot: bytes=71
2025-04-21T21:24:10.777Z [INFO]  server-89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0: snapshot restore progress: id=2-10013-1745270650777 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:10.777Z [INFO]  server-89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0: removing all old logs from log store
2025-04-21T21:24:10.777Z [INFO]  server-89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0: compacting logs: from=1 to=22
2025-04-21T21:24:10.777Z [INFO]  server-89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0: Installed remote snapshot
2025-04-21T21:24:10.777Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: pipelining replication: peer="{Voter server-89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0 89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0}"
2025-04-21T21:24:10.777Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:10.781Z [ERROR] server-8aced771-bc4e-a7d7-45e6-58793d561629: failed to get log: index=22 error="log not found"
2025-04-21T21:24:10.781Z [INFO]  creating new snapshot: path=/tmp/raft1893582428/snapshots/2-10013-1745270650781.tmp
2025-04-21T21:24:10.781Z [INFO]  server-0f072092-dd08-67a7-964e-6e262fecfb82: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:10.781Z [INFO]  server-0f072092-dd08-67a7-964e-6e262fecfb82: copied to local snapshot: bytes=71
2025-04-21T21:24:10.781Z [INFO]  server-0f072092-dd08-67a7-964e-6e262fecfb82: snapshot restore progress: id=2-10013-1745270650781 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:10.781Z [INFO]  server-0f072092-dd08-67a7-964e-6e262fecfb82: removing all old logs from log store
2025-04-21T21:24:10.781Z [INFO]  server-0f072092-dd08-67a7-964e-6e262fecfb82: compacting logs: from=1 to=22
2025-04-21T21:24:10.781Z [INFO]  server-0f072092-dd08-67a7-964e-6e262fecfb82: Installed remote snapshot
2025-04-21T21:24:10.781Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: pipelining replication: peer="{Voter server-0f072092-dd08-67a7-964e-6e262fecfb82 0f072092-dd08-67a7-964e-6e262fecfb82}"
2025-04-21T21:24:11.785Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:10.777801047 +0000 UTC m=+89.299460330 (2 nodes), 480ns from start of poll, 1.544156698s from cluster start. Timeout at 2025-04-21 21:24:11.782803767 +0000 UTC m=+90.304463090, 1.00500276s after stability
2025-04-21T21:24:11.798Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: aborting pipeline replication: peer="{Voter server-89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0 89146ab4-cb52-ae3b-1cc5-b6b50f4fb1f0}"
2025-04-21T21:24:11.798Z [INFO]  server-8aced771-bc4e-a7d7-45e6-58793d561629: aborting pipeline replication: peer="{Voter server-0f072092-dd08-67a7-964e-6e262fecfb82 0f072092-dd08-67a7-964e-6e262fecfb82}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true
2025-04-21T21:24:11.799Z [INFO]  server-84f6bc11-5bf7-42c0-6c46-3e2e7636ff80: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84f6bc11-5bf7-42c0-6c46-3e2e7636ff80 Address:84f6bc11-5bf7-42c0-6c46-3e2e7636ff80} {Suffrage:Voter ID:server-20b814ae-7c76-9ec7-2551-3eea2f162bc6 Address:20b814ae-7c76-9ec7-2551-3eea2f162bc6} {Suffrage:Voter ID:server-2d014142-d598-8f46-b424-f90c37ebfe27 Address:2d014142-d598-8f46-b424-f90c37ebfe27}]"
2025-04-21T21:24:11.799Z [INFO]  server-20b814ae-7c76-9ec7-2551-3eea2f162bc6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84f6bc11-5bf7-42c0-6c46-3e2e7636ff80 Address:84f6bc11-5bf7-42c0-6c46-3e2e7636ff80} {Suffrage:Voter ID:server-20b814ae-7c76-9ec7-2551-3eea2f162bc6 Address:20b814ae-7c76-9ec7-2551-3eea2f162bc6} {Suffrage:Voter ID:server-2d014142-d598-8f46-b424-f90c37ebfe27 Address:2d014142-d598-8f46-b424-f90c37ebfe27}]"
2025-04-21T21:24:11.799Z [INFO]  server-2d014142-d598-8f46-b424-f90c37ebfe27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84f6bc11-5bf7-42c0-6c46-3e2e7636ff80 Address:84f6bc11-5bf7-42c0-6c46-3e2e7636ff80} {Suffrage:Voter ID:server-20b814ae-7c76-9ec7-2551-3eea2f162bc6 Address:20b814ae-7c76-9ec7-2551-3eea2f162bc6} {Suffrage:Voter ID:server-2d014142-d598-8f46-b424-f90c37ebfe27 Address:2d014142-d598-8f46-b424-f90c37ebfe27}]"
2025-04-21T21:24:11.799Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:11.799Z [INFO]  server-84f6bc11-5bf7-42c0-6c46-3e2e7636ff80: entering follower state: follower="Node at 84f6bc11-5bf7-42c0-6c46-3e2e7636ff80 [Follower]" leader-address= leader-id=
2025-04-21T21:24:11.799Z [INFO]  server-20b814ae-7c76-9ec7-2551-3eea2f162bc6: entering follower state: follower="Node at 20b814ae-7c76-9ec7-2551-3eea2f162bc6 [Follower]" leader-address= leader-id=
2025-04-21T21:24:11.799Z [INFO]  server-2d014142-d598-8f46-b424-f90c37ebfe27: entering follower state: follower="Node at 2d014142-d598-8f46-b424-f90c37ebfe27 [Follower]" leader-address= leader-id=
2025-04-21T21:24:12.605Z [WARN]  server-84f6bc11-5bf7-42c0-6c46-3e2e7636ff80: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:12.605Z [INFO]  server-84f6bc11-5bf7-42c0-6c46-3e2e7636ff80: entering candidate state: node="Node at 84f6bc11-5bf7-42c0-6c46-3e2e7636ff80 [Candidate]" term=2
2025-04-21T21:24:12.605Z [INFO]  server-84f6bc11-5bf7-42c0-6c46-3e2e7636ff80: election won: term=2 tally=2
2025-04-21T21:24:12.605Z [INFO]  server-84f6bc11-5bf7-42c0-6c46-3e2e7636ff80: entering leader state: leader="Node at 84f6bc11-5bf7-42c0-6c46-3e2e7636ff80 [Leader]"
2025-04-21T21:24:12.605Z [INFO]  server-84f6bc11-5bf7-42c0-6c46-3e2e7636ff80: added peer, starting replication: peer=server-20b814ae-7c76-9ec7-2551-3eea2f162bc6
2025-04-21T21:24:12.605Z [INFO]  server-84f6bc11-5bf7-42c0-6c46-3e2e7636ff80: added peer, starting replication: peer=server-2d014142-d598-8f46-b424-f90c37ebfe27
2025-04-21T21:24:12.605Z [INFO]  server-84f6bc11-5bf7-42c0-6c46-3e2e7636ff80: pipelining replication: peer="{Voter server-2d014142-d598-8f46-b424-f90c37ebfe27 2d014142-d598-8f46-b424-f90c37ebfe27}"
2025-04-21T21:24:12.605Z [INFO]  server-84f6bc11-5bf7-42c0-6c46-3e2e7636ff80: pipelining replication: peer="{Voter server-20b814ae-7c76-9ec7-2551-3eea2f162bc6 20b814ae-7c76-9ec7-2551-3eea2f162bc6}"
2025-04-21T21:24:13.610Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:12.605749613 +0000 UTC m=+91.127408896 (1 nodes), 806.286619ms from start of poll, 806.482301ms from cluster start. Timeout at 2025-04-21 21:24:13.610762333 +0000 UTC m=+92.132421656, 1.00501276s after stability
2025-04-21T21:24:13.620Z [INFO]  server-84f6bc11-5bf7-42c0-6c46-3e2e7636ff80: starting snapshot up to: index=12
2025-04-21T21:24:13.620Z [INFO]  creating new snapshot: path=/tmp/raft3664008105/snapshots/2-12-1745270653620.tmp
2025-04-21T21:24:13.620Z [INFO]  server-84f6bc11-5bf7-42c0-6c46-3e2e7636ff80: snapshot complete up to: index=12
2025-04-21T21:24:13.621Z [INFO]  server-3a1878c7-cbc5-0752-ea08-04553c34306f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a1878c7-cbc5-0752-ea08-04553c34306f Address:3a1878c7-cbc5-0752-ea08-04553c34306f} {Suffrage:Voter ID:server-3fdc5909-aa9a-ce4a-5678-dd391a78e9aa Address:3fdc5909-aa9a-ce4a-5678-dd391a78e9aa} {Suffrage:Voter ID:server-ca374a01-a856-0ac0-e688-69f273fca0a1 Address:ca374a01-a856-0ac0-e688-69f273fca0a1}]"
2025-04-21T21:24:13.621Z [INFO]  server-3fdc5909-aa9a-ce4a-5678-dd391a78e9aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a1878c7-cbc5-0752-ea08-04553c34306f Address:3a1878c7-cbc5-0752-ea08-04553c34306f} {Suffrage:Voter ID:server-3fdc5909-aa9a-ce4a-5678-dd391a78e9aa Address:3fdc5909-aa9a-ce4a-5678-dd391a78e9aa} {Suffrage:Voter ID:server-ca374a01-a856-0ac0-e688-69f273fca0a1 Address:ca374a01-a856-0ac0-e688-69f273fca0a1}]"
2025-04-21T21:24:13.621Z [INFO]  server-ca374a01-a856-0ac0-e688-69f273fca0a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a1878c7-cbc5-0752-ea08-04553c34306f Address:3a1878c7-cbc5-0752-ea08-04553c34306f} {Suffrage:Voter ID:server-3fdc5909-aa9a-ce4a-5678-dd391a78e9aa Address:3fdc5909-aa9a-ce4a-5678-dd391a78e9aa} {Suffrage:Voter ID:server-ca374a01-a856-0ac0-e688-69f273fca0a1 Address:ca374a01-a856-0ac0-e688-69f273fca0a1}]"
2025-04-21T21:24:13.621Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:13.621Z [INFO]  server-3a1878c7-cbc5-0752-ea08-04553c34306f: entering follower state: follower="Node at 3a1878c7-cbc5-0752-ea08-04553c34306f [Follower]" leader-address= leader-id=
2025-04-21T21:24:13.621Z [INFO]  server-3fdc5909-aa9a-ce4a-5678-dd391a78e9aa: entering follower state: follower="Node at 3fdc5909-aa9a-ce4a-5678-dd391a78e9aa [Follower]" leader-address= leader-id=
2025-04-21T21:24:13.621Z [INFO]  server-ca374a01-a856-0ac0-e688-69f273fca0a1: entering follower state: follower="Node at ca374a01-a856-0ac0-e688-69f273fca0a1 [Follower]" leader-address= leader-id=
2025-04-21T21:24:14.237Z [WARN]  server-3a1878c7-cbc5-0752-ea08-04553c34306f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:14.237Z [INFO]  server-3a1878c7-cbc5-0752-ea08-04553c34306f: entering candidate state: node="Node at 3a1878c7-cbc5-0752-ea08-04553c34306f [Candidate]" term=2
2025-04-21T21:24:14.237Z [INFO]  server-3a1878c7-cbc5-0752-ea08-04553c34306f: election won: term=2 tally=2
2025-04-21T21:24:14.237Z [INFO]  server-3a1878c7-cbc5-0752-ea08-04553c34306f: entering leader state: leader="Node at 3a1878c7-cbc5-0752-ea08-04553c34306f [Leader]"
2025-04-21T21:24:14.237Z [INFO]  server-3a1878c7-cbc5-0752-ea08-04553c34306f: added peer, starting replication: peer=server-3fdc5909-aa9a-ce4a-5678-dd391a78e9aa
2025-04-21T21:24:14.237Z [INFO]  server-3a1878c7-cbc5-0752-ea08-04553c34306f: added peer, starting replication: peer=server-ca374a01-a856-0ac0-e688-69f273fca0a1
2025-04-21T21:24:14.237Z [INFO]  server-3a1878c7-cbc5-0752-ea08-04553c34306f: pipelining replication: peer="{Voter server-ca374a01-a856-0ac0-e688-69f273fca0a1 ca374a01-a856-0ac0-e688-69f273fca0a1}"
2025-04-21T21:24:14.237Z [INFO]  server-3a1878c7-cbc5-0752-ea08-04553c34306f: pipelining replication: peer="{Voter server-3fdc5909-aa9a-ce4a-5678-dd391a78e9aa 3fdc5909-aa9a-ce4a-5678-dd391a78e9aa}"
2025-04-21T21:24:15.245Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:14.237865622 +0000 UTC m=+92.759524945 (1 nodes), 616.390063ms from start of poll, 616.567464ms from cluster start. Timeout at 2025-04-21 21:24:15.242880743 +0000 UTC m=+93.764540066, 1.005015121s after stability
2025-04-21T21:24:15.245Z [INFO]  creating new snapshot: path=/tmp/raft3602862380/snapshots/2-10013-1745270655245.tmp
2025-04-21T21:24:15.245Z [INFO]  server-3a1878c7-cbc5-0752-ea08-04553c34306f: copied to local snapshot: bytes=71
2025-04-21T21:24:15.245Z [INFO]  server-3a1878c7-cbc5-0752-ea08-04553c34306f: snapshot restore progress: id=2-10013-1745270655245 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:15.245Z [INFO]  server-3a1878c7-cbc5-0752-ea08-04553c34306f: restored user snapshot: index=10013
2025-04-21T21:24:15.245Z [ERROR] server-3a1878c7-cbc5-0752-ea08-04553c34306f: failed to get log: index=3 error="log not found"
2025-04-21T21:24:15.245Z [INFO]  server-3a1878c7-cbc5-0752-ea08-04553c34306f: aborting pipeline replication: peer="{Voter server-ca374a01-a856-0ac0-e688-69f273fca0a1 ca374a01-a856-0ac0-e688-69f273fca0a1}"
2025-04-21T21:24:15.245Z [ERROR] server-3a1878c7-cbc5-0752-ea08-04553c34306f: failed to get log: index=3 error="log not found"
2025-04-21T21:24:15.245Z [INFO]  server-3a1878c7-cbc5-0752-ea08-04553c34306f: aborting pipeline replication: peer="{Voter server-3fdc5909-aa9a-ce4a-5678-dd391a78e9aa 3fdc5909-aa9a-ce4a-5678-dd391a78e9aa}"
2025-04-21T21:24:15.253Z [ERROR] server-3a1878c7-cbc5-0752-ea08-04553c34306f: failed to get log: index=3 error="log not found"
2025-04-21T21:24:15.253Z [INFO]  creating new snapshot: path=/tmp/raft1434127742/snapshots/2-10013-1745270655253.tmp
2025-04-21T21:24:15.254Z [INFO]  server-ca374a01-a856-0ac0-e688-69f273fca0a1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:15.254Z [INFO]  server-ca374a01-a856-0ac0-e688-69f273fca0a1: copied to local snapshot: bytes=71
2025-04-21T21:24:15.254Z [INFO]  server-ca374a01-a856-0ac0-e688-69f273fca0a1: snapshot restore progress: id=2-10013-1745270655253 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:15.254Z [INFO]  server-ca374a01-a856-0ac0-e688-69f273fca0a1: Installed remote snapshot
2025-04-21T21:24:15.254Z [INFO]  server-3a1878c7-cbc5-0752-ea08-04553c34306f: pipelining replication: peer="{Voter server-ca374a01-a856-0ac0-e688-69f273fca0a1 ca374a01-a856-0ac0-e688-69f273fca0a1}"
2025-04-21T21:24:15.254Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:15.261Z [ERROR] server-3a1878c7-cbc5-0752-ea08-04553c34306f: failed to get log: index=3 error="log not found"
2025-04-21T21:24:15.261Z [INFO]  creating new snapshot: path=/tmp/raft669826322/snapshots/2-10013-1745270655261.tmp
2025-04-21T21:24:15.261Z [INFO]  server-3fdc5909-aa9a-ce4a-5678-dd391a78e9aa: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:15.261Z [INFO]  server-3fdc5909-aa9a-ce4a-5678-dd391a78e9aa: copied to local snapshot: bytes=71
2025-04-21T21:24:15.261Z [INFO]  server-3fdc5909-aa9a-ce4a-5678-dd391a78e9aa: snapshot restore progress: id=2-10013-1745270655261 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:15.261Z [INFO]  server-3fdc5909-aa9a-ce4a-5678-dd391a78e9aa: Installed remote snapshot
2025-04-21T21:24:15.261Z [INFO]  server-3a1878c7-cbc5-0752-ea08-04553c34306f: pipelining replication: peer="{Voter server-3fdc5909-aa9a-ce4a-5678-dd391a78e9aa 3fdc5909-aa9a-ce4a-5678-dd391a78e9aa}"
2025-04-21T21:24:16.261Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:15.254466615 +0000 UTC m=+93.776125938 (2 nodes), 880ns from start of poll, 1.633168457s from cluster start. Timeout at 2025-04-21 21:24:16.259532576 +0000 UTC m=+94.781191899, 1.005065961s after stability
2025-04-21T21:24:16.277Z [INFO]  server-84f6bc11-5bf7-42c0-6c46-3e2e7636ff80: aborting pipeline replication: peer="{Voter server-2d014142-d598-8f46-b424-f90c37ebfe27 2d014142-d598-8f46-b424-f90c37ebfe27}"
2025-04-21T21:24:16.277Z [INFO]  server-84f6bc11-5bf7-42c0-6c46-3e2e7636ff80: aborting pipeline replication: peer="{Voter server-20b814ae-7c76-9ec7-2551-3eea2f162bc6 20b814ae-7c76-9ec7-2551-3eea2f162bc6}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true
2025-04-21T21:24:16.278Z [INFO]  server-047161b9-2497-60fe-4578-64d23b577be9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-047161b9-2497-60fe-4578-64d23b577be9 Address:047161b9-2497-60fe-4578-64d23b577be9} {Suffrage:Voter ID:server-57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e Address:57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e} {Suffrage:Voter ID:server-e8b579bd-cced-6fa6-9ac1-c03579f4e5cb Address:e8b579bd-cced-6fa6-9ac1-c03579f4e5cb}]"
2025-04-21T21:24:16.278Z [INFO]  server-57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-047161b9-2497-60fe-4578-64d23b577be9 Address:047161b9-2497-60fe-4578-64d23b577be9} {Suffrage:Voter ID:server-57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e Address:57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e} {Suffrage:Voter ID:server-e8b579bd-cced-6fa6-9ac1-c03579f4e5cb Address:e8b579bd-cced-6fa6-9ac1-c03579f4e5cb}]"
2025-04-21T21:24:16.278Z [INFO]  server-e8b579bd-cced-6fa6-9ac1-c03579f4e5cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-047161b9-2497-60fe-4578-64d23b577be9 Address:047161b9-2497-60fe-4578-64d23b577be9} {Suffrage:Voter ID:server-57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e Address:57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e} {Suffrage:Voter ID:server-e8b579bd-cced-6fa6-9ac1-c03579f4e5cb Address:e8b579bd-cced-6fa6-9ac1-c03579f4e5cb}]"
2025-04-21T21:24:16.278Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:16.278Z [INFO]  server-047161b9-2497-60fe-4578-64d23b577be9: entering follower state: follower="Node at 047161b9-2497-60fe-4578-64d23b577be9 [Follower]" leader-address= leader-id=
2025-04-21T21:24:16.278Z [INFO]  server-57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e: entering follower state: follower="Node at 57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e [Follower]" leader-address= leader-id=
2025-04-21T21:24:16.278Z [INFO]  server-e8b579bd-cced-6fa6-9ac1-c03579f4e5cb: entering follower state: follower="Node at e8b579bd-cced-6fa6-9ac1-c03579f4e5cb [Follower]" leader-address= leader-id=
2025-04-21T21:24:16.961Z [WARN]  server-57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:16.961Z [INFO]  server-57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e: entering candidate state: node="Node at 57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e [Candidate]" term=2
2025-04-21T21:24:16.961Z [INFO]  server-57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e: election won: term=2 tally=2
2025-04-21T21:24:16.961Z [INFO]  server-57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e: entering leader state: leader="Node at 57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e [Leader]"
2025-04-21T21:24:16.961Z [INFO]  server-57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e: added peer, starting replication: peer=server-047161b9-2497-60fe-4578-64d23b577be9
2025-04-21T21:24:16.961Z [INFO]  server-57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e: added peer, starting replication: peer=server-e8b579bd-cced-6fa6-9ac1-c03579f4e5cb
2025-04-21T21:24:16.961Z [INFO]  server-57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e: pipelining replication: peer="{Voter server-e8b579bd-cced-6fa6-9ac1-c03579f4e5cb e8b579bd-cced-6fa6-9ac1-c03579f4e5cb}"
2025-04-21T21:24:16.961Z [INFO]  server-57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e: pipelining replication: peer="{Voter server-047161b9-2497-60fe-4578-64d23b577be9 047161b9-2497-60fe-4578-64d23b577be9}"
2025-04-21T21:24:17.969Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:16.96171201 +0000 UTC m=+95.483371333 (1 nodes), 683.556555ms from start of poll, 683.755396ms from cluster start. Timeout at 2025-04-21 21:24:17.96673789 +0000 UTC m=+96.488397173, 1.00502584s after stability
2025-04-21T21:24:17.969Z [INFO]  server-57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e: starting snapshot up to: index=12
2025-04-21T21:24:17.969Z [INFO]  creating new snapshot: path=/tmp/raft2301215470/snapshots/2-12-1745270657969.tmp
2025-04-21T21:24:17.970Z [INFO]  server-57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e: snapshot complete up to: index=12
2025-04-21T21:24:17.970Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c Address:c51419d5-0bc4-4f93-9a75-2c6a03d2159c} {Suffrage:Voter ID:server-2efd08fe-e4cb-2146-65b4-780c5a0458c7 Address:2efd08fe-e4cb-2146-65b4-780c5a0458c7} {Suffrage:Voter ID:server-edd262ee-3226-fae1-9068-21259ff3d417 Address:edd262ee-3226-fae1-9068-21259ff3d417}]"
2025-04-21T21:24:17.970Z [INFO]  server-2efd08fe-e4cb-2146-65b4-780c5a0458c7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c Address:c51419d5-0bc4-4f93-9a75-2c6a03d2159c} {Suffrage:Voter ID:server-2efd08fe-e4cb-2146-65b4-780c5a0458c7 Address:2efd08fe-e4cb-2146-65b4-780c5a0458c7} {Suffrage:Voter ID:server-edd262ee-3226-fae1-9068-21259ff3d417 Address:edd262ee-3226-fae1-9068-21259ff3d417}]"
2025-04-21T21:24:17.970Z [INFO]  server-edd262ee-3226-fae1-9068-21259ff3d417: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c Address:c51419d5-0bc4-4f93-9a75-2c6a03d2159c} {Suffrage:Voter ID:server-2efd08fe-e4cb-2146-65b4-780c5a0458c7 Address:2efd08fe-e4cb-2146-65b4-780c5a0458c7} {Suffrage:Voter ID:server-edd262ee-3226-fae1-9068-21259ff3d417 Address:edd262ee-3226-fae1-9068-21259ff3d417}]"
2025-04-21T21:24:17.970Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:17.971Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: entering follower state: follower="Node at c51419d5-0bc4-4f93-9a75-2c6a03d2159c [Follower]" leader-address= leader-id=
2025-04-21T21:24:17.971Z [INFO]  server-2efd08fe-e4cb-2146-65b4-780c5a0458c7: entering follower state: follower="Node at 2efd08fe-e4cb-2146-65b4-780c5a0458c7 [Follower]" leader-address= leader-id=
2025-04-21T21:24:17.971Z [INFO]  server-edd262ee-3226-fae1-9068-21259ff3d417: entering follower state: follower="Node at edd262ee-3226-fae1-9068-21259ff3d417 [Follower]" leader-address= leader-id=
2025-04-21T21:24:18.705Z [WARN]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:18.705Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: entering candidate state: node="Node at c51419d5-0bc4-4f93-9a75-2c6a03d2159c [Candidate]" term=2
2025-04-21T21:24:18.705Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: election won: term=2 tally=2
2025-04-21T21:24:18.705Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: entering leader state: leader="Node at c51419d5-0bc4-4f93-9a75-2c6a03d2159c [Leader]"
2025-04-21T21:24:18.705Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: added peer, starting replication: peer=server-2efd08fe-e4cb-2146-65b4-780c5a0458c7
2025-04-21T21:24:18.705Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: added peer, starting replication: peer=server-edd262ee-3226-fae1-9068-21259ff3d417
2025-04-21T21:24:18.706Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: pipelining replication: peer="{Voter server-2efd08fe-e4cb-2146-65b4-780c5a0458c7 2efd08fe-e4cb-2146-65b4-780c5a0458c7}"
2025-04-21T21:24:18.706Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: pipelining replication: peer="{Voter server-edd262ee-3226-fae1-9068-21259ff3d417 edd262ee-3226-fae1-9068-21259ff3d417}"
2025-04-21T21:24:19.713Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:18.706164294 +0000 UTC m=+97.227823577 (1 nodes), 735.209581ms from start of poll, 735.362662ms from cluster start. Timeout at 2025-04-21 21:24:19.711181094 +0000 UTC m=+98.232840417, 1.00501684s after stability
2025-04-21T21:24:19.713Z [INFO]  creating new snapshot: path=/tmp/raft2598470646/snapshots/2-10013-1745270659713.tmp
2025-04-21T21:24:19.713Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: copied to local snapshot: bytes=71
2025-04-21T21:24:19.713Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: snapshot restore progress: id=2-10013-1745270659713 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:19.713Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: removing all old logs from log store
2025-04-21T21:24:19.713Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: compacting logs: from=1 to=2
2025-04-21T21:24:19.713Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: restored user snapshot: index=10013
2025-04-21T21:24:19.713Z [ERROR] server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: failed to get log: index=2 error="log not found"
2025-04-21T21:24:19.713Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: aborting pipeline replication: peer="{Voter server-edd262ee-3226-fae1-9068-21259ff3d417 edd262ee-3226-fae1-9068-21259ff3d417}"
2025-04-21T21:24:19.713Z [ERROR] server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: failed to get log: index=2 error="log not found"
2025-04-21T21:24:19.713Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: aborting pipeline replication: peer="{Voter server-2efd08fe-e4cb-2146-65b4-780c5a0458c7 2efd08fe-e4cb-2146-65b4-780c5a0458c7}"
2025-04-21T21:24:19.721Z [ERROR] server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: failed to get log: index=2 error="log not found"
2025-04-21T21:24:19.721Z [INFO]  creating new snapshot: path=/tmp/raft1283160188/snapshots/2-10013-1745270659721.tmp
2025-04-21T21:24:19.721Z [INFO]  server-2efd08fe-e4cb-2146-65b4-780c5a0458c7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:19.722Z [INFO]  server-2efd08fe-e4cb-2146-65b4-780c5a0458c7: copied to local snapshot: bytes=71
2025-04-21T21:24:19.722Z [ERROR] server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: failed to get log: index=2 error="log not found"
2025-04-21T21:24:19.722Z [INFO]  creating new snapshot: path=/tmp/raft2979007060/snapshots/2-10013-1745270659722.tmp
2025-04-21T21:24:19.722Z [INFO]  server-edd262ee-3226-fae1-9068-21259ff3d417: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:19.722Z [INFO]  server-edd262ee-3226-fae1-9068-21259ff3d417: copied to local snapshot: bytes=71
2025-04-21T21:24:19.722Z [INFO]  server-edd262ee-3226-fae1-9068-21259ff3d417: snapshot restore progress: id=2-10013-1745270659722 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:19.722Z [INFO]  server-edd262ee-3226-fae1-9068-21259ff3d417: removing all old logs from log store
2025-04-21T21:24:19.722Z [INFO]  server-edd262ee-3226-fae1-9068-21259ff3d417: compacting logs: from=1 to=2
2025-04-21T21:24:19.722Z [INFO]  server-edd262ee-3226-fae1-9068-21259ff3d417: Installed remote snapshot
2025-04-21T21:24:19.722Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: pipelining replication: peer="{Voter server-edd262ee-3226-fae1-9068-21259ff3d417 edd262ee-3226-fae1-9068-21259ff3d417}"
2025-04-21T21:24:19.722Z [INFO]  server-2efd08fe-e4cb-2146-65b4-780c5a0458c7: snapshot restore progress: id=2-10013-1745270659721 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-04-21T21:24:19.722Z [INFO]  server-2efd08fe-e4cb-2146-65b4-780c5a0458c7: removing all old logs from log store
2025-04-21T21:24:19.722Z [INFO]  server-2efd08fe-e4cb-2146-65b4-780c5a0458c7: compacting logs: from=1 to=2
2025-04-21T21:24:19.722Z [INFO]  server-2efd08fe-e4cb-2146-65b4-780c5a0458c7: Installed remote snapshot
2025-04-21T21:24:19.723Z [INFO]  server-c51419d5-0bc4-4f93-9a75-2c6a03d2159c: pipelining replication: peer="{Voter server-2efd08fe-e4cb-2146-65b4-780c5a0458c7 2efd08fe-e4cb-2146-65b4-780c5a0458c7}"
2025-04-21T21:24:19.723Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:20.733Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:19.723060329 +0000 UTC m=+98.244719612 (2 nodes), 480ns from start of poll, 1.752258697s from cluster start. Timeout at 2025-04-21 21:24:20.728100409 +0000 UTC m=+99.249759692, 1.00504008s after stability
2025-04-21T21:24:20.749Z [INFO]  server-57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e: aborting pipeline replication: peer="{Voter server-047161b9-2497-60fe-4578-64d23b577be9 047161b9-2497-60fe-4578-64d23b577be9}"
2025-04-21T21:24:20.749Z [INFO]  server-57cb0fcd-cf86-a77a-21f8-46f4ce5b2a5e: aborting pipeline replication: peer="{Voter server-e8b579bd-cced-6fa6-9ac1-c03579f4e5cb e8b579bd-cced-6fa6-9ac1-c03579f4e5cb}"
--- PASS: TestRaft_UserRestore (84.70s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.58s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.72s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.54s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.29s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.72s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.64s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.52s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.19s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.87s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.58s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.36s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.33s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.91s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.56s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.22s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.38s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.66s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.62s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.59s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.16s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.73s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.57s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.48s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.47s)
=== RUN   TestRaft_SendSnapshotFollower
2025-04-21T21:24:20.749Z [INFO]  server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296 Address:a7b7b7dc-13df-4878-a8b8-3d1b30cfb296} {Suffrage:Voter ID:server-589299a2-46ec-861e-70da-43c36c071608 Address:589299a2-46ec-861e-70da-43c36c071608} {Suffrage:Voter ID:server-67599731-ae24-f398-9459-dbbf07625c4a Address:67599731-ae24-f398-9459-dbbf07625c4a}]"
2025-04-21T21:24:20.750Z [INFO]  server-589299a2-46ec-861e-70da-43c36c071608: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296 Address:a7b7b7dc-13df-4878-a8b8-3d1b30cfb296} {Suffrage:Voter ID:server-589299a2-46ec-861e-70da-43c36c071608 Address:589299a2-46ec-861e-70da-43c36c071608} {Suffrage:Voter ID:server-67599731-ae24-f398-9459-dbbf07625c4a Address:67599731-ae24-f398-9459-dbbf07625c4a}]"
2025-04-21T21:24:20.750Z [INFO]  server-67599731-ae24-f398-9459-dbbf07625c4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296 Address:a7b7b7dc-13df-4878-a8b8-3d1b30cfb296} {Suffrage:Voter ID:server-589299a2-46ec-861e-70da-43c36c071608 Address:589299a2-46ec-861e-70da-43c36c071608} {Suffrage:Voter ID:server-67599731-ae24-f398-9459-dbbf07625c4a Address:67599731-ae24-f398-9459-dbbf07625c4a}]"
2025-04-21T21:24:20.750Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:20.750Z [INFO]  server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296: entering follower state: follower="Node at a7b7b7dc-13df-4878-a8b8-3d1b30cfb296 [Follower]" leader-address= leader-id=
2025-04-21T21:24:20.750Z [INFO]  server-589299a2-46ec-861e-70da-43c36c071608: entering follower state: follower="Node at 589299a2-46ec-861e-70da-43c36c071608 [Follower]" leader-address= leader-id=
2025-04-21T21:24:20.750Z [INFO]  server-67599731-ae24-f398-9459-dbbf07625c4a: entering follower state: follower="Node at 67599731-ae24-f398-9459-dbbf07625c4a [Follower]" leader-address= leader-id=
2025-04-21T21:24:20.805Z [WARN]  server-589299a2-46ec-861e-70da-43c36c071608: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:20.805Z [INFO]  server-589299a2-46ec-861e-70da-43c36c071608: entering candidate state: node="Node at 589299a2-46ec-861e-70da-43c36c071608 [Candidate]" term=2
2025-04-21T21:24:20.805Z [INFO]  server-589299a2-46ec-861e-70da-43c36c071608: election won: term=2 tally=2
2025-04-21T21:24:20.805Z [INFO]  server-589299a2-46ec-861e-70da-43c36c071608: entering leader state: leader="Node at 589299a2-46ec-861e-70da-43c36c071608 [Leader]"
2025-04-21T21:24:20.805Z [INFO]  server-589299a2-46ec-861e-70da-43c36c071608: added peer, starting replication: peer=server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296
2025-04-21T21:24:20.805Z [INFO]  server-589299a2-46ec-861e-70da-43c36c071608: added peer, starting replication: peer=server-67599731-ae24-f398-9459-dbbf07625c4a
2025-04-21T21:24:20.805Z [INFO]  server-589299a2-46ec-861e-70da-43c36c071608: pipelining replication: peer="{Voter server-67599731-ae24-f398-9459-dbbf07625c4a 67599731-ae24-f398-9459-dbbf07625c4a}"
2025-04-21T21:24:20.805Z [INFO]  server-589299a2-46ec-861e-70da-43c36c071608: pipelining replication: peer="{Voter server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296 a7b7b7dc-13df-4878-a8b8-3d1b30cfb296}"
2025-04-21T21:24:20.913Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:20.805871887 +0000 UTC m=+99.327531210 (2 nodes), 55.798735ms from start of poll, 55.999937ms from cluster start. Timeout at 2025-04-21 21:24:20.910875527 +0000 UTC m=+99.432534850, 105.00364ms after stability
2025-04-21T21:24:20.913Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:21.021Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:20.91352005 +0000 UTC m=+99.435179373 (1 nodes), 1.72µs from start of poll, 163.6481ms from cluster start. Timeout at 2025-04-21 21:24:21.01855061 +0000 UTC m=+99.540209933, 105.03056ms after stability
2025-04-21T21:24:21.021Z [ERROR] server-589299a2-46ec-861e-70da-43c36c071608: failed to pipeline appendEntries: peer="{Voter server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296 a7b7b7dc-13df-4878-a8b8-3d1b30cfb296}" error="append pipeline closed"
2025-04-21T21:24:21.021Z [INFO]  server-589299a2-46ec-861e-70da-43c36c071608: aborting pipeline replication: peer="{Voter server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296 a7b7b7dc-13df-4878-a8b8-3d1b30cfb296}"
2025-04-21T21:24:21.021Z [ERROR] server-589299a2-46ec-861e-70da-43c36c071608: failed to appendEntries to: peer="{Voter server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296 a7b7b7dc-13df-4878-a8b8-3d1b30cfb296}" error="failed to connect to peer: a7b7b7dc-13df-4878-a8b8-3d1b30cfb296"
    raft_test.go:1627: [INFO] Finished apply without behind follower
2025-04-21T21:24:21.023Z [ERROR] server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296: failed to take snapshot: error="nothing new to snapshot"
2025-04-21T21:24:21.023Z [INFO]  server-589299a2-46ec-861e-70da-43c36c071608: starting snapshot up to: index=102
2025-04-21T21:24:21.023Z [INFO]  creating new snapshot: path=/tmp/raft3574264731/snapshots/2-102-1745270661023.tmp
2025-04-21T21:24:21.024Z [INFO]  server-589299a2-46ec-861e-70da-43c36c071608: compacting logs: from=1 to=92
2025-04-21T21:24:21.024Z [INFO]  server-589299a2-46ec-861e-70da-43c36c071608: snapshot complete up to: index=102
2025-04-21T21:24:21.024Z [INFO]  server-67599731-ae24-f398-9459-dbbf07625c4a: starting snapshot up to: index=100
2025-04-21T21:24:21.024Z [INFO]  creating new snapshot: path=/tmp/raft1420341274/snapshots/2-100-1745270661024.tmp
2025-04-21T21:24:21.024Z [INFO]  server-67599731-ae24-f398-9459-dbbf07625c4a: compacting logs: from=1 to=92
2025-04-21T21:24:21.024Z [INFO]  server-67599731-ae24-f398-9459-dbbf07625c4a: snapshot complete up to: index=100
2025-04-21T21:24:21.033Z [ERROR] server-589299a2-46ec-861e-70da-43c36c071608: failed to get log: index=2 error="log not found"
2025-04-21T21:24:21.033Z [INFO]  creating new snapshot: path=/tmp/raft97402085/snapshots/2-102-1745270661033.tmp
2025-04-21T21:24:21.033Z [INFO]  server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-04-21T21:24:21.034Z [INFO]  server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296: copied to local snapshot: bytes=693
2025-04-21T21:24:21.034Z [INFO]  server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296: snapshot restore progress: id=2-102-1745270661033 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-04-21T21:24:21.034Z [INFO]  server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296: Installed remote snapshot
2025-04-21T21:24:21.034Z [WARN]  server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296: skipping application of old log: index=102
2025-04-21T21:24:21.034Z [INFO]  server-589299a2-46ec-861e-70da-43c36c071608: pipelining replication: peer="{Voter server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296 a7b7b7dc-13df-4878-a8b8-3d1b30cfb296}"
2025-04-21T21:24:21.041Z [INFO]  server-589299a2-46ec-861e-70da-43c36c071608: aborting pipeline replication: peer="{Voter server-67599731-ae24-f398-9459-dbbf07625c4a 67599731-ae24-f398-9459-dbbf07625c4a}"
2025-04-21T21:24:21.041Z [INFO]  server-589299a2-46ec-861e-70da-43c36c071608: aborting pipeline replication: peer="{Voter server-a7b7b7dc-13df-4878-a8b8-3d1b30cfb296 a7b7b7dc-13df-4878-a8b8-3d1b30cfb296}"
--- PASS: TestRaft_SendSnapshotFollower (0.29s)
=== RUN   TestRaft_SendSnapshotAndLogsFollower
2025-04-21T21:24:21.042Z [INFO]  server-a57dd488-0452-af56-2c9e-11d8f01801ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a57dd488-0452-af56-2c9e-11d8f01801ca Address:a57dd488-0452-af56-2c9e-11d8f01801ca} {Suffrage:Voter ID:server-d574d151-3397-db2f-400f-34b46d8b64ee Address:d574d151-3397-db2f-400f-34b46d8b64ee} {Suffrage:Voter ID:server-6f2ffa20-148c-62cc-1188-04495624e1e8 Address:6f2ffa20-148c-62cc-1188-04495624e1e8}]"
2025-04-21T21:24:21.042Z [INFO]  server-d574d151-3397-db2f-400f-34b46d8b64ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a57dd488-0452-af56-2c9e-11d8f01801ca Address:a57dd488-0452-af56-2c9e-11d8f01801ca} {Suffrage:Voter ID:server-d574d151-3397-db2f-400f-34b46d8b64ee Address:d574d151-3397-db2f-400f-34b46d8b64ee} {Suffrage:Voter ID:server-6f2ffa20-148c-62cc-1188-04495624e1e8 Address:6f2ffa20-148c-62cc-1188-04495624e1e8}]"
2025-04-21T21:24:21.042Z [INFO]  server-6f2ffa20-148c-62cc-1188-04495624e1e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a57dd488-0452-af56-2c9e-11d8f01801ca Address:a57dd488-0452-af56-2c9e-11d8f01801ca} {Suffrage:Voter ID:server-d574d151-3397-db2f-400f-34b46d8b64ee Address:d574d151-3397-db2f-400f-34b46d8b64ee} {Suffrage:Voter ID:server-6f2ffa20-148c-62cc-1188-04495624e1e8 Address:6f2ffa20-148c-62cc-1188-04495624e1e8}]"
2025-04-21T21:24:21.042Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:21.042Z [INFO]  server-a57dd488-0452-af56-2c9e-11d8f01801ca: entering follower state: follower="Node at a57dd488-0452-af56-2c9e-11d8f01801ca [Follower]" leader-address= leader-id=
2025-04-21T21:24:21.042Z [INFO]  server-d574d151-3397-db2f-400f-34b46d8b64ee: entering follower state: follower="Node at d574d151-3397-db2f-400f-34b46d8b64ee [Follower]" leader-address= leader-id=
2025-04-21T21:24:21.042Z [INFO]  server-6f2ffa20-148c-62cc-1188-04495624e1e8: entering follower state: follower="Node at 6f2ffa20-148c-62cc-1188-04495624e1e8 [Follower]" leader-address= leader-id=
2025-04-21T21:24:21.102Z [WARN]  server-d574d151-3397-db2f-400f-34b46d8b64ee: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:21.102Z [INFO]  server-d574d151-3397-db2f-400f-34b46d8b64ee: entering candidate state: node="Node at d574d151-3397-db2f-400f-34b46d8b64ee [Candidate]" term=2
2025-04-21T21:24:21.102Z [INFO]  server-d574d151-3397-db2f-400f-34b46d8b64ee: election won: term=2 tally=2
2025-04-21T21:24:21.102Z [INFO]  server-d574d151-3397-db2f-400f-34b46d8b64ee: entering leader state: leader="Node at d574d151-3397-db2f-400f-34b46d8b64ee [Leader]"
2025-04-21T21:24:21.102Z [INFO]  server-d574d151-3397-db2f-400f-34b46d8b64ee: added peer, starting replication: peer=server-a57dd488-0452-af56-2c9e-11d8f01801ca
2025-04-21T21:24:21.102Z [INFO]  server-d574d151-3397-db2f-400f-34b46d8b64ee: added peer, starting replication: peer=server-6f2ffa20-148c-62cc-1188-04495624e1e8
2025-04-21T21:24:21.102Z [INFO]  server-d574d151-3397-db2f-400f-34b46d8b64ee: pipelining replication: peer="{Voter server-6f2ffa20-148c-62cc-1188-04495624e1e8 6f2ffa20-148c-62cc-1188-04495624e1e8}"
2025-04-21T21:24:21.102Z [INFO]  server-d574d151-3397-db2f-400f-34b46d8b64ee: pipelining replication: peer="{Voter server-a57dd488-0452-af56-2c9e-11d8f01801ca a57dd488-0452-af56-2c9e-11d8f01801ca}"
2025-04-21T21:24:21.209Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:21.102741322 +0000 UTC m=+99.624400645 (2 nodes), 60.293365ms from start of poll, 60.464086ms from cluster start. Timeout at 2025-04-21 21:24:21.207764243 +0000 UTC m=+99.729423566, 105.022921ms after stability
2025-04-21T21:24:21.209Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:21.317Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:21.209414199 +0000 UTC m=+99.731073522 (1 nodes), 560ns from start of poll, 167.136963ms from cluster start. Timeout at 2025-04-21 21:24:21.314468279 +0000 UTC m=+99.836127602, 105.05408ms after stability
2025-04-21T21:24:21.317Z [ERROR] server-d574d151-3397-db2f-400f-34b46d8b64ee: failed to pipeline appendEntries: peer="{Voter server-a57dd488-0452-af56-2c9e-11d8f01801ca a57dd488-0452-af56-2c9e-11d8f01801ca}" error="append pipeline closed"
2025-04-21T21:24:21.317Z [INFO]  server-d574d151-3397-db2f-400f-34b46d8b64ee: aborting pipeline replication: peer="{Voter server-a57dd488-0452-af56-2c9e-11d8f01801ca a57dd488-0452-af56-2c9e-11d8f01801ca}"
2025-04-21T21:24:21.317Z [ERROR] server-d574d151-3397-db2f-400f-34b46d8b64ee: failed to heartbeat to: peer=a57dd488-0452-af56-2c9e-11d8f01801ca backoff time=10ms error="failed to connect to peer: a57dd488-0452-af56-2c9e-11d8f01801ca"
2025-04-21T21:24:21.317Z [ERROR] server-d574d151-3397-db2f-400f-34b46d8b64ee: failed to appendEntries to: peer="{Voter server-a57dd488-0452-af56-2c9e-11d8f01801ca a57dd488-0452-af56-2c9e-11d8f01801ca}" error="failed to connect to peer: a57dd488-0452-af56-2c9e-11d8f01801ca"
    raft_test.go:1669: [INFO] Finished apply without behind follower
2025-04-21T21:24:21.333Z [ERROR] server-a57dd488-0452-af56-2c9e-11d8f01801ca: failed to take snapshot: error="nothing new to snapshot"
2025-04-21T21:24:21.333Z [INFO]  server-d574d151-3397-db2f-400f-34b46d8b64ee: starting snapshot up to: index=102
2025-04-21T21:24:21.333Z [INFO]  creating new snapshot: path=/tmp/raft3351971560/snapshots/2-102-1745270661333.tmp
2025-04-21T21:24:21.334Z [INFO]  server-d574d151-3397-db2f-400f-34b46d8b64ee: compacting logs: from=1 to=92
2025-04-21T21:24:21.334Z [INFO]  server-d574d151-3397-db2f-400f-34b46d8b64ee: snapshot complete up to: index=102
2025-04-21T21:24:21.334Z [INFO]  server-6f2ffa20-148c-62cc-1188-04495624e1e8: starting snapshot up to: index=98
2025-04-21T21:24:21.334Z [INFO]  creating new snapshot: path=/tmp/raft769316057/snapshots/2-98-1745270661334.tmp
2025-04-21T21:24:21.334Z [INFO]  server-6f2ffa20-148c-62cc-1188-04495624e1e8: compacting logs: from=1 to=92
2025-04-21T21:24:21.334Z [INFO]  server-6f2ffa20-148c-62cc-1188-04495624e1e8: snapshot complete up to: index=98
2025-04-21T21:24:21.335Z [ERROR] server-d574d151-3397-db2f-400f-34b46d8b64ee: failed to get log: index=2 error="log not found"
2025-04-21T21:24:21.335Z [ERROR] server-d574d151-3397-db2f-400f-34b46d8b64ee: failed to install snapshot: id=2-102-1745270661333 error="failed to connect to peer: a57dd488-0452-af56-2c9e-11d8f01801ca"
2025-04-21T21:24:21.335Z [ERROR] server-d574d151-3397-db2f-400f-34b46d8b64ee: failed to send snapshot to: peer="{Voter server-a57dd488-0452-af56-2c9e-11d8f01801ca a57dd488-0452-af56-2c9e-11d8f01801ca}" error="failed to connect to peer: a57dd488-0452-af56-2c9e-11d8f01801ca"
2025-04-21T21:24:21.345Z [ERROR] server-d574d151-3397-db2f-400f-34b46d8b64ee: failed to heartbeat to: peer=a57dd488-0452-af56-2c9e-11d8f01801ca backoff time=10ms error="failed to connect to peer: a57dd488-0452-af56-2c9e-11d8f01801ca"
2025-04-21T21:24:21.345Z [ERROR] server-d574d151-3397-db2f-400f-34b46d8b64ee: failed to get log: index=2 error="log not found"
2025-04-21T21:24:21.345Z [ERROR] server-d574d151-3397-db2f-400f-34b46d8b64ee: failed to install snapshot: id=2-102-1745270661333 error="failed to connect to peer: a57dd488-0452-af56-2c9e-11d8f01801ca"
2025-04-21T21:24:21.345Z [ERROR] server-d574d151-3397-db2f-400f-34b46d8b64ee: failed to send snapshot to: peer="{Voter server-a57dd488-0452-af56-2c9e-11d8f01801ca a57dd488-0452-af56-2c9e-11d8f01801ca}" error="failed to connect to peer: a57dd488-0452-af56-2c9e-11d8f01801ca"
    raft_test.go:1690: [INFO] Finished apply without behind follower
2025-04-21T21:24:21.369Z [ERROR] server-d574d151-3397-db2f-400f-34b46d8b64ee: failed to get log: index=2 error="log not found"
2025-04-21T21:24:21.369Z [INFO]  creating new snapshot: path=/tmp/raft47359521/snapshots/2-102-1745270661369.tmp
2025-04-21T21:24:21.369Z [INFO]  server-a57dd488-0452-af56-2c9e-11d8f01801ca: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-04-21T21:24:21.369Z [INFO]  server-a57dd488-0452-af56-2c9e-11d8f01801ca: copied to local snapshot: bytes=693
2025-04-21T21:24:21.369Z [INFO]  server-a57dd488-0452-af56-2c9e-11d8f01801ca: snapshot restore progress: id=2-102-1745270661369 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-04-21T21:24:21.369Z [INFO]  server-a57dd488-0452-af56-2c9e-11d8f01801ca: Installed remote snapshot
2025-04-21T21:24:21.370Z [INFO]  server-d574d151-3397-db2f-400f-34b46d8b64ee: pipelining replication: peer="{Voter server-a57dd488-0452-af56-2c9e-11d8f01801ca a57dd488-0452-af56-2c9e-11d8f01801ca}"
2025-04-21T21:24:21.370Z [INFO]  server-d574d151-3397-db2f-400f-34b46d8b64ee: aborting pipeline replication: peer="{Voter server-6f2ffa20-148c-62cc-1188-04495624e1e8 6f2ffa20-148c-62cc-1188-04495624e1e8}"
2025-04-21T21:24:21.370Z [INFO]  server-d574d151-3397-db2f-400f-34b46d8b64ee: aborting pipeline replication: peer="{Voter server-a57dd488-0452-af56-2c9e-11d8f01801ca a57dd488-0452-af56-2c9e-11d8f01801ca}"
--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.33s)
=== RUN   TestRaft_ReJoinFollower
2025-04-21T21:24:21.370Z [INFO]  server-4414aa22-34ff-6b02-f46b-80b2502ecb1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4414aa22-34ff-6b02-f46b-80b2502ecb1b Address:4414aa22-34ff-6b02-f46b-80b2502ecb1b} {Suffrage:Voter ID:server-09cc6cab-bda5-747d-6ee3-aee138cd2e90 Address:09cc6cab-bda5-747d-6ee3-aee138cd2e90} {Suffrage:Voter ID:server-a099c79a-ff6c-cd98-20b1-5ab4dae07fdd Address:a099c79a-ff6c-cd98-20b1-5ab4dae07fdd}]"
2025-04-21T21:24:21.371Z [INFO]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4414aa22-34ff-6b02-f46b-80b2502ecb1b Address:4414aa22-34ff-6b02-f46b-80b2502ecb1b} {Suffrage:Voter ID:server-09cc6cab-bda5-747d-6ee3-aee138cd2e90 Address:09cc6cab-bda5-747d-6ee3-aee138cd2e90} {Suffrage:Voter ID:server-a099c79a-ff6c-cd98-20b1-5ab4dae07fdd Address:a099c79a-ff6c-cd98-20b1-5ab4dae07fdd}]"
2025-04-21T21:24:21.371Z [INFO]  server-a099c79a-ff6c-cd98-20b1-5ab4dae07fdd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4414aa22-34ff-6b02-f46b-80b2502ecb1b Address:4414aa22-34ff-6b02-f46b-80b2502ecb1b} {Suffrage:Voter ID:server-09cc6cab-bda5-747d-6ee3-aee138cd2e90 Address:09cc6cab-bda5-747d-6ee3-aee138cd2e90} {Suffrage:Voter ID:server-a099c79a-ff6c-cd98-20b1-5ab4dae07fdd Address:a099c79a-ff6c-cd98-20b1-5ab4dae07fdd}]"
2025-04-21T21:24:21.371Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:21.371Z [INFO]  server-4414aa22-34ff-6b02-f46b-80b2502ecb1b: entering follower state: follower="Node at 4414aa22-34ff-6b02-f46b-80b2502ecb1b [Follower]" leader-address= leader-id=
2025-04-21T21:24:21.371Z [INFO]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: entering follower state: follower="Node at 09cc6cab-bda5-747d-6ee3-aee138cd2e90 [Follower]" leader-address= leader-id=
2025-04-21T21:24:21.371Z [INFO]  server-a099c79a-ff6c-cd98-20b1-5ab4dae07fdd: entering follower state: follower="Node at a099c79a-ff6c-cd98-20b1-5ab4dae07fdd [Follower]" leader-address= leader-id=
2025-04-21T21:24:21.429Z [WARN]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:21.429Z [INFO]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: entering candidate state: node="Node at 09cc6cab-bda5-747d-6ee3-aee138cd2e90 [Candidate]" term=2
2025-04-21T21:24:21.429Z [INFO]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: election won: term=2 tally=2
2025-04-21T21:24:21.429Z [INFO]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: entering leader state: leader="Node at 09cc6cab-bda5-747d-6ee3-aee138cd2e90 [Leader]"
2025-04-21T21:24:21.429Z [INFO]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: added peer, starting replication: peer=server-4414aa22-34ff-6b02-f46b-80b2502ecb1b
2025-04-21T21:24:21.429Z [INFO]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: added peer, starting replication: peer=server-a099c79a-ff6c-cd98-20b1-5ab4dae07fdd
2025-04-21T21:24:21.429Z [INFO]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: pipelining replication: peer="{Voter server-a099c79a-ff6c-cd98-20b1-5ab4dae07fdd a099c79a-ff6c-cd98-20b1-5ab4dae07fdd}"
2025-04-21T21:24:21.429Z [INFO]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: pipelining replication: peer="{Voter server-4414aa22-34ff-6b02-f46b-80b2502ecb1b 4414aa22-34ff-6b02-f46b-80b2502ecb1b}"
2025-04-21T21:24:21.537Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:21.42978872 +0000 UTC m=+99.951448043 (1 nodes), 58.726394ms from start of poll, 58.904876ms from cluster start. Timeout at 2025-04-21 21:24:21.53479232 +0000 UTC m=+100.056451643, 105.0036ms after stability
2025-04-21T21:24:21.545Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:21.657Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:21.545163656 +0000 UTC m=+100.066822979 (2 nodes), 3.76µs from start of poll, 174.279812ms from cluster start. Timeout at 2025-04-21 21:24:21.650205896 +0000 UTC m=+100.171865219, 105.04224ms after stability
2025-04-21T21:24:21.657Z [INFO]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: updating configuration: command=RemoveServer server-id=server-4414aa22-34ff-6b02-f46b-80b2502ecb1b server-addr= servers="[{Suffrage:Voter ID:server-09cc6cab-bda5-747d-6ee3-aee138cd2e90 Address:09cc6cab-bda5-747d-6ee3-aee138cd2e90} {Suffrage:Voter ID:server-a099c79a-ff6c-cd98-20b1-5ab4dae07fdd Address:a099c79a-ff6c-cd98-20b1-5ab4dae07fdd}]"
2025-04-21T21:24:21.657Z [INFO]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: removed peer, stopping replication: peer=server-4414aa22-34ff-6b02-f46b-80b2502ecb1b last-index=3
2025-04-21T21:24:21.657Z [INFO]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: aborting pipeline replication: peer="{Voter server-4414aa22-34ff-6b02-f46b-80b2502ecb1b 4414aa22-34ff-6b02-f46b-80b2502ecb1b}"
2025-04-21T21:24:21.721Z [WARN]  server-4414aa22-34ff-6b02-f46b-80b2502ecb1b: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-04-21T21:24:21.765Z [INFO]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: updating configuration: command=AddVoter server-id=server-4414aa22-34ff-6b02-f46b-80b2502ecb1b server-addr=4414aa22-34ff-6b02-f46b-80b2502ecb1b servers="[{Suffrage:Voter ID:server-09cc6cab-bda5-747d-6ee3-aee138cd2e90 Address:09cc6cab-bda5-747d-6ee3-aee138cd2e90} {Suffrage:Voter ID:server-a099c79a-ff6c-cd98-20b1-5ab4dae07fdd Address:a099c79a-ff6c-cd98-20b1-5ab4dae07fdd} {Suffrage:Voter ID:server-4414aa22-34ff-6b02-f46b-80b2502ecb1b Address:4414aa22-34ff-6b02-f46b-80b2502ecb1b}]"
2025-04-21T21:24:21.765Z [INFO]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: added peer, starting replication: peer=server-4414aa22-34ff-6b02-f46b-80b2502ecb1b
2025-04-21T21:24:21.765Z [WARN]  server-4414aa22-34ff-6b02-f46b-80b2502ecb1b: failed to get previous log: previous-index=4 last-index=3 error="log not found"
2025-04-21T21:24:21.765Z [WARN]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: appendEntries rejected, sending older logs: peer="{Voter server-4414aa22-34ff-6b02-f46b-80b2502ecb1b 4414aa22-34ff-6b02-f46b-80b2502ecb1b}" next=4
2025-04-21T21:24:21.765Z [INFO]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: pipelining replication: peer="{Voter server-4414aa22-34ff-6b02-f46b-80b2502ecb1b 4414aa22-34ff-6b02-f46b-80b2502ecb1b}"
2025-04-21T21:24:21.765Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:21.873Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:21.765545977 +0000 UTC m=+100.287205300 (1 nodes), 360ns from start of poll, 394.662133ms from cluster start. Timeout at 2025-04-21 21:24:21.870604617 +0000 UTC m=+100.392263940, 105.05864ms after stability
2025-04-21T21:24:21.873Z [INFO]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: aborting pipeline replication: peer="{Voter server-a099c79a-ff6c-cd98-20b1-5ab4dae07fdd a099c79a-ff6c-cd98-20b1-5ab4dae07fdd}"
2025-04-21T21:24:21.873Z [INFO]  server-09cc6cab-bda5-747d-6ee3-aee138cd2e90: aborting pipeline replication: peer="{Voter server-4414aa22-34ff-6b02-f46b-80b2502ecb1b 4414aa22-34ff-6b02-f46b-80b2502ecb1b}"
--- PASS: TestRaft_ReJoinFollower (0.50s)
=== RUN   TestRaft_LeaderLeaseExpire
2025-04-21T21:24:21.873Z [INFO]  server-817029f3-4410-a6a8-dfff-44921d4a0316: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-817029f3-4410-a6a8-dfff-44921d4a0316 Address:817029f3-4410-a6a8-dfff-44921d4a0316} {Suffrage:Voter ID:server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84 Address:9c5b33ef-84b1-d74c-6d35-b3e936cb8c84}]"
2025-04-21T21:24:21.874Z [INFO]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-817029f3-4410-a6a8-dfff-44921d4a0316 Address:817029f3-4410-a6a8-dfff-44921d4a0316} {Suffrage:Voter ID:server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84 Address:9c5b33ef-84b1-d74c-6d35-b3e936cb8c84}]"
2025-04-21T21:24:21.874Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:21.874Z [INFO]  server-817029f3-4410-a6a8-dfff-44921d4a0316: entering follower state: follower="Node at 817029f3-4410-a6a8-dfff-44921d4a0316 [Follower]" leader-address= leader-id=
2025-04-21T21:24:21.874Z [INFO]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: entering follower state: follower="Node at 9c5b33ef-84b1-d74c-6d35-b3e936cb8c84 [Follower]" leader-address= leader-id=
2025-04-21T21:24:21.949Z [WARN]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:21.949Z [INFO]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: entering candidate state: node="Node at 9c5b33ef-84b1-d74c-6d35-b3e936cb8c84 [Candidate]" term=2
2025-04-21T21:24:21.949Z [INFO]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: election won: term=2 tally=2
2025-04-21T21:24:21.949Z [INFO]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: entering leader state: leader="Node at 9c5b33ef-84b1-d74c-6d35-b3e936cb8c84 [Leader]"
2025-04-21T21:24:21.949Z [INFO]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: added peer, starting replication: peer=server-817029f3-4410-a6a8-dfff-44921d4a0316
2025-04-21T21:24:21.949Z [INFO]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: pipelining replication: peer="{Voter server-817029f3-4410-a6a8-dfff-44921d4a0316 817029f3-4410-a6a8-dfff-44921d4a0316}"
2025-04-21T21:24:22.061Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:21.949687615 +0000 UTC m=+100.471346938 (1 nodes), 75.649589ms from start of poll, 75.81315ms from cluster start. Timeout at 2025-04-21 21:24:22.054694375 +0000 UTC m=+100.576353698, 105.00676ms after stability
2025-04-21T21:24:22.069Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:22.185Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:22.069261938 +0000 UTC m=+100.590921261 (1 nodes), 1.44µs from start of poll, 195.387473ms from cluster start. Timeout at 2025-04-21 21:24:22.174281818 +0000 UTC m=+100.695941101, 105.01984ms after stability
    raft_test.go:1799: [INFO] Disconnecting Node at 817029f3-4410-a6a8-dfff-44921d4a0316 [Follower]
2025-04-21T21:24:22.185Z [ERROR] server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: failed to heartbeat to: peer=817029f3-4410-a6a8-dfff-44921d4a0316 backoff time=10ms error="failed to connect to peer: 817029f3-4410-a6a8-dfff-44921d4a0316"
2025-04-21T21:24:22.193Z [ERROR] server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: failed to pipeline appendEntries: peer="{Voter server-817029f3-4410-a6a8-dfff-44921d4a0316 817029f3-4410-a6a8-dfff-44921d4a0316}" error="append pipeline closed"
2025-04-21T21:24:22.193Z [INFO]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: aborting pipeline replication: peer="{Voter server-817029f3-4410-a6a8-dfff-44921d4a0316 817029f3-4410-a6a8-dfff-44921d4a0316}"
2025-04-21T21:24:22.201Z [ERROR] server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: failed to appendEntries to: peer="{Voter server-817029f3-4410-a6a8-dfff-44921d4a0316 817029f3-4410-a6a8-dfff-44921d4a0316}" error="failed to connect to peer: 817029f3-4410-a6a8-dfff-44921d4a0316"
2025-04-21T21:24:22.205Z [ERROR] server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: failed to heartbeat to: peer=817029f3-4410-a6a8-dfff-44921d4a0316 backoff time=10ms error="failed to connect to peer: 817029f3-4410-a6a8-dfff-44921d4a0316"
2025-04-21T21:24:22.221Z [WARN]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: failed to contact: server-id=server-817029f3-4410-a6a8-dfff-44921d4a0316 time=51.679947ms
2025-04-21T21:24:22.221Z [WARN]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: failed to contact quorum of nodes, stepping down
2025-04-21T21:24:22.221Z [INFO]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: entering follower state: follower="Node at 9c5b33ef-84b1-d74c-6d35-b3e936cb8c84 [Follower]" leader-address= leader-id=
2025-04-21T21:24:22.221Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:22.221Z [ERROR] server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: failed to appendEntries to: peer="{Voter server-817029f3-4410-a6a8-dfff-44921d4a0316 817029f3-4410-a6a8-dfff-44921d4a0316}" error="failed to connect to peer: 817029f3-4410-a6a8-dfff-44921d4a0316"
2025-04-21T21:24:22.269Z [WARN]  server-817029f3-4410-a6a8-dfff-44921d4a0316: heartbeat timeout reached, starting election: last-leader-addr=9c5b33ef-84b1-d74c-6d35-b3e936cb8c84 last-leader-id=server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84
2025-04-21T21:24:22.269Z [INFO]  server-817029f3-4410-a6a8-dfff-44921d4a0316: entering candidate state: node="Node at 817029f3-4410-a6a8-dfff-44921d4a0316 [Candidate]" term=3
2025-04-21T21:24:22.269Z [ERROR] server-817029f3-4410-a6a8-dfff-44921d4a0316: failed to make requestVote RPC: target="{Voter server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84 9c5b33ef-84b1-d74c-6d35-b3e936cb8c84}" error="failed to connect to peer: 9c5b33ef-84b1-d74c-6d35-b3e936cb8c84" term=3
2025-04-21T21:24:22.285Z [WARN]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:22.285Z [INFO]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: entering candidate state: node="Node at 9c5b33ef-84b1-d74c-6d35-b3e936cb8c84 [Candidate]" term=3
2025-04-21T21:24:22.285Z [ERROR] server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: failed to make requestVote RPC: target="{Voter server-817029f3-4410-a6a8-dfff-44921d4a0316 817029f3-4410-a6a8-dfff-44921d4a0316}" error="failed to connect to peer: 817029f3-4410-a6a8-dfff-44921d4a0316" term=3
2025-04-21T21:24:22.357Z [WARN]  server-817029f3-4410-a6a8-dfff-44921d4a0316: Election timeout reached, restarting election
2025-04-21T21:24:22.357Z [INFO]  server-817029f3-4410-a6a8-dfff-44921d4a0316: entering candidate state: node="Node at 817029f3-4410-a6a8-dfff-44921d4a0316 [Candidate]" term=4
2025-04-21T21:24:22.357Z [ERROR] server-817029f3-4410-a6a8-dfff-44921d4a0316: failed to make requestVote RPC: target="{Voter server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84 9c5b33ef-84b1-d74c-6d35-b3e936cb8c84}" error="failed to connect to peer: 9c5b33ef-84b1-d74c-6d35-b3e936cb8c84" term=4
2025-04-21T21:24:22.357Z [WARN]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: Election timeout reached, restarting election
2025-04-21T21:24:22.357Z [INFO]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: entering candidate state: node="Node at 9c5b33ef-84b1-d74c-6d35-b3e936cb8c84 [Candidate]" term=4
2025-04-21T21:24:22.357Z [ERROR] server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: failed to make requestVote RPC: target="{Voter server-817029f3-4410-a6a8-dfff-44921d4a0316 817029f3-4410-a6a8-dfff-44921d4a0316}" error="failed to connect to peer: 817029f3-4410-a6a8-dfff-44921d4a0316" term=4
2025-04-21T21:24:22.393Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:22.285868034 +0000 UTC m=+100.807527357 (0 nodes), 64.643834ms from start of poll, 411.993569ms from cluster start. Timeout at 2025-04-21 21:24:22.390875354 +0000 UTC m=+100.912534677, 105.00732ms after stability
2025-04-21T21:24:22.441Z [WARN]  server-817029f3-4410-a6a8-dfff-44921d4a0316: Election timeout reached, restarting election
2025-04-21T21:24:22.441Z [INFO]  server-817029f3-4410-a6a8-dfff-44921d4a0316: entering candidate state: node="Node at 817029f3-4410-a6a8-dfff-44921d4a0316 [Candidate]" term=5
2025-04-21T21:24:22.441Z [ERROR] server-817029f3-4410-a6a8-dfff-44921d4a0316: failed to make requestVote RPC: target="{Voter server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84 9c5b33ef-84b1-d74c-6d35-b3e936cb8c84}" error="failed to connect to peer: 9c5b33ef-84b1-d74c-6d35-b3e936cb8c84" term=5
2025-04-21T21:24:22.449Z [WARN]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: Election timeout reached, restarting election
2025-04-21T21:24:22.449Z [INFO]  server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: entering candidate state: node="Node at 9c5b33ef-84b1-d74c-6d35-b3e936cb8c84 [Candidate]" term=5
2025-04-21T21:24:22.449Z [ERROR] server-9c5b33ef-84b1-d74c-6d35-b3e936cb8c84: failed to make requestVote RPC: target="{Voter server-817029f3-4410-a6a8-dfff-44921d4a0316 817029f3-4410-a6a8-dfff-44921d4a0316}" error="failed to connect to peer: 817029f3-4410-a6a8-dfff-44921d4a0316" term=5
--- PASS: TestRaft_LeaderLeaseExpire (0.63s)
=== RUN   TestRaft_Barrier
2025-04-21T21:24:22.501Z [INFO]  server-3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa Address:3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa} {Suffrage:Voter ID:server-f313f854-ab06-9f22-726b-61f6f957d105 Address:f313f854-ab06-9f22-726b-61f6f957d105} {Suffrage:Voter ID:server-1ec7bfcc-e2f5-603c-36c0-e101fdde55f9 Address:1ec7bfcc-e2f5-603c-36c0-e101fdde55f9}]"
2025-04-21T21:24:22.501Z [INFO]  server-f313f854-ab06-9f22-726b-61f6f957d105: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa Address:3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa} {Suffrage:Voter ID:server-f313f854-ab06-9f22-726b-61f6f957d105 Address:f313f854-ab06-9f22-726b-61f6f957d105} {Suffrage:Voter ID:server-1ec7bfcc-e2f5-603c-36c0-e101fdde55f9 Address:1ec7bfcc-e2f5-603c-36c0-e101fdde55f9}]"
2025-04-21T21:24:22.501Z [INFO]  server-1ec7bfcc-e2f5-603c-36c0-e101fdde55f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa Address:3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa} {Suffrage:Voter ID:server-f313f854-ab06-9f22-726b-61f6f957d105 Address:f313f854-ab06-9f22-726b-61f6f957d105} {Suffrage:Voter ID:server-1ec7bfcc-e2f5-603c-36c0-e101fdde55f9 Address:1ec7bfcc-e2f5-603c-36c0-e101fdde55f9}]"
2025-04-21T21:24:22.501Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:22.501Z [INFO]  server-3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa: entering follower state: follower="Node at 3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa [Follower]" leader-address= leader-id=
2025-04-21T21:24:22.502Z [INFO]  server-f313f854-ab06-9f22-726b-61f6f957d105: entering follower state: follower="Node at f313f854-ab06-9f22-726b-61f6f957d105 [Follower]" leader-address= leader-id=
2025-04-21T21:24:22.502Z [INFO]  server-1ec7bfcc-e2f5-603c-36c0-e101fdde55f9: entering follower state: follower="Node at 1ec7bfcc-e2f5-603c-36c0-e101fdde55f9 [Follower]" leader-address= leader-id=
2025-04-21T21:24:22.573Z [WARN]  server-3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:22.573Z [INFO]  server-3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa: entering candidate state: node="Node at 3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa [Candidate]" term=2
2025-04-21T21:24:22.573Z [INFO]  server-3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa: election won: term=2 tally=2
2025-04-21T21:24:22.573Z [INFO]  server-3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa: entering leader state: leader="Node at 3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa [Leader]"
2025-04-21T21:24:22.573Z [INFO]  server-3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa: added peer, starting replication: peer=server-f313f854-ab06-9f22-726b-61f6f957d105
2025-04-21T21:24:22.573Z [INFO]  server-3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa: added peer, starting replication: peer=server-1ec7bfcc-e2f5-603c-36c0-e101fdde55f9
2025-04-21T21:24:22.573Z [INFO]  server-3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa: pipelining replication: peer="{Voter server-1ec7bfcc-e2f5-603c-36c0-e101fdde55f9 1ec7bfcc-e2f5-603c-36c0-e101fdde55f9}"
2025-04-21T21:24:22.573Z [INFO]  server-3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa: pipelining replication: peer="{Voter server-f313f854-ab06-9f22-726b-61f6f957d105 f313f854-ab06-9f22-726b-61f6f957d105}"
2025-04-21T21:24:22.685Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:22.573633688 +0000 UTC m=+101.095293011 (1 nodes), 71.692962ms from start of poll, 71.878483ms from cluster start. Timeout at 2025-04-21 21:24:22.678642648 +0000 UTC m=+101.200301931, 105.00892ms after stability
2025-04-21T21:24:22.713Z [INFO]  server-3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa: aborting pipeline replication: peer="{Voter server-1ec7bfcc-e2f5-603c-36c0-e101fdde55f9 1ec7bfcc-e2f5-603c-36c0-e101fdde55f9}"
2025-04-21T21:24:22.713Z [INFO]  server-3c8ecc0c-8eb3-ffed-4762-09f77eff7bfa: aborting pipeline replication: peer="{Voter server-f313f854-ab06-9f22-726b-61f6f957d105 f313f854-ab06-9f22-726b-61f6f957d105}"
--- PASS: TestRaft_Barrier (0.21s)
=== RUN   TestRaft_VerifyLeader
2025-04-21T21:24:22.713Z [INFO]  server-4e339e00-0e70-2179-a022-bc5e3814d372: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e339e00-0e70-2179-a022-bc5e3814d372 Address:4e339e00-0e70-2179-a022-bc5e3814d372} {Suffrage:Voter ID:server-ee11e637-a1d7-7de7-3091-c163a4212ccc Address:ee11e637-a1d7-7de7-3091-c163a4212ccc} {Suffrage:Voter ID:server-be53fa17-7ff2-25e8-f914-936c59d47bf1 Address:be53fa17-7ff2-25e8-f914-936c59d47bf1}]"
2025-04-21T21:24:22.713Z [INFO]  server-ee11e637-a1d7-7de7-3091-c163a4212ccc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e339e00-0e70-2179-a022-bc5e3814d372 Address:4e339e00-0e70-2179-a022-bc5e3814d372} {Suffrage:Voter ID:server-ee11e637-a1d7-7de7-3091-c163a4212ccc Address:ee11e637-a1d7-7de7-3091-c163a4212ccc} {Suffrage:Voter ID:server-be53fa17-7ff2-25e8-f914-936c59d47bf1 Address:be53fa17-7ff2-25e8-f914-936c59d47bf1}]"
2025-04-21T21:24:22.713Z [INFO]  server-be53fa17-7ff2-25e8-f914-936c59d47bf1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e339e00-0e70-2179-a022-bc5e3814d372 Address:4e339e00-0e70-2179-a022-bc5e3814d372} {Suffrage:Voter ID:server-ee11e637-a1d7-7de7-3091-c163a4212ccc Address:ee11e637-a1d7-7de7-3091-c163a4212ccc} {Suffrage:Voter ID:server-be53fa17-7ff2-25e8-f914-936c59d47bf1 Address:be53fa17-7ff2-25e8-f914-936c59d47bf1}]"
2025-04-21T21:24:22.713Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:22.713Z [INFO]  server-4e339e00-0e70-2179-a022-bc5e3814d372: entering follower state: follower="Node at 4e339e00-0e70-2179-a022-bc5e3814d372 [Follower]" leader-address= leader-id=
2025-04-21T21:24:22.713Z [INFO]  server-ee11e637-a1d7-7de7-3091-c163a4212ccc: entering follower state: follower="Node at ee11e637-a1d7-7de7-3091-c163a4212ccc [Follower]" leader-address= leader-id=
2025-04-21T21:24:22.713Z [INFO]  server-be53fa17-7ff2-25e8-f914-936c59d47bf1: entering follower state: follower="Node at be53fa17-7ff2-25e8-f914-936c59d47bf1 [Follower]" leader-address= leader-id=
2025-04-21T21:24:22.781Z [WARN]  server-ee11e637-a1d7-7de7-3091-c163a4212ccc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:22.781Z [INFO]  server-ee11e637-a1d7-7de7-3091-c163a4212ccc: entering candidate state: node="Node at ee11e637-a1d7-7de7-3091-c163a4212ccc [Candidate]" term=2
2025-04-21T21:24:22.781Z [INFO]  server-ee11e637-a1d7-7de7-3091-c163a4212ccc: election won: term=2 tally=2
2025-04-21T21:24:22.781Z [INFO]  server-ee11e637-a1d7-7de7-3091-c163a4212ccc: entering leader state: leader="Node at ee11e637-a1d7-7de7-3091-c163a4212ccc [Leader]"
2025-04-21T21:24:22.781Z [INFO]  server-ee11e637-a1d7-7de7-3091-c163a4212ccc: added peer, starting replication: peer=server-4e339e00-0e70-2179-a022-bc5e3814d372
2025-04-21T21:24:22.781Z [INFO]  server-ee11e637-a1d7-7de7-3091-c163a4212ccc: added peer, starting replication: peer=server-be53fa17-7ff2-25e8-f914-936c59d47bf1
2025-04-21T21:24:22.781Z [INFO]  server-ee11e637-a1d7-7de7-3091-c163a4212ccc: pipelining replication: peer="{Voter server-be53fa17-7ff2-25e8-f914-936c59d47bf1 be53fa17-7ff2-25e8-f914-936c59d47bf1}"
2025-04-21T21:24:22.781Z [INFO]  server-ee11e637-a1d7-7de7-3091-c163a4212ccc: pipelining replication: peer="{Voter server-4e339e00-0e70-2179-a022-bc5e3814d372 4e339e00-0e70-2179-a022-bc5e3814d372}"
2025-04-21T21:24:22.905Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:22.781481605 +0000 UTC m=+101.303140928 (1 nodes), 67.753575ms from start of poll, 67.944216ms from cluster start. Timeout at 2025-04-21 21:24:22.886484125 +0000 UTC m=+101.408143448, 105.00252ms after stability
2025-04-21T21:24:22.915Z [INFO]  server-ee11e637-a1d7-7de7-3091-c163a4212ccc: aborting pipeline replication: peer="{Voter server-be53fa17-7ff2-25e8-f914-936c59d47bf1 be53fa17-7ff2-25e8-f914-936c59d47bf1}"
2025-04-21T21:24:22.915Z [INFO]  server-ee11e637-a1d7-7de7-3091-c163a4212ccc: aborting pipeline replication: peer="{Voter server-4e339e00-0e70-2179-a022-bc5e3814d372 4e339e00-0e70-2179-a022-bc5e3814d372}"
--- PASS: TestRaft_VerifyLeader (0.20s)
=== RUN   TestRaft_VerifyLeader_Single
2025-04-21T21:24:22.916Z [INFO]  server-414c9574-a3c5-3b59-f5d7-14d783858b9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-414c9574-a3c5-3b59-f5d7-14d783858b9d Address:414c9574-a3c5-3b59-f5d7-14d783858b9d}]"
2025-04-21T21:24:22.916Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:22.916Z [INFO]  server-414c9574-a3c5-3b59-f5d7-14d783858b9d: entering follower state: follower="Node at 414c9574-a3c5-3b59-f5d7-14d783858b9d [Follower]" leader-address= leader-id=
2025-04-21T21:24:22.981Z [WARN]  server-414c9574-a3c5-3b59-f5d7-14d783858b9d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:22.981Z [INFO]  server-414c9574-a3c5-3b59-f5d7-14d783858b9d: entering candidate state: node="Node at 414c9574-a3c5-3b59-f5d7-14d783858b9d [Candidate]" term=2
2025-04-21T21:24:22.981Z [INFO]  server-414c9574-a3c5-3b59-f5d7-14d783858b9d: election won: term=2 tally=1
2025-04-21T21:24:22.981Z [INFO]  server-414c9574-a3c5-3b59-f5d7-14d783858b9d: entering leader state: leader="Node at 414c9574-a3c5-3b59-f5d7-14d783858b9d [Leader]"
2025-04-21T21:24:23.097Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:22.981448269 +0000 UTC m=+101.503107552 (1 nodes), 65.343919ms from start of poll, 65.44496ms from cluster start. Timeout at 2025-04-21 21:24:23.086452109 +0000 UTC m=+101.608111432, 105.00388ms after stability
--- PASS: TestRaft_VerifyLeader_Single (0.18s)
=== RUN   TestRaft_VerifyLeader_Fail
2025-04-21T21:24:23.097Z [INFO]  server-dd45b9a2-0ee8-1606-bdb0-9c63c4cd3534: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd45b9a2-0ee8-1606-bdb0-9c63c4cd3534 Address:dd45b9a2-0ee8-1606-bdb0-9c63c4cd3534} {Suffrage:Voter ID:server-4be64684-5974-e82f-b590-2916f120bf8b Address:4be64684-5974-e82f-b590-2916f120bf8b}]"
2025-04-21T21:24:23.097Z [INFO]  server-4be64684-5974-e82f-b590-2916f120bf8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd45b9a2-0ee8-1606-bdb0-9c63c4cd3534 Address:dd45b9a2-0ee8-1606-bdb0-9c63c4cd3534} {Suffrage:Voter ID:server-4be64684-5974-e82f-b590-2916f120bf8b Address:4be64684-5974-e82f-b590-2916f120bf8b}]"
2025-04-21T21:24:23.097Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:23.098Z [INFO]  server-dd45b9a2-0ee8-1606-bdb0-9c63c4cd3534: entering follower state: follower="Node at dd45b9a2-0ee8-1606-bdb0-9c63c4cd3534 [Follower]" leader-address= leader-id=
2025-04-21T21:24:23.098Z [INFO]  server-4be64684-5974-e82f-b590-2916f120bf8b: entering follower state: follower="Node at 4be64684-5974-e82f-b590-2916f120bf8b [Follower]" leader-address= leader-id=
2025-04-21T21:24:23.153Z [WARN]  server-4be64684-5974-e82f-b590-2916f120bf8b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:23.153Z [INFO]  server-4be64684-5974-e82f-b590-2916f120bf8b: entering candidate state: node="Node at 4be64684-5974-e82f-b590-2916f120bf8b [Candidate]" term=2
2025-04-21T21:24:23.154Z [INFO]  server-4be64684-5974-e82f-b590-2916f120bf8b: election won: term=2 tally=2
2025-04-21T21:24:23.154Z [INFO]  server-4be64684-5974-e82f-b590-2916f120bf8b: entering leader state: leader="Node at 4be64684-5974-e82f-b590-2916f120bf8b [Leader]"
2025-04-21T21:24:23.154Z [INFO]  server-4be64684-5974-e82f-b590-2916f120bf8b: added peer, starting replication: peer=server-dd45b9a2-0ee8-1606-bdb0-9c63c4cd3534
2025-04-21T21:24:23.154Z [INFO]  server-4be64684-5974-e82f-b590-2916f120bf8b: pipelining replication: peer="{Voter server-dd45b9a2-0ee8-1606-bdb0-9c63c4cd3534 dd45b9a2-0ee8-1606-bdb0-9c63c4cd3534}"
2025-04-21T21:24:23.261Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:23.15416931 +0000 UTC m=+101.675828633 (1 nodes), 56.403419ms from start of poll, 56.54598ms from cluster start. Timeout at 2025-04-21 21:24:23.25918323 +0000 UTC m=+101.780842553, 105.01392ms after stability
2025-04-21T21:24:23.261Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:23.369Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:23.26122095 +0000 UTC m=+101.782880273 (1 nodes), 1.32µs from start of poll, 163.59762ms from cluster start. Timeout at 2025-04-21 21:24:23.36624091 +0000 UTC m=+101.887900513, 105.02024ms after stability
2025-04-21T21:24:23.377Z [ERROR] server-4be64684-5974-e82f-b590-2916f120bf8b: peer has newer term, stopping replication: peer="{Voter server-dd45b9a2-0ee8-1606-bdb0-9c63c4cd3534 dd45b9a2-0ee8-1606-bdb0-9c63c4cd3534}"
2025-04-21T21:24:23.377Z [INFO]  server-4be64684-5974-e82f-b590-2916f120bf8b: aborting pipeline replication: peer="{Voter server-dd45b9a2-0ee8-1606-bdb0-9c63c4cd3534 dd45b9a2-0ee8-1606-bdb0-9c63c4cd3534}"
2025-04-21T21:24:23.377Z [INFO]  server-4be64684-5974-e82f-b590-2916f120bf8b: entering follower state: follower="Node at 4be64684-5974-e82f-b590-2916f120bf8b [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VerifyLeader_Fail (0.28s)
=== RUN   TestRaft_VerifyLeader_PartialConnect
2025-04-21T21:24:23.377Z [INFO]  server-a359e21f-3af8-df10-0eea-168598b585f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a359e21f-3af8-df10-0eea-168598b585f0 Address:a359e21f-3af8-df10-0eea-168598b585f0} {Suffrage:Voter ID:server-cd1e97c6-ee03-d446-3ce9-0ca18cb65713 Address:cd1e97c6-ee03-d446-3ce9-0ca18cb65713} {Suffrage:Voter ID:server-000142ea-b3eb-aa94-a697-bb114ad5889c Address:000142ea-b3eb-aa94-a697-bb114ad5889c}]"
2025-04-21T21:24:23.377Z [INFO]  server-cd1e97c6-ee03-d446-3ce9-0ca18cb65713: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a359e21f-3af8-df10-0eea-168598b585f0 Address:a359e21f-3af8-df10-0eea-168598b585f0} {Suffrage:Voter ID:server-cd1e97c6-ee03-d446-3ce9-0ca18cb65713 Address:cd1e97c6-ee03-d446-3ce9-0ca18cb65713} {Suffrage:Voter ID:server-000142ea-b3eb-aa94-a697-bb114ad5889c Address:000142ea-b3eb-aa94-a697-bb114ad5889c}]"
2025-04-21T21:24:23.377Z [INFO]  server-000142ea-b3eb-aa94-a697-bb114ad5889c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a359e21f-3af8-df10-0eea-168598b585f0 Address:a359e21f-3af8-df10-0eea-168598b585f0} {Suffrage:Voter ID:server-cd1e97c6-ee03-d446-3ce9-0ca18cb65713 Address:cd1e97c6-ee03-d446-3ce9-0ca18cb65713} {Suffrage:Voter ID:server-000142ea-b3eb-aa94-a697-bb114ad5889c Address:000142ea-b3eb-aa94-a697-bb114ad5889c}]"
2025-04-21T21:24:23.377Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:23.378Z [INFO]  server-a359e21f-3af8-df10-0eea-168598b585f0: entering follower state: follower="Node at a359e21f-3af8-df10-0eea-168598b585f0 [Follower]" leader-address= leader-id=
2025-04-21T21:24:23.378Z [INFO]  server-cd1e97c6-ee03-d446-3ce9-0ca18cb65713: entering follower state: follower="Node at cd1e97c6-ee03-d446-3ce9-0ca18cb65713 [Follower]" leader-address= leader-id=
2025-04-21T21:24:23.378Z [INFO]  server-000142ea-b3eb-aa94-a697-bb114ad5889c: entering follower state: follower="Node at 000142ea-b3eb-aa94-a697-bb114ad5889c [Follower]" leader-address= leader-id=
2025-04-21T21:24:23.433Z [WARN]  server-000142ea-b3eb-aa94-a697-bb114ad5889c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:23.433Z [INFO]  server-000142ea-b3eb-aa94-a697-bb114ad5889c: entering candidate state: node="Node at 000142ea-b3eb-aa94-a697-bb114ad5889c [Candidate]" term=2
2025-04-21T21:24:23.433Z [INFO]  server-000142ea-b3eb-aa94-a697-bb114ad5889c: election won: term=2 tally=2
2025-04-21T21:24:23.433Z [INFO]  server-000142ea-b3eb-aa94-a697-bb114ad5889c: entering leader state: leader="Node at 000142ea-b3eb-aa94-a697-bb114ad5889c [Leader]"
2025-04-21T21:24:23.433Z [INFO]  server-000142ea-b3eb-aa94-a697-bb114ad5889c: added peer, starting replication: peer=server-a359e21f-3af8-df10-0eea-168598b585f0
2025-04-21T21:24:23.433Z [INFO]  server-000142ea-b3eb-aa94-a697-bb114ad5889c: added peer, starting replication: peer=server-cd1e97c6-ee03-d446-3ce9-0ca18cb65713
2025-04-21T21:24:23.433Z [INFO]  server-000142ea-b3eb-aa94-a697-bb114ad5889c: pipelining replication: peer="{Voter server-cd1e97c6-ee03-d446-3ce9-0ca18cb65713 cd1e97c6-ee03-d446-3ce9-0ca18cb65713}"
2025-04-21T21:24:23.433Z [INFO]  server-000142ea-b3eb-aa94-a697-bb114ad5889c: pipelining replication: peer="{Voter server-a359e21f-3af8-df10-0eea-168598b585f0 a359e21f-3af8-df10-0eea-168598b585f0}"
2025-04-21T21:24:23.545Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:23.433550948 +0000 UTC m=+101.955210271 (1 nodes), 55.549373ms from start of poll, 55.775575ms from cluster start. Timeout at 2025-04-21 21:24:23.538553948 +0000 UTC m=+102.060213271, 105.003ms after stability
2025-04-21T21:24:23.553Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:23.661Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:23.553408394 +0000 UTC m=+102.075067717 (2 nodes), 1.6µs from start of poll, 175.633021ms from cluster start. Timeout at 2025-04-21 21:24:23.658467474 +0000 UTC m=+102.180126797, 105.05908ms after stability
    raft_test.go:1968: [INFO] Disconnecting Node at a359e21f-3af8-df10-0eea-168598b585f0 [Follower]
2025-04-21T21:24:23.661Z [ERROR] server-000142ea-b3eb-aa94-a697-bb114ad5889c: failed to heartbeat to: peer=a359e21f-3af8-df10-0eea-168598b585f0 backoff time=10ms error="failed to connect to peer: a359e21f-3af8-df10-0eea-168598b585f0"
2025-04-21T21:24:23.661Z [INFO]  server-000142ea-b3eb-aa94-a697-bb114ad5889c: aborting pipeline replication: peer="{Voter server-cd1e97c6-ee03-d446-3ce9-0ca18cb65713 cd1e97c6-ee03-d446-3ce9-0ca18cb65713}"
2025-04-21T21:24:23.661Z [INFO]  server-000142ea-b3eb-aa94-a697-bb114ad5889c: aborting pipeline replication: peer="{Voter server-a359e21f-3af8-df10-0eea-168598b585f0 a359e21f-3af8-df10-0eea-168598b585f0}"
--- PASS: TestRaft_VerifyLeader_PartialConnect (0.28s)
=== RUN   TestRaft_NotifyCh
2025-04-21T21:24:23.662Z [INFO]  server-da37981a-088d-da37-5503-e93e7bafca78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da37981a-088d-da37-5503-e93e7bafca78 Address:da37981a-088d-da37-5503-e93e7bafca78}]"
2025-04-21T21:24:23.662Z [INFO]  server-da37981a-088d-da37-5503-e93e7bafca78: entering follower state: follower="Node at da37981a-088d-da37-5503-e93e7bafca78 [Follower]" leader-address= leader-id=
2025-04-21T21:24:23.713Z [WARN]  server-da37981a-088d-da37-5503-e93e7bafca78: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:23.713Z [INFO]  server-da37981a-088d-da37-5503-e93e7bafca78: entering candidate state: node="Node at da37981a-088d-da37-5503-e93e7bafca78 [Candidate]" term=2
2025-04-21T21:24:23.713Z [INFO]  server-da37981a-088d-da37-5503-e93e7bafca78: election won: term=2 tally=1
2025-04-21T21:24:23.713Z [INFO]  server-da37981a-088d-da37-5503-e93e7bafca78: entering leader state: leader="Node at da37981a-088d-da37-5503-e93e7bafca78 [Leader]"
--- PASS: TestRaft_NotifyCh (0.05s)
=== RUN   TestRaft_AppendEntry
2025-04-21T21:24:23.714Z [INFO]  server-e17796a8-6257-ffef-ab4d-5b43d604d3ab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e17796a8-6257-ffef-ab4d-5b43d604d3ab Address:e17796a8-6257-ffef-ab4d-5b43d604d3ab} {Suffrage:Voter ID:server-8b2f82e4-bdb3-3027-97c4-26f1e4a04966 Address:8b2f82e4-bdb3-3027-97c4-26f1e4a04966} {Suffrage:Voter ID:server-57d04b74-2b23-94ee-2466-ac5dfa670b7a Address:57d04b74-2b23-94ee-2466-ac5dfa670b7a}]"
2025-04-21T21:24:23.714Z [INFO]  server-8b2f82e4-bdb3-3027-97c4-26f1e4a04966: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e17796a8-6257-ffef-ab4d-5b43d604d3ab Address:e17796a8-6257-ffef-ab4d-5b43d604d3ab} {Suffrage:Voter ID:server-8b2f82e4-bdb3-3027-97c4-26f1e4a04966 Address:8b2f82e4-bdb3-3027-97c4-26f1e4a04966} {Suffrage:Voter ID:server-57d04b74-2b23-94ee-2466-ac5dfa670b7a Address:57d04b74-2b23-94ee-2466-ac5dfa670b7a}]"
2025-04-21T21:24:23.714Z [INFO]  server-57d04b74-2b23-94ee-2466-ac5dfa670b7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e17796a8-6257-ffef-ab4d-5b43d604d3ab Address:e17796a8-6257-ffef-ab4d-5b43d604d3ab} {Suffrage:Voter ID:server-8b2f82e4-bdb3-3027-97c4-26f1e4a04966 Address:8b2f82e4-bdb3-3027-97c4-26f1e4a04966} {Suffrage:Voter ID:server-57d04b74-2b23-94ee-2466-ac5dfa670b7a Address:57d04b74-2b23-94ee-2466-ac5dfa670b7a}]"
2025-04-21T21:24:23.714Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:23.714Z [INFO]  server-e17796a8-6257-ffef-ab4d-5b43d604d3ab: entering follower state: follower="Node at e17796a8-6257-ffef-ab4d-5b43d604d3ab [Follower]" leader-address= leader-id=
2025-04-21T21:24:23.714Z [INFO]  server-8b2f82e4-bdb3-3027-97c4-26f1e4a04966: entering follower state: follower="Node at 8b2f82e4-bdb3-3027-97c4-26f1e4a04966 [Follower]" leader-address= leader-id=
2025-04-21T21:24:23.714Z [INFO]  server-57d04b74-2b23-94ee-2466-ac5dfa670b7a: entering follower state: follower="Node at 57d04b74-2b23-94ee-2466-ac5dfa670b7a [Follower]" leader-address= leader-id=
2025-04-21T21:24:23.769Z [WARN]  server-57d04b74-2b23-94ee-2466-ac5dfa670b7a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:23.769Z [INFO]  server-57d04b74-2b23-94ee-2466-ac5dfa670b7a: entering candidate state: node="Node at 57d04b74-2b23-94ee-2466-ac5dfa670b7a [Candidate]" term=2
2025-04-21T21:24:23.769Z [INFO]  server-57d04b74-2b23-94ee-2466-ac5dfa670b7a: election won: term=2 tally=2
2025-04-21T21:24:23.769Z [INFO]  server-57d04b74-2b23-94ee-2466-ac5dfa670b7a: entering leader state: leader="Node at 57d04b74-2b23-94ee-2466-ac5dfa670b7a [Leader]"
2025-04-21T21:24:23.769Z [INFO]  server-57d04b74-2b23-94ee-2466-ac5dfa670b7a: added peer, starting replication: peer=server-e17796a8-6257-ffef-ab4d-5b43d604d3ab
2025-04-21T21:24:23.769Z [INFO]  server-57d04b74-2b23-94ee-2466-ac5dfa670b7a: added peer, starting replication: peer=server-8b2f82e4-bdb3-3027-97c4-26f1e4a04966
2025-04-21T21:24:23.769Z [INFO]  server-57d04b74-2b23-94ee-2466-ac5dfa670b7a: pipelining replication: peer="{Voter server-8b2f82e4-bdb3-3027-97c4-26f1e4a04966 8b2f82e4-bdb3-3027-97c4-26f1e4a04966}"
2025-04-21T21:24:23.769Z [WARN]  server-8b2f82e4-bdb3-3027-97c4-26f1e4a04966: rejecting vote request since our last term is greater: candidate=57d04b74-2b23-94ee-2466-ac5dfa670b7a last-term=2 last-candidate-term=1
2025-04-21T21:24:23.769Z [INFO]  server-57d04b74-2b23-94ee-2466-ac5dfa670b7a: pipelining replication: peer="{Voter server-e17796a8-6257-ffef-ab4d-5b43d604d3ab e17796a8-6257-ffef-ab4d-5b43d604d3ab}"
2025-04-21T21:24:23.877Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:23.769655527 +0000 UTC m=+102.291314810 (2 nodes), 55.487933ms from start of poll, 55.694534ms from cluster start. Timeout at 2025-04-21 21:24:23.874658447 +0000 UTC m=+102.396317770, 105.00296ms after stability
2025-04-21T21:24:23.877Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:23.985Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:23.877357811 +0000 UTC m=+102.399017094 (1 nodes), 560ns from start of poll, 163.396818ms from cluster start. Timeout at 2025-04-21 21:24:23.982395971 +0000 UTC m=+102.504055294, 105.0382ms after stability
2025-04-21T21:24:23.985Z [WARN]  server-e17796a8-6257-ffef-ab4d-5b43d604d3ab: clearing log suffix: from=1 to=2
2025-04-21T21:24:23.985Z [WARN]  server-e17796a8-6257-ffef-ab4d-5b43d604d3ab: skipping application of old log: index=1
2025-04-21T21:24:23.985Z [WARN]  server-e17796a8-6257-ffef-ab4d-5b43d604d3ab: skipping application of old log: index=1
2025-04-21T21:24:23.985Z [INFO]  server-57d04b74-2b23-94ee-2466-ac5dfa670b7a: aborting pipeline replication: peer="{Voter server-8b2f82e4-bdb3-3027-97c4-26f1e4a04966 8b2f82e4-bdb3-3027-97c4-26f1e4a04966}"
2025-04-21T21:24:23.985Z [INFO]  server-57d04b74-2b23-94ee-2466-ac5dfa670b7a: aborting pipeline replication: peer="{Voter server-e17796a8-6257-ffef-ab4d-5b43d604d3ab e17796a8-6257-ffef-ab4d-5b43d604d3ab}"
--- PASS: TestRaft_AppendEntry (0.27s)
=== RUN   TestRaft_VotingGrant_WhenLeaderAvailable
2025-04-21T21:24:23.986Z [INFO]  server-6f517b33-8fc1-5638-5082-e579fa21e973: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f517b33-8fc1-5638-5082-e579fa21e973 Address:6f517b33-8fc1-5638-5082-e579fa21e973} {Suffrage:Voter ID:server-81088c90-c5d3-9697-46b2-373f3dfbdc8f Address:81088c90-c5d3-9697-46b2-373f3dfbdc8f} {Suffrage:Voter ID:server-f2f08dea-10d7-b555-c7e8-0d86b1d96391 Address:f2f08dea-10d7-b555-c7e8-0d86b1d96391}]"
2025-04-21T21:24:23.986Z [INFO]  server-81088c90-c5d3-9697-46b2-373f3dfbdc8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f517b33-8fc1-5638-5082-e579fa21e973 Address:6f517b33-8fc1-5638-5082-e579fa21e973} {Suffrage:Voter ID:server-81088c90-c5d3-9697-46b2-373f3dfbdc8f Address:81088c90-c5d3-9697-46b2-373f3dfbdc8f} {Suffrage:Voter ID:server-f2f08dea-10d7-b555-c7e8-0d86b1d96391 Address:f2f08dea-10d7-b555-c7e8-0d86b1d96391}]"
2025-04-21T21:24:23.986Z [INFO]  server-f2f08dea-10d7-b555-c7e8-0d86b1d96391: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6f517b33-8fc1-5638-5082-e579fa21e973 Address:6f517b33-8fc1-5638-5082-e579fa21e973} {Suffrage:Voter ID:server-81088c90-c5d3-9697-46b2-373f3dfbdc8f Address:81088c90-c5d3-9697-46b2-373f3dfbdc8f} {Suffrage:Voter ID:server-f2f08dea-10d7-b555-c7e8-0d86b1d96391 Address:f2f08dea-10d7-b555-c7e8-0d86b1d96391}]"
2025-04-21T21:24:23.986Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:23.986Z [INFO]  server-6f517b33-8fc1-5638-5082-e579fa21e973: entering follower state: follower="Node at 6f517b33-8fc1-5638-5082-e579fa21e973 [Follower]" leader-address= leader-id=
2025-04-21T21:24:23.986Z [INFO]  server-81088c90-c5d3-9697-46b2-373f3dfbdc8f: entering follower state: follower="Node at 81088c90-c5d3-9697-46b2-373f3dfbdc8f [Follower]" leader-address= leader-id=
2025-04-21T21:24:23.986Z [INFO]  server-f2f08dea-10d7-b555-c7e8-0d86b1d96391: entering follower state: follower="Node at f2f08dea-10d7-b555-c7e8-0d86b1d96391 [Follower]" leader-address= leader-id=
2025-04-21T21:24:24.041Z [WARN]  server-6f517b33-8fc1-5638-5082-e579fa21e973: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:24.041Z [INFO]  server-6f517b33-8fc1-5638-5082-e579fa21e973: entering candidate state: node="Node at 6f517b33-8fc1-5638-5082-e579fa21e973 [Candidate]" term=2
2025-04-21T21:24:24.041Z [INFO]  server-6f517b33-8fc1-5638-5082-e579fa21e973: election won: term=2 tally=2
2025-04-21T21:24:24.041Z [INFO]  server-6f517b33-8fc1-5638-5082-e579fa21e973: entering leader state: leader="Node at 6f517b33-8fc1-5638-5082-e579fa21e973 [Leader]"
2025-04-21T21:24:24.041Z [INFO]  server-6f517b33-8fc1-5638-5082-e579fa21e973: added peer, starting replication: peer=server-81088c90-c5d3-9697-46b2-373f3dfbdc8f
2025-04-21T21:24:24.041Z [INFO]  server-6f517b33-8fc1-5638-5082-e579fa21e973: added peer, starting replication: peer=server-f2f08dea-10d7-b555-c7e8-0d86b1d96391
2025-04-21T21:24:24.041Z [INFO]  server-6f517b33-8fc1-5638-5082-e579fa21e973: pipelining replication: peer="{Voter server-f2f08dea-10d7-b555-c7e8-0d86b1d96391 f2f08dea-10d7-b555-c7e8-0d86b1d96391}"
2025-04-21T21:24:24.041Z [INFO]  server-6f517b33-8fc1-5638-5082-e579fa21e973: pipelining replication: peer="{Voter server-81088c90-c5d3-9697-46b2-373f3dfbdc8f 81088c90-c5d3-9697-46b2-373f3dfbdc8f}"
2025-04-21T21:24:24.147Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:24.042055238 +0000 UTC m=+102.563714521 (2 nodes), 55.769535ms from start of poll, 55.957496ms from cluster start. Timeout at 2025-04-21 21:24:24.147065398 +0000 UTC m=+102.668724721, 105.0102ms after stability
2025-04-21T21:24:24.147Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:24.257Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:24.147450026 +0000 UTC m=+102.669109349 (1 nodes), 960ns from start of poll, 161.352324ms from cluster start. Timeout at 2025-04-21 21:24:24.252511506 +0000 UTC m=+102.774170829, 105.06148ms after stability
2025-04-21T21:24:24.257Z [WARN]  server-f2f08dea-10d7-b555-c7e8-0d86b1d96391: rejecting vote request since we have a leader: from=81088c90-c5d3-9697-46b2-373f3dfbdc8f leader=6f517b33-8fc1-5638-5082-e579fa21e973 leader-id=server-6f517b33-8fc1-5638-5082-e579fa21e973
2025-04-21T21:24:24.257Z [INFO]  server-6f517b33-8fc1-5638-5082-e579fa21e973: aborting pipeline replication: peer="{Voter server-f2f08dea-10d7-b555-c7e8-0d86b1d96391 f2f08dea-10d7-b555-c7e8-0d86b1d96391}"
2025-04-21T21:24:24.257Z [INFO]  server-6f517b33-8fc1-5638-5082-e579fa21e973: aborting pipeline replication: peer="{Voter server-81088c90-c5d3-9697-46b2-373f3dfbdc8f 81088c90-c5d3-9697-46b2-373f3dfbdc8f}"
--- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.27s)
=== RUN   TestRaft_ProtocolVersion_RejectRPC
2025-04-21T21:24:24.257Z [INFO]  server-d3247077-cd7d-9737-a13c-7b189dc8cdbe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3247077-cd7d-9737-a13c-7b189dc8cdbe Address:d3247077-cd7d-9737-a13c-7b189dc8cdbe} {Suffrage:Voter ID:server-fbf98426-2d88-4ac6-1392-88fa9430c3b5 Address:fbf98426-2d88-4ac6-1392-88fa9430c3b5} {Suffrage:Voter ID:server-4c005fe7-6d33-e1a0-0301-c529da2ee6ba Address:4c005fe7-6d33-e1a0-0301-c529da2ee6ba}]"
2025-04-21T21:24:24.257Z [INFO]  server-fbf98426-2d88-4ac6-1392-88fa9430c3b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3247077-cd7d-9737-a13c-7b189dc8cdbe Address:d3247077-cd7d-9737-a13c-7b189dc8cdbe} {Suffrage:Voter ID:server-fbf98426-2d88-4ac6-1392-88fa9430c3b5 Address:fbf98426-2d88-4ac6-1392-88fa9430c3b5} {Suffrage:Voter ID:server-4c005fe7-6d33-e1a0-0301-c529da2ee6ba Address:4c005fe7-6d33-e1a0-0301-c529da2ee6ba}]"
2025-04-21T21:24:24.257Z [INFO]  server-4c005fe7-6d33-e1a0-0301-c529da2ee6ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3247077-cd7d-9737-a13c-7b189dc8cdbe Address:d3247077-cd7d-9737-a13c-7b189dc8cdbe} {Suffrage:Voter ID:server-fbf98426-2d88-4ac6-1392-88fa9430c3b5 Address:fbf98426-2d88-4ac6-1392-88fa9430c3b5} {Suffrage:Voter ID:server-4c005fe7-6d33-e1a0-0301-c529da2ee6ba Address:4c005fe7-6d33-e1a0-0301-c529da2ee6ba}]"
2025-04-21T21:24:24.257Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:24.258Z [INFO]  server-d3247077-cd7d-9737-a13c-7b189dc8cdbe: entering follower state: follower="Node at d3247077-cd7d-9737-a13c-7b189dc8cdbe [Follower]" leader-address= leader-id=
2025-04-21T21:24:24.258Z [INFO]  server-fbf98426-2d88-4ac6-1392-88fa9430c3b5: entering follower state: follower="Node at fbf98426-2d88-4ac6-1392-88fa9430c3b5 [Follower]" leader-address= leader-id=
2025-04-21T21:24:24.258Z [INFO]  server-4c005fe7-6d33-e1a0-0301-c529da2ee6ba: entering follower state: follower="Node at 4c005fe7-6d33-e1a0-0301-c529da2ee6ba [Follower]" leader-address= leader-id=
2025-04-21T21:24:24.317Z [WARN]  server-fbf98426-2d88-4ac6-1392-88fa9430c3b5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:24.317Z [INFO]  server-fbf98426-2d88-4ac6-1392-88fa9430c3b5: entering candidate state: node="Node at fbf98426-2d88-4ac6-1392-88fa9430c3b5 [Candidate]" term=2
2025-04-21T21:24:24.317Z [INFO]  server-fbf98426-2d88-4ac6-1392-88fa9430c3b5: election won: term=2 tally=2
2025-04-21T21:24:24.317Z [INFO]  server-fbf98426-2d88-4ac6-1392-88fa9430c3b5: entering leader state: leader="Node at fbf98426-2d88-4ac6-1392-88fa9430c3b5 [Leader]"
2025-04-21T21:24:24.317Z [INFO]  server-fbf98426-2d88-4ac6-1392-88fa9430c3b5: added peer, starting replication: peer=server-d3247077-cd7d-9737-a13c-7b189dc8cdbe
2025-04-21T21:24:24.317Z [INFO]  server-fbf98426-2d88-4ac6-1392-88fa9430c3b5: added peer, starting replication: peer=server-4c005fe7-6d33-e1a0-0301-c529da2ee6ba
2025-04-21T21:24:24.317Z [INFO]  server-fbf98426-2d88-4ac6-1392-88fa9430c3b5: pipelining replication: peer="{Voter server-4c005fe7-6d33-e1a0-0301-c529da2ee6ba 4c005fe7-6d33-e1a0-0301-c529da2ee6ba}"
2025-04-21T21:24:24.317Z [INFO]  server-fbf98426-2d88-4ac6-1392-88fa9430c3b5: pipelining replication: peer="{Voter server-d3247077-cd7d-9737-a13c-7b189dc8cdbe d3247077-cd7d-9737-a13c-7b189dc8cdbe}"
2025-04-21T21:24:24.425Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:24.317500529 +0000 UTC m=+102.839159852 (2 nodes), 59.49524ms from start of poll, 59.693601ms from cluster start. Timeout at 2025-04-21 21:24:24.422503849 +0000 UTC m=+102.944163172, 105.00332ms after stability
2025-04-21T21:24:24.425Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:24.533Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:24.425714217 +0000 UTC m=+102.947373540 (1 nodes), 720ns from start of poll, 167.907289ms from cluster start. Timeout at 2025-04-21 21:24:24.530773097 +0000 UTC m=+103.052432380, 105.05884ms after stability
2025-04-21T21:24:24.533Z [INFO]  server-fbf98426-2d88-4ac6-1392-88fa9430c3b5: aborting pipeline replication: peer="{Voter server-4c005fe7-6d33-e1a0-0301-c529da2ee6ba 4c005fe7-6d33-e1a0-0301-c529da2ee6ba}"
2025-04-21T21:24:24.533Z [INFO]  server-fbf98426-2d88-4ac6-1392-88fa9430c3b5: aborting pipeline replication: peer="{Voter server-d3247077-cd7d-9737-a13c-7b189dc8cdbe d3247077-cd7d-9737-a13c-7b189dc8cdbe}"
--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.28s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_1_2
    raft_test.go:2151: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_2_3
    raft_test.go:2196: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s)
=== RUN   TestRaft_LeaderID_Propagated
2025-04-21T21:24:24.533Z [INFO]  server-618f660e-3d67-49a1-648d-3a70854854d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-618f660e-3d67-49a1-648d-3a70854854d4 Address:618f660e-3d67-49a1-648d-3a70854854d4} {Suffrage:Voter ID:server-ae321ebc-8bab-43fb-8833-ba4238415163 Address:ae321ebc-8bab-43fb-8833-ba4238415163} {Suffrage:Voter ID:server-27773517-f971-fefa-bb26-f1aeb10a62ad Address:27773517-f971-fefa-bb26-f1aeb10a62ad}]"
2025-04-21T21:24:24.533Z [INFO]  server-ae321ebc-8bab-43fb-8833-ba4238415163: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-618f660e-3d67-49a1-648d-3a70854854d4 Address:618f660e-3d67-49a1-648d-3a70854854d4} {Suffrage:Voter ID:server-ae321ebc-8bab-43fb-8833-ba4238415163 Address:ae321ebc-8bab-43fb-8833-ba4238415163} {Suffrage:Voter ID:server-27773517-f971-fefa-bb26-f1aeb10a62ad Address:27773517-f971-fefa-bb26-f1aeb10a62ad}]"
2025-04-21T21:24:24.533Z [INFO]  server-27773517-f971-fefa-bb26-f1aeb10a62ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-618f660e-3d67-49a1-648d-3a70854854d4 Address:618f660e-3d67-49a1-648d-3a70854854d4} {Suffrage:Voter ID:server-ae321ebc-8bab-43fb-8833-ba4238415163 Address:ae321ebc-8bab-43fb-8833-ba4238415163} {Suffrage:Voter ID:server-27773517-f971-fefa-bb26-f1aeb10a62ad Address:27773517-f971-fefa-bb26-f1aeb10a62ad}]"
2025-04-21T21:24:24.533Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:24.533Z [INFO]  server-618f660e-3d67-49a1-648d-3a70854854d4: entering follower state: follower="Node at 618f660e-3d67-49a1-648d-3a70854854d4 [Follower]" leader-address= leader-id=
2025-04-21T21:24:24.533Z [INFO]  server-ae321ebc-8bab-43fb-8833-ba4238415163: entering follower state: follower="Node at ae321ebc-8bab-43fb-8833-ba4238415163 [Follower]" leader-address= leader-id=
2025-04-21T21:24:24.533Z [INFO]  server-27773517-f971-fefa-bb26-f1aeb10a62ad: entering follower state: follower="Node at 27773517-f971-fefa-bb26-f1aeb10a62ad [Follower]" leader-address= leader-id=
2025-04-21T21:24:24.593Z [WARN]  server-618f660e-3d67-49a1-648d-3a70854854d4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:24.593Z [INFO]  server-618f660e-3d67-49a1-648d-3a70854854d4: entering candidate state: node="Node at 618f660e-3d67-49a1-648d-3a70854854d4 [Candidate]" term=2
2025-04-21T21:24:24.593Z [INFO]  server-618f660e-3d67-49a1-648d-3a70854854d4: election won: term=2 tally=2
2025-04-21T21:24:24.593Z [INFO]  server-618f660e-3d67-49a1-648d-3a70854854d4: entering leader state: leader="Node at 618f660e-3d67-49a1-648d-3a70854854d4 [Leader]"
2025-04-21T21:24:24.593Z [INFO]  server-618f660e-3d67-49a1-648d-3a70854854d4: added peer, starting replication: peer=server-ae321ebc-8bab-43fb-8833-ba4238415163
2025-04-21T21:24:24.593Z [INFO]  server-618f660e-3d67-49a1-648d-3a70854854d4: added peer, starting replication: peer=server-27773517-f971-fefa-bb26-f1aeb10a62ad
2025-04-21T21:24:24.593Z [INFO]  server-618f660e-3d67-49a1-648d-3a70854854d4: pipelining replication: peer="{Voter server-27773517-f971-fefa-bb26-f1aeb10a62ad 27773517-f971-fefa-bb26-f1aeb10a62ad}"
2025-04-21T21:24:24.593Z [INFO]  server-618f660e-3d67-49a1-648d-3a70854854d4: pipelining replication: peer="{Voter server-ae321ebc-8bab-43fb-8833-ba4238415163 ae321ebc-8bab-43fb-8833-ba4238415163}"
2025-04-21T21:24:24.701Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:24.594048428 +0000 UTC m=+103.115707711 (1 nodes), 60.171804ms from start of poll, 60.346886ms from cluster start. Timeout at 2025-04-21 21:24:24.699062788 +0000 UTC m=+103.220722071, 105.01436ms after stability
2025-04-21T21:24:24.701Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:24.809Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:24.701293509 +0000 UTC m=+103.222952832 (1 nodes), 400ns from start of poll, 167.592007ms from cluster start. Timeout at 2025-04-21 21:24:24.80633683 +0000 UTC m=+103.327996153, 105.043321ms after stability
2025-04-21T21:24:24.809Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:24.915Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:24.809726598 +0000 UTC m=+103.331385921 (1 nodes), 440ns from start of poll, 276.025096ms from cluster start. Timeout at 2025-04-21 21:24:24.914756079 +0000 UTC m=+103.436415362, 105.029441ms after stability
2025-04-21T21:24:24.915Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:25.021Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:24.91575319 +0000 UTC m=+103.437412473 (1 nodes), 600ns from start of poll, 382.051648ms from cluster start. Timeout at 2025-04-21 21:24:25.02080239 +0000 UTC m=+103.542461713, 105.04924ms after stability
2025-04-21T21:24:25.021Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:25.129Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:25.021507341 +0000 UTC m=+103.543166664 (1 nodes), 880ns from start of poll, 487.805839ms from cluster start. Timeout at 2025-04-21 21:24:25.126543141 +0000 UTC m=+103.648202464, 105.0358ms after stability
2025-04-21T21:24:25.129Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:25.237Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:25.129616148 +0000 UTC m=+103.651275471 (1 nodes), 520ns from start of poll, 595.914646ms from cluster start. Timeout at 2025-04-21 21:24:25.234650828 +0000 UTC m=+103.756310111, 105.03464ms after stability
2025-04-21T21:24:25.345Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:25.453Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:25.345220237 +0000 UTC m=+103.866879560 (1 nodes), 1.92µs from start of poll, 811.518735ms from cluster start. Timeout at 2025-04-21 21:24:25.450272438 +0000 UTC m=+103.971931761, 105.052201ms after stability
2025-04-21T21:24:25.453Z [INFO]  server-618f660e-3d67-49a1-648d-3a70854854d4: aborting pipeline replication: peer="{Voter server-27773517-f971-fefa-bb26-f1aeb10a62ad 27773517-f971-fefa-bb26-f1aeb10a62ad}"
2025-04-21T21:24:25.453Z [INFO]  server-618f660e-3d67-49a1-648d-3a70854854d4: aborting pipeline replication: peer="{Voter server-ae321ebc-8bab-43fb-8833-ba4238415163 ae321ebc-8bab-43fb-8833-ba4238415163}"
--- PASS: TestRaft_LeaderID_Propagated (0.92s)
=== RUN   TestRaft_LeadershipTransferInProgress
--- PASS: TestRaft_LeadershipTransferInProgress (0.00s)
=== RUN   TestRaft_LeadershipTransferPickServer
--- PASS: TestRaft_LeadershipTransferPickServer (0.00s)
=== RUN   TestRaft_LeadershipTransfer
2025-04-21T21:24:25.454Z [INFO]  server-596043f9-c134-67b8-7a52-7002a20df02d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-596043f9-c134-67b8-7a52-7002a20df02d Address:596043f9-c134-67b8-7a52-7002a20df02d} {Suffrage:Voter ID:server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b Address:4ee99f58-4cd4-a044-6b58-8b7a80f8369b} {Suffrage:Voter ID:server-b3b4661d-61bf-a1ab-ee80-e886bb458945 Address:b3b4661d-61bf-a1ab-ee80-e886bb458945}]"
2025-04-21T21:24:25.454Z [INFO]  server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-596043f9-c134-67b8-7a52-7002a20df02d Address:596043f9-c134-67b8-7a52-7002a20df02d} {Suffrage:Voter ID:server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b Address:4ee99f58-4cd4-a044-6b58-8b7a80f8369b} {Suffrage:Voter ID:server-b3b4661d-61bf-a1ab-ee80-e886bb458945 Address:b3b4661d-61bf-a1ab-ee80-e886bb458945}]"
2025-04-21T21:24:25.454Z [INFO]  server-b3b4661d-61bf-a1ab-ee80-e886bb458945: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-596043f9-c134-67b8-7a52-7002a20df02d Address:596043f9-c134-67b8-7a52-7002a20df02d} {Suffrage:Voter ID:server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b Address:4ee99f58-4cd4-a044-6b58-8b7a80f8369b} {Suffrage:Voter ID:server-b3b4661d-61bf-a1ab-ee80-e886bb458945 Address:b3b4661d-61bf-a1ab-ee80-e886bb458945}]"
2025-04-21T21:24:25.454Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:25.454Z [INFO]  server-596043f9-c134-67b8-7a52-7002a20df02d: entering follower state: follower="Node at 596043f9-c134-67b8-7a52-7002a20df02d [Follower]" leader-address= leader-id=
2025-04-21T21:24:25.454Z [INFO]  server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b: entering follower state: follower="Node at 4ee99f58-4cd4-a044-6b58-8b7a80f8369b [Follower]" leader-address= leader-id=
2025-04-21T21:24:25.454Z [INFO]  server-b3b4661d-61bf-a1ab-ee80-e886bb458945: entering follower state: follower="Node at b3b4661d-61bf-a1ab-ee80-e886bb458945 [Follower]" leader-address= leader-id=
2025-04-21T21:24:25.529Z [WARN]  server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:25.529Z [INFO]  server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b: entering candidate state: node="Node at 4ee99f58-4cd4-a044-6b58-8b7a80f8369b [Candidate]" term=2
2025-04-21T21:24:25.529Z [INFO]  server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b: election won: term=2 tally=2
2025-04-21T21:24:25.529Z [INFO]  server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b: entering leader state: leader="Node at 4ee99f58-4cd4-a044-6b58-8b7a80f8369b [Leader]"
2025-04-21T21:24:25.529Z [INFO]  server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b: added peer, starting replication: peer=server-596043f9-c134-67b8-7a52-7002a20df02d
2025-04-21T21:24:25.529Z [INFO]  server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b: added peer, starting replication: peer=server-b3b4661d-61bf-a1ab-ee80-e886bb458945
2025-04-21T21:24:25.529Z [INFO]  server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b: pipelining replication: peer="{Voter server-b3b4661d-61bf-a1ab-ee80-e886bb458945 b3b4661d-61bf-a1ab-ee80-e886bb458945}"
2025-04-21T21:24:25.529Z [INFO]  server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b: pipelining replication: peer="{Voter server-596043f9-c134-67b8-7a52-7002a20df02d 596043f9-c134-67b8-7a52-7002a20df02d}"
2025-04-21T21:24:25.637Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:25.529886358 +0000 UTC m=+104.051545641 (1 nodes), 75.604068ms from start of poll, 75.795149ms from cluster start. Timeout at 2025-04-21 21:24:25.634887638 +0000 UTC m=+104.156546921, 105.00128ms after stability
2025-04-21T21:24:25.637Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:25.745Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:25.637423521 +0000 UTC m=+104.159082844 (1 nodes), 1.44µs from start of poll, 183.332352ms from cluster start. Timeout at 2025-04-21 21:24:25.742467842 +0000 UTC m=+104.264127125, 105.044281ms after stability
2025-04-21T21:24:25.745Z [INFO]  server-596043f9-c134-67b8-7a52-7002a20df02d: entering candidate state: node="Node at 596043f9-c134-67b8-7a52-7002a20df02d [Candidate]" term=3
2025-04-21T21:24:25.745Z [INFO]  server-596043f9-c134-67b8-7a52-7002a20df02d: election won: term=3 tally=2
2025-04-21T21:24:25.745Z [INFO]  server-596043f9-c134-67b8-7a52-7002a20df02d: entering leader state: leader="Node at 596043f9-c134-67b8-7a52-7002a20df02d [Leader]"
2025-04-21T21:24:25.745Z [INFO]  server-596043f9-c134-67b8-7a52-7002a20df02d: added peer, starting replication: peer=server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b
2025-04-21T21:24:25.745Z [INFO]  server-596043f9-c134-67b8-7a52-7002a20df02d: added peer, starting replication: peer=server-b3b4661d-61bf-a1ab-ee80-e886bb458945
2025-04-21T21:24:25.745Z [INFO]  server-596043f9-c134-67b8-7a52-7002a20df02d: pipelining replication: peer="{Voter server-b3b4661d-61bf-a1ab-ee80-e886bb458945 b3b4661d-61bf-a1ab-ee80-e886bb458945}"
2025-04-21T21:24:25.745Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:25.745Z [INFO]  server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b: entering follower state: follower="Node at 4ee99f58-4cd4-a044-6b58-8b7a80f8369b [Follower]" leader-address= leader-id=
2025-04-21T21:24:25.745Z [INFO]  server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b: aborting pipeline replication: peer="{Voter server-b3b4661d-61bf-a1ab-ee80-e886bb458945 b3b4661d-61bf-a1ab-ee80-e886bb458945}"
2025-04-21T21:24:25.745Z [INFO]  server-596043f9-c134-67b8-7a52-7002a20df02d: pipelining replication: peer="{Voter server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b 4ee99f58-4cd4-a044-6b58-8b7a80f8369b}"
2025-04-21T21:24:25.745Z [INFO]  server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b: aborting pipeline replication: peer="{Voter server-596043f9-c134-67b8-7a52-7002a20df02d 596043f9-c134-67b8-7a52-7002a20df02d}"
2025-04-21T21:24:25.850Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:25.745597768 +0000 UTC m=+104.267257051 (1 nodes), 155.881µs from start of poll, 291.506559ms from cluster start. Timeout at 2025-04-21 21:24:25.850631368 +0000 UTC m=+104.372290731, 105.03368ms after stability
2025-04-21T21:24:25.850Z [INFO]  server-596043f9-c134-67b8-7a52-7002a20df02d: aborting pipeline replication: peer="{Voter server-b3b4661d-61bf-a1ab-ee80-e886bb458945 b3b4661d-61bf-a1ab-ee80-e886bb458945}"
2025-04-21T21:24:25.850Z [INFO]  server-596043f9-c134-67b8-7a52-7002a20df02d: aborting pipeline replication: peer="{Voter server-4ee99f58-4cd4-a044-6b58-8b7a80f8369b 4ee99f58-4cd4-a044-6b58-8b7a80f8369b}"
--- PASS: TestRaft_LeadershipTransfer (0.40s)
=== RUN   TestRaft_LeadershipTransferWithOneNode
2025-04-21T21:24:25.851Z [INFO]  server-52e6787b-7be4-a903-46dc-b306a1200ec6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-52e6787b-7be4-a903-46dc-b306a1200ec6 Address:52e6787b-7be4-a903-46dc-b306a1200ec6}]"
2025-04-21T21:24:25.851Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:25.851Z [INFO]  server-52e6787b-7be4-a903-46dc-b306a1200ec6: entering follower state: follower="Node at 52e6787b-7be4-a903-46dc-b306a1200ec6 [Follower]" leader-address= leader-id=
2025-04-21T21:24:25.917Z [WARN]  server-52e6787b-7be4-a903-46dc-b306a1200ec6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:25.917Z [INFO]  server-52e6787b-7be4-a903-46dc-b306a1200ec6: entering candidate state: node="Node at 52e6787b-7be4-a903-46dc-b306a1200ec6 [Candidate]" term=2
2025-04-21T21:24:25.917Z [INFO]  server-52e6787b-7be4-a903-46dc-b306a1200ec6: election won: term=2 tally=1
2025-04-21T21:24:25.917Z [INFO]  server-52e6787b-7be4-a903-46dc-b306a1200ec6: entering leader state: leader="Node at 52e6787b-7be4-a903-46dc-b306a1200ec6 [Leader]"
2025-04-21T21:24:26.038Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:25.917285442 +0000 UTC m=+104.438944765 (1 nodes), 66.170885ms from start of poll, 66.259926ms from cluster start. Timeout at 2025-04-21 21:24:26.022329122 +0000 UTC m=+104.543988445, 105.04368ms after stability
--- PASS: TestRaft_LeadershipTransferWithOneNode (0.19s)
=== RUN   TestRaft_LeadershipTransferWithSevenNodes
2025-04-21T21:24:26.039Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f Address:cdfd4b49-05d5-9623-52b2-a3991eb80c5f} {Suffrage:Voter ID:server-9018f652-6eb1-3a86-5bd6-bc3cd698f4b2 Address:9018f652-6eb1-3a86-5bd6-bc3cd698f4b2} {Suffrage:Voter ID:server-48e3616d-5aaf-6c12-5096-2ceeedab4354 Address:48e3616d-5aaf-6c12-5096-2ceeedab4354} {Suffrage:Voter ID:server-440b8e06-dc12-96db-99c7-edf4024f5760 Address:440b8e06-dc12-96db-99c7-edf4024f5760} {Suffrage:Voter ID:server-0cfad761-e310-3982-14b0-c2a6655c3f83 Address:0cfad761-e310-3982-14b0-c2a6655c3f83} {Suffrage:Voter ID:server-d2c4cad2-689c-0265-2a63-8fab854ffe36 Address:d2c4cad2-689c-0265-2a63-8fab854ffe36} {Suffrage:Voter ID:server-7c7657bb-f2b8-de13-ce53-dc800b60d786 Address:7c7657bb-f2b8-de13-ce53-dc800b60d786}]"
2025-04-21T21:24:26.039Z [INFO]  server-9018f652-6eb1-3a86-5bd6-bc3cd698f4b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f Address:cdfd4b49-05d5-9623-52b2-a3991eb80c5f} {Suffrage:Voter ID:server-9018f652-6eb1-3a86-5bd6-bc3cd698f4b2 Address:9018f652-6eb1-3a86-5bd6-bc3cd698f4b2} {Suffrage:Voter ID:server-48e3616d-5aaf-6c12-5096-2ceeedab4354 Address:48e3616d-5aaf-6c12-5096-2ceeedab4354} {Suffrage:Voter ID:server-440b8e06-dc12-96db-99c7-edf4024f5760 Address:440b8e06-dc12-96db-99c7-edf4024f5760} {Suffrage:Voter ID:server-0cfad761-e310-3982-14b0-c2a6655c3f83 Address:0cfad761-e310-3982-14b0-c2a6655c3f83} {Suffrage:Voter ID:server-d2c4cad2-689c-0265-2a63-8fab854ffe36 Address:d2c4cad2-689c-0265-2a63-8fab854ffe36} {Suffrage:Voter ID:server-7c7657bb-f2b8-de13-ce53-dc800b60d786 Address:7c7657bb-f2b8-de13-ce53-dc800b60d786}]"
2025-04-21T21:24:26.039Z [INFO]  server-48e3616d-5aaf-6c12-5096-2ceeedab4354: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f Address:cdfd4b49-05d5-9623-52b2-a3991eb80c5f} {Suffrage:Voter ID:server-9018f652-6eb1-3a86-5bd6-bc3cd698f4b2 Address:9018f652-6eb1-3a86-5bd6-bc3cd698f4b2} {Suffrage:Voter ID:server-48e3616d-5aaf-6c12-5096-2ceeedab4354 Address:48e3616d-5aaf-6c12-5096-2ceeedab4354} {Suffrage:Voter ID:server-440b8e06-dc12-96db-99c7-edf4024f5760 Address:440b8e06-dc12-96db-99c7-edf4024f5760} {Suffrage:Voter ID:server-0cfad761-e310-3982-14b0-c2a6655c3f83 Address:0cfad761-e310-3982-14b0-c2a6655c3f83} {Suffrage:Voter ID:server-d2c4cad2-689c-0265-2a63-8fab854ffe36 Address:d2c4cad2-689c-0265-2a63-8fab854ffe36} {Suffrage:Voter ID:server-7c7657bb-f2b8-de13-ce53-dc800b60d786 Address:7c7657bb-f2b8-de13-ce53-dc800b60d786}]"
2025-04-21T21:24:26.039Z [INFO]  server-440b8e06-dc12-96db-99c7-edf4024f5760: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f Address:cdfd4b49-05d5-9623-52b2-a3991eb80c5f} {Suffrage:Voter ID:server-9018f652-6eb1-3a86-5bd6-bc3cd698f4b2 Address:9018f652-6eb1-3a86-5bd6-bc3cd698f4b2} {Suffrage:Voter ID:server-48e3616d-5aaf-6c12-5096-2ceeedab4354 Address:48e3616d-5aaf-6c12-5096-2ceeedab4354} {Suffrage:Voter ID:server-440b8e06-dc12-96db-99c7-edf4024f5760 Address:440b8e06-dc12-96db-99c7-edf4024f5760} {Suffrage:Voter ID:server-0cfad761-e310-3982-14b0-c2a6655c3f83 Address:0cfad761-e310-3982-14b0-c2a6655c3f83} {Suffrage:Voter ID:server-d2c4cad2-689c-0265-2a63-8fab854ffe36 Address:d2c4cad2-689c-0265-2a63-8fab854ffe36} {Suffrage:Voter ID:server-7c7657bb-f2b8-de13-ce53-dc800b60d786 Address:7c7657bb-f2b8-de13-ce53-dc800b60d786}]"
2025-04-21T21:24:26.039Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f Address:cdfd4b49-05d5-9623-52b2-a3991eb80c5f} {Suffrage:Voter ID:server-9018f652-6eb1-3a86-5bd6-bc3cd698f4b2 Address:9018f652-6eb1-3a86-5bd6-bc3cd698f4b2} {Suffrage:Voter ID:server-48e3616d-5aaf-6c12-5096-2ceeedab4354 Address:48e3616d-5aaf-6c12-5096-2ceeedab4354} {Suffrage:Voter ID:server-440b8e06-dc12-96db-99c7-edf4024f5760 Address:440b8e06-dc12-96db-99c7-edf4024f5760} {Suffrage:Voter ID:server-0cfad761-e310-3982-14b0-c2a6655c3f83 Address:0cfad761-e310-3982-14b0-c2a6655c3f83} {Suffrage:Voter ID:server-d2c4cad2-689c-0265-2a63-8fab854ffe36 Address:d2c4cad2-689c-0265-2a63-8fab854ffe36} {Suffrage:Voter ID:server-7c7657bb-f2b8-de13-ce53-dc800b60d786 Address:7c7657bb-f2b8-de13-ce53-dc800b60d786}]"
2025-04-21T21:24:26.039Z [INFO]  server-d2c4cad2-689c-0265-2a63-8fab854ffe36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f Address:cdfd4b49-05d5-9623-52b2-a3991eb80c5f} {Suffrage:Voter ID:server-9018f652-6eb1-3a86-5bd6-bc3cd698f4b2 Address:9018f652-6eb1-3a86-5bd6-bc3cd698f4b2} {Suffrage:Voter ID:server-48e3616d-5aaf-6c12-5096-2ceeedab4354 Address:48e3616d-5aaf-6c12-5096-2ceeedab4354} {Suffrage:Voter ID:server-440b8e06-dc12-96db-99c7-edf4024f5760 Address:440b8e06-dc12-96db-99c7-edf4024f5760} {Suffrage:Voter ID:server-0cfad761-e310-3982-14b0-c2a6655c3f83 Address:0cfad761-e310-3982-14b0-c2a6655c3f83} {Suffrage:Voter ID:server-d2c4cad2-689c-0265-2a63-8fab854ffe36 Address:d2c4cad2-689c-0265-2a63-8fab854ffe36} {Suffrage:Voter ID:server-7c7657bb-f2b8-de13-ce53-dc800b60d786 Address:7c7657bb-f2b8-de13-ce53-dc800b60d786}]"
2025-04-21T21:24:26.039Z [INFO]  server-7c7657bb-f2b8-de13-ce53-dc800b60d786: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f Address:cdfd4b49-05d5-9623-52b2-a3991eb80c5f} {Suffrage:Voter ID:server-9018f652-6eb1-3a86-5bd6-bc3cd698f4b2 Address:9018f652-6eb1-3a86-5bd6-bc3cd698f4b2} {Suffrage:Voter ID:server-48e3616d-5aaf-6c12-5096-2ceeedab4354 Address:48e3616d-5aaf-6c12-5096-2ceeedab4354} {Suffrage:Voter ID:server-440b8e06-dc12-96db-99c7-edf4024f5760 Address:440b8e06-dc12-96db-99c7-edf4024f5760} {Suffrage:Voter ID:server-0cfad761-e310-3982-14b0-c2a6655c3f83 Address:0cfad761-e310-3982-14b0-c2a6655c3f83} {Suffrage:Voter ID:server-d2c4cad2-689c-0265-2a63-8fab854ffe36 Address:d2c4cad2-689c-0265-2a63-8fab854ffe36} {Suffrage:Voter ID:server-7c7657bb-f2b8-de13-ce53-dc800b60d786 Address:7c7657bb-f2b8-de13-ce53-dc800b60d786}]"
2025-04-21T21:24:26.039Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:26.039Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: entering follower state: follower="Node at cdfd4b49-05d5-9623-52b2-a3991eb80c5f [Follower]" leader-address= leader-id=
2025-04-21T21:24:26.039Z [INFO]  server-9018f652-6eb1-3a86-5bd6-bc3cd698f4b2: entering follower state: follower="Node at 9018f652-6eb1-3a86-5bd6-bc3cd698f4b2 [Follower]" leader-address= leader-id=
2025-04-21T21:24:26.039Z [INFO]  server-48e3616d-5aaf-6c12-5096-2ceeedab4354: entering follower state: follower="Node at 48e3616d-5aaf-6c12-5096-2ceeedab4354 [Follower]" leader-address= leader-id=
2025-04-21T21:24:26.039Z [INFO]  server-440b8e06-dc12-96db-99c7-edf4024f5760: entering follower state: follower="Node at 440b8e06-dc12-96db-99c7-edf4024f5760 [Follower]" leader-address= leader-id=
2025-04-21T21:24:26.039Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: entering follower state: follower="Node at 0cfad761-e310-3982-14b0-c2a6655c3f83 [Follower]" leader-address= leader-id=
2025-04-21T21:24:26.039Z [INFO]  server-d2c4cad2-689c-0265-2a63-8fab854ffe36: entering follower state: follower="Node at d2c4cad2-689c-0265-2a63-8fab854ffe36 [Follower]" leader-address= leader-id=
2025-04-21T21:24:26.039Z [INFO]  server-7c7657bb-f2b8-de13-ce53-dc800b60d786: entering follower state: follower="Node at 7c7657bb-f2b8-de13-ce53-dc800b60d786 [Follower]" leader-address= leader-id=
2025-04-21T21:24:26.109Z [WARN]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:26.109Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: entering candidate state: node="Node at 0cfad761-e310-3982-14b0-c2a6655c3f83 [Candidate]" term=2
2025-04-21T21:24:26.109Z [WARN]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:26.109Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: entering candidate state: node="Node at cdfd4b49-05d5-9623-52b2-a3991eb80c5f [Candidate]" term=2
2025-04-21T21:24:26.109Z [INFO]  server-7c7657bb-f2b8-de13-ce53-dc800b60d786: duplicate requestVote for same term: term=2
2025-04-21T21:24:26.109Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: duplicate requestVote for same term: term=2
2025-04-21T21:24:26.109Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: election won: term=2 tally=4
2025-04-21T21:24:26.109Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: entering leader state: leader="Node at 0cfad761-e310-3982-14b0-c2a6655c3f83 [Leader]"
2025-04-21T21:24:26.109Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: added peer, starting replication: peer=server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f
2025-04-21T21:24:26.109Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: added peer, starting replication: peer=server-9018f652-6eb1-3a86-5bd6-bc3cd698f4b2
2025-04-21T21:24:26.109Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: added peer, starting replication: peer=server-48e3616d-5aaf-6c12-5096-2ceeedab4354
2025-04-21T21:24:26.109Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: added peer, starting replication: peer=server-440b8e06-dc12-96db-99c7-edf4024f5760
2025-04-21T21:24:26.109Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: added peer, starting replication: peer=server-d2c4cad2-689c-0265-2a63-8fab854ffe36
2025-04-21T21:24:26.109Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: added peer, starting replication: peer=server-7c7657bb-f2b8-de13-ce53-dc800b60d786
2025-04-21T21:24:26.109Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: pipelining replication: peer="{Voter server-7c7657bb-f2b8-de13-ce53-dc800b60d786 7c7657bb-f2b8-de13-ce53-dc800b60d786}"
2025-04-21T21:24:26.110Z [INFO]  server-9018f652-6eb1-3a86-5bd6-bc3cd698f4b2: duplicate requestVote for same term: term=2
2025-04-21T21:24:26.110Z [INFO]  server-48e3616d-5aaf-6c12-5096-2ceeedab4354: duplicate requestVote for same term: term=2
2025-04-21T21:24:26.110Z [INFO]  server-440b8e06-dc12-96db-99c7-edf4024f5760: duplicate requestVote for same term: term=2
2025-04-21T21:24:26.110Z [WARN]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: rejecting vote request since we have a leader: from=cdfd4b49-05d5-9623-52b2-a3991eb80c5f leader=0cfad761-e310-3982-14b0-c2a6655c3f83 leader-id=server-0cfad761-e310-3982-14b0-c2a6655c3f83
2025-04-21T21:24:26.110Z [INFO]  server-d2c4cad2-689c-0265-2a63-8fab854ffe36: duplicate requestVote for same term: term=2
2025-04-21T21:24:26.110Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: entering follower state: follower="Node at cdfd4b49-05d5-9623-52b2-a3991eb80c5f [Follower]" leader-address=0cfad761-e310-3982-14b0-c2a6655c3f83 leader-id=server-0cfad761-e310-3982-14b0-c2a6655c3f83
2025-04-21T21:24:26.110Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: pipelining replication: peer="{Voter server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f cdfd4b49-05d5-9623-52b2-a3991eb80c5f}"
2025-04-21T21:24:26.110Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: pipelining replication: peer="{Voter server-9018f652-6eb1-3a86-5bd6-bc3cd698f4b2 9018f652-6eb1-3a86-5bd6-bc3cd698f4b2}"
2025-04-21T21:24:26.110Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: pipelining replication: peer="{Voter server-48e3616d-5aaf-6c12-5096-2ceeedab4354 48e3616d-5aaf-6c12-5096-2ceeedab4354}"
2025-04-21T21:24:26.110Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: pipelining replication: peer="{Voter server-440b8e06-dc12-96db-99c7-edf4024f5760 440b8e06-dc12-96db-99c7-edf4024f5760}"
2025-04-21T21:24:26.110Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: pipelining replication: peer="{Voter server-d2c4cad2-689c-0265-2a63-8fab854ffe36 d2c4cad2-689c-0265-2a63-8fab854ffe36}"
2025-04-21T21:24:26.217Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:26.110589821 +0000 UTC m=+104.632249144 (1 nodes), 71.227279ms from start of poll, 71.661162ms from cluster start. Timeout at 2025-04-21 21:24:26.215613741 +0000 UTC m=+104.737273064, 105.02392ms after stability
2025-04-21T21:24:26.217Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:26.325Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:26.217408219 +0000 UTC m=+104.739067542 (6 nodes), 2µs from start of poll, 178.47956ms from cluster start. Timeout at 2025-04-21 21:24:26.32247018 +0000 UTC m=+104.844129503, 105.061961ms after stability
2025-04-21T21:24:26.325Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:26.441Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:26.325646306 +0000 UTC m=+104.847305629 (1 nodes), 1.2µs from start of poll, 286.717647ms from cluster start. Timeout at 2025-04-21 21:24:26.430682786 +0000 UTC m=+104.952342109, 105.03648ms after stability
2025-04-21T21:24:26.441Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: entering candidate state: node="Node at cdfd4b49-05d5-9623-52b2-a3991eb80c5f [Candidate]" term=3
2025-04-21T21:24:26.441Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:26.441Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: entering follower state: follower="Node at 0cfad761-e310-3982-14b0-c2a6655c3f83 [Follower]" leader-address= leader-id=
2025-04-21T21:24:26.441Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: aborting pipeline replication: peer="{Voter server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f cdfd4b49-05d5-9623-52b2-a3991eb80c5f}"
2025-04-21T21:24:26.441Z [ERROR] server-0cfad761-e310-3982-14b0-c2a6655c3f83: peer has newer term, stopping replication: peer="{Voter server-7c7657bb-f2b8-de13-ce53-dc800b60d786 7c7657bb-f2b8-de13-ce53-dc800b60d786}"
2025-04-21T21:24:26.442Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: election won: term=3 tally=4
2025-04-21T21:24:26.442Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: entering leader state: leader="Node at cdfd4b49-05d5-9623-52b2-a3991eb80c5f [Leader]"
2025-04-21T21:24:26.442Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: added peer, starting replication: peer=server-9018f652-6eb1-3a86-5bd6-bc3cd698f4b2
2025-04-21T21:24:26.442Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: added peer, starting replication: peer=server-48e3616d-5aaf-6c12-5096-2ceeedab4354
2025-04-21T21:24:26.442Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: added peer, starting replication: peer=server-440b8e06-dc12-96db-99c7-edf4024f5760
2025-04-21T21:24:26.442Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: added peer, starting replication: peer=server-0cfad761-e310-3982-14b0-c2a6655c3f83
2025-04-21T21:24:26.442Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: added peer, starting replication: peer=server-d2c4cad2-689c-0265-2a63-8fab854ffe36
2025-04-21T21:24:26.442Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: added peer, starting replication: peer=server-7c7657bb-f2b8-de13-ce53-dc800b60d786
2025-04-21T21:24:26.442Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: pipelining replication: peer="{Voter server-7c7657bb-f2b8-de13-ce53-dc800b60d786 7c7657bb-f2b8-de13-ce53-dc800b60d786}"
2025-04-21T21:24:26.442Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: aborting pipeline replication: peer="{Voter server-9018f652-6eb1-3a86-5bd6-bc3cd698f4b2 9018f652-6eb1-3a86-5bd6-bc3cd698f4b2}"
2025-04-21T21:24:26.442Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: aborting pipeline replication: peer="{Voter server-48e3616d-5aaf-6c12-5096-2ceeedab4354 48e3616d-5aaf-6c12-5096-2ceeedab4354}"
2025-04-21T21:24:26.442Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: aborting pipeline replication: peer="{Voter server-440b8e06-dc12-96db-99c7-edf4024f5760 440b8e06-dc12-96db-99c7-edf4024f5760}"
2025-04-21T21:24:26.442Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: aborting pipeline replication: peer="{Voter server-d2c4cad2-689c-0265-2a63-8fab854ffe36 d2c4cad2-689c-0265-2a63-8fab854ffe36}"
2025-04-21T21:24:26.442Z [INFO]  server-0cfad761-e310-3982-14b0-c2a6655c3f83: aborting pipeline replication: peer="{Voter server-7c7657bb-f2b8-de13-ce53-dc800b60d786 7c7657bb-f2b8-de13-ce53-dc800b60d786}"
2025-04-21T21:24:26.442Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: pipelining replication: peer="{Voter server-9018f652-6eb1-3a86-5bd6-bc3cd698f4b2 9018f652-6eb1-3a86-5bd6-bc3cd698f4b2}"
2025-04-21T21:24:26.442Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: pipelining replication: peer="{Voter server-48e3616d-5aaf-6c12-5096-2ceeedab4354 48e3616d-5aaf-6c12-5096-2ceeedab4354}"
2025-04-21T21:24:26.442Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: pipelining replication: peer="{Voter server-440b8e06-dc12-96db-99c7-edf4024f5760 440b8e06-dc12-96db-99c7-edf4024f5760}"
2025-04-21T21:24:26.442Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: pipelining replication: peer="{Voter server-0cfad761-e310-3982-14b0-c2a6655c3f83 0cfad761-e310-3982-14b0-c2a6655c3f83}"
2025-04-21T21:24:26.442Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: pipelining replication: peer="{Voter server-d2c4cad2-689c-0265-2a63-8fab854ffe36 d2c4cad2-689c-0265-2a63-8fab854ffe36}"
2025-04-21T21:24:26.549Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:26.44226613 +0000 UTC m=+104.963925413 (1 nodes), 544.203µs from start of poll, 403.337431ms from cluster start. Timeout at 2025-04-21 21:24:26.54731485 +0000 UTC m=+105.068974173, 105.04876ms after stability
2025-04-21T21:24:26.549Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: aborting pipeline replication: peer="{Voter server-7c7657bb-f2b8-de13-ce53-dc800b60d786 7c7657bb-f2b8-de13-ce53-dc800b60d786}"
2025-04-21T21:24:26.549Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: aborting pipeline replication: peer="{Voter server-9018f652-6eb1-3a86-5bd6-bc3cd698f4b2 9018f652-6eb1-3a86-5bd6-bc3cd698f4b2}"
2025-04-21T21:24:26.549Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: aborting pipeline replication: peer="{Voter server-48e3616d-5aaf-6c12-5096-2ceeedab4354 48e3616d-5aaf-6c12-5096-2ceeedab4354}"
2025-04-21T21:24:26.549Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: aborting pipeline replication: peer="{Voter server-440b8e06-dc12-96db-99c7-edf4024f5760 440b8e06-dc12-96db-99c7-edf4024f5760}"
2025-04-21T21:24:26.549Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: aborting pipeline replication: peer="{Voter server-0cfad761-e310-3982-14b0-c2a6655c3f83 0cfad761-e310-3982-14b0-c2a6655c3f83}"
2025-04-21T21:24:26.549Z [INFO]  server-cdfd4b49-05d5-9623-52b2-a3991eb80c5f: aborting pipeline replication: peer="{Voter server-d2c4cad2-689c-0265-2a63-8fab854ffe36 d2c4cad2-689c-0265-2a63-8fab854ffe36}"
--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.51s)
=== RUN   TestRaft_LeadershipTransferToInvalidID
2025-04-21T21:24:26.550Z [INFO]  server-6c54e212-fa97-a373-2b0e-9a315648f560: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c54e212-fa97-a373-2b0e-9a315648f560 Address:6c54e212-fa97-a373-2b0e-9a315648f560} {Suffrage:Voter ID:server-22cc16b9-2e95-8621-f19a-6f7ed0162889 Address:22cc16b9-2e95-8621-f19a-6f7ed0162889} {Suffrage:Voter ID:server-ba647214-05bb-b000-56a5-797a1cff104a Address:ba647214-05bb-b000-56a5-797a1cff104a}]"
2025-04-21T21:24:26.550Z [INFO]  server-22cc16b9-2e95-8621-f19a-6f7ed0162889: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c54e212-fa97-a373-2b0e-9a315648f560 Address:6c54e212-fa97-a373-2b0e-9a315648f560} {Suffrage:Voter ID:server-22cc16b9-2e95-8621-f19a-6f7ed0162889 Address:22cc16b9-2e95-8621-f19a-6f7ed0162889} {Suffrage:Voter ID:server-ba647214-05bb-b000-56a5-797a1cff104a Address:ba647214-05bb-b000-56a5-797a1cff104a}]"
2025-04-21T21:24:26.550Z [INFO]  server-ba647214-05bb-b000-56a5-797a1cff104a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c54e212-fa97-a373-2b0e-9a315648f560 Address:6c54e212-fa97-a373-2b0e-9a315648f560} {Suffrage:Voter ID:server-22cc16b9-2e95-8621-f19a-6f7ed0162889 Address:22cc16b9-2e95-8621-f19a-6f7ed0162889} {Suffrage:Voter ID:server-ba647214-05bb-b000-56a5-797a1cff104a Address:ba647214-05bb-b000-56a5-797a1cff104a}]"
2025-04-21T21:24:26.550Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:26.550Z [INFO]  server-6c54e212-fa97-a373-2b0e-9a315648f560: entering follower state: follower="Node at 6c54e212-fa97-a373-2b0e-9a315648f560 [Follower]" leader-address= leader-id=
2025-04-21T21:24:26.550Z [INFO]  server-22cc16b9-2e95-8621-f19a-6f7ed0162889: entering follower state: follower="Node at 22cc16b9-2e95-8621-f19a-6f7ed0162889 [Follower]" leader-address= leader-id=
2025-04-21T21:24:26.550Z [INFO]  server-ba647214-05bb-b000-56a5-797a1cff104a: entering follower state: follower="Node at ba647214-05bb-b000-56a5-797a1cff104a [Follower]" leader-address= leader-id=
2025-04-21T21:24:26.609Z [WARN]  server-6c54e212-fa97-a373-2b0e-9a315648f560: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:26.609Z [INFO]  server-6c54e212-fa97-a373-2b0e-9a315648f560: entering candidate state: node="Node at 6c54e212-fa97-a373-2b0e-9a315648f560 [Candidate]" term=2
2025-04-21T21:24:26.609Z [INFO]  server-6c54e212-fa97-a373-2b0e-9a315648f560: election won: term=2 tally=2
2025-04-21T21:24:26.609Z [INFO]  server-6c54e212-fa97-a373-2b0e-9a315648f560: entering leader state: leader="Node at 6c54e212-fa97-a373-2b0e-9a315648f560 [Leader]"
2025-04-21T21:24:26.609Z [INFO]  server-6c54e212-fa97-a373-2b0e-9a315648f560: added peer, starting replication: peer=server-22cc16b9-2e95-8621-f19a-6f7ed0162889
2025-04-21T21:24:26.609Z [INFO]  server-6c54e212-fa97-a373-2b0e-9a315648f560: added peer, starting replication: peer=server-ba647214-05bb-b000-56a5-797a1cff104a
2025-04-21T21:24:26.610Z [INFO]  server-6c54e212-fa97-a373-2b0e-9a315648f560: pipelining replication: peer="{Voter server-ba647214-05bb-b000-56a5-797a1cff104a ba647214-05bb-b000-56a5-797a1cff104a}"
2025-04-21T21:24:26.610Z [INFO]  server-6c54e212-fa97-a373-2b0e-9a315648f560: pipelining replication: peer="{Voter server-22cc16b9-2e95-8621-f19a-6f7ed0162889 22cc16b9-2e95-8621-f19a-6f7ed0162889}"
2025-04-21T21:24:26.717Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:26.610148019 +0000 UTC m=+105.131807342 (1 nodes), 59.576601ms from start of poll, 59.772642ms from cluster start. Timeout at 2025-04-21 21:24:26.715168659 +0000 UTC m=+105.236827982, 105.02064ms after stability
2025-04-21T21:24:26.717Z [INFO]  server-6c54e212-fa97-a373-2b0e-9a315648f560: aborting pipeline replication: peer="{Voter server-ba647214-05bb-b000-56a5-797a1cff104a ba647214-05bb-b000-56a5-797a1cff104a}"
2025-04-21T21:24:26.717Z [INFO]  server-6c54e212-fa97-a373-2b0e-9a315648f560: aborting pipeline replication: peer="{Voter server-22cc16b9-2e95-8621-f19a-6f7ed0162889 22cc16b9-2e95-8621-f19a-6f7ed0162889}"
--- PASS: TestRaft_LeadershipTransferToInvalidID (0.17s)
=== RUN   TestRaft_LeadershipTransferToInvalidAddress
2025-04-21T21:24:26.718Z [INFO]  server-0b17a129-e325-4276-6620-9087e60af72a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b17a129-e325-4276-6620-9087e60af72a Address:0b17a129-e325-4276-6620-9087e60af72a} {Suffrage:Voter ID:server-6c6ec71e-4e16-1b85-299a-dd775b60baa2 Address:6c6ec71e-4e16-1b85-299a-dd775b60baa2} {Suffrage:Voter ID:server-c8336b8f-1967-8215-9240-9e4b279990a6 Address:c8336b8f-1967-8215-9240-9e4b279990a6}]"
2025-04-21T21:24:26.718Z [INFO]  server-6c6ec71e-4e16-1b85-299a-dd775b60baa2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b17a129-e325-4276-6620-9087e60af72a Address:0b17a129-e325-4276-6620-9087e60af72a} {Suffrage:Voter ID:server-6c6ec71e-4e16-1b85-299a-dd775b60baa2 Address:6c6ec71e-4e16-1b85-299a-dd775b60baa2} {Suffrage:Voter ID:server-c8336b8f-1967-8215-9240-9e4b279990a6 Address:c8336b8f-1967-8215-9240-9e4b279990a6}]"
2025-04-21T21:24:26.718Z [INFO]  server-c8336b8f-1967-8215-9240-9e4b279990a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b17a129-e325-4276-6620-9087e60af72a Address:0b17a129-e325-4276-6620-9087e60af72a} {Suffrage:Voter ID:server-6c6ec71e-4e16-1b85-299a-dd775b60baa2 Address:6c6ec71e-4e16-1b85-299a-dd775b60baa2} {Suffrage:Voter ID:server-c8336b8f-1967-8215-9240-9e4b279990a6 Address:c8336b8f-1967-8215-9240-9e4b279990a6}]"
2025-04-21T21:24:26.718Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:26.718Z [INFO]  server-0b17a129-e325-4276-6620-9087e60af72a: entering follower state: follower="Node at 0b17a129-e325-4276-6620-9087e60af72a [Follower]" leader-address= leader-id=
2025-04-21T21:24:26.718Z [INFO]  server-6c6ec71e-4e16-1b85-299a-dd775b60baa2: entering follower state: follower="Node at 6c6ec71e-4e16-1b85-299a-dd775b60baa2 [Follower]" leader-address= leader-id=
2025-04-21T21:24:26.718Z [INFO]  server-c8336b8f-1967-8215-9240-9e4b279990a6: entering follower state: follower="Node at c8336b8f-1967-8215-9240-9e4b279990a6 [Follower]" leader-address= leader-id=
2025-04-21T21:24:26.777Z [WARN]  server-6c6ec71e-4e16-1b85-299a-dd775b60baa2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:26.777Z [INFO]  server-6c6ec71e-4e16-1b85-299a-dd775b60baa2: entering candidate state: node="Node at 6c6ec71e-4e16-1b85-299a-dd775b60baa2 [Candidate]" term=2
2025-04-21T21:24:26.777Z [WARN]  server-c8336b8f-1967-8215-9240-9e4b279990a6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:26.777Z [INFO]  server-c8336b8f-1967-8215-9240-9e4b279990a6: entering candidate state: node="Node at c8336b8f-1967-8215-9240-9e4b279990a6 [Candidate]" term=2
2025-04-21T21:24:26.777Z [INFO]  server-c8336b8f-1967-8215-9240-9e4b279990a6: duplicate requestVote for same term: term=2
2025-04-21T21:24:26.777Z [INFO]  server-6c6ec71e-4e16-1b85-299a-dd775b60baa2: election won: term=2 tally=2
2025-04-21T21:24:26.777Z [INFO]  server-6c6ec71e-4e16-1b85-299a-dd775b60baa2: entering leader state: leader="Node at 6c6ec71e-4e16-1b85-299a-dd775b60baa2 [Leader]"
2025-04-21T21:24:26.777Z [INFO]  server-6c6ec71e-4e16-1b85-299a-dd775b60baa2: added peer, starting replication: peer=server-0b17a129-e325-4276-6620-9087e60af72a
2025-04-21T21:24:26.777Z [INFO]  server-6c6ec71e-4e16-1b85-299a-dd775b60baa2: added peer, starting replication: peer=server-c8336b8f-1967-8215-9240-9e4b279990a6
2025-04-21T21:24:26.777Z [INFO]  server-c8336b8f-1967-8215-9240-9e4b279990a6: entering follower state: follower="Node at c8336b8f-1967-8215-9240-9e4b279990a6 [Follower]" leader-address=6c6ec71e-4e16-1b85-299a-dd775b60baa2 leader-id=server-6c6ec71e-4e16-1b85-299a-dd775b60baa2
2025-04-21T21:24:26.777Z [INFO]  server-6c6ec71e-4e16-1b85-299a-dd775b60baa2: pipelining replication: peer="{Voter server-c8336b8f-1967-8215-9240-9e4b279990a6 c8336b8f-1967-8215-9240-9e4b279990a6}"
2025-04-21T21:24:26.777Z [INFO]  server-0b17a129-e325-4276-6620-9087e60af72a: duplicate requestVote for same term: term=2
2025-04-21T21:24:26.777Z [INFO]  server-6c6ec71e-4e16-1b85-299a-dd775b60baa2: pipelining replication: peer="{Voter server-0b17a129-e325-4276-6620-9087e60af72a 0b17a129-e325-4276-6620-9087e60af72a}"
2025-04-21T21:24:26.777Z [WARN]  server-6c6ec71e-4e16-1b85-299a-dd775b60baa2: rejecting vote request since we have a leader: from=c8336b8f-1967-8215-9240-9e4b279990a6 leader=6c6ec71e-4e16-1b85-299a-dd775b60baa2 leader-id=server-6c6ec71e-4e16-1b85-299a-dd775b60baa2
2025-04-21T21:24:26.885Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:26.777462063 +0000 UTC m=+105.299121386 (2 nodes), 59.314798ms from start of poll, 59.514ms from cluster start. Timeout at 2025-04-21 21:24:26.882464583 +0000 UTC m=+105.404123906, 105.00252ms after stability
2025-04-21T21:24:26.885Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:26.993Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:26.885190147 +0000 UTC m=+105.406849470 (1 nodes), 1.92µs from start of poll, 167.242084ms from cluster start. Timeout at 2025-04-21 21:24:26.990221627 +0000 UTC m=+105.511880910, 105.03144ms after stability
2025-04-21T21:24:26.993Z [INFO]  server-6c6ec71e-4e16-1b85-299a-dd775b60baa2: aborting pipeline replication: peer="{Voter server-c8336b8f-1967-8215-9240-9e4b279990a6 c8336b8f-1967-8215-9240-9e4b279990a6}"
2025-04-21T21:24:26.993Z [INFO]  server-6c6ec71e-4e16-1b85-299a-dd775b60baa2: aborting pipeline replication: peer="{Voter server-0b17a129-e325-4276-6620-9087e60af72a 0b17a129-e325-4276-6620-9087e60af72a}"
--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.28s)
=== RUN   TestRaft_LeadershipTransferToBehindServer
2025-04-21T21:24:26.993Z [INFO]  server-4de1445d-8b40-326a-3396-e1b23615548d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4de1445d-8b40-326a-3396-e1b23615548d Address:4de1445d-8b40-326a-3396-e1b23615548d} {Suffrage:Voter ID:server-497fda13-1905-a073-6e38-f91ad78a4b9f Address:497fda13-1905-a073-6e38-f91ad78a4b9f} {Suffrage:Voter ID:server-fd6cd4d1-3117-17ff-53c3-c80dbcee79df Address:fd6cd4d1-3117-17ff-53c3-c80dbcee79df}]"
2025-04-21T21:24:26.993Z [INFO]  server-497fda13-1905-a073-6e38-f91ad78a4b9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4de1445d-8b40-326a-3396-e1b23615548d Address:4de1445d-8b40-326a-3396-e1b23615548d} {Suffrage:Voter ID:server-497fda13-1905-a073-6e38-f91ad78a4b9f Address:497fda13-1905-a073-6e38-f91ad78a4b9f} {Suffrage:Voter ID:server-fd6cd4d1-3117-17ff-53c3-c80dbcee79df Address:fd6cd4d1-3117-17ff-53c3-c80dbcee79df}]"
2025-04-21T21:24:26.993Z [INFO]  server-fd6cd4d1-3117-17ff-53c3-c80dbcee79df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4de1445d-8b40-326a-3396-e1b23615548d Address:4de1445d-8b40-326a-3396-e1b23615548d} {Suffrage:Voter ID:server-497fda13-1905-a073-6e38-f91ad78a4b9f Address:497fda13-1905-a073-6e38-f91ad78a4b9f} {Suffrage:Voter ID:server-fd6cd4d1-3117-17ff-53c3-c80dbcee79df Address:fd6cd4d1-3117-17ff-53c3-c80dbcee79df}]"
2025-04-21T21:24:26.993Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:26.993Z [INFO]  server-4de1445d-8b40-326a-3396-e1b23615548d: entering follower state: follower="Node at 4de1445d-8b40-326a-3396-e1b23615548d [Follower]" leader-address= leader-id=
2025-04-21T21:24:26.993Z [INFO]  server-497fda13-1905-a073-6e38-f91ad78a4b9f: entering follower state: follower="Node at 497fda13-1905-a073-6e38-f91ad78a4b9f [Follower]" leader-address= leader-id=
2025-04-21T21:24:26.993Z [INFO]  server-fd6cd4d1-3117-17ff-53c3-c80dbcee79df: entering follower state: follower="Node at fd6cd4d1-3117-17ff-53c3-c80dbcee79df [Follower]" leader-address= leader-id=
2025-04-21T21:24:27.073Z [WARN]  server-497fda13-1905-a073-6e38-f91ad78a4b9f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:27.073Z [INFO]  server-497fda13-1905-a073-6e38-f91ad78a4b9f: entering candidate state: node="Node at 497fda13-1905-a073-6e38-f91ad78a4b9f [Candidate]" term=2
2025-04-21T21:24:27.073Z [INFO]  server-497fda13-1905-a073-6e38-f91ad78a4b9f: election won: term=2 tally=2
2025-04-21T21:24:27.073Z [INFO]  server-497fda13-1905-a073-6e38-f91ad78a4b9f: entering leader state: leader="Node at 497fda13-1905-a073-6e38-f91ad78a4b9f [Leader]"
2025-04-21T21:24:27.073Z [INFO]  server-497fda13-1905-a073-6e38-f91ad78a4b9f: added peer, starting replication: peer=server-4de1445d-8b40-326a-3396-e1b23615548d
2025-04-21T21:24:27.073Z [INFO]  server-497fda13-1905-a073-6e38-f91ad78a4b9f: added peer, starting replication: peer=server-fd6cd4d1-3117-17ff-53c3-c80dbcee79df
2025-04-21T21:24:27.073Z [INFO]  server-497fda13-1905-a073-6e38-f91ad78a4b9f: pipelining replication: peer="{Voter server-fd6cd4d1-3117-17ff-53c3-c80dbcee79df fd6cd4d1-3117-17ff-53c3-c80dbcee79df}"
2025-04-21T21:24:27.073Z [INFO]  server-497fda13-1905-a073-6e38-f91ad78a4b9f: pipelining replication: peer="{Voter server-4de1445d-8b40-326a-3396-e1b23615548d 4de1445d-8b40-326a-3396-e1b23615548d}"
2025-04-21T21:24:27.181Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:27.073727815 +0000 UTC m=+105.595387138 (1 nodes), 79.897977ms from start of poll, 80.072659ms from cluster start. Timeout at 2025-04-21 21:24:27.178735055 +0000 UTC m=+105.700394378, 105.00724ms after stability
2025-04-21T21:24:27.181Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:27.290Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:27.181226057 +0000 UTC m=+105.702885340 (2 nodes), 1.88µs from start of poll, 187.570861ms from cluster start. Timeout at 2025-04-21 21:24:27.286269658 +0000 UTC m=+105.807928981, 105.043641ms after stability
    raft_test.go:2411: This is not supposed to error: leadership transfer timeout
2025-04-21T21:24:27.393Z [INFO]  server-497fda13-1905-a073-6e38-f91ad78a4b9f: aborting pipeline replication: peer="{Voter server-fd6cd4d1-3117-17ff-53c3-c80dbcee79df fd6cd4d1-3117-17ff-53c3-c80dbcee79df}"
2025-04-21T21:24:27.393Z [INFO]  server-497fda13-1905-a073-6e38-f91ad78a4b9f: aborting pipeline replication: peer="{Voter server-4de1445d-8b40-326a-3396-e1b23615548d 4de1445d-8b40-326a-3396-e1b23615548d}"
--- FAIL: TestRaft_LeadershipTransferToBehindServer (0.40s)
=== RUN   TestRaft_LeadershipTransferToItself
2025-04-21T21:24:27.393Z [INFO]  server-8598f0a0-ba49-5c59-fc20-789a4f207374: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8598f0a0-ba49-5c59-fc20-789a4f207374 Address:8598f0a0-ba49-5c59-fc20-789a4f207374} {Suffrage:Voter ID:server-895b57e0-31e9-573e-c253-d00b53a8cd43 Address:895b57e0-31e9-573e-c253-d00b53a8cd43} {Suffrage:Voter ID:server-c600f9cd-9f9d-48a6-b4d9-a91a010630cb Address:c600f9cd-9f9d-48a6-b4d9-a91a010630cb}]"
2025-04-21T21:24:27.393Z [INFO]  server-895b57e0-31e9-573e-c253-d00b53a8cd43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8598f0a0-ba49-5c59-fc20-789a4f207374 Address:8598f0a0-ba49-5c59-fc20-789a4f207374} {Suffrage:Voter ID:server-895b57e0-31e9-573e-c253-d00b53a8cd43 Address:895b57e0-31e9-573e-c253-d00b53a8cd43} {Suffrage:Voter ID:server-c600f9cd-9f9d-48a6-b4d9-a91a010630cb Address:c600f9cd-9f9d-48a6-b4d9-a91a010630cb}]"
2025-04-21T21:24:27.393Z [INFO]  server-c600f9cd-9f9d-48a6-b4d9-a91a010630cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8598f0a0-ba49-5c59-fc20-789a4f207374 Address:8598f0a0-ba49-5c59-fc20-789a4f207374} {Suffrage:Voter ID:server-895b57e0-31e9-573e-c253-d00b53a8cd43 Address:895b57e0-31e9-573e-c253-d00b53a8cd43} {Suffrage:Voter ID:server-c600f9cd-9f9d-48a6-b4d9-a91a010630cb Address:c600f9cd-9f9d-48a6-b4d9-a91a010630cb}]"
2025-04-21T21:24:27.393Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:27.393Z [INFO]  server-8598f0a0-ba49-5c59-fc20-789a4f207374: entering follower state: follower="Node at 8598f0a0-ba49-5c59-fc20-789a4f207374 [Follower]" leader-address= leader-id=
2025-04-21T21:24:27.393Z [INFO]  server-895b57e0-31e9-573e-c253-d00b53a8cd43: entering follower state: follower="Node at 895b57e0-31e9-573e-c253-d00b53a8cd43 [Follower]" leader-address= leader-id=
2025-04-21T21:24:27.393Z [INFO]  server-c600f9cd-9f9d-48a6-b4d9-a91a010630cb: entering follower state: follower="Node at c600f9cd-9f9d-48a6-b4d9-a91a010630cb [Follower]" leader-address= leader-id=
2025-04-21T21:24:27.481Z [WARN]  server-8598f0a0-ba49-5c59-fc20-789a4f207374: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:27.481Z [INFO]  server-8598f0a0-ba49-5c59-fc20-789a4f207374: entering candidate state: node="Node at 8598f0a0-ba49-5c59-fc20-789a4f207374 [Candidate]" term=2
2025-04-21T21:24:27.481Z [INFO]  server-8598f0a0-ba49-5c59-fc20-789a4f207374: election won: term=2 tally=2
2025-04-21T21:24:27.481Z [INFO]  server-8598f0a0-ba49-5c59-fc20-789a4f207374: entering leader state: leader="Node at 8598f0a0-ba49-5c59-fc20-789a4f207374 [Leader]"
2025-04-21T21:24:27.481Z [INFO]  server-8598f0a0-ba49-5c59-fc20-789a4f207374: added peer, starting replication: peer=server-895b57e0-31e9-573e-c253-d00b53a8cd43
2025-04-21T21:24:27.481Z [INFO]  server-8598f0a0-ba49-5c59-fc20-789a4f207374: added peer, starting replication: peer=server-c600f9cd-9f9d-48a6-b4d9-a91a010630cb
2025-04-21T21:24:27.481Z [INFO]  server-8598f0a0-ba49-5c59-fc20-789a4f207374: pipelining replication: peer="{Voter server-c600f9cd-9f9d-48a6-b4d9-a91a010630cb c600f9cd-9f9d-48a6-b4d9-a91a010630cb}"
2025-04-21T21:24:27.481Z [INFO]  server-8598f0a0-ba49-5c59-fc20-789a4f207374: pipelining replication: peer="{Voter server-895b57e0-31e9-573e-c253-d00b53a8cd43 895b57e0-31e9-573e-c253-d00b53a8cd43}"
2025-04-21T21:24:27.589Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:27.481279354 +0000 UTC m=+106.002938677 (1 nodes), 87.553469ms from start of poll, 87.71559ms from cluster start. Timeout at 2025-04-21 21:24:27.586282634 +0000 UTC m=+106.107941917, 105.00324ms after stability
2025-04-21T21:24:27.589Z [INFO]  server-8598f0a0-ba49-5c59-fc20-789a4f207374: cannot transfer leadership to itself
2025-04-21T21:24:27.589Z [INFO]  server-8598f0a0-ba49-5c59-fc20-789a4f207374: aborting pipeline replication: peer="{Voter server-c600f9cd-9f9d-48a6-b4d9-a91a010630cb c600f9cd-9f9d-48a6-b4d9-a91a010630cb}"
2025-04-21T21:24:27.589Z [INFO]  server-8598f0a0-ba49-5c59-fc20-789a4f207374: aborting pipeline replication: peer="{Voter server-895b57e0-31e9-573e-c253-d00b53a8cd43 895b57e0-31e9-573e-c253-d00b53a8cd43}"
--- PASS: TestRaft_LeadershipTransferToItself (0.20s)
=== RUN   TestRaft_LeadershipTransferLeaderRejectsClientRequests
2025-04-21T21:24:27.590Z [INFO]  server-0e6f1bc9-2754-cf5f-a000-6776fd1a1d90: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e6f1bc9-2754-cf5f-a000-6776fd1a1d90 Address:0e6f1bc9-2754-cf5f-a000-6776fd1a1d90} {Suffrage:Voter ID:server-cdbc6c11-6503-1fd6-106d-c2cb3f56f8b7 Address:cdbc6c11-6503-1fd6-106d-c2cb3f56f8b7} {Suffrage:Voter ID:server-cf732495-6a89-1bff-4e9d-44c285834d83 Address:cf732495-6a89-1bff-4e9d-44c285834d83}]"
2025-04-21T21:24:27.590Z [INFO]  server-cdbc6c11-6503-1fd6-106d-c2cb3f56f8b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e6f1bc9-2754-cf5f-a000-6776fd1a1d90 Address:0e6f1bc9-2754-cf5f-a000-6776fd1a1d90} {Suffrage:Voter ID:server-cdbc6c11-6503-1fd6-106d-c2cb3f56f8b7 Address:cdbc6c11-6503-1fd6-106d-c2cb3f56f8b7} {Suffrage:Voter ID:server-cf732495-6a89-1bff-4e9d-44c285834d83 Address:cf732495-6a89-1bff-4e9d-44c285834d83}]"
2025-04-21T21:24:27.590Z [INFO]  server-cf732495-6a89-1bff-4e9d-44c285834d83: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e6f1bc9-2754-cf5f-a000-6776fd1a1d90 Address:0e6f1bc9-2754-cf5f-a000-6776fd1a1d90} {Suffrage:Voter ID:server-cdbc6c11-6503-1fd6-106d-c2cb3f56f8b7 Address:cdbc6c11-6503-1fd6-106d-c2cb3f56f8b7} {Suffrage:Voter ID:server-cf732495-6a89-1bff-4e9d-44c285834d83 Address:cf732495-6a89-1bff-4e9d-44c285834d83}]"
2025-04-21T21:24:27.590Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:27.590Z [INFO]  server-0e6f1bc9-2754-cf5f-a000-6776fd1a1d90: entering follower state: follower="Node at 0e6f1bc9-2754-cf5f-a000-6776fd1a1d90 [Follower]" leader-address= leader-id=
2025-04-21T21:24:27.590Z [INFO]  server-cdbc6c11-6503-1fd6-106d-c2cb3f56f8b7: entering follower state: follower="Node at cdbc6c11-6503-1fd6-106d-c2cb3f56f8b7 [Follower]" leader-address= leader-id=
2025-04-21T21:24:27.590Z [INFO]  server-cf732495-6a89-1bff-4e9d-44c285834d83: entering follower state: follower="Node at cf732495-6a89-1bff-4e9d-44c285834d83 [Follower]" leader-address= leader-id=
2025-04-21T21:24:27.649Z [WARN]  server-cf732495-6a89-1bff-4e9d-44c285834d83: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:27.649Z [INFO]  server-cf732495-6a89-1bff-4e9d-44c285834d83: entering candidate state: node="Node at cf732495-6a89-1bff-4e9d-44c285834d83 [Candidate]" term=2
2025-04-21T21:24:27.649Z [INFO]  server-cf732495-6a89-1bff-4e9d-44c285834d83: election won: term=2 tally=2
2025-04-21T21:24:27.649Z [INFO]  server-cf732495-6a89-1bff-4e9d-44c285834d83: entering leader state: leader="Node at cf732495-6a89-1bff-4e9d-44c285834d83 [Leader]"
2025-04-21T21:24:27.649Z [INFO]  server-cf732495-6a89-1bff-4e9d-44c285834d83: added peer, starting replication: peer=server-0e6f1bc9-2754-cf5f-a000-6776fd1a1d90
2025-04-21T21:24:27.649Z [INFO]  server-cf732495-6a89-1bff-4e9d-44c285834d83: added peer, starting replication: peer=server-cdbc6c11-6503-1fd6-106d-c2cb3f56f8b7
2025-04-21T21:24:27.649Z [INFO]  server-cf732495-6a89-1bff-4e9d-44c285834d83: pipelining replication: peer="{Voter server-cdbc6c11-6503-1fd6-106d-c2cb3f56f8b7 cdbc6c11-6503-1fd6-106d-c2cb3f56f8b7}"
2025-04-21T21:24:27.649Z [INFO]  server-cf732495-6a89-1bff-4e9d-44c285834d83: pipelining replication: peer="{Voter server-0e6f1bc9-2754-cf5f-a000-6776fd1a1d90 0e6f1bc9-2754-cf5f-a000-6776fd1a1d90}"
2025-04-21T21:24:27.757Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:27.649608565 +0000 UTC m=+106.171267888 (1 nodes), 59.417999ms from start of poll, 59.5928ms from cluster start. Timeout at 2025-04-21 21:24:27.754611685 +0000 UTC m=+106.276270968, 105.00308ms after stability
    raft_test.go:2457: waiting on future 0
    raft_test.go:2457: waiting on future 1
    raft_test.go:2457: waiting on future 2
    raft_test.go:2457: waiting on future 3
    raft_test.go:2457: waiting on future 4
    raft_test.go:2457: waiting on future 5
    raft_test.go:2457: waiting on future 6
2025-04-21T21:24:27.757Z [INFO]  server-cf732495-6a89-1bff-4e9d-44c285834d83: aborting pipeline replication: peer="{Voter server-cdbc6c11-6503-1fd6-106d-c2cb3f56f8b7 cdbc6c11-6503-1fd6-106d-c2cb3f56f8b7}"
2025-04-21T21:24:27.757Z [INFO]  server-cf732495-6a89-1bff-4e9d-44c285834d83: aborting pipeline replication: peer="{Voter server-0e6f1bc9-2754-cf5f-a000-6776fd1a1d90 0e6f1bc9-2754-cf5f-a000-6776fd1a1d90}"
--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.17s)
=== RUN   TestRaft_LeadershipTransferLeaderReplicationTimeout
2025-04-21T21:24:27.758Z [INFO]  server-fa20c9a6-c825-d059-1d65-05e1a996ebe8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa20c9a6-c825-d059-1d65-05e1a996ebe8 Address:fa20c9a6-c825-d059-1d65-05e1a996ebe8} {Suffrage:Voter ID:server-40f84ba9-50e1-d29f-372e-d5c6d0442b3a Address:40f84ba9-50e1-d29f-372e-d5c6d0442b3a} {Suffrage:Voter ID:server-cbd53ede-8985-17ea-3233-53a451d363b1 Address:cbd53ede-8985-17ea-3233-53a451d363b1}]"
2025-04-21T21:24:27.758Z [INFO]  server-40f84ba9-50e1-d29f-372e-d5c6d0442b3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa20c9a6-c825-d059-1d65-05e1a996ebe8 Address:fa20c9a6-c825-d059-1d65-05e1a996ebe8} {Suffrage:Voter ID:server-40f84ba9-50e1-d29f-372e-d5c6d0442b3a Address:40f84ba9-50e1-d29f-372e-d5c6d0442b3a} {Suffrage:Voter ID:server-cbd53ede-8985-17ea-3233-53a451d363b1 Address:cbd53ede-8985-17ea-3233-53a451d363b1}]"
2025-04-21T21:24:27.758Z [INFO]  server-cbd53ede-8985-17ea-3233-53a451d363b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa20c9a6-c825-d059-1d65-05e1a996ebe8 Address:fa20c9a6-c825-d059-1d65-05e1a996ebe8} {Suffrage:Voter ID:server-40f84ba9-50e1-d29f-372e-d5c6d0442b3a Address:40f84ba9-50e1-d29f-372e-d5c6d0442b3a} {Suffrage:Voter ID:server-cbd53ede-8985-17ea-3233-53a451d363b1 Address:cbd53ede-8985-17ea-3233-53a451d363b1}]"
2025-04-21T21:24:27.758Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:27.758Z [INFO]  server-fa20c9a6-c825-d059-1d65-05e1a996ebe8: entering follower state: follower="Node at fa20c9a6-c825-d059-1d65-05e1a996ebe8 [Follower]" leader-address= leader-id=
2025-04-21T21:24:27.758Z [INFO]  server-40f84ba9-50e1-d29f-372e-d5c6d0442b3a: entering follower state: follower="Node at 40f84ba9-50e1-d29f-372e-d5c6d0442b3a [Follower]" leader-address= leader-id=
2025-04-21T21:24:27.758Z [INFO]  server-cbd53ede-8985-17ea-3233-53a451d363b1: entering follower state: follower="Node at cbd53ede-8985-17ea-3233-53a451d363b1 [Follower]" leader-address= leader-id=
2025-04-21T21:24:27.825Z [WARN]  server-40f84ba9-50e1-d29f-372e-d5c6d0442b3a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:27.825Z [INFO]  server-40f84ba9-50e1-d29f-372e-d5c6d0442b3a: entering candidate state: node="Node at 40f84ba9-50e1-d29f-372e-d5c6d0442b3a [Candidate]" term=2
2025-04-21T21:24:27.825Z [INFO]  server-40f84ba9-50e1-d29f-372e-d5c6d0442b3a: election won: term=2 tally=2
2025-04-21T21:24:27.825Z [INFO]  server-40f84ba9-50e1-d29f-372e-d5c6d0442b3a: entering leader state: leader="Node at 40f84ba9-50e1-d29f-372e-d5c6d0442b3a [Leader]"
2025-04-21T21:24:27.825Z [INFO]  server-40f84ba9-50e1-d29f-372e-d5c6d0442b3a: added peer, starting replication: peer=server-fa20c9a6-c825-d059-1d65-05e1a996ebe8
2025-04-21T21:24:27.825Z [INFO]  server-40f84ba9-50e1-d29f-372e-d5c6d0442b3a: added peer, starting replication: peer=server-cbd53ede-8985-17ea-3233-53a451d363b1
2025-04-21T21:24:27.825Z [INFO]  server-40f84ba9-50e1-d29f-372e-d5c6d0442b3a: pipelining replication: peer="{Voter server-cbd53ede-8985-17ea-3233-53a451d363b1 cbd53ede-8985-17ea-3233-53a451d363b1}"
2025-04-21T21:24:27.825Z [INFO]  server-40f84ba9-50e1-d29f-372e-d5c6d0442b3a: pipelining replication: peer="{Voter server-fa20c9a6-c825-d059-1d65-05e1a996ebe8 fa20c9a6-c825-d059-1d65-05e1a996ebe8}"
2025-04-21T21:24:27.933Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:27.825535508 +0000 UTC m=+106.347194831 (1 nodes), 67.303213ms from start of poll, 67.482614ms from cluster start. Timeout at 2025-04-21 21:24:27.930551748 +0000 UTC m=+106.452211071, 105.01624ms after stability
2025-04-21T21:24:27.933Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:28.041Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:27.933185231 +0000 UTC m=+106.454844554 (2 nodes), 720ns from start of poll, 175.132337ms from cluster start. Timeout at 2025-04-21 21:24:28.038238072 +0000 UTC m=+106.559897355, 105.052801ms after stability
2025-04-21T21:24:28.598Z [INFO]  server-40f84ba9-50e1-d29f-372e-d5c6d0442b3a: aborting pipeline replication: peer="{Voter server-cbd53ede-8985-17ea-3233-53a451d363b1 cbd53ede-8985-17ea-3233-53a451d363b1}"
2025-04-21T21:24:28.601Z [INFO]  server-40f84ba9-50e1-d29f-372e-d5c6d0442b3a: aborting pipeline replication: peer="{Voter server-fa20c9a6-c825-d059-1d65-05e1a996ebe8 fa20c9a6-c825-d059-1d65-05e1a996ebe8}"
--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.86s)
=== RUN   TestRaft_LeadershipTransferIgnoresNonvoters
2025-04-21T21:24:28.613Z [INFO]  server-400ba4cd-af34-4820-1c3d-e2c626c76efc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-400ba4cd-af34-4820-1c3d-e2c626c76efc Address:400ba4cd-af34-4820-1c3d-e2c626c76efc} {Suffrage:Voter ID:server-a1b4a4ee-ca82-4b3c-d28c-bf9bd14a75ae Address:a1b4a4ee-ca82-4b3c-d28c-bf9bd14a75ae}]"
2025-04-21T21:24:28.613Z [INFO]  server-a1b4a4ee-ca82-4b3c-d28c-bf9bd14a75ae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-400ba4cd-af34-4820-1c3d-e2c626c76efc Address:400ba4cd-af34-4820-1c3d-e2c626c76efc} {Suffrage:Voter ID:server-a1b4a4ee-ca82-4b3c-d28c-bf9bd14a75ae Address:a1b4a4ee-ca82-4b3c-d28c-bf9bd14a75ae}]"
2025-04-21T21:24:28.613Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:28.613Z [INFO]  server-400ba4cd-af34-4820-1c3d-e2c626c76efc: entering follower state: follower="Node at 400ba4cd-af34-4820-1c3d-e2c626c76efc [Follower]" leader-address= leader-id=
2025-04-21T21:24:28.613Z [INFO]  server-a1b4a4ee-ca82-4b3c-d28c-bf9bd14a75ae: entering follower state: follower="Node at a1b4a4ee-ca82-4b3c-d28c-bf9bd14a75ae [Follower]" leader-address= leader-id=
2025-04-21T21:24:28.669Z [WARN]  server-400ba4cd-af34-4820-1c3d-e2c626c76efc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:28.669Z [INFO]  server-400ba4cd-af34-4820-1c3d-e2c626c76efc: entering candidate state: node="Node at 400ba4cd-af34-4820-1c3d-e2c626c76efc [Candidate]" term=2
2025-04-21T21:24:28.669Z [INFO]  server-400ba4cd-af34-4820-1c3d-e2c626c76efc: election won: term=2 tally=2
2025-04-21T21:24:28.669Z [INFO]  server-400ba4cd-af34-4820-1c3d-e2c626c76efc: entering leader state: leader="Node at 400ba4cd-af34-4820-1c3d-e2c626c76efc [Leader]"
2025-04-21T21:24:28.669Z [INFO]  server-400ba4cd-af34-4820-1c3d-e2c626c76efc: added peer, starting replication: peer=server-a1b4a4ee-ca82-4b3c-d28c-bf9bd14a75ae
2025-04-21T21:24:28.669Z [INFO]  server-400ba4cd-af34-4820-1c3d-e2c626c76efc: pipelining replication: peer="{Voter server-a1b4a4ee-ca82-4b3c-d28c-bf9bd14a75ae a1b4a4ee-ca82-4b3c-d28c-bf9bd14a75ae}"
2025-04-21T21:24:28.781Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:28.669275979 +0000 UTC m=+107.190935262 (1 nodes), 55.566854ms from start of poll, 55.701615ms from cluster start. Timeout at 2025-04-21 21:24:28.774277219 +0000 UTC m=+107.295936502, 105.00124ms after stability
2025-04-21T21:24:28.781Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:28.889Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:28.781442093 +0000 UTC m=+107.303101376 (1 nodes), 520ns from start of poll, 167.867729ms from cluster start. Timeout at 2025-04-21 21:24:28.886495373 +0000 UTC m=+107.408154696, 105.05332ms after stability
2025-04-21T21:24:28.889Z [INFO]  server-400ba4cd-af34-4820-1c3d-e2c626c76efc: updating configuration: command=DemoteVoter server-id=server-a1b4a4ee-ca82-4b3c-d28c-bf9bd14a75ae server-addr= servers="[{Suffrage:Voter ID:server-400ba4cd-af34-4820-1c3d-e2c626c76efc Address:400ba4cd-af34-4820-1c3d-e2c626c76efc} {Suffrage:Nonvoter ID:server-a1b4a4ee-ca82-4b3c-d28c-bf9bd14a75ae Address:a1b4a4ee-ca82-4b3c-d28c-bf9bd14a75ae}]"
2025-04-21T21:24:28.889Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:28.997Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:28.889327938 +0000 UTC m=+107.410987221 (1 nodes), 480ns from start of poll, 275.753574ms from cluster start. Timeout at 2025-04-21 21:24:28.994367978 +0000 UTC m=+107.516027421, 105.0402ms after stability
2025-04-21T21:24:28.997Z [INFO]  server-400ba4cd-af34-4820-1c3d-e2c626c76efc: aborting pipeline replication: peer="{Voter server-a1b4a4ee-ca82-4b3c-d28c-bf9bd14a75ae a1b4a4ee-ca82-4b3c-d28c-bf9bd14a75ae}"
--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.38s)
=== RUN   TestRaft_LeadershipTransferStopRightAway
--- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s)
=== RUN   TestRaft_GetConfigurationNoBootstrap
2025-04-21T21:24:28.997Z [INFO]  server-b429e46a-90b4-2a49-c40f-61680385f091: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b429e46a-90b4-2a49-c40f-61680385f091 Address:b429e46a-90b4-2a49-c40f-61680385f091} {Suffrage:Voter ID:server-358db70f-cdd0-7529-2320-2ba8afe9e30a Address:358db70f-cdd0-7529-2320-2ba8afe9e30a}]"
2025-04-21T21:24:28.998Z [INFO]  server-358db70f-cdd0-7529-2320-2ba8afe9e30a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b429e46a-90b4-2a49-c40f-61680385f091 Address:b429e46a-90b4-2a49-c40f-61680385f091} {Suffrage:Voter ID:server-358db70f-cdd0-7529-2320-2ba8afe9e30a Address:358db70f-cdd0-7529-2320-2ba8afe9e30a}]"
2025-04-21T21:24:28.998Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:28.998Z [INFO]  server-b429e46a-90b4-2a49-c40f-61680385f091: entering follower state: follower="Node at b429e46a-90b4-2a49-c40f-61680385f091 [Follower]" leader-address= leader-id=
2025-04-21T21:24:28.998Z [INFO]  server-358db70f-cdd0-7529-2320-2ba8afe9e30a: entering follower state: follower="Node at 358db70f-cdd0-7529-2320-2ba8afe9e30a [Follower]" leader-address= leader-id=
2025-04-21T21:24:29.085Z [WARN]  server-358db70f-cdd0-7529-2320-2ba8afe9e30a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:29.085Z [INFO]  server-358db70f-cdd0-7529-2320-2ba8afe9e30a: entering candidate state: node="Node at 358db70f-cdd0-7529-2320-2ba8afe9e30a [Candidate]" term=2
2025-04-21T21:24:29.085Z [INFO]  server-358db70f-cdd0-7529-2320-2ba8afe9e30a: election won: term=2 tally=2
2025-04-21T21:24:29.085Z [INFO]  server-358db70f-cdd0-7529-2320-2ba8afe9e30a: entering leader state: leader="Node at 358db70f-cdd0-7529-2320-2ba8afe9e30a [Leader]"
2025-04-21T21:24:29.085Z [INFO]  server-358db70f-cdd0-7529-2320-2ba8afe9e30a: added peer, starting replication: peer=server-b429e46a-90b4-2a49-c40f-61680385f091
2025-04-21T21:24:29.085Z [INFO]  server-358db70f-cdd0-7529-2320-2ba8afe9e30a: pipelining replication: peer="{Voter server-b429e46a-90b4-2a49-c40f-61680385f091 b429e46a-90b4-2a49-c40f-61680385f091}"
2025-04-21T21:24:29.193Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:29.085772138 +0000 UTC m=+107.607431421 (1 nodes), 87.72087ms from start of poll, 87.86603ms from cluster start. Timeout at 2025-04-21 21:24:29.190790498 +0000 UTC m=+107.712449781, 105.01836ms after stability
2025-04-21T21:24:29.193Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:29.301Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:29.19321182 +0000 UTC m=+107.714871143 (1 nodes), 480ns from start of poll, 195.305752ms from cluster start. Timeout at 2025-04-21 21:24:29.298277941 +0000 UTC m=+107.819937264, 105.066121ms after stability
2025-04-21T21:24:29.327Z [INFO]  server-358db70f-cdd0-7529-2320-2ba8afe9e30a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b429e46a-90b4-2a49-c40f-61680385f091 Address:b429e46a-90b4-2a49-c40f-61680385f091} {Suffrage:Voter ID:server-358db70f-cdd0-7529-2320-2ba8afe9e30a Address:358db70f-cdd0-7529-2320-2ba8afe9e30a}]"
2025-04-21T21:24:29.327Z [INFO]  server-358db70f-cdd0-7529-2320-2ba8afe9e30a: aborting pipeline replication: peer="{Voter server-b429e46a-90b4-2a49-c40f-61680385f091 b429e46a-90b4-2a49-c40f-61680385f091}"
--- PASS: TestRaft_GetConfigurationNoBootstrap (0.33s)
=== RUN   TestRaft_LogStoreIsMonotonic
2025-04-21T21:24:29.327Z [INFO]  server-6081ed69-8f3f-6af4-e435-ef6f3013ccbc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6081ed69-8f3f-6af4-e435-ef6f3013ccbc Address:6081ed69-8f3f-6af4-e435-ef6f3013ccbc}]"
2025-04-21T21:24:29.327Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:29.327Z [INFO]  server-6081ed69-8f3f-6af4-e435-ef6f3013ccbc: entering follower state: follower="Node at 6081ed69-8f3f-6af4-e435-ef6f3013ccbc [Follower]" leader-address= leader-id=
2025-04-21T21:24:29.429Z [WARN]  server-6081ed69-8f3f-6af4-e435-ef6f3013ccbc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:29.429Z [INFO]  server-6081ed69-8f3f-6af4-e435-ef6f3013ccbc: entering candidate state: node="Node at 6081ed69-8f3f-6af4-e435-ef6f3013ccbc [Candidate]" term=2
2025-04-21T21:24:29.429Z [INFO]  server-6081ed69-8f3f-6af4-e435-ef6f3013ccbc: election won: term=2 tally=1
2025-04-21T21:24:29.429Z [INFO]  server-6081ed69-8f3f-6af4-e435-ef6f3013ccbc: entering leader state: leader="Node at 6081ed69-8f3f-6af4-e435-ef6f3013ccbc [Leader]"
2025-04-21T21:24:29.537Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:29.429315327 +0000 UTC m=+107.950974650 (1 nodes), 101.634403ms from start of poll, 101.702323ms from cluster start. Timeout at 2025-04-21 21:24:29.534330647 +0000 UTC m=+108.055989970, 105.01532ms after stability
--- PASS: TestRaft_LogStoreIsMonotonic (0.21s)
=== RUN   TestRaft_CacheLogWithStoreError
2025-04-21T21:24:29.537Z [INFO]  server-1359daaa-0cd8-ba59-a28f-06bfa5d81c49: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1359daaa-0cd8-ba59-a28f-06bfa5d81c49 Address:1359daaa-0cd8-ba59-a28f-06bfa5d81c49} {Suffrage:Voter ID:server-09b813f4-e97b-445c-81a4-05c2a387d9a3 Address:09b813f4-e97b-445c-81a4-05c2a387d9a3}]"
2025-04-21T21:24:29.537Z [INFO]  server-09b813f4-e97b-445c-81a4-05c2a387d9a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1359daaa-0cd8-ba59-a28f-06bfa5d81c49 Address:1359daaa-0cd8-ba59-a28f-06bfa5d81c49} {Suffrage:Voter ID:server-09b813f4-e97b-445c-81a4-05c2a387d9a3 Address:09b813f4-e97b-445c-81a4-05c2a387d9a3}]"
2025-04-21T21:24:29.537Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:29.537Z [INFO]  server-1359daaa-0cd8-ba59-a28f-06bfa5d81c49: entering follower state: follower="Node at 1359daaa-0cd8-ba59-a28f-06bfa5d81c49 [Follower]" leader-address= leader-id=
2025-04-21T21:24:29.537Z [INFO]  server-09b813f4-e97b-445c-81a4-05c2a387d9a3: entering follower state: follower="Node at 09b813f4-e97b-445c-81a4-05c2a387d9a3 [Follower]" leader-address= leader-id=
2025-04-21T21:24:29.598Z [WARN]  server-1359daaa-0cd8-ba59-a28f-06bfa5d81c49: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:29.598Z [INFO]  server-1359daaa-0cd8-ba59-a28f-06bfa5d81c49: entering candidate state: node="Node at 1359daaa-0cd8-ba59-a28f-06bfa5d81c49 [Candidate]" term=2
2025-04-21T21:24:29.598Z [INFO]  server-1359daaa-0cd8-ba59-a28f-06bfa5d81c49: election won: term=2 tally=2
2025-04-21T21:24:29.598Z [INFO]  server-1359daaa-0cd8-ba59-a28f-06bfa5d81c49: entering leader state: leader="Node at 1359daaa-0cd8-ba59-a28f-06bfa5d81c49 [Leader]"
2025-04-21T21:24:29.598Z [INFO]  server-1359daaa-0cd8-ba59-a28f-06bfa5d81c49: added peer, starting replication: peer=server-09b813f4-e97b-445c-81a4-05c2a387d9a3
2025-04-21T21:24:29.598Z [INFO]  server-1359daaa-0cd8-ba59-a28f-06bfa5d81c49: pipelining replication: peer="{Voter server-09b813f4-e97b-445c-81a4-05c2a387d9a3 09b813f4-e97b-445c-81a4-05c2a387d9a3}"
2025-04-21T21:24:29.705Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:29.599038428 +0000 UTC m=+108.120697751 (1 nodes), 61.299012ms from start of poll, 61.426213ms from cluster start. Timeout at 2025-04-21 21:24:29.704061349 +0000 UTC m=+108.225720672, 105.022921ms after stability
2025-04-21T21:24:29.705Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:29.813Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:29.705308942 +0000 UTC m=+108.226968265 (1 nodes), 1.36µs from start of poll, 167.696727ms from cluster start. Timeout at 2025-04-21 21:24:29.810349782 +0000 UTC m=+108.332009105, 105.04084ms after stability
2025-04-21T21:24:29.814Z [ERROR] server-09b813f4-e97b-445c-81a4-05c2a387d9a3: failed to append to logs: error="unable to store logs within log store, err: "some error""
2025-04-21T21:24:29.814Z [WARN]  server-09b813f4-e97b-445c-81a4-05c2a387d9a3: failed to get previous log: previous-index=9 last-index=7 error="log not found"
2025-04-21T21:24:29.814Z [INFO]  server-1359daaa-0cd8-ba59-a28f-06bfa5d81c49: aborting pipeline replication: peer="{Voter server-09b813f4-e97b-445c-81a4-05c2a387d9a3 09b813f4-e97b-445c-81a4-05c2a387d9a3}"
2025-04-21T21:24:29.825Z [INFO]  server-1359daaa-0cd8-ba59-a28f-06bfa5d81c49: pipelining replication: peer="{Voter server-09b813f4-e97b-445c-81a4-05c2a387d9a3 09b813f4-e97b-445c-81a4-05c2a387d9a3}"
2025-04-21T21:24:29.825Z [INFO]  server-09b813f4-e97b-445c-81a4-05c2a387d9a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1359daaa-0cd8-ba59-a28f-06bfa5d81c49 Address:1359daaa-0cd8-ba59-a28f-06bfa5d81c49} {Suffrage:Voter ID:server-09b813f4-e97b-445c-81a4-05c2a387d9a3 Address:09b813f4-e97b-445c-81a4-05c2a387d9a3}]"
2025-04-21T21:24:29.825Z [INFO]  server-1359daaa-0cd8-ba59-a28f-06bfa5d81c49: aborting pipeline replication: peer="{Voter server-09b813f4-e97b-445c-81a4-05c2a387d9a3 09b813f4-e97b-445c-81a4-05c2a387d9a3}"
--- PASS: TestRaft_CacheLogWithStoreError (0.29s)
=== RUN   TestRaft_ReloadConfig
2025-04-21T21:24:29.826Z [INFO]  server-37ed82c3-882e-f626-277f-eecbf512d6e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-37ed82c3-882e-f626-277f-eecbf512d6e6 Address:37ed82c3-882e-f626-277f-eecbf512d6e6}]"
--- PASS: TestRaft_ReloadConfig (0.00s)
=== RUN   TestRaft_ReloadConfigValidates
2025-04-21T21:24:29.826Z [INFO]  server-e5046797-bd07-4df2-cce7-526664acd51d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e5046797-bd07-4df2-cce7-526664acd51d Address:e5046797-bd07-4df2-cce7-526664acd51d}]"
--- PASS: TestRaft_ReloadConfigValidates (0.02s)
=== RUN   TestRaft_InstallSnapshot_InvalidPeers
2025-04-21T21:24:29.841Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)"
--- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s)
=== RUN   TestRaft_VoteNotGranted_WhenNodeNotInCluster
2025-04-21T21:24:29.841Z [INFO]  server-d1f2313a-2c38-2db4-83ab-e1eb768b85b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d1f2313a-2c38-2db4-83ab-e1eb768b85b3 Address:d1f2313a-2c38-2db4-83ab-e1eb768b85b3} {Suffrage:Voter ID:server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4 Address:fe19d8c4-bec8-5178-6d3b-da78f56eb5c4} {Suffrage:Voter ID:server-3100d411-b5ae-6270-3cd6-679aeff45c1b Address:3100d411-b5ae-6270-3cd6-679aeff45c1b}]"
2025-04-21T21:24:29.841Z [INFO]  server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d1f2313a-2c38-2db4-83ab-e1eb768b85b3 Address:d1f2313a-2c38-2db4-83ab-e1eb768b85b3} {Suffrage:Voter ID:server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4 Address:fe19d8c4-bec8-5178-6d3b-da78f56eb5c4} {Suffrage:Voter ID:server-3100d411-b5ae-6270-3cd6-679aeff45c1b Address:3100d411-b5ae-6270-3cd6-679aeff45c1b}]"
2025-04-21T21:24:29.842Z [INFO]  server-3100d411-b5ae-6270-3cd6-679aeff45c1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d1f2313a-2c38-2db4-83ab-e1eb768b85b3 Address:d1f2313a-2c38-2db4-83ab-e1eb768b85b3} {Suffrage:Voter ID:server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4 Address:fe19d8c4-bec8-5178-6d3b-da78f56eb5c4} {Suffrage:Voter ID:server-3100d411-b5ae-6270-3cd6-679aeff45c1b Address:3100d411-b5ae-6270-3cd6-679aeff45c1b}]"
2025-04-21T21:24:29.842Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:29.842Z [INFO]  server-d1f2313a-2c38-2db4-83ab-e1eb768b85b3: entering follower state: follower="Node at d1f2313a-2c38-2db4-83ab-e1eb768b85b3 [Follower]" leader-address= leader-id=
2025-04-21T21:24:29.842Z [INFO]  server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: entering follower state: follower="Node at fe19d8c4-bec8-5178-6d3b-da78f56eb5c4 [Follower]" leader-address= leader-id=
2025-04-21T21:24:29.842Z [INFO]  server-3100d411-b5ae-6270-3cd6-679aeff45c1b: entering follower state: follower="Node at 3100d411-b5ae-6270-3cd6-679aeff45c1b [Follower]" leader-address= leader-id=
2025-04-21T21:24:29.897Z [WARN]  server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:29.897Z [INFO]  server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: entering candidate state: node="Node at fe19d8c4-bec8-5178-6d3b-da78f56eb5c4 [Candidate]" term=2
2025-04-21T21:24:29.897Z [INFO]  server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: election won: term=2 tally=2
2025-04-21T21:24:29.897Z [INFO]  server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: entering leader state: leader="Node at fe19d8c4-bec8-5178-6d3b-da78f56eb5c4 [Leader]"
2025-04-21T21:24:29.897Z [INFO]  server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: added peer, starting replication: peer=server-d1f2313a-2c38-2db4-83ab-e1eb768b85b3
2025-04-21T21:24:29.897Z [INFO]  server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: added peer, starting replication: peer=server-3100d411-b5ae-6270-3cd6-679aeff45c1b
2025-04-21T21:24:29.897Z [INFO]  server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: pipelining replication: peer="{Voter server-3100d411-b5ae-6270-3cd6-679aeff45c1b 3100d411-b5ae-6270-3cd6-679aeff45c1b}"
2025-04-21T21:24:29.897Z [INFO]  server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: pipelining replication: peer="{Voter server-d1f2313a-2c38-2db4-83ab-e1eb768b85b3 d1f2313a-2c38-2db4-83ab-e1eb768b85b3}"
2025-04-21T21:24:30.005Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:29.897931437 +0000 UTC m=+108.419590760 (1 nodes), 55.906016ms from start of poll, 56.126737ms from cluster start. Timeout at 2025-04-21 21:24:30.002944957 +0000 UTC m=+108.524604280, 105.01352ms after stability
2025-04-21T21:24:30.013Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:30.122Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:30.013194132 +0000 UTC m=+108.534853415 (2 nodes), 880ns from start of poll, 171.389392ms from cluster start. Timeout at 2025-04-21 21:24:30.118236012 +0000 UTC m=+108.639895335, 105.04192ms after stability
2025-04-21T21:24:30.122Z [INFO]  server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: updating configuration: command=RemoveServer server-id=server-d1f2313a-2c38-2db4-83ab-e1eb768b85b3 server-addr= servers="[{Suffrage:Voter ID:server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4 Address:fe19d8c4-bec8-5178-6d3b-da78f56eb5c4} {Suffrage:Voter ID:server-3100d411-b5ae-6270-3cd6-679aeff45c1b Address:3100d411-b5ae-6270-3cd6-679aeff45c1b}]"
2025-04-21T21:24:30.122Z [INFO]  server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: removed peer, stopping replication: peer=server-d1f2313a-2c38-2db4-83ab-e1eb768b85b3 last-index=3
2025-04-21T21:24:30.122Z [INFO]  server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: aborting pipeline replication: peer="{Voter server-d1f2313a-2c38-2db4-83ab-e1eb768b85b3 d1f2313a-2c38-2db4-83ab-e1eb768b85b3}"
2025-04-21T21:24:30.193Z [WARN]  server-d1f2313a-2c38-2db4-83ab-e1eb768b85b3: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-04-21T21:24:30.233Z [ERROR] server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: failed to pipeline appendEntries: peer="{Voter server-3100d411-b5ae-6270-3cd6-679aeff45c1b 3100d411-b5ae-6270-3cd6-679aeff45c1b}" error="append pipeline closed"
2025-04-21T21:24:30.233Z [INFO]  server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: aborting pipeline replication: peer="{Voter server-3100d411-b5ae-6270-3cd6-679aeff45c1b 3100d411-b5ae-6270-3cd6-679aeff45c1b}"
2025-04-21T21:24:30.241Z [ERROR] server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: failed to appendEntries to: peer="{Voter server-3100d411-b5ae-6270-3cd6-679aeff45c1b 3100d411-b5ae-6270-3cd6-679aeff45c1b}" error="failed to connect to peer: 3100d411-b5ae-6270-3cd6-679aeff45c1b"
2025-04-21T21:24:30.241Z [ERROR] server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: failed to heartbeat to: peer=3100d411-b5ae-6270-3cd6-679aeff45c1b backoff time=10ms error="failed to connect to peer: 3100d411-b5ae-6270-3cd6-679aeff45c1b"
2025-04-21T21:24:30.261Z [ERROR] server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: failed to appendEntries to: peer="{Voter server-3100d411-b5ae-6270-3cd6-679aeff45c1b 3100d411-b5ae-6270-3cd6-679aeff45c1b}" error="failed to connect to peer: 3100d411-b5ae-6270-3cd6-679aeff45c1b"
2025-04-21T21:24:30.261Z [ERROR] server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: failed to heartbeat to: peer=3100d411-b5ae-6270-3cd6-679aeff45c1b backoff time=10ms error="failed to connect to peer: 3100d411-b5ae-6270-3cd6-679aeff45c1b"
2025-04-21T21:24:30.281Z [WARN]  server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: failed to contact: server-id=server-3100d411-b5ae-6270-3cd6-679aeff45c1b time=51.824428ms
2025-04-21T21:24:30.281Z [WARN]  server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: failed to contact quorum of nodes, stepping down
2025-04-21T21:24:30.281Z [INFO]  server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: entering follower state: follower="Node at fe19d8c4-bec8-5178-6d3b-da78f56eb5c4 [Follower]" leader-address= leader-id=
2025-04-21T21:24:30.281Z [ERROR] server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4: failed to appendEntries to: peer="{Voter server-3100d411-b5ae-6270-3cd6-679aeff45c1b 3100d411-b5ae-6270-3cd6-679aeff45c1b}" error="failed to connect to peer: 3100d411-b5ae-6270-3cd6-679aeff45c1b"
2025-04-21T21:24:30.333Z [WARN]  server-3100d411-b5ae-6270-3cd6-679aeff45c1b: heartbeat timeout reached, starting election: last-leader-addr=fe19d8c4-bec8-5178-6d3b-da78f56eb5c4 last-leader-id=server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4
2025-04-21T21:24:30.333Z [INFO]  server-3100d411-b5ae-6270-3cd6-679aeff45c1b: entering candidate state: node="Node at 3100d411-b5ae-6270-3cd6-679aeff45c1b [Candidate]" term=3
2025-04-21T21:24:30.333Z [ERROR] server-3100d411-b5ae-6270-3cd6-679aeff45c1b: failed to make requestVote RPC: target="{Voter server-fe19d8c4-bec8-5178-6d3b-da78f56eb5c4 fe19d8c4-bec8-5178-6d3b-da78f56eb5c4}" error="failed to connect to peer: fe19d8c4-bec8-5178-6d3b-da78f56eb5c4" term=3
2025-04-21T21:24:30.337Z [WARN]  server-3100d411-b5ae-6270-3cd6-679aeff45c1b: rejecting vote request since node is not in configuration: from=d1f2313a-2c38-2db4-83ab-e1eb768b85b3
--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.50s)
=== RUN   TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin
2025-04-21T21:24:30.337Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5eb78743-3dfc-1586-095e-59c86bb58f37 Address:5eb78743-3dfc-1586-095e-59c86bb58f37} {Suffrage:Voter ID:server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 Address:d3cc73e1-e778-d00d-7652-d8a9412ecb69} {Suffrage:Voter ID:server-a0f798de-4563-4b6c-04ef-15ee21499001 Address:a0f798de-4563-4b6c-04ef-15ee21499001}]"
2025-04-21T21:24:30.338Z [INFO]  server-d3cc73e1-e778-d00d-7652-d8a9412ecb69: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5eb78743-3dfc-1586-095e-59c86bb58f37 Address:5eb78743-3dfc-1586-095e-59c86bb58f37} {Suffrage:Voter ID:server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 Address:d3cc73e1-e778-d00d-7652-d8a9412ecb69} {Suffrage:Voter ID:server-a0f798de-4563-4b6c-04ef-15ee21499001 Address:a0f798de-4563-4b6c-04ef-15ee21499001}]"
2025-04-21T21:24:30.338Z [INFO]  server-a0f798de-4563-4b6c-04ef-15ee21499001: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5eb78743-3dfc-1586-095e-59c86bb58f37 Address:5eb78743-3dfc-1586-095e-59c86bb58f37} {Suffrage:Voter ID:server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 Address:d3cc73e1-e778-d00d-7652-d8a9412ecb69} {Suffrage:Voter ID:server-a0f798de-4563-4b6c-04ef-15ee21499001 Address:a0f798de-4563-4b6c-04ef-15ee21499001}]"
2025-04-21T21:24:30.338Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:30.338Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: entering follower state: follower="Node at 5eb78743-3dfc-1586-095e-59c86bb58f37 [Follower]" leader-address= leader-id=
2025-04-21T21:24:30.338Z [INFO]  server-d3cc73e1-e778-d00d-7652-d8a9412ecb69: entering follower state: follower="Node at d3cc73e1-e778-d00d-7652-d8a9412ecb69 [Follower]" leader-address= leader-id=
2025-04-21T21:24:30.338Z [INFO]  server-a0f798de-4563-4b6c-04ef-15ee21499001: entering follower state: follower="Node at a0f798de-4563-4b6c-04ef-15ee21499001 [Follower]" leader-address= leader-id=
2025-04-21T21:24:30.405Z [WARN]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:30.405Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: entering candidate state: node="Node at 5eb78743-3dfc-1586-095e-59c86bb58f37 [Candidate]" term=2
2025-04-21T21:24:30.405Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: election won: term=2 tally=2
2025-04-21T21:24:30.405Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: entering leader state: leader="Node at 5eb78743-3dfc-1586-095e-59c86bb58f37 [Leader]"
2025-04-21T21:24:30.405Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: added peer, starting replication: peer=server-d3cc73e1-e778-d00d-7652-d8a9412ecb69
2025-04-21T21:24:30.405Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: added peer, starting replication: peer=server-a0f798de-4563-4b6c-04ef-15ee21499001
2025-04-21T21:24:30.405Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: pipelining replication: peer="{Voter server-a0f798de-4563-4b6c-04ef-15ee21499001 a0f798de-4563-4b6c-04ef-15ee21499001}"
2025-04-21T21:24:30.405Z [WARN]  server-d3cc73e1-e778-d00d-7652-d8a9412ecb69: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:30.405Z [INFO]  server-d3cc73e1-e778-d00d-7652-d8a9412ecb69: entering candidate state: node="Node at d3cc73e1-e778-d00d-7652-d8a9412ecb69 [Candidate]" term=2
2025-04-21T21:24:30.405Z [INFO]  server-d3cc73e1-e778-d00d-7652-d8a9412ecb69: duplicate requestVote for same term: term=2
2025-04-21T21:24:30.405Z [INFO]  server-d3cc73e1-e778-d00d-7652-d8a9412ecb69: entering follower state: follower="Node at d3cc73e1-e778-d00d-7652-d8a9412ecb69 [Follower]" leader-address=5eb78743-3dfc-1586-095e-59c86bb58f37 leader-id=server-5eb78743-3dfc-1586-095e-59c86bb58f37
2025-04-21T21:24:30.405Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: pipelining replication: peer="{Voter server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 d3cc73e1-e778-d00d-7652-d8a9412ecb69}"
2025-04-21T21:24:30.406Z [WARN]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: rejecting vote request since we have a leader: from=d3cc73e1-e778-d00d-7652-d8a9412ecb69 leader=5eb78743-3dfc-1586-095e-59c86bb58f37 leader-id=server-5eb78743-3dfc-1586-095e-59c86bb58f37
2025-04-21T21:24:30.406Z [WARN]  server-a0f798de-4563-4b6c-04ef-15ee21499001: rejecting vote request since we have a leader: from=d3cc73e1-e778-d00d-7652-d8a9412ecb69 leader=5eb78743-3dfc-1586-095e-59c86bb58f37 leader-id=server-5eb78743-3dfc-1586-095e-59c86bb58f37
2025-04-21T21:24:30.513Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:30.406056292 +0000 UTC m=+108.927715615 (1 nodes), 67.983577ms from start of poll, 68.193058ms from cluster start. Timeout at 2025-04-21 21:24:30.511065772 +0000 UTC m=+109.032725055, 105.00944ms after stability
2025-04-21T21:24:30.521Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:30.629Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:30.521167306 +0000 UTC m=+109.042826629 (2 nodes), 2.48µs from start of poll, 183.304072ms from cluster start. Timeout at 2025-04-21 21:24:30.626183946 +0000 UTC m=+109.147843269, 105.01664ms after stability
2025-04-21T21:24:30.633Z [ERROR] server-5eb78743-3dfc-1586-095e-59c86bb58f37: failed to heartbeat to: peer=d3cc73e1-e778-d00d-7652-d8a9412ecb69 backoff time=10ms error="failed to connect to peer: d3cc73e1-e778-d00d-7652-d8a9412ecb69"
2025-04-21T21:24:30.633Z [ERROR] server-5eb78743-3dfc-1586-095e-59c86bb58f37: failed to pipeline appendEntries: peer="{Voter server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 d3cc73e1-e778-d00d-7652-d8a9412ecb69}" error="append pipeline closed"
2025-04-21T21:24:30.633Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: aborting pipeline replication: peer="{Voter server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 d3cc73e1-e778-d00d-7652-d8a9412ecb69}"
2025-04-21T21:24:30.645Z [ERROR] server-5eb78743-3dfc-1586-095e-59c86bb58f37: failed to appendEntries to: peer="{Voter server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 d3cc73e1-e778-d00d-7652-d8a9412ecb69}" error="failed to connect to peer: d3cc73e1-e778-d00d-7652-d8a9412ecb69"
2025-04-21T21:24:30.652Z [ERROR] server-5eb78743-3dfc-1586-095e-59c86bb58f37: failed to heartbeat to: peer=d3cc73e1-e778-d00d-7652-d8a9412ecb69 backoff time=10ms error="failed to connect to peer: d3cc73e1-e778-d00d-7652-d8a9412ecb69"
2025-04-21T21:24:30.673Z [ERROR] server-5eb78743-3dfc-1586-095e-59c86bb58f37: failed to appendEntries to: peer="{Voter server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 d3cc73e1-e778-d00d-7652-d8a9412ecb69}" error="failed to connect to peer: d3cc73e1-e778-d00d-7652-d8a9412ecb69"
2025-04-21T21:24:30.673Z [ERROR] server-5eb78743-3dfc-1586-095e-59c86bb58f37: failed to heartbeat to: peer=d3cc73e1-e778-d00d-7652-d8a9412ecb69 backoff time=10ms error="failed to connect to peer: d3cc73e1-e778-d00d-7652-d8a9412ecb69"
2025-04-21T21:24:30.689Z [WARN]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: failed to contact: server-id=server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 time=64.067031ms
2025-04-21T21:24:30.693Z [ERROR] server-5eb78743-3dfc-1586-095e-59c86bb58f37: failed to heartbeat to: peer=d3cc73e1-e778-d00d-7652-d8a9412ecb69 backoff time=20ms error="failed to connect to peer: d3cc73e1-e778-d00d-7652-d8a9412ecb69"
2025-04-21T21:24:30.697Z [ERROR] server-5eb78743-3dfc-1586-095e-59c86bb58f37: failed to appendEntries to: peer="{Voter server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 d3cc73e1-e778-d00d-7652-d8a9412ecb69}" error="failed to connect to peer: d3cc73e1-e778-d00d-7652-d8a9412ecb69"
2025-04-21T21:24:30.721Z [ERROR] server-5eb78743-3dfc-1586-095e-59c86bb58f37: failed to heartbeat to: peer=d3cc73e1-e778-d00d-7652-d8a9412ecb69 backoff time=25ms error="failed to connect to peer: d3cc73e1-e778-d00d-7652-d8a9412ecb69"
2025-04-21T21:24:30.721Z [WARN]  server-d3cc73e1-e778-d00d-7652-d8a9412ecb69: heartbeat timeout reached, starting election: last-leader-addr=5eb78743-3dfc-1586-095e-59c86bb58f37 last-leader-id=server-5eb78743-3dfc-1586-095e-59c86bb58f37
2025-04-21T21:24:30.721Z [INFO]  server-d3cc73e1-e778-d00d-7652-d8a9412ecb69: entering candidate state: node="Node at d3cc73e1-e778-d00d-7652-d8a9412ecb69 [Candidate]" term=3
2025-04-21T21:24:30.721Z [ERROR] server-d3cc73e1-e778-d00d-7652-d8a9412ecb69: failed to make requestVote RPC: target="{Voter server-a0f798de-4563-4b6c-04ef-15ee21499001 a0f798de-4563-4b6c-04ef-15ee21499001}" error="failed to connect to peer: a0f798de-4563-4b6c-04ef-15ee21499001" term=3
2025-04-21T21:24:30.721Z [ERROR] server-d3cc73e1-e778-d00d-7652-d8a9412ecb69: failed to make requestVote RPC: target="{Voter server-5eb78743-3dfc-1586-095e-59c86bb58f37 5eb78743-3dfc-1586-095e-59c86bb58f37}" error="failed to connect to peer: 5eb78743-3dfc-1586-095e-59c86bb58f37" term=3
2025-04-21T21:24:30.725Z [ERROR] server-5eb78743-3dfc-1586-095e-59c86bb58f37: failed to appendEntries to: peer="{Voter server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 d3cc73e1-e778-d00d-7652-d8a9412ecb69}" error="failed to connect to peer: d3cc73e1-e778-d00d-7652-d8a9412ecb69"
2025-04-21T21:24:30.737Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: updating configuration: command=RemoveServer server-id=server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 server-addr= servers="[{Suffrage:Voter ID:server-5eb78743-3dfc-1586-095e-59c86bb58f37 Address:5eb78743-3dfc-1586-095e-59c86bb58f37} {Suffrage:Voter ID:server-a0f798de-4563-4b6c-04ef-15ee21499001 Address:a0f798de-4563-4b6c-04ef-15ee21499001}]"
2025-04-21T21:24:30.737Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: removed peer, stopping replication: peer=server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 last-index=3
2025-04-21T21:24:30.737Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: updating configuration: command=AddNonvoter server-id=server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 server-addr=d3cc73e1-e778-d00d-7652-d8a9412ecb69 servers="[{Suffrage:Voter ID:server-5eb78743-3dfc-1586-095e-59c86bb58f37 Address:5eb78743-3dfc-1586-095e-59c86bb58f37} {Suffrage:Voter ID:server-a0f798de-4563-4b6c-04ef-15ee21499001 Address:a0f798de-4563-4b6c-04ef-15ee21499001} {Suffrage:Nonvoter ID:server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 Address:d3cc73e1-e778-d00d-7652-d8a9412ecb69}]"
2025-04-21T21:24:30.737Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: added peer, starting replication: peer=server-d3cc73e1-e778-d00d-7652-d8a9412ecb69
2025-04-21T21:24:30.737Z [ERROR] server-5eb78743-3dfc-1586-095e-59c86bb58f37: failed to appendEntries to: peer="{Nonvoter server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 d3cc73e1-e778-d00d-7652-d8a9412ecb69}" error="failed to connect to peer: d3cc73e1-e778-d00d-7652-d8a9412ecb69"
2025-04-21T21:24:30.761Z [ERROR] server-5eb78743-3dfc-1586-095e-59c86bb58f37: peer has newer term, stopping replication: peer="{Nonvoter server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 d3cc73e1-e778-d00d-7652-d8a9412ecb69}"
2025-04-21T21:24:30.761Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: entering follower state: follower="Node at 5eb78743-3dfc-1586-095e-59c86bb58f37 [Follower]" leader-address= leader-id=
2025-04-21T21:24:30.761Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: aborting pipeline replication: peer="{Voter server-a0f798de-4563-4b6c-04ef-15ee21499001 a0f798de-4563-4b6c-04ef-15ee21499001}"
2025-04-21T21:24:30.777Z [ERROR] server-5eb78743-3dfc-1586-095e-59c86bb58f37: peer has newer term, stopping replication: peer="{Voter server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 d3cc73e1-e778-d00d-7652-d8a9412ecb69}"
2025-04-21T21:24:30.801Z [WARN]  server-d3cc73e1-e778-d00d-7652-d8a9412ecb69: Election timeout reached, restarting election
2025-04-21T21:24:30.801Z [INFO]  server-d3cc73e1-e778-d00d-7652-d8a9412ecb69: entering candidate state: node="Node at d3cc73e1-e778-d00d-7652-d8a9412ecb69 [Candidate]" term=23
2025-04-21T21:24:30.801Z [WARN]  server-a0f798de-4563-4b6c-04ef-15ee21499001: rejecting vote request since we have a leader: from=d3cc73e1-e778-d00d-7652-d8a9412ecb69 leader=5eb78743-3dfc-1586-095e-59c86bb58f37 leader-id=server-5eb78743-3dfc-1586-095e-59c86bb58f37
2025-04-21T21:24:30.801Z [WARN]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: rejecting vote request since node is not a voter: from=d3cc73e1-e778-d00d-7652-d8a9412ecb69
2025-04-21T21:24:30.833Z [WARN]  server-a0f798de-4563-4b6c-04ef-15ee21499001: heartbeat timeout reached, starting election: last-leader-addr=5eb78743-3dfc-1586-095e-59c86bb58f37 last-leader-id=server-5eb78743-3dfc-1586-095e-59c86bb58f37
2025-04-21T21:24:30.833Z [INFO]  server-a0f798de-4563-4b6c-04ef-15ee21499001: entering candidate state: node="Node at a0f798de-4563-4b6c-04ef-15ee21499001 [Candidate]" term=3
2025-04-21T21:24:30.833Z [INFO]  server-a0f798de-4563-4b6c-04ef-15ee21499001: entering follower state: follower="Node at a0f798de-4563-4b6c-04ef-15ee21499001 [Follower]" leader-address= leader-id=
2025-04-21T21:24:30.845Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:30.849Z [WARN]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:30.849Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: entering candidate state: node="Node at 5eb78743-3dfc-1586-095e-59c86bb58f37 [Candidate]" term=24
2025-04-21T21:24:30.849Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: election won: term=24 tally=2
2025-04-21T21:24:30.849Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: entering leader state: leader="Node at 5eb78743-3dfc-1586-095e-59c86bb58f37 [Leader]"
2025-04-21T21:24:30.849Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: added peer, starting replication: peer=server-a0f798de-4563-4b6c-04ef-15ee21499001
2025-04-21T21:24:30.849Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: added peer, starting replication: peer=server-d3cc73e1-e778-d00d-7652-d8a9412ecb69
2025-04-21T21:24:30.849Z [WARN]  server-d3cc73e1-e778-d00d-7652-d8a9412ecb69: failed to get previous log: previous-index=4 last-index=2 error="log not found"
2025-04-21T21:24:30.849Z [INFO]  server-d3cc73e1-e778-d00d-7652-d8a9412ecb69: entering follower state: follower="Node at d3cc73e1-e778-d00d-7652-d8a9412ecb69 [Follower]" leader-address=5eb78743-3dfc-1586-095e-59c86bb58f37 leader-id=server-5eb78743-3dfc-1586-095e-59c86bb58f37
2025-04-21T21:24:30.849Z [WARN]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: appendEntries rejected, sending older logs: peer="{Nonvoter server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 d3cc73e1-e778-d00d-7652-d8a9412ecb69}" next=3
2025-04-21T21:24:30.849Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: pipelining replication: peer="{Nonvoter server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 d3cc73e1-e778-d00d-7652-d8a9412ecb69}"
2025-04-21T21:24:30.849Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: pipelining replication: peer="{Voter server-a0f798de-4563-4b6c-04ef-15ee21499001 a0f798de-4563-4b6c-04ef-15ee21499001}"
2025-04-21T21:24:30.957Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:30.849518473 +0000 UTC m=+109.371177796 (1 nodes), 4.346469ms from start of poll, 511.655239ms from cluster start. Timeout at 2025-04-21 21:24:30.954520833 +0000 UTC m=+109.476180116, 105.00232ms after stability
2025-04-21T21:24:30.989Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: updating configuration: command=RemoveServer server-id=server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 server-addr= servers="[{Suffrage:Voter ID:server-5eb78743-3dfc-1586-095e-59c86bb58f37 Address:5eb78743-3dfc-1586-095e-59c86bb58f37} {Suffrage:Voter ID:server-a0f798de-4563-4b6c-04ef-15ee21499001 Address:a0f798de-4563-4b6c-04ef-15ee21499001}]"
2025-04-21T21:24:30.989Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: removed peer, stopping replication: peer=server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 last-index=106
2025-04-21T21:24:30.989Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: aborting pipeline replication: peer="{Nonvoter server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 d3cc73e1-e778-d00d-7652-d8a9412ecb69}"
2025-04-21T21:24:30.989Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: updating configuration: command=AddVoter server-id=server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 server-addr=d3cc73e1-e778-d00d-7652-d8a9412ecb69 servers="[{Suffrage:Voter ID:server-5eb78743-3dfc-1586-095e-59c86bb58f37 Address:5eb78743-3dfc-1586-095e-59c86bb58f37} {Suffrage:Voter ID:server-a0f798de-4563-4b6c-04ef-15ee21499001 Address:a0f798de-4563-4b6c-04ef-15ee21499001} {Suffrage:Voter ID:server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 Address:d3cc73e1-e778-d00d-7652-d8a9412ecb69}]"
2025-04-21T21:24:30.989Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: added peer, starting replication: peer=server-d3cc73e1-e778-d00d-7652-d8a9412ecb69
2025-04-21T21:24:30.989Z [WARN]  server-d3cc73e1-e778-d00d-7652-d8a9412ecb69: failed to get previous log: previous-index=107 last-index=106 error="log not found"
2025-04-21T21:24:30.989Z [WARN]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: appendEntries rejected, sending older logs: peer="{Voter server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 d3cc73e1-e778-d00d-7652-d8a9412ecb69}" next=107
2025-04-21T21:24:30.989Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: pipelining replication: peer="{Voter server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 d3cc73e1-e778-d00d-7652-d8a9412ecb69}"
2025-04-21T21:24:32.073Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:32.181Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:32.073358019 +0000 UTC m=+110.595017342 (1 nodes), 1.64µs from start of poll, 1.735494785s from cluster start. Timeout at 2025-04-21 21:24:32.178362579 +0000 UTC m=+110.700021902, 105.00456ms after stability
2025-04-21T21:24:32.181Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: aborting pipeline replication: peer="{Voter server-d3cc73e1-e778-d00d-7652-d8a9412ecb69 d3cc73e1-e778-d00d-7652-d8a9412ecb69}"
2025-04-21T21:24:32.181Z [INFO]  server-5eb78743-3dfc-1586-095e-59c86bb58f37: aborting pipeline replication: peer="{Voter server-a0f798de-4563-4b6c-04ef-15ee21499001 a0f798de-4563-4b6c-04ef-15ee21499001}"
--- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.84s)
=== RUN   TestRaft_FollowerRemovalNoElection
2025-04-21T21:24:32.182Z [INFO]  server-fe05031f-7f9f-e66c-b2f8-876998b34658: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe05031f-7f9f-e66c-b2f8-876998b34658 Address:fe05031f-7f9f-e66c-b2f8-876998b34658} {Suffrage:Voter ID:server-df202865-12ca-9732-2dcd-24d0abd0d4d5 Address:df202865-12ca-9732-2dcd-24d0abd0d4d5} {Suffrage:Voter ID:server-f89fa984-fd1c-6195-938b-834057b1c184 Address:f89fa984-fd1c-6195-938b-834057b1c184}]"
2025-04-21T21:24:32.182Z [INFO]  server-df202865-12ca-9732-2dcd-24d0abd0d4d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe05031f-7f9f-e66c-b2f8-876998b34658 Address:fe05031f-7f9f-e66c-b2f8-876998b34658} {Suffrage:Voter ID:server-df202865-12ca-9732-2dcd-24d0abd0d4d5 Address:df202865-12ca-9732-2dcd-24d0abd0d4d5} {Suffrage:Voter ID:server-f89fa984-fd1c-6195-938b-834057b1c184 Address:f89fa984-fd1c-6195-938b-834057b1c184}]"
2025-04-21T21:24:32.182Z [INFO]  server-f89fa984-fd1c-6195-938b-834057b1c184: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe05031f-7f9f-e66c-b2f8-876998b34658 Address:fe05031f-7f9f-e66c-b2f8-876998b34658} {Suffrage:Voter ID:server-df202865-12ca-9732-2dcd-24d0abd0d4d5 Address:df202865-12ca-9732-2dcd-24d0abd0d4d5} {Suffrage:Voter ID:server-f89fa984-fd1c-6195-938b-834057b1c184 Address:f89fa984-fd1c-6195-938b-834057b1c184}]"
2025-04-21T21:24:32.182Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:32.182Z [INFO]  server-fe05031f-7f9f-e66c-b2f8-876998b34658: entering follower state: follower="Node at fe05031f-7f9f-e66c-b2f8-876998b34658 [Follower]" leader-address= leader-id=
2025-04-21T21:24:32.182Z [INFO]  server-df202865-12ca-9732-2dcd-24d0abd0d4d5: entering follower state: follower="Node at df202865-12ca-9732-2dcd-24d0abd0d4d5 [Follower]" leader-address= leader-id=
2025-04-21T21:24:32.182Z [INFO]  server-f89fa984-fd1c-6195-938b-834057b1c184: entering follower state: follower="Node at f89fa984-fd1c-6195-938b-834057b1c184 [Follower]" leader-address= leader-id=
2025-04-21T21:24:32.313Z [WARN]  server-f89fa984-fd1c-6195-938b-834057b1c184: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:32.313Z [INFO]  server-f89fa984-fd1c-6195-938b-834057b1c184: entering candidate state: node="Node at f89fa984-fd1c-6195-938b-834057b1c184 [Candidate]" term=2
2025-04-21T21:24:32.313Z [INFO]  server-f89fa984-fd1c-6195-938b-834057b1c184: election won: term=2 tally=2
2025-04-21T21:24:32.313Z [INFO]  server-f89fa984-fd1c-6195-938b-834057b1c184: entering leader state: leader="Node at f89fa984-fd1c-6195-938b-834057b1c184 [Leader]"
2025-04-21T21:24:32.313Z [INFO]  server-f89fa984-fd1c-6195-938b-834057b1c184: added peer, starting replication: peer=server-fe05031f-7f9f-e66c-b2f8-876998b34658
2025-04-21T21:24:32.313Z [INFO]  server-f89fa984-fd1c-6195-938b-834057b1c184: added peer, starting replication: peer=server-df202865-12ca-9732-2dcd-24d0abd0d4d5
2025-04-21T21:24:32.313Z [INFO]  server-f89fa984-fd1c-6195-938b-834057b1c184: pipelining replication: peer="{Voter server-df202865-12ca-9732-2dcd-24d0abd0d4d5 df202865-12ca-9732-2dcd-24d0abd0d4d5}"
2025-04-21T21:24:32.313Z [WARN]  server-fe05031f-7f9f-e66c-b2f8-876998b34658: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:32.313Z [INFO]  server-fe05031f-7f9f-e66c-b2f8-876998b34658: entering candidate state: node="Node at fe05031f-7f9f-e66c-b2f8-876998b34658 [Candidate]" term=2
2025-04-21T21:24:32.313Z [INFO]  server-fe05031f-7f9f-e66c-b2f8-876998b34658: duplicate requestVote for same term: term=2
2025-04-21T21:24:32.313Z [INFO]  server-fe05031f-7f9f-e66c-b2f8-876998b34658: entering follower state: follower="Node at fe05031f-7f9f-e66c-b2f8-876998b34658 [Follower]" leader-address=f89fa984-fd1c-6195-938b-834057b1c184 leader-id=server-f89fa984-fd1c-6195-938b-834057b1c184
2025-04-21T21:24:32.313Z [INFO]  server-f89fa984-fd1c-6195-938b-834057b1c184: pipelining replication: peer="{Voter server-fe05031f-7f9f-e66c-b2f8-876998b34658 fe05031f-7f9f-e66c-b2f8-876998b34658}"
2025-04-21T21:24:32.313Z [WARN]  server-df202865-12ca-9732-2dcd-24d0abd0d4d5: rejecting vote request since we have a leader: from=fe05031f-7f9f-e66c-b2f8-876998b34658 leader=f89fa984-fd1c-6195-938b-834057b1c184 leader-id=server-f89fa984-fd1c-6195-938b-834057b1c184
2025-04-21T21:24:32.313Z [WARN]  server-f89fa984-fd1c-6195-938b-834057b1c184: rejecting vote request since we have a leader: from=fe05031f-7f9f-e66c-b2f8-876998b34658 leader=f89fa984-fd1c-6195-938b-834057b1c184 leader-id=server-f89fa984-fd1c-6195-938b-834057b1c184
2025-04-21T21:24:32.521Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:32.313928556 +0000 UTC m=+110.835587879 (1 nodes), 131.722486ms from start of poll, 131.889727ms from cluster start. Timeout at 2025-04-21 21:24:32.518933436 +0000 UTC m=+111.040592759, 205.00488ms after stability
2025-04-21T21:24:32.521Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:32.729Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:32.521958234 +0000 UTC m=+111.043617557 (1 nodes), 1.44µs from start of poll, 339.919405ms from cluster start. Timeout at 2025-04-21 21:24:32.726985194 +0000 UTC m=+111.248644517, 205.02696ms after stability
2025-04-21T21:24:32.737Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:32.945Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:32.73714272 +0000 UTC m=+111.258802043 (2 nodes), 3.8µs from start of poll, 555.103891ms from cluster start. Timeout at 2025-04-21 21:24:32.942182641 +0000 UTC m=+111.463841964, 205.039921ms after stability
2025-04-21T21:24:32.945Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:33.153Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:32.94541436 +0000 UTC m=+111.467073683 (1 nodes), 1.48µs from start of poll, 763.375531ms from cluster start. Timeout at 2025-04-21 21:24:33.150457881 +0000 UTC m=+111.672117204, 205.043521ms after stability
    raft_test.go:2961: [INFO] restarting Node at fe05031f-7f9f-e66c-b2f8-876998b34658 [Follower]
2025-04-21T21:24:33.153Z [INFO]  server-fe05031f-7f9f-e66c-b2f8-876998b34658: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fe05031f-7f9f-e66c-b2f8-876998b34658 Address:fe05031f-7f9f-e66c-b2f8-876998b34658} {Suffrage:Voter ID:server-df202865-12ca-9732-2dcd-24d0abd0d4d5 Address:df202865-12ca-9732-2dcd-24d0abd0d4d5} {Suffrage:Voter ID:server-f89fa984-fd1c-6195-938b-834057b1c184 Address:f89fa984-fd1c-6195-938b-834057b1c184}]"
2025-04-21T21:24:33.153Z [INFO]  server-fe05031f-7f9f-e66c-b2f8-876998b34658: entering follower state: follower="Node at fe05031f-7f9f-e66c-b2f8-876998b34658 [Follower]" leader-address= leader-id=
2025-04-21T21:24:33.205Z [WARN]  server-f89fa984-fd1c-6195-938b-834057b1c184: failed to contact: server-id=server-fe05031f-7f9f-e66c-b2f8-876998b34658 time=55.912255ms
2025-04-21T21:24:33.253Z [WARN]  server-f89fa984-fd1c-6195-938b-834057b1c184: failed to contact: server-id=server-fe05031f-7f9f-e66c-b2f8-876998b34658 time=103.755777ms
2025-04-21T21:24:33.301Z [WARN]  server-fe05031f-7f9f-e66c-b2f8-876998b34658: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:33.301Z [INFO]  server-fe05031f-7f9f-e66c-b2f8-876998b34658: entering candidate state: node="Node at fe05031f-7f9f-e66c-b2f8-876998b34658 [Candidate]" term=3
2025-04-21T21:24:33.301Z [WARN]  server-df202865-12ca-9732-2dcd-24d0abd0d4d5: rejecting vote request since we have a leader: from=fe05031f-7f9f-e66c-b2f8-876998b34658 leader=f89fa984-fd1c-6195-938b-834057b1c184 leader-id=server-f89fa984-fd1c-6195-938b-834057b1c184
2025-04-21T21:24:33.301Z [WARN]  server-f89fa984-fd1c-6195-938b-834057b1c184: rejecting vote request since we have a leader: from=fe05031f-7f9f-e66c-b2f8-876998b34658 leader=f89fa984-fd1c-6195-938b-834057b1c184 leader-id=server-f89fa984-fd1c-6195-938b-834057b1c184
2025-04-21T21:24:33.405Z [ERROR] cluster: peer sees incorrect leader: peer="Node at fe05031f-7f9f-e66c-b2f8-876998b34658 [Candidate]" leader=[none] expected-leader=f89fa984-fd1c-6195-938b-834057b1c184
    testing.go:604: at least one peer has the wrong notion of leader
2025-04-21T21:24:33.405Z [INFO]  server-f89fa984-fd1c-6195-938b-834057b1c184: aborting pipeline replication: peer="{Voter server-df202865-12ca-9732-2dcd-24d0abd0d4d5 df202865-12ca-9732-2dcd-24d0abd0d4d5}"
2025-04-21T21:24:33.657Z [ERROR] server-f89fa984-fd1c-6195-938b-834057b1c184: failed to heartbeat to: peer=fe05031f-7f9f-e66c-b2f8-876998b34658 backoff time=10ms error="command timed out"
2025-04-21T21:24:33.813Z [ERROR] server-f89fa984-fd1c-6195-938b-834057b1c184: failed to pipeline appendEntries: peer="{Voter server-fe05031f-7f9f-e66c-b2f8-876998b34658 fe05031f-7f9f-e66c-b2f8-876998b34658}" error="command enqueue timeout"
2025-04-21T21:24:33.813Z [INFO]  server-f89fa984-fd1c-6195-938b-834057b1c184: aborting pipeline replication: peer="{Voter server-fe05031f-7f9f-e66c-b2f8-876998b34658 fe05031f-7f9f-e66c-b2f8-876998b34658}"
2025-04-21T21:24:34.189Z [ERROR] server-f89fa984-fd1c-6195-938b-834057b1c184: failed to heartbeat to: peer=fe05031f-7f9f-e66c-b2f8-876998b34658 backoff time=10ms error="command timed out"
--- FAIL: TestRaft_FollowerRemovalNoElection (2.01s)
=== RUN   TestRaft_VoteWithNoIDNoAddr
2025-04-21T21:24:34.190Z [INFO]  server-9b29f47f-ef64-69f9-2689-c00dcd4e463a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b29f47f-ef64-69f9-2689-c00dcd4e463a Address:9b29f47f-ef64-69f9-2689-c00dcd4e463a} {Suffrage:Voter ID:server-ac76d87a-1701-ab70-3922-f667831a3ec8 Address:ac76d87a-1701-ab70-3922-f667831a3ec8} {Suffrage:Voter ID:server-c6a4739b-d095-337b-18f2-6332bccb4850 Address:c6a4739b-d095-337b-18f2-6332bccb4850}]"
2025-04-21T21:24:34.190Z [INFO]  server-ac76d87a-1701-ab70-3922-f667831a3ec8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b29f47f-ef64-69f9-2689-c00dcd4e463a Address:9b29f47f-ef64-69f9-2689-c00dcd4e463a} {Suffrage:Voter ID:server-ac76d87a-1701-ab70-3922-f667831a3ec8 Address:ac76d87a-1701-ab70-3922-f667831a3ec8} {Suffrage:Voter ID:server-c6a4739b-d095-337b-18f2-6332bccb4850 Address:c6a4739b-d095-337b-18f2-6332bccb4850}]"
2025-04-21T21:24:34.190Z [INFO]  server-c6a4739b-d095-337b-18f2-6332bccb4850: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9b29f47f-ef64-69f9-2689-c00dcd4e463a Address:9b29f47f-ef64-69f9-2689-c00dcd4e463a} {Suffrage:Voter ID:server-ac76d87a-1701-ab70-3922-f667831a3ec8 Address:ac76d87a-1701-ab70-3922-f667831a3ec8} {Suffrage:Voter ID:server-c6a4739b-d095-337b-18f2-6332bccb4850 Address:c6a4739b-d095-337b-18f2-6332bccb4850}]"
2025-04-21T21:24:34.190Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:34.190Z [INFO]  server-9b29f47f-ef64-69f9-2689-c00dcd4e463a: entering follower state: follower="Node at 9b29f47f-ef64-69f9-2689-c00dcd4e463a [Follower]" leader-address= leader-id=
2025-04-21T21:24:34.190Z [INFO]  server-ac76d87a-1701-ab70-3922-f667831a3ec8: entering follower state: follower="Node at ac76d87a-1701-ab70-3922-f667831a3ec8 [Follower]" leader-address= leader-id=
2025-04-21T21:24:34.190Z [INFO]  server-c6a4739b-d095-337b-18f2-6332bccb4850: entering follower state: follower="Node at c6a4739b-d095-337b-18f2-6332bccb4850 [Follower]" leader-address= leader-id=
2025-04-21T21:24:34.277Z [WARN]  server-ac76d87a-1701-ab70-3922-f667831a3ec8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-21T21:24:34.277Z [INFO]  server-ac76d87a-1701-ab70-3922-f667831a3ec8: entering candidate state: node="Node at ac76d87a-1701-ab70-3922-f667831a3ec8 [Candidate]" term=2
2025-04-21T21:24:34.277Z [INFO]  server-ac76d87a-1701-ab70-3922-f667831a3ec8: election won: term=2 tally=2
2025-04-21T21:24:34.277Z [INFO]  server-ac76d87a-1701-ab70-3922-f667831a3ec8: entering leader state: leader="Node at ac76d87a-1701-ab70-3922-f667831a3ec8 [Leader]"
2025-04-21T21:24:34.277Z [INFO]  server-ac76d87a-1701-ab70-3922-f667831a3ec8: added peer, starting replication: peer=server-9b29f47f-ef64-69f9-2689-c00dcd4e463a
2025-04-21T21:24:34.277Z [INFO]  server-ac76d87a-1701-ab70-3922-f667831a3ec8: added peer, starting replication: peer=server-c6a4739b-d095-337b-18f2-6332bccb4850
2025-04-21T21:24:34.277Z [INFO]  server-ac76d87a-1701-ab70-3922-f667831a3ec8: pipelining replication: peer="{Voter server-c6a4739b-d095-337b-18f2-6332bccb4850 c6a4739b-d095-337b-18f2-6332bccb4850}"
2025-04-21T21:24:34.277Z [INFO]  server-ac76d87a-1701-ab70-3922-f667831a3ec8: pipelining replication: peer="{Voter server-9b29f47f-ef64-69f9-2689-c00dcd4e463a 9b29f47f-ef64-69f9-2689-c00dcd4e463a}"
2025-04-21T21:24:34.389Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:34.277604634 +0000 UTC m=+112.799263957 (1 nodes), 87.487628ms from start of poll, 87.694189ms from cluster start. Timeout at 2025-04-21 21:24:34.382607234 +0000 UTC m=+112.904266557, 105.0026ms after stability
2025-04-21T21:24:34.389Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-04-21T21:24:34.497Z [INFO]  cluster: stable state for Leader reached at 2025-04-21 21:24:34.389855309 +0000 UTC m=+112.911514632 (1 nodes), 1.44µs from start of poll, 199.944864ms from cluster start. Timeout at 2025-04-21 21:24:34.494871789 +0000 UTC m=+113.016531112, 105.01648ms after stability
2025-04-21T21:24:34.505Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-04-21T21:24:34.613Z [INFO]  cluster: stable state for Follower reached at 2025-04-21 21:24:34.505231084 +0000 UTC m=+113.026890407 (2 nodes), 2µs from start of poll, 315.320639ms from cluster start. Timeout at 2025-04-21 21:24:34.610295644 +0000 UTC m=+113.131954967, 105.06456ms after stability
2025-04-21T21:24:34.617Z [ERROR] server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to heartbeat to: peer=9b29f47f-ef64-69f9-2689-c00dcd4e463a backoff time=10ms error="failed to connect to peer: 9b29f47f-ef64-69f9-2689-c00dcd4e463a"
2025-04-21T21:24:34.617Z [ERROR] server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to pipeline appendEntries: peer="{Voter server-c6a4739b-d095-337b-18f2-6332bccb4850 c6a4739b-d095-337b-18f2-6332bccb4850}" error="append pipeline closed"
2025-04-21T21:24:34.617Z [INFO]  server-ac76d87a-1701-ab70-3922-f667831a3ec8: aborting pipeline replication: peer="{Voter server-c6a4739b-d095-337b-18f2-6332bccb4850 c6a4739b-d095-337b-18f2-6332bccb4850}"
2025-04-21T21:24:34.617Z [ERROR] server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to pipeline appendEntries: peer="{Voter server-9b29f47f-ef64-69f9-2689-c00dcd4e463a 9b29f47f-ef64-69f9-2689-c00dcd4e463a}" error="append pipeline closed"
2025-04-21T21:24:34.617Z [INFO]  server-ac76d87a-1701-ab70-3922-f667831a3ec8: aborting pipeline replication: peer="{Voter server-9b29f47f-ef64-69f9-2689-c00dcd4e463a 9b29f47f-ef64-69f9-2689-c00dcd4e463a}"
2025-04-21T21:24:34.621Z [ERROR] server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to heartbeat to: peer=c6a4739b-d095-337b-18f2-6332bccb4850 backoff time=10ms error="failed to connect to peer: c6a4739b-d095-337b-18f2-6332bccb4850"
2025-04-21T21:24:34.625Z [ERROR] server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to appendEntries to: peer="{Voter server-c6a4739b-d095-337b-18f2-6332bccb4850 c6a4739b-d095-337b-18f2-6332bccb4850}" error="failed to connect to peer: c6a4739b-d095-337b-18f2-6332bccb4850"
2025-04-21T21:24:34.625Z [ERROR] server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to appendEntries to: peer="{Voter server-9b29f47f-ef64-69f9-2689-c00dcd4e463a 9b29f47f-ef64-69f9-2689-c00dcd4e463a}" error="failed to connect to peer: 9b29f47f-ef64-69f9-2689-c00dcd4e463a"
2025-04-21T21:24:34.637Z [ERROR] server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to heartbeat to: peer=9b29f47f-ef64-69f9-2689-c00dcd4e463a backoff time=10ms error="failed to connect to peer: 9b29f47f-ef64-69f9-2689-c00dcd4e463a"
2025-04-21T21:24:34.645Z [ERROR] server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to heartbeat to: peer=c6a4739b-d095-337b-18f2-6332bccb4850 backoff time=10ms error="failed to connect to peer: c6a4739b-d095-337b-18f2-6332bccb4850"
2025-04-21T21:24:34.646Z [ERROR] server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to appendEntries to: peer="{Voter server-c6a4739b-d095-337b-18f2-6332bccb4850 c6a4739b-d095-337b-18f2-6332bccb4850}" error="failed to connect to peer: c6a4739b-d095-337b-18f2-6332bccb4850"
2025-04-21T21:24:34.657Z [WARN]  server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to contact: server-id=server-9b29f47f-ef64-69f9-2689-c00dcd4e463a time=51.980549ms
2025-04-21T21:24:34.658Z [WARN]  server-9b29f47f-ef64-69f9-2689-c00dcd4e463a: heartbeat timeout reached, starting election: last-leader-addr=ac76d87a-1701-ab70-3922-f667831a3ec8 last-leader-id=server-ac76d87a-1701-ab70-3922-f667831a3ec8
2025-04-21T21:24:34.658Z [INFO]  server-9b29f47f-ef64-69f9-2689-c00dcd4e463a: entering candidate state: node="Node at 9b29f47f-ef64-69f9-2689-c00dcd4e463a [Candidate]" term=3
2025-04-21T21:24:34.658Z [WARN]  server-c6a4739b-d095-337b-18f2-6332bccb4850: rejecting vote request since we have a leader: from=9b29f47f-ef64-69f9-2689-c00dcd4e463a leader=ac76d87a-1701-ab70-3922-f667831a3ec8 leader-id=server-ac76d87a-1701-ab70-3922-f667831a3ec8
2025-04-21T21:24:34.658Z [ERROR] server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to appendEntries to: peer="{Voter server-9b29f47f-ef64-69f9-2689-c00dcd4e463a 9b29f47f-ef64-69f9-2689-c00dcd4e463a}" error="failed to connect to peer: 9b29f47f-ef64-69f9-2689-c00dcd4e463a"
2025-04-21T21:24:34.658Z [ERROR] server-9b29f47f-ef64-69f9-2689-c00dcd4e463a: failed to make requestVote RPC: target="{Voter server-ac76d87a-1701-ab70-3922-f667831a3ec8 ac76d87a-1701-ab70-3922-f667831a3ec8}" error="failed to connect to peer: ac76d87a-1701-ab70-3922-f667831a3ec8" term=3
2025-04-21T21:24:34.665Z [ERROR] server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to appendEntries to: peer="{Voter server-c6a4739b-d095-337b-18f2-6332bccb4850 c6a4739b-d095-337b-18f2-6332bccb4850}" error="failed to connect to peer: c6a4739b-d095-337b-18f2-6332bccb4850"
2025-04-21T21:24:34.665Z [ERROR] server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to heartbeat to: peer=c6a4739b-d095-337b-18f2-6332bccb4850 backoff time=10ms error="failed to connect to peer: c6a4739b-d095-337b-18f2-6332bccb4850"
2025-04-21T21:24:34.665Z [ERROR] server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to heartbeat to: peer=9b29f47f-ef64-69f9-2689-c00dcd4e463a backoff time=10ms error="failed to connect to peer: 9b29f47f-ef64-69f9-2689-c00dcd4e463a"
2025-04-21T21:24:34.669Z [WARN]  server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to contact: server-id=server-9b29f47f-ef64-69f9-2689-c00dcd4e463a time=63.805149ms
2025-04-21T21:24:34.669Z [WARN]  server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to contact: server-id=server-c6a4739b-d095-337b-18f2-6332bccb4850 time=59.50548ms
2025-04-21T21:24:34.669Z [WARN]  server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to contact quorum of nodes, stepping down
2025-04-21T21:24:34.669Z [INFO]  server-ac76d87a-1701-ab70-3922-f667831a3ec8: entering follower state: follower="Node at ac76d87a-1701-ab70-3922-f667831a3ec8 [Follower]" leader-address= leader-id=
2025-04-21T21:24:34.677Z [ERROR] server-ac76d87a-1701-ab70-3922-f667831a3ec8: failed to appendEntries to: peer="{Voter server-9b29f47f-ef64-69f9-2689-c00dcd4e463a 9b29f47f-ef64-69f9-2689-c00dcd4e463a}" error="failed to connect to peer: 9b29f47f-ef64-69f9-2689-c00dcd4e463a"
2025-04-21T21:24:34.713Z [WARN]  server-c6a4739b-d095-337b-18f2-6332bccb4850: heartbeat timeout reached, starting election: last-leader-addr=ac76d87a-1701-ab70-3922-f667831a3ec8 last-leader-id=server-ac76d87a-1701-ab70-3922-f667831a3ec8
2025-04-21T21:24:34.713Z [INFO]  server-c6a4739b-d095-337b-18f2-6332bccb4850: entering candidate state: node="Node at c6a4739b-d095-337b-18f2-6332bccb4850 [Candidate]" term=3
2025-04-21T21:24:34.713Z [ERROR] server-c6a4739b-d095-337b-18f2-6332bccb4850: failed to make requestVote RPC: target="{Voter server-ac76d87a-1701-ab70-3922-f667831a3ec8 ac76d87a-1701-ab70-3922-f667831a3ec8}" error="failed to connect to peer: ac76d87a-1701-ab70-3922-f667831a3ec8" term=3
2025-04-21T21:24:34.713Z [INFO]  server-9b29f47f-ef64-69f9-2689-c00dcd4e463a: duplicate requestVote for same term: term=3
2025-04-21T21:24:34.721Z [INFO]  server-9b29f47f-ef64-69f9-2689-c00dcd4e463a: entering follower state: follower="Node at 9b29f47f-ef64-69f9-2689-c00dcd4e463a [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VoteWithNoIDNoAddr (0.53s)
=== RUN   TestRaft_runFollower_State_Transition
=== RUN   TestRaft_runFollower_State_Transition/NonVoter
2025-04-21T21:24:34.722Z [INFO]  127.0.0.1:33937 :: starting node: addr=127.0.0.1:33937
2025-04-21T21:24:34.722Z [INFO]  127.0.0.1:33937 :: initial configuration: index=0 servers=[]
2025-04-21T21:24:34.722Z [INFO]  127.0.0.1:33937 :: entering follower state: follower="Node at 127.0.0.1:33937 [Follower]" leader-address= leader-id=
2025-04-21T21:24:34.797Z [WARN]  127.0.0.1:33937 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
=== RUN   TestRaft_runFollower_State_Transition/Voter
2025-04-21T21:24:34.873Z [INFO]  127.0.0.1:33783 :: starting node: addr=127.0.0.1:33783
2025-04-21T21:24:34.873Z [INFO]  127.0.0.1:33783 :: initial configuration: index=0 servers=[]
2025-04-21T21:24:34.873Z [INFO]  127.0.0.1:33783 :: entering follower state: follower="Node at 127.0.0.1:33783 [Follower]" leader-address= leader-id=
2025-04-21T21:24:34.953Z [WARN]  127.0.0.1:33783 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
=== RUN   TestRaft_runFollower_State_Transition/Not_in_Config
2025-04-21T21:24:35.025Z [INFO]  127.0.0.1:37441 :: starting node: addr=127.0.0.1:37441
2025-04-21T21:24:35.025Z [INFO]  127.0.0.1:37441 :: initial configuration: index=0 servers=[]
2025-04-21T21:24:35.025Z [INFO]  127.0.0.1:37441 :: entering follower state: follower="Node at 127.0.0.1:37441 [Follower]" leader-address= leader-id=
2025-04-21T21:24:35.109Z [WARN]  127.0.0.1:37441 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
--- PASS: TestRaft_runFollower_State_Transition (0.46s)
    --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s)
=== RUN   TestRaft_runFollower_ReloadTimeoutConfigs
2025-04-21T21:24:35.177Z [INFO]  127.0.0.1:45619 :: starting node: addr=127.0.0.1:45619
2025-04-21T21:24:35.177Z [INFO]  127.0.0.1:45619 :: initial configuration: index=0 servers=[]
2025-04-21T21:24:35.177Z [INFO]  127.0.0.1:45619 :: entering follower state: follower="Node at 127.0.0.1:45619 [Follower]" leader-address= leader-id=
2025-04-21T21:24:35.177Z [WARN]  127.0.0.1:45619 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
--- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s)
=== RUN   TestSaturationMetric
=== RUN   TestSaturationMetric/without_smoothing
--- PASS: TestSaturationMetric (0.00s)
    --- PASS: TestSaturationMetric/without_smoothing (0.00s)
=== RUN   TestSaturationMetric_IncorrectUsage
=== RUN   TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_first
--- PASS: TestSaturationMetric_IncorrectUsage (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s)
=== RUN   TestTCPTransport_BadAddr
--- PASS: TestTCPTransport_BadAddr (0.00s)
=== RUN   TestTCPTransport_EmptyAddr
--- PASS: TestTCPTransport_EmptyAddr (0.00s)
=== RUN   TestTCPTransport_WithAdvertise
--- PASS: TestTCPTransport_WithAdvertise (0.00s)
=== RUN   TestTransport_StartStop
--- PASS: TestTransport_StartStop (0.00s)
=== RUN   TestTransport_AppendEntries
--- PASS: TestTransport_AppendEntries (0.00s)
=== RUN   TestTransport_AppendEntriesPipeline
--- PASS: TestTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestTransport_RequestVote
--- PASS: TestTransport_RequestVote (0.00s)
=== RUN   TestTransport_InstallSnapshot
--- PASS: TestTransport_InstallSnapshot (0.00s)
=== RUN   TestTransport_EncodeDecode
--- PASS: TestTransport_EncodeDecode (0.00s)
=== RUN   TestRandomTimeout
--- PASS: TestRandomTimeout (0.01s)
=== RUN   TestNewSeed
--- PASS: TestNewSeed (0.00s)
=== RUN   TestRandomTimeout_NoTime
--- PASS: TestRandomTimeout_NoTime (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
=== RUN   TestMax
--- PASS: TestMax (0.00s)
=== RUN   TestGenerateUUID
--- PASS: TestGenerateUUID (0.00s)
=== RUN   TestBackoff
--- PASS: TestBackoff (0.00s)
=== RUN   TestOverrideNotifyBool
--- PASS: TestOverrideNotifyBool (0.00s)
FAIL
FAIL	github.com/hashicorp/raft	113.899s
?   	github.com/hashicorp/raft/bench	[no test files]
FAIL
dh_auto_test: error: cd obj-aarch64-linux-gnu && go test -vet=off -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench returned exit code 1
make[1]: *** [debian/rules:17: override_dh_auto_test] Error 25
make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
make: *** [debian/rules:14: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
I: copying local configuration
E: Failed autobuilding of package
I: user script /srv/workspace/pbuilder/2469999/tmp/hooks/C01_cleanup starting
debug output: disk usage on i-capture-the-hostname at Mon Apr 21 21:24:35 UTC 2025
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            30G     0   30G   0% /dev/shm

I: user script /srv/workspace/pbuilder/2469999/tmp/hooks/C01_cleanup finished
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /srv/workspace/pbuilder/2469999 and its subdirectories