I: pbuilder: network access will be disabled during build
I: Current time: Sun Jan 26 16:41:04 +14 2025
I: pbuilder-time-stamp: 1737859264
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/unstable-reproducible-base.tgz]
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: policy-rc.d already exists
I: Copying source file
I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc]
I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz]
I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz]
I: Extracting source
dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: unsupported subcommand
dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying 0001-Skip-unreliable-tests.patch
I: Not using root during the build.
I: Installing the build-deps
I: user script /srv/workspace/pbuilder/778150/tmp/hooks/D01_modify_environment starting
debug: Running on codethink04-arm64.
I: Changing host+domainname to test build reproducibility
I: Adding a custom variable just for the fun of it...
I: Changing /bin/sh to bash
'/bin/sh' -> '/bin/bash'
lrwxrwxrwx 1 root root 9 Jan 26 02:41 /bin/sh -> /bin/bash
I: Setting pbuilder2's login shell to /bin/bash
I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other
I: user script /srv/workspace/pbuilder/778150/tmp/hooks/D01_modify_environment finished
I: user script /srv/workspace/pbuilder/778150/tmp/hooks/D02_print_environment starting
I: set
  BASH=/bin/sh
  BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:globskipdots:hostcomplete:interactive_comments:patsub_replacement:progcomp:promptvars:sourcepath
  BASH_ALIASES=()
  BASH_ARGC=()
  BASH_ARGV=()
  BASH_CMDS=()
  BASH_LINENO=([0]="12" [1]="0")
  BASH_LOADABLES_PATH=/usr/local/lib/bash:/usr/lib/bash:/opt/local/lib/bash:/usr/pkg/lib/bash:/opt/pkg/lib/bash:.
  BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment")
  BASH_VERSINFO=([0]="5" [1]="2" [2]="37" [3]="1" [4]="release" [5]="aarch64-unknown-linux-gnu")
  BASH_VERSION='5.2.37(1)-release'
  BUILDDIR=/build/reproducible-path
  BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other'
  BUILDUSERNAME=pbuilder2
  BUILD_ARCH=arm64
  DEBIAN_FRONTEND=noninteractive
  DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=12 '
  DIRSTACK=()
  DISTRIBUTION=unstable
  EUID=0
  FUNCNAME=([0]="Echo" [1]="main")
  GROUPS=()
  HOME=/root
  HOSTNAME=i-capture-the-hostname
  HOSTTYPE=aarch64
  HOST_ARCH=arm64
  IFS=' 	
  '
  INVOCATION_ID=605fa77672f44d08bbf11336bc83d05d
  LANG=C
  LANGUAGE=nl_BE:nl
  LC_ALL=C
  MACHTYPE=aarch64-unknown-linux-gnu
  MAIL=/var/mail/root
  OPTERR=1
  OPTIND=1
  OSTYPE=linux-gnu
  PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path
  PBCURRENTCOMMANDLINEOPERATION=build
  PBUILDER_OPERATION=build
  PBUILDER_PKGDATADIR=/usr/share/pbuilder
  PBUILDER_PKGLIBDIR=/usr/lib/pbuilder
  PBUILDER_SYSCONFDIR=/etc
  PIPESTATUS=([0]="0")
  POSIXLY_CORRECT=y
  PPID=778150
  PS4='+ '
  PWD=/
  SHELL=/bin/bash
  SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix
  SHLVL=3
  SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.dqTaEGxs/pbuilderrc_k63X --distribution unstable --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/unstable-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.dqTaEGxs/b2 --logfile b2/build.log golang-github-hashicorp-raft_1.5.0-1.dsc'
  SUDO_GID=109
  SUDO_UID=104
  SUDO_USER=jenkins
  TERM=unknown
  TZ=/usr/share/zoneinfo/Etc/GMT-14
  UID=0
  USER=root
  _='I: set'
  http_proxy=http://192.168.101.4:3128
I: uname -a
  Linux i-capture-the-hostname 6.1.0-30-cloud-arm64 #1 SMP Debian 6.1.124-1 (2025-01-12) aarch64 GNU/Linux
I: ls -l /bin
  lrwxrwxrwx 1 root root 7 Nov 22 14:40 /bin -> usr/bin
I: user script /srv/workspace/pbuilder/778150/tmp/hooks/D02_print_environment finished
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: arm64
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 19946 files and directories currently installed.)
Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ...
Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however:
  Package debhelper-compat is not installed.
 pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however:
  Package dh-sequence-golang is not installed.
 pbuilder-satisfydepends-dummy depends on golang-any; however:
  Package golang-any is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however:
  Package golang-github-armon-go-metrics-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however:
  Package golang-github-hashicorp-go-hclog-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however:
  Package golang-github-hashicorp-go-msgpack-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however:
  Package golang-github-stretchr-testify-dev is not installed.

Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
Building tag database...
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
The following NEW packages will be installed:
  autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.23-go{a} golang-1.23-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libunistring5{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} 
The following packages are RECOMMENDED but will NOT be installed:
  curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 
0 packages upgraded, 106 newly installed, 0 to remove and 0 not upgraded.
Need to get 124 MB of archives. After unpacking 693 MB will be used.
Writing extended state information...
Get: 1 http://deb.debian.org/debian unstable/main arm64 sensible-utils all 0.0.24 [24.8 kB]
Get: 2 http://deb.debian.org/debian unstable/main arm64 libmagic-mgc arm64 1:5.45-3+b1 [314 kB]
Get: 3 http://deb.debian.org/debian unstable/main arm64 libmagic1t64 arm64 1:5.45-3+b1 [102 kB]
Get: 4 http://deb.debian.org/debian unstable/main arm64 file arm64 1:5.45-3+b1 [43.4 kB]
Get: 5 http://deb.debian.org/debian unstable/main arm64 gettext-base arm64 0.23.1-1 [241 kB]
Get: 6 http://deb.debian.org/debian unstable/main arm64 libuchardet0 arm64 0.0.8-1+b2 [69.2 kB]
Get: 7 http://deb.debian.org/debian unstable/main arm64 groff-base arm64 1.23.0-7 [1129 kB]
Get: 8 http://deb.debian.org/debian unstable/main arm64 bsdextrautils arm64 2.40.4-1 [91.6 kB]
Get: 9 http://deb.debian.org/debian unstable/main arm64 libpipeline1 arm64 1.5.8-1 [40.2 kB]
Get: 10 http://deb.debian.org/debian unstable/main arm64 man-db arm64 2.13.0-1 [1404 kB]
Get: 11 http://deb.debian.org/debian unstable/main arm64 m4 arm64 1.4.19-5 [284 kB]
Get: 12 http://deb.debian.org/debian unstable/main arm64 autoconf all 2.72-3 [493 kB]
Get: 13 http://deb.debian.org/debian unstable/main arm64 autotools-dev all 20220109.1 [51.6 kB]
Get: 14 http://deb.debian.org/debian unstable/main arm64 automake all 1:1.16.5-1.3 [823 kB]
Get: 15 http://deb.debian.org/debian unstable/main arm64 autopoint all 0.23.1-1 [770 kB]
Get: 16 http://deb.debian.org/debian unstable/main arm64 libdebhelper-perl all 13.24.1 [90.9 kB]
Get: 17 http://deb.debian.org/debian unstable/main arm64 libtool all 2.5.4-2 [539 kB]
Get: 18 http://deb.debian.org/debian unstable/main arm64 dh-autoreconf all 20 [17.1 kB]
Get: 19 http://deb.debian.org/debian unstable/main arm64 libarchive-zip-perl all 1.68-1 [104 kB]
Get: 20 http://deb.debian.org/debian unstable/main arm64 libfile-stripnondeterminism-perl all 1.14.1-1 [19.6 kB]
Get: 21 http://deb.debian.org/debian unstable/main arm64 dh-strip-nondeterminism all 1.14.1-1 [8584 B]
Get: 22 http://deb.debian.org/debian unstable/main arm64 libelf1t64 arm64 0.192-4 [189 kB]
Get: 23 http://deb.debian.org/debian unstable/main arm64 dwz arm64 0.15-1+b1 [102 kB]
Get: 24 http://deb.debian.org/debian unstable/main arm64 libunistring5 arm64 1.3-1 [449 kB]
Get: 25 http://deb.debian.org/debian unstable/main arm64 libicu72 arm64 72.1-6 [9239 kB]
Get: 26 http://deb.debian.org/debian unstable/main arm64 libxml2 arm64 2.12.7+dfsg+really2.9.14-0.2+b1 [630 kB]
Get: 27 http://deb.debian.org/debian unstable/main arm64 gettext arm64 0.23.1-1 [1610 kB]
Get: 28 http://deb.debian.org/debian unstable/main arm64 intltool-debian all 0.35.0+20060710.6 [22.9 kB]
Get: 29 http://deb.debian.org/debian unstable/main arm64 po-debconf all 1.0.21+nmu1 [248 kB]
Get: 30 http://deb.debian.org/debian unstable/main arm64 debhelper all 13.24.1 [920 kB]
Get: 31 http://deb.debian.org/debian unstable/main arm64 dh-golang all 1.62 [24.7 kB]
Get: 32 http://deb.debian.org/debian unstable/main arm64 golang-1.23-src all 1.23.5-1 [19.1 MB]
Get: 33 http://deb.debian.org/debian unstable/main arm64 golang-1.23-go arm64 1.23.5-1 [23.4 MB]
Get: 34 http://deb.debian.org/debian unstable/main arm64 golang-src all 2:1.23~2 [5088 B]
Get: 35 http://deb.debian.org/debian unstable/main arm64 golang-go arm64 2:1.23~2 [44.3 kB]
Get: 36 http://deb.debian.org/debian unstable/main arm64 golang-any arm64 2:1.23~2 [5172 B]
Get: 37 http://deb.debian.org/debian unstable/main arm64 golang-github-alecthomas-units-dev all 0.0~git20240626.61d1e34-1 [9032 B]
Get: 38 http://deb.debian.org/debian unstable/main arm64 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB]
Get: 39 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB]
Get: 40 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-sys-dev all 0.22.0-1 [431 kB]
Get: 41 http://deb.debian.org/debian unstable/main arm64 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B]
Get: 42 http://deb.debian.org/debian unstable/main arm64 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB]
Get: 43 http://deb.debian.org/debian unstable/main arm64 golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB]
Get: 44 http://deb.debian.org/debian unstable/main arm64 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB]
Get: 45 http://deb.debian.org/debian unstable/main arm64 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB]
Get: 46 http://deb.debian.org/debian unstable/main arm64 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB]
Get: 47 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB]
Get: 48 http://deb.debian.org/debian unstable/main arm64 golang-github-stretchr-testify-dev all 1.9.0-1 [74.7 kB]
Get: 49 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB]
Get: 50 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB]
Get: 51 http://deb.debian.org/debian unstable/main arm64 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB]
Get: 52 http://deb.debian.org/debian unstable/main arm64 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B]
Get: 53 http://deb.debian.org/debian unstable/main arm64 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB]
Get: 54 http://deb.debian.org/debian unstable/main arm64 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB]
Get: 55 http://deb.debian.org/debian unstable/main arm64 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB]
Get: 56 http://deb.debian.org/debian unstable/main arm64 golang-google-protobuf-dev all 1.33.0-1 [625 kB]
Get: 57 http://deb.debian.org/debian unstable/main arm64 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB]
Get: 58 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B]
Get: 59 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB]
Get: 60 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB]
Get: 61 http://deb.debian.org/debian unstable/main arm64 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB]
Get: 62 http://deb.debian.org/debian unstable/main arm64 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB]
Get: 63 http://deb.debian.org/debian unstable/main arm64 golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB]
Get: 64 http://deb.debian.org/debian unstable/main arm64 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B]
Get: 65 http://deb.debian.org/debian unstable/main arm64 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB]
Get: 66 http://deb.debian.org/debian unstable/main arm64 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB]
Get: 67 http://deb.debian.org/debian unstable/main arm64 golang-github-klauspost-compress-dev all 1.17.11+ds1-1 [33.3 MB]
Get: 68 http://deb.debian.org/debian unstable/main arm64 golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB]
Get: 69 http://deb.debian.org/debian unstable/main arm64 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB]
Get: 70 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB]
Get: 71 http://deb.debian.org/debian unstable/main arm64 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB]
Get: 72 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-term-dev all 0.22.0-1 [14.8 kB]
Get: 73 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-text-dev all 0.16.0-1 [4003 kB]
Get: 74 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-crypto-dev all 1:0.25.0-1 [1682 kB]
Get: 75 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-net-dev all 1:0.27.0-1 [898 kB]
Get: 76 http://deb.debian.org/debian unstable/main arm64 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB]
Get: 77 http://deb.debian.org/debian unstable/main arm64 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB]
Get: 78 http://deb.debian.org/debian unstable/main arm64 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB]
Get: 79 http://deb.debian.org/debian unstable/main arm64 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB]
Get: 80 http://deb.debian.org/debian unstable/main arm64 golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B]
Get: 81 http://deb.debian.org/debian unstable/main arm64 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B]
Get: 82 http://deb.debian.org/debian unstable/main arm64 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB]
Get: 83 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-oauth2-dev all 0.15.0-1 [38.0 kB]
Get: 84 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB]
Get: 85 http://deb.debian.org/debian unstable/main arm64 golang-github-prometheus-common-dev all 0.61.0-1 [120 kB]
Get: 86 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB]
Get: 87 http://deb.debian.org/debian unstable/main arm64 golang-github-prometheus-procfs-dev all 0.14.0-1 [183 kB]
Get: 88 http://deb.debian.org/debian unstable/main arm64 golang-github-prometheus-client-golang-dev all 1.20.5-1 [188 kB]
Get: 89 http://deb.debian.org/debian unstable/main arm64 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB]
Get: 90 http://deb.debian.org/debian unstable/main arm64 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB]
Get: 91 http://deb.debian.org/debian unstable/main arm64 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B]
Get: 92 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB]
Get: 93 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B]
Get: 94 http://deb.debian.org/debian unstable/main arm64 libsasl2-modules-db arm64 2.1.28+dfsg1-8+b1 [20.3 kB]
Get: 95 http://deb.debian.org/debian unstable/main arm64 libsasl2-2 arm64 2.1.28+dfsg1-8+b1 [55.7 kB]
Get: 96 http://deb.debian.org/debian unstable/main arm64 libssl-dev arm64 3.4.0-2 [3234 kB]
Get: 97 http://deb.debian.org/debian unstable/main arm64 libsasl2-dev arm64 2.1.28+dfsg1-8+b1 [230 kB]
Get: 98 http://deb.debian.org/debian unstable/main arm64 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB]
Get: 99 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-mod-dev all 0.19.0-1 [104 kB]
Get: 100 http://deb.debian.org/debian unstable/main arm64 golang-github-yuin-goldmark-dev all 1.7.4-1 [113 kB]
Get: 101 http://deb.debian.org/debian unstable/main arm64 golang-golang-x-tools-dev all 1:0.23.0+ds-1 [1593 kB]
Get: 102 http://deb.debian.org/debian unstable/main arm64 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB]
Get: 103 http://deb.debian.org/debian unstable/main arm64 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB]
Get: 104 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB]
Get: 105 http://deb.debian.org/debian unstable/main arm64 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB]
Get: 106 http://deb.debian.org/debian unstable/main arm64 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB]
Fetched 124 MB in 1s (92.6 MB/s)
Preconfiguring packages ...
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 19946 files and directories currently installed.)
Preparing to unpack .../000-sensible-utils_0.0.24_all.deb ...
Unpacking sensible-utils (0.0.24) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3+b1_arm64.deb ...
Unpacking libmagic-mgc (1:5.45-3+b1) ...
Selecting previously unselected package libmagic1t64:arm64.
Preparing to unpack .../002-libmagic1t64_1%3a5.45-3+b1_arm64.deb ...
Unpacking libmagic1t64:arm64 (1:5.45-3+b1) ...
Selecting previously unselected package file.
Preparing to unpack .../003-file_1%3a5.45-3+b1_arm64.deb ...
Unpacking file (1:5.45-3+b1) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../004-gettext-base_0.23.1-1_arm64.deb ...
Unpacking gettext-base (0.23.1-1) ...
Selecting previously unselected package libuchardet0:arm64.
Preparing to unpack .../005-libuchardet0_0.0.8-1+b2_arm64.deb ...
Unpacking libuchardet0:arm64 (0.0.8-1+b2) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../006-groff-base_1.23.0-7_arm64.deb ...
Unpacking groff-base (1.23.0-7) ...
Selecting previously unselected package bsdextrautils.
Preparing to unpack .../007-bsdextrautils_2.40.4-1_arm64.deb ...
Unpacking bsdextrautils (2.40.4-1) ...
Selecting previously unselected package libpipeline1:arm64.
Preparing to unpack .../008-libpipeline1_1.5.8-1_arm64.deb ...
Unpacking libpipeline1:arm64 (1.5.8-1) ...
Selecting previously unselected package man-db.
Preparing to unpack .../009-man-db_2.13.0-1_arm64.deb ...
Unpacking man-db (2.13.0-1) ...
Selecting previously unselected package m4.
Preparing to unpack .../010-m4_1.4.19-5_arm64.deb ...
Unpacking m4 (1.4.19-5) ...
Selecting previously unselected package autoconf.
Preparing to unpack .../011-autoconf_2.72-3_all.deb ...
Unpacking autoconf (2.72-3) ...
Selecting previously unselected package autotools-dev.
Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ...
Unpacking autotools-dev (20220109.1) ...
Selecting previously unselected package automake.
Preparing to unpack .../013-automake_1%3a1.16.5-1.3_all.deb ...
Unpacking automake (1:1.16.5-1.3) ...
Selecting previously unselected package autopoint.
Preparing to unpack .../014-autopoint_0.23.1-1_all.deb ...
Unpacking autopoint (0.23.1-1) ...
Selecting previously unselected package libdebhelper-perl.
Preparing to unpack .../015-libdebhelper-perl_13.24.1_all.deb ...
Unpacking libdebhelper-perl (13.24.1) ...
Selecting previously unselected package libtool.
Preparing to unpack .../016-libtool_2.5.4-2_all.deb ...
Unpacking libtool (2.5.4-2) ...
Selecting previously unselected package dh-autoreconf.
Preparing to unpack .../017-dh-autoreconf_20_all.deb ...
Unpacking dh-autoreconf (20) ...
Selecting previously unselected package libarchive-zip-perl.
Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ...
Unpacking libarchive-zip-perl (1.68-1) ...
Selecting previously unselected package libfile-stripnondeterminism-perl.
Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.1-1_all.deb ...
Unpacking libfile-stripnondeterminism-perl (1.14.1-1) ...
Selecting previously unselected package dh-strip-nondeterminism.
Preparing to unpack .../020-dh-strip-nondeterminism_1.14.1-1_all.deb ...
Unpacking dh-strip-nondeterminism (1.14.1-1) ...
Selecting previously unselected package libelf1t64:arm64.
Preparing to unpack .../021-libelf1t64_0.192-4_arm64.deb ...
Unpacking libelf1t64:arm64 (0.192-4) ...
Selecting previously unselected package dwz.
Preparing to unpack .../022-dwz_0.15-1+b1_arm64.deb ...
Unpacking dwz (0.15-1+b1) ...
Selecting previously unselected package libunistring5:arm64.
Preparing to unpack .../023-libunistring5_1.3-1_arm64.deb ...
Unpacking libunistring5:arm64 (1.3-1) ...
Selecting previously unselected package libicu72:arm64.
Preparing to unpack .../024-libicu72_72.1-6_arm64.deb ...
Unpacking libicu72:arm64 (72.1-6) ...
Selecting previously unselected package libxml2:arm64.
Preparing to unpack .../025-libxml2_2.12.7+dfsg+really2.9.14-0.2+b1_arm64.deb ...
Unpacking libxml2:arm64 (2.12.7+dfsg+really2.9.14-0.2+b1) ...
Selecting previously unselected package gettext.
Preparing to unpack .../026-gettext_0.23.1-1_arm64.deb ...
Unpacking gettext (0.23.1-1) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../027-intltool-debian_0.35.0+20060710.6_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.6) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../028-po-debconf_1.0.21+nmu1_all.deb ...
Unpacking po-debconf (1.0.21+nmu1) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../029-debhelper_13.24.1_all.deb ...
Unpacking debhelper (13.24.1) ...
Selecting previously unselected package dh-golang.
Preparing to unpack .../030-dh-golang_1.62_all.deb ...
Unpacking dh-golang (1.62) ...
Selecting previously unselected package golang-1.23-src.
Preparing to unpack .../031-golang-1.23-src_1.23.5-1_all.deb ...
Unpacking golang-1.23-src (1.23.5-1) ...
Selecting previously unselected package golang-1.23-go.
Preparing to unpack .../032-golang-1.23-go_1.23.5-1_arm64.deb ...
Unpacking golang-1.23-go (1.23.5-1) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../033-golang-src_2%3a1.23~2_all.deb ...
Unpacking golang-src (2:1.23~2) ...
Selecting previously unselected package golang-go:arm64.
Preparing to unpack .../034-golang-go_2%3a1.23~2_arm64.deb ...
Unpacking golang-go:arm64 (2:1.23~2) ...
Selecting previously unselected package golang-any:arm64.
Preparing to unpack .../035-golang-any_2%3a1.23~2_arm64.deb ...
Unpacking golang-any:arm64 (2:1.23~2) ...
Selecting previously unselected package golang-github-alecthomas-units-dev.
Preparing to unpack .../036-golang-github-alecthomas-units-dev_0.0~git20240626.61d1e34-1_all.deb ...
Unpacking golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ...
Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev.
Preparing to unpack .../037-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ...
Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev.
Preparing to unpack .../038-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ...
Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Selecting previously unselected package golang-golang-x-sys-dev.
Preparing to unpack .../039-golang-golang-x-sys-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-sys-dev (0.22.0-1) ...
Selecting previously unselected package golang-github-mattn-go-isatty-dev.
Preparing to unpack .../040-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ...
Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Selecting previously unselected package golang-github-mattn-go-colorable-dev.
Preparing to unpack .../041-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ...
Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Selecting previously unselected package golang-github-fatih-color-dev.
Preparing to unpack .../042-golang-github-fatih-color-dev_1.17.0-1_all.deb ...
Unpacking golang-github-fatih-color-dev (1.17.0-1) ...
Selecting previously unselected package golang-github-davecgh-go-spew-dev.
Preparing to unpack .../043-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ...
Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Selecting previously unselected package golang-github-pmezard-go-difflib-dev.
Preparing to unpack .../044-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ...
Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Selecting previously unselected package golang-github-stretchr-objx-dev.
Preparing to unpack .../045-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ...
Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ...
Selecting previously unselected package golang-gopkg-yaml.v3-dev.
Preparing to unpack .../046-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ...
Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Selecting previously unselected package golang-github-stretchr-testify-dev.
Preparing to unpack .../047-golang-github-stretchr-testify-dev_1.9.0-1_all.deb ...
Unpacking golang-github-stretchr-testify-dev (1.9.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-hclog-dev.
Preparing to unpack .../048-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ...
Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev.
Preparing to unpack .../049-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ...
Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Selecting previously unselected package golang-github-pkg-errors-dev.
Preparing to unpack .../050-golang-github-pkg-errors-dev_0.9.1-3_all.deb ...
Unpacking golang-github-pkg-errors-dev (0.9.1-3) ...
Selecting previously unselected package golang-github-tv42-httpunix-dev.
Preparing to unpack .../051-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ...
Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev.
Preparing to unpack .../052-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ...
Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Selecting previously unselected package golang-github-datadog-datadog-go-dev.
Preparing to unpack .../053-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ...
Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Selecting previously unselected package golang-github-google-go-cmp-dev.
Preparing to unpack .../054-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ...
Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ...
Selecting previously unselected package golang-google-protobuf-dev.
Preparing to unpack .../055-golang-google-protobuf-dev_1.33.0-1_all.deb ...
Unpacking golang-google-protobuf-dev (1.33.0-1) ...
Selecting previously unselected package golang-github-golang-protobuf-1-5-dev.
Preparing to unpack .../056-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ...
Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Selecting previously unselected package golang-github-hashicorp-go-uuid-dev.
Preparing to unpack .../057-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ...
Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Selecting previously unselected package golang-github-hashicorp-golang-lru-dev.
Preparing to unpack .../058-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ...
Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev.
Preparing to unpack .../059-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ...
Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Selecting previously unselected package golang-github-pascaldekloe-goe-dev.
Preparing to unpack .../060-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ...
Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Selecting previously unselected package golang-github-beorn7-perks-dev.
Preparing to unpack .../061-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ...
Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ...
Selecting previously unselected package golang-github-cespare-xxhash-dev.
Preparing to unpack .../062-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ...
Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ...
Selecting previously unselected package golang-github-modern-go-concurrent-dev.
Preparing to unpack .../063-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ...
Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Selecting previously unselected package golang-github-modern-go-reflect2-dev.
Preparing to unpack .../064-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ...
Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-json-iterator-go-dev.
Preparing to unpack .../065-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ...
Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ...
Selecting previously unselected package golang-github-klauspost-compress-dev.
Preparing to unpack .../066-golang-github-klauspost-compress-dev_1.17.11+ds1-1_all.deb ...
Unpacking golang-github-klauspost-compress-dev (1.17.11+ds1-1) ...
Selecting previously unselected package golang-github-kylelemons-godebug-dev.
Preparing to unpack .../067-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ...
Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Selecting previously unselected package golang-github-prometheus-client-model-dev.
Preparing to unpack .../068-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ...
Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ...
Selecting previously unselected package golang-gopkg-yaml.v2-dev.
Preparing to unpack .../069-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ...
Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Selecting previously unselected package golang-github-jmespath-go-jmespath-dev.
Preparing to unpack .../070-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ...
Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Selecting previously unselected package golang-golang-x-term-dev.
Preparing to unpack .../071-golang-golang-x-term-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-term-dev (0.22.0-1) ...
Selecting previously unselected package golang-golang-x-text-dev.
Preparing to unpack .../072-golang-golang-x-text-dev_0.16.0-1_all.deb ...
Unpacking golang-golang-x-text-dev (0.16.0-1) ...
Selecting previously unselected package golang-golang-x-crypto-dev.
Preparing to unpack .../073-golang-golang-x-crypto-dev_1%3a0.25.0-1_all.deb ...
Unpacking golang-golang-x-crypto-dev (1:0.25.0-1) ...
Selecting previously unselected package golang-golang-x-net-dev.
Preparing to unpack .../074-golang-golang-x-net-dev_1%3a0.27.0-1_all.deb ...
Unpacking golang-golang-x-net-dev (1:0.27.0-1) ...
Selecting previously unselected package golang-github-aws-aws-sdk-go-dev.
Preparing to unpack .../075-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ...
Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Selecting previously unselected package golang-github-go-logfmt-logfmt-dev.
Preparing to unpack .../076-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ...
Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Selecting previously unselected package golang-github-go-kit-log-dev.
Preparing to unpack .../077-golang-github-go-kit-log-dev_0.2.1-1_all.deb ...
Unpacking golang-github-go-kit-log-dev (0.2.1-1) ...
Selecting previously unselected package golang-github-julienschmidt-httprouter-dev.
Preparing to unpack .../078-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ...
Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Selecting previously unselected package golang-github-munnerz-goautoneg-dev.
Preparing to unpack .../079-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ...
Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Selecting previously unselected package golang-github-jpillora-backoff-dev.
Preparing to unpack .../080-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ...
Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev.
Preparing to unpack .../081-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ...
Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Selecting previously unselected package golang-golang-x-oauth2-dev.
Preparing to unpack .../082-golang-golang-x-oauth2-dev_0.15.0-1_all.deb ...
Unpacking golang-golang-x-oauth2-dev (0.15.0-1) ...
Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev.
Preparing to unpack .../083-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ...
Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Selecting previously unselected package golang-github-prometheus-common-dev.
Preparing to unpack .../084-golang-github-prometheus-common-dev_0.61.0-1_all.deb ...
Unpacking golang-github-prometheus-common-dev (0.61.0-1) ...
Selecting previously unselected package golang-golang-x-sync-dev.
Preparing to unpack .../085-golang-golang-x-sync-dev_0.7.0-1_all.deb ...
Unpacking golang-golang-x-sync-dev (0.7.0-1) ...
Selecting previously unselected package golang-github-prometheus-procfs-dev.
Preparing to unpack .../086-golang-github-prometheus-procfs-dev_0.14.0-1_all.deb ...
Unpacking golang-github-prometheus-procfs-dev (0.14.0-1) ...
Selecting previously unselected package golang-github-prometheus-client-golang-dev.
Preparing to unpack .../087-golang-github-prometheus-client-golang-dev_1.20.5-1_all.deb ...
Unpacking golang-github-prometheus-client-golang-dev (1.20.5-1) ...
Selecting previously unselected package golang-github-armon-go-metrics-dev.
Preparing to unpack .../088-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ...
Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ...
Selecting previously unselected package golang-github-creack-pty-dev.
Preparing to unpack .../089-golang-github-creack-pty-dev_1.1.21-1_all.deb ...
Unpacking golang-github-creack-pty-dev (1.1.21-1) ...
Selecting previously unselected package golang-github-vmihailenco-tagparser-dev.
Preparing to unpack .../090-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ...
Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev.
Preparing to unpack .../091-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ...
Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Selecting previously unselected package golang-gopkg-tomb.v2-dev.
Preparing to unpack .../092-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ...
Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Selecting previously unselected package libsasl2-modules-db:arm64.
Preparing to unpack .../093-libsasl2-modules-db_2.1.28+dfsg1-8+b1_arm64.deb ...
Unpacking libsasl2-modules-db:arm64 (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package libsasl2-2:arm64.
Preparing to unpack .../094-libsasl2-2_2.1.28+dfsg1-8+b1_arm64.deb ...
Unpacking libsasl2-2:arm64 (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package libssl-dev:arm64.
Preparing to unpack .../095-libssl-dev_3.4.0-2_arm64.deb ...
Unpacking libssl-dev:arm64 (3.4.0-2) ...
Selecting previously unselected package libsasl2-dev.
Preparing to unpack .../096-libsasl2-dev_2.1.28+dfsg1-8+b1_arm64.deb ...
Unpacking libsasl2-dev (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package golang-github-kr-text-dev.
Preparing to unpack .../097-golang-github-kr-text-dev_0.2.0-2_all.deb ...
Unpacking golang-github-kr-text-dev (0.2.0-2) ...
Selecting previously unselected package golang-golang-x-mod-dev.
Preparing to unpack .../098-golang-golang-x-mod-dev_0.19.0-1_all.deb ...
Unpacking golang-golang-x-mod-dev (0.19.0-1) ...
Selecting previously unselected package golang-github-yuin-goldmark-dev.
Preparing to unpack .../099-golang-github-yuin-goldmark-dev_1.7.4-1_all.deb ...
Unpacking golang-github-yuin-goldmark-dev (1.7.4-1) ...
Selecting previously unselected package golang-golang-x-tools-dev.
Preparing to unpack .../100-golang-golang-x-tools-dev_1%3a0.23.0+ds-1_all.deb ...
Unpacking golang-golang-x-tools-dev (1:0.23.0+ds-1) ...
Selecting previously unselected package golang-github-rogpeppe-go-internal-dev.
Preparing to unpack .../101-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ...
Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Selecting previously unselected package golang-github-kr-pretty-dev.
Preparing to unpack .../102-golang-github-kr-pretty-dev_0.3.1-1_all.deb ...
Unpacking golang-github-kr-pretty-dev (0.3.1-1) ...
Selecting previously unselected package golang-gopkg-check.v1-dev.
Preparing to unpack .../103-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ...
Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Selecting previously unselected package golang-gopkg-mgo.v2-dev.
Preparing to unpack .../104-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ...
Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev.
Preparing to unpack .../105-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ...
Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up libpipeline1:arm64 (1.5.8-1) ...
Setting up golang-github-google-go-cmp-dev (0.6.0-1) ...
Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Setting up libicu72:arm64 (72.1-6) ...
Setting up bsdextrautils (2.40.4-1) ...
Setting up golang-github-creack-pty-dev (1.1.21-1) ...
Setting up golang-github-stretchr-objx-dev (0.5.2-1) ...
Setting up libmagic-mgc (1:5.45-3+b1) ...
Setting up golang-github-pkg-errors-dev (0.9.1-3) ...
Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Setting up libarchive-zip-perl (1.68-1) ...
Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Setting up libdebhelper-perl (13.24.1) ...
Setting up golang-golang-x-sys-dev (0.22.0-1) ...
Setting up libmagic1t64:arm64 (1:5.45-3+b1) ...
Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Setting up gettext-base (0.23.1-1) ...
Setting up m4 (1.4.19-5) ...
Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Setting up golang-golang-x-term-dev (0.22.0-1) ...
Setting up file (1:5.45-3+b1) ...
Setting up golang-github-beorn7-perks-dev (1.0.1-1) ...
Setting up libelf1t64:arm64 (0.192-4) ...
Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Setting up libsasl2-modules-db:arm64 (2.1.28+dfsg1-8+b1) ...
Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ...
Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Setting up autotools-dev (20220109.1) ...
Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Setting up golang-google-protobuf-dev (1.33.0-1) ...
Setting up golang-github-kr-text-dev (0.2.0-2) ...
Setting up golang-github-stretchr-testify-dev (1.9.0-1) ...
Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Setting up golang-github-yuin-goldmark-dev (1.7.4-1) ...
Setting up libunistring5:arm64 (1.3-1) ...
Setting up libssl-dev:arm64 (3.4.0-2) ...
Setting up golang-golang-x-sync-dev (0.7.0-1) ...
Setting up golang-golang-x-mod-dev (0.19.0-1) ...
Setting up golang-1.23-src (1.23.5-1) ...
Setting up autopoint (0.23.1-1) ...
Setting up golang-github-klauspost-compress-dev (1.17.11+ds1-1) ...
Setting up libsasl2-2:arm64 (2.1.28+dfsg1-8+b1) ...
Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Setting up autoconf (2.72-3) ...
Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Setting up dwz (0.15-1+b1) ...
Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Setting up sensible-utils (0.0.24) ...
Setting up golang-golang-x-text-dev (0.16.0-1) ...
Setting up libuchardet0:arm64 (0.0.8-1+b2) ...
Setting up golang-github-prometheus-procfs-dev (0.14.0-1) ...
Setting up golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ...
Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Setting up golang-golang-x-crypto-dev (1:0.25.0-1) ...
Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ...
Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Setting up golang-golang-x-oauth2-dev (0.15.0-1) ...
Setting up golang-src (2:1.23~2) ...
Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Setting up libxml2:arm64 (2.12.7+dfsg+really2.9.14-0.2+b1) ...
Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Setting up automake (1:1.16.5-1.3) ...
update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode
Setting up libfile-stripnondeterminism-perl (1.14.1-1) ...
Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Setting up golang-github-go-kit-log-dev (0.2.1-1) ...
Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Setting up gettext (0.23.1-1) ...
Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Setting up libtool (2.5.4-2) ...
Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Setting up golang-github-json-iterator-go-dev (1.1.12-2) ...
Setting up intltool-debian (0.35.0+20060710.6) ...
Setting up dh-autoreconf (20) ...
Setting up golang-1.23-go (1.23.5-1) ...
Setting up libsasl2-dev (2.1.28+dfsg1-8+b1) ...
Setting up golang-golang-x-net-dev (1:0.27.0-1) ...
Setting up dh-strip-nondeterminism (1.14.1-1) ...
Setting up groff-base (1.23.0-7) ...
Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Setting up golang-github-fatih-color-dev (1.17.0-1) ...
Setting up golang-go:arm64 (2:1.23~2) ...
Setting up po-debconf (1.0.21+nmu1) ...
Setting up golang-golang-x-tools-dev (1:0.23.0+ds-1) ...
Setting up man-db (2.13.0-1) ...
Not building database; man-db/auto-update is not 'true'.
Setting up golang-any:arm64 (2:1.23~2) ...
Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Setting up golang-github-kr-pretty-dev (0.3.1-1) ...
Setting up debhelper (13.24.1) ...
Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Setting up dh-golang (1.62) ...
Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Setting up golang-github-prometheus-common-dev (0.61.0-1) ...
Setting up golang-github-prometheus-client-golang-dev (1.20.5-1) ...
Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ...
Processing triggers for libc-bin (2.40-6) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Building tag database...
 -> Finished parsing the build-deps
I: Building the package
I: user script /srv/workspace/pbuilder/778150/tmp/hooks/A99_set_merged_usr starting
Not re-configuring usrmerge for unstable
I: user script /srv/workspace/pbuilder/778150/tmp/hooks/A99_set_merged_usr finished
hostname: Name or service not known
I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S  > ../golang-github-hashicorp-raft_1.5.0-1_source.changes
dpkg-buildpackage: info: source package golang-github-hashicorp-raft
dpkg-buildpackage: info: source version 1.5.0-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Daniel Swarbrick <dswarbrick@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture arm64
 debian/rules clean
dh clean --buildsystem=golang
   dh_auto_clean -O--buildsystem=golang
   dh_autoreconf_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules binary
dh binary --buildsystem=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-aarch64-linux-gnu && go install -trimpath -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench
internal/coverage/rtcov
internal/unsafeheader
internal/goarch
internal/byteorder
internal/godebugs
internal/goexperiment
internal/goos
internal/profilerecord
unicode
unicode/utf8
sync/atomic
internal/cpu
internal/race
internal/runtime/syscall
internal/abi
runtime/internal/math
internal/chacha8rand
container/list
runtime/internal/sys
internal/asan
internal/itoa
internal/msan
math/bits
crypto/internal/alias
crypto/subtle
crypto/internal/boring/sig
cmp
encoding
unicode/utf16
log/internal
internal/nettrace
vendor/golang.org/x/crypto/cryptobyte/asn1
vendor/golang.org/x/crypto/internal/alias
internal/bytealg
internal/runtime/atomic
math
internal/runtime/exithook
internal/stringslite
runtime
internal/reflectlite
sync
iter
internal/weak
maps
slices
internal/bisect
internal/testlog
internal/singleflight
runtime/cgo
internal/godebug
errors
sort
io
internal/oserror
strconv
path
math/rand
bytes
crypto/internal/edwards25519/field
vendor/golang.org/x/net/dns/dnsmessage
github.com/hashicorp/golang-lru/simplelru
crypto/internal/nistec/fiat
math/rand/v2
reflect
syscall
strings
crypto/internal/randutil
hash
crypto/rc4
internal/saferio
crypto
hash/crc32
crypto/internal/edwards25519
internal/concurrent
crypto/cipher
crypto/md5
vendor/golang.org/x/text/transform
unique
hash/crc64
bufio
github.com/hashicorp/go-immutable-radix
net/http/internal/ascii
crypto/internal/boring
crypto/des
regexp/syntax
crypto/aes
net/netip
crypto/sha512
crypto/hmac
crypto/sha1
vendor/golang.org/x/crypto/hkdf
crypto/sha256
html
regexp
time
internal/syscall/unix
internal/syscall/execenv
context
io/fs
internal/poll
internal/fmtsort
encoding/binary
internal/filepathlite
embed
crypto/internal/nistec
encoding/base64
vendor/golang.org/x/crypto/chacha20
vendor/golang.org/x/crypto/internal/poly1305
golang.org/x/sys/unix
os
encoding/pem
vendor/golang.org/x/crypto/chacha20poly1305
crypto/ecdh
fmt
vendor/golang.org/x/sys/cpu
io/ioutil
path/filepath
internal/sysinfo
os/signal
net
vendor/golang.org/x/crypto/sha3
math/big
encoding/json
log
compress/flate
encoding/hex
net/url
vendor/golang.org/x/text/unicode/norm
vendor/golang.org/x/net/http2/hpack
mime
mime/quotedprintable
net/http/internal
encoding/gob
vendor/golang.org/x/text/unicode/bidi
go/token
text/template/parse
flag
runtime/debug
runtime/trace
compress/gzip
github.com/mattn/go-isatty
vendor/golang.org/x/text/secure/bidirule
github.com/mattn/go-colorable
testing
github.com/fatih/color
text/template
github.com/hashicorp/go-hclog
vendor/golang.org/x/net/idna
crypto/rand
crypto/elliptic
crypto/internal/bigmod
crypto/internal/boring/bbig
crypto/dsa
encoding/asn1
crypto/ed25519
crypto/internal/hpke
crypto/internal/mlkem768
crypto/rsa
html/template
vendor/golang.org/x/crypto/cryptobyte
crypto/x509/pkix
crypto/ecdsa
crypto/x509
net/textproto
vendor/golang.org/x/net/http/httpproxy
vendor/golang.org/x/net/http/httpguts
mime/multipart
crypto/tls
net/http/httptrace
net/http
github.com/armon/go-metrics
net/rpc
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/raft
github.com/hashicorp/raft/bench
   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
dh_auto_test
	cd obj-aarch64-linux-gnu && go test -vet=off -v -p 12 github.com/hashicorp/raft github.com/hashicorp/raft/bench
?   	github.com/hashicorp/raft/bench	[no test files]
=== RUN   TestCommitment_setVoters
--- PASS: TestCommitment_setVoters (0.00s)
=== RUN   TestCommitment_match_max
--- PASS: TestCommitment_match_max (0.00s)
=== RUN   TestCommitment_match_nonVoting
--- PASS: TestCommitment_match_nonVoting (0.00s)
=== RUN   TestCommitment_recalculate
--- PASS: TestCommitment_recalculate (0.00s)
=== RUN   TestCommitment_recalculate_startIndex
--- PASS: TestCommitment_recalculate_startIndex (0.00s)
=== RUN   TestCommitment_noVoterSanity
--- PASS: TestCommitment_noVoterSanity (0.00s)
=== RUN   TestCommitment_singleVoter
--- PASS: TestCommitment_singleVoter (0.00s)
=== RUN   TestConfiguration_Configuration_Clone
--- PASS: TestConfiguration_Configuration_Clone (0.00s)
=== RUN   TestConfiguration_configurations_Clone
--- PASS: TestConfiguration_configurations_Clone (0.00s)
=== RUN   TestConfiguration_hasVote
--- PASS: TestConfiguration_hasVote (0.00s)
=== RUN   TestConfiguration_checkConfiguration
--- PASS: TestConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_nextConfiguration_table
--- PASS: TestConfiguration_nextConfiguration_table (0.00s)
=== RUN   TestConfiguration_nextConfiguration_prevIndex
--- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
=== RUN   TestConfiguration_nextConfiguration_checkConfiguration
--- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_encodeDecodePeers
--- PASS: TestConfiguration_encodeDecodePeers (0.00s)
=== RUN   TestConfiguration_encodeDecodeConfiguration
--- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
=== RUN   TestDiscardSnapshotStoreImpl
--- PASS: TestDiscardSnapshotStoreImpl (0.00s)
=== RUN   TestDiscardSnapshotSinkImpl
--- PASS: TestDiscardSnapshotSinkImpl (0.00s)
=== RUN   TestFileSnapshotStoreImpl
--- PASS: TestFileSnapshotStoreImpl (0.00s)
=== RUN   TestFileSnapshotSinkImpl
--- PASS: TestFileSnapshotSinkImpl (0.00s)
=== RUN   TestFileSS_CreateSnapshotMissingParentDir
2025-01-26T02:43:23.489Z [INFO]  creating new snapshot: path=/tmp/raft177942540/raft292656264/snapshots/3-10-1737859403489.tmp
--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s)
=== RUN   TestFileSS_CreateSnapshot
2025-01-26T02:43:23.489Z [INFO]  creating new snapshot: path=/tmp/raft1303429953/snapshots/3-10-1737859403489.tmp
2025-01-26T02:43:23.489Z [WARN]  found temporary snapshot: name=3-10-1737859403489.tmp
--- PASS: TestFileSS_CreateSnapshot (0.00s)
=== RUN   TestFileSS_CancelSnapshot
2025-01-26T02:43:23.490Z [INFO]  creating new snapshot: path=/tmp/raft1333469313/snapshots/3-10-1737859403490.tmp
--- PASS: TestFileSS_CancelSnapshot (0.00s)
=== RUN   TestFileSS_Retention
2025-01-26T02:43:23.490Z [INFO]  creating new snapshot: path=/tmp/raft4072315228/snapshots/3-10-1737859403490.tmp
2025-01-26T02:43:23.490Z [INFO]  creating new snapshot: path=/tmp/raft4072315228/snapshots/3-11-1737859403490.tmp
2025-01-26T02:43:23.490Z [INFO]  creating new snapshot: path=/tmp/raft4072315228/snapshots/3-12-1737859403490.tmp
2025-01-26T02:43:23.490Z [INFO]  reaping snapshot: path=/tmp/raft4072315228/snapshots/3-10-1737859403490
2025-01-26T02:43:23.490Z [INFO]  creating new snapshot: path=/tmp/raft4072315228/snapshots/3-13-1737859403490.tmp
2025-01-26T02:43:23.490Z [INFO]  reaping snapshot: path=/tmp/raft4072315228/snapshots/3-11-1737859403490
2025-01-26T02:43:23.490Z [INFO]  creating new snapshot: path=/tmp/raft4072315228/snapshots/3-14-1737859403490.tmp
2025-01-26T02:43:23.490Z [INFO]  reaping snapshot: path=/tmp/raft4072315228/snapshots/3-12-1737859403490
--- PASS: TestFileSS_Retention (0.00s)
=== RUN   TestFileSS_BadPerm
--- PASS: TestFileSS_BadPerm (0.00s)
=== RUN   TestFileSS_MissingParentDir
--- PASS: TestFileSS_MissingParentDir (0.00s)
=== RUN   TestFileSS_Ordering
2025-01-26T02:43:23.491Z [INFO]  creating new snapshot: path=/tmp/raft2682805471/snapshots/5-130350-1737859403491.tmp
2025-01-26T02:43:23.491Z [INFO]  creating new snapshot: path=/tmp/raft2682805471/snapshots/36-204917-1737859403491.tmp
--- PASS: TestFileSS_Ordering (0.01s)
=== RUN   TestDeferFutureSuccess
--- PASS: TestDeferFutureSuccess (0.00s)
=== RUN   TestDeferFutureError
--- PASS: TestDeferFutureError (0.00s)
=== RUN   TestDeferFutureConcurrent
--- PASS: TestDeferFutureConcurrent (0.00s)
=== RUN   TestInmemSnapshotStoreImpl
--- PASS: TestInmemSnapshotStoreImpl (0.00s)
=== RUN   TestInmemSnapshotSinkImpl
--- PASS: TestInmemSnapshotSinkImpl (0.00s)
=== RUN   TestInmemSS_CreateSnapshot
--- PASS: TestInmemSS_CreateSnapshot (0.00s)
=== RUN   TestInmemSS_OpenSnapshotTwice
--- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
=== RUN   TestInmemTransportImpl
--- PASS: TestInmemTransportImpl (0.00s)
=== RUN   TestInmemTransportWriteTimeout
--- PASS: TestInmemTransportWriteTimeout (0.01s)
=== RUN   TestRaft_Integ
--- SKIP: TestRaft_Integ (0.00s)
=== RUN   TestRaft_RestartFollower_LongInitialHeartbeat
--- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s)
=== RUN   TestLogCache
--- PASS: TestLogCache (0.00s)
=== RUN   TestLogCacheWithBackendStoreError
--- PASS: TestLogCacheWithBackendStoreError (0.00s)
=== RUN   TestOldestLog
=== RUN   TestOldestLog/empty_logs
=== RUN   TestOldestLog/simple_case
--- PASS: TestOldestLog (0.00s)
    --- PASS: TestOldestLog/empty_logs (0.00s)
    --- PASS: TestOldestLog/simple_case (0.00s)
=== RUN   TestEmitsLogStoreMetrics
--- PASS: TestEmitsLogStoreMetrics (0.01s)
=== RUN   TestNetworkTransport_CloseStreams
--- PASS: TestNetworkTransport_CloseStreams (0.04s)
=== RUN   TestNetworkTransport_StartStop
--- PASS: TestNetworkTransport_StartStop (0.00s)
=== RUN   TestNetworkTransport_Heartbeat_FastPath
--- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s)
=== RUN   TestNetworkTransport_AppendEntries
--- PASS: TestNetworkTransport_AppendEntries (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline
--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.03s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_CloseStreams
--- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.05s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2
2025-01-26T02:43:23.715Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10
2025-01-26T02:43:23.771Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.18s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.06s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.06s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s)
=== RUN   TestNetworkTransport_RequestVote
2025-01-26T02:43:23.824Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_RequestVote (0.00s)
=== RUN   TestNetworkTransport_InstallSnapshot
--- PASS: TestNetworkTransport_InstallSnapshot (0.01s)
=== RUN   TestNetworkTransport_EncodeDecode
--- PASS: TestNetworkTransport_EncodeDecode (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode_AddressProvider
--- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
=== RUN   TestNetworkTransport_PooledConn
--- PASS: TestNetworkTransport_PooledConn (0.02s)
=== RUN   TestNetworkTransport_ListenBackoff
    net_transport_test.go:839: countingWriter: 2025-01-26T02:43:23.859Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-26T02:43:23.867Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-26T02:43:23.879Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-26T02:43:23.903Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-26T02:43:23.947Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-26T02:43:24.031Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-26T02:43:24.195Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-26T02:43:24.515Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-26T02:43:25.159Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-26T02:43:26.163Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-26T02:43:27.163Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
--- PASS: TestNetworkTransport_ListenBackoff (4.00s)
=== RUN   TestPeersJSON_BadConfiguration
--- PASS: TestPeersJSON_BadConfiguration (0.00s)
=== RUN   TestPeersJSON_ReadPeersJSON
--- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
=== RUN   TestPeersJSON_ReadConfigJSON
--- PASS: TestPeersJSON_ReadConfigJSON (0.00s)
=== RUN   TestRaft_StartStop
2025-01-26T02:43:27.864Z [INFO]  server-0ac3c634-d0f3-1291-75aa-5c98b362d4d6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0ac3c634-d0f3-1291-75aa-5c98b362d4d6 Address:0ac3c634-d0f3-1291-75aa-5c98b362d4d6}]"
--- PASS: TestRaft_StartStop (0.00s)
=== RUN   TestRaft_AfterShutdown
2025-01-26T02:43:27.864Z [INFO]  server-e9fe4733-8b52-ffcc-f562-30480102ecaa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e9fe4733-8b52-ffcc-f562-30480102ecaa Address:e9fe4733-8b52-ffcc-f562-30480102ecaa}]"
--- PASS: TestRaft_AfterShutdown (0.00s)
=== RUN   TestRaft_LiveBootstrap
2025-01-26T02:43:27.864Z [INFO]  server-ce259e46-9aa6-ad0d-4db0-7e7d5d2c61f7: initial configuration: index=0 servers=[]
2025-01-26T02:43:27.864Z [INFO]  server-455096ae-753b-ee4e-d59f-c337e4fd6acd: initial configuration: index=0 servers=[]
2025-01-26T02:43:27.864Z [INFO]  server-5a66b044-7a19-910a-fa79-6bb3b7266762: initial configuration: index=0 servers=[]
2025-01-26T02:43:27.864Z [INFO]  server-ce259e46-9aa6-ad0d-4db0-7e7d5d2c61f7: entering follower state: follower="Node at ce259e46-9aa6-ad0d-4db0-7e7d5d2c61f7 [Follower]" leader-address= leader-id=
2025-01-26T02:43:27.864Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:43:27.864Z [INFO]  server-455096ae-753b-ee4e-d59f-c337e4fd6acd: entering follower state: follower="Node at 455096ae-753b-ee4e-d59f-c337e4fd6acd [Follower]" leader-address= leader-id=
2025-01-26T02:43:27.864Z [INFO]  server-5a66b044-7a19-910a-fa79-6bb3b7266762: entering follower state: follower="Node at 5a66b044-7a19-910a-fa79-6bb3b7266762 [Follower]" leader-address= leader-id=
2025-01-26T02:43:27.915Z [WARN]  server-ce259e46-9aa6-ad0d-4db0-7e7d5d2c61f7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:27.915Z [INFO]  server-ce259e46-9aa6-ad0d-4db0-7e7d5d2c61f7: entering candidate state: node="Node at ce259e46-9aa6-ad0d-4db0-7e7d5d2c61f7 [Candidate]" term=2
2025-01-26T02:43:27.915Z [INFO]  server-ce259e46-9aa6-ad0d-4db0-7e7d5d2c61f7: election won: term=2 tally=2
2025-01-26T02:43:27.915Z [INFO]  server-ce259e46-9aa6-ad0d-4db0-7e7d5d2c61f7: entering leader state: leader="Node at ce259e46-9aa6-ad0d-4db0-7e7d5d2c61f7 [Leader]"
2025-01-26T02:43:27.915Z [INFO]  server-ce259e46-9aa6-ad0d-4db0-7e7d5d2c61f7: added peer, starting replication: peer=server-455096ae-753b-ee4e-d59f-c337e4fd6acd
2025-01-26T02:43:27.915Z [INFO]  server-ce259e46-9aa6-ad0d-4db0-7e7d5d2c61f7: added peer, starting replication: peer=server-5a66b044-7a19-910a-fa79-6bb3b7266762
2025-01-26T02:43:27.915Z [WARN]  server-5a66b044-7a19-910a-fa79-6bb3b7266762: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-01-26T02:43:27.915Z [WARN]  server-ce259e46-9aa6-ad0d-4db0-7e7d5d2c61f7: appendEntries rejected, sending older logs: peer="{Voter server-5a66b044-7a19-910a-fa79-6bb3b7266762 5a66b044-7a19-910a-fa79-6bb3b7266762}" next=1
2025-01-26T02:43:27.915Z [INFO]  server-ce259e46-9aa6-ad0d-4db0-7e7d5d2c61f7: pipelining replication: peer="{Voter server-5a66b044-7a19-910a-fa79-6bb3b7266762 5a66b044-7a19-910a-fa79-6bb3b7266762}"
2025-01-26T02:43:27.915Z [WARN]  server-455096ae-753b-ee4e-d59f-c337e4fd6acd: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-01-26T02:43:27.915Z [WARN]  server-ce259e46-9aa6-ad0d-4db0-7e7d5d2c61f7: appendEntries rejected, sending older logs: peer="{Voter server-455096ae-753b-ee4e-d59f-c337e4fd6acd 455096ae-753b-ee4e-d59f-c337e4fd6acd}" next=1
2025-01-26T02:43:27.915Z [INFO]  server-ce259e46-9aa6-ad0d-4db0-7e7d5d2c61f7: pipelining replication: peer="{Voter server-455096ae-753b-ee4e-d59f-c337e4fd6acd 455096ae-753b-ee4e-d59f-c337e4fd6acd}"
2025-01-26T02:43:28.022Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:43:27.915792952 +0000 UTC m=+4.430825327 (2 nodes), 51.198311ms from start of poll, 51.342472ms from cluster start. Timeout at 2025-01-26 02:43:28.020794192 +0000 UTC m=+4.535826527, 105.0012ms after stability
2025-01-26T02:43:28.022Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:28.135Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:28.022893122 +0000 UTC m=+4.537925497 (1 nodes), 880ns from start of poll, 158.442642ms from cluster start. Timeout at 2025-01-26 02:43:28.127896442 +0000 UTC m=+4.642928777, 105.00328ms after stability
2025-01-26T02:43:28.155Z [INFO]  server-ce259e46-9aa6-ad0d-4db0-7e7d5d2c61f7: aborting pipeline replication: peer="{Voter server-5a66b044-7a19-910a-fa79-6bb3b7266762 5a66b044-7a19-910a-fa79-6bb3b7266762}"
2025-01-26T02:43:28.155Z [INFO]  server-ce259e46-9aa6-ad0d-4db0-7e7d5d2c61f7: aborting pipeline replication: peer="{Voter server-455096ae-753b-ee4e-d59f-c337e4fd6acd 455096ae-753b-ee4e-d59f-c337e4fd6acd}"
--- PASS: TestRaft_LiveBootstrap (0.29s)
=== RUN   TestRaft_LiveBootstrap_From_NonVoter
2025-01-26T02:43:28.155Z [INFO]  server-d5a1ef11-e7e5-e8e5-ff6c-5df619d4549b: initial configuration: index=0 servers=[]
2025-01-26T02:43:28.155Z [INFO]  server-bafc766e-26cb-4e4b-7413-b31c676bd61f: initial configuration: index=0 servers=[]
2025-01-26T02:43:28.155Z [INFO]  server-d5a1ef11-e7e5-e8e5-ff6c-5df619d4549b: entering follower state: follower="Node at d5a1ef11-e7e5-e8e5-ff6c-5df619d4549b [Follower]" leader-address= leader-id=
--- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s)
=== RUN   TestRaft_RecoverCluster_NoState
2025-01-26T02:43:28.156Z [INFO]  server-f306a904-79b9-7993-1df0-b4140a84a948: initial configuration: index=0 servers=[]
--- PASS: TestRaft_RecoverCluster_NoState (0.00s)
=== RUN   TestRaft_RecoverCluster
=== RUN   TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs
2025-01-26T02:43:28.156Z [INFO]  server-3bdb34ae-1f51-620a-968b-ef70c832ae28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3bdb34ae-1f51-620a-968b-ef70c832ae28 Address:3bdb34ae-1f51-620a-968b-ef70c832ae28} {Suffrage:Voter ID:server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56 Address:7a0486a2-0fcb-38c6-826b-cca85c7c0e56} {Suffrage:Voter ID:server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f Address:ec8efc49-f06a-11fc-3d0e-1597ff0dee8f}]"
2025-01-26T02:43:28.156Z [INFO]  server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3bdb34ae-1f51-620a-968b-ef70c832ae28 Address:3bdb34ae-1f51-620a-968b-ef70c832ae28} {Suffrage:Voter ID:server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56 Address:7a0486a2-0fcb-38c6-826b-cca85c7c0e56} {Suffrage:Voter ID:server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f Address:ec8efc49-f06a-11fc-3d0e-1597ff0dee8f}]"
2025-01-26T02:43:28.156Z [INFO]  server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3bdb34ae-1f51-620a-968b-ef70c832ae28 Address:3bdb34ae-1f51-620a-968b-ef70c832ae28} {Suffrage:Voter ID:server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56 Address:7a0486a2-0fcb-38c6-826b-cca85c7c0e56} {Suffrage:Voter ID:server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f Address:ec8efc49-f06a-11fc-3d0e-1597ff0dee8f}]"
2025-01-26T02:43:28.156Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:28.156Z [INFO]  server-3bdb34ae-1f51-620a-968b-ef70c832ae28: entering follower state: follower="Node at 3bdb34ae-1f51-620a-968b-ef70c832ae28 [Follower]" leader-address= leader-id=
2025-01-26T02:43:28.156Z [INFO]  server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56: entering follower state: follower="Node at 7a0486a2-0fcb-38c6-826b-cca85c7c0e56 [Follower]" leader-address= leader-id=
2025-01-26T02:43:28.156Z [INFO]  server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: entering follower state: follower="Node at ec8efc49-f06a-11fc-3d0e-1597ff0dee8f [Follower]" leader-address= leader-id=
2025-01-26T02:43:28.219Z [WARN]  server-3bdb34ae-1f51-620a-968b-ef70c832ae28: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:28.219Z [INFO]  server-3bdb34ae-1f51-620a-968b-ef70c832ae28: entering candidate state: node="Node at 3bdb34ae-1f51-620a-968b-ef70c832ae28 [Candidate]" term=2
2025-01-26T02:43:28.219Z [INFO]  server-3bdb34ae-1f51-620a-968b-ef70c832ae28: election won: term=2 tally=2
2025-01-26T02:43:28.219Z [INFO]  server-3bdb34ae-1f51-620a-968b-ef70c832ae28: entering leader state: leader="Node at 3bdb34ae-1f51-620a-968b-ef70c832ae28 [Leader]"
2025-01-26T02:43:28.219Z [INFO]  server-3bdb34ae-1f51-620a-968b-ef70c832ae28: added peer, starting replication: peer=server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56
2025-01-26T02:43:28.219Z [INFO]  server-3bdb34ae-1f51-620a-968b-ef70c832ae28: added peer, starting replication: peer=server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f
2025-01-26T02:43:28.219Z [INFO]  server-3bdb34ae-1f51-620a-968b-ef70c832ae28: pipelining replication: peer="{Voter server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f ec8efc49-f06a-11fc-3d0e-1597ff0dee8f}"
2025-01-26T02:43:28.219Z [INFO]  server-3bdb34ae-1f51-620a-968b-ef70c832ae28: pipelining replication: peer="{Voter server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56 7a0486a2-0fcb-38c6-826b-cca85c7c0e56}"
2025-01-26T02:43:28.327Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:28.219724916 +0000 UTC m=+4.734757251 (1 nodes), 63.334664ms from start of poll, 63.504145ms from cluster start. Timeout at 2025-01-26 02:43:28.324727276 +0000 UTC m=+4.839759651, 105.0024ms after stability
2025-01-26T02:43:28.327Z [INFO]  server-3bdb34ae-1f51-620a-968b-ef70c832ae28: aborting pipeline replication: peer="{Voter server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f ec8efc49-f06a-11fc-3d0e-1597ff0dee8f}"
2025-01-26T02:43:28.327Z [INFO]  server-3bdb34ae-1f51-620a-968b-ef70c832ae28: aborting pipeline replication: peer="{Voter server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56 7a0486a2-0fcb-38c6-826b-cca85c7c0e56}"
2025-01-26T02:43:28.327Z [INFO]  creating new snapshot: path=/tmp/raft2816165043/snapshots/2-2-1737859408327.tmp
2025-01-26T02:43:28.327Z [INFO]  server-3bdb34ae-1f51-620a-968b-ef70c832ae28: starting restore from snapshot: id=2-2-1737859408327 last-index=2 last-term=2 size-in-bytes=1
2025-01-26T02:43:28.328Z [INFO]  server-3bdb34ae-1f51-620a-968b-ef70c832ae28: snapshot restore progress: id=2-2-1737859408327 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-01-26T02:43:28.328Z [INFO]  server-3bdb34ae-1f51-620a-968b-ef70c832ae28: restored from snapshot: id=2-2-1737859408327 last-index=2 last-term=2 size-in-bytes=1
2025-01-26T02:43:28.328Z [INFO]  server-3bdb34ae-1f51-620a-968b-ef70c832ae28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3bdb34ae-1f51-620a-968b-ef70c832ae28 Address:3bdb34ae-1f51-620a-968b-ef70c832ae28} {Suffrage:Voter ID:server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56 Address:7a0486a2-0fcb-38c6-826b-cca85c7c0e56} {Suffrage:Voter ID:server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f Address:ec8efc49-f06a-11fc-3d0e-1597ff0dee8f}]"
2025-01-26T02:43:28.328Z [INFO]  creating new snapshot: path=/tmp/raft1754821931/snapshots/2-2-1737859408328.tmp
2025-01-26T02:43:28.328Z [INFO]  server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56: starting restore from snapshot: id=2-2-1737859408328 last-index=2 last-term=2 size-in-bytes=1
2025-01-26T02:43:28.328Z [INFO]  server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56: snapshot restore progress: id=2-2-1737859408328 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-01-26T02:43:28.328Z [INFO]  server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56: restored from snapshot: id=2-2-1737859408328 last-index=2 last-term=2 size-in-bytes=1
2025-01-26T02:43:28.328Z [INFO]  server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3bdb34ae-1f51-620a-968b-ef70c832ae28 Address:3bdb34ae-1f51-620a-968b-ef70c832ae28} {Suffrage:Voter ID:server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56 Address:7a0486a2-0fcb-38c6-826b-cca85c7c0e56} {Suffrage:Voter ID:server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f Address:ec8efc49-f06a-11fc-3d0e-1597ff0dee8f}]"
2025-01-26T02:43:28.328Z [INFO]  creating new snapshot: path=/tmp/raft1653874475/snapshots/2-2-1737859408328.tmp
2025-01-26T02:43:28.328Z [INFO]  server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: starting restore from snapshot: id=2-2-1737859408328 last-index=2 last-term=2 size-in-bytes=1
2025-01-26T02:43:28.328Z [INFO]  server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: snapshot restore progress: id=2-2-1737859408328 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-01-26T02:43:28.328Z [INFO]  server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: restored from snapshot: id=2-2-1737859408328 last-index=2 last-term=2 size-in-bytes=1
2025-01-26T02:43:28.328Z [INFO]  server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3bdb34ae-1f51-620a-968b-ef70c832ae28 Address:3bdb34ae-1f51-620a-968b-ef70c832ae28} {Suffrage:Voter ID:server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56 Address:7a0486a2-0fcb-38c6-826b-cca85c7c0e56} {Suffrage:Voter ID:server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f Address:ec8efc49-f06a-11fc-3d0e-1597ff0dee8f}]"
2025-01-26T02:43:28.328Z [INFO]  server-3bdb34ae-1f51-620a-968b-ef70c832ae28: entering follower state: follower="Node at 3bdb34ae-1f51-620a-968b-ef70c832ae28 [Follower]" leader-address= leader-id=
2025-01-26T02:43:28.328Z [INFO]  server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56: entering follower state: follower="Node at 7a0486a2-0fcb-38c6-826b-cca85c7c0e56 [Follower]" leader-address= leader-id=
2025-01-26T02:43:28.328Z [INFO]  server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: entering follower state: follower="Node at ec8efc49-f06a-11fc-3d0e-1597ff0dee8f [Follower]" leader-address= leader-id=
2025-01-26T02:43:28.399Z [WARN]  server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:28.399Z [INFO]  server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: entering candidate state: node="Node at ec8efc49-f06a-11fc-3d0e-1597ff0dee8f [Candidate]" term=3
2025-01-26T02:43:28.399Z [INFO]  server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: election won: term=3 tally=2
2025-01-26T02:43:28.399Z [INFO]  server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: entering leader state: leader="Node at ec8efc49-f06a-11fc-3d0e-1597ff0dee8f [Leader]"
2025-01-26T02:43:28.399Z [INFO]  server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: added peer, starting replication: peer=server-3bdb34ae-1f51-620a-968b-ef70c832ae28
2025-01-26T02:43:28.399Z [INFO]  server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: added peer, starting replication: peer=server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56
2025-01-26T02:43:28.399Z [INFO]  server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: pipelining replication: peer="{Voter server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56 7a0486a2-0fcb-38c6-826b-cca85c7c0e56}"
2025-01-26T02:43:28.399Z [INFO]  server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: pipelining replication: peer="{Voter server-3bdb34ae-1f51-620a-968b-ef70c832ae28 3bdb34ae-1f51-620a-968b-ef70c832ae28}"
2025-01-26T02:43:28.647Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:28.755Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:28.647568472 +0000 UTC m=+5.162600847 (1 nodes), 18.6µs from start of poll, 491.347741ms from cluster start. Timeout at 2025-01-26 02:43:28.752572472 +0000 UTC m=+5.267604807, 105.00396ms after stability
2025-01-26T02:43:28.755Z [INFO]  server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: aborting pipeline replication: peer="{Voter server-7a0486a2-0fcb-38c6-826b-cca85c7c0e56 7a0486a2-0fcb-38c6-826b-cca85c7c0e56}"
2025-01-26T02:43:28.755Z [INFO]  server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: aborting pipeline replication: peer="{Voter server-3bdb34ae-1f51-620a-968b-ef70c832ae28 3bdb34ae-1f51-620a-968b-ef70c832ae28}"
2025-01-26T02:43:29.259Z [ERROR] server-ec8efc49-f06a-11fc-3d0e-1597ff0dee8f: failed to heartbeat to: peer=3bdb34ae-1f51-620a-968b-ef70c832ae28 backoff time=10ms error="command timed out"
=== RUN   TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs
2025-01-26T02:43:29.259Z [INFO]  server-0b003ae6-76bb-258c-7524-c58b6b2f2171: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b003ae6-76bb-258c-7524-c58b6b2f2171 Address:0b003ae6-76bb-258c-7524-c58b6b2f2171} {Suffrage:Voter ID:server-1019e28c-99c8-e734-5856-5bd28f67584f Address:1019e28c-99c8-e734-5856-5bd28f67584f} {Suffrage:Voter ID:server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa Address:785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa}]"
2025-01-26T02:43:29.260Z [INFO]  server-1019e28c-99c8-e734-5856-5bd28f67584f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b003ae6-76bb-258c-7524-c58b6b2f2171 Address:0b003ae6-76bb-258c-7524-c58b6b2f2171} {Suffrage:Voter ID:server-1019e28c-99c8-e734-5856-5bd28f67584f Address:1019e28c-99c8-e734-5856-5bd28f67584f} {Suffrage:Voter ID:server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa Address:785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa}]"
2025-01-26T02:43:29.260Z [INFO]  server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b003ae6-76bb-258c-7524-c58b6b2f2171 Address:0b003ae6-76bb-258c-7524-c58b6b2f2171} {Suffrage:Voter ID:server-1019e28c-99c8-e734-5856-5bd28f67584f Address:1019e28c-99c8-e734-5856-5bd28f67584f} {Suffrage:Voter ID:server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa Address:785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa}]"
2025-01-26T02:43:29.260Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:29.260Z [INFO]  server-0b003ae6-76bb-258c-7524-c58b6b2f2171: entering follower state: follower="Node at 0b003ae6-76bb-258c-7524-c58b6b2f2171 [Follower]" leader-address= leader-id=
2025-01-26T02:43:29.260Z [INFO]  server-1019e28c-99c8-e734-5856-5bd28f67584f: entering follower state: follower="Node at 1019e28c-99c8-e734-5856-5bd28f67584f [Follower]" leader-address= leader-id=
2025-01-26T02:43:29.260Z [INFO]  server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa: entering follower state: follower="Node at 785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa [Follower]" leader-address= leader-id=
2025-01-26T02:43:29.339Z [WARN]  server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:29.339Z [INFO]  server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa: entering candidate state: node="Node at 785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa [Candidate]" term=2
2025-01-26T02:43:29.339Z [INFO]  server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa: election won: term=2 tally=2
2025-01-26T02:43:29.339Z [INFO]  server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa: entering leader state: leader="Node at 785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa [Leader]"
2025-01-26T02:43:29.339Z [INFO]  server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa: added peer, starting replication: peer=server-0b003ae6-76bb-258c-7524-c58b6b2f2171
2025-01-26T02:43:29.339Z [INFO]  server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa: added peer, starting replication: peer=server-1019e28c-99c8-e734-5856-5bd28f67584f
2025-01-26T02:43:29.339Z [INFO]  server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa: pipelining replication: peer="{Voter server-1019e28c-99c8-e734-5856-5bd28f67584f 1019e28c-99c8-e734-5856-5bd28f67584f}"
2025-01-26T02:43:29.339Z [INFO]  server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa: pipelining replication: peer="{Voter server-0b003ae6-76bb-258c-7524-c58b6b2f2171 0b003ae6-76bb-258c-7524-c58b6b2f2171}"
2025-01-26T02:43:29.447Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:29.339747672 +0000 UTC m=+5.854780047 (1 nodes), 79.643643ms from start of poll, 79.858724ms from cluster start. Timeout at 2025-01-26 02:43:29.444751312 +0000 UTC m=+5.959783647, 105.0036ms after stability
2025-01-26T02:43:29.447Z [INFO]  server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa: aborting pipeline replication: peer="{Voter server-1019e28c-99c8-e734-5856-5bd28f67584f 1019e28c-99c8-e734-5856-5bd28f67584f}"
2025-01-26T02:43:29.447Z [INFO]  server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa: aborting pipeline replication: peer="{Voter server-0b003ae6-76bb-258c-7524-c58b6b2f2171 0b003ae6-76bb-258c-7524-c58b6b2f2171}"
2025-01-26T02:43:29.447Z [INFO]  creating new snapshot: path=/tmp/raft1925790344/snapshots/2-6-1737859409447.tmp
2025-01-26T02:43:29.448Z [INFO]  server-0b003ae6-76bb-258c-7524-c58b6b2f2171: starting restore from snapshot: id=2-6-1737859409447 last-index=6 last-term=2 size-in-bytes=25
2025-01-26T02:43:29.448Z [INFO]  server-0b003ae6-76bb-258c-7524-c58b6b2f2171: snapshot restore progress: id=2-6-1737859409447 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-01-26T02:43:29.448Z [INFO]  server-0b003ae6-76bb-258c-7524-c58b6b2f2171: restored from snapshot: id=2-6-1737859409447 last-index=6 last-term=2 size-in-bytes=25
2025-01-26T02:43:29.448Z [INFO]  server-0b003ae6-76bb-258c-7524-c58b6b2f2171: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b003ae6-76bb-258c-7524-c58b6b2f2171 Address:0b003ae6-76bb-258c-7524-c58b6b2f2171} {Suffrage:Voter ID:server-1019e28c-99c8-e734-5856-5bd28f67584f Address:1019e28c-99c8-e734-5856-5bd28f67584f} {Suffrage:Voter ID:server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa Address:785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa}]"
2025-01-26T02:43:29.448Z [INFO]  creating new snapshot: path=/tmp/raft4246276009/snapshots/2-6-1737859409448.tmp
2025-01-26T02:43:29.448Z [INFO]  server-1019e28c-99c8-e734-5856-5bd28f67584f: starting restore from snapshot: id=2-6-1737859409448 last-index=6 last-term=2 size-in-bytes=25
2025-01-26T02:43:29.448Z [INFO]  server-1019e28c-99c8-e734-5856-5bd28f67584f: snapshot restore progress: id=2-6-1737859409448 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-01-26T02:43:29.448Z [INFO]  server-1019e28c-99c8-e734-5856-5bd28f67584f: restored from snapshot: id=2-6-1737859409448 last-index=6 last-term=2 size-in-bytes=25
2025-01-26T02:43:29.448Z [INFO]  server-1019e28c-99c8-e734-5856-5bd28f67584f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b003ae6-76bb-258c-7524-c58b6b2f2171 Address:0b003ae6-76bb-258c-7524-c58b6b2f2171} {Suffrage:Voter ID:server-1019e28c-99c8-e734-5856-5bd28f67584f Address:1019e28c-99c8-e734-5856-5bd28f67584f} {Suffrage:Voter ID:server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa Address:785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa}]"
2025-01-26T02:43:29.448Z [INFO]  creating new snapshot: path=/tmp/raft3889772594/snapshots/2-6-1737859409448.tmp
2025-01-26T02:43:29.448Z [INFO]  server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa: starting restore from snapshot: id=2-6-1737859409448 last-index=6 last-term=2 size-in-bytes=25
2025-01-26T02:43:29.448Z [INFO]  server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa: snapshot restore progress: id=2-6-1737859409448 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-01-26T02:43:29.448Z [INFO]  server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa: restored from snapshot: id=2-6-1737859409448 last-index=6 last-term=2 size-in-bytes=25
2025-01-26T02:43:29.448Z [INFO]  server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b003ae6-76bb-258c-7524-c58b6b2f2171 Address:0b003ae6-76bb-258c-7524-c58b6b2f2171} {Suffrage:Voter ID:server-1019e28c-99c8-e734-5856-5bd28f67584f Address:1019e28c-99c8-e734-5856-5bd28f67584f} {Suffrage:Voter ID:server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa Address:785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa}]"
2025-01-26T02:43:29.448Z [INFO]  server-0b003ae6-76bb-258c-7524-c58b6b2f2171: entering follower state: follower="Node at 0b003ae6-76bb-258c-7524-c58b6b2f2171 [Follower]" leader-address= leader-id=
2025-01-26T02:43:29.449Z [INFO]  server-1019e28c-99c8-e734-5856-5bd28f67584f: entering follower state: follower="Node at 1019e28c-99c8-e734-5856-5bd28f67584f [Follower]" leader-address= leader-id=
2025-01-26T02:43:29.449Z [INFO]  server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa: entering follower state: follower="Node at 785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa [Follower]" leader-address= leader-id=
2025-01-26T02:43:29.511Z [WARN]  server-0b003ae6-76bb-258c-7524-c58b6b2f2171: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:29.511Z [INFO]  server-0b003ae6-76bb-258c-7524-c58b6b2f2171: entering candidate state: node="Node at 0b003ae6-76bb-258c-7524-c58b6b2f2171 [Candidate]" term=3
2025-01-26T02:43:29.511Z [INFO]  server-0b003ae6-76bb-258c-7524-c58b6b2f2171: election won: term=3 tally=2
2025-01-26T02:43:29.511Z [INFO]  server-0b003ae6-76bb-258c-7524-c58b6b2f2171: entering leader state: leader="Node at 0b003ae6-76bb-258c-7524-c58b6b2f2171 [Leader]"
2025-01-26T02:43:29.511Z [INFO]  server-0b003ae6-76bb-258c-7524-c58b6b2f2171: added peer, starting replication: peer=server-1019e28c-99c8-e734-5856-5bd28f67584f
2025-01-26T02:43:29.511Z [INFO]  server-0b003ae6-76bb-258c-7524-c58b6b2f2171: added peer, starting replication: peer=server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa
2025-01-26T02:43:29.511Z [INFO]  server-0b003ae6-76bb-258c-7524-c58b6b2f2171: pipelining replication: peer="{Voter server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa 785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa}"
2025-01-26T02:43:29.512Z [INFO]  server-0b003ae6-76bb-258c-7524-c58b6b2f2171: pipelining replication: peer="{Voter server-1019e28c-99c8-e734-5856-5bd28f67584f 1019e28c-99c8-e734-5856-5bd28f67584f}"
2025-01-26T02:43:29.767Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:29.875Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:29.767589868 +0000 UTC m=+6.282622243 (1 nodes), 21.32µs from start of poll, 507.70092ms from cluster start. Timeout at 2025-01-26 02:43:29.872592588 +0000 UTC m=+6.387624923, 105.00268ms after stability
2025-01-26T02:43:29.875Z [INFO]  server-0b003ae6-76bb-258c-7524-c58b6b2f2171: aborting pipeline replication: peer="{Voter server-785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa 785ae8e9-1ff9-aa1a-eb3e-3466e4cca1aa}"
2025-01-26T02:43:29.875Z [INFO]  server-0b003ae6-76bb-258c-7524-c58b6b2f2171: aborting pipeline replication: peer="{Voter server-1019e28c-99c8-e734-5856-5bd28f67584f 1019e28c-99c8-e734-5856-5bd28f67584f}"
=== RUN   TestRaft_RecoverCluster/snapshot,_with_trailing_logs
2025-01-26T02:43:29.876Z [INFO]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83a654a4-cf14-d8d5-9d21-7acbe86d4444 Address:83a654a4-cf14-d8d5-9d21-7acbe86d4444} {Suffrage:Voter ID:server-8bb1362d-8c13-5989-4193-42ceac8986ec Address:8bb1362d-8c13-5989-4193-42ceac8986ec} {Suffrage:Voter ID:server-73ca3b53-62c0-a512-0a03-255f1dcd450b Address:73ca3b53-62c0-a512-0a03-255f1dcd450b}]"
2025-01-26T02:43:29.876Z [INFO]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83a654a4-cf14-d8d5-9d21-7acbe86d4444 Address:83a654a4-cf14-d8d5-9d21-7acbe86d4444} {Suffrage:Voter ID:server-8bb1362d-8c13-5989-4193-42ceac8986ec Address:8bb1362d-8c13-5989-4193-42ceac8986ec} {Suffrage:Voter ID:server-73ca3b53-62c0-a512-0a03-255f1dcd450b Address:73ca3b53-62c0-a512-0a03-255f1dcd450b}]"
2025-01-26T02:43:29.876Z [INFO]  server-73ca3b53-62c0-a512-0a03-255f1dcd450b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83a654a4-cf14-d8d5-9d21-7acbe86d4444 Address:83a654a4-cf14-d8d5-9d21-7acbe86d4444} {Suffrage:Voter ID:server-8bb1362d-8c13-5989-4193-42ceac8986ec Address:8bb1362d-8c13-5989-4193-42ceac8986ec} {Suffrage:Voter ID:server-73ca3b53-62c0-a512-0a03-255f1dcd450b Address:73ca3b53-62c0-a512-0a03-255f1dcd450b}]"
2025-01-26T02:43:29.876Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:29.876Z [INFO]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: entering follower state: follower="Node at 83a654a4-cf14-d8d5-9d21-7acbe86d4444 [Follower]" leader-address= leader-id=
2025-01-26T02:43:29.876Z [INFO]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: entering follower state: follower="Node at 8bb1362d-8c13-5989-4193-42ceac8986ec [Follower]" leader-address= leader-id=
2025-01-26T02:43:29.876Z [INFO]  server-73ca3b53-62c0-a512-0a03-255f1dcd450b: entering follower state: follower="Node at 73ca3b53-62c0-a512-0a03-255f1dcd450b [Follower]" leader-address= leader-id=
2025-01-26T02:43:29.935Z [WARN]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:29.935Z [INFO]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: entering candidate state: node="Node at 8bb1362d-8c13-5989-4193-42ceac8986ec [Candidate]" term=2
2025-01-26T02:43:29.935Z [INFO]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: election won: term=2 tally=2
2025-01-26T02:43:29.935Z [INFO]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: entering leader state: leader="Node at 8bb1362d-8c13-5989-4193-42ceac8986ec [Leader]"
2025-01-26T02:43:29.935Z [INFO]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: added peer, starting replication: peer=server-83a654a4-cf14-d8d5-9d21-7acbe86d4444
2025-01-26T02:43:29.935Z [INFO]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: added peer, starting replication: peer=server-73ca3b53-62c0-a512-0a03-255f1dcd450b
2025-01-26T02:43:29.935Z [INFO]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: pipelining replication: peer="{Voter server-73ca3b53-62c0-a512-0a03-255f1dcd450b 73ca3b53-62c0-a512-0a03-255f1dcd450b}"
2025-01-26T02:43:29.935Z [INFO]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: pipelining replication: peer="{Voter server-83a654a4-cf14-d8d5-9d21-7acbe86d4444 83a654a4-cf14-d8d5-9d21-7acbe86d4444}"
2025-01-26T02:43:30.043Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:29.935704528 +0000 UTC m=+6.450736903 (1 nodes), 59.567561ms from start of poll, 59.733442ms from cluster start. Timeout at 2025-01-26 02:43:30.040707688 +0000 UTC m=+6.555740063, 105.00316ms after stability
2025-01-26T02:43:30.052Z [INFO]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: aborting pipeline replication: peer="{Voter server-73ca3b53-62c0-a512-0a03-255f1dcd450b 73ca3b53-62c0-a512-0a03-255f1dcd450b}"
2025-01-26T02:43:30.052Z [INFO]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: aborting pipeline replication: peer="{Voter server-83a654a4-cf14-d8d5-9d21-7acbe86d4444 83a654a4-cf14-d8d5-9d21-7acbe86d4444}"
2025-01-26T02:43:30.052Z [INFO]  creating new snapshot: path=/tmp/raft1654390816/snapshots/2-27-1737859410052.tmp
2025-01-26T02:43:30.053Z [INFO]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: starting restore from snapshot: id=2-27-1737859410052 last-index=27 last-term=2 size-in-bytes=168
2025-01-26T02:43:30.053Z [INFO]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: snapshot restore progress: id=2-27-1737859410052 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-01-26T02:43:30.053Z [INFO]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: restored from snapshot: id=2-27-1737859410052 last-index=27 last-term=2 size-in-bytes=168
2025-01-26T02:43:30.053Z [INFO]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83a654a4-cf14-d8d5-9d21-7acbe86d4444 Address:83a654a4-cf14-d8d5-9d21-7acbe86d4444} {Suffrage:Voter ID:server-8bb1362d-8c13-5989-4193-42ceac8986ec Address:8bb1362d-8c13-5989-4193-42ceac8986ec} {Suffrage:Voter ID:server-73ca3b53-62c0-a512-0a03-255f1dcd450b Address:73ca3b53-62c0-a512-0a03-255f1dcd450b}]"
2025-01-26T02:43:30.053Z [INFO]  creating new snapshot: path=/tmp/raft3810636897/snapshots/2-27-1737859410053.tmp
2025-01-26T02:43:30.053Z [INFO]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: starting restore from snapshot: id=2-27-1737859410053 last-index=27 last-term=2 size-in-bytes=168
2025-01-26T02:43:30.053Z [INFO]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: snapshot restore progress: id=2-27-1737859410053 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-01-26T02:43:30.053Z [INFO]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: restored from snapshot: id=2-27-1737859410053 last-index=27 last-term=2 size-in-bytes=168
2025-01-26T02:43:30.053Z [INFO]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83a654a4-cf14-d8d5-9d21-7acbe86d4444 Address:83a654a4-cf14-d8d5-9d21-7acbe86d4444} {Suffrage:Voter ID:server-8bb1362d-8c13-5989-4193-42ceac8986ec Address:8bb1362d-8c13-5989-4193-42ceac8986ec} {Suffrage:Voter ID:server-73ca3b53-62c0-a512-0a03-255f1dcd450b Address:73ca3b53-62c0-a512-0a03-255f1dcd450b}]"
2025-01-26T02:43:30.053Z [INFO]  creating new snapshot: path=/tmp/raft2431066796/snapshots/2-16-1737859410053.tmp
2025-01-26T02:43:30.053Z [INFO]  server-73ca3b53-62c0-a512-0a03-255f1dcd450b: starting restore from snapshot: id=2-16-1737859410053 last-index=16 last-term=2 size-in-bytes=89
2025-01-26T02:43:30.053Z [INFO]  server-73ca3b53-62c0-a512-0a03-255f1dcd450b: snapshot restore progress: id=2-16-1737859410053 last-index=16 last-term=2 size-in-bytes=89 read-bytes=89 percent-complete=100.00%
2025-01-26T02:43:30.053Z [INFO]  server-73ca3b53-62c0-a512-0a03-255f1dcd450b: restored from snapshot: id=2-16-1737859410053 last-index=16 last-term=2 size-in-bytes=89
2025-01-26T02:43:30.053Z [INFO]  server-73ca3b53-62c0-a512-0a03-255f1dcd450b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83a654a4-cf14-d8d5-9d21-7acbe86d4444 Address:83a654a4-cf14-d8d5-9d21-7acbe86d4444} {Suffrage:Voter ID:server-8bb1362d-8c13-5989-4193-42ceac8986ec Address:8bb1362d-8c13-5989-4193-42ceac8986ec} {Suffrage:Voter ID:server-73ca3b53-62c0-a512-0a03-255f1dcd450b Address:73ca3b53-62c0-a512-0a03-255f1dcd450b}]"
2025-01-26T02:43:30.053Z [INFO]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: entering follower state: follower="Node at 83a654a4-cf14-d8d5-9d21-7acbe86d4444 [Follower]" leader-address= leader-id=
2025-01-26T02:43:30.053Z [INFO]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: entering follower state: follower="Node at 8bb1362d-8c13-5989-4193-42ceac8986ec [Follower]" leader-address= leader-id=
2025-01-26T02:43:30.053Z [INFO]  server-73ca3b53-62c0-a512-0a03-255f1dcd450b: entering follower state: follower="Node at 73ca3b53-62c0-a512-0a03-255f1dcd450b [Follower]" leader-address= leader-id=
2025-01-26T02:43:30.127Z [WARN]  server-73ca3b53-62c0-a512-0a03-255f1dcd450b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:30.127Z [INFO]  server-73ca3b53-62c0-a512-0a03-255f1dcd450b: entering candidate state: node="Node at 73ca3b53-62c0-a512-0a03-255f1dcd450b [Candidate]" term=3
2025-01-26T02:43:30.127Z [WARN]  server-8bb1362d-8c13-5989-4193-42ceac8986ec: rejecting vote request since our last index is greater: candidate=73ca3b53-62c0-a512-0a03-255f1dcd450b last-index=27 last-candidate-index=16
2025-01-26T02:43:30.127Z [WARN]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: rejecting vote request since our last index is greater: candidate=73ca3b53-62c0-a512-0a03-255f1dcd450b last-index=27 last-candidate-index=16
2025-01-26T02:43:30.131Z [WARN]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:30.131Z [INFO]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: entering candidate state: node="Node at 83a654a4-cf14-d8d5-9d21-7acbe86d4444 [Candidate]" term=4
2025-01-26T02:43:30.131Z [INFO]  server-73ca3b53-62c0-a512-0a03-255f1dcd450b: entering follower state: follower="Node at 73ca3b53-62c0-a512-0a03-255f1dcd450b [Follower]" leader-address= leader-id=
2025-01-26T02:43:30.131Z [INFO]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: election won: term=4 tally=2
2025-01-26T02:43:30.131Z [INFO]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: entering leader state: leader="Node at 83a654a4-cf14-d8d5-9d21-7acbe86d4444 [Leader]"
2025-01-26T02:43:30.131Z [INFO]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: added peer, starting replication: peer=server-8bb1362d-8c13-5989-4193-42ceac8986ec
2025-01-26T02:43:30.131Z [INFO]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: added peer, starting replication: peer=server-73ca3b53-62c0-a512-0a03-255f1dcd450b
2025-01-26T02:43:30.131Z [WARN]  server-73ca3b53-62c0-a512-0a03-255f1dcd450b: failed to get previous log: previous-index=27 last-index=16 error="log not found"
2025-01-26T02:43:30.131Z [WARN]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: appendEntries rejected, sending older logs: peer="{Voter server-73ca3b53-62c0-a512-0a03-255f1dcd450b 73ca3b53-62c0-a512-0a03-255f1dcd450b}" next=17
2025-01-26T02:43:30.131Z [ERROR] server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: failed to get log: index=16 error="log not found"
2025-01-26T02:43:30.131Z [INFO]  creating new snapshot: path=/tmp/raft2431066796/snapshots/2-27-1737859410131.tmp
2025-01-26T02:43:30.131Z [INFO]  server-73ca3b53-62c0-a512-0a03-255f1dcd450b: snapshot network transfer progress: read-bytes=168 percent-complete=100.00%
2025-01-26T02:43:30.132Z [INFO]  server-73ca3b53-62c0-a512-0a03-255f1dcd450b: copied to local snapshot: bytes=168
2025-01-26T02:43:30.132Z [INFO]  server-73ca3b53-62c0-a512-0a03-255f1dcd450b: snapshot restore progress: id=2-27-1737859410131 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-01-26T02:43:30.132Z [INFO]  server-73ca3b53-62c0-a512-0a03-255f1dcd450b: Installed remote snapshot
2025-01-26T02:43:30.132Z [INFO]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: pipelining replication: peer="{Voter server-73ca3b53-62c0-a512-0a03-255f1dcd450b 73ca3b53-62c0-a512-0a03-255f1dcd450b}"
2025-01-26T02:43:30.132Z [INFO]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: pipelining replication: peer="{Voter server-8bb1362d-8c13-5989-4193-42ceac8986ec 8bb1362d-8c13-5989-4193-42ceac8986ec}"
2025-01-26T02:43:30.371Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:30.479Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:30.371423692 +0000 UTC m=+6.886456067 (1 nodes), 22.52µs from start of poll, 495.452606ms from cluster start. Timeout at 2025-01-26 02:43:30.476433572 +0000 UTC m=+6.991465947, 105.00988ms after stability
2025-01-26T02:43:30.479Z [INFO]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: aborting pipeline replication: peer="{Voter server-73ca3b53-62c0-a512-0a03-255f1dcd450b 73ca3b53-62c0-a512-0a03-255f1dcd450b}"
2025-01-26T02:43:30.479Z [INFO]  server-83a654a4-cf14-d8d5-9d21-7acbe86d4444: aborting pipeline replication: peer="{Voter server-8bb1362d-8c13-5989-4193-42ceac8986ec 8bb1362d-8c13-5989-4193-42ceac8986ec}"
--- PASS: TestRaft_RecoverCluster (2.34s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (1.10s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.62s)
    --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.61s)
=== RUN   TestRaft_HasExistingState
    raft_test.go:258: Skipping unreliable test
--- SKIP: TestRaft_HasExistingState (0.00s)
=== RUN   TestRaft_SingleNode
2025-01-26T02:43:30.499Z [INFO]  server-f51ef8bc-5e3b-aee6-6285-34823e8bb728: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f51ef8bc-5e3b-aee6-6285-34823e8bb728 Address:f51ef8bc-5e3b-aee6-6285-34823e8bb728}]"
2025-01-26T02:43:30.499Z [INFO]  server-f51ef8bc-5e3b-aee6-6285-34823e8bb728: entering follower state: follower="Node at f51ef8bc-5e3b-aee6-6285-34823e8bb728 [Follower]" leader-address= leader-id=
2025-01-26T02:43:30.559Z [WARN]  server-f51ef8bc-5e3b-aee6-6285-34823e8bb728: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:30.559Z [INFO]  server-f51ef8bc-5e3b-aee6-6285-34823e8bb728: entering candidate state: node="Node at f51ef8bc-5e3b-aee6-6285-34823e8bb728 [Candidate]" term=2
2025-01-26T02:43:30.559Z [INFO]  server-f51ef8bc-5e3b-aee6-6285-34823e8bb728: election won: term=2 tally=1
2025-01-26T02:43:30.559Z [INFO]  server-f51ef8bc-5e3b-aee6-6285-34823e8bb728: entering leader state: leader="Node at f51ef8bc-5e3b-aee6-6285-34823e8bb728 [Leader]"
--- PASS: TestRaft_SingleNode (0.06s)
=== RUN   TestRaft_TripleNode
2025-01-26T02:43:30.559Z [INFO]  server-e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa Address:e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa} {Suffrage:Voter ID:server-1b8af0f3-6abd-1ed5-390b-dcc59e020e22 Address:1b8af0f3-6abd-1ed5-390b-dcc59e020e22} {Suffrage:Voter ID:server-9fc3fe58-e2ac-1d35-e54a-65a62ebc615b Address:9fc3fe58-e2ac-1d35-e54a-65a62ebc615b}]"
2025-01-26T02:43:30.559Z [INFO]  server-1b8af0f3-6abd-1ed5-390b-dcc59e020e22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa Address:e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa} {Suffrage:Voter ID:server-1b8af0f3-6abd-1ed5-390b-dcc59e020e22 Address:1b8af0f3-6abd-1ed5-390b-dcc59e020e22} {Suffrage:Voter ID:server-9fc3fe58-e2ac-1d35-e54a-65a62ebc615b Address:9fc3fe58-e2ac-1d35-e54a-65a62ebc615b}]"
2025-01-26T02:43:30.559Z [INFO]  server-9fc3fe58-e2ac-1d35-e54a-65a62ebc615b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa Address:e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa} {Suffrage:Voter ID:server-1b8af0f3-6abd-1ed5-390b-dcc59e020e22 Address:1b8af0f3-6abd-1ed5-390b-dcc59e020e22} {Suffrage:Voter ID:server-9fc3fe58-e2ac-1d35-e54a-65a62ebc615b Address:9fc3fe58-e2ac-1d35-e54a-65a62ebc615b}]"
2025-01-26T02:43:30.559Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:43:30.559Z [INFO]  server-e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa: entering follower state: follower="Node at e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa [Follower]" leader-address= leader-id=
2025-01-26T02:43:30.560Z [INFO]  server-1b8af0f3-6abd-1ed5-390b-dcc59e020e22: entering follower state: follower="Node at 1b8af0f3-6abd-1ed5-390b-dcc59e020e22 [Follower]" leader-address= leader-id=
2025-01-26T02:43:30.560Z [INFO]  server-9fc3fe58-e2ac-1d35-e54a-65a62ebc615b: entering follower state: follower="Node at 9fc3fe58-e2ac-1d35-e54a-65a62ebc615b [Follower]" leader-address= leader-id=
2025-01-26T02:43:30.619Z [WARN]  server-e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:30.619Z [INFO]  server-e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa: entering candidate state: node="Node at e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa [Candidate]" term=2
2025-01-26T02:43:30.619Z [INFO]  server-e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa: election won: term=2 tally=2
2025-01-26T02:43:30.619Z [INFO]  server-e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa: entering leader state: leader="Node at e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa [Leader]"
2025-01-26T02:43:30.619Z [INFO]  server-e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa: added peer, starting replication: peer=server-1b8af0f3-6abd-1ed5-390b-dcc59e020e22
2025-01-26T02:43:30.619Z [INFO]  server-e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa: added peer, starting replication: peer=server-9fc3fe58-e2ac-1d35-e54a-65a62ebc615b
2025-01-26T02:43:30.619Z [INFO]  server-e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa: pipelining replication: peer="{Voter server-9fc3fe58-e2ac-1d35-e54a-65a62ebc615b 9fc3fe58-e2ac-1d35-e54a-65a62ebc615b}"
2025-01-26T02:43:30.619Z [INFO]  server-e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa: pipelining replication: peer="{Voter server-1b8af0f3-6abd-1ed5-390b-dcc59e020e22 1b8af0f3-6abd-1ed5-390b-dcc59e020e22}"
2025-01-26T02:43:30.727Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:43:30.619706919 +0000 UTC m=+7.134739294 (2 nodes), 59.733843ms from start of poll, 59.900524ms from cluster start. Timeout at 2025-01-26 02:43:30.724709879 +0000 UTC m=+7.239742254, 105.00296ms after stability
2025-01-26T02:43:30.727Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:30.835Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:30.727469733 +0000 UTC m=+7.242502068 (1 nodes), 1.68µs from start of poll, 167.663298ms from cluster start. Timeout at 2025-01-26 02:43:30.832475973 +0000 UTC m=+7.347508348, 105.00628ms after stability
2025-01-26T02:43:30.851Z [INFO]  server-e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa: aborting pipeline replication: peer="{Voter server-9fc3fe58-e2ac-1d35-e54a-65a62ebc615b 9fc3fe58-e2ac-1d35-e54a-65a62ebc615b}"
2025-01-26T02:43:30.851Z [INFO]  server-e0d3b2ff-800f-aa64-30f8-bf84cb16f0fa: aborting pipeline replication: peer="{Voter server-1b8af0f3-6abd-1ed5-390b-dcc59e020e22 1b8af0f3-6abd-1ed5-390b-dcc59e020e22}"
--- PASS: TestRaft_TripleNode (0.29s)
=== RUN   TestRaft_LeaderFail
2025-01-26T02:43:30.851Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d83737e3-80e9-9241-3f6c-fa724ef257e7 Address:d83737e3-80e9-9241-3f6c-fa724ef257e7} {Suffrage:Voter ID:server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2 Address:b2cd91dc-494f-5bc4-ff47-177fdae3ede2} {Suffrage:Voter ID:server-eabb1e89-846a-e534-a4f2-c3650ac40193 Address:eabb1e89-846a-e534-a4f2-c3650ac40193}]"
2025-01-26T02:43:30.851Z [INFO]  server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d83737e3-80e9-9241-3f6c-fa724ef257e7 Address:d83737e3-80e9-9241-3f6c-fa724ef257e7} {Suffrage:Voter ID:server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2 Address:b2cd91dc-494f-5bc4-ff47-177fdae3ede2} {Suffrage:Voter ID:server-eabb1e89-846a-e534-a4f2-c3650ac40193 Address:eabb1e89-846a-e534-a4f2-c3650ac40193}]"
2025-01-26T02:43:30.852Z [INFO]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d83737e3-80e9-9241-3f6c-fa724ef257e7 Address:d83737e3-80e9-9241-3f6c-fa724ef257e7} {Suffrage:Voter ID:server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2 Address:b2cd91dc-494f-5bc4-ff47-177fdae3ede2} {Suffrage:Voter ID:server-eabb1e89-846a-e534-a4f2-c3650ac40193 Address:eabb1e89-846a-e534-a4f2-c3650ac40193}]"
2025-01-26T02:43:30.852Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:43:30.852Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: entering follower state: follower="Node at d83737e3-80e9-9241-3f6c-fa724ef257e7 [Follower]" leader-address= leader-id=
2025-01-26T02:43:30.852Z [INFO]  server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2: entering follower state: follower="Node at b2cd91dc-494f-5bc4-ff47-177fdae3ede2 [Follower]" leader-address= leader-id=
2025-01-26T02:43:30.852Z [INFO]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: entering follower state: follower="Node at eabb1e89-846a-e534-a4f2-c3650ac40193 [Follower]" leader-address= leader-id=
2025-01-26T02:43:30.911Z [WARN]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:30.911Z [INFO]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: entering candidate state: node="Node at eabb1e89-846a-e534-a4f2-c3650ac40193 [Candidate]" term=2
2025-01-26T02:43:30.911Z [INFO]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: election won: term=2 tally=2
2025-01-26T02:43:30.911Z [INFO]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: entering leader state: leader="Node at eabb1e89-846a-e534-a4f2-c3650ac40193 [Leader]"
2025-01-26T02:43:30.911Z [INFO]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: added peer, starting replication: peer=server-d83737e3-80e9-9241-3f6c-fa724ef257e7
2025-01-26T02:43:30.911Z [INFO]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: added peer, starting replication: peer=server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2
2025-01-26T02:43:30.911Z [INFO]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: pipelining replication: peer="{Voter server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2 b2cd91dc-494f-5bc4-ff47-177fdae3ede2}"
2025-01-26T02:43:30.911Z [INFO]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: pipelining replication: peer="{Voter server-d83737e3-80e9-9241-3f6c-fa724ef257e7 d83737e3-80e9-9241-3f6c-fa724ef257e7}"
2025-01-26T02:43:31.019Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:43:30.911706411 +0000 UTC m=+7.426738786 (2 nodes), 59.659562ms from start of poll, 59.833163ms from cluster start. Timeout at 2025-01-26 02:43:31.016707531 +0000 UTC m=+7.531739906, 105.00112ms after stability
2025-01-26T02:43:31.019Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:31.127Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:31.019504585 +0000 UTC m=+7.534536960 (1 nodes), 920ns from start of poll, 167.631337ms from cluster start. Timeout at 2025-01-26 02:43:31.124513825 +0000 UTC m=+7.639546160, 105.0092ms after stability
    raft_test.go:378: [INFO] Disconnecting Node at eabb1e89-846a-e534-a4f2-c3650ac40193 [Leader]
2025-01-26T02:43:31.143Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to heartbeat to: peer=b2cd91dc-494f-5bc4-ff47-177fdae3ede2 backoff time=10ms error="failed to connect to peer: b2cd91dc-494f-5bc4-ff47-177fdae3ede2"
2025-01-26T02:43:31.143Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:31.147Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to heartbeat to: peer=d83737e3-80e9-9241-3f6c-fa724ef257e7 backoff time=10ms error="failed to connect to peer: d83737e3-80e9-9241-3f6c-fa724ef257e7"
2025-01-26T02:43:31.151Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to pipeline appendEntries: peer="{Voter server-d83737e3-80e9-9241-3f6c-fa724ef257e7 d83737e3-80e9-9241-3f6c-fa724ef257e7}" error="append pipeline closed"
2025-01-26T02:43:31.151Z [INFO]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: aborting pipeline replication: peer="{Voter server-d83737e3-80e9-9241-3f6c-fa724ef257e7 d83737e3-80e9-9241-3f6c-fa724ef257e7}"
2025-01-26T02:43:31.151Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to pipeline appendEntries: peer="{Voter server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2 b2cd91dc-494f-5bc4-ff47-177fdae3ede2}" error="append pipeline closed"
2025-01-26T02:43:31.151Z [INFO]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: aborting pipeline replication: peer="{Voter server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2 b2cd91dc-494f-5bc4-ff47-177fdae3ede2}"
2025-01-26T02:43:31.163Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to appendEntries to: peer="{Voter server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2 b2cd91dc-494f-5bc4-ff47-177fdae3ede2}" error="failed to connect to peer: b2cd91dc-494f-5bc4-ff47-177fdae3ede2"
2025-01-26T02:43:31.163Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to heartbeat to: peer=b2cd91dc-494f-5bc4-ff47-177fdae3ede2 backoff time=10ms error="failed to connect to peer: b2cd91dc-494f-5bc4-ff47-177fdae3ede2"
2025-01-26T02:43:31.163Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to appendEntries to: peer="{Voter server-d83737e3-80e9-9241-3f6c-fa724ef257e7 d83737e3-80e9-9241-3f6c-fa724ef257e7}" error="failed to connect to peer: d83737e3-80e9-9241-3f6c-fa724ef257e7"
2025-01-26T02:43:31.175Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to heartbeat to: peer=d83737e3-80e9-9241-3f6c-fa724ef257e7 backoff time=10ms error="failed to connect to peer: d83737e3-80e9-9241-3f6c-fa724ef257e7"
2025-01-26T02:43:31.187Z [WARN]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to contact: server-id=server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2 time=51.858394ms
2025-01-26T02:43:31.187Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to heartbeat to: peer=b2cd91dc-494f-5bc4-ff47-177fdae3ede2 backoff time=10ms error="failed to connect to peer: b2cd91dc-494f-5bc4-ff47-177fdae3ede2"
2025-01-26T02:43:31.187Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to appendEntries to: peer="{Voter server-d83737e3-80e9-9241-3f6c-fa724ef257e7 d83737e3-80e9-9241-3f6c-fa724ef257e7}" error="failed to connect to peer: d83737e3-80e9-9241-3f6c-fa724ef257e7"
2025-01-26T02:43:31.187Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to appendEntries to: peer="{Voter server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2 b2cd91dc-494f-5bc4-ff47-177fdae3ede2}" error="failed to connect to peer: b2cd91dc-494f-5bc4-ff47-177fdae3ede2"
2025-01-26T02:43:31.199Z [WARN]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to contact: server-id=server-d83737e3-80e9-9241-3f6c-fa724ef257e7 time=59.942043ms
2025-01-26T02:43:31.199Z [WARN]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to contact: server-id=server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2 time=63.866707ms
2025-01-26T02:43:31.199Z [WARN]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to contact quorum of nodes, stepping down
2025-01-26T02:43:31.199Z [INFO]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: entering follower state: follower="Node at eabb1e89-846a-e534-a4f2-c3650ac40193 [Follower]" leader-address= leader-id=
2025-01-26T02:43:31.199Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to heartbeat to: peer=d83737e3-80e9-9241-3f6c-fa724ef257e7 backoff time=10ms error="failed to connect to peer: d83737e3-80e9-9241-3f6c-fa724ef257e7"
2025-01-26T02:43:31.207Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to appendEntries to: peer="{Voter server-d83737e3-80e9-9241-3f6c-fa724ef257e7 d83737e3-80e9-9241-3f6c-fa724ef257e7}" error="failed to connect to peer: d83737e3-80e9-9241-3f6c-fa724ef257e7"
2025-01-26T02:43:31.207Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to appendEntries to: peer="{Voter server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2 b2cd91dc-494f-5bc4-ff47-177fdae3ede2}" error="failed to connect to peer: b2cd91dc-494f-5bc4-ff47-177fdae3ede2"
2025-01-26T02:43:31.227Z [WARN]  server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2: heartbeat timeout reached, starting election: last-leader-addr=eabb1e89-846a-e534-a4f2-c3650ac40193 last-leader-id=server-eabb1e89-846a-e534-a4f2-c3650ac40193
2025-01-26T02:43:31.227Z [INFO]  server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2: entering candidate state: node="Node at b2cd91dc-494f-5bc4-ff47-177fdae3ede2 [Candidate]" term=3
2025-01-26T02:43:31.227Z [ERROR] server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2: failed to make requestVote RPC: target="{Voter server-eabb1e89-846a-e534-a4f2-c3650ac40193 eabb1e89-846a-e534-a4f2-c3650ac40193}" error="failed to connect to peer: eabb1e89-846a-e534-a4f2-c3650ac40193" term=3
2025-01-26T02:43:31.227Z [WARN]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: rejecting vote request since we have a leader: from=b2cd91dc-494f-5bc4-ff47-177fdae3ede2 leader=eabb1e89-846a-e534-a4f2-c3650ac40193 leader-id=server-eabb1e89-846a-e534-a4f2-c3650ac40193
2025-01-26T02:43:31.239Z [WARN]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: heartbeat timeout reached, starting election: last-leader-addr=eabb1e89-846a-e534-a4f2-c3650ac40193 last-leader-id=server-eabb1e89-846a-e534-a4f2-c3650ac40193
2025-01-26T02:43:31.239Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: entering candidate state: node="Node at d83737e3-80e9-9241-3f6c-fa724ef257e7 [Candidate]" term=3
2025-01-26T02:43:31.239Z [ERROR] server-d83737e3-80e9-9241-3f6c-fa724ef257e7: failed to make requestVote RPC: target="{Voter server-eabb1e89-846a-e534-a4f2-c3650ac40193 eabb1e89-846a-e534-a4f2-c3650ac40193}" error="failed to connect to peer: eabb1e89-846a-e534-a4f2-c3650ac40193" term=3
2025-01-26T02:43:31.239Z [INFO]  server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2: duplicate requestVote for same term: term=3
2025-01-26T02:43:31.291Z [WARN]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:31.291Z [INFO]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: entering candidate state: node="Node at eabb1e89-846a-e534-a4f2-c3650ac40193 [Candidate]" term=3
2025-01-26T02:43:31.291Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to make requestVote RPC: target="{Voter server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2 b2cd91dc-494f-5bc4-ff47-177fdae3ede2}" error="failed to connect to peer: b2cd91dc-494f-5bc4-ff47-177fdae3ede2" term=3
2025-01-26T02:43:31.291Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to make requestVote RPC: target="{Voter server-d83737e3-80e9-9241-3f6c-fa724ef257e7 d83737e3-80e9-9241-3f6c-fa724ef257e7}" error="failed to connect to peer: d83737e3-80e9-9241-3f6c-fa724ef257e7" term=3
2025-01-26T02:43:31.311Z [WARN]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: Election timeout reached, restarting election
2025-01-26T02:43:31.311Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: entering candidate state: node="Node at d83737e3-80e9-9241-3f6c-fa724ef257e7 [Candidate]" term=4
2025-01-26T02:43:31.311Z [ERROR] server-d83737e3-80e9-9241-3f6c-fa724ef257e7: failed to make requestVote RPC: target="{Voter server-eabb1e89-846a-e534-a4f2-c3650ac40193 eabb1e89-846a-e534-a4f2-c3650ac40193}" error="failed to connect to peer: eabb1e89-846a-e534-a4f2-c3650ac40193" term=4
2025-01-26T02:43:31.311Z [INFO]  server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2: entering follower state: follower="Node at b2cd91dc-494f-5bc4-ff47-177fdae3ede2 [Follower]" leader-address= leader-id=
2025-01-26T02:43:31.311Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: election won: term=4 tally=2
2025-01-26T02:43:31.311Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: entering leader state: leader="Node at d83737e3-80e9-9241-3f6c-fa724ef257e7 [Leader]"
2025-01-26T02:43:31.311Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: added peer, starting replication: peer=server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2
2025-01-26T02:43:31.311Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: added peer, starting replication: peer=server-eabb1e89-846a-e534-a4f2-c3650ac40193
2025-01-26T02:43:31.311Z [ERROR] server-d83737e3-80e9-9241-3f6c-fa724ef257e7: failed to appendEntries to: peer="{Voter server-eabb1e89-846a-e534-a4f2-c3650ac40193 eabb1e89-846a-e534-a4f2-c3650ac40193}" error="failed to connect to peer: eabb1e89-846a-e534-a4f2-c3650ac40193"
2025-01-26T02:43:31.311Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: pipelining replication: peer="{Voter server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2 b2cd91dc-494f-5bc4-ff47-177fdae3ede2}"
2025-01-26T02:43:31.319Z [ERROR] server-d83737e3-80e9-9241-3f6c-fa724ef257e7: failed to heartbeat to: peer=eabb1e89-846a-e534-a4f2-c3650ac40193 backoff time=10ms error="failed to connect to peer: eabb1e89-846a-e534-a4f2-c3650ac40193"
2025-01-26T02:43:31.331Z [ERROR] server-d83737e3-80e9-9241-3f6c-fa724ef257e7: failed to appendEntries to: peer="{Voter server-eabb1e89-846a-e534-a4f2-c3650ac40193 eabb1e89-846a-e534-a4f2-c3650ac40193}" error="failed to connect to peer: eabb1e89-846a-e534-a4f2-c3650ac40193"
2025-01-26T02:43:31.343Z [ERROR] server-d83737e3-80e9-9241-3f6c-fa724ef257e7: failed to heartbeat to: peer=eabb1e89-846a-e534-a4f2-c3650ac40193 backoff time=10ms error="failed to connect to peer: eabb1e89-846a-e534-a4f2-c3650ac40193"
2025-01-26T02:43:31.355Z [WARN]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: Election timeout reached, restarting election
2025-01-26T02:43:31.355Z [INFO]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: entering candidate state: node="Node at eabb1e89-846a-e534-a4f2-c3650ac40193 [Candidate]" term=4
2025-01-26T02:43:31.355Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to make requestVote RPC: target="{Voter server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2 b2cd91dc-494f-5bc4-ff47-177fdae3ede2}" error="failed to connect to peer: b2cd91dc-494f-5bc4-ff47-177fdae3ede2" term=4
2025-01-26T02:43:31.355Z [ERROR] server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to make requestVote RPC: target="{Voter server-d83737e3-80e9-9241-3f6c-fa724ef257e7 d83737e3-80e9-9241-3f6c-fa724ef257e7}" error="failed to connect to peer: d83737e3-80e9-9241-3f6c-fa724ef257e7" term=4
2025-01-26T02:43:31.355Z [ERROR] server-d83737e3-80e9-9241-3f6c-fa724ef257e7: failed to appendEntries to: peer="{Voter server-eabb1e89-846a-e534-a4f2-c3650ac40193 eabb1e89-846a-e534-a4f2-c3650ac40193}" error="failed to connect to peer: eabb1e89-846a-e534-a4f2-c3650ac40193"
2025-01-26T02:43:31.363Z [WARN]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: failed to contact: server-id=server-eabb1e89-846a-e534-a4f2-c3650ac40193 time=51.898635ms
2025-01-26T02:43:31.367Z [ERROR] server-d83737e3-80e9-9241-3f6c-fa724ef257e7: failed to heartbeat to: peer=eabb1e89-846a-e534-a4f2-c3650ac40193 backoff time=10ms error="failed to connect to peer: eabb1e89-846a-e534-a4f2-c3650ac40193"
2025-01-26T02:43:31.387Z [ERROR] server-d83737e3-80e9-9241-3f6c-fa724ef257e7: failed to appendEntries to: peer="{Voter server-eabb1e89-846a-e534-a4f2-c3650ac40193 eabb1e89-846a-e534-a4f2-c3650ac40193}" error="failed to connect to peer: eabb1e89-846a-e534-a4f2-c3650ac40193"
2025-01-26T02:43:31.391Z [ERROR] server-d83737e3-80e9-9241-3f6c-fa724ef257e7: failed to heartbeat to: peer=eabb1e89-846a-e534-a4f2-c3650ac40193 backoff time=20ms error="failed to connect to peer: eabb1e89-846a-e534-a4f2-c3650ac40193"
2025-01-26T02:43:31.415Z [WARN]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: failed to contact: server-id=server-eabb1e89-846a-e534-a4f2-c3650ac40193 time=103.83055ms
2025-01-26T02:43:31.419Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:31.311607357 +0000 UTC m=+7.826639732 (1 nodes), 168.11982ms from start of poll, 459.734109ms from cluster start. Timeout at 2025-01-26 02:43:31.416610757 +0000 UTC m=+7.931643132, 105.0034ms after stability
    raft_test.go:413: [INFO] Reconnecting Node at eabb1e89-846a-e534-a4f2-c3650ac40193 [Candidate]
2025-01-26T02:43:31.427Z [WARN]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: Election timeout reached, restarting election
2025-01-26T02:43:31.427Z [INFO]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: entering candidate state: node="Node at eabb1e89-846a-e534-a4f2-c3650ac40193 [Candidate]" term=5
2025-01-26T02:43:31.427Z [WARN]  server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2: rejecting vote request since we have a leader: from=eabb1e89-846a-e534-a4f2-c3650ac40193 leader=d83737e3-80e9-9241-3f6c-fa724ef257e7 leader-id=server-d83737e3-80e9-9241-3f6c-fa724ef257e7
2025-01-26T02:43:31.427Z [WARN]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: rejecting vote request since we have a leader: from=eabb1e89-846a-e534-a4f2-c3650ac40193 leader=d83737e3-80e9-9241-3f6c-fa724ef257e7 leader-id=server-d83737e3-80e9-9241-3f6c-fa724ef257e7
2025-01-26T02:43:31.435Z [ERROR] server-d83737e3-80e9-9241-3f6c-fa724ef257e7: peer has newer term, stopping replication: peer="{Voter server-eabb1e89-846a-e534-a4f2-c3650ac40193 eabb1e89-846a-e534-a4f2-c3650ac40193}"
2025-01-26T02:43:31.435Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: entering follower state: follower="Node at d83737e3-80e9-9241-3f6c-fa724ef257e7 [Follower]" leader-address= leader-id=
2025-01-26T02:43:31.435Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: aborting pipeline replication: peer="{Voter server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2 b2cd91dc-494f-5bc4-ff47-177fdae3ede2}"
2025-01-26T02:43:31.499Z [WARN]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:31.499Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: entering candidate state: node="Node at d83737e3-80e9-9241-3f6c-fa724ef257e7 [Candidate]" term=5
2025-01-26T02:43:31.499Z [INFO]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: duplicate requestVote for same term: term=5
2025-01-26T02:43:31.499Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: election won: term=5 tally=2
2025-01-26T02:43:31.499Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: entering leader state: leader="Node at d83737e3-80e9-9241-3f6c-fa724ef257e7 [Leader]"
2025-01-26T02:43:31.499Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: added peer, starting replication: peer=server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2
2025-01-26T02:43:31.499Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: added peer, starting replication: peer=server-eabb1e89-846a-e534-a4f2-c3650ac40193
2025-01-26T02:43:31.499Z [WARN]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: failed to get previous log: previous-index=5 last-index=3 error="log not found"
2025-01-26T02:43:31.499Z [INFO]  server-eabb1e89-846a-e534-a4f2-c3650ac40193: entering follower state: follower="Node at eabb1e89-846a-e534-a4f2-c3650ac40193 [Follower]" leader-address=d83737e3-80e9-9241-3f6c-fa724ef257e7 leader-id=server-d83737e3-80e9-9241-3f6c-fa724ef257e7
2025-01-26T02:43:31.499Z [WARN]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: appendEntries rejected, sending older logs: peer="{Voter server-eabb1e89-846a-e534-a4f2-c3650ac40193 eabb1e89-846a-e534-a4f2-c3650ac40193}" next=4
2025-01-26T02:43:31.499Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: pipelining replication: peer="{Voter server-eabb1e89-846a-e534-a4f2-c3650ac40193 eabb1e89-846a-e534-a4f2-c3650ac40193}"
2025-01-26T02:43:31.499Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: pipelining replication: peer="{Voter server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2 b2cd91dc-494f-5bc4-ff47-177fdae3ede2}"
2025-01-26T02:43:31.499Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: aborting pipeline replication: peer="{Voter server-eabb1e89-846a-e534-a4f2-c3650ac40193 eabb1e89-846a-e534-a4f2-c3650ac40193}"
2025-01-26T02:43:31.499Z [INFO]  server-d83737e3-80e9-9241-3f6c-fa724ef257e7: aborting pipeline replication: peer="{Voter server-b2cd91dc-494f-5bc4-ff47-177fdae3ede2 b2cd91dc-494f-5bc4-ff47-177fdae3ede2}"
--- PASS: TestRaft_LeaderFail (0.65s)
=== RUN   TestRaft_BehindFollower
2025-01-26T02:43:31.500Z [INFO]  server-697a25a2-fa32-e145-1a10-58e11476111d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-697a25a2-fa32-e145-1a10-58e11476111d Address:697a25a2-fa32-e145-1a10-58e11476111d} {Suffrage:Voter ID:server-687981b4-6eb2-0bc4-d1e7-1a93bcdcea0e Address:687981b4-6eb2-0bc4-d1e7-1a93bcdcea0e} {Suffrage:Voter ID:server-27d241c7-2c4a-1792-07d1-1da072eb8d1c Address:27d241c7-2c4a-1792-07d1-1da072eb8d1c}]"
2025-01-26T02:43:31.500Z [INFO]  server-687981b4-6eb2-0bc4-d1e7-1a93bcdcea0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-697a25a2-fa32-e145-1a10-58e11476111d Address:697a25a2-fa32-e145-1a10-58e11476111d} {Suffrage:Voter ID:server-687981b4-6eb2-0bc4-d1e7-1a93bcdcea0e Address:687981b4-6eb2-0bc4-d1e7-1a93bcdcea0e} {Suffrage:Voter ID:server-27d241c7-2c4a-1792-07d1-1da072eb8d1c Address:27d241c7-2c4a-1792-07d1-1da072eb8d1c}]"
2025-01-26T02:43:31.500Z [INFO]  server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-697a25a2-fa32-e145-1a10-58e11476111d Address:697a25a2-fa32-e145-1a10-58e11476111d} {Suffrage:Voter ID:server-687981b4-6eb2-0bc4-d1e7-1a93bcdcea0e Address:687981b4-6eb2-0bc4-d1e7-1a93bcdcea0e} {Suffrage:Voter ID:server-27d241c7-2c4a-1792-07d1-1da072eb8d1c Address:27d241c7-2c4a-1792-07d1-1da072eb8d1c}]"
2025-01-26T02:43:31.500Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:31.500Z [INFO]  server-697a25a2-fa32-e145-1a10-58e11476111d: entering follower state: follower="Node at 697a25a2-fa32-e145-1a10-58e11476111d [Follower]" leader-address= leader-id=
2025-01-26T02:43:31.500Z [INFO]  server-687981b4-6eb2-0bc4-d1e7-1a93bcdcea0e: entering follower state: follower="Node at 687981b4-6eb2-0bc4-d1e7-1a93bcdcea0e [Follower]" leader-address= leader-id=
2025-01-26T02:43:31.500Z [INFO]  server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: entering follower state: follower="Node at 27d241c7-2c4a-1792-07d1-1da072eb8d1c [Follower]" leader-address= leader-id=
2025-01-26T02:43:31.559Z [WARN]  server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:31.559Z [INFO]  server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: entering candidate state: node="Node at 27d241c7-2c4a-1792-07d1-1da072eb8d1c [Candidate]" term=2
2025-01-26T02:43:31.559Z [INFO]  server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: election won: term=2 tally=2
2025-01-26T02:43:31.559Z [INFO]  server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: entering leader state: leader="Node at 27d241c7-2c4a-1792-07d1-1da072eb8d1c [Leader]"
2025-01-26T02:43:31.559Z [INFO]  server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: added peer, starting replication: peer=server-697a25a2-fa32-e145-1a10-58e11476111d
2025-01-26T02:43:31.559Z [INFO]  server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: added peer, starting replication: peer=server-687981b4-6eb2-0bc4-d1e7-1a93bcdcea0e
2025-01-26T02:43:31.559Z [INFO]  server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: pipelining replication: peer="{Voter server-687981b4-6eb2-0bc4-d1e7-1a93bcdcea0e 687981b4-6eb2-0bc4-d1e7-1a93bcdcea0e}"
2025-01-26T02:43:31.559Z [INFO]  server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: pipelining replication: peer="{Voter server-697a25a2-fa32-e145-1a10-58e11476111d 697a25a2-fa32-e145-1a10-58e11476111d}"
2025-01-26T02:43:31.667Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:31.559716263 +0000 UTC m=+8.074748598 (1 nodes), 59.426881ms from start of poll, 59.607322ms from cluster start. Timeout at 2025-01-26 02:43:31.664718263 +0000 UTC m=+8.179750598, 105.002ms after stability
2025-01-26T02:43:31.667Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:43:31.779Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:43:31.667516117 +0000 UTC m=+8.182548492 (2 nodes), 1.48µs from start of poll, 167.407216ms from cluster start. Timeout at 2025-01-26 02:43:31.772522837 +0000 UTC m=+8.287555212, 105.00672ms after stability
2025-01-26T02:43:31.779Z [ERROR] server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: failed to pipeline appendEntries: peer="{Voter server-697a25a2-fa32-e145-1a10-58e11476111d 697a25a2-fa32-e145-1a10-58e11476111d}" error="append pipeline closed"
2025-01-26T02:43:31.779Z [INFO]  server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: aborting pipeline replication: peer="{Voter server-697a25a2-fa32-e145-1a10-58e11476111d 697a25a2-fa32-e145-1a10-58e11476111d}"
2025-01-26T02:43:31.779Z [ERROR] server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: failed to appendEntries to: peer="{Voter server-697a25a2-fa32-e145-1a10-58e11476111d 697a25a2-fa32-e145-1a10-58e11476111d}" error="failed to connect to peer: 697a25a2-fa32-e145-1a10-58e11476111d"
2025-01-26T02:43:31.781Z [ERROR] server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: failed to heartbeat to: peer=697a25a2-fa32-e145-1a10-58e11476111d backoff time=10ms error="failed to connect to peer: 697a25a2-fa32-e145-1a10-58e11476111d"
2025-01-26T02:43:31.795Z [ERROR] server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: failed to appendEntries to: peer="{Voter server-697a25a2-fa32-e145-1a10-58e11476111d 697a25a2-fa32-e145-1a10-58e11476111d}" error="failed to connect to peer: 697a25a2-fa32-e145-1a10-58e11476111d"
    raft_test.go:459: [INFO] Finished apply without behind follower
2025-01-26T02:43:31.811Z [INFO]  server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: pipelining replication: peer="{Voter server-697a25a2-fa32-e145-1a10-58e11476111d 697a25a2-fa32-e145-1a10-58e11476111d}"
2025-01-26T02:43:31.815Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:31.923Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:31.815457615 +0000 UTC m=+8.330489990 (1 nodes), 2.481µs from start of poll, 315.348714ms from cluster start. Timeout at 2025-01-26 02:43:31.920462255 +0000 UTC m=+8.435494590, 105.0046ms after stability
2025-01-26T02:43:31.923Z [INFO]  server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: aborting pipeline replication: peer="{Voter server-687981b4-6eb2-0bc4-d1e7-1a93bcdcea0e 687981b4-6eb2-0bc4-d1e7-1a93bcdcea0e}"
2025-01-26T02:43:31.923Z [INFO]  server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: aborting pipeline replication: peer="{Voter server-697a25a2-fa32-e145-1a10-58e11476111d 697a25a2-fa32-e145-1a10-58e11476111d}"
2025-01-26T02:43:32.427Z [ERROR] server-27d241c7-2c4a-1792-07d1-1da072eb8d1c: failed to heartbeat to: peer=687981b4-6eb2-0bc4-d1e7-1a93bcdcea0e backoff time=10ms error="command timed out"
--- PASS: TestRaft_BehindFollower (0.93s)
=== RUN   TestRaft_ApplyNonLeader
2025-01-26T02:43:32.427Z [INFO]  server-afd087c8-e33c-64b7-aed4-7b544d2ad8f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afd087c8-e33c-64b7-aed4-7b544d2ad8f0 Address:afd087c8-e33c-64b7-aed4-7b544d2ad8f0} {Suffrage:Voter ID:server-744ee579-67c8-eb10-3408-0afd730719e6 Address:744ee579-67c8-eb10-3408-0afd730719e6} {Suffrage:Voter ID:server-d0e11126-1007-c8e8-caf4-75a9c5167f6b Address:d0e11126-1007-c8e8-caf4-75a9c5167f6b}]"
2025-01-26T02:43:32.427Z [INFO]  server-744ee579-67c8-eb10-3408-0afd730719e6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afd087c8-e33c-64b7-aed4-7b544d2ad8f0 Address:afd087c8-e33c-64b7-aed4-7b544d2ad8f0} {Suffrage:Voter ID:server-744ee579-67c8-eb10-3408-0afd730719e6 Address:744ee579-67c8-eb10-3408-0afd730719e6} {Suffrage:Voter ID:server-d0e11126-1007-c8e8-caf4-75a9c5167f6b Address:d0e11126-1007-c8e8-caf4-75a9c5167f6b}]"
2025-01-26T02:43:32.427Z [INFO]  server-d0e11126-1007-c8e8-caf4-75a9c5167f6b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afd087c8-e33c-64b7-aed4-7b544d2ad8f0 Address:afd087c8-e33c-64b7-aed4-7b544d2ad8f0} {Suffrage:Voter ID:server-744ee579-67c8-eb10-3408-0afd730719e6 Address:744ee579-67c8-eb10-3408-0afd730719e6} {Suffrage:Voter ID:server-d0e11126-1007-c8e8-caf4-75a9c5167f6b Address:d0e11126-1007-c8e8-caf4-75a9c5167f6b}]"
2025-01-26T02:43:32.427Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:32.427Z [INFO]  server-afd087c8-e33c-64b7-aed4-7b544d2ad8f0: entering follower state: follower="Node at afd087c8-e33c-64b7-aed4-7b544d2ad8f0 [Follower]" leader-address= leader-id=
2025-01-26T02:43:32.428Z [INFO]  server-744ee579-67c8-eb10-3408-0afd730719e6: entering follower state: follower="Node at 744ee579-67c8-eb10-3408-0afd730719e6 [Follower]" leader-address= leader-id=
2025-01-26T02:43:32.428Z [INFO]  server-d0e11126-1007-c8e8-caf4-75a9c5167f6b: entering follower state: follower="Node at d0e11126-1007-c8e8-caf4-75a9c5167f6b [Follower]" leader-address= leader-id=
2025-01-26T02:43:32.487Z [WARN]  server-afd087c8-e33c-64b7-aed4-7b544d2ad8f0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:32.487Z [INFO]  server-afd087c8-e33c-64b7-aed4-7b544d2ad8f0: entering candidate state: node="Node at afd087c8-e33c-64b7-aed4-7b544d2ad8f0 [Candidate]" term=2
2025-01-26T02:43:32.487Z [INFO]  server-afd087c8-e33c-64b7-aed4-7b544d2ad8f0: election won: term=2 tally=2
2025-01-26T02:43:32.487Z [INFO]  server-afd087c8-e33c-64b7-aed4-7b544d2ad8f0: entering leader state: leader="Node at afd087c8-e33c-64b7-aed4-7b544d2ad8f0 [Leader]"
2025-01-26T02:43:32.487Z [INFO]  server-afd087c8-e33c-64b7-aed4-7b544d2ad8f0: added peer, starting replication: peer=server-744ee579-67c8-eb10-3408-0afd730719e6
2025-01-26T02:43:32.487Z [INFO]  server-afd087c8-e33c-64b7-aed4-7b544d2ad8f0: added peer, starting replication: peer=server-d0e11126-1007-c8e8-caf4-75a9c5167f6b
2025-01-26T02:43:32.487Z [INFO]  server-afd087c8-e33c-64b7-aed4-7b544d2ad8f0: pipelining replication: peer="{Voter server-d0e11126-1007-c8e8-caf4-75a9c5167f6b d0e11126-1007-c8e8-caf4-75a9c5167f6b}"
2025-01-26T02:43:32.487Z [INFO]  server-afd087c8-e33c-64b7-aed4-7b544d2ad8f0: pipelining replication: peer="{Voter server-744ee579-67c8-eb10-3408-0afd730719e6 744ee579-67c8-eb10-3408-0afd730719e6}"
2025-01-26T02:43:32.599Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:32.487719934 +0000 UTC m=+9.002752309 (1 nodes), 59.749403ms from start of poll, 59.924404ms from cluster start. Timeout at 2025-01-26 02:43:32.592725414 +0000 UTC m=+9.107757789, 105.00548ms after stability
2025-01-26T02:43:32.599Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:43:32.707Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:43:32.599434292 +0000 UTC m=+9.114466667 (2 nodes), 2.28µs from start of poll, 171.638762ms from cluster start. Timeout at 2025-01-26 02:43:32.704441972 +0000 UTC m=+9.219474347, 105.00768ms after stability
2025-01-26T02:43:32.707Z [INFO]  server-afd087c8-e33c-64b7-aed4-7b544d2ad8f0: aborting pipeline replication: peer="{Voter server-d0e11126-1007-c8e8-caf4-75a9c5167f6b d0e11126-1007-c8e8-caf4-75a9c5167f6b}"
2025-01-26T02:43:32.707Z [INFO]  server-afd087c8-e33c-64b7-aed4-7b544d2ad8f0: aborting pipeline replication: peer="{Voter server-744ee579-67c8-eb10-3408-0afd730719e6 744ee579-67c8-eb10-3408-0afd730719e6}"
--- PASS: TestRaft_ApplyNonLeader (0.28s)
=== RUN   TestRaft_ApplyConcurrent
2025-01-26T02:43:32.707Z [INFO]  server-aa8f9869-60e7-4ad0-48c4-e28656c3059c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa8f9869-60e7-4ad0-48c4-e28656c3059c Address:aa8f9869-60e7-4ad0-48c4-e28656c3059c} {Suffrage:Voter ID:server-421f5109-21b1-25a7-7a6d-a1d0311b152b Address:421f5109-21b1-25a7-7a6d-a1d0311b152b} {Suffrage:Voter ID:server-57ca8f34-5508-553f-b1b3-d9097ad58602 Address:57ca8f34-5508-553f-b1b3-d9097ad58602}]"
2025-01-26T02:43:32.707Z [INFO]  server-421f5109-21b1-25a7-7a6d-a1d0311b152b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa8f9869-60e7-4ad0-48c4-e28656c3059c Address:aa8f9869-60e7-4ad0-48c4-e28656c3059c} {Suffrage:Voter ID:server-421f5109-21b1-25a7-7a6d-a1d0311b152b Address:421f5109-21b1-25a7-7a6d-a1d0311b152b} {Suffrage:Voter ID:server-57ca8f34-5508-553f-b1b3-d9097ad58602 Address:57ca8f34-5508-553f-b1b3-d9097ad58602}]"
2025-01-26T02:43:32.708Z [INFO]  server-57ca8f34-5508-553f-b1b3-d9097ad58602: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa8f9869-60e7-4ad0-48c4-e28656c3059c Address:aa8f9869-60e7-4ad0-48c4-e28656c3059c} {Suffrage:Voter ID:server-421f5109-21b1-25a7-7a6d-a1d0311b152b Address:421f5109-21b1-25a7-7a6d-a1d0311b152b} {Suffrage:Voter ID:server-57ca8f34-5508-553f-b1b3-d9097ad58602 Address:57ca8f34-5508-553f-b1b3-d9097ad58602}]"
2025-01-26T02:43:32.708Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:32.708Z [INFO]  server-aa8f9869-60e7-4ad0-48c4-e28656c3059c: entering follower state: follower="Node at aa8f9869-60e7-4ad0-48c4-e28656c3059c [Follower]" leader-address= leader-id=
2025-01-26T02:43:32.708Z [INFO]  server-421f5109-21b1-25a7-7a6d-a1d0311b152b: entering follower state: follower="Node at 421f5109-21b1-25a7-7a6d-a1d0311b152b [Follower]" leader-address= leader-id=
2025-01-26T02:43:32.708Z [INFO]  server-57ca8f34-5508-553f-b1b3-d9097ad58602: entering follower state: follower="Node at 57ca8f34-5508-553f-b1b3-d9097ad58602 [Follower]" leader-address= leader-id=
2025-01-26T02:43:32.843Z [WARN]  server-aa8f9869-60e7-4ad0-48c4-e28656c3059c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:32.843Z [INFO]  server-aa8f9869-60e7-4ad0-48c4-e28656c3059c: entering candidate state: node="Node at aa8f9869-60e7-4ad0-48c4-e28656c3059c [Candidate]" term=2
2025-01-26T02:43:32.843Z [INFO]  server-aa8f9869-60e7-4ad0-48c4-e28656c3059c: election won: term=2 tally=2
2025-01-26T02:43:32.843Z [INFO]  server-aa8f9869-60e7-4ad0-48c4-e28656c3059c: entering leader state: leader="Node at aa8f9869-60e7-4ad0-48c4-e28656c3059c [Leader]"
2025-01-26T02:43:32.843Z [INFO]  server-aa8f9869-60e7-4ad0-48c4-e28656c3059c: added peer, starting replication: peer=server-421f5109-21b1-25a7-7a6d-a1d0311b152b
2025-01-26T02:43:32.843Z [INFO]  server-aa8f9869-60e7-4ad0-48c4-e28656c3059c: added peer, starting replication: peer=server-57ca8f34-5508-553f-b1b3-d9097ad58602
2025-01-26T02:43:32.843Z [INFO]  server-aa8f9869-60e7-4ad0-48c4-e28656c3059c: pipelining replication: peer="{Voter server-57ca8f34-5508-553f-b1b3-d9097ad58602 57ca8f34-5508-553f-b1b3-d9097ad58602}"
2025-01-26T02:43:32.843Z [INFO]  server-aa8f9869-60e7-4ad0-48c4-e28656c3059c: pipelining replication: peer="{Voter server-421f5109-21b1-25a7-7a6d-a1d0311b152b 421f5109-21b1-25a7-7a6d-a1d0311b152b}"
2025-01-26T02:43:33.051Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:32.843730214 +0000 UTC m=+9.358762589 (1 nodes), 135.703503ms from start of poll, 135.882145ms from cluster start. Timeout at 2025-01-26 02:43:33.048732814 +0000 UTC m=+9.563765189, 205.0026ms after stability
2025-01-26T02:43:33.079Z [INFO]  server-aa8f9869-60e7-4ad0-48c4-e28656c3059c: aborting pipeline replication: peer="{Voter server-57ca8f34-5508-553f-b1b3-d9097ad58602 57ca8f34-5508-553f-b1b3-d9097ad58602}"
2025-01-26T02:43:33.079Z [INFO]  server-aa8f9869-60e7-4ad0-48c4-e28656c3059c: aborting pipeline replication: peer="{Voter server-421f5109-21b1-25a7-7a6d-a1d0311b152b 421f5109-21b1-25a7-7a6d-a1d0311b152b}"
--- PASS: TestRaft_ApplyConcurrent (0.37s)
=== RUN   TestRaft_ApplyConcurrent_Timeout
2025-01-26T02:43:33.079Z [INFO]  server-8e666930-ef57-820e-80de-60a6e3efc938: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e666930-ef57-820e-80de-60a6e3efc938 Address:8e666930-ef57-820e-80de-60a6e3efc938}]"
2025-01-26T02:43:33.079Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:33.079Z [INFO]  server-8e666930-ef57-820e-80de-60a6e3efc938: entering follower state: follower="Node at 8e666930-ef57-820e-80de-60a6e3efc938 [Follower]" leader-address= leader-id=
2025-01-26T02:43:33.211Z [WARN]  server-8e666930-ef57-820e-80de-60a6e3efc938: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:33.211Z [INFO]  server-8e666930-ef57-820e-80de-60a6e3efc938: entering candidate state: node="Node at 8e666930-ef57-820e-80de-60a6e3efc938 [Candidate]" term=2
2025-01-26T02:43:33.211Z [INFO]  server-8e666930-ef57-820e-80de-60a6e3efc938: election won: term=2 tally=1
2025-01-26T02:43:33.211Z [INFO]  server-8e666930-ef57-820e-80de-60a6e3efc938: entering leader state: leader="Node at 8e666930-ef57-820e-80de-60a6e3efc938 [Leader]"
2025-01-26T02:43:33.415Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:33.211508726 +0000 UTC m=+9.726541101 (1 nodes), 131.623559ms from start of poll, 131.71696ms from cluster start. Timeout at 2025-01-26 02:43:33.412517286 +0000 UTC m=+9.927549621, 201.00852ms after stability
--- PASS: TestRaft_ApplyConcurrent_Timeout (0.34s)
=== RUN   TestRaft_JoinNode
2025-01-26T02:43:33.417Z [INFO]  server-0f16fb8e-6feb-f918-575a-bc145e5ae0e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f16fb8e-6feb-f918-575a-bc145e5ae0e1 Address:0f16fb8e-6feb-f918-575a-bc145e5ae0e1} {Suffrage:Voter ID:server-3be3412a-c24c-6d57-e62f-3f5123c3dd08 Address:3be3412a-c24c-6d57-e62f-3f5123c3dd08}]"
2025-01-26T02:43:33.417Z [INFO]  server-3be3412a-c24c-6d57-e62f-3f5123c3dd08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f16fb8e-6feb-f918-575a-bc145e5ae0e1 Address:0f16fb8e-6feb-f918-575a-bc145e5ae0e1} {Suffrage:Voter ID:server-3be3412a-c24c-6d57-e62f-3f5123c3dd08 Address:3be3412a-c24c-6d57-e62f-3f5123c3dd08}]"
2025-01-26T02:43:33.418Z [INFO]  server-b7a2ffae-01bc-0f93-f08d-a1da545f693d: initial configuration: index=0 servers=[]
2025-01-26T02:43:33.418Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:33.418Z [INFO]  server-0f16fb8e-6feb-f918-575a-bc145e5ae0e1: entering follower state: follower="Node at 0f16fb8e-6feb-f918-575a-bc145e5ae0e1 [Follower]" leader-address= leader-id=
2025-01-26T02:43:33.418Z [INFO]  server-3be3412a-c24c-6d57-e62f-3f5123c3dd08: entering follower state: follower="Node at 3be3412a-c24c-6d57-e62f-3f5123c3dd08 [Follower]" leader-address= leader-id=
2025-01-26T02:43:33.418Z [INFO]  server-b7a2ffae-01bc-0f93-f08d-a1da545f693d: entering follower state: follower="Node at b7a2ffae-01bc-0f93-f08d-a1da545f693d [Follower]" leader-address= leader-id=
2025-01-26T02:43:33.495Z [WARN]  server-3be3412a-c24c-6d57-e62f-3f5123c3dd08: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:33.495Z [INFO]  server-3be3412a-c24c-6d57-e62f-3f5123c3dd08: entering candidate state: node="Node at 3be3412a-c24c-6d57-e62f-3f5123c3dd08 [Candidate]" term=2
2025-01-26T02:43:33.495Z [INFO]  server-3be3412a-c24c-6d57-e62f-3f5123c3dd08: election won: term=2 tally=2
2025-01-26T02:43:33.495Z [INFO]  server-3be3412a-c24c-6d57-e62f-3f5123c3dd08: entering leader state: leader="Node at 3be3412a-c24c-6d57-e62f-3f5123c3dd08 [Leader]"
2025-01-26T02:43:33.495Z [INFO]  server-3be3412a-c24c-6d57-e62f-3f5123c3dd08: added peer, starting replication: peer=server-0f16fb8e-6feb-f918-575a-bc145e5ae0e1
2025-01-26T02:43:33.495Z [INFO]  server-3be3412a-c24c-6d57-e62f-3f5123c3dd08: pipelining replication: peer="{Voter server-0f16fb8e-6feb-f918-575a-bc145e5ae0e1 0f16fb8e-6feb-f918-575a-bc145e5ae0e1}"
2025-01-26T02:43:33.519Z [WARN]  server-b7a2ffae-01bc-0f93-f08d-a1da545f693d: no known peers, aborting election
2025-01-26T02:43:33.603Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:33.49561449 +0000 UTC m=+10.010646825 (1 nodes), 77.555791ms from start of poll, 77.776632ms from cluster start. Timeout at 2025-01-26 02:43:33.60061729 +0000 UTC m=+10.115649665, 105.00284ms after stability
2025-01-26T02:43:33.603Z [INFO]  server-3be3412a-c24c-6d57-e62f-3f5123c3dd08: updating configuration: command=AddVoter server-id=server-b7a2ffae-01bc-0f93-f08d-a1da545f693d server-addr=b7a2ffae-01bc-0f93-f08d-a1da545f693d servers="[{Suffrage:Voter ID:server-0f16fb8e-6feb-f918-575a-bc145e5ae0e1 Address:0f16fb8e-6feb-f918-575a-bc145e5ae0e1} {Suffrage:Voter ID:server-3be3412a-c24c-6d57-e62f-3f5123c3dd08 Address:3be3412a-c24c-6d57-e62f-3f5123c3dd08} {Suffrage:Voter ID:server-b7a2ffae-01bc-0f93-f08d-a1da545f693d Address:b7a2ffae-01bc-0f93-f08d-a1da545f693d}]"
2025-01-26T02:43:33.603Z [INFO]  server-3be3412a-c24c-6d57-e62f-3f5123c3dd08: added peer, starting replication: peer=server-b7a2ffae-01bc-0f93-f08d-a1da545f693d
2025-01-26T02:43:33.603Z [WARN]  server-b7a2ffae-01bc-0f93-f08d-a1da545f693d: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-01-26T02:43:33.603Z [WARN]  server-3be3412a-c24c-6d57-e62f-3f5123c3dd08: appendEntries rejected, sending older logs: peer="{Voter server-b7a2ffae-01bc-0f93-f08d-a1da545f693d b7a2ffae-01bc-0f93-f08d-a1da545f693d}" next=1
2025-01-26T02:43:33.603Z [INFO]  server-3be3412a-c24c-6d57-e62f-3f5123c3dd08: pipelining replication: peer="{Voter server-b7a2ffae-01bc-0f93-f08d-a1da545f693d b7a2ffae-01bc-0f93-f08d-a1da545f693d}"
2025-01-26T02:43:33.603Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:33.711Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:33.603740786 +0000 UTC m=+10.118773121 (1 nodes), 440ns from start of poll, 185.902928ms from cluster start. Timeout at 2025-01-26 02:43:33.708744146 +0000 UTC m=+10.223776521, 105.0034ms after stability
2025-01-26T02:43:33.711Z [INFO]  server-3be3412a-c24c-6d57-e62f-3f5123c3dd08: aborting pipeline replication: peer="{Voter server-b7a2ffae-01bc-0f93-f08d-a1da545f693d b7a2ffae-01bc-0f93-f08d-a1da545f693d}"
2025-01-26T02:43:33.711Z [INFO]  server-3be3412a-c24c-6d57-e62f-3f5123c3dd08: aborting pipeline replication: peer="{Voter server-0f16fb8e-6feb-f918-575a-bc145e5ae0e1 0f16fb8e-6feb-f918-575a-bc145e5ae0e1}"
--- PASS: TestRaft_JoinNode (0.29s)
=== RUN   TestRaft_JoinNode_ConfigStore
2025-01-26T02:43:33.711Z [INFO]  server-73174c35-6539-6242-d8b6-b3c23b1ebcab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73174c35-6539-6242-d8b6-b3c23b1ebcab Address:73174c35-6539-6242-d8b6-b3c23b1ebcab}]"
2025-01-26T02:43:33.711Z [INFO]  server-404294aa-2532-7265-f894-9d5518709db4: initial configuration: index=0 servers=[]
2025-01-26T02:43:33.712Z [INFO]  server-8c88c7eb-3541-649f-223d-001188913f4e: initial configuration: index=0 servers=[]
2025-01-26T02:43:33.712Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:33.712Z [INFO]  server-73174c35-6539-6242-d8b6-b3c23b1ebcab: entering follower state: follower="Node at 73174c35-6539-6242-d8b6-b3c23b1ebcab [Follower]" leader-address= leader-id=
2025-01-26T02:43:33.712Z [INFO]  server-404294aa-2532-7265-f894-9d5518709db4: entering follower state: follower="Node at 404294aa-2532-7265-f894-9d5518709db4 [Follower]" leader-address= leader-id=
2025-01-26T02:43:33.712Z [INFO]  server-8c88c7eb-3541-649f-223d-001188913f4e: entering follower state: follower="Node at 8c88c7eb-3541-649f-223d-001188913f4e [Follower]" leader-address= leader-id=
2025-01-26T02:43:33.787Z [WARN]  server-8c88c7eb-3541-649f-223d-001188913f4e: no known peers, aborting election
2025-01-26T02:43:33.795Z [WARN]  server-73174c35-6539-6242-d8b6-b3c23b1ebcab: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:33.795Z [INFO]  server-73174c35-6539-6242-d8b6-b3c23b1ebcab: entering candidate state: node="Node at 73174c35-6539-6242-d8b6-b3c23b1ebcab [Candidate]" term=2
2025-01-26T02:43:33.795Z [INFO]  server-73174c35-6539-6242-d8b6-b3c23b1ebcab: election won: term=2 tally=1
2025-01-26T02:43:33.795Z [INFO]  server-73174c35-6539-6242-d8b6-b3c23b1ebcab: entering leader state: leader="Node at 73174c35-6539-6242-d8b6-b3c23b1ebcab [Leader]"
2025-01-26T02:43:33.807Z [WARN]  server-404294aa-2532-7265-f894-9d5518709db4: no known peers, aborting election
2025-01-26T02:43:33.903Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:33.79555863 +0000 UTC m=+10.310591005 (1 nodes), 83.491107ms from start of poll, 83.704428ms from cluster start. Timeout at 2025-01-26 02:43:33.90056127 +0000 UTC m=+10.415593685, 105.00268ms after stability
2025-01-26T02:43:33.903Z [INFO]  server-73174c35-6539-6242-d8b6-b3c23b1ebcab: updating configuration: command=AddVoter server-id=server-404294aa-2532-7265-f894-9d5518709db4 server-addr=404294aa-2532-7265-f894-9d5518709db4 servers="[{Suffrage:Voter ID:server-73174c35-6539-6242-d8b6-b3c23b1ebcab Address:73174c35-6539-6242-d8b6-b3c23b1ebcab} {Suffrage:Voter ID:server-404294aa-2532-7265-f894-9d5518709db4 Address:404294aa-2532-7265-f894-9d5518709db4}]"
2025-01-26T02:43:33.903Z [INFO]  server-73174c35-6539-6242-d8b6-b3c23b1ebcab: added peer, starting replication: peer=server-404294aa-2532-7265-f894-9d5518709db4
2025-01-26T02:43:33.903Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:33.903Z [WARN]  server-404294aa-2532-7265-f894-9d5518709db4: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-01-26T02:43:33.903Z [WARN]  server-73174c35-6539-6242-d8b6-b3c23b1ebcab: appendEntries rejected, sending older logs: peer="{Voter server-404294aa-2532-7265-f894-9d5518709db4 404294aa-2532-7265-f894-9d5518709db4}" next=1
2025-01-26T02:43:33.903Z [INFO]  server-73174c35-6539-6242-d8b6-b3c23b1ebcab: pipelining replication: peer="{Voter server-404294aa-2532-7265-f894-9d5518709db4 404294aa-2532-7265-f894-9d5518709db4}"
2025-01-26T02:43:34.011Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:33.903542565 +0000 UTC m=+10.418574940 (1 nodes), 560ns from start of poll, 191.688363ms from cluster start. Timeout at 2025-01-26 02:43:34.008549205 +0000 UTC m=+10.523581580, 105.00664ms after stability
2025-01-26T02:43:34.011Z [INFO]  server-73174c35-6539-6242-d8b6-b3c23b1ebcab: updating configuration: command=AddVoter server-id=server-8c88c7eb-3541-649f-223d-001188913f4e server-addr=8c88c7eb-3541-649f-223d-001188913f4e servers="[{Suffrage:Voter ID:server-73174c35-6539-6242-d8b6-b3c23b1ebcab Address:73174c35-6539-6242-d8b6-b3c23b1ebcab} {Suffrage:Voter ID:server-404294aa-2532-7265-f894-9d5518709db4 Address:404294aa-2532-7265-f894-9d5518709db4} {Suffrage:Voter ID:server-8c88c7eb-3541-649f-223d-001188913f4e Address:8c88c7eb-3541-649f-223d-001188913f4e}]"
2025-01-26T02:43:34.011Z [INFO]  server-73174c35-6539-6242-d8b6-b3c23b1ebcab: added peer, starting replication: peer=server-8c88c7eb-3541-649f-223d-001188913f4e
2025-01-26T02:43:34.011Z [WARN]  server-8c88c7eb-3541-649f-223d-001188913f4e: failed to get previous log: previous-index=4 last-index=0 error="log not found"
2025-01-26T02:43:34.011Z [WARN]  server-73174c35-6539-6242-d8b6-b3c23b1ebcab: appendEntries rejected, sending older logs: peer="{Voter server-8c88c7eb-3541-649f-223d-001188913f4e 8c88c7eb-3541-649f-223d-001188913f4e}" next=1
2025-01-26T02:43:34.011Z [INFO]  server-73174c35-6539-6242-d8b6-b3c23b1ebcab: pipelining replication: peer="{Voter server-8c88c7eb-3541-649f-223d-001188913f4e 8c88c7eb-3541-649f-223d-001188913f4e}"
2025-01-26T02:43:34.011Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:34.119Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:34.011747701 +0000 UTC m=+10.526780076 (1 nodes), 440ns from start of poll, 299.893499ms from cluster start. Timeout at 2025-01-26 02:43:34.116752461 +0000 UTC m=+10.631784836, 105.00476ms after stability
2025-01-26T02:43:34.119Z [INFO]  server-73174c35-6539-6242-d8b6-b3c23b1ebcab: aborting pipeline replication: peer="{Voter server-404294aa-2532-7265-f894-9d5518709db4 404294aa-2532-7265-f894-9d5518709db4}"
2025-01-26T02:43:34.119Z [INFO]  server-73174c35-6539-6242-d8b6-b3c23b1ebcab: aborting pipeline replication: peer="{Voter server-8c88c7eb-3541-649f-223d-001188913f4e 8c88c7eb-3541-649f-223d-001188913f4e}"
--- PASS: TestRaft_JoinNode_ConfigStore (0.41s)
=== RUN   TestRaft_RemoveFollower
2025-01-26T02:43:34.119Z [INFO]  server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2 Address:5f97cc1e-31a6-cc32-0d65-bca87bb838a2} {Suffrage:Voter ID:server-e60c71c9-47b7-57f8-444f-ed7d2eb84aab Address:e60c71c9-47b7-57f8-444f-ed7d2eb84aab} {Suffrage:Voter ID:server-50e7d41f-31bc-a632-9655-073c7cdfbcf3 Address:50e7d41f-31bc-a632-9655-073c7cdfbcf3}]"
2025-01-26T02:43:34.119Z [INFO]  server-e60c71c9-47b7-57f8-444f-ed7d2eb84aab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2 Address:5f97cc1e-31a6-cc32-0d65-bca87bb838a2} {Suffrage:Voter ID:server-e60c71c9-47b7-57f8-444f-ed7d2eb84aab Address:e60c71c9-47b7-57f8-444f-ed7d2eb84aab} {Suffrage:Voter ID:server-50e7d41f-31bc-a632-9655-073c7cdfbcf3 Address:50e7d41f-31bc-a632-9655-073c7cdfbcf3}]"
2025-01-26T02:43:34.120Z [INFO]  server-50e7d41f-31bc-a632-9655-073c7cdfbcf3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2 Address:5f97cc1e-31a6-cc32-0d65-bca87bb838a2} {Suffrage:Voter ID:server-e60c71c9-47b7-57f8-444f-ed7d2eb84aab Address:e60c71c9-47b7-57f8-444f-ed7d2eb84aab} {Suffrage:Voter ID:server-50e7d41f-31bc-a632-9655-073c7cdfbcf3 Address:50e7d41f-31bc-a632-9655-073c7cdfbcf3}]"
2025-01-26T02:43:34.120Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:34.120Z [INFO]  server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2: entering follower state: follower="Node at 5f97cc1e-31a6-cc32-0d65-bca87bb838a2 [Follower]" leader-address= leader-id=
2025-01-26T02:43:34.120Z [INFO]  server-e60c71c9-47b7-57f8-444f-ed7d2eb84aab: entering follower state: follower="Node at e60c71c9-47b7-57f8-444f-ed7d2eb84aab [Follower]" leader-address= leader-id=
2025-01-26T02:43:34.120Z [INFO]  server-50e7d41f-31bc-a632-9655-073c7cdfbcf3: entering follower state: follower="Node at 50e7d41f-31bc-a632-9655-073c7cdfbcf3 [Follower]" leader-address= leader-id=
2025-01-26T02:43:34.183Z [WARN]  server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:34.183Z [INFO]  server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2: entering candidate state: node="Node at 5f97cc1e-31a6-cc32-0d65-bca87bb838a2 [Candidate]" term=2
2025-01-26T02:43:34.183Z [INFO]  server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2: election won: term=2 tally=2
2025-01-26T02:43:34.183Z [INFO]  server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2: entering leader state: leader="Node at 5f97cc1e-31a6-cc32-0d65-bca87bb838a2 [Leader]"
2025-01-26T02:43:34.183Z [INFO]  server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2: added peer, starting replication: peer=server-e60c71c9-47b7-57f8-444f-ed7d2eb84aab
2025-01-26T02:43:34.183Z [INFO]  server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2: added peer, starting replication: peer=server-50e7d41f-31bc-a632-9655-073c7cdfbcf3
2025-01-26T02:43:34.183Z [INFO]  server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2: pipelining replication: peer="{Voter server-50e7d41f-31bc-a632-9655-073c7cdfbcf3 50e7d41f-31bc-a632-9655-073c7cdfbcf3}"
2025-01-26T02:43:34.183Z [INFO]  server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2: pipelining replication: peer="{Voter server-e60c71c9-47b7-57f8-444f-ed7d2eb84aab e60c71c9-47b7-57f8-444f-ed7d2eb84aab}"
2025-01-26T02:43:34.295Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:34.183750305 +0000 UTC m=+10.698782680 (1 nodes), 63.689386ms from start of poll, 63.869747ms from cluster start. Timeout at 2025-01-26 02:43:34.288753745 +0000 UTC m=+10.803786120, 105.00344ms after stability
2025-01-26T02:43:34.303Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:43:34.411Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:43:34.303455351 +0000 UTC m=+10.818487726 (2 nodes), 840ns from start of poll, 183.574793ms from cluster start. Timeout at 2025-01-26 02:43:34.408467031 +0000 UTC m=+10.923499406, 105.01168ms after stability
2025-01-26T02:43:34.411Z [INFO]  server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2: updating configuration: command=RemoveServer server-id=server-e60c71c9-47b7-57f8-444f-ed7d2eb84aab server-addr= servers="[{Suffrage:Voter ID:server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2 Address:5f97cc1e-31a6-cc32-0d65-bca87bb838a2} {Suffrage:Voter ID:server-50e7d41f-31bc-a632-9655-073c7cdfbcf3 Address:50e7d41f-31bc-a632-9655-073c7cdfbcf3}]"
2025-01-26T02:43:34.411Z [INFO]  server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2: removed peer, stopping replication: peer=server-e60c71c9-47b7-57f8-444f-ed7d2eb84aab last-index=3
2025-01-26T02:43:34.411Z [INFO]  server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2: aborting pipeline replication: peer="{Voter server-e60c71c9-47b7-57f8-444f-ed7d2eb84aab e60c71c9-47b7-57f8-444f-ed7d2eb84aab}"
2025-01-26T02:43:34.495Z [WARN]  server-e60c71c9-47b7-57f8-444f-ed7d2eb84aab: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-01-26T02:43:34.519Z [INFO]  server-5f97cc1e-31a6-cc32-0d65-bca87bb838a2: aborting pipeline replication: peer="{Voter server-50e7d41f-31bc-a632-9655-073c7cdfbcf3 50e7d41f-31bc-a632-9655-073c7cdfbcf3}"
--- PASS: TestRaft_RemoveFollower (0.40s)
=== RUN   TestRaft_RemoveLeader
2025-01-26T02:43:34.520Z [INFO]  server-6129b84e-16a2-8ef6-562e-9b6543affff0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6129b84e-16a2-8ef6-562e-9b6543affff0 Address:6129b84e-16a2-8ef6-562e-9b6543affff0} {Suffrage:Voter ID:server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d Address:5f03c804-0bca-3c06-5cb0-75dc9d7bb80d} {Suffrage:Voter ID:server-81608327-342e-4336-a5f6-73c79cfb0815 Address:81608327-342e-4336-a5f6-73c79cfb0815}]"
2025-01-26T02:43:34.520Z [INFO]  server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6129b84e-16a2-8ef6-562e-9b6543affff0 Address:6129b84e-16a2-8ef6-562e-9b6543affff0} {Suffrage:Voter ID:server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d Address:5f03c804-0bca-3c06-5cb0-75dc9d7bb80d} {Suffrage:Voter ID:server-81608327-342e-4336-a5f6-73c79cfb0815 Address:81608327-342e-4336-a5f6-73c79cfb0815}]"
2025-01-26T02:43:34.520Z [INFO]  server-81608327-342e-4336-a5f6-73c79cfb0815: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6129b84e-16a2-8ef6-562e-9b6543affff0 Address:6129b84e-16a2-8ef6-562e-9b6543affff0} {Suffrage:Voter ID:server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d Address:5f03c804-0bca-3c06-5cb0-75dc9d7bb80d} {Suffrage:Voter ID:server-81608327-342e-4336-a5f6-73c79cfb0815 Address:81608327-342e-4336-a5f6-73c79cfb0815}]"
2025-01-26T02:43:34.520Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:34.520Z [INFO]  server-6129b84e-16a2-8ef6-562e-9b6543affff0: entering follower state: follower="Node at 6129b84e-16a2-8ef6-562e-9b6543affff0 [Follower]" leader-address= leader-id=
2025-01-26T02:43:34.520Z [INFO]  server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d: entering follower state: follower="Node at 5f03c804-0bca-3c06-5cb0-75dc9d7bb80d [Follower]" leader-address= leader-id=
2025-01-26T02:43:34.520Z [INFO]  server-81608327-342e-4336-a5f6-73c79cfb0815: entering follower state: follower="Node at 81608327-342e-4336-a5f6-73c79cfb0815 [Follower]" leader-address= leader-id=
2025-01-26T02:43:34.587Z [WARN]  server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:34.587Z [INFO]  server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d: entering candidate state: node="Node at 5f03c804-0bca-3c06-5cb0-75dc9d7bb80d [Candidate]" term=2
2025-01-26T02:43:34.587Z [INFO]  server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d: election won: term=2 tally=2
2025-01-26T02:43:34.587Z [INFO]  server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d: entering leader state: leader="Node at 5f03c804-0bca-3c06-5cb0-75dc9d7bb80d [Leader]"
2025-01-26T02:43:34.587Z [INFO]  server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d: added peer, starting replication: peer=server-6129b84e-16a2-8ef6-562e-9b6543affff0
2025-01-26T02:43:34.587Z [INFO]  server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d: added peer, starting replication: peer=server-81608327-342e-4336-a5f6-73c79cfb0815
2025-01-26T02:43:34.587Z [INFO]  server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d: pipelining replication: peer="{Voter server-81608327-342e-4336-a5f6-73c79cfb0815 81608327-342e-4336-a5f6-73c79cfb0815}"
2025-01-26T02:43:34.587Z [INFO]  server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d: pipelining replication: peer="{Voter server-6129b84e-16a2-8ef6-562e-9b6543affff0 6129b84e-16a2-8ef6-562e-9b6543affff0}"
2025-01-26T02:43:34.695Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:34.588005158 +0000 UTC m=+11.103037533 (1 nodes), 67.862932ms from start of poll, 68.049093ms from cluster start. Timeout at 2025-01-26 02:43:34.693010878 +0000 UTC m=+11.208043253, 105.00572ms after stability
2025-01-26T02:43:34.703Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:43:34.811Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:43:34.703423378 +0000 UTC m=+11.218455753 (2 nodes), 4.56µs from start of poll, 183.467313ms from cluster start. Timeout at 2025-01-26 02:43:34.808428858 +0000 UTC m=+11.323461233, 105.00548ms after stability
2025-01-26T02:43:34.811Z [INFO]  server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d: updating configuration: command=RemoveServer server-id=server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d server-addr= servers="[{Suffrage:Voter ID:server-6129b84e-16a2-8ef6-562e-9b6543affff0 Address:6129b84e-16a2-8ef6-562e-9b6543affff0} {Suffrage:Voter ID:server-81608327-342e-4336-a5f6-73c79cfb0815 Address:81608327-342e-4336-a5f6-73c79cfb0815}]"
2025-01-26T02:43:34.811Z [INFO]  server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d: removed ourself, shutting down
2025-01-26T02:43:34.811Z [INFO]  server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d: aborting pipeline replication: peer="{Voter server-81608327-342e-4336-a5f6-73c79cfb0815 81608327-342e-4336-a5f6-73c79cfb0815}"
2025-01-26T02:43:34.811Z [INFO]  server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d: aborting pipeline replication: peer="{Voter server-6129b84e-16a2-8ef6-562e-9b6543affff0 6129b84e-16a2-8ef6-562e-9b6543affff0}"
2025-01-26T02:43:34.927Z [WARN]  server-6129b84e-16a2-8ef6-562e-9b6543affff0: heartbeat timeout reached, starting election: last-leader-addr=5f03c804-0bca-3c06-5cb0-75dc9d7bb80d last-leader-id=server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d
2025-01-26T02:43:34.927Z [INFO]  server-6129b84e-16a2-8ef6-562e-9b6543affff0: entering candidate state: node="Node at 6129b84e-16a2-8ef6-562e-9b6543affff0 [Candidate]" term=3
2025-01-26T02:43:34.927Z [WARN]  server-81608327-342e-4336-a5f6-73c79cfb0815: rejecting vote request since we have a leader: from=6129b84e-16a2-8ef6-562e-9b6543affff0 leader=5f03c804-0bca-3c06-5cb0-75dc9d7bb80d leader-id=server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d
2025-01-26T02:43:34.927Z [WARN]  server-81608327-342e-4336-a5f6-73c79cfb0815: heartbeat timeout reached, starting election: last-leader-addr=5f03c804-0bca-3c06-5cb0-75dc9d7bb80d last-leader-id=server-5f03c804-0bca-3c06-5cb0-75dc9d7bb80d
2025-01-26T02:43:34.927Z [INFO]  server-81608327-342e-4336-a5f6-73c79cfb0815: entering candidate state: node="Node at 81608327-342e-4336-a5f6-73c79cfb0815 [Candidate]" term=3
2025-01-26T02:43:34.927Z [INFO]  server-6129b84e-16a2-8ef6-562e-9b6543affff0: duplicate requestVote for same term: term=3
2025-01-26T02:43:35.015Z [WARN]  server-6129b84e-16a2-8ef6-562e-9b6543affff0: Election timeout reached, restarting election
2025-01-26T02:43:35.015Z [INFO]  server-6129b84e-16a2-8ef6-562e-9b6543affff0: entering candidate state: node="Node at 6129b84e-16a2-8ef6-562e-9b6543affff0 [Candidate]" term=4
2025-01-26T02:43:35.015Z [INFO]  server-81608327-342e-4336-a5f6-73c79cfb0815: entering follower state: follower="Node at 81608327-342e-4336-a5f6-73c79cfb0815 [Follower]" leader-address= leader-id=
2025-01-26T02:43:35.015Z [INFO]  server-6129b84e-16a2-8ef6-562e-9b6543affff0: election won: term=4 tally=2
2025-01-26T02:43:35.015Z [INFO]  server-6129b84e-16a2-8ef6-562e-9b6543affff0: entering leader state: leader="Node at 6129b84e-16a2-8ef6-562e-9b6543affff0 [Leader]"
2025-01-26T02:43:35.015Z [INFO]  server-6129b84e-16a2-8ef6-562e-9b6543affff0: added peer, starting replication: peer=server-81608327-342e-4336-a5f6-73c79cfb0815
2025-01-26T02:43:35.015Z [INFO]  server-6129b84e-16a2-8ef6-562e-9b6543affff0: pipelining replication: peer="{Voter server-81608327-342e-4336-a5f6-73c79cfb0815 81608327-342e-4336-a5f6-73c79cfb0815}"
2025-01-26T02:43:35.027Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:35.135Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:35.027450185 +0000 UTC m=+11.542482560 (1 nodes), 31.481µs from start of poll, 507.49412ms from cluster start. Timeout at 2025-01-26 02:43:35.132454465 +0000 UTC m=+11.647486840, 105.00428ms after stability
2025-01-26T02:43:35.135Z [INFO]  server-6129b84e-16a2-8ef6-562e-9b6543affff0: aborting pipeline replication: peer="{Voter server-81608327-342e-4336-a5f6-73c79cfb0815 81608327-342e-4336-a5f6-73c79cfb0815}"
--- PASS: TestRaft_RemoveLeader (0.62s)
=== RUN   TestRaft_RemoveLeader_NoShutdown
2025-01-26T02:43:35.136Z [INFO]  server-34a0de46-11a7-40bb-22fa-9aa69faca156: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34a0de46-11a7-40bb-22fa-9aa69faca156 Address:34a0de46-11a7-40bb-22fa-9aa69faca156} {Suffrage:Voter ID:server-bcb18c17-73be-2d6d-36b2-05cc37c5c172 Address:bcb18c17-73be-2d6d-36b2-05cc37c5c172} {Suffrage:Voter ID:server-f02b3030-530a-b140-bd1a-c3381b3656bc Address:f02b3030-530a-b140-bd1a-c3381b3656bc}]"
2025-01-26T02:43:35.136Z [INFO]  server-bcb18c17-73be-2d6d-36b2-05cc37c5c172: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34a0de46-11a7-40bb-22fa-9aa69faca156 Address:34a0de46-11a7-40bb-22fa-9aa69faca156} {Suffrage:Voter ID:server-bcb18c17-73be-2d6d-36b2-05cc37c5c172 Address:bcb18c17-73be-2d6d-36b2-05cc37c5c172} {Suffrage:Voter ID:server-f02b3030-530a-b140-bd1a-c3381b3656bc Address:f02b3030-530a-b140-bd1a-c3381b3656bc}]"
2025-01-26T02:43:35.136Z [INFO]  server-f02b3030-530a-b140-bd1a-c3381b3656bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34a0de46-11a7-40bb-22fa-9aa69faca156 Address:34a0de46-11a7-40bb-22fa-9aa69faca156} {Suffrage:Voter ID:server-bcb18c17-73be-2d6d-36b2-05cc37c5c172 Address:bcb18c17-73be-2d6d-36b2-05cc37c5c172} {Suffrage:Voter ID:server-f02b3030-530a-b140-bd1a-c3381b3656bc Address:f02b3030-530a-b140-bd1a-c3381b3656bc}]"
2025-01-26T02:43:35.136Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:43:35.137Z [INFO]  server-34a0de46-11a7-40bb-22fa-9aa69faca156: entering follower state: follower="Node at 34a0de46-11a7-40bb-22fa-9aa69faca156 [Follower]" leader-address= leader-id=
2025-01-26T02:43:35.137Z [INFO]  server-bcb18c17-73be-2d6d-36b2-05cc37c5c172: entering follower state: follower="Node at bcb18c17-73be-2d6d-36b2-05cc37c5c172 [Follower]" leader-address= leader-id=
2025-01-26T02:43:35.137Z [INFO]  server-f02b3030-530a-b140-bd1a-c3381b3656bc: entering follower state: follower="Node at f02b3030-530a-b140-bd1a-c3381b3656bc [Follower]" leader-address= leader-id=
2025-01-26T02:43:35.203Z [WARN]  server-34a0de46-11a7-40bb-22fa-9aa69faca156: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:35.203Z [INFO]  server-34a0de46-11a7-40bb-22fa-9aa69faca156: entering candidate state: node="Node at 34a0de46-11a7-40bb-22fa-9aa69faca156 [Candidate]" term=2
2025-01-26T02:43:35.203Z [INFO]  server-34a0de46-11a7-40bb-22fa-9aa69faca156: election won: term=2 tally=2
2025-01-26T02:43:35.203Z [INFO]  server-34a0de46-11a7-40bb-22fa-9aa69faca156: entering leader state: leader="Node at 34a0de46-11a7-40bb-22fa-9aa69faca156 [Leader]"
2025-01-26T02:43:35.203Z [INFO]  server-34a0de46-11a7-40bb-22fa-9aa69faca156: added peer, starting replication: peer=server-bcb18c17-73be-2d6d-36b2-05cc37c5c172
2025-01-26T02:43:35.203Z [INFO]  server-34a0de46-11a7-40bb-22fa-9aa69faca156: added peer, starting replication: peer=server-f02b3030-530a-b140-bd1a-c3381b3656bc
2025-01-26T02:43:35.203Z [INFO]  server-34a0de46-11a7-40bb-22fa-9aa69faca156: pipelining replication: peer="{Voter server-f02b3030-530a-b140-bd1a-c3381b3656bc f02b3030-530a-b140-bd1a-c3381b3656bc}"
2025-01-26T02:43:35.203Z [WARN]  server-bcb18c17-73be-2d6d-36b2-05cc37c5c172: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:35.203Z [INFO]  server-bcb18c17-73be-2d6d-36b2-05cc37c5c172: entering candidate state: node="Node at bcb18c17-73be-2d6d-36b2-05cc37c5c172 [Candidate]" term=2
2025-01-26T02:43:35.203Z [WARN]  server-f02b3030-530a-b140-bd1a-c3381b3656bc: rejecting vote request since we have a leader: from=bcb18c17-73be-2d6d-36b2-05cc37c5c172 leader=34a0de46-11a7-40bb-22fa-9aa69faca156 leader-id=server-34a0de46-11a7-40bb-22fa-9aa69faca156
2025-01-26T02:43:35.203Z [INFO]  server-bcb18c17-73be-2d6d-36b2-05cc37c5c172: duplicate requestVote for same term: term=2
2025-01-26T02:43:35.203Z [INFO]  server-bcb18c17-73be-2d6d-36b2-05cc37c5c172: entering follower state: follower="Node at bcb18c17-73be-2d6d-36b2-05cc37c5c172 [Follower]" leader-address=34a0de46-11a7-40bb-22fa-9aa69faca156 leader-id=server-34a0de46-11a7-40bb-22fa-9aa69faca156
2025-01-26T02:43:35.203Z [INFO]  server-34a0de46-11a7-40bb-22fa-9aa69faca156: pipelining replication: peer="{Voter server-bcb18c17-73be-2d6d-36b2-05cc37c5c172 bcb18c17-73be-2d6d-36b2-05cc37c5c172}"
2025-01-26T02:43:35.203Z [WARN]  server-34a0de46-11a7-40bb-22fa-9aa69faca156: rejecting vote request since we have a leader: from=bcb18c17-73be-2d6d-36b2-05cc37c5c172 leader=34a0de46-11a7-40bb-22fa-9aa69faca156 leader-id=server-34a0de46-11a7-40bb-22fa-9aa69faca156
2025-01-26T02:43:35.311Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:43:35.203848095 +0000 UTC m=+11.718880470 (2 nodes), 67.49561ms from start of poll, 67.922932ms from cluster start. Timeout at 2025-01-26 02:43:35.308849175 +0000 UTC m=+11.823881550, 105.00108ms after stability
2025-01-26T02:43:35.311Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:35.419Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:35.311553469 +0000 UTC m=+11.826585844 (1 nodes), 1.081µs from start of poll, 175.628306ms from cluster start. Timeout at 2025-01-26 02:43:35.416561109 +0000 UTC m=+11.931593484, 105.00764ms after stability
2025-01-26T02:43:35.422Z [INFO]  server-34a0de46-11a7-40bb-22fa-9aa69faca156: updating configuration: command=RemoveServer server-id=server-34a0de46-11a7-40bb-22fa-9aa69faca156 server-addr= servers="[{Suffrage:Voter ID:server-bcb18c17-73be-2d6d-36b2-05cc37c5c172 Address:bcb18c17-73be-2d6d-36b2-05cc37c5c172} {Suffrage:Voter ID:server-f02b3030-530a-b140-bd1a-c3381b3656bc Address:f02b3030-530a-b140-bd1a-c3381b3656bc}]"
2025-01-26T02:43:35.422Z [INFO]  server-34a0de46-11a7-40bb-22fa-9aa69faca156: removed ourself, transitioning to follower
2025-01-26T02:43:35.422Z [INFO]  server-34a0de46-11a7-40bb-22fa-9aa69faca156: entering follower state: follower="Node at 34a0de46-11a7-40bb-22fa-9aa69faca156 [Follower]" leader-address= leader-id=
2025-01-26T02:43:35.422Z [INFO]  server-34a0de46-11a7-40bb-22fa-9aa69faca156: aborting pipeline replication: peer="{Voter server-f02b3030-530a-b140-bd1a-c3381b3656bc f02b3030-530a-b140-bd1a-c3381b3656bc}"
2025-01-26T02:43:35.422Z [INFO]  server-34a0de46-11a7-40bb-22fa-9aa69faca156: aborting pipeline replication: peer="{Voter server-bcb18c17-73be-2d6d-36b2-05cc37c5c172 bcb18c17-73be-2d6d-36b2-05cc37c5c172}"
2025-01-26T02:43:35.507Z [WARN]  server-34a0de46-11a7-40bb-22fa-9aa69faca156: not part of stable configuration, aborting election
2025-01-26T02:43:35.515Z [WARN]  server-bcb18c17-73be-2d6d-36b2-05cc37c5c172: heartbeat timeout reached, starting election: last-leader-addr=34a0de46-11a7-40bb-22fa-9aa69faca156 last-leader-id=server-34a0de46-11a7-40bb-22fa-9aa69faca156
2025-01-26T02:43:35.515Z [INFO]  server-bcb18c17-73be-2d6d-36b2-05cc37c5c172: entering candidate state: node="Node at bcb18c17-73be-2d6d-36b2-05cc37c5c172 [Candidate]" term=3
2025-01-26T02:43:35.515Z [WARN]  server-f02b3030-530a-b140-bd1a-c3381b3656bc: rejecting vote request since we have a leader: from=bcb18c17-73be-2d6d-36b2-05cc37c5c172 leader=34a0de46-11a7-40bb-22fa-9aa69faca156 leader-id=server-34a0de46-11a7-40bb-22fa-9aa69faca156
2025-01-26T02:43:35.531Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:35.555Z [WARN]  server-f02b3030-530a-b140-bd1a-c3381b3656bc: heartbeat timeout reached, starting election: last-leader-addr=34a0de46-11a7-40bb-22fa-9aa69faca156 last-leader-id=server-34a0de46-11a7-40bb-22fa-9aa69faca156
2025-01-26T02:43:35.555Z [INFO]  server-f02b3030-530a-b140-bd1a-c3381b3656bc: entering candidate state: node="Node at f02b3030-530a-b140-bd1a-c3381b3656bc [Candidate]" term=3
2025-01-26T02:43:35.555Z [INFO]  server-bcb18c17-73be-2d6d-36b2-05cc37c5c172: duplicate requestVote for same term: term=3
2025-01-26T02:43:35.587Z [WARN]  server-bcb18c17-73be-2d6d-36b2-05cc37c5c172: Election timeout reached, restarting election
2025-01-26T02:43:35.587Z [INFO]  server-bcb18c17-73be-2d6d-36b2-05cc37c5c172: entering candidate state: node="Node at bcb18c17-73be-2d6d-36b2-05cc37c5c172 [Candidate]" term=4
2025-01-26T02:43:35.587Z [INFO]  server-f02b3030-530a-b140-bd1a-c3381b3656bc: entering follower state: follower="Node at f02b3030-530a-b140-bd1a-c3381b3656bc [Follower]" leader-address= leader-id=
2025-01-26T02:43:35.587Z [INFO]  server-bcb18c17-73be-2d6d-36b2-05cc37c5c172: election won: term=4 tally=2
2025-01-26T02:43:35.587Z [INFO]  server-bcb18c17-73be-2d6d-36b2-05cc37c5c172: entering leader state: leader="Node at bcb18c17-73be-2d6d-36b2-05cc37c5c172 [Leader]"
2025-01-26T02:43:35.587Z [INFO]  server-bcb18c17-73be-2d6d-36b2-05cc37c5c172: added peer, starting replication: peer=server-f02b3030-530a-b140-bd1a-c3381b3656bc
2025-01-26T02:43:35.587Z [INFO]  server-bcb18c17-73be-2d6d-36b2-05cc37c5c172: pipelining replication: peer="{Voter server-f02b3030-530a-b140-bd1a-c3381b3656bc f02b3030-530a-b140-bd1a-c3381b3656bc}"
2025-01-26T02:43:35.695Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:35.587645064 +0000 UTC m=+12.102677439 (1 nodes), 56.235421ms from start of poll, 451.719901ms from cluster start. Timeout at 2025-01-26 02:43:35.692651224 +0000 UTC m=+12.207683599, 105.00616ms after stability
2025-01-26T02:43:35.803Z [INFO]  server-bcb18c17-73be-2d6d-36b2-05cc37c5c172: aborting pipeline replication: peer="{Voter server-f02b3030-530a-b140-bd1a-c3381b3656bc f02b3030-530a-b140-bd1a-c3381b3656bc}"
--- PASS: TestRaft_RemoveLeader_NoShutdown (0.67s)
=== RUN   TestRaft_RemoveFollower_SplitCluster
2025-01-26T02:43:35.804Z [INFO]  server-851505c1-e8bb-a5d6-c836-063210249c1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-851505c1-e8bb-a5d6-c836-063210249c1e Address:851505c1-e8bb-a5d6-c836-063210249c1e} {Suffrage:Voter ID:server-363af5ea-cbc9-7651-3865-6b38b1ab1437 Address:363af5ea-cbc9-7651-3865-6b38b1ab1437} {Suffrage:Voter ID:server-4b6064fa-0985-f146-4566-1723f7dc5d8e Address:4b6064fa-0985-f146-4566-1723f7dc5d8e} {Suffrage:Voter ID:server-f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e Address:f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e}]"
2025-01-26T02:43:35.804Z [INFO]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-851505c1-e8bb-a5d6-c836-063210249c1e Address:851505c1-e8bb-a5d6-c836-063210249c1e} {Suffrage:Voter ID:server-363af5ea-cbc9-7651-3865-6b38b1ab1437 Address:363af5ea-cbc9-7651-3865-6b38b1ab1437} {Suffrage:Voter ID:server-4b6064fa-0985-f146-4566-1723f7dc5d8e Address:4b6064fa-0985-f146-4566-1723f7dc5d8e} {Suffrage:Voter ID:server-f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e Address:f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e}]"
2025-01-26T02:43:35.804Z [INFO]  server-4b6064fa-0985-f146-4566-1723f7dc5d8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-851505c1-e8bb-a5d6-c836-063210249c1e Address:851505c1-e8bb-a5d6-c836-063210249c1e} {Suffrage:Voter ID:server-363af5ea-cbc9-7651-3865-6b38b1ab1437 Address:363af5ea-cbc9-7651-3865-6b38b1ab1437} {Suffrage:Voter ID:server-4b6064fa-0985-f146-4566-1723f7dc5d8e Address:4b6064fa-0985-f146-4566-1723f7dc5d8e} {Suffrage:Voter ID:server-f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e Address:f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e}]"
2025-01-26T02:43:35.804Z [INFO]  server-f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-851505c1-e8bb-a5d6-c836-063210249c1e Address:851505c1-e8bb-a5d6-c836-063210249c1e} {Suffrage:Voter ID:server-363af5ea-cbc9-7651-3865-6b38b1ab1437 Address:363af5ea-cbc9-7651-3865-6b38b1ab1437} {Suffrage:Voter ID:server-4b6064fa-0985-f146-4566-1723f7dc5d8e Address:4b6064fa-0985-f146-4566-1723f7dc5d8e} {Suffrage:Voter ID:server-f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e Address:f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e}]"
2025-01-26T02:43:35.804Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:35.804Z [INFO]  server-851505c1-e8bb-a5d6-c836-063210249c1e: entering follower state: follower="Node at 851505c1-e8bb-a5d6-c836-063210249c1e [Follower]" leader-address= leader-id=
2025-01-26T02:43:35.804Z [INFO]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: entering follower state: follower="Node at 363af5ea-cbc9-7651-3865-6b38b1ab1437 [Follower]" leader-address= leader-id=
2025-01-26T02:43:35.804Z [INFO]  server-4b6064fa-0985-f146-4566-1723f7dc5d8e: entering follower state: follower="Node at 4b6064fa-0985-f146-4566-1723f7dc5d8e [Follower]" leader-address= leader-id=
2025-01-26T02:43:35.804Z [INFO]  server-f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e: entering follower state: follower="Node at f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e [Follower]" leader-address= leader-id=
2025-01-26T02:43:35.859Z [WARN]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:35.859Z [INFO]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: entering candidate state: node="Node at 363af5ea-cbc9-7651-3865-6b38b1ab1437 [Candidate]" term=2
2025-01-26T02:43:35.859Z [INFO]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: election won: term=2 tally=3
2025-01-26T02:43:35.859Z [INFO]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: entering leader state: leader="Node at 363af5ea-cbc9-7651-3865-6b38b1ab1437 [Leader]"
2025-01-26T02:43:35.859Z [INFO]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: added peer, starting replication: peer=server-851505c1-e8bb-a5d6-c836-063210249c1e
2025-01-26T02:43:35.859Z [INFO]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: added peer, starting replication: peer=server-4b6064fa-0985-f146-4566-1723f7dc5d8e
2025-01-26T02:43:35.859Z [INFO]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: added peer, starting replication: peer=server-f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e
2025-01-26T02:43:35.859Z [INFO]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: pipelining replication: peer="{Voter server-f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e}"
2025-01-26T02:43:35.859Z [INFO]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: pipelining replication: peer="{Voter server-851505c1-e8bb-a5d6-c836-063210249c1e 851505c1-e8bb-a5d6-c836-063210249c1e}"
2025-01-26T02:43:35.859Z [INFO]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: pipelining replication: peer="{Voter server-4b6064fa-0985-f146-4566-1723f7dc5d8e 4b6064fa-0985-f146-4566-1723f7dc5d8e}"
2025-01-26T02:43:35.967Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:35.859733715 +0000 UTC m=+12.374766090 (1 nodes), 55.550937ms from start of poll, 55.800139ms from cluster start. Timeout at 2025-01-26 02:43:35.964749355 +0000 UTC m=+12.479781730, 105.01564ms after stability
2025-01-26T02:43:36.075Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:43:36.183Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:43:36.075510424 +0000 UTC m=+12.590542799 (3 nodes), 1.92µs from start of poll, 271.576848ms from cluster start. Timeout at 2025-01-26 02:43:36.180519344 +0000 UTC m=+12.695551719, 105.00892ms after stability
2025-01-26T02:43:36.183Z [INFO]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: updating configuration: command=RemoveServer server-id=server-f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e server-addr= servers="[{Suffrage:Voter ID:server-851505c1-e8bb-a5d6-c836-063210249c1e Address:851505c1-e8bb-a5d6-c836-063210249c1e} {Suffrage:Voter ID:server-363af5ea-cbc9-7651-3865-6b38b1ab1437 Address:363af5ea-cbc9-7651-3865-6b38b1ab1437} {Suffrage:Voter ID:server-4b6064fa-0985-f146-4566-1723f7dc5d8e Address:4b6064fa-0985-f146-4566-1723f7dc5d8e}]"
2025-01-26T02:43:36.183Z [INFO]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: removed peer, stopping replication: peer=server-f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e last-index=3
2025-01-26T02:43:36.183Z [INFO]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: aborting pipeline replication: peer="{Voter server-f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e f28f1372-1cc7-e0ad-a8a4-650fdc0b1c1e}"
2025-01-26T02:43:36.183Z [ERROR] server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to pipeline appendEntries: peer="{Voter server-851505c1-e8bb-a5d6-c836-063210249c1e 851505c1-e8bb-a5d6-c836-063210249c1e}" error="append pipeline closed"
2025-01-26T02:43:36.183Z [INFO]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: aborting pipeline replication: peer="{Voter server-851505c1-e8bb-a5d6-c836-063210249c1e 851505c1-e8bb-a5d6-c836-063210249c1e}"
2025-01-26T02:43:36.183Z [ERROR] server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to pipeline appendEntries: peer="{Voter server-4b6064fa-0985-f146-4566-1723f7dc5d8e 4b6064fa-0985-f146-4566-1723f7dc5d8e}" error="append pipeline closed"
2025-01-26T02:43:36.183Z [INFO]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: aborting pipeline replication: peer="{Voter server-4b6064fa-0985-f146-4566-1723f7dc5d8e 4b6064fa-0985-f146-4566-1723f7dc5d8e}"
2025-01-26T02:43:36.183Z [ERROR] server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to heartbeat to: peer=851505c1-e8bb-a5d6-c836-063210249c1e backoff time=10ms error="failed to connect to peer: 851505c1-e8bb-a5d6-c836-063210249c1e"
2025-01-26T02:43:36.187Z [ERROR] server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to heartbeat to: peer=4b6064fa-0985-f146-4566-1723f7dc5d8e backoff time=10ms error="failed to connect to peer: 4b6064fa-0985-f146-4566-1723f7dc5d8e"
2025-01-26T02:43:36.191Z [ERROR] server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to appendEntries to: peer="{Voter server-4b6064fa-0985-f146-4566-1723f7dc5d8e 4b6064fa-0985-f146-4566-1723f7dc5d8e}" error="failed to connect to peer: 4b6064fa-0985-f146-4566-1723f7dc5d8e"
2025-01-26T02:43:36.191Z [ERROR] server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to appendEntries to: peer="{Voter server-851505c1-e8bb-a5d6-c836-063210249c1e 851505c1-e8bb-a5d6-c836-063210249c1e}" error="failed to connect to peer: 851505c1-e8bb-a5d6-c836-063210249c1e"
2025-01-26T02:43:36.203Z [ERROR] server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to heartbeat to: peer=851505c1-e8bb-a5d6-c836-063210249c1e backoff time=10ms error="failed to connect to peer: 851505c1-e8bb-a5d6-c836-063210249c1e"
2025-01-26T02:43:36.211Z [ERROR] server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to appendEntries to: peer="{Voter server-851505c1-e8bb-a5d6-c836-063210249c1e 851505c1-e8bb-a5d6-c836-063210249c1e}" error="failed to connect to peer: 851505c1-e8bb-a5d6-c836-063210249c1e"
2025-01-26T02:43:36.211Z [ERROR] server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to heartbeat to: peer=4b6064fa-0985-f146-4566-1723f7dc5d8e backoff time=10ms error="failed to connect to peer: 4b6064fa-0985-f146-4566-1723f7dc5d8e"
2025-01-26T02:43:36.215Z [ERROR] server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to appendEntries to: peer="{Voter server-4b6064fa-0985-f146-4566-1723f7dc5d8e 4b6064fa-0985-f146-4566-1723f7dc5d8e}" error="failed to connect to peer: 4b6064fa-0985-f146-4566-1723f7dc5d8e"
2025-01-26T02:43:36.227Z [WARN]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to contact: server-id=server-851505c1-e8bb-a5d6-c836-063210249c1e time=55.93038ms
2025-01-26T02:43:36.231Z [ERROR] server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to heartbeat to: peer=851505c1-e8bb-a5d6-c836-063210249c1e backoff time=10ms error="failed to connect to peer: 851505c1-e8bb-a5d6-c836-063210249c1e"
2025-01-26T02:43:36.235Z [ERROR] server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to heartbeat to: peer=4b6064fa-0985-f146-4566-1723f7dc5d8e backoff time=10ms error="failed to connect to peer: 4b6064fa-0985-f146-4566-1723f7dc5d8e"
2025-01-26T02:43:36.235Z [ERROR] server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to appendEntries to: peer="{Voter server-851505c1-e8bb-a5d6-c836-063210249c1e 851505c1-e8bb-a5d6-c836-063210249c1e}" error="failed to connect to peer: 851505c1-e8bb-a5d6-c836-063210249c1e"
2025-01-26T02:43:36.239Z [WARN]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to contact: server-id=server-851505c1-e8bb-a5d6-c836-063210249c1e time=67.930533ms
2025-01-26T02:43:36.239Z [WARN]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to contact: server-id=server-4b6064fa-0985-f146-4566-1723f7dc5d8e time=59.913644ms
2025-01-26T02:43:36.239Z [WARN]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to contact quorum of nodes, stepping down
2025-01-26T02:43:36.239Z [INFO]  server-363af5ea-cbc9-7651-3865-6b38b1ab1437: entering follower state: follower="Node at 363af5ea-cbc9-7651-3865-6b38b1ab1437 [Follower]" leader-address= leader-id=
2025-01-26T02:43:36.239Z [ERROR] server-363af5ea-cbc9-7651-3865-6b38b1ab1437: failed to appendEntries to: peer="{Voter server-4b6064fa-0985-f146-4566-1723f7dc5d8e 4b6064fa-0985-f146-4566-1723f7dc5d8e}" error="failed to connect to peer: 4b6064fa-0985-f146-4566-1723f7dc5d8e"
--- PASS: TestRaft_RemoveFollower_SplitCluster (0.44s)
=== RUN   TestRaft_AddKnownPeer
2025-01-26T02:43:36.240Z [INFO]  server-0d0f9451-f594-4144-b4a2-e0098031c1a2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d0f9451-f594-4144-b4a2-e0098031c1a2 Address:0d0f9451-f594-4144-b4a2-e0098031c1a2} {Suffrage:Voter ID:server-54f9dc38-c39b-1a7c-ea82-7a1404a6c184 Address:54f9dc38-c39b-1a7c-ea82-7a1404a6c184} {Suffrage:Voter ID:server-88ed83b6-eae6-e9e6-3a22-b91aeb931384 Address:88ed83b6-eae6-e9e6-3a22-b91aeb931384}]"
2025-01-26T02:43:36.240Z [INFO]  server-54f9dc38-c39b-1a7c-ea82-7a1404a6c184: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d0f9451-f594-4144-b4a2-e0098031c1a2 Address:0d0f9451-f594-4144-b4a2-e0098031c1a2} {Suffrage:Voter ID:server-54f9dc38-c39b-1a7c-ea82-7a1404a6c184 Address:54f9dc38-c39b-1a7c-ea82-7a1404a6c184} {Suffrage:Voter ID:server-88ed83b6-eae6-e9e6-3a22-b91aeb931384 Address:88ed83b6-eae6-e9e6-3a22-b91aeb931384}]"
2025-01-26T02:43:36.240Z [INFO]  server-88ed83b6-eae6-e9e6-3a22-b91aeb931384: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0d0f9451-f594-4144-b4a2-e0098031c1a2 Address:0d0f9451-f594-4144-b4a2-e0098031c1a2} {Suffrage:Voter ID:server-54f9dc38-c39b-1a7c-ea82-7a1404a6c184 Address:54f9dc38-c39b-1a7c-ea82-7a1404a6c184} {Suffrage:Voter ID:server-88ed83b6-eae6-e9e6-3a22-b91aeb931384 Address:88ed83b6-eae6-e9e6-3a22-b91aeb931384}]"
2025-01-26T02:43:36.240Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:36.240Z [INFO]  server-0d0f9451-f594-4144-b4a2-e0098031c1a2: entering follower state: follower="Node at 0d0f9451-f594-4144-b4a2-e0098031c1a2 [Follower]" leader-address= leader-id=
2025-01-26T02:43:36.240Z [INFO]  server-54f9dc38-c39b-1a7c-ea82-7a1404a6c184: entering follower state: follower="Node at 54f9dc38-c39b-1a7c-ea82-7a1404a6c184 [Follower]" leader-address= leader-id=
2025-01-26T02:43:36.240Z [INFO]  server-88ed83b6-eae6-e9e6-3a22-b91aeb931384: entering follower state: follower="Node at 88ed83b6-eae6-e9e6-3a22-b91aeb931384 [Follower]" leader-address= leader-id=
2025-01-26T02:43:36.307Z [WARN]  server-54f9dc38-c39b-1a7c-ea82-7a1404a6c184: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:36.307Z [INFO]  server-54f9dc38-c39b-1a7c-ea82-7a1404a6c184: entering candidate state: node="Node at 54f9dc38-c39b-1a7c-ea82-7a1404a6c184 [Candidate]" term=2
2025-01-26T02:43:36.307Z [INFO]  server-54f9dc38-c39b-1a7c-ea82-7a1404a6c184: election won: term=2 tally=2
2025-01-26T02:43:36.307Z [INFO]  server-54f9dc38-c39b-1a7c-ea82-7a1404a6c184: entering leader state: leader="Node at 54f9dc38-c39b-1a7c-ea82-7a1404a6c184 [Leader]"
2025-01-26T02:43:36.307Z [INFO]  server-54f9dc38-c39b-1a7c-ea82-7a1404a6c184: added peer, starting replication: peer=server-0d0f9451-f594-4144-b4a2-e0098031c1a2
2025-01-26T02:43:36.307Z [INFO]  server-54f9dc38-c39b-1a7c-ea82-7a1404a6c184: added peer, starting replication: peer=server-88ed83b6-eae6-e9e6-3a22-b91aeb931384
2025-01-26T02:43:36.307Z [INFO]  server-54f9dc38-c39b-1a7c-ea82-7a1404a6c184: pipelining replication: peer="{Voter server-88ed83b6-eae6-e9e6-3a22-b91aeb931384 88ed83b6-eae6-e9e6-3a22-b91aeb931384}"
2025-01-26T02:43:36.307Z [INFO]  server-54f9dc38-c39b-1a7c-ea82-7a1404a6c184: pipelining replication: peer="{Voter server-0d0f9451-f594-4144-b4a2-e0098031c1a2 0d0f9451-f594-4144-b4a2-e0098031c1a2}"
2025-01-26T02:43:36.415Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:36.307739833 +0000 UTC m=+12.822772208 (1 nodes), 67.61693ms from start of poll, 67.810611ms from cluster start. Timeout at 2025-01-26 02:43:36.412751673 +0000 UTC m=+12.927784008, 105.0118ms after stability
2025-01-26T02:43:36.415Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:43:36.523Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:43:36.415451367 +0000 UTC m=+12.930483742 (2 nodes), 2.08µs from start of poll, 175.522145ms from cluster start. Timeout at 2025-01-26 02:43:36.520468767 +0000 UTC m=+13.035501102, 105.01736ms after stability
2025-01-26T02:43:36.523Z [INFO]  server-54f9dc38-c39b-1a7c-ea82-7a1404a6c184: updating configuration: command=AddVoter server-id=server-0d0f9451-f594-4144-b4a2-e0098031c1a2 server-addr=0d0f9451-f594-4144-b4a2-e0098031c1a2 servers="[{Suffrage:Voter ID:server-0d0f9451-f594-4144-b4a2-e0098031c1a2 Address:0d0f9451-f594-4144-b4a2-e0098031c1a2} {Suffrage:Voter ID:server-54f9dc38-c39b-1a7c-ea82-7a1404a6c184 Address:54f9dc38-c39b-1a7c-ea82-7a1404a6c184} {Suffrage:Voter ID:server-88ed83b6-eae6-e9e6-3a22-b91aeb931384 Address:88ed83b6-eae6-e9e6-3a22-b91aeb931384}]"
2025-01-26T02:43:36.523Z [INFO]  server-54f9dc38-c39b-1a7c-ea82-7a1404a6c184: aborting pipeline replication: peer="{Voter server-0d0f9451-f594-4144-b4a2-e0098031c1a2 0d0f9451-f594-4144-b4a2-e0098031c1a2}"
2025-01-26T02:43:36.523Z [INFO]  server-54f9dc38-c39b-1a7c-ea82-7a1404a6c184: aborting pipeline replication: peer="{Voter server-88ed83b6-eae6-e9e6-3a22-b91aeb931384 88ed83b6-eae6-e9e6-3a22-b91aeb931384}"
--- PASS: TestRaft_AddKnownPeer (0.28s)
=== RUN   TestRaft_RemoveUnknownPeer
2025-01-26T02:43:36.524Z [INFO]  server-91999eb9-bf4c-379d-ecdc-aca0d9703e1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91999eb9-bf4c-379d-ecdc-aca0d9703e1b Address:91999eb9-bf4c-379d-ecdc-aca0d9703e1b} {Suffrage:Voter ID:server-581449f6-6b6f-9894-e66d-78fdd336e3b1 Address:581449f6-6b6f-9894-e66d-78fdd336e3b1} {Suffrage:Voter ID:server-176b6317-0c72-e0e2-6ec4-7acfdffdbe0c Address:176b6317-0c72-e0e2-6ec4-7acfdffdbe0c}]"
2025-01-26T02:43:36.524Z [INFO]  server-581449f6-6b6f-9894-e66d-78fdd336e3b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91999eb9-bf4c-379d-ecdc-aca0d9703e1b Address:91999eb9-bf4c-379d-ecdc-aca0d9703e1b} {Suffrage:Voter ID:server-581449f6-6b6f-9894-e66d-78fdd336e3b1 Address:581449f6-6b6f-9894-e66d-78fdd336e3b1} {Suffrage:Voter ID:server-176b6317-0c72-e0e2-6ec4-7acfdffdbe0c Address:176b6317-0c72-e0e2-6ec4-7acfdffdbe0c}]"
2025-01-26T02:43:36.524Z [INFO]  server-176b6317-0c72-e0e2-6ec4-7acfdffdbe0c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-91999eb9-bf4c-379d-ecdc-aca0d9703e1b Address:91999eb9-bf4c-379d-ecdc-aca0d9703e1b} {Suffrage:Voter ID:server-581449f6-6b6f-9894-e66d-78fdd336e3b1 Address:581449f6-6b6f-9894-e66d-78fdd336e3b1} {Suffrage:Voter ID:server-176b6317-0c72-e0e2-6ec4-7acfdffdbe0c Address:176b6317-0c72-e0e2-6ec4-7acfdffdbe0c}]"
2025-01-26T02:43:36.524Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:36.525Z [INFO]  server-91999eb9-bf4c-379d-ecdc-aca0d9703e1b: entering follower state: follower="Node at 91999eb9-bf4c-379d-ecdc-aca0d9703e1b [Follower]" leader-address= leader-id=
2025-01-26T02:43:36.525Z [INFO]  server-581449f6-6b6f-9894-e66d-78fdd336e3b1: entering follower state: follower="Node at 581449f6-6b6f-9894-e66d-78fdd336e3b1 [Follower]" leader-address= leader-id=
2025-01-26T02:43:36.525Z [INFO]  server-176b6317-0c72-e0e2-6ec4-7acfdffdbe0c: entering follower state: follower="Node at 176b6317-0c72-e0e2-6ec4-7acfdffdbe0c [Follower]" leader-address= leader-id=
2025-01-26T02:43:36.579Z [WARN]  server-581449f6-6b6f-9894-e66d-78fdd336e3b1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:36.579Z [INFO]  server-581449f6-6b6f-9894-e66d-78fdd336e3b1: entering candidate state: node="Node at 581449f6-6b6f-9894-e66d-78fdd336e3b1 [Candidate]" term=2
2025-01-26T02:43:36.579Z [WARN]  server-176b6317-0c72-e0e2-6ec4-7acfdffdbe0c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:36.579Z [INFO]  server-176b6317-0c72-e0e2-6ec4-7acfdffdbe0c: entering candidate state: node="Node at 176b6317-0c72-e0e2-6ec4-7acfdffdbe0c [Candidate]" term=2
2025-01-26T02:43:36.579Z [INFO]  server-176b6317-0c72-e0e2-6ec4-7acfdffdbe0c: duplicate requestVote for same term: term=2
2025-01-26T02:43:36.579Z [INFO]  server-581449f6-6b6f-9894-e66d-78fdd336e3b1: election won: term=2 tally=2
2025-01-26T02:43:36.579Z [INFO]  server-581449f6-6b6f-9894-e66d-78fdd336e3b1: entering leader state: leader="Node at 581449f6-6b6f-9894-e66d-78fdd336e3b1 [Leader]"
2025-01-26T02:43:36.579Z [INFO]  server-581449f6-6b6f-9894-e66d-78fdd336e3b1: added peer, starting replication: peer=server-91999eb9-bf4c-379d-ecdc-aca0d9703e1b
2025-01-26T02:43:36.579Z [INFO]  server-581449f6-6b6f-9894-e66d-78fdd336e3b1: added peer, starting replication: peer=server-176b6317-0c72-e0e2-6ec4-7acfdffdbe0c
2025-01-26T02:43:36.579Z [INFO]  server-176b6317-0c72-e0e2-6ec4-7acfdffdbe0c: entering follower state: follower="Node at 176b6317-0c72-e0e2-6ec4-7acfdffdbe0c [Follower]" leader-address=581449f6-6b6f-9894-e66d-78fdd336e3b1 leader-id=server-581449f6-6b6f-9894-e66d-78fdd336e3b1
2025-01-26T02:43:36.579Z [INFO]  server-581449f6-6b6f-9894-e66d-78fdd336e3b1: pipelining replication: peer="{Voter server-176b6317-0c72-e0e2-6ec4-7acfdffdbe0c 176b6317-0c72-e0e2-6ec4-7acfdffdbe0c}"
2025-01-26T02:43:36.579Z [INFO]  server-91999eb9-bf4c-379d-ecdc-aca0d9703e1b: duplicate requestVote for same term: term=2
2025-01-26T02:43:36.579Z [INFO]  server-581449f6-6b6f-9894-e66d-78fdd336e3b1: pipelining replication: peer="{Voter server-91999eb9-bf4c-379d-ecdc-aca0d9703e1b 91999eb9-bf4c-379d-ecdc-aca0d9703e1b}"
2025-01-26T02:43:36.579Z [WARN]  server-581449f6-6b6f-9894-e66d-78fdd336e3b1: rejecting vote request since we have a leader: from=176b6317-0c72-e0e2-6ec4-7acfdffdbe0c leader=581449f6-6b6f-9894-e66d-78fdd336e3b1 leader-id=server-581449f6-6b6f-9894-e66d-78fdd336e3b1
2025-01-26T02:43:36.687Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:36.579793524 +0000 UTC m=+13.094825899 (1 nodes), 54.992934ms from start of poll, 55.480857ms from cluster start. Timeout at 2025-01-26 02:43:36.684797524 +0000 UTC m=+13.199829859, 105.00396ms after stability
2025-01-26T02:43:36.687Z [INFO]  server-581449f6-6b6f-9894-e66d-78fdd336e3b1: updating configuration: command=RemoveServer server-id=a9c6bb32-9bc0-7e4f-6903-b504f5b15a65 server-addr= servers="[{Suffrage:Voter ID:server-91999eb9-bf4c-379d-ecdc-aca0d9703e1b Address:91999eb9-bf4c-379d-ecdc-aca0d9703e1b} {Suffrage:Voter ID:server-581449f6-6b6f-9894-e66d-78fdd336e3b1 Address:581449f6-6b6f-9894-e66d-78fdd336e3b1} {Suffrage:Voter ID:server-176b6317-0c72-e0e2-6ec4-7acfdffdbe0c Address:176b6317-0c72-e0e2-6ec4-7acfdffdbe0c}]"
2025-01-26T02:43:36.687Z [INFO]  server-581449f6-6b6f-9894-e66d-78fdd336e3b1: aborting pipeline replication: peer="{Voter server-91999eb9-bf4c-379d-ecdc-aca0d9703e1b 91999eb9-bf4c-379d-ecdc-aca0d9703e1b}"
2025-01-26T02:43:36.687Z [INFO]  server-581449f6-6b6f-9894-e66d-78fdd336e3b1: aborting pipeline replication: peer="{Voter server-176b6317-0c72-e0e2-6ec4-7acfdffdbe0c 176b6317-0c72-e0e2-6ec4-7acfdffdbe0c}"
2025-01-26T02:43:37.191Z [ERROR] server-581449f6-6b6f-9894-e66d-78fdd336e3b1: failed to heartbeat to: peer=91999eb9-bf4c-379d-ecdc-aca0d9703e1b backoff time=10ms error="command timed out"
--- PASS: TestRaft_RemoveUnknownPeer (0.67s)
=== RUN   TestRaft_SnapshotRestore
2025-01-26T02:43:37.191Z [INFO]  server-402e6ab4-2f81-189e-756a-c792b511f19e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-402e6ab4-2f81-189e-756a-c792b511f19e Address:402e6ab4-2f81-189e-756a-c792b511f19e}]"
2025-01-26T02:43:37.191Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:37.191Z [INFO]  server-402e6ab4-2f81-189e-756a-c792b511f19e: entering follower state: follower="Node at 402e6ab4-2f81-189e-756a-c792b511f19e [Follower]" leader-address= leader-id=
2025-01-26T02:43:37.291Z [WARN]  server-402e6ab4-2f81-189e-756a-c792b511f19e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:37.291Z [INFO]  server-402e6ab4-2f81-189e-756a-c792b511f19e: entering candidate state: node="Node at 402e6ab4-2f81-189e-756a-c792b511f19e [Candidate]" term=2
2025-01-26T02:43:37.291Z [INFO]  server-402e6ab4-2f81-189e-756a-c792b511f19e: election won: term=2 tally=1
2025-01-26T02:43:37.291Z [INFO]  server-402e6ab4-2f81-189e-756a-c792b511f19e: entering leader state: leader="Node at 402e6ab4-2f81-189e-756a-c792b511f19e [Leader]"
2025-01-26T02:43:37.399Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:37.291512963 +0000 UTC m=+13.806545338 (1 nodes), 99.656645ms from start of poll, 99.763966ms from cluster start. Timeout at 2025-01-26 02:43:37.396516043 +0000 UTC m=+13.911548418, 105.00308ms after stability
2025-01-26T02:43:37.400Z [INFO]  server-402e6ab4-2f81-189e-756a-c792b511f19e: starting snapshot up to: index=102
2025-01-26T02:43:37.400Z [INFO]  creating new snapshot: path=/tmp/raft839147971/snapshots/2-102-1737859417400.tmp
2025-01-26T02:43:37.400Z [INFO]  server-402e6ab4-2f81-189e-756a-c792b511f19e: compacting logs: from=1 to=92
2025-01-26T02:43:37.400Z [INFO]  server-402e6ab4-2f81-189e-756a-c792b511f19e: snapshot complete up to: index=102
2025-01-26T02:43:37.400Z [INFO]  server-402e6ab4-2f81-189e-756a-c792b511f19e: starting restore from snapshot: id=2-102-1737859417400 last-index=102 last-term=2 size-in-bytes=693
2025-01-26T02:43:37.401Z [INFO]  server-402e6ab4-2f81-189e-756a-c792b511f19e: snapshot restore progress: id=2-102-1737859417400 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-01-26T02:43:37.401Z [INFO]  server-402e6ab4-2f81-189e-756a-c792b511f19e: restored from snapshot: id=2-102-1737859417400 last-index=102 last-term=2 size-in-bytes=693
2025-01-26T02:43:37.401Z [INFO]  server-402e6ab4-2f81-189e-756a-c792b511f19e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-402e6ab4-2f81-189e-756a-c792b511f19e Address:402e6ab4-2f81-189e-756a-c792b511f19e}]"
--- PASS: TestRaft_SnapshotRestore (0.21s)
=== RUN   TestRaft_RestoreSnapshotOnStartup_Monotonic
2025-01-26T02:43:37.401Z [INFO]  server-3d248312-e9b4-8044-88c1-9277c322985a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d248312-e9b4-8044-88c1-9277c322985a Address:3d248312-e9b4-8044-88c1-9277c322985a}]"
2025-01-26T02:43:37.401Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:37.401Z [INFO]  server-3d248312-e9b4-8044-88c1-9277c322985a: entering follower state: follower="Node at 3d248312-e9b4-8044-88c1-9277c322985a [Follower]" leader-address= leader-id=
2025-01-26T02:43:37.495Z [WARN]  server-3d248312-e9b4-8044-88c1-9277c322985a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:37.495Z [INFO]  server-3d248312-e9b4-8044-88c1-9277c322985a: entering candidate state: node="Node at 3d248312-e9b4-8044-88c1-9277c322985a [Candidate]" term=2
2025-01-26T02:43:37.495Z [INFO]  server-3d248312-e9b4-8044-88c1-9277c322985a: election won: term=2 tally=1
2025-01-26T02:43:37.495Z [INFO]  server-3d248312-e9b4-8044-88c1-9277c322985a: entering leader state: leader="Node at 3d248312-e9b4-8044-88c1-9277c322985a [Leader]"
2025-01-26T02:43:37.603Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:37.495514921 +0000 UTC m=+14.010547296 (1 nodes), 94.131492ms from start of poll, 94.209772ms from cluster start. Timeout at 2025-01-26 02:43:37.600520881 +0000 UTC m=+14.115553216, 105.00592ms after stability
2025-01-26T02:43:37.604Z [INFO]  server-3d248312-e9b4-8044-88c1-9277c322985a: starting snapshot up to: index=102
2025-01-26T02:43:37.604Z [INFO]  creating new snapshot: path=/tmp/raft415817844/snapshots/2-102-1737859417604.tmp
2025-01-26T02:43:37.604Z [INFO]  server-3d248312-e9b4-8044-88c1-9277c322985a: compacting logs: from=1 to=92
2025-01-26T02:43:37.604Z [INFO]  server-3d248312-e9b4-8044-88c1-9277c322985a: snapshot complete up to: index=102
2025-01-26T02:43:37.604Z [INFO]  server-3d248312-e9b4-8044-88c1-9277c322985a: starting restore from snapshot: id=2-102-1737859417604 last-index=102 last-term=2 size-in-bytes=693
2025-01-26T02:43:37.605Z [INFO]  server-3d248312-e9b4-8044-88c1-9277c322985a: snapshot restore progress: id=2-102-1737859417604 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-01-26T02:43:37.605Z [INFO]  server-3d248312-e9b4-8044-88c1-9277c322985a: restored from snapshot: id=2-102-1737859417604 last-index=102 last-term=2 size-in-bytes=693
2025-01-26T02:43:37.605Z [INFO]  server-3d248312-e9b4-8044-88c1-9277c322985a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d248312-e9b4-8044-88c1-9277c322985a Address:3d248312-e9b4-8044-88c1-9277c322985a}]"
--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.20s)
=== RUN   TestRaft_SnapshotRestore_Progress
2025-01-26T02:43:37.605Z [INFO]  server-4d0dc4c2-a7b9-2bda-67f9-0e0a031613f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d0dc4c2-a7b9-2bda-67f9-0e0a031613f7 Address:4d0dc4c2-a7b9-2bda-67f9-0e0a031613f7}]"
2025-01-26T02:43:37.605Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:37.605Z [INFO]  server-4d0dc4c2-a7b9-2bda-67f9-0e0a031613f7: entering follower state: follower="Node at 4d0dc4c2-a7b9-2bda-67f9-0e0a031613f7 [Follower]" leader-address= leader-id=
2025-01-26T02:43:37.679Z [WARN]  server-4d0dc4c2-a7b9-2bda-67f9-0e0a031613f7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:37.679Z [INFO]  server-4d0dc4c2-a7b9-2bda-67f9-0e0a031613f7: entering candidate state: node="Node at 4d0dc4c2-a7b9-2bda-67f9-0e0a031613f7 [Candidate]" term=2
2025-01-26T02:43:37.679Z [INFO]  server-4d0dc4c2-a7b9-2bda-67f9-0e0a031613f7: election won: term=2 tally=1
2025-01-26T02:43:37.679Z [INFO]  server-4d0dc4c2-a7b9-2bda-67f9-0e0a031613f7: entering leader state: leader="Node at 4d0dc4c2-a7b9-2bda-67f9-0e0a031613f7 [Leader]"
2025-01-26T02:43:37.787Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:37.679530917 +0000 UTC m=+14.194563292 (1 nodes), 74.15145ms from start of poll, 74.22657ms from cluster start. Timeout at 2025-01-26 02:43:37.784535717 +0000 UTC m=+14.299568092, 105.0048ms after stability
2025-01-26T02:43:37.788Z [INFO]  server-4d0dc4c2-a7b9-2bda-67f9-0e0a031613f7: starting snapshot up to: index=102
2025-01-26T02:43:37.788Z [INFO]  creating new snapshot: path=/tmp/raft2835185879/snapshots/2-102-1737859417788.tmp
2025-01-26T02:43:37.788Z [INFO]  server-4d0dc4c2-a7b9-2bda-67f9-0e0a031613f7: compacting logs: from=1 to=92
2025-01-26T02:43:37.788Z [INFO]  server-4d0dc4c2-a7b9-2bda-67f9-0e0a031613f7: snapshot complete up to: index=102
--- PASS: TestRaft_SnapshotRestore_Progress (0.18s)
=== RUN   TestRaft_NoRestoreOnStart
2025-01-26T02:43:37.789Z [INFO]  server-dbfed3e9-612e-a041-b79c-4b1fdfac724b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dbfed3e9-612e-a041-b79c-4b1fdfac724b Address:dbfed3e9-612e-a041-b79c-4b1fdfac724b}]"
2025-01-26T02:43:37.789Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:37.789Z [INFO]  server-dbfed3e9-612e-a041-b79c-4b1fdfac724b: entering follower state: follower="Node at dbfed3e9-612e-a041-b79c-4b1fdfac724b [Follower]" leader-address= leader-id=
2025-01-26T02:43:37.863Z [WARN]  server-dbfed3e9-612e-a041-b79c-4b1fdfac724b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:37.863Z [INFO]  server-dbfed3e9-612e-a041-b79c-4b1fdfac724b: entering candidate state: node="Node at dbfed3e9-612e-a041-b79c-4b1fdfac724b [Candidate]" term=2
2025-01-26T02:43:37.863Z [INFO]  server-dbfed3e9-612e-a041-b79c-4b1fdfac724b: election won: term=2 tally=1
2025-01-26T02:43:37.863Z [INFO]  server-dbfed3e9-612e-a041-b79c-4b1fdfac724b: entering leader state: leader="Node at dbfed3e9-612e-a041-b79c-4b1fdfac724b [Leader]"
2025-01-26T02:43:37.971Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:37.863554434 +0000 UTC m=+14.378586809 (1 nodes), 74.037529ms from start of poll, 74.11741ms from cluster start. Timeout at 2025-01-26 02:43:37.968557554 +0000 UTC m=+14.483589929, 105.00312ms after stability
2025-01-26T02:43:37.973Z [INFO]  server-dbfed3e9-612e-a041-b79c-4b1fdfac724b: starting snapshot up to: index=102
2025-01-26T02:43:37.973Z [INFO]  creating new snapshot: path=/tmp/raft2449051736/snapshots/2-102-1737859417973.tmp
2025-01-26T02:43:37.974Z [INFO]  server-dbfed3e9-612e-a041-b79c-4b1fdfac724b: compacting logs: from=1 to=92
2025-01-26T02:43:37.974Z [INFO]  server-dbfed3e9-612e-a041-b79c-4b1fdfac724b: snapshot complete up to: index=102
2025-01-26T02:43:37.974Z [INFO]  server-dbfed3e9-612e-a041-b79c-4b1fdfac724b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dbfed3e9-612e-a041-b79c-4b1fdfac724b Address:dbfed3e9-612e-a041-b79c-4b1fdfac724b}]"
--- PASS: TestRaft_NoRestoreOnStart (0.18s)
=== RUN   TestRaft_SnapshotRestore_PeerChange
    raft_test.go:1266: Skipping unreliable test
--- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
=== RUN   TestRaft_AutoSnapshot
2025-01-26T02:43:37.974Z [INFO]  server-40c46aaa-0b44-709e-6a2c-0b4a11c537f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-40c46aaa-0b44-709e-6a2c-0b4a11c537f2 Address:40c46aaa-0b44-709e-6a2c-0b4a11c537f2}]"
2025-01-26T02:43:37.974Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:37.974Z [INFO]  server-dbfed3e9-612e-a041-b79c-4b1fdfac724b: entering follower state: follower="Node at dbfed3e9-612e-a041-b79c-4b1fdfac724b [Follower]" leader-address= leader-id=
2025-01-26T02:43:37.974Z [INFO]  server-40c46aaa-0b44-709e-6a2c-0b4a11c537f2: entering follower state: follower="Node at 40c46aaa-0b44-709e-6a2c-0b4a11c537f2 [Follower]" leader-address= leader-id=
2025-01-26T02:43:38.039Z [WARN]  server-dbfed3e9-612e-a041-b79c-4b1fdfac724b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:38.039Z [INFO]  server-dbfed3e9-612e-a041-b79c-4b1fdfac724b: entering candidate state: node="Node at dbfed3e9-612e-a041-b79c-4b1fdfac724b [Candidate]" term=3
2025-01-26T02:43:38.039Z [INFO]  server-dbfed3e9-612e-a041-b79c-4b1fdfac724b: election won: term=3 tally=1
2025-01-26T02:43:38.039Z [INFO]  server-dbfed3e9-612e-a041-b79c-4b1fdfac724b: entering leader state: leader="Node at dbfed3e9-612e-a041-b79c-4b1fdfac724b [Leader]"
2025-01-26T02:43:38.047Z [WARN]  server-40c46aaa-0b44-709e-6a2c-0b4a11c537f2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:38.047Z [INFO]  server-40c46aaa-0b44-709e-6a2c-0b4a11c537f2: entering candidate state: node="Node at 40c46aaa-0b44-709e-6a2c-0b4a11c537f2 [Candidate]" term=2
2025-01-26T02:43:38.047Z [INFO]  server-40c46aaa-0b44-709e-6a2c-0b4a11c537f2: election won: term=2 tally=1
2025-01-26T02:43:38.047Z [INFO]  server-40c46aaa-0b44-709e-6a2c-0b4a11c537f2: entering leader state: leader="Node at 40c46aaa-0b44-709e-6a2c-0b4a11c537f2 [Leader]"
2025-01-26T02:43:38.155Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:38.04751527 +0000 UTC m=+14.562547645 (1 nodes), 73.043803ms from start of poll, 73.133844ms from cluster start. Timeout at 2025-01-26 02:43:38.15252123 +0000 UTC m=+14.667553565, 105.00592ms after stability
2025-01-26T02:43:38.171Z [INFO]  server-40c46aaa-0b44-709e-6a2c-0b4a11c537f2: starting snapshot up to: index=102
2025-01-26T02:43:38.171Z [INFO]  creating new snapshot: path=/tmp/raft2220126425/snapshots/2-102-1737859418171.tmp
2025-01-26T02:43:38.171Z [INFO]  server-40c46aaa-0b44-709e-6a2c-0b4a11c537f2: compacting logs: from=1 to=92
2025-01-26T02:43:38.171Z [INFO]  server-40c46aaa-0b44-709e-6a2c-0b4a11c537f2: snapshot complete up to: index=102
--- PASS: TestRaft_AutoSnapshot (0.29s)
=== RUN   TestRaft_UserSnapshot
2025-01-26T02:43:38.263Z [INFO]  server-631541a2-852f-279e-a08e-c67668709872: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-631541a2-852f-279e-a08e-c67668709872 Address:631541a2-852f-279e-a08e-c67668709872}]"
2025-01-26T02:43:38.263Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:38.263Z [INFO]  server-631541a2-852f-279e-a08e-c67668709872: entering follower state: follower="Node at 631541a2-852f-279e-a08e-c67668709872 [Follower]" leader-address= leader-id=
2025-01-26T02:43:38.339Z [WARN]  server-631541a2-852f-279e-a08e-c67668709872: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:38.339Z [INFO]  server-631541a2-852f-279e-a08e-c67668709872: entering candidate state: node="Node at 631541a2-852f-279e-a08e-c67668709872 [Candidate]" term=2
2025-01-26T02:43:38.339Z [INFO]  server-631541a2-852f-279e-a08e-c67668709872: election won: term=2 tally=1
2025-01-26T02:43:38.339Z [INFO]  server-631541a2-852f-279e-a08e-c67668709872: entering leader state: leader="Node at 631541a2-852f-279e-a08e-c67668709872 [Leader]"
2025-01-26T02:43:38.447Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:38.339516442 +0000 UTC m=+14.854548817 (1 nodes), 75.633859ms from start of poll, 75.74646ms from cluster start. Timeout at 2025-01-26 02:43:38.444522202 +0000 UTC m=+14.959554577, 105.00576ms after stability
2025-01-26T02:43:38.447Z [ERROR] server-631541a2-852f-279e-a08e-c67668709872: failed to take snapshot: error="nothing new to snapshot"
2025-01-26T02:43:38.447Z [INFO]  server-631541a2-852f-279e-a08e-c67668709872: starting snapshot up to: index=12
2025-01-26T02:43:38.447Z [INFO]  creating new snapshot: path=/tmp/raft32365340/snapshots/2-12-1737859418447.tmp
2025-01-26T02:43:38.447Z [INFO]  server-631541a2-852f-279e-a08e-c67668709872: compacting logs: from=1 to=2
2025-01-26T02:43:38.447Z [INFO]  server-631541a2-852f-279e-a08e-c67668709872: snapshot complete up to: index=12
--- PASS: TestRaft_UserSnapshot (0.18s)
=== RUN   TestRaft_UserRestore
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false
2025-01-26T02:43:38.448Z [INFO]  server-c38691cb-e304-9b26-ebfc-146c462f602d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c38691cb-e304-9b26-ebfc-146c462f602d Address:c38691cb-e304-9b26-ebfc-146c462f602d} {Suffrage:Voter ID:server-9ec03dd9-b4e3-bff8-9e34-6c7d259956a4 Address:9ec03dd9-b4e3-bff8-9e34-6c7d259956a4} {Suffrage:Voter ID:server-3c195922-014b-a26b-0059-cdc090659925 Address:3c195922-014b-a26b-0059-cdc090659925}]"
2025-01-26T02:43:38.448Z [INFO]  server-9ec03dd9-b4e3-bff8-9e34-6c7d259956a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c38691cb-e304-9b26-ebfc-146c462f602d Address:c38691cb-e304-9b26-ebfc-146c462f602d} {Suffrage:Voter ID:server-9ec03dd9-b4e3-bff8-9e34-6c7d259956a4 Address:9ec03dd9-b4e3-bff8-9e34-6c7d259956a4} {Suffrage:Voter ID:server-3c195922-014b-a26b-0059-cdc090659925 Address:3c195922-014b-a26b-0059-cdc090659925}]"
2025-01-26T02:43:38.448Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c38691cb-e304-9b26-ebfc-146c462f602d Address:c38691cb-e304-9b26-ebfc-146c462f602d} {Suffrage:Voter ID:server-9ec03dd9-b4e3-bff8-9e34-6c7d259956a4 Address:9ec03dd9-b4e3-bff8-9e34-6c7d259956a4} {Suffrage:Voter ID:server-3c195922-014b-a26b-0059-cdc090659925 Address:3c195922-014b-a26b-0059-cdc090659925}]"
2025-01-26T02:43:38.448Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:38.448Z [INFO]  server-c38691cb-e304-9b26-ebfc-146c462f602d: entering follower state: follower="Node at c38691cb-e304-9b26-ebfc-146c462f602d [Follower]" leader-address= leader-id=
2025-01-26T02:43:38.448Z [INFO]  server-9ec03dd9-b4e3-bff8-9e34-6c7d259956a4: entering follower state: follower="Node at 9ec03dd9-b4e3-bff8-9e34-6c7d259956a4 [Follower]" leader-address= leader-id=
2025-01-26T02:43:38.448Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: entering follower state: follower="Node at 3c195922-014b-a26b-0059-cdc090659925 [Follower]" leader-address= leader-id=
2025-01-26T02:43:39.150Z [WARN]  server-3c195922-014b-a26b-0059-cdc090659925: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:39.150Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: entering candidate state: node="Node at 3c195922-014b-a26b-0059-cdc090659925 [Candidate]" term=2
2025-01-26T02:43:39.150Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: election won: term=2 tally=2
2025-01-26T02:43:39.150Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: entering leader state: leader="Node at 3c195922-014b-a26b-0059-cdc090659925 [Leader]"
2025-01-26T02:43:39.150Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: added peer, starting replication: peer=server-c38691cb-e304-9b26-ebfc-146c462f602d
2025-01-26T02:43:39.150Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: added peer, starting replication: peer=server-9ec03dd9-b4e3-bff8-9e34-6c7d259956a4
2025-01-26T02:43:39.150Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: pipelining replication: peer="{Voter server-9ec03dd9-b4e3-bff8-9e34-6c7d259956a4 9ec03dd9-b4e3-bff8-9e34-6c7d259956a4}"
2025-01-26T02:43:39.150Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: pipelining replication: peer="{Voter server-c38691cb-e304-9b26-ebfc-146c462f602d c38691cb-e304-9b26-ebfc-146c462f602d}"
2025-01-26T02:43:40.159Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:39.150889645 +0000 UTC m=+15.665922020 (1 nodes), 702.479942ms from start of poll, 702.657063ms from cluster start. Timeout at 2025-01-26 02:43:40.155894445 +0000 UTC m=+16.670926780, 1.00500476s after stability
2025-01-26T02:43:40.159Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: starting snapshot up to: index=12
2025-01-26T02:43:40.159Z [INFO]  creating new snapshot: path=/tmp/raft4187367119/snapshots/2-12-1737859420159.tmp
2025-01-26T02:43:40.160Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: snapshot complete up to: index=12
2025-01-26T02:43:40.160Z [INFO]  creating new snapshot: path=/tmp/raft4187367119/snapshots/2-23-1737859420160.tmp
2025-01-26T02:43:40.161Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: copied to local snapshot: bytes=71
2025-01-26T02:43:40.161Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: snapshot restore progress: id=2-23-1737859420160 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:40.161Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: restored user snapshot: index=23
2025-01-26T02:43:40.161Z [ERROR] server-3c195922-014b-a26b-0059-cdc090659925: failed to get log: index=23 error="log not found"
2025-01-26T02:43:40.161Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: aborting pipeline replication: peer="{Voter server-9ec03dd9-b4e3-bff8-9e34-6c7d259956a4 9ec03dd9-b4e3-bff8-9e34-6c7d259956a4}"
2025-01-26T02:43:40.161Z [ERROR] server-3c195922-014b-a26b-0059-cdc090659925: failed to get log: index=23 error="log not found"
2025-01-26T02:43:40.161Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: aborting pipeline replication: peer="{Voter server-c38691cb-e304-9b26-ebfc-146c462f602d c38691cb-e304-9b26-ebfc-146c462f602d}"
2025-01-26T02:43:40.161Z [ERROR] server-3c195922-014b-a26b-0059-cdc090659925: failed to get log: index=23 error="log not found"
2025-01-26T02:43:40.161Z [INFO]  creating new snapshot: path=/tmp/raft1747701857/snapshots/2-23-1737859420161.tmp
2025-01-26T02:43:40.161Z [INFO]  server-c38691cb-e304-9b26-ebfc-146c462f602d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:40.161Z [INFO]  server-c38691cb-e304-9b26-ebfc-146c462f602d: copied to local snapshot: bytes=71
2025-01-26T02:43:40.161Z [INFO]  server-c38691cb-e304-9b26-ebfc-146c462f602d: snapshot restore progress: id=2-23-1737859420161 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:40.161Z [INFO]  server-c38691cb-e304-9b26-ebfc-146c462f602d: Installed remote snapshot
2025-01-26T02:43:40.161Z [WARN]  server-c38691cb-e304-9b26-ebfc-146c462f602d: skipping application of old log: index=22
2025-01-26T02:43:40.161Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: pipelining replication: peer="{Voter server-c38691cb-e304-9b26-ebfc-146c462f602d c38691cb-e304-9b26-ebfc-146c462f602d}"
2025-01-26T02:43:40.161Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:43:40.171Z [ERROR] server-3c195922-014b-a26b-0059-cdc090659925: failed to get log: index=23 error="log not found"
2025-01-26T02:43:40.171Z [INFO]  creating new snapshot: path=/tmp/raft1614147054/snapshots/2-23-1737859420171.tmp
2025-01-26T02:43:40.171Z [INFO]  server-9ec03dd9-b4e3-bff8-9e34-6c7d259956a4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:40.172Z [INFO]  server-9ec03dd9-b4e3-bff8-9e34-6c7d259956a4: copied to local snapshot: bytes=71
2025-01-26T02:43:40.172Z [INFO]  server-9ec03dd9-b4e3-bff8-9e34-6c7d259956a4: snapshot restore progress: id=2-23-1737859420171 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:40.172Z [INFO]  server-9ec03dd9-b4e3-bff8-9e34-6c7d259956a4: Installed remote snapshot
2025-01-26T02:43:40.172Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: pipelining replication: peer="{Voter server-9ec03dd9-b4e3-bff8-9e34-6c7d259956a4 9ec03dd9-b4e3-bff8-9e34-6c7d259956a4}"
2025-01-26T02:43:41.167Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:43:40.16182982 +0000 UTC m=+16.676862155 (2 nodes), 480ns from start of poll, 1.713597198s from cluster start. Timeout at 2025-01-26 02:43:41.16683766 +0000 UTC m=+17.681869995, 1.00500784s after stability
2025-01-26T02:43:41.183Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: aborting pipeline replication: peer="{Voter server-9ec03dd9-b4e3-bff8-9e34-6c7d259956a4 9ec03dd9-b4e3-bff8-9e34-6c7d259956a4}"
2025-01-26T02:43:41.183Z [INFO]  server-3c195922-014b-a26b-0059-cdc090659925: aborting pipeline replication: peer="{Voter server-c38691cb-e304-9b26-ebfc-146c462f602d c38691cb-e304-9b26-ebfc-146c462f602d}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false
2025-01-26T02:43:41.184Z [INFO]  server-58803dfc-6826-f2a9-0799-78c21a954531: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58803dfc-6826-f2a9-0799-78c21a954531 Address:58803dfc-6826-f2a9-0799-78c21a954531} {Suffrage:Voter ID:server-2d6ae6f6-6072-2b0e-908b-f94e57fe6120 Address:2d6ae6f6-6072-2b0e-908b-f94e57fe6120} {Suffrage:Voter ID:server-c3f9dcf1-2964-1a7c-d29e-291bb394c663 Address:c3f9dcf1-2964-1a7c-d29e-291bb394c663}]"
2025-01-26T02:43:41.184Z [INFO]  server-2d6ae6f6-6072-2b0e-908b-f94e57fe6120: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58803dfc-6826-f2a9-0799-78c21a954531 Address:58803dfc-6826-f2a9-0799-78c21a954531} {Suffrage:Voter ID:server-2d6ae6f6-6072-2b0e-908b-f94e57fe6120 Address:2d6ae6f6-6072-2b0e-908b-f94e57fe6120} {Suffrage:Voter ID:server-c3f9dcf1-2964-1a7c-d29e-291bb394c663 Address:c3f9dcf1-2964-1a7c-d29e-291bb394c663}]"
2025-01-26T02:43:41.184Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-58803dfc-6826-f2a9-0799-78c21a954531 Address:58803dfc-6826-f2a9-0799-78c21a954531} {Suffrage:Voter ID:server-2d6ae6f6-6072-2b0e-908b-f94e57fe6120 Address:2d6ae6f6-6072-2b0e-908b-f94e57fe6120} {Suffrage:Voter ID:server-c3f9dcf1-2964-1a7c-d29e-291bb394c663 Address:c3f9dcf1-2964-1a7c-d29e-291bb394c663}]"
2025-01-26T02:43:41.184Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:41.184Z [INFO]  server-58803dfc-6826-f2a9-0799-78c21a954531: entering follower state: follower="Node at 58803dfc-6826-f2a9-0799-78c21a954531 [Follower]" leader-address= leader-id=
2025-01-26T02:43:41.184Z [INFO]  server-2d6ae6f6-6072-2b0e-908b-f94e57fe6120: entering follower state: follower="Node at 2d6ae6f6-6072-2b0e-908b-f94e57fe6120 [Follower]" leader-address= leader-id=
2025-01-26T02:43:41.184Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: entering follower state: follower="Node at c3f9dcf1-2964-1a7c-d29e-291bb394c663 [Follower]" leader-address= leader-id=
2025-01-26T02:43:41.759Z [WARN]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:41.759Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: entering candidate state: node="Node at c3f9dcf1-2964-1a7c-d29e-291bb394c663 [Candidate]" term=2
2025-01-26T02:43:41.759Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: election won: term=2 tally=2
2025-01-26T02:43:41.759Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: entering leader state: leader="Node at c3f9dcf1-2964-1a7c-d29e-291bb394c663 [Leader]"
2025-01-26T02:43:41.759Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: added peer, starting replication: peer=server-58803dfc-6826-f2a9-0799-78c21a954531
2025-01-26T02:43:41.759Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: added peer, starting replication: peer=server-2d6ae6f6-6072-2b0e-908b-f94e57fe6120
2025-01-26T02:43:41.759Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: pipelining replication: peer="{Voter server-2d6ae6f6-6072-2b0e-908b-f94e57fe6120 2d6ae6f6-6072-2b0e-908b-f94e57fe6120}"
2025-01-26T02:43:41.759Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: pipelining replication: peer="{Voter server-58803dfc-6826-f2a9-0799-78c21a954531 58803dfc-6826-f2a9-0799-78c21a954531}"
2025-01-26T02:43:42.767Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:41.759719955 +0000 UTC m=+18.274752290 (1 nodes), 575.537052ms from start of poll, 575.720013ms from cluster start. Timeout at 2025-01-26 02:43:42.764725515 +0000 UTC m=+19.279757850, 1.00500556s after stability
2025-01-26T02:43:42.767Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: starting snapshot up to: index=12
2025-01-26T02:43:42.767Z [INFO]  creating new snapshot: path=/tmp/raft3167901718/snapshots/2-12-1737859422767.tmp
2025-01-26T02:43:42.768Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: snapshot complete up to: index=12
2025-01-26T02:43:42.768Z [INFO]  creating new snapshot: path=/tmp/raft3167901718/snapshots/2-23-1737859422768.tmp
2025-01-26T02:43:42.768Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: copied to local snapshot: bytes=71
2025-01-26T02:43:42.768Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: snapshot restore progress: id=2-23-1737859422768 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:42.769Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: removing all old logs from log store
2025-01-26T02:43:42.769Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: compacting logs: from=1 to=22
2025-01-26T02:43:42.769Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: restored user snapshot: index=23
2025-01-26T02:43:42.769Z [ERROR] server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: failed to get log: index=22 error="log not found"
2025-01-26T02:43:42.769Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: aborting pipeline replication: peer="{Voter server-2d6ae6f6-6072-2b0e-908b-f94e57fe6120 2d6ae6f6-6072-2b0e-908b-f94e57fe6120}"
2025-01-26T02:43:42.769Z [ERROR] server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: failed to get log: index=22 error="log not found"
2025-01-26T02:43:42.769Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: aborting pipeline replication: peer="{Voter server-58803dfc-6826-f2a9-0799-78c21a954531 58803dfc-6826-f2a9-0799-78c21a954531}"
2025-01-26T02:43:42.779Z [ERROR] server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: failed to get log: index=22 error="log not found"
2025-01-26T02:43:42.779Z [INFO]  creating new snapshot: path=/tmp/raft2990061570/snapshots/2-23-1737859422779.tmp
2025-01-26T02:43:42.781Z [ERROR] server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: failed to get log: index=22 error="log not found"
2025-01-26T02:43:42.781Z [INFO]  creating new snapshot: path=/tmp/raft206599039/snapshots/2-23-1737859422781.tmp
2025-01-26T02:43:42.781Z [INFO]  server-58803dfc-6826-f2a9-0799-78c21a954531: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:42.781Z [INFO]  server-58803dfc-6826-f2a9-0799-78c21a954531: copied to local snapshot: bytes=71
2025-01-26T02:43:42.781Z [INFO]  server-58803dfc-6826-f2a9-0799-78c21a954531: snapshot restore progress: id=2-23-1737859422781 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:42.781Z [INFO]  server-58803dfc-6826-f2a9-0799-78c21a954531: removing all old logs from log store
2025-01-26T02:43:42.781Z [INFO]  server-58803dfc-6826-f2a9-0799-78c21a954531: compacting logs: from=1 to=22
2025-01-26T02:43:42.781Z [INFO]  server-58803dfc-6826-f2a9-0799-78c21a954531: Installed remote snapshot
2025-01-26T02:43:42.781Z [WARN]  server-58803dfc-6826-f2a9-0799-78c21a954531: skipping application of old log: index=22
2025-01-26T02:43:42.781Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: pipelining replication: peer="{Voter server-58803dfc-6826-f2a9-0799-78c21a954531 58803dfc-6826-f2a9-0799-78c21a954531}"
2025-01-26T02:43:42.781Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:43:42.782Z [INFO]  server-2d6ae6f6-6072-2b0e-908b-f94e57fe6120: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:42.782Z [INFO]  server-2d6ae6f6-6072-2b0e-908b-f94e57fe6120: copied to local snapshot: bytes=71
2025-01-26T02:43:42.782Z [INFO]  server-2d6ae6f6-6072-2b0e-908b-f94e57fe6120: snapshot restore progress: id=2-23-1737859422779 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:42.782Z [INFO]  server-2d6ae6f6-6072-2b0e-908b-f94e57fe6120: removing all old logs from log store
2025-01-26T02:43:42.782Z [INFO]  server-2d6ae6f6-6072-2b0e-908b-f94e57fe6120: compacting logs: from=1 to=22
2025-01-26T02:43:42.782Z [INFO]  server-2d6ae6f6-6072-2b0e-908b-f94e57fe6120: Installed remote snapshot
2025-01-26T02:43:42.782Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: pipelining replication: peer="{Voter server-2d6ae6f6-6072-2b0e-908b-f94e57fe6120 2d6ae6f6-6072-2b0e-908b-f94e57fe6120}"
2025-01-26T02:43:43.787Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:43:42.781976878 +0000 UTC m=+19.297009253 (2 nodes), 680ns from start of poll, 1.597976976s from cluster start. Timeout at 2025-01-26 02:43:43.786990878 +0000 UTC m=+20.302023253, 1.005014s after stability
2025-01-26T02:43:43.803Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: aborting pipeline replication: peer="{Voter server-2d6ae6f6-6072-2b0e-908b-f94e57fe6120 2d6ae6f6-6072-2b0e-908b-f94e57fe6120}"
2025-01-26T02:43:43.803Z [INFO]  server-c3f9dcf1-2964-1a7c-d29e-291bb394c663: aborting pipeline replication: peer="{Voter server-58803dfc-6826-f2a9-0799-78c21a954531 58803dfc-6826-f2a9-0799-78c21a954531}"
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true
2025-01-26T02:43:43.804Z [INFO]  server-d447d30b-0328-934d-32e8-06be8ab02b2d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d447d30b-0328-934d-32e8-06be8ab02b2d Address:d447d30b-0328-934d-32e8-06be8ab02b2d} {Suffrage:Voter ID:server-05419800-9651-c356-6b59-208874488f92 Address:05419800-9651-c356-6b59-208874488f92} {Suffrage:Voter ID:server-a83d9855-3247-dd49-a893-5920fecb0050 Address:a83d9855-3247-dd49-a893-5920fecb0050}]"
2025-01-26T02:43:43.804Z [INFO]  server-05419800-9651-c356-6b59-208874488f92: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d447d30b-0328-934d-32e8-06be8ab02b2d Address:d447d30b-0328-934d-32e8-06be8ab02b2d} {Suffrage:Voter ID:server-05419800-9651-c356-6b59-208874488f92 Address:05419800-9651-c356-6b59-208874488f92} {Suffrage:Voter ID:server-a83d9855-3247-dd49-a893-5920fecb0050 Address:a83d9855-3247-dd49-a893-5920fecb0050}]"
2025-01-26T02:43:43.804Z [INFO]  server-a83d9855-3247-dd49-a893-5920fecb0050: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d447d30b-0328-934d-32e8-06be8ab02b2d Address:d447d30b-0328-934d-32e8-06be8ab02b2d} {Suffrage:Voter ID:server-05419800-9651-c356-6b59-208874488f92 Address:05419800-9651-c356-6b59-208874488f92} {Suffrage:Voter ID:server-a83d9855-3247-dd49-a893-5920fecb0050 Address:a83d9855-3247-dd49-a893-5920fecb0050}]"
2025-01-26T02:43:43.804Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:43.804Z [INFO]  server-d447d30b-0328-934d-32e8-06be8ab02b2d: entering follower state: follower="Node at d447d30b-0328-934d-32e8-06be8ab02b2d [Follower]" leader-address= leader-id=
2025-01-26T02:43:43.804Z [INFO]  server-05419800-9651-c356-6b59-208874488f92: entering follower state: follower="Node at 05419800-9651-c356-6b59-208874488f92 [Follower]" leader-address= leader-id=
2025-01-26T02:43:43.804Z [INFO]  server-a83d9855-3247-dd49-a893-5920fecb0050: entering follower state: follower="Node at a83d9855-3247-dd49-a893-5920fecb0050 [Follower]" leader-address= leader-id=
2025-01-26T02:43:44.503Z [WARN]  server-05419800-9651-c356-6b59-208874488f92: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:44.503Z [INFO]  server-05419800-9651-c356-6b59-208874488f92: entering candidate state: node="Node at 05419800-9651-c356-6b59-208874488f92 [Candidate]" term=2
2025-01-26T02:43:44.503Z [INFO]  server-05419800-9651-c356-6b59-208874488f92: election won: term=2 tally=2
2025-01-26T02:43:44.503Z [INFO]  server-05419800-9651-c356-6b59-208874488f92: entering leader state: leader="Node at 05419800-9651-c356-6b59-208874488f92 [Leader]"
2025-01-26T02:43:44.503Z [INFO]  server-05419800-9651-c356-6b59-208874488f92: added peer, starting replication: peer=server-d447d30b-0328-934d-32e8-06be8ab02b2d
2025-01-26T02:43:44.503Z [INFO]  server-05419800-9651-c356-6b59-208874488f92: added peer, starting replication: peer=server-a83d9855-3247-dd49-a893-5920fecb0050
2025-01-26T02:43:44.503Z [INFO]  server-05419800-9651-c356-6b59-208874488f92: pipelining replication: peer="{Voter server-a83d9855-3247-dd49-a893-5920fecb0050 a83d9855-3247-dd49-a893-5920fecb0050}"
2025-01-26T02:43:44.503Z [INFO]  server-05419800-9651-c356-6b59-208874488f92: pipelining replication: peer="{Voter server-d447d30b-0328-934d-32e8-06be8ab02b2d d447d30b-0328-934d-32e8-06be8ab02b2d}"
2025-01-26T02:43:45.515Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:44.503673685 +0000 UTC m=+21.018706060 (1 nodes), 699.478924ms from start of poll, 699.679445ms from cluster start. Timeout at 2025-01-26 02:43:45.508681525 +0000 UTC m=+22.023713900, 1.00500784s after stability
2025-01-26T02:43:45.515Z [INFO]  server-05419800-9651-c356-6b59-208874488f92: starting snapshot up to: index=12
2025-01-26T02:43:45.515Z [INFO]  creating new snapshot: path=/tmp/raft2568641170/snapshots/2-12-1737859425515.tmp
2025-01-26T02:43:45.516Z [INFO]  server-05419800-9651-c356-6b59-208874488f92: snapshot complete up to: index=12
2025-01-26T02:43:45.516Z [INFO]  server-98d20fe3-12a9-eb58-5493-c4390d9738b7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98d20fe3-12a9-eb58-5493-c4390d9738b7 Address:98d20fe3-12a9-eb58-5493-c4390d9738b7} {Suffrage:Voter ID:server-f2f26855-3728-2c30-7cd2-0cfe932aa214 Address:f2f26855-3728-2c30-7cd2-0cfe932aa214} {Suffrage:Voter ID:server-e50b8af7-fe52-9801-4ba9-c054c0ed9dc8 Address:e50b8af7-fe52-9801-4ba9-c054c0ed9dc8}]"
2025-01-26T02:43:45.516Z [INFO]  server-f2f26855-3728-2c30-7cd2-0cfe932aa214: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98d20fe3-12a9-eb58-5493-c4390d9738b7 Address:98d20fe3-12a9-eb58-5493-c4390d9738b7} {Suffrage:Voter ID:server-f2f26855-3728-2c30-7cd2-0cfe932aa214 Address:f2f26855-3728-2c30-7cd2-0cfe932aa214} {Suffrage:Voter ID:server-e50b8af7-fe52-9801-4ba9-c054c0ed9dc8 Address:e50b8af7-fe52-9801-4ba9-c054c0ed9dc8}]"
2025-01-26T02:43:45.516Z [INFO]  server-e50b8af7-fe52-9801-4ba9-c054c0ed9dc8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-98d20fe3-12a9-eb58-5493-c4390d9738b7 Address:98d20fe3-12a9-eb58-5493-c4390d9738b7} {Suffrage:Voter ID:server-f2f26855-3728-2c30-7cd2-0cfe932aa214 Address:f2f26855-3728-2c30-7cd2-0cfe932aa214} {Suffrage:Voter ID:server-e50b8af7-fe52-9801-4ba9-c054c0ed9dc8 Address:e50b8af7-fe52-9801-4ba9-c054c0ed9dc8}]"
2025-01-26T02:43:45.516Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:45.516Z [INFO]  server-98d20fe3-12a9-eb58-5493-c4390d9738b7: entering follower state: follower="Node at 98d20fe3-12a9-eb58-5493-c4390d9738b7 [Follower]" leader-address= leader-id=
2025-01-26T02:43:45.516Z [INFO]  server-f2f26855-3728-2c30-7cd2-0cfe932aa214: entering follower state: follower="Node at f2f26855-3728-2c30-7cd2-0cfe932aa214 [Follower]" leader-address= leader-id=
2025-01-26T02:43:45.516Z [INFO]  server-e50b8af7-fe52-9801-4ba9-c054c0ed9dc8: entering follower state: follower="Node at e50b8af7-fe52-9801-4ba9-c054c0ed9dc8 [Follower]" leader-address= leader-id=
2025-01-26T02:43:46.231Z [WARN]  server-98d20fe3-12a9-eb58-5493-c4390d9738b7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:46.231Z [INFO]  server-98d20fe3-12a9-eb58-5493-c4390d9738b7: entering candidate state: node="Node at 98d20fe3-12a9-eb58-5493-c4390d9738b7 [Candidate]" term=2
2025-01-26T02:43:46.231Z [INFO]  server-98d20fe3-12a9-eb58-5493-c4390d9738b7: election won: term=2 tally=2
2025-01-26T02:43:46.231Z [INFO]  server-98d20fe3-12a9-eb58-5493-c4390d9738b7: entering leader state: leader="Node at 98d20fe3-12a9-eb58-5493-c4390d9738b7 [Leader]"
2025-01-26T02:43:46.231Z [INFO]  server-98d20fe3-12a9-eb58-5493-c4390d9738b7: added peer, starting replication: peer=server-f2f26855-3728-2c30-7cd2-0cfe932aa214
2025-01-26T02:43:46.231Z [INFO]  server-98d20fe3-12a9-eb58-5493-c4390d9738b7: added peer, starting replication: peer=server-e50b8af7-fe52-9801-4ba9-c054c0ed9dc8
2025-01-26T02:43:46.231Z [INFO]  server-98d20fe3-12a9-eb58-5493-c4390d9738b7: pipelining replication: peer="{Voter server-e50b8af7-fe52-9801-4ba9-c054c0ed9dc8 e50b8af7-fe52-9801-4ba9-c054c0ed9dc8}"
2025-01-26T02:43:46.231Z [INFO]  server-98d20fe3-12a9-eb58-5493-c4390d9738b7: pipelining replication: peer="{Voter server-f2f26855-3728-2c30-7cd2-0cfe932aa214 f2f26855-3728-2c30-7cd2-0cfe932aa214}"
2025-01-26T02:43:47.247Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:46.231850852 +0000 UTC m=+22.746883187 (1 nodes), 715.16306ms from start of poll, 715.333701ms from cluster start. Timeout at 2025-01-26 02:43:47.236855652 +0000 UTC m=+23.751888027, 1.00500484s after stability
2025-01-26T02:43:47.247Z [INFO]  creating new snapshot: path=/tmp/raft1519690628/snapshots/2-13-1737859427247.tmp
2025-01-26T02:43:47.247Z [INFO]  server-98d20fe3-12a9-eb58-5493-c4390d9738b7: copied to local snapshot: bytes=71
2025-01-26T02:43:47.248Z [INFO]  server-98d20fe3-12a9-eb58-5493-c4390d9738b7: snapshot restore progress: id=2-13-1737859427247 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:47.248Z [INFO]  server-98d20fe3-12a9-eb58-5493-c4390d9738b7: restored user snapshot: index=13
2025-01-26T02:43:47.248Z [ERROR] server-98d20fe3-12a9-eb58-5493-c4390d9738b7: failed to get log: index=3 error="log not found"
2025-01-26T02:43:47.248Z [INFO]  server-98d20fe3-12a9-eb58-5493-c4390d9738b7: aborting pipeline replication: peer="{Voter server-e50b8af7-fe52-9801-4ba9-c054c0ed9dc8 e50b8af7-fe52-9801-4ba9-c054c0ed9dc8}"
2025-01-26T02:43:47.248Z [ERROR] server-98d20fe3-12a9-eb58-5493-c4390d9738b7: failed to get log: index=3 error="log not found"
2025-01-26T02:43:47.248Z [INFO]  server-98d20fe3-12a9-eb58-5493-c4390d9738b7: aborting pipeline replication: peer="{Voter server-f2f26855-3728-2c30-7cd2-0cfe932aa214 f2f26855-3728-2c30-7cd2-0cfe932aa214}"
2025-01-26T02:43:47.263Z [ERROR] server-98d20fe3-12a9-eb58-5493-c4390d9738b7: failed to get log: index=3 error="log not found"
2025-01-26T02:43:47.263Z [INFO]  creating new snapshot: path=/tmp/raft80771795/snapshots/2-13-1737859427263.tmp
2025-01-26T02:43:47.263Z [INFO]  server-e50b8af7-fe52-9801-4ba9-c054c0ed9dc8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:47.264Z [INFO]  server-e50b8af7-fe52-9801-4ba9-c054c0ed9dc8: copied to local snapshot: bytes=71
2025-01-26T02:43:47.264Z [INFO]  server-e50b8af7-fe52-9801-4ba9-c054c0ed9dc8: snapshot restore progress: id=2-13-1737859427263 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:47.264Z [INFO]  server-e50b8af7-fe52-9801-4ba9-c054c0ed9dc8: Installed remote snapshot
2025-01-26T02:43:47.264Z [INFO]  server-98d20fe3-12a9-eb58-5493-c4390d9738b7: pipelining replication: peer="{Voter server-e50b8af7-fe52-9801-4ba9-c054c0ed9dc8 e50b8af7-fe52-9801-4ba9-c054c0ed9dc8}"
2025-01-26T02:43:47.263Z [ERROR] server-98d20fe3-12a9-eb58-5493-c4390d9738b7: failed to get log: index=3 error="log not found"
2025-01-26T02:43:47.264Z [INFO]  creating new snapshot: path=/tmp/raft3276173600/snapshots/2-13-1737859427264.tmp
2025-01-26T02:43:47.264Z [INFO]  server-f2f26855-3728-2c30-7cd2-0cfe932aa214: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:47.264Z [INFO]  server-f2f26855-3728-2c30-7cd2-0cfe932aa214: copied to local snapshot: bytes=71
2025-01-26T02:43:47.264Z [INFO]  server-f2f26855-3728-2c30-7cd2-0cfe932aa214: snapshot restore progress: id=2-13-1737859427264 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:47.264Z [INFO]  server-f2f26855-3728-2c30-7cd2-0cfe932aa214: Installed remote snapshot
2025-01-26T02:43:47.264Z [INFO]  server-98d20fe3-12a9-eb58-5493-c4390d9738b7: pipelining replication: peer="{Voter server-f2f26855-3728-2c30-7cd2-0cfe932aa214 f2f26855-3728-2c30-7cd2-0cfe932aa214}"
2025-01-26T02:43:47.264Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:43:48.271Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:43:47.264751559 +0000 UTC m=+23.779783934 (2 nodes), 520ns from start of poll, 1.748234448s from cluster start. Timeout at 2025-01-26 02:43:48.269762199 +0000 UTC m=+24.784794574, 1.00501064s after stability
2025-01-26T02:43:48.287Z [INFO]  server-05419800-9651-c356-6b59-208874488f92: aborting pipeline replication: peer="{Voter server-a83d9855-3247-dd49-a893-5920fecb0050 a83d9855-3247-dd49-a893-5920fecb0050}"
2025-01-26T02:43:48.287Z [INFO]  server-05419800-9651-c356-6b59-208874488f92: aborting pipeline replication: peer="{Voter server-d447d30b-0328-934d-32e8-06be8ab02b2d d447d30b-0328-934d-32e8-06be8ab02b2d}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true
2025-01-26T02:43:48.288Z [INFO]  server-9fc9b99b-398e-5a67-a8cc-9f67a265d20d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9fc9b99b-398e-5a67-a8cc-9f67a265d20d Address:9fc9b99b-398e-5a67-a8cc-9f67a265d20d} {Suffrage:Voter ID:server-3b9b1201-fe08-7a5c-4962-73ac0d69f87f Address:3b9b1201-fe08-7a5c-4962-73ac0d69f87f} {Suffrage:Voter ID:server-c56d6fbc-999b-54e4-45b0-20d1fdcef799 Address:c56d6fbc-999b-54e4-45b0-20d1fdcef799}]"
2025-01-26T02:43:48.288Z [INFO]  server-3b9b1201-fe08-7a5c-4962-73ac0d69f87f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9fc9b99b-398e-5a67-a8cc-9f67a265d20d Address:9fc9b99b-398e-5a67-a8cc-9f67a265d20d} {Suffrage:Voter ID:server-3b9b1201-fe08-7a5c-4962-73ac0d69f87f Address:3b9b1201-fe08-7a5c-4962-73ac0d69f87f} {Suffrage:Voter ID:server-c56d6fbc-999b-54e4-45b0-20d1fdcef799 Address:c56d6fbc-999b-54e4-45b0-20d1fdcef799}]"
2025-01-26T02:43:48.288Z [INFO]  server-c56d6fbc-999b-54e4-45b0-20d1fdcef799: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9fc9b99b-398e-5a67-a8cc-9f67a265d20d Address:9fc9b99b-398e-5a67-a8cc-9f67a265d20d} {Suffrage:Voter ID:server-3b9b1201-fe08-7a5c-4962-73ac0d69f87f Address:3b9b1201-fe08-7a5c-4962-73ac0d69f87f} {Suffrage:Voter ID:server-c56d6fbc-999b-54e4-45b0-20d1fdcef799 Address:c56d6fbc-999b-54e4-45b0-20d1fdcef799}]"
2025-01-26T02:43:48.288Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:48.288Z [INFO]  server-9fc9b99b-398e-5a67-a8cc-9f67a265d20d: entering follower state: follower="Node at 9fc9b99b-398e-5a67-a8cc-9f67a265d20d [Follower]" leader-address= leader-id=
2025-01-26T02:43:48.288Z [INFO]  server-3b9b1201-fe08-7a5c-4962-73ac0d69f87f: entering follower state: follower="Node at 3b9b1201-fe08-7a5c-4962-73ac0d69f87f [Follower]" leader-address= leader-id=
2025-01-26T02:43:48.288Z [INFO]  server-c56d6fbc-999b-54e4-45b0-20d1fdcef799: entering follower state: follower="Node at c56d6fbc-999b-54e4-45b0-20d1fdcef799 [Follower]" leader-address= leader-id=
2025-01-26T02:43:49.039Z [WARN]  server-c56d6fbc-999b-54e4-45b0-20d1fdcef799: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:49.039Z [INFO]  server-c56d6fbc-999b-54e4-45b0-20d1fdcef799: entering candidate state: node="Node at c56d6fbc-999b-54e4-45b0-20d1fdcef799 [Candidate]" term=2
2025-01-26T02:43:49.039Z [INFO]  server-c56d6fbc-999b-54e4-45b0-20d1fdcef799: election won: term=2 tally=2
2025-01-26T02:43:49.039Z [INFO]  server-c56d6fbc-999b-54e4-45b0-20d1fdcef799: entering leader state: leader="Node at c56d6fbc-999b-54e4-45b0-20d1fdcef799 [Leader]"
2025-01-26T02:43:49.039Z [INFO]  server-c56d6fbc-999b-54e4-45b0-20d1fdcef799: added peer, starting replication: peer=server-9fc9b99b-398e-5a67-a8cc-9f67a265d20d
2025-01-26T02:43:49.039Z [INFO]  server-c56d6fbc-999b-54e4-45b0-20d1fdcef799: added peer, starting replication: peer=server-3b9b1201-fe08-7a5c-4962-73ac0d69f87f
2025-01-26T02:43:49.039Z [INFO]  server-c56d6fbc-999b-54e4-45b0-20d1fdcef799: pipelining replication: peer="{Voter server-3b9b1201-fe08-7a5c-4962-73ac0d69f87f 3b9b1201-fe08-7a5c-4962-73ac0d69f87f}"
2025-01-26T02:43:49.039Z [INFO]  server-c56d6fbc-999b-54e4-45b0-20d1fdcef799: pipelining replication: peer="{Voter server-9fc9b99b-398e-5a67-a8cc-9f67a265d20d 9fc9b99b-398e-5a67-a8cc-9f67a265d20d}"
2025-01-26T02:43:50.047Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:49.03972873 +0000 UTC m=+25.554761105 (1 nodes), 751.617561ms from start of poll, 751.799682ms from cluster start. Timeout at 2025-01-26 02:43:50.04473085 +0000 UTC m=+26.559763185, 1.00500208s after stability
2025-01-26T02:43:50.047Z [INFO]  server-c56d6fbc-999b-54e4-45b0-20d1fdcef799: starting snapshot up to: index=12
2025-01-26T02:43:50.047Z [INFO]  creating new snapshot: path=/tmp/raft1277950193/snapshots/2-12-1737859430047.tmp
2025-01-26T02:43:50.048Z [INFO]  server-c56d6fbc-999b-54e4-45b0-20d1fdcef799: snapshot complete up to: index=12
2025-01-26T02:43:50.049Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48b264cf-1aec-9751-8718-07505763cd56 Address:48b264cf-1aec-9751-8718-07505763cd56} {Suffrage:Voter ID:server-2cae26b2-490b-5216-26c2-034be424df96 Address:2cae26b2-490b-5216-26c2-034be424df96} {Suffrage:Voter ID:server-39e313e1-9223-cc14-fb54-e75d6c3e3fcf Address:39e313e1-9223-cc14-fb54-e75d6c3e3fcf}]"
2025-01-26T02:43:50.049Z [INFO]  server-2cae26b2-490b-5216-26c2-034be424df96: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48b264cf-1aec-9751-8718-07505763cd56 Address:48b264cf-1aec-9751-8718-07505763cd56} {Suffrage:Voter ID:server-2cae26b2-490b-5216-26c2-034be424df96 Address:2cae26b2-490b-5216-26c2-034be424df96} {Suffrage:Voter ID:server-39e313e1-9223-cc14-fb54-e75d6c3e3fcf Address:39e313e1-9223-cc14-fb54-e75d6c3e3fcf}]"
2025-01-26T02:43:50.049Z [INFO]  server-39e313e1-9223-cc14-fb54-e75d6c3e3fcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-48b264cf-1aec-9751-8718-07505763cd56 Address:48b264cf-1aec-9751-8718-07505763cd56} {Suffrage:Voter ID:server-2cae26b2-490b-5216-26c2-034be424df96 Address:2cae26b2-490b-5216-26c2-034be424df96} {Suffrage:Voter ID:server-39e313e1-9223-cc14-fb54-e75d6c3e3fcf Address:39e313e1-9223-cc14-fb54-e75d6c3e3fcf}]"
2025-01-26T02:43:50.049Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:50.050Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: entering follower state: follower="Node at 48b264cf-1aec-9751-8718-07505763cd56 [Follower]" leader-address= leader-id=
2025-01-26T02:43:50.055Z [INFO]  server-2cae26b2-490b-5216-26c2-034be424df96: entering follower state: follower="Node at 2cae26b2-490b-5216-26c2-034be424df96 [Follower]" leader-address= leader-id=
2025-01-26T02:43:50.055Z [INFO]  server-39e313e1-9223-cc14-fb54-e75d6c3e3fcf: entering follower state: follower="Node at 39e313e1-9223-cc14-fb54-e75d6c3e3fcf [Follower]" leader-address= leader-id=
2025-01-26T02:43:50.615Z [WARN]  server-48b264cf-1aec-9751-8718-07505763cd56: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:50.615Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: entering candidate state: node="Node at 48b264cf-1aec-9751-8718-07505763cd56 [Candidate]" term=2
2025-01-26T02:43:50.615Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: election won: term=2 tally=2
2025-01-26T02:43:50.615Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: entering leader state: leader="Node at 48b264cf-1aec-9751-8718-07505763cd56 [Leader]"
2025-01-26T02:43:50.615Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: added peer, starting replication: peer=server-2cae26b2-490b-5216-26c2-034be424df96
2025-01-26T02:43:50.615Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: added peer, starting replication: peer=server-39e313e1-9223-cc14-fb54-e75d6c3e3fcf
2025-01-26T02:43:50.615Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: pipelining replication: peer="{Voter server-39e313e1-9223-cc14-fb54-e75d6c3e3fcf 39e313e1-9223-cc14-fb54-e75d6c3e3fcf}"
2025-01-26T02:43:50.615Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: pipelining replication: peer="{Voter server-2cae26b2-490b-5216-26c2-034be424df96 2cae26b2-490b-5216-26c2-034be424df96}"
2025-01-26T02:43:51.623Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:50.615710493 +0000 UTC m=+27.130742868 (1 nodes), 566.087115ms from start of poll, 566.94212ms from cluster start. Timeout at 2025-01-26 02:43:51.620713573 +0000 UTC m=+28.135746028, 1.00500316s after stability
2025-01-26T02:43:51.623Z [INFO]  creating new snapshot: path=/tmp/raft2687011973/snapshots/2-13-1737859431623.tmp
2025-01-26T02:43:51.623Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: copied to local snapshot: bytes=71
2025-01-26T02:43:51.624Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: snapshot restore progress: id=2-13-1737859431623 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:51.624Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: removing all old logs from log store
2025-01-26T02:43:51.624Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: compacting logs: from=1 to=2
2025-01-26T02:43:51.624Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: restored user snapshot: index=13
2025-01-26T02:43:51.624Z [ERROR] server-48b264cf-1aec-9751-8718-07505763cd56: failed to get log: index=2 error="log not found"
2025-01-26T02:43:51.624Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: aborting pipeline replication: peer="{Voter server-39e313e1-9223-cc14-fb54-e75d6c3e3fcf 39e313e1-9223-cc14-fb54-e75d6c3e3fcf}"
2025-01-26T02:43:51.624Z [ERROR] server-48b264cf-1aec-9751-8718-07505763cd56: failed to get log: index=2 error="log not found"
2025-01-26T02:43:51.624Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: aborting pipeline replication: peer="{Voter server-2cae26b2-490b-5216-26c2-034be424df96 2cae26b2-490b-5216-26c2-034be424df96}"
2025-01-26T02:43:51.631Z [ERROR] server-48b264cf-1aec-9751-8718-07505763cd56: failed to get log: index=2 error="log not found"
2025-01-26T02:43:51.631Z [INFO]  creating new snapshot: path=/tmp/raft586374774/snapshots/2-13-1737859431631.tmp
2025-01-26T02:43:51.631Z [INFO]  server-2cae26b2-490b-5216-26c2-034be424df96: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:51.631Z [INFO]  server-2cae26b2-490b-5216-26c2-034be424df96: copied to local snapshot: bytes=71
2025-01-26T02:43:51.632Z [INFO]  server-2cae26b2-490b-5216-26c2-034be424df96: snapshot restore progress: id=2-13-1737859431631 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:51.632Z [INFO]  server-2cae26b2-490b-5216-26c2-034be424df96: removing all old logs from log store
2025-01-26T02:43:51.632Z [INFO]  server-2cae26b2-490b-5216-26c2-034be424df96: compacting logs: from=1 to=2
2025-01-26T02:43:51.632Z [INFO]  server-2cae26b2-490b-5216-26c2-034be424df96: Installed remote snapshot
2025-01-26T02:43:51.632Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: pipelining replication: peer="{Voter server-2cae26b2-490b-5216-26c2-034be424df96 2cae26b2-490b-5216-26c2-034be424df96}"
2025-01-26T02:43:51.632Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:43:51.632Z [ERROR] server-48b264cf-1aec-9751-8718-07505763cd56: failed to get log: index=2 error="log not found"
2025-01-26T02:43:51.632Z [INFO]  creating new snapshot: path=/tmp/raft3353982182/snapshots/2-13-1737859431632.tmp
2025-01-26T02:43:51.632Z [INFO]  server-39e313e1-9223-cc14-fb54-e75d6c3e3fcf: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:51.632Z [INFO]  server-39e313e1-9223-cc14-fb54-e75d6c3e3fcf: copied to local snapshot: bytes=71
2025-01-26T02:43:51.632Z [INFO]  server-39e313e1-9223-cc14-fb54-e75d6c3e3fcf: snapshot restore progress: id=2-13-1737859431632 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:51.632Z [INFO]  server-39e313e1-9223-cc14-fb54-e75d6c3e3fcf: removing all old logs from log store
2025-01-26T02:43:51.632Z [INFO]  server-39e313e1-9223-cc14-fb54-e75d6c3e3fcf: compacting logs: from=1 to=2
2025-01-26T02:43:51.632Z [INFO]  server-39e313e1-9223-cc14-fb54-e75d6c3e3fcf: Installed remote snapshot
2025-01-26T02:43:51.632Z [INFO]  server-48b264cf-1aec-9751-8718-07505763cd56: pipelining replication: peer="{Voter server-39e313e1-9223-cc14-fb54-e75d6c3e3fcf 39e313e1-9223-cc14-fb54-e75d6c3e3fcf}"
2025-01-26T02:43:52.639Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:43:51.632241741 +0000 UTC m=+28.147274116 (2 nodes), 520ns from start of poll, 1.583473368s from cluster start. Timeout at 2025-01-26 02:43:52.637254861 +0000 UTC m=+29.152287236, 1.00501312s after stability
2025-01-26T02:43:52.656Z [INFO]  server-c56d6fbc-999b-54e4-45b0-20d1fdcef799: aborting pipeline replication: peer="{Voter server-3b9b1201-fe08-7a5c-4962-73ac0d69f87f 3b9b1201-fe08-7a5c-4962-73ac0d69f87f}"
2025-01-26T02:43:52.656Z [INFO]  server-c56d6fbc-999b-54e4-45b0-20d1fdcef799: aborting pipeline replication: peer="{Voter server-9fc9b99b-398e-5a67-a8cc-9f67a265d20d 9fc9b99b-398e-5a67-a8cc-9f67a265d20d}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false
2025-01-26T02:43:52.656Z [INFO]  server-454a9901-90b6-92b6-aa43-b6140632ddb5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-454a9901-90b6-92b6-aa43-b6140632ddb5 Address:454a9901-90b6-92b6-aa43-b6140632ddb5} {Suffrage:Voter ID:server-7152cd0c-5107-da5e-cfbf-692a1a4d75b2 Address:7152cd0c-5107-da5e-cfbf-692a1a4d75b2} {Suffrage:Voter ID:server-8dfd5fce-d520-670f-d3ff-29e7daa38f51 Address:8dfd5fce-d520-670f-d3ff-29e7daa38f51}]"
2025-01-26T02:43:52.656Z [INFO]  server-7152cd0c-5107-da5e-cfbf-692a1a4d75b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-454a9901-90b6-92b6-aa43-b6140632ddb5 Address:454a9901-90b6-92b6-aa43-b6140632ddb5} {Suffrage:Voter ID:server-7152cd0c-5107-da5e-cfbf-692a1a4d75b2 Address:7152cd0c-5107-da5e-cfbf-692a1a4d75b2} {Suffrage:Voter ID:server-8dfd5fce-d520-670f-d3ff-29e7daa38f51 Address:8dfd5fce-d520-670f-d3ff-29e7daa38f51}]"
2025-01-26T02:43:52.656Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-454a9901-90b6-92b6-aa43-b6140632ddb5 Address:454a9901-90b6-92b6-aa43-b6140632ddb5} {Suffrage:Voter ID:server-7152cd0c-5107-da5e-cfbf-692a1a4d75b2 Address:7152cd0c-5107-da5e-cfbf-692a1a4d75b2} {Suffrage:Voter ID:server-8dfd5fce-d520-670f-d3ff-29e7daa38f51 Address:8dfd5fce-d520-670f-d3ff-29e7daa38f51}]"
2025-01-26T02:43:52.656Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:52.656Z [INFO]  server-454a9901-90b6-92b6-aa43-b6140632ddb5: entering follower state: follower="Node at 454a9901-90b6-92b6-aa43-b6140632ddb5 [Follower]" leader-address= leader-id=
2025-01-26T02:43:52.656Z [INFO]  server-7152cd0c-5107-da5e-cfbf-692a1a4d75b2: entering follower state: follower="Node at 7152cd0c-5107-da5e-cfbf-692a1a4d75b2 [Follower]" leader-address= leader-id=
2025-01-26T02:43:52.656Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: entering follower state: follower="Node at 8dfd5fce-d520-670f-d3ff-29e7daa38f51 [Follower]" leader-address= leader-id=
2025-01-26T02:43:53.283Z [WARN]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:53.283Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: entering candidate state: node="Node at 8dfd5fce-d520-670f-d3ff-29e7daa38f51 [Candidate]" term=2
2025-01-26T02:43:53.283Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: election won: term=2 tally=2
2025-01-26T02:43:53.283Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: entering leader state: leader="Node at 8dfd5fce-d520-670f-d3ff-29e7daa38f51 [Leader]"
2025-01-26T02:43:53.283Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: added peer, starting replication: peer=server-454a9901-90b6-92b6-aa43-b6140632ddb5
2025-01-26T02:43:53.283Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: added peer, starting replication: peer=server-7152cd0c-5107-da5e-cfbf-692a1a4d75b2
2025-01-26T02:43:53.283Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: pipelining replication: peer="{Voter server-7152cd0c-5107-da5e-cfbf-692a1a4d75b2 7152cd0c-5107-da5e-cfbf-692a1a4d75b2}"
2025-01-26T02:43:53.283Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: pipelining replication: peer="{Voter server-454a9901-90b6-92b6-aa43-b6140632ddb5 454a9901-90b6-92b6-aa43-b6140632ddb5}"
2025-01-26T02:43:54.291Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:53.283816723 +0000 UTC m=+29.798849098 (1 nodes), 627.181006ms from start of poll, 627.362487ms from cluster start. Timeout at 2025-01-26 02:43:54.288822403 +0000 UTC m=+30.803854778, 1.00500568s after stability
2025-01-26T02:43:54.291Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: starting snapshot up to: index=12
2025-01-26T02:43:54.291Z [INFO]  creating new snapshot: path=/tmp/raft793986731/snapshots/2-12-1737859434291.tmp
2025-01-26T02:43:54.292Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: snapshot complete up to: index=12
2025-01-26T02:43:54.292Z [INFO]  creating new snapshot: path=/tmp/raft793986731/snapshots/2-23-1737859434292.tmp
2025-01-26T02:43:54.292Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: copied to local snapshot: bytes=71
2025-01-26T02:43:54.293Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: snapshot restore progress: id=2-23-1737859434292 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:54.293Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: restored user snapshot: index=23
2025-01-26T02:43:54.293Z [ERROR] server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: failed to get log: index=23 error="log not found"
2025-01-26T02:43:54.293Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: aborting pipeline replication: peer="{Voter server-7152cd0c-5107-da5e-cfbf-692a1a4d75b2 7152cd0c-5107-da5e-cfbf-692a1a4d75b2}"
2025-01-26T02:43:54.293Z [ERROR] server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: failed to get log: index=23 error="log not found"
2025-01-26T02:43:54.293Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: aborting pipeline replication: peer="{Voter server-454a9901-90b6-92b6-aa43-b6140632ddb5 454a9901-90b6-92b6-aa43-b6140632ddb5}"
2025-01-26T02:43:54.300Z [ERROR] server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: failed to get log: index=23 error="log not found"
2025-01-26T02:43:54.301Z [INFO]  creating new snapshot: path=/tmp/raft945347729/snapshots/2-23-1737859434301.tmp
2025-01-26T02:43:54.301Z [INFO]  server-454a9901-90b6-92b6-aa43-b6140632ddb5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:54.301Z [INFO]  server-454a9901-90b6-92b6-aa43-b6140632ddb5: copied to local snapshot: bytes=71
2025-01-26T02:43:54.301Z [INFO]  server-454a9901-90b6-92b6-aa43-b6140632ddb5: snapshot restore progress: id=2-23-1737859434301 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:54.301Z [INFO]  server-454a9901-90b6-92b6-aa43-b6140632ddb5: Installed remote snapshot
2025-01-26T02:43:54.301Z [WARN]  server-454a9901-90b6-92b6-aa43-b6140632ddb5: skipping application of old log: index=22
2025-01-26T02:43:54.301Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: pipelining replication: peer="{Voter server-454a9901-90b6-92b6-aa43-b6140632ddb5 454a9901-90b6-92b6-aa43-b6140632ddb5}"
2025-01-26T02:43:54.301Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:43:54.302Z [ERROR] server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: failed to get log: index=23 error="log not found"
2025-01-26T02:43:54.302Z [INFO]  creating new snapshot: path=/tmp/raft3786977654/snapshots/2-23-1737859434302.tmp
2025-01-26T02:43:54.302Z [INFO]  server-7152cd0c-5107-da5e-cfbf-692a1a4d75b2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:54.302Z [INFO]  server-7152cd0c-5107-da5e-cfbf-692a1a4d75b2: copied to local snapshot: bytes=71
2025-01-26T02:43:54.302Z [INFO]  server-7152cd0c-5107-da5e-cfbf-692a1a4d75b2: snapshot restore progress: id=2-23-1737859434302 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:54.302Z [INFO]  server-7152cd0c-5107-da5e-cfbf-692a1a4d75b2: Installed remote snapshot
2025-01-26T02:43:54.302Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: pipelining replication: peer="{Voter server-7152cd0c-5107-da5e-cfbf-692a1a4d75b2 7152cd0c-5107-da5e-cfbf-692a1a4d75b2}"
2025-01-26T02:43:55.307Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:43:54.30182654 +0000 UTC m=+30.816858915 (2 nodes), 480ns from start of poll, 1.645372304s from cluster start. Timeout at 2025-01-26 02:43:55.30683298 +0000 UTC m=+31.821865355, 1.00500644s after stability
2025-01-26T02:43:55.323Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: aborting pipeline replication: peer="{Voter server-7152cd0c-5107-da5e-cfbf-692a1a4d75b2 7152cd0c-5107-da5e-cfbf-692a1a4d75b2}"
2025-01-26T02:43:55.323Z [INFO]  server-8dfd5fce-d520-670f-d3ff-29e7daa38f51: aborting pipeline replication: peer="{Voter server-454a9901-90b6-92b6-aa43-b6140632ddb5 454a9901-90b6-92b6-aa43-b6140632ddb5}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false
2025-01-26T02:43:55.324Z [INFO]  server-e50819f7-27a3-6112-2df8-bde4313999f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e50819f7-27a3-6112-2df8-bde4313999f6 Address:e50819f7-27a3-6112-2df8-bde4313999f6} {Suffrage:Voter ID:server-27f0dc80-508e-1139-2647-a144903efc43 Address:27f0dc80-508e-1139-2647-a144903efc43} {Suffrage:Voter ID:server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed Address:0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed}]"
2025-01-26T02:43:55.324Z [INFO]  server-27f0dc80-508e-1139-2647-a144903efc43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e50819f7-27a3-6112-2df8-bde4313999f6 Address:e50819f7-27a3-6112-2df8-bde4313999f6} {Suffrage:Voter ID:server-27f0dc80-508e-1139-2647-a144903efc43 Address:27f0dc80-508e-1139-2647-a144903efc43} {Suffrage:Voter ID:server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed Address:0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed}]"
2025-01-26T02:43:55.324Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e50819f7-27a3-6112-2df8-bde4313999f6 Address:e50819f7-27a3-6112-2df8-bde4313999f6} {Suffrage:Voter ID:server-27f0dc80-508e-1139-2647-a144903efc43 Address:27f0dc80-508e-1139-2647-a144903efc43} {Suffrage:Voter ID:server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed Address:0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed}]"
2025-01-26T02:43:55.324Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:55.324Z [INFO]  server-e50819f7-27a3-6112-2df8-bde4313999f6: entering follower state: follower="Node at e50819f7-27a3-6112-2df8-bde4313999f6 [Follower]" leader-address= leader-id=
2025-01-26T02:43:55.324Z [INFO]  server-27f0dc80-508e-1139-2647-a144903efc43: entering follower state: follower="Node at 27f0dc80-508e-1139-2647-a144903efc43 [Follower]" leader-address= leader-id=
2025-01-26T02:43:55.324Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: entering follower state: follower="Node at 0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed [Follower]" leader-address= leader-id=
2025-01-26T02:43:55.947Z [WARN]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:55.947Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: entering candidate state: node="Node at 0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed [Candidate]" term=2
2025-01-26T02:43:55.947Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: election won: term=2 tally=2
2025-01-26T02:43:55.947Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: entering leader state: leader="Node at 0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed [Leader]"
2025-01-26T02:43:55.947Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: added peer, starting replication: peer=server-e50819f7-27a3-6112-2df8-bde4313999f6
2025-01-26T02:43:55.947Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: added peer, starting replication: peer=server-27f0dc80-508e-1139-2647-a144903efc43
2025-01-26T02:43:55.947Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: pipelining replication: peer="{Voter server-27f0dc80-508e-1139-2647-a144903efc43 27f0dc80-508e-1139-2647-a144903efc43}"
2025-01-26T02:43:55.947Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: pipelining replication: peer="{Voter server-e50819f7-27a3-6112-2df8-bde4313999f6 e50819f7-27a3-6112-2df8-bde4313999f6}"
2025-01-26T02:43:56.955Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:55.947966128 +0000 UTC m=+32.462998503 (1 nodes), 623.714224ms from start of poll, 623.906225ms from cluster start. Timeout at 2025-01-26 02:43:56.952969328 +0000 UTC m=+33.468001663, 1.00500316s after stability
2025-01-26T02:43:56.955Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: starting snapshot up to: index=12
2025-01-26T02:43:56.955Z [INFO]  creating new snapshot: path=/tmp/raft3439658731/snapshots/2-12-1737859436955.tmp
2025-01-26T02:43:56.956Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: snapshot complete up to: index=12
2025-01-26T02:43:56.963Z [INFO]  creating new snapshot: path=/tmp/raft3439658731/snapshots/2-23-1737859436963.tmp
2025-01-26T02:43:56.964Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: copied to local snapshot: bytes=71
2025-01-26T02:43:56.964Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: snapshot restore progress: id=2-23-1737859436963 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:56.964Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: removing all old logs from log store
2025-01-26T02:43:56.964Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: compacting logs: from=1 to=22
2025-01-26T02:43:56.964Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: restored user snapshot: index=23
2025-01-26T02:43:56.964Z [ERROR] server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: failed to get log: index=22 error="log not found"
2025-01-26T02:43:56.964Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: aborting pipeline replication: peer="{Voter server-27f0dc80-508e-1139-2647-a144903efc43 27f0dc80-508e-1139-2647-a144903efc43}"
2025-01-26T02:43:56.964Z [ERROR] server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: failed to get log: index=22 error="log not found"
2025-01-26T02:43:56.964Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: aborting pipeline replication: peer="{Voter server-e50819f7-27a3-6112-2df8-bde4313999f6 e50819f7-27a3-6112-2df8-bde4313999f6}"
2025-01-26T02:43:56.971Z [ERROR] server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: failed to get log: index=22 error="log not found"
2025-01-26T02:43:56.971Z [INFO]  creating new snapshot: path=/tmp/raft3918774588/snapshots/2-23-1737859436971.tmp
2025-01-26T02:43:56.971Z [INFO]  server-27f0dc80-508e-1139-2647-a144903efc43: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:56.971Z [INFO]  server-27f0dc80-508e-1139-2647-a144903efc43: copied to local snapshot: bytes=71
2025-01-26T02:43:56.971Z [INFO]  server-27f0dc80-508e-1139-2647-a144903efc43: snapshot restore progress: id=2-23-1737859436971 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:56.971Z [INFO]  server-27f0dc80-508e-1139-2647-a144903efc43: removing all old logs from log store
2025-01-26T02:43:56.971Z [INFO]  server-27f0dc80-508e-1139-2647-a144903efc43: compacting logs: from=1 to=22
2025-01-26T02:43:56.971Z [INFO]  server-27f0dc80-508e-1139-2647-a144903efc43: Installed remote snapshot
2025-01-26T02:43:56.972Z [WARN]  server-27f0dc80-508e-1139-2647-a144903efc43: skipping application of old log: index=22
2025-01-26T02:43:56.972Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: pipelining replication: peer="{Voter server-27f0dc80-508e-1139-2647-a144903efc43 27f0dc80-508e-1139-2647-a144903efc43}"
2025-01-26T02:43:56.972Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:43:56.975Z [ERROR] server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: failed to get log: index=22 error="log not found"
2025-01-26T02:43:56.975Z [INFO]  creating new snapshot: path=/tmp/raft2357264204/snapshots/2-23-1737859436975.tmp
2025-01-26T02:43:56.975Z [INFO]  server-e50819f7-27a3-6112-2df8-bde4313999f6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:56.975Z [INFO]  server-e50819f7-27a3-6112-2df8-bde4313999f6: copied to local snapshot: bytes=71
2025-01-26T02:43:56.976Z [INFO]  server-e50819f7-27a3-6112-2df8-bde4313999f6: snapshot restore progress: id=2-23-1737859436975 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:43:56.976Z [INFO]  server-e50819f7-27a3-6112-2df8-bde4313999f6: removing all old logs from log store
2025-01-26T02:43:56.976Z [INFO]  server-e50819f7-27a3-6112-2df8-bde4313999f6: compacting logs: from=1 to=22
2025-01-26T02:43:56.976Z [INFO]  server-e50819f7-27a3-6112-2df8-bde4313999f6: Installed remote snapshot
2025-01-26T02:43:56.976Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: pipelining replication: peer="{Voter server-e50819f7-27a3-6112-2df8-bde4313999f6 e50819f7-27a3-6112-2df8-bde4313999f6}"
2025-01-26T02:43:57.979Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:43:56.972078543 +0000 UTC m=+33.487110918 (2 nodes), 520ns from start of poll, 1.64801864s from cluster start. Timeout at 2025-01-26 02:43:57.977080943 +0000 UTC m=+34.492113318, 1.0050024s after stability
2025-01-26T02:43:57.995Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: aborting pipeline replication: peer="{Voter server-27f0dc80-508e-1139-2647-a144903efc43 27f0dc80-508e-1139-2647-a144903efc43}"
2025-01-26T02:43:57.995Z [INFO]  server-0d2ccac3-ebc0-b844-7d9f-753cc6a4c2ed: aborting pipeline replication: peer="{Voter server-e50819f7-27a3-6112-2df8-bde4313999f6 e50819f7-27a3-6112-2df8-bde4313999f6}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true
2025-01-26T02:43:57.996Z [INFO]  server-01205d06-aaa4-d1f9-385a-4dd2dd41ea2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01205d06-aaa4-d1f9-385a-4dd2dd41ea2a Address:01205d06-aaa4-d1f9-385a-4dd2dd41ea2a} {Suffrage:Voter ID:server-c3da24e0-1ffe-82ba-47e8-0f140b8c74f8 Address:c3da24e0-1ffe-82ba-47e8-0f140b8c74f8} {Suffrage:Voter ID:server-58f0477a-32d1-9116-b97d-4031e243f062 Address:58f0477a-32d1-9116-b97d-4031e243f062}]"
2025-01-26T02:43:57.996Z [INFO]  server-c3da24e0-1ffe-82ba-47e8-0f140b8c74f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01205d06-aaa4-d1f9-385a-4dd2dd41ea2a Address:01205d06-aaa4-d1f9-385a-4dd2dd41ea2a} {Suffrage:Voter ID:server-c3da24e0-1ffe-82ba-47e8-0f140b8c74f8 Address:c3da24e0-1ffe-82ba-47e8-0f140b8c74f8} {Suffrage:Voter ID:server-58f0477a-32d1-9116-b97d-4031e243f062 Address:58f0477a-32d1-9116-b97d-4031e243f062}]"
2025-01-26T02:43:57.996Z [INFO]  server-58f0477a-32d1-9116-b97d-4031e243f062: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-01205d06-aaa4-d1f9-385a-4dd2dd41ea2a Address:01205d06-aaa4-d1f9-385a-4dd2dd41ea2a} {Suffrage:Voter ID:server-c3da24e0-1ffe-82ba-47e8-0f140b8c74f8 Address:c3da24e0-1ffe-82ba-47e8-0f140b8c74f8} {Suffrage:Voter ID:server-58f0477a-32d1-9116-b97d-4031e243f062 Address:58f0477a-32d1-9116-b97d-4031e243f062}]"
2025-01-26T02:43:57.996Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:57.996Z [INFO]  server-01205d06-aaa4-d1f9-385a-4dd2dd41ea2a: entering follower state: follower="Node at 01205d06-aaa4-d1f9-385a-4dd2dd41ea2a [Follower]" leader-address= leader-id=
2025-01-26T02:43:57.996Z [INFO]  server-c3da24e0-1ffe-82ba-47e8-0f140b8c74f8: entering follower state: follower="Node at c3da24e0-1ffe-82ba-47e8-0f140b8c74f8 [Follower]" leader-address= leader-id=
2025-01-26T02:43:57.996Z [INFO]  server-58f0477a-32d1-9116-b97d-4031e243f062: entering follower state: follower="Node at 58f0477a-32d1-9116-b97d-4031e243f062 [Follower]" leader-address= leader-id=
2025-01-26T02:43:58.807Z [WARN]  server-c3da24e0-1ffe-82ba-47e8-0f140b8c74f8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:43:58.807Z [INFO]  server-c3da24e0-1ffe-82ba-47e8-0f140b8c74f8: entering candidate state: node="Node at c3da24e0-1ffe-82ba-47e8-0f140b8c74f8 [Candidate]" term=2
2025-01-26T02:43:58.807Z [INFO]  server-c3da24e0-1ffe-82ba-47e8-0f140b8c74f8: election won: term=2 tally=2
2025-01-26T02:43:58.807Z [INFO]  server-c3da24e0-1ffe-82ba-47e8-0f140b8c74f8: entering leader state: leader="Node at c3da24e0-1ffe-82ba-47e8-0f140b8c74f8 [Leader]"
2025-01-26T02:43:58.807Z [INFO]  server-c3da24e0-1ffe-82ba-47e8-0f140b8c74f8: added peer, starting replication: peer=server-01205d06-aaa4-d1f9-385a-4dd2dd41ea2a
2025-01-26T02:43:58.807Z [INFO]  server-c3da24e0-1ffe-82ba-47e8-0f140b8c74f8: added peer, starting replication: peer=server-58f0477a-32d1-9116-b97d-4031e243f062
2025-01-26T02:43:58.807Z [INFO]  server-c3da24e0-1ffe-82ba-47e8-0f140b8c74f8: pipelining replication: peer="{Voter server-58f0477a-32d1-9116-b97d-4031e243f062 58f0477a-32d1-9116-b97d-4031e243f062}"
2025-01-26T02:43:58.807Z [INFO]  server-c3da24e0-1ffe-82ba-47e8-0f140b8c74f8: pipelining replication: peer="{Voter server-01205d06-aaa4-d1f9-385a-4dd2dd41ea2a 01205d06-aaa4-d1f9-385a-4dd2dd41ea2a}"
2025-01-26T02:43:59.815Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:43:58.807818402 +0000 UTC m=+35.322850777 (1 nodes), 811.655205ms from start of poll, 811.828806ms from cluster start. Timeout at 2025-01-26 02:43:59.812822082 +0000 UTC m=+36.327854457, 1.00500368s after stability
2025-01-26T02:43:59.815Z [INFO]  server-c3da24e0-1ffe-82ba-47e8-0f140b8c74f8: starting snapshot up to: index=12
2025-01-26T02:43:59.815Z [INFO]  creating new snapshot: path=/tmp/raft2984103512/snapshots/2-12-1737859439815.tmp
2025-01-26T02:43:59.816Z [INFO]  server-c3da24e0-1ffe-82ba-47e8-0f140b8c74f8: snapshot complete up to: index=12
2025-01-26T02:43:59.816Z [INFO]  server-77587b54-8b0a-9efd-3955-f6d36b637089: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77587b54-8b0a-9efd-3955-f6d36b637089 Address:77587b54-8b0a-9efd-3955-f6d36b637089} {Suffrage:Voter ID:server-e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a Address:e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a} {Suffrage:Voter ID:server-4e403b65-c64a-2828-ddd2-a94c0e9927cb Address:4e403b65-c64a-2828-ddd2-a94c0e9927cb}]"
2025-01-26T02:43:59.816Z [INFO]  server-e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77587b54-8b0a-9efd-3955-f6d36b637089 Address:77587b54-8b0a-9efd-3955-f6d36b637089} {Suffrage:Voter ID:server-e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a Address:e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a} {Suffrage:Voter ID:server-4e403b65-c64a-2828-ddd2-a94c0e9927cb Address:4e403b65-c64a-2828-ddd2-a94c0e9927cb}]"
2025-01-26T02:43:59.816Z [INFO]  server-4e403b65-c64a-2828-ddd2-a94c0e9927cb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-77587b54-8b0a-9efd-3955-f6d36b637089 Address:77587b54-8b0a-9efd-3955-f6d36b637089} {Suffrage:Voter ID:server-e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a Address:e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a} {Suffrage:Voter ID:server-4e403b65-c64a-2828-ddd2-a94c0e9927cb Address:4e403b65-c64a-2828-ddd2-a94c0e9927cb}]"
2025-01-26T02:43:59.816Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:43:59.816Z [INFO]  server-77587b54-8b0a-9efd-3955-f6d36b637089: entering follower state: follower="Node at 77587b54-8b0a-9efd-3955-f6d36b637089 [Follower]" leader-address= leader-id=
2025-01-26T02:43:59.816Z [INFO]  server-e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a: entering follower state: follower="Node at e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a [Follower]" leader-address= leader-id=
2025-01-26T02:43:59.817Z [INFO]  server-4e403b65-c64a-2828-ddd2-a94c0e9927cb: entering follower state: follower="Node at 4e403b65-c64a-2828-ddd2-a94c0e9927cb [Follower]" leader-address= leader-id=
2025-01-26T02:44:00.335Z [WARN]  server-77587b54-8b0a-9efd-3955-f6d36b637089: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:00.335Z [INFO]  server-77587b54-8b0a-9efd-3955-f6d36b637089: entering candidate state: node="Node at 77587b54-8b0a-9efd-3955-f6d36b637089 [Candidate]" term=2
2025-01-26T02:44:00.335Z [INFO]  server-77587b54-8b0a-9efd-3955-f6d36b637089: election won: term=2 tally=2
2025-01-26T02:44:00.335Z [INFO]  server-77587b54-8b0a-9efd-3955-f6d36b637089: entering leader state: leader="Node at 77587b54-8b0a-9efd-3955-f6d36b637089 [Leader]"
2025-01-26T02:44:00.335Z [INFO]  server-77587b54-8b0a-9efd-3955-f6d36b637089: added peer, starting replication: peer=server-e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a
2025-01-26T02:44:00.335Z [INFO]  server-77587b54-8b0a-9efd-3955-f6d36b637089: added peer, starting replication: peer=server-4e403b65-c64a-2828-ddd2-a94c0e9927cb
2025-01-26T02:44:00.335Z [INFO]  server-77587b54-8b0a-9efd-3955-f6d36b637089: pipelining replication: peer="{Voter server-4e403b65-c64a-2828-ddd2-a94c0e9927cb 4e403b65-c64a-2828-ddd2-a94c0e9927cb}"
2025-01-26T02:44:00.335Z [INFO]  server-77587b54-8b0a-9efd-3955-f6d36b637089: pipelining replication: peer="{Voter server-e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a}"
2025-01-26T02:44:01.343Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:00.335746713 +0000 UTC m=+36.850779088 (1 nodes), 518.855348ms from start of poll, 519.020029ms from cluster start. Timeout at 2025-01-26 02:44:01.340762433 +0000 UTC m=+37.855794808, 1.00501572s after stability
2025-01-26T02:44:01.343Z [INFO]  creating new snapshot: path=/tmp/raft3726528381/snapshots/2-14-1737859441343.tmp
2025-01-26T02:44:01.343Z [INFO]  server-77587b54-8b0a-9efd-3955-f6d36b637089: copied to local snapshot: bytes=71
2025-01-26T02:44:01.344Z [INFO]  server-77587b54-8b0a-9efd-3955-f6d36b637089: snapshot restore progress: id=2-14-1737859441343 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:01.344Z [INFO]  server-77587b54-8b0a-9efd-3955-f6d36b637089: restored user snapshot: index=14
2025-01-26T02:44:01.344Z [ERROR] server-77587b54-8b0a-9efd-3955-f6d36b637089: failed to get log: index=3 error="log not found"
2025-01-26T02:44:01.344Z [INFO]  server-77587b54-8b0a-9efd-3955-f6d36b637089: aborting pipeline replication: peer="{Voter server-4e403b65-c64a-2828-ddd2-a94c0e9927cb 4e403b65-c64a-2828-ddd2-a94c0e9927cb}"
2025-01-26T02:44:01.344Z [ERROR] server-77587b54-8b0a-9efd-3955-f6d36b637089: failed to get log: index=3 error="log not found"
2025-01-26T02:44:01.344Z [INFO]  server-77587b54-8b0a-9efd-3955-f6d36b637089: aborting pipeline replication: peer="{Voter server-e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a}"
2025-01-26T02:44:01.355Z [ERROR] server-77587b54-8b0a-9efd-3955-f6d36b637089: failed to get log: index=3 error="log not found"
2025-01-26T02:44:01.355Z [INFO]  creating new snapshot: path=/tmp/raft2135749081/snapshots/2-14-1737859441355.tmp
2025-01-26T02:44:01.355Z [INFO]  server-e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:01.355Z [INFO]  server-e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a: copied to local snapshot: bytes=71
2025-01-26T02:44:01.356Z [INFO]  server-e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a: snapshot restore progress: id=2-14-1737859441355 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:01.356Z [INFO]  server-e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a: Installed remote snapshot
2025-01-26T02:44:01.356Z [INFO]  server-77587b54-8b0a-9efd-3955-f6d36b637089: pipelining replication: peer="{Voter server-e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a e89cba34-9ebe-41c3-fdf9-9d1ae1b18f6a}"
2025-01-26T02:44:01.356Z [ERROR] server-77587b54-8b0a-9efd-3955-f6d36b637089: failed to get log: index=3 error="log not found"
2025-01-26T02:44:01.356Z [INFO]  creating new snapshot: path=/tmp/raft3097180542/snapshots/2-14-1737859441356.tmp
2025-01-26T02:44:01.356Z [INFO]  server-4e403b65-c64a-2828-ddd2-a94c0e9927cb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:01.356Z [INFO]  server-4e403b65-c64a-2828-ddd2-a94c0e9927cb: copied to local snapshot: bytes=71
2025-01-26T02:44:01.356Z [INFO]  server-4e403b65-c64a-2828-ddd2-a94c0e9927cb: snapshot restore progress: id=2-14-1737859441356 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:01.356Z [INFO]  server-4e403b65-c64a-2828-ddd2-a94c0e9927cb: Installed remote snapshot
2025-01-26T02:44:01.356Z [INFO]  server-77587b54-8b0a-9efd-3955-f6d36b637089: pipelining replication: peer="{Voter server-4e403b65-c64a-2828-ddd2-a94c0e9927cb 4e403b65-c64a-2828-ddd2-a94c0e9927cb}"
2025-01-26T02:44:01.356Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:44:02.363Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:44:01.356696028 +0000 UTC m=+37.871728403 (2 nodes), 640ns from start of poll, 1.539969344s from cluster start. Timeout at 2025-01-26 02:44:02.361713348 +0000 UTC m=+38.876745723, 1.00501732s after stability
2025-01-26T02:44:02.379Z [INFO]  server-c3da24e0-1ffe-82ba-47e8-0f140b8c74f8: aborting pipeline replication: peer="{Voter server-58f0477a-32d1-9116-b97d-4031e243f062 58f0477a-32d1-9116-b97d-4031e243f062}"
2025-01-26T02:44:02.379Z [INFO]  server-c3da24e0-1ffe-82ba-47e8-0f140b8c74f8: aborting pipeline replication: peer="{Voter server-01205d06-aaa4-d1f9-385a-4dd2dd41ea2a 01205d06-aaa4-d1f9-385a-4dd2dd41ea2a}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true
2025-01-26T02:44:02.380Z [INFO]  server-07cab742-b4d8-6dca-dc3a-a477923dcec1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07cab742-b4d8-6dca-dc3a-a477923dcec1 Address:07cab742-b4d8-6dca-dc3a-a477923dcec1} {Suffrage:Voter ID:server-1dd7ac8d-03de-de7b-3772-77c24c90cc88 Address:1dd7ac8d-03de-de7b-3772-77c24c90cc88} {Suffrage:Voter ID:server-8bf90fb5-981e-7386-b258-bb7fcadb528e Address:8bf90fb5-981e-7386-b258-bb7fcadb528e}]"
2025-01-26T02:44:02.380Z [INFO]  server-1dd7ac8d-03de-de7b-3772-77c24c90cc88: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07cab742-b4d8-6dca-dc3a-a477923dcec1 Address:07cab742-b4d8-6dca-dc3a-a477923dcec1} {Suffrage:Voter ID:server-1dd7ac8d-03de-de7b-3772-77c24c90cc88 Address:1dd7ac8d-03de-de7b-3772-77c24c90cc88} {Suffrage:Voter ID:server-8bf90fb5-981e-7386-b258-bb7fcadb528e Address:8bf90fb5-981e-7386-b258-bb7fcadb528e}]"
2025-01-26T02:44:02.380Z [INFO]  server-8bf90fb5-981e-7386-b258-bb7fcadb528e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07cab742-b4d8-6dca-dc3a-a477923dcec1 Address:07cab742-b4d8-6dca-dc3a-a477923dcec1} {Suffrage:Voter ID:server-1dd7ac8d-03de-de7b-3772-77c24c90cc88 Address:1dd7ac8d-03de-de7b-3772-77c24c90cc88} {Suffrage:Voter ID:server-8bf90fb5-981e-7386-b258-bb7fcadb528e Address:8bf90fb5-981e-7386-b258-bb7fcadb528e}]"
2025-01-26T02:44:02.380Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:02.380Z [INFO]  server-07cab742-b4d8-6dca-dc3a-a477923dcec1: entering follower state: follower="Node at 07cab742-b4d8-6dca-dc3a-a477923dcec1 [Follower]" leader-address= leader-id=
2025-01-26T02:44:02.380Z [INFO]  server-1dd7ac8d-03de-de7b-3772-77c24c90cc88: entering follower state: follower="Node at 1dd7ac8d-03de-de7b-3772-77c24c90cc88 [Follower]" leader-address= leader-id=
2025-01-26T02:44:02.380Z [INFO]  server-8bf90fb5-981e-7386-b258-bb7fcadb528e: entering follower state: follower="Node at 8bf90fb5-981e-7386-b258-bb7fcadb528e [Follower]" leader-address= leader-id=
2025-01-26T02:44:02.903Z [WARN]  server-1dd7ac8d-03de-de7b-3772-77c24c90cc88: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:02.903Z [INFO]  server-1dd7ac8d-03de-de7b-3772-77c24c90cc88: entering candidate state: node="Node at 1dd7ac8d-03de-de7b-3772-77c24c90cc88 [Candidate]" term=2
2025-01-26T02:44:02.903Z [INFO]  server-1dd7ac8d-03de-de7b-3772-77c24c90cc88: election won: term=2 tally=2
2025-01-26T02:44:02.903Z [INFO]  server-1dd7ac8d-03de-de7b-3772-77c24c90cc88: entering leader state: leader="Node at 1dd7ac8d-03de-de7b-3772-77c24c90cc88 [Leader]"
2025-01-26T02:44:02.903Z [INFO]  server-1dd7ac8d-03de-de7b-3772-77c24c90cc88: added peer, starting replication: peer=server-07cab742-b4d8-6dca-dc3a-a477923dcec1
2025-01-26T02:44:02.903Z [INFO]  server-1dd7ac8d-03de-de7b-3772-77c24c90cc88: added peer, starting replication: peer=server-8bf90fb5-981e-7386-b258-bb7fcadb528e
2025-01-26T02:44:02.903Z [INFO]  server-1dd7ac8d-03de-de7b-3772-77c24c90cc88: pipelining replication: peer="{Voter server-8bf90fb5-981e-7386-b258-bb7fcadb528e 8bf90fb5-981e-7386-b258-bb7fcadb528e}"
2025-01-26T02:44:02.903Z [INFO]  server-1dd7ac8d-03de-de7b-3772-77c24c90cc88: pipelining replication: peer="{Voter server-07cab742-b4d8-6dca-dc3a-a477923dcec1 07cab742-b4d8-6dca-dc3a-a477923dcec1}"
2025-01-26T02:44:03.911Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:02.903751656 +0000 UTC m=+39.418784031 (1 nodes), 523.495297ms from start of poll, 523.680698ms from cluster start. Timeout at 2025-01-26 02:44:03.908762336 +0000 UTC m=+40.423794711, 1.00501068s after stability
2025-01-26T02:44:03.911Z [INFO]  server-1dd7ac8d-03de-de7b-3772-77c24c90cc88: starting snapshot up to: index=12
2025-01-26T02:44:03.911Z [INFO]  creating new snapshot: path=/tmp/raft1968626138/snapshots/2-12-1737859443911.tmp
2025-01-26T02:44:03.912Z [INFO]  server-1dd7ac8d-03de-de7b-3772-77c24c90cc88: snapshot complete up to: index=12
2025-01-26T02:44:03.912Z [INFO]  server-23f690fe-3adf-0036-5a0b-e6b7f3232a97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-23f690fe-3adf-0036-5a0b-e6b7f3232a97 Address:23f690fe-3adf-0036-5a0b-e6b7f3232a97} {Suffrage:Voter ID:server-7c9fd10c-da82-f2d1-c043-ead6e2261f62 Address:7c9fd10c-da82-f2d1-c043-ead6e2261f62} {Suffrage:Voter ID:server-375b81c7-bd87-377e-b97f-4f501bd993f0 Address:375b81c7-bd87-377e-b97f-4f501bd993f0}]"
2025-01-26T02:44:03.912Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-23f690fe-3adf-0036-5a0b-e6b7f3232a97 Address:23f690fe-3adf-0036-5a0b-e6b7f3232a97} {Suffrage:Voter ID:server-7c9fd10c-da82-f2d1-c043-ead6e2261f62 Address:7c9fd10c-da82-f2d1-c043-ead6e2261f62} {Suffrage:Voter ID:server-375b81c7-bd87-377e-b97f-4f501bd993f0 Address:375b81c7-bd87-377e-b97f-4f501bd993f0}]"
2025-01-26T02:44:03.912Z [INFO]  server-375b81c7-bd87-377e-b97f-4f501bd993f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-23f690fe-3adf-0036-5a0b-e6b7f3232a97 Address:23f690fe-3adf-0036-5a0b-e6b7f3232a97} {Suffrage:Voter ID:server-7c9fd10c-da82-f2d1-c043-ead6e2261f62 Address:7c9fd10c-da82-f2d1-c043-ead6e2261f62} {Suffrage:Voter ID:server-375b81c7-bd87-377e-b97f-4f501bd993f0 Address:375b81c7-bd87-377e-b97f-4f501bd993f0}]"
2025-01-26T02:44:03.912Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:03.912Z [INFO]  server-23f690fe-3adf-0036-5a0b-e6b7f3232a97: entering follower state: follower="Node at 23f690fe-3adf-0036-5a0b-e6b7f3232a97 [Follower]" leader-address= leader-id=
2025-01-26T02:44:03.912Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: entering follower state: follower="Node at 7c9fd10c-da82-f2d1-c043-ead6e2261f62 [Follower]" leader-address= leader-id=
2025-01-26T02:44:03.912Z [INFO]  server-375b81c7-bd87-377e-b97f-4f501bd993f0: entering follower state: follower="Node at 375b81c7-bd87-377e-b97f-4f501bd993f0 [Follower]" leader-address= leader-id=
2025-01-26T02:44:04.747Z [WARN]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:04.747Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: entering candidate state: node="Node at 7c9fd10c-da82-f2d1-c043-ead6e2261f62 [Candidate]" term=2
2025-01-26T02:44:04.747Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: election won: term=2 tally=2
2025-01-26T02:44:04.747Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: entering leader state: leader="Node at 7c9fd10c-da82-f2d1-c043-ead6e2261f62 [Leader]"
2025-01-26T02:44:04.747Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: added peer, starting replication: peer=server-23f690fe-3adf-0036-5a0b-e6b7f3232a97
2025-01-26T02:44:04.747Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: added peer, starting replication: peer=server-375b81c7-bd87-377e-b97f-4f501bd993f0
2025-01-26T02:44:04.747Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: pipelining replication: peer="{Voter server-375b81c7-bd87-377e-b97f-4f501bd993f0 375b81c7-bd87-377e-b97f-4f501bd993f0}"
2025-01-26T02:44:04.747Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: pipelining replication: peer="{Voter server-23f690fe-3adf-0036-5a0b-e6b7f3232a97 23f690fe-3adf-0036-5a0b-e6b7f3232a97}"
2025-01-26T02:44:05.755Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:04.747743045 +0000 UTC m=+41.262775420 (1 nodes), 834.939947ms from start of poll, 835.110868ms from cluster start. Timeout at 2025-01-26 02:44:05.752748326 +0000 UTC m=+42.267780701, 1.005005281s after stability
2025-01-26T02:44:05.755Z [INFO]  creating new snapshot: path=/tmp/raft1553824538/snapshots/2-14-1737859445755.tmp
2025-01-26T02:44:05.755Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: copied to local snapshot: bytes=71
2025-01-26T02:44:05.756Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: snapshot restore progress: id=2-14-1737859445755 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:05.756Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: removing all old logs from log store
2025-01-26T02:44:05.756Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: compacting logs: from=1 to=2
2025-01-26T02:44:05.756Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: restored user snapshot: index=14
2025-01-26T02:44:05.756Z [ERROR] server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: failed to get log: index=2 error="log not found"
2025-01-26T02:44:05.756Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: aborting pipeline replication: peer="{Voter server-375b81c7-bd87-377e-b97f-4f501bd993f0 375b81c7-bd87-377e-b97f-4f501bd993f0}"
2025-01-26T02:44:05.756Z [ERROR] server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: failed to get log: index=2 error="log not found"
2025-01-26T02:44:05.756Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: aborting pipeline replication: peer="{Voter server-23f690fe-3adf-0036-5a0b-e6b7f3232a97 23f690fe-3adf-0036-5a0b-e6b7f3232a97}"
2025-01-26T02:44:05.763Z [ERROR] server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: failed to get log: index=2 error="log not found"
2025-01-26T02:44:05.763Z [INFO]  creating new snapshot: path=/tmp/raft1279413228/snapshots/2-14-1737859445763.tmp
2025-01-26T02:44:05.763Z [INFO]  server-23f690fe-3adf-0036-5a0b-e6b7f3232a97: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:05.764Z [INFO]  server-23f690fe-3adf-0036-5a0b-e6b7f3232a97: copied to local snapshot: bytes=71
2025-01-26T02:44:05.764Z [INFO]  server-23f690fe-3adf-0036-5a0b-e6b7f3232a97: snapshot restore progress: id=2-14-1737859445763 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:05.764Z [INFO]  server-23f690fe-3adf-0036-5a0b-e6b7f3232a97: removing all old logs from log store
2025-01-26T02:44:05.764Z [INFO]  server-23f690fe-3adf-0036-5a0b-e6b7f3232a97: compacting logs: from=1 to=2
2025-01-26T02:44:05.764Z [INFO]  server-23f690fe-3adf-0036-5a0b-e6b7f3232a97: Installed remote snapshot
2025-01-26T02:44:05.764Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: pipelining replication: peer="{Voter server-23f690fe-3adf-0036-5a0b-e6b7f3232a97 23f690fe-3adf-0036-5a0b-e6b7f3232a97}"
2025-01-26T02:44:05.764Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:44:05.775Z [ERROR] server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: failed to get log: index=2 error="log not found"
2025-01-26T02:44:05.776Z [INFO]  creating new snapshot: path=/tmp/raft478758050/snapshots/2-14-1737859445776.tmp
2025-01-26T02:44:05.776Z [INFO]  server-375b81c7-bd87-377e-b97f-4f501bd993f0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:05.776Z [INFO]  server-375b81c7-bd87-377e-b97f-4f501bd993f0: copied to local snapshot: bytes=71
2025-01-26T02:44:05.776Z [INFO]  server-375b81c7-bd87-377e-b97f-4f501bd993f0: snapshot restore progress: id=2-14-1737859445776 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:05.776Z [INFO]  server-375b81c7-bd87-377e-b97f-4f501bd993f0: removing all old logs from log store
2025-01-26T02:44:05.776Z [INFO]  server-375b81c7-bd87-377e-b97f-4f501bd993f0: compacting logs: from=1 to=2
2025-01-26T02:44:05.776Z [INFO]  server-375b81c7-bd87-377e-b97f-4f501bd993f0: Installed remote snapshot
2025-01-26T02:44:05.776Z [INFO]  server-7c9fd10c-da82-f2d1-c043-ead6e2261f62: pipelining replication: peer="{Voter server-375b81c7-bd87-377e-b97f-4f501bd993f0 375b81c7-bd87-377e-b97f-4f501bd993f0}"
2025-01-26T02:44:06.770Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:44:05.764276573 +0000 UTC m=+42.279308948 (2 nodes), 720ns from start of poll, 1.851644396s from cluster start. Timeout at 2025-01-26 02:44:06.769287973 +0000 UTC m=+43.284320348, 1.0050114s after stability
2025-01-26T02:44:06.784Z [INFO]  server-1dd7ac8d-03de-de7b-3772-77c24c90cc88: aborting pipeline replication: peer="{Voter server-8bf90fb5-981e-7386-b258-bb7fcadb528e 8bf90fb5-981e-7386-b258-bb7fcadb528e}"
2025-01-26T02:44:06.784Z [INFO]  server-1dd7ac8d-03de-de7b-3772-77c24c90cc88: aborting pipeline replication: peer="{Voter server-07cab742-b4d8-6dca-dc3a-a477923dcec1 07cab742-b4d8-6dca-dc3a-a477923dcec1}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false
2025-01-26T02:44:06.785Z [INFO]  server-12274929-4ed1-51bc-af4e-3cab9a43b532: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12274929-4ed1-51bc-af4e-3cab9a43b532 Address:12274929-4ed1-51bc-af4e-3cab9a43b532} {Suffrage:Voter ID:server-1a399594-5116-8a10-30eb-8a63418e5c36 Address:1a399594-5116-8a10-30eb-8a63418e5c36} {Suffrage:Voter ID:server-a513b384-96f8-41ca-592a-584efc442be8 Address:a513b384-96f8-41ca-592a-584efc442be8}]"
2025-01-26T02:44:06.785Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12274929-4ed1-51bc-af4e-3cab9a43b532 Address:12274929-4ed1-51bc-af4e-3cab9a43b532} {Suffrage:Voter ID:server-1a399594-5116-8a10-30eb-8a63418e5c36 Address:1a399594-5116-8a10-30eb-8a63418e5c36} {Suffrage:Voter ID:server-a513b384-96f8-41ca-592a-584efc442be8 Address:a513b384-96f8-41ca-592a-584efc442be8}]"
2025-01-26T02:44:06.785Z [INFO]  server-a513b384-96f8-41ca-592a-584efc442be8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12274929-4ed1-51bc-af4e-3cab9a43b532 Address:12274929-4ed1-51bc-af4e-3cab9a43b532} {Suffrage:Voter ID:server-1a399594-5116-8a10-30eb-8a63418e5c36 Address:1a399594-5116-8a10-30eb-8a63418e5c36} {Suffrage:Voter ID:server-a513b384-96f8-41ca-592a-584efc442be8 Address:a513b384-96f8-41ca-592a-584efc442be8}]"
2025-01-26T02:44:06.785Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:06.785Z [INFO]  server-12274929-4ed1-51bc-af4e-3cab9a43b532: entering follower state: follower="Node at 12274929-4ed1-51bc-af4e-3cab9a43b532 [Follower]" leader-address= leader-id=
2025-01-26T02:44:06.785Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: entering follower state: follower="Node at 1a399594-5116-8a10-30eb-8a63418e5c36 [Follower]" leader-address= leader-id=
2025-01-26T02:44:06.785Z [INFO]  server-a513b384-96f8-41ca-592a-584efc442be8: entering follower state: follower="Node at a513b384-96f8-41ca-592a-584efc442be8 [Follower]" leader-address= leader-id=
2025-01-26T02:44:07.571Z [WARN]  server-1a399594-5116-8a10-30eb-8a63418e5c36: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:07.571Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: entering candidate state: node="Node at 1a399594-5116-8a10-30eb-8a63418e5c36 [Candidate]" term=2
2025-01-26T02:44:07.571Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: election won: term=2 tally=2
2025-01-26T02:44:07.571Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: entering leader state: leader="Node at 1a399594-5116-8a10-30eb-8a63418e5c36 [Leader]"
2025-01-26T02:44:07.571Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: added peer, starting replication: peer=server-12274929-4ed1-51bc-af4e-3cab9a43b532
2025-01-26T02:44:07.571Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: added peer, starting replication: peer=server-a513b384-96f8-41ca-592a-584efc442be8
2025-01-26T02:44:07.571Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: pipelining replication: peer="{Voter server-a513b384-96f8-41ca-592a-584efc442be8 a513b384-96f8-41ca-592a-584efc442be8}"
2025-01-26T02:44:07.571Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: pipelining replication: peer="{Voter server-12274929-4ed1-51bc-af4e-3cab9a43b532 12274929-4ed1-51bc-af4e-3cab9a43b532}"
2025-01-26T02:44:08.579Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:07.572022262 +0000 UTC m=+44.087054637 (1 nodes), 786.652693ms from start of poll, 786.840254ms from cluster start. Timeout at 2025-01-26 02:44:08.577029262 +0000 UTC m=+45.092061637, 1.005007s after stability
2025-01-26T02:44:08.580Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: starting snapshot up to: index=12
2025-01-26T02:44:08.580Z [INFO]  creating new snapshot: path=/tmp/raft500838940/snapshots/2-12-1737859448580.tmp
2025-01-26T02:44:08.581Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: snapshot complete up to: index=12
2025-01-26T02:44:08.588Z [INFO]  creating new snapshot: path=/tmp/raft500838940/snapshots/2-23-1737859448588.tmp
2025-01-26T02:44:08.588Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: copied to local snapshot: bytes=71
2025-01-26T02:44:08.588Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: snapshot restore progress: id=2-23-1737859448588 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:08.588Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: restored user snapshot: index=23
2025-01-26T02:44:08.588Z [ERROR] server-1a399594-5116-8a10-30eb-8a63418e5c36: failed to get log: index=23 error="log not found"
2025-01-26T02:44:08.588Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: aborting pipeline replication: peer="{Voter server-a513b384-96f8-41ca-592a-584efc442be8 a513b384-96f8-41ca-592a-584efc442be8}"
2025-01-26T02:44:08.588Z [ERROR] server-1a399594-5116-8a10-30eb-8a63418e5c36: failed to get log: index=23 error="log not found"
2025-01-26T02:44:08.588Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: aborting pipeline replication: peer="{Voter server-12274929-4ed1-51bc-af4e-3cab9a43b532 12274929-4ed1-51bc-af4e-3cab9a43b532}"
2025-01-26T02:44:08.599Z [ERROR] server-1a399594-5116-8a10-30eb-8a63418e5c36: failed to get log: index=23 error="log not found"
2025-01-26T02:44:08.599Z [INFO]  creating new snapshot: path=/tmp/raft1290477267/snapshots/2-23-1737859448599.tmp
2025-01-26T02:44:08.599Z [INFO]  server-a513b384-96f8-41ca-592a-584efc442be8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:08.599Z [INFO]  server-a513b384-96f8-41ca-592a-584efc442be8: copied to local snapshot: bytes=71
2025-01-26T02:44:08.600Z [INFO]  server-a513b384-96f8-41ca-592a-584efc442be8: snapshot restore progress: id=2-23-1737859448599 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:08.600Z [INFO]  server-a513b384-96f8-41ca-592a-584efc442be8: Installed remote snapshot
2025-01-26T02:44:08.600Z [WARN]  server-a513b384-96f8-41ca-592a-584efc442be8: skipping application of old log: index=22
2025-01-26T02:44:08.600Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: pipelining replication: peer="{Voter server-a513b384-96f8-41ca-592a-584efc442be8 a513b384-96f8-41ca-592a-584efc442be8}"
2025-01-26T02:44:08.600Z [ERROR] server-1a399594-5116-8a10-30eb-8a63418e5c36: failed to get log: index=23 error="log not found"
2025-01-26T02:44:08.600Z [INFO]  creating new snapshot: path=/tmp/raft614170275/snapshots/2-23-1737859448600.tmp
2025-01-26T02:44:08.600Z [INFO]  server-12274929-4ed1-51bc-af4e-3cab9a43b532: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:08.600Z [INFO]  server-12274929-4ed1-51bc-af4e-3cab9a43b532: copied to local snapshot: bytes=71
2025-01-26T02:44:08.600Z [INFO]  server-12274929-4ed1-51bc-af4e-3cab9a43b532: snapshot restore progress: id=2-23-1737859448600 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:08.600Z [INFO]  server-12274929-4ed1-51bc-af4e-3cab9a43b532: Installed remote snapshot
2025-01-26T02:44:08.600Z [WARN]  server-12274929-4ed1-51bc-af4e-3cab9a43b532: skipping application of old log: index=22
2025-01-26T02:44:08.600Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: pipelining replication: peer="{Voter server-12274929-4ed1-51bc-af4e-3cab9a43b532 12274929-4ed1-51bc-af4e-3cab9a43b532}"
2025-01-26T02:44:08.600Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:44:09.607Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:44:08.600814425 +0000 UTC m=+45.115846800 (2 nodes), 600ns from start of poll, 1.815632417s from cluster start. Timeout at 2025-01-26 02:44:09.605862706 +0000 UTC m=+46.120895081, 1.005048281s after stability
2025-01-26T02:44:09.627Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: aborting pipeline replication: peer="{Voter server-a513b384-96f8-41ca-592a-584efc442be8 a513b384-96f8-41ca-592a-584efc442be8}"
2025-01-26T02:44:09.627Z [INFO]  server-1a399594-5116-8a10-30eb-8a63418e5c36: aborting pipeline replication: peer="{Voter server-12274929-4ed1-51bc-af4e-3cab9a43b532 12274929-4ed1-51bc-af4e-3cab9a43b532}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false
2025-01-26T02:44:09.628Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94f40034-4bfb-017a-e036-85a627bb1c1f Address:94f40034-4bfb-017a-e036-85a627bb1c1f} {Suffrage:Voter ID:server-45e80730-fbd2-2ecb-7d65-6249d9e80e9b Address:45e80730-fbd2-2ecb-7d65-6249d9e80e9b} {Suffrage:Voter ID:server-1d567b9a-59f5-9d22-f341-5a0e7be3c78f Address:1d567b9a-59f5-9d22-f341-5a0e7be3c78f}]"
2025-01-26T02:44:09.628Z [INFO]  server-45e80730-fbd2-2ecb-7d65-6249d9e80e9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94f40034-4bfb-017a-e036-85a627bb1c1f Address:94f40034-4bfb-017a-e036-85a627bb1c1f} {Suffrage:Voter ID:server-45e80730-fbd2-2ecb-7d65-6249d9e80e9b Address:45e80730-fbd2-2ecb-7d65-6249d9e80e9b} {Suffrage:Voter ID:server-1d567b9a-59f5-9d22-f341-5a0e7be3c78f Address:1d567b9a-59f5-9d22-f341-5a0e7be3c78f}]"
2025-01-26T02:44:09.628Z [INFO]  server-1d567b9a-59f5-9d22-f341-5a0e7be3c78f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-94f40034-4bfb-017a-e036-85a627bb1c1f Address:94f40034-4bfb-017a-e036-85a627bb1c1f} {Suffrage:Voter ID:server-45e80730-fbd2-2ecb-7d65-6249d9e80e9b Address:45e80730-fbd2-2ecb-7d65-6249d9e80e9b} {Suffrage:Voter ID:server-1d567b9a-59f5-9d22-f341-5a0e7be3c78f Address:1d567b9a-59f5-9d22-f341-5a0e7be3c78f}]"
2025-01-26T02:44:09.628Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:09.628Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: entering follower state: follower="Node at 94f40034-4bfb-017a-e036-85a627bb1c1f [Follower]" leader-address= leader-id=
2025-01-26T02:44:09.628Z [INFO]  server-45e80730-fbd2-2ecb-7d65-6249d9e80e9b: entering follower state: follower="Node at 45e80730-fbd2-2ecb-7d65-6249d9e80e9b [Follower]" leader-address= leader-id=
2025-01-26T02:44:09.628Z [INFO]  server-1d567b9a-59f5-9d22-f341-5a0e7be3c78f: entering follower state: follower="Node at 1d567b9a-59f5-9d22-f341-5a0e7be3c78f [Follower]" leader-address= leader-id=
2025-01-26T02:44:10.399Z [WARN]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:10.399Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: entering candidate state: node="Node at 94f40034-4bfb-017a-e036-85a627bb1c1f [Candidate]" term=2
2025-01-26T02:44:10.399Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: election won: term=2 tally=2
2025-01-26T02:44:10.399Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: entering leader state: leader="Node at 94f40034-4bfb-017a-e036-85a627bb1c1f [Leader]"
2025-01-26T02:44:10.399Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: added peer, starting replication: peer=server-45e80730-fbd2-2ecb-7d65-6249d9e80e9b
2025-01-26T02:44:10.399Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: added peer, starting replication: peer=server-1d567b9a-59f5-9d22-f341-5a0e7be3c78f
2025-01-26T02:44:10.399Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: pipelining replication: peer="{Voter server-1d567b9a-59f5-9d22-f341-5a0e7be3c78f 1d567b9a-59f5-9d22-f341-5a0e7be3c78f}"
2025-01-26T02:44:10.399Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: pipelining replication: peer="{Voter server-45e80730-fbd2-2ecb-7d65-6249d9e80e9b 45e80730-fbd2-2ecb-7d65-6249d9e80e9b}"
2025-01-26T02:44:11.407Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:10.39960354 +0000 UTC m=+46.914635915 (1 nodes), 771.29832ms from start of poll, 771.473641ms from cluster start. Timeout at 2025-01-26 02:44:11.4046063 +0000 UTC m=+47.919638675, 1.00500276s after stability
2025-01-26T02:44:11.408Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: starting snapshot up to: index=12
2025-01-26T02:44:11.408Z [INFO]  creating new snapshot: path=/tmp/raft844300721/snapshots/2-12-1737859451408.tmp
2025-01-26T02:44:11.408Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: snapshot complete up to: index=12
2025-01-26T02:44:11.409Z [INFO]  creating new snapshot: path=/tmp/raft844300721/snapshots/2-23-1737859451409.tmp
2025-01-26T02:44:11.409Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: copied to local snapshot: bytes=71
2025-01-26T02:44:11.409Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: snapshot restore progress: id=2-23-1737859451409 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:11.409Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: removing all old logs from log store
2025-01-26T02:44:11.409Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: compacting logs: from=1 to=22
2025-01-26T02:44:11.409Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: restored user snapshot: index=23
2025-01-26T02:44:11.409Z [ERROR] server-94f40034-4bfb-017a-e036-85a627bb1c1f: failed to get log: index=22 error="log not found"
2025-01-26T02:44:11.409Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: aborting pipeline replication: peer="{Voter server-1d567b9a-59f5-9d22-f341-5a0e7be3c78f 1d567b9a-59f5-9d22-f341-5a0e7be3c78f}"
2025-01-26T02:44:11.409Z [ERROR] server-94f40034-4bfb-017a-e036-85a627bb1c1f: failed to get log: index=22 error="log not found"
2025-01-26T02:44:11.409Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: aborting pipeline replication: peer="{Voter server-45e80730-fbd2-2ecb-7d65-6249d9e80e9b 45e80730-fbd2-2ecb-7d65-6249d9e80e9b}"
2025-01-26T02:44:11.419Z [ERROR] server-94f40034-4bfb-017a-e036-85a627bb1c1f: failed to get log: index=22 error="log not found"
2025-01-26T02:44:11.419Z [INFO]  creating new snapshot: path=/tmp/raft22127714/snapshots/2-23-1737859451419.tmp
2025-01-26T02:44:11.419Z [INFO]  server-45e80730-fbd2-2ecb-7d65-6249d9e80e9b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:11.419Z [INFO]  server-45e80730-fbd2-2ecb-7d65-6249d9e80e9b: copied to local snapshot: bytes=71
2025-01-26T02:44:11.420Z [INFO]  server-45e80730-fbd2-2ecb-7d65-6249d9e80e9b: snapshot restore progress: id=2-23-1737859451419 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:11.420Z [INFO]  server-45e80730-fbd2-2ecb-7d65-6249d9e80e9b: removing all old logs from log store
2025-01-26T02:44:11.420Z [INFO]  server-45e80730-fbd2-2ecb-7d65-6249d9e80e9b: compacting logs: from=1 to=22
2025-01-26T02:44:11.420Z [INFO]  server-45e80730-fbd2-2ecb-7d65-6249d9e80e9b: Installed remote snapshot
2025-01-26T02:44:11.420Z [WARN]  server-45e80730-fbd2-2ecb-7d65-6249d9e80e9b: skipping application of old log: index=22
2025-01-26T02:44:11.420Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: pipelining replication: peer="{Voter server-45e80730-fbd2-2ecb-7d65-6249d9e80e9b 45e80730-fbd2-2ecb-7d65-6249d9e80e9b}"
2025-01-26T02:44:11.420Z [ERROR] server-94f40034-4bfb-017a-e036-85a627bb1c1f: failed to get log: index=22 error="log not found"
2025-01-26T02:44:11.420Z [INFO]  creating new snapshot: path=/tmp/raft2503126779/snapshots/2-23-1737859451420.tmp
2025-01-26T02:44:11.420Z [INFO]  server-1d567b9a-59f5-9d22-f341-5a0e7be3c78f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:11.420Z [INFO]  server-1d567b9a-59f5-9d22-f341-5a0e7be3c78f: copied to local snapshot: bytes=71
2025-01-26T02:44:11.420Z [INFO]  server-1d567b9a-59f5-9d22-f341-5a0e7be3c78f: snapshot restore progress: id=2-23-1737859451420 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:11.420Z [INFO]  server-1d567b9a-59f5-9d22-f341-5a0e7be3c78f: removing all old logs from log store
2025-01-26T02:44:11.420Z [INFO]  server-1d567b9a-59f5-9d22-f341-5a0e7be3c78f: compacting logs: from=1 to=22
2025-01-26T02:44:11.420Z [INFO]  server-1d567b9a-59f5-9d22-f341-5a0e7be3c78f: Installed remote snapshot
2025-01-26T02:44:11.420Z [WARN]  server-1d567b9a-59f5-9d22-f341-5a0e7be3c78f: skipping application of old log: index=22
2025-01-26T02:44:11.420Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: pipelining replication: peer="{Voter server-1d567b9a-59f5-9d22-f341-5a0e7be3c78f 1d567b9a-59f5-9d22-f341-5a0e7be3c78f}"
2025-01-26T02:44:11.420Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:44:12.427Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:44:11.420799296 +0000 UTC m=+47.935831671 (2 nodes), 520ns from start of poll, 1.792669397s from cluster start. Timeout at 2025-01-26 02:44:12.425800736 +0000 UTC m=+48.940833111, 1.00500144s after stability
2025-01-26T02:44:12.443Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: aborting pipeline replication: peer="{Voter server-1d567b9a-59f5-9d22-f341-5a0e7be3c78f 1d567b9a-59f5-9d22-f341-5a0e7be3c78f}"
2025-01-26T02:44:12.443Z [INFO]  server-94f40034-4bfb-017a-e036-85a627bb1c1f: aborting pipeline replication: peer="{Voter server-45e80730-fbd2-2ecb-7d65-6249d9e80e9b 45e80730-fbd2-2ecb-7d65-6249d9e80e9b}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true
2025-01-26T02:44:12.444Z [INFO]  server-35227329-aaa9-c06f-6c21-098925406480: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35227329-aaa9-c06f-6c21-098925406480 Address:35227329-aaa9-c06f-6c21-098925406480} {Suffrage:Voter ID:server-04d0cbdf-3acf-a07c-a3c5-089bc61013bc Address:04d0cbdf-3acf-a07c-a3c5-089bc61013bc} {Suffrage:Voter ID:server-f9e480bf-a87b-7c65-a9de-978ef1457aa2 Address:f9e480bf-a87b-7c65-a9de-978ef1457aa2}]"
2025-01-26T02:44:12.444Z [INFO]  server-04d0cbdf-3acf-a07c-a3c5-089bc61013bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35227329-aaa9-c06f-6c21-098925406480 Address:35227329-aaa9-c06f-6c21-098925406480} {Suffrage:Voter ID:server-04d0cbdf-3acf-a07c-a3c5-089bc61013bc Address:04d0cbdf-3acf-a07c-a3c5-089bc61013bc} {Suffrage:Voter ID:server-f9e480bf-a87b-7c65-a9de-978ef1457aa2 Address:f9e480bf-a87b-7c65-a9de-978ef1457aa2}]"
2025-01-26T02:44:12.444Z [INFO]  server-f9e480bf-a87b-7c65-a9de-978ef1457aa2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35227329-aaa9-c06f-6c21-098925406480 Address:35227329-aaa9-c06f-6c21-098925406480} {Suffrage:Voter ID:server-04d0cbdf-3acf-a07c-a3c5-089bc61013bc Address:04d0cbdf-3acf-a07c-a3c5-089bc61013bc} {Suffrage:Voter ID:server-f9e480bf-a87b-7c65-a9de-978ef1457aa2 Address:f9e480bf-a87b-7c65-a9de-978ef1457aa2}]"
2025-01-26T02:44:12.444Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:12.444Z [INFO]  server-35227329-aaa9-c06f-6c21-098925406480: entering follower state: follower="Node at 35227329-aaa9-c06f-6c21-098925406480 [Follower]" leader-address= leader-id=
2025-01-26T02:44:12.444Z [INFO]  server-04d0cbdf-3acf-a07c-a3c5-089bc61013bc: entering follower state: follower="Node at 04d0cbdf-3acf-a07c-a3c5-089bc61013bc [Follower]" leader-address= leader-id=
2025-01-26T02:44:12.444Z [INFO]  server-f9e480bf-a87b-7c65-a9de-978ef1457aa2: entering follower state: follower="Node at f9e480bf-a87b-7c65-a9de-978ef1457aa2 [Follower]" leader-address= leader-id=
2025-01-26T02:44:12.983Z [WARN]  server-04d0cbdf-3acf-a07c-a3c5-089bc61013bc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:12.983Z [INFO]  server-04d0cbdf-3acf-a07c-a3c5-089bc61013bc: entering candidate state: node="Node at 04d0cbdf-3acf-a07c-a3c5-089bc61013bc [Candidate]" term=2
2025-01-26T02:44:12.983Z [INFO]  server-04d0cbdf-3acf-a07c-a3c5-089bc61013bc: election won: term=2 tally=2
2025-01-26T02:44:12.983Z [INFO]  server-04d0cbdf-3acf-a07c-a3c5-089bc61013bc: entering leader state: leader="Node at 04d0cbdf-3acf-a07c-a3c5-089bc61013bc [Leader]"
2025-01-26T02:44:12.983Z [INFO]  server-04d0cbdf-3acf-a07c-a3c5-089bc61013bc: added peer, starting replication: peer=server-35227329-aaa9-c06f-6c21-098925406480
2025-01-26T02:44:12.983Z [INFO]  server-04d0cbdf-3acf-a07c-a3c5-089bc61013bc: added peer, starting replication: peer=server-f9e480bf-a87b-7c65-a9de-978ef1457aa2
2025-01-26T02:44:12.983Z [INFO]  server-04d0cbdf-3acf-a07c-a3c5-089bc61013bc: pipelining replication: peer="{Voter server-f9e480bf-a87b-7c65-a9de-978ef1457aa2 f9e480bf-a87b-7c65-a9de-978ef1457aa2}"
2025-01-26T02:44:12.983Z [INFO]  server-04d0cbdf-3acf-a07c-a3c5-089bc61013bc: pipelining replication: peer="{Voter server-35227329-aaa9-c06f-6c21-098925406480 35227329-aaa9-c06f-6c21-098925406480}"
2025-01-26T02:44:13.991Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:12.983920661 +0000 UTC m=+49.498953036 (1 nodes), 539.401153ms from start of poll, 539.580674ms from cluster start. Timeout at 2025-01-26 02:44:13.988923021 +0000 UTC m=+50.503955396, 1.00500236s after stability
2025-01-26T02:44:13.994Z [INFO]  server-04d0cbdf-3acf-a07c-a3c5-089bc61013bc: starting snapshot up to: index=12
2025-01-26T02:44:13.994Z [INFO]  creating new snapshot: path=/tmp/raft327604933/snapshots/2-12-1737859453994.tmp
2025-01-26T02:44:13.994Z [INFO]  server-04d0cbdf-3acf-a07c-a3c5-089bc61013bc: snapshot complete up to: index=12
2025-01-26T02:44:13.996Z [INFO]  server-47a2ec44-7dc4-a4b8-d398-c29912422143: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47a2ec44-7dc4-a4b8-d398-c29912422143 Address:47a2ec44-7dc4-a4b8-d398-c29912422143} {Suffrage:Voter ID:server-d5e81ea7-717a-8579-2e5a-9c47465df911 Address:d5e81ea7-717a-8579-2e5a-9c47465df911} {Suffrage:Voter ID:server-164346fe-4531-11c1-0d0f-f427eda9e952 Address:164346fe-4531-11c1-0d0f-f427eda9e952}]"
2025-01-26T02:44:13.996Z [INFO]  server-d5e81ea7-717a-8579-2e5a-9c47465df911: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47a2ec44-7dc4-a4b8-d398-c29912422143 Address:47a2ec44-7dc4-a4b8-d398-c29912422143} {Suffrage:Voter ID:server-d5e81ea7-717a-8579-2e5a-9c47465df911 Address:d5e81ea7-717a-8579-2e5a-9c47465df911} {Suffrage:Voter ID:server-164346fe-4531-11c1-0d0f-f427eda9e952 Address:164346fe-4531-11c1-0d0f-f427eda9e952}]"
2025-01-26T02:44:13.996Z [INFO]  server-164346fe-4531-11c1-0d0f-f427eda9e952: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-47a2ec44-7dc4-a4b8-d398-c29912422143 Address:47a2ec44-7dc4-a4b8-d398-c29912422143} {Suffrage:Voter ID:server-d5e81ea7-717a-8579-2e5a-9c47465df911 Address:d5e81ea7-717a-8579-2e5a-9c47465df911} {Suffrage:Voter ID:server-164346fe-4531-11c1-0d0f-f427eda9e952 Address:164346fe-4531-11c1-0d0f-f427eda9e952}]"
2025-01-26T02:44:13.996Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:13.996Z [INFO]  server-47a2ec44-7dc4-a4b8-d398-c29912422143: entering follower state: follower="Node at 47a2ec44-7dc4-a4b8-d398-c29912422143 [Follower]" leader-address= leader-id=
2025-01-26T02:44:13.996Z [INFO]  server-d5e81ea7-717a-8579-2e5a-9c47465df911: entering follower state: follower="Node at d5e81ea7-717a-8579-2e5a-9c47465df911 [Follower]" leader-address= leader-id=
2025-01-26T02:44:13.996Z [INFO]  server-164346fe-4531-11c1-0d0f-f427eda9e952: entering follower state: follower="Node at 164346fe-4531-11c1-0d0f-f427eda9e952 [Follower]" leader-address= leader-id=
2025-01-26T02:44:14.587Z [WARN]  server-47a2ec44-7dc4-a4b8-d398-c29912422143: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:14.587Z [INFO]  server-47a2ec44-7dc4-a4b8-d398-c29912422143: entering candidate state: node="Node at 47a2ec44-7dc4-a4b8-d398-c29912422143 [Candidate]" term=2
2025-01-26T02:44:14.587Z [INFO]  server-47a2ec44-7dc4-a4b8-d398-c29912422143: election won: term=2 tally=2
2025-01-26T02:44:14.587Z [INFO]  server-47a2ec44-7dc4-a4b8-d398-c29912422143: entering leader state: leader="Node at 47a2ec44-7dc4-a4b8-d398-c29912422143 [Leader]"
2025-01-26T02:44:14.587Z [INFO]  server-47a2ec44-7dc4-a4b8-d398-c29912422143: added peer, starting replication: peer=server-d5e81ea7-717a-8579-2e5a-9c47465df911
2025-01-26T02:44:14.587Z [INFO]  server-47a2ec44-7dc4-a4b8-d398-c29912422143: added peer, starting replication: peer=server-164346fe-4531-11c1-0d0f-f427eda9e952
2025-01-26T02:44:14.587Z [INFO]  server-47a2ec44-7dc4-a4b8-d398-c29912422143: pipelining replication: peer="{Voter server-164346fe-4531-11c1-0d0f-f427eda9e952 164346fe-4531-11c1-0d0f-f427eda9e952}"
2025-01-26T02:44:14.588Z [WARN]  server-164346fe-4531-11c1-0d0f-f427eda9e952: rejecting vote request since our last term is greater: candidate=47a2ec44-7dc4-a4b8-d398-c29912422143 last-term=2 last-candidate-term=1
2025-01-26T02:44:14.588Z [INFO]  server-47a2ec44-7dc4-a4b8-d398-c29912422143: pipelining replication: peer="{Voter server-d5e81ea7-717a-8579-2e5a-9c47465df911 d5e81ea7-717a-8579-2e5a-9c47465df911}"
2025-01-26T02:44:15.595Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:14.588047955 +0000 UTC m=+51.103080330 (1 nodes), 591.828471ms from start of poll, 592.051833ms from cluster start. Timeout at 2025-01-26 02:44:15.593049315 +0000 UTC m=+52.108081690, 1.00500136s after stability
2025-01-26T02:44:15.595Z [INFO]  creating new snapshot: path=/tmp/raft1735915340/snapshots/2-15-1737859455595.tmp
2025-01-26T02:44:15.596Z [INFO]  server-47a2ec44-7dc4-a4b8-d398-c29912422143: copied to local snapshot: bytes=71
2025-01-26T02:44:15.596Z [INFO]  server-47a2ec44-7dc4-a4b8-d398-c29912422143: snapshot restore progress: id=2-15-1737859455595 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:15.596Z [INFO]  server-47a2ec44-7dc4-a4b8-d398-c29912422143: restored user snapshot: index=15
2025-01-26T02:44:15.596Z [ERROR] server-47a2ec44-7dc4-a4b8-d398-c29912422143: failed to get log: index=3 error="log not found"
2025-01-26T02:44:15.596Z [INFO]  server-47a2ec44-7dc4-a4b8-d398-c29912422143: aborting pipeline replication: peer="{Voter server-164346fe-4531-11c1-0d0f-f427eda9e952 164346fe-4531-11c1-0d0f-f427eda9e952}"
2025-01-26T02:44:15.596Z [ERROR] server-47a2ec44-7dc4-a4b8-d398-c29912422143: failed to get log: index=3 error="log not found"
2025-01-26T02:44:15.596Z [INFO]  server-47a2ec44-7dc4-a4b8-d398-c29912422143: aborting pipeline replication: peer="{Voter server-d5e81ea7-717a-8579-2e5a-9c47465df911 d5e81ea7-717a-8579-2e5a-9c47465df911}"
2025-01-26T02:44:15.607Z [ERROR] server-47a2ec44-7dc4-a4b8-d398-c29912422143: failed to get log: index=3 error="log not found"
2025-01-26T02:44:15.607Z [INFO]  creating new snapshot: path=/tmp/raft799844425/snapshots/2-15-1737859455607.tmp
2025-01-26T02:44:15.607Z [INFO]  server-d5e81ea7-717a-8579-2e5a-9c47465df911: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:15.607Z [INFO]  server-d5e81ea7-717a-8579-2e5a-9c47465df911: copied to local snapshot: bytes=71
2025-01-26T02:44:15.608Z [INFO]  server-d5e81ea7-717a-8579-2e5a-9c47465df911: snapshot restore progress: id=2-15-1737859455607 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:15.608Z [INFO]  server-d5e81ea7-717a-8579-2e5a-9c47465df911: Installed remote snapshot
2025-01-26T02:44:15.608Z [INFO]  server-47a2ec44-7dc4-a4b8-d398-c29912422143: pipelining replication: peer="{Voter server-d5e81ea7-717a-8579-2e5a-9c47465df911 d5e81ea7-717a-8579-2e5a-9c47465df911}"
2025-01-26T02:44:15.608Z [ERROR] server-47a2ec44-7dc4-a4b8-d398-c29912422143: failed to get log: index=3 error="log not found"
2025-01-26T02:44:15.608Z [INFO]  creating new snapshot: path=/tmp/raft129682806/snapshots/2-15-1737859455608.tmp
2025-01-26T02:44:15.608Z [INFO]  server-164346fe-4531-11c1-0d0f-f427eda9e952: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:15.608Z [INFO]  server-164346fe-4531-11c1-0d0f-f427eda9e952: copied to local snapshot: bytes=71
2025-01-26T02:44:15.608Z [INFO]  server-164346fe-4531-11c1-0d0f-f427eda9e952: snapshot restore progress: id=2-15-1737859455608 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:15.608Z [INFO]  server-164346fe-4531-11c1-0d0f-f427eda9e952: Installed remote snapshot
2025-01-26T02:44:15.608Z [INFO]  server-47a2ec44-7dc4-a4b8-d398-c29912422143: pipelining replication: peer="{Voter server-164346fe-4531-11c1-0d0f-f427eda9e952 164346fe-4531-11c1-0d0f-f427eda9e952}"
2025-01-26T02:44:15.608Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:44:16.615Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:44:15.608736028 +0000 UTC m=+52.123768403 (2 nodes), 600ns from start of poll, 1.612739906s from cluster start. Timeout at 2025-01-26 02:44:16.613760189 +0000 UTC m=+53.128792564, 1.005024161s after stability
2025-01-26T02:44:16.635Z [INFO]  server-04d0cbdf-3acf-a07c-a3c5-089bc61013bc: aborting pipeline replication: peer="{Voter server-f9e480bf-a87b-7c65-a9de-978ef1457aa2 f9e480bf-a87b-7c65-a9de-978ef1457aa2}"
2025-01-26T02:44:16.635Z [INFO]  server-04d0cbdf-3acf-a07c-a3c5-089bc61013bc: aborting pipeline replication: peer="{Voter server-35227329-aaa9-c06f-6c21-098925406480 35227329-aaa9-c06f-6c21-098925406480}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true
2025-01-26T02:44:16.636Z [INFO]  server-10091fa8-5a69-8435-82fb-1141370c2b39: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-10091fa8-5a69-8435-82fb-1141370c2b39 Address:10091fa8-5a69-8435-82fb-1141370c2b39} {Suffrage:Voter ID:server-668bd332-ae55-91b3-9c17-92500c2cb6a3 Address:668bd332-ae55-91b3-9c17-92500c2cb6a3} {Suffrage:Voter ID:server-841c31ec-44c0-7ac8-30c8-55d793e6b22a Address:841c31ec-44c0-7ac8-30c8-55d793e6b22a}]"
2025-01-26T02:44:16.636Z [INFO]  server-668bd332-ae55-91b3-9c17-92500c2cb6a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-10091fa8-5a69-8435-82fb-1141370c2b39 Address:10091fa8-5a69-8435-82fb-1141370c2b39} {Suffrage:Voter ID:server-668bd332-ae55-91b3-9c17-92500c2cb6a3 Address:668bd332-ae55-91b3-9c17-92500c2cb6a3} {Suffrage:Voter ID:server-841c31ec-44c0-7ac8-30c8-55d793e6b22a Address:841c31ec-44c0-7ac8-30c8-55d793e6b22a}]"
2025-01-26T02:44:16.636Z [INFO]  server-841c31ec-44c0-7ac8-30c8-55d793e6b22a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-10091fa8-5a69-8435-82fb-1141370c2b39 Address:10091fa8-5a69-8435-82fb-1141370c2b39} {Suffrage:Voter ID:server-668bd332-ae55-91b3-9c17-92500c2cb6a3 Address:668bd332-ae55-91b3-9c17-92500c2cb6a3} {Suffrage:Voter ID:server-841c31ec-44c0-7ac8-30c8-55d793e6b22a Address:841c31ec-44c0-7ac8-30c8-55d793e6b22a}]"
2025-01-26T02:44:16.636Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:16.636Z [INFO]  server-10091fa8-5a69-8435-82fb-1141370c2b39: entering follower state: follower="Node at 10091fa8-5a69-8435-82fb-1141370c2b39 [Follower]" leader-address= leader-id=
2025-01-26T02:44:16.636Z [INFO]  server-668bd332-ae55-91b3-9c17-92500c2cb6a3: entering follower state: follower="Node at 668bd332-ae55-91b3-9c17-92500c2cb6a3 [Follower]" leader-address= leader-id=
2025-01-26T02:44:16.636Z [INFO]  server-841c31ec-44c0-7ac8-30c8-55d793e6b22a: entering follower state: follower="Node at 841c31ec-44c0-7ac8-30c8-55d793e6b22a [Follower]" leader-address= leader-id=
2025-01-26T02:44:17.223Z [WARN]  server-841c31ec-44c0-7ac8-30c8-55d793e6b22a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:17.223Z [INFO]  server-841c31ec-44c0-7ac8-30c8-55d793e6b22a: entering candidate state: node="Node at 841c31ec-44c0-7ac8-30c8-55d793e6b22a [Candidate]" term=2
2025-01-26T02:44:17.223Z [INFO]  server-841c31ec-44c0-7ac8-30c8-55d793e6b22a: election won: term=2 tally=2
2025-01-26T02:44:17.223Z [INFO]  server-841c31ec-44c0-7ac8-30c8-55d793e6b22a: entering leader state: leader="Node at 841c31ec-44c0-7ac8-30c8-55d793e6b22a [Leader]"
2025-01-26T02:44:17.223Z [INFO]  server-841c31ec-44c0-7ac8-30c8-55d793e6b22a: added peer, starting replication: peer=server-10091fa8-5a69-8435-82fb-1141370c2b39
2025-01-26T02:44:17.223Z [INFO]  server-841c31ec-44c0-7ac8-30c8-55d793e6b22a: added peer, starting replication: peer=server-668bd332-ae55-91b3-9c17-92500c2cb6a3
2025-01-26T02:44:17.223Z [INFO]  server-841c31ec-44c0-7ac8-30c8-55d793e6b22a: pipelining replication: peer="{Voter server-668bd332-ae55-91b3-9c17-92500c2cb6a3 668bd332-ae55-91b3-9c17-92500c2cb6a3}"
2025-01-26T02:44:17.223Z [INFO]  server-841c31ec-44c0-7ac8-30c8-55d793e6b22a: pipelining replication: peer="{Voter server-10091fa8-5a69-8435-82fb-1141370c2b39 10091fa8-5a69-8435-82fb-1141370c2b39}"
2025-01-26T02:44:18.231Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:17.223844829 +0000 UTC m=+53.738877204 (1 nodes), 587.682246ms from start of poll, 587.868047ms from cluster start. Timeout at 2025-01-26 02:44:18.228849469 +0000 UTC m=+54.743881844, 1.00500464s after stability
2025-01-26T02:44:18.231Z [INFO]  server-841c31ec-44c0-7ac8-30c8-55d793e6b22a: starting snapshot up to: index=12
2025-01-26T02:44:18.231Z [INFO]  creating new snapshot: path=/tmp/raft1535230164/snapshots/2-12-1737859458231.tmp
2025-01-26T02:44:18.232Z [INFO]  server-841c31ec-44c0-7ac8-30c8-55d793e6b22a: snapshot complete up to: index=12
2025-01-26T02:44:18.232Z [INFO]  server-226b9f11-cc9f-3225-931a-3cd864fbbaaf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-226b9f11-cc9f-3225-931a-3cd864fbbaaf Address:226b9f11-cc9f-3225-931a-3cd864fbbaaf} {Suffrage:Voter ID:server-677d0a38-ff25-2b0f-445b-0b6e84b83683 Address:677d0a38-ff25-2b0f-445b-0b6e84b83683} {Suffrage:Voter ID:server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9 Address:c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9}]"
2025-01-26T02:44:18.232Z [INFO]  server-677d0a38-ff25-2b0f-445b-0b6e84b83683: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-226b9f11-cc9f-3225-931a-3cd864fbbaaf Address:226b9f11-cc9f-3225-931a-3cd864fbbaaf} {Suffrage:Voter ID:server-677d0a38-ff25-2b0f-445b-0b6e84b83683 Address:677d0a38-ff25-2b0f-445b-0b6e84b83683} {Suffrage:Voter ID:server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9 Address:c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9}]"
2025-01-26T02:44:18.232Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-226b9f11-cc9f-3225-931a-3cd864fbbaaf Address:226b9f11-cc9f-3225-931a-3cd864fbbaaf} {Suffrage:Voter ID:server-677d0a38-ff25-2b0f-445b-0b6e84b83683 Address:677d0a38-ff25-2b0f-445b-0b6e84b83683} {Suffrage:Voter ID:server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9 Address:c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9}]"
2025-01-26T02:44:18.233Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:18.233Z [INFO]  server-226b9f11-cc9f-3225-931a-3cd864fbbaaf: entering follower state: follower="Node at 226b9f11-cc9f-3225-931a-3cd864fbbaaf [Follower]" leader-address= leader-id=
2025-01-26T02:44:18.233Z [INFO]  server-677d0a38-ff25-2b0f-445b-0b6e84b83683: entering follower state: follower="Node at 677d0a38-ff25-2b0f-445b-0b6e84b83683 [Follower]" leader-address= leader-id=
2025-01-26T02:44:18.233Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: entering follower state: follower="Node at c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9 [Follower]" leader-address= leader-id=
2025-01-26T02:44:18.783Z [WARN]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:18.783Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: entering candidate state: node="Node at c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9 [Candidate]" term=2
2025-01-26T02:44:18.783Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: election won: term=2 tally=2
2025-01-26T02:44:18.783Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: entering leader state: leader="Node at c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9 [Leader]"
2025-01-26T02:44:18.783Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: added peer, starting replication: peer=server-226b9f11-cc9f-3225-931a-3cd864fbbaaf
2025-01-26T02:44:18.783Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: added peer, starting replication: peer=server-677d0a38-ff25-2b0f-445b-0b6e84b83683
2025-01-26T02:44:18.783Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: pipelining replication: peer="{Voter server-677d0a38-ff25-2b0f-445b-0b6e84b83683 677d0a38-ff25-2b0f-445b-0b6e84b83683}"
2025-01-26T02:44:18.783Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: pipelining replication: peer="{Voter server-226b9f11-cc9f-3225-931a-3cd864fbbaaf 226b9f11-cc9f-3225-931a-3cd864fbbaaf}"
2025-01-26T02:44:19.789Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:18.783965215 +0000 UTC m=+55.298997550 (1 nodes), 550.948583ms from start of poll, 551.158184ms from cluster start. Timeout at 2025-01-26 02:44:19.788967415 +0000 UTC m=+56.303999790, 1.00500224s after stability
2025-01-26T02:44:19.789Z [INFO]  creating new snapshot: path=/tmp/raft2470144406/snapshots/2-15-1737859459789.tmp
2025-01-26T02:44:19.790Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: copied to local snapshot: bytes=71
2025-01-26T02:44:19.792Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: snapshot restore progress: id=2-15-1737859459789 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:19.792Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: removing all old logs from log store
2025-01-26T02:44:19.792Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: compacting logs: from=1 to=2
2025-01-26T02:44:19.792Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: restored user snapshot: index=15
2025-01-26T02:44:19.792Z [ERROR] server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: failed to get log: index=2 error="log not found"
2025-01-26T02:44:19.792Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: aborting pipeline replication: peer="{Voter server-677d0a38-ff25-2b0f-445b-0b6e84b83683 677d0a38-ff25-2b0f-445b-0b6e84b83683}"
2025-01-26T02:44:19.792Z [ERROR] server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: failed to get log: index=2 error="log not found"
2025-01-26T02:44:19.792Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: aborting pipeline replication: peer="{Voter server-226b9f11-cc9f-3225-931a-3cd864fbbaaf 226b9f11-cc9f-3225-931a-3cd864fbbaaf}"
2025-01-26T02:44:19.799Z [ERROR] server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: failed to get log: index=2 error="log not found"
2025-01-26T02:44:19.800Z [ERROR] server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: failed to get log: index=2 error="log not found"
2025-01-26T02:44:19.800Z [INFO]  creating new snapshot: path=/tmp/raft1795441760/snapshots/2-15-1737859459800.tmp
2025-01-26T02:44:19.800Z [INFO]  server-677d0a38-ff25-2b0f-445b-0b6e84b83683: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:19.800Z [INFO]  server-677d0a38-ff25-2b0f-445b-0b6e84b83683: copied to local snapshot: bytes=71
2025-01-26T02:44:19.800Z [INFO]  server-677d0a38-ff25-2b0f-445b-0b6e84b83683: snapshot restore progress: id=2-15-1737859459800 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:19.800Z [INFO]  server-677d0a38-ff25-2b0f-445b-0b6e84b83683: removing all old logs from log store
2025-01-26T02:44:19.800Z [INFO]  server-677d0a38-ff25-2b0f-445b-0b6e84b83683: compacting logs: from=1 to=2
2025-01-26T02:44:19.800Z [INFO]  server-677d0a38-ff25-2b0f-445b-0b6e84b83683: Installed remote snapshot
2025-01-26T02:44:19.800Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: pipelining replication: peer="{Voter server-677d0a38-ff25-2b0f-445b-0b6e84b83683 677d0a38-ff25-2b0f-445b-0b6e84b83683}"
2025-01-26T02:44:19.800Z [INFO]  creating new snapshot: path=/tmp/raft2905058253/snapshots/2-15-1737859459800.tmp
2025-01-26T02:44:19.800Z [INFO]  server-226b9f11-cc9f-3225-931a-3cd864fbbaaf: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:19.801Z [INFO]  server-226b9f11-cc9f-3225-931a-3cd864fbbaaf: copied to local snapshot: bytes=71
2025-01-26T02:44:19.801Z [INFO]  server-226b9f11-cc9f-3225-931a-3cd864fbbaaf: snapshot restore progress: id=2-15-1737859459800 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:19.801Z [INFO]  server-226b9f11-cc9f-3225-931a-3cd864fbbaaf: removing all old logs from log store
2025-01-26T02:44:19.801Z [INFO]  server-226b9f11-cc9f-3225-931a-3cd864fbbaaf: compacting logs: from=1 to=2
2025-01-26T02:44:19.801Z [INFO]  server-226b9f11-cc9f-3225-931a-3cd864fbbaaf: Installed remote snapshot
2025-01-26T02:44:19.801Z [INFO]  server-c88cbab0-ee0d-d83c-b014-fdd3f0aeb7b9: pipelining replication: peer="{Voter server-226b9f11-cc9f-3225-931a-3cd864fbbaaf 226b9f11-cc9f-3225-931a-3cd864fbbaaf}"
2025-01-26T02:44:19.801Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:44:20.811Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:44:19.801268508 +0000 UTC m=+56.316300883 (2 nodes), 480ns from start of poll, 1.568461517s from cluster start. Timeout at 2025-01-26 02:44:20.806306308 +0000 UTC m=+57.321338683, 1.0050378s after stability
2025-01-26T02:44:20.827Z [INFO]  server-841c31ec-44c0-7ac8-30c8-55d793e6b22a: aborting pipeline replication: peer="{Voter server-668bd332-ae55-91b3-9c17-92500c2cb6a3 668bd332-ae55-91b3-9c17-92500c2cb6a3}"
2025-01-26T02:44:20.827Z [INFO]  server-841c31ec-44c0-7ac8-30c8-55d793e6b22a: aborting pipeline replication: peer="{Voter server-10091fa8-5a69-8435-82fb-1141370c2b39 10091fa8-5a69-8435-82fb-1141370c2b39}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false
2025-01-26T02:44:20.828Z [INFO]  server-1a35993b-9628-c4b9-4aeb-41547ae45724: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a35993b-9628-c4b9-4aeb-41547ae45724 Address:1a35993b-9628-c4b9-4aeb-41547ae45724} {Suffrage:Voter ID:server-444c113c-139a-9e9d-abb2-5d3b859f508a Address:444c113c-139a-9e9d-abb2-5d3b859f508a} {Suffrage:Voter ID:server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a Address:2c1cc75b-ee9a-189f-093a-e35b47dadf1a}]"
2025-01-26T02:44:20.828Z [INFO]  server-444c113c-139a-9e9d-abb2-5d3b859f508a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a35993b-9628-c4b9-4aeb-41547ae45724 Address:1a35993b-9628-c4b9-4aeb-41547ae45724} {Suffrage:Voter ID:server-444c113c-139a-9e9d-abb2-5d3b859f508a Address:444c113c-139a-9e9d-abb2-5d3b859f508a} {Suffrage:Voter ID:server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a Address:2c1cc75b-ee9a-189f-093a-e35b47dadf1a}]"
2025-01-26T02:44:20.828Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a35993b-9628-c4b9-4aeb-41547ae45724 Address:1a35993b-9628-c4b9-4aeb-41547ae45724} {Suffrage:Voter ID:server-444c113c-139a-9e9d-abb2-5d3b859f508a Address:444c113c-139a-9e9d-abb2-5d3b859f508a} {Suffrage:Voter ID:server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a Address:2c1cc75b-ee9a-189f-093a-e35b47dadf1a}]"
2025-01-26T02:44:20.828Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:20.828Z [INFO]  server-1a35993b-9628-c4b9-4aeb-41547ae45724: entering follower state: follower="Node at 1a35993b-9628-c4b9-4aeb-41547ae45724 [Follower]" leader-address= leader-id=
2025-01-26T02:44:20.828Z [INFO]  server-444c113c-139a-9e9d-abb2-5d3b859f508a: entering follower state: follower="Node at 444c113c-139a-9e9d-abb2-5d3b859f508a [Follower]" leader-address= leader-id=
2025-01-26T02:44:20.828Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: entering follower state: follower="Node at 2c1cc75b-ee9a-189f-093a-e35b47dadf1a [Follower]" leader-address= leader-id=
2025-01-26T02:44:21.476Z [WARN]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:21.476Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: entering candidate state: node="Node at 2c1cc75b-ee9a-189f-093a-e35b47dadf1a [Candidate]" term=2
2025-01-26T02:44:21.476Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: election won: term=2 tally=2
2025-01-26T02:44:21.476Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: entering leader state: leader="Node at 2c1cc75b-ee9a-189f-093a-e35b47dadf1a [Leader]"
2025-01-26T02:44:21.476Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: added peer, starting replication: peer=server-1a35993b-9628-c4b9-4aeb-41547ae45724
2025-01-26T02:44:21.476Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: added peer, starting replication: peer=server-444c113c-139a-9e9d-abb2-5d3b859f508a
2025-01-26T02:44:21.476Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: pipelining replication: peer="{Voter server-444c113c-139a-9e9d-abb2-5d3b859f508a 444c113c-139a-9e9d-abb2-5d3b859f508a}"
2025-01-26T02:44:21.476Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: pipelining replication: peer="{Voter server-1a35993b-9628-c4b9-4aeb-41547ae45724 1a35993b-9628-c4b9-4aeb-41547ae45724}"
2025-01-26T02:44:22.483Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:21.477062077 +0000 UTC m=+57.992094452 (1 nodes), 648.836057ms from start of poll, 649.013138ms from cluster start. Timeout at 2025-01-26 02:44:22.482091197 +0000 UTC m=+58.997123572, 1.00502912s after stability
2025-01-26T02:44:22.483Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: starting snapshot up to: index=12
2025-01-26T02:44:22.484Z [INFO]  creating new snapshot: path=/tmp/raft3951956910/snapshots/2-12-1737859462483.tmp
2025-01-26T02:44:22.484Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: snapshot complete up to: index=12
2025-01-26T02:44:22.484Z [INFO]  creating new snapshot: path=/tmp/raft3951956910/snapshots/2-113-1737859462484.tmp
2025-01-26T02:44:22.485Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: copied to local snapshot: bytes=71
2025-01-26T02:44:22.485Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: snapshot restore progress: id=2-113-1737859462484 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:22.485Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: restored user snapshot: index=113
2025-01-26T02:44:22.485Z [ERROR] server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: failed to get log: index=23 error="log not found"
2025-01-26T02:44:22.485Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: aborting pipeline replication: peer="{Voter server-444c113c-139a-9e9d-abb2-5d3b859f508a 444c113c-139a-9e9d-abb2-5d3b859f508a}"
2025-01-26T02:44:22.485Z [ERROR] server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: failed to get log: index=23 error="log not found"
2025-01-26T02:44:22.485Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: aborting pipeline replication: peer="{Voter server-1a35993b-9628-c4b9-4aeb-41547ae45724 1a35993b-9628-c4b9-4aeb-41547ae45724}"
2025-01-26T02:44:22.495Z [ERROR] server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: failed to get log: index=23 error="log not found"
2025-01-26T02:44:22.495Z [INFO]  creating new snapshot: path=/tmp/raft4025829853/snapshots/2-113-1737859462495.tmp
2025-01-26T02:44:22.495Z [INFO]  server-444c113c-139a-9e9d-abb2-5d3b859f508a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:22.495Z [INFO]  server-444c113c-139a-9e9d-abb2-5d3b859f508a: copied to local snapshot: bytes=71
2025-01-26T02:44:22.496Z [INFO]  server-444c113c-139a-9e9d-abb2-5d3b859f508a: snapshot restore progress: id=2-113-1737859462495 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:22.496Z [INFO]  server-444c113c-139a-9e9d-abb2-5d3b859f508a: Installed remote snapshot
2025-01-26T02:44:22.496Z [WARN]  server-444c113c-139a-9e9d-abb2-5d3b859f508a: skipping application of old log: index=22
2025-01-26T02:44:22.496Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: pipelining replication: peer="{Voter server-444c113c-139a-9e9d-abb2-5d3b859f508a 444c113c-139a-9e9d-abb2-5d3b859f508a}"
2025-01-26T02:44:22.496Z [ERROR] server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: failed to get log: index=23 error="log not found"
2025-01-26T02:44:22.496Z [INFO]  creating new snapshot: path=/tmp/raft1186577960/snapshots/2-113-1737859462496.tmp
2025-01-26T02:44:22.496Z [INFO]  server-1a35993b-9628-c4b9-4aeb-41547ae45724: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:22.496Z [INFO]  server-1a35993b-9628-c4b9-4aeb-41547ae45724: copied to local snapshot: bytes=71
2025-01-26T02:44:22.496Z [INFO]  server-1a35993b-9628-c4b9-4aeb-41547ae45724: snapshot restore progress: id=2-113-1737859462496 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:22.496Z [INFO]  server-1a35993b-9628-c4b9-4aeb-41547ae45724: Installed remote snapshot
2025-01-26T02:44:22.496Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: pipelining replication: peer="{Voter server-1a35993b-9628-c4b9-4aeb-41547ae45724 1a35993b-9628-c4b9-4aeb-41547ae45724}"
2025-01-26T02:44:22.496Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:44:23.511Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:44:22.496694464 +0000 UTC m=+59.011726839 (2 nodes), 440ns from start of poll, 1.668645525s from cluster start. Timeout at 2025-01-26 02:44:23.501731144 +0000 UTC m=+60.016763519, 1.00503668s after stability
2025-01-26T02:44:23.527Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: aborting pipeline replication: peer="{Voter server-444c113c-139a-9e9d-abb2-5d3b859f508a 444c113c-139a-9e9d-abb2-5d3b859f508a}"
2025-01-26T02:44:23.527Z [INFO]  server-2c1cc75b-ee9a-189f-093a-e35b47dadf1a: aborting pipeline replication: peer="{Voter server-1a35993b-9628-c4b9-4aeb-41547ae45724 1a35993b-9628-c4b9-4aeb-41547ae45724}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false
2025-01-26T02:44:23.528Z [INFO]  server-afbcf69a-bcdb-5e23-a0db-24c2a24f6941: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afbcf69a-bcdb-5e23-a0db-24c2a24f6941 Address:afbcf69a-bcdb-5e23-a0db-24c2a24f6941} {Suffrage:Voter ID:server-d29c8673-e778-4bbe-34b4-9e729d4aaec6 Address:d29c8673-e778-4bbe-34b4-9e729d4aaec6} {Suffrage:Voter ID:server-c754e811-b118-89c6-57d1-c1886dd6747a Address:c754e811-b118-89c6-57d1-c1886dd6747a}]"
2025-01-26T02:44:23.528Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afbcf69a-bcdb-5e23-a0db-24c2a24f6941 Address:afbcf69a-bcdb-5e23-a0db-24c2a24f6941} {Suffrage:Voter ID:server-d29c8673-e778-4bbe-34b4-9e729d4aaec6 Address:d29c8673-e778-4bbe-34b4-9e729d4aaec6} {Suffrage:Voter ID:server-c754e811-b118-89c6-57d1-c1886dd6747a Address:c754e811-b118-89c6-57d1-c1886dd6747a}]"
2025-01-26T02:44:23.528Z [INFO]  server-c754e811-b118-89c6-57d1-c1886dd6747a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afbcf69a-bcdb-5e23-a0db-24c2a24f6941 Address:afbcf69a-bcdb-5e23-a0db-24c2a24f6941} {Suffrage:Voter ID:server-d29c8673-e778-4bbe-34b4-9e729d4aaec6 Address:d29c8673-e778-4bbe-34b4-9e729d4aaec6} {Suffrage:Voter ID:server-c754e811-b118-89c6-57d1-c1886dd6747a Address:c754e811-b118-89c6-57d1-c1886dd6747a}]"
2025-01-26T02:44:23.528Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:23.528Z [INFO]  server-afbcf69a-bcdb-5e23-a0db-24c2a24f6941: entering follower state: follower="Node at afbcf69a-bcdb-5e23-a0db-24c2a24f6941 [Follower]" leader-address= leader-id=
2025-01-26T02:44:23.528Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: entering follower state: follower="Node at d29c8673-e778-4bbe-34b4-9e729d4aaec6 [Follower]" leader-address= leader-id=
2025-01-26T02:44:23.528Z [INFO]  server-c754e811-b118-89c6-57d1-c1886dd6747a: entering follower state: follower="Node at c754e811-b118-89c6-57d1-c1886dd6747a [Follower]" leader-address= leader-id=
2025-01-26T02:44:24.043Z [WARN]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:24.043Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: entering candidate state: node="Node at d29c8673-e778-4bbe-34b4-9e729d4aaec6 [Candidate]" term=2
2025-01-26T02:44:24.043Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: election won: term=2 tally=2
2025-01-26T02:44:24.043Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: entering leader state: leader="Node at d29c8673-e778-4bbe-34b4-9e729d4aaec6 [Leader]"
2025-01-26T02:44:24.043Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: added peer, starting replication: peer=server-afbcf69a-bcdb-5e23-a0db-24c2a24f6941
2025-01-26T02:44:24.043Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: added peer, starting replication: peer=server-c754e811-b118-89c6-57d1-c1886dd6747a
2025-01-26T02:44:24.043Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: pipelining replication: peer="{Voter server-c754e811-b118-89c6-57d1-c1886dd6747a c754e811-b118-89c6-57d1-c1886dd6747a}"
2025-01-26T02:44:24.044Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: pipelining replication: peer="{Voter server-afbcf69a-bcdb-5e23-a0db-24c2a24f6941 afbcf69a-bcdb-5e23-a0db-24c2a24f6941}"
2025-01-26T02:44:25.051Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:24.044042013 +0000 UTC m=+60.559074388 (1 nodes), 515.693249ms from start of poll, 515.87665ms from cluster start. Timeout at 2025-01-26 02:44:25.049049093 +0000 UTC m=+61.564081428, 1.00500704s after stability
2025-01-26T02:44:25.051Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: starting snapshot up to: index=12
2025-01-26T02:44:25.051Z [INFO]  creating new snapshot: path=/tmp/raft3482868141/snapshots/2-12-1737859465051.tmp
2025-01-26T02:44:25.052Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: snapshot complete up to: index=12
2025-01-26T02:44:25.052Z [INFO]  creating new snapshot: path=/tmp/raft3482868141/snapshots/2-113-1737859465052.tmp
2025-01-26T02:44:25.060Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: copied to local snapshot: bytes=71
2025-01-26T02:44:25.067Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: snapshot restore progress: id=2-113-1737859465052 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:25.067Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: removing all old logs from log store
2025-01-26T02:44:25.067Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: compacting logs: from=1 to=22
2025-01-26T02:44:25.068Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: restored user snapshot: index=113
2025-01-26T02:44:25.068Z [ERROR] server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: failed to get log: index=22 error="log not found"
2025-01-26T02:44:25.068Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: aborting pipeline replication: peer="{Voter server-afbcf69a-bcdb-5e23-a0db-24c2a24f6941 afbcf69a-bcdb-5e23-a0db-24c2a24f6941}"
2025-01-26T02:44:25.068Z [ERROR] server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: failed to get log: index=22 error="log not found"
2025-01-26T02:44:25.068Z [INFO]  creating new snapshot: path=/tmp/raft967537356/snapshots/2-113-1737859465068.tmp
2025-01-26T02:44:25.068Z [INFO]  server-afbcf69a-bcdb-5e23-a0db-24c2a24f6941: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:25.068Z [INFO]  server-afbcf69a-bcdb-5e23-a0db-24c2a24f6941: copied to local snapshot: bytes=71
2025-01-26T02:44:25.068Z [INFO]  server-afbcf69a-bcdb-5e23-a0db-24c2a24f6941: snapshot restore progress: id=2-113-1737859465068 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:25.068Z [INFO]  server-afbcf69a-bcdb-5e23-a0db-24c2a24f6941: removing all old logs from log store
2025-01-26T02:44:25.068Z [INFO]  server-afbcf69a-bcdb-5e23-a0db-24c2a24f6941: compacting logs: from=1 to=22
2025-01-26T02:44:25.068Z [INFO]  server-afbcf69a-bcdb-5e23-a0db-24c2a24f6941: Installed remote snapshot
2025-01-26T02:44:25.068Z [WARN]  server-afbcf69a-bcdb-5e23-a0db-24c2a24f6941: skipping application of old log: index=22
2025-01-26T02:44:25.068Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: pipelining replication: peer="{Voter server-afbcf69a-bcdb-5e23-a0db-24c2a24f6941 afbcf69a-bcdb-5e23-a0db-24c2a24f6941}"
2025-01-26T02:44:25.069Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:44:25.069Z [ERROR] server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: failed to get log: index=22 error="log not found"
2025-01-26T02:44:25.069Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: aborting pipeline replication: peer="{Voter server-c754e811-b118-89c6-57d1-c1886dd6747a c754e811-b118-89c6-57d1-c1886dd6747a}"
2025-01-26T02:44:25.069Z [ERROR] server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: failed to get log: index=22 error="log not found"
2025-01-26T02:44:25.069Z [INFO]  creating new snapshot: path=/tmp/raft1063326765/snapshots/2-113-1737859465069.tmp
2025-01-26T02:44:25.069Z [INFO]  server-c754e811-b118-89c6-57d1-c1886dd6747a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:25.069Z [INFO]  server-c754e811-b118-89c6-57d1-c1886dd6747a: copied to local snapshot: bytes=71
2025-01-26T02:44:25.069Z [INFO]  server-c754e811-b118-89c6-57d1-c1886dd6747a: snapshot restore progress: id=2-113-1737859465069 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:25.069Z [INFO]  server-c754e811-b118-89c6-57d1-c1886dd6747a: removing all old logs from log store
2025-01-26T02:44:25.069Z [INFO]  server-c754e811-b118-89c6-57d1-c1886dd6747a: compacting logs: from=1 to=22
2025-01-26T02:44:25.069Z [INFO]  server-c754e811-b118-89c6-57d1-c1886dd6747a: Installed remote snapshot
2025-01-26T02:44:25.069Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: pipelining replication: peer="{Voter server-c754e811-b118-89c6-57d1-c1886dd6747a c754e811-b118-89c6-57d1-c1886dd6747a}"
2025-01-26T02:44:26.075Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:44:25.069065633 +0000 UTC m=+61.584097968 (2 nodes), 600ns from start of poll, 1.54090023s from cluster start. Timeout at 2025-01-26 02:44:26.074097233 +0000 UTC m=+62.589129608, 1.00503164s after stability
2025-01-26T02:44:26.107Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: aborting pipeline replication: peer="{Voter server-c754e811-b118-89c6-57d1-c1886dd6747a c754e811-b118-89c6-57d1-c1886dd6747a}"
2025-01-26T02:44:26.107Z [INFO]  server-d29c8673-e778-4bbe-34b4-9e729d4aaec6: aborting pipeline replication: peer="{Voter server-afbcf69a-bcdb-5e23-a0db-24c2a24f6941 afbcf69a-bcdb-5e23-a0db-24c2a24f6941}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true
2025-01-26T02:44:26.108Z [INFO]  server-89fdb94d-fd97-ed77-4cf0-425d6d15643e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89fdb94d-fd97-ed77-4cf0-425d6d15643e Address:89fdb94d-fd97-ed77-4cf0-425d6d15643e} {Suffrage:Voter ID:server-43eb80ab-6735-59b5-a211-01a2d2ec2a0e Address:43eb80ab-6735-59b5-a211-01a2d2ec2a0e} {Suffrage:Voter ID:server-97540042-122b-7561-11f9-e470fa5e4b59 Address:97540042-122b-7561-11f9-e470fa5e4b59}]"
2025-01-26T02:44:26.108Z [INFO]  server-43eb80ab-6735-59b5-a211-01a2d2ec2a0e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89fdb94d-fd97-ed77-4cf0-425d6d15643e Address:89fdb94d-fd97-ed77-4cf0-425d6d15643e} {Suffrage:Voter ID:server-43eb80ab-6735-59b5-a211-01a2d2ec2a0e Address:43eb80ab-6735-59b5-a211-01a2d2ec2a0e} {Suffrage:Voter ID:server-97540042-122b-7561-11f9-e470fa5e4b59 Address:97540042-122b-7561-11f9-e470fa5e4b59}]"
2025-01-26T02:44:26.108Z [INFO]  server-97540042-122b-7561-11f9-e470fa5e4b59: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-89fdb94d-fd97-ed77-4cf0-425d6d15643e Address:89fdb94d-fd97-ed77-4cf0-425d6d15643e} {Suffrage:Voter ID:server-43eb80ab-6735-59b5-a211-01a2d2ec2a0e Address:43eb80ab-6735-59b5-a211-01a2d2ec2a0e} {Suffrage:Voter ID:server-97540042-122b-7561-11f9-e470fa5e4b59 Address:97540042-122b-7561-11f9-e470fa5e4b59}]"
2025-01-26T02:44:26.108Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:26.108Z [INFO]  server-89fdb94d-fd97-ed77-4cf0-425d6d15643e: entering follower state: follower="Node at 89fdb94d-fd97-ed77-4cf0-425d6d15643e [Follower]" leader-address= leader-id=
2025-01-26T02:44:26.108Z [INFO]  server-43eb80ab-6735-59b5-a211-01a2d2ec2a0e: entering follower state: follower="Node at 43eb80ab-6735-59b5-a211-01a2d2ec2a0e [Follower]" leader-address= leader-id=
2025-01-26T02:44:26.108Z [INFO]  server-97540042-122b-7561-11f9-e470fa5e4b59: entering follower state: follower="Node at 97540042-122b-7561-11f9-e470fa5e4b59 [Follower]" leader-address= leader-id=
2025-01-26T02:44:26.715Z [WARN]  server-89fdb94d-fd97-ed77-4cf0-425d6d15643e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:26.715Z [INFO]  server-89fdb94d-fd97-ed77-4cf0-425d6d15643e: entering candidate state: node="Node at 89fdb94d-fd97-ed77-4cf0-425d6d15643e [Candidate]" term=2
2025-01-26T02:44:26.715Z [INFO]  server-89fdb94d-fd97-ed77-4cf0-425d6d15643e: election won: term=2 tally=2
2025-01-26T02:44:26.715Z [INFO]  server-89fdb94d-fd97-ed77-4cf0-425d6d15643e: entering leader state: leader="Node at 89fdb94d-fd97-ed77-4cf0-425d6d15643e [Leader]"
2025-01-26T02:44:26.715Z [INFO]  server-89fdb94d-fd97-ed77-4cf0-425d6d15643e: added peer, starting replication: peer=server-43eb80ab-6735-59b5-a211-01a2d2ec2a0e
2025-01-26T02:44:26.715Z [INFO]  server-89fdb94d-fd97-ed77-4cf0-425d6d15643e: added peer, starting replication: peer=server-97540042-122b-7561-11f9-e470fa5e4b59
2025-01-26T02:44:26.715Z [INFO]  server-89fdb94d-fd97-ed77-4cf0-425d6d15643e: pipelining replication: peer="{Voter server-97540042-122b-7561-11f9-e470fa5e4b59 97540042-122b-7561-11f9-e470fa5e4b59}"
2025-01-26T02:44:26.715Z [INFO]  server-89fdb94d-fd97-ed77-4cf0-425d6d15643e: pipelining replication: peer="{Voter server-43eb80ab-6735-59b5-a211-01a2d2ec2a0e 43eb80ab-6735-59b5-a211-01a2d2ec2a0e}"
2025-01-26T02:44:27.723Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:26.715901306 +0000 UTC m=+63.230933681 (1 nodes), 607.321965ms from start of poll, 607.502607ms from cluster start. Timeout at 2025-01-26 02:44:27.720911026 +0000 UTC m=+64.235943401, 1.00500972s after stability
2025-01-26T02:44:27.724Z [INFO]  server-89fdb94d-fd97-ed77-4cf0-425d6d15643e: starting snapshot up to: index=12
2025-01-26T02:44:27.724Z [INFO]  creating new snapshot: path=/tmp/raft1625041668/snapshots/2-12-1737859467724.tmp
2025-01-26T02:44:27.724Z [INFO]  server-89fdb94d-fd97-ed77-4cf0-425d6d15643e: snapshot complete up to: index=12
2025-01-26T02:44:27.725Z [INFO]  server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6503adc3-18c2-33e4-aa46-e9202e3a22c0 Address:6503adc3-18c2-33e4-aa46-e9202e3a22c0} {Suffrage:Voter ID:server-a2d1c30d-dd14-99c5-0e0f-7e675036d0e7 Address:a2d1c30d-dd14-99c5-0e0f-7e675036d0e7} {Suffrage:Voter ID:server-907ec5a8-597f-0fdf-fc55-2ec897e7b3b1 Address:907ec5a8-597f-0fdf-fc55-2ec897e7b3b1}]"
2025-01-26T02:44:27.725Z [INFO]  server-a2d1c30d-dd14-99c5-0e0f-7e675036d0e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6503adc3-18c2-33e4-aa46-e9202e3a22c0 Address:6503adc3-18c2-33e4-aa46-e9202e3a22c0} {Suffrage:Voter ID:server-a2d1c30d-dd14-99c5-0e0f-7e675036d0e7 Address:a2d1c30d-dd14-99c5-0e0f-7e675036d0e7} {Suffrage:Voter ID:server-907ec5a8-597f-0fdf-fc55-2ec897e7b3b1 Address:907ec5a8-597f-0fdf-fc55-2ec897e7b3b1}]"
2025-01-26T02:44:27.725Z [INFO]  server-907ec5a8-597f-0fdf-fc55-2ec897e7b3b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6503adc3-18c2-33e4-aa46-e9202e3a22c0 Address:6503adc3-18c2-33e4-aa46-e9202e3a22c0} {Suffrage:Voter ID:server-a2d1c30d-dd14-99c5-0e0f-7e675036d0e7 Address:a2d1c30d-dd14-99c5-0e0f-7e675036d0e7} {Suffrage:Voter ID:server-907ec5a8-597f-0fdf-fc55-2ec897e7b3b1 Address:907ec5a8-597f-0fdf-fc55-2ec897e7b3b1}]"
2025-01-26T02:44:27.725Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:27.725Z [INFO]  server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: entering follower state: follower="Node at 6503adc3-18c2-33e4-aa46-e9202e3a22c0 [Follower]" leader-address= leader-id=
2025-01-26T02:44:27.725Z [INFO]  server-a2d1c30d-dd14-99c5-0e0f-7e675036d0e7: entering follower state: follower="Node at a2d1c30d-dd14-99c5-0e0f-7e675036d0e7 [Follower]" leader-address= leader-id=
2025-01-26T02:44:27.725Z [INFO]  server-907ec5a8-597f-0fdf-fc55-2ec897e7b3b1: entering follower state: follower="Node at 907ec5a8-597f-0fdf-fc55-2ec897e7b3b1 [Follower]" leader-address= leader-id=
2025-01-26T02:44:28.427Z [WARN]  server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:28.427Z [INFO]  server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: entering candidate state: node="Node at 6503adc3-18c2-33e4-aa46-e9202e3a22c0 [Candidate]" term=2
2025-01-26T02:44:28.427Z [INFO]  server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: election won: term=2 tally=2
2025-01-26T02:44:28.427Z [INFO]  server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: entering leader state: leader="Node at 6503adc3-18c2-33e4-aa46-e9202e3a22c0 [Leader]"
2025-01-26T02:44:28.427Z [INFO]  server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: added peer, starting replication: peer=server-a2d1c30d-dd14-99c5-0e0f-7e675036d0e7
2025-01-26T02:44:28.427Z [INFO]  server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: added peer, starting replication: peer=server-907ec5a8-597f-0fdf-fc55-2ec897e7b3b1
2025-01-26T02:44:28.427Z [INFO]  server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: pipelining replication: peer="{Voter server-907ec5a8-597f-0fdf-fc55-2ec897e7b3b1 907ec5a8-597f-0fdf-fc55-2ec897e7b3b1}"
2025-01-26T02:44:28.427Z [INFO]  server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: pipelining replication: peer="{Voter server-a2d1c30d-dd14-99c5-0e0f-7e675036d0e7 a2d1c30d-dd14-99c5-0e0f-7e675036d0e7}"
2025-01-26T02:44:29.435Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:28.427942254 +0000 UTC m=+64.942974629 (1 nodes), 702.641583ms from start of poll, 702.821464ms from cluster start. Timeout at 2025-01-26 02:44:29.432943414 +0000 UTC m=+65.947975789, 1.00500116s after stability
2025-01-26T02:44:29.435Z [INFO]  creating new snapshot: path=/tmp/raft95820943/snapshots/2-113-1737859469435.tmp
2025-01-26T02:44:29.436Z [INFO]  server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: copied to local snapshot: bytes=71
2025-01-26T02:44:29.436Z [INFO]  server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: snapshot restore progress: id=2-113-1737859469435 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:29.436Z [INFO]  server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: restored user snapshot: index=113
2025-01-26T02:44:29.436Z [ERROR] server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: failed to get log: index=3 error="log not found"
2025-01-26T02:44:29.436Z [INFO]  server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: aborting pipeline replication: peer="{Voter server-a2d1c30d-dd14-99c5-0e0f-7e675036d0e7 a2d1c30d-dd14-99c5-0e0f-7e675036d0e7}"
2025-01-26T02:44:29.436Z [ERROR] server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: failed to get log: index=3 error="log not found"
2025-01-26T02:44:29.436Z [INFO]  server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: aborting pipeline replication: peer="{Voter server-907ec5a8-597f-0fdf-fc55-2ec897e7b3b1 907ec5a8-597f-0fdf-fc55-2ec897e7b3b1}"
2025-01-26T02:44:29.443Z [ERROR] server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: failed to get log: index=3 error="log not found"
2025-01-26T02:44:29.443Z [ERROR] server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: failed to get log: index=3 error="log not found"
2025-01-26T02:44:29.444Z [INFO]  creating new snapshot: path=/tmp/raft3019735439/snapshots/2-113-1737859469444.tmp
2025-01-26T02:44:29.444Z [INFO]  server-907ec5a8-597f-0fdf-fc55-2ec897e7b3b1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:29.444Z [INFO]  server-907ec5a8-597f-0fdf-fc55-2ec897e7b3b1: copied to local snapshot: bytes=71
2025-01-26T02:44:29.444Z [INFO]  server-907ec5a8-597f-0fdf-fc55-2ec897e7b3b1: snapshot restore progress: id=2-113-1737859469444 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:29.444Z [INFO]  server-907ec5a8-597f-0fdf-fc55-2ec897e7b3b1: Installed remote snapshot
2025-01-26T02:44:29.444Z [INFO]  server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: pipelining replication: peer="{Voter server-907ec5a8-597f-0fdf-fc55-2ec897e7b3b1 907ec5a8-597f-0fdf-fc55-2ec897e7b3b1}"
2025-01-26T02:44:29.444Z [INFO]  creating new snapshot: path=/tmp/raft1948268586/snapshots/2-113-1737859469444.tmp
2025-01-26T02:44:29.444Z [INFO]  server-a2d1c30d-dd14-99c5-0e0f-7e675036d0e7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:29.444Z [INFO]  server-a2d1c30d-dd14-99c5-0e0f-7e675036d0e7: copied to local snapshot: bytes=71
2025-01-26T02:44:29.444Z [INFO]  server-a2d1c30d-dd14-99c5-0e0f-7e675036d0e7: snapshot restore progress: id=2-113-1737859469444 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:29.444Z [INFO]  server-a2d1c30d-dd14-99c5-0e0f-7e675036d0e7: Installed remote snapshot
2025-01-26T02:44:29.444Z [INFO]  server-6503adc3-18c2-33e4-aa46-e9202e3a22c0: pipelining replication: peer="{Voter server-a2d1c30d-dd14-99c5-0e0f-7e675036d0e7 a2d1c30d-dd14-99c5-0e0f-7e675036d0e7}"
2025-01-26T02:44:29.445Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:44:30.451Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:44:29.445024506 +0000 UTC m=+65.960056881 (2 nodes), 520ns from start of poll, 1.719903716s from cluster start. Timeout at 2025-01-26 02:44:30.450045626 +0000 UTC m=+66.965077961, 1.00502108s after stability
2025-01-26T02:44:30.468Z [INFO]  server-89fdb94d-fd97-ed77-4cf0-425d6d15643e: aborting pipeline replication: peer="{Voter server-97540042-122b-7561-11f9-e470fa5e4b59 97540042-122b-7561-11f9-e470fa5e4b59}"
2025-01-26T02:44:30.468Z [INFO]  server-89fdb94d-fd97-ed77-4cf0-425d6d15643e: aborting pipeline replication: peer="{Voter server-43eb80ab-6735-59b5-a211-01a2d2ec2a0e 43eb80ab-6735-59b5-a211-01a2d2ec2a0e}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true
2025-01-26T02:44:30.468Z [INFO]  server-2396d588-daa2-fbd1-71b4-9517af205401: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2396d588-daa2-fbd1-71b4-9517af205401 Address:2396d588-daa2-fbd1-71b4-9517af205401} {Suffrage:Voter ID:server-2873f5c5-7fe4-e63d-fdc1-8256c7e90938 Address:2873f5c5-7fe4-e63d-fdc1-8256c7e90938} {Suffrage:Voter ID:server-c7a5fcad-3087-4eec-51b4-6909ad4f7ff5 Address:c7a5fcad-3087-4eec-51b4-6909ad4f7ff5}]"
2025-01-26T02:44:30.468Z [INFO]  server-2873f5c5-7fe4-e63d-fdc1-8256c7e90938: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2396d588-daa2-fbd1-71b4-9517af205401 Address:2396d588-daa2-fbd1-71b4-9517af205401} {Suffrage:Voter ID:server-2873f5c5-7fe4-e63d-fdc1-8256c7e90938 Address:2873f5c5-7fe4-e63d-fdc1-8256c7e90938} {Suffrage:Voter ID:server-c7a5fcad-3087-4eec-51b4-6909ad4f7ff5 Address:c7a5fcad-3087-4eec-51b4-6909ad4f7ff5}]"
2025-01-26T02:44:30.468Z [INFO]  server-c7a5fcad-3087-4eec-51b4-6909ad4f7ff5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2396d588-daa2-fbd1-71b4-9517af205401 Address:2396d588-daa2-fbd1-71b4-9517af205401} {Suffrage:Voter ID:server-2873f5c5-7fe4-e63d-fdc1-8256c7e90938 Address:2873f5c5-7fe4-e63d-fdc1-8256c7e90938} {Suffrage:Voter ID:server-c7a5fcad-3087-4eec-51b4-6909ad4f7ff5 Address:c7a5fcad-3087-4eec-51b4-6909ad4f7ff5}]"
2025-01-26T02:44:30.468Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:30.468Z [INFO]  server-2396d588-daa2-fbd1-71b4-9517af205401: entering follower state: follower="Node at 2396d588-daa2-fbd1-71b4-9517af205401 [Follower]" leader-address= leader-id=
2025-01-26T02:44:30.468Z [INFO]  server-2873f5c5-7fe4-e63d-fdc1-8256c7e90938: entering follower state: follower="Node at 2873f5c5-7fe4-e63d-fdc1-8256c7e90938 [Follower]" leader-address= leader-id=
2025-01-26T02:44:30.468Z [INFO]  server-c7a5fcad-3087-4eec-51b4-6909ad4f7ff5: entering follower state: follower="Node at c7a5fcad-3087-4eec-51b4-6909ad4f7ff5 [Follower]" leader-address= leader-id=
2025-01-26T02:44:31.034Z [WARN]  server-2873f5c5-7fe4-e63d-fdc1-8256c7e90938: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:31.034Z [INFO]  server-2873f5c5-7fe4-e63d-fdc1-8256c7e90938: entering candidate state: node="Node at 2873f5c5-7fe4-e63d-fdc1-8256c7e90938 [Candidate]" term=2
2025-01-26T02:44:31.034Z [INFO]  server-2873f5c5-7fe4-e63d-fdc1-8256c7e90938: election won: term=2 tally=2
2025-01-26T02:44:31.034Z [INFO]  server-2873f5c5-7fe4-e63d-fdc1-8256c7e90938: entering leader state: leader="Node at 2873f5c5-7fe4-e63d-fdc1-8256c7e90938 [Leader]"
2025-01-26T02:44:31.034Z [INFO]  server-2873f5c5-7fe4-e63d-fdc1-8256c7e90938: added peer, starting replication: peer=server-2396d588-daa2-fbd1-71b4-9517af205401
2025-01-26T02:44:31.034Z [INFO]  server-2873f5c5-7fe4-e63d-fdc1-8256c7e90938: added peer, starting replication: peer=server-c7a5fcad-3087-4eec-51b4-6909ad4f7ff5
2025-01-26T02:44:31.034Z [INFO]  server-2873f5c5-7fe4-e63d-fdc1-8256c7e90938: pipelining replication: peer="{Voter server-c7a5fcad-3087-4eec-51b4-6909ad4f7ff5 c7a5fcad-3087-4eec-51b4-6909ad4f7ff5}"
2025-01-26T02:44:31.034Z [WARN]  server-c7a5fcad-3087-4eec-51b4-6909ad4f7ff5: rejecting vote request since our last term is greater: candidate=2873f5c5-7fe4-e63d-fdc1-8256c7e90938 last-term=2 last-candidate-term=1
2025-01-26T02:44:31.034Z [INFO]  server-2873f5c5-7fe4-e63d-fdc1-8256c7e90938: pipelining replication: peer="{Voter server-2396d588-daa2-fbd1-71b4-9517af205401 2396d588-daa2-fbd1-71b4-9517af205401}"
2025-01-26T02:44:32.044Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:31.034782592 +0000 UTC m=+67.549814967 (1 nodes), 565.964714ms from start of poll, 566.157315ms from cluster start. Timeout at 2025-01-26 02:44:32.039792112 +0000 UTC m=+68.554824487, 1.00500952s after stability
2025-01-26T02:44:32.044Z [INFO]  server-2873f5c5-7fe4-e63d-fdc1-8256c7e90938: starting snapshot up to: index=12
2025-01-26T02:44:32.044Z [INFO]  creating new snapshot: path=/tmp/raft2795221938/snapshots/2-12-1737859472044.tmp
2025-01-26T02:44:32.044Z [INFO]  server-2873f5c5-7fe4-e63d-fdc1-8256c7e90938: snapshot complete up to: index=12
2025-01-26T02:44:32.045Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0 Address:57e94ce0-bdc4-2fcf-c5d2-367059715bc0} {Suffrage:Voter ID:server-fb44e971-29a0-d4a9-5a2a-3e02b192d196 Address:fb44e971-29a0-d4a9-5a2a-3e02b192d196} {Suffrage:Voter ID:server-12be280f-2390-9888-b0e4-978cc10ffdd7 Address:12be280f-2390-9888-b0e4-978cc10ffdd7}]"
2025-01-26T02:44:32.045Z [INFO]  server-fb44e971-29a0-d4a9-5a2a-3e02b192d196: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0 Address:57e94ce0-bdc4-2fcf-c5d2-367059715bc0} {Suffrage:Voter ID:server-fb44e971-29a0-d4a9-5a2a-3e02b192d196 Address:fb44e971-29a0-d4a9-5a2a-3e02b192d196} {Suffrage:Voter ID:server-12be280f-2390-9888-b0e4-978cc10ffdd7 Address:12be280f-2390-9888-b0e4-978cc10ffdd7}]"
2025-01-26T02:44:32.045Z [INFO]  server-12be280f-2390-9888-b0e4-978cc10ffdd7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0 Address:57e94ce0-bdc4-2fcf-c5d2-367059715bc0} {Suffrage:Voter ID:server-fb44e971-29a0-d4a9-5a2a-3e02b192d196 Address:fb44e971-29a0-d4a9-5a2a-3e02b192d196} {Suffrage:Voter ID:server-12be280f-2390-9888-b0e4-978cc10ffdd7 Address:12be280f-2390-9888-b0e4-978cc10ffdd7}]"
2025-01-26T02:44:32.045Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:32.045Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: entering follower state: follower="Node at 57e94ce0-bdc4-2fcf-c5d2-367059715bc0 [Follower]" leader-address= leader-id=
2025-01-26T02:44:32.045Z [INFO]  server-fb44e971-29a0-d4a9-5a2a-3e02b192d196: entering follower state: follower="Node at fb44e971-29a0-d4a9-5a2a-3e02b192d196 [Follower]" leader-address= leader-id=
2025-01-26T02:44:32.045Z [INFO]  server-12be280f-2390-9888-b0e4-978cc10ffdd7: entering follower state: follower="Node at 12be280f-2390-9888-b0e4-978cc10ffdd7 [Follower]" leader-address= leader-id=
2025-01-26T02:44:32.579Z [WARN]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:32.579Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: entering candidate state: node="Node at 57e94ce0-bdc4-2fcf-c5d2-367059715bc0 [Candidate]" term=2
2025-01-26T02:44:32.579Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: election won: term=2 tally=2
2025-01-26T02:44:32.579Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: entering leader state: leader="Node at 57e94ce0-bdc4-2fcf-c5d2-367059715bc0 [Leader]"
2025-01-26T02:44:32.579Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: added peer, starting replication: peer=server-fb44e971-29a0-d4a9-5a2a-3e02b192d196
2025-01-26T02:44:32.579Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: added peer, starting replication: peer=server-12be280f-2390-9888-b0e4-978cc10ffdd7
2025-01-26T02:44:32.579Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: pipelining replication: peer="{Voter server-12be280f-2390-9888-b0e4-978cc10ffdd7 12be280f-2390-9888-b0e4-978cc10ffdd7}"
2025-01-26T02:44:32.579Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: pipelining replication: peer="{Voter server-fb44e971-29a0-d4a9-5a2a-3e02b192d196 fb44e971-29a0-d4a9-5a2a-3e02b192d196}"
2025-01-26T02:44:33.587Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:32.579895768 +0000 UTC m=+69.094928103 (1 nodes), 534.213522ms from start of poll, 534.386323ms from cluster start. Timeout at 2025-01-26 02:44:33.584898208 +0000 UTC m=+70.099930583, 1.00500248s after stability
2025-01-26T02:44:33.587Z [INFO]  creating new snapshot: path=/tmp/raft2628969813/snapshots/2-113-1737859473587.tmp
2025-01-26T02:44:33.588Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: copied to local snapshot: bytes=71
2025-01-26T02:44:33.588Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: snapshot restore progress: id=2-113-1737859473587 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:33.588Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: removing all old logs from log store
2025-01-26T02:44:33.588Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: compacting logs: from=1 to=2
2025-01-26T02:44:33.588Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: restored user snapshot: index=113
2025-01-26T02:44:33.588Z [ERROR] server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: failed to get log: index=2 error="log not found"
2025-01-26T02:44:33.588Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: aborting pipeline replication: peer="{Voter server-12be280f-2390-9888-b0e4-978cc10ffdd7 12be280f-2390-9888-b0e4-978cc10ffdd7}"
2025-01-26T02:44:33.588Z [ERROR] server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: failed to get log: index=2 error="log not found"
2025-01-26T02:44:33.588Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: aborting pipeline replication: peer="{Voter server-fb44e971-29a0-d4a9-5a2a-3e02b192d196 fb44e971-29a0-d4a9-5a2a-3e02b192d196}"
2025-01-26T02:44:33.595Z [ERROR] server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: failed to get log: index=2 error="log not found"
2025-01-26T02:44:33.595Z [INFO]  creating new snapshot: path=/tmp/raft1181609998/snapshots/2-113-1737859473595.tmp
2025-01-26T02:44:33.595Z [INFO]  server-12be280f-2390-9888-b0e4-978cc10ffdd7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:33.596Z [INFO]  server-12be280f-2390-9888-b0e4-978cc10ffdd7: copied to local snapshot: bytes=71
2025-01-26T02:44:33.596Z [INFO]  server-12be280f-2390-9888-b0e4-978cc10ffdd7: snapshot restore progress: id=2-113-1737859473595 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:33.596Z [INFO]  server-12be280f-2390-9888-b0e4-978cc10ffdd7: removing all old logs from log store
2025-01-26T02:44:33.596Z [INFO]  server-12be280f-2390-9888-b0e4-978cc10ffdd7: compacting logs: from=1 to=2
2025-01-26T02:44:33.596Z [INFO]  server-12be280f-2390-9888-b0e4-978cc10ffdd7: Installed remote snapshot
2025-01-26T02:44:33.596Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: pipelining replication: peer="{Voter server-12be280f-2390-9888-b0e4-978cc10ffdd7 12be280f-2390-9888-b0e4-978cc10ffdd7}"
2025-01-26T02:44:33.596Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:44:33.603Z [ERROR] server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: failed to get log: index=2 error="log not found"
2025-01-26T02:44:33.604Z [INFO]  creating new snapshot: path=/tmp/raft614158825/snapshots/2-113-1737859473604.tmp
2025-01-26T02:44:33.604Z [INFO]  server-fb44e971-29a0-d4a9-5a2a-3e02b192d196: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:33.604Z [INFO]  server-fb44e971-29a0-d4a9-5a2a-3e02b192d196: copied to local snapshot: bytes=71
2025-01-26T02:44:33.604Z [INFO]  server-fb44e971-29a0-d4a9-5a2a-3e02b192d196: snapshot restore progress: id=2-113-1737859473604 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:33.604Z [INFO]  server-fb44e971-29a0-d4a9-5a2a-3e02b192d196: removing all old logs from log store
2025-01-26T02:44:33.604Z [INFO]  server-fb44e971-29a0-d4a9-5a2a-3e02b192d196: compacting logs: from=1 to=2
2025-01-26T02:44:33.604Z [INFO]  server-fb44e971-29a0-d4a9-5a2a-3e02b192d196: Installed remote snapshot
2025-01-26T02:44:33.604Z [INFO]  server-57e94ce0-bdc4-2fcf-c5d2-367059715bc0: pipelining replication: peer="{Voter server-fb44e971-29a0-d4a9-5a2a-3e02b192d196 fb44e971-29a0-d4a9-5a2a-3e02b192d196}"
2025-01-26T02:44:34.607Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:44:33.596378456 +0000 UTC m=+70.111410831 (2 nodes), 600ns from start of poll, 1.550869051s from cluster start. Timeout at 2025-01-26 02:44:34.601404297 +0000 UTC m=+71.116436672, 1.005025841s after stability
2025-01-26T02:44:34.623Z [INFO]  server-2873f5c5-7fe4-e63d-fdc1-8256c7e90938: aborting pipeline replication: peer="{Voter server-c7a5fcad-3087-4eec-51b4-6909ad4f7ff5 c7a5fcad-3087-4eec-51b4-6909ad4f7ff5}"
2025-01-26T02:44:34.623Z [INFO]  server-2873f5c5-7fe4-e63d-fdc1-8256c7e90938: aborting pipeline replication: peer="{Voter server-2396d588-daa2-fbd1-71b4-9517af205401 2396d588-daa2-fbd1-71b4-9517af205401}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false
2025-01-26T02:44:34.624Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de06d82d-7747-2203-d981-ece70c2572a9 Address:de06d82d-7747-2203-d981-ece70c2572a9} {Suffrage:Voter ID:server-055ea6b3-6331-3189-5752-6a4d9806e488 Address:055ea6b3-6331-3189-5752-6a4d9806e488} {Suffrage:Voter ID:server-2a36c6d1-bf11-b7be-da9e-8024b3634346 Address:2a36c6d1-bf11-b7be-da9e-8024b3634346}]"
2025-01-26T02:44:34.624Z [INFO]  server-055ea6b3-6331-3189-5752-6a4d9806e488: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de06d82d-7747-2203-d981-ece70c2572a9 Address:de06d82d-7747-2203-d981-ece70c2572a9} {Suffrage:Voter ID:server-055ea6b3-6331-3189-5752-6a4d9806e488 Address:055ea6b3-6331-3189-5752-6a4d9806e488} {Suffrage:Voter ID:server-2a36c6d1-bf11-b7be-da9e-8024b3634346 Address:2a36c6d1-bf11-b7be-da9e-8024b3634346}]"
2025-01-26T02:44:34.624Z [INFO]  server-2a36c6d1-bf11-b7be-da9e-8024b3634346: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-de06d82d-7747-2203-d981-ece70c2572a9 Address:de06d82d-7747-2203-d981-ece70c2572a9} {Suffrage:Voter ID:server-055ea6b3-6331-3189-5752-6a4d9806e488 Address:055ea6b3-6331-3189-5752-6a4d9806e488} {Suffrage:Voter ID:server-2a36c6d1-bf11-b7be-da9e-8024b3634346 Address:2a36c6d1-bf11-b7be-da9e-8024b3634346}]"
2025-01-26T02:44:34.624Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:34.624Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: entering follower state: follower="Node at de06d82d-7747-2203-d981-ece70c2572a9 [Follower]" leader-address= leader-id=
2025-01-26T02:44:34.624Z [INFO]  server-055ea6b3-6331-3189-5752-6a4d9806e488: entering follower state: follower="Node at 055ea6b3-6331-3189-5752-6a4d9806e488 [Follower]" leader-address= leader-id=
2025-01-26T02:44:34.624Z [INFO]  server-2a36c6d1-bf11-b7be-da9e-8024b3634346: entering follower state: follower="Node at 2a36c6d1-bf11-b7be-da9e-8024b3634346 [Follower]" leader-address= leader-id=
2025-01-26T02:44:35.187Z [WARN]  server-de06d82d-7747-2203-d981-ece70c2572a9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:35.187Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: entering candidate state: node="Node at de06d82d-7747-2203-d981-ece70c2572a9 [Candidate]" term=2
2025-01-26T02:44:35.187Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: election won: term=2 tally=2
2025-01-26T02:44:35.187Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: entering leader state: leader="Node at de06d82d-7747-2203-d981-ece70c2572a9 [Leader]"
2025-01-26T02:44:35.187Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: added peer, starting replication: peer=server-055ea6b3-6331-3189-5752-6a4d9806e488
2025-01-26T02:44:35.187Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: added peer, starting replication: peer=server-2a36c6d1-bf11-b7be-da9e-8024b3634346
2025-01-26T02:44:35.187Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: pipelining replication: peer="{Voter server-2a36c6d1-bf11-b7be-da9e-8024b3634346 2a36c6d1-bf11-b7be-da9e-8024b3634346}"
2025-01-26T02:44:35.187Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: pipelining replication: peer="{Voter server-055ea6b3-6331-3189-5752-6a4d9806e488 055ea6b3-6331-3189-5752-6a4d9806e488}"
2025-01-26T02:44:36.194Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:35.187835913 +0000 UTC m=+71.702868288 (1 nodes), 563.65778ms from start of poll, 563.858462ms from cluster start. Timeout at 2025-01-26 02:44:36.192838593 +0000 UTC m=+72.707870968, 1.00500268s after stability
2025-01-26T02:44:36.195Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: starting snapshot up to: index=12
2025-01-26T02:44:36.195Z [INFO]  creating new snapshot: path=/tmp/raft1630039085/snapshots/2-12-1737859476195.tmp
2025-01-26T02:44:36.195Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: snapshot complete up to: index=12
2025-01-26T02:44:36.196Z [INFO]  creating new snapshot: path=/tmp/raft1630039085/snapshots/2-1013-1737859476196.tmp
2025-01-26T02:44:36.196Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: copied to local snapshot: bytes=71
2025-01-26T02:44:36.196Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: snapshot restore progress: id=2-1013-1737859476196 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:36.196Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: restored user snapshot: index=1013
2025-01-26T02:44:36.196Z [ERROR] server-de06d82d-7747-2203-d981-ece70c2572a9: failed to get log: index=23 error="log not found"
2025-01-26T02:44:36.196Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: aborting pipeline replication: peer="{Voter server-055ea6b3-6331-3189-5752-6a4d9806e488 055ea6b3-6331-3189-5752-6a4d9806e488}"
2025-01-26T02:44:36.196Z [ERROR] server-de06d82d-7747-2203-d981-ece70c2572a9: failed to get log: index=23 error="log not found"
2025-01-26T02:44:36.196Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: aborting pipeline replication: peer="{Voter server-2a36c6d1-bf11-b7be-da9e-8024b3634346 2a36c6d1-bf11-b7be-da9e-8024b3634346}"
2025-01-26T02:44:36.202Z [ERROR] server-de06d82d-7747-2203-d981-ece70c2572a9: failed to get log: index=23 error="log not found"
2025-01-26T02:44:36.202Z [INFO]  creating new snapshot: path=/tmp/raft821151206/snapshots/2-1013-1737859476202.tmp
2025-01-26T02:44:36.202Z [INFO]  server-2a36c6d1-bf11-b7be-da9e-8024b3634346: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:36.203Z [INFO]  server-2a36c6d1-bf11-b7be-da9e-8024b3634346: copied to local snapshot: bytes=71
2025-01-26T02:44:36.203Z [INFO]  server-2a36c6d1-bf11-b7be-da9e-8024b3634346: snapshot restore progress: id=2-1013-1737859476202 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:36.203Z [INFO]  server-2a36c6d1-bf11-b7be-da9e-8024b3634346: Installed remote snapshot
2025-01-26T02:44:36.203Z [WARN]  server-2a36c6d1-bf11-b7be-da9e-8024b3634346: skipping application of old log: index=22
2025-01-26T02:44:36.203Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: pipelining replication: peer="{Voter server-2a36c6d1-bf11-b7be-da9e-8024b3634346 2a36c6d1-bf11-b7be-da9e-8024b3634346}"
2025-01-26T02:44:36.203Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:44:36.207Z [ERROR] server-de06d82d-7747-2203-d981-ece70c2572a9: failed to get log: index=23 error="log not found"
2025-01-26T02:44:36.207Z [INFO]  creating new snapshot: path=/tmp/raft884507789/snapshots/2-1013-1737859476207.tmp
2025-01-26T02:44:36.208Z [INFO]  server-055ea6b3-6331-3189-5752-6a4d9806e488: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:36.208Z [INFO]  server-055ea6b3-6331-3189-5752-6a4d9806e488: copied to local snapshot: bytes=71
2025-01-26T02:44:36.208Z [INFO]  server-055ea6b3-6331-3189-5752-6a4d9806e488: snapshot restore progress: id=2-1013-1737859476207 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:36.208Z [INFO]  server-055ea6b3-6331-3189-5752-6a4d9806e488: Installed remote snapshot
2025-01-26T02:44:36.208Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: pipelining replication: peer="{Voter server-055ea6b3-6331-3189-5752-6a4d9806e488 055ea6b3-6331-3189-5752-6a4d9806e488}"
2025-01-26T02:44:37.211Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:44:36.203367355 +0000 UTC m=+72.718399690 (2 nodes), 560ns from start of poll, 1.579389864s from cluster start. Timeout at 2025-01-26 02:44:37.208404836 +0000 UTC m=+73.723437211, 1.005037521s after stability
2025-01-26T02:44:37.228Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: aborting pipeline replication: peer="{Voter server-2a36c6d1-bf11-b7be-da9e-8024b3634346 2a36c6d1-bf11-b7be-da9e-8024b3634346}"
2025-01-26T02:44:37.235Z [INFO]  server-de06d82d-7747-2203-d981-ece70c2572a9: aborting pipeline replication: peer="{Voter server-055ea6b3-6331-3189-5752-6a4d9806e488 055ea6b3-6331-3189-5752-6a4d9806e488}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false
2025-01-26T02:44:37.236Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a164c25-af2c-a0b5-15c2-744904930d63 Address:3a164c25-af2c-a0b5-15c2-744904930d63} {Suffrage:Voter ID:server-7d5d5bf1-5eb4-2a83-cc14-ee4916668681 Address:7d5d5bf1-5eb4-2a83-cc14-ee4916668681} {Suffrage:Voter ID:server-7ba80c23-988b-be0a-a363-e3d4fbb44201 Address:7ba80c23-988b-be0a-a363-e3d4fbb44201}]"
2025-01-26T02:44:37.236Z [INFO]  server-7d5d5bf1-5eb4-2a83-cc14-ee4916668681: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a164c25-af2c-a0b5-15c2-744904930d63 Address:3a164c25-af2c-a0b5-15c2-744904930d63} {Suffrage:Voter ID:server-7d5d5bf1-5eb4-2a83-cc14-ee4916668681 Address:7d5d5bf1-5eb4-2a83-cc14-ee4916668681} {Suffrage:Voter ID:server-7ba80c23-988b-be0a-a363-e3d4fbb44201 Address:7ba80c23-988b-be0a-a363-e3d4fbb44201}]"
2025-01-26T02:44:37.236Z [INFO]  server-7ba80c23-988b-be0a-a363-e3d4fbb44201: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a164c25-af2c-a0b5-15c2-744904930d63 Address:3a164c25-af2c-a0b5-15c2-744904930d63} {Suffrage:Voter ID:server-7d5d5bf1-5eb4-2a83-cc14-ee4916668681 Address:7d5d5bf1-5eb4-2a83-cc14-ee4916668681} {Suffrage:Voter ID:server-7ba80c23-988b-be0a-a363-e3d4fbb44201 Address:7ba80c23-988b-be0a-a363-e3d4fbb44201}]"
2025-01-26T02:44:37.236Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:37.236Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: entering follower state: follower="Node at 3a164c25-af2c-a0b5-15c2-744904930d63 [Follower]" leader-address= leader-id=
2025-01-26T02:44:37.236Z [INFO]  server-7d5d5bf1-5eb4-2a83-cc14-ee4916668681: entering follower state: follower="Node at 7d5d5bf1-5eb4-2a83-cc14-ee4916668681 [Follower]" leader-address= leader-id=
2025-01-26T02:44:37.236Z [INFO]  server-7ba80c23-988b-be0a-a363-e3d4fbb44201: entering follower state: follower="Node at 7ba80c23-988b-be0a-a363-e3d4fbb44201 [Follower]" leader-address= leader-id=
2025-01-26T02:44:37.807Z [WARN]  server-3a164c25-af2c-a0b5-15c2-744904930d63: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:37.807Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: entering candidate state: node="Node at 3a164c25-af2c-a0b5-15c2-744904930d63 [Candidate]" term=2
2025-01-26T02:44:37.807Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: election won: term=2 tally=2
2025-01-26T02:44:37.807Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: entering leader state: leader="Node at 3a164c25-af2c-a0b5-15c2-744904930d63 [Leader]"
2025-01-26T02:44:37.807Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: added peer, starting replication: peer=server-7d5d5bf1-5eb4-2a83-cc14-ee4916668681
2025-01-26T02:44:37.807Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: added peer, starting replication: peer=server-7ba80c23-988b-be0a-a363-e3d4fbb44201
2025-01-26T02:44:37.807Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: pipelining replication: peer="{Voter server-7ba80c23-988b-be0a-a363-e3d4fbb44201 7ba80c23-988b-be0a-a363-e3d4fbb44201}"
2025-01-26T02:44:37.807Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: pipelining replication: peer="{Voter server-7d5d5bf1-5eb4-2a83-cc14-ee4916668681 7d5d5bf1-5eb4-2a83-cc14-ee4916668681}"
2025-01-26T02:44:38.815Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:37.807891971 +0000 UTC m=+74.322924346 (1 nodes), 571.555948ms from start of poll, 571.741029ms from cluster start. Timeout at 2025-01-26 02:44:38.812898411 +0000 UTC m=+75.327930746, 1.0050064s after stability
2025-01-26T02:44:38.815Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: starting snapshot up to: index=12
2025-01-26T02:44:38.815Z [INFO]  creating new snapshot: path=/tmp/raft2201781528/snapshots/2-12-1737859478815.tmp
2025-01-26T02:44:38.816Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: snapshot complete up to: index=12
2025-01-26T02:44:38.816Z [INFO]  creating new snapshot: path=/tmp/raft2201781528/snapshots/2-1013-1737859478816.tmp
2025-01-26T02:44:38.816Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: copied to local snapshot: bytes=71
2025-01-26T02:44:38.816Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: snapshot restore progress: id=2-1013-1737859478816 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:38.816Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: removing all old logs from log store
2025-01-26T02:44:38.816Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: compacting logs: from=1 to=22
2025-01-26T02:44:38.816Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: restored user snapshot: index=1013
2025-01-26T02:44:38.816Z [ERROR] server-3a164c25-af2c-a0b5-15c2-744904930d63: failed to get log: index=20 error="log not found"
2025-01-26T02:44:38.816Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: aborting pipeline replication: peer="{Voter server-7d5d5bf1-5eb4-2a83-cc14-ee4916668681 7d5d5bf1-5eb4-2a83-cc14-ee4916668681}"
2025-01-26T02:44:38.816Z [ERROR] server-3a164c25-af2c-a0b5-15c2-744904930d63: failed to get log: index=22 error="log not found"
2025-01-26T02:44:38.816Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: aborting pipeline replication: peer="{Voter server-7ba80c23-988b-be0a-a363-e3d4fbb44201 7ba80c23-988b-be0a-a363-e3d4fbb44201}"
2025-01-26T02:44:38.816Z [ERROR] server-3a164c25-af2c-a0b5-15c2-744904930d63: failed to get log: index=20 error="log not found"
2025-01-26T02:44:38.817Z [INFO]  creating new snapshot: path=/tmp/raft1769546999/snapshots/2-1013-1737859478817.tmp
2025-01-26T02:44:38.817Z [INFO]  server-7d5d5bf1-5eb4-2a83-cc14-ee4916668681: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:38.817Z [INFO]  server-7d5d5bf1-5eb4-2a83-cc14-ee4916668681: copied to local snapshot: bytes=71
2025-01-26T02:44:38.817Z [INFO]  server-7d5d5bf1-5eb4-2a83-cc14-ee4916668681: snapshot restore progress: id=2-1013-1737859478817 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:38.817Z [INFO]  server-7d5d5bf1-5eb4-2a83-cc14-ee4916668681: removing all old logs from log store
2025-01-26T02:44:38.817Z [INFO]  server-7d5d5bf1-5eb4-2a83-cc14-ee4916668681: compacting logs: from=1 to=20
2025-01-26T02:44:38.817Z [INFO]  server-7d5d5bf1-5eb4-2a83-cc14-ee4916668681: Installed remote snapshot
2025-01-26T02:44:38.817Z [WARN]  server-7d5d5bf1-5eb4-2a83-cc14-ee4916668681: skipping application of old log: index=22
2025-01-26T02:44:38.817Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: pipelining replication: peer="{Voter server-7d5d5bf1-5eb4-2a83-cc14-ee4916668681 7d5d5bf1-5eb4-2a83-cc14-ee4916668681}"
2025-01-26T02:44:38.817Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:44:38.823Z [ERROR] server-3a164c25-af2c-a0b5-15c2-744904930d63: failed to get log: index=22 error="log not found"
2025-01-26T02:44:38.823Z [INFO]  creating new snapshot: path=/tmp/raft740507461/snapshots/2-1013-1737859478823.tmp
2025-01-26T02:44:38.831Z [INFO]  server-7ba80c23-988b-be0a-a363-e3d4fbb44201: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:38.831Z [INFO]  server-7ba80c23-988b-be0a-a363-e3d4fbb44201: copied to local snapshot: bytes=71
2025-01-26T02:44:38.832Z [INFO]  server-7ba80c23-988b-be0a-a363-e3d4fbb44201: snapshot restore progress: id=2-1013-1737859478823 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:38.832Z [INFO]  server-7ba80c23-988b-be0a-a363-e3d4fbb44201: removing all old logs from log store
2025-01-26T02:44:38.832Z [INFO]  server-7ba80c23-988b-be0a-a363-e3d4fbb44201: compacting logs: from=1 to=22
2025-01-26T02:44:38.832Z [INFO]  server-7ba80c23-988b-be0a-a363-e3d4fbb44201: Installed remote snapshot
2025-01-26T02:44:38.832Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: pipelining replication: peer="{Voter server-7ba80c23-988b-be0a-a363-e3d4fbb44201 7ba80c23-988b-be0a-a363-e3d4fbb44201}"
2025-01-26T02:44:39.823Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:44:38.817577418 +0000 UTC m=+75.332609793 (2 nodes), 520ns from start of poll, 1.581426476s from cluster start. Timeout at 2025-01-26 02:44:39.822582458 +0000 UTC m=+76.337614913, 1.00500512s after stability
2025-01-26T02:44:39.843Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: aborting pipeline replication: peer="{Voter server-7ba80c23-988b-be0a-a363-e3d4fbb44201 7ba80c23-988b-be0a-a363-e3d4fbb44201}"
2025-01-26T02:44:39.843Z [INFO]  server-3a164c25-af2c-a0b5-15c2-744904930d63: aborting pipeline replication: peer="{Voter server-7d5d5bf1-5eb4-2a83-cc14-ee4916668681 7d5d5bf1-5eb4-2a83-cc14-ee4916668681}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true
2025-01-26T02:44:39.844Z [INFO]  server-62b1af16-bda2-2c8f-ecaa-1f96789c03aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62b1af16-bda2-2c8f-ecaa-1f96789c03aa Address:62b1af16-bda2-2c8f-ecaa-1f96789c03aa} {Suffrage:Voter ID:server-bcc59535-31b6-4b87-5c59-b730c14ff914 Address:bcc59535-31b6-4b87-5c59-b730c14ff914} {Suffrage:Voter ID:server-51b82b78-2afb-c92e-4b84-70f4957b12fa Address:51b82b78-2afb-c92e-4b84-70f4957b12fa}]"
2025-01-26T02:44:39.844Z [INFO]  server-bcc59535-31b6-4b87-5c59-b730c14ff914: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62b1af16-bda2-2c8f-ecaa-1f96789c03aa Address:62b1af16-bda2-2c8f-ecaa-1f96789c03aa} {Suffrage:Voter ID:server-bcc59535-31b6-4b87-5c59-b730c14ff914 Address:bcc59535-31b6-4b87-5c59-b730c14ff914} {Suffrage:Voter ID:server-51b82b78-2afb-c92e-4b84-70f4957b12fa Address:51b82b78-2afb-c92e-4b84-70f4957b12fa}]"
2025-01-26T02:44:39.844Z [INFO]  server-51b82b78-2afb-c92e-4b84-70f4957b12fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62b1af16-bda2-2c8f-ecaa-1f96789c03aa Address:62b1af16-bda2-2c8f-ecaa-1f96789c03aa} {Suffrage:Voter ID:server-bcc59535-31b6-4b87-5c59-b730c14ff914 Address:bcc59535-31b6-4b87-5c59-b730c14ff914} {Suffrage:Voter ID:server-51b82b78-2afb-c92e-4b84-70f4957b12fa Address:51b82b78-2afb-c92e-4b84-70f4957b12fa}]"
2025-01-26T02:44:39.844Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:39.844Z [INFO]  server-62b1af16-bda2-2c8f-ecaa-1f96789c03aa: entering follower state: follower="Node at 62b1af16-bda2-2c8f-ecaa-1f96789c03aa [Follower]" leader-address= leader-id=
2025-01-26T02:44:39.844Z [INFO]  server-bcc59535-31b6-4b87-5c59-b730c14ff914: entering follower state: follower="Node at bcc59535-31b6-4b87-5c59-b730c14ff914 [Follower]" leader-address= leader-id=
2025-01-26T02:44:39.844Z [INFO]  server-51b82b78-2afb-c92e-4b84-70f4957b12fa: entering follower state: follower="Node at 51b82b78-2afb-c92e-4b84-70f4957b12fa [Follower]" leader-address= leader-id=
2025-01-26T02:44:40.487Z [WARN]  server-51b82b78-2afb-c92e-4b84-70f4957b12fa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:40.487Z [INFO]  server-51b82b78-2afb-c92e-4b84-70f4957b12fa: entering candidate state: node="Node at 51b82b78-2afb-c92e-4b84-70f4957b12fa [Candidate]" term=2
2025-01-26T02:44:40.487Z [INFO]  server-51b82b78-2afb-c92e-4b84-70f4957b12fa: election won: term=2 tally=2
2025-01-26T02:44:40.487Z [INFO]  server-51b82b78-2afb-c92e-4b84-70f4957b12fa: entering leader state: leader="Node at 51b82b78-2afb-c92e-4b84-70f4957b12fa [Leader]"
2025-01-26T02:44:40.487Z [INFO]  server-51b82b78-2afb-c92e-4b84-70f4957b12fa: added peer, starting replication: peer=server-62b1af16-bda2-2c8f-ecaa-1f96789c03aa
2025-01-26T02:44:40.487Z [INFO]  server-51b82b78-2afb-c92e-4b84-70f4957b12fa: added peer, starting replication: peer=server-bcc59535-31b6-4b87-5c59-b730c14ff914
2025-01-26T02:44:40.487Z [INFO]  server-51b82b78-2afb-c92e-4b84-70f4957b12fa: pipelining replication: peer="{Voter server-bcc59535-31b6-4b87-5c59-b730c14ff914 bcc59535-31b6-4b87-5c59-b730c14ff914}"
2025-01-26T02:44:40.487Z [INFO]  server-51b82b78-2afb-c92e-4b84-70f4957b12fa: pipelining replication: peer="{Voter server-62b1af16-bda2-2c8f-ecaa-1f96789c03aa 62b1af16-bda2-2c8f-ecaa-1f96789c03aa}"
2025-01-26T02:44:41.495Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:40.487894913 +0000 UTC m=+77.002927248 (1 nodes), 643.722506ms from start of poll, 643.907907ms from cluster start. Timeout at 2025-01-26 02:44:41.492905233 +0000 UTC m=+78.007937608, 1.00501036s after stability
2025-01-26T02:44:41.495Z [INFO]  server-51b82b78-2afb-c92e-4b84-70f4957b12fa: starting snapshot up to: index=12
2025-01-26T02:44:41.495Z [INFO]  creating new snapshot: path=/tmp/raft2288776449/snapshots/2-12-1737859481495.tmp
2025-01-26T02:44:41.496Z [INFO]  server-51b82b78-2afb-c92e-4b84-70f4957b12fa: snapshot complete up to: index=12
2025-01-26T02:44:41.496Z [INFO]  server-0cc7cf72-d804-c502-04bc-2e6809196f33: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cc7cf72-d804-c502-04bc-2e6809196f33 Address:0cc7cf72-d804-c502-04bc-2e6809196f33} {Suffrage:Voter ID:server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d Address:f20486ad-2bfc-cd8a-c7b7-80447bc22f8d} {Suffrage:Voter ID:server-7c170814-a709-fed9-3d14-22699282a391 Address:7c170814-a709-fed9-3d14-22699282a391}]"
2025-01-26T02:44:41.496Z [INFO]  server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cc7cf72-d804-c502-04bc-2e6809196f33 Address:0cc7cf72-d804-c502-04bc-2e6809196f33} {Suffrage:Voter ID:server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d Address:f20486ad-2bfc-cd8a-c7b7-80447bc22f8d} {Suffrage:Voter ID:server-7c170814-a709-fed9-3d14-22699282a391 Address:7c170814-a709-fed9-3d14-22699282a391}]"
2025-01-26T02:44:41.496Z [INFO]  server-7c170814-a709-fed9-3d14-22699282a391: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cc7cf72-d804-c502-04bc-2e6809196f33 Address:0cc7cf72-d804-c502-04bc-2e6809196f33} {Suffrage:Voter ID:server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d Address:f20486ad-2bfc-cd8a-c7b7-80447bc22f8d} {Suffrage:Voter ID:server-7c170814-a709-fed9-3d14-22699282a391 Address:7c170814-a709-fed9-3d14-22699282a391}]"
2025-01-26T02:44:41.496Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:41.496Z [INFO]  server-0cc7cf72-d804-c502-04bc-2e6809196f33: entering follower state: follower="Node at 0cc7cf72-d804-c502-04bc-2e6809196f33 [Follower]" leader-address= leader-id=
2025-01-26T02:44:41.496Z [INFO]  server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: entering follower state: follower="Node at f20486ad-2bfc-cd8a-c7b7-80447bc22f8d [Follower]" leader-address= leader-id=
2025-01-26T02:44:41.496Z [INFO]  server-7c170814-a709-fed9-3d14-22699282a391: entering follower state: follower="Node at 7c170814-a709-fed9-3d14-22699282a391 [Follower]" leader-address= leader-id=
2025-01-26T02:44:42.067Z [WARN]  server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:42.067Z [INFO]  server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: entering candidate state: node="Node at f20486ad-2bfc-cd8a-c7b7-80447bc22f8d [Candidate]" term=2
2025-01-26T02:44:42.067Z [INFO]  server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: election won: term=2 tally=2
2025-01-26T02:44:42.067Z [INFO]  server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: entering leader state: leader="Node at f20486ad-2bfc-cd8a-c7b7-80447bc22f8d [Leader]"
2025-01-26T02:44:42.067Z [INFO]  server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: added peer, starting replication: peer=server-0cc7cf72-d804-c502-04bc-2e6809196f33
2025-01-26T02:44:42.067Z [INFO]  server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: added peer, starting replication: peer=server-7c170814-a709-fed9-3d14-22699282a391
2025-01-26T02:44:42.067Z [INFO]  server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: pipelining replication: peer="{Voter server-7c170814-a709-fed9-3d14-22699282a391 7c170814-a709-fed9-3d14-22699282a391}"
2025-01-26T02:44:42.067Z [INFO]  server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: pipelining replication: peer="{Voter server-0cc7cf72-d804-c502-04bc-2e6809196f33 0cc7cf72-d804-c502-04bc-2e6809196f33}"
2025-01-26T02:44:43.079Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:42.067903941 +0000 UTC m=+78.582936316 (1 nodes), 571.016585ms from start of poll, 571.184866ms from cluster start. Timeout at 2025-01-26 02:44:43.072907701 +0000 UTC m=+79.587940076, 1.00500376s after stability
2025-01-26T02:44:43.079Z [INFO]  creating new snapshot: path=/tmp/raft4088156871/snapshots/2-1013-1737859483079.tmp
2025-01-26T02:44:43.079Z [INFO]  server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: copied to local snapshot: bytes=71
2025-01-26T02:44:43.079Z [INFO]  server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: snapshot restore progress: id=2-1013-1737859483079 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:43.079Z [INFO]  server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: restored user snapshot: index=1013
2025-01-26T02:44:43.079Z [ERROR] server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: failed to get log: index=3 error="log not found"
2025-01-26T02:44:43.079Z [INFO]  server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: aborting pipeline replication: peer="{Voter server-7c170814-a709-fed9-3d14-22699282a391 7c170814-a709-fed9-3d14-22699282a391}"
2025-01-26T02:44:43.080Z [ERROR] server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: failed to get log: index=3 error="log not found"
2025-01-26T02:44:43.080Z [INFO]  server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: aborting pipeline replication: peer="{Voter server-0cc7cf72-d804-c502-04bc-2e6809196f33 0cc7cf72-d804-c502-04bc-2e6809196f33}"
2025-01-26T02:44:43.087Z [ERROR] server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: failed to get log: index=3 error="log not found"
2025-01-26T02:44:43.088Z [INFO]  creating new snapshot: path=/tmp/raft705098242/snapshots/2-1013-1737859483088.tmp
2025-01-26T02:44:43.088Z [INFO]  server-7c170814-a709-fed9-3d14-22699282a391: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:43.088Z [INFO]  server-7c170814-a709-fed9-3d14-22699282a391: copied to local snapshot: bytes=71
2025-01-26T02:44:43.088Z [INFO]  server-7c170814-a709-fed9-3d14-22699282a391: snapshot restore progress: id=2-1013-1737859483088 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:43.088Z [INFO]  server-7c170814-a709-fed9-3d14-22699282a391: Installed remote snapshot
2025-01-26T02:44:43.088Z [INFO]  server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: pipelining replication: peer="{Voter server-7c170814-a709-fed9-3d14-22699282a391 7c170814-a709-fed9-3d14-22699282a391}"
2025-01-26T02:44:43.088Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:44:43.091Z [ERROR] server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: failed to get log: index=3 error="log not found"
2025-01-26T02:44:43.091Z [INFO]  creating new snapshot: path=/tmp/raft2294073319/snapshots/2-1013-1737859483091.tmp
2025-01-26T02:44:43.091Z [INFO]  server-0cc7cf72-d804-c502-04bc-2e6809196f33: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:43.091Z [INFO]  server-0cc7cf72-d804-c502-04bc-2e6809196f33: copied to local snapshot: bytes=71
2025-01-26T02:44:43.092Z [INFO]  server-0cc7cf72-d804-c502-04bc-2e6809196f33: snapshot restore progress: id=2-1013-1737859483091 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:43.092Z [INFO]  server-0cc7cf72-d804-c502-04bc-2e6809196f33: Installed remote snapshot
2025-01-26T02:44:43.092Z [INFO]  server-f20486ad-2bfc-cd8a-c7b7-80447bc22f8d: pipelining replication: peer="{Voter server-0cc7cf72-d804-c502-04bc-2e6809196f33 0cc7cf72-d804-c502-04bc-2e6809196f33}"
2025-01-26T02:44:44.095Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:44:43.088479733 +0000 UTC m=+79.603512068 (2 nodes), 560ns from start of poll, 1.591760618s from cluster start. Timeout at 2025-01-26 02:44:44.093496533 +0000 UTC m=+80.608528908, 1.00501684s after stability
2025-01-26T02:44:44.115Z [INFO]  server-51b82b78-2afb-c92e-4b84-70f4957b12fa: aborting pipeline replication: peer="{Voter server-bcc59535-31b6-4b87-5c59-b730c14ff914 bcc59535-31b6-4b87-5c59-b730c14ff914}"
2025-01-26T02:44:44.115Z [INFO]  server-51b82b78-2afb-c92e-4b84-70f4957b12fa: aborting pipeline replication: peer="{Voter server-62b1af16-bda2-2c8f-ecaa-1f96789c03aa 62b1af16-bda2-2c8f-ecaa-1f96789c03aa}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true
2025-01-26T02:44:44.116Z [INFO]  server-dea87440-8e6c-19a8-6989-9c6960072760: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dea87440-8e6c-19a8-6989-9c6960072760 Address:dea87440-8e6c-19a8-6989-9c6960072760} {Suffrage:Voter ID:server-90377db2-ab59-05d3-60c3-e913d4b972f9 Address:90377db2-ab59-05d3-60c3-e913d4b972f9} {Suffrage:Voter ID:server-b5ef38c7-49de-bad9-349e-55e15d19e544 Address:b5ef38c7-49de-bad9-349e-55e15d19e544}]"
2025-01-26T02:44:44.116Z [INFO]  server-90377db2-ab59-05d3-60c3-e913d4b972f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dea87440-8e6c-19a8-6989-9c6960072760 Address:dea87440-8e6c-19a8-6989-9c6960072760} {Suffrage:Voter ID:server-90377db2-ab59-05d3-60c3-e913d4b972f9 Address:90377db2-ab59-05d3-60c3-e913d4b972f9} {Suffrage:Voter ID:server-b5ef38c7-49de-bad9-349e-55e15d19e544 Address:b5ef38c7-49de-bad9-349e-55e15d19e544}]"
2025-01-26T02:44:44.116Z [INFO]  server-b5ef38c7-49de-bad9-349e-55e15d19e544: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dea87440-8e6c-19a8-6989-9c6960072760 Address:dea87440-8e6c-19a8-6989-9c6960072760} {Suffrage:Voter ID:server-90377db2-ab59-05d3-60c3-e913d4b972f9 Address:90377db2-ab59-05d3-60c3-e913d4b972f9} {Suffrage:Voter ID:server-b5ef38c7-49de-bad9-349e-55e15d19e544 Address:b5ef38c7-49de-bad9-349e-55e15d19e544}]"
2025-01-26T02:44:44.116Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:44.116Z [INFO]  server-dea87440-8e6c-19a8-6989-9c6960072760: entering follower state: follower="Node at dea87440-8e6c-19a8-6989-9c6960072760 [Follower]" leader-address= leader-id=
2025-01-26T02:44:44.116Z [INFO]  server-90377db2-ab59-05d3-60c3-e913d4b972f9: entering follower state: follower="Node at 90377db2-ab59-05d3-60c3-e913d4b972f9 [Follower]" leader-address= leader-id=
2025-01-26T02:44:44.116Z [INFO]  server-b5ef38c7-49de-bad9-349e-55e15d19e544: entering follower state: follower="Node at b5ef38c7-49de-bad9-349e-55e15d19e544 [Follower]" leader-address= leader-id=
2025-01-26T02:44:44.759Z [WARN]  server-dea87440-8e6c-19a8-6989-9c6960072760: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:44.759Z [INFO]  server-dea87440-8e6c-19a8-6989-9c6960072760: entering candidate state: node="Node at dea87440-8e6c-19a8-6989-9c6960072760 [Candidate]" term=2
2025-01-26T02:44:44.759Z [INFO]  server-dea87440-8e6c-19a8-6989-9c6960072760: election won: term=2 tally=2
2025-01-26T02:44:44.759Z [INFO]  server-dea87440-8e6c-19a8-6989-9c6960072760: entering leader state: leader="Node at dea87440-8e6c-19a8-6989-9c6960072760 [Leader]"
2025-01-26T02:44:44.759Z [INFO]  server-dea87440-8e6c-19a8-6989-9c6960072760: added peer, starting replication: peer=server-90377db2-ab59-05d3-60c3-e913d4b972f9
2025-01-26T02:44:44.759Z [INFO]  server-dea87440-8e6c-19a8-6989-9c6960072760: added peer, starting replication: peer=server-b5ef38c7-49de-bad9-349e-55e15d19e544
2025-01-26T02:44:44.760Z [INFO]  server-dea87440-8e6c-19a8-6989-9c6960072760: pipelining replication: peer="{Voter server-90377db2-ab59-05d3-60c3-e913d4b972f9 90377db2-ab59-05d3-60c3-e913d4b972f9}"
2025-01-26T02:44:44.760Z [INFO]  server-dea87440-8e6c-19a8-6989-9c6960072760: pipelining replication: peer="{Voter server-b5ef38c7-49de-bad9-349e-55e15d19e544 b5ef38c7-49de-bad9-349e-55e15d19e544}"
2025-01-26T02:44:45.767Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:44.760138437 +0000 UTC m=+81.275170812 (1 nodes), 643.926388ms from start of poll, 644.116229ms from cluster start. Timeout at 2025-01-26 02:44:45.765142237 +0000 UTC m=+82.280174612, 1.0050038s after stability
2025-01-26T02:44:45.768Z [INFO]  server-dea87440-8e6c-19a8-6989-9c6960072760: starting snapshot up to: index=12
2025-01-26T02:44:45.768Z [INFO]  creating new snapshot: path=/tmp/raft2390938649/snapshots/2-12-1737859485768.tmp
2025-01-26T02:44:45.768Z [INFO]  server-dea87440-8e6c-19a8-6989-9c6960072760: snapshot complete up to: index=12
2025-01-26T02:44:45.769Z [INFO]  server-2f607944-1674-c087-103e-dfd9cd15c1b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f607944-1674-c087-103e-dfd9cd15c1b1 Address:2f607944-1674-c087-103e-dfd9cd15c1b1} {Suffrage:Voter ID:server-62b657d9-7977-86a3-2eef-2a575c5a6355 Address:62b657d9-7977-86a3-2eef-2a575c5a6355} {Suffrage:Voter ID:server-c188c7e6-1233-3a53-6239-88c740b12e17 Address:c188c7e6-1233-3a53-6239-88c740b12e17}]"
2025-01-26T02:44:45.769Z [INFO]  server-62b657d9-7977-86a3-2eef-2a575c5a6355: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f607944-1674-c087-103e-dfd9cd15c1b1 Address:2f607944-1674-c087-103e-dfd9cd15c1b1} {Suffrage:Voter ID:server-62b657d9-7977-86a3-2eef-2a575c5a6355 Address:62b657d9-7977-86a3-2eef-2a575c5a6355} {Suffrage:Voter ID:server-c188c7e6-1233-3a53-6239-88c740b12e17 Address:c188c7e6-1233-3a53-6239-88c740b12e17}]"
2025-01-26T02:44:45.769Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2f607944-1674-c087-103e-dfd9cd15c1b1 Address:2f607944-1674-c087-103e-dfd9cd15c1b1} {Suffrage:Voter ID:server-62b657d9-7977-86a3-2eef-2a575c5a6355 Address:62b657d9-7977-86a3-2eef-2a575c5a6355} {Suffrage:Voter ID:server-c188c7e6-1233-3a53-6239-88c740b12e17 Address:c188c7e6-1233-3a53-6239-88c740b12e17}]"
2025-01-26T02:44:45.769Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:45.769Z [INFO]  server-2f607944-1674-c087-103e-dfd9cd15c1b1: entering follower state: follower="Node at 2f607944-1674-c087-103e-dfd9cd15c1b1 [Follower]" leader-address= leader-id=
2025-01-26T02:44:45.769Z [INFO]  server-62b657d9-7977-86a3-2eef-2a575c5a6355: entering follower state: follower="Node at 62b657d9-7977-86a3-2eef-2a575c5a6355 [Follower]" leader-address= leader-id=
2025-01-26T02:44:45.769Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: entering follower state: follower="Node at c188c7e6-1233-3a53-6239-88c740b12e17 [Follower]" leader-address= leader-id=
2025-01-26T02:44:46.463Z [WARN]  server-c188c7e6-1233-3a53-6239-88c740b12e17: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:46.463Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: entering candidate state: node="Node at c188c7e6-1233-3a53-6239-88c740b12e17 [Candidate]" term=2
2025-01-26T02:44:46.463Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: election won: term=2 tally=2
2025-01-26T02:44:46.463Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: entering leader state: leader="Node at c188c7e6-1233-3a53-6239-88c740b12e17 [Leader]"
2025-01-26T02:44:46.463Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: added peer, starting replication: peer=server-2f607944-1674-c087-103e-dfd9cd15c1b1
2025-01-26T02:44:46.463Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: added peer, starting replication: peer=server-62b657d9-7977-86a3-2eef-2a575c5a6355
2025-01-26T02:44:46.463Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: pipelining replication: peer="{Voter server-62b657d9-7977-86a3-2eef-2a575c5a6355 62b657d9-7977-86a3-2eef-2a575c5a6355}"
2025-01-26T02:44:46.464Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: pipelining replication: peer="{Voter server-2f607944-1674-c087-103e-dfd9cd15c1b1 2f607944-1674-c087-103e-dfd9cd15c1b1}"
2025-01-26T02:44:47.471Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:46.464069936 +0000 UTC m=+82.979102311 (1 nodes), 694.669895ms from start of poll, 694.856856ms from cluster start. Timeout at 2025-01-26 02:44:47.469079256 +0000 UTC m=+83.984111631, 1.00500932s after stability
2025-01-26T02:44:47.471Z [INFO]  creating new snapshot: path=/tmp/raft2096613472/snapshots/2-1013-1737859487471.tmp
2025-01-26T02:44:47.471Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: copied to local snapshot: bytes=71
2025-01-26T02:44:47.472Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: snapshot restore progress: id=2-1013-1737859487471 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:47.472Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: removing all old logs from log store
2025-01-26T02:44:47.472Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: compacting logs: from=1 to=2
2025-01-26T02:44:47.472Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: restored user snapshot: index=1013
2025-01-26T02:44:47.472Z [ERROR] server-c188c7e6-1233-3a53-6239-88c740b12e17: failed to get log: index=2 error="log not found"
2025-01-26T02:44:47.472Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: aborting pipeline replication: peer="{Voter server-62b657d9-7977-86a3-2eef-2a575c5a6355 62b657d9-7977-86a3-2eef-2a575c5a6355}"
2025-01-26T02:44:47.472Z [ERROR] server-c188c7e6-1233-3a53-6239-88c740b12e17: failed to get log: index=2 error="log not found"
2025-01-26T02:44:47.472Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: aborting pipeline replication: peer="{Voter server-2f607944-1674-c087-103e-dfd9cd15c1b1 2f607944-1674-c087-103e-dfd9cd15c1b1}"
2025-01-26T02:44:47.479Z [ERROR] server-c188c7e6-1233-3a53-6239-88c740b12e17: failed to get log: index=2 error="log not found"
2025-01-26T02:44:47.480Z [INFO]  creating new snapshot: path=/tmp/raft3031527360/snapshots/2-1013-1737859487480.tmp
2025-01-26T02:44:47.480Z [ERROR] server-c188c7e6-1233-3a53-6239-88c740b12e17: failed to get log: index=2 error="log not found"
2025-01-26T02:44:47.480Z [INFO]  creating new snapshot: path=/tmp/raft3608435346/snapshots/2-1013-1737859487480.tmp
2025-01-26T02:44:47.480Z [INFO]  server-62b657d9-7977-86a3-2eef-2a575c5a6355: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:47.480Z [INFO]  server-62b657d9-7977-86a3-2eef-2a575c5a6355: copied to local snapshot: bytes=71
2025-01-26T02:44:47.480Z [INFO]  server-62b657d9-7977-86a3-2eef-2a575c5a6355: snapshot restore progress: id=2-1013-1737859487480 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:47.480Z [INFO]  server-62b657d9-7977-86a3-2eef-2a575c5a6355: removing all old logs from log store
2025-01-26T02:44:47.480Z [INFO]  server-62b657d9-7977-86a3-2eef-2a575c5a6355: compacting logs: from=1 to=2
2025-01-26T02:44:47.480Z [INFO]  server-62b657d9-7977-86a3-2eef-2a575c5a6355: Installed remote snapshot
2025-01-26T02:44:47.480Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: pipelining replication: peer="{Voter server-62b657d9-7977-86a3-2eef-2a575c5a6355 62b657d9-7977-86a3-2eef-2a575c5a6355}"
2025-01-26T02:44:47.480Z [INFO]  server-2f607944-1674-c087-103e-dfd9cd15c1b1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:47.481Z [INFO]  server-2f607944-1674-c087-103e-dfd9cd15c1b1: copied to local snapshot: bytes=71
2025-01-26T02:44:47.481Z [INFO]  server-2f607944-1674-c087-103e-dfd9cd15c1b1: snapshot restore progress: id=2-1013-1737859487480 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:47.481Z [INFO]  server-2f607944-1674-c087-103e-dfd9cd15c1b1: removing all old logs from log store
2025-01-26T02:44:47.481Z [INFO]  server-2f607944-1674-c087-103e-dfd9cd15c1b1: compacting logs: from=1 to=2
2025-01-26T02:44:47.481Z [INFO]  server-2f607944-1674-c087-103e-dfd9cd15c1b1: Installed remote snapshot
2025-01-26T02:44:47.481Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:44:47.481Z [INFO]  server-c188c7e6-1233-3a53-6239-88c740b12e17: pipelining replication: peer="{Voter server-2f607944-1674-c087-103e-dfd9cd15c1b1 2f607944-1674-c087-103e-dfd9cd15c1b1}"
2025-01-26T02:44:48.487Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:44:47.481201748 +0000 UTC m=+83.996234123 (2 nodes), 480ns from start of poll, 1.711988668s from cluster start. Timeout at 2025-01-26 02:44:48.486211628 +0000 UTC m=+85.001244003, 1.00500988s after stability
2025-01-26T02:44:48.503Z [INFO]  server-dea87440-8e6c-19a8-6989-9c6960072760: aborting pipeline replication: peer="{Voter server-b5ef38c7-49de-bad9-349e-55e15d19e544 b5ef38c7-49de-bad9-349e-55e15d19e544}"
2025-01-26T02:44:48.503Z [INFO]  server-dea87440-8e6c-19a8-6989-9c6960072760: aborting pipeline replication: peer="{Voter server-90377db2-ab59-05d3-60c3-e913d4b972f9 90377db2-ab59-05d3-60c3-e913d4b972f9}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false
2025-01-26T02:44:48.504Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c Address:1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c} {Suffrage:Voter ID:server-22c5b97f-3c38-682b-1c2c-e23e3590aa31 Address:22c5b97f-3c38-682b-1c2c-e23e3590aa31} {Suffrage:Voter ID:server-866b762d-23e5-27e5-1204-9db63d55d6da Address:866b762d-23e5-27e5-1204-9db63d55d6da}]"
2025-01-26T02:44:48.504Z [INFO]  server-22c5b97f-3c38-682b-1c2c-e23e3590aa31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c Address:1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c} {Suffrage:Voter ID:server-22c5b97f-3c38-682b-1c2c-e23e3590aa31 Address:22c5b97f-3c38-682b-1c2c-e23e3590aa31} {Suffrage:Voter ID:server-866b762d-23e5-27e5-1204-9db63d55d6da Address:866b762d-23e5-27e5-1204-9db63d55d6da}]"
2025-01-26T02:44:48.504Z [INFO]  server-866b762d-23e5-27e5-1204-9db63d55d6da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c Address:1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c} {Suffrage:Voter ID:server-22c5b97f-3c38-682b-1c2c-e23e3590aa31 Address:22c5b97f-3c38-682b-1c2c-e23e3590aa31} {Suffrage:Voter ID:server-866b762d-23e5-27e5-1204-9db63d55d6da Address:866b762d-23e5-27e5-1204-9db63d55d6da}]"
2025-01-26T02:44:48.504Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:48.504Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: entering follower state: follower="Node at 1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c [Follower]" leader-address= leader-id=
2025-01-26T02:44:48.504Z [INFO]  server-22c5b97f-3c38-682b-1c2c-e23e3590aa31: entering follower state: follower="Node at 22c5b97f-3c38-682b-1c2c-e23e3590aa31 [Follower]" leader-address= leader-id=
2025-01-26T02:44:48.504Z [INFO]  server-866b762d-23e5-27e5-1204-9db63d55d6da: entering follower state: follower="Node at 866b762d-23e5-27e5-1204-9db63d55d6da [Follower]" leader-address= leader-id=
2025-01-26T02:44:49.143Z [WARN]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:49.143Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: entering candidate state: node="Node at 1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c [Candidate]" term=2
2025-01-26T02:44:49.143Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: election won: term=2 tally=2
2025-01-26T02:44:49.143Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: entering leader state: leader="Node at 1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c [Leader]"
2025-01-26T02:44:49.143Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: added peer, starting replication: peer=server-22c5b97f-3c38-682b-1c2c-e23e3590aa31
2025-01-26T02:44:49.143Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: added peer, starting replication: peer=server-866b762d-23e5-27e5-1204-9db63d55d6da
2025-01-26T02:44:49.144Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: pipelining replication: peer="{Voter server-22c5b97f-3c38-682b-1c2c-e23e3590aa31 22c5b97f-3c38-682b-1c2c-e23e3590aa31}"
2025-01-26T02:44:49.144Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: pipelining replication: peer="{Voter server-866b762d-23e5-27e5-1204-9db63d55d6da 866b762d-23e5-27e5-1204-9db63d55d6da}"
2025-01-26T02:44:50.175Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:49.144173399 +0000 UTC m=+85.659205774 (1 nodes), 639.950083ms from start of poll, 640.132324ms from cluster start. Timeout at 2025-01-26 02:44:50.149180239 +0000 UTC m=+86.664212614, 1.00500684s after stability
2025-01-26T02:44:50.175Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: starting snapshot up to: index=12
2025-01-26T02:44:50.175Z [INFO]  creating new snapshot: path=/tmp/raft1546061065/snapshots/2-12-1737859490175.tmp
2025-01-26T02:44:50.176Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: snapshot complete up to: index=12
2025-01-26T02:44:50.176Z [INFO]  creating new snapshot: path=/tmp/raft1546061065/snapshots/2-10013-1737859490176.tmp
2025-01-26T02:44:50.176Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: copied to local snapshot: bytes=71
2025-01-26T02:44:50.176Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: snapshot restore progress: id=2-10013-1737859490176 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:50.176Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: restored user snapshot: index=10013
2025-01-26T02:44:50.176Z [ERROR] server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: failed to get log: index=23 error="log not found"
2025-01-26T02:44:50.176Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: aborting pipeline replication: peer="{Voter server-866b762d-23e5-27e5-1204-9db63d55d6da 866b762d-23e5-27e5-1204-9db63d55d6da}"
2025-01-26T02:44:50.176Z [ERROR] server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: failed to get log: index=23 error="log not found"
2025-01-26T02:44:50.176Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: aborting pipeline replication: peer="{Voter server-22c5b97f-3c38-682b-1c2c-e23e3590aa31 22c5b97f-3c38-682b-1c2c-e23e3590aa31}"
2025-01-26T02:44:50.183Z [ERROR] server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: failed to get log: index=23 error="log not found"
2025-01-26T02:44:50.184Z [INFO]  creating new snapshot: path=/tmp/raft3574023023/snapshots/2-10013-1737859490184.tmp
2025-01-26T02:44:50.184Z [INFO]  server-22c5b97f-3c38-682b-1c2c-e23e3590aa31: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:50.184Z [INFO]  server-22c5b97f-3c38-682b-1c2c-e23e3590aa31: copied to local snapshot: bytes=71
2025-01-26T02:44:50.184Z [INFO]  server-22c5b97f-3c38-682b-1c2c-e23e3590aa31: snapshot restore progress: id=2-10013-1737859490184 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:50.184Z [INFO]  server-22c5b97f-3c38-682b-1c2c-e23e3590aa31: Installed remote snapshot
2025-01-26T02:44:50.184Z [WARN]  server-22c5b97f-3c38-682b-1c2c-e23e3590aa31: skipping application of old log: index=22
2025-01-26T02:44:50.184Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: pipelining replication: peer="{Voter server-22c5b97f-3c38-682b-1c2c-e23e3590aa31 22c5b97f-3c38-682b-1c2c-e23e3590aa31}"
2025-01-26T02:44:50.184Z [ERROR] server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: failed to get log: index=23 error="log not found"
2025-01-26T02:44:50.184Z [INFO]  creating new snapshot: path=/tmp/raft1682571683/snapshots/2-10013-1737859490184.tmp
2025-01-26T02:44:50.184Z [INFO]  server-866b762d-23e5-27e5-1204-9db63d55d6da: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:50.184Z [INFO]  server-866b762d-23e5-27e5-1204-9db63d55d6da: copied to local snapshot: bytes=71
2025-01-26T02:44:50.184Z [INFO]  server-866b762d-23e5-27e5-1204-9db63d55d6da: snapshot restore progress: id=2-10013-1737859490184 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:50.184Z [INFO]  server-866b762d-23e5-27e5-1204-9db63d55d6da: Installed remote snapshot
2025-01-26T02:44:50.185Z [WARN]  server-866b762d-23e5-27e5-1204-9db63d55d6da: skipping application of old log: index=22
2025-01-26T02:44:50.185Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: pipelining replication: peer="{Voter server-866b762d-23e5-27e5-1204-9db63d55d6da 866b762d-23e5-27e5-1204-9db63d55d6da}"
2025-01-26T02:44:50.185Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:44:51.191Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:44:50.185124875 +0000 UTC m=+86.700157210 (2 nodes), 600ns from start of poll, 1.68108376s from cluster start. Timeout at 2025-01-26 02:44:51.190128915 +0000 UTC m=+87.705161250, 1.00500404s after stability
2025-01-26T02:44:51.207Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: aborting pipeline replication: peer="{Voter server-866b762d-23e5-27e5-1204-9db63d55d6da 866b762d-23e5-27e5-1204-9db63d55d6da}"
2025-01-26T02:44:51.207Z [INFO]  server-1a0277b0-985d-cb4f-8f71-26d0ecbc8c5c: aborting pipeline replication: peer="{Voter server-22c5b97f-3c38-682b-1c2c-e23e3590aa31 22c5b97f-3c38-682b-1c2c-e23e3590aa31}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false
2025-01-26T02:44:51.208Z [INFO]  server-a12a9032-e106-4724-8074-35c364c81ae3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a12a9032-e106-4724-8074-35c364c81ae3 Address:a12a9032-e106-4724-8074-35c364c81ae3} {Suffrage:Voter ID:server-c1620186-0412-53f5-8457-b408413e7616 Address:c1620186-0412-53f5-8457-b408413e7616} {Suffrage:Voter ID:server-eb8c60fb-a912-3bd1-3851-33473c3e3355 Address:eb8c60fb-a912-3bd1-3851-33473c3e3355}]"
2025-01-26T02:44:51.208Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a12a9032-e106-4724-8074-35c364c81ae3 Address:a12a9032-e106-4724-8074-35c364c81ae3} {Suffrage:Voter ID:server-c1620186-0412-53f5-8457-b408413e7616 Address:c1620186-0412-53f5-8457-b408413e7616} {Suffrage:Voter ID:server-eb8c60fb-a912-3bd1-3851-33473c3e3355 Address:eb8c60fb-a912-3bd1-3851-33473c3e3355}]"
2025-01-26T02:44:51.208Z [INFO]  server-eb8c60fb-a912-3bd1-3851-33473c3e3355: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a12a9032-e106-4724-8074-35c364c81ae3 Address:a12a9032-e106-4724-8074-35c364c81ae3} {Suffrage:Voter ID:server-c1620186-0412-53f5-8457-b408413e7616 Address:c1620186-0412-53f5-8457-b408413e7616} {Suffrage:Voter ID:server-eb8c60fb-a912-3bd1-3851-33473c3e3355 Address:eb8c60fb-a912-3bd1-3851-33473c3e3355}]"
2025-01-26T02:44:51.208Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:51.208Z [INFO]  server-a12a9032-e106-4724-8074-35c364c81ae3: entering follower state: follower="Node at a12a9032-e106-4724-8074-35c364c81ae3 [Follower]" leader-address= leader-id=
2025-01-26T02:44:51.208Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: entering follower state: follower="Node at c1620186-0412-53f5-8457-b408413e7616 [Follower]" leader-address= leader-id=
2025-01-26T02:44:51.208Z [INFO]  server-eb8c60fb-a912-3bd1-3851-33473c3e3355: entering follower state: follower="Node at eb8c60fb-a912-3bd1-3851-33473c3e3355 [Follower]" leader-address= leader-id=
2025-01-26T02:44:51.887Z [WARN]  server-c1620186-0412-53f5-8457-b408413e7616: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:51.887Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: entering candidate state: node="Node at c1620186-0412-53f5-8457-b408413e7616 [Candidate]" term=2
2025-01-26T02:44:51.887Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: election won: term=2 tally=2
2025-01-26T02:44:51.887Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: entering leader state: leader="Node at c1620186-0412-53f5-8457-b408413e7616 [Leader]"
2025-01-26T02:44:51.887Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: added peer, starting replication: peer=server-a12a9032-e106-4724-8074-35c364c81ae3
2025-01-26T02:44:51.887Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: added peer, starting replication: peer=server-eb8c60fb-a912-3bd1-3851-33473c3e3355
2025-01-26T02:44:51.887Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: pipelining replication: peer="{Voter server-eb8c60fb-a912-3bd1-3851-33473c3e3355 eb8c60fb-a912-3bd1-3851-33473c3e3355}"
2025-01-26T02:44:51.887Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: pipelining replication: peer="{Voter server-a12a9032-e106-4724-8074-35c364c81ae3 a12a9032-e106-4724-8074-35c364c81ae3}"
2025-01-26T02:44:52.904Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:51.888047008 +0000 UTC m=+88.403079383 (1 nodes), 679.613323ms from start of poll, 679.787164ms from cluster start. Timeout at 2025-01-26 02:44:52.893055208 +0000 UTC m=+89.408087583, 1.0050082s after stability
2025-01-26T02:44:52.911Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: starting snapshot up to: index=12
2025-01-26T02:44:52.911Z [INFO]  creating new snapshot: path=/tmp/raft3888556742/snapshots/2-12-1737859492911.tmp
2025-01-26T02:44:52.911Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: snapshot complete up to: index=12
2025-01-26T02:44:52.912Z [INFO]  creating new snapshot: path=/tmp/raft3888556742/snapshots/2-10013-1737859492912.tmp
2025-01-26T02:44:52.919Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: copied to local snapshot: bytes=71
2025-01-26T02:44:52.920Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: snapshot restore progress: id=2-10013-1737859492912 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:52.920Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: removing all old logs from log store
2025-01-26T02:44:52.920Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: compacting logs: from=1 to=22
2025-01-26T02:44:52.920Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: restored user snapshot: index=10013
2025-01-26T02:44:52.920Z [ERROR] server-c1620186-0412-53f5-8457-b408413e7616: failed to get log: index=22 error="log not found"
2025-01-26T02:44:52.920Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: aborting pipeline replication: peer="{Voter server-eb8c60fb-a912-3bd1-3851-33473c3e3355 eb8c60fb-a912-3bd1-3851-33473c3e3355}"
2025-01-26T02:44:52.920Z [ERROR] server-c1620186-0412-53f5-8457-b408413e7616: failed to get log: index=22 error="log not found"
2025-01-26T02:44:52.920Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: aborting pipeline replication: peer="{Voter server-a12a9032-e106-4724-8074-35c364c81ae3 a12a9032-e106-4724-8074-35c364c81ae3}"
2025-01-26T02:44:52.927Z [ERROR] server-c1620186-0412-53f5-8457-b408413e7616: failed to get log: index=22 error="log not found"
2025-01-26T02:44:52.927Z [INFO]  creating new snapshot: path=/tmp/raft2569490072/snapshots/2-10013-1737859492927.tmp
2025-01-26T02:44:52.928Z [INFO]  server-eb8c60fb-a912-3bd1-3851-33473c3e3355: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:52.928Z [INFO]  server-eb8c60fb-a912-3bd1-3851-33473c3e3355: copied to local snapshot: bytes=71
2025-01-26T02:44:52.928Z [INFO]  server-eb8c60fb-a912-3bd1-3851-33473c3e3355: snapshot restore progress: id=2-10013-1737859492927 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:52.928Z [INFO]  server-eb8c60fb-a912-3bd1-3851-33473c3e3355: removing all old logs from log store
2025-01-26T02:44:52.928Z [INFO]  server-eb8c60fb-a912-3bd1-3851-33473c3e3355: compacting logs: from=1 to=22
2025-01-26T02:44:52.928Z [INFO]  server-eb8c60fb-a912-3bd1-3851-33473c3e3355: Installed remote snapshot
2025-01-26T02:44:52.928Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: pipelining replication: peer="{Voter server-eb8c60fb-a912-3bd1-3851-33473c3e3355 eb8c60fb-a912-3bd1-3851-33473c3e3355}"
2025-01-26T02:44:52.928Z [ERROR] server-c1620186-0412-53f5-8457-b408413e7616: failed to get log: index=22 error="log not found"
2025-01-26T02:44:52.928Z [INFO]  creating new snapshot: path=/tmp/raft2611916408/snapshots/2-10013-1737859492928.tmp
2025-01-26T02:44:52.928Z [INFO]  server-a12a9032-e106-4724-8074-35c364c81ae3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:52.928Z [INFO]  server-a12a9032-e106-4724-8074-35c364c81ae3: copied to local snapshot: bytes=71
2025-01-26T02:44:52.928Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:44:52.928Z [INFO]  server-a12a9032-e106-4724-8074-35c364c81ae3: snapshot restore progress: id=2-10013-1737859492928 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:52.928Z [INFO]  server-a12a9032-e106-4724-8074-35c364c81ae3: removing all old logs from log store
2025-01-26T02:44:52.928Z [INFO]  server-a12a9032-e106-4724-8074-35c364c81ae3: compacting logs: from=1 to=22
2025-01-26T02:44:52.928Z [INFO]  server-a12a9032-e106-4724-8074-35c364c81ae3: Installed remote snapshot
2025-01-26T02:44:52.928Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: pipelining replication: peer="{Voter server-a12a9032-e106-4724-8074-35c364c81ae3 a12a9032-e106-4724-8074-35c364c81ae3}"
2025-01-26T02:44:53.935Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:44:52.928737403 +0000 UTC m=+89.443769738 (2 nodes), 520ns from start of poll, 1.720477519s from cluster start. Timeout at 2025-01-26 02:44:53.933759483 +0000 UTC m=+90.448791858, 1.00502212s after stability
2025-01-26T02:44:53.951Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: aborting pipeline replication: peer="{Voter server-eb8c60fb-a912-3bd1-3851-33473c3e3355 eb8c60fb-a912-3bd1-3851-33473c3e3355}"
2025-01-26T02:44:53.951Z [INFO]  server-c1620186-0412-53f5-8457-b408413e7616: aborting pipeline replication: peer="{Voter server-a12a9032-e106-4724-8074-35c364c81ae3 a12a9032-e106-4724-8074-35c364c81ae3}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true
2025-01-26T02:44:53.952Z [INFO]  server-ef5aed55-ba74-2ab6-c5d1-7bd699aa665f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef5aed55-ba74-2ab6-c5d1-7bd699aa665f Address:ef5aed55-ba74-2ab6-c5d1-7bd699aa665f} {Suffrage:Voter ID:server-71f51baf-5dd4-dd33-2035-3d99945b37e0 Address:71f51baf-5dd4-dd33-2035-3d99945b37e0} {Suffrage:Voter ID:server-28fd144f-f59f-c9a3-7b7b-b7782ac46a00 Address:28fd144f-f59f-c9a3-7b7b-b7782ac46a00}]"
2025-01-26T02:44:53.952Z [INFO]  server-71f51baf-5dd4-dd33-2035-3d99945b37e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef5aed55-ba74-2ab6-c5d1-7bd699aa665f Address:ef5aed55-ba74-2ab6-c5d1-7bd699aa665f} {Suffrage:Voter ID:server-71f51baf-5dd4-dd33-2035-3d99945b37e0 Address:71f51baf-5dd4-dd33-2035-3d99945b37e0} {Suffrage:Voter ID:server-28fd144f-f59f-c9a3-7b7b-b7782ac46a00 Address:28fd144f-f59f-c9a3-7b7b-b7782ac46a00}]"
2025-01-26T02:44:53.952Z [INFO]  server-28fd144f-f59f-c9a3-7b7b-b7782ac46a00: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ef5aed55-ba74-2ab6-c5d1-7bd699aa665f Address:ef5aed55-ba74-2ab6-c5d1-7bd699aa665f} {Suffrage:Voter ID:server-71f51baf-5dd4-dd33-2035-3d99945b37e0 Address:71f51baf-5dd4-dd33-2035-3d99945b37e0} {Suffrage:Voter ID:server-28fd144f-f59f-c9a3-7b7b-b7782ac46a00 Address:28fd144f-f59f-c9a3-7b7b-b7782ac46a00}]"
2025-01-26T02:44:53.952Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:53.952Z [INFO]  server-ef5aed55-ba74-2ab6-c5d1-7bd699aa665f: entering follower state: follower="Node at ef5aed55-ba74-2ab6-c5d1-7bd699aa665f [Follower]" leader-address= leader-id=
2025-01-26T02:44:53.952Z [INFO]  server-71f51baf-5dd4-dd33-2035-3d99945b37e0: entering follower state: follower="Node at 71f51baf-5dd4-dd33-2035-3d99945b37e0 [Follower]" leader-address= leader-id=
2025-01-26T02:44:53.952Z [INFO]  server-28fd144f-f59f-c9a3-7b7b-b7782ac46a00: entering follower state: follower="Node at 28fd144f-f59f-c9a3-7b7b-b7782ac46a00 [Follower]" leader-address= leader-id=
2025-01-26T02:44:54.491Z [WARN]  server-ef5aed55-ba74-2ab6-c5d1-7bd699aa665f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:54.491Z [INFO]  server-ef5aed55-ba74-2ab6-c5d1-7bd699aa665f: entering candidate state: node="Node at ef5aed55-ba74-2ab6-c5d1-7bd699aa665f [Candidate]" term=2
2025-01-26T02:44:54.491Z [INFO]  server-ef5aed55-ba74-2ab6-c5d1-7bd699aa665f: election won: term=2 tally=2
2025-01-26T02:44:54.491Z [INFO]  server-ef5aed55-ba74-2ab6-c5d1-7bd699aa665f: entering leader state: leader="Node at ef5aed55-ba74-2ab6-c5d1-7bd699aa665f [Leader]"
2025-01-26T02:44:54.491Z [INFO]  server-ef5aed55-ba74-2ab6-c5d1-7bd699aa665f: added peer, starting replication: peer=server-71f51baf-5dd4-dd33-2035-3d99945b37e0
2025-01-26T02:44:54.491Z [INFO]  server-ef5aed55-ba74-2ab6-c5d1-7bd699aa665f: added peer, starting replication: peer=server-28fd144f-f59f-c9a3-7b7b-b7782ac46a00
2025-01-26T02:44:54.491Z [INFO]  server-ef5aed55-ba74-2ab6-c5d1-7bd699aa665f: pipelining replication: peer="{Voter server-28fd144f-f59f-c9a3-7b7b-b7782ac46a00 28fd144f-f59f-c9a3-7b7b-b7782ac46a00}"
2025-01-26T02:44:54.491Z [INFO]  server-ef5aed55-ba74-2ab6-c5d1-7bd699aa665f: pipelining replication: peer="{Voter server-71f51baf-5dd4-dd33-2035-3d99945b37e0 71f51baf-5dd4-dd33-2035-3d99945b37e0}"
2025-01-26T02:44:55.499Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:54.491936169 +0000 UTC m=+91.006968504 (1 nodes), 539.719315ms from start of poll, 539.892916ms from cluster start. Timeout at 2025-01-26 02:44:55.496956209 +0000 UTC m=+92.011988584, 1.00502008s after stability
2025-01-26T02:44:55.499Z [INFO]  server-ef5aed55-ba74-2ab6-c5d1-7bd699aa665f: starting snapshot up to: index=12
2025-01-26T02:44:55.499Z [INFO]  creating new snapshot: path=/tmp/raft2029209895/snapshots/2-12-1737859495499.tmp
2025-01-26T02:44:55.500Z [INFO]  server-ef5aed55-ba74-2ab6-c5d1-7bd699aa665f: snapshot complete up to: index=12
2025-01-26T02:44:55.500Z [INFO]  server-6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7 Address:6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7} {Suffrage:Voter ID:server-d51dd76c-4657-99d3-1785-78753a79ecb6 Address:d51dd76c-4657-99d3-1785-78753a79ecb6} {Suffrage:Voter ID:server-39d27254-6580-d28a-c1a0-02422658be9d Address:39d27254-6580-d28a-c1a0-02422658be9d}]"
2025-01-26T02:44:55.500Z [INFO]  server-d51dd76c-4657-99d3-1785-78753a79ecb6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7 Address:6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7} {Suffrage:Voter ID:server-d51dd76c-4657-99d3-1785-78753a79ecb6 Address:d51dd76c-4657-99d3-1785-78753a79ecb6} {Suffrage:Voter ID:server-39d27254-6580-d28a-c1a0-02422658be9d Address:39d27254-6580-d28a-c1a0-02422658be9d}]"
2025-01-26T02:44:55.500Z [INFO]  server-39d27254-6580-d28a-c1a0-02422658be9d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7 Address:6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7} {Suffrage:Voter ID:server-d51dd76c-4657-99d3-1785-78753a79ecb6 Address:d51dd76c-4657-99d3-1785-78753a79ecb6} {Suffrage:Voter ID:server-39d27254-6580-d28a-c1a0-02422658be9d Address:39d27254-6580-d28a-c1a0-02422658be9d}]"
2025-01-26T02:44:55.500Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:55.500Z [INFO]  server-6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7: entering follower state: follower="Node at 6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7 [Follower]" leader-address= leader-id=
2025-01-26T02:44:55.500Z [INFO]  server-d51dd76c-4657-99d3-1785-78753a79ecb6: entering follower state: follower="Node at d51dd76c-4657-99d3-1785-78753a79ecb6 [Follower]" leader-address= leader-id=
2025-01-26T02:44:55.500Z [INFO]  server-39d27254-6580-d28a-c1a0-02422658be9d: entering follower state: follower="Node at 39d27254-6580-d28a-c1a0-02422658be9d [Follower]" leader-address= leader-id=
2025-01-26T02:44:56.011Z [WARN]  server-d51dd76c-4657-99d3-1785-78753a79ecb6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:56.011Z [INFO]  server-d51dd76c-4657-99d3-1785-78753a79ecb6: entering candidate state: node="Node at d51dd76c-4657-99d3-1785-78753a79ecb6 [Candidate]" term=2
2025-01-26T02:44:56.011Z [INFO]  server-d51dd76c-4657-99d3-1785-78753a79ecb6: election won: term=2 tally=2
2025-01-26T02:44:56.011Z [INFO]  server-d51dd76c-4657-99d3-1785-78753a79ecb6: entering leader state: leader="Node at d51dd76c-4657-99d3-1785-78753a79ecb6 [Leader]"
2025-01-26T02:44:56.011Z [INFO]  server-d51dd76c-4657-99d3-1785-78753a79ecb6: added peer, starting replication: peer=server-6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7
2025-01-26T02:44:56.011Z [INFO]  server-d51dd76c-4657-99d3-1785-78753a79ecb6: added peer, starting replication: peer=server-39d27254-6580-d28a-c1a0-02422658be9d
2025-01-26T02:44:56.011Z [INFO]  server-d51dd76c-4657-99d3-1785-78753a79ecb6: pipelining replication: peer="{Voter server-39d27254-6580-d28a-c1a0-02422658be9d 39d27254-6580-d28a-c1a0-02422658be9d}"
2025-01-26T02:44:56.011Z [WARN]  server-39d27254-6580-d28a-c1a0-02422658be9d: rejecting vote request since our last term is greater: candidate=d51dd76c-4657-99d3-1785-78753a79ecb6 last-term=2 last-candidate-term=1
2025-01-26T02:44:56.011Z [INFO]  server-d51dd76c-4657-99d3-1785-78753a79ecb6: pipelining replication: peer="{Voter server-6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7 6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7}"
2025-01-26T02:44:57.016Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:56.011814751 +0000 UTC m=+92.526847126 (1 nodes), 511.007141ms from start of poll, 511.182262ms from cluster start. Timeout at 2025-01-26 02:44:57.016830351 +0000 UTC m=+93.531862686, 1.00501556s after stability
2025-01-26T02:44:57.017Z [INFO]  creating new snapshot: path=/tmp/raft249701129/snapshots/2-10013-1737859497017.tmp
2025-01-26T02:44:57.017Z [INFO]  server-d51dd76c-4657-99d3-1785-78753a79ecb6: copied to local snapshot: bytes=71
2025-01-26T02:44:57.017Z [INFO]  server-d51dd76c-4657-99d3-1785-78753a79ecb6: snapshot restore progress: id=2-10013-1737859497017 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:57.017Z [INFO]  server-d51dd76c-4657-99d3-1785-78753a79ecb6: restored user snapshot: index=10013
2025-01-26T02:44:57.017Z [ERROR] server-d51dd76c-4657-99d3-1785-78753a79ecb6: failed to get log: index=3 error="log not found"
2025-01-26T02:44:57.017Z [INFO]  server-d51dd76c-4657-99d3-1785-78753a79ecb6: aborting pipeline replication: peer="{Voter server-39d27254-6580-d28a-c1a0-02422658be9d 39d27254-6580-d28a-c1a0-02422658be9d}"
2025-01-26T02:44:57.017Z [ERROR] server-d51dd76c-4657-99d3-1785-78753a79ecb6: failed to get log: index=3 error="log not found"
2025-01-26T02:44:57.017Z [INFO]  server-d51dd76c-4657-99d3-1785-78753a79ecb6: aborting pipeline replication: peer="{Voter server-6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7 6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7}"
2025-01-26T02:44:57.017Z [ERROR] server-d51dd76c-4657-99d3-1785-78753a79ecb6: failed to get log: index=3 error="log not found"
2025-01-26T02:44:57.017Z [INFO]  creating new snapshot: path=/tmp/raft2895506473/snapshots/2-10013-1737859497017.tmp
2025-01-26T02:44:57.017Z [INFO]  server-6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:57.017Z [INFO]  server-6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7: copied to local snapshot: bytes=71
2025-01-26T02:44:57.017Z [INFO]  server-6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7: snapshot restore progress: id=2-10013-1737859497017 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:57.018Z [INFO]  server-6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7: Installed remote snapshot
2025-01-26T02:44:57.018Z [INFO]  server-d51dd76c-4657-99d3-1785-78753a79ecb6: pipelining replication: peer="{Voter server-6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7 6e4c48c3-9af2-06fa-7cf1-1cc22bd350e7}"
2025-01-26T02:44:57.018Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:44:57.027Z [ERROR] server-d51dd76c-4657-99d3-1785-78753a79ecb6: failed to get log: index=3 error="log not found"
2025-01-26T02:44:57.027Z [INFO]  creating new snapshot: path=/tmp/raft3598081988/snapshots/2-10013-1737859497027.tmp
2025-01-26T02:44:57.028Z [INFO]  server-39d27254-6580-d28a-c1a0-02422658be9d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:57.028Z [INFO]  server-39d27254-6580-d28a-c1a0-02422658be9d: copied to local snapshot: bytes=71
2025-01-26T02:44:57.028Z [INFO]  server-39d27254-6580-d28a-c1a0-02422658be9d: snapshot restore progress: id=2-10013-1737859497027 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:44:57.028Z [INFO]  server-39d27254-6580-d28a-c1a0-02422658be9d: Installed remote snapshot
2025-01-26T02:44:57.028Z [INFO]  server-d51dd76c-4657-99d3-1785-78753a79ecb6: pipelining replication: peer="{Voter server-39d27254-6580-d28a-c1a0-02422658be9d 39d27254-6580-d28a-c1a0-02422658be9d}"
2025-01-26T02:44:58.023Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:44:57.018230738 +0000 UTC m=+93.533263113 (2 nodes), 520ns from start of poll, 1.517598249s from cluster start. Timeout at 2025-01-26 02:44:58.023259258 +0000 UTC m=+94.538291633, 1.00502852s after stability
2025-01-26T02:44:58.039Z [INFO]  server-ef5aed55-ba74-2ab6-c5d1-7bd699aa665f: aborting pipeline replication: peer="{Voter server-71f51baf-5dd4-dd33-2035-3d99945b37e0 71f51baf-5dd4-dd33-2035-3d99945b37e0}"
2025-01-26T02:44:58.039Z [INFO]  server-ef5aed55-ba74-2ab6-c5d1-7bd699aa665f: aborting pipeline replication: peer="{Voter server-28fd144f-f59f-c9a3-7b7b-b7782ac46a00 28fd144f-f59f-c9a3-7b7b-b7782ac46a00}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true
2025-01-26T02:44:58.040Z [INFO]  server-33703acb-1fc9-d6f5-f657-cc6766e193da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33703acb-1fc9-d6f5-f657-cc6766e193da Address:33703acb-1fc9-d6f5-f657-cc6766e193da} {Suffrage:Voter ID:server-f280e300-187c-7942-ef1f-d01ec043a1b8 Address:f280e300-187c-7942-ef1f-d01ec043a1b8} {Suffrage:Voter ID:server-83a204f6-56bc-0c6b-f11c-d25fcbbe6996 Address:83a204f6-56bc-0c6b-f11c-d25fcbbe6996}]"
2025-01-26T02:44:58.040Z [INFO]  server-f280e300-187c-7942-ef1f-d01ec043a1b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33703acb-1fc9-d6f5-f657-cc6766e193da Address:33703acb-1fc9-d6f5-f657-cc6766e193da} {Suffrage:Voter ID:server-f280e300-187c-7942-ef1f-d01ec043a1b8 Address:f280e300-187c-7942-ef1f-d01ec043a1b8} {Suffrage:Voter ID:server-83a204f6-56bc-0c6b-f11c-d25fcbbe6996 Address:83a204f6-56bc-0c6b-f11c-d25fcbbe6996}]"
2025-01-26T02:44:58.040Z [INFO]  server-83a204f6-56bc-0c6b-f11c-d25fcbbe6996: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33703acb-1fc9-d6f5-f657-cc6766e193da Address:33703acb-1fc9-d6f5-f657-cc6766e193da} {Suffrage:Voter ID:server-f280e300-187c-7942-ef1f-d01ec043a1b8 Address:f280e300-187c-7942-ef1f-d01ec043a1b8} {Suffrage:Voter ID:server-83a204f6-56bc-0c6b-f11c-d25fcbbe6996 Address:83a204f6-56bc-0c6b-f11c-d25fcbbe6996}]"
2025-01-26T02:44:58.040Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:58.040Z [INFO]  server-33703acb-1fc9-d6f5-f657-cc6766e193da: entering follower state: follower="Node at 33703acb-1fc9-d6f5-f657-cc6766e193da [Follower]" leader-address= leader-id=
2025-01-26T02:44:58.040Z [INFO]  server-f280e300-187c-7942-ef1f-d01ec043a1b8: entering follower state: follower="Node at f280e300-187c-7942-ef1f-d01ec043a1b8 [Follower]" leader-address= leader-id=
2025-01-26T02:44:58.040Z [INFO]  server-83a204f6-56bc-0c6b-f11c-d25fcbbe6996: entering follower state: follower="Node at 83a204f6-56bc-0c6b-f11c-d25fcbbe6996 [Follower]" leader-address= leader-id=
2025-01-26T02:44:58.643Z [WARN]  server-83a204f6-56bc-0c6b-f11c-d25fcbbe6996: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:44:58.643Z [INFO]  server-83a204f6-56bc-0c6b-f11c-d25fcbbe6996: entering candidate state: node="Node at 83a204f6-56bc-0c6b-f11c-d25fcbbe6996 [Candidate]" term=2
2025-01-26T02:44:58.643Z [INFO]  server-83a204f6-56bc-0c6b-f11c-d25fcbbe6996: election won: term=2 tally=2
2025-01-26T02:44:58.643Z [INFO]  server-83a204f6-56bc-0c6b-f11c-d25fcbbe6996: entering leader state: leader="Node at 83a204f6-56bc-0c6b-f11c-d25fcbbe6996 [Leader]"
2025-01-26T02:44:58.643Z [INFO]  server-83a204f6-56bc-0c6b-f11c-d25fcbbe6996: added peer, starting replication: peer=server-33703acb-1fc9-d6f5-f657-cc6766e193da
2025-01-26T02:44:58.643Z [INFO]  server-83a204f6-56bc-0c6b-f11c-d25fcbbe6996: added peer, starting replication: peer=server-f280e300-187c-7942-ef1f-d01ec043a1b8
2025-01-26T02:44:58.643Z [INFO]  server-83a204f6-56bc-0c6b-f11c-d25fcbbe6996: pipelining replication: peer="{Voter server-f280e300-187c-7942-ef1f-d01ec043a1b8 f280e300-187c-7942-ef1f-d01ec043a1b8}"
2025-01-26T02:44:58.643Z [WARN]  server-f280e300-187c-7942-ef1f-d01ec043a1b8: rejecting vote request since our last term is greater: candidate=83a204f6-56bc-0c6b-f11c-d25fcbbe6996 last-term=2 last-candidate-term=1
2025-01-26T02:44:58.643Z [INFO]  server-83a204f6-56bc-0c6b-f11c-d25fcbbe6996: pipelining replication: peer="{Voter server-33703acb-1fc9-d6f5-f657-cc6766e193da 33703acb-1fc9-d6f5-f657-cc6766e193da}"
2025-01-26T02:44:59.651Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:44:58.643757522 +0000 UTC m=+95.158789897 (1 nodes), 603.629863ms from start of poll, 603.818944ms from cluster start. Timeout at 2025-01-26 02:44:59.648761602 +0000 UTC m=+96.163793977, 1.00500408s after stability
2025-01-26T02:44:59.652Z [INFO]  server-83a204f6-56bc-0c6b-f11c-d25fcbbe6996: starting snapshot up to: index=12
2025-01-26T02:44:59.652Z [INFO]  creating new snapshot: path=/tmp/raft3490641574/snapshots/2-12-1737859499652.tmp
2025-01-26T02:44:59.652Z [INFO]  server-83a204f6-56bc-0c6b-f11c-d25fcbbe6996: snapshot complete up to: index=12
2025-01-26T02:44:59.652Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24349f29-f4a4-f744-3718-f911bd92ae73 Address:24349f29-f4a4-f744-3718-f911bd92ae73} {Suffrage:Voter ID:server-b3cdc32b-536d-0a64-b4a4-3db9c004365b Address:b3cdc32b-536d-0a64-b4a4-3db9c004365b} {Suffrage:Voter ID:server-c6afb55f-5cb5-ab02-7f6a-98539e3c4ace Address:c6afb55f-5cb5-ab02-7f6a-98539e3c4ace}]"
2025-01-26T02:44:59.652Z [INFO]  server-b3cdc32b-536d-0a64-b4a4-3db9c004365b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24349f29-f4a4-f744-3718-f911bd92ae73 Address:24349f29-f4a4-f744-3718-f911bd92ae73} {Suffrage:Voter ID:server-b3cdc32b-536d-0a64-b4a4-3db9c004365b Address:b3cdc32b-536d-0a64-b4a4-3db9c004365b} {Suffrage:Voter ID:server-c6afb55f-5cb5-ab02-7f6a-98539e3c4ace Address:c6afb55f-5cb5-ab02-7f6a-98539e3c4ace}]"
2025-01-26T02:44:59.652Z [INFO]  server-c6afb55f-5cb5-ab02-7f6a-98539e3c4ace: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-24349f29-f4a4-f744-3718-f911bd92ae73 Address:24349f29-f4a4-f744-3718-f911bd92ae73} {Suffrage:Voter ID:server-b3cdc32b-536d-0a64-b4a4-3db9c004365b Address:b3cdc32b-536d-0a64-b4a4-3db9c004365b} {Suffrage:Voter ID:server-c6afb55f-5cb5-ab02-7f6a-98539e3c4ace Address:c6afb55f-5cb5-ab02-7f6a-98539e3c4ace}]"
2025-01-26T02:44:59.653Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:44:59.653Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: entering follower state: follower="Node at 24349f29-f4a4-f744-3718-f911bd92ae73 [Follower]" leader-address= leader-id=
2025-01-26T02:44:59.653Z [INFO]  server-b3cdc32b-536d-0a64-b4a4-3db9c004365b: entering follower state: follower="Node at b3cdc32b-536d-0a64-b4a4-3db9c004365b [Follower]" leader-address= leader-id=
2025-01-26T02:44:59.653Z [INFO]  server-c6afb55f-5cb5-ab02-7f6a-98539e3c4ace: entering follower state: follower="Node at c6afb55f-5cb5-ab02-7f6a-98539e3c4ace [Follower]" leader-address= leader-id=
2025-01-26T02:45:00.159Z [WARN]  server-24349f29-f4a4-f744-3718-f911bd92ae73: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:00.159Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: entering candidate state: node="Node at 24349f29-f4a4-f744-3718-f911bd92ae73 [Candidate]" term=2
2025-01-26T02:45:00.159Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: election won: term=2 tally=2
2025-01-26T02:45:00.159Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: entering leader state: leader="Node at 24349f29-f4a4-f744-3718-f911bd92ae73 [Leader]"
2025-01-26T02:45:00.159Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: added peer, starting replication: peer=server-b3cdc32b-536d-0a64-b4a4-3db9c004365b
2025-01-26T02:45:00.159Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: added peer, starting replication: peer=server-c6afb55f-5cb5-ab02-7f6a-98539e3c4ace
2025-01-26T02:45:00.160Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: pipelining replication: peer="{Voter server-c6afb55f-5cb5-ab02-7f6a-98539e3c4ace c6afb55f-5cb5-ab02-7f6a-98539e3c4ace}"
2025-01-26T02:45:00.160Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: pipelining replication: peer="{Voter server-b3cdc32b-536d-0a64-b4a4-3db9c004365b b3cdc32b-536d-0a64-b4a4-3db9c004365b}"
2025-01-26T02:45:01.167Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:00.160188723 +0000 UTC m=+96.675221098 (1 nodes), 507.167837ms from start of poll, 507.372199ms from cluster start. Timeout at 2025-01-26 02:45:01.165189643 +0000 UTC m=+97.680221978, 1.00500088s after stability
2025-01-26T02:45:01.167Z [INFO]  creating new snapshot: path=/tmp/raft2795796811/snapshots/2-10013-1737859501167.tmp
2025-01-26T02:45:01.168Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: copied to local snapshot: bytes=71
2025-01-26T02:45:01.168Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: snapshot restore progress: id=2-10013-1737859501167 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:45:01.168Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: removing all old logs from log store
2025-01-26T02:45:01.168Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: compacting logs: from=1 to=2
2025-01-26T02:45:01.168Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: restored user snapshot: index=10013
2025-01-26T02:45:01.168Z [ERROR] server-24349f29-f4a4-f744-3718-f911bd92ae73: failed to get log: index=2 error="log not found"
2025-01-26T02:45:01.168Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: aborting pipeline replication: peer="{Voter server-c6afb55f-5cb5-ab02-7f6a-98539e3c4ace c6afb55f-5cb5-ab02-7f6a-98539e3c4ace}"
2025-01-26T02:45:01.168Z [ERROR] server-24349f29-f4a4-f744-3718-f911bd92ae73: failed to get log: index=2 error="log not found"
2025-01-26T02:45:01.168Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: aborting pipeline replication: peer="{Voter server-b3cdc32b-536d-0a64-b4a4-3db9c004365b b3cdc32b-536d-0a64-b4a4-3db9c004365b}"
2025-01-26T02:45:01.176Z [ERROR] server-24349f29-f4a4-f744-3718-f911bd92ae73: failed to get log: index=2 error="log not found"
2025-01-26T02:45:01.176Z [INFO]  creating new snapshot: path=/tmp/raft684120119/snapshots/2-10013-1737859501176.tmp
2025-01-26T02:45:01.176Z [INFO]  server-c6afb55f-5cb5-ab02-7f6a-98539e3c4ace: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:45:01.176Z [ERROR] server-24349f29-f4a4-f744-3718-f911bd92ae73: failed to get log: index=2 error="log not found"
2025-01-26T02:45:01.176Z [INFO]  creating new snapshot: path=/tmp/raft2193180904/snapshots/2-10013-1737859501176.tmp
2025-01-26T02:45:01.176Z [INFO]  server-b3cdc32b-536d-0a64-b4a4-3db9c004365b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-26T02:45:01.176Z [INFO]  server-b3cdc32b-536d-0a64-b4a4-3db9c004365b: copied to local snapshot: bytes=71
2025-01-26T02:45:01.176Z [INFO]  server-b3cdc32b-536d-0a64-b4a4-3db9c004365b: snapshot restore progress: id=2-10013-1737859501176 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:45:01.176Z [INFO]  server-b3cdc32b-536d-0a64-b4a4-3db9c004365b: removing all old logs from log store
2025-01-26T02:45:01.176Z [INFO]  server-b3cdc32b-536d-0a64-b4a4-3db9c004365b: compacting logs: from=1 to=2
2025-01-26T02:45:01.176Z [INFO]  server-b3cdc32b-536d-0a64-b4a4-3db9c004365b: Installed remote snapshot
2025-01-26T02:45:01.177Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: pipelining replication: peer="{Voter server-b3cdc32b-536d-0a64-b4a4-3db9c004365b b3cdc32b-536d-0a64-b4a4-3db9c004365b}"
2025-01-26T02:45:01.177Z [INFO]  server-c6afb55f-5cb5-ab02-7f6a-98539e3c4ace: copied to local snapshot: bytes=71
2025-01-26T02:45:01.177Z [INFO]  server-c6afb55f-5cb5-ab02-7f6a-98539e3c4ace: snapshot restore progress: id=2-10013-1737859501176 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-26T02:45:01.177Z [INFO]  server-c6afb55f-5cb5-ab02-7f6a-98539e3c4ace: removing all old logs from log store
2025-01-26T02:45:01.177Z [INFO]  server-c6afb55f-5cb5-ab02-7f6a-98539e3c4ace: compacting logs: from=1 to=2
2025-01-26T02:45:01.177Z [INFO]  server-c6afb55f-5cb5-ab02-7f6a-98539e3c4ace: Installed remote snapshot
2025-01-26T02:45:01.177Z [INFO]  server-24349f29-f4a4-f744-3718-f911bd92ae73: pipelining replication: peer="{Voter server-c6afb55f-5cb5-ab02-7f6a-98539e3c4ace c6afb55f-5cb5-ab02-7f6a-98539e3c4ace}"
2025-01-26T02:45:01.177Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:02.183Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:01.177367495 +0000 UTC m=+97.692399870 (2 nodes), 480ns from start of poll, 1.524550971s from cluster start. Timeout at 2025-01-26 02:45:02.182370135 +0000 UTC m=+98.697402510, 1.00500264s after stability
2025-01-26T02:45:02.199Z [INFO]  server-83a204f6-56bc-0c6b-f11c-d25fcbbe6996: aborting pipeline replication: peer="{Voter server-f280e300-187c-7942-ef1f-d01ec043a1b8 f280e300-187c-7942-ef1f-d01ec043a1b8}"
2025-01-26T02:45:02.199Z [INFO]  server-83a204f6-56bc-0c6b-f11c-d25fcbbe6996: aborting pipeline replication: peer="{Voter server-33703acb-1fc9-d6f5-f657-cc6766e193da 33703acb-1fc9-d6f5-f657-cc6766e193da}"
--- PASS: TestRaft_UserRestore (83.75s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.74s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.62s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.48s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.37s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.67s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.67s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.38s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.41s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.84s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.82s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.19s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.19s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.70s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.58s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.36s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.16s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.27s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.39s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.70s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.74s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.09s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.16s)
=== RUN   TestRaft_SendSnapshotFollower
2025-01-26T02:45:02.200Z [INFO]  server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c148ed2-b03f-85d0-70ca-fe95fc89f924 Address:4c148ed2-b03f-85d0-70ca-fe95fc89f924} {Suffrage:Voter ID:server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb Address:b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb} {Suffrage:Voter ID:server-f033ded6-a2e9-9060-55f1-3c0a2d0c633d Address:f033ded6-a2e9-9060-55f1-3c0a2d0c633d}]"
2025-01-26T02:45:02.200Z [INFO]  server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c148ed2-b03f-85d0-70ca-fe95fc89f924 Address:4c148ed2-b03f-85d0-70ca-fe95fc89f924} {Suffrage:Voter ID:server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb Address:b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb} {Suffrage:Voter ID:server-f033ded6-a2e9-9060-55f1-3c0a2d0c633d Address:f033ded6-a2e9-9060-55f1-3c0a2d0c633d}]"
2025-01-26T02:45:02.200Z [INFO]  server-f033ded6-a2e9-9060-55f1-3c0a2d0c633d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4c148ed2-b03f-85d0-70ca-fe95fc89f924 Address:4c148ed2-b03f-85d0-70ca-fe95fc89f924} {Suffrage:Voter ID:server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb Address:b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb} {Suffrage:Voter ID:server-f033ded6-a2e9-9060-55f1-3c0a2d0c633d Address:f033ded6-a2e9-9060-55f1-3c0a2d0c633d}]"
2025-01-26T02:45:02.200Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:02.200Z [INFO]  server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: entering follower state: follower="Node at 4c148ed2-b03f-85d0-70ca-fe95fc89f924 [Follower]" leader-address= leader-id=
2025-01-26T02:45:02.200Z [INFO]  server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb: entering follower state: follower="Node at b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb [Follower]" leader-address= leader-id=
2025-01-26T02:45:02.200Z [INFO]  server-f033ded6-a2e9-9060-55f1-3c0a2d0c633d: entering follower state: follower="Node at f033ded6-a2e9-9060-55f1-3c0a2d0c633d [Follower]" leader-address= leader-id=
2025-01-26T02:45:02.271Z [WARN]  server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:02.271Z [INFO]  server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: entering candidate state: node="Node at 4c148ed2-b03f-85d0-70ca-fe95fc89f924 [Candidate]" term=2
2025-01-26T02:45:02.271Z [INFO]  server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: election won: term=2 tally=2
2025-01-26T02:45:02.271Z [INFO]  server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: entering leader state: leader="Node at 4c148ed2-b03f-85d0-70ca-fe95fc89f924 [Leader]"
2025-01-26T02:45:02.271Z [INFO]  server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: added peer, starting replication: peer=server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb
2025-01-26T02:45:02.271Z [INFO]  server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: added peer, starting replication: peer=server-f033ded6-a2e9-9060-55f1-3c0a2d0c633d
2025-01-26T02:45:02.271Z [INFO]  server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: pipelining replication: peer="{Voter server-f033ded6-a2e9-9060-55f1-3c0a2d0c633d f033ded6-a2e9-9060-55f1-3c0a2d0c633d}"
2025-01-26T02:45:02.271Z [INFO]  server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: pipelining replication: peer="{Voter server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb}"
2025-01-26T02:45:02.379Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:02.271905057 +0000 UTC m=+98.786937392 (2 nodes), 71.468954ms from start of poll, 71.671595ms from cluster start. Timeout at 2025-01-26 02:45:02.376908177 +0000 UTC m=+98.891940552, 105.00316ms after stability
2025-01-26T02:45:02.379Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:02.487Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:02.379664791 +0000 UTC m=+98.894697126 (1 nodes), 800ns from start of poll, 179.431329ms from cluster start. Timeout at 2025-01-26 02:45:02.484692231 +0000 UTC m=+98.999724646, 105.02752ms after stability
2025-01-26T02:45:02.487Z [ERROR] server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: failed to pipeline appendEntries: peer="{Voter server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb}" error="append pipeline closed"
2025-01-26T02:45:02.487Z [INFO]  server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: aborting pipeline replication: peer="{Voter server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb}"
2025-01-26T02:45:02.487Z [ERROR] server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: failed to appendEntries to: peer="{Voter server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb}" error="failed to connect to peer: b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb"
2025-01-26T02:45:02.488Z [ERROR] server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: failed to heartbeat to: peer=b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb backoff time=10ms error="failed to connect to peer: b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb"
    raft_test.go:1627: [INFO] Finished apply without behind follower
2025-01-26T02:45:02.490Z [INFO]  server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: starting snapshot up to: index=102
2025-01-26T02:45:02.490Z [INFO]  creating new snapshot: path=/tmp/raft3200268539/snapshots/2-102-1737859502490.tmp
2025-01-26T02:45:02.490Z [INFO]  server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: compacting logs: from=1 to=92
2025-01-26T02:45:02.490Z [INFO]  server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: snapshot complete up to: index=102
2025-01-26T02:45:02.490Z [ERROR] server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb: failed to take snapshot: error="nothing new to snapshot"
2025-01-26T02:45:02.490Z [INFO]  server-f033ded6-a2e9-9060-55f1-3c0a2d0c633d: starting snapshot up to: index=98
2025-01-26T02:45:02.490Z [INFO]  creating new snapshot: path=/tmp/raft3329194410/snapshots/2-98-1737859502490.tmp
2025-01-26T02:45:02.490Z [INFO]  server-f033ded6-a2e9-9060-55f1-3c0a2d0c633d: compacting logs: from=1 to=92
2025-01-26T02:45:02.490Z [INFO]  server-f033ded6-a2e9-9060-55f1-3c0a2d0c633d: snapshot complete up to: index=98
2025-01-26T02:45:02.499Z [ERROR] server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: failed to get log: index=2 error="log not found"
2025-01-26T02:45:02.499Z [INFO]  creating new snapshot: path=/tmp/raft706910992/snapshots/2-102-1737859502499.tmp
2025-01-26T02:45:02.499Z [INFO]  server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-01-26T02:45:02.500Z [INFO]  server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb: copied to local snapshot: bytes=693
2025-01-26T02:45:02.500Z [INFO]  server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb: snapshot restore progress: id=2-102-1737859502499 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-01-26T02:45:02.500Z [INFO]  server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb: Installed remote snapshot
2025-01-26T02:45:02.500Z [WARN]  server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb: skipping application of old log: index=102
2025-01-26T02:45:02.500Z [INFO]  server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: pipelining replication: peer="{Voter server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb}"
2025-01-26T02:45:02.507Z [INFO]  server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: aborting pipeline replication: peer="{Voter server-b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb b52f76e8-4e4e-9c9f-dcc7-1794c11dddeb}"
2025-01-26T02:45:02.507Z [INFO]  server-4c148ed2-b03f-85d0-70ca-fe95fc89f924: aborting pipeline replication: peer="{Voter server-f033ded6-a2e9-9060-55f1-3c0a2d0c633d f033ded6-a2e9-9060-55f1-3c0a2d0c633d}"
--- PASS: TestRaft_SendSnapshotFollower (0.31s)
=== RUN   TestRaft_SendSnapshotAndLogsFollower
2025-01-26T02:45:02.508Z [INFO]  server-84bbbba3-144d-cc08-443f-f24c5bdfedb3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84bbbba3-144d-cc08-443f-f24c5bdfedb3 Address:84bbbba3-144d-cc08-443f-f24c5bdfedb3} {Suffrage:Voter ID:server-7bb7bc7c-1e44-f191-a20e-08f8e9408b5d Address:7bb7bc7c-1e44-f191-a20e-08f8e9408b5d} {Suffrage:Voter ID:server-7aec8c78-54a2-0dab-4956-59551473aa08 Address:7aec8c78-54a2-0dab-4956-59551473aa08}]"
2025-01-26T02:45:02.508Z [INFO]  server-7bb7bc7c-1e44-f191-a20e-08f8e9408b5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84bbbba3-144d-cc08-443f-f24c5bdfedb3 Address:84bbbba3-144d-cc08-443f-f24c5bdfedb3} {Suffrage:Voter ID:server-7bb7bc7c-1e44-f191-a20e-08f8e9408b5d Address:7bb7bc7c-1e44-f191-a20e-08f8e9408b5d} {Suffrage:Voter ID:server-7aec8c78-54a2-0dab-4956-59551473aa08 Address:7aec8c78-54a2-0dab-4956-59551473aa08}]"
2025-01-26T02:45:02.508Z [INFO]  server-7aec8c78-54a2-0dab-4956-59551473aa08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84bbbba3-144d-cc08-443f-f24c5bdfedb3 Address:84bbbba3-144d-cc08-443f-f24c5bdfedb3} {Suffrage:Voter ID:server-7bb7bc7c-1e44-f191-a20e-08f8e9408b5d Address:7bb7bc7c-1e44-f191-a20e-08f8e9408b5d} {Suffrage:Voter ID:server-7aec8c78-54a2-0dab-4956-59551473aa08 Address:7aec8c78-54a2-0dab-4956-59551473aa08}]"
2025-01-26T02:45:02.508Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:02.508Z [INFO]  server-84bbbba3-144d-cc08-443f-f24c5bdfedb3: entering follower state: follower="Node at 84bbbba3-144d-cc08-443f-f24c5bdfedb3 [Follower]" leader-address= leader-id=
2025-01-26T02:45:02.508Z [INFO]  server-7bb7bc7c-1e44-f191-a20e-08f8e9408b5d: entering follower state: follower="Node at 7bb7bc7c-1e44-f191-a20e-08f8e9408b5d [Follower]" leader-address= leader-id=
2025-01-26T02:45:02.508Z [INFO]  server-7aec8c78-54a2-0dab-4956-59551473aa08: entering follower state: follower="Node at 7aec8c78-54a2-0dab-4956-59551473aa08 [Follower]" leader-address= leader-id=
2025-01-26T02:45:02.567Z [WARN]  server-7aec8c78-54a2-0dab-4956-59551473aa08: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:02.567Z [INFO]  server-7aec8c78-54a2-0dab-4956-59551473aa08: entering candidate state: node="Node at 7aec8c78-54a2-0dab-4956-59551473aa08 [Candidate]" term=2
2025-01-26T02:45:02.567Z [INFO]  server-7aec8c78-54a2-0dab-4956-59551473aa08: election won: term=2 tally=2
2025-01-26T02:45:02.567Z [INFO]  server-7aec8c78-54a2-0dab-4956-59551473aa08: entering leader state: leader="Node at 7aec8c78-54a2-0dab-4956-59551473aa08 [Leader]"
2025-01-26T02:45:02.567Z [INFO]  server-7aec8c78-54a2-0dab-4956-59551473aa08: added peer, starting replication: peer=server-84bbbba3-144d-cc08-443f-f24c5bdfedb3
2025-01-26T02:45:02.567Z [INFO]  server-7aec8c78-54a2-0dab-4956-59551473aa08: added peer, starting replication: peer=server-7bb7bc7c-1e44-f191-a20e-08f8e9408b5d
2025-01-26T02:45:02.567Z [INFO]  server-7aec8c78-54a2-0dab-4956-59551473aa08: pipelining replication: peer="{Voter server-7bb7bc7c-1e44-f191-a20e-08f8e9408b5d 7bb7bc7c-1e44-f191-a20e-08f8e9408b5d}"
2025-01-26T02:45:02.567Z [INFO]  server-7aec8c78-54a2-0dab-4956-59551473aa08: pipelining replication: peer="{Voter server-84bbbba3-144d-cc08-443f-f24c5bdfedb3 84bbbba3-144d-cc08-443f-f24c5bdfedb3}"
2025-01-26T02:45:02.676Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:02.568165855 +0000 UTC m=+99.083198190 (2 nodes), 59.769803ms from start of poll, 59.969924ms from cluster start. Timeout at 2025-01-26 02:45:02.673167975 +0000 UTC m=+99.188200350, 105.00216ms after stability
2025-01-26T02:45:02.676Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:02.783Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:02.676078949 +0000 UTC m=+99.191111324 (1 nodes), 1.12µs from start of poll, 167.883058ms from cluster start. Timeout at 2025-01-26 02:45:02.78112475 +0000 UTC m=+99.296157125, 105.045801ms after stability
2025-01-26T02:45:02.783Z [ERROR] server-7aec8c78-54a2-0dab-4956-59551473aa08: failed to pipeline appendEntries: peer="{Voter server-84bbbba3-144d-cc08-443f-f24c5bdfedb3 84bbbba3-144d-cc08-443f-f24c5bdfedb3}" error="append pipeline closed"
2025-01-26T02:45:02.783Z [INFO]  server-7aec8c78-54a2-0dab-4956-59551473aa08: aborting pipeline replication: peer="{Voter server-84bbbba3-144d-cc08-443f-f24c5bdfedb3 84bbbba3-144d-cc08-443f-f24c5bdfedb3}"
2025-01-26T02:45:02.783Z [ERROR] server-7aec8c78-54a2-0dab-4956-59551473aa08: failed to appendEntries to: peer="{Voter server-84bbbba3-144d-cc08-443f-f24c5bdfedb3 84bbbba3-144d-cc08-443f-f24c5bdfedb3}" error="failed to connect to peer: 84bbbba3-144d-cc08-443f-f24c5bdfedb3"
2025-01-26T02:45:02.784Z [ERROR] server-7aec8c78-54a2-0dab-4956-59551473aa08: failed to heartbeat to: peer=84bbbba3-144d-cc08-443f-f24c5bdfedb3 backoff time=10ms error="failed to connect to peer: 84bbbba3-144d-cc08-443f-f24c5bdfedb3"
    raft_test.go:1669: [INFO] Finished apply without behind follower
2025-01-26T02:45:02.786Z [ERROR] server-84bbbba3-144d-cc08-443f-f24c5bdfedb3: failed to take snapshot: error="nothing new to snapshot"
2025-01-26T02:45:02.786Z [INFO]  server-7bb7bc7c-1e44-f191-a20e-08f8e9408b5d: starting snapshot up to: index=99
2025-01-26T02:45:02.786Z [INFO]  creating new snapshot: path=/tmp/raft3035524853/snapshots/2-99-1737859502786.tmp
2025-01-26T02:45:02.786Z [INFO]  server-7bb7bc7c-1e44-f191-a20e-08f8e9408b5d: compacting logs: from=1 to=92
2025-01-26T02:45:02.786Z [INFO]  server-7bb7bc7c-1e44-f191-a20e-08f8e9408b5d: snapshot complete up to: index=99
2025-01-26T02:45:02.786Z [INFO]  server-7aec8c78-54a2-0dab-4956-59551473aa08: starting snapshot up to: index=102
2025-01-26T02:45:02.786Z [INFO]  creating new snapshot: path=/tmp/raft1661336344/snapshots/2-102-1737859502786.tmp
2025-01-26T02:45:02.786Z [INFO]  server-7aec8c78-54a2-0dab-4956-59551473aa08: compacting logs: from=1 to=92
2025-01-26T02:45:02.786Z [INFO]  server-7aec8c78-54a2-0dab-4956-59551473aa08: snapshot complete up to: index=102
    raft_test.go:1690: [INFO] Finished apply without behind follower
2025-01-26T02:45:02.797Z [ERROR] server-7aec8c78-54a2-0dab-4956-59551473aa08: failed to get log: index=2 error="log not found"
2025-01-26T02:45:02.798Z [INFO]  creating new snapshot: path=/tmp/raft1738011946/snapshots/2-102-1737859502798.tmp
2025-01-26T02:45:02.798Z [INFO]  server-84bbbba3-144d-cc08-443f-f24c5bdfedb3: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-01-26T02:45:02.804Z [INFO]  server-84bbbba3-144d-cc08-443f-f24c5bdfedb3: copied to local snapshot: bytes=693
2025-01-26T02:45:02.804Z [INFO]  server-84bbbba3-144d-cc08-443f-f24c5bdfedb3: snapshot restore progress: id=2-102-1737859502798 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-01-26T02:45:02.804Z [INFO]  server-84bbbba3-144d-cc08-443f-f24c5bdfedb3: Installed remote snapshot
2025-01-26T02:45:02.804Z [INFO]  server-7aec8c78-54a2-0dab-4956-59551473aa08: pipelining replication: peer="{Voter server-84bbbba3-144d-cc08-443f-f24c5bdfedb3 84bbbba3-144d-cc08-443f-f24c5bdfedb3}"
2025-01-26T02:45:02.811Z [INFO]  server-7aec8c78-54a2-0dab-4956-59551473aa08: aborting pipeline replication: peer="{Voter server-7bb7bc7c-1e44-f191-a20e-08f8e9408b5d 7bb7bc7c-1e44-f191-a20e-08f8e9408b5d}"
2025-01-26T02:45:02.811Z [INFO]  server-7aec8c78-54a2-0dab-4956-59551473aa08: aborting pipeline replication: peer="{Voter server-84bbbba3-144d-cc08-443f-f24c5bdfedb3 84bbbba3-144d-cc08-443f-f24c5bdfedb3}"
--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.30s)
=== RUN   TestRaft_ReJoinFollower
2025-01-26T02:45:02.812Z [INFO]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c5dd391-ce90-e10c-2d68-f46f2d172e65 Address:1c5dd391-ce90-e10c-2d68-f46f2d172e65} {Suffrage:Voter ID:server-0e24bb36-d1e1-5716-b188-27a73a060776 Address:0e24bb36-d1e1-5716-b188-27a73a060776} {Suffrage:Voter ID:server-f97aef4f-4ee6-669f-a98b-2d2ddbc284dc Address:f97aef4f-4ee6-669f-a98b-2d2ddbc284dc}]"
2025-01-26T02:45:02.812Z [INFO]  server-0e24bb36-d1e1-5716-b188-27a73a060776: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c5dd391-ce90-e10c-2d68-f46f2d172e65 Address:1c5dd391-ce90-e10c-2d68-f46f2d172e65} {Suffrage:Voter ID:server-0e24bb36-d1e1-5716-b188-27a73a060776 Address:0e24bb36-d1e1-5716-b188-27a73a060776} {Suffrage:Voter ID:server-f97aef4f-4ee6-669f-a98b-2d2ddbc284dc Address:f97aef4f-4ee6-669f-a98b-2d2ddbc284dc}]"
2025-01-26T02:45:02.812Z [INFO]  server-f97aef4f-4ee6-669f-a98b-2d2ddbc284dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1c5dd391-ce90-e10c-2d68-f46f2d172e65 Address:1c5dd391-ce90-e10c-2d68-f46f2d172e65} {Suffrage:Voter ID:server-0e24bb36-d1e1-5716-b188-27a73a060776 Address:0e24bb36-d1e1-5716-b188-27a73a060776} {Suffrage:Voter ID:server-f97aef4f-4ee6-669f-a98b-2d2ddbc284dc Address:f97aef4f-4ee6-669f-a98b-2d2ddbc284dc}]"
2025-01-26T02:45:02.812Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:02.812Z [INFO]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: entering follower state: follower="Node at 1c5dd391-ce90-e10c-2d68-f46f2d172e65 [Follower]" leader-address= leader-id=
2025-01-26T02:45:02.812Z [INFO]  server-0e24bb36-d1e1-5716-b188-27a73a060776: entering follower state: follower="Node at 0e24bb36-d1e1-5716-b188-27a73a060776 [Follower]" leader-address= leader-id=
2025-01-26T02:45:02.812Z [INFO]  server-f97aef4f-4ee6-669f-a98b-2d2ddbc284dc: entering follower state: follower="Node at f97aef4f-4ee6-669f-a98b-2d2ddbc284dc [Follower]" leader-address= leader-id=
2025-01-26T02:45:02.872Z [WARN]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:02.872Z [INFO]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: entering candidate state: node="Node at 1c5dd391-ce90-e10c-2d68-f46f2d172e65 [Candidate]" term=2
2025-01-26T02:45:02.872Z [INFO]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: election won: term=2 tally=2
2025-01-26T02:45:02.872Z [INFO]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: entering leader state: leader="Node at 1c5dd391-ce90-e10c-2d68-f46f2d172e65 [Leader]"
2025-01-26T02:45:02.872Z [INFO]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: added peer, starting replication: peer=server-0e24bb36-d1e1-5716-b188-27a73a060776
2025-01-26T02:45:02.872Z [INFO]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: added peer, starting replication: peer=server-f97aef4f-4ee6-669f-a98b-2d2ddbc284dc
2025-01-26T02:45:02.872Z [INFO]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: pipelining replication: peer="{Voter server-f97aef4f-4ee6-669f-a98b-2d2ddbc284dc f97aef4f-4ee6-669f-a98b-2d2ddbc284dc}"
2025-01-26T02:45:02.872Z [INFO]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: pipelining replication: peer="{Voter server-0e24bb36-d1e1-5716-b188-27a73a060776 0e24bb36-d1e1-5716-b188-27a73a060776}"
2025-01-26T02:45:02.979Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:02.87229218 +0000 UTC m=+99.387324555 (1 nodes), 59.843723ms from start of poll, 60.021804ms from cluster start. Timeout at 2025-01-26 02:45:02.9773027 +0000 UTC m=+99.492335035, 105.01048ms after stability
2025-01-26T02:45:02.987Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:03.095Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:02.987455239 +0000 UTC m=+99.502487574 (2 nodes), 3.28µs from start of poll, 175.184823ms from cluster start. Timeout at 2025-01-26 02:45:03.092493479 +0000 UTC m=+99.607525854, 105.03828ms after stability
2025-01-26T02:45:03.095Z [INFO]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: updating configuration: command=RemoveServer server-id=server-0e24bb36-d1e1-5716-b188-27a73a060776 server-addr= servers="[{Suffrage:Voter ID:server-1c5dd391-ce90-e10c-2d68-f46f2d172e65 Address:1c5dd391-ce90-e10c-2d68-f46f2d172e65} {Suffrage:Voter ID:server-f97aef4f-4ee6-669f-a98b-2d2ddbc284dc Address:f97aef4f-4ee6-669f-a98b-2d2ddbc284dc}]"
2025-01-26T02:45:03.095Z [INFO]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: removed peer, stopping replication: peer=server-0e24bb36-d1e1-5716-b188-27a73a060776 last-index=3
2025-01-26T02:45:03.096Z [INFO]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: aborting pipeline replication: peer="{Voter server-0e24bb36-d1e1-5716-b188-27a73a060776 0e24bb36-d1e1-5716-b188-27a73a060776}"
2025-01-26T02:45:03.175Z [WARN]  server-0e24bb36-d1e1-5716-b188-27a73a060776: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-01-26T02:45:03.203Z [INFO]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: updating configuration: command=AddVoter server-id=server-0e24bb36-d1e1-5716-b188-27a73a060776 server-addr=0e24bb36-d1e1-5716-b188-27a73a060776 servers="[{Suffrage:Voter ID:server-1c5dd391-ce90-e10c-2d68-f46f2d172e65 Address:1c5dd391-ce90-e10c-2d68-f46f2d172e65} {Suffrage:Voter ID:server-f97aef4f-4ee6-669f-a98b-2d2ddbc284dc Address:f97aef4f-4ee6-669f-a98b-2d2ddbc284dc} {Suffrage:Voter ID:server-0e24bb36-d1e1-5716-b188-27a73a060776 Address:0e24bb36-d1e1-5716-b188-27a73a060776}]"
2025-01-26T02:45:03.203Z [INFO]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: added peer, starting replication: peer=server-0e24bb36-d1e1-5716-b188-27a73a060776
2025-01-26T02:45:03.203Z [WARN]  server-0e24bb36-d1e1-5716-b188-27a73a060776: failed to get previous log: previous-index=4 last-index=3 error="log not found"
2025-01-26T02:45:03.203Z [WARN]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: appendEntries rejected, sending older logs: peer="{Voter server-0e24bb36-d1e1-5716-b188-27a73a060776 0e24bb36-d1e1-5716-b188-27a73a060776}" next=4
2025-01-26T02:45:03.203Z [INFO]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: pipelining replication: peer="{Voter server-0e24bb36-d1e1-5716-b188-27a73a060776 0e24bb36-d1e1-5716-b188-27a73a060776}"
2025-01-26T02:45:03.203Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:03.311Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:03.203841312 +0000 UTC m=+99.718873687 (1 nodes), 400ns from start of poll, 391.570936ms from cluster start. Timeout at 2025-01-26 02:45:03.308864152 +0000 UTC m=+99.823896527, 105.02284ms after stability
2025-01-26T02:45:03.311Z [INFO]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: aborting pipeline replication: peer="{Voter server-f97aef4f-4ee6-669f-a98b-2d2ddbc284dc f97aef4f-4ee6-669f-a98b-2d2ddbc284dc}"
2025-01-26T02:45:03.311Z [INFO]  server-1c5dd391-ce90-e10c-2d68-f46f2d172e65: aborting pipeline replication: peer="{Voter server-0e24bb36-d1e1-5716-b188-27a73a060776 0e24bb36-d1e1-5716-b188-27a73a060776}"
--- PASS: TestRaft_ReJoinFollower (0.50s)
=== RUN   TestRaft_LeaderLeaseExpire
2025-01-26T02:45:03.312Z [INFO]  server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 Address:07aecb70-f8fd-0af6-0847-ebc4fd85e0b1} {Suffrage:Voter ID:server-473cc7db-0505-2d95-0db1-668df5530837 Address:473cc7db-0505-2d95-0db1-668df5530837}]"
2025-01-26T02:45:03.312Z [INFO]  server-473cc7db-0505-2d95-0db1-668df5530837: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 Address:07aecb70-f8fd-0af6-0847-ebc4fd85e0b1} {Suffrage:Voter ID:server-473cc7db-0505-2d95-0db1-668df5530837 Address:473cc7db-0505-2d95-0db1-668df5530837}]"
2025-01-26T02:45:03.312Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:03.312Z [INFO]  server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1: entering follower state: follower="Node at 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 [Follower]" leader-address= leader-id=
2025-01-26T02:45:03.312Z [INFO]  server-473cc7db-0505-2d95-0db1-668df5530837: entering follower state: follower="Node at 473cc7db-0505-2d95-0db1-668df5530837 [Follower]" leader-address= leader-id=
2025-01-26T02:45:03.379Z [WARN]  server-473cc7db-0505-2d95-0db1-668df5530837: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:03.379Z [INFO]  server-473cc7db-0505-2d95-0db1-668df5530837: entering candidate state: node="Node at 473cc7db-0505-2d95-0db1-668df5530837 [Candidate]" term=2
2025-01-26T02:45:03.379Z [INFO]  server-473cc7db-0505-2d95-0db1-668df5530837: election won: term=2 tally=2
2025-01-26T02:45:03.379Z [INFO]  server-473cc7db-0505-2d95-0db1-668df5530837: entering leader state: leader="Node at 473cc7db-0505-2d95-0db1-668df5530837 [Leader]"
2025-01-26T02:45:03.379Z [INFO]  server-473cc7db-0505-2d95-0db1-668df5530837: added peer, starting replication: peer=server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1
2025-01-26T02:45:03.379Z [INFO]  server-473cc7db-0505-2d95-0db1-668df5530837: pipelining replication: peer="{Voter server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1}"
2025-01-26T02:45:03.487Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:03.37994002 +0000 UTC m=+99.894972395 (1 nodes), 67.55033ms from start of poll, 67.68325ms from cluster start. Timeout at 2025-01-26 02:45:03.48495182 +0000 UTC m=+99.999984195, 105.0118ms after stability
2025-01-26T02:45:03.495Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:03.603Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:03.495692963 +0000 UTC m=+100.010725338 (1 nodes), 2.12µs from start of poll, 183.436193ms from cluster start. Timeout at 2025-01-26 02:45:03.600725603 +0000 UTC m=+100.115757978, 105.03264ms after stability
    raft_test.go:1799: [INFO] Disconnecting Node at 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 [Follower]
2025-01-26T02:45:03.603Z [ERROR] server-473cc7db-0505-2d95-0db1-668df5530837: failed to heartbeat to: peer=07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 backoff time=10ms error="failed to connect to peer: 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1"
2025-01-26T02:45:03.607Z [ERROR] server-473cc7db-0505-2d95-0db1-668df5530837: failed to pipeline appendEntries: peer="{Voter server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1}" error="append pipeline closed"
2025-01-26T02:45:03.607Z [INFO]  server-473cc7db-0505-2d95-0db1-668df5530837: aborting pipeline replication: peer="{Voter server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1}"
2025-01-26T02:45:03.619Z [ERROR] server-473cc7db-0505-2d95-0db1-668df5530837: failed to appendEntries to: peer="{Voter server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1}" error="failed to connect to peer: 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1"
2025-01-26T02:45:03.623Z [ERROR] server-473cc7db-0505-2d95-0db1-668df5530837: failed to heartbeat to: peer=07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 backoff time=10ms error="failed to connect to peer: 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1"
2025-01-26T02:45:03.643Z [ERROR] server-473cc7db-0505-2d95-0db1-668df5530837: failed to appendEntries to: peer="{Voter server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1}" error="failed to connect to peer: 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1"
2025-01-26T02:45:03.655Z [ERROR] server-473cc7db-0505-2d95-0db1-668df5530837: failed to heartbeat to: peer=07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 backoff time=10ms error="failed to connect to peer: 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1"
2025-01-26T02:45:03.655Z [WARN]  server-473cc7db-0505-2d95-0db1-668df5530837: failed to contact: server-id=server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 time=52.278997ms
2025-01-26T02:45:03.655Z [WARN]  server-473cc7db-0505-2d95-0db1-668df5530837: failed to contact quorum of nodes, stepping down
2025-01-26T02:45:03.655Z [INFO]  server-473cc7db-0505-2d95-0db1-668df5530837: entering follower state: follower="Node at 473cc7db-0505-2d95-0db1-668df5530837 [Follower]" leader-address= leader-id=
2025-01-26T02:45:03.655Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:03.659Z [WARN]  server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1: heartbeat timeout reached, starting election: last-leader-addr=473cc7db-0505-2d95-0db1-668df5530837 last-leader-id=server-473cc7db-0505-2d95-0db1-668df5530837
2025-01-26T02:45:03.659Z [INFO]  server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1: entering candidate state: node="Node at 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 [Candidate]" term=3
2025-01-26T02:45:03.659Z [ERROR] server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1: failed to make requestVote RPC: target="{Voter server-473cc7db-0505-2d95-0db1-668df5530837 473cc7db-0505-2d95-0db1-668df5530837}" error="failed to connect to peer: 473cc7db-0505-2d95-0db1-668df5530837" term=3
2025-01-26T02:45:03.727Z [WARN]  server-473cc7db-0505-2d95-0db1-668df5530837: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:03.727Z [INFO]  server-473cc7db-0505-2d95-0db1-668df5530837: entering candidate state: node="Node at 473cc7db-0505-2d95-0db1-668df5530837 [Candidate]" term=3
2025-01-26T02:45:03.727Z [ERROR] server-473cc7db-0505-2d95-0db1-668df5530837: failed to make requestVote RPC: target="{Voter server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1}" error="failed to connect to peer: 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1" term=3
2025-01-26T02:45:03.751Z [WARN]  server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1: Election timeout reached, restarting election
2025-01-26T02:45:03.751Z [INFO]  server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1: entering candidate state: node="Node at 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 [Candidate]" term=4
2025-01-26T02:45:03.751Z [ERROR] server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1: failed to make requestVote RPC: target="{Voter server-473cc7db-0505-2d95-0db1-668df5530837 473cc7db-0505-2d95-0db1-668df5530837}" error="failed to connect to peer: 473cc7db-0505-2d95-0db1-668df5530837" term=4
2025-01-26T02:45:03.811Z [WARN]  server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1: Election timeout reached, restarting election
2025-01-26T02:45:03.811Z [INFO]  server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1: entering candidate state: node="Node at 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 [Candidate]" term=5
2025-01-26T02:45:03.811Z [ERROR] server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1: failed to make requestVote RPC: target="{Voter server-473cc7db-0505-2d95-0db1-668df5530837 473cc7db-0505-2d95-0db1-668df5530837}" error="failed to connect to peer: 473cc7db-0505-2d95-0db1-668df5530837" term=5
2025-01-26T02:45:03.815Z [WARN]  server-473cc7db-0505-2d95-0db1-668df5530837: Election timeout reached, restarting election
2025-01-26T02:45:03.815Z [INFO]  server-473cc7db-0505-2d95-0db1-668df5530837: entering candidate state: node="Node at 473cc7db-0505-2d95-0db1-668df5530837 [Candidate]" term=4
2025-01-26T02:45:03.815Z [ERROR] server-473cc7db-0505-2d95-0db1-668df5530837: failed to make requestVote RPC: target="{Voter server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1}" error="failed to connect to peer: 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1" term=4
2025-01-26T02:45:03.835Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:03.727939692 +0000 UTC m=+100.242972067 (0 nodes), 72.144798ms from start of poll, 415.682922ms from cluster start. Timeout at 2025-01-26 02:45:03.832942972 +0000 UTC m=+100.347975347, 105.00328ms after stability
2025-01-26T02:45:03.899Z [WARN]  server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1: Election timeout reached, restarting election
2025-01-26T02:45:03.899Z [INFO]  server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1: entering candidate state: node="Node at 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 [Candidate]" term=6
2025-01-26T02:45:03.899Z [ERROR] server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1: failed to make requestVote RPC: target="{Voter server-473cc7db-0505-2d95-0db1-668df5530837 473cc7db-0505-2d95-0db1-668df5530837}" error="failed to connect to peer: 473cc7db-0505-2d95-0db1-668df5530837" term=6
2025-01-26T02:45:03.917Z [WARN]  server-473cc7db-0505-2d95-0db1-668df5530837: Election timeout reached, restarting election
2025-01-26T02:45:03.917Z [INFO]  server-473cc7db-0505-2d95-0db1-668df5530837: entering candidate state: node="Node at 473cc7db-0505-2d95-0db1-668df5530837 [Candidate]" term=5
2025-01-26T02:45:03.917Z [ERROR] server-473cc7db-0505-2d95-0db1-668df5530837: failed to make requestVote RPC: target="{Voter server-07aecb70-f8fd-0af6-0847-ebc4fd85e0b1 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1}" error="failed to connect to peer: 07aecb70-f8fd-0af6-0847-ebc4fd85e0b1" term=5
--- PASS: TestRaft_LeaderLeaseExpire (0.63s)
=== RUN   TestRaft_Barrier
2025-01-26T02:45:03.944Z [INFO]  server-50cc4348-cacf-b61a-eeea-7ebe8e983c57: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50cc4348-cacf-b61a-eeea-7ebe8e983c57 Address:50cc4348-cacf-b61a-eeea-7ebe8e983c57} {Suffrage:Voter ID:server-412fe294-63c4-3820-8c24-7fbd1aaa24cd Address:412fe294-63c4-3820-8c24-7fbd1aaa24cd} {Suffrage:Voter ID:server-198c90ae-f370-f0ef-8f8d-121f4fbf200e Address:198c90ae-f370-f0ef-8f8d-121f4fbf200e}]"
2025-01-26T02:45:03.944Z [INFO]  server-412fe294-63c4-3820-8c24-7fbd1aaa24cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50cc4348-cacf-b61a-eeea-7ebe8e983c57 Address:50cc4348-cacf-b61a-eeea-7ebe8e983c57} {Suffrage:Voter ID:server-412fe294-63c4-3820-8c24-7fbd1aaa24cd Address:412fe294-63c4-3820-8c24-7fbd1aaa24cd} {Suffrage:Voter ID:server-198c90ae-f370-f0ef-8f8d-121f4fbf200e Address:198c90ae-f370-f0ef-8f8d-121f4fbf200e}]"
2025-01-26T02:45:03.944Z [INFO]  server-198c90ae-f370-f0ef-8f8d-121f4fbf200e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50cc4348-cacf-b61a-eeea-7ebe8e983c57 Address:50cc4348-cacf-b61a-eeea-7ebe8e983c57} {Suffrage:Voter ID:server-412fe294-63c4-3820-8c24-7fbd1aaa24cd Address:412fe294-63c4-3820-8c24-7fbd1aaa24cd} {Suffrage:Voter ID:server-198c90ae-f370-f0ef-8f8d-121f4fbf200e Address:198c90ae-f370-f0ef-8f8d-121f4fbf200e}]"
2025-01-26T02:45:03.944Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:03.944Z [INFO]  server-50cc4348-cacf-b61a-eeea-7ebe8e983c57: entering follower state: follower="Node at 50cc4348-cacf-b61a-eeea-7ebe8e983c57 [Follower]" leader-address= leader-id=
2025-01-26T02:45:03.944Z [INFO]  server-412fe294-63c4-3820-8c24-7fbd1aaa24cd: entering follower state: follower="Node at 412fe294-63c4-3820-8c24-7fbd1aaa24cd [Follower]" leader-address= leader-id=
2025-01-26T02:45:03.944Z [INFO]  server-198c90ae-f370-f0ef-8f8d-121f4fbf200e: entering follower state: follower="Node at 198c90ae-f370-f0ef-8f8d-121f4fbf200e [Follower]" leader-address= leader-id=
2025-01-26T02:45:03.999Z [WARN]  server-412fe294-63c4-3820-8c24-7fbd1aaa24cd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:03.999Z [INFO]  server-412fe294-63c4-3820-8c24-7fbd1aaa24cd: entering candidate state: node="Node at 412fe294-63c4-3820-8c24-7fbd1aaa24cd [Candidate]" term=2
2025-01-26T02:45:03.999Z [INFO]  server-412fe294-63c4-3820-8c24-7fbd1aaa24cd: election won: term=2 tally=2
2025-01-26T02:45:03.999Z [INFO]  server-412fe294-63c4-3820-8c24-7fbd1aaa24cd: entering leader state: leader="Node at 412fe294-63c4-3820-8c24-7fbd1aaa24cd [Leader]"
2025-01-26T02:45:03.999Z [INFO]  server-412fe294-63c4-3820-8c24-7fbd1aaa24cd: added peer, starting replication: peer=server-50cc4348-cacf-b61a-eeea-7ebe8e983c57
2025-01-26T02:45:03.999Z [INFO]  server-412fe294-63c4-3820-8c24-7fbd1aaa24cd: added peer, starting replication: peer=server-198c90ae-f370-f0ef-8f8d-121f4fbf200e
2025-01-26T02:45:03.999Z [INFO]  server-412fe294-63c4-3820-8c24-7fbd1aaa24cd: pipelining replication: peer="{Voter server-198c90ae-f370-f0ef-8f8d-121f4fbf200e 198c90ae-f370-f0ef-8f8d-121f4fbf200e}"
2025-01-26T02:45:03.999Z [INFO]  server-412fe294-63c4-3820-8c24-7fbd1aaa24cd: pipelining replication: peer="{Voter server-50cc4348-cacf-b61a-eeea-7ebe8e983c57 50cc4348-cacf-b61a-eeea-7ebe8e983c57}"
2025-01-26T02:45:04.107Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:04.000006703 +0000 UTC m=+100.515039078 (1 nodes), 55.527217ms from start of poll, 55.722138ms from cluster start. Timeout at 2025-01-26 02:45:04.105010223 +0000 UTC m=+100.620042598, 105.00352ms after stability
2025-01-26T02:45:04.124Z [INFO]  server-412fe294-63c4-3820-8c24-7fbd1aaa24cd: aborting pipeline replication: peer="{Voter server-198c90ae-f370-f0ef-8f8d-121f4fbf200e 198c90ae-f370-f0ef-8f8d-121f4fbf200e}"
2025-01-26T02:45:04.124Z [INFO]  server-412fe294-63c4-3820-8c24-7fbd1aaa24cd: aborting pipeline replication: peer="{Voter server-50cc4348-cacf-b61a-eeea-7ebe8e983c57 50cc4348-cacf-b61a-eeea-7ebe8e983c57}"
--- PASS: TestRaft_Barrier (0.18s)
=== RUN   TestRaft_VerifyLeader
2025-01-26T02:45:04.125Z [INFO]  server-f1b36645-79bc-9af3-418f-971fbddbaff4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f1b36645-79bc-9af3-418f-971fbddbaff4 Address:f1b36645-79bc-9af3-418f-971fbddbaff4} {Suffrage:Voter ID:server-9b9b7cdd-614c-d23c-12d6-bc9217353b80 Address:9b9b7cdd-614c-d23c-12d6-bc9217353b80} {Suffrage:Voter ID:server-bc138eac-e3f0-0e2d-0097-d673697124e0 Address:bc138eac-e3f0-0e2d-0097-d673697124e0}]"
2025-01-26T02:45:04.125Z [INFO]  server-9b9b7cdd-614c-d23c-12d6-bc9217353b80: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f1b36645-79bc-9af3-418f-971fbddbaff4 Address:f1b36645-79bc-9af3-418f-971fbddbaff4} {Suffrage:Voter ID:server-9b9b7cdd-614c-d23c-12d6-bc9217353b80 Address:9b9b7cdd-614c-d23c-12d6-bc9217353b80} {Suffrage:Voter ID:server-bc138eac-e3f0-0e2d-0097-d673697124e0 Address:bc138eac-e3f0-0e2d-0097-d673697124e0}]"
2025-01-26T02:45:04.125Z [INFO]  server-bc138eac-e3f0-0e2d-0097-d673697124e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f1b36645-79bc-9af3-418f-971fbddbaff4 Address:f1b36645-79bc-9af3-418f-971fbddbaff4} {Suffrage:Voter ID:server-9b9b7cdd-614c-d23c-12d6-bc9217353b80 Address:9b9b7cdd-614c-d23c-12d6-bc9217353b80} {Suffrage:Voter ID:server-bc138eac-e3f0-0e2d-0097-d673697124e0 Address:bc138eac-e3f0-0e2d-0097-d673697124e0}]"
2025-01-26T02:45:04.125Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:04.125Z [INFO]  server-f1b36645-79bc-9af3-418f-971fbddbaff4: entering follower state: follower="Node at f1b36645-79bc-9af3-418f-971fbddbaff4 [Follower]" leader-address= leader-id=
2025-01-26T02:45:04.125Z [INFO]  server-9b9b7cdd-614c-d23c-12d6-bc9217353b80: entering follower state: follower="Node at 9b9b7cdd-614c-d23c-12d6-bc9217353b80 [Follower]" leader-address= leader-id=
2025-01-26T02:45:04.125Z [INFO]  server-bc138eac-e3f0-0e2d-0097-d673697124e0: entering follower state: follower="Node at bc138eac-e3f0-0e2d-0097-d673697124e0 [Follower]" leader-address= leader-id=
2025-01-26T02:45:04.180Z [WARN]  server-9b9b7cdd-614c-d23c-12d6-bc9217353b80: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:04.180Z [INFO]  server-9b9b7cdd-614c-d23c-12d6-bc9217353b80: entering candidate state: node="Node at 9b9b7cdd-614c-d23c-12d6-bc9217353b80 [Candidate]" term=2
2025-01-26T02:45:04.180Z [INFO]  server-9b9b7cdd-614c-d23c-12d6-bc9217353b80: election won: term=2 tally=2
2025-01-26T02:45:04.180Z [INFO]  server-9b9b7cdd-614c-d23c-12d6-bc9217353b80: entering leader state: leader="Node at 9b9b7cdd-614c-d23c-12d6-bc9217353b80 [Leader]"
2025-01-26T02:45:04.180Z [INFO]  server-9b9b7cdd-614c-d23c-12d6-bc9217353b80: added peer, starting replication: peer=server-f1b36645-79bc-9af3-418f-971fbddbaff4
2025-01-26T02:45:04.180Z [INFO]  server-9b9b7cdd-614c-d23c-12d6-bc9217353b80: added peer, starting replication: peer=server-bc138eac-e3f0-0e2d-0097-d673697124e0
2025-01-26T02:45:04.180Z [INFO]  server-9b9b7cdd-614c-d23c-12d6-bc9217353b80: pipelining replication: peer="{Voter server-bc138eac-e3f0-0e2d-0097-d673697124e0 bc138eac-e3f0-0e2d-0097-d673697124e0}"
2025-01-26T02:45:04.180Z [WARN]  server-f1b36645-79bc-9af3-418f-971fbddbaff4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:04.180Z [INFO]  server-f1b36645-79bc-9af3-418f-971fbddbaff4: entering candidate state: node="Node at f1b36645-79bc-9af3-418f-971fbddbaff4 [Candidate]" term=2
2025-01-26T02:45:04.180Z [WARN]  server-bc138eac-e3f0-0e2d-0097-d673697124e0: rejecting vote request since we have a leader: from=f1b36645-79bc-9af3-418f-971fbddbaff4 leader=9b9b7cdd-614c-d23c-12d6-bc9217353b80 leader-id=server-9b9b7cdd-614c-d23c-12d6-bc9217353b80
2025-01-26T02:45:04.180Z [INFO]  server-f1b36645-79bc-9af3-418f-971fbddbaff4: duplicate requestVote for same term: term=2
2025-01-26T02:45:04.180Z [INFO]  server-f1b36645-79bc-9af3-418f-971fbddbaff4: entering follower state: follower="Node at f1b36645-79bc-9af3-418f-971fbddbaff4 [Follower]" leader-address=9b9b7cdd-614c-d23c-12d6-bc9217353b80 leader-id=server-9b9b7cdd-614c-d23c-12d6-bc9217353b80
2025-01-26T02:45:04.180Z [INFO]  server-9b9b7cdd-614c-d23c-12d6-bc9217353b80: pipelining replication: peer="{Voter server-f1b36645-79bc-9af3-418f-971fbddbaff4 f1b36645-79bc-9af3-418f-971fbddbaff4}"
2025-01-26T02:45:04.180Z [WARN]  server-9b9b7cdd-614c-d23c-12d6-bc9217353b80: rejecting vote request since we have a leader: from=f1b36645-79bc-9af3-418f-971fbddbaff4 leader=9b9b7cdd-614c-d23c-12d6-bc9217353b80 leader-id=server-9b9b7cdd-614c-d23c-12d6-bc9217353b80
2025-01-26T02:45:04.287Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:04.180523718 +0000 UTC m=+100.695556093 (1 nodes), 55.344736ms from start of poll, 55.548337ms from cluster start. Timeout at 2025-01-26 02:45:04.285526478 +0000 UTC m=+100.800558853, 105.00276ms after stability
2025-01-26T02:45:04.287Z [INFO]  server-9b9b7cdd-614c-d23c-12d6-bc9217353b80: aborting pipeline replication: peer="{Voter server-bc138eac-e3f0-0e2d-0097-d673697124e0 bc138eac-e3f0-0e2d-0097-d673697124e0}"
2025-01-26T02:45:04.288Z [INFO]  server-9b9b7cdd-614c-d23c-12d6-bc9217353b80: aborting pipeline replication: peer="{Voter server-f1b36645-79bc-9af3-418f-971fbddbaff4 f1b36645-79bc-9af3-418f-971fbddbaff4}"
2025-01-26T02:45:04.791Z [ERROR] server-9b9b7cdd-614c-d23c-12d6-bc9217353b80: failed to heartbeat to: peer=f1b36645-79bc-9af3-418f-971fbddbaff4 backoff time=10ms error="command timed out"
--- PASS: TestRaft_VerifyLeader (0.67s)
=== RUN   TestRaft_VerifyLeader_Single
2025-01-26T02:45:04.791Z [INFO]  server-634f0bbd-f10e-b197-095d-024b4ded5edd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-634f0bbd-f10e-b197-095d-024b4ded5edd Address:634f0bbd-f10e-b197-095d-024b4ded5edd}]"
2025-01-26T02:45:04.791Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:04.791Z [INFO]  server-634f0bbd-f10e-b197-095d-024b4ded5edd: entering follower state: follower="Node at 634f0bbd-f10e-b197-095d-024b4ded5edd [Follower]" leader-address= leader-id=
2025-01-26T02:45:04.851Z [WARN]  server-634f0bbd-f10e-b197-095d-024b4ded5edd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:04.851Z [INFO]  server-634f0bbd-f10e-b197-095d-024b4ded5edd: entering candidate state: node="Node at 634f0bbd-f10e-b197-095d-024b4ded5edd [Candidate]" term=2
2025-01-26T02:45:04.851Z [INFO]  server-634f0bbd-f10e-b197-095d-024b4ded5edd: election won: term=2 tally=1
2025-01-26T02:45:04.851Z [INFO]  server-634f0bbd-f10e-b197-095d-024b4ded5edd: entering leader state: leader="Node at 634f0bbd-f10e-b197-095d-024b4ded5edd [Leader]"
2025-01-26T02:45:04.959Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:04.851800032 +0000 UTC m=+101.366832367 (1 nodes), 59.970764ms from start of poll, 60.085525ms from cluster start. Timeout at 2025-01-26 02:45:04.956801112 +0000 UTC m=+101.471833447, 105.00108ms after stability
--- PASS: TestRaft_VerifyLeader_Single (0.17s)
=== RUN   TestRaft_VerifyLeader_Fail
2025-01-26T02:45:04.960Z [INFO]  server-c872b77e-5c76-4a37-5fcd-240b375ec270: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c872b77e-5c76-4a37-5fcd-240b375ec270 Address:c872b77e-5c76-4a37-5fcd-240b375ec270} {Suffrage:Voter ID:server-d419241b-50dc-3cd6-7d8d-d3ecb984f1b4 Address:d419241b-50dc-3cd6-7d8d-d3ecb984f1b4}]"
2025-01-26T02:45:04.960Z [INFO]  server-d419241b-50dc-3cd6-7d8d-d3ecb984f1b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c872b77e-5c76-4a37-5fcd-240b375ec270 Address:c872b77e-5c76-4a37-5fcd-240b375ec270} {Suffrage:Voter ID:server-d419241b-50dc-3cd6-7d8d-d3ecb984f1b4 Address:d419241b-50dc-3cd6-7d8d-d3ecb984f1b4}]"
2025-01-26T02:45:04.960Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:04.960Z [INFO]  server-c872b77e-5c76-4a37-5fcd-240b375ec270: entering follower state: follower="Node at c872b77e-5c76-4a37-5fcd-240b375ec270 [Follower]" leader-address= leader-id=
2025-01-26T02:45:04.960Z [INFO]  server-d419241b-50dc-3cd6-7d8d-d3ecb984f1b4: entering follower state: follower="Node at d419241b-50dc-3cd6-7d8d-d3ecb984f1b4 [Follower]" leader-address= leader-id=
2025-01-26T02:45:05.035Z [WARN]  server-c872b77e-5c76-4a37-5fcd-240b375ec270: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:05.035Z [INFO]  server-c872b77e-5c76-4a37-5fcd-240b375ec270: entering candidate state: node="Node at c872b77e-5c76-4a37-5fcd-240b375ec270 [Candidate]" term=2
2025-01-26T02:45:05.035Z [INFO]  server-c872b77e-5c76-4a37-5fcd-240b375ec270: election won: term=2 tally=2
2025-01-26T02:45:05.035Z [INFO]  server-c872b77e-5c76-4a37-5fcd-240b375ec270: entering leader state: leader="Node at c872b77e-5c76-4a37-5fcd-240b375ec270 [Leader]"
2025-01-26T02:45:05.035Z [INFO]  server-c872b77e-5c76-4a37-5fcd-240b375ec270: added peer, starting replication: peer=server-d419241b-50dc-3cd6-7d8d-d3ecb984f1b4
2025-01-26T02:45:05.036Z [INFO]  server-c872b77e-5c76-4a37-5fcd-240b375ec270: pipelining replication: peer="{Voter server-d419241b-50dc-3cd6-7d8d-d3ecb984f1b4 d419241b-50dc-3cd6-7d8d-d3ecb984f1b4}"
2025-01-26T02:45:05.143Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:05.03607755 +0000 UTC m=+101.551109925 (1 nodes), 75.76162ms from start of poll, 75.914661ms from cluster start. Timeout at 2025-01-26 02:45:05.14109903 +0000 UTC m=+101.656131405, 105.02148ms after stability
2025-01-26T02:45:05.143Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:05.251Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:05.143810603 +0000 UTC m=+101.658842978 (1 nodes), 680ns from start of poll, 183.647714ms from cluster start. Timeout at 2025-01-26 02:45:05.248831923 +0000 UTC m=+101.763864298, 105.02132ms after stability
2025-01-26T02:45:05.255Z [ERROR] server-c872b77e-5c76-4a37-5fcd-240b375ec270: peer has newer term, stopping replication: peer="{Voter server-d419241b-50dc-3cd6-7d8d-d3ecb984f1b4 d419241b-50dc-3cd6-7d8d-d3ecb984f1b4}"
2025-01-26T02:45:05.255Z [INFO]  server-c872b77e-5c76-4a37-5fcd-240b375ec270: aborting pipeline replication: peer="{Voter server-d419241b-50dc-3cd6-7d8d-d3ecb984f1b4 d419241b-50dc-3cd6-7d8d-d3ecb984f1b4}"
2025-01-26T02:45:05.255Z [INFO]  server-c872b77e-5c76-4a37-5fcd-240b375ec270: entering follower state: follower="Node at c872b77e-5c76-4a37-5fcd-240b375ec270 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VerifyLeader_Fail (0.30s)
=== RUN   TestRaft_VerifyLeader_PartialConnect
2025-01-26T02:45:05.256Z [INFO]  server-5eee92f6-e81c-ff91-f3fc-07c71ec44ed4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5eee92f6-e81c-ff91-f3fc-07c71ec44ed4 Address:5eee92f6-e81c-ff91-f3fc-07c71ec44ed4} {Suffrage:Voter ID:server-5e1d01e9-f741-5b6e-b867-52243f628036 Address:5e1d01e9-f741-5b6e-b867-52243f628036} {Suffrage:Voter ID:server-0c04750b-9be2-81d5-c9b8-68d1d2820228 Address:0c04750b-9be2-81d5-c9b8-68d1d2820228}]"
2025-01-26T02:45:05.256Z [INFO]  server-5e1d01e9-f741-5b6e-b867-52243f628036: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5eee92f6-e81c-ff91-f3fc-07c71ec44ed4 Address:5eee92f6-e81c-ff91-f3fc-07c71ec44ed4} {Suffrage:Voter ID:server-5e1d01e9-f741-5b6e-b867-52243f628036 Address:5e1d01e9-f741-5b6e-b867-52243f628036} {Suffrage:Voter ID:server-0c04750b-9be2-81d5-c9b8-68d1d2820228 Address:0c04750b-9be2-81d5-c9b8-68d1d2820228}]"
2025-01-26T02:45:05.256Z [INFO]  server-0c04750b-9be2-81d5-c9b8-68d1d2820228: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5eee92f6-e81c-ff91-f3fc-07c71ec44ed4 Address:5eee92f6-e81c-ff91-f3fc-07c71ec44ed4} {Suffrage:Voter ID:server-5e1d01e9-f741-5b6e-b867-52243f628036 Address:5e1d01e9-f741-5b6e-b867-52243f628036} {Suffrage:Voter ID:server-0c04750b-9be2-81d5-c9b8-68d1d2820228 Address:0c04750b-9be2-81d5-c9b8-68d1d2820228}]"
2025-01-26T02:45:05.256Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:05.256Z [INFO]  server-5eee92f6-e81c-ff91-f3fc-07c71ec44ed4: entering follower state: follower="Node at 5eee92f6-e81c-ff91-f3fc-07c71ec44ed4 [Follower]" leader-address= leader-id=
2025-01-26T02:45:05.256Z [INFO]  server-5e1d01e9-f741-5b6e-b867-52243f628036: entering follower state: follower="Node at 5e1d01e9-f741-5b6e-b867-52243f628036 [Follower]" leader-address= leader-id=
2025-01-26T02:45:05.256Z [INFO]  server-0c04750b-9be2-81d5-c9b8-68d1d2820228: entering follower state: follower="Node at 0c04750b-9be2-81d5-c9b8-68d1d2820228 [Follower]" leader-address= leader-id=
2025-01-26T02:45:05.352Z [WARN]  server-5e1d01e9-f741-5b6e-b867-52243f628036: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:05.352Z [INFO]  server-5e1d01e9-f741-5b6e-b867-52243f628036: entering candidate state: node="Node at 5e1d01e9-f741-5b6e-b867-52243f628036 [Candidate]" term=2
2025-01-26T02:45:05.352Z [WARN]  server-0c04750b-9be2-81d5-c9b8-68d1d2820228: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:05.352Z [INFO]  server-0c04750b-9be2-81d5-c9b8-68d1d2820228: entering candidate state: node="Node at 0c04750b-9be2-81d5-c9b8-68d1d2820228 [Candidate]" term=2
2025-01-26T02:45:05.352Z [INFO]  server-0c04750b-9be2-81d5-c9b8-68d1d2820228: duplicate requestVote for same term: term=2
2025-01-26T02:45:05.352Z [WARN]  server-5eee92f6-e81c-ff91-f3fc-07c71ec44ed4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:05.352Z [INFO]  server-5eee92f6-e81c-ff91-f3fc-07c71ec44ed4: entering candidate state: node="Node at 5eee92f6-e81c-ff91-f3fc-07c71ec44ed4 [Candidate]" term=2
2025-01-26T02:45:05.352Z [INFO]  server-0c04750b-9be2-81d5-c9b8-68d1d2820228: duplicate requestVote for same term: term=2
2025-01-26T02:45:05.352Z [INFO]  server-5eee92f6-e81c-ff91-f3fc-07c71ec44ed4: duplicate requestVote for same term: term=2
2025-01-26T02:45:05.352Z [INFO]  server-5eee92f6-e81c-ff91-f3fc-07c71ec44ed4: duplicate requestVote for same term: term=2
2025-01-26T02:45:05.352Z [INFO]  server-5e1d01e9-f741-5b6e-b867-52243f628036: duplicate requestVote for same term: term=2
2025-01-26T02:45:05.352Z [INFO]  server-5e1d01e9-f741-5b6e-b867-52243f628036: duplicate requestVote for same term: term=2
2025-01-26T02:45:05.415Z [WARN]  server-5e1d01e9-f741-5b6e-b867-52243f628036: Election timeout reached, restarting election
2025-01-26T02:45:05.415Z [INFO]  server-5e1d01e9-f741-5b6e-b867-52243f628036: entering candidate state: node="Node at 5e1d01e9-f741-5b6e-b867-52243f628036 [Candidate]" term=3
2025-01-26T02:45:05.415Z [INFO]  server-0c04750b-9be2-81d5-c9b8-68d1d2820228: entering follower state: follower="Node at 0c04750b-9be2-81d5-c9b8-68d1d2820228 [Follower]" leader-address= leader-id=
2025-01-26T02:45:05.415Z [INFO]  server-5e1d01e9-f741-5b6e-b867-52243f628036: election won: term=3 tally=2
2025-01-26T02:45:05.415Z [INFO]  server-5e1d01e9-f741-5b6e-b867-52243f628036: entering leader state: leader="Node at 5e1d01e9-f741-5b6e-b867-52243f628036 [Leader]"
2025-01-26T02:45:05.415Z [INFO]  server-5e1d01e9-f741-5b6e-b867-52243f628036: added peer, starting replication: peer=server-5eee92f6-e81c-ff91-f3fc-07c71ec44ed4
2025-01-26T02:45:05.415Z [INFO]  server-5e1d01e9-f741-5b6e-b867-52243f628036: added peer, starting replication: peer=server-0c04750b-9be2-81d5-c9b8-68d1d2820228
2025-01-26T02:45:05.415Z [INFO]  server-5e1d01e9-f741-5b6e-b867-52243f628036: pipelining replication: peer="{Voter server-0c04750b-9be2-81d5-c9b8-68d1d2820228 0c04750b-9be2-81d5-c9b8-68d1d2820228}"
2025-01-26T02:45:05.415Z [INFO]  server-5eee92f6-e81c-ff91-f3fc-07c71ec44ed4: entering follower state: follower="Node at 5eee92f6-e81c-ff91-f3fc-07c71ec44ed4 [Follower]" leader-address= leader-id=
2025-01-26T02:45:05.416Z [INFO]  server-5e1d01e9-f741-5b6e-b867-52243f628036: pipelining replication: peer="{Voter server-5eee92f6-e81c-ff91-f3fc-07c71ec44ed4 5eee92f6-e81c-ff91-f3fc-07c71ec44ed4}"
2025-01-26T02:45:05.523Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:05.415976575 +0000 UTC m=+101.931008950 (1 nodes), 159.520128ms from start of poll, 159.703289ms from cluster start. Timeout at 2025-01-26 02:45:05.520996095 +0000 UTC m=+102.036028470, 105.01952ms after stability
2025-01-26T02:45:05.531Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:05.639Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:05.531696277 +0000 UTC m=+102.046728652 (2 nodes), 1.16µs from start of poll, 275.422991ms from cluster start. Timeout at 2025-01-26 02:45:05.636729797 +0000 UTC m=+102.151762172, 105.03352ms after stability
    raft_test.go:1968: [INFO] Disconnecting Node at 5eee92f6-e81c-ff91-f3fc-07c71ec44ed4 [Follower]
2025-01-26T02:45:05.639Z [ERROR] server-5e1d01e9-f741-5b6e-b867-52243f628036: failed to heartbeat to: peer=5eee92f6-e81c-ff91-f3fc-07c71ec44ed4 backoff time=10ms error="failed to connect to peer: 5eee92f6-e81c-ff91-f3fc-07c71ec44ed4"
2025-01-26T02:45:05.639Z [INFO]  server-5e1d01e9-f741-5b6e-b867-52243f628036: aborting pipeline replication: peer="{Voter server-0c04750b-9be2-81d5-c9b8-68d1d2820228 0c04750b-9be2-81d5-c9b8-68d1d2820228}"
2025-01-26T02:45:05.639Z [INFO]  server-5e1d01e9-f741-5b6e-b867-52243f628036: aborting pipeline replication: peer="{Voter server-5eee92f6-e81c-ff91-f3fc-07c71ec44ed4 5eee92f6-e81c-ff91-f3fc-07c71ec44ed4}"
--- PASS: TestRaft_VerifyLeader_PartialConnect (0.38s)
=== RUN   TestRaft_NotifyCh
2025-01-26T02:45:05.640Z [INFO]  server-8e46f417-8e99-d1d1-dfef-92b6f881d512: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8e46f417-8e99-d1d1-dfef-92b6f881d512 Address:8e46f417-8e99-d1d1-dfef-92b6f881d512}]"
2025-01-26T02:45:05.640Z [INFO]  server-8e46f417-8e99-d1d1-dfef-92b6f881d512: entering follower state: follower="Node at 8e46f417-8e99-d1d1-dfef-92b6f881d512 [Follower]" leader-address= leader-id=
2025-01-26T02:45:05.703Z [WARN]  server-8e46f417-8e99-d1d1-dfef-92b6f881d512: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:05.703Z [INFO]  server-8e46f417-8e99-d1d1-dfef-92b6f881d512: entering candidate state: node="Node at 8e46f417-8e99-d1d1-dfef-92b6f881d512 [Candidate]" term=2
2025-01-26T02:45:05.703Z [INFO]  server-8e46f417-8e99-d1d1-dfef-92b6f881d512: election won: term=2 tally=1
2025-01-26T02:45:05.703Z [INFO]  server-8e46f417-8e99-d1d1-dfef-92b6f881d512: entering leader state: leader="Node at 8e46f417-8e99-d1d1-dfef-92b6f881d512 [Leader]"
--- PASS: TestRaft_NotifyCh (0.06s)
=== RUN   TestRaft_AppendEntry
2025-01-26T02:45:05.704Z [INFO]  server-4db61052-fa17-73de-074f-f34069791350: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4db61052-fa17-73de-074f-f34069791350 Address:4db61052-fa17-73de-074f-f34069791350} {Suffrage:Voter ID:server-c677fdc5-ca0b-f15b-53a0-8214ad3a693b Address:c677fdc5-ca0b-f15b-53a0-8214ad3a693b} {Suffrage:Voter ID:server-8efc80f0-75af-1c80-22ab-481dd115ebee Address:8efc80f0-75af-1c80-22ab-481dd115ebee}]"
2025-01-26T02:45:05.704Z [INFO]  server-c677fdc5-ca0b-f15b-53a0-8214ad3a693b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4db61052-fa17-73de-074f-f34069791350 Address:4db61052-fa17-73de-074f-f34069791350} {Suffrage:Voter ID:server-c677fdc5-ca0b-f15b-53a0-8214ad3a693b Address:c677fdc5-ca0b-f15b-53a0-8214ad3a693b} {Suffrage:Voter ID:server-8efc80f0-75af-1c80-22ab-481dd115ebee Address:8efc80f0-75af-1c80-22ab-481dd115ebee}]"
2025-01-26T02:45:05.704Z [INFO]  server-8efc80f0-75af-1c80-22ab-481dd115ebee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4db61052-fa17-73de-074f-f34069791350 Address:4db61052-fa17-73de-074f-f34069791350} {Suffrage:Voter ID:server-c677fdc5-ca0b-f15b-53a0-8214ad3a693b Address:c677fdc5-ca0b-f15b-53a0-8214ad3a693b} {Suffrage:Voter ID:server-8efc80f0-75af-1c80-22ab-481dd115ebee Address:8efc80f0-75af-1c80-22ab-481dd115ebee}]"
2025-01-26T02:45:05.704Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:05.704Z [INFO]  server-4db61052-fa17-73de-074f-f34069791350: entering follower state: follower="Node at 4db61052-fa17-73de-074f-f34069791350 [Follower]" leader-address= leader-id=
2025-01-26T02:45:05.704Z [INFO]  server-c677fdc5-ca0b-f15b-53a0-8214ad3a693b: entering follower state: follower="Node at c677fdc5-ca0b-f15b-53a0-8214ad3a693b [Follower]" leader-address= leader-id=
2025-01-26T02:45:05.704Z [INFO]  server-8efc80f0-75af-1c80-22ab-481dd115ebee: entering follower state: follower="Node at 8efc80f0-75af-1c80-22ab-481dd115ebee [Follower]" leader-address= leader-id=
2025-01-26T02:45:05.771Z [WARN]  server-c677fdc5-ca0b-f15b-53a0-8214ad3a693b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:05.771Z [INFO]  server-c677fdc5-ca0b-f15b-53a0-8214ad3a693b: entering candidate state: node="Node at c677fdc5-ca0b-f15b-53a0-8214ad3a693b [Candidate]" term=2
2025-01-26T02:45:05.771Z [INFO]  server-c677fdc5-ca0b-f15b-53a0-8214ad3a693b: election won: term=2 tally=2
2025-01-26T02:45:05.771Z [INFO]  server-c677fdc5-ca0b-f15b-53a0-8214ad3a693b: entering leader state: leader="Node at c677fdc5-ca0b-f15b-53a0-8214ad3a693b [Leader]"
2025-01-26T02:45:05.771Z [INFO]  server-c677fdc5-ca0b-f15b-53a0-8214ad3a693b: added peer, starting replication: peer=server-4db61052-fa17-73de-074f-f34069791350
2025-01-26T02:45:05.771Z [INFO]  server-c677fdc5-ca0b-f15b-53a0-8214ad3a693b: added peer, starting replication: peer=server-8efc80f0-75af-1c80-22ab-481dd115ebee
2025-01-26T02:45:05.771Z [INFO]  server-c677fdc5-ca0b-f15b-53a0-8214ad3a693b: pipelining replication: peer="{Voter server-8efc80f0-75af-1c80-22ab-481dd115ebee 8efc80f0-75af-1c80-22ab-481dd115ebee}"
2025-01-26T02:45:05.772Z [INFO]  server-c677fdc5-ca0b-f15b-53a0-8214ad3a693b: pipelining replication: peer="{Voter server-4db61052-fa17-73de-074f-f34069791350 4db61052-fa17-73de-074f-f34069791350}"
2025-01-26T02:45:05.879Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:05.772086016 +0000 UTC m=+102.287118391 (2 nodes), 67.918212ms from start of poll, 68.102453ms from cluster start. Timeout at 2025-01-26 02:45:05.877098856 +0000 UTC m=+102.392131191, 105.0128ms after stability
2025-01-26T02:45:05.879Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:05.987Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:05.879504948 +0000 UTC m=+102.394537323 (1 nodes), 1.28µs from start of poll, 175.521385ms from cluster start. Timeout at 2025-01-26 02:45:05.984531868 +0000 UTC m=+102.499564243, 105.02692ms after stability
2025-01-26T02:45:05.987Z [WARN]  server-4db61052-fa17-73de-074f-f34069791350: clearing log suffix: from=1 to=2
2025-01-26T02:45:05.987Z [WARN]  server-4db61052-fa17-73de-074f-f34069791350: skipping application of old log: index=1
2025-01-26T02:45:05.987Z [WARN]  server-4db61052-fa17-73de-074f-f34069791350: skipping application of old log: index=1
2025-01-26T02:45:05.987Z [INFO]  server-c677fdc5-ca0b-f15b-53a0-8214ad3a693b: aborting pipeline replication: peer="{Voter server-8efc80f0-75af-1c80-22ab-481dd115ebee 8efc80f0-75af-1c80-22ab-481dd115ebee}"
2025-01-26T02:45:05.991Z [INFO]  server-c677fdc5-ca0b-f15b-53a0-8214ad3a693b: aborting pipeline replication: peer="{Voter server-4db61052-fa17-73de-074f-f34069791350 4db61052-fa17-73de-074f-f34069791350}"
--- PASS: TestRaft_AppendEntry (0.29s)
=== RUN   TestRaft_VotingGrant_WhenLeaderAvailable
2025-01-26T02:45:05.992Z [INFO]  server-d55fd590-dd01-025e-5b7d-a479f16137fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d55fd590-dd01-025e-5b7d-a479f16137fc Address:d55fd590-dd01-025e-5b7d-a479f16137fc} {Suffrage:Voter ID:server-945d0ba4-e5cf-aebc-e395-818c3a5622c8 Address:945d0ba4-e5cf-aebc-e395-818c3a5622c8} {Suffrage:Voter ID:server-03e4a4ee-ca4e-0cca-2989-c39358dc1c01 Address:03e4a4ee-ca4e-0cca-2989-c39358dc1c01}]"
2025-01-26T02:45:05.992Z [INFO]  server-945d0ba4-e5cf-aebc-e395-818c3a5622c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d55fd590-dd01-025e-5b7d-a479f16137fc Address:d55fd590-dd01-025e-5b7d-a479f16137fc} {Suffrage:Voter ID:server-945d0ba4-e5cf-aebc-e395-818c3a5622c8 Address:945d0ba4-e5cf-aebc-e395-818c3a5622c8} {Suffrage:Voter ID:server-03e4a4ee-ca4e-0cca-2989-c39358dc1c01 Address:03e4a4ee-ca4e-0cca-2989-c39358dc1c01}]"
2025-01-26T02:45:05.992Z [INFO]  server-03e4a4ee-ca4e-0cca-2989-c39358dc1c01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d55fd590-dd01-025e-5b7d-a479f16137fc Address:d55fd590-dd01-025e-5b7d-a479f16137fc} {Suffrage:Voter ID:server-945d0ba4-e5cf-aebc-e395-818c3a5622c8 Address:945d0ba4-e5cf-aebc-e395-818c3a5622c8} {Suffrage:Voter ID:server-03e4a4ee-ca4e-0cca-2989-c39358dc1c01 Address:03e4a4ee-ca4e-0cca-2989-c39358dc1c01}]"
2025-01-26T02:45:05.992Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:05.992Z [INFO]  server-d55fd590-dd01-025e-5b7d-a479f16137fc: entering follower state: follower="Node at d55fd590-dd01-025e-5b7d-a479f16137fc [Follower]" leader-address= leader-id=
2025-01-26T02:45:05.992Z [INFO]  server-945d0ba4-e5cf-aebc-e395-818c3a5622c8: entering follower state: follower="Node at 945d0ba4-e5cf-aebc-e395-818c3a5622c8 [Follower]" leader-address= leader-id=
2025-01-26T02:45:05.992Z [INFO]  server-03e4a4ee-ca4e-0cca-2989-c39358dc1c01: entering follower state: follower="Node at 03e4a4ee-ca4e-0cca-2989-c39358dc1c01 [Follower]" leader-address= leader-id=
2025-01-26T02:45:06.043Z [WARN]  server-03e4a4ee-ca4e-0cca-2989-c39358dc1c01: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:06.043Z [INFO]  server-03e4a4ee-ca4e-0cca-2989-c39358dc1c01: entering candidate state: node="Node at 03e4a4ee-ca4e-0cca-2989-c39358dc1c01 [Candidate]" term=2
2025-01-26T02:45:06.043Z [INFO]  server-03e4a4ee-ca4e-0cca-2989-c39358dc1c01: election won: term=2 tally=2
2025-01-26T02:45:06.043Z [INFO]  server-03e4a4ee-ca4e-0cca-2989-c39358dc1c01: entering leader state: leader="Node at 03e4a4ee-ca4e-0cca-2989-c39358dc1c01 [Leader]"
2025-01-26T02:45:06.043Z [INFO]  server-03e4a4ee-ca4e-0cca-2989-c39358dc1c01: added peer, starting replication: peer=server-d55fd590-dd01-025e-5b7d-a479f16137fc
2025-01-26T02:45:06.043Z [INFO]  server-03e4a4ee-ca4e-0cca-2989-c39358dc1c01: added peer, starting replication: peer=server-945d0ba4-e5cf-aebc-e395-818c3a5622c8
2025-01-26T02:45:06.043Z [INFO]  server-03e4a4ee-ca4e-0cca-2989-c39358dc1c01: pipelining replication: peer="{Voter server-945d0ba4-e5cf-aebc-e395-818c3a5622c8 945d0ba4-e5cf-aebc-e395-818c3a5622c8}"
2025-01-26T02:45:06.043Z [INFO]  server-03e4a4ee-ca4e-0cca-2989-c39358dc1c01: pipelining replication: peer="{Voter server-d55fd590-dd01-025e-5b7d-a479f16137fc d55fd590-dd01-025e-5b7d-a479f16137fc}"
2025-01-26T02:45:06.152Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:06.043963545 +0000 UTC m=+102.558995920 (2 nodes), 51.797754ms from start of poll, 51.981835ms from cluster start. Timeout at 2025-01-26 02:45:06.148989145 +0000 UTC m=+102.664021520, 105.0256ms after stability
2025-01-26T02:45:06.152Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:06.259Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:06.152201722 +0000 UTC m=+102.667234057 (1 nodes), 2.64µs from start of poll, 160.219972ms from cluster start. Timeout at 2025-01-26 02:45:06.257230563 +0000 UTC m=+102.772262938, 105.028881ms after stability
2025-01-26T02:45:06.259Z [WARN]  server-945d0ba4-e5cf-aebc-e395-818c3a5622c8: rejecting vote request since we have a leader: from=d55fd590-dd01-025e-5b7d-a479f16137fc leader=03e4a4ee-ca4e-0cca-2989-c39358dc1c01 leader-id=server-03e4a4ee-ca4e-0cca-2989-c39358dc1c01
2025-01-26T02:45:06.259Z [INFO]  server-03e4a4ee-ca4e-0cca-2989-c39358dc1c01: aborting pipeline replication: peer="{Voter server-945d0ba4-e5cf-aebc-e395-818c3a5622c8 945d0ba4-e5cf-aebc-e395-818c3a5622c8}"
2025-01-26T02:45:06.259Z [INFO]  server-03e4a4ee-ca4e-0cca-2989-c39358dc1c01: aborting pipeline replication: peer="{Voter server-d55fd590-dd01-025e-5b7d-a479f16137fc d55fd590-dd01-025e-5b7d-a479f16137fc}"
--- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.27s)
=== RUN   TestRaft_ProtocolVersion_RejectRPC
2025-01-26T02:45:06.260Z [INFO]  server-731764eb-cdf6-e59e-eb8a-8fb4320772d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-731764eb-cdf6-e59e-eb8a-8fb4320772d8 Address:731764eb-cdf6-e59e-eb8a-8fb4320772d8} {Suffrage:Voter ID:server-fe1c2f7d-793e-31d5-672e-5b14af748dc2 Address:fe1c2f7d-793e-31d5-672e-5b14af748dc2} {Suffrage:Voter ID:server-2a4a3ad8-cef7-dd81-7c33-e4d811d865bf Address:2a4a3ad8-cef7-dd81-7c33-e4d811d865bf}]"
2025-01-26T02:45:06.260Z [INFO]  server-fe1c2f7d-793e-31d5-672e-5b14af748dc2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-731764eb-cdf6-e59e-eb8a-8fb4320772d8 Address:731764eb-cdf6-e59e-eb8a-8fb4320772d8} {Suffrage:Voter ID:server-fe1c2f7d-793e-31d5-672e-5b14af748dc2 Address:fe1c2f7d-793e-31d5-672e-5b14af748dc2} {Suffrage:Voter ID:server-2a4a3ad8-cef7-dd81-7c33-e4d811d865bf Address:2a4a3ad8-cef7-dd81-7c33-e4d811d865bf}]"
2025-01-26T02:45:06.260Z [INFO]  server-2a4a3ad8-cef7-dd81-7c33-e4d811d865bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-731764eb-cdf6-e59e-eb8a-8fb4320772d8 Address:731764eb-cdf6-e59e-eb8a-8fb4320772d8} {Suffrage:Voter ID:server-fe1c2f7d-793e-31d5-672e-5b14af748dc2 Address:fe1c2f7d-793e-31d5-672e-5b14af748dc2} {Suffrage:Voter ID:server-2a4a3ad8-cef7-dd81-7c33-e4d811d865bf Address:2a4a3ad8-cef7-dd81-7c33-e4d811d865bf}]"
2025-01-26T02:45:06.260Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:06.260Z [INFO]  server-731764eb-cdf6-e59e-eb8a-8fb4320772d8: entering follower state: follower="Node at 731764eb-cdf6-e59e-eb8a-8fb4320772d8 [Follower]" leader-address= leader-id=
2025-01-26T02:45:06.260Z [INFO]  server-fe1c2f7d-793e-31d5-672e-5b14af748dc2: entering follower state: follower="Node at fe1c2f7d-793e-31d5-672e-5b14af748dc2 [Follower]" leader-address= leader-id=
2025-01-26T02:45:06.260Z [INFO]  server-2a4a3ad8-cef7-dd81-7c33-e4d811d865bf: entering follower state: follower="Node at 2a4a3ad8-cef7-dd81-7c33-e4d811d865bf [Follower]" leader-address= leader-id=
2025-01-26T02:45:06.315Z [WARN]  server-731764eb-cdf6-e59e-eb8a-8fb4320772d8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:06.315Z [INFO]  server-731764eb-cdf6-e59e-eb8a-8fb4320772d8: entering candidate state: node="Node at 731764eb-cdf6-e59e-eb8a-8fb4320772d8 [Candidate]" term=2
2025-01-26T02:45:06.315Z [INFO]  server-731764eb-cdf6-e59e-eb8a-8fb4320772d8: election won: term=2 tally=2
2025-01-26T02:45:06.315Z [INFO]  server-731764eb-cdf6-e59e-eb8a-8fb4320772d8: entering leader state: leader="Node at 731764eb-cdf6-e59e-eb8a-8fb4320772d8 [Leader]"
2025-01-26T02:45:06.315Z [INFO]  server-731764eb-cdf6-e59e-eb8a-8fb4320772d8: added peer, starting replication: peer=server-fe1c2f7d-793e-31d5-672e-5b14af748dc2
2025-01-26T02:45:06.315Z [INFO]  server-731764eb-cdf6-e59e-eb8a-8fb4320772d8: added peer, starting replication: peer=server-2a4a3ad8-cef7-dd81-7c33-e4d811d865bf
2025-01-26T02:45:06.315Z [INFO]  server-731764eb-cdf6-e59e-eb8a-8fb4320772d8: pipelining replication: peer="{Voter server-2a4a3ad8-cef7-dd81-7c33-e4d811d865bf 2a4a3ad8-cef7-dd81-7c33-e4d811d865bf}"
2025-01-26T02:45:06.315Z [INFO]  server-731764eb-cdf6-e59e-eb8a-8fb4320772d8: pipelining replication: peer="{Voter server-fe1c2f7d-793e-31d5-672e-5b14af748dc2 fe1c2f7d-793e-31d5-672e-5b14af748dc2}"
2025-01-26T02:45:06.423Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:06.315828835 +0000 UTC m=+102.830861210 (2 nodes), 55.518297ms from start of poll, 55.701938ms from cluster start. Timeout at 2025-01-26 02:45:06.420858955 +0000 UTC m=+102.935891330, 105.03012ms after stability
2025-01-26T02:45:06.424Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:06.531Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:06.424011332 +0000 UTC m=+102.939043707 (1 nodes), 1.32µs from start of poll, 163.884435ms from cluster start. Timeout at 2025-01-26 02:45:06.529031492 +0000 UTC m=+103.044063987, 105.02028ms after stability
2025-01-26T02:45:06.531Z [INFO]  server-731764eb-cdf6-e59e-eb8a-8fb4320772d8: aborting pipeline replication: peer="{Voter server-2a4a3ad8-cef7-dd81-7c33-e4d811d865bf 2a4a3ad8-cef7-dd81-7c33-e4d811d865bf}"
2025-01-26T02:45:06.531Z [INFO]  server-731764eb-cdf6-e59e-eb8a-8fb4320772d8: aborting pipeline replication: peer="{Voter server-fe1c2f7d-793e-31d5-672e-5b14af748dc2 fe1c2f7d-793e-31d5-672e-5b14af748dc2}"
--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_1_2
    raft_test.go:2151: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_2_3
    raft_test.go:2196: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s)
=== RUN   TestRaft_LeaderID_Propagated
2025-01-26T02:45:06.532Z [INFO]  server-d2ddc37b-c896-123f-12f5-1b1e56822d31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2ddc37b-c896-123f-12f5-1b1e56822d31 Address:d2ddc37b-c896-123f-12f5-1b1e56822d31} {Suffrage:Voter ID:server-7a24ddf2-4191-5ee4-5615-95d446c85313 Address:7a24ddf2-4191-5ee4-5615-95d446c85313} {Suffrage:Voter ID:server-50227587-8df1-1cee-7cc6-a4473c913dc1 Address:50227587-8df1-1cee-7cc6-a4473c913dc1}]"
2025-01-26T02:45:06.532Z [INFO]  server-7a24ddf2-4191-5ee4-5615-95d446c85313: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2ddc37b-c896-123f-12f5-1b1e56822d31 Address:d2ddc37b-c896-123f-12f5-1b1e56822d31} {Suffrage:Voter ID:server-7a24ddf2-4191-5ee4-5615-95d446c85313 Address:7a24ddf2-4191-5ee4-5615-95d446c85313} {Suffrage:Voter ID:server-50227587-8df1-1cee-7cc6-a4473c913dc1 Address:50227587-8df1-1cee-7cc6-a4473c913dc1}]"
2025-01-26T02:45:06.532Z [INFO]  server-50227587-8df1-1cee-7cc6-a4473c913dc1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d2ddc37b-c896-123f-12f5-1b1e56822d31 Address:d2ddc37b-c896-123f-12f5-1b1e56822d31} {Suffrage:Voter ID:server-7a24ddf2-4191-5ee4-5615-95d446c85313 Address:7a24ddf2-4191-5ee4-5615-95d446c85313} {Suffrage:Voter ID:server-50227587-8df1-1cee-7cc6-a4473c913dc1 Address:50227587-8df1-1cee-7cc6-a4473c913dc1}]"
2025-01-26T02:45:06.532Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:06.532Z [INFO]  server-d2ddc37b-c896-123f-12f5-1b1e56822d31: entering follower state: follower="Node at d2ddc37b-c896-123f-12f5-1b1e56822d31 [Follower]" leader-address= leader-id=
2025-01-26T02:45:06.532Z [INFO]  server-7a24ddf2-4191-5ee4-5615-95d446c85313: entering follower state: follower="Node at 7a24ddf2-4191-5ee4-5615-95d446c85313 [Follower]" leader-address= leader-id=
2025-01-26T02:45:06.532Z [INFO]  server-50227587-8df1-1cee-7cc6-a4473c913dc1: entering follower state: follower="Node at 50227587-8df1-1cee-7cc6-a4473c913dc1 [Follower]" leader-address= leader-id=
2025-01-26T02:45:06.599Z [WARN]  server-d2ddc37b-c896-123f-12f5-1b1e56822d31: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:06.599Z [INFO]  server-d2ddc37b-c896-123f-12f5-1b1e56822d31: entering candidate state: node="Node at d2ddc37b-c896-123f-12f5-1b1e56822d31 [Candidate]" term=2
2025-01-26T02:45:06.599Z [INFO]  server-d2ddc37b-c896-123f-12f5-1b1e56822d31: election won: term=2 tally=2
2025-01-26T02:45:06.599Z [INFO]  server-d2ddc37b-c896-123f-12f5-1b1e56822d31: entering leader state: leader="Node at d2ddc37b-c896-123f-12f5-1b1e56822d31 [Leader]"
2025-01-26T02:45:06.599Z [INFO]  server-d2ddc37b-c896-123f-12f5-1b1e56822d31: added peer, starting replication: peer=server-7a24ddf2-4191-5ee4-5615-95d446c85313
2025-01-26T02:45:06.599Z [INFO]  server-d2ddc37b-c896-123f-12f5-1b1e56822d31: added peer, starting replication: peer=server-50227587-8df1-1cee-7cc6-a4473c913dc1
2025-01-26T02:45:06.599Z [INFO]  server-d2ddc37b-c896-123f-12f5-1b1e56822d31: pipelining replication: peer="{Voter server-7a24ddf2-4191-5ee4-5615-95d446c85313 7a24ddf2-4191-5ee4-5615-95d446c85313}"
2025-01-26T02:45:06.599Z [INFO]  server-d2ddc37b-c896-123f-12f5-1b1e56822d31: pipelining replication: peer="{Voter server-50227587-8df1-1cee-7cc6-a4473c913dc1 50227587-8df1-1cee-7cc6-a4473c913dc1}"
2025-01-26T02:45:06.708Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:06.599811118 +0000 UTC m=+103.114843493 (1 nodes), 67.66673ms from start of poll, 67.838771ms from cluster start. Timeout at 2025-01-26 02:45:06.704821959 +0000 UTC m=+103.219854334, 105.010841ms after stability
2025-01-26T02:45:06.708Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:06.815Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:06.708380217 +0000 UTC m=+103.223412592 (1 nodes), 1.44µs from start of poll, 176.40787ms from cluster start. Timeout at 2025-01-26 02:45:06.813412657 +0000 UTC m=+103.328445032, 105.03244ms after stability
2025-01-26T02:45:06.815Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:06.923Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:06.815660308 +0000 UTC m=+103.330692643 (1 nodes), 640ns from start of poll, 283.687921ms from cluster start. Timeout at 2025-01-26 02:45:06.920698228 +0000 UTC m=+103.435730603, 105.03796ms after stability
2025-01-26T02:45:06.923Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:07.031Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:06.923672083 +0000 UTC m=+103.438704458 (1 nodes), 600ns from start of poll, 391.699736ms from cluster start. Timeout at 2025-01-26 02:45:07.028713003 +0000 UTC m=+103.543745378, 105.04092ms after stability
2025-01-26T02:45:07.031Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:07.139Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:07.031696899 +0000 UTC m=+103.546729274 (1 nodes), 680ns from start of poll, 499.724552ms from cluster start. Timeout at 2025-01-26 02:45:07.136732659 +0000 UTC m=+103.651764994, 105.03572ms after stability
2025-01-26T02:45:07.139Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:07.247Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:07.139931436 +0000 UTC m=+103.654963811 (1 nodes), 520ns from start of poll, 607.959089ms from cluster start. Timeout at 2025-01-26 02:45:07.244970197 +0000 UTC m=+103.760002572, 105.038761ms after stability
2025-01-26T02:45:07.355Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:07.467Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:07.355474064 +0000 UTC m=+103.870506439 (1 nodes), 2.32µs from start of poll, 823.501717ms from cluster start. Timeout at 2025-01-26 02:45:07.460517424 +0000 UTC m=+103.975549799, 105.04336ms after stability
2025-01-26T02:45:07.467Z [INFO]  server-d2ddc37b-c896-123f-12f5-1b1e56822d31: aborting pipeline replication: peer="{Voter server-50227587-8df1-1cee-7cc6-a4473c913dc1 50227587-8df1-1cee-7cc6-a4473c913dc1}"
2025-01-26T02:45:07.467Z [INFO]  server-d2ddc37b-c896-123f-12f5-1b1e56822d31: aborting pipeline replication: peer="{Voter server-7a24ddf2-4191-5ee4-5615-95d446c85313 7a24ddf2-4191-5ee4-5615-95d446c85313}"
2025-01-26T02:45:07.967Z [ERROR] server-d2ddc37b-c896-123f-12f5-1b1e56822d31: failed to heartbeat to: peer=50227587-8df1-1cee-7cc6-a4473c913dc1 backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeaderID_Propagated (1.44s)
=== RUN   TestRaft_LeadershipTransferInProgress
--- PASS: TestRaft_LeadershipTransferInProgress (0.00s)
=== RUN   TestRaft_LeadershipTransferPickServer
--- PASS: TestRaft_LeadershipTransferPickServer (0.00s)
=== RUN   TestRaft_LeadershipTransfer
2025-01-26T02:45:07.968Z [INFO]  server-a51a044b-d3bb-1d72-d00f-6a82534c23a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a51a044b-d3bb-1d72-d00f-6a82534c23a5 Address:a51a044b-d3bb-1d72-d00f-6a82534c23a5} {Suffrage:Voter ID:server-f39c29d6-2c26-c9b8-bd01-2c59211261db Address:f39c29d6-2c26-c9b8-bd01-2c59211261db} {Suffrage:Voter ID:server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6 Address:e26bd9c2-34ed-b061-e1da-a7a661ab81c6}]"
2025-01-26T02:45:07.968Z [INFO]  server-f39c29d6-2c26-c9b8-bd01-2c59211261db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a51a044b-d3bb-1d72-d00f-6a82534c23a5 Address:a51a044b-d3bb-1d72-d00f-6a82534c23a5} {Suffrage:Voter ID:server-f39c29d6-2c26-c9b8-bd01-2c59211261db Address:f39c29d6-2c26-c9b8-bd01-2c59211261db} {Suffrage:Voter ID:server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6 Address:e26bd9c2-34ed-b061-e1da-a7a661ab81c6}]"
2025-01-26T02:45:07.968Z [INFO]  server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a51a044b-d3bb-1d72-d00f-6a82534c23a5 Address:a51a044b-d3bb-1d72-d00f-6a82534c23a5} {Suffrage:Voter ID:server-f39c29d6-2c26-c9b8-bd01-2c59211261db Address:f39c29d6-2c26-c9b8-bd01-2c59211261db} {Suffrage:Voter ID:server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6 Address:e26bd9c2-34ed-b061-e1da-a7a661ab81c6}]"
2025-01-26T02:45:07.968Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:07.968Z [INFO]  server-a51a044b-d3bb-1d72-d00f-6a82534c23a5: entering follower state: follower="Node at a51a044b-d3bb-1d72-d00f-6a82534c23a5 [Follower]" leader-address= leader-id=
2025-01-26T02:45:07.968Z [INFO]  server-f39c29d6-2c26-c9b8-bd01-2c59211261db: entering follower state: follower="Node at f39c29d6-2c26-c9b8-bd01-2c59211261db [Follower]" leader-address= leader-id=
2025-01-26T02:45:07.968Z [INFO]  server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6: entering follower state: follower="Node at e26bd9c2-34ed-b061-e1da-a7a661ab81c6 [Follower]" leader-address= leader-id=
2025-01-26T02:45:08.051Z [WARN]  server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:08.051Z [INFO]  server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6: entering candidate state: node="Node at e26bd9c2-34ed-b061-e1da-a7a661ab81c6 [Candidate]" term=2
2025-01-26T02:45:08.051Z [INFO]  server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6: election won: term=2 tally=2
2025-01-26T02:45:08.051Z [INFO]  server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6: entering leader state: leader="Node at e26bd9c2-34ed-b061-e1da-a7a661ab81c6 [Leader]"
2025-01-26T02:45:08.051Z [INFO]  server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6: added peer, starting replication: peer=server-a51a044b-d3bb-1d72-d00f-6a82534c23a5
2025-01-26T02:45:08.051Z [INFO]  server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6: added peer, starting replication: peer=server-f39c29d6-2c26-c9b8-bd01-2c59211261db
2025-01-26T02:45:08.051Z [INFO]  server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6: pipelining replication: peer="{Voter server-f39c29d6-2c26-c9b8-bd01-2c59211261db f39c29d6-2c26-c9b8-bd01-2c59211261db}"
2025-01-26T02:45:08.052Z [INFO]  server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6: pipelining replication: peer="{Voter server-a51a044b-d3bb-1d72-d00f-6a82534c23a5 a51a044b-d3bb-1d72-d00f-6a82534c23a5}"
2025-01-26T02:45:08.160Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:08.052070491 +0000 UTC m=+104.567102866 (1 nodes), 83.875029ms from start of poll, 84.063991ms from cluster start. Timeout at 2025-01-26 02:45:08.157072931 +0000 UTC m=+104.672105306, 105.00244ms after stability
2025-01-26T02:45:08.160Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:08.267Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:08.160179107 +0000 UTC m=+104.675211482 (1 nodes), 1.28µs from start of poll, 192.172607ms from cluster start. Timeout at 2025-01-26 02:45:08.265215707 +0000 UTC m=+104.780248082, 105.0366ms after stability
2025-01-26T02:45:08.267Z [INFO]  server-a51a044b-d3bb-1d72-d00f-6a82534c23a5: entering candidate state: node="Node at a51a044b-d3bb-1d72-d00f-6a82534c23a5 [Candidate]" term=3
2025-01-26T02:45:08.267Z [INFO]  server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6: entering follower state: follower="Node at e26bd9c2-34ed-b061-e1da-a7a661ab81c6 [Follower]" leader-address= leader-id=
2025-01-26T02:45:08.267Z [INFO]  server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6: aborting pipeline replication: peer="{Voter server-f39c29d6-2c26-c9b8-bd01-2c59211261db f39c29d6-2c26-c9b8-bd01-2c59211261db}"
2025-01-26T02:45:08.267Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:08.267Z [INFO]  server-a51a044b-d3bb-1d72-d00f-6a82534c23a5: election won: term=3 tally=2
2025-01-26T02:45:08.267Z [INFO]  server-a51a044b-d3bb-1d72-d00f-6a82534c23a5: entering leader state: leader="Node at a51a044b-d3bb-1d72-d00f-6a82534c23a5 [Leader]"
2025-01-26T02:45:08.267Z [INFO]  server-a51a044b-d3bb-1d72-d00f-6a82534c23a5: added peer, starting replication: peer=server-f39c29d6-2c26-c9b8-bd01-2c59211261db
2025-01-26T02:45:08.267Z [INFO]  server-a51a044b-d3bb-1d72-d00f-6a82534c23a5: added peer, starting replication: peer=server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6
2025-01-26T02:45:08.267Z [INFO]  server-a51a044b-d3bb-1d72-d00f-6a82534c23a5: pipelining replication: peer="{Voter server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6 e26bd9c2-34ed-b061-e1da-a7a661ab81c6}"
2025-01-26T02:45:08.267Z [INFO]  server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6: aborting pipeline replication: peer="{Voter server-a51a044b-d3bb-1d72-d00f-6a82534c23a5 a51a044b-d3bb-1d72-d00f-6a82534c23a5}"
2025-01-26T02:45:08.268Z [INFO]  server-a51a044b-d3bb-1d72-d00f-6a82534c23a5: pipelining replication: peer="{Voter server-f39c29d6-2c26-c9b8-bd01-2c59211261db f39c29d6-2c26-c9b8-bd01-2c59211261db}"
2025-01-26T02:45:08.380Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:08.267981801 +0000 UTC m=+104.783014176 (1 nodes), 134.481µs from start of poll, 299.975301ms from cluster start. Timeout at 2025-01-26 02:45:08.373035122 +0000 UTC m=+104.888067497, 105.053321ms after stability
2025-01-26T02:45:08.380Z [INFO]  server-a51a044b-d3bb-1d72-d00f-6a82534c23a5: aborting pipeline replication: peer="{Voter server-e26bd9c2-34ed-b061-e1da-a7a661ab81c6 e26bd9c2-34ed-b061-e1da-a7a661ab81c6}"
2025-01-26T02:45:08.380Z [INFO]  server-a51a044b-d3bb-1d72-d00f-6a82534c23a5: aborting pipeline replication: peer="{Voter server-f39c29d6-2c26-c9b8-bd01-2c59211261db f39c29d6-2c26-c9b8-bd01-2c59211261db}"
--- PASS: TestRaft_LeadershipTransfer (0.41s)
=== RUN   TestRaft_LeadershipTransferWithOneNode
2025-01-26T02:45:08.380Z [INFO]  server-45f66bf0-1813-92c7-ae43-816016f30ae2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45f66bf0-1813-92c7-ae43-816016f30ae2 Address:45f66bf0-1813-92c7-ae43-816016f30ae2}]"
2025-01-26T02:45:08.380Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:08.380Z [INFO]  server-45f66bf0-1813-92c7-ae43-816016f30ae2: entering follower state: follower="Node at 45f66bf0-1813-92c7-ae43-816016f30ae2 [Follower]" leader-address= leader-id=
2025-01-26T02:45:08.479Z [WARN]  server-45f66bf0-1813-92c7-ae43-816016f30ae2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:08.479Z [INFO]  server-45f66bf0-1813-92c7-ae43-816016f30ae2: entering candidate state: node="Node at 45f66bf0-1813-92c7-ae43-816016f30ae2 [Candidate]" term=2
2025-01-26T02:45:08.479Z [INFO]  server-45f66bf0-1813-92c7-ae43-816016f30ae2: election won: term=2 tally=1
2025-01-26T02:45:08.479Z [INFO]  server-45f66bf0-1813-92c7-ae43-816016f30ae2: entering leader state: leader="Node at 45f66bf0-1813-92c7-ae43-816016f30ae2 [Leader]"
2025-01-26T02:45:08.587Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:08.479592845 +0000 UTC m=+104.994625220 (1 nodes), 98.662239ms from start of poll, 98.75812ms from cluster start. Timeout at 2025-01-26 02:45:08.584595685 +0000 UTC m=+105.099628060, 105.00284ms after stability
--- PASS: TestRaft_LeadershipTransferWithOneNode (0.21s)
=== RUN   TestRaft_LeadershipTransferWithSevenNodes
2025-01-26T02:45:08.588Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7f0467a-2a2f-c21d-c431-a09a6aacff70 Address:c7f0467a-2a2f-c21d-c431-a09a6aacff70} {Suffrage:Voter ID:server-49c061ad-0983-d0a7-b836-304683c157a7 Address:49c061ad-0983-d0a7-b836-304683c157a7} {Suffrage:Voter ID:server-f0260d2b-3b8e-ffce-74e6-a645dbcad407 Address:f0260d2b-3b8e-ffce-74e6-a645dbcad407} {Suffrage:Voter ID:server-ca3f39cb-5bbe-e1b2-acd2-4037231cd268 Address:ca3f39cb-5bbe-e1b2-acd2-4037231cd268} {Suffrage:Voter ID:server-72670242-54b2-7ed5-9329-e7ae9a8c8d52 Address:72670242-54b2-7ed5-9329-e7ae9a8c8d52} {Suffrage:Voter ID:server-73212176-5c0b-5f0d-16b6-06539b6dc14c Address:73212176-5c0b-5f0d-16b6-06539b6dc14c} {Suffrage:Voter ID:server-f3196939-cce7-1418-1ddc-0ce46d3e275e Address:f3196939-cce7-1418-1ddc-0ce46d3e275e}]"
2025-01-26T02:45:08.588Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7f0467a-2a2f-c21d-c431-a09a6aacff70 Address:c7f0467a-2a2f-c21d-c431-a09a6aacff70} {Suffrage:Voter ID:server-49c061ad-0983-d0a7-b836-304683c157a7 Address:49c061ad-0983-d0a7-b836-304683c157a7} {Suffrage:Voter ID:server-f0260d2b-3b8e-ffce-74e6-a645dbcad407 Address:f0260d2b-3b8e-ffce-74e6-a645dbcad407} {Suffrage:Voter ID:server-ca3f39cb-5bbe-e1b2-acd2-4037231cd268 Address:ca3f39cb-5bbe-e1b2-acd2-4037231cd268} {Suffrage:Voter ID:server-72670242-54b2-7ed5-9329-e7ae9a8c8d52 Address:72670242-54b2-7ed5-9329-e7ae9a8c8d52} {Suffrage:Voter ID:server-73212176-5c0b-5f0d-16b6-06539b6dc14c Address:73212176-5c0b-5f0d-16b6-06539b6dc14c} {Suffrage:Voter ID:server-f3196939-cce7-1418-1ddc-0ce46d3e275e Address:f3196939-cce7-1418-1ddc-0ce46d3e275e}]"
2025-01-26T02:45:08.588Z [INFO]  server-f0260d2b-3b8e-ffce-74e6-a645dbcad407: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7f0467a-2a2f-c21d-c431-a09a6aacff70 Address:c7f0467a-2a2f-c21d-c431-a09a6aacff70} {Suffrage:Voter ID:server-49c061ad-0983-d0a7-b836-304683c157a7 Address:49c061ad-0983-d0a7-b836-304683c157a7} {Suffrage:Voter ID:server-f0260d2b-3b8e-ffce-74e6-a645dbcad407 Address:f0260d2b-3b8e-ffce-74e6-a645dbcad407} {Suffrage:Voter ID:server-ca3f39cb-5bbe-e1b2-acd2-4037231cd268 Address:ca3f39cb-5bbe-e1b2-acd2-4037231cd268} {Suffrage:Voter ID:server-72670242-54b2-7ed5-9329-e7ae9a8c8d52 Address:72670242-54b2-7ed5-9329-e7ae9a8c8d52} {Suffrage:Voter ID:server-73212176-5c0b-5f0d-16b6-06539b6dc14c Address:73212176-5c0b-5f0d-16b6-06539b6dc14c} {Suffrage:Voter ID:server-f3196939-cce7-1418-1ddc-0ce46d3e275e Address:f3196939-cce7-1418-1ddc-0ce46d3e275e}]"
2025-01-26T02:45:08.588Z [INFO]  server-ca3f39cb-5bbe-e1b2-acd2-4037231cd268: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7f0467a-2a2f-c21d-c431-a09a6aacff70 Address:c7f0467a-2a2f-c21d-c431-a09a6aacff70} {Suffrage:Voter ID:server-49c061ad-0983-d0a7-b836-304683c157a7 Address:49c061ad-0983-d0a7-b836-304683c157a7} {Suffrage:Voter ID:server-f0260d2b-3b8e-ffce-74e6-a645dbcad407 Address:f0260d2b-3b8e-ffce-74e6-a645dbcad407} {Suffrage:Voter ID:server-ca3f39cb-5bbe-e1b2-acd2-4037231cd268 Address:ca3f39cb-5bbe-e1b2-acd2-4037231cd268} {Suffrage:Voter ID:server-72670242-54b2-7ed5-9329-e7ae9a8c8d52 Address:72670242-54b2-7ed5-9329-e7ae9a8c8d52} {Suffrage:Voter ID:server-73212176-5c0b-5f0d-16b6-06539b6dc14c Address:73212176-5c0b-5f0d-16b6-06539b6dc14c} {Suffrage:Voter ID:server-f3196939-cce7-1418-1ddc-0ce46d3e275e Address:f3196939-cce7-1418-1ddc-0ce46d3e275e}]"
2025-01-26T02:45:08.588Z [INFO]  server-72670242-54b2-7ed5-9329-e7ae9a8c8d52: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7f0467a-2a2f-c21d-c431-a09a6aacff70 Address:c7f0467a-2a2f-c21d-c431-a09a6aacff70} {Suffrage:Voter ID:server-49c061ad-0983-d0a7-b836-304683c157a7 Address:49c061ad-0983-d0a7-b836-304683c157a7} {Suffrage:Voter ID:server-f0260d2b-3b8e-ffce-74e6-a645dbcad407 Address:f0260d2b-3b8e-ffce-74e6-a645dbcad407} {Suffrage:Voter ID:server-ca3f39cb-5bbe-e1b2-acd2-4037231cd268 Address:ca3f39cb-5bbe-e1b2-acd2-4037231cd268} {Suffrage:Voter ID:server-72670242-54b2-7ed5-9329-e7ae9a8c8d52 Address:72670242-54b2-7ed5-9329-e7ae9a8c8d52} {Suffrage:Voter ID:server-73212176-5c0b-5f0d-16b6-06539b6dc14c Address:73212176-5c0b-5f0d-16b6-06539b6dc14c} {Suffrage:Voter ID:server-f3196939-cce7-1418-1ddc-0ce46d3e275e Address:f3196939-cce7-1418-1ddc-0ce46d3e275e}]"
2025-01-26T02:45:08.588Z [INFO]  server-73212176-5c0b-5f0d-16b6-06539b6dc14c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7f0467a-2a2f-c21d-c431-a09a6aacff70 Address:c7f0467a-2a2f-c21d-c431-a09a6aacff70} {Suffrage:Voter ID:server-49c061ad-0983-d0a7-b836-304683c157a7 Address:49c061ad-0983-d0a7-b836-304683c157a7} {Suffrage:Voter ID:server-f0260d2b-3b8e-ffce-74e6-a645dbcad407 Address:f0260d2b-3b8e-ffce-74e6-a645dbcad407} {Suffrage:Voter ID:server-ca3f39cb-5bbe-e1b2-acd2-4037231cd268 Address:ca3f39cb-5bbe-e1b2-acd2-4037231cd268} {Suffrage:Voter ID:server-72670242-54b2-7ed5-9329-e7ae9a8c8d52 Address:72670242-54b2-7ed5-9329-e7ae9a8c8d52} {Suffrage:Voter ID:server-73212176-5c0b-5f0d-16b6-06539b6dc14c Address:73212176-5c0b-5f0d-16b6-06539b6dc14c} {Suffrage:Voter ID:server-f3196939-cce7-1418-1ddc-0ce46d3e275e Address:f3196939-cce7-1418-1ddc-0ce46d3e275e}]"
2025-01-26T02:45:08.588Z [INFO]  server-f3196939-cce7-1418-1ddc-0ce46d3e275e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7f0467a-2a2f-c21d-c431-a09a6aacff70 Address:c7f0467a-2a2f-c21d-c431-a09a6aacff70} {Suffrage:Voter ID:server-49c061ad-0983-d0a7-b836-304683c157a7 Address:49c061ad-0983-d0a7-b836-304683c157a7} {Suffrage:Voter ID:server-f0260d2b-3b8e-ffce-74e6-a645dbcad407 Address:f0260d2b-3b8e-ffce-74e6-a645dbcad407} {Suffrage:Voter ID:server-ca3f39cb-5bbe-e1b2-acd2-4037231cd268 Address:ca3f39cb-5bbe-e1b2-acd2-4037231cd268} {Suffrage:Voter ID:server-72670242-54b2-7ed5-9329-e7ae9a8c8d52 Address:72670242-54b2-7ed5-9329-e7ae9a8c8d52} {Suffrage:Voter ID:server-73212176-5c0b-5f0d-16b6-06539b6dc14c Address:73212176-5c0b-5f0d-16b6-06539b6dc14c} {Suffrage:Voter ID:server-f3196939-cce7-1418-1ddc-0ce46d3e275e Address:f3196939-cce7-1418-1ddc-0ce46d3e275e}]"
2025-01-26T02:45:08.588Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:08.588Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: entering follower state: follower="Node at c7f0467a-2a2f-c21d-c431-a09a6aacff70 [Follower]" leader-address= leader-id=
2025-01-26T02:45:08.588Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: entering follower state: follower="Node at 49c061ad-0983-d0a7-b836-304683c157a7 [Follower]" leader-address= leader-id=
2025-01-26T02:45:08.588Z [INFO]  server-f0260d2b-3b8e-ffce-74e6-a645dbcad407: entering follower state: follower="Node at f0260d2b-3b8e-ffce-74e6-a645dbcad407 [Follower]" leader-address= leader-id=
2025-01-26T02:45:08.588Z [INFO]  server-ca3f39cb-5bbe-e1b2-acd2-4037231cd268: entering follower state: follower="Node at ca3f39cb-5bbe-e1b2-acd2-4037231cd268 [Follower]" leader-address= leader-id=
2025-01-26T02:45:08.588Z [INFO]  server-72670242-54b2-7ed5-9329-e7ae9a8c8d52: entering follower state: follower="Node at 72670242-54b2-7ed5-9329-e7ae9a8c8d52 [Follower]" leader-address= leader-id=
2025-01-26T02:45:08.588Z [INFO]  server-73212176-5c0b-5f0d-16b6-06539b6dc14c: entering follower state: follower="Node at 73212176-5c0b-5f0d-16b6-06539b6dc14c [Follower]" leader-address= leader-id=
2025-01-26T02:45:08.588Z [INFO]  server-f3196939-cce7-1418-1ddc-0ce46d3e275e: entering follower state: follower="Node at f3196939-cce7-1418-1ddc-0ce46d3e275e [Follower]" leader-address= leader-id=
2025-01-26T02:45:08.643Z [WARN]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:08.643Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: entering candidate state: node="Node at c7f0467a-2a2f-c21d-c431-a09a6aacff70 [Candidate]" term=2
2025-01-26T02:45:08.643Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: election won: term=2 tally=4
2025-01-26T02:45:08.643Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: entering leader state: leader="Node at c7f0467a-2a2f-c21d-c431-a09a6aacff70 [Leader]"
2025-01-26T02:45:08.643Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: added peer, starting replication: peer=server-49c061ad-0983-d0a7-b836-304683c157a7
2025-01-26T02:45:08.643Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: added peer, starting replication: peer=server-f0260d2b-3b8e-ffce-74e6-a645dbcad407
2025-01-26T02:45:08.643Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: added peer, starting replication: peer=server-ca3f39cb-5bbe-e1b2-acd2-4037231cd268
2025-01-26T02:45:08.644Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: added peer, starting replication: peer=server-72670242-54b2-7ed5-9329-e7ae9a8c8d52
2025-01-26T02:45:08.644Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: added peer, starting replication: peer=server-73212176-5c0b-5f0d-16b6-06539b6dc14c
2025-01-26T02:45:08.644Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: added peer, starting replication: peer=server-f3196939-cce7-1418-1ddc-0ce46d3e275e
2025-01-26T02:45:08.644Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: pipelining replication: peer="{Voter server-f3196939-cce7-1418-1ddc-0ce46d3e275e f3196939-cce7-1418-1ddc-0ce46d3e275e}"
2025-01-26T02:45:08.644Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: pipelining replication: peer="{Voter server-49c061ad-0983-d0a7-b836-304683c157a7 49c061ad-0983-d0a7-b836-304683c157a7}"
2025-01-26T02:45:08.644Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: pipelining replication: peer="{Voter server-f0260d2b-3b8e-ffce-74e6-a645dbcad407 f0260d2b-3b8e-ffce-74e6-a645dbcad407}"
2025-01-26T02:45:08.644Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: pipelining replication: peer="{Voter server-ca3f39cb-5bbe-e1b2-acd2-4037231cd268 ca3f39cb-5bbe-e1b2-acd2-4037231cd268}"
2025-01-26T02:45:08.644Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: pipelining replication: peer="{Voter server-72670242-54b2-7ed5-9329-e7ae9a8c8d52 72670242-54b2-7ed5-9329-e7ae9a8c8d52}"
2025-01-26T02:45:08.644Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: pipelining replication: peer="{Voter server-73212176-5c0b-5f0d-16b6-06539b6dc14c 73212176-5c0b-5f0d-16b6-06539b6dc14c}"
2025-01-26T02:45:08.752Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:08.644303084 +0000 UTC m=+105.159335459 (1 nodes), 55.603577ms from start of poll, 56.03858ms from cluster start. Timeout at 2025-01-26 02:45:08.749306764 +0000 UTC m=+105.264339099, 105.00364ms after stability
2025-01-26T02:45:08.752Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:08.859Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:08.752073178 +0000 UTC m=+105.267105513 (6 nodes), 1.92µs from start of poll, 163.808634ms from cluster start. Timeout at 2025-01-26 02:45:08.857117618 +0000 UTC m=+105.372149993, 105.04448ms after stability
2025-01-26T02:45:08.859Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:08.967Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:08.85955647 +0000 UTC m=+105.374588845 (1 nodes), 1.24µs from start of poll, 271.291966ms from cluster start. Timeout at 2025-01-26 02:45:08.964591631 +0000 UTC m=+105.479624006, 105.035161ms after stability
2025-01-26T02:45:08.967Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: entering candidate state: node="Node at 49c061ad-0983-d0a7-b836-304683c157a7 [Candidate]" term=3
2025-01-26T02:45:08.967Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:08.967Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: entering follower state: follower="Node at c7f0467a-2a2f-c21d-c431-a09a6aacff70 [Follower]" leader-address= leader-id=
2025-01-26T02:45:08.967Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: aborting pipeline replication: peer="{Voter server-73212176-5c0b-5f0d-16b6-06539b6dc14c 73212176-5c0b-5f0d-16b6-06539b6dc14c}"
2025-01-26T02:45:08.967Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: election won: term=3 tally=4
2025-01-26T02:45:08.967Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: entering leader state: leader="Node at 49c061ad-0983-d0a7-b836-304683c157a7 [Leader]"
2025-01-26T02:45:08.967Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: added peer, starting replication: peer=server-c7f0467a-2a2f-c21d-c431-a09a6aacff70
2025-01-26T02:45:08.967Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: added peer, starting replication: peer=server-f0260d2b-3b8e-ffce-74e6-a645dbcad407
2025-01-26T02:45:08.967Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: added peer, starting replication: peer=server-ca3f39cb-5bbe-e1b2-acd2-4037231cd268
2025-01-26T02:45:08.967Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: added peer, starting replication: peer=server-72670242-54b2-7ed5-9329-e7ae9a8c8d52
2025-01-26T02:45:08.967Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: added peer, starting replication: peer=server-73212176-5c0b-5f0d-16b6-06539b6dc14c
2025-01-26T02:45:08.967Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: added peer, starting replication: peer=server-f3196939-cce7-1418-1ddc-0ce46d3e275e
2025-01-26T02:45:08.967Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: pipelining replication: peer="{Voter server-f3196939-cce7-1418-1ddc-0ce46d3e275e f3196939-cce7-1418-1ddc-0ce46d3e275e}"
2025-01-26T02:45:08.967Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: aborting pipeline replication: peer="{Voter server-f3196939-cce7-1418-1ddc-0ce46d3e275e f3196939-cce7-1418-1ddc-0ce46d3e275e}"
2025-01-26T02:45:08.967Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: aborting pipeline replication: peer="{Voter server-49c061ad-0983-d0a7-b836-304683c157a7 49c061ad-0983-d0a7-b836-304683c157a7}"
2025-01-26T02:45:08.967Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: aborting pipeline replication: peer="{Voter server-f0260d2b-3b8e-ffce-74e6-a645dbcad407 f0260d2b-3b8e-ffce-74e6-a645dbcad407}"
2025-01-26T02:45:08.967Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: aborting pipeline replication: peer="{Voter server-ca3f39cb-5bbe-e1b2-acd2-4037231cd268 ca3f39cb-5bbe-e1b2-acd2-4037231cd268}"
2025-01-26T02:45:08.967Z [INFO]  server-c7f0467a-2a2f-c21d-c431-a09a6aacff70: aborting pipeline replication: peer="{Voter server-72670242-54b2-7ed5-9329-e7ae9a8c8d52 72670242-54b2-7ed5-9329-e7ae9a8c8d52}"
2025-01-26T02:45:08.968Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: pipelining replication: peer="{Voter server-c7f0467a-2a2f-c21d-c431-a09a6aacff70 c7f0467a-2a2f-c21d-c431-a09a6aacff70}"
2025-01-26T02:45:08.968Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: pipelining replication: peer="{Voter server-f0260d2b-3b8e-ffce-74e6-a645dbcad407 f0260d2b-3b8e-ffce-74e6-a645dbcad407}"
2025-01-26T02:45:08.968Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: pipelining replication: peer="{Voter server-ca3f39cb-5bbe-e1b2-acd2-4037231cd268 ca3f39cb-5bbe-e1b2-acd2-4037231cd268}"
2025-01-26T02:45:08.968Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: pipelining replication: peer="{Voter server-72670242-54b2-7ed5-9329-e7ae9a8c8d52 72670242-54b2-7ed5-9329-e7ae9a8c8d52}"
2025-01-26T02:45:08.968Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: pipelining replication: peer="{Voter server-73212176-5c0b-5f0d-16b6-06539b6dc14c 73212176-5c0b-5f0d-16b6-06539b6dc14c}"
2025-01-26T02:45:09.073Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:08.967943208 +0000 UTC m=+105.482975583 (1 nodes), 262.602µs from start of poll, 379.678704ms from cluster start. Timeout at 2025-01-26 02:45:09.072971249 +0000 UTC m=+105.588003624, 105.028041ms after stability
2025-01-26T02:45:09.073Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: aborting pipeline replication: peer="{Voter server-73212176-5c0b-5f0d-16b6-06539b6dc14c 73212176-5c0b-5f0d-16b6-06539b6dc14c}"
2025-01-26T02:45:09.073Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: aborting pipeline replication: peer="{Voter server-f3196939-cce7-1418-1ddc-0ce46d3e275e f3196939-cce7-1418-1ddc-0ce46d3e275e}"
2025-01-26T02:45:09.073Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: aborting pipeline replication: peer="{Voter server-c7f0467a-2a2f-c21d-c431-a09a6aacff70 c7f0467a-2a2f-c21d-c431-a09a6aacff70}"
2025-01-26T02:45:09.073Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: aborting pipeline replication: peer="{Voter server-f0260d2b-3b8e-ffce-74e6-a645dbcad407 f0260d2b-3b8e-ffce-74e6-a645dbcad407}"
2025-01-26T02:45:09.073Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: aborting pipeline replication: peer="{Voter server-ca3f39cb-5bbe-e1b2-acd2-4037231cd268 ca3f39cb-5bbe-e1b2-acd2-4037231cd268}"
2025-01-26T02:45:09.073Z [INFO]  server-49c061ad-0983-d0a7-b836-304683c157a7: aborting pipeline replication: peer="{Voter server-72670242-54b2-7ed5-9329-e7ae9a8c8d52 72670242-54b2-7ed5-9329-e7ae9a8c8d52}"
--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.49s)
=== RUN   TestRaft_LeadershipTransferToInvalidID
2025-01-26T02:45:09.074Z [INFO]  server-f76fd8cc-86dc-60cf-6807-4e9cbca6dfdd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f76fd8cc-86dc-60cf-6807-4e9cbca6dfdd Address:f76fd8cc-86dc-60cf-6807-4e9cbca6dfdd} {Suffrage:Voter ID:server-50eb212c-d855-1772-0c90-38927775bb9f Address:50eb212c-d855-1772-0c90-38927775bb9f} {Suffrage:Voter ID:server-01b1d9ac-fcab-79f2-13c9-99056588da48 Address:01b1d9ac-fcab-79f2-13c9-99056588da48}]"
2025-01-26T02:45:09.074Z [INFO]  server-50eb212c-d855-1772-0c90-38927775bb9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f76fd8cc-86dc-60cf-6807-4e9cbca6dfdd Address:f76fd8cc-86dc-60cf-6807-4e9cbca6dfdd} {Suffrage:Voter ID:server-50eb212c-d855-1772-0c90-38927775bb9f Address:50eb212c-d855-1772-0c90-38927775bb9f} {Suffrage:Voter ID:server-01b1d9ac-fcab-79f2-13c9-99056588da48 Address:01b1d9ac-fcab-79f2-13c9-99056588da48}]"
2025-01-26T02:45:09.074Z [INFO]  server-01b1d9ac-fcab-79f2-13c9-99056588da48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f76fd8cc-86dc-60cf-6807-4e9cbca6dfdd Address:f76fd8cc-86dc-60cf-6807-4e9cbca6dfdd} {Suffrage:Voter ID:server-50eb212c-d855-1772-0c90-38927775bb9f Address:50eb212c-d855-1772-0c90-38927775bb9f} {Suffrage:Voter ID:server-01b1d9ac-fcab-79f2-13c9-99056588da48 Address:01b1d9ac-fcab-79f2-13c9-99056588da48}]"
2025-01-26T02:45:09.074Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:09.074Z [INFO]  server-f76fd8cc-86dc-60cf-6807-4e9cbca6dfdd: entering follower state: follower="Node at f76fd8cc-86dc-60cf-6807-4e9cbca6dfdd [Follower]" leader-address= leader-id=
2025-01-26T02:45:09.074Z [INFO]  server-50eb212c-d855-1772-0c90-38927775bb9f: entering follower state: follower="Node at 50eb212c-d855-1772-0c90-38927775bb9f [Follower]" leader-address= leader-id=
2025-01-26T02:45:09.074Z [INFO]  server-01b1d9ac-fcab-79f2-13c9-99056588da48: entering follower state: follower="Node at 01b1d9ac-fcab-79f2-13c9-99056588da48 [Follower]" leader-address= leader-id=
2025-01-26T02:45:09.125Z [WARN]  server-50eb212c-d855-1772-0c90-38927775bb9f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:09.125Z [INFO]  server-50eb212c-d855-1772-0c90-38927775bb9f: entering candidate state: node="Node at 50eb212c-d855-1772-0c90-38927775bb9f [Candidate]" term=2
2025-01-26T02:45:09.125Z [INFO]  server-50eb212c-d855-1772-0c90-38927775bb9f: election won: term=2 tally=2
2025-01-26T02:45:09.125Z [INFO]  server-50eb212c-d855-1772-0c90-38927775bb9f: entering leader state: leader="Node at 50eb212c-d855-1772-0c90-38927775bb9f [Leader]"
2025-01-26T02:45:09.125Z [INFO]  server-50eb212c-d855-1772-0c90-38927775bb9f: added peer, starting replication: peer=server-f76fd8cc-86dc-60cf-6807-4e9cbca6dfdd
2025-01-26T02:45:09.125Z [INFO]  server-50eb212c-d855-1772-0c90-38927775bb9f: added peer, starting replication: peer=server-01b1d9ac-fcab-79f2-13c9-99056588da48
2025-01-26T02:45:09.125Z [INFO]  server-50eb212c-d855-1772-0c90-38927775bb9f: pipelining replication: peer="{Voter server-01b1d9ac-fcab-79f2-13c9-99056588da48 01b1d9ac-fcab-79f2-13c9-99056588da48}"
2025-01-26T02:45:09.125Z [INFO]  server-50eb212c-d855-1772-0c90-38927775bb9f: pipelining replication: peer="{Voter server-f76fd8cc-86dc-60cf-6807-4e9cbca6dfdd f76fd8cc-86dc-60cf-6807-4e9cbca6dfdd}"
2025-01-26T02:45:09.232Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:09.125560244 +0000 UTC m=+105.640592619 (1 nodes), 51.23135ms from start of poll, 51.413032ms from cluster start. Timeout at 2025-01-26 02:45:09.230568684 +0000 UTC m=+105.745601059, 105.00844ms after stability
2025-01-26T02:45:09.232Z [INFO]  server-50eb212c-d855-1772-0c90-38927775bb9f: aborting pipeline replication: peer="{Voter server-01b1d9ac-fcab-79f2-13c9-99056588da48 01b1d9ac-fcab-79f2-13c9-99056588da48}"
2025-01-26T02:45:09.232Z [INFO]  server-50eb212c-d855-1772-0c90-38927775bb9f: aborting pipeline replication: peer="{Voter server-f76fd8cc-86dc-60cf-6807-4e9cbca6dfdd f76fd8cc-86dc-60cf-6807-4e9cbca6dfdd}"
--- PASS: TestRaft_LeadershipTransferToInvalidID (0.16s)
=== RUN   TestRaft_LeadershipTransferToInvalidAddress
2025-01-26T02:45:09.232Z [INFO]  server-5efed29c-34c2-8bde-e4d6-5398ab4c9480: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5efed29c-34c2-8bde-e4d6-5398ab4c9480 Address:5efed29c-34c2-8bde-e4d6-5398ab4c9480} {Suffrage:Voter ID:server-c209dee7-44a7-1af6-3ca9-e1e6bfa01bc4 Address:c209dee7-44a7-1af6-3ca9-e1e6bfa01bc4} {Suffrage:Voter ID:server-35032041-4bae-2d3a-b905-c00e09ac5da1 Address:35032041-4bae-2d3a-b905-c00e09ac5da1}]"
2025-01-26T02:45:09.232Z [INFO]  server-c209dee7-44a7-1af6-3ca9-e1e6bfa01bc4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5efed29c-34c2-8bde-e4d6-5398ab4c9480 Address:5efed29c-34c2-8bde-e4d6-5398ab4c9480} {Suffrage:Voter ID:server-c209dee7-44a7-1af6-3ca9-e1e6bfa01bc4 Address:c209dee7-44a7-1af6-3ca9-e1e6bfa01bc4} {Suffrage:Voter ID:server-35032041-4bae-2d3a-b905-c00e09ac5da1 Address:35032041-4bae-2d3a-b905-c00e09ac5da1}]"
2025-01-26T02:45:09.232Z [INFO]  server-35032041-4bae-2d3a-b905-c00e09ac5da1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5efed29c-34c2-8bde-e4d6-5398ab4c9480 Address:5efed29c-34c2-8bde-e4d6-5398ab4c9480} {Suffrage:Voter ID:server-c209dee7-44a7-1af6-3ca9-e1e6bfa01bc4 Address:c209dee7-44a7-1af6-3ca9-e1e6bfa01bc4} {Suffrage:Voter ID:server-35032041-4bae-2d3a-b905-c00e09ac5da1 Address:35032041-4bae-2d3a-b905-c00e09ac5da1}]"
2025-01-26T02:45:09.232Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:09.232Z [INFO]  server-5efed29c-34c2-8bde-e4d6-5398ab4c9480: entering follower state: follower="Node at 5efed29c-34c2-8bde-e4d6-5398ab4c9480 [Follower]" leader-address= leader-id=
2025-01-26T02:45:09.232Z [INFO]  server-c209dee7-44a7-1af6-3ca9-e1e6bfa01bc4: entering follower state: follower="Node at c209dee7-44a7-1af6-3ca9-e1e6bfa01bc4 [Follower]" leader-address= leader-id=
2025-01-26T02:45:09.232Z [INFO]  server-35032041-4bae-2d3a-b905-c00e09ac5da1: entering follower state: follower="Node at 35032041-4bae-2d3a-b905-c00e09ac5da1 [Follower]" leader-address= leader-id=
2025-01-26T02:45:09.294Z [WARN]  server-35032041-4bae-2d3a-b905-c00e09ac5da1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:09.294Z [INFO]  server-35032041-4bae-2d3a-b905-c00e09ac5da1: entering candidate state: node="Node at 35032041-4bae-2d3a-b905-c00e09ac5da1 [Candidate]" term=2
2025-01-26T02:45:09.294Z [INFO]  server-35032041-4bae-2d3a-b905-c00e09ac5da1: election won: term=2 tally=2
2025-01-26T02:45:09.294Z [INFO]  server-35032041-4bae-2d3a-b905-c00e09ac5da1: entering leader state: leader="Node at 35032041-4bae-2d3a-b905-c00e09ac5da1 [Leader]"
2025-01-26T02:45:09.294Z [INFO]  server-35032041-4bae-2d3a-b905-c00e09ac5da1: added peer, starting replication: peer=server-5efed29c-34c2-8bde-e4d6-5398ab4c9480
2025-01-26T02:45:09.294Z [INFO]  server-35032041-4bae-2d3a-b905-c00e09ac5da1: added peer, starting replication: peer=server-c209dee7-44a7-1af6-3ca9-e1e6bfa01bc4
2025-01-26T02:45:09.294Z [INFO]  server-35032041-4bae-2d3a-b905-c00e09ac5da1: pipelining replication: peer="{Voter server-c209dee7-44a7-1af6-3ca9-e1e6bfa01bc4 c209dee7-44a7-1af6-3ca9-e1e6bfa01bc4}"
2025-01-26T02:45:09.294Z [INFO]  server-35032041-4bae-2d3a-b905-c00e09ac5da1: pipelining replication: peer="{Voter server-5efed29c-34c2-8bde-e4d6-5398ab4c9480 5efed29c-34c2-8bde-e4d6-5398ab4c9480}"
2025-01-26T02:45:09.399Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:09.294678031 +0000 UTC m=+105.809710406 (2 nodes), 61.958416ms from start of poll, 62.138377ms from cluster start. Timeout at 2025-01-26 02:45:09.399680311 +0000 UTC m=+105.914712686, 105.00228ms after stability
2025-01-26T02:45:09.399Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:09.507Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:09.399877429 +0000 UTC m=+105.914909804 (1 nodes), 640ns from start of poll, 167.337775ms from cluster start. Timeout at 2025-01-26 02:45:09.504923549 +0000 UTC m=+106.019955884, 105.04608ms after stability
2025-01-26T02:45:09.507Z [INFO]  server-35032041-4bae-2d3a-b905-c00e09ac5da1: aborting pipeline replication: peer="{Voter server-c209dee7-44a7-1af6-3ca9-e1e6bfa01bc4 c209dee7-44a7-1af6-3ca9-e1e6bfa01bc4}"
2025-01-26T02:45:09.507Z [INFO]  server-35032041-4bae-2d3a-b905-c00e09ac5da1: aborting pipeline replication: peer="{Voter server-5efed29c-34c2-8bde-e4d6-5398ab4c9480 5efed29c-34c2-8bde-e4d6-5398ab4c9480}"
--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.28s)
=== RUN   TestRaft_LeadershipTransferToBehindServer
2025-01-26T02:45:09.508Z [INFO]  server-fca1639d-7715-55d7-9fd6-d311f586957a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fca1639d-7715-55d7-9fd6-d311f586957a Address:fca1639d-7715-55d7-9fd6-d311f586957a} {Suffrage:Voter ID:server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7 Address:8e15dcae-8fbe-109e-bac1-2fdd0b3437f7} {Suffrage:Voter ID:server-3d2f6d16-4826-71e0-11e2-e26000b00c2a Address:3d2f6d16-4826-71e0-11e2-e26000b00c2a}]"
2025-01-26T02:45:09.508Z [INFO]  server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fca1639d-7715-55d7-9fd6-d311f586957a Address:fca1639d-7715-55d7-9fd6-d311f586957a} {Suffrage:Voter ID:server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7 Address:8e15dcae-8fbe-109e-bac1-2fdd0b3437f7} {Suffrage:Voter ID:server-3d2f6d16-4826-71e0-11e2-e26000b00c2a Address:3d2f6d16-4826-71e0-11e2-e26000b00c2a}]"
2025-01-26T02:45:09.508Z [INFO]  server-3d2f6d16-4826-71e0-11e2-e26000b00c2a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fca1639d-7715-55d7-9fd6-d311f586957a Address:fca1639d-7715-55d7-9fd6-d311f586957a} {Suffrage:Voter ID:server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7 Address:8e15dcae-8fbe-109e-bac1-2fdd0b3437f7} {Suffrage:Voter ID:server-3d2f6d16-4826-71e0-11e2-e26000b00c2a Address:3d2f6d16-4826-71e0-11e2-e26000b00c2a}]"
2025-01-26T02:45:09.508Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:09.508Z [INFO]  server-fca1639d-7715-55d7-9fd6-d311f586957a: entering follower state: follower="Node at fca1639d-7715-55d7-9fd6-d311f586957a [Follower]" leader-address= leader-id=
2025-01-26T02:45:09.508Z [INFO]  server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7: entering follower state: follower="Node at 8e15dcae-8fbe-109e-bac1-2fdd0b3437f7 [Follower]" leader-address= leader-id=
2025-01-26T02:45:09.508Z [INFO]  server-3d2f6d16-4826-71e0-11e2-e26000b00c2a: entering follower state: follower="Node at 3d2f6d16-4826-71e0-11e2-e26000b00c2a [Follower]" leader-address= leader-id=
2025-01-26T02:45:09.563Z [WARN]  server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:09.563Z [INFO]  server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7: entering candidate state: node="Node at 8e15dcae-8fbe-109e-bac1-2fdd0b3437f7 [Candidate]" term=2
2025-01-26T02:45:09.563Z [INFO]  server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7: election won: term=2 tally=2
2025-01-26T02:45:09.563Z [INFO]  server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7: entering leader state: leader="Node at 8e15dcae-8fbe-109e-bac1-2fdd0b3437f7 [Leader]"
2025-01-26T02:45:09.563Z [INFO]  server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7: added peer, starting replication: peer=server-fca1639d-7715-55d7-9fd6-d311f586957a
2025-01-26T02:45:09.563Z [INFO]  server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7: added peer, starting replication: peer=server-3d2f6d16-4826-71e0-11e2-e26000b00c2a
2025-01-26T02:45:09.563Z [INFO]  server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7: pipelining replication: peer="{Voter server-3d2f6d16-4826-71e0-11e2-e26000b00c2a 3d2f6d16-4826-71e0-11e2-e26000b00c2a}"
2025-01-26T02:45:09.564Z [INFO]  server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7: pipelining replication: peer="{Voter server-fca1639d-7715-55d7-9fd6-d311f586957a fca1639d-7715-55d7-9fd6-d311f586957a}"
2025-01-26T02:45:09.671Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:09.564139826 +0000 UTC m=+106.079172161 (1 nodes), 55.891939ms from start of poll, 56.069741ms from cluster start. Timeout at 2025-01-26 02:45:09.669145946 +0000 UTC m=+106.184178321, 105.00616ms after stability
2025-01-26T02:45:09.671Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:09.785Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:09.671682758 +0000 UTC m=+106.186715133 (2 nodes), 1.52µs from start of poll, 163.612713ms from cluster start. Timeout at 2025-01-26 02:45:09.776741678 +0000 UTC m=+106.291774053, 105.05892ms after stability
2025-01-26T02:45:09.840Z [INFO]  server-fca1639d-7715-55d7-9fd6-d311f586957a: entering candidate state: node="Node at fca1639d-7715-55d7-9fd6-d311f586957a [Candidate]" term=3
2025-01-26T02:45:09.840Z [INFO]  server-fca1639d-7715-55d7-9fd6-d311f586957a: election won: term=3 tally=2
2025-01-26T02:45:09.840Z [INFO]  server-fca1639d-7715-55d7-9fd6-d311f586957a: entering leader state: leader="Node at fca1639d-7715-55d7-9fd6-d311f586957a [Leader]"
2025-01-26T02:45:09.840Z [INFO]  server-fca1639d-7715-55d7-9fd6-d311f586957a: added peer, starting replication: peer=server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7
2025-01-26T02:45:09.840Z [INFO]  server-fca1639d-7715-55d7-9fd6-d311f586957a: added peer, starting replication: peer=server-3d2f6d16-4826-71e0-11e2-e26000b00c2a
2025-01-26T02:45:09.840Z [INFO]  server-fca1639d-7715-55d7-9fd6-d311f586957a: pipelining replication: peer="{Voter server-3d2f6d16-4826-71e0-11e2-e26000b00c2a 3d2f6d16-4826-71e0-11e2-e26000b00c2a}"
2025-01-26T02:45:09.840Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:09.840Z [INFO]  server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7: entering follower state: follower="Node at 8e15dcae-8fbe-109e-bac1-2fdd0b3437f7 [Follower]" leader-address= leader-id=
2025-01-26T02:45:09.840Z [INFO]  server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7: aborting pipeline replication: peer="{Voter server-3d2f6d16-4826-71e0-11e2-e26000b00c2a 3d2f6d16-4826-71e0-11e2-e26000b00c2a}"
2025-01-26T02:45:09.841Z [INFO]  server-fca1639d-7715-55d7-9fd6-d311f586957a: pipelining replication: peer="{Voter server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7 8e15dcae-8fbe-109e-bac1-2fdd0b3437f7}"
2025-01-26T02:45:09.841Z [INFO]  server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7: aborting pipeline replication: peer="{Voter server-fca1639d-7715-55d7-9fd6-d311f586957a fca1639d-7715-55d7-9fd6-d311f586957a}"
2025-01-26T02:45:09.947Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:09.841038946 +0000 UTC m=+106.356071321 (1 nodes), 136.001µs from start of poll, 332.968901ms from cluster start. Timeout at 2025-01-26 02:45:09.946071986 +0000 UTC m=+106.461104361, 105.03304ms after stability
2025-01-26T02:45:09.947Z [INFO]  server-fca1639d-7715-55d7-9fd6-d311f586957a: aborting pipeline replication: peer="{Voter server-3d2f6d16-4826-71e0-11e2-e26000b00c2a 3d2f6d16-4826-71e0-11e2-e26000b00c2a}"
2025-01-26T02:45:09.947Z [INFO]  server-fca1639d-7715-55d7-9fd6-d311f586957a: aborting pipeline replication: peer="{Voter server-8e15dcae-8fbe-109e-bac1-2fdd0b3437f7 8e15dcae-8fbe-109e-bac1-2fdd0b3437f7}"
--- PASS: TestRaft_LeadershipTransferToBehindServer (0.44s)
=== RUN   TestRaft_LeadershipTransferToItself
2025-01-26T02:45:09.948Z [INFO]  server-7fbf6908-17dc-f9af-458d-21bd1f6f2182: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fbf6908-17dc-f9af-458d-21bd1f6f2182 Address:7fbf6908-17dc-f9af-458d-21bd1f6f2182} {Suffrage:Voter ID:server-3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4 Address:3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4} {Suffrage:Voter ID:server-9fb3459f-9728-af52-f3f4-96df4afef466 Address:9fb3459f-9728-af52-f3f4-96df4afef466}]"
2025-01-26T02:45:09.948Z [INFO]  server-3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fbf6908-17dc-f9af-458d-21bd1f6f2182 Address:7fbf6908-17dc-f9af-458d-21bd1f6f2182} {Suffrage:Voter ID:server-3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4 Address:3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4} {Suffrage:Voter ID:server-9fb3459f-9728-af52-f3f4-96df4afef466 Address:9fb3459f-9728-af52-f3f4-96df4afef466}]"
2025-01-26T02:45:09.948Z [INFO]  server-9fb3459f-9728-af52-f3f4-96df4afef466: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7fbf6908-17dc-f9af-458d-21bd1f6f2182 Address:7fbf6908-17dc-f9af-458d-21bd1f6f2182} {Suffrage:Voter ID:server-3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4 Address:3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4} {Suffrage:Voter ID:server-9fb3459f-9728-af52-f3f4-96df4afef466 Address:9fb3459f-9728-af52-f3f4-96df4afef466}]"
2025-01-26T02:45:09.948Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:09.948Z [INFO]  server-7fbf6908-17dc-f9af-458d-21bd1f6f2182: entering follower state: follower="Node at 7fbf6908-17dc-f9af-458d-21bd1f6f2182 [Follower]" leader-address= leader-id=
2025-01-26T02:45:09.948Z [INFO]  server-3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4: entering follower state: follower="Node at 3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4 [Follower]" leader-address= leader-id=
2025-01-26T02:45:09.948Z [INFO]  server-9fb3459f-9728-af52-f3f4-96df4afef466: entering follower state: follower="Node at 9fb3459f-9728-af52-f3f4-96df4afef466 [Follower]" leader-address= leader-id=
2025-01-26T02:45:10.019Z [WARN]  server-3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:10.019Z [INFO]  server-3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4: entering candidate state: node="Node at 3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4 [Candidate]" term=2
2025-01-26T02:45:10.019Z [INFO]  server-3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4: election won: term=2 tally=2
2025-01-26T02:45:10.019Z [INFO]  server-3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4: entering leader state: leader="Node at 3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4 [Leader]"
2025-01-26T02:45:10.019Z [INFO]  server-3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4: added peer, starting replication: peer=server-7fbf6908-17dc-f9af-458d-21bd1f6f2182
2025-01-26T02:45:10.019Z [INFO]  server-3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4: added peer, starting replication: peer=server-9fb3459f-9728-af52-f3f4-96df4afef466
2025-01-26T02:45:10.019Z [INFO]  server-3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4: pipelining replication: peer="{Voter server-9fb3459f-9728-af52-f3f4-96df4afef466 9fb3459f-9728-af52-f3f4-96df4afef466}"
2025-01-26T02:45:10.031Z [INFO]  server-3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4: pipelining replication: peer="{Voter server-7fbf6908-17dc-f9af-458d-21bd1f6f2182 7fbf6908-17dc-f9af-458d-21bd1f6f2182}"
2025-01-26T02:45:10.031Z [WARN]  server-7fbf6908-17dc-f9af-458d-21bd1f6f2182: rejecting vote request since our last term is greater: candidate=3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4 last-term=2 last-candidate-term=1
2025-01-26T02:45:10.139Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:10.032008145 +0000 UTC m=+106.547040520 (1 nodes), 83.881229ms from start of poll, 84.06563ms from cluster start. Timeout at 2025-01-26 02:45:10.137011225 +0000 UTC m=+106.652043600, 105.00308ms after stability
2025-01-26T02:45:10.139Z [INFO]  server-3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4: cannot transfer leadership to itself
2025-01-26T02:45:10.139Z [INFO]  server-3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4: aborting pipeline replication: peer="{Voter server-9fb3459f-9728-af52-f3f4-96df4afef466 9fb3459f-9728-af52-f3f4-96df4afef466}"
2025-01-26T02:45:10.139Z [INFO]  server-3dd0e0f1-72fd-dc58-b4c7-2da8c30c68b4: aborting pipeline replication: peer="{Voter server-7fbf6908-17dc-f9af-458d-21bd1f6f2182 7fbf6908-17dc-f9af-458d-21bd1f6f2182}"
--- PASS: TestRaft_LeadershipTransferToItself (0.19s)
=== RUN   TestRaft_LeadershipTransferLeaderRejectsClientRequests
2025-01-26T02:45:10.140Z [INFO]  server-6ea31b9f-2d30-64cb-9df4-fa2995cb8294: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ea31b9f-2d30-64cb-9df4-fa2995cb8294 Address:6ea31b9f-2d30-64cb-9df4-fa2995cb8294} {Suffrage:Voter ID:server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8 Address:b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8} {Suffrage:Voter ID:server-c6429895-3c55-7489-c2da-8fb81af7112b Address:c6429895-3c55-7489-c2da-8fb81af7112b}]"
2025-01-26T02:45:10.140Z [INFO]  server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ea31b9f-2d30-64cb-9df4-fa2995cb8294 Address:6ea31b9f-2d30-64cb-9df4-fa2995cb8294} {Suffrage:Voter ID:server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8 Address:b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8} {Suffrage:Voter ID:server-c6429895-3c55-7489-c2da-8fb81af7112b Address:c6429895-3c55-7489-c2da-8fb81af7112b}]"
2025-01-26T02:45:10.140Z [INFO]  server-c6429895-3c55-7489-c2da-8fb81af7112b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ea31b9f-2d30-64cb-9df4-fa2995cb8294 Address:6ea31b9f-2d30-64cb-9df4-fa2995cb8294} {Suffrage:Voter ID:server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8 Address:b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8} {Suffrage:Voter ID:server-c6429895-3c55-7489-c2da-8fb81af7112b Address:c6429895-3c55-7489-c2da-8fb81af7112b}]"
2025-01-26T02:45:10.140Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:10.140Z [INFO]  server-6ea31b9f-2d30-64cb-9df4-fa2995cb8294: entering follower state: follower="Node at 6ea31b9f-2d30-64cb-9df4-fa2995cb8294 [Follower]" leader-address= leader-id=
2025-01-26T02:45:10.140Z [INFO]  server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8: entering follower state: follower="Node at b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8 [Follower]" leader-address= leader-id=
2025-01-26T02:45:10.140Z [INFO]  server-c6429895-3c55-7489-c2da-8fb81af7112b: entering follower state: follower="Node at c6429895-3c55-7489-c2da-8fb81af7112b [Follower]" leader-address= leader-id=
2025-01-26T02:45:10.203Z [WARN]  server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:10.203Z [INFO]  server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8: entering candidate state: node="Node at b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8 [Candidate]" term=2
2025-01-26T02:45:10.203Z [INFO]  server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8: election won: term=2 tally=2
2025-01-26T02:45:10.203Z [INFO]  server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8: entering leader state: leader="Node at b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8 [Leader]"
2025-01-26T02:45:10.203Z [INFO]  server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8: added peer, starting replication: peer=server-6ea31b9f-2d30-64cb-9df4-fa2995cb8294
2025-01-26T02:45:10.203Z [INFO]  server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8: added peer, starting replication: peer=server-c6429895-3c55-7489-c2da-8fb81af7112b
2025-01-26T02:45:10.203Z [WARN]  server-6ea31b9f-2d30-64cb-9df4-fa2995cb8294: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:10.203Z [INFO]  server-6ea31b9f-2d30-64cb-9df4-fa2995cb8294: entering candidate state: node="Node at 6ea31b9f-2d30-64cb-9df4-fa2995cb8294 [Candidate]" term=2
2025-01-26T02:45:10.203Z [INFO]  server-c6429895-3c55-7489-c2da-8fb81af7112b: duplicate requestVote for same term: term=2
2025-01-26T02:45:10.203Z [INFO]  server-6ea31b9f-2d30-64cb-9df4-fa2995cb8294: duplicate requestVote for same term: term=2
2025-01-26T02:45:10.203Z [INFO]  server-6ea31b9f-2d30-64cb-9df4-fa2995cb8294: entering follower state: follower="Node at 6ea31b9f-2d30-64cb-9df4-fa2995cb8294 [Follower]" leader-address=b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8 leader-id=server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8
2025-01-26T02:45:10.203Z [INFO]  server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8: pipelining replication: peer="{Voter server-6ea31b9f-2d30-64cb-9df4-fa2995cb8294 6ea31b9f-2d30-64cb-9df4-fa2995cb8294}"
2025-01-26T02:45:10.203Z [INFO]  server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8: pipelining replication: peer="{Voter server-c6429895-3c55-7489-c2da-8fb81af7112b c6429895-3c55-7489-c2da-8fb81af7112b}"
2025-01-26T02:45:10.203Z [WARN]  server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8: rejecting vote request since we have a leader: from=6ea31b9f-2d30-64cb-9df4-fa2995cb8294 leader=b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8 leader-id=server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8
2025-01-26T02:45:10.311Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:10.203973188 +0000 UTC m=+106.719005563 (1 nodes), 63.543185ms from start of poll, 63.727627ms from cluster start. Timeout at 2025-01-26 02:45:10.308976828 +0000 UTC m=+106.824009163, 105.0036ms after stability
    raft_test.go:2457: waiting on future 0
    raft_test.go:2457: waiting on future 1
    raft_test.go:2457: waiting on future 2
    raft_test.go:2457: waiting on future 3
    raft_test.go:2457: waiting on future 4
    raft_test.go:2457: waiting on future 5
    raft_test.go:2457: waiting on future 6
2025-01-26T02:45:10.312Z [INFO]  server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8: aborting pipeline replication: peer="{Voter server-c6429895-3c55-7489-c2da-8fb81af7112b c6429895-3c55-7489-c2da-8fb81af7112b}"
2025-01-26T02:45:10.312Z [INFO]  server-b1c1dcf7-d607-1544-c6cc-1cf8b58b18d8: aborting pipeline replication: peer="{Voter server-6ea31b9f-2d30-64cb-9df4-fa2995cb8294 6ea31b9f-2d30-64cb-9df4-fa2995cb8294}"
--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.17s)
=== RUN   TestRaft_LeadershipTransferLeaderReplicationTimeout
2025-01-26T02:45:10.312Z [INFO]  server-761585f6-d154-95f2-7a99-4988dc2edcea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-761585f6-d154-95f2-7a99-4988dc2edcea Address:761585f6-d154-95f2-7a99-4988dc2edcea} {Suffrage:Voter ID:server-a09974de-32e4-5063-b7c5-1eac4023de0b Address:a09974de-32e4-5063-b7c5-1eac4023de0b} {Suffrage:Voter ID:server-10f9d6cc-3595-89d4-f07e-e97a7a00bf9c Address:10f9d6cc-3595-89d4-f07e-e97a7a00bf9c}]"
2025-01-26T02:45:10.312Z [INFO]  server-a09974de-32e4-5063-b7c5-1eac4023de0b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-761585f6-d154-95f2-7a99-4988dc2edcea Address:761585f6-d154-95f2-7a99-4988dc2edcea} {Suffrage:Voter ID:server-a09974de-32e4-5063-b7c5-1eac4023de0b Address:a09974de-32e4-5063-b7c5-1eac4023de0b} {Suffrage:Voter ID:server-10f9d6cc-3595-89d4-f07e-e97a7a00bf9c Address:10f9d6cc-3595-89d4-f07e-e97a7a00bf9c}]"
2025-01-26T02:45:10.312Z [INFO]  server-10f9d6cc-3595-89d4-f07e-e97a7a00bf9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-761585f6-d154-95f2-7a99-4988dc2edcea Address:761585f6-d154-95f2-7a99-4988dc2edcea} {Suffrage:Voter ID:server-a09974de-32e4-5063-b7c5-1eac4023de0b Address:a09974de-32e4-5063-b7c5-1eac4023de0b} {Suffrage:Voter ID:server-10f9d6cc-3595-89d4-f07e-e97a7a00bf9c Address:10f9d6cc-3595-89d4-f07e-e97a7a00bf9c}]"
2025-01-26T02:45:10.312Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:10.312Z [INFO]  server-761585f6-d154-95f2-7a99-4988dc2edcea: entering follower state: follower="Node at 761585f6-d154-95f2-7a99-4988dc2edcea [Follower]" leader-address= leader-id=
2025-01-26T02:45:10.312Z [INFO]  server-a09974de-32e4-5063-b7c5-1eac4023de0b: entering follower state: follower="Node at a09974de-32e4-5063-b7c5-1eac4023de0b [Follower]" leader-address= leader-id=
2025-01-26T02:45:10.312Z [INFO]  server-10f9d6cc-3595-89d4-f07e-e97a7a00bf9c: entering follower state: follower="Node at 10f9d6cc-3595-89d4-f07e-e97a7a00bf9c [Follower]" leader-address= leader-id=
2025-01-26T02:45:10.367Z [WARN]  server-a09974de-32e4-5063-b7c5-1eac4023de0b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:10.367Z [INFO]  server-a09974de-32e4-5063-b7c5-1eac4023de0b: entering candidate state: node="Node at a09974de-32e4-5063-b7c5-1eac4023de0b [Candidate]" term=2
2025-01-26T02:45:10.367Z [INFO]  server-a09974de-32e4-5063-b7c5-1eac4023de0b: election won: term=2 tally=2
2025-01-26T02:45:10.367Z [INFO]  server-a09974de-32e4-5063-b7c5-1eac4023de0b: entering leader state: leader="Node at a09974de-32e4-5063-b7c5-1eac4023de0b [Leader]"
2025-01-26T02:45:10.367Z [INFO]  server-a09974de-32e4-5063-b7c5-1eac4023de0b: added peer, starting replication: peer=server-761585f6-d154-95f2-7a99-4988dc2edcea
2025-01-26T02:45:10.367Z [INFO]  server-a09974de-32e4-5063-b7c5-1eac4023de0b: added peer, starting replication: peer=server-10f9d6cc-3595-89d4-f07e-e97a7a00bf9c
2025-01-26T02:45:10.367Z [INFO]  server-a09974de-32e4-5063-b7c5-1eac4023de0b: pipelining replication: peer="{Voter server-10f9d6cc-3595-89d4-f07e-e97a7a00bf9c 10f9d6cc-3595-89d4-f07e-e97a7a00bf9c}"
2025-01-26T02:45:10.367Z [INFO]  server-a09974de-32e4-5063-b7c5-1eac4023de0b: pipelining replication: peer="{Voter server-761585f6-d154-95f2-7a99-4988dc2edcea 761585f6-d154-95f2-7a99-4988dc2edcea}"
2025-01-26T02:45:10.479Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:10.367684142 +0000 UTC m=+106.882716477 (1 nodes), 54.994854ms from start of poll, 55.174375ms from cluster start. Timeout at 2025-01-26 02:45:10.472687822 +0000 UTC m=+106.987720157, 105.00368ms after stability
2025-01-26T02:45:10.479Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:10.595Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:10.479667941 +0000 UTC m=+106.994700356 (2 nodes), 1.12µs from start of poll, 167.158254ms from cluster start. Timeout at 2025-01-26 02:45:10.584735181 +0000 UTC m=+107.099767556, 105.0672ms after stability
2025-01-26T02:45:11.331Z [INFO]  server-a09974de-32e4-5063-b7c5-1eac4023de0b: aborting pipeline replication: peer="{Voter server-761585f6-d154-95f2-7a99-4988dc2edcea 761585f6-d154-95f2-7a99-4988dc2edcea}"
2025-01-26T02:45:11.823Z [ERROR] server-a09974de-32e4-5063-b7c5-1eac4023de0b: failed to pipeline appendEntries: peer="{Voter server-10f9d6cc-3595-89d4-f07e-e97a7a00bf9c 10f9d6cc-3595-89d4-f07e-e97a7a00bf9c}" error="command enqueue timeout"
2025-01-26T02:45:11.823Z [INFO]  server-a09974de-32e4-5063-b7c5-1eac4023de0b: aborting pipeline replication: peer="{Voter server-10f9d6cc-3595-89d4-f07e-e97a7a00bf9c 10f9d6cc-3595-89d4-f07e-e97a7a00bf9c}"
2025-01-26T02:45:11.827Z [ERROR] server-a09974de-32e4-5063-b7c5-1eac4023de0b: failed to heartbeat to: peer=10f9d6cc-3595-89d4-f07e-e97a7a00bf9c backoff time=10ms error="send timed out"
2025-01-26T02:45:11.831Z [ERROR] server-a09974de-32e4-5063-b7c5-1eac4023de0b: failed to appendEntries to: peer="{Voter server-761585f6-d154-95f2-7a99-4988dc2edcea 761585f6-d154-95f2-7a99-4988dc2edcea}" error="command timed out"
2025-01-26T02:45:11.839Z [ERROR] server-a09974de-32e4-5063-b7c5-1eac4023de0b: failed to heartbeat to: peer=761585f6-d154-95f2-7a99-4988dc2edcea backoff time=10ms error="command timed out"
2025-01-26T02:45:12.331Z [ERROR] server-a09974de-32e4-5063-b7c5-1eac4023de0b: failed to appendEntries to: peer="{Voter server-761585f6-d154-95f2-7a99-4988dc2edcea 761585f6-d154-95f2-7a99-4988dc2edcea}" error="command timed out"
2025-01-26T02:45:12.359Z [ERROR] server-a09974de-32e4-5063-b7c5-1eac4023de0b: failed to heartbeat to: peer=761585f6-d154-95f2-7a99-4988dc2edcea backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (2.05s)
=== RUN   TestRaft_LeadershipTransferIgnoresNonvoters
2025-01-26T02:45:12.360Z [INFO]  server-e6bc59e7-6053-bda5-fec5-63b72d233da3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6bc59e7-6053-bda5-fec5-63b72d233da3 Address:e6bc59e7-6053-bda5-fec5-63b72d233da3} {Suffrage:Voter ID:server-82b0f095-1d1c-21f9-6b32-a8f63f94ab49 Address:82b0f095-1d1c-21f9-6b32-a8f63f94ab49}]"
2025-01-26T02:45:12.360Z [INFO]  server-82b0f095-1d1c-21f9-6b32-a8f63f94ab49: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6bc59e7-6053-bda5-fec5-63b72d233da3 Address:e6bc59e7-6053-bda5-fec5-63b72d233da3} {Suffrage:Voter ID:server-82b0f095-1d1c-21f9-6b32-a8f63f94ab49 Address:82b0f095-1d1c-21f9-6b32-a8f63f94ab49}]"
2025-01-26T02:45:12.360Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:12.360Z [INFO]  server-e6bc59e7-6053-bda5-fec5-63b72d233da3: entering follower state: follower="Node at e6bc59e7-6053-bda5-fec5-63b72d233da3 [Follower]" leader-address= leader-id=
2025-01-26T02:45:12.360Z [INFO]  server-82b0f095-1d1c-21f9-6b32-a8f63f94ab49: entering follower state: follower="Node at 82b0f095-1d1c-21f9-6b32-a8f63f94ab49 [Follower]" leader-address= leader-id=
2025-01-26T02:45:12.411Z [WARN]  server-82b0f095-1d1c-21f9-6b32-a8f63f94ab49: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:12.411Z [INFO]  server-82b0f095-1d1c-21f9-6b32-a8f63f94ab49: entering candidate state: node="Node at 82b0f095-1d1c-21f9-6b32-a8f63f94ab49 [Candidate]" term=2
2025-01-26T02:45:12.411Z [INFO]  server-82b0f095-1d1c-21f9-6b32-a8f63f94ab49: election won: term=2 tally=2
2025-01-26T02:45:12.411Z [INFO]  server-82b0f095-1d1c-21f9-6b32-a8f63f94ab49: entering leader state: leader="Node at 82b0f095-1d1c-21f9-6b32-a8f63f94ab49 [Leader]"
2025-01-26T02:45:12.411Z [INFO]  server-82b0f095-1d1c-21f9-6b32-a8f63f94ab49: added peer, starting replication: peer=server-e6bc59e7-6053-bda5-fec5-63b72d233da3
2025-01-26T02:45:12.411Z [INFO]  server-82b0f095-1d1c-21f9-6b32-a8f63f94ab49: pipelining replication: peer="{Voter server-e6bc59e7-6053-bda5-fec5-63b72d233da3 e6bc59e7-6053-bda5-fec5-63b72d233da3}"
2025-01-26T02:45:12.519Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:12.411673344 +0000 UTC m=+108.926705719 (1 nodes), 51.562513ms from start of poll, 51.706633ms from cluster start. Timeout at 2025-01-26 02:45:12.516676024 +0000 UTC m=+109.031708359, 105.00264ms after stability
2025-01-26T02:45:12.519Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:12.631Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:12.519989242 +0000 UTC m=+109.035021617 (1 nodes), 1.36µs from start of poll, 160.022531ms from cluster start. Timeout at 2025-01-26 02:45:12.625012162 +0000 UTC m=+109.140044537, 105.02292ms after stability
2025-01-26T02:45:12.631Z [INFO]  server-82b0f095-1d1c-21f9-6b32-a8f63f94ab49: updating configuration: command=DemoteVoter server-id=server-e6bc59e7-6053-bda5-fec5-63b72d233da3 server-addr= servers="[{Suffrage:Nonvoter ID:server-e6bc59e7-6053-bda5-fec5-63b72d233da3 Address:e6bc59e7-6053-bda5-fec5-63b72d233da3} {Suffrage:Voter ID:server-82b0f095-1d1c-21f9-6b32-a8f63f94ab49 Address:82b0f095-1d1c-21f9-6b32-a8f63f94ab49}]"
2025-01-26T02:45:12.631Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:12.739Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:12.631698279 +0000 UTC m=+109.146730654 (1 nodes), 520ns from start of poll, 271.731568ms from cluster start. Timeout at 2025-01-26 02:45:12.736735799 +0000 UTC m=+109.251768174, 105.03752ms after stability
2025-01-26T02:45:12.739Z [INFO]  server-82b0f095-1d1c-21f9-6b32-a8f63f94ab49: aborting pipeline replication: peer="{Voter server-e6bc59e7-6053-bda5-fec5-63b72d233da3 e6bc59e7-6053-bda5-fec5-63b72d233da3}"
--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.38s)
=== RUN   TestRaft_LeadershipTransferStopRightAway
--- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s)
=== RUN   TestRaft_GetConfigurationNoBootstrap
2025-01-26T02:45:12.740Z [INFO]  server-0a1b41ed-5fae-ae95-4127-50401be5115c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0a1b41ed-5fae-ae95-4127-50401be5115c Address:0a1b41ed-5fae-ae95-4127-50401be5115c} {Suffrage:Voter ID:server-609f7c02-459d-f91f-c654-0bd4e956a706 Address:609f7c02-459d-f91f-c654-0bd4e956a706}]"
2025-01-26T02:45:12.740Z [INFO]  server-609f7c02-459d-f91f-c654-0bd4e956a706: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0a1b41ed-5fae-ae95-4127-50401be5115c Address:0a1b41ed-5fae-ae95-4127-50401be5115c} {Suffrage:Voter ID:server-609f7c02-459d-f91f-c654-0bd4e956a706 Address:609f7c02-459d-f91f-c654-0bd4e956a706}]"
2025-01-26T02:45:12.740Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:12.740Z [INFO]  server-0a1b41ed-5fae-ae95-4127-50401be5115c: entering follower state: follower="Node at 0a1b41ed-5fae-ae95-4127-50401be5115c [Follower]" leader-address= leader-id=
2025-01-26T02:45:12.740Z [INFO]  server-609f7c02-459d-f91f-c654-0bd4e956a706: entering follower state: follower="Node at 609f7c02-459d-f91f-c654-0bd4e956a706 [Follower]" leader-address= leader-id=
2025-01-26T02:45:12.815Z [WARN]  server-0a1b41ed-5fae-ae95-4127-50401be5115c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:12.815Z [INFO]  server-0a1b41ed-5fae-ae95-4127-50401be5115c: entering candidate state: node="Node at 0a1b41ed-5fae-ae95-4127-50401be5115c [Candidate]" term=2
2025-01-26T02:45:12.815Z [INFO]  server-0a1b41ed-5fae-ae95-4127-50401be5115c: election won: term=2 tally=2
2025-01-26T02:45:12.815Z [INFO]  server-0a1b41ed-5fae-ae95-4127-50401be5115c: entering leader state: leader="Node at 0a1b41ed-5fae-ae95-4127-50401be5115c [Leader]"
2025-01-26T02:45:12.815Z [INFO]  server-0a1b41ed-5fae-ae95-4127-50401be5115c: added peer, starting replication: peer=server-609f7c02-459d-f91f-c654-0bd4e956a706
2025-01-26T02:45:12.815Z [INFO]  server-0a1b41ed-5fae-ae95-4127-50401be5115c: pipelining replication: peer="{Voter server-609f7c02-459d-f91f-c654-0bd4e956a706 609f7c02-459d-f91f-c654-0bd4e956a706}"
2025-01-26T02:45:12.927Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:12.815676436 +0000 UTC m=+109.330708811 (1 nodes), 75.524099ms from start of poll, 75.654499ms from cluster start. Timeout at 2025-01-26 02:45:12.920681396 +0000 UTC m=+109.435713771, 105.00496ms after stability
2025-01-26T02:45:12.927Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:13.035Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:12.927794116 +0000 UTC m=+109.442826491 (1 nodes), 560ns from start of poll, 187.772179ms from cluster start. Timeout at 2025-01-26 02:45:13.032835276 +0000 UTC m=+109.547867651, 105.04116ms after stability
2025-01-26T02:45:13.051Z [INFO]  server-609f7c02-459d-f91f-c654-0bd4e956a706: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0a1b41ed-5fae-ae95-4127-50401be5115c Address:0a1b41ed-5fae-ae95-4127-50401be5115c} {Suffrage:Voter ID:server-609f7c02-459d-f91f-c654-0bd4e956a706 Address:609f7c02-459d-f91f-c654-0bd4e956a706}]"
2025-01-26T02:45:13.051Z [INFO]  server-0a1b41ed-5fae-ae95-4127-50401be5115c: aborting pipeline replication: peer="{Voter server-609f7c02-459d-f91f-c654-0bd4e956a706 609f7c02-459d-f91f-c654-0bd4e956a706}"
--- PASS: TestRaft_GetConfigurationNoBootstrap (0.31s)
=== RUN   TestRaft_LogStoreIsMonotonic
2025-01-26T02:45:13.051Z [INFO]  server-f9d4492c-c848-d659-65de-3e5aa756f175: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f9d4492c-c848-d659-65de-3e5aa756f175 Address:f9d4492c-c848-d659-65de-3e5aa756f175}]"
2025-01-26T02:45:13.051Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:13.051Z [INFO]  server-f9d4492c-c848-d659-65de-3e5aa756f175: entering follower state: follower="Node at f9d4492c-c848-d659-65de-3e5aa756f175 [Follower]" leader-address= leader-id=
2025-01-26T02:45:13.103Z [WARN]  server-f9d4492c-c848-d659-65de-3e5aa756f175: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:13.103Z [INFO]  server-f9d4492c-c848-d659-65de-3e5aa756f175: entering candidate state: node="Node at f9d4492c-c848-d659-65de-3e5aa756f175 [Candidate]" term=2
2025-01-26T02:45:13.103Z [INFO]  server-f9d4492c-c848-d659-65de-3e5aa756f175: election won: term=2 tally=1
2025-01-26T02:45:13.103Z [INFO]  server-f9d4492c-c848-d659-65de-3e5aa756f175: entering leader state: leader="Node at f9d4492c-c848-d659-65de-3e5aa756f175 [Leader]"
2025-01-26T02:45:13.211Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:13.103661663 +0000 UTC m=+109.618694038 (1 nodes), 51.670993ms from start of poll, 51.749274ms from cluster start. Timeout at 2025-01-26 02:45:13.208664743 +0000 UTC m=+109.723697078, 105.00304ms after stability
--- PASS: TestRaft_LogStoreIsMonotonic (0.16s)
=== RUN   TestRaft_CacheLogWithStoreError
2025-01-26T02:45:13.211Z [INFO]  server-ce5e36ad-9640-d2aa-e818-08524de61471: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce5e36ad-9640-d2aa-e818-08524de61471 Address:ce5e36ad-9640-d2aa-e818-08524de61471} {Suffrage:Voter ID:server-7698199a-38dc-be6f-1181-c56476f6d522 Address:7698199a-38dc-be6f-1181-c56476f6d522}]"
2025-01-26T02:45:13.211Z [INFO]  server-7698199a-38dc-be6f-1181-c56476f6d522: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce5e36ad-9640-d2aa-e818-08524de61471 Address:ce5e36ad-9640-d2aa-e818-08524de61471} {Suffrage:Voter ID:server-7698199a-38dc-be6f-1181-c56476f6d522 Address:7698199a-38dc-be6f-1181-c56476f6d522}]"
2025-01-26T02:45:13.211Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:13.211Z [INFO]  server-ce5e36ad-9640-d2aa-e818-08524de61471: entering follower state: follower="Node at ce5e36ad-9640-d2aa-e818-08524de61471 [Follower]" leader-address= leader-id=
2025-01-26T02:45:13.212Z [INFO]  server-7698199a-38dc-be6f-1181-c56476f6d522: entering follower state: follower="Node at 7698199a-38dc-be6f-1181-c56476f6d522 [Follower]" leader-address= leader-id=
2025-01-26T02:45:13.267Z [WARN]  server-ce5e36ad-9640-d2aa-e818-08524de61471: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:13.267Z [INFO]  server-ce5e36ad-9640-d2aa-e818-08524de61471: entering candidate state: node="Node at ce5e36ad-9640-d2aa-e818-08524de61471 [Candidate]" term=2
2025-01-26T02:45:13.267Z [INFO]  server-ce5e36ad-9640-d2aa-e818-08524de61471: election won: term=2 tally=2
2025-01-26T02:45:13.267Z [INFO]  server-ce5e36ad-9640-d2aa-e818-08524de61471: entering leader state: leader="Node at ce5e36ad-9640-d2aa-e818-08524de61471 [Leader]"
2025-01-26T02:45:13.267Z [INFO]  server-ce5e36ad-9640-d2aa-e818-08524de61471: added peer, starting replication: peer=server-7698199a-38dc-be6f-1181-c56476f6d522
2025-01-26T02:45:13.267Z [INFO]  server-ce5e36ad-9640-d2aa-e818-08524de61471: pipelining replication: peer="{Voter server-7698199a-38dc-be6f-1181-c56476f6d522 7698199a-38dc-be6f-1181-c56476f6d522}"
2025-01-26T02:45:13.375Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:13.267784459 +0000 UTC m=+109.782816834 (1 nodes), 55.880739ms from start of poll, 56.00858ms from cluster start. Timeout at 2025-01-26 02:45:13.372800659 +0000 UTC m=+109.887833034, 105.0162ms after stability
2025-01-26T02:45:13.375Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:13.483Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:13.375534393 +0000 UTC m=+109.890566768 (1 nodes), 1.32µs from start of poll, 163.758514ms from cluster start. Timeout at 2025-01-26 02:45:13.480575274 +0000 UTC m=+109.995607649, 105.040881ms after stability
2025-01-26T02:45:13.484Z [ERROR] server-7698199a-38dc-be6f-1181-c56476f6d522: failed to append to logs: error="unable to store logs within log store, err: "some error""
2025-01-26T02:45:13.484Z [WARN]  server-7698199a-38dc-be6f-1181-c56476f6d522: failed to get previous log: previous-index=9 last-index=7 error="log not found"
2025-01-26T02:45:13.484Z [INFO]  server-ce5e36ad-9640-d2aa-e818-08524de61471: aborting pipeline replication: peer="{Voter server-7698199a-38dc-be6f-1181-c56476f6d522 7698199a-38dc-be6f-1181-c56476f6d522}"
2025-01-26T02:45:13.491Z [INFO]  server-ce5e36ad-9640-d2aa-e818-08524de61471: pipelining replication: peer="{Voter server-7698199a-38dc-be6f-1181-c56476f6d522 7698199a-38dc-be6f-1181-c56476f6d522}"
2025-01-26T02:45:13.491Z [INFO]  server-7698199a-38dc-be6f-1181-c56476f6d522: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ce5e36ad-9640-d2aa-e818-08524de61471 Address:ce5e36ad-9640-d2aa-e818-08524de61471} {Suffrage:Voter ID:server-7698199a-38dc-be6f-1181-c56476f6d522 Address:7698199a-38dc-be6f-1181-c56476f6d522}]"
2025-01-26T02:45:13.491Z [INFO]  server-ce5e36ad-9640-d2aa-e818-08524de61471: aborting pipeline replication: peer="{Voter server-7698199a-38dc-be6f-1181-c56476f6d522 7698199a-38dc-be6f-1181-c56476f6d522}"
2025-01-26T02:45:13.991Z [ERROR] server-ce5e36ad-9640-d2aa-e818-08524de61471: failed to heartbeat to: peer=7698199a-38dc-be6f-1181-c56476f6d522 backoff time=10ms error="command timed out"
--- PASS: TestRaft_CacheLogWithStoreError (0.78s)
=== RUN   TestRaft_ReloadConfig
2025-01-26T02:45:13.992Z [INFO]  server-b041b74b-cd29-4cf4-9d21-0e01508ea67f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b041b74b-cd29-4cf4-9d21-0e01508ea67f Address:b041b74b-cd29-4cf4-9d21-0e01508ea67f}]"
--- PASS: TestRaft_ReloadConfig (0.00s)
=== RUN   TestRaft_ReloadConfigValidates
2025-01-26T02:45:13.992Z [INFO]  server-7956bfd6-bce4-b60e-4749-f34c23dcdbfd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7956bfd6-bce4-b60e-4749-f34c23dcdbfd Address:7956bfd6-bce4-b60e-4749-f34c23dcdbfd}]"
--- PASS: TestRaft_ReloadConfigValidates (0.00s)
=== RUN   TestRaft_InstallSnapshot_InvalidPeers
2025-01-26T02:45:13.992Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)"
--- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s)
=== RUN   TestRaft_VoteNotGranted_WhenNodeNotInCluster
2025-01-26T02:45:13.992Z [INFO]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e22ba692-0637-2f48-01de-94f5fdb431b0 Address:e22ba692-0637-2f48-01de-94f5fdb431b0} {Suffrage:Voter ID:server-51026783-c7dd-237c-7ac4-78335869886b Address:51026783-c7dd-237c-7ac4-78335869886b} {Suffrage:Voter ID:server-e8f9b925-f428-27fc-af12-ab2c762f58ff Address:e8f9b925-f428-27fc-af12-ab2c762f58ff}]"
2025-01-26T02:45:13.992Z [INFO]  server-51026783-c7dd-237c-7ac4-78335869886b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e22ba692-0637-2f48-01de-94f5fdb431b0 Address:e22ba692-0637-2f48-01de-94f5fdb431b0} {Suffrage:Voter ID:server-51026783-c7dd-237c-7ac4-78335869886b Address:51026783-c7dd-237c-7ac4-78335869886b} {Suffrage:Voter ID:server-e8f9b925-f428-27fc-af12-ab2c762f58ff Address:e8f9b925-f428-27fc-af12-ab2c762f58ff}]"
2025-01-26T02:45:13.992Z [INFO]  server-e8f9b925-f428-27fc-af12-ab2c762f58ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e22ba692-0637-2f48-01de-94f5fdb431b0 Address:e22ba692-0637-2f48-01de-94f5fdb431b0} {Suffrage:Voter ID:server-51026783-c7dd-237c-7ac4-78335869886b Address:51026783-c7dd-237c-7ac4-78335869886b} {Suffrage:Voter ID:server-e8f9b925-f428-27fc-af12-ab2c762f58ff Address:e8f9b925-f428-27fc-af12-ab2c762f58ff}]"
2025-01-26T02:45:13.992Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:13.992Z [INFO]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: entering follower state: follower="Node at e22ba692-0637-2f48-01de-94f5fdb431b0 [Follower]" leader-address= leader-id=
2025-01-26T02:45:13.992Z [INFO]  server-51026783-c7dd-237c-7ac4-78335869886b: entering follower state: follower="Node at 51026783-c7dd-237c-7ac4-78335869886b [Follower]" leader-address= leader-id=
2025-01-26T02:45:13.992Z [INFO]  server-e8f9b925-f428-27fc-af12-ab2c762f58ff: entering follower state: follower="Node at e8f9b925-f428-27fc-af12-ab2c762f58ff [Follower]" leader-address= leader-id=
2025-01-26T02:45:14.047Z [WARN]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:14.047Z [INFO]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: entering candidate state: node="Node at e22ba692-0637-2f48-01de-94f5fdb431b0 [Candidate]" term=2
2025-01-26T02:45:14.047Z [INFO]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: election won: term=2 tally=2
2025-01-26T02:45:14.047Z [INFO]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: entering leader state: leader="Node at e22ba692-0637-2f48-01de-94f5fdb431b0 [Leader]"
2025-01-26T02:45:14.047Z [INFO]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: added peer, starting replication: peer=server-51026783-c7dd-237c-7ac4-78335869886b
2025-01-26T02:45:14.047Z [INFO]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: added peer, starting replication: peer=server-e8f9b925-f428-27fc-af12-ab2c762f58ff
2025-01-26T02:45:14.047Z [INFO]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: pipelining replication: peer="{Voter server-e8f9b925-f428-27fc-af12-ab2c762f58ff e8f9b925-f428-27fc-af12-ab2c762f58ff}"
2025-01-26T02:45:14.047Z [INFO]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: pipelining replication: peer="{Voter server-51026783-c7dd-237c-7ac4-78335869886b 51026783-c7dd-237c-7ac4-78335869886b}"
2025-01-26T02:45:14.155Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:14.048002553 +0000 UTC m=+110.563034928 (1 nodes), 55.209895ms from start of poll, 55.376936ms from cluster start. Timeout at 2025-01-26 02:45:14.153015353 +0000 UTC m=+110.668047688, 105.01276ms after stability
2025-01-26T02:45:14.164Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:14.271Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:14.164153338 +0000 UTC m=+110.679185713 (2 nodes), 1.16µs from start of poll, 171.527721ms from cluster start. Timeout at 2025-01-26 02:45:14.269174218 +0000 UTC m=+110.784206593, 105.02088ms after stability
2025-01-26T02:45:14.271Z [INFO]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: updating configuration: command=RemoveServer server-id=server-51026783-c7dd-237c-7ac4-78335869886b server-addr= servers="[{Suffrage:Voter ID:server-e22ba692-0637-2f48-01de-94f5fdb431b0 Address:e22ba692-0637-2f48-01de-94f5fdb431b0} {Suffrage:Voter ID:server-e8f9b925-f428-27fc-af12-ab2c762f58ff Address:e8f9b925-f428-27fc-af12-ab2c762f58ff}]"
2025-01-26T02:45:14.271Z [INFO]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: removed peer, stopping replication: peer=server-51026783-c7dd-237c-7ac4-78335869886b last-index=3
2025-01-26T02:45:14.271Z [INFO]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: aborting pipeline replication: peer="{Voter server-51026783-c7dd-237c-7ac4-78335869886b 51026783-c7dd-237c-7ac4-78335869886b}"
2025-01-26T02:45:14.383Z [ERROR] server-e22ba692-0637-2f48-01de-94f5fdb431b0: failed to pipeline appendEntries: peer="{Voter server-e8f9b925-f428-27fc-af12-ab2c762f58ff e8f9b925-f428-27fc-af12-ab2c762f58ff}" error="append pipeline closed"
2025-01-26T02:45:14.383Z [INFO]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: aborting pipeline replication: peer="{Voter server-e8f9b925-f428-27fc-af12-ab2c762f58ff e8f9b925-f428-27fc-af12-ab2c762f58ff}"
2025-01-26T02:45:14.383Z [ERROR] server-e22ba692-0637-2f48-01de-94f5fdb431b0: failed to heartbeat to: peer=e8f9b925-f428-27fc-af12-ab2c762f58ff backoff time=10ms error="failed to connect to peer: e8f9b925-f428-27fc-af12-ab2c762f58ff"
2025-01-26T02:45:14.391Z [ERROR] server-e22ba692-0637-2f48-01de-94f5fdb431b0: failed to appendEntries to: peer="{Voter server-e8f9b925-f428-27fc-af12-ab2c762f58ff e8f9b925-f428-27fc-af12-ab2c762f58ff}" error="failed to connect to peer: e8f9b925-f428-27fc-af12-ab2c762f58ff"
2025-01-26T02:45:14.397Z [WARN]  server-51026783-c7dd-237c-7ac4-78335869886b: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-01-26T02:45:14.403Z [ERROR] server-e22ba692-0637-2f48-01de-94f5fdb431b0: failed to heartbeat to: peer=e8f9b925-f428-27fc-af12-ab2c762f58ff backoff time=10ms error="failed to connect to peer: e8f9b925-f428-27fc-af12-ab2c762f58ff"
2025-01-26T02:45:14.415Z [ERROR] server-e22ba692-0637-2f48-01de-94f5fdb431b0: failed to appendEntries to: peer="{Voter server-e8f9b925-f428-27fc-af12-ab2c762f58ff e8f9b925-f428-27fc-af12-ab2c762f58ff}" error="failed to connect to peer: e8f9b925-f428-27fc-af12-ab2c762f58ff"
2025-01-26T02:45:14.423Z [ERROR] server-e22ba692-0637-2f48-01de-94f5fdb431b0: failed to heartbeat to: peer=e8f9b925-f428-27fc-af12-ab2c762f58ff backoff time=10ms error="failed to connect to peer: e8f9b925-f428-27fc-af12-ab2c762f58ff"
2025-01-26T02:45:14.427Z [WARN]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: failed to contact: server-id=server-e8f9b925-f428-27fc-af12-ab2c762f58ff time=51.794274ms
2025-01-26T02:45:14.427Z [WARN]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: failed to contact quorum of nodes, stepping down
2025-01-26T02:45:14.427Z [INFO]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: entering follower state: follower="Node at e22ba692-0637-2f48-01de-94f5fdb431b0 [Follower]" leader-address= leader-id=
2025-01-26T02:45:14.435Z [ERROR] server-e22ba692-0637-2f48-01de-94f5fdb431b0: failed to appendEntries to: peer="{Voter server-e8f9b925-f428-27fc-af12-ab2c762f58ff e8f9b925-f428-27fc-af12-ab2c762f58ff}" error="failed to connect to peer: e8f9b925-f428-27fc-af12-ab2c762f58ff"
2025-01-26T02:45:14.507Z [WARN]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:14.507Z [INFO]  server-e22ba692-0637-2f48-01de-94f5fdb431b0: entering candidate state: node="Node at e22ba692-0637-2f48-01de-94f5fdb431b0 [Candidate]" term=3
2025-01-26T02:45:14.507Z [ERROR] server-e22ba692-0637-2f48-01de-94f5fdb431b0: failed to make requestVote RPC: target="{Voter server-e8f9b925-f428-27fc-af12-ab2c762f58ff e8f9b925-f428-27fc-af12-ab2c762f58ff}" error="failed to connect to peer: e8f9b925-f428-27fc-af12-ab2c762f58ff" term=3
2025-01-26T02:45:14.515Z [WARN]  server-e8f9b925-f428-27fc-af12-ab2c762f58ff: heartbeat timeout reached, starting election: last-leader-addr=e22ba692-0637-2f48-01de-94f5fdb431b0 last-leader-id=server-e22ba692-0637-2f48-01de-94f5fdb431b0
2025-01-26T02:45:14.515Z [INFO]  server-e8f9b925-f428-27fc-af12-ab2c762f58ff: entering candidate state: node="Node at e8f9b925-f428-27fc-af12-ab2c762f58ff [Candidate]" term=3
2025-01-26T02:45:14.515Z [ERROR] server-e8f9b925-f428-27fc-af12-ab2c762f58ff: failed to make requestVote RPC: target="{Voter server-e22ba692-0637-2f48-01de-94f5fdb431b0 e22ba692-0637-2f48-01de-94f5fdb431b0}" error="failed to connect to peer: e22ba692-0637-2f48-01de-94f5fdb431b0" term=3
2025-01-26T02:45:14.515Z [WARN]  server-e8f9b925-f428-27fc-af12-ab2c762f58ff: rejecting vote request since node is not in configuration: from=51026783-c7dd-237c-7ac4-78335869886b
--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.52s)
=== RUN   TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin
2025-01-26T02:45:14.516Z [INFO]  server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf Address:7bdbce1f-6e9c-4fcf-4af9-e916ace196cf} {Suffrage:Voter ID:server-3fbf18d0-7656-209c-dfdf-240c56b51e5d Address:3fbf18d0-7656-209c-dfdf-240c56b51e5d} {Suffrage:Voter ID:server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee Address:468a9cc0-3aad-cbe3-bdd1-b295050a76ee}]"
2025-01-26T02:45:14.516Z [INFO]  server-3fbf18d0-7656-209c-dfdf-240c56b51e5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf Address:7bdbce1f-6e9c-4fcf-4af9-e916ace196cf} {Suffrage:Voter ID:server-3fbf18d0-7656-209c-dfdf-240c56b51e5d Address:3fbf18d0-7656-209c-dfdf-240c56b51e5d} {Suffrage:Voter ID:server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee Address:468a9cc0-3aad-cbe3-bdd1-b295050a76ee}]"
2025-01-26T02:45:14.516Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf Address:7bdbce1f-6e9c-4fcf-4af9-e916ace196cf} {Suffrage:Voter ID:server-3fbf18d0-7656-209c-dfdf-240c56b51e5d Address:3fbf18d0-7656-209c-dfdf-240c56b51e5d} {Suffrage:Voter ID:server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee Address:468a9cc0-3aad-cbe3-bdd1-b295050a76ee}]"
2025-01-26T02:45:14.516Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:14.516Z [INFO]  server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf: entering follower state: follower="Node at 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf [Follower]" leader-address= leader-id=
2025-01-26T02:45:14.516Z [INFO]  server-3fbf18d0-7656-209c-dfdf-240c56b51e5d: entering follower state: follower="Node at 3fbf18d0-7656-209c-dfdf-240c56b51e5d [Follower]" leader-address= leader-id=
2025-01-26T02:45:14.516Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: entering follower state: follower="Node at 468a9cc0-3aad-cbe3-bdd1-b295050a76ee [Follower]" leader-address= leader-id=
2025-01-26T02:45:14.571Z [WARN]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:14.571Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: entering candidate state: node="Node at 468a9cc0-3aad-cbe3-bdd1-b295050a76ee [Candidate]" term=2
2025-01-26T02:45:14.572Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: election won: term=2 tally=2
2025-01-26T02:45:14.572Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: entering leader state: leader="Node at 468a9cc0-3aad-cbe3-bdd1-b295050a76ee [Leader]"
2025-01-26T02:45:14.572Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: added peer, starting replication: peer=server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf
2025-01-26T02:45:14.572Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: added peer, starting replication: peer=server-3fbf18d0-7656-209c-dfdf-240c56b51e5d
2025-01-26T02:45:14.572Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: pipelining replication: peer="{Voter server-3fbf18d0-7656-209c-dfdf-240c56b51e5d 3fbf18d0-7656-209c-dfdf-240c56b51e5d}"
2025-01-26T02:45:14.572Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: pipelining replication: peer="{Voter server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}"
2025-01-26T02:45:14.679Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:14.572391775 +0000 UTC m=+111.087424110 (1 nodes), 55.995739ms from start of poll, 56.17482ms from cluster start. Timeout at 2025-01-26 02:45:14.677411536 +0000 UTC m=+111.192443911, 105.019801ms after stability
2025-01-26T02:45:14.687Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:14.795Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:14.687691595 +0000 UTC m=+111.202723930 (2 nodes), 1.28µs from start of poll, 171.47464ms from cluster start. Timeout at 2025-01-26 02:45:14.792744876 +0000 UTC m=+111.307777251, 105.053321ms after stability
2025-01-26T02:45:14.795Z [ERROR] server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: failed to pipeline appendEntries: peer="{Voter server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}" error="append pipeline closed"
2025-01-26T02:45:14.795Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: aborting pipeline replication: peer="{Voter server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}"
2025-01-26T02:45:14.799Z [ERROR] server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: failed to heartbeat to: peer=7bdbce1f-6e9c-4fcf-4af9-e916ace196cf backoff time=10ms error="failed to connect to peer: 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf"
2025-01-26T02:45:14.803Z [ERROR] server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: failed to appendEntries to: peer="{Voter server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}" error="failed to connect to peer: 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf"
2025-01-26T02:45:14.819Z [ERROR] server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: failed to heartbeat to: peer=7bdbce1f-6e9c-4fcf-4af9-e916ace196cf backoff time=10ms error="failed to connect to peer: 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf"
2025-01-26T02:45:14.823Z [ERROR] server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: failed to appendEntries to: peer="{Voter server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}" error="failed to connect to peer: 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf"
2025-01-26T02:45:14.843Z [WARN]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: failed to contact: server-id=server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf time=51.671274ms
2025-01-26T02:45:14.843Z [ERROR] server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: failed to heartbeat to: peer=7bdbce1f-6e9c-4fcf-4af9-e916ace196cf backoff time=10ms error="failed to connect to peer: 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf"
2025-01-26T02:45:14.843Z [ERROR] server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: failed to appendEntries to: peer="{Voter server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}" error="failed to connect to peer: 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf"
2025-01-26T02:45:14.863Z [ERROR] server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: failed to heartbeat to: peer=7bdbce1f-6e9c-4fcf-4af9-e916ace196cf backoff time=20ms error="failed to connect to peer: 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf"
2025-01-26T02:45:14.871Z [ERROR] server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: failed to appendEntries to: peer="{Voter server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}" error="failed to connect to peer: 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf"
2025-01-26T02:45:14.887Z [WARN]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: failed to contact: server-id=server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf time=95.675581ms
2025-01-26T02:45:14.891Z [ERROR] server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: failed to heartbeat to: peer=7bdbce1f-6e9c-4fcf-4af9-e916ace196cf backoff time=25ms error="failed to connect to peer: 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf"
2025-01-26T02:45:14.903Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: updating configuration: command=RemoveServer server-id=server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf server-addr= servers="[{Suffrage:Voter ID:server-3fbf18d0-7656-209c-dfdf-240c56b51e5d Address:3fbf18d0-7656-209c-dfdf-240c56b51e5d} {Suffrage:Voter ID:server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee Address:468a9cc0-3aad-cbe3-bdd1-b295050a76ee}]"
2025-01-26T02:45:14.903Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: removed peer, stopping replication: peer=server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf last-index=3
2025-01-26T02:45:14.903Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: updating configuration: command=AddNonvoter server-id=server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf server-addr=7bdbce1f-6e9c-4fcf-4af9-e916ace196cf servers="[{Suffrage:Voter ID:server-3fbf18d0-7656-209c-dfdf-240c56b51e5d Address:3fbf18d0-7656-209c-dfdf-240c56b51e5d} {Suffrage:Voter ID:server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee Address:468a9cc0-3aad-cbe3-bdd1-b295050a76ee} {Suffrage:Nonvoter ID:server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf Address:7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}]"
2025-01-26T02:45:14.903Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: added peer, starting replication: peer=server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf
2025-01-26T02:45:14.904Z [ERROR] server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: failed to appendEntries to: peer="{Nonvoter server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}" error="failed to connect to peer: 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf"
2025-01-26T02:45:14.911Z [WARN]  server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf: heartbeat timeout reached, starting election: last-leader-addr=468a9cc0-3aad-cbe3-bdd1-b295050a76ee last-leader-id=server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee
2025-01-26T02:45:14.911Z [INFO]  server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf: entering candidate state: node="Node at 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf [Candidate]" term=23
2025-01-26T02:45:14.911Z [WARN]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: rejecting vote request since we have a leader: from=7bdbce1f-6e9c-4fcf-4af9-e916ace196cf leader=468a9cc0-3aad-cbe3-bdd1-b295050a76ee leader-id=server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee
2025-01-26T02:45:14.911Z [WARN]  server-3fbf18d0-7656-209c-dfdf-240c56b51e5d: rejecting vote request since we have a leader: from=7bdbce1f-6e9c-4fcf-4af9-e916ace196cf leader=468a9cc0-3aad-cbe3-bdd1-b295050a76ee leader-id=server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee
2025-01-26T02:45:14.923Z [ERROR] server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: peer has newer term, stopping replication: peer="{Voter server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}"
2025-01-26T02:45:14.923Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: entering follower state: follower="Node at 468a9cc0-3aad-cbe3-bdd1-b295050a76ee [Follower]" leader-address= leader-id=
2025-01-26T02:45:14.923Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: aborting pipeline replication: peer="{Voter server-3fbf18d0-7656-209c-dfdf-240c56b51e5d 3fbf18d0-7656-209c-dfdf-240c56b51e5d}"
2025-01-26T02:45:14.927Z [ERROR] server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: peer has newer term, stopping replication: peer="{Nonvoter server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}"
2025-01-26T02:45:14.975Z [WARN]  server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf: Election timeout reached, restarting election
2025-01-26T02:45:14.975Z [INFO]  server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf: entering candidate state: node="Node at 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf [Candidate]" term=24
2025-01-26T02:45:14.975Z [WARN]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: rejecting vote request since node is not a voter: from=7bdbce1f-6e9c-4fcf-4af9-e916ace196cf
2025-01-26T02:45:14.975Z [WARN]  server-3fbf18d0-7656-209c-dfdf-240c56b51e5d: rejecting vote request since we have a leader: from=7bdbce1f-6e9c-4fcf-4af9-e916ace196cf leader=468a9cc0-3aad-cbe3-bdd1-b295050a76ee leader-id=server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee
2025-01-26T02:45:14.991Z [WARN]  server-3fbf18d0-7656-209c-dfdf-240c56b51e5d: heartbeat timeout reached, starting election: last-leader-addr=468a9cc0-3aad-cbe3-bdd1-b295050a76ee last-leader-id=server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee
2025-01-26T02:45:14.991Z [INFO]  server-3fbf18d0-7656-209c-dfdf-240c56b51e5d: entering candidate state: node="Node at 3fbf18d0-7656-209c-dfdf-240c56b51e5d [Candidate]" term=3
2025-01-26T02:45:14.991Z [INFO]  server-3fbf18d0-7656-209c-dfdf-240c56b51e5d: entering follower state: follower="Node at 3fbf18d0-7656-209c-dfdf-240c56b51e5d [Follower]" leader-address= leader-id=
2025-01-26T02:45:15.011Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:15.011Z [WARN]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:15.011Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: entering candidate state: node="Node at 468a9cc0-3aad-cbe3-bdd1-b295050a76ee [Candidate]" term=25
2025-01-26T02:45:15.011Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: election won: term=25 tally=2
2025-01-26T02:45:15.011Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: entering leader state: leader="Node at 468a9cc0-3aad-cbe3-bdd1-b295050a76ee [Leader]"
2025-01-26T02:45:15.011Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: added peer, starting replication: peer=server-3fbf18d0-7656-209c-dfdf-240c56b51e5d
2025-01-26T02:45:15.011Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: added peer, starting replication: peer=server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf
2025-01-26T02:45:15.011Z [WARN]  server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf: failed to get previous log: previous-index=4 last-index=2 error="log not found"
2025-01-26T02:45:15.011Z [INFO]  server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf: entering follower state: follower="Node at 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf [Follower]" leader-address=468a9cc0-3aad-cbe3-bdd1-b295050a76ee leader-id=server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee
2025-01-26T02:45:15.011Z [WARN]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: appendEntries rejected, sending older logs: peer="{Nonvoter server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}" next=3
2025-01-26T02:45:15.011Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: pipelining replication: peer="{Nonvoter server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}"
2025-01-26T02:45:15.012Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: pipelining replication: peer="{Voter server-3fbf18d0-7656-209c-dfdf-240c56b51e5d 3fbf18d0-7656-209c-dfdf-240c56b51e5d}"
2025-01-26T02:45:15.119Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:15.012045403 +0000 UTC m=+111.527077738 (1 nodes), 399.602µs from start of poll, 495.828448ms from cluster start. Timeout at 2025-01-26 02:45:15.117104284 +0000 UTC m=+111.632136659, 105.058921ms after stability
2025-01-26T02:45:15.163Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: updating configuration: command=RemoveServer server-id=server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf server-addr= servers="[{Suffrage:Voter ID:server-3fbf18d0-7656-209c-dfdf-240c56b51e5d Address:3fbf18d0-7656-209c-dfdf-240c56b51e5d} {Suffrage:Voter ID:server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee Address:468a9cc0-3aad-cbe3-bdd1-b295050a76ee}]"
2025-01-26T02:45:15.163Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: removed peer, stopping replication: peer=server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf last-index=106
2025-01-26T02:45:15.163Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: updating configuration: command=AddVoter server-id=server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf server-addr=7bdbce1f-6e9c-4fcf-4af9-e916ace196cf servers="[{Suffrage:Voter ID:server-3fbf18d0-7656-209c-dfdf-240c56b51e5d Address:3fbf18d0-7656-209c-dfdf-240c56b51e5d} {Suffrage:Voter ID:server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee Address:468a9cc0-3aad-cbe3-bdd1-b295050a76ee} {Suffrage:Voter ID:server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf Address:7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}]"
2025-01-26T02:45:15.163Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: added peer, starting replication: peer=server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf
2025-01-26T02:45:15.163Z [WARN]  server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf: failed to get previous log: previous-index=107 last-index=105 error="log not found"
2025-01-26T02:45:15.163Z [WARN]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: appendEntries rejected, sending older logs: peer="{Voter server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}" next=106
2025-01-26T02:45:15.163Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: pipelining replication: peer="{Voter server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}"
2025-01-26T02:45:15.163Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: aborting pipeline replication: peer="{Nonvoter server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}"
2025-01-26T02:45:16.263Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:16.371Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:16.26404328 +0000 UTC m=+112.779075655 (1 nodes), 680ns from start of poll, 1.747826365s from cluster start. Timeout at 2025-01-26 02:45:16.36909184 +0000 UTC m=+112.884124175, 105.04852ms after stability
2025-01-26T02:45:16.371Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: aborting pipeline replication: peer="{Voter server-7bdbce1f-6e9c-4fcf-4af9-e916ace196cf 7bdbce1f-6e9c-4fcf-4af9-e916ace196cf}"
2025-01-26T02:45:16.371Z [INFO]  server-468a9cc0-3aad-cbe3-bdd1-b295050a76ee: aborting pipeline replication: peer="{Voter server-3fbf18d0-7656-209c-dfdf-240c56b51e5d 3fbf18d0-7656-209c-dfdf-240c56b51e5d}"
--- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.86s)
=== RUN   TestRaft_FollowerRemovalNoElection
2025-01-26T02:45:16.372Z [INFO]  server-253d46d2-583f-4b55-11b9-324294695e45: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-253d46d2-583f-4b55-11b9-324294695e45 Address:253d46d2-583f-4b55-11b9-324294695e45} {Suffrage:Voter ID:server-c00b30ab-ac20-40cd-90e9-6505011522ff Address:c00b30ab-ac20-40cd-90e9-6505011522ff} {Suffrage:Voter ID:server-e433932b-c3c4-2eac-f22a-540cb03cfadb Address:e433932b-c3c4-2eac-f22a-540cb03cfadb}]"
2025-01-26T02:45:16.372Z [INFO]  server-c00b30ab-ac20-40cd-90e9-6505011522ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-253d46d2-583f-4b55-11b9-324294695e45 Address:253d46d2-583f-4b55-11b9-324294695e45} {Suffrage:Voter ID:server-c00b30ab-ac20-40cd-90e9-6505011522ff Address:c00b30ab-ac20-40cd-90e9-6505011522ff} {Suffrage:Voter ID:server-e433932b-c3c4-2eac-f22a-540cb03cfadb Address:e433932b-c3c4-2eac-f22a-540cb03cfadb}]"
2025-01-26T02:45:16.372Z [INFO]  server-e433932b-c3c4-2eac-f22a-540cb03cfadb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-253d46d2-583f-4b55-11b9-324294695e45 Address:253d46d2-583f-4b55-11b9-324294695e45} {Suffrage:Voter ID:server-c00b30ab-ac20-40cd-90e9-6505011522ff Address:c00b30ab-ac20-40cd-90e9-6505011522ff} {Suffrage:Voter ID:server-e433932b-c3c4-2eac-f22a-540cb03cfadb Address:e433932b-c3c4-2eac-f22a-540cb03cfadb}]"
2025-01-26T02:45:16.372Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:16.372Z [INFO]  server-253d46d2-583f-4b55-11b9-324294695e45: entering follower state: follower="Node at 253d46d2-583f-4b55-11b9-324294695e45 [Follower]" leader-address= leader-id=
2025-01-26T02:45:16.372Z [INFO]  server-c00b30ab-ac20-40cd-90e9-6505011522ff: entering follower state: follower="Node at c00b30ab-ac20-40cd-90e9-6505011522ff [Follower]" leader-address= leader-id=
2025-01-26T02:45:16.372Z [INFO]  server-e433932b-c3c4-2eac-f22a-540cb03cfadb: entering follower state: follower="Node at e433932b-c3c4-2eac-f22a-540cb03cfadb [Follower]" leader-address= leader-id=
2025-01-26T02:45:16.515Z [WARN]  server-253d46d2-583f-4b55-11b9-324294695e45: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:16.515Z [INFO]  server-253d46d2-583f-4b55-11b9-324294695e45: entering candidate state: node="Node at 253d46d2-583f-4b55-11b9-324294695e45 [Candidate]" term=2
2025-01-26T02:45:16.515Z [WARN]  server-e433932b-c3c4-2eac-f22a-540cb03cfadb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:16.515Z [INFO]  server-e433932b-c3c4-2eac-f22a-540cb03cfadb: entering candidate state: node="Node at e433932b-c3c4-2eac-f22a-540cb03cfadb [Candidate]" term=2
2025-01-26T02:45:16.515Z [INFO]  server-e433932b-c3c4-2eac-f22a-540cb03cfadb: duplicate requestVote for same term: term=2
2025-01-26T02:45:16.515Z [INFO]  server-253d46d2-583f-4b55-11b9-324294695e45: election won: term=2 tally=2
2025-01-26T02:45:16.515Z [INFO]  server-253d46d2-583f-4b55-11b9-324294695e45: entering leader state: leader="Node at 253d46d2-583f-4b55-11b9-324294695e45 [Leader]"
2025-01-26T02:45:16.515Z [INFO]  server-253d46d2-583f-4b55-11b9-324294695e45: added peer, starting replication: peer=server-c00b30ab-ac20-40cd-90e9-6505011522ff
2025-01-26T02:45:16.515Z [INFO]  server-253d46d2-583f-4b55-11b9-324294695e45: added peer, starting replication: peer=server-e433932b-c3c4-2eac-f22a-540cb03cfadb
2025-01-26T02:45:16.515Z [INFO]  server-e433932b-c3c4-2eac-f22a-540cb03cfadb: entering follower state: follower="Node at e433932b-c3c4-2eac-f22a-540cb03cfadb [Follower]" leader-address=253d46d2-583f-4b55-11b9-324294695e45 leader-id=server-253d46d2-583f-4b55-11b9-324294695e45
2025-01-26T02:45:16.515Z [INFO]  server-253d46d2-583f-4b55-11b9-324294695e45: pipelining replication: peer="{Voter server-e433932b-c3c4-2eac-f22a-540cb03cfadb e433932b-c3c4-2eac-f22a-540cb03cfadb}"
2025-01-26T02:45:16.515Z [INFO]  server-c00b30ab-ac20-40cd-90e9-6505011522ff: duplicate requestVote for same term: term=2
2025-01-26T02:45:16.515Z [INFO]  server-253d46d2-583f-4b55-11b9-324294695e45: pipelining replication: peer="{Voter server-c00b30ab-ac20-40cd-90e9-6505011522ff c00b30ab-ac20-40cd-90e9-6505011522ff}"
2025-01-26T02:45:16.515Z [WARN]  server-253d46d2-583f-4b55-11b9-324294695e45: rejecting vote request since we have a leader: from=e433932b-c3c4-2eac-f22a-540cb03cfadb leader=253d46d2-583f-4b55-11b9-324294695e45 leader-id=server-253d46d2-583f-4b55-11b9-324294695e45
2025-01-26T02:45:16.723Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:16.515962729 +0000 UTC m=+113.030995104 (1 nodes), 143.618592ms from start of poll, 143.800553ms from cluster start. Timeout at 2025-01-26 02:45:16.720970089 +0000 UTC m=+113.236002464, 205.00736ms after stability
2025-01-26T02:45:16.723Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:16.936Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:16.72377219 +0000 UTC m=+113.238804525 (1 nodes), 1.68µs from start of poll, 351.609974ms from cluster start. Timeout at 2025-01-26 02:45:16.928825951 +0000 UTC m=+113.443858326, 205.053801ms after stability
2025-01-26T02:45:16.943Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:17.151Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:16.943453083 +0000 UTC m=+113.458485458 (2 nodes), 1.72µs from start of poll, 571.290907ms from cluster start. Timeout at 2025-01-26 02:45:17.148473003 +0000 UTC m=+113.663505378, 205.01992ms after stability
2025-01-26T02:45:17.151Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:17.359Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:17.151922308 +0000 UTC m=+113.666954643 (1 nodes), 1.56µs from start of poll, 779.760092ms from cluster start. Timeout at 2025-01-26 02:45:17.356962069 +0000 UTC m=+113.871994444, 205.039801ms after stability
    raft_test.go:2961: [INFO] restarting Node at c00b30ab-ac20-40cd-90e9-6505011522ff [Follower]
2025-01-26T02:45:17.360Z [INFO]  server-c00b30ab-ac20-40cd-90e9-6505011522ff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-253d46d2-583f-4b55-11b9-324294695e45 Address:253d46d2-583f-4b55-11b9-324294695e45} {Suffrage:Voter ID:server-c00b30ab-ac20-40cd-90e9-6505011522ff Address:c00b30ab-ac20-40cd-90e9-6505011522ff} {Suffrage:Voter ID:server-e433932b-c3c4-2eac-f22a-540cb03cfadb Address:e433932b-c3c4-2eac-f22a-540cb03cfadb}]"
2025-01-26T02:45:17.360Z [INFO]  server-c00b30ab-ac20-40cd-90e9-6505011522ff: entering follower state: follower="Node at c00b30ab-ac20-40cd-90e9-6505011522ff [Follower]" leader-address= leader-id=
2025-01-26T02:45:17.611Z [INFO]  server-253d46d2-583f-4b55-11b9-324294695e45: aborting pipeline replication: peer="{Voter server-e433932b-c3c4-2eac-f22a-540cb03cfadb e433932b-c3c4-2eac-f22a-540cb03cfadb}"
2025-01-26T02:45:18.032Z [ERROR] server-253d46d2-583f-4b55-11b9-324294695e45: failed to pipeline appendEntries: peer="{Voter server-c00b30ab-ac20-40cd-90e9-6505011522ff c00b30ab-ac20-40cd-90e9-6505011522ff}" error="command enqueue timeout"
2025-01-26T02:45:18.032Z [INFO]  server-253d46d2-583f-4b55-11b9-324294695e45: aborting pipeline replication: peer="{Voter server-c00b30ab-ac20-40cd-90e9-6505011522ff c00b30ab-ac20-40cd-90e9-6505011522ff}"
2025-01-26T02:45:18.127Z [ERROR] server-253d46d2-583f-4b55-11b9-324294695e45: failed to heartbeat to: peer=c00b30ab-ac20-40cd-90e9-6505011522ff backoff time=10ms error="command timed out"
--- PASS: TestRaft_FollowerRemovalNoElection (1.76s)
=== RUN   TestRaft_VoteWithNoIDNoAddr
2025-01-26T02:45:18.128Z [INFO]  server-e880a1be-3717-7f1f-32c6-e9faecee998e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e880a1be-3717-7f1f-32c6-e9faecee998e Address:e880a1be-3717-7f1f-32c6-e9faecee998e} {Suffrage:Voter ID:server-1994481a-2734-2402-af69-a340de6c8f73 Address:1994481a-2734-2402-af69-a340de6c8f73} {Suffrage:Voter ID:server-d4884b68-daeb-b87e-5f4d-99c8d94874e8 Address:d4884b68-daeb-b87e-5f4d-99c8d94874e8}]"
2025-01-26T02:45:18.128Z [INFO]  server-1994481a-2734-2402-af69-a340de6c8f73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e880a1be-3717-7f1f-32c6-e9faecee998e Address:e880a1be-3717-7f1f-32c6-e9faecee998e} {Suffrage:Voter ID:server-1994481a-2734-2402-af69-a340de6c8f73 Address:1994481a-2734-2402-af69-a340de6c8f73} {Suffrage:Voter ID:server-d4884b68-daeb-b87e-5f4d-99c8d94874e8 Address:d4884b68-daeb-b87e-5f4d-99c8d94874e8}]"
2025-01-26T02:45:18.128Z [INFO]  server-d4884b68-daeb-b87e-5f4d-99c8d94874e8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e880a1be-3717-7f1f-32c6-e9faecee998e Address:e880a1be-3717-7f1f-32c6-e9faecee998e} {Suffrage:Voter ID:server-1994481a-2734-2402-af69-a340de6c8f73 Address:1994481a-2734-2402-af69-a340de6c8f73} {Suffrage:Voter ID:server-d4884b68-daeb-b87e-5f4d-99c8d94874e8 Address:d4884b68-daeb-b87e-5f4d-99c8d94874e8}]"
2025-01-26T02:45:18.128Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:18.128Z [INFO]  server-e880a1be-3717-7f1f-32c6-e9faecee998e: entering follower state: follower="Node at e880a1be-3717-7f1f-32c6-e9faecee998e [Follower]" leader-address= leader-id=
2025-01-26T02:45:18.128Z [INFO]  server-1994481a-2734-2402-af69-a340de6c8f73: entering follower state: follower="Node at 1994481a-2734-2402-af69-a340de6c8f73 [Follower]" leader-address= leader-id=
2025-01-26T02:45:18.128Z [INFO]  server-d4884b68-daeb-b87e-5f4d-99c8d94874e8: entering follower state: follower="Node at d4884b68-daeb-b87e-5f4d-99c8d94874e8 [Follower]" leader-address= leader-id=
2025-01-26T02:45:18.191Z [WARN]  server-e880a1be-3717-7f1f-32c6-e9faecee998e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-26T02:45:18.191Z [INFO]  server-e880a1be-3717-7f1f-32c6-e9faecee998e: entering candidate state: node="Node at e880a1be-3717-7f1f-32c6-e9faecee998e [Candidate]" term=2
2025-01-26T02:45:18.191Z [INFO]  server-e880a1be-3717-7f1f-32c6-e9faecee998e: election won: term=2 tally=2
2025-01-26T02:45:18.191Z [INFO]  server-e880a1be-3717-7f1f-32c6-e9faecee998e: entering leader state: leader="Node at e880a1be-3717-7f1f-32c6-e9faecee998e [Leader]"
2025-01-26T02:45:18.191Z [INFO]  server-e880a1be-3717-7f1f-32c6-e9faecee998e: added peer, starting replication: peer=server-1994481a-2734-2402-af69-a340de6c8f73
2025-01-26T02:45:18.191Z [INFO]  server-e880a1be-3717-7f1f-32c6-e9faecee998e: added peer, starting replication: peer=server-d4884b68-daeb-b87e-5f4d-99c8d94874e8
2025-01-26T02:45:18.191Z [INFO]  server-e880a1be-3717-7f1f-32c6-e9faecee998e: pipelining replication: peer="{Voter server-d4884b68-daeb-b87e-5f4d-99c8d94874e8 d4884b68-daeb-b87e-5f4d-99c8d94874e8}"
2025-01-26T02:45:18.191Z [INFO]  server-e880a1be-3717-7f1f-32c6-e9faecee998e: pipelining replication: peer="{Voter server-1994481a-2734-2402-af69-a340de6c8f73 1994481a-2734-2402-af69-a340de6c8f73}"
2025-01-26T02:45:18.299Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:18.192006939 +0000 UTC m=+114.707039314 (1 nodes), 63.754867ms from start of poll, 63.935348ms from cluster start. Timeout at 2025-01-26 02:45:18.297011019 +0000 UTC m=+114.812043354, 105.00404ms after stability
2025-01-26T02:45:18.299Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-26T02:45:18.407Z [INFO]  cluster: stable state for Leader reached at 2025-01-26 02:45:18.299921034 +0000 UTC m=+114.814953369 (1 nodes), 960ns from start of poll, 171.849403ms from cluster start. Timeout at 2025-01-26 02:45:18.404962674 +0000 UTC m=+114.919995049, 105.04168ms after stability
2025-01-26T02:45:18.415Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-26T02:45:18.521Z [INFO]  cluster: stable state for Follower reached at 2025-01-26 02:45:18.415471335 +0000 UTC m=+114.930503710 (2 nodes), 1.2µs from start of poll, 287.399744ms from cluster start. Timeout at 2025-01-26 02:45:18.520517655 +0000 UTC m=+115.035549990, 105.04628ms after stability
2025-01-26T02:45:18.521Z [ERROR] server-e880a1be-3717-7f1f-32c6-e9faecee998e: failed to pipeline appendEntries: peer="{Voter server-d4884b68-daeb-b87e-5f4d-99c8d94874e8 d4884b68-daeb-b87e-5f4d-99c8d94874e8}" error="append pipeline closed"
2025-01-26T02:45:18.521Z [INFO]  server-e880a1be-3717-7f1f-32c6-e9faecee998e: aborting pipeline replication: peer="{Voter server-d4884b68-daeb-b87e-5f4d-99c8d94874e8 d4884b68-daeb-b87e-5f4d-99c8d94874e8}"
2025-01-26T02:45:18.521Z [ERROR] server-e880a1be-3717-7f1f-32c6-e9faecee998e: failed to pipeline appendEntries: peer="{Voter server-1994481a-2734-2402-af69-a340de6c8f73 1994481a-2734-2402-af69-a340de6c8f73}" error="append pipeline closed"
2025-01-26T02:45:18.521Z [INFO]  server-e880a1be-3717-7f1f-32c6-e9faecee998e: aborting pipeline replication: peer="{Voter server-1994481a-2734-2402-af69-a340de6c8f73 1994481a-2734-2402-af69-a340de6c8f73}"
2025-01-26T02:45:18.531Z [ERROR] server-e880a1be-3717-7f1f-32c6-e9faecee998e: failed to heartbeat to: peer=d4884b68-daeb-b87e-5f4d-99c8d94874e8 backoff time=10ms error="failed to connect to peer: d4884b68-daeb-b87e-5f4d-99c8d94874e8"
2025-01-26T02:45:18.531Z [ERROR] server-e880a1be-3717-7f1f-32c6-e9faecee998e: failed to appendEntries to: peer="{Voter server-d4884b68-daeb-b87e-5f4d-99c8d94874e8 d4884b68-daeb-b87e-5f4d-99c8d94874e8}" error="failed to connect to peer: d4884b68-daeb-b87e-5f4d-99c8d94874e8"
2025-01-26T02:45:18.531Z [ERROR] server-e880a1be-3717-7f1f-32c6-e9faecee998e: failed to appendEntries to: peer="{Voter server-1994481a-2734-2402-af69-a340de6c8f73 1994481a-2734-2402-af69-a340de6c8f73}" error="failed to connect to peer: 1994481a-2734-2402-af69-a340de6c8f73"
2025-01-26T02:45:18.531Z [ERROR] server-e880a1be-3717-7f1f-32c6-e9faecee998e: failed to heartbeat to: peer=1994481a-2734-2402-af69-a340de6c8f73 backoff time=10ms error="failed to connect to peer: 1994481a-2734-2402-af69-a340de6c8f73"
2025-01-26T02:45:18.555Z [ERROR] server-e880a1be-3717-7f1f-32c6-e9faecee998e: failed to heartbeat to: peer=d4884b68-daeb-b87e-5f4d-99c8d94874e8 backoff time=10ms error="failed to connect to peer: d4884b68-daeb-b87e-5f4d-99c8d94874e8"
2025-01-26T02:45:18.558Z [ERROR] server-e880a1be-3717-7f1f-32c6-e9faecee998e: failed to appendEntries to: peer="{Voter server-d4884b68-daeb-b87e-5f4d-99c8d94874e8 d4884b68-daeb-b87e-5f4d-99c8d94874e8}" error="failed to connect to peer: d4884b68-daeb-b87e-5f4d-99c8d94874e8"
2025-01-26T02:45:18.558Z [ERROR] server-e880a1be-3717-7f1f-32c6-e9faecee998e: failed to heartbeat to: peer=1994481a-2734-2402-af69-a340de6c8f73 backoff time=10ms error="failed to connect to peer: 1994481a-2734-2402-af69-a340de6c8f73"
2025-01-26T02:45:18.558Z [ERROR] server-e880a1be-3717-7f1f-32c6-e9faecee998e: failed to appendEntries to: peer="{Voter server-1994481a-2734-2402-af69-a340de6c8f73 1994481a-2734-2402-af69-a340de6c8f73}" error="failed to connect to peer: 1994481a-2734-2402-af69-a340de6c8f73"
2025-01-26T02:45:18.579Z [ERROR] server-e880a1be-3717-7f1f-32c6-e9faecee998e: failed to heartbeat to: peer=1994481a-2734-2402-af69-a340de6c8f73 backoff time=10ms error="failed to connect to peer: 1994481a-2734-2402-af69-a340de6c8f73"
2025-01-26T02:45:18.579Z [ERROR] server-e880a1be-3717-7f1f-32c6-e9faecee998e: failed to heartbeat to: peer=d4884b68-daeb-b87e-5f4d-99c8d94874e8 backoff time=10ms error="failed to connect to peer: d4884b68-daeb-b87e-5f4d-99c8d94874e8"
2025-01-26T02:45:18.579Z [ERROR] server-e880a1be-3717-7f1f-32c6-e9faecee998e: failed to appendEntries to: peer="{Voter server-d4884b68-daeb-b87e-5f4d-99c8d94874e8 d4884b68-daeb-b87e-5f4d-99c8d94874e8}" error="failed to connect to peer: d4884b68-daeb-b87e-5f4d-99c8d94874e8"
2025-01-26T02:45:18.582Z [ERROR] server-e880a1be-3717-7f1f-32c6-e9faecee998e: failed to appendEntries to: peer="{Voter server-1994481a-2734-2402-af69-a340de6c8f73 1994481a-2734-2402-af69-a340de6c8f73}" error="failed to connect to peer: 1994481a-2734-2402-af69-a340de6c8f73"
2025-01-26T02:45:18.583Z [WARN]  server-e880a1be-3717-7f1f-32c6-e9faecee998e: failed to contact: server-id=server-1994481a-2734-2402-af69-a340de6c8f73 time=63.868947ms
2025-01-26T02:45:18.583Z [WARN]  server-e880a1be-3717-7f1f-32c6-e9faecee998e: failed to contact: server-id=server-d4884b68-daeb-b87e-5f4d-99c8d94874e8 time=63.938148ms
2025-01-26T02:45:18.583Z [WARN]  server-e880a1be-3717-7f1f-32c6-e9faecee998e: failed to contact quorum of nodes, stepping down
2025-01-26T02:45:18.583Z [INFO]  server-e880a1be-3717-7f1f-32c6-e9faecee998e: entering follower state: follower="Node at e880a1be-3717-7f1f-32c6-e9faecee998e [Follower]" leader-address= leader-id=
2025-01-26T02:45:18.611Z [WARN]  server-d4884b68-daeb-b87e-5f4d-99c8d94874e8: heartbeat timeout reached, starting election: last-leader-addr=e880a1be-3717-7f1f-32c6-e9faecee998e last-leader-id=server-e880a1be-3717-7f1f-32c6-e9faecee998e
2025-01-26T02:45:18.611Z [INFO]  server-d4884b68-daeb-b87e-5f4d-99c8d94874e8: entering candidate state: node="Node at d4884b68-daeb-b87e-5f4d-99c8d94874e8 [Candidate]" term=3
2025-01-26T02:45:18.611Z [WARN]  server-1994481a-2734-2402-af69-a340de6c8f73: rejecting vote request since we have a leader: from=d4884b68-daeb-b87e-5f4d-99c8d94874e8 leader=e880a1be-3717-7f1f-32c6-e9faecee998e leader-id=server-e880a1be-3717-7f1f-32c6-e9faecee998e
2025-01-26T02:45:18.611Z [ERROR] server-d4884b68-daeb-b87e-5f4d-99c8d94874e8: failed to make requestVote RPC: target="{Voter server-e880a1be-3717-7f1f-32c6-e9faecee998e e880a1be-3717-7f1f-32c6-e9faecee998e}" error="failed to connect to peer: e880a1be-3717-7f1f-32c6-e9faecee998e" term=3
2025-01-26T02:45:18.631Z [WARN]  server-1994481a-2734-2402-af69-a340de6c8f73: heartbeat timeout reached, starting election: last-leader-addr=e880a1be-3717-7f1f-32c6-e9faecee998e last-leader-id=server-e880a1be-3717-7f1f-32c6-e9faecee998e
2025-01-26T02:45:18.631Z [INFO]  server-1994481a-2734-2402-af69-a340de6c8f73: entering candidate state: node="Node at 1994481a-2734-2402-af69-a340de6c8f73 [Candidate]" term=3
2025-01-26T02:45:18.631Z [INFO]  server-d4884b68-daeb-b87e-5f4d-99c8d94874e8: duplicate requestVote for same term: term=3
2025-01-26T02:45:18.631Z [ERROR] server-1994481a-2734-2402-af69-a340de6c8f73: failed to make requestVote RPC: target="{Voter server-e880a1be-3717-7f1f-32c6-e9faecee998e e880a1be-3717-7f1f-32c6-e9faecee998e}" error="failed to connect to peer: e880a1be-3717-7f1f-32c6-e9faecee998e" term=3
2025-01-26T02:45:18.634Z [INFO]  server-1994481a-2734-2402-af69-a340de6c8f73: entering follower state: follower="Node at 1994481a-2734-2402-af69-a340de6c8f73 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VoteWithNoIDNoAddr (0.51s)
=== RUN   TestRaft_runFollower_State_Transition
=== RUN   TestRaft_runFollower_State_Transition/NonVoter
2025-01-26T02:45:18.634Z [INFO]  127.0.0.1:38821 :: starting node: addr=127.0.0.1:38821
2025-01-26T02:45:18.634Z [INFO]  127.0.0.1:38821 :: initial configuration: index=0 servers=[]
2025-01-26T02:45:18.634Z [INFO]  127.0.0.1:38821 :: entering follower state: follower="Node at 127.0.0.1:38821 [Follower]" leader-address= leader-id=
2025-01-26T02:45:18.715Z [WARN]  127.0.0.1:38821 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
=== RUN   TestRaft_runFollower_State_Transition/Voter
2025-01-26T02:45:18.802Z [INFO]  127.0.0.1:35667 :: starting node: addr=127.0.0.1:35667
2025-01-26T02:45:18.802Z [INFO]  127.0.0.1:35667 :: initial configuration: index=0 servers=[]
2025-01-26T02:45:18.802Z [INFO]  127.0.0.1:35667 :: entering follower state: follower="Node at 127.0.0.1:35667 [Follower]" leader-address= leader-id=
2025-01-26T02:45:18.872Z [WARN]  127.0.0.1:35667 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
=== RUN   TestRaft_runFollower_State_Transition/Not_in_Config
2025-01-26T02:45:18.959Z [INFO]  127.0.0.1:44165 :: starting node: addr=127.0.0.1:44165
2025-01-26T02:45:18.959Z [INFO]  127.0.0.1:44165 :: initial configuration: index=0 servers=[]
2025-01-26T02:45:18.959Z [INFO]  127.0.0.1:44165 :: entering follower state: follower="Node at 127.0.0.1:44165 [Follower]" leader-address= leader-id=
2025-01-26T02:45:19.035Z [WARN]  127.0.0.1:44165 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
--- PASS: TestRaft_runFollower_State_Transition (0.48s)
    --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.17s)
    --- PASS: TestRaft_runFollower_State_Transition/Voter (0.16s)
    --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s)
=== RUN   TestRaft_runFollower_ReloadTimeoutConfigs
2025-01-26T02:45:19.112Z [INFO]  127.0.0.1:38223 :: starting node: addr=127.0.0.1:38223
2025-01-26T02:45:19.112Z [INFO]  127.0.0.1:38223 :: initial configuration: index=0 servers=[]
2025-01-26T02:45:19.112Z [INFO]  127.0.0.1:38223 :: entering follower state: follower="Node at 127.0.0.1:38223 [Follower]" leader-address= leader-id=
2025-01-26T02:45:19.112Z [WARN]  127.0.0.1:38223 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
--- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s)
=== RUN   TestSaturationMetric
=== RUN   TestSaturationMetric/without_smoothing
--- PASS: TestSaturationMetric (0.00s)
    --- PASS: TestSaturationMetric/without_smoothing (0.00s)
=== RUN   TestSaturationMetric_IncorrectUsage
=== RUN   TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_first
--- PASS: TestSaturationMetric_IncorrectUsage (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s)
=== RUN   TestTCPTransport_BadAddr
--- PASS: TestTCPTransport_BadAddr (0.00s)
=== RUN   TestTCPTransport_EmptyAddr
--- PASS: TestTCPTransport_EmptyAddr (0.00s)
=== RUN   TestTCPTransport_WithAdvertise
--- PASS: TestTCPTransport_WithAdvertise (0.00s)
=== RUN   TestTransport_StartStop
--- PASS: TestTransport_StartStop (0.00s)
=== RUN   TestTransport_AppendEntries
--- PASS: TestTransport_AppendEntries (0.00s)
=== RUN   TestTransport_AppendEntriesPipeline
--- PASS: TestTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestTransport_RequestVote
--- PASS: TestTransport_RequestVote (0.00s)
=== RUN   TestTransport_InstallSnapshot
--- PASS: TestTransport_InstallSnapshot (0.00s)
=== RUN   TestTransport_EncodeDecode
--- PASS: TestTransport_EncodeDecode (0.00s)
=== RUN   TestRandomTimeout
--- PASS: TestRandomTimeout (0.00s)
=== RUN   TestNewSeed
--- PASS: TestNewSeed (0.00s)
=== RUN   TestRandomTimeout_NoTime
--- PASS: TestRandomTimeout_NoTime (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
=== RUN   TestMax
--- PASS: TestMax (0.00s)
=== RUN   TestGenerateUUID
--- PASS: TestGenerateUUID (0.00s)
=== RUN   TestBackoff
--- PASS: TestBackoff (0.00s)
=== RUN   TestOverrideNotifyBool
--- PASS: TestOverrideNotifyBool (0.00s)
PASS
ok  	github.com/hashicorp/raft	115.830s
make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
   create-stamp debian/debhelper-build-stamp
   dh_testroot -O--buildsystem=golang
   dh_prep -O--buildsystem=golang
   dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang
   dh_installdocs -O--buildsystem=golang
   dh_installchangelogs -O--buildsystem=golang
   dh_installsystemduser -O--buildsystem=golang
   dh_perl -O--buildsystem=golang
   dh_link -O--buildsystem=golang
   dh_strip_nondeterminism -O--buildsystem=golang
   dh_compress -O--buildsystem=golang
   dh_fixperms -O--buildsystem=golang
   dh_missing -O--buildsystem=golang
   dh_installdeb -O--buildsystem=golang
   dh_golang -O--buildsystem=golang
   dh_gencontrol -O--buildsystem=golang
   dh_md5sums -O--buildsystem=golang
   dh_builddeb -O--buildsystem=golang
dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'.
 dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_arm64.buildinfo
 dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_arm64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-buildpackage: info: binary-only upload (no source included)
dpkg-genchanges: info: including full source code in upload
I: copying local configuration
I: user script /srv/workspace/pbuilder/778150/tmp/hooks/B01_cleanup starting
I: user script /srv/workspace/pbuilder/778150/tmp/hooks/B01_cleanup finished
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /srv/workspace/pbuilder/778150 and its subdirectories
I: Current time: Sun Jan 26 16:45:29 +14 2025
I: pbuilder-time-stamp: 1737859529