I: pbuilder: network access will be disabled during build
I: Current time: Wed Jan 29 04:49:51 +14 2025
I: pbuilder-time-stamp: 1738075791
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/unstable-reproducible-base.tgz]
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: policy-rc.d already exists
I: Copying source file
I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc]
I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz]
I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz]
I: Extracting source
dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: unsupported subcommand
dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying 0001-Skip-unreliable-tests.patch
I: Not using root during the build.
I: Installing the build-deps
I: user script /srv/workspace/pbuilder/31365/tmp/hooks/D01_modify_environment starting
debug: Running on virt64a.
I: Changing host+domainname to test build reproducibility
I: Adding a custom variable just for the fun of it...
I: Changing /bin/sh to bash
'/bin/sh' -> '/bin/bash'
lrwxrwxrwx 1 root root 9 Jan 28 14:50 /bin/sh -> /bin/bash
I: Setting pbuilder2's login shell to /bin/bash
I: Setting pbuilder2's GECOS to second user,second room,second work-phone,second home-phone,second other
I: user script /srv/workspace/pbuilder/31365/tmp/hooks/D01_modify_environment finished
I: user script /srv/workspace/pbuilder/31365/tmp/hooks/D02_print_environment starting
I: set
  BASH=/bin/sh
  BASHOPTS=checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:globskipdots:hostcomplete:interactive_comments:patsub_replacement:progcomp:promptvars:sourcepath
  BASH_ALIASES=()
  BASH_ARGC=()
  BASH_ARGV=()
  BASH_CMDS=()
  BASH_LINENO=([0]="12" [1]="0")
  BASH_LOADABLES_PATH=/usr/local/lib/bash:/usr/lib/bash:/opt/local/lib/bash:/usr/pkg/lib/bash:/opt/pkg/lib/bash:.
  BASH_SOURCE=([0]="/tmp/hooks/D02_print_environment" [1]="/tmp/hooks/D02_print_environment")
  BASH_VERSINFO=([0]="5" [1]="2" [2]="37" [3]="1" [4]="release" [5]="arm-unknown-linux-gnueabihf")
  BASH_VERSION='5.2.37(1)-release'
  BUILDDIR=/build/reproducible-path
  BUILDUSERGECOS='second user,second room,second work-phone,second home-phone,second other'
  BUILDUSERNAME=pbuilder2
  BUILD_ARCH=armhf
  DEBIAN_FRONTEND=noninteractive
  DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=4 '
  DIRSTACK=()
  DISTRIBUTION=unstable
  EUID=0
  FUNCNAME=([0]="Echo" [1]="main")
  GROUPS=()
  HOME=/root
  HOSTNAME=i-capture-the-hostname
  HOSTTYPE=arm
  HOST_ARCH=armhf
  IFS=' 	
  '
  INVOCATION_ID=1f1135c6ed76417ab154bb54404cfc95
  LANG=C
  LANGUAGE=it_CH:it
  LC_ALL=C
  MACHTYPE=arm-unknown-linux-gnueabihf
  MAIL=/var/mail/root
  OPTERR=1
  OPTIND=1
  OSTYPE=linux-gnueabihf
  PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path
  PBCURRENTCOMMANDLINEOPERATION=build
  PBUILDER_OPERATION=build
  PBUILDER_PKGDATADIR=/usr/share/pbuilder
  PBUILDER_PKGLIBDIR=/usr/lib/pbuilder
  PBUILDER_SYSCONFDIR=/etc
  PIPESTATUS=([0]="0")
  POSIXLY_CORRECT=y
  PPID=31365
  PS4='+ '
  PWD=/
  SHELL=/bin/bash
  SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:posix
  SHLVL=3
  SUDO_COMMAND='/usr/bin/timeout -k 24.1h 24h /usr/bin/ionice -c 3 /usr/bin/nice -n 11 /usr/bin/unshare --uts -- /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.mPRYFda3/pbuilderrc_yuu9 --distribution unstable --hookdir /etc/pbuilder/rebuild-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/unstable-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.mPRYFda3/b2 --logfile b2/build.log golang-github-hashicorp-raft_1.5.0-1.dsc'
  SUDO_GID=114
  SUDO_UID=108
  SUDO_USER=jenkins
  TERM=unknown
  TZ=/usr/share/zoneinfo/Etc/GMT-14
  UID=0
  USER=root
  _='I: set'
  http_proxy=http://10.0.0.15:3142/
I: uname -a
  Linux i-capture-the-hostname 6.1.0-30-arm64 #1 SMP Debian 6.1.124-1 (2025-01-12) aarch64 GNU/Linux
I: ls -l /bin
  lrwxrwxrwx 1 root root 7 Nov 22 14:40 /bin -> usr/bin
I: user script /srv/workspace/pbuilder/31365/tmp/hooks/D02_print_environment finished
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: armhf
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 19568 files and directories currently installed.)
Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ...
Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however:
  Package debhelper-compat is not installed.
 pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however:
  Package dh-sequence-golang is not installed.
 pbuilder-satisfydepends-dummy depends on golang-any; however:
  Package golang-any is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however:
  Package golang-github-armon-go-metrics-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however:
  Package golang-github-hashicorp-go-hclog-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however:
  Package golang-github-hashicorp-go-msgpack-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however:
  Package golang-github-stretchr-testify-dev is not installed.

Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
Building tag database...
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
The following NEW packages will be installed:
  autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.23-go{a} golang-1.23-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libunistring5{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} 
The following packages are RECOMMENDED but will NOT be installed:
  curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 
0 packages upgraded, 106 newly installed, 0 to remove and 0 not upgraded.
Need to get 123 MB of archives. After unpacking 681 MB will be used.
Writing extended state information...
Get: 1 http://deb.debian.org/debian unstable/main armhf sensible-utils all 0.0.24 [24.8 kB]
Get: 2 http://deb.debian.org/debian unstable/main armhf libmagic-mgc armhf 1:5.45-3+b1 [314 kB]
Get: 3 http://deb.debian.org/debian unstable/main armhf libmagic1t64 armhf 1:5.45-3+b1 [98.5 kB]
Get: 4 http://deb.debian.org/debian unstable/main armhf file armhf 1:5.45-3+b1 [42.3 kB]
Get: 5 http://deb.debian.org/debian unstable/main armhf gettext-base armhf 0.23.1-1 [238 kB]
Get: 6 http://deb.debian.org/debian unstable/main armhf libuchardet0 armhf 0.0.8-1+b2 [65.6 kB]
Get: 7 http://deb.debian.org/debian unstable/main armhf groff-base armhf 1.23.0-7 [1095 kB]
Get: 8 http://deb.debian.org/debian unstable/main armhf bsdextrautils armhf 2.40.4-2 [84.6 kB]
Get: 9 http://deb.debian.org/debian unstable/main armhf libpipeline1 armhf 1.5.8-1 [35.0 kB]
Get: 10 http://deb.debian.org/debian unstable/main armhf man-db armhf 2.13.0-1 [1382 kB]
Get: 11 http://deb.debian.org/debian unstable/main armhf m4 armhf 1.4.19-5 [272 kB]
Get: 12 http://deb.debian.org/debian unstable/main armhf autoconf all 2.72-3 [493 kB]
Get: 13 http://deb.debian.org/debian unstable/main armhf autotools-dev all 20220109.1 [51.6 kB]
Get: 14 http://deb.debian.org/debian unstable/main armhf automake all 1:1.17-2 [861 kB]
Get: 15 http://deb.debian.org/debian unstable/main armhf autopoint all 0.23.1-1 [770 kB]
Get: 16 http://deb.debian.org/debian unstable/main armhf libdebhelper-perl all 13.24.1 [90.9 kB]
Get: 17 http://deb.debian.org/debian unstable/main armhf libtool all 2.5.4-2 [539 kB]
Get: 18 http://deb.debian.org/debian unstable/main armhf dh-autoreconf all 20 [17.1 kB]
Get: 19 http://deb.debian.org/debian unstable/main armhf libarchive-zip-perl all 1.68-1 [104 kB]
Get: 20 http://deb.debian.org/debian unstable/main armhf libfile-stripnondeterminism-perl all 1.14.1-1 [19.6 kB]
Get: 21 http://deb.debian.org/debian unstable/main armhf dh-strip-nondeterminism all 1.14.1-1 [8584 B]
Get: 22 http://deb.debian.org/debian unstable/main armhf libelf1t64 armhf 0.192-4 [184 kB]
Get: 23 http://deb.debian.org/debian unstable/main armhf dwz armhf 0.15-1+b2 [106 kB]
Get: 24 http://deb.debian.org/debian unstable/main armhf libunistring5 armhf 1.3-1 [444 kB]
Get: 25 http://deb.debian.org/debian unstable/main armhf libicu72 armhf 72.1-6 [9086 kB]
Get: 26 http://deb.debian.org/debian unstable/main armhf libxml2 armhf 2.12.7+dfsg+really2.9.14-0.2+b1 [605 kB]
Get: 27 http://deb.debian.org/debian unstable/main armhf gettext armhf 0.23.1-1 [1560 kB]
Get: 28 http://deb.debian.org/debian unstable/main armhf intltool-debian all 0.35.0+20060710.6 [22.9 kB]
Get: 29 http://deb.debian.org/debian unstable/main armhf po-debconf all 1.0.21+nmu1 [248 kB]
Get: 30 http://deb.debian.org/debian unstable/main armhf debhelper all 13.24.1 [920 kB]
Get: 31 http://deb.debian.org/debian unstable/main armhf dh-golang all 1.62 [24.7 kB]
Get: 32 http://deb.debian.org/debian unstable/main armhf golang-1.23-src all 1.23.5-1 [19.1 MB]
Get: 33 http://deb.debian.org/debian unstable/main armhf golang-1.23-go armhf 1.23.5-1 [23.6 MB]
Get: 34 http://deb.debian.org/debian unstable/main armhf golang-src all 2:1.23~2 [5088 B]
Get: 35 http://deb.debian.org/debian unstable/main armhf golang-go armhf 2:1.23~2 [44.3 kB]
Get: 36 http://deb.debian.org/debian unstable/main armhf golang-any armhf 2:1.23~2 [5172 B]
Get: 37 http://deb.debian.org/debian unstable/main armhf golang-github-alecthomas-units-dev all 0.0~git20240626.61d1e34-1 [9032 B]
Get: 38 http://deb.debian.org/debian unstable/main armhf golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB]
Get: 39 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB]
Get: 40 http://deb.debian.org/debian unstable/main armhf golang-golang-x-sys-dev all 0.22.0-1 [431 kB]
Get: 41 http://deb.debian.org/debian unstable/main armhf golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B]
Get: 42 http://deb.debian.org/debian unstable/main armhf golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB]
Get: 43 http://deb.debian.org/debian unstable/main armhf golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB]
Get: 44 http://deb.debian.org/debian unstable/main armhf golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB]
Get: 45 http://deb.debian.org/debian unstable/main armhf golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB]
Get: 46 http://deb.debian.org/debian unstable/main armhf golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB]
Get: 47 http://deb.debian.org/debian unstable/main armhf golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB]
Get: 48 http://deb.debian.org/debian unstable/main armhf golang-github-stretchr-testify-dev all 1.9.0-1 [74.7 kB]
Get: 49 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB]
Get: 50 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB]
Get: 51 http://deb.debian.org/debian unstable/main armhf golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB]
Get: 52 http://deb.debian.org/debian unstable/main armhf golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B]
Get: 53 http://deb.debian.org/debian unstable/main armhf golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB]
Get: 54 http://deb.debian.org/debian unstable/main armhf golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB]
Get: 55 http://deb.debian.org/debian unstable/main armhf golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB]
Get: 56 http://deb.debian.org/debian unstable/main armhf golang-google-protobuf-dev all 1.33.0-1 [625 kB]
Get: 57 http://deb.debian.org/debian unstable/main armhf golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB]
Get: 58 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B]
Get: 59 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB]
Get: 60 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB]
Get: 61 http://deb.debian.org/debian unstable/main armhf golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB]
Get: 62 http://deb.debian.org/debian unstable/main armhf golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB]
Get: 63 http://deb.debian.org/debian unstable/main armhf golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB]
Get: 64 http://deb.debian.org/debian unstable/main armhf golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B]
Get: 65 http://deb.debian.org/debian unstable/main armhf golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB]
Get: 66 http://deb.debian.org/debian unstable/main armhf golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB]
Get: 67 http://deb.debian.org/debian unstable/main armhf golang-github-klauspost-compress-dev all 1.17.11+ds1-1 [33.3 MB]
Get: 68 http://deb.debian.org/debian unstable/main armhf golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB]
Get: 69 http://deb.debian.org/debian unstable/main armhf golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB]
Get: 70 http://deb.debian.org/debian unstable/main armhf golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB]
Get: 71 http://deb.debian.org/debian unstable/main armhf golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB]
Get: 72 http://deb.debian.org/debian unstable/main armhf golang-golang-x-term-dev all 0.22.0-1 [14.8 kB]
Get: 73 http://deb.debian.org/debian unstable/main armhf golang-golang-x-text-dev all 0.16.0-1 [4003 kB]
Get: 74 http://deb.debian.org/debian unstable/main armhf golang-golang-x-crypto-dev all 1:0.25.0-1 [1682 kB]
Get: 75 http://deb.debian.org/debian unstable/main armhf golang-golang-x-net-dev all 1:0.27.0-1 [898 kB]
Get: 76 http://deb.debian.org/debian unstable/main armhf golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB]
Get: 77 http://deb.debian.org/debian unstable/main armhf golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB]
Get: 78 http://deb.debian.org/debian unstable/main armhf golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB]
Get: 79 http://deb.debian.org/debian unstable/main armhf golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB]
Get: 80 http://deb.debian.org/debian unstable/main armhf golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B]
Get: 81 http://deb.debian.org/debian unstable/main armhf golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B]
Get: 82 http://deb.debian.org/debian unstable/main armhf golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB]
Get: 83 http://deb.debian.org/debian unstable/main armhf golang-golang-x-oauth2-dev all 0.15.0-1 [38.0 kB]
Get: 84 http://deb.debian.org/debian unstable/main armhf golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB]
Get: 85 http://deb.debian.org/debian unstable/main armhf golang-github-prometheus-common-dev all 0.61.0-1 [120 kB]
Get: 86 http://deb.debian.org/debian unstable/main armhf golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB]
Get: 87 http://deb.debian.org/debian unstable/main armhf golang-github-prometheus-procfs-dev all 0.14.0-1 [183 kB]
Get: 88 http://deb.debian.org/debian unstable/main armhf golang-github-prometheus-client-golang-dev all 1.20.5-1 [188 kB]
Get: 89 http://deb.debian.org/debian unstable/main armhf golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB]
Get: 90 http://deb.debian.org/debian unstable/main armhf golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB]
Get: 91 http://deb.debian.org/debian unstable/main armhf golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B]
Get: 92 http://deb.debian.org/debian unstable/main armhf golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB]
Get: 93 http://deb.debian.org/debian unstable/main armhf golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B]
Get: 94 http://deb.debian.org/debian unstable/main armhf libsasl2-modules-db armhf 2.1.28+dfsg1-8+b1 [18.6 kB]
Get: 95 http://deb.debian.org/debian unstable/main armhf libsasl2-2 armhf 2.1.28+dfsg1-8+b1 [50.6 kB]
Get: 96 http://deb.debian.org/debian unstable/main armhf libssl-dev armhf 3.4.0-2 [2431 kB]
Get: 97 http://deb.debian.org/debian unstable/main armhf libsasl2-dev armhf 2.1.28+dfsg1-8+b1 [219 kB]
Get: 98 http://deb.debian.org/debian unstable/main armhf golang-github-kr-text-dev all 0.2.0-2 [10.2 kB]
Get: 99 http://deb.debian.org/debian unstable/main armhf golang-golang-x-mod-dev all 0.19.0-1 [104 kB]
Get: 100 http://deb.debian.org/debian unstable/main armhf golang-github-yuin-goldmark-dev all 1.7.4-1 [113 kB]
Get: 101 http://deb.debian.org/debian unstable/main armhf golang-golang-x-tools-dev all 1:0.23.0+ds-1 [1593 kB]
Get: 102 http://deb.debian.org/debian unstable/main armhf golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB]
Get: 103 http://deb.debian.org/debian unstable/main armhf golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB]
Get: 104 http://deb.debian.org/debian unstable/main armhf golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB]
Get: 105 http://deb.debian.org/debian unstable/main armhf golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB]
Get: 106 http://deb.debian.org/debian unstable/main armhf golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB]
Fetched 123 MB in 24s (5136 kB/s)
Preconfiguring packages ...
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 19568 files and directories currently installed.)
Preparing to unpack .../000-sensible-utils_0.0.24_all.deb ...
Unpacking sensible-utils (0.0.24) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3+b1_armhf.deb ...
Unpacking libmagic-mgc (1:5.45-3+b1) ...
Selecting previously unselected package libmagic1t64:armhf.
Preparing to unpack .../002-libmagic1t64_1%3a5.45-3+b1_armhf.deb ...
Unpacking libmagic1t64:armhf (1:5.45-3+b1) ...
Selecting previously unselected package file.
Preparing to unpack .../003-file_1%3a5.45-3+b1_armhf.deb ...
Unpacking file (1:5.45-3+b1) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../004-gettext-base_0.23.1-1_armhf.deb ...
Unpacking gettext-base (0.23.1-1) ...
Selecting previously unselected package libuchardet0:armhf.
Preparing to unpack .../005-libuchardet0_0.0.8-1+b2_armhf.deb ...
Unpacking libuchardet0:armhf (0.0.8-1+b2) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../006-groff-base_1.23.0-7_armhf.deb ...
Unpacking groff-base (1.23.0-7) ...
Selecting previously unselected package bsdextrautils.
Preparing to unpack .../007-bsdextrautils_2.40.4-2_armhf.deb ...
Unpacking bsdextrautils (2.40.4-2) ...
Selecting previously unselected package libpipeline1:armhf.
Preparing to unpack .../008-libpipeline1_1.5.8-1_armhf.deb ...
Unpacking libpipeline1:armhf (1.5.8-1) ...
Selecting previously unselected package man-db.
Preparing to unpack .../009-man-db_2.13.0-1_armhf.deb ...
Unpacking man-db (2.13.0-1) ...
Selecting previously unselected package m4.
Preparing to unpack .../010-m4_1.4.19-5_armhf.deb ...
Unpacking m4 (1.4.19-5) ...
Selecting previously unselected package autoconf.
Preparing to unpack .../011-autoconf_2.72-3_all.deb ...
Unpacking autoconf (2.72-3) ...
Selecting previously unselected package autotools-dev.
Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ...
Unpacking autotools-dev (20220109.1) ...
Selecting previously unselected package automake.
Preparing to unpack .../013-automake_1%3a1.17-2_all.deb ...
Unpacking automake (1:1.17-2) ...
Selecting previously unselected package autopoint.
Preparing to unpack .../014-autopoint_0.23.1-1_all.deb ...
Unpacking autopoint (0.23.1-1) ...
Selecting previously unselected package libdebhelper-perl.
Preparing to unpack .../015-libdebhelper-perl_13.24.1_all.deb ...
Unpacking libdebhelper-perl (13.24.1) ...
Selecting previously unselected package libtool.
Preparing to unpack .../016-libtool_2.5.4-2_all.deb ...
Unpacking libtool (2.5.4-2) ...
Selecting previously unselected package dh-autoreconf.
Preparing to unpack .../017-dh-autoreconf_20_all.deb ...
Unpacking dh-autoreconf (20) ...
Selecting previously unselected package libarchive-zip-perl.
Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ...
Unpacking libarchive-zip-perl (1.68-1) ...
Selecting previously unselected package libfile-stripnondeterminism-perl.
Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.1-1_all.deb ...
Unpacking libfile-stripnondeterminism-perl (1.14.1-1) ...
Selecting previously unselected package dh-strip-nondeterminism.
Preparing to unpack .../020-dh-strip-nondeterminism_1.14.1-1_all.deb ...
Unpacking dh-strip-nondeterminism (1.14.1-1) ...
Selecting previously unselected package libelf1t64:armhf.
Preparing to unpack .../021-libelf1t64_0.192-4_armhf.deb ...
Unpacking libelf1t64:armhf (0.192-4) ...
Selecting previously unselected package dwz.
Preparing to unpack .../022-dwz_0.15-1+b2_armhf.deb ...
Unpacking dwz (0.15-1+b2) ...
Selecting previously unselected package libunistring5:armhf.
Preparing to unpack .../023-libunistring5_1.3-1_armhf.deb ...
Unpacking libunistring5:armhf (1.3-1) ...
Selecting previously unselected package libicu72:armhf.
Preparing to unpack .../024-libicu72_72.1-6_armhf.deb ...
Unpacking libicu72:armhf (72.1-6) ...
Selecting previously unselected package libxml2:armhf.
Preparing to unpack .../025-libxml2_2.12.7+dfsg+really2.9.14-0.2+b1_armhf.deb ...
Unpacking libxml2:armhf (2.12.7+dfsg+really2.9.14-0.2+b1) ...
Selecting previously unselected package gettext.
Preparing to unpack .../026-gettext_0.23.1-1_armhf.deb ...
Unpacking gettext (0.23.1-1) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../027-intltool-debian_0.35.0+20060710.6_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.6) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../028-po-debconf_1.0.21+nmu1_all.deb ...
Unpacking po-debconf (1.0.21+nmu1) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../029-debhelper_13.24.1_all.deb ...
Unpacking debhelper (13.24.1) ...
Selecting previously unselected package dh-golang.
Preparing to unpack .../030-dh-golang_1.62_all.deb ...
Unpacking dh-golang (1.62) ...
Selecting previously unselected package golang-1.23-src.
Preparing to unpack .../031-golang-1.23-src_1.23.5-1_all.deb ...
Unpacking golang-1.23-src (1.23.5-1) ...
Selecting previously unselected package golang-1.23-go.
Preparing to unpack .../032-golang-1.23-go_1.23.5-1_armhf.deb ...
Unpacking golang-1.23-go (1.23.5-1) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../033-golang-src_2%3a1.23~2_all.deb ...
Unpacking golang-src (2:1.23~2) ...
Selecting previously unselected package golang-go:armhf.
Preparing to unpack .../034-golang-go_2%3a1.23~2_armhf.deb ...
Unpacking golang-go:armhf (2:1.23~2) ...
Selecting previously unselected package golang-any:armhf.
Preparing to unpack .../035-golang-any_2%3a1.23~2_armhf.deb ...
Unpacking golang-any:armhf (2:1.23~2) ...
Selecting previously unselected package golang-github-alecthomas-units-dev.
Preparing to unpack .../036-golang-github-alecthomas-units-dev_0.0~git20240626.61d1e34-1_all.deb ...
Unpacking golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ...
Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev.
Preparing to unpack .../037-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ...
Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev.
Preparing to unpack .../038-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ...
Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Selecting previously unselected package golang-golang-x-sys-dev.
Preparing to unpack .../039-golang-golang-x-sys-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-sys-dev (0.22.0-1) ...
Selecting previously unselected package golang-github-mattn-go-isatty-dev.
Preparing to unpack .../040-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ...
Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Selecting previously unselected package golang-github-mattn-go-colorable-dev.
Preparing to unpack .../041-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ...
Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Selecting previously unselected package golang-github-fatih-color-dev.
Preparing to unpack .../042-golang-github-fatih-color-dev_1.17.0-1_all.deb ...
Unpacking golang-github-fatih-color-dev (1.17.0-1) ...
Selecting previously unselected package golang-github-davecgh-go-spew-dev.
Preparing to unpack .../043-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ...
Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Selecting previously unselected package golang-github-pmezard-go-difflib-dev.
Preparing to unpack .../044-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ...
Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Selecting previously unselected package golang-github-stretchr-objx-dev.
Preparing to unpack .../045-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ...
Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ...
Selecting previously unselected package golang-gopkg-yaml.v3-dev.
Preparing to unpack .../046-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ...
Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Selecting previously unselected package golang-github-stretchr-testify-dev.
Preparing to unpack .../047-golang-github-stretchr-testify-dev_1.9.0-1_all.deb ...
Unpacking golang-github-stretchr-testify-dev (1.9.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-hclog-dev.
Preparing to unpack .../048-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ...
Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev.
Preparing to unpack .../049-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ...
Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Selecting previously unselected package golang-github-pkg-errors-dev.
Preparing to unpack .../050-golang-github-pkg-errors-dev_0.9.1-3_all.deb ...
Unpacking golang-github-pkg-errors-dev (0.9.1-3) ...
Selecting previously unselected package golang-github-tv42-httpunix-dev.
Preparing to unpack .../051-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ...
Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev.
Preparing to unpack .../052-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ...
Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Selecting previously unselected package golang-github-datadog-datadog-go-dev.
Preparing to unpack .../053-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ...
Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Selecting previously unselected package golang-github-google-go-cmp-dev.
Preparing to unpack .../054-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ...
Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ...
Selecting previously unselected package golang-google-protobuf-dev.
Preparing to unpack .../055-golang-google-protobuf-dev_1.33.0-1_all.deb ...
Unpacking golang-google-protobuf-dev (1.33.0-1) ...
Selecting previously unselected package golang-github-golang-protobuf-1-5-dev.
Preparing to unpack .../056-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ...
Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Selecting previously unselected package golang-github-hashicorp-go-uuid-dev.
Preparing to unpack .../057-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ...
Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Selecting previously unselected package golang-github-hashicorp-golang-lru-dev.
Preparing to unpack .../058-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ...
Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev.
Preparing to unpack .../059-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ...
Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Selecting previously unselected package golang-github-pascaldekloe-goe-dev.
Preparing to unpack .../060-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ...
Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Selecting previously unselected package golang-github-beorn7-perks-dev.
Preparing to unpack .../061-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ...
Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ...
Selecting previously unselected package golang-github-cespare-xxhash-dev.
Preparing to unpack .../062-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ...
Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ...
Selecting previously unselected package golang-github-modern-go-concurrent-dev.
Preparing to unpack .../063-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ...
Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Selecting previously unselected package golang-github-modern-go-reflect2-dev.
Preparing to unpack .../064-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ...
Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-json-iterator-go-dev.
Preparing to unpack .../065-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ...
Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ...
Selecting previously unselected package golang-github-klauspost-compress-dev.
Preparing to unpack .../066-golang-github-klauspost-compress-dev_1.17.11+ds1-1_all.deb ...
Unpacking golang-github-klauspost-compress-dev (1.17.11+ds1-1) ...
Selecting previously unselected package golang-github-kylelemons-godebug-dev.
Preparing to unpack .../067-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ...
Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Selecting previously unselected package golang-github-prometheus-client-model-dev.
Preparing to unpack .../068-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ...
Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ...
Selecting previously unselected package golang-gopkg-yaml.v2-dev.
Preparing to unpack .../069-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ...
Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Selecting previously unselected package golang-github-jmespath-go-jmespath-dev.
Preparing to unpack .../070-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ...
Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Selecting previously unselected package golang-golang-x-term-dev.
Preparing to unpack .../071-golang-golang-x-term-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-term-dev (0.22.0-1) ...
Selecting previously unselected package golang-golang-x-text-dev.
Preparing to unpack .../072-golang-golang-x-text-dev_0.16.0-1_all.deb ...
Unpacking golang-golang-x-text-dev (0.16.0-1) ...
Selecting previously unselected package golang-golang-x-crypto-dev.
Preparing to unpack .../073-golang-golang-x-crypto-dev_1%3a0.25.0-1_all.deb ...
Unpacking golang-golang-x-crypto-dev (1:0.25.0-1) ...
Selecting previously unselected package golang-golang-x-net-dev.
Preparing to unpack .../074-golang-golang-x-net-dev_1%3a0.27.0-1_all.deb ...
Unpacking golang-golang-x-net-dev (1:0.27.0-1) ...
Selecting previously unselected package golang-github-aws-aws-sdk-go-dev.
Preparing to unpack .../075-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ...
Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Selecting previously unselected package golang-github-go-logfmt-logfmt-dev.
Preparing to unpack .../076-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ...
Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Selecting previously unselected package golang-github-go-kit-log-dev.
Preparing to unpack .../077-golang-github-go-kit-log-dev_0.2.1-1_all.deb ...
Unpacking golang-github-go-kit-log-dev (0.2.1-1) ...
Selecting previously unselected package golang-github-julienschmidt-httprouter-dev.
Preparing to unpack .../078-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ...
Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Selecting previously unselected package golang-github-munnerz-goautoneg-dev.
Preparing to unpack .../079-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ...
Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Selecting previously unselected package golang-github-jpillora-backoff-dev.
Preparing to unpack .../080-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ...
Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev.
Preparing to unpack .../081-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ...
Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Selecting previously unselected package golang-golang-x-oauth2-dev.
Preparing to unpack .../082-golang-golang-x-oauth2-dev_0.15.0-1_all.deb ...
Unpacking golang-golang-x-oauth2-dev (0.15.0-1) ...
Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev.
Preparing to unpack .../083-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ...
Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Selecting previously unselected package golang-github-prometheus-common-dev.
Preparing to unpack .../084-golang-github-prometheus-common-dev_0.61.0-1_all.deb ...
Unpacking golang-github-prometheus-common-dev (0.61.0-1) ...
Selecting previously unselected package golang-golang-x-sync-dev.
Preparing to unpack .../085-golang-golang-x-sync-dev_0.7.0-1_all.deb ...
Unpacking golang-golang-x-sync-dev (0.7.0-1) ...
Selecting previously unselected package golang-github-prometheus-procfs-dev.
Preparing to unpack .../086-golang-github-prometheus-procfs-dev_0.14.0-1_all.deb ...
Unpacking golang-github-prometheus-procfs-dev (0.14.0-1) ...
Selecting previously unselected package golang-github-prometheus-client-golang-dev.
Preparing to unpack .../087-golang-github-prometheus-client-golang-dev_1.20.5-1_all.deb ...
Unpacking golang-github-prometheus-client-golang-dev (1.20.5-1) ...
Selecting previously unselected package golang-github-armon-go-metrics-dev.
Preparing to unpack .../088-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ...
Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ...
Selecting previously unselected package golang-github-creack-pty-dev.
Preparing to unpack .../089-golang-github-creack-pty-dev_1.1.21-1_all.deb ...
Unpacking golang-github-creack-pty-dev (1.1.21-1) ...
Selecting previously unselected package golang-github-vmihailenco-tagparser-dev.
Preparing to unpack .../090-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ...
Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev.
Preparing to unpack .../091-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ...
Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Selecting previously unselected package golang-gopkg-tomb.v2-dev.
Preparing to unpack .../092-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ...
Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Selecting previously unselected package libsasl2-modules-db:armhf.
Preparing to unpack .../093-libsasl2-modules-db_2.1.28+dfsg1-8+b1_armhf.deb ...
Unpacking libsasl2-modules-db:armhf (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package libsasl2-2:armhf.
Preparing to unpack .../094-libsasl2-2_2.1.28+dfsg1-8+b1_armhf.deb ...
Unpacking libsasl2-2:armhf (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package libssl-dev:armhf.
Preparing to unpack .../095-libssl-dev_3.4.0-2_armhf.deb ...
Unpacking libssl-dev:armhf (3.4.0-2) ...
Selecting previously unselected package libsasl2-dev.
Preparing to unpack .../096-libsasl2-dev_2.1.28+dfsg1-8+b1_armhf.deb ...
Unpacking libsasl2-dev (2.1.28+dfsg1-8+b1) ...
Selecting previously unselected package golang-github-kr-text-dev.
Preparing to unpack .../097-golang-github-kr-text-dev_0.2.0-2_all.deb ...
Unpacking golang-github-kr-text-dev (0.2.0-2) ...
Selecting previously unselected package golang-golang-x-mod-dev.
Preparing to unpack .../098-golang-golang-x-mod-dev_0.19.0-1_all.deb ...
Unpacking golang-golang-x-mod-dev (0.19.0-1) ...
Selecting previously unselected package golang-github-yuin-goldmark-dev.
Preparing to unpack .../099-golang-github-yuin-goldmark-dev_1.7.4-1_all.deb ...
Unpacking golang-github-yuin-goldmark-dev (1.7.4-1) ...
Selecting previously unselected package golang-golang-x-tools-dev.
Preparing to unpack .../100-golang-golang-x-tools-dev_1%3a0.23.0+ds-1_all.deb ...
Unpacking golang-golang-x-tools-dev (1:0.23.0+ds-1) ...
Selecting previously unselected package golang-github-rogpeppe-go-internal-dev.
Preparing to unpack .../101-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ...
Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Selecting previously unselected package golang-github-kr-pretty-dev.
Preparing to unpack .../102-golang-github-kr-pretty-dev_0.3.1-1_all.deb ...
Unpacking golang-github-kr-pretty-dev (0.3.1-1) ...
Selecting previously unselected package golang-gopkg-check.v1-dev.
Preparing to unpack .../103-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ...
Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Selecting previously unselected package golang-gopkg-mgo.v2-dev.
Preparing to unpack .../104-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ...
Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev.
Preparing to unpack .../105-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ...
Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up libpipeline1:armhf (1.5.8-1) ...
Setting up golang-github-google-go-cmp-dev (0.6.0-1) ...
Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Setting up libicu72:armhf (72.1-6) ...
Setting up bsdextrautils (2.40.4-2) ...
Setting up golang-github-creack-pty-dev (1.1.21-1) ...
Setting up golang-github-stretchr-objx-dev (0.5.2-1) ...
Setting up libmagic-mgc (1:5.45-3+b1) ...
Setting up golang-github-pkg-errors-dev (0.9.1-3) ...
Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Setting up libarchive-zip-perl (1.68-1) ...
Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Setting up libdebhelper-perl (13.24.1) ...
Setting up golang-golang-x-sys-dev (0.22.0-1) ...
Setting up libmagic1t64:armhf (1:5.45-3+b1) ...
Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Setting up gettext-base (0.23.1-1) ...
Setting up m4 (1.4.19-5) ...
Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Setting up golang-golang-x-term-dev (0.22.0-1) ...
Setting up file (1:5.45-3+b1) ...
Setting up golang-github-beorn7-perks-dev (1.0.1-1) ...
Setting up libelf1t64:armhf (0.192-4) ...
Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Setting up libsasl2-modules-db:armhf (2.1.28+dfsg1-8+b1) ...
Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ...
Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Setting up autotools-dev (20220109.1) ...
Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Setting up golang-google-protobuf-dev (1.33.0-1) ...
Setting up golang-github-kr-text-dev (0.2.0-2) ...
Setting up golang-github-stretchr-testify-dev (1.9.0-1) ...
Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Setting up golang-github-yuin-goldmark-dev (1.7.4-1) ...
Setting up libunistring5:armhf (1.3-1) ...
Setting up libssl-dev:armhf (3.4.0-2) ...
Setting up golang-golang-x-sync-dev (0.7.0-1) ...
Setting up golang-golang-x-mod-dev (0.19.0-1) ...
Setting up golang-1.23-src (1.23.5-1) ...
Setting up autopoint (0.23.1-1) ...
Setting up golang-github-klauspost-compress-dev (1.17.11+ds1-1) ...
Setting up libsasl2-2:armhf (2.1.28+dfsg1-8+b1) ...
Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Setting up autoconf (2.72-3) ...
Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Setting up dwz (0.15-1+b2) ...
Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Setting up sensible-utils (0.0.24) ...
Setting up golang-golang-x-text-dev (0.16.0-1) ...
Setting up libuchardet0:armhf (0.0.8-1+b2) ...
Setting up golang-github-prometheus-procfs-dev (0.14.0-1) ...
Setting up golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ...
Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Setting up golang-golang-x-crypto-dev (1:0.25.0-1) ...
Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ...
Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Setting up golang-golang-x-oauth2-dev (0.15.0-1) ...
Setting up golang-src (2:1.23~2) ...
Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Setting up libxml2:armhf (2.12.7+dfsg+really2.9.14-0.2+b1) ...
Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Setting up automake (1:1.17-2) ...
update-alternatives: using /usr/bin/automake-1.17 to provide /usr/bin/automake (automake) in auto mode
Setting up libfile-stripnondeterminism-perl (1.14.1-1) ...
Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Setting up golang-github-go-kit-log-dev (0.2.1-1) ...
Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Setting up gettext (0.23.1-1) ...
Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Setting up libtool (2.5.4-2) ...
Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Setting up golang-github-json-iterator-go-dev (1.1.12-2) ...
Setting up intltool-debian (0.35.0+20060710.6) ...
Setting up dh-autoreconf (20) ...
Setting up golang-1.23-go (1.23.5-1) ...
Setting up libsasl2-dev (2.1.28+dfsg1-8+b1) ...
Setting up golang-golang-x-net-dev (1:0.27.0-1) ...
Setting up dh-strip-nondeterminism (1.14.1-1) ...
Setting up groff-base (1.23.0-7) ...
Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Setting up golang-github-fatih-color-dev (1.17.0-1) ...
Setting up golang-go:armhf (2:1.23~2) ...
Setting up po-debconf (1.0.21+nmu1) ...
Setting up golang-golang-x-tools-dev (1:0.23.0+ds-1) ...
Setting up man-db (2.13.0-1) ...
Not building database; man-db/auto-update is not 'true'.
Setting up golang-any:armhf (2:1.23~2) ...
Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Setting up golang-github-kr-pretty-dev (0.3.1-1) ...
Setting up debhelper (13.24.1) ...
Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Setting up dh-golang (1.62) ...
Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Setting up golang-github-prometheus-common-dev (0.61.0-1) ...
Setting up golang-github-prometheus-client-golang-dev (1.20.5-1) ...
Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ...
Processing triggers for libc-bin (2.40-6) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Building tag database...
 -> Finished parsing the build-deps
I: Building the package
I: user script /srv/workspace/pbuilder/31365/tmp/hooks/A99_set_merged_usr starting
Not re-configuring usrmerge for unstable
I: user script /srv/workspace/pbuilder/31365/tmp/hooks/A99_set_merged_usr finished
hostname: Name or service not known
I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/i/capture/the/path" HOME="/nonexistent/second-build" dpkg-genchanges -S  > ../golang-github-hashicorp-raft_1.5.0-1_source.changes
dpkg-buildpackage: info: source package golang-github-hashicorp-raft
dpkg-buildpackage: info: source version 1.5.0-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Daniel Swarbrick <dswarbrick@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture armhf
 debian/rules clean
dh clean --buildsystem=golang
   dh_auto_clean -O--buildsystem=golang
   dh_autoreconf_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules binary
dh binary --buildsystem=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-arm-linux-gnueabihf && go install -trimpath -v -p 4 github.com/hashicorp/raft github.com/hashicorp/raft/bench
internal/unsafeheader
internal/byteorder
internal/goarch
internal/cpu
internal/coverage/rtcov
internal/abi
internal/chacha8rand
internal/godebugs
internal/goexperiment
internal/bytealg
internal/goos
internal/profilerecord
internal/runtime/atomic
internal/runtime/syscall
internal/stringslite
runtime/internal/math
runtime/internal/sys
internal/race
sync/atomic
unicode
internal/runtime/exithook
unicode/utf8
container/list
internal/asan
internal/itoa
internal/msan
math/bits
crypto/internal/alias
crypto/internal/boring/sig
cmp
math
encoding
unicode/utf16
log/internal
internal/nettrace
runtime
vendor/golang.org/x/crypto/cryptobyte/asn1
vendor/golang.org/x/crypto/internal/alias
iter
crypto/subtle
internal/reflectlite
sync
slices
internal/weak
maps
internal/bisect
internal/testlog
internal/singleflight
runtime/cgo
errors
sort
io
internal/godebug
internal/oserror
bytes
strings
syscall
hash
strconv
bufio
crypto/cipher
crypto
crypto/internal/randutil
reflect
crypto/internal/boring
path
time
internal/syscall/unix
internal/syscall/execenv
math/rand
github.com/hashicorp/golang-lru/simplelru
github.com/hashicorp/go-immutable-radix
vendor/golang.org/x/net/dns/dnsmessage
math/rand/v2
context
io/fs
internal/poll
internal/filepathlite
internal/concurrent
hash/crc32
unique
crypto/aes
net/netip
crypto/des
os
crypto/internal/edwards25519/field
crypto/internal/nistec/fiat
internal/fmtsort
encoding/binary
crypto/sha512
fmt
encoding/base64
net
crypto/internal/edwards25519
crypto/hmac
vendor/golang.org/x/crypto/chacha20
vendor/golang.org/x/crypto/internal/poly1305
math/big
encoding/json
log
compress/flate
crypto/internal/nistec
compress/gzip
vendor/golang.org/x/crypto/chacha20poly1305
crypto/rand
crypto/internal/bigmod
crypto/internal/boring/bbig
encoding/asn1
crypto/ecdh
crypto/elliptic
crypto/ed25519
vendor/golang.org/x/crypto/hkdf
crypto/internal/hpke
vendor/golang.org/x/crypto/cryptobyte
vendor/golang.org/x/sys/cpu
crypto/md5
vendor/golang.org/x/crypto/sha3
crypto/rc4
crypto/rsa
crypto/ecdsa
crypto/internal/mlkem768
crypto/sha1
crypto/sha256
crypto/dsa
encoding/hex
encoding/pem
net/url
crypto/x509/pkix
path/filepath
vendor/golang.org/x/text/transform
vendor/golang.org/x/text/unicode/bidi
vendor/golang.org/x/text/unicode/norm
vendor/golang.org/x/net/http2/hpack
mime
vendor/golang.org/x/text/secure/bidirule
mime/quotedprintable
net/http/internal
net/http/internal/ascii
os/signal
golang.org/x/sys/unix
vendor/golang.org/x/net/idna
io/ioutil
regexp/syntax
internal/saferio
encoding/gob
regexp
crypto/x509
net/textproto
vendor/golang.org/x/net/http/httpproxy
vendor/golang.org/x/net/http/httpguts
mime/multipart
github.com/mattn/go-isatty
github.com/mattn/go-colorable
github.com/fatih/color
github.com/hashicorp/go-hclog
go/token
html
text/template/parse
hash/crc64
flag
crypto/tls
internal/sysinfo
runtime/debug
runtime/trace
testing
text/template
html/template
net/http/httptrace
net/http
net/rpc
github.com/armon/go-metrics
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/raft
github.com/hashicorp/raft/bench
   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
dh_auto_test
	cd obj-arm-linux-gnueabihf && go test -vet=off -v -p 4 github.com/hashicorp/raft github.com/hashicorp/raft/bench
?   	github.com/hashicorp/raft/bench	[no test files]
=== RUN   TestCommitment_setVoters
--- PASS: TestCommitment_setVoters (0.00s)
=== RUN   TestCommitment_match_max
--- PASS: TestCommitment_match_max (0.00s)
=== RUN   TestCommitment_match_nonVoting
--- PASS: TestCommitment_match_nonVoting (0.00s)
=== RUN   TestCommitment_recalculate
--- PASS: TestCommitment_recalculate (0.00s)
=== RUN   TestCommitment_recalculate_startIndex
--- PASS: TestCommitment_recalculate_startIndex (0.00s)
=== RUN   TestCommitment_noVoterSanity
--- PASS: TestCommitment_noVoterSanity (0.00s)
=== RUN   TestCommitment_singleVoter
--- PASS: TestCommitment_singleVoter (0.00s)
=== RUN   TestConfiguration_Configuration_Clone
--- PASS: TestConfiguration_Configuration_Clone (0.00s)
=== RUN   TestConfiguration_configurations_Clone
--- PASS: TestConfiguration_configurations_Clone (0.00s)
=== RUN   TestConfiguration_hasVote
--- PASS: TestConfiguration_hasVote (0.00s)
=== RUN   TestConfiguration_checkConfiguration
--- PASS: TestConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_nextConfiguration_table
--- PASS: TestConfiguration_nextConfiguration_table (0.00s)
=== RUN   TestConfiguration_nextConfiguration_prevIndex
--- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
=== RUN   TestConfiguration_nextConfiguration_checkConfiguration
--- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_encodeDecodePeers
--- PASS: TestConfiguration_encodeDecodePeers (0.00s)
=== RUN   TestConfiguration_encodeDecodeConfiguration
--- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
=== RUN   TestDiscardSnapshotStoreImpl
--- PASS: TestDiscardSnapshotStoreImpl (0.00s)
=== RUN   TestDiscardSnapshotSinkImpl
--- PASS: TestDiscardSnapshotSinkImpl (0.00s)
=== RUN   TestFileSnapshotStoreImpl
--- PASS: TestFileSnapshotStoreImpl (0.00s)
=== RUN   TestFileSnapshotSinkImpl
--- PASS: TestFileSnapshotSinkImpl (0.00s)
=== RUN   TestFileSS_CreateSnapshotMissingParentDir
2025-01-28T14:53:34.332Z [INFO]  creating new snapshot: path=/tmp/raft1018138551/raft3254672211/snapshots/3-10-1738076014332.tmp
--- PASS: TestFileSS_CreateSnapshotMissingParentDir (23.05s)
=== RUN   TestFileSS_CreateSnapshot
2025-01-28T14:53:57.378Z [INFO]  creating new snapshot: path=/tmp/raft1768655196/snapshots/3-10-1738076037378.tmp
2025-01-28T14:53:57.465Z [WARN]  found temporary snapshot: name=3-10-1738076037378.tmp
--- PASS: TestFileSS_CreateSnapshot (0.13s)
=== RUN   TestFileSS_CancelSnapshot
2025-01-28T14:53:57.505Z [INFO]  creating new snapshot: path=/tmp/raft220298489/snapshots/3-10-1738076037505.tmp
--- PASS: TestFileSS_CancelSnapshot (0.02s)
=== RUN   TestFileSS_Retention
2025-01-28T14:53:57.529Z [INFO]  creating new snapshot: path=/tmp/raft3110082571/snapshots/3-10-1738076037529.tmp
2025-01-28T14:53:57.584Z [INFO]  creating new snapshot: path=/tmp/raft3110082571/snapshots/3-11-1738076037584.tmp
2025-01-28T14:53:57.612Z [INFO]  creating new snapshot: path=/tmp/raft3110082571/snapshots/3-12-1738076037612.tmp
2025-01-28T14:53:57.651Z [INFO]  reaping snapshot: path=/tmp/raft3110082571/snapshots/3-10-1738076037529
2025-01-28T14:53:57.652Z [INFO]  creating new snapshot: path=/tmp/raft3110082571/snapshots/3-13-1738076037652.tmp
2025-01-28T14:53:59.683Z [INFO]  reaping snapshot: path=/tmp/raft3110082571/snapshots/3-11-1738076037584
2025-01-28T14:53:59.684Z [INFO]  creating new snapshot: path=/tmp/raft3110082571/snapshots/3-14-1738076039684.tmp
2025-01-28T14:53:59.703Z [INFO]  reaping snapshot: path=/tmp/raft3110082571/snapshots/3-12-1738076037612
--- PASS: TestFileSS_Retention (2.18s)
=== RUN   TestFileSS_BadPerm
--- PASS: TestFileSS_BadPerm (0.00s)
=== RUN   TestFileSS_MissingParentDir
--- PASS: TestFileSS_MissingParentDir (0.00s)
=== RUN   TestFileSS_Ordering
2025-01-28T14:53:59.709Z [INFO]  creating new snapshot: path=/tmp/raft787323634/snapshots/5-130350-1738076039709.tmp
2025-01-28T14:53:59.732Z [INFO]  creating new snapshot: path=/tmp/raft787323634/snapshots/36-204917-1738076039731.tmp
--- PASS: TestFileSS_Ordering (0.04s)
=== RUN   TestDeferFutureSuccess
--- PASS: TestDeferFutureSuccess (0.00s)
=== RUN   TestDeferFutureError
--- PASS: TestDeferFutureError (0.00s)
=== RUN   TestDeferFutureConcurrent
--- PASS: TestDeferFutureConcurrent (0.00s)
=== RUN   TestInmemSnapshotStoreImpl
--- PASS: TestInmemSnapshotStoreImpl (0.00s)
=== RUN   TestInmemSnapshotSinkImpl
--- PASS: TestInmemSnapshotSinkImpl (0.00s)
=== RUN   TestInmemSS_CreateSnapshot
--- PASS: TestInmemSS_CreateSnapshot (0.00s)
=== RUN   TestInmemSS_OpenSnapshotTwice
--- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
=== RUN   TestInmemTransportImpl
--- PASS: TestInmemTransportImpl (0.00s)
=== RUN   TestInmemTransportWriteTimeout
--- PASS: TestInmemTransportWriteTimeout (0.01s)
=== RUN   TestRaft_Integ
--- SKIP: TestRaft_Integ (0.00s)
=== RUN   TestRaft_RestartFollower_LongInitialHeartbeat
--- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s)
=== RUN   TestLogCache
--- PASS: TestLogCache (0.00s)
=== RUN   TestLogCacheWithBackendStoreError
--- PASS: TestLogCacheWithBackendStoreError (0.00s)
=== RUN   TestOldestLog
=== RUN   TestOldestLog/empty_logs
=== RUN   TestOldestLog/simple_case
--- PASS: TestOldestLog (0.00s)
    --- PASS: TestOldestLog/empty_logs (0.00s)
    --- PASS: TestOldestLog/simple_case (0.00s)
=== RUN   TestEmitsLogStoreMetrics
--- PASS: TestEmitsLogStoreMetrics (0.01s)
=== RUN   TestNetworkTransport_CloseStreams
--- PASS: TestNetworkTransport_CloseStreams (0.01s)
=== RUN   TestNetworkTransport_StartStop
--- PASS: TestNetworkTransport_StartStop (0.00s)
=== RUN   TestNetworkTransport_Heartbeat_FastPath
--- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s)
=== RUN   TestNetworkTransport_AppendEntries
--- PASS: TestNetworkTransport_AppendEntries (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline
--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.01s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_CloseStreams
--- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.03s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0
2025-01-28T14:53:59.879Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2
2025-01-28T14:53:59.932Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10
2025-01-28T14:53:59.986Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s)
=== RUN   TestNetworkTransport_RequestVote
--- PASS: TestNetworkTransport_RequestVote (0.00s)
=== RUN   TestNetworkTransport_InstallSnapshot
--- PASS: TestNetworkTransport_InstallSnapshot (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode
--- PASS: TestNetworkTransport_EncodeDecode (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode_AddressProvider
--- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
=== RUN   TestNetworkTransport_PooledConn
--- PASS: TestNetworkTransport_PooledConn (0.00s)
=== RUN   TestNetworkTransport_ListenBackoff
    net_transport_test.go:839: countingWriter: 2025-01-28T14:54:00.000Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-28T14:54:00.005Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-28T14:54:00.016Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-28T14:54:00.036Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-28T14:54:00.077Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-28T14:54:00.158Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-28T14:54:00.319Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-28T14:54:00.640Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-28T14:54:01.281Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-28T14:54:02.284Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2025-01-28T14:54:03.287Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
--- PASS: TestNetworkTransport_ListenBackoff (4.00s)
=== RUN   TestPeersJSON_BadConfiguration
--- PASS: TestPeersJSON_BadConfiguration (0.00s)
=== RUN   TestPeersJSON_ReadPeersJSON
--- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
=== RUN   TestPeersJSON_ReadConfigJSON
--- PASS: TestPeersJSON_ReadConfigJSON (0.00s)
=== RUN   TestRaft_StartStop
2025-01-28T14:54:04.006Z [INFO]  server-1a660542-3c61-e666-67c9-c3101f2eedb6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1a660542-3c61-e666-67c9-c3101f2eedb6 Address:1a660542-3c61-e666-67c9-c3101f2eedb6}]"
--- PASS: TestRaft_StartStop (0.00s)
=== RUN   TestRaft_AfterShutdown
2025-01-28T14:54:04.008Z [INFO]  server-dfb916f7-f85a-0916-eaa0-325d42f3738f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dfb916f7-f85a-0916-eaa0-325d42f3738f Address:dfb916f7-f85a-0916-eaa0-325d42f3738f}]"
--- PASS: TestRaft_AfterShutdown (0.00s)
=== RUN   TestRaft_LiveBootstrap
2025-01-28T14:54:04.011Z [INFO]  server-ab0ff5c1-30b6-d579-ae24-70044e9c067e: initial configuration: index=0 servers=[]
2025-01-28T14:54:04.011Z [INFO]  server-ba9b9407-a8bd-3c6d-65f8-4754fac01422: initial configuration: index=0 servers=[]
2025-01-28T14:54:04.011Z [INFO]  server-ab0ff5c1-30b6-d579-ae24-70044e9c067e: entering follower state: follower="Node at ab0ff5c1-30b6-d579-ae24-70044e9c067e [Follower]" leader-address= leader-id=
2025-01-28T14:54:04.011Z [INFO]  server-ba9b9407-a8bd-3c6d-65f8-4754fac01422: entering follower state: follower="Node at ba9b9407-a8bd-3c6d-65f8-4754fac01422 [Follower]" leader-address= leader-id=
2025-01-28T14:54:04.011Z [INFO]  server-3b972842-c6f4-61c6-b7ab-7b09a3b1c352: initial configuration: index=0 servers=[]
2025-01-28T14:54:04.012Z [INFO]  server-3b972842-c6f4-61c6-b7ab-7b09a3b1c352: entering follower state: follower="Node at 3b972842-c6f4-61c6-b7ab-7b09a3b1c352 [Follower]" leader-address= leader-id=
2025-01-28T14:54:04.012Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:04.068Z [WARN]  server-3b972842-c6f4-61c6-b7ab-7b09a3b1c352: no known peers, aborting election
2025-01-28T14:54:04.083Z [WARN]  server-ab0ff5c1-30b6-d579-ae24-70044e9c067e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:04.083Z [INFO]  server-ab0ff5c1-30b6-d579-ae24-70044e9c067e: entering candidate state: node="Node at ab0ff5c1-30b6-d579-ae24-70044e9c067e [Candidate]" term=2
2025-01-28T14:54:04.083Z [INFO]  server-ab0ff5c1-30b6-d579-ae24-70044e9c067e: election won: term=2 tally=2
2025-01-28T14:54:04.083Z [INFO]  server-ab0ff5c1-30b6-d579-ae24-70044e9c067e: entering leader state: leader="Node at ab0ff5c1-30b6-d579-ae24-70044e9c067e [Leader]"
2025-01-28T14:54:04.083Z [INFO]  server-ab0ff5c1-30b6-d579-ae24-70044e9c067e: added peer, starting replication: peer=server-ba9b9407-a8bd-3c6d-65f8-4754fac01422
2025-01-28T14:54:04.083Z [INFO]  server-ab0ff5c1-30b6-d579-ae24-70044e9c067e: added peer, starting replication: peer=server-3b972842-c6f4-61c6-b7ab-7b09a3b1c352
2025-01-28T14:54:04.084Z [WARN]  server-3b972842-c6f4-61c6-b7ab-7b09a3b1c352: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-01-28T14:54:04.084Z [WARN]  server-ba9b9407-a8bd-3c6d-65f8-4754fac01422: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2025-01-28T14:54:04.084Z [WARN]  server-ab0ff5c1-30b6-d579-ae24-70044e9c067e: appendEntries rejected, sending older logs: peer="{Voter server-3b972842-c6f4-61c6-b7ab-7b09a3b1c352 3b972842-c6f4-61c6-b7ab-7b09a3b1c352}" next=1
2025-01-28T14:54:04.084Z [WARN]  server-ab0ff5c1-30b6-d579-ae24-70044e9c067e: appendEntries rejected, sending older logs: peer="{Voter server-ba9b9407-a8bd-3c6d-65f8-4754fac01422 ba9b9407-a8bd-3c6d-65f8-4754fac01422}" next=1
2025-01-28T14:54:04.084Z [INFO]  server-ab0ff5c1-30b6-d579-ae24-70044e9c067e: pipelining replication: peer="{Voter server-3b972842-c6f4-61c6-b7ab-7b09a3b1c352 3b972842-c6f4-61c6-b7ab-7b09a3b1c352}"
2025-01-28T14:54:04.084Z [INFO]  server-ab0ff5c1-30b6-d579-ae24-70044e9c067e: pipelining replication: peer="{Voter server-ba9b9407-a8bd-3c6d-65f8-4754fac01422 ba9b9407-a8bd-3c6d-65f8-4754fac01422}"
2025-01-28T14:54:04.189Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:04.08387253 +0000 UTC m=+29.758362530 (2 nodes), 71.416523ms from start of poll, 72.365033ms from cluster start. Timeout at 2025-01-28 14:54:04.188881609 +0000 UTC m=+29.863371589, 105.009059ms after stability
2025-01-28T14:54:04.189Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:04.295Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:04.189903007 +0000 UTC m=+29.864392967 (1 nodes), 3.3µs from start of poll, 178.39547ms from cluster start. Timeout at 2025-01-28 14:54:04.294917286 +0000 UTC m=+29.969407286, 105.014319ms after stability
2025-01-28T14:54:04.306Z [INFO]  server-ab0ff5c1-30b6-d579-ae24-70044e9c067e: aborting pipeline replication: peer="{Voter server-3b972842-c6f4-61c6-b7ab-7b09a3b1c352 3b972842-c6f4-61c6-b7ab-7b09a3b1c352}"
2025-01-28T14:54:04.306Z [INFO]  server-ab0ff5c1-30b6-d579-ae24-70044e9c067e: aborting pipeline replication: peer="{Voter server-ba9b9407-a8bd-3c6d-65f8-4754fac01422 ba9b9407-a8bd-3c6d-65f8-4754fac01422}"
--- PASS: TestRaft_LiveBootstrap (0.30s)
=== RUN   TestRaft_LiveBootstrap_From_NonVoter
2025-01-28T14:54:04.310Z [INFO]  server-825aaf67-c1b2-23e0-0f42-c87d2160602f: initial configuration: index=0 servers=[]
2025-01-28T14:54:04.310Z [INFO]  server-a2c4c3c5-b8ee-1f10-36a5-a6e5f29bcfb7: initial configuration: index=0 servers=[]
2025-01-28T14:54:04.310Z [INFO]  server-825aaf67-c1b2-23e0-0f42-c87d2160602f: entering follower state: follower="Node at 825aaf67-c1b2-23e0-0f42-c87d2160602f [Follower]" leader-address= leader-id=
2025-01-28T14:54:04.310Z [INFO]  server-a2c4c3c5-b8ee-1f10-36a5-a6e5f29bcfb7: entering follower state: follower="Node at a2c4c3c5-b8ee-1f10-36a5-a6e5f29bcfb7 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s)
=== RUN   TestRaft_RecoverCluster_NoState
2025-01-28T14:54:04.312Z [INFO]  server-cf112137-089b-ebab-fcab-6a5d19f879f3: initial configuration: index=0 servers=[]
--- PASS: TestRaft_RecoverCluster_NoState (0.00s)
=== RUN   TestRaft_RecoverCluster
=== RUN   TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs
2025-01-28T14:54:04.314Z [INFO]  server-139f096b-28b6-854a-86b1-b8c9c56fe6d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-139f096b-28b6-854a-86b1-b8c9c56fe6d0 Address:139f096b-28b6-854a-86b1-b8c9c56fe6d0} {Suffrage:Voter ID:server-1321883d-4cf3-7286-8743-a5a9a9aef940 Address:1321883d-4cf3-7286-8743-a5a9a9aef940} {Suffrage:Voter ID:server-415f37d2-c85f-1819-e4e8-bb29df644aba Address:415f37d2-c85f-1819-e4e8-bb29df644aba}]"
2025-01-28T14:54:04.315Z [INFO]  server-139f096b-28b6-854a-86b1-b8c9c56fe6d0: entering follower state: follower="Node at 139f096b-28b6-854a-86b1-b8c9c56fe6d0 [Follower]" leader-address= leader-id=
2025-01-28T14:54:04.315Z [INFO]  server-1321883d-4cf3-7286-8743-a5a9a9aef940: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-139f096b-28b6-854a-86b1-b8c9c56fe6d0 Address:139f096b-28b6-854a-86b1-b8c9c56fe6d0} {Suffrage:Voter ID:server-1321883d-4cf3-7286-8743-a5a9a9aef940 Address:1321883d-4cf3-7286-8743-a5a9a9aef940} {Suffrage:Voter ID:server-415f37d2-c85f-1819-e4e8-bb29df644aba Address:415f37d2-c85f-1819-e4e8-bb29df644aba}]"
2025-01-28T14:54:04.315Z [INFO]  server-1321883d-4cf3-7286-8743-a5a9a9aef940: entering follower state: follower="Node at 1321883d-4cf3-7286-8743-a5a9a9aef940 [Follower]" leader-address= leader-id=
2025-01-28T14:54:04.315Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-139f096b-28b6-854a-86b1-b8c9c56fe6d0 Address:139f096b-28b6-854a-86b1-b8c9c56fe6d0} {Suffrage:Voter ID:server-1321883d-4cf3-7286-8743-a5a9a9aef940 Address:1321883d-4cf3-7286-8743-a5a9a9aef940} {Suffrage:Voter ID:server-415f37d2-c85f-1819-e4e8-bb29df644aba Address:415f37d2-c85f-1819-e4e8-bb29df644aba}]"
2025-01-28T14:54:04.315Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:04.315Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: entering follower state: follower="Node at 415f37d2-c85f-1819-e4e8-bb29df644aba [Follower]" leader-address= leader-id=
2025-01-28T14:54:04.391Z [WARN]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:04.391Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: entering candidate state: node="Node at 415f37d2-c85f-1819-e4e8-bb29df644aba [Candidate]" term=2
2025-01-28T14:54:04.392Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: election won: term=2 tally=2
2025-01-28T14:54:04.392Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: entering leader state: leader="Node at 415f37d2-c85f-1819-e4e8-bb29df644aba [Leader]"
2025-01-28T14:54:04.392Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: added peer, starting replication: peer=server-139f096b-28b6-854a-86b1-b8c9c56fe6d0
2025-01-28T14:54:04.392Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: added peer, starting replication: peer=server-1321883d-4cf3-7286-8743-a5a9a9aef940
2025-01-28T14:54:04.392Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: pipelining replication: peer="{Voter server-139f096b-28b6-854a-86b1-b8c9c56fe6d0 139f096b-28b6-854a-86b1-b8c9c56fe6d0}"
2025-01-28T14:54:04.392Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: pipelining replication: peer="{Voter server-1321883d-4cf3-7286-8743-a5a9a9aef940 1321883d-4cf3-7286-8743-a5a9a9aef940}"
2025-01-28T14:54:04.497Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:04.392304953 +0000 UTC m=+30.066794933 (1 nodes), 76.561318ms from start of poll, 77.562023ms from cluster start. Timeout at 2025-01-28 14:54:04.497418062 +0000 UTC m=+30.171908142, 105.113209ms after stability
2025-01-28T14:54:04.498Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: aborting pipeline replication: peer="{Voter server-1321883d-4cf3-7286-8743-a5a9a9aef940 1321883d-4cf3-7286-8743-a5a9a9aef940}"
2025-01-28T14:54:04.498Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: aborting pipeline replication: peer="{Voter server-139f096b-28b6-854a-86b1-b8c9c56fe6d0 139f096b-28b6-854a-86b1-b8c9c56fe6d0}"
2025-01-28T14:54:04.498Z [INFO]  creating new snapshot: path=/tmp/raft2030302248/snapshots/2-2-1738076044498.tmp
2025-01-28T14:54:04.501Z [INFO]  server-139f096b-28b6-854a-86b1-b8c9c56fe6d0: starting restore from snapshot: id=2-2-1738076044498 last-index=2 last-term=2 size-in-bytes=1
2025-01-28T14:54:04.501Z [INFO]  server-139f096b-28b6-854a-86b1-b8c9c56fe6d0: snapshot restore progress: id=2-2-1738076044498 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-01-28T14:54:04.501Z [INFO]  server-139f096b-28b6-854a-86b1-b8c9c56fe6d0: restored from snapshot: id=2-2-1738076044498 last-index=2 last-term=2 size-in-bytes=1
2025-01-28T14:54:04.501Z [INFO]  server-139f096b-28b6-854a-86b1-b8c9c56fe6d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-139f096b-28b6-854a-86b1-b8c9c56fe6d0 Address:139f096b-28b6-854a-86b1-b8c9c56fe6d0} {Suffrage:Voter ID:server-1321883d-4cf3-7286-8743-a5a9a9aef940 Address:1321883d-4cf3-7286-8743-a5a9a9aef940} {Suffrage:Voter ID:server-415f37d2-c85f-1819-e4e8-bb29df644aba Address:415f37d2-c85f-1819-e4e8-bb29df644aba}]"
2025-01-28T14:54:04.501Z [INFO]  server-139f096b-28b6-854a-86b1-b8c9c56fe6d0: entering follower state: follower="Node at 139f096b-28b6-854a-86b1-b8c9c56fe6d0 [Follower]" leader-address= leader-id=
2025-01-28T14:54:04.501Z [INFO]  creating new snapshot: path=/tmp/raft462386264/snapshots/2-2-1738076044501.tmp
2025-01-28T14:54:04.504Z [INFO]  server-1321883d-4cf3-7286-8743-a5a9a9aef940: starting restore from snapshot: id=2-2-1738076044501 last-index=2 last-term=2 size-in-bytes=1
2025-01-28T14:54:04.504Z [INFO]  server-1321883d-4cf3-7286-8743-a5a9a9aef940: snapshot restore progress: id=2-2-1738076044501 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-01-28T14:54:04.504Z [INFO]  server-1321883d-4cf3-7286-8743-a5a9a9aef940: restored from snapshot: id=2-2-1738076044501 last-index=2 last-term=2 size-in-bytes=1
2025-01-28T14:54:04.504Z [INFO]  server-1321883d-4cf3-7286-8743-a5a9a9aef940: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-139f096b-28b6-854a-86b1-b8c9c56fe6d0 Address:139f096b-28b6-854a-86b1-b8c9c56fe6d0} {Suffrage:Voter ID:server-1321883d-4cf3-7286-8743-a5a9a9aef940 Address:1321883d-4cf3-7286-8743-a5a9a9aef940} {Suffrage:Voter ID:server-415f37d2-c85f-1819-e4e8-bb29df644aba Address:415f37d2-c85f-1819-e4e8-bb29df644aba}]"
2025-01-28T14:54:04.504Z [INFO]  server-1321883d-4cf3-7286-8743-a5a9a9aef940: entering follower state: follower="Node at 1321883d-4cf3-7286-8743-a5a9a9aef940 [Follower]" leader-address= leader-id=
2025-01-28T14:54:04.505Z [INFO]  creating new snapshot: path=/tmp/raft1841362757/snapshots/2-2-1738076044505.tmp
2025-01-28T14:54:04.506Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: starting restore from snapshot: id=2-2-1738076044505 last-index=2 last-term=2 size-in-bytes=1
2025-01-28T14:54:04.507Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: snapshot restore progress: id=2-2-1738076044505 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2025-01-28T14:54:04.507Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: restored from snapshot: id=2-2-1738076044505 last-index=2 last-term=2 size-in-bytes=1
2025-01-28T14:54:04.507Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-139f096b-28b6-854a-86b1-b8c9c56fe6d0 Address:139f096b-28b6-854a-86b1-b8c9c56fe6d0} {Suffrage:Voter ID:server-1321883d-4cf3-7286-8743-a5a9a9aef940 Address:1321883d-4cf3-7286-8743-a5a9a9aef940} {Suffrage:Voter ID:server-415f37d2-c85f-1819-e4e8-bb29df644aba Address:415f37d2-c85f-1819-e4e8-bb29df644aba}]"
2025-01-28T14:54:04.507Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: entering follower state: follower="Node at 415f37d2-c85f-1819-e4e8-bb29df644aba [Follower]" leader-address= leader-id=
2025-01-28T14:54:04.560Z [WARN]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:04.560Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: entering candidate state: node="Node at 415f37d2-c85f-1819-e4e8-bb29df644aba [Candidate]" term=3
2025-01-28T14:54:04.560Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: election won: term=3 tally=2
2025-01-28T14:54:04.560Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: entering leader state: leader="Node at 415f37d2-c85f-1819-e4e8-bb29df644aba [Leader]"
2025-01-28T14:54:04.560Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: added peer, starting replication: peer=server-139f096b-28b6-854a-86b1-b8c9c56fe6d0
2025-01-28T14:54:04.560Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: added peer, starting replication: peer=server-1321883d-4cf3-7286-8743-a5a9a9aef940
2025-01-28T14:54:04.561Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: pipelining replication: peer="{Voter server-1321883d-4cf3-7286-8743-a5a9a9aef940 1321883d-4cf3-7286-8743-a5a9a9aef940}"
2025-01-28T14:54:04.561Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: pipelining replication: peer="{Voter server-139f096b-28b6-854a-86b1-b8c9c56fe6d0 139f096b-28b6-854a-86b1-b8c9c56fe6d0}"
2025-01-28T14:54:04.822Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:04.929Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:04.82311999 +0000 UTC m=+30.497609970 (1 nodes), 77.433µs from start of poll, 508.37706ms from cluster start. Timeout at 2025-01-28 14:54:04.928151047 +0000 UTC m=+30.602641027, 105.031057ms after stability
2025-01-28T14:54:04.929Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: aborting pipeline replication: peer="{Voter server-1321883d-4cf3-7286-8743-a5a9a9aef940 1321883d-4cf3-7286-8743-a5a9a9aef940}"
2025-01-28T14:54:04.929Z [INFO]  server-415f37d2-c85f-1819-e4e8-bb29df644aba: aborting pipeline replication: peer="{Voter server-139f096b-28b6-854a-86b1-b8c9c56fe6d0 139f096b-28b6-854a-86b1-b8c9c56fe6d0}"
2025-01-28T14:54:05.430Z [ERROR] server-415f37d2-c85f-1819-e4e8-bb29df644aba: failed to heartbeat to: peer=1321883d-4cf3-7286-8743-a5a9a9aef940 backoff time=10ms error="command timed out"
=== RUN   TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs
2025-01-28T14:54:05.436Z [INFO]  server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2 Address:0cb1976e-18c2-6d6f-cfe2-0864af19caa2} {Suffrage:Voter ID:server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9 Address:b9b1cf77-fae2-b389-93bb-9f6313fcaff9} {Suffrage:Voter ID:server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd Address:ecaea33d-0b8c-e99b-c96d-e9c72cd518dd}]"
2025-01-28T14:54:05.436Z [INFO]  server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2: entering follower state: follower="Node at 0cb1976e-18c2-6d6f-cfe2-0864af19caa2 [Follower]" leader-address= leader-id=
2025-01-28T14:54:05.436Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2 Address:0cb1976e-18c2-6d6f-cfe2-0864af19caa2} {Suffrage:Voter ID:server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9 Address:b9b1cf77-fae2-b389-93bb-9f6313fcaff9} {Suffrage:Voter ID:server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd Address:ecaea33d-0b8c-e99b-c96d-e9c72cd518dd}]"
2025-01-28T14:54:05.436Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: entering follower state: follower="Node at b9b1cf77-fae2-b389-93bb-9f6313fcaff9 [Follower]" leader-address= leader-id=
2025-01-28T14:54:05.436Z [INFO]  server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2 Address:0cb1976e-18c2-6d6f-cfe2-0864af19caa2} {Suffrage:Voter ID:server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9 Address:b9b1cf77-fae2-b389-93bb-9f6313fcaff9} {Suffrage:Voter ID:server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd Address:ecaea33d-0b8c-e99b-c96d-e9c72cd518dd}]"
2025-01-28T14:54:05.436Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:05.436Z [INFO]  server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd: entering follower state: follower="Node at ecaea33d-0b8c-e99b-c96d-e9c72cd518dd [Follower]" leader-address= leader-id=
2025-01-28T14:54:05.490Z [WARN]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:05.490Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: entering candidate state: node="Node at b9b1cf77-fae2-b389-93bb-9f6313fcaff9 [Candidate]" term=2
2025-01-28T14:54:05.490Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: election won: term=2 tally=2
2025-01-28T14:54:05.490Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: entering leader state: leader="Node at b9b1cf77-fae2-b389-93bb-9f6313fcaff9 [Leader]"
2025-01-28T14:54:05.490Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: added peer, starting replication: peer=server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2
2025-01-28T14:54:05.490Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: added peer, starting replication: peer=server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd
2025-01-28T14:54:05.491Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: pipelining replication: peer="{Voter server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd ecaea33d-0b8c-e99b-c96d-e9c72cd518dd}"
2025-01-28T14:54:05.491Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: pipelining replication: peer="{Voter server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2 0cb1976e-18c2-6d6f-cfe2-0864af19caa2}"
2025-01-28T14:54:05.596Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:05.490978704 +0000 UTC m=+31.165468664 (1 nodes), 54.000925ms from start of poll, 55.093362ms from cluster start. Timeout at 2025-01-28 14:54:05.595984143 +0000 UTC m=+31.270474183, 105.005519ms after stability
2025-01-28T14:54:05.598Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: aborting pipeline replication: peer="{Voter server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd ecaea33d-0b8c-e99b-c96d-e9c72cd518dd}"
2025-01-28T14:54:05.598Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: aborting pipeline replication: peer="{Voter server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2 0cb1976e-18c2-6d6f-cfe2-0864af19caa2}"
2025-01-28T14:54:05.598Z [INFO]  creating new snapshot: path=/tmp/raft1183262515/snapshots/2-6-1738076045598.tmp
2025-01-28T14:54:05.600Z [INFO]  server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2: starting restore from snapshot: id=2-6-1738076045598 last-index=6 last-term=2 size-in-bytes=25
2025-01-28T14:54:05.601Z [INFO]  server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2: snapshot restore progress: id=2-6-1738076045598 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-01-28T14:54:05.601Z [INFO]  server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2: restored from snapshot: id=2-6-1738076045598 last-index=6 last-term=2 size-in-bytes=25
2025-01-28T14:54:05.601Z [INFO]  server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2 Address:0cb1976e-18c2-6d6f-cfe2-0864af19caa2} {Suffrage:Voter ID:server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9 Address:b9b1cf77-fae2-b389-93bb-9f6313fcaff9} {Suffrage:Voter ID:server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd Address:ecaea33d-0b8c-e99b-c96d-e9c72cd518dd}]"
2025-01-28T14:54:05.601Z [INFO]  server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2: entering follower state: follower="Node at 0cb1976e-18c2-6d6f-cfe2-0864af19caa2 [Follower]" leader-address= leader-id=
2025-01-28T14:54:05.601Z [INFO]  creating new snapshot: path=/tmp/raft675510915/snapshots/2-6-1738076045601.tmp
2025-01-28T14:54:05.603Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: starting restore from snapshot: id=2-6-1738076045601 last-index=6 last-term=2 size-in-bytes=25
2025-01-28T14:54:05.603Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: snapshot restore progress: id=2-6-1738076045601 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-01-28T14:54:05.604Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: restored from snapshot: id=2-6-1738076045601 last-index=6 last-term=2 size-in-bytes=25
2025-01-28T14:54:05.604Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2 Address:0cb1976e-18c2-6d6f-cfe2-0864af19caa2} {Suffrage:Voter ID:server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9 Address:b9b1cf77-fae2-b389-93bb-9f6313fcaff9} {Suffrage:Voter ID:server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd Address:ecaea33d-0b8c-e99b-c96d-e9c72cd518dd}]"
2025-01-28T14:54:05.604Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: entering follower state: follower="Node at b9b1cf77-fae2-b389-93bb-9f6313fcaff9 [Follower]" leader-address= leader-id=
2025-01-28T14:54:05.604Z [INFO]  creating new snapshot: path=/tmp/raft3113940077/snapshots/2-6-1738076045604.tmp
2025-01-28T14:54:05.605Z [INFO]  server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd: starting restore from snapshot: id=2-6-1738076045604 last-index=6 last-term=2 size-in-bytes=25
2025-01-28T14:54:05.606Z [INFO]  server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd: snapshot restore progress: id=2-6-1738076045604 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2025-01-28T14:54:05.606Z [INFO]  server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd: restored from snapshot: id=2-6-1738076045604 last-index=6 last-term=2 size-in-bytes=25
2025-01-28T14:54:05.606Z [INFO]  server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2 Address:0cb1976e-18c2-6d6f-cfe2-0864af19caa2} {Suffrage:Voter ID:server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9 Address:b9b1cf77-fae2-b389-93bb-9f6313fcaff9} {Suffrage:Voter ID:server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd Address:ecaea33d-0b8c-e99b-c96d-e9c72cd518dd}]"
2025-01-28T14:54:05.606Z [INFO]  server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd: entering follower state: follower="Node at ecaea33d-0b8c-e99b-c96d-e9c72cd518dd [Follower]" leader-address= leader-id=
2025-01-28T14:54:05.685Z [WARN]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:05.685Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: entering candidate state: node="Node at b9b1cf77-fae2-b389-93bb-9f6313fcaff9 [Candidate]" term=3
2025-01-28T14:54:05.685Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: election won: term=3 tally=2
2025-01-28T14:54:05.685Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: entering leader state: leader="Node at b9b1cf77-fae2-b389-93bb-9f6313fcaff9 [Leader]"
2025-01-28T14:54:05.685Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: added peer, starting replication: peer=server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2
2025-01-28T14:54:05.685Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: added peer, starting replication: peer=server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd
2025-01-28T14:54:05.686Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: pipelining replication: peer="{Voter server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd ecaea33d-0b8c-e99b-c96d-e9c72cd518dd}"
2025-01-28T14:54:05.686Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: pipelining replication: peer="{Voter server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2 0cb1976e-18c2-6d6f-cfe2-0864af19caa2}"
2025-01-28T14:54:05.922Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:06.030Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:05.925042195 +0000 UTC m=+31.599532175 (1 nodes), 96.051µs from start of poll, 489.156873ms from cluster start. Timeout at 2025-01-28 14:54:06.030066333 +0000 UTC m=+31.704556333, 105.024158ms after stability
2025-01-28T14:54:06.030Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: aborting pipeline replication: peer="{Voter server-ecaea33d-0b8c-e99b-c96d-e9c72cd518dd ecaea33d-0b8c-e99b-c96d-e9c72cd518dd}"
2025-01-28T14:54:06.030Z [INFO]  server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: aborting pipeline replication: peer="{Voter server-0cb1976e-18c2-6d6f-cfe2-0864af19caa2 0cb1976e-18c2-6d6f-cfe2-0864af19caa2}"
2025-01-28T14:54:06.531Z [ERROR] server-b9b1cf77-fae2-b389-93bb-9f6313fcaff9: failed to heartbeat to: peer=ecaea33d-0b8c-e99b-c96d-e9c72cd518dd backoff time=10ms error="command timed out"
=== RUN   TestRaft_RecoverCluster/snapshot,_with_trailing_logs
2025-01-28T14:54:06.537Z [INFO]  server-31607b25-9e8c-51af-26f5-7ccfc34cad4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31607b25-9e8c-51af-26f5-7ccfc34cad4a Address:31607b25-9e8c-51af-26f5-7ccfc34cad4a} {Suffrage:Voter ID:server-893a3b65-5cc9-bccf-a8fa-92de7626fc78 Address:893a3b65-5cc9-bccf-a8fa-92de7626fc78} {Suffrage:Voter ID:server-b22db518-672f-70eb-3290-9cba5bc1cf78 Address:b22db518-672f-70eb-3290-9cba5bc1cf78}]"
2025-01-28T14:54:06.537Z [INFO]  server-31607b25-9e8c-51af-26f5-7ccfc34cad4a: entering follower state: follower="Node at 31607b25-9e8c-51af-26f5-7ccfc34cad4a [Follower]" leader-address= leader-id=
2025-01-28T14:54:06.537Z [INFO]  server-893a3b65-5cc9-bccf-a8fa-92de7626fc78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31607b25-9e8c-51af-26f5-7ccfc34cad4a Address:31607b25-9e8c-51af-26f5-7ccfc34cad4a} {Suffrage:Voter ID:server-893a3b65-5cc9-bccf-a8fa-92de7626fc78 Address:893a3b65-5cc9-bccf-a8fa-92de7626fc78} {Suffrage:Voter ID:server-b22db518-672f-70eb-3290-9cba5bc1cf78 Address:b22db518-672f-70eb-3290-9cba5bc1cf78}]"
2025-01-28T14:54:06.537Z [INFO]  server-893a3b65-5cc9-bccf-a8fa-92de7626fc78: entering follower state: follower="Node at 893a3b65-5cc9-bccf-a8fa-92de7626fc78 [Follower]" leader-address= leader-id=
2025-01-28T14:54:06.538Z [INFO]  server-b22db518-672f-70eb-3290-9cba5bc1cf78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31607b25-9e8c-51af-26f5-7ccfc34cad4a Address:31607b25-9e8c-51af-26f5-7ccfc34cad4a} {Suffrage:Voter ID:server-893a3b65-5cc9-bccf-a8fa-92de7626fc78 Address:893a3b65-5cc9-bccf-a8fa-92de7626fc78} {Suffrage:Voter ID:server-b22db518-672f-70eb-3290-9cba5bc1cf78 Address:b22db518-672f-70eb-3290-9cba5bc1cf78}]"
2025-01-28T14:54:06.538Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:06.538Z [INFO]  server-b22db518-672f-70eb-3290-9cba5bc1cf78: entering follower state: follower="Node at b22db518-672f-70eb-3290-9cba5bc1cf78 [Follower]" leader-address= leader-id=
2025-01-28T14:54:06.597Z [WARN]  server-31607b25-9e8c-51af-26f5-7ccfc34cad4a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:06.597Z [INFO]  server-31607b25-9e8c-51af-26f5-7ccfc34cad4a: entering candidate state: node="Node at 31607b25-9e8c-51af-26f5-7ccfc34cad4a [Candidate]" term=2
2025-01-28T14:54:06.597Z [INFO]  server-31607b25-9e8c-51af-26f5-7ccfc34cad4a: election won: term=2 tally=2
2025-01-28T14:54:06.597Z [INFO]  server-31607b25-9e8c-51af-26f5-7ccfc34cad4a: entering leader state: leader="Node at 31607b25-9e8c-51af-26f5-7ccfc34cad4a [Leader]"
2025-01-28T14:54:06.597Z [INFO]  server-31607b25-9e8c-51af-26f5-7ccfc34cad4a: added peer, starting replication: peer=server-893a3b65-5cc9-bccf-a8fa-92de7626fc78
2025-01-28T14:54:06.597Z [INFO]  server-31607b25-9e8c-51af-26f5-7ccfc34cad4a: added peer, starting replication: peer=server-b22db518-672f-70eb-3290-9cba5bc1cf78
2025-01-28T14:54:06.597Z [INFO]  server-31607b25-9e8c-51af-26f5-7ccfc34cad4a: pipelining replication: peer="{Voter server-893a3b65-5cc9-bccf-a8fa-92de7626fc78 893a3b65-5cc9-bccf-a8fa-92de7626fc78}"
2025-01-28T14:54:06.597Z [INFO]  server-31607b25-9e8c-51af-26f5-7ccfc34cad4a: pipelining replication: peer="{Voter server-b22db518-672f-70eb-3290-9cba5bc1cf78 b22db518-672f-70eb-3290-9cba5bc1cf78}"
2025-01-28T14:54:06.702Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:06.597510734 +0000 UTC m=+32.272000734 (1 nodes), 59.235211ms from start of poll, 60.31077ms from cluster start. Timeout at 2025-01-28 14:54:06.702522573 +0000 UTC m=+32.377012553, 105.011819ms after stability
2025-01-28T14:54:06.710Z [INFO]  server-31607b25-9e8c-51af-26f5-7ccfc34cad4a: aborting pipeline replication: peer="{Voter server-b22db518-672f-70eb-3290-9cba5bc1cf78 b22db518-672f-70eb-3290-9cba5bc1cf78}"
2025-01-28T14:54:06.710Z [INFO]  server-31607b25-9e8c-51af-26f5-7ccfc34cad4a: aborting pipeline replication: peer="{Voter server-893a3b65-5cc9-bccf-a8fa-92de7626fc78 893a3b65-5cc9-bccf-a8fa-92de7626fc78}"
2025-01-28T14:54:06.710Z [INFO]  creating new snapshot: path=/tmp/raft3604028932/snapshots/2-27-1738076046710.tmp
2025-01-28T14:54:06.713Z [INFO]  server-31607b25-9e8c-51af-26f5-7ccfc34cad4a: starting restore from snapshot: id=2-27-1738076046710 last-index=27 last-term=2 size-in-bytes=168
2025-01-28T14:54:06.713Z [INFO]  server-31607b25-9e8c-51af-26f5-7ccfc34cad4a: snapshot restore progress: id=2-27-1738076046710 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-01-28T14:54:06.713Z [INFO]  server-31607b25-9e8c-51af-26f5-7ccfc34cad4a: restored from snapshot: id=2-27-1738076046710 last-index=27 last-term=2 size-in-bytes=168
2025-01-28T14:54:06.713Z [INFO]  server-31607b25-9e8c-51af-26f5-7ccfc34cad4a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31607b25-9e8c-51af-26f5-7ccfc34cad4a Address:31607b25-9e8c-51af-26f5-7ccfc34cad4a} {Suffrage:Voter ID:server-893a3b65-5cc9-bccf-a8fa-92de7626fc78 Address:893a3b65-5cc9-bccf-a8fa-92de7626fc78} {Suffrage:Voter ID:server-b22db518-672f-70eb-3290-9cba5bc1cf78 Address:b22db518-672f-70eb-3290-9cba5bc1cf78}]"
2025-01-28T14:54:06.713Z [INFO]  server-31607b25-9e8c-51af-26f5-7ccfc34cad4a: entering follower state: follower="Node at 31607b25-9e8c-51af-26f5-7ccfc34cad4a [Follower]" leader-address= leader-id=
2025-01-28T14:54:06.713Z [INFO]  creating new snapshot: path=/tmp/raft3703374330/snapshots/2-27-1738076046713.tmp
2025-01-28T14:54:06.716Z [INFO]  server-893a3b65-5cc9-bccf-a8fa-92de7626fc78: starting restore from snapshot: id=2-27-1738076046713 last-index=27 last-term=2 size-in-bytes=168
2025-01-28T14:54:06.716Z [INFO]  server-893a3b65-5cc9-bccf-a8fa-92de7626fc78: snapshot restore progress: id=2-27-1738076046713 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-01-28T14:54:06.716Z [INFO]  server-893a3b65-5cc9-bccf-a8fa-92de7626fc78: restored from snapshot: id=2-27-1738076046713 last-index=27 last-term=2 size-in-bytes=168
2025-01-28T14:54:06.716Z [INFO]  server-893a3b65-5cc9-bccf-a8fa-92de7626fc78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31607b25-9e8c-51af-26f5-7ccfc34cad4a Address:31607b25-9e8c-51af-26f5-7ccfc34cad4a} {Suffrage:Voter ID:server-893a3b65-5cc9-bccf-a8fa-92de7626fc78 Address:893a3b65-5cc9-bccf-a8fa-92de7626fc78} {Suffrage:Voter ID:server-b22db518-672f-70eb-3290-9cba5bc1cf78 Address:b22db518-672f-70eb-3290-9cba5bc1cf78}]"
2025-01-28T14:54:06.716Z [INFO]  server-893a3b65-5cc9-bccf-a8fa-92de7626fc78: entering follower state: follower="Node at 893a3b65-5cc9-bccf-a8fa-92de7626fc78 [Follower]" leader-address= leader-id=
2025-01-28T14:54:06.716Z [INFO]  creating new snapshot: path=/tmp/raft1485038946/snapshots/2-27-1738076046716.tmp
2025-01-28T14:54:06.718Z [INFO]  server-b22db518-672f-70eb-3290-9cba5bc1cf78: starting restore from snapshot: id=2-27-1738076046716 last-index=27 last-term=2 size-in-bytes=168
2025-01-28T14:54:06.719Z [INFO]  server-b22db518-672f-70eb-3290-9cba5bc1cf78: snapshot restore progress: id=2-27-1738076046716 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2025-01-28T14:54:06.719Z [INFO]  server-b22db518-672f-70eb-3290-9cba5bc1cf78: restored from snapshot: id=2-27-1738076046716 last-index=27 last-term=2 size-in-bytes=168
2025-01-28T14:54:06.719Z [INFO]  server-b22db518-672f-70eb-3290-9cba5bc1cf78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31607b25-9e8c-51af-26f5-7ccfc34cad4a Address:31607b25-9e8c-51af-26f5-7ccfc34cad4a} {Suffrage:Voter ID:server-893a3b65-5cc9-bccf-a8fa-92de7626fc78 Address:893a3b65-5cc9-bccf-a8fa-92de7626fc78} {Suffrage:Voter ID:server-b22db518-672f-70eb-3290-9cba5bc1cf78 Address:b22db518-672f-70eb-3290-9cba5bc1cf78}]"
2025-01-28T14:54:06.719Z [INFO]  server-b22db518-672f-70eb-3290-9cba5bc1cf78: entering follower state: follower="Node at b22db518-672f-70eb-3290-9cba5bc1cf78 [Follower]" leader-address= leader-id=
2025-01-28T14:54:06.772Z [WARN]  server-b22db518-672f-70eb-3290-9cba5bc1cf78: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:06.772Z [INFO]  server-b22db518-672f-70eb-3290-9cba5bc1cf78: entering candidate state: node="Node at b22db518-672f-70eb-3290-9cba5bc1cf78 [Candidate]" term=3
2025-01-28T14:54:06.772Z [INFO]  server-b22db518-672f-70eb-3290-9cba5bc1cf78: election won: term=3 tally=2
2025-01-28T14:54:06.772Z [INFO]  server-b22db518-672f-70eb-3290-9cba5bc1cf78: entering leader state: leader="Node at b22db518-672f-70eb-3290-9cba5bc1cf78 [Leader]"
2025-01-28T14:54:06.772Z [INFO]  server-b22db518-672f-70eb-3290-9cba5bc1cf78: added peer, starting replication: peer=server-31607b25-9e8c-51af-26f5-7ccfc34cad4a
2025-01-28T14:54:06.772Z [INFO]  server-b22db518-672f-70eb-3290-9cba5bc1cf78: added peer, starting replication: peer=server-893a3b65-5cc9-bccf-a8fa-92de7626fc78
2025-01-28T14:54:06.773Z [INFO]  server-b22db518-672f-70eb-3290-9cba5bc1cf78: pipelining replication: peer="{Voter server-31607b25-9e8c-51af-26f5-7ccfc34cad4a 31607b25-9e8c-51af-26f5-7ccfc34cad4a}"
2025-01-28T14:54:06.773Z [INFO]  server-b22db518-672f-70eb-3290-9cba5bc1cf78: pipelining replication: peer="{Voter server-893a3b65-5cc9-bccf-a8fa-92de7626fc78 893a3b65-5cc9-bccf-a8fa-92de7626fc78}"
2025-01-28T14:54:07.035Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:07.141Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:07.035633082 +0000 UTC m=+32.710123042 (1 nodes), 66.854µs from start of poll, 498.433078ms from cluster start. Timeout at 2025-01-28 14:54:07.14065896 +0000 UTC m=+32.815148980, 105.025938ms after stability
2025-01-28T14:54:07.141Z [INFO]  server-b22db518-672f-70eb-3290-9cba5bc1cf78: aborting pipeline replication: peer="{Voter server-893a3b65-5cc9-bccf-a8fa-92de7626fc78 893a3b65-5cc9-bccf-a8fa-92de7626fc78}"
2025-01-28T14:54:07.141Z [INFO]  server-b22db518-672f-70eb-3290-9cba5bc1cf78: aborting pipeline replication: peer="{Voter server-31607b25-9e8c-51af-26f5-7ccfc34cad4a 31607b25-9e8c-51af-26f5-7ccfc34cad4a}"
--- PASS: TestRaft_RecoverCluster (2.83s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (1.12s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (1.10s)
    --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (0.61s)
=== RUN   TestRaft_HasExistingState
    raft_test.go:258: Skipping unreliable test
--- SKIP: TestRaft_HasExistingState (0.00s)
=== RUN   TestRaft_SingleNode
2025-01-28T14:54:07.145Z [INFO]  server-2c2f2f1c-87df-4c24-4c68-16a980495fc5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2c2f2f1c-87df-4c24-4c68-16a980495fc5 Address:2c2f2f1c-87df-4c24-4c68-16a980495fc5}]"
2025-01-28T14:54:07.145Z [INFO]  server-2c2f2f1c-87df-4c24-4c68-16a980495fc5: entering follower state: follower="Node at 2c2f2f1c-87df-4c24-4c68-16a980495fc5 [Follower]" leader-address= leader-id=
2025-01-28T14:54:07.235Z [WARN]  server-2c2f2f1c-87df-4c24-4c68-16a980495fc5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:07.235Z [INFO]  server-2c2f2f1c-87df-4c24-4c68-16a980495fc5: entering candidate state: node="Node at 2c2f2f1c-87df-4c24-4c68-16a980495fc5 [Candidate]" term=2
2025-01-28T14:54:07.235Z [INFO]  server-2c2f2f1c-87df-4c24-4c68-16a980495fc5: election won: term=2 tally=1
2025-01-28T14:54:07.235Z [INFO]  server-2c2f2f1c-87df-4c24-4c68-16a980495fc5: entering leader state: leader="Node at 2c2f2f1c-87df-4c24-4c68-16a980495fc5 [Leader]"
--- PASS: TestRaft_SingleNode (0.09s)
=== RUN   TestRaft_TripleNode
2025-01-28T14:54:07.239Z [INFO]  server-4176ae00-01fe-9053-9737-d187d7c52821: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4176ae00-01fe-9053-9737-d187d7c52821 Address:4176ae00-01fe-9053-9737-d187d7c52821} {Suffrage:Voter ID:server-78e757d8-8d82-847e-a62a-2541af981efc Address:78e757d8-8d82-847e-a62a-2541af981efc} {Suffrage:Voter ID:server-19964c54-a948-f5f4-f109-0505c07c6eff Address:19964c54-a948-f5f4-f109-0505c07c6eff}]"
2025-01-28T14:54:07.239Z [INFO]  server-4176ae00-01fe-9053-9737-d187d7c52821: entering follower state: follower="Node at 4176ae00-01fe-9053-9737-d187d7c52821 [Follower]" leader-address= leader-id=
2025-01-28T14:54:07.239Z [INFO]  server-78e757d8-8d82-847e-a62a-2541af981efc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4176ae00-01fe-9053-9737-d187d7c52821 Address:4176ae00-01fe-9053-9737-d187d7c52821} {Suffrage:Voter ID:server-78e757d8-8d82-847e-a62a-2541af981efc Address:78e757d8-8d82-847e-a62a-2541af981efc} {Suffrage:Voter ID:server-19964c54-a948-f5f4-f109-0505c07c6eff Address:19964c54-a948-f5f4-f109-0505c07c6eff}]"
2025-01-28T14:54:07.239Z [INFO]  server-78e757d8-8d82-847e-a62a-2541af981efc: entering follower state: follower="Node at 78e757d8-8d82-847e-a62a-2541af981efc [Follower]" leader-address= leader-id=
2025-01-28T14:54:07.239Z [INFO]  server-19964c54-a948-f5f4-f109-0505c07c6eff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4176ae00-01fe-9053-9737-d187d7c52821 Address:4176ae00-01fe-9053-9737-d187d7c52821} {Suffrage:Voter ID:server-78e757d8-8d82-847e-a62a-2541af981efc Address:78e757d8-8d82-847e-a62a-2541af981efc} {Suffrage:Voter ID:server-19964c54-a948-f5f4-f109-0505c07c6eff Address:19964c54-a948-f5f4-f109-0505c07c6eff}]"
2025-01-28T14:54:07.239Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:07.239Z [INFO]  server-19964c54-a948-f5f4-f109-0505c07c6eff: entering follower state: follower="Node at 19964c54-a948-f5f4-f109-0505c07c6eff [Follower]" leader-address= leader-id=
2025-01-28T14:54:07.318Z [WARN]  server-19964c54-a948-f5f4-f109-0505c07c6eff: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:07.318Z [INFO]  server-19964c54-a948-f5f4-f109-0505c07c6eff: entering candidate state: node="Node at 19964c54-a948-f5f4-f109-0505c07c6eff [Candidate]" term=2
2025-01-28T14:54:07.318Z [INFO]  server-19964c54-a948-f5f4-f109-0505c07c6eff: election won: term=2 tally=2
2025-01-28T14:54:07.318Z [INFO]  server-19964c54-a948-f5f4-f109-0505c07c6eff: entering leader state: leader="Node at 19964c54-a948-f5f4-f109-0505c07c6eff [Leader]"
2025-01-28T14:54:07.318Z [INFO]  server-19964c54-a948-f5f4-f109-0505c07c6eff: added peer, starting replication: peer=server-4176ae00-01fe-9053-9737-d187d7c52821
2025-01-28T14:54:07.318Z [INFO]  server-19964c54-a948-f5f4-f109-0505c07c6eff: added peer, starting replication: peer=server-78e757d8-8d82-847e-a62a-2541af981efc
2025-01-28T14:54:07.319Z [INFO]  server-19964c54-a948-f5f4-f109-0505c07c6eff: pipelining replication: peer="{Voter server-4176ae00-01fe-9053-9737-d187d7c52821 4176ae00-01fe-9053-9737-d187d7c52821}"
2025-01-28T14:54:07.319Z [INFO]  server-19964c54-a948-f5f4-f109-0505c07c6eff: pipelining replication: peer="{Voter server-78e757d8-8d82-847e-a62a-2541af981efc 78e757d8-8d82-847e-a62a-2541af981efc}"
2025-01-28T14:54:07.424Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:07.318928156 +0000 UTC m=+32.993418136 (2 nodes), 78.998167ms from start of poll, 80.176136ms from cluster start. Timeout at 2025-01-28 14:54:07.423934635 +0000 UTC m=+33.098424695, 105.006559ms after stability
2025-01-28T14:54:07.424Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:07.529Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:07.424766651 +0000 UTC m=+33.099256631 (1 nodes), 1.999µs from start of poll, 186.014631ms from cluster start. Timeout at 2025-01-28 14:54:07.529795448 +0000 UTC m=+33.204285508, 105.028877ms after stability
2025-01-28T14:54:07.541Z [INFO]  server-19964c54-a948-f5f4-f109-0505c07c6eff: aborting pipeline replication: peer="{Voter server-78e757d8-8d82-847e-a62a-2541af981efc 78e757d8-8d82-847e-a62a-2541af981efc}"
2025-01-28T14:54:07.541Z [INFO]  server-19964c54-a948-f5f4-f109-0505c07c6eff: aborting pipeline replication: peer="{Voter server-4176ae00-01fe-9053-9737-d187d7c52821 4176ae00-01fe-9053-9737-d187d7c52821}"
--- PASS: TestRaft_TripleNode (0.31s)
=== RUN   TestRaft_LeaderFail
2025-01-28T14:54:07.545Z [INFO]  server-95200575-5c4b-73ff-91a6-38f0a636df08: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95200575-5c4b-73ff-91a6-38f0a636df08 Address:95200575-5c4b-73ff-91a6-38f0a636df08} {Suffrage:Voter ID:server-ca8f3652-28d9-7f14-6bd4-c7e032de850c Address:ca8f3652-28d9-7f14-6bd4-c7e032de850c} {Suffrage:Voter ID:server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 Address:c85ee916-e3a7-2f3d-e0a7-266d2eb04f01}]"
2025-01-28T14:54:07.545Z [INFO]  server-95200575-5c4b-73ff-91a6-38f0a636df08: entering follower state: follower="Node at 95200575-5c4b-73ff-91a6-38f0a636df08 [Follower]" leader-address= leader-id=
2025-01-28T14:54:07.545Z [INFO]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95200575-5c4b-73ff-91a6-38f0a636df08 Address:95200575-5c4b-73ff-91a6-38f0a636df08} {Suffrage:Voter ID:server-ca8f3652-28d9-7f14-6bd4-c7e032de850c Address:ca8f3652-28d9-7f14-6bd4-c7e032de850c} {Suffrage:Voter ID:server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 Address:c85ee916-e3a7-2f3d-e0a7-266d2eb04f01}]"
2025-01-28T14:54:07.546Z [INFO]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: entering follower state: follower="Node at ca8f3652-28d9-7f14-6bd4-c7e032de850c [Follower]" leader-address= leader-id=
2025-01-28T14:54:07.546Z [INFO]  server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95200575-5c4b-73ff-91a6-38f0a636df08 Address:95200575-5c4b-73ff-91a6-38f0a636df08} {Suffrage:Voter ID:server-ca8f3652-28d9-7f14-6bd4-c7e032de850c Address:ca8f3652-28d9-7f14-6bd4-c7e032de850c} {Suffrage:Voter ID:server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 Address:c85ee916-e3a7-2f3d-e0a7-266d2eb04f01}]"
2025-01-28T14:54:07.546Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:07.546Z [INFO]  server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: entering follower state: follower="Node at c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 [Follower]" leader-address= leader-id=
2025-01-28T14:54:07.600Z [WARN]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:07.600Z [INFO]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: entering candidate state: node="Node at ca8f3652-28d9-7f14-6bd4-c7e032de850c [Candidate]" term=2
2025-01-28T14:54:07.600Z [INFO]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: election won: term=2 tally=2
2025-01-28T14:54:07.600Z [INFO]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: entering leader state: leader="Node at ca8f3652-28d9-7f14-6bd4-c7e032de850c [Leader]"
2025-01-28T14:54:07.600Z [INFO]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: added peer, starting replication: peer=server-95200575-5c4b-73ff-91a6-38f0a636df08
2025-01-28T14:54:07.600Z [INFO]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: added peer, starting replication: peer=server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01
2025-01-28T14:54:07.601Z [INFO]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: pipelining replication: peer="{Voter server-95200575-5c4b-73ff-91a6-38f0a636df08 95200575-5c4b-73ff-91a6-38f0a636df08}"
2025-01-28T14:54:07.601Z [INFO]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: pipelining replication: peer="{Voter server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 c85ee916-e3a7-2f3d-e0a7-266d2eb04f01}"
2025-01-28T14:54:07.706Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:07.601015604 +0000 UTC m=+33.275505584 (2 nodes), 54.614208ms from start of poll, 55.696505ms from cluster start. Timeout at 2025-01-28 14:54:07.706026423 +0000 UTC m=+33.380516403, 105.010819ms after stability
2025-01-28T14:54:07.706Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:07.812Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:07.706403822 +0000 UTC m=+33.380893802 (1 nodes), 2.1µs from start of poll, 161.084723ms from cluster start. Timeout at 2025-01-28 14:54:07.81142778 +0000 UTC m=+33.485917820, 105.024018ms after stability
    raft_test.go:378: [INFO] Disconnecting Node at ca8f3652-28d9-7f14-6bd4-c7e032de850c [Leader]
2025-01-28T14:54:07.828Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to pipeline appendEntries: peer="{Voter server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 c85ee916-e3a7-2f3d-e0a7-266d2eb04f01}" error="append pipeline closed"
2025-01-28T14:54:07.828Z [INFO]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: aborting pipeline replication: peer="{Voter server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 c85ee916-e3a7-2f3d-e0a7-266d2eb04f01}"
2025-01-28T14:54:07.829Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:07.830Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to pipeline appendEntries: peer="{Voter server-95200575-5c4b-73ff-91a6-38f0a636df08 95200575-5c4b-73ff-91a6-38f0a636df08}" error="append pipeline closed"
2025-01-28T14:54:07.830Z [INFO]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: aborting pipeline replication: peer="{Voter server-95200575-5c4b-73ff-91a6-38f0a636df08 95200575-5c4b-73ff-91a6-38f0a636df08}"
2025-01-28T14:54:07.830Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to heartbeat to: peer=95200575-5c4b-73ff-91a6-38f0a636df08 backoff time=10ms error="failed to connect to peer: 95200575-5c4b-73ff-91a6-38f0a636df08"
2025-01-28T14:54:07.832Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to heartbeat to: peer=c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 backoff time=10ms error="failed to connect to peer: c85ee916-e3a7-2f3d-e0a7-266d2eb04f01"
2025-01-28T14:54:07.835Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to appendEntries to: peer="{Voter server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 c85ee916-e3a7-2f3d-e0a7-266d2eb04f01}" error="failed to connect to peer: c85ee916-e3a7-2f3d-e0a7-266d2eb04f01"
2025-01-28T14:54:07.839Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to appendEntries to: peer="{Voter server-95200575-5c4b-73ff-91a6-38f0a636df08 95200575-5c4b-73ff-91a6-38f0a636df08}" error="failed to connect to peer: 95200575-5c4b-73ff-91a6-38f0a636df08"
2025-01-28T14:54:07.848Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to heartbeat to: peer=95200575-5c4b-73ff-91a6-38f0a636df08 backoff time=10ms error="failed to connect to peer: 95200575-5c4b-73ff-91a6-38f0a636df08"
2025-01-28T14:54:07.852Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to heartbeat to: peer=c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 backoff time=10ms error="failed to connect to peer: c85ee916-e3a7-2f3d-e0a7-266d2eb04f01"
2025-01-28T14:54:07.854Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to appendEntries to: peer="{Voter server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 c85ee916-e3a7-2f3d-e0a7-266d2eb04f01}" error="failed to connect to peer: c85ee916-e3a7-2f3d-e0a7-266d2eb04f01"
2025-01-28T14:54:07.858Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to appendEntries to: peer="{Voter server-95200575-5c4b-73ff-91a6-38f0a636df08 95200575-5c4b-73ff-91a6-38f0a636df08}" error="failed to connect to peer: 95200575-5c4b-73ff-91a6-38f0a636df08"
2025-01-28T14:54:07.868Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to heartbeat to: peer=95200575-5c4b-73ff-91a6-38f0a636df08 backoff time=10ms error="failed to connect to peer: 95200575-5c4b-73ff-91a6-38f0a636df08"
2025-01-28T14:54:07.871Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to heartbeat to: peer=c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 backoff time=10ms error="failed to connect to peer: c85ee916-e3a7-2f3d-e0a7-266d2eb04f01"
2025-01-28T14:54:07.872Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to appendEntries to: peer="{Voter server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 c85ee916-e3a7-2f3d-e0a7-266d2eb04f01}" error="failed to connect to peer: c85ee916-e3a7-2f3d-e0a7-266d2eb04f01"
2025-01-28T14:54:07.874Z [WARN]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to contact: server-id=server-95200575-5c4b-73ff-91a6-38f0a636df08 time=51.023827ms
2025-01-28T14:54:07.875Z [WARN]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to contact: server-id=server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 time=50.940775ms
2025-01-28T14:54:07.875Z [WARN]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to contact quorum of nodes, stepping down
2025-01-28T14:54:07.875Z [INFO]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: entering follower state: follower="Node at ca8f3652-28d9-7f14-6bd4-c7e032de850c [Follower]" leader-address= leader-id=
2025-01-28T14:54:07.877Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to appendEntries to: peer="{Voter server-95200575-5c4b-73ff-91a6-38f0a636df08 95200575-5c4b-73ff-91a6-38f0a636df08}" error="failed to connect to peer: 95200575-5c4b-73ff-91a6-38f0a636df08"
2025-01-28T14:54:07.906Z [WARN]  server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: heartbeat timeout reached, starting election: last-leader-addr=ca8f3652-28d9-7f14-6bd4-c7e032de850c last-leader-id=server-ca8f3652-28d9-7f14-6bd4-c7e032de850c
2025-01-28T14:54:07.906Z [INFO]  server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: entering candidate state: node="Node at c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 [Candidate]" term=3
2025-01-28T14:54:07.906Z [ERROR] server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: failed to make requestVote RPC: target="{Voter server-ca8f3652-28d9-7f14-6bd4-c7e032de850c ca8f3652-28d9-7f14-6bd4-c7e032de850c}" error="failed to connect to peer: ca8f3652-28d9-7f14-6bd4-c7e032de850c" term=3
2025-01-28T14:54:07.906Z [WARN]  server-95200575-5c4b-73ff-91a6-38f0a636df08: rejecting vote request since we have a leader: from=c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 leader=ca8f3652-28d9-7f14-6bd4-c7e032de850c leader-id=server-ca8f3652-28d9-7f14-6bd4-c7e032de850c
2025-01-28T14:54:07.937Z [WARN]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:07.937Z [INFO]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: entering candidate state: node="Node at ca8f3652-28d9-7f14-6bd4-c7e032de850c [Candidate]" term=3
2025-01-28T14:54:07.937Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to make requestVote RPC: target="{Voter server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 c85ee916-e3a7-2f3d-e0a7-266d2eb04f01}" error="failed to connect to peer: c85ee916-e3a7-2f3d-e0a7-266d2eb04f01" term=3
2025-01-28T14:54:07.937Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to make requestVote RPC: target="{Voter server-95200575-5c4b-73ff-91a6-38f0a636df08 95200575-5c4b-73ff-91a6-38f0a636df08}" error="failed to connect to peer: 95200575-5c4b-73ff-91a6-38f0a636df08" term=3
2025-01-28T14:54:07.951Z [WARN]  server-95200575-5c4b-73ff-91a6-38f0a636df08: heartbeat timeout reached, starting election: last-leader-addr=ca8f3652-28d9-7f14-6bd4-c7e032de850c last-leader-id=server-ca8f3652-28d9-7f14-6bd4-c7e032de850c
2025-01-28T14:54:07.951Z [INFO]  server-95200575-5c4b-73ff-91a6-38f0a636df08: entering candidate state: node="Node at 95200575-5c4b-73ff-91a6-38f0a636df08 [Candidate]" term=3
2025-01-28T14:54:07.951Z [INFO]  server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: duplicate requestVote for same term: term=3
2025-01-28T14:54:07.951Z [ERROR] server-95200575-5c4b-73ff-91a6-38f0a636df08: failed to make requestVote RPC: target="{Voter server-ca8f3652-28d9-7f14-6bd4-c7e032de850c ca8f3652-28d9-7f14-6bd4-c7e032de850c}" error="failed to connect to peer: ca8f3652-28d9-7f14-6bd4-c7e032de850c" term=3
2025-01-28T14:54:08.002Z [WARN]  server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: Election timeout reached, restarting election
2025-01-28T14:54:08.003Z [INFO]  server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: entering candidate state: node="Node at c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 [Candidate]" term=4
2025-01-28T14:54:08.003Z [ERROR] server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: failed to make requestVote RPC: target="{Voter server-ca8f3652-28d9-7f14-6bd4-c7e032de850c ca8f3652-28d9-7f14-6bd4-c7e032de850c}" error="failed to connect to peer: ca8f3652-28d9-7f14-6bd4-c7e032de850c" term=4
2025-01-28T14:54:08.003Z [INFO]  server-95200575-5c4b-73ff-91a6-38f0a636df08: entering follower state: follower="Node at 95200575-5c4b-73ff-91a6-38f0a636df08 [Follower]" leader-address= leader-id=
2025-01-28T14:54:08.003Z [INFO]  server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: election won: term=4 tally=2
2025-01-28T14:54:08.003Z [INFO]  server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: entering leader state: leader="Node at c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 [Leader]"
2025-01-28T14:54:08.003Z [INFO]  server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: added peer, starting replication: peer=server-95200575-5c4b-73ff-91a6-38f0a636df08
2025-01-28T14:54:08.003Z [INFO]  server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: added peer, starting replication: peer=server-ca8f3652-28d9-7f14-6bd4-c7e032de850c
2025-01-28T14:54:08.003Z [ERROR] server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: failed to appendEntries to: peer="{Voter server-ca8f3652-28d9-7f14-6bd4-c7e032de850c ca8f3652-28d9-7f14-6bd4-c7e032de850c}" error="failed to connect to peer: ca8f3652-28d9-7f14-6bd4-c7e032de850c"
2025-01-28T14:54:08.004Z [INFO]  server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: pipelining replication: peer="{Voter server-95200575-5c4b-73ff-91a6-38f0a636df08 95200575-5c4b-73ff-91a6-38f0a636df08}"
2025-01-28T14:54:08.014Z [ERROR] server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: failed to heartbeat to: peer=ca8f3652-28d9-7f14-6bd4-c7e032de850c backoff time=10ms error="failed to connect to peer: ca8f3652-28d9-7f14-6bd4-c7e032de850c"
2025-01-28T14:54:08.019Z [WARN]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: Election timeout reached, restarting election
2025-01-28T14:54:08.019Z [INFO]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: entering candidate state: node="Node at ca8f3652-28d9-7f14-6bd4-c7e032de850c [Candidate]" term=4
2025-01-28T14:54:08.019Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to make requestVote RPC: target="{Voter server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 c85ee916-e3a7-2f3d-e0a7-266d2eb04f01}" error="failed to connect to peer: c85ee916-e3a7-2f3d-e0a7-266d2eb04f01" term=4
2025-01-28T14:54:08.019Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to make requestVote RPC: target="{Voter server-95200575-5c4b-73ff-91a6-38f0a636df08 95200575-5c4b-73ff-91a6-38f0a636df08}" error="failed to connect to peer: 95200575-5c4b-73ff-91a6-38f0a636df08" term=4
2025-01-28T14:54:08.020Z [ERROR] server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: failed to appendEntries to: peer="{Voter server-ca8f3652-28d9-7f14-6bd4-c7e032de850c ca8f3652-28d9-7f14-6bd4-c7e032de850c}" error="failed to connect to peer: ca8f3652-28d9-7f14-6bd4-c7e032de850c"
2025-01-28T14:54:08.033Z [ERROR] server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: failed to heartbeat to: peer=ca8f3652-28d9-7f14-6bd4-c7e032de850c backoff time=10ms error="failed to connect to peer: ca8f3652-28d9-7f14-6bd4-c7e032de850c"
2025-01-28T14:54:08.038Z [ERROR] server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: failed to appendEntries to: peer="{Voter server-ca8f3652-28d9-7f14-6bd4-c7e032de850c ca8f3652-28d9-7f14-6bd4-c7e032de850c}" error="failed to connect to peer: ca8f3652-28d9-7f14-6bd4-c7e032de850c"
2025-01-28T14:54:08.053Z [ERROR] server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: failed to heartbeat to: peer=ca8f3652-28d9-7f14-6bd4-c7e032de850c backoff time=10ms error="failed to connect to peer: ca8f3652-28d9-7f14-6bd4-c7e032de850c"
2025-01-28T14:54:08.054Z [WARN]  server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: failed to contact: server-id=server-ca8f3652-28d9-7f14-6bd4-c7e032de850c time=51.050384ms
2025-01-28T14:54:08.067Z [ERROR] server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: failed to appendEntries to: peer="{Voter server-ca8f3652-28d9-7f14-6bd4-c7e032de850c ca8f3652-28d9-7f14-6bd4-c7e032de850c}" error="failed to connect to peer: ca8f3652-28d9-7f14-6bd4-c7e032de850c"
2025-01-28T14:54:08.072Z [ERROR] server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: failed to heartbeat to: peer=ca8f3652-28d9-7f14-6bd4-c7e032de850c backoff time=20ms error="failed to connect to peer: ca8f3652-28d9-7f14-6bd4-c7e032de850c"
2025-01-28T14:54:08.076Z [WARN]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: Election timeout reached, restarting election
2025-01-28T14:54:08.076Z [INFO]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: entering candidate state: node="Node at ca8f3652-28d9-7f14-6bd4-c7e032de850c [Candidate]" term=5
2025-01-28T14:54:08.076Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to make requestVote RPC: target="{Voter server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 c85ee916-e3a7-2f3d-e0a7-266d2eb04f01}" error="failed to connect to peer: c85ee916-e3a7-2f3d-e0a7-266d2eb04f01" term=5
2025-01-28T14:54:08.076Z [ERROR] server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to make requestVote RPC: target="{Voter server-95200575-5c4b-73ff-91a6-38f0a636df08 95200575-5c4b-73ff-91a6-38f0a636df08}" error="failed to connect to peer: 95200575-5c4b-73ff-91a6-38f0a636df08" term=5
2025-01-28T14:54:08.102Z [ERROR] server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: failed to heartbeat to: peer=ca8f3652-28d9-7f14-6bd4-c7e032de850c backoff time=25ms error="failed to connect to peer: ca8f3652-28d9-7f14-6bd4-c7e032de850c"
2025-01-28T14:54:08.104Z [WARN]  server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: failed to contact: server-id=server-ca8f3652-28d9-7f14-6bd4-c7e032de850c time=100.732677ms
2025-01-28T14:54:08.109Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:08.00375221 +0000 UTC m=+33.678242210 (1 nodes), 174.278138ms from start of poll, 458.433131ms from cluster start. Timeout at 2025-01-28 14:54:08.10875665 +0000 UTC m=+33.783247230, 105.00502ms after stability
    raft_test.go:413: [INFO] Reconnecting Node at ca8f3652-28d9-7f14-6bd4-c7e032de850c [Candidate]
2025-01-28T14:54:08.114Z [ERROR] server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: peer has newer term, stopping replication: peer="{Voter server-ca8f3652-28d9-7f14-6bd4-c7e032de850c ca8f3652-28d9-7f14-6bd4-c7e032de850c}"
2025-01-28T14:54:08.115Z [INFO]  server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: entering follower state: follower="Node at c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 [Follower]" leader-address= leader-id=
2025-01-28T14:54:08.115Z [INFO]  server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01: aborting pipeline replication: peer="{Voter server-95200575-5c4b-73ff-91a6-38f0a636df08 95200575-5c4b-73ff-91a6-38f0a636df08}"
2025-01-28T14:54:08.162Z [WARN]  server-95200575-5c4b-73ff-91a6-38f0a636df08: heartbeat timeout reached, starting election: last-leader-addr=c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 last-leader-id=server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01
2025-01-28T14:54:08.162Z [INFO]  server-95200575-5c4b-73ff-91a6-38f0a636df08: entering candidate state: node="Node at 95200575-5c4b-73ff-91a6-38f0a636df08 [Candidate]" term=5
2025-01-28T14:54:08.162Z [INFO]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: duplicate requestVote for same term: term=5
2025-01-28T14:54:08.162Z [INFO]  server-95200575-5c4b-73ff-91a6-38f0a636df08: election won: term=5 tally=2
2025-01-28T14:54:08.162Z [INFO]  server-95200575-5c4b-73ff-91a6-38f0a636df08: entering leader state: leader="Node at 95200575-5c4b-73ff-91a6-38f0a636df08 [Leader]"
2025-01-28T14:54:08.162Z [INFO]  server-95200575-5c4b-73ff-91a6-38f0a636df08: added peer, starting replication: peer=server-ca8f3652-28d9-7f14-6bd4-c7e032de850c
2025-01-28T14:54:08.162Z [INFO]  server-95200575-5c4b-73ff-91a6-38f0a636df08: added peer, starting replication: peer=server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01
2025-01-28T14:54:08.162Z [WARN]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: failed to get previous log: previous-index=5 last-index=3 error="log not found"
2025-01-28T14:54:08.162Z [INFO]  server-ca8f3652-28d9-7f14-6bd4-c7e032de850c: entering follower state: follower="Node at ca8f3652-28d9-7f14-6bd4-c7e032de850c [Follower]" leader-address=95200575-5c4b-73ff-91a6-38f0a636df08 leader-id=server-95200575-5c4b-73ff-91a6-38f0a636df08
2025-01-28T14:54:08.162Z [INFO]  server-95200575-5c4b-73ff-91a6-38f0a636df08: pipelining replication: peer="{Voter server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 c85ee916-e3a7-2f3d-e0a7-266d2eb04f01}"
2025-01-28T14:54:08.162Z [WARN]  server-95200575-5c4b-73ff-91a6-38f0a636df08: appendEntries rejected, sending older logs: peer="{Voter server-ca8f3652-28d9-7f14-6bd4-c7e032de850c ca8f3652-28d9-7f14-6bd4-c7e032de850c}" next=4
2025-01-28T14:54:08.163Z [INFO]  server-95200575-5c4b-73ff-91a6-38f0a636df08: pipelining replication: peer="{Voter server-ca8f3652-28d9-7f14-6bd4-c7e032de850c ca8f3652-28d9-7f14-6bd4-c7e032de850c}"
2025-01-28T14:54:08.168Z [INFO]  server-95200575-5c4b-73ff-91a6-38f0a636df08: aborting pipeline replication: peer="{Voter server-ca8f3652-28d9-7f14-6bd4-c7e032de850c ca8f3652-28d9-7f14-6bd4-c7e032de850c}"
2025-01-28T14:54:08.168Z [INFO]  server-95200575-5c4b-73ff-91a6-38f0a636df08: aborting pipeline replication: peer="{Voter server-c85ee916-e3a7-2f3d-e0a7-266d2eb04f01 c85ee916-e3a7-2f3d-e0a7-266d2eb04f01}"
--- PASS: TestRaft_LeaderFail (0.63s)
=== RUN   TestRaft_BehindFollower
2025-01-28T14:54:08.172Z [INFO]  server-259c2814-2c0e-8740-7c64-7fcdd0bb018f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-259c2814-2c0e-8740-7c64-7fcdd0bb018f Address:259c2814-2c0e-8740-7c64-7fcdd0bb018f} {Suffrage:Voter ID:server-cb79b771-b354-3c34-6925-3cfa59b3eab5 Address:cb79b771-b354-3c34-6925-3cfa59b3eab5} {Suffrage:Voter ID:server-d9e430cf-1cbc-2b83-e0b7-8e63354295bf Address:d9e430cf-1cbc-2b83-e0b7-8e63354295bf}]"
2025-01-28T14:54:08.172Z [INFO]  server-259c2814-2c0e-8740-7c64-7fcdd0bb018f: entering follower state: follower="Node at 259c2814-2c0e-8740-7c64-7fcdd0bb018f [Follower]" leader-address= leader-id=
2025-01-28T14:54:08.172Z [INFO]  server-cb79b771-b354-3c34-6925-3cfa59b3eab5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-259c2814-2c0e-8740-7c64-7fcdd0bb018f Address:259c2814-2c0e-8740-7c64-7fcdd0bb018f} {Suffrage:Voter ID:server-cb79b771-b354-3c34-6925-3cfa59b3eab5 Address:cb79b771-b354-3c34-6925-3cfa59b3eab5} {Suffrage:Voter ID:server-d9e430cf-1cbc-2b83-e0b7-8e63354295bf Address:d9e430cf-1cbc-2b83-e0b7-8e63354295bf}]"
2025-01-28T14:54:08.172Z [INFO]  server-cb79b771-b354-3c34-6925-3cfa59b3eab5: entering follower state: follower="Node at cb79b771-b354-3c34-6925-3cfa59b3eab5 [Follower]" leader-address= leader-id=
2025-01-28T14:54:08.172Z [INFO]  server-d9e430cf-1cbc-2b83-e0b7-8e63354295bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-259c2814-2c0e-8740-7c64-7fcdd0bb018f Address:259c2814-2c0e-8740-7c64-7fcdd0bb018f} {Suffrage:Voter ID:server-cb79b771-b354-3c34-6925-3cfa59b3eab5 Address:cb79b771-b354-3c34-6925-3cfa59b3eab5} {Suffrage:Voter ID:server-d9e430cf-1cbc-2b83-e0b7-8e63354295bf Address:d9e430cf-1cbc-2b83-e0b7-8e63354295bf}]"
2025-01-28T14:54:08.172Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:08.172Z [INFO]  server-d9e430cf-1cbc-2b83-e0b7-8e63354295bf: entering follower state: follower="Node at d9e430cf-1cbc-2b83-e0b7-8e63354295bf [Follower]" leader-address= leader-id=
2025-01-28T14:54:08.227Z [WARN]  server-259c2814-2c0e-8740-7c64-7fcdd0bb018f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:08.227Z [INFO]  server-259c2814-2c0e-8740-7c64-7fcdd0bb018f: entering candidate state: node="Node at 259c2814-2c0e-8740-7c64-7fcdd0bb018f [Candidate]" term=2
2025-01-28T14:54:08.227Z [INFO]  server-259c2814-2c0e-8740-7c64-7fcdd0bb018f: election won: term=2 tally=2
2025-01-28T14:54:08.227Z [INFO]  server-259c2814-2c0e-8740-7c64-7fcdd0bb018f: entering leader state: leader="Node at 259c2814-2c0e-8740-7c64-7fcdd0bb018f [Leader]"
2025-01-28T14:54:08.227Z [INFO]  server-259c2814-2c0e-8740-7c64-7fcdd0bb018f: added peer, starting replication: peer=server-cb79b771-b354-3c34-6925-3cfa59b3eab5
2025-01-28T14:54:08.227Z [INFO]  server-259c2814-2c0e-8740-7c64-7fcdd0bb018f: added peer, starting replication: peer=server-d9e430cf-1cbc-2b83-e0b7-8e63354295bf
2025-01-28T14:54:08.227Z [INFO]  server-259c2814-2c0e-8740-7c64-7fcdd0bb018f: pipelining replication: peer="{Voter server-cb79b771-b354-3c34-6925-3cfa59b3eab5 cb79b771-b354-3c34-6925-3cfa59b3eab5}"
2025-01-28T14:54:08.228Z [INFO]  server-259c2814-2c0e-8740-7c64-7fcdd0bb018f: pipelining replication: peer="{Voter server-d9e430cf-1cbc-2b83-e0b7-8e63354295bf d9e430cf-1cbc-2b83-e0b7-8e63354295bf}"
2025-01-28T14:54:08.333Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:08.227765297 +0000 UTC m=+33.902255397 (1 nodes), 54.840166ms from start of poll, 55.856351ms from cluster start. Timeout at 2025-01-28 14:54:08.332770216 +0000 UTC m=+34.007260276, 105.004879ms after stability
2025-01-28T14:54:08.334Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:08.439Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:08.334122683 +0000 UTC m=+34.008612663 (2 nodes), 2.8µs from start of poll, 162.213617ms from cluster start. Timeout at 2025-01-28 14:54:08.439142161 +0000 UTC m=+34.113632261, 105.019598ms after stability
2025-01-28T14:54:08.439Z [ERROR] server-259c2814-2c0e-8740-7c64-7fcdd0bb018f: failed to pipeline appendEntries: peer="{Voter server-cb79b771-b354-3c34-6925-3cfa59b3eab5 cb79b771-b354-3c34-6925-3cfa59b3eab5}" error="append pipeline closed"
2025-01-28T14:54:08.439Z [INFO]  server-259c2814-2c0e-8740-7c64-7fcdd0bb018f: aborting pipeline replication: peer="{Voter server-cb79b771-b354-3c34-6925-3cfa59b3eab5 cb79b771-b354-3c34-6925-3cfa59b3eab5}"
2025-01-28T14:54:08.439Z [ERROR] server-259c2814-2c0e-8740-7c64-7fcdd0bb018f: failed to appendEntries to: peer="{Voter server-cb79b771-b354-3c34-6925-3cfa59b3eab5 cb79b771-b354-3c34-6925-3cfa59b3eab5}" error="failed to connect to peer: cb79b771-b354-3c34-6925-3cfa59b3eab5"
2025-01-28T14:54:08.444Z [ERROR] server-259c2814-2c0e-8740-7c64-7fcdd0bb018f: failed to heartbeat to: peer=cb79b771-b354-3c34-6925-3cfa59b3eab5 backoff time=10ms error="failed to connect to peer: cb79b771-b354-3c34-6925-3cfa59b3eab5"
    raft_test.go:459: [INFO] Finished apply without behind follower
2025-01-28T14:54:08.450Z [INFO]  server-259c2814-2c0e-8740-7c64-7fcdd0bb018f: pipelining replication: peer="{Voter server-cb79b771-b354-3c34-6925-3cfa59b3eab5 cb79b771-b354-3c34-6925-3cfa59b3eab5}"
2025-01-28T14:54:08.463Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:08.568Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:08.463628286 +0000 UTC m=+34.138118266 (1 nodes), 2.36µs from start of poll, 291.71922ms from cluster start. Timeout at 2025-01-28 14:54:08.568663243 +0000 UTC m=+34.243153283, 105.035017ms after stability
2025-01-28T14:54:08.569Z [INFO]  server-259c2814-2c0e-8740-7c64-7fcdd0bb018f: aborting pipeline replication: peer="{Voter server-cb79b771-b354-3c34-6925-3cfa59b3eab5 cb79b771-b354-3c34-6925-3cfa59b3eab5}"
2025-01-28T14:54:08.569Z [INFO]  server-259c2814-2c0e-8740-7c64-7fcdd0bb018f: aborting pipeline replication: peer="{Voter server-d9e430cf-1cbc-2b83-e0b7-8e63354295bf d9e430cf-1cbc-2b83-e0b7-8e63354295bf}"
--- PASS: TestRaft_BehindFollower (0.40s)
=== RUN   TestRaft_ApplyNonLeader
2025-01-28T14:54:08.573Z [INFO]  server-97b26d48-6271-3c33-0bf4-f49dd176cbcf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97b26d48-6271-3c33-0bf4-f49dd176cbcf Address:97b26d48-6271-3c33-0bf4-f49dd176cbcf} {Suffrage:Voter ID:server-a0ca82cd-07d1-ff92-338c-434d892ac522 Address:a0ca82cd-07d1-ff92-338c-434d892ac522} {Suffrage:Voter ID:server-6ae0868c-4dbc-e09d-2375-337216515fe1 Address:6ae0868c-4dbc-e09d-2375-337216515fe1}]"
2025-01-28T14:54:08.573Z [INFO]  server-97b26d48-6271-3c33-0bf4-f49dd176cbcf: entering follower state: follower="Node at 97b26d48-6271-3c33-0bf4-f49dd176cbcf [Follower]" leader-address= leader-id=
2025-01-28T14:54:08.573Z [INFO]  server-a0ca82cd-07d1-ff92-338c-434d892ac522: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97b26d48-6271-3c33-0bf4-f49dd176cbcf Address:97b26d48-6271-3c33-0bf4-f49dd176cbcf} {Suffrage:Voter ID:server-a0ca82cd-07d1-ff92-338c-434d892ac522 Address:a0ca82cd-07d1-ff92-338c-434d892ac522} {Suffrage:Voter ID:server-6ae0868c-4dbc-e09d-2375-337216515fe1 Address:6ae0868c-4dbc-e09d-2375-337216515fe1}]"
2025-01-28T14:54:08.573Z [INFO]  server-a0ca82cd-07d1-ff92-338c-434d892ac522: entering follower state: follower="Node at a0ca82cd-07d1-ff92-338c-434d892ac522 [Follower]" leader-address= leader-id=
2025-01-28T14:54:08.573Z [INFO]  server-6ae0868c-4dbc-e09d-2375-337216515fe1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-97b26d48-6271-3c33-0bf4-f49dd176cbcf Address:97b26d48-6271-3c33-0bf4-f49dd176cbcf} {Suffrage:Voter ID:server-a0ca82cd-07d1-ff92-338c-434d892ac522 Address:a0ca82cd-07d1-ff92-338c-434d892ac522} {Suffrage:Voter ID:server-6ae0868c-4dbc-e09d-2375-337216515fe1 Address:6ae0868c-4dbc-e09d-2375-337216515fe1}]"
2025-01-28T14:54:08.574Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:08.574Z [INFO]  server-6ae0868c-4dbc-e09d-2375-337216515fe1: entering follower state: follower="Node at 6ae0868c-4dbc-e09d-2375-337216515fe1 [Follower]" leader-address= leader-id=
2025-01-28T14:54:08.628Z [WARN]  server-6ae0868c-4dbc-e09d-2375-337216515fe1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:08.628Z [INFO]  server-6ae0868c-4dbc-e09d-2375-337216515fe1: entering candidate state: node="Node at 6ae0868c-4dbc-e09d-2375-337216515fe1 [Candidate]" term=2
2025-01-28T14:54:08.628Z [WARN]  server-a0ca82cd-07d1-ff92-338c-434d892ac522: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:08.629Z [INFO]  server-a0ca82cd-07d1-ff92-338c-434d892ac522: entering candidate state: node="Node at a0ca82cd-07d1-ff92-338c-434d892ac522 [Candidate]" term=2
2025-01-28T14:54:08.629Z [INFO]  server-6ae0868c-4dbc-e09d-2375-337216515fe1: election won: term=2 tally=2
2025-01-28T14:54:08.629Z [INFO]  server-a0ca82cd-07d1-ff92-338c-434d892ac522: duplicate requestVote for same term: term=2
2025-01-28T14:54:08.629Z [INFO]  server-6ae0868c-4dbc-e09d-2375-337216515fe1: entering leader state: leader="Node at 6ae0868c-4dbc-e09d-2375-337216515fe1 [Leader]"
2025-01-28T14:54:08.629Z [INFO]  server-6ae0868c-4dbc-e09d-2375-337216515fe1: added peer, starting replication: peer=server-97b26d48-6271-3c33-0bf4-f49dd176cbcf
2025-01-28T14:54:08.629Z [INFO]  server-97b26d48-6271-3c33-0bf4-f49dd176cbcf: duplicate requestVote for same term: term=2
2025-01-28T14:54:08.629Z [INFO]  server-6ae0868c-4dbc-e09d-2375-337216515fe1: added peer, starting replication: peer=server-a0ca82cd-07d1-ff92-338c-434d892ac522
2025-01-28T14:54:08.629Z [WARN]  server-6ae0868c-4dbc-e09d-2375-337216515fe1: rejecting vote request since we have a leader: from=a0ca82cd-07d1-ff92-338c-434d892ac522 leader=6ae0868c-4dbc-e09d-2375-337216515fe1 leader-id=server-6ae0868c-4dbc-e09d-2375-337216515fe1
2025-01-28T14:54:08.629Z [INFO]  server-a0ca82cd-07d1-ff92-338c-434d892ac522: entering follower state: follower="Node at a0ca82cd-07d1-ff92-338c-434d892ac522 [Follower]" leader-address=6ae0868c-4dbc-e09d-2375-337216515fe1 leader-id=server-6ae0868c-4dbc-e09d-2375-337216515fe1
2025-01-28T14:54:08.629Z [INFO]  server-6ae0868c-4dbc-e09d-2375-337216515fe1: pipelining replication: peer="{Voter server-97b26d48-6271-3c33-0bf4-f49dd176cbcf 97b26d48-6271-3c33-0bf4-f49dd176cbcf}"
2025-01-28T14:54:08.629Z [INFO]  server-6ae0868c-4dbc-e09d-2375-337216515fe1: pipelining replication: peer="{Voter server-a0ca82cd-07d1-ff92-338c-434d892ac522 a0ca82cd-07d1-ff92-338c-434d892ac522}"
2025-01-28T14:54:08.734Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:08.62947374 +0000 UTC m=+34.303963720 (1 nodes), 55.43329ms from start of poll, 56.49447ms from cluster start. Timeout at 2025-01-28 14:54:08.734477499 +0000 UTC m=+34.408967619, 105.003899ms after stability
2025-01-28T14:54:08.734Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:08.840Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:08.734823341 +0000 UTC m=+34.409313321 (2 nodes), 2.679µs from start of poll, 161.844071ms from cluster start. Timeout at 2025-01-28 14:54:08.839858918 +0000 UTC m=+34.514348978, 105.035657ms after stability
2025-01-28T14:54:08.840Z [INFO]  server-6ae0868c-4dbc-e09d-2375-337216515fe1: aborting pipeline replication: peer="{Voter server-a0ca82cd-07d1-ff92-338c-434d892ac522 a0ca82cd-07d1-ff92-338c-434d892ac522}"
2025-01-28T14:54:08.840Z [INFO]  server-6ae0868c-4dbc-e09d-2375-337216515fe1: aborting pipeline replication: peer="{Voter server-97b26d48-6271-3c33-0bf4-f49dd176cbcf 97b26d48-6271-3c33-0bf4-f49dd176cbcf}"
2025-01-28T14:54:09.341Z [ERROR] server-6ae0868c-4dbc-e09d-2375-337216515fe1: failed to heartbeat to: peer=a0ca82cd-07d1-ff92-338c-434d892ac522 backoff time=10ms error="command timed out"
--- PASS: TestRaft_ApplyNonLeader (0.77s)
=== RUN   TestRaft_ApplyConcurrent
2025-01-28T14:54:09.345Z [INFO]  server-6e008763-cf5d-9965-aeef-6013999d31e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e008763-cf5d-9965-aeef-6013999d31e9 Address:6e008763-cf5d-9965-aeef-6013999d31e9} {Suffrage:Voter ID:server-f33119c5-e798-2b25-c714-1cf09faf7f84 Address:f33119c5-e798-2b25-c714-1cf09faf7f84} {Suffrage:Voter ID:server-d4deb887-edd7-f23c-c0b7-fdf1e8164b5b Address:d4deb887-edd7-f23c-c0b7-fdf1e8164b5b}]"
2025-01-28T14:54:09.345Z [INFO]  server-6e008763-cf5d-9965-aeef-6013999d31e9: entering follower state: follower="Node at 6e008763-cf5d-9965-aeef-6013999d31e9 [Follower]" leader-address= leader-id=
2025-01-28T14:54:09.345Z [INFO]  server-f33119c5-e798-2b25-c714-1cf09faf7f84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e008763-cf5d-9965-aeef-6013999d31e9 Address:6e008763-cf5d-9965-aeef-6013999d31e9} {Suffrage:Voter ID:server-f33119c5-e798-2b25-c714-1cf09faf7f84 Address:f33119c5-e798-2b25-c714-1cf09faf7f84} {Suffrage:Voter ID:server-d4deb887-edd7-f23c-c0b7-fdf1e8164b5b Address:d4deb887-edd7-f23c-c0b7-fdf1e8164b5b}]"
2025-01-28T14:54:09.345Z [INFO]  server-f33119c5-e798-2b25-c714-1cf09faf7f84: entering follower state: follower="Node at f33119c5-e798-2b25-c714-1cf09faf7f84 [Follower]" leader-address= leader-id=
2025-01-28T14:54:09.346Z [INFO]  server-d4deb887-edd7-f23c-c0b7-fdf1e8164b5b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e008763-cf5d-9965-aeef-6013999d31e9 Address:6e008763-cf5d-9965-aeef-6013999d31e9} {Suffrage:Voter ID:server-f33119c5-e798-2b25-c714-1cf09faf7f84 Address:f33119c5-e798-2b25-c714-1cf09faf7f84} {Suffrage:Voter ID:server-d4deb887-edd7-f23c-c0b7-fdf1e8164b5b Address:d4deb887-edd7-f23c-c0b7-fdf1e8164b5b}]"
2025-01-28T14:54:09.346Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:09.346Z [INFO]  server-d4deb887-edd7-f23c-c0b7-fdf1e8164b5b: entering follower state: follower="Node at d4deb887-edd7-f23c-c0b7-fdf1e8164b5b [Follower]" leader-address= leader-id=
2025-01-28T14:54:09.476Z [WARN]  server-6e008763-cf5d-9965-aeef-6013999d31e9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:09.476Z [INFO]  server-6e008763-cf5d-9965-aeef-6013999d31e9: entering candidate state: node="Node at 6e008763-cf5d-9965-aeef-6013999d31e9 [Candidate]" term=2
2025-01-28T14:54:09.476Z [INFO]  server-6e008763-cf5d-9965-aeef-6013999d31e9: election won: term=2 tally=2
2025-01-28T14:54:09.476Z [INFO]  server-6e008763-cf5d-9965-aeef-6013999d31e9: entering leader state: leader="Node at 6e008763-cf5d-9965-aeef-6013999d31e9 [Leader]"
2025-01-28T14:54:09.476Z [INFO]  server-6e008763-cf5d-9965-aeef-6013999d31e9: added peer, starting replication: peer=server-f33119c5-e798-2b25-c714-1cf09faf7f84
2025-01-28T14:54:09.476Z [INFO]  server-6e008763-cf5d-9965-aeef-6013999d31e9: added peer, starting replication: peer=server-d4deb887-edd7-f23c-c0b7-fdf1e8164b5b
2025-01-28T14:54:09.477Z [INFO]  server-6e008763-cf5d-9965-aeef-6013999d31e9: pipelining replication: peer="{Voter server-d4deb887-edd7-f23c-c0b7-fdf1e8164b5b d4deb887-edd7-f23c-c0b7-fdf1e8164b5b}"
2025-01-28T14:54:09.477Z [INFO]  server-6e008763-cf5d-9965-aeef-6013999d31e9: pipelining replication: peer="{Voter server-f33119c5-e798-2b25-c714-1cf09faf7f84 f33119c5-e798-2b25-c714-1cf09faf7f84}"
2025-01-28T14:54:09.682Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:09.476873077 +0000 UTC m=+35.151363057 (1 nodes), 130.578562ms from start of poll, 131.628403ms from cluster start. Timeout at 2025-01-28 14:54:09.681879916 +0000 UTC m=+35.356369956, 205.006899ms after stability
2025-01-28T14:54:09.697Z [INFO]  server-6e008763-cf5d-9965-aeef-6013999d31e9: aborting pipeline replication: peer="{Voter server-d4deb887-edd7-f23c-c0b7-fdf1e8164b5b d4deb887-edd7-f23c-c0b7-fdf1e8164b5b}"
2025-01-28T14:54:09.697Z [INFO]  server-6e008763-cf5d-9965-aeef-6013999d31e9: aborting pipeline replication: peer="{Voter server-f33119c5-e798-2b25-c714-1cf09faf7f84 f33119c5-e798-2b25-c714-1cf09faf7f84}"
--- PASS: TestRaft_ApplyConcurrent (0.36s)
=== RUN   TestRaft_ApplyConcurrent_Timeout
2025-01-28T14:54:09.699Z [INFO]  server-15df5da1-5489-9190-9dbc-5ab5efaffb73: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-15df5da1-5489-9190-9dbc-5ab5efaffb73 Address:15df5da1-5489-9190-9dbc-5ab5efaffb73}]"
2025-01-28T14:54:09.700Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:09.700Z [INFO]  server-15df5da1-5489-9190-9dbc-5ab5efaffb73: entering follower state: follower="Node at 15df5da1-5489-9190-9dbc-5ab5efaffb73 [Follower]" leader-address= leader-id=
2025-01-28T14:54:09.891Z [WARN]  server-15df5da1-5489-9190-9dbc-5ab5efaffb73: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:09.891Z [INFO]  server-15df5da1-5489-9190-9dbc-5ab5efaffb73: entering candidate state: node="Node at 15df5da1-5489-9190-9dbc-5ab5efaffb73 [Candidate]" term=2
2025-01-28T14:54:09.891Z [INFO]  server-15df5da1-5489-9190-9dbc-5ab5efaffb73: election won: term=2 tally=1
2025-01-28T14:54:09.891Z [INFO]  server-15df5da1-5489-9190-9dbc-5ab5efaffb73: entering leader state: leader="Node at 15df5da1-5489-9190-9dbc-5ab5efaffb73 [Leader]"
2025-01-28T14:54:10.093Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:09.891974176 +0000 UTC m=+35.566464156 (1 nodes), 191.927373ms from start of poll, 192.346874ms from cluster start. Timeout at 2025-01-28 14:54:10.092981015 +0000 UTC m=+35.767471135, 201.006979ms after stability
--- PASS: TestRaft_ApplyConcurrent_Timeout (0.40s)
=== RUN   TestRaft_JoinNode
2025-01-28T14:54:10.096Z [INFO]  server-327182ef-baab-9b93-33be-9143202b1706: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-327182ef-baab-9b93-33be-9143202b1706 Address:327182ef-baab-9b93-33be-9143202b1706} {Suffrage:Voter ID:server-50cc7076-8741-7987-c24e-f8a12069a615 Address:50cc7076-8741-7987-c24e-f8a12069a615}]"
2025-01-28T14:54:10.096Z [INFO]  server-327182ef-baab-9b93-33be-9143202b1706: entering follower state: follower="Node at 327182ef-baab-9b93-33be-9143202b1706 [Follower]" leader-address= leader-id=
2025-01-28T14:54:10.097Z [INFO]  server-50cc7076-8741-7987-c24e-f8a12069a615: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-327182ef-baab-9b93-33be-9143202b1706 Address:327182ef-baab-9b93-33be-9143202b1706} {Suffrage:Voter ID:server-50cc7076-8741-7987-c24e-f8a12069a615 Address:50cc7076-8741-7987-c24e-f8a12069a615}]"
2025-01-28T14:54:10.097Z [INFO]  server-50cc7076-8741-7987-c24e-f8a12069a615: entering follower state: follower="Node at 50cc7076-8741-7987-c24e-f8a12069a615 [Follower]" leader-address= leader-id=
2025-01-28T14:54:10.097Z [INFO]  server-fdc951f2-46d3-6012-50e8-e739433a6060: initial configuration: index=0 servers=[]
2025-01-28T14:54:10.097Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:10.097Z [INFO]  server-fdc951f2-46d3-6012-50e8-e739433a6060: entering follower state: follower="Node at fdc951f2-46d3-6012-50e8-e739433a6060 [Follower]" leader-address= leader-id=
2025-01-28T14:54:10.171Z [WARN]  server-fdc951f2-46d3-6012-50e8-e739433a6060: no known peers, aborting election
2025-01-28T14:54:10.177Z [WARN]  server-50cc7076-8741-7987-c24e-f8a12069a615: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:10.177Z [INFO]  server-50cc7076-8741-7987-c24e-f8a12069a615: entering candidate state: node="Node at 50cc7076-8741-7987-c24e-f8a12069a615 [Candidate]" term=2
2025-01-28T14:54:10.177Z [INFO]  server-50cc7076-8741-7987-c24e-f8a12069a615: election won: term=2 tally=2
2025-01-28T14:54:10.178Z [INFO]  server-50cc7076-8741-7987-c24e-f8a12069a615: entering leader state: leader="Node at 50cc7076-8741-7987-c24e-f8a12069a615 [Leader]"
2025-01-28T14:54:10.178Z [INFO]  server-50cc7076-8741-7987-c24e-f8a12069a615: added peer, starting replication: peer=server-327182ef-baab-9b93-33be-9143202b1706
2025-01-28T14:54:10.178Z [INFO]  server-50cc7076-8741-7987-c24e-f8a12069a615: pipelining replication: peer="{Voter server-327182ef-baab-9b93-33be-9143202b1706 327182ef-baab-9b93-33be-9143202b1706}"
2025-01-28T14:54:10.284Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:10.178178316 +0000 UTC m=+35.852668316 (1 nodes), 80.201314ms from start of poll, 81.698253ms from cluster start. Timeout at 2025-01-28 14:54:10.283185676 +0000 UTC m=+35.957675716, 105.0074ms after stability
2025-01-28T14:54:10.284Z [INFO]  server-50cc7076-8741-7987-c24e-f8a12069a615: updating configuration: command=AddVoter server-id=server-fdc951f2-46d3-6012-50e8-e739433a6060 server-addr=fdc951f2-46d3-6012-50e8-e739433a6060 servers="[{Suffrage:Voter ID:server-327182ef-baab-9b93-33be-9143202b1706 Address:327182ef-baab-9b93-33be-9143202b1706} {Suffrage:Voter ID:server-50cc7076-8741-7987-c24e-f8a12069a615 Address:50cc7076-8741-7987-c24e-f8a12069a615} {Suffrage:Voter ID:server-fdc951f2-46d3-6012-50e8-e739433a6060 Address:fdc951f2-46d3-6012-50e8-e739433a6060}]"
2025-01-28T14:54:10.284Z [INFO]  server-50cc7076-8741-7987-c24e-f8a12069a615: added peer, starting replication: peer=server-fdc951f2-46d3-6012-50e8-e739433a6060
2025-01-28T14:54:10.284Z [WARN]  server-fdc951f2-46d3-6012-50e8-e739433a6060: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-01-28T14:54:10.284Z [WARN]  server-50cc7076-8741-7987-c24e-f8a12069a615: appendEntries rejected, sending older logs: peer="{Voter server-fdc951f2-46d3-6012-50e8-e739433a6060 fdc951f2-46d3-6012-50e8-e739433a6060}" next=1
2025-01-28T14:54:10.285Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:10.285Z [INFO]  server-50cc7076-8741-7987-c24e-f8a12069a615: pipelining replication: peer="{Voter server-fdc951f2-46d3-6012-50e8-e739433a6060 fdc951f2-46d3-6012-50e8-e739433a6060}"
2025-01-28T14:54:10.390Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:10.285206859 +0000 UTC m=+35.959696879 (1 nodes), 1.86µs from start of poll, 188.726816ms from cluster start. Timeout at 2025-01-28 14:54:10.390230617 +0000 UTC m=+36.064720597, 105.023718ms after stability
2025-01-28T14:54:10.391Z [INFO]  server-50cc7076-8741-7987-c24e-f8a12069a615: aborting pipeline replication: peer="{Voter server-fdc951f2-46d3-6012-50e8-e739433a6060 fdc951f2-46d3-6012-50e8-e739433a6060}"
2025-01-28T14:54:10.391Z [INFO]  server-50cc7076-8741-7987-c24e-f8a12069a615: aborting pipeline replication: peer="{Voter server-327182ef-baab-9b93-33be-9143202b1706 327182ef-baab-9b93-33be-9143202b1706}"
--- PASS: TestRaft_JoinNode (0.30s)
=== RUN   TestRaft_JoinNode_ConfigStore
2025-01-28T14:54:10.393Z [INFO]  server-183828d8-410c-72b5-37b5-b094d1b7c6fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-183828d8-410c-72b5-37b5-b094d1b7c6fa Address:183828d8-410c-72b5-37b5-b094d1b7c6fa}]"
2025-01-28T14:54:10.393Z [INFO]  server-183828d8-410c-72b5-37b5-b094d1b7c6fa: entering follower state: follower="Node at 183828d8-410c-72b5-37b5-b094d1b7c6fa [Follower]" leader-address= leader-id=
2025-01-28T14:54:10.394Z [INFO]  server-c14a7aa7-240d-742f-6ff7-cea5db30bb7f: initial configuration: index=0 servers=[]
2025-01-28T14:54:10.394Z [INFO]  server-c14a7aa7-240d-742f-6ff7-cea5db30bb7f: entering follower state: follower="Node at c14a7aa7-240d-742f-6ff7-cea5db30bb7f [Follower]" leader-address= leader-id=
2025-01-28T14:54:10.395Z [INFO]  server-bed2a870-29c5-4ddb-d2d9-36c311fa1a1b: initial configuration: index=0 servers=[]
2025-01-28T14:54:10.395Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:10.395Z [INFO]  server-bed2a870-29c5-4ddb-d2d9-36c311fa1a1b: entering follower state: follower="Node at bed2a870-29c5-4ddb-d2d9-36c311fa1a1b [Follower]" leader-address= leader-id=
2025-01-28T14:54:10.474Z [WARN]  server-bed2a870-29c5-4ddb-d2d9-36c311fa1a1b: no known peers, aborting election
2025-01-28T14:54:10.486Z [WARN]  server-c14a7aa7-240d-742f-6ff7-cea5db30bb7f: no known peers, aborting election
2025-01-28T14:54:10.492Z [WARN]  server-183828d8-410c-72b5-37b5-b094d1b7c6fa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:10.492Z [INFO]  server-183828d8-410c-72b5-37b5-b094d1b7c6fa: entering candidate state: node="Node at 183828d8-410c-72b5-37b5-b094d1b7c6fa [Candidate]" term=2
2025-01-28T14:54:10.492Z [INFO]  server-183828d8-410c-72b5-37b5-b094d1b7c6fa: election won: term=2 tally=1
2025-01-28T14:54:10.493Z [INFO]  server-183828d8-410c-72b5-37b5-b094d1b7c6fa: entering leader state: leader="Node at 183828d8-410c-72b5-37b5-b094d1b7c6fa [Leader]"
2025-01-28T14:54:10.598Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:10.493109526 +0000 UTC m=+36.167599965 (1 nodes), 97.711361ms from start of poll, 99.609682ms from cluster start. Timeout at 2025-01-28 14:54:10.598116584 +0000 UTC m=+36.272606624, 105.006659ms after stability
2025-01-28T14:54:10.598Z [INFO]  server-183828d8-410c-72b5-37b5-b094d1b7c6fa: updating configuration: command=AddVoter server-id=server-c14a7aa7-240d-742f-6ff7-cea5db30bb7f server-addr=c14a7aa7-240d-742f-6ff7-cea5db30bb7f servers="[{Suffrage:Voter ID:server-183828d8-410c-72b5-37b5-b094d1b7c6fa Address:183828d8-410c-72b5-37b5-b094d1b7c6fa} {Suffrage:Voter ID:server-c14a7aa7-240d-742f-6ff7-cea5db30bb7f Address:c14a7aa7-240d-742f-6ff7-cea5db30bb7f}]"
2025-01-28T14:54:10.598Z [INFO]  server-183828d8-410c-72b5-37b5-b094d1b7c6fa: added peer, starting replication: peer=server-c14a7aa7-240d-742f-6ff7-cea5db30bb7f
2025-01-28T14:54:10.598Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:10.599Z [WARN]  server-c14a7aa7-240d-742f-6ff7-cea5db30bb7f: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2025-01-28T14:54:10.599Z [WARN]  server-183828d8-410c-72b5-37b5-b094d1b7c6fa: appendEntries rejected, sending older logs: peer="{Voter server-c14a7aa7-240d-742f-6ff7-cea5db30bb7f c14a7aa7-240d-742f-6ff7-cea5db30bb7f}" next=1
2025-01-28T14:54:10.599Z [INFO]  server-183828d8-410c-72b5-37b5-b094d1b7c6fa: pipelining replication: peer="{Voter server-c14a7aa7-240d-742f-6ff7-cea5db30bb7f c14a7aa7-240d-742f-6ff7-cea5db30bb7f}"
2025-01-28T14:54:10.704Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:10.599008855 +0000 UTC m=+36.273498815 (1 nodes), 2.54µs from start of poll, 205.508532ms from cluster start. Timeout at 2025-01-28 14:54:10.704032953 +0000 UTC m=+36.378522933, 105.024118ms after stability
2025-01-28T14:54:10.704Z [INFO]  server-183828d8-410c-72b5-37b5-b094d1b7c6fa: updating configuration: command=AddVoter server-id=server-bed2a870-29c5-4ddb-d2d9-36c311fa1a1b server-addr=bed2a870-29c5-4ddb-d2d9-36c311fa1a1b servers="[{Suffrage:Voter ID:server-183828d8-410c-72b5-37b5-b094d1b7c6fa Address:183828d8-410c-72b5-37b5-b094d1b7c6fa} {Suffrage:Voter ID:server-c14a7aa7-240d-742f-6ff7-cea5db30bb7f Address:c14a7aa7-240d-742f-6ff7-cea5db30bb7f} {Suffrage:Voter ID:server-bed2a870-29c5-4ddb-d2d9-36c311fa1a1b Address:bed2a870-29c5-4ddb-d2d9-36c311fa1a1b}]"
2025-01-28T14:54:10.704Z [INFO]  server-183828d8-410c-72b5-37b5-b094d1b7c6fa: added peer, starting replication: peer=server-bed2a870-29c5-4ddb-d2d9-36c311fa1a1b
2025-01-28T14:54:10.704Z [WARN]  server-bed2a870-29c5-4ddb-d2d9-36c311fa1a1b: failed to get previous log: previous-index=4 last-index=0 error="log not found"
2025-01-28T14:54:10.704Z [WARN]  server-183828d8-410c-72b5-37b5-b094d1b7c6fa: appendEntries rejected, sending older logs: peer="{Voter server-bed2a870-29c5-4ddb-d2d9-36c311fa1a1b bed2a870-29c5-4ddb-d2d9-36c311fa1a1b}" next=1
2025-01-28T14:54:10.705Z [INFO]  server-183828d8-410c-72b5-37b5-b094d1b7c6fa: pipelining replication: peer="{Voter server-bed2a870-29c5-4ddb-d2d9-36c311fa1a1b bed2a870-29c5-4ddb-d2d9-36c311fa1a1b}"
2025-01-28T14:54:10.705Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:10.810Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:10.705254292 +0000 UTC m=+36.379744272 (1 nodes), 2.02µs from start of poll, 311.753989ms from cluster start. Timeout at 2025-01-28 14:54:10.81027775 +0000 UTC m=+36.484767890, 105.023618ms after stability
2025-01-28T14:54:10.810Z [INFO]  server-183828d8-410c-72b5-37b5-b094d1b7c6fa: aborting pipeline replication: peer="{Voter server-bed2a870-29c5-4ddb-d2d9-36c311fa1a1b bed2a870-29c5-4ddb-d2d9-36c311fa1a1b}"
2025-01-28T14:54:10.810Z [INFO]  server-183828d8-410c-72b5-37b5-b094d1b7c6fa: aborting pipeline replication: peer="{Voter server-c14a7aa7-240d-742f-6ff7-cea5db30bb7f c14a7aa7-240d-742f-6ff7-cea5db30bb7f}"
--- PASS: TestRaft_JoinNode_ConfigStore (0.42s)
=== RUN   TestRaft_RemoveFollower
2025-01-28T14:54:10.816Z [INFO]  server-12f0ae44-c32d-27c7-ddc8-30c3f3b876c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12f0ae44-c32d-27c7-ddc8-30c3f3b876c5 Address:12f0ae44-c32d-27c7-ddc8-30c3f3b876c5} {Suffrage:Voter ID:server-59c54d84-49d3-c527-e7ed-304a18561415 Address:59c54d84-49d3-c527-e7ed-304a18561415} {Suffrage:Voter ID:server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df Address:18f60f93-92a5-bd66-fbd4-a4e4fe3a30df}]"
2025-01-28T14:54:10.817Z [INFO]  server-12f0ae44-c32d-27c7-ddc8-30c3f3b876c5: entering follower state: follower="Node at 12f0ae44-c32d-27c7-ddc8-30c3f3b876c5 [Follower]" leader-address= leader-id=
2025-01-28T14:54:10.817Z [INFO]  server-59c54d84-49d3-c527-e7ed-304a18561415: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12f0ae44-c32d-27c7-ddc8-30c3f3b876c5 Address:12f0ae44-c32d-27c7-ddc8-30c3f3b876c5} {Suffrage:Voter ID:server-59c54d84-49d3-c527-e7ed-304a18561415 Address:59c54d84-49d3-c527-e7ed-304a18561415} {Suffrage:Voter ID:server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df Address:18f60f93-92a5-bd66-fbd4-a4e4fe3a30df}]"
2025-01-28T14:54:10.817Z [INFO]  server-59c54d84-49d3-c527-e7ed-304a18561415: entering follower state: follower="Node at 59c54d84-49d3-c527-e7ed-304a18561415 [Follower]" leader-address= leader-id=
2025-01-28T14:54:10.817Z [INFO]  server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12f0ae44-c32d-27c7-ddc8-30c3f3b876c5 Address:12f0ae44-c32d-27c7-ddc8-30c3f3b876c5} {Suffrage:Voter ID:server-59c54d84-49d3-c527-e7ed-304a18561415 Address:59c54d84-49d3-c527-e7ed-304a18561415} {Suffrage:Voter ID:server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df Address:18f60f93-92a5-bd66-fbd4-a4e4fe3a30df}]"
2025-01-28T14:54:10.817Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:10.817Z [INFO]  server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df: entering follower state: follower="Node at 18f60f93-92a5-bd66-fbd4-a4e4fe3a30df [Follower]" leader-address= leader-id=
2025-01-28T14:54:10.872Z [WARN]  server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:10.872Z [INFO]  server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df: entering candidate state: node="Node at 18f60f93-92a5-bd66-fbd4-a4e4fe3a30df [Candidate]" term=2
2025-01-28T14:54:10.872Z [INFO]  server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df: election won: term=2 tally=2
2025-01-28T14:54:10.872Z [INFO]  server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df: entering leader state: leader="Node at 18f60f93-92a5-bd66-fbd4-a4e4fe3a30df [Leader]"
2025-01-28T14:54:10.873Z [INFO]  server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df: added peer, starting replication: peer=server-12f0ae44-c32d-27c7-ddc8-30c3f3b876c5
2025-01-28T14:54:10.873Z [INFO]  server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df: added peer, starting replication: peer=server-59c54d84-49d3-c527-e7ed-304a18561415
2025-01-28T14:54:10.873Z [INFO]  server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df: pipelining replication: peer="{Voter server-12f0ae44-c32d-27c7-ddc8-30c3f3b876c5 12f0ae44-c32d-27c7-ddc8-30c3f3b876c5}"
2025-01-28T14:54:10.873Z [INFO]  server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df: pipelining replication: peer="{Voter server-59c54d84-49d3-c527-e7ed-304a18561415 59c54d84-49d3-c527-e7ed-304a18561415}"
2025-01-28T14:54:10.978Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:10.872984828 +0000 UTC m=+36.547474808 (1 nodes), 55.201692ms from start of poll, 56.470572ms from cluster start. Timeout at 2025-01-28 14:54:10.977991708 +0000 UTC m=+36.652481768, 105.00696ms after stability
2025-01-28T14:54:10.983Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:11.089Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:10.983770777 +0000 UTC m=+36.658260757 (2 nodes), 3.1µs from start of poll, 167.256521ms from cluster start. Timeout at 2025-01-28 14:54:11.088801574 +0000 UTC m=+36.763291614, 105.030857ms after stability
2025-01-28T14:54:11.089Z [INFO]  server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df: updating configuration: command=RemoveServer server-id=server-12f0ae44-c32d-27c7-ddc8-30c3f3b876c5 server-addr= servers="[{Suffrage:Voter ID:server-59c54d84-49d3-c527-e7ed-304a18561415 Address:59c54d84-49d3-c527-e7ed-304a18561415} {Suffrage:Voter ID:server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df Address:18f60f93-92a5-bd66-fbd4-a4e4fe3a30df}]"
2025-01-28T14:54:11.089Z [INFO]  server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df: removed peer, stopping replication: peer=server-12f0ae44-c32d-27c7-ddc8-30c3f3b876c5 last-index=3
2025-01-28T14:54:11.089Z [INFO]  server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df: aborting pipeline replication: peer="{Voter server-12f0ae44-c32d-27c7-ddc8-30c3f3b876c5 12f0ae44-c32d-27c7-ddc8-30c3f3b876c5}"
2025-01-28T14:54:11.195Z [INFO]  server-18f60f93-92a5-bd66-fbd4-a4e4fe3a30df: aborting pipeline replication: peer="{Voter server-59c54d84-49d3-c527-e7ed-304a18561415 59c54d84-49d3-c527-e7ed-304a18561415}"
--- PASS: TestRaft_RemoveFollower (0.38s)
=== RUN   TestRaft_RemoveLeader
2025-01-28T14:54:11.199Z [INFO]  server-05ce62bb-e9c2-814a-bfa7-4004b680a037: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05ce62bb-e9c2-814a-bfa7-4004b680a037 Address:05ce62bb-e9c2-814a-bfa7-4004b680a037} {Suffrage:Voter ID:server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7 Address:87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7} {Suffrage:Voter ID:server-d74b3a6a-b3b2-3935-bc09-493fe8e760a6 Address:d74b3a6a-b3b2-3935-bc09-493fe8e760a6}]"
2025-01-28T14:54:11.199Z [INFO]  server-05ce62bb-e9c2-814a-bfa7-4004b680a037: entering follower state: follower="Node at 05ce62bb-e9c2-814a-bfa7-4004b680a037 [Follower]" leader-address= leader-id=
2025-01-28T14:54:11.199Z [INFO]  server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05ce62bb-e9c2-814a-bfa7-4004b680a037 Address:05ce62bb-e9c2-814a-bfa7-4004b680a037} {Suffrage:Voter ID:server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7 Address:87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7} {Suffrage:Voter ID:server-d74b3a6a-b3b2-3935-bc09-493fe8e760a6 Address:d74b3a6a-b3b2-3935-bc09-493fe8e760a6}]"
2025-01-28T14:54:11.199Z [INFO]  server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7: entering follower state: follower="Node at 87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7 [Follower]" leader-address= leader-id=
2025-01-28T14:54:11.199Z [INFO]  server-d74b3a6a-b3b2-3935-bc09-493fe8e760a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-05ce62bb-e9c2-814a-bfa7-4004b680a037 Address:05ce62bb-e9c2-814a-bfa7-4004b680a037} {Suffrage:Voter ID:server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7 Address:87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7} {Suffrage:Voter ID:server-d74b3a6a-b3b2-3935-bc09-493fe8e760a6 Address:d74b3a6a-b3b2-3935-bc09-493fe8e760a6}]"
2025-01-28T14:54:11.200Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:11.200Z [INFO]  server-d74b3a6a-b3b2-3935-bc09-493fe8e760a6: entering follower state: follower="Node at d74b3a6a-b3b2-3935-bc09-493fe8e760a6 [Follower]" leader-address= leader-id=
2025-01-28T14:54:11.250Z [WARN]  server-05ce62bb-e9c2-814a-bfa7-4004b680a037: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:11.250Z [INFO]  server-05ce62bb-e9c2-814a-bfa7-4004b680a037: entering candidate state: node="Node at 05ce62bb-e9c2-814a-bfa7-4004b680a037 [Candidate]" term=2
2025-01-28T14:54:11.250Z [INFO]  server-05ce62bb-e9c2-814a-bfa7-4004b680a037: election won: term=2 tally=2
2025-01-28T14:54:11.250Z [INFO]  server-05ce62bb-e9c2-814a-bfa7-4004b680a037: entering leader state: leader="Node at 05ce62bb-e9c2-814a-bfa7-4004b680a037 [Leader]"
2025-01-28T14:54:11.251Z [INFO]  server-05ce62bb-e9c2-814a-bfa7-4004b680a037: added peer, starting replication: peer=server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7
2025-01-28T14:54:11.251Z [INFO]  server-05ce62bb-e9c2-814a-bfa7-4004b680a037: added peer, starting replication: peer=server-d74b3a6a-b3b2-3935-bc09-493fe8e760a6
2025-01-28T14:54:11.251Z [INFO]  server-05ce62bb-e9c2-814a-bfa7-4004b680a037: pipelining replication: peer="{Voter server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7 87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7}"
2025-01-28T14:54:11.251Z [INFO]  server-05ce62bb-e9c2-814a-bfa7-4004b680a037: pipelining replication: peer="{Voter server-d74b3a6a-b3b2-3935-bc09-493fe8e760a6 d74b3a6a-b3b2-3935-bc09-493fe8e760a6}"
2025-01-28T14:54:11.357Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:11.251040723 +0000 UTC m=+36.925530683 (1 nodes), 50.963792ms from start of poll, 52.098865ms from cluster start. Timeout at 2025-01-28 14:54:11.356044362 +0000 UTC m=+37.030534382, 105.003699ms after stability
2025-01-28T14:54:11.362Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:11.469Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:11.362926768 +0000 UTC m=+37.037416748 (2 nodes), 2.86µs from start of poll, 163.98493ms from cluster start. Timeout at 2025-01-28 14:54:11.467952985 +0000 UTC m=+37.142443025, 105.026277ms after stability
2025-01-28T14:54:11.469Z [INFO]  server-05ce62bb-e9c2-814a-bfa7-4004b680a037: updating configuration: command=RemoveServer server-id=server-05ce62bb-e9c2-814a-bfa7-4004b680a037 server-addr= servers="[{Suffrage:Voter ID:server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7 Address:87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7} {Suffrage:Voter ID:server-d74b3a6a-b3b2-3935-bc09-493fe8e760a6 Address:d74b3a6a-b3b2-3935-bc09-493fe8e760a6}]"
2025-01-28T14:54:11.469Z [INFO]  server-05ce62bb-e9c2-814a-bfa7-4004b680a037: removed ourself, shutting down
2025-01-28T14:54:11.469Z [INFO]  server-05ce62bb-e9c2-814a-bfa7-4004b680a037: aborting pipeline replication: peer="{Voter server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7 87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7}"
2025-01-28T14:54:11.469Z [INFO]  server-05ce62bb-e9c2-814a-bfa7-4004b680a037: aborting pipeline replication: peer="{Voter server-d74b3a6a-b3b2-3935-bc09-493fe8e760a6 d74b3a6a-b3b2-3935-bc09-493fe8e760a6}"
2025-01-28T14:54:11.553Z [WARN]  server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7: heartbeat timeout reached, starting election: last-leader-addr=05ce62bb-e9c2-814a-bfa7-4004b680a037 last-leader-id=server-05ce62bb-e9c2-814a-bfa7-4004b680a037
2025-01-28T14:54:11.553Z [INFO]  server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7: entering candidate state: node="Node at 87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7 [Candidate]" term=3
2025-01-28T14:54:11.553Z [WARN]  server-d74b3a6a-b3b2-3935-bc09-493fe8e760a6: rejecting vote request since we have a leader: from=87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7 leader=05ce62bb-e9c2-814a-bfa7-4004b680a037 leader-id=server-05ce62bb-e9c2-814a-bfa7-4004b680a037
2025-01-28T14:54:11.585Z [WARN]  server-d74b3a6a-b3b2-3935-bc09-493fe8e760a6: heartbeat timeout reached, starting election: last-leader-addr=05ce62bb-e9c2-814a-bfa7-4004b680a037 last-leader-id=server-05ce62bb-e9c2-814a-bfa7-4004b680a037
2025-01-28T14:54:11.585Z [INFO]  server-d74b3a6a-b3b2-3935-bc09-493fe8e760a6: entering candidate state: node="Node at d74b3a6a-b3b2-3935-bc09-493fe8e760a6 [Candidate]" term=3
2025-01-28T14:54:11.585Z [INFO]  server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7: duplicate requestVote for same term: term=3
2025-01-28T14:54:11.618Z [WARN]  server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7: Election timeout reached, restarting election
2025-01-28T14:54:11.618Z [INFO]  server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7: entering candidate state: node="Node at 87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7 [Candidate]" term=4
2025-01-28T14:54:11.618Z [INFO]  server-d74b3a6a-b3b2-3935-bc09-493fe8e760a6: entering follower state: follower="Node at d74b3a6a-b3b2-3935-bc09-493fe8e760a6 [Follower]" leader-address= leader-id=
2025-01-28T14:54:11.618Z [INFO]  server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7: election won: term=4 tally=2
2025-01-28T14:54:11.618Z [INFO]  server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7: entering leader state: leader="Node at 87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7 [Leader]"
2025-01-28T14:54:11.618Z [INFO]  server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7: added peer, starting replication: peer=server-d74b3a6a-b3b2-3935-bc09-493fe8e760a6
2025-01-28T14:54:11.619Z [INFO]  server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7: pipelining replication: peer="{Voter server-d74b3a6a-b3b2-3935-bc09-493fe8e760a6 d74b3a6a-b3b2-3935-bc09-493fe8e760a6}"
2025-01-28T14:54:11.681Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:11.786Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:11.681293493 +0000 UTC m=+37.355783473 (1 nodes), 152.766µs from start of poll, 482.351655ms from cluster start. Timeout at 2025-01-28 14:54:11.786396063 +0000 UTC m=+37.460886103, 105.10263ms after stability
2025-01-28T14:54:11.786Z [INFO]  server-87e7f958-5e84-1f2c-4bf8-e7f6dc3784e7: aborting pipeline replication: peer="{Voter server-d74b3a6a-b3b2-3935-bc09-493fe8e760a6 d74b3a6a-b3b2-3935-bc09-493fe8e760a6}"
--- PASS: TestRaft_RemoveLeader (0.59s)
=== RUN   TestRaft_RemoveLeader_NoShutdown
2025-01-28T14:54:11.790Z [INFO]  server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c Address:0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c} {Suffrage:Voter ID:server-05fc388a-219d-e8b2-8b93-f540449453f4 Address:05fc388a-219d-e8b2-8b93-f540449453f4} {Suffrage:Voter ID:server-4c6e348c-2d58-77df-adde-6f86d27fa231 Address:4c6e348c-2d58-77df-adde-6f86d27fa231}]"
2025-01-28T14:54:11.790Z [INFO]  server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c: entering follower state: follower="Node at 0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c [Follower]" leader-address= leader-id=
2025-01-28T14:54:11.790Z [INFO]  server-05fc388a-219d-e8b2-8b93-f540449453f4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c Address:0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c} {Suffrage:Voter ID:server-05fc388a-219d-e8b2-8b93-f540449453f4 Address:05fc388a-219d-e8b2-8b93-f540449453f4} {Suffrage:Voter ID:server-4c6e348c-2d58-77df-adde-6f86d27fa231 Address:4c6e348c-2d58-77df-adde-6f86d27fa231}]"
2025-01-28T14:54:11.790Z [INFO]  server-05fc388a-219d-e8b2-8b93-f540449453f4: entering follower state: follower="Node at 05fc388a-219d-e8b2-8b93-f540449453f4 [Follower]" leader-address= leader-id=
2025-01-28T14:54:11.791Z [INFO]  server-4c6e348c-2d58-77df-adde-6f86d27fa231: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c Address:0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c} {Suffrage:Voter ID:server-05fc388a-219d-e8b2-8b93-f540449453f4 Address:05fc388a-219d-e8b2-8b93-f540449453f4} {Suffrage:Voter ID:server-4c6e348c-2d58-77df-adde-6f86d27fa231 Address:4c6e348c-2d58-77df-adde-6f86d27fa231}]"
2025-01-28T14:54:11.791Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:11.791Z [INFO]  server-4c6e348c-2d58-77df-adde-6f86d27fa231: entering follower state: follower="Node at 4c6e348c-2d58-77df-adde-6f86d27fa231 [Follower]" leader-address= leader-id=
2025-01-28T14:54:11.859Z [WARN]  server-05fc388a-219d-e8b2-8b93-f540449453f4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:11.859Z [INFO]  server-05fc388a-219d-e8b2-8b93-f540449453f4: entering candidate state: node="Node at 05fc388a-219d-e8b2-8b93-f540449453f4 [Candidate]" term=2
2025-01-28T14:54:11.859Z [INFO]  server-05fc388a-219d-e8b2-8b93-f540449453f4: election won: term=2 tally=2
2025-01-28T14:54:11.859Z [INFO]  server-05fc388a-219d-e8b2-8b93-f540449453f4: entering leader state: leader="Node at 05fc388a-219d-e8b2-8b93-f540449453f4 [Leader]"
2025-01-28T14:54:11.859Z [INFO]  server-05fc388a-219d-e8b2-8b93-f540449453f4: added peer, starting replication: peer=server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c
2025-01-28T14:54:11.859Z [INFO]  server-05fc388a-219d-e8b2-8b93-f540449453f4: added peer, starting replication: peer=server-4c6e348c-2d58-77df-adde-6f86d27fa231
2025-01-28T14:54:11.860Z [INFO]  server-05fc388a-219d-e8b2-8b93-f540449453f4: pipelining replication: peer="{Voter server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c 0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c}"
2025-01-28T14:54:11.860Z [INFO]  server-05fc388a-219d-e8b2-8b93-f540449453f4: pipelining replication: peer="{Voter server-4c6e348c-2d58-77df-adde-6f86d27fa231 4c6e348c-2d58-77df-adde-6f86d27fa231}"
2025-01-28T14:54:11.965Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:11.85982807 +0000 UTC m=+37.534318050 (2 nodes), 68.629645ms from start of poll, 69.697145ms from cluster start. Timeout at 2025-01-28 14:54:11.964833049 +0000 UTC m=+37.639323089, 105.005039ms after stability
2025-01-28T14:54:11.965Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:12.070Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:11.965812072 +0000 UTC m=+37.640302072 (1 nodes), 2.46µs from start of poll, 175.681167ms from cluster start. Timeout at 2025-01-28 14:54:12.07083607 +0000 UTC m=+37.745326070, 105.023998ms after stability
2025-01-28T14:54:12.077Z [INFO]  server-05fc388a-219d-e8b2-8b93-f540449453f4: updating configuration: command=RemoveServer server-id=server-05fc388a-219d-e8b2-8b93-f540449453f4 server-addr= servers="[{Suffrage:Voter ID:server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c Address:0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c} {Suffrage:Voter ID:server-4c6e348c-2d58-77df-adde-6f86d27fa231 Address:4c6e348c-2d58-77df-adde-6f86d27fa231}]"
2025-01-28T14:54:12.077Z [INFO]  server-05fc388a-219d-e8b2-8b93-f540449453f4: removed ourself, transitioning to follower
2025-01-28T14:54:12.077Z [INFO]  server-05fc388a-219d-e8b2-8b93-f540449453f4: entering follower state: follower="Node at 05fc388a-219d-e8b2-8b93-f540449453f4 [Follower]" leader-address= leader-id=
2025-01-28T14:54:12.077Z [INFO]  server-05fc388a-219d-e8b2-8b93-f540449453f4: aborting pipeline replication: peer="{Voter server-4c6e348c-2d58-77df-adde-6f86d27fa231 4c6e348c-2d58-77df-adde-6f86d27fa231}"
2025-01-28T14:54:12.077Z [INFO]  server-05fc388a-219d-e8b2-8b93-f540449453f4: aborting pipeline replication: peer="{Voter server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c 0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c}"
2025-01-28T14:54:12.141Z [WARN]  server-05fc388a-219d-e8b2-8b93-f540449453f4: not part of stable configuration, aborting election
2025-01-28T14:54:12.152Z [WARN]  server-4c6e348c-2d58-77df-adde-6f86d27fa231: heartbeat timeout reached, starting election: last-leader-addr=05fc388a-219d-e8b2-8b93-f540449453f4 last-leader-id=server-05fc388a-219d-e8b2-8b93-f540449453f4
2025-01-28T14:54:12.152Z [INFO]  server-4c6e348c-2d58-77df-adde-6f86d27fa231: entering candidate state: node="Node at 4c6e348c-2d58-77df-adde-6f86d27fa231 [Candidate]" term=3
2025-01-28T14:54:12.152Z [WARN]  server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c: rejecting vote request since we have a leader: from=4c6e348c-2d58-77df-adde-6f86d27fa231 leader=05fc388a-219d-e8b2-8b93-f540449453f4 leader-id=server-05fc388a-219d-e8b2-8b93-f540449453f4
2025-01-28T14:54:12.180Z [WARN]  server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c: heartbeat timeout reached, starting election: last-leader-addr=05fc388a-219d-e8b2-8b93-f540449453f4 last-leader-id=server-05fc388a-219d-e8b2-8b93-f540449453f4
2025-01-28T14:54:12.180Z [INFO]  server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c: entering candidate state: node="Node at 0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c [Candidate]" term=3
2025-01-28T14:54:12.181Z [INFO]  server-4c6e348c-2d58-77df-adde-6f86d27fa231: duplicate requestVote for same term: term=3
2025-01-28T14:54:12.184Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:12.238Z [WARN]  server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c: Election timeout reached, restarting election
2025-01-28T14:54:12.238Z [INFO]  server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c: entering candidate state: node="Node at 0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c [Candidate]" term=4
2025-01-28T14:54:12.238Z [INFO]  server-4c6e348c-2d58-77df-adde-6f86d27fa231: entering follower state: follower="Node at 4c6e348c-2d58-77df-adde-6f86d27fa231 [Follower]" leader-address= leader-id=
2025-01-28T14:54:12.239Z [INFO]  server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c: election won: term=4 tally=2
2025-01-28T14:54:12.239Z [INFO]  server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c: entering leader state: leader="Node at 0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c [Leader]"
2025-01-28T14:54:12.239Z [INFO]  server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c: added peer, starting replication: peer=server-4c6e348c-2d58-77df-adde-6f86d27fa231
2025-01-28T14:54:12.239Z [INFO]  server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c: pipelining replication: peer="{Voter server-4c6e348c-2d58-77df-adde-6f86d27fa231 4c6e348c-2d58-77df-adde-6f86d27fa231}"
2025-01-28T14:54:12.345Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:12.239061574 +0000 UTC m=+37.913551534 (1 nodes), 54.925039ms from start of poll, 448.930629ms from cluster start. Timeout at 2025-01-28 14:54:12.344068994 +0000 UTC m=+38.018559034, 105.0075ms after stability
2025-01-28T14:54:12.450Z [INFO]  server-0c484a7d-1415-40ba-fe0c-dcafa3dc2e1c: aborting pipeline replication: peer="{Voter server-4c6e348c-2d58-77df-adde-6f86d27fa231 4c6e348c-2d58-77df-adde-6f86d27fa231}"
--- PASS: TestRaft_RemoveLeader_NoShutdown (0.66s)
=== RUN   TestRaft_RemoveFollower_SplitCluster
2025-01-28T14:54:12.456Z [INFO]  server-42825a80-bd55-5310-334b-d02866c2b71a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42825a80-bd55-5310-334b-d02866c2b71a Address:42825a80-bd55-5310-334b-d02866c2b71a} {Suffrage:Voter ID:server-e516185a-dd82-97e2-fdae-73975d6c1ed0 Address:e516185a-dd82-97e2-fdae-73975d6c1ed0} {Suffrage:Voter ID:server-e27b9610-e5d8-7241-0ebf-ec53c025a26e Address:e27b9610-e5d8-7241-0ebf-ec53c025a26e} {Suffrage:Voter ID:server-d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927 Address:d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927}]"
2025-01-28T14:54:12.456Z [INFO]  server-42825a80-bd55-5310-334b-d02866c2b71a: entering follower state: follower="Node at 42825a80-bd55-5310-334b-d02866c2b71a [Follower]" leader-address= leader-id=
2025-01-28T14:54:12.456Z [INFO]  server-e516185a-dd82-97e2-fdae-73975d6c1ed0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42825a80-bd55-5310-334b-d02866c2b71a Address:42825a80-bd55-5310-334b-d02866c2b71a} {Suffrage:Voter ID:server-e516185a-dd82-97e2-fdae-73975d6c1ed0 Address:e516185a-dd82-97e2-fdae-73975d6c1ed0} {Suffrage:Voter ID:server-e27b9610-e5d8-7241-0ebf-ec53c025a26e Address:e27b9610-e5d8-7241-0ebf-ec53c025a26e} {Suffrage:Voter ID:server-d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927 Address:d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927}]"
2025-01-28T14:54:12.456Z [INFO]  server-e27b9610-e5d8-7241-0ebf-ec53c025a26e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42825a80-bd55-5310-334b-d02866c2b71a Address:42825a80-bd55-5310-334b-d02866c2b71a} {Suffrage:Voter ID:server-e516185a-dd82-97e2-fdae-73975d6c1ed0 Address:e516185a-dd82-97e2-fdae-73975d6c1ed0} {Suffrage:Voter ID:server-e27b9610-e5d8-7241-0ebf-ec53c025a26e Address:e27b9610-e5d8-7241-0ebf-ec53c025a26e} {Suffrage:Voter ID:server-d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927 Address:d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927}]"
2025-01-28T14:54:12.457Z [INFO]  server-d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42825a80-bd55-5310-334b-d02866c2b71a Address:42825a80-bd55-5310-334b-d02866c2b71a} {Suffrage:Voter ID:server-e516185a-dd82-97e2-fdae-73975d6c1ed0 Address:e516185a-dd82-97e2-fdae-73975d6c1ed0} {Suffrage:Voter ID:server-e27b9610-e5d8-7241-0ebf-ec53c025a26e Address:e27b9610-e5d8-7241-0ebf-ec53c025a26e} {Suffrage:Voter ID:server-d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927 Address:d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927}]"
2025-01-28T14:54:12.457Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:12.457Z [INFO]  server-e516185a-dd82-97e2-fdae-73975d6c1ed0: entering follower state: follower="Node at e516185a-dd82-97e2-fdae-73975d6c1ed0 [Follower]" leader-address= leader-id=
2025-01-28T14:54:12.457Z [INFO]  server-e27b9610-e5d8-7241-0ebf-ec53c025a26e: entering follower state: follower="Node at e27b9610-e5d8-7241-0ebf-ec53c025a26e [Follower]" leader-address= leader-id=
2025-01-28T14:54:12.457Z [INFO]  server-d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927: entering follower state: follower="Node at d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927 [Follower]" leader-address= leader-id=
2025-01-28T14:54:12.515Z [WARN]  server-42825a80-bd55-5310-334b-d02866c2b71a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:12.515Z [INFO]  server-42825a80-bd55-5310-334b-d02866c2b71a: entering candidate state: node="Node at 42825a80-bd55-5310-334b-d02866c2b71a [Candidate]" term=2
2025-01-28T14:54:12.516Z [INFO]  server-42825a80-bd55-5310-334b-d02866c2b71a: election won: term=2 tally=3
2025-01-28T14:54:12.516Z [INFO]  server-42825a80-bd55-5310-334b-d02866c2b71a: entering leader state: leader="Node at 42825a80-bd55-5310-334b-d02866c2b71a [Leader]"
2025-01-28T14:54:12.516Z [INFO]  server-42825a80-bd55-5310-334b-d02866c2b71a: added peer, starting replication: peer=server-e516185a-dd82-97e2-fdae-73975d6c1ed0
2025-01-28T14:54:12.516Z [INFO]  server-42825a80-bd55-5310-334b-d02866c2b71a: added peer, starting replication: peer=server-e27b9610-e5d8-7241-0ebf-ec53c025a26e
2025-01-28T14:54:12.516Z [INFO]  server-42825a80-bd55-5310-334b-d02866c2b71a: added peer, starting replication: peer=server-d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927
2025-01-28T14:54:12.516Z [INFO]  server-42825a80-bd55-5310-334b-d02866c2b71a: pipelining replication: peer="{Voter server-e516185a-dd82-97e2-fdae-73975d6c1ed0 e516185a-dd82-97e2-fdae-73975d6c1ed0}"
2025-01-28T14:54:12.516Z [INFO]  server-42825a80-bd55-5310-334b-d02866c2b71a: pipelining replication: peer="{Voter server-e27b9610-e5d8-7241-0ebf-ec53c025a26e e27b9610-e5d8-7241-0ebf-ec53c025a26e}"
2025-01-28T14:54:12.516Z [INFO]  server-42825a80-bd55-5310-334b-d02866c2b71a: pipelining replication: peer="{Voter server-d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927 d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927}"
2025-01-28T14:54:12.621Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:12.516322237 +0000 UTC m=+38.190812217 (1 nodes), 59.163099ms from start of poll, 60.479894ms from cluster start. Timeout at 2025-01-28 14:54:12.621335176 +0000 UTC m=+38.295825156, 105.012939ms after stability
2025-01-28T14:54:12.727Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:12.832Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:12.727272536 +0000 UTC m=+38.401762516 (3 nodes), 2.9µs from start of poll, 271.430193ms from cluster start. Timeout at 2025-01-28 14:54:12.832321711 +0000 UTC m=+38.506811711, 105.049195ms after stability
2025-01-28T14:54:12.832Z [INFO]  server-42825a80-bd55-5310-334b-d02866c2b71a: updating configuration: command=RemoveServer server-id=server-d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927 server-addr= servers="[{Suffrage:Voter ID:server-42825a80-bd55-5310-334b-d02866c2b71a Address:42825a80-bd55-5310-334b-d02866c2b71a} {Suffrage:Voter ID:server-e516185a-dd82-97e2-fdae-73975d6c1ed0 Address:e516185a-dd82-97e2-fdae-73975d6c1ed0} {Suffrage:Voter ID:server-e27b9610-e5d8-7241-0ebf-ec53c025a26e Address:e27b9610-e5d8-7241-0ebf-ec53c025a26e}]"
2025-01-28T14:54:12.832Z [INFO]  server-42825a80-bd55-5310-334b-d02866c2b71a: removed peer, stopping replication: peer=server-d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927 last-index=3
2025-01-28T14:54:12.833Z [INFO]  server-42825a80-bd55-5310-334b-d02866c2b71a: aborting pipeline replication: peer="{Voter server-d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927 d44ee3a7-d67d-8f31-b14c-1bb4d7a5b927}"
2025-01-28T14:54:12.832Z [ERROR] server-42825a80-bd55-5310-334b-d02866c2b71a: failed to pipeline appendEntries: peer="{Voter server-e516185a-dd82-97e2-fdae-73975d6c1ed0 e516185a-dd82-97e2-fdae-73975d6c1ed0}" error="append pipeline closed"
2025-01-28T14:54:12.833Z [INFO]  server-42825a80-bd55-5310-334b-d02866c2b71a: aborting pipeline replication: peer="{Voter server-e516185a-dd82-97e2-fdae-73975d6c1ed0 e516185a-dd82-97e2-fdae-73975d6c1ed0}"
2025-01-28T14:54:12.833Z [ERROR] server-42825a80-bd55-5310-334b-d02866c2b71a: failed to pipeline appendEntries: peer="{Voter server-e27b9610-e5d8-7241-0ebf-ec53c025a26e e27b9610-e5d8-7241-0ebf-ec53c025a26e}" error="append pipeline closed"
2025-01-28T14:54:12.833Z [INFO]  server-42825a80-bd55-5310-334b-d02866c2b71a: aborting pipeline replication: peer="{Voter server-e27b9610-e5d8-7241-0ebf-ec53c025a26e e27b9610-e5d8-7241-0ebf-ec53c025a26e}"
2025-01-28T14:54:12.833Z [ERROR] server-42825a80-bd55-5310-334b-d02866c2b71a: failed to appendEntries to: peer="{Voter server-e516185a-dd82-97e2-fdae-73975d6c1ed0 e516185a-dd82-97e2-fdae-73975d6c1ed0}" error="failed to connect to peer: e516185a-dd82-97e2-fdae-73975d6c1ed0"
2025-01-28T14:54:12.837Z [ERROR] server-42825a80-bd55-5310-334b-d02866c2b71a: failed to heartbeat to: peer=e516185a-dd82-97e2-fdae-73975d6c1ed0 backoff time=10ms error="failed to connect to peer: e516185a-dd82-97e2-fdae-73975d6c1ed0"
2025-01-28T14:54:12.841Z [ERROR] server-42825a80-bd55-5310-334b-d02866c2b71a: failed to heartbeat to: peer=e27b9610-e5d8-7241-0ebf-ec53c025a26e backoff time=10ms error="failed to connect to peer: e27b9610-e5d8-7241-0ebf-ec53c025a26e"
2025-01-28T14:54:12.843Z [ERROR] server-42825a80-bd55-5310-334b-d02866c2b71a: failed to appendEntries to: peer="{Voter server-e27b9610-e5d8-7241-0ebf-ec53c025a26e e27b9610-e5d8-7241-0ebf-ec53c025a26e}" error="failed to connect to peer: e27b9610-e5d8-7241-0ebf-ec53c025a26e"
2025-01-28T14:54:12.849Z [ERROR] server-42825a80-bd55-5310-334b-d02866c2b71a: failed to appendEntries to: peer="{Voter server-e516185a-dd82-97e2-fdae-73975d6c1ed0 e516185a-dd82-97e2-fdae-73975d6c1ed0}" error="failed to connect to peer: e516185a-dd82-97e2-fdae-73975d6c1ed0"
2025-01-28T14:54:12.856Z [ERROR] server-42825a80-bd55-5310-334b-d02866c2b71a: failed to heartbeat to: peer=e516185a-dd82-97e2-fdae-73975d6c1ed0 backoff time=10ms error="failed to connect to peer: e516185a-dd82-97e2-fdae-73975d6c1ed0"
2025-01-28T14:54:12.859Z [ERROR] server-42825a80-bd55-5310-334b-d02866c2b71a: failed to heartbeat to: peer=e27b9610-e5d8-7241-0ebf-ec53c025a26e backoff time=10ms error="failed to connect to peer: e27b9610-e5d8-7241-0ebf-ec53c025a26e"
2025-01-28T14:54:12.861Z [ERROR] server-42825a80-bd55-5310-334b-d02866c2b71a: failed to appendEntries to: peer="{Voter server-e27b9610-e5d8-7241-0ebf-ec53c025a26e e27b9610-e5d8-7241-0ebf-ec53c025a26e}" error="failed to connect to peer: e27b9610-e5d8-7241-0ebf-ec53c025a26e"
2025-01-28T14:54:12.869Z [ERROR] server-42825a80-bd55-5310-334b-d02866c2b71a: failed to appendEntries to: peer="{Voter server-e516185a-dd82-97e2-fdae-73975d6c1ed0 e516185a-dd82-97e2-fdae-73975d6c1ed0}" error="failed to connect to peer: e516185a-dd82-97e2-fdae-73975d6c1ed0"
2025-01-28T14:54:12.876Z [ERROR] server-42825a80-bd55-5310-334b-d02866c2b71a: failed to heartbeat to: peer=e516185a-dd82-97e2-fdae-73975d6c1ed0 backoff time=10ms error="failed to connect to peer: e516185a-dd82-97e2-fdae-73975d6c1ed0"
2025-01-28T14:54:12.879Z [ERROR] server-42825a80-bd55-5310-334b-d02866c2b71a: failed to appendEntries to: peer="{Voter server-e27b9610-e5d8-7241-0ebf-ec53c025a26e e27b9610-e5d8-7241-0ebf-ec53c025a26e}" error="failed to connect to peer: e27b9610-e5d8-7241-0ebf-ec53c025a26e"
2025-01-28T14:54:12.879Z [ERROR] server-42825a80-bd55-5310-334b-d02866c2b71a: failed to heartbeat to: peer=e27b9610-e5d8-7241-0ebf-ec53c025a26e backoff time=10ms error="failed to connect to peer: e27b9610-e5d8-7241-0ebf-ec53c025a26e"
2025-01-28T14:54:12.879Z [WARN]  server-42825a80-bd55-5310-334b-d02866c2b71a: failed to contact: server-id=server-e516185a-dd82-97e2-fdae-73975d6c1ed0 time=50.759012ms
2025-01-28T14:54:12.890Z [WARN]  server-42825a80-bd55-5310-334b-d02866c2b71a: failed to contact: server-id=server-e516185a-dd82-97e2-fdae-73975d6c1ed0 time=61.382769ms
2025-01-28T14:54:12.890Z [WARN]  server-42825a80-bd55-5310-334b-d02866c2b71a: failed to contact: server-id=server-e27b9610-e5d8-7241-0ebf-ec53c025a26e time=57.962771ms
2025-01-28T14:54:12.890Z [WARN]  server-42825a80-bd55-5310-334b-d02866c2b71a: failed to contact quorum of nodes, stepping down
2025-01-28T14:54:12.890Z [INFO]  server-42825a80-bd55-5310-334b-d02866c2b71a: entering follower state: follower="Node at 42825a80-bd55-5310-334b-d02866c2b71a [Follower]" leader-address= leader-id=
2025-01-28T14:54:12.890Z [ERROR] server-42825a80-bd55-5310-334b-d02866c2b71a: failed to appendEntries to: peer="{Voter server-e516185a-dd82-97e2-fdae-73975d6c1ed0 e516185a-dd82-97e2-fdae-73975d6c1ed0}" error="failed to connect to peer: e516185a-dd82-97e2-fdae-73975d6c1ed0"
2025-01-28T14:54:12.890Z [ERROR] server-42825a80-bd55-5310-334b-d02866c2b71a: failed to appendEntries to: peer="{Voter server-e27b9610-e5d8-7241-0ebf-ec53c025a26e e27b9610-e5d8-7241-0ebf-ec53c025a26e}" error="failed to connect to peer: e27b9610-e5d8-7241-0ebf-ec53c025a26e"
--- PASS: TestRaft_RemoveFollower_SplitCluster (0.44s)
=== RUN   TestRaft_AddKnownPeer
2025-01-28T14:54:12.894Z [INFO]  server-cd3e7900-30db-3d90-8214-e7a0b136665e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd3e7900-30db-3d90-8214-e7a0b136665e Address:cd3e7900-30db-3d90-8214-e7a0b136665e} {Suffrage:Voter ID:server-457801f5-520b-ec19-26b0-b71b98e0a840 Address:457801f5-520b-ec19-26b0-b71b98e0a840} {Suffrage:Voter ID:server-486b8bd7-7d5d-b434-93d8-ab4c8a737ff8 Address:486b8bd7-7d5d-b434-93d8-ab4c8a737ff8}]"
2025-01-28T14:54:12.894Z [INFO]  server-cd3e7900-30db-3d90-8214-e7a0b136665e: entering follower state: follower="Node at cd3e7900-30db-3d90-8214-e7a0b136665e [Follower]" leader-address= leader-id=
2025-01-28T14:54:12.895Z [INFO]  server-457801f5-520b-ec19-26b0-b71b98e0a840: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd3e7900-30db-3d90-8214-e7a0b136665e Address:cd3e7900-30db-3d90-8214-e7a0b136665e} {Suffrage:Voter ID:server-457801f5-520b-ec19-26b0-b71b98e0a840 Address:457801f5-520b-ec19-26b0-b71b98e0a840} {Suffrage:Voter ID:server-486b8bd7-7d5d-b434-93d8-ab4c8a737ff8 Address:486b8bd7-7d5d-b434-93d8-ab4c8a737ff8}]"
2025-01-28T14:54:12.895Z [INFO]  server-457801f5-520b-ec19-26b0-b71b98e0a840: entering follower state: follower="Node at 457801f5-520b-ec19-26b0-b71b98e0a840 [Follower]" leader-address= leader-id=
2025-01-28T14:54:12.895Z [INFO]  server-486b8bd7-7d5d-b434-93d8-ab4c8a737ff8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd3e7900-30db-3d90-8214-e7a0b136665e Address:cd3e7900-30db-3d90-8214-e7a0b136665e} {Suffrage:Voter ID:server-457801f5-520b-ec19-26b0-b71b98e0a840 Address:457801f5-520b-ec19-26b0-b71b98e0a840} {Suffrage:Voter ID:server-486b8bd7-7d5d-b434-93d8-ab4c8a737ff8 Address:486b8bd7-7d5d-b434-93d8-ab4c8a737ff8}]"
2025-01-28T14:54:12.895Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:12.895Z [INFO]  server-486b8bd7-7d5d-b434-93d8-ab4c8a737ff8: entering follower state: follower="Node at 486b8bd7-7d5d-b434-93d8-ab4c8a737ff8 [Follower]" leader-address= leader-id=
2025-01-28T14:54:12.967Z [WARN]  server-457801f5-520b-ec19-26b0-b71b98e0a840: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:12.967Z [INFO]  server-457801f5-520b-ec19-26b0-b71b98e0a840: entering candidate state: node="Node at 457801f5-520b-ec19-26b0-b71b98e0a840 [Candidate]" term=2
2025-01-28T14:54:12.968Z [INFO]  server-457801f5-520b-ec19-26b0-b71b98e0a840: election won: term=2 tally=2
2025-01-28T14:54:12.968Z [INFO]  server-457801f5-520b-ec19-26b0-b71b98e0a840: entering leader state: leader="Node at 457801f5-520b-ec19-26b0-b71b98e0a840 [Leader]"
2025-01-28T14:54:12.968Z [INFO]  server-457801f5-520b-ec19-26b0-b71b98e0a840: added peer, starting replication: peer=server-cd3e7900-30db-3d90-8214-e7a0b136665e
2025-01-28T14:54:12.968Z [INFO]  server-457801f5-520b-ec19-26b0-b71b98e0a840: added peer, starting replication: peer=server-486b8bd7-7d5d-b434-93d8-ab4c8a737ff8
2025-01-28T14:54:12.968Z [INFO]  server-457801f5-520b-ec19-26b0-b71b98e0a840: pipelining replication: peer="{Voter server-cd3e7900-30db-3d90-8214-e7a0b136665e cd3e7900-30db-3d90-8214-e7a0b136665e}"
2025-01-28T14:54:12.968Z [INFO]  server-457801f5-520b-ec19-26b0-b71b98e0a840: pipelining replication: peer="{Voter server-486b8bd7-7d5d-b434-93d8-ab4c8a737ff8 486b8bd7-7d5d-b434-93d8-ab4c8a737ff8}"
2025-01-28T14:54:13.074Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:12.968327775 +0000 UTC m=+38.642817755 (1 nodes), 72.730979ms from start of poll, 73.832735ms from cluster start. Timeout at 2025-01-28 14:54:13.073332295 +0000 UTC m=+38.747822295, 105.00454ms after stability
2025-01-28T14:54:13.074Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:13.179Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:13.074316536 +0000 UTC m=+38.748806496 (2 nodes), 2.86µs from start of poll, 179.821476ms from cluster start. Timeout at 2025-01-28 14:54:13.179449464 +0000 UTC m=+38.853939444, 105.132948ms after stability
2025-01-28T14:54:13.179Z [INFO]  server-457801f5-520b-ec19-26b0-b71b98e0a840: updating configuration: command=AddVoter server-id=server-cd3e7900-30db-3d90-8214-e7a0b136665e server-addr=cd3e7900-30db-3d90-8214-e7a0b136665e servers="[{Suffrage:Voter ID:server-cd3e7900-30db-3d90-8214-e7a0b136665e Address:cd3e7900-30db-3d90-8214-e7a0b136665e} {Suffrage:Voter ID:server-457801f5-520b-ec19-26b0-b71b98e0a840 Address:457801f5-520b-ec19-26b0-b71b98e0a840} {Suffrage:Voter ID:server-486b8bd7-7d5d-b434-93d8-ab4c8a737ff8 Address:486b8bd7-7d5d-b434-93d8-ab4c8a737ff8}]"
2025-01-28T14:54:13.180Z [INFO]  server-457801f5-520b-ec19-26b0-b71b98e0a840: aborting pipeline replication: peer="{Voter server-cd3e7900-30db-3d90-8214-e7a0b136665e cd3e7900-30db-3d90-8214-e7a0b136665e}"
2025-01-28T14:54:13.180Z [INFO]  server-457801f5-520b-ec19-26b0-b71b98e0a840: aborting pipeline replication: peer="{Voter server-486b8bd7-7d5d-b434-93d8-ab4c8a737ff8 486b8bd7-7d5d-b434-93d8-ab4c8a737ff8}"
--- PASS: TestRaft_AddKnownPeer (0.29s)
=== RUN   TestRaft_RemoveUnknownPeer
2025-01-28T14:54:13.184Z [INFO]  server-f493538c-403e-5872-b349-66e762fa4c28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f493538c-403e-5872-b349-66e762fa4c28 Address:f493538c-403e-5872-b349-66e762fa4c28} {Suffrage:Voter ID:server-f1617c35-8568-b033-701c-5644b234f32d Address:f1617c35-8568-b033-701c-5644b234f32d} {Suffrage:Voter ID:server-8e3a5de3-a10f-45e1-ea67-0ed109937eda Address:8e3a5de3-a10f-45e1-ea67-0ed109937eda}]"
2025-01-28T14:54:13.184Z [INFO]  server-f493538c-403e-5872-b349-66e762fa4c28: entering follower state: follower="Node at f493538c-403e-5872-b349-66e762fa4c28 [Follower]" leader-address= leader-id=
2025-01-28T14:54:13.184Z [INFO]  server-f1617c35-8568-b033-701c-5644b234f32d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f493538c-403e-5872-b349-66e762fa4c28 Address:f493538c-403e-5872-b349-66e762fa4c28} {Suffrage:Voter ID:server-f1617c35-8568-b033-701c-5644b234f32d Address:f1617c35-8568-b033-701c-5644b234f32d} {Suffrage:Voter ID:server-8e3a5de3-a10f-45e1-ea67-0ed109937eda Address:8e3a5de3-a10f-45e1-ea67-0ed109937eda}]"
2025-01-28T14:54:13.185Z [INFO]  server-f1617c35-8568-b033-701c-5644b234f32d: entering follower state: follower="Node at f1617c35-8568-b033-701c-5644b234f32d [Follower]" leader-address= leader-id=
2025-01-28T14:54:13.185Z [INFO]  server-8e3a5de3-a10f-45e1-ea67-0ed109937eda: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f493538c-403e-5872-b349-66e762fa4c28 Address:f493538c-403e-5872-b349-66e762fa4c28} {Suffrage:Voter ID:server-f1617c35-8568-b033-701c-5644b234f32d Address:f1617c35-8568-b033-701c-5644b234f32d} {Suffrage:Voter ID:server-8e3a5de3-a10f-45e1-ea67-0ed109937eda Address:8e3a5de3-a10f-45e1-ea67-0ed109937eda}]"
2025-01-28T14:54:13.185Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:13.185Z [INFO]  server-8e3a5de3-a10f-45e1-ea67-0ed109937eda: entering follower state: follower="Node at 8e3a5de3-a10f-45e1-ea67-0ed109937eda [Follower]" leader-address= leader-id=
2025-01-28T14:54:13.253Z [WARN]  server-f1617c35-8568-b033-701c-5644b234f32d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:13.253Z [INFO]  server-f1617c35-8568-b033-701c-5644b234f32d: entering candidate state: node="Node at f1617c35-8568-b033-701c-5644b234f32d [Candidate]" term=2
2025-01-28T14:54:13.254Z [INFO]  server-f1617c35-8568-b033-701c-5644b234f32d: election won: term=2 tally=2
2025-01-28T14:54:13.254Z [INFO]  server-f1617c35-8568-b033-701c-5644b234f32d: entering leader state: leader="Node at f1617c35-8568-b033-701c-5644b234f32d [Leader]"
2025-01-28T14:54:13.254Z [INFO]  server-f1617c35-8568-b033-701c-5644b234f32d: added peer, starting replication: peer=server-f493538c-403e-5872-b349-66e762fa4c28
2025-01-28T14:54:13.254Z [INFO]  server-f1617c35-8568-b033-701c-5644b234f32d: added peer, starting replication: peer=server-8e3a5de3-a10f-45e1-ea67-0ed109937eda
2025-01-28T14:54:13.254Z [INFO]  server-f1617c35-8568-b033-701c-5644b234f32d: pipelining replication: peer="{Voter server-f493538c-403e-5872-b349-66e762fa4c28 f493538c-403e-5872-b349-66e762fa4c28}"
2025-01-28T14:54:13.254Z [INFO]  server-f1617c35-8568-b033-701c-5644b234f32d: pipelining replication: peer="{Voter server-8e3a5de3-a10f-45e1-ea67-0ed109937eda 8e3a5de3-a10f-45e1-ea67-0ed109937eda}"
2025-01-28T14:54:13.360Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:13.254300437 +0000 UTC m=+38.928790417 (1 nodes), 68.954715ms from start of poll, 70.028934ms from cluster start. Timeout at 2025-01-28 14:54:13.359308936 +0000 UTC m=+39.033798996, 105.008579ms after stability
2025-01-28T14:54:13.360Z [INFO]  server-f1617c35-8568-b033-701c-5644b234f32d: updating configuration: command=RemoveServer server-id=f348d836-aee0-ef8c-a33c-c2217a91085f server-addr= servers="[{Suffrage:Voter ID:server-f493538c-403e-5872-b349-66e762fa4c28 Address:f493538c-403e-5872-b349-66e762fa4c28} {Suffrage:Voter ID:server-f1617c35-8568-b033-701c-5644b234f32d Address:f1617c35-8568-b033-701c-5644b234f32d} {Suffrage:Voter ID:server-8e3a5de3-a10f-45e1-ea67-0ed109937eda Address:8e3a5de3-a10f-45e1-ea67-0ed109937eda}]"
2025-01-28T14:54:13.361Z [INFO]  server-f1617c35-8568-b033-701c-5644b234f32d: aborting pipeline replication: peer="{Voter server-8e3a5de3-a10f-45e1-ea67-0ed109937eda 8e3a5de3-a10f-45e1-ea67-0ed109937eda}"
2025-01-28T14:54:13.361Z [INFO]  server-f1617c35-8568-b033-701c-5644b234f32d: aborting pipeline replication: peer="{Voter server-f493538c-403e-5872-b349-66e762fa4c28 f493538c-403e-5872-b349-66e762fa4c28}"
--- PASS: TestRaft_RemoveUnknownPeer (0.18s)
=== RUN   TestRaft_SnapshotRestore
2025-01-28T14:54:13.363Z [INFO]  server-696bd993-5836-9d5f-7b58-3fcf8cf0b2f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-696bd993-5836-9d5f-7b58-3fcf8cf0b2f5 Address:696bd993-5836-9d5f-7b58-3fcf8cf0b2f5}]"
2025-01-28T14:54:13.364Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:13.364Z [INFO]  server-696bd993-5836-9d5f-7b58-3fcf8cf0b2f5: entering follower state: follower="Node at 696bd993-5836-9d5f-7b58-3fcf8cf0b2f5 [Follower]" leader-address= leader-id=
2025-01-28T14:54:13.429Z [WARN]  server-696bd993-5836-9d5f-7b58-3fcf8cf0b2f5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:13.429Z [INFO]  server-696bd993-5836-9d5f-7b58-3fcf8cf0b2f5: entering candidate state: node="Node at 696bd993-5836-9d5f-7b58-3fcf8cf0b2f5 [Candidate]" term=2
2025-01-28T14:54:13.429Z [INFO]  server-696bd993-5836-9d5f-7b58-3fcf8cf0b2f5: election won: term=2 tally=1
2025-01-28T14:54:13.429Z [INFO]  server-696bd993-5836-9d5f-7b58-3fcf8cf0b2f5: entering leader state: leader="Node at 696bd993-5836-9d5f-7b58-3fcf8cf0b2f5 [Leader]"
2025-01-28T14:54:13.535Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:13.429873413 +0000 UTC m=+39.104363393 (1 nodes), 65.788313ms from start of poll, 66.185136ms from cluster start. Timeout at 2025-01-28 14:54:13.534877733 +0000 UTC m=+39.209367753, 105.00436ms after stability
2025-01-28T14:54:13.543Z [INFO]  server-696bd993-5836-9d5f-7b58-3fcf8cf0b2f5: starting snapshot up to: index=102
2025-01-28T14:54:13.544Z [INFO]  creating new snapshot: path=/tmp/raft4231758816/snapshots/2-102-1738076053543.tmp
2025-01-28T14:54:13.545Z [INFO]  server-696bd993-5836-9d5f-7b58-3fcf8cf0b2f5: compacting logs: from=1 to=92
2025-01-28T14:54:13.545Z [INFO]  server-696bd993-5836-9d5f-7b58-3fcf8cf0b2f5: snapshot complete up to: index=102
2025-01-28T14:54:13.546Z [INFO]  server-696bd993-5836-9d5f-7b58-3fcf8cf0b2f5: starting restore from snapshot: id=2-102-1738076053543 last-index=102 last-term=2 size-in-bytes=693
2025-01-28T14:54:13.547Z [INFO]  server-696bd993-5836-9d5f-7b58-3fcf8cf0b2f5: snapshot restore progress: id=2-102-1738076053543 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-01-28T14:54:13.547Z [INFO]  server-696bd993-5836-9d5f-7b58-3fcf8cf0b2f5: restored from snapshot: id=2-102-1738076053543 last-index=102 last-term=2 size-in-bytes=693
2025-01-28T14:54:13.547Z [INFO]  server-696bd993-5836-9d5f-7b58-3fcf8cf0b2f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-696bd993-5836-9d5f-7b58-3fcf8cf0b2f5 Address:696bd993-5836-9d5f-7b58-3fcf8cf0b2f5}]"
--- PASS: TestRaft_SnapshotRestore (0.19s)
=== RUN   TestRaft_RestoreSnapshotOnStartup_Monotonic
2025-01-28T14:54:13.549Z [INFO]  server-e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9 Address:e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9}]"
2025-01-28T14:54:13.549Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:13.549Z [INFO]  server-e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9: entering follower state: follower="Node at e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9 [Follower]" leader-address= leader-id=
2025-01-28T14:54:13.621Z [WARN]  server-e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:13.621Z [INFO]  server-e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9: entering candidate state: node="Node at e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9 [Candidate]" term=2
2025-01-28T14:54:13.622Z [INFO]  server-e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9: election won: term=2 tally=1
2025-01-28T14:54:13.622Z [INFO]  server-e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9: entering leader state: leader="Node at e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9 [Leader]"
2025-01-28T14:54:13.727Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:13.622115237 +0000 UTC m=+39.296605237 (1 nodes), 72.593911ms from start of poll, 72.970076ms from cluster start. Timeout at 2025-01-28 14:54:13.727122316 +0000 UTC m=+39.401612436, 105.007199ms after stability
2025-01-28T14:54:13.733Z [INFO]  server-e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9: starting snapshot up to: index=102
2025-01-28T14:54:13.733Z [INFO]  creating new snapshot: path=/tmp/raft3532354257/snapshots/2-102-1738076053733.tmp
2025-01-28T14:54:13.735Z [INFO]  server-e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9: compacting logs: from=1 to=92
2025-01-28T14:54:13.736Z [INFO]  server-e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9: snapshot complete up to: index=102
2025-01-28T14:54:13.736Z [INFO]  server-e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9: starting restore from snapshot: id=2-102-1738076053733 last-index=102 last-term=2 size-in-bytes=693
2025-01-28T14:54:13.737Z [INFO]  server-e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9: snapshot restore progress: id=2-102-1738076053733 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-01-28T14:54:13.737Z [INFO]  server-e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9: restored from snapshot: id=2-102-1738076053733 last-index=102 last-term=2 size-in-bytes=693
2025-01-28T14:54:13.737Z [INFO]  server-e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9 Address:e0e2ca0f-500c-d72e-e3a0-e673bae1c0d9}]"
--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.19s)
=== RUN   TestRaft_SnapshotRestore_Progress
2025-01-28T14:54:13.740Z [INFO]  server-e1fa01d8-1b90-d514-f0c5-886c3913a8f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e1fa01d8-1b90-d514-f0c5-886c3913a8f0 Address:e1fa01d8-1b90-d514-f0c5-886c3913a8f0}]"
2025-01-28T14:54:13.740Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:13.740Z [INFO]  server-e1fa01d8-1b90-d514-f0c5-886c3913a8f0: entering follower state: follower="Node at e1fa01d8-1b90-d514-f0c5-886c3913a8f0 [Follower]" leader-address= leader-id=
2025-01-28T14:54:13.800Z [WARN]  server-e1fa01d8-1b90-d514-f0c5-886c3913a8f0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:13.801Z [INFO]  server-e1fa01d8-1b90-d514-f0c5-886c3913a8f0: entering candidate state: node="Node at e1fa01d8-1b90-d514-f0c5-886c3913a8f0 [Candidate]" term=2
2025-01-28T14:54:13.801Z [INFO]  server-e1fa01d8-1b90-d514-f0c5-886c3913a8f0: election won: term=2 tally=1
2025-01-28T14:54:13.801Z [INFO]  server-e1fa01d8-1b90-d514-f0c5-886c3913a8f0: entering leader state: leader="Node at e1fa01d8-1b90-d514-f0c5-886c3913a8f0 [Leader]"
2025-01-28T14:54:13.907Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:13.801256057 +0000 UTC m=+39.475746037 (1 nodes), 60.539209ms from start of poll, 60.941451ms from cluster start. Timeout at 2025-01-28 14:54:13.906264337 +0000 UTC m=+39.580754377, 105.00834ms after stability
2025-01-28T14:54:13.914Z [INFO]  server-e1fa01d8-1b90-d514-f0c5-886c3913a8f0: starting snapshot up to: index=102
2025-01-28T14:54:13.914Z [INFO]  creating new snapshot: path=/tmp/raft2116801761/snapshots/2-102-1738076053914.tmp
2025-01-28T14:54:13.916Z [INFO]  server-e1fa01d8-1b90-d514-f0c5-886c3913a8f0: compacting logs: from=1 to=92
2025-01-28T14:54:13.916Z [INFO]  server-e1fa01d8-1b90-d514-f0c5-886c3913a8f0: snapshot complete up to: index=102
--- PASS: TestRaft_SnapshotRestore_Progress (0.18s)
=== RUN   TestRaft_NoRestoreOnStart
2025-01-28T14:54:13.920Z [INFO]  server-835fea57-0626-ba4b-02c0-4957702425ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-835fea57-0626-ba4b-02c0-4957702425ed Address:835fea57-0626-ba4b-02c0-4957702425ed}]"
2025-01-28T14:54:13.920Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:13.920Z [INFO]  server-835fea57-0626-ba4b-02c0-4957702425ed: entering follower state: follower="Node at 835fea57-0626-ba4b-02c0-4957702425ed [Follower]" leader-address= leader-id=
2025-01-28T14:54:14.011Z [WARN]  server-835fea57-0626-ba4b-02c0-4957702425ed: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:14.012Z [INFO]  server-835fea57-0626-ba4b-02c0-4957702425ed: entering candidate state: node="Node at 835fea57-0626-ba4b-02c0-4957702425ed [Candidate]" term=2
2025-01-28T14:54:14.012Z [INFO]  server-835fea57-0626-ba4b-02c0-4957702425ed: election won: term=2 tally=1
2025-01-28T14:54:14.012Z [INFO]  server-835fea57-0626-ba4b-02c0-4957702425ed: entering leader state: leader="Node at 835fea57-0626-ba4b-02c0-4957702425ed [Leader]"
2025-01-28T14:54:14.118Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:14.012417556 +0000 UTC m=+39.686907556 (1 nodes), 91.89435ms from start of poll, 92.283694ms from cluster start. Timeout at 2025-01-28 14:54:14.117424175 +0000 UTC m=+39.791914155, 105.006599ms after stability
2025-01-28T14:54:14.124Z [INFO]  server-835fea57-0626-ba4b-02c0-4957702425ed: starting snapshot up to: index=102
2025-01-28T14:54:14.124Z [INFO]  creating new snapshot: path=/tmp/raft1326688817/snapshots/2-102-1738076054124.tmp
2025-01-28T14:54:14.126Z [INFO]  server-835fea57-0626-ba4b-02c0-4957702425ed: compacting logs: from=1 to=92
2025-01-28T14:54:14.126Z [INFO]  server-835fea57-0626-ba4b-02c0-4957702425ed: snapshot complete up to: index=102
2025-01-28T14:54:14.126Z [INFO]  server-835fea57-0626-ba4b-02c0-4957702425ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-835fea57-0626-ba4b-02c0-4957702425ed Address:835fea57-0626-ba4b-02c0-4957702425ed}]"
--- PASS: TestRaft_NoRestoreOnStart (0.21s)
=== RUN   TestRaft_SnapshotRestore_PeerChange
2025-01-28T14:54:14.126Z [INFO]  server-835fea57-0626-ba4b-02c0-4957702425ed: entering follower state: follower="Node at 835fea57-0626-ba4b-02c0-4957702425ed [Follower]" leader-address= leader-id=
    raft_test.go:1266: Skipping unreliable test
--- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
=== RUN   TestRaft_AutoSnapshot
2025-01-28T14:54:14.127Z [INFO]  server-54d22736-1d2a-770d-74e7-06ef75daf5fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54d22736-1d2a-770d-74e7-06ef75daf5fa Address:54d22736-1d2a-770d-74e7-06ef75daf5fa}]"
2025-01-28T14:54:14.127Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:14.127Z [INFO]  server-54d22736-1d2a-770d-74e7-06ef75daf5fa: entering follower state: follower="Node at 54d22736-1d2a-770d-74e7-06ef75daf5fa [Follower]" leader-address= leader-id=
2025-01-28T14:54:14.193Z [WARN]  server-54d22736-1d2a-770d-74e7-06ef75daf5fa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:14.193Z [INFO]  server-54d22736-1d2a-770d-74e7-06ef75daf5fa: entering candidate state: node="Node at 54d22736-1d2a-770d-74e7-06ef75daf5fa [Candidate]" term=2
2025-01-28T14:54:14.194Z [INFO]  server-54d22736-1d2a-770d-74e7-06ef75daf5fa: election won: term=2 tally=1
2025-01-28T14:54:14.194Z [INFO]  server-54d22736-1d2a-770d-74e7-06ef75daf5fa: entering leader state: leader="Node at 54d22736-1d2a-770d-74e7-06ef75daf5fa [Leader]"
2025-01-28T14:54:14.219Z [WARN]  server-835fea57-0626-ba4b-02c0-4957702425ed: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:14.219Z [INFO]  server-835fea57-0626-ba4b-02c0-4957702425ed: entering candidate state: node="Node at 835fea57-0626-ba4b-02c0-4957702425ed [Candidate]" term=3
2025-01-28T14:54:14.219Z [INFO]  server-835fea57-0626-ba4b-02c0-4957702425ed: election won: term=3 tally=1
2025-01-28T14:54:14.219Z [INFO]  server-835fea57-0626-ba4b-02c0-4957702425ed: entering leader state: leader="Node at 835fea57-0626-ba4b-02c0-4957702425ed [Leader]"
2025-01-28T14:54:14.299Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:14.194240203 +0000 UTC m=+39.868730183 (1 nodes), 66.295185ms from start of poll, 66.622035ms from cluster start. Timeout at 2025-01-28 14:54:14.299295737 +0000 UTC m=+39.973785797, 105.055614ms after stability
2025-01-28T14:54:14.315Z [INFO]  server-54d22736-1d2a-770d-74e7-06ef75daf5fa: starting snapshot up to: index=102
2025-01-28T14:54:14.315Z [INFO]  creating new snapshot: path=/tmp/raft4012878319/snapshots/2-102-1738076054315.tmp
2025-01-28T14:54:14.317Z [INFO]  server-54d22736-1d2a-770d-74e7-06ef75daf5fa: compacting logs: from=1 to=92
2025-01-28T14:54:14.317Z [INFO]  server-54d22736-1d2a-770d-74e7-06ef75daf5fa: snapshot complete up to: index=102
--- PASS: TestRaft_AutoSnapshot (0.29s)
=== RUN   TestRaft_UserSnapshot
2025-01-28T14:54:14.414Z [INFO]  server-c7464b7d-5c64-7fe9-2afc-668011747a55: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c7464b7d-5c64-7fe9-2afc-668011747a55 Address:c7464b7d-5c64-7fe9-2afc-668011747a55}]"
2025-01-28T14:54:14.414Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:14.414Z [INFO]  server-c7464b7d-5c64-7fe9-2afc-668011747a55: entering follower state: follower="Node at c7464b7d-5c64-7fe9-2afc-668011747a55 [Follower]" leader-address= leader-id=
2025-01-28T14:54:14.510Z [WARN]  server-c7464b7d-5c64-7fe9-2afc-668011747a55: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:14.510Z [INFO]  server-c7464b7d-5c64-7fe9-2afc-668011747a55: entering candidate state: node="Node at c7464b7d-5c64-7fe9-2afc-668011747a55 [Candidate]" term=2
2025-01-28T14:54:14.510Z [INFO]  server-c7464b7d-5c64-7fe9-2afc-668011747a55: election won: term=2 tally=1
2025-01-28T14:54:14.510Z [INFO]  server-c7464b7d-5c64-7fe9-2afc-668011747a55: entering leader state: leader="Node at c7464b7d-5c64-7fe9-2afc-668011747a55 [Leader]"
2025-01-28T14:54:14.616Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:14.510508647 +0000 UTC m=+40.184998627 (1 nodes), 95.551446ms from start of poll, 95.947648ms from cluster start. Timeout at 2025-01-28 14:54:14.615516746 +0000 UTC m=+40.290006786, 105.008159ms after stability
2025-01-28T14:54:14.617Z [ERROR] server-c7464b7d-5c64-7fe9-2afc-668011747a55: failed to take snapshot: error="nothing new to snapshot"
2025-01-28T14:54:14.618Z [INFO]  server-c7464b7d-5c64-7fe9-2afc-668011747a55: starting snapshot up to: index=12
2025-01-28T14:54:14.618Z [INFO]  creating new snapshot: path=/tmp/raft2041857284/snapshots/2-12-1738076054618.tmp
2025-01-28T14:54:14.619Z [INFO]  server-c7464b7d-5c64-7fe9-2afc-668011747a55: compacting logs: from=1 to=2
2025-01-28T14:54:14.619Z [INFO]  server-c7464b7d-5c64-7fe9-2afc-668011747a55: snapshot complete up to: index=12
--- PASS: TestRaft_UserSnapshot (0.21s)
=== RUN   TestRaft_UserRestore
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false
2025-01-28T14:54:14.623Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761 Address:0640f2fe-9fcd-00cf-0dbf-653e06ad6761} {Suffrage:Voter ID:server-8729d1b3-48b0-0b5c-9099-236c1b1a99f2 Address:8729d1b3-48b0-0b5c-9099-236c1b1a99f2} {Suffrage:Voter ID:server-5401f093-b02f-401e-a07e-c348c61f98f7 Address:5401f093-b02f-401e-a07e-c348c61f98f7}]"
2025-01-28T14:54:14.624Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: entering follower state: follower="Node at 0640f2fe-9fcd-00cf-0dbf-653e06ad6761 [Follower]" leader-address= leader-id=
2025-01-28T14:54:14.624Z [INFO]  server-8729d1b3-48b0-0b5c-9099-236c1b1a99f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761 Address:0640f2fe-9fcd-00cf-0dbf-653e06ad6761} {Suffrage:Voter ID:server-8729d1b3-48b0-0b5c-9099-236c1b1a99f2 Address:8729d1b3-48b0-0b5c-9099-236c1b1a99f2} {Suffrage:Voter ID:server-5401f093-b02f-401e-a07e-c348c61f98f7 Address:5401f093-b02f-401e-a07e-c348c61f98f7}]"
2025-01-28T14:54:14.624Z [INFO]  server-8729d1b3-48b0-0b5c-9099-236c1b1a99f2: entering follower state: follower="Node at 8729d1b3-48b0-0b5c-9099-236c1b1a99f2 [Follower]" leader-address= leader-id=
2025-01-28T14:54:14.624Z [INFO]  server-5401f093-b02f-401e-a07e-c348c61f98f7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761 Address:0640f2fe-9fcd-00cf-0dbf-653e06ad6761} {Suffrage:Voter ID:server-8729d1b3-48b0-0b5c-9099-236c1b1a99f2 Address:8729d1b3-48b0-0b5c-9099-236c1b1a99f2} {Suffrage:Voter ID:server-5401f093-b02f-401e-a07e-c348c61f98f7 Address:5401f093-b02f-401e-a07e-c348c61f98f7}]"
2025-01-28T14:54:14.624Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:14.624Z [INFO]  server-5401f093-b02f-401e-a07e-c348c61f98f7: entering follower state: follower="Node at 5401f093-b02f-401e-a07e-c348c61f98f7 [Follower]" leader-address= leader-id=
2025-01-28T14:54:15.155Z [WARN]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:15.156Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: entering candidate state: node="Node at 0640f2fe-9fcd-00cf-0dbf-653e06ad6761 [Candidate]" term=2
2025-01-28T14:54:15.156Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: election won: term=2 tally=2
2025-01-28T14:54:15.156Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: entering leader state: leader="Node at 0640f2fe-9fcd-00cf-0dbf-653e06ad6761 [Leader]"
2025-01-28T14:54:15.156Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: added peer, starting replication: peer=server-8729d1b3-48b0-0b5c-9099-236c1b1a99f2
2025-01-28T14:54:15.156Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: added peer, starting replication: peer=server-5401f093-b02f-401e-a07e-c348c61f98f7
2025-01-28T14:54:15.156Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: pipelining replication: peer="{Voter server-8729d1b3-48b0-0b5c-9099-236c1b1a99f2 8729d1b3-48b0-0b5c-9099-236c1b1a99f2}"
2025-01-28T14:54:15.156Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: pipelining replication: peer="{Voter server-5401f093-b02f-401e-a07e-c348c61f98f7 5401f093-b02f-401e-a07e-c348c61f98f7}"
2025-01-28T14:54:16.162Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:15.156483126 +0000 UTC m=+40.830973106 (1 nodes), 531.851625ms from start of poll, 533.014376ms from cluster start. Timeout at 2025-01-28 14:54:16.161496145 +0000 UTC m=+41.835986185, 1.005013079s after stability
2025-01-28T14:54:16.163Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: starting snapshot up to: index=12
2025-01-28T14:54:16.163Z [INFO]  creating new snapshot: path=/tmp/raft3069554211/snapshots/2-12-1738076056163.tmp
2025-01-28T14:54:16.165Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: snapshot complete up to: index=12
2025-01-28T14:54:16.166Z [INFO]  creating new snapshot: path=/tmp/raft3069554211/snapshots/2-23-1738076056166.tmp
2025-01-28T14:54:16.168Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: copied to local snapshot: bytes=71
2025-01-28T14:54:16.169Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: snapshot restore progress: id=2-23-1738076056166 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:16.169Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: restored user snapshot: index=23
2025-01-28T14:54:16.169Z [ERROR] server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: failed to get log: index=23 error="log not found"
2025-01-28T14:54:16.169Z [ERROR] server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: failed to get log: index=23 error="log not found"
2025-01-28T14:54:16.169Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: aborting pipeline replication: peer="{Voter server-8729d1b3-48b0-0b5c-9099-236c1b1a99f2 8729d1b3-48b0-0b5c-9099-236c1b1a99f2}"
2025-01-28T14:54:16.169Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: aborting pipeline replication: peer="{Voter server-5401f093-b02f-401e-a07e-c348c61f98f7 5401f093-b02f-401e-a07e-c348c61f98f7}"
2025-01-28T14:54:16.175Z [ERROR] server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: failed to get log: index=23 error="log not found"
2025-01-28T14:54:16.175Z [ERROR] server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: failed to get log: index=23 error="log not found"
2025-01-28T14:54:16.176Z [INFO]  creating new snapshot: path=/tmp/raft1826080371/snapshots/2-23-1738076056176.tmp
2025-01-28T14:54:16.176Z [INFO]  creating new snapshot: path=/tmp/raft1059235554/snapshots/2-23-1738076056176.tmp
2025-01-28T14:54:16.177Z [INFO]  server-8729d1b3-48b0-0b5c-9099-236c1b1a99f2: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:16.177Z [INFO]  server-5401f093-b02f-401e-a07e-c348c61f98f7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:16.178Z [INFO]  server-5401f093-b02f-401e-a07e-c348c61f98f7: copied to local snapshot: bytes=71
2025-01-28T14:54:16.178Z [INFO]  server-8729d1b3-48b0-0b5c-9099-236c1b1a99f2: copied to local snapshot: bytes=71
2025-01-28T14:54:16.178Z [INFO]  server-5401f093-b02f-401e-a07e-c348c61f98f7: snapshot restore progress: id=2-23-1738076056176 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:16.178Z [INFO]  server-8729d1b3-48b0-0b5c-9099-236c1b1a99f2: snapshot restore progress: id=2-23-1738076056176 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:16.178Z [INFO]  server-5401f093-b02f-401e-a07e-c348c61f98f7: Installed remote snapshot
2025-01-28T14:54:16.178Z [INFO]  server-8729d1b3-48b0-0b5c-9099-236c1b1a99f2: Installed remote snapshot
2025-01-28T14:54:16.179Z [WARN]  server-5401f093-b02f-401e-a07e-c348c61f98f7: skipping application of old log: index=22
2025-01-28T14:54:16.179Z [WARN]  server-8729d1b3-48b0-0b5c-9099-236c1b1a99f2: skipping application of old log: index=22
2025-01-28T14:54:16.179Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: pipelining replication: peer="{Voter server-5401f093-b02f-401e-a07e-c348c61f98f7 5401f093-b02f-401e-a07e-c348c61f98f7}"
2025-01-28T14:54:16.179Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:16.179Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: pipelining replication: peer="{Voter server-8729d1b3-48b0-0b5c-9099-236c1b1a99f2 8729d1b3-48b0-0b5c-9099-236c1b1a99f2}"
2025-01-28T14:54:17.185Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:16.179393266 +0000 UTC m=+41.853883246 (2 nodes), 2.78µs from start of poll, 1.555924516s from cluster start. Timeout at 2025-01-28 14:54:17.184409164 +0000 UTC m=+42.858899164, 1.005015918s after stability
2025-01-28T14:54:17.198Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: aborting pipeline replication: peer="{Voter server-5401f093-b02f-401e-a07e-c348c61f98f7 5401f093-b02f-401e-a07e-c348c61f98f7}"
2025-01-28T14:54:17.198Z [INFO]  server-0640f2fe-9fcd-00cf-0dbf-653e06ad6761: aborting pipeline replication: peer="{Voter server-8729d1b3-48b0-0b5c-9099-236c1b1a99f2 8729d1b3-48b0-0b5c-9099-236c1b1a99f2}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false
2025-01-28T14:54:17.204Z [INFO]  server-9cf0ae6c-d9fe-370f-2c7c-54d3af84533b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9cf0ae6c-d9fe-370f-2c7c-54d3af84533b Address:9cf0ae6c-d9fe-370f-2c7c-54d3af84533b} {Suffrage:Voter ID:server-c040e7ec-138c-ac03-79b0-8c7b0bc18403 Address:c040e7ec-138c-ac03-79b0-8c7b0bc18403} {Suffrage:Voter ID:server-9c609124-1208-e526-d005-97b61985b125 Address:9c609124-1208-e526-d005-97b61985b125}]"
2025-01-28T14:54:17.204Z [INFO]  server-9cf0ae6c-d9fe-370f-2c7c-54d3af84533b: entering follower state: follower="Node at 9cf0ae6c-d9fe-370f-2c7c-54d3af84533b [Follower]" leader-address= leader-id=
2025-01-28T14:54:17.205Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9cf0ae6c-d9fe-370f-2c7c-54d3af84533b Address:9cf0ae6c-d9fe-370f-2c7c-54d3af84533b} {Suffrage:Voter ID:server-c040e7ec-138c-ac03-79b0-8c7b0bc18403 Address:c040e7ec-138c-ac03-79b0-8c7b0bc18403} {Suffrage:Voter ID:server-9c609124-1208-e526-d005-97b61985b125 Address:9c609124-1208-e526-d005-97b61985b125}]"
2025-01-28T14:54:17.206Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: entering follower state: follower="Node at c040e7ec-138c-ac03-79b0-8c7b0bc18403 [Follower]" leader-address= leader-id=
2025-01-28T14:54:17.206Z [INFO]  server-9c609124-1208-e526-d005-97b61985b125: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9cf0ae6c-d9fe-370f-2c7c-54d3af84533b Address:9cf0ae6c-d9fe-370f-2c7c-54d3af84533b} {Suffrage:Voter ID:server-c040e7ec-138c-ac03-79b0-8c7b0bc18403 Address:c040e7ec-138c-ac03-79b0-8c7b0bc18403} {Suffrage:Voter ID:server-9c609124-1208-e526-d005-97b61985b125 Address:9c609124-1208-e526-d005-97b61985b125}]"
2025-01-28T14:54:17.206Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:17.206Z [INFO]  server-9c609124-1208-e526-d005-97b61985b125: entering follower state: follower="Node at 9c609124-1208-e526-d005-97b61985b125 [Follower]" leader-address= leader-id=
2025-01-28T14:54:17.904Z [WARN]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:17.904Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: entering candidate state: node="Node at c040e7ec-138c-ac03-79b0-8c7b0bc18403 [Candidate]" term=2
2025-01-28T14:54:17.905Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: election won: term=2 tally=2
2025-01-28T14:54:17.905Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: entering leader state: leader="Node at c040e7ec-138c-ac03-79b0-8c7b0bc18403 [Leader]"
2025-01-28T14:54:17.905Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: added peer, starting replication: peer=server-9cf0ae6c-d9fe-370f-2c7c-54d3af84533b
2025-01-28T14:54:17.905Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: added peer, starting replication: peer=server-9c609124-1208-e526-d005-97b61985b125
2025-01-28T14:54:17.905Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: pipelining replication: peer="{Voter server-9cf0ae6c-d9fe-370f-2c7c-54d3af84533b 9cf0ae6c-d9fe-370f-2c7c-54d3af84533b}"
2025-01-28T14:54:17.905Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: pipelining replication: peer="{Voter server-9c609124-1208-e526-d005-97b61985b125 9c609124-1208-e526-d005-97b61985b125}"
2025-01-28T14:54:18.910Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:17.905109743 +0000 UTC m=+43.579599703 (1 nodes), 698.64739ms from start of poll, 701.248545ms from cluster start. Timeout at 2025-01-28 14:54:18.910117182 +0000 UTC m=+44.584607242, 1.005007539s after stability
2025-01-28T14:54:18.911Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: starting snapshot up to: index=12
2025-01-28T14:54:18.911Z [INFO]  creating new snapshot: path=/tmp/raft2401974711/snapshots/2-12-1738076058911.tmp
2025-01-28T14:54:18.913Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: snapshot complete up to: index=12
2025-01-28T14:54:18.915Z [INFO]  creating new snapshot: path=/tmp/raft2401974711/snapshots/2-23-1738076058915.tmp
2025-01-28T14:54:18.916Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: copied to local snapshot: bytes=71
2025-01-28T14:54:18.917Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: snapshot restore progress: id=2-23-1738076058915 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:18.917Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: removing all old logs from log store
2025-01-28T14:54:18.917Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: compacting logs: from=1 to=22
2025-01-28T14:54:18.917Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: restored user snapshot: index=23
2025-01-28T14:54:18.917Z [ERROR] server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: failed to get log: index=22 error="log not found"
2025-01-28T14:54:18.917Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: aborting pipeline replication: peer="{Voter server-9c609124-1208-e526-d005-97b61985b125 9c609124-1208-e526-d005-97b61985b125}"
2025-01-28T14:54:18.917Z [ERROR] server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: failed to get log: index=22 error="log not found"
2025-01-28T14:54:18.917Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: aborting pipeline replication: peer="{Voter server-9cf0ae6c-d9fe-370f-2c7c-54d3af84533b 9cf0ae6c-d9fe-370f-2c7c-54d3af84533b}"
2025-01-28T14:54:18.924Z [ERROR] server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: failed to get log: index=22 error="log not found"
2025-01-28T14:54:18.925Z [INFO]  creating new snapshot: path=/tmp/raft3936816475/snapshots/2-23-1738076058925.tmp
2025-01-28T14:54:18.926Z [INFO]  server-9cf0ae6c-d9fe-370f-2c7c-54d3af84533b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:18.927Z [INFO]  server-9cf0ae6c-d9fe-370f-2c7c-54d3af84533b: copied to local snapshot: bytes=71
2025-01-28T14:54:18.927Z [ERROR] server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: failed to get log: index=22 error="log not found"
2025-01-28T14:54:18.927Z [INFO]  server-9cf0ae6c-d9fe-370f-2c7c-54d3af84533b: snapshot restore progress: id=2-23-1738076058925 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:18.927Z [INFO]  server-9cf0ae6c-d9fe-370f-2c7c-54d3af84533b: removing all old logs from log store
2025-01-28T14:54:18.927Z [INFO]  server-9cf0ae6c-d9fe-370f-2c7c-54d3af84533b: compacting logs: from=1 to=22
2025-01-28T14:54:18.927Z [INFO]  server-9cf0ae6c-d9fe-370f-2c7c-54d3af84533b: Installed remote snapshot
2025-01-28T14:54:18.928Z [WARN]  server-9cf0ae6c-d9fe-370f-2c7c-54d3af84533b: skipping application of old log: index=22
2025-01-28T14:54:18.928Z [INFO]  creating new snapshot: path=/tmp/raft2466843163/snapshots/2-23-1738076058928.tmp
2025-01-28T14:54:18.928Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: pipelining replication: peer="{Voter server-9cf0ae6c-d9fe-370f-2c7c-54d3af84533b 9cf0ae6c-d9fe-370f-2c7c-54d3af84533b}"
2025-01-28T14:54:18.928Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:18.928Z [INFO]  server-9c609124-1208-e526-d005-97b61985b125: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:18.929Z [INFO]  server-9c609124-1208-e526-d005-97b61985b125: copied to local snapshot: bytes=71
2025-01-28T14:54:18.930Z [INFO]  server-9c609124-1208-e526-d005-97b61985b125: snapshot restore progress: id=2-23-1738076058928 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:18.930Z [INFO]  server-9c609124-1208-e526-d005-97b61985b125: removing all old logs from log store
2025-01-28T14:54:18.930Z [INFO]  server-9c609124-1208-e526-d005-97b61985b125: compacting logs: from=1 to=22
2025-01-28T14:54:18.930Z [INFO]  server-9c609124-1208-e526-d005-97b61985b125: Installed remote snapshot
2025-01-28T14:54:18.930Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: pipelining replication: peer="{Voter server-9c609124-1208-e526-d005-97b61985b125 9c609124-1208-e526-d005-97b61985b125}"
2025-01-28T14:54:19.934Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:18.928462981 +0000 UTC m=+44.602953001 (2 nodes), 3.44µs from start of poll, 1.724601843s from cluster start. Timeout at 2025-01-28 14:54:19.933497578 +0000 UTC m=+45.607987558, 1.005034557s after stability
2025-01-28T14:54:19.947Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: aborting pipeline replication: peer="{Voter server-9c609124-1208-e526-d005-97b61985b125 9c609124-1208-e526-d005-97b61985b125}"
2025-01-28T14:54:19.947Z [INFO]  server-c040e7ec-138c-ac03-79b0-8c7b0bc18403: aborting pipeline replication: peer="{Voter server-9cf0ae6c-d9fe-370f-2c7c-54d3af84533b 9cf0ae6c-d9fe-370f-2c7c-54d3af84533b}"
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true
2025-01-28T14:54:19.952Z [INFO]  server-1ef7e83a-bb69-a369-9bb8-6ca81262f962: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ef7e83a-bb69-a369-9bb8-6ca81262f962 Address:1ef7e83a-bb69-a369-9bb8-6ca81262f962} {Suffrage:Voter ID:server-9c0e008d-5edd-34cd-4663-902eb315721b Address:9c0e008d-5edd-34cd-4663-902eb315721b} {Suffrage:Voter ID:server-9e3fd7b9-09db-d49d-37e3-57a86907dd99 Address:9e3fd7b9-09db-d49d-37e3-57a86907dd99}]"
2025-01-28T14:54:19.953Z [INFO]  server-1ef7e83a-bb69-a369-9bb8-6ca81262f962: entering follower state: follower="Node at 1ef7e83a-bb69-a369-9bb8-6ca81262f962 [Follower]" leader-address= leader-id=
2025-01-28T14:54:19.953Z [INFO]  server-9c0e008d-5edd-34cd-4663-902eb315721b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ef7e83a-bb69-a369-9bb8-6ca81262f962 Address:1ef7e83a-bb69-a369-9bb8-6ca81262f962} {Suffrage:Voter ID:server-9c0e008d-5edd-34cd-4663-902eb315721b Address:9c0e008d-5edd-34cd-4663-902eb315721b} {Suffrage:Voter ID:server-9e3fd7b9-09db-d49d-37e3-57a86907dd99 Address:9e3fd7b9-09db-d49d-37e3-57a86907dd99}]"
2025-01-28T14:54:19.953Z [INFO]  server-9c0e008d-5edd-34cd-4663-902eb315721b: entering follower state: follower="Node at 9c0e008d-5edd-34cd-4663-902eb315721b [Follower]" leader-address= leader-id=
2025-01-28T14:54:19.953Z [INFO]  server-9e3fd7b9-09db-d49d-37e3-57a86907dd99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1ef7e83a-bb69-a369-9bb8-6ca81262f962 Address:1ef7e83a-bb69-a369-9bb8-6ca81262f962} {Suffrage:Voter ID:server-9c0e008d-5edd-34cd-4663-902eb315721b Address:9c0e008d-5edd-34cd-4663-902eb315721b} {Suffrage:Voter ID:server-9e3fd7b9-09db-d49d-37e3-57a86907dd99 Address:9e3fd7b9-09db-d49d-37e3-57a86907dd99}]"
2025-01-28T14:54:19.953Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:19.953Z [INFO]  server-9e3fd7b9-09db-d49d-37e3-57a86907dd99: entering follower state: follower="Node at 9e3fd7b9-09db-d49d-37e3-57a86907dd99 [Follower]" leader-address= leader-id=
2025-01-28T14:54:20.484Z [WARN]  server-9c0e008d-5edd-34cd-4663-902eb315721b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:20.484Z [INFO]  server-9c0e008d-5edd-34cd-4663-902eb315721b: entering candidate state: node="Node at 9c0e008d-5edd-34cd-4663-902eb315721b [Candidate]" term=2
2025-01-28T14:54:20.485Z [INFO]  server-9c0e008d-5edd-34cd-4663-902eb315721b: election won: term=2 tally=2
2025-01-28T14:54:20.485Z [INFO]  server-9c0e008d-5edd-34cd-4663-902eb315721b: entering leader state: leader="Node at 9c0e008d-5edd-34cd-4663-902eb315721b [Leader]"
2025-01-28T14:54:20.485Z [INFO]  server-9c0e008d-5edd-34cd-4663-902eb315721b: added peer, starting replication: peer=server-1ef7e83a-bb69-a369-9bb8-6ca81262f962
2025-01-28T14:54:20.485Z [INFO]  server-9c0e008d-5edd-34cd-4663-902eb315721b: added peer, starting replication: peer=server-9e3fd7b9-09db-d49d-37e3-57a86907dd99
2025-01-28T14:54:20.485Z [INFO]  server-9c0e008d-5edd-34cd-4663-902eb315721b: pipelining replication: peer="{Voter server-9e3fd7b9-09db-d49d-37e3-57a86907dd99 9e3fd7b9-09db-d49d-37e3-57a86907dd99}"
2025-01-28T14:54:20.485Z [INFO]  server-9c0e008d-5edd-34cd-4663-902eb315721b: pipelining replication: peer="{Voter server-1ef7e83a-bb69-a369-9bb8-6ca81262f962 1ef7e83a-bb69-a369-9bb8-6ca81262f962}"
2025-01-28T14:54:21.490Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:20.485216618 +0000 UTC m=+46.159706598 (1 nodes), 531.612528ms from start of poll, 532.722783ms from cluster start. Timeout at 2025-01-28 14:54:21.490224318 +0000 UTC m=+47.164714418, 1.00500782s after stability
2025-01-28T14:54:21.492Z [INFO]  server-9c0e008d-5edd-34cd-4663-902eb315721b: starting snapshot up to: index=12
2025-01-28T14:54:21.492Z [INFO]  creating new snapshot: path=/tmp/raft924787358/snapshots/2-12-1738076061492.tmp
2025-01-28T14:54:21.669Z [INFO]  server-9c0e008d-5edd-34cd-4663-902eb315721b: snapshot complete up to: index=12
2025-01-28T14:54:21.673Z [INFO]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-796717a3-8be9-6a90-c3ff-babcfc941ad1 Address:796717a3-8be9-6a90-c3ff-babcfc941ad1} {Suffrage:Voter ID:server-6e60e5aa-c036-8918-49be-ab3c8b222b91 Address:6e60e5aa-c036-8918-49be-ab3c8b222b91} {Suffrage:Voter ID:server-d1afff6d-4fc0-3302-952c-38c375c42ee7 Address:d1afff6d-4fc0-3302-952c-38c375c42ee7}]"
2025-01-28T14:54:21.673Z [INFO]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: entering follower state: follower="Node at 796717a3-8be9-6a90-c3ff-babcfc941ad1 [Follower]" leader-address= leader-id=
2025-01-28T14:54:21.673Z [INFO]  server-6e60e5aa-c036-8918-49be-ab3c8b222b91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-796717a3-8be9-6a90-c3ff-babcfc941ad1 Address:796717a3-8be9-6a90-c3ff-babcfc941ad1} {Suffrage:Voter ID:server-6e60e5aa-c036-8918-49be-ab3c8b222b91 Address:6e60e5aa-c036-8918-49be-ab3c8b222b91} {Suffrage:Voter ID:server-d1afff6d-4fc0-3302-952c-38c375c42ee7 Address:d1afff6d-4fc0-3302-952c-38c375c42ee7}]"
2025-01-28T14:54:21.673Z [INFO]  server-6e60e5aa-c036-8918-49be-ab3c8b222b91: entering follower state: follower="Node at 6e60e5aa-c036-8918-49be-ab3c8b222b91 [Follower]" leader-address= leader-id=
2025-01-28T14:54:21.673Z [INFO]  server-d1afff6d-4fc0-3302-952c-38c375c42ee7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-796717a3-8be9-6a90-c3ff-babcfc941ad1 Address:796717a3-8be9-6a90-c3ff-babcfc941ad1} {Suffrage:Voter ID:server-6e60e5aa-c036-8918-49be-ab3c8b222b91 Address:6e60e5aa-c036-8918-49be-ab3c8b222b91} {Suffrage:Voter ID:server-d1afff6d-4fc0-3302-952c-38c375c42ee7 Address:d1afff6d-4fc0-3302-952c-38c375c42ee7}]"
2025-01-28T14:54:21.673Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:21.673Z [INFO]  server-d1afff6d-4fc0-3302-952c-38c375c42ee7: entering follower state: follower="Node at d1afff6d-4fc0-3302-952c-38c375c42ee7 [Follower]" leader-address= leader-id=
2025-01-28T14:54:22.181Z [WARN]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:22.181Z [INFO]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: entering candidate state: node="Node at 796717a3-8be9-6a90-c3ff-babcfc941ad1 [Candidate]" term=2
2025-01-28T14:54:22.181Z [WARN]  server-d1afff6d-4fc0-3302-952c-38c375c42ee7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:22.181Z [INFO]  server-d1afff6d-4fc0-3302-952c-38c375c42ee7: entering candidate state: node="Node at d1afff6d-4fc0-3302-952c-38c375c42ee7 [Candidate]" term=2
2025-01-28T14:54:22.181Z [INFO]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: election won: term=2 tally=2
2025-01-28T14:54:22.181Z [INFO]  server-d1afff6d-4fc0-3302-952c-38c375c42ee7: duplicate requestVote for same term: term=2
2025-01-28T14:54:22.181Z [INFO]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: entering leader state: leader="Node at 796717a3-8be9-6a90-c3ff-babcfc941ad1 [Leader]"
2025-01-28T14:54:22.181Z [INFO]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: added peer, starting replication: peer=server-6e60e5aa-c036-8918-49be-ab3c8b222b91
2025-01-28T14:54:22.181Z [INFO]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: added peer, starting replication: peer=server-d1afff6d-4fc0-3302-952c-38c375c42ee7
2025-01-28T14:54:22.181Z [INFO]  server-6e60e5aa-c036-8918-49be-ab3c8b222b91: duplicate requestVote for same term: term=2
2025-01-28T14:54:22.181Z [WARN]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: rejecting vote request since we have a leader: from=d1afff6d-4fc0-3302-952c-38c375c42ee7 leader=796717a3-8be9-6a90-c3ff-babcfc941ad1 leader-id=server-796717a3-8be9-6a90-c3ff-babcfc941ad1
2025-01-28T14:54:22.181Z [INFO]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: pipelining replication: peer="{Voter server-6e60e5aa-c036-8918-49be-ab3c8b222b91 6e60e5aa-c036-8918-49be-ab3c8b222b91}"
2025-01-28T14:54:22.181Z [INFO]  server-d1afff6d-4fc0-3302-952c-38c375c42ee7: entering follower state: follower="Node at d1afff6d-4fc0-3302-952c-38c375c42ee7 [Follower]" leader-address=796717a3-8be9-6a90-c3ff-babcfc941ad1 leader-id=server-796717a3-8be9-6a90-c3ff-babcfc941ad1
2025-01-28T14:54:22.182Z [INFO]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: pipelining replication: peer="{Voter server-d1afff6d-4fc0-3302-952c-38c375c42ee7 d1afff6d-4fc0-3302-952c-38c375c42ee7}"
2025-01-28T14:54:23.187Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:22.181825242 +0000 UTC m=+47.856315222 (1 nodes), 507.973998ms from start of poll, 509.070595ms from cluster start. Timeout at 2025-01-28 14:54:23.186832821 +0000 UTC m=+48.861322941, 1.005007719s after stability
2025-01-28T14:54:23.188Z [INFO]  creating new snapshot: path=/tmp/raft4007420181/snapshots/2-13-1738076063188.tmp
2025-01-28T14:54:23.189Z [INFO]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: copied to local snapshot: bytes=71
2025-01-28T14:54:23.190Z [INFO]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: snapshot restore progress: id=2-13-1738076063188 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:23.190Z [INFO]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: restored user snapshot: index=13
2025-01-28T14:54:23.190Z [ERROR] server-796717a3-8be9-6a90-c3ff-babcfc941ad1: failed to get log: index=3 error="log not found"
2025-01-28T14:54:23.191Z [INFO]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: aborting pipeline replication: peer="{Voter server-d1afff6d-4fc0-3302-952c-38c375c42ee7 d1afff6d-4fc0-3302-952c-38c375c42ee7}"
2025-01-28T14:54:23.190Z [ERROR] server-796717a3-8be9-6a90-c3ff-babcfc941ad1: failed to get log: index=3 error="log not found"
2025-01-28T14:54:23.191Z [INFO]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: aborting pipeline replication: peer="{Voter server-6e60e5aa-c036-8918-49be-ab3c8b222b91 6e60e5aa-c036-8918-49be-ab3c8b222b91}"
2025-01-28T14:54:23.199Z [ERROR] server-796717a3-8be9-6a90-c3ff-babcfc941ad1: failed to get log: index=3 error="log not found"
2025-01-28T14:54:23.200Z [INFO]  creating new snapshot: path=/tmp/raft1823198774/snapshots/2-13-1738076063200.tmp
2025-01-28T14:54:23.200Z [ERROR] server-796717a3-8be9-6a90-c3ff-babcfc941ad1: failed to get log: index=3 error="log not found"
2025-01-28T14:54:23.201Z [INFO]  creating new snapshot: path=/tmp/raft171869902/snapshots/2-13-1738076063201.tmp
2025-01-28T14:54:23.201Z [INFO]  server-d1afff6d-4fc0-3302-952c-38c375c42ee7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:23.201Z [INFO]  server-6e60e5aa-c036-8918-49be-ab3c8b222b91: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:23.202Z [INFO]  server-d1afff6d-4fc0-3302-952c-38c375c42ee7: copied to local snapshot: bytes=71
2025-01-28T14:54:23.202Z [INFO]  server-6e60e5aa-c036-8918-49be-ab3c8b222b91: copied to local snapshot: bytes=71
2025-01-28T14:54:23.202Z [INFO]  server-d1afff6d-4fc0-3302-952c-38c375c42ee7: snapshot restore progress: id=2-13-1738076063200 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:23.202Z [INFO]  server-d1afff6d-4fc0-3302-952c-38c375c42ee7: Installed remote snapshot
2025-01-28T14:54:23.203Z [INFO]  server-6e60e5aa-c036-8918-49be-ab3c8b222b91: snapshot restore progress: id=2-13-1738076063201 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:23.203Z [INFO]  server-6e60e5aa-c036-8918-49be-ab3c8b222b91: Installed remote snapshot
2025-01-28T14:54:23.203Z [INFO]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: pipelining replication: peer="{Voter server-d1afff6d-4fc0-3302-952c-38c375c42ee7 d1afff6d-4fc0-3302-952c-38c375c42ee7}"
2025-01-28T14:54:23.203Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:23.203Z [INFO]  server-796717a3-8be9-6a90-c3ff-babcfc941ad1: pipelining replication: peer="{Voter server-6e60e5aa-c036-8918-49be-ab3c8b222b91 6e60e5aa-c036-8918-49be-ab3c8b222b91}"
2025-01-28T14:54:24.208Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:23.203659603 +0000 UTC m=+48.878149583 (2 nodes), 2.28µs from start of poll, 1.530904956s from cluster start. Timeout at 2025-01-28 14:54:24.208699619 +0000 UTC m=+49.883189739, 1.005040156s after stability
2025-01-28T14:54:24.222Z [INFO]  server-9c0e008d-5edd-34cd-4663-902eb315721b: aborting pipeline replication: peer="{Voter server-9e3fd7b9-09db-d49d-37e3-57a86907dd99 9e3fd7b9-09db-d49d-37e3-57a86907dd99}"
2025-01-28T14:54:24.222Z [INFO]  server-9c0e008d-5edd-34cd-4663-902eb315721b: aborting pipeline replication: peer="{Voter server-1ef7e83a-bb69-a369-9bb8-6ca81262f962 1ef7e83a-bb69-a369-9bb8-6ca81262f962}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true
2025-01-28T14:54:24.228Z [INFO]  server-8dc342da-9b13-39ca-70b9-cb2d4de56941: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8dc342da-9b13-39ca-70b9-cb2d4de56941 Address:8dc342da-9b13-39ca-70b9-cb2d4de56941} {Suffrage:Voter ID:server-049fa6b8-6f27-43df-aa89-37941a186534 Address:049fa6b8-6f27-43df-aa89-37941a186534} {Suffrage:Voter ID:server-df36936f-e222-5616-bdb6-1bdc60c25a23 Address:df36936f-e222-5616-bdb6-1bdc60c25a23}]"
2025-01-28T14:54:24.229Z [INFO]  server-049fa6b8-6f27-43df-aa89-37941a186534: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8dc342da-9b13-39ca-70b9-cb2d4de56941 Address:8dc342da-9b13-39ca-70b9-cb2d4de56941} {Suffrage:Voter ID:server-049fa6b8-6f27-43df-aa89-37941a186534 Address:049fa6b8-6f27-43df-aa89-37941a186534} {Suffrage:Voter ID:server-df36936f-e222-5616-bdb6-1bdc60c25a23 Address:df36936f-e222-5616-bdb6-1bdc60c25a23}]"
2025-01-28T14:54:24.229Z [INFO]  server-8dc342da-9b13-39ca-70b9-cb2d4de56941: entering follower state: follower="Node at 8dc342da-9b13-39ca-70b9-cb2d4de56941 [Follower]" leader-address= leader-id=
2025-01-28T14:54:24.229Z [INFO]  server-049fa6b8-6f27-43df-aa89-37941a186534: entering follower state: follower="Node at 049fa6b8-6f27-43df-aa89-37941a186534 [Follower]" leader-address= leader-id=
2025-01-28T14:54:24.229Z [INFO]  server-df36936f-e222-5616-bdb6-1bdc60c25a23: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8dc342da-9b13-39ca-70b9-cb2d4de56941 Address:8dc342da-9b13-39ca-70b9-cb2d4de56941} {Suffrage:Voter ID:server-049fa6b8-6f27-43df-aa89-37941a186534 Address:049fa6b8-6f27-43df-aa89-37941a186534} {Suffrage:Voter ID:server-df36936f-e222-5616-bdb6-1bdc60c25a23 Address:df36936f-e222-5616-bdb6-1bdc60c25a23}]"
2025-01-28T14:54:24.230Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:24.230Z [INFO]  server-df36936f-e222-5616-bdb6-1bdc60c25a23: entering follower state: follower="Node at df36936f-e222-5616-bdb6-1bdc60c25a23 [Follower]" leader-address= leader-id=
2025-01-28T14:54:24.741Z [WARN]  server-df36936f-e222-5616-bdb6-1bdc60c25a23: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:24.741Z [INFO]  server-df36936f-e222-5616-bdb6-1bdc60c25a23: entering candidate state: node="Node at df36936f-e222-5616-bdb6-1bdc60c25a23 [Candidate]" term=2
2025-01-28T14:54:24.742Z [INFO]  server-df36936f-e222-5616-bdb6-1bdc60c25a23: election won: term=2 tally=2
2025-01-28T14:54:24.742Z [INFO]  server-df36936f-e222-5616-bdb6-1bdc60c25a23: entering leader state: leader="Node at df36936f-e222-5616-bdb6-1bdc60c25a23 [Leader]"
2025-01-28T14:54:24.742Z [INFO]  server-df36936f-e222-5616-bdb6-1bdc60c25a23: added peer, starting replication: peer=server-8dc342da-9b13-39ca-70b9-cb2d4de56941
2025-01-28T14:54:24.742Z [INFO]  server-df36936f-e222-5616-bdb6-1bdc60c25a23: added peer, starting replication: peer=server-049fa6b8-6f27-43df-aa89-37941a186534
2025-01-28T14:54:24.742Z [INFO]  server-df36936f-e222-5616-bdb6-1bdc60c25a23: pipelining replication: peer="{Voter server-8dc342da-9b13-39ca-70b9-cb2d4de56941 8dc342da-9b13-39ca-70b9-cb2d4de56941}"
2025-01-28T14:54:24.742Z [INFO]  server-df36936f-e222-5616-bdb6-1bdc60c25a23: pipelining replication: peer="{Voter server-049fa6b8-6f27-43df-aa89-37941a186534 049fa6b8-6f27-43df-aa89-37941a186534}"
2025-01-28T14:54:25.747Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:24.742346165 +0000 UTC m=+50.416836185 (1 nodes), 512.104588ms from start of poll, 515.001475ms from cluster start. Timeout at 2025-01-28 14:54:25.747351984 +0000 UTC m=+51.421842024, 1.005005839s after stability
2025-01-28T14:54:25.748Z [INFO]  server-df36936f-e222-5616-bdb6-1bdc60c25a23: starting snapshot up to: index=12
2025-01-28T14:54:25.748Z [INFO]  creating new snapshot: path=/tmp/raft1371664513/snapshots/2-12-1738076065748.tmp
2025-01-28T14:54:25.750Z [INFO]  server-df36936f-e222-5616-bdb6-1bdc60c25a23: snapshot complete up to: index=12
2025-01-28T14:54:25.753Z [INFO]  server-3bcfe630-708f-09ed-eaef-81aa41e7668a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3bcfe630-708f-09ed-eaef-81aa41e7668a Address:3bcfe630-708f-09ed-eaef-81aa41e7668a} {Suffrage:Voter ID:server-90050a2c-afa6-9b84-fc4e-4109e8ad878e Address:90050a2c-afa6-9b84-fc4e-4109e8ad878e} {Suffrage:Voter ID:server-8eec329d-2933-8733-3329-234a5a421c3e Address:8eec329d-2933-8733-3329-234a5a421c3e}]"
2025-01-28T14:54:25.753Z [INFO]  server-3bcfe630-708f-09ed-eaef-81aa41e7668a: entering follower state: follower="Node at 3bcfe630-708f-09ed-eaef-81aa41e7668a [Follower]" leader-address= leader-id=
2025-01-28T14:54:25.753Z [INFO]  server-90050a2c-afa6-9b84-fc4e-4109e8ad878e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3bcfe630-708f-09ed-eaef-81aa41e7668a Address:3bcfe630-708f-09ed-eaef-81aa41e7668a} {Suffrage:Voter ID:server-90050a2c-afa6-9b84-fc4e-4109e8ad878e Address:90050a2c-afa6-9b84-fc4e-4109e8ad878e} {Suffrage:Voter ID:server-8eec329d-2933-8733-3329-234a5a421c3e Address:8eec329d-2933-8733-3329-234a5a421c3e}]"
2025-01-28T14:54:25.753Z [INFO]  server-90050a2c-afa6-9b84-fc4e-4109e8ad878e: entering follower state: follower="Node at 90050a2c-afa6-9b84-fc4e-4109e8ad878e [Follower]" leader-address= leader-id=
2025-01-28T14:54:25.753Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3bcfe630-708f-09ed-eaef-81aa41e7668a Address:3bcfe630-708f-09ed-eaef-81aa41e7668a} {Suffrage:Voter ID:server-90050a2c-afa6-9b84-fc4e-4109e8ad878e Address:90050a2c-afa6-9b84-fc4e-4109e8ad878e} {Suffrage:Voter ID:server-8eec329d-2933-8733-3329-234a5a421c3e Address:8eec329d-2933-8733-3329-234a5a421c3e}]"
2025-01-28T14:54:25.753Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:25.753Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: entering follower state: follower="Node at 8eec329d-2933-8733-3329-234a5a421c3e [Follower]" leader-address= leader-id=
2025-01-28T14:54:26.649Z [WARN]  server-8eec329d-2933-8733-3329-234a5a421c3e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:26.650Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: entering candidate state: node="Node at 8eec329d-2933-8733-3329-234a5a421c3e [Candidate]" term=2
2025-01-28T14:54:26.650Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: election won: term=2 tally=2
2025-01-28T14:54:26.650Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: entering leader state: leader="Node at 8eec329d-2933-8733-3329-234a5a421c3e [Leader]"
2025-01-28T14:54:26.650Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: added peer, starting replication: peer=server-3bcfe630-708f-09ed-eaef-81aa41e7668a
2025-01-28T14:54:26.650Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: added peer, starting replication: peer=server-90050a2c-afa6-9b84-fc4e-4109e8ad878e
2025-01-28T14:54:26.650Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: pipelining replication: peer="{Voter server-90050a2c-afa6-9b84-fc4e-4109e8ad878e 90050a2c-afa6-9b84-fc4e-4109e8ad878e}"
2025-01-28T14:54:26.650Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: pipelining replication: peer="{Voter server-3bcfe630-708f-09ed-eaef-81aa41e7668a 3bcfe630-708f-09ed-eaef-81aa41e7668a}"
2025-01-28T14:54:27.655Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:26.650374224 +0000 UTC m=+52.324864204 (1 nodes), 896.382256ms from start of poll, 897.423278ms from cluster start. Timeout at 2025-01-28 14:54:27.655395322 +0000 UTC m=+53.329885302, 1.005021098s after stability
2025-01-28T14:54:27.658Z [INFO]  creating new snapshot: path=/tmp/raft4049951965/snapshots/2-13-1738076067658.tmp
2025-01-28T14:54:27.661Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: copied to local snapshot: bytes=71
2025-01-28T14:54:27.661Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: snapshot restore progress: id=2-13-1738076067658 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:27.661Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: removing all old logs from log store
2025-01-28T14:54:27.662Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: compacting logs: from=1 to=2
2025-01-28T14:54:27.662Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: restored user snapshot: index=13
2025-01-28T14:54:27.662Z [ERROR] server-8eec329d-2933-8733-3329-234a5a421c3e: failed to get log: index=2 error="log not found"
2025-01-28T14:54:27.662Z [ERROR] server-8eec329d-2933-8733-3329-234a5a421c3e: failed to get log: index=2 error="log not found"
2025-01-28T14:54:27.662Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: aborting pipeline replication: peer="{Voter server-90050a2c-afa6-9b84-fc4e-4109e8ad878e 90050a2c-afa6-9b84-fc4e-4109e8ad878e}"
2025-01-28T14:54:27.662Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: aborting pipeline replication: peer="{Voter server-3bcfe630-708f-09ed-eaef-81aa41e7668a 3bcfe630-708f-09ed-eaef-81aa41e7668a}"
2025-01-28T14:54:27.669Z [ERROR] server-8eec329d-2933-8733-3329-234a5a421c3e: failed to get log: index=2 error="log not found"
2025-01-28T14:54:27.669Z [ERROR] server-8eec329d-2933-8733-3329-234a5a421c3e: failed to get log: index=2 error="log not found"
2025-01-28T14:54:27.669Z [INFO]  creating new snapshot: path=/tmp/raft1430819158/snapshots/2-13-1738076067669.tmp
2025-01-28T14:54:27.669Z [INFO]  creating new snapshot: path=/tmp/raft1329274814/snapshots/2-13-1738076067669.tmp
2025-01-28T14:54:27.670Z [INFO]  server-90050a2c-afa6-9b84-fc4e-4109e8ad878e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:27.670Z [INFO]  server-3bcfe630-708f-09ed-eaef-81aa41e7668a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:27.671Z [INFO]  server-90050a2c-afa6-9b84-fc4e-4109e8ad878e: copied to local snapshot: bytes=71
2025-01-28T14:54:27.671Z [INFO]  server-3bcfe630-708f-09ed-eaef-81aa41e7668a: copied to local snapshot: bytes=71
2025-01-28T14:54:27.672Z [INFO]  server-90050a2c-afa6-9b84-fc4e-4109e8ad878e: snapshot restore progress: id=2-13-1738076067669 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:27.672Z [INFO]  server-3bcfe630-708f-09ed-eaef-81aa41e7668a: snapshot restore progress: id=2-13-1738076067669 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:27.672Z [INFO]  server-3bcfe630-708f-09ed-eaef-81aa41e7668a: removing all old logs from log store
2025-01-28T14:54:27.672Z [INFO]  server-3bcfe630-708f-09ed-eaef-81aa41e7668a: compacting logs: from=1 to=2
2025-01-28T14:54:27.672Z [INFO]  server-3bcfe630-708f-09ed-eaef-81aa41e7668a: Installed remote snapshot
2025-01-28T14:54:27.672Z [INFO]  server-90050a2c-afa6-9b84-fc4e-4109e8ad878e: removing all old logs from log store
2025-01-28T14:54:27.672Z [INFO]  server-90050a2c-afa6-9b84-fc4e-4109e8ad878e: compacting logs: from=1 to=2
2025-01-28T14:54:27.672Z [INFO]  server-90050a2c-afa6-9b84-fc4e-4109e8ad878e: Installed remote snapshot
2025-01-28T14:54:27.672Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: pipelining replication: peer="{Voter server-3bcfe630-708f-09ed-eaef-81aa41e7668a 3bcfe630-708f-09ed-eaef-81aa41e7668a}"
2025-01-28T14:54:27.672Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:27.672Z [WARN]  server-90050a2c-afa6-9b84-fc4e-4109e8ad878e: skipping application of old log: index=13
2025-01-28T14:54:27.672Z [INFO]  server-8eec329d-2933-8733-3329-234a5a421c3e: pipelining replication: peer="{Voter server-90050a2c-afa6-9b84-fc4e-4109e8ad878e 90050a2c-afa6-9b84-fc4e-4109e8ad878e}"
2025-01-28T14:54:28.678Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:27.672698719 +0000 UTC m=+53.347188679 (2 nodes), 2.339µs from start of poll, 1.919747753s from cluster start. Timeout at 2025-01-28 14:54:28.677751254 +0000 UTC m=+54.352241234, 1.005052555s after stability
2025-01-28T14:54:28.694Z [INFO]  server-df36936f-e222-5616-bdb6-1bdc60c25a23: aborting pipeline replication: peer="{Voter server-049fa6b8-6f27-43df-aa89-37941a186534 049fa6b8-6f27-43df-aa89-37941a186534}"
2025-01-28T14:54:28.694Z [INFO]  server-df36936f-e222-5616-bdb6-1bdc60c25a23: aborting pipeline replication: peer="{Voter server-8dc342da-9b13-39ca-70b9-cb2d4de56941 8dc342da-9b13-39ca-70b9-cb2d4de56941}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false
2025-01-28T14:54:28.699Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f16a4165-49eb-daee-52d5-fc7314684fa6 Address:f16a4165-49eb-daee-52d5-fc7314684fa6} {Suffrage:Voter ID:server-180fd759-cadb-e5b5-76be-531e34f87a8b Address:180fd759-cadb-e5b5-76be-531e34f87a8b} {Suffrage:Voter ID:server-85b6ad5f-b4b0-2169-f8ba-e0b70b426553 Address:85b6ad5f-b4b0-2169-f8ba-e0b70b426553}]"
2025-01-28T14:54:28.699Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: entering follower state: follower="Node at f16a4165-49eb-daee-52d5-fc7314684fa6 [Follower]" leader-address= leader-id=
2025-01-28T14:54:28.700Z [INFO]  server-180fd759-cadb-e5b5-76be-531e34f87a8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f16a4165-49eb-daee-52d5-fc7314684fa6 Address:f16a4165-49eb-daee-52d5-fc7314684fa6} {Suffrage:Voter ID:server-180fd759-cadb-e5b5-76be-531e34f87a8b Address:180fd759-cadb-e5b5-76be-531e34f87a8b} {Suffrage:Voter ID:server-85b6ad5f-b4b0-2169-f8ba-e0b70b426553 Address:85b6ad5f-b4b0-2169-f8ba-e0b70b426553}]"
2025-01-28T14:54:28.700Z [INFO]  server-180fd759-cadb-e5b5-76be-531e34f87a8b: entering follower state: follower="Node at 180fd759-cadb-e5b5-76be-531e34f87a8b [Follower]" leader-address= leader-id=
2025-01-28T14:54:28.700Z [INFO]  server-85b6ad5f-b4b0-2169-f8ba-e0b70b426553: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f16a4165-49eb-daee-52d5-fc7314684fa6 Address:f16a4165-49eb-daee-52d5-fc7314684fa6} {Suffrage:Voter ID:server-180fd759-cadb-e5b5-76be-531e34f87a8b Address:180fd759-cadb-e5b5-76be-531e34f87a8b} {Suffrage:Voter ID:server-85b6ad5f-b4b0-2169-f8ba-e0b70b426553 Address:85b6ad5f-b4b0-2169-f8ba-e0b70b426553}]"
2025-01-28T14:54:28.700Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:28.700Z [INFO]  server-85b6ad5f-b4b0-2169-f8ba-e0b70b426553: entering follower state: follower="Node at 85b6ad5f-b4b0-2169-f8ba-e0b70b426553 [Follower]" leader-address= leader-id=
2025-01-28T14:54:29.276Z [WARN]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:29.276Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: entering candidate state: node="Node at f16a4165-49eb-daee-52d5-fc7314684fa6 [Candidate]" term=2
2025-01-28T14:54:29.277Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: election won: term=2 tally=2
2025-01-28T14:54:29.277Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: entering leader state: leader="Node at f16a4165-49eb-daee-52d5-fc7314684fa6 [Leader]"
2025-01-28T14:54:29.277Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: added peer, starting replication: peer=server-180fd759-cadb-e5b5-76be-531e34f87a8b
2025-01-28T14:54:29.277Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: added peer, starting replication: peer=server-85b6ad5f-b4b0-2169-f8ba-e0b70b426553
2025-01-28T14:54:29.277Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: pipelining replication: peer="{Voter server-180fd759-cadb-e5b5-76be-531e34f87a8b 180fd759-cadb-e5b5-76be-531e34f87a8b}"
2025-01-28T14:54:29.277Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: pipelining replication: peer="{Voter server-85b6ad5f-b4b0-2169-f8ba-e0b70b426553 85b6ad5f-b4b0-2169-f8ba-e0b70b426553}"
2025-01-28T14:54:30.283Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:29.277492085 +0000 UTC m=+54.951982085 (1 nodes), 576.973349ms from start of poll, 578.207092ms from cluster start. Timeout at 2025-01-28 14:54:30.282501664 +0000 UTC m=+55.956991664, 1.005009579s after stability
2025-01-28T14:54:30.284Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: starting snapshot up to: index=12
2025-01-28T14:54:30.284Z [INFO]  creating new snapshot: path=/tmp/raft2156609689/snapshots/2-12-1738076070284.tmp
2025-01-28T14:54:30.286Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: snapshot complete up to: index=12
2025-01-28T14:54:30.288Z [INFO]  creating new snapshot: path=/tmp/raft2156609689/snapshots/2-23-1738076070288.tmp
2025-01-28T14:54:30.290Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: copied to local snapshot: bytes=71
2025-01-28T14:54:30.290Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: snapshot restore progress: id=2-23-1738076070288 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:30.290Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: restored user snapshot: index=23
2025-01-28T14:54:30.290Z [ERROR] server-f16a4165-49eb-daee-52d5-fc7314684fa6: failed to get log: index=23 error="log not found"
2025-01-28T14:54:30.290Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: aborting pipeline replication: peer="{Voter server-85b6ad5f-b4b0-2169-f8ba-e0b70b426553 85b6ad5f-b4b0-2169-f8ba-e0b70b426553}"
2025-01-28T14:54:30.291Z [ERROR] server-f16a4165-49eb-daee-52d5-fc7314684fa6: failed to get log: index=23 error="log not found"
2025-01-28T14:54:30.291Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: aborting pipeline replication: peer="{Voter server-180fd759-cadb-e5b5-76be-531e34f87a8b 180fd759-cadb-e5b5-76be-531e34f87a8b}"
2025-01-28T14:54:30.299Z [ERROR] server-f16a4165-49eb-daee-52d5-fc7314684fa6: failed to get log: index=23 error="log not found"
2025-01-28T14:54:30.300Z [INFO]  creating new snapshot: path=/tmp/raft465191807/snapshots/2-23-1738076070300.tmp
2025-01-28T14:54:30.301Z [ERROR] server-f16a4165-49eb-daee-52d5-fc7314684fa6: failed to get log: index=23 error="log not found"
2025-01-28T14:54:30.301Z [INFO]  server-180fd759-cadb-e5b5-76be-531e34f87a8b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:30.302Z [INFO]  creating new snapshot: path=/tmp/raft762961124/snapshots/2-23-1738076070302.tmp
2025-01-28T14:54:30.302Z [INFO]  server-180fd759-cadb-e5b5-76be-531e34f87a8b: copied to local snapshot: bytes=71
2025-01-28T14:54:30.302Z [INFO]  server-85b6ad5f-b4b0-2169-f8ba-e0b70b426553: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:30.302Z [INFO]  server-180fd759-cadb-e5b5-76be-531e34f87a8b: snapshot restore progress: id=2-23-1738076070300 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:30.303Z [INFO]  server-180fd759-cadb-e5b5-76be-531e34f87a8b: Installed remote snapshot
2025-01-28T14:54:30.303Z [WARN]  server-180fd759-cadb-e5b5-76be-531e34f87a8b: skipping application of old log: index=22
2025-01-28T14:54:30.303Z [INFO]  server-85b6ad5f-b4b0-2169-f8ba-e0b70b426553: copied to local snapshot: bytes=71
2025-01-28T14:54:30.303Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: pipelining replication: peer="{Voter server-180fd759-cadb-e5b5-76be-531e34f87a8b 180fd759-cadb-e5b5-76be-531e34f87a8b}"
2025-01-28T14:54:30.303Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:30.304Z [INFO]  server-85b6ad5f-b4b0-2169-f8ba-e0b70b426553: snapshot restore progress: id=2-23-1738076070302 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:30.304Z [INFO]  server-85b6ad5f-b4b0-2169-f8ba-e0b70b426553: Installed remote snapshot
2025-01-28T14:54:30.304Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: pipelining replication: peer="{Voter server-85b6ad5f-b4b0-2169-f8ba-e0b70b426553 85b6ad5f-b4b0-2169-f8ba-e0b70b426553}"
2025-01-28T14:54:31.309Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:30.303868058 +0000 UTC m=+55.978358038 (2 nodes), 2.479µs from start of poll, 1.604583045s from cluster start. Timeout at 2025-01-28 14:54:31.308904634 +0000 UTC m=+56.983394654, 1.005036616s after stability
2025-01-28T14:54:31.321Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: aborting pipeline replication: peer="{Voter server-85b6ad5f-b4b0-2169-f8ba-e0b70b426553 85b6ad5f-b4b0-2169-f8ba-e0b70b426553}"
2025-01-28T14:54:31.321Z [INFO]  server-f16a4165-49eb-daee-52d5-fc7314684fa6: aborting pipeline replication: peer="{Voter server-180fd759-cadb-e5b5-76be-531e34f87a8b 180fd759-cadb-e5b5-76be-531e34f87a8b}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false
2025-01-28T14:54:31.326Z [INFO]  server-18d90f22-4c7a-b5d6-08af-5dba33fe2ed7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18d90f22-4c7a-b5d6-08af-5dba33fe2ed7 Address:18d90f22-4c7a-b5d6-08af-5dba33fe2ed7} {Suffrage:Voter ID:server-e3d2edfd-26c4-44da-e721-c910e7daeb59 Address:e3d2edfd-26c4-44da-e721-c910e7daeb59} {Suffrage:Voter ID:server-82c19a4e-9157-25aa-f15b-126f6d879685 Address:82c19a4e-9157-25aa-f15b-126f6d879685}]"
2025-01-28T14:54:31.326Z [INFO]  server-18d90f22-4c7a-b5d6-08af-5dba33fe2ed7: entering follower state: follower="Node at 18d90f22-4c7a-b5d6-08af-5dba33fe2ed7 [Follower]" leader-address= leader-id=
2025-01-28T14:54:31.326Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18d90f22-4c7a-b5d6-08af-5dba33fe2ed7 Address:18d90f22-4c7a-b5d6-08af-5dba33fe2ed7} {Suffrage:Voter ID:server-e3d2edfd-26c4-44da-e721-c910e7daeb59 Address:e3d2edfd-26c4-44da-e721-c910e7daeb59} {Suffrage:Voter ID:server-82c19a4e-9157-25aa-f15b-126f6d879685 Address:82c19a4e-9157-25aa-f15b-126f6d879685}]"
2025-01-28T14:54:31.326Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: entering follower state: follower="Node at e3d2edfd-26c4-44da-e721-c910e7daeb59 [Follower]" leader-address= leader-id=
2025-01-28T14:54:31.326Z [INFO]  server-82c19a4e-9157-25aa-f15b-126f6d879685: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18d90f22-4c7a-b5d6-08af-5dba33fe2ed7 Address:18d90f22-4c7a-b5d6-08af-5dba33fe2ed7} {Suffrage:Voter ID:server-e3d2edfd-26c4-44da-e721-c910e7daeb59 Address:e3d2edfd-26c4-44da-e721-c910e7daeb59} {Suffrage:Voter ID:server-82c19a4e-9157-25aa-f15b-126f6d879685 Address:82c19a4e-9157-25aa-f15b-126f6d879685}]"
2025-01-28T14:54:31.326Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:31.326Z [INFO]  server-82c19a4e-9157-25aa-f15b-126f6d879685: entering follower state: follower="Node at 82c19a4e-9157-25aa-f15b-126f6d879685 [Follower]" leader-address= leader-id=
2025-01-28T14:54:31.834Z [WARN]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:31.834Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: entering candidate state: node="Node at e3d2edfd-26c4-44da-e721-c910e7daeb59 [Candidate]" term=2
2025-01-28T14:54:31.834Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: election won: term=2 tally=2
2025-01-28T14:54:31.834Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: entering leader state: leader="Node at e3d2edfd-26c4-44da-e721-c910e7daeb59 [Leader]"
2025-01-28T14:54:31.834Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: added peer, starting replication: peer=server-18d90f22-4c7a-b5d6-08af-5dba33fe2ed7
2025-01-28T14:54:31.834Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: added peer, starting replication: peer=server-82c19a4e-9157-25aa-f15b-126f6d879685
2025-01-28T14:54:31.835Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: pipelining replication: peer="{Voter server-18d90f22-4c7a-b5d6-08af-5dba33fe2ed7 18d90f22-4c7a-b5d6-08af-5dba33fe2ed7}"
2025-01-28T14:54:31.835Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: pipelining replication: peer="{Voter server-82c19a4e-9157-25aa-f15b-126f6d879685 82c19a4e-9157-25aa-f15b-126f6d879685}"
2025-01-28T14:54:32.840Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:31.834724939 +0000 UTC m=+57.509214899 (1 nodes), 508.145122ms from start of poll, 508.921209ms from cluster start. Timeout at 2025-01-28 14:54:32.839728838 +0000 UTC m=+58.514218858, 1.005003959s after stability
2025-01-28T14:54:32.841Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: starting snapshot up to: index=12
2025-01-28T14:54:32.841Z [INFO]  creating new snapshot: path=/tmp/raft1746061664/snapshots/2-12-1738076072841.tmp
2025-01-28T14:54:33.140Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: snapshot complete up to: index=12
2025-01-28T14:54:33.141Z [INFO]  creating new snapshot: path=/tmp/raft1746061664/snapshots/2-23-1738076073141.tmp
2025-01-28T14:54:33.143Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: copied to local snapshot: bytes=71
2025-01-28T14:54:33.143Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: snapshot restore progress: id=2-23-1738076073141 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:33.143Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: removing all old logs from log store
2025-01-28T14:54:33.143Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: compacting logs: from=1 to=22
2025-01-28T14:54:33.143Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: restored user snapshot: index=23
2025-01-28T14:54:33.144Z [ERROR] server-e3d2edfd-26c4-44da-e721-c910e7daeb59: failed to get log: index=22 error="log not found"
2025-01-28T14:54:33.144Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: aborting pipeline replication: peer="{Voter server-82c19a4e-9157-25aa-f15b-126f6d879685 82c19a4e-9157-25aa-f15b-126f6d879685}"
2025-01-28T14:54:33.144Z [ERROR] server-e3d2edfd-26c4-44da-e721-c910e7daeb59: failed to get log: index=22 error="log not found"
2025-01-28T14:54:33.144Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: aborting pipeline replication: peer="{Voter server-18d90f22-4c7a-b5d6-08af-5dba33fe2ed7 18d90f22-4c7a-b5d6-08af-5dba33fe2ed7}"
2025-01-28T14:54:33.152Z [ERROR] server-e3d2edfd-26c4-44da-e721-c910e7daeb59: failed to get log: index=22 error="log not found"
2025-01-28T14:54:33.153Z [INFO]  creating new snapshot: path=/tmp/raft2668722255/snapshots/2-23-1738076073153.tmp
2025-01-28T14:54:33.153Z [ERROR] server-e3d2edfd-26c4-44da-e721-c910e7daeb59: failed to get log: index=22 error="log not found"
2025-01-28T14:54:33.154Z [INFO]  server-82c19a4e-9157-25aa-f15b-126f6d879685: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:33.154Z [INFO]  creating new snapshot: path=/tmp/raft3123973954/snapshots/2-23-1738076073154.tmp
2025-01-28T14:54:33.155Z [INFO]  server-82c19a4e-9157-25aa-f15b-126f6d879685: copied to local snapshot: bytes=71
2025-01-28T14:54:33.155Z [INFO]  server-18d90f22-4c7a-b5d6-08af-5dba33fe2ed7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:33.155Z [INFO]  server-82c19a4e-9157-25aa-f15b-126f6d879685: snapshot restore progress: id=2-23-1738076073153 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:33.155Z [INFO]  server-82c19a4e-9157-25aa-f15b-126f6d879685: removing all old logs from log store
2025-01-28T14:54:33.155Z [INFO]  server-82c19a4e-9157-25aa-f15b-126f6d879685: compacting logs: from=1 to=22
2025-01-28T14:54:33.155Z [INFO]  server-82c19a4e-9157-25aa-f15b-126f6d879685: Installed remote snapshot
2025-01-28T14:54:33.155Z [INFO]  server-18d90f22-4c7a-b5d6-08af-5dba33fe2ed7: copied to local snapshot: bytes=71
2025-01-28T14:54:33.156Z [WARN]  server-82c19a4e-9157-25aa-f15b-126f6d879685: skipping application of old log: index=22
2025-01-28T14:54:33.156Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: pipelining replication: peer="{Voter server-82c19a4e-9157-25aa-f15b-126f6d879685 82c19a4e-9157-25aa-f15b-126f6d879685}"
2025-01-28T14:54:33.156Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:33.156Z [INFO]  server-18d90f22-4c7a-b5d6-08af-5dba33fe2ed7: snapshot restore progress: id=2-23-1738076073154 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:33.156Z [INFO]  server-18d90f22-4c7a-b5d6-08af-5dba33fe2ed7: removing all old logs from log store
2025-01-28T14:54:33.156Z [INFO]  server-18d90f22-4c7a-b5d6-08af-5dba33fe2ed7: compacting logs: from=1 to=22
2025-01-28T14:54:33.156Z [INFO]  server-18d90f22-4c7a-b5d6-08af-5dba33fe2ed7: Installed remote snapshot
2025-01-28T14:54:33.156Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: pipelining replication: peer="{Voter server-18d90f22-4c7a-b5d6-08af-5dba33fe2ed7 18d90f22-4c7a-b5d6-08af-5dba33fe2ed7}"
2025-01-28T14:54:34.162Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:33.156367398 +0000 UTC m=+58.830857358 (2 nodes), 2.32µs from start of poll, 1.830563668s from cluster start. Timeout at 2025-01-28 14:54:34.161404195 +0000 UTC m=+59.835894155, 1.005036797s after stability
2025-01-28T14:54:34.175Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: aborting pipeline replication: peer="{Voter server-82c19a4e-9157-25aa-f15b-126f6d879685 82c19a4e-9157-25aa-f15b-126f6d879685}"
2025-01-28T14:54:34.175Z [INFO]  server-e3d2edfd-26c4-44da-e721-c910e7daeb59: aborting pipeline replication: peer="{Voter server-18d90f22-4c7a-b5d6-08af-5dba33fe2ed7 18d90f22-4c7a-b5d6-08af-5dba33fe2ed7}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true
2025-01-28T14:54:34.180Z [INFO]  server-c9f5320d-c1a0-128f-0032-7168be7c2b60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9f5320d-c1a0-128f-0032-7168be7c2b60 Address:c9f5320d-c1a0-128f-0032-7168be7c2b60} {Suffrage:Voter ID:server-6784c6c9-f9ef-f762-7dc7-ff3d4810281f Address:6784c6c9-f9ef-f762-7dc7-ff3d4810281f} {Suffrage:Voter ID:server-5a912079-e29a-fd62-7b1c-5bd6bda950d4 Address:5a912079-e29a-fd62-7b1c-5bd6bda950d4}]"
2025-01-28T14:54:34.180Z [INFO]  server-c9f5320d-c1a0-128f-0032-7168be7c2b60: entering follower state: follower="Node at c9f5320d-c1a0-128f-0032-7168be7c2b60 [Follower]" leader-address= leader-id=
2025-01-28T14:54:34.180Z [INFO]  server-6784c6c9-f9ef-f762-7dc7-ff3d4810281f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9f5320d-c1a0-128f-0032-7168be7c2b60 Address:c9f5320d-c1a0-128f-0032-7168be7c2b60} {Suffrage:Voter ID:server-6784c6c9-f9ef-f762-7dc7-ff3d4810281f Address:6784c6c9-f9ef-f762-7dc7-ff3d4810281f} {Suffrage:Voter ID:server-5a912079-e29a-fd62-7b1c-5bd6bda950d4 Address:5a912079-e29a-fd62-7b1c-5bd6bda950d4}]"
2025-01-28T14:54:34.180Z [INFO]  server-6784c6c9-f9ef-f762-7dc7-ff3d4810281f: entering follower state: follower="Node at 6784c6c9-f9ef-f762-7dc7-ff3d4810281f [Follower]" leader-address= leader-id=
2025-01-28T14:54:34.180Z [INFO]  server-5a912079-e29a-fd62-7b1c-5bd6bda950d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c9f5320d-c1a0-128f-0032-7168be7c2b60 Address:c9f5320d-c1a0-128f-0032-7168be7c2b60} {Suffrage:Voter ID:server-6784c6c9-f9ef-f762-7dc7-ff3d4810281f Address:6784c6c9-f9ef-f762-7dc7-ff3d4810281f} {Suffrage:Voter ID:server-5a912079-e29a-fd62-7b1c-5bd6bda950d4 Address:5a912079-e29a-fd62-7b1c-5bd6bda950d4}]"
2025-01-28T14:54:34.180Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:34.180Z [INFO]  server-5a912079-e29a-fd62-7b1c-5bd6bda950d4: entering follower state: follower="Node at 5a912079-e29a-fd62-7b1c-5bd6bda950d4 [Follower]" leader-address= leader-id=
2025-01-28T14:54:34.776Z [WARN]  server-c9f5320d-c1a0-128f-0032-7168be7c2b60: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:34.776Z [INFO]  server-c9f5320d-c1a0-128f-0032-7168be7c2b60: entering candidate state: node="Node at c9f5320d-c1a0-128f-0032-7168be7c2b60 [Candidate]" term=2
2025-01-28T14:54:34.776Z [INFO]  server-c9f5320d-c1a0-128f-0032-7168be7c2b60: election won: term=2 tally=2
2025-01-28T14:54:34.776Z [INFO]  server-c9f5320d-c1a0-128f-0032-7168be7c2b60: entering leader state: leader="Node at c9f5320d-c1a0-128f-0032-7168be7c2b60 [Leader]"
2025-01-28T14:54:34.776Z [INFO]  server-c9f5320d-c1a0-128f-0032-7168be7c2b60: added peer, starting replication: peer=server-6784c6c9-f9ef-f762-7dc7-ff3d4810281f
2025-01-28T14:54:34.776Z [INFO]  server-c9f5320d-c1a0-128f-0032-7168be7c2b60: added peer, starting replication: peer=server-5a912079-e29a-fd62-7b1c-5bd6bda950d4
2025-01-28T14:54:34.776Z [INFO]  server-c9f5320d-c1a0-128f-0032-7168be7c2b60: pipelining replication: peer="{Voter server-6784c6c9-f9ef-f762-7dc7-ff3d4810281f 6784c6c9-f9ef-f762-7dc7-ff3d4810281f}"
2025-01-28T14:54:34.776Z [INFO]  server-c9f5320d-c1a0-128f-0032-7168be7c2b60: pipelining replication: peer="{Voter server-5a912079-e29a-fd62-7b1c-5bd6bda950d4 5a912079-e29a-fd62-7b1c-5bd6bda950d4}"
2025-01-28T14:54:35.781Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:34.776401166 +0000 UTC m=+60.450891126 (1 nodes), 595.583473ms from start of poll, 596.527104ms from cluster start. Timeout at 2025-01-28 14:54:35.781416724 +0000 UTC m=+61.455906704, 1.005015578s after stability
2025-01-28T14:54:35.785Z [INFO]  server-c9f5320d-c1a0-128f-0032-7168be7c2b60: starting snapshot up to: index=12
2025-01-28T14:54:35.785Z [INFO]  creating new snapshot: path=/tmp/raft671186994/snapshots/2-12-1738076075785.tmp
2025-01-28T14:54:35.787Z [INFO]  server-c9f5320d-c1a0-128f-0032-7168be7c2b60: snapshot complete up to: index=12
2025-01-28T14:54:35.790Z [INFO]  server-6c849148-cc31-4bf7-9366-7dca4454f9d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c849148-cc31-4bf7-9366-7dca4454f9d7 Address:6c849148-cc31-4bf7-9366-7dca4454f9d7} {Suffrage:Voter ID:server-7ec272ff-ab2e-ea24-4b02-d10bef5a7b09 Address:7ec272ff-ab2e-ea24-4b02-d10bef5a7b09} {Suffrage:Voter ID:server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7 Address:e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7}]"
2025-01-28T14:54:35.791Z [INFO]  server-6c849148-cc31-4bf7-9366-7dca4454f9d7: entering follower state: follower="Node at 6c849148-cc31-4bf7-9366-7dca4454f9d7 [Follower]" leader-address= leader-id=
2025-01-28T14:54:35.791Z [INFO]  server-7ec272ff-ab2e-ea24-4b02-d10bef5a7b09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c849148-cc31-4bf7-9366-7dca4454f9d7 Address:6c849148-cc31-4bf7-9366-7dca4454f9d7} {Suffrage:Voter ID:server-7ec272ff-ab2e-ea24-4b02-d10bef5a7b09 Address:7ec272ff-ab2e-ea24-4b02-d10bef5a7b09} {Suffrage:Voter ID:server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7 Address:e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7}]"
2025-01-28T14:54:35.791Z [INFO]  server-7ec272ff-ab2e-ea24-4b02-d10bef5a7b09: entering follower state: follower="Node at 7ec272ff-ab2e-ea24-4b02-d10bef5a7b09 [Follower]" leader-address= leader-id=
2025-01-28T14:54:35.791Z [INFO]  server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6c849148-cc31-4bf7-9366-7dca4454f9d7 Address:6c849148-cc31-4bf7-9366-7dca4454f9d7} {Suffrage:Voter ID:server-7ec272ff-ab2e-ea24-4b02-d10bef5a7b09 Address:7ec272ff-ab2e-ea24-4b02-d10bef5a7b09} {Suffrage:Voter ID:server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7 Address:e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7}]"
2025-01-28T14:54:35.791Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:35.791Z [INFO]  server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: entering follower state: follower="Node at e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7 [Follower]" leader-address= leader-id=
2025-01-28T14:54:36.368Z [WARN]  server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:36.368Z [INFO]  server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: entering candidate state: node="Node at e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7 [Candidate]" term=2
2025-01-28T14:54:36.369Z [INFO]  server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: election won: term=2 tally=2
2025-01-28T14:54:36.369Z [INFO]  server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: entering leader state: leader="Node at e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7 [Leader]"
2025-01-28T14:54:36.369Z [INFO]  server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: added peer, starting replication: peer=server-6c849148-cc31-4bf7-9366-7dca4454f9d7
2025-01-28T14:54:36.369Z [INFO]  server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: added peer, starting replication: peer=server-7ec272ff-ab2e-ea24-4b02-d10bef5a7b09
2025-01-28T14:54:36.369Z [INFO]  server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: pipelining replication: peer="{Voter server-6c849148-cc31-4bf7-9366-7dca4454f9d7 6c849148-cc31-4bf7-9366-7dca4454f9d7}"
2025-01-28T14:54:36.369Z [INFO]  server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: pipelining replication: peer="{Voter server-7ec272ff-ab2e-ea24-4b02-d10bef5a7b09 7ec272ff-ab2e-ea24-4b02-d10bef5a7b09}"
2025-01-28T14:54:37.374Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:36.369160728 +0000 UTC m=+62.043650708 (1 nodes), 577.462803ms from start of poll, 578.489227ms from cluster start. Timeout at 2025-01-28 14:54:37.374173207 +0000 UTC m=+63.048663227, 1.005012519s after stability
2025-01-28T14:54:37.375Z [INFO]  creating new snapshot: path=/tmp/raft3047435954/snapshots/2-14-1738076077375.tmp
2025-01-28T14:54:37.376Z [INFO]  server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: copied to local snapshot: bytes=71
2025-01-28T14:54:37.377Z [INFO]  server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: snapshot restore progress: id=2-14-1738076077375 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:37.377Z [INFO]  server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: restored user snapshot: index=14
2025-01-28T14:54:37.377Z [ERROR] server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: failed to get log: index=3 error="log not found"
2025-01-28T14:54:37.377Z [INFO]  server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: aborting pipeline replication: peer="{Voter server-7ec272ff-ab2e-ea24-4b02-d10bef5a7b09 7ec272ff-ab2e-ea24-4b02-d10bef5a7b09}"
2025-01-28T14:54:37.377Z [ERROR] server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: failed to get log: index=3 error="log not found"
2025-01-28T14:54:37.378Z [INFO]  server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: aborting pipeline replication: peer="{Voter server-6c849148-cc31-4bf7-9366-7dca4454f9d7 6c849148-cc31-4bf7-9366-7dca4454f9d7}"
2025-01-28T14:54:37.383Z [ERROR] server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: failed to get log: index=3 error="log not found"
2025-01-28T14:54:37.384Z [INFO]  creating new snapshot: path=/tmp/raft2901170872/snapshots/2-14-1738076077384.tmp
2025-01-28T14:54:37.385Z [INFO]  server-7ec272ff-ab2e-ea24-4b02-d10bef5a7b09: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:37.386Z [INFO]  server-7ec272ff-ab2e-ea24-4b02-d10bef5a7b09: copied to local snapshot: bytes=71
2025-01-28T14:54:37.387Z [ERROR] server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: failed to get log: index=3 error="log not found"
2025-01-28T14:54:37.387Z [INFO]  server-7ec272ff-ab2e-ea24-4b02-d10bef5a7b09: snapshot restore progress: id=2-14-1738076077384 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:37.387Z [INFO]  server-7ec272ff-ab2e-ea24-4b02-d10bef5a7b09: Installed remote snapshot
2025-01-28T14:54:37.388Z [INFO]  creating new snapshot: path=/tmp/raft296555598/snapshots/2-14-1738076077388.tmp
2025-01-28T14:54:37.388Z [INFO]  server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: pipelining replication: peer="{Voter server-7ec272ff-ab2e-ea24-4b02-d10bef5a7b09 7ec272ff-ab2e-ea24-4b02-d10bef5a7b09}"
2025-01-28T14:54:37.388Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:37.389Z [INFO]  server-6c849148-cc31-4bf7-9366-7dca4454f9d7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:37.389Z [INFO]  server-6c849148-cc31-4bf7-9366-7dca4454f9d7: copied to local snapshot: bytes=71
2025-01-28T14:54:37.390Z [INFO]  server-6c849148-cc31-4bf7-9366-7dca4454f9d7: snapshot restore progress: id=2-14-1738076077388 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:37.390Z [INFO]  server-6c849148-cc31-4bf7-9366-7dca4454f9d7: Installed remote snapshot
2025-01-28T14:54:37.391Z [INFO]  server-e4dc8683-71e7-b8c9-1c1c-d49653e8d5a7: pipelining replication: peer="{Voter server-6c849148-cc31-4bf7-9366-7dca4454f9d7 6c849148-cc31-4bf7-9366-7dca4454f9d7}"
2025-01-28T14:54:38.393Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:37.388434731 +0000 UTC m=+63.062924711 (2 nodes), 2.58µs from start of poll, 1.59776323s from cluster start. Timeout at 2025-01-28 14:54:38.393482946 +0000 UTC m=+64.067972926, 1.005048215s after stability
2025-01-28T14:54:38.411Z [INFO]  server-c9f5320d-c1a0-128f-0032-7168be7c2b60: aborting pipeline replication: peer="{Voter server-5a912079-e29a-fd62-7b1c-5bd6bda950d4 5a912079-e29a-fd62-7b1c-5bd6bda950d4}"
2025-01-28T14:54:38.411Z [INFO]  server-c9f5320d-c1a0-128f-0032-7168be7c2b60: aborting pipeline replication: peer="{Voter server-6784c6c9-f9ef-f762-7dc7-ff3d4810281f 6784c6c9-f9ef-f762-7dc7-ff3d4810281f}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true
2025-01-28T14:54:38.415Z [INFO]  server-d85694f6-9270-bc6d-dfa5-cdf46aa50b17: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d85694f6-9270-bc6d-dfa5-cdf46aa50b17 Address:d85694f6-9270-bc6d-dfa5-cdf46aa50b17} {Suffrage:Voter ID:server-2e7e0eb5-9008-2a88-2729-cf2a3139fb04 Address:2e7e0eb5-9008-2a88-2729-cf2a3139fb04} {Suffrage:Voter ID:server-e07cbf22-1478-ee4d-b495-9ede2565a1a4 Address:e07cbf22-1478-ee4d-b495-9ede2565a1a4}]"
2025-01-28T14:54:38.415Z [INFO]  server-d85694f6-9270-bc6d-dfa5-cdf46aa50b17: entering follower state: follower="Node at d85694f6-9270-bc6d-dfa5-cdf46aa50b17 [Follower]" leader-address= leader-id=
2025-01-28T14:54:38.415Z [INFO]  server-2e7e0eb5-9008-2a88-2729-cf2a3139fb04: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d85694f6-9270-bc6d-dfa5-cdf46aa50b17 Address:d85694f6-9270-bc6d-dfa5-cdf46aa50b17} {Suffrage:Voter ID:server-2e7e0eb5-9008-2a88-2729-cf2a3139fb04 Address:2e7e0eb5-9008-2a88-2729-cf2a3139fb04} {Suffrage:Voter ID:server-e07cbf22-1478-ee4d-b495-9ede2565a1a4 Address:e07cbf22-1478-ee4d-b495-9ede2565a1a4}]"
2025-01-28T14:54:38.415Z [INFO]  server-2e7e0eb5-9008-2a88-2729-cf2a3139fb04: entering follower state: follower="Node at 2e7e0eb5-9008-2a88-2729-cf2a3139fb04 [Follower]" leader-address= leader-id=
2025-01-28T14:54:38.415Z [INFO]  server-e07cbf22-1478-ee4d-b495-9ede2565a1a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d85694f6-9270-bc6d-dfa5-cdf46aa50b17 Address:d85694f6-9270-bc6d-dfa5-cdf46aa50b17} {Suffrage:Voter ID:server-2e7e0eb5-9008-2a88-2729-cf2a3139fb04 Address:2e7e0eb5-9008-2a88-2729-cf2a3139fb04} {Suffrage:Voter ID:server-e07cbf22-1478-ee4d-b495-9ede2565a1a4 Address:e07cbf22-1478-ee4d-b495-9ede2565a1a4}]"
2025-01-28T14:54:38.415Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:38.415Z [INFO]  server-e07cbf22-1478-ee4d-b495-9ede2565a1a4: entering follower state: follower="Node at e07cbf22-1478-ee4d-b495-9ede2565a1a4 [Follower]" leader-address= leader-id=
2025-01-28T14:54:38.932Z [WARN]  server-d85694f6-9270-bc6d-dfa5-cdf46aa50b17: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:38.932Z [INFO]  server-d85694f6-9270-bc6d-dfa5-cdf46aa50b17: entering candidate state: node="Node at d85694f6-9270-bc6d-dfa5-cdf46aa50b17 [Candidate]" term=2
2025-01-28T14:54:38.932Z [INFO]  server-d85694f6-9270-bc6d-dfa5-cdf46aa50b17: election won: term=2 tally=2
2025-01-28T14:54:38.932Z [INFO]  server-d85694f6-9270-bc6d-dfa5-cdf46aa50b17: entering leader state: leader="Node at d85694f6-9270-bc6d-dfa5-cdf46aa50b17 [Leader]"
2025-01-28T14:54:38.932Z [INFO]  server-d85694f6-9270-bc6d-dfa5-cdf46aa50b17: added peer, starting replication: peer=server-2e7e0eb5-9008-2a88-2729-cf2a3139fb04
2025-01-28T14:54:38.932Z [INFO]  server-d85694f6-9270-bc6d-dfa5-cdf46aa50b17: added peer, starting replication: peer=server-e07cbf22-1478-ee4d-b495-9ede2565a1a4
2025-01-28T14:54:38.932Z [INFO]  server-d85694f6-9270-bc6d-dfa5-cdf46aa50b17: pipelining replication: peer="{Voter server-e07cbf22-1478-ee4d-b495-9ede2565a1a4 e07cbf22-1478-ee4d-b495-9ede2565a1a4}"
2025-01-28T14:54:38.932Z [INFO]  server-d85694f6-9270-bc6d-dfa5-cdf46aa50b17: pipelining replication: peer="{Voter server-2e7e0eb5-9008-2a88-2729-cf2a3139fb04 2e7e0eb5-9008-2a88-2729-cf2a3139fb04}"
2025-01-28T14:54:39.938Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:38.932510886 +0000 UTC m=+64.607000866 (1 nodes), 516.517313ms from start of poll, 517.61211ms from cluster start. Timeout at 2025-01-28 14:54:39.937522444 +0000 UTC m=+65.612012424, 1.005011558s after stability
2025-01-28T14:54:39.939Z [INFO]  server-d85694f6-9270-bc6d-dfa5-cdf46aa50b17: starting snapshot up to: index=12
2025-01-28T14:54:39.939Z [INFO]  creating new snapshot: path=/tmp/raft1876669511/snapshots/2-12-1738076079939.tmp
2025-01-28T14:54:39.941Z [INFO]  server-d85694f6-9270-bc6d-dfa5-cdf46aa50b17: snapshot complete up to: index=12
2025-01-28T14:54:39.944Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a958917-308c-5240-b5d2-0ea19e5f7840 Address:6a958917-308c-5240-b5d2-0ea19e5f7840} {Suffrage:Voter ID:server-4a507f22-e57a-19c4-cdee-16ed98661d9c Address:4a507f22-e57a-19c4-cdee-16ed98661d9c} {Suffrage:Voter ID:server-720e4830-346e-712f-d45b-967af9c1b10d Address:720e4830-346e-712f-d45b-967af9c1b10d}]"
2025-01-28T14:54:39.944Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: entering follower state: follower="Node at 6a958917-308c-5240-b5d2-0ea19e5f7840 [Follower]" leader-address= leader-id=
2025-01-28T14:54:39.944Z [INFO]  server-4a507f22-e57a-19c4-cdee-16ed98661d9c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a958917-308c-5240-b5d2-0ea19e5f7840 Address:6a958917-308c-5240-b5d2-0ea19e5f7840} {Suffrage:Voter ID:server-4a507f22-e57a-19c4-cdee-16ed98661d9c Address:4a507f22-e57a-19c4-cdee-16ed98661d9c} {Suffrage:Voter ID:server-720e4830-346e-712f-d45b-967af9c1b10d Address:720e4830-346e-712f-d45b-967af9c1b10d}]"
2025-01-28T14:54:39.944Z [INFO]  server-4a507f22-e57a-19c4-cdee-16ed98661d9c: entering follower state: follower="Node at 4a507f22-e57a-19c4-cdee-16ed98661d9c [Follower]" leader-address= leader-id=
2025-01-28T14:54:39.944Z [INFO]  server-720e4830-346e-712f-d45b-967af9c1b10d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6a958917-308c-5240-b5d2-0ea19e5f7840 Address:6a958917-308c-5240-b5d2-0ea19e5f7840} {Suffrage:Voter ID:server-4a507f22-e57a-19c4-cdee-16ed98661d9c Address:4a507f22-e57a-19c4-cdee-16ed98661d9c} {Suffrage:Voter ID:server-720e4830-346e-712f-d45b-967af9c1b10d Address:720e4830-346e-712f-d45b-967af9c1b10d}]"
2025-01-28T14:54:39.944Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:39.944Z [INFO]  server-720e4830-346e-712f-d45b-967af9c1b10d: entering follower state: follower="Node at 720e4830-346e-712f-d45b-967af9c1b10d [Follower]" leader-address= leader-id=
2025-01-28T14:54:40.622Z [WARN]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:40.622Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: entering candidate state: node="Node at 6a958917-308c-5240-b5d2-0ea19e5f7840 [Candidate]" term=2
2025-01-28T14:54:40.622Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: election won: term=2 tally=2
2025-01-28T14:54:40.622Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: entering leader state: leader="Node at 6a958917-308c-5240-b5d2-0ea19e5f7840 [Leader]"
2025-01-28T14:54:40.622Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: added peer, starting replication: peer=server-4a507f22-e57a-19c4-cdee-16ed98661d9c
2025-01-28T14:54:40.623Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: added peer, starting replication: peer=server-720e4830-346e-712f-d45b-967af9c1b10d
2025-01-28T14:54:40.623Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: pipelining replication: peer="{Voter server-720e4830-346e-712f-d45b-967af9c1b10d 720e4830-346e-712f-d45b-967af9c1b10d}"
2025-01-28T14:54:40.623Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: pipelining replication: peer="{Voter server-4a507f22-e57a-19c4-cdee-16ed98661d9c 4a507f22-e57a-19c4-cdee-16ed98661d9c}"
2025-01-28T14:54:41.628Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:40.623015186 +0000 UTC m=+66.297505146 (1 nodes), 678.067072ms from start of poll, 679.032241ms from cluster start. Timeout at 2025-01-28 14:54:41.628021425 +0000 UTC m=+67.302511485, 1.005006339s after stability
2025-01-28T14:54:41.628Z [INFO]  creating new snapshot: path=/tmp/raft4177038571/snapshots/2-14-1738076081628.tmp
2025-01-28T14:54:41.630Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: copied to local snapshot: bytes=71
2025-01-28T14:54:41.630Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: snapshot restore progress: id=2-14-1738076081628 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:41.631Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: removing all old logs from log store
2025-01-28T14:54:41.631Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: compacting logs: from=1 to=2
2025-01-28T14:54:41.631Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: restored user snapshot: index=14
2025-01-28T14:54:41.631Z [ERROR] server-6a958917-308c-5240-b5d2-0ea19e5f7840: failed to get log: index=2 error="log not found"
2025-01-28T14:54:41.631Z [ERROR] server-6a958917-308c-5240-b5d2-0ea19e5f7840: failed to get log: index=2 error="log not found"
2025-01-28T14:54:41.631Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: aborting pipeline replication: peer="{Voter server-4a507f22-e57a-19c4-cdee-16ed98661d9c 4a507f22-e57a-19c4-cdee-16ed98661d9c}"
2025-01-28T14:54:41.631Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: aborting pipeline replication: peer="{Voter server-720e4830-346e-712f-d45b-967af9c1b10d 720e4830-346e-712f-d45b-967af9c1b10d}"
2025-01-28T14:54:41.641Z [ERROR] server-6a958917-308c-5240-b5d2-0ea19e5f7840: failed to get log: index=2 error="log not found"
2025-01-28T14:54:41.641Z [ERROR] server-6a958917-308c-5240-b5d2-0ea19e5f7840: failed to get log: index=2 error="log not found"
2025-01-28T14:54:41.642Z [INFO]  creating new snapshot: path=/tmp/raft726200438/snapshots/2-14-1738076081642.tmp
2025-01-28T14:54:41.642Z [INFO]  creating new snapshot: path=/tmp/raft1142058795/snapshots/2-14-1738076081642.tmp
2025-01-28T14:54:41.643Z [INFO]  server-720e4830-346e-712f-d45b-967af9c1b10d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:41.643Z [INFO]  server-4a507f22-e57a-19c4-cdee-16ed98661d9c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:41.643Z [INFO]  server-4a507f22-e57a-19c4-cdee-16ed98661d9c: copied to local snapshot: bytes=71
2025-01-28T14:54:41.643Z [INFO]  server-720e4830-346e-712f-d45b-967af9c1b10d: copied to local snapshot: bytes=71
2025-01-28T14:54:41.644Z [INFO]  server-4a507f22-e57a-19c4-cdee-16ed98661d9c: snapshot restore progress: id=2-14-1738076081642 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:41.644Z [INFO]  server-4a507f22-e57a-19c4-cdee-16ed98661d9c: removing all old logs from log store
2025-01-28T14:54:41.644Z [INFO]  server-4a507f22-e57a-19c4-cdee-16ed98661d9c: compacting logs: from=1 to=2
2025-01-28T14:54:41.644Z [INFO]  server-4a507f22-e57a-19c4-cdee-16ed98661d9c: Installed remote snapshot
2025-01-28T14:54:41.644Z [INFO]  server-720e4830-346e-712f-d45b-967af9c1b10d: snapshot restore progress: id=2-14-1738076081642 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:41.644Z [INFO]  server-720e4830-346e-712f-d45b-967af9c1b10d: removing all old logs from log store
2025-01-28T14:54:41.644Z [INFO]  server-720e4830-346e-712f-d45b-967af9c1b10d: compacting logs: from=1 to=2
2025-01-28T14:54:41.644Z [INFO]  server-720e4830-346e-712f-d45b-967af9c1b10d: Installed remote snapshot
2025-01-28T14:54:41.644Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: pipelining replication: peer="{Voter server-4a507f22-e57a-19c4-cdee-16ed98661d9c 4a507f22-e57a-19c4-cdee-16ed98661d9c}"
2025-01-28T14:54:41.644Z [INFO]  server-6a958917-308c-5240-b5d2-0ea19e5f7840: pipelining replication: peer="{Voter server-720e4830-346e-712f-d45b-967af9c1b10d 720e4830-346e-712f-d45b-967af9c1b10d}"
2025-01-28T14:54:41.644Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:42.650Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:41.64483655 +0000 UTC m=+67.319326530 (2 nodes), 2.32µs from start of poll, 1.700853625s from cluster start. Timeout at 2025-01-28 14:54:42.649896385 +0000 UTC m=+68.324386405, 1.005059875s after stability
2025-01-28T14:54:42.662Z [INFO]  server-d85694f6-9270-bc6d-dfa5-cdf46aa50b17: aborting pipeline replication: peer="{Voter server-e07cbf22-1478-ee4d-b495-9ede2565a1a4 e07cbf22-1478-ee4d-b495-9ede2565a1a4}"
2025-01-28T14:54:42.662Z [INFO]  server-d85694f6-9270-bc6d-dfa5-cdf46aa50b17: aborting pipeline replication: peer="{Voter server-2e7e0eb5-9008-2a88-2729-cf2a3139fb04 2e7e0eb5-9008-2a88-2729-cf2a3139fb04}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false
2025-01-28T14:54:42.668Z [INFO]  server-8b04e788-ec63-8cae-cc75-0aac69070353: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b04e788-ec63-8cae-cc75-0aac69070353 Address:8b04e788-ec63-8cae-cc75-0aac69070353} {Suffrage:Voter ID:server-2050504c-cc88-a166-4194-a85f5a64e661 Address:2050504c-cc88-a166-4194-a85f5a64e661} {Suffrage:Voter ID:server-cd12e50e-bd0a-d539-b229-a440b0aa9b44 Address:cd12e50e-bd0a-d539-b229-a440b0aa9b44}]"
2025-01-28T14:54:42.668Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b04e788-ec63-8cae-cc75-0aac69070353 Address:8b04e788-ec63-8cae-cc75-0aac69070353} {Suffrage:Voter ID:server-2050504c-cc88-a166-4194-a85f5a64e661 Address:2050504c-cc88-a166-4194-a85f5a64e661} {Suffrage:Voter ID:server-cd12e50e-bd0a-d539-b229-a440b0aa9b44 Address:cd12e50e-bd0a-d539-b229-a440b0aa9b44}]"
2025-01-28T14:54:42.669Z [INFO]  server-cd12e50e-bd0a-d539-b229-a440b0aa9b44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8b04e788-ec63-8cae-cc75-0aac69070353 Address:8b04e788-ec63-8cae-cc75-0aac69070353} {Suffrage:Voter ID:server-2050504c-cc88-a166-4194-a85f5a64e661 Address:2050504c-cc88-a166-4194-a85f5a64e661} {Suffrage:Voter ID:server-cd12e50e-bd0a-d539-b229-a440b0aa9b44 Address:cd12e50e-bd0a-d539-b229-a440b0aa9b44}]"
2025-01-28T14:54:42.669Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:42.669Z [INFO]  server-8b04e788-ec63-8cae-cc75-0aac69070353: entering follower state: follower="Node at 8b04e788-ec63-8cae-cc75-0aac69070353 [Follower]" leader-address= leader-id=
2025-01-28T14:54:42.669Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: entering follower state: follower="Node at 2050504c-cc88-a166-4194-a85f5a64e661 [Follower]" leader-address= leader-id=
2025-01-28T14:54:42.669Z [INFO]  server-cd12e50e-bd0a-d539-b229-a440b0aa9b44: entering follower state: follower="Node at cd12e50e-bd0a-d539-b229-a440b0aa9b44 [Follower]" leader-address= leader-id=
2025-01-28T14:54:43.350Z [WARN]  server-2050504c-cc88-a166-4194-a85f5a64e661: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:43.350Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: entering candidate state: node="Node at 2050504c-cc88-a166-4194-a85f5a64e661 [Candidate]" term=2
2025-01-28T14:54:43.350Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: election won: term=2 tally=2
2025-01-28T14:54:43.350Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: entering leader state: leader="Node at 2050504c-cc88-a166-4194-a85f5a64e661 [Leader]"
2025-01-28T14:54:43.350Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: added peer, starting replication: peer=server-8b04e788-ec63-8cae-cc75-0aac69070353
2025-01-28T14:54:43.350Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: added peer, starting replication: peer=server-cd12e50e-bd0a-d539-b229-a440b0aa9b44
2025-01-28T14:54:43.350Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: pipelining replication: peer="{Voter server-cd12e50e-bd0a-d539-b229-a440b0aa9b44 cd12e50e-bd0a-d539-b229-a440b0aa9b44}"
2025-01-28T14:54:43.351Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: pipelining replication: peer="{Voter server-8b04e788-ec63-8cae-cc75-0aac69070353 8b04e788-ec63-8cae-cc75-0aac69070353}"
2025-01-28T14:54:44.355Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:43.350661725 +0000 UTC m=+69.025151705 (1 nodes), 681.447373ms from start of poll, 682.641921ms from cluster start. Timeout at 2025-01-28 14:54:44.355667305 +0000 UTC m=+70.030157345, 1.00500564s after stability
2025-01-28T14:54:44.356Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: starting snapshot up to: index=12
2025-01-28T14:54:44.356Z [INFO]  creating new snapshot: path=/tmp/raft188398636/snapshots/2-12-1738076084356.tmp
2025-01-28T14:54:44.358Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: snapshot complete up to: index=12
2025-01-28T14:54:44.359Z [INFO]  creating new snapshot: path=/tmp/raft188398636/snapshots/2-23-1738076084359.tmp
2025-01-28T14:54:44.361Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: copied to local snapshot: bytes=71
2025-01-28T14:54:44.361Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: snapshot restore progress: id=2-23-1738076084359 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:44.361Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: restored user snapshot: index=23
2025-01-28T14:54:44.361Z [ERROR] server-2050504c-cc88-a166-4194-a85f5a64e661: failed to get log: index=23 error="log not found"
2025-01-28T14:54:44.362Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: aborting pipeline replication: peer="{Voter server-8b04e788-ec63-8cae-cc75-0aac69070353 8b04e788-ec63-8cae-cc75-0aac69070353}"
2025-01-28T14:54:44.361Z [ERROR] server-2050504c-cc88-a166-4194-a85f5a64e661: failed to get log: index=23 error="log not found"
2025-01-28T14:54:44.362Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: aborting pipeline replication: peer="{Voter server-cd12e50e-bd0a-d539-b229-a440b0aa9b44 cd12e50e-bd0a-d539-b229-a440b0aa9b44}"
2025-01-28T14:54:44.371Z [ERROR] server-2050504c-cc88-a166-4194-a85f5a64e661: failed to get log: index=23 error="log not found"
2025-01-28T14:54:44.372Z [INFO]  creating new snapshot: path=/tmp/raft3424123677/snapshots/2-23-1738076084372.tmp
2025-01-28T14:54:44.372Z [ERROR] server-2050504c-cc88-a166-4194-a85f5a64e661: failed to get log: index=23 error="log not found"
2025-01-28T14:54:44.373Z [INFO]  server-8b04e788-ec63-8cae-cc75-0aac69070353: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:44.373Z [INFO]  creating new snapshot: path=/tmp/raft3075059098/snapshots/2-23-1738076084373.tmp
2025-01-28T14:54:44.373Z [INFO]  server-8b04e788-ec63-8cae-cc75-0aac69070353: copied to local snapshot: bytes=71
2025-01-28T14:54:44.373Z [INFO]  server-cd12e50e-bd0a-d539-b229-a440b0aa9b44: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:44.374Z [INFO]  server-8b04e788-ec63-8cae-cc75-0aac69070353: snapshot restore progress: id=2-23-1738076084372 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:44.374Z [INFO]  server-8b04e788-ec63-8cae-cc75-0aac69070353: Installed remote snapshot
2025-01-28T14:54:44.374Z [WARN]  server-8b04e788-ec63-8cae-cc75-0aac69070353: skipping application of old log: index=22
2025-01-28T14:54:44.374Z [INFO]  server-cd12e50e-bd0a-d539-b229-a440b0aa9b44: copied to local snapshot: bytes=71
2025-01-28T14:54:44.374Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: pipelining replication: peer="{Voter server-8b04e788-ec63-8cae-cc75-0aac69070353 8b04e788-ec63-8cae-cc75-0aac69070353}"
2025-01-28T14:54:44.374Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:44.375Z [INFO]  server-cd12e50e-bd0a-d539-b229-a440b0aa9b44: snapshot restore progress: id=2-23-1738076084373 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:44.375Z [INFO]  server-cd12e50e-bd0a-d539-b229-a440b0aa9b44: Installed remote snapshot
2025-01-28T14:54:44.375Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: pipelining replication: peer="{Voter server-cd12e50e-bd0a-d539-b229-a440b0aa9b44 cd12e50e-bd0a-d539-b229-a440b0aa9b44}"
2025-01-28T14:54:45.380Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:44.374991072 +0000 UTC m=+70.049481052 (2 nodes), 2.48µs from start of poll, 1.706971268s from cluster start. Timeout at 2025-01-28 14:54:45.380044847 +0000 UTC m=+71.054534827, 1.005053775s after stability
2025-01-28T14:54:45.393Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: aborting pipeline replication: peer="{Voter server-cd12e50e-bd0a-d539-b229-a440b0aa9b44 cd12e50e-bd0a-d539-b229-a440b0aa9b44}"
2025-01-28T14:54:45.393Z [INFO]  server-2050504c-cc88-a166-4194-a85f5a64e661: aborting pipeline replication: peer="{Voter server-8b04e788-ec63-8cae-cc75-0aac69070353 8b04e788-ec63-8cae-cc75-0aac69070353}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false
2025-01-28T14:54:45.398Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4790534a-3157-b26e-07f4-dee3e0c556e1 Address:4790534a-3157-b26e-07f4-dee3e0c556e1} {Suffrage:Voter ID:server-0a3c39e7-7a7a-1dd4-ceac-0bbe87744006 Address:0a3c39e7-7a7a-1dd4-ceac-0bbe87744006} {Suffrage:Voter ID:server-2887f76f-7354-7282-f73c-f12c386312f1 Address:2887f76f-7354-7282-f73c-f12c386312f1}]"
2025-01-28T14:54:45.399Z [INFO]  server-0a3c39e7-7a7a-1dd4-ceac-0bbe87744006: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4790534a-3157-b26e-07f4-dee3e0c556e1 Address:4790534a-3157-b26e-07f4-dee3e0c556e1} {Suffrage:Voter ID:server-0a3c39e7-7a7a-1dd4-ceac-0bbe87744006 Address:0a3c39e7-7a7a-1dd4-ceac-0bbe87744006} {Suffrage:Voter ID:server-2887f76f-7354-7282-f73c-f12c386312f1 Address:2887f76f-7354-7282-f73c-f12c386312f1}]"
2025-01-28T14:54:45.400Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: entering follower state: follower="Node at 4790534a-3157-b26e-07f4-dee3e0c556e1 [Follower]" leader-address= leader-id=
2025-01-28T14:54:45.400Z [INFO]  server-2887f76f-7354-7282-f73c-f12c386312f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4790534a-3157-b26e-07f4-dee3e0c556e1 Address:4790534a-3157-b26e-07f4-dee3e0c556e1} {Suffrage:Voter ID:server-0a3c39e7-7a7a-1dd4-ceac-0bbe87744006 Address:0a3c39e7-7a7a-1dd4-ceac-0bbe87744006} {Suffrage:Voter ID:server-2887f76f-7354-7282-f73c-f12c386312f1 Address:2887f76f-7354-7282-f73c-f12c386312f1}]"
2025-01-28T14:54:45.400Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:45.400Z [INFO]  server-2887f76f-7354-7282-f73c-f12c386312f1: entering follower state: follower="Node at 2887f76f-7354-7282-f73c-f12c386312f1 [Follower]" leader-address= leader-id=
2025-01-28T14:54:45.400Z [INFO]  server-0a3c39e7-7a7a-1dd4-ceac-0bbe87744006: entering follower state: follower="Node at 0a3c39e7-7a7a-1dd4-ceac-0bbe87744006 [Follower]" leader-address= leader-id=
2025-01-28T14:54:45.946Z [WARN]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:45.946Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: entering candidate state: node="Node at 4790534a-3157-b26e-07f4-dee3e0c556e1 [Candidate]" term=2
2025-01-28T14:54:45.947Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: election won: term=2 tally=2
2025-01-28T14:54:45.947Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: entering leader state: leader="Node at 4790534a-3157-b26e-07f4-dee3e0c556e1 [Leader]"
2025-01-28T14:54:45.947Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: added peer, starting replication: peer=server-0a3c39e7-7a7a-1dd4-ceac-0bbe87744006
2025-01-28T14:54:45.947Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: added peer, starting replication: peer=server-2887f76f-7354-7282-f73c-f12c386312f1
2025-01-28T14:54:45.947Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: pipelining replication: peer="{Voter server-2887f76f-7354-7282-f73c-f12c386312f1 2887f76f-7354-7282-f73c-f12c386312f1}"
2025-01-28T14:54:45.947Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: pipelining replication: peer="{Voter server-0a3c39e7-7a7a-1dd4-ceac-0bbe87744006 0a3c39e7-7a7a-1dd4-ceac-0bbe87744006}"
2025-01-28T14:54:46.953Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:45.947399454 +0000 UTC m=+71.621889434 (1 nodes), 546.695806ms from start of poll, 549.905243ms from cluster start. Timeout at 2025-01-28 14:54:46.952410953 +0000 UTC m=+72.626900933, 1.005011499s after stability
2025-01-28T14:54:46.954Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: starting snapshot up to: index=12
2025-01-28T14:54:46.955Z [INFO]  creating new snapshot: path=/tmp/raft3698249626/snapshots/2-12-1738076086955.tmp
2025-01-28T14:54:46.956Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: snapshot complete up to: index=12
2025-01-28T14:54:46.958Z [INFO]  creating new snapshot: path=/tmp/raft3698249626/snapshots/2-23-1738076086958.tmp
2025-01-28T14:54:46.959Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: copied to local snapshot: bytes=71
2025-01-28T14:54:46.960Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: snapshot restore progress: id=2-23-1738076086958 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:46.960Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: removing all old logs from log store
2025-01-28T14:54:46.960Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: compacting logs: from=1 to=22
2025-01-28T14:54:46.960Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: restored user snapshot: index=23
2025-01-28T14:54:46.960Z [ERROR] server-4790534a-3157-b26e-07f4-dee3e0c556e1: failed to get log: index=22 error="log not found"
2025-01-28T14:54:46.960Z [ERROR] server-4790534a-3157-b26e-07f4-dee3e0c556e1: failed to get log: index=22 error="log not found"
2025-01-28T14:54:46.960Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: aborting pipeline replication: peer="{Voter server-0a3c39e7-7a7a-1dd4-ceac-0bbe87744006 0a3c39e7-7a7a-1dd4-ceac-0bbe87744006}"
2025-01-28T14:54:46.960Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: aborting pipeline replication: peer="{Voter server-2887f76f-7354-7282-f73c-f12c386312f1 2887f76f-7354-7282-f73c-f12c386312f1}"
2025-01-28T14:54:46.969Z [ERROR] server-4790534a-3157-b26e-07f4-dee3e0c556e1: failed to get log: index=22 error="log not found"
2025-01-28T14:54:46.970Z [ERROR] server-4790534a-3157-b26e-07f4-dee3e0c556e1: failed to get log: index=22 error="log not found"
2025-01-28T14:54:46.970Z [INFO]  creating new snapshot: path=/tmp/raft1331227272/snapshots/2-23-1738076086970.tmp
2025-01-28T14:54:46.970Z [INFO]  creating new snapshot: path=/tmp/raft486685482/snapshots/2-23-1738076086970.tmp
2025-01-28T14:54:46.971Z [INFO]  server-0a3c39e7-7a7a-1dd4-ceac-0bbe87744006: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:46.972Z [INFO]  server-2887f76f-7354-7282-f73c-f12c386312f1: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:46.972Z [INFO]  server-2887f76f-7354-7282-f73c-f12c386312f1: copied to local snapshot: bytes=71
2025-01-28T14:54:46.972Z [INFO]  server-0a3c39e7-7a7a-1dd4-ceac-0bbe87744006: copied to local snapshot: bytes=71
2025-01-28T14:54:46.974Z [INFO]  server-2887f76f-7354-7282-f73c-f12c386312f1: snapshot restore progress: id=2-23-1738076086970 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:46.974Z [INFO]  server-0a3c39e7-7a7a-1dd4-ceac-0bbe87744006: snapshot restore progress: id=2-23-1738076086970 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:46.974Z [INFO]  server-0a3c39e7-7a7a-1dd4-ceac-0bbe87744006: removing all old logs from log store
2025-01-28T14:54:46.974Z [INFO]  server-0a3c39e7-7a7a-1dd4-ceac-0bbe87744006: compacting logs: from=1 to=22
2025-01-28T14:54:46.974Z [INFO]  server-0a3c39e7-7a7a-1dd4-ceac-0bbe87744006: Installed remote snapshot
2025-01-28T14:54:46.974Z [INFO]  server-2887f76f-7354-7282-f73c-f12c386312f1: removing all old logs from log store
2025-01-28T14:54:46.974Z [INFO]  server-2887f76f-7354-7282-f73c-f12c386312f1: compacting logs: from=1 to=22
2025-01-28T14:54:46.974Z [INFO]  server-2887f76f-7354-7282-f73c-f12c386312f1: Installed remote snapshot
2025-01-28T14:54:46.974Z [WARN]  server-0a3c39e7-7a7a-1dd4-ceac-0bbe87744006: skipping application of old log: index=22
2025-01-28T14:54:46.975Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: pipelining replication: peer="{Voter server-0a3c39e7-7a7a-1dd4-ceac-0bbe87744006 0a3c39e7-7a7a-1dd4-ceac-0bbe87744006}"
2025-01-28T14:54:46.975Z [WARN]  server-2887f76f-7354-7282-f73c-f12c386312f1: skipping application of old log: index=22
2025-01-28T14:54:46.975Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:46.975Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: pipelining replication: peer="{Voter server-2887f76f-7354-7282-f73c-f12c386312f1 2887f76f-7354-7282-f73c-f12c386312f1}"
2025-01-28T14:54:47.981Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:46.975348779 +0000 UTC m=+72.649838759 (2 nodes), 2.659µs from start of poll, 1.577854568s from cluster start. Timeout at 2025-01-28 14:54:47.980422652 +0000 UTC m=+73.654912712, 1.005073953s after stability
2025-01-28T14:54:47.992Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: aborting pipeline replication: peer="{Voter server-2887f76f-7354-7282-f73c-f12c386312f1 2887f76f-7354-7282-f73c-f12c386312f1}"
2025-01-28T14:54:47.992Z [INFO]  server-4790534a-3157-b26e-07f4-dee3e0c556e1: aborting pipeline replication: peer="{Voter server-0a3c39e7-7a7a-1dd4-ceac-0bbe87744006 0a3c39e7-7a7a-1dd4-ceac-0bbe87744006}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true
2025-01-28T14:54:47.998Z [INFO]  server-d13178ff-0d53-802d-08d0-32f53f675c97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d13178ff-0d53-802d-08d0-32f53f675c97 Address:d13178ff-0d53-802d-08d0-32f53f675c97} {Suffrage:Voter ID:server-72d04ff1-e490-ae1e-1a02-ecf28d1e6be0 Address:72d04ff1-e490-ae1e-1a02-ecf28d1e6be0} {Suffrage:Voter ID:server-128d5b43-dd5c-8edc-0b80-6485ef0f2ad6 Address:128d5b43-dd5c-8edc-0b80-6485ef0f2ad6}]"
2025-01-28T14:54:47.998Z [INFO]  server-d13178ff-0d53-802d-08d0-32f53f675c97: entering follower state: follower="Node at d13178ff-0d53-802d-08d0-32f53f675c97 [Follower]" leader-address= leader-id=
2025-01-28T14:54:47.998Z [INFO]  server-72d04ff1-e490-ae1e-1a02-ecf28d1e6be0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d13178ff-0d53-802d-08d0-32f53f675c97 Address:d13178ff-0d53-802d-08d0-32f53f675c97} {Suffrage:Voter ID:server-72d04ff1-e490-ae1e-1a02-ecf28d1e6be0 Address:72d04ff1-e490-ae1e-1a02-ecf28d1e6be0} {Suffrage:Voter ID:server-128d5b43-dd5c-8edc-0b80-6485ef0f2ad6 Address:128d5b43-dd5c-8edc-0b80-6485ef0f2ad6}]"
2025-01-28T14:54:47.999Z [INFO]  server-72d04ff1-e490-ae1e-1a02-ecf28d1e6be0: entering follower state: follower="Node at 72d04ff1-e490-ae1e-1a02-ecf28d1e6be0 [Follower]" leader-address= leader-id=
2025-01-28T14:54:47.999Z [INFO]  server-128d5b43-dd5c-8edc-0b80-6485ef0f2ad6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d13178ff-0d53-802d-08d0-32f53f675c97 Address:d13178ff-0d53-802d-08d0-32f53f675c97} {Suffrage:Voter ID:server-72d04ff1-e490-ae1e-1a02-ecf28d1e6be0 Address:72d04ff1-e490-ae1e-1a02-ecf28d1e6be0} {Suffrage:Voter ID:server-128d5b43-dd5c-8edc-0b80-6485ef0f2ad6 Address:128d5b43-dd5c-8edc-0b80-6485ef0f2ad6}]"
2025-01-28T14:54:47.999Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:47.999Z [INFO]  server-128d5b43-dd5c-8edc-0b80-6485ef0f2ad6: entering follower state: follower="Node at 128d5b43-dd5c-8edc-0b80-6485ef0f2ad6 [Follower]" leader-address= leader-id=
2025-01-28T14:54:48.698Z [WARN]  server-128d5b43-dd5c-8edc-0b80-6485ef0f2ad6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:48.698Z [INFO]  server-128d5b43-dd5c-8edc-0b80-6485ef0f2ad6: entering candidate state: node="Node at 128d5b43-dd5c-8edc-0b80-6485ef0f2ad6 [Candidate]" term=2
2025-01-28T14:54:48.698Z [INFO]  server-128d5b43-dd5c-8edc-0b80-6485ef0f2ad6: election won: term=2 tally=2
2025-01-28T14:54:48.698Z [INFO]  server-128d5b43-dd5c-8edc-0b80-6485ef0f2ad6: entering leader state: leader="Node at 128d5b43-dd5c-8edc-0b80-6485ef0f2ad6 [Leader]"
2025-01-28T14:54:48.698Z [INFO]  server-128d5b43-dd5c-8edc-0b80-6485ef0f2ad6: added peer, starting replication: peer=server-d13178ff-0d53-802d-08d0-32f53f675c97
2025-01-28T14:54:48.698Z [INFO]  server-128d5b43-dd5c-8edc-0b80-6485ef0f2ad6: added peer, starting replication: peer=server-72d04ff1-e490-ae1e-1a02-ecf28d1e6be0
2025-01-28T14:54:48.698Z [INFO]  server-128d5b43-dd5c-8edc-0b80-6485ef0f2ad6: pipelining replication: peer="{Voter server-d13178ff-0d53-802d-08d0-32f53f675c97 d13178ff-0d53-802d-08d0-32f53f675c97}"
2025-01-28T14:54:48.698Z [INFO]  server-128d5b43-dd5c-8edc-0b80-6485ef0f2ad6: pipelining replication: peer="{Voter server-72d04ff1-e490-ae1e-1a02-ecf28d1e6be0 72d04ff1-e490-ae1e-1a02-ecf28d1e6be0}"
2025-01-28T14:54:49.704Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:48.698670585 +0000 UTC m=+74.373160585 (1 nodes), 698.747442ms from start of poll, 700.653862ms from cluster start. Timeout at 2025-01-28 14:54:49.703677605 +0000 UTC m=+75.378167665, 1.00500708s after stability
2025-01-28T14:54:49.705Z [INFO]  server-128d5b43-dd5c-8edc-0b80-6485ef0f2ad6: starting snapshot up to: index=12
2025-01-28T14:54:49.706Z [INFO]  creating new snapshot: path=/tmp/raft3000744062/snapshots/2-12-1738076089706.tmp
2025-01-28T14:54:49.774Z [INFO]  server-128d5b43-dd5c-8edc-0b80-6485ef0f2ad6: snapshot complete up to: index=12
2025-01-28T14:54:49.778Z [INFO]  server-bb3db447-5cdd-3761-3e19-7628726f0fae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb3db447-5cdd-3761-3e19-7628726f0fae Address:bb3db447-5cdd-3761-3e19-7628726f0fae} {Suffrage:Voter ID:server-972471b3-7b48-7d02-2c42-8a3273317c9a Address:972471b3-7b48-7d02-2c42-8a3273317c9a} {Suffrage:Voter ID:server-bcec3f66-5293-d022-29a2-4ac2cb5adc8b Address:bcec3f66-5293-d022-29a2-4ac2cb5adc8b}]"
2025-01-28T14:54:49.778Z [INFO]  server-bb3db447-5cdd-3761-3e19-7628726f0fae: entering follower state: follower="Node at bb3db447-5cdd-3761-3e19-7628726f0fae [Follower]" leader-address= leader-id=
2025-01-28T14:54:49.778Z [INFO]  server-972471b3-7b48-7d02-2c42-8a3273317c9a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb3db447-5cdd-3761-3e19-7628726f0fae Address:bb3db447-5cdd-3761-3e19-7628726f0fae} {Suffrage:Voter ID:server-972471b3-7b48-7d02-2c42-8a3273317c9a Address:972471b3-7b48-7d02-2c42-8a3273317c9a} {Suffrage:Voter ID:server-bcec3f66-5293-d022-29a2-4ac2cb5adc8b Address:bcec3f66-5293-d022-29a2-4ac2cb5adc8b}]"
2025-01-28T14:54:49.779Z [INFO]  server-972471b3-7b48-7d02-2c42-8a3273317c9a: entering follower state: follower="Node at 972471b3-7b48-7d02-2c42-8a3273317c9a [Follower]" leader-address= leader-id=
2025-01-28T14:54:49.779Z [INFO]  server-bcec3f66-5293-d022-29a2-4ac2cb5adc8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb3db447-5cdd-3761-3e19-7628726f0fae Address:bb3db447-5cdd-3761-3e19-7628726f0fae} {Suffrage:Voter ID:server-972471b3-7b48-7d02-2c42-8a3273317c9a Address:972471b3-7b48-7d02-2c42-8a3273317c9a} {Suffrage:Voter ID:server-bcec3f66-5293-d022-29a2-4ac2cb5adc8b Address:bcec3f66-5293-d022-29a2-4ac2cb5adc8b}]"
2025-01-28T14:54:49.779Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:49.779Z [INFO]  server-bcec3f66-5293-d022-29a2-4ac2cb5adc8b: entering follower state: follower="Node at bcec3f66-5293-d022-29a2-4ac2cb5adc8b [Follower]" leader-address= leader-id=
2025-01-28T14:54:50.376Z [WARN]  server-972471b3-7b48-7d02-2c42-8a3273317c9a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:50.377Z [INFO]  server-972471b3-7b48-7d02-2c42-8a3273317c9a: entering candidate state: node="Node at 972471b3-7b48-7d02-2c42-8a3273317c9a [Candidate]" term=2
2025-01-28T14:54:50.377Z [INFO]  server-972471b3-7b48-7d02-2c42-8a3273317c9a: election won: term=2 tally=2
2025-01-28T14:54:50.377Z [INFO]  server-972471b3-7b48-7d02-2c42-8a3273317c9a: entering leader state: leader="Node at 972471b3-7b48-7d02-2c42-8a3273317c9a [Leader]"
2025-01-28T14:54:50.377Z [INFO]  server-972471b3-7b48-7d02-2c42-8a3273317c9a: added peer, starting replication: peer=server-bb3db447-5cdd-3761-3e19-7628726f0fae
2025-01-28T14:54:50.377Z [INFO]  server-972471b3-7b48-7d02-2c42-8a3273317c9a: added peer, starting replication: peer=server-bcec3f66-5293-d022-29a2-4ac2cb5adc8b
2025-01-28T14:54:50.377Z [INFO]  server-972471b3-7b48-7d02-2c42-8a3273317c9a: pipelining replication: peer="{Voter server-bb3db447-5cdd-3761-3e19-7628726f0fae bb3db447-5cdd-3761-3e19-7628726f0fae}"
2025-01-28T14:54:50.377Z [INFO]  server-972471b3-7b48-7d02-2c42-8a3273317c9a: pipelining replication: peer="{Voter server-bcec3f66-5293-d022-29a2-4ac2cb5adc8b bcec3f66-5293-d022-29a2-4ac2cb5adc8b}"
2025-01-28T14:54:51.383Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:50.377420155 +0000 UTC m=+76.051910135 (1 nodes), 597.999466ms from start of poll, 599.293164ms from cluster start. Timeout at 2025-01-28 14:54:51.382424335 +0000 UTC m=+77.056914375, 1.00500424s after stability
2025-01-28T14:54:51.383Z [INFO]  creating new snapshot: path=/tmp/raft1648756860/snapshots/2-15-1738076091383.tmp
2025-01-28T14:54:51.385Z [INFO]  server-972471b3-7b48-7d02-2c42-8a3273317c9a: copied to local snapshot: bytes=71
2025-01-28T14:54:51.386Z [INFO]  server-972471b3-7b48-7d02-2c42-8a3273317c9a: snapshot restore progress: id=2-15-1738076091383 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:51.386Z [INFO]  server-972471b3-7b48-7d02-2c42-8a3273317c9a: restored user snapshot: index=15
2025-01-28T14:54:51.386Z [ERROR] server-972471b3-7b48-7d02-2c42-8a3273317c9a: failed to get log: index=3 error="log not found"
2025-01-28T14:54:51.386Z [INFO]  server-972471b3-7b48-7d02-2c42-8a3273317c9a: aborting pipeline replication: peer="{Voter server-bcec3f66-5293-d022-29a2-4ac2cb5adc8b bcec3f66-5293-d022-29a2-4ac2cb5adc8b}"
2025-01-28T14:54:51.386Z [ERROR] server-972471b3-7b48-7d02-2c42-8a3273317c9a: failed to get log: index=3 error="log not found"
2025-01-28T14:54:51.386Z [INFO]  server-972471b3-7b48-7d02-2c42-8a3273317c9a: aborting pipeline replication: peer="{Voter server-bb3db447-5cdd-3761-3e19-7628726f0fae bb3db447-5cdd-3761-3e19-7628726f0fae}"
2025-01-28T14:54:51.396Z [ERROR] server-972471b3-7b48-7d02-2c42-8a3273317c9a: failed to get log: index=3 error="log not found"
2025-01-28T14:54:51.397Z [INFO]  creating new snapshot: path=/tmp/raft624318904/snapshots/2-15-1738076091397.tmp
2025-01-28T14:54:51.397Z [ERROR] server-972471b3-7b48-7d02-2c42-8a3273317c9a: failed to get log: index=3 error="log not found"
2025-01-28T14:54:51.398Z [INFO]  creating new snapshot: path=/tmp/raft1980657660/snapshots/2-15-1738076091398.tmp
2025-01-28T14:54:51.398Z [INFO]  server-bb3db447-5cdd-3761-3e19-7628726f0fae: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:51.398Z [INFO]  server-bcec3f66-5293-d022-29a2-4ac2cb5adc8b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:51.399Z [INFO]  server-bb3db447-5cdd-3761-3e19-7628726f0fae: copied to local snapshot: bytes=71
2025-01-28T14:54:51.399Z [INFO]  server-bcec3f66-5293-d022-29a2-4ac2cb5adc8b: copied to local snapshot: bytes=71
2025-01-28T14:54:51.400Z [INFO]  server-bb3db447-5cdd-3761-3e19-7628726f0fae: snapshot restore progress: id=2-15-1738076091397 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:51.400Z [INFO]  server-bb3db447-5cdd-3761-3e19-7628726f0fae: Installed remote snapshot
2025-01-28T14:54:51.400Z [INFO]  server-bcec3f66-5293-d022-29a2-4ac2cb5adc8b: snapshot restore progress: id=2-15-1738076091398 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:51.400Z [INFO]  server-bcec3f66-5293-d022-29a2-4ac2cb5adc8b: Installed remote snapshot
2025-01-28T14:54:51.400Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:51.400Z [INFO]  server-972471b3-7b48-7d02-2c42-8a3273317c9a: pipelining replication: peer="{Voter server-bb3db447-5cdd-3761-3e19-7628726f0fae bb3db447-5cdd-3761-3e19-7628726f0fae}"
2025-01-28T14:54:51.401Z [INFO]  server-972471b3-7b48-7d02-2c42-8a3273317c9a: pipelining replication: peer="{Voter server-bcec3f66-5293-d022-29a2-4ac2cb5adc8b bcec3f66-5293-d022-29a2-4ac2cb5adc8b}"
2025-01-28T14:54:52.406Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:51.400862206 +0000 UTC m=+77.075352166 (2 nodes), 2.92µs from start of poll, 1.622735195s from cluster start. Timeout at 2025-01-28 14:54:52.405907282 +0000 UTC m=+78.080397262, 1.005045096s after stability
2025-01-28T14:54:52.418Z [INFO]  server-128d5b43-dd5c-8edc-0b80-6485ef0f2ad6: aborting pipeline replication: peer="{Voter server-d13178ff-0d53-802d-08d0-32f53f675c97 d13178ff-0d53-802d-08d0-32f53f675c97}"
2025-01-28T14:54:52.418Z [INFO]  server-128d5b43-dd5c-8edc-0b80-6485ef0f2ad6: aborting pipeline replication: peer="{Voter server-72d04ff1-e490-ae1e-1a02-ecf28d1e6be0 72d04ff1-e490-ae1e-1a02-ecf28d1e6be0}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true
2025-01-28T14:54:52.422Z [INFO]  server-ede50f2d-5fb1-af3a-0a15-465889c97889: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ede50f2d-5fb1-af3a-0a15-465889c97889 Address:ede50f2d-5fb1-af3a-0a15-465889c97889} {Suffrage:Voter ID:server-8812df09-ccb8-6aab-ebb7-734d1399fcdc Address:8812df09-ccb8-6aab-ebb7-734d1399fcdc} {Suffrage:Voter ID:server-20cc3556-5843-71c7-21cc-de081e2e48fa Address:20cc3556-5843-71c7-21cc-de081e2e48fa}]"
2025-01-28T14:54:52.422Z [INFO]  server-ede50f2d-5fb1-af3a-0a15-465889c97889: entering follower state: follower="Node at ede50f2d-5fb1-af3a-0a15-465889c97889 [Follower]" leader-address= leader-id=
2025-01-28T14:54:52.422Z [INFO]  server-8812df09-ccb8-6aab-ebb7-734d1399fcdc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ede50f2d-5fb1-af3a-0a15-465889c97889 Address:ede50f2d-5fb1-af3a-0a15-465889c97889} {Suffrage:Voter ID:server-8812df09-ccb8-6aab-ebb7-734d1399fcdc Address:8812df09-ccb8-6aab-ebb7-734d1399fcdc} {Suffrage:Voter ID:server-20cc3556-5843-71c7-21cc-de081e2e48fa Address:20cc3556-5843-71c7-21cc-de081e2e48fa}]"
2025-01-28T14:54:52.423Z [INFO]  server-8812df09-ccb8-6aab-ebb7-734d1399fcdc: entering follower state: follower="Node at 8812df09-ccb8-6aab-ebb7-734d1399fcdc [Follower]" leader-address= leader-id=
2025-01-28T14:54:52.423Z [INFO]  server-20cc3556-5843-71c7-21cc-de081e2e48fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ede50f2d-5fb1-af3a-0a15-465889c97889 Address:ede50f2d-5fb1-af3a-0a15-465889c97889} {Suffrage:Voter ID:server-8812df09-ccb8-6aab-ebb7-734d1399fcdc Address:8812df09-ccb8-6aab-ebb7-734d1399fcdc} {Suffrage:Voter ID:server-20cc3556-5843-71c7-21cc-de081e2e48fa Address:20cc3556-5843-71c7-21cc-de081e2e48fa}]"
2025-01-28T14:54:52.423Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:52.423Z [INFO]  server-20cc3556-5843-71c7-21cc-de081e2e48fa: entering follower state: follower="Node at 20cc3556-5843-71c7-21cc-de081e2e48fa [Follower]" leader-address= leader-id=
2025-01-28T14:54:52.954Z [WARN]  server-ede50f2d-5fb1-af3a-0a15-465889c97889: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:52.954Z [INFO]  server-ede50f2d-5fb1-af3a-0a15-465889c97889: entering candidate state: node="Node at ede50f2d-5fb1-af3a-0a15-465889c97889 [Candidate]" term=2
2025-01-28T14:54:52.954Z [INFO]  server-ede50f2d-5fb1-af3a-0a15-465889c97889: election won: term=2 tally=2
2025-01-28T14:54:52.954Z [INFO]  server-ede50f2d-5fb1-af3a-0a15-465889c97889: entering leader state: leader="Node at ede50f2d-5fb1-af3a-0a15-465889c97889 [Leader]"
2025-01-28T14:54:52.954Z [INFO]  server-ede50f2d-5fb1-af3a-0a15-465889c97889: added peer, starting replication: peer=server-8812df09-ccb8-6aab-ebb7-734d1399fcdc
2025-01-28T14:54:52.954Z [INFO]  server-ede50f2d-5fb1-af3a-0a15-465889c97889: added peer, starting replication: peer=server-20cc3556-5843-71c7-21cc-de081e2e48fa
2025-01-28T14:54:52.954Z [INFO]  server-ede50f2d-5fb1-af3a-0a15-465889c97889: pipelining replication: peer="{Voter server-8812df09-ccb8-6aab-ebb7-734d1399fcdc 8812df09-ccb8-6aab-ebb7-734d1399fcdc}"
2025-01-28T14:54:52.954Z [INFO]  server-ede50f2d-5fb1-af3a-0a15-465889c97889: pipelining replication: peer="{Voter server-20cc3556-5843-71c7-21cc-de081e2e48fa 20cc3556-5843-71c7-21cc-de081e2e48fa}"
2025-01-28T14:54:53.960Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:52.954685861 +0000 UTC m=+78.629175821 (1 nodes), 531.350913ms from start of poll, 532.34202ms from cluster start. Timeout at 2025-01-28 14:54:53.959716038 +0000 UTC m=+79.634206058, 1.005030237s after stability
2025-01-28T14:54:53.961Z [INFO]  server-ede50f2d-5fb1-af3a-0a15-465889c97889: starting snapshot up to: index=12
2025-01-28T14:54:53.961Z [INFO]  creating new snapshot: path=/tmp/raft1107055528/snapshots/2-12-1738076093961.tmp
2025-01-28T14:54:53.962Z [INFO]  server-ede50f2d-5fb1-af3a-0a15-465889c97889: snapshot complete up to: index=12
2025-01-28T14:54:53.966Z [INFO]  server-6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf Address:6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf} {Suffrage:Voter ID:server-97b2c5a6-b641-0e49-e151-7932751d5f66 Address:97b2c5a6-b641-0e49-e151-7932751d5f66} {Suffrage:Voter ID:server-0d9871ee-1c3d-4090-9e42-bd029d482157 Address:0d9871ee-1c3d-4090-9e42-bd029d482157}]"
2025-01-28T14:54:53.966Z [INFO]  server-6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf: entering follower state: follower="Node at 6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf [Follower]" leader-address= leader-id=
2025-01-28T14:54:53.966Z [INFO]  server-97b2c5a6-b641-0e49-e151-7932751d5f66: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf Address:6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf} {Suffrage:Voter ID:server-97b2c5a6-b641-0e49-e151-7932751d5f66 Address:97b2c5a6-b641-0e49-e151-7932751d5f66} {Suffrage:Voter ID:server-0d9871ee-1c3d-4090-9e42-bd029d482157 Address:0d9871ee-1c3d-4090-9e42-bd029d482157}]"
2025-01-28T14:54:53.966Z [INFO]  server-97b2c5a6-b641-0e49-e151-7932751d5f66: entering follower state: follower="Node at 97b2c5a6-b641-0e49-e151-7932751d5f66 [Follower]" leader-address= leader-id=
2025-01-28T14:54:53.966Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf Address:6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf} {Suffrage:Voter ID:server-97b2c5a6-b641-0e49-e151-7932751d5f66 Address:97b2c5a6-b641-0e49-e151-7932751d5f66} {Suffrage:Voter ID:server-0d9871ee-1c3d-4090-9e42-bd029d482157 Address:0d9871ee-1c3d-4090-9e42-bd029d482157}]"
2025-01-28T14:54:53.967Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:53.967Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: entering follower state: follower="Node at 0d9871ee-1c3d-4090-9e42-bd029d482157 [Follower]" leader-address= leader-id=
2025-01-28T14:54:54.624Z [WARN]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:54.624Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: entering candidate state: node="Node at 0d9871ee-1c3d-4090-9e42-bd029d482157 [Candidate]" term=2
2025-01-28T14:54:54.624Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: election won: term=2 tally=2
2025-01-28T14:54:54.624Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: entering leader state: leader="Node at 0d9871ee-1c3d-4090-9e42-bd029d482157 [Leader]"
2025-01-28T14:54:54.624Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: added peer, starting replication: peer=server-6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf
2025-01-28T14:54:54.624Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: added peer, starting replication: peer=server-97b2c5a6-b641-0e49-e151-7932751d5f66
2025-01-28T14:54:54.625Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: pipelining replication: peer="{Voter server-97b2c5a6-b641-0e49-e151-7932751d5f66 97b2c5a6-b641-0e49-e151-7932751d5f66}"
2025-01-28T14:54:54.625Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: pipelining replication: peer="{Voter server-6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf 6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf}"
2025-01-28T14:54:55.630Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:54.624848982 +0000 UTC m=+80.299338962 (1 nodes), 657.735891ms from start of poll, 658.815949ms from cluster start. Timeout at 2025-01-28 14:54:55.6298629 +0000 UTC m=+81.304352860, 1.005013898s after stability
2025-01-28T14:54:55.633Z [INFO]  creating new snapshot: path=/tmp/raft3579785621/snapshots/2-15-1738076095632.tmp
2025-01-28T14:54:55.643Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: copied to local snapshot: bytes=71
2025-01-28T14:54:55.644Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: snapshot restore progress: id=2-15-1738076095632 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:55.647Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: removing all old logs from log store
2025-01-28T14:54:55.647Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: compacting logs: from=1 to=2
2025-01-28T14:54:55.647Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: restored user snapshot: index=15
2025-01-28T14:54:55.647Z [ERROR] server-0d9871ee-1c3d-4090-9e42-bd029d482157: failed to get log: index=2 error="log not found"
2025-01-28T14:54:55.647Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: aborting pipeline replication: peer="{Voter server-97b2c5a6-b641-0e49-e151-7932751d5f66 97b2c5a6-b641-0e49-e151-7932751d5f66}"
2025-01-28T14:54:55.647Z [ERROR] server-0d9871ee-1c3d-4090-9e42-bd029d482157: failed to get log: index=2 error="log not found"
2025-01-28T14:54:55.648Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: aborting pipeline replication: peer="{Voter server-6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf 6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf}"
2025-01-28T14:54:55.656Z [ERROR] server-0d9871ee-1c3d-4090-9e42-bd029d482157: failed to get log: index=2 error="log not found"
2025-01-28T14:54:55.656Z [INFO]  creating new snapshot: path=/tmp/raft4134343914/snapshots/2-15-1738076095656.tmp
2025-01-28T14:54:55.657Z [INFO]  server-6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:55.658Z [INFO]  server-6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf: copied to local snapshot: bytes=71
2025-01-28T14:54:55.658Z [ERROR] server-0d9871ee-1c3d-4090-9e42-bd029d482157: failed to get log: index=2 error="log not found"
2025-01-28T14:54:55.659Z [INFO]  creating new snapshot: path=/tmp/raft4059423763/snapshots/2-15-1738076095659.tmp
2025-01-28T14:54:55.659Z [INFO]  server-6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf: snapshot restore progress: id=2-15-1738076095656 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:55.659Z [INFO]  server-6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf: removing all old logs from log store
2025-01-28T14:54:55.659Z [INFO]  server-6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf: compacting logs: from=1 to=2
2025-01-28T14:54:55.659Z [INFO]  server-6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf: Installed remote snapshot
2025-01-28T14:54:55.659Z [INFO]  server-97b2c5a6-b641-0e49-e151-7932751d5f66: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:55.660Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: pipelining replication: peer="{Voter server-6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf 6bcf3569-ccc9-970b-3b69-5d4fa7c89fdf}"
2025-01-28T14:54:55.660Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:55.660Z [INFO]  server-97b2c5a6-b641-0e49-e151-7932751d5f66: copied to local snapshot: bytes=71
2025-01-28T14:54:55.661Z [INFO]  server-97b2c5a6-b641-0e49-e151-7932751d5f66: snapshot restore progress: id=2-15-1738076095659 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:55.661Z [INFO]  server-97b2c5a6-b641-0e49-e151-7932751d5f66: removing all old logs from log store
2025-01-28T14:54:55.661Z [INFO]  server-97b2c5a6-b641-0e49-e151-7932751d5f66: compacting logs: from=1 to=2
2025-01-28T14:54:55.661Z [INFO]  server-97b2c5a6-b641-0e49-e151-7932751d5f66: Installed remote snapshot
2025-01-28T14:54:55.661Z [INFO]  server-0d9871ee-1c3d-4090-9e42-bd029d482157: pipelining replication: peer="{Voter server-97b2c5a6-b641-0e49-e151-7932751d5f66 97b2c5a6-b641-0e49-e151-7932751d5f66}"
2025-01-28T14:54:56.666Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:55.660325258 +0000 UTC m=+81.334815218 (2 nodes), 2.38µs from start of poll, 1.694292205s from cluster start. Timeout at 2025-01-28 14:54:56.665391792 +0000 UTC m=+82.339881772, 1.005066554s after stability
2025-01-28T14:54:56.677Z [INFO]  server-ede50f2d-5fb1-af3a-0a15-465889c97889: aborting pipeline replication: peer="{Voter server-20cc3556-5843-71c7-21cc-de081e2e48fa 20cc3556-5843-71c7-21cc-de081e2e48fa}"
2025-01-28T14:54:56.677Z [INFO]  server-ede50f2d-5fb1-af3a-0a15-465889c97889: aborting pipeline replication: peer="{Voter server-8812df09-ccb8-6aab-ebb7-734d1399fcdc 8812df09-ccb8-6aab-ebb7-734d1399fcdc}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false
2025-01-28T14:54:56.681Z [INFO]  server-0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64 Address:0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64} {Suffrage:Voter ID:server-9a11845a-33b0-4ffa-107c-4ee13bafdae3 Address:9a11845a-33b0-4ffa-107c-4ee13bafdae3} {Suffrage:Voter ID:server-df0b2715-797e-cfb5-76ad-eaffe3e65769 Address:df0b2715-797e-cfb5-76ad-eaffe3e65769}]"
2025-01-28T14:54:56.682Z [INFO]  server-0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64: entering follower state: follower="Node at 0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64 [Follower]" leader-address= leader-id=
2025-01-28T14:54:56.682Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64 Address:0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64} {Suffrage:Voter ID:server-9a11845a-33b0-4ffa-107c-4ee13bafdae3 Address:9a11845a-33b0-4ffa-107c-4ee13bafdae3} {Suffrage:Voter ID:server-df0b2715-797e-cfb5-76ad-eaffe3e65769 Address:df0b2715-797e-cfb5-76ad-eaffe3e65769}]"
2025-01-28T14:54:56.682Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: entering follower state: follower="Node at 9a11845a-33b0-4ffa-107c-4ee13bafdae3 [Follower]" leader-address= leader-id=
2025-01-28T14:54:56.682Z [INFO]  server-df0b2715-797e-cfb5-76ad-eaffe3e65769: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64 Address:0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64} {Suffrage:Voter ID:server-9a11845a-33b0-4ffa-107c-4ee13bafdae3 Address:9a11845a-33b0-4ffa-107c-4ee13bafdae3} {Suffrage:Voter ID:server-df0b2715-797e-cfb5-76ad-eaffe3e65769 Address:df0b2715-797e-cfb5-76ad-eaffe3e65769}]"
2025-01-28T14:54:56.682Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:56.682Z [INFO]  server-df0b2715-797e-cfb5-76ad-eaffe3e65769: entering follower state: follower="Node at df0b2715-797e-cfb5-76ad-eaffe3e65769 [Follower]" leader-address= leader-id=
2025-01-28T14:54:57.548Z [WARN]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:54:57.548Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: entering candidate state: node="Node at 9a11845a-33b0-4ffa-107c-4ee13bafdae3 [Candidate]" term=2
2025-01-28T14:54:57.548Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: election won: term=2 tally=2
2025-01-28T14:54:57.548Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: entering leader state: leader="Node at 9a11845a-33b0-4ffa-107c-4ee13bafdae3 [Leader]"
2025-01-28T14:54:57.548Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: added peer, starting replication: peer=server-0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64
2025-01-28T14:54:57.548Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: added peer, starting replication: peer=server-df0b2715-797e-cfb5-76ad-eaffe3e65769
2025-01-28T14:54:57.549Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: pipelining replication: peer="{Voter server-0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64 0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64}"
2025-01-28T14:54:57.549Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: pipelining replication: peer="{Voter server-df0b2715-797e-cfb5-76ad-eaffe3e65769 df0b2715-797e-cfb5-76ad-eaffe3e65769}"
2025-01-28T14:54:58.554Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:54:57.548868957 +0000 UTC m=+83.223358917 (1 nodes), 866.14141ms from start of poll, 867.140896ms from cluster start. Timeout at 2025-01-28 14:54:58.553877417 +0000 UTC m=+84.228367397, 1.00500848s after stability
2025-01-28T14:54:58.556Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: starting snapshot up to: index=12
2025-01-28T14:54:58.556Z [INFO]  creating new snapshot: path=/tmp/raft2960184890/snapshots/2-12-1738076098556.tmp
2025-01-28T14:54:58.557Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: snapshot complete up to: index=12
2025-01-28T14:54:58.559Z [INFO]  creating new snapshot: path=/tmp/raft2960184890/snapshots/2-113-1738076098559.tmp
2025-01-28T14:54:58.560Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: copied to local snapshot: bytes=71
2025-01-28T14:54:58.561Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: snapshot restore progress: id=2-113-1738076098559 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:58.561Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: restored user snapshot: index=113
2025-01-28T14:54:58.561Z [ERROR] server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: failed to get log: index=23 error="log not found"
2025-01-28T14:54:58.561Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: aborting pipeline replication: peer="{Voter server-df0b2715-797e-cfb5-76ad-eaffe3e65769 df0b2715-797e-cfb5-76ad-eaffe3e65769}"
2025-01-28T14:54:58.561Z [ERROR] server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: failed to get log: index=23 error="log not found"
2025-01-28T14:54:58.561Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: aborting pipeline replication: peer="{Voter server-0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64 0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64}"
2025-01-28T14:54:58.569Z [ERROR] server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: failed to get log: index=23 error="log not found"
2025-01-28T14:54:58.570Z [ERROR] server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: failed to get log: index=23 error="log not found"
2025-01-28T14:54:58.571Z [INFO]  creating new snapshot: path=/tmp/raft2530845631/snapshots/2-113-1738076098571.tmp
2025-01-28T14:54:58.573Z [INFO]  creating new snapshot: path=/tmp/raft2314347603/snapshots/2-113-1738076098573.tmp
2025-01-28T14:54:58.573Z [INFO]  server-0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:58.574Z [INFO]  server-df0b2715-797e-cfb5-76ad-eaffe3e65769: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:58.575Z [INFO]  server-df0b2715-797e-cfb5-76ad-eaffe3e65769: copied to local snapshot: bytes=71
2025-01-28T14:54:58.575Z [INFO]  server-0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64: copied to local snapshot: bytes=71
2025-01-28T14:54:58.575Z [INFO]  server-df0b2715-797e-cfb5-76ad-eaffe3e65769: snapshot restore progress: id=2-113-1738076098573 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:58.575Z [INFO]  server-0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64: snapshot restore progress: id=2-113-1738076098571 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:54:58.575Z [INFO]  server-df0b2715-797e-cfb5-76ad-eaffe3e65769: Installed remote snapshot
2025-01-28T14:54:58.575Z [INFO]  server-0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64: Installed remote snapshot
2025-01-28T14:54:58.576Z [WARN]  server-0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64: skipping application of old log: index=113
2025-01-28T14:54:58.576Z [WARN]  server-df0b2715-797e-cfb5-76ad-eaffe3e65769: skipping application of old log: index=113
2025-01-28T14:54:58.576Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: pipelining replication: peer="{Voter server-0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64 0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64}"
2025-01-28T14:54:58.576Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:54:58.576Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: pipelining replication: peer="{Voter server-df0b2715-797e-cfb5-76ad-eaffe3e65769 df0b2715-797e-cfb5-76ad-eaffe3e65769}"
2025-01-28T14:54:59.583Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:54:58.576446038 +0000 UTC m=+84.250936018 (2 nodes), 2.66µs from start of poll, 1.894717997s from cluster start. Timeout at 2025-01-28 14:54:59.581554068 +0000 UTC m=+85.256044128, 1.00510811s after stability
2025-01-28T14:54:59.599Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: aborting pipeline replication: peer="{Voter server-df0b2715-797e-cfb5-76ad-eaffe3e65769 df0b2715-797e-cfb5-76ad-eaffe3e65769}"
2025-01-28T14:54:59.599Z [INFO]  server-9a11845a-33b0-4ffa-107c-4ee13bafdae3: aborting pipeline replication: peer="{Voter server-0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64 0e1f90fe-8b5c-6c34-fe80-c0dc24a42f64}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false
2025-01-28T14:54:59.604Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45c840c3-9891-b0c0-25ca-e35fd62c4468 Address:45c840c3-9891-b0c0-25ca-e35fd62c4468} {Suffrage:Voter ID:server-d74f6f89-3eee-9ea4-f555-95b6e56918dd Address:d74f6f89-3eee-9ea4-f555-95b6e56918dd} {Suffrage:Voter ID:server-ed822476-b77f-a013-afa7-5c4c70037fe4 Address:ed822476-b77f-a013-afa7-5c4c70037fe4}]"
2025-01-28T14:54:59.604Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: entering follower state: follower="Node at 45c840c3-9891-b0c0-25ca-e35fd62c4468 [Follower]" leader-address= leader-id=
2025-01-28T14:54:59.604Z [INFO]  server-d74f6f89-3eee-9ea4-f555-95b6e56918dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45c840c3-9891-b0c0-25ca-e35fd62c4468 Address:45c840c3-9891-b0c0-25ca-e35fd62c4468} {Suffrage:Voter ID:server-d74f6f89-3eee-9ea4-f555-95b6e56918dd Address:d74f6f89-3eee-9ea4-f555-95b6e56918dd} {Suffrage:Voter ID:server-ed822476-b77f-a013-afa7-5c4c70037fe4 Address:ed822476-b77f-a013-afa7-5c4c70037fe4}]"
2025-01-28T14:54:59.605Z [INFO]  server-d74f6f89-3eee-9ea4-f555-95b6e56918dd: entering follower state: follower="Node at d74f6f89-3eee-9ea4-f555-95b6e56918dd [Follower]" leader-address= leader-id=
2025-01-28T14:54:59.605Z [INFO]  server-ed822476-b77f-a013-afa7-5c4c70037fe4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-45c840c3-9891-b0c0-25ca-e35fd62c4468 Address:45c840c3-9891-b0c0-25ca-e35fd62c4468} {Suffrage:Voter ID:server-d74f6f89-3eee-9ea4-f555-95b6e56918dd Address:d74f6f89-3eee-9ea4-f555-95b6e56918dd} {Suffrage:Voter ID:server-ed822476-b77f-a013-afa7-5c4c70037fe4 Address:ed822476-b77f-a013-afa7-5c4c70037fe4}]"
2025-01-28T14:54:59.605Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:54:59.605Z [INFO]  server-ed822476-b77f-a013-afa7-5c4c70037fe4: entering follower state: follower="Node at ed822476-b77f-a013-afa7-5c4c70037fe4 [Follower]" leader-address= leader-id=
2025-01-28T14:55:00.175Z [WARN]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:00.175Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: entering candidate state: node="Node at 45c840c3-9891-b0c0-25ca-e35fd62c4468 [Candidate]" term=2
2025-01-28T14:55:00.175Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: election won: term=2 tally=2
2025-01-28T14:55:00.175Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: entering leader state: leader="Node at 45c840c3-9891-b0c0-25ca-e35fd62c4468 [Leader]"
2025-01-28T14:55:00.175Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: added peer, starting replication: peer=server-d74f6f89-3eee-9ea4-f555-95b6e56918dd
2025-01-28T14:55:00.175Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: added peer, starting replication: peer=server-ed822476-b77f-a013-afa7-5c4c70037fe4
2025-01-28T14:55:00.176Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: pipelining replication: peer="{Voter server-ed822476-b77f-a013-afa7-5c4c70037fe4 ed822476-b77f-a013-afa7-5c4c70037fe4}"
2025-01-28T14:55:00.176Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: pipelining replication: peer="{Voter server-d74f6f89-3eee-9ea4-f555-95b6e56918dd d74f6f89-3eee-9ea4-f555-95b6e56918dd}"
2025-01-28T14:55:01.181Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:00.175916008 +0000 UTC m=+85.850405988 (1 nodes), 570.580853ms from start of poll, 571.650732ms from cluster start. Timeout at 2025-01-28 14:55:01.180919808 +0000 UTC m=+86.855409808, 1.00500382s after stability
2025-01-28T14:55:01.182Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: starting snapshot up to: index=12
2025-01-28T14:55:01.182Z [INFO]  creating new snapshot: path=/tmp/raft3954512927/snapshots/2-12-1738076101182.tmp
2025-01-28T14:55:01.184Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: snapshot complete up to: index=12
2025-01-28T14:55:01.185Z [INFO]  creating new snapshot: path=/tmp/raft3954512927/snapshots/2-113-1738076101185.tmp
2025-01-28T14:55:01.187Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: copied to local snapshot: bytes=71
2025-01-28T14:55:01.187Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: snapshot restore progress: id=2-113-1738076101185 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:01.188Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: removing all old logs from log store
2025-01-28T14:55:01.188Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: compacting logs: from=1 to=22
2025-01-28T14:55:01.188Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: restored user snapshot: index=113
2025-01-28T14:55:01.188Z [ERROR] server-45c840c3-9891-b0c0-25ca-e35fd62c4468: failed to get log: index=22 error="log not found"
2025-01-28T14:55:01.188Z [ERROR] server-45c840c3-9891-b0c0-25ca-e35fd62c4468: failed to get log: index=22 error="log not found"
2025-01-28T14:55:01.188Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: aborting pipeline replication: peer="{Voter server-d74f6f89-3eee-9ea4-f555-95b6e56918dd d74f6f89-3eee-9ea4-f555-95b6e56918dd}"
2025-01-28T14:55:01.188Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: aborting pipeline replication: peer="{Voter server-ed822476-b77f-a013-afa7-5c4c70037fe4 ed822476-b77f-a013-afa7-5c4c70037fe4}"
2025-01-28T14:55:01.196Z [ERROR] server-45c840c3-9891-b0c0-25ca-e35fd62c4468: failed to get log: index=22 error="log not found"
2025-01-28T14:55:01.197Z [INFO]  creating new snapshot: path=/tmp/raft222557369/snapshots/2-113-1738076101197.tmp
2025-01-28T14:55:01.197Z [INFO]  server-ed822476-b77f-a013-afa7-5c4c70037fe4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:01.198Z [ERROR] server-45c840c3-9891-b0c0-25ca-e35fd62c4468: failed to get log: index=22 error="log not found"
2025-01-28T14:55:01.198Z [INFO]  server-ed822476-b77f-a013-afa7-5c4c70037fe4: copied to local snapshot: bytes=71
2025-01-28T14:55:01.199Z [INFO]  server-ed822476-b77f-a013-afa7-5c4c70037fe4: snapshot restore progress: id=2-113-1738076101197 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:01.199Z [INFO]  creating new snapshot: path=/tmp/raft2543610462/snapshots/2-113-1738076101199.tmp
2025-01-28T14:55:01.199Z [INFO]  server-ed822476-b77f-a013-afa7-5c4c70037fe4: removing all old logs from log store
2025-01-28T14:55:01.199Z [INFO]  server-ed822476-b77f-a013-afa7-5c4c70037fe4: compacting logs: from=1 to=22
2025-01-28T14:55:01.199Z [INFO]  server-ed822476-b77f-a013-afa7-5c4c70037fe4: Installed remote snapshot
2025-01-28T14:55:01.199Z [WARN]  server-ed822476-b77f-a013-afa7-5c4c70037fe4: skipping application of old log: index=22
2025-01-28T14:55:01.199Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: pipelining replication: peer="{Voter server-ed822476-b77f-a013-afa7-5c4c70037fe4 ed822476-b77f-a013-afa7-5c4c70037fe4}"
2025-01-28T14:55:01.199Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:01.199Z [INFO]  server-d74f6f89-3eee-9ea4-f555-95b6e56918dd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:01.200Z [INFO]  server-d74f6f89-3eee-9ea4-f555-95b6e56918dd: copied to local snapshot: bytes=71
2025-01-28T14:55:01.201Z [INFO]  server-d74f6f89-3eee-9ea4-f555-95b6e56918dd: snapshot restore progress: id=2-113-1738076101199 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:01.201Z [INFO]  server-d74f6f89-3eee-9ea4-f555-95b6e56918dd: removing all old logs from log store
2025-01-28T14:55:01.201Z [INFO]  server-d74f6f89-3eee-9ea4-f555-95b6e56918dd: compacting logs: from=1 to=22
2025-01-28T14:55:01.201Z [INFO]  server-d74f6f89-3eee-9ea4-f555-95b6e56918dd: Installed remote snapshot
2025-01-28T14:55:01.201Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: pipelining replication: peer="{Voter server-d74f6f89-3eee-9ea4-f555-95b6e56918dd d74f6f89-3eee-9ea4-f555-95b6e56918dd}"
2025-01-28T14:55:02.206Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:01.19997926 +0000 UTC m=+86.874469220 (2 nodes), 2.319µs from start of poll, 1.595713964s from cluster start. Timeout at 2025-01-28 14:55:02.205064532 +0000 UTC m=+87.879554592, 1.005085372s after stability
2025-01-28T14:55:02.218Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: aborting pipeline replication: peer="{Voter server-ed822476-b77f-a013-afa7-5c4c70037fe4 ed822476-b77f-a013-afa7-5c4c70037fe4}"
2025-01-28T14:55:02.218Z [INFO]  server-45c840c3-9891-b0c0-25ca-e35fd62c4468: aborting pipeline replication: peer="{Voter server-d74f6f89-3eee-9ea4-f555-95b6e56918dd d74f6f89-3eee-9ea4-f555-95b6e56918dd}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true
2025-01-28T14:55:02.224Z [INFO]  server-13bdf423-95f8-2f7e-ca4a-f9e458ddcf39: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-13bdf423-95f8-2f7e-ca4a-f9e458ddcf39 Address:13bdf423-95f8-2f7e-ca4a-f9e458ddcf39} {Suffrage:Voter ID:server-16702220-5698-7e54-81b6-f16843890f5a Address:16702220-5698-7e54-81b6-f16843890f5a} {Suffrage:Voter ID:server-c22c2d0f-2bc0-2bef-fd79-45834aef386a Address:c22c2d0f-2bc0-2bef-fd79-45834aef386a}]"
2025-01-28T14:55:02.224Z [INFO]  server-16702220-5698-7e54-81b6-f16843890f5a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-13bdf423-95f8-2f7e-ca4a-f9e458ddcf39 Address:13bdf423-95f8-2f7e-ca4a-f9e458ddcf39} {Suffrage:Voter ID:server-16702220-5698-7e54-81b6-f16843890f5a Address:16702220-5698-7e54-81b6-f16843890f5a} {Suffrage:Voter ID:server-c22c2d0f-2bc0-2bef-fd79-45834aef386a Address:c22c2d0f-2bc0-2bef-fd79-45834aef386a}]"
2025-01-28T14:55:02.224Z [INFO]  server-13bdf423-95f8-2f7e-ca4a-f9e458ddcf39: entering follower state: follower="Node at 13bdf423-95f8-2f7e-ca4a-f9e458ddcf39 [Follower]" leader-address= leader-id=
2025-01-28T14:55:02.224Z [INFO]  server-16702220-5698-7e54-81b6-f16843890f5a: entering follower state: follower="Node at 16702220-5698-7e54-81b6-f16843890f5a [Follower]" leader-address= leader-id=
2025-01-28T14:55:02.225Z [INFO]  server-c22c2d0f-2bc0-2bef-fd79-45834aef386a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-13bdf423-95f8-2f7e-ca4a-f9e458ddcf39 Address:13bdf423-95f8-2f7e-ca4a-f9e458ddcf39} {Suffrage:Voter ID:server-16702220-5698-7e54-81b6-f16843890f5a Address:16702220-5698-7e54-81b6-f16843890f5a} {Suffrage:Voter ID:server-c22c2d0f-2bc0-2bef-fd79-45834aef386a Address:c22c2d0f-2bc0-2bef-fd79-45834aef386a}]"
2025-01-28T14:55:02.225Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:02.225Z [INFO]  server-c22c2d0f-2bc0-2bef-fd79-45834aef386a: entering follower state: follower="Node at c22c2d0f-2bc0-2bef-fd79-45834aef386a [Follower]" leader-address= leader-id=
2025-01-28T14:55:02.876Z [WARN]  server-c22c2d0f-2bc0-2bef-fd79-45834aef386a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:02.876Z [INFO]  server-c22c2d0f-2bc0-2bef-fd79-45834aef386a: entering candidate state: node="Node at c22c2d0f-2bc0-2bef-fd79-45834aef386a [Candidate]" term=2
2025-01-28T14:55:02.876Z [INFO]  server-c22c2d0f-2bc0-2bef-fd79-45834aef386a: election won: term=2 tally=2
2025-01-28T14:55:02.876Z [INFO]  server-c22c2d0f-2bc0-2bef-fd79-45834aef386a: entering leader state: leader="Node at c22c2d0f-2bc0-2bef-fd79-45834aef386a [Leader]"
2025-01-28T14:55:02.876Z [INFO]  server-c22c2d0f-2bc0-2bef-fd79-45834aef386a: added peer, starting replication: peer=server-13bdf423-95f8-2f7e-ca4a-f9e458ddcf39
2025-01-28T14:55:02.876Z [INFO]  server-c22c2d0f-2bc0-2bef-fd79-45834aef386a: added peer, starting replication: peer=server-16702220-5698-7e54-81b6-f16843890f5a
2025-01-28T14:55:02.877Z [INFO]  server-c22c2d0f-2bc0-2bef-fd79-45834aef386a: pipelining replication: peer="{Voter server-13bdf423-95f8-2f7e-ca4a-f9e458ddcf39 13bdf423-95f8-2f7e-ca4a-f9e458ddcf39}"
2025-01-28T14:55:02.877Z [INFO]  server-c22c2d0f-2bc0-2bef-fd79-45834aef386a: pipelining replication: peer="{Voter server-16702220-5698-7e54-81b6-f16843890f5a 16702220-5698-7e54-81b6-f16843890f5a}"
2025-01-28T14:55:03.882Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:02.876901024 +0000 UTC m=+88.551391004 (1 nodes), 651.59769ms from start of poll, 652.835733ms from cluster start. Timeout at 2025-01-28 14:55:03.881914163 +0000 UTC m=+89.556404183, 1.005013179s after stability
2025-01-28T14:55:03.884Z [INFO]  server-c22c2d0f-2bc0-2bef-fd79-45834aef386a: starting snapshot up to: index=12
2025-01-28T14:55:03.884Z [INFO]  creating new snapshot: path=/tmp/raft2055875527/snapshots/2-12-1738076103884.tmp
2025-01-28T14:55:03.885Z [INFO]  server-c22c2d0f-2bc0-2bef-fd79-45834aef386a: snapshot complete up to: index=12
2025-01-28T14:55:03.892Z [INFO]  server-21362484-cbbf-6c03-76cc-c7a73bc664e4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21362484-cbbf-6c03-76cc-c7a73bc664e4 Address:21362484-cbbf-6c03-76cc-c7a73bc664e4} {Suffrage:Voter ID:server-05c9db10-5aff-04f6-6d99-33be0731ef44 Address:05c9db10-5aff-04f6-6d99-33be0731ef44} {Suffrage:Voter ID:server-5751c78e-cf79-0cd8-43e9-94f0ea0e3a44 Address:5751c78e-cf79-0cd8-43e9-94f0ea0e3a44}]"
2025-01-28T14:55:03.892Z [INFO]  server-05c9db10-5aff-04f6-6d99-33be0731ef44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21362484-cbbf-6c03-76cc-c7a73bc664e4 Address:21362484-cbbf-6c03-76cc-c7a73bc664e4} {Suffrage:Voter ID:server-05c9db10-5aff-04f6-6d99-33be0731ef44 Address:05c9db10-5aff-04f6-6d99-33be0731ef44} {Suffrage:Voter ID:server-5751c78e-cf79-0cd8-43e9-94f0ea0e3a44 Address:5751c78e-cf79-0cd8-43e9-94f0ea0e3a44}]"
2025-01-28T14:55:03.892Z [INFO]  server-21362484-cbbf-6c03-76cc-c7a73bc664e4: entering follower state: follower="Node at 21362484-cbbf-6c03-76cc-c7a73bc664e4 [Follower]" leader-address= leader-id=
2025-01-28T14:55:03.892Z [INFO]  server-05c9db10-5aff-04f6-6d99-33be0731ef44: entering follower state: follower="Node at 05c9db10-5aff-04f6-6d99-33be0731ef44 [Follower]" leader-address= leader-id=
2025-01-28T14:55:03.892Z [INFO]  server-5751c78e-cf79-0cd8-43e9-94f0ea0e3a44: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-21362484-cbbf-6c03-76cc-c7a73bc664e4 Address:21362484-cbbf-6c03-76cc-c7a73bc664e4} {Suffrage:Voter ID:server-05c9db10-5aff-04f6-6d99-33be0731ef44 Address:05c9db10-5aff-04f6-6d99-33be0731ef44} {Suffrage:Voter ID:server-5751c78e-cf79-0cd8-43e9-94f0ea0e3a44 Address:5751c78e-cf79-0cd8-43e9-94f0ea0e3a44}]"
2025-01-28T14:55:03.892Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:03.892Z [INFO]  server-5751c78e-cf79-0cd8-43e9-94f0ea0e3a44: entering follower state: follower="Node at 5751c78e-cf79-0cd8-43e9-94f0ea0e3a44 [Follower]" leader-address= leader-id=
2025-01-28T14:55:04.523Z [WARN]  server-05c9db10-5aff-04f6-6d99-33be0731ef44: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:04.523Z [INFO]  server-05c9db10-5aff-04f6-6d99-33be0731ef44: entering candidate state: node="Node at 05c9db10-5aff-04f6-6d99-33be0731ef44 [Candidate]" term=2
2025-01-28T14:55:04.523Z [INFO]  server-05c9db10-5aff-04f6-6d99-33be0731ef44: election won: term=2 tally=2
2025-01-28T14:55:04.523Z [INFO]  server-05c9db10-5aff-04f6-6d99-33be0731ef44: entering leader state: leader="Node at 05c9db10-5aff-04f6-6d99-33be0731ef44 [Leader]"
2025-01-28T14:55:04.523Z [INFO]  server-05c9db10-5aff-04f6-6d99-33be0731ef44: added peer, starting replication: peer=server-21362484-cbbf-6c03-76cc-c7a73bc664e4
2025-01-28T14:55:04.523Z [INFO]  server-05c9db10-5aff-04f6-6d99-33be0731ef44: added peer, starting replication: peer=server-5751c78e-cf79-0cd8-43e9-94f0ea0e3a44
2025-01-28T14:55:04.524Z [INFO]  server-05c9db10-5aff-04f6-6d99-33be0731ef44: pipelining replication: peer="{Voter server-5751c78e-cf79-0cd8-43e9-94f0ea0e3a44 5751c78e-cf79-0cd8-43e9-94f0ea0e3a44}"
2025-01-28T14:55:04.524Z [INFO]  server-05c9db10-5aff-04f6-6d99-33be0731ef44: pipelining replication: peer="{Voter server-21362484-cbbf-6c03-76cc-c7a73bc664e4 21362484-cbbf-6c03-76cc-c7a73bc664e4}"
2025-01-28T14:55:05.529Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:04.523767543 +0000 UTC m=+90.198257523 (1 nodes), 630.949438ms from start of poll, 631.77364ms from cluster start. Timeout at 2025-01-28 14:55:05.528774942 +0000 UTC m=+91.203264902, 1.005007379s after stability
2025-01-28T14:55:05.530Z [INFO]  creating new snapshot: path=/tmp/raft725788943/snapshots/2-113-1738076105530.tmp
2025-01-28T14:55:05.531Z [INFO]  server-05c9db10-5aff-04f6-6d99-33be0731ef44: copied to local snapshot: bytes=71
2025-01-28T14:55:05.532Z [INFO]  server-05c9db10-5aff-04f6-6d99-33be0731ef44: snapshot restore progress: id=2-113-1738076105530 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:05.532Z [INFO]  server-05c9db10-5aff-04f6-6d99-33be0731ef44: restored user snapshot: index=113
2025-01-28T14:55:05.532Z [ERROR] server-05c9db10-5aff-04f6-6d99-33be0731ef44: failed to get log: index=3 error="log not found"
2025-01-28T14:55:05.532Z [INFO]  server-05c9db10-5aff-04f6-6d99-33be0731ef44: aborting pipeline replication: peer="{Voter server-5751c78e-cf79-0cd8-43e9-94f0ea0e3a44 5751c78e-cf79-0cd8-43e9-94f0ea0e3a44}"
2025-01-28T14:55:05.532Z [ERROR] server-05c9db10-5aff-04f6-6d99-33be0731ef44: failed to get log: index=3 error="log not found"
2025-01-28T14:55:05.532Z [INFO]  server-05c9db10-5aff-04f6-6d99-33be0731ef44: aborting pipeline replication: peer="{Voter server-21362484-cbbf-6c03-76cc-c7a73bc664e4 21362484-cbbf-6c03-76cc-c7a73bc664e4}"
2025-01-28T14:55:05.540Z [ERROR] server-05c9db10-5aff-04f6-6d99-33be0731ef44: failed to get log: index=3 error="log not found"
2025-01-28T14:55:05.541Z [INFO]  creating new snapshot: path=/tmp/raft693359209/snapshots/2-113-1738076105541.tmp
2025-01-28T14:55:05.542Z [INFO]  server-21362484-cbbf-6c03-76cc-c7a73bc664e4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:05.542Z [ERROR] server-05c9db10-5aff-04f6-6d99-33be0731ef44: failed to get log: index=3 error="log not found"
2025-01-28T14:55:05.543Z [INFO]  server-21362484-cbbf-6c03-76cc-c7a73bc664e4: copied to local snapshot: bytes=71
2025-01-28T14:55:05.544Z [INFO]  creating new snapshot: path=/tmp/raft814231172/snapshots/2-113-1738076105544.tmp
2025-01-28T14:55:05.545Z [INFO]  server-21362484-cbbf-6c03-76cc-c7a73bc664e4: snapshot restore progress: id=2-113-1738076105541 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:05.545Z [INFO]  server-21362484-cbbf-6c03-76cc-c7a73bc664e4: Installed remote snapshot
2025-01-28T14:55:05.546Z [INFO]  server-05c9db10-5aff-04f6-6d99-33be0731ef44: pipelining replication: peer="{Voter server-21362484-cbbf-6c03-76cc-c7a73bc664e4 21362484-cbbf-6c03-76cc-c7a73bc664e4}"
2025-01-28T14:55:05.547Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:05.547Z [INFO]  server-5751c78e-cf79-0cd8-43e9-94f0ea0e3a44: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:05.548Z [INFO]  server-5751c78e-cf79-0cd8-43e9-94f0ea0e3a44: copied to local snapshot: bytes=71
2025-01-28T14:55:05.548Z [INFO]  server-5751c78e-cf79-0cd8-43e9-94f0ea0e3a44: snapshot restore progress: id=2-113-1738076105544 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:05.548Z [INFO]  server-5751c78e-cf79-0cd8-43e9-94f0ea0e3a44: Installed remote snapshot
2025-01-28T14:55:05.548Z [INFO]  server-05c9db10-5aff-04f6-6d99-33be0731ef44: pipelining replication: peer="{Voter server-5751c78e-cf79-0cd8-43e9-94f0ea0e3a44 5751c78e-cf79-0cd8-43e9-94f0ea0e3a44}"
2025-01-28T14:55:06.552Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:05.547072427 +0000 UTC m=+91.221562407 (2 nodes), 2.2µs from start of poll, 1.655078524s from cluster start. Timeout at 2025-01-28 14:55:06.552158059 +0000 UTC m=+92.226648079, 1.005085672s after stability
2025-01-28T14:55:06.564Z [INFO]  server-c22c2d0f-2bc0-2bef-fd79-45834aef386a: aborting pipeline replication: peer="{Voter server-13bdf423-95f8-2f7e-ca4a-f9e458ddcf39 13bdf423-95f8-2f7e-ca4a-f9e458ddcf39}"
2025-01-28T14:55:06.564Z [INFO]  server-c22c2d0f-2bc0-2bef-fd79-45834aef386a: aborting pipeline replication: peer="{Voter server-16702220-5698-7e54-81b6-f16843890f5a 16702220-5698-7e54-81b6-f16843890f5a}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true
2025-01-28T14:55:06.570Z [INFO]  server-3ce8adbc-fd9d-ed3f-890d-927ccf732217: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ce8adbc-fd9d-ed3f-890d-927ccf732217 Address:3ce8adbc-fd9d-ed3f-890d-927ccf732217} {Suffrage:Voter ID:server-ce8157db-7e2c-3baf-6af7-5d7ef93d5cad Address:ce8157db-7e2c-3baf-6af7-5d7ef93d5cad} {Suffrage:Voter ID:server-3859bb36-6c31-79bf-81ca-d6aed133cd80 Address:3859bb36-6c31-79bf-81ca-d6aed133cd80}]"
2025-01-28T14:55:06.570Z [INFO]  server-3ce8adbc-fd9d-ed3f-890d-927ccf732217: entering follower state: follower="Node at 3ce8adbc-fd9d-ed3f-890d-927ccf732217 [Follower]" leader-address= leader-id=
2025-01-28T14:55:06.570Z [INFO]  server-ce8157db-7e2c-3baf-6af7-5d7ef93d5cad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ce8adbc-fd9d-ed3f-890d-927ccf732217 Address:3ce8adbc-fd9d-ed3f-890d-927ccf732217} {Suffrage:Voter ID:server-ce8157db-7e2c-3baf-6af7-5d7ef93d5cad Address:ce8157db-7e2c-3baf-6af7-5d7ef93d5cad} {Suffrage:Voter ID:server-3859bb36-6c31-79bf-81ca-d6aed133cd80 Address:3859bb36-6c31-79bf-81ca-d6aed133cd80}]"
2025-01-28T14:55:06.571Z [INFO]  server-ce8157db-7e2c-3baf-6af7-5d7ef93d5cad: entering follower state: follower="Node at ce8157db-7e2c-3baf-6af7-5d7ef93d5cad [Follower]" leader-address= leader-id=
2025-01-28T14:55:06.571Z [INFO]  server-3859bb36-6c31-79bf-81ca-d6aed133cd80: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3ce8adbc-fd9d-ed3f-890d-927ccf732217 Address:3ce8adbc-fd9d-ed3f-890d-927ccf732217} {Suffrage:Voter ID:server-ce8157db-7e2c-3baf-6af7-5d7ef93d5cad Address:ce8157db-7e2c-3baf-6af7-5d7ef93d5cad} {Suffrage:Voter ID:server-3859bb36-6c31-79bf-81ca-d6aed133cd80 Address:3859bb36-6c31-79bf-81ca-d6aed133cd80}]"
2025-01-28T14:55:06.571Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:06.571Z [INFO]  server-3859bb36-6c31-79bf-81ca-d6aed133cd80: entering follower state: follower="Node at 3859bb36-6c31-79bf-81ca-d6aed133cd80 [Follower]" leader-address= leader-id=
2025-01-28T14:55:07.488Z [WARN]  server-3859bb36-6c31-79bf-81ca-d6aed133cd80: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:07.488Z [INFO]  server-3859bb36-6c31-79bf-81ca-d6aed133cd80: entering candidate state: node="Node at 3859bb36-6c31-79bf-81ca-d6aed133cd80 [Candidate]" term=2
2025-01-28T14:55:07.488Z [INFO]  server-3859bb36-6c31-79bf-81ca-d6aed133cd80: election won: term=2 tally=2
2025-01-28T14:55:07.488Z [INFO]  server-3859bb36-6c31-79bf-81ca-d6aed133cd80: entering leader state: leader="Node at 3859bb36-6c31-79bf-81ca-d6aed133cd80 [Leader]"
2025-01-28T14:55:07.488Z [INFO]  server-3859bb36-6c31-79bf-81ca-d6aed133cd80: added peer, starting replication: peer=server-3ce8adbc-fd9d-ed3f-890d-927ccf732217
2025-01-28T14:55:07.488Z [INFO]  server-3859bb36-6c31-79bf-81ca-d6aed133cd80: added peer, starting replication: peer=server-ce8157db-7e2c-3baf-6af7-5d7ef93d5cad
2025-01-28T14:55:07.488Z [INFO]  server-3859bb36-6c31-79bf-81ca-d6aed133cd80: pipelining replication: peer="{Voter server-3ce8adbc-fd9d-ed3f-890d-927ccf732217 3ce8adbc-fd9d-ed3f-890d-927ccf732217}"
2025-01-28T14:55:07.488Z [INFO]  server-3859bb36-6c31-79bf-81ca-d6aed133cd80: pipelining replication: peer="{Voter server-ce8157db-7e2c-3baf-6af7-5d7ef93d5cad ce8157db-7e2c-3baf-6af7-5d7ef93d5cad}"
2025-01-28T14:55:08.494Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:07.488687401 +0000 UTC m=+93.163177361 (1 nodes), 917.228632ms from start of poll, 918.323568ms from cluster start. Timeout at 2025-01-28 14:55:08.49369612 +0000 UTC m=+94.168186100, 1.005008739s after stability
2025-01-28T14:55:08.495Z [INFO]  server-3859bb36-6c31-79bf-81ca-d6aed133cd80: starting snapshot up to: index=12
2025-01-28T14:55:08.495Z [INFO]  creating new snapshot: path=/tmp/raft2328750132/snapshots/2-12-1738076108495.tmp
2025-01-28T14:55:08.496Z [INFO]  server-3859bb36-6c31-79bf-81ca-d6aed133cd80: snapshot complete up to: index=12
2025-01-28T14:55:08.499Z [INFO]  server-cf010aad-ad2f-4235-d9c1-e40bc80247bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf010aad-ad2f-4235-d9c1-e40bc80247bd Address:cf010aad-ad2f-4235-d9c1-e40bc80247bd} {Suffrage:Voter ID:server-7f361493-4921-4358-9436-b9aab5e113f8 Address:7f361493-4921-4358-9436-b9aab5e113f8} {Suffrage:Voter ID:server-b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a Address:b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a}]"
2025-01-28T14:55:08.500Z [INFO]  server-cf010aad-ad2f-4235-d9c1-e40bc80247bd: entering follower state: follower="Node at cf010aad-ad2f-4235-d9c1-e40bc80247bd [Follower]" leader-address= leader-id=
2025-01-28T14:55:08.500Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf010aad-ad2f-4235-d9c1-e40bc80247bd Address:cf010aad-ad2f-4235-d9c1-e40bc80247bd} {Suffrage:Voter ID:server-7f361493-4921-4358-9436-b9aab5e113f8 Address:7f361493-4921-4358-9436-b9aab5e113f8} {Suffrage:Voter ID:server-b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a Address:b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a}]"
2025-01-28T14:55:08.500Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: entering follower state: follower="Node at 7f361493-4921-4358-9436-b9aab5e113f8 [Follower]" leader-address= leader-id=
2025-01-28T14:55:08.500Z [INFO]  server-b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cf010aad-ad2f-4235-d9c1-e40bc80247bd Address:cf010aad-ad2f-4235-d9c1-e40bc80247bd} {Suffrage:Voter ID:server-7f361493-4921-4358-9436-b9aab5e113f8 Address:7f361493-4921-4358-9436-b9aab5e113f8} {Suffrage:Voter ID:server-b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a Address:b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a}]"
2025-01-28T14:55:08.500Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:08.500Z [INFO]  server-b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a: entering follower state: follower="Node at b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a [Follower]" leader-address= leader-id=
2025-01-28T14:55:09.031Z [WARN]  server-7f361493-4921-4358-9436-b9aab5e113f8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:09.031Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: entering candidate state: node="Node at 7f361493-4921-4358-9436-b9aab5e113f8 [Candidate]" term=2
2025-01-28T14:55:09.031Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: election won: term=2 tally=2
2025-01-28T14:55:09.031Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: entering leader state: leader="Node at 7f361493-4921-4358-9436-b9aab5e113f8 [Leader]"
2025-01-28T14:55:09.031Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: added peer, starting replication: peer=server-cf010aad-ad2f-4235-d9c1-e40bc80247bd
2025-01-28T14:55:09.031Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: added peer, starting replication: peer=server-b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a
2025-01-28T14:55:09.032Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: pipelining replication: peer="{Voter server-cf010aad-ad2f-4235-d9c1-e40bc80247bd cf010aad-ad2f-4235-d9c1-e40bc80247bd}"
2025-01-28T14:55:09.032Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: pipelining replication: peer="{Voter server-b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a}"
2025-01-28T14:55:10.037Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:09.031895199 +0000 UTC m=+94.706385179 (1 nodes), 531.102218ms from start of poll, 532.150459ms from cluster start. Timeout at 2025-01-28 14:55:10.036902638 +0000 UTC m=+95.711392618, 1.005007439s after stability
2025-01-28T14:55:10.037Z [INFO]  creating new snapshot: path=/tmp/raft2926251092/snapshots/2-113-1738076110037.tmp
2025-01-28T14:55:10.039Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: copied to local snapshot: bytes=71
2025-01-28T14:55:10.039Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: snapshot restore progress: id=2-113-1738076110037 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:10.040Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: removing all old logs from log store
2025-01-28T14:55:10.040Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: compacting logs: from=1 to=2
2025-01-28T14:55:10.040Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: restored user snapshot: index=113
2025-01-28T14:55:10.040Z [ERROR] server-7f361493-4921-4358-9436-b9aab5e113f8: failed to get log: index=2 error="log not found"
2025-01-28T14:55:10.040Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: aborting pipeline replication: peer="{Voter server-b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a}"
2025-01-28T14:55:10.040Z [ERROR] server-7f361493-4921-4358-9436-b9aab5e113f8: failed to get log: index=2 error="log not found"
2025-01-28T14:55:10.040Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: aborting pipeline replication: peer="{Voter server-cf010aad-ad2f-4235-d9c1-e40bc80247bd cf010aad-ad2f-4235-d9c1-e40bc80247bd}"
2025-01-28T14:55:10.047Z [ERROR] server-7f361493-4921-4358-9436-b9aab5e113f8: failed to get log: index=2 error="log not found"
2025-01-28T14:55:10.047Z [INFO]  creating new snapshot: path=/tmp/raft4191162390/snapshots/2-113-1738076110047.tmp
2025-01-28T14:55:10.047Z [ERROR] server-7f361493-4921-4358-9436-b9aab5e113f8: failed to get log: index=2 error="log not found"
2025-01-28T14:55:10.048Z [INFO]  server-b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:10.048Z [INFO]  creating new snapshot: path=/tmp/raft1426473459/snapshots/2-113-1738076110048.tmp
2025-01-28T14:55:10.049Z [INFO]  server-b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a: copied to local snapshot: bytes=71
2025-01-28T14:55:10.049Z [INFO]  server-cf010aad-ad2f-4235-d9c1-e40bc80247bd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:10.049Z [INFO]  server-b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a: snapshot restore progress: id=2-113-1738076110047 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:10.049Z [INFO]  server-b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a: removing all old logs from log store
2025-01-28T14:55:10.049Z [INFO]  server-b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a: compacting logs: from=1 to=2
2025-01-28T14:55:10.049Z [INFO]  server-b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a: Installed remote snapshot
2025-01-28T14:55:10.050Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: pipelining replication: peer="{Voter server-b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a b92cd6e7-6a01-6d82-cda1-5ac3b2a7864a}"
2025-01-28T14:55:10.050Z [INFO]  server-cf010aad-ad2f-4235-d9c1-e40bc80247bd: copied to local snapshot: bytes=71
2025-01-28T14:55:10.050Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:10.050Z [INFO]  server-cf010aad-ad2f-4235-d9c1-e40bc80247bd: snapshot restore progress: id=2-113-1738076110048 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:10.050Z [INFO]  server-cf010aad-ad2f-4235-d9c1-e40bc80247bd: removing all old logs from log store
2025-01-28T14:55:10.050Z [INFO]  server-cf010aad-ad2f-4235-d9c1-e40bc80247bd: compacting logs: from=1 to=2
2025-01-28T14:55:10.050Z [INFO]  server-cf010aad-ad2f-4235-d9c1-e40bc80247bd: Installed remote snapshot
2025-01-28T14:55:10.051Z [INFO]  server-7f361493-4921-4358-9436-b9aab5e113f8: pipelining replication: peer="{Voter server-cf010aad-ad2f-4235-d9c1-e40bc80247bd cf010aad-ad2f-4235-d9c1-e40bc80247bd}"
2025-01-28T14:55:11.056Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:10.050429653 +0000 UTC m=+95.724919633 (2 nodes), 2.22µs from start of poll, 1.550684913s from cluster start. Timeout at 2025-01-28 14:55:11.055517645 +0000 UTC m=+96.730007625, 1.005087992s after stability
2025-01-28T14:55:11.067Z [INFO]  server-3859bb36-6c31-79bf-81ca-d6aed133cd80: aborting pipeline replication: peer="{Voter server-ce8157db-7e2c-3baf-6af7-5d7ef93d5cad ce8157db-7e2c-3baf-6af7-5d7ef93d5cad}"
2025-01-28T14:55:11.067Z [INFO]  server-3859bb36-6c31-79bf-81ca-d6aed133cd80: aborting pipeline replication: peer="{Voter server-3ce8adbc-fd9d-ed3f-890d-927ccf732217 3ce8adbc-fd9d-ed3f-890d-927ccf732217}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false
2025-01-28T14:55:11.071Z [INFO]  server-bb466cd6-b8f1-2de2-0eff-bca81df6afff: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb466cd6-b8f1-2de2-0eff-bca81df6afff Address:bb466cd6-b8f1-2de2-0eff-bca81df6afff} {Suffrage:Voter ID:server-a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc Address:a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc} {Suffrage:Voter ID:server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe Address:b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe}]"
2025-01-28T14:55:11.071Z [INFO]  server-bb466cd6-b8f1-2de2-0eff-bca81df6afff: entering follower state: follower="Node at bb466cd6-b8f1-2de2-0eff-bca81df6afff [Follower]" leader-address= leader-id=
2025-01-28T14:55:11.071Z [INFO]  server-a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb466cd6-b8f1-2de2-0eff-bca81df6afff Address:bb466cd6-b8f1-2de2-0eff-bca81df6afff} {Suffrage:Voter ID:server-a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc Address:a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc} {Suffrage:Voter ID:server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe Address:b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe}]"
2025-01-28T14:55:11.071Z [INFO]  server-a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc: entering follower state: follower="Node at a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc [Follower]" leader-address= leader-id=
2025-01-28T14:55:11.071Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bb466cd6-b8f1-2de2-0eff-bca81df6afff Address:bb466cd6-b8f1-2de2-0eff-bca81df6afff} {Suffrage:Voter ID:server-a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc Address:a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc} {Suffrage:Voter ID:server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe Address:b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe}]"
2025-01-28T14:55:11.071Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:11.071Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: entering follower state: follower="Node at b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe [Follower]" leader-address= leader-id=
2025-01-28T14:55:11.653Z [WARN]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:11.653Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: entering candidate state: node="Node at b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe [Candidate]" term=2
2025-01-28T14:55:11.653Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: election won: term=2 tally=2
2025-01-28T14:55:11.653Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: entering leader state: leader="Node at b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe [Leader]"
2025-01-28T14:55:11.653Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: added peer, starting replication: peer=server-bb466cd6-b8f1-2de2-0eff-bca81df6afff
2025-01-28T14:55:11.653Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: added peer, starting replication: peer=server-a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc
2025-01-28T14:55:11.654Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: pipelining replication: peer="{Voter server-a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc}"
2025-01-28T14:55:11.654Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: pipelining replication: peer="{Voter server-bb466cd6-b8f1-2de2-0eff-bca81df6afff bb466cd6-b8f1-2de2-0eff-bca81df6afff}"
2025-01-28T14:55:12.659Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:11.653950622 +0000 UTC m=+97.328440602 (1 nodes), 581.997096ms from start of poll, 582.959466ms from cluster start. Timeout at 2025-01-28 14:55:12.658955241 +0000 UTC m=+98.333445241, 1.005004639s after stability
2025-01-28T14:55:12.660Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: starting snapshot up to: index=12
2025-01-28T14:55:12.660Z [INFO]  creating new snapshot: path=/tmp/raft1038531717/snapshots/2-12-1738076112660.tmp
2025-01-28T14:55:12.665Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: snapshot complete up to: index=12
2025-01-28T14:55:12.666Z [INFO]  creating new snapshot: path=/tmp/raft1038531717/snapshots/2-1013-1738076112666.tmp
2025-01-28T14:55:12.668Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: copied to local snapshot: bytes=71
2025-01-28T14:55:12.668Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: snapshot restore progress: id=2-1013-1738076112666 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:12.669Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: restored user snapshot: index=1013
2025-01-28T14:55:12.669Z [ERROR] server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: failed to get log: index=23 error="log not found"
2025-01-28T14:55:12.669Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: aborting pipeline replication: peer="{Voter server-a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc}"
2025-01-28T14:55:12.669Z [ERROR] server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: failed to get log: index=23 error="log not found"
2025-01-28T14:55:12.669Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: aborting pipeline replication: peer="{Voter server-bb466cd6-b8f1-2de2-0eff-bca81df6afff bb466cd6-b8f1-2de2-0eff-bca81df6afff}"
2025-01-28T14:55:12.676Z [ERROR] server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: failed to get log: index=23 error="log not found"
2025-01-28T14:55:12.677Z [INFO]  creating new snapshot: path=/tmp/raft687380656/snapshots/2-1013-1738076112677.tmp
2025-01-28T14:55:12.677Z [INFO]  server-bb466cd6-b8f1-2de2-0eff-bca81df6afff: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:12.678Z [INFO]  server-bb466cd6-b8f1-2de2-0eff-bca81df6afff: copied to local snapshot: bytes=71
2025-01-28T14:55:12.678Z [ERROR] server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: failed to get log: index=23 error="log not found"
2025-01-28T14:55:12.678Z [INFO]  server-bb466cd6-b8f1-2de2-0eff-bca81df6afff: snapshot restore progress: id=2-1013-1738076112677 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:12.679Z [INFO]  server-bb466cd6-b8f1-2de2-0eff-bca81df6afff: Installed remote snapshot
2025-01-28T14:55:12.679Z [WARN]  server-bb466cd6-b8f1-2de2-0eff-bca81df6afff: skipping application of old log: index=22
2025-01-28T14:55:12.679Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: pipelining replication: peer="{Voter server-bb466cd6-b8f1-2de2-0eff-bca81df6afff bb466cd6-b8f1-2de2-0eff-bca81df6afff}"
2025-01-28T14:55:12.679Z [INFO]  creating new snapshot: path=/tmp/raft3736291733/snapshots/2-1013-1738076112679.tmp
2025-01-28T14:55:12.679Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:12.680Z [INFO]  server-a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:12.680Z [INFO]  server-a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc: copied to local snapshot: bytes=71
2025-01-28T14:55:12.681Z [INFO]  server-a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc: snapshot restore progress: id=2-1013-1738076112679 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:12.681Z [INFO]  server-a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc: Installed remote snapshot
2025-01-28T14:55:12.681Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: pipelining replication: peer="{Voter server-a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc}"
2025-01-28T14:55:13.685Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:12.679688177 +0000 UTC m=+98.354178157 (2 nodes), 2.3µs from start of poll, 1.608697021s from cluster start. Timeout at 2025-01-28 14:55:13.684784008 +0000 UTC m=+99.359274008, 1.005095851s after stability
2025-01-28T14:55:13.697Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: aborting pipeline replication: peer="{Voter server-a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc a89d7eb1-497f-06ca-e6ee-77d5fc0d91dc}"
2025-01-28T14:55:13.697Z [INFO]  server-b6f8f0b9-53f5-0bd4-18e9-f06acaaffffe: aborting pipeline replication: peer="{Voter server-bb466cd6-b8f1-2de2-0eff-bca81df6afff bb466cd6-b8f1-2de2-0eff-bca81df6afff}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false
2025-01-28T14:55:13.702Z [INFO]  server-9c7a1ee8-5a4f-d630-10e3-905196f37b7d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c7a1ee8-5a4f-d630-10e3-905196f37b7d Address:9c7a1ee8-5a4f-d630-10e3-905196f37b7d} {Suffrage:Voter ID:server-978e1e58-ba7b-37c8-8e62-05f69c058a0d Address:978e1e58-ba7b-37c8-8e62-05f69c058a0d} {Suffrage:Voter ID:server-dad33df4-91fd-2588-99d0-ef28e498779d Address:dad33df4-91fd-2588-99d0-ef28e498779d}]"
2025-01-28T14:55:13.702Z [INFO]  server-9c7a1ee8-5a4f-d630-10e3-905196f37b7d: entering follower state: follower="Node at 9c7a1ee8-5a4f-d630-10e3-905196f37b7d [Follower]" leader-address= leader-id=
2025-01-28T14:55:13.702Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c7a1ee8-5a4f-d630-10e3-905196f37b7d Address:9c7a1ee8-5a4f-d630-10e3-905196f37b7d} {Suffrage:Voter ID:server-978e1e58-ba7b-37c8-8e62-05f69c058a0d Address:978e1e58-ba7b-37c8-8e62-05f69c058a0d} {Suffrage:Voter ID:server-dad33df4-91fd-2588-99d0-ef28e498779d Address:dad33df4-91fd-2588-99d0-ef28e498779d}]"
2025-01-28T14:55:13.702Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: entering follower state: follower="Node at 978e1e58-ba7b-37c8-8e62-05f69c058a0d [Follower]" leader-address= leader-id=
2025-01-28T14:55:13.702Z [INFO]  server-dad33df4-91fd-2588-99d0-ef28e498779d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9c7a1ee8-5a4f-d630-10e3-905196f37b7d Address:9c7a1ee8-5a4f-d630-10e3-905196f37b7d} {Suffrage:Voter ID:server-978e1e58-ba7b-37c8-8e62-05f69c058a0d Address:978e1e58-ba7b-37c8-8e62-05f69c058a0d} {Suffrage:Voter ID:server-dad33df4-91fd-2588-99d0-ef28e498779d Address:dad33df4-91fd-2588-99d0-ef28e498779d}]"
2025-01-28T14:55:13.703Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:13.703Z [INFO]  server-dad33df4-91fd-2588-99d0-ef28e498779d: entering follower state: follower="Node at dad33df4-91fd-2588-99d0-ef28e498779d [Follower]" leader-address= leader-id=
2025-01-28T14:55:14.252Z [WARN]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:14.252Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: entering candidate state: node="Node at 978e1e58-ba7b-37c8-8e62-05f69c058a0d [Candidate]" term=2
2025-01-28T14:55:14.252Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: election won: term=2 tally=2
2025-01-28T14:55:14.252Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: entering leader state: leader="Node at 978e1e58-ba7b-37c8-8e62-05f69c058a0d [Leader]"
2025-01-28T14:55:14.252Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: added peer, starting replication: peer=server-9c7a1ee8-5a4f-d630-10e3-905196f37b7d
2025-01-28T14:55:14.252Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: added peer, starting replication: peer=server-dad33df4-91fd-2588-99d0-ef28e498779d
2025-01-28T14:55:14.252Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: pipelining replication: peer="{Voter server-dad33df4-91fd-2588-99d0-ef28e498779d dad33df4-91fd-2588-99d0-ef28e498779d}"
2025-01-28T14:55:14.252Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: pipelining replication: peer="{Voter server-9c7a1ee8-5a4f-d630-10e3-905196f37b7d 9c7a1ee8-5a4f-d630-10e3-905196f37b7d}"
2025-01-28T14:55:15.258Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:14.25252776 +0000 UTC m=+99.927017740 (1 nodes), 549.41277ms from start of poll, 550.377179ms from cluster start. Timeout at 2025-01-28 14:55:15.2575307 +0000 UTC m=+100.932020740, 1.005003s after stability
2025-01-28T14:55:15.263Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: starting snapshot up to: index=12
2025-01-28T14:55:15.263Z [INFO]  creating new snapshot: path=/tmp/raft696981668/snapshots/2-12-1738076115263.tmp
2025-01-28T14:55:15.265Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: snapshot complete up to: index=12
2025-01-28T14:55:15.266Z [INFO]  creating new snapshot: path=/tmp/raft696981668/snapshots/2-1013-1738076115266.tmp
2025-01-28T14:55:15.268Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: copied to local snapshot: bytes=71
2025-01-28T14:55:15.268Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: snapshot restore progress: id=2-1013-1738076115266 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:15.268Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: removing all old logs from log store
2025-01-28T14:55:15.268Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: compacting logs: from=1 to=22
2025-01-28T14:55:15.268Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: restored user snapshot: index=1013
2025-01-28T14:55:15.269Z [ERROR] server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: failed to get log: index=22 error="log not found"
2025-01-28T14:55:15.269Z [ERROR] server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: failed to get log: index=22 error="log not found"
2025-01-28T14:55:15.269Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: aborting pipeline replication: peer="{Voter server-dad33df4-91fd-2588-99d0-ef28e498779d dad33df4-91fd-2588-99d0-ef28e498779d}"
2025-01-28T14:55:15.269Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: aborting pipeline replication: peer="{Voter server-9c7a1ee8-5a4f-d630-10e3-905196f37b7d 9c7a1ee8-5a4f-d630-10e3-905196f37b7d}"
2025-01-28T14:55:15.276Z [ERROR] server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: failed to get log: index=22 error="log not found"
2025-01-28T14:55:15.277Z [INFO]  creating new snapshot: path=/tmp/raft4006968070/snapshots/2-1013-1738076115277.tmp
2025-01-28T14:55:15.277Z [INFO]  server-9c7a1ee8-5a4f-d630-10e3-905196f37b7d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:15.278Z [INFO]  server-9c7a1ee8-5a4f-d630-10e3-905196f37b7d: copied to local snapshot: bytes=71
2025-01-28T14:55:15.278Z [ERROR] server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: failed to get log: index=22 error="log not found"
2025-01-28T14:55:15.279Z [INFO]  server-9c7a1ee8-5a4f-d630-10e3-905196f37b7d: snapshot restore progress: id=2-1013-1738076115277 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:15.279Z [INFO]  server-9c7a1ee8-5a4f-d630-10e3-905196f37b7d: removing all old logs from log store
2025-01-28T14:55:15.279Z [INFO]  server-9c7a1ee8-5a4f-d630-10e3-905196f37b7d: compacting logs: from=1 to=22
2025-01-28T14:55:15.279Z [INFO]  server-9c7a1ee8-5a4f-d630-10e3-905196f37b7d: Installed remote snapshot
2025-01-28T14:55:15.279Z [WARN]  server-9c7a1ee8-5a4f-d630-10e3-905196f37b7d: skipping application of old log: index=22
2025-01-28T14:55:15.279Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: pipelining replication: peer="{Voter server-9c7a1ee8-5a4f-d630-10e3-905196f37b7d 9c7a1ee8-5a4f-d630-10e3-905196f37b7d}"
2025-01-28T14:55:15.279Z [INFO]  creating new snapshot: path=/tmp/raft4185156934/snapshots/2-1013-1738076115279.tmp
2025-01-28T14:55:15.279Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:15.280Z [INFO]  server-dad33df4-91fd-2588-99d0-ef28e498779d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:15.281Z [INFO]  server-dad33df4-91fd-2588-99d0-ef28e498779d: copied to local snapshot: bytes=71
2025-01-28T14:55:15.281Z [INFO]  server-dad33df4-91fd-2588-99d0-ef28e498779d: snapshot restore progress: id=2-1013-1738076115279 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:15.281Z [INFO]  server-dad33df4-91fd-2588-99d0-ef28e498779d: removing all old logs from log store
2025-01-28T14:55:15.281Z [INFO]  server-dad33df4-91fd-2588-99d0-ef28e498779d: compacting logs: from=1 to=22
2025-01-28T14:55:15.281Z [INFO]  server-dad33df4-91fd-2588-99d0-ef28e498779d: Installed remote snapshot
2025-01-28T14:55:15.282Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: pipelining replication: peer="{Voter server-dad33df4-91fd-2588-99d0-ef28e498779d dad33df4-91fd-2588-99d0-ef28e498779d}"
2025-01-28T14:55:16.285Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:15.279711259 +0000 UTC m=+100.954201239 (2 nodes), 2.3µs from start of poll, 1.577560678s from cluster start. Timeout at 2025-01-28 14:55:16.284815509 +0000 UTC m=+101.959305509, 1.00510427s after stability
2025-01-28T14:55:16.297Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: aborting pipeline replication: peer="{Voter server-dad33df4-91fd-2588-99d0-ef28e498779d dad33df4-91fd-2588-99d0-ef28e498779d}"
2025-01-28T14:55:16.297Z [INFO]  server-978e1e58-ba7b-37c8-8e62-05f69c058a0d: aborting pipeline replication: peer="{Voter server-9c7a1ee8-5a4f-d630-10e3-905196f37b7d 9c7a1ee8-5a4f-d630-10e3-905196f37b7d}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true
2025-01-28T14:55:16.302Z [INFO]  server-38f2462b-6c70-a872-50af-c43450274792: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38f2462b-6c70-a872-50af-c43450274792 Address:38f2462b-6c70-a872-50af-c43450274792} {Suffrage:Voter ID:server-06719bad-053e-2626-907e-0053f45f65ea Address:06719bad-053e-2626-907e-0053f45f65ea} {Suffrage:Voter ID:server-d008feb8-48ae-d268-8a96-33bbc5fce90b Address:d008feb8-48ae-d268-8a96-33bbc5fce90b}]"
2025-01-28T14:55:16.302Z [INFO]  server-38f2462b-6c70-a872-50af-c43450274792: entering follower state: follower="Node at 38f2462b-6c70-a872-50af-c43450274792 [Follower]" leader-address= leader-id=
2025-01-28T14:55:16.302Z [INFO]  server-06719bad-053e-2626-907e-0053f45f65ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38f2462b-6c70-a872-50af-c43450274792 Address:38f2462b-6c70-a872-50af-c43450274792} {Suffrage:Voter ID:server-06719bad-053e-2626-907e-0053f45f65ea Address:06719bad-053e-2626-907e-0053f45f65ea} {Suffrage:Voter ID:server-d008feb8-48ae-d268-8a96-33bbc5fce90b Address:d008feb8-48ae-d268-8a96-33bbc5fce90b}]"
2025-01-28T14:55:16.302Z [INFO]  server-06719bad-053e-2626-907e-0053f45f65ea: entering follower state: follower="Node at 06719bad-053e-2626-907e-0053f45f65ea [Follower]" leader-address= leader-id=
2025-01-28T14:55:16.302Z [INFO]  server-d008feb8-48ae-d268-8a96-33bbc5fce90b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-38f2462b-6c70-a872-50af-c43450274792 Address:38f2462b-6c70-a872-50af-c43450274792} {Suffrage:Voter ID:server-06719bad-053e-2626-907e-0053f45f65ea Address:06719bad-053e-2626-907e-0053f45f65ea} {Suffrage:Voter ID:server-d008feb8-48ae-d268-8a96-33bbc5fce90b Address:d008feb8-48ae-d268-8a96-33bbc5fce90b}]"
2025-01-28T14:55:16.303Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:16.303Z [INFO]  server-d008feb8-48ae-d268-8a96-33bbc5fce90b: entering follower state: follower="Node at d008feb8-48ae-d268-8a96-33bbc5fce90b [Follower]" leader-address= leader-id=
2025-01-28T14:55:16.825Z [WARN]  server-38f2462b-6c70-a872-50af-c43450274792: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:16.825Z [INFO]  server-38f2462b-6c70-a872-50af-c43450274792: entering candidate state: node="Node at 38f2462b-6c70-a872-50af-c43450274792 [Candidate]" term=2
2025-01-28T14:55:16.825Z [INFO]  server-38f2462b-6c70-a872-50af-c43450274792: election won: term=2 tally=2
2025-01-28T14:55:16.825Z [INFO]  server-38f2462b-6c70-a872-50af-c43450274792: entering leader state: leader="Node at 38f2462b-6c70-a872-50af-c43450274792 [Leader]"
2025-01-28T14:55:16.825Z [INFO]  server-38f2462b-6c70-a872-50af-c43450274792: added peer, starting replication: peer=server-06719bad-053e-2626-907e-0053f45f65ea
2025-01-28T14:55:16.825Z [INFO]  server-38f2462b-6c70-a872-50af-c43450274792: added peer, starting replication: peer=server-d008feb8-48ae-d268-8a96-33bbc5fce90b
2025-01-28T14:55:16.825Z [INFO]  server-38f2462b-6c70-a872-50af-c43450274792: pipelining replication: peer="{Voter server-d008feb8-48ae-d268-8a96-33bbc5fce90b d008feb8-48ae-d268-8a96-33bbc5fce90b}"
2025-01-28T14:55:16.825Z [INFO]  server-38f2462b-6c70-a872-50af-c43450274792: pipelining replication: peer="{Voter server-06719bad-053e-2626-907e-0053f45f65ea 06719bad-053e-2626-907e-0053f45f65ea}"
2025-01-28T14:55:17.831Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:16.825633421 +0000 UTC m=+102.500123401 (1 nodes), 522.571162ms from start of poll, 523.484636ms from cluster start. Timeout at 2025-01-28 14:55:17.830648579 +0000 UTC m=+103.505138559, 1.005015158s after stability
2025-01-28T14:55:17.837Z [INFO]  server-38f2462b-6c70-a872-50af-c43450274792: starting snapshot up to: index=12
2025-01-28T14:55:17.837Z [INFO]  creating new snapshot: path=/tmp/raft4081606572/snapshots/2-12-1738076117837.tmp
2025-01-28T14:55:17.840Z [INFO]  server-38f2462b-6c70-a872-50af-c43450274792: snapshot complete up to: index=12
2025-01-28T14:55:17.843Z [INFO]  server-cd9f0463-c26e-9ad4-50bf-fdd8aecac857: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd9f0463-c26e-9ad4-50bf-fdd8aecac857 Address:cd9f0463-c26e-9ad4-50bf-fdd8aecac857} {Suffrage:Voter ID:server-1279708b-6d61-de12-355a-6937eaa3530b Address:1279708b-6d61-de12-355a-6937eaa3530b} {Suffrage:Voter ID:server-b6b8405e-9883-e6ab-ff3f-a54c520905b8 Address:b6b8405e-9883-e6ab-ff3f-a54c520905b8}]"
2025-01-28T14:55:17.843Z [INFO]  server-cd9f0463-c26e-9ad4-50bf-fdd8aecac857: entering follower state: follower="Node at cd9f0463-c26e-9ad4-50bf-fdd8aecac857 [Follower]" leader-address= leader-id=
2025-01-28T14:55:17.843Z [INFO]  server-1279708b-6d61-de12-355a-6937eaa3530b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd9f0463-c26e-9ad4-50bf-fdd8aecac857 Address:cd9f0463-c26e-9ad4-50bf-fdd8aecac857} {Suffrage:Voter ID:server-1279708b-6d61-de12-355a-6937eaa3530b Address:1279708b-6d61-de12-355a-6937eaa3530b} {Suffrage:Voter ID:server-b6b8405e-9883-e6ab-ff3f-a54c520905b8 Address:b6b8405e-9883-e6ab-ff3f-a54c520905b8}]"
2025-01-28T14:55:17.843Z [INFO]  server-1279708b-6d61-de12-355a-6937eaa3530b: entering follower state: follower="Node at 1279708b-6d61-de12-355a-6937eaa3530b [Follower]" leader-address= leader-id=
2025-01-28T14:55:17.844Z [INFO]  server-b6b8405e-9883-e6ab-ff3f-a54c520905b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cd9f0463-c26e-9ad4-50bf-fdd8aecac857 Address:cd9f0463-c26e-9ad4-50bf-fdd8aecac857} {Suffrage:Voter ID:server-1279708b-6d61-de12-355a-6937eaa3530b Address:1279708b-6d61-de12-355a-6937eaa3530b} {Suffrage:Voter ID:server-b6b8405e-9883-e6ab-ff3f-a54c520905b8 Address:b6b8405e-9883-e6ab-ff3f-a54c520905b8}]"
2025-01-28T14:55:17.844Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:17.844Z [INFO]  server-b6b8405e-9883-e6ab-ff3f-a54c520905b8: entering follower state: follower="Node at b6b8405e-9883-e6ab-ff3f-a54c520905b8 [Follower]" leader-address= leader-id=
2025-01-28T14:55:18.434Z [WARN]  server-1279708b-6d61-de12-355a-6937eaa3530b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:18.434Z [INFO]  server-1279708b-6d61-de12-355a-6937eaa3530b: entering candidate state: node="Node at 1279708b-6d61-de12-355a-6937eaa3530b [Candidate]" term=2
2025-01-28T14:55:18.435Z [INFO]  server-1279708b-6d61-de12-355a-6937eaa3530b: election won: term=2 tally=2
2025-01-28T14:55:18.435Z [INFO]  server-1279708b-6d61-de12-355a-6937eaa3530b: entering leader state: leader="Node at 1279708b-6d61-de12-355a-6937eaa3530b [Leader]"
2025-01-28T14:55:18.435Z [INFO]  server-1279708b-6d61-de12-355a-6937eaa3530b: added peer, starting replication: peer=server-cd9f0463-c26e-9ad4-50bf-fdd8aecac857
2025-01-28T14:55:18.435Z [INFO]  server-1279708b-6d61-de12-355a-6937eaa3530b: added peer, starting replication: peer=server-b6b8405e-9883-e6ab-ff3f-a54c520905b8
2025-01-28T14:55:18.435Z [INFO]  server-1279708b-6d61-de12-355a-6937eaa3530b: pipelining replication: peer="{Voter server-cd9f0463-c26e-9ad4-50bf-fdd8aecac857 cd9f0463-c26e-9ad4-50bf-fdd8aecac857}"
2025-01-28T14:55:18.435Z [INFO]  server-1279708b-6d61-de12-355a-6937eaa3530b: pipelining replication: peer="{Voter server-b6b8405e-9883-e6ab-ff3f-a54c520905b8 b6b8405e-9883-e6ab-ff3f-a54c520905b8}"
2025-01-28T14:55:19.441Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:18.435155124 +0000 UTC m=+104.109645084 (1 nodes), 590.849161ms from start of poll, 591.996633ms from cluster start. Timeout at 2025-01-28 14:55:19.440164603 +0000 UTC m=+105.114654583, 1.005009499s after stability
2025-01-28T14:55:19.442Z [INFO]  creating new snapshot: path=/tmp/raft3003612796/snapshots/2-1013-1738076119442.tmp
2025-01-28T14:55:19.443Z [INFO]  server-1279708b-6d61-de12-355a-6937eaa3530b: copied to local snapshot: bytes=71
2025-01-28T14:55:19.444Z [INFO]  server-1279708b-6d61-de12-355a-6937eaa3530b: snapshot restore progress: id=2-1013-1738076119442 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:19.444Z [INFO]  server-1279708b-6d61-de12-355a-6937eaa3530b: restored user snapshot: index=1013
2025-01-28T14:55:19.444Z [ERROR] server-1279708b-6d61-de12-355a-6937eaa3530b: failed to get log: index=3 error="log not found"
2025-01-28T14:55:19.444Z [INFO]  server-1279708b-6d61-de12-355a-6937eaa3530b: aborting pipeline replication: peer="{Voter server-b6b8405e-9883-e6ab-ff3f-a54c520905b8 b6b8405e-9883-e6ab-ff3f-a54c520905b8}"
2025-01-28T14:55:19.444Z [ERROR] server-1279708b-6d61-de12-355a-6937eaa3530b: failed to get log: index=3 error="log not found"
2025-01-28T14:55:19.444Z [INFO]  server-1279708b-6d61-de12-355a-6937eaa3530b: aborting pipeline replication: peer="{Voter server-cd9f0463-c26e-9ad4-50bf-fdd8aecac857 cd9f0463-c26e-9ad4-50bf-fdd8aecac857}"
2025-01-28T14:55:19.450Z [ERROR] server-1279708b-6d61-de12-355a-6937eaa3530b: failed to get log: index=3 error="log not found"
2025-01-28T14:55:19.452Z [INFO]  creating new snapshot: path=/tmp/raft538748978/snapshots/2-1013-1738076119452.tmp
2025-01-28T14:55:19.452Z [INFO]  server-cd9f0463-c26e-9ad4-50bf-fdd8aecac857: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:19.453Z [INFO]  server-cd9f0463-c26e-9ad4-50bf-fdd8aecac857: copied to local snapshot: bytes=71
2025-01-28T14:55:19.454Z [INFO]  server-cd9f0463-c26e-9ad4-50bf-fdd8aecac857: snapshot restore progress: id=2-1013-1738076119452 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:19.454Z [INFO]  server-cd9f0463-c26e-9ad4-50bf-fdd8aecac857: Installed remote snapshot
2025-01-28T14:55:19.454Z [INFO]  server-1279708b-6d61-de12-355a-6937eaa3530b: pipelining replication: peer="{Voter server-cd9f0463-c26e-9ad4-50bf-fdd8aecac857 cd9f0463-c26e-9ad4-50bf-fdd8aecac857}"
2025-01-28T14:55:19.455Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:19.457Z [ERROR] server-1279708b-6d61-de12-355a-6937eaa3530b: failed to get log: index=3 error="log not found"
2025-01-28T14:55:19.457Z [INFO]  creating new snapshot: path=/tmp/raft3103627393/snapshots/2-1013-1738076119457.tmp
2025-01-28T14:55:19.458Z [INFO]  server-b6b8405e-9883-e6ab-ff3f-a54c520905b8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:19.459Z [INFO]  server-b6b8405e-9883-e6ab-ff3f-a54c520905b8: copied to local snapshot: bytes=71
2025-01-28T14:55:19.459Z [INFO]  server-b6b8405e-9883-e6ab-ff3f-a54c520905b8: snapshot restore progress: id=2-1013-1738076119457 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:19.460Z [INFO]  server-b6b8405e-9883-e6ab-ff3f-a54c520905b8: Installed remote snapshot
2025-01-28T14:55:19.460Z [INFO]  server-1279708b-6d61-de12-355a-6937eaa3530b: pipelining replication: peer="{Voter server-b6b8405e-9883-e6ab-ff3f-a54c520905b8 b6b8405e-9883-e6ab-ff3f-a54c520905b8}"
2025-01-28T14:55:20.461Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:19.455082727 +0000 UTC m=+105.129572687 (2 nodes), 2.359µs from start of poll, 1.611924236s from cluster start. Timeout at 2025-01-28 14:55:20.460182778 +0000 UTC m=+106.134672838, 1.005100151s after stability
2025-01-28T14:55:20.472Z [INFO]  server-38f2462b-6c70-a872-50af-c43450274792: aborting pipeline replication: peer="{Voter server-d008feb8-48ae-d268-8a96-33bbc5fce90b d008feb8-48ae-d268-8a96-33bbc5fce90b}"
2025-01-28T14:55:20.472Z [INFO]  server-38f2462b-6c70-a872-50af-c43450274792: aborting pipeline replication: peer="{Voter server-06719bad-053e-2626-907e-0053f45f65ea 06719bad-053e-2626-907e-0053f45f65ea}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true
2025-01-28T14:55:20.476Z [INFO]  server-42a1f769-6cd6-036d-3c0b-5a133b0f1eb9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42a1f769-6cd6-036d-3c0b-5a133b0f1eb9 Address:42a1f769-6cd6-036d-3c0b-5a133b0f1eb9} {Suffrage:Voter ID:server-b9e3aad9-e9e6-85f9-361d-d215f4a207cd Address:b9e3aad9-e9e6-85f9-361d-d215f4a207cd} {Suffrage:Voter ID:server-60f4ed59-de49-f423-3161-c1381e85a1e0 Address:60f4ed59-de49-f423-3161-c1381e85a1e0}]"
2025-01-28T14:55:20.476Z [INFO]  server-42a1f769-6cd6-036d-3c0b-5a133b0f1eb9: entering follower state: follower="Node at 42a1f769-6cd6-036d-3c0b-5a133b0f1eb9 [Follower]" leader-address= leader-id=
2025-01-28T14:55:20.476Z [INFO]  server-b9e3aad9-e9e6-85f9-361d-d215f4a207cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42a1f769-6cd6-036d-3c0b-5a133b0f1eb9 Address:42a1f769-6cd6-036d-3c0b-5a133b0f1eb9} {Suffrage:Voter ID:server-b9e3aad9-e9e6-85f9-361d-d215f4a207cd Address:b9e3aad9-e9e6-85f9-361d-d215f4a207cd} {Suffrage:Voter ID:server-60f4ed59-de49-f423-3161-c1381e85a1e0 Address:60f4ed59-de49-f423-3161-c1381e85a1e0}]"
2025-01-28T14:55:20.476Z [INFO]  server-b9e3aad9-e9e6-85f9-361d-d215f4a207cd: entering follower state: follower="Node at b9e3aad9-e9e6-85f9-361d-d215f4a207cd [Follower]" leader-address= leader-id=
2025-01-28T14:55:20.476Z [INFO]  server-60f4ed59-de49-f423-3161-c1381e85a1e0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-42a1f769-6cd6-036d-3c0b-5a133b0f1eb9 Address:42a1f769-6cd6-036d-3c0b-5a133b0f1eb9} {Suffrage:Voter ID:server-b9e3aad9-e9e6-85f9-361d-d215f4a207cd Address:b9e3aad9-e9e6-85f9-361d-d215f4a207cd} {Suffrage:Voter ID:server-60f4ed59-de49-f423-3161-c1381e85a1e0 Address:60f4ed59-de49-f423-3161-c1381e85a1e0}]"
2025-01-28T14:55:20.476Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:20.477Z [INFO]  server-60f4ed59-de49-f423-3161-c1381e85a1e0: entering follower state: follower="Node at 60f4ed59-de49-f423-3161-c1381e85a1e0 [Follower]" leader-address= leader-id=
2025-01-28T14:55:21.126Z [WARN]  server-b9e3aad9-e9e6-85f9-361d-d215f4a207cd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:21.126Z [INFO]  server-b9e3aad9-e9e6-85f9-361d-d215f4a207cd: entering candidate state: node="Node at b9e3aad9-e9e6-85f9-361d-d215f4a207cd [Candidate]" term=2
2025-01-28T14:55:21.126Z [INFO]  server-b9e3aad9-e9e6-85f9-361d-d215f4a207cd: election won: term=2 tally=2
2025-01-28T14:55:21.126Z [INFO]  server-b9e3aad9-e9e6-85f9-361d-d215f4a207cd: entering leader state: leader="Node at b9e3aad9-e9e6-85f9-361d-d215f4a207cd [Leader]"
2025-01-28T14:55:21.126Z [INFO]  server-b9e3aad9-e9e6-85f9-361d-d215f4a207cd: added peer, starting replication: peer=server-42a1f769-6cd6-036d-3c0b-5a133b0f1eb9
2025-01-28T14:55:21.126Z [INFO]  server-b9e3aad9-e9e6-85f9-361d-d215f4a207cd: added peer, starting replication: peer=server-60f4ed59-de49-f423-3161-c1381e85a1e0
2025-01-28T14:55:21.126Z [INFO]  server-b9e3aad9-e9e6-85f9-361d-d215f4a207cd: pipelining replication: peer="{Voter server-60f4ed59-de49-f423-3161-c1381e85a1e0 60f4ed59-de49-f423-3161-c1381e85a1e0}"
2025-01-28T14:55:21.127Z [INFO]  server-b9e3aad9-e9e6-85f9-361d-d215f4a207cd: pipelining replication: peer="{Voter server-42a1f769-6cd6-036d-3c0b-5a133b0f1eb9 42a1f769-6cd6-036d-3c0b-5a133b0f1eb9}"
2025-01-28T14:55:22.132Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:21.126675635 +0000 UTC m=+106.801165615 (1 nodes), 649.649914ms from start of poll, 650.446619ms from cluster start. Timeout at 2025-01-28 14:55:22.131685434 +0000 UTC m=+107.806175414, 1.005009799s after stability
2025-01-28T14:55:22.133Z [INFO]  server-b9e3aad9-e9e6-85f9-361d-d215f4a207cd: starting snapshot up to: index=12
2025-01-28T14:55:22.133Z [INFO]  creating new snapshot: path=/tmp/raft3002474244/snapshots/2-12-1738076122133.tmp
2025-01-28T14:55:22.136Z [INFO]  server-b9e3aad9-e9e6-85f9-361d-d215f4a207cd: snapshot complete up to: index=12
2025-01-28T14:55:22.141Z [INFO]  server-bcae2595-402b-7bec-9a33-21ade35672fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcae2595-402b-7bec-9a33-21ade35672fe Address:bcae2595-402b-7bec-9a33-21ade35672fe} {Suffrage:Voter ID:server-4b888034-289d-dca9-34f0-95dc436fecc5 Address:4b888034-289d-dca9-34f0-95dc436fecc5} {Suffrage:Voter ID:server-47773f58-222a-9b0f-314b-edf27ff170af Address:47773f58-222a-9b0f-314b-edf27ff170af}]"
2025-01-28T14:55:22.141Z [INFO]  server-bcae2595-402b-7bec-9a33-21ade35672fe: entering follower state: follower="Node at bcae2595-402b-7bec-9a33-21ade35672fe [Follower]" leader-address= leader-id=
2025-01-28T14:55:22.141Z [INFO]  server-4b888034-289d-dca9-34f0-95dc436fecc5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcae2595-402b-7bec-9a33-21ade35672fe Address:bcae2595-402b-7bec-9a33-21ade35672fe} {Suffrage:Voter ID:server-4b888034-289d-dca9-34f0-95dc436fecc5 Address:4b888034-289d-dca9-34f0-95dc436fecc5} {Suffrage:Voter ID:server-47773f58-222a-9b0f-314b-edf27ff170af Address:47773f58-222a-9b0f-314b-edf27ff170af}]"
2025-01-28T14:55:22.141Z [INFO]  server-4b888034-289d-dca9-34f0-95dc436fecc5: entering follower state: follower="Node at 4b888034-289d-dca9-34f0-95dc436fecc5 [Follower]" leader-address= leader-id=
2025-01-28T14:55:22.141Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bcae2595-402b-7bec-9a33-21ade35672fe Address:bcae2595-402b-7bec-9a33-21ade35672fe} {Suffrage:Voter ID:server-4b888034-289d-dca9-34f0-95dc436fecc5 Address:4b888034-289d-dca9-34f0-95dc436fecc5} {Suffrage:Voter ID:server-47773f58-222a-9b0f-314b-edf27ff170af Address:47773f58-222a-9b0f-314b-edf27ff170af}]"
2025-01-28T14:55:22.141Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:22.141Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: entering follower state: follower="Node at 47773f58-222a-9b0f-314b-edf27ff170af [Follower]" leader-address= leader-id=
2025-01-28T14:55:22.747Z [WARN]  server-47773f58-222a-9b0f-314b-edf27ff170af: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:22.748Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: entering candidate state: node="Node at 47773f58-222a-9b0f-314b-edf27ff170af [Candidate]" term=2
2025-01-28T14:55:22.748Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: election won: term=2 tally=2
2025-01-28T14:55:22.748Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: entering leader state: leader="Node at 47773f58-222a-9b0f-314b-edf27ff170af [Leader]"
2025-01-28T14:55:22.749Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: added peer, starting replication: peer=server-bcae2595-402b-7bec-9a33-21ade35672fe
2025-01-28T14:55:22.749Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: added peer, starting replication: peer=server-4b888034-289d-dca9-34f0-95dc436fecc5
2025-01-28T14:55:22.751Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: pipelining replication: peer="{Voter server-bcae2595-402b-7bec-9a33-21ade35672fe bcae2595-402b-7bec-9a33-21ade35672fe}"
2025-01-28T14:55:22.752Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: pipelining replication: peer="{Voter server-4b888034-289d-dca9-34f0-95dc436fecc5 4b888034-289d-dca9-34f0-95dc436fecc5}"
2025-01-28T14:55:23.754Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:22.749302562 +0000 UTC m=+108.423792542 (1 nodes), 607.353884ms from start of poll, 608.200145ms from cluster start. Timeout at 2025-01-28 14:55:23.754310241 +0000 UTC m=+109.428800221, 1.005007679s after stability
2025-01-28T14:55:23.755Z [INFO]  creating new snapshot: path=/tmp/raft3203437808/snapshots/2-1013-1738076123755.tmp
2025-01-28T14:55:23.757Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: copied to local snapshot: bytes=71
2025-01-28T14:55:23.758Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: snapshot restore progress: id=2-1013-1738076123755 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:23.758Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: removing all old logs from log store
2025-01-28T14:55:23.758Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: compacting logs: from=1 to=2
2025-01-28T14:55:23.758Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: restored user snapshot: index=1013
2025-01-28T14:55:23.758Z [ERROR] server-47773f58-222a-9b0f-314b-edf27ff170af: failed to get log: index=2 error="log not found"
2025-01-28T14:55:23.758Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: aborting pipeline replication: peer="{Voter server-4b888034-289d-dca9-34f0-95dc436fecc5 4b888034-289d-dca9-34f0-95dc436fecc5}"
2025-01-28T14:55:23.758Z [ERROR] server-47773f58-222a-9b0f-314b-edf27ff170af: failed to get log: index=2 error="log not found"
2025-01-28T14:55:23.758Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: aborting pipeline replication: peer="{Voter server-bcae2595-402b-7bec-9a33-21ade35672fe bcae2595-402b-7bec-9a33-21ade35672fe}"
2025-01-28T14:55:23.763Z [ERROR] server-47773f58-222a-9b0f-314b-edf27ff170af: failed to get log: index=2 error="log not found"
2025-01-28T14:55:23.764Z [INFO]  creating new snapshot: path=/tmp/raft2496577681/snapshots/2-1013-1738076123764.tmp
2025-01-28T14:55:23.764Z [ERROR] server-47773f58-222a-9b0f-314b-edf27ff170af: failed to get log: index=2 error="log not found"
2025-01-28T14:55:23.765Z [INFO]  server-4b888034-289d-dca9-34f0-95dc436fecc5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:23.765Z [INFO]  creating new snapshot: path=/tmp/raft2947565179/snapshots/2-1013-1738076123765.tmp
2025-01-28T14:55:23.766Z [INFO]  server-4b888034-289d-dca9-34f0-95dc436fecc5: copied to local snapshot: bytes=71
2025-01-28T14:55:23.766Z [INFO]  server-bcae2595-402b-7bec-9a33-21ade35672fe: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:23.767Z [INFO]  server-bcae2595-402b-7bec-9a33-21ade35672fe: copied to local snapshot: bytes=71
2025-01-28T14:55:23.767Z [INFO]  server-4b888034-289d-dca9-34f0-95dc436fecc5: snapshot restore progress: id=2-1013-1738076123764 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:23.769Z [INFO]  server-4b888034-289d-dca9-34f0-95dc436fecc5: removing all old logs from log store
2025-01-28T14:55:23.769Z [INFO]  server-4b888034-289d-dca9-34f0-95dc436fecc5: compacting logs: from=1 to=2
2025-01-28T14:55:23.769Z [INFO]  server-4b888034-289d-dca9-34f0-95dc436fecc5: Installed remote snapshot
2025-01-28T14:55:23.770Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: pipelining replication: peer="{Voter server-4b888034-289d-dca9-34f0-95dc436fecc5 4b888034-289d-dca9-34f0-95dc436fecc5}"
2025-01-28T14:55:23.771Z [INFO]  server-bcae2595-402b-7bec-9a33-21ade35672fe: snapshot restore progress: id=2-1013-1738076123765 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:23.771Z [INFO]  server-bcae2595-402b-7bec-9a33-21ade35672fe: removing all old logs from log store
2025-01-28T14:55:23.771Z [INFO]  server-bcae2595-402b-7bec-9a33-21ade35672fe: compacting logs: from=1 to=2
2025-01-28T14:55:23.771Z [INFO]  server-bcae2595-402b-7bec-9a33-21ade35672fe: Installed remote snapshot
2025-01-28T14:55:23.771Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:23.771Z [INFO]  server-47773f58-222a-9b0f-314b-edf27ff170af: pipelining replication: peer="{Voter server-bcae2595-402b-7bec-9a33-21ade35672fe bcae2595-402b-7bec-9a33-21ade35672fe}"
2025-01-28T14:55:24.777Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:23.771577484 +0000 UTC m=+109.446067464 (2 nodes), 2.2µs from start of poll, 1.630475067s from cluster start. Timeout at 2025-01-28 14:55:24.776734489 +0000 UTC m=+110.451224449, 1.005156985s after stability
2025-01-28T14:55:24.789Z [INFO]  server-b9e3aad9-e9e6-85f9-361d-d215f4a207cd: aborting pipeline replication: peer="{Voter server-60f4ed59-de49-f423-3161-c1381e85a1e0 60f4ed59-de49-f423-3161-c1381e85a1e0}"
2025-01-28T14:55:24.789Z [INFO]  server-b9e3aad9-e9e6-85f9-361d-d215f4a207cd: aborting pipeline replication: peer="{Voter server-42a1f769-6cd6-036d-3c0b-5a133b0f1eb9 42a1f769-6cd6-036d-3c0b-5a133b0f1eb9}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false
2025-01-28T14:55:24.793Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c Address:1b0a5d29-a527-f3af-3fd5-278ce33bad5c} {Suffrage:Voter ID:server-71a325e5-0e8a-6458-1fe9-ab9fbaf89058 Address:71a325e5-0e8a-6458-1fe9-ab9fbaf89058} {Suffrage:Voter ID:server-f4cb0155-f07c-36ec-da11-8420eadfa1b6 Address:f4cb0155-f07c-36ec-da11-8420eadfa1b6}]"
2025-01-28T14:55:24.793Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: entering follower state: follower="Node at 1b0a5d29-a527-f3af-3fd5-278ce33bad5c [Follower]" leader-address= leader-id=
2025-01-28T14:55:24.793Z [INFO]  server-71a325e5-0e8a-6458-1fe9-ab9fbaf89058: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c Address:1b0a5d29-a527-f3af-3fd5-278ce33bad5c} {Suffrage:Voter ID:server-71a325e5-0e8a-6458-1fe9-ab9fbaf89058 Address:71a325e5-0e8a-6458-1fe9-ab9fbaf89058} {Suffrage:Voter ID:server-f4cb0155-f07c-36ec-da11-8420eadfa1b6 Address:f4cb0155-f07c-36ec-da11-8420eadfa1b6}]"
2025-01-28T14:55:24.793Z [INFO]  server-71a325e5-0e8a-6458-1fe9-ab9fbaf89058: entering follower state: follower="Node at 71a325e5-0e8a-6458-1fe9-ab9fbaf89058 [Follower]" leader-address= leader-id=
2025-01-28T14:55:24.794Z [INFO]  server-f4cb0155-f07c-36ec-da11-8420eadfa1b6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c Address:1b0a5d29-a527-f3af-3fd5-278ce33bad5c} {Suffrage:Voter ID:server-71a325e5-0e8a-6458-1fe9-ab9fbaf89058 Address:71a325e5-0e8a-6458-1fe9-ab9fbaf89058} {Suffrage:Voter ID:server-f4cb0155-f07c-36ec-da11-8420eadfa1b6 Address:f4cb0155-f07c-36ec-da11-8420eadfa1b6}]"
2025-01-28T14:55:24.794Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:24.794Z [INFO]  server-f4cb0155-f07c-36ec-da11-8420eadfa1b6: entering follower state: follower="Node at f4cb0155-f07c-36ec-da11-8420eadfa1b6 [Follower]" leader-address= leader-id=
2025-01-28T14:55:25.559Z [WARN]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:25.559Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: entering candidate state: node="Node at 1b0a5d29-a527-f3af-3fd5-278ce33bad5c [Candidate]" term=2
2025-01-28T14:55:25.560Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: election won: term=2 tally=2
2025-01-28T14:55:25.560Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: entering leader state: leader="Node at 1b0a5d29-a527-f3af-3fd5-278ce33bad5c [Leader]"
2025-01-28T14:55:25.560Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: added peer, starting replication: peer=server-71a325e5-0e8a-6458-1fe9-ab9fbaf89058
2025-01-28T14:55:25.560Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: added peer, starting replication: peer=server-f4cb0155-f07c-36ec-da11-8420eadfa1b6
2025-01-28T14:55:25.560Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: pipelining replication: peer="{Voter server-71a325e5-0e8a-6458-1fe9-ab9fbaf89058 71a325e5-0e8a-6458-1fe9-ab9fbaf89058}"
2025-01-28T14:55:25.560Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: pipelining replication: peer="{Voter server-f4cb0155-f07c-36ec-da11-8420eadfa1b6 f4cb0155-f07c-36ec-da11-8420eadfa1b6}"
2025-01-28T14:55:26.565Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:25.560191532 +0000 UTC m=+111.234681512 (1 nodes), 765.854512ms from start of poll, 766.919751ms from cluster start. Timeout at 2025-01-28 14:55:26.565193211 +0000 UTC m=+112.239683211, 1.005001699s after stability
2025-01-28T14:55:26.566Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: starting snapshot up to: index=12
2025-01-28T14:55:26.566Z [INFO]  creating new snapshot: path=/tmp/raft449509607/snapshots/2-12-1738076126566.tmp
2025-01-28T14:55:26.567Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: snapshot complete up to: index=12
2025-01-28T14:55:26.569Z [INFO]  creating new snapshot: path=/tmp/raft449509607/snapshots/2-10013-1738076126569.tmp
2025-01-28T14:55:26.570Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: copied to local snapshot: bytes=71
2025-01-28T14:55:26.571Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: snapshot restore progress: id=2-10013-1738076126569 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:26.571Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: restored user snapshot: index=10013
2025-01-28T14:55:26.571Z [ERROR] server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: failed to get log: index=23 error="log not found"
2025-01-28T14:55:26.571Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: aborting pipeline replication: peer="{Voter server-71a325e5-0e8a-6458-1fe9-ab9fbaf89058 71a325e5-0e8a-6458-1fe9-ab9fbaf89058}"
2025-01-28T14:55:26.571Z [ERROR] server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: failed to get log: index=23 error="log not found"
2025-01-28T14:55:26.571Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: aborting pipeline replication: peer="{Voter server-f4cb0155-f07c-36ec-da11-8420eadfa1b6 f4cb0155-f07c-36ec-da11-8420eadfa1b6}"
2025-01-28T14:55:26.579Z [ERROR] server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: failed to get log: index=23 error="log not found"
2025-01-28T14:55:26.580Z [INFO]  creating new snapshot: path=/tmp/raft2780785844/snapshots/2-10013-1738076126580.tmp
2025-01-28T14:55:26.580Z [ERROR] server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: failed to get log: index=23 error="log not found"
2025-01-28T14:55:26.580Z [INFO]  server-71a325e5-0e8a-6458-1fe9-ab9fbaf89058: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:26.581Z [INFO]  creating new snapshot: path=/tmp/raft1465756817/snapshots/2-10013-1738076126581.tmp
2025-01-28T14:55:26.581Z [INFO]  server-71a325e5-0e8a-6458-1fe9-ab9fbaf89058: copied to local snapshot: bytes=71
2025-01-28T14:55:26.581Z [INFO]  server-f4cb0155-f07c-36ec-da11-8420eadfa1b6: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:26.582Z [INFO]  server-71a325e5-0e8a-6458-1fe9-ab9fbaf89058: snapshot restore progress: id=2-10013-1738076126580 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:26.582Z [INFO]  server-71a325e5-0e8a-6458-1fe9-ab9fbaf89058: Installed remote snapshot
2025-01-28T14:55:26.582Z [WARN]  server-71a325e5-0e8a-6458-1fe9-ab9fbaf89058: skipping application of old log: index=22
2025-01-28T14:55:26.582Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: pipelining replication: peer="{Voter server-71a325e5-0e8a-6458-1fe9-ab9fbaf89058 71a325e5-0e8a-6458-1fe9-ab9fbaf89058}"
2025-01-28T14:55:26.582Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:26.582Z [INFO]  server-f4cb0155-f07c-36ec-da11-8420eadfa1b6: copied to local snapshot: bytes=71
2025-01-28T14:55:26.582Z [INFO]  server-f4cb0155-f07c-36ec-da11-8420eadfa1b6: snapshot restore progress: id=2-10013-1738076126581 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:26.583Z [INFO]  server-f4cb0155-f07c-36ec-da11-8420eadfa1b6: Installed remote snapshot
2025-01-28T14:55:26.583Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: pipelining replication: peer="{Voter server-f4cb0155-f07c-36ec-da11-8420eadfa1b6 f4cb0155-f07c-36ec-da11-8420eadfa1b6}"
2025-01-28T14:55:27.587Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:26.582559786 +0000 UTC m=+112.257049766 (2 nodes), 2.46µs from start of poll, 1.789288005s from cluster start. Timeout at 2025-01-28 14:55:27.587688494 +0000 UTC m=+113.262178474, 1.005128708s after stability
2025-01-28T14:55:27.602Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: aborting pipeline replication: peer="{Voter server-71a325e5-0e8a-6458-1fe9-ab9fbaf89058 71a325e5-0e8a-6458-1fe9-ab9fbaf89058}"
2025-01-28T14:55:27.602Z [INFO]  server-1b0a5d29-a527-f3af-3fd5-278ce33bad5c: aborting pipeline replication: peer="{Voter server-f4cb0155-f07c-36ec-da11-8420eadfa1b6 f4cb0155-f07c-36ec-da11-8420eadfa1b6}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false
2025-01-28T14:55:27.606Z [INFO]  server-04f6db8c-16a6-107a-6dda-cbb76cd6025d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04f6db8c-16a6-107a-6dda-cbb76cd6025d Address:04f6db8c-16a6-107a-6dda-cbb76cd6025d} {Suffrage:Voter ID:server-c0e4b846-4b00-2b4e-a1d3-4b8075da1062 Address:c0e4b846-4b00-2b4e-a1d3-4b8075da1062} {Suffrage:Voter ID:server-c5bdfb74-bba4-3b2e-8174-33b86652d143 Address:c5bdfb74-bba4-3b2e-8174-33b86652d143}]"
2025-01-28T14:55:27.607Z [INFO]  server-04f6db8c-16a6-107a-6dda-cbb76cd6025d: entering follower state: follower="Node at 04f6db8c-16a6-107a-6dda-cbb76cd6025d [Follower]" leader-address= leader-id=
2025-01-28T14:55:27.607Z [INFO]  server-c0e4b846-4b00-2b4e-a1d3-4b8075da1062: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04f6db8c-16a6-107a-6dda-cbb76cd6025d Address:04f6db8c-16a6-107a-6dda-cbb76cd6025d} {Suffrage:Voter ID:server-c0e4b846-4b00-2b4e-a1d3-4b8075da1062 Address:c0e4b846-4b00-2b4e-a1d3-4b8075da1062} {Suffrage:Voter ID:server-c5bdfb74-bba4-3b2e-8174-33b86652d143 Address:c5bdfb74-bba4-3b2e-8174-33b86652d143}]"
2025-01-28T14:55:27.607Z [INFO]  server-c0e4b846-4b00-2b4e-a1d3-4b8075da1062: entering follower state: follower="Node at c0e4b846-4b00-2b4e-a1d3-4b8075da1062 [Follower]" leader-address= leader-id=
2025-01-28T14:55:27.607Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04f6db8c-16a6-107a-6dda-cbb76cd6025d Address:04f6db8c-16a6-107a-6dda-cbb76cd6025d} {Suffrage:Voter ID:server-c0e4b846-4b00-2b4e-a1d3-4b8075da1062 Address:c0e4b846-4b00-2b4e-a1d3-4b8075da1062} {Suffrage:Voter ID:server-c5bdfb74-bba4-3b2e-8174-33b86652d143 Address:c5bdfb74-bba4-3b2e-8174-33b86652d143}]"
2025-01-28T14:55:27.607Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:27.607Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: entering follower state: follower="Node at c5bdfb74-bba4-3b2e-8174-33b86652d143 [Follower]" leader-address= leader-id=
2025-01-28T14:55:28.295Z [WARN]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:28.295Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: entering candidate state: node="Node at c5bdfb74-bba4-3b2e-8174-33b86652d143 [Candidate]" term=2
2025-01-28T14:55:28.296Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: election won: term=2 tally=2
2025-01-28T14:55:28.296Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: entering leader state: leader="Node at c5bdfb74-bba4-3b2e-8174-33b86652d143 [Leader]"
2025-01-28T14:55:28.296Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: added peer, starting replication: peer=server-04f6db8c-16a6-107a-6dda-cbb76cd6025d
2025-01-28T14:55:28.296Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: added peer, starting replication: peer=server-c0e4b846-4b00-2b4e-a1d3-4b8075da1062
2025-01-28T14:55:28.296Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: pipelining replication: peer="{Voter server-04f6db8c-16a6-107a-6dda-cbb76cd6025d 04f6db8c-16a6-107a-6dda-cbb76cd6025d}"
2025-01-28T14:55:28.296Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: pipelining replication: peer="{Voter server-c0e4b846-4b00-2b4e-a1d3-4b8075da1062 c0e4b846-4b00-2b4e-a1d3-4b8075da1062}"
2025-01-28T14:55:29.301Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:28.296199547 +0000 UTC m=+113.970689527 (1 nodes), 688.421117ms from start of poll, 689.459299ms from cluster start. Timeout at 2025-01-28 14:55:29.301203447 +0000 UTC m=+114.975693467, 1.00500394s after stability
2025-01-28T14:55:29.302Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: starting snapshot up to: index=12
2025-01-28T14:55:29.303Z [INFO]  creating new snapshot: path=/tmp/raft3367337895/snapshots/2-12-1738076129303.tmp
2025-01-28T14:55:29.304Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: snapshot complete up to: index=12
2025-01-28T14:55:29.306Z [INFO]  creating new snapshot: path=/tmp/raft3367337895/snapshots/2-10013-1738076129306.tmp
2025-01-28T14:55:29.308Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: copied to local snapshot: bytes=71
2025-01-28T14:55:29.308Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: snapshot restore progress: id=2-10013-1738076129306 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:29.308Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: removing all old logs from log store
2025-01-28T14:55:29.308Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: compacting logs: from=1 to=22
2025-01-28T14:55:29.308Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: restored user snapshot: index=10013
2025-01-28T14:55:29.308Z [ERROR] server-c5bdfb74-bba4-3b2e-8174-33b86652d143: failed to get log: index=22 error="log not found"
2025-01-28T14:55:29.308Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: aborting pipeline replication: peer="{Voter server-c0e4b846-4b00-2b4e-a1d3-4b8075da1062 c0e4b846-4b00-2b4e-a1d3-4b8075da1062}"
2025-01-28T14:55:29.308Z [ERROR] server-c5bdfb74-bba4-3b2e-8174-33b86652d143: failed to get log: index=22 error="log not found"
2025-01-28T14:55:29.308Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: aborting pipeline replication: peer="{Voter server-04f6db8c-16a6-107a-6dda-cbb76cd6025d 04f6db8c-16a6-107a-6dda-cbb76cd6025d}"
2025-01-28T14:55:29.316Z [ERROR] server-c5bdfb74-bba4-3b2e-8174-33b86652d143: failed to get log: index=22 error="log not found"
2025-01-28T14:55:29.317Z [INFO]  creating new snapshot: path=/tmp/raft1246544600/snapshots/2-10013-1738076129317.tmp
2025-01-28T14:55:29.317Z [INFO]  server-04f6db8c-16a6-107a-6dda-cbb76cd6025d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:29.318Z [INFO]  server-04f6db8c-16a6-107a-6dda-cbb76cd6025d: copied to local snapshot: bytes=71
2025-01-28T14:55:29.318Z [ERROR] server-c5bdfb74-bba4-3b2e-8174-33b86652d143: failed to get log: index=22 error="log not found"
2025-01-28T14:55:29.319Z [INFO]  server-04f6db8c-16a6-107a-6dda-cbb76cd6025d: snapshot restore progress: id=2-10013-1738076129317 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:29.319Z [INFO]  server-04f6db8c-16a6-107a-6dda-cbb76cd6025d: removing all old logs from log store
2025-01-28T14:55:29.319Z [INFO]  server-04f6db8c-16a6-107a-6dda-cbb76cd6025d: compacting logs: from=1 to=22
2025-01-28T14:55:29.319Z [INFO]  server-04f6db8c-16a6-107a-6dda-cbb76cd6025d: Installed remote snapshot
2025-01-28T14:55:29.319Z [WARN]  server-04f6db8c-16a6-107a-6dda-cbb76cd6025d: skipping application of old log: index=22
2025-01-28T14:55:29.319Z [INFO]  creating new snapshot: path=/tmp/raft3390697256/snapshots/2-10013-1738076129319.tmp
2025-01-28T14:55:29.319Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: pipelining replication: peer="{Voter server-04f6db8c-16a6-107a-6dda-cbb76cd6025d 04f6db8c-16a6-107a-6dda-cbb76cd6025d}"
2025-01-28T14:55:29.319Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:29.320Z [INFO]  server-c0e4b846-4b00-2b4e-a1d3-4b8075da1062: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:29.321Z [INFO]  server-c0e4b846-4b00-2b4e-a1d3-4b8075da1062: copied to local snapshot: bytes=71
2025-01-28T14:55:29.321Z [INFO]  server-c0e4b846-4b00-2b4e-a1d3-4b8075da1062: snapshot restore progress: id=2-10013-1738076129319 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:29.321Z [INFO]  server-c0e4b846-4b00-2b4e-a1d3-4b8075da1062: removing all old logs from log store
2025-01-28T14:55:29.321Z [INFO]  server-c0e4b846-4b00-2b4e-a1d3-4b8075da1062: compacting logs: from=1 to=22
2025-01-28T14:55:29.321Z [INFO]  server-c0e4b846-4b00-2b4e-a1d3-4b8075da1062: Installed remote snapshot
2025-01-28T14:55:29.322Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: pipelining replication: peer="{Voter server-c0e4b846-4b00-2b4e-a1d3-4b8075da1062 c0e4b846-4b00-2b4e-a1d3-4b8075da1062}"
2025-01-28T14:55:30.325Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:29.319862279 +0000 UTC m=+114.994352259 (2 nodes), 2.16µs from start of poll, 1.713122031s from cluster start. Timeout at 2025-01-28 14:55:30.324999585 +0000 UTC m=+115.999489625, 1.005137366s after stability
2025-01-28T14:55:30.338Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: aborting pipeline replication: peer="{Voter server-c0e4b846-4b00-2b4e-a1d3-4b8075da1062 c0e4b846-4b00-2b4e-a1d3-4b8075da1062}"
2025-01-28T14:55:30.338Z [INFO]  server-c5bdfb74-bba4-3b2e-8174-33b86652d143: aborting pipeline replication: peer="{Voter server-04f6db8c-16a6-107a-6dda-cbb76cd6025d 04f6db8c-16a6-107a-6dda-cbb76cd6025d}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true
2025-01-28T14:55:30.344Z [INFO]  server-dd0ab195-ec18-fd17-33af-5cbc8ef3d20e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd0ab195-ec18-fd17-33af-5cbc8ef3d20e Address:dd0ab195-ec18-fd17-33af-5cbc8ef3d20e} {Suffrage:Voter ID:server-72c1e8bd-8f17-13e4-0964-413c5ff8d25f Address:72c1e8bd-8f17-13e4-0964-413c5ff8d25f} {Suffrage:Voter ID:server-d01ad084-15fb-8aed-0d0f-655a24fbf32c Address:d01ad084-15fb-8aed-0d0f-655a24fbf32c}]"
2025-01-28T14:55:30.344Z [INFO]  server-dd0ab195-ec18-fd17-33af-5cbc8ef3d20e: entering follower state: follower="Node at dd0ab195-ec18-fd17-33af-5cbc8ef3d20e [Follower]" leader-address= leader-id=
2025-01-28T14:55:30.344Z [INFO]  server-72c1e8bd-8f17-13e4-0964-413c5ff8d25f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd0ab195-ec18-fd17-33af-5cbc8ef3d20e Address:dd0ab195-ec18-fd17-33af-5cbc8ef3d20e} {Suffrage:Voter ID:server-72c1e8bd-8f17-13e4-0964-413c5ff8d25f Address:72c1e8bd-8f17-13e4-0964-413c5ff8d25f} {Suffrage:Voter ID:server-d01ad084-15fb-8aed-0d0f-655a24fbf32c Address:d01ad084-15fb-8aed-0d0f-655a24fbf32c}]"
2025-01-28T14:55:30.345Z [INFO]  server-72c1e8bd-8f17-13e4-0964-413c5ff8d25f: entering follower state: follower="Node at 72c1e8bd-8f17-13e4-0964-413c5ff8d25f [Follower]" leader-address= leader-id=
2025-01-28T14:55:30.345Z [INFO]  server-d01ad084-15fb-8aed-0d0f-655a24fbf32c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd0ab195-ec18-fd17-33af-5cbc8ef3d20e Address:dd0ab195-ec18-fd17-33af-5cbc8ef3d20e} {Suffrage:Voter ID:server-72c1e8bd-8f17-13e4-0964-413c5ff8d25f Address:72c1e8bd-8f17-13e4-0964-413c5ff8d25f} {Suffrage:Voter ID:server-d01ad084-15fb-8aed-0d0f-655a24fbf32c Address:d01ad084-15fb-8aed-0d0f-655a24fbf32c}]"
2025-01-28T14:55:30.345Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:30.345Z [INFO]  server-d01ad084-15fb-8aed-0d0f-655a24fbf32c: entering follower state: follower="Node at d01ad084-15fb-8aed-0d0f-655a24fbf32c [Follower]" leader-address= leader-id=
2025-01-28T14:55:30.891Z [WARN]  server-72c1e8bd-8f17-13e4-0964-413c5ff8d25f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:30.891Z [INFO]  server-72c1e8bd-8f17-13e4-0964-413c5ff8d25f: entering candidate state: node="Node at 72c1e8bd-8f17-13e4-0964-413c5ff8d25f [Candidate]" term=2
2025-01-28T14:55:30.891Z [INFO]  server-72c1e8bd-8f17-13e4-0964-413c5ff8d25f: election won: term=2 tally=2
2025-01-28T14:55:30.891Z [INFO]  server-72c1e8bd-8f17-13e4-0964-413c5ff8d25f: entering leader state: leader="Node at 72c1e8bd-8f17-13e4-0964-413c5ff8d25f [Leader]"
2025-01-28T14:55:30.891Z [INFO]  server-72c1e8bd-8f17-13e4-0964-413c5ff8d25f: added peer, starting replication: peer=server-dd0ab195-ec18-fd17-33af-5cbc8ef3d20e
2025-01-28T14:55:30.891Z [INFO]  server-72c1e8bd-8f17-13e4-0964-413c5ff8d25f: added peer, starting replication: peer=server-d01ad084-15fb-8aed-0d0f-655a24fbf32c
2025-01-28T14:55:30.892Z [INFO]  server-72c1e8bd-8f17-13e4-0964-413c5ff8d25f: pipelining replication: peer="{Voter server-d01ad084-15fb-8aed-0d0f-655a24fbf32c d01ad084-15fb-8aed-0d0f-655a24fbf32c}"
2025-01-28T14:55:30.892Z [INFO]  server-72c1e8bd-8f17-13e4-0964-413c5ff8d25f: pipelining replication: peer="{Voter server-dd0ab195-ec18-fd17-33af-5cbc8ef3d20e dd0ab195-ec18-fd17-33af-5cbc8ef3d20e}"
2025-01-28T14:55:31.896Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:30.891697897 +0000 UTC m=+116.566187877 (1 nodes), 546.410093ms from start of poll, 547.405159ms from cluster start. Timeout at 2025-01-28 14:55:31.896701017 +0000 UTC m=+117.571191017, 1.00500314s after stability
2025-01-28T14:55:31.898Z [INFO]  server-72c1e8bd-8f17-13e4-0964-413c5ff8d25f: starting snapshot up to: index=12
2025-01-28T14:55:31.898Z [INFO]  creating new snapshot: path=/tmp/raft146499436/snapshots/2-12-1738076131898.tmp
2025-01-28T14:55:31.900Z [INFO]  server-72c1e8bd-8f17-13e4-0964-413c5ff8d25f: snapshot complete up to: index=12
2025-01-28T14:55:31.903Z [INFO]  server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f Address:d9eb3999-75b8-bfe3-8539-cfc480e4fb0f} {Suffrage:Voter ID:server-b451307e-bf96-5198-3f54-a03762055e2c Address:b451307e-bf96-5198-3f54-a03762055e2c} {Suffrage:Voter ID:server-808e21be-0bb8-9a48-5555-3c8e9f1ac813 Address:808e21be-0bb8-9a48-5555-3c8e9f1ac813}]"
2025-01-28T14:55:31.904Z [INFO]  server-b451307e-bf96-5198-3f54-a03762055e2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f Address:d9eb3999-75b8-bfe3-8539-cfc480e4fb0f} {Suffrage:Voter ID:server-b451307e-bf96-5198-3f54-a03762055e2c Address:b451307e-bf96-5198-3f54-a03762055e2c} {Suffrage:Voter ID:server-808e21be-0bb8-9a48-5555-3c8e9f1ac813 Address:808e21be-0bb8-9a48-5555-3c8e9f1ac813}]"
2025-01-28T14:55:31.904Z [INFO]  server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: entering follower state: follower="Node at d9eb3999-75b8-bfe3-8539-cfc480e4fb0f [Follower]" leader-address= leader-id=
2025-01-28T14:55:31.904Z [INFO]  server-b451307e-bf96-5198-3f54-a03762055e2c: entering follower state: follower="Node at b451307e-bf96-5198-3f54-a03762055e2c [Follower]" leader-address= leader-id=
2025-01-28T14:55:31.904Z [INFO]  server-808e21be-0bb8-9a48-5555-3c8e9f1ac813: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f Address:d9eb3999-75b8-bfe3-8539-cfc480e4fb0f} {Suffrage:Voter ID:server-b451307e-bf96-5198-3f54-a03762055e2c Address:b451307e-bf96-5198-3f54-a03762055e2c} {Suffrage:Voter ID:server-808e21be-0bb8-9a48-5555-3c8e9f1ac813 Address:808e21be-0bb8-9a48-5555-3c8e9f1ac813}]"
2025-01-28T14:55:31.904Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:31.904Z [INFO]  server-808e21be-0bb8-9a48-5555-3c8e9f1ac813: entering follower state: follower="Node at 808e21be-0bb8-9a48-5555-3c8e9f1ac813 [Follower]" leader-address= leader-id=
2025-01-28T14:55:32.570Z [WARN]  server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:32.570Z [INFO]  server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: entering candidate state: node="Node at d9eb3999-75b8-bfe3-8539-cfc480e4fb0f [Candidate]" term=2
2025-01-28T14:55:32.570Z [INFO]  server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: election won: term=2 tally=2
2025-01-28T14:55:32.570Z [INFO]  server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: entering leader state: leader="Node at d9eb3999-75b8-bfe3-8539-cfc480e4fb0f [Leader]"
2025-01-28T14:55:32.571Z [INFO]  server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: added peer, starting replication: peer=server-b451307e-bf96-5198-3f54-a03762055e2c
2025-01-28T14:55:32.571Z [INFO]  server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: added peer, starting replication: peer=server-808e21be-0bb8-9a48-5555-3c8e9f1ac813
2025-01-28T14:55:32.571Z [INFO]  server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: pipelining replication: peer="{Voter server-b451307e-bf96-5198-3f54-a03762055e2c b451307e-bf96-5198-3f54-a03762055e2c}"
2025-01-28T14:55:32.571Z [INFO]  server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: pipelining replication: peer="{Voter server-808e21be-0bb8-9a48-5555-3c8e9f1ac813 808e21be-0bb8-9a48-5555-3c8e9f1ac813}"
2025-01-28T14:55:33.576Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:32.571033555 +0000 UTC m=+118.245523535 (1 nodes), 666.359438ms from start of poll, 667.39854ms from cluster start. Timeout at 2025-01-28 14:55:33.576040335 +0000 UTC m=+119.250530415, 1.00500688s after stability
2025-01-28T14:55:33.576Z [INFO]  creating new snapshot: path=/tmp/raft4183933472/snapshots/2-10013-1738076133576.tmp
2025-01-28T14:55:33.578Z [INFO]  server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: copied to local snapshot: bytes=71
2025-01-28T14:55:33.579Z [INFO]  server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: snapshot restore progress: id=2-10013-1738076133576 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:33.579Z [INFO]  server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: restored user snapshot: index=10013
2025-01-28T14:55:33.579Z [ERROR] server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: failed to get log: index=3 error="log not found"
2025-01-28T14:55:33.579Z [INFO]  server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: aborting pipeline replication: peer="{Voter server-808e21be-0bb8-9a48-5555-3c8e9f1ac813 808e21be-0bb8-9a48-5555-3c8e9f1ac813}"
2025-01-28T14:55:33.579Z [ERROR] server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: failed to get log: index=3 error="log not found"
2025-01-28T14:55:33.580Z [INFO]  server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: aborting pipeline replication: peer="{Voter server-b451307e-bf96-5198-3f54-a03762055e2c b451307e-bf96-5198-3f54-a03762055e2c}"
2025-01-28T14:55:33.589Z [ERROR] server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: failed to get log: index=3 error="log not found"
2025-01-28T14:55:33.590Z [ERROR] server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: failed to get log: index=3 error="log not found"
2025-01-28T14:55:33.590Z [INFO]  creating new snapshot: path=/tmp/raft2639486270/snapshots/2-10013-1738076133590.tmp
2025-01-28T14:55:33.591Z [INFO]  creating new snapshot: path=/tmp/raft3354896893/snapshots/2-10013-1738076133591.tmp
2025-01-28T14:55:33.591Z [INFO]  server-b451307e-bf96-5198-3f54-a03762055e2c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:33.592Z [INFO]  server-808e21be-0bb8-9a48-5555-3c8e9f1ac813: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:33.592Z [INFO]  server-b451307e-bf96-5198-3f54-a03762055e2c: copied to local snapshot: bytes=71
2025-01-28T14:55:33.592Z [INFO]  server-808e21be-0bb8-9a48-5555-3c8e9f1ac813: copied to local snapshot: bytes=71
2025-01-28T14:55:33.593Z [INFO]  server-b451307e-bf96-5198-3f54-a03762055e2c: snapshot restore progress: id=2-10013-1738076133590 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:33.593Z [INFO]  server-b451307e-bf96-5198-3f54-a03762055e2c: Installed remote snapshot
2025-01-28T14:55:33.593Z [INFO]  server-808e21be-0bb8-9a48-5555-3c8e9f1ac813: snapshot restore progress: id=2-10013-1738076133591 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:33.593Z [WARN]  server-b451307e-bf96-5198-3f54-a03762055e2c: skipping application of old log: index=10013
2025-01-28T14:55:33.593Z [INFO]  server-808e21be-0bb8-9a48-5555-3c8e9f1ac813: Installed remote snapshot
2025-01-28T14:55:33.593Z [INFO]  server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: pipelining replication: peer="{Voter server-b451307e-bf96-5198-3f54-a03762055e2c b451307e-bf96-5198-3f54-a03762055e2c}"
2025-01-28T14:55:33.593Z [WARN]  server-808e21be-0bb8-9a48-5555-3c8e9f1ac813: skipping application of old log: index=10013
2025-01-28T14:55:33.593Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:33.593Z [INFO]  server-d9eb3999-75b8-bfe3-8539-cfc480e4fb0f: pipelining replication: peer="{Voter server-808e21be-0bb8-9a48-5555-3c8e9f1ac813 808e21be-0bb8-9a48-5555-3c8e9f1ac813}"
2025-01-28T14:55:34.599Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:33.593945718 +0000 UTC m=+119.268435698 (2 nodes), 2.86µs from start of poll, 1.690310703s from cluster start. Timeout at 2025-01-28 14:55:34.599189875 +0000 UTC m=+120.273679855, 1.005244157s after stability
2025-01-28T14:55:34.612Z [INFO]  server-72c1e8bd-8f17-13e4-0964-413c5ff8d25f: aborting pipeline replication: peer="{Voter server-dd0ab195-ec18-fd17-33af-5cbc8ef3d20e dd0ab195-ec18-fd17-33af-5cbc8ef3d20e}"
2025-01-28T14:55:34.612Z [INFO]  server-72c1e8bd-8f17-13e4-0964-413c5ff8d25f: aborting pipeline replication: peer="{Voter server-d01ad084-15fb-8aed-0d0f-655a24fbf32c d01ad084-15fb-8aed-0d0f-655a24fbf32c}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true
2025-01-28T14:55:34.621Z [INFO]  server-104efdcf-a081-9f52-7794-f9a4ae371d5d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-104efdcf-a081-9f52-7794-f9a4ae371d5d Address:104efdcf-a081-9f52-7794-f9a4ae371d5d} {Suffrage:Voter ID:server-373cd188-51fc-85d5-97b3-492821fe0623 Address:373cd188-51fc-85d5-97b3-492821fe0623} {Suffrage:Voter ID:server-d4c1da89-f157-768b-7e09-66e59e01dfa5 Address:d4c1da89-f157-768b-7e09-66e59e01dfa5}]"
2025-01-28T14:55:34.621Z [INFO]  server-104efdcf-a081-9f52-7794-f9a4ae371d5d: entering follower state: follower="Node at 104efdcf-a081-9f52-7794-f9a4ae371d5d [Follower]" leader-address= leader-id=
2025-01-28T14:55:34.621Z [INFO]  server-373cd188-51fc-85d5-97b3-492821fe0623: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-104efdcf-a081-9f52-7794-f9a4ae371d5d Address:104efdcf-a081-9f52-7794-f9a4ae371d5d} {Suffrage:Voter ID:server-373cd188-51fc-85d5-97b3-492821fe0623 Address:373cd188-51fc-85d5-97b3-492821fe0623} {Suffrage:Voter ID:server-d4c1da89-f157-768b-7e09-66e59e01dfa5 Address:d4c1da89-f157-768b-7e09-66e59e01dfa5}]"
2025-01-28T14:55:34.621Z [INFO]  server-373cd188-51fc-85d5-97b3-492821fe0623: entering follower state: follower="Node at 373cd188-51fc-85d5-97b3-492821fe0623 [Follower]" leader-address= leader-id=
2025-01-28T14:55:34.621Z [INFO]  server-d4c1da89-f157-768b-7e09-66e59e01dfa5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-104efdcf-a081-9f52-7794-f9a4ae371d5d Address:104efdcf-a081-9f52-7794-f9a4ae371d5d} {Suffrage:Voter ID:server-373cd188-51fc-85d5-97b3-492821fe0623 Address:373cd188-51fc-85d5-97b3-492821fe0623} {Suffrage:Voter ID:server-d4c1da89-f157-768b-7e09-66e59e01dfa5 Address:d4c1da89-f157-768b-7e09-66e59e01dfa5}]"
2025-01-28T14:55:34.621Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:34.621Z [INFO]  server-d4c1da89-f157-768b-7e09-66e59e01dfa5: entering follower state: follower="Node at d4c1da89-f157-768b-7e09-66e59e01dfa5 [Follower]" leader-address= leader-id=
2025-01-28T14:55:35.135Z [WARN]  server-d4c1da89-f157-768b-7e09-66e59e01dfa5: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:35.135Z [INFO]  server-d4c1da89-f157-768b-7e09-66e59e01dfa5: entering candidate state: node="Node at d4c1da89-f157-768b-7e09-66e59e01dfa5 [Candidate]" term=2
2025-01-28T14:55:35.135Z [INFO]  server-d4c1da89-f157-768b-7e09-66e59e01dfa5: election won: term=2 tally=2
2025-01-28T14:55:35.135Z [INFO]  server-d4c1da89-f157-768b-7e09-66e59e01dfa5: entering leader state: leader="Node at d4c1da89-f157-768b-7e09-66e59e01dfa5 [Leader]"
2025-01-28T14:55:35.135Z [INFO]  server-d4c1da89-f157-768b-7e09-66e59e01dfa5: added peer, starting replication: peer=server-104efdcf-a081-9f52-7794-f9a4ae371d5d
2025-01-28T14:55:35.135Z [INFO]  server-d4c1da89-f157-768b-7e09-66e59e01dfa5: added peer, starting replication: peer=server-373cd188-51fc-85d5-97b3-492821fe0623
2025-01-28T14:55:35.136Z [INFO]  server-d4c1da89-f157-768b-7e09-66e59e01dfa5: pipelining replication: peer="{Voter server-373cd188-51fc-85d5-97b3-492821fe0623 373cd188-51fc-85d5-97b3-492821fe0623}"
2025-01-28T14:55:35.136Z [INFO]  server-d4c1da89-f157-768b-7e09-66e59e01dfa5: pipelining replication: peer="{Voter server-104efdcf-a081-9f52-7794-f9a4ae371d5d 104efdcf-a081-9f52-7794-f9a4ae371d5d}"
2025-01-28T14:55:36.141Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:35.135727952 +0000 UTC m=+120.810217932 (1 nodes), 514.057766ms from start of poll, 514.951362ms from cluster start. Timeout at 2025-01-28 14:55:36.140751889 +0000 UTC m=+121.815241869, 1.005023937s after stability
2025-01-28T14:55:36.142Z [INFO]  server-d4c1da89-f157-768b-7e09-66e59e01dfa5: starting snapshot up to: index=12
2025-01-28T14:55:36.142Z [INFO]  creating new snapshot: path=/tmp/raft291674726/snapshots/2-12-1738076136142.tmp
2025-01-28T14:55:36.144Z [INFO]  server-d4c1da89-f157-768b-7e09-66e59e01dfa5: snapshot complete up to: index=12
2025-01-28T14:55:36.147Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1db11d7-518d-da10-aff3-ca6b3ab71779 Address:b1db11d7-518d-da10-aff3-ca6b3ab71779} {Suffrage:Voter ID:server-adfb35ab-4c3a-5d71-ff65-714e5999c9af Address:adfb35ab-4c3a-5d71-ff65-714e5999c9af} {Suffrage:Voter ID:server-41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc Address:41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc}]"
2025-01-28T14:55:36.148Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: entering follower state: follower="Node at b1db11d7-518d-da10-aff3-ca6b3ab71779 [Follower]" leader-address= leader-id=
2025-01-28T14:55:36.148Z [INFO]  server-adfb35ab-4c3a-5d71-ff65-714e5999c9af: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1db11d7-518d-da10-aff3-ca6b3ab71779 Address:b1db11d7-518d-da10-aff3-ca6b3ab71779} {Suffrage:Voter ID:server-adfb35ab-4c3a-5d71-ff65-714e5999c9af Address:adfb35ab-4c3a-5d71-ff65-714e5999c9af} {Suffrage:Voter ID:server-41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc Address:41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc}]"
2025-01-28T14:55:36.148Z [INFO]  server-adfb35ab-4c3a-5d71-ff65-714e5999c9af: entering follower state: follower="Node at adfb35ab-4c3a-5d71-ff65-714e5999c9af [Follower]" leader-address= leader-id=
2025-01-28T14:55:36.148Z [INFO]  server-41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b1db11d7-518d-da10-aff3-ca6b3ab71779 Address:b1db11d7-518d-da10-aff3-ca6b3ab71779} {Suffrage:Voter ID:server-adfb35ab-4c3a-5d71-ff65-714e5999c9af Address:adfb35ab-4c3a-5d71-ff65-714e5999c9af} {Suffrage:Voter ID:server-41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc Address:41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc}]"
2025-01-28T14:55:36.148Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:36.148Z [INFO]  server-41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc: entering follower state: follower="Node at 41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc [Follower]" leader-address= leader-id=
2025-01-28T14:55:36.723Z [WARN]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:36.724Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: entering candidate state: node="Node at b1db11d7-518d-da10-aff3-ca6b3ab71779 [Candidate]" term=2
2025-01-28T14:55:36.724Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: election won: term=2 tally=2
2025-01-28T14:55:36.724Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: entering leader state: leader="Node at b1db11d7-518d-da10-aff3-ca6b3ab71779 [Leader]"
2025-01-28T14:55:36.724Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: added peer, starting replication: peer=server-adfb35ab-4c3a-5d71-ff65-714e5999c9af
2025-01-28T14:55:36.724Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: added peer, starting replication: peer=server-41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc
2025-01-28T14:55:36.724Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: pipelining replication: peer="{Voter server-adfb35ab-4c3a-5d71-ff65-714e5999c9af adfb35ab-4c3a-5d71-ff65-714e5999c9af}"
2025-01-28T14:55:36.724Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: pipelining replication: peer="{Voter server-41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc 41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc}"
2025-01-28T14:55:37.729Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:36.724406042 +0000 UTC m=+122.398896022 (1 nodes), 575.747586ms from start of poll, 576.949993ms from cluster start. Timeout at 2025-01-28 14:55:37.729418681 +0000 UTC m=+123.403908641, 1.005012619s after stability
2025-01-28T14:55:37.730Z [INFO]  creating new snapshot: path=/tmp/raft2894792163/snapshots/2-10013-1738076137730.tmp
2025-01-28T14:55:37.731Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: copied to local snapshot: bytes=71
2025-01-28T14:55:37.732Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: snapshot restore progress: id=2-10013-1738076137730 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:37.732Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: removing all old logs from log store
2025-01-28T14:55:37.732Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: compacting logs: from=1 to=2
2025-01-28T14:55:37.732Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: restored user snapshot: index=10013
2025-01-28T14:55:37.732Z [ERROR] server-b1db11d7-518d-da10-aff3-ca6b3ab71779: failed to get log: index=2 error="log not found"
2025-01-28T14:55:37.733Z [ERROR] server-b1db11d7-518d-da10-aff3-ca6b3ab71779: failed to get log: index=2 error="log not found"
2025-01-28T14:55:37.733Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: aborting pipeline replication: peer="{Voter server-adfb35ab-4c3a-5d71-ff65-714e5999c9af adfb35ab-4c3a-5d71-ff65-714e5999c9af}"
2025-01-28T14:55:37.733Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: aborting pipeline replication: peer="{Voter server-41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc 41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc}"
2025-01-28T14:55:37.739Z [ERROR] server-b1db11d7-518d-da10-aff3-ca6b3ab71779: failed to get log: index=2 error="log not found"
2025-01-28T14:55:37.740Z [INFO]  creating new snapshot: path=/tmp/raft531455212/snapshots/2-10013-1738076137740.tmp
2025-01-28T14:55:37.741Z [INFO]  server-adfb35ab-4c3a-5d71-ff65-714e5999c9af: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:37.742Z [ERROR] server-b1db11d7-518d-da10-aff3-ca6b3ab71779: failed to get log: index=2 error="log not found"
2025-01-28T14:55:37.742Z [INFO]  server-adfb35ab-4c3a-5d71-ff65-714e5999c9af: copied to local snapshot: bytes=71
2025-01-28T14:55:37.743Z [INFO]  creating new snapshot: path=/tmp/raft1391662181/snapshots/2-10013-1738076137743.tmp
2025-01-28T14:55:37.743Z [INFO]  server-adfb35ab-4c3a-5d71-ff65-714e5999c9af: snapshot restore progress: id=2-10013-1738076137740 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:37.743Z [INFO]  server-adfb35ab-4c3a-5d71-ff65-714e5999c9af: removing all old logs from log store
2025-01-28T14:55:37.743Z [INFO]  server-adfb35ab-4c3a-5d71-ff65-714e5999c9af: compacting logs: from=1 to=2
2025-01-28T14:55:37.743Z [INFO]  server-adfb35ab-4c3a-5d71-ff65-714e5999c9af: Installed remote snapshot
2025-01-28T14:55:37.743Z [WARN]  server-adfb35ab-4c3a-5d71-ff65-714e5999c9af: skipping application of old log: index=10013
2025-01-28T14:55:37.743Z [INFO]  server-41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:37.743Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: pipelining replication: peer="{Voter server-adfb35ab-4c3a-5d71-ff65-714e5999c9af adfb35ab-4c3a-5d71-ff65-714e5999c9af}"
2025-01-28T14:55:37.743Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:37.744Z [INFO]  server-41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc: copied to local snapshot: bytes=71
2025-01-28T14:55:37.745Z [INFO]  server-41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc: snapshot restore progress: id=2-10013-1738076137743 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2025-01-28T14:55:37.745Z [INFO]  server-41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc: removing all old logs from log store
2025-01-28T14:55:37.745Z [INFO]  server-41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc: compacting logs: from=1 to=2
2025-01-28T14:55:37.745Z [INFO]  server-41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc: Installed remote snapshot
2025-01-28T14:55:37.746Z [INFO]  server-b1db11d7-518d-da10-aff3-ca6b3ab71779: pipelining replication: peer="{Voter server-41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc 41ef8dd7-3fc5-2de4-d9d4-1acbe98d81dc}"
2025-01-28T14:55:38.750Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:37.74397652 +0000 UTC m=+123.418466500 (2 nodes), 2.24µs from start of poll, 1.596520471s from cluster start. Timeout at 2025-01-28 14:55:38.749233836 +0000 UTC m=+124.423723876, 1.005257376s after stability
2025-01-28T14:55:38.762Z [INFO]  server-d4c1da89-f157-768b-7e09-66e59e01dfa5: aborting pipeline replication: peer="{Voter server-104efdcf-a081-9f52-7794-f9a4ae371d5d 104efdcf-a081-9f52-7794-f9a4ae371d5d}"
2025-01-28T14:55:38.762Z [INFO]  server-d4c1da89-f157-768b-7e09-66e59e01dfa5: aborting pipeline replication: peer="{Voter server-373cd188-51fc-85d5-97b3-492821fe0623 373cd188-51fc-85d5-97b3-492821fe0623}"
--- PASS: TestRaft_UserRestore (84.14s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.58s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.75s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.27s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.47s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.63s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.85s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.23s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.25s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.73s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.60s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.42s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.26s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.92s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.62s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.35s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.50s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.63s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.60s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.17s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.32s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.81s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.74s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.28s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.15s)
=== RUN   TestRaft_SendSnapshotFollower
2025-01-28T14:55:38.767Z [INFO]  server-dcc7b185-d18b-cc0c-f895-c87db52ff555: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dcc7b185-d18b-cc0c-f895-c87db52ff555 Address:dcc7b185-d18b-cc0c-f895-c87db52ff555} {Suffrage:Voter ID:server-3f84320c-7a18-ccee-4e2c-bb37f3c08d1e Address:3f84320c-7a18-ccee-4e2c-bb37f3c08d1e} {Suffrage:Voter ID:server-69650491-b359-5331-f391-71ddfce09298 Address:69650491-b359-5331-f391-71ddfce09298}]"
2025-01-28T14:55:38.767Z [INFO]  server-dcc7b185-d18b-cc0c-f895-c87db52ff555: entering follower state: follower="Node at dcc7b185-d18b-cc0c-f895-c87db52ff555 [Follower]" leader-address= leader-id=
2025-01-28T14:55:38.767Z [INFO]  server-3f84320c-7a18-ccee-4e2c-bb37f3c08d1e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dcc7b185-d18b-cc0c-f895-c87db52ff555 Address:dcc7b185-d18b-cc0c-f895-c87db52ff555} {Suffrage:Voter ID:server-3f84320c-7a18-ccee-4e2c-bb37f3c08d1e Address:3f84320c-7a18-ccee-4e2c-bb37f3c08d1e} {Suffrage:Voter ID:server-69650491-b359-5331-f391-71ddfce09298 Address:69650491-b359-5331-f391-71ddfce09298}]"
2025-01-28T14:55:38.767Z [INFO]  server-3f84320c-7a18-ccee-4e2c-bb37f3c08d1e: entering follower state: follower="Node at 3f84320c-7a18-ccee-4e2c-bb37f3c08d1e [Follower]" leader-address= leader-id=
2025-01-28T14:55:38.768Z [INFO]  server-69650491-b359-5331-f391-71ddfce09298: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dcc7b185-d18b-cc0c-f895-c87db52ff555 Address:dcc7b185-d18b-cc0c-f895-c87db52ff555} {Suffrage:Voter ID:server-3f84320c-7a18-ccee-4e2c-bb37f3c08d1e Address:3f84320c-7a18-ccee-4e2c-bb37f3c08d1e} {Suffrage:Voter ID:server-69650491-b359-5331-f391-71ddfce09298 Address:69650491-b359-5331-f391-71ddfce09298}]"
2025-01-28T14:55:38.768Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:38.768Z [INFO]  server-69650491-b359-5331-f391-71ddfce09298: entering follower state: follower="Node at 69650491-b359-5331-f391-71ddfce09298 [Follower]" leader-address= leader-id=
2025-01-28T14:55:38.845Z [WARN]  server-69650491-b359-5331-f391-71ddfce09298: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:38.845Z [INFO]  server-69650491-b359-5331-f391-71ddfce09298: entering candidate state: node="Node at 69650491-b359-5331-f391-71ddfce09298 [Candidate]" term=2
2025-01-28T14:55:38.846Z [INFO]  server-69650491-b359-5331-f391-71ddfce09298: election won: term=2 tally=2
2025-01-28T14:55:38.846Z [INFO]  server-69650491-b359-5331-f391-71ddfce09298: entering leader state: leader="Node at 69650491-b359-5331-f391-71ddfce09298 [Leader]"
2025-01-28T14:55:38.846Z [INFO]  server-69650491-b359-5331-f391-71ddfce09298: added peer, starting replication: peer=server-dcc7b185-d18b-cc0c-f895-c87db52ff555
2025-01-28T14:55:38.846Z [INFO]  server-69650491-b359-5331-f391-71ddfce09298: added peer, starting replication: peer=server-3f84320c-7a18-ccee-4e2c-bb37f3c08d1e
2025-01-28T14:55:38.846Z [INFO]  server-69650491-b359-5331-f391-71ddfce09298: pipelining replication: peer="{Voter server-dcc7b185-d18b-cc0c-f895-c87db52ff555 dcc7b185-d18b-cc0c-f895-c87db52ff555}"
2025-01-28T14:55:38.846Z [INFO]  server-69650491-b359-5331-f391-71ddfce09298: pipelining replication: peer="{Voter server-3f84320c-7a18-ccee-4e2c-bb37f3c08d1e 3f84320c-7a18-ccee-4e2c-bb37f3c08d1e}"
2025-01-28T14:55:38.952Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:38.846319029 +0000 UTC m=+124.520808989 (2 nodes), 78.221181ms from start of poll, 79.241544ms from cluster start. Timeout at 2025-01-28 14:55:38.951328188 +0000 UTC m=+124.625818168, 105.009179ms after stability
2025-01-28T14:55:38.952Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:39.057Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:38.952176623 +0000 UTC m=+124.626666603 (1 nodes), 2.34µs from start of poll, 185.099158ms from cluster start. Timeout at 2025-01-28 14:55:39.057329529 +0000 UTC m=+124.731819649, 105.153046ms after stability
2025-01-28T14:55:39.058Z [ERROR] server-69650491-b359-5331-f391-71ddfce09298: failed to pipeline appendEntries: peer="{Voter server-dcc7b185-d18b-cc0c-f895-c87db52ff555 dcc7b185-d18b-cc0c-f895-c87db52ff555}" error="append pipeline closed"
2025-01-28T14:55:39.058Z [INFO]  server-69650491-b359-5331-f391-71ddfce09298: aborting pipeline replication: peer="{Voter server-dcc7b185-d18b-cc0c-f895-c87db52ff555 dcc7b185-d18b-cc0c-f895-c87db52ff555}"
2025-01-28T14:55:39.058Z [ERROR] server-69650491-b359-5331-f391-71ddfce09298: failed to appendEntries to: peer="{Voter server-dcc7b185-d18b-cc0c-f895-c87db52ff555 dcc7b185-d18b-cc0c-f895-c87db52ff555}" error="failed to connect to peer: dcc7b185-d18b-cc0c-f895-c87db52ff555"
2025-01-28T14:55:39.061Z [ERROR] server-69650491-b359-5331-f391-71ddfce09298: failed to heartbeat to: peer=dcc7b185-d18b-cc0c-f895-c87db52ff555 backoff time=10ms error="failed to connect to peer: dcc7b185-d18b-cc0c-f895-c87db52ff555"
    raft_test.go:1627: [INFO] Finished apply without behind follower
2025-01-28T14:55:39.066Z [ERROR] server-dcc7b185-d18b-cc0c-f895-c87db52ff555: failed to take snapshot: error="nothing new to snapshot"
2025-01-28T14:55:39.066Z [INFO]  server-3f84320c-7a18-ccee-4e2c-bb37f3c08d1e: starting snapshot up to: index=99
2025-01-28T14:55:39.066Z [INFO]  creating new snapshot: path=/tmp/raft2545984320/snapshots/2-99-1738076139066.tmp
2025-01-28T14:55:39.067Z [INFO]  server-3f84320c-7a18-ccee-4e2c-bb37f3c08d1e: compacting logs: from=1 to=92
2025-01-28T14:55:39.068Z [INFO]  server-3f84320c-7a18-ccee-4e2c-bb37f3c08d1e: snapshot complete up to: index=99
2025-01-28T14:55:39.068Z [INFO]  server-69650491-b359-5331-f391-71ddfce09298: starting snapshot up to: index=102
2025-01-28T14:55:39.068Z [INFO]  creating new snapshot: path=/tmp/raft642241934/snapshots/2-102-1738076139068.tmp
2025-01-28T14:55:39.068Z [ERROR] server-69650491-b359-5331-f391-71ddfce09298: failed to appendEntries to: peer="{Voter server-dcc7b185-d18b-cc0c-f895-c87db52ff555 dcc7b185-d18b-cc0c-f895-c87db52ff555}" error="failed to connect to peer: dcc7b185-d18b-cc0c-f895-c87db52ff555"
2025-01-28T14:55:39.069Z [INFO]  server-69650491-b359-5331-f391-71ddfce09298: compacting logs: from=1 to=92
2025-01-28T14:55:39.069Z [INFO]  server-69650491-b359-5331-f391-71ddfce09298: snapshot complete up to: index=102
2025-01-28T14:55:39.079Z [ERROR] server-69650491-b359-5331-f391-71ddfce09298: failed to get log: index=2 error="log not found"
2025-01-28T14:55:39.080Z [INFO]  creating new snapshot: path=/tmp/raft547618876/snapshots/2-102-1738076139080.tmp
2025-01-28T14:55:39.081Z [INFO]  server-dcc7b185-d18b-cc0c-f895-c87db52ff555: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-01-28T14:55:39.081Z [INFO]  server-dcc7b185-d18b-cc0c-f895-c87db52ff555: copied to local snapshot: bytes=693
2025-01-28T14:55:39.082Z [INFO]  server-dcc7b185-d18b-cc0c-f895-c87db52ff555: snapshot restore progress: id=2-102-1738076139080 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-01-28T14:55:39.082Z [INFO]  server-dcc7b185-d18b-cc0c-f895-c87db52ff555: Installed remote snapshot
2025-01-28T14:55:39.082Z [INFO]  server-69650491-b359-5331-f391-71ddfce09298: aborting pipeline replication: peer="{Voter server-3f84320c-7a18-ccee-4e2c-bb37f3c08d1e 3f84320c-7a18-ccee-4e2c-bb37f3c08d1e}"
--- PASS: TestRaft_SendSnapshotFollower (0.32s)
=== RUN   TestRaft_SendSnapshotAndLogsFollower
2025-01-28T14:55:39.086Z [INFO]  server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b Address:dd8e445e-8394-7ca9-3e23-a84b340ccc8b} {Suffrage:Voter ID:server-c604e684-364d-3dc5-bdcd-f79bc77965e1 Address:c604e684-364d-3dc5-bdcd-f79bc77965e1} {Suffrage:Voter ID:server-44ceefb6-23a5-1f80-f094-962dd4830e58 Address:44ceefb6-23a5-1f80-f094-962dd4830e58}]"
2025-01-28T14:55:39.086Z [INFO]  server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: entering follower state: follower="Node at dd8e445e-8394-7ca9-3e23-a84b340ccc8b [Follower]" leader-address= leader-id=
2025-01-28T14:55:39.087Z [INFO]  server-c604e684-364d-3dc5-bdcd-f79bc77965e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b Address:dd8e445e-8394-7ca9-3e23-a84b340ccc8b} {Suffrage:Voter ID:server-c604e684-364d-3dc5-bdcd-f79bc77965e1 Address:c604e684-364d-3dc5-bdcd-f79bc77965e1} {Suffrage:Voter ID:server-44ceefb6-23a5-1f80-f094-962dd4830e58 Address:44ceefb6-23a5-1f80-f094-962dd4830e58}]"
2025-01-28T14:55:39.087Z [INFO]  server-c604e684-364d-3dc5-bdcd-f79bc77965e1: entering follower state: follower="Node at c604e684-364d-3dc5-bdcd-f79bc77965e1 [Follower]" leader-address= leader-id=
2025-01-28T14:55:39.087Z [INFO]  server-44ceefb6-23a5-1f80-f094-962dd4830e58: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b Address:dd8e445e-8394-7ca9-3e23-a84b340ccc8b} {Suffrage:Voter ID:server-c604e684-364d-3dc5-bdcd-f79bc77965e1 Address:c604e684-364d-3dc5-bdcd-f79bc77965e1} {Suffrage:Voter ID:server-44ceefb6-23a5-1f80-f094-962dd4830e58 Address:44ceefb6-23a5-1f80-f094-962dd4830e58}]"
2025-01-28T14:55:39.087Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:39.087Z [INFO]  server-44ceefb6-23a5-1f80-f094-962dd4830e58: entering follower state: follower="Node at 44ceefb6-23a5-1f80-f094-962dd4830e58 [Follower]" leader-address= leader-id=
2025-01-28T14:55:39.164Z [WARN]  server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:39.164Z [INFO]  server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: entering candidate state: node="Node at dd8e445e-8394-7ca9-3e23-a84b340ccc8b [Candidate]" term=2
2025-01-28T14:55:39.164Z [INFO]  server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: election won: term=2 tally=2
2025-01-28T14:55:39.164Z [INFO]  server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: entering leader state: leader="Node at dd8e445e-8394-7ca9-3e23-a84b340ccc8b [Leader]"
2025-01-28T14:55:39.164Z [INFO]  server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: added peer, starting replication: peer=server-c604e684-364d-3dc5-bdcd-f79bc77965e1
2025-01-28T14:55:39.164Z [INFO]  server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: added peer, starting replication: peer=server-44ceefb6-23a5-1f80-f094-962dd4830e58
2025-01-28T14:55:39.164Z [INFO]  server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: pipelining replication: peer="{Voter server-c604e684-364d-3dc5-bdcd-f79bc77965e1 c604e684-364d-3dc5-bdcd-f79bc77965e1}"
2025-01-28T14:55:39.165Z [INFO]  server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: pipelining replication: peer="{Voter server-44ceefb6-23a5-1f80-f094-962dd4830e58 44ceefb6-23a5-1f80-f094-962dd4830e58}"
2025-01-28T14:55:39.269Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:39.164795626 +0000 UTC m=+124.839285586 (2 nodes), 77.360783ms from start of poll, 78.23724ms from cluster start. Timeout at 2025-01-28 14:55:39.269800466 +0000 UTC m=+124.944290466, 105.00488ms after stability
2025-01-28T14:55:39.269Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:39.375Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:39.269999641 +0000 UTC m=+124.944489621 (1 nodes), 2.06µs from start of poll, 183.441275ms from cluster start. Timeout at 2025-01-28 14:55:39.375181344 +0000 UTC m=+125.049671304, 105.181683ms after stability
2025-01-28T14:55:39.375Z [ERROR] server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: failed to pipeline appendEntries: peer="{Voter server-c604e684-364d-3dc5-bdcd-f79bc77965e1 c604e684-364d-3dc5-bdcd-f79bc77965e1}" error="append pipeline closed"
2025-01-28T14:55:39.375Z [INFO]  server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: aborting pipeline replication: peer="{Voter server-c604e684-364d-3dc5-bdcd-f79bc77965e1 c604e684-364d-3dc5-bdcd-f79bc77965e1}"
2025-01-28T14:55:39.375Z [ERROR] server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: failed to appendEntries to: peer="{Voter server-c604e684-364d-3dc5-bdcd-f79bc77965e1 c604e684-364d-3dc5-bdcd-f79bc77965e1}" error="failed to connect to peer: c604e684-364d-3dc5-bdcd-f79bc77965e1"
2025-01-28T14:55:39.385Z [ERROR] server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: failed to heartbeat to: peer=c604e684-364d-3dc5-bdcd-f79bc77965e1 backoff time=10ms error="failed to connect to peer: c604e684-364d-3dc5-bdcd-f79bc77965e1"
2025-01-28T14:55:39.385Z [ERROR] server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: failed to appendEntries to: peer="{Voter server-c604e684-364d-3dc5-bdcd-f79bc77965e1 c604e684-364d-3dc5-bdcd-f79bc77965e1}" error="failed to connect to peer: c604e684-364d-3dc5-bdcd-f79bc77965e1"
    raft_test.go:1669: [INFO] Finished apply without behind follower
2025-01-28T14:55:39.387Z [INFO]  server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: starting snapshot up to: index=102
2025-01-28T14:55:39.387Z [INFO]  creating new snapshot: path=/tmp/raft342791242/snapshots/2-102-1738076139387.tmp
2025-01-28T14:55:39.389Z [INFO]  server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: compacting logs: from=1 to=92
2025-01-28T14:55:39.389Z [INFO]  server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: snapshot complete up to: index=102
2025-01-28T14:55:39.389Z [ERROR] server-c604e684-364d-3dc5-bdcd-f79bc77965e1: failed to take snapshot: error="nothing new to snapshot"
2025-01-28T14:55:39.389Z [INFO]  server-44ceefb6-23a5-1f80-f094-962dd4830e58: starting snapshot up to: index=98
2025-01-28T14:55:39.390Z [INFO]  creating new snapshot: path=/tmp/raft3205141601/snapshots/2-98-1738076139389.tmp
2025-01-28T14:55:39.391Z [INFO]  server-44ceefb6-23a5-1f80-f094-962dd4830e58: compacting logs: from=1 to=92
2025-01-28T14:55:39.391Z [INFO]  server-44ceefb6-23a5-1f80-f094-962dd4830e58: snapshot complete up to: index=98
2025-01-28T14:55:39.396Z [ERROR] server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: failed to get log: index=2 error="log not found"
2025-01-28T14:55:39.396Z [ERROR] server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: failed to install snapshot: id=2-102-1738076139387 error="failed to connect to peer: c604e684-364d-3dc5-bdcd-f79bc77965e1"
2025-01-28T14:55:39.396Z [ERROR] server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: failed to send snapshot to: peer="{Voter server-c604e684-364d-3dc5-bdcd-f79bc77965e1 c604e684-364d-3dc5-bdcd-f79bc77965e1}" error="failed to connect to peer: c604e684-364d-3dc5-bdcd-f79bc77965e1"
    raft_test.go:1690: [INFO] Finished apply without behind follower
2025-01-28T14:55:39.417Z [ERROR] server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: failed to get log: index=2 error="log not found"
2025-01-28T14:55:39.418Z [INFO]  creating new snapshot: path=/tmp/raft2693038633/snapshots/2-102-1738076139418.tmp
2025-01-28T14:55:39.419Z [INFO]  server-c604e684-364d-3dc5-bdcd-f79bc77965e1: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2025-01-28T14:55:39.419Z [INFO]  server-c604e684-364d-3dc5-bdcd-f79bc77965e1: copied to local snapshot: bytes=693
2025-01-28T14:55:39.420Z [INFO]  server-c604e684-364d-3dc5-bdcd-f79bc77965e1: snapshot restore progress: id=2-102-1738076139418 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2025-01-28T14:55:39.420Z [INFO]  server-c604e684-364d-3dc5-bdcd-f79bc77965e1: Installed remote snapshot
2025-01-28T14:55:39.421Z [INFO]  server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: pipelining replication: peer="{Voter server-c604e684-364d-3dc5-bdcd-f79bc77965e1 c604e684-364d-3dc5-bdcd-f79bc77965e1}"
2025-01-28T14:55:39.426Z [INFO]  server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: aborting pipeline replication: peer="{Voter server-44ceefb6-23a5-1f80-f094-962dd4830e58 44ceefb6-23a5-1f80-f094-962dd4830e58}"
2025-01-28T14:55:39.426Z [INFO]  server-dd8e445e-8394-7ca9-3e23-a84b340ccc8b: aborting pipeline replication: peer="{Voter server-c604e684-364d-3dc5-bdcd-f79bc77965e1 c604e684-364d-3dc5-bdcd-f79bc77965e1}"
--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.34s)
=== RUN   TestRaft_ReJoinFollower
2025-01-28T14:55:39.430Z [INFO]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf Address:f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf} {Suffrage:Voter ID:server-33fc2ea1-4723-c033-f027-777b733dea56 Address:33fc2ea1-4723-c033-f027-777b733dea56} {Suffrage:Voter ID:server-013a953e-cba9-58ed-a479-fe8165deb59f Address:013a953e-cba9-58ed-a479-fe8165deb59f}]"
2025-01-28T14:55:39.430Z [INFO]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: entering follower state: follower="Node at f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf [Follower]" leader-address= leader-id=
2025-01-28T14:55:39.430Z [INFO]  server-33fc2ea1-4723-c033-f027-777b733dea56: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf Address:f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf} {Suffrage:Voter ID:server-33fc2ea1-4723-c033-f027-777b733dea56 Address:33fc2ea1-4723-c033-f027-777b733dea56} {Suffrage:Voter ID:server-013a953e-cba9-58ed-a479-fe8165deb59f Address:013a953e-cba9-58ed-a479-fe8165deb59f}]"
2025-01-28T14:55:39.430Z [INFO]  server-33fc2ea1-4723-c033-f027-777b733dea56: entering follower state: follower="Node at 33fc2ea1-4723-c033-f027-777b733dea56 [Follower]" leader-address= leader-id=
2025-01-28T14:55:39.431Z [INFO]  server-013a953e-cba9-58ed-a479-fe8165deb59f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf Address:f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf} {Suffrage:Voter ID:server-33fc2ea1-4723-c033-f027-777b733dea56 Address:33fc2ea1-4723-c033-f027-777b733dea56} {Suffrage:Voter ID:server-013a953e-cba9-58ed-a479-fe8165deb59f Address:013a953e-cba9-58ed-a479-fe8165deb59f}]"
2025-01-28T14:55:39.431Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:39.431Z [INFO]  server-013a953e-cba9-58ed-a479-fe8165deb59f: entering follower state: follower="Node at 013a953e-cba9-58ed-a479-fe8165deb59f [Follower]" leader-address= leader-id=
2025-01-28T14:55:39.481Z [WARN]  server-013a953e-cba9-58ed-a479-fe8165deb59f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:39.481Z [WARN]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:39.481Z [INFO]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: entering candidate state: node="Node at f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf [Candidate]" term=2
2025-01-28T14:55:39.481Z [INFO]  server-013a953e-cba9-58ed-a479-fe8165deb59f: entering candidate state: node="Node at 013a953e-cba9-58ed-a479-fe8165deb59f [Candidate]" term=2
2025-01-28T14:55:39.481Z [INFO]  server-013a953e-cba9-58ed-a479-fe8165deb59f: duplicate requestVote for same term: term=2
2025-01-28T14:55:39.481Z [INFO]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: election won: term=2 tally=2
2025-01-28T14:55:39.482Z [INFO]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: entering leader state: leader="Node at f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf [Leader]"
2025-01-28T14:55:39.482Z [INFO]  server-33fc2ea1-4723-c033-f027-777b733dea56: duplicate requestVote for same term: term=2
2025-01-28T14:55:39.482Z [INFO]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: added peer, starting replication: peer=server-33fc2ea1-4723-c033-f027-777b733dea56
2025-01-28T14:55:39.482Z [INFO]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: added peer, starting replication: peer=server-013a953e-cba9-58ed-a479-fe8165deb59f
2025-01-28T14:55:39.482Z [WARN]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: rejecting vote request since we have a leader: from=013a953e-cba9-58ed-a479-fe8165deb59f leader=f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf leader-id=server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf
2025-01-28T14:55:39.482Z [INFO]  server-013a953e-cba9-58ed-a479-fe8165deb59f: entering follower state: follower="Node at 013a953e-cba9-58ed-a479-fe8165deb59f [Follower]" leader-address=f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf leader-id=server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf
2025-01-28T14:55:39.482Z [INFO]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: pipelining replication: peer="{Voter server-33fc2ea1-4723-c033-f027-777b733dea56 33fc2ea1-4723-c033-f027-777b733dea56}"
2025-01-28T14:55:39.482Z [INFO]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: pipelining replication: peer="{Voter server-013a953e-cba9-58ed-a479-fe8165deb59f 013a953e-cba9-58ed-a479-fe8165deb59f}"
2025-01-28T14:55:39.588Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:39.482318912 +0000 UTC m=+125.156808872 (1 nodes), 50.983191ms from start of poll, 51.998395ms from cluster start. Timeout at 2025-01-28 14:55:39.587330271 +0000 UTC m=+125.261820291, 105.011419ms after stability
2025-01-28T14:55:39.594Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:39.700Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:39.594188918 +0000 UTC m=+125.268678898 (2 nodes), 2.399µs from start of poll, 163.868421ms from cluster start. Timeout at 2025-01-28 14:55:39.699358282 +0000 UTC m=+125.373848262, 105.169364ms after stability
2025-01-28T14:55:39.700Z [INFO]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: updating configuration: command=RemoveServer server-id=server-33fc2ea1-4723-c033-f027-777b733dea56 server-addr= servers="[{Suffrage:Voter ID:server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf Address:f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf} {Suffrage:Voter ID:server-013a953e-cba9-58ed-a479-fe8165deb59f Address:013a953e-cba9-58ed-a479-fe8165deb59f}]"
2025-01-28T14:55:39.700Z [INFO]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: removed peer, stopping replication: peer=server-33fc2ea1-4723-c033-f027-777b733dea56 last-index=3
2025-01-28T14:55:39.701Z [INFO]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: aborting pipeline replication: peer="{Voter server-33fc2ea1-4723-c033-f027-777b733dea56 33fc2ea1-4723-c033-f027-777b733dea56}"
2025-01-28T14:55:39.804Z [WARN]  server-33fc2ea1-4723-c033-f027-777b733dea56: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-01-28T14:55:39.806Z [INFO]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: updating configuration: command=AddVoter server-id=server-33fc2ea1-4723-c033-f027-777b733dea56 server-addr=33fc2ea1-4723-c033-f027-777b733dea56 servers="[{Suffrage:Voter ID:server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf Address:f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf} {Suffrage:Voter ID:server-013a953e-cba9-58ed-a479-fe8165deb59f Address:013a953e-cba9-58ed-a479-fe8165deb59f} {Suffrage:Voter ID:server-33fc2ea1-4723-c033-f027-777b733dea56 Address:33fc2ea1-4723-c033-f027-777b733dea56}]"
2025-01-28T14:55:39.806Z [INFO]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: added peer, starting replication: peer=server-33fc2ea1-4723-c033-f027-777b733dea56
2025-01-28T14:55:39.806Z [WARN]  server-33fc2ea1-4723-c033-f027-777b733dea56: failed to get previous log: previous-index=4 last-index=3 error="log not found"
2025-01-28T14:55:39.806Z [WARN]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: appendEntries rejected, sending older logs: peer="{Voter server-33fc2ea1-4723-c033-f027-777b733dea56 33fc2ea1-4723-c033-f027-777b733dea56}" next=4
2025-01-28T14:55:39.807Z [INFO]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: pipelining replication: peer="{Voter server-33fc2ea1-4723-c033-f027-777b733dea56 33fc2ea1-4723-c033-f027-777b733dea56}"
2025-01-28T14:55:39.807Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:39.913Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:39.807374507 +0000 UTC m=+125.481864487 (1 nodes), 1.62µs from start of poll, 377.05401ms from cluster start. Timeout at 2025-01-28 14:55:39.912561389 +0000 UTC m=+125.587051469, 105.186982ms after stability
2025-01-28T14:55:39.913Z [INFO]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: aborting pipeline replication: peer="{Voter server-013a953e-cba9-58ed-a479-fe8165deb59f 013a953e-cba9-58ed-a479-fe8165deb59f}"
2025-01-28T14:55:39.913Z [INFO]  server-f0ff7cb0-b6bb-25bf-50da-cb3fec9201bf: aborting pipeline replication: peer="{Voter server-33fc2ea1-4723-c033-f027-777b733dea56 33fc2ea1-4723-c033-f027-777b733dea56}"
--- PASS: TestRaft_ReJoinFollower (0.49s)
=== RUN   TestRaft_LeaderLeaseExpire
2025-01-28T14:55:39.919Z [INFO]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd Address:f6d2dff6-b296-cafa-6dc0-7072a484f0fd} {Suffrage:Voter ID:server-5a3c7731-9dba-e855-7774-7b5c03278f2b Address:5a3c7731-9dba-e855-7774-7b5c03278f2b}]"
2025-01-28T14:55:39.921Z [INFO]  server-5a3c7731-9dba-e855-7774-7b5c03278f2b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd Address:f6d2dff6-b296-cafa-6dc0-7072a484f0fd} {Suffrage:Voter ID:server-5a3c7731-9dba-e855-7774-7b5c03278f2b Address:5a3c7731-9dba-e855-7774-7b5c03278f2b}]"
2025-01-28T14:55:39.921Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:39.922Z [INFO]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: entering follower state: follower="Node at f6d2dff6-b296-cafa-6dc0-7072a484f0fd [Follower]" leader-address= leader-id=
2025-01-28T14:55:39.923Z [INFO]  server-5a3c7731-9dba-e855-7774-7b5c03278f2b: entering follower state: follower="Node at 5a3c7731-9dba-e855-7774-7b5c03278f2b [Follower]" leader-address= leader-id=
2025-01-28T14:55:39.976Z [WARN]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:39.976Z [INFO]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: entering candidate state: node="Node at f6d2dff6-b296-cafa-6dc0-7072a484f0fd [Candidate]" term=2
2025-01-28T14:55:39.977Z [INFO]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: election won: term=2 tally=2
2025-01-28T14:55:39.977Z [INFO]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: entering leader state: leader="Node at f6d2dff6-b296-cafa-6dc0-7072a484f0fd [Leader]"
2025-01-28T14:55:39.977Z [INFO]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: added peer, starting replication: peer=server-5a3c7731-9dba-e855-7774-7b5c03278f2b
2025-01-28T14:55:39.977Z [INFO]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: pipelining replication: peer="{Voter server-5a3c7731-9dba-e855-7774-7b5c03278f2b 5a3c7731-9dba-e855-7774-7b5c03278f2b}"
2025-01-28T14:55:40.083Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:39.977129993 +0000 UTC m=+125.651619973 (1 nodes), 55.900306ms from start of poll, 57.731514ms from cluster start. Timeout at 2025-01-28 14:55:40.082132833 +0000 UTC m=+125.756622833, 105.00286ms after stability
2025-01-28T14:55:40.088Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:40.194Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:40.089042856 +0000 UTC m=+125.763532836 (1 nodes), 1.74µs from start of poll, 169.644377ms from cluster start. Timeout at 2025-01-28 14:55:40.194183063 +0000 UTC m=+125.868673123, 105.140287ms after stability
    raft_test.go:1799: [INFO] Disconnecting Node at 5a3c7731-9dba-e855-7774-7b5c03278f2b [Follower]
2025-01-28T14:55:40.194Z [ERROR] server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: failed to pipeline appendEntries: peer="{Voter server-5a3c7731-9dba-e855-7774-7b5c03278f2b 5a3c7731-9dba-e855-7774-7b5c03278f2b}" error="append pipeline closed"
2025-01-28T14:55:40.194Z [INFO]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: aborting pipeline replication: peer="{Voter server-5a3c7731-9dba-e855-7774-7b5c03278f2b 5a3c7731-9dba-e855-7774-7b5c03278f2b}"
2025-01-28T14:55:40.196Z [ERROR] server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: failed to heartbeat to: peer=5a3c7731-9dba-e855-7774-7b5c03278f2b backoff time=10ms error="failed to connect to peer: 5a3c7731-9dba-e855-7774-7b5c03278f2b"
2025-01-28T14:55:40.204Z [ERROR] server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: failed to appendEntries to: peer="{Voter server-5a3c7731-9dba-e855-7774-7b5c03278f2b 5a3c7731-9dba-e855-7774-7b5c03278f2b}" error="failed to connect to peer: 5a3c7731-9dba-e855-7774-7b5c03278f2b"
2025-01-28T14:55:40.216Z [ERROR] server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: failed to heartbeat to: peer=5a3c7731-9dba-e855-7774-7b5c03278f2b backoff time=10ms error="failed to connect to peer: 5a3c7731-9dba-e855-7774-7b5c03278f2b"
2025-01-28T14:55:40.223Z [ERROR] server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: failed to appendEntries to: peer="{Voter server-5a3c7731-9dba-e855-7774-7b5c03278f2b 5a3c7731-9dba-e855-7774-7b5c03278f2b}" error="failed to connect to peer: 5a3c7731-9dba-e855-7774-7b5c03278f2b"
2025-01-28T14:55:40.236Z [ERROR] server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: failed to heartbeat to: peer=5a3c7731-9dba-e855-7774-7b5c03278f2b backoff time=10ms error="failed to connect to peer: 5a3c7731-9dba-e855-7774-7b5c03278f2b"
2025-01-28T14:55:40.239Z [WARN]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: failed to contact: server-id=server-5a3c7731-9dba-e855-7774-7b5c03278f2b time=50.212463ms
2025-01-28T14:55:40.239Z [WARN]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: failed to contact quorum of nodes, stepping down
2025-01-28T14:55:40.239Z [INFO]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: entering follower state: follower="Node at f6d2dff6-b296-cafa-6dc0-7072a484f0fd [Follower]" leader-address= leader-id=
2025-01-28T14:55:40.239Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:40.243Z [ERROR] server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: failed to appendEntries to: peer="{Voter server-5a3c7731-9dba-e855-7774-7b5c03278f2b 5a3c7731-9dba-e855-7774-7b5c03278f2b}" error="failed to connect to peer: 5a3c7731-9dba-e855-7774-7b5c03278f2b"
2025-01-28T14:55:40.288Z [WARN]  server-5a3c7731-9dba-e855-7774-7b5c03278f2b: heartbeat timeout reached, starting election: last-leader-addr=f6d2dff6-b296-cafa-6dc0-7072a484f0fd last-leader-id=server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd
2025-01-28T14:55:40.288Z [INFO]  server-5a3c7731-9dba-e855-7774-7b5c03278f2b: entering candidate state: node="Node at 5a3c7731-9dba-e855-7774-7b5c03278f2b [Candidate]" term=3
2025-01-28T14:55:40.288Z [ERROR] server-5a3c7731-9dba-e855-7774-7b5c03278f2b: failed to make requestVote RPC: target="{Voter server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd f6d2dff6-b296-cafa-6dc0-7072a484f0fd}" error="failed to connect to peer: f6d2dff6-b296-cafa-6dc0-7072a484f0fd" term=3
2025-01-28T14:55:40.333Z [WARN]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:40.333Z [INFO]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: entering candidate state: node="Node at f6d2dff6-b296-cafa-6dc0-7072a484f0fd [Candidate]" term=3
2025-01-28T14:55:40.333Z [ERROR] server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: failed to make requestVote RPC: target="{Voter server-5a3c7731-9dba-e855-7774-7b5c03278f2b 5a3c7731-9dba-e855-7774-7b5c03278f2b}" error="failed to connect to peer: 5a3c7731-9dba-e855-7774-7b5c03278f2b" term=3
2025-01-28T14:55:40.339Z [WARN]  server-5a3c7731-9dba-e855-7774-7b5c03278f2b: Election timeout reached, restarting election
2025-01-28T14:55:40.339Z [INFO]  server-5a3c7731-9dba-e855-7774-7b5c03278f2b: entering candidate state: node="Node at 5a3c7731-9dba-e855-7774-7b5c03278f2b [Candidate]" term=4
2025-01-28T14:55:40.339Z [ERROR] server-5a3c7731-9dba-e855-7774-7b5c03278f2b: failed to make requestVote RPC: target="{Voter server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd f6d2dff6-b296-cafa-6dc0-7072a484f0fd}" error="failed to connect to peer: f6d2dff6-b296-cafa-6dc0-7072a484f0fd" term=4
2025-01-28T14:55:40.414Z [WARN]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: Election timeout reached, restarting election
2025-01-28T14:55:40.415Z [INFO]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: entering candidate state: node="Node at f6d2dff6-b296-cafa-6dc0-7072a484f0fd [Candidate]" term=4
2025-01-28T14:55:40.415Z [ERROR] server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: failed to make requestVote RPC: target="{Voter server-5a3c7731-9dba-e855-7774-7b5c03278f2b 5a3c7731-9dba-e855-7774-7b5c03278f2b}" error="failed to connect to peer: 5a3c7731-9dba-e855-7774-7b5c03278f2b" term=4
2025-01-28T14:55:40.436Z [WARN]  server-5a3c7731-9dba-e855-7774-7b5c03278f2b: Election timeout reached, restarting election
2025-01-28T14:55:40.436Z [INFO]  server-5a3c7731-9dba-e855-7774-7b5c03278f2b: entering candidate state: node="Node at 5a3c7731-9dba-e855-7774-7b5c03278f2b [Candidate]" term=5
2025-01-28T14:55:40.436Z [ERROR] server-5a3c7731-9dba-e855-7774-7b5c03278f2b: failed to make requestVote RPC: target="{Voter server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd f6d2dff6-b296-cafa-6dc0-7072a484f0fd}" error="failed to connect to peer: f6d2dff6-b296-cafa-6dc0-7072a484f0fd" term=5
2025-01-28T14:55:40.439Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:40.333846782 +0000 UTC m=+126.008336762 (0 nodes), 94.089704ms from start of poll, 414.448303ms from cluster start. Timeout at 2025-01-28 14:55:40.43886818 +0000 UTC m=+126.113358200, 105.021438ms after stability
2025-01-28T14:55:40.504Z [WARN]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: Election timeout reached, restarting election
2025-01-28T14:55:40.504Z [INFO]  server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: entering candidate state: node="Node at f6d2dff6-b296-cafa-6dc0-7072a484f0fd [Candidate]" term=5
2025-01-28T14:55:40.504Z [ERROR] server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd: failed to make requestVote RPC: target="{Voter server-5a3c7731-9dba-e855-7774-7b5c03278f2b 5a3c7731-9dba-e855-7774-7b5c03278f2b}" error="failed to connect to peer: 5a3c7731-9dba-e855-7774-7b5c03278f2b" term=5
2025-01-28T14:55:40.527Z [WARN]  server-5a3c7731-9dba-e855-7774-7b5c03278f2b: Election timeout reached, restarting election
2025-01-28T14:55:40.528Z [INFO]  server-5a3c7731-9dba-e855-7774-7b5c03278f2b: entering candidate state: node="Node at 5a3c7731-9dba-e855-7774-7b5c03278f2b [Candidate]" term=6
2025-01-28T14:55:40.528Z [ERROR] server-5a3c7731-9dba-e855-7774-7b5c03278f2b: failed to make requestVote RPC: target="{Voter server-f6d2dff6-b296-cafa-6dc0-7072a484f0fd f6d2dff6-b296-cafa-6dc0-7072a484f0fd}" error="failed to connect to peer: f6d2dff6-b296-cafa-6dc0-7072a484f0fd" term=6
--- PASS: TestRaft_LeaderLeaseExpire (0.63s)
=== RUN   TestRaft_Barrier
2025-01-28T14:55:40.549Z [INFO]  server-d768e263-4d16-2bb6-c0b7-ac3a6f49bd24: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d768e263-4d16-2bb6-c0b7-ac3a6f49bd24 Address:d768e263-4d16-2bb6-c0b7-ac3a6f49bd24} {Suffrage:Voter ID:server-9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28 Address:9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28} {Suffrage:Voter ID:server-40d862e6-1667-eb8c-e8e4-dede8fbdc63f Address:40d862e6-1667-eb8c-e8e4-dede8fbdc63f}]"
2025-01-28T14:55:40.549Z [INFO]  server-9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d768e263-4d16-2bb6-c0b7-ac3a6f49bd24 Address:d768e263-4d16-2bb6-c0b7-ac3a6f49bd24} {Suffrage:Voter ID:server-9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28 Address:9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28} {Suffrage:Voter ID:server-40d862e6-1667-eb8c-e8e4-dede8fbdc63f Address:40d862e6-1667-eb8c-e8e4-dede8fbdc63f}]"
2025-01-28T14:55:40.549Z [INFO]  server-d768e263-4d16-2bb6-c0b7-ac3a6f49bd24: entering follower state: follower="Node at d768e263-4d16-2bb6-c0b7-ac3a6f49bd24 [Follower]" leader-address= leader-id=
2025-01-28T14:55:40.549Z [INFO]  server-9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28: entering follower state: follower="Node at 9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28 [Follower]" leader-address= leader-id=
2025-01-28T14:55:40.549Z [INFO]  server-40d862e6-1667-eb8c-e8e4-dede8fbdc63f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d768e263-4d16-2bb6-c0b7-ac3a6f49bd24 Address:d768e263-4d16-2bb6-c0b7-ac3a6f49bd24} {Suffrage:Voter ID:server-9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28 Address:9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28} {Suffrage:Voter ID:server-40d862e6-1667-eb8c-e8e4-dede8fbdc63f Address:40d862e6-1667-eb8c-e8e4-dede8fbdc63f}]"
2025-01-28T14:55:40.549Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:40.549Z [INFO]  server-40d862e6-1667-eb8c-e8e4-dede8fbdc63f: entering follower state: follower="Node at 40d862e6-1667-eb8c-e8e4-dede8fbdc63f [Follower]" leader-address= leader-id=
2025-01-28T14:55:40.608Z [WARN]  server-9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:40.608Z [INFO]  server-9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28: entering candidate state: node="Node at 9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28 [Candidate]" term=2
2025-01-28T14:55:40.608Z [INFO]  server-9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28: election won: term=2 tally=2
2025-01-28T14:55:40.608Z [INFO]  server-9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28: entering leader state: leader="Node at 9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28 [Leader]"
2025-01-28T14:55:40.608Z [INFO]  server-9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28: added peer, starting replication: peer=server-d768e263-4d16-2bb6-c0b7-ac3a6f49bd24
2025-01-28T14:55:40.608Z [INFO]  server-9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28: added peer, starting replication: peer=server-40d862e6-1667-eb8c-e8e4-dede8fbdc63f
2025-01-28T14:55:40.608Z [INFO]  server-9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28: pipelining replication: peer="{Voter server-d768e263-4d16-2bb6-c0b7-ac3a6f49bd24 d768e263-4d16-2bb6-c0b7-ac3a6f49bd24}"
2025-01-28T14:55:40.608Z [INFO]  server-9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28: pipelining replication: peer="{Voter server-40d862e6-1667-eb8c-e8e4-dede8fbdc63f 40d862e6-1667-eb8c-e8e4-dede8fbdc63f}"
2025-01-28T14:55:40.714Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:40.608504372 +0000 UTC m=+126.282994352 (1 nodes), 58.657826ms from start of poll, 59.67533ms from cluster start. Timeout at 2025-01-28 14:55:40.713508212 +0000 UTC m=+126.387998252, 105.0039ms after stability
2025-01-28T14:55:40.733Z [INFO]  server-9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28: aborting pipeline replication: peer="{Voter server-40d862e6-1667-eb8c-e8e4-dede8fbdc63f 40d862e6-1667-eb8c-e8e4-dede8fbdc63f}"
2025-01-28T14:55:40.733Z [INFO]  server-9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28: aborting pipeline replication: peer="{Voter server-d768e263-4d16-2bb6-c0b7-ac3a6f49bd24 d768e263-4d16-2bb6-c0b7-ac3a6f49bd24}"
2025-01-28T14:55:41.234Z [ERROR] server-9c1dadd7-1a54-09f6-2eb0-c6149f3c1c28: failed to heartbeat to: peer=d768e263-4d16-2bb6-c0b7-ac3a6f49bd24 backoff time=10ms error="command timed out"
--- PASS: TestRaft_Barrier (0.69s)
=== RUN   TestRaft_VerifyLeader
2025-01-28T14:55:41.238Z [INFO]  server-dfb5e5c1-6f86-2ccc-c6cc-431cbe8f29a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dfb5e5c1-6f86-2ccc-c6cc-431cbe8f29a8 Address:dfb5e5c1-6f86-2ccc-c6cc-431cbe8f29a8} {Suffrage:Voter ID:server-64c34394-dc1f-2562-d666-122ceb03db6c Address:64c34394-dc1f-2562-d666-122ceb03db6c} {Suffrage:Voter ID:server-c07244a4-770d-3caf-ddab-c27998ead838 Address:c07244a4-770d-3caf-ddab-c27998ead838}]"
2025-01-28T14:55:41.238Z [INFO]  server-dfb5e5c1-6f86-2ccc-c6cc-431cbe8f29a8: entering follower state: follower="Node at dfb5e5c1-6f86-2ccc-c6cc-431cbe8f29a8 [Follower]" leader-address= leader-id=
2025-01-28T14:55:41.239Z [INFO]  server-64c34394-dc1f-2562-d666-122ceb03db6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dfb5e5c1-6f86-2ccc-c6cc-431cbe8f29a8 Address:dfb5e5c1-6f86-2ccc-c6cc-431cbe8f29a8} {Suffrage:Voter ID:server-64c34394-dc1f-2562-d666-122ceb03db6c Address:64c34394-dc1f-2562-d666-122ceb03db6c} {Suffrage:Voter ID:server-c07244a4-770d-3caf-ddab-c27998ead838 Address:c07244a4-770d-3caf-ddab-c27998ead838}]"
2025-01-28T14:55:41.239Z [INFO]  server-64c34394-dc1f-2562-d666-122ceb03db6c: entering follower state: follower="Node at 64c34394-dc1f-2562-d666-122ceb03db6c [Follower]" leader-address= leader-id=
2025-01-28T14:55:41.239Z [INFO]  server-c07244a4-770d-3caf-ddab-c27998ead838: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dfb5e5c1-6f86-2ccc-c6cc-431cbe8f29a8 Address:dfb5e5c1-6f86-2ccc-c6cc-431cbe8f29a8} {Suffrage:Voter ID:server-64c34394-dc1f-2562-d666-122ceb03db6c Address:64c34394-dc1f-2562-d666-122ceb03db6c} {Suffrage:Voter ID:server-c07244a4-770d-3caf-ddab-c27998ead838 Address:c07244a4-770d-3caf-ddab-c27998ead838}]"
2025-01-28T14:55:41.239Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:41.239Z [INFO]  server-c07244a4-770d-3caf-ddab-c27998ead838: entering follower state: follower="Node at c07244a4-770d-3caf-ddab-c27998ead838 [Follower]" leader-address= leader-id=
2025-01-28T14:55:41.304Z [WARN]  server-c07244a4-770d-3caf-ddab-c27998ead838: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:41.304Z [INFO]  server-c07244a4-770d-3caf-ddab-c27998ead838: entering candidate state: node="Node at c07244a4-770d-3caf-ddab-c27998ead838 [Candidate]" term=2
2025-01-28T14:55:41.304Z [INFO]  server-c07244a4-770d-3caf-ddab-c27998ead838: election won: term=2 tally=2
2025-01-28T14:55:41.304Z [INFO]  server-c07244a4-770d-3caf-ddab-c27998ead838: entering leader state: leader="Node at c07244a4-770d-3caf-ddab-c27998ead838 [Leader]"
2025-01-28T14:55:41.304Z [INFO]  server-c07244a4-770d-3caf-ddab-c27998ead838: added peer, starting replication: peer=server-dfb5e5c1-6f86-2ccc-c6cc-431cbe8f29a8
2025-01-28T14:55:41.304Z [INFO]  server-c07244a4-770d-3caf-ddab-c27998ead838: added peer, starting replication: peer=server-64c34394-dc1f-2562-d666-122ceb03db6c
2025-01-28T14:55:41.304Z [INFO]  server-c07244a4-770d-3caf-ddab-c27998ead838: pipelining replication: peer="{Voter server-dfb5e5c1-6f86-2ccc-c6cc-431cbe8f29a8 dfb5e5c1-6f86-2ccc-c6cc-431cbe8f29a8}"
2025-01-28T14:55:41.304Z [INFO]  server-c07244a4-770d-3caf-ddab-c27998ead838: pipelining replication: peer="{Voter server-64c34394-dc1f-2562-d666-122ceb03db6c 64c34394-dc1f-2562-d666-122ceb03db6c}"
2025-01-28T14:55:41.410Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:41.304442 +0000 UTC m=+126.978931980 (1 nodes), 64.849062ms from start of poll, 66.098304ms from cluster start. Timeout at 2025-01-28 14:55:41.409458819 +0000 UTC m=+127.083948779, 105.016799ms after stability
2025-01-28T14:55:41.410Z [INFO]  server-c07244a4-770d-3caf-ddab-c27998ead838: aborting pipeline replication: peer="{Voter server-64c34394-dc1f-2562-d666-122ceb03db6c 64c34394-dc1f-2562-d666-122ceb03db6c}"
2025-01-28T14:55:41.410Z [INFO]  server-c07244a4-770d-3caf-ddab-c27998ead838: aborting pipeline replication: peer="{Voter server-dfb5e5c1-6f86-2ccc-c6cc-431cbe8f29a8 dfb5e5c1-6f86-2ccc-c6cc-431cbe8f29a8}"
--- PASS: TestRaft_VerifyLeader (0.18s)
=== RUN   TestRaft_VerifyLeader_Single
2025-01-28T14:55:41.413Z [INFO]  server-08ac384d-0d60-95fc-0e43-6d42ad6aad15: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-08ac384d-0d60-95fc-0e43-6d42ad6aad15 Address:08ac384d-0d60-95fc-0e43-6d42ad6aad15}]"
2025-01-28T14:55:41.413Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:41.414Z [INFO]  server-08ac384d-0d60-95fc-0e43-6d42ad6aad15: entering follower state: follower="Node at 08ac384d-0d60-95fc-0e43-6d42ad6aad15 [Follower]" leader-address= leader-id=
2025-01-28T14:55:41.478Z [WARN]  server-08ac384d-0d60-95fc-0e43-6d42ad6aad15: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:41.478Z [INFO]  server-08ac384d-0d60-95fc-0e43-6d42ad6aad15: entering candidate state: node="Node at 08ac384d-0d60-95fc-0e43-6d42ad6aad15 [Candidate]" term=2
2025-01-28T14:55:41.478Z [INFO]  server-08ac384d-0d60-95fc-0e43-6d42ad6aad15: election won: term=2 tally=1
2025-01-28T14:55:41.478Z [INFO]  server-08ac384d-0d60-95fc-0e43-6d42ad6aad15: entering leader state: leader="Node at 08ac384d-0d60-95fc-0e43-6d42ad6aad15 [Leader]"
2025-01-28T14:55:41.584Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:41.47881831 +0000 UTC m=+127.153308270 (1 nodes), 64.806946ms from start of poll, 65.237326ms from cluster start. Timeout at 2025-01-28 14:55:41.583822189 +0000 UTC m=+127.258312209, 105.003939ms after stability
--- PASS: TestRaft_VerifyLeader_Single (0.17s)
=== RUN   TestRaft_VerifyLeader_Fail
2025-01-28T14:55:41.587Z [INFO]  server-62c70e57-c45c-4eee-17fd-27d5920c3174: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62c70e57-c45c-4eee-17fd-27d5920c3174 Address:62c70e57-c45c-4eee-17fd-27d5920c3174} {Suffrage:Voter ID:server-3b9817b0-8e76-ff5f-db22-edc543bd75fa Address:3b9817b0-8e76-ff5f-db22-edc543bd75fa}]"
2025-01-28T14:55:41.587Z [INFO]  server-3b9817b0-8e76-ff5f-db22-edc543bd75fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-62c70e57-c45c-4eee-17fd-27d5920c3174 Address:62c70e57-c45c-4eee-17fd-27d5920c3174} {Suffrage:Voter ID:server-3b9817b0-8e76-ff5f-db22-edc543bd75fa Address:3b9817b0-8e76-ff5f-db22-edc543bd75fa}]"
2025-01-28T14:55:41.587Z [INFO]  server-62c70e57-c45c-4eee-17fd-27d5920c3174: entering follower state: follower="Node at 62c70e57-c45c-4eee-17fd-27d5920c3174 [Follower]" leader-address= leader-id=
2025-01-28T14:55:41.587Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:41.587Z [INFO]  server-3b9817b0-8e76-ff5f-db22-edc543bd75fa: entering follower state: follower="Node at 3b9817b0-8e76-ff5f-db22-edc543bd75fa [Follower]" leader-address= leader-id=
2025-01-28T14:55:41.670Z [WARN]  server-62c70e57-c45c-4eee-17fd-27d5920c3174: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:41.670Z [INFO]  server-62c70e57-c45c-4eee-17fd-27d5920c3174: entering candidate state: node="Node at 62c70e57-c45c-4eee-17fd-27d5920c3174 [Candidate]" term=2
2025-01-28T14:55:41.670Z [INFO]  server-62c70e57-c45c-4eee-17fd-27d5920c3174: election won: term=2 tally=2
2025-01-28T14:55:41.670Z [INFO]  server-62c70e57-c45c-4eee-17fd-27d5920c3174: entering leader state: leader="Node at 62c70e57-c45c-4eee-17fd-27d5920c3174 [Leader]"
2025-01-28T14:55:41.670Z [INFO]  server-62c70e57-c45c-4eee-17fd-27d5920c3174: added peer, starting replication: peer=server-3b9817b0-8e76-ff5f-db22-edc543bd75fa
2025-01-28T14:55:41.670Z [INFO]  server-62c70e57-c45c-4eee-17fd-27d5920c3174: pipelining replication: peer="{Voter server-3b9817b0-8e76-ff5f-db22-edc543bd75fa 3b9817b0-8e76-ff5f-db22-edc543bd75fa}"
2025-01-28T14:55:41.776Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:41.67068655 +0000 UTC m=+127.345176530 (1 nodes), 82.981992ms from start of poll, 83.699304ms from cluster start. Timeout at 2025-01-28 14:55:41.775690869 +0000 UTC m=+127.450180909, 105.004379ms after stability
2025-01-28T14:55:41.776Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:41.881Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:41.776399558 +0000 UTC m=+127.450889538 (1 nodes), 1.48µs from start of poll, 189.412312ms from cluster start. Timeout at 2025-01-28 14:55:41.881566782 +0000 UTC m=+127.556056742, 105.167204ms after stability
2025-01-28T14:55:41.883Z [ERROR] server-62c70e57-c45c-4eee-17fd-27d5920c3174: peer has newer term, stopping replication: peer="{Voter server-3b9817b0-8e76-ff5f-db22-edc543bd75fa 3b9817b0-8e76-ff5f-db22-edc543bd75fa}"
2025-01-28T14:55:41.883Z [INFO]  server-62c70e57-c45c-4eee-17fd-27d5920c3174: aborting pipeline replication: peer="{Voter server-3b9817b0-8e76-ff5f-db22-edc543bd75fa 3b9817b0-8e76-ff5f-db22-edc543bd75fa}"
2025-01-28T14:55:41.883Z [INFO]  server-62c70e57-c45c-4eee-17fd-27d5920c3174: entering follower state: follower="Node at 62c70e57-c45c-4eee-17fd-27d5920c3174 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VerifyLeader_Fail (0.30s)
=== RUN   TestRaft_VerifyLeader_PartialConnect
2025-01-28T14:55:41.886Z [INFO]  server-55b86cf6-6a23-3b03-15df-f5aefa95ce66: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-55b86cf6-6a23-3b03-15df-f5aefa95ce66 Address:55b86cf6-6a23-3b03-15df-f5aefa95ce66} {Suffrage:Voter ID:server-27655c63-2055-a7b1-0052-2fbdd5ead241 Address:27655c63-2055-a7b1-0052-2fbdd5ead241} {Suffrage:Voter ID:server-b981a2b9-bc07-43c9-cb5e-0ce94217a371 Address:b981a2b9-bc07-43c9-cb5e-0ce94217a371}]"
2025-01-28T14:55:41.886Z [INFO]  server-27655c63-2055-a7b1-0052-2fbdd5ead241: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-55b86cf6-6a23-3b03-15df-f5aefa95ce66 Address:55b86cf6-6a23-3b03-15df-f5aefa95ce66} {Suffrage:Voter ID:server-27655c63-2055-a7b1-0052-2fbdd5ead241 Address:27655c63-2055-a7b1-0052-2fbdd5ead241} {Suffrage:Voter ID:server-b981a2b9-bc07-43c9-cb5e-0ce94217a371 Address:b981a2b9-bc07-43c9-cb5e-0ce94217a371}]"
2025-01-28T14:55:41.886Z [INFO]  server-55b86cf6-6a23-3b03-15df-f5aefa95ce66: entering follower state: follower="Node at 55b86cf6-6a23-3b03-15df-f5aefa95ce66 [Follower]" leader-address= leader-id=
2025-01-28T14:55:41.886Z [INFO]  server-27655c63-2055-a7b1-0052-2fbdd5ead241: entering follower state: follower="Node at 27655c63-2055-a7b1-0052-2fbdd5ead241 [Follower]" leader-address= leader-id=
2025-01-28T14:55:41.887Z [INFO]  server-b981a2b9-bc07-43c9-cb5e-0ce94217a371: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-55b86cf6-6a23-3b03-15df-f5aefa95ce66 Address:55b86cf6-6a23-3b03-15df-f5aefa95ce66} {Suffrage:Voter ID:server-27655c63-2055-a7b1-0052-2fbdd5ead241 Address:27655c63-2055-a7b1-0052-2fbdd5ead241} {Suffrage:Voter ID:server-b981a2b9-bc07-43c9-cb5e-0ce94217a371 Address:b981a2b9-bc07-43c9-cb5e-0ce94217a371}]"
2025-01-28T14:55:41.887Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:41.887Z [INFO]  server-b981a2b9-bc07-43c9-cb5e-0ce94217a371: entering follower state: follower="Node at b981a2b9-bc07-43c9-cb5e-0ce94217a371 [Follower]" leader-address= leader-id=
2025-01-28T14:55:41.952Z [WARN]  server-b981a2b9-bc07-43c9-cb5e-0ce94217a371: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:41.953Z [INFO]  server-b981a2b9-bc07-43c9-cb5e-0ce94217a371: entering candidate state: node="Node at b981a2b9-bc07-43c9-cb5e-0ce94217a371 [Candidate]" term=2
2025-01-28T14:55:41.953Z [INFO]  server-b981a2b9-bc07-43c9-cb5e-0ce94217a371: election won: term=2 tally=2
2025-01-28T14:55:41.953Z [INFO]  server-b981a2b9-bc07-43c9-cb5e-0ce94217a371: entering leader state: leader="Node at b981a2b9-bc07-43c9-cb5e-0ce94217a371 [Leader]"
2025-01-28T14:55:41.953Z [INFO]  server-b981a2b9-bc07-43c9-cb5e-0ce94217a371: added peer, starting replication: peer=server-55b86cf6-6a23-3b03-15df-f5aefa95ce66
2025-01-28T14:55:41.953Z [INFO]  server-b981a2b9-bc07-43c9-cb5e-0ce94217a371: added peer, starting replication: peer=server-27655c63-2055-a7b1-0052-2fbdd5ead241
2025-01-28T14:55:41.953Z [INFO]  server-b981a2b9-bc07-43c9-cb5e-0ce94217a371: pipelining replication: peer="{Voter server-55b86cf6-6a23-3b03-15df-f5aefa95ce66 55b86cf6-6a23-3b03-15df-f5aefa95ce66}"
2025-01-28T14:55:41.953Z [INFO]  server-b981a2b9-bc07-43c9-cb5e-0ce94217a371: pipelining replication: peer="{Voter server-27655c63-2055-a7b1-0052-2fbdd5ead241 27655c63-2055-a7b1-0052-2fbdd5ead241}"
2025-01-28T14:55:42.059Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:41.953311948 +0000 UTC m=+127.627801928 (1 nodes), 66.134201ms from start of poll, 67.060413ms from cluster start. Timeout at 2025-01-28 14:55:42.058319647 +0000 UTC m=+127.732809627, 105.007699ms after stability
2025-01-28T14:55:42.065Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:42.170Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:42.065298884 +0000 UTC m=+127.739788864 (2 nodes), 2.08µs from start of poll, 179.047349ms from cluster start. Timeout at 2025-01-28 14:55:42.170494045 +0000 UTC m=+127.844984025, 105.195161ms after stability
    raft_test.go:1968: [INFO] Disconnecting Node at 55b86cf6-6a23-3b03-15df-f5aefa95ce66 [Follower]
2025-01-28T14:55:42.171Z [ERROR] server-b981a2b9-bc07-43c9-cb5e-0ce94217a371: failed to heartbeat to: peer=55b86cf6-6a23-3b03-15df-f5aefa95ce66 backoff time=10ms error="failed to connect to peer: 55b86cf6-6a23-3b03-15df-f5aefa95ce66"
2025-01-28T14:55:42.171Z [INFO]  server-b981a2b9-bc07-43c9-cb5e-0ce94217a371: aborting pipeline replication: peer="{Voter server-27655c63-2055-a7b1-0052-2fbdd5ead241 27655c63-2055-a7b1-0052-2fbdd5ead241}"
2025-01-28T14:55:42.171Z [INFO]  server-b981a2b9-bc07-43c9-cb5e-0ce94217a371: aborting pipeline replication: peer="{Voter server-55b86cf6-6a23-3b03-15df-f5aefa95ce66 55b86cf6-6a23-3b03-15df-f5aefa95ce66}"
--- PASS: TestRaft_VerifyLeader_PartialConnect (0.29s)
=== RUN   TestRaft_NotifyCh
2025-01-28T14:55:42.173Z [INFO]  server-80b55dfb-55f3-519c-4e51-67d2a95bc98b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-80b55dfb-55f3-519c-4e51-67d2a95bc98b Address:80b55dfb-55f3-519c-4e51-67d2a95bc98b}]"
2025-01-28T14:55:42.173Z [INFO]  server-80b55dfb-55f3-519c-4e51-67d2a95bc98b: entering follower state: follower="Node at 80b55dfb-55f3-519c-4e51-67d2a95bc98b [Follower]" leader-address= leader-id=
2025-01-28T14:55:42.258Z [WARN]  server-80b55dfb-55f3-519c-4e51-67d2a95bc98b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:42.258Z [INFO]  server-80b55dfb-55f3-519c-4e51-67d2a95bc98b: entering candidate state: node="Node at 80b55dfb-55f3-519c-4e51-67d2a95bc98b [Candidate]" term=2
2025-01-28T14:55:42.258Z [INFO]  server-80b55dfb-55f3-519c-4e51-67d2a95bc98b: election won: term=2 tally=1
2025-01-28T14:55:42.258Z [INFO]  server-80b55dfb-55f3-519c-4e51-67d2a95bc98b: entering leader state: leader="Node at 80b55dfb-55f3-519c-4e51-67d2a95bc98b [Leader]"
--- PASS: TestRaft_NotifyCh (0.09s)
=== RUN   TestRaft_AppendEntry
2025-01-28T14:55:42.263Z [INFO]  server-676b8923-473c-ba66-a195-c005c4ff1866: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-676b8923-473c-ba66-a195-c005c4ff1866 Address:676b8923-473c-ba66-a195-c005c4ff1866} {Suffrage:Voter ID:server-191c856e-43d0-57e6-ac74-7bd8b76f4125 Address:191c856e-43d0-57e6-ac74-7bd8b76f4125} {Suffrage:Voter ID:server-0a1710f5-c74f-de9f-75d6-2fed7b785acb Address:0a1710f5-c74f-de9f-75d6-2fed7b785acb}]"
2025-01-28T14:55:42.263Z [INFO]  server-191c856e-43d0-57e6-ac74-7bd8b76f4125: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-676b8923-473c-ba66-a195-c005c4ff1866 Address:676b8923-473c-ba66-a195-c005c4ff1866} {Suffrage:Voter ID:server-191c856e-43d0-57e6-ac74-7bd8b76f4125 Address:191c856e-43d0-57e6-ac74-7bd8b76f4125} {Suffrage:Voter ID:server-0a1710f5-c74f-de9f-75d6-2fed7b785acb Address:0a1710f5-c74f-de9f-75d6-2fed7b785acb}]"
2025-01-28T14:55:42.264Z [INFO]  server-0a1710f5-c74f-de9f-75d6-2fed7b785acb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-676b8923-473c-ba66-a195-c005c4ff1866 Address:676b8923-473c-ba66-a195-c005c4ff1866} {Suffrage:Voter ID:server-191c856e-43d0-57e6-ac74-7bd8b76f4125 Address:191c856e-43d0-57e6-ac74-7bd8b76f4125} {Suffrage:Voter ID:server-0a1710f5-c74f-de9f-75d6-2fed7b785acb Address:0a1710f5-c74f-de9f-75d6-2fed7b785acb}]"
2025-01-28T14:55:42.264Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:42.265Z [INFO]  server-676b8923-473c-ba66-a195-c005c4ff1866: entering follower state: follower="Node at 676b8923-473c-ba66-a195-c005c4ff1866 [Follower]" leader-address= leader-id=
2025-01-28T14:55:42.266Z [INFO]  server-191c856e-43d0-57e6-ac74-7bd8b76f4125: entering follower state: follower="Node at 191c856e-43d0-57e6-ac74-7bd8b76f4125 [Follower]" leader-address= leader-id=
2025-01-28T14:55:42.266Z [INFO]  server-0a1710f5-c74f-de9f-75d6-2fed7b785acb: entering follower state: follower="Node at 0a1710f5-c74f-de9f-75d6-2fed7b785acb [Follower]" leader-address= leader-id=
2025-01-28T14:55:42.317Z [WARN]  server-0a1710f5-c74f-de9f-75d6-2fed7b785acb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:42.317Z [INFO]  server-0a1710f5-c74f-de9f-75d6-2fed7b785acb: entering candidate state: node="Node at 0a1710f5-c74f-de9f-75d6-2fed7b785acb [Candidate]" term=2
2025-01-28T14:55:42.317Z [INFO]  server-0a1710f5-c74f-de9f-75d6-2fed7b785acb: election won: term=2 tally=2
2025-01-28T14:55:42.318Z [INFO]  server-0a1710f5-c74f-de9f-75d6-2fed7b785acb: entering leader state: leader="Node at 0a1710f5-c74f-de9f-75d6-2fed7b785acb [Leader]"
2025-01-28T14:55:42.318Z [INFO]  server-0a1710f5-c74f-de9f-75d6-2fed7b785acb: added peer, starting replication: peer=server-676b8923-473c-ba66-a195-c005c4ff1866
2025-01-28T14:55:42.318Z [INFO]  server-0a1710f5-c74f-de9f-75d6-2fed7b785acb: added peer, starting replication: peer=server-191c856e-43d0-57e6-ac74-7bd8b76f4125
2025-01-28T14:55:42.318Z [INFO]  server-0a1710f5-c74f-de9f-75d6-2fed7b785acb: pipelining replication: peer="{Voter server-191c856e-43d0-57e6-ac74-7bd8b76f4125 191c856e-43d0-57e6-ac74-7bd8b76f4125}"
2025-01-28T14:55:42.318Z [INFO]  server-0a1710f5-c74f-de9f-75d6-2fed7b785acb: pipelining replication: peer="{Voter server-676b8923-473c-ba66-a195-c005c4ff1866 676b8923-473c-ba66-a195-c005c4ff1866}"
2025-01-28T14:55:42.424Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:42.318172449 +0000 UTC m=+127.992662429 (2 nodes), 53.475535ms from start of poll, 55.390355ms from cluster start. Timeout at 2025-01-28 14:55:42.423178008 +0000 UTC m=+128.097668088, 105.005659ms after stability
2025-01-28T14:55:42.424Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:42.530Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:42.424440564 +0000 UTC m=+128.098930544 (1 nodes), 2.5µs from start of poll, 161.65847ms from cluster start. Timeout at 2025-01-28 14:55:42.529610948 +0000 UTC m=+128.204100908, 105.170364ms after stability
2025-01-28T14:55:42.530Z [WARN]  server-676b8923-473c-ba66-a195-c005c4ff1866: clearing log suffix: from=1 to=2
2025-01-28T14:55:42.530Z [WARN]  server-676b8923-473c-ba66-a195-c005c4ff1866: skipping application of old log: index=1
2025-01-28T14:55:42.531Z [WARN]  server-676b8923-473c-ba66-a195-c005c4ff1866: skipping application of old log: index=1
2025-01-28T14:55:42.531Z [INFO]  server-0a1710f5-c74f-de9f-75d6-2fed7b785acb: aborting pipeline replication: peer="{Voter server-191c856e-43d0-57e6-ac74-7bd8b76f4125 191c856e-43d0-57e6-ac74-7bd8b76f4125}"
2025-01-28T14:55:42.531Z [INFO]  server-0a1710f5-c74f-de9f-75d6-2fed7b785acb: aborting pipeline replication: peer="{Voter server-676b8923-473c-ba66-a195-c005c4ff1866 676b8923-473c-ba66-a195-c005c4ff1866}"
--- PASS: TestRaft_AppendEntry (0.27s)
=== RUN   TestRaft_VotingGrant_WhenLeaderAvailable
2025-01-28T14:55:42.538Z [INFO]  server-374863fa-c980-13b6-5302-d34b4dafb566: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-374863fa-c980-13b6-5302-d34b4dafb566 Address:374863fa-c980-13b6-5302-d34b4dafb566} {Suffrage:Voter ID:server-c1cbb197-190d-1668-0f50-a7519d9cd324 Address:c1cbb197-190d-1668-0f50-a7519d9cd324} {Suffrage:Voter ID:server-018c737c-0d71-5c52-28b2-890d72878f19 Address:018c737c-0d71-5c52-28b2-890d72878f19}]"
2025-01-28T14:55:42.538Z [INFO]  server-c1cbb197-190d-1668-0f50-a7519d9cd324: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-374863fa-c980-13b6-5302-d34b4dafb566 Address:374863fa-c980-13b6-5302-d34b4dafb566} {Suffrage:Voter ID:server-c1cbb197-190d-1668-0f50-a7519d9cd324 Address:c1cbb197-190d-1668-0f50-a7519d9cd324} {Suffrage:Voter ID:server-018c737c-0d71-5c52-28b2-890d72878f19 Address:018c737c-0d71-5c52-28b2-890d72878f19}]"
2025-01-28T14:55:42.539Z [INFO]  server-018c737c-0d71-5c52-28b2-890d72878f19: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-374863fa-c980-13b6-5302-d34b4dafb566 Address:374863fa-c980-13b6-5302-d34b4dafb566} {Suffrage:Voter ID:server-c1cbb197-190d-1668-0f50-a7519d9cd324 Address:c1cbb197-190d-1668-0f50-a7519d9cd324} {Suffrage:Voter ID:server-018c737c-0d71-5c52-28b2-890d72878f19 Address:018c737c-0d71-5c52-28b2-890d72878f19}]"
2025-01-28T14:55:42.539Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:42.540Z [INFO]  server-374863fa-c980-13b6-5302-d34b4dafb566: entering follower state: follower="Node at 374863fa-c980-13b6-5302-d34b4dafb566 [Follower]" leader-address= leader-id=
2025-01-28T14:55:42.540Z [INFO]  server-c1cbb197-190d-1668-0f50-a7519d9cd324: entering follower state: follower="Node at c1cbb197-190d-1668-0f50-a7519d9cd324 [Follower]" leader-address= leader-id=
2025-01-28T14:55:42.541Z [INFO]  server-018c737c-0d71-5c52-28b2-890d72878f19: entering follower state: follower="Node at 018c737c-0d71-5c52-28b2-890d72878f19 [Follower]" leader-address= leader-id=
2025-01-28T14:55:42.599Z [WARN]  server-c1cbb197-190d-1668-0f50-a7519d9cd324: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:42.599Z [INFO]  server-c1cbb197-190d-1668-0f50-a7519d9cd324: entering candidate state: node="Node at c1cbb197-190d-1668-0f50-a7519d9cd324 [Candidate]" term=2
2025-01-28T14:55:42.599Z [INFO]  server-c1cbb197-190d-1668-0f50-a7519d9cd324: election won: term=2 tally=2
2025-01-28T14:55:42.599Z [INFO]  server-c1cbb197-190d-1668-0f50-a7519d9cd324: entering leader state: leader="Node at c1cbb197-190d-1668-0f50-a7519d9cd324 [Leader]"
2025-01-28T14:55:42.599Z [INFO]  server-c1cbb197-190d-1668-0f50-a7519d9cd324: added peer, starting replication: peer=server-374863fa-c980-13b6-5302-d34b4dafb566
2025-01-28T14:55:42.599Z [INFO]  server-c1cbb197-190d-1668-0f50-a7519d9cd324: added peer, starting replication: peer=server-018c737c-0d71-5c52-28b2-890d72878f19
2025-01-28T14:55:42.599Z [INFO]  server-c1cbb197-190d-1668-0f50-a7519d9cd324: pipelining replication: peer="{Voter server-374863fa-c980-13b6-5302-d34b4dafb566 374863fa-c980-13b6-5302-d34b4dafb566}"
2025-01-28T14:55:42.599Z [INFO]  server-c1cbb197-190d-1668-0f50-a7519d9cd324: pipelining replication: peer="{Voter server-018c737c-0d71-5c52-28b2-890d72878f19 018c737c-0d71-5c52-28b2-890d72878f19}"
2025-01-28T14:55:42.705Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:42.599611319 +0000 UTC m=+128.274101299 (2 nodes), 59.863792ms from start of poll, 65.044684ms from cluster start. Timeout at 2025-01-28 14:55:42.704628977 +0000 UTC m=+128.379118957, 105.017658ms after stability
2025-01-28T14:55:42.705Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:42.810Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:42.705323427 +0000 UTC m=+128.379813507 (1 nodes), 2.02µs from start of poll, 170.756892ms from cluster start. Timeout at 2025-01-28 14:55:42.810497951 +0000 UTC m=+128.484987931, 105.174424ms after stability
2025-01-28T14:55:42.810Z [WARN]  server-018c737c-0d71-5c52-28b2-890d72878f19: rejecting vote request since we have a leader: from=374863fa-c980-13b6-5302-d34b4dafb566 leader=c1cbb197-190d-1668-0f50-a7519d9cd324 leader-id=server-c1cbb197-190d-1668-0f50-a7519d9cd324
2025-01-28T14:55:42.816Z [INFO]  server-c1cbb197-190d-1668-0f50-a7519d9cd324: aborting pipeline replication: peer="{Voter server-018c737c-0d71-5c52-28b2-890d72878f19 018c737c-0d71-5c52-28b2-890d72878f19}"
2025-01-28T14:55:42.816Z [ERROR] server-c1cbb197-190d-1668-0f50-a7519d9cd324: peer has newer term, stopping replication: peer="{Voter server-374863fa-c980-13b6-5302-d34b4dafb566 374863fa-c980-13b6-5302-d34b4dafb566}"
2025-01-28T14:55:42.816Z [INFO]  server-c1cbb197-190d-1668-0f50-a7519d9cd324: aborting pipeline replication: peer="{Voter server-374863fa-c980-13b6-5302-d34b4dafb566 374863fa-c980-13b6-5302-d34b4dafb566}"
--- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.29s)
=== RUN   TestRaft_ProtocolVersion_RejectRPC
2025-01-28T14:55:42.820Z [INFO]  server-8f956ec1-4b23-9326-071b-cd8cfab8f882: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f956ec1-4b23-9326-071b-cd8cfab8f882 Address:8f956ec1-4b23-9326-071b-cd8cfab8f882} {Suffrage:Voter ID:server-207670f0-993b-d09f-9dc8-7eb3ac0d9070 Address:207670f0-993b-d09f-9dc8-7eb3ac0d9070} {Suffrage:Voter ID:server-b551991d-4bcd-596e-328c-4dcdaf19a40f Address:b551991d-4bcd-596e-328c-4dcdaf19a40f}]"
2025-01-28T14:55:42.820Z [INFO]  server-8f956ec1-4b23-9326-071b-cd8cfab8f882: entering follower state: follower="Node at 8f956ec1-4b23-9326-071b-cd8cfab8f882 [Follower]" leader-address= leader-id=
2025-01-28T14:55:42.820Z [INFO]  server-207670f0-993b-d09f-9dc8-7eb3ac0d9070: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f956ec1-4b23-9326-071b-cd8cfab8f882 Address:8f956ec1-4b23-9326-071b-cd8cfab8f882} {Suffrage:Voter ID:server-207670f0-993b-d09f-9dc8-7eb3ac0d9070 Address:207670f0-993b-d09f-9dc8-7eb3ac0d9070} {Suffrage:Voter ID:server-b551991d-4bcd-596e-328c-4dcdaf19a40f Address:b551991d-4bcd-596e-328c-4dcdaf19a40f}]"
2025-01-28T14:55:42.820Z [INFO]  server-207670f0-993b-d09f-9dc8-7eb3ac0d9070: entering follower state: follower="Node at 207670f0-993b-d09f-9dc8-7eb3ac0d9070 [Follower]" leader-address= leader-id=
2025-01-28T14:55:42.821Z [INFO]  server-b551991d-4bcd-596e-328c-4dcdaf19a40f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8f956ec1-4b23-9326-071b-cd8cfab8f882 Address:8f956ec1-4b23-9326-071b-cd8cfab8f882} {Suffrage:Voter ID:server-207670f0-993b-d09f-9dc8-7eb3ac0d9070 Address:207670f0-993b-d09f-9dc8-7eb3ac0d9070} {Suffrage:Voter ID:server-b551991d-4bcd-596e-328c-4dcdaf19a40f Address:b551991d-4bcd-596e-328c-4dcdaf19a40f}]"
2025-01-28T14:55:42.821Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:42.821Z [INFO]  server-b551991d-4bcd-596e-328c-4dcdaf19a40f: entering follower state: follower="Node at b551991d-4bcd-596e-328c-4dcdaf19a40f [Follower]" leader-address= leader-id=
2025-01-28T14:55:42.871Z [WARN]  server-b551991d-4bcd-596e-328c-4dcdaf19a40f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:42.871Z [INFO]  server-b551991d-4bcd-596e-328c-4dcdaf19a40f: entering candidate state: node="Node at b551991d-4bcd-596e-328c-4dcdaf19a40f [Candidate]" term=2
2025-01-28T14:55:42.871Z [INFO]  server-b551991d-4bcd-596e-328c-4dcdaf19a40f: election won: term=2 tally=2
2025-01-28T14:55:42.871Z [INFO]  server-b551991d-4bcd-596e-328c-4dcdaf19a40f: entering leader state: leader="Node at b551991d-4bcd-596e-328c-4dcdaf19a40f [Leader]"
2025-01-28T14:55:42.871Z [INFO]  server-b551991d-4bcd-596e-328c-4dcdaf19a40f: added peer, starting replication: peer=server-8f956ec1-4b23-9326-071b-cd8cfab8f882
2025-01-28T14:55:42.871Z [INFO]  server-b551991d-4bcd-596e-328c-4dcdaf19a40f: added peer, starting replication: peer=server-207670f0-993b-d09f-9dc8-7eb3ac0d9070
2025-01-28T14:55:42.872Z [INFO]  server-b551991d-4bcd-596e-328c-4dcdaf19a40f: pipelining replication: peer="{Voter server-8f956ec1-4b23-9326-071b-cd8cfab8f882 8f956ec1-4b23-9326-071b-cd8cfab8f882}"
2025-01-28T14:55:42.872Z [INFO]  server-b551991d-4bcd-596e-328c-4dcdaf19a40f: pipelining replication: peer="{Voter server-207670f0-993b-d09f-9dc8-7eb3ac0d9070 207670f0-993b-d09f-9dc8-7eb3ac0d9070}"
2025-01-28T14:55:42.977Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:42.871859736 +0000 UTC m=+128.546349696 (2 nodes), 50.66248ms from start of poll, 51.587773ms from cluster start. Timeout at 2025-01-28 14:55:42.976865895 +0000 UTC m=+128.651355915, 105.006219ms after stability
2025-01-28T14:55:42.977Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:43.082Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:42.977198519 +0000 UTC m=+128.651688499 (1 nodes), 2.06µs from start of poll, 156.926576ms from cluster start. Timeout at 2025-01-28 14:55:43.082372783 +0000 UTC m=+128.756862783, 105.174284ms after stability
2025-01-28T14:55:43.082Z [INFO]  server-b551991d-4bcd-596e-328c-4dcdaf19a40f: aborting pipeline replication: peer="{Voter server-8f956ec1-4b23-9326-071b-cd8cfab8f882 8f956ec1-4b23-9326-071b-cd8cfab8f882}"
2025-01-28T14:55:43.082Z [INFO]  server-b551991d-4bcd-596e-328c-4dcdaf19a40f: aborting pipeline replication: peer="{Voter server-207670f0-993b-d09f-9dc8-7eb3ac0d9070 207670f0-993b-d09f-9dc8-7eb3ac0d9070}"
--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_1_2
    raft_test.go:2151: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_2_3
    raft_test.go:2196: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s)
=== RUN   TestRaft_LeaderID_Propagated
2025-01-28T14:55:43.087Z [INFO]  server-68c9cb15-984b-6ca2-c9bd-2def67a97228: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68c9cb15-984b-6ca2-c9bd-2def67a97228 Address:68c9cb15-984b-6ca2-c9bd-2def67a97228} {Suffrage:Voter ID:server-09f7553a-5545-8684-f197-f0e35932a4d1 Address:09f7553a-5545-8684-f197-f0e35932a4d1} {Suffrage:Voter ID:server-d23f2521-39d4-26ef-028d-f07ea2c33c6f Address:d23f2521-39d4-26ef-028d-f07ea2c33c6f}]"
2025-01-28T14:55:43.088Z [INFO]  server-09f7553a-5545-8684-f197-f0e35932a4d1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68c9cb15-984b-6ca2-c9bd-2def67a97228 Address:68c9cb15-984b-6ca2-c9bd-2def67a97228} {Suffrage:Voter ID:server-09f7553a-5545-8684-f197-f0e35932a4d1 Address:09f7553a-5545-8684-f197-f0e35932a4d1} {Suffrage:Voter ID:server-d23f2521-39d4-26ef-028d-f07ea2c33c6f Address:d23f2521-39d4-26ef-028d-f07ea2c33c6f}]"
2025-01-28T14:55:43.088Z [INFO]  server-d23f2521-39d4-26ef-028d-f07ea2c33c6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-68c9cb15-984b-6ca2-c9bd-2def67a97228 Address:68c9cb15-984b-6ca2-c9bd-2def67a97228} {Suffrage:Voter ID:server-09f7553a-5545-8684-f197-f0e35932a4d1 Address:09f7553a-5545-8684-f197-f0e35932a4d1} {Suffrage:Voter ID:server-d23f2521-39d4-26ef-028d-f07ea2c33c6f Address:d23f2521-39d4-26ef-028d-f07ea2c33c6f}]"
2025-01-28T14:55:43.088Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:43.090Z [INFO]  server-68c9cb15-984b-6ca2-c9bd-2def67a97228: entering follower state: follower="Node at 68c9cb15-984b-6ca2-c9bd-2def67a97228 [Follower]" leader-address= leader-id=
2025-01-28T14:55:43.091Z [INFO]  server-d23f2521-39d4-26ef-028d-f07ea2c33c6f: entering follower state: follower="Node at d23f2521-39d4-26ef-028d-f07ea2c33c6f [Follower]" leader-address= leader-id=
2025-01-28T14:55:43.091Z [INFO]  server-09f7553a-5545-8684-f197-f0e35932a4d1: entering follower state: follower="Node at 09f7553a-5545-8684-f197-f0e35932a4d1 [Follower]" leader-address= leader-id=
2025-01-28T14:55:43.142Z [WARN]  server-d23f2521-39d4-26ef-028d-f07ea2c33c6f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:43.142Z [INFO]  server-d23f2521-39d4-26ef-028d-f07ea2c33c6f: entering candidate state: node="Node at d23f2521-39d4-26ef-028d-f07ea2c33c6f [Candidate]" term=2
2025-01-28T14:55:43.142Z [INFO]  server-d23f2521-39d4-26ef-028d-f07ea2c33c6f: election won: term=2 tally=2
2025-01-28T14:55:43.142Z [INFO]  server-d23f2521-39d4-26ef-028d-f07ea2c33c6f: entering leader state: leader="Node at d23f2521-39d4-26ef-028d-f07ea2c33c6f [Leader]"
2025-01-28T14:55:43.142Z [INFO]  server-d23f2521-39d4-26ef-028d-f07ea2c33c6f: added peer, starting replication: peer=server-68c9cb15-984b-6ca2-c9bd-2def67a97228
2025-01-28T14:55:43.142Z [INFO]  server-d23f2521-39d4-26ef-028d-f07ea2c33c6f: added peer, starting replication: peer=server-09f7553a-5545-8684-f197-f0e35932a4d1
2025-01-28T14:55:43.142Z [INFO]  server-d23f2521-39d4-26ef-028d-f07ea2c33c6f: pipelining replication: peer="{Voter server-68c9cb15-984b-6ca2-c9bd-2def67a97228 68c9cb15-984b-6ca2-c9bd-2def67a97228}"
2025-01-28T14:55:43.142Z [INFO]  server-d23f2521-39d4-26ef-028d-f07ea2c33c6f: pipelining replication: peer="{Voter server-09f7553a-5545-8684-f197-f0e35932a4d1 09f7553a-5545-8684-f197-f0e35932a4d1}"
2025-01-28T14:55:43.248Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:43.142463329 +0000 UTC m=+128.816953289 (1 nodes), 54.000866ms from start of poll, 55.076324ms from cluster start. Timeout at 2025-01-28 14:55:43.247477588 +0000 UTC m=+128.921967568, 105.014279ms after stability
2025-01-28T14:55:43.248Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:43.353Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:43.248278587 +0000 UTC m=+128.922768567 (1 nodes), 1.64µs from start of poll, 160.891602ms from cluster start. Timeout at 2025-01-28 14:55:43.35345401 +0000 UTC m=+129.027944110, 105.175543ms after stability
2025-01-28T14:55:43.353Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:43.459Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:43.354007613 +0000 UTC m=+129.028497593 (1 nodes), 1.74µs from start of poll, 266.620628ms from cluster start. Timeout at 2025-01-28 14:55:43.4591465 +0000 UTC m=+129.133636540, 105.138947ms after stability
2025-01-28T14:55:43.460Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:43.566Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:43.460415355 +0000 UTC m=+129.134905315 (1 nodes), 1.88µs from start of poll, 373.02835ms from cluster start. Timeout at 2025-01-28 14:55:43.565550162 +0000 UTC m=+129.240040242, 105.134927ms after stability
2025-01-28T14:55:43.567Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:43.672Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:43.56720868 +0000 UTC m=+129.241698640 (1 nodes), 2.18µs from start of poll, 479.821675ms from cluster start. Timeout at 2025-01-28 14:55:43.672390203 +0000 UTC m=+129.346880183, 105.181543ms after stability
2025-01-28T14:55:43.673Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:43.779Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:43.673347547 +0000 UTC m=+129.347837507 (1 nodes), 1.839µs from start of poll, 585.960542ms from cluster start. Timeout at 2025-01-28 14:55:43.778509891 +0000 UTC m=+129.452999871, 105.162364ms after stability
2025-01-28T14:55:43.885Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:43.991Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:43.885174385 +0000 UTC m=+129.559664365 (1 nodes), 1.84µs from start of poll, 797.7874ms from cluster start. Timeout at 2025-01-28 14:55:43.990271676 +0000 UTC m=+129.664761656, 105.097291ms after stability
2025-01-28T14:55:43.994Z [INFO]  server-d23f2521-39d4-26ef-028d-f07ea2c33c6f: aborting pipeline replication: peer="{Voter server-09f7553a-5545-8684-f197-f0e35932a4d1 09f7553a-5545-8684-f197-f0e35932a4d1}"
2025-01-28T14:55:43.994Z [INFO]  server-d23f2521-39d4-26ef-028d-f07ea2c33c6f: aborting pipeline replication: peer="{Voter server-68c9cb15-984b-6ca2-c9bd-2def67a97228 68c9cb15-984b-6ca2-c9bd-2def67a97228}"
2025-01-28T14:55:44.495Z [ERROR] server-d23f2521-39d4-26ef-028d-f07ea2c33c6f: failed to heartbeat to: peer=09f7553a-5545-8684-f197-f0e35932a4d1 backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeaderID_Propagated (1.41s)
=== RUN   TestRaft_LeadershipTransferInProgress
--- PASS: TestRaft_LeadershipTransferInProgress (0.00s)
=== RUN   TestRaft_LeadershipTransferPickServer
--- PASS: TestRaft_LeadershipTransferPickServer (0.00s)
=== RUN   TestRaft_LeadershipTransfer
2025-01-28T14:55:44.499Z [INFO]  server-955c351e-e622-2746-f130-abbd0c0ab2ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-955c351e-e622-2746-f130-abbd0c0ab2ca Address:955c351e-e622-2746-f130-abbd0c0ab2ca} {Suffrage:Voter ID:server-b060373a-87ff-c769-2110-10dcff83778e Address:b060373a-87ff-c769-2110-10dcff83778e} {Suffrage:Voter ID:server-7535ea26-0093-6cdc-5060-927a8bee1d14 Address:7535ea26-0093-6cdc-5060-927a8bee1d14}]"
2025-01-28T14:55:44.499Z [INFO]  server-955c351e-e622-2746-f130-abbd0c0ab2ca: entering follower state: follower="Node at 955c351e-e622-2746-f130-abbd0c0ab2ca [Follower]" leader-address= leader-id=
2025-01-28T14:55:44.499Z [INFO]  server-b060373a-87ff-c769-2110-10dcff83778e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-955c351e-e622-2746-f130-abbd0c0ab2ca Address:955c351e-e622-2746-f130-abbd0c0ab2ca} {Suffrage:Voter ID:server-b060373a-87ff-c769-2110-10dcff83778e Address:b060373a-87ff-c769-2110-10dcff83778e} {Suffrage:Voter ID:server-7535ea26-0093-6cdc-5060-927a8bee1d14 Address:7535ea26-0093-6cdc-5060-927a8bee1d14}]"
2025-01-28T14:55:44.499Z [INFO]  server-b060373a-87ff-c769-2110-10dcff83778e: entering follower state: follower="Node at b060373a-87ff-c769-2110-10dcff83778e [Follower]" leader-address= leader-id=
2025-01-28T14:55:44.500Z [INFO]  server-7535ea26-0093-6cdc-5060-927a8bee1d14: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-955c351e-e622-2746-f130-abbd0c0ab2ca Address:955c351e-e622-2746-f130-abbd0c0ab2ca} {Suffrage:Voter ID:server-b060373a-87ff-c769-2110-10dcff83778e Address:b060373a-87ff-c769-2110-10dcff83778e} {Suffrage:Voter ID:server-7535ea26-0093-6cdc-5060-927a8bee1d14 Address:7535ea26-0093-6cdc-5060-927a8bee1d14}]"
2025-01-28T14:55:44.500Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:44.500Z [INFO]  server-7535ea26-0093-6cdc-5060-927a8bee1d14: entering follower state: follower="Node at 7535ea26-0093-6cdc-5060-927a8bee1d14 [Follower]" leader-address= leader-id=
2025-01-28T14:55:44.553Z [WARN]  server-b060373a-87ff-c769-2110-10dcff83778e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:44.553Z [INFO]  server-b060373a-87ff-c769-2110-10dcff83778e: entering candidate state: node="Node at b060373a-87ff-c769-2110-10dcff83778e [Candidate]" term=2
2025-01-28T14:55:44.553Z [INFO]  server-b060373a-87ff-c769-2110-10dcff83778e: election won: term=2 tally=2
2025-01-28T14:55:44.553Z [INFO]  server-b060373a-87ff-c769-2110-10dcff83778e: entering leader state: leader="Node at b060373a-87ff-c769-2110-10dcff83778e [Leader]"
2025-01-28T14:55:44.553Z [INFO]  server-b060373a-87ff-c769-2110-10dcff83778e: added peer, starting replication: peer=server-955c351e-e622-2746-f130-abbd0c0ab2ca
2025-01-28T14:55:44.553Z [INFO]  server-b060373a-87ff-c769-2110-10dcff83778e: added peer, starting replication: peer=server-7535ea26-0093-6cdc-5060-927a8bee1d14
2025-01-28T14:55:44.553Z [INFO]  server-b060373a-87ff-c769-2110-10dcff83778e: pipelining replication: peer="{Voter server-955c351e-e622-2746-f130-abbd0c0ab2ca 955c351e-e622-2746-f130-abbd0c0ab2ca}"
2025-01-28T14:55:44.553Z [INFO]  server-b060373a-87ff-c769-2110-10dcff83778e: pipelining replication: peer="{Voter server-7535ea26-0093-6cdc-5060-927a8bee1d14 7535ea26-0093-6cdc-5060-927a8bee1d14}"
2025-01-28T14:55:44.658Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:44.553400047 +0000 UTC m=+130.227890027 (1 nodes), 53.243677ms from start of poll, 54.27406ms from cluster start. Timeout at 2025-01-28 14:55:44.658404827 +0000 UTC m=+130.332894807, 105.00478ms after stability
2025-01-28T14:55:44.658Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:44.763Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:44.658619301 +0000 UTC m=+130.333109281 (1 nodes), 1.88µs from start of poll, 159.493314ms from cluster start. Timeout at 2025-01-28 14:55:44.763784965 +0000 UTC m=+130.438274925, 105.165644ms after stability
2025-01-28T14:55:44.764Z [INFO]  server-955c351e-e622-2746-f130-abbd0c0ab2ca: entering candidate state: node="Node at 955c351e-e622-2746-f130-abbd0c0ab2ca [Candidate]" term=3
2025-01-28T14:55:44.764Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:44.764Z [INFO]  server-b060373a-87ff-c769-2110-10dcff83778e: entering follower state: follower="Node at b060373a-87ff-c769-2110-10dcff83778e [Follower]" leader-address= leader-id=
2025-01-28T14:55:44.764Z [INFO]  server-955c351e-e622-2746-f130-abbd0c0ab2ca: election won: term=3 tally=2
2025-01-28T14:55:44.764Z [INFO]  server-b060373a-87ff-c769-2110-10dcff83778e: aborting pipeline replication: peer="{Voter server-7535ea26-0093-6cdc-5060-927a8bee1d14 7535ea26-0093-6cdc-5060-927a8bee1d14}"
2025-01-28T14:55:44.764Z [INFO]  server-955c351e-e622-2746-f130-abbd0c0ab2ca: entering leader state: leader="Node at 955c351e-e622-2746-f130-abbd0c0ab2ca [Leader]"
2025-01-28T14:55:44.764Z [INFO]  server-955c351e-e622-2746-f130-abbd0c0ab2ca: added peer, starting replication: peer=server-b060373a-87ff-c769-2110-10dcff83778e
2025-01-28T14:55:44.764Z [INFO]  server-955c351e-e622-2746-f130-abbd0c0ab2ca: added peer, starting replication: peer=server-7535ea26-0093-6cdc-5060-927a8bee1d14
2025-01-28T14:55:44.764Z [INFO]  server-b060373a-87ff-c769-2110-10dcff83778e: aborting pipeline replication: peer="{Voter server-955c351e-e622-2746-f130-abbd0c0ab2ca 955c351e-e622-2746-f130-abbd0c0ab2ca}"
2025-01-28T14:55:44.764Z [INFO]  server-955c351e-e622-2746-f130-abbd0c0ab2ca: pipelining replication: peer="{Voter server-7535ea26-0093-6cdc-5060-927a8bee1d14 7535ea26-0093-6cdc-5060-927a8bee1d14}"
2025-01-28T14:55:44.764Z [INFO]  server-955c351e-e622-2746-f130-abbd0c0ab2ca: pipelining replication: peer="{Voter server-b060373a-87ff-c769-2110-10dcff83778e b060373a-87ff-c769-2110-10dcff83778e}"
2025-01-28T14:55:44.869Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:44.76463604 +0000 UTC m=+130.439126020 (1 nodes), 242.897µs from start of poll, 265.510053ms from cluster start. Timeout at 2025-01-28 14:55:44.869761448 +0000 UTC m=+130.544251448, 105.125428ms after stability
2025-01-28T14:55:44.870Z [INFO]  server-955c351e-e622-2746-f130-abbd0c0ab2ca: aborting pipeline replication: peer="{Voter server-7535ea26-0093-6cdc-5060-927a8bee1d14 7535ea26-0093-6cdc-5060-927a8bee1d14}"
2025-01-28T14:55:44.870Z [INFO]  server-955c351e-e622-2746-f130-abbd0c0ab2ca: aborting pipeline replication: peer="{Voter server-b060373a-87ff-c769-2110-10dcff83778e b060373a-87ff-c769-2110-10dcff83778e}"
--- PASS: TestRaft_LeadershipTransfer (0.37s)
=== RUN   TestRaft_LeadershipTransferWithOneNode
2025-01-28T14:55:44.872Z [INFO]  server-aa009046-31a1-409c-2e42-174b9a155edb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-aa009046-31a1-409c-2e42-174b9a155edb Address:aa009046-31a1-409c-2e42-174b9a155edb}]"
2025-01-28T14:55:44.872Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:44.873Z [INFO]  server-aa009046-31a1-409c-2e42-174b9a155edb: entering follower state: follower="Node at aa009046-31a1-409c-2e42-174b9a155edb [Follower]" leader-address= leader-id=
2025-01-28T14:55:44.949Z [WARN]  server-aa009046-31a1-409c-2e42-174b9a155edb: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:44.949Z [INFO]  server-aa009046-31a1-409c-2e42-174b9a155edb: entering candidate state: node="Node at aa009046-31a1-409c-2e42-174b9a155edb [Candidate]" term=2
2025-01-28T14:55:44.949Z [INFO]  server-aa009046-31a1-409c-2e42-174b9a155edb: election won: term=2 tally=1
2025-01-28T14:55:44.949Z [INFO]  server-aa009046-31a1-409c-2e42-174b9a155edb: entering leader state: leader="Node at aa009046-31a1-409c-2e42-174b9a155edb [Leader]"
2025-01-28T14:55:45.055Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:44.949814771 +0000 UTC m=+130.624304731 (1 nodes), 76.826313ms from start of poll, 77.159261ms from cluster start. Timeout at 2025-01-28 14:55:45.054839628 +0000 UTC m=+130.729329688, 105.024957ms after stability
--- PASS: TestRaft_LeadershipTransferWithOneNode (0.19s)
=== RUN   TestRaft_LeadershipTransferWithSevenNodes
2025-01-28T14:55:45.069Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67f33648-437a-847e-a9cf-6bbd6b981188 Address:67f33648-437a-847e-a9cf-6bbd6b981188} {Suffrage:Voter ID:server-48509be3-7c81-5153-ded7-68466927abae Address:48509be3-7c81-5153-ded7-68466927abae} {Suffrage:Voter ID:server-f73da7ab-19fa-ec89-0c4f-7831a90fffdc Address:f73da7ab-19fa-ec89-0c4f-7831a90fffdc} {Suffrage:Voter ID:server-7b554427-bd99-fe04-c520-fd1722bcebdf Address:7b554427-bd99-fe04-c520-fd1722bcebdf} {Suffrage:Voter ID:server-56b0cb4f-a302-06ed-3363-683976f5efa9 Address:56b0cb4f-a302-06ed-3363-683976f5efa9} {Suffrage:Voter ID:server-5b1e3e51-dcf4-3845-5297-0841920f5ae9 Address:5b1e3e51-dcf4-3845-5297-0841920f5ae9} {Suffrage:Voter ID:server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e Address:e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e}]"
2025-01-28T14:55:45.069Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: entering follower state: follower="Node at 67f33648-437a-847e-a9cf-6bbd6b981188 [Follower]" leader-address= leader-id=
2025-01-28T14:55:45.069Z [INFO]  server-48509be3-7c81-5153-ded7-68466927abae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67f33648-437a-847e-a9cf-6bbd6b981188 Address:67f33648-437a-847e-a9cf-6bbd6b981188} {Suffrage:Voter ID:server-48509be3-7c81-5153-ded7-68466927abae Address:48509be3-7c81-5153-ded7-68466927abae} {Suffrage:Voter ID:server-f73da7ab-19fa-ec89-0c4f-7831a90fffdc Address:f73da7ab-19fa-ec89-0c4f-7831a90fffdc} {Suffrage:Voter ID:server-7b554427-bd99-fe04-c520-fd1722bcebdf Address:7b554427-bd99-fe04-c520-fd1722bcebdf} {Suffrage:Voter ID:server-56b0cb4f-a302-06ed-3363-683976f5efa9 Address:56b0cb4f-a302-06ed-3363-683976f5efa9} {Suffrage:Voter ID:server-5b1e3e51-dcf4-3845-5297-0841920f5ae9 Address:5b1e3e51-dcf4-3845-5297-0841920f5ae9} {Suffrage:Voter ID:server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e Address:e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e}]"
2025-01-28T14:55:45.069Z [INFO]  server-48509be3-7c81-5153-ded7-68466927abae: entering follower state: follower="Node at 48509be3-7c81-5153-ded7-68466927abae [Follower]" leader-address= leader-id=
2025-01-28T14:55:45.069Z [INFO]  server-f73da7ab-19fa-ec89-0c4f-7831a90fffdc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67f33648-437a-847e-a9cf-6bbd6b981188 Address:67f33648-437a-847e-a9cf-6bbd6b981188} {Suffrage:Voter ID:server-48509be3-7c81-5153-ded7-68466927abae Address:48509be3-7c81-5153-ded7-68466927abae} {Suffrage:Voter ID:server-f73da7ab-19fa-ec89-0c4f-7831a90fffdc Address:f73da7ab-19fa-ec89-0c4f-7831a90fffdc} {Suffrage:Voter ID:server-7b554427-bd99-fe04-c520-fd1722bcebdf Address:7b554427-bd99-fe04-c520-fd1722bcebdf} {Suffrage:Voter ID:server-56b0cb4f-a302-06ed-3363-683976f5efa9 Address:56b0cb4f-a302-06ed-3363-683976f5efa9} {Suffrage:Voter ID:server-5b1e3e51-dcf4-3845-5297-0841920f5ae9 Address:5b1e3e51-dcf4-3845-5297-0841920f5ae9} {Suffrage:Voter ID:server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e Address:e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e}]"
2025-01-28T14:55:45.070Z [INFO]  server-f73da7ab-19fa-ec89-0c4f-7831a90fffdc: entering follower state: follower="Node at f73da7ab-19fa-ec89-0c4f-7831a90fffdc [Follower]" leader-address= leader-id=
2025-01-28T14:55:45.070Z [INFO]  server-7b554427-bd99-fe04-c520-fd1722bcebdf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67f33648-437a-847e-a9cf-6bbd6b981188 Address:67f33648-437a-847e-a9cf-6bbd6b981188} {Suffrage:Voter ID:server-48509be3-7c81-5153-ded7-68466927abae Address:48509be3-7c81-5153-ded7-68466927abae} {Suffrage:Voter ID:server-f73da7ab-19fa-ec89-0c4f-7831a90fffdc Address:f73da7ab-19fa-ec89-0c4f-7831a90fffdc} {Suffrage:Voter ID:server-7b554427-bd99-fe04-c520-fd1722bcebdf Address:7b554427-bd99-fe04-c520-fd1722bcebdf} {Suffrage:Voter ID:server-56b0cb4f-a302-06ed-3363-683976f5efa9 Address:56b0cb4f-a302-06ed-3363-683976f5efa9} {Suffrage:Voter ID:server-5b1e3e51-dcf4-3845-5297-0841920f5ae9 Address:5b1e3e51-dcf4-3845-5297-0841920f5ae9} {Suffrage:Voter ID:server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e Address:e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e}]"
2025-01-28T14:55:45.070Z [INFO]  server-7b554427-bd99-fe04-c520-fd1722bcebdf: entering follower state: follower="Node at 7b554427-bd99-fe04-c520-fd1722bcebdf [Follower]" leader-address= leader-id=
2025-01-28T14:55:45.070Z [INFO]  server-56b0cb4f-a302-06ed-3363-683976f5efa9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67f33648-437a-847e-a9cf-6bbd6b981188 Address:67f33648-437a-847e-a9cf-6bbd6b981188} {Suffrage:Voter ID:server-48509be3-7c81-5153-ded7-68466927abae Address:48509be3-7c81-5153-ded7-68466927abae} {Suffrage:Voter ID:server-f73da7ab-19fa-ec89-0c4f-7831a90fffdc Address:f73da7ab-19fa-ec89-0c4f-7831a90fffdc} {Suffrage:Voter ID:server-7b554427-bd99-fe04-c520-fd1722bcebdf Address:7b554427-bd99-fe04-c520-fd1722bcebdf} {Suffrage:Voter ID:server-56b0cb4f-a302-06ed-3363-683976f5efa9 Address:56b0cb4f-a302-06ed-3363-683976f5efa9} {Suffrage:Voter ID:server-5b1e3e51-dcf4-3845-5297-0841920f5ae9 Address:5b1e3e51-dcf4-3845-5297-0841920f5ae9} {Suffrage:Voter ID:server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e Address:e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e}]"
2025-01-28T14:55:45.070Z [INFO]  server-56b0cb4f-a302-06ed-3363-683976f5efa9: entering follower state: follower="Node at 56b0cb4f-a302-06ed-3363-683976f5efa9 [Follower]" leader-address= leader-id=
2025-01-28T14:55:45.070Z [INFO]  server-5b1e3e51-dcf4-3845-5297-0841920f5ae9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67f33648-437a-847e-a9cf-6bbd6b981188 Address:67f33648-437a-847e-a9cf-6bbd6b981188} {Suffrage:Voter ID:server-48509be3-7c81-5153-ded7-68466927abae Address:48509be3-7c81-5153-ded7-68466927abae} {Suffrage:Voter ID:server-f73da7ab-19fa-ec89-0c4f-7831a90fffdc Address:f73da7ab-19fa-ec89-0c4f-7831a90fffdc} {Suffrage:Voter ID:server-7b554427-bd99-fe04-c520-fd1722bcebdf Address:7b554427-bd99-fe04-c520-fd1722bcebdf} {Suffrage:Voter ID:server-56b0cb4f-a302-06ed-3363-683976f5efa9 Address:56b0cb4f-a302-06ed-3363-683976f5efa9} {Suffrage:Voter ID:server-5b1e3e51-dcf4-3845-5297-0841920f5ae9 Address:5b1e3e51-dcf4-3845-5297-0841920f5ae9} {Suffrage:Voter ID:server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e Address:e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e}]"
2025-01-28T14:55:45.071Z [INFO]  server-5b1e3e51-dcf4-3845-5297-0841920f5ae9: entering follower state: follower="Node at 5b1e3e51-dcf4-3845-5297-0841920f5ae9 [Follower]" leader-address= leader-id=
2025-01-28T14:55:45.071Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-67f33648-437a-847e-a9cf-6bbd6b981188 Address:67f33648-437a-847e-a9cf-6bbd6b981188} {Suffrage:Voter ID:server-48509be3-7c81-5153-ded7-68466927abae Address:48509be3-7c81-5153-ded7-68466927abae} {Suffrage:Voter ID:server-f73da7ab-19fa-ec89-0c4f-7831a90fffdc Address:f73da7ab-19fa-ec89-0c4f-7831a90fffdc} {Suffrage:Voter ID:server-7b554427-bd99-fe04-c520-fd1722bcebdf Address:7b554427-bd99-fe04-c520-fd1722bcebdf} {Suffrage:Voter ID:server-56b0cb4f-a302-06ed-3363-683976f5efa9 Address:56b0cb4f-a302-06ed-3363-683976f5efa9} {Suffrage:Voter ID:server-5b1e3e51-dcf4-3845-5297-0841920f5ae9 Address:5b1e3e51-dcf4-3845-5297-0841920f5ae9} {Suffrage:Voter ID:server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e Address:e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e}]"
2025-01-28T14:55:45.071Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:45.071Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: entering follower state: follower="Node at e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e [Follower]" leader-address= leader-id=
2025-01-28T14:55:45.130Z [WARN]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:45.130Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: entering candidate state: node="Node at e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e [Candidate]" term=2
2025-01-28T14:55:45.130Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: election won: term=2 tally=4
2025-01-28T14:55:45.130Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: entering leader state: leader="Node at e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e [Leader]"
2025-01-28T14:55:45.130Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: added peer, starting replication: peer=server-67f33648-437a-847e-a9cf-6bbd6b981188
2025-01-28T14:55:45.130Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: added peer, starting replication: peer=server-48509be3-7c81-5153-ded7-68466927abae
2025-01-28T14:55:45.130Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: added peer, starting replication: peer=server-f73da7ab-19fa-ec89-0c4f-7831a90fffdc
2025-01-28T14:55:45.130Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: added peer, starting replication: peer=server-7b554427-bd99-fe04-c520-fd1722bcebdf
2025-01-28T14:55:45.130Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: added peer, starting replication: peer=server-56b0cb4f-a302-06ed-3363-683976f5efa9
2025-01-28T14:55:45.130Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: added peer, starting replication: peer=server-5b1e3e51-dcf4-3845-5297-0841920f5ae9
2025-01-28T14:55:45.131Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: pipelining replication: peer="{Voter server-67f33648-437a-847e-a9cf-6bbd6b981188 67f33648-437a-847e-a9cf-6bbd6b981188}"
2025-01-28T14:55:45.131Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: pipelining replication: peer="{Voter server-f73da7ab-19fa-ec89-0c4f-7831a90fffdc f73da7ab-19fa-ec89-0c4f-7831a90fffdc}"
2025-01-28T14:55:45.131Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: pipelining replication: peer="{Voter server-5b1e3e51-dcf4-3845-5297-0841920f5ae9 5b1e3e51-dcf4-3845-5297-0841920f5ae9}"
2025-01-28T14:55:45.131Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: pipelining replication: peer="{Voter server-7b554427-bd99-fe04-c520-fd1722bcebdf 7b554427-bd99-fe04-c520-fd1722bcebdf}"
2025-01-28T14:55:45.131Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: pipelining replication: peer="{Voter server-48509be3-7c81-5153-ded7-68466927abae 48509be3-7c81-5153-ded7-68466927abae}"
2025-01-28T14:55:45.131Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: pipelining replication: peer="{Voter server-56b0cb4f-a302-06ed-3363-683976f5efa9 56b0cb4f-a302-06ed-3363-683976f5efa9}"
2025-01-28T14:55:45.236Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:45.130735044 +0000 UTC m=+130.805225064 (1 nodes), 59.298646ms from start of poll, 61.891102ms from cluster start. Timeout at 2025-01-28 14:55:45.235754322 +0000 UTC m=+130.910244302, 105.019238ms after stability
2025-01-28T14:55:45.236Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:45.342Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:45.236650592 +0000 UTC m=+130.911140592 (6 nodes), 3.26µs from start of poll, 167.80663ms from cluster start. Timeout at 2025-01-28 14:55:45.341802738 +0000 UTC m=+131.016292738, 105.152146ms after stability
2025-01-28T14:55:45.342Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:45.447Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:45.342272508 +0000 UTC m=+131.016762488 (1 nodes), 2.22µs from start of poll, 273.428526ms from cluster start. Timeout at 2025-01-28 14:55:45.44746263 +0000 UTC m=+131.121952630, 105.190142ms after stability
2025-01-28T14:55:45.447Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: entering candidate state: node="Node at 67f33648-437a-847e-a9cf-6bbd6b981188 [Candidate]" term=3
2025-01-28T14:55:45.447Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:45.448Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: entering follower state: follower="Node at e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e [Follower]" leader-address= leader-id=
2025-01-28T14:55:45.448Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: election won: term=3 tally=4
2025-01-28T14:55:45.448Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: aborting pipeline replication: peer="{Voter server-7b554427-bd99-fe04-c520-fd1722bcebdf 7b554427-bd99-fe04-c520-fd1722bcebdf}"
2025-01-28T14:55:45.448Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: entering leader state: leader="Node at 67f33648-437a-847e-a9cf-6bbd6b981188 [Leader]"
2025-01-28T14:55:45.448Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: aborting pipeline replication: peer="{Voter server-67f33648-437a-847e-a9cf-6bbd6b981188 67f33648-437a-847e-a9cf-6bbd6b981188}"
2025-01-28T14:55:45.448Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: added peer, starting replication: peer=server-48509be3-7c81-5153-ded7-68466927abae
2025-01-28T14:55:45.448Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: added peer, starting replication: peer=server-f73da7ab-19fa-ec89-0c4f-7831a90fffdc
2025-01-28T14:55:45.448Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: aborting pipeline replication: peer="{Voter server-56b0cb4f-a302-06ed-3363-683976f5efa9 56b0cb4f-a302-06ed-3363-683976f5efa9}"
2025-01-28T14:55:45.448Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: added peer, starting replication: peer=server-7b554427-bd99-fe04-c520-fd1722bcebdf
2025-01-28T14:55:45.448Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: aborting pipeline replication: peer="{Voter server-48509be3-7c81-5153-ded7-68466927abae 48509be3-7c81-5153-ded7-68466927abae}"
2025-01-28T14:55:45.448Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: added peer, starting replication: peer=server-56b0cb4f-a302-06ed-3363-683976f5efa9
2025-01-28T14:55:45.448Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: added peer, starting replication: peer=server-5b1e3e51-dcf4-3845-5297-0841920f5ae9
2025-01-28T14:55:45.448Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: added peer, starting replication: peer=server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e
2025-01-28T14:55:45.448Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: pipelining replication: peer="{Voter server-56b0cb4f-a302-06ed-3363-683976f5efa9 56b0cb4f-a302-06ed-3363-683976f5efa9}"
2025-01-28T14:55:45.448Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: aborting pipeline replication: peer="{Voter server-5b1e3e51-dcf4-3845-5297-0841920f5ae9 5b1e3e51-dcf4-3845-5297-0841920f5ae9}"
2025-01-28T14:55:45.448Z [INFO]  server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e: aborting pipeline replication: peer="{Voter server-f73da7ab-19fa-ec89-0c4f-7831a90fffdc f73da7ab-19fa-ec89-0c4f-7831a90fffdc}"
2025-01-28T14:55:45.448Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: pipelining replication: peer="{Voter server-f73da7ab-19fa-ec89-0c4f-7831a90fffdc f73da7ab-19fa-ec89-0c4f-7831a90fffdc}"
2025-01-28T14:55:45.448Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: pipelining replication: peer="{Voter server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e}"
2025-01-28T14:55:45.448Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: pipelining replication: peer="{Voter server-48509be3-7c81-5153-ded7-68466927abae 48509be3-7c81-5153-ded7-68466927abae}"
2025-01-28T14:55:45.448Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: pipelining replication: peer="{Voter server-7b554427-bd99-fe04-c520-fd1722bcebdf 7b554427-bd99-fe04-c520-fd1722bcebdf}"
2025-01-28T14:55:45.448Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: pipelining replication: peer="{Voter server-5b1e3e51-dcf4-3845-5297-0841920f5ae9 5b1e3e51-dcf4-3845-5297-0841920f5ae9}"
2025-01-28T14:55:45.553Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:45.448345142 +0000 UTC m=+131.122835122 (1 nodes), 278.134µs from start of poll, 379.50116ms from cluster start. Timeout at 2025-01-28 14:55:45.553349042 +0000 UTC m=+131.227839042, 105.00392ms after stability
2025-01-28T14:55:45.553Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: aborting pipeline replication: peer="{Voter server-56b0cb4f-a302-06ed-3363-683976f5efa9 56b0cb4f-a302-06ed-3363-683976f5efa9}"
2025-01-28T14:55:45.553Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: aborting pipeline replication: peer="{Voter server-5b1e3e51-dcf4-3845-5297-0841920f5ae9 5b1e3e51-dcf4-3845-5297-0841920f5ae9}"
2025-01-28T14:55:45.553Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: aborting pipeline replication: peer="{Voter server-f73da7ab-19fa-ec89-0c4f-7831a90fffdc f73da7ab-19fa-ec89-0c4f-7831a90fffdc}"
2025-01-28T14:55:45.553Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: aborting pipeline replication: peer="{Voter server-e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e e9bd0c8f-47f0-0d6f-d302-92b1caf70f8e}"
2025-01-28T14:55:45.553Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: aborting pipeline replication: peer="{Voter server-7b554427-bd99-fe04-c520-fd1722bcebdf 7b554427-bd99-fe04-c520-fd1722bcebdf}"
2025-01-28T14:55:45.553Z [INFO]  server-67f33648-437a-847e-a9cf-6bbd6b981188: aborting pipeline replication: peer="{Voter server-48509be3-7c81-5153-ded7-68466927abae 48509be3-7c81-5153-ded7-68466927abae}"
--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.49s)
=== RUN   TestRaft_LeadershipTransferToInvalidID
2025-01-28T14:55:45.559Z [INFO]  server-6d2a0a7b-da66-3929-6249-abc675f7b1f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d2a0a7b-da66-3929-6249-abc675f7b1f0 Address:6d2a0a7b-da66-3929-6249-abc675f7b1f0} {Suffrage:Voter ID:server-4800695e-99c8-aed5-49e7-1a183d4648f9 Address:4800695e-99c8-aed5-49e7-1a183d4648f9} {Suffrage:Voter ID:server-a901fd98-466c-c269-31f1-d8806e950d4c Address:a901fd98-466c-c269-31f1-d8806e950d4c}]"
2025-01-28T14:55:45.560Z [INFO]  server-6d2a0a7b-da66-3929-6249-abc675f7b1f0: entering follower state: follower="Node at 6d2a0a7b-da66-3929-6249-abc675f7b1f0 [Follower]" leader-address= leader-id=
2025-01-28T14:55:45.560Z [INFO]  server-4800695e-99c8-aed5-49e7-1a183d4648f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d2a0a7b-da66-3929-6249-abc675f7b1f0 Address:6d2a0a7b-da66-3929-6249-abc675f7b1f0} {Suffrage:Voter ID:server-4800695e-99c8-aed5-49e7-1a183d4648f9 Address:4800695e-99c8-aed5-49e7-1a183d4648f9} {Suffrage:Voter ID:server-a901fd98-466c-c269-31f1-d8806e950d4c Address:a901fd98-466c-c269-31f1-d8806e950d4c}]"
2025-01-28T14:55:45.560Z [INFO]  server-4800695e-99c8-aed5-49e7-1a183d4648f9: entering follower state: follower="Node at 4800695e-99c8-aed5-49e7-1a183d4648f9 [Follower]" leader-address= leader-id=
2025-01-28T14:55:45.560Z [INFO]  server-a901fd98-466c-c269-31f1-d8806e950d4c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d2a0a7b-da66-3929-6249-abc675f7b1f0 Address:6d2a0a7b-da66-3929-6249-abc675f7b1f0} {Suffrage:Voter ID:server-4800695e-99c8-aed5-49e7-1a183d4648f9 Address:4800695e-99c8-aed5-49e7-1a183d4648f9} {Suffrage:Voter ID:server-a901fd98-466c-c269-31f1-d8806e950d4c Address:a901fd98-466c-c269-31f1-d8806e950d4c}]"
2025-01-28T14:55:45.560Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:45.560Z [INFO]  server-a901fd98-466c-c269-31f1-d8806e950d4c: entering follower state: follower="Node at a901fd98-466c-c269-31f1-d8806e950d4c [Follower]" leader-address= leader-id=
2025-01-28T14:55:45.612Z [WARN]  server-4800695e-99c8-aed5-49e7-1a183d4648f9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:45.612Z [INFO]  server-4800695e-99c8-aed5-49e7-1a183d4648f9: entering candidate state: node="Node at 4800695e-99c8-aed5-49e7-1a183d4648f9 [Candidate]" term=2
2025-01-28T14:55:45.612Z [INFO]  server-4800695e-99c8-aed5-49e7-1a183d4648f9: election won: term=2 tally=2
2025-01-28T14:55:45.612Z [INFO]  server-4800695e-99c8-aed5-49e7-1a183d4648f9: entering leader state: leader="Node at 4800695e-99c8-aed5-49e7-1a183d4648f9 [Leader]"
2025-01-28T14:55:45.612Z [INFO]  server-4800695e-99c8-aed5-49e7-1a183d4648f9: added peer, starting replication: peer=server-6d2a0a7b-da66-3929-6249-abc675f7b1f0
2025-01-28T14:55:45.612Z [INFO]  server-4800695e-99c8-aed5-49e7-1a183d4648f9: added peer, starting replication: peer=server-a901fd98-466c-c269-31f1-d8806e950d4c
2025-01-28T14:55:45.612Z [INFO]  server-4800695e-99c8-aed5-49e7-1a183d4648f9: pipelining replication: peer="{Voter server-a901fd98-466c-c269-31f1-d8806e950d4c a901fd98-466c-c269-31f1-d8806e950d4c}"
2025-01-28T14:55:45.612Z [INFO]  server-4800695e-99c8-aed5-49e7-1a183d4648f9: pipelining replication: peer="{Voter server-6d2a0a7b-da66-3929-6249-abc675f7b1f0 6d2a0a7b-da66-3929-6249-abc675f7b1f0}"
2025-01-28T14:55:45.718Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:45.612416104 +0000 UTC m=+131.286906064 (1 nodes), 51.680725ms from start of poll, 52.68519ms from cluster start. Timeout at 2025-01-28 14:55:45.717420284 +0000 UTC m=+131.391910304, 105.00424ms after stability
2025-01-28T14:55:45.718Z [INFO]  server-4800695e-99c8-aed5-49e7-1a183d4648f9: aborting pipeline replication: peer="{Voter server-a901fd98-466c-c269-31f1-d8806e950d4c a901fd98-466c-c269-31f1-d8806e950d4c}"
2025-01-28T14:55:45.718Z [INFO]  server-4800695e-99c8-aed5-49e7-1a183d4648f9: aborting pipeline replication: peer="{Voter server-6d2a0a7b-da66-3929-6249-abc675f7b1f0 6d2a0a7b-da66-3929-6249-abc675f7b1f0}"
--- PASS: TestRaft_LeadershipTransferToInvalidID (0.16s)
=== RUN   TestRaft_LeadershipTransferToInvalidAddress
2025-01-28T14:55:45.722Z [INFO]  server-ee87fc46-708f-2642-6a7b-fc1ad5437880: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee87fc46-708f-2642-6a7b-fc1ad5437880 Address:ee87fc46-708f-2642-6a7b-fc1ad5437880} {Suffrage:Voter ID:server-f79ba009-bdf9-08dc-ba05-2dcd422dc759 Address:f79ba009-bdf9-08dc-ba05-2dcd422dc759} {Suffrage:Voter ID:server-df93ffd3-70da-146c-c89c-579662282942 Address:df93ffd3-70da-146c-c89c-579662282942}]"
2025-01-28T14:55:45.723Z [INFO]  server-ee87fc46-708f-2642-6a7b-fc1ad5437880: entering follower state: follower="Node at ee87fc46-708f-2642-6a7b-fc1ad5437880 [Follower]" leader-address= leader-id=
2025-01-28T14:55:45.723Z [INFO]  server-f79ba009-bdf9-08dc-ba05-2dcd422dc759: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee87fc46-708f-2642-6a7b-fc1ad5437880 Address:ee87fc46-708f-2642-6a7b-fc1ad5437880} {Suffrage:Voter ID:server-f79ba009-bdf9-08dc-ba05-2dcd422dc759 Address:f79ba009-bdf9-08dc-ba05-2dcd422dc759} {Suffrage:Voter ID:server-df93ffd3-70da-146c-c89c-579662282942 Address:df93ffd3-70da-146c-c89c-579662282942}]"
2025-01-28T14:55:45.723Z [INFO]  server-f79ba009-bdf9-08dc-ba05-2dcd422dc759: entering follower state: follower="Node at f79ba009-bdf9-08dc-ba05-2dcd422dc759 [Follower]" leader-address= leader-id=
2025-01-28T14:55:45.723Z [INFO]  server-df93ffd3-70da-146c-c89c-579662282942: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee87fc46-708f-2642-6a7b-fc1ad5437880 Address:ee87fc46-708f-2642-6a7b-fc1ad5437880} {Suffrage:Voter ID:server-f79ba009-bdf9-08dc-ba05-2dcd422dc759 Address:f79ba009-bdf9-08dc-ba05-2dcd422dc759} {Suffrage:Voter ID:server-df93ffd3-70da-146c-c89c-579662282942 Address:df93ffd3-70da-146c-c89c-579662282942}]"
2025-01-28T14:55:45.723Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:45.723Z [INFO]  server-df93ffd3-70da-146c-c89c-579662282942: entering follower state: follower="Node at df93ffd3-70da-146c-c89c-579662282942 [Follower]" leader-address= leader-id=
2025-01-28T14:55:45.796Z [WARN]  server-df93ffd3-70da-146c-c89c-579662282942: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:45.796Z [INFO]  server-df93ffd3-70da-146c-c89c-579662282942: entering candidate state: node="Node at df93ffd3-70da-146c-c89c-579662282942 [Candidate]" term=2
2025-01-28T14:55:45.796Z [INFO]  server-df93ffd3-70da-146c-c89c-579662282942: election won: term=2 tally=2
2025-01-28T14:55:45.796Z [INFO]  server-df93ffd3-70da-146c-c89c-579662282942: entering leader state: leader="Node at df93ffd3-70da-146c-c89c-579662282942 [Leader]"
2025-01-28T14:55:45.796Z [INFO]  server-df93ffd3-70da-146c-c89c-579662282942: added peer, starting replication: peer=server-ee87fc46-708f-2642-6a7b-fc1ad5437880
2025-01-28T14:55:45.796Z [INFO]  server-df93ffd3-70da-146c-c89c-579662282942: added peer, starting replication: peer=server-f79ba009-bdf9-08dc-ba05-2dcd422dc759
2025-01-28T14:55:45.796Z [INFO]  server-df93ffd3-70da-146c-c89c-579662282942: pipelining replication: peer="{Voter server-ee87fc46-708f-2642-6a7b-fc1ad5437880 ee87fc46-708f-2642-6a7b-fc1ad5437880}"
2025-01-28T14:55:45.797Z [INFO]  server-df93ffd3-70da-146c-c89c-579662282942: pipelining replication: peer="{Voter server-f79ba009-bdf9-08dc-ba05-2dcd422dc759 f79ba009-bdf9-08dc-ba05-2dcd422dc759}"
2025-01-28T14:55:45.902Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:45.79648412 +0000 UTC m=+131.470974100 (2 nodes), 72.786894ms from start of poll, 73.898569ms from cluster start. Timeout at 2025-01-28 14:55:45.90148868 +0000 UTC m=+131.575978720, 105.00462ms after stability
2025-01-28T14:55:45.902Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:46.009Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:45.902975294 +0000 UTC m=+131.577465274 (1 nodes), 2.7µs from start of poll, 180.389743ms from cluster start. Timeout at 2025-01-28 14:55:46.008094662 +0000 UTC m=+131.682584682, 105.119408ms after stability
2025-01-28T14:55:46.009Z [INFO]  server-df93ffd3-70da-146c-c89c-579662282942: aborting pipeline replication: peer="{Voter server-f79ba009-bdf9-08dc-ba05-2dcd422dc759 f79ba009-bdf9-08dc-ba05-2dcd422dc759}"
2025-01-28T14:55:46.009Z [INFO]  server-df93ffd3-70da-146c-c89c-579662282942: aborting pipeline replication: peer="{Voter server-ee87fc46-708f-2642-6a7b-fc1ad5437880 ee87fc46-708f-2642-6a7b-fc1ad5437880}"
--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.29s)
=== RUN   TestRaft_LeadershipTransferToBehindServer
2025-01-28T14:55:46.013Z [INFO]  server-351f5591-9548-908e-f887-eed3eb778e21: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-351f5591-9548-908e-f887-eed3eb778e21 Address:351f5591-9548-908e-f887-eed3eb778e21} {Suffrage:Voter ID:server-5bdb543d-ac5a-db90-a73e-36e36a7d7c6d Address:5bdb543d-ac5a-db90-a73e-36e36a7d7c6d} {Suffrage:Voter ID:server-a37e61c3-cf06-f465-d155-cad27c2b1fd9 Address:a37e61c3-cf06-f465-d155-cad27c2b1fd9}]"
2025-01-28T14:55:46.013Z [INFO]  server-351f5591-9548-908e-f887-eed3eb778e21: entering follower state: follower="Node at 351f5591-9548-908e-f887-eed3eb778e21 [Follower]" leader-address= leader-id=
2025-01-28T14:55:46.013Z [INFO]  server-5bdb543d-ac5a-db90-a73e-36e36a7d7c6d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-351f5591-9548-908e-f887-eed3eb778e21 Address:351f5591-9548-908e-f887-eed3eb778e21} {Suffrage:Voter ID:server-5bdb543d-ac5a-db90-a73e-36e36a7d7c6d Address:5bdb543d-ac5a-db90-a73e-36e36a7d7c6d} {Suffrage:Voter ID:server-a37e61c3-cf06-f465-d155-cad27c2b1fd9 Address:a37e61c3-cf06-f465-d155-cad27c2b1fd9}]"
2025-01-28T14:55:46.013Z [INFO]  server-5bdb543d-ac5a-db90-a73e-36e36a7d7c6d: entering follower state: follower="Node at 5bdb543d-ac5a-db90-a73e-36e36a7d7c6d [Follower]" leader-address= leader-id=
2025-01-28T14:55:46.014Z [INFO]  server-a37e61c3-cf06-f465-d155-cad27c2b1fd9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-351f5591-9548-908e-f887-eed3eb778e21 Address:351f5591-9548-908e-f887-eed3eb778e21} {Suffrage:Voter ID:server-5bdb543d-ac5a-db90-a73e-36e36a7d7c6d Address:5bdb543d-ac5a-db90-a73e-36e36a7d7c6d} {Suffrage:Voter ID:server-a37e61c3-cf06-f465-d155-cad27c2b1fd9 Address:a37e61c3-cf06-f465-d155-cad27c2b1fd9}]"
2025-01-28T14:55:46.014Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:46.014Z [INFO]  server-a37e61c3-cf06-f465-d155-cad27c2b1fd9: entering follower state: follower="Node at a37e61c3-cf06-f465-d155-cad27c2b1fd9 [Follower]" leader-address= leader-id=
2025-01-28T14:55:46.070Z [WARN]  server-351f5591-9548-908e-f887-eed3eb778e21: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:46.070Z [INFO]  server-351f5591-9548-908e-f887-eed3eb778e21: entering candidate state: node="Node at 351f5591-9548-908e-f887-eed3eb778e21 [Candidate]" term=2
2025-01-28T14:55:46.071Z [INFO]  server-351f5591-9548-908e-f887-eed3eb778e21: election won: term=2 tally=2
2025-01-28T14:55:46.071Z [INFO]  server-351f5591-9548-908e-f887-eed3eb778e21: entering leader state: leader="Node at 351f5591-9548-908e-f887-eed3eb778e21 [Leader]"
2025-01-28T14:55:46.071Z [INFO]  server-351f5591-9548-908e-f887-eed3eb778e21: added peer, starting replication: peer=server-5bdb543d-ac5a-db90-a73e-36e36a7d7c6d
2025-01-28T14:55:46.071Z [INFO]  server-351f5591-9548-908e-f887-eed3eb778e21: added peer, starting replication: peer=server-a37e61c3-cf06-f465-d155-cad27c2b1fd9
2025-01-28T14:55:46.071Z [INFO]  server-351f5591-9548-908e-f887-eed3eb778e21: pipelining replication: peer="{Voter server-5bdb543d-ac5a-db90-a73e-36e36a7d7c6d 5bdb543d-ac5a-db90-a73e-36e36a7d7c6d}"
2025-01-28T14:55:46.071Z [INFO]  server-351f5591-9548-908e-f887-eed3eb778e21: pipelining replication: peer="{Voter server-a37e61c3-cf06-f465-d155-cad27c2b1fd9 a37e61c3-cf06-f465-d155-cad27c2b1fd9}"
2025-01-28T14:55:46.176Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:46.071263778 +0000 UTC m=+131.745753758 (1 nodes), 56.976305ms from start of poll, 58.113197ms from cluster start. Timeout at 2025-01-28 14:55:46.176310354 +0000 UTC m=+131.850800334, 105.046576ms after stability
2025-01-28T14:55:46.176Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:46.282Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:46.176536627 +0000 UTC m=+131.851026627 (2 nodes), 2.779µs from start of poll, 163.386066ms from cluster start. Timeout at 2025-01-28 14:55:46.28172025 +0000 UTC m=+131.956210230, 105.183603ms after stability
2025-01-28T14:55:46.385Z [INFO]  server-5bdb543d-ac5a-db90-a73e-36e36a7d7c6d: entering candidate state: node="Node at 5bdb543d-ac5a-db90-a73e-36e36a7d7c6d [Candidate]" term=3
2025-01-28T14:55:46.385Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:46.385Z [INFO]  server-5bdb543d-ac5a-db90-a73e-36e36a7d7c6d: election won: term=3 tally=2
2025-01-28T14:55:46.385Z [INFO]  server-5bdb543d-ac5a-db90-a73e-36e36a7d7c6d: entering leader state: leader="Node at 5bdb543d-ac5a-db90-a73e-36e36a7d7c6d [Leader]"
2025-01-28T14:55:46.385Z [INFO]  server-5bdb543d-ac5a-db90-a73e-36e36a7d7c6d: added peer, starting replication: peer=server-351f5591-9548-908e-f887-eed3eb778e21
2025-01-28T14:55:46.385Z [INFO]  server-5bdb543d-ac5a-db90-a73e-36e36a7d7c6d: added peer, starting replication: peer=server-a37e61c3-cf06-f465-d155-cad27c2b1fd9
2025-01-28T14:55:46.385Z [INFO]  server-351f5591-9548-908e-f887-eed3eb778e21: entering follower state: follower="Node at 351f5591-9548-908e-f887-eed3eb778e21 [Follower]" leader-address= leader-id=
2025-01-28T14:55:46.385Z [INFO]  server-351f5591-9548-908e-f887-eed3eb778e21: aborting pipeline replication: peer="{Voter server-5bdb543d-ac5a-db90-a73e-36e36a7d7c6d 5bdb543d-ac5a-db90-a73e-36e36a7d7c6d}"
2025-01-28T14:55:46.385Z [INFO]  server-351f5591-9548-908e-f887-eed3eb778e21: aborting pipeline replication: peer="{Voter server-a37e61c3-cf06-f465-d155-cad27c2b1fd9 a37e61c3-cf06-f465-d155-cad27c2b1fd9}"
2025-01-28T14:55:46.385Z [INFO]  server-5bdb543d-ac5a-db90-a73e-36e36a7d7c6d: pipelining replication: peer="{Voter server-a37e61c3-cf06-f465-d155-cad27c2b1fd9 a37e61c3-cf06-f465-d155-cad27c2b1fd9}"
2025-01-28T14:55:46.386Z [INFO]  server-5bdb543d-ac5a-db90-a73e-36e36a7d7c6d: pipelining replication: peer="{Voter server-351f5591-9548-908e-f887-eed3eb778e21 351f5591-9548-908e-f887-eed3eb778e21}"
2025-01-28T14:55:46.491Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:46.38586748 +0000 UTC m=+132.060357460 (1 nodes), 334.028µs from start of poll, 372.716899ms from cluster start. Timeout at 2025-01-28 14:55:46.491141914 +0000 UTC m=+132.165631894, 105.274434ms after stability
2025-01-28T14:55:46.491Z [INFO]  server-5bdb543d-ac5a-db90-a73e-36e36a7d7c6d: aborting pipeline replication: peer="{Voter server-a37e61c3-cf06-f465-d155-cad27c2b1fd9 a37e61c3-cf06-f465-d155-cad27c2b1fd9}"
2025-01-28T14:55:46.491Z [INFO]  server-5bdb543d-ac5a-db90-a73e-36e36a7d7c6d: aborting pipeline replication: peer="{Voter server-351f5591-9548-908e-f887-eed3eb778e21 351f5591-9548-908e-f887-eed3eb778e21}"
--- PASS: TestRaft_LeadershipTransferToBehindServer (0.48s)
=== RUN   TestRaft_LeadershipTransferToItself
2025-01-28T14:55:46.495Z [INFO]  server-4ec4f3e4-f559-0bcb-677d-e0afc991249e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ec4f3e4-f559-0bcb-677d-e0afc991249e Address:4ec4f3e4-f559-0bcb-677d-e0afc991249e} {Suffrage:Voter ID:server-7612cd24-58bd-6876-f82b-22544ea61fc3 Address:7612cd24-58bd-6876-f82b-22544ea61fc3} {Suffrage:Voter ID:server-cc064f2c-0843-57fd-a345-8ef186b2cba1 Address:cc064f2c-0843-57fd-a345-8ef186b2cba1}]"
2025-01-28T14:55:46.495Z [INFO]  server-4ec4f3e4-f559-0bcb-677d-e0afc991249e: entering follower state: follower="Node at 4ec4f3e4-f559-0bcb-677d-e0afc991249e [Follower]" leader-address= leader-id=
2025-01-28T14:55:46.495Z [INFO]  server-7612cd24-58bd-6876-f82b-22544ea61fc3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ec4f3e4-f559-0bcb-677d-e0afc991249e Address:4ec4f3e4-f559-0bcb-677d-e0afc991249e} {Suffrage:Voter ID:server-7612cd24-58bd-6876-f82b-22544ea61fc3 Address:7612cd24-58bd-6876-f82b-22544ea61fc3} {Suffrage:Voter ID:server-cc064f2c-0843-57fd-a345-8ef186b2cba1 Address:cc064f2c-0843-57fd-a345-8ef186b2cba1}]"
2025-01-28T14:55:46.495Z [INFO]  server-7612cd24-58bd-6876-f82b-22544ea61fc3: entering follower state: follower="Node at 7612cd24-58bd-6876-f82b-22544ea61fc3 [Follower]" leader-address= leader-id=
2025-01-28T14:55:46.496Z [INFO]  server-cc064f2c-0843-57fd-a345-8ef186b2cba1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ec4f3e4-f559-0bcb-677d-e0afc991249e Address:4ec4f3e4-f559-0bcb-677d-e0afc991249e} {Suffrage:Voter ID:server-7612cd24-58bd-6876-f82b-22544ea61fc3 Address:7612cd24-58bd-6876-f82b-22544ea61fc3} {Suffrage:Voter ID:server-cc064f2c-0843-57fd-a345-8ef186b2cba1 Address:cc064f2c-0843-57fd-a345-8ef186b2cba1}]"
2025-01-28T14:55:46.496Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:46.496Z [INFO]  server-cc064f2c-0843-57fd-a345-8ef186b2cba1: entering follower state: follower="Node at cc064f2c-0843-57fd-a345-8ef186b2cba1 [Follower]" leader-address= leader-id=
2025-01-28T14:55:46.548Z [WARN]  server-4ec4f3e4-f559-0bcb-677d-e0afc991249e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:46.548Z [INFO]  server-4ec4f3e4-f559-0bcb-677d-e0afc991249e: entering candidate state: node="Node at 4ec4f3e4-f559-0bcb-677d-e0afc991249e [Candidate]" term=2
2025-01-28T14:55:46.549Z [INFO]  server-4ec4f3e4-f559-0bcb-677d-e0afc991249e: election won: term=2 tally=2
2025-01-28T14:55:46.549Z [INFO]  server-4ec4f3e4-f559-0bcb-677d-e0afc991249e: entering leader state: leader="Node at 4ec4f3e4-f559-0bcb-677d-e0afc991249e [Leader]"
2025-01-28T14:55:46.549Z [INFO]  server-4ec4f3e4-f559-0bcb-677d-e0afc991249e: added peer, starting replication: peer=server-7612cd24-58bd-6876-f82b-22544ea61fc3
2025-01-28T14:55:46.549Z [INFO]  server-4ec4f3e4-f559-0bcb-677d-e0afc991249e: added peer, starting replication: peer=server-cc064f2c-0843-57fd-a345-8ef186b2cba1
2025-01-28T14:55:46.549Z [INFO]  server-4ec4f3e4-f559-0bcb-677d-e0afc991249e: pipelining replication: peer="{Voter server-7612cd24-58bd-6876-f82b-22544ea61fc3 7612cd24-58bd-6876-f82b-22544ea61fc3}"
2025-01-28T14:55:46.549Z [INFO]  server-4ec4f3e4-f559-0bcb-677d-e0afc991249e: pipelining replication: peer="{Voter server-cc064f2c-0843-57fd-a345-8ef186b2cba1 cc064f2c-0843-57fd-a345-8ef186b2cba1}"
2025-01-28T14:55:46.655Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:46.549259266 +0000 UTC m=+132.223749246 (1 nodes), 53.076893ms from start of poll, 54.154031ms from cluster start. Timeout at 2025-01-28 14:55:46.654266846 +0000 UTC m=+132.328756846, 105.0076ms after stability
2025-01-28T14:55:46.655Z [INFO]  server-4ec4f3e4-f559-0bcb-677d-e0afc991249e: cannot transfer leadership to itself
2025-01-28T14:55:46.655Z [INFO]  server-4ec4f3e4-f559-0bcb-677d-e0afc991249e: aborting pipeline replication: peer="{Voter server-cc064f2c-0843-57fd-a345-8ef186b2cba1 cc064f2c-0843-57fd-a345-8ef186b2cba1}"
2025-01-28T14:55:46.655Z [INFO]  server-4ec4f3e4-f559-0bcb-677d-e0afc991249e: aborting pipeline replication: peer="{Voter server-7612cd24-58bd-6876-f82b-22544ea61fc3 7612cd24-58bd-6876-f82b-22544ea61fc3}"
--- PASS: TestRaft_LeadershipTransferToItself (0.16s)
=== RUN   TestRaft_LeadershipTransferLeaderRejectsClientRequests
2025-01-28T14:55:46.658Z [INFO]  server-8d995b7c-3e4f-48bc-db12-9401e9370a43: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d995b7c-3e4f-48bc-db12-9401e9370a43 Address:8d995b7c-3e4f-48bc-db12-9401e9370a43} {Suffrage:Voter ID:server-3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8 Address:3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8} {Suffrage:Voter ID:server-99861c09-c48b-b03d-f3f8-5bfee700e81e Address:99861c09-c48b-b03d-f3f8-5bfee700e81e}]"
2025-01-28T14:55:46.659Z [INFO]  server-3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d995b7c-3e4f-48bc-db12-9401e9370a43 Address:8d995b7c-3e4f-48bc-db12-9401e9370a43} {Suffrage:Voter ID:server-3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8 Address:3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8} {Suffrage:Voter ID:server-99861c09-c48b-b03d-f3f8-5bfee700e81e Address:99861c09-c48b-b03d-f3f8-5bfee700e81e}]"
2025-01-28T14:55:46.659Z [INFO]  server-8d995b7c-3e4f-48bc-db12-9401e9370a43: entering follower state: follower="Node at 8d995b7c-3e4f-48bc-db12-9401e9370a43 [Follower]" leader-address= leader-id=
2025-01-28T14:55:46.659Z [INFO]  server-3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8: entering follower state: follower="Node at 3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8 [Follower]" leader-address= leader-id=
2025-01-28T14:55:46.659Z [INFO]  server-99861c09-c48b-b03d-f3f8-5bfee700e81e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8d995b7c-3e4f-48bc-db12-9401e9370a43 Address:8d995b7c-3e4f-48bc-db12-9401e9370a43} {Suffrage:Voter ID:server-3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8 Address:3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8} {Suffrage:Voter ID:server-99861c09-c48b-b03d-f3f8-5bfee700e81e Address:99861c09-c48b-b03d-f3f8-5bfee700e81e}]"
2025-01-28T14:55:46.659Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:46.659Z [INFO]  server-99861c09-c48b-b03d-f3f8-5bfee700e81e: entering follower state: follower="Node at 99861c09-c48b-b03d-f3f8-5bfee700e81e [Follower]" leader-address= leader-id=
2025-01-28T14:55:46.722Z [WARN]  server-3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:46.722Z [INFO]  server-3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8: entering candidate state: node="Node at 3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8 [Candidate]" term=2
2025-01-28T14:55:46.723Z [INFO]  server-3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8: election won: term=2 tally=2
2025-01-28T14:55:46.723Z [INFO]  server-3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8: entering leader state: leader="Node at 3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8 [Leader]"
2025-01-28T14:55:46.723Z [INFO]  server-3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8: added peer, starting replication: peer=server-8d995b7c-3e4f-48bc-db12-9401e9370a43
2025-01-28T14:55:46.723Z [INFO]  server-3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8: added peer, starting replication: peer=server-99861c09-c48b-b03d-f3f8-5bfee700e81e
2025-01-28T14:55:46.723Z [INFO]  server-3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8: pipelining replication: peer="{Voter server-8d995b7c-3e4f-48bc-db12-9401e9370a43 8d995b7c-3e4f-48bc-db12-9401e9370a43}"
2025-01-28T14:55:46.723Z [INFO]  server-3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8: pipelining replication: peer="{Voter server-99861c09-c48b-b03d-f3f8-5bfee700e81e 99861c09-c48b-b03d-f3f8-5bfee700e81e}"
2025-01-28T14:55:46.828Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:46.723192638 +0000 UTC m=+132.397682618 (1 nodes), 63.592161ms from start of poll, 64.510574ms from cluster start. Timeout at 2025-01-28 14:55:46.828239793 +0000 UTC m=+132.502729773, 105.047155ms after stability
    raft_test.go:2457: waiting on future 0
    raft_test.go:2457: waiting on future 1
    raft_test.go:2457: waiting on future 2
    raft_test.go:2457: waiting on future 3
    raft_test.go:2457: waiting on future 4
    raft_test.go:2457: waiting on future 5
    raft_test.go:2457: waiting on future 6
2025-01-28T14:55:46.829Z [INFO]  server-3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8: aborting pipeline replication: peer="{Voter server-8d995b7c-3e4f-48bc-db12-9401e9370a43 8d995b7c-3e4f-48bc-db12-9401e9370a43}"
2025-01-28T14:55:46.829Z [INFO]  server-3ca88f25-d55c-ca06-ab1d-bf92f86cf9c8: aborting pipeline replication: peer="{Voter server-99861c09-c48b-b03d-f3f8-5bfee700e81e 99861c09-c48b-b03d-f3f8-5bfee700e81e}"
--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.17s)
=== RUN   TestRaft_LeadershipTransferLeaderReplicationTimeout
2025-01-28T14:55:46.832Z [INFO]  server-0419cf48-f9a9-9056-bee6-d2aa9e986112: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0419cf48-f9a9-9056-bee6-d2aa9e986112 Address:0419cf48-f9a9-9056-bee6-d2aa9e986112} {Suffrage:Voter ID:server-200a3c74-e46a-a974-6330-d711217f9fb1 Address:200a3c74-e46a-a974-6330-d711217f9fb1} {Suffrage:Voter ID:server-9a546608-1506-737c-6c0a-d0a273c6a27c Address:9a546608-1506-737c-6c0a-d0a273c6a27c}]"
2025-01-28T14:55:46.832Z [INFO]  server-0419cf48-f9a9-9056-bee6-d2aa9e986112: entering follower state: follower="Node at 0419cf48-f9a9-9056-bee6-d2aa9e986112 [Follower]" leader-address= leader-id=
2025-01-28T14:55:46.833Z [INFO]  server-200a3c74-e46a-a974-6330-d711217f9fb1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0419cf48-f9a9-9056-bee6-d2aa9e986112 Address:0419cf48-f9a9-9056-bee6-d2aa9e986112} {Suffrage:Voter ID:server-200a3c74-e46a-a974-6330-d711217f9fb1 Address:200a3c74-e46a-a974-6330-d711217f9fb1} {Suffrage:Voter ID:server-9a546608-1506-737c-6c0a-d0a273c6a27c Address:9a546608-1506-737c-6c0a-d0a273c6a27c}]"
2025-01-28T14:55:46.833Z [INFO]  server-200a3c74-e46a-a974-6330-d711217f9fb1: entering follower state: follower="Node at 200a3c74-e46a-a974-6330-d711217f9fb1 [Follower]" leader-address= leader-id=
2025-01-28T14:55:46.833Z [INFO]  server-9a546608-1506-737c-6c0a-d0a273c6a27c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0419cf48-f9a9-9056-bee6-d2aa9e986112 Address:0419cf48-f9a9-9056-bee6-d2aa9e986112} {Suffrage:Voter ID:server-200a3c74-e46a-a974-6330-d711217f9fb1 Address:200a3c74-e46a-a974-6330-d711217f9fb1} {Suffrage:Voter ID:server-9a546608-1506-737c-6c0a-d0a273c6a27c Address:9a546608-1506-737c-6c0a-d0a273c6a27c}]"
2025-01-28T14:55:46.833Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:46.833Z [INFO]  server-9a546608-1506-737c-6c0a-d0a273c6a27c: entering follower state: follower="Node at 9a546608-1506-737c-6c0a-d0a273c6a27c [Follower]" leader-address= leader-id=
2025-01-28T14:55:46.894Z [WARN]  server-200a3c74-e46a-a974-6330-d711217f9fb1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:46.894Z [INFO]  server-200a3c74-e46a-a974-6330-d711217f9fb1: entering candidate state: node="Node at 200a3c74-e46a-a974-6330-d711217f9fb1 [Candidate]" term=2
2025-01-28T14:55:46.894Z [INFO]  server-200a3c74-e46a-a974-6330-d711217f9fb1: election won: term=2 tally=2
2025-01-28T14:55:46.894Z [INFO]  server-200a3c74-e46a-a974-6330-d711217f9fb1: entering leader state: leader="Node at 200a3c74-e46a-a974-6330-d711217f9fb1 [Leader]"
2025-01-28T14:55:46.895Z [INFO]  server-200a3c74-e46a-a974-6330-d711217f9fb1: added peer, starting replication: peer=server-0419cf48-f9a9-9056-bee6-d2aa9e986112
2025-01-28T14:55:46.895Z [INFO]  server-200a3c74-e46a-a974-6330-d711217f9fb1: added peer, starting replication: peer=server-9a546608-1506-737c-6c0a-d0a273c6a27c
2025-01-28T14:55:46.895Z [INFO]  server-200a3c74-e46a-a974-6330-d711217f9fb1: pipelining replication: peer="{Voter server-0419cf48-f9a9-9056-bee6-d2aa9e986112 0419cf48-f9a9-9056-bee6-d2aa9e986112}"
2025-01-28T14:55:46.895Z [INFO]  server-200a3c74-e46a-a974-6330-d711217f9fb1: pipelining replication: peer="{Voter server-9a546608-1506-737c-6c0a-d0a273c6a27c 9a546608-1506-737c-6c0a-d0a273c6a27c}"
2025-01-28T14:55:47.000Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:46.89500769 +0000 UTC m=+132.569497670 (1 nodes), 61.675202ms from start of poll, 62.588456ms from cluster start. Timeout at 2025-01-28 14:55:47.00001003 +0000 UTC m=+132.674500050, 105.00238ms after stability
2025-01-28T14:55:47.000Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:47.105Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:47.00026722 +0000 UTC m=+132.674757200 (2 nodes), 2.24µs from start of poll, 167.847986ms from cluster start. Timeout at 2025-01-28 14:55:47.105430585 +0000 UTC m=+132.779920565, 105.163365ms after stability
2025-01-28T14:55:48.253Z [INFO]  server-200a3c74-e46a-a974-6330-d711217f9fb1: aborting pipeline replication: peer="{Voter server-0419cf48-f9a9-9056-bee6-d2aa9e986112 0419cf48-f9a9-9056-bee6-d2aa9e986112}"
2025-01-28T14:55:48.253Z [INFO]  server-200a3c74-e46a-a974-6330-d711217f9fb1: aborting pipeline replication: peer="{Voter server-9a546608-1506-737c-6c0a-d0a273c6a27c 9a546608-1506-737c-6c0a-d0a273c6a27c}"
--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (1.42s)
=== RUN   TestRaft_LeadershipTransferIgnoresNonvoters
2025-01-28T14:55:48.257Z [INFO]  server-e22cb081-05ea-6bc7-bc5c-a1830f1dc522: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e22cb081-05ea-6bc7-bc5c-a1830f1dc522 Address:e22cb081-05ea-6bc7-bc5c-a1830f1dc522} {Suffrage:Voter ID:server-a50880e4-77ae-ffc1-a203-e50d4c0d5e52 Address:a50880e4-77ae-ffc1-a203-e50d4c0d5e52}]"
2025-01-28T14:55:48.257Z [INFO]  server-e22cb081-05ea-6bc7-bc5c-a1830f1dc522: entering follower state: follower="Node at e22cb081-05ea-6bc7-bc5c-a1830f1dc522 [Follower]" leader-address= leader-id=
2025-01-28T14:55:48.257Z [INFO]  server-a50880e4-77ae-ffc1-a203-e50d4c0d5e52: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e22cb081-05ea-6bc7-bc5c-a1830f1dc522 Address:e22cb081-05ea-6bc7-bc5c-a1830f1dc522} {Suffrage:Voter ID:server-a50880e4-77ae-ffc1-a203-e50d4c0d5e52 Address:a50880e4-77ae-ffc1-a203-e50d4c0d5e52}]"
2025-01-28T14:55:48.257Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:48.257Z [INFO]  server-a50880e4-77ae-ffc1-a203-e50d4c0d5e52: entering follower state: follower="Node at a50880e4-77ae-ffc1-a203-e50d4c0d5e52 [Follower]" leader-address= leader-id=
2025-01-28T14:55:48.322Z [WARN]  server-e22cb081-05ea-6bc7-bc5c-a1830f1dc522: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:48.322Z [INFO]  server-e22cb081-05ea-6bc7-bc5c-a1830f1dc522: entering candidate state: node="Node at e22cb081-05ea-6bc7-bc5c-a1830f1dc522 [Candidate]" term=2
2025-01-28T14:55:48.322Z [INFO]  server-e22cb081-05ea-6bc7-bc5c-a1830f1dc522: election won: term=2 tally=2
2025-01-28T14:55:48.322Z [INFO]  server-e22cb081-05ea-6bc7-bc5c-a1830f1dc522: entering leader state: leader="Node at e22cb081-05ea-6bc7-bc5c-a1830f1dc522 [Leader]"
2025-01-28T14:55:48.322Z [INFO]  server-e22cb081-05ea-6bc7-bc5c-a1830f1dc522: added peer, starting replication: peer=server-a50880e4-77ae-ffc1-a203-e50d4c0d5e52
2025-01-28T14:55:48.323Z [INFO]  server-e22cb081-05ea-6bc7-bc5c-a1830f1dc522: pipelining replication: peer="{Voter server-a50880e4-77ae-ffc1-a203-e50d4c0d5e52 a50880e4-77ae-ffc1-a203-e50d4c0d5e52}"
2025-01-28T14:55:48.428Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:48.322940585 +0000 UTC m=+133.997430585 (1 nodes), 65.340036ms from start of poll, 65.947099ms from cluster start. Timeout at 2025-01-28 14:55:48.427946825 +0000 UTC m=+134.102436805, 105.00622ms after stability
2025-01-28T14:55:48.428Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:48.534Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:48.428954324 +0000 UTC m=+134.103444304 (1 nodes), 1.68µs from start of poll, 171.960818ms from cluster start. Timeout at 2025-01-28 14:55:48.534346807 +0000 UTC m=+134.208836767, 105.392463ms after stability
2025-01-28T14:55:48.534Z [INFO]  server-e22cb081-05ea-6bc7-bc5c-a1830f1dc522: updating configuration: command=DemoteVoter server-id=server-a50880e4-77ae-ffc1-a203-e50d4c0d5e52 server-addr= servers="[{Suffrage:Voter ID:server-e22cb081-05ea-6bc7-bc5c-a1830f1dc522 Address:e22cb081-05ea-6bc7-bc5c-a1830f1dc522} {Suffrage:Nonvoter ID:server-a50880e4-77ae-ffc1-a203-e50d4c0d5e52 Address:a50880e4-77ae-ffc1-a203-e50d4c0d5e52}]"
2025-01-28T14:55:48.535Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:48.641Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:48.535153486 +0000 UTC m=+134.209643466 (1 nodes), 2.12µs from start of poll, 278.15998ms from cluster start. Timeout at 2025-01-28 14:55:48.640421421 +0000 UTC m=+134.314911401, 105.267935ms after stability
2025-01-28T14:55:48.641Z [INFO]  server-e22cb081-05ea-6bc7-bc5c-a1830f1dc522: aborting pipeline replication: peer="{Voter server-a50880e4-77ae-ffc1-a203-e50d4c0d5e52 a50880e4-77ae-ffc1-a203-e50d4c0d5e52}"
--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.39s)
=== RUN   TestRaft_LeadershipTransferStopRightAway
--- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s)
=== RUN   TestRaft_GetConfigurationNoBootstrap
2025-01-28T14:55:48.644Z [INFO]  server-9504af7e-f5a7-c933-2c5d-630cb7d275a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9504af7e-f5a7-c933-2c5d-630cb7d275a9 Address:9504af7e-f5a7-c933-2c5d-630cb7d275a9} {Suffrage:Voter ID:server-1a9d936b-b713-e5f2-0aab-c55d0efada1b Address:1a9d936b-b713-e5f2-0aab-c55d0efada1b}]"
2025-01-28T14:55:48.645Z [INFO]  server-1a9d936b-b713-e5f2-0aab-c55d0efada1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9504af7e-f5a7-c933-2c5d-630cb7d275a9 Address:9504af7e-f5a7-c933-2c5d-630cb7d275a9} {Suffrage:Voter ID:server-1a9d936b-b713-e5f2-0aab-c55d0efada1b Address:1a9d936b-b713-e5f2-0aab-c55d0efada1b}]"
2025-01-28T14:55:48.645Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:48.645Z [INFO]  server-1a9d936b-b713-e5f2-0aab-c55d0efada1b: entering follower state: follower="Node at 1a9d936b-b713-e5f2-0aab-c55d0efada1b [Follower]" leader-address= leader-id=
2025-01-28T14:55:48.645Z [INFO]  server-9504af7e-f5a7-c933-2c5d-630cb7d275a9: entering follower state: follower="Node at 9504af7e-f5a7-c933-2c5d-630cb7d275a9 [Follower]" leader-address= leader-id=
2025-01-28T14:55:48.718Z [WARN]  server-9504af7e-f5a7-c933-2c5d-630cb7d275a9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:48.718Z [INFO]  server-9504af7e-f5a7-c933-2c5d-630cb7d275a9: entering candidate state: node="Node at 9504af7e-f5a7-c933-2c5d-630cb7d275a9 [Candidate]" term=2
2025-01-28T14:55:48.719Z [INFO]  server-9504af7e-f5a7-c933-2c5d-630cb7d275a9: election won: term=2 tally=2
2025-01-28T14:55:48.719Z [INFO]  server-9504af7e-f5a7-c933-2c5d-630cb7d275a9: entering leader state: leader="Node at 9504af7e-f5a7-c933-2c5d-630cb7d275a9 [Leader]"
2025-01-28T14:55:48.719Z [INFO]  server-9504af7e-f5a7-c933-2c5d-630cb7d275a9: added peer, starting replication: peer=server-1a9d936b-b713-e5f2-0aab-c55d0efada1b
2025-01-28T14:55:48.719Z [INFO]  server-9504af7e-f5a7-c933-2c5d-630cb7d275a9: pipelining replication: peer="{Voter server-1a9d936b-b713-e5f2-0aab-c55d0efada1b 1a9d936b-b713-e5f2-0aab-c55d0efada1b}"
2025-01-28T14:55:48.824Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:48.719202464 +0000 UTC m=+134.393692443 (1 nodes), 73.888129ms from start of poll, 74.457216ms from cluster start. Timeout at 2025-01-28 14:55:48.824212582 +0000 UTC m=+134.498702562, 105.010119ms after stability
2025-01-28T14:55:48.825Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:48.931Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:48.82556383 +0000 UTC m=+134.500053790 (1 nodes), 1.98µs from start of poll, 180.818563ms from cluster start. Timeout at 2025-01-28 14:55:48.930759011 +0000 UTC m=+134.605248971, 105.195181ms after stability
2025-01-28T14:55:48.943Z [INFO]  server-1a9d936b-b713-e5f2-0aab-c55d0efada1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9504af7e-f5a7-c933-2c5d-630cb7d275a9 Address:9504af7e-f5a7-c933-2c5d-630cb7d275a9} {Suffrage:Voter ID:server-1a9d936b-b713-e5f2-0aab-c55d0efada1b Address:1a9d936b-b713-e5f2-0aab-c55d0efada1b}]"
2025-01-28T14:55:48.944Z [INFO]  server-9504af7e-f5a7-c933-2c5d-630cb7d275a9: aborting pipeline replication: peer="{Voter server-1a9d936b-b713-e5f2-0aab-c55d0efada1b 1a9d936b-b713-e5f2-0aab-c55d0efada1b}"
--- PASS: TestRaft_GetConfigurationNoBootstrap (0.30s)
=== RUN   TestRaft_LogStoreIsMonotonic
2025-01-28T14:55:48.946Z [INFO]  server-6075dfdb-c5c3-c542-65d2-1b0d11eba32f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6075dfdb-c5c3-c542-65d2-1b0d11eba32f Address:6075dfdb-c5c3-c542-65d2-1b0d11eba32f}]"
2025-01-28T14:55:48.946Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:48.946Z [INFO]  server-6075dfdb-c5c3-c542-65d2-1b0d11eba32f: entering follower state: follower="Node at 6075dfdb-c5c3-c542-65d2-1b0d11eba32f [Follower]" leader-address= leader-id=
2025-01-28T14:55:49.021Z [WARN]  server-6075dfdb-c5c3-c542-65d2-1b0d11eba32f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:49.021Z [INFO]  server-6075dfdb-c5c3-c542-65d2-1b0d11eba32f: entering candidate state: node="Node at 6075dfdb-c5c3-c542-65d2-1b0d11eba32f [Candidate]" term=2
2025-01-28T14:55:49.021Z [INFO]  server-6075dfdb-c5c3-c542-65d2-1b0d11eba32f: election won: term=2 tally=1
2025-01-28T14:55:49.021Z [INFO]  server-6075dfdb-c5c3-c542-65d2-1b0d11eba32f: entering leader state: leader="Node at 6075dfdb-c5c3-c542-65d2-1b0d11eba32f [Leader]"
2025-01-28T14:55:49.127Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:49.021611716 +0000 UTC m=+134.696101696 (1 nodes), 75.05206ms from start of poll, 75.436344ms from cluster start. Timeout at 2025-01-28 14:55:49.126616196 +0000 UTC m=+134.801106316, 105.00462ms after stability
--- PASS: TestRaft_LogStoreIsMonotonic (0.18s)
=== RUN   TestRaft_CacheLogWithStoreError
2025-01-28T14:55:49.129Z [INFO]  server-b0dbc48b-3325-97a5-3b59-17f6ac95743f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b0dbc48b-3325-97a5-3b59-17f6ac95743f Address:b0dbc48b-3325-97a5-3b59-17f6ac95743f} {Suffrage:Voter ID:server-87b305a3-5544-05f9-6f42-0e5bdae05c67 Address:87b305a3-5544-05f9-6f42-0e5bdae05c67}]"
2025-01-28T14:55:49.130Z [INFO]  server-87b305a3-5544-05f9-6f42-0e5bdae05c67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b0dbc48b-3325-97a5-3b59-17f6ac95743f Address:b0dbc48b-3325-97a5-3b59-17f6ac95743f} {Suffrage:Voter ID:server-87b305a3-5544-05f9-6f42-0e5bdae05c67 Address:87b305a3-5544-05f9-6f42-0e5bdae05c67}]"
2025-01-28T14:55:49.130Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:49.134Z [INFO]  server-b0dbc48b-3325-97a5-3b59-17f6ac95743f: entering follower state: follower="Node at b0dbc48b-3325-97a5-3b59-17f6ac95743f [Follower]" leader-address= leader-id=
2025-01-28T14:55:49.134Z [INFO]  server-87b305a3-5544-05f9-6f42-0e5bdae05c67: entering follower state: follower="Node at 87b305a3-5544-05f9-6f42-0e5bdae05c67 [Follower]" leader-address= leader-id=
2025-01-28T14:55:49.194Z [WARN]  server-87b305a3-5544-05f9-6f42-0e5bdae05c67: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:49.195Z [INFO]  server-87b305a3-5544-05f9-6f42-0e5bdae05c67: entering candidate state: node="Node at 87b305a3-5544-05f9-6f42-0e5bdae05c67 [Candidate]" term=2
2025-01-28T14:55:49.195Z [INFO]  server-87b305a3-5544-05f9-6f42-0e5bdae05c67: election won: term=2 tally=2
2025-01-28T14:55:49.195Z [INFO]  server-87b305a3-5544-05f9-6f42-0e5bdae05c67: entering leader state: leader="Node at 87b305a3-5544-05f9-6f42-0e5bdae05c67 [Leader]"
2025-01-28T14:55:49.195Z [INFO]  server-87b305a3-5544-05f9-6f42-0e5bdae05c67: added peer, starting replication: peer=server-b0dbc48b-3325-97a5-3b59-17f6ac95743f
2025-01-28T14:55:49.195Z [INFO]  server-87b305a3-5544-05f9-6f42-0e5bdae05c67: pipelining replication: peer="{Voter server-b0dbc48b-3325-97a5-3b59-17f6ac95743f b0dbc48b-3325-97a5-3b59-17f6ac95743f}"
2025-01-28T14:55:49.301Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:49.19540772 +0000 UTC m=+134.869897700 (1 nodes), 64.368587ms from start of poll, 65.701762ms from cluster start. Timeout at 2025-01-28 14:55:49.300431378 +0000 UTC m=+134.974921358, 105.023658ms after stability
2025-01-28T14:55:49.301Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:49.406Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:49.301459196 +0000 UTC m=+134.975949176 (1 nodes), 1.48µs from start of poll, 171.753238ms from cluster start. Timeout at 2025-01-28 14:55:49.406649538 +0000 UTC m=+135.081139518, 105.190342ms after stability
2025-01-28T14:55:49.408Z [ERROR] server-b0dbc48b-3325-97a5-3b59-17f6ac95743f: failed to append to logs: error="unable to store logs within log store, err: "some error""
2025-01-28T14:55:49.408Z [WARN]  server-b0dbc48b-3325-97a5-3b59-17f6ac95743f: failed to get previous log: previous-index=9 last-index=7 error="log not found"
2025-01-28T14:55:49.408Z [INFO]  server-87b305a3-5544-05f9-6f42-0e5bdae05c67: aborting pipeline replication: peer="{Voter server-b0dbc48b-3325-97a5-3b59-17f6ac95743f b0dbc48b-3325-97a5-3b59-17f6ac95743f}"
2025-01-28T14:55:49.418Z [INFO]  server-87b305a3-5544-05f9-6f42-0e5bdae05c67: pipelining replication: peer="{Voter server-b0dbc48b-3325-97a5-3b59-17f6ac95743f b0dbc48b-3325-97a5-3b59-17f6ac95743f}"
2025-01-28T14:55:49.418Z [INFO]  server-b0dbc48b-3325-97a5-3b59-17f6ac95743f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b0dbc48b-3325-97a5-3b59-17f6ac95743f Address:b0dbc48b-3325-97a5-3b59-17f6ac95743f} {Suffrage:Voter ID:server-87b305a3-5544-05f9-6f42-0e5bdae05c67 Address:87b305a3-5544-05f9-6f42-0e5bdae05c67}]"
2025-01-28T14:55:49.418Z [INFO]  server-87b305a3-5544-05f9-6f42-0e5bdae05c67: aborting pipeline replication: peer="{Voter server-b0dbc48b-3325-97a5-3b59-17f6ac95743f b0dbc48b-3325-97a5-3b59-17f6ac95743f}"
--- PASS: TestRaft_CacheLogWithStoreError (0.29s)
=== RUN   TestRaft_ReloadConfig
2025-01-28T14:55:49.420Z [INFO]  server-afbd0443-ec9d-5a6f-1b46-d9896ecd3b3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afbd0443-ec9d-5a6f-1b46-d9896ecd3b3a Address:afbd0443-ec9d-5a6f-1b46-d9896ecd3b3a}]"
--- PASS: TestRaft_ReloadConfig (0.00s)
=== RUN   TestRaft_ReloadConfigValidates
2025-01-28T14:55:49.422Z [INFO]  server-cb2322c6-30f1-1ee3-4961-f36c5d1768e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cb2322c6-30f1-1ee3-4961-f36c5d1768e7 Address:cb2322c6-30f1-1ee3-4961-f36c5d1768e7}]"
--- PASS: TestRaft_ReloadConfigValidates (0.00s)
=== RUN   TestRaft_InstallSnapshot_InvalidPeers
2025-01-28T14:55:49.423Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)"
--- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s)
=== RUN   TestRaft_VoteNotGranted_WhenNodeNotInCluster
2025-01-28T14:55:49.425Z [INFO]  server-fc9db554-9723-5936-cfa0-bb7cb96e09a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc9db554-9723-5936-cfa0-bb7cb96e09a3 Address:fc9db554-9723-5936-cfa0-bb7cb96e09a3} {Suffrage:Voter ID:server-51497266-e076-d678-05fa-8e87d0870285 Address:51497266-e076-d678-05fa-8e87d0870285} {Suffrage:Voter ID:server-e42861cd-4a11-860f-9cf8-f69f06cd95c0 Address:e42861cd-4a11-860f-9cf8-f69f06cd95c0}]"
2025-01-28T14:55:49.425Z [INFO]  server-fc9db554-9723-5936-cfa0-bb7cb96e09a3: entering follower state: follower="Node at fc9db554-9723-5936-cfa0-bb7cb96e09a3 [Follower]" leader-address= leader-id=
2025-01-28T14:55:49.426Z [INFO]  server-51497266-e076-d678-05fa-8e87d0870285: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc9db554-9723-5936-cfa0-bb7cb96e09a3 Address:fc9db554-9723-5936-cfa0-bb7cb96e09a3} {Suffrage:Voter ID:server-51497266-e076-d678-05fa-8e87d0870285 Address:51497266-e076-d678-05fa-8e87d0870285} {Suffrage:Voter ID:server-e42861cd-4a11-860f-9cf8-f69f06cd95c0 Address:e42861cd-4a11-860f-9cf8-f69f06cd95c0}]"
2025-01-28T14:55:49.426Z [INFO]  server-51497266-e076-d678-05fa-8e87d0870285: entering follower state: follower="Node at 51497266-e076-d678-05fa-8e87d0870285 [Follower]" leader-address= leader-id=
2025-01-28T14:55:49.426Z [INFO]  server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fc9db554-9723-5936-cfa0-bb7cb96e09a3 Address:fc9db554-9723-5936-cfa0-bb7cb96e09a3} {Suffrage:Voter ID:server-51497266-e076-d678-05fa-8e87d0870285 Address:51497266-e076-d678-05fa-8e87d0870285} {Suffrage:Voter ID:server-e42861cd-4a11-860f-9cf8-f69f06cd95c0 Address:e42861cd-4a11-860f-9cf8-f69f06cd95c0}]"
2025-01-28T14:55:49.426Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:49.426Z [INFO]  server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: entering follower state: follower="Node at e42861cd-4a11-860f-9cf8-f69f06cd95c0 [Follower]" leader-address= leader-id=
2025-01-28T14:55:49.478Z [WARN]  server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:49.478Z [INFO]  server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: entering candidate state: node="Node at e42861cd-4a11-860f-9cf8-f69f06cd95c0 [Candidate]" term=2
2025-01-28T14:55:49.478Z [INFO]  server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: election won: term=2 tally=2
2025-01-28T14:55:49.478Z [INFO]  server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: entering leader state: leader="Node at e42861cd-4a11-860f-9cf8-f69f06cd95c0 [Leader]"
2025-01-28T14:55:49.478Z [INFO]  server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: added peer, starting replication: peer=server-fc9db554-9723-5936-cfa0-bb7cb96e09a3
2025-01-28T14:55:49.478Z [INFO]  server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: added peer, starting replication: peer=server-51497266-e076-d678-05fa-8e87d0870285
2025-01-28T14:55:49.478Z [INFO]  server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: pipelining replication: peer="{Voter server-fc9db554-9723-5936-cfa0-bb7cb96e09a3 fc9db554-9723-5936-cfa0-bb7cb96e09a3}"
2025-01-28T14:55:49.479Z [INFO]  server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: pipelining replication: peer="{Voter server-51497266-e076-d678-05fa-8e87d0870285 51497266-e076-d678-05fa-8e87d0870285}"
2025-01-28T14:55:49.583Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:49.478777609 +0000 UTC m=+135.153267589 (1 nodes), 52.382119ms from start of poll, 53.300952ms from cluster start. Timeout at 2025-01-28 14:55:49.583787828 +0000 UTC m=+135.258277808, 105.010219ms after stability
2025-01-28T14:55:49.589Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:49.695Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:49.589603094 +0000 UTC m=+135.264093074 (2 nodes), 2.22µs from start of poll, 164.126437ms from cluster start. Timeout at 2025-01-28 14:55:49.694772458 +0000 UTC m=+135.369262518, 105.169444ms after stability
2025-01-28T14:55:49.695Z [INFO]  server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: updating configuration: command=RemoveServer server-id=server-fc9db554-9723-5936-cfa0-bb7cb96e09a3 server-addr= servers="[{Suffrage:Voter ID:server-51497266-e076-d678-05fa-8e87d0870285 Address:51497266-e076-d678-05fa-8e87d0870285} {Suffrage:Voter ID:server-e42861cd-4a11-860f-9cf8-f69f06cd95c0 Address:e42861cd-4a11-860f-9cf8-f69f06cd95c0}]"
2025-01-28T14:55:49.696Z [INFO]  server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: removed peer, stopping replication: peer=server-fc9db554-9723-5936-cfa0-bb7cb96e09a3 last-index=3
2025-01-28T14:55:49.696Z [INFO]  server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: aborting pipeline replication: peer="{Voter server-fc9db554-9723-5936-cfa0-bb7cb96e09a3 fc9db554-9723-5936-cfa0-bb7cb96e09a3}"
2025-01-28T14:55:49.797Z [WARN]  server-fc9db554-9723-5936-cfa0-bb7cb96e09a3: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2025-01-28T14:55:49.805Z [ERROR] server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: failed to pipeline appendEntries: peer="{Voter server-51497266-e076-d678-05fa-8e87d0870285 51497266-e076-d678-05fa-8e87d0870285}" error="append pipeline closed"
2025-01-28T14:55:49.805Z [INFO]  server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: aborting pipeline replication: peer="{Voter server-51497266-e076-d678-05fa-8e87d0870285 51497266-e076-d678-05fa-8e87d0870285}"
2025-01-28T14:55:49.808Z [ERROR] server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: failed to heartbeat to: peer=51497266-e076-d678-05fa-8e87d0870285 backoff time=10ms error="failed to connect to peer: 51497266-e076-d678-05fa-8e87d0870285"
2025-01-28T14:55:49.817Z [ERROR] server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: failed to appendEntries to: peer="{Voter server-51497266-e076-d678-05fa-8e87d0870285 51497266-e076-d678-05fa-8e87d0870285}" error="failed to connect to peer: 51497266-e076-d678-05fa-8e87d0870285"
2025-01-28T14:55:49.828Z [ERROR] server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: failed to heartbeat to: peer=51497266-e076-d678-05fa-8e87d0870285 backoff time=10ms error="failed to connect to peer: 51497266-e076-d678-05fa-8e87d0870285"
2025-01-28T14:55:49.835Z [ERROR] server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: failed to appendEntries to: peer="{Voter server-51497266-e076-d678-05fa-8e87d0870285 51497266-e076-d678-05fa-8e87d0870285}" error="failed to connect to peer: 51497266-e076-d678-05fa-8e87d0870285"
2025-01-28T14:55:49.847Z [ERROR] server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: failed to heartbeat to: peer=51497266-e076-d678-05fa-8e87d0870285 backoff time=10ms error="failed to connect to peer: 51497266-e076-d678-05fa-8e87d0870285"
2025-01-28T14:55:49.849Z [WARN]  server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: failed to contact: server-id=server-51497266-e076-d678-05fa-8e87d0870285 time=50.548611ms
2025-01-28T14:55:49.849Z [WARN]  server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: failed to contact quorum of nodes, stepping down
2025-01-28T14:55:49.850Z [INFO]  server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: entering follower state: follower="Node at e42861cd-4a11-860f-9cf8-f69f06cd95c0 [Follower]" leader-address= leader-id=
2025-01-28T14:55:49.854Z [ERROR] server-e42861cd-4a11-860f-9cf8-f69f06cd95c0: failed to appendEntries to: peer="{Voter server-51497266-e076-d678-05fa-8e87d0870285 51497266-e076-d678-05fa-8e87d0870285}" error="failed to connect to peer: 51497266-e076-d678-05fa-8e87d0870285"
2025-01-28T14:55:49.883Z [WARN]  server-51497266-e076-d678-05fa-8e87d0870285: heartbeat timeout reached, starting election: last-leader-addr=e42861cd-4a11-860f-9cf8-f69f06cd95c0 last-leader-id=server-e42861cd-4a11-860f-9cf8-f69f06cd95c0
2025-01-28T14:55:49.883Z [INFO]  server-51497266-e076-d678-05fa-8e87d0870285: entering candidate state: node="Node at 51497266-e076-d678-05fa-8e87d0870285 [Candidate]" term=3
2025-01-28T14:55:49.883Z [ERROR] server-51497266-e076-d678-05fa-8e87d0870285: failed to make requestVote RPC: target="{Voter server-e42861cd-4a11-860f-9cf8-f69f06cd95c0 e42861cd-4a11-860f-9cf8-f69f06cd95c0}" error="failed to connect to peer: e42861cd-4a11-860f-9cf8-f69f06cd95c0" term=3
2025-01-28T14:55:49.907Z [WARN]  server-51497266-e076-d678-05fa-8e87d0870285: rejecting vote request since node is not in configuration: from=fc9db554-9723-5936-cfa0-bb7cb96e09a3
--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.49s)
=== RUN   TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin
2025-01-28T14:55:49.911Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c6d0317c-047e-3058-733b-aa9010a436ca Address:c6d0317c-047e-3058-733b-aa9010a436ca} {Suffrage:Voter ID:server-dbec255f-f4af-0920-107a-83477fbee278 Address:dbec255f-f4af-0920-107a-83477fbee278} {Suffrage:Voter ID:server-acfb0190-f267-5721-2143-bca1045c67d5 Address:acfb0190-f267-5721-2143-bca1045c67d5}]"
2025-01-28T14:55:49.912Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: entering follower state: follower="Node at c6d0317c-047e-3058-733b-aa9010a436ca [Follower]" leader-address= leader-id=
2025-01-28T14:55:49.912Z [INFO]  server-dbec255f-f4af-0920-107a-83477fbee278: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c6d0317c-047e-3058-733b-aa9010a436ca Address:c6d0317c-047e-3058-733b-aa9010a436ca} {Suffrage:Voter ID:server-dbec255f-f4af-0920-107a-83477fbee278 Address:dbec255f-f4af-0920-107a-83477fbee278} {Suffrage:Voter ID:server-acfb0190-f267-5721-2143-bca1045c67d5 Address:acfb0190-f267-5721-2143-bca1045c67d5}]"
2025-01-28T14:55:49.912Z [INFO]  server-dbec255f-f4af-0920-107a-83477fbee278: entering follower state: follower="Node at dbec255f-f4af-0920-107a-83477fbee278 [Follower]" leader-address= leader-id=
2025-01-28T14:55:49.912Z [INFO]  server-acfb0190-f267-5721-2143-bca1045c67d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c6d0317c-047e-3058-733b-aa9010a436ca Address:c6d0317c-047e-3058-733b-aa9010a436ca} {Suffrage:Voter ID:server-dbec255f-f4af-0920-107a-83477fbee278 Address:dbec255f-f4af-0920-107a-83477fbee278} {Suffrage:Voter ID:server-acfb0190-f267-5721-2143-bca1045c67d5 Address:acfb0190-f267-5721-2143-bca1045c67d5}]"
2025-01-28T14:55:49.912Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:49.912Z [INFO]  server-acfb0190-f267-5721-2143-bca1045c67d5: entering follower state: follower="Node at acfb0190-f267-5721-2143-bca1045c67d5 [Follower]" leader-address= leader-id=
2025-01-28T14:55:49.974Z [WARN]  server-c6d0317c-047e-3058-733b-aa9010a436ca: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:49.974Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: entering candidate state: node="Node at c6d0317c-047e-3058-733b-aa9010a436ca [Candidate]" term=2
2025-01-28T14:55:49.974Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: election won: term=2 tally=2
2025-01-28T14:55:49.974Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: entering leader state: leader="Node at c6d0317c-047e-3058-733b-aa9010a436ca [Leader]"
2025-01-28T14:55:49.974Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: added peer, starting replication: peer=server-dbec255f-f4af-0920-107a-83477fbee278
2025-01-28T14:55:49.974Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: added peer, starting replication: peer=server-acfb0190-f267-5721-2143-bca1045c67d5
2025-01-28T14:55:49.975Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: pipelining replication: peer="{Voter server-acfb0190-f267-5721-2143-bca1045c67d5 acfb0190-f267-5721-2143-bca1045c67d5}"
2025-01-28T14:55:49.975Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: pipelining replication: peer="{Voter server-dbec255f-f4af-0920-107a-83477fbee278 dbec255f-f4af-0920-107a-83477fbee278}"
2025-01-28T14:55:50.080Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:49.975026075 +0000 UTC m=+135.649516095 (1 nodes), 62.389835ms from start of poll, 63.293469ms from cluster start. Timeout at 2025-01-28 14:55:50.080033015 +0000 UTC m=+135.754523055, 105.00696ms after stability
2025-01-28T14:55:50.085Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:50.191Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:50.085581525 +0000 UTC m=+135.760071505 (2 nodes), 2.959µs from start of poll, 173.848879ms from cluster start. Timeout at 2025-01-28 14:55:50.190765028 +0000 UTC m=+135.865255108, 105.183603ms after stability
2025-01-28T14:55:50.197Z [ERROR] server-c6d0317c-047e-3058-733b-aa9010a436ca: failed to heartbeat to: peer=dbec255f-f4af-0920-107a-83477fbee278 backoff time=10ms error="failed to connect to peer: dbec255f-f4af-0920-107a-83477fbee278"
2025-01-28T14:55:50.198Z [ERROR] server-c6d0317c-047e-3058-733b-aa9010a436ca: failed to pipeline appendEntries: peer="{Voter server-dbec255f-f4af-0920-107a-83477fbee278 dbec255f-f4af-0920-107a-83477fbee278}" error="append pipeline closed"
2025-01-28T14:55:50.198Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: aborting pipeline replication: peer="{Voter server-dbec255f-f4af-0920-107a-83477fbee278 dbec255f-f4af-0920-107a-83477fbee278}"
2025-01-28T14:55:50.206Z [ERROR] server-c6d0317c-047e-3058-733b-aa9010a436ca: failed to appendEntries to: peer="{Voter server-dbec255f-f4af-0920-107a-83477fbee278 dbec255f-f4af-0920-107a-83477fbee278}" error="failed to connect to peer: dbec255f-f4af-0920-107a-83477fbee278"
2025-01-28T14:55:50.215Z [ERROR] server-c6d0317c-047e-3058-733b-aa9010a436ca: failed to heartbeat to: peer=dbec255f-f4af-0920-107a-83477fbee278 backoff time=10ms error="failed to connect to peer: dbec255f-f4af-0920-107a-83477fbee278"
2025-01-28T14:55:50.227Z [ERROR] server-c6d0317c-047e-3058-733b-aa9010a436ca: failed to appendEntries to: peer="{Voter server-dbec255f-f4af-0920-107a-83477fbee278 dbec255f-f4af-0920-107a-83477fbee278}" error="failed to connect to peer: dbec255f-f4af-0920-107a-83477fbee278"
2025-01-28T14:55:50.234Z [ERROR] server-c6d0317c-047e-3058-733b-aa9010a436ca: failed to heartbeat to: peer=dbec255f-f4af-0920-107a-83477fbee278 backoff time=10ms error="failed to connect to peer: dbec255f-f4af-0920-107a-83477fbee278"
2025-01-28T14:55:50.237Z [WARN]  server-c6d0317c-047e-3058-733b-aa9010a436ca: failed to contact: server-id=server-dbec255f-f4af-0920-107a-83477fbee278 time=50.365269ms
2025-01-28T14:55:50.244Z [ERROR] server-c6d0317c-047e-3058-733b-aa9010a436ca: failed to appendEntries to: peer="{Voter server-dbec255f-f4af-0920-107a-83477fbee278 dbec255f-f4af-0920-107a-83477fbee278}" error="failed to connect to peer: dbec255f-f4af-0920-107a-83477fbee278"
2025-01-28T14:55:50.254Z [ERROR] server-c6d0317c-047e-3058-733b-aa9010a436ca: failed to heartbeat to: peer=dbec255f-f4af-0920-107a-83477fbee278 backoff time=20ms error="failed to connect to peer: dbec255f-f4af-0920-107a-83477fbee278"
2025-01-28T14:55:50.273Z [ERROR] server-c6d0317c-047e-3058-733b-aa9010a436ca: failed to appendEntries to: peer="{Voter server-dbec255f-f4af-0920-107a-83477fbee278 dbec255f-f4af-0920-107a-83477fbee278}" error="failed to connect to peer: dbec255f-f4af-0920-107a-83477fbee278"
2025-01-28T14:55:50.282Z [ERROR] server-c6d0317c-047e-3058-733b-aa9010a436ca: failed to heartbeat to: peer=dbec255f-f4af-0920-107a-83477fbee278 backoff time=25ms error="failed to connect to peer: dbec255f-f4af-0920-107a-83477fbee278"
2025-01-28T14:55:50.283Z [WARN]  server-dbec255f-f4af-0920-107a-83477fbee278: heartbeat timeout reached, starting election: last-leader-addr=c6d0317c-047e-3058-733b-aa9010a436ca last-leader-id=server-c6d0317c-047e-3058-733b-aa9010a436ca
2025-01-28T14:55:50.283Z [WARN]  server-c6d0317c-047e-3058-733b-aa9010a436ca: failed to contact: server-id=server-dbec255f-f4af-0920-107a-83477fbee278 time=96.848564ms
2025-01-28T14:55:50.283Z [INFO]  server-dbec255f-f4af-0920-107a-83477fbee278: entering candidate state: node="Node at dbec255f-f4af-0920-107a-83477fbee278 [Candidate]" term=3
2025-01-28T14:55:50.284Z [ERROR] server-dbec255f-f4af-0920-107a-83477fbee278: failed to make requestVote RPC: target="{Voter server-acfb0190-f267-5721-2143-bca1045c67d5 acfb0190-f267-5721-2143-bca1045c67d5}" error="failed to connect to peer: acfb0190-f267-5721-2143-bca1045c67d5" term=3
2025-01-28T14:55:50.284Z [ERROR] server-dbec255f-f4af-0920-107a-83477fbee278: failed to make requestVote RPC: target="{Voter server-c6d0317c-047e-3058-733b-aa9010a436ca c6d0317c-047e-3058-733b-aa9010a436ca}" error="failed to connect to peer: c6d0317c-047e-3058-733b-aa9010a436ca" term=3
2025-01-28T14:55:50.296Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: updating configuration: command=RemoveServer server-id=server-dbec255f-f4af-0920-107a-83477fbee278 server-addr= servers="[{Suffrage:Voter ID:server-c6d0317c-047e-3058-733b-aa9010a436ca Address:c6d0317c-047e-3058-733b-aa9010a436ca} {Suffrage:Voter ID:server-acfb0190-f267-5721-2143-bca1045c67d5 Address:acfb0190-f267-5721-2143-bca1045c67d5}]"
2025-01-28T14:55:50.297Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: removed peer, stopping replication: peer=server-dbec255f-f4af-0920-107a-83477fbee278 last-index=3
2025-01-28T14:55:50.297Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: updating configuration: command=AddNonvoter server-id=server-dbec255f-f4af-0920-107a-83477fbee278 server-addr=dbec255f-f4af-0920-107a-83477fbee278 servers="[{Suffrage:Voter ID:server-c6d0317c-047e-3058-733b-aa9010a436ca Address:c6d0317c-047e-3058-733b-aa9010a436ca} {Suffrage:Voter ID:server-acfb0190-f267-5721-2143-bca1045c67d5 Address:acfb0190-f267-5721-2143-bca1045c67d5} {Suffrage:Nonvoter ID:server-dbec255f-f4af-0920-107a-83477fbee278 Address:dbec255f-f4af-0920-107a-83477fbee278}]"
2025-01-28T14:55:50.297Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: added peer, starting replication: peer=server-dbec255f-f4af-0920-107a-83477fbee278
2025-01-28T14:55:50.297Z [ERROR] server-c6d0317c-047e-3058-733b-aa9010a436ca: failed to appendEntries to: peer="{Nonvoter server-dbec255f-f4af-0920-107a-83477fbee278 dbec255f-f4af-0920-107a-83477fbee278}" error="failed to connect to peer: dbec255f-f4af-0920-107a-83477fbee278"
2025-01-28T14:55:50.319Z [ERROR] server-c6d0317c-047e-3058-733b-aa9010a436ca: peer has newer term, stopping replication: peer="{Nonvoter server-dbec255f-f4af-0920-107a-83477fbee278 dbec255f-f4af-0920-107a-83477fbee278}"
2025-01-28T14:55:50.320Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: entering follower state: follower="Node at c6d0317c-047e-3058-733b-aa9010a436ca [Follower]" leader-address= leader-id=
2025-01-28T14:55:50.320Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: aborting pipeline replication: peer="{Voter server-acfb0190-f267-5721-2143-bca1045c67d5 acfb0190-f267-5721-2143-bca1045c67d5}"
2025-01-28T14:55:50.323Z [ERROR] server-c6d0317c-047e-3058-733b-aa9010a436ca: peer has newer term, stopping replication: peer="{Voter server-dbec255f-f4af-0920-107a-83477fbee278 dbec255f-f4af-0920-107a-83477fbee278}"
2025-01-28T14:55:50.347Z [WARN]  server-dbec255f-f4af-0920-107a-83477fbee278: Election timeout reached, restarting election
2025-01-28T14:55:50.347Z [INFO]  server-dbec255f-f4af-0920-107a-83477fbee278: entering candidate state: node="Node at dbec255f-f4af-0920-107a-83477fbee278 [Candidate]" term=23
2025-01-28T14:55:50.347Z [WARN]  server-acfb0190-f267-5721-2143-bca1045c67d5: rejecting vote request since we have a leader: from=dbec255f-f4af-0920-107a-83477fbee278 leader=c6d0317c-047e-3058-733b-aa9010a436ca leader-id=server-c6d0317c-047e-3058-733b-aa9010a436ca
2025-01-28T14:55:50.347Z [WARN]  server-c6d0317c-047e-3058-733b-aa9010a436ca: rejecting vote request since node is not a voter: from=dbec255f-f4af-0920-107a-83477fbee278
2025-01-28T14:55:50.372Z [WARN]  server-acfb0190-f267-5721-2143-bca1045c67d5: heartbeat timeout reached, starting election: last-leader-addr=c6d0317c-047e-3058-733b-aa9010a436ca last-leader-id=server-c6d0317c-047e-3058-733b-aa9010a436ca
2025-01-28T14:55:50.372Z [INFO]  server-acfb0190-f267-5721-2143-bca1045c67d5: entering candidate state: node="Node at acfb0190-f267-5721-2143-bca1045c67d5 [Candidate]" term=3
2025-01-28T14:55:50.372Z [INFO]  server-acfb0190-f267-5721-2143-bca1045c67d5: entering follower state: follower="Node at acfb0190-f267-5721-2143-bca1045c67d5 [Follower]" leader-address= leader-id=
2025-01-28T14:55:50.394Z [WARN]  server-c6d0317c-047e-3058-733b-aa9010a436ca: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:50.394Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: entering candidate state: node="Node at c6d0317c-047e-3058-733b-aa9010a436ca [Candidate]" term=24
2025-01-28T14:55:50.395Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: election won: term=24 tally=2
2025-01-28T14:55:50.395Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: entering leader state: leader="Node at c6d0317c-047e-3058-733b-aa9010a436ca [Leader]"
2025-01-28T14:55:50.395Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: added peer, starting replication: peer=server-acfb0190-f267-5721-2143-bca1045c67d5
2025-01-28T14:55:50.395Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: added peer, starting replication: peer=server-dbec255f-f4af-0920-107a-83477fbee278
2025-01-28T14:55:50.395Z [WARN]  server-dbec255f-f4af-0920-107a-83477fbee278: failed to get previous log: previous-index=4 last-index=2 error="log not found"
2025-01-28T14:55:50.395Z [INFO]  server-dbec255f-f4af-0920-107a-83477fbee278: entering follower state: follower="Node at dbec255f-f4af-0920-107a-83477fbee278 [Follower]" leader-address=c6d0317c-047e-3058-733b-aa9010a436ca leader-id=server-c6d0317c-047e-3058-733b-aa9010a436ca
2025-01-28T14:55:50.395Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: pipelining replication: peer="{Voter server-acfb0190-f267-5721-2143-bca1045c67d5 acfb0190-f267-5721-2143-bca1045c67d5}"
2025-01-28T14:55:50.395Z [WARN]  server-c6d0317c-047e-3058-733b-aa9010a436ca: appendEntries rejected, sending older logs: peer="{Nonvoter server-dbec255f-f4af-0920-107a-83477fbee278 dbec255f-f4af-0920-107a-83477fbee278}" next=3
2025-01-28T14:55:50.395Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: pipelining replication: peer="{Nonvoter server-dbec255f-f4af-0920-107a-83477fbee278 dbec255f-f4af-0920-107a-83477fbee278}"
2025-01-28T14:55:50.404Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:50.509Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:50.404208848 +0000 UTC m=+136.078698828 (1 nodes), 146.907µs from start of poll, 492.476202ms from cluster start. Timeout at 2025-01-28 14:55:50.509421268 +0000 UTC m=+136.183911248, 105.21242ms after stability
2025-01-28T14:55:50.549Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: updating configuration: command=RemoveServer server-id=server-dbec255f-f4af-0920-107a-83477fbee278 server-addr= servers="[{Suffrage:Voter ID:server-c6d0317c-047e-3058-733b-aa9010a436ca Address:c6d0317c-047e-3058-733b-aa9010a436ca} {Suffrage:Voter ID:server-acfb0190-f267-5721-2143-bca1045c67d5 Address:acfb0190-f267-5721-2143-bca1045c67d5}]"
2025-01-28T14:55:50.549Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: removed peer, stopping replication: peer=server-dbec255f-f4af-0920-107a-83477fbee278 last-index=106
2025-01-28T14:55:50.549Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: aborting pipeline replication: peer="{Nonvoter server-dbec255f-f4af-0920-107a-83477fbee278 dbec255f-f4af-0920-107a-83477fbee278}"
2025-01-28T14:55:50.550Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: updating configuration: command=AddVoter server-id=server-dbec255f-f4af-0920-107a-83477fbee278 server-addr=dbec255f-f4af-0920-107a-83477fbee278 servers="[{Suffrage:Voter ID:server-c6d0317c-047e-3058-733b-aa9010a436ca Address:c6d0317c-047e-3058-733b-aa9010a436ca} {Suffrage:Voter ID:server-acfb0190-f267-5721-2143-bca1045c67d5 Address:acfb0190-f267-5721-2143-bca1045c67d5} {Suffrage:Voter ID:server-dbec255f-f4af-0920-107a-83477fbee278 Address:dbec255f-f4af-0920-107a-83477fbee278}]"
2025-01-28T14:55:50.551Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: added peer, starting replication: peer=server-dbec255f-f4af-0920-107a-83477fbee278
2025-01-28T14:55:50.552Z [WARN]  server-dbec255f-f4af-0920-107a-83477fbee278: failed to get previous log: previous-index=107 last-index=106 error="log not found"
2025-01-28T14:55:50.554Z [WARN]  server-c6d0317c-047e-3058-733b-aa9010a436ca: appendEntries rejected, sending older logs: peer="{Voter server-dbec255f-f4af-0920-107a-83477fbee278 dbec255f-f4af-0920-107a-83477fbee278}" next=107
2025-01-28T14:55:50.556Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: pipelining replication: peer="{Voter server-dbec255f-f4af-0920-107a-83477fbee278 dbec255f-f4af-0920-107a-83477fbee278}"
2025-01-28T14:55:51.651Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:51.758Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:51.65164901 +0000 UTC m=+137.326138990 (1 nodes), 2.36µs from start of poll, 1.739916364s from cluster start. Timeout at 2025-01-28 14:55:51.756866529 +0000 UTC m=+137.431356549, 105.217559ms after stability
2025-01-28T14:55:51.758Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: aborting pipeline replication: peer="{Voter server-dbec255f-f4af-0920-107a-83477fbee278 dbec255f-f4af-0920-107a-83477fbee278}"
2025-01-28T14:55:51.758Z [INFO]  server-c6d0317c-047e-3058-733b-aa9010a436ca: aborting pipeline replication: peer="{Voter server-acfb0190-f267-5721-2143-bca1045c67d5 acfb0190-f267-5721-2143-bca1045c67d5}"
--- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.85s)
=== RUN   TestRaft_FollowerRemovalNoElection
2025-01-28T14:55:51.762Z [INFO]  server-1916c62d-8327-2bb8-b754-91600e9a3b40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1916c62d-8327-2bb8-b754-91600e9a3b40 Address:1916c62d-8327-2bb8-b754-91600e9a3b40} {Suffrage:Voter ID:server-a06d4b15-6ea8-86ad-16af-df0261e2bfc2 Address:a06d4b15-6ea8-86ad-16af-df0261e2bfc2} {Suffrage:Voter ID:server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0 Address:c9b312d1-6b96-ac50-ba4d-5d8bff2356d0}]"
2025-01-28T14:55:51.763Z [INFO]  server-1916c62d-8327-2bb8-b754-91600e9a3b40: entering follower state: follower="Node at 1916c62d-8327-2bb8-b754-91600e9a3b40 [Follower]" leader-address= leader-id=
2025-01-28T14:55:51.763Z [INFO]  server-a06d4b15-6ea8-86ad-16af-df0261e2bfc2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1916c62d-8327-2bb8-b754-91600e9a3b40 Address:1916c62d-8327-2bb8-b754-91600e9a3b40} {Suffrage:Voter ID:server-a06d4b15-6ea8-86ad-16af-df0261e2bfc2 Address:a06d4b15-6ea8-86ad-16af-df0261e2bfc2} {Suffrage:Voter ID:server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0 Address:c9b312d1-6b96-ac50-ba4d-5d8bff2356d0}]"
2025-01-28T14:55:51.763Z [INFO]  server-a06d4b15-6ea8-86ad-16af-df0261e2bfc2: entering follower state: follower="Node at a06d4b15-6ea8-86ad-16af-df0261e2bfc2 [Follower]" leader-address= leader-id=
2025-01-28T14:55:51.763Z [INFO]  server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1916c62d-8327-2bb8-b754-91600e9a3b40 Address:1916c62d-8327-2bb8-b754-91600e9a3b40} {Suffrage:Voter ID:server-a06d4b15-6ea8-86ad-16af-df0261e2bfc2 Address:a06d4b15-6ea8-86ad-16af-df0261e2bfc2} {Suffrage:Voter ID:server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0 Address:c9b312d1-6b96-ac50-ba4d-5d8bff2356d0}]"
2025-01-28T14:55:51.763Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:51.763Z [INFO]  server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0: entering follower state: follower="Node at c9b312d1-6b96-ac50-ba4d-5d8bff2356d0 [Follower]" leader-address= leader-id=
2025-01-28T14:55:51.889Z [WARN]  server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:51.889Z [INFO]  server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0: entering candidate state: node="Node at c9b312d1-6b96-ac50-ba4d-5d8bff2356d0 [Candidate]" term=2
2025-01-28T14:55:51.890Z [INFO]  server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0: election won: term=2 tally=2
2025-01-28T14:55:51.890Z [INFO]  server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0: entering leader state: leader="Node at c9b312d1-6b96-ac50-ba4d-5d8bff2356d0 [Leader]"
2025-01-28T14:55:51.890Z [INFO]  server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0: added peer, starting replication: peer=server-1916c62d-8327-2bb8-b754-91600e9a3b40
2025-01-28T14:55:51.890Z [INFO]  server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0: added peer, starting replication: peer=server-a06d4b15-6ea8-86ad-16af-df0261e2bfc2
2025-01-28T14:55:51.890Z [INFO]  server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0: pipelining replication: peer="{Voter server-1916c62d-8327-2bb8-b754-91600e9a3b40 1916c62d-8327-2bb8-b754-91600e9a3b40}"
2025-01-28T14:55:51.890Z [INFO]  server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0: pipelining replication: peer="{Voter server-a06d4b15-6ea8-86ad-16af-df0261e2bfc2 a06d4b15-6ea8-86ad-16af-df0261e2bfc2}"
2025-01-28T14:55:52.096Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:51.890235523 +0000 UTC m=+137.564725503 (1 nodes), 126.613996ms from start of poll, 127.683775ms from cluster start. Timeout at 2025-01-28 14:55:52.095239522 +0000 UTC m=+137.769729542, 205.004039ms after stability
2025-01-28T14:55:52.096Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:52.302Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:52.09664893 +0000 UTC m=+137.771138890 (1 nodes), 2.119µs from start of poll, 334.097162ms from cluster start. Timeout at 2025-01-28 14:55:52.301913825 +0000 UTC m=+137.976403805, 205.264915ms after stability
2025-01-28T14:55:52.307Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:52.513Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:52.307877764 +0000 UTC m=+137.982367744 (2 nodes), 2.86µs from start of poll, 545.326016ms from cluster start. Timeout at 2025-01-28 14:55:52.512982894 +0000 UTC m=+138.187472934, 205.10519ms after stability
2025-01-28T14:55:52.513Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:52.719Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:52.513834835 +0000 UTC m=+138.188324835 (1 nodes), 2.08µs from start of poll, 751.283107ms from cluster start. Timeout at 2025-01-28 14:55:52.7189997 +0000 UTC m=+138.393489680, 205.164845ms after stability
    raft_test.go:2961: [INFO] restarting Node at 1916c62d-8327-2bb8-b754-91600e9a3b40 [Follower]
2025-01-28T14:55:52.720Z [INFO]  server-1916c62d-8327-2bb8-b754-91600e9a3b40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1916c62d-8327-2bb8-b754-91600e9a3b40 Address:1916c62d-8327-2bb8-b754-91600e9a3b40} {Suffrage:Voter ID:server-a06d4b15-6ea8-86ad-16af-df0261e2bfc2 Address:a06d4b15-6ea8-86ad-16af-df0261e2bfc2} {Suffrage:Voter ID:server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0 Address:c9b312d1-6b96-ac50-ba4d-5d8bff2356d0}]"
2025-01-28T14:55:52.720Z [INFO]  server-1916c62d-8327-2bb8-b754-91600e9a3b40: entering follower state: follower="Node at 1916c62d-8327-2bb8-b754-91600e9a3b40 [Follower]" leader-address= leader-id=
2025-01-28T14:55:52.971Z [INFO]  server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0: aborting pipeline replication: peer="{Voter server-a06d4b15-6ea8-86ad-16af-df0261e2bfc2 a06d4b15-6ea8-86ad-16af-df0261e2bfc2}"
2025-01-28T14:55:53.349Z [ERROR] server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0: failed to pipeline appendEntries: peer="{Voter server-1916c62d-8327-2bb8-b754-91600e9a3b40 1916c62d-8327-2bb8-b754-91600e9a3b40}" error="command enqueue timeout"
2025-01-28T14:55:53.349Z [INFO]  server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0: aborting pipeline replication: peer="{Voter server-1916c62d-8327-2bb8-b754-91600e9a3b40 1916c62d-8327-2bb8-b754-91600e9a3b40}"
2025-01-28T14:55:53.483Z [ERROR] server-c9b312d1-6b96-ac50-ba4d-5d8bff2356d0: failed to heartbeat to: peer=1916c62d-8327-2bb8-b754-91600e9a3b40 backoff time=10ms error="command timed out"
--- PASS: TestRaft_FollowerRemovalNoElection (1.72s)
=== RUN   TestRaft_VoteWithNoIDNoAddr
2025-01-28T14:55:53.486Z [INFO]  server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2 Address:3f95db3c-8f17-8b74-4c5c-2a59bde16fe2} {Suffrage:Voter ID:server-1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a Address:1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a} {Suffrage:Voter ID:server-71e5a17f-d253-b322-3d77-b5e98015a4b0 Address:71e5a17f-d253-b322-3d77-b5e98015a4b0}]"
2025-01-28T14:55:53.487Z [INFO]  server-1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2 Address:3f95db3c-8f17-8b74-4c5c-2a59bde16fe2} {Suffrage:Voter ID:server-1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a Address:1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a} {Suffrage:Voter ID:server-71e5a17f-d253-b322-3d77-b5e98015a4b0 Address:71e5a17f-d253-b322-3d77-b5e98015a4b0}]"
2025-01-28T14:55:53.487Z [INFO]  server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2: entering follower state: follower="Node at 3f95db3c-8f17-8b74-4c5c-2a59bde16fe2 [Follower]" leader-address= leader-id=
2025-01-28T14:55:53.487Z [INFO]  server-1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a: entering follower state: follower="Node at 1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a [Follower]" leader-address= leader-id=
2025-01-28T14:55:53.487Z [INFO]  server-71e5a17f-d253-b322-3d77-b5e98015a4b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2 Address:3f95db3c-8f17-8b74-4c5c-2a59bde16fe2} {Suffrage:Voter ID:server-1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a Address:1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a} {Suffrage:Voter ID:server-71e5a17f-d253-b322-3d77-b5e98015a4b0 Address:71e5a17f-d253-b322-3d77-b5e98015a4b0}]"
2025-01-28T14:55:53.487Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:53.487Z [INFO]  server-71e5a17f-d253-b322-3d77-b5e98015a4b0: entering follower state: follower="Node at 71e5a17f-d253-b322-3d77-b5e98015a4b0 [Follower]" leader-address= leader-id=
2025-01-28T14:55:53.540Z [WARN]  server-71e5a17f-d253-b322-3d77-b5e98015a4b0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-01-28T14:55:53.540Z [INFO]  server-71e5a17f-d253-b322-3d77-b5e98015a4b0: entering candidate state: node="Node at 71e5a17f-d253-b322-3d77-b5e98015a4b0 [Candidate]" term=2
2025-01-28T14:55:53.540Z [INFO]  server-71e5a17f-d253-b322-3d77-b5e98015a4b0: election won: term=2 tally=2
2025-01-28T14:55:53.540Z [INFO]  server-71e5a17f-d253-b322-3d77-b5e98015a4b0: entering leader state: leader="Node at 71e5a17f-d253-b322-3d77-b5e98015a4b0 [Leader]"
2025-01-28T14:55:53.540Z [INFO]  server-71e5a17f-d253-b322-3d77-b5e98015a4b0: added peer, starting replication: peer=server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2
2025-01-28T14:55:53.540Z [INFO]  server-71e5a17f-d253-b322-3d77-b5e98015a4b0: added peer, starting replication: peer=server-1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a
2025-01-28T14:55:53.541Z [INFO]  server-71e5a17f-d253-b322-3d77-b5e98015a4b0: pipelining replication: peer="{Voter server-1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a 1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a}"
2025-01-28T14:55:53.541Z [INFO]  server-71e5a17f-d253-b322-3d77-b5e98015a4b0: pipelining replication: peer="{Voter server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2 3f95db3c-8f17-8b74-4c5c-2a59bde16fe2}"
2025-01-28T14:55:53.647Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:53.540989478 +0000 UTC m=+139.215479498 (1 nodes), 53.198101ms from start of poll, 54.321076ms from cluster start. Timeout at 2025-01-28 14:55:53.645995038 +0000 UTC m=+139.320485198, 105.0057ms after stability
2025-01-28T14:55:53.647Z [INFO]  cluster: starting stability test: raft-state=Leader
2025-01-28T14:55:53.752Z [INFO]  cluster: stable state for Leader reached at 2025-01-28 14:55:53.647179541 +0000 UTC m=+139.321669541 (1 nodes), 2.32µs from start of poll, 160.511119ms from cluster start. Timeout at 2025-01-28 14:55:53.752367963 +0000 UTC m=+139.426858003, 105.188462ms after stability
2025-01-28T14:55:53.757Z [INFO]  cluster: starting stability test: raft-state=Follower
2025-01-28T14:55:53.863Z [INFO]  cluster: stable state for Follower reached at 2025-01-28 14:55:53.757755689 +0000 UTC m=+139.432245669 (2 nodes), 3.659µs from start of poll, 271.087247ms from cluster start. Timeout at 2025-01-28 14:55:53.862931413 +0000 UTC m=+139.537421413, 105.175744ms after stability
2025-01-28T14:55:53.863Z [ERROR] server-71e5a17f-d253-b322-3d77-b5e98015a4b0: failed to pipeline appendEntries: peer="{Voter server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2 3f95db3c-8f17-8b74-4c5c-2a59bde16fe2}" error="append pipeline closed"
2025-01-28T14:55:53.863Z [INFO]  server-71e5a17f-d253-b322-3d77-b5e98015a4b0: aborting pipeline replication: peer="{Voter server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2 3f95db3c-8f17-8b74-4c5c-2a59bde16fe2}"
2025-01-28T14:55:53.863Z [ERROR] server-71e5a17f-d253-b322-3d77-b5e98015a4b0: failed to heartbeat to: peer=1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a backoff time=10ms error="failed to connect to peer: 1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a"
2025-01-28T14:55:53.866Z [ERROR] server-71e5a17f-d253-b322-3d77-b5e98015a4b0: failed to heartbeat to: peer=3f95db3c-8f17-8b74-4c5c-2a59bde16fe2 backoff time=10ms error="failed to connect to peer: 3f95db3c-8f17-8b74-4c5c-2a59bde16fe2"
2025-01-28T14:55:53.870Z [ERROR] server-71e5a17f-d253-b322-3d77-b5e98015a4b0: failed to pipeline appendEntries: peer="{Voter server-1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a 1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a}" error="append pipeline closed"
2025-01-28T14:55:53.870Z [INFO]  server-71e5a17f-d253-b322-3d77-b5e98015a4b0: aborting pipeline replication: peer="{Voter server-1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a 1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a}"
2025-01-28T14:55:53.870Z [ERROR] server-71e5a17f-d253-b322-3d77-b5e98015a4b0: failed to appendEntries to: peer="{Voter server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2 3f95db3c-8f17-8b74-4c5c-2a59bde16fe2}" error="failed to connect to peer: 3f95db3c-8f17-8b74-4c5c-2a59bde16fe2"
2025-01-28T14:55:53.879Z [ERROR] server-71e5a17f-d253-b322-3d77-b5e98015a4b0: failed to appendEntries to: peer="{Voter server-1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a 1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a}" error="failed to connect to peer: 1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a"
2025-01-28T14:55:53.885Z [ERROR] server-71e5a17f-d253-b322-3d77-b5e98015a4b0: failed to heartbeat to: peer=1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a backoff time=10ms error="failed to connect to peer: 1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a"
2025-01-28T14:55:53.886Z [ERROR] server-71e5a17f-d253-b322-3d77-b5e98015a4b0: failed to heartbeat to: peer=3f95db3c-8f17-8b74-4c5c-2a59bde16fe2 backoff time=10ms error="failed to connect to peer: 3f95db3c-8f17-8b74-4c5c-2a59bde16fe2"
2025-01-28T14:55:53.889Z [ERROR] server-71e5a17f-d253-b322-3d77-b5e98015a4b0: failed to appendEntries to: peer="{Voter server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2 3f95db3c-8f17-8b74-4c5c-2a59bde16fe2}" error="failed to connect to peer: 3f95db3c-8f17-8b74-4c5c-2a59bde16fe2"
2025-01-28T14:55:53.896Z [ERROR] server-71e5a17f-d253-b322-3d77-b5e98015a4b0: failed to appendEntries to: peer="{Voter server-1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a 1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a}" error="failed to connect to peer: 1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a"
2025-01-28T14:55:53.902Z [ERROR] server-71e5a17f-d253-b322-3d77-b5e98015a4b0: failed to heartbeat to: peer=1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a backoff time=10ms error="failed to connect to peer: 1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a"
2025-01-28T14:55:53.905Z [ERROR] server-71e5a17f-d253-b322-3d77-b5e98015a4b0: failed to appendEntries to: peer="{Voter server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2 3f95db3c-8f17-8b74-4c5c-2a59bde16fe2}" error="failed to connect to peer: 3f95db3c-8f17-8b74-4c5c-2a59bde16fe2"
2025-01-28T14:55:53.906Z [ERROR] server-71e5a17f-d253-b322-3d77-b5e98015a4b0: failed to heartbeat to: peer=3f95db3c-8f17-8b74-4c5c-2a59bde16fe2 backoff time=10ms error="failed to connect to peer: 3f95db3c-8f17-8b74-4c5c-2a59bde16fe2"
2025-01-28T14:55:53.910Z [WARN]  server-71e5a17f-d253-b322-3d77-b5e98015a4b0: failed to contact: server-id=server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2 time=50.746872ms
2025-01-28T14:55:53.910Z [WARN]  server-71e5a17f-d253-b322-3d77-b5e98015a4b0: failed to contact: server-id=server-1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a time=50.432002ms
2025-01-28T14:55:53.910Z [WARN]  server-71e5a17f-d253-b322-3d77-b5e98015a4b0: failed to contact quorum of nodes, stepping down
2025-01-28T14:55:53.910Z [INFO]  server-71e5a17f-d253-b322-3d77-b5e98015a4b0: entering follower state: follower="Node at 71e5a17f-d253-b322-3d77-b5e98015a4b0 [Follower]" leader-address= leader-id=
2025-01-28T14:55:53.914Z [ERROR] server-71e5a17f-d253-b322-3d77-b5e98015a4b0: failed to appendEntries to: peer="{Voter server-1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a 1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a}" error="failed to connect to peer: 1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a"
2025-01-28T14:55:53.940Z [WARN]  server-1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a: heartbeat timeout reached, starting election: last-leader-addr=71e5a17f-d253-b322-3d77-b5e98015a4b0 last-leader-id=server-71e5a17f-d253-b322-3d77-b5e98015a4b0
2025-01-28T14:55:53.941Z [INFO]  server-1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a: entering candidate state: node="Node at 1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a [Candidate]" term=3
2025-01-28T14:55:53.941Z [ERROR] server-1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a: failed to make requestVote RPC: target="{Voter server-71e5a17f-d253-b322-3d77-b5e98015a4b0 71e5a17f-d253-b322-3d77-b5e98015a4b0}" error="failed to connect to peer: 71e5a17f-d253-b322-3d77-b5e98015a4b0" term=3
2025-01-28T14:55:53.941Z [WARN]  server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2: rejecting vote request since we have a leader: from=1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a leader=71e5a17f-d253-b322-3d77-b5e98015a4b0 leader-id=server-71e5a17f-d253-b322-3d77-b5e98015a4b0
2025-01-28T14:55:53.946Z [WARN]  server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2: heartbeat timeout reached, starting election: last-leader-addr=71e5a17f-d253-b322-3d77-b5e98015a4b0 last-leader-id=server-71e5a17f-d253-b322-3d77-b5e98015a4b0
2025-01-28T14:55:53.946Z [INFO]  server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2: entering candidate state: node="Node at 3f95db3c-8f17-8b74-4c5c-2a59bde16fe2 [Candidate]" term=3
2025-01-28T14:55:53.946Z [ERROR] server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2: failed to make requestVote RPC: target="{Voter server-71e5a17f-d253-b322-3d77-b5e98015a4b0 71e5a17f-d253-b322-3d77-b5e98015a4b0}" error="failed to connect to peer: 71e5a17f-d253-b322-3d77-b5e98015a4b0" term=3
2025-01-28T14:55:53.946Z [INFO]  server-1111ca03-b3b9-82b4-1d43-82fc4cb0fc8a: duplicate requestVote for same term: term=3
2025-01-28T14:55:53.968Z [INFO]  server-3f95db3c-8f17-8b74-4c5c-2a59bde16fe2: entering follower state: follower="Node at 3f95db3c-8f17-8b74-4c5c-2a59bde16fe2 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VoteWithNoIDNoAddr (0.49s)
=== RUN   TestRaft_runFollower_State_Transition
=== RUN   TestRaft_runFollower_State_Transition/NonVoter
2025-01-28T14:55:53.972Z [INFO]  127.0.0.1:33367 :: starting node: addr=127.0.0.1:33367
2025-01-28T14:55:53.972Z [INFO]  127.0.0.1:33367 :: initial configuration: index=0 servers=[]
2025-01-28T14:55:53.972Z [INFO]  127.0.0.1:33367 :: entering follower state: follower="Node at 127.0.0.1:33367 [Follower]" leader-address= leader-id=
2025-01-28T14:55:54.066Z [WARN]  127.0.0.1:33367 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
=== RUN   TestRaft_runFollower_State_Transition/Voter
2025-01-28T14:55:54.124Z [INFO]  127.0.0.1:43511 :: starting node: addr=127.0.0.1:43511
2025-01-28T14:55:54.124Z [INFO]  127.0.0.1:43511 :: initial configuration: index=0 servers=[]
2025-01-28T14:55:54.124Z [INFO]  127.0.0.1:43511 :: entering follower state: follower="Node at 127.0.0.1:43511 [Follower]" leader-address= leader-id=
2025-01-28T14:55:54.189Z [WARN]  127.0.0.1:43511 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
=== RUN   TestRaft_runFollower_State_Transition/Not_in_Config
2025-01-28T14:55:54.276Z [INFO]  127.0.0.1:44733 :: starting node: addr=127.0.0.1:44733
2025-01-28T14:55:54.276Z [INFO]  127.0.0.1:44733 :: initial configuration: index=0 servers=[]
2025-01-28T14:55:54.276Z [INFO]  127.0.0.1:44733 :: entering follower state: follower="Node at 127.0.0.1:44733 [Follower]" leader-address= leader-id=
2025-01-28T14:55:54.340Z [WARN]  127.0.0.1:44733 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
--- PASS: TestRaft_runFollower_State_Transition (0.46s)
    --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Voter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s)
=== RUN   TestRaft_runFollower_ReloadTimeoutConfigs
2025-01-28T14:55:54.428Z [INFO]  127.0.0.1:45233 :: starting node: addr=127.0.0.1:45233
2025-01-28T14:55:54.428Z [INFO]  127.0.0.1:45233 :: initial configuration: index=0 servers=[]
2025-01-28T14:55:54.428Z [INFO]  127.0.0.1:45233 :: entering follower state: follower="Node at 127.0.0.1:45233 [Follower]" leader-address= leader-id=
2025-01-28T14:55:54.428Z [WARN]  127.0.0.1:45233 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
--- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s)
=== RUN   TestSaturationMetric
=== RUN   TestSaturationMetric/without_smoothing
--- PASS: TestSaturationMetric (0.00s)
    --- PASS: TestSaturationMetric/without_smoothing (0.00s)
=== RUN   TestSaturationMetric_IncorrectUsage
=== RUN   TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_first
--- PASS: TestSaturationMetric_IncorrectUsage (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s)
=== RUN   TestTCPTransport_BadAddr
--- PASS: TestTCPTransport_BadAddr (0.00s)
=== RUN   TestTCPTransport_EmptyAddr
--- PASS: TestTCPTransport_EmptyAddr (0.00s)
=== RUN   TestTCPTransport_WithAdvertise
--- PASS: TestTCPTransport_WithAdvertise (0.00s)
=== RUN   TestTransport_StartStop
--- PASS: TestTransport_StartStop (0.00s)
=== RUN   TestTransport_AppendEntries
--- PASS: TestTransport_AppendEntries (0.00s)
=== RUN   TestTransport_AppendEntriesPipeline
--- PASS: TestTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestTransport_RequestVote
--- PASS: TestTransport_RequestVote (0.00s)
=== RUN   TestTransport_InstallSnapshot
--- PASS: TestTransport_InstallSnapshot (0.00s)
=== RUN   TestTransport_EncodeDecode
--- PASS: TestTransport_EncodeDecode (0.00s)
=== RUN   TestRandomTimeout
--- PASS: TestRandomTimeout (0.00s)
=== RUN   TestNewSeed
--- PASS: TestNewSeed (0.01s)
=== RUN   TestRandomTimeout_NoTime
--- PASS: TestRandomTimeout_NoTime (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
=== RUN   TestMax
--- PASS: TestMax (0.00s)
=== RUN   TestGenerateUUID
--- PASS: TestGenerateUUID (0.00s)
=== RUN   TestBackoff
--- PASS: TestBackoff (0.00s)
=== RUN   TestOverrideNotifyBool
--- PASS: TestOverrideNotifyBool (0.00s)
PASS
ok  	github.com/hashicorp/raft	140.280s
make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
   create-stamp debian/debhelper-build-stamp
   dh_testroot -O--buildsystem=golang
   dh_prep -O--buildsystem=golang
   dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang
   dh_installdocs -O--buildsystem=golang
   dh_installchangelogs -O--buildsystem=golang
   dh_installsystemduser -O--buildsystem=golang
   dh_perl -O--buildsystem=golang
   dh_link -O--buildsystem=golang
   dh_strip_nondeterminism -O--buildsystem=golang
   dh_compress -O--buildsystem=golang
   dh_fixperms -O--buildsystem=golang
   dh_missing -O--buildsystem=golang
   dh_installdeb -O--buildsystem=golang
   dh_golang -O--buildsystem=golang
   dh_gencontrol -O--buildsystem=golang
   dh_md5sums -O--buildsystem=golang
   dh_builddeb -O--buildsystem=golang
dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'.
 dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_armhf.buildinfo
 dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_armhf.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-buildpackage: info: binary-only upload (no source included)
dpkg-genchanges: info: including full source code in upload
I: copying local configuration
I: user script /srv/workspace/pbuilder/31365/tmp/hooks/B01_cleanup starting
I: user script /srv/workspace/pbuilder/31365/tmp/hooks/B01_cleanup finished
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /srv/workspace/pbuilder/31365 and its subdirectories
I: Current time: Wed Jan 29 04:56:10 +14 2025
I: pbuilder-time-stamp: 1738076170