I: pbuilder: network access will be disabled during build
I: Current time: Tue Dec  3 13:29:28 -12 2024
I: pbuilder-time-stamp: 1733275768
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/trixie-reproducible-base.tgz]
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: policy-rc.d already exists
I: using eatmydata during job
I: Copying source file
I: copying [golang-github-hashicorp-raft_1.5.0-1.dsc]
I: copying [./golang-github-hashicorp-raft_1.5.0.orig.tar.gz]
I: copying [./golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz]
I: Extracting source
gpgv: Signature made Thu Jul 27 15:15:46 2023 
gpgv:                using RSA key 303F687A4EFCB1AE7C1A35A3B700173FBB805A29
gpgv:                issuer "dswarbrick@debian.org"
gpgv: Can't check signature: No public key
dpkg-source: warning: cannot verify inline signature for ./golang-github-hashicorp-raft_1.5.0-1.dsc: no acceptable signature found
dpkg-source: info: extracting golang-github-hashicorp-raft in golang-github-hashicorp-raft-1.5.0
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0.orig.tar.gz
dpkg-source: info: unpacking golang-github-hashicorp-raft_1.5.0-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying 0001-Skip-unreliable-tests.patch
I: Not using root during the build.
I: Installing the build-deps
I: user script /srv/workspace/pbuilder/25276/tmp/hooks/D02_print_environment starting
I: set
  BUILDDIR='/build/reproducible-path'
  BUILDUSERGECOS='first user,first room,first work-phone,first home-phone,first other'
  BUILDUSERNAME='pbuilder1'
  BUILD_ARCH='i386'
  DEBIAN_FRONTEND='noninteractive'
  DEB_BUILD_OPTIONS='buildinfo=+all reproducible=+all parallel=11 '
  DISTRIBUTION='trixie'
  HOME='/root'
  HOST_ARCH='i386'
  IFS=' 	
  '
  INVOCATION_ID='96a4616a6bba41e0ab7fae82ea5b9a74'
  LANG='C'
  LANGUAGE='en_US:en'
  LC_ALL='C'
  LD_LIBRARY_PATH='/usr/lib/libeatmydata'
  LD_PRELOAD='libeatmydata.so'
  MAIL='/var/mail/root'
  OPTIND='1'
  PATH='/usr/sbin:/usr/bin:/sbin:/bin:/usr/games'
  PBCURRENTCOMMANDLINEOPERATION='build'
  PBUILDER_OPERATION='build'
  PBUILDER_PKGDATADIR='/usr/share/pbuilder'
  PBUILDER_PKGLIBDIR='/usr/lib/pbuilder'
  PBUILDER_SYSCONFDIR='/etc'
  PPID='25276'
  PS1='# '
  PS2='> '
  PS4='+ '
  PWD='/'
  SHELL='/bin/bash'
  SHLVL='2'
  SUDO_COMMAND='/usr/bin/timeout -k 18.1h 18h /usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/pbuilder --build --configfile /srv/reproducible-results/rbuild-debian/r-b-build.s7UhExSL/pbuilderrc_pbZA --distribution trixie --hookdir /etc/pbuilder/first-build-hooks --debbuildopts -b --basetgz /var/cache/pbuilder/trixie-reproducible-base.tgz --buildresult /srv/reproducible-results/rbuild-debian/r-b-build.s7UhExSL/b1 --logfile b1/build.log golang-github-hashicorp-raft_1.5.0-1.dsc'
  SUDO_GID='112'
  SUDO_UID='107'
  SUDO_USER='jenkins'
  TERM='unknown'
  TZ='/usr/share/zoneinfo/Etc/GMT+12'
  USER='root'
  _='/usr/bin/systemd-run'
  http_proxy='http://46.16.76.132:3128'
I: uname -a
  Linux ionos2-i386 6.1.0-28-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.119-1 (2024-11-22) x86_64 GNU/Linux
I: ls -l /bin
  lrwxrwxrwx 1 root root 7 Nov 22 14:40 /bin -> usr/bin
I: user script /srv/workspace/pbuilder/25276/tmp/hooks/D02_print_environment finished
 -> Attempting to satisfy build-dependencies
 -> Creating pbuilder-satisfydepends-dummy package
Package: pbuilder-satisfydepends-dummy
Version: 0.invalid.0
Architecture: i386
Maintainer: Debian Pbuilder Team <pbuilder-maint@lists.alioth.debian.org>
Description: Dummy package to satisfy dependencies with aptitude - created by pbuilder
 This package was created automatically by pbuilder to satisfy the
 build-dependencies of the package being currently built.
Depends: debhelper-compat (= 13), dh-sequence-golang, golang-any, golang-github-armon-go-metrics-dev, golang-github-hashicorp-go-hclog-dev (>= 0.9.2~), golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~), golang-github-stretchr-testify-dev
dpkg-deb: building package 'pbuilder-satisfydepends-dummy' in '/tmp/satisfydepends-aptitude/pbuilder-satisfydepends-dummy.deb'.
Selecting previously unselected package pbuilder-satisfydepends-dummy.
(Reading database ... 19950 files and directories currently installed.)
Preparing to unpack .../pbuilder-satisfydepends-dummy.deb ...
Unpacking pbuilder-satisfydepends-dummy (0.invalid.0) ...
dpkg: pbuilder-satisfydepends-dummy: dependency problems, but configuring anyway as you requested:
 pbuilder-satisfydepends-dummy depends on debhelper-compat (= 13); however:
  Package debhelper-compat is not installed.
 pbuilder-satisfydepends-dummy depends on dh-sequence-golang; however:
  Package dh-sequence-golang is not installed.
 pbuilder-satisfydepends-dummy depends on golang-any; however:
  Package golang-any is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-armon-go-metrics-dev; however:
  Package golang-github-armon-go-metrics-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-hclog-dev (>= 0.9.2~); however:
  Package golang-github-hashicorp-go-hclog-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-hashicorp-go-msgpack-dev (>= 0.5.5~); however:
  Package golang-github-hashicorp-go-msgpack-dev is not installed.
 pbuilder-satisfydepends-dummy depends on golang-github-stretchr-testify-dev; however:
  Package golang-github-stretchr-testify-dev is not installed.

Setting up pbuilder-satisfydepends-dummy (0.invalid.0) ...
Reading package lists...
Building dependency tree...
Reading state information...
Initializing package states...
Writing extended state information...
Building tag database...
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
pbuilder-satisfydepends-dummy is already installed at the requested version (0.invalid.0)
The following NEW packages will be installed:
  autoconf{a} automake{a} autopoint{a} autotools-dev{a} bsdextrautils{a} debhelper{a} dh-autoreconf{a} dh-golang{a} dh-strip-nondeterminism{a} dwz{a} file{a} gettext{a} gettext-base{a} golang-1.23-go{a} golang-1.23-src{a} golang-any{a} golang-github-alecthomas-units-dev{a} golang-github-armon-go-metrics-dev{a} golang-github-aws-aws-sdk-go-dev{a} golang-github-beorn7-perks-dev{a} golang-github-cespare-xxhash-dev{a} golang-github-circonus-labs-circonus-gometrics-dev{a} golang-github-circonus-labs-circonusllhist-dev{a} golang-github-creack-pty-dev{a} golang-github-datadog-datadog-go-dev{a} golang-github-davecgh-go-spew-dev{a} golang-github-fatih-color-dev{a} golang-github-go-kit-log-dev{a} golang-github-go-logfmt-logfmt-dev{a} golang-github-golang-protobuf-1-5-dev{a} golang-github-google-go-cmp-dev{a} golang-github-hashicorp-go-cleanhttp-dev{a} golang-github-hashicorp-go-hclog-dev{a} golang-github-hashicorp-go-immutable-radix-dev{a} golang-github-hashicorp-go-msgpack-dev{a} golang-github-hashicorp-go-retryablehttp-dev{a} golang-github-hashicorp-go-uuid-dev{a} golang-github-hashicorp-golang-lru-dev{a} golang-github-jmespath-go-jmespath-dev{a} golang-github-jpillora-backoff-dev{a} golang-github-json-iterator-go-dev{a} golang-github-julienschmidt-httprouter-dev{a} golang-github-klauspost-compress-dev{a} golang-github-kr-pretty-dev{a} golang-github-kr-text-dev{a} golang-github-kylelemons-godebug-dev{a} golang-github-mattn-go-colorable-dev{a} golang-github-mattn-go-isatty-dev{a} golang-github-modern-go-concurrent-dev{a} golang-github-modern-go-reflect2-dev{a} golang-github-munnerz-goautoneg-dev{a} golang-github-mwitkow-go-conntrack-dev{a} golang-github-pascaldekloe-goe-dev{a} golang-github-pkg-errors-dev{a} golang-github-pmezard-go-difflib-dev{a} golang-github-prometheus-client-golang-dev{a} golang-github-prometheus-client-model-dev{a} golang-github-prometheus-common-dev{a} golang-github-prometheus-procfs-dev{a} golang-github-rogpeppe-go-internal-dev{a} golang-github-stretchr-objx-dev{a} golang-github-stretchr-testify-dev{a} golang-github-tv42-httpunix-dev{a} golang-github-vmihailenco-tagparser-dev{a} golang-github-yuin-goldmark-dev{a} golang-go{a} golang-golang-x-crypto-dev{a} golang-golang-x-mod-dev{a} golang-golang-x-net-dev{a} golang-golang-x-oauth2-dev{a} golang-golang-x-sync-dev{a} golang-golang-x-sys-dev{a} golang-golang-x-term-dev{a} golang-golang-x-text-dev{a} golang-golang-x-tools-dev{a} golang-google-protobuf-dev{a} golang-gopkg-alecthomas-kingpin.v2-dev{a} golang-gopkg-check.v1-dev{a} golang-gopkg-mgo.v2-dev{a} golang-gopkg-tomb.v2-dev{a} golang-gopkg-vmihailenco-msgpack.v2-dev{a} golang-gopkg-yaml.v2-dev{a} golang-gopkg-yaml.v3-dev{a} golang-src{a} groff-base{a} intltool-debian{a} libarchive-zip-perl{a} libdebhelper-perl{a} libelf1t64{a} libfile-stripnondeterminism-perl{a} libicu72{a} libmagic-mgc{a} libmagic1t64{a} libpipeline1{a} libsasl2-2{a} libsasl2-dev{a} libsasl2-modules-db{a} libssl-dev{a} libtool{a} libuchardet0{a} libxml2{a} m4{a} man-db{a} po-debconf{a} sensible-utils{a} 
The following packages are RECOMMENDED but will NOT be installed:
  curl libarchive-cpio-perl libltdl-dev libmail-sendmail-perl libsasl2-modules lynx pkgconf wget 
0 packages upgraded, 105 newly installed, 0 to remove and 0 not upgraded.
Need to get 125 MB of archives. After unpacking 682 MB will be used.
Writing extended state information...
Get: 1 http://deb.debian.org/debian trixie/main i386 sensible-utils all 0.0.24 [24.8 kB]
Get: 2 http://deb.debian.org/debian trixie/main i386 libmagic-mgc i386 1:5.45-3+b1 [314 kB]
Get: 3 http://deb.debian.org/debian trixie/main i386 libmagic1t64 i386 1:5.45-3+b1 [115 kB]
Get: 4 http://deb.debian.org/debian trixie/main i386 file i386 1:5.45-3+b1 [43.2 kB]
Get: 5 http://deb.debian.org/debian trixie/main i386 gettext-base i386 0.22.5-2 [201 kB]
Get: 6 http://deb.debian.org/debian trixie/main i386 libuchardet0 i386 0.0.8-1+b2 [69.2 kB]
Get: 7 http://deb.debian.org/debian trixie/main i386 groff-base i386 1.23.0-5 [1196 kB]
Get: 8 http://deb.debian.org/debian trixie/main i386 bsdextrautils i386 2.40.2-11 [95.6 kB]
Get: 9 http://deb.debian.org/debian trixie/main i386 libpipeline1 i386 1.5.8-1 [41.2 kB]
Get: 10 http://deb.debian.org/debian trixie/main i386 man-db i386 2.13.0-1 [1428 kB]
Get: 11 http://deb.debian.org/debian trixie/main i386 m4 i386 1.4.19-4 [293 kB]
Get: 12 http://deb.debian.org/debian trixie/main i386 autoconf all 2.72-3 [493 kB]
Get: 13 http://deb.debian.org/debian trixie/main i386 autotools-dev all 20220109.1 [51.6 kB]
Get: 14 http://deb.debian.org/debian trixie/main i386 automake all 1:1.16.5-1.3 [823 kB]
Get: 15 http://deb.debian.org/debian trixie/main i386 autopoint all 0.22.5-2 [723 kB]
Get: 16 http://deb.debian.org/debian trixie/main i386 libdebhelper-perl all 13.20 [89.7 kB]
Get: 17 http://deb.debian.org/debian trixie/main i386 libtool all 2.4.7-8 [517 kB]
Get: 18 http://deb.debian.org/debian trixie/main i386 dh-autoreconf all 20 [17.1 kB]
Get: 19 http://deb.debian.org/debian trixie/main i386 libarchive-zip-perl all 1.68-1 [104 kB]
Get: 20 http://deb.debian.org/debian trixie/main i386 libfile-stripnondeterminism-perl all 1.14.0-1 [19.5 kB]
Get: 21 http://deb.debian.org/debian trixie/main i386 dh-strip-nondeterminism all 1.14.0-1 [8448 B]
Get: 22 http://deb.debian.org/debian trixie/main i386 libelf1t64 i386 0.192-4 [195 kB]
Get: 23 http://deb.debian.org/debian trixie/main i386 dwz i386 0.15-1+b1 [116 kB]
Get: 24 http://deb.debian.org/debian trixie/main i386 libicu72 i386 72.1-5+b1 [9583 kB]
Get: 25 http://deb.debian.org/debian trixie/main i386 libxml2 i386 2.12.7+dfsg+really2.9.14-0.2+b1 [734 kB]
Get: 26 http://deb.debian.org/debian trixie/main i386 gettext i386 0.22.5-2 [1631 kB]
Get: 27 http://deb.debian.org/debian trixie/main i386 intltool-debian all 0.35.0+20060710.6 [22.9 kB]
Get: 28 http://deb.debian.org/debian trixie/main i386 po-debconf all 1.0.21+nmu1 [248 kB]
Get: 29 http://deb.debian.org/debian trixie/main i386 debhelper all 13.20 [915 kB]
Get: 30 http://deb.debian.org/debian trixie/main i386 dh-golang all 1.62 [24.7 kB]
Get: 31 http://deb.debian.org/debian trixie/main i386 golang-1.23-src all 1.23.3-2 [19.1 MB]
Get: 32 http://deb.debian.org/debian trixie/main i386 golang-1.23-go i386 1.23.3-2 [25.1 MB]
Get: 33 http://deb.debian.org/debian trixie/main i386 golang-src all 2:1.23~2 [5088 B]
Get: 34 http://deb.debian.org/debian trixie/main i386 golang-go i386 2:1.23~2 [44.3 kB]
Get: 35 http://deb.debian.org/debian trixie/main i386 golang-any i386 2:1.23~2 [5168 B]
Get: 36 http://deb.debian.org/debian trixie/main i386 golang-github-alecthomas-units-dev all 0.0~git20240626.61d1e34-1 [9032 B]
Get: 37 http://deb.debian.org/debian trixie/main i386 golang-github-circonus-labs-circonusllhist-dev all 0.0~git20191022.ec08cde-1 [11.1 kB]
Get: 38 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-cleanhttp-dev all 0.5.2-1 [10.8 kB]
Get: 39 http://deb.debian.org/debian trixie/main i386 golang-golang-x-sys-dev all 0.22.0-1 [431 kB]
Get: 40 http://deb.debian.org/debian trixie/main i386 golang-github-mattn-go-isatty-dev all 0.0.20-1 [6080 B]
Get: 41 http://deb.debian.org/debian trixie/main i386 golang-github-mattn-go-colorable-dev all 0.1.13-1 [10.3 kB]
Get: 42 http://deb.debian.org/debian trixie/main i386 golang-github-fatih-color-dev all 1.17.0-1 [13.9 kB]
Get: 43 http://deb.debian.org/debian trixie/main i386 golang-github-davecgh-go-spew-dev all 1.1.1-3 [30.1 kB]
Get: 44 http://deb.debian.org/debian trixie/main i386 golang-github-pmezard-go-difflib-dev all 1.0.0-3 [12.3 kB]
Get: 45 http://deb.debian.org/debian trixie/main i386 golang-github-stretchr-objx-dev all 0.5.2-1 [25.5 kB]
Get: 46 http://deb.debian.org/debian trixie/main i386 golang-gopkg-yaml.v3-dev all 3.0.1-3 [74.3 kB]
Get: 47 http://deb.debian.org/debian trixie/main i386 golang-github-stretchr-testify-dev all 1.9.0-1 [74.7 kB]
Get: 48 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-hclog-dev all 0.11.0-1 [17.9 kB]
Get: 49 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-retryablehttp-dev all 0.7.1-1 [21.3 kB]
Get: 50 http://deb.debian.org/debian trixie/main i386 golang-github-pkg-errors-dev all 0.9.1-3 [12.9 kB]
Get: 51 http://deb.debian.org/debian trixie/main i386 golang-github-tv42-httpunix-dev all 0.0~git20150427.b75d861-3 [3956 B]
Get: 52 http://deb.debian.org/debian trixie/main i386 golang-github-circonus-labs-circonus-gometrics-dev all 2.3.1-4 [65.7 kB]
Get: 53 http://deb.debian.org/debian trixie/main i386 golang-github-datadog-datadog-go-dev all 2.1.0-3 [14.9 kB]
Get: 54 http://deb.debian.org/debian trixie/main i386 golang-github-google-go-cmp-dev all 0.6.0-1 [86.6 kB]
Get: 55 http://deb.debian.org/debian trixie/main i386 golang-google-protobuf-dev all 1.33.0-1 [625 kB]
Get: 56 http://deb.debian.org/debian trixie/main i386 golang-github-golang-protobuf-1-5-dev all 1.5.4-1 [136 kB]
Get: 57 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-uuid-dev all 1.0.3-1 [8636 B]
Get: 58 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-golang-lru-dev all 0.5.4-2 [14.7 kB]
Get: 59 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-immutable-radix-dev all 1.3.1-3 [26.2 kB]
Get: 60 http://deb.debian.org/debian trixie/main i386 golang-github-pascaldekloe-goe-dev all 0.1.0-4 [10.9 kB]
Get: 61 http://deb.debian.org/debian trixie/main i386 golang-github-beorn7-perks-dev all 1.0.1-1 [12.1 kB]
Get: 62 http://deb.debian.org/debian trixie/main i386 golang-github-cespare-xxhash-dev all 2.3.0-1 [10.8 kB]
Get: 63 http://deb.debian.org/debian trixie/main i386 golang-github-modern-go-concurrent-dev all 1.0.3-1.1 [4668 B]
Get: 64 http://deb.debian.org/debian trixie/main i386 golang-github-modern-go-reflect2-dev all 1.0.2-2 [10.9 kB]
Get: 65 http://deb.debian.org/debian trixie/main i386 golang-github-json-iterator-go-dev all 1.1.12-2 [62.2 kB]
Get: 66 http://deb.debian.org/debian trixie/main i386 golang-github-klauspost-compress-dev all 1.17.11+ds1-1 [33.3 MB]
Get: 67 http://deb.debian.org/debian trixie/main i386 golang-github-kylelemons-godebug-dev all 1.1.0-2 [14.8 kB]
Get: 68 http://deb.debian.org/debian trixie/main i386 golang-github-prometheus-client-model-dev all 0.6.0-1 [11.2 kB]
Get: 69 http://deb.debian.org/debian trixie/main i386 golang-gopkg-yaml.v2-dev all 2.4.0-4 [61.5 kB]
Get: 70 http://deb.debian.org/debian trixie/main i386 golang-github-jmespath-go-jmespath-dev all 0.4.0-2 [91.6 kB]
Get: 71 http://deb.debian.org/debian trixie/main i386 golang-golang-x-term-dev all 0.22.0-1 [14.8 kB]
Get: 72 http://deb.debian.org/debian trixie/main i386 golang-golang-x-text-dev all 0.16.0-1 [4003 kB]
Get: 73 http://deb.debian.org/debian trixie/main i386 golang-golang-x-crypto-dev all 1:0.25.0-1 [1682 kB]
Get: 74 http://deb.debian.org/debian trixie/main i386 golang-golang-x-net-dev all 1:0.27.0-1 [898 kB]
Get: 75 http://deb.debian.org/debian trixie/main i386 golang-github-aws-aws-sdk-go-dev all 1.49.0-2 [12.9 MB]
Get: 76 http://deb.debian.org/debian trixie/main i386 golang-github-go-logfmt-logfmt-dev all 0.5.0-2 [13.4 kB]
Get: 77 http://deb.debian.org/debian trixie/main i386 golang-github-go-kit-log-dev all 0.2.1-1 [26.9 kB]
Get: 78 http://deb.debian.org/debian trixie/main i386 golang-github-julienschmidt-httprouter-dev all 1.3.0-2 [19.3 kB]
Get: 79 http://deb.debian.org/debian trixie/main i386 golang-github-munnerz-goautoneg-dev all 0.0~git20191010.a7dc8b6-2 [4308 B]
Get: 80 http://deb.debian.org/debian trixie/main i386 golang-github-jpillora-backoff-dev all 1.0.0-1.1 [3712 B]
Get: 81 http://deb.debian.org/debian trixie/main i386 golang-github-mwitkow-go-conntrack-dev all 0.0~git20190716.2f06839-3 [14.6 kB]
Get: 82 http://deb.debian.org/debian trixie/main i386 golang-golang-x-oauth2-dev all 0.15.0-1 [38.0 kB]
Get: 83 http://deb.debian.org/debian trixie/main i386 golang-gopkg-alecthomas-kingpin.v2-dev all 2.2.6-4 [42.8 kB]
Get: 84 http://deb.debian.org/debian trixie/main i386 golang-github-prometheus-common-dev all 0.60.1-1 [120 kB]
Get: 85 http://deb.debian.org/debian trixie/main i386 golang-golang-x-sync-dev all 0.7.0-1 [19.0 kB]
Get: 86 http://deb.debian.org/debian trixie/main i386 golang-github-prometheus-procfs-dev all 0.14.0-1 [183 kB]
Get: 87 http://deb.debian.org/debian trixie/main i386 golang-github-prometheus-client-golang-dev all 1.20.5-1 [188 kB]
Get: 88 http://deb.debian.org/debian trixie/main i386 golang-github-armon-go-metrics-dev all 0.4.1-1 [35.1 kB]
Get: 89 http://deb.debian.org/debian trixie/main i386 golang-github-creack-pty-dev all 1.1.21-1 [13.0 kB]
Get: 90 http://deb.debian.org/debian trixie/main i386 golang-github-vmihailenco-tagparser-dev all 0.1.1-4 [4684 B]
Get: 91 http://deb.debian.org/debian trixie/main i386 golang-gopkg-vmihailenco-msgpack.v2-dev all 4.3.1-2 [27.5 kB]
Get: 92 http://deb.debian.org/debian trixie/main i386 golang-gopkg-tomb.v2-dev all 0.0~git20161208.d5d1b58-3 [6840 B]
Get: 93 http://deb.debian.org/debian trixie/main i386 libsasl2-modules-db i386 2.1.28+dfsg1-8 [20.6 kB]
Get: 94 http://deb.debian.org/debian trixie/main i386 libsasl2-2 i386 2.1.28+dfsg1-8 [61.0 kB]
Get: 95 http://deb.debian.org/debian trixie/main i386 libssl-dev i386 3.3.2-2 [2790 kB]
Get: 96 http://deb.debian.org/debian trixie/main i386 libsasl2-dev i386 2.1.28+dfsg1-8 [247 kB]
Get: 97 http://deb.debian.org/debian trixie/main i386 golang-github-kr-text-dev all 0.2.0-2 [10.2 kB]
Get: 98 http://deb.debian.org/debian trixie/main i386 golang-golang-x-mod-dev all 0.19.0-1 [104 kB]
Get: 99 http://deb.debian.org/debian trixie/main i386 golang-github-yuin-goldmark-dev all 1.7.4-1 [113 kB]
Get: 100 http://deb.debian.org/debian trixie/main i386 golang-golang-x-tools-dev all 1:0.23.0+ds-1 [1593 kB]
Get: 101 http://deb.debian.org/debian trixie/main i386 golang-github-rogpeppe-go-internal-dev all 1.12.0-3 [103 kB]
Get: 102 http://deb.debian.org/debian trixie/main i386 golang-github-kr-pretty-dev all 0.3.1-1 [11.9 kB]
Get: 103 http://deb.debian.org/debian trixie/main i386 golang-gopkg-check.v1-dev all 0.0+git20200902.038fdea-1 [32.1 kB]
Get: 104 http://deb.debian.org/debian trixie/main i386 golang-gopkg-mgo.v2-dev all 2016.08.01-7 [316 kB]
Get: 105 http://deb.debian.org/debian trixie/main i386 golang-github-hashicorp-go-msgpack-dev all 0.5.5-1 [43.3 kB]
Fetched 125 MB in 2s (51.0 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package sensible-utils.
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 19950 files and directories currently installed.)
Preparing to unpack .../000-sensible-utils_0.0.24_all.deb ...
Unpacking sensible-utils (0.0.24) ...
Selecting previously unselected package libmagic-mgc.
Preparing to unpack .../001-libmagic-mgc_1%3a5.45-3+b1_i386.deb ...
Unpacking libmagic-mgc (1:5.45-3+b1) ...
Selecting previously unselected package libmagic1t64:i386.
Preparing to unpack .../002-libmagic1t64_1%3a5.45-3+b1_i386.deb ...
Unpacking libmagic1t64:i386 (1:5.45-3+b1) ...
Selecting previously unselected package file.
Preparing to unpack .../003-file_1%3a5.45-3+b1_i386.deb ...
Unpacking file (1:5.45-3+b1) ...
Selecting previously unselected package gettext-base.
Preparing to unpack .../004-gettext-base_0.22.5-2_i386.deb ...
Unpacking gettext-base (0.22.5-2) ...
Selecting previously unselected package libuchardet0:i386.
Preparing to unpack .../005-libuchardet0_0.0.8-1+b2_i386.deb ...
Unpacking libuchardet0:i386 (0.0.8-1+b2) ...
Selecting previously unselected package groff-base.
Preparing to unpack .../006-groff-base_1.23.0-5_i386.deb ...
Unpacking groff-base (1.23.0-5) ...
Selecting previously unselected package bsdextrautils.
Preparing to unpack .../007-bsdextrautils_2.40.2-11_i386.deb ...
Unpacking bsdextrautils (2.40.2-11) ...
Selecting previously unselected package libpipeline1:i386.
Preparing to unpack .../008-libpipeline1_1.5.8-1_i386.deb ...
Unpacking libpipeline1:i386 (1.5.8-1) ...
Selecting previously unselected package man-db.
Preparing to unpack .../009-man-db_2.13.0-1_i386.deb ...
Unpacking man-db (2.13.0-1) ...
Selecting previously unselected package m4.
Preparing to unpack .../010-m4_1.4.19-4_i386.deb ...
Unpacking m4 (1.4.19-4) ...
Selecting previously unselected package autoconf.
Preparing to unpack .../011-autoconf_2.72-3_all.deb ...
Unpacking autoconf (2.72-3) ...
Selecting previously unselected package autotools-dev.
Preparing to unpack .../012-autotools-dev_20220109.1_all.deb ...
Unpacking autotools-dev (20220109.1) ...
Selecting previously unselected package automake.
Preparing to unpack .../013-automake_1%3a1.16.5-1.3_all.deb ...
Unpacking automake (1:1.16.5-1.3) ...
Selecting previously unselected package autopoint.
Preparing to unpack .../014-autopoint_0.22.5-2_all.deb ...
Unpacking autopoint (0.22.5-2) ...
Selecting previously unselected package libdebhelper-perl.
Preparing to unpack .../015-libdebhelper-perl_13.20_all.deb ...
Unpacking libdebhelper-perl (13.20) ...
Selecting previously unselected package libtool.
Preparing to unpack .../016-libtool_2.4.7-8_all.deb ...
Unpacking libtool (2.4.7-8) ...
Selecting previously unselected package dh-autoreconf.
Preparing to unpack .../017-dh-autoreconf_20_all.deb ...
Unpacking dh-autoreconf (20) ...
Selecting previously unselected package libarchive-zip-perl.
Preparing to unpack .../018-libarchive-zip-perl_1.68-1_all.deb ...
Unpacking libarchive-zip-perl (1.68-1) ...
Selecting previously unselected package libfile-stripnondeterminism-perl.
Preparing to unpack .../019-libfile-stripnondeterminism-perl_1.14.0-1_all.deb ...
Unpacking libfile-stripnondeterminism-perl (1.14.0-1) ...
Selecting previously unselected package dh-strip-nondeterminism.
Preparing to unpack .../020-dh-strip-nondeterminism_1.14.0-1_all.deb ...
Unpacking dh-strip-nondeterminism (1.14.0-1) ...
Selecting previously unselected package libelf1t64:i386.
Preparing to unpack .../021-libelf1t64_0.192-4_i386.deb ...
Unpacking libelf1t64:i386 (0.192-4) ...
Selecting previously unselected package dwz.
Preparing to unpack .../022-dwz_0.15-1+b1_i386.deb ...
Unpacking dwz (0.15-1+b1) ...
Selecting previously unselected package libicu72:i386.
Preparing to unpack .../023-libicu72_72.1-5+b1_i386.deb ...
Unpacking libicu72:i386 (72.1-5+b1) ...
Selecting previously unselected package libxml2:i386.
Preparing to unpack .../024-libxml2_2.12.7+dfsg+really2.9.14-0.2+b1_i386.deb ...
Unpacking libxml2:i386 (2.12.7+dfsg+really2.9.14-0.2+b1) ...
Selecting previously unselected package gettext.
Preparing to unpack .../025-gettext_0.22.5-2_i386.deb ...
Unpacking gettext (0.22.5-2) ...
Selecting previously unselected package intltool-debian.
Preparing to unpack .../026-intltool-debian_0.35.0+20060710.6_all.deb ...
Unpacking intltool-debian (0.35.0+20060710.6) ...
Selecting previously unselected package po-debconf.
Preparing to unpack .../027-po-debconf_1.0.21+nmu1_all.deb ...
Unpacking po-debconf (1.0.21+nmu1) ...
Selecting previously unselected package debhelper.
Preparing to unpack .../028-debhelper_13.20_all.deb ...
Unpacking debhelper (13.20) ...
Selecting previously unselected package dh-golang.
Preparing to unpack .../029-dh-golang_1.62_all.deb ...
Unpacking dh-golang (1.62) ...
Selecting previously unselected package golang-1.23-src.
Preparing to unpack .../030-golang-1.23-src_1.23.3-2_all.deb ...
Unpacking golang-1.23-src (1.23.3-2) ...
Selecting previously unselected package golang-1.23-go.
Preparing to unpack .../031-golang-1.23-go_1.23.3-2_i386.deb ...
Unpacking golang-1.23-go (1.23.3-2) ...
Selecting previously unselected package golang-src.
Preparing to unpack .../032-golang-src_2%3a1.23~2_all.deb ...
Unpacking golang-src (2:1.23~2) ...
Selecting previously unselected package golang-go:i386.
Preparing to unpack .../033-golang-go_2%3a1.23~2_i386.deb ...
Unpacking golang-go:i386 (2:1.23~2) ...
Selecting previously unselected package golang-any:i386.
Preparing to unpack .../034-golang-any_2%3a1.23~2_i386.deb ...
Unpacking golang-any:i386 (2:1.23~2) ...
Selecting previously unselected package golang-github-alecthomas-units-dev.
Preparing to unpack .../035-golang-github-alecthomas-units-dev_0.0~git20240626.61d1e34-1_all.deb ...
Unpacking golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ...
Selecting previously unselected package golang-github-circonus-labs-circonusllhist-dev.
Preparing to unpack .../036-golang-github-circonus-labs-circonusllhist-dev_0.0~git20191022.ec08cde-1_all.deb ...
Unpacking golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Selecting previously unselected package golang-github-hashicorp-go-cleanhttp-dev.
Preparing to unpack .../037-golang-github-hashicorp-go-cleanhttp-dev_0.5.2-1_all.deb ...
Unpacking golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Selecting previously unselected package golang-golang-x-sys-dev.
Preparing to unpack .../038-golang-golang-x-sys-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-sys-dev (0.22.0-1) ...
Selecting previously unselected package golang-github-mattn-go-isatty-dev.
Preparing to unpack .../039-golang-github-mattn-go-isatty-dev_0.0.20-1_all.deb ...
Unpacking golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Selecting previously unselected package golang-github-mattn-go-colorable-dev.
Preparing to unpack .../040-golang-github-mattn-go-colorable-dev_0.1.13-1_all.deb ...
Unpacking golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Selecting previously unselected package golang-github-fatih-color-dev.
Preparing to unpack .../041-golang-github-fatih-color-dev_1.17.0-1_all.deb ...
Unpacking golang-github-fatih-color-dev (1.17.0-1) ...
Selecting previously unselected package golang-github-davecgh-go-spew-dev.
Preparing to unpack .../042-golang-github-davecgh-go-spew-dev_1.1.1-3_all.deb ...
Unpacking golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Selecting previously unselected package golang-github-pmezard-go-difflib-dev.
Preparing to unpack .../043-golang-github-pmezard-go-difflib-dev_1.0.0-3_all.deb ...
Unpacking golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Selecting previously unselected package golang-github-stretchr-objx-dev.
Preparing to unpack .../044-golang-github-stretchr-objx-dev_0.5.2-1_all.deb ...
Unpacking golang-github-stretchr-objx-dev (0.5.2-1) ...
Selecting previously unselected package golang-gopkg-yaml.v3-dev.
Preparing to unpack .../045-golang-gopkg-yaml.v3-dev_3.0.1-3_all.deb ...
Unpacking golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Selecting previously unselected package golang-github-stretchr-testify-dev.
Preparing to unpack .../046-golang-github-stretchr-testify-dev_1.9.0-1_all.deb ...
Unpacking golang-github-stretchr-testify-dev (1.9.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-hclog-dev.
Preparing to unpack .../047-golang-github-hashicorp-go-hclog-dev_0.11.0-1_all.deb ...
Unpacking golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Selecting previously unselected package golang-github-hashicorp-go-retryablehttp-dev.
Preparing to unpack .../048-golang-github-hashicorp-go-retryablehttp-dev_0.7.1-1_all.deb ...
Unpacking golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Selecting previously unselected package golang-github-pkg-errors-dev.
Preparing to unpack .../049-golang-github-pkg-errors-dev_0.9.1-3_all.deb ...
Unpacking golang-github-pkg-errors-dev (0.9.1-3) ...
Selecting previously unselected package golang-github-tv42-httpunix-dev.
Preparing to unpack .../050-golang-github-tv42-httpunix-dev_0.0~git20150427.b75d861-3_all.deb ...
Unpacking golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Selecting previously unselected package golang-github-circonus-labs-circonus-gometrics-dev.
Preparing to unpack .../051-golang-github-circonus-labs-circonus-gometrics-dev_2.3.1-4_all.deb ...
Unpacking golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Selecting previously unselected package golang-github-datadog-datadog-go-dev.
Preparing to unpack .../052-golang-github-datadog-datadog-go-dev_2.1.0-3_all.deb ...
Unpacking golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Selecting previously unselected package golang-github-google-go-cmp-dev.
Preparing to unpack .../053-golang-github-google-go-cmp-dev_0.6.0-1_all.deb ...
Unpacking golang-github-google-go-cmp-dev (0.6.0-1) ...
Selecting previously unselected package golang-google-protobuf-dev.
Preparing to unpack .../054-golang-google-protobuf-dev_1.33.0-1_all.deb ...
Unpacking golang-google-protobuf-dev (1.33.0-1) ...
Selecting previously unselected package golang-github-golang-protobuf-1-5-dev.
Preparing to unpack .../055-golang-github-golang-protobuf-1-5-dev_1.5.4-1_all.deb ...
Unpacking golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Selecting previously unselected package golang-github-hashicorp-go-uuid-dev.
Preparing to unpack .../056-golang-github-hashicorp-go-uuid-dev_1.0.3-1_all.deb ...
Unpacking golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Selecting previously unselected package golang-github-hashicorp-golang-lru-dev.
Preparing to unpack .../057-golang-github-hashicorp-golang-lru-dev_0.5.4-2_all.deb ...
Unpacking golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Selecting previously unselected package golang-github-hashicorp-go-immutable-radix-dev.
Preparing to unpack .../058-golang-github-hashicorp-go-immutable-radix-dev_1.3.1-3_all.deb ...
Unpacking golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Selecting previously unselected package golang-github-pascaldekloe-goe-dev.
Preparing to unpack .../059-golang-github-pascaldekloe-goe-dev_0.1.0-4_all.deb ...
Unpacking golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Selecting previously unselected package golang-github-beorn7-perks-dev.
Preparing to unpack .../060-golang-github-beorn7-perks-dev_1.0.1-1_all.deb ...
Unpacking golang-github-beorn7-perks-dev (1.0.1-1) ...
Selecting previously unselected package golang-github-cespare-xxhash-dev.
Preparing to unpack .../061-golang-github-cespare-xxhash-dev_2.3.0-1_all.deb ...
Unpacking golang-github-cespare-xxhash-dev (2.3.0-1) ...
Selecting previously unselected package golang-github-modern-go-concurrent-dev.
Preparing to unpack .../062-golang-github-modern-go-concurrent-dev_1.0.3-1.1_all.deb ...
Unpacking golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Selecting previously unselected package golang-github-modern-go-reflect2-dev.
Preparing to unpack .../063-golang-github-modern-go-reflect2-dev_1.0.2-2_all.deb ...
Unpacking golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Selecting previously unselected package golang-github-json-iterator-go-dev.
Preparing to unpack .../064-golang-github-json-iterator-go-dev_1.1.12-2_all.deb ...
Unpacking golang-github-json-iterator-go-dev (1.1.12-2) ...
Selecting previously unselected package golang-github-klauspost-compress-dev.
Preparing to unpack .../065-golang-github-klauspost-compress-dev_1.17.11+ds1-1_all.deb ...
Unpacking golang-github-klauspost-compress-dev (1.17.11+ds1-1) ...
Selecting previously unselected package golang-github-kylelemons-godebug-dev.
Preparing to unpack .../066-golang-github-kylelemons-godebug-dev_1.1.0-2_all.deb ...
Unpacking golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Selecting previously unselected package golang-github-prometheus-client-model-dev.
Preparing to unpack .../067-golang-github-prometheus-client-model-dev_0.6.0-1_all.deb ...
Unpacking golang-github-prometheus-client-model-dev (0.6.0-1) ...
Selecting previously unselected package golang-gopkg-yaml.v2-dev.
Preparing to unpack .../068-golang-gopkg-yaml.v2-dev_2.4.0-4_all.deb ...
Unpacking golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Selecting previously unselected package golang-github-jmespath-go-jmespath-dev.
Preparing to unpack .../069-golang-github-jmespath-go-jmespath-dev_0.4.0-2_all.deb ...
Unpacking golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Selecting previously unselected package golang-golang-x-term-dev.
Preparing to unpack .../070-golang-golang-x-term-dev_0.22.0-1_all.deb ...
Unpacking golang-golang-x-term-dev (0.22.0-1) ...
Selecting previously unselected package golang-golang-x-text-dev.
Preparing to unpack .../071-golang-golang-x-text-dev_0.16.0-1_all.deb ...
Unpacking golang-golang-x-text-dev (0.16.0-1) ...
Selecting previously unselected package golang-golang-x-crypto-dev.
Preparing to unpack .../072-golang-golang-x-crypto-dev_1%3a0.25.0-1_all.deb ...
Unpacking golang-golang-x-crypto-dev (1:0.25.0-1) ...
Selecting previously unselected package golang-golang-x-net-dev.
Preparing to unpack .../073-golang-golang-x-net-dev_1%3a0.27.0-1_all.deb ...
Unpacking golang-golang-x-net-dev (1:0.27.0-1) ...
Selecting previously unselected package golang-github-aws-aws-sdk-go-dev.
Preparing to unpack .../074-golang-github-aws-aws-sdk-go-dev_1.49.0-2_all.deb ...
Unpacking golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Selecting previously unselected package golang-github-go-logfmt-logfmt-dev.
Preparing to unpack .../075-golang-github-go-logfmt-logfmt-dev_0.5.0-2_all.deb ...
Unpacking golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Selecting previously unselected package golang-github-go-kit-log-dev.
Preparing to unpack .../076-golang-github-go-kit-log-dev_0.2.1-1_all.deb ...
Unpacking golang-github-go-kit-log-dev (0.2.1-1) ...
Selecting previously unselected package golang-github-julienschmidt-httprouter-dev.
Preparing to unpack .../077-golang-github-julienschmidt-httprouter-dev_1.3.0-2_all.deb ...
Unpacking golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Selecting previously unselected package golang-github-munnerz-goautoneg-dev.
Preparing to unpack .../078-golang-github-munnerz-goautoneg-dev_0.0~git20191010.a7dc8b6-2_all.deb ...
Unpacking golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Selecting previously unselected package golang-github-jpillora-backoff-dev.
Preparing to unpack .../079-golang-github-jpillora-backoff-dev_1.0.0-1.1_all.deb ...
Unpacking golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Selecting previously unselected package golang-github-mwitkow-go-conntrack-dev.
Preparing to unpack .../080-golang-github-mwitkow-go-conntrack-dev_0.0~git20190716.2f06839-3_all.deb ...
Unpacking golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Selecting previously unselected package golang-golang-x-oauth2-dev.
Preparing to unpack .../081-golang-golang-x-oauth2-dev_0.15.0-1_all.deb ...
Unpacking golang-golang-x-oauth2-dev (0.15.0-1) ...
Selecting previously unselected package golang-gopkg-alecthomas-kingpin.v2-dev.
Preparing to unpack .../082-golang-gopkg-alecthomas-kingpin.v2-dev_2.2.6-4_all.deb ...
Unpacking golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Selecting previously unselected package golang-github-prometheus-common-dev.
Preparing to unpack .../083-golang-github-prometheus-common-dev_0.60.1-1_all.deb ...
Unpacking golang-github-prometheus-common-dev (0.60.1-1) ...
Selecting previously unselected package golang-golang-x-sync-dev.
Preparing to unpack .../084-golang-golang-x-sync-dev_0.7.0-1_all.deb ...
Unpacking golang-golang-x-sync-dev (0.7.0-1) ...
Selecting previously unselected package golang-github-prometheus-procfs-dev.
Preparing to unpack .../085-golang-github-prometheus-procfs-dev_0.14.0-1_all.deb ...
Unpacking golang-github-prometheus-procfs-dev (0.14.0-1) ...
Selecting previously unselected package golang-github-prometheus-client-golang-dev.
Preparing to unpack .../086-golang-github-prometheus-client-golang-dev_1.20.5-1_all.deb ...
Unpacking golang-github-prometheus-client-golang-dev (1.20.5-1) ...
Selecting previously unselected package golang-github-armon-go-metrics-dev.
Preparing to unpack .../087-golang-github-armon-go-metrics-dev_0.4.1-1_all.deb ...
Unpacking golang-github-armon-go-metrics-dev (0.4.1-1) ...
Selecting previously unselected package golang-github-creack-pty-dev.
Preparing to unpack .../088-golang-github-creack-pty-dev_1.1.21-1_all.deb ...
Unpacking golang-github-creack-pty-dev (1.1.21-1) ...
Selecting previously unselected package golang-github-vmihailenco-tagparser-dev.
Preparing to unpack .../089-golang-github-vmihailenco-tagparser-dev_0.1.1-4_all.deb ...
Unpacking golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Selecting previously unselected package golang-gopkg-vmihailenco-msgpack.v2-dev.
Preparing to unpack .../090-golang-gopkg-vmihailenco-msgpack.v2-dev_4.3.1-2_all.deb ...
Unpacking golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Selecting previously unselected package golang-gopkg-tomb.v2-dev.
Preparing to unpack .../091-golang-gopkg-tomb.v2-dev_0.0~git20161208.d5d1b58-3_all.deb ...
Unpacking golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Selecting previously unselected package libsasl2-modules-db:i386.
Preparing to unpack .../092-libsasl2-modules-db_2.1.28+dfsg1-8_i386.deb ...
Unpacking libsasl2-modules-db:i386 (2.1.28+dfsg1-8) ...
Selecting previously unselected package libsasl2-2:i386.
Preparing to unpack .../093-libsasl2-2_2.1.28+dfsg1-8_i386.deb ...
Unpacking libsasl2-2:i386 (2.1.28+dfsg1-8) ...
Selecting previously unselected package libssl-dev:i386.
Preparing to unpack .../094-libssl-dev_3.3.2-2_i386.deb ...
Unpacking libssl-dev:i386 (3.3.2-2) ...
Selecting previously unselected package libsasl2-dev.
Preparing to unpack .../095-libsasl2-dev_2.1.28+dfsg1-8_i386.deb ...
Unpacking libsasl2-dev (2.1.28+dfsg1-8) ...
Selecting previously unselected package golang-github-kr-text-dev.
Preparing to unpack .../096-golang-github-kr-text-dev_0.2.0-2_all.deb ...
Unpacking golang-github-kr-text-dev (0.2.0-2) ...
Selecting previously unselected package golang-golang-x-mod-dev.
Preparing to unpack .../097-golang-golang-x-mod-dev_0.19.0-1_all.deb ...
Unpacking golang-golang-x-mod-dev (0.19.0-1) ...
Selecting previously unselected package golang-github-yuin-goldmark-dev.
Preparing to unpack .../098-golang-github-yuin-goldmark-dev_1.7.4-1_all.deb ...
Unpacking golang-github-yuin-goldmark-dev (1.7.4-1) ...
Selecting previously unselected package golang-golang-x-tools-dev.
Preparing to unpack .../099-golang-golang-x-tools-dev_1%3a0.23.0+ds-1_all.deb ...
Unpacking golang-golang-x-tools-dev (1:0.23.0+ds-1) ...
Selecting previously unselected package golang-github-rogpeppe-go-internal-dev.
Preparing to unpack .../100-golang-github-rogpeppe-go-internal-dev_1.12.0-3_all.deb ...
Unpacking golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Selecting previously unselected package golang-github-kr-pretty-dev.
Preparing to unpack .../101-golang-github-kr-pretty-dev_0.3.1-1_all.deb ...
Unpacking golang-github-kr-pretty-dev (0.3.1-1) ...
Selecting previously unselected package golang-gopkg-check.v1-dev.
Preparing to unpack .../102-golang-gopkg-check.v1-dev_0.0+git20200902.038fdea-1_all.deb ...
Unpacking golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Selecting previously unselected package golang-gopkg-mgo.v2-dev.
Preparing to unpack .../103-golang-gopkg-mgo.v2-dev_2016.08.01-7_all.deb ...
Unpacking golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Selecting previously unselected package golang-github-hashicorp-go-msgpack-dev.
Preparing to unpack .../104-golang-github-hashicorp-go-msgpack-dev_0.5.5-1_all.deb ...
Unpacking golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up libpipeline1:i386 (1.5.8-1) ...
Setting up golang-github-google-go-cmp-dev (0.6.0-1) ...
Setting up golang-gopkg-yaml.v2-dev (2.4.0-4) ...
Setting up golang-github-hashicorp-go-uuid-dev (1.0.3-1) ...
Setting up libicu72:i386 (72.1-5+b1) ...
Setting up bsdextrautils (2.40.2-11) ...
Setting up golang-github-creack-pty-dev (1.1.21-1) ...
Setting up golang-github-stretchr-objx-dev (0.5.2-1) ...
Setting up libmagic-mgc (1:5.45-3+b1) ...
Setting up golang-github-pkg-errors-dev (0.9.1-3) ...
Setting up golang-github-hashicorp-golang-lru-dev (0.5.4-2) ...
Setting up libarchive-zip-perl (1.68-1) ...
Setting up golang-github-kylelemons-godebug-dev (1.1.0-2) ...
Setting up golang-gopkg-yaml.v3-dev (3.0.1-3) ...
Setting up libdebhelper-perl (13.20) ...
Setting up golang-golang-x-sys-dev (0.22.0-1) ...
Setting up libmagic1t64:i386 (1:5.45-3+b1) ...
Setting up golang-github-pmezard-go-difflib-dev (1.0.0-3) ...
Setting up golang-github-modern-go-concurrent-dev (1.0.3-1.1) ...
Setting up gettext-base (0.22.5-2) ...
Setting up m4 (1.4.19-4) ...
Setting up golang-github-circonus-labs-circonusllhist-dev (0.0~git20191022.ec08cde-1) ...
Setting up golang-golang-x-term-dev (0.22.0-1) ...
Setting up file (1:5.45-3+b1) ...
Setting up golang-github-beorn7-perks-dev (1.0.1-1) ...
Setting up libelf1t64:i386 (0.192-4) ...
Setting up golang-github-hashicorp-go-cleanhttp-dev (0.5.2-1) ...
Setting up libsasl2-modules-db:i386 (2.1.28+dfsg1-8) ...
Setting up golang-github-cespare-xxhash-dev (2.3.0-1) ...
Setting up golang-gopkg-tomb.v2-dev (0.0~git20161208.d5d1b58-3) ...
Setting up golang-github-jpillora-backoff-dev (1.0.0-1.1) ...
Setting up golang-github-davecgh-go-spew-dev (1.1.1-3) ...
Setting up autotools-dev (20220109.1) ...
Setting up golang-github-pascaldekloe-goe-dev (0.1.0-4) ...
Setting up golang-github-go-logfmt-logfmt-dev (0.5.0-2) ...
Setting up golang-google-protobuf-dev (1.33.0-1) ...
Setting up golang-github-kr-text-dev (0.2.0-2) ...
Setting up golang-github-stretchr-testify-dev (1.9.0-1) ...
Setting up golang-github-hashicorp-go-immutable-radix-dev (1.3.1-3) ...
Setting up golang-github-yuin-goldmark-dev (1.7.4-1) ...
Setting up libssl-dev:i386 (3.3.2-2) ...
Setting up golang-golang-x-sync-dev (0.7.0-1) ...
Setting up golang-golang-x-mod-dev (0.19.0-1) ...
Setting up golang-1.23-src (1.23.3-2) ...
Setting up autopoint (0.22.5-2) ...
Setting up golang-github-klauspost-compress-dev (1.17.11+ds1-1) ...
Setting up libsasl2-2:i386 (2.1.28+dfsg1-8) ...
Setting up golang-github-julienschmidt-httprouter-dev (1.3.0-2) ...
Setting up autoconf (2.72-3) ...
Setting up golang-github-tv42-httpunix-dev (0.0~git20150427.b75d861-3) ...
Setting up dwz (0.15-1+b1) ...
Setting up golang-github-munnerz-goautoneg-dev (0.0~git20191010.a7dc8b6-2) ...
Setting up sensible-utils (0.0.24) ...
Setting up golang-golang-x-text-dev (0.16.0-1) ...
Setting up libuchardet0:i386 (0.0.8-1+b2) ...
Setting up golang-github-prometheus-procfs-dev (0.14.0-1) ...
Setting up golang-github-alecthomas-units-dev (0.0~git20240626.61d1e34-1) ...
Setting up golang-github-modern-go-reflect2-dev (1.0.2-2) ...
Setting up golang-golang-x-crypto-dev (1:0.25.0-1) ...
Setting up golang-github-prometheus-client-model-dev (0.6.0-1) ...
Setting up golang-github-datadog-datadog-go-dev (2.1.0-3) ...
Setting up golang-golang-x-oauth2-dev (0.15.0-1) ...
Setting up golang-src (2:1.23~2) ...
Setting up golang-github-vmihailenco-tagparser-dev (0.1.1-4) ...
Setting up libxml2:i386 (2.12.7+dfsg+really2.9.14-0.2+b1) ...
Setting up golang-github-jmespath-go-jmespath-dev (0.4.0-2) ...
Setting up automake (1:1.16.5-1.3) ...
update-alternatives: using /usr/bin/automake-1.16 to provide /usr/bin/automake (automake) in auto mode
Setting up libfile-stripnondeterminism-perl (1.14.0-1) ...
Setting up golang-github-mattn-go-isatty-dev (0.0.20-1) ...
Setting up golang-github-go-kit-log-dev (0.2.1-1) ...
Setting up golang-github-golang-protobuf-1-5-dev (1.5.4-1) ...
Setting up gettext (0.22.5-2) ...
Setting up golang-gopkg-vmihailenco-msgpack.v2-dev (4.3.1-2) ...
Setting up golang-gopkg-alecthomas-kingpin.v2-dev (2.2.6-4) ...
Setting up libtool (2.4.7-8) ...
Setting up golang-github-mattn-go-colorable-dev (0.1.13-1) ...
Setting up golang-github-json-iterator-go-dev (1.1.12-2) ...
Setting up intltool-debian (0.35.0+20060710.6) ...
Setting up dh-autoreconf (20) ...
Setting up golang-1.23-go (1.23.3-2) ...
Setting up libsasl2-dev (2.1.28+dfsg1-8) ...
Setting up golang-golang-x-net-dev (1:0.27.0-1) ...
Setting up dh-strip-nondeterminism (1.14.0-1) ...
Setting up groff-base (1.23.0-5) ...
Setting up golang-github-aws-aws-sdk-go-dev (1.49.0-2) ...
Setting up golang-github-fatih-color-dev (1.17.0-1) ...
Setting up golang-go:i386 (2:1.23~2) ...
Setting up po-debconf (1.0.21+nmu1) ...
Setting up golang-golang-x-tools-dev (1:0.23.0+ds-1) ...
Setting up man-db (2.13.0-1) ...
Not building database; man-db/auto-update is not 'true'.
Setting up golang-any:i386 (2:1.23~2) ...
Setting up golang-github-rogpeppe-go-internal-dev (1.12.0-3) ...
Setting up golang-github-hashicorp-go-hclog-dev (0.11.0-1) ...
Setting up golang-github-kr-pretty-dev (0.3.1-1) ...
Setting up debhelper (13.20) ...
Setting up golang-gopkg-check.v1-dev (0.0+git20200902.038fdea-1) ...
Setting up golang-gopkg-mgo.v2-dev (2016.08.01-7) ...
Setting up golang-github-hashicorp-go-retryablehttp-dev (0.7.1-1) ...
Setting up dh-golang (1.62) ...
Setting up golang-github-hashicorp-go-msgpack-dev (0.5.5-1) ...
Setting up golang-github-circonus-labs-circonus-gometrics-dev (2.3.1-4) ...
Setting up golang-github-mwitkow-go-conntrack-dev (0.0~git20190716.2f06839-3) ...
Setting up golang-github-prometheus-common-dev (0.60.1-1) ...
Setting up golang-github-prometheus-client-golang-dev (1.20.5-1) ...
Setting up golang-github-armon-go-metrics-dev (0.4.1-1) ...
Processing triggers for libc-bin (2.40-3) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Building tag database...
 -> Finished parsing the build-deps
I: Building the package
I: Running cd /build/reproducible-path/golang-github-hashicorp-raft-1.5.0/ && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-buildpackage -us -uc -b && env PATH="/usr/sbin:/usr/bin:/sbin:/bin:/usr/games" HOME="/nonexistent/first-build" dpkg-genchanges -S  > ../golang-github-hashicorp-raft_1.5.0-1_source.changes
dpkg-buildpackage: info: source package golang-github-hashicorp-raft
dpkg-buildpackage: info: source version 1.5.0-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Daniel Swarbrick <dswarbrick@debian.org>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture i386
 debian/rules clean
dh clean --buildsystem=golang
   dh_auto_clean -O--buildsystem=golang
   dh_autoreconf_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules binary
dh binary --buildsystem=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-i686-linux-gnu && go install -trimpath -v -p 11 github.com/hashicorp/raft github.com/hashicorp/raft/bench
internal/profilerecord
internal/godebugs
internal/coverage/rtcov
internal/unsafeheader
internal/byteorder
internal/goarch
internal/goexperiment
internal/goos
internal/race
internal/runtime/atomic
internal/cpu
internal/runtime/syscall
unicode
runtime/internal/math
runtime/internal/sys
container/list
internal/chacha8rand
sync/atomic
internal/abi
unicode/utf8
internal/asan
internal/itoa
internal/msan
math/bits
crypto/internal/alias
crypto/internal/boring/sig
cmp
encoding
unicode/utf16
log/internal
internal/nettrace
internal/runtime/exithook
vendor/golang.org/x/crypto/cryptobyte/asn1
vendor/golang.org/x/crypto/internal/alias
internal/bytealg
math
internal/stringslite
runtime
internal/reflectlite
crypto/subtle
iter
sync
internal/weak
slices
maps
internal/bisect
internal/testlog
internal/singleflight
errors
sort
internal/godebug
internal/oserror
path
github.com/hashicorp/golang-lru/simplelru
math/rand/v2
vendor/golang.org/x/net/dns/dnsmessage
io
strconv
crypto/internal/edwards25519/field
runtime/cgo
syscall
math/rand
crypto/internal/nistec/fiat
crypto/internal/edwards25519
bytes
hash
strings
crypto/internal/randutil
internal/concurrent
hash/crc32
internal/saferio
unique
hash/crc64
crypto
reflect
crypto/cipher
bufio
crypto/rc4
net/netip
github.com/hashicorp/go-immutable-radix
net/http/internal/ascii
crypto/md5
vendor/golang.org/x/text/transform
regexp/syntax
crypto/internal/boring
crypto/des
html
crypto/sha512
crypto/aes
crypto/hmac
crypto/sha1
crypto/sha256
vendor/golang.org/x/crypto/hkdf
internal/syscall/unix
internal/syscall/execenv
time
regexp
crypto/internal/nistec
internal/fmtsort
encoding/binary
io/fs
context
internal/poll
crypto/ecdh
internal/filepathlite
vendor/golang.org/x/crypto/chacha20
vendor/golang.org/x/crypto/internal/poly1305
encoding/base64
golang.org/x/sys/unix
vendor/golang.org/x/crypto/chacha20poly1305
encoding/pem
os
fmt
path/filepath
vendor/golang.org/x/sys/cpu
net
os/signal
io/ioutil
internal/sysinfo
vendor/golang.org/x/crypto/sha3
log
encoding/hex
encoding/json
net/url
vendor/golang.org/x/net/http2/hpack
compress/flate
vendor/golang.org/x/text/unicode/norm
mime
math/big
mime/quotedprintable
net/http/internal
github.com/mattn/go-isatty
vendor/golang.org/x/text/unicode/bidi
github.com/mattn/go-colorable
encoding/gob
github.com/fatih/color
go/token
compress/gzip
text/template/parse
flag
runtime/debug
runtime/trace
vendor/golang.org/x/text/secure/bidirule
testing
github.com/hashicorp/go-hclog
vendor/golang.org/x/net/idna
text/template
crypto/rand
html/template
crypto/internal/boring/bbig
crypto/internal/bigmod
crypto/dsa
encoding/asn1
crypto/elliptic
crypto/internal/mlkem768
crypto/internal/hpke
crypto/ed25519
crypto/rsa
crypto/x509/pkix
vendor/golang.org/x/crypto/cryptobyte
crypto/ecdsa
net/textproto
vendor/golang.org/x/net/http/httpproxy
crypto/x509
vendor/golang.org/x/net/http/httpguts
mime/multipart
crypto/tls
net/http/httptrace
net/http
github.com/armon/go-metrics
net/rpc
github.com/hashicorp/go-msgpack/codec
github.com/hashicorp/raft
github.com/hashicorp/raft/bench
   debian/rules override_dh_auto_test
make[1]: Entering directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
dh_auto_test
	cd obj-i686-linux-gnu && go test -vet=off -v -p 11 github.com/hashicorp/raft github.com/hashicorp/raft/bench
?   	github.com/hashicorp/raft/bench	[no test files]
=== RUN   TestCommitment_setVoters
--- PASS: TestCommitment_setVoters (0.00s)
=== RUN   TestCommitment_match_max
--- PASS: TestCommitment_match_max (0.00s)
=== RUN   TestCommitment_match_nonVoting
--- PASS: TestCommitment_match_nonVoting (0.00s)
=== RUN   TestCommitment_recalculate
--- PASS: TestCommitment_recalculate (0.00s)
=== RUN   TestCommitment_recalculate_startIndex
--- PASS: TestCommitment_recalculate_startIndex (0.00s)
=== RUN   TestCommitment_noVoterSanity
--- PASS: TestCommitment_noVoterSanity (0.00s)
=== RUN   TestCommitment_singleVoter
--- PASS: TestCommitment_singleVoter (0.00s)
=== RUN   TestConfiguration_Configuration_Clone
--- PASS: TestConfiguration_Configuration_Clone (0.00s)
=== RUN   TestConfiguration_configurations_Clone
--- PASS: TestConfiguration_configurations_Clone (0.00s)
=== RUN   TestConfiguration_hasVote
--- PASS: TestConfiguration_hasVote (0.00s)
=== RUN   TestConfiguration_checkConfiguration
--- PASS: TestConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_nextConfiguration_table
--- PASS: TestConfiguration_nextConfiguration_table (0.00s)
=== RUN   TestConfiguration_nextConfiguration_prevIndex
--- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s)
=== RUN   TestConfiguration_nextConfiguration_checkConfiguration
--- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s)
=== RUN   TestConfiguration_encodeDecodePeers
--- PASS: TestConfiguration_encodeDecodePeers (0.00s)
=== RUN   TestConfiguration_encodeDecodeConfiguration
--- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s)
=== RUN   TestDiscardSnapshotStoreImpl
--- PASS: TestDiscardSnapshotStoreImpl (0.00s)
=== RUN   TestDiscardSnapshotSinkImpl
--- PASS: TestDiscardSnapshotSinkImpl (0.00s)
=== RUN   TestFileSnapshotStoreImpl
--- PASS: TestFileSnapshotStoreImpl (0.00s)
=== RUN   TestFileSnapshotSinkImpl
--- PASS: TestFileSnapshotSinkImpl (0.00s)
=== RUN   TestFileSS_CreateSnapshotMissingParentDir
2024-12-04T01:31:49.171Z [INFO]  creating new snapshot: path=/tmp/raft619976792/raft1256474951/snapshots/3-10-1733275909171.tmp
--- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s)
=== RUN   TestFileSS_CreateSnapshot
2024-12-04T01:31:49.175Z [INFO]  creating new snapshot: path=/tmp/raft2788028519/snapshots/3-10-1733275909175.tmp
2024-12-04T01:31:49.176Z [WARN]  found temporary snapshot: name=3-10-1733275909175.tmp
--- PASS: TestFileSS_CreateSnapshot (0.01s)
=== RUN   TestFileSS_CancelSnapshot
2024-12-04T01:31:49.181Z [INFO]  creating new snapshot: path=/tmp/raft3414242366/snapshots/3-10-1733275909181.tmp
--- PASS: TestFileSS_CancelSnapshot (0.00s)
=== RUN   TestFileSS_Retention
2024-12-04T01:31:49.184Z [INFO]  creating new snapshot: path=/tmp/raft914659978/snapshots/3-10-1733275909184.tmp
2024-12-04T01:31:49.189Z [INFO]  creating new snapshot: path=/tmp/raft914659978/snapshots/3-11-1733275909189.tmp
2024-12-04T01:31:49.193Z [INFO]  creating new snapshot: path=/tmp/raft914659978/snapshots/3-12-1733275909193.tmp
2024-12-04T01:31:49.198Z [INFO]  reaping snapshot: path=/tmp/raft914659978/snapshots/3-10-1733275909184
2024-12-04T01:31:49.198Z [INFO]  creating new snapshot: path=/tmp/raft914659978/snapshots/3-13-1733275909198.tmp
2024-12-04T01:31:49.202Z [INFO]  reaping snapshot: path=/tmp/raft914659978/snapshots/3-11-1733275909189
2024-12-04T01:31:49.203Z [INFO]  creating new snapshot: path=/tmp/raft914659978/snapshots/3-14-1733275909203.tmp
2024-12-04T01:31:49.207Z [INFO]  reaping snapshot: path=/tmp/raft914659978/snapshots/3-12-1733275909193
--- PASS: TestFileSS_Retention (0.02s)
=== RUN   TestFileSS_BadPerm
--- PASS: TestFileSS_BadPerm (0.00s)
=== RUN   TestFileSS_MissingParentDir
--- PASS: TestFileSS_MissingParentDir (0.00s)
=== RUN   TestFileSS_Ordering
2024-12-04T01:31:49.209Z [INFO]  creating new snapshot: path=/tmp/raft4035479814/snapshots/5-130350-1733275909209.tmp
2024-12-04T01:31:49.213Z [INFO]  creating new snapshot: path=/tmp/raft4035479814/snapshots/36-204917-1733275909213.tmp
--- PASS: TestFileSS_Ordering (0.01s)
=== RUN   TestDeferFutureSuccess
--- PASS: TestDeferFutureSuccess (0.00s)
=== RUN   TestDeferFutureError
--- PASS: TestDeferFutureError (0.00s)
=== RUN   TestDeferFutureConcurrent
--- PASS: TestDeferFutureConcurrent (0.00s)
=== RUN   TestInmemSnapshotStoreImpl
--- PASS: TestInmemSnapshotStoreImpl (0.00s)
=== RUN   TestInmemSnapshotSinkImpl
--- PASS: TestInmemSnapshotSinkImpl (0.00s)
=== RUN   TestInmemSS_CreateSnapshot
--- PASS: TestInmemSS_CreateSnapshot (0.00s)
=== RUN   TestInmemSS_OpenSnapshotTwice
--- PASS: TestInmemSS_OpenSnapshotTwice (0.00s)
=== RUN   TestInmemTransportImpl
--- PASS: TestInmemTransportImpl (0.00s)
=== RUN   TestInmemTransportWriteTimeout
--- PASS: TestInmemTransportWriteTimeout (0.01s)
=== RUN   TestRaft_Integ
--- SKIP: TestRaft_Integ (0.00s)
=== RUN   TestRaft_RestartFollower_LongInitialHeartbeat
--- SKIP: TestRaft_RestartFollower_LongInitialHeartbeat (0.00s)
=== RUN   TestLogCache
--- PASS: TestLogCache (0.00s)
=== RUN   TestLogCacheWithBackendStoreError
--- PASS: TestLogCacheWithBackendStoreError (0.00s)
=== RUN   TestOldestLog
=== RUN   TestOldestLog/empty_logs
=== RUN   TestOldestLog/simple_case
--- PASS: TestOldestLog (0.00s)
    --- PASS: TestOldestLog/empty_logs (0.00s)
    --- PASS: TestOldestLog/simple_case (0.00s)
=== RUN   TestEmitsLogStoreMetrics
--- PASS: TestEmitsLogStoreMetrics (0.01s)
=== RUN   TestNetworkTransport_CloseStreams
--- PASS: TestNetworkTransport_CloseStreams (0.01s)
=== RUN   TestNetworkTransport_StartStop
--- PASS: TestNetworkTransport_StartStop (0.00s)
=== RUN   TestNetworkTransport_Heartbeat_FastPath
--- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s)
=== RUN   TestNetworkTransport_AppendEntries
--- PASS: TestNetworkTransport_AppendEntries (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline
--- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_CloseStreams
--- PASS: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.03s)
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0
2024-12-04T01:31:49.330Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2
2024-12-04T01:31:49.383Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
=== RUN   TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10
--- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight (0.16s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=0 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=1 (0.00s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=2 (0.05s)
    --- PASS: TestNetworkTransport_AppendEntriesPipeline_MaxRPCsInFlight/max=10 (0.05s)
=== RUN   TestNetworkTransport_RequestVote
2024-12-04T01:31:49.438Z [ERROR] raft-net: failed to decode incoming command: error="transport shutdown"
--- PASS: TestNetworkTransport_RequestVote (0.00s)
=== RUN   TestNetworkTransport_InstallSnapshot
--- PASS: TestNetworkTransport_InstallSnapshot (0.01s)
=== RUN   TestNetworkTransport_EncodeDecode
--- PASS: TestNetworkTransport_EncodeDecode (0.00s)
=== RUN   TestNetworkTransport_EncodeDecode_AddressProvider
--- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s)
=== RUN   TestNetworkTransport_PooledConn
--- PASS: TestNetworkTransport_PooledConn (0.01s)
=== RUN   TestNetworkTransport_ListenBackoff
    net_transport_test.go:839: countingWriter: 2024-12-04T01:31:49.458Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-12-04T01:31:49.464Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-12-04T01:31:49.475Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-12-04T01:31:49.495Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-12-04T01:31:49.536Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-12-04T01:31:49.617Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-12-04T01:31:49.778Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-12-04T01:31:50.098Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-12-04T01:31:50.739Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-12-04T01:31:51.742Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
    net_transport_test.go:839: countingWriter: 2024-12-04T01:31:52.746Z [ERROR] test: failed to accept connection: error="intentional error in test"
        
--- PASS: TestNetworkTransport_ListenBackoff (4.00s)
=== RUN   TestPeersJSON_BadConfiguration
--- PASS: TestPeersJSON_BadConfiguration (0.00s)
=== RUN   TestPeersJSON_ReadPeersJSON
--- PASS: TestPeersJSON_ReadPeersJSON (0.00s)
=== RUN   TestPeersJSON_ReadConfigJSON
--- PASS: TestPeersJSON_ReadConfigJSON (0.00s)
=== RUN   TestRaft_StartStop
2024-12-04T01:31:53.464Z [INFO]  server-c1796311-0cd4-520c-fade-e02541dd9e62: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c1796311-0cd4-520c-fade-e02541dd9e62 Address:c1796311-0cd4-520c-fade-e02541dd9e62}]"
--- PASS: TestRaft_StartStop (0.00s)
=== RUN   TestRaft_AfterShutdown
2024-12-04T01:31:53.465Z [INFO]  server-ee8936e0-1858-6673-cd53-3acbedb5a36a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ee8936e0-1858-6673-cd53-3acbedb5a36a Address:ee8936e0-1858-6673-cd53-3acbedb5a36a}]"
--- PASS: TestRaft_AfterShutdown (0.00s)
=== RUN   TestRaft_LiveBootstrap
2024-12-04T01:31:53.466Z [INFO]  server-47954a67-1c90-e9ac-b287-fd01c5f0e470: initial configuration: index=0 servers=[]
2024-12-04T01:31:53.466Z [INFO]  server-8156af79-7e4c-11f4-3673-341fc694f3aa: initial configuration: index=0 servers=[]
2024-12-04T01:31:53.466Z [INFO]  server-2b0954e7-141c-66a4-5b20-ea3bc11920eb: initial configuration: index=0 servers=[]
2024-12-04T01:31:53.466Z [INFO]  server-47954a67-1c90-e9ac-b287-fd01c5f0e470: entering follower state: follower="Node at 47954a67-1c90-e9ac-b287-fd01c5f0e470 [Follower]" leader-address= leader-id=
2024-12-04T01:31:53.466Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:31:53.466Z [INFO]  server-8156af79-7e4c-11f4-3673-341fc694f3aa: entering follower state: follower="Node at 8156af79-7e4c-11f4-3673-341fc694f3aa [Follower]" leader-address= leader-id=
2024-12-04T01:31:53.467Z [INFO]  server-2b0954e7-141c-66a4-5b20-ea3bc11920eb: entering follower state: follower="Node at 2b0954e7-141c-66a4-5b20-ea3bc11920eb [Follower]" leader-address= leader-id=
2024-12-04T01:31:53.534Z [WARN]  server-47954a67-1c90-e9ac-b287-fd01c5f0e470: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:53.534Z [INFO]  server-47954a67-1c90-e9ac-b287-fd01c5f0e470: entering candidate state: node="Node at 47954a67-1c90-e9ac-b287-fd01c5f0e470 [Candidate]" term=2
2024-12-04T01:31:53.534Z [WARN]  server-2b0954e7-141c-66a4-5b20-ea3bc11920eb: no known peers, aborting election
2024-12-04T01:31:53.534Z [INFO]  server-47954a67-1c90-e9ac-b287-fd01c5f0e470: election won: term=2 tally=2
2024-12-04T01:31:53.534Z [INFO]  server-47954a67-1c90-e9ac-b287-fd01c5f0e470: entering leader state: leader="Node at 47954a67-1c90-e9ac-b287-fd01c5f0e470 [Leader]"
2024-12-04T01:31:53.534Z [INFO]  server-47954a67-1c90-e9ac-b287-fd01c5f0e470: added peer, starting replication: peer=server-8156af79-7e4c-11f4-3673-341fc694f3aa
2024-12-04T01:31:53.534Z [INFO]  server-47954a67-1c90-e9ac-b287-fd01c5f0e470: added peer, starting replication: peer=server-2b0954e7-141c-66a4-5b20-ea3bc11920eb
2024-12-04T01:31:53.534Z [WARN]  server-2b0954e7-141c-66a4-5b20-ea3bc11920eb: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2024-12-04T01:31:53.534Z [WARN]  server-47954a67-1c90-e9ac-b287-fd01c5f0e470: appendEntries rejected, sending older logs: peer="{Voter server-2b0954e7-141c-66a4-5b20-ea3bc11920eb 2b0954e7-141c-66a4-5b20-ea3bc11920eb}" next=1
2024-12-04T01:31:53.534Z [INFO]  server-47954a67-1c90-e9ac-b287-fd01c5f0e470: pipelining replication: peer="{Voter server-2b0954e7-141c-66a4-5b20-ea3bc11920eb 2b0954e7-141c-66a4-5b20-ea3bc11920eb}"
2024-12-04T01:31:53.535Z [WARN]  server-8156af79-7e4c-11f4-3673-341fc694f3aa: failed to get previous log: previous-index=1 last-index=0 error="log not found"
2024-12-04T01:31:53.535Z [WARN]  server-47954a67-1c90-e9ac-b287-fd01c5f0e470: appendEntries rejected, sending older logs: peer="{Voter server-8156af79-7e4c-11f4-3673-341fc694f3aa 8156af79-7e4c-11f4-3673-341fc694f3aa}" next=1
2024-12-04T01:31:53.535Z [INFO]  server-47954a67-1c90-e9ac-b287-fd01c5f0e470: pipelining replication: peer="{Voter server-8156af79-7e4c-11f4-3673-341fc694f3aa 8156af79-7e4c-11f4-3673-341fc694f3aa}"
2024-12-04T01:31:53.642Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:31:53.535433765 +0000 UTC m=+4.367276522 (2 nodes), 68.489238ms from start of poll, 69.01937ms from cluster start. Timeout at 2024-12-04 01:31:53.640440223 +0000 UTC m=+4.472282948, 105.006426ms after stability
2024-12-04T01:31:53.642Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:53.750Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:53.64260473 +0000 UTC m=+4.474447429 (1 nodes), 2.151µs from start of poll, 176.190277ms from cluster start. Timeout at 2024-12-04 01:31:53.747611632 +0000 UTC m=+4.579454605, 105.007176ms after stability
2024-12-04T01:31:53.762Z [INFO]  server-47954a67-1c90-e9ac-b287-fd01c5f0e470: aborting pipeline replication: peer="{Voter server-8156af79-7e4c-11f4-3673-341fc694f3aa 8156af79-7e4c-11f4-3673-341fc694f3aa}"
2024-12-04T01:31:53.762Z [INFO]  server-47954a67-1c90-e9ac-b287-fd01c5f0e470: aborting pipeline replication: peer="{Voter server-2b0954e7-141c-66a4-5b20-ea3bc11920eb 2b0954e7-141c-66a4-5b20-ea3bc11920eb}"
--- PASS: TestRaft_LiveBootstrap (0.30s)
=== RUN   TestRaft_LiveBootstrap_From_NonVoter
2024-12-04T01:31:53.763Z [INFO]  server-19a2998f-fb2f-f104-c30d-31a92d921b10: initial configuration: index=0 servers=[]
2024-12-04T01:31:53.763Z [INFO]  server-ec43193b-ac6a-8604-1174-f58faa8c2cf8: initial configuration: index=0 servers=[]
2024-12-04T01:31:53.763Z [INFO]  server-19a2998f-fb2f-f104-c30d-31a92d921b10: entering follower state: follower="Node at 19a2998f-fb2f-f104-c30d-31a92d921b10 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_LiveBootstrap_From_NonVoter (0.00s)
=== RUN   TestRaft_RecoverCluster_NoState
2024-12-04T01:31:53.764Z [INFO]  server-beac5bbf-f540-1f59-6305-0d1244c9617c: initial configuration: index=0 servers=[]
--- PASS: TestRaft_RecoverCluster_NoState (0.00s)
=== RUN   TestRaft_RecoverCluster
=== RUN   TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs
2024-12-04T01:31:53.765Z [INFO]  server-952f59bd-6d40-f082-dbfe-7e0749d4e34f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-952f59bd-6d40-f082-dbfe-7e0749d4e34f Address:952f59bd-6d40-f082-dbfe-7e0749d4e34f} {Suffrage:Voter ID:server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8 Address:09d40183-e7ed-53d2-04f0-af8efd9c5fd8} {Suffrage:Voter ID:server-460bb384-1906-f4e2-824b-0a508784c2fe Address:460bb384-1906-f4e2-824b-0a508784c2fe}]"
2024-12-04T01:31:53.765Z [INFO]  server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-952f59bd-6d40-f082-dbfe-7e0749d4e34f Address:952f59bd-6d40-f082-dbfe-7e0749d4e34f} {Suffrage:Voter ID:server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8 Address:09d40183-e7ed-53d2-04f0-af8efd9c5fd8} {Suffrage:Voter ID:server-460bb384-1906-f4e2-824b-0a508784c2fe Address:460bb384-1906-f4e2-824b-0a508784c2fe}]"
2024-12-04T01:31:53.765Z [INFO]  server-460bb384-1906-f4e2-824b-0a508784c2fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-952f59bd-6d40-f082-dbfe-7e0749d4e34f Address:952f59bd-6d40-f082-dbfe-7e0749d4e34f} {Suffrage:Voter ID:server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8 Address:09d40183-e7ed-53d2-04f0-af8efd9c5fd8} {Suffrage:Voter ID:server-460bb384-1906-f4e2-824b-0a508784c2fe Address:460bb384-1906-f4e2-824b-0a508784c2fe}]"
2024-12-04T01:31:53.765Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:53.765Z [INFO]  server-952f59bd-6d40-f082-dbfe-7e0749d4e34f: entering follower state: follower="Node at 952f59bd-6d40-f082-dbfe-7e0749d4e34f [Follower]" leader-address= leader-id=
2024-12-04T01:31:53.765Z [INFO]  server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8: entering follower state: follower="Node at 09d40183-e7ed-53d2-04f0-af8efd9c5fd8 [Follower]" leader-address= leader-id=
2024-12-04T01:31:53.765Z [INFO]  server-460bb384-1906-f4e2-824b-0a508784c2fe: entering follower state: follower="Node at 460bb384-1906-f4e2-824b-0a508784c2fe [Follower]" leader-address= leader-id=
2024-12-04T01:31:53.834Z [WARN]  server-460bb384-1906-f4e2-824b-0a508784c2fe: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:53.834Z [INFO]  server-460bb384-1906-f4e2-824b-0a508784c2fe: entering candidate state: node="Node at 460bb384-1906-f4e2-824b-0a508784c2fe [Candidate]" term=2
2024-12-04T01:31:53.834Z [INFO]  server-460bb384-1906-f4e2-824b-0a508784c2fe: election won: term=2 tally=2
2024-12-04T01:31:53.834Z [INFO]  server-460bb384-1906-f4e2-824b-0a508784c2fe: entering leader state: leader="Node at 460bb384-1906-f4e2-824b-0a508784c2fe [Leader]"
2024-12-04T01:31:53.834Z [INFO]  server-460bb384-1906-f4e2-824b-0a508784c2fe: added peer, starting replication: peer=server-952f59bd-6d40-f082-dbfe-7e0749d4e34f
2024-12-04T01:31:53.834Z [INFO]  server-460bb384-1906-f4e2-824b-0a508784c2fe: added peer, starting replication: peer=server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8
2024-12-04T01:31:53.834Z [INFO]  server-460bb384-1906-f4e2-824b-0a508784c2fe: pipelining replication: peer="{Voter server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8 09d40183-e7ed-53d2-04f0-af8efd9c5fd8}"
2024-12-04T01:31:53.835Z [INFO]  server-460bb384-1906-f4e2-824b-0a508784c2fe: pipelining replication: peer="{Voter server-952f59bd-6d40-f082-dbfe-7e0749d4e34f 952f59bd-6d40-f082-dbfe-7e0749d4e34f}"
2024-12-04T01:31:53.942Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:53.835114983 +0000 UTC m=+4.666957648 (1 nodes), 69.232624ms from start of poll, 69.650386ms from cluster start. Timeout at 2024-12-04 01:31:53.940121665 +0000 UTC m=+4.771964349, 105.006701ms after stability
2024-12-04T01:31:53.942Z [INFO]  server-460bb384-1906-f4e2-824b-0a508784c2fe: aborting pipeline replication: peer="{Voter server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8 09d40183-e7ed-53d2-04f0-af8efd9c5fd8}"
2024-12-04T01:31:53.942Z [INFO]  server-460bb384-1906-f4e2-824b-0a508784c2fe: aborting pipeline replication: peer="{Voter server-952f59bd-6d40-f082-dbfe-7e0749d4e34f 952f59bd-6d40-f082-dbfe-7e0749d4e34f}"
2024-12-04T01:31:53.942Z [INFO]  creating new snapshot: path=/tmp/raft901199499/snapshots/2-2-1733275913942.tmp
2024-12-04T01:31:53.943Z [INFO]  server-952f59bd-6d40-f082-dbfe-7e0749d4e34f: starting restore from snapshot: id=2-2-1733275913942 last-index=2 last-term=2 size-in-bytes=1
2024-12-04T01:31:53.944Z [INFO]  server-952f59bd-6d40-f082-dbfe-7e0749d4e34f: snapshot restore progress: id=2-2-1733275913942 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2024-12-04T01:31:53.944Z [INFO]  server-952f59bd-6d40-f082-dbfe-7e0749d4e34f: restored from snapshot: id=2-2-1733275913942 last-index=2 last-term=2 size-in-bytes=1
2024-12-04T01:31:53.944Z [INFO]  server-952f59bd-6d40-f082-dbfe-7e0749d4e34f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-952f59bd-6d40-f082-dbfe-7e0749d4e34f Address:952f59bd-6d40-f082-dbfe-7e0749d4e34f} {Suffrage:Voter ID:server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8 Address:09d40183-e7ed-53d2-04f0-af8efd9c5fd8} {Suffrage:Voter ID:server-460bb384-1906-f4e2-824b-0a508784c2fe Address:460bb384-1906-f4e2-824b-0a508784c2fe}]"
2024-12-04T01:31:53.944Z [INFO]  creating new snapshot: path=/tmp/raft1517460956/snapshots/2-2-1733275913944.tmp
2024-12-04T01:31:53.945Z [INFO]  server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8: starting restore from snapshot: id=2-2-1733275913944 last-index=2 last-term=2 size-in-bytes=1
2024-12-04T01:31:53.945Z [INFO]  server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8: snapshot restore progress: id=2-2-1733275913944 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2024-12-04T01:31:53.945Z [INFO]  server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8: restored from snapshot: id=2-2-1733275913944 last-index=2 last-term=2 size-in-bytes=1
2024-12-04T01:31:53.945Z [INFO]  server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-952f59bd-6d40-f082-dbfe-7e0749d4e34f Address:952f59bd-6d40-f082-dbfe-7e0749d4e34f} {Suffrage:Voter ID:server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8 Address:09d40183-e7ed-53d2-04f0-af8efd9c5fd8} {Suffrage:Voter ID:server-460bb384-1906-f4e2-824b-0a508784c2fe Address:460bb384-1906-f4e2-824b-0a508784c2fe}]"
2024-12-04T01:31:53.945Z [INFO]  creating new snapshot: path=/tmp/raft675415027/snapshots/2-2-1733275913945.tmp
2024-12-04T01:31:53.946Z [INFO]  server-952f59bd-6d40-f082-dbfe-7e0749d4e34f: entering follower state: follower="Node at 952f59bd-6d40-f082-dbfe-7e0749d4e34f [Follower]" leader-address= leader-id=
2024-12-04T01:31:53.946Z [INFO]  server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8: entering follower state: follower="Node at 09d40183-e7ed-53d2-04f0-af8efd9c5fd8 [Follower]" leader-address= leader-id=
2024-12-04T01:31:53.946Z [INFO]  server-460bb384-1906-f4e2-824b-0a508784c2fe: starting restore from snapshot: id=2-2-1733275913945 last-index=2 last-term=2 size-in-bytes=1
2024-12-04T01:31:53.947Z [INFO]  server-460bb384-1906-f4e2-824b-0a508784c2fe: snapshot restore progress: id=2-2-1733275913945 last-index=2 last-term=2 size-in-bytes=1 read-bytes=1 percent-complete=100.00%
2024-12-04T01:31:53.947Z [INFO]  server-460bb384-1906-f4e2-824b-0a508784c2fe: restored from snapshot: id=2-2-1733275913945 last-index=2 last-term=2 size-in-bytes=1
2024-12-04T01:31:53.947Z [INFO]  server-460bb384-1906-f4e2-824b-0a508784c2fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-952f59bd-6d40-f082-dbfe-7e0749d4e34f Address:952f59bd-6d40-f082-dbfe-7e0749d4e34f} {Suffrage:Voter ID:server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8 Address:09d40183-e7ed-53d2-04f0-af8efd9c5fd8} {Suffrage:Voter ID:server-460bb384-1906-f4e2-824b-0a508784c2fe Address:460bb384-1906-f4e2-824b-0a508784c2fe}]"
2024-12-04T01:31:53.947Z [INFO]  server-460bb384-1906-f4e2-824b-0a508784c2fe: entering follower state: follower="Node at 460bb384-1906-f4e2-824b-0a508784c2fe [Follower]" leader-address= leader-id=
2024-12-04T01:31:54.014Z [WARN]  server-952f59bd-6d40-f082-dbfe-7e0749d4e34f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:54.014Z [INFO]  server-952f59bd-6d40-f082-dbfe-7e0749d4e34f: entering candidate state: node="Node at 952f59bd-6d40-f082-dbfe-7e0749d4e34f [Candidate]" term=3
2024-12-04T01:31:54.014Z [INFO]  server-952f59bd-6d40-f082-dbfe-7e0749d4e34f: election won: term=3 tally=2
2024-12-04T01:31:54.014Z [INFO]  server-952f59bd-6d40-f082-dbfe-7e0749d4e34f: entering leader state: leader="Node at 952f59bd-6d40-f082-dbfe-7e0749d4e34f [Leader]"
2024-12-04T01:31:54.014Z [INFO]  server-952f59bd-6d40-f082-dbfe-7e0749d4e34f: added peer, starting replication: peer=server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8
2024-12-04T01:31:54.014Z [INFO]  server-952f59bd-6d40-f082-dbfe-7e0749d4e34f: added peer, starting replication: peer=server-460bb384-1906-f4e2-824b-0a508784c2fe
2024-12-04T01:31:54.014Z [INFO]  server-952f59bd-6d40-f082-dbfe-7e0749d4e34f: pipelining replication: peer="{Voter server-460bb384-1906-f4e2-824b-0a508784c2fe 460bb384-1906-f4e2-824b-0a508784c2fe}"
2024-12-04T01:31:54.014Z [INFO]  server-952f59bd-6d40-f082-dbfe-7e0749d4e34f: pipelining replication: peer="{Voter server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8 09d40183-e7ed-53d2-04f0-af8efd9c5fd8}"
2024-12-04T01:31:54.262Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:54.370Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:54.262863531 +0000 UTC m=+5.094706607 (1 nodes), 66.635µs from start of poll, 497.399345ms from cluster start. Timeout at 2024-12-04 01:31:54.367888757 +0000 UTC m=+5.199732727, 105.02612ms after stability
2024-12-04T01:31:54.370Z [INFO]  server-952f59bd-6d40-f082-dbfe-7e0749d4e34f: aborting pipeline replication: peer="{Voter server-09d40183-e7ed-53d2-04f0-af8efd9c5fd8 09d40183-e7ed-53d2-04f0-af8efd9c5fd8}"
2024-12-04T01:31:54.370Z [INFO]  server-952f59bd-6d40-f082-dbfe-7e0749d4e34f: aborting pipeline replication: peer="{Voter server-460bb384-1906-f4e2-824b-0a508784c2fe 460bb384-1906-f4e2-824b-0a508784c2fe}"
=== RUN   TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs
2024-12-04T01:31:54.374Z [INFO]  server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243 Address:a1792d44-1d9d-7fd9-9dd5-b71771ebb243} {Suffrage:Voter ID:server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684 Address:403f75e4-fd9f-ba4d-f8b6-c8f5ce848684} {Suffrage:Voter ID:server-65dd9823-12a8-672e-58d4-2b2c17b76a1c Address:65dd9823-12a8-672e-58d4-2b2c17b76a1c}]"
2024-12-04T01:31:54.374Z [INFO]  server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243 Address:a1792d44-1d9d-7fd9-9dd5-b71771ebb243} {Suffrage:Voter ID:server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684 Address:403f75e4-fd9f-ba4d-f8b6-c8f5ce848684} {Suffrage:Voter ID:server-65dd9823-12a8-672e-58d4-2b2c17b76a1c Address:65dd9823-12a8-672e-58d4-2b2c17b76a1c}]"
2024-12-04T01:31:54.374Z [INFO]  server-65dd9823-12a8-672e-58d4-2b2c17b76a1c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243 Address:a1792d44-1d9d-7fd9-9dd5-b71771ebb243} {Suffrage:Voter ID:server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684 Address:403f75e4-fd9f-ba4d-f8b6-c8f5ce848684} {Suffrage:Voter ID:server-65dd9823-12a8-672e-58d4-2b2c17b76a1c Address:65dd9823-12a8-672e-58d4-2b2c17b76a1c}]"
2024-12-04T01:31:54.374Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:54.374Z [INFO]  server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243: entering follower state: follower="Node at a1792d44-1d9d-7fd9-9dd5-b71771ebb243 [Follower]" leader-address= leader-id=
2024-12-04T01:31:54.375Z [INFO]  server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684: entering follower state: follower="Node at 403f75e4-fd9f-ba4d-f8b6-c8f5ce848684 [Follower]" leader-address= leader-id=
2024-12-04T01:31:54.375Z [INFO]  server-65dd9823-12a8-672e-58d4-2b2c17b76a1c: entering follower state: follower="Node at 65dd9823-12a8-672e-58d4-2b2c17b76a1c [Follower]" leader-address= leader-id=
2024-12-04T01:31:54.438Z [WARN]  server-65dd9823-12a8-672e-58d4-2b2c17b76a1c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:54.438Z [INFO]  server-65dd9823-12a8-672e-58d4-2b2c17b76a1c: entering candidate state: node="Node at 65dd9823-12a8-672e-58d4-2b2c17b76a1c [Candidate]" term=2
2024-12-04T01:31:54.438Z [INFO]  server-65dd9823-12a8-672e-58d4-2b2c17b76a1c: election won: term=2 tally=2
2024-12-04T01:31:54.438Z [INFO]  server-65dd9823-12a8-672e-58d4-2b2c17b76a1c: entering leader state: leader="Node at 65dd9823-12a8-672e-58d4-2b2c17b76a1c [Leader]"
2024-12-04T01:31:54.438Z [INFO]  server-65dd9823-12a8-672e-58d4-2b2c17b76a1c: added peer, starting replication: peer=server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243
2024-12-04T01:31:54.438Z [INFO]  server-65dd9823-12a8-672e-58d4-2b2c17b76a1c: added peer, starting replication: peer=server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684
2024-12-04T01:31:54.439Z [INFO]  server-65dd9823-12a8-672e-58d4-2b2c17b76a1c: pipelining replication: peer="{Voter server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684 403f75e4-fd9f-ba4d-f8b6-c8f5ce848684}"
2024-12-04T01:31:54.439Z [INFO]  server-65dd9823-12a8-672e-58d4-2b2c17b76a1c: pipelining replication: peer="{Voter server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243 a1792d44-1d9d-7fd9-9dd5-b71771ebb243}"
2024-12-04T01:31:54.548Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:54.439409694 +0000 UTC m=+5.271252308 (1 nodes), 64.504079ms from start of poll, 65.508537ms from cluster start. Timeout at 2024-12-04 01:31:54.544565435 +0000 UTC m=+5.376408325, 105.156017ms after stability
2024-12-04T01:31:54.550Z [INFO]  server-65dd9823-12a8-672e-58d4-2b2c17b76a1c: aborting pipeline replication: peer="{Voter server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684 403f75e4-fd9f-ba4d-f8b6-c8f5ce848684}"
2024-12-04T01:31:54.550Z [INFO]  server-65dd9823-12a8-672e-58d4-2b2c17b76a1c: aborting pipeline replication: peer="{Voter server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243 a1792d44-1d9d-7fd9-9dd5-b71771ebb243}"
2024-12-04T01:31:54.551Z [INFO]  creating new snapshot: path=/tmp/raft3301856245/snapshots/2-6-1733275914551.tmp
2024-12-04T01:31:54.568Z [INFO]  server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243: starting restore from snapshot: id=2-6-1733275914551 last-index=6 last-term=2 size-in-bytes=25
2024-12-04T01:31:54.569Z [INFO]  server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243: snapshot restore progress: id=2-6-1733275914551 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2024-12-04T01:31:54.569Z [INFO]  server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243: restored from snapshot: id=2-6-1733275914551 last-index=6 last-term=2 size-in-bytes=25
2024-12-04T01:31:54.569Z [INFO]  server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243 Address:a1792d44-1d9d-7fd9-9dd5-b71771ebb243} {Suffrage:Voter ID:server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684 Address:403f75e4-fd9f-ba4d-f8b6-c8f5ce848684} {Suffrage:Voter ID:server-65dd9823-12a8-672e-58d4-2b2c17b76a1c Address:65dd9823-12a8-672e-58d4-2b2c17b76a1c}]"
2024-12-04T01:31:54.569Z [INFO]  creating new snapshot: path=/tmp/raft3525732306/snapshots/2-6-1733275914569.tmp
2024-12-04T01:31:54.570Z [INFO]  server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684: starting restore from snapshot: id=2-6-1733275914569 last-index=6 last-term=2 size-in-bytes=25
2024-12-04T01:31:54.571Z [INFO]  server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684: snapshot restore progress: id=2-6-1733275914569 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2024-12-04T01:31:54.571Z [INFO]  server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684: restored from snapshot: id=2-6-1733275914569 last-index=6 last-term=2 size-in-bytes=25
2024-12-04T01:31:54.571Z [INFO]  server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243 Address:a1792d44-1d9d-7fd9-9dd5-b71771ebb243} {Suffrage:Voter ID:server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684 Address:403f75e4-fd9f-ba4d-f8b6-c8f5ce848684} {Suffrage:Voter ID:server-65dd9823-12a8-672e-58d4-2b2c17b76a1c Address:65dd9823-12a8-672e-58d4-2b2c17b76a1c}]"
2024-12-04T01:31:54.571Z [INFO]  creating new snapshot: path=/tmp/raft1881738341/snapshots/2-6-1733275914571.tmp
2024-12-04T01:31:54.572Z [INFO]  server-65dd9823-12a8-672e-58d4-2b2c17b76a1c: starting restore from snapshot: id=2-6-1733275914571 last-index=6 last-term=2 size-in-bytes=25
2024-12-04T01:31:54.572Z [INFO]  server-65dd9823-12a8-672e-58d4-2b2c17b76a1c: snapshot restore progress: id=2-6-1733275914571 last-index=6 last-term=2 size-in-bytes=25 read-bytes=25 percent-complete=100.00%
2024-12-04T01:31:54.572Z [INFO]  server-65dd9823-12a8-672e-58d4-2b2c17b76a1c: restored from snapshot: id=2-6-1733275914571 last-index=6 last-term=2 size-in-bytes=25
2024-12-04T01:31:54.572Z [INFO]  server-65dd9823-12a8-672e-58d4-2b2c17b76a1c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243 Address:a1792d44-1d9d-7fd9-9dd5-b71771ebb243} {Suffrage:Voter ID:server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684 Address:403f75e4-fd9f-ba4d-f8b6-c8f5ce848684} {Suffrage:Voter ID:server-65dd9823-12a8-672e-58d4-2b2c17b76a1c Address:65dd9823-12a8-672e-58d4-2b2c17b76a1c}]"
2024-12-04T01:31:54.572Z [INFO]  server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684: entering follower state: follower="Node at 403f75e4-fd9f-ba4d-f8b6-c8f5ce848684 [Follower]" leader-address= leader-id=
2024-12-04T01:31:54.572Z [INFO]  server-65dd9823-12a8-672e-58d4-2b2c17b76a1c: entering follower state: follower="Node at 65dd9823-12a8-672e-58d4-2b2c17b76a1c [Follower]" leader-address= leader-id=
2024-12-04T01:31:54.572Z [INFO]  server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243: entering follower state: follower="Node at a1792d44-1d9d-7fd9-9dd5-b71771ebb243 [Follower]" leader-address= leader-id=
2024-12-04T01:31:54.623Z [WARN]  server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:54.623Z [INFO]  server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243: entering candidate state: node="Node at a1792d44-1d9d-7fd9-9dd5-b71771ebb243 [Candidate]" term=3
2024-12-04T01:31:54.623Z [INFO]  server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243: election won: term=3 tally=2
2024-12-04T01:31:54.623Z [INFO]  server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243: entering leader state: leader="Node at a1792d44-1d9d-7fd9-9dd5-b71771ebb243 [Leader]"
2024-12-04T01:31:54.623Z [INFO]  server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243: added peer, starting replication: peer=server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684
2024-12-04T01:31:54.623Z [INFO]  server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243: added peer, starting replication: peer=server-65dd9823-12a8-672e-58d4-2b2c17b76a1c
2024-12-04T01:31:54.623Z [INFO]  server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243: pipelining replication: peer="{Voter server-65dd9823-12a8-672e-58d4-2b2c17b76a1c 65dd9823-12a8-672e-58d4-2b2c17b76a1c}"
2024-12-04T01:31:54.623Z [INFO]  server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243: pipelining replication: peer="{Voter server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684 403f75e4-fd9f-ba4d-f8b6-c8f5ce848684}"
2024-12-04T01:31:54.887Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:54.998Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:54.887938466 +0000 UTC m=+5.719781113 (1 nodes), 43.568µs from start of poll, 514.037342ms from cluster start. Timeout at 2024-12-04 01:31:54.992954958 +0000 UTC m=+5.824797633, 105.01652ms after stability
2024-12-04T01:31:54.998Z [INFO]  server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243: aborting pipeline replication: peer="{Voter server-65dd9823-12a8-672e-58d4-2b2c17b76a1c 65dd9823-12a8-672e-58d4-2b2c17b76a1c}"
2024-12-04T01:31:54.998Z [INFO]  server-a1792d44-1d9d-7fd9-9dd5-b71771ebb243: aborting pipeline replication: peer="{Voter server-403f75e4-fd9f-ba4d-f8b6-c8f5ce848684 403f75e4-fd9f-ba4d-f8b6-c8f5ce848684}"
=== RUN   TestRaft_RecoverCluster/snapshot,_with_trailing_logs
2024-12-04T01:31:55.001Z [INFO]  server-0973e852-0bae-db92-7046-d405cfbe4c26: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0973e852-0bae-db92-7046-d405cfbe4c26 Address:0973e852-0bae-db92-7046-d405cfbe4c26} {Suffrage:Voter ID:server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 Address:88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8} {Suffrage:Voter ID:server-97d25109-9d18-7b31-4235-bd3a87c52dc2 Address:97d25109-9d18-7b31-4235-bd3a87c52dc2}]"
2024-12-04T01:31:55.001Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0973e852-0bae-db92-7046-d405cfbe4c26 Address:0973e852-0bae-db92-7046-d405cfbe4c26} {Suffrage:Voter ID:server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 Address:88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8} {Suffrage:Voter ID:server-97d25109-9d18-7b31-4235-bd3a87c52dc2 Address:97d25109-9d18-7b31-4235-bd3a87c52dc2}]"
2024-12-04T01:31:55.001Z [INFO]  server-97d25109-9d18-7b31-4235-bd3a87c52dc2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0973e852-0bae-db92-7046-d405cfbe4c26 Address:0973e852-0bae-db92-7046-d405cfbe4c26} {Suffrage:Voter ID:server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 Address:88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8} {Suffrage:Voter ID:server-97d25109-9d18-7b31-4235-bd3a87c52dc2 Address:97d25109-9d18-7b31-4235-bd3a87c52dc2}]"
2024-12-04T01:31:55.001Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:55.001Z [INFO]  server-0973e852-0bae-db92-7046-d405cfbe4c26: entering follower state: follower="Node at 0973e852-0bae-db92-7046-d405cfbe4c26 [Follower]" leader-address= leader-id=
2024-12-04T01:31:55.001Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: entering follower state: follower="Node at 88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 [Follower]" leader-address= leader-id=
2024-12-04T01:31:55.002Z [INFO]  server-97d25109-9d18-7b31-4235-bd3a87c52dc2: entering follower state: follower="Node at 97d25109-9d18-7b31-4235-bd3a87c52dc2 [Follower]" leader-address= leader-id=
2024-12-04T01:31:55.066Z [WARN]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:55.066Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: entering candidate state: node="Node at 88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 [Candidate]" term=2
2024-12-04T01:31:55.066Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: election won: term=2 tally=2
2024-12-04T01:31:55.066Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: entering leader state: leader="Node at 88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 [Leader]"
2024-12-04T01:31:55.066Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: added peer, starting replication: peer=server-0973e852-0bae-db92-7046-d405cfbe4c26
2024-12-04T01:31:55.066Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: added peer, starting replication: peer=server-97d25109-9d18-7b31-4235-bd3a87c52dc2
2024-12-04T01:31:55.066Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: pipelining replication: peer="{Voter server-97d25109-9d18-7b31-4235-bd3a87c52dc2 97d25109-9d18-7b31-4235-bd3a87c52dc2}"
2024-12-04T01:31:55.067Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: pipelining replication: peer="{Voter server-0973e852-0bae-db92-7046-d405cfbe4c26 0973e852-0bae-db92-7046-d405cfbe4c26}"
2024-12-04T01:31:55.174Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:55.067171282 +0000 UTC m=+5.899014634 (1 nodes), 65.33998ms from start of poll, 65.867451ms from cluster start. Timeout at 2024-12-04 01:31:55.172189512 +0000 UTC m=+6.004032102, 105.017468ms after stability
2024-12-04T01:31:55.187Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: aborting pipeline replication: peer="{Voter server-97d25109-9d18-7b31-4235-bd3a87c52dc2 97d25109-9d18-7b31-4235-bd3a87c52dc2}"
2024-12-04T01:31:55.187Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: aborting pipeline replication: peer="{Voter server-0973e852-0bae-db92-7046-d405cfbe4c26 0973e852-0bae-db92-7046-d405cfbe4c26}"
2024-12-04T01:31:55.187Z [INFO]  creating new snapshot: path=/tmp/raft2430582065/snapshots/2-27-1733275915187.tmp
2024-12-04T01:31:55.207Z [INFO]  server-0973e852-0bae-db92-7046-d405cfbe4c26: starting restore from snapshot: id=2-27-1733275915187 last-index=27 last-term=2 size-in-bytes=168
2024-12-04T01:31:55.207Z [INFO]  server-0973e852-0bae-db92-7046-d405cfbe4c26: snapshot restore progress: id=2-27-1733275915187 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2024-12-04T01:31:55.207Z [INFO]  server-0973e852-0bae-db92-7046-d405cfbe4c26: restored from snapshot: id=2-27-1733275915187 last-index=27 last-term=2 size-in-bytes=168
2024-12-04T01:31:55.207Z [INFO]  server-0973e852-0bae-db92-7046-d405cfbe4c26: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0973e852-0bae-db92-7046-d405cfbe4c26 Address:0973e852-0bae-db92-7046-d405cfbe4c26} {Suffrage:Voter ID:server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 Address:88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8} {Suffrage:Voter ID:server-97d25109-9d18-7b31-4235-bd3a87c52dc2 Address:97d25109-9d18-7b31-4235-bd3a87c52dc2}]"
2024-12-04T01:31:55.207Z [INFO]  creating new snapshot: path=/tmp/raft409274211/snapshots/2-27-1733275915207.tmp
2024-12-04T01:31:55.210Z [INFO]  server-0973e852-0bae-db92-7046-d405cfbe4c26: entering follower state: follower="Node at 0973e852-0bae-db92-7046-d405cfbe4c26 [Follower]" leader-address= leader-id=
2024-12-04T01:31:55.304Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: starting restore from snapshot: id=2-27-1733275915207 last-index=27 last-term=2 size-in-bytes=168
2024-12-04T01:31:55.304Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: snapshot restore progress: id=2-27-1733275915207 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2024-12-04T01:31:55.304Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: restored from snapshot: id=2-27-1733275915207 last-index=27 last-term=2 size-in-bytes=168
2024-12-04T01:31:55.304Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0973e852-0bae-db92-7046-d405cfbe4c26 Address:0973e852-0bae-db92-7046-d405cfbe4c26} {Suffrage:Voter ID:server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 Address:88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8} {Suffrage:Voter ID:server-97d25109-9d18-7b31-4235-bd3a87c52dc2 Address:97d25109-9d18-7b31-4235-bd3a87c52dc2}]"
2024-12-04T01:31:55.304Z [INFO]  creating new snapshot: path=/tmp/raft745615982/snapshots/2-27-1733275915304.tmp
2024-12-04T01:31:55.305Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: entering follower state: follower="Node at 88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 [Follower]" leader-address= leader-id=
2024-12-04T01:31:55.306Z [WARN]  server-0973e852-0bae-db92-7046-d405cfbe4c26: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:55.306Z [INFO]  server-0973e852-0bae-db92-7046-d405cfbe4c26: entering candidate state: node="Node at 0973e852-0bae-db92-7046-d405cfbe4c26 [Candidate]" term=3
2024-12-04T01:31:55.306Z [ERROR] server-0973e852-0bae-db92-7046-d405cfbe4c26: failed to make requestVote RPC: target="{Voter server-97d25109-9d18-7b31-4235-bd3a87c52dc2 97d25109-9d18-7b31-4235-bd3a87c52dc2}" error="failed to connect to peer: 97d25109-9d18-7b31-4235-bd3a87c52dc2" term=3
2024-12-04T01:31:55.306Z [ERROR] server-0973e852-0bae-db92-7046-d405cfbe4c26: failed to make requestVote RPC: target="{Voter server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8}" error="failed to connect to peer: 88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8" term=3
2024-12-04T01:31:55.376Z [WARN]  server-0973e852-0bae-db92-7046-d405cfbe4c26: Election timeout reached, restarting election
2024-12-04T01:31:55.376Z [INFO]  server-0973e852-0bae-db92-7046-d405cfbe4c26: entering candidate state: node="Node at 0973e852-0bae-db92-7046-d405cfbe4c26 [Candidate]" term=4
2024-12-04T01:31:55.382Z [ERROR] server-0973e852-0bae-db92-7046-d405cfbe4c26: failed to make requestVote RPC: target="{Voter server-97d25109-9d18-7b31-4235-bd3a87c52dc2 97d25109-9d18-7b31-4235-bd3a87c52dc2}" error="failed to connect to peer: 97d25109-9d18-7b31-4235-bd3a87c52dc2" term=4
2024-12-04T01:31:55.382Z [ERROR] server-0973e852-0bae-db92-7046-d405cfbe4c26: failed to make requestVote RPC: target="{Voter server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8}" error="failed to connect to peer: 88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8" term=4
2024-12-04T01:31:55.382Z [INFO]  server-97d25109-9d18-7b31-4235-bd3a87c52dc2: starting restore from snapshot: id=2-27-1733275915304 last-index=27 last-term=2 size-in-bytes=168
2024-12-04T01:31:55.383Z [INFO]  server-97d25109-9d18-7b31-4235-bd3a87c52dc2: snapshot restore progress: id=2-27-1733275915304 last-index=27 last-term=2 size-in-bytes=168 read-bytes=168 percent-complete=100.00%
2024-12-04T01:31:55.383Z [INFO]  server-97d25109-9d18-7b31-4235-bd3a87c52dc2: restored from snapshot: id=2-27-1733275915304 last-index=27 last-term=2 size-in-bytes=168
2024-12-04T01:31:55.383Z [INFO]  server-97d25109-9d18-7b31-4235-bd3a87c52dc2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0973e852-0bae-db92-7046-d405cfbe4c26 Address:0973e852-0bae-db92-7046-d405cfbe4c26} {Suffrage:Voter ID:server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 Address:88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8} {Suffrage:Voter ID:server-97d25109-9d18-7b31-4235-bd3a87c52dc2 Address:97d25109-9d18-7b31-4235-bd3a87c52dc2}]"
2024-12-04T01:31:55.383Z [INFO]  server-97d25109-9d18-7b31-4235-bd3a87c52dc2: entering follower state: follower="Node at 97d25109-9d18-7b31-4235-bd3a87c52dc2 [Follower]" leader-address= leader-id=
2024-12-04T01:31:55.383Z [WARN]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:55.383Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: entering candidate state: node="Node at 88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 [Candidate]" term=3
2024-12-04T01:31:55.383Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: election won: term=3 tally=2
2024-12-04T01:31:55.383Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: entering leader state: leader="Node at 88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 [Leader]"
2024-12-04T01:31:55.383Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: added peer, starting replication: peer=server-0973e852-0bae-db92-7046-d405cfbe4c26
2024-12-04T01:31:55.383Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: added peer, starting replication: peer=server-97d25109-9d18-7b31-4235-bd3a87c52dc2
2024-12-04T01:31:55.383Z [ERROR] server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: peer has newer term, stopping replication: peer="{Voter server-0973e852-0bae-db92-7046-d405cfbe4c26 0973e852-0bae-db92-7046-d405cfbe4c26}"
2024-12-04T01:31:55.383Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: entering follower state: follower="Node at 88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 [Follower]" leader-address= leader-id=
2024-12-04T01:31:55.383Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: pipelining replication: peer="{Voter server-97d25109-9d18-7b31-4235-bd3a87c52dc2 97d25109-9d18-7b31-4235-bd3a87c52dc2}"
2024-12-04T01:31:55.383Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: aborting pipeline replication: peer="{Voter server-97d25109-9d18-7b31-4235-bd3a87c52dc2 97d25109-9d18-7b31-4235-bd3a87c52dc2}"
2024-12-04T01:31:55.437Z [WARN]  server-0973e852-0bae-db92-7046-d405cfbe4c26: Election timeout reached, restarting election
2024-12-04T01:31:55.437Z [INFO]  server-0973e852-0bae-db92-7046-d405cfbe4c26: entering candidate state: node="Node at 0973e852-0bae-db92-7046-d405cfbe4c26 [Candidate]" term=5
2024-12-04T01:31:55.438Z [WARN]  server-97d25109-9d18-7b31-4235-bd3a87c52dc2: rejecting vote request since we have a leader: from=0973e852-0bae-db92-7046-d405cfbe4c26 leader=88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 leader-id=server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8
2024-12-04T01:31:55.438Z [WARN]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: rejecting vote request since our last term is greater: candidate=0973e852-0bae-db92-7046-d405cfbe4c26 last-term=3 last-candidate-term=2
2024-12-04T01:31:55.473Z [WARN]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:55.473Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: entering candidate state: node="Node at 88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 [Candidate]" term=6
2024-12-04T01:31:55.473Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: election won: term=6 tally=2
2024-12-04T01:31:55.473Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: entering leader state: leader="Node at 88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8 [Leader]"
2024-12-04T01:31:55.473Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: added peer, starting replication: peer=server-0973e852-0bae-db92-7046-d405cfbe4c26
2024-12-04T01:31:55.473Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: added peer, starting replication: peer=server-97d25109-9d18-7b31-4235-bd3a87c52dc2
2024-12-04T01:31:55.473Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: pipelining replication: peer="{Voter server-97d25109-9d18-7b31-4235-bd3a87c52dc2 97d25109-9d18-7b31-4235-bd3a87c52dc2}"
2024-12-04T01:31:55.473Z [INFO]  server-0973e852-0bae-db92-7046-d405cfbe4c26: entering follower state: follower="Node at 0973e852-0bae-db92-7046-d405cfbe4c26 [Follower]" leader-address= leader-id=
2024-12-04T01:31:55.473Z [WARN]  server-0973e852-0bae-db92-7046-d405cfbe4c26: failed to get previous log: previous-index=28 last-index=27 error="log not found"
2024-12-04T01:31:55.473Z [WARN]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: appendEntries rejected, sending older logs: peer="{Voter server-0973e852-0bae-db92-7046-d405cfbe4c26 0973e852-0bae-db92-7046-d405cfbe4c26}" next=28
2024-12-04T01:31:55.473Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: pipelining replication: peer="{Voter server-0973e852-0bae-db92-7046-d405cfbe4c26 0973e852-0bae-db92-7046-d405cfbe4c26}"
2024-12-04T01:31:55.698Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:55.806Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:55.698698888 +0000 UTC m=+6.530541649 (1 nodes), 54.589µs from start of poll, 697.394466ms from cluster start. Timeout at 2024-12-04 01:31:55.803725057 +0000 UTC m=+6.635567654, 105.026005ms after stability
2024-12-04T01:31:55.806Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: aborting pipeline replication: peer="{Voter server-0973e852-0bae-db92-7046-d405cfbe4c26 0973e852-0bae-db92-7046-d405cfbe4c26}"
2024-12-04T01:31:55.806Z [INFO]  server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: aborting pipeline replication: peer="{Voter server-97d25109-9d18-7b31-4235-bd3a87c52dc2 97d25109-9d18-7b31-4235-bd3a87c52dc2}"
2024-12-04T01:31:56.310Z [ERROR] server-88b0abc6-2dcb-54b7-6f2e-9ecf02e99bf8: failed to heartbeat to: peer=0973e852-0bae-db92-7046-d405cfbe4c26 backoff time=10ms error="command timed out"
--- PASS: TestRaft_RecoverCluster (2.55s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_no_trailing_logs (0.61s)
    --- PASS: TestRaft_RecoverCluster/no_snapshot,_some_trailing_logs (0.63s)
    --- PASS: TestRaft_RecoverCluster/snapshot,_with_trailing_logs (1.31s)
=== RUN   TestRaft_HasExistingState
    raft_test.go:258: Skipping unreliable test
--- SKIP: TestRaft_HasExistingState (0.00s)
=== RUN   TestRaft_SingleNode
2024-12-04T01:31:56.312Z [INFO]  server-1fc14935-76e8-6fbf-5ef0-6902a368729e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1fc14935-76e8-6fbf-5ef0-6902a368729e Address:1fc14935-76e8-6fbf-5ef0-6902a368729e}]"
2024-12-04T01:31:56.312Z [INFO]  server-1fc14935-76e8-6fbf-5ef0-6902a368729e: entering follower state: follower="Node at 1fc14935-76e8-6fbf-5ef0-6902a368729e [Follower]" leader-address= leader-id=
2024-12-04T01:31:56.378Z [WARN]  server-1fc14935-76e8-6fbf-5ef0-6902a368729e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:56.378Z [INFO]  server-1fc14935-76e8-6fbf-5ef0-6902a368729e: entering candidate state: node="Node at 1fc14935-76e8-6fbf-5ef0-6902a368729e [Candidate]" term=2
2024-12-04T01:31:56.378Z [INFO]  server-1fc14935-76e8-6fbf-5ef0-6902a368729e: election won: term=2 tally=1
2024-12-04T01:31:56.378Z [INFO]  server-1fc14935-76e8-6fbf-5ef0-6902a368729e: entering leader state: leader="Node at 1fc14935-76e8-6fbf-5ef0-6902a368729e [Leader]"
--- PASS: TestRaft_SingleNode (0.07s)
=== RUN   TestRaft_TripleNode
2024-12-04T01:31:56.380Z [INFO]  server-09c8ba06-d706-572a-bf90-3c02ee727cf7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09c8ba06-d706-572a-bf90-3c02ee727cf7 Address:09c8ba06-d706-572a-bf90-3c02ee727cf7} {Suffrage:Voter ID:server-92d391f9-7805-2c1c-756c-cde54100c67e Address:92d391f9-7805-2c1c-756c-cde54100c67e} {Suffrage:Voter ID:server-57e1e272-e215-1be2-f6e1-143bc3c30535 Address:57e1e272-e215-1be2-f6e1-143bc3c30535}]"
2024-12-04T01:31:56.380Z [INFO]  server-92d391f9-7805-2c1c-756c-cde54100c67e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09c8ba06-d706-572a-bf90-3c02ee727cf7 Address:09c8ba06-d706-572a-bf90-3c02ee727cf7} {Suffrage:Voter ID:server-92d391f9-7805-2c1c-756c-cde54100c67e Address:92d391f9-7805-2c1c-756c-cde54100c67e} {Suffrage:Voter ID:server-57e1e272-e215-1be2-f6e1-143bc3c30535 Address:57e1e272-e215-1be2-f6e1-143bc3c30535}]"
2024-12-04T01:31:56.380Z [INFO]  server-57e1e272-e215-1be2-f6e1-143bc3c30535: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09c8ba06-d706-572a-bf90-3c02ee727cf7 Address:09c8ba06-d706-572a-bf90-3c02ee727cf7} {Suffrage:Voter ID:server-92d391f9-7805-2c1c-756c-cde54100c67e Address:92d391f9-7805-2c1c-756c-cde54100c67e} {Suffrage:Voter ID:server-57e1e272-e215-1be2-f6e1-143bc3c30535 Address:57e1e272-e215-1be2-f6e1-143bc3c30535}]"
2024-12-04T01:31:56.380Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:31:56.380Z [INFO]  server-09c8ba06-d706-572a-bf90-3c02ee727cf7: entering follower state: follower="Node at 09c8ba06-d706-572a-bf90-3c02ee727cf7 [Follower]" leader-address= leader-id=
2024-12-04T01:31:56.380Z [INFO]  server-92d391f9-7805-2c1c-756c-cde54100c67e: entering follower state: follower="Node at 92d391f9-7805-2c1c-756c-cde54100c67e [Follower]" leader-address= leader-id=
2024-12-04T01:31:56.380Z [INFO]  server-57e1e272-e215-1be2-f6e1-143bc3c30535: entering follower state: follower="Node at 57e1e272-e215-1be2-f6e1-143bc3c30535 [Follower]" leader-address= leader-id=
2024-12-04T01:31:56.450Z [WARN]  server-57e1e272-e215-1be2-f6e1-143bc3c30535: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:56.450Z [INFO]  server-57e1e272-e215-1be2-f6e1-143bc3c30535: entering candidate state: node="Node at 57e1e272-e215-1be2-f6e1-143bc3c30535 [Candidate]" term=2
2024-12-04T01:31:56.450Z [INFO]  server-57e1e272-e215-1be2-f6e1-143bc3c30535: election won: term=2 tally=2
2024-12-04T01:31:56.450Z [INFO]  server-57e1e272-e215-1be2-f6e1-143bc3c30535: entering leader state: leader="Node at 57e1e272-e215-1be2-f6e1-143bc3c30535 [Leader]"
2024-12-04T01:31:56.450Z [INFO]  server-57e1e272-e215-1be2-f6e1-143bc3c30535: added peer, starting replication: peer=server-09c8ba06-d706-572a-bf90-3c02ee727cf7
2024-12-04T01:31:56.450Z [INFO]  server-57e1e272-e215-1be2-f6e1-143bc3c30535: added peer, starting replication: peer=server-92d391f9-7805-2c1c-756c-cde54100c67e
2024-12-04T01:31:56.450Z [INFO]  server-57e1e272-e215-1be2-f6e1-143bc3c30535: pipelining replication: peer="{Voter server-92d391f9-7805-2c1c-756c-cde54100c67e 92d391f9-7805-2c1c-756c-cde54100c67e}"
2024-12-04T01:31:56.451Z [INFO]  server-57e1e272-e215-1be2-f6e1-143bc3c30535: pipelining replication: peer="{Voter server-09c8ba06-d706-572a-bf90-3c02ee727cf7 09c8ba06-d706-572a-bf90-3c02ee727cf7}"
2024-12-04T01:31:56.574Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:31:56.451383916 +0000 UTC m=+7.283226612 (2 nodes), 70.881673ms from start of poll, 71.380586ms from cluster start. Timeout at 2024-12-04 01:31:56.556393447 +0000 UTC m=+7.388236099, 105.009487ms after stability
2024-12-04T01:31:56.574Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:56.681Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:56.57499453 +0000 UTC m=+7.406837154 (1 nodes), 2.206µs from start of poll, 194.991128ms from cluster start. Timeout at 2024-12-04 01:31:56.680008676 +0000 UTC m=+7.511851412, 105.014258ms after stability
2024-12-04T01:31:56.698Z [INFO]  server-57e1e272-e215-1be2-f6e1-143bc3c30535: aborting pipeline replication: peer="{Voter server-92d391f9-7805-2c1c-756c-cde54100c67e 92d391f9-7805-2c1c-756c-cde54100c67e}"
2024-12-04T01:31:56.698Z [INFO]  server-57e1e272-e215-1be2-f6e1-143bc3c30535: aborting pipeline replication: peer="{Voter server-09c8ba06-d706-572a-bf90-3c02ee727cf7 09c8ba06-d706-572a-bf90-3c02ee727cf7}"
--- PASS: TestRaft_TripleNode (0.32s)
=== RUN   TestRaft_LeaderFail
2024-12-04T01:31:56.700Z [INFO]  server-3a729bdb-291e-7129-3759-e3284e4e39d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a729bdb-291e-7129-3759-e3284e4e39d7 Address:3a729bdb-291e-7129-3759-e3284e4e39d7} {Suffrage:Voter ID:server-9c2d41c7-b347-97d5-6de1-71256fa62dee Address:9c2d41c7-b347-97d5-6de1-71256fa62dee} {Suffrage:Voter ID:server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa Address:90c9edc4-a781-2a80-3c72-d0587ad9e7fa}]"
2024-12-04T01:31:56.700Z [INFO]  server-9c2d41c7-b347-97d5-6de1-71256fa62dee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a729bdb-291e-7129-3759-e3284e4e39d7 Address:3a729bdb-291e-7129-3759-e3284e4e39d7} {Suffrage:Voter ID:server-9c2d41c7-b347-97d5-6de1-71256fa62dee Address:9c2d41c7-b347-97d5-6de1-71256fa62dee} {Suffrage:Voter ID:server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa Address:90c9edc4-a781-2a80-3c72-d0587ad9e7fa}]"
2024-12-04T01:31:56.700Z [INFO]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a729bdb-291e-7129-3759-e3284e4e39d7 Address:3a729bdb-291e-7129-3759-e3284e4e39d7} {Suffrage:Voter ID:server-9c2d41c7-b347-97d5-6de1-71256fa62dee Address:9c2d41c7-b347-97d5-6de1-71256fa62dee} {Suffrage:Voter ID:server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa Address:90c9edc4-a781-2a80-3c72-d0587ad9e7fa}]"
2024-12-04T01:31:56.700Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:31:56.700Z [INFO]  server-3a729bdb-291e-7129-3759-e3284e4e39d7: entering follower state: follower="Node at 3a729bdb-291e-7129-3759-e3284e4e39d7 [Follower]" leader-address= leader-id=
2024-12-04T01:31:56.700Z [INFO]  server-9c2d41c7-b347-97d5-6de1-71256fa62dee: entering follower state: follower="Node at 9c2d41c7-b347-97d5-6de1-71256fa62dee [Follower]" leader-address= leader-id=
2024-12-04T01:31:56.700Z [INFO]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: entering follower state: follower="Node at 90c9edc4-a781-2a80-3c72-d0587ad9e7fa [Follower]" leader-address= leader-id=
2024-12-04T01:31:56.772Z [WARN]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:56.772Z [INFO]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: entering candidate state: node="Node at 90c9edc4-a781-2a80-3c72-d0587ad9e7fa [Candidate]" term=2
2024-12-04T01:31:56.772Z [INFO]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: election won: term=2 tally=2
2024-12-04T01:31:56.772Z [INFO]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: entering leader state: leader="Node at 90c9edc4-a781-2a80-3c72-d0587ad9e7fa [Leader]"
2024-12-04T01:31:56.772Z [INFO]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: added peer, starting replication: peer=server-3a729bdb-291e-7129-3759-e3284e4e39d7
2024-12-04T01:31:56.772Z [INFO]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: added peer, starting replication: peer=server-9c2d41c7-b347-97d5-6de1-71256fa62dee
2024-12-04T01:31:56.772Z [INFO]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: pipelining replication: peer="{Voter server-9c2d41c7-b347-97d5-6de1-71256fa62dee 9c2d41c7-b347-97d5-6de1-71256fa62dee}"
2024-12-04T01:31:56.772Z [INFO]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: pipelining replication: peer="{Voter server-3a729bdb-291e-7129-3759-e3284e4e39d7 3a729bdb-291e-7129-3759-e3284e4e39d7}"
2024-12-04T01:31:56.878Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:31:56.772891073 +0000 UTC m=+7.604733696 (2 nodes), 72.344467ms from start of poll, 72.796474ms from cluster start. Timeout at 2024-12-04 01:31:56.877897747 +0000 UTC m=+7.709740900, 105.007204ms after stability
2024-12-04T01:31:56.878Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:56.987Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:56.878321152 +0000 UTC m=+7.710163795 (1 nodes), 3.046µs from start of poll, 178.226573ms from cluster start. Timeout at 2024-12-04 01:31:56.983355098 +0000 UTC m=+7.815197683, 105.033888ms after stability
    raft_test.go:378: [INFO] Disconnecting Node at 90c9edc4-a781-2a80-3c72-d0587ad9e7fa [Leader]
2024-12-04T01:31:57.003Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to heartbeat to: peer=9c2d41c7-b347-97d5-6de1-71256fa62dee backoff time=10ms error="failed to connect to peer: 9c2d41c7-b347-97d5-6de1-71256fa62dee"
2024-12-04T01:31:57.004Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to pipeline appendEntries: peer="{Voter server-3a729bdb-291e-7129-3759-e3284e4e39d7 3a729bdb-291e-7129-3759-e3284e4e39d7}" error="append pipeline closed"
2024-12-04T01:31:57.004Z [INFO]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: aborting pipeline replication: peer="{Voter server-3a729bdb-291e-7129-3759-e3284e4e39d7 3a729bdb-291e-7129-3759-e3284e4e39d7}"
2024-12-04T01:31:57.004Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to pipeline appendEntries: peer="{Voter server-9c2d41c7-b347-97d5-6de1-71256fa62dee 9c2d41c7-b347-97d5-6de1-71256fa62dee}" error="append pipeline closed"
2024-12-04T01:31:57.004Z [INFO]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: aborting pipeline replication: peer="{Voter server-9c2d41c7-b347-97d5-6de1-71256fa62dee 9c2d41c7-b347-97d5-6de1-71256fa62dee}"
2024-12-04T01:31:57.004Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to heartbeat to: peer=3a729bdb-291e-7129-3759-e3284e4e39d7 backoff time=10ms error="failed to connect to peer: 3a729bdb-291e-7129-3759-e3284e4e39d7"
2024-12-04T01:31:57.004Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:57.012Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to appendEntries to: peer="{Voter server-3a729bdb-291e-7129-3759-e3284e4e39d7 3a729bdb-291e-7129-3759-e3284e4e39d7}" error="failed to connect to peer: 3a729bdb-291e-7129-3759-e3284e4e39d7"
2024-12-04T01:31:57.012Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to appendEntries to: peer="{Voter server-9c2d41c7-b347-97d5-6de1-71256fa62dee 9c2d41c7-b347-97d5-6de1-71256fa62dee}" error="failed to connect to peer: 9c2d41c7-b347-97d5-6de1-71256fa62dee"
2024-12-04T01:31:57.021Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to heartbeat to: peer=9c2d41c7-b347-97d5-6de1-71256fa62dee backoff time=10ms error="failed to connect to peer: 9c2d41c7-b347-97d5-6de1-71256fa62dee"
2024-12-04T01:31:57.021Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to heartbeat to: peer=3a729bdb-291e-7129-3759-e3284e4e39d7 backoff time=10ms error="failed to connect to peer: 3a729bdb-291e-7129-3759-e3284e4e39d7"
2024-12-04T01:31:57.029Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to appendEntries to: peer="{Voter server-3a729bdb-291e-7129-3759-e3284e4e39d7 3a729bdb-291e-7129-3759-e3284e4e39d7}" error="failed to connect to peer: 3a729bdb-291e-7129-3759-e3284e4e39d7"
2024-12-04T01:31:57.032Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to appendEntries to: peer="{Voter server-9c2d41c7-b347-97d5-6de1-71256fa62dee 9c2d41c7-b347-97d5-6de1-71256fa62dee}" error="failed to connect to peer: 9c2d41c7-b347-97d5-6de1-71256fa62dee"
2024-12-04T01:31:57.038Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to heartbeat to: peer=9c2d41c7-b347-97d5-6de1-71256fa62dee backoff time=10ms error="failed to connect to peer: 9c2d41c7-b347-97d5-6de1-71256fa62dee"
2024-12-04T01:31:57.042Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to heartbeat to: peer=3a729bdb-291e-7129-3759-e3284e4e39d7 backoff time=10ms error="failed to connect to peer: 3a729bdb-291e-7129-3759-e3284e4e39d7"
2024-12-04T01:31:57.047Z [WARN]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to contact: server-id=server-3a729bdb-291e-7129-3759-e3284e4e39d7 time=51.780978ms
2024-12-04T01:31:57.047Z [WARN]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to contact: server-id=server-9c2d41c7-b347-97d5-6de1-71256fa62dee time=51.870426ms
2024-12-04T01:31:57.047Z [WARN]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to contact quorum of nodes, stepping down
2024-12-04T01:31:57.047Z [INFO]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: entering follower state: follower="Node at 90c9edc4-a781-2a80-3c72-d0587ad9e7fa [Follower]" leader-address= leader-id=
2024-12-04T01:31:57.049Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to appendEntries to: peer="{Voter server-3a729bdb-291e-7129-3759-e3284e4e39d7 3a729bdb-291e-7129-3759-e3284e4e39d7}" error="failed to connect to peer: 3a729bdb-291e-7129-3759-e3284e4e39d7"
2024-12-04T01:31:57.053Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to appendEntries to: peer="{Voter server-9c2d41c7-b347-97d5-6de1-71256fa62dee 9c2d41c7-b347-97d5-6de1-71256fa62dee}" error="failed to connect to peer: 9c2d41c7-b347-97d5-6de1-71256fa62dee"
2024-12-04T01:31:57.061Z [WARN]  server-9c2d41c7-b347-97d5-6de1-71256fa62dee: heartbeat timeout reached, starting election: last-leader-addr=90c9edc4-a781-2a80-3c72-d0587ad9e7fa last-leader-id=server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa
2024-12-04T01:31:57.061Z [INFO]  server-9c2d41c7-b347-97d5-6de1-71256fa62dee: entering candidate state: node="Node at 9c2d41c7-b347-97d5-6de1-71256fa62dee [Candidate]" term=3
2024-12-04T01:31:57.061Z [ERROR] server-9c2d41c7-b347-97d5-6de1-71256fa62dee: failed to make requestVote RPC: target="{Voter server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa 90c9edc4-a781-2a80-3c72-d0587ad9e7fa}" error="failed to connect to peer: 90c9edc4-a781-2a80-3c72-d0587ad9e7fa" term=3
2024-12-04T01:31:57.062Z [WARN]  server-3a729bdb-291e-7129-3759-e3284e4e39d7: rejecting vote request since we have a leader: from=9c2d41c7-b347-97d5-6de1-71256fa62dee leader=90c9edc4-a781-2a80-3c72-d0587ad9e7fa leader-id=server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa
2024-12-04T01:31:57.070Z [WARN]  server-3a729bdb-291e-7129-3759-e3284e4e39d7: heartbeat timeout reached, starting election: last-leader-addr=90c9edc4-a781-2a80-3c72-d0587ad9e7fa last-leader-id=server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa
2024-12-04T01:31:57.070Z [INFO]  server-3a729bdb-291e-7129-3759-e3284e4e39d7: entering candidate state: node="Node at 3a729bdb-291e-7129-3759-e3284e4e39d7 [Candidate]" term=3
2024-12-04T01:31:57.070Z [ERROR] server-3a729bdb-291e-7129-3759-e3284e4e39d7: failed to make requestVote RPC: target="{Voter server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa 90c9edc4-a781-2a80-3c72-d0587ad9e7fa}" error="failed to connect to peer: 90c9edc4-a781-2a80-3c72-d0587ad9e7fa" term=3
2024-12-04T01:31:57.070Z [INFO]  server-9c2d41c7-b347-97d5-6de1-71256fa62dee: duplicate requestVote for same term: term=3
2024-12-04T01:31:57.122Z [WARN]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:57.122Z [INFO]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: entering candidate state: node="Node at 90c9edc4-a781-2a80-3c72-d0587ad9e7fa [Candidate]" term=3
2024-12-04T01:31:57.122Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to make requestVote RPC: target="{Voter server-9c2d41c7-b347-97d5-6de1-71256fa62dee 9c2d41c7-b347-97d5-6de1-71256fa62dee}" error="failed to connect to peer: 9c2d41c7-b347-97d5-6de1-71256fa62dee" term=3
2024-12-04T01:31:57.122Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to make requestVote RPC: target="{Voter server-3a729bdb-291e-7129-3759-e3284e4e39d7 3a729bdb-291e-7129-3759-e3284e4e39d7}" error="failed to connect to peer: 3a729bdb-291e-7129-3759-e3284e4e39d7" term=3
2024-12-04T01:31:57.134Z [WARN]  server-9c2d41c7-b347-97d5-6de1-71256fa62dee: Election timeout reached, restarting election
2024-12-04T01:31:57.134Z [INFO]  server-9c2d41c7-b347-97d5-6de1-71256fa62dee: entering candidate state: node="Node at 9c2d41c7-b347-97d5-6de1-71256fa62dee [Candidate]" term=4
2024-12-04T01:31:57.134Z [ERROR] server-9c2d41c7-b347-97d5-6de1-71256fa62dee: failed to make requestVote RPC: target="{Voter server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa 90c9edc4-a781-2a80-3c72-d0587ad9e7fa}" error="failed to connect to peer: 90c9edc4-a781-2a80-3c72-d0587ad9e7fa" term=4
2024-12-04T01:31:57.134Z [INFO]  server-3a729bdb-291e-7129-3759-e3284e4e39d7: entering follower state: follower="Node at 3a729bdb-291e-7129-3759-e3284e4e39d7 [Follower]" leader-address= leader-id=
2024-12-04T01:31:57.134Z [INFO]  server-9c2d41c7-b347-97d5-6de1-71256fa62dee: election won: term=4 tally=2
2024-12-04T01:31:57.134Z [INFO]  server-9c2d41c7-b347-97d5-6de1-71256fa62dee: entering leader state: leader="Node at 9c2d41c7-b347-97d5-6de1-71256fa62dee [Leader]"
2024-12-04T01:31:57.134Z [INFO]  server-9c2d41c7-b347-97d5-6de1-71256fa62dee: added peer, starting replication: peer=server-3a729bdb-291e-7129-3759-e3284e4e39d7
2024-12-04T01:31:57.134Z [INFO]  server-9c2d41c7-b347-97d5-6de1-71256fa62dee: added peer, starting replication: peer=server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa
2024-12-04T01:31:57.134Z [ERROR] server-9c2d41c7-b347-97d5-6de1-71256fa62dee: failed to appendEntries to: peer="{Voter server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa 90c9edc4-a781-2a80-3c72-d0587ad9e7fa}" error="failed to connect to peer: 90c9edc4-a781-2a80-3c72-d0587ad9e7fa"
2024-12-04T01:31:57.134Z [INFO]  server-9c2d41c7-b347-97d5-6de1-71256fa62dee: pipelining replication: peer="{Voter server-3a729bdb-291e-7129-3759-e3284e4e39d7 3a729bdb-291e-7129-3759-e3284e4e39d7}"
2024-12-04T01:31:57.142Z [ERROR] server-9c2d41c7-b347-97d5-6de1-71256fa62dee: failed to heartbeat to: peer=90c9edc4-a781-2a80-3c72-d0587ad9e7fa backoff time=10ms error="failed to connect to peer: 90c9edc4-a781-2a80-3c72-d0587ad9e7fa"
2024-12-04T01:31:57.157Z [ERROR] server-9c2d41c7-b347-97d5-6de1-71256fa62dee: failed to appendEntries to: peer="{Voter server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa 90c9edc4-a781-2a80-3c72-d0587ad9e7fa}" error="failed to connect to peer: 90c9edc4-a781-2a80-3c72-d0587ad9e7fa"
2024-12-04T01:31:57.161Z [ERROR] server-9c2d41c7-b347-97d5-6de1-71256fa62dee: failed to heartbeat to: peer=90c9edc4-a781-2a80-3c72-d0587ad9e7fa backoff time=10ms error="failed to connect to peer: 90c9edc4-a781-2a80-3c72-d0587ad9e7fa"
2024-12-04T01:31:57.177Z [ERROR] server-9c2d41c7-b347-97d5-6de1-71256fa62dee: failed to appendEntries to: peer="{Voter server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa 90c9edc4-a781-2a80-3c72-d0587ad9e7fa}" error="failed to connect to peer: 90c9edc4-a781-2a80-3c72-d0587ad9e7fa"
2024-12-04T01:31:57.182Z [ERROR] server-9c2d41c7-b347-97d5-6de1-71256fa62dee: failed to heartbeat to: peer=90c9edc4-a781-2a80-3c72-d0587ad9e7fa backoff time=10ms error="failed to connect to peer: 90c9edc4-a781-2a80-3c72-d0587ad9e7fa"
2024-12-04T01:31:57.185Z [WARN]  server-9c2d41c7-b347-97d5-6de1-71256fa62dee: failed to contact: server-id=server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa time=50.898313ms
2024-12-04T01:31:57.188Z [WARN]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: Election timeout reached, restarting election
2024-12-04T01:31:57.188Z [INFO]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: entering candidate state: node="Node at 90c9edc4-a781-2a80-3c72-d0587ad9e7fa [Candidate]" term=4
2024-12-04T01:31:57.188Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to make requestVote RPC: target="{Voter server-9c2d41c7-b347-97d5-6de1-71256fa62dee 9c2d41c7-b347-97d5-6de1-71256fa62dee}" error="failed to connect to peer: 9c2d41c7-b347-97d5-6de1-71256fa62dee" term=4
2024-12-04T01:31:57.188Z [ERROR] server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: failed to make requestVote RPC: target="{Voter server-3a729bdb-291e-7129-3759-e3284e4e39d7 3a729bdb-291e-7129-3759-e3284e4e39d7}" error="failed to connect to peer: 3a729bdb-291e-7129-3759-e3284e4e39d7" term=4
2024-12-04T01:31:57.206Z [ERROR] server-9c2d41c7-b347-97d5-6de1-71256fa62dee: failed to appendEntries to: peer="{Voter server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa 90c9edc4-a781-2a80-3c72-d0587ad9e7fa}" error="failed to connect to peer: 90c9edc4-a781-2a80-3c72-d0587ad9e7fa"
2024-12-04T01:31:57.206Z [ERROR] server-9c2d41c7-b347-97d5-6de1-71256fa62dee: failed to heartbeat to: peer=90c9edc4-a781-2a80-3c72-d0587ad9e7fa backoff time=20ms error="failed to connect to peer: 90c9edc4-a781-2a80-3c72-d0587ad9e7fa"
2024-12-04T01:31:57.229Z [WARN]  server-9c2d41c7-b347-97d5-6de1-71256fa62dee: failed to contact: server-id=server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa time=94.435638ms
2024-12-04T01:31:57.235Z [ERROR] server-9c2d41c7-b347-97d5-6de1-71256fa62dee: failed to heartbeat to: peer=90c9edc4-a781-2a80-3c72-d0587ad9e7fa backoff time=25ms error="failed to connect to peer: 90c9edc4-a781-2a80-3c72-d0587ad9e7fa"
2024-12-04T01:31:57.240Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:57.134775169 +0000 UTC m=+7.966617752 (1 nodes), 130.628725ms from start of poll, 434.68053ms from cluster start. Timeout at 2024-12-04 01:31:57.239788667 +0000 UTC m=+8.071631312, 105.01356ms after stability
    raft_test.go:413: [INFO] Reconnecting Node at 90c9edc4-a781-2a80-3c72-d0587ad9e7fa [Candidate]
2024-12-04T01:31:57.260Z [INFO]  server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa: entering follower state: follower="Node at 90c9edc4-a781-2a80-3c72-d0587ad9e7fa [Follower]" leader-address=9c2d41c7-b347-97d5-6de1-71256fa62dee leader-id=server-9c2d41c7-b347-97d5-6de1-71256fa62dee
2024-12-04T01:31:57.260Z [INFO]  server-9c2d41c7-b347-97d5-6de1-71256fa62dee: pipelining replication: peer="{Voter server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa 90c9edc4-a781-2a80-3c72-d0587ad9e7fa}"
2024-12-04T01:31:57.260Z [INFO]  server-9c2d41c7-b347-97d5-6de1-71256fa62dee: aborting pipeline replication: peer="{Voter server-90c9edc4-a781-2a80-3c72-d0587ad9e7fa 90c9edc4-a781-2a80-3c72-d0587ad9e7fa}"
2024-12-04T01:31:57.260Z [INFO]  server-9c2d41c7-b347-97d5-6de1-71256fa62dee: aborting pipeline replication: peer="{Voter server-3a729bdb-291e-7129-3759-e3284e4e39d7 3a729bdb-291e-7129-3759-e3284e4e39d7}"
--- PASS: TestRaft_LeaderFail (0.56s)
=== RUN   TestRaft_BehindFollower
2024-12-04T01:31:57.262Z [INFO]  server-692403bd-f458-441e-671e-1f271c120898: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-692403bd-f458-441e-671e-1f271c120898 Address:692403bd-f458-441e-671e-1f271c120898} {Suffrage:Voter ID:server-bbd47d9e-fb29-4d40-6ed7-0006b334af27 Address:bbd47d9e-fb29-4d40-6ed7-0006b334af27} {Suffrage:Voter ID:server-57dc74c6-cadd-260d-e374-63cc2d9cb780 Address:57dc74c6-cadd-260d-e374-63cc2d9cb780}]"
2024-12-04T01:31:57.262Z [INFO]  server-bbd47d9e-fb29-4d40-6ed7-0006b334af27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-692403bd-f458-441e-671e-1f271c120898 Address:692403bd-f458-441e-671e-1f271c120898} {Suffrage:Voter ID:server-bbd47d9e-fb29-4d40-6ed7-0006b334af27 Address:bbd47d9e-fb29-4d40-6ed7-0006b334af27} {Suffrage:Voter ID:server-57dc74c6-cadd-260d-e374-63cc2d9cb780 Address:57dc74c6-cadd-260d-e374-63cc2d9cb780}]"
2024-12-04T01:31:57.262Z [INFO]  server-57dc74c6-cadd-260d-e374-63cc2d9cb780: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-692403bd-f458-441e-671e-1f271c120898 Address:692403bd-f458-441e-671e-1f271c120898} {Suffrage:Voter ID:server-bbd47d9e-fb29-4d40-6ed7-0006b334af27 Address:bbd47d9e-fb29-4d40-6ed7-0006b334af27} {Suffrage:Voter ID:server-57dc74c6-cadd-260d-e374-63cc2d9cb780 Address:57dc74c6-cadd-260d-e374-63cc2d9cb780}]"
2024-12-04T01:31:57.262Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:57.262Z [INFO]  server-692403bd-f458-441e-671e-1f271c120898: entering follower state: follower="Node at 692403bd-f458-441e-671e-1f271c120898 [Follower]" leader-address= leader-id=
2024-12-04T01:31:57.262Z [INFO]  server-bbd47d9e-fb29-4d40-6ed7-0006b334af27: entering follower state: follower="Node at bbd47d9e-fb29-4d40-6ed7-0006b334af27 [Follower]" leader-address= leader-id=
2024-12-04T01:31:57.262Z [INFO]  server-57dc74c6-cadd-260d-e374-63cc2d9cb780: entering follower state: follower="Node at 57dc74c6-cadd-260d-e374-63cc2d9cb780 [Follower]" leader-address= leader-id=
2024-12-04T01:31:57.314Z [WARN]  server-bbd47d9e-fb29-4d40-6ed7-0006b334af27: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:57.314Z [INFO]  server-bbd47d9e-fb29-4d40-6ed7-0006b334af27: entering candidate state: node="Node at bbd47d9e-fb29-4d40-6ed7-0006b334af27 [Candidate]" term=2
2024-12-04T01:31:57.314Z [INFO]  server-bbd47d9e-fb29-4d40-6ed7-0006b334af27: election won: term=2 tally=2
2024-12-04T01:31:57.314Z [INFO]  server-bbd47d9e-fb29-4d40-6ed7-0006b334af27: entering leader state: leader="Node at bbd47d9e-fb29-4d40-6ed7-0006b334af27 [Leader]"
2024-12-04T01:31:57.314Z [INFO]  server-bbd47d9e-fb29-4d40-6ed7-0006b334af27: added peer, starting replication: peer=server-692403bd-f458-441e-671e-1f271c120898
2024-12-04T01:31:57.314Z [INFO]  server-bbd47d9e-fb29-4d40-6ed7-0006b334af27: added peer, starting replication: peer=server-57dc74c6-cadd-260d-e374-63cc2d9cb780
2024-12-04T01:31:57.314Z [INFO]  server-bbd47d9e-fb29-4d40-6ed7-0006b334af27: pipelining replication: peer="{Voter server-57dc74c6-cadd-260d-e374-63cc2d9cb780 57dc74c6-cadd-260d-e374-63cc2d9cb780}"
2024-12-04T01:31:57.315Z [INFO]  server-bbd47d9e-fb29-4d40-6ed7-0006b334af27: pipelining replication: peer="{Voter server-692403bd-f458-441e-671e-1f271c120898 692403bd-f458-441e-671e-1f271c120898}"
2024-12-04T01:31:57.423Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:57.315088763 +0000 UTC m=+8.146931505 (1 nodes), 52.495994ms from start of poll, 52.892231ms from cluster start. Timeout at 2024-12-04 01:31:57.420099264 +0000 UTC m=+8.251941883, 105.010378ms after stability
2024-12-04T01:31:57.423Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:31:57.529Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:31:57.423176209 +0000 UTC m=+8.255018784 (2 nodes), 3.565µs from start of poll, 160.97951ms from cluster start. Timeout at 2024-12-04 01:31:57.528193643 +0000 UTC m=+8.360036475, 105.017691ms after stability
2024-12-04T01:31:57.529Z [ERROR] server-bbd47d9e-fb29-4d40-6ed7-0006b334af27: failed to pipeline appendEntries: peer="{Voter server-692403bd-f458-441e-671e-1f271c120898 692403bd-f458-441e-671e-1f271c120898}" error="append pipeline closed"
2024-12-04T01:31:57.529Z [INFO]  server-bbd47d9e-fb29-4d40-6ed7-0006b334af27: aborting pipeline replication: peer="{Voter server-692403bd-f458-441e-671e-1f271c120898 692403bd-f458-441e-671e-1f271c120898}"
2024-12-04T01:31:57.530Z [ERROR] server-bbd47d9e-fb29-4d40-6ed7-0006b334af27: failed to appendEntries to: peer="{Voter server-692403bd-f458-441e-671e-1f271c120898 692403bd-f458-441e-671e-1f271c120898}" error="failed to connect to peer: 692403bd-f458-441e-671e-1f271c120898"
2024-12-04T01:31:57.530Z [ERROR] server-bbd47d9e-fb29-4d40-6ed7-0006b334af27: failed to heartbeat to: peer=692403bd-f458-441e-671e-1f271c120898 backoff time=10ms error="failed to connect to peer: 692403bd-f458-441e-671e-1f271c120898"
    raft_test.go:459: [INFO] Finished apply without behind follower
2024-12-04T01:31:57.550Z [INFO]  server-bbd47d9e-fb29-4d40-6ed7-0006b334af27: pipelining replication: peer="{Voter server-692403bd-f458-441e-671e-1f271c120898 692403bd-f458-441e-671e-1f271c120898}"
2024-12-04T01:31:57.570Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:57.675Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:57.570379494 +0000 UTC m=+8.402222111 (1 nodes), 2.263µs from start of poll, 308.182837ms from cluster start. Timeout at 2024-12-04 01:31:57.675398183 +0000 UTC m=+8.507240807, 105.018696ms after stability
2024-12-04T01:31:57.675Z [INFO]  server-bbd47d9e-fb29-4d40-6ed7-0006b334af27: aborting pipeline replication: peer="{Voter server-57dc74c6-cadd-260d-e374-63cc2d9cb780 57dc74c6-cadd-260d-e374-63cc2d9cb780}"
2024-12-04T01:31:57.675Z [INFO]  server-bbd47d9e-fb29-4d40-6ed7-0006b334af27: aborting pipeline replication: peer="{Voter server-692403bd-f458-441e-671e-1f271c120898 692403bd-f458-441e-671e-1f271c120898}"
--- PASS: TestRaft_BehindFollower (0.41s)
=== RUN   TestRaft_ApplyNonLeader
2024-12-04T01:31:57.677Z [INFO]  server-df939bd4-9f74-f9fc-7247-51dd77158b4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df939bd4-9f74-f9fc-7247-51dd77158b4d Address:df939bd4-9f74-f9fc-7247-51dd77158b4d} {Suffrage:Voter ID:server-469ecd1e-3abd-6343-37d3-7bc711bfc3d8 Address:469ecd1e-3abd-6343-37d3-7bc711bfc3d8} {Suffrage:Voter ID:server-79e782e9-efac-88d9-d1b9-a5b1425ae81d Address:79e782e9-efac-88d9-d1b9-a5b1425ae81d}]"
2024-12-04T01:31:57.677Z [INFO]  server-469ecd1e-3abd-6343-37d3-7bc711bfc3d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df939bd4-9f74-f9fc-7247-51dd77158b4d Address:df939bd4-9f74-f9fc-7247-51dd77158b4d} {Suffrage:Voter ID:server-469ecd1e-3abd-6343-37d3-7bc711bfc3d8 Address:469ecd1e-3abd-6343-37d3-7bc711bfc3d8} {Suffrage:Voter ID:server-79e782e9-efac-88d9-d1b9-a5b1425ae81d Address:79e782e9-efac-88d9-d1b9-a5b1425ae81d}]"
2024-12-04T01:31:57.677Z [INFO]  server-79e782e9-efac-88d9-d1b9-a5b1425ae81d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df939bd4-9f74-f9fc-7247-51dd77158b4d Address:df939bd4-9f74-f9fc-7247-51dd77158b4d} {Suffrage:Voter ID:server-469ecd1e-3abd-6343-37d3-7bc711bfc3d8 Address:469ecd1e-3abd-6343-37d3-7bc711bfc3d8} {Suffrage:Voter ID:server-79e782e9-efac-88d9-d1b9-a5b1425ae81d Address:79e782e9-efac-88d9-d1b9-a5b1425ae81d}]"
2024-12-04T01:31:57.677Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:57.677Z [INFO]  server-df939bd4-9f74-f9fc-7247-51dd77158b4d: entering follower state: follower="Node at df939bd4-9f74-f9fc-7247-51dd77158b4d [Follower]" leader-address= leader-id=
2024-12-04T01:31:57.677Z [INFO]  server-469ecd1e-3abd-6343-37d3-7bc711bfc3d8: entering follower state: follower="Node at 469ecd1e-3abd-6343-37d3-7bc711bfc3d8 [Follower]" leader-address= leader-id=
2024-12-04T01:31:57.677Z [INFO]  server-79e782e9-efac-88d9-d1b9-a5b1425ae81d: entering follower state: follower="Node at 79e782e9-efac-88d9-d1b9-a5b1425ae81d [Follower]" leader-address= leader-id=
2024-12-04T01:31:57.734Z [WARN]  server-469ecd1e-3abd-6343-37d3-7bc711bfc3d8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:57.734Z [INFO]  server-469ecd1e-3abd-6343-37d3-7bc711bfc3d8: entering candidate state: node="Node at 469ecd1e-3abd-6343-37d3-7bc711bfc3d8 [Candidate]" term=2
2024-12-04T01:31:57.734Z [INFO]  server-469ecd1e-3abd-6343-37d3-7bc711bfc3d8: election won: term=2 tally=2
2024-12-04T01:31:57.734Z [INFO]  server-469ecd1e-3abd-6343-37d3-7bc711bfc3d8: entering leader state: leader="Node at 469ecd1e-3abd-6343-37d3-7bc711bfc3d8 [Leader]"
2024-12-04T01:31:57.734Z [INFO]  server-469ecd1e-3abd-6343-37d3-7bc711bfc3d8: added peer, starting replication: peer=server-df939bd4-9f74-f9fc-7247-51dd77158b4d
2024-12-04T01:31:57.734Z [INFO]  server-469ecd1e-3abd-6343-37d3-7bc711bfc3d8: added peer, starting replication: peer=server-79e782e9-efac-88d9-d1b9-a5b1425ae81d
2024-12-04T01:31:57.735Z [INFO]  server-469ecd1e-3abd-6343-37d3-7bc711bfc3d8: pipelining replication: peer="{Voter server-79e782e9-efac-88d9-d1b9-a5b1425ae81d 79e782e9-efac-88d9-d1b9-a5b1425ae81d}"
2024-12-04T01:31:57.735Z [INFO]  server-469ecd1e-3abd-6343-37d3-7bc711bfc3d8: pipelining replication: peer="{Voter server-df939bd4-9f74-f9fc-7247-51dd77158b4d df939bd4-9f74-f9fc-7247-51dd77158b4d}"
2024-12-04T01:31:57.842Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:57.735317291 +0000 UTC m=+8.567159852 (1 nodes), 57.729558ms from start of poll, 58.142509ms from cluster start. Timeout at 2024-12-04 01:31:57.840324959 +0000 UTC m=+8.672167812, 105.00796ms after stability
2024-12-04T01:31:57.842Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:31:57.949Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:31:57.842527197 +0000 UTC m=+8.674369766 (2 nodes), 3.724µs from start of poll, 165.352423ms from cluster start. Timeout at 2024-12-04 01:31:57.947583215 +0000 UTC m=+8.779425779, 105.056013ms after stability
2024-12-04T01:31:57.949Z [INFO]  server-469ecd1e-3abd-6343-37d3-7bc711bfc3d8: aborting pipeline replication: peer="{Voter server-79e782e9-efac-88d9-d1b9-a5b1425ae81d 79e782e9-efac-88d9-d1b9-a5b1425ae81d}"
2024-12-04T01:31:57.949Z [INFO]  server-469ecd1e-3abd-6343-37d3-7bc711bfc3d8: aborting pipeline replication: peer="{Voter server-df939bd4-9f74-f9fc-7247-51dd77158b4d df939bd4-9f74-f9fc-7247-51dd77158b4d}"
--- PASS: TestRaft_ApplyNonLeader (0.27s)
=== RUN   TestRaft_ApplyConcurrent
2024-12-04T01:31:57.950Z [INFO]  server-492736ec-e2a8-d8aa-f906-62849b1bce4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-492736ec-e2a8-d8aa-f906-62849b1bce4d Address:492736ec-e2a8-d8aa-f906-62849b1bce4d} {Suffrage:Voter ID:server-d111116f-8564-5797-d010-00a76e724804 Address:d111116f-8564-5797-d010-00a76e724804} {Suffrage:Voter ID:server-3a272c0f-7bc9-cf57-d693-5c7e03da9aab Address:3a272c0f-7bc9-cf57-d693-5c7e03da9aab}]"
2024-12-04T01:31:57.951Z [INFO]  server-d111116f-8564-5797-d010-00a76e724804: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-492736ec-e2a8-d8aa-f906-62849b1bce4d Address:492736ec-e2a8-d8aa-f906-62849b1bce4d} {Suffrage:Voter ID:server-d111116f-8564-5797-d010-00a76e724804 Address:d111116f-8564-5797-d010-00a76e724804} {Suffrage:Voter ID:server-3a272c0f-7bc9-cf57-d693-5c7e03da9aab Address:3a272c0f-7bc9-cf57-d693-5c7e03da9aab}]"
2024-12-04T01:31:57.951Z [INFO]  server-3a272c0f-7bc9-cf57-d693-5c7e03da9aab: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-492736ec-e2a8-d8aa-f906-62849b1bce4d Address:492736ec-e2a8-d8aa-f906-62849b1bce4d} {Suffrage:Voter ID:server-d111116f-8564-5797-d010-00a76e724804 Address:d111116f-8564-5797-d010-00a76e724804} {Suffrage:Voter ID:server-3a272c0f-7bc9-cf57-d693-5c7e03da9aab Address:3a272c0f-7bc9-cf57-d693-5c7e03da9aab}]"
2024-12-04T01:31:57.951Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:57.951Z [INFO]  server-492736ec-e2a8-d8aa-f906-62849b1bce4d: entering follower state: follower="Node at 492736ec-e2a8-d8aa-f906-62849b1bce4d [Follower]" leader-address= leader-id=
2024-12-04T01:31:57.951Z [INFO]  server-d111116f-8564-5797-d010-00a76e724804: entering follower state: follower="Node at d111116f-8564-5797-d010-00a76e724804 [Follower]" leader-address= leader-id=
2024-12-04T01:31:57.951Z [INFO]  server-3a272c0f-7bc9-cf57-d693-5c7e03da9aab: entering follower state: follower="Node at 3a272c0f-7bc9-cf57-d693-5c7e03da9aab [Follower]" leader-address= leader-id=
2024-12-04T01:31:58.058Z [WARN]  server-d111116f-8564-5797-d010-00a76e724804: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:58.058Z [INFO]  server-d111116f-8564-5797-d010-00a76e724804: entering candidate state: node="Node at d111116f-8564-5797-d010-00a76e724804 [Candidate]" term=2
2024-12-04T01:31:58.059Z [INFO]  server-d111116f-8564-5797-d010-00a76e724804: election won: term=2 tally=2
2024-12-04T01:31:58.059Z [INFO]  server-d111116f-8564-5797-d010-00a76e724804: entering leader state: leader="Node at d111116f-8564-5797-d010-00a76e724804 [Leader]"
2024-12-04T01:31:58.059Z [INFO]  server-d111116f-8564-5797-d010-00a76e724804: added peer, starting replication: peer=server-492736ec-e2a8-d8aa-f906-62849b1bce4d
2024-12-04T01:31:58.059Z [INFO]  server-d111116f-8564-5797-d010-00a76e724804: added peer, starting replication: peer=server-3a272c0f-7bc9-cf57-d693-5c7e03da9aab
2024-12-04T01:31:58.059Z [INFO]  server-d111116f-8564-5797-d010-00a76e724804: pipelining replication: peer="{Voter server-3a272c0f-7bc9-cf57-d693-5c7e03da9aab 3a272c0f-7bc9-cf57-d693-5c7e03da9aab}"
2024-12-04T01:31:58.059Z [INFO]  server-d111116f-8564-5797-d010-00a76e724804: pipelining replication: peer="{Voter server-492736ec-e2a8-d8aa-f906-62849b1bce4d 492736ec-e2a8-d8aa-f906-62849b1bce4d}"
2024-12-04T01:31:58.266Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:58.059760948 +0000 UTC m=+8.891603579 (1 nodes), 108.468786ms from start of poll, 108.958925ms from cluster start. Timeout at 2024-12-04 01:31:58.264771723 +0000 UTC m=+9.096614621, 205.011042ms after stability
2024-12-04T01:31:58.294Z [INFO]  server-d111116f-8564-5797-d010-00a76e724804: aborting pipeline replication: peer="{Voter server-3a272c0f-7bc9-cf57-d693-5c7e03da9aab 3a272c0f-7bc9-cf57-d693-5c7e03da9aab}"
2024-12-04T01:31:58.294Z [INFO]  server-d111116f-8564-5797-d010-00a76e724804: aborting pipeline replication: peer="{Voter server-492736ec-e2a8-d8aa-f906-62849b1bce4d 492736ec-e2a8-d8aa-f906-62849b1bce4d}"
--- PASS: TestRaft_ApplyConcurrent (0.35s)
=== RUN   TestRaft_ApplyConcurrent_Timeout
2024-12-04T01:31:58.296Z [INFO]  server-372b2f61-8f48-7f11-cb89-f6d8dc225e3d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-372b2f61-8f48-7f11-cb89-f6d8dc225e3d Address:372b2f61-8f48-7f11-cb89-f6d8dc225e3d}]"
2024-12-04T01:31:58.296Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:58.296Z [INFO]  server-372b2f61-8f48-7f11-cb89-f6d8dc225e3d: entering follower state: follower="Node at 372b2f61-8f48-7f11-cb89-f6d8dc225e3d [Follower]" leader-address= leader-id=
2024-12-04T01:31:58.456Z [WARN]  server-372b2f61-8f48-7f11-cb89-f6d8dc225e3d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:58.456Z [INFO]  server-372b2f61-8f48-7f11-cb89-f6d8dc225e3d: entering candidate state: node="Node at 372b2f61-8f48-7f11-cb89-f6d8dc225e3d [Candidate]" term=2
2024-12-04T01:31:58.456Z [INFO]  server-372b2f61-8f48-7f11-cb89-f6d8dc225e3d: election won: term=2 tally=1
2024-12-04T01:31:58.456Z [INFO]  server-372b2f61-8f48-7f11-cb89-f6d8dc225e3d: entering leader state: leader="Node at 372b2f61-8f48-7f11-cb89-f6d8dc225e3d [Leader]"
2024-12-04T01:31:58.658Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:58.456779151 +0000 UTC m=+9.288621809 (1 nodes), 160.531095ms from start of poll, 160.736812ms from cluster start. Timeout at 2024-12-04 01:31:58.657793639 +0000 UTC m=+9.489636709, 201.0149ms after stability
--- PASS: TestRaft_ApplyConcurrent_Timeout (0.37s)
=== RUN   TestRaft_JoinNode
2024-12-04T01:31:58.663Z [INFO]  server-8309a3f5-527b-d9a4-47fe-186c2b4a3e4c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8309a3f5-527b-d9a4-47fe-186c2b4a3e4c Address:8309a3f5-527b-d9a4-47fe-186c2b4a3e4c} {Suffrage:Voter ID:server-f51604be-3a32-9405-2ea4-04d4a284dd67 Address:f51604be-3a32-9405-2ea4-04d4a284dd67}]"
2024-12-04T01:31:58.663Z [INFO]  server-f51604be-3a32-9405-2ea4-04d4a284dd67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8309a3f5-527b-d9a4-47fe-186c2b4a3e4c Address:8309a3f5-527b-d9a4-47fe-186c2b4a3e4c} {Suffrage:Voter ID:server-f51604be-3a32-9405-2ea4-04d4a284dd67 Address:f51604be-3a32-9405-2ea4-04d4a284dd67}]"
2024-12-04T01:31:58.664Z [INFO]  server-f4028940-7c0f-6b72-039a-927bba5f0861: initial configuration: index=0 servers=[]
2024-12-04T01:31:58.663Z [INFO]  server-f51604be-3a32-9405-2ea4-04d4a284dd67: entering follower state: follower="Node at f51604be-3a32-9405-2ea4-04d4a284dd67 [Follower]" leader-address= leader-id=
2024-12-04T01:31:58.664Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:58.664Z [INFO]  server-8309a3f5-527b-d9a4-47fe-186c2b4a3e4c: entering follower state: follower="Node at 8309a3f5-527b-d9a4-47fe-186c2b4a3e4c [Follower]" leader-address= leader-id=
2024-12-04T01:31:58.664Z [INFO]  server-f4028940-7c0f-6b72-039a-927bba5f0861: entering follower state: follower="Node at f4028940-7c0f-6b72-039a-927bba5f0861 [Follower]" leader-address= leader-id=
2024-12-04T01:31:58.730Z [WARN]  server-f4028940-7c0f-6b72-039a-927bba5f0861: no known peers, aborting election
2024-12-04T01:31:58.734Z [WARN]  server-f51604be-3a32-9405-2ea4-04d4a284dd67: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:58.734Z [INFO]  server-f51604be-3a32-9405-2ea4-04d4a284dd67: entering candidate state: node="Node at f51604be-3a32-9405-2ea4-04d4a284dd67 [Candidate]" term=2
2024-12-04T01:31:58.734Z [INFO]  server-f51604be-3a32-9405-2ea4-04d4a284dd67: election won: term=2 tally=2
2024-12-04T01:31:58.734Z [INFO]  server-f51604be-3a32-9405-2ea4-04d4a284dd67: entering leader state: leader="Node at f51604be-3a32-9405-2ea4-04d4a284dd67 [Leader]"
2024-12-04T01:31:58.734Z [INFO]  server-f51604be-3a32-9405-2ea4-04d4a284dd67: added peer, starting replication: peer=server-8309a3f5-527b-d9a4-47fe-186c2b4a3e4c
2024-12-04T01:31:58.734Z [INFO]  server-f51604be-3a32-9405-2ea4-04d4a284dd67: pipelining replication: peer="{Voter server-8309a3f5-527b-d9a4-47fe-186c2b4a3e4c 8309a3f5-527b-d9a4-47fe-186c2b4a3e4c}"
2024-12-04T01:31:58.840Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:58.735014498 +0000 UTC m=+9.566857599 (1 nodes), 70.945994ms from start of poll, 71.603992ms from cluster start. Timeout at 2024-12-04 01:31:58.840021462 +0000 UTC m=+9.671864310, 105.006711ms after stability
2024-12-04T01:31:58.840Z [INFO]  server-f51604be-3a32-9405-2ea4-04d4a284dd67: updating configuration: command=AddVoter server-id=server-f4028940-7c0f-6b72-039a-927bba5f0861 server-addr=f4028940-7c0f-6b72-039a-927bba5f0861 servers="[{Suffrage:Voter ID:server-8309a3f5-527b-d9a4-47fe-186c2b4a3e4c Address:8309a3f5-527b-d9a4-47fe-186c2b4a3e4c} {Suffrage:Voter ID:server-f51604be-3a32-9405-2ea4-04d4a284dd67 Address:f51604be-3a32-9405-2ea4-04d4a284dd67} {Suffrage:Voter ID:server-f4028940-7c0f-6b72-039a-927bba5f0861 Address:f4028940-7c0f-6b72-039a-927bba5f0861}]"
2024-12-04T01:31:58.840Z [INFO]  server-f51604be-3a32-9405-2ea4-04d4a284dd67: added peer, starting replication: peer=server-f4028940-7c0f-6b72-039a-927bba5f0861
2024-12-04T01:31:58.840Z [WARN]  server-f4028940-7c0f-6b72-039a-927bba5f0861: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2024-12-04T01:31:58.840Z [WARN]  server-f51604be-3a32-9405-2ea4-04d4a284dd67: appendEntries rejected, sending older logs: peer="{Voter server-f4028940-7c0f-6b72-039a-927bba5f0861 f4028940-7c0f-6b72-039a-927bba5f0861}" next=1
2024-12-04T01:31:58.841Z [INFO]  server-f51604be-3a32-9405-2ea4-04d4a284dd67: pipelining replication: peer="{Voter server-f4028940-7c0f-6b72-039a-927bba5f0861 f4028940-7c0f-6b72-039a-927bba5f0861}"
2024-12-04T01:31:58.841Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:58.946Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:58.841340351 +0000 UTC m=+9.673183047 (1 nodes), 2.157µs from start of poll, 177.92944ms from cluster start. Timeout at 2024-12-04 01:31:58.946350443 +0000 UTC m=+9.778193195, 105.010148ms after stability
2024-12-04T01:31:58.946Z [INFO]  server-f51604be-3a32-9405-2ea4-04d4a284dd67: aborting pipeline replication: peer="{Voter server-f4028940-7c0f-6b72-039a-927bba5f0861 f4028940-7c0f-6b72-039a-927bba5f0861}"
2024-12-04T01:31:58.946Z [INFO]  server-f51604be-3a32-9405-2ea4-04d4a284dd67: aborting pipeline replication: peer="{Voter server-8309a3f5-527b-d9a4-47fe-186c2b4a3e4c 8309a3f5-527b-d9a4-47fe-186c2b4a3e4c}"
--- PASS: TestRaft_JoinNode (0.29s)
=== RUN   TestRaft_JoinNode_ConfigStore
2024-12-04T01:31:58.948Z [INFO]  server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e Address:f4ee9b6d-7961-a588-14f9-9b6d5b1c683e}]"
2024-12-04T01:31:58.948Z [INFO]  server-89af06de-1599-0b6d-0927-26ce85d7fa61: initial configuration: index=0 servers=[]
2024-12-04T01:31:58.949Z [INFO]  server-ea10414d-5605-22ae-3429-2c99e2903951: initial configuration: index=0 servers=[]
2024-12-04T01:31:58.949Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:58.949Z [INFO]  server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e: entering follower state: follower="Node at f4ee9b6d-7961-a588-14f9-9b6d5b1c683e [Follower]" leader-address= leader-id=
2024-12-04T01:31:58.949Z [INFO]  server-89af06de-1599-0b6d-0927-26ce85d7fa61: entering follower state: follower="Node at 89af06de-1599-0b6d-0927-26ce85d7fa61 [Follower]" leader-address= leader-id=
2024-12-04T01:31:58.949Z [INFO]  server-ea10414d-5605-22ae-3429-2c99e2903951: entering follower state: follower="Node at ea10414d-5605-22ae-3429-2c99e2903951 [Follower]" leader-address= leader-id=
2024-12-04T01:31:59.006Z [WARN]  server-89af06de-1599-0b6d-0927-26ce85d7fa61: no known peers, aborting election
2024-12-04T01:31:59.010Z [WARN]  server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:59.010Z [INFO]  server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e: entering candidate state: node="Node at f4ee9b6d-7961-a588-14f9-9b6d5b1c683e [Candidate]" term=2
2024-12-04T01:31:59.010Z [INFO]  server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e: election won: term=2 tally=1
2024-12-04T01:31:59.010Z [INFO]  server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e: entering leader state: leader="Node at f4ee9b6d-7961-a588-14f9-9b6d5b1c683e [Leader]"
2024-12-04T01:31:59.043Z [WARN]  server-ea10414d-5605-22ae-3429-2c99e2903951: no known peers, aborting election
2024-12-04T01:31:59.118Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:59.01051352 +0000 UTC m=+9.842356399 (1 nodes), 61.352153ms from start of poll, 62.2182ms from cluster start. Timeout at 2024-12-04 01:31:59.115524149 +0000 UTC m=+9.947366948, 105.010549ms after stability
2024-12-04T01:31:59.118Z [INFO]  server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e: updating configuration: command=AddVoter server-id=server-89af06de-1599-0b6d-0927-26ce85d7fa61 server-addr=89af06de-1599-0b6d-0927-26ce85d7fa61 servers="[{Suffrage:Voter ID:server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e Address:f4ee9b6d-7961-a588-14f9-9b6d5b1c683e} {Suffrage:Voter ID:server-89af06de-1599-0b6d-0927-26ce85d7fa61 Address:89af06de-1599-0b6d-0927-26ce85d7fa61}]"
2024-12-04T01:31:59.118Z [INFO]  server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e: added peer, starting replication: peer=server-89af06de-1599-0b6d-0927-26ce85d7fa61
2024-12-04T01:31:59.118Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:59.118Z [WARN]  server-89af06de-1599-0b6d-0927-26ce85d7fa61: failed to get previous log: previous-index=3 last-index=0 error="log not found"
2024-12-04T01:31:59.118Z [WARN]  server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e: appendEntries rejected, sending older logs: peer="{Voter server-89af06de-1599-0b6d-0927-26ce85d7fa61 89af06de-1599-0b6d-0927-26ce85d7fa61}" next=1
2024-12-04T01:31:59.118Z [INFO]  server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e: pipelining replication: peer="{Voter server-89af06de-1599-0b6d-0927-26ce85d7fa61 89af06de-1599-0b6d-0927-26ce85d7fa61}"
2024-12-04T01:31:59.226Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:59.118664909 +0000 UTC m=+9.950507469 (1 nodes), 2.18µs from start of poll, 170.36927ms from cluster start. Timeout at 2024-12-04 01:31:59.22368902 +0000 UTC m=+10.055532045, 105.024576ms after stability
2024-12-04T01:31:59.226Z [INFO]  server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e: updating configuration: command=AddVoter server-id=server-ea10414d-5605-22ae-3429-2c99e2903951 server-addr=ea10414d-5605-22ae-3429-2c99e2903951 servers="[{Suffrage:Voter ID:server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e Address:f4ee9b6d-7961-a588-14f9-9b6d5b1c683e} {Suffrage:Voter ID:server-89af06de-1599-0b6d-0927-26ce85d7fa61 Address:89af06de-1599-0b6d-0927-26ce85d7fa61} {Suffrage:Voter ID:server-ea10414d-5605-22ae-3429-2c99e2903951 Address:ea10414d-5605-22ae-3429-2c99e2903951}]"
2024-12-04T01:31:59.226Z [INFO]  server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e: added peer, starting replication: peer=server-ea10414d-5605-22ae-3429-2c99e2903951
2024-12-04T01:31:59.226Z [WARN]  server-ea10414d-5605-22ae-3429-2c99e2903951: failed to get previous log: previous-index=4 last-index=0 error="log not found"
2024-12-04T01:31:59.226Z [WARN]  server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e: appendEntries rejected, sending older logs: peer="{Voter server-ea10414d-5605-22ae-3429-2c99e2903951 ea10414d-5605-22ae-3429-2c99e2903951}" next=1
2024-12-04T01:31:59.226Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:59.227Z [INFO]  server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e: pipelining replication: peer="{Voter server-ea10414d-5605-22ae-3429-2c99e2903951 ea10414d-5605-22ae-3429-2c99e2903951}"
2024-12-04T01:31:59.334Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:59.227036751 +0000 UTC m=+10.058879512 (1 nodes), 3.881µs from start of poll, 278.741313ms from cluster start. Timeout at 2024-12-04 01:31:59.332058388 +0000 UTC m=+10.163900986, 105.021474ms after stability
2024-12-04T01:31:59.334Z [INFO]  server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e: aborting pipeline replication: peer="{Voter server-ea10414d-5605-22ae-3429-2c99e2903951 ea10414d-5605-22ae-3429-2c99e2903951}"
2024-12-04T01:31:59.334Z [INFO]  server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e: aborting pipeline replication: peer="{Voter server-89af06de-1599-0b6d-0927-26ce85d7fa61 89af06de-1599-0b6d-0927-26ce85d7fa61}"
2024-12-04T01:31:59.838Z [ERROR] server-f4ee9b6d-7961-a588-14f9-9b6d5b1c683e: failed to heartbeat to: peer=89af06de-1599-0b6d-0927-26ce85d7fa61 backoff time=10ms error="command timed out"
--- PASS: TestRaft_JoinNode_ConfigStore (0.89s)
=== RUN   TestRaft_RemoveFollower
2024-12-04T01:31:59.840Z [INFO]  server-9f602161-15a1-f7de-7cd1-779d9a70e945: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f602161-15a1-f7de-7cd1-779d9a70e945 Address:9f602161-15a1-f7de-7cd1-779d9a70e945} {Suffrage:Voter ID:server-0fee33a8-f40c-dcac-9f44-568182f39d21 Address:0fee33a8-f40c-dcac-9f44-568182f39d21} {Suffrage:Voter ID:server-ccaec0e5-ccd7-6852-8b42-366327004344 Address:ccaec0e5-ccd7-6852-8b42-366327004344}]"
2024-12-04T01:31:59.840Z [INFO]  server-0fee33a8-f40c-dcac-9f44-568182f39d21: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f602161-15a1-f7de-7cd1-779d9a70e945 Address:9f602161-15a1-f7de-7cd1-779d9a70e945} {Suffrage:Voter ID:server-0fee33a8-f40c-dcac-9f44-568182f39d21 Address:0fee33a8-f40c-dcac-9f44-568182f39d21} {Suffrage:Voter ID:server-ccaec0e5-ccd7-6852-8b42-366327004344 Address:ccaec0e5-ccd7-6852-8b42-366327004344}]"
2024-12-04T01:31:59.840Z [INFO]  server-ccaec0e5-ccd7-6852-8b42-366327004344: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9f602161-15a1-f7de-7cd1-779d9a70e945 Address:9f602161-15a1-f7de-7cd1-779d9a70e945} {Suffrage:Voter ID:server-0fee33a8-f40c-dcac-9f44-568182f39d21 Address:0fee33a8-f40c-dcac-9f44-568182f39d21} {Suffrage:Voter ID:server-ccaec0e5-ccd7-6852-8b42-366327004344 Address:ccaec0e5-ccd7-6852-8b42-366327004344}]"
2024-12-04T01:31:59.840Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:31:59.841Z [INFO]  server-9f602161-15a1-f7de-7cd1-779d9a70e945: entering follower state: follower="Node at 9f602161-15a1-f7de-7cd1-779d9a70e945 [Follower]" leader-address= leader-id=
2024-12-04T01:31:59.841Z [INFO]  server-0fee33a8-f40c-dcac-9f44-568182f39d21: entering follower state: follower="Node at 0fee33a8-f40c-dcac-9f44-568182f39d21 [Follower]" leader-address= leader-id=
2024-12-04T01:31:59.841Z [INFO]  server-ccaec0e5-ccd7-6852-8b42-366327004344: entering follower state: follower="Node at ccaec0e5-ccd7-6852-8b42-366327004344 [Follower]" leader-address= leader-id=
2024-12-04T01:31:59.904Z [WARN]  server-9f602161-15a1-f7de-7cd1-779d9a70e945: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:31:59.904Z [INFO]  server-9f602161-15a1-f7de-7cd1-779d9a70e945: entering candidate state: node="Node at 9f602161-15a1-f7de-7cd1-779d9a70e945 [Candidate]" term=2
2024-12-04T01:31:59.904Z [INFO]  server-9f602161-15a1-f7de-7cd1-779d9a70e945: election won: term=2 tally=2
2024-12-04T01:31:59.905Z [INFO]  server-9f602161-15a1-f7de-7cd1-779d9a70e945: entering leader state: leader="Node at 9f602161-15a1-f7de-7cd1-779d9a70e945 [Leader]"
2024-12-04T01:31:59.905Z [INFO]  server-9f602161-15a1-f7de-7cd1-779d9a70e945: added peer, starting replication: peer=server-0fee33a8-f40c-dcac-9f44-568182f39d21
2024-12-04T01:31:59.905Z [INFO]  server-9f602161-15a1-f7de-7cd1-779d9a70e945: added peer, starting replication: peer=server-ccaec0e5-ccd7-6852-8b42-366327004344
2024-12-04T01:31:59.905Z [INFO]  server-9f602161-15a1-f7de-7cd1-779d9a70e945: pipelining replication: peer="{Voter server-ccaec0e5-ccd7-6852-8b42-366327004344 ccaec0e5-ccd7-6852-8b42-366327004344}"
2024-12-04T01:31:59.905Z [INFO]  server-9f602161-15a1-f7de-7cd1-779d9a70e945: pipelining replication: peer="{Voter server-0fee33a8-f40c-dcac-9f44-568182f39d21 0fee33a8-f40c-dcac-9f44-568182f39d21}"
2024-12-04T01:32:00.014Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:31:59.905698144 +0000 UTC m=+10.737540784 (1 nodes), 64.683893ms from start of poll, 65.131211ms from cluster start. Timeout at 2024-12-04 01:32:00.010704775 +0000 UTC m=+10.842547401, 105.006617ms after stability
2024-12-04T01:32:00.020Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:00.126Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:00.0204337 +0000 UTC m=+10.852276306 (2 nodes), 3.889µs from start of poll, 179.866733ms from cluster start. Timeout at 2024-12-04 01:32:00.125471962 +0000 UTC m=+10.957314893, 105.038587ms after stability
2024-12-04T01:32:00.126Z [INFO]  server-9f602161-15a1-f7de-7cd1-779d9a70e945: updating configuration: command=RemoveServer server-id=server-0fee33a8-f40c-dcac-9f44-568182f39d21 server-addr= servers="[{Suffrage:Voter ID:server-9f602161-15a1-f7de-7cd1-779d9a70e945 Address:9f602161-15a1-f7de-7cd1-779d9a70e945} {Suffrage:Voter ID:server-ccaec0e5-ccd7-6852-8b42-366327004344 Address:ccaec0e5-ccd7-6852-8b42-366327004344}]"
2024-12-04T01:32:00.126Z [INFO]  server-9f602161-15a1-f7de-7cd1-779d9a70e945: removed peer, stopping replication: peer=server-0fee33a8-f40c-dcac-9f44-568182f39d21 last-index=3
2024-12-04T01:32:00.126Z [INFO]  server-9f602161-15a1-f7de-7cd1-779d9a70e945: aborting pipeline replication: peer="{Voter server-0fee33a8-f40c-dcac-9f44-568182f39d21 0fee33a8-f40c-dcac-9f44-568182f39d21}"
2024-12-04T01:32:00.235Z [INFO]  server-9f602161-15a1-f7de-7cd1-779d9a70e945: aborting pipeline replication: peer="{Voter server-ccaec0e5-ccd7-6852-8b42-366327004344 ccaec0e5-ccd7-6852-8b42-366327004344}"
--- PASS: TestRaft_RemoveFollower (0.40s)
=== RUN   TestRaft_RemoveLeader
2024-12-04T01:32:00.237Z [INFO]  server-9644dce8-f1bd-df17-e888-d151572fca2b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9644dce8-f1bd-df17-e888-d151572fca2b Address:9644dce8-f1bd-df17-e888-d151572fca2b} {Suffrage:Voter ID:server-eea71904-f419-b82c-0087-12c57c36ad0d Address:eea71904-f419-b82c-0087-12c57c36ad0d} {Suffrage:Voter ID:server-a37b6b47-af49-fa3a-b76a-9d4acfe97e22 Address:a37b6b47-af49-fa3a-b76a-9d4acfe97e22}]"
2024-12-04T01:32:00.237Z [INFO]  server-eea71904-f419-b82c-0087-12c57c36ad0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9644dce8-f1bd-df17-e888-d151572fca2b Address:9644dce8-f1bd-df17-e888-d151572fca2b} {Suffrage:Voter ID:server-eea71904-f419-b82c-0087-12c57c36ad0d Address:eea71904-f419-b82c-0087-12c57c36ad0d} {Suffrage:Voter ID:server-a37b6b47-af49-fa3a-b76a-9d4acfe97e22 Address:a37b6b47-af49-fa3a-b76a-9d4acfe97e22}]"
2024-12-04T01:32:00.237Z [INFO]  server-a37b6b47-af49-fa3a-b76a-9d4acfe97e22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9644dce8-f1bd-df17-e888-d151572fca2b Address:9644dce8-f1bd-df17-e888-d151572fca2b} {Suffrage:Voter ID:server-eea71904-f419-b82c-0087-12c57c36ad0d Address:eea71904-f419-b82c-0087-12c57c36ad0d} {Suffrage:Voter ID:server-a37b6b47-af49-fa3a-b76a-9d4acfe97e22 Address:a37b6b47-af49-fa3a-b76a-9d4acfe97e22}]"
2024-12-04T01:32:00.237Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:00.237Z [INFO]  server-9644dce8-f1bd-df17-e888-d151572fca2b: entering follower state: follower="Node at 9644dce8-f1bd-df17-e888-d151572fca2b [Follower]" leader-address= leader-id=
2024-12-04T01:32:00.237Z [INFO]  server-eea71904-f419-b82c-0087-12c57c36ad0d: entering follower state: follower="Node at eea71904-f419-b82c-0087-12c57c36ad0d [Follower]" leader-address= leader-id=
2024-12-04T01:32:00.237Z [INFO]  server-a37b6b47-af49-fa3a-b76a-9d4acfe97e22: entering follower state: follower="Node at a37b6b47-af49-fa3a-b76a-9d4acfe97e22 [Follower]" leader-address= leader-id=
2024-12-04T01:32:00.290Z [WARN]  server-eea71904-f419-b82c-0087-12c57c36ad0d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:00.290Z [INFO]  server-eea71904-f419-b82c-0087-12c57c36ad0d: entering candidate state: node="Node at eea71904-f419-b82c-0087-12c57c36ad0d [Candidate]" term=2
2024-12-04T01:32:00.290Z [INFO]  server-eea71904-f419-b82c-0087-12c57c36ad0d: election won: term=2 tally=2
2024-12-04T01:32:00.290Z [INFO]  server-eea71904-f419-b82c-0087-12c57c36ad0d: entering leader state: leader="Node at eea71904-f419-b82c-0087-12c57c36ad0d [Leader]"
2024-12-04T01:32:00.290Z [INFO]  server-eea71904-f419-b82c-0087-12c57c36ad0d: added peer, starting replication: peer=server-9644dce8-f1bd-df17-e888-d151572fca2b
2024-12-04T01:32:00.290Z [INFO]  server-eea71904-f419-b82c-0087-12c57c36ad0d: added peer, starting replication: peer=server-a37b6b47-af49-fa3a-b76a-9d4acfe97e22
2024-12-04T01:32:00.290Z [INFO]  server-eea71904-f419-b82c-0087-12c57c36ad0d: pipelining replication: peer="{Voter server-a37b6b47-af49-fa3a-b76a-9d4acfe97e22 a37b6b47-af49-fa3a-b76a-9d4acfe97e22}"
2024-12-04T01:32:00.291Z [INFO]  server-eea71904-f419-b82c-0087-12c57c36ad0d: pipelining replication: peer="{Voter server-9644dce8-f1bd-df17-e888-d151572fca2b 9644dce8-f1bd-df17-e888-d151572fca2b}"
2024-12-04T01:32:00.398Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:00.291255898 +0000 UTC m=+11.123098661 (1 nodes), 53.419645ms from start of poll, 53.904051ms from cluster start. Timeout at 2024-12-04 01:32:00.396262755 +0000 UTC m=+11.228105623, 105.006962ms after stability
2024-12-04T01:32:00.406Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:00.514Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:00.40664987 +0000 UTC m=+11.238492528 (2 nodes), 4.604µs from start of poll, 169.297918ms from cluster start. Timeout at 2024-12-04 01:32:00.511680552 +0000 UTC m=+11.343523689, 105.031161ms after stability
2024-12-04T01:32:00.514Z [INFO]  server-eea71904-f419-b82c-0087-12c57c36ad0d: updating configuration: command=RemoveServer server-id=server-eea71904-f419-b82c-0087-12c57c36ad0d server-addr= servers="[{Suffrage:Voter ID:server-9644dce8-f1bd-df17-e888-d151572fca2b Address:9644dce8-f1bd-df17-e888-d151572fca2b} {Suffrage:Voter ID:server-a37b6b47-af49-fa3a-b76a-9d4acfe97e22 Address:a37b6b47-af49-fa3a-b76a-9d4acfe97e22}]"
2024-12-04T01:32:00.514Z [INFO]  server-eea71904-f419-b82c-0087-12c57c36ad0d: removed ourself, shutting down
2024-12-04T01:32:00.514Z [INFO]  server-eea71904-f419-b82c-0087-12c57c36ad0d: aborting pipeline replication: peer="{Voter server-a37b6b47-af49-fa3a-b76a-9d4acfe97e22 a37b6b47-af49-fa3a-b76a-9d4acfe97e22}"
2024-12-04T01:32:00.515Z [INFO]  server-eea71904-f419-b82c-0087-12c57c36ad0d: aborting pipeline replication: peer="{Voter server-9644dce8-f1bd-df17-e888-d151572fca2b 9644dce8-f1bd-df17-e888-d151572fca2b}"
2024-12-04T01:32:00.594Z [WARN]  server-9644dce8-f1bd-df17-e888-d151572fca2b: heartbeat timeout reached, starting election: last-leader-addr=eea71904-f419-b82c-0087-12c57c36ad0d last-leader-id=server-eea71904-f419-b82c-0087-12c57c36ad0d
2024-12-04T01:32:00.594Z [INFO]  server-9644dce8-f1bd-df17-e888-d151572fca2b: entering candidate state: node="Node at 9644dce8-f1bd-df17-e888-d151572fca2b [Candidate]" term=3
2024-12-04T01:32:00.594Z [WARN]  server-a37b6b47-af49-fa3a-b76a-9d4acfe97e22: rejecting vote request since we have a leader: from=9644dce8-f1bd-df17-e888-d151572fca2b leader=eea71904-f419-b82c-0087-12c57c36ad0d leader-id=server-eea71904-f419-b82c-0087-12c57c36ad0d
2024-12-04T01:32:00.602Z [WARN]  server-a37b6b47-af49-fa3a-b76a-9d4acfe97e22: heartbeat timeout reached, starting election: last-leader-addr=eea71904-f419-b82c-0087-12c57c36ad0d last-leader-id=server-eea71904-f419-b82c-0087-12c57c36ad0d
2024-12-04T01:32:00.602Z [INFO]  server-a37b6b47-af49-fa3a-b76a-9d4acfe97e22: entering candidate state: node="Node at a37b6b47-af49-fa3a-b76a-9d4acfe97e22 [Candidate]" term=3
2024-12-04T01:32:00.602Z [INFO]  server-9644dce8-f1bd-df17-e888-d151572fca2b: duplicate requestVote for same term: term=3
2024-12-04T01:32:00.658Z [WARN]  server-9644dce8-f1bd-df17-e888-d151572fca2b: Election timeout reached, restarting election
2024-12-04T01:32:00.658Z [INFO]  server-9644dce8-f1bd-df17-e888-d151572fca2b: entering candidate state: node="Node at 9644dce8-f1bd-df17-e888-d151572fca2b [Candidate]" term=4
2024-12-04T01:32:00.658Z [INFO]  server-a37b6b47-af49-fa3a-b76a-9d4acfe97e22: entering follower state: follower="Node at a37b6b47-af49-fa3a-b76a-9d4acfe97e22 [Follower]" leader-address= leader-id=
2024-12-04T01:32:00.658Z [INFO]  server-9644dce8-f1bd-df17-e888-d151572fca2b: election won: term=4 tally=2
2024-12-04T01:32:00.658Z [INFO]  server-9644dce8-f1bd-df17-e888-d151572fca2b: entering leader state: leader="Node at 9644dce8-f1bd-df17-e888-d151572fca2b [Leader]"
2024-12-04T01:32:00.658Z [INFO]  server-9644dce8-f1bd-df17-e888-d151572fca2b: added peer, starting replication: peer=server-a37b6b47-af49-fa3a-b76a-9d4acfe97e22
2024-12-04T01:32:00.658Z [INFO]  server-9644dce8-f1bd-df17-e888-d151572fca2b: pipelining replication: peer="{Voter server-a37b6b47-af49-fa3a-b76a-9d4acfe97e22 a37b6b47-af49-fa3a-b76a-9d4acfe97e22}"
2024-12-04T01:32:00.733Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:00.838Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:00.733520654 +0000 UTC m=+11.565363409 (1 nodes), 107.593µs from start of poll, 496.168799ms from cluster start. Timeout at 2024-12-04 01:32:00.838553817 +0000 UTC m=+11.670396446, 105.033037ms after stability
2024-12-04T01:32:00.838Z [INFO]  server-9644dce8-f1bd-df17-e888-d151572fca2b: aborting pipeline replication: peer="{Voter server-a37b6b47-af49-fa3a-b76a-9d4acfe97e22 a37b6b47-af49-fa3a-b76a-9d4acfe97e22}"
--- PASS: TestRaft_RemoveLeader (0.60s)
=== RUN   TestRaft_RemoveLeader_NoShutdown
2024-12-04T01:32:00.840Z [INFO]  server-2d380865-a14c-b9a2-b825-f1c5b5742e68: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d380865-a14c-b9a2-b825-f1c5b5742e68 Address:2d380865-a14c-b9a2-b825-f1c5b5742e68} {Suffrage:Voter ID:server-16703d1e-7b3b-73d4-9d42-39f256664f6c Address:16703d1e-7b3b-73d4-9d42-39f256664f6c} {Suffrage:Voter ID:server-f1658132-61b8-f4c9-19fa-c231a97327b9 Address:f1658132-61b8-f4c9-19fa-c231a97327b9}]"
2024-12-04T01:32:00.840Z [INFO]  server-16703d1e-7b3b-73d4-9d42-39f256664f6c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d380865-a14c-b9a2-b825-f1c5b5742e68 Address:2d380865-a14c-b9a2-b825-f1c5b5742e68} {Suffrage:Voter ID:server-16703d1e-7b3b-73d4-9d42-39f256664f6c Address:16703d1e-7b3b-73d4-9d42-39f256664f6c} {Suffrage:Voter ID:server-f1658132-61b8-f4c9-19fa-c231a97327b9 Address:f1658132-61b8-f4c9-19fa-c231a97327b9}]"
2024-12-04T01:32:00.840Z [INFO]  server-f1658132-61b8-f4c9-19fa-c231a97327b9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d380865-a14c-b9a2-b825-f1c5b5742e68 Address:2d380865-a14c-b9a2-b825-f1c5b5742e68} {Suffrage:Voter ID:server-16703d1e-7b3b-73d4-9d42-39f256664f6c Address:16703d1e-7b3b-73d4-9d42-39f256664f6c} {Suffrage:Voter ID:server-f1658132-61b8-f4c9-19fa-c231a97327b9 Address:f1658132-61b8-f4c9-19fa-c231a97327b9}]"
2024-12-04T01:32:00.841Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:00.841Z [INFO]  server-2d380865-a14c-b9a2-b825-f1c5b5742e68: entering follower state: follower="Node at 2d380865-a14c-b9a2-b825-f1c5b5742e68 [Follower]" leader-address= leader-id=
2024-12-04T01:32:00.841Z [INFO]  server-16703d1e-7b3b-73d4-9d42-39f256664f6c: entering follower state: follower="Node at 16703d1e-7b3b-73d4-9d42-39f256664f6c [Follower]" leader-address= leader-id=
2024-12-04T01:32:00.841Z [INFO]  server-f1658132-61b8-f4c9-19fa-c231a97327b9: entering follower state: follower="Node at f1658132-61b8-f4c9-19fa-c231a97327b9 [Follower]" leader-address= leader-id=
2024-12-04T01:32:00.894Z [WARN]  server-f1658132-61b8-f4c9-19fa-c231a97327b9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:00.894Z [INFO]  server-f1658132-61b8-f4c9-19fa-c231a97327b9: entering candidate state: node="Node at f1658132-61b8-f4c9-19fa-c231a97327b9 [Candidate]" term=2
2024-12-04T01:32:00.894Z [INFO]  server-f1658132-61b8-f4c9-19fa-c231a97327b9: election won: term=2 tally=2
2024-12-04T01:32:00.894Z [INFO]  server-f1658132-61b8-f4c9-19fa-c231a97327b9: entering leader state: leader="Node at f1658132-61b8-f4c9-19fa-c231a97327b9 [Leader]"
2024-12-04T01:32:00.894Z [INFO]  server-f1658132-61b8-f4c9-19fa-c231a97327b9: added peer, starting replication: peer=server-2d380865-a14c-b9a2-b825-f1c5b5742e68
2024-12-04T01:32:00.895Z [INFO]  server-f1658132-61b8-f4c9-19fa-c231a97327b9: added peer, starting replication: peer=server-16703d1e-7b3b-73d4-9d42-39f256664f6c
2024-12-04T01:32:00.895Z [INFO]  server-f1658132-61b8-f4c9-19fa-c231a97327b9: pipelining replication: peer="{Voter server-16703d1e-7b3b-73d4-9d42-39f256664f6c 16703d1e-7b3b-73d4-9d42-39f256664f6c}"
2024-12-04T01:32:00.895Z [INFO]  server-f1658132-61b8-f4c9-19fa-c231a97327b9: pipelining replication: peer="{Voter server-2d380865-a14c-b9a2-b825-f1c5b5742e68 2d380865-a14c-b9a2-b825-f1c5b5742e68}"
2024-12-04T01:32:01.002Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:00.895562212 +0000 UTC m=+11.727404810 (2 nodes), 54.522034ms from start of poll, 54.986328ms from cluster start. Timeout at 2024-12-04 01:32:01.000581976 +0000 UTC m=+11.832424899, 105.020089ms after stability
2024-12-04T01:32:01.002Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:01.107Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:01.002355857 +0000 UTC m=+11.834198486 (1 nodes), 3.114µs from start of poll, 161.780004ms from cluster start. Timeout at 2024-12-04 01:32:01.107370467 +0000 UTC m=+11.939213356, 105.01487ms after stability
2024-12-04T01:32:01.130Z [INFO]  server-f1658132-61b8-f4c9-19fa-c231a97327b9: updating configuration: command=RemoveServer server-id=server-f1658132-61b8-f4c9-19fa-c231a97327b9 server-addr= servers="[{Suffrage:Voter ID:server-2d380865-a14c-b9a2-b825-f1c5b5742e68 Address:2d380865-a14c-b9a2-b825-f1c5b5742e68} {Suffrage:Voter ID:server-16703d1e-7b3b-73d4-9d42-39f256664f6c Address:16703d1e-7b3b-73d4-9d42-39f256664f6c}]"
2024-12-04T01:32:01.131Z [INFO]  server-f1658132-61b8-f4c9-19fa-c231a97327b9: removed ourself, transitioning to follower
2024-12-04T01:32:01.131Z [INFO]  server-f1658132-61b8-f4c9-19fa-c231a97327b9: entering follower state: follower="Node at f1658132-61b8-f4c9-19fa-c231a97327b9 [Follower]" leader-address= leader-id=
2024-12-04T01:32:01.131Z [INFO]  server-f1658132-61b8-f4c9-19fa-c231a97327b9: aborting pipeline replication: peer="{Voter server-2d380865-a14c-b9a2-b825-f1c5b5742e68 2d380865-a14c-b9a2-b825-f1c5b5742e68}"
2024-12-04T01:32:01.131Z [INFO]  server-f1658132-61b8-f4c9-19fa-c231a97327b9: aborting pipeline replication: peer="{Voter server-16703d1e-7b3b-73d4-9d42-39f256664f6c 16703d1e-7b3b-73d4-9d42-39f256664f6c}"
2024-12-04T01:32:01.186Z [WARN]  server-16703d1e-7b3b-73d4-9d42-39f256664f6c: heartbeat timeout reached, starting election: last-leader-addr=f1658132-61b8-f4c9-19fa-c231a97327b9 last-leader-id=server-f1658132-61b8-f4c9-19fa-c231a97327b9
2024-12-04T01:32:01.186Z [INFO]  server-16703d1e-7b3b-73d4-9d42-39f256664f6c: entering candidate state: node="Node at 16703d1e-7b3b-73d4-9d42-39f256664f6c [Candidate]" term=3
2024-12-04T01:32:01.186Z [WARN]  server-2d380865-a14c-b9a2-b825-f1c5b5742e68: rejecting vote request since we have a leader: from=16703d1e-7b3b-73d4-9d42-39f256664f6c leader=f1658132-61b8-f4c9-19fa-c231a97327b9 leader-id=server-f1658132-61b8-f4c9-19fa-c231a97327b9
2024-12-04T01:32:01.190Z [WARN]  server-2d380865-a14c-b9a2-b825-f1c5b5742e68: heartbeat timeout reached, starting election: last-leader-addr=f1658132-61b8-f4c9-19fa-c231a97327b9 last-leader-id=server-f1658132-61b8-f4c9-19fa-c231a97327b9
2024-12-04T01:32:01.190Z [INFO]  server-2d380865-a14c-b9a2-b825-f1c5b5742e68: entering candidate state: node="Node at 2d380865-a14c-b9a2-b825-f1c5b5742e68 [Candidate]" term=3
2024-12-04T01:32:01.190Z [INFO]  server-16703d1e-7b3b-73d4-9d42-39f256664f6c: duplicate requestVote for same term: term=3
2024-12-04T01:32:01.198Z [WARN]  server-f1658132-61b8-f4c9-19fa-c231a97327b9: not part of stable configuration, aborting election
2024-12-04T01:32:01.238Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:01.262Z [WARN]  server-16703d1e-7b3b-73d4-9d42-39f256664f6c: Election timeout reached, restarting election
2024-12-04T01:32:01.262Z [INFO]  server-16703d1e-7b3b-73d4-9d42-39f256664f6c: entering candidate state: node="Node at 16703d1e-7b3b-73d4-9d42-39f256664f6c [Candidate]" term=4
2024-12-04T01:32:01.262Z [INFO]  server-2d380865-a14c-b9a2-b825-f1c5b5742e68: entering follower state: follower="Node at 2d380865-a14c-b9a2-b825-f1c5b5742e68 [Follower]" leader-address= leader-id=
2024-12-04T01:32:01.262Z [INFO]  server-16703d1e-7b3b-73d4-9d42-39f256664f6c: election won: term=4 tally=2
2024-12-04T01:32:01.262Z [INFO]  server-16703d1e-7b3b-73d4-9d42-39f256664f6c: entering leader state: leader="Node at 16703d1e-7b3b-73d4-9d42-39f256664f6c [Leader]"
2024-12-04T01:32:01.262Z [INFO]  server-16703d1e-7b3b-73d4-9d42-39f256664f6c: added peer, starting replication: peer=server-2d380865-a14c-b9a2-b825-f1c5b5742e68
2024-12-04T01:32:01.262Z [INFO]  server-16703d1e-7b3b-73d4-9d42-39f256664f6c: pipelining replication: peer="{Voter server-2d380865-a14c-b9a2-b825-f1c5b5742e68 2d380865-a14c-b9a2-b825-f1c5b5742e68}"
2024-12-04T01:32:01.370Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:01.2629577 +0000 UTC m=+12.094800372 (1 nodes), 24.573736ms from start of poll, 422.38189ms from cluster start. Timeout at 2024-12-04 01:32:01.367968311 +0000 UTC m=+12.199811578, 105.011206ms after stability
2024-12-04T01:32:01.478Z [INFO]  server-16703d1e-7b3b-73d4-9d42-39f256664f6c: aborting pipeline replication: peer="{Voter server-2d380865-a14c-b9a2-b825-f1c5b5742e68 2d380865-a14c-b9a2-b825-f1c5b5742e68}"
--- PASS: TestRaft_RemoveLeader_NoShutdown (0.64s)
=== RUN   TestRaft_RemoveFollower_SplitCluster
2024-12-04T01:32:01.480Z [INFO]  server-93d24590-d6ca-fe30-a853-edb54841b0bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93d24590-d6ca-fe30-a853-edb54841b0bf Address:93d24590-d6ca-fe30-a853-edb54841b0bf} {Suffrage:Voter ID:server-a333f361-abe6-dcb9-1b46-a836bd431517 Address:a333f361-abe6-dcb9-1b46-a836bd431517} {Suffrage:Voter ID:server-358d716f-3e9d-2c44-2522-0876eb2b22d7 Address:358d716f-3e9d-2c44-2522-0876eb2b22d7} {Suffrage:Voter ID:server-e256e742-a277-8824-c635-abde1a0fc824 Address:e256e742-a277-8824-c635-abde1a0fc824}]"
2024-12-04T01:32:01.480Z [INFO]  server-a333f361-abe6-dcb9-1b46-a836bd431517: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93d24590-d6ca-fe30-a853-edb54841b0bf Address:93d24590-d6ca-fe30-a853-edb54841b0bf} {Suffrage:Voter ID:server-a333f361-abe6-dcb9-1b46-a836bd431517 Address:a333f361-abe6-dcb9-1b46-a836bd431517} {Suffrage:Voter ID:server-358d716f-3e9d-2c44-2522-0876eb2b22d7 Address:358d716f-3e9d-2c44-2522-0876eb2b22d7} {Suffrage:Voter ID:server-e256e742-a277-8824-c635-abde1a0fc824 Address:e256e742-a277-8824-c635-abde1a0fc824}]"
2024-12-04T01:32:01.480Z [INFO]  server-358d716f-3e9d-2c44-2522-0876eb2b22d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93d24590-d6ca-fe30-a853-edb54841b0bf Address:93d24590-d6ca-fe30-a853-edb54841b0bf} {Suffrage:Voter ID:server-a333f361-abe6-dcb9-1b46-a836bd431517 Address:a333f361-abe6-dcb9-1b46-a836bd431517} {Suffrage:Voter ID:server-358d716f-3e9d-2c44-2522-0876eb2b22d7 Address:358d716f-3e9d-2c44-2522-0876eb2b22d7} {Suffrage:Voter ID:server-e256e742-a277-8824-c635-abde1a0fc824 Address:e256e742-a277-8824-c635-abde1a0fc824}]"
2024-12-04T01:32:01.480Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-93d24590-d6ca-fe30-a853-edb54841b0bf Address:93d24590-d6ca-fe30-a853-edb54841b0bf} {Suffrage:Voter ID:server-a333f361-abe6-dcb9-1b46-a836bd431517 Address:a333f361-abe6-dcb9-1b46-a836bd431517} {Suffrage:Voter ID:server-358d716f-3e9d-2c44-2522-0876eb2b22d7 Address:358d716f-3e9d-2c44-2522-0876eb2b22d7} {Suffrage:Voter ID:server-e256e742-a277-8824-c635-abde1a0fc824 Address:e256e742-a277-8824-c635-abde1a0fc824}]"
2024-12-04T01:32:01.480Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:01.480Z [INFO]  server-93d24590-d6ca-fe30-a853-edb54841b0bf: entering follower state: follower="Node at 93d24590-d6ca-fe30-a853-edb54841b0bf [Follower]" leader-address= leader-id=
2024-12-04T01:32:01.480Z [INFO]  server-a333f361-abe6-dcb9-1b46-a836bd431517: entering follower state: follower="Node at a333f361-abe6-dcb9-1b46-a836bd431517 [Follower]" leader-address= leader-id=
2024-12-04T01:32:01.480Z [INFO]  server-358d716f-3e9d-2c44-2522-0876eb2b22d7: entering follower state: follower="Node at 358d716f-3e9d-2c44-2522-0876eb2b22d7 [Follower]" leader-address= leader-id=
2024-12-04T01:32:01.481Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: entering follower state: follower="Node at e256e742-a277-8824-c635-abde1a0fc824 [Follower]" leader-address= leader-id=
2024-12-04T01:32:01.558Z [WARN]  server-e256e742-a277-8824-c635-abde1a0fc824: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:01.558Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: entering candidate state: node="Node at e256e742-a277-8824-c635-abde1a0fc824 [Candidate]" term=2
2024-12-04T01:32:01.558Z [WARN]  server-a333f361-abe6-dcb9-1b46-a836bd431517: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:01.558Z [INFO]  server-a333f361-abe6-dcb9-1b46-a836bd431517: entering candidate state: node="Node at a333f361-abe6-dcb9-1b46-a836bd431517 [Candidate]" term=2
2024-12-04T01:32:01.558Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: duplicate requestVote for same term: term=2
2024-12-04T01:32:01.558Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: election won: term=2 tally=3
2024-12-04T01:32:01.558Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: entering leader state: leader="Node at e256e742-a277-8824-c635-abde1a0fc824 [Leader]"
2024-12-04T01:32:01.558Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: added peer, starting replication: peer=server-93d24590-d6ca-fe30-a853-edb54841b0bf
2024-12-04T01:32:01.558Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: added peer, starting replication: peer=server-a333f361-abe6-dcb9-1b46-a836bd431517
2024-12-04T01:32:01.558Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: added peer, starting replication: peer=server-358d716f-3e9d-2c44-2522-0876eb2b22d7
2024-12-04T01:32:01.558Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: pipelining replication: peer="{Voter server-358d716f-3e9d-2c44-2522-0876eb2b22d7 358d716f-3e9d-2c44-2522-0876eb2b22d7}"
2024-12-04T01:32:01.559Z [INFO]  server-a333f361-abe6-dcb9-1b46-a836bd431517: duplicate requestVote for same term: term=2
2024-12-04T01:32:01.559Z [INFO]  server-93d24590-d6ca-fe30-a853-edb54841b0bf: duplicate requestVote for same term: term=2
2024-12-04T01:32:01.559Z [WARN]  server-358d716f-3e9d-2c44-2522-0876eb2b22d7: rejecting vote request since we have a leader: from=a333f361-abe6-dcb9-1b46-a836bd431517 leader=e256e742-a277-8824-c635-abde1a0fc824 leader-id=server-e256e742-a277-8824-c635-abde1a0fc824
2024-12-04T01:32:01.559Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: pipelining replication: peer="{Voter server-93d24590-d6ca-fe30-a853-edb54841b0bf 93d24590-d6ca-fe30-a853-edb54841b0bf}"
2024-12-04T01:32:01.559Z [INFO]  server-a333f361-abe6-dcb9-1b46-a836bd431517: entering follower state: follower="Node at a333f361-abe6-dcb9-1b46-a836bd431517 [Follower]" leader-address=e256e742-a277-8824-c635-abde1a0fc824 leader-id=server-e256e742-a277-8824-c635-abde1a0fc824
2024-12-04T01:32:01.559Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: pipelining replication: peer="{Voter server-a333f361-abe6-dcb9-1b46-a836bd431517 a333f361-abe6-dcb9-1b46-a836bd431517}"
2024-12-04T01:32:01.666Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:01.559566469 +0000 UTC m=+12.391409138 (1 nodes), 78.722045ms from start of poll, 79.318567ms from cluster start. Timeout at 2024-12-04 01:32:01.664571005 +0000 UTC m=+12.496413885, 105.004747ms after stability
2024-12-04T01:32:01.774Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:01.882Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:01.774357988 +0000 UTC m=+12.606200751 (3 nodes), 2.651µs from start of poll, 294.11018ms from cluster start. Timeout at 2024-12-04 01:32:01.879377707 +0000 UTC m=+12.711220368, 105.019617ms after stability
2024-12-04T01:32:01.882Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: updating configuration: command=RemoveServer server-id=server-358d716f-3e9d-2c44-2522-0876eb2b22d7 server-addr= servers="[{Suffrage:Voter ID:server-93d24590-d6ca-fe30-a853-edb54841b0bf Address:93d24590-d6ca-fe30-a853-edb54841b0bf} {Suffrage:Voter ID:server-a333f361-abe6-dcb9-1b46-a836bd431517 Address:a333f361-abe6-dcb9-1b46-a836bd431517} {Suffrage:Voter ID:server-e256e742-a277-8824-c635-abde1a0fc824 Address:e256e742-a277-8824-c635-abde1a0fc824}]"
2024-12-04T01:32:01.882Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: removed peer, stopping replication: peer=server-358d716f-3e9d-2c44-2522-0876eb2b22d7 last-index=3
2024-12-04T01:32:01.882Z [ERROR] server-e256e742-a277-8824-c635-abde1a0fc824: failed to heartbeat to: peer=a333f361-abe6-dcb9-1b46-a836bd431517 backoff time=10ms error="failed to connect to peer: a333f361-abe6-dcb9-1b46-a836bd431517"
2024-12-04T01:32:01.882Z [ERROR] server-e256e742-a277-8824-c635-abde1a0fc824: failed to pipeline appendEntries: peer="{Voter server-93d24590-d6ca-fe30-a853-edb54841b0bf 93d24590-d6ca-fe30-a853-edb54841b0bf}" error="append pipeline closed"
2024-12-04T01:32:01.882Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: aborting pipeline replication: peer="{Voter server-93d24590-d6ca-fe30-a853-edb54841b0bf 93d24590-d6ca-fe30-a853-edb54841b0bf}"
2024-12-04T01:32:01.882Z [ERROR] server-e256e742-a277-8824-c635-abde1a0fc824: failed to pipeline appendEntries: peer="{Voter server-a333f361-abe6-dcb9-1b46-a836bd431517 a333f361-abe6-dcb9-1b46-a836bd431517}" error="append pipeline closed"
2024-12-04T01:32:01.883Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: aborting pipeline replication: peer="{Voter server-a333f361-abe6-dcb9-1b46-a836bd431517 a333f361-abe6-dcb9-1b46-a836bd431517}"
2024-12-04T01:32:01.883Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: aborting pipeline replication: peer="{Voter server-358d716f-3e9d-2c44-2522-0876eb2b22d7 358d716f-3e9d-2c44-2522-0876eb2b22d7}"
2024-12-04T01:32:01.890Z [ERROR] server-e256e742-a277-8824-c635-abde1a0fc824: failed to appendEntries to: peer="{Voter server-a333f361-abe6-dcb9-1b46-a836bd431517 a333f361-abe6-dcb9-1b46-a836bd431517}" error="failed to connect to peer: a333f361-abe6-dcb9-1b46-a836bd431517"
2024-12-04T01:32:01.891Z [ERROR] server-e256e742-a277-8824-c635-abde1a0fc824: failed to heartbeat to: peer=93d24590-d6ca-fe30-a853-edb54841b0bf backoff time=10ms error="failed to connect to peer: 93d24590-d6ca-fe30-a853-edb54841b0bf"
2024-12-04T01:32:01.891Z [ERROR] server-e256e742-a277-8824-c635-abde1a0fc824: failed to appendEntries to: peer="{Voter server-93d24590-d6ca-fe30-a853-edb54841b0bf 93d24590-d6ca-fe30-a853-edb54841b0bf}" error="failed to connect to peer: 93d24590-d6ca-fe30-a853-edb54841b0bf"
2024-12-04T01:32:01.906Z [ERROR] server-e256e742-a277-8824-c635-abde1a0fc824: failed to heartbeat to: peer=a333f361-abe6-dcb9-1b46-a836bd431517 backoff time=10ms error="failed to connect to peer: a333f361-abe6-dcb9-1b46-a836bd431517"
2024-12-04T01:32:01.910Z [ERROR] server-e256e742-a277-8824-c635-abde1a0fc824: failed to heartbeat to: peer=93d24590-d6ca-fe30-a853-edb54841b0bf backoff time=10ms error="failed to connect to peer: 93d24590-d6ca-fe30-a853-edb54841b0bf"
2024-12-04T01:32:01.910Z [ERROR] server-e256e742-a277-8824-c635-abde1a0fc824: failed to appendEntries to: peer="{Voter server-a333f361-abe6-dcb9-1b46-a836bd431517 a333f361-abe6-dcb9-1b46-a836bd431517}" error="failed to connect to peer: a333f361-abe6-dcb9-1b46-a836bd431517"
2024-12-04T01:32:01.914Z [ERROR] server-e256e742-a277-8824-c635-abde1a0fc824: failed to appendEntries to: peer="{Voter server-93d24590-d6ca-fe30-a853-edb54841b0bf 93d24590-d6ca-fe30-a853-edb54841b0bf}" error="failed to connect to peer: 93d24590-d6ca-fe30-a853-edb54841b0bf"
2024-12-04T01:32:01.926Z [WARN]  server-e256e742-a277-8824-c635-abde1a0fc824: failed to contact: server-id=server-a333f361-abe6-dcb9-1b46-a836bd431517 time=51.908488ms
2024-12-04T01:32:01.926Z [ERROR] server-e256e742-a277-8824-c635-abde1a0fc824: failed to heartbeat to: peer=a333f361-abe6-dcb9-1b46-a836bd431517 backoff time=10ms error="failed to connect to peer: a333f361-abe6-dcb9-1b46-a836bd431517"
2024-12-04T01:32:01.934Z [ERROR] server-e256e742-a277-8824-c635-abde1a0fc824: failed to appendEntries to: peer="{Voter server-a333f361-abe6-dcb9-1b46-a836bd431517 a333f361-abe6-dcb9-1b46-a836bd431517}" error="failed to connect to peer: a333f361-abe6-dcb9-1b46-a836bd431517"
2024-12-04T01:32:01.934Z [ERROR] server-e256e742-a277-8824-c635-abde1a0fc824: failed to heartbeat to: peer=93d24590-d6ca-fe30-a853-edb54841b0bf backoff time=10ms error="failed to connect to peer: 93d24590-d6ca-fe30-a853-edb54841b0bf"
2024-12-04T01:32:01.939Z [ERROR] server-e256e742-a277-8824-c635-abde1a0fc824: failed to appendEntries to: peer="{Voter server-93d24590-d6ca-fe30-a853-edb54841b0bf 93d24590-d6ca-fe30-a853-edb54841b0bf}" error="failed to connect to peer: 93d24590-d6ca-fe30-a853-edb54841b0bf"
2024-12-04T01:32:01.939Z [WARN]  server-e256e742-a277-8824-c635-abde1a0fc824: failed to contact: server-id=server-93d24590-d6ca-fe30-a853-edb54841b0bf time=57.208188ms
2024-12-04T01:32:01.939Z [WARN]  server-e256e742-a277-8824-c635-abde1a0fc824: failed to contact: server-id=server-a333f361-abe6-dcb9-1b46-a836bd431517 time=65.116545ms
2024-12-04T01:32:01.939Z [WARN]  server-e256e742-a277-8824-c635-abde1a0fc824: failed to contact quorum of nodes, stepping down
2024-12-04T01:32:01.939Z [INFO]  server-e256e742-a277-8824-c635-abde1a0fc824: entering follower state: follower="Node at e256e742-a277-8824-c635-abde1a0fc824 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_RemoveFollower_SplitCluster (0.46s)
=== RUN   TestRaft_AddKnownPeer
2024-12-04T01:32:01.941Z [INFO]  server-12efa9e7-73df-44ee-d002-b0d0f9fab36c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12efa9e7-73df-44ee-d002-b0d0f9fab36c Address:12efa9e7-73df-44ee-d002-b0d0f9fab36c} {Suffrage:Voter ID:server-092e8925-5f9d-e3fc-8574-7fb986394d47 Address:092e8925-5f9d-e3fc-8574-7fb986394d47} {Suffrage:Voter ID:server-4c2696b9-9026-bd1f-42be-7655d6f51c31 Address:4c2696b9-9026-bd1f-42be-7655d6f51c31}]"
2024-12-04T01:32:01.942Z [INFO]  server-092e8925-5f9d-e3fc-8574-7fb986394d47: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12efa9e7-73df-44ee-d002-b0d0f9fab36c Address:12efa9e7-73df-44ee-d002-b0d0f9fab36c} {Suffrage:Voter ID:server-092e8925-5f9d-e3fc-8574-7fb986394d47 Address:092e8925-5f9d-e3fc-8574-7fb986394d47} {Suffrage:Voter ID:server-4c2696b9-9026-bd1f-42be-7655d6f51c31 Address:4c2696b9-9026-bd1f-42be-7655d6f51c31}]"
2024-12-04T01:32:01.942Z [INFO]  server-4c2696b9-9026-bd1f-42be-7655d6f51c31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-12efa9e7-73df-44ee-d002-b0d0f9fab36c Address:12efa9e7-73df-44ee-d002-b0d0f9fab36c} {Suffrage:Voter ID:server-092e8925-5f9d-e3fc-8574-7fb986394d47 Address:092e8925-5f9d-e3fc-8574-7fb986394d47} {Suffrage:Voter ID:server-4c2696b9-9026-bd1f-42be-7655d6f51c31 Address:4c2696b9-9026-bd1f-42be-7655d6f51c31}]"
2024-12-04T01:32:01.942Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:01.942Z [INFO]  server-092e8925-5f9d-e3fc-8574-7fb986394d47: entering follower state: follower="Node at 092e8925-5f9d-e3fc-8574-7fb986394d47 [Follower]" leader-address= leader-id=
2024-12-04T01:32:01.942Z [INFO]  server-12efa9e7-73df-44ee-d002-b0d0f9fab36c: entering follower state: follower="Node at 12efa9e7-73df-44ee-d002-b0d0f9fab36c [Follower]" leader-address= leader-id=
2024-12-04T01:32:01.942Z [INFO]  server-4c2696b9-9026-bd1f-42be-7655d6f51c31: entering follower state: follower="Node at 4c2696b9-9026-bd1f-42be-7655d6f51c31 [Follower]" leader-address= leader-id=
2024-12-04T01:32:02.006Z [WARN]  server-12efa9e7-73df-44ee-d002-b0d0f9fab36c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:02.006Z [INFO]  server-12efa9e7-73df-44ee-d002-b0d0f9fab36c: entering candidate state: node="Node at 12efa9e7-73df-44ee-d002-b0d0f9fab36c [Candidate]" term=2
2024-12-04T01:32:02.006Z [INFO]  server-12efa9e7-73df-44ee-d002-b0d0f9fab36c: election won: term=2 tally=2
2024-12-04T01:32:02.006Z [INFO]  server-12efa9e7-73df-44ee-d002-b0d0f9fab36c: entering leader state: leader="Node at 12efa9e7-73df-44ee-d002-b0d0f9fab36c [Leader]"
2024-12-04T01:32:02.006Z [INFO]  server-12efa9e7-73df-44ee-d002-b0d0f9fab36c: added peer, starting replication: peer=server-092e8925-5f9d-e3fc-8574-7fb986394d47
2024-12-04T01:32:02.006Z [INFO]  server-12efa9e7-73df-44ee-d002-b0d0f9fab36c: added peer, starting replication: peer=server-4c2696b9-9026-bd1f-42be-7655d6f51c31
2024-12-04T01:32:02.006Z [INFO]  server-12efa9e7-73df-44ee-d002-b0d0f9fab36c: pipelining replication: peer="{Voter server-4c2696b9-9026-bd1f-42be-7655d6f51c31 4c2696b9-9026-bd1f-42be-7655d6f51c31}"
2024-12-04T01:32:02.007Z [INFO]  server-12efa9e7-73df-44ee-d002-b0d0f9fab36c: pipelining replication: peer="{Voter server-092e8925-5f9d-e3fc-8574-7fb986394d47 092e8925-5f9d-e3fc-8574-7fb986394d47}"
2024-12-04T01:32:02.113Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:02.007314551 +0000 UTC m=+12.839157276 (1 nodes), 65.045683ms from start of poll, 65.591324ms from cluster start. Timeout at 2024-12-04 01:32:02.112330908 +0000 UTC m=+12.944173621, 105.016345ms after stability
2024-12-04T01:32:02.113Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:02.222Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:02.113840807 +0000 UTC m=+12.945683445 (2 nodes), 4.216µs from start of poll, 172.117493ms from cluster start. Timeout at 2024-12-04 01:32:02.21886431 +0000 UTC m=+13.050707148, 105.023703ms after stability
2024-12-04T01:32:02.222Z [INFO]  server-12efa9e7-73df-44ee-d002-b0d0f9fab36c: updating configuration: command=AddVoter server-id=server-092e8925-5f9d-e3fc-8574-7fb986394d47 server-addr=092e8925-5f9d-e3fc-8574-7fb986394d47 servers="[{Suffrage:Voter ID:server-12efa9e7-73df-44ee-d002-b0d0f9fab36c Address:12efa9e7-73df-44ee-d002-b0d0f9fab36c} {Suffrage:Voter ID:server-092e8925-5f9d-e3fc-8574-7fb986394d47 Address:092e8925-5f9d-e3fc-8574-7fb986394d47} {Suffrage:Voter ID:server-4c2696b9-9026-bd1f-42be-7655d6f51c31 Address:4c2696b9-9026-bd1f-42be-7655d6f51c31}]"
2024-12-04T01:32:02.222Z [INFO]  server-12efa9e7-73df-44ee-d002-b0d0f9fab36c: aborting pipeline replication: peer="{Voter server-4c2696b9-9026-bd1f-42be-7655d6f51c31 4c2696b9-9026-bd1f-42be-7655d6f51c31}"
2024-12-04T01:32:02.223Z [INFO]  server-12efa9e7-73df-44ee-d002-b0d0f9fab36c: aborting pipeline replication: peer="{Voter server-092e8925-5f9d-e3fc-8574-7fb986394d47 092e8925-5f9d-e3fc-8574-7fb986394d47}"
--- PASS: TestRaft_AddKnownPeer (0.28s)
=== RUN   TestRaft_RemoveUnknownPeer
2024-12-04T01:32:02.224Z [INFO]  server-a9b74f30-e06e-500b-b090-88e8e852f52f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9b74f30-e06e-500b-b090-88e8e852f52f Address:a9b74f30-e06e-500b-b090-88e8e852f52f} {Suffrage:Voter ID:server-3fc6f426-0584-c12f-f889-f5b39ea7ab9b Address:3fc6f426-0584-c12f-f889-f5b39ea7ab9b} {Suffrage:Voter ID:server-93a1bcd8-d5ff-3c82-442f-642c91f175d0 Address:93a1bcd8-d5ff-3c82-442f-642c91f175d0}]"
2024-12-04T01:32:02.224Z [INFO]  server-3fc6f426-0584-c12f-f889-f5b39ea7ab9b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9b74f30-e06e-500b-b090-88e8e852f52f Address:a9b74f30-e06e-500b-b090-88e8e852f52f} {Suffrage:Voter ID:server-3fc6f426-0584-c12f-f889-f5b39ea7ab9b Address:3fc6f426-0584-c12f-f889-f5b39ea7ab9b} {Suffrage:Voter ID:server-93a1bcd8-d5ff-3c82-442f-642c91f175d0 Address:93a1bcd8-d5ff-3c82-442f-642c91f175d0}]"
2024-12-04T01:32:02.225Z [INFO]  server-93a1bcd8-d5ff-3c82-442f-642c91f175d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a9b74f30-e06e-500b-b090-88e8e852f52f Address:a9b74f30-e06e-500b-b090-88e8e852f52f} {Suffrage:Voter ID:server-3fc6f426-0584-c12f-f889-f5b39ea7ab9b Address:3fc6f426-0584-c12f-f889-f5b39ea7ab9b} {Suffrage:Voter ID:server-93a1bcd8-d5ff-3c82-442f-642c91f175d0 Address:93a1bcd8-d5ff-3c82-442f-642c91f175d0}]"
2024-12-04T01:32:02.225Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:02.225Z [INFO]  server-a9b74f30-e06e-500b-b090-88e8e852f52f: entering follower state: follower="Node at a9b74f30-e06e-500b-b090-88e8e852f52f [Follower]" leader-address= leader-id=
2024-12-04T01:32:02.225Z [INFO]  server-3fc6f426-0584-c12f-f889-f5b39ea7ab9b: entering follower state: follower="Node at 3fc6f426-0584-c12f-f889-f5b39ea7ab9b [Follower]" leader-address= leader-id=
2024-12-04T01:32:02.225Z [INFO]  server-93a1bcd8-d5ff-3c82-442f-642c91f175d0: entering follower state: follower="Node at 93a1bcd8-d5ff-3c82-442f-642c91f175d0 [Follower]" leader-address= leader-id=
2024-12-04T01:32:02.283Z [WARN]  server-3fc6f426-0584-c12f-f889-f5b39ea7ab9b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:02.284Z [INFO]  server-3fc6f426-0584-c12f-f889-f5b39ea7ab9b: entering candidate state: node="Node at 3fc6f426-0584-c12f-f889-f5b39ea7ab9b [Candidate]" term=2
2024-12-04T01:32:02.284Z [INFO]  server-3fc6f426-0584-c12f-f889-f5b39ea7ab9b: election won: term=2 tally=2
2024-12-04T01:32:02.284Z [INFO]  server-3fc6f426-0584-c12f-f889-f5b39ea7ab9b: entering leader state: leader="Node at 3fc6f426-0584-c12f-f889-f5b39ea7ab9b [Leader]"
2024-12-04T01:32:02.284Z [INFO]  server-3fc6f426-0584-c12f-f889-f5b39ea7ab9b: added peer, starting replication: peer=server-a9b74f30-e06e-500b-b090-88e8e852f52f
2024-12-04T01:32:02.284Z [INFO]  server-3fc6f426-0584-c12f-f889-f5b39ea7ab9b: added peer, starting replication: peer=server-93a1bcd8-d5ff-3c82-442f-642c91f175d0
2024-12-04T01:32:02.284Z [INFO]  server-3fc6f426-0584-c12f-f889-f5b39ea7ab9b: pipelining replication: peer="{Voter server-93a1bcd8-d5ff-3c82-442f-642c91f175d0 93a1bcd8-d5ff-3c82-442f-642c91f175d0}"
2024-12-04T01:32:02.284Z [INFO]  server-3fc6f426-0584-c12f-f889-f5b39ea7ab9b: pipelining replication: peer="{Voter server-a9b74f30-e06e-500b-b090-88e8e852f52f a9b74f30-e06e-500b-b090-88e8e852f52f}"
2024-12-04T01:32:02.390Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:02.28489432 +0000 UTC m=+13.116736971 (1 nodes), 59.773241ms from start of poll, 60.227337ms from cluster start. Timeout at 2024-12-04 01:32:02.389935442 +0000 UTC m=+13.221778298, 105.041327ms after stability
2024-12-04T01:32:02.390Z [INFO]  server-3fc6f426-0584-c12f-f889-f5b39ea7ab9b: updating configuration: command=RemoveServer server-id=a84567be-efeb-d9e1-3f28-a103ea028b73 server-addr= servers="[{Suffrage:Voter ID:server-a9b74f30-e06e-500b-b090-88e8e852f52f Address:a9b74f30-e06e-500b-b090-88e8e852f52f} {Suffrage:Voter ID:server-3fc6f426-0584-c12f-f889-f5b39ea7ab9b Address:3fc6f426-0584-c12f-f889-f5b39ea7ab9b} {Suffrage:Voter ID:server-93a1bcd8-d5ff-3c82-442f-642c91f175d0 Address:93a1bcd8-d5ff-3c82-442f-642c91f175d0}]"
2024-12-04T01:32:02.391Z [INFO]  server-3fc6f426-0584-c12f-f889-f5b39ea7ab9b: aborting pipeline replication: peer="{Voter server-93a1bcd8-d5ff-3c82-442f-642c91f175d0 93a1bcd8-d5ff-3c82-442f-642c91f175d0}"
2024-12-04T01:32:02.391Z [INFO]  server-3fc6f426-0584-c12f-f889-f5b39ea7ab9b: aborting pipeline replication: peer="{Voter server-a9b74f30-e06e-500b-b090-88e8e852f52f a9b74f30-e06e-500b-b090-88e8e852f52f}"
--- PASS: TestRaft_RemoveUnknownPeer (0.17s)
=== RUN   TestRaft_SnapshotRestore
2024-12-04T01:32:02.392Z [INFO]  server-ba3d8251-7055-39d3-99dc-5781e6744534: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba3d8251-7055-39d3-99dc-5781e6744534 Address:ba3d8251-7055-39d3-99dc-5781e6744534}]"
2024-12-04T01:32:02.392Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:02.392Z [INFO]  server-ba3d8251-7055-39d3-99dc-5781e6744534: entering follower state: follower="Node at ba3d8251-7055-39d3-99dc-5781e6744534 [Follower]" leader-address= leader-id=
2024-12-04T01:32:02.466Z [WARN]  server-ba3d8251-7055-39d3-99dc-5781e6744534: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:02.466Z [INFO]  server-ba3d8251-7055-39d3-99dc-5781e6744534: entering candidate state: node="Node at ba3d8251-7055-39d3-99dc-5781e6744534 [Candidate]" term=2
2024-12-04T01:32:02.466Z [INFO]  server-ba3d8251-7055-39d3-99dc-5781e6744534: election won: term=2 tally=1
2024-12-04T01:32:02.466Z [INFO]  server-ba3d8251-7055-39d3-99dc-5781e6744534: entering leader state: leader="Node at ba3d8251-7055-39d3-99dc-5781e6744534 [Leader]"
2024-12-04T01:32:02.574Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:02.466622637 +0000 UTC m=+13.298465249 (1 nodes), 73.746707ms from start of poll, 73.930148ms from cluster start. Timeout at 2024-12-04 01:32:02.5716281 +0000 UTC m=+13.403470722, 105.005473ms after stability
2024-12-04T01:32:02.579Z [INFO]  server-ba3d8251-7055-39d3-99dc-5781e6744534: starting snapshot up to: index=102
2024-12-04T01:32:02.579Z [INFO]  creating new snapshot: path=/tmp/raft4176043144/snapshots/2-102-1733275922579.tmp
2024-12-04T01:32:02.581Z [INFO]  server-ba3d8251-7055-39d3-99dc-5781e6744534: compacting logs: from=1 to=92
2024-12-04T01:32:02.581Z [INFO]  server-ba3d8251-7055-39d3-99dc-5781e6744534: snapshot complete up to: index=102
2024-12-04T01:32:02.581Z [INFO]  server-ba3d8251-7055-39d3-99dc-5781e6744534: starting restore from snapshot: id=2-102-1733275922579 last-index=102 last-term=2 size-in-bytes=693
2024-12-04T01:32:02.581Z [INFO]  server-ba3d8251-7055-39d3-99dc-5781e6744534: snapshot restore progress: id=2-102-1733275922579 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2024-12-04T01:32:02.581Z [INFO]  server-ba3d8251-7055-39d3-99dc-5781e6744534: restored from snapshot: id=2-102-1733275922579 last-index=102 last-term=2 size-in-bytes=693
2024-12-04T01:32:02.581Z [INFO]  server-ba3d8251-7055-39d3-99dc-5781e6744534: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba3d8251-7055-39d3-99dc-5781e6744534 Address:ba3d8251-7055-39d3-99dc-5781e6744534}]"
--- PASS: TestRaft_SnapshotRestore (0.19s)
=== RUN   TestRaft_RestoreSnapshotOnStartup_Monotonic
2024-12-04T01:32:02.587Z [INFO]  server-73498094-afff-db56-8a00-92e7fbb7cbf2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73498094-afff-db56-8a00-92e7fbb7cbf2 Address:73498094-afff-db56-8a00-92e7fbb7cbf2}]"
2024-12-04T01:32:02.587Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:02.587Z [INFO]  server-73498094-afff-db56-8a00-92e7fbb7cbf2: entering follower state: follower="Node at 73498094-afff-db56-8a00-92e7fbb7cbf2 [Follower]" leader-address= leader-id=
2024-12-04T01:32:02.642Z [WARN]  server-73498094-afff-db56-8a00-92e7fbb7cbf2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:02.642Z [INFO]  server-73498094-afff-db56-8a00-92e7fbb7cbf2: entering candidate state: node="Node at 73498094-afff-db56-8a00-92e7fbb7cbf2 [Candidate]" term=2
2024-12-04T01:32:02.642Z [INFO]  server-73498094-afff-db56-8a00-92e7fbb7cbf2: election won: term=2 tally=1
2024-12-04T01:32:02.642Z [INFO]  server-73498094-afff-db56-8a00-92e7fbb7cbf2: entering leader state: leader="Node at 73498094-afff-db56-8a00-92e7fbb7cbf2 [Leader]"
2024-12-04T01:32:02.754Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:02.643112033 +0000 UTC m=+13.474954666 (1 nodes), 55.837485ms from start of poll, 56.070913ms from cluster start. Timeout at 2024-12-04 01:32:02.74812647 +0000 UTC m=+13.579969272, 105.014606ms after stability
2024-12-04T01:32:02.758Z [INFO]  server-73498094-afff-db56-8a00-92e7fbb7cbf2: starting snapshot up to: index=102
2024-12-04T01:32:02.758Z [INFO]  creating new snapshot: path=/tmp/raft565852466/snapshots/2-102-1733275922758.tmp
2024-12-04T01:32:02.759Z [INFO]  server-73498094-afff-db56-8a00-92e7fbb7cbf2: compacting logs: from=1 to=92
2024-12-04T01:32:02.759Z [INFO]  server-73498094-afff-db56-8a00-92e7fbb7cbf2: snapshot complete up to: index=102
2024-12-04T01:32:02.760Z [INFO]  server-73498094-afff-db56-8a00-92e7fbb7cbf2: starting restore from snapshot: id=2-102-1733275922758 last-index=102 last-term=2 size-in-bytes=693
2024-12-04T01:32:02.760Z [INFO]  server-73498094-afff-db56-8a00-92e7fbb7cbf2: snapshot restore progress: id=2-102-1733275922758 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2024-12-04T01:32:02.760Z [INFO]  server-73498094-afff-db56-8a00-92e7fbb7cbf2: restored from snapshot: id=2-102-1733275922758 last-index=102 last-term=2 size-in-bytes=693
2024-12-04T01:32:02.760Z [INFO]  server-73498094-afff-db56-8a00-92e7fbb7cbf2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-73498094-afff-db56-8a00-92e7fbb7cbf2 Address:73498094-afff-db56-8a00-92e7fbb7cbf2}]"
--- PASS: TestRaft_RestoreSnapshotOnStartup_Monotonic (0.17s)
=== RUN   TestRaft_SnapshotRestore_Progress
2024-12-04T01:32:02.761Z [INFO]  server-2cea1cfc-ad88-9cd3-1ee1-43ce12b08c91: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2cea1cfc-ad88-9cd3-1ee1-43ce12b08c91 Address:2cea1cfc-ad88-9cd3-1ee1-43ce12b08c91}]"
2024-12-04T01:32:02.761Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:02.761Z [INFO]  server-2cea1cfc-ad88-9cd3-1ee1-43ce12b08c91: entering follower state: follower="Node at 2cea1cfc-ad88-9cd3-1ee1-43ce12b08c91 [Follower]" leader-address= leader-id=
2024-12-04T01:32:02.854Z [WARN]  server-2cea1cfc-ad88-9cd3-1ee1-43ce12b08c91: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:02.854Z [INFO]  server-2cea1cfc-ad88-9cd3-1ee1-43ce12b08c91: entering candidate state: node="Node at 2cea1cfc-ad88-9cd3-1ee1-43ce12b08c91 [Candidate]" term=2
2024-12-04T01:32:02.854Z [INFO]  server-2cea1cfc-ad88-9cd3-1ee1-43ce12b08c91: election won: term=2 tally=1
2024-12-04T01:32:02.854Z [INFO]  server-2cea1cfc-ad88-9cd3-1ee1-43ce12b08c91: entering leader state: leader="Node at 2cea1cfc-ad88-9cd3-1ee1-43ce12b08c91 [Leader]"
2024-12-04T01:32:02.962Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:02.854626476 +0000 UTC m=+13.686469444 (1 nodes), 93.309437ms from start of poll, 93.484127ms from cluster start. Timeout at 2024-12-04 01:32:02.959640424 +0000 UTC m=+13.791483504, 105.01406ms after stability
2024-12-04T01:32:02.968Z [INFO]  server-2cea1cfc-ad88-9cd3-1ee1-43ce12b08c91: starting snapshot up to: index=102
2024-12-04T01:32:02.968Z [INFO]  creating new snapshot: path=/tmp/raft4046980649/snapshots/2-102-1733275922968.tmp
2024-12-04T01:32:02.978Z [INFO]  server-2cea1cfc-ad88-9cd3-1ee1-43ce12b08c91: compacting logs: from=1 to=92
2024-12-04T01:32:02.978Z [INFO]  server-2cea1cfc-ad88-9cd3-1ee1-43ce12b08c91: snapshot complete up to: index=102
--- PASS: TestRaft_SnapshotRestore_Progress (0.22s)
=== RUN   TestRaft_NoRestoreOnStart
2024-12-04T01:32:02.986Z [INFO]  server-185082fb-1bc2-7680-62d8-31b92692d490: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-185082fb-1bc2-7680-62d8-31b92692d490 Address:185082fb-1bc2-7680-62d8-31b92692d490}]"
2024-12-04T01:32:02.986Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:02.990Z [INFO]  server-185082fb-1bc2-7680-62d8-31b92692d490: entering follower state: follower="Node at 185082fb-1bc2-7680-62d8-31b92692d490 [Follower]" leader-address= leader-id=
2024-12-04T01:32:03.046Z [WARN]  server-185082fb-1bc2-7680-62d8-31b92692d490: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:03.046Z [INFO]  server-185082fb-1bc2-7680-62d8-31b92692d490: entering candidate state: node="Node at 185082fb-1bc2-7680-62d8-31b92692d490 [Candidate]" term=2
2024-12-04T01:32:03.046Z [INFO]  server-185082fb-1bc2-7680-62d8-31b92692d490: election won: term=2 tally=1
2024-12-04T01:32:03.046Z [INFO]  server-185082fb-1bc2-7680-62d8-31b92692d490: entering leader state: leader="Node at 185082fb-1bc2-7680-62d8-31b92692d490 [Leader]"
2024-12-04T01:32:03.154Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:03.046704547 +0000 UTC m=+13.878547165 (1 nodes), 59.736796ms from start of poll, 66.17094ms from cluster start. Timeout at 2024-12-04 01:32:03.151719641 +0000 UTC m=+13.983562576, 105.015411ms after stability
2024-12-04T01:32:03.160Z [INFO]  server-185082fb-1bc2-7680-62d8-31b92692d490: starting snapshot up to: index=102
2024-12-04T01:32:03.160Z [INFO]  creating new snapshot: path=/tmp/raft3162575913/snapshots/2-102-1733275923160.tmp
2024-12-04T01:32:03.162Z [INFO]  server-185082fb-1bc2-7680-62d8-31b92692d490: compacting logs: from=1 to=92
2024-12-04T01:32:03.162Z [INFO]  server-185082fb-1bc2-7680-62d8-31b92692d490: snapshot complete up to: index=102
2024-12-04T01:32:03.162Z [INFO]  server-185082fb-1bc2-7680-62d8-31b92692d490: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-185082fb-1bc2-7680-62d8-31b92692d490 Address:185082fb-1bc2-7680-62d8-31b92692d490}]"
--- PASS: TestRaft_NoRestoreOnStart (0.18s)
=== RUN   TestRaft_SnapshotRestore_PeerChange
    raft_test.go:1266: Skipping unreliable test
--- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s)
=== RUN   TestRaft_AutoSnapshot
2024-12-04T01:32:03.163Z [INFO]  server-3370fc71-4e6e-9133-ff75-a1d33bcdce48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3370fc71-4e6e-9133-ff75-a1d33bcdce48 Address:3370fc71-4e6e-9133-ff75-a1d33bcdce48}]"
2024-12-04T01:32:03.163Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:03.163Z [INFO]  server-185082fb-1bc2-7680-62d8-31b92692d490: entering follower state: follower="Node at 185082fb-1bc2-7680-62d8-31b92692d490 [Follower]" leader-address= leader-id=
2024-12-04T01:32:03.163Z [INFO]  server-3370fc71-4e6e-9133-ff75-a1d33bcdce48: entering follower state: follower="Node at 3370fc71-4e6e-9133-ff75-a1d33bcdce48 [Follower]" leader-address= leader-id=
2024-12-04T01:32:03.218Z [WARN]  server-3370fc71-4e6e-9133-ff75-a1d33bcdce48: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:03.218Z [INFO]  server-3370fc71-4e6e-9133-ff75-a1d33bcdce48: entering candidate state: node="Node at 3370fc71-4e6e-9133-ff75-a1d33bcdce48 [Candidate]" term=2
2024-12-04T01:32:03.218Z [INFO]  server-3370fc71-4e6e-9133-ff75-a1d33bcdce48: election won: term=2 tally=1
2024-12-04T01:32:03.218Z [INFO]  server-3370fc71-4e6e-9133-ff75-a1d33bcdce48: entering leader state: leader="Node at 3370fc71-4e6e-9133-ff75-a1d33bcdce48 [Leader]"
2024-12-04T01:32:03.250Z [WARN]  server-185082fb-1bc2-7680-62d8-31b92692d490: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:03.250Z [INFO]  server-185082fb-1bc2-7680-62d8-31b92692d490: entering candidate state: node="Node at 185082fb-1bc2-7680-62d8-31b92692d490 [Candidate]" term=3
2024-12-04T01:32:03.250Z [INFO]  server-185082fb-1bc2-7680-62d8-31b92692d490: election won: term=3 tally=1
2024-12-04T01:32:03.250Z [INFO]  server-185082fb-1bc2-7680-62d8-31b92692d490: entering leader state: leader="Node at 185082fb-1bc2-7680-62d8-31b92692d490 [Leader]"
2024-12-04T01:32:03.330Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:03.218683741 +0000 UTC m=+14.050526397 (1 nodes), 55.497623ms from start of poll, 55.661751ms from cluster start. Timeout at 2024-12-04 01:32:03.32370084 +0000 UTC m=+14.155543549, 105.017152ms after stability
2024-12-04T01:32:03.346Z [INFO]  server-3370fc71-4e6e-9133-ff75-a1d33bcdce48: starting snapshot up to: index=102
2024-12-04T01:32:03.346Z [INFO]  creating new snapshot: path=/tmp/raft44492217/snapshots/2-102-1733275923346.tmp
2024-12-04T01:32:03.347Z [INFO]  server-3370fc71-4e6e-9133-ff75-a1d33bcdce48: compacting logs: from=1 to=92
2024-12-04T01:32:03.347Z [INFO]  server-3370fc71-4e6e-9133-ff75-a1d33bcdce48: snapshot complete up to: index=102
--- PASS: TestRaft_AutoSnapshot (0.28s)
=== RUN   TestRaft_UserSnapshot
2024-12-04T01:32:03.444Z [INFO]  server-3a5e4385-97ac-df47-9025-3a4c4e9d2e0d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3a5e4385-97ac-df47-9025-3a4c4e9d2e0d Address:3a5e4385-97ac-df47-9025-3a4c4e9d2e0d}]"
2024-12-04T01:32:03.444Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:03.444Z [INFO]  server-3a5e4385-97ac-df47-9025-3a4c4e9d2e0d: entering follower state: follower="Node at 3a5e4385-97ac-df47-9025-3a4c4e9d2e0d [Follower]" leader-address= leader-id=
2024-12-04T01:32:03.530Z [WARN]  server-3a5e4385-97ac-df47-9025-3a4c4e9d2e0d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:03.530Z [INFO]  server-3a5e4385-97ac-df47-9025-3a4c4e9d2e0d: entering candidate state: node="Node at 3a5e4385-97ac-df47-9025-3a4c4e9d2e0d [Candidate]" term=2
2024-12-04T01:32:03.530Z [INFO]  server-3a5e4385-97ac-df47-9025-3a4c4e9d2e0d: election won: term=2 tally=1
2024-12-04T01:32:03.530Z [INFO]  server-3a5e4385-97ac-df47-9025-3a4c4e9d2e0d: entering leader state: leader="Node at 3a5e4385-97ac-df47-9025-3a4c4e9d2e0d [Leader]"
2024-12-04T01:32:03.638Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:03.530605774 +0000 UTC m=+14.362448397 (1 nodes), 86.220419ms from start of poll, 86.495176ms from cluster start. Timeout at 2024-12-04 01:32:03.63561007 +0000 UTC m=+14.467452737, 105.00434ms after stability
2024-12-04T01:32:03.638Z [ERROR] server-3a5e4385-97ac-df47-9025-3a4c4e9d2e0d: failed to take snapshot: error="nothing new to snapshot"
2024-12-04T01:32:03.639Z [INFO]  server-3a5e4385-97ac-df47-9025-3a4c4e9d2e0d: starting snapshot up to: index=12
2024-12-04T01:32:03.639Z [INFO]  creating new snapshot: path=/tmp/raft2689298857/snapshots/2-12-1733275923639.tmp
2024-12-04T01:32:03.640Z [INFO]  server-3a5e4385-97ac-df47-9025-3a4c4e9d2e0d: compacting logs: from=1 to=2
2024-12-04T01:32:03.640Z [INFO]  server-3a5e4385-97ac-df47-9025-3a4c4e9d2e0d: snapshot complete up to: index=12
--- PASS: TestRaft_UserSnapshot (0.20s)
=== RUN   TestRaft_UserRestore
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false
2024-12-04T01:32:03.645Z [INFO]  server-b80cfee2-6c73-6a12-dd15-e6c3f64b4e27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b80cfee2-6c73-6a12-dd15-e6c3f64b4e27 Address:b80cfee2-6c73-6a12-dd15-e6c3f64b4e27} {Suffrage:Voter ID:server-a4c57718-635d-bdbf-f703-133c723fbb42 Address:a4c57718-635d-bdbf-f703-133c723fbb42} {Suffrage:Voter ID:server-058788d3-d45f-6aa0-0c52-9a46dde3c102 Address:058788d3-d45f-6aa0-0c52-9a46dde3c102}]"
2024-12-04T01:32:03.645Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b80cfee2-6c73-6a12-dd15-e6c3f64b4e27 Address:b80cfee2-6c73-6a12-dd15-e6c3f64b4e27} {Suffrage:Voter ID:server-a4c57718-635d-bdbf-f703-133c723fbb42 Address:a4c57718-635d-bdbf-f703-133c723fbb42} {Suffrage:Voter ID:server-058788d3-d45f-6aa0-0c52-9a46dde3c102 Address:058788d3-d45f-6aa0-0c52-9a46dde3c102}]"
2024-12-04T01:32:03.645Z [INFO]  server-058788d3-d45f-6aa0-0c52-9a46dde3c102: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b80cfee2-6c73-6a12-dd15-e6c3f64b4e27 Address:b80cfee2-6c73-6a12-dd15-e6c3f64b4e27} {Suffrage:Voter ID:server-a4c57718-635d-bdbf-f703-133c723fbb42 Address:a4c57718-635d-bdbf-f703-133c723fbb42} {Suffrage:Voter ID:server-058788d3-d45f-6aa0-0c52-9a46dde3c102 Address:058788d3-d45f-6aa0-0c52-9a46dde3c102}]"
2024-12-04T01:32:03.645Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:03.645Z [INFO]  server-b80cfee2-6c73-6a12-dd15-e6c3f64b4e27: entering follower state: follower="Node at b80cfee2-6c73-6a12-dd15-e6c3f64b4e27 [Follower]" leader-address= leader-id=
2024-12-04T01:32:03.645Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: entering follower state: follower="Node at a4c57718-635d-bdbf-f703-133c723fbb42 [Follower]" leader-address= leader-id=
2024-12-04T01:32:03.645Z [INFO]  server-058788d3-d45f-6aa0-0c52-9a46dde3c102: entering follower state: follower="Node at 058788d3-d45f-6aa0-0c52-9a46dde3c102 [Follower]" leader-address= leader-id=
2024-12-04T01:32:04.190Z [WARN]  server-a4c57718-635d-bdbf-f703-133c723fbb42: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:04.190Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: entering candidate state: node="Node at a4c57718-635d-bdbf-f703-133c723fbb42 [Candidate]" term=2
2024-12-04T01:32:04.190Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: election won: term=2 tally=2
2024-12-04T01:32:04.190Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: entering leader state: leader="Node at a4c57718-635d-bdbf-f703-133c723fbb42 [Leader]"
2024-12-04T01:32:04.190Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: added peer, starting replication: peer=server-b80cfee2-6c73-6a12-dd15-e6c3f64b4e27
2024-12-04T01:32:04.190Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: added peer, starting replication: peer=server-058788d3-d45f-6aa0-0c52-9a46dde3c102
2024-12-04T01:32:04.191Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: pipelining replication: peer="{Voter server-058788d3-d45f-6aa0-0c52-9a46dde3c102 058788d3-d45f-6aa0-0c52-9a46dde3c102}"
2024-12-04T01:32:04.191Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: pipelining replication: peer="{Voter server-b80cfee2-6c73-6a12-dd15-e6c3f64b4e27 b80cfee2-6c73-6a12-dd15-e6c3f64b4e27}"
2024-12-04T01:32:05.198Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:04.191625754 +0000 UTC m=+15.023468469 (1 nodes), 545.970165ms from start of poll, 546.510826ms from cluster start. Timeout at 2024-12-04 01:32:05.196629489 +0000 UTC m=+16.028472545, 1.005004076s after stability
2024-12-04T01:32:05.199Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: starting snapshot up to: index=12
2024-12-04T01:32:05.199Z [INFO]  creating new snapshot: path=/tmp/raft332580266/snapshots/2-12-1733275925199.tmp
2024-12-04T01:32:05.201Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: snapshot complete up to: index=12
2024-12-04T01:32:05.203Z [INFO]  creating new snapshot: path=/tmp/raft332580266/snapshots/2-23-1733275925203.tmp
2024-12-04T01:32:05.204Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: copied to local snapshot: bytes=71
2024-12-04T01:32:05.204Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: snapshot restore progress: id=2-23-1733275925203 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:05.204Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: restored user snapshot: index=23
2024-12-04T01:32:05.204Z [ERROR] server-a4c57718-635d-bdbf-f703-133c723fbb42: failed to get log: index=23 error="log not found"
2024-12-04T01:32:05.204Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: aborting pipeline replication: peer="{Voter server-058788d3-d45f-6aa0-0c52-9a46dde3c102 058788d3-d45f-6aa0-0c52-9a46dde3c102}"
2024-12-04T01:32:05.204Z [ERROR] server-a4c57718-635d-bdbf-f703-133c723fbb42: failed to get log: index=23 error="log not found"
2024-12-04T01:32:05.204Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: aborting pipeline replication: peer="{Voter server-b80cfee2-6c73-6a12-dd15-e6c3f64b4e27 b80cfee2-6c73-6a12-dd15-e6c3f64b4e27}"
2024-12-04T01:32:05.214Z [ERROR] server-a4c57718-635d-bdbf-f703-133c723fbb42: failed to get log: index=23 error="log not found"
2024-12-04T01:32:05.215Z [INFO]  creating new snapshot: path=/tmp/raft3532699377/snapshots/2-23-1733275925215.tmp
2024-12-04T01:32:05.215Z [INFO]  server-b80cfee2-6c73-6a12-dd15-e6c3f64b4e27: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:05.218Z [ERROR] server-a4c57718-635d-bdbf-f703-133c723fbb42: failed to get log: index=23 error="log not found"
2024-12-04T01:32:05.226Z [INFO]  server-b80cfee2-6c73-6a12-dd15-e6c3f64b4e27: copied to local snapshot: bytes=71
2024-12-04T01:32:05.238Z [INFO]  creating new snapshot: path=/tmp/raft3377418910/snapshots/2-23-1733275925238.tmp
2024-12-04T01:32:05.238Z [INFO]  server-058788d3-d45f-6aa0-0c52-9a46dde3c102: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:05.239Z [INFO]  server-058788d3-d45f-6aa0-0c52-9a46dde3c102: copied to local snapshot: bytes=71
2024-12-04T01:32:05.239Z [INFO]  server-058788d3-d45f-6aa0-0c52-9a46dde3c102: snapshot restore progress: id=2-23-1733275925238 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:05.239Z [INFO]  server-058788d3-d45f-6aa0-0c52-9a46dde3c102: Installed remote snapshot
2024-12-04T01:32:05.239Z [WARN]  server-058788d3-d45f-6aa0-0c52-9a46dde3c102: skipping application of old log: index=22
2024-12-04T01:32:05.239Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: pipelining replication: peer="{Voter server-058788d3-d45f-6aa0-0c52-9a46dde3c102 058788d3-d45f-6aa0-0c52-9a46dde3c102}"
2024-12-04T01:32:05.239Z [INFO]  server-b80cfee2-6c73-6a12-dd15-e6c3f64b4e27: snapshot restore progress: id=2-23-1733275925215 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:05.240Z [INFO]  server-b80cfee2-6c73-6a12-dd15-e6c3f64b4e27: Installed remote snapshot
2024-12-04T01:32:05.240Z [WARN]  server-b80cfee2-6c73-6a12-dd15-e6c3f64b4e27: skipping application of old log: index=22
2024-12-04T01:32:05.240Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: pipelining replication: peer="{Voter server-b80cfee2-6c73-6a12-dd15-e6c3f64b4e27 b80cfee2-6c73-6a12-dd15-e6c3f64b4e27}"
2024-12-04T01:32:05.240Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:06.246Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:05.240239311 +0000 UTC m=+16.072081875 (2 nodes), 1.945µs from start of poll, 1.595124232s from cluster start. Timeout at 2024-12-04 01:32:06.245259175 +0000 UTC m=+17.077102362, 1.005020487s after stability
2024-12-04T01:32:06.259Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: aborting pipeline replication: peer="{Voter server-058788d3-d45f-6aa0-0c52-9a46dde3c102 058788d3-d45f-6aa0-0c52-9a46dde3c102}"
2024-12-04T01:32:06.260Z [INFO]  server-a4c57718-635d-bdbf-f703-133c723fbb42: aborting pipeline replication: peer="{Voter server-b80cfee2-6c73-6a12-dd15-e6c3f64b4e27 b80cfee2-6c73-6a12-dd15-e6c3f64b4e27}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false
2024-12-04T01:32:06.262Z [INFO]  server-7c11fe86-eeb1-1455-5b50-8b9279650ee4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c11fe86-eeb1-1455-5b50-8b9279650ee4 Address:7c11fe86-eeb1-1455-5b50-8b9279650ee4} {Suffrage:Voter ID:server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4 Address:f8215e1e-d092-5017-ed73-fac8ae4ca0a4} {Suffrage:Voter ID:server-43a2a824-0129-1a54-57c6-6c2a87ca8105 Address:43a2a824-0129-1a54-57c6-6c2a87ca8105}]"
2024-12-04T01:32:06.263Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c11fe86-eeb1-1455-5b50-8b9279650ee4 Address:7c11fe86-eeb1-1455-5b50-8b9279650ee4} {Suffrage:Voter ID:server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4 Address:f8215e1e-d092-5017-ed73-fac8ae4ca0a4} {Suffrage:Voter ID:server-43a2a824-0129-1a54-57c6-6c2a87ca8105 Address:43a2a824-0129-1a54-57c6-6c2a87ca8105}]"
2024-12-04T01:32:06.263Z [INFO]  server-43a2a824-0129-1a54-57c6-6c2a87ca8105: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7c11fe86-eeb1-1455-5b50-8b9279650ee4 Address:7c11fe86-eeb1-1455-5b50-8b9279650ee4} {Suffrage:Voter ID:server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4 Address:f8215e1e-d092-5017-ed73-fac8ae4ca0a4} {Suffrage:Voter ID:server-43a2a824-0129-1a54-57c6-6c2a87ca8105 Address:43a2a824-0129-1a54-57c6-6c2a87ca8105}]"
2024-12-04T01:32:06.263Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:06.263Z [INFO]  server-7c11fe86-eeb1-1455-5b50-8b9279650ee4: entering follower state: follower="Node at 7c11fe86-eeb1-1455-5b50-8b9279650ee4 [Follower]" leader-address= leader-id=
2024-12-04T01:32:06.263Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: entering follower state: follower="Node at f8215e1e-d092-5017-ed73-fac8ae4ca0a4 [Follower]" leader-address= leader-id=
2024-12-04T01:32:06.263Z [INFO]  server-43a2a824-0129-1a54-57c6-6c2a87ca8105: entering follower state: follower="Node at 43a2a824-0129-1a54-57c6-6c2a87ca8105 [Follower]" leader-address= leader-id=
2024-12-04T01:32:06.794Z [WARN]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:06.794Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: entering candidate state: node="Node at f8215e1e-d092-5017-ed73-fac8ae4ca0a4 [Candidate]" term=2
2024-12-04T01:32:06.794Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: election won: term=2 tally=2
2024-12-04T01:32:06.794Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: entering leader state: leader="Node at f8215e1e-d092-5017-ed73-fac8ae4ca0a4 [Leader]"
2024-12-04T01:32:06.794Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: added peer, starting replication: peer=server-7c11fe86-eeb1-1455-5b50-8b9279650ee4
2024-12-04T01:32:06.794Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: added peer, starting replication: peer=server-43a2a824-0129-1a54-57c6-6c2a87ca8105
2024-12-04T01:32:06.795Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: pipelining replication: peer="{Voter server-43a2a824-0129-1a54-57c6-6c2a87ca8105 43a2a824-0129-1a54-57c6-6c2a87ca8105}"
2024-12-04T01:32:06.795Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: pipelining replication: peer="{Voter server-7c11fe86-eeb1-1455-5b50-8b9279650ee4 7c11fe86-eeb1-1455-5b50-8b9279650ee4}"
2024-12-04T01:32:07.807Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:06.795401836 +0000 UTC m=+17.627244463 (1 nodes), 532.117662ms from start of poll, 532.686735ms from cluster start. Timeout at 2024-12-04 01:32:07.800422793 +0000 UTC m=+18.632265430, 1.005020967s after stability
2024-12-04T01:32:07.819Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: starting snapshot up to: index=12
2024-12-04T01:32:07.819Z [INFO]  creating new snapshot: path=/tmp/raft483219704/snapshots/2-12-1733275927819.tmp
2024-12-04T01:32:07.820Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: snapshot complete up to: index=12
2024-12-04T01:32:07.824Z [INFO]  creating new snapshot: path=/tmp/raft483219704/snapshots/2-23-1733275927824.tmp
2024-12-04T01:32:07.825Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: copied to local snapshot: bytes=71
2024-12-04T01:32:07.825Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: snapshot restore progress: id=2-23-1733275927824 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:07.825Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: removing all old logs from log store
2024-12-04T01:32:07.825Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: compacting logs: from=1 to=22
2024-12-04T01:32:07.825Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: restored user snapshot: index=23
2024-12-04T01:32:07.825Z [ERROR] server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: failed to get log: index=22 error="log not found"
2024-12-04T01:32:07.825Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: aborting pipeline replication: peer="{Voter server-43a2a824-0129-1a54-57c6-6c2a87ca8105 43a2a824-0129-1a54-57c6-6c2a87ca8105}"
2024-12-04T01:32:07.826Z [ERROR] server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: failed to get log: index=22 error="log not found"
2024-12-04T01:32:07.826Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: aborting pipeline replication: peer="{Voter server-7c11fe86-eeb1-1455-5b50-8b9279650ee4 7c11fe86-eeb1-1455-5b50-8b9279650ee4}"
2024-12-04T01:32:07.834Z [ERROR] server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: failed to get log: index=22 error="log not found"
2024-12-04T01:32:07.835Z [INFO]  creating new snapshot: path=/tmp/raft1854873004/snapshots/2-23-1733275927835.tmp
2024-12-04T01:32:07.835Z [INFO]  server-7c11fe86-eeb1-1455-5b50-8b9279650ee4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:07.836Z [INFO]  server-7c11fe86-eeb1-1455-5b50-8b9279650ee4: copied to local snapshot: bytes=71
2024-12-04T01:32:07.836Z [INFO]  server-7c11fe86-eeb1-1455-5b50-8b9279650ee4: snapshot restore progress: id=2-23-1733275927835 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:07.836Z [INFO]  server-7c11fe86-eeb1-1455-5b50-8b9279650ee4: removing all old logs from log store
2024-12-04T01:32:07.836Z [INFO]  server-7c11fe86-eeb1-1455-5b50-8b9279650ee4: compacting logs: from=1 to=22
2024-12-04T01:32:07.836Z [INFO]  server-7c11fe86-eeb1-1455-5b50-8b9279650ee4: Installed remote snapshot
2024-12-04T01:32:07.836Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: pipelining replication: peer="{Voter server-7c11fe86-eeb1-1455-5b50-8b9279650ee4 7c11fe86-eeb1-1455-5b50-8b9279650ee4}"
2024-12-04T01:32:07.837Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:07.838Z [ERROR] server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: failed to get log: index=22 error="log not found"
2024-12-04T01:32:07.839Z [INFO]  creating new snapshot: path=/tmp/raft2555374835/snapshots/2-23-1733275927839.tmp
2024-12-04T01:32:07.839Z [INFO]  server-43a2a824-0129-1a54-57c6-6c2a87ca8105: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:07.840Z [INFO]  server-43a2a824-0129-1a54-57c6-6c2a87ca8105: copied to local snapshot: bytes=71
2024-12-04T01:32:07.840Z [INFO]  server-43a2a824-0129-1a54-57c6-6c2a87ca8105: snapshot restore progress: id=2-23-1733275927839 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:07.840Z [INFO]  server-43a2a824-0129-1a54-57c6-6c2a87ca8105: removing all old logs from log store
2024-12-04T01:32:07.840Z [INFO]  server-43a2a824-0129-1a54-57c6-6c2a87ca8105: compacting logs: from=1 to=22
2024-12-04T01:32:07.840Z [INFO]  server-43a2a824-0129-1a54-57c6-6c2a87ca8105: Installed remote snapshot
2024-12-04T01:32:07.840Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: pipelining replication: peer="{Voter server-43a2a824-0129-1a54-57c6-6c2a87ca8105 43a2a824-0129-1a54-57c6-6c2a87ca8105}"
2024-12-04T01:32:08.854Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:07.837068389 +0000 UTC m=+18.668911027 (2 nodes), 2.291µs from start of poll, 1.574353299s from cluster start. Timeout at 2024-12-04 01:32:08.842083684 +0000 UTC m=+19.673926526, 1.005015499s after stability
2024-12-04T01:32:08.878Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: aborting pipeline replication: peer="{Voter server-43a2a824-0129-1a54-57c6-6c2a87ca8105 43a2a824-0129-1a54-57c6-6c2a87ca8105}"
2024-12-04T01:32:08.878Z [INFO]  server-f8215e1e-d092-5017-ed73-fac8ae4ca0a4: aborting pipeline replication: peer="{Voter server-7c11fe86-eeb1-1455-5b50-8b9279650ee4 7c11fe86-eeb1-1455-5b50-8b9279650ee4}"
=== RUN   TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true
2024-12-04T01:32:08.881Z [INFO]  server-afeaa4fc-e681-4e2c-d090-d1a3f9833741: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afeaa4fc-e681-4e2c-d090-d1a3f9833741 Address:afeaa4fc-e681-4e2c-d090-d1a3f9833741} {Suffrage:Voter ID:server-24b6ee03-c208-b54d-7dad-37d465570640 Address:24b6ee03-c208-b54d-7dad-37d465570640} {Suffrage:Voter ID:server-d6521f80-eacf-76c4-4394-188e5d8e93be Address:d6521f80-eacf-76c4-4394-188e5d8e93be}]"
2024-12-04T01:32:08.881Z [INFO]  server-24b6ee03-c208-b54d-7dad-37d465570640: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afeaa4fc-e681-4e2c-d090-d1a3f9833741 Address:afeaa4fc-e681-4e2c-d090-d1a3f9833741} {Suffrage:Voter ID:server-24b6ee03-c208-b54d-7dad-37d465570640 Address:24b6ee03-c208-b54d-7dad-37d465570640} {Suffrage:Voter ID:server-d6521f80-eacf-76c4-4394-188e5d8e93be Address:d6521f80-eacf-76c4-4394-188e5d8e93be}]"
2024-12-04T01:32:08.881Z [INFO]  server-d6521f80-eacf-76c4-4394-188e5d8e93be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afeaa4fc-e681-4e2c-d090-d1a3f9833741 Address:afeaa4fc-e681-4e2c-d090-d1a3f9833741} {Suffrage:Voter ID:server-24b6ee03-c208-b54d-7dad-37d465570640 Address:24b6ee03-c208-b54d-7dad-37d465570640} {Suffrage:Voter ID:server-d6521f80-eacf-76c4-4394-188e5d8e93be Address:d6521f80-eacf-76c4-4394-188e5d8e93be}]"
2024-12-04T01:32:08.881Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:08.881Z [INFO]  server-afeaa4fc-e681-4e2c-d090-d1a3f9833741: entering follower state: follower="Node at afeaa4fc-e681-4e2c-d090-d1a3f9833741 [Follower]" leader-address= leader-id=
2024-12-04T01:32:08.881Z [INFO]  server-24b6ee03-c208-b54d-7dad-37d465570640: entering follower state: follower="Node at 24b6ee03-c208-b54d-7dad-37d465570640 [Follower]" leader-address= leader-id=
2024-12-04T01:32:08.881Z [INFO]  server-d6521f80-eacf-76c4-4394-188e5d8e93be: entering follower state: follower="Node at d6521f80-eacf-76c4-4394-188e5d8e93be [Follower]" leader-address= leader-id=
2024-12-04T01:32:09.630Z [WARN]  server-d6521f80-eacf-76c4-4394-188e5d8e93be: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:09.630Z [INFO]  server-d6521f80-eacf-76c4-4394-188e5d8e93be: entering candidate state: node="Node at d6521f80-eacf-76c4-4394-188e5d8e93be [Candidate]" term=2
2024-12-04T01:32:09.630Z [INFO]  server-d6521f80-eacf-76c4-4394-188e5d8e93be: election won: term=2 tally=2
2024-12-04T01:32:09.630Z [INFO]  server-d6521f80-eacf-76c4-4394-188e5d8e93be: entering leader state: leader="Node at d6521f80-eacf-76c4-4394-188e5d8e93be [Leader]"
2024-12-04T01:32:09.630Z [INFO]  server-d6521f80-eacf-76c4-4394-188e5d8e93be: added peer, starting replication: peer=server-afeaa4fc-e681-4e2c-d090-d1a3f9833741
2024-12-04T01:32:09.630Z [INFO]  server-d6521f80-eacf-76c4-4394-188e5d8e93be: added peer, starting replication: peer=server-24b6ee03-c208-b54d-7dad-37d465570640
2024-12-04T01:32:09.631Z [INFO]  server-d6521f80-eacf-76c4-4394-188e5d8e93be: pipelining replication: peer="{Voter server-24b6ee03-c208-b54d-7dad-37d465570640 24b6ee03-c208-b54d-7dad-37d465570640}"
2024-12-04T01:32:09.631Z [INFO]  server-d6521f80-eacf-76c4-4394-188e5d8e93be: pipelining replication: peer="{Voter server-afeaa4fc-e681-4e2c-d090-d1a3f9833741 afeaa4fc-e681-4e2c-d090-d1a3f9833741}"
2024-12-04T01:32:10.638Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:09.631530431 +0000 UTC m=+20.463373023 (1 nodes), 750.083051ms from start of poll, 750.633188ms from cluster start. Timeout at 2024-12-04 01:32:10.636535824 +0000 UTC m=+21.468378652, 1.005005629s after stability
2024-12-04T01:32:10.639Z [INFO]  server-d6521f80-eacf-76c4-4394-188e5d8e93be: starting snapshot up to: index=12
2024-12-04T01:32:10.639Z [INFO]  creating new snapshot: path=/tmp/raft3424111819/snapshots/2-12-1733275930639.tmp
2024-12-04T01:32:10.641Z [INFO]  server-d6521f80-eacf-76c4-4394-188e5d8e93be: snapshot complete up to: index=12
2024-12-04T01:32:10.644Z [INFO]  server-87a6cd17-230d-b1a5-e7e9-15043f753040: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-87a6cd17-230d-b1a5-e7e9-15043f753040 Address:87a6cd17-230d-b1a5-e7e9-15043f753040} {Suffrage:Voter ID:server-857734a9-6a22-75b3-ce66-269e0ef6a3ea Address:857734a9-6a22-75b3-ce66-269e0ef6a3ea} {Suffrage:Voter ID:server-5b4c44d4-eda4-6bbe-a17b-02504e5823db Address:5b4c44d4-eda4-6bbe-a17b-02504e5823db}]"
2024-12-04T01:32:10.644Z [INFO]  server-87a6cd17-230d-b1a5-e7e9-15043f753040: entering follower state: follower="Node at 87a6cd17-230d-b1a5-e7e9-15043f753040 [Follower]" leader-address= leader-id=
2024-12-04T01:32:10.644Z [INFO]  server-857734a9-6a22-75b3-ce66-269e0ef6a3ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-87a6cd17-230d-b1a5-e7e9-15043f753040 Address:87a6cd17-230d-b1a5-e7e9-15043f753040} {Suffrage:Voter ID:server-857734a9-6a22-75b3-ce66-269e0ef6a3ea Address:857734a9-6a22-75b3-ce66-269e0ef6a3ea} {Suffrage:Voter ID:server-5b4c44d4-eda4-6bbe-a17b-02504e5823db Address:5b4c44d4-eda4-6bbe-a17b-02504e5823db}]"
2024-12-04T01:32:10.644Z [INFO]  server-857734a9-6a22-75b3-ce66-269e0ef6a3ea: entering follower state: follower="Node at 857734a9-6a22-75b3-ce66-269e0ef6a3ea [Follower]" leader-address= leader-id=
2024-12-04T01:32:10.644Z [INFO]  server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-87a6cd17-230d-b1a5-e7e9-15043f753040 Address:87a6cd17-230d-b1a5-e7e9-15043f753040} {Suffrage:Voter ID:server-857734a9-6a22-75b3-ce66-269e0ef6a3ea Address:857734a9-6a22-75b3-ce66-269e0ef6a3ea} {Suffrage:Voter ID:server-5b4c44d4-eda4-6bbe-a17b-02504e5823db Address:5b4c44d4-eda4-6bbe-a17b-02504e5823db}]"
2024-12-04T01:32:10.644Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:10.644Z [INFO]  server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: entering follower state: follower="Node at 5b4c44d4-eda4-6bbe-a17b-02504e5823db [Follower]" leader-address= leader-id=
2024-12-04T01:32:11.174Z [WARN]  server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:11.174Z [INFO]  server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: entering candidate state: node="Node at 5b4c44d4-eda4-6bbe-a17b-02504e5823db [Candidate]" term=2
2024-12-04T01:32:11.174Z [INFO]  server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: election won: term=2 tally=2
2024-12-04T01:32:11.174Z [INFO]  server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: entering leader state: leader="Node at 5b4c44d4-eda4-6bbe-a17b-02504e5823db [Leader]"
2024-12-04T01:32:11.174Z [INFO]  server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: added peer, starting replication: peer=server-87a6cd17-230d-b1a5-e7e9-15043f753040
2024-12-04T01:32:11.174Z [INFO]  server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: added peer, starting replication: peer=server-857734a9-6a22-75b3-ce66-269e0ef6a3ea
2024-12-04T01:32:11.174Z [INFO]  server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: pipelining replication: peer="{Voter server-857734a9-6a22-75b3-ce66-269e0ef6a3ea 857734a9-6a22-75b3-ce66-269e0ef6a3ea}"
2024-12-04T01:32:11.175Z [INFO]  server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: pipelining replication: peer="{Voter server-87a6cd17-230d-b1a5-e7e9-15043f753040 87a6cd17-230d-b1a5-e7e9-15043f753040}"
2024-12-04T01:32:12.182Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:11.175384978 +0000 UTC m=+22.007227608 (1 nodes), 530.672196ms from start of poll, 531.268648ms from cluster start. Timeout at 2024-12-04 01:32:12.180390719 +0000 UTC m=+23.012233502, 1.005005894s after stability
2024-12-04T01:32:12.182Z [INFO]  creating new snapshot: path=/tmp/raft1652603710/snapshots/2-13-1733275932182.tmp
2024-12-04T01:32:12.183Z [INFO]  server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: copied to local snapshot: bytes=71
2024-12-04T01:32:12.184Z [INFO]  server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: snapshot restore progress: id=2-13-1733275932182 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:12.184Z [INFO]  server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: restored user snapshot: index=13
2024-12-04T01:32:12.184Z [ERROR] server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: failed to get log: index=3 error="log not found"
2024-12-04T01:32:12.184Z [INFO]  server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: aborting pipeline replication: peer="{Voter server-857734a9-6a22-75b3-ce66-269e0ef6a3ea 857734a9-6a22-75b3-ce66-269e0ef6a3ea}"
2024-12-04T01:32:12.184Z [ERROR] server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: failed to get log: index=3 error="log not found"
2024-12-04T01:32:12.184Z [INFO]  server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: aborting pipeline replication: peer="{Voter server-87a6cd17-230d-b1a5-e7e9-15043f753040 87a6cd17-230d-b1a5-e7e9-15043f753040}"
2024-12-04T01:32:12.190Z [ERROR] server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: failed to get log: index=3 error="log not found"
2024-12-04T01:32:12.190Z [INFO]  creating new snapshot: path=/tmp/raft240867072/snapshots/2-13-1733275932190.tmp
2024-12-04T01:32:12.191Z [INFO]  server-87a6cd17-230d-b1a5-e7e9-15043f753040: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:12.192Z [INFO]  server-87a6cd17-230d-b1a5-e7e9-15043f753040: copied to local snapshot: bytes=71
2024-12-04T01:32:12.192Z [INFO]  server-87a6cd17-230d-b1a5-e7e9-15043f753040: snapshot restore progress: id=2-13-1733275932190 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:12.192Z [INFO]  server-87a6cd17-230d-b1a5-e7e9-15043f753040: Installed remote snapshot
2024-12-04T01:32:12.192Z [INFO]  server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: pipelining replication: peer="{Voter server-87a6cd17-230d-b1a5-e7e9-15043f753040 87a6cd17-230d-b1a5-e7e9-15043f753040}"
2024-12-04T01:32:12.192Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:12.193Z [ERROR] server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: failed to get log: index=3 error="log not found"
2024-12-04T01:32:12.193Z [INFO]  creating new snapshot: path=/tmp/raft3171753059/snapshots/2-13-1733275932193.tmp
2024-12-04T01:32:12.194Z [INFO]  server-857734a9-6a22-75b3-ce66-269e0ef6a3ea: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:12.194Z [INFO]  server-857734a9-6a22-75b3-ce66-269e0ef6a3ea: copied to local snapshot: bytes=71
2024-12-04T01:32:12.195Z [INFO]  server-857734a9-6a22-75b3-ce66-269e0ef6a3ea: snapshot restore progress: id=2-13-1733275932193 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:12.195Z [INFO]  server-857734a9-6a22-75b3-ce66-269e0ef6a3ea: Installed remote snapshot
2024-12-04T01:32:12.195Z [INFO]  server-5b4c44d4-eda4-6bbe-a17b-02504e5823db: pipelining replication: peer="{Voter server-857734a9-6a22-75b3-ce66-269e0ef6a3ea 857734a9-6a22-75b3-ce66-269e0ef6a3ea}"
2024-12-04T01:32:13.198Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:12.192866481 +0000 UTC m=+23.024709077 (2 nodes), 1.761µs from start of poll, 1.548750117s from cluster start. Timeout at 2024-12-04 01:32:13.197884205 +0000 UTC m=+24.029726975, 1.005017898s after stability
2024-12-04T01:32:13.210Z [INFO]  server-d6521f80-eacf-76c4-4394-188e5d8e93be: aborting pipeline replication: peer="{Voter server-24b6ee03-c208-b54d-7dad-37d465570640 24b6ee03-c208-b54d-7dad-37d465570640}"
2024-12-04T01:32:13.210Z [INFO]  server-d6521f80-eacf-76c4-4394-188e5d8e93be: aborting pipeline replication: peer="{Voter server-afeaa4fc-e681-4e2c-d090-d1a3f9833741 afeaa4fc-e681-4e2c-d090-d1a3f9833741}"
=== RUN   TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true
2024-12-04T01:32:13.213Z [INFO]  server-daf24d0c-d35d-1549-d834-0b008718fcf4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daf24d0c-d35d-1549-d834-0b008718fcf4 Address:daf24d0c-d35d-1549-d834-0b008718fcf4} {Suffrage:Voter ID:server-8704cc6d-3863-9f0c-3059-0f76e7b93d92 Address:8704cc6d-3863-9f0c-3059-0f76e7b93d92} {Suffrage:Voter ID:server-4f07a3a4-a490-3c92-9af9-f72f4847055c Address:4f07a3a4-a490-3c92-9af9-f72f4847055c}]"
2024-12-04T01:32:13.213Z [INFO]  server-8704cc6d-3863-9f0c-3059-0f76e7b93d92: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daf24d0c-d35d-1549-d834-0b008718fcf4 Address:daf24d0c-d35d-1549-d834-0b008718fcf4} {Suffrage:Voter ID:server-8704cc6d-3863-9f0c-3059-0f76e7b93d92 Address:8704cc6d-3863-9f0c-3059-0f76e7b93d92} {Suffrage:Voter ID:server-4f07a3a4-a490-3c92-9af9-f72f4847055c Address:4f07a3a4-a490-3c92-9af9-f72f4847055c}]"
2024-12-04T01:32:13.213Z [INFO]  server-4f07a3a4-a490-3c92-9af9-f72f4847055c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-daf24d0c-d35d-1549-d834-0b008718fcf4 Address:daf24d0c-d35d-1549-d834-0b008718fcf4} {Suffrage:Voter ID:server-8704cc6d-3863-9f0c-3059-0f76e7b93d92 Address:8704cc6d-3863-9f0c-3059-0f76e7b93d92} {Suffrage:Voter ID:server-4f07a3a4-a490-3c92-9af9-f72f4847055c Address:4f07a3a4-a490-3c92-9af9-f72f4847055c}]"
2024-12-04T01:32:13.213Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:13.213Z [INFO]  server-daf24d0c-d35d-1549-d834-0b008718fcf4: entering follower state: follower="Node at daf24d0c-d35d-1549-d834-0b008718fcf4 [Follower]" leader-address= leader-id=
2024-12-04T01:32:13.213Z [INFO]  server-8704cc6d-3863-9f0c-3059-0f76e7b93d92: entering follower state: follower="Node at 8704cc6d-3863-9f0c-3059-0f76e7b93d92 [Follower]" leader-address= leader-id=
2024-12-04T01:32:13.213Z [INFO]  server-4f07a3a4-a490-3c92-9af9-f72f4847055c: entering follower state: follower="Node at 4f07a3a4-a490-3c92-9af9-f72f4847055c [Follower]" leader-address= leader-id=
2024-12-04T01:32:13.750Z [WARN]  server-4f07a3a4-a490-3c92-9af9-f72f4847055c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:13.750Z [INFO]  server-4f07a3a4-a490-3c92-9af9-f72f4847055c: entering candidate state: node="Node at 4f07a3a4-a490-3c92-9af9-f72f4847055c [Candidate]" term=2
2024-12-04T01:32:13.750Z [INFO]  server-4f07a3a4-a490-3c92-9af9-f72f4847055c: election won: term=2 tally=2
2024-12-04T01:32:13.750Z [INFO]  server-4f07a3a4-a490-3c92-9af9-f72f4847055c: entering leader state: leader="Node at 4f07a3a4-a490-3c92-9af9-f72f4847055c [Leader]"
2024-12-04T01:32:13.750Z [INFO]  server-4f07a3a4-a490-3c92-9af9-f72f4847055c: added peer, starting replication: peer=server-daf24d0c-d35d-1549-d834-0b008718fcf4
2024-12-04T01:32:13.750Z [INFO]  server-4f07a3a4-a490-3c92-9af9-f72f4847055c: added peer, starting replication: peer=server-8704cc6d-3863-9f0c-3059-0f76e7b93d92
2024-12-04T01:32:13.750Z [INFO]  server-4f07a3a4-a490-3c92-9af9-f72f4847055c: pipelining replication: peer="{Voter server-8704cc6d-3863-9f0c-3059-0f76e7b93d92 8704cc6d-3863-9f0c-3059-0f76e7b93d92}"
2024-12-04T01:32:13.751Z [INFO]  server-4f07a3a4-a490-3c92-9af9-f72f4847055c: pipelining replication: peer="{Voter server-daf24d0c-d35d-1549-d834-0b008718fcf4 daf24d0c-d35d-1549-d834-0b008718fcf4}"
2024-12-04T01:32:14.757Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:13.751351261 +0000 UTC m=+24.583194255 (1 nodes), 537.728034ms from start of poll, 538.283777ms from cluster start. Timeout at 2024-12-04 01:32:14.756357119 +0000 UTC m=+25.588199889, 1.005005634s after stability
2024-12-04T01:32:14.758Z [INFO]  server-4f07a3a4-a490-3c92-9af9-f72f4847055c: starting snapshot up to: index=12
2024-12-04T01:32:14.758Z [INFO]  creating new snapshot: path=/tmp/raft3123417605/snapshots/2-12-1733275934758.tmp
2024-12-04T01:32:14.759Z [INFO]  server-4f07a3a4-a490-3c92-9af9-f72f4847055c: snapshot complete up to: index=12
2024-12-04T01:32:14.763Z [INFO]  server-29febbd6-a91a-4961-703e-f6e54abbacf7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29febbd6-a91a-4961-703e-f6e54abbacf7 Address:29febbd6-a91a-4961-703e-f6e54abbacf7} {Suffrage:Voter ID:server-b4616e6d-98ca-03b9-e8db-aa00fbe4edc0 Address:b4616e6d-98ca-03b9-e8db-aa00fbe4edc0} {Suffrage:Voter ID:server-87d72ccc-820f-5ad5-9985-4d6870253f69 Address:87d72ccc-820f-5ad5-9985-4d6870253f69}]"
2024-12-04T01:32:14.763Z [INFO]  server-b4616e6d-98ca-03b9-e8db-aa00fbe4edc0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29febbd6-a91a-4961-703e-f6e54abbacf7 Address:29febbd6-a91a-4961-703e-f6e54abbacf7} {Suffrage:Voter ID:server-b4616e6d-98ca-03b9-e8db-aa00fbe4edc0 Address:b4616e6d-98ca-03b9-e8db-aa00fbe4edc0} {Suffrage:Voter ID:server-87d72ccc-820f-5ad5-9985-4d6870253f69 Address:87d72ccc-820f-5ad5-9985-4d6870253f69}]"
2024-12-04T01:32:14.763Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29febbd6-a91a-4961-703e-f6e54abbacf7 Address:29febbd6-a91a-4961-703e-f6e54abbacf7} {Suffrage:Voter ID:server-b4616e6d-98ca-03b9-e8db-aa00fbe4edc0 Address:b4616e6d-98ca-03b9-e8db-aa00fbe4edc0} {Suffrage:Voter ID:server-87d72ccc-820f-5ad5-9985-4d6870253f69 Address:87d72ccc-820f-5ad5-9985-4d6870253f69}]"
2024-12-04T01:32:14.764Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:14.764Z [INFO]  server-29febbd6-a91a-4961-703e-f6e54abbacf7: entering follower state: follower="Node at 29febbd6-a91a-4961-703e-f6e54abbacf7 [Follower]" leader-address= leader-id=
2024-12-04T01:32:14.764Z [INFO]  server-b4616e6d-98ca-03b9-e8db-aa00fbe4edc0: entering follower state: follower="Node at b4616e6d-98ca-03b9-e8db-aa00fbe4edc0 [Follower]" leader-address= leader-id=
2024-12-04T01:32:14.764Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: entering follower state: follower="Node at 87d72ccc-820f-5ad5-9985-4d6870253f69 [Follower]" leader-address= leader-id=
2024-12-04T01:32:15.310Z [WARN]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:15.310Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: entering candidate state: node="Node at 87d72ccc-820f-5ad5-9985-4d6870253f69 [Candidate]" term=2
2024-12-04T01:32:15.310Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: election won: term=2 tally=2
2024-12-04T01:32:15.310Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: entering leader state: leader="Node at 87d72ccc-820f-5ad5-9985-4d6870253f69 [Leader]"
2024-12-04T01:32:15.310Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: added peer, starting replication: peer=server-29febbd6-a91a-4961-703e-f6e54abbacf7
2024-12-04T01:32:15.310Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: added peer, starting replication: peer=server-b4616e6d-98ca-03b9-e8db-aa00fbe4edc0
2024-12-04T01:32:15.310Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: pipelining replication: peer="{Voter server-b4616e6d-98ca-03b9-e8db-aa00fbe4edc0 b4616e6d-98ca-03b9-e8db-aa00fbe4edc0}"
2024-12-04T01:32:15.311Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: pipelining replication: peer="{Voter server-29febbd6-a91a-4961-703e-f6e54abbacf7 29febbd6-a91a-4961-703e-f6e54abbacf7}"
2024-12-04T01:32:16.318Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:15.311294271 +0000 UTC m=+26.143136894 (1 nodes), 547.258377ms from start of poll, 547.772837ms from cluster start. Timeout at 2024-12-04 01:32:16.316306475 +0000 UTC m=+27.148149766, 1.005012872s after stability
2024-12-04T01:32:16.319Z [INFO]  creating new snapshot: path=/tmp/raft1715220824/snapshots/2-13-1733275936319.tmp
2024-12-04T01:32:16.320Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: copied to local snapshot: bytes=71
2024-12-04T01:32:16.321Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: snapshot restore progress: id=2-13-1733275936319 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:16.321Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: removing all old logs from log store
2024-12-04T01:32:16.321Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: compacting logs: from=1 to=2
2024-12-04T01:32:16.321Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: restored user snapshot: index=13
2024-12-04T01:32:16.321Z [ERROR] server-87d72ccc-820f-5ad5-9985-4d6870253f69: failed to get log: index=2 error="log not found"
2024-12-04T01:32:16.321Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: aborting pipeline replication: peer="{Voter server-b4616e6d-98ca-03b9-e8db-aa00fbe4edc0 b4616e6d-98ca-03b9-e8db-aa00fbe4edc0}"
2024-12-04T01:32:16.321Z [ERROR] server-87d72ccc-820f-5ad5-9985-4d6870253f69: failed to get log: index=2 error="log not found"
2024-12-04T01:32:16.321Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: aborting pipeline replication: peer="{Voter server-29febbd6-a91a-4961-703e-f6e54abbacf7 29febbd6-a91a-4961-703e-f6e54abbacf7}"
2024-12-04T01:32:16.330Z [ERROR] server-87d72ccc-820f-5ad5-9985-4d6870253f69: failed to get log: index=2 error="log not found"
2024-12-04T01:32:16.334Z [ERROR] server-87d72ccc-820f-5ad5-9985-4d6870253f69: failed to get log: index=2 error="log not found"
2024-12-04T01:32:16.350Z [INFO]  creating new snapshot: path=/tmp/raft1643435596/snapshots/2-13-1733275936350.tmp
2024-12-04T01:32:16.351Z [INFO]  server-b4616e6d-98ca-03b9-e8db-aa00fbe4edc0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:16.351Z [INFO]  creating new snapshot: path=/tmp/raft1317250023/snapshots/2-13-1733275936351.tmp
2024-12-04T01:32:16.352Z [INFO]  server-29febbd6-a91a-4961-703e-f6e54abbacf7: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:16.352Z [INFO]  server-b4616e6d-98ca-03b9-e8db-aa00fbe4edc0: copied to local snapshot: bytes=71
2024-12-04T01:32:16.352Z [INFO]  server-b4616e6d-98ca-03b9-e8db-aa00fbe4edc0: snapshot restore progress: id=2-13-1733275936350 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:16.352Z [INFO]  server-b4616e6d-98ca-03b9-e8db-aa00fbe4edc0: removing all old logs from log store
2024-12-04T01:32:16.352Z [INFO]  server-b4616e6d-98ca-03b9-e8db-aa00fbe4edc0: compacting logs: from=1 to=2
2024-12-04T01:32:16.352Z [INFO]  server-b4616e6d-98ca-03b9-e8db-aa00fbe4edc0: Installed remote snapshot
2024-12-04T01:32:16.352Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: pipelining replication: peer="{Voter server-b4616e6d-98ca-03b9-e8db-aa00fbe4edc0 b4616e6d-98ca-03b9-e8db-aa00fbe4edc0}"
2024-12-04T01:32:16.353Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:16.358Z [INFO]  server-29febbd6-a91a-4961-703e-f6e54abbacf7: copied to local snapshot: bytes=71
2024-12-04T01:32:16.358Z [INFO]  server-29febbd6-a91a-4961-703e-f6e54abbacf7: snapshot restore progress: id=2-13-1733275936351 last-index=13 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:16.359Z [INFO]  server-29febbd6-a91a-4961-703e-f6e54abbacf7: removing all old logs from log store
2024-12-04T01:32:16.359Z [INFO]  server-29febbd6-a91a-4961-703e-f6e54abbacf7: compacting logs: from=1 to=2
2024-12-04T01:32:16.359Z [INFO]  server-29febbd6-a91a-4961-703e-f6e54abbacf7: Installed remote snapshot
2024-12-04T01:32:16.359Z [INFO]  server-87d72ccc-820f-5ad5-9985-4d6870253f69: pipelining replication: peer="{Voter server-29febbd6-a91a-4961-703e-f6e54abbacf7 29febbd6-a91a-4961-703e-f6e54abbacf7}"
2024-12-04T01:32:17.358Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:16.353107186 +0000 UTC m=+27.184951723 (2 nodes), 6.188µs from start of poll, 1.589587666s from cluster start. Timeout at 2024-12-04 01:32:17.358137807 +0000 UTC m=+28.189981240, 1.005029517s after stability
2024-12-04T01:32:17.382Z [INFO]  server-4f07a3a4-a490-3c92-9af9-f72f4847055c: aborting pipeline replication: peer="{Voter server-8704cc6d-3863-9f0c-3059-0f76e7b93d92 8704cc6d-3863-9f0c-3059-0f76e7b93d92}"
2024-12-04T01:32:17.382Z [INFO]  server-4f07a3a4-a490-3c92-9af9-f72f4847055c: aborting pipeline replication: peer="{Voter server-daf24d0c-d35d-1549-d834-0b008718fcf4 daf24d0c-d35d-1549-d834-0b008718fcf4}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false
2024-12-04T01:32:17.384Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34a0a104-343d-c46e-d1b9-b9f6428ddee6 Address:34a0a104-343d-c46e-d1b9-b9f6428ddee6} {Suffrage:Voter ID:server-89821237-ad3f-0d71-cb34-8d7e7d13c294 Address:89821237-ad3f-0d71-cb34-8d7e7d13c294} {Suffrage:Voter ID:server-8ab2ed57-c68e-29d3-20f6-539c43b91b37 Address:8ab2ed57-c68e-29d3-20f6-539c43b91b37}]"
2024-12-04T01:32:17.384Z [INFO]  server-89821237-ad3f-0d71-cb34-8d7e7d13c294: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34a0a104-343d-c46e-d1b9-b9f6428ddee6 Address:34a0a104-343d-c46e-d1b9-b9f6428ddee6} {Suffrage:Voter ID:server-89821237-ad3f-0d71-cb34-8d7e7d13c294 Address:89821237-ad3f-0d71-cb34-8d7e7d13c294} {Suffrage:Voter ID:server-8ab2ed57-c68e-29d3-20f6-539c43b91b37 Address:8ab2ed57-c68e-29d3-20f6-539c43b91b37}]"
2024-12-04T01:32:17.384Z [INFO]  server-8ab2ed57-c68e-29d3-20f6-539c43b91b37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-34a0a104-343d-c46e-d1b9-b9f6428ddee6 Address:34a0a104-343d-c46e-d1b9-b9f6428ddee6} {Suffrage:Voter ID:server-89821237-ad3f-0d71-cb34-8d7e7d13c294 Address:89821237-ad3f-0d71-cb34-8d7e7d13c294} {Suffrage:Voter ID:server-8ab2ed57-c68e-29d3-20f6-539c43b91b37 Address:8ab2ed57-c68e-29d3-20f6-539c43b91b37}]"
2024-12-04T01:32:17.384Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:17.384Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: entering follower state: follower="Node at 34a0a104-343d-c46e-d1b9-b9f6428ddee6 [Follower]" leader-address= leader-id=
2024-12-04T01:32:17.384Z [INFO]  server-89821237-ad3f-0d71-cb34-8d7e7d13c294: entering follower state: follower="Node at 89821237-ad3f-0d71-cb34-8d7e7d13c294 [Follower]" leader-address= leader-id=
2024-12-04T01:32:17.384Z [INFO]  server-8ab2ed57-c68e-29d3-20f6-539c43b91b37: entering follower state: follower="Node at 8ab2ed57-c68e-29d3-20f6-539c43b91b37 [Follower]" leader-address= leader-id=
2024-12-04T01:32:17.890Z [WARN]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:17.890Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: entering candidate state: node="Node at 34a0a104-343d-c46e-d1b9-b9f6428ddee6 [Candidate]" term=2
2024-12-04T01:32:17.890Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: election won: term=2 tally=2
2024-12-04T01:32:17.890Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: entering leader state: leader="Node at 34a0a104-343d-c46e-d1b9-b9f6428ddee6 [Leader]"
2024-12-04T01:32:17.890Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: added peer, starting replication: peer=server-89821237-ad3f-0d71-cb34-8d7e7d13c294
2024-12-04T01:32:17.890Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: added peer, starting replication: peer=server-8ab2ed57-c68e-29d3-20f6-539c43b91b37
2024-12-04T01:32:17.890Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: pipelining replication: peer="{Voter server-8ab2ed57-c68e-29d3-20f6-539c43b91b37 8ab2ed57-c68e-29d3-20f6-539c43b91b37}"
2024-12-04T01:32:17.891Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: pipelining replication: peer="{Voter server-89821237-ad3f-0d71-cb34-8d7e7d13c294 89821237-ad3f-0d71-cb34-8d7e7d13c294}"
2024-12-04T01:32:18.902Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:17.891327343 +0000 UTC m=+28.723171249 (1 nodes), 506.491094ms from start of poll, 506.983938ms from cluster start. Timeout at 2024-12-04 01:32:18.896364324 +0000 UTC m=+29.728207032, 1.005035783s after stability
2024-12-04T01:32:18.904Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: starting snapshot up to: index=12
2024-12-04T01:32:18.904Z [INFO]  creating new snapshot: path=/tmp/raft9497488/snapshots/2-12-1733275938904.tmp
2024-12-04T01:32:18.905Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: snapshot complete up to: index=12
2024-12-04T01:32:18.907Z [INFO]  creating new snapshot: path=/tmp/raft9497488/snapshots/2-23-1733275938907.tmp
2024-12-04T01:32:18.908Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: copied to local snapshot: bytes=71
2024-12-04T01:32:18.909Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: snapshot restore progress: id=2-23-1733275938907 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:18.909Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: restored user snapshot: index=23
2024-12-04T01:32:18.909Z [ERROR] server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: failed to get log: index=23 error="log not found"
2024-12-04T01:32:18.909Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: aborting pipeline replication: peer="{Voter server-89821237-ad3f-0d71-cb34-8d7e7d13c294 89821237-ad3f-0d71-cb34-8d7e7d13c294}"
2024-12-04T01:32:18.909Z [ERROR] server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: failed to get log: index=23 error="log not found"
2024-12-04T01:32:18.909Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: aborting pipeline replication: peer="{Voter server-8ab2ed57-c68e-29d3-20f6-539c43b91b37 8ab2ed57-c68e-29d3-20f6-539c43b91b37}"
2024-12-04T01:32:18.918Z [ERROR] server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: failed to get log: index=23 error="log not found"
2024-12-04T01:32:18.919Z [INFO]  creating new snapshot: path=/tmp/raft1459926777/snapshots/2-23-1733275938919.tmp
2024-12-04T01:32:18.930Z [INFO]  server-8ab2ed57-c68e-29d3-20f6-539c43b91b37: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:18.931Z [INFO]  server-8ab2ed57-c68e-29d3-20f6-539c43b91b37: copied to local snapshot: bytes=71
2024-12-04T01:32:18.932Z [INFO]  server-8ab2ed57-c68e-29d3-20f6-539c43b91b37: snapshot restore progress: id=2-23-1733275938919 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:18.932Z [INFO]  server-8ab2ed57-c68e-29d3-20f6-539c43b91b37: Installed remote snapshot
2024-12-04T01:32:18.932Z [WARN]  server-8ab2ed57-c68e-29d3-20f6-539c43b91b37: skipping application of old log: index=22
2024-12-04T01:32:18.932Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: pipelining replication: peer="{Voter server-8ab2ed57-c68e-29d3-20f6-539c43b91b37 8ab2ed57-c68e-29d3-20f6-539c43b91b37}"
2024-12-04T01:32:18.932Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:18.933Z [ERROR] server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: failed to get log: index=23 error="log not found"
2024-12-04T01:32:18.942Z [INFO]  creating new snapshot: path=/tmp/raft1398953196/snapshots/2-23-1733275938942.tmp
2024-12-04T01:32:18.943Z [INFO]  server-89821237-ad3f-0d71-cb34-8d7e7d13c294: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:18.944Z [INFO]  server-89821237-ad3f-0d71-cb34-8d7e7d13c294: copied to local snapshot: bytes=71
2024-12-04T01:32:18.944Z [INFO]  server-89821237-ad3f-0d71-cb34-8d7e7d13c294: snapshot restore progress: id=2-23-1733275938942 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:18.944Z [INFO]  server-89821237-ad3f-0d71-cb34-8d7e7d13c294: Installed remote snapshot
2024-12-04T01:32:18.945Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: pipelining replication: peer="{Voter server-89821237-ad3f-0d71-cb34-8d7e7d13c294 89821237-ad3f-0d71-cb34-8d7e7d13c294}"
2024-12-04T01:32:19.938Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:18.932707766 +0000 UTC m=+29.764550495 (2 nodes), 2.744µs from start of poll, 1.548363184s from cluster start. Timeout at 2024-12-04 01:32:19.93773655 +0000 UTC m=+30.769579552, 1.005029057s after stability
2024-12-04T01:32:19.958Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: aborting pipeline replication: peer="{Voter server-8ab2ed57-c68e-29d3-20f6-539c43b91b37 8ab2ed57-c68e-29d3-20f6-539c43b91b37}"
2024-12-04T01:32:19.958Z [INFO]  server-34a0a104-343d-c46e-d1b9-b9f6428ddee6: aborting pipeline replication: peer="{Voter server-89821237-ad3f-0d71-cb34-8d7e7d13c294 89821237-ad3f-0d71-cb34-8d7e7d13c294}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false
2024-12-04T01:32:19.960Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b46a7678-5f9c-565f-a180-62db4de3d739 Address:b46a7678-5f9c-565f-a180-62db4de3d739} {Suffrage:Voter ID:server-2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f Address:2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f} {Suffrage:Voter ID:server-3448766e-3b39-a5b9-c5c2-37b539effa52 Address:3448766e-3b39-a5b9-c5c2-37b539effa52}]"
2024-12-04T01:32:19.960Z [INFO]  server-2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b46a7678-5f9c-565f-a180-62db4de3d739 Address:b46a7678-5f9c-565f-a180-62db4de3d739} {Suffrage:Voter ID:server-2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f Address:2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f} {Suffrage:Voter ID:server-3448766e-3b39-a5b9-c5c2-37b539effa52 Address:3448766e-3b39-a5b9-c5c2-37b539effa52}]"
2024-12-04T01:32:19.960Z [INFO]  server-3448766e-3b39-a5b9-c5c2-37b539effa52: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b46a7678-5f9c-565f-a180-62db4de3d739 Address:b46a7678-5f9c-565f-a180-62db4de3d739} {Suffrage:Voter ID:server-2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f Address:2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f} {Suffrage:Voter ID:server-3448766e-3b39-a5b9-c5c2-37b539effa52 Address:3448766e-3b39-a5b9-c5c2-37b539effa52}]"
2024-12-04T01:32:19.960Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:19.960Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: entering follower state: follower="Node at b46a7678-5f9c-565f-a180-62db4de3d739 [Follower]" leader-address= leader-id=
2024-12-04T01:32:19.960Z [INFO]  server-2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f: entering follower state: follower="Node at 2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f [Follower]" leader-address= leader-id=
2024-12-04T01:32:19.960Z [INFO]  server-3448766e-3b39-a5b9-c5c2-37b539effa52: entering follower state: follower="Node at 3448766e-3b39-a5b9-c5c2-37b539effa52 [Follower]" leader-address= leader-id=
2024-12-04T01:32:20.733Z [WARN]  server-b46a7678-5f9c-565f-a180-62db4de3d739: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:20.733Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: entering candidate state: node="Node at b46a7678-5f9c-565f-a180-62db4de3d739 [Candidate]" term=2
2024-12-04T01:32:20.733Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: election won: term=2 tally=2
2024-12-04T01:32:20.733Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: entering leader state: leader="Node at b46a7678-5f9c-565f-a180-62db4de3d739 [Leader]"
2024-12-04T01:32:20.733Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: added peer, starting replication: peer=server-2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f
2024-12-04T01:32:20.733Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: added peer, starting replication: peer=server-3448766e-3b39-a5b9-c5c2-37b539effa52
2024-12-04T01:32:20.733Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: pipelining replication: peer="{Voter server-3448766e-3b39-a5b9-c5c2-37b539effa52 3448766e-3b39-a5b9-c5c2-37b539effa52}"
2024-12-04T01:32:20.734Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: pipelining replication: peer="{Voter server-2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f 2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f}"
2024-12-04T01:32:21.740Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:20.734188567 +0000 UTC m=+31.566031595 (1 nodes), 773.561633ms from start of poll, 773.929994ms from cluster start. Timeout at 2024-12-04 01:32:21.739194917 +0000 UTC m=+32.571037629, 1.005006034s after stability
2024-12-04T01:32:21.742Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: starting snapshot up to: index=12
2024-12-04T01:32:21.742Z [INFO]  creating new snapshot: path=/tmp/raft3336434848/snapshots/2-12-1733275941742.tmp
2024-12-04T01:32:21.743Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: snapshot complete up to: index=12
2024-12-04T01:32:21.745Z [INFO]  creating new snapshot: path=/tmp/raft3336434848/snapshots/2-23-1733275941745.tmp
2024-12-04T01:32:21.747Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: copied to local snapshot: bytes=71
2024-12-04T01:32:21.747Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: snapshot restore progress: id=2-23-1733275941745 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:21.747Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: removing all old logs from log store
2024-12-04T01:32:21.747Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: compacting logs: from=1 to=22
2024-12-04T01:32:21.747Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: restored user snapshot: index=23
2024-12-04T01:32:21.747Z [ERROR] server-b46a7678-5f9c-565f-a180-62db4de3d739: failed to get log: index=22 error="log not found"
2024-12-04T01:32:21.747Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: aborting pipeline replication: peer="{Voter server-3448766e-3b39-a5b9-c5c2-37b539effa52 3448766e-3b39-a5b9-c5c2-37b539effa52}"
2024-12-04T01:32:21.747Z [ERROR] server-b46a7678-5f9c-565f-a180-62db4de3d739: failed to get log: index=22 error="log not found"
2024-12-04T01:32:21.747Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: aborting pipeline replication: peer="{Voter server-2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f 2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f}"
2024-12-04T01:32:21.753Z [ERROR] server-b46a7678-5f9c-565f-a180-62db4de3d739: failed to get log: index=22 error="log not found"
2024-12-04T01:32:21.754Z [INFO]  creating new snapshot: path=/tmp/raft3642441212/snapshots/2-23-1733275941754.tmp
2024-12-04T01:32:21.755Z [INFO]  server-2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:21.755Z [INFO]  server-2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f: copied to local snapshot: bytes=71
2024-12-04T01:32:21.755Z [ERROR] server-b46a7678-5f9c-565f-a180-62db4de3d739: failed to get log: index=22 error="log not found"
2024-12-04T01:32:21.755Z [INFO]  server-2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f: snapshot restore progress: id=2-23-1733275941754 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:21.755Z [INFO]  server-2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f: removing all old logs from log store
2024-12-04T01:32:21.755Z [INFO]  server-2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f: compacting logs: from=1 to=22
2024-12-04T01:32:21.756Z [INFO]  server-2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f: Installed remote snapshot
2024-12-04T01:32:21.756Z [WARN]  server-2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f: skipping application of old log: index=22
2024-12-04T01:32:21.756Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: pipelining replication: peer="{Voter server-2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f 2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f}"
2024-12-04T01:32:21.756Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:21.756Z [INFO]  creating new snapshot: path=/tmp/raft2204142557/snapshots/2-23-1733275941756.tmp
2024-12-04T01:32:21.756Z [INFO]  server-3448766e-3b39-a5b9-c5c2-37b539effa52: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:21.757Z [INFO]  server-3448766e-3b39-a5b9-c5c2-37b539effa52: copied to local snapshot: bytes=71
2024-12-04T01:32:21.757Z [INFO]  server-3448766e-3b39-a5b9-c5c2-37b539effa52: snapshot restore progress: id=2-23-1733275941756 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:21.757Z [INFO]  server-3448766e-3b39-a5b9-c5c2-37b539effa52: removing all old logs from log store
2024-12-04T01:32:21.757Z [INFO]  server-3448766e-3b39-a5b9-c5c2-37b539effa52: compacting logs: from=1 to=22
2024-12-04T01:32:21.757Z [INFO]  server-3448766e-3b39-a5b9-c5c2-37b539effa52: Installed remote snapshot
2024-12-04T01:32:21.757Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: pipelining replication: peer="{Voter server-3448766e-3b39-a5b9-c5c2-37b539effa52 3448766e-3b39-a5b9-c5c2-37b539effa52}"
2024-12-04T01:32:22.761Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:21.756312864 +0000 UTC m=+32.588155512 (2 nodes), 2.212µs from start of poll, 1.796053911s from cluster start. Timeout at 2024-12-04 01:32:22.761327402 +0000 UTC m=+33.593170319, 1.005014807s after stability
2024-12-04T01:32:22.774Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: aborting pipeline replication: peer="{Voter server-3448766e-3b39-a5b9-c5c2-37b539effa52 3448766e-3b39-a5b9-c5c2-37b539effa52}"
2024-12-04T01:32:22.775Z [INFO]  server-b46a7678-5f9c-565f-a180-62db4de3d739: aborting pipeline replication: peer="{Voter server-2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f 2f2c41d1-e4f7-c3ba-94ae-e1e3100d5f4f}"
=== RUN   TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true
2024-12-04T01:32:22.776Z [INFO]  server-bd5e24c6-5c28-97dd-5a7e-d73889ed1c32: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bd5e24c6-5c28-97dd-5a7e-d73889ed1c32 Address:bd5e24c6-5c28-97dd-5a7e-d73889ed1c32} {Suffrage:Voter ID:server-d71a4153-d7cc-af01-f0e0-d7aae6283866 Address:d71a4153-d7cc-af01-f0e0-d7aae6283866} {Suffrage:Voter ID:server-26f0b634-25d6-389d-067a-0cf35f254022 Address:26f0b634-25d6-389d-067a-0cf35f254022}]"
2024-12-04T01:32:22.777Z [INFO]  server-d71a4153-d7cc-af01-f0e0-d7aae6283866: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bd5e24c6-5c28-97dd-5a7e-d73889ed1c32 Address:bd5e24c6-5c28-97dd-5a7e-d73889ed1c32} {Suffrage:Voter ID:server-d71a4153-d7cc-af01-f0e0-d7aae6283866 Address:d71a4153-d7cc-af01-f0e0-d7aae6283866} {Suffrage:Voter ID:server-26f0b634-25d6-389d-067a-0cf35f254022 Address:26f0b634-25d6-389d-067a-0cf35f254022}]"
2024-12-04T01:32:22.777Z [INFO]  server-26f0b634-25d6-389d-067a-0cf35f254022: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bd5e24c6-5c28-97dd-5a7e-d73889ed1c32 Address:bd5e24c6-5c28-97dd-5a7e-d73889ed1c32} {Suffrage:Voter ID:server-d71a4153-d7cc-af01-f0e0-d7aae6283866 Address:d71a4153-d7cc-af01-f0e0-d7aae6283866} {Suffrage:Voter ID:server-26f0b634-25d6-389d-067a-0cf35f254022 Address:26f0b634-25d6-389d-067a-0cf35f254022}]"
2024-12-04T01:32:22.777Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:22.777Z [INFO]  server-bd5e24c6-5c28-97dd-5a7e-d73889ed1c32: entering follower state: follower="Node at bd5e24c6-5c28-97dd-5a7e-d73889ed1c32 [Follower]" leader-address= leader-id=
2024-12-04T01:32:22.777Z [INFO]  server-d71a4153-d7cc-af01-f0e0-d7aae6283866: entering follower state: follower="Node at d71a4153-d7cc-af01-f0e0-d7aae6283866 [Follower]" leader-address= leader-id=
2024-12-04T01:32:22.777Z [INFO]  server-26f0b634-25d6-389d-067a-0cf35f254022: entering follower state: follower="Node at 26f0b634-25d6-389d-067a-0cf35f254022 [Follower]" leader-address= leader-id=
2024-12-04T01:32:23.395Z [WARN]  server-bd5e24c6-5c28-97dd-5a7e-d73889ed1c32: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:23.395Z [INFO]  server-bd5e24c6-5c28-97dd-5a7e-d73889ed1c32: entering candidate state: node="Node at bd5e24c6-5c28-97dd-5a7e-d73889ed1c32 [Candidate]" term=2
2024-12-04T01:32:23.396Z [INFO]  server-bd5e24c6-5c28-97dd-5a7e-d73889ed1c32: election won: term=2 tally=2
2024-12-04T01:32:23.396Z [INFO]  server-bd5e24c6-5c28-97dd-5a7e-d73889ed1c32: entering leader state: leader="Node at bd5e24c6-5c28-97dd-5a7e-d73889ed1c32 [Leader]"
2024-12-04T01:32:23.396Z [INFO]  server-bd5e24c6-5c28-97dd-5a7e-d73889ed1c32: added peer, starting replication: peer=server-d71a4153-d7cc-af01-f0e0-d7aae6283866
2024-12-04T01:32:23.396Z [INFO]  server-bd5e24c6-5c28-97dd-5a7e-d73889ed1c32: added peer, starting replication: peer=server-26f0b634-25d6-389d-067a-0cf35f254022
2024-12-04T01:32:23.396Z [INFO]  server-bd5e24c6-5c28-97dd-5a7e-d73889ed1c32: pipelining replication: peer="{Voter server-26f0b634-25d6-389d-067a-0cf35f254022 26f0b634-25d6-389d-067a-0cf35f254022}"
2024-12-04T01:32:23.396Z [INFO]  server-bd5e24c6-5c28-97dd-5a7e-d73889ed1c32: pipelining replication: peer="{Voter server-d71a4153-d7cc-af01-f0e0-d7aae6283866 d71a4153-d7cc-af01-f0e0-d7aae6283866}"
2024-12-04T01:32:24.402Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:23.396766019 +0000 UTC m=+34.228608994 (1 nodes), 619.521945ms from start of poll, 619.918461ms from cluster start. Timeout at 2024-12-04 01:32:24.401777053 +0000 UTC m=+35.233619622, 1.005010628s after stability
2024-12-04T01:32:24.403Z [INFO]  server-bd5e24c6-5c28-97dd-5a7e-d73889ed1c32: starting snapshot up to: index=12
2024-12-04T01:32:24.403Z [INFO]  creating new snapshot: path=/tmp/raft492742055/snapshots/2-12-1733275944403.tmp
2024-12-04T01:32:24.404Z [INFO]  server-bd5e24c6-5c28-97dd-5a7e-d73889ed1c32: snapshot complete up to: index=12
2024-12-04T01:32:24.411Z [INFO]  server-95c17bee-31f9-f031-ddd4-b03cf09a152a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95c17bee-31f9-f031-ddd4-b03cf09a152a Address:95c17bee-31f9-f031-ddd4-b03cf09a152a} {Suffrage:Voter ID:server-0b37bd6f-037d-09a5-1900-e0cff3a354c8 Address:0b37bd6f-037d-09a5-1900-e0cff3a354c8} {Suffrage:Voter ID:server-a650852f-764b-a053-4d07-525a1ed7154b Address:a650852f-764b-a053-4d07-525a1ed7154b}]"
2024-12-04T01:32:24.411Z [INFO]  server-0b37bd6f-037d-09a5-1900-e0cff3a354c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95c17bee-31f9-f031-ddd4-b03cf09a152a Address:95c17bee-31f9-f031-ddd4-b03cf09a152a} {Suffrage:Voter ID:server-0b37bd6f-037d-09a5-1900-e0cff3a354c8 Address:0b37bd6f-037d-09a5-1900-e0cff3a354c8} {Suffrage:Voter ID:server-a650852f-764b-a053-4d07-525a1ed7154b Address:a650852f-764b-a053-4d07-525a1ed7154b}]"
2024-12-04T01:32:24.411Z [INFO]  server-a650852f-764b-a053-4d07-525a1ed7154b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-95c17bee-31f9-f031-ddd4-b03cf09a152a Address:95c17bee-31f9-f031-ddd4-b03cf09a152a} {Suffrage:Voter ID:server-0b37bd6f-037d-09a5-1900-e0cff3a354c8 Address:0b37bd6f-037d-09a5-1900-e0cff3a354c8} {Suffrage:Voter ID:server-a650852f-764b-a053-4d07-525a1ed7154b Address:a650852f-764b-a053-4d07-525a1ed7154b}]"
2024-12-04T01:32:24.411Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:24.412Z [INFO]  server-95c17bee-31f9-f031-ddd4-b03cf09a152a: entering follower state: follower="Node at 95c17bee-31f9-f031-ddd4-b03cf09a152a [Follower]" leader-address= leader-id=
2024-12-04T01:32:24.412Z [INFO]  server-0b37bd6f-037d-09a5-1900-e0cff3a354c8: entering follower state: follower="Node at 0b37bd6f-037d-09a5-1900-e0cff3a354c8 [Follower]" leader-address= leader-id=
2024-12-04T01:32:24.412Z [INFO]  server-a650852f-764b-a053-4d07-525a1ed7154b: entering follower state: follower="Node at a650852f-764b-a053-4d07-525a1ed7154b [Follower]" leader-address= leader-id=
2024-12-04T01:32:25.012Z [WARN]  server-95c17bee-31f9-f031-ddd4-b03cf09a152a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:25.012Z [INFO]  server-95c17bee-31f9-f031-ddd4-b03cf09a152a: entering candidate state: node="Node at 95c17bee-31f9-f031-ddd4-b03cf09a152a [Candidate]" term=2
2024-12-04T01:32:25.012Z [INFO]  server-95c17bee-31f9-f031-ddd4-b03cf09a152a: election won: term=2 tally=2
2024-12-04T01:32:25.012Z [INFO]  server-95c17bee-31f9-f031-ddd4-b03cf09a152a: entering leader state: leader="Node at 95c17bee-31f9-f031-ddd4-b03cf09a152a [Leader]"
2024-12-04T01:32:25.012Z [INFO]  server-95c17bee-31f9-f031-ddd4-b03cf09a152a: added peer, starting replication: peer=server-0b37bd6f-037d-09a5-1900-e0cff3a354c8
2024-12-04T01:32:25.012Z [INFO]  server-95c17bee-31f9-f031-ddd4-b03cf09a152a: added peer, starting replication: peer=server-a650852f-764b-a053-4d07-525a1ed7154b
2024-12-04T01:32:25.012Z [INFO]  server-95c17bee-31f9-f031-ddd4-b03cf09a152a: pipelining replication: peer="{Voter server-a650852f-764b-a053-4d07-525a1ed7154b a650852f-764b-a053-4d07-525a1ed7154b}"
2024-12-04T01:32:25.012Z [INFO]  server-95c17bee-31f9-f031-ddd4-b03cf09a152a: pipelining replication: peer="{Voter server-0b37bd6f-037d-09a5-1900-e0cff3a354c8 0b37bd6f-037d-09a5-1900-e0cff3a354c8}"
2024-12-04T01:32:26.018Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:25.013155715 +0000 UTC m=+35.844998407 (1 nodes), 601.208538ms from start of poll, 601.752078ms from cluster start. Timeout at 2024-12-04 01:32:26.018172061 +0000 UTC m=+36.850014634, 1.005016227s after stability
2024-12-04T01:32:26.018Z [INFO]  creating new snapshot: path=/tmp/raft245050750/snapshots/2-14-1733275946018.tmp
2024-12-04T01:32:26.050Z [INFO]  server-95c17bee-31f9-f031-ddd4-b03cf09a152a: copied to local snapshot: bytes=71
2024-12-04T01:32:26.050Z [INFO]  server-95c17bee-31f9-f031-ddd4-b03cf09a152a: snapshot restore progress: id=2-14-1733275946018 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:26.050Z [INFO]  server-95c17bee-31f9-f031-ddd4-b03cf09a152a: restored user snapshot: index=14
2024-12-04T01:32:26.050Z [ERROR] server-95c17bee-31f9-f031-ddd4-b03cf09a152a: failed to get log: index=3 error="log not found"
2024-12-04T01:32:26.050Z [INFO]  server-95c17bee-31f9-f031-ddd4-b03cf09a152a: aborting pipeline replication: peer="{Voter server-a650852f-764b-a053-4d07-525a1ed7154b a650852f-764b-a053-4d07-525a1ed7154b}"
2024-12-04T01:32:26.050Z [ERROR] server-95c17bee-31f9-f031-ddd4-b03cf09a152a: failed to get log: index=3 error="log not found"
2024-12-04T01:32:26.050Z [INFO]  server-95c17bee-31f9-f031-ddd4-b03cf09a152a: aborting pipeline replication: peer="{Voter server-0b37bd6f-037d-09a5-1900-e0cff3a354c8 0b37bd6f-037d-09a5-1900-e0cff3a354c8}"
2024-12-04T01:32:26.057Z [ERROR] server-95c17bee-31f9-f031-ddd4-b03cf09a152a: failed to get log: index=3 error="log not found"
2024-12-04T01:32:26.059Z [INFO]  creating new snapshot: path=/tmp/raft1403250403/snapshots/2-14-1733275946059.tmp
2024-12-04T01:32:26.059Z [INFO]  server-0b37bd6f-037d-09a5-1900-e0cff3a354c8: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:26.060Z [ERROR] server-95c17bee-31f9-f031-ddd4-b03cf09a152a: failed to get log: index=3 error="log not found"
2024-12-04T01:32:26.061Z [INFO]  creating new snapshot: path=/tmp/raft3724258599/snapshots/2-14-1733275946061.tmp
2024-12-04T01:32:26.062Z [INFO]  server-a650852f-764b-a053-4d07-525a1ed7154b: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:26.092Z [INFO]  server-0b37bd6f-037d-09a5-1900-e0cff3a354c8: copied to local snapshot: bytes=71
2024-12-04T01:32:26.092Z [INFO]  server-0b37bd6f-037d-09a5-1900-e0cff3a354c8: snapshot restore progress: id=2-14-1733275946059 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:26.092Z [INFO]  server-0b37bd6f-037d-09a5-1900-e0cff3a354c8: Installed remote snapshot
2024-12-04T01:32:26.092Z [INFO]  server-95c17bee-31f9-f031-ddd4-b03cf09a152a: pipelining replication: peer="{Voter server-0b37bd6f-037d-09a5-1900-e0cff3a354c8 0b37bd6f-037d-09a5-1900-e0cff3a354c8}"
2024-12-04T01:32:26.092Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:26.094Z [INFO]  server-a650852f-764b-a053-4d07-525a1ed7154b: copied to local snapshot: bytes=71
2024-12-04T01:32:26.094Z [INFO]  server-a650852f-764b-a053-4d07-525a1ed7154b: snapshot restore progress: id=2-14-1733275946061 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:26.094Z [INFO]  server-a650852f-764b-a053-4d07-525a1ed7154b: Installed remote snapshot
2024-12-04T01:32:26.095Z [INFO]  server-95c17bee-31f9-f031-ddd4-b03cf09a152a: pipelining replication: peer="{Voter server-a650852f-764b-a053-4d07-525a1ed7154b a650852f-764b-a053-4d07-525a1ed7154b}"
2024-12-04T01:32:27.098Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:26.092762113 +0000 UTC m=+36.924604679 (2 nodes), 1.796µs from start of poll, 1.68135835s from cluster start. Timeout at 2024-12-04 01:32:27.097784442 +0000 UTC m=+37.929627047, 1.005022368s after stability
2024-12-04T01:32:27.111Z [INFO]  server-bd5e24c6-5c28-97dd-5a7e-d73889ed1c32: aborting pipeline replication: peer="{Voter server-26f0b634-25d6-389d-067a-0cf35f254022 26f0b634-25d6-389d-067a-0cf35f254022}"
2024-12-04T01:32:27.111Z [INFO]  server-bd5e24c6-5c28-97dd-5a7e-d73889ed1c32: aborting pipeline replication: peer="{Voter server-d71a4153-d7cc-af01-f0e0-d7aae6283866 d71a4153-d7cc-af01-f0e0-d7aae6283866}"
=== RUN   TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true
2024-12-04T01:32:27.112Z [INFO]  server-dca907f5-d6b8-ddcd-ac91-f4f71c1492f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dca907f5-d6b8-ddcd-ac91-f4f71c1492f2 Address:dca907f5-d6b8-ddcd-ac91-f4f71c1492f2} {Suffrage:Voter ID:server-985705b5-69d6-9612-6bf1-7f5e77c58aa2 Address:985705b5-69d6-9612-6bf1-7f5e77c58aa2} {Suffrage:Voter ID:server-c3f3c9a6-c6ea-63a6-4449-a5621556c399 Address:c3f3c9a6-c6ea-63a6-4449-a5621556c399}]"
2024-12-04T01:32:27.113Z [INFO]  server-985705b5-69d6-9612-6bf1-7f5e77c58aa2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dca907f5-d6b8-ddcd-ac91-f4f71c1492f2 Address:dca907f5-d6b8-ddcd-ac91-f4f71c1492f2} {Suffrage:Voter ID:server-985705b5-69d6-9612-6bf1-7f5e77c58aa2 Address:985705b5-69d6-9612-6bf1-7f5e77c58aa2} {Suffrage:Voter ID:server-c3f3c9a6-c6ea-63a6-4449-a5621556c399 Address:c3f3c9a6-c6ea-63a6-4449-a5621556c399}]"
2024-12-04T01:32:27.113Z [INFO]  server-c3f3c9a6-c6ea-63a6-4449-a5621556c399: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dca907f5-d6b8-ddcd-ac91-f4f71c1492f2 Address:dca907f5-d6b8-ddcd-ac91-f4f71c1492f2} {Suffrage:Voter ID:server-985705b5-69d6-9612-6bf1-7f5e77c58aa2 Address:985705b5-69d6-9612-6bf1-7f5e77c58aa2} {Suffrage:Voter ID:server-c3f3c9a6-c6ea-63a6-4449-a5621556c399 Address:c3f3c9a6-c6ea-63a6-4449-a5621556c399}]"
2024-12-04T01:32:27.113Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:27.113Z [INFO]  server-dca907f5-d6b8-ddcd-ac91-f4f71c1492f2: entering follower state: follower="Node at dca907f5-d6b8-ddcd-ac91-f4f71c1492f2 [Follower]" leader-address= leader-id=
2024-12-04T01:32:27.113Z [INFO]  server-985705b5-69d6-9612-6bf1-7f5e77c58aa2: entering follower state: follower="Node at 985705b5-69d6-9612-6bf1-7f5e77c58aa2 [Follower]" leader-address= leader-id=
2024-12-04T01:32:27.113Z [INFO]  server-c3f3c9a6-c6ea-63a6-4449-a5621556c399: entering follower state: follower="Node at c3f3c9a6-c6ea-63a6-4449-a5621556c399 [Follower]" leader-address= leader-id=
2024-12-04T01:32:27.655Z [WARN]  server-c3f3c9a6-c6ea-63a6-4449-a5621556c399: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:27.655Z [INFO]  server-c3f3c9a6-c6ea-63a6-4449-a5621556c399: entering candidate state: node="Node at c3f3c9a6-c6ea-63a6-4449-a5621556c399 [Candidate]" term=2
2024-12-04T01:32:27.655Z [INFO]  server-c3f3c9a6-c6ea-63a6-4449-a5621556c399: election won: term=2 tally=2
2024-12-04T01:32:27.655Z [INFO]  server-c3f3c9a6-c6ea-63a6-4449-a5621556c399: entering leader state: leader="Node at c3f3c9a6-c6ea-63a6-4449-a5621556c399 [Leader]"
2024-12-04T01:32:27.655Z [INFO]  server-c3f3c9a6-c6ea-63a6-4449-a5621556c399: added peer, starting replication: peer=server-dca907f5-d6b8-ddcd-ac91-f4f71c1492f2
2024-12-04T01:32:27.655Z [INFO]  server-c3f3c9a6-c6ea-63a6-4449-a5621556c399: added peer, starting replication: peer=server-985705b5-69d6-9612-6bf1-7f5e77c58aa2
2024-12-04T01:32:27.657Z [INFO]  server-c3f3c9a6-c6ea-63a6-4449-a5621556c399: pipelining replication: peer="{Voter server-985705b5-69d6-9612-6bf1-7f5e77c58aa2 985705b5-69d6-9612-6bf1-7f5e77c58aa2}"
2024-12-04T01:32:27.657Z [INFO]  server-c3f3c9a6-c6ea-63a6-4449-a5621556c399: pipelining replication: peer="{Voter server-dca907f5-d6b8-ddcd-ac91-f4f71c1492f2 dca907f5-d6b8-ddcd-ac91-f4f71c1492f2}"
2024-12-04T01:32:28.660Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:27.655449319 +0000 UTC m=+38.487292506 (1 nodes), 542.172705ms from start of poll, 542.606906ms from cluster start. Timeout at 2024-12-04 01:32:28.660455114 +0000 UTC m=+39.492297703, 1.005005197s after stability
2024-12-04T01:32:28.662Z [INFO]  server-c3f3c9a6-c6ea-63a6-4449-a5621556c399: starting snapshot up to: index=12
2024-12-04T01:32:28.662Z [INFO]  creating new snapshot: path=/tmp/raft310529170/snapshots/2-12-1733275948662.tmp
2024-12-04T01:32:28.691Z [INFO]  server-c3f3c9a6-c6ea-63a6-4449-a5621556c399: snapshot complete up to: index=12
2024-12-04T01:32:28.693Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-55513658-02d6-d8ac-72cf-e5d53c7d7e60 Address:55513658-02d6-d8ac-72cf-e5d53c7d7e60} {Suffrage:Voter ID:server-698cf176-f243-9f5a-fdef-b7b82d3a91d5 Address:698cf176-f243-9f5a-fdef-b7b82d3a91d5} {Suffrage:Voter ID:server-5b7d0de1-44ea-ea9d-58a2-efdc294af76e Address:5b7d0de1-44ea-ea9d-58a2-efdc294af76e}]"
2024-12-04T01:32:28.693Z [INFO]  server-698cf176-f243-9f5a-fdef-b7b82d3a91d5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-55513658-02d6-d8ac-72cf-e5d53c7d7e60 Address:55513658-02d6-d8ac-72cf-e5d53c7d7e60} {Suffrage:Voter ID:server-698cf176-f243-9f5a-fdef-b7b82d3a91d5 Address:698cf176-f243-9f5a-fdef-b7b82d3a91d5} {Suffrage:Voter ID:server-5b7d0de1-44ea-ea9d-58a2-efdc294af76e Address:5b7d0de1-44ea-ea9d-58a2-efdc294af76e}]"
2024-12-04T01:32:28.693Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: entering follower state: follower="Node at 55513658-02d6-d8ac-72cf-e5d53c7d7e60 [Follower]" leader-address= leader-id=
2024-12-04T01:32:28.694Z [INFO]  server-698cf176-f243-9f5a-fdef-b7b82d3a91d5: entering follower state: follower="Node at 698cf176-f243-9f5a-fdef-b7b82d3a91d5 [Follower]" leader-address= leader-id=
2024-12-04T01:32:28.694Z [INFO]  server-5b7d0de1-44ea-ea9d-58a2-efdc294af76e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-55513658-02d6-d8ac-72cf-e5d53c7d7e60 Address:55513658-02d6-d8ac-72cf-e5d53c7d7e60} {Suffrage:Voter ID:server-698cf176-f243-9f5a-fdef-b7b82d3a91d5 Address:698cf176-f243-9f5a-fdef-b7b82d3a91d5} {Suffrage:Voter ID:server-5b7d0de1-44ea-ea9d-58a2-efdc294af76e Address:5b7d0de1-44ea-ea9d-58a2-efdc294af76e}]"
2024-12-04T01:32:28.694Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:28.694Z [INFO]  server-5b7d0de1-44ea-ea9d-58a2-efdc294af76e: entering follower state: follower="Node at 5b7d0de1-44ea-ea9d-58a2-efdc294af76e [Follower]" leader-address= leader-id=
2024-12-04T01:32:29.268Z [WARN]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:29.268Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: entering candidate state: node="Node at 55513658-02d6-d8ac-72cf-e5d53c7d7e60 [Candidate]" term=2
2024-12-04T01:32:29.268Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: election won: term=2 tally=2
2024-12-04T01:32:29.268Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: entering leader state: leader="Node at 55513658-02d6-d8ac-72cf-e5d53c7d7e60 [Leader]"
2024-12-04T01:32:29.268Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: added peer, starting replication: peer=server-698cf176-f243-9f5a-fdef-b7b82d3a91d5
2024-12-04T01:32:29.268Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: added peer, starting replication: peer=server-5b7d0de1-44ea-ea9d-58a2-efdc294af76e
2024-12-04T01:32:29.268Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: pipelining replication: peer="{Voter server-698cf176-f243-9f5a-fdef-b7b82d3a91d5 698cf176-f243-9f5a-fdef-b7b82d3a91d5}"
2024-12-04T01:32:29.269Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: pipelining replication: peer="{Voter server-5b7d0de1-44ea-ea9d-58a2-efdc294af76e 5b7d0de1-44ea-ea9d-58a2-efdc294af76e}"
2024-12-04T01:32:30.274Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:29.268779539 +0000 UTC m=+40.100622226 (1 nodes), 574.579915ms from start of poll, 575.127625ms from cluster start. Timeout at 2024-12-04 01:32:30.273790328 +0000 UTC m=+41.105633104, 1.005010878s after stability
2024-12-04T01:32:30.274Z [INFO]  creating new snapshot: path=/tmp/raft1096514151/snapshots/2-14-1733275950274.tmp
2024-12-04T01:32:30.275Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: copied to local snapshot: bytes=71
2024-12-04T01:32:30.275Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: snapshot restore progress: id=2-14-1733275950274 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:30.275Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: removing all old logs from log store
2024-12-04T01:32:30.275Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: compacting logs: from=1 to=2
2024-12-04T01:32:30.275Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: restored user snapshot: index=14
2024-12-04T01:32:30.275Z [ERROR] server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: failed to get log: index=2 error="log not found"
2024-12-04T01:32:30.275Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: aborting pipeline replication: peer="{Voter server-5b7d0de1-44ea-ea9d-58a2-efdc294af76e 5b7d0de1-44ea-ea9d-58a2-efdc294af76e}"
2024-12-04T01:32:30.275Z [ERROR] server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: failed to get log: index=2 error="log not found"
2024-12-04T01:32:30.275Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: aborting pipeline replication: peer="{Voter server-698cf176-f243-9f5a-fdef-b7b82d3a91d5 698cf176-f243-9f5a-fdef-b7b82d3a91d5}"
2024-12-04T01:32:30.282Z [ERROR] server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: failed to get log: index=2 error="log not found"
2024-12-04T01:32:30.283Z [INFO]  creating new snapshot: path=/tmp/raft2583738041/snapshots/2-14-1733275950283.tmp
2024-12-04T01:32:30.283Z [INFO]  server-5b7d0de1-44ea-ea9d-58a2-efdc294af76e: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:30.283Z [INFO]  server-5b7d0de1-44ea-ea9d-58a2-efdc294af76e: copied to local snapshot: bytes=71
2024-12-04T01:32:30.282Z [ERROR] server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: failed to get log: index=2 error="log not found"
2024-12-04T01:32:30.284Z [INFO]  creating new snapshot: path=/tmp/raft3836434387/snapshots/2-14-1733275950284.tmp
2024-12-04T01:32:30.284Z [INFO]  server-698cf176-f243-9f5a-fdef-b7b82d3a91d5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:30.284Z [INFO]  server-698cf176-f243-9f5a-fdef-b7b82d3a91d5: copied to local snapshot: bytes=71
2024-12-04T01:32:30.285Z [INFO]  server-5b7d0de1-44ea-ea9d-58a2-efdc294af76e: snapshot restore progress: id=2-14-1733275950283 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:30.285Z [INFO]  server-5b7d0de1-44ea-ea9d-58a2-efdc294af76e: removing all old logs from log store
2024-12-04T01:32:30.285Z [INFO]  server-5b7d0de1-44ea-ea9d-58a2-efdc294af76e: compacting logs: from=1 to=2
2024-12-04T01:32:30.285Z [INFO]  server-5b7d0de1-44ea-ea9d-58a2-efdc294af76e: Installed remote snapshot
2024-12-04T01:32:30.285Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: pipelining replication: peer="{Voter server-5b7d0de1-44ea-ea9d-58a2-efdc294af76e 5b7d0de1-44ea-ea9d-58a2-efdc294af76e}"
2024-12-04T01:32:30.285Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:30.285Z [INFO]  server-698cf176-f243-9f5a-fdef-b7b82d3a91d5: snapshot restore progress: id=2-14-1733275950284 last-index=14 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:30.285Z [INFO]  server-698cf176-f243-9f5a-fdef-b7b82d3a91d5: removing all old logs from log store
2024-12-04T01:32:30.285Z [INFO]  server-698cf176-f243-9f5a-fdef-b7b82d3a91d5: compacting logs: from=1 to=2
2024-12-04T01:32:30.285Z [INFO]  server-698cf176-f243-9f5a-fdef-b7b82d3a91d5: Installed remote snapshot
2024-12-04T01:32:30.285Z [INFO]  server-55513658-02d6-d8ac-72cf-e5d53c7d7e60: pipelining replication: peer="{Voter server-698cf176-f243-9f5a-fdef-b7b82d3a91d5 698cf176-f243-9f5a-fdef-b7b82d3a91d5}"
2024-12-04T01:32:31.291Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:30.285617317 +0000 UTC m=+41.117460086 (2 nodes), 2.358µs from start of poll, 1.591965485s from cluster start. Timeout at 2024-12-04 01:32:31.290649163 +0000 UTC m=+42.122491974, 1.005031888s after stability
2024-12-04T01:32:31.303Z [INFO]  server-c3f3c9a6-c6ea-63a6-4449-a5621556c399: aborting pipeline replication: peer="{Voter server-985705b5-69d6-9612-6bf1-7f5e77c58aa2 985705b5-69d6-9612-6bf1-7f5e77c58aa2}"
2024-12-04T01:32:31.303Z [INFO]  server-c3f3c9a6-c6ea-63a6-4449-a5621556c399: aborting pipeline replication: peer="{Voter server-dca907f5-d6b8-ddcd-ac91-f4f71c1492f2 dca907f5-d6b8-ddcd-ac91-f4f71c1492f2}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false
2024-12-04T01:32:31.308Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b Address:b5da9e4c-9071-14c0-a4b8-d6245fa7d71b} {Suffrage:Voter ID:server-2e822c87-9fdf-6de2-5cc5-21f566509631 Address:2e822c87-9fdf-6de2-5cc5-21f566509631} {Suffrage:Voter ID:server-c3c7a48e-f893-ae34-aff1-a93b782b4d3a Address:c3c7a48e-f893-ae34-aff1-a93b782b4d3a}]"
2024-12-04T01:32:31.308Z [INFO]  server-2e822c87-9fdf-6de2-5cc5-21f566509631: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b Address:b5da9e4c-9071-14c0-a4b8-d6245fa7d71b} {Suffrage:Voter ID:server-2e822c87-9fdf-6de2-5cc5-21f566509631 Address:2e822c87-9fdf-6de2-5cc5-21f566509631} {Suffrage:Voter ID:server-c3c7a48e-f893-ae34-aff1-a93b782b4d3a Address:c3c7a48e-f893-ae34-aff1-a93b782b4d3a}]"
2024-12-04T01:32:31.308Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: entering follower state: follower="Node at b5da9e4c-9071-14c0-a4b8-d6245fa7d71b [Follower]" leader-address= leader-id=
2024-12-04T01:32:31.308Z [INFO]  server-c3c7a48e-f893-ae34-aff1-a93b782b4d3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b Address:b5da9e4c-9071-14c0-a4b8-d6245fa7d71b} {Suffrage:Voter ID:server-2e822c87-9fdf-6de2-5cc5-21f566509631 Address:2e822c87-9fdf-6de2-5cc5-21f566509631} {Suffrage:Voter ID:server-c3c7a48e-f893-ae34-aff1-a93b782b4d3a Address:c3c7a48e-f893-ae34-aff1-a93b782b4d3a}]"
2024-12-04T01:32:31.308Z [INFO]  server-2e822c87-9fdf-6de2-5cc5-21f566509631: entering follower state: follower="Node at 2e822c87-9fdf-6de2-5cc5-21f566509631 [Follower]" leader-address= leader-id=
2024-12-04T01:32:31.308Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:31.308Z [INFO]  server-c3c7a48e-f893-ae34-aff1-a93b782b4d3a: entering follower state: follower="Node at c3c7a48e-f893-ae34-aff1-a93b782b4d3a [Follower]" leader-address= leader-id=
2024-12-04T01:32:32.074Z [WARN]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:32.074Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: entering candidate state: node="Node at b5da9e4c-9071-14c0-a4b8-d6245fa7d71b [Candidate]" term=2
2024-12-04T01:32:32.074Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: election won: term=2 tally=2
2024-12-04T01:32:32.074Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: entering leader state: leader="Node at b5da9e4c-9071-14c0-a4b8-d6245fa7d71b [Leader]"
2024-12-04T01:32:32.074Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: added peer, starting replication: peer=server-2e822c87-9fdf-6de2-5cc5-21f566509631
2024-12-04T01:32:32.074Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: added peer, starting replication: peer=server-c3c7a48e-f893-ae34-aff1-a93b782b4d3a
2024-12-04T01:32:32.074Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: pipelining replication: peer="{Voter server-c3c7a48e-f893-ae34-aff1-a93b782b4d3a c3c7a48e-f893-ae34-aff1-a93b782b4d3a}"
2024-12-04T01:32:32.075Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: pipelining replication: peer="{Voter server-2e822c87-9fdf-6de2-5cc5-21f566509631 2e822c87-9fdf-6de2-5cc5-21f566509631}"
2024-12-04T01:32:33.080Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:32.075265948 +0000 UTC m=+42.907108527 (1 nodes), 766.814325ms from start of poll, 767.297416ms from cluster start. Timeout at 2024-12-04 01:32:33.080268142 +0000 UTC m=+43.912110717, 1.00500219s after stability
2024-12-04T01:32:33.081Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: starting snapshot up to: index=12
2024-12-04T01:32:33.081Z [INFO]  creating new snapshot: path=/tmp/raft3616368818/snapshots/2-12-1733275953081.tmp
2024-12-04T01:32:33.082Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: snapshot complete up to: index=12
2024-12-04T01:32:33.083Z [INFO]  creating new snapshot: path=/tmp/raft3616368818/snapshots/2-23-1733275953083.tmp
2024-12-04T01:32:33.084Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: copied to local snapshot: bytes=71
2024-12-04T01:32:33.084Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: snapshot restore progress: id=2-23-1733275953083 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:33.084Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: restored user snapshot: index=23
2024-12-04T01:32:33.084Z [ERROR] server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: failed to get log: index=23 error="log not found"
2024-12-04T01:32:33.084Z [ERROR] server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: failed to get log: index=23 error="log not found"
2024-12-04T01:32:33.084Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: aborting pipeline replication: peer="{Voter server-c3c7a48e-f893-ae34-aff1-a93b782b4d3a c3c7a48e-f893-ae34-aff1-a93b782b4d3a}"
2024-12-04T01:32:33.084Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: aborting pipeline replication: peer="{Voter server-2e822c87-9fdf-6de2-5cc5-21f566509631 2e822c87-9fdf-6de2-5cc5-21f566509631}"
2024-12-04T01:32:33.092Z [ERROR] server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: failed to get log: index=23 error="log not found"
2024-12-04T01:32:33.093Z [INFO]  creating new snapshot: path=/tmp/raft616843157/snapshots/2-23-1733275953093.tmp
2024-12-04T01:32:33.093Z [INFO]  server-2e822c87-9fdf-6de2-5cc5-21f566509631: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:33.094Z [INFO]  server-2e822c87-9fdf-6de2-5cc5-21f566509631: copied to local snapshot: bytes=71
2024-12-04T01:32:33.094Z [ERROR] server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: failed to get log: index=23 error="log not found"
2024-12-04T01:32:33.094Z [INFO]  server-2e822c87-9fdf-6de2-5cc5-21f566509631: snapshot restore progress: id=2-23-1733275953093 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:33.094Z [INFO]  server-2e822c87-9fdf-6de2-5cc5-21f566509631: Installed remote snapshot
2024-12-04T01:32:33.094Z [WARN]  server-2e822c87-9fdf-6de2-5cc5-21f566509631: skipping application of old log: index=22
2024-12-04T01:32:33.094Z [INFO]  creating new snapshot: path=/tmp/raft2065088511/snapshots/2-23-1733275953094.tmp
2024-12-04T01:32:33.094Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: pipelining replication: peer="{Voter server-2e822c87-9fdf-6de2-5cc5-21f566509631 2e822c87-9fdf-6de2-5cc5-21f566509631}"
2024-12-04T01:32:33.094Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:33.095Z [INFO]  server-c3c7a48e-f893-ae34-aff1-a93b782b4d3a: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:33.095Z [INFO]  server-c3c7a48e-f893-ae34-aff1-a93b782b4d3a: copied to local snapshot: bytes=71
2024-12-04T01:32:33.095Z [INFO]  server-c3c7a48e-f893-ae34-aff1-a93b782b4d3a: snapshot restore progress: id=2-23-1733275953094 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:33.095Z [INFO]  server-c3c7a48e-f893-ae34-aff1-a93b782b4d3a: Installed remote snapshot
2024-12-04T01:32:33.095Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: pipelining replication: peer="{Voter server-c3c7a48e-f893-ae34-aff1-a93b782b4d3a c3c7a48e-f893-ae34-aff1-a93b782b4d3a}"
2024-12-04T01:32:34.100Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:33.094846575 +0000 UTC m=+43.926689329 (2 nodes), 2.091µs from start of poll, 1.786878218s from cluster start. Timeout at 2024-12-04 01:32:34.099881679 +0000 UTC m=+44.931724586, 1.005035257s after stability
2024-12-04T01:32:34.113Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: aborting pipeline replication: peer="{Voter server-c3c7a48e-f893-ae34-aff1-a93b782b4d3a c3c7a48e-f893-ae34-aff1-a93b782b4d3a}"
2024-12-04T01:32:34.113Z [INFO]  server-b5da9e4c-9071-14c0-a4b8-d6245fa7d71b: aborting pipeline replication: peer="{Voter server-2e822c87-9fdf-6de2-5cc5-21f566509631 2e822c87-9fdf-6de2-5cc5-21f566509631}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false
2024-12-04T01:32:34.115Z [INFO]  server-bfbbf2e1-a8ec-73de-3436-3b03fbafa453: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfbbf2e1-a8ec-73de-3436-3b03fbafa453 Address:bfbbf2e1-a8ec-73de-3436-3b03fbafa453} {Suffrage:Voter ID:server-eba18960-fb17-dfb6-7c06-723baf3c920f Address:eba18960-fb17-dfb6-7c06-723baf3c920f} {Suffrage:Voter ID:server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a Address:561c9bb7-e27b-01b2-2f9d-b79047e78b9a}]"
2024-12-04T01:32:34.115Z [INFO]  server-eba18960-fb17-dfb6-7c06-723baf3c920f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfbbf2e1-a8ec-73de-3436-3b03fbafa453 Address:bfbbf2e1-a8ec-73de-3436-3b03fbafa453} {Suffrage:Voter ID:server-eba18960-fb17-dfb6-7c06-723baf3c920f Address:eba18960-fb17-dfb6-7c06-723baf3c920f} {Suffrage:Voter ID:server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a Address:561c9bb7-e27b-01b2-2f9d-b79047e78b9a}]"
2024-12-04T01:32:34.115Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bfbbf2e1-a8ec-73de-3436-3b03fbafa453 Address:bfbbf2e1-a8ec-73de-3436-3b03fbafa453} {Suffrage:Voter ID:server-eba18960-fb17-dfb6-7c06-723baf3c920f Address:eba18960-fb17-dfb6-7c06-723baf3c920f} {Suffrage:Voter ID:server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a Address:561c9bb7-e27b-01b2-2f9d-b79047e78b9a}]"
2024-12-04T01:32:34.115Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:34.116Z [INFO]  server-bfbbf2e1-a8ec-73de-3436-3b03fbafa453: entering follower state: follower="Node at bfbbf2e1-a8ec-73de-3436-3b03fbafa453 [Follower]" leader-address= leader-id=
2024-12-04T01:32:34.116Z [INFO]  server-eba18960-fb17-dfb6-7c06-723baf3c920f: entering follower state: follower="Node at eba18960-fb17-dfb6-7c06-723baf3c920f [Follower]" leader-address= leader-id=
2024-12-04T01:32:34.116Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: entering follower state: follower="Node at 561c9bb7-e27b-01b2-2f9d-b79047e78b9a [Follower]" leader-address= leader-id=
2024-12-04T01:32:34.934Z [WARN]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:34.934Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: entering candidate state: node="Node at 561c9bb7-e27b-01b2-2f9d-b79047e78b9a [Candidate]" term=2
2024-12-04T01:32:34.934Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: election won: term=2 tally=2
2024-12-04T01:32:34.934Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: entering leader state: leader="Node at 561c9bb7-e27b-01b2-2f9d-b79047e78b9a [Leader]"
2024-12-04T01:32:34.934Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: added peer, starting replication: peer=server-bfbbf2e1-a8ec-73de-3436-3b03fbafa453
2024-12-04T01:32:34.934Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: added peer, starting replication: peer=server-eba18960-fb17-dfb6-7c06-723baf3c920f
2024-12-04T01:32:34.935Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: pipelining replication: peer="{Voter server-eba18960-fb17-dfb6-7c06-723baf3c920f eba18960-fb17-dfb6-7c06-723baf3c920f}"
2024-12-04T01:32:34.935Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: pipelining replication: peer="{Voter server-bfbbf2e1-a8ec-73de-3436-3b03fbafa453 bfbbf2e1-a8ec-73de-3436-3b03fbafa453}"
2024-12-04T01:32:35.940Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:34.935362728 +0000 UTC m=+45.767205417 (1 nodes), 819.350044ms from start of poll, 819.73186ms from cluster start. Timeout at 2024-12-04 01:32:35.940367294 +0000 UTC m=+46.772209871, 1.005004454s after stability
2024-12-04T01:32:35.941Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: starting snapshot up to: index=12
2024-12-04T01:32:35.941Z [INFO]  creating new snapshot: path=/tmp/raft3631319639/snapshots/2-12-1733275955941.tmp
2024-12-04T01:32:35.942Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: snapshot complete up to: index=12
2024-12-04T01:32:35.943Z [INFO]  creating new snapshot: path=/tmp/raft3631319639/snapshots/2-23-1733275955943.tmp
2024-12-04T01:32:35.944Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: copied to local snapshot: bytes=71
2024-12-04T01:32:35.944Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: snapshot restore progress: id=2-23-1733275955943 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:35.944Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: removing all old logs from log store
2024-12-04T01:32:35.944Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: compacting logs: from=1 to=22
2024-12-04T01:32:35.944Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: restored user snapshot: index=23
2024-12-04T01:32:35.944Z [ERROR] server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: failed to get log: index=22 error="log not found"
2024-12-04T01:32:35.944Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: aborting pipeline replication: peer="{Voter server-eba18960-fb17-dfb6-7c06-723baf3c920f eba18960-fb17-dfb6-7c06-723baf3c920f}"
2024-12-04T01:32:35.944Z [ERROR] server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: failed to get log: index=22 error="log not found"
2024-12-04T01:32:35.944Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: aborting pipeline replication: peer="{Voter server-bfbbf2e1-a8ec-73de-3436-3b03fbafa453 bfbbf2e1-a8ec-73de-3436-3b03fbafa453}"
2024-12-04T01:32:35.951Z [ERROR] server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: failed to get log: index=22 error="log not found"
2024-12-04T01:32:35.951Z [INFO]  creating new snapshot: path=/tmp/raft3095770263/snapshots/2-23-1733275955951.tmp
2024-12-04T01:32:35.952Z [INFO]  server-bfbbf2e1-a8ec-73de-3436-3b03fbafa453: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:35.952Z [INFO]  server-bfbbf2e1-a8ec-73de-3436-3b03fbafa453: copied to local snapshot: bytes=71
2024-12-04T01:32:35.952Z [INFO]  server-bfbbf2e1-a8ec-73de-3436-3b03fbafa453: snapshot restore progress: id=2-23-1733275955951 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:35.952Z [INFO]  server-bfbbf2e1-a8ec-73de-3436-3b03fbafa453: removing all old logs from log store
2024-12-04T01:32:35.952Z [INFO]  server-bfbbf2e1-a8ec-73de-3436-3b03fbafa453: compacting logs: from=1 to=22
2024-12-04T01:32:35.952Z [INFO]  server-bfbbf2e1-a8ec-73de-3436-3b03fbafa453: Installed remote snapshot
2024-12-04T01:32:35.952Z [WARN]  server-bfbbf2e1-a8ec-73de-3436-3b03fbafa453: skipping application of old log: index=22
2024-12-04T01:32:35.952Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: pipelining replication: peer="{Voter server-bfbbf2e1-a8ec-73de-3436-3b03fbafa453 bfbbf2e1-a8ec-73de-3436-3b03fbafa453}"
2024-12-04T01:32:35.952Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:35.954Z [ERROR] server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: failed to get log: index=22 error="log not found"
2024-12-04T01:32:35.954Z [INFO]  creating new snapshot: path=/tmp/raft1294164258/snapshots/2-23-1733275955954.tmp
2024-12-04T01:32:35.954Z [INFO]  server-eba18960-fb17-dfb6-7c06-723baf3c920f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:35.955Z [INFO]  server-eba18960-fb17-dfb6-7c06-723baf3c920f: copied to local snapshot: bytes=71
2024-12-04T01:32:35.958Z [INFO]  server-eba18960-fb17-dfb6-7c06-723baf3c920f: snapshot restore progress: id=2-23-1733275955954 last-index=23 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:35.958Z [INFO]  server-eba18960-fb17-dfb6-7c06-723baf3c920f: removing all old logs from log store
2024-12-04T01:32:35.958Z [INFO]  server-eba18960-fb17-dfb6-7c06-723baf3c920f: compacting logs: from=1 to=22
2024-12-04T01:32:35.958Z [INFO]  server-eba18960-fb17-dfb6-7c06-723baf3c920f: Installed remote snapshot
2024-12-04T01:32:35.959Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: pipelining replication: peer="{Voter server-eba18960-fb17-dfb6-7c06-723baf3c920f eba18960-fb17-dfb6-7c06-723baf3c920f}"
2024-12-04T01:32:36.958Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:35.952919072 +0000 UTC m=+46.784761747 (2 nodes), 2.694µs from start of poll, 1.83728819s from cluster start. Timeout at 2024-12-04 01:32:36.957951881 +0000 UTC m=+47.789794892, 1.005033145s after stability
2024-12-04T01:32:36.971Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: aborting pipeline replication: peer="{Voter server-eba18960-fb17-dfb6-7c06-723baf3c920f eba18960-fb17-dfb6-7c06-723baf3c920f}"
2024-12-04T01:32:36.972Z [INFO]  server-561c9bb7-e27b-01b2-2f9d-b79047e78b9a: aborting pipeline replication: peer="{Voter server-bfbbf2e1-a8ec-73de-3436-3b03fbafa453 bfbbf2e1-a8ec-73de-3436-3b03fbafa453}"
=== RUN   TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true
2024-12-04T01:32:36.973Z [INFO]  server-fb2c9ba9-3484-8318-ccbb-0236fd73746e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb2c9ba9-3484-8318-ccbb-0236fd73746e Address:fb2c9ba9-3484-8318-ccbb-0236fd73746e} {Suffrage:Voter ID:server-7cccf1d7-9d13-139c-2b98-489c3ffc3905 Address:7cccf1d7-9d13-139c-2b98-489c3ffc3905} {Suffrage:Voter ID:server-8ba91a1a-53cf-aae2-c603-7ae024b960c8 Address:8ba91a1a-53cf-aae2-c603-7ae024b960c8}]"
2024-12-04T01:32:36.973Z [INFO]  server-7cccf1d7-9d13-139c-2b98-489c3ffc3905: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb2c9ba9-3484-8318-ccbb-0236fd73746e Address:fb2c9ba9-3484-8318-ccbb-0236fd73746e} {Suffrage:Voter ID:server-7cccf1d7-9d13-139c-2b98-489c3ffc3905 Address:7cccf1d7-9d13-139c-2b98-489c3ffc3905} {Suffrage:Voter ID:server-8ba91a1a-53cf-aae2-c603-7ae024b960c8 Address:8ba91a1a-53cf-aae2-c603-7ae024b960c8}]"
2024-12-04T01:32:36.974Z [INFO]  server-8ba91a1a-53cf-aae2-c603-7ae024b960c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fb2c9ba9-3484-8318-ccbb-0236fd73746e Address:fb2c9ba9-3484-8318-ccbb-0236fd73746e} {Suffrage:Voter ID:server-7cccf1d7-9d13-139c-2b98-489c3ffc3905 Address:7cccf1d7-9d13-139c-2b98-489c3ffc3905} {Suffrage:Voter ID:server-8ba91a1a-53cf-aae2-c603-7ae024b960c8 Address:8ba91a1a-53cf-aae2-c603-7ae024b960c8}]"
2024-12-04T01:32:36.974Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:36.974Z [INFO]  server-fb2c9ba9-3484-8318-ccbb-0236fd73746e: entering follower state: follower="Node at fb2c9ba9-3484-8318-ccbb-0236fd73746e [Follower]" leader-address= leader-id=
2024-12-04T01:32:36.974Z [INFO]  server-7cccf1d7-9d13-139c-2b98-489c3ffc3905: entering follower state: follower="Node at 7cccf1d7-9d13-139c-2b98-489c3ffc3905 [Follower]" leader-address= leader-id=
2024-12-04T01:32:36.974Z [INFO]  server-8ba91a1a-53cf-aae2-c603-7ae024b960c8: entering follower state: follower="Node at 8ba91a1a-53cf-aae2-c603-7ae024b960c8 [Follower]" leader-address= leader-id=
2024-12-04T01:32:37.677Z [WARN]  server-8ba91a1a-53cf-aae2-c603-7ae024b960c8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:37.677Z [INFO]  server-8ba91a1a-53cf-aae2-c603-7ae024b960c8: entering candidate state: node="Node at 8ba91a1a-53cf-aae2-c603-7ae024b960c8 [Candidate]" term=2
2024-12-04T01:32:37.677Z [INFO]  server-8ba91a1a-53cf-aae2-c603-7ae024b960c8: election won: term=2 tally=2
2024-12-04T01:32:37.677Z [INFO]  server-8ba91a1a-53cf-aae2-c603-7ae024b960c8: entering leader state: leader="Node at 8ba91a1a-53cf-aae2-c603-7ae024b960c8 [Leader]"
2024-12-04T01:32:37.677Z [INFO]  server-8ba91a1a-53cf-aae2-c603-7ae024b960c8: added peer, starting replication: peer=server-fb2c9ba9-3484-8318-ccbb-0236fd73746e
2024-12-04T01:32:37.677Z [INFO]  server-8ba91a1a-53cf-aae2-c603-7ae024b960c8: added peer, starting replication: peer=server-7cccf1d7-9d13-139c-2b98-489c3ffc3905
2024-12-04T01:32:37.677Z [INFO]  server-8ba91a1a-53cf-aae2-c603-7ae024b960c8: pipelining replication: peer="{Voter server-7cccf1d7-9d13-139c-2b98-489c3ffc3905 7cccf1d7-9d13-139c-2b98-489c3ffc3905}"
2024-12-04T01:32:37.677Z [INFO]  server-8ba91a1a-53cf-aae2-c603-7ae024b960c8: pipelining replication: peer="{Voter server-fb2c9ba9-3484-8318-ccbb-0236fd73746e fb2c9ba9-3484-8318-ccbb-0236fd73746e}"
2024-12-04T01:32:38.683Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:37.677737283 +0000 UTC m=+48.509579975 (1 nodes), 703.562585ms from start of poll, 703.978968ms from cluster start. Timeout at 2024-12-04 01:32:38.682747698 +0000 UTC m=+49.514590253, 1.005010278s after stability
2024-12-04T01:32:38.684Z [INFO]  server-8ba91a1a-53cf-aae2-c603-7ae024b960c8: starting snapshot up to: index=12
2024-12-04T01:32:38.684Z [INFO]  creating new snapshot: path=/tmp/raft3783114333/snapshots/2-12-1733275958684.tmp
2024-12-04T01:32:38.713Z [INFO]  server-8ba91a1a-53cf-aae2-c603-7ae024b960c8: snapshot complete up to: index=12
2024-12-04T01:32:38.715Z [INFO]  server-4cd2493b-29f4-36d8-6490-ecbaabd1b080: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cd2493b-29f4-36d8-6490-ecbaabd1b080 Address:4cd2493b-29f4-36d8-6490-ecbaabd1b080} {Suffrage:Voter ID:server-b27633d4-d509-6af4-9b23-e366836008ba Address:b27633d4-d509-6af4-9b23-e366836008ba} {Suffrage:Voter ID:server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1 Address:335d7b03-5dc3-cc83-af3a-a40a5bac70c1}]"
2024-12-04T01:32:38.715Z [INFO]  server-b27633d4-d509-6af4-9b23-e366836008ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cd2493b-29f4-36d8-6490-ecbaabd1b080 Address:4cd2493b-29f4-36d8-6490-ecbaabd1b080} {Suffrage:Voter ID:server-b27633d4-d509-6af4-9b23-e366836008ba Address:b27633d4-d509-6af4-9b23-e366836008ba} {Suffrage:Voter ID:server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1 Address:335d7b03-5dc3-cc83-af3a-a40a5bac70c1}]"
2024-12-04T01:32:38.715Z [INFO]  server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cd2493b-29f4-36d8-6490-ecbaabd1b080 Address:4cd2493b-29f4-36d8-6490-ecbaabd1b080} {Suffrage:Voter ID:server-b27633d4-d509-6af4-9b23-e366836008ba Address:b27633d4-d509-6af4-9b23-e366836008ba} {Suffrage:Voter ID:server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1 Address:335d7b03-5dc3-cc83-af3a-a40a5bac70c1}]"
2024-12-04T01:32:38.715Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:38.715Z [INFO]  server-4cd2493b-29f4-36d8-6490-ecbaabd1b080: entering follower state: follower="Node at 4cd2493b-29f4-36d8-6490-ecbaabd1b080 [Follower]" leader-address= leader-id=
2024-12-04T01:32:38.715Z [INFO]  server-b27633d4-d509-6af4-9b23-e366836008ba: entering follower state: follower="Node at b27633d4-d509-6af4-9b23-e366836008ba [Follower]" leader-address= leader-id=
2024-12-04T01:32:38.715Z [INFO]  server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: entering follower state: follower="Node at 335d7b03-5dc3-cc83-af3a-a40a5bac70c1 [Follower]" leader-address= leader-id=
2024-12-04T01:32:39.361Z [WARN]  server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:39.361Z [INFO]  server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: entering candidate state: node="Node at 335d7b03-5dc3-cc83-af3a-a40a5bac70c1 [Candidate]" term=2
2024-12-04T01:32:39.362Z [INFO]  server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: election won: term=2 tally=2
2024-12-04T01:32:39.362Z [INFO]  server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: entering leader state: leader="Node at 335d7b03-5dc3-cc83-af3a-a40a5bac70c1 [Leader]"
2024-12-04T01:32:39.362Z [INFO]  server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: added peer, starting replication: peer=server-4cd2493b-29f4-36d8-6490-ecbaabd1b080
2024-12-04T01:32:39.362Z [INFO]  server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: added peer, starting replication: peer=server-b27633d4-d509-6af4-9b23-e366836008ba
2024-12-04T01:32:39.362Z [INFO]  server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: pipelining replication: peer="{Voter server-b27633d4-d509-6af4-9b23-e366836008ba b27633d4-d509-6af4-9b23-e366836008ba}"
2024-12-04T01:32:39.362Z [INFO]  server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: pipelining replication: peer="{Voter server-4cd2493b-29f4-36d8-6490-ecbaabd1b080 4cd2493b-29f4-36d8-6490-ecbaabd1b080}"
2024-12-04T01:32:40.367Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:39.362258988 +0000 UTC m=+50.194101760 (1 nodes), 646.559289ms from start of poll, 646.951137ms from cluster start. Timeout at 2024-12-04 01:32:40.367269243 +0000 UTC m=+51.199111808, 1.005010048s after stability
2024-12-04T01:32:40.367Z [INFO]  creating new snapshot: path=/tmp/raft3173323546/snapshots/2-15-1733275960367.tmp
2024-12-04T01:32:40.368Z [INFO]  server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: copied to local snapshot: bytes=71
2024-12-04T01:32:40.374Z [INFO]  server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: snapshot restore progress: id=2-15-1733275960367 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:40.374Z [INFO]  server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: restored user snapshot: index=15
2024-12-04T01:32:40.375Z [ERROR] server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: failed to get log: index=3 error="log not found"
2024-12-04T01:32:40.375Z [INFO]  server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: aborting pipeline replication: peer="{Voter server-b27633d4-d509-6af4-9b23-e366836008ba b27633d4-d509-6af4-9b23-e366836008ba}"
2024-12-04T01:32:40.375Z [ERROR] server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: failed to get log: index=3 error="log not found"
2024-12-04T01:32:40.375Z [INFO]  server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: aborting pipeline replication: peer="{Voter server-4cd2493b-29f4-36d8-6490-ecbaabd1b080 4cd2493b-29f4-36d8-6490-ecbaabd1b080}"
2024-12-04T01:32:40.382Z [ERROR] server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: failed to get log: index=3 error="log not found"
2024-12-04T01:32:40.383Z [INFO]  creating new snapshot: path=/tmp/raft1009601780/snapshots/2-15-1733275960383.tmp
2024-12-04T01:32:40.383Z [INFO]  server-b27633d4-d509-6af4-9b23-e366836008ba: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:40.383Z [INFO]  server-b27633d4-d509-6af4-9b23-e366836008ba: copied to local snapshot: bytes=71
2024-12-04T01:32:40.384Z [INFO]  server-b27633d4-d509-6af4-9b23-e366836008ba: snapshot restore progress: id=2-15-1733275960383 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:40.384Z [INFO]  server-b27633d4-d509-6af4-9b23-e366836008ba: Installed remote snapshot
2024-12-04T01:32:40.384Z [INFO]  server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: pipelining replication: peer="{Voter server-b27633d4-d509-6af4-9b23-e366836008ba b27633d4-d509-6af4-9b23-e366836008ba}"
2024-12-04T01:32:40.384Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:40.384Z [ERROR] server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: failed to get log: index=3 error="log not found"
2024-12-04T01:32:40.385Z [INFO]  creating new snapshot: path=/tmp/raft2189105975/snapshots/2-15-1733275960385.tmp
2024-12-04T01:32:40.385Z [INFO]  server-4cd2493b-29f4-36d8-6490-ecbaabd1b080: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:40.385Z [INFO]  server-4cd2493b-29f4-36d8-6490-ecbaabd1b080: copied to local snapshot: bytes=71
2024-12-04T01:32:40.386Z [INFO]  server-4cd2493b-29f4-36d8-6490-ecbaabd1b080: snapshot restore progress: id=2-15-1733275960385 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:40.386Z [INFO]  server-4cd2493b-29f4-36d8-6490-ecbaabd1b080: Installed remote snapshot
2024-12-04T01:32:40.386Z [INFO]  server-335d7b03-5dc3-cc83-af3a-a40a5bac70c1: pipelining replication: peer="{Voter server-4cd2493b-29f4-36d8-6490-ecbaabd1b080 4cd2493b-29f4-36d8-6490-ecbaabd1b080}"
2024-12-04T01:32:41.389Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:40.384563613 +0000 UTC m=+51.216406440 (2 nodes), 2.417µs from start of poll, 1.669255817s from cluster start. Timeout at 2024-12-04 01:32:41.389600444 +0000 UTC m=+52.221443010, 1.00503657s after stability
2024-12-04T01:32:41.401Z [INFO]  server-8ba91a1a-53cf-aae2-c603-7ae024b960c8: aborting pipeline replication: peer="{Voter server-fb2c9ba9-3484-8318-ccbb-0236fd73746e fb2c9ba9-3484-8318-ccbb-0236fd73746e}"
2024-12-04T01:32:41.401Z [INFO]  server-8ba91a1a-53cf-aae2-c603-7ae024b960c8: aborting pipeline replication: peer="{Voter server-7cccf1d7-9d13-139c-2b98-489c3ffc3905 7cccf1d7-9d13-139c-2b98-489c3ffc3905}"
=== RUN   TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true
2024-12-04T01:32:41.403Z [INFO]  server-8169796c-0079-e4e0-457d-386b19325adb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8169796c-0079-e4e0-457d-386b19325adb Address:8169796c-0079-e4e0-457d-386b19325adb} {Suffrage:Voter ID:server-5999c552-1bec-c6f9-5b92-cbf748719bec Address:5999c552-1bec-c6f9-5b92-cbf748719bec} {Suffrage:Voter ID:server-667a1fab-c002-4b85-dfc5-b6ce2334bbb9 Address:667a1fab-c002-4b85-dfc5-b6ce2334bbb9}]"
2024-12-04T01:32:41.403Z [INFO]  server-5999c552-1bec-c6f9-5b92-cbf748719bec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8169796c-0079-e4e0-457d-386b19325adb Address:8169796c-0079-e4e0-457d-386b19325adb} {Suffrage:Voter ID:server-5999c552-1bec-c6f9-5b92-cbf748719bec Address:5999c552-1bec-c6f9-5b92-cbf748719bec} {Suffrage:Voter ID:server-667a1fab-c002-4b85-dfc5-b6ce2334bbb9 Address:667a1fab-c002-4b85-dfc5-b6ce2334bbb9}]"
2024-12-04T01:32:41.403Z [INFO]  server-667a1fab-c002-4b85-dfc5-b6ce2334bbb9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8169796c-0079-e4e0-457d-386b19325adb Address:8169796c-0079-e4e0-457d-386b19325adb} {Suffrage:Voter ID:server-5999c552-1bec-c6f9-5b92-cbf748719bec Address:5999c552-1bec-c6f9-5b92-cbf748719bec} {Suffrage:Voter ID:server-667a1fab-c002-4b85-dfc5-b6ce2334bbb9 Address:667a1fab-c002-4b85-dfc5-b6ce2334bbb9}]"
2024-12-04T01:32:41.403Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:41.404Z [INFO]  server-8169796c-0079-e4e0-457d-386b19325adb: entering follower state: follower="Node at 8169796c-0079-e4e0-457d-386b19325adb [Follower]" leader-address= leader-id=
2024-12-04T01:32:41.404Z [INFO]  server-5999c552-1bec-c6f9-5b92-cbf748719bec: entering follower state: follower="Node at 5999c552-1bec-c6f9-5b92-cbf748719bec [Follower]" leader-address= leader-id=
2024-12-04T01:32:41.404Z [INFO]  server-667a1fab-c002-4b85-dfc5-b6ce2334bbb9: entering follower state: follower="Node at 667a1fab-c002-4b85-dfc5-b6ce2334bbb9 [Follower]" leader-address= leader-id=
2024-12-04T01:32:41.961Z [WARN]  server-667a1fab-c002-4b85-dfc5-b6ce2334bbb9: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:41.961Z [INFO]  server-667a1fab-c002-4b85-dfc5-b6ce2334bbb9: entering candidate state: node="Node at 667a1fab-c002-4b85-dfc5-b6ce2334bbb9 [Candidate]" term=2
2024-12-04T01:32:41.962Z [INFO]  server-667a1fab-c002-4b85-dfc5-b6ce2334bbb9: election won: term=2 tally=2
2024-12-04T01:32:41.962Z [INFO]  server-667a1fab-c002-4b85-dfc5-b6ce2334bbb9: entering leader state: leader="Node at 667a1fab-c002-4b85-dfc5-b6ce2334bbb9 [Leader]"
2024-12-04T01:32:41.962Z [INFO]  server-667a1fab-c002-4b85-dfc5-b6ce2334bbb9: added peer, starting replication: peer=server-8169796c-0079-e4e0-457d-386b19325adb
2024-12-04T01:32:41.962Z [INFO]  server-667a1fab-c002-4b85-dfc5-b6ce2334bbb9: added peer, starting replication: peer=server-5999c552-1bec-c6f9-5b92-cbf748719bec
2024-12-04T01:32:41.962Z [INFO]  server-667a1fab-c002-4b85-dfc5-b6ce2334bbb9: pipelining replication: peer="{Voter server-5999c552-1bec-c6f9-5b92-cbf748719bec 5999c552-1bec-c6f9-5b92-cbf748719bec}"
2024-12-04T01:32:41.962Z [INFO]  server-667a1fab-c002-4b85-dfc5-b6ce2334bbb9: pipelining replication: peer="{Voter server-8169796c-0079-e4e0-457d-386b19325adb 8169796c-0079-e4e0-457d-386b19325adb}"
2024-12-04T01:32:42.967Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:41.962483767 +0000 UTC m=+52.794326618 (1 nodes), 558.487642ms from start of poll, 558.852392ms from cluster start. Timeout at 2024-12-04 01:32:42.967490966 +0000 UTC m=+53.799333551, 1.005006933s after stability
2024-12-04T01:32:42.968Z [INFO]  server-667a1fab-c002-4b85-dfc5-b6ce2334bbb9: starting snapshot up to: index=12
2024-12-04T01:32:42.968Z [INFO]  creating new snapshot: path=/tmp/raft1902415029/snapshots/2-12-1733275962968.tmp
2024-12-04T01:32:42.974Z [INFO]  server-667a1fab-c002-4b85-dfc5-b6ce2334bbb9: snapshot complete up to: index=12
2024-12-04T01:32:42.975Z [INFO]  server-d1c659fc-7197-85d8-c98f-ae81c3ef6987: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d1c659fc-7197-85d8-c98f-ae81c3ef6987 Address:d1c659fc-7197-85d8-c98f-ae81c3ef6987} {Suffrage:Voter ID:server-e8617c35-271d-6e15-ea59-c30dd771d791 Address:e8617c35-271d-6e15-ea59-c30dd771d791} {Suffrage:Voter ID:server-d4a5366e-8b4d-cc41-d14d-87f293a4f128 Address:d4a5366e-8b4d-cc41-d14d-87f293a4f128}]"
2024-12-04T01:32:42.976Z [INFO]  server-e8617c35-271d-6e15-ea59-c30dd771d791: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d1c659fc-7197-85d8-c98f-ae81c3ef6987 Address:d1c659fc-7197-85d8-c98f-ae81c3ef6987} {Suffrage:Voter ID:server-e8617c35-271d-6e15-ea59-c30dd771d791 Address:e8617c35-271d-6e15-ea59-c30dd771d791} {Suffrage:Voter ID:server-d4a5366e-8b4d-cc41-d14d-87f293a4f128 Address:d4a5366e-8b4d-cc41-d14d-87f293a4f128}]"
2024-12-04T01:32:42.976Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d1c659fc-7197-85d8-c98f-ae81c3ef6987 Address:d1c659fc-7197-85d8-c98f-ae81c3ef6987} {Suffrage:Voter ID:server-e8617c35-271d-6e15-ea59-c30dd771d791 Address:e8617c35-271d-6e15-ea59-c30dd771d791} {Suffrage:Voter ID:server-d4a5366e-8b4d-cc41-d14d-87f293a4f128 Address:d4a5366e-8b4d-cc41-d14d-87f293a4f128}]"
2024-12-04T01:32:42.976Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:42.976Z [INFO]  server-d1c659fc-7197-85d8-c98f-ae81c3ef6987: entering follower state: follower="Node at d1c659fc-7197-85d8-c98f-ae81c3ef6987 [Follower]" leader-address= leader-id=
2024-12-04T01:32:42.976Z [INFO]  server-e8617c35-271d-6e15-ea59-c30dd771d791: entering follower state: follower="Node at e8617c35-271d-6e15-ea59-c30dd771d791 [Follower]" leader-address= leader-id=
2024-12-04T01:32:42.976Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: entering follower state: follower="Node at d4a5366e-8b4d-cc41-d14d-87f293a4f128 [Follower]" leader-address= leader-id=
2024-12-04T01:32:43.545Z [WARN]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:43.546Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: entering candidate state: node="Node at d4a5366e-8b4d-cc41-d14d-87f293a4f128 [Candidate]" term=2
2024-12-04T01:32:43.546Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: election won: term=2 tally=2
2024-12-04T01:32:43.546Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: entering leader state: leader="Node at d4a5366e-8b4d-cc41-d14d-87f293a4f128 [Leader]"
2024-12-04T01:32:43.546Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: added peer, starting replication: peer=server-d1c659fc-7197-85d8-c98f-ae81c3ef6987
2024-12-04T01:32:43.546Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: added peer, starting replication: peer=server-e8617c35-271d-6e15-ea59-c30dd771d791
2024-12-04T01:32:43.546Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: pipelining replication: peer="{Voter server-e8617c35-271d-6e15-ea59-c30dd771d791 e8617c35-271d-6e15-ea59-c30dd771d791}"
2024-12-04T01:32:43.546Z [WARN]  server-e8617c35-271d-6e15-ea59-c30dd771d791: rejecting vote request since our last term is greater: candidate=d4a5366e-8b4d-cc41-d14d-87f293a4f128 last-term=2 last-candidate-term=1
2024-12-04T01:32:43.546Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: pipelining replication: peer="{Voter server-d1c659fc-7197-85d8-c98f-ae81c3ef6987 d1c659fc-7197-85d8-c98f-ae81c3ef6987}"
2024-12-04T01:32:44.552Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:43.546806022 +0000 UTC m=+54.378648698 (1 nodes), 570.663969ms from start of poll, 571.011344ms from cluster start. Timeout at 2024-12-04 01:32:44.551810167 +0000 UTC m=+55.383652760, 1.005004062s after stability
2024-12-04T01:32:44.553Z [INFO]  creating new snapshot: path=/tmp/raft2249911889/snapshots/2-15-1733275964553.tmp
2024-12-04T01:32:44.554Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: copied to local snapshot: bytes=71
2024-12-04T01:32:44.554Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: snapshot restore progress: id=2-15-1733275964553 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:44.554Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: removing all old logs from log store
2024-12-04T01:32:44.554Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: compacting logs: from=1 to=2
2024-12-04T01:32:44.554Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: restored user snapshot: index=15
2024-12-04T01:32:44.554Z [ERROR] server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: failed to get log: index=2 error="log not found"
2024-12-04T01:32:44.554Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: aborting pipeline replication: peer="{Voter server-e8617c35-271d-6e15-ea59-c30dd771d791 e8617c35-271d-6e15-ea59-c30dd771d791}"
2024-12-04T01:32:44.554Z [ERROR] server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: failed to get log: index=2 error="log not found"
2024-12-04T01:32:44.554Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: aborting pipeline replication: peer="{Voter server-d1c659fc-7197-85d8-c98f-ae81c3ef6987 d1c659fc-7197-85d8-c98f-ae81c3ef6987}"
2024-12-04T01:32:44.561Z [ERROR] server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: failed to get log: index=2 error="log not found"
2024-12-04T01:32:44.562Z [INFO]  creating new snapshot: path=/tmp/raft4292976709/snapshots/2-15-1733275964562.tmp
2024-12-04T01:32:44.562Z [INFO]  server-e8617c35-271d-6e15-ea59-c30dd771d791: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:44.562Z [INFO]  server-e8617c35-271d-6e15-ea59-c30dd771d791: copied to local snapshot: bytes=71
2024-12-04T01:32:44.563Z [INFO]  server-e8617c35-271d-6e15-ea59-c30dd771d791: snapshot restore progress: id=2-15-1733275964562 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:44.563Z [INFO]  server-e8617c35-271d-6e15-ea59-c30dd771d791: removing all old logs from log store
2024-12-04T01:32:44.563Z [INFO]  server-e8617c35-271d-6e15-ea59-c30dd771d791: compacting logs: from=1 to=2
2024-12-04T01:32:44.563Z [INFO]  server-e8617c35-271d-6e15-ea59-c30dd771d791: Installed remote snapshot
2024-12-04T01:32:44.563Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: pipelining replication: peer="{Voter server-e8617c35-271d-6e15-ea59-c30dd771d791 e8617c35-271d-6e15-ea59-c30dd771d791}"
2024-12-04T01:32:44.563Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:44.563Z [ERROR] server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: failed to get log: index=2 error="log not found"
2024-12-04T01:32:44.563Z [INFO]  creating new snapshot: path=/tmp/raft3893385076/snapshots/2-15-1733275964563.tmp
2024-12-04T01:32:44.564Z [INFO]  server-d1c659fc-7197-85d8-c98f-ae81c3ef6987: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:44.564Z [INFO]  server-d1c659fc-7197-85d8-c98f-ae81c3ef6987: copied to local snapshot: bytes=71
2024-12-04T01:32:44.564Z [INFO]  server-d1c659fc-7197-85d8-c98f-ae81c3ef6987: snapshot restore progress: id=2-15-1733275964563 last-index=15 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:44.564Z [INFO]  server-d1c659fc-7197-85d8-c98f-ae81c3ef6987: removing all old logs from log store
2024-12-04T01:32:44.564Z [INFO]  server-d1c659fc-7197-85d8-c98f-ae81c3ef6987: compacting logs: from=1 to=2
2024-12-04T01:32:44.564Z [INFO]  server-d1c659fc-7197-85d8-c98f-ae81c3ef6987: Installed remote snapshot
2024-12-04T01:32:44.565Z [INFO]  server-d4a5366e-8b4d-cc41-d14d-87f293a4f128: pipelining replication: peer="{Voter server-d1c659fc-7197-85d8-c98f-ae81c3ef6987 d1c659fc-7197-85d8-c98f-ae81c3ef6987}"
2024-12-04T01:32:45.569Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:44.563482959 +0000 UTC m=+55.395325528 (2 nodes), 1.828µs from start of poll, 1.587688174s from cluster start. Timeout at 2024-12-04 01:32:45.568535996 +0000 UTC m=+56.400380622, 1.005055094s after stability
2024-12-04T01:32:45.582Z [INFO]  server-667a1fab-c002-4b85-dfc5-b6ce2334bbb9: aborting pipeline replication: peer="{Voter server-5999c552-1bec-c6f9-5b92-cbf748719bec 5999c552-1bec-c6f9-5b92-cbf748719bec}"
2024-12-04T01:32:45.582Z [INFO]  server-667a1fab-c002-4b85-dfc5-b6ce2334bbb9: aborting pipeline replication: peer="{Voter server-8169796c-0079-e4e0-457d-386b19325adb 8169796c-0079-e4e0-457d-386b19325adb}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false
2024-12-04T01:32:45.584Z [INFO]  server-602229ad-3d5a-5f8b-a50a-0166d7fca6b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-602229ad-3d5a-5f8b-a50a-0166d7fca6b5 Address:602229ad-3d5a-5f8b-a50a-0166d7fca6b5} {Suffrage:Voter ID:server-c28147ed-01a0-34f3-9fb8-b0b56e7fe488 Address:c28147ed-01a0-34f3-9fb8-b0b56e7fe488} {Suffrage:Voter ID:server-6828deef-3ee7-d7ae-e32b-fc41686e51f8 Address:6828deef-3ee7-d7ae-e32b-fc41686e51f8}]"
2024-12-04T01:32:45.584Z [INFO]  server-c28147ed-01a0-34f3-9fb8-b0b56e7fe488: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-602229ad-3d5a-5f8b-a50a-0166d7fca6b5 Address:602229ad-3d5a-5f8b-a50a-0166d7fca6b5} {Suffrage:Voter ID:server-c28147ed-01a0-34f3-9fb8-b0b56e7fe488 Address:c28147ed-01a0-34f3-9fb8-b0b56e7fe488} {Suffrage:Voter ID:server-6828deef-3ee7-d7ae-e32b-fc41686e51f8 Address:6828deef-3ee7-d7ae-e32b-fc41686e51f8}]"
2024-12-04T01:32:45.584Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-602229ad-3d5a-5f8b-a50a-0166d7fca6b5 Address:602229ad-3d5a-5f8b-a50a-0166d7fca6b5} {Suffrage:Voter ID:server-c28147ed-01a0-34f3-9fb8-b0b56e7fe488 Address:c28147ed-01a0-34f3-9fb8-b0b56e7fe488} {Suffrage:Voter ID:server-6828deef-3ee7-d7ae-e32b-fc41686e51f8 Address:6828deef-3ee7-d7ae-e32b-fc41686e51f8}]"
2024-12-04T01:32:45.584Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:45.584Z [INFO]  server-602229ad-3d5a-5f8b-a50a-0166d7fca6b5: entering follower state: follower="Node at 602229ad-3d5a-5f8b-a50a-0166d7fca6b5 [Follower]" leader-address= leader-id=
2024-12-04T01:32:45.584Z [INFO]  server-c28147ed-01a0-34f3-9fb8-b0b56e7fe488: entering follower state: follower="Node at c28147ed-01a0-34f3-9fb8-b0b56e7fe488 [Follower]" leader-address= leader-id=
2024-12-04T01:32:45.584Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: entering follower state: follower="Node at 6828deef-3ee7-d7ae-e32b-fc41686e51f8 [Follower]" leader-address= leader-id=
2024-12-04T01:32:46.169Z [WARN]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:46.169Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: entering candidate state: node="Node at 6828deef-3ee7-d7ae-e32b-fc41686e51f8 [Candidate]" term=2
2024-12-04T01:32:46.169Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: election won: term=2 tally=2
2024-12-04T01:32:46.169Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: entering leader state: leader="Node at 6828deef-3ee7-d7ae-e32b-fc41686e51f8 [Leader]"
2024-12-04T01:32:46.169Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: added peer, starting replication: peer=server-602229ad-3d5a-5f8b-a50a-0166d7fca6b5
2024-12-04T01:32:46.169Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: added peer, starting replication: peer=server-c28147ed-01a0-34f3-9fb8-b0b56e7fe488
2024-12-04T01:32:46.169Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: pipelining replication: peer="{Voter server-c28147ed-01a0-34f3-9fb8-b0b56e7fe488 c28147ed-01a0-34f3-9fb8-b0b56e7fe488}"
2024-12-04T01:32:46.169Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: pipelining replication: peer="{Voter server-602229ad-3d5a-5f8b-a50a-0166d7fca6b5 602229ad-3d5a-5f8b-a50a-0166d7fca6b5}"
2024-12-04T01:32:47.175Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:46.169979675 +0000 UTC m=+57.001822275 (1 nodes), 585.235808ms from start of poll, 585.711572ms from cluster start. Timeout at 2024-12-04 01:32:47.174988878 +0000 UTC m=+58.006831487, 1.005009212s after stability
2024-12-04T01:32:47.176Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: starting snapshot up to: index=12
2024-12-04T01:32:47.177Z [INFO]  creating new snapshot: path=/tmp/raft1950843005/snapshots/2-12-1733275967177.tmp
2024-12-04T01:32:47.177Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: snapshot complete up to: index=12
2024-12-04T01:32:47.179Z [INFO]  creating new snapshot: path=/tmp/raft1950843005/snapshots/2-113-1733275967179.tmp
2024-12-04T01:32:47.180Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: copied to local snapshot: bytes=71
2024-12-04T01:32:47.180Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: snapshot restore progress: id=2-113-1733275967179 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:47.180Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: restored user snapshot: index=113
2024-12-04T01:32:47.180Z [ERROR] server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: failed to get log: index=23 error="log not found"
2024-12-04T01:32:47.180Z [ERROR] server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: failed to get log: index=23 error="log not found"
2024-12-04T01:32:47.180Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: aborting pipeline replication: peer="{Voter server-602229ad-3d5a-5f8b-a50a-0166d7fca6b5 602229ad-3d5a-5f8b-a50a-0166d7fca6b5}"
2024-12-04T01:32:47.180Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: aborting pipeline replication: peer="{Voter server-c28147ed-01a0-34f3-9fb8-b0b56e7fe488 c28147ed-01a0-34f3-9fb8-b0b56e7fe488}"
2024-12-04T01:32:47.186Z [ERROR] server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: failed to get log: index=23 error="log not found"
2024-12-04T01:32:47.187Z [INFO]  creating new snapshot: path=/tmp/raft1593175449/snapshots/2-113-1733275967187.tmp
2024-12-04T01:32:47.187Z [ERROR] server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: failed to get log: index=23 error="log not found"
2024-12-04T01:32:47.187Z [INFO]  server-602229ad-3d5a-5f8b-a50a-0166d7fca6b5: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:47.188Z [INFO]  creating new snapshot: path=/tmp/raft3968213472/snapshots/2-113-1733275967188.tmp
2024-12-04T01:32:47.188Z [INFO]  server-602229ad-3d5a-5f8b-a50a-0166d7fca6b5: copied to local snapshot: bytes=71
2024-12-04T01:32:47.188Z [INFO]  server-c28147ed-01a0-34f3-9fb8-b0b56e7fe488: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:47.188Z [INFO]  server-c28147ed-01a0-34f3-9fb8-b0b56e7fe488: copied to local snapshot: bytes=71
2024-12-04T01:32:47.188Z [INFO]  server-602229ad-3d5a-5f8b-a50a-0166d7fca6b5: snapshot restore progress: id=2-113-1733275967187 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:47.188Z [INFO]  server-602229ad-3d5a-5f8b-a50a-0166d7fca6b5: Installed remote snapshot
2024-12-04T01:32:47.188Z [WARN]  server-602229ad-3d5a-5f8b-a50a-0166d7fca6b5: skipping application of old log: index=22
2024-12-04T01:32:47.188Z [INFO]  server-c28147ed-01a0-34f3-9fb8-b0b56e7fe488: snapshot restore progress: id=2-113-1733275967188 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:47.188Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: pipelining replication: peer="{Voter server-602229ad-3d5a-5f8b-a50a-0166d7fca6b5 602229ad-3d5a-5f8b-a50a-0166d7fca6b5}"
2024-12-04T01:32:47.189Z [INFO]  server-c28147ed-01a0-34f3-9fb8-b0b56e7fe488: Installed remote snapshot
2024-12-04T01:32:47.189Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:47.194Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: pipelining replication: peer="{Voter server-c28147ed-01a0-34f3-9fb8-b0b56e7fe488 c28147ed-01a0-34f3-9fb8-b0b56e7fe488}"
2024-12-04T01:32:48.195Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:47.189084529 +0000 UTC m=+58.020927101 (2 nodes), 1.91µs from start of poll, 1.604816398s from cluster start. Timeout at 2024-12-04 01:32:48.194133461 +0000 UTC m=+59.025976581, 1.00504948s after stability
2024-12-04T01:32:48.214Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: aborting pipeline replication: peer="{Voter server-c28147ed-01a0-34f3-9fb8-b0b56e7fe488 c28147ed-01a0-34f3-9fb8-b0b56e7fe488}"
2024-12-04T01:32:48.214Z [INFO]  server-6828deef-3ee7-d7ae-e32b-fc41686e51f8: aborting pipeline replication: peer="{Voter server-602229ad-3d5a-5f8b-a50a-0166d7fca6b5 602229ad-3d5a-5f8b-a50a-0166d7fca6b5}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false
2024-12-04T01:32:48.216Z [INFO]  server-989a3f4c-d875-27be-15e4-77879d6ae5c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-989a3f4c-d875-27be-15e4-77879d6ae5c3 Address:989a3f4c-d875-27be-15e4-77879d6ae5c3} {Suffrage:Voter ID:server-78de21a9-8620-9d07-db20-c26e8cbaa9bf Address:78de21a9-8620-9d07-db20-c26e8cbaa9bf} {Suffrage:Voter ID:server-a9ba1ac9-7369-9462-0643-cd01fabbba64 Address:a9ba1ac9-7369-9462-0643-cd01fabbba64}]"
2024-12-04T01:32:48.216Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-989a3f4c-d875-27be-15e4-77879d6ae5c3 Address:989a3f4c-d875-27be-15e4-77879d6ae5c3} {Suffrage:Voter ID:server-78de21a9-8620-9d07-db20-c26e8cbaa9bf Address:78de21a9-8620-9d07-db20-c26e8cbaa9bf} {Suffrage:Voter ID:server-a9ba1ac9-7369-9462-0643-cd01fabbba64 Address:a9ba1ac9-7369-9462-0643-cd01fabbba64}]"
2024-12-04T01:32:48.217Z [INFO]  server-a9ba1ac9-7369-9462-0643-cd01fabbba64: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-989a3f4c-d875-27be-15e4-77879d6ae5c3 Address:989a3f4c-d875-27be-15e4-77879d6ae5c3} {Suffrage:Voter ID:server-78de21a9-8620-9d07-db20-c26e8cbaa9bf Address:78de21a9-8620-9d07-db20-c26e8cbaa9bf} {Suffrage:Voter ID:server-a9ba1ac9-7369-9462-0643-cd01fabbba64 Address:a9ba1ac9-7369-9462-0643-cd01fabbba64}]"
2024-12-04T01:32:48.217Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:48.217Z [INFO]  server-989a3f4c-d875-27be-15e4-77879d6ae5c3: entering follower state: follower="Node at 989a3f4c-d875-27be-15e4-77879d6ae5c3 [Follower]" leader-address= leader-id=
2024-12-04T01:32:48.217Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: entering follower state: follower="Node at 78de21a9-8620-9d07-db20-c26e8cbaa9bf [Follower]" leader-address= leader-id=
2024-12-04T01:32:48.217Z [INFO]  server-a9ba1ac9-7369-9462-0643-cd01fabbba64: entering follower state: follower="Node at a9ba1ac9-7369-9462-0643-cd01fabbba64 [Follower]" leader-address= leader-id=
2024-12-04T01:32:48.880Z [WARN]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:48.880Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: entering candidate state: node="Node at 78de21a9-8620-9d07-db20-c26e8cbaa9bf [Candidate]" term=2
2024-12-04T01:32:48.880Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: election won: term=2 tally=2
2024-12-04T01:32:48.880Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: entering leader state: leader="Node at 78de21a9-8620-9d07-db20-c26e8cbaa9bf [Leader]"
2024-12-04T01:32:48.880Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: added peer, starting replication: peer=server-989a3f4c-d875-27be-15e4-77879d6ae5c3
2024-12-04T01:32:48.880Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: added peer, starting replication: peer=server-a9ba1ac9-7369-9462-0643-cd01fabbba64
2024-12-04T01:32:48.881Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: pipelining replication: peer="{Voter server-a9ba1ac9-7369-9462-0643-cd01fabbba64 a9ba1ac9-7369-9462-0643-cd01fabbba64}"
2024-12-04T01:32:48.881Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: pipelining replication: peer="{Voter server-989a3f4c-d875-27be-15e4-77879d6ae5c3 989a3f4c-d875-27be-15e4-77879d6ae5c3}"
2024-12-04T01:32:49.887Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:48.881053342 +0000 UTC m=+59.712895995 (1 nodes), 663.913515ms from start of poll, 664.333286ms from cluster start. Timeout at 2024-12-04 01:32:49.886058372 +0000 UTC m=+60.717901029, 1.005005034s after stability
2024-12-04T01:32:49.889Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: starting snapshot up to: index=12
2024-12-04T01:32:49.889Z [INFO]  creating new snapshot: path=/tmp/raft1663158233/snapshots/2-12-1733275969889.tmp
2024-12-04T01:32:49.890Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: snapshot complete up to: index=12
2024-12-04T01:32:49.893Z [INFO]  creating new snapshot: path=/tmp/raft1663158233/snapshots/2-113-1733275969893.tmp
2024-12-04T01:32:49.894Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: copied to local snapshot: bytes=71
2024-12-04T01:32:49.894Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: snapshot restore progress: id=2-113-1733275969893 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:49.894Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: removing all old logs from log store
2024-12-04T01:32:49.894Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: compacting logs: from=1 to=22
2024-12-04T01:32:49.894Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: restored user snapshot: index=113
2024-12-04T01:32:49.894Z [ERROR] server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: failed to get log: index=22 error="log not found"
2024-12-04T01:32:49.894Z [ERROR] server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: failed to get log: index=22 error="log not found"
2024-12-04T01:32:49.894Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: aborting pipeline replication: peer="{Voter server-989a3f4c-d875-27be-15e4-77879d6ae5c3 989a3f4c-d875-27be-15e4-77879d6ae5c3}"
2024-12-04T01:32:49.894Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: aborting pipeline replication: peer="{Voter server-a9ba1ac9-7369-9462-0643-cd01fabbba64 a9ba1ac9-7369-9462-0643-cd01fabbba64}"
2024-12-04T01:32:49.905Z [ERROR] server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: failed to get log: index=22 error="log not found"
2024-12-04T01:32:49.905Z [ERROR] server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: failed to get log: index=22 error="log not found"
2024-12-04T01:32:49.906Z [INFO]  creating new snapshot: path=/tmp/raft1333913885/snapshots/2-113-1733275969906.tmp
2024-12-04T01:32:49.907Z [INFO]  server-989a3f4c-d875-27be-15e4-77879d6ae5c3: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:49.907Z [INFO]  server-989a3f4c-d875-27be-15e4-77879d6ae5c3: copied to local snapshot: bytes=71
2024-12-04T01:32:49.908Z [INFO]  server-989a3f4c-d875-27be-15e4-77879d6ae5c3: snapshot restore progress: id=2-113-1733275969906 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:49.908Z [INFO]  server-989a3f4c-d875-27be-15e4-77879d6ae5c3: removing all old logs from log store
2024-12-04T01:32:49.908Z [INFO]  server-989a3f4c-d875-27be-15e4-77879d6ae5c3: compacting logs: from=1 to=22
2024-12-04T01:32:49.908Z [INFO]  server-989a3f4c-d875-27be-15e4-77879d6ae5c3: Installed remote snapshot
2024-12-04T01:32:49.908Z [WARN]  server-989a3f4c-d875-27be-15e4-77879d6ae5c3: skipping application of old log: index=22
2024-12-04T01:32:49.908Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: pipelining replication: peer="{Voter server-989a3f4c-d875-27be-15e4-77879d6ae5c3 989a3f4c-d875-27be-15e4-77879d6ae5c3}"
2024-12-04T01:32:49.908Z [INFO]  creating new snapshot: path=/tmp/raft775607423/snapshots/2-113-1733275969908.tmp
2024-12-04T01:32:49.908Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:49.908Z [INFO]  server-a9ba1ac9-7369-9462-0643-cd01fabbba64: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:49.909Z [INFO]  server-a9ba1ac9-7369-9462-0643-cd01fabbba64: copied to local snapshot: bytes=71
2024-12-04T01:32:49.909Z [INFO]  server-a9ba1ac9-7369-9462-0643-cd01fabbba64: snapshot restore progress: id=2-113-1733275969908 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:49.909Z [INFO]  server-a9ba1ac9-7369-9462-0643-cd01fabbba64: removing all old logs from log store
2024-12-04T01:32:49.909Z [INFO]  server-a9ba1ac9-7369-9462-0643-cd01fabbba64: compacting logs: from=1 to=22
2024-12-04T01:32:49.909Z [INFO]  server-a9ba1ac9-7369-9462-0643-cd01fabbba64: Installed remote snapshot
2024-12-04T01:32:49.909Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: pipelining replication: peer="{Voter server-a9ba1ac9-7369-9462-0643-cd01fabbba64 a9ba1ac9-7369-9462-0643-cd01fabbba64}"
2024-12-04T01:32:50.914Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:49.908609031 +0000 UTC m=+60.740451758 (2 nodes), 2.325µs from start of poll, 1.691889049s from cluster start. Timeout at 2024-12-04 01:32:50.913678788 +0000 UTC m=+61.745521380, 1.005069622s after stability
2024-12-04T01:32:50.926Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: aborting pipeline replication: peer="{Voter server-a9ba1ac9-7369-9462-0643-cd01fabbba64 a9ba1ac9-7369-9462-0643-cd01fabbba64}"
2024-12-04T01:32:50.926Z [INFO]  server-78de21a9-8620-9d07-db20-c26e8cbaa9bf: aborting pipeline replication: peer="{Voter server-989a3f4c-d875-27be-15e4-77879d6ae5c3 989a3f4c-d875-27be-15e4-77879d6ae5c3}"
=== RUN   TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true
2024-12-04T01:32:50.928Z [INFO]  server-0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc Address:0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc} {Suffrage:Voter ID:server-a2918b7a-b745-02cd-0907-5a7f291c66b3 Address:a2918b7a-b745-02cd-0907-5a7f291c66b3} {Suffrage:Voter ID:server-d7dec38e-4ebf-f18d-806b-cd8472ba02fe Address:d7dec38e-4ebf-f18d-806b-cd8472ba02fe}]"
2024-12-04T01:32:50.928Z [INFO]  server-a2918b7a-b745-02cd-0907-5a7f291c66b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc Address:0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc} {Suffrage:Voter ID:server-a2918b7a-b745-02cd-0907-5a7f291c66b3 Address:a2918b7a-b745-02cd-0907-5a7f291c66b3} {Suffrage:Voter ID:server-d7dec38e-4ebf-f18d-806b-cd8472ba02fe Address:d7dec38e-4ebf-f18d-806b-cd8472ba02fe}]"
2024-12-04T01:32:50.928Z [INFO]  server-d7dec38e-4ebf-f18d-806b-cd8472ba02fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc Address:0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc} {Suffrage:Voter ID:server-a2918b7a-b745-02cd-0907-5a7f291c66b3 Address:a2918b7a-b745-02cd-0907-5a7f291c66b3} {Suffrage:Voter ID:server-d7dec38e-4ebf-f18d-806b-cd8472ba02fe Address:d7dec38e-4ebf-f18d-806b-cd8472ba02fe}]"
2024-12-04T01:32:50.928Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:50.928Z [INFO]  server-0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc: entering follower state: follower="Node at 0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc [Follower]" leader-address= leader-id=
2024-12-04T01:32:50.928Z [INFO]  server-a2918b7a-b745-02cd-0907-5a7f291c66b3: entering follower state: follower="Node at a2918b7a-b745-02cd-0907-5a7f291c66b3 [Follower]" leader-address= leader-id=
2024-12-04T01:32:50.928Z [INFO]  server-d7dec38e-4ebf-f18d-806b-cd8472ba02fe: entering follower state: follower="Node at d7dec38e-4ebf-f18d-806b-cd8472ba02fe [Follower]" leader-address= leader-id=
2024-12-04T01:32:51.543Z [WARN]  server-0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:51.543Z [INFO]  server-0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc: entering candidate state: node="Node at 0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc [Candidate]" term=2
2024-12-04T01:32:51.543Z [INFO]  server-0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc: election won: term=2 tally=2
2024-12-04T01:32:51.543Z [INFO]  server-0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc: entering leader state: leader="Node at 0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc [Leader]"
2024-12-04T01:32:51.543Z [INFO]  server-0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc: added peer, starting replication: peer=server-a2918b7a-b745-02cd-0907-5a7f291c66b3
2024-12-04T01:32:51.543Z [INFO]  server-0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc: added peer, starting replication: peer=server-d7dec38e-4ebf-f18d-806b-cd8472ba02fe
2024-12-04T01:32:51.544Z [INFO]  server-0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc: pipelining replication: peer="{Voter server-d7dec38e-4ebf-f18d-806b-cd8472ba02fe d7dec38e-4ebf-f18d-806b-cd8472ba02fe}"
2024-12-04T01:32:51.544Z [INFO]  server-0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc: pipelining replication: peer="{Voter server-a2918b7a-b745-02cd-0907-5a7f291c66b3 a2918b7a-b745-02cd-0907-5a7f291c66b3}"
2024-12-04T01:32:52.549Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:51.544448088 +0000 UTC m=+62.376290766 (1 nodes), 615.992198ms from start of poll, 616.343134ms from cluster start. Timeout at 2024-12-04 01:32:52.54945451 +0000 UTC m=+63.381297071, 1.005006305s after stability
2024-12-04T01:32:52.551Z [INFO]  server-0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc: starting snapshot up to: index=12
2024-12-04T01:32:52.551Z [INFO]  creating new snapshot: path=/tmp/raft1671586395/snapshots/2-12-1733275972551.tmp
2024-12-04T01:32:52.552Z [INFO]  server-0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc: snapshot complete up to: index=12
2024-12-04T01:32:52.555Z [INFO]  server-d7c045d8-81aa-028a-0a60-868a6401cc10: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d7c045d8-81aa-028a-0a60-868a6401cc10 Address:d7c045d8-81aa-028a-0a60-868a6401cc10} {Suffrage:Voter ID:server-02b04359-aa63-c059-358e-7e713b1e2428 Address:02b04359-aa63-c059-358e-7e713b1e2428} {Suffrage:Voter ID:server-23b992f4-136f-d5f5-dc2b-e85a297491dd Address:23b992f4-136f-d5f5-dc2b-e85a297491dd}]"
2024-12-04T01:32:52.555Z [INFO]  server-02b04359-aa63-c059-358e-7e713b1e2428: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d7c045d8-81aa-028a-0a60-868a6401cc10 Address:d7c045d8-81aa-028a-0a60-868a6401cc10} {Suffrage:Voter ID:server-02b04359-aa63-c059-358e-7e713b1e2428 Address:02b04359-aa63-c059-358e-7e713b1e2428} {Suffrage:Voter ID:server-23b992f4-136f-d5f5-dc2b-e85a297491dd Address:23b992f4-136f-d5f5-dc2b-e85a297491dd}]"
2024-12-04T01:32:52.556Z [INFO]  server-23b992f4-136f-d5f5-dc2b-e85a297491dd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d7c045d8-81aa-028a-0a60-868a6401cc10 Address:d7c045d8-81aa-028a-0a60-868a6401cc10} {Suffrage:Voter ID:server-02b04359-aa63-c059-358e-7e713b1e2428 Address:02b04359-aa63-c059-358e-7e713b1e2428} {Suffrage:Voter ID:server-23b992f4-136f-d5f5-dc2b-e85a297491dd Address:23b992f4-136f-d5f5-dc2b-e85a297491dd}]"
2024-12-04T01:32:52.556Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:52.556Z [INFO]  server-d7c045d8-81aa-028a-0a60-868a6401cc10: entering follower state: follower="Node at d7c045d8-81aa-028a-0a60-868a6401cc10 [Follower]" leader-address= leader-id=
2024-12-04T01:32:52.556Z [INFO]  server-02b04359-aa63-c059-358e-7e713b1e2428: entering follower state: follower="Node at 02b04359-aa63-c059-358e-7e713b1e2428 [Follower]" leader-address= leader-id=
2024-12-04T01:32:52.556Z [INFO]  server-23b992f4-136f-d5f5-dc2b-e85a297491dd: entering follower state: follower="Node at 23b992f4-136f-d5f5-dc2b-e85a297491dd [Follower]" leader-address= leader-id=
2024-12-04T01:32:53.086Z [WARN]  server-02b04359-aa63-c059-358e-7e713b1e2428: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:53.086Z [INFO]  server-02b04359-aa63-c059-358e-7e713b1e2428: entering candidate state: node="Node at 02b04359-aa63-c059-358e-7e713b1e2428 [Candidate]" term=2
2024-12-04T01:32:53.086Z [INFO]  server-02b04359-aa63-c059-358e-7e713b1e2428: election won: term=2 tally=2
2024-12-04T01:32:53.086Z [INFO]  server-02b04359-aa63-c059-358e-7e713b1e2428: entering leader state: leader="Node at 02b04359-aa63-c059-358e-7e713b1e2428 [Leader]"
2024-12-04T01:32:53.086Z [INFO]  server-02b04359-aa63-c059-358e-7e713b1e2428: added peer, starting replication: peer=server-d7c045d8-81aa-028a-0a60-868a6401cc10
2024-12-04T01:32:53.086Z [INFO]  server-02b04359-aa63-c059-358e-7e713b1e2428: added peer, starting replication: peer=server-23b992f4-136f-d5f5-dc2b-e85a297491dd
2024-12-04T01:32:53.086Z [INFO]  server-02b04359-aa63-c059-358e-7e713b1e2428: pipelining replication: peer="{Voter server-23b992f4-136f-d5f5-dc2b-e85a297491dd 23b992f4-136f-d5f5-dc2b-e85a297491dd}"
2024-12-04T01:32:53.087Z [WARN]  server-23b992f4-136f-d5f5-dc2b-e85a297491dd: rejecting vote request since our last term is greater: candidate=02b04359-aa63-c059-358e-7e713b1e2428 last-term=2 last-candidate-term=1
2024-12-04T01:32:53.087Z [INFO]  server-02b04359-aa63-c059-358e-7e713b1e2428: pipelining replication: peer="{Voter server-d7c045d8-81aa-028a-0a60-868a6401cc10 d7c045d8-81aa-028a-0a60-868a6401cc10}"
2024-12-04T01:32:54.093Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:53.087145576 +0000 UTC m=+63.918988147 (1 nodes), 531.057005ms from start of poll, 531.439231ms from cluster start. Timeout at 2024-12-04 01:32:54.092163208 +0000 UTC m=+64.924005776, 1.005017629s after stability
2024-12-04T01:32:54.093Z [INFO]  creating new snapshot: path=/tmp/raft2822592699/snapshots/2-113-1733275974093.tmp
2024-12-04T01:32:54.094Z [INFO]  server-02b04359-aa63-c059-358e-7e713b1e2428: copied to local snapshot: bytes=71
2024-12-04T01:32:54.094Z [INFO]  server-02b04359-aa63-c059-358e-7e713b1e2428: snapshot restore progress: id=2-113-1733275974093 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:54.094Z [INFO]  server-02b04359-aa63-c059-358e-7e713b1e2428: restored user snapshot: index=113
2024-12-04T01:32:54.094Z [ERROR] server-02b04359-aa63-c059-358e-7e713b1e2428: failed to get log: index=3 error="log not found"
2024-12-04T01:32:54.094Z [INFO]  server-02b04359-aa63-c059-358e-7e713b1e2428: aborting pipeline replication: peer="{Voter server-23b992f4-136f-d5f5-dc2b-e85a297491dd 23b992f4-136f-d5f5-dc2b-e85a297491dd}"
2024-12-04T01:32:54.095Z [ERROR] server-02b04359-aa63-c059-358e-7e713b1e2428: failed to get log: index=3 error="log not found"
2024-12-04T01:32:54.095Z [INFO]  server-02b04359-aa63-c059-358e-7e713b1e2428: aborting pipeline replication: peer="{Voter server-d7c045d8-81aa-028a-0a60-868a6401cc10 d7c045d8-81aa-028a-0a60-868a6401cc10}"
2024-12-04T01:32:54.100Z [ERROR] server-02b04359-aa63-c059-358e-7e713b1e2428: failed to get log: index=3 error="log not found"
2024-12-04T01:32:54.100Z [INFO]  creating new snapshot: path=/tmp/raft3142193666/snapshots/2-113-1733275974100.tmp
2024-12-04T01:32:54.100Z [INFO]  server-23b992f4-136f-d5f5-dc2b-e85a297491dd: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:54.101Z [INFO]  server-23b992f4-136f-d5f5-dc2b-e85a297491dd: copied to local snapshot: bytes=71
2024-12-04T01:32:54.101Z [INFO]  server-23b992f4-136f-d5f5-dc2b-e85a297491dd: snapshot restore progress: id=2-113-1733275974100 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:54.101Z [INFO]  server-23b992f4-136f-d5f5-dc2b-e85a297491dd: Installed remote snapshot
2024-12-04T01:32:54.101Z [INFO]  server-02b04359-aa63-c059-358e-7e713b1e2428: pipelining replication: peer="{Voter server-23b992f4-136f-d5f5-dc2b-e85a297491dd 23b992f4-136f-d5f5-dc2b-e85a297491dd}"
2024-12-04T01:32:54.101Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:54.102Z [ERROR] server-02b04359-aa63-c059-358e-7e713b1e2428: failed to get log: index=3 error="log not found"
2024-12-04T01:32:54.102Z [INFO]  creating new snapshot: path=/tmp/raft205660685/snapshots/2-113-1733275974102.tmp
2024-12-04T01:32:54.103Z [INFO]  server-d7c045d8-81aa-028a-0a60-868a6401cc10: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:54.103Z [INFO]  server-d7c045d8-81aa-028a-0a60-868a6401cc10: copied to local snapshot: bytes=71
2024-12-04T01:32:54.110Z [INFO]  server-d7c045d8-81aa-028a-0a60-868a6401cc10: snapshot restore progress: id=2-113-1733275974102 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:54.111Z [INFO]  server-d7c045d8-81aa-028a-0a60-868a6401cc10: Installed remote snapshot
2024-12-04T01:32:54.111Z [INFO]  server-02b04359-aa63-c059-358e-7e713b1e2428: pipelining replication: peer="{Voter server-d7c045d8-81aa-028a-0a60-868a6401cc10 d7c045d8-81aa-028a-0a60-868a6401cc10}"
2024-12-04T01:32:55.107Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:54.101964782 +0000 UTC m=+64.933807344 (2 nodes), 2.058µs from start of poll, 1.546258428s from cluster start. Timeout at 2024-12-04 01:32:55.106973168 +0000 UTC m=+65.938815787, 1.005008443s after stability
2024-12-04T01:32:55.119Z [INFO]  server-0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc: aborting pipeline replication: peer="{Voter server-a2918b7a-b745-02cd-0907-5a7f291c66b3 a2918b7a-b745-02cd-0907-5a7f291c66b3}"
2024-12-04T01:32:55.119Z [INFO]  server-0cc2c0c9-ccdb-7d40-479e-6cdaac60e0dc: aborting pipeline replication: peer="{Voter server-d7dec38e-4ebf-f18d-806b-cd8472ba02fe d7dec38e-4ebf-f18d-806b-cd8472ba02fe}"
=== RUN   TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true
2024-12-04T01:32:55.121Z [INFO]  server-449ccf34-e9f1-40f8-343f-1acbae14385d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-449ccf34-e9f1-40f8-343f-1acbae14385d Address:449ccf34-e9f1-40f8-343f-1acbae14385d} {Suffrage:Voter ID:server-3cb4f35b-fdc5-1172-9b68-796a7e0cb1a3 Address:3cb4f35b-fdc5-1172-9b68-796a7e0cb1a3} {Suffrage:Voter ID:server-11db41d1-eaf6-bb7e-4bb0-79f0887dba17 Address:11db41d1-eaf6-bb7e-4bb0-79f0887dba17}]"
2024-12-04T01:32:55.121Z [INFO]  server-3cb4f35b-fdc5-1172-9b68-796a7e0cb1a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-449ccf34-e9f1-40f8-343f-1acbae14385d Address:449ccf34-e9f1-40f8-343f-1acbae14385d} {Suffrage:Voter ID:server-3cb4f35b-fdc5-1172-9b68-796a7e0cb1a3 Address:3cb4f35b-fdc5-1172-9b68-796a7e0cb1a3} {Suffrage:Voter ID:server-11db41d1-eaf6-bb7e-4bb0-79f0887dba17 Address:11db41d1-eaf6-bb7e-4bb0-79f0887dba17}]"
2024-12-04T01:32:55.121Z [INFO]  server-11db41d1-eaf6-bb7e-4bb0-79f0887dba17: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-449ccf34-e9f1-40f8-343f-1acbae14385d Address:449ccf34-e9f1-40f8-343f-1acbae14385d} {Suffrage:Voter ID:server-3cb4f35b-fdc5-1172-9b68-796a7e0cb1a3 Address:3cb4f35b-fdc5-1172-9b68-796a7e0cb1a3} {Suffrage:Voter ID:server-11db41d1-eaf6-bb7e-4bb0-79f0887dba17 Address:11db41d1-eaf6-bb7e-4bb0-79f0887dba17}]"
2024-12-04T01:32:55.121Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:55.121Z [INFO]  server-449ccf34-e9f1-40f8-343f-1acbae14385d: entering follower state: follower="Node at 449ccf34-e9f1-40f8-343f-1acbae14385d [Follower]" leader-address= leader-id=
2024-12-04T01:32:55.121Z [INFO]  server-3cb4f35b-fdc5-1172-9b68-796a7e0cb1a3: entering follower state: follower="Node at 3cb4f35b-fdc5-1172-9b68-796a7e0cb1a3 [Follower]" leader-address= leader-id=
2024-12-04T01:32:55.121Z [INFO]  server-11db41d1-eaf6-bb7e-4bb0-79f0887dba17: entering follower state: follower="Node at 11db41d1-eaf6-bb7e-4bb0-79f0887dba17 [Follower]" leader-address= leader-id=
2024-12-04T01:32:55.836Z [WARN]  server-449ccf34-e9f1-40f8-343f-1acbae14385d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:55.836Z [INFO]  server-449ccf34-e9f1-40f8-343f-1acbae14385d: entering candidate state: node="Node at 449ccf34-e9f1-40f8-343f-1acbae14385d [Candidate]" term=2
2024-12-04T01:32:55.836Z [INFO]  server-449ccf34-e9f1-40f8-343f-1acbae14385d: election won: term=2 tally=2
2024-12-04T01:32:55.836Z [INFO]  server-449ccf34-e9f1-40f8-343f-1acbae14385d: entering leader state: leader="Node at 449ccf34-e9f1-40f8-343f-1acbae14385d [Leader]"
2024-12-04T01:32:55.836Z [INFO]  server-449ccf34-e9f1-40f8-343f-1acbae14385d: added peer, starting replication: peer=server-3cb4f35b-fdc5-1172-9b68-796a7e0cb1a3
2024-12-04T01:32:55.836Z [INFO]  server-449ccf34-e9f1-40f8-343f-1acbae14385d: added peer, starting replication: peer=server-11db41d1-eaf6-bb7e-4bb0-79f0887dba17
2024-12-04T01:32:55.836Z [INFO]  server-449ccf34-e9f1-40f8-343f-1acbae14385d: pipelining replication: peer="{Voter server-11db41d1-eaf6-bb7e-4bb0-79f0887dba17 11db41d1-eaf6-bb7e-4bb0-79f0887dba17}"
2024-12-04T01:32:55.836Z [INFO]  server-449ccf34-e9f1-40f8-343f-1acbae14385d: pipelining replication: peer="{Voter server-3cb4f35b-fdc5-1172-9b68-796a7e0cb1a3 3cb4f35b-fdc5-1172-9b68-796a7e0cb1a3}"
2024-12-04T01:32:56.841Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:55.836322064 +0000 UTC m=+66.668164655 (1 nodes), 714.915702ms from start of poll, 715.386787ms from cluster start. Timeout at 2024-12-04 01:32:56.84133763 +0000 UTC m=+67.673180244, 1.005015589s after stability
2024-12-04T01:32:56.843Z [INFO]  server-449ccf34-e9f1-40f8-343f-1acbae14385d: starting snapshot up to: index=12
2024-12-04T01:32:56.843Z [INFO]  creating new snapshot: path=/tmp/raft89969975/snapshots/2-12-1733275976843.tmp
2024-12-04T01:32:56.844Z [INFO]  server-449ccf34-e9f1-40f8-343f-1acbae14385d: snapshot complete up to: index=12
2024-12-04T01:32:56.846Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a Address:ec19a2c0-1df9-c146-0d93-2b38cc77d69a} {Suffrage:Voter ID:server-e8722844-fcf1-940d-a0c8-64b206efb425 Address:e8722844-fcf1-940d-a0c8-64b206efb425} {Suffrage:Voter ID:server-70890530-7d0a-838b-bdb5-c5e4b42754b4 Address:70890530-7d0a-838b-bdb5-c5e4b42754b4}]"
2024-12-04T01:32:56.846Z [INFO]  server-e8722844-fcf1-940d-a0c8-64b206efb425: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a Address:ec19a2c0-1df9-c146-0d93-2b38cc77d69a} {Suffrage:Voter ID:server-e8722844-fcf1-940d-a0c8-64b206efb425 Address:e8722844-fcf1-940d-a0c8-64b206efb425} {Suffrage:Voter ID:server-70890530-7d0a-838b-bdb5-c5e4b42754b4 Address:70890530-7d0a-838b-bdb5-c5e4b42754b4}]"
2024-12-04T01:32:56.846Z [INFO]  server-70890530-7d0a-838b-bdb5-c5e4b42754b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a Address:ec19a2c0-1df9-c146-0d93-2b38cc77d69a} {Suffrage:Voter ID:server-e8722844-fcf1-940d-a0c8-64b206efb425 Address:e8722844-fcf1-940d-a0c8-64b206efb425} {Suffrage:Voter ID:server-70890530-7d0a-838b-bdb5-c5e4b42754b4 Address:70890530-7d0a-838b-bdb5-c5e4b42754b4}]"
2024-12-04T01:32:56.846Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:56.847Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: entering follower state: follower="Node at ec19a2c0-1df9-c146-0d93-2b38cc77d69a [Follower]" leader-address= leader-id=
2024-12-04T01:32:56.847Z [INFO]  server-e8722844-fcf1-940d-a0c8-64b206efb425: entering follower state: follower="Node at e8722844-fcf1-940d-a0c8-64b206efb425 [Follower]" leader-address= leader-id=
2024-12-04T01:32:56.847Z [INFO]  server-70890530-7d0a-838b-bdb5-c5e4b42754b4: entering follower state: follower="Node at 70890530-7d0a-838b-bdb5-c5e4b42754b4 [Follower]" leader-address= leader-id=
2024-12-04T01:32:57.391Z [WARN]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:32:57.391Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: entering candidate state: node="Node at ec19a2c0-1df9-c146-0d93-2b38cc77d69a [Candidate]" term=2
2024-12-04T01:32:57.391Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: election won: term=2 tally=2
2024-12-04T01:32:57.392Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: entering leader state: leader="Node at ec19a2c0-1df9-c146-0d93-2b38cc77d69a [Leader]"
2024-12-04T01:32:57.392Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: added peer, starting replication: peer=server-e8722844-fcf1-940d-a0c8-64b206efb425
2024-12-04T01:32:57.392Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: added peer, starting replication: peer=server-70890530-7d0a-838b-bdb5-c5e4b42754b4
2024-12-04T01:32:57.392Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: pipelining replication: peer="{Voter server-70890530-7d0a-838b-bdb5-c5e4b42754b4 70890530-7d0a-838b-bdb5-c5e4b42754b4}"
2024-12-04T01:32:57.392Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: pipelining replication: peer="{Voter server-e8722844-fcf1-940d-a0c8-64b206efb425 e8722844-fcf1-940d-a0c8-64b206efb425}"
2024-12-04T01:32:58.397Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:32:57.392134291 +0000 UTC m=+68.223977109 (1 nodes), 545.139931ms from start of poll, 545.598486ms from cluster start. Timeout at 2024-12-04 01:32:58.39713745 +0000 UTC m=+69.228980221, 1.005003112s after stability
2024-12-04T01:32:58.397Z [INFO]  creating new snapshot: path=/tmp/raft4282060103/snapshots/2-113-1733275978397.tmp
2024-12-04T01:32:58.398Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: copied to local snapshot: bytes=71
2024-12-04T01:32:58.399Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: snapshot restore progress: id=2-113-1733275978397 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:58.399Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: removing all old logs from log store
2024-12-04T01:32:58.399Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: compacting logs: from=1 to=2
2024-12-04T01:32:58.399Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: restored user snapshot: index=113
2024-12-04T01:32:58.399Z [ERROR] server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: failed to get log: index=2 error="log not found"
2024-12-04T01:32:58.399Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: aborting pipeline replication: peer="{Voter server-70890530-7d0a-838b-bdb5-c5e4b42754b4 70890530-7d0a-838b-bdb5-c5e4b42754b4}"
2024-12-04T01:32:58.399Z [ERROR] server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: failed to get log: index=2 error="log not found"
2024-12-04T01:32:58.399Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: aborting pipeline replication: peer="{Voter server-e8722844-fcf1-940d-a0c8-64b206efb425 e8722844-fcf1-940d-a0c8-64b206efb425}"
2024-12-04T01:32:58.405Z [ERROR] server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: failed to get log: index=2 error="log not found"
2024-12-04T01:32:58.410Z [INFO]  creating new snapshot: path=/tmp/raft60855430/snapshots/2-113-1733275978410.tmp
2024-12-04T01:32:58.410Z [ERROR] server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: failed to get log: index=2 error="log not found"
2024-12-04T01:32:58.410Z [INFO]  server-e8722844-fcf1-940d-a0c8-64b206efb425: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:58.411Z [INFO]  creating new snapshot: path=/tmp/raft1454593316/snapshots/2-113-1733275978411.tmp
2024-12-04T01:32:58.411Z [INFO]  server-e8722844-fcf1-940d-a0c8-64b206efb425: copied to local snapshot: bytes=71
2024-12-04T01:32:58.411Z [INFO]  server-70890530-7d0a-838b-bdb5-c5e4b42754b4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:58.411Z [INFO]  server-e8722844-fcf1-940d-a0c8-64b206efb425: snapshot restore progress: id=2-113-1733275978410 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:58.411Z [INFO]  server-e8722844-fcf1-940d-a0c8-64b206efb425: removing all old logs from log store
2024-12-04T01:32:58.411Z [INFO]  server-e8722844-fcf1-940d-a0c8-64b206efb425: compacting logs: from=1 to=2
2024-12-04T01:32:58.411Z [INFO]  server-e8722844-fcf1-940d-a0c8-64b206efb425: Installed remote snapshot
2024-12-04T01:32:58.411Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: pipelining replication: peer="{Voter server-e8722844-fcf1-940d-a0c8-64b206efb425 e8722844-fcf1-940d-a0c8-64b206efb425}"
2024-12-04T01:32:58.411Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:32:58.411Z [INFO]  server-70890530-7d0a-838b-bdb5-c5e4b42754b4: copied to local snapshot: bytes=71
2024-12-04T01:32:58.413Z [INFO]  server-70890530-7d0a-838b-bdb5-c5e4b42754b4: snapshot restore progress: id=2-113-1733275978411 last-index=113 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:32:58.413Z [INFO]  server-70890530-7d0a-838b-bdb5-c5e4b42754b4: removing all old logs from log store
2024-12-04T01:32:58.413Z [INFO]  server-70890530-7d0a-838b-bdb5-c5e4b42754b4: compacting logs: from=1 to=2
2024-12-04T01:32:58.413Z [INFO]  server-70890530-7d0a-838b-bdb5-c5e4b42754b4: Installed remote snapshot
2024-12-04T01:32:58.413Z [INFO]  server-ec19a2c0-1df9-c146-0d93-2b38cc77d69a: pipelining replication: peer="{Voter server-70890530-7d0a-838b-bdb5-c5e4b42754b4 70890530-7d0a-838b-bdb5-c5e4b42754b4}"
2024-12-04T01:32:59.418Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:32:58.411923053 +0000 UTC m=+69.243765635 (2 nodes), 1.797µs from start of poll, 1.565387012s from cluster start. Timeout at 2024-12-04 01:32:59.416944325 +0000 UTC m=+70.248786990, 1.005021355s after stability
2024-12-04T01:32:59.429Z [INFO]  server-449ccf34-e9f1-40f8-343f-1acbae14385d: aborting pipeline replication: peer="{Voter server-11db41d1-eaf6-bb7e-4bb0-79f0887dba17 11db41d1-eaf6-bb7e-4bb0-79f0887dba17}"
2024-12-04T01:32:59.429Z [INFO]  server-449ccf34-e9f1-40f8-343f-1acbae14385d: aborting pipeline replication: peer="{Voter server-3cb4f35b-fdc5-1172-9b68-796a7e0cb1a3 3cb4f35b-fdc5-1172-9b68-796a7e0cb1a3}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false
2024-12-04T01:32:59.431Z [INFO]  server-0086809b-0686-318b-2162-e6ff815cd2de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0086809b-0686-318b-2162-e6ff815cd2de Address:0086809b-0686-318b-2162-e6ff815cd2de} {Suffrage:Voter ID:server-b51398d9-ef2e-796b-ec92-f4a653d029a4 Address:b51398d9-ef2e-796b-ec92-f4a653d029a4} {Suffrage:Voter ID:server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad Address:f567ffbf-02d4-a36f-dd4b-3aeda9a62fad}]"
2024-12-04T01:32:59.431Z [INFO]  server-b51398d9-ef2e-796b-ec92-f4a653d029a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0086809b-0686-318b-2162-e6ff815cd2de Address:0086809b-0686-318b-2162-e6ff815cd2de} {Suffrage:Voter ID:server-b51398d9-ef2e-796b-ec92-f4a653d029a4 Address:b51398d9-ef2e-796b-ec92-f4a653d029a4} {Suffrage:Voter ID:server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad Address:f567ffbf-02d4-a36f-dd4b-3aeda9a62fad}]"
2024-12-04T01:32:59.431Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0086809b-0686-318b-2162-e6ff815cd2de Address:0086809b-0686-318b-2162-e6ff815cd2de} {Suffrage:Voter ID:server-b51398d9-ef2e-796b-ec92-f4a653d029a4 Address:b51398d9-ef2e-796b-ec92-f4a653d029a4} {Suffrage:Voter ID:server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad Address:f567ffbf-02d4-a36f-dd4b-3aeda9a62fad}]"
2024-12-04T01:32:59.431Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:32:59.431Z [INFO]  server-0086809b-0686-318b-2162-e6ff815cd2de: entering follower state: follower="Node at 0086809b-0686-318b-2162-e6ff815cd2de [Follower]" leader-address= leader-id=
2024-12-04T01:32:59.431Z [INFO]  server-b51398d9-ef2e-796b-ec92-f4a653d029a4: entering follower state: follower="Node at b51398d9-ef2e-796b-ec92-f4a653d029a4 [Follower]" leader-address= leader-id=
2024-12-04T01:32:59.432Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: entering follower state: follower="Node at f567ffbf-02d4-a36f-dd4b-3aeda9a62fad [Follower]" leader-address= leader-id=
2024-12-04T01:33:00.001Z [WARN]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:00.001Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: entering candidate state: node="Node at f567ffbf-02d4-a36f-dd4b-3aeda9a62fad [Candidate]" term=2
2024-12-04T01:33:00.001Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: election won: term=2 tally=2
2024-12-04T01:33:00.001Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: entering leader state: leader="Node at f567ffbf-02d4-a36f-dd4b-3aeda9a62fad [Leader]"
2024-12-04T01:33:00.001Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: added peer, starting replication: peer=server-0086809b-0686-318b-2162-e6ff815cd2de
2024-12-04T01:33:00.001Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: added peer, starting replication: peer=server-b51398d9-ef2e-796b-ec92-f4a653d029a4
2024-12-04T01:33:00.001Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: pipelining replication: peer="{Voter server-b51398d9-ef2e-796b-ec92-f4a653d029a4 b51398d9-ef2e-796b-ec92-f4a653d029a4}"
2024-12-04T01:33:00.002Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: pipelining replication: peer="{Voter server-0086809b-0686-318b-2162-e6ff815cd2de 0086809b-0686-318b-2162-e6ff815cd2de}"
2024-12-04T01:33:01.007Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:00.001936979 +0000 UTC m=+70.833779737 (1 nodes), 570.107599ms from start of poll, 570.539278ms from cluster start. Timeout at 2024-12-04 01:33:01.006941113 +0000 UTC m=+71.838783757, 1.00500402s after stability
2024-12-04T01:33:01.008Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: starting snapshot up to: index=12
2024-12-04T01:33:01.009Z [INFO]  creating new snapshot: path=/tmp/raft3718538488/snapshots/2-12-1733275981009.tmp
2024-12-04T01:33:01.009Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: snapshot complete up to: index=12
2024-12-04T01:33:01.011Z [INFO]  creating new snapshot: path=/tmp/raft3718538488/snapshots/2-1013-1733275981011.tmp
2024-12-04T01:33:01.012Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: copied to local snapshot: bytes=71
2024-12-04T01:33:01.012Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: snapshot restore progress: id=2-1013-1733275981011 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:01.012Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: restored user snapshot: index=1013
2024-12-04T01:33:01.012Z [ERROR] server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: failed to get log: index=23 error="log not found"
2024-12-04T01:33:01.012Z [ERROR] server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: failed to get log: index=23 error="log not found"
2024-12-04T01:33:01.012Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: aborting pipeline replication: peer="{Voter server-0086809b-0686-318b-2162-e6ff815cd2de 0086809b-0686-318b-2162-e6ff815cd2de}"
2024-12-04T01:33:01.012Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: aborting pipeline replication: peer="{Voter server-b51398d9-ef2e-796b-ec92-f4a653d029a4 b51398d9-ef2e-796b-ec92-f4a653d029a4}"
2024-12-04T01:33:01.021Z [ERROR] server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: failed to get log: index=23 error="log not found"
2024-12-04T01:33:01.022Z [ERROR] server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: failed to get log: index=23 error="log not found"
2024-12-04T01:33:01.023Z [INFO]  creating new snapshot: path=/tmp/raft3150503945/snapshots/2-1013-1733275981023.tmp
2024-12-04T01:33:01.023Z [INFO]  creating new snapshot: path=/tmp/raft3319781051/snapshots/2-1013-1733275981023.tmp
2024-12-04T01:33:01.024Z [INFO]  server-b51398d9-ef2e-796b-ec92-f4a653d029a4: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:01.024Z [INFO]  server-0086809b-0686-318b-2162-e6ff815cd2de: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:01.024Z [INFO]  server-b51398d9-ef2e-796b-ec92-f4a653d029a4: copied to local snapshot: bytes=71
2024-12-04T01:33:01.024Z [INFO]  server-0086809b-0686-318b-2162-e6ff815cd2de: copied to local snapshot: bytes=71
2024-12-04T01:33:01.024Z [INFO]  server-b51398d9-ef2e-796b-ec92-f4a653d029a4: snapshot restore progress: id=2-1013-1733275981023 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:01.024Z [INFO]  server-b51398d9-ef2e-796b-ec92-f4a653d029a4: Installed remote snapshot
2024-12-04T01:33:01.024Z [INFO]  server-0086809b-0686-318b-2162-e6ff815cd2de: snapshot restore progress: id=2-1013-1733275981023 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:01.024Z [INFO]  server-0086809b-0686-318b-2162-e6ff815cd2de: Installed remote snapshot
2024-12-04T01:33:01.025Z [WARN]  server-b51398d9-ef2e-796b-ec92-f4a653d029a4: skipping application of old log: index=22
2024-12-04T01:33:01.025Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: pipelining replication: peer="{Voter server-b51398d9-ef2e-796b-ec92-f4a653d029a4 b51398d9-ef2e-796b-ec92-f4a653d029a4}"
2024-12-04T01:33:01.025Z [WARN]  server-0086809b-0686-318b-2162-e6ff815cd2de: skipping application of old log: index=22
2024-12-04T01:33:01.025Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:01.025Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: pipelining replication: peer="{Voter server-0086809b-0686-318b-2162-e6ff815cd2de 0086809b-0686-318b-2162-e6ff815cd2de}"
2024-12-04T01:33:02.030Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:01.025197439 +0000 UTC m=+71.857040195 (2 nodes), 2.008µs from start of poll, 1.593799736s from cluster start. Timeout at 2024-12-04 01:33:02.030234809 +0000 UTC m=+72.862077588, 1.005037393s after stability
2024-12-04T01:33:02.043Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: aborting pipeline replication: peer="{Voter server-b51398d9-ef2e-796b-ec92-f4a653d029a4 b51398d9-ef2e-796b-ec92-f4a653d029a4}"
2024-12-04T01:33:02.043Z [INFO]  server-f567ffbf-02d4-a36f-dd4b-3aeda9a62fad: aborting pipeline replication: peer="{Voter server-0086809b-0686-318b-2162-e6ff815cd2de 0086809b-0686-318b-2162-e6ff815cd2de}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false
2024-12-04T01:33:02.045Z [INFO]  server-28898bf1-8f92-ffc5-28d5-62b6c9b19b5c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-28898bf1-8f92-ffc5-28d5-62b6c9b19b5c Address:28898bf1-8f92-ffc5-28d5-62b6c9b19b5c} {Suffrage:Voter ID:server-be56da09-a9ff-0c65-bfb6-36a3061378a6 Address:be56da09-a9ff-0c65-bfb6-36a3061378a6} {Suffrage:Voter ID:server-46b6529b-5267-4883-e247-03adf4819edb Address:46b6529b-5267-4883-e247-03adf4819edb}]"
2024-12-04T01:33:02.045Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-28898bf1-8f92-ffc5-28d5-62b6c9b19b5c Address:28898bf1-8f92-ffc5-28d5-62b6c9b19b5c} {Suffrage:Voter ID:server-be56da09-a9ff-0c65-bfb6-36a3061378a6 Address:be56da09-a9ff-0c65-bfb6-36a3061378a6} {Suffrage:Voter ID:server-46b6529b-5267-4883-e247-03adf4819edb Address:46b6529b-5267-4883-e247-03adf4819edb}]"
2024-12-04T01:33:02.045Z [INFO]  server-46b6529b-5267-4883-e247-03adf4819edb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-28898bf1-8f92-ffc5-28d5-62b6c9b19b5c Address:28898bf1-8f92-ffc5-28d5-62b6c9b19b5c} {Suffrage:Voter ID:server-be56da09-a9ff-0c65-bfb6-36a3061378a6 Address:be56da09-a9ff-0c65-bfb6-36a3061378a6} {Suffrage:Voter ID:server-46b6529b-5267-4883-e247-03adf4819edb Address:46b6529b-5267-4883-e247-03adf4819edb}]"
2024-12-04T01:33:02.045Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:02.045Z [INFO]  server-28898bf1-8f92-ffc5-28d5-62b6c9b19b5c: entering follower state: follower="Node at 28898bf1-8f92-ffc5-28d5-62b6c9b19b5c [Follower]" leader-address= leader-id=
2024-12-04T01:33:02.045Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: entering follower state: follower="Node at be56da09-a9ff-0c65-bfb6-36a3061378a6 [Follower]" leader-address= leader-id=
2024-12-04T01:33:02.045Z [INFO]  server-46b6529b-5267-4883-e247-03adf4819edb: entering follower state: follower="Node at 46b6529b-5267-4883-e247-03adf4819edb [Follower]" leader-address= leader-id=
2024-12-04T01:33:02.720Z [WARN]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:02.720Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: entering candidate state: node="Node at be56da09-a9ff-0c65-bfb6-36a3061378a6 [Candidate]" term=2
2024-12-04T01:33:02.720Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: election won: term=2 tally=2
2024-12-04T01:33:02.720Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: entering leader state: leader="Node at be56da09-a9ff-0c65-bfb6-36a3061378a6 [Leader]"
2024-12-04T01:33:02.720Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: added peer, starting replication: peer=server-28898bf1-8f92-ffc5-28d5-62b6c9b19b5c
2024-12-04T01:33:02.720Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: added peer, starting replication: peer=server-46b6529b-5267-4883-e247-03adf4819edb
2024-12-04T01:33:02.721Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: pipelining replication: peer="{Voter server-46b6529b-5267-4883-e247-03adf4819edb 46b6529b-5267-4883-e247-03adf4819edb}"
2024-12-04T01:33:02.721Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: pipelining replication: peer="{Voter server-28898bf1-8f92-ffc5-28d5-62b6c9b19b5c 28898bf1-8f92-ffc5-28d5-62b6c9b19b5c}"
2024-12-04T01:33:03.726Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:02.721026792 +0000 UTC m=+73.552869524 (1 nodes), 675.466195ms from start of poll, 675.814832ms from cluster start. Timeout at 2024-12-04 01:33:03.726041491 +0000 UTC m=+74.557884115, 1.005014591s after stability
2024-12-04T01:33:03.727Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: starting snapshot up to: index=12
2024-12-04T01:33:03.727Z [INFO]  creating new snapshot: path=/tmp/raft583367495/snapshots/2-12-1733275983727.tmp
2024-12-04T01:33:03.728Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: snapshot complete up to: index=12
2024-12-04T01:33:03.729Z [INFO]  creating new snapshot: path=/tmp/raft583367495/snapshots/2-1013-1733275983729.tmp
2024-12-04T01:33:03.730Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: copied to local snapshot: bytes=71
2024-12-04T01:33:03.731Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: snapshot restore progress: id=2-1013-1733275983729 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:03.731Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: removing all old logs from log store
2024-12-04T01:33:03.731Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: compacting logs: from=1 to=22
2024-12-04T01:33:03.731Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: restored user snapshot: index=1013
2024-12-04T01:33:03.731Z [ERROR] server-be56da09-a9ff-0c65-bfb6-36a3061378a6: failed to get log: index=22 error="log not found"
2024-12-04T01:33:03.731Z [ERROR] server-be56da09-a9ff-0c65-bfb6-36a3061378a6: failed to get log: index=22 error="log not found"
2024-12-04T01:33:03.731Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: aborting pipeline replication: peer="{Voter server-28898bf1-8f92-ffc5-28d5-62b6c9b19b5c 28898bf1-8f92-ffc5-28d5-62b6c9b19b5c}"
2024-12-04T01:33:03.731Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: aborting pipeline replication: peer="{Voter server-46b6529b-5267-4883-e247-03adf4819edb 46b6529b-5267-4883-e247-03adf4819edb}"
2024-12-04T01:33:03.737Z [ERROR] server-be56da09-a9ff-0c65-bfb6-36a3061378a6: failed to get log: index=22 error="log not found"
2024-12-04T01:33:03.738Z [INFO]  creating new snapshot: path=/tmp/raft1845503433/snapshots/2-1013-1733275983738.tmp
2024-12-04T01:33:03.738Z [ERROR] server-be56da09-a9ff-0c65-bfb6-36a3061378a6: failed to get log: index=22 error="log not found"
2024-12-04T01:33:03.738Z [INFO]  server-46b6529b-5267-4883-e247-03adf4819edb: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:03.739Z [INFO]  server-46b6529b-5267-4883-e247-03adf4819edb: copied to local snapshot: bytes=71
2024-12-04T01:33:03.739Z [INFO]  creating new snapshot: path=/tmp/raft3160316726/snapshots/2-1013-1733275983739.tmp
2024-12-04T01:33:03.739Z [INFO]  server-46b6529b-5267-4883-e247-03adf4819edb: snapshot restore progress: id=2-1013-1733275983738 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:03.739Z [INFO]  server-46b6529b-5267-4883-e247-03adf4819edb: removing all old logs from log store
2024-12-04T01:33:03.739Z [INFO]  server-46b6529b-5267-4883-e247-03adf4819edb: compacting logs: from=1 to=22
2024-12-04T01:33:03.739Z [INFO]  server-46b6529b-5267-4883-e247-03adf4819edb: Installed remote snapshot
2024-12-04T01:33:03.739Z [INFO]  server-28898bf1-8f92-ffc5-28d5-62b6c9b19b5c: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:03.740Z [WARN]  server-46b6529b-5267-4883-e247-03adf4819edb: skipping application of old log: index=22
2024-12-04T01:33:03.740Z [INFO]  server-28898bf1-8f92-ffc5-28d5-62b6c9b19b5c: copied to local snapshot: bytes=71
2024-12-04T01:33:03.740Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: pipelining replication: peer="{Voter server-46b6529b-5267-4883-e247-03adf4819edb 46b6529b-5267-4883-e247-03adf4819edb}"
2024-12-04T01:33:03.740Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:03.740Z [INFO]  server-28898bf1-8f92-ffc5-28d5-62b6c9b19b5c: snapshot restore progress: id=2-1013-1733275983739 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:03.740Z [INFO]  server-28898bf1-8f92-ffc5-28d5-62b6c9b19b5c: removing all old logs from log store
2024-12-04T01:33:03.740Z [INFO]  server-28898bf1-8f92-ffc5-28d5-62b6c9b19b5c: compacting logs: from=1 to=22
2024-12-04T01:33:03.740Z [INFO]  server-28898bf1-8f92-ffc5-28d5-62b6c9b19b5c: Installed remote snapshot
2024-12-04T01:33:03.740Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: pipelining replication: peer="{Voter server-28898bf1-8f92-ffc5-28d5-62b6c9b19b5c 28898bf1-8f92-ffc5-28d5-62b6c9b19b5c}"
2024-12-04T01:33:04.746Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:03.740238101 +0000 UTC m=+74.572080744 (2 nodes), 1.962µs from start of poll, 1.695026052s from cluster start. Timeout at 2024-12-04 01:33:04.745249886 +0000 UTC m=+75.577092593, 1.005011849s after stability
2024-12-04T01:33:04.759Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: aborting pipeline replication: peer="{Voter server-46b6529b-5267-4883-e247-03adf4819edb 46b6529b-5267-4883-e247-03adf4819edb}"
2024-12-04T01:33:04.759Z [INFO]  server-be56da09-a9ff-0c65-bfb6-36a3061378a6: aborting pipeline replication: peer="{Voter server-28898bf1-8f92-ffc5-28d5-62b6c9b19b5c 28898bf1-8f92-ffc5-28d5-62b6c9b19b5c}"
=== RUN   TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true
2024-12-04T01:33:04.761Z [INFO]  server-6e92f8a3-062f-42b9-626f-a7f4c8c0dd58: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e92f8a3-062f-42b9-626f-a7f4c8c0dd58 Address:6e92f8a3-062f-42b9-626f-a7f4c8c0dd58} {Suffrage:Voter ID:server-9490f051-dd3f-acce-7111-37e8680744f9 Address:9490f051-dd3f-acce-7111-37e8680744f9} {Suffrage:Voter ID:server-a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a Address:a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a}]"
2024-12-04T01:33:04.761Z [INFO]  server-9490f051-dd3f-acce-7111-37e8680744f9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e92f8a3-062f-42b9-626f-a7f4c8c0dd58 Address:6e92f8a3-062f-42b9-626f-a7f4c8c0dd58} {Suffrage:Voter ID:server-9490f051-dd3f-acce-7111-37e8680744f9 Address:9490f051-dd3f-acce-7111-37e8680744f9} {Suffrage:Voter ID:server-a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a Address:a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a}]"
2024-12-04T01:33:04.761Z [INFO]  server-a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6e92f8a3-062f-42b9-626f-a7f4c8c0dd58 Address:6e92f8a3-062f-42b9-626f-a7f4c8c0dd58} {Suffrage:Voter ID:server-9490f051-dd3f-acce-7111-37e8680744f9 Address:9490f051-dd3f-acce-7111-37e8680744f9} {Suffrage:Voter ID:server-a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a Address:a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a}]"
2024-12-04T01:33:04.761Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:04.761Z [INFO]  server-6e92f8a3-062f-42b9-626f-a7f4c8c0dd58: entering follower state: follower="Node at 6e92f8a3-062f-42b9-626f-a7f4c8c0dd58 [Follower]" leader-address= leader-id=
2024-12-04T01:33:04.761Z [INFO]  server-9490f051-dd3f-acce-7111-37e8680744f9: entering follower state: follower="Node at 9490f051-dd3f-acce-7111-37e8680744f9 [Follower]" leader-address= leader-id=
2024-12-04T01:33:04.761Z [INFO]  server-a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a: entering follower state: follower="Node at a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a [Follower]" leader-address= leader-id=
2024-12-04T01:33:05.358Z [WARN]  server-a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:05.358Z [INFO]  server-a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a: entering candidate state: node="Node at a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a [Candidate]" term=2
2024-12-04T01:33:05.359Z [INFO]  server-a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a: election won: term=2 tally=2
2024-12-04T01:33:05.359Z [INFO]  server-a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a: entering leader state: leader="Node at a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a [Leader]"
2024-12-04T01:33:05.359Z [INFO]  server-a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a: added peer, starting replication: peer=server-6e92f8a3-062f-42b9-626f-a7f4c8c0dd58
2024-12-04T01:33:05.359Z [INFO]  server-a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a: added peer, starting replication: peer=server-9490f051-dd3f-acce-7111-37e8680744f9
2024-12-04T01:33:05.359Z [INFO]  server-a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a: pipelining replication: peer="{Voter server-9490f051-dd3f-acce-7111-37e8680744f9 9490f051-dd3f-acce-7111-37e8680744f9}"
2024-12-04T01:33:05.359Z [INFO]  server-a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a: pipelining replication: peer="{Voter server-6e92f8a3-062f-42b9-626f-a7f4c8c0dd58 6e92f8a3-062f-42b9-626f-a7f4c8c0dd58}"
2024-12-04T01:33:06.364Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:05.359102877 +0000 UTC m=+76.190945465 (1 nodes), 597.31571ms from start of poll, 597.676877ms from cluster start. Timeout at 2024-12-04 01:33:06.364106858 +0000 UTC m=+77.195949651, 1.005004186s after stability
2024-12-04T01:33:06.366Z [INFO]  server-a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a: starting snapshot up to: index=12
2024-12-04T01:33:06.366Z [INFO]  creating new snapshot: path=/tmp/raft4033502626/snapshots/2-12-1733275986366.tmp
2024-12-04T01:33:06.368Z [INFO]  server-a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a: snapshot complete up to: index=12
2024-12-04T01:33:06.376Z [INFO]  server-1cec4897-5280-bae7-37fa-56fded64164f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cec4897-5280-bae7-37fa-56fded64164f Address:1cec4897-5280-bae7-37fa-56fded64164f} {Suffrage:Voter ID:server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b Address:6ff1e93b-b47c-b387-0a1d-83375b9b2c4b} {Suffrage:Voter ID:server-0fcef7c9-452d-ed92-d546-8cce036ad70d Address:0fcef7c9-452d-ed92-d546-8cce036ad70d}]"
2024-12-04T01:33:06.378Z [INFO]  server-1cec4897-5280-bae7-37fa-56fded64164f: entering follower state: follower="Node at 1cec4897-5280-bae7-37fa-56fded64164f [Follower]" leader-address= leader-id=
2024-12-04T01:33:06.378Z [INFO]  server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cec4897-5280-bae7-37fa-56fded64164f Address:1cec4897-5280-bae7-37fa-56fded64164f} {Suffrage:Voter ID:server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b Address:6ff1e93b-b47c-b387-0a1d-83375b9b2c4b} {Suffrage:Voter ID:server-0fcef7c9-452d-ed92-d546-8cce036ad70d Address:0fcef7c9-452d-ed92-d546-8cce036ad70d}]"
2024-12-04T01:33:06.378Z [INFO]  server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: entering follower state: follower="Node at 6ff1e93b-b47c-b387-0a1d-83375b9b2c4b [Follower]" leader-address= leader-id=
2024-12-04T01:33:06.378Z [INFO]  server-0fcef7c9-452d-ed92-d546-8cce036ad70d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cec4897-5280-bae7-37fa-56fded64164f Address:1cec4897-5280-bae7-37fa-56fded64164f} {Suffrage:Voter ID:server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b Address:6ff1e93b-b47c-b387-0a1d-83375b9b2c4b} {Suffrage:Voter ID:server-0fcef7c9-452d-ed92-d546-8cce036ad70d Address:0fcef7c9-452d-ed92-d546-8cce036ad70d}]"
2024-12-04T01:33:06.378Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:06.378Z [INFO]  server-0fcef7c9-452d-ed92-d546-8cce036ad70d: entering follower state: follower="Node at 0fcef7c9-452d-ed92-d546-8cce036ad70d [Follower]" leader-address= leader-id=
2024-12-04T01:33:07.208Z [WARN]  server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:07.208Z [INFO]  server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: entering candidate state: node="Node at 6ff1e93b-b47c-b387-0a1d-83375b9b2c4b [Candidate]" term=2
2024-12-04T01:33:07.208Z [INFO]  server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: election won: term=2 tally=2
2024-12-04T01:33:07.208Z [INFO]  server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: entering leader state: leader="Node at 6ff1e93b-b47c-b387-0a1d-83375b9b2c4b [Leader]"
2024-12-04T01:33:07.208Z [INFO]  server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: added peer, starting replication: peer=server-1cec4897-5280-bae7-37fa-56fded64164f
2024-12-04T01:33:07.208Z [INFO]  server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: added peer, starting replication: peer=server-0fcef7c9-452d-ed92-d546-8cce036ad70d
2024-12-04T01:33:07.208Z [INFO]  server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: pipelining replication: peer="{Voter server-1cec4897-5280-bae7-37fa-56fded64164f 1cec4897-5280-bae7-37fa-56fded64164f}"
2024-12-04T01:33:07.208Z [INFO]  server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: pipelining replication: peer="{Voter server-0fcef7c9-452d-ed92-d546-8cce036ad70d 0fcef7c9-452d-ed92-d546-8cce036ad70d}"
2024-12-04T01:33:08.215Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:07.209170234 +0000 UTC m=+78.041013001 (1 nodes), 830.617868ms from start of poll, 832.959882ms from cluster start. Timeout at 2024-12-04 01:33:08.214206883 +0000 UTC m=+79.046049658, 1.005036657s after stability
2024-12-04T01:33:08.215Z [INFO]  creating new snapshot: path=/tmp/raft1313977242/snapshots/2-1013-1733275988215.tmp
2024-12-04T01:33:08.216Z [INFO]  server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: copied to local snapshot: bytes=71
2024-12-04T01:33:08.216Z [INFO]  server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: snapshot restore progress: id=2-1013-1733275988215 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:08.216Z [INFO]  server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: restored user snapshot: index=1013
2024-12-04T01:33:08.217Z [ERROR] server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: failed to get log: index=3 error="log not found"
2024-12-04T01:33:08.217Z [INFO]  server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: aborting pipeline replication: peer="{Voter server-0fcef7c9-452d-ed92-d546-8cce036ad70d 0fcef7c9-452d-ed92-d546-8cce036ad70d}"
2024-12-04T01:33:08.217Z [ERROR] server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: failed to get log: index=3 error="log not found"
2024-12-04T01:33:08.217Z [INFO]  server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: aborting pipeline replication: peer="{Voter server-1cec4897-5280-bae7-37fa-56fded64164f 1cec4897-5280-bae7-37fa-56fded64164f}"
2024-12-04T01:33:08.226Z [ERROR] server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: failed to get log: index=3 error="log not found"
2024-12-04T01:33:08.227Z [ERROR] server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: failed to get log: index=3 error="log not found"
2024-12-04T01:33:08.227Z [INFO]  creating new snapshot: path=/tmp/raft154768772/snapshots/2-1013-1733275988227.tmp
2024-12-04T01:33:08.227Z [INFO]  server-1cec4897-5280-bae7-37fa-56fded64164f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:08.228Z [INFO]  server-1cec4897-5280-bae7-37fa-56fded64164f: copied to local snapshot: bytes=71
2024-12-04T01:33:08.228Z [INFO]  server-1cec4897-5280-bae7-37fa-56fded64164f: snapshot restore progress: id=2-1013-1733275988227 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:08.228Z [INFO]  server-1cec4897-5280-bae7-37fa-56fded64164f: Installed remote snapshot
2024-12-04T01:33:08.228Z [INFO]  server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: pipelining replication: peer="{Voter server-1cec4897-5280-bae7-37fa-56fded64164f 1cec4897-5280-bae7-37fa-56fded64164f}"
2024-12-04T01:33:08.228Z [INFO]  creating new snapshot: path=/tmp/raft2764043934/snapshots/2-1013-1733275988228.tmp
2024-12-04T01:33:08.229Z [INFO]  server-0fcef7c9-452d-ed92-d546-8cce036ad70d: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:08.229Z [INFO]  server-0fcef7c9-452d-ed92-d546-8cce036ad70d: copied to local snapshot: bytes=71
2024-12-04T01:33:08.229Z [INFO]  server-0fcef7c9-452d-ed92-d546-8cce036ad70d: snapshot restore progress: id=2-1013-1733275988228 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:08.229Z [INFO]  server-0fcef7c9-452d-ed92-d546-8cce036ad70d: Installed remote snapshot
2024-12-04T01:33:08.229Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:08.230Z [INFO]  server-6ff1e93b-b47c-b387-0a1d-83375b9b2c4b: pipelining replication: peer="{Voter server-0fcef7c9-452d-ed92-d546-8cce036ad70d 0fcef7c9-452d-ed92-d546-8cce036ad70d}"
2024-12-04T01:33:09.236Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:08.229929167 +0000 UTC m=+79.061771930 (2 nodes), 2.145µs from start of poll, 1.853718811s from cluster start. Timeout at 2024-12-04 01:33:09.234962868 +0000 UTC m=+80.066805478, 1.005033548s after stability
2024-12-04T01:33:09.247Z [INFO]  server-a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a: aborting pipeline replication: peer="{Voter server-9490f051-dd3f-acce-7111-37e8680744f9 9490f051-dd3f-acce-7111-37e8680744f9}"
2024-12-04T01:33:09.247Z [INFO]  server-a1503b5d-df7b-c64b-4f6f-3a5f0ea1af7a: aborting pipeline replication: peer="{Voter server-6e92f8a3-062f-42b9-626f-a7f4c8c0dd58 6e92f8a3-062f-42b9-626f-a7f4c8c0dd58}"
=== RUN   TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true
2024-12-04T01:33:09.249Z [INFO]  server-e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd Address:e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd} {Suffrage:Voter ID:server-a3d404d3-0e4f-892e-87a3-7a74660e188b Address:a3d404d3-0e4f-892e-87a3-7a74660e188b} {Suffrage:Voter ID:server-c44a5ef2-8de6-0cf0-f1a8-bcbd41ed5fca Address:c44a5ef2-8de6-0cf0-f1a8-bcbd41ed5fca}]"
2024-12-04T01:33:09.249Z [INFO]  server-a3d404d3-0e4f-892e-87a3-7a74660e188b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd Address:e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd} {Suffrage:Voter ID:server-a3d404d3-0e4f-892e-87a3-7a74660e188b Address:a3d404d3-0e4f-892e-87a3-7a74660e188b} {Suffrage:Voter ID:server-c44a5ef2-8de6-0cf0-f1a8-bcbd41ed5fca Address:c44a5ef2-8de6-0cf0-f1a8-bcbd41ed5fca}]"
2024-12-04T01:33:09.249Z [INFO]  server-c44a5ef2-8de6-0cf0-f1a8-bcbd41ed5fca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd Address:e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd} {Suffrage:Voter ID:server-a3d404d3-0e4f-892e-87a3-7a74660e188b Address:a3d404d3-0e4f-892e-87a3-7a74660e188b} {Suffrage:Voter ID:server-c44a5ef2-8de6-0cf0-f1a8-bcbd41ed5fca Address:c44a5ef2-8de6-0cf0-f1a8-bcbd41ed5fca}]"
2024-12-04T01:33:09.249Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:09.249Z [INFO]  server-e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd: entering follower state: follower="Node at e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd [Follower]" leader-address= leader-id=
2024-12-04T01:33:09.249Z [INFO]  server-a3d404d3-0e4f-892e-87a3-7a74660e188b: entering follower state: follower="Node at a3d404d3-0e4f-892e-87a3-7a74660e188b [Follower]" leader-address= leader-id=
2024-12-04T01:33:09.249Z [INFO]  server-c44a5ef2-8de6-0cf0-f1a8-bcbd41ed5fca: entering follower state: follower="Node at c44a5ef2-8de6-0cf0-f1a8-bcbd41ed5fca [Follower]" leader-address= leader-id=
2024-12-04T01:33:09.801Z [WARN]  server-e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:09.801Z [INFO]  server-e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd: entering candidate state: node="Node at e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd [Candidate]" term=2
2024-12-04T01:33:09.801Z [INFO]  server-e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd: election won: term=2 tally=2
2024-12-04T01:33:09.801Z [INFO]  server-e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd: entering leader state: leader="Node at e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd [Leader]"
2024-12-04T01:33:09.801Z [INFO]  server-e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd: added peer, starting replication: peer=server-a3d404d3-0e4f-892e-87a3-7a74660e188b
2024-12-04T01:33:09.801Z [INFO]  server-e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd: added peer, starting replication: peer=server-c44a5ef2-8de6-0cf0-f1a8-bcbd41ed5fca
2024-12-04T01:33:09.802Z [INFO]  server-e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd: pipelining replication: peer="{Voter server-a3d404d3-0e4f-892e-87a3-7a74660e188b a3d404d3-0e4f-892e-87a3-7a74660e188b}"
2024-12-04T01:33:09.802Z [INFO]  server-e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd: pipelining replication: peer="{Voter server-c44a5ef2-8de6-0cf0-f1a8-bcbd41ed5fca c44a5ef2-8de6-0cf0-f1a8-bcbd41ed5fca}"
2024-12-04T01:33:10.808Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:09.802469594 +0000 UTC m=+80.634312220 (1 nodes), 552.863109ms from start of poll, 553.218076ms from cluster start. Timeout at 2024-12-04 01:33:10.8074863 +0000 UTC m=+81.639328959, 1.005016739s after stability
2024-12-04T01:33:10.809Z [INFO]  server-e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd: starting snapshot up to: index=12
2024-12-04T01:33:10.809Z [INFO]  creating new snapshot: path=/tmp/raft2973888912/snapshots/2-12-1733275990809.tmp
2024-12-04T01:33:10.810Z [INFO]  server-e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd: snapshot complete up to: index=12
2024-12-04T01:33:10.822Z [INFO]  server-e986bf1b-cbdb-7884-ac70-3512ba4886b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e986bf1b-cbdb-7884-ac70-3512ba4886b0 Address:e986bf1b-cbdb-7884-ac70-3512ba4886b0} {Suffrage:Voter ID:server-0988744e-6273-8b65-38f9-6e48d4d80427 Address:0988744e-6273-8b65-38f9-6e48d4d80427} {Suffrage:Voter ID:server-26a79d3b-075f-b6fa-1b4d-f8dc658aecc0 Address:26a79d3b-075f-b6fa-1b4d-f8dc658aecc0}]"
2024-12-04T01:33:10.822Z [INFO]  server-e986bf1b-cbdb-7884-ac70-3512ba4886b0: entering follower state: follower="Node at e986bf1b-cbdb-7884-ac70-3512ba4886b0 [Follower]" leader-address= leader-id=
2024-12-04T01:33:10.822Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e986bf1b-cbdb-7884-ac70-3512ba4886b0 Address:e986bf1b-cbdb-7884-ac70-3512ba4886b0} {Suffrage:Voter ID:server-0988744e-6273-8b65-38f9-6e48d4d80427 Address:0988744e-6273-8b65-38f9-6e48d4d80427} {Suffrage:Voter ID:server-26a79d3b-075f-b6fa-1b4d-f8dc658aecc0 Address:26a79d3b-075f-b6fa-1b4d-f8dc658aecc0}]"
2024-12-04T01:33:10.822Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: entering follower state: follower="Node at 0988744e-6273-8b65-38f9-6e48d4d80427 [Follower]" leader-address= leader-id=
2024-12-04T01:33:10.822Z [INFO]  server-26a79d3b-075f-b6fa-1b4d-f8dc658aecc0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e986bf1b-cbdb-7884-ac70-3512ba4886b0 Address:e986bf1b-cbdb-7884-ac70-3512ba4886b0} {Suffrage:Voter ID:server-0988744e-6273-8b65-38f9-6e48d4d80427 Address:0988744e-6273-8b65-38f9-6e48d4d80427} {Suffrage:Voter ID:server-26a79d3b-075f-b6fa-1b4d-f8dc658aecc0 Address:26a79d3b-075f-b6fa-1b4d-f8dc658aecc0}]"
2024-12-04T01:33:10.823Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:10.823Z [INFO]  server-26a79d3b-075f-b6fa-1b4d-f8dc658aecc0: entering follower state: follower="Node at 26a79d3b-075f-b6fa-1b4d-f8dc658aecc0 [Follower]" leader-address= leader-id=
2024-12-04T01:33:11.387Z [WARN]  server-0988744e-6273-8b65-38f9-6e48d4d80427: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:11.387Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: entering candidate state: node="Node at 0988744e-6273-8b65-38f9-6e48d4d80427 [Candidate]" term=2
2024-12-04T01:33:11.387Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: election won: term=2 tally=2
2024-12-04T01:33:11.387Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: entering leader state: leader="Node at 0988744e-6273-8b65-38f9-6e48d4d80427 [Leader]"
2024-12-04T01:33:11.387Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: added peer, starting replication: peer=server-e986bf1b-cbdb-7884-ac70-3512ba4886b0
2024-12-04T01:33:11.387Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: added peer, starting replication: peer=server-26a79d3b-075f-b6fa-1b4d-f8dc658aecc0
2024-12-04T01:33:11.388Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: pipelining replication: peer="{Voter server-26a79d3b-075f-b6fa-1b4d-f8dc658aecc0 26a79d3b-075f-b6fa-1b4d-f8dc658aecc0}"
2024-12-04T01:33:11.388Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: pipelining replication: peer="{Voter server-e986bf1b-cbdb-7884-ac70-3512ba4886b0 e986bf1b-cbdb-7884-ac70-3512ba4886b0}"
2024-12-04T01:33:12.393Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:11.388046693 +0000 UTC m=+82.219889473 (1 nodes), 565.013557ms from start of poll, 565.543576ms from cluster start. Timeout at 2024-12-04 01:33:12.393053199 +0000 UTC m=+83.224895794, 1.005006321s after stability
2024-12-04T01:33:12.394Z [INFO]  creating new snapshot: path=/tmp/raft570407288/snapshots/2-1013-1733275992394.tmp
2024-12-04T01:33:12.394Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: copied to local snapshot: bytes=71
2024-12-04T01:33:12.395Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: snapshot restore progress: id=2-1013-1733275992394 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:12.395Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: removing all old logs from log store
2024-12-04T01:33:12.395Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: compacting logs: from=1 to=2
2024-12-04T01:33:12.395Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: restored user snapshot: index=1013
2024-12-04T01:33:12.395Z [ERROR] server-0988744e-6273-8b65-38f9-6e48d4d80427: failed to get log: index=2 error="log not found"
2024-12-04T01:33:12.395Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: aborting pipeline replication: peer="{Voter server-26a79d3b-075f-b6fa-1b4d-f8dc658aecc0 26a79d3b-075f-b6fa-1b4d-f8dc658aecc0}"
2024-12-04T01:33:12.395Z [ERROR] server-0988744e-6273-8b65-38f9-6e48d4d80427: failed to get log: index=2 error="log not found"
2024-12-04T01:33:12.395Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: aborting pipeline replication: peer="{Voter server-e986bf1b-cbdb-7884-ac70-3512ba4886b0 e986bf1b-cbdb-7884-ac70-3512ba4886b0}"
2024-12-04T01:33:12.404Z [ERROR] server-0988744e-6273-8b65-38f9-6e48d4d80427: failed to get log: index=2 error="log not found"
2024-12-04T01:33:12.405Z [ERROR] server-0988744e-6273-8b65-38f9-6e48d4d80427: failed to get log: index=2 error="log not found"
2024-12-04T01:33:12.405Z [INFO]  creating new snapshot: path=/tmp/raft2339044309/snapshots/2-1013-1733275992405.tmp
2024-12-04T01:33:12.405Z [INFO]  server-e986bf1b-cbdb-7884-ac70-3512ba4886b0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:12.406Z [INFO]  creating new snapshot: path=/tmp/raft2807863142/snapshots/2-1013-1733275992406.tmp
2024-12-04T01:33:12.406Z [INFO]  server-e986bf1b-cbdb-7884-ac70-3512ba4886b0: copied to local snapshot: bytes=71
2024-12-04T01:33:12.406Z [INFO]  server-26a79d3b-075f-b6fa-1b4d-f8dc658aecc0: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:12.406Z [INFO]  server-e986bf1b-cbdb-7884-ac70-3512ba4886b0: snapshot restore progress: id=2-1013-1733275992405 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:12.406Z [INFO]  server-e986bf1b-cbdb-7884-ac70-3512ba4886b0: removing all old logs from log store
2024-12-04T01:33:12.406Z [INFO]  server-e986bf1b-cbdb-7884-ac70-3512ba4886b0: compacting logs: from=1 to=2
2024-12-04T01:33:12.406Z [INFO]  server-e986bf1b-cbdb-7884-ac70-3512ba4886b0: Installed remote snapshot
2024-12-04T01:33:12.406Z [INFO]  server-26a79d3b-075f-b6fa-1b4d-f8dc658aecc0: copied to local snapshot: bytes=71
2024-12-04T01:33:12.407Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: pipelining replication: peer="{Voter server-e986bf1b-cbdb-7884-ac70-3512ba4886b0 e986bf1b-cbdb-7884-ac70-3512ba4886b0}"
2024-12-04T01:33:12.407Z [INFO]  server-26a79d3b-075f-b6fa-1b4d-f8dc658aecc0: snapshot restore progress: id=2-1013-1733275992406 last-index=1013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:12.407Z [INFO]  server-26a79d3b-075f-b6fa-1b4d-f8dc658aecc0: removing all old logs from log store
2024-12-04T01:33:12.407Z [INFO]  server-26a79d3b-075f-b6fa-1b4d-f8dc658aecc0: compacting logs: from=1 to=2
2024-12-04T01:33:12.407Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:12.407Z [INFO]  server-26a79d3b-075f-b6fa-1b4d-f8dc658aecc0: Installed remote snapshot
2024-12-04T01:33:12.407Z [INFO]  server-0988744e-6273-8b65-38f9-6e48d4d80427: pipelining replication: peer="{Voter server-26a79d3b-075f-b6fa-1b4d-f8dc658aecc0 26a79d3b-075f-b6fa-1b4d-f8dc658aecc0}"
2024-12-04T01:33:13.412Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:12.407379509 +0000 UTC m=+83.239222090 (2 nodes), 2.364µs from start of poll, 1.584876193s from cluster start. Timeout at 2024-12-04 01:33:13.412436084 +0000 UTC m=+84.244278691, 1.005056601s after stability
2024-12-04T01:33:13.425Z [INFO]  server-e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd: aborting pipeline replication: peer="{Voter server-a3d404d3-0e4f-892e-87a3-7a74660e188b a3d404d3-0e4f-892e-87a3-7a74660e188b}"
2024-12-04T01:33:13.425Z [INFO]  server-e089e4a7-2ea0-e1b4-b3b2-b624f21f56fd: aborting pipeline replication: peer="{Voter server-c44a5ef2-8de6-0cf0-f1a8-bcbd41ed5fca c44a5ef2-8de6-0cf0-f1a8-bcbd41ed5fca}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false
2024-12-04T01:33:13.427Z [INFO]  server-4eb10287-37d7-2223-d1df-174fbdcb0166: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4eb10287-37d7-2223-d1df-174fbdcb0166 Address:4eb10287-37d7-2223-d1df-174fbdcb0166} {Suffrage:Voter ID:server-e8bb5bec-860e-bb08-b23b-c320293fe114 Address:e8bb5bec-860e-bb08-b23b-c320293fe114} {Suffrage:Voter ID:server-e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc Address:e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc}]"
2024-12-04T01:33:13.427Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4eb10287-37d7-2223-d1df-174fbdcb0166 Address:4eb10287-37d7-2223-d1df-174fbdcb0166} {Suffrage:Voter ID:server-e8bb5bec-860e-bb08-b23b-c320293fe114 Address:e8bb5bec-860e-bb08-b23b-c320293fe114} {Suffrage:Voter ID:server-e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc Address:e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc}]"
2024-12-04T01:33:13.427Z [INFO]  server-e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4eb10287-37d7-2223-d1df-174fbdcb0166 Address:4eb10287-37d7-2223-d1df-174fbdcb0166} {Suffrage:Voter ID:server-e8bb5bec-860e-bb08-b23b-c320293fe114 Address:e8bb5bec-860e-bb08-b23b-c320293fe114} {Suffrage:Voter ID:server-e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc Address:e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc}]"
2024-12-04T01:33:13.427Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:13.427Z [INFO]  server-4eb10287-37d7-2223-d1df-174fbdcb0166: entering follower state: follower="Node at 4eb10287-37d7-2223-d1df-174fbdcb0166 [Follower]" leader-address= leader-id=
2024-12-04T01:33:13.427Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: entering follower state: follower="Node at e8bb5bec-860e-bb08-b23b-c320293fe114 [Follower]" leader-address= leader-id=
2024-12-04T01:33:13.427Z [INFO]  server-e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc: entering follower state: follower="Node at e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc [Follower]" leader-address= leader-id=
2024-12-04T01:33:14.112Z [WARN]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:14.112Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: entering candidate state: node="Node at e8bb5bec-860e-bb08-b23b-c320293fe114 [Candidate]" term=2
2024-12-04T01:33:14.112Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: election won: term=2 tally=2
2024-12-04T01:33:14.112Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: entering leader state: leader="Node at e8bb5bec-860e-bb08-b23b-c320293fe114 [Leader]"
2024-12-04T01:33:14.112Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: added peer, starting replication: peer=server-4eb10287-37d7-2223-d1df-174fbdcb0166
2024-12-04T01:33:14.112Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: added peer, starting replication: peer=server-e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc
2024-12-04T01:33:14.112Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: pipelining replication: peer="{Voter server-4eb10287-37d7-2223-d1df-174fbdcb0166 4eb10287-37d7-2223-d1df-174fbdcb0166}"
2024-12-04T01:33:14.112Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: pipelining replication: peer="{Voter server-e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc}"
2024-12-04T01:33:15.118Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:14.112606149 +0000 UTC m=+84.944448939 (1 nodes), 685.071732ms from start of poll, 685.434739ms from cluster start. Timeout at 2024-12-04 01:33:15.117635042 +0000 UTC m=+85.949478663, 1.005029724s after stability
2024-12-04T01:33:15.119Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: starting snapshot up to: index=12
2024-12-04T01:33:15.119Z [INFO]  creating new snapshot: path=/tmp/raft3531418008/snapshots/2-12-1733275995119.tmp
2024-12-04T01:33:15.120Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: snapshot complete up to: index=12
2024-12-04T01:33:15.121Z [INFO]  creating new snapshot: path=/tmp/raft3531418008/snapshots/2-10013-1733275995121.tmp
2024-12-04T01:33:15.122Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: copied to local snapshot: bytes=71
2024-12-04T01:33:15.123Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: snapshot restore progress: id=2-10013-1733275995121 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:15.123Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: restored user snapshot: index=10013
2024-12-04T01:33:15.123Z [ERROR] server-e8bb5bec-860e-bb08-b23b-c320293fe114: failed to get log: index=23 error="log not found"
2024-12-04T01:33:15.123Z [ERROR] server-e8bb5bec-860e-bb08-b23b-c320293fe114: failed to get log: index=23 error="log not found"
2024-12-04T01:33:15.123Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: aborting pipeline replication: peer="{Voter server-e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc}"
2024-12-04T01:33:15.123Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: aborting pipeline replication: peer="{Voter server-4eb10287-37d7-2223-d1df-174fbdcb0166 4eb10287-37d7-2223-d1df-174fbdcb0166}"
2024-12-04T01:33:15.132Z [ERROR] server-e8bb5bec-860e-bb08-b23b-c320293fe114: failed to get log: index=23 error="log not found"
2024-12-04T01:33:15.132Z [INFO]  creating new snapshot: path=/tmp/raft2091929124/snapshots/2-10013-1733275995132.tmp
2024-12-04T01:33:15.133Z [INFO]  server-4eb10287-37d7-2223-d1df-174fbdcb0166: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:15.133Z [INFO]  server-4eb10287-37d7-2223-d1df-174fbdcb0166: copied to local snapshot: bytes=71
2024-12-04T01:33:15.133Z [INFO]  server-4eb10287-37d7-2223-d1df-174fbdcb0166: snapshot restore progress: id=2-10013-1733275995132 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:15.133Z [INFO]  server-4eb10287-37d7-2223-d1df-174fbdcb0166: Installed remote snapshot
2024-12-04T01:33:15.134Z [WARN]  server-4eb10287-37d7-2223-d1df-174fbdcb0166: skipping application of old log: index=22
2024-12-04T01:33:15.134Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: pipelining replication: peer="{Voter server-4eb10287-37d7-2223-d1df-174fbdcb0166 4eb10287-37d7-2223-d1df-174fbdcb0166}"
2024-12-04T01:33:15.134Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:15.134Z [ERROR] server-e8bb5bec-860e-bb08-b23b-c320293fe114: failed to get log: index=23 error="log not found"
2024-12-04T01:33:15.134Z [INFO]  creating new snapshot: path=/tmp/raft102566250/snapshots/2-10013-1733275995134.tmp
2024-12-04T01:33:15.135Z [INFO]  server-e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:15.135Z [INFO]  server-e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc: copied to local snapshot: bytes=71
2024-12-04T01:33:15.135Z [INFO]  server-e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc: snapshot restore progress: id=2-10013-1733275995134 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:15.135Z [INFO]  server-e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc: Installed remote snapshot
2024-12-04T01:33:15.136Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: pipelining replication: peer="{Voter server-e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc}"
2024-12-04T01:33:16.140Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:15.134371076 +0000 UTC m=+85.966213971 (2 nodes), 2.425µs from start of poll, 1.707199771s from cluster start. Timeout at 2024-12-04 01:33:16.139428953 +0000 UTC m=+86.971271755, 1.005057784s after stability
2024-12-04T01:33:16.155Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: aborting pipeline replication: peer="{Voter server-4eb10287-37d7-2223-d1df-174fbdcb0166 4eb10287-37d7-2223-d1df-174fbdcb0166}"
2024-12-04T01:33:16.155Z [INFO]  server-e8bb5bec-860e-bb08-b23b-c320293fe114: aborting pipeline replication: peer="{Voter server-e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc e18bde1b-ede5-0a28-8d5b-65afd6b2a8bc}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false
2024-12-04T01:33:16.158Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18238386-148f-29d9-0d5a-1b4100d5f514 Address:18238386-148f-29d9-0d5a-1b4100d5f514} {Suffrage:Voter ID:server-1fe75b9b-9d96-8b5b-9299-7ed5b6193c70 Address:1fe75b9b-9d96-8b5b-9299-7ed5b6193c70} {Suffrage:Voter ID:server-f519eaba-b942-5645-c943-863bbfc9c3fe Address:f519eaba-b942-5645-c943-863bbfc9c3fe}]"
2024-12-04T01:33:16.158Z [INFO]  server-1fe75b9b-9d96-8b5b-9299-7ed5b6193c70: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18238386-148f-29d9-0d5a-1b4100d5f514 Address:18238386-148f-29d9-0d5a-1b4100d5f514} {Suffrage:Voter ID:server-1fe75b9b-9d96-8b5b-9299-7ed5b6193c70 Address:1fe75b9b-9d96-8b5b-9299-7ed5b6193c70} {Suffrage:Voter ID:server-f519eaba-b942-5645-c943-863bbfc9c3fe Address:f519eaba-b942-5645-c943-863bbfc9c3fe}]"
2024-12-04T01:33:16.158Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: entering follower state: follower="Node at 18238386-148f-29d9-0d5a-1b4100d5f514 [Follower]" leader-address= leader-id=
2024-12-04T01:33:16.158Z [INFO]  server-1fe75b9b-9d96-8b5b-9299-7ed5b6193c70: entering follower state: follower="Node at 1fe75b9b-9d96-8b5b-9299-7ed5b6193c70 [Follower]" leader-address= leader-id=
2024-12-04T01:33:16.158Z [INFO]  server-f519eaba-b942-5645-c943-863bbfc9c3fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-18238386-148f-29d9-0d5a-1b4100d5f514 Address:18238386-148f-29d9-0d5a-1b4100d5f514} {Suffrage:Voter ID:server-1fe75b9b-9d96-8b5b-9299-7ed5b6193c70 Address:1fe75b9b-9d96-8b5b-9299-7ed5b6193c70} {Suffrage:Voter ID:server-f519eaba-b942-5645-c943-863bbfc9c3fe Address:f519eaba-b942-5645-c943-863bbfc9c3fe}]"
2024-12-04T01:33:16.158Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:16.158Z [INFO]  server-f519eaba-b942-5645-c943-863bbfc9c3fe: entering follower state: follower="Node at f519eaba-b942-5645-c943-863bbfc9c3fe [Follower]" leader-address= leader-id=
2024-12-04T01:33:16.971Z [WARN]  server-18238386-148f-29d9-0d5a-1b4100d5f514: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:16.971Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: entering candidate state: node="Node at 18238386-148f-29d9-0d5a-1b4100d5f514 [Candidate]" term=2
2024-12-04T01:33:16.971Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: election won: term=2 tally=2
2024-12-04T01:33:16.971Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: entering leader state: leader="Node at 18238386-148f-29d9-0d5a-1b4100d5f514 [Leader]"
2024-12-04T01:33:16.971Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: added peer, starting replication: peer=server-1fe75b9b-9d96-8b5b-9299-7ed5b6193c70
2024-12-04T01:33:16.971Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: added peer, starting replication: peer=server-f519eaba-b942-5645-c943-863bbfc9c3fe
2024-12-04T01:33:16.972Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: pipelining replication: peer="{Voter server-f519eaba-b942-5645-c943-863bbfc9c3fe f519eaba-b942-5645-c943-863bbfc9c3fe}"
2024-12-04T01:33:16.972Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: pipelining replication: peer="{Voter server-1fe75b9b-9d96-8b5b-9299-7ed5b6193c70 1fe75b9b-9d96-8b5b-9299-7ed5b6193c70}"
2024-12-04T01:33:17.984Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:16.972379927 +0000 UTC m=+87.804222510 (1 nodes), 813.925288ms from start of poll, 814.466926ms from cluster start. Timeout at 2024-12-04 01:33:17.977388491 +0000 UTC m=+88.809231303, 1.005008793s after stability
2024-12-04T01:33:17.985Z [WARN]  server-f519eaba-b942-5645-c943-863bbfc9c3fe: skipping application of old log: index=2
2024-12-04T01:33:17.986Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: starting snapshot up to: index=12
2024-12-04T01:33:17.986Z [INFO]  creating new snapshot: path=/tmp/raft1652221274/snapshots/2-12-1733275997986.tmp
2024-12-04T01:33:17.987Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: snapshot complete up to: index=12
2024-12-04T01:33:17.988Z [INFO]  creating new snapshot: path=/tmp/raft1652221274/snapshots/2-10013-1733275997988.tmp
2024-12-04T01:33:17.988Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: copied to local snapshot: bytes=71
2024-12-04T01:33:17.989Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: snapshot restore progress: id=2-10013-1733275997988 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:17.989Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: removing all old logs from log store
2024-12-04T01:33:17.989Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: compacting logs: from=1 to=22
2024-12-04T01:33:17.989Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: restored user snapshot: index=10013
2024-12-04T01:33:17.989Z [ERROR] server-18238386-148f-29d9-0d5a-1b4100d5f514: failed to get log: index=22 error="log not found"
2024-12-04T01:33:17.989Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: aborting pipeline replication: peer="{Voter server-f519eaba-b942-5645-c943-863bbfc9c3fe f519eaba-b942-5645-c943-863bbfc9c3fe}"
2024-12-04T01:33:17.989Z [ERROR] server-18238386-148f-29d9-0d5a-1b4100d5f514: failed to get log: index=22 error="log not found"
2024-12-04T01:33:17.989Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: aborting pipeline replication: peer="{Voter server-1fe75b9b-9d96-8b5b-9299-7ed5b6193c70 1fe75b9b-9d96-8b5b-9299-7ed5b6193c70}"
2024-12-04T01:33:17.996Z [ERROR] server-18238386-148f-29d9-0d5a-1b4100d5f514: failed to get log: index=22 error="log not found"
2024-12-04T01:33:17.997Z [INFO]  creating new snapshot: path=/tmp/raft1040133129/snapshots/2-10013-1733275997997.tmp
2024-12-04T01:33:17.997Z [INFO]  server-f519eaba-b942-5645-c943-863bbfc9c3fe: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:17.998Z [INFO]  server-f519eaba-b942-5645-c943-863bbfc9c3fe: copied to local snapshot: bytes=71
2024-12-04T01:33:17.998Z [INFO]  server-f519eaba-b942-5645-c943-863bbfc9c3fe: snapshot restore progress: id=2-10013-1733275997997 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:17.998Z [INFO]  server-f519eaba-b942-5645-c943-863bbfc9c3fe: removing all old logs from log store
2024-12-04T01:33:17.998Z [INFO]  server-f519eaba-b942-5645-c943-863bbfc9c3fe: compacting logs: from=1 to=22
2024-12-04T01:33:17.998Z [INFO]  server-f519eaba-b942-5645-c943-863bbfc9c3fe: Installed remote snapshot
2024-12-04T01:33:17.998Z [WARN]  server-f519eaba-b942-5645-c943-863bbfc9c3fe: skipping application of old log: index=22
2024-12-04T01:33:17.998Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: pipelining replication: peer="{Voter server-f519eaba-b942-5645-c943-863bbfc9c3fe f519eaba-b942-5645-c943-863bbfc9c3fe}"
2024-12-04T01:33:17.998Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:17.999Z [ERROR] server-18238386-148f-29d9-0d5a-1b4100d5f514: failed to get log: index=22 error="log not found"
2024-12-04T01:33:17.999Z [INFO]  creating new snapshot: path=/tmp/raft3814065374/snapshots/2-10013-1733275997999.tmp
2024-12-04T01:33:17.999Z [INFO]  server-1fe75b9b-9d96-8b5b-9299-7ed5b6193c70: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:17.999Z [INFO]  server-1fe75b9b-9d96-8b5b-9299-7ed5b6193c70: copied to local snapshot: bytes=71
2024-12-04T01:33:18.000Z [INFO]  server-1fe75b9b-9d96-8b5b-9299-7ed5b6193c70: snapshot restore progress: id=2-10013-1733275997999 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:18.000Z [INFO]  server-1fe75b9b-9d96-8b5b-9299-7ed5b6193c70: removing all old logs from log store
2024-12-04T01:33:18.000Z [INFO]  server-1fe75b9b-9d96-8b5b-9299-7ed5b6193c70: compacting logs: from=1 to=22
2024-12-04T01:33:18.000Z [INFO]  server-1fe75b9b-9d96-8b5b-9299-7ed5b6193c70: Installed remote snapshot
2024-12-04T01:33:18.000Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: pipelining replication: peer="{Voter server-1fe75b9b-9d96-8b5b-9299-7ed5b6193c70 1fe75b9b-9d96-8b5b-9299-7ed5b6193c70}"
2024-12-04T01:33:19.003Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:17.998796214 +0000 UTC m=+88.830638783 (2 nodes), 1.626µs from start of poll, 1.840883199s from cluster start. Timeout at 2024-12-04 01:33:19.003857185 +0000 UTC m=+89.835699881, 1.005061098s after stability
2024-12-04T01:33:19.015Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: aborting pipeline replication: peer="{Voter server-f519eaba-b942-5645-c943-863bbfc9c3fe f519eaba-b942-5645-c943-863bbfc9c3fe}"
2024-12-04T01:33:19.015Z [INFO]  server-18238386-148f-29d9-0d5a-1b4100d5f514: aborting pipeline replication: peer="{Voter server-1fe75b9b-9d96-8b5b-9299-7ed5b6193c70 1fe75b9b-9d96-8b5b-9299-7ed5b6193c70}"
=== RUN   TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true
2024-12-04T01:33:19.018Z [INFO]  server-7bbc8515-3d92-622f-c8ff-ab60a5cfebfe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bbc8515-3d92-622f-c8ff-ab60a5cfebfe Address:7bbc8515-3d92-622f-c8ff-ab60a5cfebfe} {Suffrage:Voter ID:server-9e545368-ecff-3e8f-0e2f-3ae2b1afd4c8 Address:9e545368-ecff-3e8f-0e2f-3ae2b1afd4c8} {Suffrage:Voter ID:server-cb96701f-432e-d4f3-dc9c-48aed9e23099 Address:cb96701f-432e-d4f3-dc9c-48aed9e23099}]"
2024-12-04T01:33:19.019Z [INFO]  server-9e545368-ecff-3e8f-0e2f-3ae2b1afd4c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bbc8515-3d92-622f-c8ff-ab60a5cfebfe Address:7bbc8515-3d92-622f-c8ff-ab60a5cfebfe} {Suffrage:Voter ID:server-9e545368-ecff-3e8f-0e2f-3ae2b1afd4c8 Address:9e545368-ecff-3e8f-0e2f-3ae2b1afd4c8} {Suffrage:Voter ID:server-cb96701f-432e-d4f3-dc9c-48aed9e23099 Address:cb96701f-432e-d4f3-dc9c-48aed9e23099}]"
2024-12-04T01:33:19.019Z [INFO]  server-cb96701f-432e-d4f3-dc9c-48aed9e23099: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7bbc8515-3d92-622f-c8ff-ab60a5cfebfe Address:7bbc8515-3d92-622f-c8ff-ab60a5cfebfe} {Suffrage:Voter ID:server-9e545368-ecff-3e8f-0e2f-3ae2b1afd4c8 Address:9e545368-ecff-3e8f-0e2f-3ae2b1afd4c8} {Suffrage:Voter ID:server-cb96701f-432e-d4f3-dc9c-48aed9e23099 Address:cb96701f-432e-d4f3-dc9c-48aed9e23099}]"
2024-12-04T01:33:19.019Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:19.024Z [INFO]  server-9e545368-ecff-3e8f-0e2f-3ae2b1afd4c8: entering follower state: follower="Node at 9e545368-ecff-3e8f-0e2f-3ae2b1afd4c8 [Follower]" leader-address= leader-id=
2024-12-04T01:33:19.024Z [INFO]  server-7bbc8515-3d92-622f-c8ff-ab60a5cfebfe: entering follower state: follower="Node at 7bbc8515-3d92-622f-c8ff-ab60a5cfebfe [Follower]" leader-address= leader-id=
2024-12-04T01:33:19.024Z [INFO]  server-cb96701f-432e-d4f3-dc9c-48aed9e23099: entering follower state: follower="Node at cb96701f-432e-d4f3-dc9c-48aed9e23099 [Follower]" leader-address= leader-id=
2024-12-04T01:33:19.787Z [WARN]  server-cb96701f-432e-d4f3-dc9c-48aed9e23099: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:19.787Z [INFO]  server-cb96701f-432e-d4f3-dc9c-48aed9e23099: entering candidate state: node="Node at cb96701f-432e-d4f3-dc9c-48aed9e23099 [Candidate]" term=2
2024-12-04T01:33:19.787Z [INFO]  server-cb96701f-432e-d4f3-dc9c-48aed9e23099: election won: term=2 tally=2
2024-12-04T01:33:19.787Z [INFO]  server-cb96701f-432e-d4f3-dc9c-48aed9e23099: entering leader state: leader="Node at cb96701f-432e-d4f3-dc9c-48aed9e23099 [Leader]"
2024-12-04T01:33:19.787Z [INFO]  server-cb96701f-432e-d4f3-dc9c-48aed9e23099: added peer, starting replication: peer=server-7bbc8515-3d92-622f-c8ff-ab60a5cfebfe
2024-12-04T01:33:19.787Z [INFO]  server-cb96701f-432e-d4f3-dc9c-48aed9e23099: added peer, starting replication: peer=server-9e545368-ecff-3e8f-0e2f-3ae2b1afd4c8
2024-12-04T01:33:19.787Z [INFO]  server-cb96701f-432e-d4f3-dc9c-48aed9e23099: pipelining replication: peer="{Voter server-7bbc8515-3d92-622f-c8ff-ab60a5cfebfe 7bbc8515-3d92-622f-c8ff-ab60a5cfebfe}"
2024-12-04T01:33:19.787Z [INFO]  server-cb96701f-432e-d4f3-dc9c-48aed9e23099: pipelining replication: peer="{Voter server-9e545368-ecff-3e8f-0e2f-3ae2b1afd4c8 9e545368-ecff-3e8f-0e2f-3ae2b1afd4c8}"
2024-12-04T01:33:20.792Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:19.78749411 +0000 UTC m=+90.619337148 (1 nodes), 767.760547ms from start of poll, 769.909892ms from cluster start. Timeout at 2024-12-04 01:33:20.792501056 +0000 UTC m=+91.624343813, 1.005006665s after stability
2024-12-04T01:33:20.793Z [INFO]  server-cb96701f-432e-d4f3-dc9c-48aed9e23099: starting snapshot up to: index=12
2024-12-04T01:33:20.793Z [INFO]  creating new snapshot: path=/tmp/raft2385143088/snapshots/2-12-1733276000793.tmp
2024-12-04T01:33:20.794Z [INFO]  server-cb96701f-432e-d4f3-dc9c-48aed9e23099: snapshot complete up to: index=12
2024-12-04T01:33:20.796Z [INFO]  server-607b5fc7-6df0-698b-8fd1-220e0f711566: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-607b5fc7-6df0-698b-8fd1-220e0f711566 Address:607b5fc7-6df0-698b-8fd1-220e0f711566} {Suffrage:Voter ID:server-fe6feb56-d386-14b7-29b8-8936cafed818 Address:fe6feb56-d386-14b7-29b8-8936cafed818} {Suffrage:Voter ID:server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0 Address:699eca48-d40b-0dcd-fdb2-3e9f798d7cf0}]"
2024-12-04T01:33:20.796Z [INFO]  server-fe6feb56-d386-14b7-29b8-8936cafed818: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-607b5fc7-6df0-698b-8fd1-220e0f711566 Address:607b5fc7-6df0-698b-8fd1-220e0f711566} {Suffrage:Voter ID:server-fe6feb56-d386-14b7-29b8-8936cafed818 Address:fe6feb56-d386-14b7-29b8-8936cafed818} {Suffrage:Voter ID:server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0 Address:699eca48-d40b-0dcd-fdb2-3e9f798d7cf0}]"
2024-12-04T01:33:20.797Z [INFO]  server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-607b5fc7-6df0-698b-8fd1-220e0f711566 Address:607b5fc7-6df0-698b-8fd1-220e0f711566} {Suffrage:Voter ID:server-fe6feb56-d386-14b7-29b8-8936cafed818 Address:fe6feb56-d386-14b7-29b8-8936cafed818} {Suffrage:Voter ID:server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0 Address:699eca48-d40b-0dcd-fdb2-3e9f798d7cf0}]"
2024-12-04T01:33:20.797Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:20.797Z [INFO]  server-607b5fc7-6df0-698b-8fd1-220e0f711566: entering follower state: follower="Node at 607b5fc7-6df0-698b-8fd1-220e0f711566 [Follower]" leader-address= leader-id=
2024-12-04T01:33:20.797Z [INFO]  server-fe6feb56-d386-14b7-29b8-8936cafed818: entering follower state: follower="Node at fe6feb56-d386-14b7-29b8-8936cafed818 [Follower]" leader-address= leader-id=
2024-12-04T01:33:20.797Z [INFO]  server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: entering follower state: follower="Node at 699eca48-d40b-0dcd-fdb2-3e9f798d7cf0 [Follower]" leader-address= leader-id=
2024-12-04T01:33:21.466Z [WARN]  server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:21.466Z [INFO]  server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: entering candidate state: node="Node at 699eca48-d40b-0dcd-fdb2-3e9f798d7cf0 [Candidate]" term=2
2024-12-04T01:33:21.466Z [INFO]  server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: election won: term=2 tally=2
2024-12-04T01:33:21.466Z [INFO]  server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: entering leader state: leader="Node at 699eca48-d40b-0dcd-fdb2-3e9f798d7cf0 [Leader]"
2024-12-04T01:33:21.466Z [INFO]  server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: added peer, starting replication: peer=server-607b5fc7-6df0-698b-8fd1-220e0f711566
2024-12-04T01:33:21.466Z [INFO]  server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: added peer, starting replication: peer=server-fe6feb56-d386-14b7-29b8-8936cafed818
2024-12-04T01:33:21.467Z [INFO]  server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: pipelining replication: peer="{Voter server-fe6feb56-d386-14b7-29b8-8936cafed818 fe6feb56-d386-14b7-29b8-8936cafed818}"
2024-12-04T01:33:21.467Z [INFO]  server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: pipelining replication: peer="{Voter server-607b5fc7-6df0-698b-8fd1-220e0f711566 607b5fc7-6df0-698b-8fd1-220e0f711566}"
2024-12-04T01:33:22.473Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:21.467482493 +0000 UTC m=+92.299325110 (1 nodes), 670.378842ms from start of poll, 670.710497ms from cluster start. Timeout at 2024-12-04 01:33:22.472488477 +0000 UTC m=+93.304331667, 1.005006557s after stability
2024-12-04T01:33:22.473Z [INFO]  creating new snapshot: path=/tmp/raft3912290263/snapshots/2-10013-1733276002473.tmp
2024-12-04T01:33:22.474Z [INFO]  server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: copied to local snapshot: bytes=71
2024-12-04T01:33:22.475Z [INFO]  server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: snapshot restore progress: id=2-10013-1733276002473 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:22.475Z [INFO]  server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: restored user snapshot: index=10013
2024-12-04T01:33:22.475Z [ERROR] server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: failed to get log: index=3 error="log not found"
2024-12-04T01:33:22.475Z [INFO]  server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: aborting pipeline replication: peer="{Voter server-fe6feb56-d386-14b7-29b8-8936cafed818 fe6feb56-d386-14b7-29b8-8936cafed818}"
2024-12-04T01:33:22.475Z [ERROR] server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: failed to get log: index=3 error="log not found"
2024-12-04T01:33:22.475Z [INFO]  server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: aborting pipeline replication: peer="{Voter server-607b5fc7-6df0-698b-8fd1-220e0f711566 607b5fc7-6df0-698b-8fd1-220e0f711566}"
2024-12-04T01:33:22.482Z [ERROR] server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: failed to get log: index=3 error="log not found"
2024-12-04T01:33:22.482Z [INFO]  creating new snapshot: path=/tmp/raft1452686420/snapshots/2-10013-1733276002482.tmp
2024-12-04T01:33:22.483Z [INFO]  server-fe6feb56-d386-14b7-29b8-8936cafed818: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:22.483Z [INFO]  server-fe6feb56-d386-14b7-29b8-8936cafed818: copied to local snapshot: bytes=71
2024-12-04T01:33:22.483Z [INFO]  server-fe6feb56-d386-14b7-29b8-8936cafed818: snapshot restore progress: id=2-10013-1733276002482 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:22.483Z [INFO]  server-fe6feb56-d386-14b7-29b8-8936cafed818: Installed remote snapshot
2024-12-04T01:33:22.483Z [INFO]  server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: pipelining replication: peer="{Voter server-fe6feb56-d386-14b7-29b8-8936cafed818 fe6feb56-d386-14b7-29b8-8936cafed818}"
2024-12-04T01:33:22.483Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:22.487Z [ERROR] server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: failed to get log: index=3 error="log not found"
2024-12-04T01:33:22.488Z [INFO]  creating new snapshot: path=/tmp/raft656230083/snapshots/2-10013-1733276002488.tmp
2024-12-04T01:33:22.488Z [INFO]  server-607b5fc7-6df0-698b-8fd1-220e0f711566: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:22.489Z [INFO]  server-607b5fc7-6df0-698b-8fd1-220e0f711566: copied to local snapshot: bytes=71
2024-12-04T01:33:22.489Z [INFO]  server-607b5fc7-6df0-698b-8fd1-220e0f711566: snapshot restore progress: id=2-10013-1733276002488 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:22.489Z [INFO]  server-607b5fc7-6df0-698b-8fd1-220e0f711566: Installed remote snapshot
2024-12-04T01:33:22.489Z [INFO]  server-699eca48-d40b-0dcd-fdb2-3e9f798d7cf0: pipelining replication: peer="{Voter server-607b5fc7-6df0-698b-8fd1-220e0f711566 607b5fc7-6df0-698b-8fd1-220e0f711566}"
2024-12-04T01:33:23.489Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:22.483974449 +0000 UTC m=+93.315817029 (2 nodes), 2.16µs from start of poll, 1.687202416s from cluster start. Timeout at 2024-12-04 01:33:23.48897936 +0000 UTC m=+94.320821962, 1.005004933s after stability
2024-12-04T01:33:23.502Z [INFO]  server-cb96701f-432e-d4f3-dc9c-48aed9e23099: aborting pipeline replication: peer="{Voter server-9e545368-ecff-3e8f-0e2f-3ae2b1afd4c8 9e545368-ecff-3e8f-0e2f-3ae2b1afd4c8}"
2024-12-04T01:33:23.502Z [INFO]  server-cb96701f-432e-d4f3-dc9c-48aed9e23099: aborting pipeline replication: peer="{Voter server-7bbc8515-3d92-622f-c8ff-ab60a5cfebfe 7bbc8515-3d92-622f-c8ff-ab60a5cfebfe}"
=== RUN   TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true
2024-12-04T01:33:23.503Z [INFO]  server-668c95bf-87c8-d6e9-7bc7-1f34a623fd5a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-668c95bf-87c8-d6e9-7bc7-1f34a623fd5a Address:668c95bf-87c8-d6e9-7bc7-1f34a623fd5a} {Suffrage:Voter ID:server-733a9e15-1373-1ff0-0683-182146dd193c Address:733a9e15-1373-1ff0-0683-182146dd193c} {Suffrage:Voter ID:server-68e63392-b9a0-c959-004c-ac8e6d1f72c3 Address:68e63392-b9a0-c959-004c-ac8e6d1f72c3}]"
2024-12-04T01:33:23.503Z [INFO]  server-733a9e15-1373-1ff0-0683-182146dd193c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-668c95bf-87c8-d6e9-7bc7-1f34a623fd5a Address:668c95bf-87c8-d6e9-7bc7-1f34a623fd5a} {Suffrage:Voter ID:server-733a9e15-1373-1ff0-0683-182146dd193c Address:733a9e15-1373-1ff0-0683-182146dd193c} {Suffrage:Voter ID:server-68e63392-b9a0-c959-004c-ac8e6d1f72c3 Address:68e63392-b9a0-c959-004c-ac8e6d1f72c3}]"
2024-12-04T01:33:23.504Z [INFO]  server-68e63392-b9a0-c959-004c-ac8e6d1f72c3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-668c95bf-87c8-d6e9-7bc7-1f34a623fd5a Address:668c95bf-87c8-d6e9-7bc7-1f34a623fd5a} {Suffrage:Voter ID:server-733a9e15-1373-1ff0-0683-182146dd193c Address:733a9e15-1373-1ff0-0683-182146dd193c} {Suffrage:Voter ID:server-68e63392-b9a0-c959-004c-ac8e6d1f72c3 Address:68e63392-b9a0-c959-004c-ac8e6d1f72c3}]"
2024-12-04T01:33:23.504Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:23.504Z [INFO]  server-668c95bf-87c8-d6e9-7bc7-1f34a623fd5a: entering follower state: follower="Node at 668c95bf-87c8-d6e9-7bc7-1f34a623fd5a [Follower]" leader-address= leader-id=
2024-12-04T01:33:23.504Z [INFO]  server-733a9e15-1373-1ff0-0683-182146dd193c: entering follower state: follower="Node at 733a9e15-1373-1ff0-0683-182146dd193c [Follower]" leader-address= leader-id=
2024-12-04T01:33:23.504Z [INFO]  server-68e63392-b9a0-c959-004c-ac8e6d1f72c3: entering follower state: follower="Node at 68e63392-b9a0-c959-004c-ac8e6d1f72c3 [Follower]" leader-address= leader-id=
2024-12-04T01:33:24.324Z [WARN]  server-668c95bf-87c8-d6e9-7bc7-1f34a623fd5a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:24.324Z [INFO]  server-668c95bf-87c8-d6e9-7bc7-1f34a623fd5a: entering candidate state: node="Node at 668c95bf-87c8-d6e9-7bc7-1f34a623fd5a [Candidate]" term=2
2024-12-04T01:33:24.324Z [INFO]  server-668c95bf-87c8-d6e9-7bc7-1f34a623fd5a: election won: term=2 tally=2
2024-12-04T01:33:24.324Z [INFO]  server-668c95bf-87c8-d6e9-7bc7-1f34a623fd5a: entering leader state: leader="Node at 668c95bf-87c8-d6e9-7bc7-1f34a623fd5a [Leader]"
2024-12-04T01:33:24.324Z [INFO]  server-668c95bf-87c8-d6e9-7bc7-1f34a623fd5a: added peer, starting replication: peer=server-733a9e15-1373-1ff0-0683-182146dd193c
2024-12-04T01:33:24.324Z [INFO]  server-668c95bf-87c8-d6e9-7bc7-1f34a623fd5a: added peer, starting replication: peer=server-68e63392-b9a0-c959-004c-ac8e6d1f72c3
2024-12-04T01:33:24.325Z [INFO]  server-668c95bf-87c8-d6e9-7bc7-1f34a623fd5a: pipelining replication: peer="{Voter server-68e63392-b9a0-c959-004c-ac8e6d1f72c3 68e63392-b9a0-c959-004c-ac8e6d1f72c3}"
2024-12-04T01:33:24.325Z [INFO]  server-668c95bf-87c8-d6e9-7bc7-1f34a623fd5a: pipelining replication: peer="{Voter server-733a9e15-1373-1ff0-0683-182146dd193c 733a9e15-1373-1ff0-0683-182146dd193c}"
2024-12-04T01:33:25.330Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:24.325401445 +0000 UTC m=+95.157244119 (1 nodes), 821.27052ms from start of poll, 821.68259ms from cluster start. Timeout at 2024-12-04 01:33:25.330417888 +0000 UTC m=+96.162260717, 1.005016598s after stability
2024-12-04T01:33:25.332Z [INFO]  server-668c95bf-87c8-d6e9-7bc7-1f34a623fd5a: starting snapshot up to: index=12
2024-12-04T01:33:25.332Z [INFO]  creating new snapshot: path=/tmp/raft399871215/snapshots/2-12-1733276005332.tmp
2024-12-04T01:33:25.333Z [INFO]  server-668c95bf-87c8-d6e9-7bc7-1f34a623fd5a: snapshot complete up to: index=12
2024-12-04T01:33:25.336Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9fd3f96a-8b08-db9f-7b6f-135594270b69 Address:9fd3f96a-8b08-db9f-7b6f-135594270b69} {Suffrage:Voter ID:server-8aee3f45-f8b9-839a-a526-27b04711f68f Address:8aee3f45-f8b9-839a-a526-27b04711f68f} {Suffrage:Voter ID:server-8fcde477-05b0-a4a4-33ec-fa74199febf9 Address:8fcde477-05b0-a4a4-33ec-fa74199febf9}]"
2024-12-04T01:33:25.336Z [INFO]  server-8aee3f45-f8b9-839a-a526-27b04711f68f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9fd3f96a-8b08-db9f-7b6f-135594270b69 Address:9fd3f96a-8b08-db9f-7b6f-135594270b69} {Suffrage:Voter ID:server-8aee3f45-f8b9-839a-a526-27b04711f68f Address:8aee3f45-f8b9-839a-a526-27b04711f68f} {Suffrage:Voter ID:server-8fcde477-05b0-a4a4-33ec-fa74199febf9 Address:8fcde477-05b0-a4a4-33ec-fa74199febf9}]"
2024-12-04T01:33:25.336Z [INFO]  server-8fcde477-05b0-a4a4-33ec-fa74199febf9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9fd3f96a-8b08-db9f-7b6f-135594270b69 Address:9fd3f96a-8b08-db9f-7b6f-135594270b69} {Suffrage:Voter ID:server-8aee3f45-f8b9-839a-a526-27b04711f68f Address:8aee3f45-f8b9-839a-a526-27b04711f68f} {Suffrage:Voter ID:server-8fcde477-05b0-a4a4-33ec-fa74199febf9 Address:8fcde477-05b0-a4a4-33ec-fa74199febf9}]"
2024-12-04T01:33:25.336Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:25.336Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: entering follower state: follower="Node at 9fd3f96a-8b08-db9f-7b6f-135594270b69 [Follower]" leader-address= leader-id=
2024-12-04T01:33:25.336Z [INFO]  server-8aee3f45-f8b9-839a-a526-27b04711f68f: entering follower state: follower="Node at 8aee3f45-f8b9-839a-a526-27b04711f68f [Follower]" leader-address= leader-id=
2024-12-04T01:33:25.336Z [INFO]  server-8fcde477-05b0-a4a4-33ec-fa74199febf9: entering follower state: follower="Node at 8fcde477-05b0-a4a4-33ec-fa74199febf9 [Follower]" leader-address= leader-id=
2024-12-04T01:33:25.882Z [WARN]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:25.882Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: entering candidate state: node="Node at 9fd3f96a-8b08-db9f-7b6f-135594270b69 [Candidate]" term=2
2024-12-04T01:33:25.882Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: election won: term=2 tally=2
2024-12-04T01:33:25.882Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: entering leader state: leader="Node at 9fd3f96a-8b08-db9f-7b6f-135594270b69 [Leader]"
2024-12-04T01:33:25.882Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: added peer, starting replication: peer=server-8aee3f45-f8b9-839a-a526-27b04711f68f
2024-12-04T01:33:25.882Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: added peer, starting replication: peer=server-8fcde477-05b0-a4a4-33ec-fa74199febf9
2024-12-04T01:33:25.882Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: pipelining replication: peer="{Voter server-8aee3f45-f8b9-839a-a526-27b04711f68f 8aee3f45-f8b9-839a-a526-27b04711f68f}"
2024-12-04T01:33:25.882Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: pipelining replication: peer="{Voter server-8fcde477-05b0-a4a4-33ec-fa74199febf9 8fcde477-05b0-a4a4-33ec-fa74199febf9}"
2024-12-04T01:33:26.888Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:25.882521799 +0000 UTC m=+96.714364483 (1 nodes), 546.064981ms from start of poll, 546.530765ms from cluster start. Timeout at 2024-12-04 01:33:26.887528312 +0000 UTC m=+97.719370897, 1.005006414s after stability
2024-12-04T01:33:26.889Z [INFO]  creating new snapshot: path=/tmp/raft397424570/snapshots/2-10013-1733276006889.tmp
2024-12-04T01:33:26.889Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: copied to local snapshot: bytes=71
2024-12-04T01:33:26.890Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: snapshot restore progress: id=2-10013-1733276006889 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:26.890Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: removing all old logs from log store
2024-12-04T01:33:26.890Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: compacting logs: from=1 to=2
2024-12-04T01:33:26.890Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: restored user snapshot: index=10013
2024-12-04T01:33:26.890Z [ERROR] server-9fd3f96a-8b08-db9f-7b6f-135594270b69: failed to get log: index=2 error="log not found"
2024-12-04T01:33:26.890Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: aborting pipeline replication: peer="{Voter server-8fcde477-05b0-a4a4-33ec-fa74199febf9 8fcde477-05b0-a4a4-33ec-fa74199febf9}"
2024-12-04T01:33:26.890Z [ERROR] server-9fd3f96a-8b08-db9f-7b6f-135594270b69: failed to get log: index=2 error="log not found"
2024-12-04T01:33:26.890Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: aborting pipeline replication: peer="{Voter server-8aee3f45-f8b9-839a-a526-27b04711f68f 8aee3f45-f8b9-839a-a526-27b04711f68f}"
2024-12-04T01:33:26.896Z [ERROR] server-9fd3f96a-8b08-db9f-7b6f-135594270b69: failed to get log: index=2 error="log not found"
2024-12-04T01:33:26.896Z [INFO]  creating new snapshot: path=/tmp/raft2478638120/snapshots/2-10013-1733276006896.tmp
2024-12-04T01:33:26.896Z [INFO]  server-8fcde477-05b0-a4a4-33ec-fa74199febf9: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:26.897Z [INFO]  server-8fcde477-05b0-a4a4-33ec-fa74199febf9: copied to local snapshot: bytes=71
2024-12-04T01:33:26.897Z [ERROR] server-9fd3f96a-8b08-db9f-7b6f-135594270b69: failed to get log: index=2 error="log not found"
2024-12-04T01:33:26.897Z [INFO]  creating new snapshot: path=/tmp/raft1177553450/snapshots/2-10013-1733276006897.tmp
2024-12-04T01:33:26.897Z [INFO]  server-8fcde477-05b0-a4a4-33ec-fa74199febf9: snapshot restore progress: id=2-10013-1733276006896 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:26.897Z [INFO]  server-8fcde477-05b0-a4a4-33ec-fa74199febf9: removing all old logs from log store
2024-12-04T01:33:26.897Z [INFO]  server-8fcde477-05b0-a4a4-33ec-fa74199febf9: compacting logs: from=1 to=2
2024-12-04T01:33:26.897Z [INFO]  server-8fcde477-05b0-a4a4-33ec-fa74199febf9: Installed remote snapshot
2024-12-04T01:33:26.898Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: pipelining replication: peer="{Voter server-8fcde477-05b0-a4a4-33ec-fa74199febf9 8fcde477-05b0-a4a4-33ec-fa74199febf9}"
2024-12-04T01:33:26.897Z [INFO]  server-8aee3f45-f8b9-839a-a526-27b04711f68f: snapshot network transfer progress: read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:26.898Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:26.898Z [INFO]  server-8aee3f45-f8b9-839a-a526-27b04711f68f: copied to local snapshot: bytes=71
2024-12-04T01:33:26.898Z [INFO]  server-8aee3f45-f8b9-839a-a526-27b04711f68f: snapshot restore progress: id=2-10013-1733276006897 last-index=10013 last-term=2 size-in-bytes=71 read-bytes=71 percent-complete=100.00%
2024-12-04T01:33:26.898Z [INFO]  server-8aee3f45-f8b9-839a-a526-27b04711f68f: removing all old logs from log store
2024-12-04T01:33:26.898Z [INFO]  server-8aee3f45-f8b9-839a-a526-27b04711f68f: compacting logs: from=1 to=2
2024-12-04T01:33:26.898Z [INFO]  server-8aee3f45-f8b9-839a-a526-27b04711f68f: Installed remote snapshot
2024-12-04T01:33:26.899Z [INFO]  server-9fd3f96a-8b08-db9f-7b6f-135594270b69: pipelining replication: peer="{Voter server-8aee3f45-f8b9-839a-a526-27b04711f68f 8aee3f45-f8b9-839a-a526-27b04711f68f}"
2024-12-04T01:33:27.903Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:26.898270722 +0000 UTC m=+97.730113297 (2 nodes), 2.015µs from start of poll, 1.562279579s from cluster start. Timeout at 2024-12-04 01:33:27.903344882 +0000 UTC m=+98.735187472, 1.005074175s after stability
2024-12-04T01:33:27.915Z [INFO]  server-668c95bf-87c8-d6e9-7bc7-1f34a623fd5a: aborting pipeline replication: peer="{Voter server-68e63392-b9a0-c959-004c-ac8e6d1f72c3 68e63392-b9a0-c959-004c-ac8e6d1f72c3}"
2024-12-04T01:33:27.915Z [INFO]  server-668c95bf-87c8-d6e9-7bc7-1f34a623fd5a: aborting pipeline replication: peer="{Voter server-733a9e15-1373-1ff0-0683-182146dd193c 733a9e15-1373-1ff0-0683-182146dd193c}"
--- PASS: TestRaft_UserRestore (84.27s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_false (2.62s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_false (2.62s)
    --- PASS: TestRaft_UserRestore/case_0_|_restored_to_new_cluster:_true (4.33s)
    --- PASS: TestRaft_UserRestore/monotonic_case_0_|_restored_to_new_cluster:_true (4.17s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_false (2.58s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_false (2.82s)
    --- PASS: TestRaft_UserRestore/case_1_|_restored_to_new_cluster:_true (4.34s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1_|_restored_to_new_cluster:_true (4.19s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_false (2.81s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_false (2.86s)
    --- PASS: TestRaft_UserRestore/case_2_|_restored_to_new_cluster:_true (4.43s)
    --- PASS: TestRaft_UserRestore/monotonic_case_2_|_restored_to_new_cluster:_true (4.18s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_false (2.63s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_false (2.71s)
    --- PASS: TestRaft_UserRestore/case_100_|_restored_to_new_cluster:_true (4.19s)
    --- PASS: TestRaft_UserRestore/monotonic_case_100_|_restored_to_new_cluster:_true (4.31s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_false (2.61s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_false (2.72s)
    --- PASS: TestRaft_UserRestore/case_1000_|_restored_to_new_cluster:_true (4.49s)
    --- PASS: TestRaft_UserRestore/monotonic_case_1000_|_restored_to_new_cluster:_true (4.18s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_false (2.73s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_false (2.86s)
    --- PASS: TestRaft_UserRestore/case_10000_|_restored_to_new_cluster:_true (4.49s)
    --- PASS: TestRaft_UserRestore/monotonic_case_10000_|_restored_to_new_cluster:_true (4.41s)
=== RUN   TestRaft_SendSnapshotFollower
2024-12-04T01:33:27.917Z [INFO]  server-4fa59701-b8a8-5338-529e-d96d739464f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fa59701-b8a8-5338-529e-d96d739464f2 Address:4fa59701-b8a8-5338-529e-d96d739464f2} {Suffrage:Voter ID:server-84e2e66c-907e-160c-e013-d0801ca4a2fc Address:84e2e66c-907e-160c-e013-d0801ca4a2fc} {Suffrage:Voter ID:server-d16e76af-284d-6843-c544-bb012f49d6a8 Address:d16e76af-284d-6843-c544-bb012f49d6a8}]"
2024-12-04T01:33:27.917Z [INFO]  server-84e2e66c-907e-160c-e013-d0801ca4a2fc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fa59701-b8a8-5338-529e-d96d739464f2 Address:4fa59701-b8a8-5338-529e-d96d739464f2} {Suffrage:Voter ID:server-84e2e66c-907e-160c-e013-d0801ca4a2fc Address:84e2e66c-907e-160c-e013-d0801ca4a2fc} {Suffrage:Voter ID:server-d16e76af-284d-6843-c544-bb012f49d6a8 Address:d16e76af-284d-6843-c544-bb012f49d6a8}]"
2024-12-04T01:33:27.917Z [INFO]  server-d16e76af-284d-6843-c544-bb012f49d6a8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4fa59701-b8a8-5338-529e-d96d739464f2 Address:4fa59701-b8a8-5338-529e-d96d739464f2} {Suffrage:Voter ID:server-84e2e66c-907e-160c-e013-d0801ca4a2fc Address:84e2e66c-907e-160c-e013-d0801ca4a2fc} {Suffrage:Voter ID:server-d16e76af-284d-6843-c544-bb012f49d6a8 Address:d16e76af-284d-6843-c544-bb012f49d6a8}]"
2024-12-04T01:33:27.917Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:27.917Z [INFO]  server-4fa59701-b8a8-5338-529e-d96d739464f2: entering follower state: follower="Node at 4fa59701-b8a8-5338-529e-d96d739464f2 [Follower]" leader-address= leader-id=
2024-12-04T01:33:27.917Z [INFO]  server-84e2e66c-907e-160c-e013-d0801ca4a2fc: entering follower state: follower="Node at 84e2e66c-907e-160c-e013-d0801ca4a2fc [Follower]" leader-address= leader-id=
2024-12-04T01:33:27.917Z [INFO]  server-d16e76af-284d-6843-c544-bb012f49d6a8: entering follower state: follower="Node at d16e76af-284d-6843-c544-bb012f49d6a8 [Follower]" leader-address= leader-id=
2024-12-04T01:33:27.975Z [WARN]  server-84e2e66c-907e-160c-e013-d0801ca4a2fc: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:27.975Z [INFO]  server-84e2e66c-907e-160c-e013-d0801ca4a2fc: entering candidate state: node="Node at 84e2e66c-907e-160c-e013-d0801ca4a2fc [Candidate]" term=2
2024-12-04T01:33:27.976Z [INFO]  server-84e2e66c-907e-160c-e013-d0801ca4a2fc: election won: term=2 tally=2
2024-12-04T01:33:27.976Z [INFO]  server-84e2e66c-907e-160c-e013-d0801ca4a2fc: entering leader state: leader="Node at 84e2e66c-907e-160c-e013-d0801ca4a2fc [Leader]"
2024-12-04T01:33:27.976Z [INFO]  server-84e2e66c-907e-160c-e013-d0801ca4a2fc: added peer, starting replication: peer=server-4fa59701-b8a8-5338-529e-d96d739464f2
2024-12-04T01:33:27.976Z [INFO]  server-84e2e66c-907e-160c-e013-d0801ca4a2fc: added peer, starting replication: peer=server-d16e76af-284d-6843-c544-bb012f49d6a8
2024-12-04T01:33:27.976Z [INFO]  server-84e2e66c-907e-160c-e013-d0801ca4a2fc: pipelining replication: peer="{Voter server-d16e76af-284d-6843-c544-bb012f49d6a8 d16e76af-284d-6843-c544-bb012f49d6a8}"
2024-12-04T01:33:27.976Z [INFO]  server-84e2e66c-907e-160c-e013-d0801ca4a2fc: pipelining replication: peer="{Voter server-4fa59701-b8a8-5338-529e-d96d739464f2 4fa59701-b8a8-5338-529e-d96d739464f2}"
2024-12-04T01:33:28.081Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:27.976494249 +0000 UTC m=+98.808336841 (2 nodes), 58.865514ms from start of poll, 59.255226ms from cluster start. Timeout at 2024-12-04 01:33:28.081506938 +0000 UTC m=+98.913349526, 105.012685ms after stability
2024-12-04T01:33:28.081Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:28.186Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:28.081621523 +0000 UTC m=+98.913464089 (1 nodes), 1.869µs from start of poll, 164.382474ms from cluster start. Timeout at 2024-12-04 01:33:28.186681895 +0000 UTC m=+99.018524662, 105.060573ms after stability
2024-12-04T01:33:28.186Z [ERROR] server-84e2e66c-907e-160c-e013-d0801ca4a2fc: failed to heartbeat to: peer=4fa59701-b8a8-5338-529e-d96d739464f2 backoff time=10ms error="failed to connect to peer: 4fa59701-b8a8-5338-529e-d96d739464f2"
2024-12-04T01:33:28.186Z [ERROR] server-84e2e66c-907e-160c-e013-d0801ca4a2fc: failed to pipeline appendEntries: peer="{Voter server-4fa59701-b8a8-5338-529e-d96d739464f2 4fa59701-b8a8-5338-529e-d96d739464f2}" error="append pipeline closed"
2024-12-04T01:33:28.187Z [INFO]  server-84e2e66c-907e-160c-e013-d0801ca4a2fc: aborting pipeline replication: peer="{Voter server-4fa59701-b8a8-5338-529e-d96d739464f2 4fa59701-b8a8-5338-529e-d96d739464f2}"
2024-12-04T01:33:28.187Z [ERROR] server-84e2e66c-907e-160c-e013-d0801ca4a2fc: failed to appendEntries to: peer="{Voter server-4fa59701-b8a8-5338-529e-d96d739464f2 4fa59701-b8a8-5338-529e-d96d739464f2}" error="failed to connect to peer: 4fa59701-b8a8-5338-529e-d96d739464f2"
    raft_test.go:1627: [INFO] Finished apply without behind follower
2024-12-04T01:33:28.195Z [ERROR] server-4fa59701-b8a8-5338-529e-d96d739464f2: failed to take snapshot: error="nothing new to snapshot"
2024-12-04T01:33:28.195Z [INFO]  server-84e2e66c-907e-160c-e013-d0801ca4a2fc: starting snapshot up to: index=102
2024-12-04T01:33:28.195Z [INFO]  creating new snapshot: path=/tmp/raft1422377437/snapshots/2-102-1733276008195.tmp
2024-12-04T01:33:28.195Z [INFO]  server-84e2e66c-907e-160c-e013-d0801ca4a2fc: compacting logs: from=1 to=92
2024-12-04T01:33:28.195Z [INFO]  server-84e2e66c-907e-160c-e013-d0801ca4a2fc: snapshot complete up to: index=102
2024-12-04T01:33:28.195Z [INFO]  server-d16e76af-284d-6843-c544-bb012f49d6a8: starting snapshot up to: index=99
2024-12-04T01:33:28.195Z [INFO]  creating new snapshot: path=/tmp/raft3457020468/snapshots/2-99-1733276008195.tmp
2024-12-04T01:33:28.196Z [INFO]  server-d16e76af-284d-6843-c544-bb012f49d6a8: compacting logs: from=1 to=92
2024-12-04T01:33:28.196Z [INFO]  server-d16e76af-284d-6843-c544-bb012f49d6a8: snapshot complete up to: index=99
2024-12-04T01:33:28.197Z [ERROR] server-84e2e66c-907e-160c-e013-d0801ca4a2fc: failed to get log: index=2 error="log not found"
2024-12-04T01:33:28.198Z [INFO]  creating new snapshot: path=/tmp/raft1856741363/snapshots/2-102-1733276008198.tmp
2024-12-04T01:33:28.198Z [INFO]  server-4fa59701-b8a8-5338-529e-d96d739464f2: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2024-12-04T01:33:28.198Z [INFO]  server-4fa59701-b8a8-5338-529e-d96d739464f2: copied to local snapshot: bytes=693
2024-12-04T01:33:28.199Z [INFO]  server-4fa59701-b8a8-5338-529e-d96d739464f2: snapshot restore progress: id=2-102-1733276008198 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2024-12-04T01:33:28.199Z [INFO]  server-4fa59701-b8a8-5338-529e-d96d739464f2: Installed remote snapshot
2024-12-04T01:33:28.199Z [WARN]  server-4fa59701-b8a8-5338-529e-d96d739464f2: skipping application of old log: index=102
2024-12-04T01:33:28.199Z [INFO]  server-84e2e66c-907e-160c-e013-d0801ca4a2fc: pipelining replication: peer="{Voter server-4fa59701-b8a8-5338-529e-d96d739464f2 4fa59701-b8a8-5338-529e-d96d739464f2}"
2024-12-04T01:33:28.204Z [INFO]  server-84e2e66c-907e-160c-e013-d0801ca4a2fc: aborting pipeline replication: peer="{Voter server-d16e76af-284d-6843-c544-bb012f49d6a8 d16e76af-284d-6843-c544-bb012f49d6a8}"
2024-12-04T01:33:28.204Z [INFO]  server-84e2e66c-907e-160c-e013-d0801ca4a2fc: aborting pipeline replication: peer="{Voter server-4fa59701-b8a8-5338-529e-d96d739464f2 4fa59701-b8a8-5338-529e-d96d739464f2}"
--- PASS: TestRaft_SendSnapshotFollower (0.29s)
=== RUN   TestRaft_SendSnapshotAndLogsFollower
2024-12-04T01:33:28.206Z [INFO]  server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e Address:0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e} {Suffrage:Voter ID:server-12173a0e-dfae-24b2-6270-71bafd89c88e Address:12173a0e-dfae-24b2-6270-71bafd89c88e} {Suffrage:Voter ID:server-ae382408-df1f-7969-53f7-d87ec60ede77 Address:ae382408-df1f-7969-53f7-d87ec60ede77}]"
2024-12-04T01:33:28.206Z [INFO]  server-12173a0e-dfae-24b2-6270-71bafd89c88e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e Address:0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e} {Suffrage:Voter ID:server-12173a0e-dfae-24b2-6270-71bafd89c88e Address:12173a0e-dfae-24b2-6270-71bafd89c88e} {Suffrage:Voter ID:server-ae382408-df1f-7969-53f7-d87ec60ede77 Address:ae382408-df1f-7969-53f7-d87ec60ede77}]"
2024-12-04T01:33:28.206Z [INFO]  server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e: entering follower state: follower="Node at 0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e [Follower]" leader-address= leader-id=
2024-12-04T01:33:28.206Z [INFO]  server-12173a0e-dfae-24b2-6270-71bafd89c88e: entering follower state: follower="Node at 12173a0e-dfae-24b2-6270-71bafd89c88e [Follower]" leader-address= leader-id=
2024-12-04T01:33:28.206Z [INFO]  server-ae382408-df1f-7969-53f7-d87ec60ede77: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e Address:0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e} {Suffrage:Voter ID:server-12173a0e-dfae-24b2-6270-71bafd89c88e Address:12173a0e-dfae-24b2-6270-71bafd89c88e} {Suffrage:Voter ID:server-ae382408-df1f-7969-53f7-d87ec60ede77 Address:ae382408-df1f-7969-53f7-d87ec60ede77}]"
2024-12-04T01:33:28.206Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:28.206Z [INFO]  server-ae382408-df1f-7969-53f7-d87ec60ede77: entering follower state: follower="Node at ae382408-df1f-7969-53f7-d87ec60ede77 [Follower]" leader-address= leader-id=
2024-12-04T01:33:28.257Z [WARN]  server-ae382408-df1f-7969-53f7-d87ec60ede77: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:28.257Z [INFO]  server-ae382408-df1f-7969-53f7-d87ec60ede77: entering candidate state: node="Node at ae382408-df1f-7969-53f7-d87ec60ede77 [Candidate]" term=2
2024-12-04T01:33:28.257Z [INFO]  server-ae382408-df1f-7969-53f7-d87ec60ede77: election won: term=2 tally=2
2024-12-04T01:33:28.257Z [INFO]  server-ae382408-df1f-7969-53f7-d87ec60ede77: entering leader state: leader="Node at ae382408-df1f-7969-53f7-d87ec60ede77 [Leader]"
2024-12-04T01:33:28.257Z [INFO]  server-ae382408-df1f-7969-53f7-d87ec60ede77: added peer, starting replication: peer=server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e
2024-12-04T01:33:28.257Z [INFO]  server-ae382408-df1f-7969-53f7-d87ec60ede77: added peer, starting replication: peer=server-12173a0e-dfae-24b2-6270-71bafd89c88e
2024-12-04T01:33:28.257Z [INFO]  server-ae382408-df1f-7969-53f7-d87ec60ede77: pipelining replication: peer="{Voter server-12173a0e-dfae-24b2-6270-71bafd89c88e 12173a0e-dfae-24b2-6270-71bafd89c88e}"
2024-12-04T01:33:28.258Z [INFO]  server-ae382408-df1f-7969-53f7-d87ec60ede77: pipelining replication: peer="{Voter server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e 0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e}"
2024-12-04T01:33:28.363Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:28.258448122 +0000 UTC m=+99.090290696 (2 nodes), 51.715924ms from start of poll, 52.088421ms from cluster start. Timeout at 2024-12-04 01:33:28.36345571 +0000 UTC m=+99.195298267, 105.007571ms after stability
2024-12-04T01:33:28.363Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:28.469Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:28.363586179 +0000 UTC m=+99.195428751 (1 nodes), 1.763µs from start of poll, 157.226476ms from cluster start. Timeout at 2024-12-04 01:33:28.468655102 +0000 UTC m=+99.300497717, 105.068966ms after stability
2024-12-04T01:33:28.469Z [ERROR] server-ae382408-df1f-7969-53f7-d87ec60ede77: failed to pipeline appendEntries: peer="{Voter server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e 0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e}" error="append pipeline closed"
2024-12-04T01:33:28.469Z [INFO]  server-ae382408-df1f-7969-53f7-d87ec60ede77: aborting pipeline replication: peer="{Voter server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e 0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e}"
2024-12-04T01:33:28.469Z [ERROR] server-ae382408-df1f-7969-53f7-d87ec60ede77: failed to appendEntries to: peer="{Voter server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e 0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e}" error="failed to connect to peer: 0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e"
2024-12-04T01:33:28.479Z [ERROR] server-ae382408-df1f-7969-53f7-d87ec60ede77: failed to heartbeat to: peer=0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e backoff time=10ms error="failed to connect to peer: 0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e"
2024-12-04T01:33:28.480Z [ERROR] server-ae382408-df1f-7969-53f7-d87ec60ede77: failed to appendEntries to: peer="{Voter server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e 0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e}" error="failed to connect to peer: 0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e"
    raft_test.go:1669: [INFO] Finished apply without behind follower
2024-12-04T01:33:28.487Z [ERROR] server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e: failed to take snapshot: error="nothing new to snapshot"
2024-12-04T01:33:28.488Z [INFO]  server-12173a0e-dfae-24b2-6270-71bafd89c88e: starting snapshot up to: index=93
2024-12-04T01:33:28.488Z [INFO]  creating new snapshot: path=/tmp/raft3173530078/snapshots/2-93-1733276008488.tmp
2024-12-04T01:33:28.488Z [INFO]  server-12173a0e-dfae-24b2-6270-71bafd89c88e: compacting logs: from=1 to=92
2024-12-04T01:33:28.488Z [INFO]  server-12173a0e-dfae-24b2-6270-71bafd89c88e: snapshot complete up to: index=93
2024-12-04T01:33:28.488Z [INFO]  server-ae382408-df1f-7969-53f7-d87ec60ede77: starting snapshot up to: index=102
2024-12-04T01:33:28.488Z [INFO]  creating new snapshot: path=/tmp/raft3722848832/snapshots/2-102-1733276008488.tmp
2024-12-04T01:33:28.489Z [INFO]  server-ae382408-df1f-7969-53f7-d87ec60ede77: compacting logs: from=1 to=92
2024-12-04T01:33:28.489Z [INFO]  server-ae382408-df1f-7969-53f7-d87ec60ede77: snapshot complete up to: index=102
2024-12-04T01:33:28.490Z [ERROR] server-ae382408-df1f-7969-53f7-d87ec60ede77: failed to get log: index=2 error="log not found"
2024-12-04T01:33:28.491Z [ERROR] server-ae382408-df1f-7969-53f7-d87ec60ede77: failed to install snapshot: id=2-102-1733276008488 error="failed to connect to peer: 0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e"
2024-12-04T01:33:28.491Z [ERROR] server-ae382408-df1f-7969-53f7-d87ec60ede77: failed to send snapshot to: peer="{Voter server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e 0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e}" error="failed to connect to peer: 0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e"
    raft_test.go:1690: [INFO] Finished apply without behind follower
2024-12-04T01:33:28.511Z [ERROR] server-ae382408-df1f-7969-53f7-d87ec60ede77: failed to get log: index=2 error="log not found"
2024-12-04T01:33:28.512Z [INFO]  creating new snapshot: path=/tmp/raft2086640809/snapshots/2-102-1733276008512.tmp
2024-12-04T01:33:28.512Z [INFO]  server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e: snapshot network transfer progress: read-bytes=693 percent-complete=100.00%
2024-12-04T01:33:28.513Z [INFO]  server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e: copied to local snapshot: bytes=693
2024-12-04T01:33:28.513Z [INFO]  server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e: snapshot restore progress: id=2-102-1733276008512 last-index=102 last-term=2 size-in-bytes=693 read-bytes=693 percent-complete=100.00%
2024-12-04T01:33:28.513Z [INFO]  server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e: Installed remote snapshot
2024-12-04T01:33:28.513Z [INFO]  server-ae382408-df1f-7969-53f7-d87ec60ede77: pipelining replication: peer="{Voter server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e 0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e}"
2024-12-04T01:33:28.524Z [INFO]  server-ae382408-df1f-7969-53f7-d87ec60ede77: aborting pipeline replication: peer="{Voter server-12173a0e-dfae-24b2-6270-71bafd89c88e 12173a0e-dfae-24b2-6270-71bafd89c88e}"
2024-12-04T01:33:28.525Z [INFO]  server-ae382408-df1f-7969-53f7-d87ec60ede77: aborting pipeline replication: peer="{Voter server-0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e 0a6ffcfc-7d93-ec68-dbf1-d8a44d68759e}"
--- PASS: TestRaft_SendSnapshotAndLogsFollower (0.32s)
=== RUN   TestRaft_ReJoinFollower
2024-12-04T01:33:28.526Z [INFO]  server-998dd173-be3b-85ba-1601-8b9281a267de: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-998dd173-be3b-85ba-1601-8b9281a267de Address:998dd173-be3b-85ba-1601-8b9281a267de} {Suffrage:Voter ID:server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a Address:bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a} {Suffrage:Voter ID:server-4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372 Address:4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372}]"
2024-12-04T01:33:28.526Z [INFO]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-998dd173-be3b-85ba-1601-8b9281a267de Address:998dd173-be3b-85ba-1601-8b9281a267de} {Suffrage:Voter ID:server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a Address:bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a} {Suffrage:Voter ID:server-4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372 Address:4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372}]"
2024-12-04T01:33:28.526Z [INFO]  server-4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-998dd173-be3b-85ba-1601-8b9281a267de Address:998dd173-be3b-85ba-1601-8b9281a267de} {Suffrage:Voter ID:server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a Address:bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a} {Suffrage:Voter ID:server-4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372 Address:4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372}]"
2024-12-04T01:33:28.526Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:28.526Z [INFO]  server-4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372: entering follower state: follower="Node at 4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372 [Follower]" leader-address= leader-id=
2024-12-04T01:33:28.527Z [INFO]  server-998dd173-be3b-85ba-1601-8b9281a267de: entering follower state: follower="Node at 998dd173-be3b-85ba-1601-8b9281a267de [Follower]" leader-address= leader-id=
2024-12-04T01:33:28.527Z [INFO]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: entering follower state: follower="Node at bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a [Follower]" leader-address= leader-id=
2024-12-04T01:33:28.585Z [WARN]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:28.585Z [INFO]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: entering candidate state: node="Node at bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a [Candidate]" term=2
2024-12-04T01:33:28.585Z [INFO]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: election won: term=2 tally=2
2024-12-04T01:33:28.585Z [INFO]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: entering leader state: leader="Node at bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a [Leader]"
2024-12-04T01:33:28.586Z [INFO]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: added peer, starting replication: peer=server-998dd173-be3b-85ba-1601-8b9281a267de
2024-12-04T01:33:28.586Z [INFO]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: added peer, starting replication: peer=server-4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372
2024-12-04T01:33:28.586Z [INFO]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: pipelining replication: peer="{Voter server-4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372 4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372}"
2024-12-04T01:33:28.586Z [INFO]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: pipelining replication: peer="{Voter server-998dd173-be3b-85ba-1601-8b9281a267de 998dd173-be3b-85ba-1601-8b9281a267de}"
2024-12-04T01:33:28.691Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:28.586181935 +0000 UTC m=+99.418024518 (1 nodes), 59.262952ms from start of poll, 59.601947ms from cluster start. Timeout at 2024-12-04 01:33:28.691204956 +0000 UTC m=+99.523047575, 105.023057ms after stability
2024-12-04T01:33:28.697Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:28.803Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:28.697365837 +0000 UTC m=+99.529208613 (2 nodes), 2.575µs from start of poll, 170.786042ms from cluster start. Timeout at 2024-12-04 01:33:28.802441381 +0000 UTC m=+99.634284147, 105.075534ms after stability
2024-12-04T01:33:28.803Z [INFO]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: updating configuration: command=RemoveServer server-id=server-998dd173-be3b-85ba-1601-8b9281a267de server-addr= servers="[{Suffrage:Voter ID:server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a Address:bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a} {Suffrage:Voter ID:server-4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372 Address:4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372}]"
2024-12-04T01:33:28.803Z [INFO]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: removed peer, stopping replication: peer=server-998dd173-be3b-85ba-1601-8b9281a267de last-index=3
2024-12-04T01:33:28.803Z [INFO]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: aborting pipeline replication: peer="{Voter server-998dd173-be3b-85ba-1601-8b9281a267de 998dd173-be3b-85ba-1601-8b9281a267de}"
2024-12-04T01:33:28.893Z [WARN]  server-998dd173-be3b-85ba-1601-8b9281a267de: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2024-12-04T01:33:28.910Z [INFO]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: updating configuration: command=AddVoter server-id=server-998dd173-be3b-85ba-1601-8b9281a267de server-addr=998dd173-be3b-85ba-1601-8b9281a267de servers="[{Suffrage:Voter ID:server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a Address:bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a} {Suffrage:Voter ID:server-4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372 Address:4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372} {Suffrage:Voter ID:server-998dd173-be3b-85ba-1601-8b9281a267de Address:998dd173-be3b-85ba-1601-8b9281a267de}]"
2024-12-04T01:33:28.910Z [INFO]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: added peer, starting replication: peer=server-998dd173-be3b-85ba-1601-8b9281a267de
2024-12-04T01:33:28.910Z [WARN]  server-998dd173-be3b-85ba-1601-8b9281a267de: failed to get previous log: previous-index=4 last-index=3 error="log not found"
2024-12-04T01:33:28.910Z [WARN]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: appendEntries rejected, sending older logs: peer="{Voter server-998dd173-be3b-85ba-1601-8b9281a267de 998dd173-be3b-85ba-1601-8b9281a267de}" next=4
2024-12-04T01:33:28.910Z [INFO]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: pipelining replication: peer="{Voter server-998dd173-be3b-85ba-1601-8b9281a267de 998dd173-be3b-85ba-1601-8b9281a267de}"
2024-12-04T01:33:28.910Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:29.016Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:28.91093633 +0000 UTC m=+99.742779620 (1 nodes), 4.36µs from start of poll, 384.357049ms from cluster start. Timeout at 2024-12-04 01:33:29.016003166 +0000 UTC m=+99.847845853, 105.066233ms after stability
2024-12-04T01:33:29.016Z [INFO]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: aborting pipeline replication: peer="{Voter server-4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372 4e3f51c8-d8d7-c390-ab6c-6c3e85a0e372}"
2024-12-04T01:33:29.016Z [INFO]  server-bd3cd6e5-95dc-b7d6-ae23-d2c707e19b1a: aborting pipeline replication: peer="{Voter server-998dd173-be3b-85ba-1601-8b9281a267de 998dd173-be3b-85ba-1601-8b9281a267de}"
--- PASS: TestRaft_ReJoinFollower (0.49s)
=== RUN   TestRaft_LeaderLeaseExpire
2024-12-04T01:33:29.018Z [INFO]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7 Address:afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7} {Suffrage:Voter ID:server-6587cf8e-cf41-50a2-6e45-43e4338486ba Address:6587cf8e-cf41-50a2-6e45-43e4338486ba}]"
2024-12-04T01:33:29.018Z [INFO]  server-6587cf8e-cf41-50a2-6e45-43e4338486ba: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7 Address:afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7} {Suffrage:Voter ID:server-6587cf8e-cf41-50a2-6e45-43e4338486ba Address:6587cf8e-cf41-50a2-6e45-43e4338486ba}]"
2024-12-04T01:33:29.018Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:29.018Z [INFO]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: entering follower state: follower="Node at afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7 [Follower]" leader-address= leader-id=
2024-12-04T01:33:29.018Z [INFO]  server-6587cf8e-cf41-50a2-6e45-43e4338486ba: entering follower state: follower="Node at 6587cf8e-cf41-50a2-6e45-43e4338486ba [Follower]" leader-address= leader-id=
2024-12-04T01:33:29.078Z [WARN]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:29.078Z [INFO]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: entering candidate state: node="Node at afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7 [Candidate]" term=2
2024-12-04T01:33:29.078Z [INFO]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: election won: term=2 tally=2
2024-12-04T01:33:29.078Z [INFO]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: entering leader state: leader="Node at afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7 [Leader]"
2024-12-04T01:33:29.078Z [INFO]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: added peer, starting replication: peer=server-6587cf8e-cf41-50a2-6e45-43e4338486ba
2024-12-04T01:33:29.078Z [INFO]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: pipelining replication: peer="{Voter server-6587cf8e-cf41-50a2-6e45-43e4338486ba 6587cf8e-cf41-50a2-6e45-43e4338486ba}"
2024-12-04T01:33:29.184Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:29.078991812 +0000 UTC m=+99.910834437 (1 nodes), 60.229601ms from start of poll, 60.632422ms from cluster start. Timeout at 2024-12-04 01:33:29.183994489 +0000 UTC m=+100.015837204, 105.002767ms after stability
2024-12-04T01:33:29.189Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:29.295Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:29.189548625 +0000 UTC m=+100.021391255 (1 nodes), 1.729µs from start of poll, 171.18924ms from cluster start. Timeout at 2024-12-04 01:33:29.294624706 +0000 UTC m=+100.126467394, 105.076139ms after stability
    raft_test.go:1799: [INFO] Disconnecting Node at 6587cf8e-cf41-50a2-6e45-43e4338486ba [Follower]
2024-12-04T01:33:29.298Z [ERROR] server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: failed to heartbeat to: peer=6587cf8e-cf41-50a2-6e45-43e4338486ba backoff time=10ms error="failed to connect to peer: 6587cf8e-cf41-50a2-6e45-43e4338486ba"
2024-12-04T01:33:29.299Z [ERROR] server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: failed to pipeline appendEntries: peer="{Voter server-6587cf8e-cf41-50a2-6e45-43e4338486ba 6587cf8e-cf41-50a2-6e45-43e4338486ba}" error="append pipeline closed"
2024-12-04T01:33:29.299Z [INFO]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: aborting pipeline replication: peer="{Voter server-6587cf8e-cf41-50a2-6e45-43e4338486ba 6587cf8e-cf41-50a2-6e45-43e4338486ba}"
2024-12-04T01:33:29.306Z [ERROR] server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: failed to appendEntries to: peer="{Voter server-6587cf8e-cf41-50a2-6e45-43e4338486ba 6587cf8e-cf41-50a2-6e45-43e4338486ba}" error="failed to connect to peer: 6587cf8e-cf41-50a2-6e45-43e4338486ba"
2024-12-04T01:33:29.316Z [ERROR] server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: failed to heartbeat to: peer=6587cf8e-cf41-50a2-6e45-43e4338486ba backoff time=10ms error="failed to connect to peer: 6587cf8e-cf41-50a2-6e45-43e4338486ba"
2024-12-04T01:33:29.324Z [ERROR] server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: failed to appendEntries to: peer="{Voter server-6587cf8e-cf41-50a2-6e45-43e4338486ba 6587cf8e-cf41-50a2-6e45-43e4338486ba}" error="failed to connect to peer: 6587cf8e-cf41-50a2-6e45-43e4338486ba"
2024-12-04T01:33:29.337Z [ERROR] server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: failed to heartbeat to: peer=6587cf8e-cf41-50a2-6e45-43e4338486ba backoff time=10ms error="failed to connect to peer: 6587cf8e-cf41-50a2-6e45-43e4338486ba"
2024-12-04T01:33:29.347Z [ERROR] server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: failed to appendEntries to: peer="{Voter server-6587cf8e-cf41-50a2-6e45-43e4338486ba 6587cf8e-cf41-50a2-6e45-43e4338486ba}" error="failed to connect to peer: 6587cf8e-cf41-50a2-6e45-43e4338486ba"
2024-12-04T01:33:29.348Z [WARN]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: failed to contact: server-id=server-6587cf8e-cf41-50a2-6e45-43e4338486ba time=54.115649ms
2024-12-04T01:33:29.348Z [WARN]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: failed to contact quorum of nodes, stepping down
2024-12-04T01:33:29.348Z [INFO]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: entering follower state: follower="Node at afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7 [Follower]" leader-address= leader-id=
2024-12-04T01:33:29.348Z [WARN]  server-6587cf8e-cf41-50a2-6e45-43e4338486ba: heartbeat timeout reached, starting election: last-leader-addr=afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7 last-leader-id=server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7
2024-12-04T01:33:29.349Z [INFO]  server-6587cf8e-cf41-50a2-6e45-43e4338486ba: entering candidate state: node="Node at 6587cf8e-cf41-50a2-6e45-43e4338486ba [Candidate]" term=3
2024-12-04T01:33:29.350Z [ERROR] server-6587cf8e-cf41-50a2-6e45-43e4338486ba: failed to make requestVote RPC: target="{Voter server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7 afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7}" error="failed to connect to peer: afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7" term=3
2024-12-04T01:33:29.350Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:29.416Z [WARN]  server-6587cf8e-cf41-50a2-6e45-43e4338486ba: Election timeout reached, restarting election
2024-12-04T01:33:29.416Z [INFO]  server-6587cf8e-cf41-50a2-6e45-43e4338486ba: entering candidate state: node="Node at 6587cf8e-cf41-50a2-6e45-43e4338486ba [Candidate]" term=4
2024-12-04T01:33:29.416Z [ERROR] server-6587cf8e-cf41-50a2-6e45-43e4338486ba: failed to make requestVote RPC: target="{Voter server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7 afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7}" error="failed to connect to peer: afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7" term=4
2024-12-04T01:33:29.434Z [WARN]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:29.434Z [INFO]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: entering candidate state: node="Node at afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7 [Candidate]" term=3
2024-12-04T01:33:29.434Z [ERROR] server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: failed to make requestVote RPC: target="{Voter server-6587cf8e-cf41-50a2-6e45-43e4338486ba 6587cf8e-cf41-50a2-6e45-43e4338486ba}" error="failed to connect to peer: 6587cf8e-cf41-50a2-6e45-43e4338486ba" term=3
2024-12-04T01:33:29.505Z [WARN]  server-6587cf8e-cf41-50a2-6e45-43e4338486ba: Election timeout reached, restarting election
2024-12-04T01:33:29.505Z [INFO]  server-6587cf8e-cf41-50a2-6e45-43e4338486ba: entering candidate state: node="Node at 6587cf8e-cf41-50a2-6e45-43e4338486ba [Candidate]" term=5
2024-12-04T01:33:29.505Z [ERROR] server-6587cf8e-cf41-50a2-6e45-43e4338486ba: failed to make requestVote RPC: target="{Voter server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7 afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7}" error="failed to connect to peer: afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7" term=5
2024-12-04T01:33:29.527Z [WARN]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: Election timeout reached, restarting election
2024-12-04T01:33:29.527Z [INFO]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: entering candidate state: node="Node at afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7 [Candidate]" term=4
2024-12-04T01:33:29.528Z [ERROR] server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: failed to make requestVote RPC: target="{Voter server-6587cf8e-cf41-50a2-6e45-43e4338486ba 6587cf8e-cf41-50a2-6e45-43e4338486ba}" error="failed to connect to peer: 6587cf8e-cf41-50a2-6e45-43e4338486ba" term=4
2024-12-04T01:33:29.540Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:29.434501722 +0000 UTC m=+100.266344761 (0 nodes), 83.409054ms from start of poll, 416.142746ms from cluster start. Timeout at 2024-12-04 01:33:29.539512743 +0000 UTC m=+100.371355430, 105.010669ms after stability
2024-12-04T01:33:29.563Z [WARN]  server-6587cf8e-cf41-50a2-6e45-43e4338486ba: Election timeout reached, restarting election
2024-12-04T01:33:29.564Z [INFO]  server-6587cf8e-cf41-50a2-6e45-43e4338486ba: entering candidate state: node="Node at 6587cf8e-cf41-50a2-6e45-43e4338486ba [Candidate]" term=6
2024-12-04T01:33:29.564Z [ERROR] server-6587cf8e-cf41-50a2-6e45-43e4338486ba: failed to make requestVote RPC: target="{Voter server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7 afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7}" error="failed to connect to peer: afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7" term=6
2024-12-04T01:33:29.590Z [WARN]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: Election timeout reached, restarting election
2024-12-04T01:33:29.590Z [INFO]  server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: entering candidate state: node="Node at afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7 [Candidate]" term=5
2024-12-04T01:33:29.590Z [ERROR] server-afc5a6a8-ac33-6f2d-bede-7aaa23cc1eb7: failed to make requestVote RPC: target="{Voter server-6587cf8e-cf41-50a2-6e45-43e4338486ba 6587cf8e-cf41-50a2-6e45-43e4338486ba}" error="failed to connect to peer: 6587cf8e-cf41-50a2-6e45-43e4338486ba" term=5
--- PASS: TestRaft_LeaderLeaseExpire (0.63s)
=== RUN   TestRaft_Barrier
2024-12-04T01:33:29.648Z [INFO]  server-c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84 Address:c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84} {Suffrage:Voter ID:server-b5bf06a8-16ce-d054-a499-bf23171a1d06 Address:b5bf06a8-16ce-d054-a499-bf23171a1d06} {Suffrage:Voter ID:server-cceac104-4ec6-5a31-e4aa-3c7fdcbb3259 Address:cceac104-4ec6-5a31-e4aa-3c7fdcbb3259}]"
2024-12-04T01:33:29.648Z [INFO]  server-b5bf06a8-16ce-d054-a499-bf23171a1d06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84 Address:c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84} {Suffrage:Voter ID:server-b5bf06a8-16ce-d054-a499-bf23171a1d06 Address:b5bf06a8-16ce-d054-a499-bf23171a1d06} {Suffrage:Voter ID:server-cceac104-4ec6-5a31-e4aa-3c7fdcbb3259 Address:cceac104-4ec6-5a31-e4aa-3c7fdcbb3259}]"
2024-12-04T01:33:29.649Z [INFO]  server-cceac104-4ec6-5a31-e4aa-3c7fdcbb3259: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84 Address:c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84} {Suffrage:Voter ID:server-b5bf06a8-16ce-d054-a499-bf23171a1d06 Address:b5bf06a8-16ce-d054-a499-bf23171a1d06} {Suffrage:Voter ID:server-cceac104-4ec6-5a31-e4aa-3c7fdcbb3259 Address:cceac104-4ec6-5a31-e4aa-3c7fdcbb3259}]"
2024-12-04T01:33:29.649Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:29.649Z [INFO]  server-c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84: entering follower state: follower="Node at c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84 [Follower]" leader-address= leader-id=
2024-12-04T01:33:29.649Z [INFO]  server-b5bf06a8-16ce-d054-a499-bf23171a1d06: entering follower state: follower="Node at b5bf06a8-16ce-d054-a499-bf23171a1d06 [Follower]" leader-address= leader-id=
2024-12-04T01:33:29.649Z [INFO]  server-cceac104-4ec6-5a31-e4aa-3c7fdcbb3259: entering follower state: follower="Node at cceac104-4ec6-5a31-e4aa-3c7fdcbb3259 [Follower]" leader-address= leader-id=
2024-12-04T01:33:29.702Z [WARN]  server-c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:29.702Z [INFO]  server-c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84: entering candidate state: node="Node at c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84 [Candidate]" term=2
2024-12-04T01:33:29.702Z [INFO]  server-c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84: election won: term=2 tally=2
2024-12-04T01:33:29.702Z [INFO]  server-c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84: entering leader state: leader="Node at c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84 [Leader]"
2024-12-04T01:33:29.702Z [INFO]  server-c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84: added peer, starting replication: peer=server-b5bf06a8-16ce-d054-a499-bf23171a1d06
2024-12-04T01:33:29.702Z [INFO]  server-c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84: added peer, starting replication: peer=server-cceac104-4ec6-5a31-e4aa-3c7fdcbb3259
2024-12-04T01:33:29.703Z [INFO]  server-c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84: pipelining replication: peer="{Voter server-cceac104-4ec6-5a31-e4aa-3c7fdcbb3259 cceac104-4ec6-5a31-e4aa-3c7fdcbb3259}"
2024-12-04T01:33:29.703Z [INFO]  server-c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84: pipelining replication: peer="{Voter server-b5bf06a8-16ce-d054-a499-bf23171a1d06 b5bf06a8-16ce-d054-a499-bf23171a1d06}"
2024-12-04T01:33:29.808Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:29.702957949 +0000 UTC m=+100.534801146 (1 nodes), 53.848725ms from start of poll, 54.473267ms from cluster start. Timeout at 2024-12-04 01:33:29.807961673 +0000 UTC m=+100.639804346, 105.0032ms after stability
2024-12-04T01:33:29.848Z [INFO]  server-c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84: aborting pipeline replication: peer="{Voter server-cceac104-4ec6-5a31-e4aa-3c7fdcbb3259 cceac104-4ec6-5a31-e4aa-3c7fdcbb3259}"
2024-12-04T01:33:29.848Z [INFO]  server-c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84: aborting pipeline replication: peer="{Voter server-b5bf06a8-16ce-d054-a499-bf23171a1d06 b5bf06a8-16ce-d054-a499-bf23171a1d06}"
2024-12-04T01:33:30.350Z [ERROR] server-c0561ca6-d974-5ef8-cc49-3f2e0a9e7d84: failed to heartbeat to: peer=cceac104-4ec6-5a31-e4aa-3c7fdcbb3259 backoff time=10ms error="command timed out"
--- PASS: TestRaft_Barrier (0.70s)
=== RUN   TestRaft_VerifyLeader
2024-12-04T01:33:30.352Z [INFO]  server-8cde776d-28c3-7fb3-6be3-70c4d3f4c022: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8cde776d-28c3-7fb3-6be3-70c4d3f4c022 Address:8cde776d-28c3-7fb3-6be3-70c4d3f4c022} {Suffrage:Voter ID:server-f1f5a995-6d51-4bb3-9941-ee660be5eaa7 Address:f1f5a995-6d51-4bb3-9941-ee660be5eaa7} {Suffrage:Voter ID:server-3560bebf-5624-10fc-1fac-2264860e5bc9 Address:3560bebf-5624-10fc-1fac-2264860e5bc9}]"
2024-12-04T01:33:30.352Z [INFO]  server-f1f5a995-6d51-4bb3-9941-ee660be5eaa7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8cde776d-28c3-7fb3-6be3-70c4d3f4c022 Address:8cde776d-28c3-7fb3-6be3-70c4d3f4c022} {Suffrage:Voter ID:server-f1f5a995-6d51-4bb3-9941-ee660be5eaa7 Address:f1f5a995-6d51-4bb3-9941-ee660be5eaa7} {Suffrage:Voter ID:server-3560bebf-5624-10fc-1fac-2264860e5bc9 Address:3560bebf-5624-10fc-1fac-2264860e5bc9}]"
2024-12-04T01:33:30.352Z [INFO]  server-3560bebf-5624-10fc-1fac-2264860e5bc9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8cde776d-28c3-7fb3-6be3-70c4d3f4c022 Address:8cde776d-28c3-7fb3-6be3-70c4d3f4c022} {Suffrage:Voter ID:server-f1f5a995-6d51-4bb3-9941-ee660be5eaa7 Address:f1f5a995-6d51-4bb3-9941-ee660be5eaa7} {Suffrage:Voter ID:server-3560bebf-5624-10fc-1fac-2264860e5bc9 Address:3560bebf-5624-10fc-1fac-2264860e5bc9}]"
2024-12-04T01:33:30.352Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:30.353Z [INFO]  server-8cde776d-28c3-7fb3-6be3-70c4d3f4c022: entering follower state: follower="Node at 8cde776d-28c3-7fb3-6be3-70c4d3f4c022 [Follower]" leader-address= leader-id=
2024-12-04T01:33:30.353Z [INFO]  server-f1f5a995-6d51-4bb3-9941-ee660be5eaa7: entering follower state: follower="Node at f1f5a995-6d51-4bb3-9941-ee660be5eaa7 [Follower]" leader-address= leader-id=
2024-12-04T01:33:30.353Z [INFO]  server-3560bebf-5624-10fc-1fac-2264860e5bc9: entering follower state: follower="Node at 3560bebf-5624-10fc-1fac-2264860e5bc9 [Follower]" leader-address= leader-id=
2024-12-04T01:33:30.412Z [WARN]  server-f1f5a995-6d51-4bb3-9941-ee660be5eaa7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:30.412Z [INFO]  server-f1f5a995-6d51-4bb3-9941-ee660be5eaa7: entering candidate state: node="Node at f1f5a995-6d51-4bb3-9941-ee660be5eaa7 [Candidate]" term=2
2024-12-04T01:33:30.413Z [INFO]  server-f1f5a995-6d51-4bb3-9941-ee660be5eaa7: election won: term=2 tally=2
2024-12-04T01:33:30.413Z [INFO]  server-f1f5a995-6d51-4bb3-9941-ee660be5eaa7: entering leader state: leader="Node at f1f5a995-6d51-4bb3-9941-ee660be5eaa7 [Leader]"
2024-12-04T01:33:30.413Z [INFO]  server-f1f5a995-6d51-4bb3-9941-ee660be5eaa7: added peer, starting replication: peer=server-8cde776d-28c3-7fb3-6be3-70c4d3f4c022
2024-12-04T01:33:30.413Z [INFO]  server-f1f5a995-6d51-4bb3-9941-ee660be5eaa7: added peer, starting replication: peer=server-3560bebf-5624-10fc-1fac-2264860e5bc9
2024-12-04T01:33:30.413Z [INFO]  server-f1f5a995-6d51-4bb3-9941-ee660be5eaa7: pipelining replication: peer="{Voter server-8cde776d-28c3-7fb3-6be3-70c4d3f4c022 8cde776d-28c3-7fb3-6be3-70c4d3f4c022}"
2024-12-04T01:33:30.413Z [INFO]  server-f1f5a995-6d51-4bb3-9941-ee660be5eaa7: pipelining replication: peer="{Voter server-3560bebf-5624-10fc-1fac-2264860e5bc9 3560bebf-5624-10fc-1fac-2264860e5bc9}"
2024-12-04T01:33:30.518Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:30.413182196 +0000 UTC m=+101.245024856 (1 nodes), 60.187662ms from start of poll, 60.674501ms from cluster start. Timeout at 2024-12-04 01:33:30.518196247 +0000 UTC m=+101.350038962, 105.014106ms after stability
2024-12-04T01:33:30.518Z [INFO]  server-f1f5a995-6d51-4bb3-9941-ee660be5eaa7: aborting pipeline replication: peer="{Voter server-3560bebf-5624-10fc-1fac-2264860e5bc9 3560bebf-5624-10fc-1fac-2264860e5bc9}"
2024-12-04T01:33:30.518Z [INFO]  server-f1f5a995-6d51-4bb3-9941-ee660be5eaa7: aborting pipeline replication: peer="{Voter server-8cde776d-28c3-7fb3-6be3-70c4d3f4c022 8cde776d-28c3-7fb3-6be3-70c4d3f4c022}"
2024-12-04T01:33:31.018Z [ERROR] server-f1f5a995-6d51-4bb3-9941-ee660be5eaa7: failed to heartbeat to: peer=8cde776d-28c3-7fb3-6be3-70c4d3f4c022 backoff time=10ms error="command timed out"
--- PASS: TestRaft_VerifyLeader (0.67s)
=== RUN   TestRaft_VerifyLeader_Single
2024-12-04T01:33:31.020Z [INFO]  server-bea8c76b-2ea5-c911-7916-4dfaab060947: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bea8c76b-2ea5-c911-7916-4dfaab060947 Address:bea8c76b-2ea5-c911-7916-4dfaab060947}]"
2024-12-04T01:33:31.020Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:31.020Z [INFO]  server-bea8c76b-2ea5-c911-7916-4dfaab060947: entering follower state: follower="Node at bea8c76b-2ea5-c911-7916-4dfaab060947 [Follower]" leader-address= leader-id=
2024-12-04T01:33:31.072Z [WARN]  server-bea8c76b-2ea5-c911-7916-4dfaab060947: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:31.073Z [INFO]  server-bea8c76b-2ea5-c911-7916-4dfaab060947: entering candidate state: node="Node at bea8c76b-2ea5-c911-7916-4dfaab060947 [Candidate]" term=2
2024-12-04T01:33:31.073Z [INFO]  server-bea8c76b-2ea5-c911-7916-4dfaab060947: election won: term=2 tally=1
2024-12-04T01:33:31.073Z [INFO]  server-bea8c76b-2ea5-c911-7916-4dfaab060947: entering leader state: leader="Node at bea8c76b-2ea5-c911-7916-4dfaab060947 [Leader]"
2024-12-04T01:33:31.196Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:31.073230865 +0000 UTC m=+101.905073457 (1 nodes), 52.77185ms from start of poll, 53.326093ms from cluster start. Timeout at 2024-12-04 01:33:31.178268607 +0000 UTC m=+102.010111362, 105.037905ms after stability
--- PASS: TestRaft_VerifyLeader_Single (0.18s)
=== RUN   TestRaft_VerifyLeader_Fail
2024-12-04T01:33:31.197Z [INFO]  server-f8929a88-3331-54a9-087a-34edfb4d88c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8929a88-3331-54a9-087a-34edfb4d88c8 Address:f8929a88-3331-54a9-087a-34edfb4d88c8} {Suffrage:Voter ID:server-fbe8a84f-3e50-16bc-ee5d-d9aec0a326a4 Address:fbe8a84f-3e50-16bc-ee5d-d9aec0a326a4}]"
2024-12-04T01:33:31.198Z [INFO]  server-fbe8a84f-3e50-16bc-ee5d-d9aec0a326a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f8929a88-3331-54a9-087a-34edfb4d88c8 Address:f8929a88-3331-54a9-087a-34edfb4d88c8} {Suffrage:Voter ID:server-fbe8a84f-3e50-16bc-ee5d-d9aec0a326a4 Address:fbe8a84f-3e50-16bc-ee5d-d9aec0a326a4}]"
2024-12-04T01:33:31.197Z [INFO]  server-f8929a88-3331-54a9-087a-34edfb4d88c8: entering follower state: follower="Node at f8929a88-3331-54a9-087a-34edfb4d88c8 [Follower]" leader-address= leader-id=
2024-12-04T01:33:31.198Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:31.198Z [INFO]  server-fbe8a84f-3e50-16bc-ee5d-d9aec0a326a4: entering follower state: follower="Node at fbe8a84f-3e50-16bc-ee5d-d9aec0a326a4 [Follower]" leader-address= leader-id=
2024-12-04T01:33:31.277Z [WARN]  server-f8929a88-3331-54a9-087a-34edfb4d88c8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:31.277Z [INFO]  server-f8929a88-3331-54a9-087a-34edfb4d88c8: entering candidate state: node="Node at f8929a88-3331-54a9-087a-34edfb4d88c8 [Candidate]" term=2
2024-12-04T01:33:31.277Z [INFO]  server-f8929a88-3331-54a9-087a-34edfb4d88c8: election won: term=2 tally=2
2024-12-04T01:33:31.277Z [INFO]  server-f8929a88-3331-54a9-087a-34edfb4d88c8: entering leader state: leader="Node at f8929a88-3331-54a9-087a-34edfb4d88c8 [Leader]"
2024-12-04T01:33:31.277Z [INFO]  server-f8929a88-3331-54a9-087a-34edfb4d88c8: added peer, starting replication: peer=server-fbe8a84f-3e50-16bc-ee5d-d9aec0a326a4
2024-12-04T01:33:31.277Z [INFO]  server-f8929a88-3331-54a9-087a-34edfb4d88c8: pipelining replication: peer="{Voter server-fbe8a84f-3e50-16bc-ee5d-d9aec0a326a4 fbe8a84f-3e50-16bc-ee5d-d9aec0a326a4}"
2024-12-04T01:33:31.383Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:31.277973162 +0000 UTC m=+102.109815817 (1 nodes), 79.882772ms from start of poll, 80.234573ms from cluster start. Timeout at 2024-12-04 01:33:31.382982644 +0000 UTC m=+102.214825215, 105.009398ms after stability
2024-12-04T01:33:31.383Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:31.489Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:31.383576215 +0000 UTC m=+102.215418799 (1 nodes), 1.687µs from start of poll, 185.837555ms from cluster start. Timeout at 2024-12-04 01:33:31.48864427 +0000 UTC m=+102.320487085, 105.068286ms after stability
2024-12-04T01:33:31.490Z [ERROR] server-f8929a88-3331-54a9-087a-34edfb4d88c8: peer has newer term, stopping replication: peer="{Voter server-fbe8a84f-3e50-16bc-ee5d-d9aec0a326a4 fbe8a84f-3e50-16bc-ee5d-d9aec0a326a4}"
2024-12-04T01:33:31.490Z [INFO]  server-f8929a88-3331-54a9-087a-34edfb4d88c8: aborting pipeline replication: peer="{Voter server-fbe8a84f-3e50-16bc-ee5d-d9aec0a326a4 fbe8a84f-3e50-16bc-ee5d-d9aec0a326a4}"
2024-12-04T01:33:31.490Z [INFO]  server-f8929a88-3331-54a9-087a-34edfb4d88c8: entering follower state: follower="Node at f8929a88-3331-54a9-087a-34edfb4d88c8 [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VerifyLeader_Fail (0.29s)
=== RUN   TestRaft_VerifyLeader_PartialConnect
2024-12-04T01:33:31.491Z [INFO]  server-8c9680de-b780-1863-f49a-650942378d21: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c9680de-b780-1863-f49a-650942378d21 Address:8c9680de-b780-1863-f49a-650942378d21} {Suffrage:Voter ID:server-4c00478b-3165-a139-0854-2184e54114d7 Address:4c00478b-3165-a139-0854-2184e54114d7} {Suffrage:Voter ID:server-16d9f4b1-1bb1-6255-f0d9-7e5951140db3 Address:16d9f4b1-1bb1-6255-f0d9-7e5951140db3}]"
2024-12-04T01:33:31.491Z [INFO]  server-4c00478b-3165-a139-0854-2184e54114d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c9680de-b780-1863-f49a-650942378d21 Address:8c9680de-b780-1863-f49a-650942378d21} {Suffrage:Voter ID:server-4c00478b-3165-a139-0854-2184e54114d7 Address:4c00478b-3165-a139-0854-2184e54114d7} {Suffrage:Voter ID:server-16d9f4b1-1bb1-6255-f0d9-7e5951140db3 Address:16d9f4b1-1bb1-6255-f0d9-7e5951140db3}]"
2024-12-04T01:33:31.492Z [INFO]  server-16d9f4b1-1bb1-6255-f0d9-7e5951140db3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c9680de-b780-1863-f49a-650942378d21 Address:8c9680de-b780-1863-f49a-650942378d21} {Suffrage:Voter ID:server-4c00478b-3165-a139-0854-2184e54114d7 Address:4c00478b-3165-a139-0854-2184e54114d7} {Suffrage:Voter ID:server-16d9f4b1-1bb1-6255-f0d9-7e5951140db3 Address:16d9f4b1-1bb1-6255-f0d9-7e5951140db3}]"
2024-12-04T01:33:31.492Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:31.492Z [INFO]  server-8c9680de-b780-1863-f49a-650942378d21: entering follower state: follower="Node at 8c9680de-b780-1863-f49a-650942378d21 [Follower]" leader-address= leader-id=
2024-12-04T01:33:31.492Z [INFO]  server-4c00478b-3165-a139-0854-2184e54114d7: entering follower state: follower="Node at 4c00478b-3165-a139-0854-2184e54114d7 [Follower]" leader-address= leader-id=
2024-12-04T01:33:31.492Z [INFO]  server-16d9f4b1-1bb1-6255-f0d9-7e5951140db3: entering follower state: follower="Node at 16d9f4b1-1bb1-6255-f0d9-7e5951140db3 [Follower]" leader-address= leader-id=
2024-12-04T01:33:31.558Z [WARN]  server-4c00478b-3165-a139-0854-2184e54114d7: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:31.558Z [INFO]  server-4c00478b-3165-a139-0854-2184e54114d7: entering candidate state: node="Node at 4c00478b-3165-a139-0854-2184e54114d7 [Candidate]" term=2
2024-12-04T01:33:31.558Z [INFO]  server-4c00478b-3165-a139-0854-2184e54114d7: election won: term=2 tally=2
2024-12-04T01:33:31.558Z [INFO]  server-4c00478b-3165-a139-0854-2184e54114d7: entering leader state: leader="Node at 4c00478b-3165-a139-0854-2184e54114d7 [Leader]"
2024-12-04T01:33:31.559Z [INFO]  server-4c00478b-3165-a139-0854-2184e54114d7: added peer, starting replication: peer=server-8c9680de-b780-1863-f49a-650942378d21
2024-12-04T01:33:31.559Z [INFO]  server-4c00478b-3165-a139-0854-2184e54114d7: added peer, starting replication: peer=server-16d9f4b1-1bb1-6255-f0d9-7e5951140db3
2024-12-04T01:33:31.559Z [INFO]  server-4c00478b-3165-a139-0854-2184e54114d7: pipelining replication: peer="{Voter server-16d9f4b1-1bb1-6255-f0d9-7e5951140db3 16d9f4b1-1bb1-6255-f0d9-7e5951140db3}"
2024-12-04T01:33:31.559Z [INFO]  server-4c00478b-3165-a139-0854-2184e54114d7: pipelining replication: peer="{Voter server-8c9680de-b780-1863-f49a-650942378d21 8c9680de-b780-1863-f49a-650942378d21}"
2024-12-04T01:33:31.665Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:31.559696951 +0000 UTC m=+102.391539615 (1 nodes), 67.620355ms from start of poll, 68.080559ms from cluster start. Timeout at 2024-12-04 01:33:31.66470045 +0000 UTC m=+102.496543034, 105.003419ms after stability
2024-12-04T01:33:31.670Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:31.775Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:31.670626239 +0000 UTC m=+102.502468816 (2 nodes), 2.576µs from start of poll, 179.00976ms from cluster start. Timeout at 2024-12-04 01:33:31.775725743 +0000 UTC m=+102.607568577, 105.099761ms after stability
    raft_test.go:1968: [INFO] Disconnecting Node at 8c9680de-b780-1863-f49a-650942378d21 [Follower]
2024-12-04T01:33:31.776Z [ERROR] server-4c00478b-3165-a139-0854-2184e54114d7: failed to heartbeat to: peer=8c9680de-b780-1863-f49a-650942378d21 backoff time=10ms error="failed to connect to peer: 8c9680de-b780-1863-f49a-650942378d21"
2024-12-04T01:33:31.776Z [INFO]  server-4c00478b-3165-a139-0854-2184e54114d7: aborting pipeline replication: peer="{Voter server-16d9f4b1-1bb1-6255-f0d9-7e5951140db3 16d9f4b1-1bb1-6255-f0d9-7e5951140db3}"
2024-12-04T01:33:31.776Z [INFO]  server-4c00478b-3165-a139-0854-2184e54114d7: aborting pipeline replication: peer="{Voter server-8c9680de-b780-1863-f49a-650942378d21 8c9680de-b780-1863-f49a-650942378d21}"
--- PASS: TestRaft_VerifyLeader_PartialConnect (0.29s)
=== RUN   TestRaft_NotifyCh
2024-12-04T01:33:31.777Z [INFO]  server-653d286a-2c23-503c-6133-84b793abdce4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-653d286a-2c23-503c-6133-84b793abdce4 Address:653d286a-2c23-503c-6133-84b793abdce4}]"
2024-12-04T01:33:31.777Z [INFO]  server-653d286a-2c23-503c-6133-84b793abdce4: entering follower state: follower="Node at 653d286a-2c23-503c-6133-84b793abdce4 [Follower]" leader-address= leader-id=
2024-12-04T01:33:31.877Z [WARN]  server-653d286a-2c23-503c-6133-84b793abdce4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:31.877Z [INFO]  server-653d286a-2c23-503c-6133-84b793abdce4: entering candidate state: node="Node at 653d286a-2c23-503c-6133-84b793abdce4 [Candidate]" term=2
2024-12-04T01:33:31.877Z [INFO]  server-653d286a-2c23-503c-6133-84b793abdce4: election won: term=2 tally=1
2024-12-04T01:33:31.877Z [INFO]  server-653d286a-2c23-503c-6133-84b793abdce4: entering leader state: leader="Node at 653d286a-2c23-503c-6133-84b793abdce4 [Leader]"
--- PASS: TestRaft_NotifyCh (0.10s)
=== RUN   TestRaft_AppendEntry
2024-12-04T01:33:31.878Z [INFO]  server-df32ddb8-c6ca-abc0-0b63-059c9408d823: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df32ddb8-c6ca-abc0-0b63-059c9408d823 Address:df32ddb8-c6ca-abc0-0b63-059c9408d823} {Suffrage:Voter ID:server-ec6861ea-1331-7e52-7eb1-9ecaf2975f31 Address:ec6861ea-1331-7e52-7eb1-9ecaf2975f31} {Suffrage:Voter ID:server-ffe5ec40-d3e4-5343-0a3e-9a69ca46f48d Address:ffe5ec40-d3e4-5343-0a3e-9a69ca46f48d}]"
2024-12-04T01:33:31.878Z [INFO]  server-ec6861ea-1331-7e52-7eb1-9ecaf2975f31: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df32ddb8-c6ca-abc0-0b63-059c9408d823 Address:df32ddb8-c6ca-abc0-0b63-059c9408d823} {Suffrage:Voter ID:server-ec6861ea-1331-7e52-7eb1-9ecaf2975f31 Address:ec6861ea-1331-7e52-7eb1-9ecaf2975f31} {Suffrage:Voter ID:server-ffe5ec40-d3e4-5343-0a3e-9a69ca46f48d Address:ffe5ec40-d3e4-5343-0a3e-9a69ca46f48d}]"
2024-12-04T01:33:31.878Z [INFO]  server-ffe5ec40-d3e4-5343-0a3e-9a69ca46f48d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-df32ddb8-c6ca-abc0-0b63-059c9408d823 Address:df32ddb8-c6ca-abc0-0b63-059c9408d823} {Suffrage:Voter ID:server-ec6861ea-1331-7e52-7eb1-9ecaf2975f31 Address:ec6861ea-1331-7e52-7eb1-9ecaf2975f31} {Suffrage:Voter ID:server-ffe5ec40-d3e4-5343-0a3e-9a69ca46f48d Address:ffe5ec40-d3e4-5343-0a3e-9a69ca46f48d}]"
2024-12-04T01:33:31.878Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:31.878Z [INFO]  server-df32ddb8-c6ca-abc0-0b63-059c9408d823: entering follower state: follower="Node at df32ddb8-c6ca-abc0-0b63-059c9408d823 [Follower]" leader-address= leader-id=
2024-12-04T01:33:31.879Z [INFO]  server-ec6861ea-1331-7e52-7eb1-9ecaf2975f31: entering follower state: follower="Node at ec6861ea-1331-7e52-7eb1-9ecaf2975f31 [Follower]" leader-address= leader-id=
2024-12-04T01:33:31.879Z [INFO]  server-ffe5ec40-d3e4-5343-0a3e-9a69ca46f48d: entering follower state: follower="Node at ffe5ec40-d3e4-5343-0a3e-9a69ca46f48d [Follower]" leader-address= leader-id=
2024-12-04T01:33:31.944Z [WARN]  server-df32ddb8-c6ca-abc0-0b63-059c9408d823: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:31.944Z [INFO]  server-df32ddb8-c6ca-abc0-0b63-059c9408d823: entering candidate state: node="Node at df32ddb8-c6ca-abc0-0b63-059c9408d823 [Candidate]" term=2
2024-12-04T01:33:31.944Z [INFO]  server-df32ddb8-c6ca-abc0-0b63-059c9408d823: election won: term=2 tally=2
2024-12-04T01:33:31.944Z [INFO]  server-df32ddb8-c6ca-abc0-0b63-059c9408d823: entering leader state: leader="Node at df32ddb8-c6ca-abc0-0b63-059c9408d823 [Leader]"
2024-12-04T01:33:31.944Z [INFO]  server-df32ddb8-c6ca-abc0-0b63-059c9408d823: added peer, starting replication: peer=server-ec6861ea-1331-7e52-7eb1-9ecaf2975f31
2024-12-04T01:33:31.944Z [INFO]  server-df32ddb8-c6ca-abc0-0b63-059c9408d823: added peer, starting replication: peer=server-ffe5ec40-d3e4-5343-0a3e-9a69ca46f48d
2024-12-04T01:33:31.944Z [INFO]  server-df32ddb8-c6ca-abc0-0b63-059c9408d823: pipelining replication: peer="{Voter server-ec6861ea-1331-7e52-7eb1-9ecaf2975f31 ec6861ea-1331-7e52-7eb1-9ecaf2975f31}"
2024-12-04T01:33:31.944Z [INFO]  server-df32ddb8-c6ca-abc0-0b63-059c9408d823: pipelining replication: peer="{Voter server-ffe5ec40-d3e4-5343-0a3e-9a69ca46f48d ffe5ec40-d3e4-5343-0a3e-9a69ca46f48d}"
2024-12-04T01:33:32.050Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:31.944322625 +0000 UTC m=+102.776165206 (2 nodes), 65.389226ms from start of poll, 65.781808ms from cluster start. Timeout at 2024-12-04 01:33:32.049333225 +0000 UTC m=+102.881175790, 105.010584ms after stability
2024-12-04T01:33:32.050Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:32.156Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:32.0503166 +0000 UTC m=+102.882159158 (1 nodes), 1.912µs from start of poll, 171.77576ms from cluster start. Timeout at 2024-12-04 01:33:32.155452749 +0000 UTC m=+102.987295300, 105.136142ms after stability
2024-12-04T01:33:32.156Z [WARN]  server-ec6861ea-1331-7e52-7eb1-9ecaf2975f31: clearing log suffix: from=1 to=2
2024-12-04T01:33:32.156Z [WARN]  server-ec6861ea-1331-7e52-7eb1-9ecaf2975f31: skipping application of old log: index=1
2024-12-04T01:33:32.156Z [WARN]  server-ec6861ea-1331-7e52-7eb1-9ecaf2975f31: skipping application of old log: index=1
2024-12-04T01:33:32.156Z [INFO]  server-df32ddb8-c6ca-abc0-0b63-059c9408d823: aborting pipeline replication: peer="{Voter server-ec6861ea-1331-7e52-7eb1-9ecaf2975f31 ec6861ea-1331-7e52-7eb1-9ecaf2975f31}"
2024-12-04T01:33:32.156Z [INFO]  server-df32ddb8-c6ca-abc0-0b63-059c9408d823: aborting pipeline replication: peer="{Voter server-ffe5ec40-d3e4-5343-0a3e-9a69ca46f48d ffe5ec40-d3e4-5343-0a3e-9a69ca46f48d}"
--- PASS: TestRaft_AppendEntry (0.28s)
=== RUN   TestRaft_VotingGrant_WhenLeaderAvailable
2024-12-04T01:33:32.158Z [INFO]  server-4108c97e-3612-fd2d-cb21-daf919827af0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4108c97e-3612-fd2d-cb21-daf919827af0 Address:4108c97e-3612-fd2d-cb21-daf919827af0} {Suffrage:Voter ID:server-3a79787a-7335-57cf-4665-7bc3b4d1e735 Address:3a79787a-7335-57cf-4665-7bc3b4d1e735} {Suffrage:Voter ID:server-007d467f-7770-edaf-d5af-9266961f7077 Address:007d467f-7770-edaf-d5af-9266961f7077}]"
2024-12-04T01:33:32.158Z [INFO]  server-3a79787a-7335-57cf-4665-7bc3b4d1e735: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4108c97e-3612-fd2d-cb21-daf919827af0 Address:4108c97e-3612-fd2d-cb21-daf919827af0} {Suffrage:Voter ID:server-3a79787a-7335-57cf-4665-7bc3b4d1e735 Address:3a79787a-7335-57cf-4665-7bc3b4d1e735} {Suffrage:Voter ID:server-007d467f-7770-edaf-d5af-9266961f7077 Address:007d467f-7770-edaf-d5af-9266961f7077}]"
2024-12-04T01:33:32.158Z [INFO]  server-007d467f-7770-edaf-d5af-9266961f7077: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4108c97e-3612-fd2d-cb21-daf919827af0 Address:4108c97e-3612-fd2d-cb21-daf919827af0} {Suffrage:Voter ID:server-3a79787a-7335-57cf-4665-7bc3b4d1e735 Address:3a79787a-7335-57cf-4665-7bc3b4d1e735} {Suffrage:Voter ID:server-007d467f-7770-edaf-d5af-9266961f7077 Address:007d467f-7770-edaf-d5af-9266961f7077}]"
2024-12-04T01:33:32.158Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:32.158Z [INFO]  server-4108c97e-3612-fd2d-cb21-daf919827af0: entering follower state: follower="Node at 4108c97e-3612-fd2d-cb21-daf919827af0 [Follower]" leader-address= leader-id=
2024-12-04T01:33:32.158Z [INFO]  server-3a79787a-7335-57cf-4665-7bc3b4d1e735: entering follower state: follower="Node at 3a79787a-7335-57cf-4665-7bc3b4d1e735 [Follower]" leader-address= leader-id=
2024-12-04T01:33:32.158Z [INFO]  server-007d467f-7770-edaf-d5af-9266961f7077: entering follower state: follower="Node at 007d467f-7770-edaf-d5af-9266961f7077 [Follower]" leader-address= leader-id=
2024-12-04T01:33:32.224Z [WARN]  server-4108c97e-3612-fd2d-cb21-daf919827af0: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:32.224Z [INFO]  server-4108c97e-3612-fd2d-cb21-daf919827af0: entering candidate state: node="Node at 4108c97e-3612-fd2d-cb21-daf919827af0 [Candidate]" term=2
2024-12-04T01:33:32.224Z [INFO]  server-4108c97e-3612-fd2d-cb21-daf919827af0: election won: term=2 tally=2
2024-12-04T01:33:32.224Z [INFO]  server-4108c97e-3612-fd2d-cb21-daf919827af0: entering leader state: leader="Node at 4108c97e-3612-fd2d-cb21-daf919827af0 [Leader]"
2024-12-04T01:33:32.224Z [INFO]  server-4108c97e-3612-fd2d-cb21-daf919827af0: added peer, starting replication: peer=server-3a79787a-7335-57cf-4665-7bc3b4d1e735
2024-12-04T01:33:32.224Z [INFO]  server-4108c97e-3612-fd2d-cb21-daf919827af0: added peer, starting replication: peer=server-007d467f-7770-edaf-d5af-9266961f7077
2024-12-04T01:33:32.224Z [INFO]  server-4108c97e-3612-fd2d-cb21-daf919827af0: pipelining replication: peer="{Voter server-3a79787a-7335-57cf-4665-7bc3b4d1e735 3a79787a-7335-57cf-4665-7bc3b4d1e735}"
2024-12-04T01:33:32.224Z [INFO]  server-4108c97e-3612-fd2d-cb21-daf919827af0: pipelining replication: peer="{Voter server-007d467f-7770-edaf-d5af-9266961f7077 007d467f-7770-edaf-d5af-9266961f7077}"
2024-12-04T01:33:32.331Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:32.224976011 +0000 UTC m=+103.056818579 (2 nodes), 66.379449ms from start of poll, 66.759431ms from cluster start. Timeout at 2024-12-04 01:33:32.329982569 +0000 UTC m=+103.161825171, 105.006592ms after stability
2024-12-04T01:33:32.331Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:32.438Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:32.331782176 +0000 UTC m=+103.163624748 (1 nodes), 1.7µs from start of poll, 173.5656ms from cluster start. Timeout at 2024-12-04 01:33:32.436892301 +0000 UTC m=+103.268734862, 105.110114ms after stability
2024-12-04T01:33:32.438Z [WARN]  server-007d467f-7770-edaf-d5af-9266961f7077: rejecting vote request since we have a leader: from=3a79787a-7335-57cf-4665-7bc3b4d1e735 leader=4108c97e-3612-fd2d-cb21-daf919827af0 leader-id=server-4108c97e-3612-fd2d-cb21-daf919827af0
2024-12-04T01:33:32.438Z [INFO]  server-4108c97e-3612-fd2d-cb21-daf919827af0: aborting pipeline replication: peer="{Voter server-007d467f-7770-edaf-d5af-9266961f7077 007d467f-7770-edaf-d5af-9266961f7077}"
2024-12-04T01:33:32.438Z [INFO]  server-4108c97e-3612-fd2d-cb21-daf919827af0: aborting pipeline replication: peer="{Voter server-3a79787a-7335-57cf-4665-7bc3b4d1e735 3a79787a-7335-57cf-4665-7bc3b4d1e735}"
--- PASS: TestRaft_VotingGrant_WhenLeaderAvailable (0.28s)
=== RUN   TestRaft_ProtocolVersion_RejectRPC
2024-12-04T01:33:32.440Z [INFO]  server-17cf29cc-0c07-b537-d56e-6317896cce3c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17cf29cc-0c07-b537-d56e-6317896cce3c Address:17cf29cc-0c07-b537-d56e-6317896cce3c} {Suffrage:Voter ID:server-7957d2ff-00a6-b113-4ea7-7d3bd35c615d Address:7957d2ff-00a6-b113-4ea7-7d3bd35c615d} {Suffrage:Voter ID:server-5584cadf-cd1a-fda5-23f9-6491ab195b97 Address:5584cadf-cd1a-fda5-23f9-6491ab195b97}]"
2024-12-04T01:33:32.440Z [INFO]  server-7957d2ff-00a6-b113-4ea7-7d3bd35c615d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17cf29cc-0c07-b537-d56e-6317896cce3c Address:17cf29cc-0c07-b537-d56e-6317896cce3c} {Suffrage:Voter ID:server-7957d2ff-00a6-b113-4ea7-7d3bd35c615d Address:7957d2ff-00a6-b113-4ea7-7d3bd35c615d} {Suffrage:Voter ID:server-5584cadf-cd1a-fda5-23f9-6491ab195b97 Address:5584cadf-cd1a-fda5-23f9-6491ab195b97}]"
2024-12-04T01:33:32.440Z [INFO]  server-5584cadf-cd1a-fda5-23f9-6491ab195b97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-17cf29cc-0c07-b537-d56e-6317896cce3c Address:17cf29cc-0c07-b537-d56e-6317896cce3c} {Suffrage:Voter ID:server-7957d2ff-00a6-b113-4ea7-7d3bd35c615d Address:7957d2ff-00a6-b113-4ea7-7d3bd35c615d} {Suffrage:Voter ID:server-5584cadf-cd1a-fda5-23f9-6491ab195b97 Address:5584cadf-cd1a-fda5-23f9-6491ab195b97}]"
2024-12-04T01:33:32.440Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:32.440Z [INFO]  server-17cf29cc-0c07-b537-d56e-6317896cce3c: entering follower state: follower="Node at 17cf29cc-0c07-b537-d56e-6317896cce3c [Follower]" leader-address= leader-id=
2024-12-04T01:33:32.440Z [INFO]  server-7957d2ff-00a6-b113-4ea7-7d3bd35c615d: entering follower state: follower="Node at 7957d2ff-00a6-b113-4ea7-7d3bd35c615d [Follower]" leader-address= leader-id=
2024-12-04T01:33:32.440Z [INFO]  server-5584cadf-cd1a-fda5-23f9-6491ab195b97: entering follower state: follower="Node at 5584cadf-cd1a-fda5-23f9-6491ab195b97 [Follower]" leader-address= leader-id=
2024-12-04T01:33:32.493Z [WARN]  server-7957d2ff-00a6-b113-4ea7-7d3bd35c615d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:32.493Z [INFO]  server-7957d2ff-00a6-b113-4ea7-7d3bd35c615d: entering candidate state: node="Node at 7957d2ff-00a6-b113-4ea7-7d3bd35c615d [Candidate]" term=2
2024-12-04T01:33:32.493Z [INFO]  server-7957d2ff-00a6-b113-4ea7-7d3bd35c615d: election won: term=2 tally=2
2024-12-04T01:33:32.493Z [INFO]  server-7957d2ff-00a6-b113-4ea7-7d3bd35c615d: entering leader state: leader="Node at 7957d2ff-00a6-b113-4ea7-7d3bd35c615d [Leader]"
2024-12-04T01:33:32.493Z [INFO]  server-7957d2ff-00a6-b113-4ea7-7d3bd35c615d: added peer, starting replication: peer=server-17cf29cc-0c07-b537-d56e-6317896cce3c
2024-12-04T01:33:32.493Z [INFO]  server-7957d2ff-00a6-b113-4ea7-7d3bd35c615d: added peer, starting replication: peer=server-5584cadf-cd1a-fda5-23f9-6491ab195b97
2024-12-04T01:33:32.493Z [INFO]  server-7957d2ff-00a6-b113-4ea7-7d3bd35c615d: pipelining replication: peer="{Voter server-5584cadf-cd1a-fda5-23f9-6491ab195b97 5584cadf-cd1a-fda5-23f9-6491ab195b97}"
2024-12-04T01:33:32.493Z [INFO]  server-7957d2ff-00a6-b113-4ea7-7d3bd35c615d: pipelining replication: peer="{Voter server-17cf29cc-0c07-b537-d56e-6317896cce3c 17cf29cc-0c07-b537-d56e-6317896cce3c}"
2024-12-04T01:33:32.599Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:32.493800974 +0000 UTC m=+103.325643554 (2 nodes), 53.402392ms from start of poll, 53.769949ms from cluster start. Timeout at 2024-12-04 01:33:32.598822932 +0000 UTC m=+103.430665494, 105.02194ms after stability
2024-12-04T01:33:32.599Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:32.705Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:32.600016829 +0000 UTC m=+103.431859395 (1 nodes), 2.42µs from start of poll, 159.98579ms from cluster start. Timeout at 2024-12-04 01:33:32.705267113 +0000 UTC m=+103.537109840, 105.250445ms after stability
2024-12-04T01:33:32.706Z [INFO]  server-7957d2ff-00a6-b113-4ea7-7d3bd35c615d: aborting pipeline replication: peer="{Voter server-5584cadf-cd1a-fda5-23f9-6491ab195b97 5584cadf-cd1a-fda5-23f9-6491ab195b97}"
2024-12-04T01:33:32.706Z [INFO]  server-7957d2ff-00a6-b113-4ea7-7d3bd35c615d: aborting pipeline replication: peer="{Voter server-17cf29cc-0c07-b537-d56e-6317896cce3c 17cf29cc-0c07-b537-d56e-6317896cce3c}"
--- PASS: TestRaft_ProtocolVersion_RejectRPC (0.27s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_1_2
    raft_test.go:2151: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_1_2 (0.00s)
=== RUN   TestRaft_ProtocolVersion_Upgrade_2_3
    raft_test.go:2196: Skipping unreliable test
--- SKIP: TestRaft_ProtocolVersion_Upgrade_2_3 (0.00s)
=== RUN   TestRaft_LeaderID_Propagated
2024-12-04T01:33:32.708Z [INFO]  server-ff3873c3-04b7-9015-9dfc-641d18432f09: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff3873c3-04b7-9015-9dfc-641d18432f09 Address:ff3873c3-04b7-9015-9dfc-641d18432f09} {Suffrage:Voter ID:server-d281cf0d-59b0-ec82-5174-cc8aaf1c94a7 Address:d281cf0d-59b0-ec82-5174-cc8aaf1c94a7} {Suffrage:Voter ID:server-ab379d9c-8cb3-6a05-5c22-0f0937e4f92f Address:ab379d9c-8cb3-6a05-5c22-0f0937e4f92f}]"
2024-12-04T01:33:32.708Z [INFO]  server-d281cf0d-59b0-ec82-5174-cc8aaf1c94a7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff3873c3-04b7-9015-9dfc-641d18432f09 Address:ff3873c3-04b7-9015-9dfc-641d18432f09} {Suffrage:Voter ID:server-d281cf0d-59b0-ec82-5174-cc8aaf1c94a7 Address:d281cf0d-59b0-ec82-5174-cc8aaf1c94a7} {Suffrage:Voter ID:server-ab379d9c-8cb3-6a05-5c22-0f0937e4f92f Address:ab379d9c-8cb3-6a05-5c22-0f0937e4f92f}]"
2024-12-04T01:33:32.708Z [INFO]  server-ff3873c3-04b7-9015-9dfc-641d18432f09: entering follower state: follower="Node at ff3873c3-04b7-9015-9dfc-641d18432f09 [Follower]" leader-address= leader-id=
2024-12-04T01:33:32.708Z [INFO]  server-d281cf0d-59b0-ec82-5174-cc8aaf1c94a7: entering follower state: follower="Node at d281cf0d-59b0-ec82-5174-cc8aaf1c94a7 [Follower]" leader-address= leader-id=
2024-12-04T01:33:32.708Z [INFO]  server-ab379d9c-8cb3-6a05-5c22-0f0937e4f92f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ff3873c3-04b7-9015-9dfc-641d18432f09 Address:ff3873c3-04b7-9015-9dfc-641d18432f09} {Suffrage:Voter ID:server-d281cf0d-59b0-ec82-5174-cc8aaf1c94a7 Address:d281cf0d-59b0-ec82-5174-cc8aaf1c94a7} {Suffrage:Voter ID:server-ab379d9c-8cb3-6a05-5c22-0f0937e4f92f Address:ab379d9c-8cb3-6a05-5c22-0f0937e4f92f}]"
2024-12-04T01:33:32.708Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:32.708Z [INFO]  server-ab379d9c-8cb3-6a05-5c22-0f0937e4f92f: entering follower state: follower="Node at ab379d9c-8cb3-6a05-5c22-0f0937e4f92f [Follower]" leader-address= leader-id=
2024-12-04T01:33:32.764Z [WARN]  server-ab379d9c-8cb3-6a05-5c22-0f0937e4f92f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:32.764Z [INFO]  server-ab379d9c-8cb3-6a05-5c22-0f0937e4f92f: entering candidate state: node="Node at ab379d9c-8cb3-6a05-5c22-0f0937e4f92f [Candidate]" term=2
2024-12-04T01:33:32.764Z [INFO]  server-ab379d9c-8cb3-6a05-5c22-0f0937e4f92f: election won: term=2 tally=2
2024-12-04T01:33:32.764Z [INFO]  server-ab379d9c-8cb3-6a05-5c22-0f0937e4f92f: entering leader state: leader="Node at ab379d9c-8cb3-6a05-5c22-0f0937e4f92f [Leader]"
2024-12-04T01:33:32.764Z [INFO]  server-ab379d9c-8cb3-6a05-5c22-0f0937e4f92f: added peer, starting replication: peer=server-ff3873c3-04b7-9015-9dfc-641d18432f09
2024-12-04T01:33:32.764Z [INFO]  server-ab379d9c-8cb3-6a05-5c22-0f0937e4f92f: added peer, starting replication: peer=server-d281cf0d-59b0-ec82-5174-cc8aaf1c94a7
2024-12-04T01:33:32.764Z [INFO]  server-ab379d9c-8cb3-6a05-5c22-0f0937e4f92f: pipelining replication: peer="{Voter server-d281cf0d-59b0-ec82-5174-cc8aaf1c94a7 d281cf0d-59b0-ec82-5174-cc8aaf1c94a7}"
2024-12-04T01:33:32.765Z [INFO]  server-ab379d9c-8cb3-6a05-5c22-0f0937e4f92f: pipelining replication: peer="{Voter server-ff3873c3-04b7-9015-9dfc-641d18432f09 ff3873c3-04b7-9015-9dfc-641d18432f09}"
2024-12-04T01:33:32.871Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:32.764710196 +0000 UTC m=+103.596552777 (1 nodes), 56.387384ms from start of poll, 56.843752ms from cluster start. Timeout at 2024-12-04 01:33:32.869717744 +0000 UTC m=+103.701560503, 105.007726ms after stability
2024-12-04T01:33:32.871Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:32.977Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:32.871163777 +0000 UTC m=+103.703006419 (1 nodes), 1.759µs from start of poll, 163.297394ms from cluster start. Timeout at 2024-12-04 01:33:32.976233542 +0000 UTC m=+103.808076232, 105.069813ms after stability
2024-12-04T01:33:32.978Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:33.084Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:32.978483681 +0000 UTC m=+103.810326608 (1 nodes), 2.297µs from start of poll, 270.617583ms from cluster start. Timeout at 2024-12-04 01:33:33.083575014 +0000 UTC m=+103.915417579, 105.090971ms after stability
2024-12-04T01:33:33.084Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:33.189Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:33.084478622 +0000 UTC m=+103.916321190 (1 nodes), 1.683µs from start of poll, 376.612165ms from cluster start. Timeout at 2024-12-04 01:33:33.189524083 +0000 UTC m=+104.021366672, 105.045482ms after stability
2024-12-04T01:33:33.189Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:33.295Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:33.189871878 +0000 UTC m=+104.021714448 (1 nodes), 1.69µs from start of poll, 482.005423ms from cluster start. Timeout at 2024-12-04 01:33:33.294945363 +0000 UTC m=+104.126787967, 105.073519ms after stability
2024-12-04T01:33:33.295Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:33.401Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:33.295355933 +0000 UTC m=+104.127198517 (1 nodes), 1.819µs from start of poll, 587.489492ms from cluster start. Timeout at 2024-12-04 01:33:33.40044754 +0000 UTC m=+104.232290122, 105.091605ms after stability
2024-12-04T01:33:33.508Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:33.613Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:33.508429954 +0000 UTC m=+104.340272512 (1 nodes), 1.512µs from start of poll, 800.563487ms from cluster start. Timeout at 2024-12-04 01:33:33.613503534 +0000 UTC m=+104.445346123, 105.073611ms after stability
2024-12-04T01:33:33.614Z [INFO]  server-ab379d9c-8cb3-6a05-5c22-0f0937e4f92f: aborting pipeline replication: peer="{Voter server-d281cf0d-59b0-ec82-5174-cc8aaf1c94a7 d281cf0d-59b0-ec82-5174-cc8aaf1c94a7}"
2024-12-04T01:33:33.614Z [INFO]  server-ab379d9c-8cb3-6a05-5c22-0f0937e4f92f: aborting pipeline replication: peer="{Voter server-ff3873c3-04b7-9015-9dfc-641d18432f09 ff3873c3-04b7-9015-9dfc-641d18432f09}"
--- PASS: TestRaft_LeaderID_Propagated (0.91s)
=== RUN   TestRaft_LeadershipTransferInProgress
--- PASS: TestRaft_LeadershipTransferInProgress (0.00s)
=== RUN   TestRaft_LeadershipTransferPickServer
--- PASS: TestRaft_LeadershipTransferPickServer (0.00s)
=== RUN   TestRaft_LeadershipTransfer
2024-12-04T01:33:33.616Z [INFO]  server-ea4f2430-d91e-c04c-1896-020dbad198ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea4f2430-d91e-c04c-1896-020dbad198ee Address:ea4f2430-d91e-c04c-1896-020dbad198ee} {Suffrage:Voter ID:server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27 Address:4d824add-bcdc-f4b4-c07e-ceeee7b24c27} {Suffrage:Voter ID:server-bce2cce7-a838-967a-474d-b484db440107 Address:bce2cce7-a838-967a-474d-b484db440107}]"
2024-12-04T01:33:33.616Z [INFO]  server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea4f2430-d91e-c04c-1896-020dbad198ee Address:ea4f2430-d91e-c04c-1896-020dbad198ee} {Suffrage:Voter ID:server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27 Address:4d824add-bcdc-f4b4-c07e-ceeee7b24c27} {Suffrage:Voter ID:server-bce2cce7-a838-967a-474d-b484db440107 Address:bce2cce7-a838-967a-474d-b484db440107}]"
2024-12-04T01:33:33.616Z [INFO]  server-bce2cce7-a838-967a-474d-b484db440107: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ea4f2430-d91e-c04c-1896-020dbad198ee Address:ea4f2430-d91e-c04c-1896-020dbad198ee} {Suffrage:Voter ID:server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27 Address:4d824add-bcdc-f4b4-c07e-ceeee7b24c27} {Suffrage:Voter ID:server-bce2cce7-a838-967a-474d-b484db440107 Address:bce2cce7-a838-967a-474d-b484db440107}]"
2024-12-04T01:33:33.616Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:33.616Z [INFO]  server-ea4f2430-d91e-c04c-1896-020dbad198ee: entering follower state: follower="Node at ea4f2430-d91e-c04c-1896-020dbad198ee [Follower]" leader-address= leader-id=
2024-12-04T01:33:33.616Z [INFO]  server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27: entering follower state: follower="Node at 4d824add-bcdc-f4b4-c07e-ceeee7b24c27 [Follower]" leader-address= leader-id=
2024-12-04T01:33:33.616Z [INFO]  server-bce2cce7-a838-967a-474d-b484db440107: entering follower state: follower="Node at bce2cce7-a838-967a-474d-b484db440107 [Follower]" leader-address= leader-id=
2024-12-04T01:33:33.681Z [WARN]  server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:33.681Z [INFO]  server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27: entering candidate state: node="Node at 4d824add-bcdc-f4b4-c07e-ceeee7b24c27 [Candidate]" term=2
2024-12-04T01:33:33.681Z [INFO]  server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27: election won: term=2 tally=2
2024-12-04T01:33:33.681Z [INFO]  server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27: entering leader state: leader="Node at 4d824add-bcdc-f4b4-c07e-ceeee7b24c27 [Leader]"
2024-12-04T01:33:33.681Z [INFO]  server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27: added peer, starting replication: peer=server-ea4f2430-d91e-c04c-1896-020dbad198ee
2024-12-04T01:33:33.681Z [INFO]  server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27: added peer, starting replication: peer=server-bce2cce7-a838-967a-474d-b484db440107
2024-12-04T01:33:33.681Z [INFO]  server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27: pipelining replication: peer="{Voter server-ea4f2430-d91e-c04c-1896-020dbad198ee ea4f2430-d91e-c04c-1896-020dbad198ee}"
2024-12-04T01:33:33.681Z [INFO]  server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27: pipelining replication: peer="{Voter server-bce2cce7-a838-967a-474d-b484db440107 bce2cce7-a838-967a-474d-b484db440107}"
2024-12-04T01:33:33.786Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:33.681594718 +0000 UTC m=+104.513437306 (1 nodes), 65.290089ms from start of poll, 65.663856ms from cluster start. Timeout at 2024-12-04 01:33:33.786601295 +0000 UTC m=+104.618443980, 105.006674ms after stability
2024-12-04T01:33:33.786Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:33.892Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:33.786772378 +0000 UTC m=+104.618615235 (1 nodes), 2.993µs from start of poll, 170.841785ms from cluster start. Timeout at 2024-12-04 01:33:33.891870096 +0000 UTC m=+104.723712866, 105.097631ms after stability
2024-12-04T01:33:33.892Z [INFO]  server-ea4f2430-d91e-c04c-1896-020dbad198ee: entering candidate state: node="Node at ea4f2430-d91e-c04c-1896-020dbad198ee [Candidate]" term=3
2024-12-04T01:33:33.892Z [INFO]  server-ea4f2430-d91e-c04c-1896-020dbad198ee: election won: term=3 tally=2
2024-12-04T01:33:33.892Z [INFO]  server-ea4f2430-d91e-c04c-1896-020dbad198ee: entering leader state: leader="Node at ea4f2430-d91e-c04c-1896-020dbad198ee [Leader]"
2024-12-04T01:33:33.892Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:33.892Z [INFO]  server-ea4f2430-d91e-c04c-1896-020dbad198ee: added peer, starting replication: peer=server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27
2024-12-04T01:33:33.892Z [INFO]  server-ea4f2430-d91e-c04c-1896-020dbad198ee: added peer, starting replication: peer=server-bce2cce7-a838-967a-474d-b484db440107
2024-12-04T01:33:33.893Z [INFO]  server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27: entering follower state: follower="Node at 4d824add-bcdc-f4b4-c07e-ceeee7b24c27 [Follower]" leader-address= leader-id=
2024-12-04T01:33:33.893Z [INFO]  server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27: aborting pipeline replication: peer="{Voter server-bce2cce7-a838-967a-474d-b484db440107 bce2cce7-a838-967a-474d-b484db440107}"
2024-12-04T01:33:33.893Z [INFO]  server-ea4f2430-d91e-c04c-1896-020dbad198ee: pipelining replication: peer="{Voter server-bce2cce7-a838-967a-474d-b484db440107 bce2cce7-a838-967a-474d-b484db440107}"
2024-12-04T01:33:33.893Z [INFO]  server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27: aborting pipeline replication: peer="{Voter server-ea4f2430-d91e-c04c-1896-020dbad198ee ea4f2430-d91e-c04c-1896-020dbad198ee}"
2024-12-04T01:33:33.893Z [INFO]  server-ea4f2430-d91e-c04c-1896-020dbad198ee: pipelining replication: peer="{Voter server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27 4d824add-bcdc-f4b4-c07e-ceeee7b24c27}"
2024-12-04T01:33:33.999Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:33.893245931 +0000 UTC m=+104.725088733 (1 nodes), 325.566µs from start of poll, 277.315283ms from cluster start. Timeout at 2024-12-04 01:33:33.998276679 +0000 UTC m=+104.830119259, 105.030526ms after stability
2024-12-04T01:33:33.999Z [INFO]  server-ea4f2430-d91e-c04c-1896-020dbad198ee: aborting pipeline replication: peer="{Voter server-bce2cce7-a838-967a-474d-b484db440107 bce2cce7-a838-967a-474d-b484db440107}"
2024-12-04T01:33:34.000Z [INFO]  server-ea4f2430-d91e-c04c-1896-020dbad198ee: aborting pipeline replication: peer="{Voter server-4d824add-bcdc-f4b4-c07e-ceeee7b24c27 4d824add-bcdc-f4b4-c07e-ceeee7b24c27}"
--- PASS: TestRaft_LeadershipTransfer (0.39s)
=== RUN   TestRaft_LeadershipTransferWithOneNode
2024-12-04T01:33:34.001Z [INFO]  server-b393b5c4-ece5-a4d9-eb78-0997722c991e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b393b5c4-ece5-a4d9-eb78-0997722c991e Address:b393b5c4-ece5-a4d9-eb78-0997722c991e}]"
2024-12-04T01:33:34.001Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:34.001Z [INFO]  server-b393b5c4-ece5-a4d9-eb78-0997722c991e: entering follower state: follower="Node at b393b5c4-ece5-a4d9-eb78-0997722c991e [Follower]" leader-address= leader-id=
2024-12-04T01:33:34.075Z [WARN]  server-b393b5c4-ece5-a4d9-eb78-0997722c991e: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:34.075Z [INFO]  server-b393b5c4-ece5-a4d9-eb78-0997722c991e: entering candidate state: node="Node at b393b5c4-ece5-a4d9-eb78-0997722c991e [Candidate]" term=2
2024-12-04T01:33:34.075Z [INFO]  server-b393b5c4-ece5-a4d9-eb78-0997722c991e: election won: term=2 tally=1
2024-12-04T01:33:34.075Z [INFO]  server-b393b5c4-ece5-a4d9-eb78-0997722c991e: entering leader state: leader="Node at b393b5c4-ece5-a4d9-eb78-0997722c991e [Leader]"
2024-12-04T01:33:34.181Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:34.075726426 +0000 UTC m=+104.907569093 (1 nodes), 74.136461ms from start of poll, 74.310398ms from cluster start. Timeout at 2024-12-04 01:33:34.180732936 +0000 UTC m=+105.012575499, 105.006406ms after stability
--- PASS: TestRaft_LeadershipTransferWithOneNode (0.18s)
=== RUN   TestRaft_LeadershipTransferWithSevenNodes
2024-12-04T01:33:34.183Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f Address:81873bbe-e50a-3ac7-4dcb-e0f390134b2f} {Suffrage:Voter ID:server-ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d Address:ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d} {Suffrage:Voter ID:server-212c403a-2d26-8ba3-575c-aa2772a9b425 Address:212c403a-2d26-8ba3-575c-aa2772a9b425} {Suffrage:Voter ID:server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2 Address:7e0be5aa-fae9-876c-3d61-f6b8723a9dc2} {Suffrage:Voter ID:server-7f547572-7c09-e574-11ee-2b7b398e3092 Address:7f547572-7c09-e574-11ee-2b7b398e3092} {Suffrage:Voter ID:server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c Address:9193b88b-e3a9-feaa-a678-5ae4ccdd709c} {Suffrage:Voter ID:server-e5ea8cee-33ee-c6a4-6f76-3d5d99194f40 Address:e5ea8cee-33ee-c6a4-6f76-3d5d99194f40}]"
2024-12-04T01:33:34.183Z [INFO]  server-ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f Address:81873bbe-e50a-3ac7-4dcb-e0f390134b2f} {Suffrage:Voter ID:server-ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d Address:ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d} {Suffrage:Voter ID:server-212c403a-2d26-8ba3-575c-aa2772a9b425 Address:212c403a-2d26-8ba3-575c-aa2772a9b425} {Suffrage:Voter ID:server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2 Address:7e0be5aa-fae9-876c-3d61-f6b8723a9dc2} {Suffrage:Voter ID:server-7f547572-7c09-e574-11ee-2b7b398e3092 Address:7f547572-7c09-e574-11ee-2b7b398e3092} {Suffrage:Voter ID:server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c Address:9193b88b-e3a9-feaa-a678-5ae4ccdd709c} {Suffrage:Voter ID:server-e5ea8cee-33ee-c6a4-6f76-3d5d99194f40 Address:e5ea8cee-33ee-c6a4-6f76-3d5d99194f40}]"
2024-12-04T01:33:34.183Z [INFO]  server-212c403a-2d26-8ba3-575c-aa2772a9b425: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f Address:81873bbe-e50a-3ac7-4dcb-e0f390134b2f} {Suffrage:Voter ID:server-ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d Address:ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d} {Suffrage:Voter ID:server-212c403a-2d26-8ba3-575c-aa2772a9b425 Address:212c403a-2d26-8ba3-575c-aa2772a9b425} {Suffrage:Voter ID:server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2 Address:7e0be5aa-fae9-876c-3d61-f6b8723a9dc2} {Suffrage:Voter ID:server-7f547572-7c09-e574-11ee-2b7b398e3092 Address:7f547572-7c09-e574-11ee-2b7b398e3092} {Suffrage:Voter ID:server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c Address:9193b88b-e3a9-feaa-a678-5ae4ccdd709c} {Suffrage:Voter ID:server-e5ea8cee-33ee-c6a4-6f76-3d5d99194f40 Address:e5ea8cee-33ee-c6a4-6f76-3d5d99194f40}]"
2024-12-04T01:33:34.183Z [INFO]  server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f Address:81873bbe-e50a-3ac7-4dcb-e0f390134b2f} {Suffrage:Voter ID:server-ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d Address:ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d} {Suffrage:Voter ID:server-212c403a-2d26-8ba3-575c-aa2772a9b425 Address:212c403a-2d26-8ba3-575c-aa2772a9b425} {Suffrage:Voter ID:server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2 Address:7e0be5aa-fae9-876c-3d61-f6b8723a9dc2} {Suffrage:Voter ID:server-7f547572-7c09-e574-11ee-2b7b398e3092 Address:7f547572-7c09-e574-11ee-2b7b398e3092} {Suffrage:Voter ID:server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c Address:9193b88b-e3a9-feaa-a678-5ae4ccdd709c} {Suffrage:Voter ID:server-e5ea8cee-33ee-c6a4-6f76-3d5d99194f40 Address:e5ea8cee-33ee-c6a4-6f76-3d5d99194f40}]"
2024-12-04T01:33:34.183Z [INFO]  server-7f547572-7c09-e574-11ee-2b7b398e3092: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f Address:81873bbe-e50a-3ac7-4dcb-e0f390134b2f} {Suffrage:Voter ID:server-ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d Address:ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d} {Suffrage:Voter ID:server-212c403a-2d26-8ba3-575c-aa2772a9b425 Address:212c403a-2d26-8ba3-575c-aa2772a9b425} {Suffrage:Voter ID:server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2 Address:7e0be5aa-fae9-876c-3d61-f6b8723a9dc2} {Suffrage:Voter ID:server-7f547572-7c09-e574-11ee-2b7b398e3092 Address:7f547572-7c09-e574-11ee-2b7b398e3092} {Suffrage:Voter ID:server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c Address:9193b88b-e3a9-feaa-a678-5ae4ccdd709c} {Suffrage:Voter ID:server-e5ea8cee-33ee-c6a4-6f76-3d5d99194f40 Address:e5ea8cee-33ee-c6a4-6f76-3d5d99194f40}]"
2024-12-04T01:33:34.183Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f Address:81873bbe-e50a-3ac7-4dcb-e0f390134b2f} {Suffrage:Voter ID:server-ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d Address:ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d} {Suffrage:Voter ID:server-212c403a-2d26-8ba3-575c-aa2772a9b425 Address:212c403a-2d26-8ba3-575c-aa2772a9b425} {Suffrage:Voter ID:server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2 Address:7e0be5aa-fae9-876c-3d61-f6b8723a9dc2} {Suffrage:Voter ID:server-7f547572-7c09-e574-11ee-2b7b398e3092 Address:7f547572-7c09-e574-11ee-2b7b398e3092} {Suffrage:Voter ID:server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c Address:9193b88b-e3a9-feaa-a678-5ae4ccdd709c} {Suffrage:Voter ID:server-e5ea8cee-33ee-c6a4-6f76-3d5d99194f40 Address:e5ea8cee-33ee-c6a4-6f76-3d5d99194f40}]"
2024-12-04T01:33:34.183Z [INFO]  server-e5ea8cee-33ee-c6a4-6f76-3d5d99194f40: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f Address:81873bbe-e50a-3ac7-4dcb-e0f390134b2f} {Suffrage:Voter ID:server-ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d Address:ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d} {Suffrage:Voter ID:server-212c403a-2d26-8ba3-575c-aa2772a9b425 Address:212c403a-2d26-8ba3-575c-aa2772a9b425} {Suffrage:Voter ID:server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2 Address:7e0be5aa-fae9-876c-3d61-f6b8723a9dc2} {Suffrage:Voter ID:server-7f547572-7c09-e574-11ee-2b7b398e3092 Address:7f547572-7c09-e574-11ee-2b7b398e3092} {Suffrage:Voter ID:server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c Address:9193b88b-e3a9-feaa-a678-5ae4ccdd709c} {Suffrage:Voter ID:server-e5ea8cee-33ee-c6a4-6f76-3d5d99194f40 Address:e5ea8cee-33ee-c6a4-6f76-3d5d99194f40}]"
2024-12-04T01:33:34.183Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: entering follower state: follower="Node at 81873bbe-e50a-3ac7-4dcb-e0f390134b2f [Follower]" leader-address= leader-id=
2024-12-04T01:33:34.184Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:34.184Z [INFO]  server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2: entering follower state: follower="Node at 7e0be5aa-fae9-876c-3d61-f6b8723a9dc2 [Follower]" leader-address= leader-id=
2024-12-04T01:33:34.184Z [INFO]  server-ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d: entering follower state: follower="Node at ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d [Follower]" leader-address= leader-id=
2024-12-04T01:33:34.184Z [INFO]  server-7f547572-7c09-e574-11ee-2b7b398e3092: entering follower state: follower="Node at 7f547572-7c09-e574-11ee-2b7b398e3092 [Follower]" leader-address= leader-id=
2024-12-04T01:33:34.184Z [INFO]  server-212c403a-2d26-8ba3-575c-aa2772a9b425: entering follower state: follower="Node at 212c403a-2d26-8ba3-575c-aa2772a9b425 [Follower]" leader-address= leader-id=
2024-12-04T01:33:34.184Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: entering follower state: follower="Node at 9193b88b-e3a9-feaa-a678-5ae4ccdd709c [Follower]" leader-address= leader-id=
2024-12-04T01:33:34.184Z [INFO]  server-e5ea8cee-33ee-c6a4-6f76-3d5d99194f40: entering follower state: follower="Node at e5ea8cee-33ee-c6a4-6f76-3d5d99194f40 [Follower]" leader-address= leader-id=
2024-12-04T01:33:34.238Z [WARN]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:34.238Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: entering candidate state: node="Node at 9193b88b-e3a9-feaa-a678-5ae4ccdd709c [Candidate]" term=2
2024-12-04T01:33:34.239Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: election won: term=2 tally=4
2024-12-04T01:33:34.239Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: entering leader state: leader="Node at 9193b88b-e3a9-feaa-a678-5ae4ccdd709c [Leader]"
2024-12-04T01:33:34.239Z [WARN]  server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:34.239Z [INFO]  server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2: entering candidate state: node="Node at 7e0be5aa-fae9-876c-3d61-f6b8723a9dc2 [Candidate]" term=2
2024-12-04T01:33:34.239Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: added peer, starting replication: peer=server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f
2024-12-04T01:33:34.239Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: added peer, starting replication: peer=server-ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d
2024-12-04T01:33:34.239Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: added peer, starting replication: peer=server-212c403a-2d26-8ba3-575c-aa2772a9b425
2024-12-04T01:33:34.239Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: added peer, starting replication: peer=server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2
2024-12-04T01:33:34.239Z [INFO]  server-e5ea8cee-33ee-c6a4-6f76-3d5d99194f40: duplicate requestVote for same term: term=2
2024-12-04T01:33:34.239Z [INFO]  server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2: entering follower state: follower="Node at 7e0be5aa-fae9-876c-3d61-f6b8723a9dc2 [Follower]" leader-address=9193b88b-e3a9-feaa-a678-5ae4ccdd709c leader-id=server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c
2024-12-04T01:33:34.239Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: duplicate requestVote for same term: term=2
2024-12-04T01:33:34.239Z [INFO]  server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2: duplicate requestVote for same term: term=2
2024-12-04T01:33:34.239Z [INFO]  server-ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d: duplicate requestVote for same term: term=2
2024-12-04T01:33:34.239Z [INFO]  server-212c403a-2d26-8ba3-575c-aa2772a9b425: duplicate requestVote for same term: term=2
2024-12-04T01:33:34.239Z [INFO]  server-7f547572-7c09-e574-11ee-2b7b398e3092: duplicate requestVote for same term: term=2
2024-12-04T01:33:34.239Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: added peer, starting replication: peer=server-7f547572-7c09-e574-11ee-2b7b398e3092
2024-12-04T01:33:34.239Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: added peer, starting replication: peer=server-e5ea8cee-33ee-c6a4-6f76-3d5d99194f40
2024-12-04T01:33:34.239Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: pipelining replication: peer="{Voter server-212c403a-2d26-8ba3-575c-aa2772a9b425 212c403a-2d26-8ba3-575c-aa2772a9b425}"
2024-12-04T01:33:34.239Z [WARN]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: rejecting vote request since we have a leader: from=7e0be5aa-fae9-876c-3d61-f6b8723a9dc2 leader=9193b88b-e3a9-feaa-a678-5ae4ccdd709c leader-id=server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c
2024-12-04T01:33:34.239Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: pipelining replication: peer="{Voter server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f 81873bbe-e50a-3ac7-4dcb-e0f390134b2f}"
2024-12-04T01:33:34.239Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: pipelining replication: peer="{Voter server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2 7e0be5aa-fae9-876c-3d61-f6b8723a9dc2}"
2024-12-04T01:33:34.240Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: pipelining replication: peer="{Voter server-7f547572-7c09-e574-11ee-2b7b398e3092 7f547572-7c09-e574-11ee-2b7b398e3092}"
2024-12-04T01:33:34.239Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: pipelining replication: peer="{Voter server-ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d}"
2024-12-04T01:33:34.240Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: pipelining replication: peer="{Voter server-e5ea8cee-33ee-c6a4-6f76-3d5d99194f40 e5ea8cee-33ee-c6a4-6f76-3d5d99194f40}"
2024-12-04T01:33:34.345Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:34.239952598 +0000 UTC m=+105.071795368 (1 nodes), 55.912106ms from start of poll, 57.008238ms from cluster start. Timeout at 2024-12-04 01:33:34.344959093 +0000 UTC m=+105.176801870, 105.006502ms after stability
2024-12-04T01:33:34.345Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:34.450Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:34.345081305 +0000 UTC m=+105.176923864 (6 nodes), 2.498µs from start of poll, 162.136734ms from cluster start. Timeout at 2024-12-04 01:33:34.450183072 +0000 UTC m=+105.282025821, 105.101957ms after stability
2024-12-04T01:33:34.450Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:34.557Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:34.450605829 +0000 UTC m=+105.282448744 (1 nodes), 3.353µs from start of poll, 267.661614ms from cluster start. Timeout at 2024-12-04 01:33:34.555675268 +0000 UTC m=+105.387518059, 105.069315ms after stability
2024-12-04T01:33:34.557Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: entering candidate state: node="Node at 81873bbe-e50a-3ac7-4dcb-e0f390134b2f [Candidate]" term=3
2024-12-04T01:33:34.557Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:34.557Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: entering follower state: follower="Node at 9193b88b-e3a9-feaa-a678-5ae4ccdd709c [Follower]" leader-address= leader-id=
2024-12-04T01:33:34.557Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: aborting pipeline replication: peer="{Voter server-e5ea8cee-33ee-c6a4-6f76-3d5d99194f40 e5ea8cee-33ee-c6a4-6f76-3d5d99194f40}"
2024-12-04T01:33:34.557Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: election won: term=3 tally=4
2024-12-04T01:33:34.557Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: entering leader state: leader="Node at 81873bbe-e50a-3ac7-4dcb-e0f390134b2f [Leader]"
2024-12-04T01:33:34.557Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: added peer, starting replication: peer=server-ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d
2024-12-04T01:33:34.557Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: added peer, starting replication: peer=server-212c403a-2d26-8ba3-575c-aa2772a9b425
2024-12-04T01:33:34.557Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: added peer, starting replication: peer=server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2
2024-12-04T01:33:34.557Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: added peer, starting replication: peer=server-7f547572-7c09-e574-11ee-2b7b398e3092
2024-12-04T01:33:34.557Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: added peer, starting replication: peer=server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c
2024-12-04T01:33:34.557Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: added peer, starting replication: peer=server-e5ea8cee-33ee-c6a4-6f76-3d5d99194f40
2024-12-04T01:33:34.557Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: pipelining replication: peer="{Voter server-e5ea8cee-33ee-c6a4-6f76-3d5d99194f40 e5ea8cee-33ee-c6a4-6f76-3d5d99194f40}"
2024-12-04T01:33:34.558Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: pipelining replication: peer="{Voter server-ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d}"
2024-12-04T01:33:34.558Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: pipelining replication: peer="{Voter server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2 7e0be5aa-fae9-876c-3d61-f6b8723a9dc2}"
2024-12-04T01:33:34.557Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: aborting pipeline replication: peer="{Voter server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f 81873bbe-e50a-3ac7-4dcb-e0f390134b2f}"
2024-12-04T01:33:34.558Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: pipelining replication: peer="{Voter server-7f547572-7c09-e574-11ee-2b7b398e3092 7f547572-7c09-e574-11ee-2b7b398e3092}"
2024-12-04T01:33:34.557Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: aborting pipeline replication: peer="{Voter server-ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d}"
2024-12-04T01:33:34.557Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: aborting pipeline replication: peer="{Voter server-212c403a-2d26-8ba3-575c-aa2772a9b425 212c403a-2d26-8ba3-575c-aa2772a9b425}"
2024-12-04T01:33:34.557Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: aborting pipeline replication: peer="{Voter server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2 7e0be5aa-fae9-876c-3d61-f6b8723a9dc2}"
2024-12-04T01:33:34.557Z [INFO]  server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c: aborting pipeline replication: peer="{Voter server-7f547572-7c09-e574-11ee-2b7b398e3092 7f547572-7c09-e574-11ee-2b7b398e3092}"
2024-12-04T01:33:34.558Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: pipelining replication: peer="{Voter server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c 9193b88b-e3a9-feaa-a678-5ae4ccdd709c}"
2024-12-04T01:33:34.558Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: pipelining replication: peer="{Voter server-212c403a-2d26-8ba3-575c-aa2772a9b425 212c403a-2d26-8ba3-575c-aa2772a9b425}"
2024-12-04T01:33:34.664Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:34.557807095 +0000 UTC m=+105.389649683 (1 nodes), 332.441µs from start of poll, 374.862553ms from cluster start. Timeout at 2024-12-04 01:33:34.662812567 +0000 UTC m=+105.494655442, 105.005759ms after stability
2024-12-04T01:33:34.664Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: aborting pipeline replication: peer="{Voter server-e5ea8cee-33ee-c6a4-6f76-3d5d99194f40 e5ea8cee-33ee-c6a4-6f76-3d5d99194f40}"
2024-12-04T01:33:34.664Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: aborting pipeline replication: peer="{Voter server-ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d ae823a66-c4b1-77bb-5ce9-c6f0b3cd067d}"
2024-12-04T01:33:34.664Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: aborting pipeline replication: peer="{Voter server-7e0be5aa-fae9-876c-3d61-f6b8723a9dc2 7e0be5aa-fae9-876c-3d61-f6b8723a9dc2}"
2024-12-04T01:33:34.664Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: aborting pipeline replication: peer="{Voter server-212c403a-2d26-8ba3-575c-aa2772a9b425 212c403a-2d26-8ba3-575c-aa2772a9b425}"
2024-12-04T01:33:34.664Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: aborting pipeline replication: peer="{Voter server-7f547572-7c09-e574-11ee-2b7b398e3092 7f547572-7c09-e574-11ee-2b7b398e3092}"
2024-12-04T01:33:34.664Z [INFO]  server-81873bbe-e50a-3ac7-4dcb-e0f390134b2f: aborting pipeline replication: peer="{Voter server-9193b88b-e3a9-feaa-a678-5ae4ccdd709c 9193b88b-e3a9-feaa-a678-5ae4ccdd709c}"
--- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.48s)
=== RUN   TestRaft_LeadershipTransferToInvalidID
2024-12-04T01:33:34.666Z [INFO]  server-172600e6-5798-05ef-18b0-c8c75991f02f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-172600e6-5798-05ef-18b0-c8c75991f02f Address:172600e6-5798-05ef-18b0-c8c75991f02f} {Suffrage:Voter ID:server-1bfa904a-456d-2801-7159-1e13837213cd Address:1bfa904a-456d-2801-7159-1e13837213cd} {Suffrage:Voter ID:server-b92a5b3e-137c-31ad-5314-57668580d56a Address:b92a5b3e-137c-31ad-5314-57668580d56a}]"
2024-12-04T01:33:34.666Z [INFO]  server-1bfa904a-456d-2801-7159-1e13837213cd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-172600e6-5798-05ef-18b0-c8c75991f02f Address:172600e6-5798-05ef-18b0-c8c75991f02f} {Suffrage:Voter ID:server-1bfa904a-456d-2801-7159-1e13837213cd Address:1bfa904a-456d-2801-7159-1e13837213cd} {Suffrage:Voter ID:server-b92a5b3e-137c-31ad-5314-57668580d56a Address:b92a5b3e-137c-31ad-5314-57668580d56a}]"
2024-12-04T01:33:34.666Z [INFO]  server-172600e6-5798-05ef-18b0-c8c75991f02f: entering follower state: follower="Node at 172600e6-5798-05ef-18b0-c8c75991f02f [Follower]" leader-address= leader-id=
2024-12-04T01:33:34.666Z [INFO]  server-b92a5b3e-137c-31ad-5314-57668580d56a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-172600e6-5798-05ef-18b0-c8c75991f02f Address:172600e6-5798-05ef-18b0-c8c75991f02f} {Suffrage:Voter ID:server-1bfa904a-456d-2801-7159-1e13837213cd Address:1bfa904a-456d-2801-7159-1e13837213cd} {Suffrage:Voter ID:server-b92a5b3e-137c-31ad-5314-57668580d56a Address:b92a5b3e-137c-31ad-5314-57668580d56a}]"
2024-12-04T01:33:34.666Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:34.666Z [INFO]  server-1bfa904a-456d-2801-7159-1e13837213cd: entering follower state: follower="Node at 1bfa904a-456d-2801-7159-1e13837213cd [Follower]" leader-address= leader-id=
2024-12-04T01:33:34.666Z [INFO]  server-b92a5b3e-137c-31ad-5314-57668580d56a: entering follower state: follower="Node at b92a5b3e-137c-31ad-5314-57668580d56a [Follower]" leader-address= leader-id=
2024-12-04T01:33:34.725Z [WARN]  server-172600e6-5798-05ef-18b0-c8c75991f02f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:34.725Z [INFO]  server-172600e6-5798-05ef-18b0-c8c75991f02f: entering candidate state: node="Node at 172600e6-5798-05ef-18b0-c8c75991f02f [Candidate]" term=2
2024-12-04T01:33:34.725Z [INFO]  server-172600e6-5798-05ef-18b0-c8c75991f02f: election won: term=2 tally=2
2024-12-04T01:33:34.725Z [INFO]  server-172600e6-5798-05ef-18b0-c8c75991f02f: entering leader state: leader="Node at 172600e6-5798-05ef-18b0-c8c75991f02f [Leader]"
2024-12-04T01:33:34.725Z [INFO]  server-172600e6-5798-05ef-18b0-c8c75991f02f: added peer, starting replication: peer=server-1bfa904a-456d-2801-7159-1e13837213cd
2024-12-04T01:33:34.725Z [INFO]  server-172600e6-5798-05ef-18b0-c8c75991f02f: added peer, starting replication: peer=server-b92a5b3e-137c-31ad-5314-57668580d56a
2024-12-04T01:33:34.725Z [INFO]  server-172600e6-5798-05ef-18b0-c8c75991f02f: pipelining replication: peer="{Voter server-b92a5b3e-137c-31ad-5314-57668580d56a b92a5b3e-137c-31ad-5314-57668580d56a}"
2024-12-04T01:33:34.725Z [INFO]  server-172600e6-5798-05ef-18b0-c8c75991f02f: pipelining replication: peer="{Voter server-1bfa904a-456d-2801-7159-1e13837213cd 1bfa904a-456d-2801-7159-1e13837213cd}"
2024-12-04T01:33:34.830Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:34.725510984 +0000 UTC m=+105.557353562 (1 nodes), 58.596565ms from start of poll, 58.986607ms from cluster start. Timeout at 2024-12-04 01:33:34.830520778 +0000 UTC m=+105.662363597, 105.010035ms after stability
2024-12-04T01:33:34.831Z [INFO]  server-172600e6-5798-05ef-18b0-c8c75991f02f: aborting pipeline replication: peer="{Voter server-b92a5b3e-137c-31ad-5314-57668580d56a b92a5b3e-137c-31ad-5314-57668580d56a}"
2024-12-04T01:33:34.831Z [INFO]  server-172600e6-5798-05ef-18b0-c8c75991f02f: aborting pipeline replication: peer="{Voter server-1bfa904a-456d-2801-7159-1e13837213cd 1bfa904a-456d-2801-7159-1e13837213cd}"
--- PASS: TestRaft_LeadershipTransferToInvalidID (0.17s)
=== RUN   TestRaft_LeadershipTransferToInvalidAddress
2024-12-04T01:33:34.832Z [INFO]  server-66a2c84b-faf4-9081-71a5-1e204c472dc9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66a2c84b-faf4-9081-71a5-1e204c472dc9 Address:66a2c84b-faf4-9081-71a5-1e204c472dc9} {Suffrage:Voter ID:server-7c4e8904-0f5c-b6b2-b4f2-960b48236428 Address:7c4e8904-0f5c-b6b2-b4f2-960b48236428} {Suffrage:Voter ID:server-f3b4950f-387a-219e-dd35-87376a2cacb5 Address:f3b4950f-387a-219e-dd35-87376a2cacb5}]"
2024-12-04T01:33:34.832Z [INFO]  server-7c4e8904-0f5c-b6b2-b4f2-960b48236428: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66a2c84b-faf4-9081-71a5-1e204c472dc9 Address:66a2c84b-faf4-9081-71a5-1e204c472dc9} {Suffrage:Voter ID:server-7c4e8904-0f5c-b6b2-b4f2-960b48236428 Address:7c4e8904-0f5c-b6b2-b4f2-960b48236428} {Suffrage:Voter ID:server-f3b4950f-387a-219e-dd35-87376a2cacb5 Address:f3b4950f-387a-219e-dd35-87376a2cacb5}]"
2024-12-04T01:33:34.832Z [INFO]  server-f3b4950f-387a-219e-dd35-87376a2cacb5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-66a2c84b-faf4-9081-71a5-1e204c472dc9 Address:66a2c84b-faf4-9081-71a5-1e204c472dc9} {Suffrage:Voter ID:server-7c4e8904-0f5c-b6b2-b4f2-960b48236428 Address:7c4e8904-0f5c-b6b2-b4f2-960b48236428} {Suffrage:Voter ID:server-f3b4950f-387a-219e-dd35-87376a2cacb5 Address:f3b4950f-387a-219e-dd35-87376a2cacb5}]"
2024-12-04T01:33:34.832Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:34.832Z [INFO]  server-66a2c84b-faf4-9081-71a5-1e204c472dc9: entering follower state: follower="Node at 66a2c84b-faf4-9081-71a5-1e204c472dc9 [Follower]" leader-address= leader-id=
2024-12-04T01:33:34.833Z [INFO]  server-7c4e8904-0f5c-b6b2-b4f2-960b48236428: entering follower state: follower="Node at 7c4e8904-0f5c-b6b2-b4f2-960b48236428 [Follower]" leader-address= leader-id=
2024-12-04T01:33:34.833Z [INFO]  server-f3b4950f-387a-219e-dd35-87376a2cacb5: entering follower state: follower="Node at f3b4950f-387a-219e-dd35-87376a2cacb5 [Follower]" leader-address= leader-id=
2024-12-04T01:33:34.891Z [WARN]  server-7c4e8904-0f5c-b6b2-b4f2-960b48236428: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:34.891Z [INFO]  server-7c4e8904-0f5c-b6b2-b4f2-960b48236428: entering candidate state: node="Node at 7c4e8904-0f5c-b6b2-b4f2-960b48236428 [Candidate]" term=2
2024-12-04T01:33:34.891Z [INFO]  server-7c4e8904-0f5c-b6b2-b4f2-960b48236428: election won: term=2 tally=2
2024-12-04T01:33:34.891Z [INFO]  server-7c4e8904-0f5c-b6b2-b4f2-960b48236428: entering leader state: leader="Node at 7c4e8904-0f5c-b6b2-b4f2-960b48236428 [Leader]"
2024-12-04T01:33:34.892Z [INFO]  server-7c4e8904-0f5c-b6b2-b4f2-960b48236428: added peer, starting replication: peer=server-66a2c84b-faf4-9081-71a5-1e204c472dc9
2024-12-04T01:33:34.892Z [INFO]  server-7c4e8904-0f5c-b6b2-b4f2-960b48236428: added peer, starting replication: peer=server-f3b4950f-387a-219e-dd35-87376a2cacb5
2024-12-04T01:33:34.892Z [INFO]  server-7c4e8904-0f5c-b6b2-b4f2-960b48236428: pipelining replication: peer="{Voter server-66a2c84b-faf4-9081-71a5-1e204c472dc9 66a2c84b-faf4-9081-71a5-1e204c472dc9}"
2024-12-04T01:33:34.892Z [INFO]  server-7c4e8904-0f5c-b6b2-b4f2-960b48236428: pipelining replication: peer="{Voter server-f3b4950f-387a-219e-dd35-87376a2cacb5 f3b4950f-387a-219e-dd35-87376a2cacb5}"
2024-12-04T01:33:34.997Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:34.892042957 +0000 UTC m=+105.723885526 (2 nodes), 59.086707ms from start of poll, 59.484802ms from cluster start. Timeout at 2024-12-04 01:33:34.997057177 +0000 UTC m=+105.828899884, 105.014358ms after stability
2024-12-04T01:33:34.997Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:35.102Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:34.997235021 +0000 UTC m=+105.829077651 (1 nodes), 2.124µs from start of poll, 164.676927ms from cluster start. Timeout at 2024-12-04 01:33:35.10229519 +0000 UTC m=+105.934137874, 105.060223ms after stability
2024-12-04T01:33:35.102Z [INFO]  server-7c4e8904-0f5c-b6b2-b4f2-960b48236428: aborting pipeline replication: peer="{Voter server-f3b4950f-387a-219e-dd35-87376a2cacb5 f3b4950f-387a-219e-dd35-87376a2cacb5}"
2024-12-04T01:33:35.102Z [INFO]  server-7c4e8904-0f5c-b6b2-b4f2-960b48236428: aborting pipeline replication: peer="{Voter server-66a2c84b-faf4-9081-71a5-1e204c472dc9 66a2c84b-faf4-9081-71a5-1e204c472dc9}"
--- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.27s)
=== RUN   TestRaft_LeadershipTransferToBehindServer
2024-12-04T01:33:35.104Z [INFO]  server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad Address:4ec72229-fa4d-155c-970b-bbf81ca7a3ad} {Suffrage:Voter ID:server-f86c9bcc-9853-54d8-bcdb-e84c3313178b Address:f86c9bcc-9853-54d8-bcdb-e84c3313178b} {Suffrage:Voter ID:server-c5b4e4e3-3391-6c13-a469-98e9a2b013e1 Address:c5b4e4e3-3391-6c13-a469-98e9a2b013e1}]"
2024-12-04T01:33:35.104Z [INFO]  server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad: entering follower state: follower="Node at 4ec72229-fa4d-155c-970b-bbf81ca7a3ad [Follower]" leader-address= leader-id=
2024-12-04T01:33:35.104Z [INFO]  server-f86c9bcc-9853-54d8-bcdb-e84c3313178b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad Address:4ec72229-fa4d-155c-970b-bbf81ca7a3ad} {Suffrage:Voter ID:server-f86c9bcc-9853-54d8-bcdb-e84c3313178b Address:f86c9bcc-9853-54d8-bcdb-e84c3313178b} {Suffrage:Voter ID:server-c5b4e4e3-3391-6c13-a469-98e9a2b013e1 Address:c5b4e4e3-3391-6c13-a469-98e9a2b013e1}]"
2024-12-04T01:33:35.104Z [INFO]  server-c5b4e4e3-3391-6c13-a469-98e9a2b013e1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad Address:4ec72229-fa4d-155c-970b-bbf81ca7a3ad} {Suffrage:Voter ID:server-f86c9bcc-9853-54d8-bcdb-e84c3313178b Address:f86c9bcc-9853-54d8-bcdb-e84c3313178b} {Suffrage:Voter ID:server-c5b4e4e3-3391-6c13-a469-98e9a2b013e1 Address:c5b4e4e3-3391-6c13-a469-98e9a2b013e1}]"
2024-12-04T01:33:35.104Z [INFO]  server-f86c9bcc-9853-54d8-bcdb-e84c3313178b: entering follower state: follower="Node at f86c9bcc-9853-54d8-bcdb-e84c3313178b [Follower]" leader-address= leader-id=
2024-12-04T01:33:35.104Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:35.104Z [INFO]  server-c5b4e4e3-3391-6c13-a469-98e9a2b013e1: entering follower state: follower="Node at c5b4e4e3-3391-6c13-a469-98e9a2b013e1 [Follower]" leader-address= leader-id=
2024-12-04T01:33:35.156Z [WARN]  server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:35.156Z [INFO]  server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad: entering candidate state: node="Node at 4ec72229-fa4d-155c-970b-bbf81ca7a3ad [Candidate]" term=2
2024-12-04T01:33:35.157Z [INFO]  server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad: election won: term=2 tally=2
2024-12-04T01:33:35.157Z [INFO]  server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad: entering leader state: leader="Node at 4ec72229-fa4d-155c-970b-bbf81ca7a3ad [Leader]"
2024-12-04T01:33:35.157Z [INFO]  server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad: added peer, starting replication: peer=server-f86c9bcc-9853-54d8-bcdb-e84c3313178b
2024-12-04T01:33:35.157Z [INFO]  server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad: added peer, starting replication: peer=server-c5b4e4e3-3391-6c13-a469-98e9a2b013e1
2024-12-04T01:33:35.157Z [INFO]  server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad: pipelining replication: peer="{Voter server-c5b4e4e3-3391-6c13-a469-98e9a2b013e1 c5b4e4e3-3391-6c13-a469-98e9a2b013e1}"
2024-12-04T01:33:35.157Z [INFO]  server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad: pipelining replication: peer="{Voter server-f86c9bcc-9853-54d8-bcdb-e84c3313178b f86c9bcc-9853-54d8-bcdb-e84c3313178b}"
2024-12-04T01:33:35.262Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:35.157301038 +0000 UTC m=+105.989143939 (1 nodes), 52.748143ms from start of poll, 53.230282ms from cluster start. Timeout at 2024-12-04 01:33:35.26231521 +0000 UTC m=+106.094157845, 105.013906ms after stability
2024-12-04T01:33:35.263Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:35.369Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:35.263060123 +0000 UTC m=+106.094902830 (2 nodes), 2.445µs from start of poll, 158.989173ms from cluster start. Timeout at 2024-12-04 01:33:35.368132408 +0000 UTC m=+106.199975328, 105.072498ms after stability
2024-12-04T01:33:35.496Z [INFO]  server-f86c9bcc-9853-54d8-bcdb-e84c3313178b: entering candidate state: node="Node at f86c9bcc-9853-54d8-bcdb-e84c3313178b [Candidate]" term=3
2024-12-04T01:33:35.496Z [INFO]  server-f86c9bcc-9853-54d8-bcdb-e84c3313178b: election won: term=3 tally=2
2024-12-04T01:33:35.496Z [INFO]  server-f86c9bcc-9853-54d8-bcdb-e84c3313178b: entering leader state: leader="Node at f86c9bcc-9853-54d8-bcdb-e84c3313178b [Leader]"
2024-12-04T01:33:35.496Z [INFO]  server-f86c9bcc-9853-54d8-bcdb-e84c3313178b: added peer, starting replication: peer=server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad
2024-12-04T01:33:35.496Z [INFO]  server-f86c9bcc-9853-54d8-bcdb-e84c3313178b: added peer, starting replication: peer=server-c5b4e4e3-3391-6c13-a469-98e9a2b013e1
2024-12-04T01:33:35.496Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:35.497Z [INFO]  server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad: entering follower state: follower="Node at 4ec72229-fa4d-155c-970b-bbf81ca7a3ad [Follower]" leader-address= leader-id=
2024-12-04T01:33:35.497Z [INFO]  server-f86c9bcc-9853-54d8-bcdb-e84c3313178b: pipelining replication: peer="{Voter server-c5b4e4e3-3391-6c13-a469-98e9a2b013e1 c5b4e4e3-3391-6c13-a469-98e9a2b013e1}"
2024-12-04T01:33:35.497Z [INFO]  server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad: aborting pipeline replication: peer="{Voter server-f86c9bcc-9853-54d8-bcdb-e84c3313178b f86c9bcc-9853-54d8-bcdb-e84c3313178b}"
2024-12-04T01:33:35.497Z [INFO]  server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad: aborting pipeline replication: peer="{Voter server-c5b4e4e3-3391-6c13-a469-98e9a2b013e1 c5b4e4e3-3391-6c13-a469-98e9a2b013e1}"
2024-12-04T01:33:35.497Z [INFO]  server-f86c9bcc-9853-54d8-bcdb-e84c3313178b: pipelining replication: peer="{Voter server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad 4ec72229-fa4d-155c-970b-bbf81ca7a3ad}"
2024-12-04T01:33:35.602Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:35.497256625 +0000 UTC m=+106.329099194 (1 nodes), 220.695µs from start of poll, 393.185537ms from cluster start. Timeout at 2024-12-04 01:33:35.60226407 +0000 UTC m=+106.434106835, 105.007641ms after stability
2024-12-04T01:33:35.602Z [INFO]  server-f86c9bcc-9853-54d8-bcdb-e84c3313178b: aborting pipeline replication: peer="{Voter server-c5b4e4e3-3391-6c13-a469-98e9a2b013e1 c5b4e4e3-3391-6c13-a469-98e9a2b013e1}"
2024-12-04T01:33:35.602Z [INFO]  server-f86c9bcc-9853-54d8-bcdb-e84c3313178b: aborting pipeline replication: peer="{Voter server-4ec72229-fa4d-155c-970b-bbf81ca7a3ad 4ec72229-fa4d-155c-970b-bbf81ca7a3ad}"
--- PASS: TestRaft_LeadershipTransferToBehindServer (0.50s)
=== RUN   TestRaft_LeadershipTransferToItself
2024-12-04T01:33:35.604Z [INFO]  server-781927ab-97da-724a-6d48-5117fe7e3476: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-781927ab-97da-724a-6d48-5117fe7e3476 Address:781927ab-97da-724a-6d48-5117fe7e3476} {Suffrage:Voter ID:server-cde6d845-93b6-e4d5-648f-49b02657c612 Address:cde6d845-93b6-e4d5-648f-49b02657c612} {Suffrage:Voter ID:server-ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63 Address:ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63}]"
2024-12-04T01:33:35.604Z [INFO]  server-cde6d845-93b6-e4d5-648f-49b02657c612: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-781927ab-97da-724a-6d48-5117fe7e3476 Address:781927ab-97da-724a-6d48-5117fe7e3476} {Suffrage:Voter ID:server-cde6d845-93b6-e4d5-648f-49b02657c612 Address:cde6d845-93b6-e4d5-648f-49b02657c612} {Suffrage:Voter ID:server-ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63 Address:ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63}]"
2024-12-04T01:33:35.604Z [INFO]  server-781927ab-97da-724a-6d48-5117fe7e3476: entering follower state: follower="Node at 781927ab-97da-724a-6d48-5117fe7e3476 [Follower]" leader-address= leader-id=
2024-12-04T01:33:35.604Z [INFO]  server-cde6d845-93b6-e4d5-648f-49b02657c612: entering follower state: follower="Node at cde6d845-93b6-e4d5-648f-49b02657c612 [Follower]" leader-address= leader-id=
2024-12-04T01:33:35.604Z [INFO]  server-ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-781927ab-97da-724a-6d48-5117fe7e3476 Address:781927ab-97da-724a-6d48-5117fe7e3476} {Suffrage:Voter ID:server-cde6d845-93b6-e4d5-648f-49b02657c612 Address:cde6d845-93b6-e4d5-648f-49b02657c612} {Suffrage:Voter ID:server-ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63 Address:ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63}]"
2024-12-04T01:33:35.604Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:35.604Z [INFO]  server-ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63: entering follower state: follower="Node at ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63 [Follower]" leader-address= leader-id=
2024-12-04T01:33:35.664Z [WARN]  server-ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:35.664Z [INFO]  server-ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63: entering candidate state: node="Node at ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63 [Candidate]" term=2
2024-12-04T01:33:35.664Z [INFO]  server-ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63: election won: term=2 tally=2
2024-12-04T01:33:35.664Z [INFO]  server-ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63: entering leader state: leader="Node at ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63 [Leader]"
2024-12-04T01:33:35.664Z [INFO]  server-ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63: added peer, starting replication: peer=server-781927ab-97da-724a-6d48-5117fe7e3476
2024-12-04T01:33:35.664Z [INFO]  server-ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63: added peer, starting replication: peer=server-cde6d845-93b6-e4d5-648f-49b02657c612
2024-12-04T01:33:35.664Z [INFO]  server-ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63: pipelining replication: peer="{Voter server-cde6d845-93b6-e4d5-648f-49b02657c612 cde6d845-93b6-e4d5-648f-49b02657c612}"
2024-12-04T01:33:35.664Z [INFO]  server-ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63: pipelining replication: peer="{Voter server-781927ab-97da-724a-6d48-5117fe7e3476 781927ab-97da-724a-6d48-5117fe7e3476}"
2024-12-04T01:33:35.770Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:35.664677413 +0000 UTC m=+106.496520280 (1 nodes), 60.207772ms from start of poll, 60.692391ms from cluster start. Timeout at 2024-12-04 01:33:35.769716354 +0000 UTC m=+106.601558920, 105.03864ms after stability
2024-12-04T01:33:35.770Z [INFO]  server-ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63: cannot transfer leadership to itself
2024-12-04T01:33:35.770Z [INFO]  server-ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63: aborting pipeline replication: peer="{Voter server-cde6d845-93b6-e4d5-648f-49b02657c612 cde6d845-93b6-e4d5-648f-49b02657c612}"
2024-12-04T01:33:35.770Z [INFO]  server-ec7d6f6f-25ca-0eef-1814-8b26ffd4fb63: aborting pipeline replication: peer="{Voter server-781927ab-97da-724a-6d48-5117fe7e3476 781927ab-97da-724a-6d48-5117fe7e3476}"
--- PASS: TestRaft_LeadershipTransferToItself (0.17s)
=== RUN   TestRaft_LeadershipTransferLeaderRejectsClientRequests
2024-12-04T01:33:35.772Z [INFO]  server-30ab036e-0410-672d-33fc-755e14c7f10c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30ab036e-0410-672d-33fc-755e14c7f10c Address:30ab036e-0410-672d-33fc-755e14c7f10c} {Suffrage:Voter ID:server-564a1172-ed72-ce27-f1ba-3f72460a8662 Address:564a1172-ed72-ce27-f1ba-3f72460a8662} {Suffrage:Voter ID:server-7b5af205-cdf7-1f5c-40e6-5635980a5887 Address:7b5af205-cdf7-1f5c-40e6-5635980a5887}]"
2024-12-04T01:33:35.772Z [INFO]  server-564a1172-ed72-ce27-f1ba-3f72460a8662: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30ab036e-0410-672d-33fc-755e14c7f10c Address:30ab036e-0410-672d-33fc-755e14c7f10c} {Suffrage:Voter ID:server-564a1172-ed72-ce27-f1ba-3f72460a8662 Address:564a1172-ed72-ce27-f1ba-3f72460a8662} {Suffrage:Voter ID:server-7b5af205-cdf7-1f5c-40e6-5635980a5887 Address:7b5af205-cdf7-1f5c-40e6-5635980a5887}]"
2024-12-04T01:33:35.772Z [INFO]  server-7b5af205-cdf7-1f5c-40e6-5635980a5887: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-30ab036e-0410-672d-33fc-755e14c7f10c Address:30ab036e-0410-672d-33fc-755e14c7f10c} {Suffrage:Voter ID:server-564a1172-ed72-ce27-f1ba-3f72460a8662 Address:564a1172-ed72-ce27-f1ba-3f72460a8662} {Suffrage:Voter ID:server-7b5af205-cdf7-1f5c-40e6-5635980a5887 Address:7b5af205-cdf7-1f5c-40e6-5635980a5887}]"
2024-12-04T01:33:35.772Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:35.772Z [INFO]  server-30ab036e-0410-672d-33fc-755e14c7f10c: entering follower state: follower="Node at 30ab036e-0410-672d-33fc-755e14c7f10c [Follower]" leader-address= leader-id=
2024-12-04T01:33:35.772Z [INFO]  server-7b5af205-cdf7-1f5c-40e6-5635980a5887: entering follower state: follower="Node at 7b5af205-cdf7-1f5c-40e6-5635980a5887 [Follower]" leader-address= leader-id=
2024-12-04T01:33:35.772Z [INFO]  server-564a1172-ed72-ce27-f1ba-3f72460a8662: entering follower state: follower="Node at 564a1172-ed72-ce27-f1ba-3f72460a8662 [Follower]" leader-address= leader-id=
2024-12-04T01:33:35.824Z [WARN]  server-30ab036e-0410-672d-33fc-755e14c7f10c: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:35.824Z [INFO]  server-30ab036e-0410-672d-33fc-755e14c7f10c: entering candidate state: node="Node at 30ab036e-0410-672d-33fc-755e14c7f10c [Candidate]" term=2
2024-12-04T01:33:35.824Z [INFO]  server-30ab036e-0410-672d-33fc-755e14c7f10c: election won: term=2 tally=2
2024-12-04T01:33:35.824Z [INFO]  server-30ab036e-0410-672d-33fc-755e14c7f10c: entering leader state: leader="Node at 30ab036e-0410-672d-33fc-755e14c7f10c [Leader]"
2024-12-04T01:33:35.824Z [INFO]  server-30ab036e-0410-672d-33fc-755e14c7f10c: added peer, starting replication: peer=server-564a1172-ed72-ce27-f1ba-3f72460a8662
2024-12-04T01:33:35.824Z [INFO]  server-30ab036e-0410-672d-33fc-755e14c7f10c: added peer, starting replication: peer=server-7b5af205-cdf7-1f5c-40e6-5635980a5887
2024-12-04T01:33:35.824Z [INFO]  server-30ab036e-0410-672d-33fc-755e14c7f10c: pipelining replication: peer="{Voter server-7b5af205-cdf7-1f5c-40e6-5635980a5887 7b5af205-cdf7-1f5c-40e6-5635980a5887}"
2024-12-04T01:33:35.825Z [INFO]  server-30ab036e-0410-672d-33fc-755e14c7f10c: pipelining replication: peer="{Voter server-564a1172-ed72-ce27-f1ba-3f72460a8662 564a1172-ed72-ce27-f1ba-3f72460a8662}"
2024-12-04T01:33:35.930Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:35.825359525 +0000 UTC m=+106.657202298 (1 nodes), 52.770571ms from start of poll, 53.214657ms from cluster start. Timeout at 2024-12-04 01:33:35.930397717 +0000 UTC m=+106.762240283, 105.037985ms after stability
    raft_test.go:2457: waiting on future 0
    raft_test.go:2457: waiting on future 1
    raft_test.go:2457: waiting on future 2
    raft_test.go:2457: waiting on future 3
    raft_test.go:2457: waiting on future 4
    raft_test.go:2457: waiting on future 5
    raft_test.go:2457: waiting on future 6
2024-12-04T01:33:35.930Z [INFO]  server-30ab036e-0410-672d-33fc-755e14c7f10c: aborting pipeline replication: peer="{Voter server-7b5af205-cdf7-1f5c-40e6-5635980a5887 7b5af205-cdf7-1f5c-40e6-5635980a5887}"
2024-12-04T01:33:35.931Z [INFO]  server-30ab036e-0410-672d-33fc-755e14c7f10c: aborting pipeline replication: peer="{Voter server-564a1172-ed72-ce27-f1ba-3f72460a8662 564a1172-ed72-ce27-f1ba-3f72460a8662}"
--- PASS: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.16s)
=== RUN   TestRaft_LeadershipTransferLeaderReplicationTimeout
2024-12-04T01:33:35.932Z [INFO]  server-b6f6a4d3-363a-58a9-8faf-66aa0da00872: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6f6a4d3-363a-58a9-8faf-66aa0da00872 Address:b6f6a4d3-363a-58a9-8faf-66aa0da00872} {Suffrage:Voter ID:server-e6ae6dd1-ef97-f4e1-27e0-8c6fbec95171 Address:e6ae6dd1-ef97-f4e1-27e0-8c6fbec95171} {Suffrage:Voter ID:server-673bd8b3-f024-5a1d-7133-b20c71fcdaea Address:673bd8b3-f024-5a1d-7133-b20c71fcdaea}]"
2024-12-04T01:33:35.932Z [INFO]  server-e6ae6dd1-ef97-f4e1-27e0-8c6fbec95171: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6f6a4d3-363a-58a9-8faf-66aa0da00872 Address:b6f6a4d3-363a-58a9-8faf-66aa0da00872} {Suffrage:Voter ID:server-e6ae6dd1-ef97-f4e1-27e0-8c6fbec95171 Address:e6ae6dd1-ef97-f4e1-27e0-8c6fbec95171} {Suffrage:Voter ID:server-673bd8b3-f024-5a1d-7133-b20c71fcdaea Address:673bd8b3-f024-5a1d-7133-b20c71fcdaea}]"
2024-12-04T01:33:35.932Z [INFO]  server-b6f6a4d3-363a-58a9-8faf-66aa0da00872: entering follower state: follower="Node at b6f6a4d3-363a-58a9-8faf-66aa0da00872 [Follower]" leader-address= leader-id=
2024-12-04T01:33:35.932Z [INFO]  server-e6ae6dd1-ef97-f4e1-27e0-8c6fbec95171: entering follower state: follower="Node at e6ae6dd1-ef97-f4e1-27e0-8c6fbec95171 [Follower]" leader-address= leader-id=
2024-12-04T01:33:35.932Z [INFO]  server-673bd8b3-f024-5a1d-7133-b20c71fcdaea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b6f6a4d3-363a-58a9-8faf-66aa0da00872 Address:b6f6a4d3-363a-58a9-8faf-66aa0da00872} {Suffrage:Voter ID:server-e6ae6dd1-ef97-f4e1-27e0-8c6fbec95171 Address:e6ae6dd1-ef97-f4e1-27e0-8c6fbec95171} {Suffrage:Voter ID:server-673bd8b3-f024-5a1d-7133-b20c71fcdaea Address:673bd8b3-f024-5a1d-7133-b20c71fcdaea}]"
2024-12-04T01:33:35.932Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:35.932Z [INFO]  server-673bd8b3-f024-5a1d-7133-b20c71fcdaea: entering follower state: follower="Node at 673bd8b3-f024-5a1d-7133-b20c71fcdaea [Follower]" leader-address= leader-id=
2024-12-04T01:33:35.996Z [WARN]  server-673bd8b3-f024-5a1d-7133-b20c71fcdaea: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:35.996Z [INFO]  server-673bd8b3-f024-5a1d-7133-b20c71fcdaea: entering candidate state: node="Node at 673bd8b3-f024-5a1d-7133-b20c71fcdaea [Candidate]" term=2
2024-12-04T01:33:35.997Z [INFO]  server-673bd8b3-f024-5a1d-7133-b20c71fcdaea: election won: term=2 tally=2
2024-12-04T01:33:35.997Z [INFO]  server-673bd8b3-f024-5a1d-7133-b20c71fcdaea: entering leader state: leader="Node at 673bd8b3-f024-5a1d-7133-b20c71fcdaea [Leader]"
2024-12-04T01:33:35.997Z [INFO]  server-673bd8b3-f024-5a1d-7133-b20c71fcdaea: added peer, starting replication: peer=server-b6f6a4d3-363a-58a9-8faf-66aa0da00872
2024-12-04T01:33:35.997Z [INFO]  server-673bd8b3-f024-5a1d-7133-b20c71fcdaea: added peer, starting replication: peer=server-e6ae6dd1-ef97-f4e1-27e0-8c6fbec95171
2024-12-04T01:33:35.997Z [INFO]  server-673bd8b3-f024-5a1d-7133-b20c71fcdaea: pipelining replication: peer="{Voter server-e6ae6dd1-ef97-f4e1-27e0-8c6fbec95171 e6ae6dd1-ef97-f4e1-27e0-8c6fbec95171}"
2024-12-04T01:33:35.997Z [INFO]  server-673bd8b3-f024-5a1d-7133-b20c71fcdaea: pipelining replication: peer="{Voter server-b6f6a4d3-363a-58a9-8faf-66aa0da00872 b6f6a4d3-363a-58a9-8faf-66aa0da00872}"
2024-12-04T01:33:36.103Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:35.997110499 +0000 UTC m=+106.828953088 (1 nodes), 64.30664ms from start of poll, 64.70595ms from cluster start. Timeout at 2024-12-04 01:33:36.102130536 +0000 UTC m=+106.933973323, 105.020235ms after stability
2024-12-04T01:33:36.103Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:36.209Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:36.103288117 +0000 UTC m=+106.935130888 (2 nodes), 2.486µs from start of poll, 170.88375ms from cluster start. Timeout at 2024-12-04 01:33:36.208341979 +0000 UTC m=+107.040184642, 105.053754ms after stability
2024-12-04T01:33:37.818Z [INFO]  server-673bd8b3-f024-5a1d-7133-b20c71fcdaea: aborting pipeline replication: peer="{Voter server-b6f6a4d3-363a-58a9-8faf-66aa0da00872 b6f6a4d3-363a-58a9-8faf-66aa0da00872}"
2024-12-04T01:33:37.818Z [INFO]  server-673bd8b3-f024-5a1d-7133-b20c71fcdaea: aborting pipeline replication: peer="{Voter server-e6ae6dd1-ef97-f4e1-27e0-8c6fbec95171 e6ae6dd1-ef97-f4e1-27e0-8c6fbec95171}"
2024-12-04T01:33:38.320Z [ERROR] server-673bd8b3-f024-5a1d-7133-b20c71fcdaea: failed to heartbeat to: peer=e6ae6dd1-ef97-f4e1-27e0-8c6fbec95171 backoff time=10ms error="command timed out"
--- PASS: TestRaft_LeadershipTransferLeaderReplicationTimeout (2.39s)
=== RUN   TestRaft_LeadershipTransferIgnoresNonvoters
2024-12-04T01:33:38.322Z [INFO]  server-00557cdb-e86f-5906-7ccf-feaccd8c28b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00557cdb-e86f-5906-7ccf-feaccd8c28b4 Address:00557cdb-e86f-5906-7ccf-feaccd8c28b4} {Suffrage:Voter ID:server-8307b409-6fa8-b2df-d8a7-4de4ffc84c4d Address:8307b409-6fa8-b2df-d8a7-4de4ffc84c4d}]"
2024-12-04T01:33:38.322Z [INFO]  server-8307b409-6fa8-b2df-d8a7-4de4ffc84c4d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-00557cdb-e86f-5906-7ccf-feaccd8c28b4 Address:00557cdb-e86f-5906-7ccf-feaccd8c28b4} {Suffrage:Voter ID:server-8307b409-6fa8-b2df-d8a7-4de4ffc84c4d Address:8307b409-6fa8-b2df-d8a7-4de4ffc84c4d}]"
2024-12-04T01:33:38.322Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:38.322Z [INFO]  server-00557cdb-e86f-5906-7ccf-feaccd8c28b4: entering follower state: follower="Node at 00557cdb-e86f-5906-7ccf-feaccd8c28b4 [Follower]" leader-address= leader-id=
2024-12-04T01:33:38.322Z [INFO]  server-8307b409-6fa8-b2df-d8a7-4de4ffc84c4d: entering follower state: follower="Node at 8307b409-6fa8-b2df-d8a7-4de4ffc84c4d [Follower]" leader-address= leader-id=
2024-12-04T01:33:38.418Z [WARN]  server-8307b409-6fa8-b2df-d8a7-4de4ffc84c4d: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:38.418Z [INFO]  server-8307b409-6fa8-b2df-d8a7-4de4ffc84c4d: entering candidate state: node="Node at 8307b409-6fa8-b2df-d8a7-4de4ffc84c4d [Candidate]" term=2
2024-12-04T01:33:38.419Z [INFO]  server-8307b409-6fa8-b2df-d8a7-4de4ffc84c4d: election won: term=2 tally=2
2024-12-04T01:33:38.419Z [INFO]  server-8307b409-6fa8-b2df-d8a7-4de4ffc84c4d: entering leader state: leader="Node at 8307b409-6fa8-b2df-d8a7-4de4ffc84c4d [Leader]"
2024-12-04T01:33:38.419Z [INFO]  server-8307b409-6fa8-b2df-d8a7-4de4ffc84c4d: added peer, starting replication: peer=server-00557cdb-e86f-5906-7ccf-feaccd8c28b4
2024-12-04T01:33:38.419Z [INFO]  server-8307b409-6fa8-b2df-d8a7-4de4ffc84c4d: pipelining replication: peer="{Voter server-00557cdb-e86f-5906-7ccf-feaccd8c28b4 00557cdb-e86f-5906-7ccf-feaccd8c28b4}"
2024-12-04T01:33:38.527Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:38.419799557 +0000 UTC m=+109.251642198 (1 nodes), 96.928595ms from start of poll, 97.286155ms from cluster start. Timeout at 2024-12-04 01:33:38.524803575 +0000 UTC m=+109.356646231, 105.004033ms after stability
2024-12-04T01:33:38.527Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:38.633Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:38.527896723 +0000 UTC m=+109.359739372 (1 nodes), 1.907µs from start of poll, 205.383329ms from cluster start. Timeout at 2024-12-04 01:33:38.632970944 +0000 UTC m=+109.464813694, 105.074322ms after stability
2024-12-04T01:33:38.633Z [INFO]  server-8307b409-6fa8-b2df-d8a7-4de4ffc84c4d: updating configuration: command=DemoteVoter server-id=server-00557cdb-e86f-5906-7ccf-feaccd8c28b4 server-addr= servers="[{Suffrage:Nonvoter ID:server-00557cdb-e86f-5906-7ccf-feaccd8c28b4 Address:00557cdb-e86f-5906-7ccf-feaccd8c28b4} {Suffrage:Voter ID:server-8307b409-6fa8-b2df-d8a7-4de4ffc84c4d Address:8307b409-6fa8-b2df-d8a7-4de4ffc84c4d}]"
2024-12-04T01:33:38.633Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:38.747Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:38.633359977 +0000 UTC m=+109.465202629 (1 nodes), 2.098µs from start of poll, 310.846586ms from cluster start. Timeout at 2024-12-04 01:33:38.738490815 +0000 UTC m=+109.570333620, 105.130991ms after stability
2024-12-04T01:33:38.748Z [INFO]  server-8307b409-6fa8-b2df-d8a7-4de4ffc84c4d: aborting pipeline replication: peer="{Voter server-00557cdb-e86f-5906-7ccf-feaccd8c28b4 00557cdb-e86f-5906-7ccf-feaccd8c28b4}"
--- PASS: TestRaft_LeadershipTransferIgnoresNonvoters (0.43s)
=== RUN   TestRaft_LeadershipTransferStopRightAway
--- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s)
=== RUN   TestRaft_GetConfigurationNoBootstrap
2024-12-04T01:33:38.750Z [INFO]  server-c2a130ea-5fac-6257-751a-148341fb98cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2a130ea-5fac-6257-751a-148341fb98cc Address:c2a130ea-5fac-6257-751a-148341fb98cc} {Suffrage:Voter ID:server-95f86859-a8ed-8d33-bb57-50f6dc7b6bf1 Address:95f86859-a8ed-8d33-bb57-50f6dc7b6bf1}]"
2024-12-04T01:33:38.750Z [INFO]  server-95f86859-a8ed-8d33-bb57-50f6dc7b6bf1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2a130ea-5fac-6257-751a-148341fb98cc Address:c2a130ea-5fac-6257-751a-148341fb98cc} {Suffrage:Voter ID:server-95f86859-a8ed-8d33-bb57-50f6dc7b6bf1 Address:95f86859-a8ed-8d33-bb57-50f6dc7b6bf1}]"
2024-12-04T01:33:38.750Z [INFO]  server-c2a130ea-5fac-6257-751a-148341fb98cc: entering follower state: follower="Node at c2a130ea-5fac-6257-751a-148341fb98cc [Follower]" leader-address= leader-id=
2024-12-04T01:33:38.750Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:38.750Z [INFO]  server-95f86859-a8ed-8d33-bb57-50f6dc7b6bf1: entering follower state: follower="Node at 95f86859-a8ed-8d33-bb57-50f6dc7b6bf1 [Follower]" leader-address= leader-id=
2024-12-04T01:33:38.831Z [WARN]  server-95f86859-a8ed-8d33-bb57-50f6dc7b6bf1: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:38.831Z [INFO]  server-95f86859-a8ed-8d33-bb57-50f6dc7b6bf1: entering candidate state: node="Node at 95f86859-a8ed-8d33-bb57-50f6dc7b6bf1 [Candidate]" term=2
2024-12-04T01:33:38.831Z [INFO]  server-95f86859-a8ed-8d33-bb57-50f6dc7b6bf1: election won: term=2 tally=2
2024-12-04T01:33:38.831Z [INFO]  server-95f86859-a8ed-8d33-bb57-50f6dc7b6bf1: entering leader state: leader="Node at 95f86859-a8ed-8d33-bb57-50f6dc7b6bf1 [Leader]"
2024-12-04T01:33:38.831Z [INFO]  server-95f86859-a8ed-8d33-bb57-50f6dc7b6bf1: added peer, starting replication: peer=server-c2a130ea-5fac-6257-751a-148341fb98cc
2024-12-04T01:33:38.832Z [INFO]  server-95f86859-a8ed-8d33-bb57-50f6dc7b6bf1: pipelining replication: peer="{Voter server-c2a130ea-5fac-6257-751a-148341fb98cc c2a130ea-5fac-6257-751a-148341fb98cc}"
2024-12-04T01:33:38.938Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:38.832141014 +0000 UTC m=+109.663983736 (1 nodes), 81.743048ms from start of poll, 82.106271ms from cluster start. Timeout at 2024-12-04 01:33:38.937164667 +0000 UTC m=+109.769007334, 105.023598ms after stability
2024-12-04T01:33:38.938Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:39.046Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:38.938427942 +0000 UTC m=+109.770270596 (1 nodes), 2.054µs from start of poll, 188.393131ms from cluster start. Timeout at 2024-12-04 01:33:39.043541661 +0000 UTC m=+109.875384336, 105.11374ms after stability
2024-12-04T01:33:39.062Z [INFO]  server-95f86859-a8ed-8d33-bb57-50f6dc7b6bf1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c2a130ea-5fac-6257-751a-148341fb98cc Address:c2a130ea-5fac-6257-751a-148341fb98cc} {Suffrage:Voter ID:server-95f86859-a8ed-8d33-bb57-50f6dc7b6bf1 Address:95f86859-a8ed-8d33-bb57-50f6dc7b6bf1}]"
2024-12-04T01:33:39.063Z [INFO]  server-95f86859-a8ed-8d33-bb57-50f6dc7b6bf1: aborting pipeline replication: peer="{Voter server-c2a130ea-5fac-6257-751a-148341fb98cc c2a130ea-5fac-6257-751a-148341fb98cc}"
2024-12-04T01:33:39.566Z [ERROR] server-95f86859-a8ed-8d33-bb57-50f6dc7b6bf1: failed to heartbeat to: peer=c2a130ea-5fac-6257-751a-148341fb98cc backoff time=10ms error="command timed out"
--- PASS: TestRaft_GetConfigurationNoBootstrap (0.82s)
=== RUN   TestRaft_LogStoreIsMonotonic
2024-12-04T01:33:39.567Z [INFO]  server-a45e41d9-4bb4-2db6-4a9f-24cf1fb5d313: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a45e41d9-4bb4-2db6-4a9f-24cf1fb5d313 Address:a45e41d9-4bb4-2db6-4a9f-24cf1fb5d313}]"
2024-12-04T01:33:39.568Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:39.568Z [INFO]  server-a45e41d9-4bb4-2db6-4a9f-24cf1fb5d313: entering follower state: follower="Node at a45e41d9-4bb4-2db6-4a9f-24cf1fb5d313 [Follower]" leader-address= leader-id=
2024-12-04T01:33:39.646Z [WARN]  server-a45e41d9-4bb4-2db6-4a9f-24cf1fb5d313: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:39.646Z [INFO]  server-a45e41d9-4bb4-2db6-4a9f-24cf1fb5d313: entering candidate state: node="Node at a45e41d9-4bb4-2db6-4a9f-24cf1fb5d313 [Candidate]" term=2
2024-12-04T01:33:39.646Z [INFO]  server-a45e41d9-4bb4-2db6-4a9f-24cf1fb5d313: election won: term=2 tally=1
2024-12-04T01:33:39.646Z [INFO]  server-a45e41d9-4bb4-2db6-4a9f-24cf1fb5d313: entering leader state: leader="Node at a45e41d9-4bb4-2db6-4a9f-24cf1fb5d313 [Leader]"
2024-12-04T01:33:39.754Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:39.646712963 +0000 UTC m=+110.478555601 (1 nodes), 78.646731ms from start of poll, 78.864294ms from cluster start. Timeout at 2024-12-04 01:33:39.751720214 +0000 UTC m=+110.583563297, 105.007696ms after stability
--- PASS: TestRaft_LogStoreIsMonotonic (0.19s)
=== RUN   TestRaft_CacheLogWithStoreError
2024-12-04T01:33:39.755Z [INFO]  server-4d34b521-4847-20c4-af52-df00fdd734b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d34b521-4847-20c4-af52-df00fdd734b4 Address:4d34b521-4847-20c4-af52-df00fdd734b4} {Suffrage:Voter ID:server-1ae8af9a-74c1-1330-61df-c0c560062168 Address:1ae8af9a-74c1-1330-61df-c0c560062168}]"
2024-12-04T01:33:39.755Z [INFO]  server-1ae8af9a-74c1-1330-61df-c0c560062168: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d34b521-4847-20c4-af52-df00fdd734b4 Address:4d34b521-4847-20c4-af52-df00fdd734b4} {Suffrage:Voter ID:server-1ae8af9a-74c1-1330-61df-c0c560062168 Address:1ae8af9a-74c1-1330-61df-c0c560062168}]"
2024-12-04T01:33:39.755Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:39.755Z [INFO]  server-4d34b521-4847-20c4-af52-df00fdd734b4: entering follower state: follower="Node at 4d34b521-4847-20c4-af52-df00fdd734b4 [Follower]" leader-address= leader-id=
2024-12-04T01:33:39.755Z [INFO]  server-1ae8af9a-74c1-1330-61df-c0c560062168: entering follower state: follower="Node at 1ae8af9a-74c1-1330-61df-c0c560062168 [Follower]" leader-address= leader-id=
2024-12-04T01:33:39.834Z [WARN]  server-4d34b521-4847-20c4-af52-df00fdd734b4: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:39.834Z [INFO]  server-4d34b521-4847-20c4-af52-df00fdd734b4: entering candidate state: node="Node at 4d34b521-4847-20c4-af52-df00fdd734b4 [Candidate]" term=2
2024-12-04T01:33:39.835Z [INFO]  server-4d34b521-4847-20c4-af52-df00fdd734b4: election won: term=2 tally=2
2024-12-04T01:33:39.835Z [INFO]  server-4d34b521-4847-20c4-af52-df00fdd734b4: entering leader state: leader="Node at 4d34b521-4847-20c4-af52-df00fdd734b4 [Leader]"
2024-12-04T01:33:39.835Z [INFO]  server-4d34b521-4847-20c4-af52-df00fdd734b4: added peer, starting replication: peer=server-1ae8af9a-74c1-1330-61df-c0c560062168
2024-12-04T01:33:39.835Z [INFO]  server-4d34b521-4847-20c4-af52-df00fdd734b4: pipelining replication: peer="{Voter server-1ae8af9a-74c1-1330-61df-c0c560062168 1ae8af9a-74c1-1330-61df-c0c560062168}"
2024-12-04T01:33:39.942Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:39.835430092 +0000 UTC m=+110.667272707 (1 nodes), 79.523ms from start of poll, 79.832631ms from cluster start. Timeout at 2024-12-04 01:33:39.940452815 +0000 UTC m=+110.772295778, 105.023071ms after stability
2024-12-04T01:33:39.942Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:40.054Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:39.942393246 +0000 UTC m=+110.774236202 (1 nodes), 3.267µs from start of poll, 186.796126ms from cluster start. Timeout at 2024-12-04 01:33:40.047656263 +0000 UTC m=+110.879498879, 105.262677ms after stability
2024-12-04T01:33:40.057Z [ERROR] server-1ae8af9a-74c1-1330-61df-c0c560062168: failed to append to logs: error="unable to store logs within log store, err: "some error""
2024-12-04T01:33:40.057Z [WARN]  server-1ae8af9a-74c1-1330-61df-c0c560062168: failed to get previous log: previous-index=9 last-index=7 error="log not found"
2024-12-04T01:33:40.057Z [INFO]  server-4d34b521-4847-20c4-af52-df00fdd734b4: aborting pipeline replication: peer="{Voter server-1ae8af9a-74c1-1330-61df-c0c560062168 1ae8af9a-74c1-1330-61df-c0c560062168}"
2024-12-04T01:33:40.078Z [INFO]  server-4d34b521-4847-20c4-af52-df00fdd734b4: pipelining replication: peer="{Voter server-1ae8af9a-74c1-1330-61df-c0c560062168 1ae8af9a-74c1-1330-61df-c0c560062168}"
2024-12-04T01:33:40.079Z [INFO]  server-1ae8af9a-74c1-1330-61df-c0c560062168: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4d34b521-4847-20c4-af52-df00fdd734b4 Address:4d34b521-4847-20c4-af52-df00fdd734b4} {Suffrage:Voter ID:server-1ae8af9a-74c1-1330-61df-c0c560062168 Address:1ae8af9a-74c1-1330-61df-c0c560062168}]"
2024-12-04T01:33:40.079Z [INFO]  server-4d34b521-4847-20c4-af52-df00fdd734b4: aborting pipeline replication: peer="{Voter server-1ae8af9a-74c1-1330-61df-c0c560062168 1ae8af9a-74c1-1330-61df-c0c560062168}"
--- PASS: TestRaft_CacheLogWithStoreError (0.33s)
=== RUN   TestRaft_ReloadConfig
2024-12-04T01:33:40.080Z [INFO]  server-981e75d7-7351-c563-0597-fe4d6b8d2af6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-981e75d7-7351-c563-0597-fe4d6b8d2af6 Address:981e75d7-7351-c563-0597-fe4d6b8d2af6}]"
--- PASS: TestRaft_ReloadConfig (0.00s)
=== RUN   TestRaft_ReloadConfigValidates
2024-12-04T01:33:40.081Z [INFO]  server-039281e4-15a6-9ce3-7b33-04550e1ea717: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-039281e4-15a6-9ce3-7b33-04550e1ea717 Address:039281e4-15a6-9ce3-7b33-04550e1ea717}]"
--- PASS: TestRaft_ReloadConfigValidates (0.00s)
=== RUN   TestRaft_InstallSnapshot_InvalidPeers
2024-12-04T01:33:40.081Z [ERROR] failed to install snapshot: error="failed to decode peers: codec.decoder: Only encoded map or array can be decoded into a slice. (valueType: 2)"
--- PASS: TestRaft_InstallSnapshot_InvalidPeers (0.00s)
=== RUN   TestRaft_VoteNotGranted_WhenNodeNotInCluster
2024-12-04T01:33:40.082Z [INFO]  server-afe8c4a2-7225-eb25-0570-802ddaf3dbfa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afe8c4a2-7225-eb25-0570-802ddaf3dbfa Address:afe8c4a2-7225-eb25-0570-802ddaf3dbfa} {Suffrage:Voter ID:server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97 Address:f5e98731-90b3-b8fe-20ec-d3d019ed4d97} {Suffrage:Voter ID:server-55dacce4-5659-98e6-fd05-2db70844f91b Address:55dacce4-5659-98e6-fd05-2db70844f91b}]"
2024-12-04T01:33:40.082Z [INFO]  server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afe8c4a2-7225-eb25-0570-802ddaf3dbfa Address:afe8c4a2-7225-eb25-0570-802ddaf3dbfa} {Suffrage:Voter ID:server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97 Address:f5e98731-90b3-b8fe-20ec-d3d019ed4d97} {Suffrage:Voter ID:server-55dacce4-5659-98e6-fd05-2db70844f91b Address:55dacce4-5659-98e6-fd05-2db70844f91b}]"
2024-12-04T01:33:40.082Z [INFO]  server-55dacce4-5659-98e6-fd05-2db70844f91b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-afe8c4a2-7225-eb25-0570-802ddaf3dbfa Address:afe8c4a2-7225-eb25-0570-802ddaf3dbfa} {Suffrage:Voter ID:server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97 Address:f5e98731-90b3-b8fe-20ec-d3d019ed4d97} {Suffrage:Voter ID:server-55dacce4-5659-98e6-fd05-2db70844f91b Address:55dacce4-5659-98e6-fd05-2db70844f91b}]"
2024-12-04T01:33:40.082Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:40.083Z [INFO]  server-afe8c4a2-7225-eb25-0570-802ddaf3dbfa: entering follower state: follower="Node at afe8c4a2-7225-eb25-0570-802ddaf3dbfa [Follower]" leader-address= leader-id=
2024-12-04T01:33:40.083Z [INFO]  server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: entering follower state: follower="Node at f5e98731-90b3-b8fe-20ec-d3d019ed4d97 [Follower]" leader-address= leader-id=
2024-12-04T01:33:40.083Z [INFO]  server-55dacce4-5659-98e6-fd05-2db70844f91b: entering follower state: follower="Node at 55dacce4-5659-98e6-fd05-2db70844f91b [Follower]" leader-address= leader-id=
2024-12-04T01:33:40.146Z [WARN]  server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:40.146Z [INFO]  server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: entering candidate state: node="Node at f5e98731-90b3-b8fe-20ec-d3d019ed4d97 [Candidate]" term=2
2024-12-04T01:33:40.146Z [INFO]  server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: election won: term=2 tally=2
2024-12-04T01:33:40.146Z [INFO]  server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: entering leader state: leader="Node at f5e98731-90b3-b8fe-20ec-d3d019ed4d97 [Leader]"
2024-12-04T01:33:40.146Z [INFO]  server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: added peer, starting replication: peer=server-afe8c4a2-7225-eb25-0570-802ddaf3dbfa
2024-12-04T01:33:40.146Z [INFO]  server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: added peer, starting replication: peer=server-55dacce4-5659-98e6-fd05-2db70844f91b
2024-12-04T01:33:40.146Z [INFO]  server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: pipelining replication: peer="{Voter server-55dacce4-5659-98e6-fd05-2db70844f91b 55dacce4-5659-98e6-fd05-2db70844f91b}"
2024-12-04T01:33:40.147Z [INFO]  server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: pipelining replication: peer="{Voter server-afe8c4a2-7225-eb25-0570-802ddaf3dbfa afe8c4a2-7225-eb25-0570-802ddaf3dbfa}"
2024-12-04T01:33:40.253Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:40.147625541 +0000 UTC m=+110.979468213 (1 nodes), 64.663337ms from start of poll, 65.094393ms from cluster start. Timeout at 2024-12-04 01:33:40.252634758 +0000 UTC m=+111.084477449, 105.009236ms after stability
2024-12-04T01:33:40.259Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:40.366Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:40.259530925 +0000 UTC m=+111.091373659 (2 nodes), 2.378µs from start of poll, 176.999839ms from cluster start. Timeout at 2024-12-04 01:33:40.364663839 +0000 UTC m=+111.196506648, 105.132989ms after stability
2024-12-04T01:33:40.366Z [INFO]  server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: updating configuration: command=RemoveServer server-id=server-afe8c4a2-7225-eb25-0570-802ddaf3dbfa server-addr= servers="[{Suffrage:Voter ID:server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97 Address:f5e98731-90b3-b8fe-20ec-d3d019ed4d97} {Suffrage:Voter ID:server-55dacce4-5659-98e6-fd05-2db70844f91b Address:55dacce4-5659-98e6-fd05-2db70844f91b}]"
2024-12-04T01:33:40.366Z [INFO]  server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: removed peer, stopping replication: peer=server-afe8c4a2-7225-eb25-0570-802ddaf3dbfa last-index=3
2024-12-04T01:33:40.367Z [INFO]  server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: aborting pipeline replication: peer="{Voter server-afe8c4a2-7225-eb25-0570-802ddaf3dbfa afe8c4a2-7225-eb25-0570-802ddaf3dbfa}"
2024-12-04T01:33:40.474Z [WARN]  server-afe8c4a2-7225-eb25-0570-802ddaf3dbfa: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
2024-12-04T01:33:40.475Z [ERROR] server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: failed to heartbeat to: peer=55dacce4-5659-98e6-fd05-2db70844f91b backoff time=10ms error="failed to connect to peer: 55dacce4-5659-98e6-fd05-2db70844f91b"
2024-12-04T01:33:40.482Z [ERROR] server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: failed to pipeline appendEntries: peer="{Voter server-55dacce4-5659-98e6-fd05-2db70844f91b 55dacce4-5659-98e6-fd05-2db70844f91b}" error="append pipeline closed"
2024-12-04T01:33:40.482Z [INFO]  server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: aborting pipeline replication: peer="{Voter server-55dacce4-5659-98e6-fd05-2db70844f91b 55dacce4-5659-98e6-fd05-2db70844f91b}"
2024-12-04T01:33:40.500Z [ERROR] server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: failed to heartbeat to: peer=55dacce4-5659-98e6-fd05-2db70844f91b backoff time=10ms error="failed to connect to peer: 55dacce4-5659-98e6-fd05-2db70844f91b"
2024-12-04T01:33:40.502Z [ERROR] server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: failed to appendEntries to: peer="{Voter server-55dacce4-5659-98e6-fd05-2db70844f91b 55dacce4-5659-98e6-fd05-2db70844f91b}" error="failed to connect to peer: 55dacce4-5659-98e6-fd05-2db70844f91b"
2024-12-04T01:33:40.517Z [ERROR] server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: failed to heartbeat to: peer=55dacce4-5659-98e6-fd05-2db70844f91b backoff time=10ms error="failed to connect to peer: 55dacce4-5659-98e6-fd05-2db70844f91b"
2024-12-04T01:33:40.519Z [ERROR] server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: failed to appendEntries to: peer="{Voter server-55dacce4-5659-98e6-fd05-2db70844f91b 55dacce4-5659-98e6-fd05-2db70844f91b}" error="failed to connect to peer: 55dacce4-5659-98e6-fd05-2db70844f91b"
2024-12-04T01:33:40.534Z [WARN]  server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: failed to contact: server-id=server-55dacce4-5659-98e6-fd05-2db70844f91b time=59.990852ms
2024-12-04T01:33:40.534Z [WARN]  server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: failed to contact quorum of nodes, stepping down
2024-12-04T01:33:40.534Z [INFO]  server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: entering follower state: follower="Node at f5e98731-90b3-b8fe-20ec-d3d019ed4d97 [Follower]" leader-address= leader-id=
2024-12-04T01:33:40.539Z [ERROR] server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: failed to appendEntries to: peer="{Voter server-55dacce4-5659-98e6-fd05-2db70844f91b 55dacce4-5659-98e6-fd05-2db70844f91b}" error="failed to connect to peer: 55dacce4-5659-98e6-fd05-2db70844f91b"
2024-12-04T01:33:40.539Z [ERROR] server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97: failed to heartbeat to: peer=55dacce4-5659-98e6-fd05-2db70844f91b backoff time=20ms error="failed to connect to peer: 55dacce4-5659-98e6-fd05-2db70844f91b"
2024-12-04T01:33:40.578Z [WARN]  server-55dacce4-5659-98e6-fd05-2db70844f91b: heartbeat timeout reached, starting election: last-leader-addr=f5e98731-90b3-b8fe-20ec-d3d019ed4d97 last-leader-id=server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97
2024-12-04T01:33:40.578Z [INFO]  server-55dacce4-5659-98e6-fd05-2db70844f91b: entering candidate state: node="Node at 55dacce4-5659-98e6-fd05-2db70844f91b [Candidate]" term=3
2024-12-04T01:33:40.578Z [ERROR] server-55dacce4-5659-98e6-fd05-2db70844f91b: failed to make requestVote RPC: target="{Voter server-f5e98731-90b3-b8fe-20ec-d3d019ed4d97 f5e98731-90b3-b8fe-20ec-d3d019ed4d97}" error="failed to connect to peer: f5e98731-90b3-b8fe-20ec-d3d019ed4d97" term=3
2024-12-04T01:33:40.583Z [WARN]  server-55dacce4-5659-98e6-fd05-2db70844f91b: rejecting vote request since node is not in configuration: from=afe8c4a2-7225-eb25-0570-802ddaf3dbfa
--- PASS: TestRaft_VoteNotGranted_WhenNodeNotInCluster (0.50s)
=== RUN   TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin
2024-12-04T01:33:40.585Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0055605-816d-197c-87f1-5ab5c5f3c0f8 Address:a0055605-816d-197c-87f1-5ab5c5f3c0f8} {Suffrage:Voter ID:server-0a4e885e-92c9-5f41-d708-12779755686d Address:0a4e885e-92c9-5f41-d708-12779755686d} {Suffrage:Voter ID:server-aee340b3-3de5-9e80-f876-ec2f010e6abb Address:aee340b3-3de5-9e80-f876-ec2f010e6abb}]"
2024-12-04T01:33:40.586Z [INFO]  server-0a4e885e-92c9-5f41-d708-12779755686d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0055605-816d-197c-87f1-5ab5c5f3c0f8 Address:a0055605-816d-197c-87f1-5ab5c5f3c0f8} {Suffrage:Voter ID:server-0a4e885e-92c9-5f41-d708-12779755686d Address:0a4e885e-92c9-5f41-d708-12779755686d} {Suffrage:Voter ID:server-aee340b3-3de5-9e80-f876-ec2f010e6abb Address:aee340b3-3de5-9e80-f876-ec2f010e6abb}]"
2024-12-04T01:33:40.586Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: entering follower state: follower="Node at a0055605-816d-197c-87f1-5ab5c5f3c0f8 [Follower]" leader-address= leader-id=
2024-12-04T01:33:40.586Z [INFO]  server-0a4e885e-92c9-5f41-d708-12779755686d: entering follower state: follower="Node at 0a4e885e-92c9-5f41-d708-12779755686d [Follower]" leader-address= leader-id=
2024-12-04T01:33:40.586Z [INFO]  server-aee340b3-3de5-9e80-f876-ec2f010e6abb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a0055605-816d-197c-87f1-5ab5c5f3c0f8 Address:a0055605-816d-197c-87f1-5ab5c5f3c0f8} {Suffrage:Voter ID:server-0a4e885e-92c9-5f41-d708-12779755686d Address:0a4e885e-92c9-5f41-d708-12779755686d} {Suffrage:Voter ID:server-aee340b3-3de5-9e80-f876-ec2f010e6abb Address:aee340b3-3de5-9e80-f876-ec2f010e6abb}]"
2024-12-04T01:33:40.587Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:40.587Z [INFO]  server-aee340b3-3de5-9e80-f876-ec2f010e6abb: entering follower state: follower="Node at aee340b3-3de5-9e80-f876-ec2f010e6abb [Follower]" leader-address= leader-id=
2024-12-04T01:33:40.638Z [WARN]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:40.639Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: entering candidate state: node="Node at a0055605-816d-197c-87f1-5ab5c5f3c0f8 [Candidate]" term=2
2024-12-04T01:33:40.639Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: election won: term=2 tally=2
2024-12-04T01:33:40.639Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: entering leader state: leader="Node at a0055605-816d-197c-87f1-5ab5c5f3c0f8 [Leader]"
2024-12-04T01:33:40.639Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: added peer, starting replication: peer=server-0a4e885e-92c9-5f41-d708-12779755686d
2024-12-04T01:33:40.639Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: added peer, starting replication: peer=server-aee340b3-3de5-9e80-f876-ec2f010e6abb
2024-12-04T01:33:40.640Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: pipelining replication: peer="{Voter server-aee340b3-3de5-9e80-f876-ec2f010e6abb aee340b3-3de5-9e80-f876-ec2f010e6abb}"
2024-12-04T01:33:40.640Z [WARN]  server-aee340b3-3de5-9e80-f876-ec2f010e6abb: rejecting vote request since our last term is greater: candidate=a0055605-816d-197c-87f1-5ab5c5f3c0f8 last-term=2 last-candidate-term=1
2024-12-04T01:33:40.640Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: pipelining replication: peer="{Voter server-0a4e885e-92c9-5f41-d708-12779755686d 0a4e885e-92c9-5f41-d708-12779755686d}"
2024-12-04T01:33:40.756Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:40.640417254 +0000 UTC m=+111.472259878 (1 nodes), 53.38729ms from start of poll, 54.65669ms from cluster start. Timeout at 2024-12-04 01:33:40.745455623 +0000 UTC m=+111.577298281, 105.038403ms after stability
2024-12-04T01:33:40.766Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:40.874Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:40.766576713 +0000 UTC m=+111.598419364 (2 nodes), 2.727µs from start of poll, 180.816176ms from cluster start. Timeout at 2024-12-04 01:33:40.87160995 +0000 UTC m=+111.703452797, 105.033433ms after stability
2024-12-04T01:33:40.878Z [ERROR] server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: failed to heartbeat to: peer=0a4e885e-92c9-5f41-d708-12779755686d backoff time=10ms error="failed to connect to peer: 0a4e885e-92c9-5f41-d708-12779755686d"
2024-12-04T01:33:40.879Z [ERROR] server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: failed to pipeline appendEntries: peer="{Voter server-0a4e885e-92c9-5f41-d708-12779755686d 0a4e885e-92c9-5f41-d708-12779755686d}" error="append pipeline closed"
2024-12-04T01:33:40.879Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: aborting pipeline replication: peer="{Voter server-0a4e885e-92c9-5f41-d708-12779755686d 0a4e885e-92c9-5f41-d708-12779755686d}"
2024-12-04T01:33:40.890Z [ERROR] server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: failed to appendEntries to: peer="{Voter server-0a4e885e-92c9-5f41-d708-12779755686d 0a4e885e-92c9-5f41-d708-12779755686d}" error="failed to connect to peer: 0a4e885e-92c9-5f41-d708-12779755686d"
2024-12-04T01:33:40.902Z [ERROR] server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: failed to heartbeat to: peer=0a4e885e-92c9-5f41-d708-12779755686d backoff time=10ms error="failed to connect to peer: 0a4e885e-92c9-5f41-d708-12779755686d"
2024-12-04T01:33:40.913Z [ERROR] server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: failed to appendEntries to: peer="{Voter server-0a4e885e-92c9-5f41-d708-12779755686d 0a4e885e-92c9-5f41-d708-12779755686d}" error="failed to connect to peer: 0a4e885e-92c9-5f41-d708-12779755686d"
2024-12-04T01:33:40.922Z [WARN]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: failed to contact: server-id=server-0a4e885e-92c9-5f41-d708-12779755686d time=50.497413ms
2024-12-04T01:33:40.926Z [ERROR] server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: failed to heartbeat to: peer=0a4e885e-92c9-5f41-d708-12779755686d backoff time=10ms error="failed to connect to peer: 0a4e885e-92c9-5f41-d708-12779755686d"
2024-12-04T01:33:40.942Z [ERROR] server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: failed to appendEntries to: peer="{Voter server-0a4e885e-92c9-5f41-d708-12779755686d 0a4e885e-92c9-5f41-d708-12779755686d}" error="failed to connect to peer: 0a4e885e-92c9-5f41-d708-12779755686d"
2024-12-04T01:33:40.950Z [ERROR] server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: failed to heartbeat to: peer=0a4e885e-92c9-5f41-d708-12779755686d backoff time=20ms error="failed to connect to peer: 0a4e885e-92c9-5f41-d708-12779755686d"
2024-12-04T01:33:40.974Z [ERROR] server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: failed to appendEntries to: peer="{Voter server-0a4e885e-92c9-5f41-d708-12779755686d 0a4e885e-92c9-5f41-d708-12779755686d}" error="failed to connect to peer: 0a4e885e-92c9-5f41-d708-12779755686d"
2024-12-04T01:33:40.974Z [WARN]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: failed to contact: server-id=server-0a4e885e-92c9-5f41-d708-12779755686d time=102.552593ms
2024-12-04T01:33:40.982Z [ERROR] server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: failed to heartbeat to: peer=0a4e885e-92c9-5f41-d708-12779755686d backoff time=25ms error="failed to connect to peer: 0a4e885e-92c9-5f41-d708-12779755686d"
2024-12-04T01:33:40.983Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: updating configuration: command=RemoveServer server-id=server-0a4e885e-92c9-5f41-d708-12779755686d server-addr= servers="[{Suffrage:Voter ID:server-a0055605-816d-197c-87f1-5ab5c5f3c0f8 Address:a0055605-816d-197c-87f1-5ab5c5f3c0f8} {Suffrage:Voter ID:server-aee340b3-3de5-9e80-f876-ec2f010e6abb Address:aee340b3-3de5-9e80-f876-ec2f010e6abb}]"
2024-12-04T01:33:40.983Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: removed peer, stopping replication: peer=server-0a4e885e-92c9-5f41-d708-12779755686d last-index=3
2024-12-04T01:33:40.984Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: updating configuration: command=AddNonvoter server-id=server-0a4e885e-92c9-5f41-d708-12779755686d server-addr=0a4e885e-92c9-5f41-d708-12779755686d servers="[{Suffrage:Voter ID:server-a0055605-816d-197c-87f1-5ab5c5f3c0f8 Address:a0055605-816d-197c-87f1-5ab5c5f3c0f8} {Suffrage:Voter ID:server-aee340b3-3de5-9e80-f876-ec2f010e6abb Address:aee340b3-3de5-9e80-f876-ec2f010e6abb} {Suffrage:Nonvoter ID:server-0a4e885e-92c9-5f41-d708-12779755686d Address:0a4e885e-92c9-5f41-d708-12779755686d}]"
2024-12-04T01:33:40.984Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: added peer, starting replication: peer=server-0a4e885e-92c9-5f41-d708-12779755686d
2024-12-04T01:33:40.984Z [ERROR] server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: failed to appendEntries to: peer="{Nonvoter server-0a4e885e-92c9-5f41-d708-12779755686d 0a4e885e-92c9-5f41-d708-12779755686d}" error="failed to connect to peer: 0a4e885e-92c9-5f41-d708-12779755686d"
2024-12-04T01:33:40.990Z [WARN]  server-0a4e885e-92c9-5f41-d708-12779755686d: heartbeat timeout reached, starting election: last-leader-addr=a0055605-816d-197c-87f1-5ab5c5f3c0f8 last-leader-id=server-a0055605-816d-197c-87f1-5ab5c5f3c0f8
2024-12-04T01:33:40.990Z [INFO]  server-0a4e885e-92c9-5f41-d708-12779755686d: entering candidate state: node="Node at 0a4e885e-92c9-5f41-d708-12779755686d [Candidate]" term=23
2024-12-04T01:33:40.990Z [WARN]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: rejecting vote request since we have a leader: from=0a4e885e-92c9-5f41-d708-12779755686d leader=a0055605-816d-197c-87f1-5ab5c5f3c0f8 leader-id=server-a0055605-816d-197c-87f1-5ab5c5f3c0f8
2024-12-04T01:33:40.990Z [WARN]  server-aee340b3-3de5-9e80-f876-ec2f010e6abb: rejecting vote request since we have a leader: from=0a4e885e-92c9-5f41-d708-12779755686d leader=a0055605-816d-197c-87f1-5ab5c5f3c0f8 leader-id=server-a0055605-816d-197c-87f1-5ab5c5f3c0f8
2024-12-04T01:33:41.010Z [ERROR] server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: peer has newer term, stopping replication: peer="{Nonvoter server-0a4e885e-92c9-5f41-d708-12779755686d 0a4e885e-92c9-5f41-d708-12779755686d}"
2024-12-04T01:33:41.010Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: entering follower state: follower="Node at a0055605-816d-197c-87f1-5ab5c5f3c0f8 [Follower]" leader-address= leader-id=
2024-12-04T01:33:41.010Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: aborting pipeline replication: peer="{Voter server-aee340b3-3de5-9e80-f876-ec2f010e6abb aee340b3-3de5-9e80-f876-ec2f010e6abb}"
2024-12-04T01:33:41.025Z [ERROR] server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: peer has newer term, stopping replication: peer="{Voter server-0a4e885e-92c9-5f41-d708-12779755686d 0a4e885e-92c9-5f41-d708-12779755686d}"
2024-12-04T01:33:41.070Z [WARN]  server-0a4e885e-92c9-5f41-d708-12779755686d: Election timeout reached, restarting election
2024-12-04T01:33:41.086Z [INFO]  server-0a4e885e-92c9-5f41-d708-12779755686d: entering candidate state: node="Node at 0a4e885e-92c9-5f41-d708-12779755686d [Candidate]" term=24
2024-12-04T01:33:41.086Z [WARN]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: rejecting vote request since node is not a voter: from=0a4e885e-92c9-5f41-d708-12779755686d
2024-12-04T01:33:41.086Z [WARN]  server-aee340b3-3de5-9e80-f876-ec2f010e6abb: heartbeat timeout reached, starting election: last-leader-addr=a0055605-816d-197c-87f1-5ab5c5f3c0f8 last-leader-id=server-a0055605-816d-197c-87f1-5ab5c5f3c0f8
2024-12-04T01:33:41.086Z [INFO]  server-aee340b3-3de5-9e80-f876-ec2f010e6abb: entering candidate state: node="Node at aee340b3-3de5-9e80-f876-ec2f010e6abb [Candidate]" term=3
2024-12-04T01:33:41.086Z [WARN]  server-aee340b3-3de5-9e80-f876-ec2f010e6abb: rejecting vote request since node is not a voter: from=0a4e885e-92c9-5f41-d708-12779755686d
2024-12-04T01:33:41.086Z [INFO]  server-aee340b3-3de5-9e80-f876-ec2f010e6abb: entering follower state: follower="Node at aee340b3-3de5-9e80-f876-ec2f010e6abb [Follower]" leader-address= leader-id=
2024-12-04T01:33:41.090Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:41.090Z [WARN]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:41.090Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: entering candidate state: node="Node at a0055605-816d-197c-87f1-5ab5c5f3c0f8 [Candidate]" term=25
2024-12-04T01:33:41.090Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: election won: term=25 tally=2
2024-12-04T01:33:41.090Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: entering leader state: leader="Node at a0055605-816d-197c-87f1-5ab5c5f3c0f8 [Leader]"
2024-12-04T01:33:41.090Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: added peer, starting replication: peer=server-aee340b3-3de5-9e80-f876-ec2f010e6abb
2024-12-04T01:33:41.090Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: added peer, starting replication: peer=server-0a4e885e-92c9-5f41-d708-12779755686d
2024-12-04T01:33:41.090Z [WARN]  server-0a4e885e-92c9-5f41-d708-12779755686d: failed to get previous log: previous-index=4 last-index=2 error="log not found"
2024-12-04T01:33:41.090Z [INFO]  server-0a4e885e-92c9-5f41-d708-12779755686d: entering follower state: follower="Node at 0a4e885e-92c9-5f41-d708-12779755686d [Follower]" leader-address=a0055605-816d-197c-87f1-5ab5c5f3c0f8 leader-id=server-a0055605-816d-197c-87f1-5ab5c5f3c0f8
2024-12-04T01:33:41.090Z [WARN]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: appendEntries rejected, sending older logs: peer="{Nonvoter server-0a4e885e-92c9-5f41-d708-12779755686d 0a4e885e-92c9-5f41-d708-12779755686d}" next=3
2024-12-04T01:33:41.090Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: pipelining replication: peer="{Nonvoter server-0a4e885e-92c9-5f41-d708-12779755686d 0a4e885e-92c9-5f41-d708-12779755686d}"
2024-12-04T01:33:41.091Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: pipelining replication: peer="{Voter server-aee340b3-3de5-9e80-f876-ec2f010e6abb aee340b3-3de5-9e80-f876-ec2f010e6abb}"
2024-12-04T01:33:41.202Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:41.091077047 +0000 UTC m=+111.922919660 (1 nodes), 797.8µs from start of poll, 505.316472ms from cluster start. Timeout at 2024-12-04 01:33:41.196130957 +0000 UTC m=+112.027973827, 105.054167ms after stability
2024-12-04T01:33:41.318Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: updating configuration: command=RemoveServer server-id=server-0a4e885e-92c9-5f41-d708-12779755686d server-addr= servers="[{Suffrage:Voter ID:server-a0055605-816d-197c-87f1-5ab5c5f3c0f8 Address:a0055605-816d-197c-87f1-5ab5c5f3c0f8} {Suffrage:Voter ID:server-aee340b3-3de5-9e80-f876-ec2f010e6abb Address:aee340b3-3de5-9e80-f876-ec2f010e6abb}]"
2024-12-04T01:33:41.318Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: removed peer, stopping replication: peer=server-0a4e885e-92c9-5f41-d708-12779755686d last-index=106
2024-12-04T01:33:41.318Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: aborting pipeline replication: peer="{Nonvoter server-0a4e885e-92c9-5f41-d708-12779755686d 0a4e885e-92c9-5f41-d708-12779755686d}"
2024-12-04T01:33:41.318Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: updating configuration: command=AddVoter server-id=server-0a4e885e-92c9-5f41-d708-12779755686d server-addr=0a4e885e-92c9-5f41-d708-12779755686d servers="[{Suffrage:Voter ID:server-a0055605-816d-197c-87f1-5ab5c5f3c0f8 Address:a0055605-816d-197c-87f1-5ab5c5f3c0f8} {Suffrage:Voter ID:server-aee340b3-3de5-9e80-f876-ec2f010e6abb Address:aee340b3-3de5-9e80-f876-ec2f010e6abb} {Suffrage:Voter ID:server-0a4e885e-92c9-5f41-d708-12779755686d Address:0a4e885e-92c9-5f41-d708-12779755686d}]"
2024-12-04T01:33:41.318Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: added peer, starting replication: peer=server-0a4e885e-92c9-5f41-d708-12779755686d
2024-12-04T01:33:41.319Z [WARN]  server-0a4e885e-92c9-5f41-d708-12779755686d: failed to get previous log: previous-index=107 last-index=106 error="log not found"
2024-12-04T01:33:41.319Z [WARN]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: appendEntries rejected, sending older logs: peer="{Voter server-0a4e885e-92c9-5f41-d708-12779755686d 0a4e885e-92c9-5f41-d708-12779755686d}" next=107
2024-12-04T01:33:41.319Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: pipelining replication: peer="{Voter server-0a4e885e-92c9-5f41-d708-12779755686d 0a4e885e-92c9-5f41-d708-12779755686d}"
2024-12-04T01:33:42.448Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:42.559Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:42.448142637 +0000 UTC m=+113.279985390 (1 nodes), 2.124µs from start of poll, 1.862382202s from cluster start. Timeout at 2024-12-04 01:33:42.553238057 +0000 UTC m=+113.385080718, 105.095328ms after stability
2024-12-04T01:33:42.559Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: aborting pipeline replication: peer="{Voter server-0a4e885e-92c9-5f41-d708-12779755686d 0a4e885e-92c9-5f41-d708-12779755686d}"
2024-12-04T01:33:42.559Z [INFO]  server-a0055605-816d-197c-87f1-5ab5c5f3c0f8: aborting pipeline replication: peer="{Voter server-aee340b3-3de5-9e80-f876-ec2f010e6abb aee340b3-3de5-9e80-f876-ec2f010e6abb}"
--- PASS: TestRaft_ClusterCanRegainStability_WhenNonVoterWithHigherTermJoin (1.98s)
=== RUN   TestRaft_FollowerRemovalNoElection
2024-12-04T01:33:42.563Z [INFO]  server-83f26e72-b4af-a421-c5d3-e4ad592a7a35: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83f26e72-b4af-a421-c5d3-e4ad592a7a35 Address:83f26e72-b4af-a421-c5d3-e4ad592a7a35} {Suffrage:Voter ID:server-bf0352c1-ac1b-987c-66db-cddf874d8a0b Address:bf0352c1-ac1b-987c-66db-cddf874d8a0b} {Suffrage:Voter ID:server-e430f761-28ad-995a-c437-e0df8a2989ce Address:e430f761-28ad-995a-c437-e0df8a2989ce}]"
2024-12-04T01:33:42.563Z [INFO]  server-bf0352c1-ac1b-987c-66db-cddf874d8a0b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83f26e72-b4af-a421-c5d3-e4ad592a7a35 Address:83f26e72-b4af-a421-c5d3-e4ad592a7a35} {Suffrage:Voter ID:server-bf0352c1-ac1b-987c-66db-cddf874d8a0b Address:bf0352c1-ac1b-987c-66db-cddf874d8a0b} {Suffrage:Voter ID:server-e430f761-28ad-995a-c437-e0df8a2989ce Address:e430f761-28ad-995a-c437-e0df8a2989ce}]"
2024-12-04T01:33:42.563Z [INFO]  server-e430f761-28ad-995a-c437-e0df8a2989ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83f26e72-b4af-a421-c5d3-e4ad592a7a35 Address:83f26e72-b4af-a421-c5d3-e4ad592a7a35} {Suffrage:Voter ID:server-bf0352c1-ac1b-987c-66db-cddf874d8a0b Address:bf0352c1-ac1b-987c-66db-cddf874d8a0b} {Suffrage:Voter ID:server-e430f761-28ad-995a-c437-e0df8a2989ce Address:e430f761-28ad-995a-c437-e0df8a2989ce}]"
2024-12-04T01:33:42.563Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:42.563Z [INFO]  server-83f26e72-b4af-a421-c5d3-e4ad592a7a35: entering follower state: follower="Node at 83f26e72-b4af-a421-c5d3-e4ad592a7a35 [Follower]" leader-address= leader-id=
2024-12-04T01:33:42.563Z [INFO]  server-bf0352c1-ac1b-987c-66db-cddf874d8a0b: entering follower state: follower="Node at bf0352c1-ac1b-987c-66db-cddf874d8a0b [Follower]" leader-address= leader-id=
2024-12-04T01:33:42.563Z [INFO]  server-e430f761-28ad-995a-c437-e0df8a2989ce: entering follower state: follower="Node at e430f761-28ad-995a-c437-e0df8a2989ce [Follower]" leader-address= leader-id=
2024-12-04T01:33:42.683Z [WARN]  server-83f26e72-b4af-a421-c5d3-e4ad592a7a35: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:42.683Z [INFO]  server-83f26e72-b4af-a421-c5d3-e4ad592a7a35: entering candidate state: node="Node at 83f26e72-b4af-a421-c5d3-e4ad592a7a35 [Candidate]" term=2
2024-12-04T01:33:42.683Z [INFO]  server-83f26e72-b4af-a421-c5d3-e4ad592a7a35: election won: term=2 tally=2
2024-12-04T01:33:42.683Z [INFO]  server-83f26e72-b4af-a421-c5d3-e4ad592a7a35: entering leader state: leader="Node at 83f26e72-b4af-a421-c5d3-e4ad592a7a35 [Leader]"
2024-12-04T01:33:42.683Z [INFO]  server-83f26e72-b4af-a421-c5d3-e4ad592a7a35: added peer, starting replication: peer=server-bf0352c1-ac1b-987c-66db-cddf874d8a0b
2024-12-04T01:33:42.683Z [INFO]  server-83f26e72-b4af-a421-c5d3-e4ad592a7a35: added peer, starting replication: peer=server-e430f761-28ad-995a-c437-e0df8a2989ce
2024-12-04T01:33:42.684Z [INFO]  server-83f26e72-b4af-a421-c5d3-e4ad592a7a35: pipelining replication: peer="{Voter server-e430f761-28ad-995a-c437-e0df8a2989ce e430f761-28ad-995a-c437-e0df8a2989ce}"
2024-12-04T01:33:42.684Z [INFO]  server-83f26e72-b4af-a421-c5d3-e4ad592a7a35: pipelining replication: peer="{Voter server-bf0352c1-ac1b-987c-66db-cddf874d8a0b bf0352c1-ac1b-987c-66db-cddf874d8a0b}"
2024-12-04T01:33:42.890Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:42.684377853 +0000 UTC m=+113.516220471 (1 nodes), 120.830454ms from start of poll, 122.282311ms from cluster start. Timeout at 2024-12-04 01:33:42.889387705 +0000 UTC m=+113.721230347, 205.009876ms after stability
2024-12-04T01:33:42.890Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:43.105Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:42.890799051 +0000 UTC m=+113.722641807 (1 nodes), 2.68µs from start of poll, 328.703647ms from cluster start. Timeout at 2024-12-04 01:33:43.095921386 +0000 UTC m=+113.927764124, 205.122317ms after stability
2024-12-04T01:33:43.111Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:43.317Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:43.112026501 +0000 UTC m=+113.943869435 (2 nodes), 2.732µs from start of poll, 549.931275ms from cluster start. Timeout at 2024-12-04 01:33:43.317082414 +0000 UTC m=+114.148925204, 205.055769ms after stability
2024-12-04T01:33:43.317Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:43.528Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:43.317424703 +0000 UTC m=+114.149267408 (1 nodes), 2.341µs from start of poll, 755.329248ms from cluster start. Timeout at 2024-12-04 01:33:43.522477056 +0000 UTC m=+114.354319968, 205.05256ms after stability
    raft_test.go:2961: [INFO] restarting Node at bf0352c1-ac1b-987c-66db-cddf874d8a0b [Follower]
2024-12-04T01:33:43.528Z [INFO]  server-bf0352c1-ac1b-987c-66db-cddf874d8a0b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-83f26e72-b4af-a421-c5d3-e4ad592a7a35 Address:83f26e72-b4af-a421-c5d3-e4ad592a7a35} {Suffrage:Voter ID:server-bf0352c1-ac1b-987c-66db-cddf874d8a0b Address:bf0352c1-ac1b-987c-66db-cddf874d8a0b} {Suffrage:Voter ID:server-e430f761-28ad-995a-c437-e0df8a2989ce Address:e430f761-28ad-995a-c437-e0df8a2989ce}]"
2024-12-04T01:33:43.528Z [INFO]  server-bf0352c1-ac1b-987c-66db-cddf874d8a0b: entering follower state: follower="Node at bf0352c1-ac1b-987c-66db-cddf874d8a0b [Follower]" leader-address= leader-id=
2024-12-04T01:33:43.780Z [INFO]  server-83f26e72-b4af-a421-c5d3-e4ad592a7a35: aborting pipeline replication: peer="{Voter server-e430f761-28ad-995a-c437-e0df8a2989ce e430f761-28ad-995a-c437-e0df8a2989ce}"
2024-12-04T01:33:44.194Z [ERROR] server-83f26e72-b4af-a421-c5d3-e4ad592a7a35: failed to pipeline appendEntries: peer="{Voter server-bf0352c1-ac1b-987c-66db-cddf874d8a0b bf0352c1-ac1b-987c-66db-cddf874d8a0b}" error="command enqueue timeout"
2024-12-04T01:33:44.194Z [INFO]  server-83f26e72-b4af-a421-c5d3-e4ad592a7a35: aborting pipeline replication: peer="{Voter server-bf0352c1-ac1b-987c-66db-cddf874d8a0b bf0352c1-ac1b-987c-66db-cddf874d8a0b}"
2024-12-04T01:33:44.292Z [ERROR] server-83f26e72-b4af-a421-c5d3-e4ad592a7a35: failed to heartbeat to: peer=bf0352c1-ac1b-987c-66db-cddf874d8a0b backoff time=10ms error="command timed out"
--- PASS: TestRaft_FollowerRemovalNoElection (1.73s)
=== RUN   TestRaft_VoteWithNoIDNoAddr
2024-12-04T01:33:44.293Z [INFO]  server-792cb120-2830-85de-c39a-8ff803acb90a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-792cb120-2830-85de-c39a-8ff803acb90a Address:792cb120-2830-85de-c39a-8ff803acb90a} {Suffrage:Voter ID:server-ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48 Address:ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48} {Suffrage:Voter ID:server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f Address:f557cf4c-ea51-7239-3c16-3e7f9fa6e72f}]"
2024-12-04T01:33:44.294Z [INFO]  server-ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-792cb120-2830-85de-c39a-8ff803acb90a Address:792cb120-2830-85de-c39a-8ff803acb90a} {Suffrage:Voter ID:server-ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48 Address:ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48} {Suffrage:Voter ID:server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f Address:f557cf4c-ea51-7239-3c16-3e7f9fa6e72f}]"
2024-12-04T01:33:44.294Z [INFO]  server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-792cb120-2830-85de-c39a-8ff803acb90a Address:792cb120-2830-85de-c39a-8ff803acb90a} {Suffrage:Voter ID:server-ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48 Address:ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48} {Suffrage:Voter ID:server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f Address:f557cf4c-ea51-7239-3c16-3e7f9fa6e72f}]"
2024-12-04T01:33:44.294Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:44.294Z [INFO]  server-792cb120-2830-85de-c39a-8ff803acb90a: entering follower state: follower="Node at 792cb120-2830-85de-c39a-8ff803acb90a [Follower]" leader-address= leader-id=
2024-12-04T01:33:44.294Z [INFO]  server-ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48: entering follower state: follower="Node at ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48 [Follower]" leader-address= leader-id=
2024-12-04T01:33:44.294Z [INFO]  server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: entering follower state: follower="Node at f557cf4c-ea51-7239-3c16-3e7f9fa6e72f [Follower]" leader-address= leader-id=
2024-12-04T01:33:44.350Z [WARN]  server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-12-04T01:33:44.350Z [INFO]  server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: entering candidate state: node="Node at f557cf4c-ea51-7239-3c16-3e7f9fa6e72f [Candidate]" term=2
2024-12-04T01:33:44.351Z [INFO]  server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: election won: term=2 tally=2
2024-12-04T01:33:44.351Z [INFO]  server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: entering leader state: leader="Node at f557cf4c-ea51-7239-3c16-3e7f9fa6e72f [Leader]"
2024-12-04T01:33:44.351Z [INFO]  server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: added peer, starting replication: peer=server-792cb120-2830-85de-c39a-8ff803acb90a
2024-12-04T01:33:44.351Z [INFO]  server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: added peer, starting replication: peer=server-ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48
2024-12-04T01:33:44.351Z [INFO]  server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: pipelining replication: peer="{Voter server-ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48 ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48}"
2024-12-04T01:33:44.352Z [INFO]  server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: pipelining replication: peer="{Voter server-792cb120-2830-85de-c39a-8ff803acb90a 792cb120-2830-85de-c39a-8ff803acb90a}"
2024-12-04T01:33:44.458Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:44.352271423 +0000 UTC m=+115.184114050 (1 nodes), 57.986505ms from start of poll, 58.501925ms from cluster start. Timeout at 2024-12-04 01:33:44.457331413 +0000 UTC m=+115.289174048, 105.059998ms after stability
2024-12-04T01:33:44.458Z [INFO]  cluster: starting stability test: raft-state=Leader
2024-12-04T01:33:44.574Z [INFO]  cluster: stable state for Leader reached at 2024-12-04 01:33:44.458824562 +0000 UTC m=+115.290667224 (1 nodes), 2.357µs from start of poll, 165.055099ms from cluster start. Timeout at 2024-12-04 01:33:44.563895304 +0000 UTC m=+115.395737996, 105.070772ms after stability
2024-12-04T01:33:44.579Z [INFO]  cluster: starting stability test: raft-state=Follower
2024-12-04T01:33:44.686Z [INFO]  cluster: stable state for Follower reached at 2024-12-04 01:33:44.579596678 +0000 UTC m=+115.411439248 (2 nodes), 2.07µs from start of poll, 285.827123ms from cluster start. Timeout at 2024-12-04 01:33:44.684683695 +0000 UTC m=+115.516526579, 105.087331ms after stability
2024-12-04T01:33:44.686Z [ERROR] server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: failed to heartbeat to: peer=ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48 backoff time=10ms error="failed to connect to peer: ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48"
2024-12-04T01:33:44.687Z [ERROR] server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: failed to pipeline appendEntries: peer="{Voter server-ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48 ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48}" error="append pipeline closed"
2024-12-04T01:33:44.687Z [INFO]  server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: aborting pipeline replication: peer="{Voter server-ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48 ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48}"
2024-12-04T01:33:44.688Z [ERROR] server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: failed to pipeline appendEntries: peer="{Voter server-792cb120-2830-85de-c39a-8ff803acb90a 792cb120-2830-85de-c39a-8ff803acb90a}" error="append pipeline closed"
2024-12-04T01:33:44.688Z [INFO]  server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: aborting pipeline replication: peer="{Voter server-792cb120-2830-85de-c39a-8ff803acb90a 792cb120-2830-85de-c39a-8ff803acb90a}"
2024-12-04T01:33:44.689Z [ERROR] server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: failed to heartbeat to: peer=792cb120-2830-85de-c39a-8ff803acb90a backoff time=10ms error="failed to connect to peer: 792cb120-2830-85de-c39a-8ff803acb90a"
2024-12-04T01:33:44.694Z [ERROR] server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: failed to appendEntries to: peer="{Voter server-ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48 ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48}" error="failed to connect to peer: ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48"
2024-12-04T01:33:44.696Z [ERROR] server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: failed to appendEntries to: peer="{Voter server-792cb120-2830-85de-c39a-8ff803acb90a 792cb120-2830-85de-c39a-8ff803acb90a}" error="failed to connect to peer: 792cb120-2830-85de-c39a-8ff803acb90a"
2024-12-04T01:33:44.710Z [ERROR] server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: failed to heartbeat to: peer=ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48 backoff time=10ms error="failed to connect to peer: ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48"
2024-12-04T01:33:44.711Z [ERROR] server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: failed to heartbeat to: peer=792cb120-2830-85de-c39a-8ff803acb90a backoff time=10ms error="failed to connect to peer: 792cb120-2830-85de-c39a-8ff803acb90a"
2024-12-04T01:33:44.719Z [ERROR] server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: failed to appendEntries to: peer="{Voter server-ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48 ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48}" error="failed to connect to peer: ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48"
2024-12-04T01:33:44.720Z [ERROR] server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: failed to appendEntries to: peer="{Voter server-792cb120-2830-85de-c39a-8ff803acb90a 792cb120-2830-85de-c39a-8ff803acb90a}" error="failed to connect to peer: 792cb120-2830-85de-c39a-8ff803acb90a"
2024-12-04T01:33:44.727Z [ERROR] server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: failed to heartbeat to: peer=ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48 backoff time=10ms error="failed to connect to peer: ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48"
2024-12-04T01:33:44.728Z [ERROR] server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: failed to heartbeat to: peer=792cb120-2830-85de-c39a-8ff803acb90a backoff time=10ms error="failed to connect to peer: 792cb120-2830-85de-c39a-8ff803acb90a"
2024-12-04T01:33:44.735Z [WARN]  server-792cb120-2830-85de-c39a-8ff803acb90a: heartbeat timeout reached, starting election: last-leader-addr=f557cf4c-ea51-7239-3c16-3e7f9fa6e72f last-leader-id=server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f
2024-12-04T01:33:44.735Z [INFO]  server-792cb120-2830-85de-c39a-8ff803acb90a: entering candidate state: node="Node at 792cb120-2830-85de-c39a-8ff803acb90a [Candidate]" term=3
2024-12-04T01:33:44.735Z [ERROR] server-792cb120-2830-85de-c39a-8ff803acb90a: failed to make requestVote RPC: target="{Voter server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f f557cf4c-ea51-7239-3c16-3e7f9fa6e72f}" error="failed to connect to peer: f557cf4c-ea51-7239-3c16-3e7f9fa6e72f" term=3
2024-12-04T01:33:44.735Z [WARN]  server-ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48: rejecting vote request since we have a leader: from=792cb120-2830-85de-c39a-8ff803acb90a leader=f557cf4c-ea51-7239-3c16-3e7f9fa6e72f leader-id=server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f
2024-12-04T01:33:44.736Z [WARN]  server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: failed to contact: server-id=server-792cb120-2830-85de-c39a-8ff803acb90a time=55.577445ms
2024-12-04T01:33:44.736Z [WARN]  server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: failed to contact: server-id=server-ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48 time=56.883205ms
2024-12-04T01:33:44.736Z [WARN]  server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: failed to contact quorum of nodes, stepping down
2024-12-04T01:33:44.736Z [INFO]  server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: entering follower state: follower="Node at f557cf4c-ea51-7239-3c16-3e7f9fa6e72f [Follower]" leader-address= leader-id=
2024-12-04T01:33:44.737Z [ERROR] server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: failed to appendEntries to: peer="{Voter server-792cb120-2830-85de-c39a-8ff803acb90a 792cb120-2830-85de-c39a-8ff803acb90a}" error="failed to connect to peer: 792cb120-2830-85de-c39a-8ff803acb90a"
2024-12-04T01:33:44.743Z [ERROR] server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f: failed to appendEntries to: peer="{Voter server-ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48 ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48}" error="failed to connect to peer: ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48"
2024-12-04T01:33:44.783Z [WARN]  server-ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48: heartbeat timeout reached, starting election: last-leader-addr=f557cf4c-ea51-7239-3c16-3e7f9fa6e72f last-leader-id=server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f
2024-12-04T01:33:44.783Z [INFO]  server-ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48: entering candidate state: node="Node at ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48 [Candidate]" term=3
2024-12-04T01:33:44.783Z [ERROR] server-ddb17ea2-faf0-41c5-2dcc-46f0ad31dc48: failed to make requestVote RPC: target="{Voter server-f557cf4c-ea51-7239-3c16-3e7f9fa6e72f f557cf4c-ea51-7239-3c16-3e7f9fa6e72f}" error="failed to connect to peer: f557cf4c-ea51-7239-3c16-3e7f9fa6e72f" term=3
2024-12-04T01:33:44.784Z [INFO]  server-792cb120-2830-85de-c39a-8ff803acb90a: duplicate requestVote for same term: term=3
2024-12-04T01:33:44.791Z [INFO]  server-792cb120-2830-85de-c39a-8ff803acb90a: entering follower state: follower="Node at 792cb120-2830-85de-c39a-8ff803acb90a [Follower]" leader-address= leader-id=
--- PASS: TestRaft_VoteWithNoIDNoAddr (0.50s)
=== RUN   TestRaft_runFollower_State_Transition
=== RUN   TestRaft_runFollower_State_Transition/NonVoter
2024-12-04T01:33:44.793Z [INFO]  127.0.0.1:43877 :: starting node: addr=127.0.0.1:43877
2024-12-04T01:33:44.793Z [INFO]  127.0.0.1:43877 :: initial configuration: index=0 servers=[]
2024-12-04T01:33:44.793Z [INFO]  127.0.0.1:43877 :: entering follower state: follower="Node at 127.0.0.1:43877 [Follower]" leader-address= leader-id=
2024-12-04T01:33:44.910Z [WARN]  127.0.0.1:43877 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
=== RUN   TestRaft_runFollower_State_Transition/Voter
2024-12-04T01:33:44.947Z [INFO]  127.0.0.1:39057 :: starting node: addr=127.0.0.1:39057
2024-12-04T01:33:44.947Z [INFO]  127.0.0.1:39057 :: initial configuration: index=0 servers=[]
2024-12-04T01:33:44.947Z [INFO]  127.0.0.1:39057 :: entering follower state: follower="Node at 127.0.0.1:39057 [Follower]" leader-address= leader-id=
2024-12-04T01:33:45.011Z [WARN]  127.0.0.1:39057 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
=== RUN   TestRaft_runFollower_State_Transition/Not_in_Config
2024-12-04T01:33:45.105Z [INFO]  127.0.0.1:41395 :: starting node: addr=127.0.0.1:41395
2024-12-04T01:33:45.105Z [INFO]  127.0.0.1:41395 :: initial configuration: index=0 servers=[]
2024-12-04T01:33:45.105Z [INFO]  127.0.0.1:41395 :: entering follower state: follower="Node at 127.0.0.1:41395 [Follower]" leader-address= leader-id=
2024-12-04T01:33:45.198Z [WARN]  127.0.0.1:41395 :: heartbeat timeout reached, not part of a stable configuration or a non-voter, not triggering a leader election
--- PASS: TestRaft_runFollower_State_Transition (0.46s)
    --- PASS: TestRaft_runFollower_State_Transition/NonVoter (0.15s)
    --- PASS: TestRaft_runFollower_State_Transition/Voter (0.16s)
    --- PASS: TestRaft_runFollower_State_Transition/Not_in_Config (0.15s)
=== RUN   TestRaft_runFollower_ReloadTimeoutConfigs
2024-12-04T01:33:45.256Z [INFO]  127.0.0.1:40129 :: starting node: addr=127.0.0.1:40129
2024-12-04T01:33:45.257Z [INFO]  127.0.0.1:40129 :: initial configuration: index=0 servers=[]
2024-12-04T01:33:45.257Z [INFO]  127.0.0.1:40129 :: entering follower state: follower="Node at 127.0.0.1:40129 [Follower]" leader-address= leader-id=
2024-12-04T01:33:45.257Z [WARN]  127.0.0.1:40129 :: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
--- PASS: TestRaft_runFollower_ReloadTimeoutConfigs (0.15s)
=== RUN   TestSaturationMetric
=== RUN   TestSaturationMetric/without_smoothing
--- PASS: TestSaturationMetric (0.00s)
    --- PASS: TestSaturationMetric/without_smoothing (0.00s)
=== RUN   TestSaturationMetric_IncorrectUsage
=== RUN   TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_consecutively
=== RUN   TestSaturationMetric_IncorrectUsage/calling_working()_first
--- PASS: TestSaturationMetric_IncorrectUsage (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_sleeping()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_consecutively (0.00s)
    --- PASS: TestSaturationMetric_IncorrectUsage/calling_working()_first (0.00s)
=== RUN   TestTCPTransport_BadAddr
--- PASS: TestTCPTransport_BadAddr (0.00s)
=== RUN   TestTCPTransport_EmptyAddr
--- PASS: TestTCPTransport_EmptyAddr (0.00s)
=== RUN   TestTCPTransport_WithAdvertise
--- PASS: TestTCPTransport_WithAdvertise (0.00s)
=== RUN   TestTransport_StartStop
--- PASS: TestTransport_StartStop (0.00s)
=== RUN   TestTransport_AppendEntries
--- PASS: TestTransport_AppendEntries (0.00s)
=== RUN   TestTransport_AppendEntriesPipeline
--- PASS: TestTransport_AppendEntriesPipeline (0.00s)
=== RUN   TestTransport_RequestVote
--- PASS: TestTransport_RequestVote (0.00s)
=== RUN   TestTransport_InstallSnapshot
--- PASS: TestTransport_InstallSnapshot (0.00s)
=== RUN   TestTransport_EncodeDecode
--- PASS: TestTransport_EncodeDecode (0.00s)
=== RUN   TestRandomTimeout
--- PASS: TestRandomTimeout (0.00s)
=== RUN   TestNewSeed
--- PASS: TestNewSeed (0.01s)
=== RUN   TestRandomTimeout_NoTime
--- PASS: TestRandomTimeout_NoTime (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
=== RUN   TestMax
--- PASS: TestMax (0.00s)
=== RUN   TestGenerateUUID
--- PASS: TestGenerateUUID (0.00s)
=== RUN   TestBackoff
--- PASS: TestBackoff (0.00s)
=== RUN   TestOverrideNotifyBool
--- PASS: TestOverrideNotifyBool (0.00s)
PASS
ok  	github.com/hashicorp/raft	116.273s
make[1]: Leaving directory '/build/reproducible-path/golang-github-hashicorp-raft-1.5.0'
   create-stamp debian/debhelper-build-stamp
   dh_testroot -O--buildsystem=golang
   dh_prep -O--buildsystem=golang
   dh_auto_install --destdir=debian/golang-github-hashicorp-raft-dev/ -O--buildsystem=golang
   dh_installdocs -O--buildsystem=golang
   dh_installchangelogs -O--buildsystem=golang
   dh_installsystemduser -O--buildsystem=golang
   dh_perl -O--buildsystem=golang
   dh_link -O--buildsystem=golang
   dh_strip_nondeterminism -O--buildsystem=golang
   dh_compress -O--buildsystem=golang
   dh_fixperms -O--buildsystem=golang
   dh_missing -O--buildsystem=golang
   dh_installdeb -O--buildsystem=golang
   dh_golang -O--buildsystem=golang
   dh_gencontrol -O--buildsystem=golang
   dh_md5sums -O--buildsystem=golang
   dh_builddeb -O--buildsystem=golang
dpkg-deb: building package 'golang-github-hashicorp-raft-dev' in '../golang-github-hashicorp-raft-dev_1.5.0-1_all.deb'.
 dpkg-genbuildinfo --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_i386.buildinfo
 dpkg-genchanges --build=binary -O../golang-github-hashicorp-raft_1.5.0-1_i386.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-buildpackage: info: binary-only upload (no source included)
dpkg-genchanges: info: including full source code in upload
I: copying local configuration
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: cleaning the build env 
I: removing directory /srv/workspace/pbuilder/25276 and its subdirectories
I: Current time: Tue Dec  3 13:33:55 -12 2024
I: pbuilder-time-stamp: 1733276035